2024-03-18T18:11:07.120 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-18T18:11:07.120 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606246 --verbose --name yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} -- /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606246/orig.config.yaml 2024-03-18T18:11:07.165 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T18:11:07.246 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606246 branch: wip-yuri10-testing-2024-03-15-1653-reef description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} email: yweinste@redhat.com first_in_suite: false job_id: '7606246' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri10-testing-2024-03-15-1653-reef ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 49bdc51142759622a281d3e5327eebd07cbb8bb3 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 49bdc51142759622a281d3e5327eebd07cbb8bb3 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri10-testing-2024-03-15-1653-reef sha1: 49bdc51142759622a281d3e5327eebd07cbb8bb3 owner: scheduled_yuriw@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 8313 sha1: 49bdc51142759622a281d3e5327eebd07cbb8bb3 sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri10-testing-2024-03-15-1653-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_49bdc51142759622a281d3e5327eebd07cbb8bb3/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 49bdc51142759622a281d3e5327eebd07cbb8bb3 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi001.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAG5rtJ0ZhNMR5UGSUXb7UXuV0i2c8I2qy7VHzgQ9LhnisB7m9uAD+tqlNDM5PLIXclL2I2FcSts8ljJ4fNGHP4= smithi165.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKlrV/zKFmHgntFhZLft2rs+fc929sATSRWOM5Wdio8Ows/4qHxwsjOObNBEEGsHqIpnPNQ3+d0ferKQgPJGLZ4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-16_15:03:17 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-18T18:11:07.246 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_49bdc51142759622a281d3e5327eebd07cbb8bb3/qa; will attempt to use it 2024-03-18T18:11:07.247 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_49bdc51142759622a281d3e5327eebd07cbb8bb3/qa/tasks 2024-03-18T18:11:07.247 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-18T18:11:07.249 INFO:teuthology.task.internal:Checking packages... 2024-03-18T18:11:07.273 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '49bdc51142759622a281d3e5327eebd07cbb8bb3' 2024-03-18T18:11:07.273 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-18T18:11:07.273 INFO:teuthology.packaging:ref: None 2024-03-18T18:11:07.273 INFO:teuthology.packaging:tag: None 2024-03-18T18:11:07.273 INFO:teuthology.packaging:branch: wip-yuri10-testing-2024-03-15-1653-reef 2024-03-18T18:11:07.273 INFO:teuthology.packaging:sha1: 49bdc51142759622a281d3e5327eebd07cbb8bb3 2024-03-18T18:11:07.273 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri10-testing-2024-03-15-1653-reef 2024-03-18T18:11:07.417 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-914.g49bdc511 2024-03-18T18:11:07.419 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-18T18:11:07.425 INFO:teuthology.task.internal:no buildpackages task found 2024-03-18T18:11:07.425 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-18T18:11:07.446 INFO:teuthology.task.internal:Saving configuration 2024-03-18T18:11:07.462 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-18T18:11:07.468 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T18:11:07.490 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606246', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 18:06:01.370357', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAG5rtJ0ZhNMR5UGSUXb7UXuV0i2c8I2qy7VHzgQ9LhnisB7m9uAD+tqlNDM5PLIXclL2I2FcSts8ljJ4fNGHP4='} 2024-03-18T18:11:07.513 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606246', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 18:06:01.368621', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKlrV/zKFmHgntFhZLft2rs+fc929sATSRWOM5Wdio8Ows/4qHxwsjOObNBEEGsHqIpnPNQ3+d0ferKQgPJGLZ4='} 2024-03-18T18:11:07.513 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-18T18:11:07.519 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-18T18:11:07.519 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-18T18:11:07.520 INFO:teuthology.run_tasks:Running task console_log... 2024-03-18T18:11:07.600 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f66c4465160>, signals=[15]) 2024-03-18T18:11:07.601 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-18T18:11:07.607 INFO:teuthology.task.internal:Opening connections... 2024-03-18T18:11:07.607 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T18:11:07.609 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T18:11:07.682 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T18:11:07.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T18:11:07.755 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-18T18:11:07.761 DEBUG:teuthology.orchestra.run.smithi001:> uname -m 2024-03-18T18:11:07.784 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64 2024-03-18T18:11:07.784 DEBUG:teuthology.orchestra.run.smithi001:> cat /etc/os-release 2024-03-18T18:11:07.845 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="CentOS Stream" 2024-03-18T18:11:07.845 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="8" 2024-03-18T18:11:07.845 INFO:teuthology.orchestra.run.smithi001.stdout:ID="centos" 2024-03-18T18:11:07.845 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE="rhel fedora" 2024-03-18T18:11:07.845 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="8" 2024-03-18T18:11:07.846 INFO:teuthology.orchestra.run.smithi001.stdout:PLATFORM_ID="platform:el8" 2024-03-18T18:11:07.846 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-18T18:11:07.846 INFO:teuthology.orchestra.run.smithi001.stdout:ANSI_COLOR="0;31" 2024-03-18T18:11:07.846 INFO:teuthology.orchestra.run.smithi001.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-18T18:11:07.846 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://centos.org/" 2024-03-18T18:11:07.846 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T18:11:07.846 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-18T18:11:07.846 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T18:11:07.847 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2024-03-18T18:11:07.871 DEBUG:teuthology.orchestra.run.smithi165:> uname -m 2024-03-18T18:11:07.893 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64 2024-03-18T18:11:07.893 DEBUG:teuthology.orchestra.run.smithi165:> cat /etc/os-release 2024-03-18T18:11:07.954 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="CentOS Stream" 2024-03-18T18:11:07.954 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="8" 2024-03-18T18:11:07.954 INFO:teuthology.orchestra.run.smithi165.stdout:ID="centos" 2024-03-18T18:11:07.954 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE="rhel fedora" 2024-03-18T18:11:07.954 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="8" 2024-03-18T18:11:07.954 INFO:teuthology.orchestra.run.smithi165.stdout:PLATFORM_ID="platform:el8" 2024-03-18T18:11:07.954 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-18T18:11:07.955 INFO:teuthology.orchestra.run.smithi165.stdout:ANSI_COLOR="0;31" 2024-03-18T18:11:07.955 INFO:teuthology.orchestra.run.smithi165.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-18T18:11:07.955 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://centos.org/" 2024-03-18T18:11:07.955 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T18:11:07.955 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-18T18:11:07.955 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T18:11:07.955 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server 2024-03-18T18:11:07.978 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-18T18:11:07.988 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-18T18:11:08.070 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-18T18:11:08.070 DEBUG:teuthology.orchestra.run.smithi001:> test '!' -e /home/ubuntu/cephtest 2024-03-18T18:11:08.073 DEBUG:teuthology.orchestra.run.smithi165:> test '!' -e /home/ubuntu/cephtest 2024-03-18T18:11:08.094 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-18T18:11:08.100 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-18T18:11:08.100 DEBUG:teuthology.orchestra.run.smithi001:> test -z $(ls -A /var/lib/ceph) 2024-03-18T18:11:08.134 DEBUG:teuthology.orchestra.run.smithi165:> test -z $(ls -A /var/lib/ceph) 2024-03-18T18:11:08.173 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-18T18:11:08.230 INFO:teuthology.run_tasks:Running task kernel... 2024-03-18T18:11:08.244 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-18T18:11:08.244 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-18T18:11:08.244 DEBUG:teuthology.orchestra.run.smithi001:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T18:11:08.245 DEBUG:teuthology.orchestra.run.smithi165:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T18:11:08.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T18:11:08.265 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-03-18T18:11:08.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T18:11:08.266 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-03-18T18:11:08.326 INFO:teuthology.orchestra.run.smithi165.stdout:4.18.0-489.el8.x86_64 2024-03-18T18:11:08.326 INFO:teuthology.task.kernel:Running kernel on smithi165: 4.18.0-489.el8.x86_64 2024-03-18T18:11:08.326 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum install -y kernel 2024-03-18T18:11:08.329 INFO:teuthology.orchestra.run.smithi001.stdout:4.18.0-489.el8.x86_64 2024-03-18T18:11:08.329 INFO:teuthology.task.kernel:Running kernel on smithi001: 4.18.0-489.el8.x86_64 2024-03-18T18:11:08.330 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum install -y kernel 2024-03-18T18:11:13.346 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 8 - AppStream 11 MB/s | 28 MB 00:02 2024-03-18T18:11:15.019 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 8 - AppStream 6.7 MB/s | 28 MB 00:04 2024-03-18T18:19:41.384 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 8 - BaseOS 21 kB/s | 10 MB 08:21 2024-03-18T18:19:45.442 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 8 - Extras 7.3 kB/s | 18 kB 00:02 2024-03-18T18:19:45.863 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 8 - Extras common packages 24 kB/s | 7.5 kB 00:00 2024-03-18T18:19:46.102 INFO:teuthology.orchestra.run.smithi165.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-03-18T18:19:46.953 INFO:teuthology.orchestra.run.smithi165.stdout:Extra Packages for Enterprise Linux 21 MB/s | 16 MB 00:00 2024-03-18T18:19:50.703 INFO:teuthology.orchestra.run.smithi165.stdout:lab-extras 560 kB/s | 24 kB 00:00 2024-03-18T18:19:53.658 INFO:teuthology.orchestra.run.smithi165.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-03-18T18:19:53.658 INFO:teuthology.orchestra.run.smithi165.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-03-18T18:19:53.784 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-03-18T18:19:53.784 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-03-18T18:19:53.784 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-03-18T18:19:53.785 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-03-18T18:19:53.785 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-03-18T18:19:53.785 INFO:teuthology.orchestra.run.smithi165.stdout: kernel x86_64 4.18.0-544.el8 baseos 10 M 2024-03-18T18:19:53.785 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-03-18T18:19:53.785 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-core x86_64 4.18.0-544.el8 baseos 44 M 2024-03-18T18:19:53.785 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules x86_64 4.18.0-544.el8 baseos 36 M 2024-03-18T18:19:53.785 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:19:53.785 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-03-18T18:19:53.785 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-03-18T18:19:53.785 INFO:teuthology.orchestra.run.smithi165.stdout:Install 3 Packages 2024-03-18T18:19:53.785 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:19:53.786 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 90 M 2024-03-18T18:19:53.786 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 97 M 2024-03-18T18:19:53.786 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-03-18T18:24:16.221 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 8 - BaseOS 14 kB/s | 10 MB 12:57 2024-03-18T18:24:18.537 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 8 - Extras 30 kB/s | 18 kB 00:00 2024-03-18T18:24:19.128 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 8 - Extras common packages 16 kB/s | 7.5 kB 00:00 2024-03-18T18:24:19.381 INFO:teuthology.orchestra.run.smithi001.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-03-18T18:24:22.391 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 5.6 MB/s | 16 MB 00:02 2024-03-18T18:24:26.340 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 429 kB/s | 24 kB 00:00 2024-03-18T18:24:29.593 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-03-18T18:24:29.593 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-03-18T18:24:29.712 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-18T18:24:29.713 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-18T18:24:29.713 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-03-18T18:24:29.713 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-18T18:24:29.713 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-18T18:24:29.713 INFO:teuthology.orchestra.run.smithi001.stdout: kernel x86_64 4.18.0-544.el8 baseos 10 M 2024-03-18T18:24:29.713 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-03-18T18:24:29.713 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-core x86_64 4.18.0-544.el8 baseos 44 M 2024-03-18T18:24:29.713 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules x86_64 4.18.0-544.el8 baseos 36 M 2024-03-18T18:24:29.713 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:24:29.713 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-18T18:24:29.713 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-18T18:24:29.714 INFO:teuthology.orchestra.run.smithi001.stdout:Install 3 Packages 2024-03-18T18:24:29.714 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:24:29.714 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 90 M 2024-03-18T18:24:29.714 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 97 M 2024-03-18T18:24:29.714 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-18T18:27:27.571 INFO:teuthology.orchestra.run.smithi165.stdout:[MIRROR] kernel-modules-4.18.0-544.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.raystedman.org/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-modules-4.18.0-544.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2024-03-18T18:27:29.478 INFO:teuthology.orchestra.run.smithi165.stdout:(1/3): kernel-modules-4.18.0-544.el8.x86_64.rpm 82 kB/s | 36 MB 07:35 2024-03-18T18:31:20.557 INFO:teuthology.orchestra.run.smithi165.stdout:(2/3): kernel-4.18.0-544.el8.x86_64.rpm 15 kB/s | 10 MB 11:26 2024-03-18T18:32:30.274 INFO:teuthology.orchestra.run.smithi001.stdout:(1/3): kernel-4.18.0-544.el8.x86_64.rpm 22 kB/s | 10 MB 08:00 2024-03-18T18:36:43.122 INFO:teuthology.orchestra.run.smithi001.stdout:(2/3): kernel-modules-4.18.0-544.el8.x86_64.rpm 51 kB/s | 36 MB 12:13 2024-03-18T18:36:55.101 INFO:teuthology.orchestra.run.smithi165.stdout:(3/3): kernel-core-4.18.0-544.el8.x86_64.rpm 44 kB/s | 44 MB 17:01 2024-03-18T18:36:55.101 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-03-18T18:36:55.101 INFO:teuthology.orchestra.run.smithi165.stdout:Total 91 kB/s | 90 MB 17:01 2024-03-18T18:36:55.777 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-03-18T18:36:56.211 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-03-18T18:36:56.211 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-03-18T18:36:57.471 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-03-18T18:36:57.501 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-03-18T18:37:00.293 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-03-18T18:37:00.810 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-core-4.18.0-544.el8.x86_64 1/3 2024-03-18T18:37:04.410 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-core-4.18.0-544.el8.x86_64 1/3 2024-03-18T18:37:04.843 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-modules-4.18.0-544.el8.x86_64 2/3 2024-03-18T18:37:08.282 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-4.18.0-544.el8.x86_64 2/3 2024-03-18T18:37:08.600 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-4.18.0-544.el8.x86_64 3/3 2024-03-18T18:37:12.860 INFO:teuthology.orchestra.run.smithi001.stdout:(3/3): kernel-core-4.18.0-544.el8.x86_64.rpm 59 kB/s | 44 MB 12:42 2024-03-18T18:37:12.860 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-18T18:37:12.860 INFO:teuthology.orchestra.run.smithi001.stdout:Total 121 kB/s | 90 MB 12:43 2024-03-18T18:37:13.575 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-18T18:37:14.035 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-18T18:37:14.035 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-18T18:37:15.343 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-18T18:37:15.382 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-18T18:37:18.330 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-18T18:37:18.849 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-core-4.18.0-544.el8.x86_64 1/3 2024-03-18T18:37:22.557 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-4.18.0-544.el8.x86_64 1/3 2024-03-18T18:37:23.032 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-modules-4.18.0-544.el8.x86_64 2/3 2024-03-18T18:37:26.719 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-4.18.0-544.el8.x86_64 2/3 2024-03-18T18:37:27.001 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-4.18.0-544.el8.x86_64 3/3 2024-03-18T18:37:30.632 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-core-4.18.0-544.el8.x86_64 3/3 2024-03-18T18:37:30.654 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-4.18.0-544.el8.x86_64 3/3 2024-03-18T18:37:30.988 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-4.18.0-544.el8.x86_64 3/3 2024-03-18T18:37:30.988 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-4.18.0-544.el8.x86_64 1/3 2024-03-18T18:37:30.988 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-core-4.18.0-544.el8.x86_64 2/3 2024-03-18T18:37:31.519 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-modules-4.18.0-544.el8.x86_64 3/3 2024-03-18T18:37:31.519 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:37:31.519 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-03-18T18:37:31.519 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-4.18.0-544.el8.x86_64 kernel-core-4.18.0-544.el8.x86_64 2024-03-18T18:37:31.519 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules-4.18.0-544.el8.x86_64 2024-03-18T18:37:31.519 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:37:31.519 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-03-18T18:37:31.726 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T18:37:31.939 INFO:teuthology.orchestra.run.smithi165.stdout:kernel-4.18.0-544.el8.x86_64 2024-03-18T18:37:31.940 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-544.el8.x86_64 2024-03-18T18:37:31.940 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-544.el8.x86_64 2024-03-18T18:37:31.940 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-18T18:37:31.940 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-18T18:37:31.940 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-18T18:37:31.940 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi165.front.sepia.ceph.com, path=None, version=distro) 2024-03-18T18:37:31.941 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T18:37:32.152 INFO:teuthology.orchestra.run.smithi165.stdout:kernel-4.18.0-544.el8.x86_64 2024-03-18T18:37:32.152 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-544.el8.x86_64 2024-03-18T18:37:32.153 DEBUG:teuthology.orchestra.run.smithi165:> sudo rpm -qi grub2-tools 2024-03-18T18:37:32.187 INFO:teuthology.orchestra.run.smithi165.stdout:Name : grub2-tools 2024-03-18T18:37:32.187 INFO:teuthology.orchestra.run.smithi165.stdout:Epoch : 1 2024-03-18T18:37:32.187 INFO:teuthology.orchestra.run.smithi165.stdout:Version : 2.02 2024-03-18T18:37:32.187 INFO:teuthology.orchestra.run.smithi165.stdout:Release : 129.el8 2024-03-18T18:37:32.187 INFO:teuthology.orchestra.run.smithi165.stdout:Architecture: x86_64 2024-03-18T18:37:32.187 INFO:teuthology.orchestra.run.smithi165.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-03-18T18:37:32.187 INFO:teuthology.orchestra.run.smithi165.stdout:Group : System Environment/Base 2024-03-18T18:37:32.187 INFO:teuthology.orchestra.run.smithi165.stdout:Size : 9265586 2024-03-18T18:37:32.187 INFO:teuthology.orchestra.run.smithi165.stdout:License : GPLv3+ 2024-03-18T18:37:32.187 INFO:teuthology.orchestra.run.smithi165.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-03-18T18:37:32.188 INFO:teuthology.orchestra.run.smithi165.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-03-18T18:37:32.188 INFO:teuthology.orchestra.run.smithi165.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-03-18T18:37:32.188 INFO:teuthology.orchestra.run.smithi165.stdout:Build Host : kbuilder.bsys.centos.org 2024-03-18T18:37:32.188 INFO:teuthology.orchestra.run.smithi165.stdout:Relocations : (not relocatable) 2024-03-18T18:37:32.188 INFO:teuthology.orchestra.run.smithi165.stdout:Packager : CentOS BuildSystem 2024-03-18T18:37:32.188 INFO:teuthology.orchestra.run.smithi165.stdout:Vendor : CentOS 2024-03-18T18:37:32.188 INFO:teuthology.orchestra.run.smithi165.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-18T18:37:32.188 INFO:teuthology.orchestra.run.smithi165.stdout:Summary : Support tools for GRUB. 2024-03-18T18:37:32.188 INFO:teuthology.orchestra.run.smithi165.stdout:Description : 2024-03-18T18:37:32.188 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:37:32.188 INFO:teuthology.orchestra.run.smithi165.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-18T18:37:32.188 INFO:teuthology.orchestra.run.smithi165.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-18T18:37:32.188 INFO:teuthology.orchestra.run.smithi165.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-18T18:37:32.188 INFO:teuthology.orchestra.run.smithi165.stdout:hardware devices. 2024-03-18T18:37:32.188 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:37:32.189 INFO:teuthology.orchestra.run.smithi165.stdout:This subpackage provides tools for support of all platforms. 2024-03-18T18:37:32.189 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-18T18:37:32.190 INFO:teuthology.task.kernel:Updating grub on smithi165 to boot 4.18.0-544.el8.x86_64 2024-03-18T18:37:32.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-18T18:37:33.982 INFO:teuthology.orchestra.run.smithi165.stderr:Generating grub configuration file ... 2024-03-18T18:37:36.269 INFO:teuthology.orchestra.run.smithi165.stderr:done 2024-03-18T18:37:36.270 DEBUG:teuthology.orchestra.run.smithi165:> mktemp 2024-03-18T18:37:36.293 INFO:teuthology.orchestra.run.smithi165.stdout:/tmp/tmp.uSHNd5BCIe 2024-03-18T18:37:36.294 DEBUG:teuthology.orchestra.run.smithi165:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.uSHNd5BCIe 2024-03-18T18:37:36.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0666 /tmp/tmp.uSHNd5BCIe 2024-03-18T18:37:36.509 DEBUG:teuthology.orchestra.remote:smithi165:/tmp/tmp.uSHNd5BCIe is 6KB 2024-03-18T18:37:36.557 DEBUG:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.uSHNd5BCIe 2024-03-18T18:37:36.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo /bin/ls /boot/loader/entries || true 2024-03-18T18:37:36.639 INFO:teuthology.orchestra.run.smithi165.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-03-18T18:37:36.639 INFO:teuthology.orchestra.run.smithi165.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-03-18T18:37:36.640 INFO:teuthology.orchestra.run.smithi165.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-03-18T18:37:36.640 INFO:teuthology.orchestra.run.smithi165.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-544.el8.x86_64.conf 2024-03-18T18:37:36.641 DEBUG:teuthology.orchestra.run.smithi165:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-544.el8.x86_64 2024-03-18T18:37:36.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo shutdown -r now 2024-03-18T18:37:50.465 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-4.18.0-544.el8.x86_64 3/3 2024-03-18T18:37:50.489 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-4.18.0-544.el8.x86_64 3/3 2024-03-18T18:37:50.875 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-4.18.0-544.el8.x86_64 3/3 2024-03-18T18:37:50.875 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-4.18.0-544.el8.x86_64 1/3 2024-03-18T18:37:50.875 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-core-4.18.0-544.el8.x86_64 2/3 2024-03-18T18:37:51.394 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-modules-4.18.0-544.el8.x86_64 3/3 2024-03-18T18:37:51.395 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:37:51.395 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-18T18:37:51.395 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-4.18.0-544.el8.x86_64 kernel-core-4.18.0-544.el8.x86_64 2024-03-18T18:37:51.395 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-4.18.0-544.el8.x86_64 2024-03-18T18:37:51.395 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:37:51.395 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-18T18:37:51.549 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T18:37:51.780 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-4.18.0-544.el8.x86_64 2024-03-18T18:37:51.781 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-544.el8.x86_64 2024-03-18T18:37:51.781 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-544.el8.x86_64 2024-03-18T18:37:51.781 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-18T18:37:51.781 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-18T18:37:51.781 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-18T18:37:51.781 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi001.front.sepia.ceph.com, path=None, version=distro) 2024-03-18T18:37:51.781 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T18:37:52.002 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-4.18.0-544.el8.x86_64 2024-03-18T18:37:52.002 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-544.el8.x86_64 2024-03-18T18:37:52.002 DEBUG:teuthology.orchestra.run.smithi001:> sudo rpm -qi grub2-tools 2024-03-18T18:37:52.037 INFO:teuthology.orchestra.run.smithi001.stdout:Name : grub2-tools 2024-03-18T18:37:52.037 INFO:teuthology.orchestra.run.smithi001.stdout:Epoch : 1 2024-03-18T18:37:52.037 INFO:teuthology.orchestra.run.smithi001.stdout:Version : 2.02 2024-03-18T18:37:52.037 INFO:teuthology.orchestra.run.smithi001.stdout:Release : 129.el8 2024-03-18T18:37:52.037 INFO:teuthology.orchestra.run.smithi001.stdout:Architecture: x86_64 2024-03-18T18:37:52.037 INFO:teuthology.orchestra.run.smithi001.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-03-18T18:37:52.037 INFO:teuthology.orchestra.run.smithi001.stdout:Group : System Environment/Base 2024-03-18T18:37:52.037 INFO:teuthology.orchestra.run.smithi001.stdout:Size : 9265586 2024-03-18T18:37:52.037 INFO:teuthology.orchestra.run.smithi001.stdout:License : GPLv3+ 2024-03-18T18:37:52.037 INFO:teuthology.orchestra.run.smithi001.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-03-18T18:37:52.037 INFO:teuthology.orchestra.run.smithi001.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-03-18T18:37:52.038 INFO:teuthology.orchestra.run.smithi001.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-03-18T18:37:52.038 INFO:teuthology.orchestra.run.smithi001.stdout:Build Host : kbuilder.bsys.centos.org 2024-03-18T18:37:52.038 INFO:teuthology.orchestra.run.smithi001.stdout:Relocations : (not relocatable) 2024-03-18T18:37:52.038 INFO:teuthology.orchestra.run.smithi001.stdout:Packager : CentOS BuildSystem 2024-03-18T18:37:52.038 INFO:teuthology.orchestra.run.smithi001.stdout:Vendor : CentOS 2024-03-18T18:37:52.038 INFO:teuthology.orchestra.run.smithi001.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-18T18:37:52.038 INFO:teuthology.orchestra.run.smithi001.stdout:Summary : Support tools for GRUB. 2024-03-18T18:37:52.038 INFO:teuthology.orchestra.run.smithi001.stdout:Description : 2024-03-18T18:37:52.038 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:37:52.038 INFO:teuthology.orchestra.run.smithi001.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-18T18:37:52.038 INFO:teuthology.orchestra.run.smithi001.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-18T18:37:52.038 INFO:teuthology.orchestra.run.smithi001.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-18T18:37:52.038 INFO:teuthology.orchestra.run.smithi001.stdout:hardware devices. 2024-03-18T18:37:52.039 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:37:52.039 INFO:teuthology.orchestra.run.smithi001.stdout:This subpackage provides tools for support of all platforms. 2024-03-18T18:37:52.039 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-18T18:37:52.039 INFO:teuthology.task.kernel:Updating grub on smithi001 to boot 4.18.0-544.el8.x86_64 2024-03-18T18:37:52.040 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-18T18:37:53.467 INFO:teuthology.orchestra.run.smithi001.stderr:Generating grub configuration file ... 2024-03-18T18:37:55.697 INFO:teuthology.orchestra.run.smithi001.stderr:done 2024-03-18T18:37:55.698 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-03-18T18:37:55.731 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.TBQYvsbaUf 2024-03-18T18:37:55.732 DEBUG:teuthology.orchestra.run.smithi001:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TBQYvsbaUf 2024-03-18T18:37:55.804 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.TBQYvsbaUf 2024-03-18T18:37:55.955 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.TBQYvsbaUf is 6KB 2024-03-18T18:37:56.003 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.TBQYvsbaUf 2024-03-18T18:37:56.021 DEBUG:teuthology.orchestra.run.smithi001:> sudo /bin/ls /boot/loader/entries || true 2024-03-18T18:37:56.139 INFO:teuthology.orchestra.run.smithi001.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-03-18T18:37:56.139 INFO:teuthology.orchestra.run.smithi001.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-03-18T18:37:56.140 INFO:teuthology.orchestra.run.smithi001.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-03-18T18:37:56.140 INFO:teuthology.orchestra.run.smithi001.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-544.el8.x86_64.conf 2024-03-18T18:37:56.141 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-544.el8.x86_64 2024-03-18T18:37:56.263 DEBUG:teuthology.orchestra.run.smithi001:> sudo shutdown -r now 2024-03-18T18:38:06.820 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-18T18:38:06.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-18T18:38:06.821 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T18:38:25.311 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2024-03-18T18:38:26.267 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-18T18:38:26.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-18T18:38:26.268 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T18:38:34.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-18T18:38:34.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T18:38:37.374 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2024-03-18T18:38:44.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.1 2024-03-18T18:38:49.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-18T18:38:49.379 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T18:38:53.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-18T18:38:53.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T18:39:49.436 DEBUG:teuthology.orchestra.remote:timed out 2024-03-18T18:39:53.777 DEBUG:teuthology.orchestra.remote:timed out 2024-03-18T18:40:04.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-18T18:40:04.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T18:40:04.847 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-03-18T18:40:05.126 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-18T18:40:05.126 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-544.el8.x86_64"... 2024-03-18T18:40:05.126 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-03-18T18:40:05.145 INFO:teuthology.orchestra.run.smithi165.stdout:4.18.0-544.el8.x86_64 2024-03-18T18:40:05.145 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-544.el8.x86_64 vs 4.18.0-544.el8.x86_64 2024-03-18T18:40:05.145 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-18T18:40:05.146 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-18T18:40:05.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-18T18:40:05.779 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T18:40:06.092 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-03-18T18:40:06.146 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-18T18:40:06.147 DEBUG:teuthology.orchestra.run.smithi165:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-18T18:40:06.239 INFO:teuthology.orchestra.run.smithi165.stdout:ttyS1 2024-03-18T18:40:06.253 DEBUG:teuthology.parallel:result is None 2024-03-18T18:40:06.395 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-18T18:40:06.395 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-544.el8.x86_64"... 2024-03-18T18:40:06.396 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-03-18T18:40:06.415 INFO:teuthology.orchestra.run.smithi001.stdout:4.18.0-544.el8.x86_64 2024-03-18T18:40:06.415 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-544.el8.x86_64 vs 4.18.0-544.el8.x86_64 2024-03-18T18:40:06.415 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-18T18:40:06.416 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-18T18:40:07.416 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-18T18:40:07.416 DEBUG:teuthology.orchestra.run.smithi001:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-18T18:40:07.503 INFO:teuthology.orchestra.run.smithi001.stdout:ttyS1 2024-03-18T18:40:07.519 DEBUG:teuthology.parallel:result is None 2024-03-18T18:40:07.519 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-18T18:40:07.529 INFO:teuthology.task.internal:Creating test directory... 2024-03-18T18:40:07.529 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T18:40:07.531 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T18:40:07.553 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-18T18:40:07.595 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-18T18:40:07.628 INFO:teuthology.task.internal:Creating archive directory... 2024-03-18T18:40:07.628 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T18:40:07.630 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T18:40:07.712 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-18T18:40:07.719 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-18T18:40:07.719 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T18:40:07.722 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T18:40:07.766 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T18:40:07.772 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T18:40:07.777 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T18:40:07.783 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T18:40:07.784 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-18T18:40:07.791 INFO:teuthology.task.internal:Configuring sudo... 2024-03-18T18:40:07.792 DEBUG:teuthology.orchestra.run.smithi001:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T18:40:07.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T18:40:07.824 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-18T18:40:07.906 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-18T18:40:07.906 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T18:40:07.909 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T18:40:07.927 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T18:40:08.014 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T18:40:08.071 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T18:40:08.158 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T18:40:08.203 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:40:08.203 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T18:40:08.273 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T18:40:08.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T18:40:08.376 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T18:40:08.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T18:40:08.506 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:40:08.506 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T18:40:08.573 DEBUG:teuthology.orchestra.run.smithi001:> sudo service rsyslog restart 2024-03-18T18:40:08.579 DEBUG:teuthology.orchestra.run.smithi165:> sudo service rsyslog restart 2024-03-18T18:40:08.647 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T18:40:08.666 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T18:40:08.990 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-18T18:40:08.999 INFO:teuthology.task.internal:Starting timer... 2024-03-18T18:40:08.999 INFO:teuthology.run_tasks:Running task pcp... 2024-03-18T18:40:09.026 INFO:teuthology.run_tasks:Running task selinux... 2024-03-18T18:40:09.061 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-18T18:40:09.061 DEBUG:teuthology.orchestra.run.smithi001:> sudo service auditd rotate 2024-03-18T18:40:09.146 INFO:teuthology.orchestra.run.smithi001.stdout:Rotating logs: 2024-03-18T18:40:09.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo service auditd rotate 2024-03-18T18:40:09.214 INFO:teuthology.orchestra.run.smithi165.stdout:Rotating logs: 2024-03-18T18:40:09.216 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-18T18:40:09.216 DEBUG:teuthology.orchestra.run.smithi001:> /usr/sbin/getenforce 2024-03-18T18:40:09.242 INFO:teuthology.orchestra.run.smithi001.stdout:Permissive 2024-03-18T18:40:09.242 DEBUG:teuthology.orchestra.run.smithi165:> /usr/sbin/getenforce 2024-03-18T18:40:09.275 INFO:teuthology.orchestra.run.smithi165.stdout:Permissive 2024-03-18T18:40:09.275 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi001.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi165.front.sepia.ceph.com': 'permissive'} 2024-03-18T18:40:09.275 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-18T18:40:09.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T18:40:09.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-18T18:40:09.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T18:40:09.343 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-18T18:40:09.343 DEBUG:teuthology.orchestra.run.smithi001:> sudo /usr/sbin/setenforce permissive 2024-03-18T18:40:09.385 DEBUG:teuthology.orchestra.run.smithi165:> sudo /usr/sbin/setenforce permissive 2024-03-18T18:40:09.412 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-18T18:40:09.423 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-18T18:40:09.428 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-18T18:40:09.542 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-18T18:40:09.559 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-18T18:40:09.560 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi001.front.sepia.ceph.com,smithi165.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-18T18:46:07.839 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi001.front.sepia.ceph.com'), Remote(name='ubuntu@smithi165.front.sepia.ceph.com')] 2024-03-18T18:46:07.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-18T18:46:07.842 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T18:46:07.919 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-03-18T18:46:07.991 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-18T18:46:07.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-18T18:46:07.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T18:46:08.068 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-03-18T18:46:08.137 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-18T18:46:08.137 INFO:teuthology.run_tasks:Running task clock... 2024-03-18T18:46:08.153 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-18T18:46:08.154 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T18:46:08.154 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T18:46:08.157 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T18:46:08.157 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T18:46:08.191 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T18:46:08.206 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T18:46:08.210 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T18:46:08.224 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T18:46:08.286 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: ntpd: command not found 2024-03-18T18:46:08.286 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: ntpd: command not found 2024-03-18T18:46:08.298 INFO:teuthology.orchestra.run.smithi001.stdout:506 Cannot talk to daemon 2024-03-18T18:46:08.298 INFO:teuthology.orchestra.run.smithi165.stdout:506 Cannot talk to daemon 2024-03-18T18:46:08.312 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T18:46:08.312 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T18:46:08.325 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T18:46:08.326 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T18:46:08.393 INFO:teuthology.orchestra.run.smithi165.stderr:bash: ntpq: command not found 2024-03-18T18:46:08.397 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T18:46:08.397 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-03-18T18:46:08.398 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T18:46:08.398 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T18:46:08.398 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T18:46:08.398 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T18:46:08.398 INFO:teuthology.orchestra.run.smithi001.stderr:bash: ntpq: command not found 2024-03-18T18:46:08.401 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T18:46:08.401 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-03-18T18:46:08.401 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T18:46:08.401 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T18:46:08.401 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T18:46:08.401 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T18:46:08.402 INFO:teuthology.run_tasks:Running task install... 2024-03-18T18:46:08.411 DEBUG:teuthology.task.install:project ceph 2024-03-18T18:46:08.412 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '49bdc51142759622a281d3e5327eebd07cbb8bb3'}} 2024-03-18T18:46:08.412 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '49bdc51142759622a281d3e5327eebd07cbb8bb3'} 2024-03-18T18:46:08.412 INFO:teuthology.task.install:Using flavor: default 2024-03-18T18:46:08.418 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-18T18:46:08.418 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-03-18T18:46:08.419 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '49bdc51142759622a281d3e5327eebd07cbb8bb3', 'tag': None, 'wait_for_package': False} 2024-03-18T18:46:08.419 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=49bdc51142759622a281d3e5327eebd07cbb8bb3 2024-03-18T18:46:08.421 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '49bdc51142759622a281d3e5327eebd07cbb8bb3', 'tag': None, 'wait_for_package': False} 2024-03-18T18:46:08.421 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=49bdc51142759622a281d3e5327eebd07cbb8bb3 2024-03-18T18:46:08.564 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/ 2024-03-18T18:46:08.565 INFO:teuthology.task.install.rpm:Package version is 18.2.2-914.g49bdc511 2024-03-18T18:46:08.642 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/ 2024-03-18T18:46:08.642 INFO:teuthology.task.install.rpm:Package version is 18.2.2-914.g49bdc511 2024-03-18T18:46:08.675 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T18:46:08.675 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:46:08.675 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T18:46:08.702 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T18:46:08.703 DEBUG:teuthology.orchestra.run.smithi165:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/49bdc51142759622a281d3e5327eebd07cbb8bb3/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T18:46:08.771 DEBUG:teuthology.orchestra.run.smithi165:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T18:46:08.812 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-15-1653-reef/49bdc51142759622a281d3e5327eebd07cbb8bb3/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T18:46:08.812 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:46:08.812 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T18:46:08.843 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T18:46:08.843 DEBUG:teuthology.orchestra.run.smithi001:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/49bdc51142759622a281d3e5327eebd07cbb8bb3/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T18:46:08.848 DEBUG:teuthology.orchestra.run.smithi165:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T18:46:08.915 DEBUG:teuthology.orchestra.run.smithi001:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T18:46:08.916 INFO:teuthology.orchestra.run.smithi165.stdout:check_obsoletes = 1 2024-03-18T18:46:08.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum clean all 2024-03-18T18:46:08.996 DEBUG:teuthology.orchestra.run.smithi001:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T18:46:09.064 INFO:teuthology.orchestra.run.smithi001.stdout:check_obsoletes = 1 2024-03-18T18:46:09.066 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum clean all 2024-03-18T18:46:09.188 INFO:teuthology.orchestra.run.smithi165.stdout:76 files removed 2024-03-18T18:46:09.216 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T18:46:09.357 INFO:teuthology.orchestra.run.smithi001.stdout:76 files removed 2024-03-18T18:46:09.387 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T18:46:09.921 INFO:teuthology.orchestra.run.smithi165.stdout:ceph packages for x86_64 266 kB/s | 80 kB 00:00 2024-03-18T18:46:10.211 INFO:teuthology.orchestra.run.smithi001.stdout:ceph packages for x86_64 201 kB/s | 80 kB 00:00 2024-03-18T18:46:10.363 INFO:teuthology.orchestra.run.smithi165.stdout:ceph noarch packages 25 kB/s | 9.9 kB 00:00 2024-03-18T18:46:10.562 INFO:teuthology.orchestra.run.smithi001.stdout:ceph noarch packages 32 kB/s | 9.9 kB 00:00 2024-03-18T18:46:10.730 INFO:teuthology.orchestra.run.smithi165.stdout:ceph source packages 4.4 kB/s | 1.5 kB 00:00 2024-03-18T18:46:10.896 INFO:teuthology.orchestra.run.smithi001.stdout:ceph source packages 5.0 kB/s | 1.5 kB 00:00 2024-03-18T18:46:12.864 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 8 - AppStream 13 MB/s | 28 MB 00:02 2024-03-18T18:46:15.238 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 8 - AppStream 6.4 MB/s | 28 MB 00:04 2024-03-18T18:46:18.523 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 8 - BaseOS 15 MB/s | 10 MB 00:00 2024-03-18T18:46:20.357 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 8 - Extras 73 kB/s | 18 kB 00:00 2024-03-18T18:46:20.915 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 7.5 kB 00:00 2024-03-18T18:46:21.425 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 8 - PowerTools 12 MB/s | 4.9 MB 00:00 2024-03-18T18:46:21.671 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 8 - BaseOS 9.9 MB/s | 10 MB 00:01 2024-03-18T18:46:22.601 INFO:teuthology.orchestra.run.smithi165.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2024-03-18T18:46:23.739 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 8 - Extras 52 kB/s | 18 kB 00:00 2024-03-18T18:46:24.079 INFO:teuthology.orchestra.run.smithi165.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2024-03-18T18:46:24.239 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 7.5 kB 00:00 2024-03-18T18:46:24.807 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 8 - PowerTools 10 MB/s | 4.9 MB 00:00 2024-03-18T18:46:25.925 INFO:teuthology.orchestra.run.smithi001.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2024-03-18T18:46:26.676 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 26 MB/s | 16 MB 00:00 2024-03-18T18:46:27.870 INFO:teuthology.orchestra.run.smithi165.stdout:lab-extras 449 kB/s | 24 kB 00:00 2024-03-18T18:46:30.710 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 359 kB/s | 24 kB 00:00 2024-03-18T18:46:31.411 INFO:teuthology.orchestra.run.smithi165.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-18T18:46:31.411 INFO:teuthology.orchestra.run.smithi165.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-18T18:46:31.421 INFO:teuthology.orchestra.run.smithi165.stdout:Package NetworkManager-1:1.40.16-4.el8.x86_64 is already installed. 2024-03-18T18:46:31.421 INFO:teuthology.orchestra.run.smithi165.stdout:Package iproute-6.2.0-1.el8.x86_64 is already installed. 2024-03-18T18:46:31.422 INFO:teuthology.orchestra.run.smithi165.stdout:Package util-linux-2.32.1-41.el8.x86_64 is already installed. 2024-03-18T18:46:31.423 INFO:teuthology.orchestra.run.smithi165.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-03-18T18:46:31.424 INFO:teuthology.orchestra.run.smithi165.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-03-18T18:46:31.426 INFO:teuthology.orchestra.run.smithi165.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-03-18T18:46:31.427 INFO:teuthology.orchestra.run.smithi165.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-03-18T18:46:31.506 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-03-18T18:46:31.521 INFO:teuthology.orchestra.run.smithi165.stdout:============================================================================================ 2024-03-18T18:46:31.521 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-03-18T18:46:31.521 INFO:teuthology.orchestra.run.smithi165.stdout:============================================================================================ 2024-03-18T18:46:31.521 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-03-18T18:46:31.521 INFO:teuthology.orchestra.run.smithi165.stdout: bison x86_64 3.0.4-10.el8 appstream 688 k 2024-03-18T18:46:31.521 INFO:teuthology.orchestra.run.smithi165.stdout: ceph x86_64 2:18.2.2-914.g49bdc511.el8 ceph 6.4 k 2024-03-18T18:46:31.522 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base x86_64 2:18.2.2-914.g49bdc511.el8 ceph 5.4 M 2024-03-18T18:46:31.522 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-fuse x86_64 2:18.2.2-914.g49bdc511.el8 ceph 854 k 2024-03-18T18:46:31.522 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-914.g49bdc511.el8 ceph 165 k 2024-03-18T18:46:31.522 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr x86_64 2:18.2.2-914.g49bdc511.el8 ceph 1.6 M 2024-03-18T18:46:31.522 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-cephadm noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 138 k 2024-03-18T18:46:31.522 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 1.7 M 2024-03-18T18:46:31.522 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 7.4 M 2024-03-18T18:46:31.523 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-rook noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 49 k 2024-03-18T18:46:31.523 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-radosgw x86_64 2:18.2.2-914.g49bdc511.el8 ceph 13 M 2024-03-18T18:46:31.523 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-test x86_64 2:18.2.2-914.g49bdc511.el8 ceph 47 M 2024-03-18T18:46:31.524 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-volume noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 263 k 2024-03-18T18:46:31.524 INFO:teuthology.orchestra.run.smithi165.stdout: cephadm noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 172 k 2024-03-18T18:46:31.524 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-mirror x86_64 2:18.2.2-914.g49bdc511.el8 ceph 261 k 2024-03-18T18:46:31.524 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-top noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 18 k 2024-03-18T18:46:31.524 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-libelf-devel x86_64 0.190-2.el8 baseos 62 k 2024-03-18T18:46:31.524 INFO:teuthology.orchestra.run.smithi165.stdout: flex x86_64 2.6.1-9.el8 appstream 320 k 2024-03-18T18:46:31.524 INFO:teuthology.orchestra.run.smithi165.stdout: libaio-devel x86_64 0.3.112-1.el8 baseos 19 k 2024-03-18T18:46:31.524 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-devel x86_64 2:18.2.2-914.g49bdc511.el8 ceph 31 k 2024-03-18T18:46:31.524 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs2 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 775 k 2024-03-18T18:46:31.524 INFO:teuthology.orchestra.run.smithi165.stdout: librados-devel x86_64 2:18.2.2-914.g49bdc511.el8 ceph 139 k 2024-03-18T18:46:31.524 INFO:teuthology.orchestra.run.smithi165.stdout: libtool x86_64 2.4.6-25.el8 appstream 709 k 2024-03-18T18:46:31.524 INFO:teuthology.orchestra.run.smithi165.stdout: libuuid-devel x86_64 2.32.1-46.el8 baseos 100 k 2024-03-18T18:46:31.525 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-devel x86_64 1:1.1.1k-12.el8 baseos 3.2 M 2024-03-18T18:46:31.525 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql x86_64 10.23-2.module_el8+572+929c87ac appstream 1.6 M 2024-03-18T18:46:31.525 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-contrib x86_64 10.23-2.module_el8+572+929c87ac appstream 901 k 2024-03-18T18:46:31.525 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-server x86_64 10.23-2.module_el8+572+929c87ac appstream 5.1 M 2024-03-18T18:46:31.525 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cephfs x86_64 2:18.2.2-914.g49bdc511.el8 ceph 196 k 2024-03-18T18:46:31.525 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rados x86_64 2:18.2.2-914.g49bdc511.el8 ceph 370 k 2024-03-18T18:46:31.525 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rbd x86_64 2:18.2.2-914.g49bdc511.el8 ceph 365 k 2024-03-18T18:46:31.525 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rgw x86_64 2:18.2.2-914.g49bdc511.el8 ceph 107 k 2024-03-18T18:46:31.525 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-fuse x86_64 2:18.2.2-914.g49bdc511.el8 ceph 95 k 2024-03-18T18:46:31.525 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-mirror x86_64 2:18.2.2-914.g49bdc511.el8 ceph 3.5 M 2024-03-18T18:46:31.525 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-nbd x86_64 2:18.2.2-914.g49bdc511.el8 ceph 182 k 2024-03-18T18:46:31.525 INFO:teuthology.orchestra.run.smithi165.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 baseos 74 k 2024-03-18T18:46:31.525 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrading: 2024-03-18T18:46:31.526 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager x86_64 1:1.40.16-15.el8 baseos 2.3 M 2024-03-18T18:46:31.526 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-libnm x86_64 1:1.40.16-15.el8 baseos 1.9 M 2024-03-18T18:46:31.526 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-team x86_64 1:1.40.16-15.el8 baseos 161 k 2024-03-18T18:46:31.526 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-tui x86_64 1:1.40.16-15.el8 baseos 356 k 2024-03-18T18:46:31.526 INFO:teuthology.orchestra.run.smithi165.stdout: acl x86_64 2.2.53-3.el8 baseos 83 k 2024-03-18T18:46:31.526 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-debuginfod-client x86_64 0.190-2.el8 baseos 77 k 2024-03-18T18:46:31.526 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-libelf x86_64 0.190-2.el8 baseos 233 k 2024-03-18T18:46:31.526 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-libs x86_64 0.190-2.el8 baseos 305 k 2024-03-18T18:46:31.527 INFO:teuthology.orchestra.run.smithi165.stdout: iproute x86_64 6.2.0-5.el8 baseos 854 k 2024-03-18T18:46:31.527 INFO:teuthology.orchestra.run.smithi165.stdout: libacl x86_64 2.2.53-3.el8 baseos 35 k 2024-03-18T18:46:31.527 INFO:teuthology.orchestra.run.smithi165.stdout: libacl-devel x86_64 2.2.53-3.el8 baseos 85 k 2024-03-18T18:46:31.527 INFO:teuthology.orchestra.run.smithi165.stdout: libblkid x86_64 2.32.1-46.el8 baseos 221 k 2024-03-18T18:46:31.527 INFO:teuthology.orchestra.run.smithi165.stdout: libfdisk x86_64 2.32.1-46.el8 baseos 254 k 2024-03-18T18:46:31.527 INFO:teuthology.orchestra.run.smithi165.stdout: libmount x86_64 2.32.1-46.el8 baseos 237 k 2024-03-18T18:46:31.527 INFO:teuthology.orchestra.run.smithi165.stdout: librados2 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 3.7 M 2024-03-18T18:46:31.527 INFO:teuthology.orchestra.run.smithi165.stdout: librbd1 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 3.6 M 2024-03-18T18:46:31.527 INFO:teuthology.orchestra.run.smithi165.stdout: libsmartcols x86_64 2.32.1-46.el8 baseos 180 k 2024-03-18T18:46:31.527 INFO:teuthology.orchestra.run.smithi165.stdout: libuuid x86_64 2.32.1-46.el8 baseos 99 k 2024-03-18T18:46:31.527 INFO:teuthology.orchestra.run.smithi165.stdout: openssl x86_64 1:1.1.1k-12.el8 baseos 738 k 2024-03-18T18:46:31.527 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-libs x86_64 1:1.1.1k-12.el8 baseos 1.5 M 2024-03-18T18:46:31.527 INFO:teuthology.orchestra.run.smithi165.stdout: util-linux x86_64 2.32.1-46.el8 baseos 2.6 M 2024-03-18T18:46:31.528 INFO:teuthology.orchestra.run.smithi165.stdout: util-linux-user x86_64 2.32.1-46.el8 baseos 103 k 2024-03-18T18:46:31.528 INFO:teuthology.orchestra.run.smithi165.stdout: xfsdump x86_64 3.1.8-7.el8 baseos 334 k 2024-03-18T18:46:31.528 INFO:teuthology.orchestra.run.smithi165.stdout: xfsprogs x86_64 5.0.0-12.el8 baseos 1.1 M 2024-03-18T18:46:31.528 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-03-18T18:46:31.528 INFO:teuthology.orchestra.run.smithi165.stdout: automake noarch 1.16.1-8.el8 appstream 740 k 2024-03-18T18:46:31.528 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-common x86_64 2:18.2.2-914.g49bdc511.el8 ceph 22 M 2024-03-18T18:46:31.528 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-grafana-dashboards noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 23 k 2024-03-18T18:46:31.528 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mds x86_64 2:18.2.2-914.g49bdc511.el8 ceph 2.3 M 2024-03-18T18:46:31.528 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 245 k 2024-03-18T18:46:31.528 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mon x86_64 2:18.2.2-914.g49bdc511.el8 ceph 4.6 M 2024-03-18T18:46:31.528 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-osd x86_64 2:18.2.2-914.g49bdc511.el8 ceph 18 M 2024-03-18T18:46:31.528 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-prometheus-alerts noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 15 k 2024-03-18T18:46:31.528 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-selinux x86_64 2:18.2.2-914.g49bdc511.el8 ceph 24 k 2024-03-18T18:46:31.529 INFO:teuthology.orchestra.run.smithi165.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-03-18T18:46:31.529 INFO:teuthology.orchestra.run.smithi165.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-18T18:46:31.529 INFO:teuthology.orchestra.run.smithi165.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-03-18T18:46:31.529 INFO:teuthology.orchestra.run.smithi165.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 baseos 48 k 2024-03-18T18:46:31.529 INFO:teuthology.orchestra.run.smithi165.stdout: krb5-devel x86_64 1.18.2-22.el8 baseos 561 k 2024-03-18T18:46:31.529 INFO:teuthology.orchestra.run.smithi165.stdout: libcephsqlite x86_64 2:18.2.2-914.g49bdc511.el8 ceph 142 k 2024-03-18T18:46:31.529 INFO:teuthology.orchestra.run.smithi165.stdout: libcom_err-devel x86_64 1.45.6-5.el8 baseos 39 k 2024-03-18T18:46:31.529 INFO:teuthology.orchestra.run.smithi165.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-03-18T18:46:31.529 INFO:teuthology.orchestra.run.smithi165.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-18T18:46:31.529 INFO:teuthology.orchestra.run.smithi165.stdout: libpq x86_64 13.5-1.el8 appstream 198 k 2024-03-18T18:46:31.529 INFO:teuthology.orchestra.run.smithi165.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-03-18T18:46:31.529 INFO:teuthology.orchestra.run.smithi165.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-03-18T18:46:31.529 INFO:teuthology.orchestra.run.smithi165.stdout: libradosstriper1 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 474 k 2024-03-18T18:46:31.529 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-03-18T18:46:31.530 INFO:teuthology.orchestra.run.smithi165.stdout: librgw2 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 6.5 M 2024-03-18T18:46:31.530 INFO:teuthology.orchestra.run.smithi165.stdout: libselinux-devel x86_64 2.9-8.el8 baseos 200 k 2024-03-18T18:46:31.530 INFO:teuthology.orchestra.run.smithi165.stdout: libsepol-devel x86_64 2.9-3.el8 baseos 87 k 2024-03-18T18:46:31.530 INFO:teuthology.orchestra.run.smithi165.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-18T18:46:31.530 INFO:teuthology.orchestra.run.smithi165.stdout: libverto-devel x86_64 0.3.2-2.el8 baseos 18 k 2024-03-18T18:46:31.530 INFO:teuthology.orchestra.run.smithi165.stdout: libzstd-devel x86_64 1.4.4-1.el8 baseos 44 k 2024-03-18T18:46:31.530 INFO:teuthology.orchestra.run.smithi165.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-03-18T18:46:31.530 INFO:teuthology.orchestra.run.smithi165.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-03-18T18:46:31.530 INFO:teuthology.orchestra.run.smithi165.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-03-18T18:46:31.530 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-03-18T18:46:31.531 INFO:teuthology.orchestra.run.smithi165.stdout: pcre2-devel x86_64 10.32-3.el8 baseos 605 k 2024-03-18T18:46:31.531 INFO:teuthology.orchestra.run.smithi165.stdout: pcre2-utf16 x86_64 10.32-3.el8 baseos 229 k 2024-03-18T18:46:31.531 INFO:teuthology.orchestra.run.smithi165.stdout: pcre2-utf32 x86_64 10.32-3.el8 baseos 220 k 2024-03-18T18:46:31.531 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-18T18:46:31.531 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-03-18T18:46:31.531 INFO:teuthology.orchestra.run.smithi165.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-18T18:46:31.531 INFO:teuthology.orchestra.run.smithi165.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-18T18:46:31.531 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-18T18:46:31.531 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse x86_64 2:18.2.2-914.g49bdc511.el8 ceph 45 k 2024-03-18T18:46:31.531 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common x86_64 2:18.2.2-914.g49bdc511.el8 ceph 126 k 2024-03-18T18:46:31.531 INFO:teuthology.orchestra.run.smithi165.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-18T18:46:31.531 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-03-18T18:46:31.531 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-18T18:46:31.531 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-18T18:46:31.532 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-03-18T18:46:31.532 INFO:teuthology.orchestra.run.smithi165.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-18T18:46:31.532 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-18T18:46:31.532 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-18T18:46:31.532 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-03-18T18:46:31.532 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-03-18T18:46:31.532 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-03-18T18:46:31.532 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-03-18T18:46:31.532 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-18T18:46:31.532 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-18T18:46:31.532 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-03-18T18:46:31.532 INFO:teuthology.orchestra.run.smithi165.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-03-18T18:46:31.532 INFO:teuthology.orchestra.run.smithi165.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-18T18:46:31.532 INFO:teuthology.orchestra.run.smithi165.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-18T18:46:31.533 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-03-18T18:46:31.533 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-03-18T18:46:31.533 INFO:teuthology.orchestra.run.smithi165.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-03-18T18:46:31.533 INFO:teuthology.orchestra.run.smithi165.stdout: python3-packaging noarch 16.8-9.el8 powertools 48 k 2024-03-18T18:46:31.533 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-18T18:46:31.533 INFO:teuthology.orchestra.run.smithi165.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-18T18:46:31.533 INFO:teuthology.orchestra.run.smithi165.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-03-18T18:46:31.533 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-03-18T18:46:31.533 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-03-18T18:46:31.533 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-03-18T18:46:31.533 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-03-18T18:46:31.533 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-03-18T18:46:31.533 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-18T18:46:31.534 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-03-18T18:46:31.534 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-18T18:46:31.534 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-18T18:46:31.534 INFO:teuthology.orchestra.run.smithi165.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-03-18T18:46:31.534 INFO:teuthology.orchestra.run.smithi165.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-18T18:46:31.534 INFO:teuthology.orchestra.run.smithi165.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-18T18:46:31.534 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-18T18:46:31.534 INFO:teuthology.orchestra.run.smithi165.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-18T18:46:31.534 INFO:teuthology.orchestra.run.smithi165.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-18T18:46:31.534 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-18T18:46:31.534 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-18T18:46:31.534 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-18T18:46:31.534 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-03-18T18:46:31.534 INFO:teuthology.orchestra.run.smithi165.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-18T18:46:31.535 INFO:teuthology.orchestra.run.smithi165.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-03-18T18:46:31.535 INFO:teuthology.orchestra.run.smithi165.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-18T18:46:31.535 INFO:teuthology.orchestra.run.smithi165.stdout: uuid x86_64 1.6.2-43.el8 appstream 64 k 2024-03-18T18:46:31.535 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-03-18T18:46:31.535 INFO:teuthology.orchestra.run.smithi165.stdout: zlib-devel x86_64 1.2.11-21.el8 baseos 58 k 2024-03-18T18:46:31.535 INFO:teuthology.orchestra.run.smithi165.stdout:Enabling module streams: 2024-03-18T18:46:31.535 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql 10 2024-03-18T18:46:31.535 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:46:31.535 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-03-18T18:46:31.535 INFO:teuthology.orchestra.run.smithi165.stdout:============================================================================================ 2024-03-18T18:46:31.535 INFO:teuthology.orchestra.run.smithi165.stdout:Install 129 Packages 2024-03-18T18:46:31.535 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrade 24 Packages 2024-03-18T18:46:31.535 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:46:31.536 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 218 M 2024-03-18T18:46:31.536 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-03-18T18:46:32.546 INFO:teuthology.orchestra.run.smithi165.stdout:(1/153): ceph-18.2.2-914.g49bdc511.el8.x86_64.r 29 kB/s | 6.4 kB 00:00 2024-03-18T18:46:32.938 INFO:teuthology.orchestra.run.smithi165.stdout:(2/153): ceph-fuse-18.2.2-914.g49bdc511.el8.x86 2.1 MB/s | 854 kB 00:00 2024-03-18T18:46:33.038 INFO:teuthology.orchestra.run.smithi165.stdout:(3/153): ceph-base-18.2.2-914.g49bdc511.el8.x86 7.6 MB/s | 5.4 MB 00:00 2024-03-18T18:46:33.063 INFO:teuthology.orchestra.run.smithi165.stdout:(4/153): ceph-immutable-object-cache-18.2.2-914 1.3 MB/s | 165 kB 00:00 2024-03-18T18:46:33.239 INFO:teuthology.orchestra.run.smithi165.stdout:(5/153): ceph-mds-18.2.2-914.g49bdc511.el8.x86_ 11 MB/s | 2.3 MB 00:00 2024-03-18T18:46:33.606 INFO:teuthology.orchestra.run.smithi165.stdout:(6/153): ceph-mgr-18.2.2-914.g49bdc511.el8.x86_ 2.9 MB/s | 1.6 MB 00:00 2024-03-18T18:46:33.673 INFO:teuthology.orchestra.run.smithi165.stdout:(7/153): ceph-mon-18.2.2-914.g49bdc511.el8.x86_ 11 MB/s | 4.6 MB 00:00 2024-03-18T18:46:34.567 INFO:teuthology.orchestra.run.smithi001.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-18T18:46:34.568 INFO:teuthology.orchestra.run.smithi001.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-18T18:46:34.578 INFO:teuthology.orchestra.run.smithi001.stdout:Package NetworkManager-1:1.40.16-4.el8.x86_64 is already installed. 2024-03-18T18:46:34.578 INFO:teuthology.orchestra.run.smithi001.stdout:Package iproute-6.2.0-1.el8.x86_64 is already installed. 2024-03-18T18:46:34.579 INFO:teuthology.orchestra.run.smithi001.stdout:Package util-linux-2.32.1-41.el8.x86_64 is already installed. 2024-03-18T18:46:34.580 INFO:teuthology.orchestra.run.smithi001.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-03-18T18:46:34.582 INFO:teuthology.orchestra.run.smithi001.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-03-18T18:46:34.584 INFO:teuthology.orchestra.run.smithi001.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-03-18T18:46:34.585 INFO:teuthology.orchestra.run.smithi001.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-03-18T18:46:34.668 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-18T18:46:34.684 INFO:teuthology.orchestra.run.smithi001.stdout:============================================================================================ 2024-03-18T18:46:34.684 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-03-18T18:46:34.684 INFO:teuthology.orchestra.run.smithi001.stdout:============================================================================================ 2024-03-18T18:46:34.684 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-18T18:46:34.685 INFO:teuthology.orchestra.run.smithi001.stdout: bison x86_64 3.0.4-10.el8 appstream 688 k 2024-03-18T18:46:34.685 INFO:teuthology.orchestra.run.smithi001.stdout: ceph x86_64 2:18.2.2-914.g49bdc511.el8 ceph 6.4 k 2024-03-18T18:46:34.685 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base x86_64 2:18.2.2-914.g49bdc511.el8 ceph 5.4 M 2024-03-18T18:46:34.685 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse x86_64 2:18.2.2-914.g49bdc511.el8 ceph 854 k 2024-03-18T18:46:34.685 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-914.g49bdc511.el8 ceph 165 k 2024-03-18T18:46:34.685 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr x86_64 2:18.2.2-914.g49bdc511.el8 ceph 1.6 M 2024-03-18T18:46:34.685 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 138 k 2024-03-18T18:46:34.685 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 1.7 M 2024-03-18T18:46:34.685 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 7.4 M 2024-03-18T18:46:34.685 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 49 k 2024-03-18T18:46:34.685 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw x86_64 2:18.2.2-914.g49bdc511.el8 ceph 13 M 2024-03-18T18:46:34.685 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test x86_64 2:18.2.2-914.g49bdc511.el8 ceph 47 M 2024-03-18T18:46:34.685 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 263 k 2024-03-18T18:46:34.685 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 172 k 2024-03-18T18:46:34.686 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-mirror x86_64 2:18.2.2-914.g49bdc511.el8 ceph 261 k 2024-03-18T18:46:34.686 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-top noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 18 k 2024-03-18T18:46:34.686 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libelf-devel x86_64 0.190-2.el8 baseos 62 k 2024-03-18T18:46:34.686 INFO:teuthology.orchestra.run.smithi001.stdout: flex x86_64 2.6.1-9.el8 appstream 320 k 2024-03-18T18:46:34.686 INFO:teuthology.orchestra.run.smithi001.stdout: libaio-devel x86_64 0.3.112-1.el8 baseos 19 k 2024-03-18T18:46:34.686 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel x86_64 2:18.2.2-914.g49bdc511.el8 ceph 31 k 2024-03-18T18:46:34.686 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 775 k 2024-03-18T18:46:34.686 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel x86_64 2:18.2.2-914.g49bdc511.el8 ceph 139 k 2024-03-18T18:46:34.686 INFO:teuthology.orchestra.run.smithi001.stdout: libtool x86_64 2.4.6-25.el8 appstream 709 k 2024-03-18T18:46:34.686 INFO:teuthology.orchestra.run.smithi001.stdout: libuuid-devel x86_64 2.32.1-46.el8 baseos 100 k 2024-03-18T18:46:34.686 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-devel x86_64 1:1.1.1k-12.el8 baseos 3.2 M 2024-03-18T18:46:34.686 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql x86_64 10.23-2.module_el8+572+929c87ac appstream 1.6 M 2024-03-18T18:46:34.687 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-contrib x86_64 10.23-2.module_el8+572+929c87ac appstream 901 k 2024-03-18T18:46:34.687 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-server x86_64 10.23-2.module_el8+572+929c87ac appstream 5.1 M 2024-03-18T18:46:34.687 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs x86_64 2:18.2.2-914.g49bdc511.el8 ceph 196 k 2024-03-18T18:46:34.687 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados x86_64 2:18.2.2-914.g49bdc511.el8 ceph 370 k 2024-03-18T18:46:34.687 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd x86_64 2:18.2.2-914.g49bdc511.el8 ceph 365 k 2024-03-18T18:46:34.687 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw x86_64 2:18.2.2-914.g49bdc511.el8 ceph 107 k 2024-03-18T18:46:34.687 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse x86_64 2:18.2.2-914.g49bdc511.el8 ceph 95 k 2024-03-18T18:46:34.687 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror x86_64 2:18.2.2-914.g49bdc511.el8 ceph 3.5 M 2024-03-18T18:46:34.687 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd x86_64 2:18.2.2-914.g49bdc511.el8 ceph 182 k 2024-03-18T18:46:34.687 INFO:teuthology.orchestra.run.smithi001.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 baseos 74 k 2024-03-18T18:46:34.687 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrading: 2024-03-18T18:46:34.687 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager x86_64 1:1.40.16-15.el8 baseos 2.3 M 2024-03-18T18:46:34.687 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-libnm x86_64 1:1.40.16-15.el8 baseos 1.9 M 2024-03-18T18:46:34.687 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-team x86_64 1:1.40.16-15.el8 baseos 161 k 2024-03-18T18:46:34.688 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-tui x86_64 1:1.40.16-15.el8 baseos 356 k 2024-03-18T18:46:34.688 INFO:teuthology.orchestra.run.smithi001.stdout: acl x86_64 2.2.53-3.el8 baseos 83 k 2024-03-18T18:46:34.688 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-debuginfod-client x86_64 0.190-2.el8 baseos 77 k 2024-03-18T18:46:34.688 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libelf x86_64 0.190-2.el8 baseos 233 k 2024-03-18T18:46:34.688 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libs x86_64 0.190-2.el8 baseos 305 k 2024-03-18T18:46:34.688 INFO:teuthology.orchestra.run.smithi001.stdout: iproute x86_64 6.2.0-5.el8 baseos 854 k 2024-03-18T18:46:34.688 INFO:teuthology.orchestra.run.smithi001.stdout: libacl x86_64 2.2.53-3.el8 baseos 35 k 2024-03-18T18:46:34.688 INFO:teuthology.orchestra.run.smithi001.stdout: libacl-devel x86_64 2.2.53-3.el8 baseos 85 k 2024-03-18T18:46:34.688 INFO:teuthology.orchestra.run.smithi001.stdout: libblkid x86_64 2.32.1-46.el8 baseos 221 k 2024-03-18T18:46:34.688 INFO:teuthology.orchestra.run.smithi001.stdout: libfdisk x86_64 2.32.1-46.el8 baseos 254 k 2024-03-18T18:46:34.688 INFO:teuthology.orchestra.run.smithi001.stdout: libmount x86_64 2.32.1-46.el8 baseos 237 k 2024-03-18T18:46:34.688 INFO:teuthology.orchestra.run.smithi001.stdout: librados2 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 3.7 M 2024-03-18T18:46:34.688 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 3.6 M 2024-03-18T18:46:34.688 INFO:teuthology.orchestra.run.smithi001.stdout: libsmartcols x86_64 2.32.1-46.el8 baseos 180 k 2024-03-18T18:46:34.689 INFO:teuthology.orchestra.run.smithi001.stdout: libuuid x86_64 2.32.1-46.el8 baseos 99 k 2024-03-18T18:46:34.689 INFO:teuthology.orchestra.run.smithi001.stdout: openssl x86_64 1:1.1.1k-12.el8 baseos 738 k 2024-03-18T18:46:34.689 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-libs x86_64 1:1.1.1k-12.el8 baseos 1.5 M 2024-03-18T18:46:34.689 INFO:teuthology.orchestra.run.smithi001.stdout: util-linux x86_64 2.32.1-46.el8 baseos 2.6 M 2024-03-18T18:46:34.689 INFO:teuthology.orchestra.run.smithi001.stdout: util-linux-user x86_64 2.32.1-46.el8 baseos 103 k 2024-03-18T18:46:34.689 INFO:teuthology.orchestra.run.smithi001.stdout: xfsdump x86_64 3.1.8-7.el8 baseos 334 k 2024-03-18T18:46:34.689 INFO:teuthology.orchestra.run.smithi001.stdout: xfsprogs x86_64 5.0.0-12.el8 baseos 1.1 M 2024-03-18T18:46:34.689 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-03-18T18:46:34.689 INFO:teuthology.orchestra.run.smithi001.stdout: automake noarch 1.16.1-8.el8 appstream 740 k 2024-03-18T18:46:34.690 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common x86_64 2:18.2.2-914.g49bdc511.el8 ceph 22 M 2024-03-18T18:46:34.690 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 23 k 2024-03-18T18:46:34.690 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds x86_64 2:18.2.2-914.g49bdc511.el8 ceph 2.3 M 2024-03-18T18:46:34.690 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 245 k 2024-03-18T18:46:34.690 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon x86_64 2:18.2.2-914.g49bdc511.el8 ceph 4.6 M 2024-03-18T18:46:34.690 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd x86_64 2:18.2.2-914.g49bdc511.el8 ceph 18 M 2024-03-18T18:46:34.690 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts noarch 2:18.2.2-914.g49bdc511.el8 ceph-noarch 15 k 2024-03-18T18:46:34.690 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux x86_64 2:18.2.2-914.g49bdc511.el8 ceph 24 k 2024-03-18T18:46:34.691 INFO:teuthology.orchestra.run.smithi001.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-03-18T18:46:34.691 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-18T18:46:34.691 INFO:teuthology.orchestra.run.smithi001.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-03-18T18:46:34.691 INFO:teuthology.orchestra.run.smithi001.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 baseos 48 k 2024-03-18T18:46:34.691 INFO:teuthology.orchestra.run.smithi001.stdout: krb5-devel x86_64 1.18.2-22.el8 baseos 561 k 2024-03-18T18:46:34.691 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite x86_64 2:18.2.2-914.g49bdc511.el8 ceph 142 k 2024-03-18T18:46:34.691 INFO:teuthology.orchestra.run.smithi001.stdout: libcom_err-devel x86_64 1.45.6-5.el8 baseos 39 k 2024-03-18T18:46:34.691 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-03-18T18:46:34.691 INFO:teuthology.orchestra.run.smithi001.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-18T18:46:34.691 INFO:teuthology.orchestra.run.smithi001.stdout: libpq x86_64 13.5-1.el8 appstream 198 k 2024-03-18T18:46:34.691 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-03-18T18:46:34.691 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-03-18T18:46:34.691 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 474 k 2024-03-18T18:46:34.692 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-03-18T18:46:34.692 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2 x86_64 2:18.2.2-914.g49bdc511.el8 ceph 6.5 M 2024-03-18T18:46:34.693 INFO:teuthology.orchestra.run.smithi001.stdout: libselinux-devel x86_64 2.9-8.el8 baseos 200 k 2024-03-18T18:46:34.693 INFO:teuthology.orchestra.run.smithi001.stdout: libsepol-devel x86_64 2.9-3.el8 baseos 87 k 2024-03-18T18:46:34.693 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-18T18:46:34.693 INFO:teuthology.orchestra.run.smithi001.stdout: libverto-devel x86_64 0.3.2-2.el8 baseos 18 k 2024-03-18T18:46:34.693 INFO:teuthology.orchestra.run.smithi001.stdout: libzstd-devel x86_64 1.4.4-1.el8 baseos 44 k 2024-03-18T18:46:34.693 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-03-18T18:46:34.693 INFO:teuthology.orchestra.run.smithi001.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-03-18T18:46:34.693 INFO:teuthology.orchestra.run.smithi001.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-03-18T18:46:34.693 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-03-18T18:46:34.693 INFO:teuthology.orchestra.run.smithi001.stdout: pcre2-devel x86_64 10.32-3.el8 baseos 605 k 2024-03-18T18:46:34.693 INFO:teuthology.orchestra.run.smithi001.stdout: pcre2-utf16 x86_64 10.32-3.el8 baseos 229 k 2024-03-18T18:46:34.693 INFO:teuthology.orchestra.run.smithi001.stdout: pcre2-utf32 x86_64 10.32-3.el8 baseos 220 k 2024-03-18T18:46:34.694 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-18T18:46:34.694 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-03-18T18:46:34.694 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-18T18:46:34.694 INFO:teuthology.orchestra.run.smithi001.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-18T18:46:34.694 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-18T18:46:34.694 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse x86_64 2:18.2.2-914.g49bdc511.el8 ceph 45 k 2024-03-18T18:46:34.694 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common x86_64 2:18.2.2-914.g49bdc511.el8 ceph 126 k 2024-03-18T18:46:34.694 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-18T18:46:34.694 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-03-18T18:46:34.694 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-18T18:46:34.694 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-18T18:46:34.694 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-03-18T18:46:34.694 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-18T18:46:34.694 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-18T18:46:34.695 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-18T18:46:34.695 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-03-18T18:46:34.695 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-03-18T18:46:34.695 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-03-18T18:46:34.695 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-03-18T18:46:34.695 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-18T18:46:34.695 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-18T18:46:34.695 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-03-18T18:46:34.695 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-03-18T18:46:34.695 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-18T18:46:34.695 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-18T18:46:34.695 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-03-18T18:46:34.696 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-03-18T18:46:34.696 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-03-18T18:46:34.696 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging noarch 16.8-9.el8 powertools 48 k 2024-03-18T18:46:34.696 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-18T18:46:34.696 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-18T18:46:34.696 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-03-18T18:46:34.696 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-03-18T18:46:34.696 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-03-18T18:46:34.696 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-03-18T18:46:34.696 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-03-18T18:46:34.696 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-03-18T18:46:34.697 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-18T18:46:34.697 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-03-18T18:46:34.697 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-18T18:46:34.697 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-18T18:46:34.697 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-03-18T18:46:34.697 INFO:teuthology.orchestra.run.smithi001.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-18T18:46:34.697 INFO:teuthology.orchestra.run.smithi001.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-18T18:46:34.697 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-18T18:46:34.697 INFO:teuthology.orchestra.run.smithi001.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-18T18:46:34.697 INFO:teuthology.orchestra.run.smithi001.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-18T18:46:34.697 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-18T18:46:34.697 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-18T18:46:34.698 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-18T18:46:34.698 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-03-18T18:46:34.698 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-18T18:46:34.698 INFO:teuthology.orchestra.run.smithi001.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-03-18T18:46:34.698 INFO:teuthology.orchestra.run.smithi001.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-18T18:46:34.698 INFO:teuthology.orchestra.run.smithi001.stdout: uuid x86_64 1.6.2-43.el8 appstream 64 k 2024-03-18T18:46:34.698 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-03-18T18:46:34.698 INFO:teuthology.orchestra.run.smithi001.stdout: zlib-devel x86_64 1.2.11-21.el8 baseos 58 k 2024-03-18T18:46:34.698 INFO:teuthology.orchestra.run.smithi001.stdout:Enabling module streams: 2024-03-18T18:46:34.698 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql 10 2024-03-18T18:46:34.698 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:46:34.698 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-18T18:46:34.698 INFO:teuthology.orchestra.run.smithi001.stdout:============================================================================================ 2024-03-18T18:46:34.699 INFO:teuthology.orchestra.run.smithi001.stdout:Install 129 Packages 2024-03-18T18:46:34.699 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrade 24 Packages 2024-03-18T18:46:34.699 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:46:34.699 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 218 M 2024-03-18T18:46:34.699 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-18T18:46:34.706 INFO:teuthology.orchestra.run.smithi165.stdout:(8/153): ceph-common-18.2.2-914.g49bdc511.el8.x 9.1 MB/s | 22 MB 00:02 2024-03-18T18:46:34.799 INFO:teuthology.orchestra.run.smithi165.stdout:(9/153): ceph-selinux-18.2.2-914.g49bdc511.el8. 266 kB/s | 24 kB 00:00 2024-03-18T18:46:35.057 INFO:teuthology.orchestra.run.smithi165.stdout:(10/153): ceph-radosgw-18.2.2-914.g49bdc511.el8 9.5 MB/s | 13 MB 00:01 2024-03-18T18:46:35.403 INFO:teuthology.orchestra.run.smithi001.stdout:(1/153): ceph-18.2.2-914.g49bdc511.el8.x86_64.r 23 kB/s | 6.4 kB 00:00 2024-03-18T18:46:36.057 INFO:teuthology.orchestra.run.smithi165.stdout:(11/153): cephfs-mirror-18.2.2-914.g49bdc511.el 261 kB/s | 261 kB 00:00 2024-03-18T18:46:36.062 INFO:teuthology.orchestra.run.smithi001.stdout:(2/153): ceph-fuse-18.2.2-914.g49bdc511.el8.x86 1.3 MB/s | 854 kB 00:00 2024-03-18T18:46:36.174 INFO:teuthology.orchestra.run.smithi165.stdout:(12/153): libcephfs-devel-18.2.2-914.g49bdc511. 265 kB/s | 31 kB 00:00 2024-03-18T18:46:36.329 INFO:teuthology.orchestra.run.smithi001.stdout:(3/153): ceph-immutable-object-cache-18.2.2-914 621 kB/s | 165 kB 00:00 2024-03-18T18:46:36.357 INFO:teuthology.orchestra.run.smithi165.stdout:(13/153): libcephfs2-18.2.2-914.g49bdc511.el8.x 4.1 MB/s | 775 kB 00:00 2024-03-18T18:46:36.458 INFO:teuthology.orchestra.run.smithi165.stdout:(14/153): libcephsqlite-18.2.2-914.g49bdc511.el 1.4 MB/s | 142 kB 00:00 2024-03-18T18:46:36.641 INFO:teuthology.orchestra.run.smithi165.stdout:(15/153): ceph-osd-18.2.2-914.g49bdc511.el8.x86 5.8 MB/s | 18 MB 00:03 2024-03-18T18:46:36.667 INFO:teuthology.orchestra.run.smithi165.stdout:(16/153): librados-devel-18.2.2-914.g49bdc511.e 666 kB/s | 139 kB 00:00 2024-03-18T18:46:36.850 INFO:teuthology.orchestra.run.smithi165.stdout:(17/153): libradosstriper1-18.2.2-914.g49bdc511 2.2 MB/s | 474 kB 00:00 2024-03-18T18:46:36.967 INFO:teuthology.orchestra.run.smithi165.stdout:(18/153): python3-ceph-argparse-18.2.2-914.g49b 381 kB/s | 45 kB 00:00 2024-03-18T18:46:37.754 INFO:teuthology.orchestra.run.smithi001.stdout:(4/153): ceph-mds-18.2.2-914.g49bdc511.el8.x86_ 1.6 MB/s | 2.3 MB 00:01 2024-03-18T18:46:37.801 INFO:teuthology.orchestra.run.smithi165.stdout:(19/153): python3-ceph-common-18.2.2-914.g49bdc 151 kB/s | 126 kB 00:00 2024-03-18T18:46:37.926 INFO:teuthology.orchestra.run.smithi165.stdout:(20/153): python3-cephfs-18.2.2-914.g49bdc511.e 1.5 MB/s | 196 kB 00:00 2024-03-18T18:46:38.118 INFO:teuthology.orchestra.run.smithi165.stdout:(21/153): python3-rados-18.2.2-914.g49bdc511.el 1.9 MB/s | 370 kB 00:00 2024-03-18T18:46:38.319 INFO:teuthology.orchestra.run.smithi165.stdout:(22/153): python3-rbd-18.2.2-914.g49bdc511.el8. 1.8 MB/s | 365 kB 00:00 2024-03-18T18:46:38.494 INFO:teuthology.orchestra.run.smithi165.stdout:(23/153): librgw2-18.2.2-914.g49bdc511.el8.x86_ 3.6 MB/s | 6.5 MB 00:01 2024-03-18T18:46:38.519 INFO:teuthology.orchestra.run.smithi165.stdout:(24/153): python3-rgw-18.2.2-914.g49bdc511.el8. 532 kB/s | 107 kB 00:00 2024-03-18T18:46:38.594 INFO:teuthology.orchestra.run.smithi165.stdout:(25/153): rbd-fuse-18.2.2-914.g49bdc511.el8.x86 950 kB/s | 95 kB 00:00 2024-03-18T18:46:38.696 INFO:teuthology.orchestra.run.smithi001.stdout:(5/153): ceph-base-18.2.2-914.g49bdc511.el8.x86 1.5 MB/s | 5.4 MB 00:03 2024-03-18T18:46:38.703 INFO:teuthology.orchestra.run.smithi165.stdout:(26/153): rbd-nbd-18.2.2-914.g49bdc511.el8.x86_ 1.6 MB/s | 182 kB 00:00 2024-03-18T18:46:38.730 INFO:teuthology.orchestra.run.smithi001.stdout:(6/153): ceph-mgr-18.2.2-914.g49bdc511.el8.x86_ 1.6 MB/s | 1.6 MB 00:00 2024-03-18T18:46:38.795 INFO:teuthology.orchestra.run.smithi165.stdout:(27/153): ceph-grafana-dashboards-18.2.2-914.g4 254 kB/s | 23 kB 00:00 2024-03-18T18:46:38.912 INFO:teuthology.orchestra.run.smithi165.stdout:(28/153): ceph-mgr-cephadm-18.2.2-914.g49bdc511 1.2 MB/s | 138 kB 00:00 2024-03-18T18:46:39.387 INFO:teuthology.orchestra.run.smithi165.stdout:(29/153): ceph-mgr-dashboard-18.2.2-914.g49bdc5 3.7 MB/s | 1.7 MB 00:00 2024-03-18T18:46:39.838 INFO:teuthology.orchestra.run.smithi165.stdout:(30/153): rbd-mirror-18.2.2-914.g49bdc511.el8.x 2.6 MB/s | 3.5 MB 00:01 2024-03-18T18:46:39.963 INFO:teuthology.orchestra.run.smithi165.stdout:(31/153): ceph-mgr-modules-core-18.2.2-914.g49b 1.9 MB/s | 245 kB 00:00 2024-03-18T18:46:40.080 INFO:teuthology.orchestra.run.smithi165.stdout:(32/153): ceph-mgr-rook-18.2.2-914.g49bdc511.el 421 kB/s | 49 kB 00:00 2024-03-18T18:46:40.180 INFO:teuthology.orchestra.run.smithi165.stdout:(33/153): ceph-prometheus-alerts-18.2.2-914.g49 147 kB/s | 15 kB 00:00 2024-03-18T18:46:40.241 INFO:teuthology.orchestra.run.smithi001.stdout:(7/153): ceph-common-18.2.2-914.g49bdc511.el8.x 4.2 MB/s | 22 MB 00:05 2024-03-18T18:46:40.322 INFO:teuthology.orchestra.run.smithi165.stdout:(34/153): ceph-volume-18.2.2-914.g49bdc511.el8. 1.8 MB/s | 263 kB 00:00 2024-03-18T18:46:40.423 INFO:teuthology.orchestra.run.smithi165.stdout:(35/153): cephadm-18.2.2-914.g49bdc511.el8.noar 1.7 MB/s | 172 kB 00:00 2024-03-18T18:46:40.523 INFO:teuthology.orchestra.run.smithi165.stdout:(36/153): cephfs-top-18.2.2-914.g49bdc511.el8.n 181 kB/s | 18 kB 00:00 2024-03-18T18:46:40.756 INFO:teuthology.orchestra.run.smithi165.stdout:(37/153): automake-1.16.1-8.el8.noarch.rpm 3.1 MB/s | 740 kB 00:00 2024-03-18T18:46:40.879 INFO:teuthology.orchestra.run.smithi165.stdout:(38/153): bison-3.0.4-10.el8.x86_64.rpm 5.5 MB/s | 688 kB 00:00 2024-03-18T18:46:40.949 INFO:teuthology.orchestra.run.smithi165.stdout:(39/153): flex-2.6.1-9.el8.x86_64.rpm 4.5 MB/s | 320 kB 00:00 2024-03-18T18:46:41.008 INFO:teuthology.orchestra.run.smithi165.stdout:(40/153): jq-1.6-8.el8.x86_64.rpm 3.4 MB/s | 203 kB 00:00 2024-03-18T18:46:41.101 INFO:teuthology.orchestra.run.smithi165.stdout:(41/153): ceph-mgr-diskprediction-local-18.2.2- 4.3 MB/s | 7.4 MB 00:01 2024-03-18T18:46:41.125 INFO:teuthology.orchestra.run.smithi165.stdout:(42/153): libpq-13.5-1.el8.x86_64.rpm 1.7 MB/s | 198 kB 00:00 2024-03-18T18:46:41.225 INFO:teuthology.orchestra.run.smithi165.stdout:(43/153): libtool-2.4.6-25.el8.x86_64.rpm 6.9 MB/s | 709 kB 00:00 2024-03-18T18:46:41.334 INFO:teuthology.orchestra.run.smithi165.stdout:(44/153): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.3 MB/s | 259 kB 00:00 2024-03-18T18:46:41.384 INFO:teuthology.orchestra.run.smithi165.stdout:(45/153): oniguruma-6.8.2-3.el8.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-03-18T18:46:41.405 INFO:teuthology.orchestra.run.smithi001.stdout:(8/153): ceph-mon-18.2.2-914.g49bdc511.el8.x86_ 1.7 MB/s | 4.6 MB 00:02 2024-03-18T18:46:41.443 INFO:teuthology.orchestra.run.smithi165.stdout:(46/153): librdkafka-1.6.1-1.el8.x86_64.rpm 1.9 MB/s | 677 kB 00:00 2024-03-18T18:46:41.480 INFO:teuthology.orchestra.run.smithi001.stdout:(9/153): ceph-selinux-18.2.2-914.g49bdc511.el8. 325 kB/s | 24 kB 00:00 2024-03-18T18:46:41.760 INFO:teuthology.orchestra.run.smithi165.stdout:(47/153): openblas-0.3.15-4.el8.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2024-03-18T18:46:41.827 INFO:teuthology.orchestra.run.smithi165.stdout:(48/153): openblas-threads-0.3.15-4.el8.x86_64. 13 MB/s | 4.9 MB 00:00 2024-03-18T18:46:41.910 INFO:teuthology.orchestra.run.smithi165.stdout:(49/153): postgresql-10.23-2.module_el8+572+929 10 MB/s | 1.6 MB 00:00 2024-03-18T18:46:41.936 INFO:teuthology.orchestra.run.smithi165.stdout:(50/153): postgresql-contrib-10.23-2.module_el8 8.1 MB/s | 901 kB 00:00 2024-03-18T18:46:42.203 INFO:teuthology.orchestra.run.smithi165.stdout:(51/153): postgresql-server-10.23-2.module_el8+ 18 MB/s | 5.1 MB 00:00 2024-03-18T18:46:42.261 INFO:teuthology.orchestra.run.smithi165.stdout:(52/153): python3-babel-2.5.1-7.el8.noarch.rpm 15 MB/s | 4.8 MB 00:00 2024-03-18T18:46:42.287 INFO:teuthology.orchestra.run.smithi165.stdout:(53/153): python3-jinja2-2.10.1-4.el8.noarch.rp 6.8 MB/s | 578 kB 00:00 2024-03-18T18:46:42.304 INFO:teuthology.orchestra.run.smithi165.stdout:(54/153): python3-jsonpatch-1.21-2.el8.noarch.r 634 kB/s | 27 kB 00:00 2024-03-18T18:46:42.337 INFO:teuthology.orchestra.run.smithi165.stdout:(55/153): python3-jsonpointer-1.10-11.el8.noarc 396 kB/s | 20 kB 00:00 2024-03-18T18:46:42.362 INFO:teuthology.orchestra.run.smithi165.stdout:(56/153): python3-mako-1.0.6-14.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2024-03-18T18:46:42.388 INFO:teuthology.orchestra.run.smithi165.stdout:(57/153): python3-markupsafe-0.23-19.el8.x86_64 770 kB/s | 39 kB 00:00 2024-03-18T18:46:42.438 INFO:teuthology.orchestra.run.smithi165.stdout:(58/153): python3-numpy-f2py-1.14.3-10.el8.x86_ 4.4 MB/s | 225 kB 00:00 2024-03-18T18:46:42.489 INFO:teuthology.orchestra.run.smithi165.stdout:(59/153): python3-prettytable-0.7.2-14.el8.noar 875 kB/s | 44 kB 00:00 2024-03-18T18:46:42.547 INFO:teuthology.orchestra.run.smithi165.stdout:(60/153): python3-pyOpenSSL-19.0.0-1.el8.noarch 1.7 MB/s | 103 kB 00:00 2024-03-18T18:46:42.656 INFO:teuthology.orchestra.run.smithi165.stdout:(61/153): python3-numpy-1.14.3-10.el8.x86_64.rp 13 MB/s | 3.7 MB 00:00 2024-03-18T18:46:42.681 INFO:teuthology.orchestra.run.smithi165.stdout:(62/153): python3-pyasn1-0.3.7-6.el8.noarch.rpm 945 kB/s | 126 kB 00:00 2024-03-18T18:46:42.707 INFO:teuthology.orchestra.run.smithi165.stdout:(63/153): python3-pyasn1-modules-0.3.7-6.el8.no 2.1 MB/s | 110 kB 00:00 2024-03-18T18:46:42.732 INFO:teuthology.orchestra.run.smithi165.stdout:(64/153): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2024-03-18T18:46:42.807 INFO:teuthology.orchestra.run.smithi165.stdout:(65/153): python3-werkzeug-0.12.2-4.el8.noarch. 6.0 MB/s | 457 kB 00:00 2024-03-18T18:46:42.891 INFO:teuthology.orchestra.run.smithi165.stdout:(66/153): socat-1.7.4.1-1.el8.x86_64.rpm 3.8 MB/s | 323 kB 00:00 2024-03-18T18:46:42.983 INFO:teuthology.orchestra.run.smithi165.stdout:(67/153): uuid-1.6.2-43.el8.x86_64.rpm 697 kB/s | 64 kB 00:00 2024-03-18T18:46:43.092 INFO:teuthology.orchestra.run.smithi165.stdout:(68/153): xmlstarlet-1.6.1-20.el8.x86_64.rpm 650 kB/s | 71 kB 00:00 2024-03-18T18:46:43.150 INFO:teuthology.orchestra.run.smithi165.stdout:(69/153): elfutils-libelf-devel-0.190-2.el8.x86 1.0 MB/s | 62 kB 00:00 2024-03-18T18:46:43.217 INFO:teuthology.orchestra.run.smithi165.stdout:(70/153): fuse-2.9.7-19.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2024-03-18T18:46:43.284 INFO:teuthology.orchestra.run.smithi165.stdout:(71/153): keyutils-libs-devel-1.5.10-9.el8.x86_ 721 kB/s | 48 kB 00:00 2024-03-18T18:46:43.347 INFO:teuthology.orchestra.run.smithi001.stdout:(10/153): ceph-radosgw-18.2.2-914.g49bdc511.el8 4.2 MB/s | 13 MB 00:03 2024-03-18T18:46:43.489 INFO:teuthology.orchestra.run.smithi001.stdout:(11/153): cephfs-mirror-18.2.2-914.g49bdc511.el 1.8 MB/s | 261 kB 00:00 2024-03-18T18:46:43.527 INFO:teuthology.orchestra.run.smithi165.stdout:(72/153): python3-scipy-1.0.0-21.module_el8+762 17 MB/s | 14 MB 00:00 2024-03-18T18:46:43.548 INFO:teuthology.orchestra.run.smithi001.stdout:(12/153): libcephfs-devel-18.2.2-914.g49bdc511. 530 kB/s | 31 kB 00:00 2024-03-18T18:46:43.568 INFO:teuthology.orchestra.run.smithi165.stdout:(73/153): krb5-devel-1.18.2-22.el8.x86_64.rpm 1.9 MB/s | 561 kB 00:00 2024-03-18T18:46:43.585 INFO:teuthology.orchestra.run.smithi165.stdout:(74/153): libaio-devel-0.3.112-1.el8.x86_64.rpm 319 kB/s | 19 kB 00:00 2024-03-18T18:46:43.619 INFO:teuthology.orchestra.run.smithi165.stdout:(75/153): libcom_err-devel-1.45.6-5.el8.x86_64. 771 kB/s | 39 kB 00:00 2024-03-18T18:46:43.694 INFO:teuthology.orchestra.run.smithi165.stdout:(76/153): libquadmath-8.5.0-21.el8.x86_64.rpm 2.3 MB/s | 172 kB 00:00 2024-03-18T18:46:43.761 INFO:teuthology.orchestra.run.smithi165.stdout:(77/153): librabbitmq-0.9.0-5.el8.x86_64.rpm 705 kB/s | 47 kB 00:00 2024-03-18T18:46:43.787 INFO:teuthology.orchestra.run.smithi165.stdout:(78/153): libgfortran-8.5.0-21.el8.x86_64.rpm 3.1 MB/s | 645 kB 00:00 2024-03-18T18:46:43.806 INFO:teuthology.orchestra.run.smithi001.stdout:(13/153): libcephfs2-18.2.2-914.g49bdc511.el8.x 2.9 MB/s | 775 kB 00:00 2024-03-18T18:46:43.853 INFO:teuthology.orchestra.run.smithi165.stdout:(79/153): libsepol-devel-2.9-3.el8.x86_64.rpm 1.3 MB/s | 87 kB 00:00 2024-03-18T18:46:43.887 INFO:teuthology.orchestra.run.smithi165.stdout:(80/153): libselinux-devel-2.9-8.el8.x86_64.rpm 1.6 MB/s | 200 kB 00:00 2024-03-18T18:46:43.890 INFO:teuthology.orchestra.run.smithi001.stdout:(14/153): libcephsqlite-18.2.2-914.g49bdc511.el 1.7 MB/s | 142 kB 00:00 2024-03-18T18:46:43.938 INFO:teuthology.orchestra.run.smithi165.stdout:(81/153): libverto-devel-0.3.2-2.el8.x86_64.rpm 361 kB/s | 18 kB 00:00 2024-03-18T18:46:43.963 INFO:teuthology.orchestra.run.smithi165.stdout:(82/153): libuuid-devel-2.32.1-46.el8.x86_64.rp 920 kB/s | 100 kB 00:00 2024-03-18T18:46:44.004 INFO:teuthology.orchestra.run.smithi001.stdout:(15/153): librados-devel-18.2.2-914.g49bdc511.e 1.2 MB/s | 139 kB 00:00 2024-03-18T18:46:44.013 INFO:teuthology.orchestra.run.smithi165.stdout:(83/153): libzstd-devel-1.4.4-1.el8.x86_64.rpm 583 kB/s | 44 kB 00:00 2024-03-18T18:46:44.138 INFO:teuthology.orchestra.run.smithi165.stdout:(84/153): pcre2-devel-10.32-3.el8.x86_64.rpm 4.7 MB/s | 605 kB 00:00 2024-03-18T18:46:44.157 INFO:teuthology.orchestra.run.smithi001.stdout:(16/153): libradosstriper1-18.2.2-914.g49bdc511 3.0 MB/s | 474 kB 00:00 2024-03-18T18:46:44.214 INFO:teuthology.orchestra.run.smithi165.stdout:(85/153): pcre2-utf16-10.32-3.el8.x86_64.rpm 3.0 MB/s | 229 kB 00:00 2024-03-18T18:46:44.345 INFO:teuthology.orchestra.run.smithi165.stdout:(86/153): openssl-devel-1.1.1k-12.el8.x86_64.rp 8.3 MB/s | 3.2 MB 00:00 2024-03-18T18:46:44.381 INFO:teuthology.orchestra.run.smithi165.stdout:(87/153): pcre2-utf32-10.32-3.el8.x86_64.rpm 1.3 MB/s | 220 kB 00:00 2024-03-18T18:46:44.448 INFO:teuthology.orchestra.run.smithi165.stdout:(88/153): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.4 MB/s | 249 kB 00:00 2024-03-18T18:46:44.506 INFO:teuthology.orchestra.run.smithi165.stdout:(89/153): python3-cryptography-3.2.1-7.el8.x86_ 5.0 MB/s | 641 kB 00:00 2024-03-18T18:46:44.532 INFO:teuthology.orchestra.run.smithi165.stdout:(90/153): python3-jwt-1.6.1-2.el8.noarch.rpm 510 kB/s | 43 kB 00:00 2024-03-18T18:46:44.590 INFO:teuthology.orchestra.run.smithi165.stdout:(91/153): python3-oauthlib-2.1.0-1.el8.noarch.r 1.8 MB/s | 155 kB 00:00 2024-03-18T18:46:44.617 INFO:teuthology.orchestra.run.smithi165.stdout:(92/153): python3-pycparser-2.14-14.el8.noarch. 1.3 MB/s | 109 kB 00:00 2024-03-18T18:46:44.666 INFO:teuthology.orchestra.run.smithi165.stdout:(93/153): python3-requests-oauthlib-1.0.0-1.el8 566 kB/s | 43 kB 00:00 2024-03-18T18:46:44.692 INFO:teuthology.orchestra.run.smithi165.stdout:(94/153): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 991 kB/s | 74 kB 00:00 2024-03-18T18:46:44.750 INFO:teuthology.orchestra.run.smithi165.stdout:(95/153): zlib-devel-1.2.11-21.el8.x86_64.rpm 694 kB/s | 58 kB 00:00 2024-03-18T18:46:44.776 INFO:teuthology.orchestra.run.smithi165.stdout:(96/153): python3-packaging-16.8-9.el8.noarch.r 575 kB/s | 48 kB 00:00 2024-03-18T18:46:44.918 INFO:teuthology.orchestra.run.smithi165.stdout:(97/153): liboath-2.6.2-3.el8.x86_64.rpm 415 kB/s | 59 kB 00:00 2024-03-18T18:46:44.968 INFO:teuthology.orchestra.run.smithi165.stdout:(98/153): gperftools-libs-2.7-9.el8.x86_64.rpm 1.4 MB/s | 306 kB 00:00 2024-03-18T18:46:44.993 INFO:teuthology.orchestra.run.smithi165.stdout:(99/153): libunwind-1.3.1-3.el8.x86_64.rpm 998 kB/s | 75 kB 00:00 2024-03-18T18:46:45.044 INFO:teuthology.orchestra.run.smithi165.stdout:(100/153): python3-bcrypt-3.1.6-2.el8.1.x86_64. 877 kB/s | 44 kB 00:00 2024-03-18T18:46:45.069 INFO:teuthology.orchestra.run.smithi165.stdout:(101/153): python3-asyncssh-2.7.0-2.el8.noarch. 4.6 MB/s | 476 kB 00:00 2024-03-18T18:46:45.111 INFO:teuthology.orchestra.run.smithi165.stdout:(102/153): python3-cachetools-3.1.1-4.el8.noarc 791 kB/s | 33 kB 00:00 2024-03-18T18:46:45.136 INFO:teuthology.orchestra.run.smithi165.stdout:(103/153): python3-beautifulsoup4-4.6.3-2.el8.1 2.0 MB/s | 185 kB 00:00 2024-03-18T18:46:45.162 INFO:teuthology.orchestra.run.smithi165.stdout:(104/153): python3-certifi-2018.10.15-7.el8.noa 325 kB/s | 16 kB 00:00 2024-03-18T18:46:45.195 INFO:teuthology.orchestra.run.smithi165.stdout:(105/153): python3-cheroot-8.5.2-1.el8.noarch.r 2.9 MB/s | 173 kB 00:00 2024-03-18T18:46:45.262 INFO:teuthology.orchestra.run.smithi165.stdout:(106/153): python3-cherrypy-18.4.0-1.el8.noarch 3.8 MB/s | 384 kB 00:00 2024-03-18T18:46:45.288 INFO:teuthology.orchestra.run.smithi165.stdout:(107/153): python3-google-auth-1.1.1-10.el8.noa 886 kB/s | 81 kB 00:00 2024-03-18T18:46:45.313 INFO:teuthology.orchestra.run.smithi165.stdout:(108/153): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2024-03-18T18:46:45.330 INFO:teuthology.orchestra.run.smithi165.stdout:(109/153): python3-jaraco-functools-2.0-4.el8.n 420 kB/s | 18 kB 00:00 2024-03-18T18:46:45.364 INFO:teuthology.orchestra.run.smithi165.stdout:(110/153): python3-logutils-0.3.5-11.el8.noarch 1.4 MB/s | 49 kB 00:00 2024-03-18T18:46:45.422 INFO:teuthology.orchestra.run.smithi165.stdout:(111/153): python3-kubernetes-11.0.0-6.el8.noar 11 MB/s | 1.2 MB 00:00 2024-03-18T18:46:45.439 INFO:teuthology.orchestra.run.smithi165.stdout:(112/153): python3-more-itertools-7.2.0-3.el8.n 780 kB/s | 59 kB 00:00 2024-03-18T18:46:45.464 INFO:teuthology.orchestra.run.smithi165.stdout:(113/153): python3-natsort-7.1.1-2.el8.noarch.r 1.4 MB/s | 60 kB 00:00 2024-03-18T18:46:45.466 INFO:teuthology.orchestra.run.smithi001.stdout:(17/153): ceph-osd-18.2.2-914.g49bdc511.el8.x86 2.6 MB/s | 18 MB 00:06 2024-03-18T18:46:45.506 INFO:teuthology.orchestra.run.smithi165.stdout:(114/153): python3-pecan-1.3.2-9.el8.noarch.rpm 4.1 MB/s | 283 kB 00:00 2024-03-18T18:46:45.532 INFO:teuthology.orchestra.run.smithi165.stdout:(115/153): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2024-03-18T18:46:45.541 INFO:teuthology.orchestra.run.smithi001.stdout:(18/153): python3-ceph-argparse-18.2.2-914.g49b 598 kB/s | 45 kB 00:00 2024-03-18T18:46:45.557 INFO:teuthology.orchestra.run.smithi165.stdout:(116/153): python3-repoze-lru-0.7-6.el8.noarch. 672 kB/s | 34 kB 00:00 2024-03-18T18:46:45.591 INFO:teuthology.orchestra.run.smithi165.stdout:(117/153): python3-routes-2.4.1-12.el8.noarch.r 3.3 MB/s | 196 kB 00:00 2024-03-18T18:46:45.608 INFO:teuthology.orchestra.run.smithi165.stdout:(118/153): python3-rsa-4.9-2.el8.noarch.rpm 1.2 MB/s | 64 kB 00:00 2024-03-18T18:46:45.633 INFO:teuthology.orchestra.run.smithi165.stdout:(119/153): python3-simplegeneric-0.8.1-17.el8.n 448 kB/s | 19 kB 00:00 2024-03-18T18:46:45.658 INFO:teuthology.orchestra.run.smithi165.stdout:(120/153): python3-singledispatch-3.4.0.3-18.el 477 kB/s | 24 kB 00:00 2024-03-18T18:46:45.691 INFO:teuthology.orchestra.run.smithi001.stdout:(19/153): python3-ceph-common-18.2.2-914.g49bdc 839 kB/s | 126 kB 00:00 2024-03-18T18:46:45.692 INFO:teuthology.orchestra.run.smithi165.stdout:(121/153): python3-tempora-1.14.1-5.el8.noarch. 496 kB/s | 29 kB 00:00 2024-03-18T18:46:45.718 INFO:teuthology.orchestra.run.smithi165.stdout:(122/153): python3-trustme-0.6.0-4.el8.noarch.r 466 kB/s | 27 kB 00:00 2024-03-18T18:46:45.743 INFO:teuthology.orchestra.run.smithi165.stdout:(123/153): python3-waitress-1.4.3-1.el8.noarch. 5.0 MB/s | 255 kB 00:00 2024-03-18T18:46:45.768 INFO:teuthology.orchestra.run.smithi165.stdout:(124/153): python3-webob-1.8.5-1.el8.1.noarch.r 4.9 MB/s | 251 kB 00:00 2024-03-18T18:46:45.817 INFO:teuthology.orchestra.run.smithi001.stdout:(20/153): librgw2-18.2.2-914.g49bdc511.el8.x86_ 3.9 MB/s | 6.5 MB 00:01 2024-03-18T18:46:45.842 INFO:teuthology.orchestra.run.smithi001.stdout:(21/153): python3-cephfs-18.2.2-914.g49bdc511.e 1.3 MB/s | 196 kB 00:00 2024-03-18T18:46:45.943 INFO:teuthology.orchestra.run.smithi001.stdout:(22/153): python3-rados-18.2.2-914.g49bdc511.el 2.9 MB/s | 370 kB 00:00 2024-03-18T18:46:45.968 INFO:teuthology.orchestra.run.smithi001.stdout:(23/153): python3-rbd-18.2.2-914.g49bdc511.el8. 2.8 MB/s | 365 kB 00:00 2024-03-18T18:46:46.043 INFO:teuthology.orchestra.run.smithi001.stdout:(24/153): python3-rgw-18.2.2-914.g49bdc511.el8. 1.0 MB/s | 107 kB 00:00 2024-03-18T18:46:46.068 INFO:teuthology.orchestra.run.smithi001.stdout:(25/153): rbd-fuse-18.2.2-914.g49bdc511.el8.x86 947 kB/s | 95 kB 00:00 2024-03-18T18:46:46.144 INFO:teuthology.orchestra.run.smithi001.stdout:(26/153): rbd-nbd-18.2.2-914.g49bdc511.el8.x86_ 2.4 MB/s | 182 kB 00:00 2024-03-18T18:46:46.202 INFO:teuthology.orchestra.run.smithi165.stdout:(125/153): ceph-test-18.2.2-914.g49bdc511.el8.x 4.1 MB/s | 47 MB 00:11 2024-03-18T18:46:46.219 INFO:teuthology.orchestra.run.smithi001.stdout:(27/153): ceph-grafana-dashboards-18.2.2-914.g4 311 kB/s | 23 kB 00:00 2024-03-18T18:46:46.227 INFO:teuthology.orchestra.run.smithi165.stdout:(126/153): python3-websocket-client-0.56.0-5.el 126 kB/s | 61 kB 00:00 2024-03-18T18:46:46.252 INFO:teuthology.orchestra.run.smithi165.stdout:(127/153): python3-webtest-2.0.33-1.el8.noarch. 176 kB/s | 85 kB 00:00 2024-03-18T18:46:46.277 INFO:teuthology.orchestra.run.smithi165.stdout:(128/153): python3-zc-lockfile-2.0-2.el8.noarch 459 kB/s | 23 kB 00:00 2024-03-18T18:46:46.294 INFO:teuthology.orchestra.run.smithi001.stdout:(28/153): ceph-mgr-cephadm-18.2.2-914.g49bdc511 1.8 MB/s | 138 kB 00:00 2024-03-18T18:46:46.469 INFO:teuthology.orchestra.run.smithi165.stdout:(129/153): thrift-0.13.0-2.el8.x86_64.rpm 7.2 MB/s | 1.7 MB 00:00 2024-03-18T18:46:46.845 INFO:teuthology.orchestra.run.smithi165.stdout:(130/153): NetworkManager-1.40.16-15.el8.x86_64 6.1 MB/s | 2.3 MB 00:00 2024-03-18T18:46:46.928 INFO:teuthology.orchestra.run.smithi001.stdout:(29/153): ceph-mgr-dashboard-18.2.2-914.g49bdc5 2.8 MB/s | 1.7 MB 00:00 2024-03-18T18:46:47.045 INFO:teuthology.orchestra.run.smithi001.stdout:(30/153): rbd-mirror-18.2.2-914.g49bdc511.el8.x 3.4 MB/s | 3.5 MB 00:01 2024-03-18T18:46:47.145 INFO:teuthology.orchestra.run.smithi001.stdout:(31/153): ceph-mgr-modules-core-18.2.2-914.g49b 2.4 MB/s | 245 kB 00:00 2024-03-18T18:46:47.178 INFO:teuthology.orchestra.run.smithi165.stdout:(131/153): NetworkManager-libnm-1.40.16-15.el8. 5.7 MB/s | 1.9 MB 00:00 2024-03-18T18:46:47.237 INFO:teuthology.orchestra.run.smithi001.stdout:(32/153): ceph-mgr-rook-18.2.2-914.g49bdc511.el 536 kB/s | 49 kB 00:00 2024-03-18T18:46:47.253 INFO:teuthology.orchestra.run.smithi165.stdout:(132/153): NetworkManager-team-1.40.16-15.el8.x 2.1 MB/s | 161 kB 00:00 2024-03-18T18:46:47.360 INFO:teuthology.orchestra.run.smithi001.stdout:(33/153): ceph-prometheus-alerts-18.2.2-914.g49 120 kB/s | 15 kB 00:00 2024-03-18T18:46:47.395 INFO:teuthology.orchestra.run.smithi165.stdout:(133/153): NetworkManager-tui-1.40.16-15.el8.x8 2.5 MB/s | 356 kB 00:00 2024-03-18T18:46:47.479 INFO:teuthology.orchestra.run.smithi165.stdout:(134/153): librbd1-18.2.2-914.g49bdc511.el8.x86 3.0 MB/s | 3.6 MB 00:01 2024-03-18T18:46:47.504 INFO:teuthology.orchestra.run.smithi165.stdout:(135/153): acl-2.2.53-3.el8.x86_64.rpm 762 kB/s | 83 kB 00:00 2024-03-18T18:46:47.546 INFO:teuthology.orchestra.run.smithi001.stdout:(34/153): ceph-volume-18.2.2-914.g49bdc511.el8. 1.4 MB/s | 263 kB 00:00 2024-03-18T18:46:47.554 INFO:teuthology.orchestra.run.smithi165.stdout:(136/153): elfutils-debuginfod-client-0.190-2.e 1.0 MB/s | 77 kB 00:00 2024-03-18T18:46:47.588 INFO:teuthology.orchestra.run.smithi165.stdout:(137/153): elfutils-libelf-0.190-2.el8.x86_64.r 2.7 MB/s | 233 kB 00:00 2024-03-18T18:46:47.638 INFO:teuthology.orchestra.run.smithi001.stdout:(35/153): cephadm-18.2.2-914.g49bdc511.el8.noar 1.8 MB/s | 172 kB 00:00 2024-03-18T18:46:47.672 INFO:teuthology.orchestra.run.smithi165.stdout:(138/153): elfutils-libs-0.190-2.el8.x86_64.rpm 2.5 MB/s | 305 kB 00:00 2024-03-18T18:46:47.721 INFO:teuthology.orchestra.run.smithi001.stdout:(36/153): cephfs-top-18.2.2-914.g49bdc511.el8.n 217 kB/s | 18 kB 00:00 2024-03-18T18:46:47.722 INFO:teuthology.orchestra.run.smithi165.stdout:(139/153): libacl-2.2.53-3.el8.x86_64.rpm 698 kB/s | 35 kB 00:00 2024-03-18T18:46:47.781 INFO:teuthology.orchestra.run.smithi165.stdout:(140/153): libacl-devel-2.2.53-3.el8.x86_64.rpm 1.4 MB/s | 85 kB 00:00 2024-03-18T18:46:47.881 INFO:teuthology.orchestra.run.smithi165.stdout:(141/153): iproute-6.2.0-5.el8.x86_64.rpm 2.9 MB/s | 854 kB 00:00 2024-03-18T18:46:47.907 INFO:teuthology.orchestra.run.smithi165.stdout:(142/153): libblkid-2.32.1-46.el8.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-03-18T18:46:47.972 INFO:teuthology.orchestra.run.smithi001.stdout:(37/153): automake-1.16.1-8.el8.noarch.rpm 2.9 MB/s | 740 kB 00:00 2024-03-18T18:46:48.072 INFO:teuthology.orchestra.run.smithi001.stdout:(38/153): bison-3.0.4-10.el8.x86_64.rpm 6.7 MB/s | 688 kB 00:00 2024-03-18T18:46:48.131 INFO:teuthology.orchestra.run.smithi001.stdout:(39/153): flex-2.6.1-9.el8.x86_64.rpm 5.4 MB/s | 320 kB 00:00 2024-03-18T18:46:48.157 INFO:teuthology.orchestra.run.smithi165.stdout:(143/153): libmount-2.32.1-46.el8.x86_64.rpm 948 kB/s | 237 kB 00:00 2024-03-18T18:46:48.189 INFO:teuthology.orchestra.run.smithi001.stdout:(40/153): jq-1.6-8.el8.x86_64.rpm 3.4 MB/s | 203 kB 00:00 2024-03-18T18:46:48.248 INFO:teuthology.orchestra.run.smithi001.stdout:(41/153): libpq-13.5-1.el8.x86_64.rpm 3.3 MB/s | 198 kB 00:00 2024-03-18T18:46:48.291 INFO:teuthology.orchestra.run.smithi165.stdout:(144/153): libfdisk-2.32.1-46.el8.x86_64.rpm 621 kB/s | 254 kB 00:00 2024-03-18T18:46:48.341 INFO:teuthology.orchestra.run.smithi165.stdout:(145/153): librados2-18.2.2-914.g49bdc511.el8.x 1.8 MB/s | 3.7 MB 00:02 2024-03-18T18:46:48.375 INFO:teuthology.orchestra.run.smithi165.stdout:(146/153): libsmartcols-2.32.1-46.el8.x86_64.rp 828 kB/s | 180 kB 00:00 2024-03-18T18:46:48.390 INFO:teuthology.orchestra.run.smithi001.stdout:(42/153): librdkafka-1.6.1-1.el8.x86_64.rpm 4.7 MB/s | 677 kB 00:00 2024-03-18T18:46:48.433 INFO:teuthology.orchestra.run.smithi165.stdout:(147/153): libuuid-2.32.1-46.el8.x86_64.rpm 698 kB/s | 99 kB 00:00 2024-03-18T18:46:48.490 INFO:teuthology.orchestra.run.smithi001.stdout:(43/153): libtool-2.4.6-25.el8.x86_64.rpm 6.9 MB/s | 709 kB 00:00 2024-03-18T18:46:48.492 INFO:teuthology.orchestra.run.smithi165.stdout:(148/153): openssl-1.1.1k-12.el8.x86_64.rpm 4.8 MB/s | 738 kB 00:00 2024-03-18T18:46:48.549 INFO:teuthology.orchestra.run.smithi001.stdout:(44/153): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.3 MB/s | 259 kB 00:00 2024-03-18T18:46:48.567 INFO:teuthology.orchestra.run.smithi165.stdout:(149/153): openssl-libs-1.1.1k-12.el8.x86_64.rp 7.7 MB/s | 1.5 MB 00:00 2024-03-18T18:46:48.593 INFO:teuthology.orchestra.run.smithi165.stdout:(150/153): util-linux-user-2.32.1-46.el8.x86_64 1.0 MB/s | 103 kB 00:00 2024-03-18T18:46:48.608 INFO:teuthology.orchestra.run.smithi001.stdout:(45/153): oniguruma-6.8.2-3.el8.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-03-18T18:46:48.718 INFO:teuthology.orchestra.run.smithi165.stdout:(151/153): xfsdump-3.1.8-7.el8.x86_64.rpm 2.2 MB/s | 334 kB 00:00 2024-03-18T18:46:48.843 INFO:teuthology.orchestra.run.smithi165.stdout:(152/153): util-linux-2.32.1-46.el8.x86_64.rpm 6.3 MB/s | 2.6 MB 00:00 2024-03-18T18:46:48.901 INFO:teuthology.orchestra.run.smithi165.stdout:(153/153): xfsprogs-5.0.0-12.el8.x86_64.rpm 3.5 MB/s | 1.1 MB 00:00 2024-03-18T18:46:48.905 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-03-18T18:46:48.905 INFO:teuthology.orchestra.run.smithi165.stdout:Total 13 MB/s | 218 MB 00:17 2024-03-18T18:46:48.917 INFO:teuthology.orchestra.run.smithi001.stdout:(46/153): openblas-0.3.15-4.el8.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-03-18T18:46:49.092 INFO:teuthology.orchestra.run.smithi001.stdout:(47/153): ceph-mgr-diskprediction-local-18.2.2- 3.4 MB/s | 7.4 MB 00:02 2024-03-18T18:46:49.209 INFO:teuthology.orchestra.run.smithi001.stdout:(48/153): openblas-threads-0.3.15-4.el8.x86_64. 17 MB/s | 4.9 MB 00:00 2024-03-18T18:46:49.293 INFO:teuthology.orchestra.run.smithi001.stdout:(49/153): postgresql-contrib-10.23-2.module_el8 11 MB/s | 901 kB 00:00 2024-03-18T18:46:49.476 INFO:teuthology.orchestra.run.smithi001.stdout:(50/153): postgresql-10.23-2.module_el8+572+929 4.1 MB/s | 1.6 MB 00:00 2024-03-18T18:46:49.610 INFO:teuthology.orchestra.run.smithi001.stdout:(51/153): postgresql-server-10.23-2.module_el8+ 16 MB/s | 5.1 MB 00:00 2024-03-18T18:46:49.685 INFO:teuthology.orchestra.run.smithi001.stdout:(52/153): python3-jinja2-2.10.1-4.el8.noarch.rp 7.6 MB/s | 578 kB 00:00 2024-03-18T18:46:49.760 INFO:teuthology.orchestra.run.smithi001.stdout:(53/153): python3-jsonpatch-1.21-2.el8.noarch.r 354 kB/s | 27 kB 00:00 2024-03-18T18:46:49.819 INFO:teuthology.orchestra.run.smithi001.stdout:(54/153): python3-jsonpointer-1.10-11.el8.noarc 342 kB/s | 20 kB 00:00 2024-03-18T18:46:49.886 INFO:teuthology.orchestra.run.smithi001.stdout:(55/153): python3-babel-2.5.1-7.el8.noarch.rpm 12 MB/s | 4.8 MB 00:00 2024-03-18T18:46:49.911 INFO:teuthology.orchestra.run.smithi001.stdout:(56/153): python3-mako-1.0.6-14.el8.noarch.rpm 1.7 MB/s | 157 kB 00:00 2024-03-18T18:46:49.927 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-03-18T18:46:49.945 INFO:teuthology.orchestra.run.smithi001.stdout:(57/153): python3-markupsafe-0.23-19.el8.x86_64 663 kB/s | 39 kB 00:00 2024-03-18T18:46:50.004 INFO:teuthology.orchestra.run.smithi001.stdout:(58/153): python3-numpy-f2py-1.14.3-10.el8.x86_ 3.8 MB/s | 225 kB 00:00 2024-03-18T18:46:50.087 INFO:teuthology.orchestra.run.smithi001.stdout:(59/153): python3-prettytable-0.7.2-14.el8.noar 526 kB/s | 44 kB 00:00 2024-03-18T18:46:50.146 INFO:teuthology.orchestra.run.smithi001.stdout:(60/153): python3-pyOpenSSL-19.0.0-1.el8.noarch 1.7 MB/s | 103 kB 00:00 2024-03-18T18:46:50.196 INFO:teuthology.orchestra.run.smithi001.stdout:(61/153): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2024-03-18T18:46:50.288 INFO:teuthology.orchestra.run.smithi001.stdout:(62/153): python3-numpy-1.14.3-10.el8.x86_64.rp 9.9 MB/s | 3.7 MB 00:00 2024-03-18T18:46:50.314 INFO:teuthology.orchestra.run.smithi001.stdout:(63/153): python3-pyasn1-modules-0.3.7-6.el8.no 940 kB/s | 110 kB 00:00 2024-03-18T18:46:50.339 INFO:teuthology.orchestra.run.smithi001.stdout:(64/153): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2024-03-18T18:46:50.401 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-03-18T18:46:50.401 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-03-18T18:46:50.431 INFO:teuthology.orchestra.run.smithi001.stdout:(65/153): python3-werkzeug-0.12.2-4.el8.noarch. 4.9 MB/s | 457 kB 00:00 2024-03-18T18:46:50.489 INFO:teuthology.orchestra.run.smithi001.stdout:(66/153): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2024-03-18T18:46:50.540 INFO:teuthology.orchestra.run.smithi001.stdout:(67/153): uuid-1.6.2-43.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-03-18T18:46:50.590 INFO:teuthology.orchestra.run.smithi001.stdout:(68/153): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-03-18T18:46:50.674 INFO:teuthology.orchestra.run.smithi001.stdout:(69/153): elfutils-libelf-devel-0.190-2.el8.x86 746 kB/s | 62 kB 00:00 2024-03-18T18:46:50.766 INFO:teuthology.orchestra.run.smithi001.stdout:(70/153): fuse-2.9.7-19.el8.x86_64.rpm 909 kB/s | 83 kB 00:00 2024-03-18T18:46:50.816 INFO:teuthology.orchestra.run.smithi001.stdout:(71/153): keyutils-libs-devel-1.5.10-9.el8.x86_ 967 kB/s | 48 kB 00:00 2024-03-18T18:46:50.992 INFO:teuthology.orchestra.run.smithi001.stdout:(72/153): krb5-devel-1.18.2-22.el8.x86_64.rpm 3.1 MB/s | 561 kB 00:00 2024-03-18T18:46:51.175 INFO:teuthology.orchestra.run.smithi001.stdout:(73/153): python3-scipy-1.0.0-21.module_el8+762 16 MB/s | 14 MB 00:00 2024-03-18T18:46:51.192 INFO:teuthology.orchestra.run.smithi001.stdout:(74/153): libaio-devel-0.3.112-1.el8.x86_64.rpm 93 kB/s | 19 kB 00:00 2024-03-18T18:46:51.226 INFO:teuthology.orchestra.run.smithi001.stdout:(75/153): libcom_err-devel-1.45.6-5.el8.x86_64. 777 kB/s | 39 kB 00:00 2024-03-18T18:46:51.310 INFO:teuthology.orchestra.run.smithi001.stdout:(76/153): libquadmath-8.5.0-21.el8.x86_64.rpm 2.0 MB/s | 172 kB 00:00 2024-03-18T18:46:51.418 INFO:teuthology.orchestra.run.smithi001.stdout:(77/153): libgfortran-8.5.0-21.el8.x86_64.rpm 2.8 MB/s | 645 kB 00:00 2024-03-18T18:46:51.435 INFO:teuthology.orchestra.run.smithi001.stdout:(78/153): librabbitmq-0.9.0-5.el8.x86_64.rpm 376 kB/s | 47 kB 00:00 2024-03-18T18:46:51.510 INFO:teuthology.orchestra.run.smithi001.stdout:(79/153): libselinux-devel-2.9-8.el8.x86_64.rpm 2.1 MB/s | 200 kB 00:00 2024-03-18T18:46:51.535 INFO:teuthology.orchestra.run.smithi001.stdout:(80/153): libsepol-devel-2.9-3.el8.x86_64.rpm 868 kB/s | 87 kB 00:00 2024-03-18T18:46:51.594 INFO:teuthology.orchestra.run.smithi001.stdout:(81/153): libverto-devel-0.3.2-2.el8.x86_64.rpm 311 kB/s | 18 kB 00:00 2024-03-18T18:46:51.619 INFO:teuthology.orchestra.run.smithi001.stdout:(82/153): libuuid-devel-2.32.1-46.el8.x86_64.rp 921 kB/s | 100 kB 00:00 2024-03-18T18:46:51.670 INFO:teuthology.orchestra.run.smithi001.stdout:(83/153): libzstd-devel-1.4.4-1.el8.x86_64.rpm 582 kB/s | 44 kB 00:00 2024-03-18T18:46:51.770 INFO:teuthology.orchestra.run.smithi001.stdout:(84/153): pcre2-devel-10.32-3.el8.x86_64.rpm 5.9 MB/s | 605 kB 00:00 2024-03-18T18:46:51.870 INFO:teuthology.orchestra.run.smithi001.stdout:(85/153): pcre2-utf16-10.32-3.el8.x86_64.rpm 2.2 MB/s | 229 kB 00:00 2024-03-18T18:46:51.947 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-03-18T18:46:51.950 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-03-18T18:46:51.971 INFO:teuthology.orchestra.run.smithi001.stdout:(86/153): pcre2-utf32-10.32-3.el8.x86_64.rpm 2.2 MB/s | 220 kB 00:00 2024-03-18T18:46:52.096 INFO:teuthology.orchestra.run.smithi001.stdout:(87/153): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.9 MB/s | 249 kB 00:00 2024-03-18T18:46:52.213 INFO:teuthology.orchestra.run.smithi001.stdout:(88/153): openssl-devel-1.1.1k-12.el8.x86_64.rp 5.4 MB/s | 3.2 MB 00:00 2024-03-18T18:46:52.271 INFO:teuthology.orchestra.run.smithi001.stdout:(89/153): python3-cryptography-3.2.1-7.el8.x86_ 3.6 MB/s | 641 kB 00:00 2024-03-18T18:46:52.347 INFO:teuthology.orchestra.run.smithi001.stdout:(90/153): python3-oauthlib-2.1.0-1.el8.noarch.r 2.0 MB/s | 155 kB 00:00 2024-03-18T18:46:52.822 INFO:teuthology.orchestra.run.smithi001.stdout:(91/153): ceph-test-18.2.2-914.g49bdc511.el8.x8 4.1 MB/s | 47 MB 00:11 2024-03-18T18:46:52.847 INFO:teuthology.orchestra.run.smithi001.stdout:(92/153): python3-jwt-1.6.1-2.el8.noarch.rpm 67 kB/s | 43 kB 00:00 2024-03-18T18:46:52.873 INFO:teuthology.orchestra.run.smithi001.stdout:(93/153): python3-pycparser-2.14-14.el8.noarch. 207 kB/s | 109 kB 00:00 2024-03-18T18:46:52.898 INFO:teuthology.orchestra.run.smithi001.stdout:(94/153): python3-requests-oauthlib-1.0.0-1.el8 568 kB/s | 43 kB 00:00 2024-03-18T18:46:52.957 INFO:teuthology.orchestra.run.smithi001.stdout:(95/153): zlib-devel-1.2.11-21.el8.x86_64.rpm 695 kB/s | 58 kB 00:00 2024-03-18T18:46:53.415 INFO:teuthology.orchestra.run.smithi001.stdout:(96/153): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 131 kB/s | 74 kB 00:00 2024-03-18T18:46:53.932 INFO:teuthology.orchestra.run.smithi001.stdout:(97/153): python3-packaging-16.8-9.el8.noarch.r 47 kB/s | 48 kB 00:01 2024-03-18T18:46:54.332 INFO:teuthology.orchestra.run.smithi001.stdout:(98/153): liboath-2.6.2-3.el8.x86_64.rpm 64 kB/s | 59 kB 00:00 2024-03-18T18:46:54.458 INFO:teuthology.orchestra.run.smithi001.stdout:(99/153): libunwind-1.3.1-3.el8.x86_64.rpm 143 kB/s | 75 kB 00:00 2024-03-18T18:46:54.608 INFO:teuthology.orchestra.run.smithi001.stdout:(100/153): python3-bcrypt-3.1.6-2.el8.1.x86_64. 290 kB/s | 44 kB 00:00 2024-03-18T18:46:54.633 INFO:teuthology.orchestra.run.smithi001.stdout:(101/153): gperftools-libs-2.7-9.el8.x86_64.rpm 183 kB/s | 306 kB 00:01 2024-03-18T18:46:54.909 INFO:teuthology.orchestra.run.smithi001.stdout:(102/153): python3-beautifulsoup4-4.6.3-2.el8.1 617 kB/s | 185 kB 00:00 2024-03-18T18:46:54.934 INFO:teuthology.orchestra.run.smithi001.stdout:(103/153): python3-asyncssh-2.7.0-2.el8.noarch. 791 kB/s | 476 kB 00:00 2024-03-18T18:46:55.001 INFO:teuthology.orchestra.run.smithi001.stdout:(104/153): python3-certifi-2018.10.15-7.el8.noa 177 kB/s | 16 kB 00:00 2024-03-18T18:46:55.143 INFO:teuthology.orchestra.run.smithi001.stdout:(105/153): python3-cheroot-8.5.2-1.el8.noarch.r 828 kB/s | 173 kB 00:00 2024-03-18T18:46:55.161 INFO:teuthology.orchestra.run.smithi001.stdout:(106/153): python3-cachetools-3.1.1-4.el8.noarc 63 kB/s | 33 kB 00:00 2024-03-18T18:46:55.268 INFO:teuthology.orchestra.run.smithi001.stdout:(107/153): python3-google-auth-1.1.1-10.el8.noa 648 kB/s | 81 kB 00:00 2024-03-18T18:46:55.302 INFO:teuthology.orchestra.run.smithi001.stdout:(108/153): python3-cherrypy-18.4.0-1.el8.noarch 1.2 MB/s | 384 kB 00:00 2024-03-18T18:46:55.327 INFO:teuthology.orchestra.run.smithi001.stdout:(109/153): python3-jaraco-6.2-6.el8.noarch.rpm 66 kB/s | 11 kB 00:00 2024-03-18T18:46:55.361 INFO:teuthology.orchestra.run.smithi001.stdout:(110/153): python3-jaraco-functools-2.0-4.el8.n 191 kB/s | 18 kB 00:00 2024-03-18T18:46:55.469 INFO:teuthology.orchestra.run.smithi001.stdout:(111/153): python3-more-itertools-7.2.0-3.el8.n 542 kB/s | 59 kB 00:00 2024-03-18T18:46:55.586 INFO:teuthology.orchestra.run.smithi001.stdout:(112/153): python3-logutils-0.3.5-11.el8.noarch 188 kB/s | 49 kB 00:00 2024-03-18T18:46:55.653 INFO:teuthology.orchestra.run.smithi001.stdout:(113/153): python3-natsort-7.1.1-2.el8.noarch.r 328 kB/s | 60 kB 00:00 2024-03-18T18:46:55.745 INFO:teuthology.orchestra.run.smithi001.stdout:(114/153): python3-kubernetes-11.0.0-6.el8.noar 2.6 MB/s | 1.2 MB 00:00 2024-03-18T18:46:55.771 INFO:teuthology.orchestra.run.smithi001.stdout:(115/153): python3-portend-2.6-1.el8.noarch.rpm 133 kB/s | 16 kB 00:00 2024-03-18T18:46:55.921 INFO:teuthology.orchestra.run.smithi001.stdout:(116/153): python3-routes-2.4.1-12.el8.noarch.r 1.3 MB/s | 196 kB 00:00 2024-03-18T18:46:55.946 INFO:teuthology.orchestra.run.smithi001.stdout:(117/153): python3-repoze-lru-0.7-6.el8.noarch. 168 kB/s | 34 kB 00:00 2024-03-18T18:46:56.013 INFO:teuthology.orchestra.run.smithi001.stdout:(118/153): python3-pecan-1.3.2-9.el8.noarch.rpm 663 kB/s | 283 kB 00:00 2024-03-18T18:46:56.038 INFO:teuthology.orchestra.run.smithi001.stdout:(119/153): python3-rsa-4.9-2.el8.noarch.rpm 548 kB/s | 64 kB 00:00 2024-03-18T18:46:56.063 INFO:teuthology.orchestra.run.smithi001.stdout:(120/153): python3-simplegeneric-0.8.1-17.el8.n 160 kB/s | 19 kB 00:00 2024-03-18T18:46:56.131 INFO:teuthology.orchestra.run.smithi001.stdout:(121/153): python3-singledispatch-3.4.0.3-18.el 204 kB/s | 24 kB 00:00 2024-03-18T18:46:56.147 INFO:teuthology.orchestra.run.smithi001.stdout:(122/153): python3-tempora-1.14.1-5.el8.noarch. 269 kB/s | 29 kB 00:00 2024-03-18T18:46:56.214 INFO:teuthology.orchestra.run.smithi001.stdout:(123/153): python3-trustme-0.6.0-4.el8.noarch.r 183 kB/s | 27 kB 00:00 2024-03-18T18:46:56.439 INFO:teuthology.orchestra.run.smithi001.stdout:(124/153): python3-webob-1.8.5-1.el8.1.noarch.r 860 kB/s | 251 kB 00:00 2024-03-18T18:46:56.456 INFO:teuthology.orchestra.run.smithi001.stdout:(125/153): python3-websocket-client-0.56.0-5.el 253 kB/s | 61 kB 00:00 2024-03-18T18:46:56.482 INFO:teuthology.orchestra.run.smithi001.stdout:(126/153): python3-waitress-1.4.3-1.el8.noarch. 727 kB/s | 255 kB 00:00 2024-03-18T18:46:56.624 INFO:teuthology.orchestra.run.smithi001.stdout:(127/153): python3-webtest-2.0.33-1.el8.noarch. 464 kB/s | 85 kB 00:00 2024-03-18T18:46:56.649 INFO:teuthology.orchestra.run.smithi001.stdout:(128/153): python3-zc-lockfile-2.0-2.el8.noarch 119 kB/s | 23 kB 00:00 2024-03-18T18:46:57.000 INFO:teuthology.orchestra.run.smithi001.stdout:(129/153): librados2-18.2.2-914.g49bdc511.el8.x 9.8 MB/s | 3.7 MB 00:00 2024-03-18T18:46:57.117 INFO:teuthology.orchestra.run.smithi001.stdout:(130/153): librbd1-18.2.2-914.g49bdc511.el8.x86 7.6 MB/s | 3.6 MB 00:00 2024-03-18T18:46:57.283 INFO:teuthology.orchestra.run.smithi001.stdout:(131/153): NetworkManager-libnm-1.40.16-15.el8. 11 MB/s | 1.9 MB 00:00 2024-03-18T18:46:57.359 INFO:teuthology.orchestra.run.smithi001.stdout:(132/153): NetworkManager-team-1.40.16-15.el8.x 2.1 MB/s | 161 kB 00:00 2024-03-18T18:46:57.651 INFO:teuthology.orchestra.run.smithi001.stdout:(133/153): NetworkManager-tui-1.40.16-15.el8.x8 1.2 MB/s | 356 kB 00:00 2024-03-18T18:46:57.693 INFO:teuthology.orchestra.run.smithi001.stdout:(134/153): NetworkManager-1.40.16-15.el8.x86_64 3.3 MB/s | 2.3 MB 00:00 2024-03-18T18:46:57.752 INFO:teuthology.orchestra.run.smithi001.stdout:(135/153): acl-2.2.53-3.el8.x86_64.rpm 827 kB/s | 83 kB 00:00 2024-03-18T18:46:57.885 INFO:teuthology.orchestra.run.smithi001.stdout:(136/153): elfutils-libelf-0.190-2.el8.x86_64.r 1.7 MB/s | 233 kB 00:00 2024-03-18T18:46:57.971 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-03-18T18:46:58.093 INFO:teuthology.orchestra.run.smithi001.stdout:(137/153): thrift-0.13.0-2.el8.x86_64.rpm 1.1 MB/s | 1.7 MB 00:01 2024-03-18T18:46:58.150 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8.x86_64 1/1 2024-03-18T18:46:58.269 INFO:teuthology.orchestra.run.smithi001.stdout:(138/153): iproute-6.2.0-5.el8.x86_64.rpm 4.8 MB/s | 854 kB 00:00 2024-03-18T18:46:58.327 INFO:teuthology.orchestra.run.smithi001.stdout:(139/153): libacl-2.2.53-3.el8.x86_64.rpm 598 kB/s | 35 kB 00:00 2024-03-18T18:46:58.349 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8.x86_64 1/177 2024-03-18T18:46:58.402 INFO:teuthology.orchestra.run.smithi001.stdout:(140/153): libacl-devel-2.2.53-3.el8.x86_64.rpm 1.1 MB/s | 85 kB 00:00 2024-03-18T18:46:58.537 INFO:teuthology.orchestra.run.smithi001.stdout:(141/153): libblkid-2.32.1-46.el8.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-03-18T18:46:58.574 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8.x86_64 1/177 2024-03-18T18:46:58.636 INFO:teuthology.orchestra.run.smithi001.stdout:(142/153): libfdisk-2.32.1-46.el8.x86_64.rpm 2.5 MB/s | 254 kB 00:00 2024-03-18T18:46:58.728 INFO:teuthology.orchestra.run.smithi001.stdout:(143/153): elfutils-libs-0.190-2.el8.x86_64.rpm 362 kB/s | 305 kB 00:00 2024-03-18T18:46:58.862 INFO:teuthology.orchestra.run.smithi001.stdout:(144/153): libsmartcols-2.32.1-46.el8.x86_64.rp 1.3 MB/s | 180 kB 00:00 2024-03-18T18:46:58.921 INFO:teuthology.orchestra.run.smithi001.stdout:(145/153): elfutils-debuginfod-client-0.190-2.e 62 kB/s | 77 kB 00:01 2024-03-18T18:46:58.946 INFO:teuthology.orchestra.run.smithi001.stdout:(146/153): libmount-2.32.1-46.el8.x86_64.rpm 767 kB/s | 237 kB 00:00 2024-03-18T18:46:58.971 INFO:teuthology.orchestra.run.smithi001.stdout:(147/153): libuuid-2.32.1-46.el8.x86_64.rpm 916 kB/s | 99 kB 00:00 2024-03-18T18:46:59.007 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/177 2024-03-18T18:46:59.096 INFO:teuthology.orchestra.run.smithi001.stdout:(148/153): openssl-libs-1.1.1k-12.el8.x86_64.rp 9.8 MB/s | 1.5 MB 00:00 2024-03-18T18:46:59.131 INFO:teuthology.orchestra.run.smithi001.stdout:(149/153): openssl-1.1.1k-12.el8.x86_64.rpm 3.4 MB/s | 738 kB 00:00 2024-03-18T18:46:59.141 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : libuuid-2.32.1-46.el8.x86_64 3/177 2024-03-18T18:46:59.214 INFO:teuthology.orchestra.run.smithi001.stdout:(150/153): util-linux-user-2.32.1-46.el8.x86_64 881 kB/s | 103 kB 00:00 2024-03-18T18:46:59.215 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libuuid-2.32.1-46.el8.x86_64 3/177 2024-03-18T18:46:59.272 INFO:teuthology.orchestra.run.smithi001.stdout:(151/153): util-linux-2.32.1-46.el8.x86_64.rpm 8.6 MB/s | 2.6 MB 00:00 2024-03-18T18:46:59.297 INFO:teuthology.orchestra.run.smithi001.stdout:(152/153): xfsdump-3.1.8-7.el8.x86_64.rpm 2.0 MB/s | 334 kB 00:00 2024-03-18T18:46:59.408 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : libblkid-2.32.1-46.el8.x86_64 4/177 2024-03-18T18:46:59.482 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libblkid-2.32.1-46.el8.x86_64 4/177 2024-03-18T18:46:59.710 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/177 2024-03-18T18:46:59.797 INFO:teuthology.orchestra.run.smithi001.stdout:(153/153): xfsprogs-5.0.0-12.el8.x86_64.rpm 1.9 MB/s | 1.1 MB 00:00 2024-03-18T18:46:59.801 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-18T18:46:59.801 INFO:teuthology.orchestra.run.smithi001.stdout:Total 8.7 MB/s | 218 MB 00:25 2024-03-18T18:46:59.947 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 6/177 2024-03-18T18:47:00.132 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/177 2024-03-18T18:47:00.359 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : elfutils-libelf-0.190-2.el8.x86_64 8/177 2024-03-18T18:47:00.507 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 9/177 2024-03-18T18:47:00.835 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-18T18:47:00.917 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 9/177 2024-03-18T18:47:01.099 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : librados2-2:18.2.2-914.g49bdc511.el8.x86_64 10/177 2024-03-18T18:47:01.339 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-18T18:47:01.339 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-18T18:47:01.479 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados2-2:18.2.2-914.g49bdc511.el8.x86_64 10/177 2024-03-18T18:47:01.659 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 11/177 2024-03-18T18:47:01.790 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 11/177 2024-03-18T18:47:02.007 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 12/177 2024-03-18T18:47:02.103 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 12/177 2024-03-18T18:47:02.447 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rados-2:18.2.2-914.g49bdc511.el8.x86_64 13/177 2024-03-18T18:47:02.711 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 14/177 2024-03-18T18:47:02.777 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 14/177 2024-03-18T18:47:02.927 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-18T18:47:02.930 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-18T18:47:02.943 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libpq-13.5-1.el8.x86_64 15/177 2024-03-18T18:47:03.119 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 16/177 2024-03-18T18:47:03.571 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 17/177 2024-03-18T18:47:03.794 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : postgresql-10.23-2.module_el8+572+929c87ac.x86_6 18/177 2024-03-18T18:47:03.957 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 19/177 2024-03-18T18:47:04.045 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 19/177 2024-03-18T18:47:04.231 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_ 20/177 2024-03-18T18:47:04.281 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_ 20/177 2024-03-18T18:47:04.752 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : libacl-2.2.53-3.el8.x86_64 21/177 2024-03-18T18:47:05.031 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : NetworkManager-libnm-1:1.40.16-15.el8.x86_64 22/177 2024-03-18T18:47:05.092 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: NetworkManager-libnm-1:1.40.16-15.el8.x86_64 22/177 2024-03-18T18:47:05.345 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: NetworkManager-1:1.40.16-15.el8.x86_64 23/177 2024-03-18T18:47:05.565 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : NetworkManager-1:1.40.16-15.el8.x86_64 23/177 2024-03-18T18:47:05.937 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: NetworkManager-1:1.40.16-15.el8.x86_64 23/177 2024-03-18T18:47:06.112 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 24/177 2024-03-18T18:47:06.262 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : zlib-devel-1.2.11-21.el8.x86_64 25/177 2024-03-18T18:47:06.398 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 26/177 2024-03-18T18:47:06.519 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 26/177 2024-03-18T18:47:06.706 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 27/177 2024-03-18T18:47:07.301 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 27/177 2024-03-18T18:47:07.539 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openblas-0.3.15-4.el8.x86_64 28/177 2024-03-18T18:47:08.140 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 28/177 2024-03-18T18:47:08.364 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 29/177 2024-03-18T18:47:09.038 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 29/177 2024-03-18T18:47:09.390 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 30/177 2024-03-18T18:47:09.578 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 31/177 2024-03-18T18:47:09.715 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-18T18:47:09.734 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 32/177 2024-03-18T18:47:09.905 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8.x86_64 1/1 2024-03-18T18:47:09.931 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 33/177 2024-03-18T18:47:10.170 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ceph-common-2:18.2.2-914.g49bdc511.el8.x 34/177 2024-03-18T18:47:10.171 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8.x86_64 1/177 2024-03-18T18:47:10.333 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ceph-argparse-2:18.2.2-914.g49bdc511.el8 35/177 2024-03-18T18:47:10.423 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8.x86_64 1/177 2024-03-18T18:47:10.566 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cephfs-2:18.2.2-914.g49bdc511.el8.x86_64 36/177 2024-03-18T18:47:10.595 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/177 2024-03-18T18:47:10.805 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libuuid-2.32.1-46.el8.x86_64 3/177 2024-03-18T18:47:10.891 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libuuid-2.32.1-46.el8.x86_64 3/177 2024-03-18T18:47:11.279 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libblkid-2.32.1-46.el8.x86_64 4/177 2024-03-18T18:47:11.355 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libblkid-2.32.1-46.el8.x86_64 4/177 2024-03-18T18:47:11.523 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 37/177 2024-03-18T18:47:11.637 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/177 2024-03-18T18:47:11.860 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 6/177 2024-03-18T18:47:11.868 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 38/177 2024-03-18T18:47:12.060 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/177 2024-03-18T18:47:12.125 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 39/177 2024-03-18T18:47:12.314 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 40/177 2024-03-18T18:47:12.338 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : elfutils-libelf-0.190-2.el8.x86_64 8/177 2024-03-18T18:47:12.538 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 9/177 2024-03-18T18:47:12.961 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 9/177 2024-03-18T18:47:13.321 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librados2-2:18.2.2-914.g49bdc511.el8.x86_64 10/177 2024-03-18T18:47:13.715 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:18.2.2-914.g49bdc511.el8.x86_64 10/177 2024-03-18T18:47:13.946 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 11/177 2024-03-18T18:47:14.082 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 11/177 2024-03-18T18:47:14.098 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 40/177 2024-03-18T18:47:14.321 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 12/177 2024-03-18T18:47:14.420 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 12/177 2024-03-18T18:47:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 41/177 2024-03-18T18:47:14.613 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 42/177 2024-03-18T18:47:14.762 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rados-2:18.2.2-914.g49bdc511.el8.x86_64 13/177 2024-03-18T18:47:14.831 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rsa-4.9-2.el8.noarch 43/177 2024-03-18T18:47:14.993 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rbd-2:18.2.2-914.g49bdc511.el8.x86_64 44/177 2024-03-18T18:47:15.133 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 14/177 2024-03-18T18:47:15.236 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librados-devel-2:18.2.2-914.g49bdc511.el8.x86_64 45/177 2024-03-18T18:47:15.260 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 14/177 2024-03-18T18:47:15.417 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libpq-13.5-1.el8.x86_64 15/177 2024-03-18T18:47:15.434 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : elfutils-libs-0.190-2.el8.x86_64 46/177 2024-03-18T18:47:15.608 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el8.x86_64 47/177 2024-03-18T18:47:15.663 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 16/177 2024-03-18T18:47:15.813 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : libfdisk-2.32.1-46.el8.x86_64 48/177 2024-03-18T18:47:15.889 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libfdisk-2.32.1-46.el8.x86_64 48/177 2024-03-18T18:47:16.130 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : libmount-2.32.1-46.el8.x86_64 49/177 2024-03-18T18:47:16.147 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 17/177 2024-03-18T18:47:16.199 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libmount-2.32.1-46.el8.x86_64 49/177 2024-03-18T18:47:16.353 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libuuid-devel-2.32.1-46.el8.x86_64 50/177 2024-03-18T18:47:16.425 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-10.23-2.module_el8+572+929c87ac.x86_6 18/177 2024-03-18T18:47:16.500 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 51/177 2024-03-18T18:47:16.620 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 19/177 2024-03-18T18:47:16.692 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : libsmartcols-2.32.1-46.el8.x86_64 52/177 2024-03-18T18:47:16.729 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 19/177 2024-03-18T18:47:16.954 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_ 20/177 2024-03-18T18:47:17.021 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_ 20/177 2024-03-18T18:47:17.175 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libsmartcols-2.32.1-46.el8.x86_64 52/177 2024-03-18T18:47:17.429 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : util-linux-2.32.1-46.el8.x86_64 53/177 2024-03-18T18:47:17.437 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libacl-2.2.53-3.el8.x86_64 21/177 2024-03-18T18:47:17.443 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: util-linux-2.32.1-46.el8.x86_64 53/177 2024-03-18T18:47:17.648 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 54/177 2024-03-18T18:47:17.704 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : NetworkManager-libnm-1:1.40.16-15.el8.x86_64 22/177 2024-03-18T18:47:17.765 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-libnm-1:1.40.16-15.el8.x86_64 22/177 2024-03-18T18:47:17.871 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 55/177 2024-03-18T18:47:18.061 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 56/177 2024-03-18T18:47:18.135 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-1:1.40.16-15.el8.x86_64 23/177 2024-03-18T18:47:18.225 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 57/177 2024-03-18T18:47:18.381 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 58/177 2024-03-18T18:47:18.463 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : NetworkManager-1:1.40.16-15.el8.x86_64 23/177 2024-03-18T18:47:18.620 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 59/177 2024-03-18T18:47:18.840 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 60/177 2024-03-18T18:47:18.858 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-1:1.40.16-15.el8.x86_64 23/177 2024-03-18T18:47:19.031 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 61/177 2024-03-18T18:47:19.110 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 24/177 2024-03-18T18:47:19.174 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 62/177 2024-03-18T18:47:19.302 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : zlib-devel-1.2.11-21.el8.x86_64 25/177 2024-03-18T18:47:19.377 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 63/177 2024-03-18T18:47:19.453 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 26/177 2024-03-18T18:47:19.541 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 64/177 2024-03-18T18:47:19.571 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 26/177 2024-03-18T18:47:19.685 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 65/177 2024-03-18T18:47:19.762 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 27/177 2024-03-18T18:47:20.388 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-portend-2.6-1.el8.noarch 66/177 2024-03-18T18:47:20.393 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 27/177 2024-03-18T18:47:20.554 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 67/177 2024-03-18T18:47:20.637 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-0.3.15-4.el8.x86_64 28/177 2024-03-18T18:47:20.761 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 68/177 2024-03-18T18:47:20.985 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 69/177 2024-03-18T18:47:21.223 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 70/177 2024-03-18T18:47:21.291 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 28/177 2024-03-18T18:47:21.461 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 29/177 2024-03-18T18:47:21.571 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 71/177 2024-03-18T18:47:21.821 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 72/177 2024-03-18T18:47:22.061 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 73/177 2024-03-18T18:47:22.183 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 29/177 2024-03-18T18:47:22.544 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 30/177 2024-03-18T18:47:22.793 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 31/177 2024-03-18T18:47:22.800 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 74/177 2024-03-18T18:47:22.946 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 32/177 2024-03-18T18:47:22.987 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 75/177 2024-03-18T18:47:23.062 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 75/177 2024-03-18T18:47:23.176 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 33/177 2024-03-18T18:47:23.247 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rgw-2:18.2.2-914.g49bdc511.el8.x86_64 76/177 2024-03-18T18:47:23.401 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-common-2:18.2.2-914.g49bdc511.el8.x 34/177 2024-03-18T18:47:23.488 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-packaging-16.8-9.el8.noarch 77/177 2024-03-18T18:47:23.566 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-argparse-2:18.2.2-914.g49bdc511.el8 35/177 2024-03-18T18:47:23.755 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 78/177 2024-03-18T18:47:23.866 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cephfs-2:18.2.2-914.g49bdc511.el8.x86_64 36/177 2024-03-18T18:47:24.101 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 79/177 2024-03-18T18:47:24.322 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 80/177 2024-03-18T18:47:24.530 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 81/177 2024-03-18T18:47:24.757 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 82/177 2024-03-18T18:47:24.928 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 37/177 2024-03-18T18:47:24.965 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 83/177 2024-03-18T18:47:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 38/177 2024-03-18T18:47:25.647 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 39/177 2024-03-18T18:47:25.886 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 40/177 2024-03-18T18:47:25.984 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 84/177 2024-03-18T18:47:26.361 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 85/177 2024-03-18T18:47:26.577 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 86/177 2024-03-18T18:47:26.785 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 87/177 2024-03-18T18:47:27.057 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 88/177 2024-03-18T18:47:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 89/177 2024-03-18T18:47:27.554 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 90/177 2024-03-18T18:47:27.747 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : pcre2-utf32-10.32-3.el8.x86_64 91/177 2024-03-18T18:47:27.787 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 40/177 2024-03-18T18:47:28.107 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : pcre2-utf16-10.32-3.el8.x86_64 92/177 2024-03-18T18:47:28.109 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 41/177 2024-03-18T18:47:28.296 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 42/177 2024-03-18T18:47:28.349 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : pcre2-devel-10.32-3.el8.x86_64 93/177 2024-03-18T18:47:28.504 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 94/177 2024-03-18T18:47:28.573 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rsa-4.9-2.el8.noarch 43/177 2024-03-18T18:47:28.682 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libverto-devel-0.3.2-2.el8.x86_64 95/177 2024-03-18T18:47:28.792 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rbd-2:18.2.2-914.g49bdc511.el8.x86_64 44/177 2024-03-18T18:47:29.002 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librados-devel-2:18.2.2-914.g49bdc511.el8.x86_64 45/177 2024-03-18T18:47:29.022 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 96/177 2024-03-18T18:47:29.226 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : elfutils-libs-0.190-2.el8.x86_64 46/177 2024-03-18T18:47:29.231 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libselinux-devel-2.9-8.el8.x86_64 97/177 2024-03-18T18:47:29.390 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcom_err-devel-1.45.6-5.el8.x86_64 98/177 2024-03-18T18:47:29.432 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el8.x86_64 47/177 2024-03-18T18:47:29.586 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 99/177 2024-03-18T18:47:29.619 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libfdisk-2.32.1-46.el8.x86_64 48/177 2024-03-18T18:47:29.720 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libfdisk-2.32.1-46.el8.x86_64 48/177 2024-03-18T18:47:29.831 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : krb5-devel-1.18.2-22.el8.x86_64 100/177 2024-03-18T18:47:29.936 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libmount-2.32.1-46.el8.x86_64 49/177 2024-03-18T18:47:29.995 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libmount-2.32.1-46.el8.x86_64 49/177 2024-03-18T18:47:30.045 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : fuse-2.9.7-19.el8.x86_64 101/177 2024-03-18T18:47:30.221 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libuuid-devel-2.32.1-46.el8.x86_64 50/177 2024-03-18T18:47:30.252 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 102/177 2024-03-18T18:47:30.440 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 51/177 2024-03-18T18:47:30.520 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : uuid-1.6.2-43.el8.x86_64 103/177 2024-03-18T18:47:30.652 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 103/177 2024-03-18T18:47:30.661 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libsmartcols-2.32.1-46.el8.x86_64 52/177 2024-03-18T18:47:30.794 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 104/177 2024-03-18T18:47:31.203 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libsmartcols-2.32.1-46.el8.x86_64 52/177 2024-03-18T18:47:31.519 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : util-linux-2.32.1-46.el8.x86_64 53/177 2024-03-18T18:47:31.534 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: util-linux-2.32.1-46.el8.x86_64 53/177 2024-03-18T18:47:31.707 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 54/177 2024-03-18T18:47:32.004 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 55/177 2024-03-18T18:47:32.235 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 56/177 2024-03-18T18:47:32.407 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 57/177 2024-03-18T18:47:32.606 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 58/177 2024-03-18T18:47:32.790 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 59/177 2024-03-18T18:47:33.016 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 60/177 2024-03-18T18:47:33.257 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 61/177 2024-03-18T18:47:33.285 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 105/177 2024-03-18T18:47:33.414 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 62/177 2024-03-18T18:47:33.502 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 105/177 2024-03-18T18:47:33.608 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 63/177 2024-03-18T18:47:33.765 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 64/177 2024-03-18T18:47:33.991 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 65/177 2024-03-18T18:47:34.083 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 105/177 2024-03-18T18:47:34.210 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-portend-2.6-1.el8.noarch 66/177 2024-03-18T18:47:34.360 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 106/177 2024-03-18T18:47:34.369 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 67/177 2024-03-18T18:47:34.470 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 106/177 2024-03-18T18:47:34.643 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 107/177 2024-03-18T18:47:34.647 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 68/177 2024-03-18T18:47:34.929 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 69/177 2024-03-18T18:47:35.115 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 70/177 2024-03-18T18:47:35.494 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 71/177 2024-03-18T18:47:35.744 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 72/177 2024-03-18T18:47:35.964 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 73/177 2024-03-18T18:47:36.763 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 74/177 2024-03-18T18:47:37.026 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 75/177 2024-03-18T18:47:37.090 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 75/177 2024-03-18T18:47:37.346 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rgw-2:18.2.2-914.g49bdc511.el8.x86_64 76/177 2024-03-18T18:47:37.653 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-packaging-16.8-9.el8.noarch 77/177 2024-03-18T18:47:37.885 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 78/177 2024-03-18T18:47:38.260 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 79/177 2024-03-18T18:47:38.497 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 80/177 2024-03-18T18:47:38.721 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 81/177 2024-03-18T18:47:39.005 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 82/177 2024-03-18T18:47:39.283 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 83/177 2024-03-18T18:47:40.394 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 84/177 2024-03-18T18:47:40.802 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 85/177 2024-03-18T18:47:41.042 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 86/177 2024-03-18T18:47:42.059 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 87/177 2024-03-18T18:47:42.452 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 88/177 2024-03-18T18:47:42.720 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 89/177 2024-03-18T18:47:43.006 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 90/177 2024-03-18T18:47:43.273 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : pcre2-utf32-10.32-3.el8.x86_64 91/177 2024-03-18T18:47:43.624 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : pcre2-utf16-10.32-3.el8.x86_64 92/177 2024-03-18T18:47:43.864 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : pcre2-devel-10.32-3.el8.x86_64 93/177 2024-03-18T18:47:44.036 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 94/177 2024-03-18T18:47:44.202 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libverto-devel-0.3.2-2.el8.x86_64 95/177 2024-03-18T18:47:44.554 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 96/177 2024-03-18T18:47:44.913 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libselinux-devel-2.9-8.el8.x86_64 97/177 2024-03-18T18:47:45.048 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcom_err-devel-1.45.6-5.el8.x86_64 98/177 2024-03-18T18:47:45.247 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 99/177 2024-03-18T18:47:45.503 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : krb5-devel-1.18.2-22.el8.x86_64 100/177 2024-03-18T18:47:45.753 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : fuse-2.9.7-19.el8.x86_64 101/177 2024-03-18T18:47:45.995 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 102/177 2024-03-18T18:47:46.158 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : uuid-1.6.2-43.el8.x86_64 103/177 2024-03-18T18:47:46.268 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 103/177 2024-03-18T18:47:46.417 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 104/177 2024-03-18T18:47:49.197 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 105/177 2024-03-18T18:47:49.476 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 105/177 2024-03-18T18:47:50.090 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 105/177 2024-03-18T18:47:50.442 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 106/177 2024-03-18T18:47:50.549 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 106/177 2024-03-18T18:47:50.899 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 107/177 2024-03-18T18:47:54.565 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 107/177 2024-03-18T18:47:54.565 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /sys 2024-03-18T18:47:54.565 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /proc 2024-03-18T18:47:54.565 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /dev 2024-03-18T18:47:54.565 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /run 2024-03-18T18:47:54.565 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /mnt 2024-03-18T18:47:54.565 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /var/tmp 2024-03-18T18:47:54.566 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /home 2024-03-18T18:47:54.566 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /tmp 2024-03-18T18:47:54.566 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /dev 2024-03-18T18:47:54.566 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:47:54.910 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 108/177 2024-03-18T18:47:56.469 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 108/177 2024-03-18T18:47:56.678 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 109/177 2024-03-18T18:47:56.892 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 109/177 2024-03-18T18:47:57.047 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 110/177 2024-03-18T18:47:57.503 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 110/177 2024-03-18T18:47:57.731 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 111/177 2024-03-18T18:47:57.754 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 111/177 2024-03-18T18:47:57.945 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 112/177 2024-03-18T18:47:58.141 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 113/177 2024-03-18T18:47:58.288 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 114/177 2024-03-18T18:47:58.405 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 114/177 2024-03-18T18:47:58.691 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : jq-1.6-8.el8.x86_64 115/177 2024-03-18T18:47:58.905 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : automake-1.16.1-8.el8.noarch 116/177 2024-03-18T18:47:59.432 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: cephadm-2:18.2.2-914.g49bdc511.el8.noarch 117/177 2024-03-18T18:47:59.600 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephadm-2:18.2.2-914.g49bdc511.el8.noarch 117/177 2024-03-18T18:47:59.736 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-914.g49bdc511.el 118/177 2024-03-18T18:47:59.923 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-914.g49bdc511.e 119/177 2024-03-18T18:48:00.188 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noar 120/177 2024-03-18T18:48:00.403 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noar 120/177 2024-03-18T18:48:00.655 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.no 121/177 2024-03-18T18:48:01.620 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.no 121/177 2024-03-18T18:48:01.846 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-914.g49bd 122/177 2024-03-18T18:48:01.941 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-914.g49bd 122/177 2024-03-18T18:48:02.277 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-914.g49bdc511.el8 123/177 2024-03-18T18:48:02.463 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 124/177 2024-03-18T18:48:02.500 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 124/177 2024-03-18T18:48:02.696 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 125/177 2024-03-18T18:48:02.707 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 125/177 2024-03-18T18:48:02.916 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-2:18.2.2-914.g49bdc511.el8.x86_64 126/177 2024-03-18T18:48:03.079 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libtool-2.4.6-25.el8.x86_64 127/177 2024-03-18T18:48:07.567 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 127/177 2024-03-18T18:48:08.998 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-test-2:18.2.2-914.g49bdc511.el8.x86_64 128/177 2024-03-18T18:48:09.195 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 129/177 2024-03-18T18:48:09.229 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 129/177 2024-03-18T18:48:09.470 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-914.g49bdc5 130/177 2024-03-18T18:48:09.512 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-914.g49bdc5 130/177 2024-03-18T18:48:09.711 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 131/177 2024-03-18T18:48:10.057 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 131/177 2024-03-18T18:48:10.320 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 132/177 2024-03-18T18:48:10.535 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 132/177 2024-03-18T18:48:10.851 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : postgresql-contrib-10.23-2.module_el8+572+929c87 133/177 2024-03-18T18:48:11.425 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 107/177 2024-03-18T18:48:11.426 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /sys 2024-03-18T18:48:11.426 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /proc 2024-03-18T18:48:11.426 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /dev 2024-03-18T18:48:11.426 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /run 2024-03-18T18:48:11.426 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /mnt 2024-03-18T18:48:11.426 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /var/tmp 2024-03-18T18:48:11.426 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /home 2024-03-18T18:48:11.426 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /tmp 2024-03-18T18:48:11.426 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /dev 2024-03-18T18:48:11.426 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:11.817 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 134/177 2024-03-18T18:48:11.822 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 108/177 2024-03-18T18:48:12.157 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openssl-devel-1:1.1.1k-12.el8.x86_64 135/177 2024-03-18T18:48:12.341 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : elfutils-libelf-devel-0.190-2.el8.x86_64 136/177 2024-03-18T18:48:12.555 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : util-linux-user-2.32.1-46.el8.x86_64 137/177 2024-03-18T18:48:12.732 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 138/177 2024-03-18T18:48:12.948 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephfs-devel-2:18.2.2-914.g49bdc511.el8.x86_6 139/177 2024-03-18T18:48:13.140 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : NetworkManager-team-1:1.40.16-15.el8.x86_64 140/177 2024-03-18T18:48:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : NetworkManager-tui-1:1.40.16-15.el8.x86_64 141/177 2024-03-18T18:48:13.482 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 108/177 2024-03-18T18:48:13.604 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : acl-2.2.53-3.el8.x86_64 142/177 2024-03-18T18:48:13.755 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 109/177 2024-03-18T18:48:13.828 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : libacl-devel-2.2.53-3.el8.x86_64 143/177 2024-03-18T18:48:13.988 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 109/177 2024-03-18T18:48:14.197 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 110/177 2024-03-18T18:48:14.678 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 110/177 2024-03-18T18:48:14.823 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: postgresql-server-10.23-2.module_el8+572+929c87a 144/177 2024-03-18T18:48:14.897 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 111/177 2024-03-18T18:48:14.922 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 111/177 2024-03-18T18:48:15.051 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : postgresql-server-10.23-2.module_el8+572+929c87a 144/177 2024-03-18T18:48:15.105 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: postgresql-server-10.23-2.module_el8+572+929c87a 144/177 2024-03-18T18:48:15.154 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 112/177 2024-03-18T18:48:15.356 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : xfsdump-3.1.8-7.el8.x86_64 145/177 2024-03-18T18:48:15.394 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 113/177 2024-03-18T18:48:15.544 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephfs-top-2:18.2.2-914.g49bdc511.el8.noarch 146/177 2024-03-18T18:48:15.680 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 114/177 2024-03-18T18:48:15.760 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 147/177 2024-03-18T18:48:15.763 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 114/177 2024-03-18T18:48:16.026 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : jq-1.6-8.el8.x86_64 115/177 2024-03-18T18:48:16.068 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-nbd-2:18.2.2-914.g49bdc511.el8.x86_64 148/177 2024-03-18T18:48:16.246 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : automake-1.16.1-8.el8.noarch 116/177 2024-03-18T18:48:16.447 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : iproute-6.2.0-5.el8.x86_64 149/177 2024-03-18T18:48:16.695 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : openssl-1:1.1.1k-12.el8.x86_64 150/177 2024-03-18T18:48:16.792 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:18.2.2-914.g49bdc511.el8.noarch 117/177 2024-03-18T18:48:16.877 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 151/177 2024-03-18T18:48:16.967 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephadm-2:18.2.2-914.g49bdc511.el8.noarch 117/177 2024-03-18T18:48:17.134 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flex-2.6.1-9.el8.x86_64 152/177 2024-03-18T18:48:17.154 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-914.g49bdc511.el 118/177 2024-03-18T18:48:17.219 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 152/177 2024-03-18T18:48:17.402 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-914.g49bdc511.e 119/177 2024-03-18T18:48:17.418 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : bison-3.0.4-10.el8.x86_64 153/177 2024-03-18T18:48:17.430 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 153/177 2024-03-18T18:48:17.562 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : NetworkManager-tui-1:1.40.16-4.el8.x86_64 154/177 2024-03-18T18:48:17.703 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : elfutils-debuginfod-client-0.189-1.el8.x86_64 155/177 2024-03-18T18:48:17.779 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noar 120/177 2024-03-18T18:48:17.889 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : elfutils-libs-0.189-1.el8.x86_64 156/177 2024-03-18T18:48:18.013 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noar 120/177 2024-03-18T18:48:18.054 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : openssl-1:1.1.1k-9.el8.x86_64 157/177 2024-03-18T18:48:18.213 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 158/177 2024-03-18T18:48:18.214 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 159/177 2024-03-18T18:48:18.221 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.no 121/177 2024-03-18T18:48:18.437 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 159/177 2024-03-18T18:48:18.438 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 160/177 2024-03-18T18:48:18.672 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 160/177 2024-03-18T18:48:18.672 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 161/177 2024-03-18T18:48:18.929 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 161/177 2024-03-18T18:48:19.078 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : acl-2.2.53-1.el8.x86_64 162/177 2024-03-18T18:48:19.206 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : libacl-devel-2.2.53-1.el8.x86_64 163/177 2024-03-18T18:48:19.241 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.no 121/177 2024-03-18T18:48:19.386 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : iproute-6.2.0-1.el8.x86_64 164/177 2024-03-18T18:48:19.579 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-914.g49bd 122/177 2024-03-18T18:48:19.598 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : util-linux-user-2.32.1-41.el8.x86_64 165/177 2024-03-18T18:48:19.677 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-914.g49bd 122/177 2024-03-18T18:48:19.828 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : util-linux-2.32.1-41.el8.x86_64 166/177 2024-03-18T18:48:19.828 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : libfdisk-2.32.1-41.el8.x86_64 167/177 2024-03-18T18:48:20.029 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libfdisk-2.32.1-41.el8.x86_64 167/177 2024-03-18T18:48:20.029 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : libmount-2.32.1-41.el8.x86_64 168/177 2024-03-18T18:48:20.067 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-914.g49bdc511.el8 123/177 2024-03-18T18:48:20.220 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libmount-2.32.1-41.el8.x86_64 168/177 2024-03-18T18:48:20.221 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : libblkid-2.32.1-41.el8.x86_64 169/177 2024-03-18T18:48:20.304 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 124/177 2024-03-18T18:48:20.343 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 124/177 2024-03-18T18:48:20.471 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libblkid-2.32.1-41.el8.x86_64 169/177 2024-03-18T18:48:20.546 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 125/177 2024-03-18T18:48:20.557 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 125/177 2024-03-18T18:48:20.611 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : NetworkManager-team-1:1.40.16-4.el8.x86_64 170/177 2024-03-18T18:48:20.621 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: NetworkManager-1:1.40.16-4.el8.x86_64 171/177 2024-03-18T18:48:20.621 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : NetworkManager-1:1.40.16-4.el8.x86_64 171/177 2024-03-18T18:48:20.814 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-2:18.2.2-914.g49bdc511.el8.x86_64 126/177 2024-03-18T18:48:21.045 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libtool-2.4.6-25.el8.x86_64 127/177 2024-03-18T18:48:21.063 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: NetworkManager-1:1.40.16-4.el8.x86_64 171/177 2024-03-18T18:48:21.063 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : NetworkManager-libnm-1:1.40.16-4.el8.x86_64 172/177 2024-03-18T18:48:21.337 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: NetworkManager-libnm-1:1.40.16-4.el8.x86_64 172/177 2024-03-18T18:48:21.337 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : libuuid-2.32.1-41.el8.x86_64 173/177 2024-03-18T18:48:21.511 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libuuid-2.32.1-41.el8.x86_64 173/177 2024-03-18T18:48:21.512 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : libsmartcols-2.32.1-41.el8.x86_64 174/177 2024-03-18T18:48:21.670 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libsmartcols-2.32.1-41.el8.x86_64 174/177 2024-03-18T18:48:21.827 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : elfutils-libelf-0.189-1.el8.x86_64 175/177 2024-03-18T18:48:21.962 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : libacl-2.2.53-1.el8.x86_64 176/177 2024-03-18T18:48:21.963 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : openssl-libs-1:1.1.1k-9.el8.x86_64 177/177 2024-03-18T18:48:22.875 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: openssl-libs-1:1.1.1k-9.el8.x86_64 177/177 2024-03-18T18:48:22.876 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-2:18.2.2-914.g49bdc511.el8.x86_64 1/177 2024-03-18T18:48:22.876 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 2/177 2024-03-18T18:48:22.876 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 3/177 2024-03-18T18:48:22.876 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 4/177 2024-03-18T18:48:22.876 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-914.g49bdc5 5/177 2024-03-18T18:48:22.876 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 6/177 2024-03-18T18:48:22.876 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 7/177 2024-03-18T18:48:22.876 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 8/177 2024-03-18T18:48:22.876 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 9/177 2024-03-18T18:48:22.876 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 10/177 2024-03-18T18:48:22.876 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 11/177 2024-03-18T18:48:22.876 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-test-2:18.2.2-914.g49bdc511.el8.x86_64 12/177 2024-03-18T18:48:22.877 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 13/177 2024-03-18T18:48:22.877 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephfs-devel-2:18.2.2-914.g49bdc511.el8.x86_6 14/177 2024-03-18T18:48:22.877 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 15/177 2024-03-18T18:48:22.877 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 16/177 2024-03-18T18:48:22.877 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados-devel-2:18.2.2-914.g49bdc511.el8.x86_64 17/177 2024-03-18T18:48:22.877 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_ 18/177 2024-03-18T18:48:22.877 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 19/177 2024-03-18T18:48:22.877 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ceph-argparse-2:18.2.2-914.g49bdc511.el8 20/177 2024-03-18T18:48:22.877 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ceph-common-2:18.2.2-914.g49bdc511.el8.x 21/177 2024-03-18T18:48:22.877 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cephfs-2:18.2.2-914.g49bdc511.el8.x86_64 22/177 2024-03-18T18:48:22.877 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rados-2:18.2.2-914.g49bdc511.el8.x86_64 23/177 2024-03-18T18:48:22.877 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rbd-2:18.2.2-914.g49bdc511.el8.x86_64 24/177 2024-03-18T18:48:22.877 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rgw-2:18.2.2-914.g49bdc511.el8.x86_64 25/177 2024-03-18T18:48:22.878 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 26/177 2024-03-18T18:48:22.878 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 27/177 2024-03-18T18:48:22.878 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-nbd-2:18.2.2-914.g49bdc511.el8.x86_64 28/177 2024-03-18T18:48:22.878 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-914.g49bdc511.e 29/177 2024-03-18T18:48:22.879 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noar 30/177 2024-03-18T18:48:22.879 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.no 31/177 2024-03-18T18:48:22.879 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-914.g49bd 32/177 2024-03-18T18:48:22.879 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-914.g49bdc511.el8 33/177 2024-03-18T18:48:22.879 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 34/177 2024-03-18T18:48:22.879 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-914.g49bdc511.el 35/177 2024-03-18T18:48:22.879 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 36/177 2024-03-18T18:48:22.879 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephadm-2:18.2.2-914.g49bdc511.el8.noarch 37/177 2024-03-18T18:48:22.879 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephfs-top-2:18.2.2-914.g49bdc511.el8.noarch 38/177 2024-03-18T18:48:22.879 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : automake-1.16.1-8.el8.noarch 39/177 2024-03-18T18:48:22.880 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : bison-3.0.4-10.el8.x86_64 40/177 2024-03-18T18:48:22.880 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flex-2.6.1-9.el8.x86_64 41/177 2024-03-18T18:48:22.880 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : jq-1.6-8.el8.x86_64 42/177 2024-03-18T18:48:22.880 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libpq-13.5-1.el8.x86_64 43/177 2024-03-18T18:48:22.880 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 44/177 2024-03-18T18:48:22.880 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 45/177 2024-03-18T18:48:22.880 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 46/177 2024-03-18T18:48:22.880 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 47/177 2024-03-18T18:48:22.880 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 48/177 2024-03-18T18:48:22.880 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 49/177 2024-03-18T18:48:22.880 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : postgresql-10.23-2.module_el8+572+929c87ac.x86_6 50/177 2024-03-18T18:48:22.880 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : postgresql-contrib-10.23-2.module_el8+572+929c87 51/177 2024-03-18T18:48:22.880 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : postgresql-server-10.23-2.module_el8+572+929c87a 52/177 2024-03-18T18:48:22.880 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 53/177 2024-03-18T18:48:22.881 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 54/177 2024-03-18T18:48:22.881 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 55/177 2024-03-18T18:48:22.881 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 56/177 2024-03-18T18:48:22.881 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 57/177 2024-03-18T18:48:22.881 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 58/177 2024-03-18T18:48:22.881 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 59/177 2024-03-18T18:48:22.881 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 60/177 2024-03-18T18:48:22.881 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 61/177 2024-03-18T18:48:22.881 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 62/177 2024-03-18T18:48:22.881 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 63/177 2024-03-18T18:48:22.881 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 64/177 2024-03-18T18:48:22.881 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 65/177 2024-03-18T18:48:22.881 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 66/177 2024-03-18T18:48:22.881 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 67/177 2024-03-18T18:48:22.882 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 68/177 2024-03-18T18:48:22.882 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 69/177 2024-03-18T18:48:22.882 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 70/177 2024-03-18T18:48:22.882 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : elfutils-libelf-devel-0.190-2.el8.x86_64 71/177 2024-03-18T18:48:22.882 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 72/177 2024-03-18T18:48:22.882 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 73/177 2024-03-18T18:48:22.882 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : krb5-devel-1.18.2-22.el8.x86_64 74/177 2024-03-18T18:48:22.883 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 75/177 2024-03-18T18:48:22.883 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcom_err-devel-1.45.6-5.el8.x86_64 76/177 2024-03-18T18:48:22.883 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 77/177 2024-03-18T18:48:22.883 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 78/177 2024-03-18T18:48:22.883 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 79/177 2024-03-18T18:48:22.883 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libselinux-devel-2.9-8.el8.x86_64 80/177 2024-03-18T18:48:22.883 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 81/177 2024-03-18T18:48:22.883 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libuuid-devel-2.32.1-46.el8.x86_64 82/177 2024-03-18T18:48:22.883 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libverto-devel-0.3.2-2.el8.x86_64 83/177 2024-03-18T18:48:22.883 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 84/177 2024-03-18T18:48:22.883 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8.x86_64 85/177 2024-03-18T18:48:22.883 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : pcre2-devel-10.32-3.el8.x86_64 86/177 2024-03-18T18:48:22.883 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : pcre2-utf16-10.32-3.el8.x86_64 87/177 2024-03-18T18:48:22.884 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : pcre2-utf32-10.32-3.el8.x86_64 88/177 2024-03-18T18:48:22.884 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 89/177 2024-03-18T18:48:22.884 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 90/177 2024-03-18T18:48:22.884 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 91/177 2024-03-18T18:48:22.884 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 92/177 2024-03-18T18:48:22.884 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 93/177 2024-03-18T18:48:22.884 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 94/177 2024-03-18T18:48:22.884 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 95/177 2024-03-18T18:48:22.884 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : zlib-devel-1.2.11-21.el8.x86_64 96/177 2024-03-18T18:48:22.884 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 97/177 2024-03-18T18:48:22.884 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 98/177 2024-03-18T18:48:22.884 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 99/177 2024-03-18T18:48:22.884 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 100/177 2024-03-18T18:48:22.884 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 101/177 2024-03-18T18:48:22.884 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 102/177 2024-03-18T18:48:22.885 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 103/177 2024-03-18T18:48:22.885 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 104/177 2024-03-18T18:48:22.885 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 105/177 2024-03-18T18:48:22.885 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 106/177 2024-03-18T18:48:22.885 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 107/177 2024-03-18T18:48:22.885 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 108/177 2024-03-18T18:48:22.885 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 109/177 2024-03-18T18:48:22.885 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 110/177 2024-03-18T18:48:22.885 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 111/177 2024-03-18T18:48:22.886 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 112/177 2024-03-18T18:48:22.886 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 113/177 2024-03-18T18:48:22.886 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 114/177 2024-03-18T18:48:22.886 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 115/177 2024-03-18T18:48:22.886 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-portend-2.6-1.el8.noarch 116/177 2024-03-18T18:48:22.886 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 117/177 2024-03-18T18:48:22.886 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 118/177 2024-03-18T18:48:22.886 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 119/177 2024-03-18T18:48:22.886 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 120/177 2024-03-18T18:48:22.886 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 121/177 2024-03-18T18:48:22.886 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 122/177 2024-03-18T18:48:22.886 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 123/177 2024-03-18T18:48:22.887 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 124/177 2024-03-18T18:48:22.887 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 125/177 2024-03-18T18:48:22.887 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 126/177 2024-03-18T18:48:22.887 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 127/177 2024-03-18T18:48:22.887 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 128/177 2024-03-18T18:48:22.887 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 129/177 2024-03-18T18:48:22.887 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados2-2:18.2.2-914.g49bdc511.el8.x86_64 130/177 2024-03-18T18:48:22.887 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 131/177 2024-03-18T18:48:22.887 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 132/177 2024-03-18T18:48:22.887 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 133/177 2024-03-18T18:48:22.887 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-1:1.40.16-15.el8.x86_64 134/177 2024-03-18T18:48:22.887 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-1:1.40.16-4.el8.x86_64 135/177 2024-03-18T18:48:22.887 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-libnm-1:1.40.16-15.el8.x86_64 136/177 2024-03-18T18:48:22.887 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-libnm-1:1.40.16-4.el8.x86_64 137/177 2024-03-18T18:48:22.888 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-team-1:1.40.16-15.el8.x86_64 138/177 2024-03-18T18:48:22.888 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-team-1:1.40.16-4.el8.x86_64 139/177 2024-03-18T18:48:22.888 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-tui-1:1.40.16-15.el8.x86_64 140/177 2024-03-18T18:48:22.888 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-tui-1:1.40.16-4.el8.x86_64 141/177 2024-03-18T18:48:22.888 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : acl-2.2.53-3.el8.x86_64 142/177 2024-03-18T18:48:22.888 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : acl-2.2.53-1.el8.x86_64 143/177 2024-03-18T18:48:22.888 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el8.x86_64 144/177 2024-03-18T18:48:22.888 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : elfutils-debuginfod-client-0.189-1.el8.x86_64 145/177 2024-03-18T18:48:22.888 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : elfutils-libelf-0.190-2.el8.x86_64 146/177 2024-03-18T18:48:22.888 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : elfutils-libelf-0.189-1.el8.x86_64 147/177 2024-03-18T18:48:22.888 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : elfutils-libs-0.190-2.el8.x86_64 148/177 2024-03-18T18:48:22.888 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : elfutils-libs-0.189-1.el8.x86_64 149/177 2024-03-18T18:48:22.888 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : iproute-6.2.0-5.el8.x86_64 150/177 2024-03-18T18:48:22.888 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : iproute-6.2.0-1.el8.x86_64 151/177 2024-03-18T18:48:22.889 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libacl-2.2.53-3.el8.x86_64 152/177 2024-03-18T18:48:22.889 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libacl-2.2.53-1.el8.x86_64 153/177 2024-03-18T18:48:22.889 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libacl-devel-2.2.53-3.el8.x86_64 154/177 2024-03-18T18:48:22.889 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libacl-devel-2.2.53-1.el8.x86_64 155/177 2024-03-18T18:48:22.889 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libblkid-2.32.1-46.el8.x86_64 156/177 2024-03-18T18:48:22.889 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libblkid-2.32.1-41.el8.x86_64 157/177 2024-03-18T18:48:22.889 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libfdisk-2.32.1-46.el8.x86_64 158/177 2024-03-18T18:48:22.889 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libfdisk-2.32.1-41.el8.x86_64 159/177 2024-03-18T18:48:22.889 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libmount-2.32.1-46.el8.x86_64 160/177 2024-03-18T18:48:22.889 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libmount-2.32.1-41.el8.x86_64 161/177 2024-03-18T18:48:22.889 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libsmartcols-2.32.1-46.el8.x86_64 162/177 2024-03-18T18:48:22.889 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libsmartcols-2.32.1-41.el8.x86_64 163/177 2024-03-18T18:48:22.889 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libuuid-2.32.1-46.el8.x86_64 164/177 2024-03-18T18:48:22.889 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libuuid-2.32.1-41.el8.x86_64 165/177 2024-03-18T18:48:22.890 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-1:1.1.1k-12.el8.x86_64 166/177 2024-03-18T18:48:22.890 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-1:1.1.1k-9.el8.x86_64 167/177 2024-03-18T18:48:22.890 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8.x86_64 168/177 2024-03-18T18:48:22.890 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-libs-1:1.1.1k-9.el8.x86_64 169/177 2024-03-18T18:48:22.890 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : util-linux-2.32.1-46.el8.x86_64 170/177 2024-03-18T18:48:22.890 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : util-linux-2.32.1-41.el8.x86_64 171/177 2024-03-18T18:48:22.890 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : util-linux-user-2.32.1-46.el8.x86_64 172/177 2024-03-18T18:48:22.890 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : util-linux-user-2.32.1-41.el8.x86_64 173/177 2024-03-18T18:48:22.890 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xfsdump-3.1.8-7.el8.x86_64 174/177 2024-03-18T18:48:22.890 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 175/177 2024-03-18T18:48:22.890 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 176/177 2024-03-18T18:48:25.472 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 177/177 2024-03-18T18:48:25.473 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:25.473 INFO:teuthology.orchestra.run.smithi165.stdout:Upgraded: 2024-03-18T18:48:25.473 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-1:1.40.16-15.el8.x86_64 2024-03-18T18:48:25.473 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-libnm-1:1.40.16-15.el8.x86_64 2024-03-18T18:48:25.473 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-team-1:1.40.16-15.el8.x86_64 2024-03-18T18:48:25.473 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-tui-1:1.40.16-15.el8.x86_64 2024-03-18T18:48:25.473 INFO:teuthology.orchestra.run.smithi165.stdout: acl-2.2.53-3.el8.x86_64 2024-03-18T18:48:25.473 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-debuginfod-client-0.190-2.el8.x86_64 2024-03-18T18:48:25.473 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-libelf-0.190-2.el8.x86_64 2024-03-18T18:48:25.473 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-libs-0.190-2.el8.x86_64 2024-03-18T18:48:25.473 INFO:teuthology.orchestra.run.smithi165.stdout: iproute-6.2.0-5.el8.x86_64 2024-03-18T18:48:25.473 INFO:teuthology.orchestra.run.smithi165.stdout: libacl-2.2.53-3.el8.x86_64 2024-03-18T18:48:25.474 INFO:teuthology.orchestra.run.smithi165.stdout: libacl-devel-2.2.53-3.el8.x86_64 2024-03-18T18:48:25.474 INFO:teuthology.orchestra.run.smithi165.stdout: libblkid-2.32.1-46.el8.x86_64 2024-03-18T18:48:25.474 INFO:teuthology.orchestra.run.smithi165.stdout: libfdisk-2.32.1-46.el8.x86_64 2024-03-18T18:48:25.474 INFO:teuthology.orchestra.run.smithi165.stdout: libmount-2.32.1-46.el8.x86_64 2024-03-18T18:48:25.474 INFO:teuthology.orchestra.run.smithi165.stdout: librados2-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.474 INFO:teuthology.orchestra.run.smithi165.stdout: librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.474 INFO:teuthology.orchestra.run.smithi165.stdout: libsmartcols-2.32.1-46.el8.x86_64 2024-03-18T18:48:25.474 INFO:teuthology.orchestra.run.smithi165.stdout: libuuid-2.32.1-46.el8.x86_64 2024-03-18T18:48:25.474 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-1:1.1.1k-12.el8.x86_64 2024-03-18T18:48:25.474 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-libs-1:1.1.1k-12.el8.x86_64 2024-03-18T18:48:25.474 INFO:teuthology.orchestra.run.smithi165.stdout: util-linux-2.32.1-46.el8.x86_64 2024-03-18T18:48:25.474 INFO:teuthology.orchestra.run.smithi165.stdout: util-linux-user-2.32.1-46.el8.x86_64 2024-03-18T18:48:25.474 INFO:teuthology.orchestra.run.smithi165.stdout: xfsdump-3.1.8-7.el8.x86_64 2024-03-18T18:48:25.475 INFO:teuthology.orchestra.run.smithi165.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-03-18T18:48:25.475 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-03-18T18:48:25.475 INFO:teuthology.orchestra.run.smithi165.stdout: automake-1.16.1-8.el8.noarch 2024-03-18T18:48:25.475 INFO:teuthology.orchestra.run.smithi165.stdout: bison-3.0.4-10.el8.x86_64 2024-03-18T18:48:25.475 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.475 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.475 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.475 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.475 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-grafana-dashboards-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:25.475 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-immutable-object-cache-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.475 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.475 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.475 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:25.476 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:25.476 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-diskprediction-local-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:25.476 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:25.476 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:25.476 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.476 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.476 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-prometheus-alerts-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:25.476 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.476 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.476 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-test-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.476 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:25.476 INFO:teuthology.orchestra.run.smithi165.stdout: cephadm-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:25.476 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.476 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-top-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:25.477 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-libelf-devel-0.190-2.el8.x86_64 2024-03-18T18:48:25.477 INFO:teuthology.orchestra.run.smithi165.stdout: flex-2.6.1-9.el8.x86_64 2024-03-18T18:48:25.477 INFO:teuthology.orchestra.run.smithi165.stdout: fuse-2.9.7-19.el8.x86_64 2024-03-18T18:48:25.477 INFO:teuthology.orchestra.run.smithi165.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-18T18:48:25.477 INFO:teuthology.orchestra.run.smithi165.stdout: jq-1.6-8.el8.x86_64 2024-03-18T18:48:25.477 INFO:teuthology.orchestra.run.smithi165.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-03-18T18:48:25.477 INFO:teuthology.orchestra.run.smithi165.stdout: krb5-devel-1.18.2-22.el8.x86_64 2024-03-18T18:48:25.477 INFO:teuthology.orchestra.run.smithi165.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-03-18T18:48:25.477 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-devel-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.477 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.477 INFO:teuthology.orchestra.run.smithi165.stdout: libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.477 INFO:teuthology.orchestra.run.smithi165.stdout: libcom_err-devel-1.45.6-5.el8.x86_64 2024-03-18T18:48:25.477 INFO:teuthology.orchestra.run.smithi165.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-03-18T18:48:25.477 INFO:teuthology.orchestra.run.smithi165.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-18T18:48:25.478 INFO:teuthology.orchestra.run.smithi165.stdout: libpq-13.5-1.el8.x86_64 2024-03-18T18:48:25.478 INFO:teuthology.orchestra.run.smithi165.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-03-18T18:48:25.478 INFO:teuthology.orchestra.run.smithi165.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-03-18T18:48:25.478 INFO:teuthology.orchestra.run.smithi165.stdout: librados-devel-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.478 INFO:teuthology.orchestra.run.smithi165.stdout: libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.478 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-03-18T18:48:25.478 INFO:teuthology.orchestra.run.smithi165.stdout: librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.478 INFO:teuthology.orchestra.run.smithi165.stdout: libselinux-devel-2.9-8.el8.x86_64 2024-03-18T18:48:25.478 INFO:teuthology.orchestra.run.smithi165.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-03-18T18:48:25.478 INFO:teuthology.orchestra.run.smithi165.stdout: libtool-2.4.6-25.el8.x86_64 2024-03-18T18:48:25.478 INFO:teuthology.orchestra.run.smithi165.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-18T18:48:25.478 INFO:teuthology.orchestra.run.smithi165.stdout: libuuid-devel-2.32.1-46.el8.x86_64 2024-03-18T18:48:25.478 INFO:teuthology.orchestra.run.smithi165.stdout: libverto-devel-0.3.2-2.el8.x86_64 2024-03-18T18:48:25.478 INFO:teuthology.orchestra.run.smithi165.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-03-18T18:48:25.479 INFO:teuthology.orchestra.run.smithi165.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-18T18:48:25.479 INFO:teuthology.orchestra.run.smithi165.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-03-18T18:48:25.479 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-0.3.15-4.el8.x86_64 2024-03-18T18:48:25.479 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-03-18T18:48:25.479 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-devel-1:1.1.1k-12.el8.x86_64 2024-03-18T18:48:25.479 INFO:teuthology.orchestra.run.smithi165.stdout: pcre2-devel-10.32-3.el8.x86_64 2024-03-18T18:48:25.479 INFO:teuthology.orchestra.run.smithi165.stdout: pcre2-utf16-10.32-3.el8.x86_64 2024-03-18T18:48:25.479 INFO:teuthology.orchestra.run.smithi165.stdout: pcre2-utf32-10.32-3.el8.x86_64 2024-03-18T18:48:25.479 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-10.23-2.module_el8+572+929c87ac.x86_64 2024-03-18T18:48:25.479 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-contrib-10.23-2.module_el8+572+929c87ac.x86_64 2024-03-18T18:48:25.479 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-server-10.23-2.module_el8+572+929c87ac.x86_64 2024-03-18T18:48:25.479 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-18T18:48:25.479 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-18T18:48:25.479 INFO:teuthology.orchestra.run.smithi165.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-18T18:48:25.480 INFO:teuthology.orchestra.run.smithi165.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-18T18:48:25.480 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-18T18:48:25.480 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.480 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.480 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cephfs-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.480 INFO:teuthology.orchestra.run.smithi165.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-18T18:48:25.480 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-18T18:48:25.480 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-18T18:48:25.480 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-18T18:48:25.480 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-03-18T18:48:25.480 INFO:teuthology.orchestra.run.smithi165.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-18T18:48:25.480 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-18T18:48:25.480 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-18T18:48:25.480 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-03-18T18:48:25.481 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-18T18:48:25.481 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-18T18:48:25.481 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-18T18:48:25.481 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-18T18:48:25.481 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-18T18:48:25.481 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-18T18:48:25.481 INFO:teuthology.orchestra.run.smithi165.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-18T18:48:25.481 INFO:teuthology.orchestra.run.smithi165.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-18T18:48:25.481 INFO:teuthology.orchestra.run.smithi165.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-18T18:48:25.481 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-18T18:48:25.481 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-18T18:48:25.482 INFO:teuthology.orchestra.run.smithi165.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-18T18:48:25.482 INFO:teuthology.orchestra.run.smithi165.stdout: python3-packaging-16.8-9.el8.noarch 2024-03-18T18:48:25.482 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-18T18:48:25.482 INFO:teuthology.orchestra.run.smithi165.stdout: python3-portend-2.6-1.el8.noarch 2024-03-18T18:48:25.482 INFO:teuthology.orchestra.run.smithi165.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-18T18:48:25.482 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-18T18:48:25.482 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-18T18:48:25.482 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-18T18:48:25.482 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-18T18:48:25.482 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-18T18:48:25.482 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rados-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.482 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rbd-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.482 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-18T18:48:25.482 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-18T18:48:25.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rgw-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-18T18:48:25.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-18T18:48:25.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-03-18T18:48:25.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-18T18:48:25.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-18T18:48:25.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-18T18:48:25.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-18T18:48:25.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-18T18:48:25.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-18T18:48:25.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-18T18:48:25.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-18T18:48:25.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-18T18:48:25.483 INFO:teuthology.orchestra.run.smithi165.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-18T18:48:25.484 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.484 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.484 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-nbd-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:25.484 INFO:teuthology.orchestra.run.smithi165.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-18T18:48:25.484 INFO:teuthology.orchestra.run.smithi165.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-18T18:48:25.484 INFO:teuthology.orchestra.run.smithi165.stdout: uuid-1.6.2-43.el8.x86_64 2024-03-18T18:48:25.484 INFO:teuthology.orchestra.run.smithi165.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-03-18T18:48:25.484 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-18T18:48:25.484 INFO:teuthology.orchestra.run.smithi165.stdout: zlib-devel-1.2.11-21.el8.x86_64 2024-03-18T18:48:25.484 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:25.484 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-03-18T18:48:25.811 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 127/177 2024-03-18T18:48:25.901 DEBUG:teuthology.parallel:result is None 2024-03-18T18:48:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-test-2:18.2.2-914.g49bdc511.el8.x86_64 128/177 2024-03-18T18:48:27.628 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 129/177 2024-03-18T18:48:27.663 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 129/177 2024-03-18T18:48:27.961 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-914.g49bdc5 130/177 2024-03-18T18:48:28.006 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-914.g49bdc5 130/177 2024-03-18T18:48:28.262 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 131/177 2024-03-18T18:48:28.630 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 131/177 2024-03-18T18:48:28.895 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 132/177 2024-03-18T18:48:29.139 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 132/177 2024-03-18T18:48:29.515 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-contrib-10.23-2.module_el8+572+929c87 133/177 2024-03-18T18:48:30.543 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 134/177 2024-03-18T18:48:30.924 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openssl-devel-1:1.1.1k-12.el8.x86_64 135/177 2024-03-18T18:48:31.291 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : elfutils-libelf-devel-0.190-2.el8.x86_64 136/177 2024-03-18T18:48:31.541 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : util-linux-user-2.32.1-46.el8.x86_64 137/177 2024-03-18T18:48:31.741 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 138/177 2024-03-18T18:48:31.973 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs-devel-2:18.2.2-914.g49bdc511.el8.x86_6 139/177 2024-03-18T18:48:32.159 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : NetworkManager-team-1:1.40.16-15.el8.x86_64 140/177 2024-03-18T18:48:32.401 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : NetworkManager-tui-1:1.40.16-15.el8.x86_64 141/177 2024-03-18T18:48:32.730 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : acl-2.2.53-3.el8.x86_64 142/177 2024-03-18T18:48:32.895 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : libacl-devel-2.2.53-3.el8.x86_64 143/177 2024-03-18T18:48:33.993 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: postgresql-server-10.23-2.module_el8+572+929c87a 144/177 2024-03-18T18:48:34.228 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-server-10.23-2.module_el8+572+929c87a 144/177 2024-03-18T18:48:34.297 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: postgresql-server-10.23-2.module_el8+572+929c87a 144/177 2024-03-18T18:48:34.549 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : xfsdump-3.1.8-7.el8.x86_64 145/177 2024-03-18T18:48:34.812 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephfs-top-2:18.2.2-914.g49bdc511.el8.noarch 146/177 2024-03-18T18:48:35.098 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 147/177 2024-03-18T18:48:35.505 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-nbd-2:18.2.2-914.g49bdc511.el8.x86_64 148/177 2024-03-18T18:48:35.911 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : iproute-6.2.0-5.el8.x86_64 149/177 2024-03-18T18:48:36.187 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : openssl-1:1.1.1k-12.el8.x86_64 150/177 2024-03-18T18:48:36.390 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 151/177 2024-03-18T18:48:36.610 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flex-2.6.1-9.el8.x86_64 152/177 2024-03-18T18:48:36.714 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 152/177 2024-03-18T18:48:36.935 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : bison-3.0.4-10.el8.x86_64 153/177 2024-03-18T18:48:36.949 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 153/177 2024-03-18T18:48:37.121 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : NetworkManager-tui-1:1.40.16-4.el8.x86_64 154/177 2024-03-18T18:48:37.296 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : elfutils-debuginfod-client-0.189-1.el8.x86_64 155/177 2024-03-18T18:48:37.515 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : elfutils-libs-0.189-1.el8.x86_64 156/177 2024-03-18T18:48:37.787 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : openssl-1:1.1.1k-9.el8.x86_64 157/177 2024-03-18T18:48:37.940 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 158/177 2024-03-18T18:48:37.940 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 159/177 2024-03-18T18:48:38.255 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 159/177 2024-03-18T18:48:38.255 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 160/177 2024-03-18T18:48:38.540 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 160/177 2024-03-18T18:48:38.540 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 161/177 2024-03-18T18:48:38.839 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 161/177 2024-03-18T18:48:39.004 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : acl-2.2.53-1.el8.x86_64 162/177 2024-03-18T18:48:39.165 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libacl-devel-2.2.53-1.el8.x86_64 163/177 2024-03-18T18:48:39.362 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : iproute-6.2.0-1.el8.x86_64 164/177 2024-03-18T18:48:39.561 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : util-linux-user-2.32.1-41.el8.x86_64 165/177 2024-03-18T18:48:39.821 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : util-linux-2.32.1-41.el8.x86_64 166/177 2024-03-18T18:48:39.821 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libfdisk-2.32.1-41.el8.x86_64 167/177 2024-03-18T18:48:40.030 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libfdisk-2.32.1-41.el8.x86_64 167/177 2024-03-18T18:48:40.030 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libmount-2.32.1-41.el8.x86_64 168/177 2024-03-18T18:48:40.279 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libmount-2.32.1-41.el8.x86_64 168/177 2024-03-18T18:48:40.280 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libblkid-2.32.1-41.el8.x86_64 169/177 2024-03-18T18:48:40.546 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libblkid-2.32.1-41.el8.x86_64 169/177 2024-03-18T18:48:40.670 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : NetworkManager-team-1:1.40.16-4.el8.x86_64 170/177 2024-03-18T18:48:40.680 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-1:1.40.16-4.el8.x86_64 171/177 2024-03-18T18:48:40.681 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : NetworkManager-1:1.40.16-4.el8.x86_64 171/177 2024-03-18T18:48:41.181 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-1:1.40.16-4.el8.x86_64 171/177 2024-03-18T18:48:41.181 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : NetworkManager-libnm-1:1.40.16-4.el8.x86_64 172/177 2024-03-18T18:48:41.379 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-libnm-1:1.40.16-4.el8.x86_64 172/177 2024-03-18T18:48:41.379 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libuuid-2.32.1-41.el8.x86_64 173/177 2024-03-18T18:48:41.612 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libuuid-2.32.1-41.el8.x86_64 173/177 2024-03-18T18:48:41.612 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libsmartcols-2.32.1-41.el8.x86_64 174/177 2024-03-18T18:48:41.846 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libsmartcols-2.32.1-41.el8.x86_64 174/177 2024-03-18T18:48:42.028 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : elfutils-libelf-0.189-1.el8.x86_64 175/177 2024-03-18T18:48:42.205 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : libacl-2.2.53-1.el8.x86_64 176/177 2024-03-18T18:48:42.205 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : openssl-libs-1:1.1.1k-9.el8.x86_64 177/177 2024-03-18T18:48:43.183 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: openssl-libs-1:1.1.1k-9.el8.x86_64 177/177 2024-03-18T18:48:43.183 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-2:18.2.2-914.g49bdc511.el8.x86_64 1/177 2024-03-18T18:48:43.183 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 2/177 2024-03-18T18:48:43.183 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 3/177 2024-03-18T18:48:43.183 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 4/177 2024-03-18T18:48:43.183 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-914.g49bdc5 5/177 2024-03-18T18:48:43.183 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 6/177 2024-03-18T18:48:43.183 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 7/177 2024-03-18T18:48:43.183 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 8/177 2024-03-18T18:48:43.183 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 9/177 2024-03-18T18:48:43.184 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 10/177 2024-03-18T18:48:43.184 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 11/177 2024-03-18T18:48:43.184 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-test-2:18.2.2-914.g49bdc511.el8.x86_64 12/177 2024-03-18T18:48:43.184 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 13/177 2024-03-18T18:48:43.184 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs-devel-2:18.2.2-914.g49bdc511.el8.x86_6 14/177 2024-03-18T18:48:43.184 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 15/177 2024-03-18T18:48:43.184 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 16/177 2024-03-18T18:48:43.184 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados-devel-2:18.2.2-914.g49bdc511.el8.x86_64 17/177 2024-03-18T18:48:43.185 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_ 18/177 2024-03-18T18:48:43.185 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 19/177 2024-03-18T18:48:43.185 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-argparse-2:18.2.2-914.g49bdc511.el8 20/177 2024-03-18T18:48:43.185 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-common-2:18.2.2-914.g49bdc511.el8.x 21/177 2024-03-18T18:48:43.185 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cephfs-2:18.2.2-914.g49bdc511.el8.x86_64 22/177 2024-03-18T18:48:43.185 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rados-2:18.2.2-914.g49bdc511.el8.x86_64 23/177 2024-03-18T18:48:43.185 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rbd-2:18.2.2-914.g49bdc511.el8.x86_64 24/177 2024-03-18T18:48:43.185 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rgw-2:18.2.2-914.g49bdc511.el8.x86_64 25/177 2024-03-18T18:48:43.185 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 26/177 2024-03-18T18:48:43.185 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 27/177 2024-03-18T18:48:43.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-nbd-2:18.2.2-914.g49bdc511.el8.x86_64 28/177 2024-03-18T18:48:43.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-914.g49bdc511.e 29/177 2024-03-18T18:48:43.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noar 30/177 2024-03-18T18:48:43.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.no 31/177 2024-03-18T18:48:43.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-914.g49bd 32/177 2024-03-18T18:48:43.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-914.g49bdc511.el8 33/177 2024-03-18T18:48:43.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 34/177 2024-03-18T18:48:43.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-914.g49bdc511.el 35/177 2024-03-18T18:48:43.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 36/177 2024-03-18T18:48:43.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephadm-2:18.2.2-914.g49bdc511.el8.noarch 37/177 2024-03-18T18:48:43.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephfs-top-2:18.2.2-914.g49bdc511.el8.noarch 38/177 2024-03-18T18:48:43.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : automake-1.16.1-8.el8.noarch 39/177 2024-03-18T18:48:43.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : bison-3.0.4-10.el8.x86_64 40/177 2024-03-18T18:48:43.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flex-2.6.1-9.el8.x86_64 41/177 2024-03-18T18:48:43.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : jq-1.6-8.el8.x86_64 42/177 2024-03-18T18:48:43.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libpq-13.5-1.el8.x86_64 43/177 2024-03-18T18:48:43.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 44/177 2024-03-18T18:48:43.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 45/177 2024-03-18T18:48:43.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 46/177 2024-03-18T18:48:43.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 47/177 2024-03-18T18:48:43.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 48/177 2024-03-18T18:48:43.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 49/177 2024-03-18T18:48:43.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-10.23-2.module_el8+572+929c87ac.x86_6 50/177 2024-03-18T18:48:43.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-contrib-10.23-2.module_el8+572+929c87 51/177 2024-03-18T18:48:43.202 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-server-10.23-2.module_el8+572+929c87a 52/177 2024-03-18T18:48:43.202 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 53/177 2024-03-18T18:48:43.202 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 54/177 2024-03-18T18:48:43.202 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 55/177 2024-03-18T18:48:43.202 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 56/177 2024-03-18T18:48:43.202 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 57/177 2024-03-18T18:48:43.202 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 58/177 2024-03-18T18:48:43.202 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 59/177 2024-03-18T18:48:43.202 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 60/177 2024-03-18T18:48:43.202 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 61/177 2024-03-18T18:48:43.202 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 62/177 2024-03-18T18:48:43.202 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 63/177 2024-03-18T18:48:43.202 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 64/177 2024-03-18T18:48:43.203 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 65/177 2024-03-18T18:48:43.203 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 66/177 2024-03-18T18:48:43.203 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 67/177 2024-03-18T18:48:43.203 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 68/177 2024-03-18T18:48:43.203 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 69/177 2024-03-18T18:48:43.203 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 70/177 2024-03-18T18:48:43.203 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-libelf-devel-0.190-2.el8.x86_64 71/177 2024-03-18T18:48:43.203 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 72/177 2024-03-18T18:48:43.203 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 73/177 2024-03-18T18:48:43.203 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : krb5-devel-1.18.2-22.el8.x86_64 74/177 2024-03-18T18:48:43.203 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 75/177 2024-03-18T18:48:43.203 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcom_err-devel-1.45.6-5.el8.x86_64 76/177 2024-03-18T18:48:43.203 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 77/177 2024-03-18T18:48:43.203 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 78/177 2024-03-18T18:48:43.204 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 79/177 2024-03-18T18:48:43.204 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libselinux-devel-2.9-8.el8.x86_64 80/177 2024-03-18T18:48:43.204 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 81/177 2024-03-18T18:48:43.204 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libuuid-devel-2.32.1-46.el8.x86_64 82/177 2024-03-18T18:48:43.204 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libverto-devel-0.3.2-2.el8.x86_64 83/177 2024-03-18T18:48:43.204 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 84/177 2024-03-18T18:48:43.204 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8.x86_64 85/177 2024-03-18T18:48:43.204 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : pcre2-devel-10.32-3.el8.x86_64 86/177 2024-03-18T18:48:43.204 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : pcre2-utf16-10.32-3.el8.x86_64 87/177 2024-03-18T18:48:43.204 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : pcre2-utf32-10.32-3.el8.x86_64 88/177 2024-03-18T18:48:43.204 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 89/177 2024-03-18T18:48:43.204 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 90/177 2024-03-18T18:48:43.204 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 91/177 2024-03-18T18:48:43.204 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 92/177 2024-03-18T18:48:43.205 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 93/177 2024-03-18T18:48:43.205 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 94/177 2024-03-18T18:48:43.205 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 95/177 2024-03-18T18:48:43.205 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : zlib-devel-1.2.11-21.el8.x86_64 96/177 2024-03-18T18:48:43.205 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 97/177 2024-03-18T18:48:43.205 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 98/177 2024-03-18T18:48:43.205 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 99/177 2024-03-18T18:48:43.205 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 100/177 2024-03-18T18:48:43.205 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 101/177 2024-03-18T18:48:43.205 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 102/177 2024-03-18T18:48:43.205 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 103/177 2024-03-18T18:48:43.205 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 104/177 2024-03-18T18:48:43.205 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 105/177 2024-03-18T18:48:43.205 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 106/177 2024-03-18T18:48:43.206 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 107/177 2024-03-18T18:48:43.206 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 108/177 2024-03-18T18:48:43.206 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 109/177 2024-03-18T18:48:43.206 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 110/177 2024-03-18T18:48:43.206 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 111/177 2024-03-18T18:48:43.206 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 112/177 2024-03-18T18:48:43.206 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 113/177 2024-03-18T18:48:43.206 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 114/177 2024-03-18T18:48:43.206 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 115/177 2024-03-18T18:48:43.207 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-portend-2.6-1.el8.noarch 116/177 2024-03-18T18:48:43.207 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 117/177 2024-03-18T18:48:43.207 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 118/177 2024-03-18T18:48:43.207 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 119/177 2024-03-18T18:48:43.207 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 120/177 2024-03-18T18:48:43.207 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 121/177 2024-03-18T18:48:43.207 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 122/177 2024-03-18T18:48:43.207 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 123/177 2024-03-18T18:48:43.207 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 124/177 2024-03-18T18:48:43.207 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 125/177 2024-03-18T18:48:43.207 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 126/177 2024-03-18T18:48:43.207 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 127/177 2024-03-18T18:48:43.207 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 128/177 2024-03-18T18:48:43.207 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 129/177 2024-03-18T18:48:43.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:18.2.2-914.g49bdc511.el8.x86_64 130/177 2024-03-18T18:48:43.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 131/177 2024-03-18T18:48:43.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 132/177 2024-03-18T18:48:43.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 133/177 2024-03-18T18:48:43.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-1:1.40.16-15.el8.x86_64 134/177 2024-03-18T18:48:43.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-1:1.40.16-4.el8.x86_64 135/177 2024-03-18T18:48:43.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-libnm-1:1.40.16-15.el8.x86_64 136/177 2024-03-18T18:48:43.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-libnm-1:1.40.16-4.el8.x86_64 137/177 2024-03-18T18:48:43.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-team-1:1.40.16-15.el8.x86_64 138/177 2024-03-18T18:48:43.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-team-1:1.40.16-4.el8.x86_64 139/177 2024-03-18T18:48:43.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-tui-1:1.40.16-15.el8.x86_64 140/177 2024-03-18T18:48:43.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-tui-1:1.40.16-4.el8.x86_64 141/177 2024-03-18T18:48:43.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : acl-2.2.53-3.el8.x86_64 142/177 2024-03-18T18:48:43.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : acl-2.2.53-1.el8.x86_64 143/177 2024-03-18T18:48:43.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el8.x86_64 144/177 2024-03-18T18:48:43.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-debuginfod-client-0.189-1.el8.x86_64 145/177 2024-03-18T18:48:43.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-libelf-0.190-2.el8.x86_64 146/177 2024-03-18T18:48:43.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-libelf-0.189-1.el8.x86_64 147/177 2024-03-18T18:48:43.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-libs-0.190-2.el8.x86_64 148/177 2024-03-18T18:48:43.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-libs-0.189-1.el8.x86_64 149/177 2024-03-18T18:48:43.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : iproute-6.2.0-5.el8.x86_64 150/177 2024-03-18T18:48:43.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : iproute-6.2.0-1.el8.x86_64 151/177 2024-03-18T18:48:43.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libacl-2.2.53-3.el8.x86_64 152/177 2024-03-18T18:48:43.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libacl-2.2.53-1.el8.x86_64 153/177 2024-03-18T18:48:43.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libacl-devel-2.2.53-3.el8.x86_64 154/177 2024-03-18T18:48:43.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libacl-devel-2.2.53-1.el8.x86_64 155/177 2024-03-18T18:48:43.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libblkid-2.32.1-46.el8.x86_64 156/177 2024-03-18T18:48:43.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libblkid-2.32.1-41.el8.x86_64 157/177 2024-03-18T18:48:43.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libfdisk-2.32.1-46.el8.x86_64 158/177 2024-03-18T18:48:43.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libfdisk-2.32.1-41.el8.x86_64 159/177 2024-03-18T18:48:43.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libmount-2.32.1-46.el8.x86_64 160/177 2024-03-18T18:48:43.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libmount-2.32.1-41.el8.x86_64 161/177 2024-03-18T18:48:43.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libsmartcols-2.32.1-46.el8.x86_64 162/177 2024-03-18T18:48:43.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libsmartcols-2.32.1-41.el8.x86_64 163/177 2024-03-18T18:48:43.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libuuid-2.32.1-46.el8.x86_64 164/177 2024-03-18T18:48:43.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libuuid-2.32.1-41.el8.x86_64 165/177 2024-03-18T18:48:43.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-1:1.1.1k-12.el8.x86_64 166/177 2024-03-18T18:48:43.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-1:1.1.1k-9.el8.x86_64 167/177 2024-03-18T18:48:43.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8.x86_64 168/177 2024-03-18T18:48:43.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-libs-1:1.1.1k-9.el8.x86_64 169/177 2024-03-18T18:48:43.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : util-linux-2.32.1-46.el8.x86_64 170/177 2024-03-18T18:48:43.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : util-linux-2.32.1-41.el8.x86_64 171/177 2024-03-18T18:48:43.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : util-linux-user-2.32.1-46.el8.x86_64 172/177 2024-03-18T18:48:43.211 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : util-linux-user-2.32.1-41.el8.x86_64 173/177 2024-03-18T18:48:43.211 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xfsdump-3.1.8-7.el8.x86_64 174/177 2024-03-18T18:48:43.211 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 175/177 2024-03-18T18:48:43.211 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 176/177 2024-03-18T18:48:45.435 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 177/177 2024-03-18T18:48:45.435 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:45.435 INFO:teuthology.orchestra.run.smithi001.stdout:Upgraded: 2024-03-18T18:48:45.436 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-1:1.40.16-15.el8.x86_64 2024-03-18T18:48:45.436 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-libnm-1:1.40.16-15.el8.x86_64 2024-03-18T18:48:45.436 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-team-1:1.40.16-15.el8.x86_64 2024-03-18T18:48:45.436 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-tui-1:1.40.16-15.el8.x86_64 2024-03-18T18:48:45.436 INFO:teuthology.orchestra.run.smithi001.stdout: acl-2.2.53-3.el8.x86_64 2024-03-18T18:48:45.436 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-debuginfod-client-0.190-2.el8.x86_64 2024-03-18T18:48:45.436 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libelf-0.190-2.el8.x86_64 2024-03-18T18:48:45.436 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libs-0.190-2.el8.x86_64 2024-03-18T18:48:45.436 INFO:teuthology.orchestra.run.smithi001.stdout: iproute-6.2.0-5.el8.x86_64 2024-03-18T18:48:45.437 INFO:teuthology.orchestra.run.smithi001.stdout: libacl-2.2.53-3.el8.x86_64 2024-03-18T18:48:45.437 INFO:teuthology.orchestra.run.smithi001.stdout: libacl-devel-2.2.53-3.el8.x86_64 2024-03-18T18:48:45.437 INFO:teuthology.orchestra.run.smithi001.stdout: libblkid-2.32.1-46.el8.x86_64 2024-03-18T18:48:45.437 INFO:teuthology.orchestra.run.smithi001.stdout: libfdisk-2.32.1-46.el8.x86_64 2024-03-18T18:48:45.437 INFO:teuthology.orchestra.run.smithi001.stdout: libmount-2.32.1-46.el8.x86_64 2024-03-18T18:48:45.437 INFO:teuthology.orchestra.run.smithi001.stdout: librados2-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.437 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.437 INFO:teuthology.orchestra.run.smithi001.stdout: libsmartcols-2.32.1-46.el8.x86_64 2024-03-18T18:48:45.437 INFO:teuthology.orchestra.run.smithi001.stdout: libuuid-2.32.1-46.el8.x86_64 2024-03-18T18:48:45.437 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-1:1.1.1k-12.el8.x86_64 2024-03-18T18:48:45.437 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-libs-1:1.1.1k-12.el8.x86_64 2024-03-18T18:48:45.437 INFO:teuthology.orchestra.run.smithi001.stdout: util-linux-2.32.1-46.el8.x86_64 2024-03-18T18:48:45.437 INFO:teuthology.orchestra.run.smithi001.stdout: util-linux-user-2.32.1-46.el8.x86_64 2024-03-18T18:48:45.437 INFO:teuthology.orchestra.run.smithi001.stdout: xfsdump-3.1.8-7.el8.x86_64 2024-03-18T18:48:45.438 INFO:teuthology.orchestra.run.smithi001.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-03-18T18:48:45.438 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-18T18:48:45.438 INFO:teuthology.orchestra.run.smithi001.stdout: automake-1.16.1-8.el8.noarch 2024-03-18T18:48:45.438 INFO:teuthology.orchestra.run.smithi001.stdout: bison-3.0.4-10.el8.x86_64 2024-03-18T18:48:45.438 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.438 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.438 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.438 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.438 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:45.438 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.438 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.439 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.439 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:45.439 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:45.439 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:45.439 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:45.439 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:45.439 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.439 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.439 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:45.439 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.439 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.439 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.439 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:45.439 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:45.440 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.440 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-top-2:18.2.2-914.g49bdc511.el8.noarch 2024-03-18T18:48:45.440 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libelf-devel-0.190-2.el8.x86_64 2024-03-18T18:48:45.440 INFO:teuthology.orchestra.run.smithi001.stdout: flex-2.6.1-9.el8.x86_64 2024-03-18T18:48:45.440 INFO:teuthology.orchestra.run.smithi001.stdout: fuse-2.9.7-19.el8.x86_64 2024-03-18T18:48:45.440 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-18T18:48:45.440 INFO:teuthology.orchestra.run.smithi001.stdout: jq-1.6-8.el8.x86_64 2024-03-18T18:48:45.440 INFO:teuthology.orchestra.run.smithi001.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-03-18T18:48:45.440 INFO:teuthology.orchestra.run.smithi001.stdout: krb5-devel-1.18.2-22.el8.x86_64 2024-03-18T18:48:45.440 INFO:teuthology.orchestra.run.smithi001.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-03-18T18:48:45.440 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.440 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.441 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.441 INFO:teuthology.orchestra.run.smithi001.stdout: libcom_err-devel-1.45.6-5.el8.x86_64 2024-03-18T18:48:45.441 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-03-18T18:48:45.441 INFO:teuthology.orchestra.run.smithi001.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-18T18:48:45.441 INFO:teuthology.orchestra.run.smithi001.stdout: libpq-13.5-1.el8.x86_64 2024-03-18T18:48:45.441 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-03-18T18:48:45.441 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-03-18T18:48:45.441 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.441 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.441 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-03-18T18:48:45.441 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.441 INFO:teuthology.orchestra.run.smithi001.stdout: libselinux-devel-2.9-8.el8.x86_64 2024-03-18T18:48:45.441 INFO:teuthology.orchestra.run.smithi001.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-03-18T18:48:45.442 INFO:teuthology.orchestra.run.smithi001.stdout: libtool-2.4.6-25.el8.x86_64 2024-03-18T18:48:45.442 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-18T18:48:45.442 INFO:teuthology.orchestra.run.smithi001.stdout: libuuid-devel-2.32.1-46.el8.x86_64 2024-03-18T18:48:45.442 INFO:teuthology.orchestra.run.smithi001.stdout: libverto-devel-0.3.2-2.el8.x86_64 2024-03-18T18:48:45.442 INFO:teuthology.orchestra.run.smithi001.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-03-18T18:48:45.442 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-18T18:48:45.442 INFO:teuthology.orchestra.run.smithi001.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-03-18T18:48:45.442 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-0.3.15-4.el8.x86_64 2024-03-18T18:48:45.442 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-03-18T18:48:45.442 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-devel-1:1.1.1k-12.el8.x86_64 2024-03-18T18:48:45.442 INFO:teuthology.orchestra.run.smithi001.stdout: pcre2-devel-10.32-3.el8.x86_64 2024-03-18T18:48:45.442 INFO:teuthology.orchestra.run.smithi001.stdout: pcre2-utf16-10.32-3.el8.x86_64 2024-03-18T18:48:45.442 INFO:teuthology.orchestra.run.smithi001.stdout: pcre2-utf32-10.32-3.el8.x86_64 2024-03-18T18:48:45.442 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-10.23-2.module_el8+572+929c87ac.x86_64 2024-03-18T18:48:45.443 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-contrib-10.23-2.module_el8+572+929c87ac.x86_64 2024-03-18T18:48:45.443 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-server-10.23-2.module_el8+572+929c87ac.x86_64 2024-03-18T18:48:45.443 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-18T18:48:45.443 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-18T18:48:45.443 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-18T18:48:45.443 INFO:teuthology.orchestra.run.smithi001.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-18T18:48:45.443 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-18T18:48:45.443 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.443 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.443 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.443 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-18T18:48:45.443 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-18T18:48:45.444 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-18T18:48:45.444 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-18T18:48:45.444 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-03-18T18:48:45.444 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-18T18:48:45.444 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-18T18:48:45.444 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-18T18:48:45.444 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-03-18T18:48:45.444 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-18T18:48:45.444 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-18T18:48:45.444 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-18T18:48:45.444 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-18T18:48:45.444 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-18T18:48:45.444 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-18T18:48:45.444 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-18T18:48:45.445 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-18T18:48:45.445 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-18T18:48:45.445 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-18T18:48:45.445 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-18T18:48:45.445 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-18T18:48:45.445 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging-16.8-9.el8.noarch 2024-03-18T18:48:45.445 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-18T18:48:45.445 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend-2.6-1.el8.noarch 2024-03-18T18:48:45.445 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-18T18:48:45.445 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-18T18:48:45.445 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-18T18:48:45.445 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-18T18:48:45.445 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-18T18:48:45.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-18T18:48:45.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-18T18:48:45.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-18T18:48:45.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-18T18:48:45.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-18T18:48:45.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-03-18T18:48:45.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-18T18:48:45.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-18T18:48:45.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-18T18:48:45.446 INFO:teuthology.orchestra.run.smithi001.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-18T18:48:45.447 INFO:teuthology.orchestra.run.smithi001.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-18T18:48:45.447 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-18T18:48:45.447 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-18T18:48:45.447 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-18T18:48:45.447 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-18T18:48:45.447 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-18T18:48:45.447 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.447 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.447 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd-2:18.2.2-914.g49bdc511.el8.x86_64 2024-03-18T18:48:45.447 INFO:teuthology.orchestra.run.smithi001.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-18T18:48:45.447 INFO:teuthology.orchestra.run.smithi001.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-18T18:48:45.447 INFO:teuthology.orchestra.run.smithi001.stdout: uuid-1.6.2-43.el8.x86_64 2024-03-18T18:48:45.447 INFO:teuthology.orchestra.run.smithi001.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-03-18T18:48:45.447 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-18T18:48:45.448 INFO:teuthology.orchestra.run.smithi001.stdout: zlib-devel-1.2.11-21.el8.x86_64 2024-03-18T18:48:45.448 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:45.448 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-18T18:48:45.678 DEBUG:teuthology.parallel:result is None 2024-03-18T18:48:45.679 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=49bdc51142759622a281d3e5327eebd07cbb8bb3 2024-03-18T18:48:45.924 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T18:48:45.949 INFO:teuthology.orchestra.run.smithi001.stdout:18.2.2-914.g49bdc511.el8 2024-03-18T18:48:45.950 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-914.g49bdc511.el8 2024-03-18T18:48:45.950 INFO:teuthology.task.install:The correct ceph version 18.2.2-914.g49bdc511 is installed. 2024-03-18T18:48:45.952 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=49bdc51142759622a281d3e5327eebd07cbb8bb3 2024-03-18T18:48:46.093 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T18:48:46.122 INFO:teuthology.orchestra.run.smithi165.stdout:18.2.2-914.g49bdc511.el8 2024-03-18T18:48:46.122 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-914.g49bdc511.el8 2024-03-18T18:48:46.122 INFO:teuthology.task.install:The correct ceph version 18.2.2-914.g49bdc511 is installed. 2024-03-18T18:48:46.124 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-18T18:48:46.124 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:48:46.124 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T18:48:46.157 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:48:46.157 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T18:48:46.196 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-18T18:48:46.197 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:48:46.197 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T18:48:46.225 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T18:48:46.294 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:48:46.294 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T18:48:46.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T18:48:46.393 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-18T18:48:46.394 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:48:46.394 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T18:48:46.421 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T18:48:46.489 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:48:46.489 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T18:48:46.517 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T18:48:46.586 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-18T18:48:46.586 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:48:46.586 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T18:48:46.613 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T18:48:46.680 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:48:46.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T18:48:46.708 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T18:48:46.775 INFO:teuthology.run_tasks:Running task ceph... 2024-03-18T18:48:46.850 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-18T18:48:46.851 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 777 /var/log/ceph 2024-03-18T18:48:46.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 777 /var/log/ceph 2024-03-18T18:48:46.881 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-18T18:48:46.881 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-18T18:48:46.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-18T18:48:46.965 INFO:tasks.ceph:Creating extra log directories... 2024-03-18T18:48:46.965 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-18T18:48:46.967 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-18T18:48:47.022 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-18T18:48:47.023 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-18T18:48:47.023 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-18T18:48:47.023 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:48:47.023 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-18T18:48:47.211 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-18T18:48:47.287 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-18T18:48:47.316 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:48:47.316 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-18T18:48:47.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-18T18:48:47.448 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-18T18:48:47.481 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-18T18:48:47.481 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-18T18:48:47.481 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606246', 'branch': 'wip-yuri10-testing-2024-03-15-1653-reef', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7606246', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri10-testing-2024-03-15-1653-reef'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '49bdc51142759622a281d3e5327eebd07cbb8bb3'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '49bdc51142759622a281d3e5327eebd07cbb8bb3'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri10-testing-2024-03-15-1653-reef', 'sha1': '49bdc51142759622a281d3e5327eebd07cbb8bb3'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 8313, 'sha1': '49bdc51142759622a281d3e5327eebd07cbb8bb3', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'fs', 'suite_branch': 'wip-yuri10-testing-2024-03-15-1653-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_49bdc51142759622a281d3e5327eebd07cbb8bb3/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '49bdc51142759622a281d3e5327eebd07cbb8bb3', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi001.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAG5rtJ0ZhNMR5UGSUXb7UXuV0i2c8I2qy7VHzgQ9LhnisB7m9uAD+tqlNDM5PLIXclL2I2FcSts8ljJ4fNGHP4=', 'smithi165.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKlrV/zKFmHgntFhZLft2rs+fc929sATSRWOM5Wdio8Ows/4qHxwsjOObNBEEGsHqIpnPNQ3+d0ferKQgPJGLZ4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '49bdc51142759622a281d3e5327eebd07cbb8bb3'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '49bdc51142759622a281d3e5327eebd07cbb8bb3', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-16_15:03:17', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-18T18:48:47.482 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-18T18:48:47.484 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-18T18:48:47.533 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/run/ceph 2024-03-18T18:48:47.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m0777 -- /var/run/ceph 2024-03-18T18:48:47.599 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:48:47.599 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T18:48:47.627 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T18:48:47.627 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_1 2024-03-18T18:48:47.685 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T18:48:47.685 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T18:48:47.685 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 6h/6d Inode: 197893 Links: 1 2024-03-18T18:48:47.685 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T18:48:47.686 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T18:48:47.686 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-18 18:48:42.776546118 +0000 2024-03-18T18:48:47.686 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-18 18:45:37.004438686 +0000 2024-03-18T18:48:47.686 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-18 18:45:37.004438686 +0000 2024-03-18T18:48:47.686 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: - 2024-03-18T18:48:47.686 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T18:48:47.752 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-18T18:48:47.752 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-18T18:48:47.752 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.3024e-05 s, 5.5 MB/s 2024-03-18T18:48:47.753 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T18:48:47.812 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_2 2024-03-18T18:48:47.871 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T18:48:47.871 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T18:48:47.871 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 6h/6d Inode: 193134 Links: 1 2024-03-18T18:48:47.871 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T18:48:47.871 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T18:48:47.871 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-18 18:48:42.776546118 +0000 2024-03-18T18:48:47.871 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-18 18:45:37.362433570 +0000 2024-03-18T18:48:47.872 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-18 18:45:37.362433570 +0000 2024-03-18T18:48:47.872 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: - 2024-03-18T18:48:47.872 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T18:48:47.937 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-18T18:48:47.938 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-18T18:48:47.938 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.4934e-05 s, 5.4 MB/s 2024-03-18T18:48:47.939 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T18:48:47.998 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_3 2024-03-18T18:48:48.057 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T18:48:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T18:48:48.058 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 6h/6d Inode: 196067 Links: 1 2024-03-18T18:48:48.058 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T18:48:48.058 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T18:48:48.058 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-18 18:48:42.776546118 +0000 2024-03-18T18:48:48.058 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-18 18:45:37.703428697 +0000 2024-03-18T18:48:48.058 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-18 18:45:37.703428697 +0000 2024-03-18T18:48:48.058 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: - 2024-03-18T18:48:48.059 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T18:48:48.124 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-18T18:48:48.124 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-18T18:48:48.124 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 8.3581e-05 s, 6.1 MB/s 2024-03-18T18:48:48.125 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T18:48:48.184 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_4 2024-03-18T18:48:48.242 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T18:48:48.242 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T18:48:48.243 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 6h/6d Inode: 193209 Links: 1 2024-03-18T18:48:48.243 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T18:48:48.243 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T18:48:48.243 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-18 18:48:42.776546118 +0000 2024-03-18T18:48:48.243 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-18 18:45:38.047423781 +0000 2024-03-18T18:48:48.243 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-18 18:45:38.047423781 +0000 2024-03-18T18:48:48.243 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: - 2024-03-18T18:48:48.243 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T18:48:48.310 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-18T18:48:48.310 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-18T18:48:48.310 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.9541e-05 s, 5.1 MB/s 2024-03-18T18:48:48.311 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T18:48:48.370 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-18T18:48:48.370 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:48:48.370 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T18:48:48.391 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T18:48:48.391 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_1 2024-03-18T18:48:48.454 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T18:48:48.454 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T18:48:48.454 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 6h/6d Inode: 188265 Links: 1 2024-03-18T18:48:48.454 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T18:48:48.454 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T18:48:48.454 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-18 18:48:22.471603019 +0000 2024-03-18T18:48:48.454 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-18 18:45:25.100938599 +0000 2024-03-18T18:48:48.455 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-18 18:45:25.100938599 +0000 2024-03-18T18:48:48.455 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2024-03-18T18:48:48.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T18:48:48.525 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-18T18:48:48.525 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-18T18:48:48.526 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000274488 s, 1.9 MB/s 2024-03-18T18:48:48.527 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T18:48:48.589 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_2 2024-03-18T18:48:48.652 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T18:48:48.652 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T18:48:48.652 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 6h/6d Inode: 197136 Links: 1 2024-03-18T18:48:48.652 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T18:48:48.652 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T18:48:48.652 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-18 18:48:22.471603019 +0000 2024-03-18T18:48:48.653 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-18 18:45:25.427933359 +0000 2024-03-18T18:48:48.653 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-18 18:45:25.427933359 +0000 2024-03-18T18:48:48.653 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2024-03-18T18:48:48.653 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T18:48:48.721 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-18T18:48:48.721 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-18T18:48:48.721 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000253231 s, 2.0 MB/s 2024-03-18T18:48:48.722 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T18:48:48.783 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_3 2024-03-18T18:48:48.843 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T18:48:48.843 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T18:48:48.843 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 6h/6d Inode: 194228 Links: 1 2024-03-18T18:48:48.843 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T18:48:48.843 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T18:48:48.843 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-18 18:48:22.471603019 +0000 2024-03-18T18:48:48.843 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-18 18:45:25.783927655 +0000 2024-03-18T18:48:48.843 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-18 18:45:25.783927655 +0000 2024-03-18T18:48:48.843 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2024-03-18T18:48:48.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T18:48:48.912 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-18T18:48:48.912 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-18T18:48:48.913 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000209162 s, 2.4 MB/s 2024-03-18T18:48:48.914 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T18:48:48.973 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_4 2024-03-18T18:48:49.032 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T18:48:49.032 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T18:48:49.032 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 6h/6d Inode: 194338 Links: 1 2024-03-18T18:48:49.032 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T18:48:49.033 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T18:48:49.033 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-18 18:48:22.471603019 +0000 2024-03-18T18:48:49.033 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-18 18:45:26.131922080 +0000 2024-03-18T18:48:49.033 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-18 18:45:26.131922080 +0000 2024-03-18T18:48:49.033 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2024-03-18T18:48:49.033 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T18:48:49.099 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-18T18:48:49.099 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-18T18:48:49.099 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000230944 s, 2.2 MB/s 2024-03-18T18:48:49.100 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T18:48:49.160 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-18T18:48:49.161 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi165.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-18T18:48:49.161 INFO:tasks.ceph:Generating config... 2024-03-18T18:48:49.162 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-18T18:48:49.162 INFO:tasks.ceph:[client] debug client = 20 2024-03-18T18:48:49.162 INFO:tasks.ceph:[client] debug ms = 1 2024-03-18T18:48:49.162 INFO:tasks.ceph:[client] pid file = 2024-03-18T18:48:49.162 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-18T18:48:49.162 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-18T18:48:49.163 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-18T18:48:49.163 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-18T18:48:49.163 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-18T18:48:49.163 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-18T18:48:49.163 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-18T18:48:49.163 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-18T18:48:49.163 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-18T18:48:49.163 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-18T18:48:49.163 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-18T18:48:49.163 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-18T18:48:49.163 INFO:tasks.ceph:[mgr] debug client = 20 2024-03-18T18:48:49.163 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-03-18T18:48:49.163 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-18T18:48:49.163 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-18T18:48:49.164 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-18T18:48:49.164 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-18T18:48:49.164 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-18T18:48:49.164 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-18T18:48:49.164 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-18T18:48:49.164 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-18T18:48:49.164 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-18T18:48:49.164 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-18T18:48:49.164 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-18T18:48:49.164 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-18T18:48:49.164 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-18T18:48:49.164 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-18T18:48:49.164 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-18T18:48:49.164 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-18T18:48:49.164 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-18T18:48:49.165 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-18T18:48:49.166 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-18T18:48:49.166 INFO:tasks.ceph:Setting up mon.a... 2024-03-18T18:48:49.166 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-18T18:48:49.228 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/ceph.keyring 2024-03-18T18:48:49.232 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-18T18:48:49.282 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-18T18:48:49.311 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.1'), ('mon.b', '172.21.15.165'), ('mon.c', '[v2:172.21.15.165:3301,v1:172.21.15.165:6790]')] 2024-03-18T18:48:49.311 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.1,172.21.15.165,[v2:172.21.15.165:3301,v1:172.21.15.165:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-18T18:48:49.312 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:48:49.312 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-18T18:48:49.371 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.1 --add b 172.21.15.165 --addv c '[v2:172.21.15.165:3301,v1:172.21.15.165:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-18T18:48:49.463 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-18T18:48:49.463 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: generated fsid 9c1c697f-8655-4ef3-8a13-da568d2cbe86 2024-03-18T18:48:49.463 INFO:teuthology.orchestra.run.smithi001.stdout:setting min_mon_release = pacific 2024-03-18T18:48:49.463 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 0 2024-03-18T18:48:49.464 INFO:teuthology.orchestra.run.smithi001.stdout:fsid 9c1c697f-8655-4ef3-8a13-da568d2cbe86 2024-03-18T18:48:49.466 INFO:teuthology.orchestra.run.smithi001.stdout:last_changed 2024-03-18T18:48:49.467475+0000 2024-03-18T18:48:49.466 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-03-18T18:48:49.467475+0000 2024-03-18T18:48:49.466 INFO:teuthology.orchestra.run.smithi001.stdout:min_mon_release 16 (pacific) 2024-03-18T18:48:49.467 INFO:teuthology.orchestra.run.smithi001.stdout:election_strategy: 1 2024-03-18T18:48:49.467 INFO:teuthology.orchestra.run.smithi001.stdout:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-03-18T18:48:49.467 INFO:teuthology.orchestra.run.smithi001.stdout:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-03-18T18:48:49.467 INFO:teuthology.orchestra.run.smithi001.stdout:2: [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] mon.c 2024-03-18T18:48:49.467 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-18T18:48:49.470 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-18T18:48:49.490 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 9c1c697f-8655-4ef3-8a13-da568d2cbe86... 2024-03-18T18:48:49.491 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-18T18:48:49.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-18T18:48:49.576 INFO:teuthology.orchestra.run.smithi001.stdout:[global] 2024-03-18T18:48:49.577 INFO:teuthology.orchestra.run.smithi001.stdout: chdir = "" 2024-03-18T18:48:49.577 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-18T18:48:49.577 INFO:teuthology.orchestra.run.smithi001.stdout: auth supported = cephx 2024-03-18T18:48:49.577 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.577 INFO:teuthology.orchestra.run.smithi001.stdout: filestore xattr use omap = true 2024-03-18T18:48:49.577 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.577 INFO:teuthology.orchestra.run.smithi001.stdout: mon clock drift allowed = 1.000 2024-03-18T18:48:49.577 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.578 INFO:teuthology.orchestra.run.smithi001.stdout: osd crush chooseleaf type = 0 2024-03-18T18:48:49.578 INFO:teuthology.orchestra.run.smithi001.stdout: auth debug = true 2024-03-18T18:48:49.578 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.578 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on old message = true 2024-03-18T18:48:49.578 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on bug = true 2024-03-18T18:48:49.578 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.578 INFO:teuthology.orchestra.run.smithi001.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-18T18:48:49.578 INFO:teuthology.orchestra.run.smithi001.stdout: mon pg warn max object skew = 0 2024-03-18T18:48:49.579 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.579 INFO:teuthology.orchestra.run.smithi001.stdout: # disable pg_autoscaler by default for new pools 2024-03-18T18:48:49.579 INFO:teuthology.orchestra.run.smithi001.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-18T18:48:49.579 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.579 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default size = 2 2024-03-18T18:48:49.579 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.579 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow primary affinity = true 2024-03-18T18:48:49.580 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow pg remap = true 2024-03-18T18:48:49.580 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on legacy crush tunables = false 2024-03-18T18:48:49.580 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on crush straw calc version zero = false 2024-03-18T18:48:49.580 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on no sortbitwise = false 2024-03-18T18:48:49.580 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on osd down out interval zero = false 2024-03-18T18:48:49.580 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on too few osds = false 2024-03-18T18:48:49.580 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-18T18:48:49.580 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-18T18:48:49.581 INFO:teuthology.orchestra.run.smithi001.stdout: mon_allow_pool_size_one = true 2024-03-18T18:48:49.581 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.581 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-18T18:48:49.581 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.581 INFO:teuthology.orchestra.run.smithi001.stdout: osd default data pool replay window = 5 2024-03-18T18:48:49.581 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.581 INFO:teuthology.orchestra.run.smithi001.stdout: mon allow pool delete = true 2024-03-18T18:48:49.581 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.581 INFO:teuthology.orchestra.run.smithi001.stdout: mon cluster log file level = debug 2024-03-18T18:48:49.582 INFO:teuthology.orchestra.run.smithi001.stdout: debug asserts on shutdown = true 2024-03-18T18:48:49.582 INFO:teuthology.orchestra.run.smithi001.stdout: mon health detail to clog = false 2024-03-18T18:48:49.582 INFO:teuthology.orchestra.run.smithi001.stdout: mon host = "172.21.15.1,172.21.15.165,[v2:172.21.15.165:3301,v1:172.21.15.165:6790]" 2024-03-18T18:48:49.582 INFO:teuthology.orchestra.run.smithi001.stdout: fsid = 9c1c697f-8655-4ef3-8a13-da568d2cbe86 2024-03-18T18:48:49.582 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.583 INFO:teuthology.orchestra.run.smithi001.stdout:[osd] 2024-03-18T18:48:49.583 INFO:teuthology.orchestra.run.smithi001.stdout: osd journal size = 100 2024-03-18T18:48:49.583 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.583 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub load threshold = 5.0 2024-03-18T18:48:49.583 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub max interval = 600 2024-03-18T18:48:49.583 INFO:teuthology.orchestra.run.smithi001.stdout: osd mclock profile = high_recovery_ops 2024-03-18T18:48:49.583 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.583 INFO:teuthology.orchestra.run.smithi001.stdout: osd recover clone overlap = true 2024-03-18T18:48:49.583 INFO:teuthology.orchestra.run.smithi001.stdout: osd recovery max chunk = 1048576 2024-03-18T18:48:49.584 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.584 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug shutdown = true 2024-03-18T18:48:49.584 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug op order = true 2024-03-18T18:48:49.584 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify stray on activate = true 2024-03-18T18:48:49.584 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.584 INFO:teuthology.orchestra.run.smithi001.stdout: osd open classes on start = true 2024-03-18T18:48:49.584 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug pg log writeout = true 2024-03-18T18:48:49.584 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.585 INFO:teuthology.orchestra.run.smithi001.stdout: osd deep scrub update digest min age = 30 2024-03-18T18:48:49.585 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.585 INFO:teuthology.orchestra.run.smithi001.stdout: osd map max advance = 10 2024-03-18T18:48:49.585 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.585 INFO:teuthology.orchestra.run.smithi001.stdout: journal zero on create = true 2024-03-18T18:48:49.585 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.585 INFO:teuthology.orchestra.run.smithi001.stdout: filestore ondisk finisher threads = 3 2024-03-18T18:48:49.590 INFO:teuthology.orchestra.run.smithi001.stdout: filestore apply finisher threads = 3 2024-03-18T18:48:49.590 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.590 INFO:teuthology.orchestra.run.smithi001.stdout: bdev debug aio = true 2024-03-18T18:48:49.590 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug misdirected ops = true 2024-03-18T18:48:49.590 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore block size = 96636764160 2024-03-18T18:48:49.590 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore fsck on mount = True 2024-03-18T18:48:49.590 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluefs = 20 2024-03-18T18:48:49.590 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluestore = 20 2024-03-18T18:48:49.591 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-18T18:48:49.591 INFO:teuthology.orchestra.run.smithi001.stdout: debug osd = 20 2024-03-18T18:48:49.591 INFO:teuthology.orchestra.run.smithi001.stdout: debug rocksdb = 10 2024-03-18T18:48:49.591 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd backfillfull_ratio = 0.85 2024-03-18T18:48:49.591 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd full ratio = 0.9 2024-03-18T18:48:49.591 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd nearfull ratio = 0.8 2024-03-18T18:48:49.591 INFO:teuthology.orchestra.run.smithi001.stdout: osd failsafe full ratio = 0.95 2024-03-18T18:48:49.591 INFO:teuthology.orchestra.run.smithi001.stdout: osd objectstore = bluestore 2024-03-18T18:48:49.591 INFO:teuthology.orchestra.run.smithi001.stdout: osd op complaint time = 180 2024-03-18T18:48:49.591 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.591 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr] 2024-03-18T18:48:49.591 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-18T18:48:49.591 INFO:teuthology.orchestra.run.smithi001.stdout: debug mgr = 20 2024-03-18T18:48:49.592 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-03-18T18:48:49.592 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-03-18T18:48:49.592 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-03-18T18:48:49.592 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-03-18T18:48:49.592 INFO:teuthology.orchestra.run.smithi001.stdout: mgr/telemetry/nag = false 2024-03-18T18:48:49.592 INFO:teuthology.orchestra.run.smithi001.stdout: debug client = 20 2024-03-18T18:48:49.592 INFO:teuthology.orchestra.run.smithi001.stdout: debug finisher = 20 2024-03-18T18:48:49.592 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.592 INFO:teuthology.orchestra.run.smithi001.stdout:[mon] 2024-03-18T18:48:49.592 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-18T18:48:49.592 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-03-18T18:48:49.592 INFO:teuthology.orchestra.run.smithi001.stdout: debug paxos = 20 2024-03-18T18:48:49.593 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-03-18T18:48:49.593 INFO:teuthology.orchestra.run.smithi001.stdout: mon data avail warn = 5 2024-03-18T18:48:49.593 INFO:teuthology.orchestra.run.smithi001.stdout: mon mgr mkfs grace = 240 2024-03-18T18:48:49.593 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-03-18T18:48:49.593 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd reporter subtree level = osd 2024-03-18T18:48:49.593 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd prime pg temp = true 2024-03-18T18:48:49.593 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-03-18T18:48:49.593 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.593 INFO:teuthology.orchestra.run.smithi001.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-18T18:48:49.593 INFO:teuthology.orchestra.run.smithi001.stdout: auth mon ticket ttl = 660 # 11m 2024-03-18T18:48:49.593 INFO:teuthology.orchestra.run.smithi001.stdout: auth service ticket ttl = 240 # 4m 2024-03-18T18:48:49.593 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.593 INFO:teuthology.orchestra.run.smithi001.stdout: # don't complain about insecure global_id in the test suite 2024-03-18T18:48:49.594 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-18T18:48:49.594 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-18T18:48:49.594 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.594 INFO:teuthology.orchestra.run.smithi001.stdout: # 1m isn't quite enough 2024-03-18T18:48:49.594 INFO:teuthology.orchestra.run.smithi001.stdout: mon_down_mkfs_grace = 2m 2024-03-18T18:48:49.594 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.594 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_filestore_osds = false 2024-03-18T18:48:49.594 INFO:teuthology.orchestra.run.smithi001.stdout: mon op complaint time = 120 2024-03-18T18:48:49.594 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:48:49.594 INFO:teuthology.orchestra.run.smithi001.stdout:[client] 2024-03-18T18:48:49.594 INFO:teuthology.orchestra.run.smithi001.stdout: rgw cache enabled = true 2024-03-18T18:48:49.594 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable ops log = true 2024-03-18T18:48:49.594 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable usage log = true 2024-03-18T18:48:49.594 INFO:teuthology.orchestra.run.smithi001.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-18T18:48:49.595 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-18T18:48:49.595 INFO:teuthology.orchestra.run.smithi001.stdout: client mount timeout = 600 2024-03-18T18:48:49.595 INFO:teuthology.orchestra.run.smithi001.stdout: debug client = 20 2024-03-18T18:48:49.595 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-18T18:48:49.595 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = "" 2024-03-18T18:48:49.595 INFO:teuthology.orchestra.run.smithi001.stdout: rados mon op timeout = 900 2024-03-18T18:48:49.595 INFO:teuthology.orchestra.run.smithi001.stdout: rados osd op timeout = 900 2024-03-18T18:48:49.595 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.a] 2024-03-18T18:48:49.595 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.b] 2024-03-18T18:48:49.595 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.c] 2024-03-18T18:48:49.595 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.a] 2024-03-18T18:48:49.595 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.c] 2024-03-18T18:48:49.595 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.b] 2024-03-18T18:48:49.595 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.d] 2024-03-18T18:48:49.596 INFO:teuthology.orchestra.run.smithi001.stdout:[mds] 2024-03-18T18:48:49.596 INFO:teuthology.orchestra.run.smithi001.stdout: debug mds = 20 2024-03-18T18:48:49.596 INFO:teuthology.orchestra.run.smithi001.stdout: debug mds balancer = 20 2024-03-18T18:48:49.596 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-18T18:48:49.596 INFO:teuthology.orchestra.run.smithi001.stdout: mds debug frag = True 2024-03-18T18:48:49.596 INFO:teuthology.orchestra.run.smithi001.stdout: mds debug scatterstat = True 2024-03-18T18:48:49.596 INFO:teuthology.orchestra.run.smithi001.stdout: mds op complaint time = 180 2024-03-18T18:48:49.596 INFO:teuthology.orchestra.run.smithi001.stdout: mds verify scatter = True 2024-03-18T18:48:49.596 INFO:teuthology.orchestra.run.smithi001.stdout: osd op complaint time = 180 2024-03-18T18:48:49.596 INFO:teuthology.orchestra.run.smithi001.stdout: rados mon op timeout = 900 2024-03-18T18:48:49.596 INFO:teuthology.orchestra.run.smithi001.stdout: rados osd op timeout = 900 2024-03-18T18:48:49.598 INFO:teuthology.orchestra.run.smithi165.stdout:[global] 2024-03-18T18:48:49.598 INFO:teuthology.orchestra.run.smithi165.stdout: chdir = "" 2024-03-18T18:48:49.598 INFO:teuthology.orchestra.run.smithi165.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-18T18:48:49.598 INFO:teuthology.orchestra.run.smithi165.stdout: auth supported = cephx 2024-03-18T18:48:49.599 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.599 INFO:teuthology.orchestra.run.smithi165.stdout: filestore xattr use omap = true 2024-03-18T18:48:49.599 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.599 INFO:teuthology.orchestra.run.smithi165.stdout: mon clock drift allowed = 1.000 2024-03-18T18:48:49.599 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.599 INFO:teuthology.orchestra.run.smithi165.stdout: osd crush chooseleaf type = 0 2024-03-18T18:48:49.599 INFO:teuthology.orchestra.run.smithi165.stdout: auth debug = true 2024-03-18T18:48:49.599 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.599 INFO:teuthology.orchestra.run.smithi165.stdout: ms die on old message = true 2024-03-18T18:48:49.599 INFO:teuthology.orchestra.run.smithi165.stdout: ms die on bug = true 2024-03-18T18:48:49.599 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.599 INFO:teuthology.orchestra.run.smithi165.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-18T18:48:49.599 INFO:teuthology.orchestra.run.smithi165.stdout: mon pg warn max object skew = 0 2024-03-18T18:48:49.600 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.600 INFO:teuthology.orchestra.run.smithi165.stdout: # disable pg_autoscaler by default for new pools 2024-03-18T18:48:49.600 INFO:teuthology.orchestra.run.smithi165.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-18T18:48:49.600 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.600 INFO:teuthology.orchestra.run.smithi165.stdout: osd pool default size = 2 2024-03-18T18:48:49.600 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.600 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd allow primary affinity = true 2024-03-18T18:48:49.600 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd allow pg remap = true 2024-03-18T18:48:49.600 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on legacy crush tunables = false 2024-03-18T18:48:49.600 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on crush straw calc version zero = false 2024-03-18T18:48:49.600 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on no sortbitwise = false 2024-03-18T18:48:49.600 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on osd down out interval zero = false 2024-03-18T18:48:49.600 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on too few osds = false 2024-03-18T18:48:49.601 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-18T18:48:49.601 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-18T18:48:49.601 INFO:teuthology.orchestra.run.smithi165.stdout: mon_allow_pool_size_one = true 2024-03-18T18:48:49.601 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.601 INFO:teuthology.orchestra.run.smithi165.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-18T18:48:49.601 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.601 INFO:teuthology.orchestra.run.smithi165.stdout: osd default data pool replay window = 5 2024-03-18T18:48:49.601 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.601 INFO:teuthology.orchestra.run.smithi165.stdout: mon allow pool delete = true 2024-03-18T18:48:49.601 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.602 INFO:teuthology.orchestra.run.smithi165.stdout: mon cluster log file level = debug 2024-03-18T18:48:49.602 INFO:teuthology.orchestra.run.smithi165.stdout: debug asserts on shutdown = true 2024-03-18T18:48:49.602 INFO:teuthology.orchestra.run.smithi165.stdout: mon health detail to clog = false 2024-03-18T18:48:49.602 INFO:teuthology.orchestra.run.smithi165.stdout: mon host = "172.21.15.1,172.21.15.165,[v2:172.21.15.165:3301,v1:172.21.15.165:6790]" 2024-03-18T18:48:49.602 INFO:teuthology.orchestra.run.smithi165.stdout: fsid = 9c1c697f-8655-4ef3-8a13-da568d2cbe86 2024-03-18T18:48:49.602 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.602 INFO:teuthology.orchestra.run.smithi165.stdout:[osd] 2024-03-18T18:48:49.602 INFO:teuthology.orchestra.run.smithi165.stdout: osd journal size = 100 2024-03-18T18:48:49.602 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.602 INFO:teuthology.orchestra.run.smithi165.stdout: osd scrub load threshold = 5.0 2024-03-18T18:48:49.602 INFO:teuthology.orchestra.run.smithi165.stdout: osd scrub max interval = 600 2024-03-18T18:48:49.602 INFO:teuthology.orchestra.run.smithi165.stdout: osd mclock profile = high_recovery_ops 2024-03-18T18:48:49.603 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.603 INFO:teuthology.orchestra.run.smithi165.stdout: osd recover clone overlap = true 2024-03-18T18:48:49.603 INFO:teuthology.orchestra.run.smithi165.stdout: osd recovery max chunk = 1048576 2024-03-18T18:48:49.603 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.603 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug shutdown = true 2024-03-18T18:48:49.603 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug op order = true 2024-03-18T18:48:49.603 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug verify stray on activate = true 2024-03-18T18:48:49.603 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.603 INFO:teuthology.orchestra.run.smithi165.stdout: osd open classes on start = true 2024-03-18T18:48:49.603 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug pg log writeout = true 2024-03-18T18:48:49.603 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.603 INFO:teuthology.orchestra.run.smithi165.stdout: osd deep scrub update digest min age = 30 2024-03-18T18:48:49.603 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.603 INFO:teuthology.orchestra.run.smithi165.stdout: osd map max advance = 10 2024-03-18T18:48:49.604 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.604 INFO:teuthology.orchestra.run.smithi165.stdout: journal zero on create = true 2024-03-18T18:48:49.604 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.604 INFO:teuthology.orchestra.run.smithi165.stdout: filestore ondisk finisher threads = 3 2024-03-18T18:48:49.604 INFO:teuthology.orchestra.run.smithi165.stdout: filestore apply finisher threads = 3 2024-03-18T18:48:49.604 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.604 INFO:teuthology.orchestra.run.smithi165.stdout: bdev debug aio = true 2024-03-18T18:48:49.604 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug misdirected ops = true 2024-03-18T18:48:49.604 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore block size = 96636764160 2024-03-18T18:48:49.604 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore fsck on mount = True 2024-03-18T18:48:49.604 INFO:teuthology.orchestra.run.smithi165.stdout: debug bluefs = 20 2024-03-18T18:48:49.604 INFO:teuthology.orchestra.run.smithi165.stdout: debug bluestore = 20 2024-03-18T18:48:49.604 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-03-18T18:48:49.605 INFO:teuthology.orchestra.run.smithi165.stdout: debug osd = 20 2024-03-18T18:48:49.605 INFO:teuthology.orchestra.run.smithi165.stdout: debug rocksdb = 10 2024-03-18T18:48:49.605 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd backfillfull_ratio = 0.85 2024-03-18T18:48:49.605 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd full ratio = 0.9 2024-03-18T18:48:49.605 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd nearfull ratio = 0.8 2024-03-18T18:48:49.605 INFO:teuthology.orchestra.run.smithi165.stdout: osd failsafe full ratio = 0.95 2024-03-18T18:48:49.605 INFO:teuthology.orchestra.run.smithi165.stdout: osd objectstore = bluestore 2024-03-18T18:48:49.605 INFO:teuthology.orchestra.run.smithi165.stdout: osd op complaint time = 180 2024-03-18T18:48:49.605 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.605 INFO:teuthology.orchestra.run.smithi165.stdout:[mgr] 2024-03-18T18:48:49.605 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-03-18T18:48:49.605 INFO:teuthology.orchestra.run.smithi165.stdout: debug mgr = 20 2024-03-18T18:48:49.605 INFO:teuthology.orchestra.run.smithi165.stdout: debug mon = 20 2024-03-18T18:48:49.606 INFO:teuthology.orchestra.run.smithi165.stdout: debug auth = 20 2024-03-18T18:48:49.606 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min pgs per osd = 4 2024-03-18T18:48:49.606 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min bytes per osd = 10 2024-03-18T18:48:49.606 INFO:teuthology.orchestra.run.smithi165.stdout: mgr/telemetry/nag = false 2024-03-18T18:48:49.606 INFO:teuthology.orchestra.run.smithi165.stdout: debug client = 20 2024-03-18T18:48:49.606 INFO:teuthology.orchestra.run.smithi165.stdout: debug finisher = 20 2024-03-18T18:48:49.606 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.606 INFO:teuthology.orchestra.run.smithi165.stdout:[mon] 2024-03-18T18:48:49.606 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-03-18T18:48:49.606 INFO:teuthology.orchestra.run.smithi165.stdout: debug mon = 20 2024-03-18T18:48:49.606 INFO:teuthology.orchestra.run.smithi165.stdout: debug paxos = 20 2024-03-18T18:48:49.606 INFO:teuthology.orchestra.run.smithi165.stdout: debug auth = 20 2024-03-18T18:48:49.606 INFO:teuthology.orchestra.run.smithi165.stdout: mon data avail warn = 5 2024-03-18T18:48:49.607 INFO:teuthology.orchestra.run.smithi165.stdout: mon mgr mkfs grace = 240 2024-03-18T18:48:49.607 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min pgs per osd = 4 2024-03-18T18:48:49.607 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd reporter subtree level = osd 2024-03-18T18:48:49.607 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd prime pg temp = true 2024-03-18T18:48:49.607 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min bytes per osd = 10 2024-03-18T18:48:49.607 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.607 INFO:teuthology.orchestra.run.smithi165.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-18T18:48:49.607 INFO:teuthology.orchestra.run.smithi165.stdout: auth mon ticket ttl = 660 # 11m 2024-03-18T18:48:49.607 INFO:teuthology.orchestra.run.smithi165.stdout: auth service ticket ttl = 240 # 4m 2024-03-18T18:48:49.607 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.607 INFO:teuthology.orchestra.run.smithi165.stdout: # don't complain about insecure global_id in the test suite 2024-03-18T18:48:49.607 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-18T18:48:49.607 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-18T18:48:49.607 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.608 INFO:teuthology.orchestra.run.smithi165.stdout: # 1m isn't quite enough 2024-03-18T18:48:49.608 INFO:teuthology.orchestra.run.smithi165.stdout: mon_down_mkfs_grace = 2m 2024-03-18T18:48:49.608 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.608 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_filestore_osds = false 2024-03-18T18:48:49.608 INFO:teuthology.orchestra.run.smithi165.stdout: mon op complaint time = 120 2024-03-18T18:48:49.608 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T18:48:49.608 INFO:teuthology.orchestra.run.smithi165.stdout:[client] 2024-03-18T18:48:49.608 INFO:teuthology.orchestra.run.smithi165.stdout: rgw cache enabled = true 2024-03-18T18:48:49.608 INFO:teuthology.orchestra.run.smithi165.stdout: rgw enable ops log = true 2024-03-18T18:48:49.608 INFO:teuthology.orchestra.run.smithi165.stdout: rgw enable usage log = true 2024-03-18T18:48:49.608 INFO:teuthology.orchestra.run.smithi165.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-18T18:48:49.608 INFO:teuthology.orchestra.run.smithi165.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-18T18:48:49.608 INFO:teuthology.orchestra.run.smithi165.stdout: client mount timeout = 600 2024-03-18T18:48:49.608 INFO:teuthology.orchestra.run.smithi165.stdout: debug client = 20 2024-03-18T18:48:49.608 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-03-18T18:48:49.609 INFO:teuthology.orchestra.run.smithi165.stdout: pid file = "" 2024-03-18T18:48:49.609 INFO:teuthology.orchestra.run.smithi165.stdout: rados mon op timeout = 900 2024-03-18T18:48:49.609 INFO:teuthology.orchestra.run.smithi165.stdout: rados osd op timeout = 900 2024-03-18T18:48:49.609 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.a] 2024-03-18T18:48:49.609 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.b] 2024-03-18T18:48:49.609 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.c] 2024-03-18T18:48:49.609 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.a] 2024-03-18T18:48:49.609 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.c] 2024-03-18T18:48:49.609 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.b] 2024-03-18T18:48:49.609 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.d] 2024-03-18T18:48:49.609 INFO:teuthology.orchestra.run.smithi165.stdout:[mds] 2024-03-18T18:48:49.609 INFO:teuthology.orchestra.run.smithi165.stdout: debug mds = 20 2024-03-18T18:48:49.609 INFO:teuthology.orchestra.run.smithi165.stdout: debug mds balancer = 20 2024-03-18T18:48:49.609 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-03-18T18:48:49.610 INFO:teuthology.orchestra.run.smithi165.stdout: mds debug frag = True 2024-03-18T18:48:49.610 INFO:teuthology.orchestra.run.smithi165.stdout: mds debug scatterstat = True 2024-03-18T18:48:49.610 INFO:teuthology.orchestra.run.smithi165.stdout: mds op complaint time = 180 2024-03-18T18:48:49.610 INFO:teuthology.orchestra.run.smithi165.stdout: mds verify scatter = True 2024-03-18T18:48:49.610 INFO:teuthology.orchestra.run.smithi165.stdout: osd op complaint time = 180 2024-03-18T18:48:49.610 INFO:teuthology.orchestra.run.smithi165.stdout: rados mon op timeout = 900 2024-03-18T18:48:49.610 INFO:teuthology.orchestra.run.smithi165.stdout: rados osd op timeout = 900 2024-03-18T18:48:49.610 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-18T18:48:49.610 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-18T18:48:49.697 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-18T18:48:49.697 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:48:49.697 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-18T18:48:49.715 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:48:49.715 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-18T18:48:49.773 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T18:48:49.773 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:48:49.773 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-18T18:48:49.773 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-18T18:48:49.850 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:48:49.850 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-18T18:48:49.908 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T18:48:49.908 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:48:49.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-18T18:48:49.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-18T18:48:49.952 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:48:49.953 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-18T18:48:50.013 INFO:tasks.ceph:Setting up mon nodes... 2024-03-18T18:48:50.013 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-18T18:48:50.013 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-18T18:48:50.068 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-18T18:48:50.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-18T18:48:50.138 INFO:teuthology.orchestra.run.smithi165.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-18T18:48:50.141 INFO:tasks.ceph:Setting up mds nodes... 2024-03-18T18:48:50.142 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-18T18:48:50.196 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-18T18:48:50.199 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-18T18:48:50.226 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-18T18:48:50.321 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-18T18:48:50.323 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-18T18:48:50.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-18T18:48:50.410 INFO:teuthology.orchestra.run.smithi165.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-18T18:48:50.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-18T18:48:50.441 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-18T18:48:50.535 INFO:teuthology.orchestra.run.smithi165.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-18T18:48:50.538 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-18T18:48:50.565 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-18T18:48:50.566 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-18T18:48:50.611 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-18T18:48:50.623 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-18T18:48:50.669 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-18T18:48:50.681 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-18T18:48:50.725 INFO:teuthology.orchestra.run.smithi165.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-18T18:48:50.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-18T18:48:50.821 INFO:teuthology.orchestra.run.smithi165.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-18T18:48:50.833 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-18T18:48:50.833 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi165.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-18T18:48:50.833 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-18T18:48:50.862 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-18T18:48:50.862 INFO:tasks.ceph:role: osd.0 2024-03-18T18:48:50.862 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T18:48:50.862 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-18T18:48:50.930 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T18:48:50.930 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T18:48:50.930 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T18:48:50.930 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-18T18:48:50.930 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T18:48:50.930 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-18T18:48:50.930 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T18:48:50.930 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-18T18:48:50.930 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T18:48:50.930 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T18:48:50.952 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-18T18:48:50.953 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-18T18:48:50.954 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-18T18:48:51.035 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-18T18:48:51.108 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-18T18:48:51.176 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-18T18:48:51.176 INFO:tasks.ceph:role: osd.1 2024-03-18T18:48:51.176 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T18:48:51.177 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-18T18:48:51.243 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T18:48:51.243 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T18:48:51.244 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T18:48:51.244 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-18T18:48:51.244 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T18:48:51.244 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-18T18:48:51.244 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T18:48:51.244 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-18T18:48:51.244 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T18:48:51.244 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T18:48:51.266 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-18T18:48:51.268 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-18T18:48:51.268 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-18T18:48:51.348 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-18T18:48:51.421 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-18T18:48:51.489 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-18T18:48:51.489 INFO:tasks.ceph:role: osd.2 2024-03-18T18:48:51.489 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T18:48:51.489 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-18T18:48:51.556 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T18:48:51.556 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T18:48:51.556 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T18:48:51.556 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-18T18:48:51.556 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T18:48:51.556 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-18T18:48:51.556 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T18:48:51.556 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-18T18:48:51.557 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T18:48:51.557 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T18:48:51.579 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-18T18:48:51.580 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-18T18:48:51.580 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-18T18:48:51.661 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-18T18:48:51.733 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-18T18:48:51.801 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-18T18:48:51.801 INFO:tasks.ceph:role: osd.3 2024-03-18T18:48:51.802 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T18:48:51.802 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-18T18:48:51.869 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T18:48:51.869 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T18:48:51.869 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T18:48:51.869 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-18T18:48:51.869 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T18:48:51.869 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-03-18T18:48:51.869 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T18:48:51.870 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-18T18:48:51.870 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T18:48:51.870 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T18:48:51.891 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-03-18T18:48:51.893 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-03-18T18:48:51.893 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-18T18:48:51.973 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-18T18:48:52.045 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T18:48:52.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:48:52.171+0000 7f0831f52680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-18T18:48:52.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:48:52.171+0000 7f0831f52680 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-18T18:48:52.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:48:52.171+0000 7f0831f52680 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-18T18:48:52.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:48:52.171+0000 7f0831f52680 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-18T18:48:52.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:48:52.171+0000 7f0831f52680 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-18T18:48:52.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:48:52.172+0000 7f0831f52680 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-18T18:48:55.010 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-18T18:48:55.044 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T18:48:55.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:48:55.142+0000 7f1b0c7bb680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-18T18:48:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:48:55.142+0000 7f1b0c7bb680 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-18T18:48:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:48:55.142+0000 7f1b0c7bb680 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-18T18:48:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:48:55.142+0000 7f1b0c7bb680 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-18T18:48:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:48:55.142+0000 7f1b0c7bb680 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-18T18:48:55.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:48:55.142+0000 7f1b0c7bb680 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-18T18:48:57.981 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-18T18:48:58.014 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T18:48:58.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:48:58.112+0000 7f40bec29680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-18T18:48:58.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:48:58.112+0000 7f40bec29680 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-18T18:48:58.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:48:58.112+0000 7f40bec29680 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-18T18:48:58.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:48:58.112+0000 7f40bec29680 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-18T18:48:58.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:48:58.112+0000 7f40bec29680 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-18T18:48:58.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:48:58.112+0000 7f40bec29680 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-18T18:49:00.956 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-18T18:49:00.989 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T18:49:01.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:01.086+0000 7fe0e5c82680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-18T18:49:01.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:01.087+0000 7fe0e5c82680 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-18T18:49:01.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:01.087+0000 7fe0e5c82680 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-18T18:49:01.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:01.087+0000 7fe0e5c82680 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-18T18:49:01.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:01.087+0000 7fe0e5c82680 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-18T18:49:01.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:01.087+0000 7fe0e5c82680 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-18T18:49:03.934 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-18T18:49:03.967 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-18T18:49:04.000 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-18T18:49:04.001 INFO:tasks.ceph:role: osd.4 2024-03-18T18:49:04.001 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T18:49:04.001 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-18T18:49:04.078 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T18:49:04.079 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T18:49:04.079 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T18:49:04.079 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-18T18:49:04.079 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T18:49:04.079 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2024-03-18T18:49:04.079 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T18:49:04.079 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-18T18:49:04.079 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T18:49:04.079 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T18:49:04.507 INFO:teuthology.orchestra.run.smithi165.stdout:Discarding blocks...Done. 2024-03-18T18:49:04.508 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2024-03-18T18:49:04.509 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-18T18:49:04.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-18T18:49:04.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-18T18:49:04.786 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-18T18:49:04.786 INFO:tasks.ceph:role: osd.5 2024-03-18T18:49:04.786 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T18:49:04.786 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-18T18:49:04.862 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T18:49:04.862 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T18:49:04.862 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T18:49:04.863 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-18T18:49:04.863 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T18:49:04.863 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2024-03-18T18:49:04.863 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T18:49:04.863 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-18T18:49:04.863 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T18:49:04.863 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T18:49:05.288 INFO:teuthology.orchestra.run.smithi165.stdout:Discarding blocks...Done. 2024-03-18T18:49:05.289 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2024-03-18T18:49:05.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-18T18:49:05.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-18T18:49:05.437 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-18T18:49:05.506 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-18T18:49:05.507 INFO:tasks.ceph:role: osd.6 2024-03-18T18:49:05.507 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T18:49:05.507 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-18T18:49:05.582 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T18:49:05.582 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T18:49:05.582 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T18:49:05.582 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-18T18:49:05.583 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T18:49:05.583 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2024-03-18T18:49:05.583 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T18:49:05.583 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-18T18:49:05.583 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T18:49:05.583 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T18:49:06.024 INFO:teuthology.orchestra.run.smithi165.stdout:Discarding blocks...Done. 2024-03-18T18:49:06.026 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2024-03-18T18:49:06.026 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-18T18:49:06.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-18T18:49:06.246 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-18T18:49:06.316 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-18T18:49:06.316 INFO:tasks.ceph:role: osd.7 2024-03-18T18:49:06.316 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T18:49:06.316 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-18T18:49:06.392 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T18:49:06.392 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T18:49:06.392 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T18:49:06.392 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-18T18:49:06.392 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T18:49:06.393 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2024-03-18T18:49:06.393 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T18:49:06.393 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-18T18:49:06.393 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T18:49:06.393 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T18:49:06.825 INFO:teuthology.orchestra.run.smithi165.stdout:Discarding blocks...Done. 2024-03-18T18:49:06.827 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2024-03-18T18:49:06.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-18T18:49:07.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-18T18:49:07.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T18:49:07.161 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:07.165+0000 7f6be48c9680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-18T18:49:07.163 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:07.166+0000 7f6be48c9680 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-18T18:49:07.163 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:07.166+0000 7f6be48c9680 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-18T18:49:07.163 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:07.166+0000 7f6be48c9680 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-18T18:49:07.163 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:07.166+0000 7f6be48c9680 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-18T18:49:07.163 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:07.166+0000 7f6be48c9680 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-18T18:49:10.012 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-18T18:49:10.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T18:49:10.136 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:10.139+0000 7f6302c22680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-18T18:49:10.136 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:10.139+0000 7f6302c22680 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-18T18:49:10.136 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:10.139+0000 7f6302c22680 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-18T18:49:10.136 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:10.139+0000 7f6302c22680 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-18T18:49:10.137 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:10.139+0000 7f6302c22680 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-18T18:49:10.137 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:10.140+0000 7f6302c22680 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-18T18:49:12.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-18T18:49:13.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T18:49:13.108 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:13.110+0000 7fb5dbdc3680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-18T18:49:13.108 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:13.111+0000 7fb5dbdc3680 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-18T18:49:13.108 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:13.111+0000 7fb5dbdc3680 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-18T18:49:13.108 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:13.111+0000 7fb5dbdc3680 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-18T18:49:13.109 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:13.111+0000 7fb5dbdc3680 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-18T18:49:13.109 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:13.111+0000 7fb5dbdc3680 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-18T18:49:15.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-18T18:49:15.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T18:49:16.087 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:16.090+0000 7f0b404d0680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-18T18:49:16.088 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:16.090+0000 7f0b404d0680 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-18T18:49:16.088 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:16.090+0000 7f0b404d0680 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-18T18:49:16.088 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:16.090+0000 7f0b404d0680 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-18T18:49:16.088 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:16.090+0000 7f0b404d0680 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-18T18:49:16.088 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:16.091+0000 7f0b404d0680 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-18T18:49:17.483 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:49:17.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:49:18.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-18T18:49:18.970 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-18T18:49:18.970 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:49:18.971 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-18T18:49:19.004 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:49:19.004 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-18T18:49:19.077 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:49:19.077 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-18T18:49:19.148 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:49:19.148 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-18T18:49:19.222 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:49:19.222 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-18T18:49:19.293 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:49:19.293 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-18T18:49:19.363 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:49:19.363 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-18T18:49:19.432 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:49:19.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-18T18:49:19.465 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:49:19.465 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-18T18:49:19.536 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:49:19.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-18T18:49:19.606 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:49:19.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-18T18:49:19.675 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:49:19.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-18T18:49:19.744 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:49:19.744 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-18T18:49:19.811 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:49:19.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-18T18:49:19.879 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:49:19.879 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-18T18:49:19.901 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:49:19.901 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-18T18:49:19.963 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:49:19.963 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-18T18:49:19.981 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:49:19.981 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-18T18:49:20.043 INFO:tasks.ceph:Adding keys to all mons... 2024-03-18T18:49:20.043 DEBUG:teuthology.orchestra.run.smithi001:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-18T18:49:20.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-18T18:49:20.086 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr.y] 2024-03-18T18:49:20.086 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCSjPhlL0pTBBAAdFiPcdwwz514cuLAvioxYA== 2024-03-18T18:49:20.086 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.a] 2024-03-18T18:49:20.086 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCSjPhlR93/CxAAbpe+5n3wHRur9xi8QzdCnQ== 2024-03-18T18:49:20.086 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.c] 2024-03-18T18:49:20.086 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCSjPhlJZFoExAAmI16oxwNKbf8dKetPfyk+g== 2024-03-18T18:49:20.086 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.0] 2024-03-18T18:49:20.086 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCUjPhlYOVHChAARFQrksZmMIdTlntd8TgvrQ== 2024-03-18T18:49:20.087 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.1] 2024-03-18T18:49:20.087 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCXjPhl2zSDCBAA6Al1gkWVB79ZI9djGpVChA== 2024-03-18T18:49:20.087 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.2] 2024-03-18T18:49:20.087 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCajPhlSoG/BhAAR93USDe6DCQW+hREOsnTlw== 2024-03-18T18:49:20.087 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.3] 2024-03-18T18:49:20.087 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCdjPhlhJg5BRAAWuQhcnxyjR5UOiM13yKdAQ== 2024-03-18T18:49:20.087 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr.x] 2024-03-18T18:49:20.087 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCSjPhleIuOCBAA9CH+AHewQbn+gpVr+6lJyw== 2024-03-18T18:49:20.087 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.b] 2024-03-18T18:49:20.087 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCSjPhlJ3OmGBAA37u227nxrMsh/JHt71HBxQ== 2024-03-18T18:49:20.087 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.d] 2024-03-18T18:49:20.087 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCSjPhlQrcvIBAA1LMv+EgC7S6wspX6QF49Pg== 2024-03-18T18:49:20.087 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.4] 2024-03-18T18:49:20.088 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCjjPhlCFnpCRAAKIyOQyVdjXZUTnAS6Nw/ng== 2024-03-18T18:49:20.088 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.5] 2024-03-18T18:49:20.088 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCmjPhlAeFkCBAAJcEFpRk/03lQJxj+l2hEPA== 2024-03-18T18:49:20.088 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.6] 2024-03-18T18:49:20.088 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCpjPhl/xuuBhAA0LgKfXEvN2O11ZlRQJVOkw== 2024-03-18T18:49:20.088 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.7] 2024-03-18T18:49:20.088 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCsjPhldiR4BRAA571oh6U8PzOv/4LzjYLF/A== 2024-03-18T18:49:20.088 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-18T18:49:20.088 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCSjPhlu+2xJBAANnXGGjfeqP6S2FTecsfOsg== 2024-03-18T18:49:20.088 INFO:teuthology.orchestra.run.smithi001.stdout:[client.2] 2024-03-18T18:49:20.088 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCSjPhlxBUpKBAAIGTwmLWyj+P2Yvl55QVcWA== 2024-03-18T18:49:20.088 INFO:teuthology.orchestra.run.smithi001.stdout:[client.1] 2024-03-18T18:49:20.088 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCSjPhlhJ9+KxAAEnrQrCrgJR6xI+pucwzAXA== 2024-03-18T18:49:20.088 INFO:teuthology.orchestra.run.smithi001.stdout:[client.3] 2024-03-18T18:49:20.088 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCSjPhlmao3MRAAHeiQ6eyBh9QjjKPCtxkC8g== 2024-03-18T18:49:20.108 INFO:teuthology.orchestra.run.smithi165.stdout:[mgr.y] 2024-03-18T18:49:20.109 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCSjPhlL0pTBBAAdFiPcdwwz514cuLAvioxYA== 2024-03-18T18:49:20.109 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.a] 2024-03-18T18:49:20.109 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCSjPhlR93/CxAAbpe+5n3wHRur9xi8QzdCnQ== 2024-03-18T18:49:20.109 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.c] 2024-03-18T18:49:20.109 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCSjPhlJZFoExAAmI16oxwNKbf8dKetPfyk+g== 2024-03-18T18:49:20.109 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.0] 2024-03-18T18:49:20.110 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCUjPhlYOVHChAARFQrksZmMIdTlntd8TgvrQ== 2024-03-18T18:49:20.110 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.1] 2024-03-18T18:49:20.110 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCXjPhl2zSDCBAA6Al1gkWVB79ZI9djGpVChA== 2024-03-18T18:49:20.110 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.2] 2024-03-18T18:49:20.110 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCajPhlSoG/BhAAR93USDe6DCQW+hREOsnTlw== 2024-03-18T18:49:20.110 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.3] 2024-03-18T18:49:20.110 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCdjPhlhJg5BRAAWuQhcnxyjR5UOiM13yKdAQ== 2024-03-18T18:49:20.111 INFO:teuthology.orchestra.run.smithi165.stdout:[mgr.x] 2024-03-18T18:49:20.111 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCSjPhleIuOCBAA9CH+AHewQbn+gpVr+6lJyw== 2024-03-18T18:49:20.111 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.b] 2024-03-18T18:49:20.111 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCSjPhlJ3OmGBAA37u227nxrMsh/JHt71HBxQ== 2024-03-18T18:49:20.112 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.d] 2024-03-18T18:49:20.112 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCSjPhlQrcvIBAA1LMv+EgC7S6wspX6QF49Pg== 2024-03-18T18:49:20.112 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.4] 2024-03-18T18:49:20.112 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCjjPhlCFnpCRAAKIyOQyVdjXZUTnAS6Nw/ng== 2024-03-18T18:49:20.112 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.5] 2024-03-18T18:49:20.112 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCmjPhlAeFkCBAAJcEFpRk/03lQJxj+l2hEPA== 2024-03-18T18:49:20.112 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.6] 2024-03-18T18:49:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCpjPhl/xuuBhAA0LgKfXEvN2O11ZlRQJVOkw== 2024-03-18T18:49:20.113 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.7] 2024-03-18T18:49:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCsjPhldiR4BRAA571oh6U8PzOv/4LzjYLF/A== 2024-03-18T18:49:20.113 INFO:teuthology.orchestra.run.smithi165.stdout:[client.0] 2024-03-18T18:49:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCSjPhlu+2xJBAANnXGGjfeqP6S2FTecsfOsg== 2024-03-18T18:49:20.113 INFO:teuthology.orchestra.run.smithi165.stdout:[client.2] 2024-03-18T18:49:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCSjPhlxBUpKBAAIGTwmLWyj+P2Yvl55QVcWA== 2024-03-18T18:49:20.113 INFO:teuthology.orchestra.run.smithi165.stdout:[client.1] 2024-03-18T18:49:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCSjPhlhJ9+KxAAEnrQrCrgJR6xI+pucwzAXA== 2024-03-18T18:49:20.113 INFO:teuthology.orchestra.run.smithi165.stdout:[client.3] 2024-03-18T18:49:20.113 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCSjPhlmao3MRAAHeiQ6eyBh9QjjKPCtxkC8g== 2024-03-18T18:49:20.115 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-18T18:49:20.133 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-18T18:49:20.209 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T18:49:20.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T18:49:20.271 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T18:49:20.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T18:49:20.330 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T18:49:20.332 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T18:49:20.385 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T18:49:20.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T18:49:20.440 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T18:49:20.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T18:49:20.495 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T18:49:20.497 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T18:49:20.550 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-18T18:49:20.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-18T18:49:20.606 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T18:49:20.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T18:49:20.664 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T18:49:20.667 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T18:49:20.723 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T18:49:20.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T18:49:20.778 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T18:49:20.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T18:49:20.833 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T18:49:20.835 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T18:49:20.889 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T18:49:20.891 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T18:49:20.945 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-18T18:49:20.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-18T18:49:21.004 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-18T18:49:21.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-18T18:49:21.062 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-18T18:49:21.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-18T18:49:21.120 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-18T18:49:21.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-18T18:49:21.181 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-18T18:49:21.181 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-18T18:49:21.209 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-18T18:49:21.310 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-18T18:49:21.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-18T18:49:21.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-18T18:49:21.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-18T18:49:21.493 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-18T18:49:21.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-18T18:49:21.659 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-18T18:49:21.687 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-18T18:49:21.689 DEBUG:teuthology.orchestra.run.smithi165:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-18T18:49:21.744 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-18T18:49:21.745 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-18T18:49:21.745 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-18T18:49:21.748 INFO:tasks.ceph.mon.a:Started 2024-03-18T18:49:21.748 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-18T18:49:21.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-18T18:49:21.787 INFO:tasks.ceph.mon.b:Started 2024-03-18T18:49:21.787 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-18T18:49:21.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-18T18:49:21.789 INFO:tasks.ceph.mon.c:Started 2024-03-18T18:49:21.789 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-18T18:49:21.790 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-18T18:49:21.790 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-18T18:49:21.792 INFO:tasks.ceph.mgr.y:Started 2024-03-18T18:49:21.792 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-18T18:49:21.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-18T18:49:21.794 INFO:tasks.ceph.mgr.x:Started 2024-03-18T18:49:21.794 INFO:tasks.ceph:Setting crush tunables to default 2024-03-18T18:49:21.795 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph osd crush tunables default 2024-03-18T18:49:21.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.925+0000 7f2fc173c700 1 Processor -- start 2024-03-18T18:49:21.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.925+0000 7f2fc173c700 1 -- start start 2024-03-18T18:49:21.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.925+0000 7f2fc173c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 0x7f2fbc0cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:21.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.925+0000 7f2fc173c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 0x7f2fbc0cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:21.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.925+0000 7f2fc173c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fbc0ce090 con 0x7f2fbc0ce5a0 2024-03-18T18:49:21.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.925+0000 7f2fc173c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fbc0cf120 con 0x7f2fbc09dd40 2024-03-18T18:49:21.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.925+0000 7f2fc173c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc06db30 con 0x7f2fbc15d780 2024-03-18T18:49:21.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.926+0000 7f2fba7fc700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 0x7f2fbc0cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:21.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.926+0000 7f2fbaffd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 0x7f2fbc0cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:21.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.926+0000 7f2fba7fc700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 0x7f2fbc0cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:44306/0 (socket says 172.21.15.1:44306) 2024-03-18T18:49:21.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.926+0000 7f2fbb7fe700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f2fbc0ce5a0 0x7f2fbc0cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:43990/0 (socket says 172.21.15.1:43990) 2024-03-18T18:49:21.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.926+0000 7f2fbaffd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 0x7f2fbc0cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:50900/0 (socket says 172.21.15.1:50900) 2024-03-18T18:49:21.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.926+0000 7f2fba7fc700 1 -- 172.21.15.1:0/256672949 learned_addr learned my addr 172.21.15.1:0/256672949 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:21.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.926+0000 7f2fba7fc700 1 -- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 msgr2=0x7f2fbc0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-18T18:49:21.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.926+0000 7f2fba7fc700 1 -- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 msgr2=0x7f2fbc0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T18:49:21.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.926+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 0x7f2fbc0cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T18:49:21.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.926+0000 7f2fbaffd700 1 -- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 msgr2=0x7f2fbc0cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-18T18:49:21.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.926+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 0x7f2fbc0cad70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T18:49:21.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.926+0000 7f2fbaffd700 1 -- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 msgr2=0x7f2fbc0cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T18:49:21.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.926+0000 7f2fbaffd700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 0x7f2fbc0cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T18:49:21.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:21.926+0000 7f2fbaffd700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 0x7f2fbc0cdb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T18:49:22.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.127+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 0x7f2fbc0cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:22.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.127+0000 7f2fbaffd700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 0x7f2fbc0cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:22.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.127+0000 7f2fba7fc700 1 -- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 msgr2=0x7f2fbc0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-18T18:49:22.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.127+0000 7f2fba7fc700 1 -- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 msgr2=0x7f2fbc0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T18:49:22.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.127+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 0x7f2fbc0cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T18:49:22.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.127+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 0x7f2fbc0cad70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T18:49:22.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.127+0000 7f2fbaffd700 1 -- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 msgr2=0x7f2fbc0cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-18T18:49:22.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.127+0000 7f2fbaffd700 1 -- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 msgr2=0x7f2fbc0cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T18:49:22.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.127+0000 7f2fbaffd700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 0x7f2fbc0cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T18:49:22.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.127+0000 7f2fbaffd700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 0x7f2fbc0cdb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T18:49:22.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.528+0000 7f2fbaffd700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 0x7f2fbc0cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:22.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.528+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 0x7f2fbc0cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:22.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.528+0000 7f2fba7fc700 1 -- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 msgr2=0x7f2fbc0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-18T18:49:22.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.528+0000 7f2fba7fc700 1 -- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 msgr2=0x7f2fbc0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T18:49:22.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.528+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 0x7f2fbc0cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T18:49:22.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.528+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 0x7f2fbc0cad70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T18:49:22.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.528+0000 7f2fbaffd700 1 -- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 msgr2=0x7f2fbc0cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-18T18:49:22.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.528+0000 7f2fbaffd700 1 -- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 msgr2=0x7f2fbc0cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T18:49:22.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.528+0000 7f2fbaffd700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 0x7f2fbc0cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T18:49:22.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:22.528+0000 7f2fbaffd700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 0x7f2fbc0cdb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T18:49:23.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:23.329+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 0x7f2fbc0cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:23.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:23.329+0000 7f2fbaffd700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 0x7f2fbc0cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:23.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:23.329+0000 7f2fba7fc700 1 -- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 msgr2=0x7f2fbc0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-18T18:49:23.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:23.329+0000 7f2fba7fc700 1 -- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 msgr2=0x7f2fbc0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T18:49:23.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:23.329+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 0x7f2fbc0cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T18:49:23.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:23.329+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 0x7f2fbc0cad70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T18:49:23.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:23.329+0000 7f2fbaffd700 1 -- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 msgr2=0x7f2fbc0cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-18T18:49:23.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:23.329+0000 7f2fbaffd700 1 -- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 msgr2=0x7f2fbc0cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T18:49:23.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:23.329+0000 7f2fbaffd700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 0x7f2fbc0cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T18:49:23.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:23.330+0000 7f2fbaffd700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 0x7f2fbc0cdb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T18:49:24.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:24.926+0000 7f2fb8ff9700 1 -- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 msgr2=0x7f2fbc0cdb50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T18:49:24.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:24.926+0000 7f2fb8ff9700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 0x7f2fbc0cdb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:24.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:24.926+0000 7f2fb8ff9700 1 -- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 msgr2=0x7f2fbc0cad70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T18:49:24.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:24.926+0000 7f2fb8ff9700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.165:3300/0 conn(0x7f2fbc09dd40 0x7f2fbc0cad70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:24.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:24.926+0000 7f2fb8ff9700 1 -- 172.21.15.1:0/256672949 >> v1:172.21.15.165:6789/0 conn(0x7f2fbc0ce5a0 legacy=0x7f2fbc0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:24.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:24.926+0000 7f2fb8ff9700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 0x7f2fa8009880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:24.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:24.926+0000 7f2fb8ff9700 1 -- 172.21.15.1:0/256672949 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fbc06d520 con 0x7f2fa8003750 2024-03-18T18:49:24.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:24.926+0000 7f2fb8ff9700 1 -- 172.21.15.1:0/256672949 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fbc06db30 con 0x7f2fa80073e0 2024-03-18T18:49:24.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:24.926+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 0x7f2fa8009880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:24.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:24.926+0000 7f2fba7fc700 1 -- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 msgr2=0x7f2fa8009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-18T18:49:24.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:24.926+0000 7f2fba7fc700 1 -- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 msgr2=0x7f2fa8009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T18:49:24.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:24.926+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 0x7f2fa8009880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T18:49:24.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:24.926+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 0x7f2fa8009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T18:49:25.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:25.127+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 0x7f2fa8009880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:25.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:25.127+0000 7f2fba7fc700 1 -- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 msgr2=0x7f2fa8009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-18T18:49:25.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:25.127+0000 7f2fba7fc700 1 -- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 msgr2=0x7f2fa8009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T18:49:25.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:25.127+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 0x7f2fa8009880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T18:49:25.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:25.127+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 0x7f2fa8009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T18:49:25.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:25.528+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 0x7f2fa8009880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:25.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:25.528+0000 7f2fba7fc700 1 -- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 msgr2=0x7f2fa8009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-18T18:49:25.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:25.528+0000 7f2fba7fc700 1 -- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 msgr2=0x7f2fa8009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T18:49:25.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:25.528+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 0x7f2fa8009880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T18:49:25.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:25.528+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 0x7f2fa8009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T18:49:26.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:26.329+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 0x7f2fa8009880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:26.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:26.329+0000 7f2fba7fc700 1 -- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 msgr2=0x7f2fa8009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-18T18:49:26.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:26.329+0000 7f2fba7fc700 1 -- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 msgr2=0x7f2fa8009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T18:49:26.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:26.329+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 0x7f2fa8009880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T18:49:26.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:26.329+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 0x7f2fa8009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T18:49:27.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.143+0000 7f2fb9ffb700 1 -- 172.21.15.1:0/256672949 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180294123 0 0) 0x7f2fbc06d520 con 0x7f2fa8003750 2024-03-18T18:49:27.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.143+0000 7f2fb9ffb700 1 -- 172.21.15.1:0/256672949 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa40035b0 con 0x7f2fa8003750 2024-03-18T18:49:27.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.143+0000 7f2fb9ffb700 1 -- 172.21.15.1:0/256672949 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1151713740 0 0) 0x7f2fa40035b0 con 0x7f2fa8003750 2024-03-18T18:49:27.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.143+0000 7f2fb9ffb700 1 -- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 msgr2=0x7f2fa8009880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T18:49:27.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.143+0000 7f2fb9ffb700 1 --2- 172.21.15.1:0/256672949 >> v2:172.21.15.1:3300/0 conn(0x7f2fa80073e0 0x7f2fa8009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.143+0000 7f2fb9ffb700 1 -- 172.21.15.1:0/256672949 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa800af80 con 0x7f2fa8003750 2024-03-18T18:49:27.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.143+0000 7f2fb9ffb700 1 -- 172.21.15.1:0/256672949 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f2fac00e590 con 0x7f2fa8003750 2024-03-18T18:49:27.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.143+0000 7f2fb9ffb700 1 -- 172.21.15.1:0/256672949 >> v1:172.21.15.1:6789/0 conn(0x7f2fa8003750 legacy=0x7f2fa8005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:27.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.143+0000 7f2fbb7fe700 1 -- 172.21.15.1:0/256672949 reap_dead start 2024-03-18T18:49:27.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.143+0000 7f2fb9ffb700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fa4003a00 0x7f2fa4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:27.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.143+0000 7f2fb9ffb700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 0x7f2fa4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:27.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.143+0000 7f2fb9ffb700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fa4004de0 0x7f2fa4005260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:27.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.143+0000 7f2fb9ffb700 1 -- 172.21.15.1:0/256672949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc06db30 con 0x7f2fa4003a00 2024-03-18T18:49:27.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.143+0000 7f2fb9ffb700 1 -- 172.21.15.1:0/256672949 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa400bc90 con 0x7f2fa4004de0 2024-03-18T18:49:27.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.143+0000 7f2fb9ffb700 1 -- 172.21.15.1:0/256672949 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa400be50 con 0x7f2fbc15d780 2024-03-18T18:49:27.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.143+0000 7f2fbaffd700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fa4003a00 0x7f2fa4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:27.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.143+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 0x7f2fa4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:27.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.144+0000 7f2fbb7fe700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fa4004de0 0x7f2fa4005260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:27.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.144+0000 7f2fbaffd700 1 -- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 msgr2=0x7f2fa4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:27.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.144+0000 7f2fbaffd700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 0x7f2fa4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.144+0000 7f2fbaffd700 1 -- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fa4004de0 msgr2=0x7f2fa4005260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.144+0000 7f2fbaffd700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fa4004de0 0x7f2fa4005260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.144+0000 7f2fbaffd700 1 -- 172.21.15.1:0/256672949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa40035b0 con 0x7f2fa4003a00 2024-03-18T18:49:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.144+0000 7f2fbaffd700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fa4003a00 0x7f2fa4003e00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009100 tx=0x7f2fa4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d86f5ca98f49da server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.144+0000 7f2fb9ffb700 1 -- 172.21.15.1:0/256672949 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac014c20 con 0x7f2fa4003a00 2024-03-18T18:49:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.144+0000 7f2fb9ffb700 1 -- 172.21.15.1:0/256672949 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fac00c070 con 0x7f2fa4003a00 2024-03-18T18:49:27.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.144+0000 7f2fb9ffb700 1 -- 172.21.15.1:0/256672949 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac004020 con 0x7f2fa4003a00 2024-03-18T18:49:27.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.145+0000 7f2fc173c700 1 -- 172.21.15.1:0/256672949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fa4003a00 msgr2=0x7f2fa4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:27.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.145+0000 7f2fc173c700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fa4003a00 0x7f2fa4003e00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009100 tx=0x7f2fa4005880 comp rx=0 tx=0).stop 2024-03-18T18:49:27.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.146+0000 7f2fc173c700 1 -- 172.21.15.1:0/256672949 shutdown_connections 2024-03-18T18:49:27.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.146+0000 7f2fc173c700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fa4004de0 0x7f2fa4005260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.146+0000 7f2fc173c700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fa4003a00 0x7f2fa4003e00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.146+0000 7f2fc173c700 1 --2- 172.21.15.1:0/256672949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fbc15d780 0x7f2fa4004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.146+0000 7f2fc173c700 1 -- 172.21.15.1:0/256672949 >> 172.21.15.1:0/256672949 conn(0x7f2fbc0cca00 msgr2=0x7f2fbc0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:27.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.146+0000 7f2fc173c700 1 -- 172.21.15.1:0/256672949 shutdown_connections 2024-03-18T18:49:27.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.146+0000 7f2fc173c700 1 -- 172.21.15.1:0/256672949 wait complete. 2024-03-18T18:49:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.148+0000 7f2fc173c700 1 Processor -- start 2024-03-18T18:49:27.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.149+0000 7f2fc173c700 1 -- start start 2024-03-18T18:49:27.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.150+0000 7f2fc173c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fa4003a00 0x7f2fbc1f3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:27.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.150+0000 7f2fc173c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fa4004de0 0x7f2fbc0c39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:27.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.150+0000 7f2fc173c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fbc15d780 0x7f2fbc0b0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:27.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.150+0000 7f2fc173c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc0c3f20 con 0x7f2fa4004de0 2024-03-18T18:49:27.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.150+0000 7f2fc173c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc0c81d0 con 0x7f2fbc15d780 2024-03-18T18:49:27.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.150+0000 7f2fc173c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc0c84b0 con 0x7f2fa4003a00 2024-03-18T18:49:27.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.150+0000 7f2fba7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fa4004de0 0x7f2fbc0c39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:27.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.150+0000 7f2fbaffd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fa4003a00 0x7f2fbc1f3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:27.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.150+0000 7f2fbb7fe700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fbc15d780 0x7f2fbc0b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:27.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.150+0000 7f2fbaffd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fa4003a00 0x7f2fbc1f3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:51956/0 (socket says 172.21.15.1:51956) 2024-03-18T18:49:27.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.150+0000 7f2fbaffd700 1 -- 172.21.15.1:0/1228981295 learned_addr learned my addr 172.21.15.1:0/1228981295 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:27.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.151+0000 7f2fba7fc700 1 -- 172.21.15.1:0/1228981295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fa4003a00 msgr2=0x7f2fbc1f3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:27.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.151+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/1228981295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fa4003a00 0x7f2fbc1f3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.151+0000 7f2fba7fc700 1 -- 172.21.15.1:0/1228981295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fbc15d780 msgr2=0x7f2fbc0b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:27.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.151+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/1228981295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fbc15d780 0x7f2fbc0b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.151+0000 7f2fba7fc700 1 -- 172.21.15.1:0/1228981295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fbc0c8790 con 0x7f2fa4004de0 2024-03-18T18:49:27.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.151+0000 7f2fbb7fe700 1 --2- 172.21.15.1:0/1228981295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fbc15d780 0x7f2fbc0b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:49:27.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.151+0000 7f2fba7fc700 1 --2- 172.21.15.1:0/1228981295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fa4004de0 0x7f2fbc0c39e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0011830 tx=0x7f2fb0011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:27.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.151+0000 7f2fa37fe700 1 -- 172.21.15.1:0/1228981295 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb000ba50 con 0x7f2fa4004de0 2024-03-18T18:49:27.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.151+0000 7f2fa37fe700 1 -- 172.21.15.1:0/1228981295 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb0017040 con 0x7f2fa4004de0 2024-03-18T18:49:27.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.151+0000 7f2fc173c700 1 -- 172.21.15.1:0/1228981295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fbc0c8a70 con 0x7f2fa4004de0 2024-03-18T18:49:27.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.151+0000 7f2fa37fe700 1 -- 172.21.15.1:0/1228981295 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb000bdf0 con 0x7f2fa4004de0 2024-03-18T18:49:27.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.151+0000 7f2fc173c700 1 -- 172.21.15.1:0/1228981295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fbc0b0c90 con 0x7f2fa4004de0 2024-03-18T18:49:27.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.151+0000 7f2fa37fe700 1 -- 172.21.15.1:0/1228981295 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2fb0026750 con 0x7f2fa4004de0 2024-03-18T18:49:27.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.152+0000 7f2fa37fe700 1 -- 172.21.15.1:0/1228981295 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2fb0005a20 con 0x7f2fa4004de0 2024-03-18T18:49:27.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.152+0000 7f2fc173c700 1 -- 172.21.15.1:0/1228981295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fbc0c8f90 con 0x7f2fa4004de0 2024-03-18T18:49:27.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.155+0000 7f2fa37fe700 1 -- 172.21.15.1:0/1228981295 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f2fb0045b50 con 0x7f2fa4004de0 2024-03-18T18:49:27.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.235+0000 7f2fc173c700 1 -- 172.21.15.1:0/1228981295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f2fbc0676a0 con 0x7f2fa4004de0 2024-03-18T18:49:27.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.237+0000 7f2fa37fe700 1 -- 172.21.15.1:0/1228981295 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f2fb0034070 con 0x7f2fa4004de0 2024-03-18T18:49:27.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.238+0000 7f2fc173c700 1 -- 172.21.15.1:0/1228981295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fa4004de0 msgr2=0x7f2fbc0c39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:27.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.238+0000 7f2fc173c700 1 --2- 172.21.15.1:0/1228981295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fa4004de0 0x7f2fbc0c39e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0011830 tx=0x7f2fb0011b40 comp rx=0 tx=0).stop 2024-03-18T18:49:27.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.238+0000 7f2fc173c700 1 -- 172.21.15.1:0/1228981295 shutdown_connections 2024-03-18T18:49:27.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.238+0000 7f2fc173c700 1 --2- 172.21.15.1:0/1228981295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fa4003a00 0x7f2fbc1f3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.238+0000 7f2fc173c700 1 --2- 172.21.15.1:0/1228981295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fbc15d780 0x7f2fbc0b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.238+0000 7f2fc173c700 1 --2- 172.21.15.1:0/1228981295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fa4004de0 0x7f2fbc0c39e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.238+0000 7f2fc173c700 1 -- 172.21.15.1:0/1228981295 >> 172.21.15.1:0/1228981295 conn(0x7f2fbc0cca00 msgr2=0x7f2fbc0b1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:27.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.238+0000 7f2fc173c700 1 -- 172.21.15.1:0/1228981295 shutdown_connections 2024-03-18T18:49:27.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:27.238+0000 7f2fc173c700 1 -- 172.21.15.1:0/1228981295 wait complete. 2024-03-18T18:49:27.237 INFO:teuthology.orchestra.run.smithi001.stderr:adjusted tunables profile to default 2024-03-18T18:49:27.249 INFO:tasks.ceph:check_enable_crimson: False 2024-03-18T18:49:27.249 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-18T18:49:27.250 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:49:27.250 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-18T18:49:27.278 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:49:27.279 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-18T18:49:27.280 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:27.282+0000 7fbea244d240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T18:49:27.307 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:49:27.307 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-18T18:49:27.375 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:49:27.375 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-18T18:49:27.461 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:49:27.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-18T18:49:27.473 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:27.476+0000 7fbea244d240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T18:49:27.495 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:49:27.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-18T18:49:27.565 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:49:27.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-18T18:49:27.636 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:49:27.637 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-18T18:49:27.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 380d0a40-8340-4654-9353-97df2416c043 0 2024-03-18T18:49:27.871 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.874+0000 7f07a0376700 1 Processor -- start 2024-03-18T18:49:27.871 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.874+0000 7f07a0376700 1 -- start start 2024-03-18T18:49:27.873 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.874+0000 7f07a0376700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0798141720 0x7f07980cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:27.873 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.875+0000 7f07a0376700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f079815c0e0 0x7f07980cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:27.873 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.875+0000 7f07a0376700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07980cbb10 con 0x7f07980cf760 2024-03-18T18:49:27.873 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.875+0000 7f07a0376700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f079815c550 con 0x7f0798141720 2024-03-18T18:49:27.873 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.875+0000 7f07a0376700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07980d0130 con 0x7f079815c0e0 2024-03-18T18:49:27.873 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.875+0000 7f079e913700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f079815c0e0 0x7f07980cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:27.873 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.875+0000 7f079e112700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0798141720 0x7f07980cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:27.873 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.875+0000 7f079e112700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0798141720 0x7f07980cc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:54168/0 (socket says 172.21.15.165:54168) 2024-03-18T18:49:27.873 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.875+0000 7f079e913700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f079815c0e0 0x7f07980cfbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:59630/0 (socket says 172.21.15.165:59630) 2024-03-18T18:49:27.874 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.875+0000 7f079e112700 1 -- 172.21.15.165:0/3658890627 learned_addr learned my addr 172.21.15.165:0/3658890627 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-18T18:49:27.874 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.875+0000 7f079e913700 1 -- 172.21.15.165:0/3658890627 >> v2:172.21.15.165:3300/0 conn(0x7f0798141720 msgr2=0x7f07980cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:27.874 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.875+0000 7f079e913700 1 --2- 172.21.15.165:0/3658890627 >> v2:172.21.15.165:3300/0 conn(0x7f0798141720 0x7f07980cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.874 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.875+0000 7f079e913700 1 -- 172.21.15.165:0/3658890627 >> v1:172.21.15.165:6789/0 conn(0x7f07980cf760 legacy=0x7f079815b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:27.874 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.875+0000 7f079e913700 1 -- 172.21.15.165:0/3658890627 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07980cbdf0 con 0x7f079815c0e0 2024-03-18T18:49:27.874 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.875+0000 7f079e913700 1 --2- 172.21.15.165:0/3658890627 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f079815c0e0 0x7f07980cfbf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0790009d90 tx=0x7f079000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f3fcf080f654ecb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:27.874 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.875+0000 7f079d110700 1 -- 172.21.15.165:0/3658890627 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079000e070 con 0x7f079815c0e0 2024-03-18T18:49:27.874 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.875+0000 7f079d110700 1 -- 172.21.15.165:0/3658890627 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079000bb20 con 0x7f079815c0e0 2024-03-18T18:49:27.874 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.875+0000 7f079d110700 1 -- 172.21.15.165:0/3658890627 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0790012430 con 0x7f079815c0e0 2024-03-18T18:49:27.874 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.876+0000 7f07a0376700 1 -- 172.21.15.165:0/3658890627 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f079815c0e0 msgr2=0x7f07980cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:27.874 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.876+0000 7f07a0376700 1 --2- 172.21.15.165:0/3658890627 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f079815c0e0 0x7f07980cfbf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0790009d90 tx=0x7f079000b330 comp rx=0 tx=0).stop 2024-03-18T18:49:27.874 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.876+0000 7f07a0376700 1 -- 172.21.15.165:0/3658890627 shutdown_connections 2024-03-18T18:49:27.874 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.876+0000 7f07a0376700 1 --2- 172.21.15.165:0/3658890627 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f079815c0e0 0x7f07980cfbf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.875 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.876+0000 7f07a0376700 1 --2- 172.21.15.165:0/3658890627 >> v2:172.21.15.165:3300/0 conn(0x7f0798141720 0x7f07980cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.875 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.876+0000 7f07a0376700 1 -- 172.21.15.165:0/3658890627 >> 172.21.15.165:0/3658890627 conn(0x7f0798066b70 msgr2=0x7f079806a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:27.875 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.876+0000 7f07a0376700 1 -- 172.21.15.165:0/3658890627 shutdown_connections 2024-03-18T18:49:27.875 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.876+0000 7f07a0376700 1 -- 172.21.15.165:0/3658890627 wait complete. 2024-03-18T18:49:27.875 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:27.875+0000 7fbea244d240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T18:49:27.877 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.877+0000 7f07a0376700 1 Processor -- start 2024-03-18T18:49:27.877 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.877+0000 7f07a0376700 1 -- start start 2024-03-18T18:49:27.877 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.878+0000 7f07a0376700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07980cf760 0x7f07981f0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:27.877 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.878+0000 7f07a0376700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0798141720 0x7f079814b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:27.877 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.878+0000 7f07a0376700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f079815c0e0 0x7f07981382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:27.877 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.878+0000 7f07a0376700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079813e7b0 con 0x7f07980cf760 2024-03-18T18:49:27.878 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.878+0000 7f07a0376700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079813ea50 con 0x7f0798141720 2024-03-18T18:49:27.878 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.878+0000 7f07a0376700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f079813ed30 con 0x7f079815c0e0 2024-03-18T18:49:27.878 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.878+0000 7f079e913700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f079815c0e0 0x7f07981382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:27.878 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.878+0000 7f079d911700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0798141720 0x7f079814b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:27.878 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.878+0000 7f079e112700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07980cf760 0x7f07981f0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:27.878 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.878+0000 7f079e913700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f079815c0e0 0x7f07981382f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:59640/0 (socket says 172.21.15.165:59640) 2024-03-18T18:49:27.878 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.878+0000 7f079e913700 1 -- 172.21.15.165:0/1029997778 learned_addr learned my addr 172.21.15.165:0/1029997778 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-18T18:49:27.878 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.879+0000 7f079e112700 1 -- 172.21.15.165:0/1029997778 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f079815c0e0 msgr2=0x7f07981382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:27.878 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.879+0000 7f079e112700 1 --2- 172.21.15.165:0/1029997778 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f079815c0e0 0x7f07981382f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.878 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.879+0000 7f079e112700 1 -- 172.21.15.165:0/1029997778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0798141720 msgr2=0x7f079814b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:27.878 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.879+0000 7f079e112700 1 --2- 172.21.15.165:0/1029997778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0798141720 0x7f079814b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.878 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.879+0000 7f079e112700 1 -- 172.21.15.165:0/1029997778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0790009a70 con 0x7f07980cf760 2024-03-18T18:49:27.878 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.879+0000 7f079d911700 1 --2- 172.21.15.165:0/1029997778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0798141720 0x7f079814b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:49:27.879 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.879+0000 7f079e112700 1 --2- 172.21.15.165:0/1029997778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07980cf760 0x7f07981f0bc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f078800e8e0 tx=0x7f078800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:27.879 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.879+0000 7f078effd700 1 -- 172.21.15.165:0/1029997778 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0788013680 con 0x7f07980cf760 2024-03-18T18:49:27.879 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.879+0000 7f078effd700 1 -- 172.21.15.165:0/1029997778 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07880137e0 con 0x7f07980cf760 2024-03-18T18:49:27.879 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.879+0000 7f078effd700 1 -- 172.21.15.165:0/1029997778 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0788013b70 con 0x7f07980cf760 2024-03-18T18:49:27.879 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.879+0000 7f07a0376700 1 -- 172.21.15.165:0/1029997778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f079813f010 con 0x7f07980cf760 2024-03-18T18:49:27.879 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.879+0000 7f07a0376700 1 -- 172.21.15.165:0/1029997778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07981f1100 con 0x7f07980cf760 2024-03-18T18:49:27.879 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.879+0000 7f078effd700 1 -- 172.21.15.165:0/1029997778 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0788005a20 con 0x7f07980cf760 2024-03-18T18:49:27.879 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.879+0000 7f078effd700 1 -- 172.21.15.165:0/1029997778 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f078801c790 con 0x7f07980cf760 2024-03-18T18:49:27.879 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.880+0000 7f07a0376700 1 -- 172.21.15.165:0/1029997778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07981f12f0 con 0x7f07980cf760 2024-03-18T18:49:27.879 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.882+0000 7f078effd700 1 -- 172.21.15.165:0/1029997778 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f0788021060 con 0x7f07980cf760 2024-03-18T18:49:27.962 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.965+0000 7f07a0376700 1 -- 172.21.15.165:0/1029997778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd new", "uuid": "380d0a40-8340-4654-9353-97df2416c043", "id": 0} v 0) v1 -- 0x7f079813f660 con 0x7f07980cf760 2024-03-18T18:49:27.964 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.967+0000 7f078effd700 1 -- 172.21.15.165:0/1029997778 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "380d0a40-8340-4654-9353-97df2416c043", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0788017070 con 0x7f07980cf760 2024-03-18T18:49:27.964 INFO:teuthology.orchestra.run.smithi165.stdout:0 2024-03-18T18:49:27.965 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.968+0000 7f07a0376700 1 -- 172.21.15.165:0/1029997778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07980cf760 msgr2=0x7f07981f0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:27.965 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.968+0000 7f07a0376700 1 --2- 172.21.15.165:0/1029997778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07980cf760 0x7f07981f0bc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f078800e8e0 tx=0x7f078800ebf0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.965 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.968+0000 7f07a0376700 1 -- 172.21.15.165:0/1029997778 shutdown_connections 2024-03-18T18:49:27.966 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.968+0000 7f07a0376700 1 --2- 172.21.15.165:0/1029997778 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f079815c0e0 0x7f07981382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.966 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.968+0000 7f07a0376700 1 --2- 172.21.15.165:0/1029997778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07980cf760 0x7f07981f0bc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.966 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.968+0000 7f07a0376700 1 --2- 172.21.15.165:0/1029997778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0798141720 0x7f079814b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:27.966 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.968+0000 7f07a0376700 1 -- 172.21.15.165:0/1029997778 >> 172.21.15.165:0/1029997778 conn(0x7f0798066b70 msgr2=0x7f079806e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:27.966 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.968+0000 7f07a0376700 1 -- 172.21.15.165:0/1029997778 shutdown_connections 2024-03-18T18:49:27.966 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:27.968+0000 7f07a0376700 1 -- 172.21.15.165:0/1029997778 wait complete. 2024-03-18T18:49:27.977 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 9c851da3-1c5d-4839-aba9-2d35a12ac26f 1 2024-03-18T18:49:28.007 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:28.010+0000 7ff33f4e5240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T18:49:28.100 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.103+0000 7fdb001fd700 1 Processor -- start 2024-03-18T18:49:28.100 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.103+0000 7fdb001fd700 1 -- start start 2024-03-18T18:49:28.101 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.105+0000 7fdb001fd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdaf815a270 0x7fdaf815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:28.101 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.105+0000 7fdb001fd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaf806e310 0x7fdaf8070dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:28.101 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.105+0000 7fdb001fd700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf815cb80 con 0x7fdaf8071310 2024-03-18T18:49:28.102 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.105+0000 7fdb001fd700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaf8071790 con 0x7fdaf815a270 2024-03-18T18:49:28.102 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.105+0000 7fdb001fd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf80aedf0 con 0x7fdaf806e310 2024-03-18T18:49:28.102 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.105+0000 7fdafd798700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaf806e310 0x7fdaf8070dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:28.102 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.105+0000 7fdafd798700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaf806e310 0x7fdaf8070dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:59648/0 (socket says 172.21.15.165:59648) 2024-03-18T18:49:28.102 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.105+0000 7fdafd798700 1 -- 172.21.15.165:0/1017070118 learned_addr learned my addr 172.21.15.165:0/1017070118 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-18T18:49:28.103 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.106+0000 7fdafdf99700 1 --2- 172.21.15.165:0/1017070118 >> v2:172.21.15.1:3300/0 conn(0x7fdaf815a270 0x7fdaf815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:28.103 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.106+0000 7fdafd798700 1 -- 172.21.15.165:0/1017070118 >> v2:172.21.15.1:3300/0 conn(0x7fdaf815a270 msgr2=0x7fdaf815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:28.103 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.106+0000 7fdafd798700 1 --2- 172.21.15.165:0/1017070118 >> v2:172.21.15.1:3300/0 conn(0x7fdaf815a270 0x7fdaf815e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.103 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.106+0000 7fdafd798700 1 -- 172.21.15.165:0/1017070118 >> v1:172.21.15.165:6789/0 conn(0x7fdaf8071310 legacy=0x7fdaf806ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.103 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.106+0000 7fdafd798700 1 -- 172.21.15.165:0/1017070118 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf815a940 con 0x7fdaf806e310 2024-03-18T18:49:28.103 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.106+0000 7fdafdf99700 1 --2- 172.21.15.165:0/1017070118 >> v2:172.21.15.1:3300/0 conn(0x7fdaf815a270 0x7fdaf815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:49:28.103 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.106+0000 7fdafd798700 1 --2- 172.21.15.165:0/1017070118 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaf806e310 0x7fdaf8070dd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdaf40098f0 tx=0x7fdaf4009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d656ef48719e20a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:28.104 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.106+0000 7fdafcf97700 1 -- 172.21.15.165:0/1017070118 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf400b8a0 con 0x7fdaf806e310 2024-03-18T18:49:28.104 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.106+0000 7fdafcf97700 1 -- 172.21.15.165:0/1017070118 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf400ba00 con 0x7fdaf806e310 2024-03-18T18:49:28.104 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.106+0000 7fdafcf97700 1 -- 172.21.15.165:0/1017070118 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf400bd90 con 0x7fdaf806e310 2024-03-18T18:49:28.105 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.107+0000 7fdb001fd700 1 -- 172.21.15.165:0/1017070118 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaf806e310 msgr2=0x7fdaf8070dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.105 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.107+0000 7fdb001fd700 1 --2- 172.21.15.165:0/1017070118 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaf806e310 0x7fdaf8070dd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdaf40098f0 tx=0x7fdaf4009c00 comp rx=0 tx=0).stop 2024-03-18T18:49:28.105 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.107+0000 7fdb001fd700 1 -- 172.21.15.165:0/1017070118 shutdown_connections 2024-03-18T18:49:28.105 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.107+0000 7fdb001fd700 1 --2- 172.21.15.165:0/1017070118 >> v2:172.21.15.1:3300/0 conn(0x7fdaf815a270 0x7fdaf815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.105 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.107+0000 7fdb001fd700 1 --2- 172.21.15.165:0/1017070118 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaf806e310 0x7fdaf8070dd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.105 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.107+0000 7fdb001fd700 1 -- 172.21.15.165:0/1017070118 >> 172.21.15.165:0/1017070118 conn(0x7fdaf8066b70 msgr2=0x7fdaf806a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:28.105 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.107+0000 7fdb001fd700 1 -- 172.21.15.165:0/1017070118 shutdown_connections 2024-03-18T18:49:28.105 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.108+0000 7fdb001fd700 1 -- 172.21.15.165:0/1017070118 wait complete. 2024-03-18T18:49:28.106 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.109+0000 7fdb001fd700 1 Processor -- start 2024-03-18T18:49:28.106 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.109+0000 7fdb001fd700 1 -- start start 2024-03-18T18:49:28.110 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.110+0000 7fdb001fd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaf806e310 0x7fdaf80b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:28.110 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.110+0000 7fdb001fd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaf8071310 0x7fdaf8148e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:28.110 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.110+0000 7fdb001fd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaf815a270 0x7fdaf814e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:28.110 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.110+0000 7fdb001fd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf814c450 con 0x7fdaf8071310 2024-03-18T18:49:28.110 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.110+0000 7fdb001fd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf80b0e90 con 0x7fdaf806e310 2024-03-18T18:49:28.110 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.110+0000 7fdb001fd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf80b1170 con 0x7fdaf815a270 2024-03-18T18:49:28.110 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.111+0000 7fdafdf99700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaf806e310 0x7fdaf80b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:28.110 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.111+0000 7fdafdf99700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaf806e310 0x7fdaf80b1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:54216/0 (socket says 172.21.15.165:54216) 2024-03-18T18:49:28.110 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.111+0000 7fdafdf99700 1 -- 172.21.15.165:0/1348310397 learned_addr learned my addr 172.21.15.165:0/1348310397 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-18T18:49:28.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.111+0000 7fdafe79a700 1 --2- 172.21.15.165:0/1348310397 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaf815a270 0x7fdaf814e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:28.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.111+0000 7fdafdf99700 1 -- 172.21.15.165:0/1348310397 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaf815a270 msgr2=0x7fdaf814e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.111+0000 7fdafdf99700 1 --2- 172.21.15.165:0/1348310397 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaf815a270 0x7fdaf814e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.111+0000 7fdafdf99700 1 -- 172.21.15.165:0/1348310397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaf8071310 msgr2=0x7fdaf8148e40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T18:49:28.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.111+0000 7fdafdf99700 1 --2- 172.21.15.165:0/1348310397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaf8071310 0x7fdaf8148e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.111+0000 7fdafdf99700 1 -- 172.21.15.165:0/1348310397 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf4009550 con 0x7fdaf806e310 2024-03-18T18:49:28.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.111+0000 7fdafdf99700 1 --2- 172.21.15.165:0/1348310397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaf806e310 0x7fdaf80b1f30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdaf000cf60 tx=0x7fdaf000cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:28.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.111+0000 7fdaee7fc700 1 -- 172.21.15.165:0/1348310397 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf000ae50 con 0x7fdaf806e310 2024-03-18T18:49:28.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.112+0000 7fdaee7fc700 1 -- 172.21.15.165:0/1348310397 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf0015440 con 0x7fdaf806e310 2024-03-18T18:49:28.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.112+0000 7fdaee7fc700 1 -- 172.21.15.165:0/1348310397 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf00157d0 con 0x7fdaf806e310 2024-03-18T18:49:28.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.112+0000 7fdb001fd700 1 -- 172.21.15.165:0/1348310397 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaf80b1450 con 0x7fdaf806e310 2024-03-18T18:49:28.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.112+0000 7fdb001fd700 1 -- 172.21.15.165:0/1348310397 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf80b1a10 con 0x7fdaf806e310 2024-03-18T18:49:28.112 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.112+0000 7fdaee7fc700 1 -- 172.21.15.165:0/1348310397 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fdaf0015c30 con 0x7fdaf806e310 2024-03-18T18:49:28.112 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.112+0000 7fdaee7fc700 1 -- 172.21.15.165:0/1348310397 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fdaf001f7f0 con 0x7fdaf806e310 2024-03-18T18:49:28.112 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.113+0000 7fdb001fd700 1 -- 172.21.15.165:0/1348310397 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf80b12b0 con 0x7fdaf806e310 2024-03-18T18:49:28.113 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.116+0000 7fdaee7fc700 1 -- 172.21.15.165:0/1348310397 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fdaf80b12b0 con 0x7fdaf806e310 2024-03-18T18:49:28.184 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:28.188+0000 7ff33f4e5240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T18:49:28.222 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.225+0000 7fdb001fd700 1 -- 172.21.15.165:0/1348310397 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9c851da3-1c5d-4839-aba9-2d35a12ac26f", "id": 1} v 0) v1 -- 0x7fdaf80b1c30 con 0x7fdaf806e310 2024-03-18T18:49:28.226 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.229+0000 7fdaee7fc700 1 -- 172.21.15.165:0/1348310397 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9c851da3-1c5d-4839-aba9-2d35a12ac26f", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fdaf002d0b0 con 0x7fdaf806e310 2024-03-18T18:49:28.226 INFO:teuthology.orchestra.run.smithi165.stdout:1 2024-03-18T18:49:28.227 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.230+0000 7fdb001fd700 1 -- 172.21.15.165:0/1348310397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaf806e310 msgr2=0x7fdaf80b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.227 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.230+0000 7fdb001fd700 1 --2- 172.21.15.165:0/1348310397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaf806e310 0x7fdaf80b1f30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdaf000cf60 tx=0x7fdaf000cf90 comp rx=0 tx=0).stop 2024-03-18T18:49:28.227 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.230+0000 7fdb001fd700 1 -- 172.21.15.165:0/1348310397 shutdown_connections 2024-03-18T18:49:28.227 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.230+0000 7fdb001fd700 1 --2- 172.21.15.165:0/1348310397 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaf815a270 0x7fdaf814e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.227 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.230+0000 7fdb001fd700 1 --2- 172.21.15.165:0/1348310397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaf806e310 0x7fdaf80b1f30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.227 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.230+0000 7fdb001fd700 1 --2- 172.21.15.165:0/1348310397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaf8071310 0x7fdaf8148e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.227 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.230+0000 7fdb001fd700 1 -- 172.21.15.165:0/1348310397 >> 172.21.15.165:0/1348310397 conn(0x7fdaf8066b70 msgr2=0x7fdaf806a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:28.228 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.231+0000 7fdb001fd700 1 -- 172.21.15.165:0/1348310397 shutdown_connections 2024-03-18T18:49:28.228 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.231+0000 7fdb001fd700 1 -- 172.21.15.165:0/1348310397 wait complete. 2024-03-18T18:49:28.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 36be270f-cbee-4db9-a511-9bbb18e21a2f 2 2024-03-18T18:49:28.384 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:28.386+0000 7fbea244d240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T18:49:28.401 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.404+0000 7fdbed57a700 1 Processor -- start 2024-03-18T18:49:28.401 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.405+0000 7fdbed57a700 1 -- start start 2024-03-18T18:49:28.404 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.405+0000 7fdbed57a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdbe806a750 0x7fdbe806ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:28.404 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.405+0000 7fdbed57a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdbe815b900 0x7fdbe815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:28.404 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.405+0000 7fdbed57a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe806b070 con 0x7fdbe806d560 2024-03-18T18:49:28.404 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.405+0000 7fdbed57a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe815ef00 con 0x7fdbe815b900 2024-03-18T18:49:28.404 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.405+0000 7fdbed57a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe815f1e0 con 0x7fdbe806a750 2024-03-18T18:49:28.404 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.405+0000 7fdbe6ffd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdbe806a750 0x7fdbe806ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:28.404 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.405+0000 7fdbe6ffd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdbe806a750 0x7fdbe806ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:54222/0 (socket says 172.21.15.165:54222) 2024-03-18T18:49:28.404 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.405+0000 7fdbe6ffd700 1 -- 172.21.15.165:0/1242942455 learned_addr learned my addr 172.21.15.165:0/1242942455 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-18T18:49:28.404 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.406+0000 7fdbe77fe700 1 --2- 172.21.15.165:0/1242942455 >> v2:172.21.15.1:3300/0 conn(0x7fdbe815b900 0x7fdbe815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:28.404 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.406+0000 7fdbe6ffd700 1 -- 172.21.15.165:0/1242942455 >> v2:172.21.15.1:3300/0 conn(0x7fdbe815b900 msgr2=0x7fdbe815bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:28.405 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.406+0000 7fdbe6ffd700 1 --2- 172.21.15.165:0/1242942455 >> v2:172.21.15.1:3300/0 conn(0x7fdbe815b900 0x7fdbe815bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.405 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.406+0000 7fdbe6ffd700 1 -- 172.21.15.165:0/1242942455 >> v1:172.21.15.1:6789/0 conn(0x7fdbe806d560 legacy=0x7fdbe80aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.405 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.406+0000 7fdbe6ffd700 1 -- 172.21.15.165:0/1242942455 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe815f4c0 con 0x7fdbe806a750 2024-03-18T18:49:28.405 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.406+0000 7fdbe6ffd700 1 --2- 172.21.15.165:0/1242942455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbe806a750 0x7fdbe806ab30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fdbe80b2a70 tx=0x7fdbe000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb1c40d1eb59f601 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:28.405 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.406+0000 7fdbe5ffb700 1 -- 172.21.15.165:0/1242942455 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe000e070 con 0x7fdbe806a750 2024-03-18T18:49:28.405 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.406+0000 7fdbe5ffb700 1 -- 172.21.15.165:0/1242942455 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe000b910 con 0x7fdbe806a750 2024-03-18T18:49:28.405 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.406+0000 7fdbe5ffb700 1 -- 172.21.15.165:0/1242942455 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe000bcc0 con 0x7fdbe806a750 2024-03-18T18:49:28.405 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.407+0000 7fdbed57a700 1 -- 172.21.15.165:0/1242942455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbe806a750 msgr2=0x7fdbe806ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.405 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.407+0000 7fdbed57a700 1 --2- 172.21.15.165:0/1242942455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbe806a750 0x7fdbe806ab30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fdbe80b2a70 tx=0x7fdbe000b330 comp rx=0 tx=0).stop 2024-03-18T18:49:28.405 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.407+0000 7fdbed57a700 1 -- 172.21.15.165:0/1242942455 shutdown_connections 2024-03-18T18:49:28.405 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.407+0000 7fdbed57a700 1 --2- 172.21.15.165:0/1242942455 >> v2:172.21.15.1:3300/0 conn(0x7fdbe815b900 0x7fdbe815bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.405 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.407+0000 7fdbed57a700 1 --2- 172.21.15.165:0/1242942455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbe806a750 0x7fdbe806ab30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.405 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.407+0000 7fdbed57a700 1 -- 172.21.15.165:0/1242942455 >> 172.21.15.165:0/1242942455 conn(0x7fdbe80cff30 msgr2=0x7fdbe80d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:28.405 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.407+0000 7fdbed57a700 1 -- 172.21.15.165:0/1242942455 shutdown_connections 2024-03-18T18:49:28.406 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.407+0000 7fdbed57a700 1 -- 172.21.15.165:0/1242942455 wait complete. 2024-03-18T18:49:28.408 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.408+0000 7fdbed57a700 1 Processor -- start 2024-03-18T18:49:28.408 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.408+0000 7fdbed57a700 1 -- start start 2024-03-18T18:49:28.408 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbed57a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbe806a750 0x7fdbe8155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:28.408 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbed57a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdbe806d560 0x7fdbe80b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:28.408 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbed57a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbe815b900 0x7fdbe815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:28.408 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbed57a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe80b1700 con 0x7fdbe815b900 2024-03-18T18:49:28.408 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbed57a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe8155fb0 con 0x7fdbe806a750 2024-03-18T18:49:28.408 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbed57a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe80b50e0 con 0x7fdbe806d560 2024-03-18T18:49:28.408 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbe6ffd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbe806a750 0x7fdbe8155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:28.409 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbe6ffd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbe806a750 0x7fdbe8155a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:54230/0 (socket says 172.21.15.165:54230) 2024-03-18T18:49:28.409 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbe6ffd700 1 -- 172.21.15.165:0/4060726946 learned_addr learned my addr 172.21.15.165:0/4060726946 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-18T18:49:28.409 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbe67fc700 1 --2- 172.21.15.165:0/4060726946 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdbe806d560 0x7fdbe80b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:28.409 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbe77fe700 1 --2- 172.21.15.165:0/4060726946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbe815b900 0x7fdbe815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:28.409 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbe6ffd700 1 -- 172.21.15.165:0/4060726946 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdbe806d560 msgr2=0x7fdbe80b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.409 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbe6ffd700 1 --2- 172.21.15.165:0/4060726946 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdbe806d560 0x7fdbe80b1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.409 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbe6ffd700 1 -- 172.21.15.165:0/4060726946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbe815b900 msgr2=0x7fdbe815bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.409 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbe6ffd700 1 --2- 172.21.15.165:0/4060726946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbe815b900 0x7fdbe815bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.409 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbe6ffd700 1 -- 172.21.15.165:0/4060726946 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe0009b30 con 0x7fdbe806a750 2024-03-18T18:49:28.409 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbe67fc700 1 --2- 172.21.15.165:0/4060726946 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdbe806d560 0x7fdbe80b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:49:28.409 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbe6ffd700 1 --2- 172.21.15.165:0/4060726946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbe806a750 0x7fdbe8155a70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdbe815f320 tx=0x7fdbe0004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:28.409 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbcf7fe700 1 -- 172.21.15.165:0/4060726946 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe00048c0 con 0x7fdbe806a750 2024-03-18T18:49:28.409 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbcf7fe700 1 -- 172.21.15.165:0/4060726946 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe00099b0 con 0x7fdbe806a750 2024-03-18T18:49:28.409 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbcf7fe700 1 -- 172.21.15.165:0/4060726946 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe001c680 con 0x7fdbe806a750 2024-03-18T18:49:28.410 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.409+0000 7fdbed57a700 1 -- 172.21.15.165:0/4060726946 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe80b53c0 con 0x7fdbe806a750 2024-03-18T18:49:28.410 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.410+0000 7fdbed57a700 1 -- 172.21.15.165:0/4060726946 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe80b1320 con 0x7fdbe806a750 2024-03-18T18:49:28.410 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.410+0000 7fdbcf7fe700 1 -- 172.21.15.165:0/4060726946 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fdbe001cae0 con 0x7fdbe806a750 2024-03-18T18:49:28.410 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.410+0000 7fdbcf7fe700 1 -- 172.21.15.165:0/4060726946 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fdbe0025cd0 con 0x7fdbe806a750 2024-03-18T18:49:28.410 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.410+0000 7fdbed57a700 1 -- 172.21.15.165:0/4060726946 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe80b1970 con 0x7fdbe806a750 2024-03-18T18:49:28.410 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.413+0000 7fdbcf7fe700 1 -- 172.21.15.165:0/4060726946 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fdbe0043b50 con 0x7fdbe806a750 2024-03-18T18:49:28.492 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.495+0000 7fdbed57a700 1 -- 172.21.15.165:0/4060726946 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd new", "uuid": "36be270f-cbee-4db9-a511-9bbb18e21a2f", "id": 2} v 0) v1 -- 0x7fdbe80676a0 con 0x7fdbe806a750 2024-03-18T18:49:28.496 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.499+0000 7fdbcf7fe700 1 -- 172.21.15.165:0/4060726946 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "36be270f-cbee-4db9-a511-9bbb18e21a2f", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fdbe0032030 con 0x7fdbe806a750 2024-03-18T18:49:28.496 INFO:teuthology.orchestra.run.smithi165.stdout:2 2024-03-18T18:49:28.498 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.500+0000 7fdbed57a700 1 -- 172.21.15.165:0/4060726946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbe806a750 msgr2=0x7fdbe8155a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.498 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.500+0000 7fdbed57a700 1 --2- 172.21.15.165:0/4060726946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbe806a750 0x7fdbe8155a70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdbe815f320 tx=0x7fdbe0004ad0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.498 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.500+0000 7fdbed57a700 1 -- 172.21.15.165:0/4060726946 shutdown_connections 2024-03-18T18:49:28.498 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.500+0000 7fdbed57a700 1 --2- 172.21.15.165:0/4060726946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbe815b900 0x7fdbe815bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.498 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.500+0000 7fdbed57a700 1 --2- 172.21.15.165:0/4060726946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbe806a750 0x7fdbe8155a70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.498 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.500+0000 7fdbed57a700 1 --2- 172.21.15.165:0/4060726946 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdbe806d560 0x7fdbe80b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.498 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.500+0000 7fdbed57a700 1 -- 172.21.15.165:0/4060726946 >> 172.21.15.165:0/4060726946 conn(0x7fdbe80cff30 msgr2=0x7fdbe80ca560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:28.499 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.500+0000 7fdbed57a700 1 -- 172.21.15.165:0/4060726946 shutdown_connections 2024-03-18T18:49:28.499 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.500+0000 7fdbed57a700 1 -- 172.21.15.165:0/4060726946 wait complete. 2024-03-18T18:49:28.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 428759f1-b3c5-4d09-9ec1-d39c92985a96 3 2024-03-18T18:49:28.597 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:28.600+0000 7ff33f4e5240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T18:49:28.607 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:28.610+0000 7fbea244d240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T18:49:28.632 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.635+0000 7f7d97861700 1 Processor -- start 2024-03-18T18:49:28.632 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.635+0000 7f7d97861700 1 -- start start 2024-03-18T18:49:28.634 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.636+0000 7f7d97861700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d900cb7b0 0x7f7d90099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:28.634 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.636+0000 7f7d97861700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7d90099f80 0x7f7d900cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:28.634 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.636+0000 7f7d97861700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9009a3e0 con 0x7f7d9015d500 2024-03-18T18:49:28.634 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.636+0000 7f7d97861700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d9006f490 con 0x7f7d90099f80 2024-03-18T18:49:28.635 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.636+0000 7f7d97861700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9006e880 con 0x7f7d900cb7b0 2024-03-18T18:49:28.635 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.636+0000 7f7d94dfc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d900cb7b0 0x7f7d90099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:28.635 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.636+0000 7f7d95dfe700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7d90099f80 0x7f7d900cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:28.635 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.636+0000 7f7d94dfc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d900cb7b0 0x7f7d90099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:59674/0 (socket says 172.21.15.165:59674) 2024-03-18T18:49:28.635 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.636+0000 7f7d94dfc700 1 -- 172.21.15.165:0/4028129738 learned_addr learned my addr 172.21.15.165:0/4028129738 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-18T18:49:28.635 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.636+0000 7f7d94dfc700 1 -- 172.21.15.165:0/4028129738 >> v2:172.21.15.165:3300/0 conn(0x7f7d90099f80 msgr2=0x7f7d900cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:28.635 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.636+0000 7f7d94dfc700 1 --2- 172.21.15.165:0/4028129738 >> v2:172.21.15.165:3300/0 conn(0x7f7d90099f80 0x7f7d900cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.635 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.636+0000 7f7d94dfc700 1 -- 172.21.15.165:0/4028129738 >> v1:172.21.15.165:6789/0 conn(0x7f7d9015d500 legacy=0x7f7d9006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.635 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.636+0000 7f7d94dfc700 1 -- 172.21.15.165:0/4028129738 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9006eb60 con 0x7f7d900cb7b0 2024-03-18T18:49:28.635 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.636+0000 7f7d95dfe700 1 --2- 172.21.15.165:0/4028129738 >> v2:172.21.15.165:3300/0 conn(0x7f7d90099f80 0x7f7d900cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:49:28.635 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.636+0000 7f7d94dfc700 1 --2- 172.21.15.165:0/4028129738 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d900cb7b0 0x7f7d90099a40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7d800098f0 tx=0x7f7d80009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=617507b00ec4f516 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:28.635 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.636+0000 7f7d87fff700 1 -- 172.21.15.165:0/4028129738 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d80011830 con 0x7f7d900cb7b0 2024-03-18T18:49:28.635 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.636+0000 7f7d87fff700 1 -- 172.21.15.165:0/4028129738 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d80011990 con 0x7f7d900cb7b0 2024-03-18T18:49:28.635 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.636+0000 7f7d87fff700 1 -- 172.21.15.165:0/4028129738 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d80011d20 con 0x7f7d900cb7b0 2024-03-18T18:49:28.636 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.637+0000 7f7d97861700 1 -- 172.21.15.165:0/4028129738 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d900cb7b0 msgr2=0x7f7d90099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.636 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.637+0000 7f7d97861700 1 --2- 172.21.15.165:0/4028129738 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d900cb7b0 0x7f7d90099a40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7d800098f0 tx=0x7f7d80009c00 comp rx=0 tx=0).stop 2024-03-18T18:49:28.636 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.637+0000 7f7d97861700 1 -- 172.21.15.165:0/4028129738 shutdown_connections 2024-03-18T18:49:28.636 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.637+0000 7f7d97861700 1 --2- 172.21.15.165:0/4028129738 >> v2:172.21.15.165:3300/0 conn(0x7f7d90099f80 0x7f7d900cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.636 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.637+0000 7f7d97861700 1 --2- 172.21.15.165:0/4028129738 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d900cb7b0 0x7f7d90099a40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.636 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.637+0000 7f7d97861700 1 -- 172.21.15.165:0/4028129738 >> 172.21.15.165:0/4028129738 conn(0x7f7d900b2c50 msgr2=0x7f7d900b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:28.636 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.637+0000 7f7d97861700 1 -- 172.21.15.165:0/4028129738 shutdown_connections 2024-03-18T18:49:28.636 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.637+0000 7f7d97861700 1 -- 172.21.15.165:0/4028129738 wait complete. 2024-03-18T18:49:28.640 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.638+0000 7f7d97861700 1 Processor -- start 2024-03-18T18:49:28.640 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.638+0000 7f7d97861700 1 -- start start 2024-03-18T18:49:28.640 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.639+0000 7f7d97861700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d90099f80 0x7f7d901529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:28.640 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.639+0000 7f7d97861700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d900cb7b0 0x7f7d901f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:28.641 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.639+0000 7f7d97861700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d9015d500 0x7f7d901f3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:28.641 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.639+0000 7f7d97861700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d900c4c90 con 0x7f7d900cb7b0 2024-03-18T18:49:28.641 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.639+0000 7f7d97861700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9015db70 con 0x7f7d90099f80 2024-03-18T18:49:28.641 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.639+0000 7f7d97861700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d90152f00 con 0x7f7d9015d500 2024-03-18T18:49:28.641 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.639+0000 7f7d95dfe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d9015d500 0x7f7d901f3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:28.641 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.639+0000 7f7d955fd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d90099f80 0x7f7d901529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:28.642 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.639+0000 7f7d95dfe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d9015d500 0x7f7d901f3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:59684/0 (socket says 172.21.15.165:59684) 2024-03-18T18:49:28.642 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.639+0000 7f7d955fd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d90099f80 0x7f7d901529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:54260/0 (socket says 172.21.15.165:54260) 2024-03-18T18:49:28.642 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.639+0000 7f7d95dfe700 1 -- 172.21.15.165:0/989037423 learned_addr learned my addr 172.21.15.165:0/989037423 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-18T18:49:28.642 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.639+0000 7f7d94dfc700 1 --2- 172.21.15.165:0/989037423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d900cb7b0 0x7f7d901f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:28.642 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.640+0000 7f7d955fd700 1 -- 172.21.15.165:0/989037423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d9015d500 msgr2=0x7f7d901f3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.642 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.640+0000 7f7d955fd700 1 --2- 172.21.15.165:0/989037423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d9015d500 0x7f7d901f3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.643 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.640+0000 7f7d955fd700 1 -- 172.21.15.165:0/989037423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d900cb7b0 msgr2=0x7f7d901f2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.643 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.640+0000 7f7d955fd700 1 --2- 172.21.15.165:0/989037423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d900cb7b0 0x7f7d901f2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.643 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.640+0000 7f7d955fd700 1 -- 172.21.15.165:0/989037423 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d88009a70 con 0x7f7d90099f80 2024-03-18T18:49:28.643 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.640+0000 7f7d95dfe700 1 --2- 172.21.15.165:0/989037423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d9015d500 0x7f7d901f3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:49:28.643 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.640+0000 7f7d955fd700 1 --2- 172.21.15.165:0/989037423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d90099f80 0x7f7d901529c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c00c400 tx=0x7f7d8c00c710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:28.643 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.640+0000 7f7d85ffb700 1 -- 172.21.15.165:0/989037423 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c0052d0 con 0x7f7d90099f80 2024-03-18T18:49:28.643 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.640+0000 7f7d85ffb700 1 -- 172.21.15.165:0/989037423 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8c005430 con 0x7f7d90099f80 2024-03-18T18:49:28.644 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.640+0000 7f7d85ffb700 1 -- 172.21.15.165:0/989037423 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c0038a0 con 0x7f7d90099f80 2024-03-18T18:49:28.644 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.640+0000 7f7d97861700 1 -- 172.21.15.165:0/989037423 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d80009550 con 0x7f7d90099f80 2024-03-18T18:49:28.644 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.640+0000 7f7d85ffb700 1 -- 172.21.15.165:0/989037423 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7d8c003d00 con 0x7f7d90099f80 2024-03-18T18:49:28.644 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.640+0000 7f7d97861700 1 -- 172.21.15.165:0/989037423 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d900c4ea0 con 0x7f7d90099f80 2024-03-18T18:49:28.644 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.640+0000 7f7d85ffb700 1 -- 172.21.15.165:0/989037423 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f7d8c012c90 con 0x7f7d90099f80 2024-03-18T18:49:28.644 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.641+0000 7f7d97861700 1 -- 172.21.15.165:0/989037423 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d900676a0 con 0x7f7d90099f80 2024-03-18T18:49:28.644 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.643+0000 7f7d85ffb700 1 -- 172.21.15.165:0/989037423 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f7d8c017070 con 0x7f7d90099f80 2024-03-18T18:49:28.723 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.726+0000 7f7d97861700 1 -- 172.21.15.165:0/989037423 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd new", "uuid": "428759f1-b3c5-4d09-9ec1-d39c92985a96", "id": 3} v 0) v1 -- 0x7f7d900b1820 con 0x7f7d90099f80 2024-03-18T18:49:28.726 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.730+0000 7f7d85ffb700 1 -- 172.21.15.165:0/989037423 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "428759f1-b3c5-4d09-9ec1-d39c92985a96", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7d8c012e50 con 0x7f7d90099f80 2024-03-18T18:49:28.727 INFO:teuthology.orchestra.run.smithi165.stdout:3 2024-03-18T18:49:28.728 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.731+0000 7f7d97861700 1 -- 172.21.15.165:0/989037423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d90099f80 msgr2=0x7f7d901529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.728 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.731+0000 7f7d97861700 1 --2- 172.21.15.165:0/989037423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d90099f80 0x7f7d901529c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c00c400 tx=0x7f7d8c00c710 comp rx=0 tx=0).stop 2024-03-18T18:49:28.728 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.731+0000 7f7d97861700 1 -- 172.21.15.165:0/989037423 shutdown_connections 2024-03-18T18:49:28.728 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.731+0000 7f7d97861700 1 --2- 172.21.15.165:0/989037423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d9015d500 0x7f7d901f3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.728 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.731+0000 7f7d97861700 1 --2- 172.21.15.165:0/989037423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d90099f80 0x7f7d901529c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.729 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.731+0000 7f7d97861700 1 --2- 172.21.15.165:0/989037423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d900cb7b0 0x7f7d901f2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.729 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.731+0000 7f7d97861700 1 -- 172.21.15.165:0/989037423 >> 172.21.15.165:0/989037423 conn(0x7f7d900b2c50 msgr2=0x7f7d900cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:28.729 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.731+0000 7f7d97861700 1 -- 172.21.15.165:0/989037423 shutdown_connections 2024-03-18T18:49:28.729 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.731+0000 7f7d97861700 1 -- 172.21.15.165:0/989037423 wait complete. 2024-03-18T18:49:28.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 3543b3ec-2d8d-48b9-9a55-f7911b347e90 4 2024-03-18T18:49:28.863 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.866+0000 7f46475d4700 1 Processor -- start 2024-03-18T18:49:28.863 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.866+0000 7f46475d4700 1 -- start start 2024-03-18T18:49:28.875 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.867+0000 7f46475d4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f464006fa50 0x7f464006fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:28.876 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.867+0000 7f46475d4700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46400b3aa0 con 0x7f46400ade00 2024-03-18T18:49:28.876 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.867+0000 7f46475d4700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46400ae280 con 0x7f4640099a40 2024-03-18T18:49:28.877 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.867+0000 7f46475d4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f464006e9b0 con 0x7f464006fa50 2024-03-18T18:49:28.877 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.867+0000 7f4645370700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f4640099a40 0x7f4640099e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:44460/0 (socket says 172.21.15.165:44460) 2024-03-18T18:49:28.877 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.867+0000 7f4645370700 1 -- 172.21.15.165:0/3758502220 learned_addr learned my addr 172.21.15.165:0/3758502220 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-03-18T18:49:28.877 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.867+0000 7f4645b71700 1 --2- 172.21.15.165:0/3758502220 >> v2:172.21.15.1:3300/0 conn(0x7f464006fa50 0x7f464006fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:28.877 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.867+0000 7f4637fff700 1 -- 172.21.15.165:0/3758502220 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701377975 0 0) 0x7f46400ae280 con 0x7f4640099a40 2024-03-18T18:49:28.877 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.867+0000 7f4637fff700 1 -- 172.21.15.165:0/3758502220 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f462c0035b0 con 0x7f4640099a40 2024-03-18T18:49:28.878 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.867+0000 7f4637fff700 1 -- 172.21.15.165:0/3758502220 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586384410 0 0) 0x7f46400b3aa0 con 0x7f46400ade00 2024-03-18T18:49:28.878 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.867+0000 7f4637fff700 1 -- 172.21.15.165:0/3758502220 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46400ae280 con 0x7f46400ade00 2024-03-18T18:49:28.878 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.867+0000 7f4637fff700 1 -- 172.21.15.165:0/3758502220 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 376932476 0 0) 0x7f462c0035b0 con 0x7f4640099a40 2024-03-18T18:49:28.879 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.867+0000 7f4645b71700 1 -- 172.21.15.165:0/3758502220 >> v1:172.21.15.165:6789/0 conn(0x7f4640099a40 legacy=0x7f4640099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.879 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.867+0000 7f4645b71700 1 -- 172.21.15.165:0/3758502220 >> v1:172.21.15.1:6789/0 conn(0x7f46400ade00 legacy=0x7f46400cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.879 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.867+0000 7f4645b71700 1 -- 172.21.15.165:0/3758502220 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464006ec90 con 0x7f464006fa50 2024-03-18T18:49:28.879 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.868+0000 7f4645b71700 1 --2- 172.21.15.165:0/3758502220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f464006fa50 0x7f464006fee0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4638009d90 tx=0x7f463800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f274a47dc63f44 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:28.879 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.868+0000 7f4637fff700 1 -- 172.21.15.165:0/3758502220 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463800e070 con 0x7f464006fa50 2024-03-18T18:49:28.879 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.868+0000 7f4637fff700 1 -- 172.21.15.165:0/3758502220 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4638003a20 con 0x7f464006fa50 2024-03-18T18:49:28.879 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.868+0000 7f4637fff700 1 -- 172.21.15.165:0/3758502220 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c001670 con 0x7f464006fa50 2024-03-18T18:49:28.880 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.868+0000 7f46475d4700 1 -- 172.21.15.165:0/3758502220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f464006fa50 msgr2=0x7f464006fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.880 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.868+0000 7f46475d4700 1 --2- 172.21.15.165:0/3758502220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f464006fa50 0x7f464006fee0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4638009d90 tx=0x7f463800b330 comp rx=0 tx=0).stop 2024-03-18T18:49:28.880 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.868+0000 7f46475d4700 1 -- 172.21.15.165:0/3758502220 shutdown_connections 2024-03-18T18:49:28.881 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.868+0000 7f46475d4700 1 --2- 172.21.15.165:0/3758502220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f464006fa50 0x7f464006fee0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.881 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.868+0000 7f46475d4700 1 -- 172.21.15.165:0/3758502220 >> 172.21.15.165:0/3758502220 conn(0x7f46400b2c50 msgr2=0x7f46400b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:28.881 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.869+0000 7f46475d4700 1 -- 172.21.15.165:0/3758502220 shutdown_connections 2024-03-18T18:49:28.881 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.869+0000 7f46475d4700 1 -- 172.21.15.165:0/3758502220 wait complete. 2024-03-18T18:49:28.881 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.870+0000 7f46475d4700 1 Processor -- start 2024-03-18T18:49:28.882 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.870+0000 7f46475d4700 1 -- start start 2024-03-18T18:49:28.882 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.870+0000 7f46475d4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f464006fa50 0x7f46400c55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:28.882 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.870+0000 7f46475d4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4640099a40 0x7f46400c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:28.882 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.870+0000 7f46475d4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46400ade00 0x7f4640155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:28.882 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.870+0000 7f46475d4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46400c6930 con 0x7f46400ade00 2024-03-18T18:49:28.882 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.870+0000 7f46475d4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46400c7a50 con 0x7f464006fa50 2024-03-18T18:49:28.883 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.870+0000 7f46475d4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46400c5cb0 con 0x7f4640099a40 2024-03-18T18:49:28.883 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.871+0000 7f4645370700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f464006fa50 0x7f46400c55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:28.883 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.871+0000 7f4644b6f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4640099a40 0x7f46400c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:28.883 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.871+0000 7f4645370700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f464006fa50 0x7f46400c55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:54266/0 (socket says 172.21.15.165:54266) 2024-03-18T18:49:28.883 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.871+0000 7f4645370700 1 -- 172.21.15.165:0/3536973638 learned_addr learned my addr 172.21.15.165:0/3536973638 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-18T18:49:28.883 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.871+0000 7f4644b6f700 1 --2- 172.21.15.165:0/3536973638 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4640099a40 0x7f46400c7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:59694/0 (socket says 172.21.15.165:59694) 2024-03-18T18:49:28.884 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.871+0000 7f4645b71700 1 --2- 172.21.15.165:0/3536973638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46400ade00 0x7f4640155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:28.884 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.871+0000 7f4644b6f700 1 -- 172.21.15.165:0/3536973638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f464006fa50 msgr2=0x7f46400c55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.884 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.871+0000 7f4644b6f700 1 --2- 172.21.15.165:0/3536973638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f464006fa50 0x7f46400c55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.884 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.871+0000 7f4644b6f700 1 -- 172.21.15.165:0/3536973638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46400ade00 msgr2=0x7f4640155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.884 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.871+0000 7f4644b6f700 1 --2- 172.21.15.165:0/3536973638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46400ade00 0x7f4640155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.884 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.871+0000 7f4644b6f700 1 -- 172.21.15.165:0/3536973638 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4638009a70 con 0x7f4640099a40 2024-03-18T18:49:28.885 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.871+0000 7f4645370700 1 --2- 172.21.15.165:0/3536973638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f464006fa50 0x7f46400c55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:49:28.885 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.871+0000 7f4645b71700 1 --2- 172.21.15.165:0/3536973638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46400ade00 0x7f4640155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:49:28.885 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.871+0000 7f4644b6f700 1 --2- 172.21.15.165:0/3536973638 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4640099a40 0x7f46400c7510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f463000c480 tx=0x7f463000c790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:28.885 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.871+0000 7f4635ffb700 1 -- 172.21.15.165:0/3536973638 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46300052d0 con 0x7f4640099a40 2024-03-18T18:49:28.885 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.871+0000 7f4635ffb700 1 -- 172.21.15.165:0/3536973638 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4630005430 con 0x7f4640099a40 2024-03-18T18:49:28.886 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.871+0000 7f46475d4700 1 -- 172.21.15.165:0/3536973638 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46400c5f90 con 0x7f4640099a40 2024-03-18T18:49:28.886 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.871+0000 7f4635ffb700 1 -- 172.21.15.165:0/3536973638 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46300038c0 con 0x7f4640099a40 2024-03-18T18:49:28.886 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.871+0000 7f4635ffb700 1 -- 172.21.15.165:0/3536973638 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4630003d20 con 0x7f4640099a40 2024-03-18T18:49:28.886 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.872+0000 7f46475d4700 1 -- 172.21.15.165:0/3536973638 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46400c6550 con 0x7f4640099a40 2024-03-18T18:49:28.886 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.872+0000 7f4635ffb700 1 -- 172.21.15.165:0/3536973638 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f4630012420 con 0x7f4640099a40 2024-03-18T18:49:28.886 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.872+0000 7f46475d4700 1 -- 172.21.15.165:0/3536973638 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46400c6a70 con 0x7f4640099a40 2024-03-18T18:49:28.887 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.875+0000 7f4635ffb700 1 -- 172.21.15.165:0/3536973638 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f46400c6a70 con 0x7f4640099a40 2024-03-18T18:49:28.970 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.973+0000 7f46475d4700 1 -- 172.21.15.165:0/3536973638 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3543b3ec-2d8d-48b9-9a55-f7911b347e90", "id": 4} v 0) v1 -- 0x7f46400676a0 con 0x7f4640099a40 2024-03-18T18:49:28.973 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.977+0000 7f4635ffb700 1 -- 172.21.15.165:0/3536973638 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3543b3ec-2d8d-48b9-9a55-f7911b347e90", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4630017070 con 0x7f4640099a40 2024-03-18T18:49:28.974 INFO:teuthology.orchestra.run.smithi165.stdout:4 2024-03-18T18:49:28.975 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.977+0000 7f46475d4700 1 -- 172.21.15.165:0/3536973638 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4640099a40 msgr2=0x7f46400c7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:28.975 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.977+0000 7f46475d4700 1 --2- 172.21.15.165:0/3536973638 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4640099a40 0x7f46400c7510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f463000c480 tx=0x7f463000c790 comp rx=0 tx=0).stop 2024-03-18T18:49:28.975 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.978+0000 7f46475d4700 1 -- 172.21.15.165:0/3536973638 shutdown_connections 2024-03-18T18:49:28.975 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.978+0000 7f46475d4700 1 --2- 172.21.15.165:0/3536973638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46400ade00 0x7f4640155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.975 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.978+0000 7f46475d4700 1 --2- 172.21.15.165:0/3536973638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f464006fa50 0x7f46400c55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.975 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.978+0000 7f46475d4700 1 --2- 172.21.15.165:0/3536973638 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4640099a40 0x7f46400c7510 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:28.975 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.978+0000 7f46475d4700 1 -- 172.21.15.165:0/3536973638 >> 172.21.15.165:0/3536973638 conn(0x7f46400b2c50 msgr2=0x7f464006fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:28.975 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.978+0000 7f46475d4700 1 -- 172.21.15.165:0/3536973638 shutdown_connections 2024-03-18T18:49:28.976 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:28.978+0000 7f46475d4700 1 -- 172.21.15.165:0/3536973638 wait complete. 2024-03-18T18:49:28.986 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 0a8b4e7c-f079-4266-9142-75791dbe3b04 5 2024-03-18T18:49:28.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:28.997+0000 7fbea244d240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T18:49:29.107 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.110+0000 7f5ae730e700 1 Processor -- start 2024-03-18T18:49:29.108 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.111+0000 7f5ae730e700 1 -- start start 2024-03-18T18:49:29.110 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.111+0000 7f5ae730e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5ae00aecb0 0x7f5ae0070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.110 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.111+0000 7f5ae730e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5ae0071380 0x7f5ae006d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.110 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.111+0000 7f5ae730e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae015b750 con 0x7f5ae015b370 2024-03-18T18:49:29.110 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.111+0000 7f5ae730e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ae00cf510 con 0x7f5ae00aecb0 2024-03-18T18:49:29.110 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.111+0000 7f5ae730e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ae0071830 con 0x7f5ae0071380 2024-03-18T18:49:29.110 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.112+0000 7f5ae58ab700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5ae0071380 0x7f5ae006d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.110 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.112+0000 7f5ae58ab700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5ae0071380 0x7f5ae006d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:54270/0 (socket says 172.21.15.165:54270) 2024-03-18T18:49:29.110 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.112+0000 7f5ae50aa700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f5ae015b370 0x7f5ae00cec70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:44472/0 (socket says 172.21.15.165:44472) 2024-03-18T18:49:29.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.112+0000 7f5ae58ab700 1 -- 172.21.15.165:0/3681535402 learned_addr learned my addr 172.21.15.165:0/3681535402 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-18T18:49:29.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.112+0000 7f5ae48a9700 1 --2- 172.21.15.165:0/3681535402 >> v2:172.21.15.1:3300/0 conn(0x7f5ae00aecb0 0x7f5ae0070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.112+0000 7f5ae58ab700 1 -- 172.21.15.165:0/3681535402 >> v2:172.21.15.1:3300/0 conn(0x7f5ae00aecb0 msgr2=0x7f5ae0070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:29.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.112+0000 7f5ae58ab700 1 --2- 172.21.15.165:0/3681535402 >> v2:172.21.15.1:3300/0 conn(0x7f5ae00aecb0 0x7f5ae0070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.112+0000 7f5ae58ab700 1 -- 172.21.15.165:0/3681535402 >> v1:172.21.15.165:6789/0 conn(0x7f5ae015b370 legacy=0x7f5ae00cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.112+0000 7f5ae58ab700 1 -- 172.21.15.165:0/3681535402 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae00ccc20 con 0x7f5ae0071380 2024-03-18T18:49:29.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.112+0000 7f5ae58ab700 1 --2- 172.21.15.165:0/3681535402 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ae0071380 0x7f5ae006d080 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5ae00b26e0 tx=0x7f5ad800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5cc20c1e1c8bda server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:29.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.112+0000 7f5ad7fff700 1 -- 172.21.15.165:0/3681535402 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad800e070 con 0x7f5ae0071380 2024-03-18T18:49:29.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.112+0000 7f5ad7fff700 1 -- 172.21.15.165:0/3681535402 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad800b910 con 0x7f5ae0071380 2024-03-18T18:49:29.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.112+0000 7f5ad7fff700 1 -- 172.21.15.165:0/3681535402 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad800bcc0 con 0x7f5ae0071380 2024-03-18T18:49:29.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.113+0000 7f5ae730e700 1 -- 172.21.15.165:0/3681535402 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ae0071380 msgr2=0x7f5ae006d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.113+0000 7f5ae730e700 1 --2- 172.21.15.165:0/3681535402 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ae0071380 0x7f5ae006d080 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5ae00b26e0 tx=0x7f5ad800b330 comp rx=0 tx=0).stop 2024-03-18T18:49:29.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.113+0000 7f5ae730e700 1 -- 172.21.15.165:0/3681535402 shutdown_connections 2024-03-18T18:49:29.111 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.113+0000 7f5ae730e700 1 --2- 172.21.15.165:0/3681535402 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ae0071380 0x7f5ae006d080 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.112 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.113+0000 7f5ae730e700 1 --2- 172.21.15.165:0/3681535402 >> v2:172.21.15.1:3300/0 conn(0x7f5ae00aecb0 0x7f5ae0070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.112 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.113+0000 7f5ae730e700 1 -- 172.21.15.165:0/3681535402 >> 172.21.15.165:0/3681535402 conn(0x7f5ae00b1350 msgr2=0x7f5ae00b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:29.112 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.113+0000 7f5ae730e700 1 -- 172.21.15.165:0/3681535402 shutdown_connections 2024-03-18T18:49:29.112 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.113+0000 7f5ae730e700 1 -- 172.21.15.165:0/3681535402 wait complete. 2024-03-18T18:49:29.115 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.115+0000 7f5ae730e700 1 Processor -- start 2024-03-18T18:49:29.115 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.115+0000 7f5ae730e700 1 -- start start 2024-03-18T18:49:29.115 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.115+0000 7f5ae730e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ae0071380 0x7f5ae0158e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.115 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.115+0000 7f5ae730e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ae00aecb0 0x7f5ae00a9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.116 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.115+0000 7f5ae730e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5ae015b370 0x7f5ae0149190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.116 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.115+0000 7f5ae730e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae01594b0 con 0x7f5ae00aecb0 2024-03-18T18:49:29.116 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.115+0000 7f5ae730e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae0159790 con 0x7f5ae0071380 2024-03-18T18:49:29.116 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ae730e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae0159a70 con 0x7f5ae015b370 2024-03-18T18:49:29.116 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ae50aa700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ae0071380 0x7f5ae0158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.116 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ae58ab700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5ae015b370 0x7f5ae0149190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.116 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ae58ab700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5ae015b370 0x7f5ae0149190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:59700/0 (socket says 172.21.15.165:59700) 2024-03-18T18:49:29.116 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ae50aa700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ae0071380 0x7f5ae0158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:54278/0 (socket says 172.21.15.165:54278) 2024-03-18T18:49:29.116 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ae58ab700 1 -- 172.21.15.165:0/716862706 learned_addr learned my addr 172.21.15.165:0/716862706 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-18T18:49:29.116 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ae48a9700 1 --2- 172.21.15.165:0/716862706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ae00aecb0 0x7f5ae00a9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.116 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ae50aa700 1 -- 172.21.15.165:0/716862706 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5ae015b370 msgr2=0x7f5ae0149190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.116 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ae50aa700 1 --2- 172.21.15.165:0/716862706 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5ae015b370 0x7f5ae0149190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.116 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ae50aa700 1 -- 172.21.15.165:0/716862706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ae00aecb0 msgr2=0x7f5ae00a9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.117 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ae50aa700 1 --2- 172.21.15.165:0/716862706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ae00aecb0 0x7f5ae00a9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.117 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ae50aa700 1 -- 172.21.15.165:0/716862706 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad8009b30 con 0x7f5ae0071380 2024-03-18T18:49:29.117 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ae58ab700 1 --2- 172.21.15.165:0/716862706 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5ae015b370 0x7f5ae0149190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:49:29.117 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ae48a9700 1 --2- 172.21.15.165:0/716862706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ae00aecb0 0x7f5ae00a9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:49:29.117 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ae50aa700 1 --2- 172.21.15.165:0/716862706 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ae0071380 0x7f5ae0158e80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5adc00c3c0 tx=0x7f5adc00c6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:29.117 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ad5ffb700 1 -- 172.21.15.165:0/716862706 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc004a70 con 0x7f5ae0071380 2024-03-18T18:49:29.117 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ad5ffb700 1 -- 172.21.15.165:0/716862706 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5adc004bd0 con 0x7f5ae0071380 2024-03-18T18:49:29.117 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ae730e700 1 -- 172.21.15.165:0/716862706 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae0159d50 con 0x7f5ae0071380 2024-03-18T18:49:29.117 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ad5ffb700 1 -- 172.21.15.165:0/716862706 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc0070d0 con 0x7f5ae0071380 2024-03-18T18:49:29.117 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.116+0000 7f5ae730e700 1 -- 172.21.15.165:0/716862706 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae00a95b0 con 0x7f5ae0071380 2024-03-18T18:49:29.118 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.117+0000 7f5ad5ffb700 1 -- 172.21.15.165:0/716862706 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5adc007530 con 0x7f5ae0071380 2024-03-18T18:49:29.118 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.117+0000 7f5ad5ffb700 1 -- 172.21.15.165:0/716862706 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f5adc013d10 con 0x7f5ae0071380 2024-03-18T18:49:29.118 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.117+0000 7f5ae730e700 1 -- 172.21.15.165:0/716862706 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae00676a0 con 0x7f5ae0071380 2024-03-18T18:49:29.118 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.119+0000 7f5ad5ffb700 1 -- 172.21.15.165:0/716862706 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f5adc0134b0 con 0x7f5ae0071380 2024-03-18T18:49:29.120 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:29.124+0000 7ff33f4e5240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T18:49:29.200 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.203+0000 7f5ae730e700 1 -- 172.21.15.165:0/716862706 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0a8b4e7c-f079-4266-9142-75791dbe3b04", "id": 5} v 0) v1 -- 0x7f5ae0073db0 con 0x7f5ae0071380 2024-03-18T18:49:29.203 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.206+0000 7f5ad5ffb700 1 -- 172.21.15.165:0/716862706 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0a8b4e7c-f079-4266-9142-75791dbe3b04", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5adc013720 con 0x7f5ae0071380 2024-03-18T18:49:29.203 INFO:teuthology.orchestra.run.smithi165.stdout:5 2024-03-18T18:49:29.205 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.208+0000 7f5ae730e700 1 -- 172.21.15.165:0/716862706 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ae0071380 msgr2=0x7f5ae0158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.205 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.208+0000 7f5ae730e700 1 --2- 172.21.15.165:0/716862706 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ae0071380 0x7f5ae0158e80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5adc00c3c0 tx=0x7f5adc00c6d0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.205 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.208+0000 7f5ae730e700 1 -- 172.21.15.165:0/716862706 shutdown_connections 2024-03-18T18:49:29.206 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.208+0000 7f5ae730e700 1 --2- 172.21.15.165:0/716862706 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5ae015b370 0x7f5ae0149190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.206 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.208+0000 7f5ae730e700 1 --2- 172.21.15.165:0/716862706 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ae0071380 0x7f5ae0158e80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.206 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.208+0000 7f5ae730e700 1 --2- 172.21.15.165:0/716862706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ae00aecb0 0x7f5ae00a9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.206 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.208+0000 7f5ae730e700 1 -- 172.21.15.165:0/716862706 >> 172.21.15.165:0/716862706 conn(0x7f5ae00b1350 msgr2=0x7f5ae00b0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:29.206 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.208+0000 7f5ae730e700 1 -- 172.21.15.165:0/716862706 shutdown_connections 2024-03-18T18:49:29.206 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.208+0000 7f5ae730e700 1 -- 172.21.15.165:0/716862706 wait complete. 2024-03-18T18:49:29.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new e7af56bf-343c-469e-9f54-fb2f0cb3d51d 6 2024-03-18T18:49:29.348 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.347+0000 7fb5edacd700 1 Processor -- start 2024-03-18T18:49:29.348 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.347+0000 7fb5edacd700 1 -- start start 2024-03-18T18:49:29.349 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.348+0000 7fb5edacd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb5e815d780 0x7fb5e80cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.349 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.348+0000 7fb5edacd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5e809dd40 0x7fb5e80cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.349 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.348+0000 7fb5edacd700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e80ce090 con 0x7fb5e80ce5a0 2024-03-18T18:49:29.349 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.348+0000 7fb5edacd700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e806db30 con 0x7fb5e815d780 2024-03-18T18:49:29.349 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.348+0000 7fb5edacd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e806d520 con 0x7fb5e809dd40 2024-03-18T18:49:29.349 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.348+0000 7fb5e6ffd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5e809dd40 0x7fb5e80cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.349 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.348+0000 7fb5e77fe700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb5e815d780 0x7fb5e80cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.349 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.348+0000 7fb5e77fe700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb5e815d780 0x7fb5e80cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:54288/0 (socket says 172.21.15.165:54288) 2024-03-18T18:49:29.350 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.348+0000 7fb5e6ffd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5e809dd40 0x7fb5e80cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:59708/0 (socket says 172.21.15.165:59708) 2024-03-18T18:49:29.350 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.348+0000 7fb5e77fe700 1 -- 172.21.15.165:0/812408067 learned_addr learned my addr 172.21.15.165:0/812408067 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-18T18:49:29.350 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.348+0000 7fb5e6ffd700 1 -- 172.21.15.165:0/812408067 >> v2:172.21.15.165:3300/0 conn(0x7fb5e815d780 msgr2=0x7fb5e80cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:29.350 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.348+0000 7fb5e6ffd700 1 --2- 172.21.15.165:0/812408067 >> v2:172.21.15.165:3300/0 conn(0x7fb5e815d780 0x7fb5e80cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.350 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.348+0000 7fb5e6ffd700 1 -- 172.21.15.165:0/812408067 >> v1:172.21.15.1:6789/0 conn(0x7fb5e80ce5a0 legacy=0x7fb5e80cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.350 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.348+0000 7fb5e6ffd700 1 -- 172.21.15.165:0/812408067 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e806f460 con 0x7fb5e809dd40 2024-03-18T18:49:29.350 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.348+0000 7fb5e77fe700 1 --2- 172.21.15.165:0/812408067 >> v2:172.21.15.165:3300/0 conn(0x7fb5e815d780 0x7fb5e80cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:49:29.350 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.348+0000 7fb5e6ffd700 1 --2- 172.21.15.165:0/812408067 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5e809dd40 0x7fb5e80cad70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc009840 tx=0x7fb5dc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc8d10aff0545598 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:29.351 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.349+0000 7fb5e67fc700 1 -- 172.21.15.165:0/812408067 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc011830 con 0x7fb5e809dd40 2024-03-18T18:49:29.351 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.349+0000 7fb5e67fc700 1 -- 172.21.15.165:0/812408067 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5dc011990 con 0x7fb5e809dd40 2024-03-18T18:49:29.351 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.349+0000 7fb5e67fc700 1 -- 172.21.15.165:0/812408067 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc011d20 con 0x7fb5e809dd40 2024-03-18T18:49:29.351 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.349+0000 7fb5edacd700 1 -- 172.21.15.165:0/812408067 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5e809dd40 msgr2=0x7fb5e80cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.351 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.349+0000 7fb5edacd700 1 --2- 172.21.15.165:0/812408067 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5e809dd40 0x7fb5e80cad70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc009840 tx=0x7fb5dc009b50 comp rx=0 tx=0).stop 2024-03-18T18:49:29.351 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.349+0000 7fb5edacd700 1 -- 172.21.15.165:0/812408067 shutdown_connections 2024-03-18T18:49:29.351 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.349+0000 7fb5edacd700 1 --2- 172.21.15.165:0/812408067 >> v2:172.21.15.165:3300/0 conn(0x7fb5e815d780 0x7fb5e80cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.351 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.349+0000 7fb5edacd700 1 --2- 172.21.15.165:0/812408067 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5e809dd40 0x7fb5e80cad70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.352 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.349+0000 7fb5edacd700 1 -- 172.21.15.165:0/812408067 >> 172.21.15.165:0/812408067 conn(0x7fb5e80cca00 msgr2=0x7fb5e80cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:29.352 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.349+0000 7fb5edacd700 1 -- 172.21.15.165:0/812408067 shutdown_connections 2024-03-18T18:49:29.352 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.349+0000 7fb5edacd700 1 -- 172.21.15.165:0/812408067 wait complete. 2024-03-18T18:49:29.352 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:29.350+0000 7ff33f4e5240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T18:49:29.357 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.350+0000 7fb5edacd700 1 Processor -- start 2024-03-18T18:49:29.357 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.351+0000 7fb5edacd700 1 -- start start 2024-03-18T18:49:29.357 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.351+0000 7fb5edacd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5e809dd40 0x7fb5e80c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.358 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.351+0000 7fb5edacd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5e80ce5a0 0x7fb5e80ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.358 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.351+0000 7fb5edacd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5e815d780 0x7fb5e81e4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.358 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.351+0000 7fb5edacd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e80c8a60 con 0x7fb5e80ce5a0 2024-03-18T18:49:29.358 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.351+0000 7fb5edacd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e80c8d40 con 0x7fb5e815d780 2024-03-18T18:49:29.358 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.351+0000 7fb5edacd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e80c9020 con 0x7fb5e809dd40 2024-03-18T18:49:29.358 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.352+0000 7fb5e77fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5e809dd40 0x7fb5e80c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.358 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.352+0000 7fb5e7fff700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5e815d780 0x7fb5e81e4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.358 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.352+0000 7fb5e7fff700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5e815d780 0x7fb5e81e4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:54294/0 (socket says 172.21.15.165:54294) 2024-03-18T18:49:29.358 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.352+0000 7fb5e7fff700 1 -- 172.21.15.165:0/1424481570 learned_addr learned my addr 172.21.15.165:0/1424481570 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-18T18:49:29.358 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.352+0000 7fb5e77fe700 1 --2- 172.21.15.165:0/1424481570 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5e809dd40 0x7fb5e80c9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:59714/0 (socket says 172.21.15.165:59714) 2024-03-18T18:49:29.358 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.352+0000 7fb5e6ffd700 1 --2- 172.21.15.165:0/1424481570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5e80ce5a0 0x7fb5e80ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.358 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.352+0000 7fb5e77fe700 1 -- 172.21.15.165:0/1424481570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5e815d780 msgr2=0x7fb5e81e4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.359 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.352+0000 7fb5e77fe700 1 --2- 172.21.15.165:0/1424481570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5e815d780 0x7fb5e81e4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.359 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.352+0000 7fb5e77fe700 1 -- 172.21.15.165:0/1424481570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5e80ce5a0 msgr2=0x7fb5e80ae1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.359 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.352+0000 7fb5e77fe700 1 --2- 172.21.15.165:0/1424481570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5e80ce5a0 0x7fb5e80ae1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.359 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.352+0000 7fb5e77fe700 1 -- 172.21.15.165:0/1424481570 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d8009550 con 0x7fb5e809dd40 2024-03-18T18:49:29.359 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.352+0000 7fb5e77fe700 1 --2- 172.21.15.165:0/1424481570 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5e809dd40 0x7fb5e80c9b00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb5d800e980 tx=0x7fb5d800ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:29.359 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.352+0000 7fb5cffff700 1 -- 172.21.15.165:0/1424481570 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8004030 con 0x7fb5e809dd40 2024-03-18T18:49:29.359 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.352+0000 7fb5cffff700 1 -- 172.21.15.165:0/1424481570 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d8004190 con 0x7fb5e809dd40 2024-03-18T18:49:29.359 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.352+0000 7fb5cffff700 1 -- 172.21.15.165:0/1424481570 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8004520 con 0x7fb5e809dd40 2024-03-18T18:49:29.359 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.352+0000 7fb5edacd700 1 -- 172.21.15.165:0/1424481570 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5dc0094a0 con 0x7fb5e809dd40 2024-03-18T18:49:29.359 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.352+0000 7fb5edacd700 1 -- 172.21.15.165:0/1424481570 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e80c95e0 con 0x7fb5e809dd40 2024-03-18T18:49:29.360 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.353+0000 7fb5cffff700 1 -- 172.21.15.165:0/1424481570 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb5d8005d20 con 0x7fb5e809dd40 2024-03-18T18:49:29.360 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.353+0000 7fb5cffff700 1 -- 172.21.15.165:0/1424481570 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fb5d8013d10 con 0x7fb5e809dd40 2024-03-18T18:49:29.360 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.353+0000 7fb5edacd700 1 -- 172.21.15.165:0/1424481570 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e806d7a0 con 0x7fb5e809dd40 2024-03-18T18:49:29.360 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.355+0000 7fb5cffff700 1 -- 172.21.15.165:0/1424481570 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fb5e806d7a0 con 0x7fb5e809dd40 2024-03-18T18:49:29.435 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.438+0000 7fb5edacd700 1 -- 172.21.15.165:0/1424481570 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e7af56bf-343c-469e-9f54-fb2f0cb3d51d", "id": 6} v 0) v1 -- 0x7fb5e80676a0 con 0x7fb5e809dd40 2024-03-18T18:49:29.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.447+0000 7fb5cffff700 1 -- 172.21.15.165:0/1424481570 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e7af56bf-343c-469e-9f54-fb2f0cb3d51d", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb5d8020060 con 0x7fb5e809dd40 2024-03-18T18:49:29.443 INFO:teuthology.orchestra.run.smithi165.stdout:6 2024-03-18T18:49:29.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.448+0000 7fb5edacd700 1 -- 172.21.15.165:0/1424481570 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5e809dd40 msgr2=0x7fb5e80c9b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.448+0000 7fb5edacd700 1 --2- 172.21.15.165:0/1424481570 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5e809dd40 0x7fb5e80c9b00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb5d800e980 tx=0x7fb5d800ec90 comp rx=0 tx=0).stop 2024-03-18T18:49:29.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.448+0000 7fb5edacd700 1 -- 172.21.15.165:0/1424481570 shutdown_connections 2024-03-18T18:49:29.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.448+0000 7fb5edacd700 1 --2- 172.21.15.165:0/1424481570 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5e809dd40 0x7fb5e80c9b00 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.448+0000 7fb5edacd700 1 --2- 172.21.15.165:0/1424481570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5e815d780 0x7fb5e81e4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.448+0000 7fb5edacd700 1 --2- 172.21.15.165:0/1424481570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5e80ce5a0 0x7fb5e80ae1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.448+0000 7fb5edacd700 1 -- 172.21.15.165:0/1424481570 >> 172.21.15.165:0/1424481570 conn(0x7fb5e80cca00 msgr2=0x7fb5e80aef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:29.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.448+0000 7fb5edacd700 1 -- 172.21.15.165:0/1424481570 shutdown_connections 2024-03-18T18:49:29.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.448+0000 7fb5edacd700 1 -- 172.21.15.165:0/1424481570 wait complete. 2024-03-18T18:49:29.456 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 2ca85776-3586-4679-b571-dc8d294d9dbe 7 2024-03-18T18:49:29.577 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.580+0000 7f19c7b8f700 1 Processor -- start 2024-03-18T18:49:29.577 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.580+0000 7f19c7b8f700 1 -- start start 2024-03-18T18:49:29.579 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.581+0000 7f19c7b8f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f19c006d560 0x7f19c00aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.579 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.581+0000 7f19c7b8f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19c00af510 0x7f19c015b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.579 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.581+0000 7f19c7b8f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c015ef00 con 0x7f19c006a750 2024-03-18T18:49:29.579 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.581+0000 7f19c7b8f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19c015f1e0 con 0x7f19c006d560 2024-03-18T18:49:29.580 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.581+0000 7f19c7b8f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c015f4c0 con 0x7f19c00af510 2024-03-18T18:49:29.580 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.581+0000 7f19c512a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f19c006d560 0x7f19c00aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.580 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.581+0000 7f19c612c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19c00af510 0x7f19c015b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.580 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.581+0000 7f19c512a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f19c006d560 0x7f19c00aefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:54310/0 (socket says 172.21.15.165:54310) 2024-03-18T18:49:29.580 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.581+0000 7f19c612c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19c00af510 0x7f19c015b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:59718/0 (socket says 172.21.15.165:59718) 2024-03-18T18:49:29.580 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.581+0000 7f19c512a700 1 -- 172.21.15.165:0/2883536172 learned_addr learned my addr 172.21.15.165:0/2883536172 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-18T18:49:29.580 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.581+0000 7f19c612c700 1 -- 172.21.15.165:0/2883536172 >> v2:172.21.15.165:3300/0 conn(0x7f19c006d560 msgr2=0x7f19c00aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:29.580 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.581+0000 7f19c612c700 1 --2- 172.21.15.165:0/2883536172 >> v2:172.21.15.165:3300/0 conn(0x7f19c006d560 0x7f19c00aefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.580 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.581+0000 7f19c612c700 1 -- 172.21.15.165:0/2883536172 >> v1:172.21.15.1:6789/0 conn(0x7f19c006a750 legacy=0x7f19c006ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.580 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.581+0000 7f19c612c700 1 -- 172.21.15.165:0/2883536172 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c015f7a0 con 0x7f19c00af510 2024-03-18T18:49:29.580 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.581+0000 7f19c512a700 1 --2- 172.21.15.165:0/2883536172 >> v2:172.21.15.165:3300/0 conn(0x7f19c006d560 0x7f19c00aefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:49:29.580 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.581+0000 7f19c612c700 1 --2- 172.21.15.165:0/2883536172 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19c00af510 0x7f19c015b900 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f19b8009d90 tx=0x7f19b800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85712ab4fb934bbe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:29.580 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.581+0000 7f19c4929700 1 -- 172.21.15.165:0/2883536172 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b800e070 con 0x7f19c00af510 2024-03-18T18:49:29.581 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.581+0000 7f19c4929700 1 -- 172.21.15.165:0/2883536172 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b800bb20 con 0x7f19c00af510 2024-03-18T18:49:29.581 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.581+0000 7f19c4929700 1 -- 172.21.15.165:0/2883536172 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b8012430 con 0x7f19c00af510 2024-03-18T18:49:29.581 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.582+0000 7f19c7b8f700 1 -- 172.21.15.165:0/2883536172 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19c00af510 msgr2=0x7f19c015b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.581 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.582+0000 7f19c7b8f700 1 --2- 172.21.15.165:0/2883536172 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19c00af510 0x7f19c015b900 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f19b8009d90 tx=0x7f19b800b330 comp rx=0 tx=0).stop 2024-03-18T18:49:29.581 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.582+0000 7f19c7b8f700 1 -- 172.21.15.165:0/2883536172 shutdown_connections 2024-03-18T18:49:29.581 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.582+0000 7f19c7b8f700 1 --2- 172.21.15.165:0/2883536172 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19c00af510 0x7f19c015b900 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.581 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.582+0000 7f19c7b8f700 1 --2- 172.21.15.165:0/2883536172 >> v2:172.21.15.165:3300/0 conn(0x7f19c006d560 0x7f19c00aefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.581 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.582+0000 7f19c7b8f700 1 -- 172.21.15.165:0/2883536172 >> 172.21.15.165:0/2883536172 conn(0x7f19c00cff30 msgr2=0x7f19c00d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:29.581 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.582+0000 7f19c7b8f700 1 -- 172.21.15.165:0/2883536172 shutdown_connections 2024-03-18T18:49:29.581 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.582+0000 7f19c7b8f700 1 -- 172.21.15.165:0/2883536172 wait complete. 2024-03-18T18:49:29.584 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.583+0000 7f19c7b8f700 1 Processor -- start 2024-03-18T18:49:29.584 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.583+0000 7f19c7b8f700 1 -- start start 2024-03-18T18:49:29.584 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.584+0000 7f19c7b8f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19c006a750 0x7f19c00b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.584 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.584+0000 7f19c7b8f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19c006d560 0x7f19c00b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.584 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.584+0000 7f19c7b8f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19c00af510 0x7f19c00b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.584 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.584+0000 7f19c7b8f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c00b5190 con 0x7f19c00af510 2024-03-18T18:49:29.584 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.584+0000 7f19c7b8f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c00b1760 con 0x7f19c006a750 2024-03-18T18:49:29.584 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.584+0000 7f19c7b8f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c00b0ec0 con 0x7f19c006d560 2024-03-18T18:49:29.584 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.584+0000 7f19c512a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19c006d560 0x7f19c00b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.584 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.584+0000 7f19c512a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19c006d560 0x7f19c00b21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.165:59720/0 (socket says 172.21.15.165:59720) 2024-03-18T18:49:29.585 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.584+0000 7f19c512a700 1 -- 172.21.15.165:0/1383271410 learned_addr learned my addr 172.21.15.165:0/1383271410 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-18T18:49:29.585 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.584+0000 7f19c592b700 1 --2- 172.21.15.165:0/1383271410 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19c006a750 0x7f19c00b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.585 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.584+0000 7f19c612c700 1 --2- 172.21.15.165:0/1383271410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19c00af510 0x7f19c00b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.585 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.584+0000 7f19c512a700 1 -- 172.21.15.165:0/1383271410 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19c006a750 msgr2=0x7f19c00b4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.585 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.584+0000 7f19c512a700 1 --2- 172.21.15.165:0/1383271410 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19c006a750 0x7f19c00b4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.585 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.584+0000 7f19c512a700 1 -- 172.21.15.165:0/1383271410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19c00af510 msgr2=0x7f19c00b39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.585 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.584+0000 7f19c512a700 1 --2- 172.21.15.165:0/1383271410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19c00af510 0x7f19c00b39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.585 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.584+0000 7f19c512a700 1 -- 172.21.15.165:0/1383271410 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b0009550 con 0x7f19c006d560 2024-03-18T18:49:29.585 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.585+0000 7f19c512a700 1 --2- 172.21.15.165:0/1383271410 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19c006d560 0x7f19c00b21d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f19b000ec10 tx=0x7f19b000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:29.585 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.585+0000 7f19b67fc700 1 -- 172.21.15.165:0/1383271410 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b000c850 con 0x7f19c006d560 2024-03-18T18:49:29.585 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.585+0000 7f19b67fc700 1 -- 172.21.15.165:0/1383271410 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b000c9b0 con 0x7f19c006d560 2024-03-18T18:49:29.585 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.585+0000 7f19b67fc700 1 -- 172.21.15.165:0/1383271410 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b000cd40 con 0x7f19c006d560 2024-03-18T18:49:29.585 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.585+0000 7f19c7b8f700 1 -- 172.21.15.165:0/1383271410 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b8009a70 con 0x7f19c006d560 2024-03-18T18:49:29.586 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.585+0000 7f19c7b8f700 1 -- 172.21.15.165:0/1383271410 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c00b1480 con 0x7f19c006d560 2024-03-18T18:49:29.586 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.585+0000 7f19b67fc700 1 -- 172.21.15.165:0/1383271410 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f19b0005d20 con 0x7f19c006d560 2024-03-18T18:49:29.586 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.585+0000 7f19b67fc700 1 -- 172.21.15.165:0/1383271410 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f19b0006890 con 0x7f19c006d560 2024-03-18T18:49:29.586 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.586+0000 7f19c7b8f700 1 -- 172.21.15.165:0/1383271410 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c00676a0 con 0x7f19c006d560 2024-03-18T18:49:29.586 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.589+0000 7f19b67fc700 1 -- 172.21.15.165:0/1383271410 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f19b0021060 con 0x7f19c006d560 2024-03-18T18:49:29.667 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.670+0000 7f19c7b8f700 1 -- 172.21.15.165:0/1383271410 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2ca85776-3586-4679-b571-dc8d294d9dbe", "id": 7} v 0) v1 -- 0x7f19c006e960 con 0x7f19c006d560 2024-03-18T18:49:29.676 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.679+0000 7f19b67fc700 1 -- 172.21.15.165:0/1383271410 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2ca85776-3586-4679-b571-dc8d294d9dbe", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f19b0017070 con 0x7f19c006d560 2024-03-18T18:49:29.676 INFO:teuthology.orchestra.run.smithi165.stdout:7 2024-03-18T18:49:29.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.680+0000 7f19c7b8f700 1 -- 172.21.15.165:0/1383271410 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19c006d560 msgr2=0x7f19c00b21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.680+0000 7f19c7b8f700 1 --2- 172.21.15.165:0/1383271410 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19c006d560 0x7f19c00b21d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f19b000ec10 tx=0x7f19b000ef20 comp rx=0 tx=0).stop 2024-03-18T18:49:29.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.680+0000 7f19c7b8f700 1 -- 172.21.15.165:0/1383271410 shutdown_connections 2024-03-18T18:49:29.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.680+0000 7f19c7b8f700 1 --2- 172.21.15.165:0/1383271410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19c00af510 0x7f19c00b39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.680+0000 7f19c7b8f700 1 --2- 172.21.15.165:0/1383271410 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19c006a750 0x7f19c00b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.680+0000 7f19c7b8f700 1 --2- 172.21.15.165:0/1383271410 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19c006d560 0x7f19c00b21d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.680+0000 7f19c7b8f700 1 -- 172.21.15.165:0/1383271410 >> 172.21.15.165:0/1383271410 conn(0x7f19c00cff30 msgr2=0x7f19c00c7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:29.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.680+0000 7f19c7b8f700 1 -- 172.21.15.165:0/1383271410 shutdown_connections 2024-03-18T18:49:29.679 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-18T18:49:29.680+0000 7f19c7b8f700 1 -- 172.21.15.165:0/1383271410 wait complete. 2024-03-18T18:49:29.688 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-18T18:49:29.689 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-18T18:49:29.691 INFO:tasks.ceph.osd.0:Started 2024-03-18T18:49:29.691 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-18T18:49:29.691 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-18T18:49:29.693 INFO:tasks.ceph.osd.1:Started 2024-03-18T18:49:29.693 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-18T18:49:29.693 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-18T18:49:29.695 INFO:tasks.ceph.osd.2:Started 2024-03-18T18:49:29.695 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-18T18:49:29.695 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-18T18:49:29.697 INFO:tasks.ceph.osd.3:Started 2024-03-18T18:49:29.697 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-18T18:49:29.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-18T18:49:29.699 INFO:tasks.ceph.osd.4:Started 2024-03-18T18:49:29.699 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-18T18:49:29.700 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-18T18:49:29.701 INFO:tasks.ceph.osd.5:Started 2024-03-18T18:49:29.702 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-18T18:49:29.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-18T18:49:29.704 INFO:tasks.ceph.osd.6:Started 2024-03-18T18:49:29.704 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-18T18:49:29.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-18T18:49:29.706 INFO:tasks.ceph.osd.7:Started 2024-03-18T18:49:29.707 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:49:29.733 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:29.736+0000 7fbea244d240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T18:49:29.761 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:29.764+0000 7ff33f4e5240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T18:49:29.838 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:29.841+0000 7fbea244d240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T18:49:29.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.859+0000 7f06a3524700 1 Processor -- start 2024-03-18T18:49:29.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.859+0000 7f06a3524700 1 -- start start 2024-03-18T18:49:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.860+0000 7f06a3524700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f069c0b6b60 0x7f069c0cc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.860+0000 7f06a3524700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f069c0cfda0 0x7f069c0b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.860+0000 7f06a3524700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c0cd0d0 con 0x7f069c0cee90 2024-03-18T18:49:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.860+0000 7f06a3524700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f069c0b3f00 con 0x7f069c0b6b60 2024-03-18T18:49:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.860+0000 7f06a3524700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f069c0b41e0 con 0x7f069c0cfda0 2024-03-18T18:49:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.860+0000 7f06a12c0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f069c0b6b60 0x7f069c0cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.860+0000 7f06a12c0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f069c0b6b60 0x7f069c0cc830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43266/0 (socket says 172.21.15.1:43266) 2024-03-18T18:49:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.860+0000 7f06a12c0700 1 -- 172.21.15.1:0/248444104 learned_addr learned my addr 172.21.15.1:0/248444104 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:29.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.860+0000 7f06a1ac1700 1 --2- 172.21.15.1:0/248444104 >> v2:172.21.15.165:3300/0 conn(0x7f069c0cfda0 0x7f069c0b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.860+0000 7f06a12c0700 1 -- 172.21.15.1:0/248444104 >> v2:172.21.15.165:3300/0 conn(0x7f069c0cfda0 msgr2=0x7f069c0b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:29.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.860+0000 7f06a12c0700 1 --2- 172.21.15.1:0/248444104 >> v2:172.21.15.165:3300/0 conn(0x7f069c0cfda0 0x7f069c0b4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.860+0000 7f06a12c0700 1 -- 172.21.15.1:0/248444104 >> v1:172.21.15.165:6789/0 conn(0x7f069c0cee90 legacy=0x7f069c0cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.860+0000 7f06a12c0700 1 -- 172.21.15.1:0/248444104 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069c0b44c0 con 0x7f069c0b6b60 2024-03-18T18:49:29.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.860+0000 7f06a1ac1700 1 --2- 172.21.15.1:0/248444104 >> v2:172.21.15.165:3300/0 conn(0x7f069c0cfda0 0x7f069c0b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:29.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.861+0000 7f06a12c0700 1 --2- 172.21.15.1:0/248444104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f069c0b6b60 0x7f069c0cc830 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f069c071f80 tx=0x7f0698009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab81f44313add6d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:29.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.861+0000 7f0693fff700 1 -- 172.21.15.1:0/248444104 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069800b870 con 0x7f069c0b6b60 2024-03-18T18:49:29.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.861+0000 7f0693fff700 1 -- 172.21.15.1:0/248444104 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069800b9d0 con 0x7f069c0b6b60 2024-03-18T18:49:29.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.861+0000 7f0693fff700 1 -- 172.21.15.1:0/248444104 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069800bd60 con 0x7f069c0b6b60 2024-03-18T18:49:29.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.861+0000 7f06a3524700 1 -- 172.21.15.1:0/248444104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f069c0b6b60 msgr2=0x7f069c0cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.861+0000 7f06a3524700 1 --2- 172.21.15.1:0/248444104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f069c0b6b60 0x7f069c0cc830 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f069c071f80 tx=0x7f0698009bd0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.861+0000 7f06a3524700 1 -- 172.21.15.1:0/248444104 shutdown_connections 2024-03-18T18:49:29.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.861+0000 7f06a3524700 1 --2- 172.21.15.1:0/248444104 >> v2:172.21.15.165:3300/0 conn(0x7f069c0cfda0 0x7f069c0b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.861+0000 7f06a3524700 1 --2- 172.21.15.1:0/248444104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f069c0b6b60 0x7f069c0cc830 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.861+0000 7f06a3524700 1 -- 172.21.15.1:0/248444104 >> 172.21.15.1:0/248444104 conn(0x7f069c0b2ba0 msgr2=0x7f069c0b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:29.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.861+0000 7f06a3524700 1 -- 172.21.15.1:0/248444104 shutdown_connections 2024-03-18T18:49:29.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.861+0000 7f06a3524700 1 -- 172.21.15.1:0/248444104 wait complete. 2024-03-18T18:49:29.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.863+0000 7f06a3524700 1 Processor -- start 2024-03-18T18:49:29.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.863+0000 7f06a3524700 1 -- start start 2024-03-18T18:49:29.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.863+0000 7f06a3524700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f069c0b6b60 0x7f069c0c7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.863+0000 7f06a3524700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f069c0cee90 0x7f069c157960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.863+0000 7f06a3524700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f069c0cfda0 0x7f069c0cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:29.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.863+0000 7f06a3524700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c0c6ed0 con 0x7f069c0b6b60 2024-03-18T18:49:29.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.863+0000 7f06a3524700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c0c65a0 con 0x7f069c0cee90 2024-03-18T18:49:29.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.863+0000 7f06a3524700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069c0c6880 con 0x7f069c0cfda0 2024-03-18T18:49:29.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.863+0000 7f06a12c0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f069c0b6b60 0x7f069c0c7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.863+0000 7f06a12c0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f069c0b6b60 0x7f069c0c7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43268/0 (socket says 172.21.15.1:43268) 2024-03-18T18:49:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.864+0000 7f06a12c0700 1 -- 172.21.15.1:0/3638531207 learned_addr learned my addr 172.21.15.1:0/3638531207 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.864+0000 7f06a1ac1700 1 --2- 172.21.15.1:0/3638531207 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f069c0cfda0 0x7f069c0cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.864+0000 7f06a0abf700 1 --2- 172.21.15.1:0/3638531207 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f069c0cee90 0x7f069c157960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.864+0000 7f06a12c0700 1 -- 172.21.15.1:0/3638531207 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f069c0cfda0 msgr2=0x7f069c0cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.864+0000 7f06a12c0700 1 --2- 172.21.15.1:0/3638531207 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f069c0cfda0 0x7f069c0cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.864+0000 7f06a12c0700 1 -- 172.21.15.1:0/3638531207 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f069c0cee90 msgr2=0x7f069c157960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.864+0000 7f06a12c0700 1 --2- 172.21.15.1:0/3638531207 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f069c0cee90 0x7f069c157960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.864+0000 7f06a12c0700 1 -- 172.21.15.1:0/3638531207 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0698009550 con 0x7f069c0b6b60 2024-03-18T18:49:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.864+0000 7f06a12c0700 1 --2- 172.21.15.1:0/3638531207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f069c0b6b60 0x7f069c0c7360 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0698000c00 tx=0x7f0698004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.864+0000 7f0691ffb700 1 -- 172.21.15.1:0/3638531207 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0698004480 con 0x7f069c0b6b60 2024-03-18T18:49:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.864+0000 7f06a3524700 1 -- 172.21.15.1:0/3638531207 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069c0c6b60 con 0x7f069c0b6b60 2024-03-18T18:49:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.864+0000 7f06a3524700 1 -- 172.21.15.1:0/3638531207 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f069c158120 con 0x7f069c0b6b60 2024-03-18T18:49:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.864+0000 7f0691ffb700 1 -- 172.21.15.1:0/3638531207 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06980045e0 con 0x7f069c0b6b60 2024-03-18T18:49:29.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.864+0000 7f0691ffb700 1 -- 172.21.15.1:0/3638531207 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06980038a0 con 0x7f069c0b6b60 2024-03-18T18:49:29.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.865+0000 7f0691ffb700 1 -- 172.21.15.1:0/3638531207 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0698021070 con 0x7f069c0b6b60 2024-03-18T18:49:29.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.865+0000 7f0691ffb700 1 -- 172.21.15.1:0/3638531207 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f069801bb00 con 0x7f069c0b6b60 2024-03-18T18:49:29.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.865+0000 7f06a3524700 1 -- 172.21.15.1:0/3638531207 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069c0c71b0 con 0x7f069c0b6b60 2024-03-18T18:49:29.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.867+0000 7f0691ffb700 1 -- 172.21.15.1:0/3638531207 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f0698015020 con 0x7f069c0b6b60 2024-03-18T18:49:29.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.939+0000 7f06a3524700 1 -- 172.21.15.1:0/3638531207 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f069c0676a0 con 0x7f069c0b6b60 2024-03-18T18:49:29.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.939+0000 7f0691ffb700 1 -- 172.21.15.1:0/3638531207 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f069801be20 con 0x7f069c0b6b60 2024-03-18T18:49:29.937 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:49:29.937 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":10,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:29.672898+0000","last_up_change":"0.000000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:49:29.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.940+0000 7f06a3524700 1 -- 172.21.15.1:0/3638531207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f069c0b6b60 msgr2=0x7f069c0c7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:29.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.940+0000 7f06a3524700 1 --2- 172.21.15.1:0/3638531207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f069c0b6b60 0x7f069c0c7360 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0698000c00 tx=0x7f0698004750 comp rx=0 tx=0).stop 2024-03-18T18:49:29.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.940+0000 7f06a3524700 1 -- 172.21.15.1:0/3638531207 shutdown_connections 2024-03-18T18:49:29.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.940+0000 7f06a3524700 1 --2- 172.21.15.1:0/3638531207 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f069c0cfda0 0x7f069c0cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.940+0000 7f06a3524700 1 --2- 172.21.15.1:0/3638531207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f069c0b6b60 0x7f069c0c7360 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.940+0000 7f06a3524700 1 --2- 172.21.15.1:0/3638531207 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f069c0cee90 0x7f069c157960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:29.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.940+0000 7f06a3524700 1 -- 172.21.15.1:0/3638531207 >> 172.21.15.1:0/3638531207 conn(0x7f069c0b2ba0 msgr2=0x7f069c06f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:29.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.940+0000 7f06a3524700 1 -- 172.21.15.1:0/3638531207 shutdown_connections 2024-03-18T18:49:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:29.940+0000 7f06a3524700 1 -- 172.21.15.1:0/3638531207 wait complete. 2024-03-18T18:49:29.950 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-18T18:49:29.950 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-18T18:49:29.982 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:29.984+0000 7fbea244d240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T18:49:30.164 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:30.166+0000 7fbea244d240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T18:49:30.240 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:30.243+0000 7fbea244d240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T18:49:30.298 INFO:tasks.ceph.osd.5.smithi165.stderr:2024-03-18T18:49:30.301+0000 7f3ec74ec680 -1 Falling back to public interface 2024-03-18T18:49:30.302 INFO:tasks.ceph.osd.4.smithi165.stderr:2024-03-18T18:49:30.306+0000 7feff26aa680 -1 Falling back to public interface 2024-03-18T18:49:30.312 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-03-18T18:49:30.314+0000 7fb15870d680 -1 Falling back to public interface 2024-03-18T18:49:30.322 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:30.325+0000 7fbea244d240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T18:49:30.323 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-03-18T18:49:30.326+0000 7f283252a680 -1 Falling back to public interface 2024-03-18T18:49:30.323 INFO:tasks.ceph.osd.6.smithi165.stderr:2024-03-18T18:49:30.326+0000 7fbb6f44e680 -1 Falling back to public interface 2024-03-18T18:49:30.332 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-18T18:49:30.335+0000 7f6fafab5680 -1 Falling back to public interface 2024-03-18T18:49:30.335 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-03-18T18:49:30.338+0000 7f0018468680 -1 Falling back to public interface 2024-03-18T18:49:30.338 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-03-18T18:49:30.341+0000 7fea9c8d1680 -1 Falling back to public interface 2024-03-18T18:49:30.409 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:30.411+0000 7fbea244d240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T18:49:30.463 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:30.466+0000 7ff33f4e5240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T18:49:30.496 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:30.498+0000 7fbea244d240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T18:49:30.536 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:30.539+0000 7ff33f4e5240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T18:49:30.671 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:30.674+0000 7ff33f4e5240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T18:49:30.700 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:30.702+0000 7fbea244d240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T18:49:30.756 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-18T18:49:30.846 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:30.849+0000 7ff33f4e5240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T18:49:30.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.878+0000 7f573cbb1700 1 Processor -- start 2024-03-18T18:49:30.876 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.878+0000 7f573cbb1700 1 -- start start 2024-03-18T18:49:30.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.880+0000 7f573cbb1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f57381597a0 0x7f57380cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:30.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.880+0000 7f573cbb1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5738071070 0x7f57380714a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:30.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.880+0000 7f573cbb1700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57380719e0 con 0x7f57380ccdf0 2024-03-18T18:49:30.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.880+0000 7f573cbb1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f573806d000 con 0x7f57381597a0 2024-03-18T18:49:30.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.880+0000 7f573cbb1700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f573806d2e0 con 0x7f5738071070 2024-03-18T18:49:30.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.880+0000 7f573659c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f57381597a0 0x7f57380cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:30.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.880+0000 7f573659c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f57381597a0 0x7f57380cc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43280/0 (socket says 172.21.15.1:43280) 2024-03-18T18:49:30.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.880+0000 7f573659c700 1 -- 172.21.15.1:0/3886814166 learned_addr learned my addr 172.21.15.1:0/3886814166 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:30.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.880+0000 7f5736d9d700 1 --2- 172.21.15.1:0/3886814166 >> v2:172.21.15.165:3300/0 conn(0x7f5738071070 0x7f57380714a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:30.879 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.880+0000 7f573659c700 1 -- 172.21.15.1:0/3886814166 >> v2:172.21.15.165:3300/0 conn(0x7f5738071070 msgr2=0x7f57380714a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:30.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.880+0000 7f573659c700 1 --2- 172.21.15.1:0/3886814166 >> v2:172.21.15.165:3300/0 conn(0x7f5738071070 0x7f57380714a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:30.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.880+0000 7f573659c700 1 -- 172.21.15.1:0/3886814166 >> v1:172.21.15.165:6789/0 conn(0x7f57380ccdf0 legacy=0x7f57380b3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:30.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.880+0000 7f573659c700 1 -- 172.21.15.1:0/3886814166 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573806d5c0 con 0x7f57381597a0 2024-03-18T18:49:30.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.880+0000 7f5736d9d700 1 --2- 172.21.15.1:0/3886814166 >> v2:172.21.15.165:3300/0 conn(0x7f5738071070 0x7f57380714a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:30.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.880+0000 7f573659c700 1 --2- 172.21.15.1:0/3886814166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57381597a0 0x7f57380cc8b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f57380721e0 tx=0x7f572800ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=980446fc219221f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:30.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.881+0000 7f573559a700 1 -- 172.21.15.1:0/3886814166 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572800cb70 con 0x7f57381597a0 2024-03-18T18:49:30.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.881+0000 7f573559a700 1 -- 172.21.15.1:0/3886814166 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572800ccd0 con 0x7f57381597a0 2024-03-18T18:49:30.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.881+0000 7f573559a700 1 -- 172.21.15.1:0/3886814166 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5728018620 con 0x7f57381597a0 2024-03-18T18:49:30.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.881+0000 7f573cbb1700 1 -- 172.21.15.1:0/3886814166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57381597a0 msgr2=0x7f57380cc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:30.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.881+0000 7f573cbb1700 1 --2- 172.21.15.1:0/3886814166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57381597a0 0x7f57380cc8b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f57380721e0 tx=0x7f572800ad70 comp rx=0 tx=0).stop 2024-03-18T18:49:30.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.881+0000 7f573cbb1700 1 -- 172.21.15.1:0/3886814166 shutdown_connections 2024-03-18T18:49:30.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.881+0000 7f573cbb1700 1 --2- 172.21.15.1:0/3886814166 >> v2:172.21.15.165:3300/0 conn(0x7f5738071070 0x7f57380714a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:30.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.881+0000 7f573cbb1700 1 --2- 172.21.15.1:0/3886814166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57381597a0 0x7f57380cc8b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:30.880 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.881+0000 7f573cbb1700 1 -- 172.21.15.1:0/3886814166 >> 172.21.15.1:0/3886814166 conn(0x7f57380b1560 msgr2=0x7f57380b1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:30.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.881+0000 7f573cbb1700 1 -- 172.21.15.1:0/3886814166 shutdown_connections 2024-03-18T18:49:30.881 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.881+0000 7f573cbb1700 1 -- 172.21.15.1:0/3886814166 wait complete. 2024-03-18T18:49:30.883 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.883+0000 7f573cbb1700 1 Processor -- start 2024-03-18T18:49:30.883 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.883+0000 7f573cbb1700 1 -- start start 2024-03-18T18:49:30.883 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.883+0000 7f573cbb1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5738071070 0x7f5738150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:30.883 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.883+0000 7f573cbb1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57380ccdf0 0x7f573814c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:30.883 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.883+0000 7f573cbb1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57381597a0 0x7f57380cf570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:30.883 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f573cbb1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57381526a0 con 0x7f57381597a0 2024-03-18T18:49:30.883 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f573cbb1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5738152980 con 0x7f5738071070 2024-03-18T18:49:30.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f573cbb1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57380cdca0 con 0x7f57380ccdf0 2024-03-18T18:49:30.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f5736d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57381597a0 0x7f57380cf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:30.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f5735d9b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57380ccdf0 0x7f573814c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:30.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f573659c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5738071070 0x7f5738150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:30.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f5736d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57381597a0 0x7f57380cf570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43296/0 (socket says 172.21.15.1:43296) 2024-03-18T18:49:30.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f5735d9b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57380ccdf0 0x7f573814c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:51994/0 (socket says 172.21.15.1:51994) 2024-03-18T18:49:30.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f5736d9d700 1 -- 172.21.15.1:0/2555995327 learned_addr learned my addr 172.21.15.1:0/2555995327 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:30.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f573659c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5738071070 0x7f5738150af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:36740/0 (socket says 172.21.15.1:36740) 2024-03-18T18:49:30.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f5736d9d700 1 -- 172.21.15.1:0/2555995327 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57380ccdf0 msgr2=0x7f573814c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:30.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f5736d9d700 1 --2- 172.21.15.1:0/2555995327 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57380ccdf0 0x7f573814c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:30.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f5736d9d700 1 -- 172.21.15.1:0/2555995327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5738071070 msgr2=0x7f5738150af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:30.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f5736d9d700 1 --2- 172.21.15.1:0/2555995327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5738071070 0x7f5738150af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:30.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f5736d9d700 1 -- 172.21.15.1:0/2555995327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572800a6f0 con 0x7f57381597a0 2024-03-18T18:49:30.884 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f5736d9d700 1 --2- 172.21.15.1:0/2555995327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57381597a0 0x7f57380cf570 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5730007ef0 tx=0x7f573000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:30.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f5726ffd700 1 -- 172.21.15.1:0/2555995327 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573000dd80 con 0x7f57381597a0 2024-03-18T18:49:30.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f5726ffd700 1 -- 172.21.15.1:0/2555995327 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573000f040 con 0x7f57381597a0 2024-03-18T18:49:30.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f573cbb1700 1 -- 172.21.15.1:0/2555995327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57380cdf80 con 0x7f57381597a0 2024-03-18T18:49:30.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f5726ffd700 1 -- 172.21.15.1:0/2555995327 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5730003c60 con 0x7f57381597a0 2024-03-18T18:49:30.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.884+0000 7f573cbb1700 1 -- 172.21.15.1:0/2555995327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573814c8f0 con 0x7f57381597a0 2024-03-18T18:49:30.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.885+0000 7f5726ffd700 1 -- 172.21.15.1:0/2555995327 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5730004330 con 0x7f57381597a0 2024-03-18T18:49:30.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.885+0000 7f5726ffd700 1 -- 172.21.15.1:0/2555995327 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f5730015b10 con 0x7f57381597a0 2024-03-18T18:49:30.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.885+0000 7f573cbb1700 1 -- 172.21.15.1:0/2555995327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57380ce4a0 con 0x7f57381597a0 2024-03-18T18:49:30.885 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.888+0000 7f5726ffd700 1 -- 172.21.15.1:0/2555995327 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f57380ce4a0 con 0x7f57381597a0 2024-03-18T18:49:30.923 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:30.925+0000 7ff33f4e5240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T18:49:30.925 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:30.927+0000 7fbea244d240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T18:49:30.955 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.958+0000 7f573cbb1700 1 -- 172.21.15.1:0/2555995327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57380676e0 con 0x7f57381597a0 2024-03-18T18:49:30.956 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.959+0000 7f5726ffd700 1 -- 172.21.15.1:0/2555995327 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f573001a070 con 0x7f57381597a0 2024-03-18T18:49:30.956 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-18T18:49:30.956 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:29.672898+0000","last_up_change":"0.000000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:49:30.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.960+0000 7f573cbb1700 1 -- 172.21.15.1:0/2555995327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57381597a0 msgr2=0x7f57380cf570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:30.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.960+0000 7f573cbb1700 1 --2- 172.21.15.1:0/2555995327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57381597a0 0x7f57380cf570 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5730007ef0 tx=0x7f573000d3b0 comp rx=0 tx=0).stop 2024-03-18T18:49:30.957 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.960+0000 7f573cbb1700 1 -- 172.21.15.1:0/2555995327 shutdown_connections 2024-03-18T18:49:30.958 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.960+0000 7f573cbb1700 1 --2- 172.21.15.1:0/2555995327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57381597a0 0x7f57380cf570 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:30.958 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.960+0000 7f573cbb1700 1 --2- 172.21.15.1:0/2555995327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5738071070 0x7f5738150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:30.958 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.960+0000 7f573cbb1700 1 --2- 172.21.15.1:0/2555995327 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57380ccdf0 0x7f573814c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:30.958 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.960+0000 7f573cbb1700 1 -- 172.21.15.1:0/2555995327 >> 172.21.15.1:0/2555995327 conn(0x7f57380b1560 msgr2=0x7f5738154700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:30.958 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.960+0000 7f573cbb1700 1 -- 172.21.15.1:0/2555995327 shutdown_connections 2024-03-18T18:49:30.958 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:30.960+0000 7f573cbb1700 1 -- 172.21.15.1:0/2555995327 wait complete. 2024-03-18T18:49:30.970 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-18T18:49:31.000 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:31.003+0000 7ff33f4e5240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T18:49:31.077 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:31.080+0000 7ff33f4e5240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T18:49:31.089 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:31.092+0000 7fbea244d240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T18:49:31.164 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:31.166+0000 7fbea244d240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T18:49:31.174 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:31.178+0000 7ff33f4e5240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T18:49:31.370 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:31.373+0000 7ff33f4e5240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T18:49:31.588 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:31.591+0000 7ff33f4e5240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T18:49:31.745 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:31.748+0000 7ff33f4e5240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T18:49:31.815 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:31.819+0000 7ff33f4e5240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T18:49:31.871 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:31.874+0000 7fbea244d240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T18:49:32.043 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:32.046+0000 7fbea244d240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T18:49:32.208 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:32.210+0000 7fbea244d240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T18:49:32.507 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:32.510+0000 7ff33f4e5240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T18:49:32.634 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:32.637+0000 7fbea244d240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T18:49:32.672 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:32.675+0000 7ff33f4e5240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T18:49:32.808 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:32.811+0000 7fbea244d240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T18:49:32.840 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:32.843+0000 7ff33f4e5240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T18:49:33.258 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:33.261+0000 7fbea244d240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T18:49:33.330 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:33.333+0000 7ff33f4e5240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T18:49:33.362 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:33.365+0000 7fbea244d240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T18:49:33.496 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:33.499+0000 7ff33f4e5240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T18:49:33.804 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-03-18T18:49:33.807+0000 7fb15870d680 -1 osd.1 0 log_to_monitors true 2024-03-18T18:49:33.816 INFO:tasks.ceph.osd.4.smithi165.stderr:2024-03-18T18:49:33.819+0000 7feff26aa680 -1 osd.4 0 log_to_monitors true 2024-03-18T18:49:33.851 INFO:tasks.ceph.osd.5.smithi165.stderr:2024-03-18T18:49:33.854+0000 7f3ec74ec680 -1 osd.5 0 log_to_monitors true 2024-03-18T18:49:33.851 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-03-18T18:49:33.854+0000 7f283252a680 -1 osd.7 0 log_to_monitors true 2024-03-18T18:49:33.856 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-03-18T18:49:33.859+0000 7fea9c8d1680 -1 osd.0 0 log_to_monitors true 2024-03-18T18:49:33.859 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-18T18:49:33.862+0000 7f6fafab5680 -1 osd.2 0 log_to_monitors true 2024-03-18T18:49:33.864 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-03-18T18:49:33.867+0000 7f0018468680 -1 osd.3 0 log_to_monitors true 2024-03-18T18:49:33.867 INFO:tasks.ceph.osd.6.smithi165.stderr:2024-03-18T18:49:33.870+0000 7fbb6f44e680 -1 osd.6 0 log_to_monitors true 2024-03-18T18:49:33.879 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:33.882+0000 7ff33f4e5240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T18:49:33.980 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:49:33.984+0000 7ff33f4e5240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T18:49:34.387 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:34.390+0000 7fbe6385f700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-18T18:49:34.388 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:49:34.391+0000 7fbe6385f700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-18T18:49:36.720 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-03-18T18:49:36.722+0000 7fea90c71700 -1 osd.0 0 waiting for initial osdmap 2024-03-18T18:49:36.742 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-03-18T18:49:36.745+0000 7fea8884b700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T18:49:36.747 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-18T18:49:36.750+0000 7f6fa3e55700 -1 osd.2 0 waiting for initial osdmap 2024-03-18T18:49:36.762 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-03-18T18:49:36.765+0000 7f6f9ba2f700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T18:49:36.767 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-03-18T18:49:36.769+0000 7fb14caad700 -1 osd.1 0 waiting for initial osdmap 2024-03-18T18:49:36.777 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-03-18T18:49:36.780+0000 7fb144687700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T18:49:36.805 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-03-18T18:49:36.807+0000 7f000c808700 -1 osd.3 0 waiting for initial osdmap 2024-03-18T18:49:36.842 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-03-18T18:49:36.845+0000 7f00043e2700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T18:49:37.446 INFO:tasks.ceph.osd.5.smithi165.stderr:2024-03-18T18:49:37.448+0000 7f3ebb88c700 -1 osd.5 0 waiting for initial osdmap 2024-03-18T18:49:37.464 INFO:tasks.ceph.osd.5.smithi165.stderr:2024-03-18T18:49:37.467+0000 7f3eb3466700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T18:49:37.467 INFO:tasks.ceph.osd.6.smithi165.stderr:2024-03-18T18:49:37.470+0000 7fbb637ee700 -1 osd.6 0 waiting for initial osdmap 2024-03-18T18:49:37.482 INFO:tasks.ceph.osd.6.smithi165.stderr:2024-03-18T18:49:37.486+0000 7fbb5b3c8700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T18:49:37.494 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-03-18T18:49:37.497+0000 7f28268ca700 -1 osd.7 0 waiting for initial osdmap 2024-03-18T18:49:37.502 INFO:tasks.ceph.osd.4.smithi165.stderr:2024-03-18T18:49:37.505+0000 7fefe6a4a700 -1 osd.4 0 waiting for initial osdmap 2024-03-18T18:49:37.506 INFO:tasks.ceph.osd.7.smithi165.stderr:2024-03-18T18:49:37.510+0000 7f281e4a4700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T18:49:37.514 INFO:tasks.ceph.osd.4.smithi165.stderr:2024-03-18T18:49:37.518+0000 7fefde624700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T18:49:37.777 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-18T18:49:37.904 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.907+0000 7f4a0425f700 1 Processor -- start 2024-03-18T18:49:37.905 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.907+0000 7f4a0425f700 1 -- start start 2024-03-18T18:49:37.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.908+0000 7f4a0425f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f49fc160cd0 0x7f49fc0d12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:37.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.908+0000 7f4a0425f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f49fc0ce940 0x7f49fc0ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:37.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.908+0000 7f4a0425f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49fc0cf2e0 0x7f49fc161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:37.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.908+0000 7f4a0425f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49fc06c6a0 con 0x7f49fc0ce940 2024-03-18T18:49:37.907 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.908+0000 7f4a0425f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49fc06c980 con 0x7f49fc160cd0 2024-03-18T18:49:37.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.908+0000 7f4a0425f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49fc06c070 con 0x7f49fc0cf2e0 2024-03-18T18:49:37.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.908+0000 7f4a017fa700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f49fc0ce940 0x7f49fc0ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:37.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.908+0000 7f4a017fa700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f49fc0ce940 0x7f49fc0ceda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57956/0 (socket says 172.21.15.1:57956) 2024-03-18T18:49:37.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.908+0000 7f4a017fa700 1 -- 172.21.15.1:0/2107993622 learned_addr learned my addr 172.21.15.1:0/2107993622 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:37.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.908+0000 7f4a027fc700 1 --2- 172.21.15.1:0/2107993622 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49fc0cf2e0 0x7f49fc161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:37.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.908+0000 7f4a01ffb700 1 --2- 172.21.15.1:0/2107993622 >> v2:172.21.15.165:3300/0 conn(0x7f49fc160cd0 0x7f49fc0d12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:37.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.908+0000 7f4a017fa700 1 -- 172.21.15.1:0/2107993622 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49fc0cf2e0 msgr2=0x7f49fc161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:37.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.908+0000 7f4a017fa700 1 --2- 172.21.15.1:0/2107993622 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49fc0cf2e0 0x7f49fc161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:37.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.908+0000 7f4a017fa700 1 -- 172.21.15.1:0/2107993622 >> v2:172.21.15.165:3300/0 conn(0x7f49fc160cd0 msgr2=0x7f49fc0d12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:37.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.908+0000 7f4a017fa700 1 --2- 172.21.15.1:0/2107993622 >> v2:172.21.15.165:3300/0 conn(0x7f49fc160cd0 0x7f49fc0d12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:37.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.908+0000 7f4a017fa700 1 -- 172.21.15.1:0/2107993622 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49fc0d3ea0 con 0x7f49fc0ce940 2024-03-18T18:49:37.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.908+0000 7f4a017fa700 1 --2- 172.21.15.1:0/2107993622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49fc0ce940 0x7f49fc0ceda0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f49fc08cc40 tx=0x7f49ec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5295adf0780223 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:37.908 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.909+0000 7f4a00ff9700 1 -- 172.21.15.1:0/2107993622 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ec00b870 con 0x7f49fc0ce940 2024-03-18T18:49:37.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.909+0000 7f4a00ff9700 1 -- 172.21.15.1:0/2107993622 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49ec00b9d0 con 0x7f49fc0ce940 2024-03-18T18:49:37.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.909+0000 7f4a00ff9700 1 -- 172.21.15.1:0/2107993622 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ec00bd60 con 0x7f49fc0ce940 2024-03-18T18:49:37.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.909+0000 7f4a0425f700 1 -- 172.21.15.1:0/2107993622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49fc0ce940 msgr2=0x7f49fc0ceda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:37.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.909+0000 7f4a0425f700 1 --2- 172.21.15.1:0/2107993622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49fc0ce940 0x7f49fc0ceda0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f49fc08cc40 tx=0x7f49ec009bd0 comp rx=0 tx=0).stop 2024-03-18T18:49:37.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.909+0000 7f4a0425f700 1 -- 172.21.15.1:0/2107993622 shutdown_connections 2024-03-18T18:49:37.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.909+0000 7f4a0425f700 1 --2- 172.21.15.1:0/2107993622 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49fc0cf2e0 0x7f49fc161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:37.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.909+0000 7f4a0425f700 1 --2- 172.21.15.1:0/2107993622 >> v2:172.21.15.165:3300/0 conn(0x7f49fc160cd0 0x7f49fc0d12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:37.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.909+0000 7f4a0425f700 1 --2- 172.21.15.1:0/2107993622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49fc0ce940 0x7f49fc0ceda0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:37.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.909+0000 7f4a0425f700 1 -- 172.21.15.1:0/2107993622 >> 172.21.15.1:0/2107993622 conn(0x7f49fc08afd0 msgr2=0x7f49fc0d33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:37.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.909+0000 7f4a0425f700 1 -- 172.21.15.1:0/2107993622 shutdown_connections 2024-03-18T18:49:37.909 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.909+0000 7f4a0425f700 1 -- 172.21.15.1:0/2107993622 wait complete. 2024-03-18T18:49:37.912 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.911+0000 7f4a0425f700 1 Processor -- start 2024-03-18T18:49:37.912 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.911+0000 7f4a0425f700 1 -- start start 2024-03-18T18:49:37.912 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.911+0000 7f4a0425f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49fc0ce940 0x7f49fc151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:37.912 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.911+0000 7f4a0425f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49fc0cf2e0 0x7f49fc0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:37.912 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.911+0000 7f4a0425f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49fc160cd0 0x7f49fc0b6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:37.912 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.911+0000 7f4a0425f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49fc0b7d00 con 0x7f49fc0cf2e0 2024-03-18T18:49:37.912 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.911+0000 7f4a0425f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49fc0b6e80 con 0x7f49fc0ce940 2024-03-18T18:49:37.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.911+0000 7f4a0425f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49fc0b70f0 con 0x7f49fc160cd0 2024-03-18T18:49:37.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.911+0000 7f4a017fa700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49fc0cf2e0 0x7f49fc0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:37.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.911+0000 7f4a017fa700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49fc0cf2e0 0x7f49fc0cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57966/0 (socket says 172.21.15.1:57966) 2024-03-18T18:49:37.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.911+0000 7f4a017fa700 1 -- 172.21.15.1:0/1602112655 learned_addr learned my addr 172.21.15.1:0/1602112655 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:37.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.912+0000 7f4a027fc700 1 --2- 172.21.15.1:0/1602112655 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49fc160cd0 0x7f49fc0b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:37.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.912+0000 7f4a01ffb700 1 --2- 172.21.15.1:0/1602112655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49fc0ce940 0x7f49fc151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:37.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.912+0000 7f4a017fa700 1 -- 172.21.15.1:0/1602112655 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49fc160cd0 msgr2=0x7f49fc0b6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:37.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.912+0000 7f4a017fa700 1 --2- 172.21.15.1:0/1602112655 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49fc160cd0 0x7f49fc0b6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:37.913 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.912+0000 7f4a017fa700 1 -- 172.21.15.1:0/1602112655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49fc0ce940 msgr2=0x7f49fc151b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:37.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.912+0000 7f4a017fa700 1 --2- 172.21.15.1:0/1602112655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49fc0ce940 0x7f49fc151b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:37.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.912+0000 7f4a017fa700 1 -- 172.21.15.1:0/1602112655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ec009550 con 0x7f49fc0cf2e0 2024-03-18T18:49:37.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.912+0000 7f4a027fc700 1 --2- 172.21.15.1:0/1602112655 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49fc160cd0 0x7f49fc0b6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:37.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.912+0000 7f4a017fa700 1 --2- 172.21.15.1:0/1602112655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49fc0cf2e0 0x7f49fc0cc740 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f49ec000c00 tx=0x7f49ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:37.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.912+0000 7f49f27fc700 1 -- 172.21.15.1:0/1602112655 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ec004520 con 0x7f49fc0cf2e0 2024-03-18T18:49:37.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.912+0000 7f49f27fc700 1 -- 172.21.15.1:0/1602112655 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49ec01e070 con 0x7f49fc0cf2e0 2024-03-18T18:49:37.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.912+0000 7f4a0425f700 1 -- 172.21.15.1:0/1602112655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49fc0b73d0 con 0x7f49fc0cf2e0 2024-03-18T18:49:37.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.912+0000 7f49f27fc700 1 -- 172.21.15.1:0/1602112655 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ec0038c0 con 0x7f49fc0cf2e0 2024-03-18T18:49:37.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.912+0000 7f4a0425f700 1 -- 172.21.15.1:0/1602112655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49fc0b7990 con 0x7f49fc0cf2e0 2024-03-18T18:49:37.914 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.913+0000 7f4a0425f700 1 -- 172.21.15.1:0/1602112655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49fc0b7e70 con 0x7f49fc0cf2e0 2024-03-18T18:49:37.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.913+0000 7f49f27fc700 1 -- 172.21.15.1:0/1602112655 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7f49ec003a20 con 0x7f49fc0cf2e0 2024-03-18T18:49:37.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.914+0000 7f49f27fc700 1 --2- 172.21.15.1:0/1602112655 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49e006ac00 0x7f49e006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:37.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.914+0000 7f4a01ffb700 1 --2- 172.21.15.1:0/1602112655 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49e006ac00 0x7f49e006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:37.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.914+0000 7f49f27fc700 1 -- 172.21.15.1:0/1602112655 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f49ec020070 con 0x7f49fc0cf2e0 2024-03-18T18:49:37.915 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.914+0000 7f4a01ffb700 1 --2- 172.21.15.1:0/1602112655 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49e006ac00 0x7f49e006d0c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f49f8006b20 tx=0x7f49f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:37.916 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:37.919+0000 7f49f27fc700 1 -- 172.21.15.1:0/1602112655 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f49ec055a20 con 0x7f49fc0cf2e0 2024-03-18T18:49:38.118 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:38.121+0000 7f4a0425f700 1 -- 172.21.15.1:0/1602112655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49fc0676e0 con 0x7f49fc0cf2e0 2024-03-18T18:49:38.119 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:38.121+0000 7f49f27fc700 1 -- 172.21.15.1:0/1602112655 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10392 (secure 0 0 0) 0x7f49ec059040 con 0x7f49fc0cf2e0 2024-03-18T18:49:38.119 INFO:teuthology.misc.health.smithi001.stdout: 2024-03-18T18:49:38.119 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":13,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:37.757484+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:49:38.121 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:38.123+0000 7f4a0425f700 1 -- 172.21.15.1:0/1602112655 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49e006ac00 msgr2=0x7f49e006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:38.121 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:38.123+0000 7f4a0425f700 1 --2- 172.21.15.1:0/1602112655 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49e006ac00 0x7f49e006d0c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f49f8006b20 tx=0x7f49f8008040 comp rx=0 tx=0).stop 2024-03-18T18:49:38.121 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:38.123+0000 7f4a0425f700 1 -- 172.21.15.1:0/1602112655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49fc0cf2e0 msgr2=0x7f49fc0cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:38.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:38.123+0000 7f4a0425f700 1 --2- 172.21.15.1:0/1602112655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49fc0cf2e0 0x7f49fc0cc740 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f49ec000c00 tx=0x7f49ec004750 comp rx=0 tx=0).stop 2024-03-18T18:49:38.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:38.123+0000 7f4a0425f700 1 -- 172.21.15.1:0/1602112655 shutdown_connections 2024-03-18T18:49:38.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:38.123+0000 7f4a0425f700 1 --2- 172.21.15.1:0/1602112655 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49e006ac00 0x7f49e006d0c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:38.123+0000 7f4a0425f700 1 --2- 172.21.15.1:0/1602112655 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49fc160cd0 0x7f49fc0b6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:38.123+0000 7f4a0425f700 1 --2- 172.21.15.1:0/1602112655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49fc0ce940 0x7f49fc151b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:38.123+0000 7f4a0425f700 1 --2- 172.21.15.1:0/1602112655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49fc0cf2e0 0x7f49fc0cc740 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:38.123+0000 7f4a0425f700 1 -- 172.21.15.1:0/1602112655 >> 172.21.15.1:0/1602112655 conn(0x7f49fc08afd0 msgr2=0x7f49fc0b1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:38.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:38.124+0000 7f4a0425f700 1 -- 172.21.15.1:0/1602112655 shutdown_connections 2024-03-18T18:49:38.122 INFO:teuthology.misc.health.smithi001.stderr:2024-03-18T18:49:38.124+0000 7f4a0425f700 1 -- 172.21.15.1:0/1602112655 wait complete. 2024-03-18T18:49:38.132 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-18T18:49:38.132 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-18T18:49:38.132 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T18:49:38.133 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-18T18:49:38.135 INFO:tasks.ceph.mds.a:Started 2024-03-18T18:49:38.135 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T18:49:38.135 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-18T18:49:38.137 INFO:tasks.ceph.mds.c:Started 2024-03-18T18:49:38.137 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T18:49:38.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-18T18:49:38.140 INFO:tasks.ceph.mds.b:Started 2024-03-18T18:49:38.140 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T18:49:38.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-18T18:49:38.142 INFO:tasks.ceph.mds.d:Started 2024-03-18T18:49:38.143 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-18T18:49:38.143 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:49:38.216 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T18:49:38.216 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-18T18:49:38.219 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-03-18T18:49:38.219 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-03-18T18:49:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.276+0000 7f03fdf71700 1 Processor -- start 2024-03-18T18:49:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.276+0000 7f03fdf71700 1 -- start start 2024-03-18T18:49:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.277+0000 7f03fdf71700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f03f815bb90 0x7f03f8099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.277+0000 7f03fdf71700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f80d01b0 con 0x7f03f815b230 2024-03-18T18:49:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.277+0000 7f03fdf71700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f80cc6e0 con 0x7f03f815b610 2024-03-18T18:49:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.277+0000 7f03fdf71700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f80ced80 con 0x7f03f815bb90 2024-03-18T18:49:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.277+0000 7f03f77fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f03f815b230 0x7f03f80cbce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35564/0 (socket says 172.21.15.1:35564) 2024-03-18T18:49:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.277+0000 7f03f77fe700 1 -- 172.21.15.1:0/1500454827 learned_addr learned my addr 172.21.15.1:0/1500454827 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:49:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.277+0000 7f03f6ffd700 1 --2- 172.21.15.1:0/1500454827 >> v2:172.21.15.165:3300/0 conn(0x7f03f815bb90 0x7f03f8099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.277+0000 7f03f67fc700 1 -- 172.21.15.1:0/1500454827 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034384748 0 0) 0x7f03ec001110 con 0x7f03f815b230 2024-03-18T18:49:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.278+0000 7f03f67fc700 1 -- 172.21.15.1:0/1500454827 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03e00035b0 con 0x7f03f815b230 2024-03-18T18:49:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.278+0000 7f03f67fc700 1 -- 172.21.15.1:0/1500454827 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653500149 0 0) 0x7f03f80cc6e0 con 0x7f03f815b610 2024-03-18T18:49:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.278+0000 7f03f67fc700 1 -- 172.21.15.1:0/1500454827 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03ec001110 con 0x7f03f815b610 2024-03-18T18:49:38.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.278+0000 7f03f6ffd700 1 -- 172.21.15.1:0/1500454827 >> v1:172.21.15.165:6789/0 conn(0x7f03f815b610 legacy=0x7f03f80afe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:38.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.278+0000 7f03f67fc700 1 -- 172.21.15.1:0/1500454827 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2666665312 0 0) 0x7f03e00035b0 con 0x7f03f815b230 2024-03-18T18:49:38.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.278+0000 7f03f6ffd700 1 -- 172.21.15.1:0/1500454827 >> v1:172.21.15.1:6789/0 conn(0x7f03f815b230 legacy=0x7f03f80cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:38.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.278+0000 7f03f6ffd700 1 -- 172.21.15.1:0/1500454827 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f80cc9c0 con 0x7f03f815bb90 2024-03-18T18:49:38.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.278+0000 7f03f6ffd700 1 --2- 172.21.15.1:0/1500454827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03f815bb90 0x7f03f8099a40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009840 tx=0x7f03e8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b993936f291d7847 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:38.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.278+0000 7f03f67fc700 1 -- 172.21.15.1:0/1500454827 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec001800 con 0x7f03f815bb90 2024-03-18T18:49:38.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.278+0000 7f03f67fc700 1 -- 172.21.15.1:0/1500454827 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e8004030 con 0x7f03f815bb90 2024-03-18T18:49:38.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.278+0000 7f03f67fc700 1 -- 172.21.15.1:0/1500454827 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e80043c0 con 0x7f03f815bb90 2024-03-18T18:49:38.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.279+0000 7f03fdf71700 1 -- 172.21.15.1:0/1500454827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03f815bb90 msgr2=0x7f03f8099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:38.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.279+0000 7f03fdf71700 1 --2- 172.21.15.1:0/1500454827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03f815bb90 0x7f03f8099a40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009840 tx=0x7f03e8009b50 comp rx=0 tx=0).stop 2024-03-18T18:49:38.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.279+0000 7f03fdf71700 1 -- 172.21.15.1:0/1500454827 shutdown_connections 2024-03-18T18:49:38.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.279+0000 7f03fdf71700 1 --2- 172.21.15.1:0/1500454827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03f815bb90 0x7f03f8099a40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.279+0000 7f03fdf71700 1 -- 172.21.15.1:0/1500454827 >> 172.21.15.1:0/1500454827 conn(0x7f03f80b2c30 msgr2=0x7f03f80b3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:38.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.279+0000 7f03fdf71700 1 -- 172.21.15.1:0/1500454827 shutdown_connections 2024-03-18T18:49:38.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.279+0000 7f03fdf71700 1 -- 172.21.15.1:0/1500454827 wait complete. 2024-03-18T18:49:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.280+0000 7f03fdf71700 1 Processor -- start 2024-03-18T18:49:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.280+0000 7f03fdf71700 1 -- start start 2024-03-18T18:49:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.281+0000 7f03fdf71700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03f815b230 0x7f03f80b5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.281+0000 7f03fdf71700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03f815b610 0x7f03f80c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:38.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.281+0000 7f03fdf71700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03f815bb90 0x7f03f81593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.281+0000 7f03fdf71700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f80b50c0 con 0x7f03f815b230 2024-03-18T18:49:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.281+0000 7f03fdf71700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f80b53a0 con 0x7f03f815b610 2024-03-18T18:49:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.281+0000 7f03fdf71700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f80b5680 con 0x7f03f815bb90 2024-03-18T18:49:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.281+0000 7f03f77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03f815b230 0x7f03f80b5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.281+0000 7f03f77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03f815b230 0x7f03f80b5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57984/0 (socket says 172.21.15.1:57984) 2024-03-18T18:49:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.281+0000 7f03f77fe700 1 -- 172.21.15.1:0/3137035327 learned_addr learned my addr 172.21.15.1:0/3137035327 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.281+0000 7f03f6ffd700 1 --2- 172.21.15.1:0/3137035327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03f815b610 0x7f03f80c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.281+0000 7f03f7fff700 1 --2- 172.21.15.1:0/3137035327 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03f815bb90 0x7f03f81593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.281+0000 7f03f77fe700 1 -- 172.21.15.1:0/3137035327 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03f815bb90 msgr2=0x7f03f81593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.281+0000 7f03f77fe700 1 --2- 172.21.15.1:0/3137035327 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03f815bb90 0x7f03f81593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.281+0000 7f03f77fe700 1 -- 172.21.15.1:0/3137035327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03f815b610 msgr2=0x7f03f80c99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:38.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.281+0000 7f03f77fe700 1 --2- 172.21.15.1:0/3137035327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03f815b610 0x7f03f80c99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.281+0000 7f03f77fe700 1 -- 172.21.15.1:0/3137035327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e80094a0 con 0x7f03f815b230 2024-03-18T18:49:38.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.281+0000 7f03f6ffd700 1 --2- 172.21.15.1:0/3137035327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03f815b610 0x7f03f80c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:38.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.281+0000 7f03f77fe700 1 --2- 172.21.15.1:0/3137035327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03f815b230 0x7f03f80b5e80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f03ec00cad0 tx=0x7f03ec00ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:38.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.281+0000 7f03dffff700 1 -- 172.21.15.1:0/3137035327 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec004a80 con 0x7f03f815b230 2024-03-18T18:49:38.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.282+0000 7f03dffff700 1 -- 172.21.15.1:0/3137035327 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ec004c00 con 0x7f03f815b230 2024-03-18T18:49:38.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.282+0000 7f03fdf71700 1 -- 172.21.15.1:0/3137035327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f80b5960 con 0x7f03f815b230 2024-03-18T18:49:38.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.282+0000 7f03dffff700 1 -- 172.21.15.1:0/3137035327 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec0070d0 con 0x7f03f815b230 2024-03-18T18:49:38.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.282+0000 7f03fdf71700 1 -- 172.21.15.1:0/3137035327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f815a840 con 0x7f03f815b230 2024-03-18T18:49:38.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.283+0000 7f03fdf71700 1 -- 172.21.15.1:0/3137035327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f80b5ba0 con 0x7f03f815b230 2024-03-18T18:49:38.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.283+0000 7f03dffff700 1 -- 172.21.15.1:0/3137035327 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7f03ec007230 con 0x7f03f815b230 2024-03-18T18:49:38.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.283+0000 7f03dffff700 1 --2- 172.21.15.1:0/3137035327 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f03e406ac00 0x7f03e406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:38.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.283+0000 7f03f6ffd700 1 --2- 172.21.15.1:0/3137035327 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f03e406ac00 0x7f03e406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:38.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.284+0000 7f03dffff700 1 -- 172.21.15.1:0/3137035327 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f03ec0880d0 con 0x7f03f815b230 2024-03-18T18:49:38.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.284+0000 7f03f6ffd700 1 --2- 172.21.15.1:0/3137035327 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f03e406ac00 0x7f03e406d0c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f03f8072600 tx=0x7f03e8013040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:38.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.288+0000 7f03dffff700 1 -- 172.21.15.1:0/3137035327 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f03f80b5ba0 con 0x7f03f815b230 2024-03-18T18:49:38.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.497+0000 7f03fdf71700 1 -- 172.21.15.1:0/3137035327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03f80d01b0 con 0x7f03f815b230 2024-03-18T18:49:38.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.498+0000 7f03dffff700 1 -- 172.21.15.1:0/3137035327 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10392 (secure 0 0 0) 0x7f03f80d01b0 con 0x7f03f815b230 2024-03-18T18:49:38.495 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:49:38.495 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":13,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:37.757484+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:49:38.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.500+0000 7f03fdf71700 1 -- 172.21.15.1:0/3137035327 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f03e406ac00 msgr2=0x7f03e406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:38.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.500+0000 7f03fdf71700 1 --2- 172.21.15.1:0/3137035327 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f03e406ac00 0x7f03e406d0c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f03f8072600 tx=0x7f03e8013040 comp rx=0 tx=0).stop 2024-03-18T18:49:38.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.500+0000 7f03fdf71700 1 -- 172.21.15.1:0/3137035327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03f815b230 msgr2=0x7f03f80b5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:38.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.500+0000 7f03fdf71700 1 --2- 172.21.15.1:0/3137035327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03f815b230 0x7f03f80b5e80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f03ec00cad0 tx=0x7f03ec00ce90 comp rx=0 tx=0).stop 2024-03-18T18:49:38.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.500+0000 7f03fdf71700 1 -- 172.21.15.1:0/3137035327 shutdown_connections 2024-03-18T18:49:38.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.500+0000 7f03fdf71700 1 --2- 172.21.15.1:0/3137035327 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f03e406ac00 0x7f03e406d0c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.500+0000 7f03fdf71700 1 --2- 172.21.15.1:0/3137035327 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03f815bb90 0x7f03f81593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.500+0000 7f03fdf71700 1 --2- 172.21.15.1:0/3137035327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03f815b230 0x7f03f80b5e80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.500+0000 7f03fdf71700 1 --2- 172.21.15.1:0/3137035327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03f815b610 0x7f03f80c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.500+0000 7f03fdf71700 1 -- 172.21.15.1:0/3137035327 >> 172.21.15.1:0/3137035327 conn(0x7f03f80b2c30 msgr2=0x7f03f80cbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:38.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.500+0000 7f03fdf71700 1 -- 172.21.15.1:0/3137035327 shutdown_connections 2024-03-18T18:49:38.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.500+0000 7f03fdf71700 1 -- 172.21.15.1:0/3137035327 wait complete. 2024-03-18T18:49:38.509 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-03-18T18:49:38.509 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T18:49:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.637+0000 7ff6b7942700 1 Processor -- start 2024-03-18T18:49:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.637+0000 7ff6b7942700 1 -- start start 2024-03-18T18:49:38.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.637+0000 7ff6b7942700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff6b01465a0 0x7ff6b0168660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:38.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.637+0000 7ff6b7942700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6b0072660 con 0x7ff6b006ae60 2024-03-18T18:49:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.637+0000 7ff6b7942700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6b006ebd0 con 0x7ff6b0146190 2024-03-18T18:49:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.637+0000 7ff6b7942700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b006f850 con 0x7ff6b01465a0 2024-03-18T18:49:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.637+0000 7ff6b4edd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff6b006ae60 0x7ff6b0164f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35572/0 (socket says 172.21.15.1:35572) 2024-03-18T18:49:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.637+0000 7ff6b4edd700 1 -- 172.21.15.1:0/794524449 learned_addr learned my addr 172.21.15.1:0/794524449 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:49:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6b5edf700 1 --2- 172.21.15.1:0/794524449 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff6b01465a0 0x7ff6b0168660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6a3fff700 1 -- 172.21.15.1:0/794524449 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868444953 0 0) 0x7ff6b0072660 con 0x7ff6b006ae60 2024-03-18T18:49:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6a3fff700 1 -- 172.21.15.1:0/794524449 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff69c0035b0 con 0x7ff6b006ae60 2024-03-18T18:49:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6a3fff700 1 -- 172.21.15.1:0/794524449 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2669191240 0 0) 0x7ff69c0035b0 con 0x7ff6b006ae60 2024-03-18T18:49:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6a3fff700 1 -- 172.21.15.1:0/794524449 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff6b01465a0 msgr2=0x7ff6b0168660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6a3fff700 1 --2- 172.21.15.1:0/794524449 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff6b01465a0 0x7ff6b0168660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6a3fff700 1 -- 172.21.15.1:0/794524449 >> v1:172.21.15.165:6789/0 conn(0x7ff6b0146190 legacy=0x7ff6b006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6a3fff700 1 -- 172.21.15.1:0/794524449 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b006eeb0 con 0x7ff6b006ae60 2024-03-18T18:49:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6a3fff700 1 -- 172.21.15.1:0/794524449 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7ff6ac001670 con 0x7ff6b006ae60 2024-03-18T18:49:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6a3fff700 1 -- 172.21.15.1:0/794524449 >> v1:172.21.15.1:6789/0 conn(0x7ff6b006ae60 legacy=0x7ff6b0164f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6a3fff700 1 --2- 172.21.15.1:0/794524449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff69c003a30 0x7ff69c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6a3fff700 1 --2- 172.21.15.1:0/794524449 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff69c0043e0 0x7ff69c0048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6a3fff700 1 --2- 172.21.15.1:0/794524449 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff6b01465a0 0x7ff69c005110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6a3fff700 1 -- 172.21.15.1:0/794524449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b006f850 con 0x7ff69c003a30 2024-03-18T18:49:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6a3fff700 1 -- 172.21.15.1:0/794524449 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c00ba10 con 0x7ff69c0043e0 2024-03-18T18:49:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6a3fff700 1 -- 172.21.15.1:0/794524449 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c00bbd0 con 0x7ff6b01465a0 2024-03-18T18:49:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6b56de700 1 --2- 172.21.15.1:0/794524449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff69c003a30 0x7ff69c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6b4edd700 1 --2- 172.21.15.1:0/794524449 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff69c0043e0 0x7ff69c0048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6b5edf700 1 --2- 172.21.15.1:0/794524449 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff6b01465a0 0x7ff69c005110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6b56de700 1 -- 172.21.15.1:0/794524449 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff6b01465a0 msgr2=0x7ff69c005110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6b56de700 1 --2- 172.21.15.1:0/794524449 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff6b01465a0 0x7ff69c005110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6b56de700 1 -- 172.21.15.1:0/794524449 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff69c0043e0 msgr2=0x7ff69c0048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6b56de700 1 --2- 172.21.15.1:0/794524449 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff69c0043e0 0x7ff69c0048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6b56de700 1 -- 172.21.15.1:0/794524449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69c0035b0 con 0x7ff69c003a30 2024-03-18T18:49:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.638+0000 7ff6b4edd700 1 --2- 172.21.15.1:0/794524449 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff69c0043e0 0x7ff69c0048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.639+0000 7ff6b56de700 1 --2- 172.21.15.1:0/794524449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff69c003a30 0x7ff69c003e10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4001900 tx=0x7ff69c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=860379f4046b718a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.639+0000 7ff6a3fff700 1 -- 172.21.15.1:0/794524449 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a4016070 con 0x7ff69c003a30 2024-03-18T18:49:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.639+0000 7ff6a3fff700 1 -- 172.21.15.1:0/794524449 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a4001d30 con 0x7ff69c003a30 2024-03-18T18:49:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.639+0000 7ff6a3fff700 1 -- 172.21.15.1:0/794524449 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a40013f0 con 0x7ff69c003a30 2024-03-18T18:49:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.639+0000 7ff6b7942700 1 -- 172.21.15.1:0/794524449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff69c003a30 msgr2=0x7ff69c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.639+0000 7ff6b7942700 1 --2- 172.21.15.1:0/794524449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff69c003a30 0x7ff69c003e10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4001900 tx=0x7ff69c005880 comp rx=0 tx=0).stop 2024-03-18T18:49:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.639+0000 7ff6b5edf700 1 -- 172.21.15.1:0/794524449 reap_dead start 2024-03-18T18:49:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.639+0000 7ff6b7942700 1 -- 172.21.15.1:0/794524449 shutdown_connections 2024-03-18T18:49:38.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.639+0000 7ff6b7942700 1 -- 172.21.15.1:0/794524449 >> 172.21.15.1:0/794524449 conn(0x7ff6b00b1d20 msgr2=0x7ff6b00b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:38.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.639+0000 7ff6b7942700 1 -- 172.21.15.1:0/794524449 shutdown_connections 2024-03-18T18:49:38.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.639+0000 7ff6b7942700 1 -- 172.21.15.1:0/794524449 wait complete. 2024-03-18T18:49:38.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.641+0000 7ff6b7942700 1 Processor -- start 2024-03-18T18:49:38.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.641+0000 7ff6b7942700 1 -- start start 2024-03-18T18:49:38.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.642+0000 7ff6b7942700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff69c003a30 0x7ff6b00c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:38.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.642+0000 7ff6b7942700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6b00ca780 0x7ff6b015cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:38.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.642+0000 7ff6b7942700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff6b00afc70 0x7ff6b00b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:38.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.642+0000 7ff6b7942700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b0143220 con 0x7ff6b00ca780 2024-03-18T18:49:38.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.642+0000 7ff6b7942700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b00b0630 con 0x7ff6b00afc70 2024-03-18T18:49:38.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.642+0000 7ff6b7942700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b00cac00 con 0x7ff69c003a30 2024-03-18T18:49:38.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.642+0000 7ff6b4edd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6b00ca780 0x7ff6b015cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:38.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.642+0000 7ff6b4edd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6b00ca780 0x7ff6b015cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58000/0 (socket says 172.21.15.1:58000) 2024-03-18T18:49:38.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.642+0000 7ff6b4edd700 1 -- 172.21.15.1:0/3222191055 learned_addr learned my addr 172.21.15.1:0/3222191055 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:38.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.642+0000 7ff6b56de700 1 --2- 172.21.15.1:0/3222191055 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff69c003a30 0x7ff6b00c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:38.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.642+0000 7ff6b5edf700 1 --2- 172.21.15.1:0/3222191055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff6b00afc70 0x7ff6b00b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:38.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.642+0000 7ff6b4edd700 1 -- 172.21.15.1:0/3222191055 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff69c003a30 msgr2=0x7ff6b00c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.642+0000 7ff6b4edd700 1 --2- 172.21.15.1:0/3222191055 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff69c003a30 0x7ff6b00c8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.642+0000 7ff6b4edd700 1 -- 172.21.15.1:0/3222191055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff6b00afc70 msgr2=0x7ff6b00b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.642+0000 7ff6b4edd700 1 --2- 172.21.15.1:0/3222191055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff6b00afc70 0x7ff6b00b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.642+0000 7ff6b4edd700 1 -- 172.21.15.1:0/3222191055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b00caed0 con 0x7ff6b00ca780 2024-03-18T18:49:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.642+0000 7ff6b4edd700 1 --2- 172.21.15.1:0/3222191055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6b00ca780 0x7ff6b015cb30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac001200 tx=0x7ff6ac00e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.642+0000 7ff6a1ffb700 1 -- 172.21.15.1:0/3222191055 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac010510 con 0x7ff6b00ca780 2024-03-18T18:49:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.643+0000 7ff6a1ffb700 1 -- 172.21.15.1:0/3222191055 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6ac010670 con 0x7ff6b00ca780 2024-03-18T18:49:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.643+0000 7ff6b7942700 1 -- 172.21.15.1:0/3222191055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b00cb1b0 con 0x7ff6b00ca780 2024-03-18T18:49:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.643+0000 7ff6a1ffb700 1 -- 172.21.15.1:0/3222191055 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac0109e0 con 0x7ff6b00ca780 2024-03-18T18:49:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.643+0000 7ff6b7942700 1 -- 172.21.15.1:0/3222191055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b00cb770 con 0x7ff6b00ca780 2024-03-18T18:49:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.644+0000 7ff6b7942700 1 -- 172.21.15.1:0/3222191055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b00676a0 con 0x7ff6b00ca780 2024-03-18T18:49:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.644+0000 7ff6a1ffb700 1 -- 172.21.15.1:0/3222191055 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7ff6ac010b40 con 0x7ff6b00ca780 2024-03-18T18:49:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.644+0000 7ff6a1ffb700 1 --2- 172.21.15.1:0/3222191055 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff69406ac00 0x7ff69406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:38.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.645+0000 7ff6a1ffb700 1 -- 172.21.15.1:0/3222191055 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7ff6ac088460 con 0x7ff6b00ca780 2024-03-18T18:49:38.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.645+0000 7ff6b56de700 1 --2- 172.21.15.1:0/3222191055 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff69406ac00 0x7ff69406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:38.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.645+0000 7ff6b56de700 1 --2- 172.21.15.1:0/3222191055 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff69406ac00 0x7ff69406d0c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4014040 tx=0x7ff6a400a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:38.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.650+0000 7ff6a1ffb700 1 -- 172.21.15.1:0/3222191055 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff6ac053c00 con 0x7ff6b00ca780 2024-03-18T18:49:38.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.901+0000 7ff6b7942700 1 -- 172.21.15.1:0/3222191055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6b00cc5a0 con 0x7ff6b00ca780 2024-03-18T18:49:38.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.901+0000 7ff6a1ffb700 1 -- 172.21.15.1:0/3222191055 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1253 (secure 0 0 0) 0x7ff6ac057220 con 0x7ff6b00ca780 2024-03-18T18:49:38.899 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:49:38.899 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2118908012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2118908012},{"type":"v1","addr":"172.21.15.1:6837","nonce":2118908012}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2024-03-18T18:49:38.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.904+0000 7ff6b7942700 1 -- 172.21.15.1:0/3222191055 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff69406ac00 msgr2=0x7ff69406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:38.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.904+0000 7ff6b7942700 1 --2- 172.21.15.1:0/3222191055 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff69406ac00 0x7ff69406d0c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4014040 tx=0x7ff6a400a000 comp rx=0 tx=0).stop 2024-03-18T18:49:38.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.904+0000 7ff6b7942700 1 -- 172.21.15.1:0/3222191055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6b00ca780 msgr2=0x7ff6b015cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:38.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.904+0000 7ff6b7942700 1 --2- 172.21.15.1:0/3222191055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6b00ca780 0x7ff6b015cb30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac001200 tx=0x7ff6ac00e9e0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.904+0000 7ff6b7942700 1 -- 172.21.15.1:0/3222191055 shutdown_connections 2024-03-18T18:49:38.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.904+0000 7ff6b7942700 1 --2- 172.21.15.1:0/3222191055 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff69406ac00 0x7ff69406d0c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.904+0000 7ff6b7942700 1 --2- 172.21.15.1:0/3222191055 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff69c003a30 0x7ff6b00c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.904+0000 7ff6b7942700 1 --2- 172.21.15.1:0/3222191055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff6b00afc70 0x7ff6b00b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.904+0000 7ff6b7942700 1 --2- 172.21.15.1:0/3222191055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6b00ca780 0x7ff6b015cb30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:38.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.904+0000 7ff6b7942700 1 -- 172.21.15.1:0/3222191055 >> 172.21.15.1:0/3222191055 conn(0x7ff6b00b1d20 msgr2=0x7ff6b00b3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:38.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.904+0000 7ff6b7942700 1 -- 172.21.15.1:0/3222191055 shutdown_connections 2024-03-18T18:49:38.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:38.904+0000 7ff6b7942700 1 -- 172.21.15.1:0/3222191055 wait complete. 2024-03-18T18:49:38.903 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 2 2024-03-18T18:49:40.914 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T18:49:41.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.057+0000 7f6a70a1f700 1 Processor -- start 2024-03-18T18:49:41.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.057+0000 7f6a70a1f700 1 -- start start 2024-03-18T18:49:41.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.058+0000 7f6a70a1f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a6c071a80 0x7f6a6c06e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:41.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.058+0000 7f6a70a1f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a6c0acd80 con 0x7f6a6c15a320 2024-03-18T18:49:41.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.058+0000 7f6a70a1f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a6c0cf220 con 0x7f6a6c071600 2024-03-18T18:49:41.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.058+0000 7f6a70a1f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c071f10 con 0x7f6a6c071a80 2024-03-18T18:49:41.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.058+0000 7f6a6a59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6a6c15a320 0x7f6a6c0ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35578/0 (socket says 172.21.15.1:35578) 2024-03-18T18:49:41.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.058+0000 7f6a6a59c700 1 -- 172.21.15.1:0/2185510259 learned_addr learned my addr 172.21.15.1:0/2185510259 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:49:41.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.058+0000 7f6a6ad9d700 1 --2- 172.21.15.1:0/2185510259 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a6c071a80 0x7f6a6c06e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:41.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.058+0000 7f6a6959a700 1 -- 172.21.15.1:0/2185510259 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 666090075 0 0) 0x7f6a6c0acd80 con 0x7f6a6c15a320 2024-03-18T18:49:41.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.058+0000 7f6a6959a700 1 -- 172.21.15.1:0/2185510259 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a540057c0 con 0x7f6a6c15a320 2024-03-18T18:49:41.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.059+0000 7f6a6959a700 1 -- 172.21.15.1:0/2185510259 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138391820 0 0) 0x7f6a5c001110 con 0x7f6a6c071600 2024-03-18T18:49:41.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.059+0000 7f6a6959a700 1 -- 172.21.15.1:0/2185510259 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a6c0acd80 con 0x7f6a6c071600 2024-03-18T18:49:41.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.059+0000 7f6a6959a700 1 -- 172.21.15.1:0/2185510259 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1362848416 0 0) 0x7f6a540057c0 con 0x7f6a6c15a320 2024-03-18T18:49:41.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.059+0000 7f6a6ad9d700 1 -- 172.21.15.1:0/2185510259 >> v1:172.21.15.165:6789/0 conn(0x7f6a6c071600 legacy=0x7f6a6c06e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:41.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.059+0000 7f6a6ad9d700 1 -- 172.21.15.1:0/2185510259 >> v1:172.21.15.1:6789/0 conn(0x7f6a6c15a320 legacy=0x7f6a6c0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:41.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.059+0000 7f6a6ad9d700 1 -- 172.21.15.1:0/2185510259 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6c0cef40 con 0x7f6a6c071a80 2024-03-18T18:49:41.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.059+0000 7f6a6ad9d700 1 --2- 172.21.15.1:0/2185510259 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a6c071a80 0x7f6a6c06e9d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6a64009dc0 tx=0x7f6a6400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=260ae760938dedb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:41.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.059+0000 7f6a6959a700 1 -- 172.21.15.1:0/2185510259 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6400e070 con 0x7f6a6c071a80 2024-03-18T18:49:41.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.059+0000 7f6a6959a700 1 -- 172.21.15.1:0/2185510259 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a64003a20 con 0x7f6a6c071a80 2024-03-18T18:49:41.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.059+0000 7f6a6959a700 1 -- 172.21.15.1:0/2185510259 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60001670 con 0x7f6a6c071a80 2024-03-18T18:49:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.060+0000 7f6a70a1f700 1 -- 172.21.15.1:0/2185510259 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a6c071a80 msgr2=0x7f6a6c06e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.060+0000 7f6a70a1f700 1 --2- 172.21.15.1:0/2185510259 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a6c071a80 0x7f6a6c06e9d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6a64009dc0 tx=0x7f6a6400b330 comp rx=0 tx=0).stop 2024-03-18T18:49:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.060+0000 7f6a70a1f700 1 -- 172.21.15.1:0/2185510259 shutdown_connections 2024-03-18T18:49:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.060+0000 7f6a70a1f700 1 --2- 172.21.15.1:0/2185510259 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a6c071a80 0x7f6a6c06e9d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.060+0000 7f6a70a1f700 1 -- 172.21.15.1:0/2185510259 >> 172.21.15.1:0/2185510259 conn(0x7f6a6c0cc990 msgr2=0x7f6a6c0ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.060+0000 7f6a70a1f700 1 -- 172.21.15.1:0/2185510259 shutdown_connections 2024-03-18T18:49:41.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.060+0000 7f6a70a1f700 1 -- 172.21.15.1:0/2185510259 wait complete. 2024-03-18T18:49:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.061+0000 7f6a70a1f700 1 Processor -- start 2024-03-18T18:49:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.061+0000 7f6a70a1f700 1 -- start start 2024-03-18T18:49:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.062+0000 7f6a70a1f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a6c071600 0x7f6a6c156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.062+0000 7f6a70a1f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a6c071a80 0x7f6a6c0c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.062+0000 7f6a70a1f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a6c0b2b50 0x7f6a6c0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.062+0000 7f6a70a1f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c0ca220 con 0x7f6a6c071600 2024-03-18T18:49:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.062+0000 7f6a70a1f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c0b30c0 con 0x7f6a6c071a80 2024-03-18T18:49:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.062+0000 7f6a70a1f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c15b800 con 0x7f6a6c0b2b50 2024-03-18T18:49:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.062+0000 7f6a6a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a6c071600 0x7f6a6c156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.062+0000 7f6a6a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a6c071600 0x7f6a6c156fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58004/0 (socket says 172.21.15.1:58004) 2024-03-18T18:49:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.062+0000 7f6a6a59c700 1 -- 172.21.15.1:0/2315343181 learned_addr learned my addr 172.21.15.1:0/2315343181 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.062+0000 7f6a69d9b700 1 --2- 172.21.15.1:0/2315343181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a6c071a80 0x7f6a6c0c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.062+0000 7f6a6a59c700 1 -- 172.21.15.1:0/2315343181 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a6c0b2b50 msgr2=0x7f6a6c0b4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.062+0000 7f6a6a59c700 1 --2- 172.21.15.1:0/2315343181 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a6c0b2b50 0x7f6a6c0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.062+0000 7f6a6a59c700 1 -- 172.21.15.1:0/2315343181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a6c071a80 msgr2=0x7f6a6c0c6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.062+0000 7f6a6a59c700 1 --2- 172.21.15.1:0/2315343181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a6c071a80 0x7f6a6c0c6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.062+0000 7f6a6a59c700 1 -- 172.21.15.1:0/2315343181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a64009a70 con 0x7f6a6c071600 2024-03-18T18:49:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.062+0000 7f6a6a59c700 1 --2- 172.21.15.1:0/2315343181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a6c071600 0x7f6a6c156fd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6a6000c730 tx=0x7f6a6000caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.062+0000 7f6a5affd700 1 -- 172.21.15.1:0/2315343181 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60004c80 con 0x7f6a6c071600 2024-03-18T18:49:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.063+0000 7f6a5affd700 1 -- 172.21.15.1:0/2315343181 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a60004e00 con 0x7f6a6c071600 2024-03-18T18:49:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.063+0000 7f6a70a1f700 1 -- 172.21.15.1:0/2315343181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a6c15bae0 con 0x7f6a6c071600 2024-03-18T18:49:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.063+0000 7f6a5affd700 1 -- 172.21.15.1:0/2315343181 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60007310 con 0x7f6a6c071600 2024-03-18T18:49:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.063+0000 7f6a70a1f700 1 -- 172.21.15.1:0/2315343181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a6c15c0a0 con 0x7f6a6c071600 2024-03-18T18:49:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.063+0000 7f6a70a1f700 1 -- 172.21.15.1:0/2315343181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6c0676a0 con 0x7f6a6c071600 2024-03-18T18:49:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.064+0000 7f6a5affd700 1 -- 172.21.15.1:0/2315343181 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7f6a60007470 con 0x7f6a6c071600 2024-03-18T18:49:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.065+0000 7f6a5affd700 1 --2- 172.21.15.1:0/2315343181 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6a4c06ac00 0x7f6a4c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.065+0000 7f6a69d9b700 1 --2- 172.21.15.1:0/2315343181 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6a4c06ac00 0x7f6a4c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.065+0000 7f6a5affd700 1 -- 172.21.15.1:0/2315343181 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4846+0+0 (secure 0 0 0) 0x7f6a60088cb0 con 0x7f6a6c071600 2024-03-18T18:49:41.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.065+0000 7f6a69d9b700 1 --2- 172.21.15.1:0/2315343181 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6a4c06ac00 0x7f6a4c06d0c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c001270 tx=0x7f6a5c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:41.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.070+0000 7f6a5affd700 1 -- 172.21.15.1:0/2315343181 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6a60054360 con 0x7f6a6c071600 2024-03-18T18:49:41.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.317+0000 7f6a70a1f700 1 -- 172.21.15.1:0/2315343181 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a6c0cc340 con 0x7f6a6c071600 2024-03-18T18:49:41.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.317+0000 7f6a5affd700 1 -- 172.21.15.1:0/2315343181 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3521 (secure 0 0 0) 0x7f6a60057980 con 0x7f6a6c071600 2024-03-18T18:49:41.314 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:49:41.314 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/883056851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":883056851},{"type":"v1","addr":"172.21.15.165:6834","nonce":883056851}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1388214718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1388214718},{"type":"v1","addr":"172.21.15.165:6835","nonce":1388214718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3020786191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3020786191},{"type":"v1","addr":"172.21.15.1:6836","nonce":3020786191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4273,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2118908012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2118908012},{"type":"v1","addr":"172.21.15.1:6837","nonce":2118908012}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2024-03-18T18:49:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.319+0000 7f6a70a1f700 1 -- 172.21.15.1:0/2315343181 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6a4c06ac00 msgr2=0x7f6a4c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.319+0000 7f6a70a1f700 1 --2- 172.21.15.1:0/2315343181 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6a4c06ac00 0x7f6a4c06d0c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c001270 tx=0x7f6a5c009040 comp rx=0 tx=0).stop 2024-03-18T18:49:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.319+0000 7f6a70a1f700 1 -- 172.21.15.1:0/2315343181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a6c071600 msgr2=0x7f6a6c156fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.319+0000 7f6a70a1f700 1 --2- 172.21.15.1:0/2315343181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a6c071600 0x7f6a6c156fd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6a6000c730 tx=0x7f6a6000caf0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.319+0000 7f6a70a1f700 1 -- 172.21.15.1:0/2315343181 shutdown_connections 2024-03-18T18:49:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.319+0000 7f6a70a1f700 1 --2- 172.21.15.1:0/2315343181 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6a4c06ac00 0x7f6a4c06d0c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.319+0000 7f6a70a1f700 1 --2- 172.21.15.1:0/2315343181 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a6c0b2b50 0x7f6a6c0b4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.319+0000 7f6a70a1f700 1 --2- 172.21.15.1:0/2315343181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a6c071600 0x7f6a6c156fd0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.319+0000 7f6a70a1f700 1 --2- 172.21.15.1:0/2315343181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a6c071a80 0x7f6a6c0c6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.319+0000 7f6a70a1f700 1 -- 172.21.15.1:0/2315343181 >> 172.21.15.1:0/2315343181 conn(0x7f6a6c0cc990 msgr2=0x7f6a6c06a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.319+0000 7f6a70a1f700 1 -- 172.21.15.1:0/2315343181 shutdown_connections 2024-03-18T18:49:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.319+0000 7f6a70a1f700 1 -- 172.21.15.1:0/2315343181 wait complete. 2024-03-18T18:49:41.317 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 3 2024-03-18T18:49:41.327 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:49:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.456+0000 7fc0c1982700 1 Processor -- start 2024-03-18T18:49:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.456+0000 7fc0c1982700 1 -- start start 2024-03-18T18:49:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.457+0000 7fc0c1982700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc0bc146190 0x7fc0bc06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.457+0000 7fc0c1982700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc0bc1465a0 0x7fc0bc06ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.457+0000 7fc0c1982700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0bc146af0 0x7fc0bc071650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.457+0000 7fc0c1982700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0bc06f850 con 0x7fc0bc146190 2024-03-18T18:49:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.457+0000 7fc0c1982700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0bc06ebd0 con 0x7fc0bc1465a0 2024-03-18T18:49:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.457+0000 7fc0c1982700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc06eeb0 con 0x7fc0bc146af0 2024-03-18T18:49:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.457+0000 7fc0baffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc0bc146190 0x7fc0bc06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.457+0000 7fc0baffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc0bc146190 0x7fc0bc06a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58014/0 (socket says 172.21.15.1:58014) 2024-03-18T18:49:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.457+0000 7fc0baffd700 1 -- 172.21.15.1:0/1158907480 learned_addr learned my addr 172.21.15.1:0/1158907480 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:41.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.457+0000 7fc0ba7fc700 1 --2- 172.21.15.1:0/1158907480 >> v2:172.21.15.165:3300/0 conn(0x7fc0bc1465a0 0x7fc0bc06ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.457+0000 7fc0bb7fe700 1 --2- 172.21.15.1:0/1158907480 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0bc146af0 0x7fc0bc071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.458+0000 7fc0baffd700 1 -- 172.21.15.1:0/1158907480 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0bc146af0 msgr2=0x7fc0bc071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.458+0000 7fc0baffd700 1 --2- 172.21.15.1:0/1158907480 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0bc146af0 0x7fc0bc071650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.458+0000 7fc0baffd700 1 -- 172.21.15.1:0/1158907480 >> v2:172.21.15.165:3300/0 conn(0x7fc0bc1465a0 msgr2=0x7fc0bc06ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.458+0000 7fc0baffd700 1 --2- 172.21.15.1:0/1158907480 >> v2:172.21.15.165:3300/0 conn(0x7fc0bc1465a0 0x7fc0bc06ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.458+0000 7fc0baffd700 1 -- 172.21.15.1:0/1158907480 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0bc06f190 con 0x7fc0bc146190 2024-03-18T18:49:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.458+0000 7fc0baffd700 1 --2- 172.21.15.1:0/1158907480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc146190 0x7fc0bc06a750 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc0731a0 tx=0x7fc0b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d332a8b5fdb53 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.458+0000 7fc0b9ffb700 1 -- 172.21.15.1:0/1158907480 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b000b870 con 0x7fc0bc146190 2024-03-18T18:49:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.458+0000 7fc0b9ffb700 1 -- 172.21.15.1:0/1158907480 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b000b9d0 con 0x7fc0bc146190 2024-03-18T18:49:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.458+0000 7fc0b9ffb700 1 -- 172.21.15.1:0/1158907480 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b000bd60 con 0x7fc0bc146190 2024-03-18T18:49:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.458+0000 7fc0c1982700 1 -- 172.21.15.1:0/1158907480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc146190 msgr2=0x7fc0bc06a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.458+0000 7fc0c1982700 1 --2- 172.21.15.1:0/1158907480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc146190 0x7fc0bc06a750 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc0731a0 tx=0x7fc0b0009bd0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.458+0000 7fc0c1982700 1 -- 172.21.15.1:0/1158907480 shutdown_connections 2024-03-18T18:49:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.458+0000 7fc0c1982700 1 --2- 172.21.15.1:0/1158907480 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0bc146af0 0x7fc0bc071650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.458+0000 7fc0c1982700 1 --2- 172.21.15.1:0/1158907480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc146190 0x7fc0bc06a750 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.458+0000 7fc0c1982700 1 --2- 172.21.15.1:0/1158907480 >> v2:172.21.15.165:3300/0 conn(0x7fc0bc1465a0 0x7fc0bc06ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.458+0000 7fc0c1982700 1 -- 172.21.15.1:0/1158907480 >> 172.21.15.1:0/1158907480 conn(0x7fc0bc0b1d20 msgr2=0x7fc0bc0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.458+0000 7fc0c1982700 1 -- 172.21.15.1:0/1158907480 shutdown_connections 2024-03-18T18:49:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.459+0000 7fc0c1982700 1 -- 172.21.15.1:0/1158907480 wait complete. 2024-03-18T18:49:41.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.460+0000 7fc0c1982700 1 Processor -- start 2024-03-18T18:49:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.460+0000 7fc0c1982700 1 -- start start 2024-03-18T18:49:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0c1982700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc1465a0 0x7fc0bc0ce460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0c1982700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0bc146af0 0x7fc0bc13ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0c1982700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0bc140080 0x7fc0bc0c50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0c1982700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc142810 con 0x7fc0bc1465a0 2024-03-18T18:49:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0c1982700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc0cf400 con 0x7fc0bc146af0 2024-03-18T18:49:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0c1982700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc13f1f0 con 0x7fc0bc140080 2024-03-18T18:49:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0baffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc1465a0 0x7fc0bc0ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0baffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc1465a0 0x7fc0bc0ce460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58030/0 (socket says 172.21.15.1:58030) 2024-03-18T18:49:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0baffd700 1 -- 172.21.15.1:0/1042719695 learned_addr learned my addr 172.21.15.1:0/1042719695 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0bb7fe700 1 --2- 172.21.15.1:0/1042719695 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0bc140080 0x7fc0bc0c50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0ba7fc700 1 --2- 172.21.15.1:0/1042719695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0bc146af0 0x7fc0bc13ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:41.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0baffd700 1 -- 172.21.15.1:0/1042719695 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0bc140080 msgr2=0x7fc0bc0c50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0baffd700 1 --2- 172.21.15.1:0/1042719695 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0bc140080 0x7fc0bc0c50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0baffd700 1 -- 172.21.15.1:0/1042719695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0bc146af0 msgr2=0x7fc0bc13ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0baffd700 1 --2- 172.21.15.1:0/1042719695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0bc146af0 0x7fc0bc13ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0baffd700 1 -- 172.21.15.1:0/1042719695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b0009550 con 0x7fc0bc1465a0 2024-03-18T18:49:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0baffd700 1 --2- 172.21.15.1:0/1042719695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc1465a0 0x7fc0bc0ce460 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0000c00 tx=0x7fc0b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0a3fff700 1 -- 172.21.15.1:0/1042719695 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0004310 con 0x7fc0bc1465a0 2024-03-18T18:49:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0a3fff700 1 -- 172.21.15.1:0/1042719695 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b001e070 con 0x7fc0bc1465a0 2024-03-18T18:49:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0c1982700 1 -- 172.21.15.1:0/1042719695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0bc0ceb60 con 0x7fc0bc1465a0 2024-03-18T18:49:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0a3fff700 1 -- 172.21.15.1:0/1042719695 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b00036a0 con 0x7fc0bc1465a0 2024-03-18T18:49:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.461+0000 7fc0c1982700 1 -- 172.21.15.1:0/1042719695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0bc0cf120 con 0x7fc0bc1465a0 2024-03-18T18:49:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.462+0000 7fc0c1982700 1 -- 172.21.15.1:0/1042719695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0bc0b4a10 con 0x7fc0bc1465a0 2024-03-18T18:49:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.463+0000 7fc0a3fff700 1 -- 172.21.15.1:0/1042719695 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7fc0b0020030 con 0x7fc0bc1465a0 2024-03-18T18:49:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.463+0000 7fc0a3fff700 1 --2- 172.21.15.1:0/1042719695 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc0a806ac30 0x7fc0a806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:41.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.463+0000 7fc0a3fff700 1 -- 172.21.15.1:0/1042719695 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc0b0089c60 con 0x7fc0bc1465a0 2024-03-18T18:49:41.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.463+0000 7fc0ba7fc700 1 --2- 172.21.15.1:0/1042719695 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc0a806ac30 0x7fc0a806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:41.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.464+0000 7fc0ba7fc700 1 --2- 172.21.15.1:0/1042719695 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc0a806ac30 0x7fc0a806d0f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac009950 tx=0x7fc0ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:41.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.468+0000 7fc0a3fff700 1 -- 172.21.15.1:0/1042719695 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc0b0055300 con 0x7fc0bc1465a0 2024-03-18T18:49:41.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.670+0000 7fc0c1982700 1 -- 172.21.15.1:0/1042719695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0bc073350 con 0x7fc0bc1465a0 2024-03-18T18:49:41.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.671+0000 7fc0a3fff700 1 -- 172.21.15.1:0/1042719695 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12347 (secure 0 0 0) 0x7fc0b0058920 con 0x7fc0bc1465a0 2024-03-18T18:49:41.667 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:49:41.667 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":16,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:41.245267+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:49:41.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.673+0000 7fc0c1982700 1 -- 172.21.15.1:0/1042719695 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc0a806ac30 msgr2=0x7fc0a806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.673+0000 7fc0c1982700 1 --2- 172.21.15.1:0/1042719695 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc0a806ac30 0x7fc0a806d0f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac009950 tx=0x7fc0ac008040 comp rx=0 tx=0).stop 2024-03-18T18:49:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.673+0000 7fc0c1982700 1 -- 172.21.15.1:0/1042719695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc1465a0 msgr2=0x7fc0bc0ce460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.673+0000 7fc0c1982700 1 --2- 172.21.15.1:0/1042719695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc1465a0 0x7fc0bc0ce460 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0000c00 tx=0x7fc0b0004750 comp rx=0 tx=0).stop 2024-03-18T18:49:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.673+0000 7fc0c1982700 1 -- 172.21.15.1:0/1042719695 shutdown_connections 2024-03-18T18:49:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.673+0000 7fc0c1982700 1 --2- 172.21.15.1:0/1042719695 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc0a806ac30 0x7fc0a806d0f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.673+0000 7fc0c1982700 1 --2- 172.21.15.1:0/1042719695 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0bc140080 0x7fc0bc0c50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.673+0000 7fc0c1982700 1 --2- 172.21.15.1:0/1042719695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0bc1465a0 0x7fc0bc0ce460 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.673+0000 7fc0c1982700 1 --2- 172.21.15.1:0/1042719695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0bc146af0 0x7fc0bc13ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.673+0000 7fc0c1982700 1 -- 172.21.15.1:0/1042719695 >> 172.21.15.1:0/1042719695 conn(0x7fc0bc0b1d20 msgr2=0x7fc0bc0acab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:41.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.673+0000 7fc0c1982700 1 -- 172.21.15.1:0/1042719695 shutdown_connections 2024-03-18T18:49:41.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.673+0000 7fc0c1982700 1 -- 172.21.15.1:0/1042719695 wait complete. 2024-03-18T18:49:41.681 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T18:49:41.682 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T18:49:41.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.818+0000 7f594ed26700 1 Processor -- start 2024-03-18T18:49:41.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.818+0000 7f594ed26700 1 -- start start 2024-03-18T18:49:41.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.819+0000 7f594ed26700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f594815d780 0x7f59480cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:41.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.819+0000 7f594ed26700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59480ce090 con 0x7f594809dd40 2024-03-18T18:49:41.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.819+0000 7f594ed26700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59480cf0e0 con 0x7f59480ce5a0 2024-03-18T18:49:41.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.819+0000 7f594ed26700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59480cb480 con 0x7f594815d780 2024-03-18T18:49:41.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.819+0000 7f594cac2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f594815d780 0x7f59480cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:41.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.819+0000 7f5947fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f594809dd40 0x7f59480cad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35584/0 (socket says 172.21.15.1:35584) 2024-03-18T18:49:41.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.819+0000 7f594cac2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f594815d780 0x7f59480cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58046/0 (socket says 172.21.15.1:58046) 2024-03-18T18:49:41.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.819+0000 7f5947fff700 1 -- 172.21.15.1:0/616844683 learned_addr learned my addr 172.21.15.1:0/616844683 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:49:41.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.819+0000 7f594cac2700 1 -- 172.21.15.1:0/616844683 >> v1:172.21.15.165:6789/0 conn(0x7f59480ce5a0 legacy=0x7f59480ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:41.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.819+0000 7f594cac2700 1 -- 172.21.15.1:0/616844683 >> v1:172.21.15.1:6789/0 conn(0x7f594809dd40 legacy=0x7f59480cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:41.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.819+0000 7f594cac2700 1 -- 172.21.15.1:0/616844683 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594806db30 con 0x7f594815d780 2024-03-18T18:49:41.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.819+0000 7f59477fe700 1 -- 172.21.15.1:0/616844683 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592859806 0 0) 0x7f59480ce090 con 0x7f594809dd40 2024-03-18T18:49:41.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.819+0000 7f594cac2700 1 --2- 172.21.15.1:0/616844683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f594815d780 0x7f59480cdb50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f593c009840 tx=0x7f593c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17b4076d79d7881e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:41.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.820+0000 7f59477fe700 1 -- 172.21.15.1:0/616844683 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c011830 con 0x7f594815d780 2024-03-18T18:49:41.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.820+0000 7f59477fe700 1 -- 172.21.15.1:0/616844683 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593c011990 con 0x7f594815d780 2024-03-18T18:49:41.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.820+0000 7f59477fe700 1 -- 172.21.15.1:0/616844683 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c011d20 con 0x7f594815d780 2024-03-18T18:49:41.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.820+0000 7f594ed26700 1 -- 172.21.15.1:0/616844683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f594815d780 msgr2=0x7f59480cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:41.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.820+0000 7f594ed26700 1 --2- 172.21.15.1:0/616844683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f594815d780 0x7f59480cdb50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f593c009840 tx=0x7f593c009b50 comp rx=0 tx=0).stop 2024-03-18T18:49:41.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.820+0000 7f594ed26700 1 -- 172.21.15.1:0/616844683 shutdown_connections 2024-03-18T18:49:41.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.820+0000 7f594ed26700 1 --2- 172.21.15.1:0/616844683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f594815d780 0x7f59480cdb50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.820+0000 7f594ed26700 1 -- 172.21.15.1:0/616844683 >> 172.21.15.1:0/616844683 conn(0x7f59480cca00 msgr2=0x7f59480cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:41.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.820+0000 7f594ed26700 1 -- 172.21.15.1:0/616844683 shutdown_connections 2024-03-18T18:49:41.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.821+0000 7f594ed26700 1 -- 172.21.15.1:0/616844683 wait complete. 2024-03-18T18:49:41.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.822+0000 7f594ed26700 1 Processor -- start 2024-03-18T18:49:41.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.822+0000 7f594ed26700 1 -- start start 2024-03-18T18:49:41.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f594ed26700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f594809dd40 0x7f5948153390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:41.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f594ed26700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59480ce5a0 0x7f59480c9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:41.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f594ed26700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f594815d780 0x7f594809ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:41.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f594ed26700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5948151d00 con 0x7f59480ce5a0 2024-03-18T18:49:41.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f594ed26700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5948099a40 con 0x7f594809dd40 2024-03-18T18:49:41.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f594ed26700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59480c9d60 con 0x7f594815d780 2024-03-18T18:49:41.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f5947fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59480ce5a0 0x7f59480c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:41.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f5947fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59480ce5a0 0x7f59480c9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58060/0 (socket says 172.21.15.1:58060) 2024-03-18T18:49:41.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f5947fff700 1 -- 172.21.15.1:0/744366991 learned_addr learned my addr 172.21.15.1:0/744366991 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:41.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f594cac2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f594809dd40 0x7f5948153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:41.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f594d2c3700 1 --2- 172.21.15.1:0/744366991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f594815d780 0x7f594809ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:41.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f5947fff700 1 -- 172.21.15.1:0/744366991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f594815d780 msgr2=0x7f594809ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:41.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f5947fff700 1 --2- 172.21.15.1:0/744366991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f594815d780 0x7f594809ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f5947fff700 1 -- 172.21.15.1:0/744366991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f594809dd40 msgr2=0x7f5948153390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:41.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f5947fff700 1 --2- 172.21.15.1:0/744366991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f594809dd40 0x7f5948153390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:41.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f5947fff700 1 -- 172.21.15.1:0/744366991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c0094a0 con 0x7f59480ce5a0 2024-03-18T18:49:41.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f594d2c3700 1 --2- 172.21.15.1:0/744366991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f594815d780 0x7f594809ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:41.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f5947fff700 1 --2- 172.21.15.1:0/744366991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59480ce5a0 0x7f59480c9660 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f59340011e0 tx=0x7f593400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:41.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f59457fa700 1 -- 172.21.15.1:0/744366991 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934004c80 con 0x7f59480ce5a0 2024-03-18T18:49:41.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f59457fa700 1 -- 172.21.15.1:0/744366991 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5934004de0 con 0x7f59480ce5a0 2024-03-18T18:49:41.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.823+0000 7f59457fa700 1 -- 172.21.15.1:0/744366991 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934007310 con 0x7f59480ce5a0 2024-03-18T18:49:41.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.824+0000 7f594ed26700 1 -- 172.21.15.1:0/744366991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5948099ce0 con 0x7f59480ce5a0 2024-03-18T18:49:41.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.824+0000 7f594ed26700 1 -- 172.21.15.1:0/744366991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594809a2a0 con 0x7f59480ce5a0 2024-03-18T18:49:41.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.825+0000 7f594ed26700 1 -- 172.21.15.1:0/744366991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59480cade0 con 0x7f59480ce5a0 2024-03-18T18:49:41.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.825+0000 7f59457fa700 1 -- 172.21.15.1:0/744366991 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7f5934007470 con 0x7f59480ce5a0 2024-03-18T18:49:41.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.825+0000 7f59457fa700 1 --2- 172.21.15.1:0/744366991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f592c06ac30 0x7f592c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:41.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.826+0000 7f59457fa700 1 -- 172.21.15.1:0/744366991 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5934088bc0 con 0x7f59480ce5a0 2024-03-18T18:49:41.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.826+0000 7f594cac2700 1 --2- 172.21.15.1:0/744366991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f592c06ac30 0x7f592c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:41.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.826+0000 7f594cac2700 1 --2- 172.21.15.1:0/744366991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f592c06ac30 0x7f592c06d0f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f593c0090b0 tx=0x7f593c01c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:41.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:41.831+0000 7f59457fa700 1 -- 172.21.15.1:0/744366991 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f59340541d0 con 0x7f59480ce5a0 2024-03-18T18:49:42.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.041+0000 7f594ed26700 1 -- 172.21.15.1:0/744366991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f594809a490 con 0x7f59480ce5a0 2024-03-18T18:49:42.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.041+0000 7f59457fa700 1 -- 172.21.15.1:0/744366991 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f59340577f0 con 0x7f59480ce5a0 2024-03-18T18:49:42.038 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T18:49:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.043+0000 7f594ed26700 1 -- 172.21.15.1:0/744366991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f592c06ac30 msgr2=0x7f592c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.043+0000 7f594ed26700 1 --2- 172.21.15.1:0/744366991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f592c06ac30 0x7f592c06d0f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f593c0090b0 tx=0x7f593c01c040 comp rx=0 tx=0).stop 2024-03-18T18:49:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.043+0000 7f594ed26700 1 -- 172.21.15.1:0/744366991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59480ce5a0 msgr2=0x7f59480c9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.043+0000 7f594ed26700 1 --2- 172.21.15.1:0/744366991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59480ce5a0 0x7f59480c9660 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f59340011e0 tx=0x7f593400ca30 comp rx=0 tx=0).stop 2024-03-18T18:49:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.043+0000 7f594ed26700 1 -- 172.21.15.1:0/744366991 shutdown_connections 2024-03-18T18:49:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.043+0000 7f594ed26700 1 --2- 172.21.15.1:0/744366991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f592c06ac30 0x7f592c06d0f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.043+0000 7f594ed26700 1 --2- 172.21.15.1:0/744366991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f594815d780 0x7f594809ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:42.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.043+0000 7f594ed26700 1 --2- 172.21.15.1:0/744366991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f594809dd40 0x7f5948153390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:42.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.043+0000 7f594ed26700 1 --2- 172.21.15.1:0/744366991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59480ce5a0 0x7f59480c9660 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:42.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.043+0000 7f594ed26700 1 -- 172.21.15.1:0/744366991 >> 172.21.15.1:0/744366991 conn(0x7f59480cca00 msgr2=0x7f59480c5ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:42.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.043+0000 7f594ed26700 1 -- 172.21.15.1:0/744366991 shutdown_connections 2024-03-18T18:49:42.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.044+0000 7f594ed26700 1 -- 172.21.15.1:0/744366991 wait complete. 2024-03-18T18:49:42.052 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T18:49:42.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.183+0000 7f1644586700 1 Processor -- start 2024-03-18T18:49:42.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.183+0000 7f1644586700 1 -- start start 2024-03-18T18:49:42.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.185+0000 7f1644586700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f163c0cc170 0x7f163c0ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:42.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.185+0000 7f1644586700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f163c06e270 con 0x7f163c15a2e0 2024-03-18T18:49:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.185+0000 7f1644586700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f163c0ae050 con 0x7f163c0ae600 2024-03-18T18:49:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.185+0000 7f1644586700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f163c06f690 con 0x7f163c0cc170 2024-03-18T18:49:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.185+0000 7f1641b21700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f163c15a2e0 0x7f163c06ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35598/0 (socket says 172.21.15.1:35598) 2024-03-18T18:49:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.185+0000 7f1641b21700 1 -- 172.21.15.1:0/4152932253 learned_addr learned my addr 172.21.15.1:0/4152932253 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:49:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.185+0000 7f1642b23700 1 --2- 172.21.15.1:0/4152932253 >> v2:172.21.15.165:3300/0 conn(0x7f163c0cc170 0x7f163c0ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.185+0000 7f1641320700 1 -- 172.21.15.1:0/4152932253 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461154395 0 0) 0x7f163c06e270 con 0x7f163c15a2e0 2024-03-18T18:49:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.185+0000 7f1641320700 1 -- 172.21.15.1:0/4152932253 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16280035b0 con 0x7f163c15a2e0 2024-03-18T18:49:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.186+0000 7f1642b23700 1 -- 172.21.15.1:0/4152932253 >> v1:172.21.15.165:6789/0 conn(0x7f163c0ae600 legacy=0x7f163c0ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.186+0000 7f1642b23700 1 -- 172.21.15.1:0/4152932253 >> v1:172.21.15.1:6789/0 conn(0x7f163c15a2e0 legacy=0x7f163c06ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.186+0000 7f1642b23700 1 -- 172.21.15.1:0/4152932253 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163c0add70 con 0x7f163c0cc170 2024-03-18T18:49:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.186+0000 7f1642b23700 1 --2- 172.21.15.1:0/4152932253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f163c0cc170 0x7f163c0ad790 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1634009d90 tx=0x7f163400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f031093185e4c552 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.186+0000 7f1641320700 1 -- 172.21.15.1:0/4152932253 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163400e070 con 0x7f163c0cc170 2024-03-18T18:49:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.186+0000 7f1641320700 1 -- 172.21.15.1:0/4152932253 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163400bb20 con 0x7f163c0cc170 2024-03-18T18:49:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.186+0000 7f1641320700 1 -- 172.21.15.1:0/4152932253 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1634012430 con 0x7f163c0cc170 2024-03-18T18:49:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.187+0000 7f1644586700 1 -- 172.21.15.1:0/4152932253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f163c0cc170 msgr2=0x7f163c0ad790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:42.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.187+0000 7f1644586700 1 --2- 172.21.15.1:0/4152932253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f163c0cc170 0x7f163c0ad790 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1634009d90 tx=0x7f163400b330 comp rx=0 tx=0).stop 2024-03-18T18:49:42.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.187+0000 7f1644586700 1 -- 172.21.15.1:0/4152932253 shutdown_connections 2024-03-18T18:49:42.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.187+0000 7f1644586700 1 --2- 172.21.15.1:0/4152932253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f163c0cc170 0x7f163c0ad790 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:42.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.187+0000 7f1644586700 1 -- 172.21.15.1:0/4152932253 >> 172.21.15.1:0/4152932253 conn(0x7f163c06b860 msgr2=0x7f163c06bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:42.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.187+0000 7f1644586700 1 -- 172.21.15.1:0/4152932253 shutdown_connections 2024-03-18T18:49:42.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.187+0000 7f1644586700 1 -- 172.21.15.1:0/4152932253 wait complete. 2024-03-18T18:49:42.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.190+0000 7f1644586700 1 Processor -- start 2024-03-18T18:49:42.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.190+0000 7f1644586700 1 -- start start 2024-03-18T18:49:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.191+0000 7f1644586700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f163c0ae600 0x7f163c150550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.191+0000 7f1644586700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0cc170 0x7f163c151950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.191+0000 7f1644586700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f163c15a2e0 0x7f163c14e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.191+0000 7f1644586700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163c153240 con 0x7f163c0cc170 2024-03-18T18:49:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.191+0000 7f1644586700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163c1521a0 con 0x7f163c15a2e0 2024-03-18T18:49:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.191+0000 7f1644586700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163c152480 con 0x7f163c0ae600 2024-03-18T18:49:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.192+0000 7f1641b21700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0cc170 0x7f163c151950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.192+0000 7f1641b21700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0cc170 0x7f163c151950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58086/0 (socket says 172.21.15.1:58086) 2024-03-18T18:49:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.192+0000 7f1641b21700 1 -- 172.21.15.1:0/80246458 learned_addr learned my addr 172.21.15.1:0/80246458 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.192+0000 7f1642b23700 1 --2- 172.21.15.1:0/80246458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f163c15a2e0 0x7f163c14e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:42.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.192+0000 7f1642322700 1 --2- 172.21.15.1:0/80246458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f163c0ae600 0x7f163c150550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:42.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.192+0000 7f1641b21700 1 -- 172.21.15.1:0/80246458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f163c0ae600 msgr2=0x7f163c150550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:42.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.192+0000 7f1641b21700 1 --2- 172.21.15.1:0/80246458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f163c0ae600 0x7f163c150550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:42.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.192+0000 7f1641b21700 1 -- 172.21.15.1:0/80246458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f163c15a2e0 msgr2=0x7f163c14e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:42.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.192+0000 7f1641b21700 1 --2- 172.21.15.1:0/80246458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f163c15a2e0 0x7f163c14e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:42.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.192+0000 7f1641b21700 1 -- 172.21.15.1:0/80246458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1634009a70 con 0x7f163c0cc170 2024-03-18T18:49:42.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.192+0000 7f1642322700 1 --2- 172.21.15.1:0/80246458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f163c0ae600 0x7f163c150550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:49:42.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.192+0000 7f1641b21700 1 --2- 172.21.15.1:0/80246458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0cc170 0x7f163c151950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f162c001eb0 tx=0x7f162c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:42.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.192+0000 7f1632ffd700 1 -- 172.21.15.1:0/80246458 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c005d70 con 0x7f163c0cc170 2024-03-18T18:49:42.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.192+0000 7f1632ffd700 1 -- 172.21.15.1:0/80246458 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c0070d0 con 0x7f163c0cc170 2024-03-18T18:49:42.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.192+0000 7f1644586700 1 -- 172.21.15.1:0/80246458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f163c152760 con 0x7f163c0cc170 2024-03-18T18:49:42.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.192+0000 7f1632ffd700 1 -- 172.21.15.1:0/80246458 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c007460 con 0x7f163c0cc170 2024-03-18T18:49:42.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.192+0000 7f1644586700 1 -- 172.21.15.1:0/80246458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f163c152d20 con 0x7f163c0cc170 2024-03-18T18:49:42.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.193+0000 7f1644586700 1 -- 172.21.15.1:0/80246458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f163c0676a0 con 0x7f163c0cc170 2024-03-18T18:49:42.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.194+0000 7f1632ffd700 1 -- 172.21.15.1:0/80246458 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7f162c0075c0 con 0x7f163c0cc170 2024-03-18T18:49:42.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.195+0000 7f1632ffd700 1 --2- 172.21.15.1:0/80246458 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f162006ac00 0x7f162006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:42.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.195+0000 7f1642322700 1 --2- 172.21.15.1:0/80246458 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f162006ac00 0x7f162006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:42.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.195+0000 7f1632ffd700 1 -- 172.21.15.1:0/80246458 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f162c088400 con 0x7f163c0cc170 2024-03-18T18:49:42.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.195+0000 7f1642322700 1 --2- 172.21.15.1:0/80246458 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f162006ac00 0x7f162006d0c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1638000f60 tx=0x7f1638009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:42.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.199+0000 7f1632ffd700 1 -- 172.21.15.1:0/80246458 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f162c053a10 con 0x7f163c0cc170 2024-03-18T18:49:42.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.409+0000 7f1644586700 1 -- 172.21.15.1:0/80246458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f163c06ffc0 con 0x7f163c0cc170 2024-03-18T18:49:42.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.410+0000 7f1632ffd700 1 -- 172.21.15.1:0/80246458 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7f162c017060 con 0x7f163c0cc170 2024-03-18T18:49:42.406 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:49:42.406 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T18:49:42.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:49:42.406 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-18T18:49:42.407 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-18T18:49:42.407 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:49:42.407 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T18:49:42.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.412+0000 7f1644586700 1 -- 172.21.15.1:0/80246458 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f162006ac00 msgr2=0x7f162006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:42.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.412+0000 7f1644586700 1 --2- 172.21.15.1:0/80246458 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f162006ac00 0x7f162006d0c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1638000f60 tx=0x7f1638009040 comp rx=0 tx=0).stop 2024-03-18T18:49:42.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.412+0000 7f1644586700 1 -- 172.21.15.1:0/80246458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0cc170 msgr2=0x7f163c151950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:42.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.412+0000 7f1644586700 1 --2- 172.21.15.1:0/80246458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0cc170 0x7f163c151950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f162c001eb0 tx=0x7f162c00c860 comp rx=0 tx=0).stop 2024-03-18T18:49:42.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.412+0000 7f1644586700 1 -- 172.21.15.1:0/80246458 shutdown_connections 2024-03-18T18:49:42.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.412+0000 7f1644586700 1 --2- 172.21.15.1:0/80246458 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f162006ac00 0x7f162006d0c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:42.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.412+0000 7f1644586700 1 --2- 172.21.15.1:0/80246458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f163c0ae600 0x7f163c150550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:42.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.412+0000 7f1644586700 1 --2- 172.21.15.1:0/80246458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f163c15a2e0 0x7f163c14e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:42.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.412+0000 7f1644586700 1 --2- 172.21.15.1:0/80246458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0cc170 0x7f163c151950 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:42.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.412+0000 7f1644586700 1 -- 172.21.15.1:0/80246458 >> 172.21.15.1:0/80246458 conn(0x7f163c06b860 msgr2=0x7f163c0ad3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:42.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.412+0000 7f1644586700 1 -- 172.21.15.1:0/80246458 shutdown_connections 2024-03-18T18:49:42.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.412+0000 7f1644586700 1 -- 172.21.15.1:0/80246458 wait complete. 2024-03-18T18:49:42.420 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T18:49:42.420 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T18:49:42.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.554+0000 7facd74e8700 1 Processor -- start 2024-03-18T18:49:42.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.554+0000 7facd74e8700 1 -- start start 2024-03-18T18:49:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.555+0000 7facd74e8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facd00b3a80 0x7facd015dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.555+0000 7facd74e8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7facd00b3ee0 0x7facd015e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.555+0000 7facd74e8700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd0154c30 con 0x7facd0154850 2024-03-18T18:49:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.555+0000 7facd74e8700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd0072ab0 con 0x7facd00b3ee0 2024-03-18T18:49:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.555+0000 7facd74e8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd015eb40 con 0x7facd00b3a80 2024-03-18T18:49:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.555+0000 7facd5a85700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7facd00b3ee0 0x7facd015e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:42.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.555+0000 7facd5a85700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7facd00b3ee0 0x7facd015e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58102/0 (socket says 172.21.15.1:58102) 2024-03-18T18:49:42.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.555+0000 7facd5a85700 1 -- 172.21.15.1:0/3777467303 learned_addr learned my addr 172.21.15.1:0/3777467303 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:42.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.555+0000 7facd4a83700 1 --2- 172.21.15.1:0/3777467303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facd00b3a80 0x7facd015dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:42.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.555+0000 7facd5a85700 1 -- 172.21.15.1:0/3777467303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facd00b3a80 msgr2=0x7facd015dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:42.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.555+0000 7facd5a85700 1 --2- 172.21.15.1:0/3777467303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facd00b3a80 0x7facd015dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:42.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.555+0000 7facd5a85700 1 -- 172.21.15.1:0/3777467303 >> v1:172.21.15.165:6789/0 conn(0x7facd0154850 legacy=0x7facd00723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:42.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.555+0000 7facd5a85700 1 -- 172.21.15.1:0/3777467303 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd00b49e0 con 0x7facd00b3ee0 2024-03-18T18:49:42.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.555+0000 7facd5a85700 1 --2- 172.21.15.1:0/3777467303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facd00b3ee0 0x7facd015e270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7facc8009e50 tx=0x7facc800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f6f339e9d8fbb45 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:42.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.555+0000 7facc7fff700 1 -- 172.21.15.1:0/3777467303 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc800e070 con 0x7facd00b3ee0 2024-03-18T18:49:42.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.555+0000 7facc7fff700 1 -- 172.21.15.1:0/3777467303 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc800b910 con 0x7facd00b3ee0 2024-03-18T18:49:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.555+0000 7facc7fff700 1 -- 172.21.15.1:0/3777467303 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc800bcc0 con 0x7facd00b3ee0 2024-03-18T18:49:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.556+0000 7facd74e8700 1 -- 172.21.15.1:0/3777467303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facd00b3ee0 msgr2=0x7facd015e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.556+0000 7facd74e8700 1 --2- 172.21.15.1:0/3777467303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facd00b3ee0 0x7facd015e270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7facc8009e50 tx=0x7facc800b330 comp rx=0 tx=0).stop 2024-03-18T18:49:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.556+0000 7facd74e8700 1 -- 172.21.15.1:0/3777467303 shutdown_connections 2024-03-18T18:49:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.556+0000 7facd74e8700 1 --2- 172.21.15.1:0/3777467303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facd00b3ee0 0x7facd015e270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.556+0000 7facd74e8700 1 --2- 172.21.15.1:0/3777467303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facd00b3a80 0x7facd015dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.556+0000 7facd74e8700 1 -- 172.21.15.1:0/3777467303 >> 172.21.15.1:0/3777467303 conn(0x7facd00d00a0 msgr2=0x7facd00d04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.556+0000 7facd74e8700 1 -- 172.21.15.1:0/3777467303 shutdown_connections 2024-03-18T18:49:42.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.556+0000 7facd74e8700 1 -- 172.21.15.1:0/3777467303 wait complete. 2024-03-18T18:49:42.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.558+0000 7facd74e8700 1 Processor -- start 2024-03-18T18:49:42.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.558+0000 7facd74e8700 1 -- start start 2024-03-18T18:49:42.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facd74e8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facd00b3a80 0x7facd00a8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:42.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facd74e8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facd00b3ee0 0x7facd0151a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:42.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facd74e8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facd0154850 0x7facd0151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:42.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facd74e8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd01524f0 con 0x7facd0154850 2024-03-18T18:49:42.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facd74e8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd0159120 con 0x7facd00b3a80 2024-03-18T18:49:42.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facd74e8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd01586e0 con 0x7facd00b3ee0 2024-03-18T18:49:42.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facd5a85700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facd0154850 0x7facd0151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:42.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facd5a85700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facd0154850 0x7facd0151fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58116/0 (socket says 172.21.15.1:58116) 2024-03-18T18:49:42.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facd5a85700 1 -- 172.21.15.1:0/3167179860 learned_addr learned my addr 172.21.15.1:0/3167179860 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:42.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facd4a83700 1 --2- 172.21.15.1:0/3167179860 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facd00b3ee0 0x7facd0151a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:42.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facd5284700 1 --2- 172.21.15.1:0/3167179860 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facd00b3a80 0x7facd00a8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:42.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facd5a85700 1 -- 172.21.15.1:0/3167179860 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facd00b3ee0 msgr2=0x7facd0151a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:42.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facd5a85700 1 --2- 172.21.15.1:0/3167179860 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facd00b3ee0 0x7facd0151a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:42.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facd5a85700 1 -- 172.21.15.1:0/3167179860 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facd00b3a80 msgr2=0x7facd00a8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:42.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facd5a85700 1 --2- 172.21.15.1:0/3167179860 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facd00b3a80 0x7facd00a8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:42.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facd5a85700 1 -- 172.21.15.1:0/3167179860 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc8009b30 con 0x7facd0154850 2024-03-18T18:49:42.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facd5a85700 1 --2- 172.21.15.1:0/3167179860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facd0154850 0x7facd0151fb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7facd0071480 tx=0x7facc8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:42.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facc5ffb700 1 -- 172.21.15.1:0/3167179860 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8003ea0 con 0x7facd0154850 2024-03-18T18:49:42.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facc5ffb700 1 -- 172.21.15.1:0/3167179860 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc80048e0 con 0x7facd0154850 2024-03-18T18:49:42.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facd74e8700 1 -- 172.21.15.1:0/3167179860 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd01593e0 con 0x7facd0154850 2024-03-18T18:49:42.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facc5ffb700 1 -- 172.21.15.1:0/3167179860 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc801a700 con 0x7facd0154850 2024-03-18T18:49:42.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.559+0000 7facd74e8700 1 -- 172.21.15.1:0/3167179860 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd01599a0 con 0x7facd0154850 2024-03-18T18:49:42.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.560+0000 7facd74e8700 1 -- 172.21.15.1:0/3167179860 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd00cd6a0 con 0x7facd0154850 2024-03-18T18:49:42.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.561+0000 7facc5ffb700 1 -- 172.21.15.1:0/3167179860 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7facc801a860 con 0x7facd0154850 2024-03-18T18:49:42.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.561+0000 7facc5ffb700 1 --2- 172.21.15.1:0/3167179860 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7facb406ac00 0x7facb406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:42.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.561+0000 7facc5ffb700 1 -- 172.21.15.1:0/3167179860 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7facc808c350 con 0x7facd0154850 2024-03-18T18:49:42.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.561+0000 7facd5284700 1 --2- 172.21.15.1:0/3167179860 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7facb406ac00 0x7facb406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:42.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.562+0000 7facd5284700 1 --2- 172.21.15.1:0/3167179860 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7facb406ac00 0x7facb406d0c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7facd00b1570 tx=0x7facc0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:42.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.566+0000 7facc5ffb700 1 -- 172.21.15.1:0/3167179860 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7facc8057950 con 0x7facd0154850 2024-03-18T18:49:42.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:42.770+0000 7facd74e8700 1 -- 172.21.15.1:0/3167179860 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7facd00b4610 con 0x7facd0154850 2024-03-18T18:49:43.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.369+0000 7facc5ffb700 1 -- 172.21.15.1:0/3167179860 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7facc805af70 con 0x7facd0154850 2024-03-18T18:49:43.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.374+0000 7facd74e8700 1 -- 172.21.15.1:0/3167179860 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7facb406ac00 msgr2=0x7facb406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:43.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.374+0000 7facd74e8700 1 --2- 172.21.15.1:0/3167179860 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7facb406ac00 0x7facb406d0c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7facd00b1570 tx=0x7facc0009040 comp rx=0 tx=0).stop 2024-03-18T18:49:43.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.374+0000 7facd74e8700 1 -- 172.21.15.1:0/3167179860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facd0154850 msgr2=0x7facd0151fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.374+0000 7facd74e8700 1 --2- 172.21.15.1:0/3167179860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facd0154850 0x7facd0151fb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7facd0071480 tx=0x7facc8004ad0 comp rx=0 tx=0).stop 2024-03-18T18:49:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.374+0000 7facd74e8700 1 -- 172.21.15.1:0/3167179860 shutdown_connections 2024-03-18T18:49:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.374+0000 7facd74e8700 1 --2- 172.21.15.1:0/3167179860 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7facb406ac00 0x7facb406d0c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.374+0000 7facd74e8700 1 --2- 172.21.15.1:0/3167179860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facd0154850 0x7facd0151fb0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.374+0000 7facd74e8700 1 --2- 172.21.15.1:0/3167179860 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facd00b3a80 0x7facd00a8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.374+0000 7facd74e8700 1 --2- 172.21.15.1:0/3167179860 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facd00b3ee0 0x7facd0151a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.374+0000 7facd74e8700 1 -- 172.21.15.1:0/3167179860 >> 172.21.15.1:0/3167179860 conn(0x7facd00d00a0 msgr2=0x7facd015bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.375+0000 7facd74e8700 1 -- 172.21.15.1:0/3167179860 shutdown_connections 2024-03-18T18:49:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.376+0000 7facd74e8700 1 -- 172.21.15.1:0/3167179860 wait complete. 2024-03-18T18:49:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-18T18:49:43.389 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-18T18:49:43.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.531+0000 7f7ce089f700 1 Processor -- start 2024-03-18T18:49:43.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.531+0000 7f7ce089f700 1 -- start start 2024-03-18T18:49:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.532+0000 7f7ce089f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7cdc15d3b0 0x7f7cdc06cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.532+0000 7f7ce089f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cdc071550 0x7f7cdc0b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.532+0000 7f7ce089f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cdc0b2b80 con 0x7f7cdc06d430 2024-03-18T18:49:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.532+0000 7f7ce089f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cdc0b3800 con 0x7f7cdc15d3b0 2024-03-18T18:49:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.532+0000 7f7ce089f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc0b2e60 con 0x7f7cdc071550 2024-03-18T18:49:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.533+0000 7f7cda59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7cdc15d3b0 0x7f7cdc06cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.533+0000 7f7cda59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7cdc15d3b0 0x7f7cdc06cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58136/0 (socket says 172.21.15.1:58136) 2024-03-18T18:49:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.533+0000 7f7cda59c700 1 -- 172.21.15.1:0/1435713314 learned_addr learned my addr 172.21.15.1:0/1435713314 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.533+0000 7f7cdad9d700 1 --2- 172.21.15.1:0/1435713314 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cdc071550 0x7f7cdc0b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.533+0000 7f7cda59c700 1 -- 172.21.15.1:0/1435713314 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cdc071550 msgr2=0x7f7cdc0b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.533+0000 7f7cda59c700 1 --2- 172.21.15.1:0/1435713314 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cdc071550 0x7f7cdc0b3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.533+0000 7f7cda59c700 1 -- 172.21.15.1:0/1435713314 >> v1:172.21.15.165:6789/0 conn(0x7f7cdc06d430 legacy=0x7f7cdc070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.533+0000 7f7cda59c700 1 -- 172.21.15.1:0/1435713314 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cdc0b3140 con 0x7f7cdc15d3b0 2024-03-18T18:49:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.533+0000 7f7cdad9d700 1 --2- 172.21.15.1:0/1435713314 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cdc071550 0x7f7cdc0b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:49:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.533+0000 7f7cda59c700 1 --2- 172.21.15.1:0/1435713314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cdc15d3b0 0x7f7cdc06cef0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7cd00098f0 tx=0x7f7cd0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25584e337b8b9efe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.533+0000 7f7cd959a700 1 -- 172.21.15.1:0/1435713314 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd0004230 con 0x7f7cdc15d3b0 2024-03-18T18:49:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.533+0000 7f7cd959a700 1 -- 172.21.15.1:0/1435713314 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd0004390 con 0x7f7cdc15d3b0 2024-03-18T18:49:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.533+0000 7f7cd959a700 1 -- 172.21.15.1:0/1435713314 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd00036a0 con 0x7f7cdc15d3b0 2024-03-18T18:49:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.534+0000 7f7ce089f700 1 -- 172.21.15.1:0/1435713314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cdc15d3b0 msgr2=0x7f7cdc06cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.534+0000 7f7ce089f700 1 --2- 172.21.15.1:0/1435713314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cdc15d3b0 0x7f7cdc06cef0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7cd00098f0 tx=0x7f7cd0009c00 comp rx=0 tx=0).stop 2024-03-18T18:49:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.534+0000 7f7ce089f700 1 -- 172.21.15.1:0/1435713314 shutdown_connections 2024-03-18T18:49:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.534+0000 7f7ce089f700 1 --2- 172.21.15.1:0/1435713314 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cdc071550 0x7f7cdc0b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.534+0000 7f7ce089f700 1 --2- 172.21.15.1:0/1435713314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cdc15d3b0 0x7f7cdc06cef0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.534+0000 7f7ce089f700 1 -- 172.21.15.1:0/1435713314 >> 172.21.15.1:0/1435713314 conn(0x7f7cdc0b1350 msgr2=0x7f7cdc0b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.534+0000 7f7ce089f700 1 -- 172.21.15.1:0/1435713314 shutdown_connections 2024-03-18T18:49:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.534+0000 7f7ce089f700 1 -- 172.21.15.1:0/1435713314 wait complete. 2024-03-18T18:49:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.535+0000 7f7ce089f700 1 Processor -- start 2024-03-18T18:49:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.535+0000 7f7ce089f700 1 -- start start 2024-03-18T18:49:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.536+0000 7f7ce089f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cdc06d430 0x7f7cdc156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.536+0000 7f7ce089f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cdc071550 0x7f7cdc0b0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.536+0000 7f7ce089f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7cdc15d3b0 0x7f7cdc144b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.536+0000 7f7ce089f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc156b40 con 0x7f7cdc071550 2024-03-18T18:49:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.536+0000 7f7ce089f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc157980 con 0x7f7cdc15d3b0 2024-03-18T18:49:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.536+0000 7f7ce089f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc156d70 con 0x7f7cdc06d430 2024-03-18T18:49:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.536+0000 7f7cd9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cdc071550 0x7f7cdc0b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.536+0000 7f7cd9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cdc071550 0x7f7cdc0b0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58142/0 (socket says 172.21.15.1:58142) 2024-03-18T18:49:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.536+0000 7f7cd9d9b700 1 -- 172.21.15.1:0/1965874508 learned_addr learned my addr 172.21.15.1:0/1965874508 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.536+0000 7f7cdad9d700 1 --2- 172.21.15.1:0/1965874508 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7cdc15d3b0 0x7f7cdc144b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.536+0000 7f7cda59c700 1 --2- 172.21.15.1:0/1965874508 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cdc06d430 0x7f7cdc156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.537+0000 7f7cd9d9b700 1 -- 172.21.15.1:0/1965874508 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cdc06d430 msgr2=0x7f7cdc156600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.537+0000 7f7cd9d9b700 1 --2- 172.21.15.1:0/1965874508 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cdc06d430 0x7f7cdc156600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.537+0000 7f7cd9d9b700 1 -- 172.21.15.1:0/1965874508 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7cdc15d3b0 msgr2=0x7f7cdc144b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.537+0000 7f7cd9d9b700 1 --2- 172.21.15.1:0/1965874508 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7cdc15d3b0 0x7f7cdc144b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.537+0000 7f7cd9d9b700 1 -- 172.21.15.1:0/1965874508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd0009550 con 0x7f7cdc071550 2024-03-18T18:49:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.537+0000 7f7cd9d9b700 1 --2- 172.21.15.1:0/1965874508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cdc071550 0x7f7cdc0b0880 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00c4d0 tx=0x7f7ccc00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.537+0000 7f7ccaffd700 1 -- 172.21.15.1:0/1965874508 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc011790 con 0x7f7cdc071550 2024-03-18T18:49:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.537+0000 7f7ccaffd700 1 -- 172.21.15.1:0/1965874508 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ccc0118f0 con 0x7f7cdc071550 2024-03-18T18:49:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.537+0000 7f7ce089f700 1 -- 172.21.15.1:0/1965874508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cdc157050 con 0x7f7cdc071550 2024-03-18T18:49:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.537+0000 7f7ccaffd700 1 -- 172.21.15.1:0/1965874508 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc011ca0 con 0x7f7cdc071550 2024-03-18T18:49:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.537+0000 7f7ce089f700 1 -- 172.21.15.1:0/1965874508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cdc157610 con 0x7f7cdc071550 2024-03-18T18:49:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.538+0000 7f7ce089f700 1 -- 172.21.15.1:0/1965874508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cdc157b30 con 0x7f7cdc071550 2024-03-18T18:49:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.538+0000 7f7ccaffd700 1 -- 172.21.15.1:0/1965874508 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7f7ccc01c460 con 0x7f7cdc071550 2024-03-18T18:49:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.539+0000 7f7ccaffd700 1 --2- 172.21.15.1:0/1965874508 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7cbc06ac00 0x7f7cbc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.539+0000 7f7ccaffd700 1 -- 172.21.15.1:0/1965874508 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f7ccc088cd0 con 0x7f7cdc071550 2024-03-18T18:49:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.539+0000 7f7cda59c700 1 --2- 172.21.15.1:0/1965874508 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7cbc06ac00 0x7f7cbc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.539+0000 7f7cda59c700 1 --2- 172.21.15.1:0/1965874508 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7cbc06ac00 0x7f7cbc06d0c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009130 tx=0x7f7cd0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.543+0000 7f7ccaffd700 1 -- 172.21.15.1:0/1965874508 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7cdc157b30 con 0x7f7cdc071550 2024-03-18T18:49:43.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:43.746+0000 7f7ce089f700 1 -- 172.21.15.1:0/1965874508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f7cdc157af0 con 0x7f7cdc071550 2024-03-18T18:49:44.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.383+0000 7f7ccaffd700 1 -- 172.21.15.1:0/1965874508 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7ccc054540 con 0x7f7cdc071550 2024-03-18T18:49:44.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.385+0000 7f7ccaffd700 1 -- 172.21.15.1:0/1965874508 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v19) v1 ==== 171+0+0 (secure 0 0 0) 0x7f7ccc086f90 con 0x7f7cdc071550 2024-03-18T18:49:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.391+0000 7f7ce089f700 1 -- 172.21.15.1:0/1965874508 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7cbc06ac00 msgr2=0x7f7cbc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.391+0000 7f7ce089f700 1 --2- 172.21.15.1:0/1965874508 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7cbc06ac00 0x7f7cbc06d0c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009130 tx=0x7f7cd0008030 comp rx=0 tx=0).stop 2024-03-18T18:49:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.391+0000 7f7ce089f700 1 -- 172.21.15.1:0/1965874508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cdc071550 msgr2=0x7f7cdc0b0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.391+0000 7f7ce089f700 1 --2- 172.21.15.1:0/1965874508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cdc071550 0x7f7cdc0b0880 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00c4d0 tx=0x7f7ccc00c7e0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.393+0000 7f7ce089f700 1 -- 172.21.15.1:0/1965874508 shutdown_connections 2024-03-18T18:49:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.393+0000 7f7ce089f700 1 --2- 172.21.15.1:0/1965874508 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7cbc06ac00 0x7f7cbc06d0c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.393+0000 7f7ce089f700 1 --2- 172.21.15.1:0/1965874508 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7cdc06d430 0x7f7cdc156600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.393+0000 7f7ce089f700 1 --2- 172.21.15.1:0/1965874508 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7cdc15d3b0 0x7f7cdc144b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.393+0000 7f7ce089f700 1 --2- 172.21.15.1:0/1965874508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cdc071550 0x7f7cdc0b0880 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.393+0000 7f7ce089f700 1 -- 172.21.15.1:0/1965874508 >> 172.21.15.1:0/1965874508 conn(0x7f7cdc0b1350 msgr2=0x7f7cdc160070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.394+0000 7f7ce089f700 1 -- 172.21.15.1:0/1965874508 shutdown_connections 2024-03-18T18:49:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.394+0000 7f7ce089f700 1 -- 172.21.15.1:0/1965874508 wait complete. 2024-03-18T18:49:44.396 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-18T18:49:44.410 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T18:49:44.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.543+0000 7fe5ebe7d700 1 Processor -- start 2024-03-18T18:49:44.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.544+0000 7fe5ebe7d700 1 -- start start 2024-03-18T18:49:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.544+0000 7fe5ebe7d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe5e4071d30 0x7fe5e40add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.544+0000 7fe5ebe7d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5e40cb9e0 0x7fe5e40cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.544+0000 7fe5ebe7d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e40ce8f0 con 0x7fe5e415a1d0 2024-03-18T18:49:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.544+0000 7fe5ebe7d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e40cdb00 con 0x7fe5e4071d30 2024-03-18T18:49:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.544+0000 7fe5ebe7d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e415a5b0 con 0x7fe5e40cb9e0 2024-03-18T18:49:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.544+0000 7fe5e9c19700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe5e415a1d0 0x7fe5e4071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58284/0 (socket says 172.21.15.1:58284) 2024-03-18T18:49:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.544+0000 7fe5e9c19700 1 -- 172.21.15.1:0/820613773 learned_addr learned my addr 172.21.15.1:0/820613773 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:49:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.545+0000 7fe5e9418700 1 --2- 172.21.15.1:0/820613773 >> v2:172.21.15.165:3300/0 conn(0x7fe5e4071d30 0x7fe5e40add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.545+0000 7fe5ea41a700 1 --2- 172.21.15.1:0/820613773 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5e40cb9e0 0x7fe5e40cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.545+0000 7fe5e8c17700 1 -- 172.21.15.1:0/820613773 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801167523 0 0) 0x7fe5e40ce8f0 con 0x7fe5e415a1d0 2024-03-18T18:49:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.545+0000 7fe5e8c17700 1 -- 172.21.15.1:0/820613773 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5d00035b0 con 0x7fe5e415a1d0 2024-03-18T18:49:44.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.545+0000 7fe5e9418700 1 -- 172.21.15.1:0/820613773 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5e40cb9e0 msgr2=0x7fe5e40cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.545+0000 7fe5e9418700 1 --2- 172.21.15.1:0/820613773 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5e40cb9e0 0x7fe5e40cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.545+0000 7fe5e9418700 1 -- 172.21.15.1:0/820613773 >> v1:172.21.15.1:6789/0 conn(0x7fe5e415a1d0 legacy=0x7fe5e4071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.545+0000 7fe5e9418700 1 -- 172.21.15.1:0/820613773 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e40ae260 con 0x7fe5e4071d30 2024-03-18T18:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.545+0000 7fe5e9418700 1 --2- 172.21.15.1:0/820613773 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5e4071d30 0x7fe5e40add20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0009840 tx=0x7fe5e0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=506b86f1c524310b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.545+0000 7fe5e8c17700 1 -- 172.21.15.1:0/820613773 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0011830 con 0x7fe5e4071d30 2024-03-18T18:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.545+0000 7fe5e8c17700 1 -- 172.21.15.1:0/820613773 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e0011990 con 0x7fe5e4071d30 2024-03-18T18:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.545+0000 7fe5e8c17700 1 -- 172.21.15.1:0/820613773 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0011d20 con 0x7fe5e4071d30 2024-03-18T18:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.546+0000 7fe5ebe7d700 1 -- 172.21.15.1:0/820613773 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5e4071d30 msgr2=0x7fe5e40add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.546+0000 7fe5ebe7d700 1 --2- 172.21.15.1:0/820613773 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5e4071d30 0x7fe5e40add20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0009840 tx=0x7fe5e0009b50 comp rx=0 tx=0).stop 2024-03-18T18:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.546+0000 7fe5ebe7d700 1 -- 172.21.15.1:0/820613773 shutdown_connections 2024-03-18T18:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.546+0000 7fe5ebe7d700 1 --2- 172.21.15.1:0/820613773 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5e40cb9e0 0x7fe5e40cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.546+0000 7fe5ebe7d700 1 --2- 172.21.15.1:0/820613773 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5e4071d30 0x7fe5e40add20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.546+0000 7fe5ebe7d700 1 -- 172.21.15.1:0/820613773 >> 172.21.15.1:0/820613773 conn(0x7fe5e40cc9b0 msgr2=0x7fe5e40ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.546+0000 7fe5ebe7d700 1 -- 172.21.15.1:0/820613773 shutdown_connections 2024-03-18T18:49:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.546+0000 7fe5ebe7d700 1 -- 172.21.15.1:0/820613773 wait complete. 2024-03-18T18:49:44.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.547+0000 7fe5ebe7d700 1 Processor -- start 2024-03-18T18:49:44.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.547+0000 7fe5ebe7d700 1 -- start start 2024-03-18T18:49:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.548+0000 7fe5ebe7d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5e4071d30 0x7fe5e41f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.548+0000 7fe5ebe7d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5e40cb9e0 0x7fe5e4158260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.548+0000 7fe5ebe7d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5e415a1d0 0x7fe5e41587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:44.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.548+0000 7fe5ebe7d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e40c7350 con 0x7fe5e4071d30 2024-03-18T18:49:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.548+0000 7fe5ebe7d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e40c7000 con 0x7fe5e415a1d0 2024-03-18T18:49:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.548+0000 7fe5ebe7d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e41f1430 con 0x7fe5e40cb9e0 2024-03-18T18:49:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.548+0000 7fe5e9c19700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5e4071d30 0x7fe5e41f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.548+0000 7fe5e9c19700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5e4071d30 0x7fe5e41f1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50968/0 (socket says 172.21.15.1:50968) 2024-03-18T18:49:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.548+0000 7fe5e9c19700 1 -- 172.21.15.1:0/2316127468 learned_addr learned my addr 172.21.15.1:0/2316127468 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.548+0000 7fe5e9418700 1 --2- 172.21.15.1:0/2316127468 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5e40cb9e0 0x7fe5e4158260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.548+0000 7fe5ea41a700 1 --2- 172.21.15.1:0/2316127468 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5e415a1d0 0x7fe5e41587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:44.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.548+0000 7fe5e9c19700 1 -- 172.21.15.1:0/2316127468 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5e40cb9e0 msgr2=0x7fe5e4158260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.548+0000 7fe5e9c19700 1 --2- 172.21.15.1:0/2316127468 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5e40cb9e0 0x7fe5e4158260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.548+0000 7fe5e9c19700 1 -- 172.21.15.1:0/2316127468 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5e415a1d0 msgr2=0x7fe5e41587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.548+0000 7fe5e9c19700 1 --2- 172.21.15.1:0/2316127468 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5e415a1d0 0x7fe5e41587a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.548+0000 7fe5e9c19700 1 -- 172.21.15.1:0/2316127468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e00094a0 con 0x7fe5e4071d30 2024-03-18T18:49:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.548+0000 7fe5e9c19700 1 --2- 172.21.15.1:0/2316127468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5e4071d30 0x7fe5e41f1a90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4001260 tx=0x7fe5d400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.549+0000 7fe5da7fc700 1 -- 172.21.15.1:0/2316127468 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d40074e0 con 0x7fe5e4071d30 2024-03-18T18:49:44.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.549+0000 7fe5da7fc700 1 -- 172.21.15.1:0/2316127468 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d4007640 con 0x7fe5e4071d30 2024-03-18T18:49:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.549+0000 7fe5ebe7d700 1 -- 172.21.15.1:0/2316127468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e40c6100 con 0x7fe5e4071d30 2024-03-18T18:49:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.549+0000 7fe5da7fc700 1 -- 172.21.15.1:0/2316127468 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4005a50 con 0x7fe5e4071d30 2024-03-18T18:49:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.549+0000 7fe5ebe7d700 1 -- 172.21.15.1:0/2316127468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e40c66c0 con 0x7fe5e4071d30 2024-03-18T18:49:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.549+0000 7fe5ebe7d700 1 -- 172.21.15.1:0/2316127468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e40676a0 con 0x7fe5e4071d30 2024-03-18T18:49:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.550+0000 7fe5da7fc700 1 -- 172.21.15.1:0/2316127468 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe5d4005bb0 con 0x7fe5e4071d30 2024-03-18T18:49:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.550+0000 7fe5da7fc700 1 --2- 172.21.15.1:0/2316127468 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe5c806ac90 0x7fe5c806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.551+0000 7fe5da7fc700 1 -- 172.21.15.1:0/2316127468 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fe5d4088690 con 0x7fe5e4071d30 2024-03-18T18:49:44.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.551+0000 7fe5e9418700 1 --2- 172.21.15.1:0/2316127468 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe5c806ac90 0x7fe5c806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:44.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.551+0000 7fe5e9418700 1 --2- 172.21.15.1:0/2316127468 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe5c806ac90 0x7fe5c806d150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0000c00 tx=0x7fe5e0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:44.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.555+0000 7fe5da7fc700 1 -- 172.21.15.1:0/2316127468 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe5d40538e0 con 0x7fe5e4071d30 2024-03-18T18:49:44.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.808+0000 7fe5ebe7d700 1 -- 172.21.15.1:0/2316127468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe5e40c6850 con 0x7fe5e4071d30 2024-03-18T18:49:44.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.813+0000 7fe5da7fc700 1 -- 172.21.15.1:0/2316127468 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fe5d4007e60 con 0x7fe5e4071d30 2024-03-18T18:49:44.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.823+0000 7fe5ebe7d700 1 -- 172.21.15.1:0/2316127468 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe5c806ac90 msgr2=0x7fe5c806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:44.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.823+0000 7fe5ebe7d700 1 --2- 172.21.15.1:0/2316127468 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe5c806ac90 0x7fe5c806d150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0000c00 tx=0x7fe5e0008030 comp rx=0 tx=0).stop 2024-03-18T18:49:44.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.823+0000 7fe5ebe7d700 1 -- 172.21.15.1:0/2316127468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5e4071d30 msgr2=0x7fe5e41f1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:44.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.823+0000 7fe5ebe7d700 1 --2- 172.21.15.1:0/2316127468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5e4071d30 0x7fe5e41f1a90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe5d4001260 tx=0x7fe5d400c7b0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.823+0000 7fe5ebe7d700 1 -- 172.21.15.1:0/2316127468 shutdown_connections 2024-03-18T18:49:44.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.823+0000 7fe5ebe7d700 1 --2- 172.21.15.1:0/2316127468 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe5c806ac90 0x7fe5c806d150 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0000c00 tx=0x7fe5e0008030 comp rx=0 tx=0).stop 2024-03-18T18:49:44.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.823+0000 7fe5ebe7d700 1 --2- 172.21.15.1:0/2316127468 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5e415a1d0 0x7fe5e41587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.823+0000 7fe5ebe7d700 1 --2- 172.21.15.1:0/2316127468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5e4071d30 0x7fe5e41f1a90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.823+0000 7fe5ebe7d700 1 --2- 172.21.15.1:0/2316127468 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5e40cb9e0 0x7fe5e4158260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.823+0000 7fe5ebe7d700 1 -- 172.21.15.1:0/2316127468 >> 172.21.15.1:0/2316127468 conn(0x7fe5e40cc9b0 msgr2=0x7fe5e40684d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:44.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.824+0000 7fe5ebe7d700 1 -- 172.21.15.1:0/2316127468 shutdown_connections 2024-03-18T18:49:44.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.825+0000 7fe5ebe7d700 1 -- 172.21.15.1:0/2316127468 wait complete. 2024-03-18T18:49:44.824 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 2 and data pool 3 2024-03-18T18:49:44.843 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-18T18:49:44.843 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-18T18:49:44.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.977+0000 7f38120e4700 1 Processor -- start 2024-03-18T18:49:44.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.977+0000 7f38120e4700 1 -- start start 2024-03-18T18:49:44.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.978+0000 7f38120e4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f380c0af0a0 0x7f380c071600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:44.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.978+0000 7f38120e4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f380c15c4b0 0x7f380c071b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:44.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.978+0000 7f38120e4700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380c071170 con 0x7f380c0aecc0 2024-03-18T18:49:44.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.978+0000 7f38120e4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f380c070e20 con 0x7f380c0af0a0 2024-03-18T18:49:44.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.978+0000 7f38120e4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f380c141360 con 0x7f380c15c4b0 2024-03-18T18:49:44.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.978+0000 7f380affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f380c0af0a0 0x7f380c071600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:44.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.978+0000 7f380affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f380c0af0a0 0x7f380c071600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50976/0 (socket says 172.21.15.1:50976) 2024-03-18T18:49:44.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.978+0000 7f380affd700 1 -- 172.21.15.1:0/401982909 learned_addr learned my addr 172.21.15.1:0/401982909 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:44.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.978+0000 7f380bfff700 1 --2- 172.21.15.1:0/401982909 >> v2:172.21.15.165:3300/0 conn(0x7f380c15c4b0 0x7f380c071b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:44.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.978+0000 7f380affd700 1 -- 172.21.15.1:0/401982909 >> v2:172.21.15.165:3300/0 conn(0x7f380c15c4b0 msgr2=0x7f380c071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:44.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.978+0000 7f380affd700 1 --2- 172.21.15.1:0/401982909 >> v2:172.21.15.165:3300/0 conn(0x7f380c15c4b0 0x7f380c071b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.978+0000 7f380affd700 1 -- 172.21.15.1:0/401982909 >> v1:172.21.15.165:6789/0 conn(0x7f380c0aecc0 legacy=0x7f380c140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:44.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.978+0000 7f380affd700 1 -- 172.21.15.1:0/401982909 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380c141640 con 0x7f380c0af0a0 2024-03-18T18:49:44.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.978+0000 7f380bfff700 1 --2- 172.21.15.1:0/401982909 >> v2:172.21.15.165:3300/0 conn(0x7f380c15c4b0 0x7f380c071b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:44.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.978+0000 7f380affd700 1 --2- 172.21.15.1:0/401982909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f380c0af0a0 0x7f380c071600 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f380c0741a0 tx=0x7f37f4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc6959af1f5c2dcf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:44.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.979+0000 7f380a7fc700 1 -- 172.21.15.1:0/401982909 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f400b870 con 0x7f380c0af0a0 2024-03-18T18:49:44.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.979+0000 7f380a7fc700 1 -- 172.21.15.1:0/401982909 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f400b9d0 con 0x7f380c0af0a0 2024-03-18T18:49:44.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.979+0000 7f380a7fc700 1 -- 172.21.15.1:0/401982909 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f400bd60 con 0x7f380c0af0a0 2024-03-18T18:49:44.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.979+0000 7f38120e4700 1 -- 172.21.15.1:0/401982909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f380c0af0a0 msgr2=0x7f380c071600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:44.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.979+0000 7f38120e4700 1 --2- 172.21.15.1:0/401982909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f380c0af0a0 0x7f380c071600 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f380c0741a0 tx=0x7f37f4009bd0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.979+0000 7f38120e4700 1 -- 172.21.15.1:0/401982909 shutdown_connections 2024-03-18T18:49:44.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.979+0000 7f38120e4700 1 --2- 172.21.15.1:0/401982909 >> v2:172.21.15.165:3300/0 conn(0x7f380c15c4b0 0x7f380c071b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.979+0000 7f38120e4700 1 --2- 172.21.15.1:0/401982909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f380c0af0a0 0x7f380c071600 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.979+0000 7f38120e4700 1 -- 172.21.15.1:0/401982909 >> 172.21.15.1:0/401982909 conn(0x7f380c06a750 msgr2=0x7f380c06ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:44.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.979+0000 7f38120e4700 1 -- 172.21.15.1:0/401982909 shutdown_connections 2024-03-18T18:49:44.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.979+0000 7f38120e4700 1 -- 172.21.15.1:0/401982909 wait complete. 2024-03-18T18:49:44.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.981+0000 7f38120e4700 1 Processor -- start 2024-03-18T18:49:44.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.981+0000 7f38120e4700 1 -- start start 2024-03-18T18:49:44.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.982+0000 7f38120e4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f380c0aecc0 0x7f380c143100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:44.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.982+0000 7f38120e4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f380c0af0a0 0x7f380c13fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:44.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.982+0000 7f38120e4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f380c15c4b0 0x7f380c1e8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:44.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.982+0000 7f38120e4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c140140 con 0x7f380c0af0a0 2024-03-18T18:49:44.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.982+0000 7f38120e4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c1e6780 con 0x7f380c15c4b0 2024-03-18T18:49:44.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.982+0000 7f38120e4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380c0ad9c0 con 0x7f380c0aecc0 2024-03-18T18:49:44.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.982+0000 7f380affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f380c0af0a0 0x7f380c13fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:44.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.982+0000 7f380affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f380c0af0a0 0x7f380c13fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50982/0 (socket says 172.21.15.1:50982) 2024-03-18T18:49:44.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.982+0000 7f380affd700 1 -- 172.21.15.1:0/3637440263 learned_addr learned my addr 172.21.15.1:0/3637440263 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:44.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.982+0000 7f380bfff700 1 --2- 172.21.15.1:0/3637440263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f380c15c4b0 0x7f380c1e8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:44.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.982+0000 7f380b7fe700 1 --2- 172.21.15.1:0/3637440263 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f380c0aecc0 0x7f380c143100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:44.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.982+0000 7f380affd700 1 -- 172.21.15.1:0/3637440263 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f380c0aecc0 msgr2=0x7f380c143100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:44.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.982+0000 7f380affd700 1 --2- 172.21.15.1:0/3637440263 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f380c0aecc0 0x7f380c143100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.982+0000 7f380affd700 1 -- 172.21.15.1:0/3637440263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f380c15c4b0 msgr2=0x7f380c1e8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:44.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.982+0000 7f380affd700 1 --2- 172.21.15.1:0/3637440263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f380c15c4b0 0x7f380c1e8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:44.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.982+0000 7f380affd700 1 -- 172.21.15.1:0/3637440263 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f4009550 con 0x7f380c0af0a0 2024-03-18T18:49:44.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.982+0000 7f380affd700 1 --2- 172.21.15.1:0/3637440263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f380c0af0a0 0x7f380c13fc00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f37f4000c00 tx=0x7f37f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:44.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.982+0000 7f3803fff700 1 -- 172.21.15.1:0/3637440263 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f4004540 con 0x7f380c0af0a0 2024-03-18T18:49:44.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.983+0000 7f3803fff700 1 -- 172.21.15.1:0/3637440263 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f40036a0 con 0x7f380c0af0a0 2024-03-18T18:49:44.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.983+0000 7f38120e4700 1 -- 172.21.15.1:0/3637440263 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f380c0adca0 con 0x7f380c0af0a0 2024-03-18T18:49:44.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.983+0000 7f3803fff700 1 -- 172.21.15.1:0/3637440263 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f4003a10 con 0x7f380c0af0a0 2024-03-18T18:49:44.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.983+0000 7f38120e4700 1 -- 172.21.15.1:0/3637440263 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f380c0ae260 con 0x7f380c0af0a0 2024-03-18T18:49:44.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.984+0000 7f38120e4700 1 -- 172.21.15.1:0/3637440263 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f380c1e6940 con 0x7f380c0af0a0 2024-03-18T18:49:44.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.984+0000 7f3803fff700 1 -- 172.21.15.1:0/3637440263 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f37f4021070 con 0x7f380c0af0a0 2024-03-18T18:49:44.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.984+0000 7f3803fff700 1 --2- 172.21.15.1:0/3637440263 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f37ec06ac90 0x7f37ec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:44.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.985+0000 7f3803fff700 1 -- 172.21.15.1:0/3637440263 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37f408a9b0 con 0x7f380c0af0a0 2024-03-18T18:49:44.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.985+0000 7f380b7fe700 1 --2- 172.21.15.1:0/3637440263 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f37ec06ac90 0x7f37ec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:44.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.985+0000 7f380b7fe700 1 --2- 172.21.15.1:0/3637440263 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f37ec06ac90 0x7f37ec06d150 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f37fc000f60 tx=0x7f37fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:44.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:44.989+0000 7f3803fff700 1 -- 172.21.15.1:0/3637440263 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f37f4055b80 con 0x7f380c0af0a0 2024-03-18T18:49:45.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.202+0000 7f38120e4700 1 -- 172.21.15.1:0/3637440263 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f380c0676a0 con 0x7f380c0af0a0 2024-03-18T18:49:45.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.817+0000 7f3803fff700 1 -- 172.21.15.1:0/3637440263 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v21) v1 ==== 157+0+0 (secure 0 0 0) 0x7f37f40290e0 con 0x7f380c0af0a0 2024-03-18T18:49:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.820+0000 7f38120e4700 1 -- 172.21.15.1:0/3637440263 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f37ec06ac90 msgr2=0x7f37ec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.820+0000 7f38120e4700 1 --2- 172.21.15.1:0/3637440263 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f37ec06ac90 0x7f37ec06d150 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f37fc000f60 tx=0x7f37fc009040 comp rx=0 tx=0).stop 2024-03-18T18:49:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.820+0000 7f38120e4700 1 -- 172.21.15.1:0/3637440263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f380c0af0a0 msgr2=0x7f380c13fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.820+0000 7f38120e4700 1 --2- 172.21.15.1:0/3637440263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f380c0af0a0 0x7f380c13fc00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f37f4000c00 tx=0x7f37f4004750 comp rx=0 tx=0).stop 2024-03-18T18:49:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.820+0000 7f38120e4700 1 -- 172.21.15.1:0/3637440263 shutdown_connections 2024-03-18T18:49:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.820+0000 7f38120e4700 1 --2- 172.21.15.1:0/3637440263 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f37ec06ac90 0x7f37ec06d150 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.820+0000 7f38120e4700 1 --2- 172.21.15.1:0/3637440263 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f380c0aecc0 0x7f380c143100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.820+0000 7f38120e4700 1 --2- 172.21.15.1:0/3637440263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f380c15c4b0 0x7f380c1e8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.820+0000 7f38120e4700 1 --2- 172.21.15.1:0/3637440263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f380c0af0a0 0x7f380c13fc00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:45.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.820+0000 7f38120e4700 1 -- 172.21.15.1:0/3637440263 >> 172.21.15.1:0/3637440263 conn(0x7f380c06a750 msgr2=0x7f380c0cc200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:45.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.821+0000 7f38120e4700 1 -- 172.21.15.1:0/3637440263 shutdown_connections 2024-03-18T18:49:45.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.821+0000 7f38120e4700 1 -- 172.21.15.1:0/3637440263 wait complete. 2024-03-18T18:49:45.832 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T18:49:45.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.976+0000 7fbf0fb77700 1 Processor -- start 2024-03-18T18:49:45.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.976+0000 7fbf0fb77700 1 -- start start 2024-03-18T18:49:45.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.977+0000 7fbf0fb77700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbf08150fd0 0x7fbf080b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:45.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.977+0000 7fbf0fb77700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf08071500 0x7fbf0806ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:45.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.977+0000 7fbf0fb77700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf0815b610 con 0x7fbf0815a620 2024-03-18T18:49:45.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.977+0000 7fbf0fb77700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf0815b050 con 0x7fbf08071500 2024-03-18T18:49:45.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.977+0000 7fbf0fb77700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf0815b330 con 0x7fbf08150fd0 2024-03-18T18:49:45.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.977+0000 7fbf0e114700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf08071500 0x7fbf0806ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:45.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.977+0000 7fbf0e114700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf08071500 0x7fbf0806ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50996/0 (socket says 172.21.15.1:50996) 2024-03-18T18:49:45.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.977+0000 7fbf0e114700 1 -- 172.21.15.1:0/2697302352 learned_addr learned my addr 172.21.15.1:0/2697302352 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:45.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.977+0000 7fbf0d913700 1 --2- 172.21.15.1:0/2697302352 >> v2:172.21.15.165:3300/0 conn(0x7fbf08150fd0 0x7fbf080b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:45.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.977+0000 7fbf0e114700 1 -- 172.21.15.1:0/2697302352 >> v2:172.21.15.165:3300/0 conn(0x7fbf08150fd0 msgr2=0x7fbf080b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:45.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.977+0000 7fbf0e114700 1 --2- 172.21.15.1:0/2697302352 >> v2:172.21.15.165:3300/0 conn(0x7fbf08150fd0 0x7fbf080b2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:45.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.977+0000 7fbf0e114700 1 -- 172.21.15.1:0/2697302352 >> v1:172.21.15.165:6789/0 conn(0x7fbf0815a620 legacy=0x7fbf08070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:45.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.977+0000 7fbf0e114700 1 -- 172.21.15.1:0/2697302352 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf080b2780 con 0x7fbf08071500 2024-03-18T18:49:45.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.977+0000 7fbf0d913700 1 --2- 172.21.15.1:0/2697302352 >> v2:172.21.15.165:3300/0 conn(0x7fbf08150fd0 0x7fbf080b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:49:45.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.978+0000 7fbf0e114700 1 --2- 172.21.15.1:0/2697302352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf08071500 0x7fbf0806ce30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbf080b2410 tx=0x7fbf0000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f7dcd8e7aaf0f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:45.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.978+0000 7fbf0c911700 1 -- 172.21.15.1:0/2697302352 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf00010070 con 0x7fbf08071500 2024-03-18T18:49:45.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.978+0000 7fbf0c911700 1 -- 172.21.15.1:0/2697302352 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0000dd50 con 0x7fbf08071500 2024-03-18T18:49:45.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.978+0000 7fbf0c911700 1 -- 172.21.15.1:0/2697302352 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0000b310 con 0x7fbf08071500 2024-03-18T18:49:45.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.978+0000 7fbf0fb77700 1 -- 172.21.15.1:0/2697302352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf08071500 msgr2=0x7fbf0806ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:45.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.978+0000 7fbf0fb77700 1 --2- 172.21.15.1:0/2697302352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf08071500 0x7fbf0806ce30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbf080b2410 tx=0x7fbf0000d430 comp rx=0 tx=0).stop 2024-03-18T18:49:45.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.978+0000 7fbf0fb77700 1 -- 172.21.15.1:0/2697302352 shutdown_connections 2024-03-18T18:49:45.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.978+0000 7fbf0fb77700 1 --2- 172.21.15.1:0/2697302352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf08071500 0x7fbf0806ce30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:45.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.978+0000 7fbf0fb77700 1 --2- 172.21.15.1:0/2697302352 >> v2:172.21.15.165:3300/0 conn(0x7fbf08150fd0 0x7fbf080b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.978+0000 7fbf0fb77700 1 -- 172.21.15.1:0/2697302352 >> 172.21.15.1:0/2697302352 conn(0x7fbf08066930 msgr2=0x7fbf0806a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.978+0000 7fbf0fb77700 1 -- 172.21.15.1:0/2697302352 shutdown_connections 2024-03-18T18:49:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.978+0000 7fbf0fb77700 1 -- 172.21.15.1:0/2697302352 wait complete. 2024-03-18T18:49:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.980+0000 7fbf0fb77700 1 Processor -- start 2024-03-18T18:49:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.980+0000 7fbf0fb77700 1 -- start start 2024-03-18T18:49:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.980+0000 7fbf0fb77700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf08071500 0x7fbf08159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.980+0000 7fbf0fb77700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf08150fd0 0x7fbf0814e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.980+0000 7fbf0fb77700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf0815a620 0x7fbf0814ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.980+0000 7fbf0fb77700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf081f33c0 con 0x7fbf08150fd0 2024-03-18T18:49:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.980+0000 7fbf0fb77700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0814f250 con 0x7fbf08071500 2024-03-18T18:49:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.980+0000 7fbf0fb77700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf081f4340 con 0x7fbf0815a620 2024-03-18T18:49:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.980+0000 7fbf0d112700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf08150fd0 0x7fbf0814e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.980+0000 7fbf0d112700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf08150fd0 0x7fbf0814e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51012/0 (socket says 172.21.15.1:51012) 2024-03-18T18:49:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.980+0000 7fbf0d112700 1 -- 172.21.15.1:0/1493657232 learned_addr learned my addr 172.21.15.1:0/1493657232 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.981+0000 7fbf0d913700 1 --2- 172.21.15.1:0/1493657232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf08071500 0x7fbf08159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.981+0000 7fbf0e114700 1 --2- 172.21.15.1:0/1493657232 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf0815a620 0x7fbf0814ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.981+0000 7fbf0d112700 1 -- 172.21.15.1:0/1493657232 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf0815a620 msgr2=0x7fbf0814ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.981+0000 7fbf0d112700 1 --2- 172.21.15.1:0/1493657232 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf0815a620 0x7fbf0814ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.981+0000 7fbf0d112700 1 -- 172.21.15.1:0/1493657232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf08071500 msgr2=0x7fbf08159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.981+0000 7fbf0d112700 1 --2- 172.21.15.1:0/1493657232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf08071500 0x7fbf08159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.981+0000 7fbf0d112700 1 -- 172.21.15.1:0/1493657232 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0000bd40 con 0x7fbf08150fd0 2024-03-18T18:49:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.981+0000 7fbf0e114700 1 --2- 172.21.15.1:0/1493657232 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf0815a620 0x7fbf0814ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:49:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.981+0000 7fbf0d112700 1 --2- 172.21.15.1:0/1493657232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf08150fd0 0x7fbf0814e7d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbf0400b4a0 tx=0x7fbf0400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.981+0000 7fbefe7fc700 1 -- 172.21.15.1:0/1493657232 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf04011830 con 0x7fbf08150fd0 2024-03-18T18:49:45.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.981+0000 7fbefe7fc700 1 -- 172.21.15.1:0/1493657232 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf04011990 con 0x7fbf08150fd0 2024-03-18T18:49:45.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.981+0000 7fbefe7fc700 1 -- 172.21.15.1:0/1493657232 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf04011d40 con 0x7fbf08150fd0 2024-03-18T18:49:45.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.981+0000 7fbf0fb77700 1 -- 172.21.15.1:0/1493657232 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf080b3640 con 0x7fbf08150fd0 2024-03-18T18:49:45.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.981+0000 7fbf0fb77700 1 -- 172.21.15.1:0/1493657232 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf081f4900 con 0x7fbf08150fd0 2024-03-18T18:49:45.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.982+0000 7fbf0fb77700 1 -- 172.21.15.1:0/1493657232 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf0806b6e0 con 0x7fbf08150fd0 2024-03-18T18:49:45.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.982+0000 7fbefe7fc700 1 -- 172.21.15.1:0/1493657232 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fbf0401a400 con 0x7fbf08150fd0 2024-03-18T18:49:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.983+0000 7fbefe7fc700 1 --2- 172.21.15.1:0/1493657232 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbeec06ac90 0x7fbeec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.983+0000 7fbf0d913700 1 --2- 172.21.15.1:0/1493657232 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbeec06ac90 0x7fbeec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.983+0000 7fbefe7fc700 1 -- 172.21.15.1:0/1493657232 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5935+0+0 (secure 0 0 0) 0x7fbf04089130 con 0x7fbf08150fd0 2024-03-18T18:49:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.983+0000 7fbf0d913700 1 --2- 172.21.15.1:0/1493657232 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbeec06ac90 0x7fbeec06d150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbef8006fd0 tx=0x7fbef8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:45.987+0000 7fbefe7fc700 1 -- 172.21.15.1:0/1493657232 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbf04054230 con 0x7fbf08150fd0 2024-03-18T18:49:46.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:46.192+0000 7fbf0fb77700 1 -- 172.21.15.1:0/1493657232 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fbf080676a0 con 0x7fbf08150fd0 2024-03-18T18:49:47.617 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:49:47.620 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:49:47.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:47.824+0000 7fbefe7fc700 1 -- 172.21.15.1:0/1493657232 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v23) v1 ==== 228+0+0 (secure 0 0 0) 0x7fbf04057850 con 0x7fbf08150fd0 2024-03-18T18:49:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:47.834+0000 7fbf0fb77700 1 -- 172.21.15.1:0/1493657232 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbeec06ac90 msgr2=0x7fbeec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:47.834+0000 7fbf0fb77700 1 --2- 172.21.15.1:0/1493657232 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbeec06ac90 0x7fbeec06d150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbef8006fd0 tx=0x7fbef8008040 comp rx=0 tx=0).stop 2024-03-18T18:49:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:47.834+0000 7fbf0fb77700 1 -- 172.21.15.1:0/1493657232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf08150fd0 msgr2=0x7fbf0814e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:47.834+0000 7fbf0fb77700 1 --2- 172.21.15.1:0/1493657232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf08150fd0 0x7fbf0814e7d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbf0400b4a0 tx=0x7fbf0400b860 comp rx=0 tx=0).stop 2024-03-18T18:49:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:47.835+0000 7fbf0fb77700 1 -- 172.21.15.1:0/1493657232 shutdown_connections 2024-03-18T18:49:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:47.835+0000 7fbf0fb77700 1 --2- 172.21.15.1:0/1493657232 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbeec06ac90 0x7fbeec06d150 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:47.835+0000 7fbf0fb77700 1 --2- 172.21.15.1:0/1493657232 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf0815a620 0x7fbf0814ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:47.835+0000 7fbf0fb77700 1 --2- 172.21.15.1:0/1493657232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf08071500 0x7fbf08159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:47.835+0000 7fbf0fb77700 1 --2- 172.21.15.1:0/1493657232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf08150fd0 0x7fbf0814e7d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:47.835+0000 7fbf0fb77700 1 -- 172.21.15.1:0/1493657232 >> 172.21.15.1:0/1493657232 conn(0x7fbf08066930 msgr2=0x7fbf0814f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:47.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:47.835+0000 7fbf0fb77700 1 -- 172.21.15.1:0/1493657232 shutdown_connections 2024-03-18T18:49:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:47.835+0000 7fbf0fb77700 1 -- 172.21.15.1:0/1493657232 wait complete. 2024-03-18T18:49:47.834 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data_ec' created 2024-03-18T18:49:47.848 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-18T18:49:48.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.031+0000 7f4fe19bb700 1 Processor -- start 2024-03-18T18:49:48.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.031+0000 7f4fe19bb700 1 -- start start 2024-03-18T18:49:48.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.031+0000 7f4fe19bb700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4fdc15d8b0 0x7f4fdc0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:48.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.031+0000 7f4fe19bb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fdc0d0040 0x7f4fdc0cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:48.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.031+0000 7f4fe19bb700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fdc15dc90 con 0x7f4fdc0d1170 2024-03-18T18:49:48.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.031+0000 7f4fe19bb700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fdc15ec90 con 0x7f4fdc15d8b0 2024-03-18T18:49:48.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.031+0000 7f4fe19bb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc15ded0 con 0x7f4fdc0d0040 2024-03-18T18:49:48.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.032+0000 7f4fda7fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4fdc0d1170 0x7f4fdc0cf930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58292/0 (socket says 172.21.15.1:58292) 2024-03-18T18:49:48.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.032+0000 7f4fda7fc700 1 -- 172.21.15.1:0/1535241662 learned_addr learned my addr 172.21.15.1:0/1535241662 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:49:48.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.032+0000 7f4fdaffd700 1 --2- 172.21.15.1:0/1535241662 >> v2:172.21.15.165:3300/0 conn(0x7f4fdc15d8b0 0x7f4fdc0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:48.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.032+0000 7f4fdb7fe700 1 --2- 172.21.15.1:0/1535241662 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fdc0d0040 0x7f4fdc0cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:48.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.032+0000 7f4fd9ffb700 1 -- 172.21.15.1:0/1535241662 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743226095 0 0) 0x7f4fd0001110 con 0x7f4fdc0d1170 2024-03-18T18:49:48.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.032+0000 7f4fd9ffb700 1 -- 172.21.15.1:0/1535241662 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fc40035b0 con 0x7f4fdc0d1170 2024-03-18T18:49:48.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.032+0000 7f4fd9ffb700 1 -- 172.21.15.1:0/1535241662 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 77052250 0 0) 0x7f4fc40035b0 con 0x7f4fdc0d1170 2024-03-18T18:49:48.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.032+0000 7f4fdaffd700 1 -- 172.21.15.1:0/1535241662 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fdc0d0040 msgr2=0x7f4fdc0cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:48.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.032+0000 7f4fdaffd700 1 --2- 172.21.15.1:0/1535241662 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fdc0d0040 0x7f4fdc0cde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:48.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.032+0000 7f4fdaffd700 1 -- 172.21.15.1:0/1535241662 >> v1:172.21.15.1:6789/0 conn(0x7f4fdc0d1170 legacy=0x7f4fdc0cf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:48.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.032+0000 7f4fdaffd700 1 -- 172.21.15.1:0/1535241662 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc15e1b0 con 0x7f4fdc15d8b0 2024-03-18T18:49:48.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.032+0000 7f4fdb7fe700 1 --2- 172.21.15.1:0/1535241662 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fdc0d0040 0x7f4fdc0cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:49:48.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.032+0000 7f4fdaffd700 1 --2- 172.21.15.1:0/1535241662 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4fdc15d8b0 0x7f4fdc0d3c30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009840 tx=0x7f4fcc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8069d7a7c7f55c2e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:48.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.033+0000 7f4fd9ffb700 1 -- 172.21.15.1:0/1535241662 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0001800 con 0x7f4fdc15d8b0 2024-03-18T18:49:48.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.033+0000 7f4fd9ffb700 1 -- 172.21.15.1:0/1535241662 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fcc004030 con 0x7f4fdc15d8b0 2024-03-18T18:49:48.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.033+0000 7f4fd9ffb700 1 -- 172.21.15.1:0/1535241662 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc0043c0 con 0x7f4fdc15d8b0 2024-03-18T18:49:48.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.033+0000 7f4fe19bb700 1 -- 172.21.15.1:0/1535241662 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4fdc15d8b0 msgr2=0x7f4fdc0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:48.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.033+0000 7f4fe19bb700 1 --2- 172.21.15.1:0/1535241662 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4fdc15d8b0 0x7f4fdc0d3c30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009840 tx=0x7f4fcc009b50 comp rx=0 tx=0).stop 2024-03-18T18:49:48.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.033+0000 7f4fe19bb700 1 -- 172.21.15.1:0/1535241662 shutdown_connections 2024-03-18T18:49:48.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.033+0000 7f4fe19bb700 1 --2- 172.21.15.1:0/1535241662 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4fdc15d8b0 0x7f4fdc0d3c30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:48.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.033+0000 7f4fe19bb700 1 --2- 172.21.15.1:0/1535241662 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fdc0d0040 0x7f4fdc0cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:48.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.033+0000 7f4fe19bb700 1 -- 172.21.15.1:0/1535241662 >> 172.21.15.1:0/1535241662 conn(0x7f4fdc09c640 msgr2=0x7f4fdc0d3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:48.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.033+0000 7f4fe19bb700 1 -- 172.21.15.1:0/1535241662 shutdown_connections 2024-03-18T18:49:48.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.033+0000 7f4fe19bb700 1 -- 172.21.15.1:0/1535241662 wait complete. 2024-03-18T18:49:48.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.034+0000 7f4fe19bb700 1 Processor -- start 2024-03-18T18:49:48.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.035+0000 7f4fe19bb700 1 -- start start 2024-03-18T18:49:48.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.035+0000 7f4fe19bb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fdc0d0040 0x7f4fdc1f43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:48.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.035+0000 7f4fe19bb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fdc0d1170 0x7f4fdc1533c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:48.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.035+0000 7f4fe19bb700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4fdc15d8b0 0x7f4fdc0cc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:48.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.035+0000 7f4fe19bb700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc161600 con 0x7f4fdc0d0040 2024-03-18T18:49:48.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.035+0000 7f4fe19bb700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc161320 con 0x7f4fdc15d8b0 2024-03-18T18:49:48.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.035+0000 7f4fe19bb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc0ca570 con 0x7f4fdc0d1170 2024-03-18T18:49:48.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.035+0000 7f4fdaffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fdc0d0040 0x7f4fdc1f43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:48.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.035+0000 7f4fdaffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fdc0d0040 0x7f4fdc1f43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51016/0 (socket says 172.21.15.1:51016) 2024-03-18T18:49:48.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.035+0000 7f4fdaffd700 1 -- 172.21.15.1:0/277275299 learned_addr learned my addr 172.21.15.1:0/277275299 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:48.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.036+0000 7f4fda7fc700 1 --2- 172.21.15.1:0/277275299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fdc0d1170 0x7f4fdc1533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:48.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.036+0000 7f4fdb7fe700 1 --2- 172.21.15.1:0/277275299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4fdc15d8b0 0x7f4fdc0cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:48.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.036+0000 7f4fdaffd700 1 -- 172.21.15.1:0/277275299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fdc0d1170 msgr2=0x7f4fdc1533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:48.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.036+0000 7f4fdaffd700 1 --2- 172.21.15.1:0/277275299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fdc0d1170 0x7f4fdc1533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:48.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.036+0000 7f4fdaffd700 1 -- 172.21.15.1:0/277275299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4fdc15d8b0 msgr2=0x7f4fdc0cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:48.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.036+0000 7f4fdaffd700 1 --2- 172.21.15.1:0/277275299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4fdc15d8b0 0x7f4fdc0cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:48.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.036+0000 7f4fdaffd700 1 -- 172.21.15.1:0/277275299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd4009a70 con 0x7f4fdc0d0040 2024-03-18T18:49:48.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.036+0000 7f4fdaffd700 1 --2- 172.21.15.1:0/277275299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fdc0d0040 0x7f4fdc1f43f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc00a010 tx=0x7f4fcc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:48.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.036+0000 7f4fc3fff700 1 -- 172.21.15.1:0/277275299 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc003ad0 con 0x7f4fdc0d0040 2024-03-18T18:49:48.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.036+0000 7f4fc3fff700 1 -- 172.21.15.1:0/277275299 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fcc01f070 con 0x7f4fdc0d0040 2024-03-18T18:49:48.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.036+0000 7f4fe19bb700 1 -- 172.21.15.1:0/277275299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fcc0094a0 con 0x7f4fdc0d0040 2024-03-18T18:49:48.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.036+0000 7f4fc3fff700 1 -- 172.21.15.1:0/277275299 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc003e90 con 0x7f4fdc0d0040 2024-03-18T18:49:48.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.036+0000 7f4fe19bb700 1 -- 172.21.15.1:0/277275299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fdc0cab30 con 0x7f4fdc0d0040 2024-03-18T18:49:48.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.037+0000 7f4fe19bb700 1 -- 172.21.15.1:0/277275299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fdc0ca3b0 con 0x7f4fdc0d0040 2024-03-18T18:49:48.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.038+0000 7f4fc3fff700 1 -- 172.21.15.1:0/277275299 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4fcc021030 con 0x7f4fdc0d0040 2024-03-18T18:49:48.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.038+0000 7f4fc3fff700 1 --2- 172.21.15.1:0/277275299 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4fc806ac90 0x7f4fc806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:48.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.038+0000 7f4fc3fff700 1 -- 172.21.15.1:0/277275299 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6428+0+0 (secure 0 0 0) 0x7f4fcc08a760 con 0x7f4fdc0d0040 2024-03-18T18:49:48.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.038+0000 7f4fda7fc700 1 --2- 172.21.15.1:0/277275299 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4fc806ac90 0x7f4fc806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:48.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.039+0000 7f4fda7fc700 1 --2- 172.21.15.1:0/277275299 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4fc806ac90 0x7f4fc806d150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc15a2b0 tx=0x7f4fd00013c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:48.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.043+0000 7f4fc3fff700 1 -- 172.21.15.1:0/277275299 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4fcc055700 con 0x7f4fdc0d0040 2024-03-18T18:49:48.239 INFO:tasks.ceph.mds.a.smithi001.stderr:2024-03-18T18:49:48.243+0000 7f460b344700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T18:49:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.245+0000 7f4fe19bb700 1 -- 172.21.15.1:0/277275299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f4fdc0d1590 con 0x7f4fdc0d0040 2024-03-18T18:49:48.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.835+0000 7f4fc3fff700 1 -- 172.21.15.1:0/277275299 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v24) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4fcc058d20 con 0x7f4fdc0d0040 2024-03-18T18:49:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.842+0000 7f4fe19bb700 1 -- 172.21.15.1:0/277275299 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4fc806ac90 msgr2=0x7f4fc806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.842+0000 7f4fe19bb700 1 --2- 172.21.15.1:0/277275299 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4fc806ac90 0x7f4fc806d150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc15a2b0 tx=0x7f4fd00013c0 comp rx=0 tx=0).stop 2024-03-18T18:49:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.842+0000 7f4fe19bb700 1 -- 172.21.15.1:0/277275299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fdc0d0040 msgr2=0x7f4fdc1f43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.842+0000 7f4fe19bb700 1 --2- 172.21.15.1:0/277275299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fdc0d0040 0x7f4fdc1f43f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc00a010 tx=0x7f4fcc004750 comp rx=0 tx=0).stop 2024-03-18T18:49:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.842+0000 7f4fe19bb700 1 -- 172.21.15.1:0/277275299 shutdown_connections 2024-03-18T18:49:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.842+0000 7f4fe19bb700 1 --2- 172.21.15.1:0/277275299 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4fc806ac90 0x7f4fc806d150 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.842+0000 7f4fe19bb700 1 --2- 172.21.15.1:0/277275299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4fdc15d8b0 0x7f4fdc0cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.842+0000 7f4fe19bb700 1 --2- 172.21.15.1:0/277275299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fdc0d0040 0x7f4fdc1f43f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.842+0000 7f4fe19bb700 1 --2- 172.21.15.1:0/277275299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fdc0d1170 0x7f4fdc1533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.842+0000 7f4fe19bb700 1 -- 172.21.15.1:0/277275299 >> 172.21.15.1:0/277275299 conn(0x7f4fdc09c640 msgr2=0x7f4fdc06cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:48.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.845+0000 7f4fe19bb700 1 -- 172.21.15.1:0/277275299 shutdown_connections 2024-03-18T18:49:48.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:48.846+0000 7f4fe19bb700 1 -- 172.21.15.1:0/277275299 wait complete. 2024-03-18T18:49:48.848 INFO:teuthology.orchestra.run.smithi001.stderr:set pool 4 allow_ec_overwrites to true 2024-03-18T18:49:48.866 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-18T18:49:49.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.006+0000 7f7814faf700 1 Processor -- start 2024-03-18T18:49:49.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.006+0000 7f7814faf700 1 -- start start 2024-03-18T18:49:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.007+0000 7f7814faf700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f781015a1d0 0x7f78100add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.007+0000 7f7814faf700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78100ce8f0 con 0x7f78100cc0f0 2024-03-18T18:49:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.007+0000 7f7814faf700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78100ae260 con 0x7f7810071620 2024-03-18T18:49:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.007+0000 7f7814faf700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78100cdb00 con 0x7f781015a1d0 2024-03-18T18:49:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.007+0000 7f780e59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f781015a1d0 0x7f78100add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.007+0000 7f780ed9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f78100cc0f0 0x7f781015ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58300/0 (socket says 172.21.15.1:58300) 2024-03-18T18:49:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.007+0000 7f780ed9d700 1 -- 172.21.15.1:0/3076555147 learned_addr learned my addr 172.21.15.1:0/3076555147 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:49:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.008+0000 7f780e59c700 1 -- 172.21.15.1:0/3076555147 >> v1:172.21.15.165:6789/0 conn(0x7f7810071620 legacy=0x7f78100cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.008+0000 7f780e59c700 1 -- 172.21.15.1:0/3076555147 >> v1:172.21.15.1:6789/0 conn(0x7f78100cc0f0 legacy=0x7f781015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:49.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.008+0000 7f780d59a700 1 -- 172.21.15.1:0/3076555147 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714598021 0 0) 0x7f78100ce8f0 con 0x7f78100cc0f0 2024-03-18T18:49:49.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.008+0000 7f780e59c700 1 -- 172.21.15.1:0/3076555147 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78100cdce0 con 0x7f781015a1d0 2024-03-18T18:49:49.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.008+0000 7f780e59c700 1 --2- 172.21.15.1:0/3076555147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f781015a1d0 0x7f78100add20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7800009820 tx=0x7f7800009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93438fbb24169fa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:49.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.008+0000 7f780d59a700 1 -- 172.21.15.1:0/3076555147 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7800005e30 con 0x7f781015a1d0 2024-03-18T18:49:49.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.008+0000 7f780d59a700 1 -- 172.21.15.1:0/3076555147 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7800004030 con 0x7f781015a1d0 2024-03-18T18:49:49.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.008+0000 7f780d59a700 1 -- 172.21.15.1:0/3076555147 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78000043c0 con 0x7f781015a1d0 2024-03-18T18:49:49.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.008+0000 7f7814faf700 1 -- 172.21.15.1:0/3076555147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f781015a1d0 msgr2=0x7f78100add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:49.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.008+0000 7f7814faf700 1 --2- 172.21.15.1:0/3076555147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f781015a1d0 0x7f78100add20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7800009820 tx=0x7f7800009b30 comp rx=0 tx=0).stop 2024-03-18T18:49:49.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.009+0000 7f7814faf700 1 -- 172.21.15.1:0/3076555147 shutdown_connections 2024-03-18T18:49:49.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.009+0000 7f7814faf700 1 --2- 172.21.15.1:0/3076555147 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f781015a1d0 0x7f78100add20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.009+0000 7f7814faf700 1 -- 172.21.15.1:0/3076555147 >> 172.21.15.1:0/3076555147 conn(0x7f78100cc9b0 msgr2=0x7f78100ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:49.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.009+0000 7f7814faf700 1 -- 172.21.15.1:0/3076555147 shutdown_connections 2024-03-18T18:49:49.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.009+0000 7f7814faf700 1 -- 172.21.15.1:0/3076555147 wait complete. 2024-03-18T18:49:49.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.010+0000 7f7814faf700 1 Processor -- start 2024-03-18T18:49:49.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.010+0000 7f7814faf700 1 -- start start 2024-03-18T18:49:49.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f7814faf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7810071620 0x7f78100b1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:49.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f7814faf700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f78100cc0f0 0x7f78100c9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:49.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f7814faf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f781015a1d0 0x7f78100c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:49.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f7814faf700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781015a5b0 con 0x7f7810071620 2024-03-18T18:49:49.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f7814faf700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78100c72c0 con 0x7f78100cc0f0 2024-03-18T18:49:49.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f7814faf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f781015a7c0 con 0x7f781015a1d0 2024-03-18T18:49:49.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f780e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7810071620 0x7f78100b1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:49.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f780e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7810071620 0x7f78100b1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51030/0 (socket says 172.21.15.1:51030) 2024-03-18T18:49:49.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f780e59c700 1 -- 172.21.15.1:0/3791689442 learned_addr learned my addr 172.21.15.1:0/3791689442 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:49.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f780dd9b700 1 --2- 172.21.15.1:0/3791689442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f78100cc0f0 0x7f78100c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:49.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f780ed9d700 1 --2- 172.21.15.1:0/3791689442 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f781015a1d0 0x7f78100c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:49.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f780e59c700 1 -- 172.21.15.1:0/3791689442 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f781015a1d0 msgr2=0x7f78100c5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:49.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f780e59c700 1 --2- 172.21.15.1:0/3791689442 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f781015a1d0 0x7f78100c5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f780e59c700 1 -- 172.21.15.1:0/3791689442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f78100cc0f0 msgr2=0x7f78100c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:49.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f780e59c700 1 --2- 172.21.15.1:0/3791689442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f78100cc0f0 0x7f78100c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f780e59c700 1 -- 172.21.15.1:0/3791689442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7800009430 con 0x7f7810071620 2024-03-18T18:49:49.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f780ed9d700 1 --2- 172.21.15.1:0/3791689442 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f781015a1d0 0x7f78100c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:49.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f780e59c700 1 --2- 172.21.15.1:0/3791689442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7810071620 0x7f78100b1090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7800000c00 tx=0x7f7800004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:49.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f77feffd700 1 -- 172.21.15.1:0/3791689442 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780001c420 con 0x7f7810071620 2024-03-18T18:49:49.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f77feffd700 1 -- 172.21.15.1:0/3791689442 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780001c5a0 con 0x7f7810071620 2024-03-18T18:49:49.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f7814faf700 1 -- 172.21.15.1:0/3791689442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f781015aaa0 con 0x7f7810071620 2024-03-18T18:49:49.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f77feffd700 1 -- 172.21.15.1:0/3791689442 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780001c950 con 0x7f7810071620 2024-03-18T18:49:49.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.011+0000 7f7814faf700 1 -- 172.21.15.1:0/3791689442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f781015b060 con 0x7f7810071620 2024-03-18T18:49:49.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.012+0000 7f7814faf700 1 -- 172.21.15.1:0/3791689442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7810071a60 con 0x7f7810071620 2024-03-18T18:49:49.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.013+0000 7f77feffd700 1 -- 172.21.15.1:0/3791689442 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f780001cab0 con 0x7f7810071620 2024-03-18T18:49:49.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.013+0000 7f77feffd700 1 --2- 172.21.15.1:0/3791689442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f77f006ac90 0x7f77f006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:49.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.014+0000 7f780dd9b700 1 --2- 172.21.15.1:0/3791689442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f77f006ac90 0x7f77f006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:49.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.014+0000 7f77feffd700 1 -- 172.21.15.1:0/3791689442 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6428+0+0 (secure 0 0 0) 0x7f780008a9f0 con 0x7f7810071620 2024-03-18T18:49:49.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.014+0000 7f780dd9b700 1 --2- 172.21.15.1:0/3791689442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f77f006ac90 0x7f77f006d150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f78101f6660 tx=0x7f7804007ec0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:49.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.018+0000 7f77feffd700 1 -- 172.21.15.1:0/3791689442 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7810071a60 con 0x7f7810071620 2024-03-18T18:49:49.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.275+0000 7f7814faf700 1 -- 172.21.15.1:0/3791689442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f7810071a60 con 0x7f7810071620 2024-03-18T18:49:49.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.279+0000 7f77feffd700 1 -- 172.21.15.1:0/3791689442 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) v1 ==== 137+0+0 (secure 0 0 0) 0x7f7810071a60 con 0x7f7810071620 2024-03-18T18:49:49.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.282+0000 7f7814faf700 1 -- 172.21.15.1:0/3791689442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f77f006ac90 msgr2=0x7f77f006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:49.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.282+0000 7f7814faf700 1 --2- 172.21.15.1:0/3791689442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f77f006ac90 0x7f77f006d150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f78101f6660 tx=0x7f7804007ec0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.282+0000 7f7814faf700 1 -- 172.21.15.1:0/3791689442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7810071620 msgr2=0x7f78100b1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.282+0000 7f7814faf700 1 --2- 172.21.15.1:0/3791689442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7810071620 0x7f78100b1090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7800000c00 tx=0x7f7800004750 comp rx=0 tx=0).stop 2024-03-18T18:49:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.282+0000 7f7814faf700 1 -- 172.21.15.1:0/3791689442 shutdown_connections 2024-03-18T18:49:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.282+0000 7f7814faf700 1 --2- 172.21.15.1:0/3791689442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f77f006ac90 0x7f77f006d150 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.282+0000 7f7814faf700 1 --2- 172.21.15.1:0/3791689442 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f781015a1d0 0x7f78100c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.282+0000 7f7814faf700 1 --2- 172.21.15.1:0/3791689442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7810071620 0x7f78100b1090 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.282+0000 7f7814faf700 1 --2- 172.21.15.1:0/3791689442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f78100cc0f0 0x7f78100c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.282+0000 7f7814faf700 1 -- 172.21.15.1:0/3791689442 >> 172.21.15.1:0/3791689442 conn(0x7f78100cc9b0 msgr2=0x7f78100ccd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.282+0000 7f7814faf700 1 -- 172.21.15.1:0/3791689442 shutdown_connections 2024-03-18T18:49:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.282+0000 7f7814faf700 1 -- 172.21.15.1:0/3791689442 wait complete. 2024-03-18T18:49:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:added data pool 4 to fsmap 2024-03-18T18:49:49.290 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T18:49:49.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.419+0000 7fc6b0b8a700 1 Processor -- start 2024-03-18T18:49:49.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.419+0000 7fc6b0b8a700 1 -- start start 2024-03-18T18:49:49.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.420+0000 7fc6b0b8a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc6ac071600 0x7fc6ac071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:49.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.420+0000 7fc6b0b8a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6ac06e2c0 0x7fc6ac06e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:49.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.420+0000 7fc6b0b8a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6ac0acd80 con 0x7fc6ac15a320 2024-03-18T18:49:49.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.420+0000 7fc6b0b8a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6ac071fc0 con 0x7fc6ac071600 2024-03-18T18:49:49.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.420+0000 7fc6b0b8a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac0cf220 con 0x7fc6ac06e2c0 2024-03-18T18:49:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.420+0000 7fc6aad9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6ac06e2c0 0x7fc6ac06e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.420+0000 7fc6a9d9b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc6ac071600 0x7fc6ac071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.420+0000 7fc6aa59c700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fc6ac15a320 0x7fc6ac0ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:46408/0 (socket says 172.21.15.1:46408) 2024-03-18T18:49:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.420+0000 7fc6aad9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6ac06e2c0 0x7fc6ac06e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:58108/0 (socket says 172.21.15.1:58108) 2024-03-18T18:49:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.420+0000 7fc6a9d9b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc6ac071600 0x7fc6ac071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:52274/0 (socket says 172.21.15.1:52274) 2024-03-18T18:49:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.420+0000 7fc6aa59c700 1 -- 172.21.15.1:0/2938525893 learned_addr learned my addr 172.21.15.1:0/2938525893 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:49:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.421+0000 7fc6a9d9b700 1 -- 172.21.15.1:0/2938525893 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6ac06e2c0 msgr2=0x7fc6ac06e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.421+0000 7fc6a9d9b700 1 --2- 172.21.15.1:0/2938525893 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6ac06e2c0 0x7fc6ac06e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.421+0000 7fc6a9d9b700 1 -- 172.21.15.1:0/2938525893 >> v1:172.21.15.165:6789/0 conn(0x7fc6ac15a320 legacy=0x7fc6ac0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.421+0000 7fc6a9d9b700 1 -- 172.21.15.1:0/2938525893 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ac0cef40 con 0x7fc6ac071600 2024-03-18T18:49:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.421+0000 7fc6a9d9b700 1 --2- 172.21.15.1:0/2938525893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6ac071600 0x7fc6ac071a80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc69c009840 tx=0x7fc69c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6efcaf87e2df0f67 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.421+0000 7fc6a959a700 1 -- 172.21.15.1:0/2938525893 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c011830 con 0x7fc6ac071600 2024-03-18T18:49:49.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.421+0000 7fc6a959a700 1 -- 172.21.15.1:0/2938525893 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69c011990 con 0x7fc6ac071600 2024-03-18T18:49:49.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.421+0000 7fc6a959a700 1 -- 172.21.15.1:0/2938525893 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c011d20 con 0x7fc6ac071600 2024-03-18T18:49:49.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.422+0000 7fc6b0b8a700 1 -- 172.21.15.1:0/2938525893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6ac071600 msgr2=0x7fc6ac071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:49.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.422+0000 7fc6b0b8a700 1 --2- 172.21.15.1:0/2938525893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6ac071600 0x7fc6ac071a80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc69c009840 tx=0x7fc69c009b50 comp rx=0 tx=0).stop 2024-03-18T18:49:49.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.422+0000 7fc6b0b8a700 1 -- 172.21.15.1:0/2938525893 shutdown_connections 2024-03-18T18:49:49.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.422+0000 7fc6b0b8a700 1 --2- 172.21.15.1:0/2938525893 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6ac06e2c0 0x7fc6ac06e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.422+0000 7fc6b0b8a700 1 --2- 172.21.15.1:0/2938525893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6ac071600 0x7fc6ac071a80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.422+0000 7fc6b0b8a700 1 -- 172.21.15.1:0/2938525893 >> 172.21.15.1:0/2938525893 conn(0x7fc6ac0cc990 msgr2=0x7fc6ac0ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:49.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.422+0000 7fc6b0b8a700 1 -- 172.21.15.1:0/2938525893 shutdown_connections 2024-03-18T18:49:49.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.422+0000 7fc6b0b8a700 1 -- 172.21.15.1:0/2938525893 wait complete. 2024-03-18T18:49:49.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.423+0000 7fc6b0b8a700 1 Processor -- start 2024-03-18T18:49:49.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.423+0000 7fc6b0b8a700 1 -- start start 2024-03-18T18:49:49.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.424+0000 7fc6b0b8a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6ac06e2c0 0x7fc6ac0b5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:49.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.424+0000 7fc6b0b8a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ac071600 0x7fc6ac0c79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:49.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.424+0000 7fc6b0b8a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6ac152470 0x7fc6ac156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:49.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.424+0000 7fc6b0b8a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac0c8650 con 0x7fc6ac071600 2024-03-18T18:49:49.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.424+0000 7fc6b0b8a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac0c7ef0 con 0x7fc6ac152470 2024-03-18T18:49:49.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.424+0000 7fc6b0b8a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac0c88a0 con 0x7fc6ac06e2c0 2024-03-18T18:49:49.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.424+0000 7fc6a9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ac071600 0x7fc6ac0c79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.424+0000 7fc6a9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ac071600 0x7fc6ac0c79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51042/0 (socket says 172.21.15.1:51042) 2024-03-18T18:49:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.424+0000 7fc6a9d9b700 1 -- 172.21.15.1:0/3966518675 learned_addr learned my addr 172.21.15.1:0/3966518675 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.424+0000 7fc6aa59c700 1 --2- 172.21.15.1:0/3966518675 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6ac06e2c0 0x7fc6ac0b5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.424+0000 7fc6aad9d700 1 --2- 172.21.15.1:0/3966518675 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6ac152470 0x7fc6ac156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.424+0000 7fc6a9d9b700 1 -- 172.21.15.1:0/3966518675 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6ac06e2c0 msgr2=0x7fc6ac0b5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.424+0000 7fc6a9d9b700 1 --2- 172.21.15.1:0/3966518675 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6ac06e2c0 0x7fc6ac0b5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.424+0000 7fc6a9d9b700 1 -- 172.21.15.1:0/3966518675 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6ac152470 msgr2=0x7fc6ac156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.424+0000 7fc6a9d9b700 1 --2- 172.21.15.1:0/3966518675 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6ac152470 0x7fc6ac156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.424+0000 7fc6a9d9b700 1 -- 172.21.15.1:0/3966518675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c0094a0 con 0x7fc6ac071600 2024-03-18T18:49:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.424+0000 7fc6aad9d700 1 --2- 172.21.15.1:0/3966518675 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6ac152470 0x7fc6ac156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:49:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.424+0000 7fc6a9d9b700 1 --2- 172.21.15.1:0/3966518675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ac071600 0x7fc6ac0c79b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc69c009fd0 tx=0x7fc69c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.425+0000 7fc69affd700 1 -- 172.21.15.1:0/3966518675 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c0036a0 con 0x7fc6ac071600 2024-03-18T18:49:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.425+0000 7fc69affd700 1 -- 172.21.15.1:0/3966518675 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69c01e070 con 0x7fc6ac071600 2024-03-18T18:49:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.425+0000 7fc6b0b8a700 1 -- 172.21.15.1:0/3966518675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6ac0c8b80 con 0x7fc6ac071600 2024-03-18T18:49:49.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.425+0000 7fc69affd700 1 -- 172.21.15.1:0/3966518675 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c003a60 con 0x7fc6ac071600 2024-03-18T18:49:49.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.425+0000 7fc6b0b8a700 1 -- 172.21.15.1:0/3966518675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ac0c9140 con 0x7fc6ac071600 2024-03-18T18:49:49.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.426+0000 7fc6b0b8a700 1 -- 172.21.15.1:0/3966518675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6ac0676a0 con 0x7fc6ac071600 2024-03-18T18:49:49.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.426+0000 7fc69affd700 1 -- 172.21.15.1:0/3966518675 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc69c003bc0 con 0x7fc6ac071600 2024-03-18T18:49:49.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.427+0000 7fc69affd700 1 --2- 172.21.15.1:0/3966518675 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc68c06ac90 0x7fc68c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:49.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.427+0000 7fc6aa59c700 1 --2- 172.21.15.1:0/3966518675 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc68c06ac90 0x7fc68c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:49.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.427+0000 7fc69affd700 1 -- 172.21.15.1:0/3966518675 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc69c08a5e0 con 0x7fc6ac071600 2024-03-18T18:49:49.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.427+0000 7fc6aa59c700 1 --2- 172.21.15.1:0/3966518675 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc68c06ac90 0x7fc68c06d150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0005910 tx=0x7fc6a0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:49.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.431+0000 7fc69affd700 1 -- 172.21.15.1:0/3966518675 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc69c055560 con 0x7fc6ac071600 2024-03-18T18:49:49.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.682+0000 7fc6b0b8a700 1 -- 172.21.15.1:0/3966518675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6ac159440 con 0x7fc6ac071600 2024-03-18T18:49:49.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.682+0000 7fc69affd700 1 -- 172.21.15.1:0/3966518675 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4727 (secure 0 0 0) 0x7fc69c058b80 con 0x7fc6ac071600 2024-03-18T18:49:49.679 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:49:49.679 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/883056851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":883056851},{"type":"v1","addr":"172.21.15.165:6834","nonce":883056851}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1388214718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1388214718},{"type":"v1","addr":"172.21.15.165:6835","nonce":1388214718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3020786191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3020786191},{"type":"v1","addr":"172.21.15.1:6836","nonce":3020786191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T18:49:44.809495+0000","modified":"2024-03-18T18:49:49.277178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2118908012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2118908012},{"type":"v1","addr":"172.21.15.1:6837","nonce":2118908012}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-18T18:49:49.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.684+0000 7fc6b0b8a700 1 -- 172.21.15.1:0/3966518675 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc68c06ac90 msgr2=0x7fc68c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:49.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.684+0000 7fc6b0b8a700 1 --2- 172.21.15.1:0/3966518675 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc68c06ac90 0x7fc68c06d150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0005910 tx=0x7fc6a0009040 comp rx=0 tx=0).stop 2024-03-18T18:49:49.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.684+0000 7fc6b0b8a700 1 -- 172.21.15.1:0/3966518675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ac071600 msgr2=0x7fc6ac0c79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:49.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.684+0000 7fc6b0b8a700 1 --2- 172.21.15.1:0/3966518675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ac071600 0x7fc6ac0c79b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc69c009fd0 tx=0x7fc69c004750 comp rx=0 tx=0).stop 2024-03-18T18:49:49.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.684+0000 7fc6b0b8a700 1 -- 172.21.15.1:0/3966518675 shutdown_connections 2024-03-18T18:49:49.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.684+0000 7fc6b0b8a700 1 --2- 172.21.15.1:0/3966518675 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc68c06ac90 0x7fc68c06d150 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.684+0000 7fc6b0b8a700 1 --2- 172.21.15.1:0/3966518675 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6ac06e2c0 0x7fc6ac0b5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.684+0000 7fc6b0b8a700 1 --2- 172.21.15.1:0/3966518675 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6ac152470 0x7fc6ac156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.684+0000 7fc6b0b8a700 1 --2- 172.21.15.1:0/3966518675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6ac071600 0x7fc6ac0c79b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.684+0000 7fc6b0b8a700 1 -- 172.21.15.1:0/3966518675 >> 172.21.15.1:0/3966518675 conn(0x7fc6ac0cc990 msgr2=0x7fc6ac06a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:49.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.684+0000 7fc6b0b8a700 1 -- 172.21.15.1:0/3966518675 shutdown_connections 2024-03-18T18:49:49.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.684+0000 7fc6b0b8a700 1 -- 172.21.15.1:0/3966518675 wait complete. 2024-03-18T18:49:49.682 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 7 2024-03-18T18:49:49.693 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:49:49.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.821+0000 7fc19c679700 1 Processor -- start 2024-03-18T18:49:49.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.821+0000 7fc19c679700 1 -- start start 2024-03-18T18:49:49.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.822+0000 7fc19c679700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc19415b300 0x7fc1940cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:49.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.822+0000 7fc19c679700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc19406ef20 0x7fc194099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:49.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.822+0000 7fc19c679700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19415b6e0 con 0x7fc1940cc5e0 2024-03-18T18:49:49.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.822+0000 7fc19c679700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1940cffd0 con 0x7fc19406ef20 2024-03-18T18:49:49.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.822+0000 7fc19c679700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1940cce60 con 0x7fc19415b300 2024-03-18T18:49:49.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.822+0000 7fc19ac16700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc19406ef20 0x7fc194099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:49.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.822+0000 7fc19ac16700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc19406ef20 0x7fc194099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51054/0 (socket says 172.21.15.1:51054) 2024-03-18T18:49:49.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.822+0000 7fc19ac16700 1 -- 172.21.15.1:0/2683652793 learned_addr learned my addr 172.21.15.1:0/2683652793 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:49.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.823+0000 7fc19a415700 1 --2- 172.21.15.1:0/2683652793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc19415b300 0x7fc1940cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:49.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.823+0000 7fc19ac16700 1 -- 172.21.15.1:0/2683652793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc19415b300 msgr2=0x7fc1940cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:49.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.823+0000 7fc19ac16700 1 --2- 172.21.15.1:0/2683652793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc19415b300 0x7fc1940cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.823+0000 7fc19ac16700 1 -- 172.21.15.1:0/2683652793 >> v1:172.21.15.165:6789/0 conn(0x7fc1940cc5e0 legacy=0x7fc19406e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:49.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.823+0000 7fc19ac16700 1 -- 172.21.15.1:0/2683652793 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1940cecd0 con 0x7fc19406ef20 2024-03-18T18:49:49.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.823+0000 7fc19ac16700 1 --2- 172.21.15.1:0/2683652793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc19406ef20 0x7fc194099a40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc18c009e50 tx=0x7fc18c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2704cb7b3b836de0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:49.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.823+0000 7fc199413700 1 -- 172.21.15.1:0/2683652793 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c00e070 con 0x7fc19406ef20 2024-03-18T18:49:49.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.823+0000 7fc199413700 1 -- 172.21.15.1:0/2683652793 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18c00b9c0 con 0x7fc19406ef20 2024-03-18T18:49:49.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.823+0000 7fc199413700 1 -- 172.21.15.1:0/2683652793 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c00bd70 con 0x7fc19406ef20 2024-03-18T18:49:49.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.823+0000 7fc19c679700 1 -- 172.21.15.1:0/2683652793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc19406ef20 msgr2=0x7fc194099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:49.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.823+0000 7fc19c679700 1 --2- 172.21.15.1:0/2683652793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc19406ef20 0x7fc194099a40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc18c009e50 tx=0x7fc18c00b330 comp rx=0 tx=0).stop 2024-03-18T18:49:49.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.824+0000 7fc19c679700 1 -- 172.21.15.1:0/2683652793 shutdown_connections 2024-03-18T18:49:49.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.824+0000 7fc19c679700 1 --2- 172.21.15.1:0/2683652793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc19406ef20 0x7fc194099a40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.824+0000 7fc19c679700 1 --2- 172.21.15.1:0/2683652793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc19415b300 0x7fc1940cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.824+0000 7fc19c679700 1 -- 172.21.15.1:0/2683652793 >> 172.21.15.1:0/2683652793 conn(0x7fc1940b2b50 msgr2=0x7fc1940b2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:49.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.824+0000 7fc19c679700 1 -- 172.21.15.1:0/2683652793 shutdown_connections 2024-03-18T18:49:49.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.824+0000 7fc19c679700 1 -- 172.21.15.1:0/2683652793 wait complete. 2024-03-18T18:49:49.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.825+0000 7fc19c679700 1 Processor -- start 2024-03-18T18:49:49.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.825+0000 7fc19c679700 1 -- start start 2024-03-18T18:49:49.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc19c679700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc19406ef20 0x7fc1941f0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:49.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc19c679700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1940cc5e0 0x7fc1940b4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:49.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc19c679700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc19415b300 0x7fc1940b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:49.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc19c679700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1941f12c0 con 0x7fc1940cc5e0 2024-03-18T18:49:49.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc19c679700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1941476d0 con 0x7fc19406ef20 2024-03-18T18:49:49.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc19c679700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc194147990 con 0x7fc19415b300 2024-03-18T18:49:49.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc199c14700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1940cc5e0 0x7fc1940b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:49.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc199c14700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1940cc5e0 0x7fc1940b4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51068/0 (socket says 172.21.15.1:51068) 2024-03-18T18:49:49.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc199c14700 1 -- 172.21.15.1:0/1580314536 learned_addr learned my addr 172.21.15.1:0/1580314536 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:49.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc19ac16700 1 --2- 172.21.15.1:0/1580314536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc19415b300 0x7fc1940b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:49.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc19a415700 1 --2- 172.21.15.1:0/1580314536 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc19406ef20 0x7fc1941f0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:49.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc199c14700 1 -- 172.21.15.1:0/1580314536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc19415b300 msgr2=0x7fc1940b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:49.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc199c14700 1 --2- 172.21.15.1:0/1580314536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc19415b300 0x7fc1940b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc199c14700 1 -- 172.21.15.1:0/1580314536 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc19406ef20 msgr2=0x7fc1941f0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:49.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc199c14700 1 --2- 172.21.15.1:0/1580314536 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc19406ef20 0x7fc1941f0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:49.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc199c14700 1 -- 172.21.15.1:0/1580314536 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc18c009b30 con 0x7fc1940cc5e0 2024-03-18T18:49:49.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc19ac16700 1 --2- 172.21.15.1:0/1580314536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc19415b300 0x7fc1940b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:49:49.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc19a415700 1 --2- 172.21.15.1:0/1580314536 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc19406ef20 0x7fc1941f0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:49:49.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc199c14700 1 --2- 172.21.15.1:0/1580314536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1940cc5e0 0x7fc1940b4810 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc19000c4e0 tx=0x7fc19000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:49.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc18affd700 1 -- 172.21.15.1:0/1580314536 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1900052d0 con 0x7fc1940cc5e0 2024-03-18T18:49:49.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc18affd700 1 -- 172.21.15.1:0/1580314536 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc190005430 con 0x7fc1940cc5e0 2024-03-18T18:49:49.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc19c679700 1 -- 172.21.15.1:0/1580314536 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc194147c70 con 0x7fc1940cc5e0 2024-03-18T18:49:49.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc18affd700 1 -- 172.21.15.1:0/1580314536 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1900038c0 con 0x7fc1940cc5e0 2024-03-18T18:49:49.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.826+0000 7fc19c679700 1 -- 172.21.15.1:0/1580314536 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1940c5560 con 0x7fc1940cc5e0 2024-03-18T18:49:49.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.827+0000 7fc19c679700 1 -- 172.21.15.1:0/1580314536 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc194148190 con 0x7fc1940cc5e0 2024-03-18T18:49:49.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.828+0000 7fc18affd700 1 -- 172.21.15.1:0/1580314536 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc190003a20 con 0x7fc1940cc5e0 2024-03-18T18:49:49.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.828+0000 7fc18affd700 1 --2- 172.21.15.1:0/1580314536 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc17806ac90 0x7fc17806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:49.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.828+0000 7fc19a415700 1 --2- 172.21.15.1:0/1580314536 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc17806ac90 0x7fc17806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:49.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.828+0000 7fc18affd700 1 -- 172.21.15.1:0/1580314536 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc190088a40 con 0x7fc1940cc5e0 2024-03-18T18:49:49.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.829+0000 7fc19a415700 1 --2- 172.21.15.1:0/1580314536 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc17806ac90 0x7fc17806d150 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc184009820 tx=0x7fc184008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:49.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:49.833+0000 7fc18affd700 1 -- 172.21.15.1:0/1580314536 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc194148190 con 0x7fc1940cc5e0 2024-03-18T18:49:50.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.034+0000 7fc19c679700 1 -- 172.21.15.1:0/1580314536 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc194148190 con 0x7fc1940cc5e0 2024-03-18T18:49:50.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.035+0000 7fc18affd700 1 -- 172.21.15.1:0/1580314536 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17830 (secure 0 0 0) 0x7fc194148190 con 0x7fc1940cc5e0 2024-03-18T18:49:50.032 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:49:50.032 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:49.276961+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:49:50.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.037+0000 7fc19c679700 1 -- 172.21.15.1:0/1580314536 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc17806ac90 msgr2=0x7fc17806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.037+0000 7fc19c679700 1 --2- 172.21.15.1:0/1580314536 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc17806ac90 0x7fc17806d150 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc184009820 tx=0x7fc184008040 comp rx=0 tx=0).stop 2024-03-18T18:49:50.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.037+0000 7fc19c679700 1 -- 172.21.15.1:0/1580314536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1940cc5e0 msgr2=0x7fc1940b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.037+0000 7fc19c679700 1 --2- 172.21.15.1:0/1580314536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1940cc5e0 0x7fc1940b4810 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc19000c4e0 tx=0x7fc19000c8a0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.037+0000 7fc19c679700 1 -- 172.21.15.1:0/1580314536 shutdown_connections 2024-03-18T18:49:50.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.037+0000 7fc19c679700 1 --2- 172.21.15.1:0/1580314536 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc17806ac90 0x7fc17806d150 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.037+0000 7fc19c679700 1 --2- 172.21.15.1:0/1580314536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc19415b300 0x7fc1940b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.037+0000 7fc19c679700 1 --2- 172.21.15.1:0/1580314536 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc19406ef20 0x7fc1941f0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.037+0000 7fc19c679700 1 --2- 172.21.15.1:0/1580314536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1940cc5e0 0x7fc1940b4810 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.037+0000 7fc19c679700 1 -- 172.21.15.1:0/1580314536 >> 172.21.15.1:0/1580314536 conn(0x7fc1940b2b50 msgr2=0x7fc1940b21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:50.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.037+0000 7fc19c679700 1 -- 172.21.15.1:0/1580314536 shutdown_connections 2024-03-18T18:49:50.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.037+0000 7fc19c679700 1 -- 172.21.15.1:0/1580314536 wait complete. 2024-03-18T18:49:50.046 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:49:50.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.175+0000 7f70a613f700 1 Processor -- start 2024-03-18T18:49:50.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.175+0000 7f70a613f700 1 -- start start 2024-03-18T18:49:50.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.176+0000 7f70a613f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f70a0143c20 0x7f70a00b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.176+0000 7f70a613f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70a006f240 0x7f70a00cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.176+0000 7f70a613f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a0144000 con 0x7f70a00b3f40 2024-03-18T18:49:50.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.176+0000 7f70a613f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a00b43c0 con 0x7f70a0143c20 2024-03-18T18:49:50.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.176+0000 7f70a613f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a00b51b0 con 0x7f70a006f240 2024-03-18T18:49:50.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.176+0000 7f709f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f70a0143c20 0x7f70a00b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.176+0000 7f709f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f70a0143c20 0x7f70a00b3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51076/0 (socket says 172.21.15.1:51076) 2024-03-18T18:49:50.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.176+0000 7f709f7fe700 1 -- 172.21.15.1:0/134234472 learned_addr learned my addr 172.21.15.1:0/134234472 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:50.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.176+0000 7f709ffff700 1 --2- 172.21.15.1:0/134234472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70a006f240 0x7f70a00cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.176+0000 7f709f7fe700 1 -- 172.21.15.1:0/134234472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70a006f240 msgr2=0x7f70a00cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:50.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.176+0000 7f709f7fe700 1 --2- 172.21.15.1:0/134234472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70a006f240 0x7f70a00cddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.176+0000 7f709f7fe700 1 -- 172.21.15.1:0/134234472 >> v1:172.21.15.165:6789/0 conn(0x7f70a00b3f40 legacy=0x7f70a006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.176+0000 7f709f7fe700 1 -- 172.21.15.1:0/134234472 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a00cd8d0 con 0x7f70a0143c20 2024-03-18T18:49:50.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.177+0000 7f709f7fe700 1 --2- 172.21.15.1:0/134234472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70a0143c20 0x7f70a00b3a00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7088009920 tx=0x7f7088009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc176e7b5c51f053 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:50.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.177+0000 7f709e7fc700 1 -- 172.21.15.1:0/134234472 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708800ba20 con 0x7f70a0143c20 2024-03-18T18:49:50.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.177+0000 7f709e7fc700 1 -- 172.21.15.1:0/134234472 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708800bb80 con 0x7f70a0143c20 2024-03-18T18:49:50.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.177+0000 7f709e7fc700 1 -- 172.21.15.1:0/134234472 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7088009e70 con 0x7f70a0143c20 2024-03-18T18:49:50.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.177+0000 7f70a613f700 1 -- 172.21.15.1:0/134234472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70a0143c20 msgr2=0x7f70a00b3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.177+0000 7f70a613f700 1 --2- 172.21.15.1:0/134234472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70a0143c20 0x7f70a00b3a00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7088009920 tx=0x7f7088009c30 comp rx=0 tx=0).stop 2024-03-18T18:49:50.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.177+0000 7f70a613f700 1 -- 172.21.15.1:0/134234472 shutdown_connections 2024-03-18T18:49:50.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.177+0000 7f70a613f700 1 --2- 172.21.15.1:0/134234472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70a006f240 0x7f70a00cddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.177+0000 7f70a613f700 1 --2- 172.21.15.1:0/134234472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70a0143c20 0x7f70a00b3a00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.177+0000 7f70a613f700 1 -- 172.21.15.1:0/134234472 >> 172.21.15.1:0/134234472 conn(0x7f70a00b1d20 msgr2=0x7f70a00b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:50.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.177+0000 7f70a613f700 1 -- 172.21.15.1:0/134234472 shutdown_connections 2024-03-18T18:49:50.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.177+0000 7f70a613f700 1 -- 172.21.15.1:0/134234472 wait complete. 2024-03-18T18:49:50.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.179+0000 7f70a613f700 1 Processor -- start 2024-03-18T18:49:50.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.179+0000 7f70a613f700 1 -- start start 2024-03-18T18:49:50.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.180+0000 7f70a613f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70a006f240 0x7f70a01e3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.180+0000 7f70a613f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70a00b3f40 0x7f70a01e3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.180+0000 7f70a613f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70a0143c20 0x7f70a0141300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.180+0000 7f70a613f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a013a130 con 0x7f70a00b3f40 2024-03-18T18:49:50.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.180+0000 7f70a613f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a013a380 con 0x7f70a0143c20 2024-03-18T18:49:50.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.180+0000 7f70a613f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a013a660 con 0x7f70a006f240 2024-03-18T18:49:50.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.180+0000 7f709effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70a00b3f40 0x7f70a01e3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.180+0000 7f709effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70a00b3f40 0x7f70a01e3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51080/0 (socket says 172.21.15.1:51080) 2024-03-18T18:49:50.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.180+0000 7f709effd700 1 -- 172.21.15.1:0/1952464097 learned_addr learned my addr 172.21.15.1:0/1952464097 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:50.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.180+0000 7f709ffff700 1 --2- 172.21.15.1:0/1952464097 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70a0143c20 0x7f70a0141300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.180+0000 7f709f7fe700 1 --2- 172.21.15.1:0/1952464097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70a006f240 0x7f70a01e3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.181+0000 7f709effd700 1 -- 172.21.15.1:0/1952464097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70a006f240 msgr2=0x7f70a01e3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.181+0000 7f709effd700 1 --2- 172.21.15.1:0/1952464097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70a006f240 0x7f70a01e3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.181+0000 7f709effd700 1 -- 172.21.15.1:0/1952464097 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70a0143c20 msgr2=0x7f70a0141300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.181+0000 7f709effd700 1 --2- 172.21.15.1:0/1952464097 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70a0143c20 0x7f70a0141300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.181+0000 7f709effd700 1 -- 172.21.15.1:0/1952464097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7088009550 con 0x7f70a00b3f40 2024-03-18T18:49:50.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.181+0000 7f709f7fe700 1 --2- 172.21.15.1:0/1952464097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70a006f240 0x7f70a01e3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:50.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.181+0000 7f709ffff700 1 --2- 172.21.15.1:0/1952464097 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70a0143c20 0x7f70a0141300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:50.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.181+0000 7f709effd700 1 --2- 172.21.15.1:0/1952464097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70a00b3f40 0x7f70a01e3f40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f709000c4b0 tx=0x7f709000c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:50.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.181+0000 7f7097fff700 1 -- 172.21.15.1:0/1952464097 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70900052d0 con 0x7f70a00b3f40 2024-03-18T18:49:50.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.181+0000 7f7097fff700 1 -- 172.21.15.1:0/1952464097 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7090005430 con 0x7f70a00b3f40 2024-03-18T18:49:50.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.181+0000 7f70a613f700 1 -- 172.21.15.1:0/1952464097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a013a940 con 0x7f70a00b3f40 2024-03-18T18:49:50.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.181+0000 7f7097fff700 1 -- 172.21.15.1:0/1952464097 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70900038c0 con 0x7f70a00b3f40 2024-03-18T18:49:50.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.181+0000 7f70a613f700 1 -- 172.21.15.1:0/1952464097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a013b140 con 0x7f70a00b3f40 2024-03-18T18:49:50.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.182+0000 7f70a613f700 1 -- 172.21.15.1:0/1952464097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a013ae60 con 0x7f70a00b3f40 2024-03-18T18:49:50.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.182+0000 7f7097fff700 1 -- 172.21.15.1:0/1952464097 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7090003a20 con 0x7f70a00b3f40 2024-03-18T18:49:50.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.183+0000 7f7097fff700 1 --2- 172.21.15.1:0/1952464097 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f708006ac90 0x7f708006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.183+0000 7f7097fff700 1 -- 172.21.15.1:0/1952464097 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f70900889f0 con 0x7f70a00b3f40 2024-03-18T18:49:50.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.183+0000 7f709f7fe700 1 --2- 172.21.15.1:0/1952464097 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f708006ac90 0x7f708006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.183+0000 7f709f7fe700 1 --2- 172.21.15.1:0/1952464097 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f708006ac90 0x7f708006d150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7088000c00 tx=0x7f7088008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:50.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.187+0000 7f7097fff700 1 -- 172.21.15.1:0/1952464097 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f70a013ae60 con 0x7f70a00b3f40 2024-03-18T18:49:50.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.392+0000 7f70a613f700 1 -- 172.21.15.1:0/1952464097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70a013ae60 con 0x7f70a00b3f40 2024-03-18T18:49:50.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.394+0000 7f7097fff700 1 -- 172.21.15.1:0/1952464097 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17830 (secure 0 0 0) 0x7f70a013ae60 con 0x7f70a00b3f40 2024-03-18T18:49:50.390 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:49:50.390 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:49.276961+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:49:50.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.396+0000 7f70a613f700 1 -- 172.21.15.1:0/1952464097 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f708006ac90 msgr2=0x7f708006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.396+0000 7f70a613f700 1 --2- 172.21.15.1:0/1952464097 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f708006ac90 0x7f708006d150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7088000c00 tx=0x7f7088008030 comp rx=0 tx=0).stop 2024-03-18T18:49:50.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.396+0000 7f70a613f700 1 -- 172.21.15.1:0/1952464097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70a00b3f40 msgr2=0x7f70a01e3f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.396+0000 7f70a613f700 1 --2- 172.21.15.1:0/1952464097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70a00b3f40 0x7f70a01e3f40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f709000c4b0 tx=0x7f709000c870 comp rx=0 tx=0).stop 2024-03-18T18:49:50.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.396+0000 7f70a613f700 1 -- 172.21.15.1:0/1952464097 shutdown_connections 2024-03-18T18:49:50.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.396+0000 7f70a613f700 1 --2- 172.21.15.1:0/1952464097 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f708006ac90 0x7f708006d150 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.396+0000 7f70a613f700 1 --2- 172.21.15.1:0/1952464097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70a006f240 0x7f70a01e3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.396+0000 7f70a613f700 1 --2- 172.21.15.1:0/1952464097 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70a0143c20 0x7f70a0141300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.396+0000 7f70a613f700 1 --2- 172.21.15.1:0/1952464097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70a00b3f40 0x7f70a01e3f40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.396+0000 7f70a613f700 1 -- 172.21.15.1:0/1952464097 >> 172.21.15.1:0/1952464097 conn(0x7f70a00b1d20 msgr2=0x7f70a00b3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:50.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.396+0000 7f70a613f700 1 -- 172.21.15.1:0/1952464097 shutdown_connections 2024-03-18T18:49:50.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.396+0000 7f70a613f700 1 -- 172.21.15.1:0/1952464097 wait complete. 2024-03-18T18:49:50.423 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T18:49:50.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.550+0000 7fc728d8f700 1 Processor -- start 2024-03-18T18:49:50.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.550+0000 7fc728d8f700 1 -- start start 2024-03-18T18:49:50.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.551+0000 7fc728d8f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc724071a80 0x7fc72406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.551+0000 7fc728d8f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7240acd80 con 0x7fc724071600 2024-03-18T18:49:50.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.551+0000 7fc728d8f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7240cf220 con 0x7fc72415a320 2024-03-18T18:49:50.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.551+0000 7fc728d8f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc724071f10 con 0x7fc724071a80 2024-03-18T18:49:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.551+0000 7fc721d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc724071600 0x7fc72406e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58308/0 (socket says 172.21.15.1:58308) 2024-03-18T18:49:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.551+0000 7fc721d9b700 1 -- 172.21.15.1:0/3708067426 learned_addr learned my addr 172.21.15.1:0/3708067426 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:49:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc72159a700 1 -- 172.21.15.1:0/3708067426 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670853150 0 0) 0x7fc7240acd80 con 0x7fc724071600 2024-03-18T18:49:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc722d9d700 1 --2- 172.21.15.1:0/3708067426 >> v2:172.21.15.165:3300/0 conn(0x7fc724071a80 0x7fc72406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc72159a700 1 -- 172.21.15.1:0/3708067426 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc70c0035b0 con 0x7fc724071600 2024-03-18T18:49:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc72159a700 1 -- 172.21.15.1:0/3708067426 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 682042979 0 0) 0x7fc70c0035b0 con 0x7fc724071600 2024-03-18T18:49:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc72159a700 1 -- 172.21.15.1:0/3708067426 >> v2:172.21.15.165:3300/0 conn(0x7fc724071a80 msgr2=0x7fc72406e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc72159a700 1 --2- 172.21.15.1:0/3708067426 >> v2:172.21.15.165:3300/0 conn(0x7fc724071a80 0x7fc72406e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc72159a700 1 -- 172.21.15.1:0/3708067426 >> v1:172.21.15.165:6789/0 conn(0x7fc72415a320 legacy=0x7fc7240ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc72159a700 1 -- 172.21.15.1:0/3708067426 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7240cef40 con 0x7fc724071600 2024-03-18T18:49:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc72159a700 1 -- 172.21.15.1:0/3708067426 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fc718001670 con 0x7fc724071600 2024-03-18T18:49:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc72159a700 1 -- 172.21.15.1:0/3708067426 >> v1:172.21.15.1:6789/0 conn(0x7fc724071600 legacy=0x7fc72406e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc72159a700 1 --2- 172.21.15.1:0/3708067426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc70c003a00 0x7fc70c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc72159a700 1 --2- 172.21.15.1:0/3708067426 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc70c004850 0x7fc70c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc72159a700 1 --2- 172.21.15.1:0/3708067426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc70c005210 0x7fc70c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc72159a700 1 -- 172.21.15.1:0/3708067426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc724071f10 con 0x7fc70c005210 2024-03-18T18:49:50.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc72159a700 1 -- 172.21.15.1:0/3708067426 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c00c0b0 con 0x7fc70c004850 2024-03-18T18:49:50.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc72159a700 1 -- 172.21.15.1:0/3708067426 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c00c270 con 0x7fc70c003a00 2024-03-18T18:49:50.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc722d9d700 1 --2- 172.21.15.1:0/3708067426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc70c005210 0x7fc70c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc721d9b700 1 --2- 172.21.15.1:0/3708067426 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc70c004850 0x7fc70c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc72259c700 1 --2- 172.21.15.1:0/3708067426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc70c003a00 0x7fc70c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc722d9d700 1 -- 172.21.15.1:0/3708067426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc70c003a00 msgr2=0x7fc70c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:50.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc722d9d700 1 --2- 172.21.15.1:0/3708067426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc70c003a00 0x7fc70c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc722d9d700 1 -- 172.21.15.1:0/3708067426 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc70c004850 msgr2=0x7fc70c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:50.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc722d9d700 1 --2- 172.21.15.1:0/3708067426 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc70c004850 0x7fc70c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc722d9d700 1 -- 172.21.15.1:0/3708067426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c0035b0 con 0x7fc70c005210 2024-03-18T18:49:50.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc721d9b700 1 --2- 172.21.15.1:0/3708067426 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc70c004850 0x7fc70c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:50.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc722d9d700 1 -- 172.21.15.1:0/3708067426 reap_dead start 2024-03-18T18:49:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.552+0000 7fc722d9d700 1 --2- 172.21.15.1:0/3708067426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc70c005210 0x7fc70c00ba10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009fc0 tx=0x7fc70c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a407360f9ac2446c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.553+0000 7fc72159a700 1 -- 172.21.15.1:0/3708067426 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c003a20 con 0x7fc70c005210 2024-03-18T18:49:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.553+0000 7fc72159a700 1 -- 172.21.15.1:0/3708067426 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71c00f040 con 0x7fc70c005210 2024-03-18T18:49:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.553+0000 7fc72159a700 1 -- 172.21.15.1:0/3708067426 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c003de0 con 0x7fc70c005210 2024-03-18T18:49:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.553+0000 7fc728d8f700 1 -- 172.21.15.1:0/3708067426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc70c005210 msgr2=0x7fc70c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.553+0000 7fc728d8f700 1 --2- 172.21.15.1:0/3708067426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc70c005210 0x7fc70c00ba10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009fc0 tx=0x7fc70c005880 comp rx=0 tx=0).stop 2024-03-18T18:49:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.553+0000 7fc728d8f700 1 -- 172.21.15.1:0/3708067426 shutdown_connections 2024-03-18T18:49:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.553+0000 7fc728d8f700 1 --2- 172.21.15.1:0/3708067426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc70c005210 0x7fc70c00ba10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.553+0000 7fc728d8f700 1 -- 172.21.15.1:0/3708067426 >> 172.21.15.1:0/3708067426 conn(0x7fc7240cc990 msgr2=0x7fc7240ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.553+0000 7fc728d8f700 1 -- 172.21.15.1:0/3708067426 shutdown_connections 2024-03-18T18:49:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.553+0000 7fc728d8f700 1 -- 172.21.15.1:0/3708067426 wait complete. 2024-03-18T18:49:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.555+0000 7fc728d8f700 1 Processor -- start 2024-03-18T18:49:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.555+0000 7fc728d8f700 1 -- start start 2024-03-18T18:49:50.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.555+0000 7fc728d8f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc70c005210 0x7fc7241f62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.555+0000 7fc728d8f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc72415ccb0 0x7fc7241f3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.555+0000 7fc728d8f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc72415add0 0x7fc72415b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.555+0000 7fc728d8f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7241f7220 con 0x7fc72415ccb0 2024-03-18T18:49:50.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.555+0000 7fc728d8f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72415a9b0 con 0x7fc72415add0 2024-03-18T18:49:50.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.555+0000 7fc728d8f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc724153c90 con 0x7fc70c005210 2024-03-18T18:49:50.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.555+0000 7fc721d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc72415ccb0 0x7fc7241f3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.555+0000 7fc721d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc72415ccb0 0x7fc7241f3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51088/0 (socket says 172.21.15.1:51088) 2024-03-18T18:49:50.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.555+0000 7fc721d9b700 1 -- 172.21.15.1:0/1358126399 learned_addr learned my addr 172.21.15.1:0/1358126399 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:50.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.556+0000 7fc722d9d700 1 --2- 172.21.15.1:0/1358126399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc72415add0 0x7fc72415b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.556+0000 7fc72259c700 1 --2- 172.21.15.1:0/1358126399 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc70c005210 0x7fc7241f62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.556+0000 7fc721d9b700 1 -- 172.21.15.1:0/1358126399 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc70c005210 msgr2=0x7fc7241f62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.556+0000 7fc721d9b700 1 --2- 172.21.15.1:0/1358126399 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc70c005210 0x7fc7241f62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.556+0000 7fc721d9b700 1 -- 172.21.15.1:0/1358126399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc72415add0 msgr2=0x7fc72415b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.556+0000 7fc721d9b700 1 --2- 172.21.15.1:0/1358126399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc72415add0 0x7fc72415b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.556+0000 7fc721d9b700 1 -- 172.21.15.1:0/1358126399 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72415d130 con 0x7fc72415ccb0 2024-03-18T18:49:50.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.556+0000 7fc722d9d700 1 --2- 172.21.15.1:0/1358126399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc72415add0 0x7fc72415b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:50.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.556+0000 7fc721d9b700 1 --2- 172.21.15.1:0/1358126399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc72415ccb0 0x7fc7241f3780 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc71800e780 tx=0x7fc71800eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:50.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.556+0000 7fc712ffd700 1 -- 172.21.15.1:0/1358126399 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc718012590 con 0x7fc72415ccb0 2024-03-18T18:49:50.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.556+0000 7fc712ffd700 1 -- 172.21.15.1:0/1358126399 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7180126f0 con 0x7fc72415ccb0 2024-03-18T18:49:50.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.556+0000 7fc712ffd700 1 -- 172.21.15.1:0/1358126399 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc718012a80 con 0x7fc72415ccb0 2024-03-18T18:49:50.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.556+0000 7fc728d8f700 1 -- 172.21.15.1:0/1358126399 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7241f6980 con 0x7fc72415ccb0 2024-03-18T18:49:50.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.556+0000 7fc728d8f700 1 -- 172.21.15.1:0/1358126399 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7241f6f40 con 0x7fc72415ccb0 2024-03-18T18:49:50.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.557+0000 7fc728d8f700 1 -- 172.21.15.1:0/1358126399 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc72415a710 con 0x7fc72415ccb0 2024-03-18T18:49:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.558+0000 7fc712ffd700 1 -- 172.21.15.1:0/1358126399 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc718012be0 con 0x7fc72415ccb0 2024-03-18T18:49:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.558+0000 7fc712ffd700 1 --2- 172.21.15.1:0/1358126399 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc70406ac90 0x7fc70406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.558+0000 7fc712ffd700 1 -- 172.21.15.1:0/1358126399 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc718089440 con 0x7fc72415ccb0 2024-03-18T18:49:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.558+0000 7fc72259c700 1 --2- 172.21.15.1:0/1358126399 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc70406ac90 0x7fc70406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.558+0000 7fc72259c700 1 --2- 172.21.15.1:0/1358126399 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc70406ac90 0x7fc70406d150 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc714000c00 tx=0x7fc714009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.563+0000 7fc712ffd700 1 -- 172.21.15.1:0/1358126399 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc72415a710 con 0x7fc72415ccb0 2024-03-18T18:49:50.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.818+0000 7fc728d8f700 1 -- 172.21.15.1:0/1358126399 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc72415aaf0 con 0x7fc72415ccb0 2024-03-18T18:49:50.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.818+0000 7fc712ffd700 1 -- 172.21.15.1:0/1358126399 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4727 (secure 0 0 0) 0x7fc72415aaf0 con 0x7fc72415ccb0 2024-03-18T18:49:50.815 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:49:50.815 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/883056851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":883056851},{"type":"v1","addr":"172.21.15.165:6834","nonce":883056851}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1388214718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1388214718},{"type":"v1","addr":"172.21.15.165:6835","nonce":1388214718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3020786191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3020786191},{"type":"v1","addr":"172.21.15.1:6836","nonce":3020786191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T18:49:44.809495+0000","modified":"2024-03-18T18:49:49.277178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2118908012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2118908012},{"type":"v1","addr":"172.21.15.1:6837","nonce":2118908012}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-18T18:49:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.820+0000 7fc728d8f700 1 -- 172.21.15.1:0/1358126399 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc70406ac90 msgr2=0x7fc70406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.820+0000 7fc728d8f700 1 --2- 172.21.15.1:0/1358126399 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc70406ac90 0x7fc70406d150 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc714000c00 tx=0x7fc714009040 comp rx=0 tx=0).stop 2024-03-18T18:49:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.820+0000 7fc728d8f700 1 -- 172.21.15.1:0/1358126399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc72415ccb0 msgr2=0x7fc7241f3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.820+0000 7fc728d8f700 1 --2- 172.21.15.1:0/1358126399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc72415ccb0 0x7fc7241f3780 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc71800e780 tx=0x7fc71800eb40 comp rx=0 tx=0).stop 2024-03-18T18:49:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.820+0000 7fc728d8f700 1 -- 172.21.15.1:0/1358126399 shutdown_connections 2024-03-18T18:49:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.820+0000 7fc728d8f700 1 --2- 172.21.15.1:0/1358126399 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc70406ac90 0x7fc70406d150 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.820+0000 7fc728d8f700 1 --2- 172.21.15.1:0/1358126399 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc70c005210 0x7fc7241f62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.820+0000 7fc728d8f700 1 --2- 172.21.15.1:0/1358126399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc72415add0 0x7fc72415b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.820+0000 7fc728d8f700 1 --2- 172.21.15.1:0/1358126399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc72415ccb0 0x7fc7241f3780 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.820+0000 7fc728d8f700 1 -- 172.21.15.1:0/1358126399 >> 172.21.15.1:0/1358126399 conn(0x7fc7240cc990 msgr2=0x7fc72406dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.821+0000 7fc728d8f700 1 -- 172.21.15.1:0/1358126399 shutdown_connections 2024-03-18T18:49:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.821+0000 7fc728d8f700 1 -- 172.21.15.1:0/1358126399 wait complete. 2024-03-18T18:49:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 7 2024-03-18T18:49:50.828 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:49:50.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.956+0000 7f4f9e89e700 1 Processor -- start 2024-03-18T18:49:50.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.956+0000 7f4f9e89e700 1 -- start start 2024-03-18T18:49:50.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.957+0000 7f4f9e89e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4f981597a0 0x7f4f9815dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.957+0000 7f4f9e89e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f9815e2c0 con 0x7f4f9806e0c0 2024-03-18T18:49:50.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.957+0000 7f4f9e89e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f980b4b20 con 0x7f4f980cfbe0 2024-03-18T18:49:50.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.957+0000 7f4f9e89e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f98071bd0 con 0x7f4f981597a0 2024-03-18T18:49:50.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.957+0000 7f4f9ce3b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4f9806e0c0 0x7f4f980714c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58316/0 (socket says 172.21.15.1:58316) 2024-03-18T18:49:50.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.957+0000 7f4f9ce3b700 1 -- 172.21.15.1:0/2273659429 learned_addr learned my addr 172.21.15.1:0/2273659429 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:49:50.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f97fff700 1 --2- 172.21.15.1:0/2273659429 >> v2:172.21.15.165:3300/0 conn(0x7f4f981597a0 0x7f4f9815dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f96ffd700 1 -- 172.21.15.1:0/2273659429 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497201397 0 0) 0x7f4f90001490 con 0x7f4f9806e0c0 2024-03-18T18:49:50.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f96ffd700 1 -- 172.21.15.1:0/2273659429 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f800035b0 con 0x7f4f9806e0c0 2024-03-18T18:49:50.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f96ffd700 1 -- 172.21.15.1:0/2273659429 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3095925804 0 0) 0x7f4f800035b0 con 0x7f4f9806e0c0 2024-03-18T18:49:50.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f96ffd700 1 -- 172.21.15.1:0/2273659429 >> v2:172.21.15.165:3300/0 conn(0x7f4f981597a0 msgr2=0x7f4f9815dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:50.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f96ffd700 1 --2- 172.21.15.1:0/2273659429 >> v2:172.21.15.165:3300/0 conn(0x7f4f981597a0 0x7f4f9815dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f96ffd700 1 -- 172.21.15.1:0/2273659429 >> v1:172.21.15.165:6789/0 conn(0x7f4f980cfbe0 legacy=0x7f4f98070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f96ffd700 1 -- 172.21.15.1:0/2273659429 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f980b3f30 con 0x7f4f9806e0c0 2024-03-18T18:49:50.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f96ffd700 1 -- 172.21.15.1:0/2273659429 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f4f90001b80 con 0x7f4f9806e0c0 2024-03-18T18:49:50.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f97fff700 1 --2- 172.21.15.1:0/2273659429 >> v2:172.21.15.165:3300/0 conn(0x7f4f981597a0 0x7f4f9815dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:49:50.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f96ffd700 1 -- 172.21.15.1:0/2273659429 >> v1:172.21.15.1:6789/0 conn(0x7f4f9806e0c0 legacy=0x7f4f980714c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f96ffd700 1 --2- 172.21.15.1:0/2273659429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f80003a00 0x7f4f80003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f96ffd700 1 --2- 172.21.15.1:0/2273659429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f80004400 0x7f4f800048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f96ffd700 1 --2- 172.21.15.1:0/2273659429 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f800050d0 0x7f4f8000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f96ffd700 1 -- 172.21.15.1:0/2273659429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f98071bd0 con 0x7f4f80003a00 2024-03-18T18:49:50.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f96ffd700 1 -- 172.21.15.1:0/2273659429 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8000c030 con 0x7f4f80004400 2024-03-18T18:49:50.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f96ffd700 1 -- 172.21.15.1:0/2273659429 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8000c1f0 con 0x7f4f800050d0 2024-03-18T18:49:50.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f97fff700 1 --2- 172.21.15.1:0/2273659429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f80003a00 0x7f4f80003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f9ce3b700 1 --2- 172.21.15.1:0/2273659429 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f800050d0 0x7f4f8000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f977fe700 1 --2- 172.21.15.1:0/2273659429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f80004400 0x7f4f800048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f97fff700 1 -- 172.21.15.1:0/2273659429 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f800050d0 msgr2=0x7f4f8000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f97fff700 1 --2- 172.21.15.1:0/2273659429 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f800050d0 0x7f4f8000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f97fff700 1 -- 172.21.15.1:0/2273659429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f80004400 msgr2=0x7f4f800048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f97fff700 1 --2- 172.21.15.1:0/2273659429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f80004400 0x7f4f800048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f9ce3b700 1 -- 172.21.15.1:0/2273659429 reap_dead start 2024-03-18T18:49:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.958+0000 7f4f97fff700 1 -- 172.21.15.1:0/2273659429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f800035b0 con 0x7f4f80003a00 2024-03-18T18:49:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.959+0000 7f4f97fff700 1 --2- 172.21.15.1:0/2273659429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f80003a00 0x7f4f80003de0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4f88009d30 tx=0x7f4f80005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b8b94f84399663 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.959+0000 7f4f96ffd700 1 -- 172.21.15.1:0/2273659429 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f88009e80 con 0x7f4f80003a00 2024-03-18T18:49:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.959+0000 7f4f96ffd700 1 -- 172.21.15.1:0/2273659429 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8800e9a0 con 0x7f4f80003a00 2024-03-18T18:49:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.959+0000 7f4f96ffd700 1 -- 172.21.15.1:0/2273659429 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8800ed10 con 0x7f4f80003a00 2024-03-18T18:49:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.959+0000 7f4f9e89e700 1 -- 172.21.15.1:0/2273659429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f80003a00 msgr2=0x7f4f80003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.959+0000 7f4f9e89e700 1 --2- 172.21.15.1:0/2273659429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f80003a00 0x7f4f80003de0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4f88009d30 tx=0x7f4f80005880 comp rx=0 tx=0).stop 2024-03-18T18:49:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.959+0000 7f4f9e89e700 1 -- 172.21.15.1:0/2273659429 shutdown_connections 2024-03-18T18:49:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.959+0000 7f4f9e89e700 1 --2- 172.21.15.1:0/2273659429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f80003a00 0x7f4f80003de0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.959+0000 7f4f9e89e700 1 -- 172.21.15.1:0/2273659429 >> 172.21.15.1:0/2273659429 conn(0x7f4f9806ab70 msgr2=0x7f4f9806af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:50.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.959+0000 7f4f9e89e700 1 -- 172.21.15.1:0/2273659429 shutdown_connections 2024-03-18T18:49:50.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.959+0000 7f4f9e89e700 1 -- 172.21.15.1:0/2273659429 wait complete. 2024-03-18T18:49:50.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.961+0000 7f4f9e89e700 1 Processor -- start 2024-03-18T18:49:50.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.961+0000 7f4f9e89e700 1 -- start start 2024-03-18T18:49:50.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.961+0000 7f4f9e89e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f80003a00 0x7f4f980ab8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.961+0000 7f4f9e89e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f980cd620 0x7f4f980cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.961+0000 7f4f9e89e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f98157fc0 0x7f4f98158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.961+0000 7f4f9e89e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f980cdfe0 con 0x7f4f98157fc0 2024-03-18T18:49:50.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.961+0000 7f4f9e89e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f980a92c0 con 0x7f4f980cd620 2024-03-18T18:49:50.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.961+0000 7f4f9e89e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f98158d10 con 0x7f4f80003a00 2024-03-18T18:49:50.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.962+0000 7f4f9ce3b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f98157fc0 0x7f4f98158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.962+0000 7f4f9ce3b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f98157fc0 0x7f4f98158430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51102/0 (socket says 172.21.15.1:51102) 2024-03-18T18:49:50.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.962+0000 7f4f9ce3b700 1 -- 172.21.15.1:0/2457804299 learned_addr learned my addr 172.21.15.1:0/2457804299 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:50.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.962+0000 7f4f977fe700 1 --2- 172.21.15.1:0/2457804299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f980cd620 0x7f4f980cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.962+0000 7f4f97fff700 1 --2- 172.21.15.1:0/2457804299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f80003a00 0x7f4f980ab8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.962+0000 7f4f9ce3b700 1 -- 172.21.15.1:0/2457804299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f80003a00 msgr2=0x7f4f980ab8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.962+0000 7f4f9ce3b700 1 --2- 172.21.15.1:0/2457804299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f80003a00 0x7f4f980ab8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.962+0000 7f4f9ce3b700 1 -- 172.21.15.1:0/2457804299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f980cd620 msgr2=0x7f4f980cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:50.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.962+0000 7f4f9ce3b700 1 --2- 172.21.15.1:0/2457804299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f980cd620 0x7f4f980cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:50.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.962+0000 7f4f9ce3b700 1 -- 172.21.15.1:0/2457804299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f980a8990 con 0x7f4f98157fc0 2024-03-18T18:49:50.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.962+0000 7f4f9ce3b700 1 --2- 172.21.15.1:0/2457804299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f98157fc0 0x7f4f98158430 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4f980b2320 tx=0x7f4f90001390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:50.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.962+0000 7f4f94ff9700 1 -- 172.21.15.1:0/2457804299 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f90007b00 con 0x7f4f98157fc0 2024-03-18T18:49:50.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.962+0000 7f4f94ff9700 1 -- 172.21.15.1:0/2457804299 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f900133f0 con 0x7f4f98157fc0 2024-03-18T18:49:50.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.962+0000 7f4f94ff9700 1 -- 172.21.15.1:0/2457804299 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f900137a0 con 0x7f4f98157fc0 2024-03-18T18:49:50.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.962+0000 7f4f9e89e700 1 -- 172.21.15.1:0/2457804299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f980a8c70 con 0x7f4f98157fc0 2024-03-18T18:49:50.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.962+0000 7f4f9e89e700 1 -- 172.21.15.1:0/2457804299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f980abde0 con 0x7f4f98157fc0 2024-03-18T18:49:50.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.963+0000 7f4f9e89e700 1 -- 172.21.15.1:0/2457804299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f980a9450 con 0x7f4f98157fc0 2024-03-18T18:49:50.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.963+0000 7f4f94ff9700 1 -- 172.21.15.1:0/2457804299 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4f90013900 con 0x7f4f98157fc0 2024-03-18T18:49:50.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.964+0000 7f4f94ff9700 1 --2- 172.21.15.1:0/2457804299 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4f8406ac90 0x7f4f8406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:50.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.964+0000 7f4f97fff700 1 --2- 172.21.15.1:0/2457804299 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4f8406ac90 0x7f4f8406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:50.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.964+0000 7f4f94ff9700 1 -- 172.21.15.1:0/2457804299 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4f9008a040 con 0x7f4f98157fc0 2024-03-18T18:49:50.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.964+0000 7f4f97fff700 1 --2- 172.21.15.1:0/2457804299 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4f8406ac90 0x7f4f8406d150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4f88000c00 tx=0x7f4f8801e040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:50.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:50.969+0000 7f4f94ff9700 1 -- 172.21.15.1:0/2457804299 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4f90054fc0 con 0x7f4f98157fc0 2024-03-18T18:49:51.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.176+0000 7f4f9e89e700 1 -- 172.21.15.1:0/2457804299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f9815e2c0 con 0x7f4f98157fc0 2024-03-18T18:49:51.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.177+0000 7f4f94ff9700 1 -- 172.21.15.1:0/2457804299 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17830 (secure 0 0 0) 0x7f4f9815e2c0 con 0x7f4f98157fc0 2024-03-18T18:49:51.173 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:49:51.173 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:49.276961+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:49:51.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.179+0000 7f4f9e89e700 1 -- 172.21.15.1:0/2457804299 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4f8406ac90 msgr2=0x7f4f8406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:51.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.179+0000 7f4f9e89e700 1 --2- 172.21.15.1:0/2457804299 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4f8406ac90 0x7f4f8406d150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4f88000c00 tx=0x7f4f8801e040 comp rx=0 tx=0).stop 2024-03-18T18:49:51.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.179+0000 7f4f9e89e700 1 -- 172.21.15.1:0/2457804299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f98157fc0 msgr2=0x7f4f98158430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:51.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.179+0000 7f4f9e89e700 1 --2- 172.21.15.1:0/2457804299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f98157fc0 0x7f4f98158430 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4f980b2320 tx=0x7f4f90001390 comp rx=0 tx=0).stop 2024-03-18T18:49:51.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.179+0000 7f4f9e89e700 1 -- 172.21.15.1:0/2457804299 shutdown_connections 2024-03-18T18:49:51.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.179+0000 7f4f9e89e700 1 --2- 172.21.15.1:0/2457804299 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4f8406ac90 0x7f4f8406d150 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.179+0000 7f4f9e89e700 1 --2- 172.21.15.1:0/2457804299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f80003a00 0x7f4f980ab8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.179+0000 7f4f9e89e700 1 --2- 172.21.15.1:0/2457804299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f98157fc0 0x7f4f98158430 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.179+0000 7f4f9e89e700 1 --2- 172.21.15.1:0/2457804299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f980cd620 0x7f4f980cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.179+0000 7f4f9e89e700 1 -- 172.21.15.1:0/2457804299 >> 172.21.15.1:0/2457804299 conn(0x7f4f9806ab70 msgr2=0x7f4f98159df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:51.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.179+0000 7f4f9e89e700 1 -- 172.21.15.1:0/2457804299 shutdown_connections 2024-03-18T18:49:51.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.179+0000 7f4f9e89e700 1 -- 172.21.15.1:0/2457804299 wait complete. 2024-03-18T18:49:51.593 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T18:49:44.809495+0000', 'modified': '2024-03-18T18:49:49.277178+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4273}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4273': {'gid': 4273, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6837/2118908012', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6835', 'nonce': 2118908012}, {'type': 'v1', 'addr': '172.21.15.1:6837', 'nonce': 2118908012}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-03-18T18:49:51.593 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T18:49:51.594 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-18T18:49:51.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.738+0000 7fb4de7c7700 1 Processor -- start 2024-03-18T18:49:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.738+0000 7fb4de7c7700 1 -- start start 2024-03-18T18:49:51.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.739+0000 7fb4de7c7700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb4d815a1d0 0x7fb4d80add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.739+0000 7fb4de7c7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4d8071620 0x7fb4d8071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.739+0000 7fb4de7c7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4d80ce8f0 con 0x7fb4d80cb9e0 2024-03-18T18:49:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.739+0000 7fb4de7c7700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d80cdb00 con 0x7fb4d815a1d0 2024-03-18T18:49:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.739+0000 7fb4de7c7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d815a5b0 con 0x7fb4d8071620 2024-03-18T18:49:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.739+0000 7fb4dcd64700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb4d80cb9e0 0x7fb4d80cbe70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58330/0 (socket says 172.21.15.1:58330) 2024-03-18T18:49:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.739+0000 7fb4dcd64700 1 -- 172.21.15.1:0/1870776252 learned_addr learned my addr 172.21.15.1:0/1870776252 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:49:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.739+0000 7fb4d7fff700 1 --2- 172.21.15.1:0/1870776252 >> v2:172.21.15.165:3300/0 conn(0x7fb4d815a1d0 0x7fb4d80add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.739+0000 7fb4cffff700 1 --2- 172.21.15.1:0/1870776252 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4d8071620 0x7fb4d8071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4d77fe700 1 -- 172.21.15.1:0/1870776252 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885377151 0 0) 0x7fb4d80ce8f0 con 0x7fb4d80cb9e0 2024-03-18T18:49:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4d77fe700 1 -- 172.21.15.1:0/1870776252 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4c40057c0 con 0x7fb4d80cb9e0 2024-03-18T18:49:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4d77fe700 1 -- 172.21.15.1:0/1870776252 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3999050325 0 0) 0x7fb4c40057c0 con 0x7fb4d80cb9e0 2024-03-18T18:49:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4d77fe700 1 -- 172.21.15.1:0/1870776252 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4d8071620 msgr2=0x7fb4d8071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4d77fe700 1 --2- 172.21.15.1:0/1870776252 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4d8071620 0x7fb4d8071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4d77fe700 1 -- 172.21.15.1:0/1870776252 >> v2:172.21.15.165:3300/0 conn(0x7fb4d815a1d0 msgr2=0x7fb4d80add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4d77fe700 1 --2- 172.21.15.1:0/1870776252 >> v2:172.21.15.165:3300/0 conn(0x7fb4d815a1d0 0x7fb4d80add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4d77fe700 1 -- 172.21.15.1:0/1870776252 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d8071fc0 con 0x7fb4d80cb9e0 2024-03-18T18:49:51.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4d77fe700 1 -- 172.21.15.1:0/1870776252 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fb4d00019f0 con 0x7fb4d80cb9e0 2024-03-18T18:49:51.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4cffff700 1 --2- 172.21.15.1:0/1870776252 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4d8071620 0x7fb4d8071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:49:51.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4d77fe700 1 -- 172.21.15.1:0/1870776252 >> v1:172.21.15.1:6789/0 conn(0x7fb4d80cb9e0 legacy=0x7fb4d80cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:51.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4d77fe700 1 --2- 172.21.15.1:0/1870776252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4c4005c10 0x7fb4c4006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4d77fe700 1 --2- 172.21.15.1:0/1870776252 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4d8071620 0x7fb4c4006a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4d77fe700 1 --2- 172.21.15.1:0/1870776252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4c4006fa0 0x7fb4c4007420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4d77fe700 1 -- 172.21.15.1:0/1870776252 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d80cdb00 con 0x7fb4c4006fa0 2024-03-18T18:49:51.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4d77fe700 1 -- 172.21.15.1:0/1870776252 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c400dc20 con 0x7fb4c4005c10 2024-03-18T18:49:51.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4d77fe700 1 -- 172.21.15.1:0/1870776252 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c400dd60 con 0x7fb4d8071620 2024-03-18T18:49:51.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4d7fff700 1 --2- 172.21.15.1:0/1870776252 >> v2:172.21.15.165:3300/0 conn(0x7fb4d815a1d0 0x7fb4d80add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:49:51.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4dcd64700 1 --2- 172.21.15.1:0/1870776252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4c4006fa0 0x7fb4c4007420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4cffff700 1 --2- 172.21.15.1:0/1870776252 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4d8071620 0x7fb4c4006a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4d7fff700 1 --2- 172.21.15.1:0/1870776252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4c4005c10 0x7fb4c4006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4dcd64700 1 -- 172.21.15.1:0/1870776252 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4d8071620 msgr2=0x7fb4c4006a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:51.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4dcd64700 1 --2- 172.21.15.1:0/1870776252 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4d8071620 0x7fb4c4006a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4dcd64700 1 -- 172.21.15.1:0/1870776252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4c4005c10 msgr2=0x7fb4c4006030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:51.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4dcd64700 1 --2- 172.21.15.1:0/1870776252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4c4005c10 0x7fb4c4006030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.740+0000 7fb4dcd64700 1 -- 172.21.15.1:0/1870776252 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c00094a0 con 0x7fb4c4006fa0 2024-03-18T18:49:51.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.741+0000 7fb4dcd64700 1 --2- 172.21.15.1:0/1870776252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4c4006fa0 0x7fb4c4007420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb4d000a860 tx=0x7fb4c4007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8060375a584da01 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:51.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.741+0000 7fb4d77fe700 1 -- 172.21.15.1:0/1870776252 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0012600 con 0x7fb4c4006fa0 2024-03-18T18:49:51.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.741+0000 7fb4d77fe700 1 -- 172.21.15.1:0/1870776252 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d0012780 con 0x7fb4c4006fa0 2024-03-18T18:49:51.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.741+0000 7fb4d77fe700 1 -- 172.21.15.1:0/1870776252 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0012b30 con 0x7fb4c4006fa0 2024-03-18T18:49:51.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.741+0000 7fb4de7c7700 1 -- 172.21.15.1:0/1870776252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4c4006fa0 msgr2=0x7fb4c4007420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:51.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.741+0000 7fb4de7c7700 1 --2- 172.21.15.1:0/1870776252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4c4006fa0 0x7fb4c4007420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb4d000a860 tx=0x7fb4c4007a90 comp rx=0 tx=0).stop 2024-03-18T18:49:51.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.741+0000 7fb4dcd64700 1 -- 172.21.15.1:0/1870776252 reap_dead start 2024-03-18T18:49:51.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.741+0000 7fb4de7c7700 1 -- 172.21.15.1:0/1870776252 shutdown_connections 2024-03-18T18:49:51.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.741+0000 7fb4de7c7700 1 -- 172.21.15.1:0/1870776252 >> 172.21.15.1:0/1870776252 conn(0x7fb4d80cc9b0 msgr2=0x7fb4d80ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:51.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.741+0000 7fb4de7c7700 1 -- 172.21.15.1:0/1870776252 shutdown_connections 2024-03-18T18:49:51.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.741+0000 7fb4de7c7700 1 -- 172.21.15.1:0/1870776252 wait complete. 2024-03-18T18:49:51.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.743+0000 7fb4de7c7700 1 Processor -- start 2024-03-18T18:49:51.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.743+0000 7fb4de7c7700 1 -- start start 2024-03-18T18:49:51.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.743+0000 7fb4de7c7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4d81ee4d0 0x7fb4d815a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.743+0000 7fb4de7c7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4d815bc30 0x7fb4d814bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.743+0000 7fb4de7c7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4d815cae0 0x7fb4d815cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.743+0000 7fb4de7c7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d815b810 con 0x7fb4d815cae0 2024-03-18T18:49:51.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.743+0000 7fb4de7c7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d815d4a0 con 0x7fb4d815bc30 2024-03-18T18:49:51.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.743+0000 7fb4de7c7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d81ee8b0 con 0x7fb4d81ee4d0 2024-03-18T18:49:51.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.743+0000 7fb4dcd64700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4d815cae0 0x7fb4d815cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.744+0000 7fb4dcd64700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4d815cae0 0x7fb4d815cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51118/0 (socket says 172.21.15.1:51118) 2024-03-18T18:49:51.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.744+0000 7fb4dcd64700 1 -- 172.21.15.1:0/292199583 learned_addr learned my addr 172.21.15.1:0/292199583 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:51.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.744+0000 7fb4cffff700 1 --2- 172.21.15.1:0/292199583 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4d815bc30 0x7fb4d814bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.744+0000 7fb4d7fff700 1 --2- 172.21.15.1:0/292199583 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4d81ee4d0 0x7fb4d815a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.744+0000 7fb4dcd64700 1 -- 172.21.15.1:0/292199583 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4d81ee4d0 msgr2=0x7fb4d815a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:51.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.744+0000 7fb4dcd64700 1 --2- 172.21.15.1:0/292199583 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4d81ee4d0 0x7fb4d815a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.744+0000 7fb4dcd64700 1 -- 172.21.15.1:0/292199583 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4d815bc30 msgr2=0x7fb4d814bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:51.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.744+0000 7fb4dcd64700 1 --2- 172.21.15.1:0/292199583 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4d815bc30 0x7fb4d814bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.744+0000 7fb4dcd64700 1 -- 172.21.15.1:0/292199583 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d815ab90 con 0x7fb4d815cae0 2024-03-18T18:49:51.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.744+0000 7fb4dcd64700 1 --2- 172.21.15.1:0/292199583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4d815cae0 0x7fb4d815cf60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0015040 tx=0x7fb4d0001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:51.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.744+0000 7fb4d57fa700 1 -- 172.21.15.1:0/292199583 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d00019b0 con 0x7fb4d815cae0 2024-03-18T18:49:51.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.744+0000 7fb4d57fa700 1 -- 172.21.15.1:0/292199583 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d0012610 con 0x7fb4d815cae0 2024-03-18T18:49:51.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.744+0000 7fb4de7c7700 1 -- 172.21.15.1:0/292199583 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d815ae70 con 0x7fb4d815cae0 2024-03-18T18:49:51.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.744+0000 7fb4d57fa700 1 -- 172.21.15.1:0/292199583 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d001d610 con 0x7fb4d815cae0 2024-03-18T18:49:51.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.744+0000 7fb4de7c7700 1 -- 172.21.15.1:0/292199583 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d815b430 con 0x7fb4d815cae0 2024-03-18T18:49:51.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.746+0000 7fb4d57fa700 1 -- 172.21.15.1:0/292199583 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb4d0019070 con 0x7fb4d815cae0 2024-03-18T18:49:51.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.746+0000 7fb4d57fa700 1 --2- 172.21.15.1:0/292199583 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb4b806ac90 0x7fb4b806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.746+0000 7fb4d57fa700 1 -- 172.21.15.1:0/292199583 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb4d008aae0 con 0x7fb4d815cae0 2024-03-18T18:49:51.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.746+0000 7fb4d7fff700 1 --2- 172.21.15.1:0/292199583 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb4b806ac90 0x7fb4b806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.747+0000 7fb4d7fff700 1 --2- 172.21.15.1:0/292199583 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb4b806ac90 0x7fb4b806d150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8156d30 tx=0x7fb4c80091c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:51.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.748+0000 7fb4ea970b80 1 Processor -- start 2024-03-18T18:49:51.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.748+0000 7fb4ea970b80 1 -- start start 2024-03-18T18:49:51.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.749+0000 7fb4ea970b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56093177ec10 0x56093177eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.749+0000 7fb4ea970b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5609317500a0 0x56093179b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.749+0000 7fb4ea970b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560931790630 0x560931753160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.749+0000 7fb4ea970b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609316c9f20 con 0x56093177ec10 2024-03-18T18:49:51.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.749+0000 7fb4ea970b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609316abc30 con 0x560931790630 2024-03-18T18:49:51.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.749+0000 7fb4ea970b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x5609316a9430 con 0x5609317500a0 2024-03-18T18:49:51.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.749+0000 7fb4d7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56093177ec10 0x56093177eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.749+0000 7fb4d7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56093177ec10 0x56093177eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51124/0 (socket says 172.21.15.1:51124) 2024-03-18T18:49:51.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.749+0000 7fb4d7fff700 1 -- 172.21.15.1:0/3179664284 learned_addr learned my addr 172.21.15.1:0/3179664284 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:51.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.749+0000 7fb4cffff700 1 --2- 172.21.15.1:0/3179664284 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5609317500a0 0x56093179b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.749+0000 7fb4dcd64700 1 --2- 172.21.15.1:0/3179664284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560931790630 0x560931753160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.749+0000 7fb4d7fff700 1 -- 172.21.15.1:0/3179664284 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5609317500a0 msgr2=0x56093179b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:51.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.749+0000 7fb4d7fff700 1 --2- 172.21.15.1:0/3179664284 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5609317500a0 0x56093179b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.749+0000 7fb4d7fff700 1 -- 172.21.15.1:0/3179664284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560931790630 msgr2=0x560931753160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:51.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.749+0000 7fb4d7fff700 1 --2- 172.21.15.1:0/3179664284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560931790630 0x560931753160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.750+0000 7fb4d7fff700 1 -- 172.21.15.1:0/3179664284 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560931498790 con 0x56093177ec10 2024-03-18T18:49:51.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.750+0000 7fb4d7fff700 1 --2- 172.21.15.1:0/3179664284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56093177ec10 0x56093177eff0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8004550 tx=0x7fb4c8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e31c1dc10bdef0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.750+0000 7fb4cdffb700 1 -- 172.21.15.1:0/3179664284 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c8004580 con 0x56093177ec10 2024-03-18T18:49:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.750+0000 7fb4cdffb700 1 -- 172.21.15.1:0/3179664284 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c80038b0 con 0x56093177ec10 2024-03-18T18:49:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.750+0000 7fb4cdffb700 1 -- 172.21.15.1:0/3179664284 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c8003c20 con 0x56093177ec10 2024-03-18T18:49:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.750+0000 7fb4ea970b80 1 -- 172.21.15.1:0/3179664284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56093177ec10 msgr2=0x56093177eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.750+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/3179664284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56093177ec10 0x56093177eff0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8004550 tx=0x7fb4c8017000 comp rx=0 tx=0).stop 2024-03-18T18:49:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.750+0000 7fb4ea970b80 1 -- 172.21.15.1:0/3179664284 shutdown_connections 2024-03-18T18:49:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.750+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/3179664284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560931790630 0x560931753160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.750+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/3179664284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56093177ec10 0x56093177eff0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.750+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/3179664284 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5609317500a0 0x56093179b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.750+0000 7fb4ea970b80 1 -- 172.21.15.1:0/3179664284 >> 172.21.15.1:0/3179664284 conn(0x5609316b2cf0 msgr2=0x560931752a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.751+0000 7fb4ea970b80 1 -- 172.21.15.1:0/3179664284 shutdown_connections 2024-03-18T18:49:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.751+0000 7fb4ea970b80 1 -- 172.21.15.1:0/3179664284 wait complete. 2024-03-18T18:49:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.751+0000 7fb4ea970b80 1 -- start start 2024-03-18T18:49:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.751+0000 7fb4ea970b80 1 Processor -- start 2024-03-18T18:49:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.752+0000 7fb4ea970b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5609317500a0 0x560931773830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.752+0000 7fb4ea970b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56093177ec10 0x560931776b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.752+0000 7fb4ea970b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560931790630 0x560931777060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.752+0000 7fb4ea970b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609316a4d20 con 0x560931790630 2024-03-18T18:49:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.752+0000 7fb4ea970b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x560931640770 con 0x5609317500a0 2024-03-18T18:49:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.752+0000 7fb4ea970b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x5609316636f0 con 0x56093177ec10 2024-03-18T18:49:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.752+0000 7fb4ea970b80 10 client.0 _ll_register_callbacks cb 0x5609314ac5a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T18:49:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.752+0000 7fb4dcd64700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560931790630 0x560931777060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.752+0000 7fb4dcd64700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560931790630 0x560931777060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51134/0 (socket says 172.21.15.1:51134) 2024-03-18T18:49:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.752+0000 7fb4d7fff700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5609317500a0 0x560931773830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.752+0000 7fb4dcd64700 1 -- 172.21.15.1:0/1014044122 learned_addr learned my addr 172.21.15.1:0/1014044122 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.752+0000 7fb4cffff700 1 --2- 172.21.15.1:0/1014044122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56093177ec10 0x560931776b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4dcd64700 1 -- 172.21.15.1:0/1014044122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56093177ec10 msgr2=0x560931776b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4dcd64700 1 --2- 172.21.15.1:0/1014044122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56093177ec10 0x560931776b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4dcd64700 1 -- 172.21.15.1:0/1014044122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5609317500a0 msgr2=0x560931773830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4dcd64700 1 --2- 172.21.15.1:0/1014044122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5609317500a0 0x560931773830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4dcd64700 1 -- 172.21.15.1:0/1014044122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c8004200 con 0x560931790630 2024-03-18T18:49:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4d7fff700 1 --2- 172.21.15.1:0/1014044122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5609317500a0 0x560931773830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:49:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4dcd64700 1 --2- 172.21.15.1:0/1014044122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560931790630 0x560931777060 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0001cb0 tx=0x7fb4d00597d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8befd0242ad6f17d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4b6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T18:49:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4b6ffd700 1 -- 172.21.15.1:0/1014044122 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0061a90 con 0x560931790630 2024-03-18T18:49:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4b6ffd700 1 -- 172.21.15.1:0/1014044122 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb4d005d0d0 con 0x560931790630 2024-03-18T18:49:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4ea970b80 1 -- 172.21.15.1:0/1014044122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560931746e90 con 0x560931790630 2024-03-18T18:49:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4b6ffd700 1 -- 172.21.15.1:0/1014044122 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4d005d290 con 0x560931790630 2024-03-18T18:49:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4ea970b80 10 client.4380 fetch_fsmap learned FSMap version 7 2024-03-18T18:49:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4ea970b80 1 -- 172.21.15.1:0/1014044122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5609315850a0 con 0x560931790630 2024-03-18T18:49:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4b6ffd700 1 -- 172.21.15.1:0/1014044122 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560931746e90 con 0x560931790630 2024-03-18T18:49:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4b6ffd700 1 -- 172.21.15.1:0/1014044122 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 7) v1 ==== 3122+0+0 (secure 0 0 0) 0x7fb4d0072060 con 0x560931790630 2024-03-18T18:49:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4ea970b80 10 client.4380 fetch_fsmap finished waiting for FSMap version 7 2024-03-18T18:49:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4ea970b80 10 client.4380 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-18T18:49:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4ea970b80 20 client.4380 populate_metadata read hostname 'smithi001' 2024-03-18T18:49:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/1014044122 >> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] conn(0x5609317536e0 0x560931753ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4ea970b80 4 client.4380 mds_command: new command op to 4273 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T18:49:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.753+0000 7fb4ea970b80 1 -- 172.21.15.1:0/1014044122 --> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560931724010 con 0x5609317536e0 2024-03-18T18:49:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.754+0000 7fb4d7fff700 1 --2- 172.21.15.1:0/1014044122 >> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] conn(0x5609317536e0 0x560931753ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.754+0000 7fb4b6ffd700 10 client.4380 ms_handle_connect on v2:172.21.15.1:6835/2118908012 2024-03-18T18:49:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.754+0000 7fb4d7fff700 1 --2- 172.21.15.1:0/1014044122 >> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] conn(0x5609317536e0 0x560931753ac0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ac5287340b13969 server_cookie=ea6d1dd099d0125e in_seq=0 out_seq=0 2024-03-18T18:49:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.754+0000 7fb4b6ffd700 1 -- 172.21.15.1:0/1014044122 <== mds.0 v2:172.21.15.1:6835/2118908012 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560931724010 con 0x5609317536e0 2024-03-18T18:49:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.754+0000 7fb4b6ffd700 10 client.4380 handle_command_reply: tid=0 2024-03-18T18:49:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.754+0000 7fb4ea970b80 1 client.4380 shutdown 2024-03-18T18:49:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.755+0000 7fb4ea970b80 1 -- 172.21.15.1:0/1014044122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560931790630 msgr2=0x560931777060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.755+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/1014044122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560931790630 0x560931777060 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0001cb0 tx=0x7fb4d00597d0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.755+0000 7fb4ea970b80 1 -- 172.21.15.1:0/1014044122 shutdown_connections 2024-03-18T18:49:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.755+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/1014044122 >> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] conn(0x5609317536e0 0x560931753ac0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.755+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/1014044122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560931790630 0x560931777060 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.755+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/1014044122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5609317500a0 0x560931773830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.755+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/1014044122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56093177ec10 0x560931776b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.755+0000 7fb4ea970b80 1 -- 172.21.15.1:0/1014044122 >> 172.21.15.1:0/1014044122 conn(0x5609316b2cf0 msgr2=0x56093174ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.755+0000 7fb4b6ffd700 0 client.4380 ms_handle_reset on v2:172.21.15.1:6835/2118908012 2024-03-18T18:49:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.755+0000 7fb4ea970b80 1 -- 172.21.15.1:0/1014044122 shutdown_connections 2024-03-18T18:49:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.755+0000 7fb4ea970b80 1 -- 172.21.15.1:0/1014044122 wait complete. 2024-03-18T18:49:51.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.755+0000 7fb4ea970b80 20 client.4380 trim_cache size 0 max 16384 2024-03-18T18:49:51.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.773+0000 7fb4ea970b80 1 Processor -- start 2024-03-18T18:49:51.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.773+0000 7fb4ea970b80 1 -- start start 2024-03-18T18:49:51.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.773+0000 7fb4ea970b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5609317500a0 0x56093179a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.773+0000 7fb4ea970b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56093177ec10 0x56093174f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.773+0000 7fb4ea970b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560931790630 0x560931777790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.773+0000 7fb4ea970b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x560931499390 con 0x56093177ec10 2024-03-18T18:49:51.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.773+0000 7fb4ea970b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609317807c0 con 0x560931790630 2024-03-18T18:49:51.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.773+0000 7fb4ea970b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x56093177cc70 con 0x5609317500a0 2024-03-18T18:49:51.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.774+0000 7fb4cffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56093177ec10 0x56093174f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.774+0000 7fb4cffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56093177ec10 0x56093174f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51150/0 (socket says 172.21.15.1:51150) 2024-03-18T18:49:51.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.774+0000 7fb4cffff700 1 -- 172.21.15.1:0/27013067 learned_addr learned my addr 172.21.15.1:0/27013067 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:51.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.774+0000 7fb4d7fff700 1 --2- 172.21.15.1:0/27013067 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5609317500a0 0x56093179a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.774+0000 7fb4dcd64700 1 --2- 172.21.15.1:0/27013067 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560931790630 0x560931777790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.774+0000 7fb4cffff700 1 -- 172.21.15.1:0/27013067 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5609317500a0 msgr2=0x56093179a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.774+0000 7fb4cffff700 1 --2- 172.21.15.1:0/27013067 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5609317500a0 0x56093179a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.774+0000 7fb4cffff700 1 -- 172.21.15.1:0/27013067 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560931790630 msgr2=0x560931777790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.774+0000 7fb4cffff700 1 --2- 172.21.15.1:0/27013067 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560931790630 0x560931777790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.774+0000 7fb4cffff700 1 -- 172.21.15.1:0/27013067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56093166b010 con 0x56093177ec10 2024-03-18T18:49:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.774+0000 7fb4d7fff700 1 --2- 172.21.15.1:0/27013067 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5609317500a0 0x56093179a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.774+0000 7fb4cffff700 1 --2- 172.21.15.1:0/27013067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56093177ec10 0x56093174f140 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb4c00154a0 tx=0x7fb4c0009210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a63e7fc0098c8a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.774+0000 7fb4ce7fc700 1 -- 172.21.15.1:0/27013067 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c001a050 con 0x56093177ec10 2024-03-18T18:49:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.774+0000 7fb4ce7fc700 1 -- 172.21.15.1:0/27013067 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c001b040 con 0x56093177ec10 2024-03-18T18:49:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.774+0000 7fb4ce7fc700 1 -- 172.21.15.1:0/27013067 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0004030 con 0x56093177ec10 2024-03-18T18:49:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.775+0000 7fb4ea970b80 1 -- 172.21.15.1:0/27013067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56093177ec10 msgr2=0x56093174f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.775+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/27013067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56093177ec10 0x56093174f140 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb4c00154a0 tx=0x7fb4c0009210 comp rx=0 tx=0).stop 2024-03-18T18:49:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.775+0000 7fb4ea970b80 1 -- 172.21.15.1:0/27013067 shutdown_connections 2024-03-18T18:49:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.775+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/27013067 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5609317500a0 0x56093179a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.775+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/27013067 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x560931790630 0x560931777790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.775+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/27013067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56093177ec10 0x56093174f140 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.775+0000 7fb4ea970b80 1 -- 172.21.15.1:0/27013067 >> 172.21.15.1:0/27013067 conn(0x5609316b2cf0 msgr2=0x560931773640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.775+0000 7fb4ea970b80 1 -- 172.21.15.1:0/27013067 shutdown_connections 2024-03-18T18:49:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.775+0000 7fb4ea970b80 1 -- 172.21.15.1:0/27013067 wait complete. 2024-03-18T18:49:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.775+0000 7fb4ea970b80 1 -- start start 2024-03-18T18:49:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.776+0000 7fb4ea970b80 1 Processor -- start 2024-03-18T18:49:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.776+0000 7fb4ea970b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5609317500a0 0x5609317ed290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.776+0000 7fb4ea970b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56093177ec10 0x56093177d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.776+0000 7fb4ea970b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x560931790630 0x56093177def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.776+0000 7fb4ea970b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x56093179cbf0 con 0x5609317500a0 2024-03-18T18:49:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.776+0000 7fb4ea970b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x56093177e430 con 0x56093177ec10 2024-03-18T18:49:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.776+0000 7fb4ea970b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x560931804d50 con 0x560931790630 2024-03-18T18:49:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.776+0000 7fb4ea970b80 10 client.0 _ll_register_callbacks cb 0x7fb4d0059340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T18:49:51.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.776+0000 7fb4d7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5609317500a0 0x5609317ed290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.776+0000 7fb4d7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5609317500a0 0x5609317ed290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51160/0 (socket says 172.21.15.1:51160) 2024-03-18T18:49:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.776+0000 7fb4d7fff700 1 -- 172.21.15.1:0/3040803373 learned_addr learned my addr 172.21.15.1:0/3040803373 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4cffff700 1 --2- 172.21.15.1:0/3040803373 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56093177ec10 0x56093177d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4dcd64700 1 --2- 172.21.15.1:0/3040803373 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x560931790630 0x56093177def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4d7fff700 1 -- 172.21.15.1:0/3040803373 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x560931790630 msgr2=0x56093177def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4d7fff700 1 --2- 172.21.15.1:0/3040803373 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x560931790630 0x56093177def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4d7fff700 1 -- 172.21.15.1:0/3040803373 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56093177ec10 msgr2=0x56093177d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4d7fff700 1 --2- 172.21.15.1:0/3040803373 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56093177ec10 0x56093177d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4d7fff700 1 -- 172.21.15.1:0/3040803373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c0009d30 con 0x5609317500a0 2024-03-18T18:49:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4d7fff700 1 --2- 172.21.15.1:0/3040803373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5609317500a0 0x5609317ed290 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8016000 tx=0x7fb4c8017070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a8fcdce83e3ef27 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4b6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T18:49:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4b6ffd700 1 -- 172.21.15.1:0/3040803373 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c802f040 con 0x5609317500a0 2024-03-18T18:49:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4b6ffd700 1 -- 172.21.15.1:0/3040803373 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb4c8038c60 con 0x5609317500a0 2024-03-18T18:49:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4ea970b80 1 -- 172.21.15.1:0/3040803373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb4c80038b0 con 0x5609317500a0 2024-03-18T18:49:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4b6ffd700 1 -- 172.21.15.1:0/3040803373 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4c8005e70 con 0x5609317500a0 2024-03-18T18:49:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4ea970b80 10 client.4386 fetch_fsmap learned FSMap version 7 2024-03-18T18:49:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4ea970b80 1 -- 172.21.15.1:0/3040803373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5609314f91d0 con 0x5609317500a0 2024-03-18T18:49:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4b6ffd700 1 -- 172.21.15.1:0/3040803373 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c80038b0 con 0x5609317500a0 2024-03-18T18:49:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4b6ffd700 1 -- 172.21.15.1:0/3040803373 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 7) v1 ==== 3122+0+0 (secure 0 0 0) 0x7fb4c8003a20 con 0x5609317500a0 2024-03-18T18:49:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4ea970b80 10 client.4386 fetch_fsmap finished waiting for FSMap version 7 2024-03-18T18:49:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4ea970b80 10 client.4386 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-18T18:49:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4ea970b80 20 client.4386 populate_metadata read hostname 'smithi001' 2024-03-18T18:49:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/3040803373 >> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] conn(0x560931807730 0x560931807b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4ea970b80 4 client.4386 mds_command: new command op to 4273 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T18:49:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.777+0000 7fb4ea970b80 1 -- 172.21.15.1:0/3040803373 --> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5609314de040 con 0x560931807730 2024-03-18T18:49:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.778+0000 7fb4cffff700 1 --2- 172.21.15.1:0/3040803373 >> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] conn(0x560931807730 0x560931807b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.778+0000 7fb4b6ffd700 10 client.4386 ms_handle_connect on v2:172.21.15.1:6835/2118908012 2024-03-18T18:49:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.778+0000 7fb4cffff700 1 --2- 172.21.15.1:0/3040803373 >> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] conn(0x560931807730 0x560931807b10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f6b2d54ed2f43c server_cookie=4ba64d55f7a6b10e in_seq=0 out_seq=0 2024-03-18T18:49:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.778+0000 7fb4b6ffd700 1 -- 172.21.15.1:0/3040803373 <== mds.0 v2:172.21.15.1:6835/2118908012 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x5609314de040 con 0x560931807730 2024-03-18T18:49:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.778+0000 7fb4b6ffd700 10 client.4386 handle_command_reply: tid=0 2024-03-18T18:49:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.778+0000 7fb4ea970b80 1 client.4386 shutdown 2024-03-18T18:49:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.778+0000 7fb4ea970b80 1 -- 172.21.15.1:0/3040803373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5609317500a0 msgr2=0x5609317ed290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.778+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/3040803373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5609317500a0 0x5609317ed290 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8016000 tx=0x7fb4c8017070 comp rx=0 tx=0).stop 2024-03-18T18:49:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.778+0000 7fb4ea970b80 1 -- 172.21.15.1:0/3040803373 shutdown_connections 2024-03-18T18:49:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.778+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/3040803373 >> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] conn(0x560931807730 0x560931807b10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.778+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/3040803373 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x560931790630 0x56093177def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.778+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/3040803373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5609317500a0 0x5609317ed290 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.778+0000 7fb4b6ffd700 0 client.4386 ms_handle_reset on v2:172.21.15.1:6835/2118908012 2024-03-18T18:49:51.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.778+0000 7fb4ea970b80 1 --2- 172.21.15.1:0/3040803373 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56093177ec10 0x56093177d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.778+0000 7fb4ea970b80 1 -- 172.21.15.1:0/3040803373 >> 172.21.15.1:0/3040803373 conn(0x5609316b2cf0 msgr2=0x560931777580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:51.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.778+0000 7fb4ea970b80 1 -- 172.21.15.1:0/3040803373 shutdown_connections 2024-03-18T18:49:51.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.778+0000 7fb4ea970b80 1 -- 172.21.15.1:0/3040803373 wait complete. 2024-03-18T18:49:51.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.779+0000 7fb4ea970b80 20 client.4386 trim_cache size 0 max 16384 2024-03-18T18:49:51.782 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T18:49:51.782 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "9c1c697f-8655-4ef3-8a13-da568d2cbe86", 2024-03-18T18:49:51.782 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-18T18:49:51.782 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 4273, 2024-03-18T18:49:51.782 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-18T18:49:51.783 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-18T18:49:51.783 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-18T18:49:51.783 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 6.9652081319999999, 2024-03-18T18:49:51.783 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 7, 2024-03-18T18:49:51.783 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 25, 2024-03-18T18:49:51.783 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 24, 2024-03-18T18:49:51.783 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 13.556266351 2024-03-18T18:49:51.783 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T18:49:51.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.779+0000 7fb4de7c7700 1 -- 172.21.15.1:0/292199583 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb4b806ac90 msgr2=0x7fb4b806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:51.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.779+0000 7fb4de7c7700 1 --2- 172.21.15.1:0/292199583 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb4b806ac90 0x7fb4b806d150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8156d30 tx=0x7fb4c80091c0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.779+0000 7fb4de7c7700 1 -- 172.21.15.1:0/292199583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4d815cae0 msgr2=0x7fb4d815cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:51.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.779+0000 7fb4de7c7700 1 --2- 172.21.15.1:0/292199583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4d815cae0 0x7fb4d815cf60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0015040 tx=0x7fb4d0001d80 comp rx=0 tx=0).stop 2024-03-18T18:49:51.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.779+0000 7fb4de7c7700 1 -- 172.21.15.1:0/292199583 shutdown_connections 2024-03-18T18:49:51.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.779+0000 7fb4de7c7700 1 --2- 172.21.15.1:0/292199583 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb4b806ac90 0x7fb4b806d150 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.779+0000 7fb4de7c7700 1 --2- 172.21.15.1:0/292199583 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4d81ee4d0 0x7fb4d815a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.779+0000 7fb4de7c7700 1 --2- 172.21.15.1:0/292199583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4d815cae0 0x7fb4d815cf60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.779+0000 7fb4de7c7700 1 --2- 172.21.15.1:0/292199583 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4d815bc30 0x7fb4d814bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.779+0000 7fb4de7c7700 1 -- 172.21.15.1:0/292199583 >> 172.21.15.1:0/292199583 conn(0x7fb4d80cc9b0 msgr2=0x7fb4d806fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:51.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.779+0000 7fb4de7c7700 1 -- 172.21.15.1:0/292199583 shutdown_connections 2024-03-18T18:49:51.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.779+0000 7fb4de7c7700 1 -- 172.21.15.1:0/292199583 wait complete. 2024-03-18T18:49:51.787 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T18:49:51.787 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-18T18:49:51.788 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T18:49:51.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.917+0000 7fee733f2700 1 Processor -- start 2024-03-18T18:49:51.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.917+0000 7fee733f2700 1 -- start start 2024-03-18T18:49:51.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.918+0000 7fee733f2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee6c071600 0x7fee6c071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.918+0000 7fee733f2700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee6c0acd80 con 0x7fee6c06e2c0 2024-03-18T18:49:51.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.918+0000 7fee733f2700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee6c0cf220 con 0x7fee6c15a320 2024-03-18T18:49:51.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.918+0000 7fee733f2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c071fc0 con 0x7fee6c071600 2024-03-18T18:49:51.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.918+0000 7fee7198f700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fee6c06e2c0 0x7fee6c06e750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58346/0 (socket says 172.21.15.1:58346) 2024-03-18T18:49:51.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.918+0000 7fee7198f700 1 -- 172.21.15.1:0/4160616952 learned_addr learned my addr 172.21.15.1:0/4160616952 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:49:51.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.918+0000 7fee7098d700 1 --2- 172.21.15.1:0/4160616952 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee6c071600 0x7fee6c071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.919+0000 7fee63fff700 1 -- 172.21.15.1:0/4160616952 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134105142 0 0) 0x7fee6c0acd80 con 0x7fee6c06e2c0 2024-03-18T18:49:51.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.919+0000 7fee63fff700 1 -- 172.21.15.1:0/4160616952 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee580035b0 con 0x7fee6c06e2c0 2024-03-18T18:49:51.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.919+0000 7fee63fff700 1 -- 172.21.15.1:0/4160616952 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926194346 0 0) 0x7fee68001110 con 0x7fee6c15a320 2024-03-18T18:49:51.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.919+0000 7fee63fff700 1 -- 172.21.15.1:0/4160616952 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee6c0acd80 con 0x7fee6c15a320 2024-03-18T18:49:51.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.919+0000 7fee7098d700 1 -- 172.21.15.1:0/4160616952 >> v1:172.21.15.165:6789/0 conn(0x7fee6c15a320 legacy=0x7fee6c0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:51.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.919+0000 7fee7098d700 1 -- 172.21.15.1:0/4160616952 >> v1:172.21.15.1:6789/0 conn(0x7fee6c06e2c0 legacy=0x7fee6c06e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:51.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.919+0000 7fee7098d700 1 -- 172.21.15.1:0/4160616952 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6c0cef40 con 0x7fee6c071600 2024-03-18T18:49:51.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.919+0000 7fee7098d700 1 --2- 172.21.15.1:0/4160616952 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee6c071600 0x7fee6c071a80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fee5c0098a0 tx=0x7fee5c009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c46aa2cf582e8b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:51.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.919+0000 7fee63fff700 1 -- 172.21.15.1:0/4160616952 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5c011830 con 0x7fee6c071600 2024-03-18T18:49:51.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.919+0000 7fee63fff700 1 -- 172.21.15.1:0/4160616952 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee5c011990 con 0x7fee6c071600 2024-03-18T18:49:51.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.919+0000 7fee63fff700 1 -- 172.21.15.1:0/4160616952 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5c011d20 con 0x7fee6c071600 2024-03-18T18:49:51.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.920+0000 7fee733f2700 1 -- 172.21.15.1:0/4160616952 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee6c071600 msgr2=0x7fee6c071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:51.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.920+0000 7fee733f2700 1 --2- 172.21.15.1:0/4160616952 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee6c071600 0x7fee6c071a80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fee5c0098a0 tx=0x7fee5c009bb0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.920+0000 7fee733f2700 1 -- 172.21.15.1:0/4160616952 shutdown_connections 2024-03-18T18:49:51.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.920+0000 7fee733f2700 1 --2- 172.21.15.1:0/4160616952 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee6c071600 0x7fee6c071a80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.920+0000 7fee733f2700 1 -- 172.21.15.1:0/4160616952 >> 172.21.15.1:0/4160616952 conn(0x7fee6c0cc990 msgr2=0x7fee6c0ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:51.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.920+0000 7fee733f2700 1 -- 172.21.15.1:0/4160616952 shutdown_connections 2024-03-18T18:49:51.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.920+0000 7fee733f2700 1 -- 172.21.15.1:0/4160616952 wait complete. 2024-03-18T18:49:51.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.921+0000 7fee733f2700 1 Processor -- start 2024-03-18T18:49:51.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.921+0000 7fee733f2700 1 -- start start 2024-03-18T18:49:51.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee733f2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee6c06e2c0 0x7fee6c15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee733f2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee6c071600 0x7fee6c0c95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee733f2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee6c0c9b30 0x7fee6c0c3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee733f2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c15b380 con 0x7fee6c071600 2024-03-18T18:49:51.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee733f2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c0c8550 con 0x7fee6c0c9b30 2024-03-18T18:49:51.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee733f2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c0c8830 con 0x7fee6c06e2c0 2024-03-18T18:49:51.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee7098d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee6c071600 0x7fee6c0c95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee7098d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee6c071600 0x7fee6c0c95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51162/0 (socket says 172.21.15.1:51162) 2024-03-18T18:49:51.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee7098d700 1 -- 172.21.15.1:0/3272025932 learned_addr learned my addr 172.21.15.1:0/3272025932 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:51.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee7118e700 1 --2- 172.21.15.1:0/3272025932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee6c06e2c0 0x7fee6c15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee7198f700 1 --2- 172.21.15.1:0/3272025932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee6c0c9b30 0x7fee6c0c3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee7098d700 1 -- 172.21.15.1:0/3272025932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee6c06e2c0 msgr2=0x7fee6c15ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:51.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee7098d700 1 --2- 172.21.15.1:0/3272025932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee6c06e2c0 0x7fee6c15ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee7098d700 1 -- 172.21.15.1:0/3272025932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee6c0c9b30 msgr2=0x7fee6c0c3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:51.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee7098d700 1 --2- 172.21.15.1:0/3272025932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee6c0c9b30 0x7fee6c0c3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:51.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee7098d700 1 -- 172.21.15.1:0/3272025932 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5c0094a0 con 0x7fee6c071600 2024-03-18T18:49:51.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee7198f700 1 --2- 172.21.15.1:0/3272025932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee6c0c9b30 0x7fee6c0c3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:49:51.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee7098d700 1 --2- 172.21.15.1:0/3272025932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee6c071600 0x7fee6c0c95f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fee5c0097c0 tx=0x7fee5c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:51.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee61ffb700 1 -- 172.21.15.1:0/3272025932 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5c0036a0 con 0x7fee6c071600 2024-03-18T18:49:51.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee61ffb700 1 -- 172.21.15.1:0/3272025932 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee5c01f070 con 0x7fee6c071600 2024-03-18T18:49:51.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee733f2700 1 -- 172.21.15.1:0/3272025932 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee6c0c8b10 con 0x7fee6c071600 2024-03-18T18:49:51.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.922+0000 7fee61ffb700 1 -- 172.21.15.1:0/3272025932 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5c003a60 con 0x7fee6c071600 2024-03-18T18:49:51.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.923+0000 7fee733f2700 1 -- 172.21.15.1:0/3272025932 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee6c0c90d0 con 0x7fee6c071600 2024-03-18T18:49:51.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.924+0000 7fee733f2700 1 -- 172.21.15.1:0/3272025932 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee6c0ca600 con 0x7fee6c071600 2024-03-18T18:49:51.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.924+0000 7fee61ffb700 1 -- 172.21.15.1:0/3272025932 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fee5c003bc0 con 0x7fee6c071600 2024-03-18T18:49:51.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.924+0000 7fee61ffb700 1 --2- 172.21.15.1:0/3272025932 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee5006ac90 0x7fee5006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:51.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.924+0000 7fee7118e700 1 --2- 172.21.15.1:0/3272025932 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee5006ac90 0x7fee5006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:51.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.924+0000 7fee61ffb700 1 -- 172.21.15.1:0/3272025932 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fee5c021070 con 0x7fee6c071600 2024-03-18T18:49:51.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.925+0000 7fee7118e700 1 --2- 172.21.15.1:0/3272025932 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee5006ac90 0x7fee5006d150 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fee68001270 tx=0x7fee68009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:51.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:51.929+0000 7fee61ffb700 1 -- 172.21.15.1:0/3272025932 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fee5c055ed0 con 0x7fee6c071600 2024-03-18T18:49:52.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:52.179+0000 7fee733f2700 1 -- 172.21.15.1:0/3272025932 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fee6c0676a0 con 0x7fee6c071600 2024-03-18T18:49:52.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:52.179+0000 7fee61ffb700 1 -- 172.21.15.1:0/3272025932 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7fee5c02b070 con 0x7fee6c071600 2024-03-18T18:49:52.175 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T18:49:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:52.181+0000 7fee733f2700 1 -- 172.21.15.1:0/3272025932 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee5006ac90 msgr2=0x7fee5006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:52.181+0000 7fee733f2700 1 --2- 172.21.15.1:0/3272025932 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee5006ac90 0x7fee5006d150 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fee68001270 tx=0x7fee68009040 comp rx=0 tx=0).stop 2024-03-18T18:49:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:52.181+0000 7fee733f2700 1 -- 172.21.15.1:0/3272025932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee6c071600 msgr2=0x7fee6c0c95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:52.181+0000 7fee733f2700 1 --2- 172.21.15.1:0/3272025932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee6c071600 0x7fee6c0c95f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fee5c0097c0 tx=0x7fee5c004750 comp rx=0 tx=0).stop 2024-03-18T18:49:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:52.181+0000 7fee733f2700 1 -- 172.21.15.1:0/3272025932 shutdown_connections 2024-03-18T18:49:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:52.181+0000 7fee733f2700 1 --2- 172.21.15.1:0/3272025932 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee5006ac90 0x7fee5006d150 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:52.181+0000 7fee733f2700 1 --2- 172.21.15.1:0/3272025932 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee6c06e2c0 0x7fee6c15ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:52.181+0000 7fee733f2700 1 --2- 172.21.15.1:0/3272025932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee6c0c9b30 0x7fee6c0c3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:52.181+0000 7fee733f2700 1 --2- 172.21.15.1:0/3272025932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee6c071600 0x7fee6c0c95f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:52.181+0000 7fee733f2700 1 -- 172.21.15.1:0/3272025932 >> 172.21.15.1:0/3272025932 conn(0x7fee6c0cc990 msgr2=0x7fee6c0cd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:52.181+0000 7fee733f2700 1 -- 172.21.15.1:0/3272025932 shutdown_connections 2024-03-18T18:49:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:52.181+0000 7fee733f2700 1 -- 172.21.15.1:0/3272025932 wait complete. 2024-03-18T18:49:52.189 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-18T18:49:52.189 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T18:49:52.190 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-18T18:49:52.190 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-18T18:49:52.190 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-18T18:49:52.190 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-18T18:49:52.190 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T18:49:52.190 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T18:49:52.190 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-18T18:49:52.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:49:52.190 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-18T18:49:52.209 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T18:49:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T18:49:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T18:49:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:49:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T18:49:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:49:52.210 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T18:49:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-18T18:49:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-18T18:49:52.210 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T18:49:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-18T18:49:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-18T18:49:52.210 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T18:49:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-18T18:49:52.210 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-18T18:49:52.211 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T18:49:52.211 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 42577sec preferred_lft 42577sec 2024-03-18T18:49:52.211 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link 2024-03-18T18:49:52.211 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:49:52.211 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T18:49:52.211 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-18T18:49:52.211 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-18T18:49:52.212 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T18:49:52.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:49:52.212 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T18:49:52.212 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T18:49:52.212 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add name ceph-brx type bridge 2024-03-18T18:49:52.212 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr flush dev ceph-brx 2024-03-18T18:49:52.212 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set ceph-brx up 2024-03-18T18:49:52.212 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T18:49:52.212 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T18:49:52.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:49:52.391 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T18:49:52.425 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-18T18:49:52.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:49:52.427 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-18T18:49:52.506 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-18T18:49:52.506 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T18:49:52.506 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T18:49:52.506 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T18:49:52.507 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T18:49:52.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:49:52.507 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T18:49:52.507 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T18:49:52.507 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T18:49:52.507 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T18:49:52.507 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T18:49:52.508 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T18:49:52.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:49:52.804 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T18:49:52.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:49:52.824 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list-id 2024-03-18T18:49:52.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:49:52.883 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T18:49:52.883 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T18:49:52.883 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T18:49:52.883 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-18T18:49:52.883 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T18:49:52.979 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-18T18:49:52.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:49:52.980 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T18:49:52.980 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T18:49:52.980 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-18T18:49:52.980 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T18:49:52.980 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-18T18:49:52.980 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-18T18:49:52.980 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-18T18:49:52.980 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T18:49:53.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:49:53.181 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T18:49:53.181 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T18:49:53.181 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-18T18:49:53.181 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T18:49:53.182 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T18:49:53.280 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-18T18:49:53.280 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T18:49:53.280 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-18T18:49:53.299 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-18T18:49:53.299 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T18:49:53.299 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-18T18:49:53.357 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T18:49:53.425 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T18:49:53.426 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T18:49:53.493 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T18:49:53.494 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T18:49:53.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:49:53.495 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T18:49:53.553 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T18:49:53.553 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-18T18:49:53.596 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T18:49:53.623 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T18:49:53.623 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T18:49:53.685 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi001.stderr:2024-03-18T18:49:53.689+0000 7f75400c95c0 -1 init, newargv = 0x5577e4464d90 newargc=15 2024-03-18T18:49:53.686 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi001.stderr:ceph-fuse[84348]: starting ceph client 2024-03-18T18:49:53.692 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi001.stderr:ceph-fuse[84348]: starting fuse 2024-03-18T18:49:53.692 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T18:49:53.693 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T18:49:53.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:49:53.694 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T18:49:53.753 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T18:49:53.754 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-03-18T18:49:53.754 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T18:49:53.754 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-18T18:49:53.754 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-18T18:49:53.754 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-18T18:49:53.755 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-18T18:49:53.755 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T18:49:53.755 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-18T18:49:53.755 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-18T18:49:53.755 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-18T18:49:53.755 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-18T18:49:53.756 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T18:49:53.756 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-18T18:49:53.756 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-18T18:49:53.756 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-18T18:49:53.756 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-18T18:49:53.756 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T18:49:53.757 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-18T18:49:53.757 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T18:49:53.757 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T18:49:53.757 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T18:49:53.758 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-18T18:49:53.758 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-18T18:49:53.758 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-18T18:49:53.758 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T18:49:53.758 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T18:49:53.758 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.admin")) 2024-03-18T18:49:53.759 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-18T18:49:53.879 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.admin.84348.asok 2024-03-18T18:49:54.062 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.84348.asok 2024-03-18T18:49:54.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:49:54.062 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.84348.asok status 2024-03-18T18:49:54.224 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T18:49:54.224 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-18T18:49:54.224 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T18:49:54.225 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T18:49:54.225 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "admin", 2024-03-18T18:49:54.225 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-18T18:49:54.225 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-18T18:49:54.225 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "84348", 2024-03-18T18:49:54.225 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-18T18:49:54.225 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:49:54.225 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-18T18:49:54.225 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-18T18:49:54.225 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 4398, 2024-03-18T18:49:54.225 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-18T18:49:54.225 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-18T18:49:54.225 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-18T18:49:54.225 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4398 2024-03-18T18:49:54.226 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:49:54.226 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T18:49:54.226 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T18:49:54.226 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T18:49:54.226 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 476171360 2024-03-18T18:49:54.226 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:49:54.226 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:49:54.226 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T18:49:54.226 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T18:49:54.226 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T18:49:54.226 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 476171360 2024-03-18T18:49:54.226 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:49:54.226 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.4398 192.168.0.1:0/476171360", 2024-03-18T18:49:54.227 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/476171360", 2024-03-18T18:49:54.227 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-18T18:49:54.227 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 7, 2024-03-18T18:49:54.227 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 25, 2024-03-18T18:49:54.227 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-18T18:49:54.227 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-18T18:49:54.227 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-18T18:49:54.227 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T18:49:54.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:49:54.235 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-18T18:49:54.235 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-18T18:49:54.255 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-18T18:49:54.255 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-18T18:49:54.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:49:54.255 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-18T18:49:54.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:49:54.324 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-18T18:49:54.440 INFO:teuthology.orchestra.run.smithi001.stdout:# file: . 2024-03-18T18:49:54.440 INFO:teuthology.orchestra.run.smithi001.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-18T18:49:54.440 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:49:54.441 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:49:54.441 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T18:49:54.464 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:49:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.632+0000 7f038af8a700 1 Processor -- start 2024-03-18T18:49:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.632+0000 7f038af8a700 1 -- start start 2024-03-18T18:49:54.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.633+0000 7f038af8a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03840ae580 0x7f0384070490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:54.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.633+0000 7f038af8a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03840abd70 0x7f03840709d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:54.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.633+0000 7f038af8a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03840ac230 con 0x7f03840716b0 2024-03-18T18:49:54.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.633+0000 7f038af8a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f038406c620 con 0x7f03840abd70 2024-03-18T18:49:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.633+0000 7f038af8a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f038406c900 con 0x7f03840ae580 2024-03-18T18:49:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.633+0000 7f0389527700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03840abd70 0x7f03840709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.633+0000 7f0389527700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03840abd70 0x7f03840709d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49292/0 (socket says 172.21.15.1:49292) 2024-03-18T18:49:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.633+0000 7f0389527700 1 -- 172.21.15.1:0/432296729 learned_addr learned my addr 172.21.15.1:0/432296729 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.633+0000 7f0383fff700 1 --2- 172.21.15.1:0/432296729 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03840ae580 0x7f0384070490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.634+0000 7f0389527700 1 -- 172.21.15.1:0/432296729 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03840ae580 msgr2=0x7f0384070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.634+0000 7f0389527700 1 --2- 172.21.15.1:0/432296729 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03840ae580 0x7f0384070490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.634+0000 7f0389527700 1 -- 172.21.15.1:0/432296729 >> v1:172.21.15.165:6789/0 conn(0x7f03840716b0 legacy=0x7f0384071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.634+0000 7f0389527700 1 -- 172.21.15.1:0/432296729 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038406cbe0 con 0x7f03840abd70 2024-03-18T18:49:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.634+0000 7f0389527700 1 --2- 172.21.15.1:0/432296729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03840abd70 0x7f03840709d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f037c009e50 tx=0x7f037c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5099978d64f490ad server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.634+0000 7f03837fe700 1 -- 172.21.15.1:0/432296729 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c00e070 con 0x7f03840abd70 2024-03-18T18:49:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.634+0000 7f03837fe700 1 -- 172.21.15.1:0/432296729 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037c003a20 con 0x7f03840abd70 2024-03-18T18:49:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.634+0000 7f03837fe700 1 -- 172.21.15.1:0/432296729 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c003dd0 con 0x7f03840abd70 2024-03-18T18:49:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.634+0000 7f038af8a700 1 -- 172.21.15.1:0/432296729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03840abd70 msgr2=0x7f03840709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.634+0000 7f038af8a700 1 --2- 172.21.15.1:0/432296729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03840abd70 0x7f03840709d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f037c009e50 tx=0x7f037c00b330 comp rx=0 tx=0).stop 2024-03-18T18:49:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.634+0000 7f038af8a700 1 -- 172.21.15.1:0/432296729 shutdown_connections 2024-03-18T18:49:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.634+0000 7f038af8a700 1 --2- 172.21.15.1:0/432296729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03840abd70 0x7f03840709d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.634+0000 7f038af8a700 1 --2- 172.21.15.1:0/432296729 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03840ae580 0x7f0384070490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.634+0000 7f038af8a700 1 -- 172.21.15.1:0/432296729 >> 172.21.15.1:0/432296729 conn(0x7f03840cee10 msgr2=0x7f03840cf220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.635+0000 7f038af8a700 1 -- 172.21.15.1:0/432296729 shutdown_connections 2024-03-18T18:49:54.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.635+0000 7f038af8a700 1 -- 172.21.15.1:0/432296729 wait complete. 2024-03-18T18:49:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.636+0000 7f038af8a700 1 Processor -- start 2024-03-18T18:49:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.636+0000 7f038af8a700 1 -- start start 2024-03-18T18:49:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f038af8a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03840716b0 0x7f0384144880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f038af8a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03840abd70 0x7f038409be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f038af8a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03841f4e40 0x7f038409c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f038af8a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0384144dc0 con 0x7f03840abd70 2024-03-18T18:49:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f038af8a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03841569e0 con 0x7f03841f4e40 2024-03-18T18:49:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f038af8a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03841f4360 con 0x7f03840716b0 2024-03-18T18:49:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f0383fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03840abd70 0x7f038409be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f0383fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03840abd70 0x7f038409be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49298/0 (socket says 172.21.15.1:49298) 2024-03-18T18:49:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f0383fff700 1 -- 172.21.15.1:0/1430808297 learned_addr learned my addr 172.21.15.1:0/1430808297 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f0388d26700 1 --2- 172.21.15.1:0/1430808297 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03840716b0 0x7f0384144880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:54.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f0389527700 1 --2- 172.21.15.1:0/1430808297 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03841f4e40 0x7f038409c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:54.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f0383fff700 1 -- 172.21.15.1:0/1430808297 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03840716b0 msgr2=0x7f0384144880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:54.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f0383fff700 1 --2- 172.21.15.1:0/1430808297 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03840716b0 0x7f0384144880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:54.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f0383fff700 1 -- 172.21.15.1:0/1430808297 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03841f4e40 msgr2=0x7f038409c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:54.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f0383fff700 1 --2- 172.21.15.1:0/1430808297 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03841f4e40 0x7f038409c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:54.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f0383fff700 1 -- 172.21.15.1:0/1430808297 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037c009b30 con 0x7f03840abd70 2024-03-18T18:49:54.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f0383fff700 1 --2- 172.21.15.1:0/1430808297 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03840abd70 0x7f038409be50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f037000c7f0 tx=0x7f037000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:54.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f03817fa700 1 -- 172.21.15.1:0/1430808297 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0370004220 con 0x7f03840abd70 2024-03-18T18:49:54.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f03817fa700 1 -- 172.21.15.1:0/1430808297 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0370004380 con 0x7f03840abd70 2024-03-18T18:49:54.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f038af8a700 1 -- 172.21.15.1:0/1430808297 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03841f4640 con 0x7f03840abd70 2024-03-18T18:49:54.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.637+0000 7f03817fa700 1 -- 172.21.15.1:0/1430808297 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03700036a0 con 0x7f03840abd70 2024-03-18T18:49:54.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.638+0000 7f038af8a700 1 -- 172.21.15.1:0/1430808297 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0384155610 con 0x7f03840abd70 2024-03-18T18:49:54.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.638+0000 7f038af8a700 1 -- 172.21.15.1:0/1430808297 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03841f41c0 con 0x7f03840abd70 2024-03-18T18:49:54.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.639+0000 7f03817fa700 1 -- 172.21.15.1:0/1430808297 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0370003800 con 0x7f03840abd70 2024-03-18T18:49:54.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.639+0000 7f03817fa700 1 --2- 172.21.15.1:0/1430808297 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f036806ac90 0x7f036806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:54.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.640+0000 7f03817fa700 1 -- 172.21.15.1:0/1430808297 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0370089060 con 0x7f03840abd70 2024-03-18T18:49:54.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.640+0000 7f0388d26700 1 --2- 172.21.15.1:0/1430808297 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f036806ac90 0x7f036806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:54.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.640+0000 7f0388d26700 1 --2- 172.21.15.1:0/1430808297 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f036806ac90 0x7f036806d150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0378007fd0 tx=0x7f0378007ec0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:54.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.644+0000 7f03817fa700 1 -- 172.21.15.1:0/1430808297 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f03841f41c0 con 0x7f03840abd70 2024-03-18T18:49:54.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.848+0000 7f038af8a700 1 -- 172.21.15.1:0/1430808297 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03841f41c0 con 0x7f03840abd70 2024-03-18T18:49:54.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.849+0000 7f03817fa700 1 -- 172.21.15.1:0/1430808297 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17830 (secure 0 0 0) 0x7f03841f41c0 con 0x7f03840abd70 2024-03-18T18:49:54.846 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:49:54.846 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:49.276961+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:49:54.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.851+0000 7f038af8a700 1 -- 172.21.15.1:0/1430808297 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f036806ac90 msgr2=0x7f036806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:54.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.851+0000 7f038af8a700 1 --2- 172.21.15.1:0/1430808297 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f036806ac90 0x7f036806d150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0378007fd0 tx=0x7f0378007ec0 comp rx=0 tx=0).stop 2024-03-18T18:49:54.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.851+0000 7f038af8a700 1 -- 172.21.15.1:0/1430808297 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03840abd70 msgr2=0x7f038409be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:54.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.851+0000 7f038af8a700 1 --2- 172.21.15.1:0/1430808297 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03840abd70 0x7f038409be50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f037000c7f0 tx=0x7f037000cb00 comp rx=0 tx=0).stop 2024-03-18T18:49:54.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.851+0000 7f038af8a700 1 -- 172.21.15.1:0/1430808297 shutdown_connections 2024-03-18T18:49:54.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.851+0000 7f038af8a700 1 --2- 172.21.15.1:0/1430808297 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f036806ac90 0x7f036806d150 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:54.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.851+0000 7f038af8a700 1 --2- 172.21.15.1:0/1430808297 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03840716b0 0x7f0384144880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:54.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.851+0000 7f038af8a700 1 --2- 172.21.15.1:0/1430808297 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03841f4e40 0x7f038409c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:54.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.851+0000 7f038af8a700 1 --2- 172.21.15.1:0/1430808297 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03840abd70 0x7f038409be50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:54.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.851+0000 7f038af8a700 1 -- 172.21.15.1:0/1430808297 >> 172.21.15.1:0/1430808297 conn(0x7f03840cee10 msgr2=0x7f0384071a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:54.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.851+0000 7f038af8a700 1 -- 172.21.15.1:0/1430808297 shutdown_connections 2024-03-18T18:49:54.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.851+0000 7f038af8a700 1 -- 172.21.15.1:0/1430808297 wait complete. 2024-03-18T18:49:54.860 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T18:49:42.800801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T18:49:43.776684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T18:49:46.854608+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T18:49:54.860 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T18:49:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.992+0000 7f0cf676b700 1 Processor -- start 2024-03-18T18:49:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.992+0000 7f0cf676b700 1 -- start start 2024-03-18T18:49:54.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.993+0000 7f0cf676b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0cf00cc9c0 0x7f0cf0071570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:54.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.993+0000 7f0cf676b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0cf006a750 0x7f0cf006ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:54.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.993+0000 7f0cf676b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf006ef60 con 0x7f0cf0144ac0 2024-03-18T18:49:54.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.993+0000 7f0cf676b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf00cfdc0 con 0x7f0cf006a750 2024-03-18T18:49:54.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.993+0000 7f0cf676b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf006b0b0 con 0x7f0cf00cc9c0 2024-03-18T18:49:54.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.993+0000 7f0cf4d08700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0cf006a750 0x7f0cf006ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:54.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.993+0000 7f0cf4d08700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0cf006a750 0x7f0cf006ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49312/0 (socket says 172.21.15.1:49312) 2024-03-18T18:49:54.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.993+0000 7f0cf4d08700 1 -- 172.21.15.1:0/2377099038 learned_addr learned my addr 172.21.15.1:0/2377099038 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:54.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.993+0000 7f0cef7fe700 1 --2- 172.21.15.1:0/2377099038 >> v2:172.21.15.165:3300/0 conn(0x7f0cf00cc9c0 0x7f0cf0071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:54.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.993+0000 7f0cf4d08700 1 -- 172.21.15.1:0/2377099038 >> v2:172.21.15.165:3300/0 conn(0x7f0cf00cc9c0 msgr2=0x7f0cf0071570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:54.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.993+0000 7f0cf4d08700 1 --2- 172.21.15.1:0/2377099038 >> v2:172.21.15.165:3300/0 conn(0x7f0cf00cc9c0 0x7f0cf0071570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:54.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.993+0000 7f0cf4d08700 1 -- 172.21.15.1:0/2377099038 >> v1:172.21.15.165:6789/0 conn(0x7f0cf0144ac0 legacy=0x7f0cf00cf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:54.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.993+0000 7f0cf4d08700 1 -- 172.21.15.1:0/2377099038 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf0144ea0 con 0x7f0cf006a750 2024-03-18T18:49:54.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.993+0000 7f0cf4d08700 1 --2- 172.21.15.1:0/2377099038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf006a750 0x7f0cf006ab70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0cf00b2ab0 tx=0x7f0ce800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3655626ae60beef6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:54.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.993+0000 7f0ceeffd700 1 -- 172.21.15.1:0/2377099038 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce800e070 con 0x7f0cf006a750 2024-03-18T18:49:54.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.993+0000 7f0ceeffd700 1 -- 172.21.15.1:0/2377099038 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce800b9c0 con 0x7f0cf006a750 2024-03-18T18:49:54.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.993+0000 7f0ceeffd700 1 -- 172.21.15.1:0/2377099038 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce800bd70 con 0x7f0cf006a750 2024-03-18T18:49:54.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.994+0000 7f0cf676b700 1 -- 172.21.15.1:0/2377099038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf006a750 msgr2=0x7f0cf006ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:54.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.994+0000 7f0cf676b700 1 --2- 172.21.15.1:0/2377099038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf006a750 0x7f0cf006ab70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0cf00b2ab0 tx=0x7f0ce800b330 comp rx=0 tx=0).stop 2024-03-18T18:49:54.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.994+0000 7f0cf676b700 1 -- 172.21.15.1:0/2377099038 shutdown_connections 2024-03-18T18:49:54.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.994+0000 7f0cf676b700 1 --2- 172.21.15.1:0/2377099038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf006a750 0x7f0cf006ab70 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:54.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.994+0000 7f0cf676b700 1 --2- 172.21.15.1:0/2377099038 >> v2:172.21.15.165:3300/0 conn(0x7f0cf00cc9c0 0x7f0cf0071570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:54.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.994+0000 7f0cf676b700 1 -- 172.21.15.1:0/2377099038 >> 172.21.15.1:0/2377099038 conn(0x7f0cf00b1d20 msgr2=0x7f0cf00b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:54.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.994+0000 7f0cf676b700 1 -- 172.21.15.1:0/2377099038 shutdown_connections 2024-03-18T18:49:54.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.994+0000 7f0cf676b700 1 -- 172.21.15.1:0/2377099038 wait complete. 2024-03-18T18:49:54.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.995+0000 7f0cf676b700 1 Processor -- start 2024-03-18T18:49:54.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.995+0000 7f0cf676b700 1 -- start start 2024-03-18T18:49:54.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.996+0000 7f0cf676b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf006a750 0x7f0cf013c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:54.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.996+0000 7f0cf676b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cf00cc9c0 0x7f0cf0137810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:54.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.996+0000 7f0cf676b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0cf0144ac0 0x7f0cf0139020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:54.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.996+0000 7f0cf676b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf0137d50 con 0x7f0cf006a750 2024-03-18T18:49:54.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.996+0000 7f0cf676b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf0138b90 con 0x7f0cf00cc9c0 2024-03-18T18:49:54.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.996+0000 7f0cf676b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf0139560 con 0x7f0cf0144ac0 2024-03-18T18:49:54.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.996+0000 7f0ceffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf006a750 0x7f0cf013c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:54.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.996+0000 7f0ceffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf006a750 0x7f0cf013c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49320/0 (socket says 172.21.15.1:49320) 2024-03-18T18:49:54.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.996+0000 7f0ceffff700 1 -- 172.21.15.1:0/3023245472 learned_addr learned my addr 172.21.15.1:0/3023245472 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:54.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.996+0000 7f0cef7fe700 1 --2- 172.21.15.1:0/3023245472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cf00cc9c0 0x7f0cf0137810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:54.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.996+0000 7f0cf4d08700 1 --2- 172.21.15.1:0/3023245472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0cf0144ac0 0x7f0cf0139020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:54.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.997+0000 7f0ceffff700 1 -- 172.21.15.1:0/3023245472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0cf0144ac0 msgr2=0x7f0cf0139020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:54.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.997+0000 7f0ceffff700 1 --2- 172.21.15.1:0/3023245472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0cf0144ac0 0x7f0cf0139020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:54.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.997+0000 7f0ceffff700 1 -- 172.21.15.1:0/3023245472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cf00cc9c0 msgr2=0x7f0cf0137810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:54.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.997+0000 7f0ceffff700 1 --2- 172.21.15.1:0/3023245472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cf00cc9c0 0x7f0cf0137810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:54.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.997+0000 7f0ceffff700 1 -- 172.21.15.1:0/3023245472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce8009b30 con 0x7f0cf006a750 2024-03-18T18:49:54.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.997+0000 7f0ceffff700 1 --2- 172.21.15.1:0/3023245472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf006a750 0x7f0cf013c270 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0ce000c4d0 tx=0x7f0ce000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:54.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.997+0000 7f0cecff9700 1 -- 172.21.15.1:0/3023245472 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce00052d0 con 0x7f0cf006a750 2024-03-18T18:49:54.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.997+0000 7f0cecff9700 1 -- 172.21.15.1:0/3023245472 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce0005430 con 0x7f0cf006a750 2024-03-18T18:49:54.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.997+0000 7f0cecff9700 1 -- 172.21.15.1:0/3023245472 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce00038a0 con 0x7f0cf006a750 2024-03-18T18:49:54.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.997+0000 7f0cf676b700 1 -- 172.21.15.1:0/3023245472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf0137f80 con 0x7f0cf006a750 2024-03-18T18:49:54.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.997+0000 7f0cf676b700 1 -- 172.21.15.1:0/3023245472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf0138540 con 0x7f0cf006a750 2024-03-18T18:49:54.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.998+0000 7f0cf676b700 1 -- 172.21.15.1:0/3023245472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf0138e70 con 0x7f0cf006a750 2024-03-18T18:49:54.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.998+0000 7f0cecff9700 1 -- 172.21.15.1:0/3023245472 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0ce0003a00 con 0x7f0cf006a750 2024-03-18T18:49:54.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.999+0000 7f0cecff9700 1 --2- 172.21.15.1:0/3023245472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0cdc06acc0 0x7f0cdc06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:54.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.999+0000 7f0cecff9700 1 -- 172.21.15.1:0/3023245472 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0ce0088af0 con 0x7f0cf006a750 2024-03-18T18:49:54.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.999+0000 7f0cef7fe700 1 --2- 172.21.15.1:0/3023245472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0cdc06acc0 0x7f0cdc06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:54.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:54.999+0000 7f0cef7fe700 1 --2- 172.21.15.1:0/3023245472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0cdc06acc0 0x7f0cdc06d180 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0139ab0 tx=0x7f0ce4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:55.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.003+0000 7f0cecff9700 1 -- 172.21.15.1:0/3023245472 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0ce00538b0 con 0x7f0cf006a750 2024-03-18T18:49:55.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.205+0000 7f0cf676b700 1 -- 172.21.15.1:0/3023245472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0cf00676a0 con 0x7f0cf006a750 2024-03-18T18:49:55.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.205+0000 7f0cecff9700 1 -- 172.21.15.1:0/3023245472 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0ce0017020 con 0x7f0cf006a750 2024-03-18T18:49:55.201 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T18:49:55.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.207+0000 7f0cf676b700 1 -- 172.21.15.1:0/3023245472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0cdc06acc0 msgr2=0x7f0cdc06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:55.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.207+0000 7f0cf676b700 1 --2- 172.21.15.1:0/3023245472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0cdc06acc0 0x7f0cdc06d180 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0139ab0 tx=0x7f0ce4008040 comp rx=0 tx=0).stop 2024-03-18T18:49:55.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.207+0000 7f0cf676b700 1 -- 172.21.15.1:0/3023245472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf006a750 msgr2=0x7f0cf013c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:55.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.207+0000 7f0cf676b700 1 --2- 172.21.15.1:0/3023245472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf006a750 0x7f0cf013c270 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0ce000c4d0 tx=0x7f0ce000c7e0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.207+0000 7f0cf676b700 1 -- 172.21.15.1:0/3023245472 shutdown_connections 2024-03-18T18:49:55.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.207+0000 7f0cf676b700 1 --2- 172.21.15.1:0/3023245472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0cdc06acc0 0x7f0cdc06d180 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.207+0000 7f0cf676b700 1 --2- 172.21.15.1:0/3023245472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0cf0144ac0 0x7f0cf0139020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.207+0000 7f0cf676b700 1 --2- 172.21.15.1:0/3023245472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf006a750 0x7f0cf013c270 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.207+0000 7f0cf676b700 1 --2- 172.21.15.1:0/3023245472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cf00cc9c0 0x7f0cf0137810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.207+0000 7f0cf676b700 1 -- 172.21.15.1:0/3023245472 >> 172.21.15.1:0/3023245472 conn(0x7f0cf00b1d20 msgr2=0x7f0cf00b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:55.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.207+0000 7f0cf676b700 1 -- 172.21.15.1:0/3023245472 shutdown_connections 2024-03-18T18:49:55.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.207+0000 7f0cf676b700 1 -- 172.21.15.1:0/3023245472 wait complete. 2024-03-18T18:49:55.215 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T18:49:55.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.347+0000 7fe5d43c2700 1 Processor -- start 2024-03-18T18:49:55.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.347+0000 7fe5d43c2700 1 -- start start 2024-03-18T18:49:55.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.348+0000 7fe5d43c2700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe5cc15d5a0 0x7fe5cc0ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:55.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.348+0000 7fe5d43c2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5cc06ae60 0x7fe5cc167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:55.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.348+0000 7fe5d43c2700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5cc0cf600 con 0x7fe5cc0ce550 2024-03-18T18:49:55.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.348+0000 7fe5d43c2700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5cc0ce9f0 con 0x7fe5cc15d5a0 2024-03-18T18:49:55.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.348+0000 7fe5d43c2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc0cecd0 con 0x7fe5cc06ae60 2024-03-18T18:49:55.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.348+0000 7fe5d195d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe5cc0ce550 0x7fe5cc06a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54410/0 (socket says 172.21.15.1:54410) 2024-03-18T18:49:55.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.348+0000 7fe5d195d700 1 -- 172.21.15.1:0/3068078500 learned_addr learned my addr 172.21.15.1:0/3068078500 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:49:55.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d215e700 1 --2- 172.21.15.1:0/3068078500 >> v2:172.21.15.165:3300/0 conn(0x7fe5cc15d5a0 0x7fe5cc0ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:55.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d295f700 1 --2- 172.21.15.1:0/3068078500 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5cc06ae60 0x7fe5cc167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:55.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d115c700 1 -- 172.21.15.1:0/3068078500 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018032433 0 0) 0x7fe5cc0cf600 con 0x7fe5cc0ce550 2024-03-18T18:49:55.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d115c700 1 -- 172.21.15.1:0/3068078500 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5b80035b0 con 0x7fe5cc0ce550 2024-03-18T18:49:55.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d115c700 1 -- 172.21.15.1:0/3068078500 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3996960654 0 0) 0x7fe5b80035b0 con 0x7fe5cc0ce550 2024-03-18T18:49:55.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d115c700 1 -- 172.21.15.1:0/3068078500 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5cc06ae60 msgr2=0x7fe5cc167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:55.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d115c700 1 --2- 172.21.15.1:0/3068078500 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5cc06ae60 0x7fe5cc167140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d115c700 1 -- 172.21.15.1:0/3068078500 >> v2:172.21.15.165:3300/0 conn(0x7fe5cc15d5a0 msgr2=0x7fe5cc0ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:55.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d115c700 1 --2- 172.21.15.1:0/3068078500 >> v2:172.21.15.165:3300/0 conn(0x7fe5cc15d5a0 0x7fe5cc0ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d115c700 1 -- 172.21.15.1:0/3068078500 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5cc0cefb0 con 0x7fe5cc0ce550 2024-03-18T18:49:55.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d115c700 1 -- 172.21.15.1:0/3068078500 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fe5c8001670 con 0x7fe5cc0ce550 2024-03-18T18:49:55.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d115c700 1 -- 172.21.15.1:0/3068078500 >> v1:172.21.15.1:6789/0 conn(0x7fe5cc0ce550 legacy=0x7fe5cc06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:55.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d115c700 1 --2- 172.21.15.1:0/3068078500 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5b8003a00 0x7fe5b8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:55.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d115c700 1 --2- 172.21.15.1:0/3068078500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b80043f0 0x7fe5b80048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:55.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d115c700 1 --2- 172.21.15.1:0/3068078500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5b8004f40 0x7fe5b800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:55.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d115c700 1 -- 172.21.15.1:0/3068078500 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc0ce9f0 con 0x7fe5b80043f0 2024-03-18T18:49:55.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d115c700 1 -- 172.21.15.1:0/3068078500 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc0cecd0 con 0x7fe5b8004f40 2024-03-18T18:49:55.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d115c700 1 -- 172.21.15.1:0/3068078500 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b800bf50 con 0x7fe5b8003a00 2024-03-18T18:49:55.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d295f700 1 --2- 172.21.15.1:0/3068078500 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5cc06ae60 0x7fe5cc167140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:49:55.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d215e700 1 --2- 172.21.15.1:0/3068078500 >> v2:172.21.15.165:3300/0 conn(0x7fe5cc15d5a0 0x7fe5cc0ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:49:55.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d195d700 1 --2- 172.21.15.1:0/3068078500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b80043f0 0x7fe5b80048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:55.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d295f700 1 --2- 172.21.15.1:0/3068078500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5b8004f40 0x7fe5b800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:55.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d215e700 1 --2- 172.21.15.1:0/3068078500 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5b8003a00 0x7fe5b8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:55.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d195d700 1 -- 172.21.15.1:0/3068078500 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5b8003a00 msgr2=0x7fe5b8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:55.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d195d700 1 --2- 172.21.15.1:0/3068078500 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5b8003a00 0x7fe5b8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d195d700 1 -- 172.21.15.1:0/3068078500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5b8004f40 msgr2=0x7fe5b800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:55.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d195d700 1 --2- 172.21.15.1:0/3068078500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5b8004f40 0x7fe5b800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d195d700 1 -- 172.21.15.1:0/3068078500 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b80035b0 con 0x7fe5b80043f0 2024-03-18T18:49:55.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.349+0000 7fe5d215e700 1 --2- 172.21.15.1:0/3068078500 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5b8003a00 0x7fe5b8003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.350+0000 7fe5d195d700 1 --2- 172.21.15.1:0/3068078500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b80043f0 0x7fe5b80048d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe5c800c320 tx=0x7fe5b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7686c7923b1769ac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.350+0000 7fe5d115c700 1 -- 172.21.15.1:0/3068078500 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c800ce60 con 0x7fe5b80043f0 2024-03-18T18:49:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.350+0000 7fe5d115c700 1 -- 172.21.15.1:0/3068078500 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c8005960 con 0x7fe5b80043f0 2024-03-18T18:49:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.350+0000 7fe5d115c700 1 -- 172.21.15.1:0/3068078500 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8005cf0 con 0x7fe5b80043f0 2024-03-18T18:49:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.350+0000 7fe5d43c2700 1 -- 172.21.15.1:0/3068078500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b80043f0 msgr2=0x7fe5b80048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.350+0000 7fe5d43c2700 1 --2- 172.21.15.1:0/3068078500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b80043f0 0x7fe5b80048d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe5c800c320 tx=0x7fe5b8005880 comp rx=0 tx=0).stop 2024-03-18T18:49:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.350+0000 7fe5d295f700 1 -- 172.21.15.1:0/3068078500 reap_dead start 2024-03-18T18:49:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.350+0000 7fe5d43c2700 1 -- 172.21.15.1:0/3068078500 shutdown_connections 2024-03-18T18:49:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.350+0000 7fe5d43c2700 1 -- 172.21.15.1:0/3068078500 >> 172.21.15.1:0/3068078500 conn(0x7fe5cc0cc9b0 msgr2=0x7fe5cc0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:55.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.350+0000 7fe5d43c2700 1 -- 172.21.15.1:0/3068078500 shutdown_connections 2024-03-18T18:49:55.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.350+0000 7fe5d43c2700 1 -- 172.21.15.1:0/3068078500 wait complete. 2024-03-18T18:49:55.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.352+0000 7fe5d43c2700 1 Processor -- start 2024-03-18T18:49:55.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.352+0000 7fe5d43c2700 1 -- start start 2024-03-18T18:49:55.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.352+0000 7fe5d43c2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b80043f0 0x7fe5cc0c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:55.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.352+0000 7fe5d43c2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5cc159c30 0x7fe5cc149a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:55.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.352+0000 7fe5d43c2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5cc149fc0 0x7fe5cc0b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:55.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.352+0000 7fe5d43c2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc0b0f00 con 0x7fe5b80043f0 2024-03-18T18:49:55.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.352+0000 7fe5d43c2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc0cb100 con 0x7fe5cc159c30 2024-03-18T18:49:55.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.352+0000 7fe5d43c2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc0cab40 con 0x7fe5cc149fc0 2024-03-18T18:49:55.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.352+0000 7fe5d215e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b80043f0 0x7fe5cc0c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:55.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.352+0000 7fe5d215e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b80043f0 0x7fe5cc0c7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49336/0 (socket says 172.21.15.1:49336) 2024-03-18T18:49:55.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.352+0000 7fe5d215e700 1 -- 172.21.15.1:0/382411906 learned_addr learned my addr 172.21.15.1:0/382411906 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:55.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.353+0000 7fe5d195d700 1 --2- 172.21.15.1:0/382411906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5cc159c30 0x7fe5cc149a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:55.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.353+0000 7fe5d295f700 1 --2- 172.21.15.1:0/382411906 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5cc149fc0 0x7fe5cc0b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:55.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.353+0000 7fe5d215e700 1 -- 172.21.15.1:0/382411906 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5cc149fc0 msgr2=0x7fe5cc0b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.353+0000 7fe5d215e700 1 --2- 172.21.15.1:0/382411906 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5cc149fc0 0x7fe5cc0b4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.353+0000 7fe5d215e700 1 -- 172.21.15.1:0/382411906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5cc159c30 msgr2=0x7fe5cc149a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.353+0000 7fe5d215e700 1 --2- 172.21.15.1:0/382411906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5cc159c30 0x7fe5cc149a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.353+0000 7fe5d215e700 1 -- 172.21.15.1:0/382411906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5cc0cae20 con 0x7fe5b80043f0 2024-03-18T18:49:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.353+0000 7fe5d195d700 1 --2- 172.21.15.1:0/382411906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5cc159c30 0x7fe5cc149a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:49:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.353+0000 7fe5d295f700 1 --2- 172.21.15.1:0/382411906 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5cc149fc0 0x7fe5cc0b4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:49:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.353+0000 7fe5d215e700 1 --2- 172.21.15.1:0/382411906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b80043f0 0x7fe5cc0c7a10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc011900 tx=0x7fe5bc011cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.353+0000 7fe5c2ffd700 1 -- 172.21.15.1:0/382411906 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc00f8d0 con 0x7fe5b80043f0 2024-03-18T18:49:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.353+0000 7fe5c2ffd700 1 -- 172.21.15.1:0/382411906 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5bc00fa30 con 0x7fe5b80043f0 2024-03-18T18:49:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.353+0000 7fe5d43c2700 1 -- 172.21.15.1:0/382411906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5cc14a670 con 0x7fe5b80043f0 2024-03-18T18:49:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.353+0000 7fe5c2ffd700 1 -- 172.21.15.1:0/382411906 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc00fda0 con 0x7fe5b80043f0 2024-03-18T18:49:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.353+0000 7fe5d43c2700 1 -- 172.21.15.1:0/382411906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5cc0c7f50 con 0x7fe5b80043f0 2024-03-18T18:49:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.354+0000 7fe5d43c2700 1 -- 172.21.15.1:0/382411906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5cc14acc0 con 0x7fe5b80043f0 2024-03-18T18:49:55.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.355+0000 7fe5c2ffd700 1 -- 172.21.15.1:0/382411906 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe5bc015460 con 0x7fe5b80043f0 2024-03-18T18:49:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.355+0000 7fe5c2ffd700 1 --2- 172.21.15.1:0/382411906 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe5b006ac90 0x7fe5b006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.355+0000 7fe5c2ffd700 1 -- 172.21.15.1:0/382411906 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe5bc089e60 con 0x7fe5b80043f0 2024-03-18T18:49:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.355+0000 7fe5d195d700 1 --2- 172.21.15.1:0/382411906 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe5b006ac90 0x7fe5b006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.355+0000 7fe5d195d700 1 --2- 172.21.15.1:0/382411906 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe5b006ac90 0x7fe5b006d150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe5b80047d0 tx=0x7fe5c800a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:55.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.360+0000 7fe5c2ffd700 1 -- 172.21.15.1:0/382411906 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe5bc054de0 con 0x7fe5b80043f0 2024-03-18T18:49:55.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.562+0000 7fe5d43c2700 1 -- 172.21.15.1:0/382411906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe5cc0676a0 con 0x7fe5b80043f0 2024-03-18T18:49:55.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.563+0000 7fe5c2ffd700 1 -- 172.21.15.1:0/382411906 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe5bc058400 con 0x7fe5b80043f0 2024-03-18T18:49:55.559 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:49:55.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.565+0000 7fe5d43c2700 1 -- 172.21.15.1:0/382411906 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe5b006ac90 msgr2=0x7fe5b006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:55.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.565+0000 7fe5d43c2700 1 --2- 172.21.15.1:0/382411906 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe5b006ac90 0x7fe5b006d150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe5b80047d0 tx=0x7fe5c800a000 comp rx=0 tx=0).stop 2024-03-18T18:49:55.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.565+0000 7fe5d43c2700 1 -- 172.21.15.1:0/382411906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b80043f0 msgr2=0x7fe5cc0c7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:55.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.565+0000 7fe5d43c2700 1 --2- 172.21.15.1:0/382411906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b80043f0 0x7fe5cc0c7a10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc011900 tx=0x7fe5bc011cc0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.565+0000 7fe5d43c2700 1 -- 172.21.15.1:0/382411906 shutdown_connections 2024-03-18T18:49:55.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.565+0000 7fe5d43c2700 1 --2- 172.21.15.1:0/382411906 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe5b006ac90 0x7fe5b006d150 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.565+0000 7fe5d43c2700 1 --2- 172.21.15.1:0/382411906 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5cc149fc0 0x7fe5cc0b4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.565+0000 7fe5d43c2700 1 --2- 172.21.15.1:0/382411906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5b80043f0 0x7fe5cc0c7a10 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.565+0000 7fe5d43c2700 1 --2- 172.21.15.1:0/382411906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5cc159c30 0x7fe5cc149a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.565+0000 7fe5d43c2700 1 -- 172.21.15.1:0/382411906 >> 172.21.15.1:0/382411906 conn(0x7fe5cc0cc9b0 msgr2=0x7fe5cc071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:55.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.565+0000 7fe5d43c2700 1 -- 172.21.15.1:0/382411906 shutdown_connections 2024-03-18T18:49:55.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.565+0000 7fe5d43c2700 1 -- 172.21.15.1:0/382411906 wait complete. 2024-03-18T18:49:55.573 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T18:49:55.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.704+0000 7ffb88369700 1 Processor -- start 2024-03-18T18:49:55.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.704+0000 7ffb88369700 1 -- start start 2024-03-18T18:49:55.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.705+0000 7ffb88369700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ffb800aecb0 0x7ffb80070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:55.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.705+0000 7ffb88369700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb8015b750 con 0x7ffb80071380 2024-03-18T18:49:55.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.705+0000 7ffb88369700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb800ccc20 con 0x7ffb8015b370 2024-03-18T18:49:55.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.705+0000 7ffb88369700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb800cf510 con 0x7ffb800aecb0 2024-03-18T18:49:55.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.705+0000 7ffb86906700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ffb80071380 0x7ffb8006d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54422/0 (socket says 172.21.15.1:54422) 2024-03-18T18:49:55.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.705+0000 7ffb86906700 1 -- 172.21.15.1:0/4178378759 learned_addr learned my addr 172.21.15.1:0/4178378759 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:49:55.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.705+0000 7ffb85904700 1 --2- 172.21.15.1:0/4178378759 >> v2:172.21.15.165:3300/0 conn(0x7ffb800aecb0 0x7ffb80070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:55.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.705+0000 7ffb85103700 1 -- 172.21.15.1:0/4178378759 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981081519 0 0) 0x7ffb8015b750 con 0x7ffb80071380 2024-03-18T18:49:55.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.705+0000 7ffb85103700 1 -- 172.21.15.1:0/4178378759 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb6c0035b0 con 0x7ffb80071380 2024-03-18T18:49:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.705+0000 7ffb85103700 1 -- 172.21.15.1:0/4178378759 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 164670377 0 0) 0x7ffb6c0035b0 con 0x7ffb80071380 2024-03-18T18:49:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.705+0000 7ffb85103700 1 -- 172.21.15.1:0/4178378759 >> v2:172.21.15.165:3300/0 conn(0x7ffb800aecb0 msgr2=0x7ffb80070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.705+0000 7ffb85103700 1 --2- 172.21.15.1:0/4178378759 >> v2:172.21.15.165:3300/0 conn(0x7ffb800aecb0 0x7ffb80070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85103700 1 -- 172.21.15.1:0/4178378759 >> v1:172.21.15.165:6789/0 conn(0x7ffb8015b370 legacy=0x7ffb800cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85103700 1 -- 172.21.15.1:0/4178378759 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb800cc2f0 con 0x7ffb80071380 2024-03-18T18:49:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85103700 1 -- 172.21.15.1:0/4178378759 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7ffb780019f0 con 0x7ffb80071380 2024-03-18T18:49:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85103700 1 -- 172.21.15.1:0/4178378759 >> v1:172.21.15.1:6789/0 conn(0x7ffb80071380 legacy=0x7ffb8006d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85103700 1 --2- 172.21.15.1:0/4178378759 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb6c003a00 0x7ffb6c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85103700 1 --2- 172.21.15.1:0/4178378759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb6c004400 0x7ffb6c0048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85103700 1 --2- 172.21.15.1:0/4178378759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb6c005170 0x7ffb6c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85103700 1 -- 172.21.15.1:0/4178378759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb800cf510 con 0x7ffb6c004400 2024-03-18T18:49:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85103700 1 -- 172.21.15.1:0/4178378759 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c00c0b0 con 0x7ffb6c005170 2024-03-18T18:49:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85103700 1 -- 172.21.15.1:0/4178378759 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c00c270 con 0x7ffb6c003a00 2024-03-18T18:49:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85904700 1 --2- 172.21.15.1:0/4178378759 >> v2:172.21.15.165:3300/0 conn(0x7ffb800aecb0 0x7ffb80070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:49:55.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85904700 1 --2- 172.21.15.1:0/4178378759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb6c004400 0x7ffb6c0048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:55.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb86105700 1 --2- 172.21.15.1:0/4178378759 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb6c003a00 0x7ffb6c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:55.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb86906700 1 --2- 172.21.15.1:0/4178378759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb6c005170 0x7ffb6c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:55.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85904700 1 -- 172.21.15.1:0/4178378759 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb6c003a00 msgr2=0x7ffb6c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:55.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85904700 1 --2- 172.21.15.1:0/4178378759 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb6c003a00 0x7ffb6c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85904700 1 -- 172.21.15.1:0/4178378759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb6c005170 msgr2=0x7ffb6c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:55.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85904700 1 --2- 172.21.15.1:0/4178378759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb6c005170 0x7ffb6c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85904700 1 -- 172.21.15.1:0/4178378759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6c0035b0 con 0x7ffb6c004400 2024-03-18T18:49:55.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb86906700 1 -- 172.21.15.1:0/4178378759 reap_dead start 2024-03-18T18:49:55.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85904700 1 --2- 172.21.15.1:0/4178378759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb6c004400 0x7ffb6c0048d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c009d10 tx=0x7ffb6c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610bfae82edceada server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:55.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85103700 1 -- 172.21.15.1:0/4178378759 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c009e60 con 0x7ffb6c004400 2024-03-18T18:49:55.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85103700 1 -- 172.21.15.1:0/4178378759 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb7c00e9a0 con 0x7ffb6c004400 2024-03-18T18:49:55.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.706+0000 7ffb85103700 1 -- 172.21.15.1:0/4178378759 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c00ed30 con 0x7ffb6c004400 2024-03-18T18:49:55.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.707+0000 7ffb88369700 1 -- 172.21.15.1:0/4178378759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb6c004400 msgr2=0x7ffb6c0048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:55.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.707+0000 7ffb88369700 1 --2- 172.21.15.1:0/4178378759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb6c004400 0x7ffb6c0048d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c009d10 tx=0x7ffb6c005880 comp rx=0 tx=0).stop 2024-03-18T18:49:55.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.707+0000 7ffb88369700 1 -- 172.21.15.1:0/4178378759 shutdown_connections 2024-03-18T18:49:55.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.707+0000 7ffb88369700 1 --2- 172.21.15.1:0/4178378759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb6c004400 0x7ffb6c0048d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.707+0000 7ffb88369700 1 -- 172.21.15.1:0/4178378759 >> 172.21.15.1:0/4178378759 conn(0x7ffb800b1350 msgr2=0x7ffb800b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:55.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.707+0000 7ffb88369700 1 -- 172.21.15.1:0/4178378759 shutdown_connections 2024-03-18T18:49:55.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.707+0000 7ffb88369700 1 -- 172.21.15.1:0/4178378759 wait complete. 2024-03-18T18:49:55.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.708+0000 7ffb88369700 1 Processor -- start 2024-03-18T18:49:55.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.708+0000 7ffb88369700 1 -- start start 2024-03-18T18:49:55.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.709+0000 7ffb88369700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb6c004400 0x7ffb80153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:55.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.709+0000 7ffb88369700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb80148410 0x7ffb800ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:55.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.709+0000 7ffb88369700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb801515e0 0x7ffb80151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:55.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.709+0000 7ffb88369700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb80071830 con 0x7ffb6c004400 2024-03-18T18:49:55.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.709+0000 7ffb88369700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb800b3060 con 0x7ffb80148410 2024-03-18T18:49:55.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.709+0000 7ffb88369700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb80151fd0 con 0x7ffb801515e0 2024-03-18T18:49:55.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.709+0000 7ffb86105700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb6c004400 0x7ffb80153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:55.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.709+0000 7ffb86105700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb6c004400 0x7ffb80153a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49348/0 (socket says 172.21.15.1:49348) 2024-03-18T18:49:55.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.709+0000 7ffb86105700 1 -- 172.21.15.1:0/3059427195 learned_addr learned my addr 172.21.15.1:0/3059427195 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.709+0000 7ffb86906700 1 --2- 172.21.15.1:0/3059427195 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb801515e0 0x7ffb80151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.709+0000 7ffb85904700 1 --2- 172.21.15.1:0/3059427195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb80148410 0x7ffb800ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.709+0000 7ffb86105700 1 -- 172.21.15.1:0/3059427195 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb801515e0 msgr2=0x7ffb80151a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.709+0000 7ffb86105700 1 --2- 172.21.15.1:0/3059427195 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb801515e0 0x7ffb80151a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.709+0000 7ffb86105700 1 -- 172.21.15.1:0/3059427195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb80148410 msgr2=0x7ffb800ad2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.709+0000 7ffb86105700 1 --2- 172.21.15.1:0/3059427195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb80148410 0x7ffb800ad2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.709+0000 7ffb86105700 1 -- 172.21.15.1:0/3059427195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb800ad800 con 0x7ffb6c004400 2024-03-18T18:49:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.710+0000 7ffb86105700 1 --2- 172.21.15.1:0/3059427195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb6c004400 0x7ffb80153a00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ffb7000f7b0 tx=0x7ffb7000fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.710+0000 7ffb76ffd700 1 -- 172.21.15.1:0/3059427195 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb700052e0 con 0x7ffb6c004400 2024-03-18T18:49:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.710+0000 7ffb76ffd700 1 -- 172.21.15.1:0/3059427195 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb70005460 con 0x7ffb6c004400 2024-03-18T18:49:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.710+0000 7ffb88369700 1 -- 172.21.15.1:0/3059427195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb800adaa0 con 0x7ffb6c004400 2024-03-18T18:49:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.710+0000 7ffb76ffd700 1 -- 172.21.15.1:0/3059427195 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb700038c0 con 0x7ffb6c004400 2024-03-18T18:49:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.710+0000 7ffb88369700 1 -- 172.21.15.1:0/3059427195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb800ae060 con 0x7ffb6c004400 2024-03-18T18:49:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.711+0000 7ffb88369700 1 -- 172.21.15.1:0/3059427195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb800b31a0 con 0x7ffb6c004400 2024-03-18T18:49:55.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.711+0000 7ffb76ffd700 1 -- 172.21.15.1:0/3059427195 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ffb70003a20 con 0x7ffb6c004400 2024-03-18T18:49:55.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.712+0000 7ffb76ffd700 1 --2- 172.21.15.1:0/3059427195 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffb6406ac90 0x7ffb6406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:55.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.712+0000 7ffb76ffd700 1 -- 172.21.15.1:0/3059427195 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ffb700893c0 con 0x7ffb6c004400 2024-03-18T18:49:55.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.712+0000 7ffb85904700 1 --2- 172.21.15.1:0/3059427195 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffb6406ac90 0x7ffb6406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:55.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.712+0000 7ffb85904700 1 --2- 172.21.15.1:0/3059427195 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffb6406ac90 0x7ffb6406d150 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c0090b0 tx=0x7ffb7c01e040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:55.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.716+0000 7ffb76ffd700 1 -- 172.21.15.1:0/3059427195 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffb800b31a0 con 0x7ffb6c004400 2024-03-18T18:49:55.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.918+0000 7ffb88369700 1 -- 172.21.15.1:0/3059427195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffb800b31a0 con 0x7ffb6c004400 2024-03-18T18:49:55.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.918+0000 7ffb76ffd700 1 -- 172.21.15.1:0/3059427195 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffb800b31a0 con 0x7ffb6c004400 2024-03-18T18:49:55.915 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T18:49:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.920+0000 7ffb88369700 1 -- 172.21.15.1:0/3059427195 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffb6406ac90 msgr2=0x7ffb6406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.920+0000 7ffb88369700 1 --2- 172.21.15.1:0/3059427195 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffb6406ac90 0x7ffb6406d150 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c0090b0 tx=0x7ffb7c01e040 comp rx=0 tx=0).stop 2024-03-18T18:49:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.920+0000 7ffb88369700 1 -- 172.21.15.1:0/3059427195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb6c004400 msgr2=0x7ffb80153a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.920+0000 7ffb88369700 1 --2- 172.21.15.1:0/3059427195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb6c004400 0x7ffb80153a00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ffb7000f7b0 tx=0x7ffb7000fac0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.920+0000 7ffb88369700 1 -- 172.21.15.1:0/3059427195 shutdown_connections 2024-03-18T18:49:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.920+0000 7ffb88369700 1 --2- 172.21.15.1:0/3059427195 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffb6406ac90 0x7ffb6406d150 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.920+0000 7ffb88369700 1 --2- 172.21.15.1:0/3059427195 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb801515e0 0x7ffb80151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.920+0000 7ffb88369700 1 --2- 172.21.15.1:0/3059427195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb6c004400 0x7ffb80153a00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.920+0000 7ffb88369700 1 --2- 172.21.15.1:0/3059427195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb80148410 0x7ffb800ad2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.920+0000 7ffb88369700 1 -- 172.21.15.1:0/3059427195 >> 172.21.15.1:0/3059427195 conn(0x7ffb800b1350 msgr2=0x7ffb8006de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.920+0000 7ffb88369700 1 -- 172.21.15.1:0/3059427195 shutdown_connections 2024-03-18T18:49:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:55.920+0000 7ffb88369700 1 -- 172.21.15.1:0/3059427195 wait complete. 2024-03-18T18:49:55.929 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T18:49:56.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.060+0000 7ff4eba35700 1 Processor -- start 2024-03-18T18:49:56.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.060+0000 7ff4eba35700 1 -- start start 2024-03-18T18:49:56.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.061+0000 7ff4eba35700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4e415a1d0 0x7ff4e40add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:56.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.061+0000 7ff4eba35700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff4e4071620 0x7ff4e4071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:56.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.061+0000 7ff4eba35700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e40ce8f0 con 0x7ff4e40cb9e0 2024-03-18T18:49:56.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.061+0000 7ff4eba35700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4e40cdb00 con 0x7ff4e4071620 2024-03-18T18:49:56.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.061+0000 7ff4eba35700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e415a5b0 con 0x7ff4e415a1d0 2024-03-18T18:49:56.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.061+0000 7ff4e9fd2700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff4e40cb9e0 0x7ff4e40cbe70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54432/0 (socket says 172.21.15.1:54432) 2024-03-18T18:49:56.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.061+0000 7ff4e8fd0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff4e4071620 0x7ff4e4071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:56.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.061+0000 7ff4e9fd2700 1 -- 172.21.15.1:0/1147040111 learned_addr learned my addr 172.21.15.1:0/1147040111 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:49:56.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.062+0000 7ff4e97d1700 1 --2- 172.21.15.1:0/1147040111 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4e415a1d0 0x7ff4e40add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:56.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.062+0000 7ff4dbfff700 1 -- 172.21.15.1:0/1147040111 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992437173 0 0) 0x7ff4e40ce8f0 con 0x7ff4e40cb9e0 2024-03-18T18:49:56.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.062+0000 7ff4dbfff700 1 -- 172.21.15.1:0/1147040111 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d00035b0 con 0x7ff4e40cb9e0 2024-03-18T18:49:56.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.062+0000 7ff4e8fd0700 1 -- 172.21.15.1:0/1147040111 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4e415a1d0 msgr2=0x7ff4e40add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:56.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.062+0000 7ff4e8fd0700 1 --2- 172.21.15.1:0/1147040111 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4e415a1d0 0x7ff4e40add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.062+0000 7ff4e8fd0700 1 -- 172.21.15.1:0/1147040111 >> v1:172.21.15.1:6789/0 conn(0x7ff4e40cb9e0 legacy=0x7ff4e40cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:56.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.062+0000 7ff4e8fd0700 1 -- 172.21.15.1:0/1147040111 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e40ae260 con 0x7ff4e4071620 2024-03-18T18:49:56.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.062+0000 7ff4e8fd0700 1 --2- 172.21.15.1:0/1147040111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e4071620 0x7ff4e4071aa0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0009840 tx=0x7ff4e0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce8f50bf0be558d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:56.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.062+0000 7ff4dbfff700 1 -- 172.21.15.1:0/1147040111 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0011830 con 0x7ff4e4071620 2024-03-18T18:49:56.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.062+0000 7ff4dbfff700 1 -- 172.21.15.1:0/1147040111 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e0011990 con 0x7ff4e4071620 2024-03-18T18:49:56.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.062+0000 7ff4dbfff700 1 -- 172.21.15.1:0/1147040111 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0011d00 con 0x7ff4e4071620 2024-03-18T18:49:56.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.062+0000 7ff4eba35700 1 -- 172.21.15.1:0/1147040111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e4071620 msgr2=0x7ff4e4071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:56.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.062+0000 7ff4eba35700 1 --2- 172.21.15.1:0/1147040111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e4071620 0x7ff4e4071aa0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0009840 tx=0x7ff4e0009b50 comp rx=0 tx=0).stop 2024-03-18T18:49:56.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.063+0000 7ff4eba35700 1 -- 172.21.15.1:0/1147040111 shutdown_connections 2024-03-18T18:49:56.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.063+0000 7ff4eba35700 1 --2- 172.21.15.1:0/1147040111 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4e415a1d0 0x7ff4e40add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.063+0000 7ff4eba35700 1 --2- 172.21.15.1:0/1147040111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e4071620 0x7ff4e4071aa0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.063+0000 7ff4eba35700 1 -- 172.21.15.1:0/1147040111 >> 172.21.15.1:0/1147040111 conn(0x7ff4e40cc9b0 msgr2=0x7ff4e40ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:56.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.063+0000 7ff4eba35700 1 -- 172.21.15.1:0/1147040111 shutdown_connections 2024-03-18T18:49:56.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.063+0000 7ff4eba35700 1 -- 172.21.15.1:0/1147040111 wait complete. 2024-03-18T18:49:56.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.064+0000 7ff4eba35700 1 Processor -- start 2024-03-18T18:49:56.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.064+0000 7ff4eba35700 1 -- start start 2024-03-18T18:49:56.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.065+0000 7ff4eba35700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4e4071620 0x7ff4e415cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:56.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.065+0000 7ff4eba35700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4e40cb9e0 0x7ff4e40b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:56.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.065+0000 7ff4eba35700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e415a1d0 0x7ff4e40b10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:56.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.065+0000 7ff4eba35700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e40c3b00 con 0x7ff4e415a1d0 2024-03-18T18:49:56.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.065+0000 7ff4eba35700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e40c3da0 con 0x7ff4e4071620 2024-03-18T18:49:56.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.065+0000 7ff4eba35700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e40c4080 con 0x7ff4e40cb9e0 2024-03-18T18:49:56.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.065+0000 7ff4e9fd2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e415a1d0 0x7ff4e40b10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:56.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.065+0000 7ff4e9fd2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e415a1d0 0x7ff4e40b10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49364/0 (socket says 172.21.15.1:49364) 2024-03-18T18:49:56.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.065+0000 7ff4e9fd2700 1 -- 172.21.15.1:0/2954863868 learned_addr learned my addr 172.21.15.1:0/2954863868 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:56.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.065+0000 7ff4e8fd0700 1 --2- 172.21.15.1:0/2954863868 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4e40cb9e0 0x7ff4e40b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:56.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.065+0000 7ff4e97d1700 1 --2- 172.21.15.1:0/2954863868 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4e4071620 0x7ff4e415cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:56.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.066+0000 7ff4e9fd2700 1 -- 172.21.15.1:0/2954863868 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4e40cb9e0 msgr2=0x7ff4e40b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:56.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.066+0000 7ff4e9fd2700 1 --2- 172.21.15.1:0/2954863868 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4e40cb9e0 0x7ff4e40b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.066+0000 7ff4e9fd2700 1 -- 172.21.15.1:0/2954863868 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4e4071620 msgr2=0x7ff4e415cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:56.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.066+0000 7ff4e9fd2700 1 --2- 172.21.15.1:0/2954863868 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4e4071620 0x7ff4e415cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.066+0000 7ff4e9fd2700 1 -- 172.21.15.1:0/2954863868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e00094a0 con 0x7ff4e415a1d0 2024-03-18T18:49:56.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.066+0000 7ff4e97d1700 1 --2- 172.21.15.1:0/2954863868 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4e4071620 0x7ff4e415cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:56.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.066+0000 7ff4e9fd2700 1 --2- 172.21.15.1:0/2954863868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e415a1d0 0x7ff4e40b10d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc00cb40 tx=0x7ff4dc00ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:56.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.066+0000 7ff4d9ffb700 1 -- 172.21.15.1:0/2954863868 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc00aaa0 con 0x7ff4e415a1d0 2024-03-18T18:49:56.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.066+0000 7ff4d9ffb700 1 -- 172.21.15.1:0/2954863868 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4dc00ac00 con 0x7ff4e415a1d0 2024-03-18T18:49:56.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.066+0000 7ff4eba35700 1 -- 172.21.15.1:0/2954863868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e40c4360 con 0x7ff4e415a1d0 2024-03-18T18:49:56.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.066+0000 7ff4d9ffb700 1 -- 172.21.15.1:0/2954863868 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc012400 con 0x7ff4e415a1d0 2024-03-18T18:49:56.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.066+0000 7ff4eba35700 1 -- 172.21.15.1:0/2954863868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e415d2a0 con 0x7ff4e415a1d0 2024-03-18T18:49:56.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.067+0000 7ff4eba35700 1 -- 172.21.15.1:0/2954863868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e415d550 con 0x7ff4e415a1d0 2024-03-18T18:49:56.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.067+0000 7ff4d9ffb700 1 -- 172.21.15.1:0/2954863868 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff4dc01b3b0 con 0x7ff4e415a1d0 2024-03-18T18:49:56.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.068+0000 7ff4d9ffb700 1 --2- 172.21.15.1:0/2954863868 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff4c806ac90 0x7ff4c806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:56.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.068+0000 7ff4d9ffb700 1 -- 172.21.15.1:0/2954863868 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff4dc089370 con 0x7ff4e415a1d0 2024-03-18T18:49:56.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.068+0000 7ff4e97d1700 1 --2- 172.21.15.1:0/2954863868 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff4c806ac90 0x7ff4c806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:56.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.068+0000 7ff4e97d1700 1 --2- 172.21.15.1:0/2954863868 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff4c806ac90 0x7ff4c806d150 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4006fd0 tx=0x7ff4d4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:56.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.072+0000 7ff4d9ffb700 1 -- 172.21.15.1:0/2954863868 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff4dc054260 con 0x7ff4e415a1d0 2024-03-18T18:49:56.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.272+0000 7ff4eba35700 1 -- 172.21.15.1:0/2954863868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff4e40676a0 con 0x7ff4e415a1d0 2024-03-18T18:49:56.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.273+0000 7ff4d9ffb700 1 -- 172.21.15.1:0/2954863868 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff4dc057880 con 0x7ff4e415a1d0 2024-03-18T18:49:56.270 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:49:56.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.275+0000 7ff4eba35700 1 -- 172.21.15.1:0/2954863868 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff4c806ac90 msgr2=0x7ff4c806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:56.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.275+0000 7ff4eba35700 1 --2- 172.21.15.1:0/2954863868 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff4c806ac90 0x7ff4c806d150 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4006fd0 tx=0x7ff4d4008040 comp rx=0 tx=0).stop 2024-03-18T18:49:56.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.275+0000 7ff4eba35700 1 -- 172.21.15.1:0/2954863868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e415a1d0 msgr2=0x7ff4e40b10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:56.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.275+0000 7ff4eba35700 1 --2- 172.21.15.1:0/2954863868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e415a1d0 0x7ff4e40b10d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc00cb40 tx=0x7ff4dc00ce50 comp rx=0 tx=0).stop 2024-03-18T18:49:56.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.275+0000 7ff4eba35700 1 -- 172.21.15.1:0/2954863868 shutdown_connections 2024-03-18T18:49:56.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.275+0000 7ff4eba35700 1 --2- 172.21.15.1:0/2954863868 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff4c806ac90 0x7ff4c806d150 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.275+0000 7ff4eba35700 1 --2- 172.21.15.1:0/2954863868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4e415a1d0 0x7ff4e40b10d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.275+0000 7ff4eba35700 1 --2- 172.21.15.1:0/2954863868 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4e4071620 0x7ff4e415cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.275+0000 7ff4eba35700 1 --2- 172.21.15.1:0/2954863868 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4e40cb9e0 0x7ff4e40b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.275+0000 7ff4eba35700 1 -- 172.21.15.1:0/2954863868 >> 172.21.15.1:0/2954863868 conn(0x7ff4e40cc9b0 msgr2=0x7ff4e406ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:56.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.275+0000 7ff4eba35700 1 -- 172.21.15.1:0/2954863868 shutdown_connections 2024-03-18T18:49:56.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.275+0000 7ff4eba35700 1 -- 172.21.15.1:0/2954863868 wait complete. 2024-03-18T18:49:56.284 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T18:49:56.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.413+0000 7f8bb3534700 1 Processor -- start 2024-03-18T18:49:56.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.413+0000 7f8bb3534700 1 -- start start 2024-03-18T18:49:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.414+0000 7f8bb3534700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bac071d30 0x7f8bac0add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.414+0000 7f8bb3534700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8bac0cb9e0 0x7f8bac0cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.414+0000 7f8bb3534700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bac0ce8f0 con 0x7f8bac15a1d0 2024-03-18T18:49:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.414+0000 7f8bb3534700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bac0cdb00 con 0x7f8bac0cb9e0 2024-03-18T18:49:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.414+0000 7f8bb3534700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac15a5b0 con 0x7f8bac071d30 2024-03-18T18:49:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.414+0000 7f8bb1ad1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8bac0cb9e0 0x7f8bac0cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.414+0000 7f8bb1ad1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8bac0cb9e0 0x7f8bac0cbe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49380/0 (socket says 172.21.15.1:49380) 2024-03-18T18:49:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.414+0000 7f8bb1ad1700 1 -- 172.21.15.1:0/393157680 learned_addr learned my addr 172.21.15.1:0/393157680 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.414+0000 7f8bb0acf700 1 --2- 172.21.15.1:0/393157680 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bac071d30 0x7f8bac0add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.414+0000 7f8bb1ad1700 1 -- 172.21.15.1:0/393157680 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bac071d30 msgr2=0x7f8bac0add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.414+0000 7f8bb1ad1700 1 --2- 172.21.15.1:0/393157680 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bac071d30 0x7f8bac0add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.414+0000 7f8bb1ad1700 1 -- 172.21.15.1:0/393157680 >> v1:172.21.15.165:6789/0 conn(0x7f8bac15a1d0 legacy=0x7f8bac071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.414+0000 7f8bb1ad1700 1 -- 172.21.15.1:0/393157680 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bac0ae260 con 0x7f8bac0cb9e0 2024-03-18T18:49:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.415+0000 7f8bb1ad1700 1 --2- 172.21.15.1:0/393157680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bac0cb9e0 0x7f8bac0cbe70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8ba400d150 tx=0x7f8ba400d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e718e1ac88d4276 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.415+0000 7f8ba3fff700 1 -- 172.21.15.1:0/393157680 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4010070 con 0x7f8bac0cb9e0 2024-03-18T18:49:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.415+0000 7f8ba3fff700 1 -- 172.21.15.1:0/393157680 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba400ddc0 con 0x7f8bac0cb9e0 2024-03-18T18:49:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.415+0000 7f8ba3fff700 1 -- 172.21.15.1:0/393157680 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba400b310 con 0x7f8bac0cb9e0 2024-03-18T18:49:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.415+0000 7f8bb3534700 1 -- 172.21.15.1:0/393157680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bac0cb9e0 msgr2=0x7f8bac0cbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.415+0000 7f8bb3534700 1 --2- 172.21.15.1:0/393157680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bac0cb9e0 0x7f8bac0cbe70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8ba400d150 tx=0x7f8ba400d460 comp rx=0 tx=0).stop 2024-03-18T18:49:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.415+0000 7f8bb3534700 1 -- 172.21.15.1:0/393157680 shutdown_connections 2024-03-18T18:49:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.415+0000 7f8bb3534700 1 --2- 172.21.15.1:0/393157680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bac0cb9e0 0x7f8bac0cbe70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.415+0000 7f8bb3534700 1 --2- 172.21.15.1:0/393157680 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bac071d30 0x7f8bac0add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.415+0000 7f8bb3534700 1 -- 172.21.15.1:0/393157680 >> 172.21.15.1:0/393157680 conn(0x7f8bac0cc9b0 msgr2=0x7f8bac0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.415+0000 7f8bb3534700 1 -- 172.21.15.1:0/393157680 shutdown_connections 2024-03-18T18:49:56.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.415+0000 7f8bb3534700 1 -- 172.21.15.1:0/393157680 wait complete. 2024-03-18T18:49:56.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.417+0000 7f8bb3534700 1 Processor -- start 2024-03-18T18:49:56.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.417+0000 7f8bb3534700 1 -- start start 2024-03-18T18:49:56.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.417+0000 7f8bb3534700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bac071d30 0x7f8bac0c57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:56.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.417+0000 7f8bb3534700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8bac0cb9e0 0x7f8bac0c6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:56.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.417+0000 7f8bb3534700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bac15a1d0 0x7f8bac09cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:56.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.417+0000 7f8bb3534700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac0c4720 con 0x7f8bac15a1d0 2024-03-18T18:49:56.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.417+0000 7f8bb3534700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac0c4a00 con 0x7f8bac0cb9e0 2024-03-18T18:49:56.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.417+0000 7f8bb3534700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac0c4ce0 con 0x7f8bac071d30 2024-03-18T18:49:56.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.417+0000 7f8bb1ad1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bac15a1d0 0x7f8bac09cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:56.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.417+0000 7f8bb1ad1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bac15a1d0 0x7f8bac09cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49384/0 (socket says 172.21.15.1:49384) 2024-03-18T18:49:56.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.417+0000 7f8bb1ad1700 1 -- 172.21.15.1:0/1389384527 learned_addr learned my addr 172.21.15.1:0/1389384527 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:56.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.418+0000 7f8bb0acf700 1 --2- 172.21.15.1:0/1389384527 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8bac0cb9e0 0x7f8bac0c6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.418+0000 7f8bb12d0700 1 --2- 172.21.15.1:0/1389384527 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bac071d30 0x7f8bac0c57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.418+0000 7f8bb1ad1700 1 -- 172.21.15.1:0/1389384527 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bac071d30 msgr2=0x7f8bac0c57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.418+0000 7f8bb1ad1700 1 --2- 172.21.15.1:0/1389384527 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bac071d30 0x7f8bac0c57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.418+0000 7f8bb1ad1700 1 -- 172.21.15.1:0/1389384527 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8bac0cb9e0 msgr2=0x7f8bac0c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.418+0000 7f8bb1ad1700 1 --2- 172.21.15.1:0/1389384527 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8bac0cb9e0 0x7f8bac0c6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.418+0000 7f8bb1ad1700 1 -- 172.21.15.1:0/1389384527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba400bd40 con 0x7f8bac15a1d0 2024-03-18T18:49:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.418+0000 7f8bb0acf700 1 --2- 172.21.15.1:0/1389384527 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8bac0cb9e0 0x7f8bac0c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:49:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.418+0000 7f8bb12d0700 1 --2- 172.21.15.1:0/1389384527 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bac071d30 0x7f8bac0c57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:49:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.418+0000 7f8bb1ad1700 1 --2- 172.21.15.1:0/1389384527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bac15a1d0 0x7f8bac09cb80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8bac06cfd0 tx=0x7f8ba4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.418+0000 7f8ba1ffb700 1 -- 172.21.15.1:0/1389384527 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4009a10 con 0x7f8bac15a1d0 2024-03-18T18:49:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.418+0000 7f8ba1ffb700 1 -- 172.21.15.1:0/1389384527 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba4009b70 con 0x7f8bac15a1d0 2024-03-18T18:49:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.418+0000 7f8bb3534700 1 -- 172.21.15.1:0/1389384527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bac0c4fc0 con 0x7f8bac15a1d0 2024-03-18T18:49:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.418+0000 7f8ba1ffb700 1 -- 172.21.15.1:0/1389384527 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba400b4c0 con 0x7f8bac15a1d0 2024-03-18T18:49:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.418+0000 7f8bb3534700 1 -- 172.21.15.1:0/1389384527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bac0c5d00 con 0x7f8bac15a1d0 2024-03-18T18:49:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.419+0000 7f8bb3534700 1 -- 172.21.15.1:0/1389384527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bac0acd80 con 0x7f8bac15a1d0 2024-03-18T18:49:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.420+0000 7f8ba1ffb700 1 -- 172.21.15.1:0/1389384527 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8ba4004040 con 0x7f8bac15a1d0 2024-03-18T18:49:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.420+0000 7f8ba1ffb700 1 --2- 172.21.15.1:0/1389384527 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8b9006ac90 0x7f8b9006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.420+0000 7f8ba1ffb700 1 -- 172.21.15.1:0/1389384527 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f8ba408aca0 con 0x7f8bac15a1d0 2024-03-18T18:49:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.420+0000 7f8bb12d0700 1 --2- 172.21.15.1:0/1389384527 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8b9006ac90 0x7f8b9006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.421+0000 7f8bb12d0700 1 --2- 172.21.15.1:0/1389384527 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8b9006ac90 0x7f8b9006d150 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c000fb0 tx=0x7f8b9c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:56.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.425+0000 7f8ba1ffb700 1 -- 172.21.15.1:0/1389384527 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8ba4055ba0 con 0x7f8bac15a1d0 2024-03-18T18:49:56.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.641+0000 7f8bb3534700 1 -- 172.21.15.1:0/1389384527 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8bac0676a0 con 0x7f8bac15a1d0 2024-03-18T18:49:56.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.641+0000 7f8ba1ffb700 1 -- 172.21.15.1:0/1389384527 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8ba40591c0 con 0x7f8bac15a1d0 2024-03-18T18:49:56.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.643+0000 7f8bb3534700 1 -- 172.21.15.1:0/1389384527 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8b9006ac90 msgr2=0x7f8b9006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:56.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.643+0000 7f8bb3534700 1 --2- 172.21.15.1:0/1389384527 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8b9006ac90 0x7f8b9006d150 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c000fb0 tx=0x7f8b9c009040 comp rx=0 tx=0).stop 2024-03-18T18:49:56.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.643+0000 7f8bb3534700 1 -- 172.21.15.1:0/1389384527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bac15a1d0 msgr2=0x7f8bac09cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:56.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.643+0000 7f8bb3534700 1 --2- 172.21.15.1:0/1389384527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bac15a1d0 0x7f8bac09cb80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8bac06cfd0 tx=0x7f8ba4004ad0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.643+0000 7f8bb3534700 1 -- 172.21.15.1:0/1389384527 shutdown_connections 2024-03-18T18:49:56.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.643+0000 7f8bb3534700 1 --2- 172.21.15.1:0/1389384527 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8b9006ac90 0x7f8b9006d150 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.643+0000 7f8bb3534700 1 --2- 172.21.15.1:0/1389384527 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8bac071d30 0x7f8bac0c57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.643+0000 7f8bb3534700 1 --2- 172.21.15.1:0/1389384527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8bac15a1d0 0x7f8bac09cb80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.643+0000 7f8bb3534700 1 --2- 172.21.15.1:0/1389384527 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8bac0cb9e0 0x7f8bac0c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.643+0000 7f8bb3534700 1 -- 172.21.15.1:0/1389384527 >> 172.21.15.1:0/1389384527 conn(0x7f8bac0cc9b0 msgr2=0x7f8bac06f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:56.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.643+0000 7f8bb3534700 1 -- 172.21.15.1:0/1389384527 shutdown_connections 2024-03-18T18:49:56.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.643+0000 7f8bb3534700 1 -- 172.21.15.1:0/1389384527 wait complete. 2024-03-18T18:49:56.641 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-18T18:49:56.651 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:49:56.652 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T18:49:56.671 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:49:56.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.841+0000 7f72760e3700 1 Processor -- start 2024-03-18T18:49:56.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.841+0000 7f72760e3700 1 -- start start 2024-03-18T18:49:56.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.842+0000 7f72760e3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72700b6b60 0x7f72700cc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:56.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.842+0000 7f72760e3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f72700cee90 0x7f72700cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:56.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.842+0000 7f72760e3700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f72700cfbd0 0x7f7270099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:56.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.842+0000 7f72760e3700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72700d0060 con 0x7f72700cee90 2024-03-18T18:49:56.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.842+0000 7f72760e3700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72700cd0d0 con 0x7f72700cfbd0 2024-03-18T18:49:56.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.842+0000 7f72760e3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72700b3f00 con 0x7f72700b6b60 2024-03-18T18:49:56.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.842+0000 7f726effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f72700cee90 0x7f72700cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:56.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.842+0000 7f726effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f72700cee90 0x7f72700cf690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49388/0 (socket says 172.21.15.1:49388) 2024-03-18T18:49:56.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.842+0000 7f726effd700 1 -- 172.21.15.1:0/454217869 learned_addr learned my addr 172.21.15.1:0/454217869 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:56.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.842+0000 7f726f7fe700 1 --2- 172.21.15.1:0/454217869 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72700b6b60 0x7f72700cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:56.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.842+0000 7f726ffff700 1 --2- 172.21.15.1:0/454217869 >> v2:172.21.15.165:3300/0 conn(0x7f72700cfbd0 0x7f7270099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:56.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.842+0000 7f726effd700 1 -- 172.21.15.1:0/454217869 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72700b6b60 msgr2=0x7f72700cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:56.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.842+0000 7f726effd700 1 --2- 172.21.15.1:0/454217869 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72700b6b60 0x7f72700cc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.842+0000 7f726effd700 1 -- 172.21.15.1:0/454217869 >> v2:172.21.15.165:3300/0 conn(0x7f72700cfbd0 msgr2=0x7f7270099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:56.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.842+0000 7f726effd700 1 --2- 172.21.15.1:0/454217869 >> v2:172.21.15.165:3300/0 conn(0x7f72700cfbd0 0x7f7270099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.842+0000 7f726effd700 1 -- 172.21.15.1:0/454217869 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72700b41e0 con 0x7f72700cee90 2024-03-18T18:49:56.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.843+0000 7f726effd700 1 --2- 172.21.15.1:0/454217869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72700cee90 0x7f72700cf690 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f72700c8130 tx=0x7f7258009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f96e190570ea1c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:56.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.843+0000 7f726e7fc700 1 -- 172.21.15.1:0/454217869 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725800b870 con 0x7f72700cee90 2024-03-18T18:49:56.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.843+0000 7f726e7fc700 1 -- 172.21.15.1:0/454217869 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725800b9d0 con 0x7f72700cee90 2024-03-18T18:49:56.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.843+0000 7f726e7fc700 1 -- 172.21.15.1:0/454217869 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725800bd60 con 0x7f72700cee90 2024-03-18T18:49:56.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.843+0000 7f72760e3700 1 -- 172.21.15.1:0/454217869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72700cee90 msgr2=0x7f72700cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:56.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.843+0000 7f72760e3700 1 --2- 172.21.15.1:0/454217869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72700cee90 0x7f72700cf690 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f72700c8130 tx=0x7f7258009bd0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.843+0000 7f72760e3700 1 -- 172.21.15.1:0/454217869 shutdown_connections 2024-03-18T18:49:56.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.843+0000 7f72760e3700 1 --2- 172.21.15.1:0/454217869 >> v2:172.21.15.165:3300/0 conn(0x7f72700cfbd0 0x7f7270099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.843+0000 7f72760e3700 1 --2- 172.21.15.1:0/454217869 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72700b6b60 0x7f72700cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.843+0000 7f72760e3700 1 --2- 172.21.15.1:0/454217869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72700cee90 0x7f72700cf690 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.843+0000 7f72760e3700 1 -- 172.21.15.1:0/454217869 >> 172.21.15.1:0/454217869 conn(0x7f72700b2ba0 msgr2=0x7f72700b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:56.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.843+0000 7f72760e3700 1 -- 172.21.15.1:0/454217869 shutdown_connections 2024-03-18T18:49:56.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.843+0000 7f72760e3700 1 -- 172.21.15.1:0/454217869 wait complete. 2024-03-18T18:49:56.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.844+0000 7f72760e3700 1 Processor -- start 2024-03-18T18:49:56.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.845+0000 7f72760e3700 1 -- start start 2024-03-18T18:49:56.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.845+0000 7f72760e3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72700b6b60 0x7f7270152560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:56.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f72760e3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72700cee90 0x7f727009b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:56.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f72760e3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72700cfbd0 0x7f727009bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:56.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f72760e3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72700c6390 con 0x7f72700cee90 2024-03-18T18:49:56.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f72760e3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727006bbd0 con 0x7f72700cfbd0 2024-03-18T18:49:56.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f72760e3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7270152aa0 con 0x7f72700b6b60 2024-03-18T18:49:56.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f726effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72700cee90 0x7f727009b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:56.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f726effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72700cee90 0x7f727009b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49398/0 (socket says 172.21.15.1:49398) 2024-03-18T18:49:56.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f726effd700 1 -- 172.21.15.1:0/3691211856 learned_addr learned my addr 172.21.15.1:0/3691211856 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:56.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f726ffff700 1 --2- 172.21.15.1:0/3691211856 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72700cfbd0 0x7f727009bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:56.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f726f7fe700 1 --2- 172.21.15.1:0/3691211856 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72700b6b60 0x7f7270152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:56.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f726effd700 1 -- 172.21.15.1:0/3691211856 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72700b6b60 msgr2=0x7f7270152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:56.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f726effd700 1 --2- 172.21.15.1:0/3691211856 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72700b6b60 0x7f7270152560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f726effd700 1 -- 172.21.15.1:0/3691211856 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72700cfbd0 msgr2=0x7f727009bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:56.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f726effd700 1 --2- 172.21.15.1:0/3691211856 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72700cfbd0 0x7f727009bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:56.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f726effd700 1 -- 172.21.15.1:0/3691211856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7258009550 con 0x7f72700cee90 2024-03-18T18:49:56.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f726f7fe700 1 --2- 172.21.15.1:0/3691211856 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72700b6b60 0x7f7270152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:49:56.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f726effd700 1 --2- 172.21.15.1:0/3691211856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72700cee90 0x7f727009b790 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7258000c00 tx=0x7f7258004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:56.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f7267fff700 1 -- 172.21.15.1:0/3691211856 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7258004540 con 0x7f72700cee90 2024-03-18T18:49:56.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f7267fff700 1 -- 172.21.15.1:0/3691211856 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72580036a0 con 0x7f72700cee90 2024-03-18T18:49:56.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f72760e3700 1 -- 172.21.15.1:0/3691211856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f727009e740 con 0x7f72700cee90 2024-03-18T18:49:56.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f7267fff700 1 -- 172.21.15.1:0/3691211856 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7258003a50 con 0x7f72700cee90 2024-03-18T18:49:56.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.846+0000 7f72760e3700 1 -- 172.21.15.1:0/3691211856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f727009ed00 con 0x7f72700cee90 2024-03-18T18:49:56.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.847+0000 7f72760e3700 1 -- 172.21.15.1:0/3691211856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72700676a0 con 0x7f72700cee90 2024-03-18T18:49:56.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.848+0000 7f7267fff700 1 -- 172.21.15.1:0/3691211856 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7258021070 con 0x7f72700cee90 2024-03-18T18:49:56.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.848+0000 7f7267fff700 1 --2- 172.21.15.1:0/3691211856 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f725006ac90 0x7f725006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:56.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.848+0000 7f7267fff700 1 -- 172.21.15.1:0/3691211856 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f725808ac60 con 0x7f72700cee90 2024-03-18T18:49:56.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.848+0000 7f726f7fe700 1 --2- 172.21.15.1:0/3691211856 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f725006ac90 0x7f725006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:56.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.849+0000 7f726f7fe700 1 --2- 172.21.15.1:0/3691211856 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f725006ac90 0x7f725006d150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f72600099e0 tx=0x7f7260008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:56.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:56.853+0000 7f7267fff700 1 -- 172.21.15.1:0/3691211856 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7258055be0 con 0x7f72700cee90 2024-03-18T18:49:57.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.058+0000 7f72760e3700 1 -- 172.21.15.1:0/3691211856 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72700b58b0 con 0x7f72700cee90 2024-03-18T18:49:57.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.059+0000 7f7267fff700 1 -- 172.21.15.1:0/3691211856 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17830 (secure 0 0 0) 0x7f7258029090 con 0x7f72700cee90 2024-03-18T18:49:57.056 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:49:57.056 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:49.276961+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:49:57.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.061+0000 7f72760e3700 1 -- 172.21.15.1:0/3691211856 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f725006ac90 msgr2=0x7f725006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:57.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.061+0000 7f72760e3700 1 --2- 172.21.15.1:0/3691211856 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f725006ac90 0x7f725006d150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f72600099e0 tx=0x7f7260008040 comp rx=0 tx=0).stop 2024-03-18T18:49:57.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.061+0000 7f72760e3700 1 -- 172.21.15.1:0/3691211856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72700cee90 msgr2=0x7f727009b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:57.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.061+0000 7f72760e3700 1 --2- 172.21.15.1:0/3691211856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72700cee90 0x7f727009b790 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7258000c00 tx=0x7f7258004750 comp rx=0 tx=0).stop 2024-03-18T18:49:57.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.061+0000 7f72760e3700 1 -- 172.21.15.1:0/3691211856 shutdown_connections 2024-03-18T18:49:57.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.061+0000 7f72760e3700 1 --2- 172.21.15.1:0/3691211856 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f725006ac90 0x7f725006d150 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.061+0000 7f72760e3700 1 --2- 172.21.15.1:0/3691211856 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72700b6b60 0x7f7270152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.061+0000 7f72760e3700 1 --2- 172.21.15.1:0/3691211856 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72700cfbd0 0x7f727009bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.061+0000 7f72760e3700 1 --2- 172.21.15.1:0/3691211856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72700cee90 0x7f727009b790 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.061+0000 7f72760e3700 1 -- 172.21.15.1:0/3691211856 >> 172.21.15.1:0/3691211856 conn(0x7f72700b2ba0 msgr2=0x7f72700cd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:57.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.062+0000 7f72760e3700 1 -- 172.21.15.1:0/3691211856 shutdown_connections 2024-03-18T18:49:57.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.062+0000 7f72760e3700 1 -- 172.21.15.1:0/3691211856 wait complete. 2024-03-18T18:49:57.070 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T18:49:42.800801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T18:49:43.776684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T18:49:46.854608+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T18:49:57.071 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T18:49:57.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.202+0000 7ff28bd6e700 1 Processor -- start 2024-03-18T18:49:57.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.203+0000 7ff28bd6e700 1 -- start start 2024-03-18T18:49:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.203+0000 7ff28bd6e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff28409dd40 0x7ff2840cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.203+0000 7ff28bd6e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff2840ce5a0 0x7ff2840cea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.203+0000 7ff28bd6e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2840ce090 con 0x7ff28415d780 2024-03-18T18:49:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.203+0000 7ff28bd6e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2840cb480 con 0x7ff2840ce5a0 2024-03-18T18:49:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.203+0000 7ff28bd6e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff28406db30 con 0x7ff28409dd40 2024-03-18T18:49:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.203+0000 7ff289b0a700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff28415d780 0x7ff2840cad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54434/0 (socket says 172.21.15.1:54434) 2024-03-18T18:49:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff289b0a700 1 -- 172.21.15.1:0/2350459517 learned_addr learned my addr 172.21.15.1:0/2350459517 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:49:57.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff28a30b700 1 --2- 172.21.15.1:0/2350459517 >> v2:172.21.15.165:3300/0 conn(0x7ff2840ce5a0 0x7ff2840cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff289309700 1 --2- 172.21.15.1:0/2350459517 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff28409dd40 0x7ff2840cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff288b08700 1 -- 172.21.15.1:0/2350459517 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771740243 0 0) 0x7ff2840ce090 con 0x7ff28415d780 2024-03-18T18:49:57.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff288b08700 1 -- 172.21.15.1:0/2350459517 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2700035b0 con 0x7ff28415d780 2024-03-18T18:49:57.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff288b08700 1 -- 172.21.15.1:0/2350459517 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2352755382 0 0) 0x7ff2700035b0 con 0x7ff28415d780 2024-03-18T18:49:57.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff288b08700 1 -- 172.21.15.1:0/2350459517 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff28409dd40 msgr2=0x7ff2840cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:57.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff288b08700 1 --2- 172.21.15.1:0/2350459517 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff28409dd40 0x7ff2840cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff288b08700 1 -- 172.21.15.1:0/2350459517 >> v2:172.21.15.165:3300/0 conn(0x7ff2840ce5a0 msgr2=0x7ff2840cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:57.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff288b08700 1 --2- 172.21.15.1:0/2350459517 >> v2:172.21.15.165:3300/0 conn(0x7ff2840ce5a0 0x7ff2840cea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff288b08700 1 -- 172.21.15.1:0/2350459517 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28406d520 con 0x7ff28415d780 2024-03-18T18:49:57.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff288b08700 1 -- 172.21.15.1:0/2350459517 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7ff280001670 con 0x7ff28415d780 2024-03-18T18:49:57.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff288b08700 1 -- 172.21.15.1:0/2350459517 >> v1:172.21.15.1:6789/0 conn(0x7ff28415d780 legacy=0x7ff2840cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:57.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff288b08700 1 --2- 172.21.15.1:0/2350459517 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff270003a00 0x7ff270003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff288b08700 1 --2- 172.21.15.1:0/2350459517 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff28409dd40 0x7ff270004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff288b08700 1 --2- 172.21.15.1:0/2350459517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff270004d90 0x7ff270005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff288b08700 1 -- 172.21.15.1:0/2350459517 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2840cb480 con 0x7ff270004d90 2024-03-18T18:49:57.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff288b08700 1 -- 172.21.15.1:0/2350459517 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28406db30 con 0x7ff270003a00 2024-03-18T18:49:57.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff288b08700 1 -- 172.21.15.1:0/2350459517 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff27000ba10 con 0x7ff28409dd40 2024-03-18T18:49:57.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff28a30b700 1 --2- 172.21.15.1:0/2350459517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff270004d90 0x7ff270005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff289309700 1 --2- 172.21.15.1:0/2350459517 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff28409dd40 0x7ff270004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.204+0000 7ff289b0a700 1 --2- 172.21.15.1:0/2350459517 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff270003a00 0x7ff270003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.205+0000 7ff28a30b700 1 -- 172.21.15.1:0/2350459517 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff28409dd40 msgr2=0x7ff270004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:57.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.205+0000 7ff28a30b700 1 --2- 172.21.15.1:0/2350459517 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff28409dd40 0x7ff270004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.205+0000 7ff28a30b700 1 -- 172.21.15.1:0/2350459517 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff270003a00 msgr2=0x7ff270003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:57.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.205+0000 7ff28a30b700 1 --2- 172.21.15.1:0/2350459517 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff270003a00 0x7ff270003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.205+0000 7ff28a30b700 1 -- 172.21.15.1:0/2350459517 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2700035b0 con 0x7ff270004d90 2024-03-18T18:49:57.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.205+0000 7ff28a30b700 1 --2- 172.21.15.1:0/2350459517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff270004d90 0x7ff270005210 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff27c00e170 tx=0x7ff270005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd67e7c98ba87fee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:57.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.205+0000 7ff288b08700 1 -- 172.21.15.1:0/2350459517 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27c00ec90 con 0x7ff270004d90 2024-03-18T18:49:57.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.205+0000 7ff288b08700 1 -- 172.21.15.1:0/2350459517 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff27c00f040 con 0x7ff270004d90 2024-03-18T18:49:57.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.205+0000 7ff288b08700 1 -- 172.21.15.1:0/2350459517 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27c014480 con 0x7ff270004d90 2024-03-18T18:49:57.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.205+0000 7ff28bd6e700 1 -- 172.21.15.1:0/2350459517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff270004d90 msgr2=0x7ff270005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:57.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.205+0000 7ff28bd6e700 1 --2- 172.21.15.1:0/2350459517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff270004d90 0x7ff270005210 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff27c00e170 tx=0x7ff270005880 comp rx=0 tx=0).stop 2024-03-18T18:49:57.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.205+0000 7ff28a30b700 1 -- 172.21.15.1:0/2350459517 reap_dead start 2024-03-18T18:49:57.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.205+0000 7ff28bd6e700 1 -- 172.21.15.1:0/2350459517 shutdown_connections 2024-03-18T18:49:57.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.205+0000 7ff28bd6e700 1 -- 172.21.15.1:0/2350459517 >> 172.21.15.1:0/2350459517 conn(0x7ff2840cca00 msgr2=0x7ff2840cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:57.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.206+0000 7ff28bd6e700 1 -- 172.21.15.1:0/2350459517 shutdown_connections 2024-03-18T18:49:57.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.206+0000 7ff28bd6e700 1 -- 172.21.15.1:0/2350459517 wait complete. 2024-03-18T18:49:57.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.207+0000 7ff28bd6e700 1 Processor -- start 2024-03-18T18:49:57.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.207+0000 7ff28bd6e700 1 -- start start 2024-03-18T18:49:57.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.207+0000 7ff28bd6e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2840c9490 0x7ff2841e4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.207+0000 7ff28bd6e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff284159430 0x7ff2841588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.207+0000 7ff28bd6e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2840c9870 0x7ff284158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.207+0000 7ff28bd6e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2841e5670 con 0x7ff2840c9490 2024-03-18T18:49:57.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.207+0000 7ff28bd6e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2841e4a60 con 0x7ff2840c9870 2024-03-18T18:49:57.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.207+0000 7ff28bd6e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2841e4d40 con 0x7ff284159430 2024-03-18T18:49:57.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.208+0000 7ff289b0a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2840c9490 0x7ff2841e4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.208+0000 7ff289b0a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2840c9490 0x7ff2841e4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49422/0 (socket says 172.21.15.1:49422) 2024-03-18T18:49:57.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.208+0000 7ff289b0a700 1 -- 172.21.15.1:0/3383659247 learned_addr learned my addr 172.21.15.1:0/3383659247 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:57.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.208+0000 7ff28a30b700 1 --2- 172.21.15.1:0/3383659247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2840c9870 0x7ff284158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.208+0000 7ff289309700 1 --2- 172.21.15.1:0/3383659247 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff284159430 0x7ff2841588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.208+0000 7ff289b0a700 1 -- 172.21.15.1:0/3383659247 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff284159430 msgr2=0x7ff2841588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:57.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.208+0000 7ff289b0a700 1 --2- 172.21.15.1:0/3383659247 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff284159430 0x7ff2841588a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.208+0000 7ff289b0a700 1 -- 172.21.15.1:0/3383659247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2840c9870 msgr2=0x7ff284158de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:57.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.208+0000 7ff289b0a700 1 --2- 172.21.15.1:0/3383659247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2840c9870 0x7ff284158de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.208+0000 7ff289b0a700 1 -- 172.21.15.1:0/3383659247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2841e5020 con 0x7ff2840c9490 2024-03-18T18:49:57.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.208+0000 7ff289b0a700 1 --2- 172.21.15.1:0/3383659247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2840c9490 0x7ff2841e4210 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff280001050 tx=0x7ff2800077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:57.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.208+0000 7ff2767fc700 1 -- 172.21.15.1:0/3383659247 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff280011a60 con 0x7ff2840c9490 2024-03-18T18:49:57.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.208+0000 7ff2767fc700 1 -- 172.21.15.1:0/3383659247 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff280011bc0 con 0x7ff2840c9490 2024-03-18T18:49:57.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.208+0000 7ff28bd6e700 1 -- 172.21.15.1:0/3383659247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2841e5300 con 0x7ff2840c9490 2024-03-18T18:49:57.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.208+0000 7ff2767fc700 1 -- 172.21.15.1:0/3383659247 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff280014370 con 0x7ff2840c9490 2024-03-18T18:49:57.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.208+0000 7ff28bd6e700 1 -- 172.21.15.1:0/3383659247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2841e28c0 con 0x7ff2840c9490 2024-03-18T18:49:57.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.209+0000 7ff28bd6e700 1 -- 172.21.15.1:0/3383659247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2840ca950 con 0x7ff2840c9490 2024-03-18T18:49:57.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.210+0000 7ff2767fc700 1 -- 172.21.15.1:0/3383659247 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff2800144d0 con 0x7ff2840c9490 2024-03-18T18:49:57.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.210+0000 7ff2767fc700 1 --2- 172.21.15.1:0/3383659247 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff26806ac90 0x7ff26806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.210+0000 7ff2767fc700 1 -- 172.21.15.1:0/3383659247 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff28008b5f0 con 0x7ff2840c9490 2024-03-18T18:49:57.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.210+0000 7ff289309700 1 --2- 172.21.15.1:0/3383659247 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff26806ac90 0x7ff26806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.211+0000 7ff289309700 1 --2- 172.21.15.1:0/3383659247 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff26806ac90 0x7ff26806d150 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff2840c6490 tx=0x7ff278009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:57.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.215+0000 7ff2767fc700 1 -- 172.21.15.1:0/3383659247 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff2800564e0 con 0x7ff2840c9490 2024-03-18T18:49:57.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.414+0000 7ff28bd6e700 1 -- 172.21.15.1:0/3383659247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff2840676a0 con 0x7ff2840c9490 2024-03-18T18:49:57.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.414+0000 7ff2767fc700 1 -- 172.21.15.1:0/3383659247 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff280059b00 con 0x7ff2840c9490 2024-03-18T18:49:57.411 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T18:49:57.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.416+0000 7ff28bd6e700 1 -- 172.21.15.1:0/3383659247 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff26806ac90 msgr2=0x7ff26806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:57.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.416+0000 7ff28bd6e700 1 --2- 172.21.15.1:0/3383659247 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff26806ac90 0x7ff26806d150 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff2840c6490 tx=0x7ff278009210 comp rx=0 tx=0).stop 2024-03-18T18:49:57.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.416+0000 7ff28bd6e700 1 -- 172.21.15.1:0/3383659247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2840c9490 msgr2=0x7ff2841e4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:57.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.416+0000 7ff28bd6e700 1 --2- 172.21.15.1:0/3383659247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2840c9490 0x7ff2841e4210 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff280001050 tx=0x7ff2800077e0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.416+0000 7ff28bd6e700 1 -- 172.21.15.1:0/3383659247 shutdown_connections 2024-03-18T18:49:57.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.416+0000 7ff28bd6e700 1 --2- 172.21.15.1:0/3383659247 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff26806ac90 0x7ff26806d150 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.416+0000 7ff28bd6e700 1 --2- 172.21.15.1:0/3383659247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2840c9870 0x7ff284158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.416+0000 7ff28bd6e700 1 --2- 172.21.15.1:0/3383659247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2840c9490 0x7ff2841e4210 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.416+0000 7ff28bd6e700 1 --2- 172.21.15.1:0/3383659247 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff284159430 0x7ff2841588a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.416+0000 7ff28bd6e700 1 -- 172.21.15.1:0/3383659247 >> 172.21.15.1:0/3383659247 conn(0x7ff2840cca00 msgr2=0x7ff28415ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:57.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.417+0000 7ff28bd6e700 1 -- 172.21.15.1:0/3383659247 shutdown_connections 2024-03-18T18:49:57.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.417+0000 7ff28bd6e700 1 -- 172.21.15.1:0/3383659247 wait complete. 2024-03-18T18:49:57.425 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T18:49:57.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.556+0000 7fed47579700 1 Processor -- start 2024-03-18T18:49:57.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.556+0000 7fed47579700 1 -- start start 2024-03-18T18:49:57.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.557+0000 7fed47579700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fed400727b0 0x7fed40161610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.557+0000 7fed47579700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fed40161b50 0x7fed4006a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.557+0000 7fed47579700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed40160ea0 con 0x7fed4015ad70 2024-03-18T18:49:57.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.557+0000 7fed47579700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed400725b0 con 0x7fed400727b0 2024-03-18T18:49:57.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.557+0000 7fed47579700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed4015e090 con 0x7fed40161b50 2024-03-18T18:49:57.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.557+0000 7fed44b14700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fed400727b0 0x7fed40161610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.557+0000 7fed44b14700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fed400727b0 0x7fed40161610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49434/0 (socket says 172.21.15.1:49434) 2024-03-18T18:49:57.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.557+0000 7fed44b14700 1 -- 172.21.15.1:0/86512975 learned_addr learned my addr 172.21.15.1:0/86512975 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:57.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.558+0000 7fed45b16700 1 --2- 172.21.15.1:0/86512975 >> v2:172.21.15.165:3300/0 conn(0x7fed40161b50 0x7fed4006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.558+0000 7fed44b14700 1 -- 172.21.15.1:0/86512975 >> v2:172.21.15.165:3300/0 conn(0x7fed40161b50 msgr2=0x7fed4006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:57.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.558+0000 7fed44b14700 1 --2- 172.21.15.1:0/86512975 >> v2:172.21.15.165:3300/0 conn(0x7fed40161b50 0x7fed4006a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.558+0000 7fed44b14700 1 -- 172.21.15.1:0/86512975 >> v1:172.21.15.165:6789/0 conn(0x7fed4015ad70 legacy=0x7fed4015d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:57.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.558+0000 7fed44b14700 1 -- 172.21.15.1:0/86512975 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4015e370 con 0x7fed400727b0 2024-03-18T18:49:57.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.558+0000 7fed45b16700 1 --2- 172.21.15.1:0/86512975 >> v2:172.21.15.165:3300/0 conn(0x7fed40161b50 0x7fed4006a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:57.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.558+0000 7fed44b14700 1 --2- 172.21.15.1:0/86512975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed400727b0 0x7fed40161610 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fed40071600 tx=0x7fed3c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12201170a631f043 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:57.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.558+0000 7fed37fff700 1 -- 172.21.15.1:0/86512975 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c00b870 con 0x7fed400727b0 2024-03-18T18:49:57.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.558+0000 7fed37fff700 1 -- 172.21.15.1:0/86512975 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3c00b9d0 con 0x7fed400727b0 2024-03-18T18:49:57.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.558+0000 7fed37fff700 1 -- 172.21.15.1:0/86512975 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c00bd60 con 0x7fed400727b0 2024-03-18T18:49:57.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.558+0000 7fed47579700 1 -- 172.21.15.1:0/86512975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed400727b0 msgr2=0x7fed40161610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:57.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.558+0000 7fed47579700 1 --2- 172.21.15.1:0/86512975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed400727b0 0x7fed40161610 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fed40071600 tx=0x7fed3c009bd0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.559+0000 7fed47579700 1 -- 172.21.15.1:0/86512975 shutdown_connections 2024-03-18T18:49:57.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.559+0000 7fed47579700 1 --2- 172.21.15.1:0/86512975 >> v2:172.21.15.165:3300/0 conn(0x7fed40161b50 0x7fed4006a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.559+0000 7fed47579700 1 --2- 172.21.15.1:0/86512975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed400727b0 0x7fed40161610 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.559+0000 7fed47579700 1 -- 172.21.15.1:0/86512975 >> 172.21.15.1:0/86512975 conn(0x7fed400b1cb0 msgr2=0x7fed400b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:57.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.559+0000 7fed47579700 1 -- 172.21.15.1:0/86512975 shutdown_connections 2024-03-18T18:49:57.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.559+0000 7fed47579700 1 -- 172.21.15.1:0/86512975 wait complete. 2024-03-18T18:49:57.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.560+0000 7fed47579700 1 Processor -- start 2024-03-18T18:49:57.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.560+0000 7fed47579700 1 -- start start 2024-03-18T18:49:57.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed47579700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed400727b0 0x7fed400ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed47579700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed4015ad70 0x7fed4015b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed47579700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fed40161b50 0x7fed400aee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed47579700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4015c730 con 0x7fed4015ad70 2024-03-18T18:49:57.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed47579700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4015bb20 con 0x7fed400727b0 2024-03-18T18:49:57.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed47579700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed4015be00 con 0x7fed40161b50 2024-03-18T18:49:57.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed44b14700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed4015ad70 0x7fed4015b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed44b14700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed4015ad70 0x7fed4015b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49448/0 (socket says 172.21.15.1:49448) 2024-03-18T18:49:57.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed44b14700 1 -- 172.21.15.1:0/3277181751 learned_addr learned my addr 172.21.15.1:0/3277181751 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:57.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed45b16700 1 --2- 172.21.15.1:0/3277181751 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fed40161b50 0x7fed400aee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed45315700 1 --2- 172.21.15.1:0/3277181751 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed400727b0 0x7fed400ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed44b14700 1 -- 172.21.15.1:0/3277181751 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fed40161b50 msgr2=0x7fed400aee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:57.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed44b14700 1 --2- 172.21.15.1:0/3277181751 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fed40161b50 0x7fed400aee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed44b14700 1 -- 172.21.15.1:0/3277181751 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed400727b0 msgr2=0x7fed400ad2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:57.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed44b14700 1 --2- 172.21.15.1:0/3277181751 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed400727b0 0x7fed400ad2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed44b14700 1 -- 172.21.15.1:0/3277181751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3c009550 con 0x7fed4015ad70 2024-03-18T18:49:57.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed44b14700 1 --2- 172.21.15.1:0/3277181751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed4015ad70 0x7fed4015b2d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fed3c000c00 tx=0x7fed3c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:57.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed35ffb700 1 -- 172.21.15.1:0/3277181751 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c004480 con 0x7fed4015ad70 2024-03-18T18:49:57.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed35ffb700 1 -- 172.21.15.1:0/3277181751 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3c0045e0 con 0x7fed4015ad70 2024-03-18T18:49:57.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed47579700 1 -- 172.21.15.1:0/3277181751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed4015c0e0 con 0x7fed4015ad70 2024-03-18T18:49:57.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.561+0000 7fed35ffb700 1 -- 172.21.15.1:0/3277181751 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c0038e0 con 0x7fed4015ad70 2024-03-18T18:49:57.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.562+0000 7fed47579700 1 -- 172.21.15.1:0/3277181751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed4015cbc0 con 0x7fed4015ad70 2024-03-18T18:49:57.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.562+0000 7fed47579700 1 -- 172.21.15.1:0/3277181751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed4015c8a0 con 0x7fed4015ad70 2024-03-18T18:49:57.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.563+0000 7fed35ffb700 1 -- 172.21.15.1:0/3277181751 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fed3c021070 con 0x7fed4015ad70 2024-03-18T18:49:57.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.563+0000 7fed35ffb700 1 --2- 172.21.15.1:0/3277181751 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fed2406ac90 0x7fed2406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.563+0000 7fed35ffb700 1 -- 172.21.15.1:0/3277181751 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fed3c08ac40 con 0x7fed4015ad70 2024-03-18T18:49:57.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.564+0000 7fed45315700 1 --2- 172.21.15.1:0/3277181751 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fed2406ac90 0x7fed2406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.564+0000 7fed45315700 1 --2- 172.21.15.1:0/3277181751 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fed2406ac90 0x7fed2406d150 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fed30007fd0 tx=0x7fed30009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:57.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.568+0000 7fed35ffb700 1 -- 172.21.15.1:0/3277181751 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fed3c055b40 con 0x7fed4015ad70 2024-03-18T18:49:57.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.770+0000 7fed47579700 1 -- 172.21.15.1:0/3277181751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fed400676a0 con 0x7fed4015ad70 2024-03-18T18:49:57.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.770+0000 7fed35ffb700 1 -- 172.21.15.1:0/3277181751 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7fed3c059160 con 0x7fed4015ad70 2024-03-18T18:49:57.767 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:49:57.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.773+0000 7fed47579700 1 -- 172.21.15.1:0/3277181751 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fed2406ac90 msgr2=0x7fed2406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:57.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.773+0000 7fed47579700 1 --2- 172.21.15.1:0/3277181751 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fed2406ac90 0x7fed2406d150 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fed30007fd0 tx=0x7fed30009040 comp rx=0 tx=0).stop 2024-03-18T18:49:57.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.773+0000 7fed47579700 1 -- 172.21.15.1:0/3277181751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed4015ad70 msgr2=0x7fed4015b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:57.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.773+0000 7fed47579700 1 --2- 172.21.15.1:0/3277181751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed4015ad70 0x7fed4015b2d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fed3c000c00 tx=0x7fed3c004750 comp rx=0 tx=0).stop 2024-03-18T18:49:57.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.773+0000 7fed47579700 1 -- 172.21.15.1:0/3277181751 shutdown_connections 2024-03-18T18:49:57.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.773+0000 7fed47579700 1 --2- 172.21.15.1:0/3277181751 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fed2406ac90 0x7fed2406d150 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.773+0000 7fed47579700 1 --2- 172.21.15.1:0/3277181751 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fed40161b50 0x7fed400aee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.773+0000 7fed47579700 1 --2- 172.21.15.1:0/3277181751 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed400727b0 0x7fed400ad2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.773+0000 7fed47579700 1 --2- 172.21.15.1:0/3277181751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fed4015ad70 0x7fed4015b2d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.773+0000 7fed47579700 1 -- 172.21.15.1:0/3277181751 >> 172.21.15.1:0/3277181751 conn(0x7fed400b1cb0 msgr2=0x7fed400b07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:57.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.773+0000 7fed47579700 1 -- 172.21.15.1:0/3277181751 shutdown_connections 2024-03-18T18:49:57.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.773+0000 7fed47579700 1 -- 172.21.15.1:0/3277181751 wait complete. 2024-03-18T18:49:57.781 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T18:49:57.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.913+0000 7f2b5b17f700 1 Processor -- start 2024-03-18T18:49:57.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.913+0000 7f2b5b17f700 1 -- start start 2024-03-18T18:49:57.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.914+0000 7f2b5b17f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2b54164000 0x7f2b541663f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.914+0000 7f2b5b17f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b5415c340 con 0x7f2b5406a750 2024-03-18T18:49:57.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.914+0000 7f2b5b17f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b5415b7a0 con 0x7f2b540ace80 2024-03-18T18:49:57.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.914+0000 7f2b5b17f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b5415fc90 con 0x7f2b54164000 2024-03-18T18:49:57.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.914+0000 7f2b58f1b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2b5406a750 0x7f2b5406ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54442/0 (socket says 172.21.15.1:54442) 2024-03-18T18:49:57.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.914+0000 7f2b58f1b700 1 -- 172.21.15.1:0/1751031584 learned_addr learned my addr 172.21.15.1:0/1751031584 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:49:57.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.914+0000 7f2b537fe700 1 -- 172.21.15.1:0/1751031584 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4026612397 0 0) 0x7f2b5415c340 con 0x7f2b5406a750 2024-03-18T18:49:57.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.914+0000 7f2b537fe700 1 -- 172.21.15.1:0/1751031584 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b440035b0 con 0x7f2b5406a750 2024-03-18T18:49:57.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.914+0000 7f2b5971c700 1 --2- 172.21.15.1:0/1751031584 >> v2:172.21.15.165:3300/0 conn(0x7f2b54164000 0x7f2b541663f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.914+0000 7f2b537fe700 1 -- 172.21.15.1:0/1751031584 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2384957057 0 0) 0x7f2b440035b0 con 0x7f2b5406a750 2024-03-18T18:49:57.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.914+0000 7f2b537fe700 1 -- 172.21.15.1:0/1751031584 >> v2:172.21.15.165:3300/0 conn(0x7f2b54164000 msgr2=0x7f2b541663f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:57.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.914+0000 7f2b537fe700 1 --2- 172.21.15.1:0/1751031584 >> v2:172.21.15.165:3300/0 conn(0x7f2b54164000 0x7f2b541663f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.914+0000 7f2b537fe700 1 -- 172.21.15.1:0/1751031584 >> v1:172.21.15.165:6789/0 conn(0x7f2b540ace80 legacy=0x7f2b540ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:57.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.914+0000 7f2b537fe700 1 -- 172.21.15.1:0/1751031584 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5415ba10 con 0x7f2b5406a750 2024-03-18T18:49:57.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.914+0000 7f2b537fe700 1 -- 172.21.15.1:0/1751031584 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f2b480017f0 con 0x7f2b5406a750 2024-03-18T18:49:57.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.914+0000 7f2b5971c700 1 --2- 172.21.15.1:0/1751031584 >> v2:172.21.15.165:3300/0 conn(0x7f2b54164000 0x7f2b541663f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:57.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.914+0000 7f2b537fe700 1 -- 172.21.15.1:0/1751031584 >> v1:172.21.15.1:6789/0 conn(0x7f2b5406a750 legacy=0x7f2b5406ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:57.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.914+0000 7f2b537fe700 1 --2- 172.21.15.1:0/1751031584 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b44003a50 0x7f2b44003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.914+0000 7f2b537fe700 1 --2- 172.21.15.1:0/1751031584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b44004850 0x7f2b44004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.915+0000 7f2b537fe700 1 --2- 172.21.15.1:0/1751031584 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b44005260 0x7f2b4400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.915+0000 7f2b537fe700 1 -- 172.21.15.1:0/1751031584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5415fc90 con 0x7f2b44004850 2024-03-18T18:49:57.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.915+0000 7f2b537fe700 1 -- 172.21.15.1:0/1751031584 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4400c1b0 con 0x7f2b44005260 2024-03-18T18:49:57.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.915+0000 7f2b537fe700 1 -- 172.21.15.1:0/1751031584 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4400c370 con 0x7f2b44003a50 2024-03-18T18:49:57.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.915+0000 7f2b53fff700 1 --2- 172.21.15.1:0/1751031584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b44004850 0x7f2b44004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.915+0000 7f2b5971c700 1 --2- 172.21.15.1:0/1751031584 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b44005260 0x7f2b4400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.915+0000 7f2b58f1b700 1 --2- 172.21.15.1:0/1751031584 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b44003a50 0x7f2b44003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.915+0000 7f2b53fff700 1 -- 172.21.15.1:0/1751031584 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b44003a50 msgr2=0x7f2b44003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:57.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.915+0000 7f2b53fff700 1 --2- 172.21.15.1:0/1751031584 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b44003a50 0x7f2b44003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.915+0000 7f2b53fff700 1 -- 172.21.15.1:0/1751031584 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b44005260 msgr2=0x7f2b4400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:57.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.915+0000 7f2b53fff700 1 --2- 172.21.15.1:0/1751031584 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b44005260 0x7f2b4400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.915+0000 7f2b53fff700 1 -- 172.21.15.1:0/1751031584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b440035b0 con 0x7f2b44004850 2024-03-18T18:49:57.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.915+0000 7f2b5971c700 1 -- 172.21.15.1:0/1751031584 reap_dead start 2024-03-18T18:49:57.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.915+0000 7f2b53fff700 1 --2- 172.21.15.1:0/1751031584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b44004850 0x7f2b44004cd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2b400017c0 tx=0x7f2b44005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d75e37ad77d1905 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:57.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.915+0000 7f2b537fe700 1 -- 172.21.15.1:0/1751031584 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b400052d0 con 0x7f2b44004850 2024-03-18T18:49:57.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.915+0000 7f2b537fe700 1 -- 172.21.15.1:0/1751031584 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b40005450 con 0x7f2b44004850 2024-03-18T18:49:57.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.915+0000 7f2b537fe700 1 -- 172.21.15.1:0/1751031584 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b400038c0 con 0x7f2b44004850 2024-03-18T18:49:57.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.916+0000 7f2b5b17f700 1 -- 172.21.15.1:0/1751031584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b44004850 msgr2=0x7f2b44004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:57.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.916+0000 7f2b5b17f700 1 --2- 172.21.15.1:0/1751031584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b44004850 0x7f2b44004cd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2b400017c0 tx=0x7f2b44005880 comp rx=0 tx=0).stop 2024-03-18T18:49:57.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.916+0000 7f2b5b17f700 1 -- 172.21.15.1:0/1751031584 shutdown_connections 2024-03-18T18:49:57.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.916+0000 7f2b5b17f700 1 --2- 172.21.15.1:0/1751031584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b44004850 0x7f2b44004cd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.916+0000 7f2b5b17f700 1 -- 172.21.15.1:0/1751031584 >> 172.21.15.1:0/1751031584 conn(0x7f2b540cc9b0 msgr2=0x7f2b540ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:57.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.916+0000 7f2b5b17f700 1 -- 172.21.15.1:0/1751031584 shutdown_connections 2024-03-18T18:49:57.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.916+0000 7f2b5b17f700 1 -- 172.21.15.1:0/1751031584 wait complete. 2024-03-18T18:49:57.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.917+0000 7f2b5b17f700 1 Processor -- start 2024-03-18T18:49:57.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.917+0000 7f2b5b17f700 1 -- start start 2024-03-18T18:49:57.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.918+0000 7f2b5b17f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b44004850 0x7f2b5415dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.918+0000 7f2b5b17f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b54154320 0x7f2b54153300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.918+0000 7f2b5b17f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b54071ec0 0x7f2b54072370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.918+0000 7f2b5b17f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b540728b0 con 0x7f2b44004850 2024-03-18T18:49:57.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.918+0000 7f2b5b17f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b54154170 con 0x7f2b54071ec0 2024-03-18T18:49:57.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.918+0000 7f2b5b17f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5406e0f0 con 0x7f2b54154320 2024-03-18T18:49:57.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.918+0000 7f2b58f1b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b44004850 0x7f2b5415dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.918+0000 7f2b58f1b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b44004850 0x7f2b5415dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49464/0 (socket says 172.21.15.1:49464) 2024-03-18T18:49:57.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.918+0000 7f2b58f1b700 1 -- 172.21.15.1:0/1210322336 learned_addr learned my addr 172.21.15.1:0/1210322336 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:57.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.918+0000 7f2b53fff700 1 --2- 172.21.15.1:0/1210322336 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b54154320 0x7f2b54153300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.918+0000 7f2b5971c700 1 --2- 172.21.15.1:0/1210322336 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b54071ec0 0x7f2b54072370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.918+0000 7f2b58f1b700 1 -- 172.21.15.1:0/1210322336 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b54154320 msgr2=0x7f2b54153300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:57.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.918+0000 7f2b58f1b700 1 --2- 172.21.15.1:0/1210322336 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b54154320 0x7f2b54153300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.918+0000 7f2b58f1b700 1 -- 172.21.15.1:0/1210322336 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b54071ec0 msgr2=0x7f2b54072370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.918+0000 7f2b58f1b700 1 --2- 172.21.15.1:0/1210322336 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b54071ec0 0x7f2b54072370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.918+0000 7f2b58f1b700 1 -- 172.21.15.1:0/1210322336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5406e3d0 con 0x7f2b44004850 2024-03-18T18:49:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.918+0000 7f2b58f1b700 1 --2- 172.21.15.1:0/1210322336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b44004850 0x7f2b5415dde0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2b4800f7f0 tx=0x7f2b4800fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.918+0000 7f2b517fa700 1 -- 172.21.15.1:0/1210322336 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b480072e0 con 0x7f2b44004850 2024-03-18T18:49:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.919+0000 7f2b517fa700 1 -- 172.21.15.1:0/1210322336 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b48007460 con 0x7f2b44004850 2024-03-18T18:49:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.919+0000 7f2b5b17f700 1 -- 172.21.15.1:0/1210322336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b5406e6b0 con 0x7f2b44004850 2024-03-18T18:49:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.919+0000 7f2b517fa700 1 -- 172.21.15.1:0/1210322336 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b48005830 con 0x7f2b44004850 2024-03-18T18:49:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.919+0000 7f2b5b17f700 1 -- 172.21.15.1:0/1210322336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b5415e320 con 0x7f2b44004850 2024-03-18T18:49:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.920+0000 7f2b5b17f700 1 -- 172.21.15.1:0/1210322336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b540676a0 con 0x7f2b44004850 2024-03-18T18:49:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.920+0000 7f2b517fa700 1 -- 172.21.15.1:0/1210322336 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2b48005990 con 0x7f2b44004850 2024-03-18T18:49:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.920+0000 7f2b517fa700 1 --2- 172.21.15.1:0/1210322336 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2b3806ac90 0x7f2b3806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:57.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.920+0000 7f2b517fa700 1 -- 172.21.15.1:0/1210322336 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2b480894e0 con 0x7f2b44004850 2024-03-18T18:49:57.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.921+0000 7f2b53fff700 1 --2- 172.21.15.1:0/1210322336 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2b3806ac90 0x7f2b3806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:57.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.921+0000 7f2b53fff700 1 --2- 172.21.15.1:0/1210322336 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2b3806ac90 0x7f2b3806d150 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2b40005040 tx=0x7f2b40009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:57.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:57.925+0000 7f2b517fa700 1 -- 172.21.15.1:0/1210322336 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2b48054460 con 0x7f2b44004850 2024-03-18T18:49:58.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.126+0000 7f2b5b17f700 1 -- 172.21.15.1:0/1210322336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2b540cd640 con 0x7f2b44004850 2024-03-18T18:49:58.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.126+0000 7f2b517fa700 1 -- 172.21.15.1:0/1210322336 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2b48057a80 con 0x7f2b44004850 2024-03-18T18:49:58.123 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T18:49:58.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.128+0000 7f2b5b17f700 1 -- 172.21.15.1:0/1210322336 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2b3806ac90 msgr2=0x7f2b3806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:58.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.128+0000 7f2b5b17f700 1 --2- 172.21.15.1:0/1210322336 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2b3806ac90 0x7f2b3806d150 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2b40005040 tx=0x7f2b40009030 comp rx=0 tx=0).stop 2024-03-18T18:49:58.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.128+0000 7f2b5b17f700 1 -- 172.21.15.1:0/1210322336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b44004850 msgr2=0x7f2b5415dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:58.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.128+0000 7f2b5b17f700 1 --2- 172.21.15.1:0/1210322336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b44004850 0x7f2b5415dde0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2b4800f7f0 tx=0x7f2b4800fbb0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.129+0000 7f2b5b17f700 1 -- 172.21.15.1:0/1210322336 shutdown_connections 2024-03-18T18:49:58.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.129+0000 7f2b5b17f700 1 --2- 172.21.15.1:0/1210322336 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2b3806ac90 0x7f2b3806d150 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.129+0000 7f2b5b17f700 1 --2- 172.21.15.1:0/1210322336 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b54071ec0 0x7f2b54072370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.129+0000 7f2b5b17f700 1 --2- 172.21.15.1:0/1210322336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b44004850 0x7f2b5415dde0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.129+0000 7f2b5b17f700 1 --2- 172.21.15.1:0/1210322336 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b54154320 0x7f2b54153300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.129+0000 7f2b5b17f700 1 -- 172.21.15.1:0/1210322336 >> 172.21.15.1:0/1210322336 conn(0x7f2b540cc9b0 msgr2=0x7f2b540ad290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:58.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.129+0000 7f2b5b17f700 1 -- 172.21.15.1:0/1210322336 shutdown_connections 2024-03-18T18:49:58.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.129+0000 7f2b5b17f700 1 -- 172.21.15.1:0/1210322336 wait complete. 2024-03-18T18:49:58.137 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T18:49:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.269+0000 7f23de36d700 1 Processor -- start 2024-03-18T18:49:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.269+0000 7f23de36d700 1 -- start start 2024-03-18T18:49:58.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.269+0000 7f23de36d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f23d80b3c30 0x7f23d80b2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:58.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.269+0000 7f23de36d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23d80cef70 0x7f23d80b4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.269+0000 7f23de36d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d80b3560 con 0x7f23d806ce40 2024-03-18T18:49:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.269+0000 7f23de36d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d80b4550 con 0x7f23d80b3c30 2024-03-18T18:49:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.269+0000 7f23de36d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d8071a80 con 0x7f23d80cef70 2024-03-18T18:49:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.270+0000 7f23d7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f23d80b3c30 0x7f23d80b2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.270+0000 7f23d7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f23d80b3c30 0x7f23d80b2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49474/0 (socket says 172.21.15.1:49474) 2024-03-18T18:49:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.270+0000 7f23d7fff700 1 -- 172.21.15.1:0/3547641388 learned_addr learned my addr 172.21.15.1:0/3547641388 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.270+0000 7f23d77fe700 1 --2- 172.21.15.1:0/3547641388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23d80cef70 0x7f23d80b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.270+0000 7f23d7fff700 1 -- 172.21.15.1:0/3547641388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23d80cef70 msgr2=0x7f23d80b4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.270+0000 7f23d7fff700 1 --2- 172.21.15.1:0/3547641388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23d80cef70 0x7f23d80b4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.270+0000 7f23d7fff700 1 -- 172.21.15.1:0/3547641388 >> v1:172.21.15.1:6789/0 conn(0x7f23d806ce40 legacy=0x7f23d815da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.270+0000 7f23d6ffd700 1 -- 172.21.15.1:0/3547641388 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49649691 0 0) 0x7f23d80b3560 con 0x7f23d806ce40 2024-03-18T18:49:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.270+0000 7f23d7fff700 1 -- 172.21.15.1:0/3547641388 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d80b2cc0 con 0x7f23d80b3c30 2024-03-18T18:49:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.270+0000 7f23d7fff700 1 --2- 172.21.15.1:0/3547641388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d80b3c30 0x7f23d80b2650 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f23c80098f0 tx=0x7f23c8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc45a4430f32d453 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.270+0000 7f23d6ffd700 1 -- 172.21.15.1:0/3547641388 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c800b8a0 con 0x7f23d80b3c30 2024-03-18T18:49:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.270+0000 7f23d6ffd700 1 -- 172.21.15.1:0/3547641388 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c800ba00 con 0x7f23d80b3c30 2024-03-18T18:49:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.270+0000 7f23d6ffd700 1 -- 172.21.15.1:0/3547641388 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c800bd90 con 0x7f23d80b3c30 2024-03-18T18:49:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.271+0000 7f23de36d700 1 -- 172.21.15.1:0/3547641388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d80b3c30 msgr2=0x7f23d80b2650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.271+0000 7f23de36d700 1 --2- 172.21.15.1:0/3547641388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d80b3c30 0x7f23d80b2650 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f23c80098f0 tx=0x7f23c8009c00 comp rx=0 tx=0).stop 2024-03-18T18:49:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.271+0000 7f23de36d700 1 -- 172.21.15.1:0/3547641388 shutdown_connections 2024-03-18T18:49:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.271+0000 7f23de36d700 1 --2- 172.21.15.1:0/3547641388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d80b3c30 0x7f23d80b2650 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.271+0000 7f23de36d700 1 --2- 172.21.15.1:0/3547641388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23d80cef70 0x7f23d80b4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.271+0000 7f23de36d700 1 -- 172.21.15.1:0/3547641388 >> 172.21.15.1:0/3547641388 conn(0x7f23d80b1350 msgr2=0x7f23d80b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.271+0000 7f23de36d700 1 -- 172.21.15.1:0/3547641388 shutdown_connections 2024-03-18T18:49:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.271+0000 7f23de36d700 1 -- 172.21.15.1:0/3547641388 wait complete. 2024-03-18T18:49:58.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.272+0000 7f23de36d700 1 Processor -- start 2024-03-18T18:49:58.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.272+0000 7f23de36d700 1 -- start start 2024-03-18T18:49:58.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.273+0000 7f23de36d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d806ce40 0x7f23d80add40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:58.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.273+0000 7f23de36d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23d80b3c30 0x7f23d81659d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:58.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.273+0000 7f23de36d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23d80cef70 0x7f23d80ad020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:58.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.273+0000 7f23de36d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d80ac8b0 con 0x7f23d806ce40 2024-03-18T18:49:58.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.273+0000 7f23de36d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d8150b60 con 0x7f23d80b3c30 2024-03-18T18:49:58.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.273+0000 7f23de36d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d8150e40 con 0x7f23d80cef70 2024-03-18T18:49:58.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.273+0000 7f23d7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d806ce40 0x7f23d80add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:58.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.273+0000 7f23d7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d806ce40 0x7f23d80add40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49490/0 (socket says 172.21.15.1:49490) 2024-03-18T18:49:58.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.273+0000 7f23d7fff700 1 -- 172.21.15.1:0/3651831036 learned_addr learned my addr 172.21.15.1:0/3651831036 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:58.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.273+0000 7f23d77fe700 1 --2- 172.21.15.1:0/3651831036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23d80b3c30 0x7f23d81659d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:58.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.273+0000 7f23dc90a700 1 --2- 172.21.15.1:0/3651831036 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23d80cef70 0x7f23d80ad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:58.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.273+0000 7f23d7fff700 1 -- 172.21.15.1:0/3651831036 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23d80cef70 msgr2=0x7f23d80ad020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:58.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.273+0000 7f23d7fff700 1 --2- 172.21.15.1:0/3651831036 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23d80cef70 0x7f23d80ad020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.273+0000 7f23d7fff700 1 -- 172.21.15.1:0/3651831036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23d80b3c30 msgr2=0x7f23d81659d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:58.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.273+0000 7f23d7fff700 1 --2- 172.21.15.1:0/3651831036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23d80b3c30 0x7f23d81659d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.273+0000 7f23d7fff700 1 -- 172.21.15.1:0/3651831036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c8009550 con 0x7f23d806ce40 2024-03-18T18:49:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.273+0000 7f23d77fe700 1 --2- 172.21.15.1:0/3651831036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23d80b3c30 0x7f23d81659d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.273+0000 7f23dc90a700 1 --2- 172.21.15.1:0/3651831036 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23d80cef70 0x7f23d80ad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.274+0000 7f23d7fff700 1 --2- 172.21.15.1:0/3651831036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d806ce40 0x7f23d80add40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f23c8009c40 tx=0x7f23c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.274+0000 7f23d4ff9700 1 -- 172.21.15.1:0/3651831036 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c8004380 con 0x7f23d806ce40 2024-03-18T18:49:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.274+0000 7f23d4ff9700 1 -- 172.21.15.1:0/3651831036 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c801f070 con 0x7f23d806ce40 2024-03-18T18:49:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.274+0000 7f23de36d700 1 -- 172.21.15.1:0/3651831036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d8151120 con 0x7f23d806ce40 2024-03-18T18:49:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.274+0000 7f23d4ff9700 1 -- 172.21.15.1:0/3651831036 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c80036a0 con 0x7f23d806ce40 2024-03-18T18:49:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.274+0000 7f23de36d700 1 -- 172.21.15.1:0/3651831036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d80ae280 con 0x7f23d806ce40 2024-03-18T18:49:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.275+0000 7f23de36d700 1 -- 172.21.15.1:0/3651831036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d80acd40 con 0x7f23d806ce40 2024-03-18T18:49:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.276+0000 7f23d4ff9700 1 -- 172.21.15.1:0/3651831036 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f23c8021030 con 0x7f23d806ce40 2024-03-18T18:49:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.276+0000 7f23d4ff9700 1 --2- 172.21.15.1:0/3651831036 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f23c406ac90 0x7f23c406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.276+0000 7f23d4ff9700 1 -- 172.21.15.1:0/3651831036 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f23c808add0 con 0x7f23d806ce40 2024-03-18T18:49:58.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.276+0000 7f23d77fe700 1 --2- 172.21.15.1:0/3651831036 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f23c406ac90 0x7f23c406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:58.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.276+0000 7f23d77fe700 1 --2- 172.21.15.1:0/3651831036 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f23c406ac90 0x7f23c406d150 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f23d81f6bd0 tx=0x7f23cc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:58.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.280+0000 7f23d4ff9700 1 -- 172.21.15.1:0/3651831036 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f23c8055e30 con 0x7f23d806ce40 2024-03-18T18:49:58.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.483+0000 7f23de36d700 1 -- 172.21.15.1:0/3651831036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f23d80b3970 con 0x7f23d806ce40 2024-03-18T18:49:58.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.483+0000 7f23d4ff9700 1 -- 172.21.15.1:0/3651831036 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f23c8015020 con 0x7f23d806ce40 2024-03-18T18:49:58.480 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:49:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.485+0000 7f23de36d700 1 -- 172.21.15.1:0/3651831036 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f23c406ac90 msgr2=0x7f23c406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.485+0000 7f23de36d700 1 --2- 172.21.15.1:0/3651831036 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f23c406ac90 0x7f23c406d150 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f23d81f6bd0 tx=0x7f23cc008040 comp rx=0 tx=0).stop 2024-03-18T18:49:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.485+0000 7f23de36d700 1 -- 172.21.15.1:0/3651831036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d806ce40 msgr2=0x7f23d80add40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.485+0000 7f23de36d700 1 --2- 172.21.15.1:0/3651831036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d806ce40 0x7f23d80add40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f23c8009c40 tx=0x7f23c8004750 comp rx=0 tx=0).stop 2024-03-18T18:49:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.485+0000 7f23de36d700 1 -- 172.21.15.1:0/3651831036 shutdown_connections 2024-03-18T18:49:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.485+0000 7f23de36d700 1 --2- 172.21.15.1:0/3651831036 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f23c406ac90 0x7f23c406d150 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.485+0000 7f23de36d700 1 --2- 172.21.15.1:0/3651831036 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23d80cef70 0x7f23d80ad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.485+0000 7f23de36d700 1 --2- 172.21.15.1:0/3651831036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d806ce40 0x7f23d80add40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.485+0000 7f23de36d700 1 --2- 172.21.15.1:0/3651831036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23d80b3c30 0x7f23d81659d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.485+0000 7f23de36d700 1 -- 172.21.15.1:0/3651831036 >> 172.21.15.1:0/3651831036 conn(0x7f23d80b1350 msgr2=0x7f23d80afa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.485+0000 7f23de36d700 1 -- 172.21.15.1:0/3651831036 shutdown_connections 2024-03-18T18:49:58.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.485+0000 7f23de36d700 1 -- 172.21.15.1:0/3651831036 wait complete. 2024-03-18T18:49:58.494 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T18:49:58.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.623+0000 7f306e611700 1 Processor -- start 2024-03-18T18:49:58.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.623+0000 7f306e611700 1 -- start start 2024-03-18T18:49:58.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.624+0000 7f306e611700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f306815d780 0x7f30680cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:58.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.624+0000 7f306e611700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f306809dd40 0x7f30680cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:58.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.624+0000 7f306e611700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30680ce090 con 0x7f30680ce5a0 2024-03-18T18:49:58.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.624+0000 7f306e611700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f306806db30 con 0x7f306809dd40 2024-03-18T18:49:58.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.624+0000 7f306e611700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f306806d520 con 0x7f306815d780 2024-03-18T18:49:58.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.624+0000 7f30677fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f306809dd40 0x7f30680cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:58.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.624+0000 7f30677fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f306809dd40 0x7f30680cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49502/0 (socket says 172.21.15.1:49502) 2024-03-18T18:49:58.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.624+0000 7f30677fe700 1 -- 172.21.15.1:0/4194907410 learned_addr learned my addr 172.21.15.1:0/4194907410 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:58.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.624+0000 7f3067fff700 1 --2- 172.21.15.1:0/4194907410 >> v2:172.21.15.165:3300/0 conn(0x7f306815d780 0x7f30680cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:58.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.624+0000 7f30677fe700 1 -- 172.21.15.1:0/4194907410 >> v2:172.21.15.165:3300/0 conn(0x7f306815d780 msgr2=0x7f30680cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:49:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.624+0000 7f30677fe700 1 --2- 172.21.15.1:0/4194907410 >> v2:172.21.15.165:3300/0 conn(0x7f306815d780 0x7f30680cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.624+0000 7f30677fe700 1 -- 172.21.15.1:0/4194907410 >> v1:172.21.15.165:6789/0 conn(0x7f30680ce5a0 legacy=0x7f30680ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.624+0000 7f30677fe700 1 -- 172.21.15.1:0/4194907410 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30680cf0e0 con 0x7f306809dd40 2024-03-18T18:49:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.624+0000 7f30677fe700 1 --2- 172.21.15.1:0/4194907410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f306809dd40 0x7f30680cad70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f30680674b0 tx=0x7f305c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfacdab47be4fb30 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.624+0000 7f3066ffd700 1 -- 172.21.15.1:0/4194907410 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c011830 con 0x7f306809dd40 2024-03-18T18:49:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.624+0000 7f3066ffd700 1 -- 172.21.15.1:0/4194907410 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305c011990 con 0x7f306809dd40 2024-03-18T18:49:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.625+0000 7f3066ffd700 1 -- 172.21.15.1:0/4194907410 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c011d20 con 0x7f306809dd40 2024-03-18T18:49:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.625+0000 7f306e611700 1 -- 172.21.15.1:0/4194907410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f306809dd40 msgr2=0x7f30680cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.625+0000 7f306e611700 1 --2- 172.21.15.1:0/4194907410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f306809dd40 0x7f30680cad70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f30680674b0 tx=0x7f305c009bd0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.625+0000 7f306e611700 1 -- 172.21.15.1:0/4194907410 shutdown_connections 2024-03-18T18:49:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.625+0000 7f306e611700 1 --2- 172.21.15.1:0/4194907410 >> v2:172.21.15.165:3300/0 conn(0x7f306815d780 0x7f30680cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.625+0000 7f306e611700 1 --2- 172.21.15.1:0/4194907410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f306809dd40 0x7f30680cad70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.625+0000 7f306e611700 1 -- 172.21.15.1:0/4194907410 >> 172.21.15.1:0/4194907410 conn(0x7f30680cca00 msgr2=0x7f30680cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:58.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.625+0000 7f306e611700 1 -- 172.21.15.1:0/4194907410 shutdown_connections 2024-03-18T18:49:58.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.625+0000 7f306e611700 1 -- 172.21.15.1:0/4194907410 wait complete. 2024-03-18T18:49:58.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.626+0000 7f306e611700 1 Processor -- start 2024-03-18T18:49:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.627+0000 7f306e611700 1 -- start start 2024-03-18T18:49:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.627+0000 7f306e611700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f306809dd40 0x7f306815c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.627+0000 7f306e611700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30680ce5a0 0x7f30680ad800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.627+0000 7f306e611700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f306815d780 0x7f30680b0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.627+0000 7f306e611700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30680af4a0 con 0x7f306815d780 2024-03-18T18:49:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.627+0000 7f306e611700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30680ae400 con 0x7f306809dd40 2024-03-18T18:49:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.627+0000 7f306e611700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30680ae6e0 con 0x7f30680ce5a0 2024-03-18T18:49:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.627+0000 7f306cbae700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f306815d780 0x7f30680b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.627+0000 7f306cbae700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f306815d780 0x7f30680b0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49514/0 (socket says 172.21.15.1:49514) 2024-03-18T18:49:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.627+0000 7f306cbae700 1 -- 172.21.15.1:0/3864877454 learned_addr learned my addr 172.21.15.1:0/3864877454 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:49:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.628+0000 7f30677fe700 1 --2- 172.21.15.1:0/3864877454 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30680ce5a0 0x7f30680ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:58.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.628+0000 7f3067fff700 1 --2- 172.21.15.1:0/3864877454 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f306809dd40 0x7f306815c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.628+0000 7f306cbae700 1 -- 172.21.15.1:0/3864877454 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30680ce5a0 msgr2=0x7f30680ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.628+0000 7f306cbae700 1 --2- 172.21.15.1:0/3864877454 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30680ce5a0 0x7f30680ad800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.628+0000 7f306cbae700 1 -- 172.21.15.1:0/3864877454 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f306809dd40 msgr2=0x7f306815c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.628+0000 7f306cbae700 1 --2- 172.21.15.1:0/3864877454 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f306809dd40 0x7f306815c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.628+0000 7f306cbae700 1 -- 172.21.15.1:0/3864877454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305c009550 con 0x7f306815d780 2024-03-18T18:49:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.628+0000 7f30677fe700 1 --2- 172.21.15.1:0/3864877454 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30680ce5a0 0x7f30680ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:49:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.628+0000 7f306cbae700 1 --2- 172.21.15.1:0/3864877454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f306815d780 0x7f30680b0080 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f306000cc50 tx=0x7f306000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.628+0000 7f3064ff9700 1 -- 172.21.15.1:0/3864877454 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3060003c00 con 0x7f306815d780 2024-03-18T18:49:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.628+0000 7f3064ff9700 1 -- 172.21.15.1:0/3864877454 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3060003d60 con 0x7f306815d780 2024-03-18T18:49:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.628+0000 7f306e611700 1 -- 172.21.15.1:0/3864877454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30680ae9c0 con 0x7f306815d780 2024-03-18T18:49:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.628+0000 7f3064ff9700 1 -- 172.21.15.1:0/3864877454 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306000ab60 con 0x7f306815d780 2024-03-18T18:49:58.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.628+0000 7f306e611700 1 -- 172.21.15.1:0/3864877454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30680aef80 con 0x7f306815d780 2024-03-18T18:49:58.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.629+0000 7f306e611700 1 -- 172.21.15.1:0/3864877454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30680676a0 con 0x7f306815d780 2024-03-18T18:49:58.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.629+0000 7f3064ff9700 1 -- 172.21.15.1:0/3864877454 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f306000acc0 con 0x7f306815d780 2024-03-18T18:49:58.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.630+0000 7f3064ff9700 1 --2- 172.21.15.1:0/3864877454 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f305406ac90 0x7f305406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:49:58.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.630+0000 7f3064ff9700 1 -- 172.21.15.1:0/3864877454 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f30600895d0 con 0x7f306815d780 2024-03-18T18:49:58.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.630+0000 7f3067fff700 1 --2- 172.21.15.1:0/3864877454 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f305406ac90 0x7f305406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:49:58.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.631+0000 7f3067fff700 1 --2- 172.21.15.1:0/3864877454 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f305406ac90 0x7f305406d150 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f306815cfe0 tx=0x7f3058009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:49:58.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.635+0000 7f3064ff9700 1 -- 172.21.15.1:0/3864877454 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3060054550 con 0x7f306815d780 2024-03-18T18:49:58.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.852+0000 7f306e611700 1 -- 172.21.15.1:0/3864877454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3068072490 con 0x7f306815d780 2024-03-18T18:49:58.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.852+0000 7f3064ff9700 1 -- 172.21.15.1:0/3864877454 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3060057b70 con 0x7f306815d780 2024-03-18T18:49:58.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.854+0000 7f306e611700 1 -- 172.21.15.1:0/3864877454 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f305406ac90 msgr2=0x7f305406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:58.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.854+0000 7f306e611700 1 --2- 172.21.15.1:0/3864877454 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f305406ac90 0x7f305406d150 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f306815cfe0 tx=0x7f3058009210 comp rx=0 tx=0).stop 2024-03-18T18:49:58.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.854+0000 7f306e611700 1 -- 172.21.15.1:0/3864877454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f306815d780 msgr2=0x7f30680b0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:49:58.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.854+0000 7f306e611700 1 --2- 172.21.15.1:0/3864877454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f306815d780 0x7f30680b0080 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f306000cc50 tx=0x7f306000cf60 comp rx=0 tx=0).stop 2024-03-18T18:49:58.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.854+0000 7f306e611700 1 -- 172.21.15.1:0/3864877454 shutdown_connections 2024-03-18T18:49:58.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.854+0000 7f306e611700 1 --2- 172.21.15.1:0/3864877454 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f305406ac90 0x7f305406d150 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.854+0000 7f306e611700 1 --2- 172.21.15.1:0/3864877454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f306815d780 0x7f30680b0080 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.854+0000 7f306e611700 1 --2- 172.21.15.1:0/3864877454 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f306809dd40 0x7f306815c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.854+0000 7f306e611700 1 --2- 172.21.15.1:0/3864877454 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30680ce5a0 0x7f30680ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:49:58.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.854+0000 7f306e611700 1 -- 172.21.15.1:0/3864877454 >> 172.21.15.1:0/3864877454 conn(0x7f30680cca00 msgr2=0x7f306806a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:49:58.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.855+0000 7f306e611700 1 -- 172.21.15.1:0/3864877454 shutdown_connections 2024-03-18T18:49:58.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:49:58.855+0000 7f306e611700 1 -- 172.21.15.1:0/3864877454 wait complete. 2024-03-18T18:49:58.852 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-18T18:49:58.863 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-18T18:49:58.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:49:58.863 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-18T18:49:58.900 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi001.stderr:ceph-fuse[84348]: fuse finished with error 0 and tester_r 0 2024-03-18T18:49:58.901 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T18:50:05.003 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T18:50:05.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:05.003 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-18T18:50:05.046 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:05.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.230+0000 7f5846031700 1 Processor -- start 2024-03-18T18:50:05.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.230+0000 7f5846031700 1 -- start start 2024-03-18T18:50:05.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.231+0000 7f5846031700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f584015b4d0 0x7f584009a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:05.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.231+0000 7f5846031700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f58400cf750 0x7f5840099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:05.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.231+0000 7f5846031700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f584006f070 con 0x7f5840099f80 2024-03-18T18:50:05.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.231+0000 7f5846031700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f584009a410 con 0x7f58400cf750 2024-03-18T18:50:05.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.231+0000 7f5846031700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58400d0260 con 0x7f584015b4d0 2024-03-18T18:50:05.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.231+0000 7f583effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f58400cf750 0x7f5840099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:05.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.231+0000 7f583effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f58400cf750 0x7f5840099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44976/0 (socket says 172.21.15.1:44976) 2024-03-18T18:50:05.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.231+0000 7f583effd700 1 -- 172.21.15.1:0/1608057814 learned_addr learned my addr 172.21.15.1:0/1608057814 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:05.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.231+0000 7f583f7fe700 1 --2- 172.21.15.1:0/1608057814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f584015b4d0 0x7f584009a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:05.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.231+0000 7f583effd700 1 -- 172.21.15.1:0/1608057814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f584015b4d0 msgr2=0x7f584009a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:05.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.231+0000 7f583effd700 1 --2- 172.21.15.1:0/1608057814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f584015b4d0 0x7f584009a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.231+0000 7f583effd700 1 -- 172.21.15.1:0/1608057814 >> v1:172.21.15.165:6789/0 conn(0x7f5840099f80 legacy=0x7f58400cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:05.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.231+0000 7f583effd700 1 -- 172.21.15.1:0/1608057814 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58400cff80 con 0x7f58400cf750 2024-03-18T18:50:05.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.231+0000 7f583effd700 1 --2- 172.21.15.1:0/1608057814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58400cf750 0x7f5840099a40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f58340098f0 tx=0x7f5834009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1724b68b7a068037 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:05.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.231+0000 7f583e7fc700 1 -- 172.21.15.1:0/1608057814 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5834004230 con 0x7f58400cf750 2024-03-18T18:50:05.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.231+0000 7f583e7fc700 1 -- 172.21.15.1:0/1608057814 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5834004390 con 0x7f58400cf750 2024-03-18T18:50:05.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.231+0000 7f583e7fc700 1 -- 172.21.15.1:0/1608057814 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58340036a0 con 0x7f58400cf750 2024-03-18T18:50:05.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.232+0000 7f5846031700 1 -- 172.21.15.1:0/1608057814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58400cf750 msgr2=0x7f5840099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:05.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.232+0000 7f5846031700 1 --2- 172.21.15.1:0/1608057814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58400cf750 0x7f5840099a40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f58340098f0 tx=0x7f5834009c00 comp rx=0 tx=0).stop 2024-03-18T18:50:05.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.232+0000 7f5846031700 1 -- 172.21.15.1:0/1608057814 shutdown_connections 2024-03-18T18:50:05.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.232+0000 7f5846031700 1 --2- 172.21.15.1:0/1608057814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f584015b4d0 0x7f584009a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.232+0000 7f5846031700 1 --2- 172.21.15.1:0/1608057814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58400cf750 0x7f5840099a40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.232+0000 7f5846031700 1 -- 172.21.15.1:0/1608057814 >> 172.21.15.1:0/1608057814 conn(0x7f58400b2cb0 msgr2=0x7f58400b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:05.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.232+0000 7f5846031700 1 -- 172.21.15.1:0/1608057814 shutdown_connections 2024-03-18T18:50:05.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.232+0000 7f5846031700 1 -- 172.21.15.1:0/1608057814 wait complete. 2024-03-18T18:50:05.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.233+0000 7f5846031700 1 Processor -- start 2024-03-18T18:50:05.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.234+0000 7f5846031700 1 -- start start 2024-03-18T18:50:05.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.234+0000 7f5846031700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5840099f80 0x7f5840151660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:05.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.234+0000 7f5846031700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58400cf750 0x7f58400c7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:05.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.234+0000 7f5846031700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f584015b4d0 0x7f584015de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:05.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.234+0000 7f5846031700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58400c6860 con 0x7f58400cf750 2024-03-18T18:50:05.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.234+0000 7f5846031700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58400c6b40 con 0x7f584015b4d0 2024-03-18T18:50:05.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.234+0000 7f5846031700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58400c6e20 con 0x7f5840099f80 2024-03-18T18:50:05.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.234+0000 7f583effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58400cf750 0x7f58400c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:05.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.234+0000 7f583effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58400cf750 0x7f58400c7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44984/0 (socket says 172.21.15.1:44984) 2024-03-18T18:50:05.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.234+0000 7f583effd700 1 -- 172.21.15.1:0/4080445472 learned_addr learned my addr 172.21.15.1:0/4080445472 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:05.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.235+0000 7f583ffff700 1 --2- 172.21.15.1:0/4080445472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f584015b4d0 0x7f584015de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:05.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.235+0000 7f583f7fe700 1 --2- 172.21.15.1:0/4080445472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5840099f80 0x7f5840151660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:05.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.235+0000 7f583effd700 1 -- 172.21.15.1:0/4080445472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5840099f80 msgr2=0x7f5840151660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:05.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.235+0000 7f583effd700 1 --2- 172.21.15.1:0/4080445472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5840099f80 0x7f5840151660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.235+0000 7f583effd700 1 -- 172.21.15.1:0/4080445472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f584015b4d0 msgr2=0x7f584015de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:05.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.235+0000 7f583effd700 1 --2- 172.21.15.1:0/4080445472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f584015b4d0 0x7f584015de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.235+0000 7f583effd700 1 -- 172.21.15.1:0/4080445472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5834009550 con 0x7f58400cf750 2024-03-18T18:50:05.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.235+0000 7f583f7fe700 1 --2- 172.21.15.1:0/4080445472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5840099f80 0x7f5840151660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:05.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.235+0000 7f583effd700 1 --2- 172.21.15.1:0/4080445472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58400cf750 0x7f58400c7420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5840072230 tx=0x7f5834004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:05.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.235+0000 7f5827fff700 1 -- 172.21.15.1:0/4080445472 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5834003dc0 con 0x7f58400cf750 2024-03-18T18:50:05.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.235+0000 7f5827fff700 1 -- 172.21.15.1:0/4080445472 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583401f070 con 0x7f58400cf750 2024-03-18T18:50:05.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.235+0000 7f5846031700 1 -- 172.21.15.1:0/4080445472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58400c9380 con 0x7f58400cf750 2024-03-18T18:50:05.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.235+0000 7f5827fff700 1 -- 172.21.15.1:0/4080445472 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583400b8a0 con 0x7f58400cf750 2024-03-18T18:50:05.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.235+0000 7f5846031700 1 -- 172.21.15.1:0/4080445472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5840151ba0 con 0x7f58400cf750 2024-03-18T18:50:05.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.236+0000 7f5846031700 1 -- 172.21.15.1:0/4080445472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5840068240 con 0x7f58400cf750 2024-03-18T18:50:05.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.237+0000 7f5827fff700 1 -- 172.21.15.1:0/4080445472 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5834021030 con 0x7f58400cf750 2024-03-18T18:50:05.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.237+0000 7f5827fff700 1 --2- 172.21.15.1:0/4080445472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f582c06ac90 0x7f582c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:05.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.237+0000 7f5827fff700 1 -- 172.21.15.1:0/4080445472 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f583408ac00 con 0x7f58400cf750 2024-03-18T18:50:05.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.237+0000 7f583f7fe700 1 --2- 172.21.15.1:0/4080445472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f582c06ac90 0x7f582c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:05.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.237+0000 7f583f7fe700 1 --2- 172.21.15.1:0/4080445472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f582c06ac90 0x7f582c06d150 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f583000a9b0 tx=0x7f5830008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:05.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.242+0000 7f5827fff700 1 -- 172.21.15.1:0/4080445472 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5834091080 con 0x7f58400cf750 2024-03-18T18:50:05.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.443+0000 7f5846031700 1 -- 172.21.15.1:0/4080445472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5840068240 con 0x7f58400cf750 2024-03-18T18:50:05.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.444+0000 7f5827fff700 1 -- 172.21.15.1:0/4080445472 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17830 (secure 0 0 0) 0x7f5834055ca0 con 0x7f58400cf750 2024-03-18T18:50:05.441 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:05.441 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:49.276961+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:05.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.447+0000 7f5846031700 1 -- 172.21.15.1:0/4080445472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f582c06ac90 msgr2=0x7f582c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:05.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.447+0000 7f5846031700 1 --2- 172.21.15.1:0/4080445472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f582c06ac90 0x7f582c06d150 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f583000a9b0 tx=0x7f5830008040 comp rx=0 tx=0).stop 2024-03-18T18:50:05.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.447+0000 7f5846031700 1 -- 172.21.15.1:0/4080445472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58400cf750 msgr2=0x7f58400c7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:05.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.447+0000 7f5846031700 1 --2- 172.21.15.1:0/4080445472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58400cf750 0x7f58400c7420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5840072230 tx=0x7f5834004750 comp rx=0 tx=0).stop 2024-03-18T18:50:05.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.447+0000 7f5846031700 1 -- 172.21.15.1:0/4080445472 shutdown_connections 2024-03-18T18:50:05.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.447+0000 7f5846031700 1 --2- 172.21.15.1:0/4080445472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f582c06ac90 0x7f582c06d150 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.447+0000 7f5846031700 1 --2- 172.21.15.1:0/4080445472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5840099f80 0x7f5840151660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.447+0000 7f5846031700 1 --2- 172.21.15.1:0/4080445472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f584015b4d0 0x7f584015de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.447+0000 7f5846031700 1 --2- 172.21.15.1:0/4080445472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58400cf750 0x7f58400c7420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.447+0000 7f5846031700 1 -- 172.21.15.1:0/4080445472 >> 172.21.15.1:0/4080445472 conn(0x7f58400b2cb0 msgr2=0x7f58400affc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:05.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.447+0000 7f5846031700 1 -- 172.21.15.1:0/4080445472 shutdown_connections 2024-03-18T18:50:05.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.447+0000 7f5846031700 1 -- 172.21.15.1:0/4080445472 wait complete. 2024-03-18T18:50:05.456 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:05.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.584+0000 7fa177768700 1 Processor -- start 2024-03-18T18:50:05.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.584+0000 7fa177768700 1 -- start start 2024-03-18T18:50:05.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.585+0000 7fa177768700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa17006ae60 0x7fa170164f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:05.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.585+0000 7fa177768700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa1701465a0 0x7fa170167480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:05.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.585+0000 7fa177768700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa170072660 con 0x7fa170146190 2024-03-18T18:50:05.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.585+0000 7fa177768700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa17006f850 con 0x7fa17006ae60 2024-03-18T18:50:05.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.585+0000 7fa177768700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa17006ebd0 con 0x7fa1701465a0 2024-03-18T18:50:05.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.585+0000 7fa174d03700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa17006ae60 0x7fa170164f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.585+0000 7fa174d03700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa17006ae60 0x7fa170164f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44996/0 (socket says 172.21.15.1:44996) 2024-03-18T18:50:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.585+0000 7fa174d03700 1 -- 172.21.15.1:0/4046547076 learned_addr learned my addr 172.21.15.1:0/4046547076 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.585+0000 7fa175d05700 1 --2- 172.21.15.1:0/4046547076 >> v2:172.21.15.165:3300/0 conn(0x7fa1701465a0 0x7fa170167480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.586+0000 7fa174d03700 1 -- 172.21.15.1:0/4046547076 >> v2:172.21.15.165:3300/0 conn(0x7fa1701465a0 msgr2=0x7fa170167480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.586+0000 7fa174d03700 1 --2- 172.21.15.1:0/4046547076 >> v2:172.21.15.165:3300/0 conn(0x7fa1701465a0 0x7fa170167480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.586+0000 7fa174d03700 1 -- 172.21.15.1:0/4046547076 >> v1:172.21.15.165:6789/0 conn(0x7fa170146190 legacy=0x7fa17006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.586+0000 7fa174d03700 1 -- 172.21.15.1:0/4046547076 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17006eeb0 con 0x7fa17006ae60 2024-03-18T18:50:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.586+0000 7fa175d05700 1 --2- 172.21.15.1:0/4046547076 >> v2:172.21.15.165:3300/0 conn(0x7fa1701465a0 0x7fa170167480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.586+0000 7fa174d03700 1 --2- 172.21.15.1:0/4046547076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17006ae60 0x7fa170164f30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa1700715e0 tx=0x7fa16c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43aa45b930e71e7b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.586+0000 7fa167fff700 1 -- 172.21.15.1:0/4046547076 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c00b870 con 0x7fa17006ae60 2024-03-18T18:50:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.586+0000 7fa167fff700 1 -- 172.21.15.1:0/4046547076 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16c00b9d0 con 0x7fa17006ae60 2024-03-18T18:50:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.586+0000 7fa167fff700 1 -- 172.21.15.1:0/4046547076 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c00bd60 con 0x7fa17006ae60 2024-03-18T18:50:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.586+0000 7fa177768700 1 -- 172.21.15.1:0/4046547076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17006ae60 msgr2=0x7fa170164f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.586+0000 7fa177768700 1 --2- 172.21.15.1:0/4046547076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17006ae60 0x7fa170164f30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa1700715e0 tx=0x7fa16c009bd0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.586+0000 7fa177768700 1 -- 172.21.15.1:0/4046547076 shutdown_connections 2024-03-18T18:50:05.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.586+0000 7fa177768700 1 --2- 172.21.15.1:0/4046547076 >> v2:172.21.15.165:3300/0 conn(0x7fa1701465a0 0x7fa170167480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.586+0000 7fa177768700 1 --2- 172.21.15.1:0/4046547076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17006ae60 0x7fa170164f30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.586+0000 7fa177768700 1 -- 172.21.15.1:0/4046547076 >> 172.21.15.1:0/4046547076 conn(0x7fa1700b1d20 msgr2=0x7fa1700b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:05.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.587+0000 7fa177768700 1 -- 172.21.15.1:0/4046547076 shutdown_connections 2024-03-18T18:50:05.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.587+0000 7fa177768700 1 -- 172.21.15.1:0/4046547076 wait complete. 2024-03-18T18:50:05.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.588+0000 7fa177768700 1 Processor -- start 2024-03-18T18:50:05.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.588+0000 7fa177768700 1 -- start start 2024-03-18T18:50:05.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.588+0000 7fa177768700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17006ae60 0x7fa1700adc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:05.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.588+0000 7fa177768700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1701465a0 0x7fa17015c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:05.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.588+0000 7fa177768700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1700af300 0x7fa170144870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:05.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.588+0000 7fa177768700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17015cf20 con 0x7fa17006ae60 2024-03-18T18:50:05.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.588+0000 7fa177768700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1700ae260 con 0x7fa1700af300 2024-03-18T18:50:05.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.588+0000 7fa177768700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1700ae540 con 0x7fa1701465a0 2024-03-18T18:50:05.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.589+0000 7fa175504700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17006ae60 0x7fa1700adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:05.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.589+0000 7fa175504700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17006ae60 0x7fa1700adc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45008/0 (socket says 172.21.15.1:45008) 2024-03-18T18:50:05.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.589+0000 7fa175504700 1 -- 172.21.15.1:0/3970938545 learned_addr learned my addr 172.21.15.1:0/3970938545 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:05.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.589+0000 7fa175d05700 1 --2- 172.21.15.1:0/3970938545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1700af300 0x7fa170144870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:05.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.589+0000 7fa174d03700 1 --2- 172.21.15.1:0/3970938545 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1701465a0 0x7fa17015c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:05.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.589+0000 7fa175504700 1 -- 172.21.15.1:0/3970938545 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1701465a0 msgr2=0x7fa17015c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:05.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.589+0000 7fa175504700 1 --2- 172.21.15.1:0/3970938545 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1701465a0 0x7fa17015c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.589+0000 7fa175504700 1 -- 172.21.15.1:0/3970938545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1700af300 msgr2=0x7fa170144870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:05.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.589+0000 7fa175504700 1 --2- 172.21.15.1:0/3970938545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1700af300 0x7fa170144870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.589+0000 7fa175504700 1 -- 172.21.15.1:0/3970938545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16c009550 con 0x7fa17006ae60 2024-03-18T18:50:05.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.589+0000 7fa174d03700 1 --2- 172.21.15.1:0/3970938545 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1701465a0 0x7fa17015c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:05.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.589+0000 7fa175d05700 1 --2- 172.21.15.1:0/3970938545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1700af300 0x7fa170144870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:05.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.589+0000 7fa175504700 1 --2- 172.21.15.1:0/3970938545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17006ae60 0x7fa1700adc80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa16000c4e0 tx=0x7fa16000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:05.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.589+0000 7fa165ffb700 1 -- 172.21.15.1:0/3970938545 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1600052d0 con 0x7fa17006ae60 2024-03-18T18:50:05.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.589+0000 7fa165ffb700 1 -- 172.21.15.1:0/3970938545 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa160005430 con 0x7fa17006ae60 2024-03-18T18:50:05.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.589+0000 7fa177768700 1 -- 172.21.15.1:0/3970938545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1700ae820 con 0x7fa17006ae60 2024-03-18T18:50:05.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.589+0000 7fa165ffb700 1 -- 172.21.15.1:0/3970938545 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1600038c0 con 0x7fa17006ae60 2024-03-18T18:50:05.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.589+0000 7fa177768700 1 -- 172.21.15.1:0/3970938545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1700aede0 con 0x7fa17006ae60 2024-03-18T18:50:05.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.590+0000 7fa177768700 1 -- 172.21.15.1:0/3970938545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1700676a0 con 0x7fa17006ae60 2024-03-18T18:50:05.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.591+0000 7fa165ffb700 1 -- 172.21.15.1:0/3970938545 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa160003a20 con 0x7fa17006ae60 2024-03-18T18:50:05.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.591+0000 7fa165ffb700 1 --2- 172.21.15.1:0/3970938545 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa15406ac90 0x7fa15406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:05.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.591+0000 7fa165ffb700 1 -- 172.21.15.1:0/3970938545 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa160088a60 con 0x7fa17006ae60 2024-03-18T18:50:05.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.591+0000 7fa174d03700 1 --2- 172.21.15.1:0/3970938545 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa15406ac90 0x7fa15406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:05.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.592+0000 7fa174d03700 1 --2- 172.21.15.1:0/3970938545 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa15406ac90 0x7fa15406d150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa1700cc210 tx=0x7fa16c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:05.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.596+0000 7fa165ffb700 1 -- 172.21.15.1:0/3970938545 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa1600539e0 con 0x7fa17006ae60 2024-03-18T18:50:05.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.799+0000 7fa177768700 1 -- 172.21.15.1:0/3970938545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa170146e90 con 0x7fa17006ae60 2024-03-18T18:50:05.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.800+0000 7fa165ffb700 1 -- 172.21.15.1:0/3970938545 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17830 (secure 0 0 0) 0x7fa160057000 con 0x7fa17006ae60 2024-03-18T18:50:05.797 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:05.797 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:49.276961+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:05.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.802+0000 7fa177768700 1 -- 172.21.15.1:0/3970938545 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa15406ac90 msgr2=0x7fa15406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:05.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.802+0000 7fa177768700 1 --2- 172.21.15.1:0/3970938545 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa15406ac90 0x7fa15406d150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa1700cc210 tx=0x7fa16c008030 comp rx=0 tx=0).stop 2024-03-18T18:50:05.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.802+0000 7fa177768700 1 -- 172.21.15.1:0/3970938545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17006ae60 msgr2=0x7fa1700adc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:05.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.803+0000 7fa177768700 1 --2- 172.21.15.1:0/3970938545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17006ae60 0x7fa1700adc80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa16000c4e0 tx=0x7fa16000c8a0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.803+0000 7fa177768700 1 -- 172.21.15.1:0/3970938545 shutdown_connections 2024-03-18T18:50:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.803+0000 7fa177768700 1 --2- 172.21.15.1:0/3970938545 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa15406ac90 0x7fa15406d150 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.803+0000 7fa177768700 1 --2- 172.21.15.1:0/3970938545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1700af300 0x7fa170144870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.803+0000 7fa177768700 1 --2- 172.21.15.1:0/3970938545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa17006ae60 0x7fa1700adc80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.803+0000 7fa177768700 1 --2- 172.21.15.1:0/3970938545 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1701465a0 0x7fa17015c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.803+0000 7fa177768700 1 -- 172.21.15.1:0/3970938545 >> 172.21.15.1:0/3970938545 conn(0x7fa1700b1d20 msgr2=0x7fa17013ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.803+0000 7fa177768700 1 -- 172.21.15.1:0/3970938545 shutdown_connections 2024-03-18T18:50:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.803+0000 7fa177768700 1 -- 172.21.15.1:0/3970938545 wait complete. 2024-03-18T18:50:05.812 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T18:50:05.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.944+0000 7f254fb24700 1 Processor -- start 2024-03-18T18:50:05.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.944+0000 7f254fb24700 1 -- start start 2024-03-18T18:50:05.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.945+0000 7f254fb24700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f254815a1d0 0x7f25480add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:05.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.945+0000 7f254fb24700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f25480cc0f0 0x7f2548071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:05.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.945+0000 7f254fb24700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25480ce8f0 con 0x7f2548071620 2024-03-18T18:50:05.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.945+0000 7f254fb24700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25480cdb00 con 0x7f254815a1d0 2024-03-18T18:50:05.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.945+0000 7f254fb24700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f254815a5b0 con 0x7f25480cc0f0 2024-03-18T18:50:05.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.945+0000 7f254d8c0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f254815a1d0 0x7f25480add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:05.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.945+0000 7f254d8c0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f254815a1d0 0x7f25480add20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45016/0 (socket says 172.21.15.1:45016) 2024-03-18T18:50:05.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.945+0000 7f254d8c0700 1 -- 172.21.15.1:0/1419364233 learned_addr learned my addr 172.21.15.1:0/1419364233 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:05.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.945+0000 7f254e0c1700 1 --2- 172.21.15.1:0/1419364233 >> v2:172.21.15.165:3300/0 conn(0x7f25480cc0f0 0x7f2548071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:05.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.945+0000 7f254d8c0700 1 -- 172.21.15.1:0/1419364233 >> v2:172.21.15.165:3300/0 conn(0x7f25480cc0f0 msgr2=0x7f2548071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:05.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.945+0000 7f254d8c0700 1 --2- 172.21.15.1:0/1419364233 >> v2:172.21.15.165:3300/0 conn(0x7f25480cc0f0 0x7f2548071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.945+0000 7f254d8c0700 1 -- 172.21.15.1:0/1419364233 >> v1:172.21.15.165:6789/0 conn(0x7f2548071620 legacy=0x7f25480cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:05.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.946+0000 7f254d8c0700 1 -- 172.21.15.1:0/1419364233 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2548071fc0 con 0x7f254815a1d0 2024-03-18T18:50:05.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.946+0000 7f254e0c1700 1 --2- 172.21.15.1:0/1419364233 >> v2:172.21.15.165:3300/0 conn(0x7f25480cc0f0 0x7f2548071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:05.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.946+0000 7f254d8c0700 1 --2- 172.21.15.1:0/1419364233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f254815a1d0 0x7f25480add20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2548072200 tx=0x7f2538009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d84f2ec70bf9a92 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:05.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.946+0000 7f254c8be700 1 -- 172.21.15.1:0/1419364233 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538005e30 con 0x7f254815a1d0 2024-03-18T18:50:05.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.946+0000 7f254c8be700 1 -- 172.21.15.1:0/1419364233 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2538006030 con 0x7f254815a1d0 2024-03-18T18:50:05.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.946+0000 7f254c8be700 1 -- 172.21.15.1:0/1419364233 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25380063e0 con 0x7f254815a1d0 2024-03-18T18:50:05.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.946+0000 7f254fb24700 1 -- 172.21.15.1:0/1419364233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f254815a1d0 msgr2=0x7f25480add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:05.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.946+0000 7f254fb24700 1 --2- 172.21.15.1:0/1419364233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f254815a1d0 0x7f25480add20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2548072200 tx=0x7f2538009bd0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.946+0000 7f254fb24700 1 -- 172.21.15.1:0/1419364233 shutdown_connections 2024-03-18T18:50:05.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.946+0000 7f254fb24700 1 --2- 172.21.15.1:0/1419364233 >> v2:172.21.15.165:3300/0 conn(0x7f25480cc0f0 0x7f2548071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.946+0000 7f254fb24700 1 --2- 172.21.15.1:0/1419364233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f254815a1d0 0x7f25480add20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.946+0000 7f254fb24700 1 -- 172.21.15.1:0/1419364233 >> 172.21.15.1:0/1419364233 conn(0x7f25480cc9b0 msgr2=0x7f25480ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:05.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.946+0000 7f254fb24700 1 -- 172.21.15.1:0/1419364233 shutdown_connections 2024-03-18T18:50:05.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.947+0000 7f254fb24700 1 -- 172.21.15.1:0/1419364233 wait complete. 2024-03-18T18:50:05.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.948+0000 7f254fb24700 1 Processor -- start 2024-03-18T18:50:05.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.948+0000 7f254fb24700 1 -- start start 2024-03-18T18:50:05.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.948+0000 7f254fb24700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2548071620 0x7f254806eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:05.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.948+0000 7f254fb24700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25480cc0f0 0x7f254815cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:05.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.948+0000 7f254fb24700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f254815a1d0 0x7f25481f6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:05.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.948+0000 7f254fb24700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254806f030 con 0x7f25480cc0f0 2024-03-18T18:50:05.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.948+0000 7f254fb24700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254815bfc0 con 0x7f2548071620 2024-03-18T18:50:05.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.948+0000 7f254fb24700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25480c96a0 con 0x7f254815a1d0 2024-03-18T18:50:05.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.949+0000 7f254d0bf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25480cc0f0 0x7f254815cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:05.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.949+0000 7f254d0bf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25480cc0f0 0x7f254815cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45030/0 (socket says 172.21.15.1:45030) 2024-03-18T18:50:05.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.949+0000 7f254d0bf700 1 -- 172.21.15.1:0/3887041190 learned_addr learned my addr 172.21.15.1:0/3887041190 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:05.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.949+0000 7f254d8c0700 1 --2- 172.21.15.1:0/3887041190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2548071620 0x7f254806eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:05.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.949+0000 7f254e0c1700 1 --2- 172.21.15.1:0/3887041190 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f254815a1d0 0x7f25481f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:05.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.949+0000 7f254d0bf700 1 -- 172.21.15.1:0/3887041190 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f254815a1d0 msgr2=0x7f25481f6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:05.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.949+0000 7f254d0bf700 1 --2- 172.21.15.1:0/3887041190 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f254815a1d0 0x7f25481f6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.949+0000 7f254d0bf700 1 -- 172.21.15.1:0/3887041190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2548071620 msgr2=0x7f254806eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:05.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.949+0000 7f254d0bf700 1 --2- 172.21.15.1:0/3887041190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2548071620 0x7f254806eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.949+0000 7f254d0bf700 1 -- 172.21.15.1:0/3887041190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2538009500 con 0x7f25480cc0f0 2024-03-18T18:50:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.949+0000 7f254e0c1700 1 --2- 172.21.15.1:0/3887041190 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f254815a1d0 0x7f25481f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.949+0000 7f254d0bf700 1 --2- 172.21.15.1:0/3887041190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25480cc0f0 0x7f254815cd40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f254400b4a0 tx=0x7f254400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.949+0000 7f253e7fc700 1 -- 172.21.15.1:0/3887041190 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2544011830 con 0x7f25480cc0f0 2024-03-18T18:50:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.949+0000 7f253e7fc700 1 -- 172.21.15.1:0/3887041190 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2544011990 con 0x7f25480cc0f0 2024-03-18T18:50:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.949+0000 7f254fb24700 1 -- 172.21.15.1:0/3887041190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25480c8d70 con 0x7f25480cc0f0 2024-03-18T18:50:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.949+0000 7f253e7fc700 1 -- 172.21.15.1:0/3887041190 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2544011d40 con 0x7f25480cc0f0 2024-03-18T18:50:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.949+0000 7f254fb24700 1 -- 172.21.15.1:0/3887041190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25480c9330 con 0x7f25480cc0f0 2024-03-18T18:50:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.950+0000 7f254fb24700 1 -- 172.21.15.1:0/3887041190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25480676a0 con 0x7f25480cc0f0 2024-03-18T18:50:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.951+0000 7f253e7fc700 1 -- 172.21.15.1:0/3887041190 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f254401a400 con 0x7f25480cc0f0 2024-03-18T18:50:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.951+0000 7f253e7fc700 1 --2- 172.21.15.1:0/3887041190 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f252c06ac90 0x7f252c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.951+0000 7f254d8c0700 1 --2- 172.21.15.1:0/3887041190 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f252c06ac90 0x7f252c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.951+0000 7f253e7fc700 1 -- 172.21.15.1:0/3887041190 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2544089300 con 0x7f25480cc0f0 2024-03-18T18:50:05.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.952+0000 7f254d8c0700 1 --2- 172.21.15.1:0/3887041190 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f252c06ac90 0x7f252c06d150 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2548072200 tx=0x7f2538008070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:05.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:05.955+0000 7f253e7fc700 1 -- 172.21.15.1:0/3887041190 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f25440541f0 con 0x7f25480cc0f0 2024-03-18T18:50:06.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:06.203+0000 7f254fb24700 1 -- 172.21.15.1:0/3887041190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f25480cb590 con 0x7f25480cc0f0 2024-03-18T18:50:06.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:06.887+0000 7f253e7fc700 1 -- 172.21.15.1:0/3887041190 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2544057810 con 0x7f25480cc0f0 2024-03-18T18:50:06.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:06.890+0000 7f254fb24700 1 -- 172.21.15.1:0/3887041190 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f252c06ac90 msgr2=0x7f252c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:06.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:06.890+0000 7f254fb24700 1 --2- 172.21.15.1:0/3887041190 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f252c06ac90 0x7f252c06d150 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2548072200 tx=0x7f2538008070 comp rx=0 tx=0).stop 2024-03-18T18:50:06.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:06.890+0000 7f254fb24700 1 -- 172.21.15.1:0/3887041190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25480cc0f0 msgr2=0x7f254815cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:06.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:06.890+0000 7f254fb24700 1 --2- 172.21.15.1:0/3887041190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25480cc0f0 0x7f254815cd40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f254400b4a0 tx=0x7f254400b860 comp rx=0 tx=0).stop 2024-03-18T18:50:06.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:06.890+0000 7f254fb24700 1 -- 172.21.15.1:0/3887041190 shutdown_connections 2024-03-18T18:50:06.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:06.890+0000 7f254fb24700 1 --2- 172.21.15.1:0/3887041190 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f252c06ac90 0x7f252c06d150 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:06.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:06.890+0000 7f254fb24700 1 --2- 172.21.15.1:0/3887041190 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f254815a1d0 0x7f25481f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:06.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:06.890+0000 7f254fb24700 1 --2- 172.21.15.1:0/3887041190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2548071620 0x7f254806eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:06.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:06.890+0000 7f254fb24700 1 --2- 172.21.15.1:0/3887041190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25480cc0f0 0x7f254815cd40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:06.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:06.890+0000 7f254fb24700 1 -- 172.21.15.1:0/3887041190 >> 172.21.15.1:0/3887041190 conn(0x7f25480cc9b0 msgr2=0x7f254806c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:06.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:06.890+0000 7f254fb24700 1 -- 172.21.15.1:0/3887041190 shutdown_connections 2024-03-18T18:50:06.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:06.890+0000 7f254fb24700 1 -- 172.21.15.1:0/3887041190 wait complete. 2024-03-18T18:50:06.902 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-03-18T18:50:06.902 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T18:50:07.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.040+0000 7fb957cf9700 1 Processor -- start 2024-03-18T18:50:07.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.041+0000 7fb957cf9700 1 -- start start 2024-03-18T18:50:07.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.041+0000 7fb957cf9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb9500cf9d0 0x7fb9500717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:07.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.041+0000 7fb957cf9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9500cfdb0 0x7fb950141010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:07.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.041+0000 7fb957cf9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb950071d20 con 0x7fb950144500 2024-03-18T18:50:07.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.041+0000 7fb957cf9700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb95015ed40 con 0x7fb9500cf9d0 2024-03-18T18:50:07.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.041+0000 7fb957cf9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95015ef90 con 0x7fb9500cfdb0 2024-03-18T18:50:07.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.042+0000 7fb955294700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb950144500 0x7fb9500ce4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60182/0 (socket says 172.21.15.1:60182) 2024-03-18T18:50:07.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.042+0000 7fb955294700 1 -- 172.21.15.1:0/950593577 learned_addr learned my addr 172.21.15.1:0/950593577 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:07.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.042+0000 7fb955a95700 1 --2- 172.21.15.1:0/950593577 >> v2:172.21.15.165:3300/0 conn(0x7fb9500cf9d0 0x7fb9500717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:07.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.042+0000 7fb956296700 1 --2- 172.21.15.1:0/950593577 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9500cfdb0 0x7fb950141010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:07.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.042+0000 7fb954a93700 1 -- 172.21.15.1:0/950593577 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294647361 0 0) 0x7fb94c001110 con 0x7fb950144500 2024-03-18T18:50:07.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.042+0000 7fb954a93700 1 -- 172.21.15.1:0/950593577 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb93c0035b0 con 0x7fb950144500 2024-03-18T18:50:07.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.042+0000 7fb955a95700 1 -- 172.21.15.1:0/950593577 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9500cfdb0 msgr2=0x7fb950141010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:07.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.042+0000 7fb955a95700 1 --2- 172.21.15.1:0/950593577 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9500cfdb0 0x7fb950141010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:07.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.042+0000 7fb955a95700 1 -- 172.21.15.1:0/950593577 >> v1:172.21.15.1:6789/0 conn(0x7fb950144500 legacy=0x7fb9500ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:07.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.042+0000 7fb955a95700 1 -- 172.21.15.1:0/950593577 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95015f270 con 0x7fb9500cf9d0 2024-03-18T18:50:07.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.042+0000 7fb955a95700 1 --2- 172.21.15.1:0/950593577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9500cf9d0 0x7fb9500717e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb940009870 tx=0x7fb940009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bbc946d64784470 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:07.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.042+0000 7fb954a93700 1 -- 172.21.15.1:0/950593577 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb940011830 con 0x7fb9500cf9d0 2024-03-18T18:50:07.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.042+0000 7fb954a93700 1 -- 172.21.15.1:0/950593577 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb940011990 con 0x7fb9500cf9d0 2024-03-18T18:50:07.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.042+0000 7fb954a93700 1 -- 172.21.15.1:0/950593577 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb940011d20 con 0x7fb9500cf9d0 2024-03-18T18:50:07.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.043+0000 7fb957cf9700 1 -- 172.21.15.1:0/950593577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9500cf9d0 msgr2=0x7fb9500717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:07.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.043+0000 7fb957cf9700 1 --2- 172.21.15.1:0/950593577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9500cf9d0 0x7fb9500717e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb940009870 tx=0x7fb940009b80 comp rx=0 tx=0).stop 2024-03-18T18:50:07.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.043+0000 7fb957cf9700 1 -- 172.21.15.1:0/950593577 shutdown_connections 2024-03-18T18:50:07.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.043+0000 7fb957cf9700 1 --2- 172.21.15.1:0/950593577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9500cf9d0 0x7fb9500717e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:07.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.043+0000 7fb957cf9700 1 --2- 172.21.15.1:0/950593577 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9500cfdb0 0x7fb950141010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:07.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.043+0000 7fb957cf9700 1 -- 172.21.15.1:0/950593577 >> 172.21.15.1:0/950593577 conn(0x7fb950066b70 msgr2=0x7fb95006a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:07.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.043+0000 7fb957cf9700 1 -- 172.21.15.1:0/950593577 shutdown_connections 2024-03-18T18:50:07.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.043+0000 7fb957cf9700 1 -- 172.21.15.1:0/950593577 wait complete. 2024-03-18T18:50:07.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.044+0000 7fb957cf9700 1 Processor -- start 2024-03-18T18:50:07.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.044+0000 7fb957cf9700 1 -- start start 2024-03-18T18:50:07.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.045+0000 7fb957cf9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9500cf9d0 0x7fb95014e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:07.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.045+0000 7fb957cf9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9500cfdb0 0x7fb9501f6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:07.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.045+0000 7fb957cf9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb950144500 0x7fb9501f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:07.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.045+0000 7fb957cf9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9500ae150 con 0x7fb9500cfdb0 2024-03-18T18:50:07.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.045+0000 7fb957cf9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9500ad2c0 con 0x7fb9500cf9d0 2024-03-18T18:50:07.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.045+0000 7fb957cf9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9500ad4d0 con 0x7fb950144500 2024-03-18T18:50:07.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.045+0000 7fb955294700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9500cfdb0 0x7fb9501f6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:07.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.045+0000 7fb955294700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9500cfdb0 0x7fb9501f6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45032/0 (socket says 172.21.15.1:45032) 2024-03-18T18:50:07.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.045+0000 7fb955294700 1 -- 172.21.15.1:0/2631024441 learned_addr learned my addr 172.21.15.1:0/2631024441 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:07.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.045+0000 7fb955a95700 1 --2- 172.21.15.1:0/2631024441 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9500cf9d0 0x7fb95014e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:07.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.045+0000 7fb956296700 1 --2- 172.21.15.1:0/2631024441 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb950144500 0x7fb9501f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:07.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.046+0000 7fb955294700 1 -- 172.21.15.1:0/2631024441 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb950144500 msgr2=0x7fb9501f6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:07.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.046+0000 7fb955294700 1 --2- 172.21.15.1:0/2631024441 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb950144500 0x7fb9501f6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:07.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.046+0000 7fb955294700 1 -- 172.21.15.1:0/2631024441 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9500cf9d0 msgr2=0x7fb95014e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:07.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.046+0000 7fb955294700 1 --2- 172.21.15.1:0/2631024441 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9500cf9d0 0x7fb95014e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:07.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.046+0000 7fb955294700 1 -- 172.21.15.1:0/2631024441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9400094a0 con 0x7fb9500cfdb0 2024-03-18T18:50:07.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.046+0000 7fb955294700 1 --2- 172.21.15.1:0/2631024441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9500cfdb0 0x7fb9501f6780 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb94c00c500 tx=0x7fb94c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:07.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.046+0000 7fb9467fc700 1 -- 172.21.15.1:0/2631024441 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c005d70 con 0x7fb9500cfdb0 2024-03-18T18:50:07.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.046+0000 7fb9467fc700 1 -- 172.21.15.1:0/2631024441 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94c0070d0 con 0x7fb9500cfdb0 2024-03-18T18:50:07.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.046+0000 7fb957cf9700 1 -- 172.21.15.1:0/2631024441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9500ad7b0 con 0x7fb9500cfdb0 2024-03-18T18:50:07.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.046+0000 7fb9467fc700 1 -- 172.21.15.1:0/2631024441 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c007460 con 0x7fb9500cfdb0 2024-03-18T18:50:07.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.046+0000 7fb957cf9700 1 -- 172.21.15.1:0/2631024441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9500add70 con 0x7fb9500cfdb0 2024-03-18T18:50:07.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.047+0000 7fb957cf9700 1 -- 172.21.15.1:0/2631024441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb950073350 con 0x7fb9500cfdb0 2024-03-18T18:50:07.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.047+0000 7fb9467fc700 1 -- 172.21.15.1:0/2631024441 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb94c0075c0 con 0x7fb9500cfdb0 2024-03-18T18:50:07.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.048+0000 7fb9467fc700 1 --2- 172.21.15.1:0/2631024441 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb93406ac90 0x7fb93406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:07.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.048+0000 7fb9467fc700 1 -- 172.21.15.1:0/2631024441 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb94c0887e0 con 0x7fb9500cfdb0 2024-03-18T18:50:07.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.048+0000 7fb955a95700 1 --2- 172.21.15.1:0/2631024441 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb93406ac90 0x7fb93406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:07.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.048+0000 7fb955a95700 1 --2- 172.21.15.1:0/2631024441 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb93406ac90 0x7fb93406d150 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb9400090b0 tx=0x7fb940008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:07.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.053+0000 7fb9467fc700 1 -- 172.21.15.1:0/2631024441 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb94c053760 con 0x7fb9500cfdb0 2024-03-18T18:50:07.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.299+0000 7fb957cf9700 1 -- 172.21.15.1:0/2631024441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb950071d20 con 0x7fb9500cfdb0 2024-03-18T18:50:07.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.894+0000 7fb9467fc700 1 -- 172.21.15.1:0/2631024441 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb950071d20 con 0x7fb9500cfdb0 2024-03-18T18:50:07.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.897+0000 7fb957cf9700 1 -- 172.21.15.1:0/2631024441 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb93406ac90 msgr2=0x7fb93406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:07.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.897+0000 7fb957cf9700 1 --2- 172.21.15.1:0/2631024441 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb93406ac90 0x7fb93406d150 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb9400090b0 tx=0x7fb940008030 comp rx=0 tx=0).stop 2024-03-18T18:50:07.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.897+0000 7fb957cf9700 1 -- 172.21.15.1:0/2631024441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9500cfdb0 msgr2=0x7fb9501f6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:07.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.897+0000 7fb957cf9700 1 --2- 172.21.15.1:0/2631024441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9500cfdb0 0x7fb9501f6780 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb94c00c500 tx=0x7fb94c00c810 comp rx=0 tx=0).stop 2024-03-18T18:50:07.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.897+0000 7fb957cf9700 1 -- 172.21.15.1:0/2631024441 shutdown_connections 2024-03-18T18:50:07.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.897+0000 7fb957cf9700 1 --2- 172.21.15.1:0/2631024441 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb93406ac90 0x7fb93406d150 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:07.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.897+0000 7fb957cf9700 1 --2- 172.21.15.1:0/2631024441 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb950144500 0x7fb9501f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:07.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.897+0000 7fb957cf9700 1 --2- 172.21.15.1:0/2631024441 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9500cf9d0 0x7fb95014e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:07.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.897+0000 7fb957cf9700 1 --2- 172.21.15.1:0/2631024441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9500cfdb0 0x7fb9501f6780 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:07.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.897+0000 7fb957cf9700 1 -- 172.21.15.1:0/2631024441 >> 172.21.15.1:0/2631024441 conn(0x7fb950066b70 msgr2=0x7fb95006ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:07.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.898+0000 7fb957cf9700 1 -- 172.21.15.1:0/2631024441 shutdown_connections 2024-03-18T18:50:07.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:07.898+0000 7fb957cf9700 1 -- 172.21.15.1:0/2631024441 wait complete. 2024-03-18T18:50:07.909 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T18:50:07.909 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T18:50:08.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.044+0000 7efdf21c8700 1 Processor -- start 2024-03-18T18:50:08.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.044+0000 7efdf21c8700 1 -- start start 2024-03-18T18:50:08.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.044+0000 7efdf21c8700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7efdec06ae60 0x7efdec166270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:08.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.044+0000 7efdf21c8700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdec072720 con 0x7efdec15b500 2024-03-18T18:50:08.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.044+0000 7efdf21c8700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdec06ffc0 con 0x7efdec15b120 2024-03-18T18:50:08.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.044+0000 7efdf21c8700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdec06b520 con 0x7efdec06ae60 2024-03-18T18:50:08.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.045+0000 7efdeaffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efdec15b500 0x7efdec06a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60192/0 (socket says 172.21.15.1:60192) 2024-03-18T18:50:08.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.045+0000 7efdeaffd700 1 -- 172.21.15.1:0/938792635 learned_addr learned my addr 172.21.15.1:0/938792635 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:08.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.045+0000 7efdebfff700 1 --2- 172.21.15.1:0/938792635 >> v2:172.21.15.165:3300/0 conn(0x7efdec06ae60 0x7efdec166270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:08.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.045+0000 7efdea7fc700 1 -- 172.21.15.1:0/938792635 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101737721 0 0) 0x7efde0001110 con 0x7efdec15b500 2024-03-18T18:50:08.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.045+0000 7efdea7fc700 1 -- 172.21.15.1:0/938792635 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdd40035b0 con 0x7efdec15b500 2024-03-18T18:50:08.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.045+0000 7efdebfff700 1 -- 172.21.15.1:0/938792635 >> v1:172.21.15.165:6789/0 conn(0x7efdec15b120 legacy=0x7efdec072010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:08.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.045+0000 7efdebfff700 1 -- 172.21.15.1:0/938792635 >> v1:172.21.15.1:6789/0 conn(0x7efdec15b500 legacy=0x7efdec06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:08.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.045+0000 7efdebfff700 1 -- 172.21.15.1:0/938792635 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdec06ef20 con 0x7efdec06ae60 2024-03-18T18:50:08.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.045+0000 7efdebfff700 1 --2- 172.21.15.1:0/938792635 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdec06ae60 0x7efdec166270 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efde4009840 tx=0x7efde4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb15bb2b93132059 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:08.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.045+0000 7efdea7fc700 1 -- 172.21.15.1:0/938792635 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde4011830 con 0x7efdec06ae60 2024-03-18T18:50:08.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.045+0000 7efdea7fc700 1 -- 172.21.15.1:0/938792635 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde4011990 con 0x7efdec06ae60 2024-03-18T18:50:08.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.045+0000 7efdea7fc700 1 -- 172.21.15.1:0/938792635 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde4011d20 con 0x7efdec06ae60 2024-03-18T18:50:08.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.046+0000 7efdf21c8700 1 -- 172.21.15.1:0/938792635 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdec06ae60 msgr2=0x7efdec166270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:08.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.046+0000 7efdf21c8700 1 --2- 172.21.15.1:0/938792635 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdec06ae60 0x7efdec166270 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efde4009840 tx=0x7efde4009b50 comp rx=0 tx=0).stop 2024-03-18T18:50:08.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.046+0000 7efdf21c8700 1 -- 172.21.15.1:0/938792635 shutdown_connections 2024-03-18T18:50:08.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.046+0000 7efdf21c8700 1 --2- 172.21.15.1:0/938792635 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdec06ae60 0x7efdec166270 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.046+0000 7efdf21c8700 1 -- 172.21.15.1:0/938792635 >> 172.21.15.1:0/938792635 conn(0x7efdec0ccab0 msgr2=0x7efdec0ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:08.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.046+0000 7efdf21c8700 1 -- 172.21.15.1:0/938792635 shutdown_connections 2024-03-18T18:50:08.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.046+0000 7efdf21c8700 1 -- 172.21.15.1:0/938792635 wait complete. 2024-03-18T18:50:08.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.047+0000 7efdf21c8700 1 Processor -- start 2024-03-18T18:50:08.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.047+0000 7efdf21c8700 1 -- start start 2024-03-18T18:50:08.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.048+0000 7efdf21c8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdec06ae60 0x7efdec0cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:08.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.048+0000 7efdf21c8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdec15b120 0x7efdec14cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:08.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.048+0000 7efdf21c8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdec15b500 0x7efdec0cb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:08.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.048+0000 7efdf21c8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec1f3530 con 0x7efdec15b120 2024-03-18T18:50:08.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.048+0000 7efdf21c8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec14d2d0 con 0x7efdec15b500 2024-03-18T18:50:08.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.048+0000 7efdf21c8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdec0c4990 con 0x7efdec06ae60 2024-03-18T18:50:08.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.048+0000 7efdeaffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdec15b120 0x7efdec14cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:08.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.048+0000 7efdeaffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdec15b120 0x7efdec14cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45040/0 (socket says 172.21.15.1:45040) 2024-03-18T18:50:08.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.048+0000 7efdeaffd700 1 -- 172.21.15.1:0/1857626435 learned_addr learned my addr 172.21.15.1:0/1857626435 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:08.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.048+0000 7efdebfff700 1 --2- 172.21.15.1:0/1857626435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdec15b500 0x7efdec0cb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:08.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.048+0000 7efdeb7fe700 1 --2- 172.21.15.1:0/1857626435 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdec06ae60 0x7efdec0cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:08.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.049+0000 7efdeaffd700 1 -- 172.21.15.1:0/1857626435 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdec06ae60 msgr2=0x7efdec0cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:08.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.049+0000 7efdeaffd700 1 --2- 172.21.15.1:0/1857626435 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdec06ae60 0x7efdec0cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.049+0000 7efdeaffd700 1 -- 172.21.15.1:0/1857626435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdec15b500 msgr2=0x7efdec0cb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:08.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.049+0000 7efdeaffd700 1 --2- 172.21.15.1:0/1857626435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdec15b500 0x7efdec0cb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.049+0000 7efdeaffd700 1 -- 172.21.15.1:0/1857626435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde40094a0 con 0x7efdec15b120 2024-03-18T18:50:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.049+0000 7efdeb7fe700 1 --2- 172.21.15.1:0/1857626435 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdec06ae60 0x7efdec0cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.049+0000 7efdeaffd700 1 --2- 172.21.15.1:0/1857626435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdec15b120 0x7efdec14cd90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efde000c4d0 tx=0x7efde000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.049+0000 7efdd3fff700 1 -- 172.21.15.1:0/1857626435 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde0005d70 con 0x7efdec15b120 2024-03-18T18:50:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.049+0000 7efdd3fff700 1 -- 172.21.15.1:0/1857626435 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde00070d0 con 0x7efdec15b120 2024-03-18T18:50:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.049+0000 7efdd3fff700 1 -- 172.21.15.1:0/1857626435 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde0007460 con 0x7efdec15b120 2024-03-18T18:50:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.049+0000 7efdf21c8700 1 -- 172.21.15.1:0/1857626435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdec0c4c70 con 0x7efdec15b120 2024-03-18T18:50:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.049+0000 7efdf21c8700 1 -- 172.21.15.1:0/1857626435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdec0c5230 con 0x7efdec15b120 2024-03-18T18:50:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.050+0000 7efdf21c8700 1 -- 172.21.15.1:0/1857626435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdec06eb80 con 0x7efdec15b120 2024-03-18T18:50:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.050+0000 7efdd3fff700 1 -- 172.21.15.1:0/1857626435 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7efde00075c0 con 0x7efdec15b120 2024-03-18T18:50:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.051+0000 7efdd3fff700 1 --2- 172.21.15.1:0/1857626435 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efdd806ac90 0x7efdd806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.051+0000 7efdd3fff700 1 -- 172.21.15.1:0/1857626435 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7efde00888d0 con 0x7efdec15b120 2024-03-18T18:50:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.051+0000 7efdeb7fe700 1 --2- 172.21.15.1:0/1857626435 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efdd806ac90 0x7efdd806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:08.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.051+0000 7efdeb7fe700 1 --2- 172.21.15.1:0/1857626435 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efdd806ac90 0x7efdd806d150 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efddc0012e0 tx=0x7efddc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:08.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.055+0000 7efdd3fff700 1 -- 172.21.15.1:0/1857626435 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efde0053850 con 0x7efdec15b120 2024-03-18T18:50:08.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.303+0000 7efdf21c8700 1 -- 172.21.15.1:0/1857626435 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdec072720 con 0x7efdec15b120 2024-03-18T18:50:08.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.304+0000 7efdd3fff700 1 -- 172.21.15.1:0/1857626435 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4727 (secure 0 0 0) 0x7efdec072720 con 0x7efdec15b120 2024-03-18T18:50:08.300 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:08.300 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/883056851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":883056851},{"type":"v1","addr":"172.21.15.165:6834","nonce":883056851}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1388214718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1388214718},{"type":"v1","addr":"172.21.15.165:6835","nonce":1388214718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3020786191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3020786191},{"type":"v1","addr":"172.21.15.1:6836","nonce":3020786191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T18:49:44.809495+0000","modified":"2024-03-18T18:50:07.888953+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2118908012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2118908012},{"type":"v1","addr":"172.21.15.1:6837","nonce":2118908012}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-18T18:50:08.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.306+0000 7efdf21c8700 1 -- 172.21.15.1:0/1857626435 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efdd806ac90 msgr2=0x7efdd806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:08.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.306+0000 7efdf21c8700 1 --2- 172.21.15.1:0/1857626435 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efdd806ac90 0x7efdd806d150 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efddc0012e0 tx=0x7efddc009040 comp rx=0 tx=0).stop 2024-03-18T18:50:08.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.306+0000 7efdf21c8700 1 -- 172.21.15.1:0/1857626435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdec15b120 msgr2=0x7efdec14cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:08.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.306+0000 7efdf21c8700 1 --2- 172.21.15.1:0/1857626435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdec15b120 0x7efdec14cd90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efde000c4d0 tx=0x7efde000c7e0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.306+0000 7efdf21c8700 1 -- 172.21.15.1:0/1857626435 shutdown_connections 2024-03-18T18:50:08.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.306+0000 7efdf21c8700 1 --2- 172.21.15.1:0/1857626435 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efdd806ac90 0x7efdd806d150 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.306+0000 7efdf21c8700 1 --2- 172.21.15.1:0/1857626435 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdec06ae60 0x7efdec0cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.306+0000 7efdf21c8700 1 --2- 172.21.15.1:0/1857626435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdec15b500 0x7efdec0cb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.306+0000 7efdf21c8700 1 --2- 172.21.15.1:0/1857626435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdec15b120 0x7efdec14cd90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.306+0000 7efdf21c8700 1 -- 172.21.15.1:0/1857626435 >> 172.21.15.1:0/1857626435 conn(0x7efdec0ccab0 msgr2=0x7efdec159e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:08.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.306+0000 7efdf21c8700 1 -- 172.21.15.1:0/1857626435 shutdown_connections 2024-03-18T18:50:08.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.306+0000 7efdf21c8700 1 -- 172.21.15.1:0/1857626435 wait complete. 2024-03-18T18:50:08.304 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 9 2024-03-18T18:50:08.315 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:08.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.444+0000 7f3dd70bd700 1 Processor -- start 2024-03-18T18:50:08.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.444+0000 7f3dd70bd700 1 -- start start 2024-03-18T18:50:08.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.445+0000 7f3dd70bd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3dd006a750 0x7f3dd006ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:08.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.445+0000 7f3dd70bd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3dd015d6b0 0x7f3dd0160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:08.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.445+0000 7f3dd70bd700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd00acfb0 con 0x7f3dd00ad790 2024-03-18T18:50:08.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.445+0000 7f3dd70bd700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dd00af7a0 con 0x7f3dd006a750 2024-03-18T18:50:08.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.445+0000 7f3dd70bd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd006b070 con 0x7f3dd015d6b0 2024-03-18T18:50:08.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.445+0000 7f3dd4e59700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3dd006a750 0x7f3dd006ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:08.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.445+0000 7f3dd4e59700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3dd006a750 0x7f3dd006ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45052/0 (socket says 172.21.15.1:45052) 2024-03-18T18:50:08.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.445+0000 7f3dd4e59700 1 -- 172.21.15.1:0/3799060275 learned_addr learned my addr 172.21.15.1:0/3799060275 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:08.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.446+0000 7f3dcffff700 1 --2- 172.21.15.1:0/3799060275 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3dd015d6b0 0x7f3dd0160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:08.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.446+0000 7f3dd4e59700 1 -- 172.21.15.1:0/3799060275 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3dd015d6b0 msgr2=0x7f3dd0160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:08.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.446+0000 7f3dd4e59700 1 --2- 172.21.15.1:0/3799060275 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3dd015d6b0 0x7f3dd0160ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.446+0000 7f3dd4e59700 1 -- 172.21.15.1:0/3799060275 >> v1:172.21.15.165:6789/0 conn(0x7f3dd00ad790 legacy=0x7f3dd0161530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:08.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.446+0000 7f3dd4e59700 1 -- 172.21.15.1:0/3799060275 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd00b50a0 con 0x7f3dd006a750 2024-03-18T18:50:08.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.446+0000 7f3dcffff700 1 --2- 172.21.15.1:0/3799060275 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3dd015d6b0 0x7f3dd0160ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:08.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.446+0000 7f3dd4e59700 1 --2- 172.21.15.1:0/3799060275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dd006a750 0x7f3dd006ab30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3dc40098f0 tx=0x7f3dc4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998d88dfcff699cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:08.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.446+0000 7f3dcf7fe700 1 -- 172.21.15.1:0/3799060275 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc400b8a0 con 0x7f3dd006a750 2024-03-18T18:50:08.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.446+0000 7f3dcf7fe700 1 -- 172.21.15.1:0/3799060275 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc400ba00 con 0x7f3dd006a750 2024-03-18T18:50:08.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.446+0000 7f3dcf7fe700 1 -- 172.21.15.1:0/3799060275 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc400bd90 con 0x7f3dd006a750 2024-03-18T18:50:08.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.447+0000 7f3dd70bd700 1 -- 172.21.15.1:0/3799060275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dd006a750 msgr2=0x7f3dd006ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:08.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.447+0000 7f3dd70bd700 1 --2- 172.21.15.1:0/3799060275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dd006a750 0x7f3dd006ab30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3dc40098f0 tx=0x7f3dc4009c00 comp rx=0 tx=0).stop 2024-03-18T18:50:08.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.447+0000 7f3dd70bd700 1 -- 172.21.15.1:0/3799060275 shutdown_connections 2024-03-18T18:50:08.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.447+0000 7f3dd70bd700 1 --2- 172.21.15.1:0/3799060275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dd006a750 0x7f3dd006ab30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.447+0000 7f3dd70bd700 1 --2- 172.21.15.1:0/3799060275 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3dd015d6b0 0x7f3dd0160ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.447+0000 7f3dd70bd700 1 -- 172.21.15.1:0/3799060275 >> 172.21.15.1:0/3799060275 conn(0x7f3dd006b860 msgr2=0x7f3dd006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:08.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.447+0000 7f3dd70bd700 1 -- 172.21.15.1:0/3799060275 shutdown_connections 2024-03-18T18:50:08.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.447+0000 7f3dd70bd700 1 -- 172.21.15.1:0/3799060275 wait complete. 2024-03-18T18:50:08.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.448+0000 7f3dd70bd700 1 Processor -- start 2024-03-18T18:50:08.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.448+0000 7f3dd70bd700 1 -- start start 2024-03-18T18:50:08.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.449+0000 7f3dd70bd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3dd006a750 0x7f3dd00ce8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:08.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.449+0000 7f3dd70bd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dd00ad790 0x7f3dd015bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:08.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.449+0000 7f3dd70bd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dd015d6b0 0x7f3dd015c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:08.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.449+0000 7f3dd70bd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd015bbf0 con 0x7f3dd00ad790 2024-03-18T18:50:08.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.449+0000 7f3dd70bd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd015b630 con 0x7f3dd015d6b0 2024-03-18T18:50:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.449+0000 7f3dd70bd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd015b910 con 0x7f3dd006a750 2024-03-18T18:50:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.449+0000 7f3dcffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dd00ad790 0x7f3dd015bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.449+0000 7f3dcffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dd00ad790 0x7f3dd015bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45060/0 (socket says 172.21.15.1:45060) 2024-03-18T18:50:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.449+0000 7f3dcffff700 1 -- 172.21.15.1:0/1756787636 learned_addr learned my addr 172.21.15.1:0/1756787636 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.449+0000 7f3dd565a700 1 --2- 172.21.15.1:0/1756787636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dd015d6b0 0x7f3dd015c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.449+0000 7f3dd4e59700 1 --2- 172.21.15.1:0/1756787636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3dd006a750 0x7f3dd00ce8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.449+0000 7f3dcffff700 1 -- 172.21.15.1:0/1756787636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3dd006a750 msgr2=0x7f3dd00ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.449+0000 7f3dcffff700 1 --2- 172.21.15.1:0/1756787636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3dd006a750 0x7f3dd00ce8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.449+0000 7f3dcffff700 1 -- 172.21.15.1:0/1756787636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dd015d6b0 msgr2=0x7f3dd015c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.449+0000 7f3dcffff700 1 --2- 172.21.15.1:0/1756787636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dd015d6b0 0x7f3dd015c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.449+0000 7f3dcffff700 1 -- 172.21.15.1:0/1756787636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc4009550 con 0x7f3dd00ad790 2024-03-18T18:50:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.450+0000 7f3dcffff700 1 --2- 172.21.15.1:0/1756787636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dd00ad790 0x7f3dd015bdc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc00c7f0 tx=0x7f3dbc00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.450+0000 7f3dcd7fa700 1 -- 172.21.15.1:0/1756787636 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc011800 con 0x7f3dd00ad790 2024-03-18T18:50:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.450+0000 7f3dcd7fa700 1 -- 172.21.15.1:0/1756787636 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dbc011960 con 0x7f3dd00ad790 2024-03-18T18:50:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.450+0000 7f3dd70bd700 1 -- 172.21.15.1:0/1756787636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd014ffb0 con 0x7f3dd00ad790 2024-03-18T18:50:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.450+0000 7f3dcd7fa700 1 -- 172.21.15.1:0/1756787636 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc011cf0 con 0x7f3dd00ad790 2024-03-18T18:50:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.450+0000 7f3dd70bd700 1 -- 172.21.15.1:0/1756787636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd01504d0 con 0x7f3dd00ad790 2024-03-18T18:50:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.451+0000 7f3dd70bd700 1 -- 172.21.15.1:0/1756787636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dd00ad420 con 0x7f3dd00ad790 2024-03-18T18:50:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.451+0000 7f3dcd7fa700 1 -- 172.21.15.1:0/1756787636 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3dbc018360 con 0x7f3dd00ad790 2024-03-18T18:50:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.452+0000 7f3dcd7fa700 1 --2- 172.21.15.1:0/1756787636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3db406ac90 0x7f3db406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.452+0000 7f3dcd7fa700 1 -- 172.21.15.1:0/1756787636 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3dbc0894f0 con 0x7f3dd00ad790 2024-03-18T18:50:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.452+0000 7f3dd4e59700 1 --2- 172.21.15.1:0/1756787636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3db406ac90 0x7f3db406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.452+0000 7f3dd4e59700 1 --2- 172.21.15.1:0/1756787636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3db406ac90 0x7f3db406d150 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009c40 tx=0x7f3dc4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:08.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.457+0000 7f3dcd7fa700 1 -- 172.21.15.1:0/1756787636 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3dd00ad420 con 0x7f3dd00ad790 2024-03-18T18:50:08.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.660+0000 7f3dd70bd700 1 -- 172.21.15.1:0/1756787636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3dd00ad420 con 0x7f3dd00ad790 2024-03-18T18:50:08.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.661+0000 7f3dcd7fa700 1 -- 172.21.15.1:0/1756787636 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17830 (secure 0 0 0) 0x7f3dd00ad420 con 0x7f3dd00ad790 2024-03-18T18:50:08.658 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:08.658 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:49.276961+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:08.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.664+0000 7f3dd70bd700 1 -- 172.21.15.1:0/1756787636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3db406ac90 msgr2=0x7f3db406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:08.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.664+0000 7f3dd70bd700 1 --2- 172.21.15.1:0/1756787636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3db406ac90 0x7f3db406d150 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009c40 tx=0x7f3dc4008030 comp rx=0 tx=0).stop 2024-03-18T18:50:08.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.664+0000 7f3dd70bd700 1 -- 172.21.15.1:0/1756787636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dd00ad790 msgr2=0x7f3dd015bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:08.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.664+0000 7f3dd70bd700 1 --2- 172.21.15.1:0/1756787636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dd00ad790 0x7f3dd015bdc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc00c7f0 tx=0x7f3dbc00cb00 comp rx=0 tx=0).stop 2024-03-18T18:50:08.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.664+0000 7f3dd70bd700 1 -- 172.21.15.1:0/1756787636 shutdown_connections 2024-03-18T18:50:08.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.664+0000 7f3dd70bd700 1 --2- 172.21.15.1:0/1756787636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3db406ac90 0x7f3db406d150 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.664+0000 7f3dd70bd700 1 --2- 172.21.15.1:0/1756787636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3dd006a750 0x7f3dd00ce8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.664+0000 7f3dd70bd700 1 --2- 172.21.15.1:0/1756787636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dd015d6b0 0x7f3dd015c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.664+0000 7f3dd70bd700 1 --2- 172.21.15.1:0/1756787636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dd00ad790 0x7f3dd015bdc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.664+0000 7f3dd70bd700 1 -- 172.21.15.1:0/1756787636 >> 172.21.15.1:0/1756787636 conn(0x7f3dd006b860 msgr2=0x7f3dd00b1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:08.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.664+0000 7f3dd70bd700 1 -- 172.21.15.1:0/1756787636 shutdown_connections 2024-03-18T18:50:08.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.664+0000 7f3dd70bd700 1 -- 172.21.15.1:0/1756787636 wait complete. 2024-03-18T18:50:08.672 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T18:50:08.673 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T18:50:08.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.801+0000 7fa5ccada700 1 Processor -- start 2024-03-18T18:50:08.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.801+0000 7fa5ccada700 1 -- start start 2024-03-18T18:50:08.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.802+0000 7fa5ccada700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa5c806e290 0x7fa5c8072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:08.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.802+0000 7fa5ccada700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5c806dd50 0x7fa5c80cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:08.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.802+0000 7fa5ccada700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c806f490 con 0x7fa5c815d580 2024-03-18T18:50:08.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.802+0000 7fa5ccada700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c8071cf0 con 0x7fa5c806e290 2024-03-18T18:50:08.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.802+0000 7fa5ccada700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c806f1b0 con 0x7fa5c806dd50 2024-03-18T18:50:08.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.802+0000 7fa5c5d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa5c806e290 0x7fa5c8072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:08.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.802+0000 7fa5c659c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa5c815d580 0x7fa5c806d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60208/0 (socket says 172.21.15.1:60208) 2024-03-18T18:50:08.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.802+0000 7fa5c659c700 1 -- 172.21.15.1:0/3887818855 learned_addr learned my addr 172.21.15.1:0/3887818855 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:08.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.802+0000 7fa5c6d9d700 1 --2- 172.21.15.1:0/3887818855 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5c806dd50 0x7fa5c80cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:08.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.802+0000 7fa5c5d9b700 1 -- 172.21.15.1:0/3887818855 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5c806dd50 msgr2=0x7fa5c80cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:08.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.802+0000 7fa5c5d9b700 1 --2- 172.21.15.1:0/3887818855 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5c806dd50 0x7fa5c80cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.802+0000 7fa5c5d9b700 1 -- 172.21.15.1:0/3887818855 >> v1:172.21.15.1:6789/0 conn(0x7fa5c815d580 legacy=0x7fa5c806d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:08.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.802+0000 7fa5c559a700 1 -- 172.21.15.1:0/3887818855 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298915496 0 0) 0x7fa5c806f490 con 0x7fa5c815d580 2024-03-18T18:50:08.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.802+0000 7fa5c5d9b700 1 -- 172.21.15.1:0/3887818855 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c80716a0 con 0x7fa5c806e290 2024-03-18T18:50:08.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.803+0000 7fa5c6d9d700 1 --2- 172.21.15.1:0/3887818855 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5c806dd50 0x7fa5c80cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:08.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.803+0000 7fa5c5d9b700 1 --2- 172.21.15.1:0/3887818855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5c806e290 0x7fa5c8072180 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009840 tx=0x7fa5b8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2298f07343410107 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:08.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.803+0000 7fa5c559a700 1 -- 172.21.15.1:0/3887818855 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8011830 con 0x7fa5c806e290 2024-03-18T18:50:08.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.803+0000 7fa5c559a700 1 -- 172.21.15.1:0/3887818855 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b8011990 con 0x7fa5c806e290 2024-03-18T18:50:08.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.803+0000 7fa5c559a700 1 -- 172.21.15.1:0/3887818855 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8011d20 con 0x7fa5c806e290 2024-03-18T18:50:08.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.803+0000 7fa5ccada700 1 -- 172.21.15.1:0/3887818855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5c806e290 msgr2=0x7fa5c8072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:08.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.803+0000 7fa5ccada700 1 --2- 172.21.15.1:0/3887818855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5c806e290 0x7fa5c8072180 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009840 tx=0x7fa5b8009b50 comp rx=0 tx=0).stop 2024-03-18T18:50:08.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.803+0000 7fa5ccada700 1 -- 172.21.15.1:0/3887818855 shutdown_connections 2024-03-18T18:50:08.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.803+0000 7fa5ccada700 1 --2- 172.21.15.1:0/3887818855 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5c806dd50 0x7fa5c80cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.803+0000 7fa5ccada700 1 --2- 172.21.15.1:0/3887818855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5c806e290 0x7fa5c8072180 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.803+0000 7fa5ccada700 1 -- 172.21.15.1:0/3887818855 >> 172.21.15.1:0/3887818855 conn(0x7fa5c80cc990 msgr2=0x7fa5c80ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:08.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.803+0000 7fa5ccada700 1 -- 172.21.15.1:0/3887818855 shutdown_connections 2024-03-18T18:50:08.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.804+0000 7fa5ccada700 1 -- 172.21.15.1:0/3887818855 wait complete. 2024-03-18T18:50:08.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.805+0000 7fa5ccada700 1 Processor -- start 2024-03-18T18:50:08.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.805+0000 7fa5ccada700 1 -- start start 2024-03-18T18:50:08.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5ccada700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5c806dd50 0x7fa5c81593d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:08.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5ccada700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5c815d580 0x7fa5c80b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:08.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5ccada700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5c81f36a0 0x7fa5c80b11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:08.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5ccada700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c81f3210 con 0x7fa5c806dd50 2024-03-18T18:50:08.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5ccada700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c81f6cd0 con 0x7fa5c81f36a0 2024-03-18T18:50:08.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5ccada700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c81f2bc0 con 0x7fa5c815d580 2024-03-18T18:50:08.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5c659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5c806dd50 0x7fa5c81593d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:08.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5c659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5c806dd50 0x7fa5c81593d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45074/0 (socket says 172.21.15.1:45074) 2024-03-18T18:50:08.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5c659c700 1 -- 172.21.15.1:0/2650253252 learned_addr learned my addr 172.21.15.1:0/2650253252 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:08.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5c5d9b700 1 --2- 172.21.15.1:0/2650253252 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5c815d580 0x7fa5c80b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:08.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5c6d9d700 1 --2- 172.21.15.1:0/2650253252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5c81f36a0 0x7fa5c80b11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:08.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5c659c700 1 -- 172.21.15.1:0/2650253252 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5c815d580 msgr2=0x7fa5c80b0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:08.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5c659c700 1 --2- 172.21.15.1:0/2650253252 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5c815d580 0x7fa5c80b0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5c659c700 1 -- 172.21.15.1:0/2650253252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5c81f36a0 msgr2=0x7fa5c80b11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:08.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5c659c700 1 --2- 172.21.15.1:0/2650253252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5c81f36a0 0x7fa5c80b11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:08.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5c659c700 1 -- 172.21.15.1:0/2650253252 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b80094a0 con 0x7fa5c806dd50 2024-03-18T18:50:08.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5c659c700 1 --2- 172.21.15.1:0/2650253252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5c806dd50 0x7fa5c81593d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc00c550 tx=0x7fa5bc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:08.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5b6ffd700 1 -- 172.21.15.1:0/2650253252 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc004c80 con 0x7fa5c806dd50 2024-03-18T18:50:08.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5b6ffd700 1 -- 172.21.15.1:0/2650253252 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5bc004e00 con 0x7fa5c806dd50 2024-03-18T18:50:08.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5b6ffd700 1 -- 172.21.15.1:0/2650253252 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc007310 con 0x7fa5c806dd50 2024-03-18T18:50:08.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5ccada700 1 -- 172.21.15.1:0/2650253252 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c81f2ea0 con 0x7fa5c806dd50 2024-03-18T18:50:08.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.806+0000 7fa5ccada700 1 -- 172.21.15.1:0/2650253252 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c8159910 con 0x7fa5c806dd50 2024-03-18T18:50:08.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.807+0000 7fa5ccada700 1 -- 172.21.15.1:0/2650253252 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c81f34f0 con 0x7fa5c806dd50 2024-03-18T18:50:08.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.808+0000 7fa5b6ffd700 1 -- 172.21.15.1:0/2650253252 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa5bc007470 con 0x7fa5c806dd50 2024-03-18T18:50:08.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.808+0000 7fa5b6ffd700 1 --2- 172.21.15.1:0/2650253252 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa5a806ac90 0x7fa5a806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:08.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.808+0000 7fa5b6ffd700 1 -- 172.21.15.1:0/2650253252 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa5bc089370 con 0x7fa5c806dd50 2024-03-18T18:50:08.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.808+0000 7fa5c5d9b700 1 --2- 172.21.15.1:0/2650253252 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa5a806ac90 0x7fa5a806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:08.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.809+0000 7fa5c5d9b700 1 --2- 172.21.15.1:0/2650253252 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa5a806ac90 0x7fa5a806d150 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8160820 tx=0x7fa5b8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:08.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:08.813+0000 7fa5b6ffd700 1 -- 172.21.15.1:0/2650253252 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa5bc054260 con 0x7fa5c806dd50 2024-03-18T18:50:09.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.015+0000 7fa5ccada700 1 -- 172.21.15.1:0/2650253252 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa5c806ee40 con 0x7fa5a806ac90 2024-03-18T18:50:09.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.026+0000 7fa5b6ffd700 1 -- 172.21.15.1:0/2650253252 <== mgr.4104 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345190 (secure 0 0 0) 0x7fa5b8011b80 con 0x7fa5a806ac90 2024-03-18T18:50:09.027 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.029+0000 7fa5ccada700 1 -- 172.21.15.1:0/2650253252 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa5a806ac90 msgr2=0x7fa5a806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.029+0000 7fa5ccada700 1 --2- 172.21.15.1:0/2650253252 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa5a806ac90 0x7fa5a806d150 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8160820 tx=0x7fa5b8008030 comp rx=0 tx=0).stop 2024-03-18T18:50:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.029+0000 7fa5ccada700 1 -- 172.21.15.1:0/2650253252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5c806dd50 msgr2=0x7fa5c81593d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.029+0000 7fa5ccada700 1 --2- 172.21.15.1:0/2650253252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5c806dd50 0x7fa5c81593d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc00c550 tx=0x7fa5bc00c910 comp rx=0 tx=0).stop 2024-03-18T18:50:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.029+0000 7fa5ccada700 1 -- 172.21.15.1:0/2650253252 shutdown_connections 2024-03-18T18:50:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.029+0000 7fa5ccada700 1 --2- 172.21.15.1:0/2650253252 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa5a806ac90 0x7fa5a806d150 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.029+0000 7fa5ccada700 1 --2- 172.21.15.1:0/2650253252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5c81f36a0 0x7fa5c80b11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.029+0000 7fa5ccada700 1 --2- 172.21.15.1:0/2650253252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5c806dd50 0x7fa5c81593d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.029+0000 7fa5ccada700 1 --2- 172.21.15.1:0/2650253252 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5c815d580 0x7fa5c80b0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.029+0000 7fa5ccada700 1 -- 172.21.15.1:0/2650253252 >> 172.21.15.1:0/2650253252 conn(0x7fa5c80cc990 msgr2=0x7fa5c80684d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.030+0000 7fa5ccada700 1 -- 172.21.15.1:0/2650253252 shutdown_connections 2024-03-18T18:50:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.030+0000 7fa5ccada700 1 -- 172.21.15.1:0/2650253252 wait complete. 2024-03-18T18:50:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T18:50:09.039 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-03-18T18:50:07.392187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594961,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219464,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755256,"statfs":{"total":773094113280,"available":772869382144,"internally_reserved":0,"allocated":3383296,"data_stored":2025530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2231,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003016"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976876+0000","last_change":"2024-03-18T18:49:44.831395+0000","last_active":"2024-03-18T18:49:48.976876+0000","last_peered":"2024-03-18T18:49:48.976876+0000","last_clean":"2024-03-18T18:49:48.976876+0000","last_became_active":"2024-03-18T18:49:44.830706+0000","last_became_peered":"2024-03-18T18:49:44.830706+0000","last_unstale":"2024-03-18T18:49:48.976876+0000","last_undegraded":"2024-03-18T18:49:48.976876+0000","last_fullsized":"2024-03-18T18:49:48.976876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:20:43.395005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841775+0000","last_change":"2024-03-18T18:49:44.829492+0000","last_active":"2024-03-18T18:49:48.841775+0000","last_peered":"2024-03-18T18:49:48.841775+0000","last_clean":"2024-03-18T18:49:48.841775+0000","last_became_active":"2024-03-18T18:49:44.829223+0000","last_became_peered":"2024-03-18T18:49:44.829223+0000","last_unstale":"2024-03-18T18:49:48.841775+0000","last_undegraded":"2024-03-18T18:49:48.841775+0000","last_fullsized":"2024-03-18T18:49:48.841775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.845869+0000","last_change":"2024-03-18T18:49:44.831909+0000","last_active":"2024-03-18T18:49:48.845869+0000","last_peered":"2024-03-18T18:49:48.845869+0000","last_clean":"2024-03-18T18:49:48.845869+0000","last_became_active":"2024-03-18T18:49:44.831661+0000","last_became_peered":"2024-03-18T18:49:44.831661+0000","last_unstale":"2024-03-18T18:49:48.845869+0000","last_undegraded":"2024-03-18T18:49:48.845869+0000","last_fullsized":"2024-03-18T18:49:48.845869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:14.055199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840094+0000","last_change":"2024-03-18T18:49:44.838298+0000","last_active":"2024-03-18T18:49:48.840094+0000","last_peered":"2024-03-18T18:49:48.840094+0000","last_clean":"2024-03-18T18:49:48.840094+0000","last_became_active":"2024-03-18T18:49:44.837689+0000","last_became_peered":"2024-03-18T18:49:44.837689+0000","last_unstale":"2024-03-18T18:49:48.840094+0000","last_undegraded":"2024-03-18T18:49:48.840094+0000","last_fullsized":"2024-03-18T18:49:48.840094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:50.238592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975645+0000","last_change":"2024-03-18T18:49:44.841081+0000","last_active":"2024-03-18T18:49:48.975645+0000","last_peered":"2024-03-18T18:49:48.975645+0000","last_clean":"2024-03-18T18:49:48.975645+0000","last_became_active":"2024-03-18T18:49:44.840798+0000","last_became_peered":"2024-03-18T18:49:44.840798+0000","last_unstale":"2024-03-18T18:49:48.975645+0000","last_undegraded":"2024-03-18T18:49:48.975645+0000","last_fullsized":"2024-03-18T18:49:48.975645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:27:55.580010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839590+0000","last_change":"2024-03-18T18:49:44.836710+0000","last_active":"2024-03-18T18:49:48.839590+0000","last_peered":"2024-03-18T18:49:48.839590+0000","last_clean":"2024-03-18T18:49:48.839590+0000","last_became_active":"2024-03-18T18:49:44.836473+0000","last_became_peered":"2024-03-18T18:49:44.836473+0000","last_unstale":"2024-03-18T18:49:48.839590+0000","last_undegraded":"2024-03-18T18:49:48.839590+0000","last_fullsized":"2024-03-18T18:49:48.839590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:59:20.616256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976795+0000","last_change":"2024-03-18T18:49:44.829680+0000","last_active":"2024-03-18T18:49:48.976795+0000","last_peered":"2024-03-18T18:49:48.976795+0000","last_clean":"2024-03-18T18:49:48.976795+0000","last_became_active":"2024-03-18T18:49:44.829139+0000","last_became_peered":"2024-03-18T18:49:44.829139+0000","last_unstale":"2024-03-18T18:49:48.976795+0000","last_undegraded":"2024-03-18T18:49:48.976795+0000","last_fullsized":"2024-03-18T18:49:48.976795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:59:20.616256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839889+0000","last_change":"2024-03-18T18:49:44.837035+0000","last_active":"2024-03-18T18:49:48.839889+0000","last_peered":"2024-03-18T18:49:48.839889+0000","last_clean":"2024-03-18T18:49:48.839889+0000","last_became_active":"2024-03-18T18:49:44.836598+0000","last_became_peered":"2024-03-18T18:49:44.836598+0000","last_unstale":"2024-03-18T18:49:48.839889+0000","last_undegraded":"2024-03-18T18:49:48.839889+0000","last_fullsized":"2024-03-18T18:49:48.839889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:50.238592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976827+0000","last_change":"2024-03-18T18:49:44.835014+0000","last_active":"2024-03-18T18:49:48.976827+0000","last_peered":"2024-03-18T18:49:48.976827+0000","last_clean":"2024-03-18T18:49:48.976827+0000","last_became_active":"2024-03-18T18:49:44.834438+0000","last_became_peered":"2024-03-18T18:49:44.834438+0000","last_unstale":"2024-03-18T18:49:48.976827+0000","last_undegraded":"2024-03-18T18:49:48.976827+0000","last_fullsized":"2024-03-18T18:49:48.976827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:07.013090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835727+0000","last_change":"2024-03-18T18:49:44.837039+0000","last_active":"2024-03-18T18:49:48.835727+0000","last_peered":"2024-03-18T18:49:48.835727+0000","last_clean":"2024-03-18T18:49:48.835727+0000","last_became_active":"2024-03-18T18:49:44.836749+0000","last_became_peered":"2024-03-18T18:49:44.836749+0000","last_unstale":"2024-03-18T18:49:48.835727+0000","last_undegraded":"2024-03-18T18:49:48.835727+0000","last_fullsized":"2024-03-18T18:49:48.835727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836415+0000","last_change":"2024-03-18T18:49:44.832546+0000","last_active":"2024-03-18T18:49:48.836415+0000","last_peered":"2024-03-18T18:49:48.836415+0000","last_clean":"2024-03-18T18:49:48.836415+0000","last_became_active":"2024-03-18T18:49:44.832333+0000","last_became_peered":"2024-03-18T18:49:44.832333+0000","last_unstale":"2024-03-18T18:49:48.836415+0000","last_undegraded":"2024-03-18T18:49:48.836415+0000","last_fullsized":"2024-03-18T18:49:48.836415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842011+0000","last_change":"2024-03-18T18:49:44.834366+0000","last_active":"2024-03-18T18:49:48.842011+0000","last_peered":"2024-03-18T18:49:48.842011+0000","last_clean":"2024-03-18T18:49:48.842011+0000","last_became_active":"2024-03-18T18:49:44.833912+0000","last_became_peered":"2024-03-18T18:49:44.833912+0000","last_unstale":"2024-03-18T18:49:48.842011+0000","last_undegraded":"2024-03-18T18:49:48.842011+0000","last_fullsized":"2024-03-18T18:49:48.842011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:09:08.248420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839130+0000","last_change":"2024-03-18T18:49:44.823814+0000","last_active":"2024-03-18T18:49:48.839130+0000","last_peered":"2024-03-18T18:49:48.839130+0000","last_clean":"2024-03-18T18:49:48.839130+0000","last_became_active":"2024-03-18T18:49:44.823554+0000","last_became_peered":"2024-03-18T18:49:44.823554+0000","last_unstale":"2024-03-18T18:49:48.839130+0000","last_undegraded":"2024-03-18T18:49:48.839130+0000","last_fullsized":"2024-03-18T18:49:48.839130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:05.661348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853566+0000","last_change":"2024-03-18T18:49:44.828163+0000","last_active":"2024-03-18T18:49:48.853566+0000","last_peered":"2024-03-18T18:49:48.853566+0000","last_clean":"2024-03-18T18:49:48.853566+0000","last_became_active":"2024-03-18T18:49:44.827352+0000","last_became_peered":"2024-03-18T18:49:44.827352+0000","last_unstale":"2024-03-18T18:49:48.853566+0000","last_undegraded":"2024-03-18T18:49:48.853566+0000","last_fullsized":"2024-03-18T18:49:48.853566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:30.009397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840573+0000","last_change":"2024-03-18T18:49:44.839200+0000","last_active":"2024-03-18T18:49:48.840573+0000","last_peered":"2024-03-18T18:49:48.840573+0000","last_clean":"2024-03-18T18:49:48.840573+0000","last_became_active":"2024-03-18T18:49:44.838967+0000","last_became_peered":"2024-03-18T18:49:44.838967+0000","last_unstale":"2024-03-18T18:49:48.840573+0000","last_undegraded":"2024-03-18T18:49:48.840573+0000","last_fullsized":"2024-03-18T18:49:48.840573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:14.055199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839664+0000","last_change":"2024-03-18T18:49:44.834022+0000","last_active":"2024-03-18T18:49:48.839664+0000","last_peered":"2024-03-18T18:49:48.839664+0000","last_clean":"2024-03-18T18:49:48.839664+0000","last_became_active":"2024-03-18T18:49:44.833758+0000","last_became_peered":"2024-03-18T18:49:44.833758+0000","last_unstale":"2024-03-18T18:49:48.839664+0000","last_undegraded":"2024-03-18T18:49:48.839664+0000","last_fullsized":"2024-03-18T18:49:48.839664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:25:14.491414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840752+0000","last_change":"2024-03-18T18:49:44.842296+0000","last_active":"2024-03-18T18:49:48.840752+0000","last_peered":"2024-03-18T18:49:48.840752+0000","last_clean":"2024-03-18T18:49:48.840752+0000","last_became_active":"2024-03-18T18:49:44.841720+0000","last_became_peered":"2024-03-18T18:49:44.841720+0000","last_unstale":"2024-03-18T18:49:48.840752+0000","last_undegraded":"2024-03-18T18:49:48.840752+0000","last_fullsized":"2024-03-18T18:49:48.840752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836699+0000","last_change":"2024-03-18T18:49:44.828352+0000","last_active":"2024-03-18T18:49:48.836699+0000","last_peered":"2024-03-18T18:49:48.836699+0000","last_clean":"2024-03-18T18:49:48.836699+0000","last_became_active":"2024-03-18T18:49:44.828059+0000","last_became_peered":"2024-03-18T18:49:44.828059+0000","last_unstale":"2024-03-18T18:49:48.836699+0000","last_undegraded":"2024-03-18T18:49:48.836699+0000","last_fullsized":"2024-03-18T18:49:48.836699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:29.829759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976858+0000","last_change":"2024-03-18T18:49:44.841034+0000","last_active":"2024-03-18T18:49:48.976858+0000","last_peered":"2024-03-18T18:49:48.976858+0000","last_clean":"2024-03-18T18:49:48.976858+0000","last_became_active":"2024-03-18T18:49:44.840765+0000","last_became_peered":"2024-03-18T18:49:44.840765+0000","last_unstale":"2024-03-18T18:49:48.976858+0000","last_undegraded":"2024-03-18T18:49:48.976858+0000","last_fullsized":"2024-03-18T18:49:48.976858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:34.102274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.982406+0000","last_change":"2024-03-18T18:49:48.981322+0000","last_active":"2024-03-18T18:49:48.982406+0000","last_peered":"2024-03-18T18:49:48.982406+0000","last_clean":"2024-03-18T18:49:48.982406+0000","last_became_active":"2024-03-18T18:49:48.980927+0000","last_became_peered":"2024-03-18T18:49:48.980927+0000","last_unstale":"2024-03-18T18:49:48.982406+0000","last_undegraded":"2024-03-18T18:49:48.982406+0000","last_fullsized":"2024-03-18T18:49:48.982406+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:57:44.834422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.847587+0000","last_change":"2024-03-18T18:49:44.396090+0000","last_active":"2024-03-18T18:49:48.847587+0000","last_peered":"2024-03-18T18:49:48.847587+0000","last_clean":"2024-03-18T18:49:48.847587+0000","last_became_active":"2024-03-18T18:49:44.395823+0000","last_became_peered":"2024-03-18T18:49:44.395823+0000","last_unstale":"2024-03-18T18:49:48.847587+0000","last_undegraded":"2024-03-18T18:49:48.847587+0000","last_fullsized":"2024-03-18T18:49:48.847587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:00:40.075042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840491+0000","last_change":"2024-03-18T18:49:44.821381+0000","last_active":"2024-03-18T18:49:48.840491+0000","last_peered":"2024-03-18T18:49:48.840491+0000","last_clean":"2024-03-18T18:49:48.840491+0000","last_became_active":"2024-03-18T18:49:44.821129+0000","last_became_peered":"2024-03-18T18:49:44.821129+0000","last_unstale":"2024-03-18T18:49:48.840491+0000","last_undegraded":"2024-03-18T18:49:48.840491+0000","last_fullsized":"2024-03-18T18:49:48.840491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853754+0000","last_change":"2024-03-18T18:49:48.852436+0000","last_active":"2024-03-18T18:49:48.853754+0000","last_peered":"2024-03-18T18:49:48.853754+0000","last_clean":"2024-03-18T18:49:48.853754+0000","last_became_active":"2024-03-18T18:49:48.851959+0000","last_became_peered":"2024-03-18T18:49:48.851959+0000","last_unstale":"2024-03-18T18:49:48.853754+0000","last_undegraded":"2024-03-18T18:49:48.853754+0000","last_fullsized":"2024-03-18T18:49:48.853754+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:31:52.237923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840602+0000","last_change":"2024-03-18T18:49:44.399646+0000","last_active":"2024-03-18T18:49:48.840602+0000","last_peered":"2024-03-18T18:49:48.840602+0000","last_clean":"2024-03-18T18:49:48.840602+0000","last_became_active":"2024-03-18T18:49:44.399368+0000","last_became_peered":"2024-03-18T18:49:44.399368+0000","last_unstale":"2024-03-18T18:49:48.840602+0000","last_undegraded":"2024-03-18T18:49:48.840602+0000","last_fullsized":"2024-03-18T18:49:48.840602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:29.763642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.847482+0000","last_change":"2024-03-18T18:49:44.832977+0000","last_active":"2024-03-18T18:49:48.847482+0000","last_peered":"2024-03-18T18:49:48.847482+0000","last_clean":"2024-03-18T18:49:48.847482+0000","last_became_active":"2024-03-18T18:49:44.832624+0000","last_became_peered":"2024-03-18T18:49:44.832624+0000","last_unstale":"2024-03-18T18:49:48.847482+0000","last_undegraded":"2024-03-18T18:49:48.847482+0000","last_fullsized":"2024-03-18T18:49:48.847482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853697+0000","last_change":"2024-03-18T18:49:48.852585+0000","last_active":"2024-03-18T18:49:48.853697+0000","last_peered":"2024-03-18T18:49:48.853697+0000","last_clean":"2024-03-18T18:49:48.853697+0000","last_became_active":"2024-03-18T18:49:48.852137+0000","last_became_peered":"2024-03-18T18:49:48.852137+0000","last_unstale":"2024-03-18T18:49:48.853697+0000","last_undegraded":"2024-03-18T18:49:48.853697+0000","last_fullsized":"2024-03-18T18:49:48.853697+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:50:12.397097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975894+0000","last_change":"2024-03-18T18:49:44.394010+0000","last_active":"2024-03-18T18:49:48.975894+0000","last_peered":"2024-03-18T18:49:48.975894+0000","last_clean":"2024-03-18T18:49:48.975894+0000","last_became_active":"2024-03-18T18:49:44.393512+0000","last_became_peered":"2024-03-18T18:49:44.393512+0000","last_unstale":"2024-03-18T18:49:48.975894+0000","last_undegraded":"2024-03-18T18:49:48.975894+0000","last_fullsized":"2024-03-18T18:49:48.975894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:10:24.082597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839873+0000","last_change":"2024-03-18T18:49:44.836167+0000","last_active":"2024-03-18T18:49:48.839873+0000","last_peered":"2024-03-18T18:49:48.839873+0000","last_clean":"2024-03-18T18:49:48.839873+0000","last_became_active":"2024-03-18T18:49:44.835887+0000","last_became_peered":"2024-03-18T18:49:44.835887+0000","last_unstale":"2024-03-18T18:49:48.839873+0000","last_undegraded":"2024-03-18T18:49:48.839873+0000","last_fullsized":"2024-03-18T18:49:48.839873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:34.102274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.857961+0000","last_change":"2024-03-18T18:49:48.857276+0000","last_active":"2024-03-18T18:49:48.857961+0000","last_peered":"2024-03-18T18:49:48.857961+0000","last_clean":"2024-03-18T18:49:48.857961+0000","last_became_active":"2024-03-18T18:49:48.856870+0000","last_became_peered":"2024-03-18T18:49:48.856870+0000","last_unstale":"2024-03-18T18:49:48.857961+0000","last_undegraded":"2024-03-18T18:49:48.857961+0000","last_fullsized":"2024-03-18T18:49:48.857961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:31.426438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975669+0000","last_change":"2024-03-18T18:49:44.401760+0000","last_active":"2024-03-18T18:49:48.975669+0000","last_peered":"2024-03-18T18:49:48.975669+0000","last_clean":"2024-03-18T18:49:48.975669+0000","last_became_active":"2024-03-18T18:49:44.401565+0000","last_became_peered":"2024-03-18T18:49:44.401565+0000","last_unstale":"2024-03-18T18:49:48.975669+0000","last_undegraded":"2024-03-18T18:49:48.975669+0000","last_fullsized":"2024-03-18T18:49:48.975669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:09:07.245729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840474+0000","last_change":"2024-03-18T18:49:44.834985+0000","last_active":"2024-03-18T18:49:48.840474+0000","last_peered":"2024-03-18T18:49:48.840474+0000","last_clean":"2024-03-18T18:49:48.840474+0000","last_became_active":"2024-03-18T18:49:44.834719+0000","last_became_peered":"2024-03-18T18:49:44.834719+0000","last_unstale":"2024-03-18T18:49:48.840474+0000","last_undegraded":"2024-03-18T18:49:48.840474+0000","last_fullsized":"2024-03-18T18:49:48.840474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:27:55.580010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.854528+0000","last_change":"2024-03-18T18:49:48.852823+0000","last_active":"2024-03-18T18:49:48.854528+0000","last_peered":"2024-03-18T18:49:48.854528+0000","last_clean":"2024-03-18T18:49:48.854528+0000","last_became_active":"2024-03-18T18:49:48.852431+0000","last_became_peered":"2024-03-18T18:49:48.852431+0000","last_unstale":"2024-03-18T18:49:48.854528+0000","last_undegraded":"2024-03-18T18:49:48.854528+0000","last_fullsized":"2024-03-18T18:49:48.854528+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:10:04.268147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839292+0000","last_change":"2024-03-18T18:49:44.402292+0000","last_active":"2024-03-18T18:49:48.839292+0000","last_peered":"2024-03-18T18:49:48.839292+0000","last_clean":"2024-03-18T18:49:48.839292+0000","last_became_active":"2024-03-18T18:49:44.402028+0000","last_became_peered":"2024-03-18T18:49:44.402028+0000","last_unstale":"2024-03-18T18:49:48.839292+0000","last_undegraded":"2024-03-18T18:49:48.839292+0000","last_fullsized":"2024-03-18T18:49:48.839292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.843349+0000","last_change":"2024-03-18T18:49:44.832023+0000","last_active":"2024-03-18T18:49:48.843349+0000","last_peered":"2024-03-18T18:49:48.843349+0000","last_clean":"2024-03-18T18:49:48.843349+0000","last_became_active":"2024-03-18T18:49:44.831734+0000","last_became_peered":"2024-03-18T18:49:44.831734+0000","last_unstale":"2024-03-18T18:49:48.843349+0000","last_undegraded":"2024-03-18T18:49:48.843349+0000","last_fullsized":"2024-03-18T18:49:48.843349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:10:25.085288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.856842+0000","last_change":"2024-03-18T18:49:48.852370+0000","last_active":"2024-03-18T18:49:48.856842+0000","last_peered":"2024-03-18T18:49:48.856842+0000","last_clean":"2024-03-18T18:49:48.856842+0000","last_became_active":"2024-03-18T18:49:48.851953+0000","last_became_peered":"2024-03-18T18:49:48.851953+0000","last_unstale":"2024-03-18T18:49:48.856842+0000","last_undegraded":"2024-03-18T18:49:48.856842+0000","last_fullsized":"2024-03-18T18:49:48.856842+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:17.506065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836324+0000","last_change":"2024-03-18T18:49:44.394163+0000","last_active":"2024-03-18T18:49:48.836324+0000","last_peered":"2024-03-18T18:49:48.836324+0000","last_clean":"2024-03-18T18:49:48.836324+0000","last_became_active":"2024-03-18T18:49:44.393105+0000","last_became_peered":"2024-03-18T18:49:44.393105+0000","last_unstale":"2024-03-18T18:49:48.836324+0000","last_undegraded":"2024-03-18T18:49:48.836324+0000","last_fullsized":"2024-03-18T18:49:48.836324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:30.124822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.846218+0000","last_change":"2024-03-18T18:49:44.834950+0000","last_active":"2024-03-18T18:49:48.846218+0000","last_peered":"2024-03-18T18:49:48.846218+0000","last_clean":"2024-03-18T18:49:48.846218+0000","last_became_active":"2024-03-18T18:49:44.834765+0000","last_became_peered":"2024-03-18T18:49:44.834765+0000","last_unstale":"2024-03-18T18:49:48.846218+0000","last_undegraded":"2024-03-18T18:49:48.846218+0000","last_fullsized":"2024-03-18T18:49:48.846218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:49.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.857842+0000","last_change":"2024-03-18T18:49:48.857127+0000","last_active":"2024-03-18T18:49:48.857842+0000","last_peered":"2024-03-18T18:49:48.857842+0000","last_clean":"2024-03-18T18:49:48.857842+0000","last_became_active":"2024-03-18T18:49:48.856745+0000","last_became_peered":"2024-03-18T18:49:48.856745+0000","last_unstale":"2024-03-18T18:49:48.857842+0000","last_undegraded":"2024-03-18T18:49:48.857842+0000","last_fullsized":"2024-03-18T18:49:48.857842+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:46:22.763891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840897+0000","last_change":"2024-03-18T18:49:44.396730+0000","last_active":"2024-03-18T18:49:48.840897+0000","last_peered":"2024-03-18T18:49:48.840897+0000","last_clean":"2024-03-18T18:49:48.840897+0000","last_became_active":"2024-03-18T18:49:44.396415+0000","last_became_peered":"2024-03-18T18:49:44.396415+0000","last_unstale":"2024-03-18T18:49:48.840897+0000","last_undegraded":"2024-03-18T18:49:48.840897+0000","last_fullsized":"2024-03-18T18:49:48.840897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:28.338064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839641+0000","last_change":"2024-03-18T18:49:44.836855+0000","last_active":"2024-03-18T18:49:48.839641+0000","last_peered":"2024-03-18T18:49:48.839641+0000","last_clean":"2024-03-18T18:49:48.839641+0000","last_became_active":"2024-03-18T18:49:44.836275+0000","last_became_peered":"2024-03-18T18:49:44.836275+0000","last_unstale":"2024-03-18T18:49:48.839641+0000","last_undegraded":"2024-03-18T18:49:48.839641+0000","last_fullsized":"2024-03-18T18:49:48.839641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:49.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980433+0000","last_change":"2024-03-18T18:49:48.978139+0000","last_active":"2024-03-18T18:49:48.980433+0000","last_peered":"2024-03-18T18:49:48.980433+0000","last_clean":"2024-03-18T18:49:48.980433+0000","last_became_active":"2024-03-18T18:49:48.977755+0000","last_became_peered":"2024-03-18T18:49:48.977755+0000","last_unstale":"2024-03-18T18:49:48.980433+0000","last_undegraded":"2024-03-18T18:49:48.980433+0000","last_fullsized":"2024-03-18T18:49:48.980433+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:43.744272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839770+0000","last_change":"2024-03-18T18:49:44.400836+0000","last_active":"2024-03-18T18:49:48.839770+0000","last_peered":"2024-03-18T18:49:48.839770+0000","last_clean":"2024-03-18T18:49:48.839770+0000","last_became_active":"2024-03-18T18:49:44.400587+0000","last_became_peered":"2024-03-18T18:49:44.400587+0000","last_unstale":"2024-03-18T18:49:48.839770+0000","last_undegraded":"2024-03-18T18:49:48.839770+0000","last_fullsized":"2024-03-18T18:49:48.839770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:30.124822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840775+0000","last_change":"2024-03-18T18:49:44.822283+0000","last_active":"2024-03-18T18:49:48.840775+0000","last_peered":"2024-03-18T18:49:48.840775+0000","last_clean":"2024-03-18T18:49:48.840775+0000","last_became_active":"2024-03-18T18:49:44.822016+0000","last_became_peered":"2024-03-18T18:49:44.822016+0000","last_unstale":"2024-03-18T18:49:48.840775+0000","last_undegraded":"2024-03-18T18:49:48.840775+0000","last_fullsized":"2024-03-18T18:49:48.840775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981750+0000","last_change":"2024-03-18T18:49:48.978066+0000","last_active":"2024-03-18T18:49:48.981750+0000","last_peered":"2024-03-18T18:49:48.981750+0000","last_clean":"2024-03-18T18:49:48.981750+0000","last_became_active":"2024-03-18T18:49:48.977682+0000","last_became_peered":"2024-03-18T18:49:48.977682+0000","last_unstale":"2024-03-18T18:49:48.981750+0000","last_undegraded":"2024-03-18T18:49:48.981750+0000","last_fullsized":"2024-03-18T18:49:48.981750+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:31:52.237923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836616+0000","last_change":"2024-03-18T18:49:44.402381+0000","last_active":"2024-03-18T18:49:48.836616+0000","last_peered":"2024-03-18T18:49:48.836616+0000","last_clean":"2024-03-18T18:49:48.836616+0000","last_became_active":"2024-03-18T18:49:44.402101+0000","last_became_peered":"2024-03-18T18:49:44.402101+0000","last_unstale":"2024-03-18T18:49:48.836616+0000","last_undegraded":"2024-03-18T18:49:48.836616+0000","last_fullsized":"2024-03-18T18:49:48.836616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:39.899537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840754+0000","last_change":"2024-03-18T18:49:44.841434+0000","last_active":"2024-03-18T18:49:48.840754+0000","last_peered":"2024-03-18T18:49:48.840754+0000","last_clean":"2024-03-18T18:49:48.840754+0000","last_became_active":"2024-03-18T18:49:44.841186+0000","last_became_peered":"2024-03-18T18:49:44.841186+0000","last_unstale":"2024-03-18T18:49:48.840754+0000","last_undegraded":"2024-03-18T18:49:48.840754+0000","last_fullsized":"2024-03-18T18:49:48.840754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.850697+0000","last_change":"2024-03-18T18:49:48.849692+0000","last_active":"2024-03-18T18:49:48.850697+0000","last_peered":"2024-03-18T18:49:48.850697+0000","last_clean":"2024-03-18T18:49:48.850697+0000","last_became_active":"2024-03-18T18:49:48.849230+0000","last_became_peered":"2024-03-18T18:49:48.849230+0000","last_unstale":"2024-03-18T18:49:48.850697+0000","last_undegraded":"2024-03-18T18:49:48.850697+0000","last_fullsized":"2024-03-18T18:49:48.850697+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:45:34.390044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840676+0000","last_change":"2024-03-18T18:49:44.400484+0000","last_active":"2024-03-18T18:49:48.840676+0000","last_peered":"2024-03-18T18:49:48.840676+0000","last_clean":"2024-03-18T18:49:48.840676+0000","last_became_active":"2024-03-18T18:49:44.398334+0000","last_became_peered":"2024-03-18T18:49:44.398334+0000","last_unstale":"2024-03-18T18:49:48.840676+0000","last_undegraded":"2024-03-18T18:49:48.840676+0000","last_fullsized":"2024-03-18T18:49:48.840676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:29.763642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.847751+0000","last_change":"2024-03-18T18:49:44.832690+0000","last_active":"2024-03-18T18:49:48.847751+0000","last_peered":"2024-03-18T18:49:48.847751+0000","last_clean":"2024-03-18T18:49:48.847751+0000","last_became_active":"2024-03-18T18:49:44.832317+0000","last_became_peered":"2024-03-18T18:49:44.832317+0000","last_unstale":"2024-03-18T18:49:48.847751+0000","last_undegraded":"2024-03-18T18:49:48.847751+0000","last_fullsized":"2024-03-18T18:49:48.847751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:34.102274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.850171+0000","last_change":"2024-03-18T18:49:48.849391+0000","last_active":"2024-03-18T18:49:48.850171+0000","last_peered":"2024-03-18T18:49:48.850171+0000","last_clean":"2024-03-18T18:49:48.850171+0000","last_became_active":"2024-03-18T18:49:48.849018+0000","last_became_peered":"2024-03-18T18:49:48.849018+0000","last_unstale":"2024-03-18T18:49:48.850171+0000","last_undegraded":"2024-03-18T18:49:48.850171+0000","last_fullsized":"2024-03-18T18:49:48.850171+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:20:46.845872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841507+0000","last_change":"2024-03-18T18:49:44.402300+0000","last_active":"2024-03-18T18:49:48.841507+0000","last_peered":"2024-03-18T18:49:48.841507+0000","last_clean":"2024-03-18T18:49:48.841507+0000","last_became_active":"2024-03-18T18:49:44.401998+0000","last_became_peered":"2024-03-18T18:49:44.401998+0000","last_unstale":"2024-03-18T18:49:48.841507+0000","last_undegraded":"2024-03-18T18:49:48.841507+0000","last_fullsized":"2024-03-18T18:49:48.841507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:09:07.245729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841888+0000","last_change":"2024-03-18T18:49:44.835310+0000","last_active":"2024-03-18T18:49:48.841888+0000","last_peered":"2024-03-18T18:49:48.841888+0000","last_clean":"2024-03-18T18:49:48.841888+0000","last_became_active":"2024-03-18T18:49:44.835035+0000","last_became_peered":"2024-03-18T18:49:44.835035+0000","last_unstale":"2024-03-18T18:49:48.841888+0000","last_undegraded":"2024-03-18T18:49:48.841888+0000","last_fullsized":"2024-03-18T18:49:48.841888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:29.829759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.865513+0000","last_change":"2024-03-18T18:49:48.863905+0000","last_active":"2024-03-18T18:49:48.865513+0000","last_peered":"2024-03-18T18:49:48.865513+0000","last_clean":"2024-03-18T18:49:48.865513+0000","last_became_active":"2024-03-18T18:49:48.863234+0000","last_became_peered":"2024-03-18T18:49:48.863234+0000","last_unstale":"2024-03-18T18:49:48.865513+0000","last_undegraded":"2024-03-18T18:49:48.865513+0000","last_fullsized":"2024-03-18T18:49:48.865513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:16:08.575913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T18:49:54.442358+0000","last_change":"2024-03-18T18:49:44.402443+0000","last_active":"2024-03-18T18:49:54.442358+0000","last_peered":"2024-03-18T18:49:54.442358+0000","last_clean":"2024-03-18T18:49:54.442358+0000","last_became_active":"2024-03-18T18:49:44.402144+0000","last_became_peered":"2024-03-18T18:49:44.402144+0000","last_unstale":"2024-03-18T18:49:54.442358+0000","last_undegraded":"2024-03-18T18:49:54.442358+0000","last_fullsized":"2024-03-18T18:49:54.442358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:06.816608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3795,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976826+0000","last_change":"2024-03-18T18:49:44.829814+0000","last_active":"2024-03-18T18:49:48.976826+0000","last_peered":"2024-03-18T18:49:48.976826+0000","last_clean":"2024-03-18T18:49:48.976826+0000","last_became_active":"2024-03-18T18:49:44.829548+0000","last_became_peered":"2024-03-18T18:49:44.829548+0000","last_unstale":"2024-03-18T18:49:48.976826+0000","last_undegraded":"2024-03-18T18:49:48.976826+0000","last_fullsized":"2024-03-18T18:49:48.976826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:02:06.725321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980345+0000","last_change":"2024-03-18T18:49:48.978265+0000","last_active":"2024-03-18T18:49:48.980345+0000","last_peered":"2024-03-18T18:49:48.980345+0000","last_clean":"2024-03-18T18:49:48.980345+0000","last_became_active":"2024-03-18T18:49:48.977853+0000","last_became_peered":"2024-03-18T18:49:48.977853+0000","last_unstale":"2024-03-18T18:49:48.980345+0000","last_undegraded":"2024-03-18T18:49:48.980345+0000","last_fullsized":"2024-03-18T18:49:48.980345+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:43.744272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.846561+0000","last_change":"2024-03-18T18:49:44.392639+0000","last_active":"2024-03-18T18:49:48.846561+0000","last_peered":"2024-03-18T18:49:48.846561+0000","last_clean":"2024-03-18T18:49:48.846561+0000","last_became_active":"2024-03-18T18:49:44.392339+0000","last_became_peered":"2024-03-18T18:49:44.392339+0000","last_unstale":"2024-03-18T18:49:48.846561+0000","last_undegraded":"2024-03-18T18:49:48.846561+0000","last_fullsized":"2024-03-18T18:49:48.846561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:49.491669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853657+0000","last_change":"2024-03-18T18:49:44.836441+0000","last_active":"2024-03-18T18:49:48.853657+0000","last_peered":"2024-03-18T18:49:48.853657+0000","last_clean":"2024-03-18T18:49:48.853657+0000","last_became_active":"2024-03-18T18:49:44.836156+0000","last_became_peered":"2024-03-18T18:49:44.836156+0000","last_unstale":"2024-03-18T18:49:48.853657+0000","last_undegraded":"2024-03-18T18:49:48.853657+0000","last_fullsized":"2024-03-18T18:49:48.853657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:27.913188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980346+0000","last_change":"2024-03-18T18:49:48.977065+0000","last_active":"2024-03-18T18:49:48.980346+0000","last_peered":"2024-03-18T18:49:48.980346+0000","last_clean":"2024-03-18T18:49:48.980346+0000","last_became_active":"2024-03-18T18:49:48.976568+0000","last_became_peered":"2024-03-18T18:49:48.976568+0000","last_unstale":"2024-03-18T18:49:48.980346+0000","last_undegraded":"2024-03-18T18:49:48.980346+0000","last_fullsized":"2024-03-18T18:49:48.980346+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:21.533140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839322+0000","last_change":"2024-03-18T18:49:44.404269+0000","last_active":"2024-03-18T18:49:48.839322+0000","last_peered":"2024-03-18T18:49:48.839322+0000","last_clean":"2024-03-18T18:49:48.839322+0000","last_became_active":"2024-03-18T18:49:44.403992+0000","last_became_peered":"2024-03-18T18:49:44.403992+0000","last_unstale":"2024-03-18T18:49:48.839322+0000","last_undegraded":"2024-03-18T18:49:48.839322+0000","last_fullsized":"2024-03-18T18:49:48.839322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975684+0000","last_change":"2024-03-18T18:49:44.834354+0000","last_active":"2024-03-18T18:49:48.975684+0000","last_peered":"2024-03-18T18:49:48.975684+0000","last_clean":"2024-03-18T18:49:48.975684+0000","last_became_active":"2024-03-18T18:49:44.834114+0000","last_became_peered":"2024-03-18T18:49:44.834114+0000","last_unstale":"2024-03-18T18:49:48.975684+0000","last_undegraded":"2024-03-18T18:49:48.975684+0000","last_fullsized":"2024-03-18T18:49:48.975684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:09:46.686770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981775+0000","last_change":"2024-03-18T18:49:48.978466+0000","last_active":"2024-03-18T18:49:48.981775+0000","last_peered":"2024-03-18T18:49:48.981775+0000","last_clean":"2024-03-18T18:49:48.981775+0000","last_became_active":"2024-03-18T18:49:48.977877+0000","last_became_peered":"2024-03-18T18:49:48.977877+0000","last_unstale":"2024-03-18T18:49:48.981775+0000","last_undegraded":"2024-03-18T18:49:48.981775+0000","last_fullsized":"2024-03-18T18:49:48.981775+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:56.090012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836197+0000","last_change":"2024-03-18T18:49:44.393777+0000","last_active":"2024-03-18T18:49:48.836197+0000","last_peered":"2024-03-18T18:49:48.836197+0000","last_clean":"2024-03-18T18:49:48.836197+0000","last_became_active":"2024-03-18T18:49:44.392861+0000","last_became_peered":"2024-03-18T18:49:44.392861+0000","last_unstale":"2024-03-18T18:49:48.836197+0000","last_undegraded":"2024-03-18T18:49:48.836197+0000","last_fullsized":"2024-03-18T18:49:48.836197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835596+0000","last_change":"2024-03-18T18:49:44.831121+0000","last_active":"2024-03-18T18:49:48.835596+0000","last_peered":"2024-03-18T18:49:48.835596+0000","last_clean":"2024-03-18T18:49:48.835596+0000","last_became_active":"2024-03-18T18:49:44.830864+0000","last_became_peered":"2024-03-18T18:49:44.830864+0000","last_unstale":"2024-03-18T18:49:48.835596+0000","last_undegraded":"2024-03-18T18:49:48.835596+0000","last_fullsized":"2024-03-18T18:49:48.835596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:23.558014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.860292+0000","last_change":"2024-03-18T18:49:48.858618+0000","last_active":"2024-03-18T18:49:48.860292+0000","last_peered":"2024-03-18T18:49:48.860292+0000","last_clean":"2024-03-18T18:49:48.860292+0000","last_became_active":"2024-03-18T18:49:48.858170+0000","last_became_peered":"2024-03-18T18:49:48.858170+0000","last_unstale":"2024-03-18T18:49:48.860292+0000","last_undegraded":"2024-03-18T18:49:48.860292+0000","last_fullsized":"2024-03-18T18:49:48.860292+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:31.426438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836418+0000","last_change":"2024-03-18T18:49:44.397131+0000","last_active":"2024-03-18T18:49:48.836418+0000","last_peered":"2024-03-18T18:49:48.836418+0000","last_clean":"2024-03-18T18:49:48.836418+0000","last_became_active":"2024-03-18T18:49:44.396856+0000","last_became_peered":"2024-03-18T18:49:44.396856+0000","last_unstale":"2024-03-18T18:49:48.836418+0000","last_undegraded":"2024-03-18T18:49:48.836418+0000","last_fullsized":"2024-03-18T18:49:48.836418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976093+0000","last_change":"2024-03-18T18:49:44.838701+0000","last_active":"2024-03-18T18:49:48.976093+0000","last_peered":"2024-03-18T18:49:48.976093+0000","last_clean":"2024-03-18T18:49:48.976093+0000","last_became_active":"2024-03-18T18:49:44.838424+0000","last_became_peered":"2024-03-18T18:49:44.838424+0000","last_unstale":"2024-03-18T18:49:48.976093+0000","last_undegraded":"2024-03-18T18:49:48.976093+0000","last_fullsized":"2024-03-18T18:49:48.976093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:25:14.491414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853778+0000","last_change":"2024-03-18T18:49:48.852659+0000","last_active":"2024-03-18T18:49:48.853778+0000","last_peered":"2024-03-18T18:49:48.853778+0000","last_clean":"2024-03-18T18:49:48.853778+0000","last_became_active":"2024-03-18T18:49:48.852248+0000","last_became_peered":"2024-03-18T18:49:48.852248+0000","last_unstale":"2024-03-18T18:49:48.853778+0000","last_undegraded":"2024-03-18T18:49:48.853778+0000","last_fullsized":"2024-03-18T18:49:48.853778+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:38.261593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841901+0000","last_change":"2024-03-18T18:49:44.399701+0000","last_active":"2024-03-18T18:49:48.841901+0000","last_peered":"2024-03-18T18:49:48.841901+0000","last_clean":"2024-03-18T18:49:48.841901+0000","last_became_active":"2024-03-18T18:49:44.399371+0000","last_became_peered":"2024-03-18T18:49:44.399371+0000","last_unstale":"2024-03-18T18:49:48.841901+0000","last_undegraded":"2024-03-18T18:49:48.841901+0000","last_fullsized":"2024-03-18T18:49:48.841901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:49.491669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975930+0000","last_change":"2024-03-18T18:49:44.834635+0000","last_active":"2024-03-18T18:49:48.975930+0000","last_peered":"2024-03-18T18:49:48.975930+0000","last_clean":"2024-03-18T18:49:48.975930+0000","last_became_active":"2024-03-18T18:49:44.834321+0000","last_became_peered":"2024-03-18T18:49:44.834321+0000","last_unstale":"2024-03-18T18:49:48.975930+0000","last_undegraded":"2024-03-18T18:49:48.975930+0000","last_fullsized":"2024-03-18T18:49:48.975930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:50.238592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.865503+0000","last_change":"2024-03-18T18:49:48.863585+0000","last_active":"2024-03-18T18:49:48.865503+0000","last_peered":"2024-03-18T18:49:48.865503+0000","last_clean":"2024-03-18T18:49:48.865503+0000","last_became_active":"2024-03-18T18:49:48.862710+0000","last_became_peered":"2024-03-18T18:49:48.862710+0000","last_unstale":"2024-03-18T18:49:48.865503+0000","last_undegraded":"2024-03-18T18:49:48.865503+0000","last_fullsized":"2024-03-18T18:49:48.865503+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:31.426438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842830+0000","last_change":"2024-03-18T18:49:44.399251+0000","last_active":"2024-03-18T18:49:48.842830+0000","last_peered":"2024-03-18T18:49:48.842830+0000","last_clean":"2024-03-18T18:49:48.842830+0000","last_became_active":"2024-03-18T18:49:44.398969+0000","last_became_peered":"2024-03-18T18:49:44.398969+0000","last_unstale":"2024-03-18T18:49:48.842830+0000","last_undegraded":"2024-03-18T18:49:48.842830+0000","last_fullsized":"2024-03-18T18:49:48.842830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:39.899537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840553+0000","last_change":"2024-03-18T18:49:44.841934+0000","last_active":"2024-03-18T18:49:48.840553+0000","last_peered":"2024-03-18T18:49:48.840553+0000","last_clean":"2024-03-18T18:49:48.840553+0000","last_became_active":"2024-03-18T18:49:44.840657+0000","last_became_peered":"2024-03-18T18:49:44.840657+0000","last_unstale":"2024-03-18T18:49:48.840553+0000","last_undegraded":"2024-03-18T18:49:48.840553+0000","last_fullsized":"2024-03-18T18:49:48.840553+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.865367+0000","last_change":"2024-03-18T18:49:48.862242+0000","last_active":"2024-03-18T18:49:48.865367+0000","last_peered":"2024-03-18T18:49:48.865367+0000","last_clean":"2024-03-18T18:49:48.865367+0000","last_became_active":"2024-03-18T18:49:48.861779+0000","last_became_peered":"2024-03-18T18:49:48.861779+0000","last_unstale":"2024-03-18T18:49:48.865367+0000","last_undegraded":"2024-03-18T18:49:48.865367+0000","last_fullsized":"2024-03-18T18:49:48.865367+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:49:52.529190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839229+0000","last_change":"2024-03-18T18:49:44.404171+0000","last_active":"2024-03-18T18:49:48.839229+0000","last_peered":"2024-03-18T18:49:48.839229+0000","last_clean":"2024-03-18T18:49:48.839229+0000","last_became_active":"2024-03-18T18:49:44.403758+0000","last_became_peered":"2024-03-18T18:49:44.403758+0000","last_unstale":"2024-03-18T18:49:48.839229+0000","last_undegraded":"2024-03-18T18:49:48.839229+0000","last_fullsized":"2024-03-18T18:49:48.839229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:55.666762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841896+0000","last_change":"2024-03-18T18:49:44.839405+0000","last_active":"2024-03-18T18:49:48.841896+0000","last_peered":"2024-03-18T18:49:48.841896+0000","last_clean":"2024-03-18T18:49:48.841896+0000","last_became_active":"2024-03-18T18:49:44.838946+0000","last_became_peered":"2024-03-18T18:49:44.838946+0000","last_unstale":"2024-03-18T18:49:48.841896+0000","last_undegraded":"2024-03-18T18:49:48.841896+0000","last_fullsized":"2024-03-18T18:49:48.841896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:07.819299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.865474+0000","last_change":"2024-03-18T18:49:48.863506+0000","last_active":"2024-03-18T18:49:48.865474+0000","last_peered":"2024-03-18T18:49:48.865474+0000","last_clean":"2024-03-18T18:49:48.865474+0000","last_became_active":"2024-03-18T18:49:48.863081+0000","last_became_peered":"2024-03-18T18:49:48.863081+0000","last_unstale":"2024-03-18T18:49:48.865474+0000","last_undegraded":"2024-03-18T18:49:48.865474+0000","last_fullsized":"2024-03-18T18:49:48.865474+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:47.284063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839273+0000","last_change":"2024-03-18T18:49:44.833724+0000","last_active":"2024-03-18T18:49:48.839273+0000","last_peered":"2024-03-18T18:49:48.839273+0000","last_clean":"2024-03-18T18:49:48.839273+0000","last_became_active":"2024-03-18T18:49:44.833231+0000","last_became_peered":"2024-03-18T18:49:44.833231+0000","last_unstale":"2024-03-18T18:49:48.839273+0000","last_undegraded":"2024-03-18T18:49:48.839273+0000","last_fullsized":"2024-03-18T18:49:48.839273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:30.009397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836484+0000","last_change":"2024-03-18T18:49:44.402405+0000","last_active":"2024-03-18T18:49:48.836484+0000","last_peered":"2024-03-18T18:49:48.836484+0000","last_clean":"2024-03-18T18:49:48.836484+0000","last_became_active":"2024-03-18T18:49:44.402133+0000","last_became_peered":"2024-03-18T18:49:44.402133+0000","last_unstale":"2024-03-18T18:49:48.836484+0000","last_undegraded":"2024-03-18T18:49:48.836484+0000","last_fullsized":"2024-03-18T18:49:48.836484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:29.763642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981114+0000","last_change":"2024-03-18T18:49:48.978691+0000","last_active":"2024-03-18T18:49:48.981114+0000","last_peered":"2024-03-18T18:49:48.981114+0000","last_clean":"2024-03-18T18:49:48.981114+0000","last_became_active":"2024-03-18T18:49:48.977741+0000","last_became_peered":"2024-03-18T18:49:48.977741+0000","last_unstale":"2024-03-18T18:49:48.981114+0000","last_undegraded":"2024-03-18T18:49:48.981114+0000","last_fullsized":"2024-03-18T18:49:48.981114+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:07:03.479681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840954+0000","last_change":"2024-03-18T18:49:44.833327+0000","last_active":"2024-03-18T18:49:48.840954+0000","last_peered":"2024-03-18T18:49:48.840954+0000","last_clean":"2024-03-18T18:49:48.840954+0000","last_became_active":"2024-03-18T18:49:44.833073+0000","last_became_peered":"2024-03-18T18:49:44.833073+0000","last_unstale":"2024-03-18T18:49:48.840954+0000","last_undegraded":"2024-03-18T18:49:48.840954+0000","last_fullsized":"2024-03-18T18:49:48.840954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:27:55.580010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839748+0000","last_change":"2024-03-18T18:49:44.401250+0000","last_active":"2024-03-18T18:49:48.839748+0000","last_peered":"2024-03-18T18:49:48.839748+0000","last_clean":"2024-03-18T18:49:48.839748+0000","last_became_active":"2024-03-18T18:49:44.398256+0000","last_became_peered":"2024-03-18T18:49:44.398256+0000","last_unstale":"2024-03-18T18:49:48.839748+0000","last_undegraded":"2024-03-18T18:49:48.839748+0000","last_fullsized":"2024-03-18T18:49:48.839748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:06.816608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":167,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975479+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T18:49:48.975479+0000","last_peered":"2024-03-18T18:49:48.975479+0000","last_clean":"2024-03-18T18:49:48.975479+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T18:49:48.975479+0000","last_undegraded":"2024-03-18T18:49:48.975479+0000","last_fullsized":"2024-03-18T18:49:48.975479+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980453+0000","last_change":"2024-03-18T18:49:48.978188+0000","last_active":"2024-03-18T18:49:48.980453+0000","last_peered":"2024-03-18T18:49:48.980453+0000","last_clean":"2024-03-18T18:49:48.980453+0000","last_became_active":"2024-03-18T18:49:48.977852+0000","last_became_peered":"2024-03-18T18:49:48.977852+0000","last_unstale":"2024-03-18T18:49:48.980453+0000","last_undegraded":"2024-03-18T18:49:48.980453+0000","last_fullsized":"2024-03-18T18:49:48.980453+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:38.261593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.848312+0000","last_change":"2024-03-18T18:49:44.839127+0000","last_active":"2024-03-18T18:49:48.848312+0000","last_peered":"2024-03-18T18:49:48.848312+0000","last_clean":"2024-03-18T18:49:48.848312+0000","last_became_active":"2024-03-18T18:49:44.838948+0000","last_became_peered":"2024-03-18T18:49:44.838948+0000","last_unstale":"2024-03-18T18:49:48.848312+0000","last_undegraded":"2024-03-18T18:49:48.848312+0000","last_fullsized":"2024-03-18T18:49:48.848312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:23.558014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975591+0000","last_change":"2024-03-18T18:49:44.402282+0000","last_active":"2024-03-18T18:49:48.975591+0000","last_peered":"2024-03-18T18:49:48.975591+0000","last_clean":"2024-03-18T18:49:48.975591+0000","last_became_active":"2024-03-18T18:49:44.402003+0000","last_became_peered":"2024-03-18T18:49:44.402003+0000","last_unstale":"2024-03-18T18:49:48.975591+0000","last_undegraded":"2024-03-18T18:49:48.975591+0000","last_fullsized":"2024-03-18T18:49:48.975591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:28.827068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981915+0000","last_change":"2024-03-18T18:49:48.978958+0000","last_active":"2024-03-18T18:49:48.981915+0000","last_peered":"2024-03-18T18:49:48.981915+0000","last_clean":"2024-03-18T18:49:48.981915+0000","last_became_active":"2024-03-18T18:49:48.978608+0000","last_became_peered":"2024-03-18T18:49:48.978608+0000","last_unstale":"2024-03-18T18:49:48.981915+0000","last_undegraded":"2024-03-18T18:49:48.981915+0000","last_fullsized":"2024-03-18T18:49:48.981915+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:08:46.162308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975789+0000","last_change":"2024-03-18T18:49:44.397512+0000","last_active":"2024-03-18T18:49:48.975789+0000","last_peered":"2024-03-18T18:49:48.975789+0000","last_clean":"2024-03-18T18:49:48.975789+0000","last_became_active":"2024-03-18T18:49:44.397271+0000","last_became_peered":"2024-03-18T18:49:44.397271+0000","last_unstale":"2024-03-18T18:49:48.975789+0000","last_undegraded":"2024-03-18T18:49:48.975789+0000","last_fullsized":"2024-03-18T18:49:48.975789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:55.666762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842833+0000","last_change":"2024-03-18T18:49:44.838715+0000","last_active":"2024-03-18T18:49:48.842833+0000","last_peered":"2024-03-18T18:49:48.842833+0000","last_clean":"2024-03-18T18:49:48.842833+0000","last_became_active":"2024-03-18T18:49:44.838452+0000","last_became_peered":"2024-03-18T18:49:44.838452+0000","last_unstale":"2024-03-18T18:49:48.842833+0000","last_undegraded":"2024-03-18T18:49:48.842833+0000","last_fullsized":"2024-03-18T18:49:48.842833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:23.558014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981853+0000","last_change":"2024-03-18T18:49:48.978525+0000","last_active":"2024-03-18T18:49:48.981853+0000","last_peered":"2024-03-18T18:49:48.981853+0000","last_clean":"2024-03-18T18:49:48.981853+0000","last_became_active":"2024-03-18T18:49:48.977651+0000","last_became_peered":"2024-03-18T18:49:48.977651+0000","last_unstale":"2024-03-18T18:49:48.981853+0000","last_undegraded":"2024-03-18T18:49:48.981853+0000","last_fullsized":"2024-03-18T18:49:48.981853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:31:53.057656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841616+0000","last_change":"2024-03-18T18:49:44.397630+0000","last_active":"2024-03-18T18:49:48.841616+0000","last_peered":"2024-03-18T18:49:48.841616+0000","last_clean":"2024-03-18T18:49:48.841616+0000","last_became_active":"2024-03-18T18:49:44.396306+0000","last_became_peered":"2024-03-18T18:49:44.396306+0000","last_unstale":"2024-03-18T18:49:48.841616+0000","last_undegraded":"2024-03-18T18:49:48.841616+0000","last_fullsized":"2024-03-18T18:49:48.841616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:10:24.082597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842664+0000","last_change":"2024-03-18T18:49:44.821273+0000","last_active":"2024-03-18T18:49:48.842664+0000","last_peered":"2024-03-18T18:49:48.842664+0000","last_clean":"2024-03-18T18:49:48.842664+0000","last_became_active":"2024-03-18T18:49:44.821036+0000","last_became_peered":"2024-03-18T18:49:44.821036+0000","last_unstale":"2024-03-18T18:49:48.842664+0000","last_undegraded":"2024-03-18T18:49:48.842664+0000","last_fullsized":"2024-03-18T18:49:48.842664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:30.009397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.982011+0000","last_change":"2024-03-18T18:49:48.978130+0000","last_active":"2024-03-18T18:49:48.982011+0000","last_peered":"2024-03-18T18:49:48.982011+0000","last_clean":"2024-03-18T18:49:48.982011+0000","last_became_active":"2024-03-18T18:49:48.977715+0000","last_became_peered":"2024-03-18T18:49:48.977715+0000","last_unstale":"2024-03-18T18:49:48.982011+0000","last_undegraded":"2024-03-18T18:49:48.982011+0000","last_fullsized":"2024-03-18T18:49:48.982011+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:47.284063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836478+0000","last_change":"2024-03-18T18:49:44.394139+0000","last_active":"2024-03-18T18:49:48.836478+0000","last_peered":"2024-03-18T18:49:48.836478+0000","last_clean":"2024-03-18T18:49:48.836478+0000","last_became_active":"2024-03-18T18:49:44.393405+0000","last_became_peered":"2024-03-18T18:49:44.393405+0000","last_unstale":"2024-03-18T18:49:48.836478+0000","last_undegraded":"2024-03-18T18:49:48.836478+0000","last_fullsized":"2024-03-18T18:49:48.836478+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:55.666762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853791+0000","last_change":"2024-03-18T18:49:44.829271+0000","last_active":"2024-03-18T18:49:48.853791+0000","last_peered":"2024-03-18T18:49:48.853791+0000","last_clean":"2024-03-18T18:49:48.853791+0000","last_became_active":"2024-03-18T18:49:44.829019+0000","last_became_peered":"2024-03-18T18:49:44.829019+0000","last_unstale":"2024-03-18T18:49:48.853791+0000","last_undegraded":"2024-03-18T18:49:48.853791+0000","last_fullsized":"2024-03-18T18:49:48.853791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853721+0000","last_change":"2024-03-18T18:49:48.852590+0000","last_active":"2024-03-18T18:49:48.853721+0000","last_peered":"2024-03-18T18:49:48.853721+0000","last_clean":"2024-03-18T18:49:48.853721+0000","last_became_active":"2024-03-18T18:49:48.852088+0000","last_became_peered":"2024-03-18T18:49:48.852088+0000","last_unstale":"2024-03-18T18:49:48.853721+0000","last_undegraded":"2024-03-18T18:49:48.853721+0000","last_fullsized":"2024-03-18T18:49:48.853721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:16:08.575913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975464+0000","last_change":"2024-03-18T18:49:44.393305+0000","last_active":"2024-03-18T18:49:48.975464+0000","last_peered":"2024-03-18T18:49:48.975464+0000","last_clean":"2024-03-18T18:49:48.975464+0000","last_became_active":"2024-03-18T18:49:44.392494+0000","last_became_peered":"2024-03-18T18:49:44.392494+0000","last_unstale":"2024-03-18T18:49:48.975464+0000","last_undegraded":"2024-03-18T18:49:48.975464+0000","last_fullsized":"2024-03-18T18:49:48.975464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:06.816608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841483+0000","last_change":"2024-03-18T18:49:44.837726+0000","last_active":"2024-03-18T18:49:48.841483+0000","last_peered":"2024-03-18T18:49:48.841483+0000","last_clean":"2024-03-18T18:49:48.841483+0000","last_became_active":"2024-03-18T18:49:44.837364+0000","last_became_peered":"2024-03-18T18:49:44.837364+0000","last_unstale":"2024-03-18T18:49:48.841483+0000","last_undegraded":"2024-03-18T18:49:48.841483+0000","last_fullsized":"2024-03-18T18:49:48.841483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:40.902228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.982312+0000","last_change":"2024-03-18T18:49:48.980106+0000","last_active":"2024-03-18T18:49:48.982312+0000","last_peered":"2024-03-18T18:49:48.982312+0000","last_clean":"2024-03-18T18:49:48.982312+0000","last_became_active":"2024-03-18T18:49:48.979709+0000","last_became_peered":"2024-03-18T18:49:48.979709+0000","last_unstale":"2024-03-18T18:49:48.982312+0000","last_undegraded":"2024-03-18T18:49:48.982312+0000","last_fullsized":"2024-03-18T18:49:48.982312+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:21.533140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840674+0000","last_change":"2024-03-18T18:49:44.398886+0000","last_active":"2024-03-18T18:49:48.840674+0000","last_peered":"2024-03-18T18:49:48.840674+0000","last_clean":"2024-03-18T18:49:48.840674+0000","last_became_active":"2024-03-18T18:49:44.398578+0000","last_became_peered":"2024-03-18T18:49:44.398578+0000","last_unstale":"2024-03-18T18:49:48.840674+0000","last_undegraded":"2024-03-18T18:49:48.840674+0000","last_fullsized":"2024-03-18T18:49:48.840674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839766+0000","last_change":"2024-03-18T18:49:44.826751+0000","last_active":"2024-03-18T18:49:48.839766+0000","last_peered":"2024-03-18T18:49:48.839766+0000","last_clean":"2024-03-18T18:49:48.839766+0000","last_became_active":"2024-03-18T18:49:44.825975+0000","last_became_peered":"2024-03-18T18:49:44.825975+0000","last_unstale":"2024-03-18T18:49:48.839766+0000","last_undegraded":"2024-03-18T18:49:48.839766+0000","last_fullsized":"2024-03-18T18:49:48.839766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:02:06.725321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853727+0000","last_change":"2024-03-18T18:49:48.852554+0000","last_active":"2024-03-18T18:49:48.853727+0000","last_peered":"2024-03-18T18:49:48.853727+0000","last_clean":"2024-03-18T18:49:48.853727+0000","last_became_active":"2024-03-18T18:49:48.852121+0000","last_became_peered":"2024-03-18T18:49:48.852121+0000","last_unstale":"2024-03-18T18:49:48.853727+0000","last_undegraded":"2024-03-18T18:49:48.853727+0000","last_fullsized":"2024-03-18T18:49:48.853727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:41:26.757218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841024+0000","last_change":"2024-03-18T18:49:44.402526+0000","last_active":"2024-03-18T18:49:48.841024+0000","last_peered":"2024-03-18T18:49:48.841024+0000","last_clean":"2024-03-18T18:49:48.841024+0000","last_became_active":"2024-03-18T18:49:44.402277+0000","last_became_peered":"2024-03-18T18:49:44.402277+0000","last_unstale":"2024-03-18T18:49:48.841024+0000","last_undegraded":"2024-03-18T18:49:48.841024+0000","last_fullsized":"2024-03-18T18:49:48.841024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:06.816608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839383+0000","last_change":"2024-03-18T18:49:44.830629+0000","last_active":"2024-03-18T18:49:48.839383+0000","last_peered":"2024-03-18T18:49:48.839383+0000","last_clean":"2024-03-18T18:49:48.839383+0000","last_became_active":"2024-03-18T18:49:44.830313+0000","last_became_peered":"2024-03-18T18:49:44.830313+0000","last_unstale":"2024-03-18T18:49:48.839383+0000","last_undegraded":"2024-03-18T18:49:48.839383+0000","last_fullsized":"2024-03-18T18:49:48.839383+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:27.913188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.860287+0000","last_change":"2024-03-18T18:49:48.859810+0000","last_active":"2024-03-18T18:49:48.860287+0000","last_peered":"2024-03-18T18:49:48.860287+0000","last_clean":"2024-03-18T18:49:48.860287+0000","last_became_active":"2024-03-18T18:49:48.859453+0000","last_became_peered":"2024-03-18T18:49:48.859453+0000","last_unstale":"2024-03-18T18:49:48.860287+0000","last_undegraded":"2024-03-18T18:49:48.860287+0000","last_fullsized":"2024-03-18T18:49:48.860287+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:11:09.238467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975758+0000","last_change":"2024-03-18T18:49:44.396856+0000","last_active":"2024-03-18T18:49:48.975758+0000","last_peered":"2024-03-18T18:49:48.975758+0000","last_clean":"2024-03-18T18:49:48.975758+0000","last_became_active":"2024-03-18T18:49:44.395207+0000","last_became_peered":"2024-03-18T18:49:44.395207+0000","last_unstale":"2024-03-18T18:49:48.975758+0000","last_undegraded":"2024-03-18T18:49:48.975758+0000","last_fullsized":"2024-03-18T18:49:48.975758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:39.899537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836368+0000","last_change":"2024-03-18T18:49:44.836659+0000","last_active":"2024-03-18T18:49:48.836368+0000","last_peered":"2024-03-18T18:49:48.836368+0000","last_clean":"2024-03-18T18:49:48.836368+0000","last_became_active":"2024-03-18T18:49:44.836362+0000","last_became_peered":"2024-03-18T18:49:44.836362+0000","last_unstale":"2024-03-18T18:49:48.836368+0000","last_undegraded":"2024-03-18T18:49:48.836368+0000","last_fullsized":"2024-03-18T18:49:48.836368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:49.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980755+0000","last_change":"2024-03-18T18:49:48.978204+0000","last_active":"2024-03-18T18:49:48.980755+0000","last_peered":"2024-03-18T18:49:48.980755+0000","last_clean":"2024-03-18T18:49:48.980755+0000","last_became_active":"2024-03-18T18:49:48.977745+0000","last_became_peered":"2024-03-18T18:49:48.977745+0000","last_unstale":"2024-03-18T18:49:48.980755+0000","last_undegraded":"2024-03-18T18:49:48.980755+0000","last_fullsized":"2024-03-18T18:49:48.980755+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:08:46.162308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.837352+0000","last_change":"2024-03-18T18:49:44.398731+0000","last_active":"2024-03-18T18:49:48.837352+0000","last_peered":"2024-03-18T18:49:48.837352+0000","last_clean":"2024-03-18T18:49:48.837352+0000","last_became_active":"2024-03-18T18:49:44.398448+0000","last_became_peered":"2024-03-18T18:49:44.398448+0000","last_unstale":"2024-03-18T18:49:48.837352+0000","last_undegraded":"2024-03-18T18:49:48.837352+0000","last_fullsized":"2024-03-18T18:49:48.837352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:30.124822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976056+0000","last_change":"2024-03-18T18:49:44.833300+0000","last_active":"2024-03-18T18:49:48.976056+0000","last_peered":"2024-03-18T18:49:48.976056+0000","last_clean":"2024-03-18T18:49:48.976056+0000","last_became_active":"2024-03-18T18:49:44.833027+0000","last_became_peered":"2024-03-18T18:49:44.833027+0000","last_unstale":"2024-03-18T18:49:48.976056+0000","last_undegraded":"2024-03-18T18:49:48.976056+0000","last_fullsized":"2024-03-18T18:49:48.976056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:11:05.787600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.860344+0000","last_change":"2024-03-18T18:49:48.859259+0000","last_active":"2024-03-18T18:49:48.860344+0000","last_peered":"2024-03-18T18:49:48.860344+0000","last_clean":"2024-03-18T18:49:48.860344+0000","last_became_active":"2024-03-18T18:49:48.857100+0000","last_became_peered":"2024-03-18T18:49:48.857100+0000","last_unstale":"2024-03-18T18:49:48.860344+0000","last_undegraded":"2024-03-18T18:49:48.860344+0000","last_fullsized":"2024-03-18T18:49:48.860344+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:47.284063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841409+0000","last_change":"2024-03-18T18:49:44.395491+0000","last_active":"2024-03-18T18:49:48.841409+0000","last_peered":"2024-03-18T18:49:48.841409+0000","last_clean":"2024-03-18T18:49:48.841409+0000","last_became_active":"2024-03-18T18:49:44.394529+0000","last_became_peered":"2024-03-18T18:49:44.394529+0000","last_unstale":"2024-03-18T18:49:48.841409+0000","last_undegraded":"2024-03-18T18:49:48.841409+0000","last_fullsized":"2024-03-18T18:49:48.841409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:55.666762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839957+0000","last_change":"2024-03-18T18:49:44.838466+0000","last_active":"2024-03-18T18:49:48.839957+0000","last_peered":"2024-03-18T18:49:48.839957+0000","last_clean":"2024-03-18T18:49:48.839957+0000","last_became_active":"2024-03-18T18:49:44.838027+0000","last_became_peered":"2024-03-18T18:49:44.838027+0000","last_unstale":"2024-03-18T18:49:48.839957+0000","last_undegraded":"2024-03-18T18:49:48.839957+0000","last_fullsized":"2024-03-18T18:49:48.839957+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:09:46.686770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981225+0000","last_change":"2024-03-18T18:49:48.979431+0000","last_active":"2024-03-18T18:49:48.981225+0000","last_peered":"2024-03-18T18:49:48.981225+0000","last_clean":"2024-03-18T18:49:48.981225+0000","last_became_active":"2024-03-18T18:49:48.978587+0000","last_became_peered":"2024-03-18T18:49:48.978587+0000","last_unstale":"2024-03-18T18:49:48.981225+0000","last_undegraded":"2024-03-18T18:49:48.981225+0000","last_fullsized":"2024-03-18T18:49:48.981225+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:47.284063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836616+0000","last_change":"2024-03-18T18:49:44.403374+0000","last_active":"2024-03-18T18:49:48.836616+0000","last_peered":"2024-03-18T18:49:48.836616+0000","last_clean":"2024-03-18T18:49:48.836616+0000","last_became_active":"2024-03-18T18:49:44.403082+0000","last_became_peered":"2024-03-18T18:49:44.403082+0000","last_unstale":"2024-03-18T18:49:48.836616+0000","last_undegraded":"2024-03-18T18:49:48.836616+0000","last_fullsized":"2024-03-18T18:49:48.836616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:28.338064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976741+0000","last_change":"2024-03-18T18:49:44.835166+0000","last_active":"2024-03-18T18:49:48.976741+0000","last_peered":"2024-03-18T18:49:48.976741+0000","last_clean":"2024-03-18T18:49:48.976741+0000","last_became_active":"2024-03-18T18:49:44.834808+0000","last_became_peered":"2024-03-18T18:49:44.834808+0000","last_unstale":"2024-03-18T18:49:48.976741+0000","last_undegraded":"2024-03-18T18:49:48.976741+0000","last_fullsized":"2024-03-18T18:49:48.976741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:20.816344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835598+0000","last_change":"2024-03-18T18:49:44.829141+0000","last_active":"2024-03-18T18:49:48.835598+0000","last_peered":"2024-03-18T18:49:48.835598+0000","last_clean":"2024-03-18T18:49:48.835598+0000","last_became_active":"2024-03-18T18:49:44.828863+0000","last_became_peered":"2024-03-18T18:49:44.828863+0000","last_unstale":"2024-03-18T18:49:48.835598+0000","last_undegraded":"2024-03-18T18:49:48.835598+0000","last_fullsized":"2024-03-18T18:49:48.835598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:30.009397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836583+0000","last_change":"2024-03-18T18:49:44.831132+0000","last_active":"2024-03-18T18:49:48.836583+0000","last_peered":"2024-03-18T18:49:48.836583+0000","last_clean":"2024-03-18T18:49:48.836583+0000","last_became_active":"2024-03-18T18:49:44.830873+0000","last_became_peered":"2024-03-18T18:49:44.830873+0000","last_unstale":"2024-03-18T18:49:48.836583+0000","last_undegraded":"2024-03-18T18:49:48.836583+0000","last_fullsized":"2024-03-18T18:49:48.836583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:09:08.248420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839404+0000","last_change":"2024-03-18T18:49:44.841166+0000","last_active":"2024-03-18T18:49:48.839404+0000","last_peered":"2024-03-18T18:49:48.839404+0000","last_clean":"2024-03-18T18:49:48.839404+0000","last_became_active":"2024-03-18T18:49:44.840477+0000","last_became_peered":"2024-03-18T18:49:44.840477+0000","last_unstale":"2024-03-18T18:49:48.839404+0000","last_undegraded":"2024-03-18T18:49:48.839404+0000","last_fullsized":"2024-03-18T18:49:48.839404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:20.816344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.846309+0000","last_change":"2024-03-18T18:49:44.838132+0000","last_active":"2024-03-18T18:49:48.846309+0000","last_peered":"2024-03-18T18:49:48.846309+0000","last_clean":"2024-03-18T18:49:48.846309+0000","last_became_active":"2024-03-18T18:49:44.837914+0000","last_became_peered":"2024-03-18T18:49:44.837914+0000","last_unstale":"2024-03-18T18:49:48.846309+0000","last_undegraded":"2024-03-18T18:49:48.846309+0000","last_fullsized":"2024-03-18T18:49:48.846309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840730+0000","last_change":"2024-03-18T18:49:44.835464+0000","last_active":"2024-03-18T18:49:48.840730+0000","last_peered":"2024-03-18T18:49:48.840730+0000","last_clean":"2024-03-18T18:49:48.840730+0000","last_became_active":"2024-03-18T18:49:44.835190+0000","last_became_peered":"2024-03-18T18:49:44.835190+0000","last_unstale":"2024-03-18T18:49:48.840730+0000","last_undegraded":"2024-03-18T18:49:48.840730+0000","last_fullsized":"2024-03-18T18:49:48.840730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841309+0000","last_change":"2024-03-18T18:49:44.819962+0000","last_active":"2024-03-18T18:49:48.841309+0000","last_peered":"2024-03-18T18:49:48.841309+0000","last_clean":"2024-03-18T18:49:48.841309+0000","last_became_active":"2024-03-18T18:49:44.819698+0000","last_became_peered":"2024-03-18T18:49:44.819698+0000","last_unstale":"2024-03-18T18:49:48.841309+0000","last_undegraded":"2024-03-18T18:49:48.841309+0000","last_fullsized":"2024-03-18T18:49:48.841309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835920+0000","last_change":"2024-03-18T18:49:44.830928+0000","last_active":"2024-03-18T18:49:48.835920+0000","last_peered":"2024-03-18T18:49:48.835920+0000","last_clean":"2024-03-18T18:49:48.835920+0000","last_became_active":"2024-03-18T18:49:44.830658+0000","last_became_peered":"2024-03-18T18:49:44.830658+0000","last_unstale":"2024-03-18T18:49:48.835920+0000","last_undegraded":"2024-03-18T18:49:48.835920+0000","last_fullsized":"2024-03-18T18:49:48.835920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842763+0000","last_change":"2024-03-18T18:49:44.838837+0000","last_active":"2024-03-18T18:49:48.842763+0000","last_peered":"2024-03-18T18:49:48.842763+0000","last_clean":"2024-03-18T18:49:48.842763+0000","last_became_active":"2024-03-18T18:49:44.838564+0000","last_became_peered":"2024-03-18T18:49:44.838564+0000","last_unstale":"2024-03-18T18:49:48.842763+0000","last_undegraded":"2024-03-18T18:49:48.842763+0000","last_fullsized":"2024-03-18T18:49:48.842763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:34.102274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839856+0000","last_change":"2024-03-18T18:49:44.841576+0000","last_active":"2024-03-18T18:49:48.839856+0000","last_peered":"2024-03-18T18:49:48.839856+0000","last_clean":"2024-03-18T18:49:48.839856+0000","last_became_active":"2024-03-18T18:49:44.840814+0000","last_became_peered":"2024-03-18T18:49:44.840814+0000","last_unstale":"2024-03-18T18:49:48.839856+0000","last_undegraded":"2024-03-18T18:49:48.839856+0000","last_fullsized":"2024-03-18T18:49:48.839856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:02:06.725321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840423+0000","last_change":"2024-03-18T18:49:44.840212+0000","last_active":"2024-03-18T18:49:48.840423+0000","last_peered":"2024-03-18T18:49:48.840423+0000","last_clean":"2024-03-18T18:49:48.840423+0000","last_became_active":"2024-03-18T18:49:44.839960+0000","last_became_peered":"2024-03-18T18:49:44.839960+0000","last_unstale":"2024-03-18T18:49:48.840423+0000","last_undegraded":"2024-03-18T18:49:48.840423+0000","last_fullsized":"2024-03-18T18:49:48.840423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839485+0000","last_change":"2024-03-18T18:49:44.841770+0000","last_active":"2024-03-18T18:49:48.839485+0000","last_peered":"2024-03-18T18:49:48.839485+0000","last_clean":"2024-03-18T18:49:48.839485+0000","last_became_active":"2024-03-18T18:49:44.840939+0000","last_became_peered":"2024-03-18T18:49:44.840939+0000","last_unstale":"2024-03-18T18:49:48.839485+0000","last_undegraded":"2024-03-18T18:49:48.839485+0000","last_fullsized":"2024-03-18T18:49:48.839485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:20:43.395005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.846397+0000","last_change":"2024-03-18T18:49:44.841290+0000","last_active":"2024-03-18T18:49:48.846397+0000","last_peered":"2024-03-18T18:49:48.846397+0000","last_clean":"2024-03-18T18:49:48.846397+0000","last_became_active":"2024-03-18T18:49:44.840949+0000","last_became_peered":"2024-03-18T18:49:44.840949+0000","last_unstale":"2024-03-18T18:49:48.846397+0000","last_undegraded":"2024-03-18T18:49:48.846397+0000","last_fullsized":"2024-03-18T18:49:48.846397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:05.661348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835876+0000","last_change":"2024-03-18T18:49:44.837671+0000","last_active":"2024-03-18T18:49:48.835876+0000","last_peered":"2024-03-18T18:49:48.835876+0000","last_clean":"2024-03-18T18:49:48.835876+0000","last_became_active":"2024-03-18T18:49:44.837265+0000","last_became_peered":"2024-03-18T18:49:44.837265+0000","last_unstale":"2024-03-18T18:49:48.835876+0000","last_undegraded":"2024-03-18T18:49:48.835876+0000","last_fullsized":"2024-03-18T18:49:48.835876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:14.055199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4593,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":695401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":104611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0760000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.999}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":695457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":104451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":104577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":104541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T18:50:09.045 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T18:50:09.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.207+0000 7fe1fe277700 1 Processor -- start 2024-03-18T18:50:09.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.207+0000 7fe1fe277700 1 -- start start 2024-03-18T18:50:09.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.208+0000 7fe1fe277700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe1f806e290 0x7fe1f8072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:09.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.208+0000 7fe1fe277700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f806f490 con 0x7fe1f806dd50 2024-03-18T18:50:09.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.208+0000 7fe1fe277700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f8071cf0 con 0x7fe1f815d580 2024-03-18T18:50:09.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.208+0000 7fe1fe277700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f806f1b0 con 0x7fe1f806e290 2024-03-18T18:50:09.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.208+0000 7fe1f77fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe1f806e290 0x7fe1f8072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:09.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.208+0000 7fe1fc814700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe1f806dd50 0x7fe1f80ca560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60212/0 (socket says 172.21.15.1:60212) 2024-03-18T18:50:09.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.208+0000 7fe1fc814700 1 -- 172.21.15.1:0/3664577266 learned_addr learned my addr 172.21.15.1:0/3664577266 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:09.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.208+0000 7fe1f77fe700 1 -- 172.21.15.1:0/3664577266 >> v1:172.21.15.165:6789/0 conn(0x7fe1f815d580 legacy=0x7fe1f806d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:09.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.208+0000 7fe1f77fe700 1 -- 172.21.15.1:0/3664577266 >> v1:172.21.15.1:6789/0 conn(0x7fe1f806dd50 legacy=0x7fe1f80ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:09.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.208+0000 7fe1f6ffd700 1 -- 172.21.15.1:0/3664577266 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1289306861 0 0) 0x7fe1f806f490 con 0x7fe1f806dd50 2024-03-18T18:50:09.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.208+0000 7fe1f77fe700 1 -- 172.21.15.1:0/3664577266 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f80716a0 con 0x7fe1f806e290 2024-03-18T18:50:09.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.209+0000 7fe1f77fe700 1 --2- 172.21.15.1:0/3664577266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f806e290 0x7fe1f8072180 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0009840 tx=0x7fe1e0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a069b2f67538d2c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:09.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.209+0000 7fe1f6ffd700 1 -- 172.21.15.1:0/3664577266 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e0011830 con 0x7fe1f806e290 2024-03-18T18:50:09.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.209+0000 7fe1f6ffd700 1 -- 172.21.15.1:0/3664577266 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e0011990 con 0x7fe1f806e290 2024-03-18T18:50:09.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.209+0000 7fe1f6ffd700 1 -- 172.21.15.1:0/3664577266 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e0011d20 con 0x7fe1f806e290 2024-03-18T18:50:09.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.209+0000 7fe1fe277700 1 -- 172.21.15.1:0/3664577266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f806e290 msgr2=0x7fe1f8072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:09.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.209+0000 7fe1fe277700 1 --2- 172.21.15.1:0/3664577266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f806e290 0x7fe1f8072180 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0009840 tx=0x7fe1e0009b50 comp rx=0 tx=0).stop 2024-03-18T18:50:09.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.209+0000 7fe1fe277700 1 -- 172.21.15.1:0/3664577266 shutdown_connections 2024-03-18T18:50:09.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.209+0000 7fe1fe277700 1 --2- 172.21.15.1:0/3664577266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f806e290 0x7fe1f8072180 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.209+0000 7fe1fe277700 1 -- 172.21.15.1:0/3664577266 >> 172.21.15.1:0/3664577266 conn(0x7fe1f80cc990 msgr2=0x7fe1f80ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:09.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.209+0000 7fe1fe277700 1 -- 172.21.15.1:0/3664577266 shutdown_connections 2024-03-18T18:50:09.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.209+0000 7fe1fe277700 1 -- 172.21.15.1:0/3664577266 wait complete. 2024-03-18T18:50:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.211+0000 7fe1fe277700 1 Processor -- start 2024-03-18T18:50:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.211+0000 7fe1fe277700 1 -- start start 2024-03-18T18:50:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.212+0000 7fe1fe277700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f806dd50 0x7fe1f80b5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.212+0000 7fe1fe277700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1f815d580 0x7fe1f8158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.212+0000 7fe1fe277700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1f80c6fe0 0x7fe1f80adfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.212+0000 7fe1fe277700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f815e0e0 con 0x7fe1f806dd50 2024-03-18T18:50:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.212+0000 7fe1fe277700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f80b4730 con 0x7fe1f80c6fe0 2024-03-18T18:50:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.212+0000 7fe1fe277700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f80b4450 con 0x7fe1f815d580 2024-03-18T18:50:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.212+0000 7fe1f7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f806dd50 0x7fe1f80b5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.212+0000 7fe1f7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f806dd50 0x7fe1f80b5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45104/0 (socket says 172.21.15.1:45104) 2024-03-18T18:50:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.212+0000 7fe1f7fff700 1 -- 172.21.15.1:0/3211103802 learned_addr learned my addr 172.21.15.1:0/3211103802 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.212+0000 7fe1fc814700 1 --2- 172.21.15.1:0/3211103802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1f80c6fe0 0x7fe1f80adfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.212+0000 7fe1f77fe700 1 --2- 172.21.15.1:0/3211103802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1f815d580 0x7fe1f8158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.212+0000 7fe1f7fff700 1 -- 172.21.15.1:0/3211103802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1f815d580 msgr2=0x7fe1f8158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.212+0000 7fe1f7fff700 1 --2- 172.21.15.1:0/3211103802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1f815d580 0x7fe1f8158850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.212+0000 7fe1f7fff700 1 -- 172.21.15.1:0/3211103802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1f80c6fe0 msgr2=0x7fe1f80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.212+0000 7fe1f7fff700 1 --2- 172.21.15.1:0/3211103802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1f80c6fe0 0x7fe1f80adfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.212+0000 7fe1f7fff700 1 -- 172.21.15.1:0/3211103802 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e00094a0 con 0x7fe1f806dd50 2024-03-18T18:50:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.212+0000 7fe1fc814700 1 --2- 172.21.15.1:0/3211103802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1f80c6fe0 0x7fe1f80adfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.213+0000 7fe1f7fff700 1 --2- 172.21.15.1:0/3211103802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f806dd50 0x7fe1f80b5410 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800c450 tx=0x7fe1e800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.213+0000 7fe1f4ff9700 1 -- 172.21.15.1:0/3211103802 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e80052d0 con 0x7fe1f806dd50 2024-03-18T18:50:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.213+0000 7fe1f4ff9700 1 -- 172.21.15.1:0/3211103802 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e8005430 con 0x7fe1f806dd50 2024-03-18T18:50:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.213+0000 7fe1fe277700 1 -- 172.21.15.1:0/3211103802 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f815da90 con 0x7fe1f806dd50 2024-03-18T18:50:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.213+0000 7fe1f4ff9700 1 -- 172.21.15.1:0/3211103802 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e80038c0 con 0x7fe1f806dd50 2024-03-18T18:50:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.213+0000 7fe1fe277700 1 -- 172.21.15.1:0/3211103802 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f8158d90 con 0x7fe1f806dd50 2024-03-18T18:50:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.214+0000 7fe1fe277700 1 -- 172.21.15.1:0/3211103802 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f80676a0 con 0x7fe1f806dd50 2024-03-18T18:50:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.214+0000 7fe1f4ff9700 1 -- 172.21.15.1:0/3211103802 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe1e8003a20 con 0x7fe1f806dd50 2024-03-18T18:50:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.215+0000 7fe1f4ff9700 1 --2- 172.21.15.1:0/3211103802 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1d806ac90 0x7fe1d806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:09.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.215+0000 7fe1f77fe700 1 --2- 172.21.15.1:0/3211103802 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1d806ac90 0x7fe1d806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:09.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.215+0000 7fe1f4ff9700 1 -- 172.21.15.1:0/3211103802 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe1e8088930 con 0x7fe1f806dd50 2024-03-18T18:50:09.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.215+0000 7fe1f77fe700 1 --2- 172.21.15.1:0/3211103802 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1d806ac90 0x7fe1d806d150 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe1f809de20 tx=0x7fe1e001c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:09.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.219+0000 7fe1f4ff9700 1 -- 172.21.15.1:0/3211103802 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe1e80538b0 con 0x7fe1f806dd50 2024-03-18T18:50:09.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.417+0000 7fe1fe277700 1 -- 172.21.15.1:0/3211103802 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe1f806ee40 con 0x7fe1d806ac90 2024-03-18T18:50:09.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.427+0000 7fe1f4ff9700 1 -- 172.21.15.1:0/3211103802 <== mgr.4104 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345203 (secure 0 0 0) 0x7fe1e0011b80 con 0x7fe1d806ac90 2024-03-18T18:50:09.427 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:09.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.430+0000 7fe1fe277700 1 -- 172.21.15.1:0/3211103802 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1d806ac90 msgr2=0x7fe1d806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:09.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.430+0000 7fe1fe277700 1 --2- 172.21.15.1:0/3211103802 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1d806ac90 0x7fe1d806d150 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe1f809de20 tx=0x7fe1e001c040 comp rx=0 tx=0).stop 2024-03-18T18:50:09.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.430+0000 7fe1fe277700 1 -- 172.21.15.1:0/3211103802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f806dd50 msgr2=0x7fe1f80b5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:09.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.430+0000 7fe1fe277700 1 --2- 172.21.15.1:0/3211103802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f806dd50 0x7fe1f80b5410 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800c450 tx=0x7fe1e800c810 comp rx=0 tx=0).stop 2024-03-18T18:50:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.430+0000 7fe1fe277700 1 -- 172.21.15.1:0/3211103802 shutdown_connections 2024-03-18T18:50:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.430+0000 7fe1fe277700 1 --2- 172.21.15.1:0/3211103802 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1d806ac90 0x7fe1d806d150 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.430+0000 7fe1fe277700 1 --2- 172.21.15.1:0/3211103802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1f80c6fe0 0x7fe1f80adfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.430+0000 7fe1fe277700 1 --2- 172.21.15.1:0/3211103802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f806dd50 0x7fe1f80b5410 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.430+0000 7fe1fe277700 1 --2- 172.21.15.1:0/3211103802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1f815d580 0x7fe1f8158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.430+0000 7fe1fe277700 1 -- 172.21.15.1:0/3211103802 >> 172.21.15.1:0/3211103802 conn(0x7fe1f80cc990 msgr2=0x7fe1f806a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.430+0000 7fe1fe277700 1 -- 172.21.15.1:0/3211103802 shutdown_connections 2024-03-18T18:50:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.430+0000 7fe1fe277700 1 -- 172.21.15.1:0/3211103802 wait complete. 2024-03-18T18:50:09.431 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T18:50:09.439 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-18T18:50:09.392499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594961,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219464,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755256,"statfs":{"total":773094113280,"available":772869382144,"internally_reserved":0,"allocated":3383296,"data_stored":2025530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2231,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002788"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976876+0000","last_change":"2024-03-18T18:49:44.831395+0000","last_active":"2024-03-18T18:49:48.976876+0000","last_peered":"2024-03-18T18:49:48.976876+0000","last_clean":"2024-03-18T18:49:48.976876+0000","last_became_active":"2024-03-18T18:49:44.830706+0000","last_became_peered":"2024-03-18T18:49:44.830706+0000","last_unstale":"2024-03-18T18:49:48.976876+0000","last_undegraded":"2024-03-18T18:49:48.976876+0000","last_fullsized":"2024-03-18T18:49:48.976876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:20:43.395005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841775+0000","last_change":"2024-03-18T18:49:44.829492+0000","last_active":"2024-03-18T18:49:48.841775+0000","last_peered":"2024-03-18T18:49:48.841775+0000","last_clean":"2024-03-18T18:49:48.841775+0000","last_became_active":"2024-03-18T18:49:44.829223+0000","last_became_peered":"2024-03-18T18:49:44.829223+0000","last_unstale":"2024-03-18T18:49:48.841775+0000","last_undegraded":"2024-03-18T18:49:48.841775+0000","last_fullsized":"2024-03-18T18:49:48.841775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.845869+0000","last_change":"2024-03-18T18:49:44.831909+0000","last_active":"2024-03-18T18:49:48.845869+0000","last_peered":"2024-03-18T18:49:48.845869+0000","last_clean":"2024-03-18T18:49:48.845869+0000","last_became_active":"2024-03-18T18:49:44.831661+0000","last_became_peered":"2024-03-18T18:49:44.831661+0000","last_unstale":"2024-03-18T18:49:48.845869+0000","last_undegraded":"2024-03-18T18:49:48.845869+0000","last_fullsized":"2024-03-18T18:49:48.845869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:14.055199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840094+0000","last_change":"2024-03-18T18:49:44.838298+0000","last_active":"2024-03-18T18:49:48.840094+0000","last_peered":"2024-03-18T18:49:48.840094+0000","last_clean":"2024-03-18T18:49:48.840094+0000","last_became_active":"2024-03-18T18:49:44.837689+0000","last_became_peered":"2024-03-18T18:49:44.837689+0000","last_unstale":"2024-03-18T18:49:48.840094+0000","last_undegraded":"2024-03-18T18:49:48.840094+0000","last_fullsized":"2024-03-18T18:49:48.840094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:50.238592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975645+0000","last_change":"2024-03-18T18:49:44.841081+0000","last_active":"2024-03-18T18:49:48.975645+0000","last_peered":"2024-03-18T18:49:48.975645+0000","last_clean":"2024-03-18T18:49:48.975645+0000","last_became_active":"2024-03-18T18:49:44.840798+0000","last_became_peered":"2024-03-18T18:49:44.840798+0000","last_unstale":"2024-03-18T18:49:48.975645+0000","last_undegraded":"2024-03-18T18:49:48.975645+0000","last_fullsized":"2024-03-18T18:49:48.975645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:27:55.580010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839590+0000","last_change":"2024-03-18T18:49:44.836710+0000","last_active":"2024-03-18T18:49:48.839590+0000","last_peered":"2024-03-18T18:49:48.839590+0000","last_clean":"2024-03-18T18:49:48.839590+0000","last_became_active":"2024-03-18T18:49:44.836473+0000","last_became_peered":"2024-03-18T18:49:44.836473+0000","last_unstale":"2024-03-18T18:49:48.839590+0000","last_undegraded":"2024-03-18T18:49:48.839590+0000","last_fullsized":"2024-03-18T18:49:48.839590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:59:20.616256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976795+0000","last_change":"2024-03-18T18:49:44.829680+0000","last_active":"2024-03-18T18:49:48.976795+0000","last_peered":"2024-03-18T18:49:48.976795+0000","last_clean":"2024-03-18T18:49:48.976795+0000","last_became_active":"2024-03-18T18:49:44.829139+0000","last_became_peered":"2024-03-18T18:49:44.829139+0000","last_unstale":"2024-03-18T18:49:48.976795+0000","last_undegraded":"2024-03-18T18:49:48.976795+0000","last_fullsized":"2024-03-18T18:49:48.976795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:59:20.616256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839889+0000","last_change":"2024-03-18T18:49:44.837035+0000","last_active":"2024-03-18T18:49:48.839889+0000","last_peered":"2024-03-18T18:49:48.839889+0000","last_clean":"2024-03-18T18:49:48.839889+0000","last_became_active":"2024-03-18T18:49:44.836598+0000","last_became_peered":"2024-03-18T18:49:44.836598+0000","last_unstale":"2024-03-18T18:49:48.839889+0000","last_undegraded":"2024-03-18T18:49:48.839889+0000","last_fullsized":"2024-03-18T18:49:48.839889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:50.238592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976827+0000","last_change":"2024-03-18T18:49:44.835014+0000","last_active":"2024-03-18T18:49:48.976827+0000","last_peered":"2024-03-18T18:49:48.976827+0000","last_clean":"2024-03-18T18:49:48.976827+0000","last_became_active":"2024-03-18T18:49:44.834438+0000","last_became_peered":"2024-03-18T18:49:44.834438+0000","last_unstale":"2024-03-18T18:49:48.976827+0000","last_undegraded":"2024-03-18T18:49:48.976827+0000","last_fullsized":"2024-03-18T18:49:48.976827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:07.013090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835727+0000","last_change":"2024-03-18T18:49:44.837039+0000","last_active":"2024-03-18T18:49:48.835727+0000","last_peered":"2024-03-18T18:49:48.835727+0000","last_clean":"2024-03-18T18:49:48.835727+0000","last_became_active":"2024-03-18T18:49:44.836749+0000","last_became_peered":"2024-03-18T18:49:44.836749+0000","last_unstale":"2024-03-18T18:49:48.835727+0000","last_undegraded":"2024-03-18T18:49:48.835727+0000","last_fullsized":"2024-03-18T18:49:48.835727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836415+0000","last_change":"2024-03-18T18:49:44.832546+0000","last_active":"2024-03-18T18:49:48.836415+0000","last_peered":"2024-03-18T18:49:48.836415+0000","last_clean":"2024-03-18T18:49:48.836415+0000","last_became_active":"2024-03-18T18:49:44.832333+0000","last_became_peered":"2024-03-18T18:49:44.832333+0000","last_unstale":"2024-03-18T18:49:48.836415+0000","last_undegraded":"2024-03-18T18:49:48.836415+0000","last_fullsized":"2024-03-18T18:49:48.836415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842011+0000","last_change":"2024-03-18T18:49:44.834366+0000","last_active":"2024-03-18T18:49:48.842011+0000","last_peered":"2024-03-18T18:49:48.842011+0000","last_clean":"2024-03-18T18:49:48.842011+0000","last_became_active":"2024-03-18T18:49:44.833912+0000","last_became_peered":"2024-03-18T18:49:44.833912+0000","last_unstale":"2024-03-18T18:49:48.842011+0000","last_undegraded":"2024-03-18T18:49:48.842011+0000","last_fullsized":"2024-03-18T18:49:48.842011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:09:08.248420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839130+0000","last_change":"2024-03-18T18:49:44.823814+0000","last_active":"2024-03-18T18:49:48.839130+0000","last_peered":"2024-03-18T18:49:48.839130+0000","last_clean":"2024-03-18T18:49:48.839130+0000","last_became_active":"2024-03-18T18:49:44.823554+0000","last_became_peered":"2024-03-18T18:49:44.823554+0000","last_unstale":"2024-03-18T18:49:48.839130+0000","last_undegraded":"2024-03-18T18:49:48.839130+0000","last_fullsized":"2024-03-18T18:49:48.839130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:05.661348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853566+0000","last_change":"2024-03-18T18:49:44.828163+0000","last_active":"2024-03-18T18:49:48.853566+0000","last_peered":"2024-03-18T18:49:48.853566+0000","last_clean":"2024-03-18T18:49:48.853566+0000","last_became_active":"2024-03-18T18:49:44.827352+0000","last_became_peered":"2024-03-18T18:49:44.827352+0000","last_unstale":"2024-03-18T18:49:48.853566+0000","last_undegraded":"2024-03-18T18:49:48.853566+0000","last_fullsized":"2024-03-18T18:49:48.853566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:30.009397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840573+0000","last_change":"2024-03-18T18:49:44.839200+0000","last_active":"2024-03-18T18:49:48.840573+0000","last_peered":"2024-03-18T18:49:48.840573+0000","last_clean":"2024-03-18T18:49:48.840573+0000","last_became_active":"2024-03-18T18:49:44.838967+0000","last_became_peered":"2024-03-18T18:49:44.838967+0000","last_unstale":"2024-03-18T18:49:48.840573+0000","last_undegraded":"2024-03-18T18:49:48.840573+0000","last_fullsized":"2024-03-18T18:49:48.840573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:14.055199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839664+0000","last_change":"2024-03-18T18:49:44.834022+0000","last_active":"2024-03-18T18:49:48.839664+0000","last_peered":"2024-03-18T18:49:48.839664+0000","last_clean":"2024-03-18T18:49:48.839664+0000","last_became_active":"2024-03-18T18:49:44.833758+0000","last_became_peered":"2024-03-18T18:49:44.833758+0000","last_unstale":"2024-03-18T18:49:48.839664+0000","last_undegraded":"2024-03-18T18:49:48.839664+0000","last_fullsized":"2024-03-18T18:49:48.839664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:25:14.491414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840752+0000","last_change":"2024-03-18T18:49:44.842296+0000","last_active":"2024-03-18T18:49:48.840752+0000","last_peered":"2024-03-18T18:49:48.840752+0000","last_clean":"2024-03-18T18:49:48.840752+0000","last_became_active":"2024-03-18T18:49:44.841720+0000","last_became_peered":"2024-03-18T18:49:44.841720+0000","last_unstale":"2024-03-18T18:49:48.840752+0000","last_undegraded":"2024-03-18T18:49:48.840752+0000","last_fullsized":"2024-03-18T18:49:48.840752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836699+0000","last_change":"2024-03-18T18:49:44.828352+0000","last_active":"2024-03-18T18:49:48.836699+0000","last_peered":"2024-03-18T18:49:48.836699+0000","last_clean":"2024-03-18T18:49:48.836699+0000","last_became_active":"2024-03-18T18:49:44.828059+0000","last_became_peered":"2024-03-18T18:49:44.828059+0000","last_unstale":"2024-03-18T18:49:48.836699+0000","last_undegraded":"2024-03-18T18:49:48.836699+0000","last_fullsized":"2024-03-18T18:49:48.836699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:29.829759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976858+0000","last_change":"2024-03-18T18:49:44.841034+0000","last_active":"2024-03-18T18:49:48.976858+0000","last_peered":"2024-03-18T18:49:48.976858+0000","last_clean":"2024-03-18T18:49:48.976858+0000","last_became_active":"2024-03-18T18:49:44.840765+0000","last_became_peered":"2024-03-18T18:49:44.840765+0000","last_unstale":"2024-03-18T18:49:48.976858+0000","last_undegraded":"2024-03-18T18:49:48.976858+0000","last_fullsized":"2024-03-18T18:49:48.976858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:34.102274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.982406+0000","last_change":"2024-03-18T18:49:48.981322+0000","last_active":"2024-03-18T18:49:48.982406+0000","last_peered":"2024-03-18T18:49:48.982406+0000","last_clean":"2024-03-18T18:49:48.982406+0000","last_became_active":"2024-03-18T18:49:48.980927+0000","last_became_peered":"2024-03-18T18:49:48.980927+0000","last_unstale":"2024-03-18T18:49:48.982406+0000","last_undegraded":"2024-03-18T18:49:48.982406+0000","last_fullsized":"2024-03-18T18:49:48.982406+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:57:44.834422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.847587+0000","last_change":"2024-03-18T18:49:44.396090+0000","last_active":"2024-03-18T18:49:48.847587+0000","last_peered":"2024-03-18T18:49:48.847587+0000","last_clean":"2024-03-18T18:49:48.847587+0000","last_became_active":"2024-03-18T18:49:44.395823+0000","last_became_peered":"2024-03-18T18:49:44.395823+0000","last_unstale":"2024-03-18T18:49:48.847587+0000","last_undegraded":"2024-03-18T18:49:48.847587+0000","last_fullsized":"2024-03-18T18:49:48.847587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:00:40.075042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840491+0000","last_change":"2024-03-18T18:49:44.821381+0000","last_active":"2024-03-18T18:49:48.840491+0000","last_peered":"2024-03-18T18:49:48.840491+0000","last_clean":"2024-03-18T18:49:48.840491+0000","last_became_active":"2024-03-18T18:49:44.821129+0000","last_became_peered":"2024-03-18T18:49:44.821129+0000","last_unstale":"2024-03-18T18:49:48.840491+0000","last_undegraded":"2024-03-18T18:49:48.840491+0000","last_fullsized":"2024-03-18T18:49:48.840491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853754+0000","last_change":"2024-03-18T18:49:48.852436+0000","last_active":"2024-03-18T18:49:48.853754+0000","last_peered":"2024-03-18T18:49:48.853754+0000","last_clean":"2024-03-18T18:49:48.853754+0000","last_became_active":"2024-03-18T18:49:48.851959+0000","last_became_peered":"2024-03-18T18:49:48.851959+0000","last_unstale":"2024-03-18T18:49:48.853754+0000","last_undegraded":"2024-03-18T18:49:48.853754+0000","last_fullsized":"2024-03-18T18:49:48.853754+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:31:52.237923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840602+0000","last_change":"2024-03-18T18:49:44.399646+0000","last_active":"2024-03-18T18:49:48.840602+0000","last_peered":"2024-03-18T18:49:48.840602+0000","last_clean":"2024-03-18T18:49:48.840602+0000","last_became_active":"2024-03-18T18:49:44.399368+0000","last_became_peered":"2024-03-18T18:49:44.399368+0000","last_unstale":"2024-03-18T18:49:48.840602+0000","last_undegraded":"2024-03-18T18:49:48.840602+0000","last_fullsized":"2024-03-18T18:49:48.840602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:29.763642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.847482+0000","last_change":"2024-03-18T18:49:44.832977+0000","last_active":"2024-03-18T18:49:48.847482+0000","last_peered":"2024-03-18T18:49:48.847482+0000","last_clean":"2024-03-18T18:49:48.847482+0000","last_became_active":"2024-03-18T18:49:44.832624+0000","last_became_peered":"2024-03-18T18:49:44.832624+0000","last_unstale":"2024-03-18T18:49:48.847482+0000","last_undegraded":"2024-03-18T18:49:48.847482+0000","last_fullsized":"2024-03-18T18:49:48.847482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853697+0000","last_change":"2024-03-18T18:49:48.852585+0000","last_active":"2024-03-18T18:49:48.853697+0000","last_peered":"2024-03-18T18:49:48.853697+0000","last_clean":"2024-03-18T18:49:48.853697+0000","last_became_active":"2024-03-18T18:49:48.852137+0000","last_became_peered":"2024-03-18T18:49:48.852137+0000","last_unstale":"2024-03-18T18:49:48.853697+0000","last_undegraded":"2024-03-18T18:49:48.853697+0000","last_fullsized":"2024-03-18T18:49:48.853697+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:50:12.397097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975894+0000","last_change":"2024-03-18T18:49:44.394010+0000","last_active":"2024-03-18T18:49:48.975894+0000","last_peered":"2024-03-18T18:49:48.975894+0000","last_clean":"2024-03-18T18:49:48.975894+0000","last_became_active":"2024-03-18T18:49:44.393512+0000","last_became_peered":"2024-03-18T18:49:44.393512+0000","last_unstale":"2024-03-18T18:49:48.975894+0000","last_undegraded":"2024-03-18T18:49:48.975894+0000","last_fullsized":"2024-03-18T18:49:48.975894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:10:24.082597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839873+0000","last_change":"2024-03-18T18:49:44.836167+0000","last_active":"2024-03-18T18:49:48.839873+0000","last_peered":"2024-03-18T18:49:48.839873+0000","last_clean":"2024-03-18T18:49:48.839873+0000","last_became_active":"2024-03-18T18:49:44.835887+0000","last_became_peered":"2024-03-18T18:49:44.835887+0000","last_unstale":"2024-03-18T18:49:48.839873+0000","last_undegraded":"2024-03-18T18:49:48.839873+0000","last_fullsized":"2024-03-18T18:49:48.839873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:34.102274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.857961+0000","last_change":"2024-03-18T18:49:48.857276+0000","last_active":"2024-03-18T18:49:48.857961+0000","last_peered":"2024-03-18T18:49:48.857961+0000","last_clean":"2024-03-18T18:49:48.857961+0000","last_became_active":"2024-03-18T18:49:48.856870+0000","last_became_peered":"2024-03-18T18:49:48.856870+0000","last_unstale":"2024-03-18T18:49:48.857961+0000","last_undegraded":"2024-03-18T18:49:48.857961+0000","last_fullsized":"2024-03-18T18:49:48.857961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:31.426438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975669+0000","last_change":"2024-03-18T18:49:44.401760+0000","last_active":"2024-03-18T18:49:48.975669+0000","last_peered":"2024-03-18T18:49:48.975669+0000","last_clean":"2024-03-18T18:49:48.975669+0000","last_became_active":"2024-03-18T18:49:44.401565+0000","last_became_peered":"2024-03-18T18:49:44.401565+0000","last_unstale":"2024-03-18T18:49:48.975669+0000","last_undegraded":"2024-03-18T18:49:48.975669+0000","last_fullsized":"2024-03-18T18:49:48.975669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:09:07.245729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840474+0000","last_change":"2024-03-18T18:49:44.834985+0000","last_active":"2024-03-18T18:49:48.840474+0000","last_peered":"2024-03-18T18:49:48.840474+0000","last_clean":"2024-03-18T18:49:48.840474+0000","last_became_active":"2024-03-18T18:49:44.834719+0000","last_became_peered":"2024-03-18T18:49:44.834719+0000","last_unstale":"2024-03-18T18:49:48.840474+0000","last_undegraded":"2024-03-18T18:49:48.840474+0000","last_fullsized":"2024-03-18T18:49:48.840474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:27:55.580010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.854528+0000","last_change":"2024-03-18T18:49:48.852823+0000","last_active":"2024-03-18T18:49:48.854528+0000","last_peered":"2024-03-18T18:49:48.854528+0000","last_clean":"2024-03-18T18:49:48.854528+0000","last_became_active":"2024-03-18T18:49:48.852431+0000","last_became_peered":"2024-03-18T18:49:48.852431+0000","last_unstale":"2024-03-18T18:49:48.854528+0000","last_undegraded":"2024-03-18T18:49:48.854528+0000","last_fullsized":"2024-03-18T18:49:48.854528+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:10:04.268147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839292+0000","last_change":"2024-03-18T18:49:44.402292+0000","last_active":"2024-03-18T18:49:48.839292+0000","last_peered":"2024-03-18T18:49:48.839292+0000","last_clean":"2024-03-18T18:49:48.839292+0000","last_became_active":"2024-03-18T18:49:44.402028+0000","last_became_peered":"2024-03-18T18:49:44.402028+0000","last_unstale":"2024-03-18T18:49:48.839292+0000","last_undegraded":"2024-03-18T18:49:48.839292+0000","last_fullsized":"2024-03-18T18:49:48.839292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.843349+0000","last_change":"2024-03-18T18:49:44.832023+0000","last_active":"2024-03-18T18:49:48.843349+0000","last_peered":"2024-03-18T18:49:48.843349+0000","last_clean":"2024-03-18T18:49:48.843349+0000","last_became_active":"2024-03-18T18:49:44.831734+0000","last_became_peered":"2024-03-18T18:49:44.831734+0000","last_unstale":"2024-03-18T18:49:48.843349+0000","last_undegraded":"2024-03-18T18:49:48.843349+0000","last_fullsized":"2024-03-18T18:49:48.843349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:10:25.085288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.856842+0000","last_change":"2024-03-18T18:49:48.852370+0000","last_active":"2024-03-18T18:49:48.856842+0000","last_peered":"2024-03-18T18:49:48.856842+0000","last_clean":"2024-03-18T18:49:48.856842+0000","last_became_active":"2024-03-18T18:49:48.851953+0000","last_became_peered":"2024-03-18T18:49:48.851953+0000","last_unstale":"2024-03-18T18:49:48.856842+0000","last_undegraded":"2024-03-18T18:49:48.856842+0000","last_fullsized":"2024-03-18T18:49:48.856842+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:17.506065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836324+0000","last_change":"2024-03-18T18:49:44.394163+0000","last_active":"2024-03-18T18:49:48.836324+0000","last_peered":"2024-03-18T18:49:48.836324+0000","last_clean":"2024-03-18T18:49:48.836324+0000","last_became_active":"2024-03-18T18:49:44.393105+0000","last_became_peered":"2024-03-18T18:49:44.393105+0000","last_unstale":"2024-03-18T18:49:48.836324+0000","last_undegraded":"2024-03-18T18:49:48.836324+0000","last_fullsized":"2024-03-18T18:49:48.836324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:30.124822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.846218+0000","last_change":"2024-03-18T18:49:44.834950+0000","last_active":"2024-03-18T18:49:48.846218+0000","last_peered":"2024-03-18T18:49:48.846218+0000","last_clean":"2024-03-18T18:49:48.846218+0000","last_became_active":"2024-03-18T18:49:44.834765+0000","last_became_peered":"2024-03-18T18:49:44.834765+0000","last_unstale":"2024-03-18T18:49:48.846218+0000","last_undegraded":"2024-03-18T18:49:48.846218+0000","last_fullsized":"2024-03-18T18:49:48.846218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:49.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.857842+0000","last_change":"2024-03-18T18:49:48.857127+0000","last_active":"2024-03-18T18:49:48.857842+0000","last_peered":"2024-03-18T18:49:48.857842+0000","last_clean":"2024-03-18T18:49:48.857842+0000","last_became_active":"2024-03-18T18:49:48.856745+0000","last_became_peered":"2024-03-18T18:49:48.856745+0000","last_unstale":"2024-03-18T18:49:48.857842+0000","last_undegraded":"2024-03-18T18:49:48.857842+0000","last_fullsized":"2024-03-18T18:49:48.857842+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:46:22.763891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840897+0000","last_change":"2024-03-18T18:49:44.396730+0000","last_active":"2024-03-18T18:49:48.840897+0000","last_peered":"2024-03-18T18:49:48.840897+0000","last_clean":"2024-03-18T18:49:48.840897+0000","last_became_active":"2024-03-18T18:49:44.396415+0000","last_became_peered":"2024-03-18T18:49:44.396415+0000","last_unstale":"2024-03-18T18:49:48.840897+0000","last_undegraded":"2024-03-18T18:49:48.840897+0000","last_fullsized":"2024-03-18T18:49:48.840897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:28.338064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839641+0000","last_change":"2024-03-18T18:49:44.836855+0000","last_active":"2024-03-18T18:49:48.839641+0000","last_peered":"2024-03-18T18:49:48.839641+0000","last_clean":"2024-03-18T18:49:48.839641+0000","last_became_active":"2024-03-18T18:49:44.836275+0000","last_became_peered":"2024-03-18T18:49:44.836275+0000","last_unstale":"2024-03-18T18:49:48.839641+0000","last_undegraded":"2024-03-18T18:49:48.839641+0000","last_fullsized":"2024-03-18T18:49:48.839641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:49.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980433+0000","last_change":"2024-03-18T18:49:48.978139+0000","last_active":"2024-03-18T18:49:48.980433+0000","last_peered":"2024-03-18T18:49:48.980433+0000","last_clean":"2024-03-18T18:49:48.980433+0000","last_became_active":"2024-03-18T18:49:48.977755+0000","last_became_peered":"2024-03-18T18:49:48.977755+0000","last_unstale":"2024-03-18T18:49:48.980433+0000","last_undegraded":"2024-03-18T18:49:48.980433+0000","last_fullsized":"2024-03-18T18:49:48.980433+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:43.744272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839770+0000","last_change":"2024-03-18T18:49:44.400836+0000","last_active":"2024-03-18T18:49:48.839770+0000","last_peered":"2024-03-18T18:49:48.839770+0000","last_clean":"2024-03-18T18:49:48.839770+0000","last_became_active":"2024-03-18T18:49:44.400587+0000","last_became_peered":"2024-03-18T18:49:44.400587+0000","last_unstale":"2024-03-18T18:49:48.839770+0000","last_undegraded":"2024-03-18T18:49:48.839770+0000","last_fullsized":"2024-03-18T18:49:48.839770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:30.124822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840775+0000","last_change":"2024-03-18T18:49:44.822283+0000","last_active":"2024-03-18T18:49:48.840775+0000","last_peered":"2024-03-18T18:49:48.840775+0000","last_clean":"2024-03-18T18:49:48.840775+0000","last_became_active":"2024-03-18T18:49:44.822016+0000","last_became_peered":"2024-03-18T18:49:44.822016+0000","last_unstale":"2024-03-18T18:49:48.840775+0000","last_undegraded":"2024-03-18T18:49:48.840775+0000","last_fullsized":"2024-03-18T18:49:48.840775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981750+0000","last_change":"2024-03-18T18:49:48.978066+0000","last_active":"2024-03-18T18:49:48.981750+0000","last_peered":"2024-03-18T18:49:48.981750+0000","last_clean":"2024-03-18T18:49:48.981750+0000","last_became_active":"2024-03-18T18:49:48.977682+0000","last_became_peered":"2024-03-18T18:49:48.977682+0000","last_unstale":"2024-03-18T18:49:48.981750+0000","last_undegraded":"2024-03-18T18:49:48.981750+0000","last_fullsized":"2024-03-18T18:49:48.981750+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:31:52.237923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836616+0000","last_change":"2024-03-18T18:49:44.402381+0000","last_active":"2024-03-18T18:49:48.836616+0000","last_peered":"2024-03-18T18:49:48.836616+0000","last_clean":"2024-03-18T18:49:48.836616+0000","last_became_active":"2024-03-18T18:49:44.402101+0000","last_became_peered":"2024-03-18T18:49:44.402101+0000","last_unstale":"2024-03-18T18:49:48.836616+0000","last_undegraded":"2024-03-18T18:49:48.836616+0000","last_fullsized":"2024-03-18T18:49:48.836616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:39.899537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840754+0000","last_change":"2024-03-18T18:49:44.841434+0000","last_active":"2024-03-18T18:49:48.840754+0000","last_peered":"2024-03-18T18:49:48.840754+0000","last_clean":"2024-03-18T18:49:48.840754+0000","last_became_active":"2024-03-18T18:49:44.841186+0000","last_became_peered":"2024-03-18T18:49:44.841186+0000","last_unstale":"2024-03-18T18:49:48.840754+0000","last_undegraded":"2024-03-18T18:49:48.840754+0000","last_fullsized":"2024-03-18T18:49:48.840754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.850697+0000","last_change":"2024-03-18T18:49:48.849692+0000","last_active":"2024-03-18T18:49:48.850697+0000","last_peered":"2024-03-18T18:49:48.850697+0000","last_clean":"2024-03-18T18:49:48.850697+0000","last_became_active":"2024-03-18T18:49:48.849230+0000","last_became_peered":"2024-03-18T18:49:48.849230+0000","last_unstale":"2024-03-18T18:49:48.850697+0000","last_undegraded":"2024-03-18T18:49:48.850697+0000","last_fullsized":"2024-03-18T18:49:48.850697+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:45:34.390044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840676+0000","last_change":"2024-03-18T18:49:44.400484+0000","last_active":"2024-03-18T18:49:48.840676+0000","last_peered":"2024-03-18T18:49:48.840676+0000","last_clean":"2024-03-18T18:49:48.840676+0000","last_became_active":"2024-03-18T18:49:44.398334+0000","last_became_peered":"2024-03-18T18:49:44.398334+0000","last_unstale":"2024-03-18T18:49:48.840676+0000","last_undegraded":"2024-03-18T18:49:48.840676+0000","last_fullsized":"2024-03-18T18:49:48.840676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:29.763642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.847751+0000","last_change":"2024-03-18T18:49:44.832690+0000","last_active":"2024-03-18T18:49:48.847751+0000","last_peered":"2024-03-18T18:49:48.847751+0000","last_clean":"2024-03-18T18:49:48.847751+0000","last_became_active":"2024-03-18T18:49:44.832317+0000","last_became_peered":"2024-03-18T18:49:44.832317+0000","last_unstale":"2024-03-18T18:49:48.847751+0000","last_undegraded":"2024-03-18T18:49:48.847751+0000","last_fullsized":"2024-03-18T18:49:48.847751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:34.102274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.850171+0000","last_change":"2024-03-18T18:49:48.849391+0000","last_active":"2024-03-18T18:49:48.850171+0000","last_peered":"2024-03-18T18:49:48.850171+0000","last_clean":"2024-03-18T18:49:48.850171+0000","last_became_active":"2024-03-18T18:49:48.849018+0000","last_became_peered":"2024-03-18T18:49:48.849018+0000","last_unstale":"2024-03-18T18:49:48.850171+0000","last_undegraded":"2024-03-18T18:49:48.850171+0000","last_fullsized":"2024-03-18T18:49:48.850171+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:20:46.845872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841507+0000","last_change":"2024-03-18T18:49:44.402300+0000","last_active":"2024-03-18T18:49:48.841507+0000","last_peered":"2024-03-18T18:49:48.841507+0000","last_clean":"2024-03-18T18:49:48.841507+0000","last_became_active":"2024-03-18T18:49:44.401998+0000","last_became_peered":"2024-03-18T18:49:44.401998+0000","last_unstale":"2024-03-18T18:49:48.841507+0000","last_undegraded":"2024-03-18T18:49:48.841507+0000","last_fullsized":"2024-03-18T18:49:48.841507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:09:07.245729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841888+0000","last_change":"2024-03-18T18:49:44.835310+0000","last_active":"2024-03-18T18:49:48.841888+0000","last_peered":"2024-03-18T18:49:48.841888+0000","last_clean":"2024-03-18T18:49:48.841888+0000","last_became_active":"2024-03-18T18:49:44.835035+0000","last_became_peered":"2024-03-18T18:49:44.835035+0000","last_unstale":"2024-03-18T18:49:48.841888+0000","last_undegraded":"2024-03-18T18:49:48.841888+0000","last_fullsized":"2024-03-18T18:49:48.841888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:29.829759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.865513+0000","last_change":"2024-03-18T18:49:48.863905+0000","last_active":"2024-03-18T18:49:48.865513+0000","last_peered":"2024-03-18T18:49:48.865513+0000","last_clean":"2024-03-18T18:49:48.865513+0000","last_became_active":"2024-03-18T18:49:48.863234+0000","last_became_peered":"2024-03-18T18:49:48.863234+0000","last_unstale":"2024-03-18T18:49:48.865513+0000","last_undegraded":"2024-03-18T18:49:48.865513+0000","last_fullsized":"2024-03-18T18:49:48.865513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:16:08.575913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T18:49:54.442358+0000","last_change":"2024-03-18T18:49:44.402443+0000","last_active":"2024-03-18T18:49:54.442358+0000","last_peered":"2024-03-18T18:49:54.442358+0000","last_clean":"2024-03-18T18:49:54.442358+0000","last_became_active":"2024-03-18T18:49:44.402144+0000","last_became_peered":"2024-03-18T18:49:44.402144+0000","last_unstale":"2024-03-18T18:49:54.442358+0000","last_undegraded":"2024-03-18T18:49:54.442358+0000","last_fullsized":"2024-03-18T18:49:54.442358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:06.816608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3795,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976826+0000","last_change":"2024-03-18T18:49:44.829814+0000","last_active":"2024-03-18T18:49:48.976826+0000","last_peered":"2024-03-18T18:49:48.976826+0000","last_clean":"2024-03-18T18:49:48.976826+0000","last_became_active":"2024-03-18T18:49:44.829548+0000","last_became_peered":"2024-03-18T18:49:44.829548+0000","last_unstale":"2024-03-18T18:49:48.976826+0000","last_undegraded":"2024-03-18T18:49:48.976826+0000","last_fullsized":"2024-03-18T18:49:48.976826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:02:06.725321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980345+0000","last_change":"2024-03-18T18:49:48.978265+0000","last_active":"2024-03-18T18:49:48.980345+0000","last_peered":"2024-03-18T18:49:48.980345+0000","last_clean":"2024-03-18T18:49:48.980345+0000","last_became_active":"2024-03-18T18:49:48.977853+0000","last_became_peered":"2024-03-18T18:49:48.977853+0000","last_unstale":"2024-03-18T18:49:48.980345+0000","last_undegraded":"2024-03-18T18:49:48.980345+0000","last_fullsized":"2024-03-18T18:49:48.980345+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:43.744272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.846561+0000","last_change":"2024-03-18T18:49:44.392639+0000","last_active":"2024-03-18T18:49:48.846561+0000","last_peered":"2024-03-18T18:49:48.846561+0000","last_clean":"2024-03-18T18:49:48.846561+0000","last_became_active":"2024-03-18T18:49:44.392339+0000","last_became_peered":"2024-03-18T18:49:44.392339+0000","last_unstale":"2024-03-18T18:49:48.846561+0000","last_undegraded":"2024-03-18T18:49:48.846561+0000","last_fullsized":"2024-03-18T18:49:48.846561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:49.491669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853657+0000","last_change":"2024-03-18T18:49:44.836441+0000","last_active":"2024-03-18T18:49:48.853657+0000","last_peered":"2024-03-18T18:49:48.853657+0000","last_clean":"2024-03-18T18:49:48.853657+0000","last_became_active":"2024-03-18T18:49:44.836156+0000","last_became_peered":"2024-03-18T18:49:44.836156+0000","last_unstale":"2024-03-18T18:49:48.853657+0000","last_undegraded":"2024-03-18T18:49:48.853657+0000","last_fullsized":"2024-03-18T18:49:48.853657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:27.913188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980346+0000","last_change":"2024-03-18T18:49:48.977065+0000","last_active":"2024-03-18T18:49:48.980346+0000","last_peered":"2024-03-18T18:49:48.980346+0000","last_clean":"2024-03-18T18:49:48.980346+0000","last_became_active":"2024-03-18T18:49:48.976568+0000","last_became_peered":"2024-03-18T18:49:48.976568+0000","last_unstale":"2024-03-18T18:49:48.980346+0000","last_undegraded":"2024-03-18T18:49:48.980346+0000","last_fullsized":"2024-03-18T18:49:48.980346+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:21.533140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839322+0000","last_change":"2024-03-18T18:49:44.404269+0000","last_active":"2024-03-18T18:49:48.839322+0000","last_peered":"2024-03-18T18:49:48.839322+0000","last_clean":"2024-03-18T18:49:48.839322+0000","last_became_active":"2024-03-18T18:49:44.403992+0000","last_became_peered":"2024-03-18T18:49:44.403992+0000","last_unstale":"2024-03-18T18:49:48.839322+0000","last_undegraded":"2024-03-18T18:49:48.839322+0000","last_fullsized":"2024-03-18T18:49:48.839322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975684+0000","last_change":"2024-03-18T18:49:44.834354+0000","last_active":"2024-03-18T18:49:48.975684+0000","last_peered":"2024-03-18T18:49:48.975684+0000","last_clean":"2024-03-18T18:49:48.975684+0000","last_became_active":"2024-03-18T18:49:44.834114+0000","last_became_peered":"2024-03-18T18:49:44.834114+0000","last_unstale":"2024-03-18T18:49:48.975684+0000","last_undegraded":"2024-03-18T18:49:48.975684+0000","last_fullsized":"2024-03-18T18:49:48.975684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:09:46.686770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981775+0000","last_change":"2024-03-18T18:49:48.978466+0000","last_active":"2024-03-18T18:49:48.981775+0000","last_peered":"2024-03-18T18:49:48.981775+0000","last_clean":"2024-03-18T18:49:48.981775+0000","last_became_active":"2024-03-18T18:49:48.977877+0000","last_became_peered":"2024-03-18T18:49:48.977877+0000","last_unstale":"2024-03-18T18:49:48.981775+0000","last_undegraded":"2024-03-18T18:49:48.981775+0000","last_fullsized":"2024-03-18T18:49:48.981775+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:56.090012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836197+0000","last_change":"2024-03-18T18:49:44.393777+0000","last_active":"2024-03-18T18:49:48.836197+0000","last_peered":"2024-03-18T18:49:48.836197+0000","last_clean":"2024-03-18T18:49:48.836197+0000","last_became_active":"2024-03-18T18:49:44.392861+0000","last_became_peered":"2024-03-18T18:49:44.392861+0000","last_unstale":"2024-03-18T18:49:48.836197+0000","last_undegraded":"2024-03-18T18:49:48.836197+0000","last_fullsized":"2024-03-18T18:49:48.836197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835596+0000","last_change":"2024-03-18T18:49:44.831121+0000","last_active":"2024-03-18T18:49:48.835596+0000","last_peered":"2024-03-18T18:49:48.835596+0000","last_clean":"2024-03-18T18:49:48.835596+0000","last_became_active":"2024-03-18T18:49:44.830864+0000","last_became_peered":"2024-03-18T18:49:44.830864+0000","last_unstale":"2024-03-18T18:49:48.835596+0000","last_undegraded":"2024-03-18T18:49:48.835596+0000","last_fullsized":"2024-03-18T18:49:48.835596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:23.558014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.860292+0000","last_change":"2024-03-18T18:49:48.858618+0000","last_active":"2024-03-18T18:49:48.860292+0000","last_peered":"2024-03-18T18:49:48.860292+0000","last_clean":"2024-03-18T18:49:48.860292+0000","last_became_active":"2024-03-18T18:49:48.858170+0000","last_became_peered":"2024-03-18T18:49:48.858170+0000","last_unstale":"2024-03-18T18:49:48.860292+0000","last_undegraded":"2024-03-18T18:49:48.860292+0000","last_fullsized":"2024-03-18T18:49:48.860292+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:31.426438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836418+0000","last_change":"2024-03-18T18:49:44.397131+0000","last_active":"2024-03-18T18:49:48.836418+0000","last_peered":"2024-03-18T18:49:48.836418+0000","last_clean":"2024-03-18T18:49:48.836418+0000","last_became_active":"2024-03-18T18:49:44.396856+0000","last_became_peered":"2024-03-18T18:49:44.396856+0000","last_unstale":"2024-03-18T18:49:48.836418+0000","last_undegraded":"2024-03-18T18:49:48.836418+0000","last_fullsized":"2024-03-18T18:49:48.836418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976093+0000","last_change":"2024-03-18T18:49:44.838701+0000","last_active":"2024-03-18T18:49:48.976093+0000","last_peered":"2024-03-18T18:49:48.976093+0000","last_clean":"2024-03-18T18:49:48.976093+0000","last_became_active":"2024-03-18T18:49:44.838424+0000","last_became_peered":"2024-03-18T18:49:44.838424+0000","last_unstale":"2024-03-18T18:49:48.976093+0000","last_undegraded":"2024-03-18T18:49:48.976093+0000","last_fullsized":"2024-03-18T18:49:48.976093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:25:14.491414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853778+0000","last_change":"2024-03-18T18:49:48.852659+0000","last_active":"2024-03-18T18:49:48.853778+0000","last_peered":"2024-03-18T18:49:48.853778+0000","last_clean":"2024-03-18T18:49:48.853778+0000","last_became_active":"2024-03-18T18:49:48.852248+0000","last_became_peered":"2024-03-18T18:49:48.852248+0000","last_unstale":"2024-03-18T18:49:48.853778+0000","last_undegraded":"2024-03-18T18:49:48.853778+0000","last_fullsized":"2024-03-18T18:49:48.853778+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:38.261593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841901+0000","last_change":"2024-03-18T18:49:44.399701+0000","last_active":"2024-03-18T18:49:48.841901+0000","last_peered":"2024-03-18T18:49:48.841901+0000","last_clean":"2024-03-18T18:49:48.841901+0000","last_became_active":"2024-03-18T18:49:44.399371+0000","last_became_peered":"2024-03-18T18:49:44.399371+0000","last_unstale":"2024-03-18T18:49:48.841901+0000","last_undegraded":"2024-03-18T18:49:48.841901+0000","last_fullsized":"2024-03-18T18:49:48.841901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:49.491669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975930+0000","last_change":"2024-03-18T18:49:44.834635+0000","last_active":"2024-03-18T18:49:48.975930+0000","last_peered":"2024-03-18T18:49:48.975930+0000","last_clean":"2024-03-18T18:49:48.975930+0000","last_became_active":"2024-03-18T18:49:44.834321+0000","last_became_peered":"2024-03-18T18:49:44.834321+0000","last_unstale":"2024-03-18T18:49:48.975930+0000","last_undegraded":"2024-03-18T18:49:48.975930+0000","last_fullsized":"2024-03-18T18:49:48.975930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:50.238592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.865503+0000","last_change":"2024-03-18T18:49:48.863585+0000","last_active":"2024-03-18T18:49:48.865503+0000","last_peered":"2024-03-18T18:49:48.865503+0000","last_clean":"2024-03-18T18:49:48.865503+0000","last_became_active":"2024-03-18T18:49:48.862710+0000","last_became_peered":"2024-03-18T18:49:48.862710+0000","last_unstale":"2024-03-18T18:49:48.865503+0000","last_undegraded":"2024-03-18T18:49:48.865503+0000","last_fullsized":"2024-03-18T18:49:48.865503+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:31.426438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842830+0000","last_change":"2024-03-18T18:49:44.399251+0000","last_active":"2024-03-18T18:49:48.842830+0000","last_peered":"2024-03-18T18:49:48.842830+0000","last_clean":"2024-03-18T18:49:48.842830+0000","last_became_active":"2024-03-18T18:49:44.398969+0000","last_became_peered":"2024-03-18T18:49:44.398969+0000","last_unstale":"2024-03-18T18:49:48.842830+0000","last_undegraded":"2024-03-18T18:49:48.842830+0000","last_fullsized":"2024-03-18T18:49:48.842830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:39.899537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840553+0000","last_change":"2024-03-18T18:49:44.841934+0000","last_active":"2024-03-18T18:49:48.840553+0000","last_peered":"2024-03-18T18:49:48.840553+0000","last_clean":"2024-03-18T18:49:48.840553+0000","last_became_active":"2024-03-18T18:49:44.840657+0000","last_became_peered":"2024-03-18T18:49:44.840657+0000","last_unstale":"2024-03-18T18:49:48.840553+0000","last_undegraded":"2024-03-18T18:49:48.840553+0000","last_fullsized":"2024-03-18T18:49:48.840553+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.865367+0000","last_change":"2024-03-18T18:49:48.862242+0000","last_active":"2024-03-18T18:49:48.865367+0000","last_peered":"2024-03-18T18:49:48.865367+0000","last_clean":"2024-03-18T18:49:48.865367+0000","last_became_active":"2024-03-18T18:49:48.861779+0000","last_became_peered":"2024-03-18T18:49:48.861779+0000","last_unstale":"2024-03-18T18:49:48.865367+0000","last_undegraded":"2024-03-18T18:49:48.865367+0000","last_fullsized":"2024-03-18T18:49:48.865367+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:49:52.529190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839229+0000","last_change":"2024-03-18T18:49:44.404171+0000","last_active":"2024-03-18T18:49:48.839229+0000","last_peered":"2024-03-18T18:49:48.839229+0000","last_clean":"2024-03-18T18:49:48.839229+0000","last_became_active":"2024-03-18T18:49:44.403758+0000","last_became_peered":"2024-03-18T18:49:44.403758+0000","last_unstale":"2024-03-18T18:49:48.839229+0000","last_undegraded":"2024-03-18T18:49:48.839229+0000","last_fullsized":"2024-03-18T18:49:48.839229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:55.666762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841896+0000","last_change":"2024-03-18T18:49:44.839405+0000","last_active":"2024-03-18T18:49:48.841896+0000","last_peered":"2024-03-18T18:49:48.841896+0000","last_clean":"2024-03-18T18:49:48.841896+0000","last_became_active":"2024-03-18T18:49:44.838946+0000","last_became_peered":"2024-03-18T18:49:44.838946+0000","last_unstale":"2024-03-18T18:49:48.841896+0000","last_undegraded":"2024-03-18T18:49:48.841896+0000","last_fullsized":"2024-03-18T18:49:48.841896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:07.819299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.865474+0000","last_change":"2024-03-18T18:49:48.863506+0000","last_active":"2024-03-18T18:49:48.865474+0000","last_peered":"2024-03-18T18:49:48.865474+0000","last_clean":"2024-03-18T18:49:48.865474+0000","last_became_active":"2024-03-18T18:49:48.863081+0000","last_became_peered":"2024-03-18T18:49:48.863081+0000","last_unstale":"2024-03-18T18:49:48.865474+0000","last_undegraded":"2024-03-18T18:49:48.865474+0000","last_fullsized":"2024-03-18T18:49:48.865474+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:47.284063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839273+0000","last_change":"2024-03-18T18:49:44.833724+0000","last_active":"2024-03-18T18:49:48.839273+0000","last_peered":"2024-03-18T18:49:48.839273+0000","last_clean":"2024-03-18T18:49:48.839273+0000","last_became_active":"2024-03-18T18:49:44.833231+0000","last_became_peered":"2024-03-18T18:49:44.833231+0000","last_unstale":"2024-03-18T18:49:48.839273+0000","last_undegraded":"2024-03-18T18:49:48.839273+0000","last_fullsized":"2024-03-18T18:49:48.839273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:30.009397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836484+0000","last_change":"2024-03-18T18:49:44.402405+0000","last_active":"2024-03-18T18:49:48.836484+0000","last_peered":"2024-03-18T18:49:48.836484+0000","last_clean":"2024-03-18T18:49:48.836484+0000","last_became_active":"2024-03-18T18:49:44.402133+0000","last_became_peered":"2024-03-18T18:49:44.402133+0000","last_unstale":"2024-03-18T18:49:48.836484+0000","last_undegraded":"2024-03-18T18:49:48.836484+0000","last_fullsized":"2024-03-18T18:49:48.836484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:29.763642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981114+0000","last_change":"2024-03-18T18:49:48.978691+0000","last_active":"2024-03-18T18:49:48.981114+0000","last_peered":"2024-03-18T18:49:48.981114+0000","last_clean":"2024-03-18T18:49:48.981114+0000","last_became_active":"2024-03-18T18:49:48.977741+0000","last_became_peered":"2024-03-18T18:49:48.977741+0000","last_unstale":"2024-03-18T18:49:48.981114+0000","last_undegraded":"2024-03-18T18:49:48.981114+0000","last_fullsized":"2024-03-18T18:49:48.981114+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:07:03.479681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840954+0000","last_change":"2024-03-18T18:49:44.833327+0000","last_active":"2024-03-18T18:49:48.840954+0000","last_peered":"2024-03-18T18:49:48.840954+0000","last_clean":"2024-03-18T18:49:48.840954+0000","last_became_active":"2024-03-18T18:49:44.833073+0000","last_became_peered":"2024-03-18T18:49:44.833073+0000","last_unstale":"2024-03-18T18:49:48.840954+0000","last_undegraded":"2024-03-18T18:49:48.840954+0000","last_fullsized":"2024-03-18T18:49:48.840954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:27:55.580010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839748+0000","last_change":"2024-03-18T18:49:44.401250+0000","last_active":"2024-03-18T18:49:48.839748+0000","last_peered":"2024-03-18T18:49:48.839748+0000","last_clean":"2024-03-18T18:49:48.839748+0000","last_became_active":"2024-03-18T18:49:44.398256+0000","last_became_peered":"2024-03-18T18:49:44.398256+0000","last_unstale":"2024-03-18T18:49:48.839748+0000","last_undegraded":"2024-03-18T18:49:48.839748+0000","last_fullsized":"2024-03-18T18:49:48.839748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:06.816608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":167,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975479+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T18:49:48.975479+0000","last_peered":"2024-03-18T18:49:48.975479+0000","last_clean":"2024-03-18T18:49:48.975479+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T18:49:48.975479+0000","last_undegraded":"2024-03-18T18:49:48.975479+0000","last_fullsized":"2024-03-18T18:49:48.975479+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980453+0000","last_change":"2024-03-18T18:49:48.978188+0000","last_active":"2024-03-18T18:49:48.980453+0000","last_peered":"2024-03-18T18:49:48.980453+0000","last_clean":"2024-03-18T18:49:48.980453+0000","last_became_active":"2024-03-18T18:49:48.977852+0000","last_became_peered":"2024-03-18T18:49:48.977852+0000","last_unstale":"2024-03-18T18:49:48.980453+0000","last_undegraded":"2024-03-18T18:49:48.980453+0000","last_fullsized":"2024-03-18T18:49:48.980453+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:38.261593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.848312+0000","last_change":"2024-03-18T18:49:44.839127+0000","last_active":"2024-03-18T18:49:48.848312+0000","last_peered":"2024-03-18T18:49:48.848312+0000","last_clean":"2024-03-18T18:49:48.848312+0000","last_became_active":"2024-03-18T18:49:44.838948+0000","last_became_peered":"2024-03-18T18:49:44.838948+0000","last_unstale":"2024-03-18T18:49:48.848312+0000","last_undegraded":"2024-03-18T18:49:48.848312+0000","last_fullsized":"2024-03-18T18:49:48.848312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:23.558014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975591+0000","last_change":"2024-03-18T18:49:44.402282+0000","last_active":"2024-03-18T18:49:48.975591+0000","last_peered":"2024-03-18T18:49:48.975591+0000","last_clean":"2024-03-18T18:49:48.975591+0000","last_became_active":"2024-03-18T18:49:44.402003+0000","last_became_peered":"2024-03-18T18:49:44.402003+0000","last_unstale":"2024-03-18T18:49:48.975591+0000","last_undegraded":"2024-03-18T18:49:48.975591+0000","last_fullsized":"2024-03-18T18:49:48.975591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:28.827068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981915+0000","last_change":"2024-03-18T18:49:48.978958+0000","last_active":"2024-03-18T18:49:48.981915+0000","last_peered":"2024-03-18T18:49:48.981915+0000","last_clean":"2024-03-18T18:49:48.981915+0000","last_became_active":"2024-03-18T18:49:48.978608+0000","last_became_peered":"2024-03-18T18:49:48.978608+0000","last_unstale":"2024-03-18T18:49:48.981915+0000","last_undegraded":"2024-03-18T18:49:48.981915+0000","last_fullsized":"2024-03-18T18:49:48.981915+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:08:46.162308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975789+0000","last_change":"2024-03-18T18:49:44.397512+0000","last_active":"2024-03-18T18:49:48.975789+0000","last_peered":"2024-03-18T18:49:48.975789+0000","last_clean":"2024-03-18T18:49:48.975789+0000","last_became_active":"2024-03-18T18:49:44.397271+0000","last_became_peered":"2024-03-18T18:49:44.397271+0000","last_unstale":"2024-03-18T18:49:48.975789+0000","last_undegraded":"2024-03-18T18:49:48.975789+0000","last_fullsized":"2024-03-18T18:49:48.975789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:55.666762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842833+0000","last_change":"2024-03-18T18:49:44.838715+0000","last_active":"2024-03-18T18:49:48.842833+0000","last_peered":"2024-03-18T18:49:48.842833+0000","last_clean":"2024-03-18T18:49:48.842833+0000","last_became_active":"2024-03-18T18:49:44.838452+0000","last_became_peered":"2024-03-18T18:49:44.838452+0000","last_unstale":"2024-03-18T18:49:48.842833+0000","last_undegraded":"2024-03-18T18:49:48.842833+0000","last_fullsized":"2024-03-18T18:49:48.842833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:23.558014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981853+0000","last_change":"2024-03-18T18:49:48.978525+0000","last_active":"2024-03-18T18:49:48.981853+0000","last_peered":"2024-03-18T18:49:48.981853+0000","last_clean":"2024-03-18T18:49:48.981853+0000","last_became_active":"2024-03-18T18:49:48.977651+0000","last_became_peered":"2024-03-18T18:49:48.977651+0000","last_unstale":"2024-03-18T18:49:48.981853+0000","last_undegraded":"2024-03-18T18:49:48.981853+0000","last_fullsized":"2024-03-18T18:49:48.981853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:31:53.057656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841616+0000","last_change":"2024-03-18T18:49:44.397630+0000","last_active":"2024-03-18T18:49:48.841616+0000","last_peered":"2024-03-18T18:49:48.841616+0000","last_clean":"2024-03-18T18:49:48.841616+0000","last_became_active":"2024-03-18T18:49:44.396306+0000","last_became_peered":"2024-03-18T18:49:44.396306+0000","last_unstale":"2024-03-18T18:49:48.841616+0000","last_undegraded":"2024-03-18T18:49:48.841616+0000","last_fullsized":"2024-03-18T18:49:48.841616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:10:24.082597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842664+0000","last_change":"2024-03-18T18:49:44.821273+0000","last_active":"2024-03-18T18:49:48.842664+0000","last_peered":"2024-03-18T18:49:48.842664+0000","last_clean":"2024-03-18T18:49:48.842664+0000","last_became_active":"2024-03-18T18:49:44.821036+0000","last_became_peered":"2024-03-18T18:49:44.821036+0000","last_unstale":"2024-03-18T18:49:48.842664+0000","last_undegraded":"2024-03-18T18:49:48.842664+0000","last_fullsized":"2024-03-18T18:49:48.842664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:30.009397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.982011+0000","last_change":"2024-03-18T18:49:48.978130+0000","last_active":"2024-03-18T18:49:48.982011+0000","last_peered":"2024-03-18T18:49:48.982011+0000","last_clean":"2024-03-18T18:49:48.982011+0000","last_became_active":"2024-03-18T18:49:48.977715+0000","last_became_peered":"2024-03-18T18:49:48.977715+0000","last_unstale":"2024-03-18T18:49:48.982011+0000","last_undegraded":"2024-03-18T18:49:48.982011+0000","last_fullsized":"2024-03-18T18:49:48.982011+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:47.284063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836478+0000","last_change":"2024-03-18T18:49:44.394139+0000","last_active":"2024-03-18T18:49:48.836478+0000","last_peered":"2024-03-18T18:49:48.836478+0000","last_clean":"2024-03-18T18:49:48.836478+0000","last_became_active":"2024-03-18T18:49:44.393405+0000","last_became_peered":"2024-03-18T18:49:44.393405+0000","last_unstale":"2024-03-18T18:49:48.836478+0000","last_undegraded":"2024-03-18T18:49:48.836478+0000","last_fullsized":"2024-03-18T18:49:48.836478+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:55.666762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853791+0000","last_change":"2024-03-18T18:49:44.829271+0000","last_active":"2024-03-18T18:49:48.853791+0000","last_peered":"2024-03-18T18:49:48.853791+0000","last_clean":"2024-03-18T18:49:48.853791+0000","last_became_active":"2024-03-18T18:49:44.829019+0000","last_became_peered":"2024-03-18T18:49:44.829019+0000","last_unstale":"2024-03-18T18:49:48.853791+0000","last_undegraded":"2024-03-18T18:49:48.853791+0000","last_fullsized":"2024-03-18T18:49:48.853791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853721+0000","last_change":"2024-03-18T18:49:48.852590+0000","last_active":"2024-03-18T18:49:48.853721+0000","last_peered":"2024-03-18T18:49:48.853721+0000","last_clean":"2024-03-18T18:49:48.853721+0000","last_became_active":"2024-03-18T18:49:48.852088+0000","last_became_peered":"2024-03-18T18:49:48.852088+0000","last_unstale":"2024-03-18T18:49:48.853721+0000","last_undegraded":"2024-03-18T18:49:48.853721+0000","last_fullsized":"2024-03-18T18:49:48.853721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:16:08.575913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975464+0000","last_change":"2024-03-18T18:49:44.393305+0000","last_active":"2024-03-18T18:49:48.975464+0000","last_peered":"2024-03-18T18:49:48.975464+0000","last_clean":"2024-03-18T18:49:48.975464+0000","last_became_active":"2024-03-18T18:49:44.392494+0000","last_became_peered":"2024-03-18T18:49:44.392494+0000","last_unstale":"2024-03-18T18:49:48.975464+0000","last_undegraded":"2024-03-18T18:49:48.975464+0000","last_fullsized":"2024-03-18T18:49:48.975464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:06.816608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841483+0000","last_change":"2024-03-18T18:49:44.837726+0000","last_active":"2024-03-18T18:49:48.841483+0000","last_peered":"2024-03-18T18:49:48.841483+0000","last_clean":"2024-03-18T18:49:48.841483+0000","last_became_active":"2024-03-18T18:49:44.837364+0000","last_became_peered":"2024-03-18T18:49:44.837364+0000","last_unstale":"2024-03-18T18:49:48.841483+0000","last_undegraded":"2024-03-18T18:49:48.841483+0000","last_fullsized":"2024-03-18T18:49:48.841483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:40.902228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.982312+0000","last_change":"2024-03-18T18:49:48.980106+0000","last_active":"2024-03-18T18:49:48.982312+0000","last_peered":"2024-03-18T18:49:48.982312+0000","last_clean":"2024-03-18T18:49:48.982312+0000","last_became_active":"2024-03-18T18:49:48.979709+0000","last_became_peered":"2024-03-18T18:49:48.979709+0000","last_unstale":"2024-03-18T18:49:48.982312+0000","last_undegraded":"2024-03-18T18:49:48.982312+0000","last_fullsized":"2024-03-18T18:49:48.982312+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:21.533140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840674+0000","last_change":"2024-03-18T18:49:44.398886+0000","last_active":"2024-03-18T18:49:48.840674+0000","last_peered":"2024-03-18T18:49:48.840674+0000","last_clean":"2024-03-18T18:49:48.840674+0000","last_became_active":"2024-03-18T18:49:44.398578+0000","last_became_peered":"2024-03-18T18:49:44.398578+0000","last_unstale":"2024-03-18T18:49:48.840674+0000","last_undegraded":"2024-03-18T18:49:48.840674+0000","last_fullsized":"2024-03-18T18:49:48.840674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839766+0000","last_change":"2024-03-18T18:49:44.826751+0000","last_active":"2024-03-18T18:49:48.839766+0000","last_peered":"2024-03-18T18:49:48.839766+0000","last_clean":"2024-03-18T18:49:48.839766+0000","last_became_active":"2024-03-18T18:49:44.825975+0000","last_became_peered":"2024-03-18T18:49:44.825975+0000","last_unstale":"2024-03-18T18:49:48.839766+0000","last_undegraded":"2024-03-18T18:49:48.839766+0000","last_fullsized":"2024-03-18T18:49:48.839766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:02:06.725321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853727+0000","last_change":"2024-03-18T18:49:48.852554+0000","last_active":"2024-03-18T18:49:48.853727+0000","last_peered":"2024-03-18T18:49:48.853727+0000","last_clean":"2024-03-18T18:49:48.853727+0000","last_became_active":"2024-03-18T18:49:48.852121+0000","last_became_peered":"2024-03-18T18:49:48.852121+0000","last_unstale":"2024-03-18T18:49:48.853727+0000","last_undegraded":"2024-03-18T18:49:48.853727+0000","last_fullsized":"2024-03-18T18:49:48.853727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:41:26.757218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841024+0000","last_change":"2024-03-18T18:49:44.402526+0000","last_active":"2024-03-18T18:49:48.841024+0000","last_peered":"2024-03-18T18:49:48.841024+0000","last_clean":"2024-03-18T18:49:48.841024+0000","last_became_active":"2024-03-18T18:49:44.402277+0000","last_became_peered":"2024-03-18T18:49:44.402277+0000","last_unstale":"2024-03-18T18:49:48.841024+0000","last_undegraded":"2024-03-18T18:49:48.841024+0000","last_fullsized":"2024-03-18T18:49:48.841024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:06.816608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839383+0000","last_change":"2024-03-18T18:49:44.830629+0000","last_active":"2024-03-18T18:49:48.839383+0000","last_peered":"2024-03-18T18:49:48.839383+0000","last_clean":"2024-03-18T18:49:48.839383+0000","last_became_active":"2024-03-18T18:49:44.830313+0000","last_became_peered":"2024-03-18T18:49:44.830313+0000","last_unstale":"2024-03-18T18:49:48.839383+0000","last_undegraded":"2024-03-18T18:49:48.839383+0000","last_fullsized":"2024-03-18T18:49:48.839383+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:27.913188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.860287+0000","last_change":"2024-03-18T18:49:48.859810+0000","last_active":"2024-03-18T18:49:48.860287+0000","last_peered":"2024-03-18T18:49:48.860287+0000","last_clean":"2024-03-18T18:49:48.860287+0000","last_became_active":"2024-03-18T18:49:48.859453+0000","last_became_peered":"2024-03-18T18:49:48.859453+0000","last_unstale":"2024-03-18T18:49:48.860287+0000","last_undegraded":"2024-03-18T18:49:48.860287+0000","last_fullsized":"2024-03-18T18:49:48.860287+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:11:09.238467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975758+0000","last_change":"2024-03-18T18:49:44.396856+0000","last_active":"2024-03-18T18:49:48.975758+0000","last_peered":"2024-03-18T18:49:48.975758+0000","last_clean":"2024-03-18T18:49:48.975758+0000","last_became_active":"2024-03-18T18:49:44.395207+0000","last_became_peered":"2024-03-18T18:49:44.395207+0000","last_unstale":"2024-03-18T18:49:48.975758+0000","last_undegraded":"2024-03-18T18:49:48.975758+0000","last_fullsized":"2024-03-18T18:49:48.975758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:39.899537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836368+0000","last_change":"2024-03-18T18:49:44.836659+0000","last_active":"2024-03-18T18:49:48.836368+0000","last_peered":"2024-03-18T18:49:48.836368+0000","last_clean":"2024-03-18T18:49:48.836368+0000","last_became_active":"2024-03-18T18:49:44.836362+0000","last_became_peered":"2024-03-18T18:49:44.836362+0000","last_unstale":"2024-03-18T18:49:48.836368+0000","last_undegraded":"2024-03-18T18:49:48.836368+0000","last_fullsized":"2024-03-18T18:49:48.836368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:49.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980755+0000","last_change":"2024-03-18T18:49:48.978204+0000","last_active":"2024-03-18T18:49:48.980755+0000","last_peered":"2024-03-18T18:49:48.980755+0000","last_clean":"2024-03-18T18:49:48.980755+0000","last_became_active":"2024-03-18T18:49:48.977745+0000","last_became_peered":"2024-03-18T18:49:48.977745+0000","last_unstale":"2024-03-18T18:49:48.980755+0000","last_undegraded":"2024-03-18T18:49:48.980755+0000","last_fullsized":"2024-03-18T18:49:48.980755+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:08:46.162308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.837352+0000","last_change":"2024-03-18T18:49:44.398731+0000","last_active":"2024-03-18T18:49:48.837352+0000","last_peered":"2024-03-18T18:49:48.837352+0000","last_clean":"2024-03-18T18:49:48.837352+0000","last_became_active":"2024-03-18T18:49:44.398448+0000","last_became_peered":"2024-03-18T18:49:44.398448+0000","last_unstale":"2024-03-18T18:49:48.837352+0000","last_undegraded":"2024-03-18T18:49:48.837352+0000","last_fullsized":"2024-03-18T18:49:48.837352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:30.124822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976056+0000","last_change":"2024-03-18T18:49:44.833300+0000","last_active":"2024-03-18T18:49:48.976056+0000","last_peered":"2024-03-18T18:49:48.976056+0000","last_clean":"2024-03-18T18:49:48.976056+0000","last_became_active":"2024-03-18T18:49:44.833027+0000","last_became_peered":"2024-03-18T18:49:44.833027+0000","last_unstale":"2024-03-18T18:49:48.976056+0000","last_undegraded":"2024-03-18T18:49:48.976056+0000","last_fullsized":"2024-03-18T18:49:48.976056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:11:05.787600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.860344+0000","last_change":"2024-03-18T18:49:48.859259+0000","last_active":"2024-03-18T18:49:48.860344+0000","last_peered":"2024-03-18T18:49:48.860344+0000","last_clean":"2024-03-18T18:49:48.860344+0000","last_became_active":"2024-03-18T18:49:48.857100+0000","last_became_peered":"2024-03-18T18:49:48.857100+0000","last_unstale":"2024-03-18T18:49:48.860344+0000","last_undegraded":"2024-03-18T18:49:48.860344+0000","last_fullsized":"2024-03-18T18:49:48.860344+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:47.284063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841409+0000","last_change":"2024-03-18T18:49:44.395491+0000","last_active":"2024-03-18T18:49:48.841409+0000","last_peered":"2024-03-18T18:49:48.841409+0000","last_clean":"2024-03-18T18:49:48.841409+0000","last_became_active":"2024-03-18T18:49:44.394529+0000","last_became_peered":"2024-03-18T18:49:44.394529+0000","last_unstale":"2024-03-18T18:49:48.841409+0000","last_undegraded":"2024-03-18T18:49:48.841409+0000","last_fullsized":"2024-03-18T18:49:48.841409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:55.666762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839957+0000","last_change":"2024-03-18T18:49:44.838466+0000","last_active":"2024-03-18T18:49:48.839957+0000","last_peered":"2024-03-18T18:49:48.839957+0000","last_clean":"2024-03-18T18:49:48.839957+0000","last_became_active":"2024-03-18T18:49:44.838027+0000","last_became_peered":"2024-03-18T18:49:44.838027+0000","last_unstale":"2024-03-18T18:49:48.839957+0000","last_undegraded":"2024-03-18T18:49:48.839957+0000","last_fullsized":"2024-03-18T18:49:48.839957+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:09:46.686770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981225+0000","last_change":"2024-03-18T18:49:48.979431+0000","last_active":"2024-03-18T18:49:48.981225+0000","last_peered":"2024-03-18T18:49:48.981225+0000","last_clean":"2024-03-18T18:49:48.981225+0000","last_became_active":"2024-03-18T18:49:48.978587+0000","last_became_peered":"2024-03-18T18:49:48.978587+0000","last_unstale":"2024-03-18T18:49:48.981225+0000","last_undegraded":"2024-03-18T18:49:48.981225+0000","last_fullsized":"2024-03-18T18:49:48.981225+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:47.284063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836616+0000","last_change":"2024-03-18T18:49:44.403374+0000","last_active":"2024-03-18T18:49:48.836616+0000","last_peered":"2024-03-18T18:49:48.836616+0000","last_clean":"2024-03-18T18:49:48.836616+0000","last_became_active":"2024-03-18T18:49:44.403082+0000","last_became_peered":"2024-03-18T18:49:44.403082+0000","last_unstale":"2024-03-18T18:49:48.836616+0000","last_undegraded":"2024-03-18T18:49:48.836616+0000","last_fullsized":"2024-03-18T18:49:48.836616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:28.338064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976741+0000","last_change":"2024-03-18T18:49:44.835166+0000","last_active":"2024-03-18T18:49:48.976741+0000","last_peered":"2024-03-18T18:49:48.976741+0000","last_clean":"2024-03-18T18:49:48.976741+0000","last_became_active":"2024-03-18T18:49:44.834808+0000","last_became_peered":"2024-03-18T18:49:44.834808+0000","last_unstale":"2024-03-18T18:49:48.976741+0000","last_undegraded":"2024-03-18T18:49:48.976741+0000","last_fullsized":"2024-03-18T18:49:48.976741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:20.816344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835598+0000","last_change":"2024-03-18T18:49:44.829141+0000","last_active":"2024-03-18T18:49:48.835598+0000","last_peered":"2024-03-18T18:49:48.835598+0000","last_clean":"2024-03-18T18:49:48.835598+0000","last_became_active":"2024-03-18T18:49:44.828863+0000","last_became_peered":"2024-03-18T18:49:44.828863+0000","last_unstale":"2024-03-18T18:49:48.835598+0000","last_undegraded":"2024-03-18T18:49:48.835598+0000","last_fullsized":"2024-03-18T18:49:48.835598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:30.009397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836583+0000","last_change":"2024-03-18T18:49:44.831132+0000","last_active":"2024-03-18T18:49:48.836583+0000","last_peered":"2024-03-18T18:49:48.836583+0000","last_clean":"2024-03-18T18:49:48.836583+0000","last_became_active":"2024-03-18T18:49:44.830873+0000","last_became_peered":"2024-03-18T18:49:44.830873+0000","last_unstale":"2024-03-18T18:49:48.836583+0000","last_undegraded":"2024-03-18T18:49:48.836583+0000","last_fullsized":"2024-03-18T18:49:48.836583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:09:08.248420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839404+0000","last_change":"2024-03-18T18:49:44.841166+0000","last_active":"2024-03-18T18:49:48.839404+0000","last_peered":"2024-03-18T18:49:48.839404+0000","last_clean":"2024-03-18T18:49:48.839404+0000","last_became_active":"2024-03-18T18:49:44.840477+0000","last_became_peered":"2024-03-18T18:49:44.840477+0000","last_unstale":"2024-03-18T18:49:48.839404+0000","last_undegraded":"2024-03-18T18:49:48.839404+0000","last_fullsized":"2024-03-18T18:49:48.839404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:20.816344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.846309+0000","last_change":"2024-03-18T18:49:44.838132+0000","last_active":"2024-03-18T18:49:48.846309+0000","last_peered":"2024-03-18T18:49:48.846309+0000","last_clean":"2024-03-18T18:49:48.846309+0000","last_became_active":"2024-03-18T18:49:44.837914+0000","last_became_peered":"2024-03-18T18:49:44.837914+0000","last_unstale":"2024-03-18T18:49:48.846309+0000","last_undegraded":"2024-03-18T18:49:48.846309+0000","last_fullsized":"2024-03-18T18:49:48.846309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840730+0000","last_change":"2024-03-18T18:49:44.835464+0000","last_active":"2024-03-18T18:49:48.840730+0000","last_peered":"2024-03-18T18:49:48.840730+0000","last_clean":"2024-03-18T18:49:48.840730+0000","last_became_active":"2024-03-18T18:49:44.835190+0000","last_became_peered":"2024-03-18T18:49:44.835190+0000","last_unstale":"2024-03-18T18:49:48.840730+0000","last_undegraded":"2024-03-18T18:49:48.840730+0000","last_fullsized":"2024-03-18T18:49:48.840730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841309+0000","last_change":"2024-03-18T18:49:44.819962+0000","last_active":"2024-03-18T18:49:48.841309+0000","last_peered":"2024-03-18T18:49:48.841309+0000","last_clean":"2024-03-18T18:49:48.841309+0000","last_became_active":"2024-03-18T18:49:44.819698+0000","last_became_peered":"2024-03-18T18:49:44.819698+0000","last_unstale":"2024-03-18T18:49:48.841309+0000","last_undegraded":"2024-03-18T18:49:48.841309+0000","last_fullsized":"2024-03-18T18:49:48.841309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835920+0000","last_change":"2024-03-18T18:49:44.830928+0000","last_active":"2024-03-18T18:49:48.835920+0000","last_peered":"2024-03-18T18:49:48.835920+0000","last_clean":"2024-03-18T18:49:48.835920+0000","last_became_active":"2024-03-18T18:49:44.830658+0000","last_became_peered":"2024-03-18T18:49:44.830658+0000","last_unstale":"2024-03-18T18:49:48.835920+0000","last_undegraded":"2024-03-18T18:49:48.835920+0000","last_fullsized":"2024-03-18T18:49:48.835920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842763+0000","last_change":"2024-03-18T18:49:44.838837+0000","last_active":"2024-03-18T18:49:48.842763+0000","last_peered":"2024-03-18T18:49:48.842763+0000","last_clean":"2024-03-18T18:49:48.842763+0000","last_became_active":"2024-03-18T18:49:44.838564+0000","last_became_peered":"2024-03-18T18:49:44.838564+0000","last_unstale":"2024-03-18T18:49:48.842763+0000","last_undegraded":"2024-03-18T18:49:48.842763+0000","last_fullsized":"2024-03-18T18:49:48.842763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:34.102274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839856+0000","last_change":"2024-03-18T18:49:44.841576+0000","last_active":"2024-03-18T18:49:48.839856+0000","last_peered":"2024-03-18T18:49:48.839856+0000","last_clean":"2024-03-18T18:49:48.839856+0000","last_became_active":"2024-03-18T18:49:44.840814+0000","last_became_peered":"2024-03-18T18:49:44.840814+0000","last_unstale":"2024-03-18T18:49:48.839856+0000","last_undegraded":"2024-03-18T18:49:48.839856+0000","last_fullsized":"2024-03-18T18:49:48.839856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:02:06.725321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840423+0000","last_change":"2024-03-18T18:49:44.840212+0000","last_active":"2024-03-18T18:49:48.840423+0000","last_peered":"2024-03-18T18:49:48.840423+0000","last_clean":"2024-03-18T18:49:48.840423+0000","last_became_active":"2024-03-18T18:49:44.839960+0000","last_became_peered":"2024-03-18T18:49:44.839960+0000","last_unstale":"2024-03-18T18:49:48.840423+0000","last_undegraded":"2024-03-18T18:49:48.840423+0000","last_fullsized":"2024-03-18T18:49:48.840423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839485+0000","last_change":"2024-03-18T18:49:44.841770+0000","last_active":"2024-03-18T18:49:48.839485+0000","last_peered":"2024-03-18T18:49:48.839485+0000","last_clean":"2024-03-18T18:49:48.839485+0000","last_became_active":"2024-03-18T18:49:44.840939+0000","last_became_peered":"2024-03-18T18:49:44.840939+0000","last_unstale":"2024-03-18T18:49:48.839485+0000","last_undegraded":"2024-03-18T18:49:48.839485+0000","last_fullsized":"2024-03-18T18:49:48.839485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:20:43.395005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.846397+0000","last_change":"2024-03-18T18:49:44.841290+0000","last_active":"2024-03-18T18:49:48.846397+0000","last_peered":"2024-03-18T18:49:48.846397+0000","last_clean":"2024-03-18T18:49:48.846397+0000","last_became_active":"2024-03-18T18:49:44.840949+0000","last_became_peered":"2024-03-18T18:49:44.840949+0000","last_unstale":"2024-03-18T18:49:48.846397+0000","last_undegraded":"2024-03-18T18:49:48.846397+0000","last_fullsized":"2024-03-18T18:49:48.846397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:05.661348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835876+0000","last_change":"2024-03-18T18:49:44.837671+0000","last_active":"2024-03-18T18:49:48.835876+0000","last_peered":"2024-03-18T18:49:48.835876+0000","last_clean":"2024-03-18T18:49:48.835876+0000","last_became_active":"2024-03-18T18:49:44.837265+0000","last_became_peered":"2024-03-18T18:49:44.837265+0000","last_unstale":"2024-03-18T18:49:48.835876+0000","last_undegraded":"2024-03-18T18:49:48.835876+0000","last_fullsized":"2024-03-18T18:49:48.835876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:14.055199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4593,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":695401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":104611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":695457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.19600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":104451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":104577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":104541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T18:50:09.445 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T18:50:09.445 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-18T18:50:09.445 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-18T18:50:09.445 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-18T18:50:09.446 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-18T18:50:09.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.607+0000 7f68fef95700 1 Processor -- start 2024-03-18T18:50:09.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.608+0000 7f68fef95700 1 -- start start 2024-03-18T18:50:09.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.608+0000 7f68fef95700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f68f80ca5f0 0x7f68f806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:09.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.608+0000 7f68fef95700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f68f80ace80 0x7f68f80ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:09.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.608+0000 7f68fef95700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68f80caa70 con 0x7f68f806a750 2024-03-18T18:50:09.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.608+0000 7f68fef95700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68f806e1f0 con 0x7f68f80ace80 2024-03-18T18:50:09.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.608+0000 7f68fef95700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68f806e4d0 con 0x7f68f80ca5f0 2024-03-18T18:50:09.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.608+0000 7f68fd532700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f68f80ace80 0x7f68f80ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:09.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.608+0000 7f68fcd31700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f68f806a750 0x7f68f806ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60228/0 (socket says 172.21.15.1:60228) 2024-03-18T18:50:09.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.608+0000 7f68fcd31700 1 -- 172.21.15.1:0/1179804828 learned_addr learned my addr 172.21.15.1:0/1179804828 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:09.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.609+0000 7f68f7fff700 1 --2- 172.21.15.1:0/1179804828 >> v2:172.21.15.165:3300/0 conn(0x7f68f80ca5f0 0x7f68f806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:09.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.609+0000 7f68fd532700 1 -- 172.21.15.1:0/1179804828 >> v2:172.21.15.165:3300/0 conn(0x7f68f80ca5f0 msgr2=0x7f68f806ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:09.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.609+0000 7f68fd532700 1 --2- 172.21.15.1:0/1179804828 >> v2:172.21.15.165:3300/0 conn(0x7f68f80ca5f0 0x7f68f806ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.609+0000 7f68fd532700 1 -- 172.21.15.1:0/1179804828 >> v1:172.21.15.1:6789/0 conn(0x7f68f806a750 legacy=0x7f68f806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:09.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.609+0000 7f68fd532700 1 -- 172.21.15.1:0/1179804828 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f80cc2a0 con 0x7f68f80ace80 2024-03-18T18:50:09.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.609+0000 7f68f77fe700 1 -- 172.21.15.1:0/1179804828 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159428320 0 0) 0x7f68f80caa70 con 0x7f68f806a750 2024-03-18T18:50:09.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.609+0000 7f68fd532700 1 --2- 172.21.15.1:0/1179804828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f80ace80 0x7f68f80ad310 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f68f80701b0 tx=0x7f68f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c498ed0b9584c1f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:09.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.609+0000 7f68f77fe700 1 -- 172.21.15.1:0/1179804828 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f000e070 con 0x7f68f80ace80 2024-03-18T18:50:09.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.609+0000 7f68f77fe700 1 -- 172.21.15.1:0/1179804828 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f000bbd0 con 0x7f68f80ace80 2024-03-18T18:50:09.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.609+0000 7f68f77fe700 1 -- 172.21.15.1:0/1179804828 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f0012430 con 0x7f68f80ace80 2024-03-18T18:50:09.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.610+0000 7f68fef95700 1 -- 172.21.15.1:0/1179804828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f80ace80 msgr2=0x7f68f80ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:09.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.610+0000 7f68fef95700 1 --2- 172.21.15.1:0/1179804828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f80ace80 0x7f68f80ad310 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f68f80701b0 tx=0x7f68f000b330 comp rx=0 tx=0).stop 2024-03-18T18:50:09.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.610+0000 7f68fef95700 1 -- 172.21.15.1:0/1179804828 shutdown_connections 2024-03-18T18:50:09.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.610+0000 7f68fef95700 1 --2- 172.21.15.1:0/1179804828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f80ace80 0x7f68f80ad310 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.610+0000 7f68fef95700 1 --2- 172.21.15.1:0/1179804828 >> v2:172.21.15.165:3300/0 conn(0x7f68f80ca5f0 0x7f68f806ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.610+0000 7f68fef95700 1 -- 172.21.15.1:0/1179804828 >> 172.21.15.1:0/1179804828 conn(0x7f68f80cc990 msgr2=0x7f68f80ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:09.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.610+0000 7f68fef95700 1 -- 172.21.15.1:0/1179804828 shutdown_connections 2024-03-18T18:50:09.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.610+0000 7f68fef95700 1 -- 172.21.15.1:0/1179804828 wait complete. 2024-03-18T18:50:09.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.611+0000 7f68fef95700 1 Processor -- start 2024-03-18T18:50:09.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.611+0000 7f68fef95700 1 -- start start 2024-03-18T18:50:09.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.612+0000 7f68fef95700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f806a750 0x7f68f8150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:09.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.612+0000 7f68fef95700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68f80ace80 0x7f68f815a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:09.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.612+0000 7f68fef95700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68f80ca5f0 0x7f68f80c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:09.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.612+0000 7f68fef95700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f80ad600 con 0x7f68f806a750 2024-03-18T18:50:09.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.612+0000 7f68fef95700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f8159000 con 0x7f68f80ace80 2024-03-18T18:50:09.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.612+0000 7f68fef95700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f81592e0 con 0x7f68f80ca5f0 2024-03-18T18:50:09.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.612+0000 7f68fcd31700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f806a750 0x7f68f8150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:09.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.612+0000 7f68fcd31700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f806a750 0x7f68f8150420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45122/0 (socket says 172.21.15.1:45122) 2024-03-18T18:50:09.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.612+0000 7f68fcd31700 1 -- 172.21.15.1:0/2980165837 learned_addr learned my addr 172.21.15.1:0/2980165837 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:09.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.612+0000 7f68f7fff700 1 --2- 172.21.15.1:0/2980165837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68f80ace80 0x7f68f815a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:09.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.612+0000 7f68fd532700 1 --2- 172.21.15.1:0/2980165837 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68f80ca5f0 0x7f68f80c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:09.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.612+0000 7f68fcd31700 1 -- 172.21.15.1:0/2980165837 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68f80ca5f0 msgr2=0x7f68f80c5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:09.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.612+0000 7f68fcd31700 1 --2- 172.21.15.1:0/2980165837 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68f80ca5f0 0x7f68f80c5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.612+0000 7f68fcd31700 1 -- 172.21.15.1:0/2980165837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68f80ace80 msgr2=0x7f68f815a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:09.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.612+0000 7f68fcd31700 1 --2- 172.21.15.1:0/2980165837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68f80ace80 0x7f68f815a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.612+0000 7f68fcd31700 1 -- 172.21.15.1:0/2980165837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f0009a70 con 0x7f68f806a750 2024-03-18T18:50:09.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.612+0000 7f68fcd31700 1 --2- 172.21.15.1:0/2980165837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f806a750 0x7f68f8150420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f68ec0011e0 tx=0x7f68ec00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:09.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.613+0000 7f68f57fa700 1 -- 172.21.15.1:0/2980165837 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec004c80 con 0x7f68f806a750 2024-03-18T18:50:09.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.613+0000 7f68f57fa700 1 -- 172.21.15.1:0/2980165837 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ec004e00 con 0x7f68f806a750 2024-03-18T18:50:09.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.613+0000 7f68fef95700 1 -- 172.21.15.1:0/2980165837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f81595c0 con 0x7f68f806a750 2024-03-18T18:50:09.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.613+0000 7f68f57fa700 1 -- 172.21.15.1:0/2980165837 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec007310 con 0x7f68f806a750 2024-03-18T18:50:09.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.613+0000 7f68fef95700 1 -- 172.21.15.1:0/2980165837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f8159b80 con 0x7f68f806a750 2024-03-18T18:50:09.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.614+0000 7f68fef95700 1 -- 172.21.15.1:0/2980165837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f8159da0 con 0x7f68f806a750 2024-03-18T18:50:09.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.614+0000 7f68f57fa700 1 -- 172.21.15.1:0/2980165837 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f68ec007470 con 0x7f68f806a750 2024-03-18T18:50:09.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.614+0000 7f68f57fa700 1 --2- 172.21.15.1:0/2980165837 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f68dc06ac90 0x7f68dc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:09.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.615+0000 7f68f57fa700 1 -- 172.21.15.1:0/2980165837 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f68ec0893a0 con 0x7f68f806a750 2024-03-18T18:50:09.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.615+0000 7f68f7fff700 1 --2- 172.21.15.1:0/2980165837 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f68dc06ac90 0x7f68dc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:09.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.615+0000 7f68f7fff700 1 --2- 172.21.15.1:0/2980165837 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f68dc06ac90 0x7f68dc06d150 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f68f81f6b50 tx=0x7f68e4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:09.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.619+0000 7f68f57fa700 1 -- 172.21.15.1:0/2980165837 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f68ec054290 con 0x7f68f806a750 2024-03-18T18:50:09.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.881+0000 7f68fef95700 1 -- 172.21.15.1:0/2980165837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f68f80676a0 con 0x7f68f806a750 2024-03-18T18:50:09.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.884+0000 7f68f57fa700 1 -- 172.21.15.1:0/2980165837 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+170918 (secure 0 0 0) 0x7f68ec0578b0 con 0x7f68f806a750 2024-03-18T18:50:09.882 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:09.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.887+0000 7f68fef95700 1 -- 172.21.15.1:0/2980165837 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f68dc06ac90 msgr2=0x7f68dc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:09.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.887+0000 7f68fef95700 1 --2- 172.21.15.1:0/2980165837 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f68dc06ac90 0x7f68dc06d150 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f68f81f6b50 tx=0x7f68e4008040 comp rx=0 tx=0).stop 2024-03-18T18:50:09.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.887+0000 7f68fef95700 1 -- 172.21.15.1:0/2980165837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f806a750 msgr2=0x7f68f8150420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:09.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.887+0000 7f68fef95700 1 --2- 172.21.15.1:0/2980165837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f806a750 0x7f68f8150420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f68ec0011e0 tx=0x7f68ec00cae0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.887+0000 7f68fef95700 1 -- 172.21.15.1:0/2980165837 shutdown_connections 2024-03-18T18:50:09.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.887+0000 7f68fef95700 1 --2- 172.21.15.1:0/2980165837 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f68dc06ac90 0x7f68dc06d150 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.887+0000 7f68fef95700 1 --2- 172.21.15.1:0/2980165837 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68f80ca5f0 0x7f68f80c5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.887+0000 7f68fef95700 1 --2- 172.21.15.1:0/2980165837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f806a750 0x7f68f8150420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.887+0000 7f68fef95700 1 --2- 172.21.15.1:0/2980165837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68f80ace80 0x7f68f815a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.887+0000 7f68fef95700 1 -- 172.21.15.1:0/2980165837 >> 172.21.15.1:0/2980165837 conn(0x7f68f80cc990 msgr2=0x7f68f80c7180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.887+0000 7f68fef95700 1 -- 172.21.15.1:0/2980165837 shutdown_connections 2024-03-18T18:50:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:09.887+0000 7f68fef95700 1 -- 172.21.15.1:0/2980165837 wait complete. 2024-03-18T18:50:09.896 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":5,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":76998},{"type":"v1","addr":"172.21.15.1:6833","nonce":76998}]},"active_addr":"172.21.15.1:6833/76998","active_change":"2024-03-18T18:49:33.379710+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4109,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":72803206}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":4026407635}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":2322425425}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":1992330207}]}]} 2024-03-18T18:50:09.901 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-18T18:50:09.901 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-18T18:50:09.901 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:10.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.030+0000 7fbfe0e38700 1 Processor -- start 2024-03-18T18:50:10.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.030+0000 7fbfe0e38700 1 -- start start 2024-03-18T18:50:10.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.031+0000 7fbfe0e38700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfdc146190 0x7fbfdc06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.031+0000 7fbfe0e38700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbfdc1465a0 0x7fbfdc06ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.031+0000 7fbfe0e38700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc06ebd0 con 0x7fbfdc146af0 2024-03-18T18:50:10.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.031+0000 7fbfe0e38700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfdc06f850 con 0x7fbfdc1465a0 2024-03-18T18:50:10.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.031+0000 7fbfe0e38700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc06eeb0 con 0x7fbfdc146190 2024-03-18T18:50:10.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.031+0000 7fbfd9d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbfdc1465a0 0x7fbfdc06ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.031+0000 7fbfd9d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbfdc1465a0 0x7fbfdc06ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45124/0 (socket says 172.21.15.1:45124) 2024-03-18T18:50:10.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.031+0000 7fbfd9d9b700 1 -- 172.21.15.1:0/778062776 learned_addr learned my addr 172.21.15.1:0/778062776 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.032+0000 7fbfda59c700 1 --2- 172.21.15.1:0/778062776 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfdc146190 0x7fbfdc06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.032+0000 7fbfd9d9b700 1 -- 172.21.15.1:0/778062776 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfdc146190 msgr2=0x7fbfdc06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.032+0000 7fbfd9d9b700 1 --2- 172.21.15.1:0/778062776 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfdc146190 0x7fbfdc06a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.032+0000 7fbfd9d9b700 1 -- 172.21.15.1:0/778062776 >> v1:172.21.15.165:6789/0 conn(0x7fbfdc146af0 legacy=0x7fbfdc071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.032+0000 7fbfd9d9b700 1 -- 172.21.15.1:0/778062776 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc06f190 con 0x7fbfdc1465a0 2024-03-18T18:50:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.032+0000 7fbfd9d9b700 1 --2- 172.21.15.1:0/778062776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfdc1465a0 0x7fbfdc06ac90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc0098f0 tx=0x7fbfcc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a0703a71ce62145 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.032+0000 7fbfd959a700 1 -- 172.21.15.1:0/778062776 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc004230 con 0x7fbfdc1465a0 2024-03-18T18:50:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.032+0000 7fbfd959a700 1 -- 172.21.15.1:0/778062776 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfcc004390 con 0x7fbfdc1465a0 2024-03-18T18:50:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.032+0000 7fbfd959a700 1 -- 172.21.15.1:0/778062776 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc0036a0 con 0x7fbfdc1465a0 2024-03-18T18:50:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.032+0000 7fbfe0e38700 1 -- 172.21.15.1:0/778062776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfdc1465a0 msgr2=0x7fbfdc06ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.032+0000 7fbfe0e38700 1 --2- 172.21.15.1:0/778062776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfdc1465a0 0x7fbfdc06ac90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc0098f0 tx=0x7fbfcc009c00 comp rx=0 tx=0).stop 2024-03-18T18:50:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.033+0000 7fbfe0e38700 1 -- 172.21.15.1:0/778062776 shutdown_connections 2024-03-18T18:50:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.033+0000 7fbfe0e38700 1 --2- 172.21.15.1:0/778062776 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfdc146190 0x7fbfdc06a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.033+0000 7fbfe0e38700 1 --2- 172.21.15.1:0/778062776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfdc1465a0 0x7fbfdc06ac90 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.033+0000 7fbfe0e38700 1 -- 172.21.15.1:0/778062776 >> 172.21.15.1:0/778062776 conn(0x7fbfdc0b1d20 msgr2=0x7fbfdc0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.033+0000 7fbfe0e38700 1 -- 172.21.15.1:0/778062776 shutdown_connections 2024-03-18T18:50:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.033+0000 7fbfe0e38700 1 -- 172.21.15.1:0/778062776 wait complete. 2024-03-18T18:50:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.034+0000 7fbfe0e38700 1 Processor -- start 2024-03-18T18:50:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.034+0000 7fbfe0e38700 1 -- start start 2024-03-18T18:50:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.035+0000 7fbfe0e38700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfdc1465a0 0x7fbfdc1e3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.035+0000 7fbfe0e38700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfdc146af0 0x7fbfdc1e6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.035+0000 7fbfe0e38700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfdc0c59a0 0x7fbfdc0cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.035+0000 7fbfe0e38700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc0c5f10 con 0x7fbfdc146af0 2024-03-18T18:50:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.035+0000 7fbfe0e38700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc0c61f0 con 0x7fbfdc0c59a0 2024-03-18T18:50:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.035+0000 7fbfe0e38700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc0c64d0 con 0x7fbfdc1465a0 2024-03-18T18:50:10.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.035+0000 7fbfd9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfdc146af0 0x7fbfdc1e6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.035+0000 7fbfd9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfdc146af0 0x7fbfdc1e6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45136/0 (socket says 172.21.15.1:45136) 2024-03-18T18:50:10.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.035+0000 7fbfd9d9b700 1 -- 172.21.15.1:0/2653824692 learned_addr learned my addr 172.21.15.1:0/2653824692 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:10.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.036+0000 7fbfdad9d700 1 --2- 172.21.15.1:0/2653824692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfdc0c59a0 0x7fbfdc0cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.036+0000 7fbfda59c700 1 --2- 172.21.15.1:0/2653824692 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfdc1465a0 0x7fbfdc1e3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.036+0000 7fbfd9d9b700 1 -- 172.21.15.1:0/2653824692 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfdc1465a0 msgr2=0x7fbfdc1e3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.036+0000 7fbfd9d9b700 1 --2- 172.21.15.1:0/2653824692 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfdc1465a0 0x7fbfdc1e3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.036+0000 7fbfd9d9b700 1 -- 172.21.15.1:0/2653824692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfdc0c59a0 msgr2=0x7fbfdc0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.036+0000 7fbfd9d9b700 1 --2- 172.21.15.1:0/2653824692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfdc0c59a0 0x7fbfdc0cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.036+0000 7fbfd9d9b700 1 -- 172.21.15.1:0/2653824692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfcc009550 con 0x7fbfdc146af0 2024-03-18T18:50:10.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.036+0000 7fbfd9d9b700 1 --2- 172.21.15.1:0/2653824692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfdc146af0 0x7fbfdc1e6b40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc009130 tx=0x7fbfcc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.036+0000 7fbfcaffd700 1 -- 172.21.15.1:0/2653824692 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc003d10 con 0x7fbfdc146af0 2024-03-18T18:50:10.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.036+0000 7fbfcaffd700 1 -- 172.21.15.1:0/2653824692 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfcc01f070 con 0x7fbfdc146af0 2024-03-18T18:50:10.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.036+0000 7fbfe0e38700 1 -- 172.21.15.1:0/2653824692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfdc0c67b0 con 0x7fbfdc146af0 2024-03-18T18:50:10.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.036+0000 7fbfcaffd700 1 -- 172.21.15.1:0/2653824692 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc00b8a0 con 0x7fbfdc146af0 2024-03-18T18:50:10.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.036+0000 7fbfe0e38700 1 -- 172.21.15.1:0/2653824692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfdc1e7080 con 0x7fbfdc146af0 2024-03-18T18:50:10.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.037+0000 7fbfe0e38700 1 -- 172.21.15.1:0/2653824692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfdc0c6cd0 con 0x7fbfdc146af0 2024-03-18T18:50:10.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.038+0000 7fbfcaffd700 1 -- 172.21.15.1:0/2653824692 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fbfcc021030 con 0x7fbfdc146af0 2024-03-18T18:50:10.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.038+0000 7fbfcaffd700 1 --2- 172.21.15.1:0/2653824692 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbfbc06ac90 0x7fbfbc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.038+0000 7fbfcaffd700 1 -- 172.21.15.1:0/2653824692 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbfcc08ab70 con 0x7fbfdc146af0 2024-03-18T18:50:10.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.038+0000 7fbfda59c700 1 --2- 172.21.15.1:0/2653824692 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbfbc06ac90 0x7fbfbc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.038+0000 7fbfda59c700 1 --2- 172.21.15.1:0/2653824692 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbfbc06ac90 0x7fbfbc06d150 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000ab60 tx=0x7fbfd0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.043+0000 7fbfcaffd700 1 -- 172.21.15.1:0/2653824692 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbfdc0c6cd0 con 0x7fbfdc146af0 2024-03-18T18:50:10.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.248+0000 7fbfe0e38700 1 -- 172.21.15.1:0/2653824692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfdc0c6cd0 con 0x7fbfdc146af0 2024-03-18T18:50:10.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.249+0000 7fbfcaffd700 1 -- 172.21.15.1:0/2653824692 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17830 (secure 0 0 0) 0x7fbfdc0c6cd0 con 0x7fbfdc146af0 2024-03-18T18:50:10.246 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:10.246 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:49.276961+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.251+0000 7fbfe0e38700 1 -- 172.21.15.1:0/2653824692 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbfbc06ac90 msgr2=0x7fbfbc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.251+0000 7fbfe0e38700 1 --2- 172.21.15.1:0/2653824692 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbfbc06ac90 0x7fbfbc06d150 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000ab60 tx=0x7fbfd0008040 comp rx=0 tx=0).stop 2024-03-18T18:50:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.251+0000 7fbfe0e38700 1 -- 172.21.15.1:0/2653824692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfdc146af0 msgr2=0x7fbfdc1e6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.251+0000 7fbfe0e38700 1 --2- 172.21.15.1:0/2653824692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfdc146af0 0x7fbfdc1e6b40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc009130 tx=0x7fbfcc004750 comp rx=0 tx=0).stop 2024-03-18T18:50:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.251+0000 7fbfe0e38700 1 -- 172.21.15.1:0/2653824692 shutdown_connections 2024-03-18T18:50:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.251+0000 7fbfe0e38700 1 --2- 172.21.15.1:0/2653824692 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbfbc06ac90 0x7fbfbc06d150 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.251+0000 7fbfe0e38700 1 --2- 172.21.15.1:0/2653824692 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfdc1465a0 0x7fbfdc1e3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.251+0000 7fbfe0e38700 1 --2- 172.21.15.1:0/2653824692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfdc0c59a0 0x7fbfdc0cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.251+0000 7fbfe0e38700 1 --2- 172.21.15.1:0/2653824692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfdc146af0 0x7fbfdc1e6b40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.251+0000 7fbfe0e38700 1 -- 172.21.15.1:0/2653824692 >> 172.21.15.1:0/2653824692 conn(0x7fbfdc0b1d20 msgr2=0x7fbfdc0ae650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.251+0000 7fbfe0e38700 1 -- 172.21.15.1:0/2653824692 shutdown_connections 2024-03-18T18:50:10.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.251+0000 7fbfe0e38700 1 -- 172.21.15.1:0/2653824692 wait complete. 2024-03-18T18:50:10.260 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-18T18:50:10.262 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:10.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.391+0000 7efffdaed700 1 Processor -- start 2024-03-18T18:50:10.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.391+0000 7efffdaed700 1 -- start start 2024-03-18T18:50:10.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.391+0000 7efffdaed700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efff8071600 0x7efff8071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.391+0000 7efffdaed700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7efff806e2c0 0x7efff806e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.391+0000 7efffdaed700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff80acd80 con 0x7efff815a320 2024-03-18T18:50:10.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.391+0000 7efffdaed700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efff8071fc0 con 0x7efff8071600 2024-03-18T18:50:10.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.391+0000 7efffdaed700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efff80cf220 con 0x7efff806e2c0 2024-03-18T18:50:10.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.391+0000 7efff6ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efff8071600 0x7efff8071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.391+0000 7efff77fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efff815a320 0x7efff80ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60240/0 (socket says 172.21.15.1:60240) 2024-03-18T18:50:10.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.391+0000 7efff77fe700 1 -- 172.21.15.1:0/3296151042 learned_addr learned my addr 172.21.15.1:0/3296151042 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:10.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.392+0000 7efff7fff700 1 --2- 172.21.15.1:0/3296151042 >> v2:172.21.15.165:3300/0 conn(0x7efff806e2c0 0x7efff806e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.392+0000 7efff6ffd700 1 -- 172.21.15.1:0/3296151042 >> v2:172.21.15.165:3300/0 conn(0x7efff806e2c0 msgr2=0x7efff806e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:10.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.392+0000 7efff6ffd700 1 --2- 172.21.15.1:0/3296151042 >> v2:172.21.15.165:3300/0 conn(0x7efff806e2c0 0x7efff806e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.392+0000 7efff6ffd700 1 -- 172.21.15.1:0/3296151042 >> v1:172.21.15.1:6789/0 conn(0x7efff815a320 legacy=0x7efff80ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.392+0000 7efff6ffd700 1 -- 172.21.15.1:0/3296151042 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff80cef40 con 0x7efff8071600 2024-03-18T18:50:10.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.392+0000 7efff6ffd700 1 --2- 172.21.15.1:0/3296151042 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff8071600 0x7efff8071a80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efff80b2950 tx=0x7effec009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f94c32440de8549 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.392+0000 7efff67fc700 1 -- 172.21.15.1:0/3296151042 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effec011830 con 0x7efff8071600 2024-03-18T18:50:10.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.392+0000 7efff67fc700 1 -- 172.21.15.1:0/3296151042 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effec011990 con 0x7efff8071600 2024-03-18T18:50:10.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.392+0000 7efff67fc700 1 -- 172.21.15.1:0/3296151042 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effec011d20 con 0x7efff8071600 2024-03-18T18:50:10.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.393+0000 7efffdaed700 1 -- 172.21.15.1:0/3296151042 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff8071600 msgr2=0x7efff8071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.393+0000 7efffdaed700 1 --2- 172.21.15.1:0/3296151042 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff8071600 0x7efff8071a80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efff80b2950 tx=0x7effec009b20 comp rx=0 tx=0).stop 2024-03-18T18:50:10.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.393+0000 7efffdaed700 1 -- 172.21.15.1:0/3296151042 shutdown_connections 2024-03-18T18:50:10.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.393+0000 7efffdaed700 1 --2- 172.21.15.1:0/3296151042 >> v2:172.21.15.165:3300/0 conn(0x7efff806e2c0 0x7efff806e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.393+0000 7efffdaed700 1 --2- 172.21.15.1:0/3296151042 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff8071600 0x7efff8071a80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.393+0000 7efffdaed700 1 -- 172.21.15.1:0/3296151042 >> 172.21.15.1:0/3296151042 conn(0x7efff80cc990 msgr2=0x7efff80ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.393+0000 7efffdaed700 1 -- 172.21.15.1:0/3296151042 shutdown_connections 2024-03-18T18:50:10.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.393+0000 7efffdaed700 1 -- 172.21.15.1:0/3296151042 wait complete. 2024-03-18T18:50:10.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.394+0000 7efffdaed700 1 Processor -- start 2024-03-18T18:50:10.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.394+0000 7efffdaed700 1 -- start start 2024-03-18T18:50:10.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.395+0000 7efffdaed700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efff806e2c0 0x7efff8159260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.395+0000 7efffdaed700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efff8071600 0x7efff80c6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.395+0000 7efffdaed700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff81e6ae0 0x7efff81e6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.395+0000 7efffdaed700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff81e74b0 con 0x7efff81e6ae0 2024-03-18T18:50:10.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.395+0000 7efffdaed700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff80adcb0 con 0x7efff806e2c0 2024-03-18T18:50:10.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.395+0000 7efffdaed700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efff80adf90 con 0x7efff8071600 2024-03-18T18:50:10.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.395+0000 7efff7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff81e6ae0 0x7efff81e6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.395+0000 7efff7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff81e6ae0 0x7efff81e6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45140/0 (socket says 172.21.15.1:45140) 2024-03-18T18:50:10.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.395+0000 7efff7fff700 1 -- 172.21.15.1:0/153509039 learned_addr learned my addr 172.21.15.1:0/153509039 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:10.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.395+0000 7efff77fe700 1 --2- 172.21.15.1:0/153509039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efff806e2c0 0x7efff8159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.395+0000 7efff6ffd700 1 --2- 172.21.15.1:0/153509039 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efff8071600 0x7efff80c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.395+0000 7efff7fff700 1 -- 172.21.15.1:0/153509039 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efff8071600 msgr2=0x7efff80c6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.395+0000 7efff7fff700 1 --2- 172.21.15.1:0/153509039 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efff8071600 0x7efff80c6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.395+0000 7efff7fff700 1 -- 172.21.15.1:0/153509039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efff806e2c0 msgr2=0x7efff8159260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.395+0000 7efff7fff700 1 --2- 172.21.15.1:0/153509039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efff806e2c0 0x7efff8159260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.395+0000 7efff7fff700 1 -- 172.21.15.1:0/153509039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effec0094a0 con 0x7efff81e6ae0 2024-03-18T18:50:10.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.395+0000 7efff77fe700 1 --2- 172.21.15.1:0/153509039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efff806e2c0 0x7efff8159260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:10.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.395+0000 7efff7fff700 1 --2- 172.21.15.1:0/153509039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff81e6ae0 0x7efff81e6f70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efff0007e60 tx=0x7efff000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.395+0000 7effdffff700 1 -- 172.21.15.1:0/153509039 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff000ede0 con 0x7efff81e6ae0 2024-03-18T18:50:10.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.396+0000 7effdffff700 1 -- 172.21.15.1:0/153509039 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff0014420 con 0x7efff81e6ae0 2024-03-18T18:50:10.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.396+0000 7effdffff700 1 -- 172.21.15.1:0/153509039 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff00147d0 con 0x7efff81e6ae0 2024-03-18T18:50:10.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.396+0000 7efffdaed700 1 -- 172.21.15.1:0/153509039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff80ae270 con 0x7efff81e6ae0 2024-03-18T18:50:10.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.396+0000 7efffdaed700 1 -- 172.21.15.1:0/153509039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efff80ae830 con 0x7efff81e6ae0 2024-03-18T18:50:10.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.397+0000 7efffdaed700 1 -- 172.21.15.1:0/153509039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff80aea20 con 0x7efff81e6ae0 2024-03-18T18:50:10.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.397+0000 7effdffff700 1 -- 172.21.15.1:0/153509039 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7efff0014930 con 0x7efff81e6ae0 2024-03-18T18:50:10.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.397+0000 7effdffff700 1 --2- 172.21.15.1:0/153509039 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effe406ac90 0x7effe406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.398+0000 7effdffff700 1 -- 172.21.15.1:0/153509039 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7efff0089da0 con 0x7efff81e6ae0 2024-03-18T18:50:10.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.398+0000 7efff77fe700 1 --2- 172.21.15.1:0/153509039 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effe406ac90 0x7effe406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.398+0000 7efff77fe700 1 --2- 172.21.15.1:0/153509039 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effe406ac90 0x7effe406d150 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7effe8007fd0 tx=0x7effe8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.402+0000 7effdffff700 1 -- 172.21.15.1:0/153509039 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efff0054c90 con 0x7efff81e6ae0 2024-03-18T18:50:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.602+0000 7efffdaed700 1 -- 172.21.15.1:0/153509039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efff80676a0 con 0x7efff81e6ae0 2024-03-18T18:50:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.603+0000 7effdffff700 1 -- 172.21.15.1:0/153509039 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17830 (secure 0 0 0) 0x7efff00582b0 con 0x7efff81e6ae0 2024-03-18T18:50:10.600 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:10.601 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:49.276961+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.605+0000 7efffdaed700 1 -- 172.21.15.1:0/153509039 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effe406ac90 msgr2=0x7effe406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.605+0000 7efffdaed700 1 --2- 172.21.15.1:0/153509039 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effe406ac90 0x7effe406d150 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7effe8007fd0 tx=0x7effe8009040 comp rx=0 tx=0).stop 2024-03-18T18:50:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.605+0000 7efffdaed700 1 -- 172.21.15.1:0/153509039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff81e6ae0 msgr2=0x7efff81e6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.605+0000 7efffdaed700 1 --2- 172.21.15.1:0/153509039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff81e6ae0 0x7efff81e6f70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efff0007e60 tx=0x7efff000e3c0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.605+0000 7efffdaed700 1 -- 172.21.15.1:0/153509039 shutdown_connections 2024-03-18T18:50:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.605+0000 7efffdaed700 1 --2- 172.21.15.1:0/153509039 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effe406ac90 0x7effe406d150 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.605+0000 7efffdaed700 1 --2- 172.21.15.1:0/153509039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efff81e6ae0 0x7efff81e6f70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.605+0000 7efffdaed700 1 --2- 172.21.15.1:0/153509039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efff806e2c0 0x7efff8159260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.605+0000 7efffdaed700 1 --2- 172.21.15.1:0/153509039 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efff8071600 0x7efff80c6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.605+0000 7efffdaed700 1 -- 172.21.15.1:0/153509039 >> 172.21.15.1:0/153509039 conn(0x7efff80cc990 msgr2=0x7efff80af9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.606+0000 7efffdaed700 1 -- 172.21.15.1:0/153509039 shutdown_connections 2024-03-18T18:50:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.606+0000 7efffdaed700 1 -- 172.21.15.1:0/153509039 wait complete. 2024-03-18T18:50:10.615 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-18T18:50:10.616 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-18T18:50:10.616 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-18T18:50:10.617 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-18T18:50:10.617 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-18T18:50:10.618 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-18T18:50:10.618 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-18T18:50:10.619 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-18T18:50:10.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.841+0000 7f3045d73700 1 Processor -- start 2024-03-18T18:50:10.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.842+0000 7f3045d73700 1 -- start start 2024-03-18T18:50:10.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.843+0000 7f3045d73700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3040070e40 0x7f3040162850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.843+0000 7f3045d73700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304006d810 con 0x7f304006dca0 2024-03-18T18:50:10.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.843+0000 7f3045d73700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30400b31b0 con 0x7f30400b2d60 2024-03-18T18:50:10.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.843+0000 7f3045d73700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3040162d90 con 0x7f3040070e40 2024-03-18T18:50:10.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.843+0000 7f303effd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f304006dca0 0x7f30400b2650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60246/0 (socket says 172.21.15.1:60246) 2024-03-18T18:50:10.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.843+0000 7f303effd700 1 -- 172.21.15.1:0/2491443199 learned_addr learned my addr 172.21.15.1:0/2491443199 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:10.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.844+0000 7fe88c4a0700 1 Processor -- start 2024-03-18T18:50:10.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.844+0000 7fc98d11d700 1 Processor -- start 2024-03-18T18:50:10.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.844+0000 7f31a85f9700 1 Processor -- start 2024-03-18T18:50:10.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.844+0000 7fe7627ab700 1 Processor -- start 2024-03-18T18:50:10.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.844+0000 7fc98d11d700 1 -- start start 2024-03-18T18:50:10.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.844+0000 7fe7627ab700 1 -- start start 2024-03-18T18:50:10.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.844+0000 7f303f7fe700 1 --2- 172.21.15.1:0/2491443199 >> v2:172.21.15.165:3300/0 conn(0x7f3040070e40 0x7f3040162850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.845+0000 7fc98d11d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc988160e50 0x7fc9880cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.845+0000 7fc98d11d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9881612d0 0x7fc9881616f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.845+0000 7fc98d11d700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc988162170 con 0x7fc988160a70 2024-03-18T18:50:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.845+0000 7fc98d11d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc988161c30 con 0x7fc988160e50 2024-03-18T18:50:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.845+0000 7fc98d11d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9880cef30 con 0x7fc9881612d0 2024-03-18T18:50:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.845+0000 7fe7627ab700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe75c0cfb50 0x7fe75c0d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.845+0000 7fe7627ab700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe75c0cff30 0x7fe75c0b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.845+0000 7fe7627ab700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75c0b3370 con 0x7fe75c0b3d40 2024-03-18T18:50:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.845+0000 7fe7627ab700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe75c0b2760 con 0x7fe75c0cff30 2024-03-18T18:50:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.845+0000 7fe7627ab700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe75c0b2a40 con 0x7fe75c0cfb50 2024-03-18T18:50:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.844+0000 7fe88c4a0700 1 -- start start 2024-03-18T18:50:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.845+0000 7fe88c4a0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe88414a480 0x7fe88406cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.845+0000 7fe88c4a0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe8840b3c60 0x7fe8840cc770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.845+0000 7fe88c4a0700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe884070580 con 0x7fe88406d080 2024-03-18T18:50:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.845+0000 7fe88c4a0700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe88406d500 con 0x7fe8840b3c60 2024-03-18T18:50:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.845+0000 7fe88c4a0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840713e0 con 0x7fe88414a480 2024-03-18T18:50:10.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.844+0000 7f31a85f9700 1 -- start start 2024-03-18T18:50:10.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.845+0000 7f31a85f9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f31a0071df0 0x7f31a00721d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.845+0000 7f31a85f9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f31a00b2210 0x7f31a00b2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7f31a85f9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a006d690 con 0x7f31a00ce450 2024-03-18T18:50:10.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7f31a85f9700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31a00ce060 con 0x7f31a0071df0 2024-03-18T18:50:10.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7f31a85f9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a01597c0 con 0x7f31a00b2210 2024-03-18T18:50:10.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fc98759e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9881612d0 0x7fc9881616f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fc986d9d700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fc988160a70 0x7fc9880d3b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:38310/0 (socket says 172.21.15.1:38310) 2024-03-18T18:50:10.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fc986d9d700 1 -- 172.21.15.1:0/1713039032 learned_addr learned my addr 172.21.15.1:0/1713039032 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:10.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fe889a3b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe88406d080 0x7fe8840b3550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60250/0 (socket says 172.21.15.1:60250) 2024-03-18T18:50:10.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fe889a3b700 1 -- 172.21.15.1:0/4176959435 learned_addr learned my addr 172.21.15.1:0/4176959435 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:10.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fe88a23c700 1 --2- 172.21.15.1:0/4176959435 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe88414a480 0x7fe88406cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fe760d48700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fe75c0b3d40 0x7fe75c0b4190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:38314/0 (socket says 172.21.15.1:38314) 2024-03-18T18:50:10.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fe760d48700 1 -- 172.21.15.1:0/3694741955 learned_addr learned my addr 172.21.15.1:0/3694741955 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:10.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fe75b7fe700 1 --2- 172.21.15.1:0/3694741955 >> v2:172.21.15.1:3300/0 conn(0x7fe75c0cff30 0x7fe75c0b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7f303f7fe700 1 -- 172.21.15.1:0/2491443199 >> v1:172.21.15.165:6789/0 conn(0x7f30400b2d60 legacy=0x7f3040071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7f303f7fe700 1 -- 172.21.15.1:0/2491443199 >> v1:172.21.15.1:6789/0 conn(0x7f304006dca0 legacy=0x7f30400b2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7f303f7fe700 1 -- 172.21.15.1:0/2491443199 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304006cee0 con 0x7f3040070e40 2024-03-18T18:50:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fe88a23c700 1 -- 172.21.15.1:0/4176959435 >> v2:172.21.15.1:3300/0 conn(0x7fe8840b3c60 msgr2=0x7fe8840cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fe88a23c700 1 --2- 172.21.15.1:0/4176959435 >> v2:172.21.15.1:3300/0 conn(0x7fe8840b3c60 0x7fe8840cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fe88a23c700 1 -- 172.21.15.1:0/4176959435 >> v1:172.21.15.1:6789/0 conn(0x7fe88406d080 legacy=0x7fe8840b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fe88a23c700 1 -- 172.21.15.1:0/4176959435 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8840707d0 con 0x7fe88414a480 2024-03-18T18:50:10.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.844+0000 7fca77a36700 1 Processor -- start 2024-03-18T18:50:10.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.845+0000 7fca77a36700 1 -- start start 2024-03-18T18:50:10.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fca77a36700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca700ade00 0x7fca700cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fca77a36700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fca700ccc20 0x7fca7006fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fca77a36700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca700b3aa0 con 0x7fca70099a40 2024-03-18T18:50:10.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fca77a36700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca700ae280 con 0x7fca700ccc20 2024-03-18T18:50:10.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fca77a36700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7006e9b0 con 0x7fca700ade00 2024-03-18T18:50:10.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fca74fd1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca700ade00 0x7fca700cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fca74fd1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca700ade00 0x7fca700cc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:59988/0 (socket says 172.21.15.1:59988) 2024-03-18T18:50:10.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7fca74fd1700 1 -- 172.21.15.1:0/1232568578 learned_addr learned my addr 172.21.15.1:0/1232568578 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:10.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fca75fd3700 1 --2- 172.21.15.1:0/1232568578 >> v2:172.21.15.1:3300/0 conn(0x7fca700ccc20 0x7fca7006fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fc985d9b700 1 -- 172.21.15.1:0/1713039032 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2857237078 0 0) 0x7fc980001490 con 0x7fc988160a70 2024-03-18T18:50:10.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fc985d9b700 1 -- 172.21.15.1:0/1713039032 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9700035b0 con 0x7fc988160a70 2024-03-18T18:50:10.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fc98659c700 1 --2- 172.21.15.1:0/1713039032 >> v2:172.21.15.165:3300/0 conn(0x7fc988160e50 0x7fc9880cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fc985d9b700 1 -- 172.21.15.1:0/1713039032 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2045710323 0 0) 0x7fc9700035b0 con 0x7fc988160a70 2024-03-18T18:50:10.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fc985d9b700 1 -- 172.21.15.1:0/1713039032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9881612d0 msgr2=0x7fc9881616f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:10.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fc985d9b700 1 --2- 172.21.15.1:0/1713039032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9881612d0 0x7fc9881616f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fc985d9b700 1 -- 172.21.15.1:0/1713039032 >> v2:172.21.15.165:3300/0 conn(0x7fc988160e50 msgr2=0x7fc9880cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:10.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fc985d9b700 1 --2- 172.21.15.1:0/1713039032 >> v2:172.21.15.165:3300/0 conn(0x7fc988160e50 0x7fc9880cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fc985d9b700 1 -- 172.21.15.1:0/1713039032 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9880cf210 con 0x7fc988160a70 2024-03-18T18:50:10.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fc985d9b700 1 -- 172.21.15.1:0/1713039032 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fc980001b80 con 0x7fc988160a70 2024-03-18T18:50:10.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fc985d9b700 1 -- 172.21.15.1:0/1713039032 >> v1:172.21.15.165:6789/0 conn(0x7fc988160a70 legacy=0x7fc9880d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fe88a23c700 1 --2- 172.21.15.1:0/4176959435 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe88414a480 0x7fe88406cb40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009d90 tx=0x7fe87c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef7b7d8eeaf6401f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7f303f7fe700 1 --2- 172.21.15.1:0/2491443199 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3040070e40 0x7f3040162850 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3030009840 tx=0x7f3030009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b48bd67e3f318274 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7f303e7fc700 1 -- 172.21.15.1:0/2491443199 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3030011830 con 0x7f3040070e40 2024-03-18T18:50:10.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7f303e7fc700 1 -- 172.21.15.1:0/2491443199 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3030011990 con 0x7f3040070e40 2024-03-18T18:50:10.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7f303e7fc700 1 -- 172.21.15.1:0/2491443199 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3030011d00 con 0x7f3040070e40 2024-03-18T18:50:10.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fca74fd1700 1 -- 172.21.15.1:0/1232568578 >> v2:172.21.15.1:3300/0 conn(0x7fca700ccc20 msgr2=0x7fca7006fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:10.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fca74fd1700 1 --2- 172.21.15.1:0/1232568578 >> v2:172.21.15.1:3300/0 conn(0x7fca700ccc20 0x7fca7006fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fca74fd1700 1 -- 172.21.15.1:0/1232568578 >> v1:172.21.15.1:6789/0 conn(0x7fca70099a40 legacy=0x7fca70099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:10.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fca74fd1700 1 -- 172.21.15.1:0/1232568578 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7006ec90 con 0x7fca700ade00 2024-03-18T18:50:10.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fe75b7fe700 1 -- 172.21.15.1:0/3694741955 >> v2:172.21.15.165:3300/0 conn(0x7fe75c0cfb50 msgr2=0x7fe75c0d3c60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T18:50:10.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fe75b7fe700 1 --2- 172.21.15.1:0/3694741955 >> v2:172.21.15.165:3300/0 conn(0x7fe75c0cfb50 0x7fe75c0d3c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fe75b7fe700 1 -- 172.21.15.1:0/3694741955 >> v1:172.21.15.165:6789/0 conn(0x7fe75c0b3d40 legacy=0x7fe75c0b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fe75b7fe700 1 -- 172.21.15.1:0/3694741955 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75c0b2d20 con 0x7fe75c0cff30 2024-03-18T18:50:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fc985d9b700 1 --2- 172.21.15.1:0/1713039032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc970003a00 0x7fc970003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fc985d9b700 1 --2- 172.21.15.1:0/1713039032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc970004850 0x7fc970004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fc985d9b700 1 --2- 172.21.15.1:0/1713039032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc970005210 0x7fc97000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fc985d9b700 1 -- 172.21.15.1:0/1713039032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc988161c30 con 0x7fc970004850 2024-03-18T18:50:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fc985d9b700 1 -- 172.21.15.1:0/1713039032 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9880cef30 con 0x7fc970003a00 2024-03-18T18:50:10.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fc985d9b700 1 -- 172.21.15.1:0/1713039032 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97000c050 con 0x7fc970005210 2024-03-18T18:50:10.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fca74fd1700 1 --2- 172.21.15.1:0/1232568578 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca700ade00 0x7fca700cc6e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fca68009e50 tx=0x7fca6800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af8997a8a663d174 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fe88923a700 1 -- 172.21.15.1:0/4176959435 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c00e070 con 0x7fe88414a480 2024-03-18T18:50:10.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fe75b7fe700 1 --2- 172.21.15.1:0/3694741955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe75c0cff30 0x7fe75c0b3800 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0b5c90 tx=0x7fe750009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e814f017d04947 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7f31a6395700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f31a0071df0 0x7f31a00721d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7f31a6395700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f31a0071df0 0x7f31a00721d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45164/0 (socket says 172.21.15.1:45164) 2024-03-18T18:50:10.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.846+0000 7f31a6395700 1 -- 172.21.15.1:0/983189854 learned_addr learned my addr 172.21.15.1:0/983189854 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:10.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7f31a6395700 1 -- 172.21.15.1:0/983189854 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f31a00b2210 msgr2=0x7f31a00b2680 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T18:50:10.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7f31a6395700 1 --2- 172.21.15.1:0/983189854 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f31a00b2210 0x7f31a00b2680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7f31a6395700 1 -- 172.21.15.1:0/983189854 >> v1:172.21.15.1:6789/0 conn(0x7f31a00ce450 legacy=0x7f31a00cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7f31a6395700 1 -- 172.21.15.1:0/983189854 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a00cde00 con 0x7f31a0071df0 2024-03-18T18:50:10.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.848+0000 7fc986d9d700 1 --2- 172.21.15.1:0/1713039032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc970003a00 0x7fc970003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.848+0000 7fe75affd700 1 -- 172.21.15.1:0/3694741955 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750004230 con 0x7fe75c0cff30 2024-03-18T18:50:10.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.848+0000 7fe75affd700 1 -- 172.21.15.1:0/3694741955 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe750004390 con 0x7fe75c0cff30 2024-03-18T18:50:10.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.848+0000 7fe75affd700 1 -- 172.21.15.1:0/3694741955 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7500036a0 con 0x7fe75c0cff30 2024-03-18T18:50:10.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fe88923a700 1 -- 172.21.15.1:0/4176959435 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87c00bbd0 con 0x7fe88414a480 2024-03-18T18:50:10.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.847+0000 7fe88923a700 1 -- 172.21.15.1:0/4176959435 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c012430 con 0x7fe88414a480 2024-03-18T18:50:10.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.848+0000 7fc986d9d700 1 -- 172.21.15.1:0/1713039032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc970005210 msgr2=0x7fc97000ba10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T18:50:10.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.848+0000 7fc986d9d700 1 --2- 172.21.15.1:0/1713039032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc970005210 0x7fc97000ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.848+0000 7fc986d9d700 1 -- 172.21.15.1:0/1713039032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc970004850 msgr2=0x7fc970004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:10.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.848+0000 7fc986d9d700 1 --2- 172.21.15.1:0/1713039032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc970004850 0x7fc970004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.848+0000 7fc986d9d700 1 -- 172.21.15.1:0/1713039032 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9880cf210 con 0x7fc970003a00 2024-03-18T18:50:10.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.848+0000 7f31a6395700 1 --2- 172.21.15.1:0/983189854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a0071df0 0x7f31a00721d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f319c00a990 tx=0x7f319c009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efd62374e80ee6f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.848+0000 7fc986d9d700 1 --2- 172.21.15.1:0/1713039032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc970003a00 0x7fc970003e20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc98000a8b0 tx=0x7fc970005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15192e5efc025bc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.848+0000 7f31a5393700 1 -- 172.21.15.1:0/983189854 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c009bb0 con 0x7f31a0071df0 2024-03-18T18:50:10.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.848+0000 7f31a5393700 1 -- 172.21.15.1:0/983189854 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319c009d10 con 0x7f31a0071df0 2024-03-18T18:50:10.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.848+0000 7f31a5393700 1 -- 172.21.15.1:0/983189854 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c00a0a0 con 0x7f31a0071df0 2024-03-18T18:50:10.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.848+0000 7fe88c4a0700 1 -- 172.21.15.1:0/4176959435 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe88414a480 msgr2=0x7fe88406cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.848+0000 7fe88c4a0700 1 --2- 172.21.15.1:0/4176959435 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe88414a480 0x7fe88406cb40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009d90 tx=0x7fe87c00b330 comp rx=0 tx=0).stop 2024-03-18T18:50:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.848+0000 7f3045d73700 1 -- 172.21.15.1:0/2491443199 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3040070e40 msgr2=0x7f3040162850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.848+0000 7f3045d73700 1 --2- 172.21.15.1:0/2491443199 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3040070e40 0x7f3040162850 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3030009840 tx=0x7f3030009b50 comp rx=0 tx=0).stop 2024-03-18T18:50:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7f3045d73700 1 -- 172.21.15.1:0/2491443199 shutdown_connections 2024-03-18T18:50:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7f3045d73700 1 --2- 172.21.15.1:0/2491443199 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3040070e40 0x7f3040162850 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7f3045d73700 1 -- 172.21.15.1:0/2491443199 >> 172.21.15.1:0/2491443199 conn(0x7f30400b1350 msgr2=0x7f30400b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7fe88c4a0700 1 -- 172.21.15.1:0/4176959435 shutdown_connections 2024-03-18T18:50:10.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7fe88c4a0700 1 --2- 172.21.15.1:0/4176959435 >> v2:172.21.15.1:3300/0 conn(0x7fe8840b3c60 0x7fe8840cc770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7fe88c4a0700 1 --2- 172.21.15.1:0/4176959435 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe88414a480 0x7fe88406cb40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7fe88c4a0700 1 -- 172.21.15.1:0/4176959435 >> 172.21.15.1:0/4176959435 conn(0x7fe88406ab70 msgr2=0x7fe88406af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7fe88c4a0700 1 -- 172.21.15.1:0/4176959435 shutdown_connections 2024-03-18T18:50:10.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7fe88c4a0700 1 -- 172.21.15.1:0/4176959435 wait complete. 2024-03-18T18:50:10.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7fc985d9b700 1 -- 172.21.15.1:0/1713039032 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980012600 con 0x7fc970003a00 2024-03-18T18:50:10.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7fc985d9b700 1 -- 172.21.15.1:0/1713039032 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc980012760 con 0x7fc970003a00 2024-03-18T18:50:10.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7fc985d9b700 1 -- 172.21.15.1:0/1713039032 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980012b10 con 0x7fc970003a00 2024-03-18T18:50:10.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7fe7627ab700 1 -- 172.21.15.1:0/3694741955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe75c0cff30 msgr2=0x7fe75c0b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7fe7627ab700 1 --2- 172.21.15.1:0/3694741955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe75c0cff30 0x7fe75c0b3800 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0b5c90 tx=0x7fe750009b20 comp rx=0 tx=0).stop 2024-03-18T18:50:10.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7fe7627ab700 1 -- 172.21.15.1:0/3694741955 shutdown_connections 2024-03-18T18:50:10.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7fe7627ab700 1 --2- 172.21.15.1:0/3694741955 >> v2:172.21.15.165:3300/0 conn(0x7fe75c0cfb50 0x7fe75c0d3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7fe7627ab700 1 --2- 172.21.15.1:0/3694741955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe75c0cff30 0x7fe75c0b3800 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7fe7627ab700 1 -- 172.21.15.1:0/3694741955 >> 172.21.15.1:0/3694741955 conn(0x7fe75c0d3140 msgr2=0x7fe75c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7f31a85f9700 1 -- 172.21.15.1:0/983189854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a0071df0 msgr2=0x7f31a00721d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7f31a85f9700 1 --2- 172.21.15.1:0/983189854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a0071df0 0x7f31a00721d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f319c00a990 tx=0x7f319c009520 comp rx=0 tx=0).stop 2024-03-18T18:50:10.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.849+0000 7f3045d73700 1 -- 172.21.15.1:0/2491443199 shutdown_connections 2024-03-18T18:50:10.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.850+0000 7fca67fff700 1 -- 172.21.15.1:0/1232568578 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6800e070 con 0x7fca700ade00 2024-03-18T18:50:10.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.850+0000 7fca67fff700 1 -- 172.21.15.1:0/1232568578 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca68003a20 con 0x7fca700ade00 2024-03-18T18:50:10.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.850+0000 7fca67fff700 1 -- 172.21.15.1:0/1232568578 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca68003db0 con 0x7fca700ade00 2024-03-18T18:50:10.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.850+0000 7fe7627ab700 1 -- 172.21.15.1:0/3694741955 shutdown_connections 2024-03-18T18:50:10.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.850+0000 7fe7627ab700 1 -- 172.21.15.1:0/3694741955 wait complete. 2024-03-18T18:50:10.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.850+0000 7fc98d11d700 1 -- 172.21.15.1:0/1713039032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc970003a00 msgr2=0x7fc970003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.850+0000 7fc98d11d700 1 --2- 172.21.15.1:0/1713039032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc970003a00 0x7fc970003e20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc98000a8b0 tx=0x7fc970005880 comp rx=0 tx=0).stop 2024-03-18T18:50:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.850+0000 7f31a85f9700 1 -- 172.21.15.1:0/983189854 shutdown_connections 2024-03-18T18:50:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.850+0000 7f31a85f9700 1 --2- 172.21.15.1:0/983189854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a0071df0 0x7f31a00721d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.850+0000 7f31a85f9700 1 --2- 172.21.15.1:0/983189854 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f31a00b2210 0x7f31a00b2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.850+0000 7f31a85f9700 1 -- 172.21.15.1:0/983189854 >> 172.21.15.1:0/983189854 conn(0x7f31a006a750 msgr2=0x7f31a006ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.851+0000 7f31a85f9700 1 -- 172.21.15.1:0/983189854 shutdown_connections 2024-03-18T18:50:10.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.850+0000 7f3045d73700 1 -- 172.21.15.1:0/2491443199 wait complete. 2024-03-18T18:50:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.851+0000 7fca77a36700 1 -- 172.21.15.1:0/1232568578 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca700ade00 msgr2=0x7fca700cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.851+0000 7fca77a36700 1 --2- 172.21.15.1:0/1232568578 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca700ade00 0x7fca700cc6e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fca68009e50 tx=0x7fca6800b330 comp rx=0 tx=0).stop 2024-03-18T18:50:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.851+0000 7fca77a36700 1 -- 172.21.15.1:0/1232568578 shutdown_connections 2024-03-18T18:50:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.850+0000 7fe88c4a0700 1 Processor -- start 2024-03-18T18:50:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.850+0000 7fe88c4a0700 1 -- start start 2024-03-18T18:50:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.851+0000 7fca77a36700 1 --2- 172.21.15.1:0/1232568578 >> v2:172.21.15.1:3300/0 conn(0x7fca700ccc20 0x7fca7006fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.851+0000 7fca77a36700 1 --2- 172.21.15.1:0/1232568578 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca700ade00 0x7fca700cc6e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.851+0000 7fca77a36700 1 -- 172.21.15.1:0/1232568578 >> 172.21.15.1:0/1232568578 conn(0x7fca700b2c50 msgr2=0x7fca700b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.852+0000 7fe7627ab700 1 Processor -- start 2024-03-18T18:50:10.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.852+0000 7fc98759e700 1 -- 172.21.15.1:0/1713039032 reap_dead start 2024-03-18T18:50:10.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.852+0000 7fc98d11d700 1 -- 172.21.15.1:0/1713039032 shutdown_connections 2024-03-18T18:50:10.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.852+0000 7fc98d11d700 1 -- 172.21.15.1:0/1713039032 >> 172.21.15.1:0/1713039032 conn(0x7fc98809c560 msgr2=0x7fc9880d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.852+0000 7fe88c4a0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe88406d080 0x7fe8841353a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.852+0000 7fe88c4a0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8840b3c60 0x7fe884148480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.852+0000 7fe88c4a0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe88414a480 0x7fe884161d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.852+0000 7fe88c4a0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8841363b0 con 0x7fe8840b3c60 2024-03-18T18:50:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.852+0000 7fe88c4a0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8841358e0 con 0x7fe88406d080 2024-03-18T18:50:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.852+0000 7fe88c4a0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8841366d0 con 0x7fe88414a480 2024-03-18T18:50:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.852+0000 7fe88aa3d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe88414a480 0x7fe884161d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.852+0000 7fe889a3b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8840b3c60 0x7fe884148480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.852+0000 7fe889a3b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8840b3c60 0x7fe884148480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45220/0 (socket says 172.21.15.1:45220) 2024-03-18T18:50:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.852+0000 7fe889a3b700 1 -- 172.21.15.1:0/788342128 learned_addr learned my addr 172.21.15.1:0/788342128 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7fe889a3b700 1 -- 172.21.15.1:0/788342128 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe88414a480 msgr2=0x7fe884161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7fe889a3b700 1 --2- 172.21.15.1:0/788342128 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe88414a480 0x7fe884161d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7fe889a3b700 1 -- 172.21.15.1:0/788342128 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe88406d080 msgr2=0x7fe8841353a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T18:50:10.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7fe889a3b700 1 --2- 172.21.15.1:0/788342128 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe88406d080 0x7fe8841353a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7fe889a3b700 1 -- 172.21.15.1:0/788342128 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c009a70 con 0x7fe8840b3c60 2024-03-18T18:50:10.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7fe889a3b700 1 --2- 172.21.15.1:0/788342128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8840b3c60 0x7fe884148480 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe88000c480 tx=0x7fe88000c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7fe87affd700 1 -- 172.21.15.1:0/788342128 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880005d70 con 0x7fe8840b3c60 2024-03-18T18:50:10.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7fe88c4a0700 1 -- 172.21.15.1:0/788342128 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe884135b10 con 0x7fe8840b3c60 2024-03-18T18:50:10.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7fe88c4a0700 1 -- 172.21.15.1:0/788342128 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8841360d0 con 0x7fe8840b3c60 2024-03-18T18:50:10.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.852+0000 7f3045d73700 1 Processor -- start 2024-03-18T18:50:10.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.852+0000 7f3045d73700 1 -- start start 2024-03-18T18:50:10.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7f3045d73700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f304006dca0 0x7f30400cf050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7f3045d73700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3040070e40 0x7f3040156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7f3045d73700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30400b2d60 0x7f30400af350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7f3045d73700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30401fff90 con 0x7f3040070e40 2024-03-18T18:50:10.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7f3045d73700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30400cf590 con 0x7f304006dca0 2024-03-18T18:50:10.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7f3045d73700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304015d8c0 con 0x7f30400b2d60 2024-03-18T18:50:10.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.852+0000 7fca77a36700 1 -- 172.21.15.1:0/1232568578 shutdown_connections 2024-03-18T18:50:10.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.852+0000 7fca77a36700 1 -- 172.21.15.1:0/1232568578 wait complete. 2024-03-18T18:50:10.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.854+0000 7fca77a36700 1 Processor -- start 2024-03-18T18:50:10.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.852+0000 7fe7627ab700 1 -- start start 2024-03-18T18:50:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7fe7627ab700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe75c0b3d40 0x7fe75c151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7fe7627ab700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe75c0cfb50 0x7fe75c0b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7fe7627ab700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe75c0cff30 0x7fe75c154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7fe7627ab700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c151b60 con 0x7fe75c0cfb50 2024-03-18T18:50:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7fe7627ab700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c150c00 con 0x7fe75c0cff30 2024-03-18T18:50:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7fe7627ab700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c150ee0 con 0x7fe75c0b3d40 2024-03-18T18:50:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7fe760d48700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe75c0cff30 0x7fe75c154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7fe75bfff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe75c0b3d40 0x7fe75c151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7fe75bfff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe75c0b3d40 0x7fe75c151f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:60018/0 (socket says 172.21.15.1:60018) 2024-03-18T18:50:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.853+0000 7fe75bfff700 1 -- 172.21.15.1:0/1673380968 learned_addr learned my addr 172.21.15.1:0/1673380968 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:10.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.854+0000 7fe87affd700 1 -- 172.21.15.1:0/788342128 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8800070d0 con 0x7fe8840b3c60 2024-03-18T18:50:10.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.854+0000 7fe87affd700 1 -- 172.21.15.1:0/788342128 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880007440 con 0x7fe8840b3c60 2024-03-18T18:50:10.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.854+0000 7fe75bfff700 1 -- 172.21.15.1:0/1673380968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe75c0cff30 msgr2=0x7fe75c154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.854+0000 7fe75bfff700 1 --2- 172.21.15.1:0/1673380968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe75c0cff30 0x7fe75c154af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.854+0000 7fe75bfff700 1 -- 172.21.15.1:0/1673380968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe75c0cfb50 msgr2=0x7fe75c0b5f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T18:50:10.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.854+0000 7fe75bfff700 1 --2- 172.21.15.1:0/1673380968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe75c0cfb50 0x7fe75c0b5f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.854+0000 7fe75bfff700 1 -- 172.21.15.1:0/1673380968 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7500094a0 con 0x7fe75c0b3d40 2024-03-18T18:50:10.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.854+0000 7fe75bfff700 1 --2- 172.21.15.1:0/1673380968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe75c0b3d40 0x7fe75c151f80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe75400ba50 tx=0x7fe75400bd60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.854+0000 7f303ffff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30400b2d60 0x7f30400af350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.854+0000 7f303f7fe700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f304006dca0 0x7f30400cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.854+0000 7f303effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3040070e40 0x7f3040156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.854+0000 7f303effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3040070e40 0x7f3040156bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45234/0 (socket says 172.21.15.1:45234) 2024-03-18T18:50:10.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.854+0000 7f303effd700 1 -- 172.21.15.1:0/3339264209 learned_addr learned my addr 172.21.15.1:0/3339264209 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:10.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.854+0000 7fca77a36700 1 -- start start 2024-03-18T18:50:10.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.855+0000 7f303f7fe700 1 -- 172.21.15.1:0/3339264209 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30400b2d60 msgr2=0x7f30400af350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.855+0000 7f303f7fe700 1 --2- 172.21.15.1:0/3339264209 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30400b2d60 0x7f30400af350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.855+0000 7f303f7fe700 1 -- 172.21.15.1:0/3339264209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3040070e40 msgr2=0x7f3040156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.855+0000 7f303f7fe700 1 --2- 172.21.15.1:0/3339264209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3040070e40 0x7f3040156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.855+0000 7f303f7fe700 1 -- 172.21.15.1:0/3339264209 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30300094a0 con 0x7f304006dca0 2024-03-18T18:50:10.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.855+0000 7fc98d11d700 1 -- 172.21.15.1:0/1713039032 shutdown_connections 2024-03-18T18:50:10.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.855+0000 7fca77a36700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca70099a40 0x7fca701f8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.855+0000 7fca77a36700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca700ade00 0x7fca701fa430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.855+0000 7fca77a36700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca700ccc20 0x7fca700cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.855+0000 7fca77a36700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7006d5d0 con 0x7fca700ccc20 2024-03-18T18:50:10.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.855+0000 7fca77a36700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca701f85a0 con 0x7fca700ade00 2024-03-18T18:50:10.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.855+0000 7fca77a36700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca701fa970 con 0x7fca70099a40 2024-03-18T18:50:10.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.855+0000 7fe88c4a0700 1 -- 172.21.15.1:0/788342128 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe86c000fc0 con 0x7fe8840b3c60 2024-03-18T18:50:10.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.855+0000 7fc98d11d700 1 -- 172.21.15.1:0/1713039032 wait complete. 2024-03-18T18:50:10.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.856+0000 7fca75fd3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca700ccc20 0x7fca700cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.856+0000 7fca757d2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca70099a40 0x7fca701f8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.856+0000 7fca75fd3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca700ccc20 0x7fca700cf6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45248/0 (socket says 172.21.15.1:45248) 2024-03-18T18:50:10.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.856+0000 7fca75fd3700 1 -- 172.21.15.1:0/2519178950 learned_addr learned my addr 172.21.15.1:0/2519178950 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:10.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.856+0000 7fca74fd1700 1 --2- 172.21.15.1:0/2519178950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca700ade00 0x7fca701fa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.856+0000 7f303f7fe700 1 --2- 172.21.15.1:0/3339264209 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f304006dca0 0x7f30400cf050 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3030016000 tx=0x7f3030004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.856+0000 7fca757d2700 1 -- 172.21.15.1:0/2519178950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca700ade00 msgr2=0x7fca701fa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.856+0000 7fca757d2700 1 --2- 172.21.15.1:0/2519178950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca700ade00 0x7fca701fa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.856+0000 7fca757d2700 1 -- 172.21.15.1:0/2519178950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca700ccc20 msgr2=0x7fca700cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.856+0000 7fca757d2700 1 --2- 172.21.15.1:0/2519178950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca700ccc20 0x7fca700cf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.856+0000 7fca757d2700 1 -- 172.21.15.1:0/2519178950 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca68009b30 con 0x7fca70099a40 2024-03-18T18:50:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.855+0000 7f31a85f9700 1 -- 172.21.15.1:0/983189854 wait complete. 2024-03-18T18:50:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.856+0000 7f3027fff700 1 -- 172.21.15.1:0/3339264209 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30300038a0 con 0x7f304006dca0 2024-03-18T18:50:10.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.856+0000 7f3027fff700 1 -- 172.21.15.1:0/3339264209 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3030003a20 con 0x7f304006dca0 2024-03-18T18:50:10.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.856+0000 7f3027fff700 1 -- 172.21.15.1:0/3339264209 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3030003dd0 con 0x7f304006dca0 2024-03-18T18:50:10.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.857+0000 7f3045d73700 1 -- 172.21.15.1:0/3339264209 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f304015e440 con 0x7f304006dca0 2024-03-18T18:50:10.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.857+0000 7f3045d73700 1 -- 172.21.15.1:0/3339264209 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3040157130 con 0x7f304006dca0 2024-03-18T18:50:10.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.857+0000 7fca757d2700 1 --2- 172.21.15.1:0/2519178950 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca70099a40 0x7fca701f8060 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fca6c00b3c0 tx=0x7fca6c00b6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.857+0000 7f31a85f9700 1 Processor -- start 2024-03-18T18:50:10.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.857+0000 7f31a85f9700 1 -- start start 2024-03-18T18:50:10.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.858+0000 7fc98d11d700 1 Processor -- start 2024-03-18T18:50:10.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.858+0000 7fca65ffb700 1 -- 172.21.15.1:0/2519178950 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c011620 con 0x7fca70099a40 2024-03-18T18:50:10.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.858+0000 7fca65ffb700 1 -- 172.21.15.1:0/2519178950 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6c0117a0 con 0x7fca70099a40 2024-03-18T18:50:10.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.858+0000 7fca65ffb700 1 -- 172.21.15.1:0/2519178950 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c011b50 con 0x7fca70099a40 2024-03-18T18:50:10.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.858+0000 7fca77a36700 1 -- 172.21.15.1:0/2519178950 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca700b5380 con 0x7fca70099a40 2024-03-18T18:50:10.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.858+0000 7fca77a36700 1 -- 172.21.15.1:0/2519178950 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca700b5940 con 0x7fca70099a40 2024-03-18T18:50:10.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.858+0000 7f31a85f9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f31a0071df0 0x7f31a013f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.858+0000 7f31a85f9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31a00b2210 0x7f31a0148560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.858+0000 7f31a85f9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a00ce450 0x7f31a014c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.858+0000 7f31a85f9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a013fc20 con 0x7f31a00ce450 2024-03-18T18:50:10.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.858+0000 7f31a85f9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a0147810 con 0x7f31a00b2210 2024-03-18T18:50:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.858+0000 7f31a85f9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a0146770 con 0x7f31a0071df0 2024-03-18T18:50:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.859+0000 7f31a6395700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f31a0071df0 0x7f31a013f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.857+0000 7fe87affd700 1 -- 172.21.15.1:0/788342128 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe8800075a0 con 0x7fe8840b3c60 2024-03-18T18:50:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.857+0000 7fe87affd700 1 --2- 172.21.15.1:0/788342128 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe86806ac90 0x7fe86806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.858+0000 7fe87affd700 1 -- 172.21.15.1:0/788342128 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe880088910 con 0x7fe8840b3c60 2024-03-18T18:50:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.858+0000 7fe88a23c700 1 --2- 172.21.15.1:0/788342128 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe86806ac90 0x7fe86806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.858+0000 7fe87affd700 1 --2- 172.21.15.1:0/788342128 >> [v2:172.21.15.1:6808/77305,v1:172.21.15.1:6809/77305] conn(0x7fe868074250 0x7fe868076690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.858+0000 7fe87affd700 1 -- 172.21.15.1:0/788342128 --> [v2:172.21.15.1:6808/77305,v1:172.21.15.1:6809/77305] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe868076d60 con 0x7fe868074250 2024-03-18T18:50:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.858+0000 7fe88aa3d700 1 --2- 172.21.15.1:0/788342128 >> [v2:172.21.15.1:6808/77305,v1:172.21.15.1:6809/77305] conn(0x7fe868074250 0x7fe868076690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.859+0000 7ff35bc38700 1 Processor -- start 2024-03-18T18:50:10.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.859+0000 7fe758ff9700 1 -- 172.21.15.1:0/1673380968 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75400c9e0 con 0x7fe75c0b3d40 2024-03-18T18:50:10.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.859+0000 7fe758ff9700 1 -- 172.21.15.1:0/1673380968 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75400cb40 con 0x7fe75c0b3d40 2024-03-18T18:50:10.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.859+0000 7fe758ff9700 1 -- 172.21.15.1:0/1673380968 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe754014440 con 0x7fe75c0b3d40 2024-03-18T18:50:10.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.859+0000 7fe7627ab700 1 -- 172.21.15.1:0/1673380968 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe75c1511c0 con 0x7fe75c0b3d40 2024-03-18T18:50:10.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.859+0000 7fe7627ab700 1 -- 172.21.15.1:0/1673380968 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75c151780 con 0x7fe75c0b3d40 2024-03-18T18:50:10.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.859+0000 7fc98d11d700 1 -- start start 2024-03-18T18:50:10.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7fc98d11d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc98814aa80 0x7fc9881529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7fc98d11d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc98814ae60 0x7fc9880ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7fc98d11d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9880b70c0 0x7fc988153380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7fc98d11d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9880b7860 con 0x7fc9880b70c0 2024-03-18T18:50:10.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7fc98d11d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9880b7580 con 0x7fc98814aa80 2024-03-18T18:50:10.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7fc98d11d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc988151ec0 con 0x7fc98814ae60 2024-03-18T18:50:10.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7fc986d9d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc98814aa80 0x7fc9881529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7fc98659c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc98814ae60 0x7fc9880ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7fc98659c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc98814ae60 0x7fc9880ccf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:60048/0 (socket says 172.21.15.1:60048) 2024-03-18T18:50:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7fc98659c700 1 -- 172.21.15.1:0/372364865 learned_addr learned my addr 172.21.15.1:0/372364865 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7fc98659c700 1 -- 172.21.15.1:0/372364865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc98814aa80 msgr2=0x7fc9881529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7fc98659c700 1 --2- 172.21.15.1:0/372364865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc98814aa80 0x7fc9881529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7fc98659c700 1 -- 172.21.15.1:0/372364865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9880b70c0 msgr2=0x7fc988153380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7fc98659c700 1 --2- 172.21.15.1:0/372364865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9880b70c0 0x7fc988153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7fc98659c700 1 -- 172.21.15.1:0/372364865 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9881521a0 con 0x7fc98814ae60 2024-03-18T18:50:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7fc98659c700 1 --2- 172.21.15.1:0/372364865 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc98814ae60 0x7fc9880ccf50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00e810 tx=0x7fc97c00eb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7f31a6b96700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a00ce450 0x7f31a014c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7f31a6b96700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a00ce450 0x7f31a014c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45262/0 (socket says 172.21.15.1:45262) 2024-03-18T18:50:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7f31a6b96700 1 -- 172.21.15.1:0/3982518009 learned_addr learned my addr 172.21.15.1:0/3982518009 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:10.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7f31a5b94700 1 --2- 172.21.15.1:0/3982518009 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31a00b2210 0x7f31a0148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7fe88a23c700 1 --2- 172.21.15.1:0/788342128 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe86806ac90 0x7fe86806d150 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe884148e80 tx=0x7fe87c011000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.860+0000 7f3027fff700 1 -- 172.21.15.1:0/3339264209 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f303002e450 con 0x7f304006dca0 2024-03-18T18:50:10.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.860+0000 7f3027fff700 1 --2- 172.21.15.1:0/3339264209 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f302c06ad60 0x7f302c06d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.860+0000 7f3027fff700 1 -- 172.21.15.1:0/3339264209 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f303008ab60 con 0x7f304006dca0 2024-03-18T18:50:10.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7f3045d73700 1 --2- 172.21.15.1:0/3339264209 >> [v2:172.21.15.165:6808/77474,v1:172.21.15.165:6809/77474] conn(0x7f3040071220 0x7f30400611f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7f3045d73700 1 -- 172.21.15.1:0/3339264209 --> [v2:172.21.15.165:6808/77474,v1:172.21.15.165:6809/77474] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f304006d4a0 con 0x7f3040071220 2024-03-18T18:50:10.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7f303ffff700 1 --2- 172.21.15.1:0/3339264209 >> [v2:172.21.15.165:6808/77474,v1:172.21.15.165:6809/77474] conn(0x7f3040071220 0x7f30400611f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.860+0000 7fca65ffb700 1 -- 172.21.15.1:0/2519178950 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fca6c011cb0 con 0x7fca70099a40 2024-03-18T18:50:10.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7fca65ffb700 1 --2- 172.21.15.1:0/2519178950 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fca5406ad90 0x7fca5406d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7fca65ffb700 1 -- 172.21.15.1:0/2519178950 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fca6c0890a0 con 0x7fca70099a40 2024-03-18T18:50:10.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7fca77a36700 1 --2- 172.21.15.1:0/2519178950 >> [v2:172.21.15.165:6800/77475,v1:172.21.15.165:6801/77475] conn(0x7fca700cfc10 0x7fca700b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7fca77a36700 1 -- 172.21.15.1:0/2519178950 --> [v2:172.21.15.165:6800/77475,v1:172.21.15.165:6801/77475] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fca7006f250 con 0x7fca700cfc10 2024-03-18T18:50:10.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7fca75fd3700 1 --2- 172.21.15.1:0/2519178950 >> [v2:172.21.15.165:6800/77475,v1:172.21.15.165:6801/77475] conn(0x7fca700cfc10 0x7fca700b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7fca74fd1700 1 --2- 172.21.15.1:0/2519178950 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fca5406ad90 0x7fca5406d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7fe758ff9700 1 -- 172.21.15.1:0/1673380968 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe75401d3b0 con 0x7fe75c0b3d40 2024-03-18T18:50:10.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.859+0000 7ff35bc38700 1 -- start start 2024-03-18T18:50:10.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7ff35bc38700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3541612d0 0x7ff3541616f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7ff35bc38700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff354162170 con 0x7ff354160e50 2024-03-18T18:50:10.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7ff35bc38700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff354161c30 con 0x7ff354160a70 2024-03-18T18:50:10.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7ff35bc38700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3540d06e0 con 0x7ff3541612d0 2024-03-18T18:50:10.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7ff3599d4700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7ff354160a70 0x7ff3540d3b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:38328/0 (socket says 172.21.15.1:38328) 2024-03-18T18:50:10.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7ff3599d4700 1 -- 172.21.15.1:0/1505435054 learned_addr learned my addr 172.21.15.1:0/1505435054 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:10.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.861+0000 7ff35a1d5700 1 --2- 172.21.15.1:0/1505435054 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3541612d0 0x7ff3541616f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7ff3589d2700 1 -- 172.21.15.1:0/1505435054 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374699687 0 0) 0x7ff354161c30 con 0x7ff354160a70 2024-03-18T18:50:10.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7ff3589d2700 1 -- 172.21.15.1:0/1505435054 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3400035b0 con 0x7ff354160a70 2024-03-18T18:50:10.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7ff35a1d5700 1 -- 172.21.15.1:0/1505435054 >> v1:172.21.15.165:6789/0 conn(0x7ff354160a70 legacy=0x7ff3540d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7ff35a1d5700 1 -- 172.21.15.1:0/1505435054 >> v1:172.21.15.1:6789/0 conn(0x7ff354160e50 legacy=0x7ff3540cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:10.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7ff35a1d5700 1 -- 172.21.15.1:0/1505435054 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3540cef30 con 0x7ff3541612d0 2024-03-18T18:50:10.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7f303ffff700 1 --2- 172.21.15.1:0/3339264209 >> [v2:172.21.15.165:6808/77474,v1:172.21.15.165:6809/77474] conn(0x7f3040071220 0x7f30400611f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7fe758ff9700 1 --2- 172.21.15.1:0/1673380968 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe74806ace0 0x7fe74806d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7f31a6b96700 1 -- 172.21.15.1:0/3982518009 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f31a0071df0 msgr2=0x7f31a013f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7f31a6b96700 1 --2- 172.21.15.1:0/3982518009 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f31a0071df0 0x7f31a013f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7f31a6b96700 1 -- 172.21.15.1:0/3982518009 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31a00b2210 msgr2=0x7f31a0148560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7f31a6b96700 1 --2- 172.21.15.1:0/3982518009 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31a00b2210 0x7f31a0148560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7f31a6b96700 1 -- 172.21.15.1:0/3982518009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319c00a5f0 con 0x7f31a00ce450 2024-03-18T18:50:10.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7fe758ff9700 1 -- 172.21.15.1:0/1673380968 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe754089300 con 0x7fe75c0b3d40 2024-03-18T18:50:10.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7fca75fd3700 1 --2- 172.21.15.1:0/2519178950 >> [v2:172.21.15.165:6800/77475,v1:172.21.15.165:6801/77475] conn(0x7fca700cfc10 0x7fca700b5b30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7fe87affd700 1 -- 172.21.15.1:0/788342128 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe880053810 con 0x7fe8840b3c60 2024-03-18T18:50:10.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.862+0000 7f303effd700 1 --2- 172.21.15.1:0/3339264209 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f302c06ad60 0x7f302c06d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7fe75b7fe700 1 --2- 172.21.15.1:0/1673380968 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe74806ace0 0x7fe74806d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7fca74fd1700 1 --2- 172.21.15.1:0/2519178950 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fca5406ad90 0x7fca5406d250 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fca68000f80 tx=0x7fca68009000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7ff35a1d5700 1 --2- 172.21.15.1:0/1505435054 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3541612d0 0x7ff3541616f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff34c009dc0 tx=0x7ff34c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d853eef97b902f02 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7fc96f7fe700 1 -- 172.21.15.1:0/372364865 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c00c070 con 0x7fc98814ae60 2024-03-18T18:50:10.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7fe88aa3d700 1 --2- 172.21.15.1:0/788342128 >> [v2:172.21.15.1:6808/77305,v1:172.21.15.1:6809/77305] conn(0x7fe868074250 0x7fe868076690 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7fe7627ab700 1 --2- 172.21.15.1:0/1673380968 >> [v2:172.21.15.165:6817/77477,v1:172.21.15.165:6819/77477] conn(0x7fe73c0015e0 0x7fe73c003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7fe7627ab700 1 -- 172.21.15.1:0/1673380968 --> [v2:172.21.15.165:6817/77477,v1:172.21.15.165:6819/77477] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe73c006bd0 con 0x7fe73c0015e0 2024-03-18T18:50:10.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7fc96f7fe700 1 -- 172.21.15.1:0/372364865 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc97c004270 con 0x7fc98814ae60 2024-03-18T18:50:10.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7fc96f7fe700 1 -- 172.21.15.1:0/372364865 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c0036a0 con 0x7fc98814ae60 2024-03-18T18:50:10.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7fc98d11d700 1 -- 172.21.15.1:0/372364865 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc988152480 con 0x7fc98814ae60 2024-03-18T18:50:10.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7fc98d11d700 1 -- 172.21.15.1:0/372364865 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9880cd490 con 0x7fc98814ae60 2024-03-18T18:50:10.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7ff3589d2700 1 -- 172.21.15.1:0/1505435054 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c00e070 con 0x7ff3541612d0 2024-03-18T18:50:10.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7ff3589d2700 1 -- 172.21.15.1:0/1505435054 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34c00baa0 con 0x7ff3541612d0 2024-03-18T18:50:10.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7ff3589d2700 1 -- 172.21.15.1:0/1505435054 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c00be50 con 0x7ff3541612d0 2024-03-18T18:50:10.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7f31a6b96700 1 --2- 172.21.15.1:0/3982518009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a00ce450 0x7f31a014c260 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f319800b780 tx=0x7f319800ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7fe75b7fe700 1 --2- 172.21.15.1:0/1673380968 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe74806ace0 0x7fe74806d1a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0b5c90 tx=0x7fe750008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7f303effd700 1 --2- 172.21.15.1:0/3339264209 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f302c06ad60 0x7f302c06d220 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f304015ad00 tx=0x7f3034009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7f3027fff700 1 -- 172.21.15.1:0/3339264209 <== osd.4 v2:172.21.15.165:6808/77474 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f3038001ea0 con 0x7f3040071220 2024-03-18T18:50:10.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7fca65ffb700 1 -- 172.21.15.1:0/2519178950 <== osd.5 v2:172.21.15.165:6800/77475 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fca60009aa0 con 0x7fca700cfc10 2024-03-18T18:50:10.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.863+0000 7fe760d48700 1 --2- 172.21.15.1:0/1673380968 >> [v2:172.21.15.165:6817/77477,v1:172.21.15.165:6819/77477] conn(0x7fe73c0015e0 0x7fe73c003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.864+0000 7f3196ffd700 1 -- 172.21.15.1:0/3982518009 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3198010860 con 0x7f31a00ce450 2024-03-18T18:50:10.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.864+0000 7f3196ffd700 1 -- 172.21.15.1:0/3982518009 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31980109c0 con 0x7f31a00ce450 2024-03-18T18:50:10.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.864+0000 7f31a85f9700 1 -- 172.21.15.1:0/3982518009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a0146a50 con 0x7f31a00ce450 2024-03-18T18:50:10.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.864+0000 7f3196ffd700 1 -- 172.21.15.1:0/3982518009 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3198010d70 con 0x7f31a00ce450 2024-03-18T18:50:10.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.864+0000 7f31a85f9700 1 -- 172.21.15.1:0/3982518009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a0147010 con 0x7f31a00ce450 2024-03-18T18:50:10.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.865+0000 7f31a85f9700 1 -- 172.21.15.1:0/3982518009 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f31a006d0d0 con 0x7f31a00ce450 2024-03-18T18:50:10.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.867+0000 7f3196ffd700 1 -- 172.21.15.1:0/3982518009 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f319800f3b0 con 0x7f31a00ce450 2024-03-18T18:50:10.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.867+0000 7f3196ffd700 1 --2- 172.21.15.1:0/3982518009 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f318406ad60 0x7f318406d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.867+0000 7f3196ffd700 1 -- 172.21.15.1:0/3982518009 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f31980899a0 con 0x7f31a00ce450 2024-03-18T18:50:10.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.864+0000 7fe87affd700 1 -- 172.21.15.1:0/788342128 <== osd.2 v2:172.21.15.1:6808/77305 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe868076d60 con 0x7fe868074250 2024-03-18T18:50:10.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.864+0000 7fe760d48700 1 --2- 172.21.15.1:0/1673380968 >> [v2:172.21.15.165:6817/77477,v1:172.21.15.165:6819/77477] conn(0x7fe73c0015e0 0x7fe73c003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.865+0000 7fe758ff9700 1 -- 172.21.15.1:0/1673380968 <== osd.6 v2:172.21.15.165:6817/77477 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe73c006bd0 con 0x7fe73c0015e0 2024-03-18T18:50:10.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.864+0000 7ff35bc38700 1 -- 172.21.15.1:0/1505435054 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3541612d0 msgr2=0x7ff3541616f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.864+0000 7ff35bc38700 1 --2- 172.21.15.1:0/1505435054 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3541612d0 0x7ff3541616f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff34c009dc0 tx=0x7ff34c00b330 comp rx=0 tx=0).stop 2024-03-18T18:50:10.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.864+0000 7ff35bc38700 1 -- 172.21.15.1:0/1505435054 shutdown_connections 2024-03-18T18:50:10.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.864+0000 7ff35bc38700 1 --2- 172.21.15.1:0/1505435054 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3541612d0 0x7ff3541616f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.864+0000 7ff35bc38700 1 -- 172.21.15.1:0/1505435054 >> 172.21.15.1:0/1505435054 conn(0x7ff35409c560 msgr2=0x7ff3540d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.864+0000 7ff35bc38700 1 -- 172.21.15.1:0/1505435054 shutdown_connections 2024-03-18T18:50:10.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.864+0000 7ff35bc38700 1 -- 172.21.15.1:0/1505435054 wait complete. 2024-03-18T18:50:10.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.866+0000 7ff35bc38700 1 Processor -- start 2024-03-18T18:50:10.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.866+0000 7ff35bc38700 1 -- start start 2024-03-18T18:50:10.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.868+0000 7ff35bc38700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff354160a70 0x7ff3540c8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.868+0000 7ff35bc38700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff354160e50 0x7ff3540c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.868+0000 7ff35bc38700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3541612d0 0x7ff35415f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.868+0000 7ff35bc38700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3540c70d0 con 0x7ff354160e50 2024-03-18T18:50:10.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.868+0000 7ff35bc38700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3540c72e0 con 0x7ff354160a70 2024-03-18T18:50:10.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.868+0000 7ff35bc38700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3540c75c0 con 0x7ff3541612d0 2024-03-18T18:50:10.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.868+0000 7f3196ffd700 1 --2- 172.21.15.1:0/3982518009 >> [v2:172.21.15.1:6800/77295,v1:172.21.15.1:6801/77295] conn(0x7f3184074320 0x7f3184076760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.868+0000 7f3196ffd700 1 -- 172.21.15.1:0/3982518009 --> [v2:172.21.15.1:6800/77295,v1:172.21.15.1:6801/77295] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3184076e30 con 0x7f3184074320 2024-03-18T18:50:10.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.868+0000 7f3196ffd700 1 -- 172.21.15.1:0/3982518009 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31980098d0 con 0x7f31a00ce450 2024-03-18T18:50:10.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.869+0000 7f31a6395700 1 --2- 172.21.15.1:0/3982518009 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f318406ad60 0x7f318406d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.870+0000 7f31a6395700 1 --2- 172.21.15.1:0/3982518009 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f318406ad60 0x7f318406d220 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f319c000c00 tx=0x7f319c01a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.869+0000 7fc96f7fe700 1 -- 172.21.15.1:0/372364865 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc97c003800 con 0x7fc98814ae60 2024-03-18T18:50:10.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.869+0000 7fc96f7fe700 1 --2- 172.21.15.1:0/372364865 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc97406ad60 0x7fc97406d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.869+0000 7fc96f7fe700 1 -- 172.21.15.1:0/372364865 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc97c0892a0 con 0x7fc98814ae60 2024-03-18T18:50:10.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.870+0000 7fc986d9d700 1 --2- 172.21.15.1:0/372364865 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc97406ad60 0x7fc97406d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.870+0000 7fc986d9d700 1 --2- 172.21.15.1:0/372364865 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc97406ad60 0x7fc97406d220 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc980001370 tx=0x7fc980007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.871+0000 7ff35a1d5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3541612d0 0x7ff35415f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.870+0000 7fc98d11d700 1 --2- 172.21.15.1:0/372364865 >> [v2:172.21.15.1:6824/77296,v1:172.21.15.1:6825/77296] conn(0x7fc9680015e0 0x7fc968003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.871+0000 7fc98d11d700 1 -- 172.21.15.1:0/372364865 --> [v2:172.21.15.1:6824/77296,v1:172.21.15.1:6825/77296] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc968006bd0 con 0x7fc9680015e0 2024-03-18T18:50:10.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.871+0000 7ff35a1d5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3541612d0 0x7ff35415f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:60076/0 (socket says 172.21.15.1:60076) 2024-03-18T18:50:10.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.871+0000 7ff35a1d5700 1 -- 172.21.15.1:0/3719720830 learned_addr learned my addr 172.21.15.1:0/3719720830 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:10.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.871+0000 7fb996984700 1 Processor -- start 2024-03-18T18:50:10.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.871+0000 7ff35a1d5700 1 -- 172.21.15.1:0/3719720830 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff354160a70 msgr2=0x7ff3540c8380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T18:50:10.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.871+0000 7ff35a1d5700 1 --2- 172.21.15.1:0/3719720830 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff354160a70 0x7ff3540c8380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.871+0000 7ff35a1d5700 1 -- 172.21.15.1:0/3719720830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff354160e50 msgr2=0x7ff3540c6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.871+0000 7ff35a1d5700 1 --2- 172.21.15.1:0/3719720830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff354160e50 0x7ff3540c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.871+0000 7ff35a1d5700 1 -- 172.21.15.1:0/3719720830 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34c009a70 con 0x7ff3541612d0 2024-03-18T18:50:10.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.871+0000 7ff35a1d5700 1 --2- 172.21.15.1:0/3719720830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3541612d0 0x7ff35415f0b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff34c00d010 tx=0x7ff34c004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.871+0000 7fb996984700 1 -- start start 2024-03-18T18:50:10.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.872+0000 7ff3467fc700 1 -- 172.21.15.1:0/3719720830 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c004950 con 0x7ff3541612d0 2024-03-18T18:50:10.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.872+0000 7ff35bc38700 1 -- 172.21.15.1:0/3719720830 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3540c78a0 con 0x7ff3541612d0 2024-03-18T18:50:10.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.872+0000 7ff35bc38700 1 -- 172.21.15.1:0/3719720830 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3540c7e60 con 0x7ff3541612d0 2024-03-18T18:50:10.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.872+0000 7fc98759e700 1 --2- 172.21.15.1:0/372364865 >> [v2:172.21.15.1:6824/77296,v1:172.21.15.1:6825/77296] conn(0x7fc9680015e0 0x7fc968003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.873+0000 7fc98759e700 1 --2- 172.21.15.1:0/372364865 >> [v2:172.21.15.1:6824/77296,v1:172.21.15.1:6825/77296] conn(0x7fc9680015e0 0x7fc968003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.872+0000 7fb996984700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9900b3c30 0x7fb9900b2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.873+0000 7fb996984700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb99006ce40 0x7fb99006d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.873+0000 7fb996984700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9900b3560 con 0x7fb9900cef70 2024-03-18T18:50:10.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.873+0000 7fb996984700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb990071a80 con 0x7fb99006ce40 2024-03-18T18:50:10.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.873+0000 7fb996984700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9900b2cc0 con 0x7fb9900b3c30 2024-03-18T18:50:10.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.872+0000 7ff3467fc700 1 -- 172.21.15.1:0/3719720830 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34c009450 con 0x7ff3541612d0 2024-03-18T18:50:10.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.872+0000 7ff3467fc700 1 -- 172.21.15.1:0/3719720830 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c026680 con 0x7ff3541612d0 2024-03-18T18:50:10.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.873+0000 7ff35bc38700 1 -- 172.21.15.1:0/3719720830 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff33c000fc0 con 0x7ff3541612d0 2024-03-18T18:50:10.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.873+0000 7fb98f7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb9900cef70 0x7fb9900b4010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60282/0 (socket says 172.21.15.1:60282) 2024-03-18T18:50:10.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.873+0000 7fb98f7fe700 1 -- 172.21.15.1:0/2859522275 learned_addr learned my addr 172.21.15.1:0/2859522275 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:10.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.874+0000 7fb994f21700 1 --2- 172.21.15.1:0/2859522275 >> v2:172.21.15.165:3300/0 conn(0x7fb99006ce40 0x7fb99006d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.874+0000 7fb994f21700 1 -- 172.21.15.1:0/2859522275 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9900b3c30 msgr2=0x7fb9900b2650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T18:50:10.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.874+0000 7fb994f21700 1 --2- 172.21.15.1:0/2859522275 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9900b3c30 0x7fb9900b2650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.874+0000 7fb994f21700 1 -- 172.21.15.1:0/2859522275 >> v1:172.21.15.1:6789/0 conn(0x7fb9900cef70 legacy=0x7fb9900b4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.874+0000 7fb994f21700 1 -- 172.21.15.1:0/2859522275 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9900b2fa0 con 0x7fb99006ce40 2024-03-18T18:50:10.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.874+0000 7fb994f21700 1 --2- 172.21.15.1:0/2859522275 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb99006ce40 0x7fb99006d2d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb98800bd50 tx=0x7fb988010730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76ff4bca89964a57 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.874+0000 7f31a5b94700 1 --2- 172.21.15.1:0/3982518009 >> [v2:172.21.15.1:6800/77295,v1:172.21.15.1:6801/77295] conn(0x7f3184074320 0x7f3184076760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.874+0000 7fb98effd700 1 -- 172.21.15.1:0/2859522275 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb988010d80 con 0x7fb99006ce40 2024-03-18T18:50:10.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.874+0000 7fb98effd700 1 -- 172.21.15.1:0/2859522275 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb988018400 con 0x7fb99006ce40 2024-03-18T18:50:10.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.874+0000 7fb98effd700 1 -- 172.21.15.1:0/2859522275 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9880187b0 con 0x7fb99006ce40 2024-03-18T18:50:10.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.875+0000 7ff3467fc700 1 -- 172.21.15.1:0/3719720830 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff34c0267e0 con 0x7ff3541612d0 2024-03-18T18:50:10.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.875+0000 7ff3467fc700 1 --2- 172.21.15.1:0/3719720830 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff33806ac90 0x7ff33806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.875+0000 7ff3467fc700 1 -- 172.21.15.1:0/3719720830 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff34c08bcd0 con 0x7ff3541612d0 2024-03-18T18:50:10.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.875+0000 7fb996984700 1 -- 172.21.15.1:0/2859522275 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb99006ce40 msgr2=0x7fb99006d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.875+0000 7fb996984700 1 --2- 172.21.15.1:0/2859522275 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb99006ce40 0x7fb99006d2d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb98800bd50 tx=0x7fb988010730 comp rx=0 tx=0).stop 2024-03-18T18:50:10.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.875+0000 7f31a5b94700 1 --2- 172.21.15.1:0/3982518009 >> [v2:172.21.15.1:6800/77295,v1:172.21.15.1:6801/77295] conn(0x7f3184074320 0x7f3184076760 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.876+0000 7fb996984700 1 -- 172.21.15.1:0/2859522275 shutdown_connections 2024-03-18T18:50:10.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.876+0000 7fb996984700 1 --2- 172.21.15.1:0/2859522275 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb99006ce40 0x7fb99006d2d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.876+0000 7fb996984700 1 --2- 172.21.15.1:0/2859522275 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9900b3c30 0x7fb9900b2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.876+0000 7fb996984700 1 -- 172.21.15.1:0/2859522275 >> 172.21.15.1:0/2859522275 conn(0x7fb9900b1350 msgr2=0x7fb9900b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.875+0000 7ff3467fc700 1 --2- 172.21.15.1:0/3719720830 >> [v2:172.21.15.165:6816/77476,v1:172.21.15.165:6818/77476] conn(0x7ff338074250 0x7ff338076690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.875+0000 7ff3467fc700 1 -- 172.21.15.1:0/3719720830 --> [v2:172.21.15.165:6816/77476,v1:172.21.15.165:6818/77476] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff338076d60 con 0x7ff338074250 2024-03-18T18:50:10.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.875+0000 7ff3467fc700 1 -- 172.21.15.1:0/3719720830 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff34c056cd0 con 0x7ff3541612d0 2024-03-18T18:50:10.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.876+0000 7fc96f7fe700 1 -- 172.21.15.1:0/372364865 <== osd.0 v2:172.21.15.1:6824/77296 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fc968006bd0 con 0x7fc9680015e0 2024-03-18T18:50:10.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.876+0000 7ff3599d4700 1 --2- 172.21.15.1:0/3719720830 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff33806ac90 0x7ff33806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.876+0000 7f3196ffd700 1 -- 172.21.15.1:0/3982518009 <== osd.1 v2:172.21.15.1:6800/77295 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f3184076e30 con 0x7f3184074320 2024-03-18T18:50:10.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.876+0000 7fb996984700 1 -- 172.21.15.1:0/2859522275 shutdown_connections 2024-03-18T18:50:10.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.876+0000 7ff3591d3700 1 --2- 172.21.15.1:0/3719720830 >> [v2:172.21.15.165:6816/77476,v1:172.21.15.165:6818/77476] conn(0x7ff338074250 0x7ff338076690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.877+0000 7ff3599d4700 1 --2- 172.21.15.1:0/3719720830 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff33806ac90 0x7ff33806d150 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff350000c00 tx=0x7ff350007a50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.877+0000 7ff3591d3700 1 --2- 172.21.15.1:0/3719720830 >> [v2:172.21.15.165:6816/77476,v1:172.21.15.165:6818/77476] conn(0x7ff338074250 0x7ff338076690 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.876+0000 7fb996984700 1 -- 172.21.15.1:0/2859522275 wait complete. 2024-03-18T18:50:10.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.878+0000 7fb996984700 1 Processor -- start 2024-03-18T18:50:10.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.880+0000 7fb996984700 1 -- start start 2024-03-18T18:50:10.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.881+0000 7fb996984700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb99006ce40 0x7fb9901f45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.881+0000 7fb996984700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9900b3c30 0x7fb9901f2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.881+0000 7fb996984700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9900cef70 0x7fb9901f3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.881+0000 7fb996984700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb990156240 con 0x7fb9900cef70 2024-03-18T18:50:10.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.881+0000 7fb996984700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb990156660 con 0x7fb9900b3c30 2024-03-18T18:50:10.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.881+0000 7fb996984700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9901555c0 con 0x7fb99006ce40 2024-03-18T18:50:10.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.882+0000 7fb994f21700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9900cef70 0x7fb9901f3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.882+0000 7fb994f21700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9900cef70 0x7fb9901f3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45298/0 (socket says 172.21.15.1:45298) 2024-03-18T18:50:10.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.882+0000 7fb994f21700 1 -- 172.21.15.1:0/744693426 learned_addr learned my addr 172.21.15.1:0/744693426 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:10.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.883+0000 7fb994f21700 1 -- 172.21.15.1:0/744693426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb99006ce40 msgr2=0x7fb9901f45b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T18:50:10.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.883+0000 7fb994f21700 1 --2- 172.21.15.1:0/744693426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb99006ce40 0x7fb9901f45b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.883+0000 7fb994f21700 1 -- 172.21.15.1:0/744693426 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9900b3c30 msgr2=0x7fb9901f2d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T18:50:10.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.883+0000 7fb994f21700 1 --2- 172.21.15.1:0/744693426 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9900b3c30 0x7fb9901f2d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.883+0000 7fb994f21700 1 -- 172.21.15.1:0/744693426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98800b980 con 0x7fb9900cef70 2024-03-18T18:50:10.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.883+0000 7fb994f21700 1 --2- 172.21.15.1:0/744693426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9900cef70 0x7fb9901f3290 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb98800a2c0 tx=0x7fb988004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.884+0000 7fb98cff9700 1 -- 172.21.15.1:0/744693426 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb988010070 con 0x7fb9900cef70 2024-03-18T18:50:10.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.884+0000 7fb98cff9700 1 -- 172.21.15.1:0/744693426 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb98800a870 con 0x7fb9900cef70 2024-03-18T18:50:10.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.884+0000 7fb98cff9700 1 -- 172.21.15.1:0/744693426 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb98800ac20 con 0x7fb9900cef70 2024-03-18T18:50:10.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.884+0000 7fb996984700 1 -- 172.21.15.1:0/744693426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9901558a0 con 0x7fb9900cef70 2024-03-18T18:50:10.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.884+0000 7fb996984700 1 -- 172.21.15.1:0/744693426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb990155e60 con 0x7fb9900cef70 2024-03-18T18:50:10.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.885+0000 7fca77a36700 1 -- 172.21.15.1:0/2519178950 --> [v2:172.21.15.165:6800/77475,v1:172.21.15.165:6801/77475] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fca7015b630 con 0x7fca700cfc10 2024-03-18T18:50:10.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.886+0000 7fca65ffb700 1 -- 172.21.15.1:0/2519178950 <== osd.5 v2:172.21.15.165:6800/77475 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fca60009c10 con 0x7fca700cfc10 2024-03-18T18:50:10.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.887+0000 7fb98cff9700 1 -- 172.21.15.1:0/744693426 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb9880173b0 con 0x7fb9900cef70 2024-03-18T18:50:10.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.887+0000 7fb98cff9700 1 --2- 172.21.15.1:0/744693426 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb97c06ac90 0x7fb97c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.887+0000 7fb98cff9700 1 -- 172.21.15.1:0/744693426 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb98808a9e0 con 0x7fb9900cef70 2024-03-18T18:50:10.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.887+0000 7fb996984700 1 --2- 172.21.15.1:0/744693426 >> [v2:172.21.15.1:6816/77310,v1:172.21.15.1:6817/77310] conn(0x7fb990154890 0x7fb990154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:10.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.887+0000 7fb996984700 1 -- 172.21.15.1:0/744693426 --> [v2:172.21.15.1:6816/77310,v1:172.21.15.1:6817/77310] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb990073940 con 0x7fb990154890 2024-03-18T18:50:10.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.887+0000 7ff3467fc700 1 -- 172.21.15.1:0/3719720830 <== osd.7 v2:172.21.15.165:6816/77476 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7ff338076d60 con 0x7ff338074250 2024-03-18T18:50:10.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.890+0000 7fb98ffff700 1 --2- 172.21.15.1:0/744693426 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb97c06ac90 0x7fb97c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.890+0000 7fb98ffff700 1 --2- 172.21.15.1:0/744693426 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb97c06ac90 0x7fb97c06d150 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb984009710 tx=0x7fb984006c60 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.891+0000 7fb98f7fe700 1 --2- 172.21.15.1:0/744693426 >> [v2:172.21.15.1:6816/77310,v1:172.21.15.1:6817/77310] conn(0x7fb990154890 0x7fb990154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:10.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.891+0000 7fb98f7fe700 1 --2- 172.21.15.1:0/744693426 >> [v2:172.21.15.1:6816/77310,v1:172.21.15.1:6817/77310] conn(0x7fb990154890 0x7fb990154c70 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:10.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.891+0000 7fca77a36700 1 -- 172.21.15.1:0/2519178950 >> [v2:172.21.15.165:6800/77475,v1:172.21.15.165:6801/77475] conn(0x7fca700cfc10 msgr2=0x7fca700b5b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.891+0000 7fca77a36700 1 --2- 172.21.15.1:0/2519178950 >> [v2:172.21.15.165:6800/77475,v1:172.21.15.165:6801/77475] conn(0x7fca700cfc10 0x7fca700b5b30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.892+0000 7fca77a36700 1 -- 172.21.15.1:0/2519178950 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fca5406ad90 msgr2=0x7fca5406d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.892+0000 7fca77a36700 1 --2- 172.21.15.1:0/2519178950 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fca5406ad90 0x7fca5406d250 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fca68000f80 tx=0x7fca68009000 comp rx=0 tx=0).stop 2024-03-18T18:50:10.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.892+0000 7fca77a36700 1 -- 172.21.15.1:0/2519178950 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca70099a40 msgr2=0x7fca701f8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.892+0000 7fca77a36700 1 --2- 172.21.15.1:0/2519178950 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca70099a40 0x7fca701f8060 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fca6c00b3c0 tx=0x7fca6c00b6d0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.892+0000 7fca75fd3700 1 -- 172.21.15.1:0/2519178950 reap_dead start 2024-03-18T18:50:10.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.892+0000 7fca77a36700 1 -- 172.21.15.1:0/2519178950 shutdown_connections 2024-03-18T18:50:10.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.892+0000 7fca77a36700 1 -- 172.21.15.1:0/2519178950 >> 172.21.15.1:0/2519178950 conn(0x7fca700b2c50 msgr2=0x7fca700b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.892+0000 7fca77a36700 1 -- 172.21.15.1:0/2519178950 shutdown_connections 2024-03-18T18:50:10.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.893+0000 7fca77a36700 1 -- 172.21.15.1:0/2519178950 wait complete. 2024-03-18T18:50:10.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.893+0000 7fb98cff9700 1 -- 172.21.15.1:0/744693426 <== osd.3 v2:172.21.15.1:6816/77310 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb9800011a0 con 0x7fb990154890 2024-03-18T18:50:10.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.894+0000 7fe88c4a0700 1 -- 172.21.15.1:0/788342128 --> [v2:172.21.15.1:6808/77305,v1:172.21.15.1:6809/77305] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe86c002d40 con 0x7fe868074250 2024-03-18T18:50:10.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.895+0000 7fe87affd700 1 -- 172.21.15.1:0/788342128 <== osd.2 v2:172.21.15.1:6808/77305 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe86c002d40 con 0x7fe868074250 2024-03-18T18:50:10.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.896+0000 7f3045d73700 1 -- 172.21.15.1:0/3339264209 --> [v2:172.21.15.165:6808/77474,v1:172.21.15.165:6809/77474] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f304006e100 con 0x7f3040071220 2024-03-18T18:50:10.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.896+0000 7f3027fff700 1 -- 172.21.15.1:0/3339264209 <== osd.4 v2:172.21.15.165:6808/77474 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3038013390 con 0x7f3040071220 2024-03-18T18:50:10.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.897+0000 7fe88c4a0700 1 -- 172.21.15.1:0/788342128 >> [v2:172.21.15.1:6808/77305,v1:172.21.15.1:6809/77305] conn(0x7fe868074250 msgr2=0x7fe868076690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.897+0000 7fe88c4a0700 1 --2- 172.21.15.1:0/788342128 >> [v2:172.21.15.1:6808/77305,v1:172.21.15.1:6809/77305] conn(0x7fe868074250 0x7fe868076690 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.897+0000 7f3045d73700 1 -- 172.21.15.1:0/3339264209 >> [v2:172.21.15.165:6808/77474,v1:172.21.15.165:6809/77474] conn(0x7f3040071220 msgr2=0x7f30400611f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.897+0000 7f3045d73700 1 --2- 172.21.15.1:0/3339264209 >> [v2:172.21.15.165:6808/77474,v1:172.21.15.165:6809/77474] conn(0x7f3040071220 0x7f30400611f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.897+0000 7f3045d73700 1 -- 172.21.15.1:0/3339264209 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f302c06ad60 msgr2=0x7f302c06d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.897+0000 7f3045d73700 1 --2- 172.21.15.1:0/3339264209 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f302c06ad60 0x7f302c06d220 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f304015ad00 tx=0x7f3034009040 comp rx=0 tx=0).stop 2024-03-18T18:50:10.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.897+0000 7f3045d73700 1 -- 172.21.15.1:0/3339264209 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f304006dca0 msgr2=0x7f30400cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.897+0000 7f3045d73700 1 --2- 172.21.15.1:0/3339264209 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f304006dca0 0x7f30400cf050 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3030016000 tx=0x7f3030004750 comp rx=0 tx=0).stop 2024-03-18T18:50:10.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.897+0000 7f303ffff700 1 -- 172.21.15.1:0/3339264209 reap_dead start 2024-03-18T18:50:10.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.898+0000 7fe88c4a0700 1 -- 172.21.15.1:0/788342128 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe86806ac90 msgr2=0x7fe86806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.898+0000 7fe88c4a0700 1 --2- 172.21.15.1:0/788342128 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe86806ac90 0x7fe86806d150 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe884148e80 tx=0x7fe87c011000 comp rx=0 tx=0).stop 2024-03-18T18:50:10.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.898+0000 7fe88c4a0700 1 -- 172.21.15.1:0/788342128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8840b3c60 msgr2=0x7fe884148480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.898+0000 7fe88c4a0700 1 --2- 172.21.15.1:0/788342128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8840b3c60 0x7fe884148480 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe88000c480 tx=0x7fe88000c790 comp rx=0 tx=0).stop 2024-03-18T18:50:10.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.901+0000 7fe88aa3d700 1 -- 172.21.15.1:0/788342128 reap_dead start 2024-03-18T18:50:10.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.901+0000 7fe88c4a0700 1 -- 172.21.15.1:0/788342128 shutdown_connections 2024-03-18T18:50:10.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.901+0000 7fe88c4a0700 1 -- 172.21.15.1:0/788342128 >> 172.21.15.1:0/788342128 conn(0x7fe88406ab70 msgr2=0x7fe884137450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.901+0000 7fe7627ab700 1 -- 172.21.15.1:0/1673380968 --> [v2:172.21.15.165:6817/77477,v1:172.21.15.165:6819/77477] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe73c005cb0 con 0x7fe73c0015e0 2024-03-18T18:50:10.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.902+0000 7fe758ff9700 1 -- 172.21.15.1:0/1673380968 <== osd.6 v2:172.21.15.165:6817/77477 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe73c005cb0 con 0x7fe73c0015e0 2024-03-18T18:50:10.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.903+0000 7fe7627ab700 1 -- 172.21.15.1:0/1673380968 >> [v2:172.21.15.165:6817/77477,v1:172.21.15.165:6819/77477] conn(0x7fe73c0015e0 msgr2=0x7fe73c003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.903+0000 7fe7627ab700 1 --2- 172.21.15.1:0/1673380968 >> [v2:172.21.15.165:6817/77477,v1:172.21.15.165:6819/77477] conn(0x7fe73c0015e0 0x7fe73c003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.903+0000 7fe88c4a0700 1 -- 172.21.15.1:0/788342128 shutdown_connections 2024-03-18T18:50:10.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.903+0000 7fe88c4a0700 1 -- 172.21.15.1:0/788342128 wait complete. 2024-03-18T18:50:10.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.905+0000 7f31a85f9700 1 -- 172.21.15.1:0/3982518009 --> [v2:172.21.15.1:6800/77295,v1:172.21.15.1:6801/77295] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f31a006cdf0 con 0x7f3184074320 2024-03-18T18:50:10.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.906+0000 7fc98d11d700 1 -- 172.21.15.1:0/372364865 --> [v2:172.21.15.1:6824/77296,v1:172.21.15.1:6825/77296] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc968005c50 con 0x7fc9680015e0 2024-03-18T18:50:10.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.906+0000 7fe7627ab700 1 -- 172.21.15.1:0/1673380968 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe74806ace0 msgr2=0x7fe74806d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.906+0000 7fe7627ab700 1 --2- 172.21.15.1:0/1673380968 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe74806ace0 0x7fe74806d1a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0b5c90 tx=0x7fe750008030 comp rx=0 tx=0).stop 2024-03-18T18:50:10.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.906+0000 7fe7627ab700 1 -- 172.21.15.1:0/1673380968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe75c0b3d40 msgr2=0x7fe75c151f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.906+0000 7fe7627ab700 1 --2- 172.21.15.1:0/1673380968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe75c0b3d40 0x7fe75c151f80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe75400ba50 tx=0x7fe75400bd60 comp rx=0 tx=0).stop 2024-03-18T18:50:10.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.906+0000 7fe760d48700 1 -- 172.21.15.1:0/1673380968 reap_dead start 2024-03-18T18:50:10.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.906+0000 7fe7627ab700 1 -- 172.21.15.1:0/1673380968 shutdown_connections 2024-03-18T18:50:10.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.906+0000 7fe7627ab700 1 -- 172.21.15.1:0/1673380968 >> 172.21.15.1:0/1673380968 conn(0x7fe75c0d3140 msgr2=0x7fe75c06c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.906+0000 7fc96f7fe700 1 -- 172.21.15.1:0/372364865 <== osd.0 v2:172.21.15.1:6824/77296 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc968005c50 con 0x7fc9680015e0 2024-03-18T18:50:10.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.907+0000 7f3045d73700 1 -- 172.21.15.1:0/3339264209 shutdown_connections 2024-03-18T18:50:10.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.907+0000 7f3045d73700 1 -- 172.21.15.1:0/3339264209 >> 172.21.15.1:0/3339264209 conn(0x7f30400b1350 msgr2=0x7f30401596e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.907+0000 7fc98d11d700 1 -- 172.21.15.1:0/372364865 >> [v2:172.21.15.1:6824/77296,v1:172.21.15.1:6825/77296] conn(0x7fc9680015e0 msgr2=0x7fc968003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.907+0000 7fc98d11d700 1 --2- 172.21.15.1:0/372364865 >> [v2:172.21.15.1:6824/77296,v1:172.21.15.1:6825/77296] conn(0x7fc9680015e0 0x7fc968003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.907+0000 7fc98d11d700 1 -- 172.21.15.1:0/372364865 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc97406ad60 msgr2=0x7fc97406d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.907+0000 7fc98d11d700 1 --2- 172.21.15.1:0/372364865 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc97406ad60 0x7fc97406d220 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc980001370 tx=0x7fc980007000 comp rx=0 tx=0).stop 2024-03-18T18:50:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.907+0000 7fc98d11d700 1 -- 172.21.15.1:0/372364865 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc98814ae60 msgr2=0x7fc9880ccf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.907+0000 7fc98d11d700 1 --2- 172.21.15.1:0/372364865 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc98814ae60 0x7fc9880ccf50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00e810 tx=0x7fc97c00eb20 comp rx=0 tx=0).stop 2024-03-18T18:50:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.907+0000 7fc98759e700 1 -- 172.21.15.1:0/372364865 reap_dead start 2024-03-18T18:50:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.907+0000 7fc98d11d700 1 -- 172.21.15.1:0/372364865 shutdown_connections 2024-03-18T18:50:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.907+0000 7fc98d11d700 1 -- 172.21.15.1:0/372364865 >> 172.21.15.1:0/372364865 conn(0x7fc98809c560 msgr2=0x7fc9880b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.907+0000 7f3045d73700 1 -- 172.21.15.1:0/3339264209 shutdown_connections 2024-03-18T18:50:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.907+0000 7f3045d73700 1 -- 172.21.15.1:0/3339264209 wait complete. 2024-03-18T18:50:10.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.908+0000 7fe7627ab700 1 -- 172.21.15.1:0/1673380968 shutdown_connections 2024-03-18T18:50:10.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.912+0000 7fe7627ab700 1 -- 172.21.15.1:0/1673380968 wait complete. 2024-03-18T18:50:10.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.912+0000 7fc98d11d700 1 -- 172.21.15.1:0/372364865 shutdown_connections 2024-03-18T18:50:10.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.912+0000 7fc98d11d700 1 -- 172.21.15.1:0/372364865 wait complete. 2024-03-18T18:50:10.933 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-18T18:50:10.933 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-18T18:50:10.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.914+0000 7fb996984700 1 -- 172.21.15.1:0/744693426 --> [v2:172.21.15.1:6816/77310,v1:172.21.15.1:6817/77310] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb990156380 con 0x7fb990154890 2024-03-18T18:50:10.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.915+0000 7fb98cff9700 1 -- 172.21.15.1:0/744693426 <== osd.3 v2:172.21.15.1:6816/77310 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb990156380 con 0x7fb990154890 2024-03-18T18:50:10.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.915+0000 7fb996984700 1 -- 172.21.15.1:0/744693426 >> [v2:172.21.15.1:6816/77310,v1:172.21.15.1:6817/77310] conn(0x7fb990154890 msgr2=0x7fb990154c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.915+0000 7fb996984700 1 --2- 172.21.15.1:0/744693426 >> [v2:172.21.15.1:6816/77310,v1:172.21.15.1:6817/77310] conn(0x7fb990154890 0x7fb990154c70 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.918+0000 7f3196ffd700 1 -- 172.21.15.1:0/3982518009 <== osd.1 v2:172.21.15.1:6800/77295 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f31900014c0 con 0x7f3184074320 2024-03-18T18:50:10.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.919+0000 7ff35bc38700 1 -- 172.21.15.1:0/3719720830 --> [v2:172.21.15.165:6816/77476,v1:172.21.15.165:6818/77476] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff33c002d40 con 0x7ff338074250 2024-03-18T18:50:10.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.918+0000 7fb996984700 1 -- 172.21.15.1:0/744693426 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb97c06ac90 msgr2=0x7fb97c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.918+0000 7fb996984700 1 --2- 172.21.15.1:0/744693426 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb97c06ac90 0x7fb97c06d150 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb984009710 tx=0x7fb984006c60 comp rx=0 tx=0).stop 2024-03-18T18:50:10.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.918+0000 7fb996984700 1 -- 172.21.15.1:0/744693426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9900cef70 msgr2=0x7fb9901f3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.918+0000 7fb996984700 1 --2- 172.21.15.1:0/744693426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9900cef70 0x7fb9901f3290 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb98800a2c0 tx=0x7fb988004ad0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.918+0000 7fb994f21700 1 -- 172.21.15.1:0/744693426 reap_dead start 2024-03-18T18:50:10.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.919+0000 7fb996984700 1 -- 172.21.15.1:0/744693426 shutdown_connections 2024-03-18T18:50:10.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.919+0000 7fb996984700 1 -- 172.21.15.1:0/744693426 >> 172.21.15.1:0/744693426 conn(0x7fb9900b1350 msgr2=0x7fb9900afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.919+0000 7fb996984700 1 -- 172.21.15.1:0/744693426 shutdown_connections 2024-03-18T18:50:10.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.919+0000 7fb996984700 1 -- 172.21.15.1:0/744693426 wait complete. 2024-03-18T18:50:10.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.923+0000 7ff3467fc700 1 -- 172.21.15.1:0/3719720830 <== osd.7 v2:172.21.15.165:6816/77476 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff33c002d40 con 0x7ff338074250 2024-03-18T18:50:10.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.923+0000 7ff35bc38700 1 -- 172.21.15.1:0/3719720830 >> [v2:172.21.15.165:6816/77476,v1:172.21.15.165:6818/77476] conn(0x7ff338074250 msgr2=0x7ff338076690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.923+0000 7ff35bc38700 1 --2- 172.21.15.1:0/3719720830 >> [v2:172.21.15.165:6816/77476,v1:172.21.15.165:6818/77476] conn(0x7ff338074250 0x7ff338076690 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.923+0000 7ff35bc38700 1 -- 172.21.15.1:0/3719720830 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff33806ac90 msgr2=0x7ff33806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.923+0000 7ff35bc38700 1 --2- 172.21.15.1:0/3719720830 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff33806ac90 0x7ff33806d150 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff350000c00 tx=0x7ff350007a50 comp rx=0 tx=0).stop 2024-03-18T18:50:10.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.923+0000 7ff35bc38700 1 -- 172.21.15.1:0/3719720830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3541612d0 msgr2=0x7ff35415f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.923+0000 7ff35bc38700 1 --2- 172.21.15.1:0/3719720830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3541612d0 0x7ff35415f0b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff34c00d010 tx=0x7ff34c004ad0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.923+0000 7ff35a1d5700 1 -- 172.21.15.1:0/3719720830 reap_dead start 2024-03-18T18:50:10.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.925+0000 7f31a85f9700 1 -- 172.21.15.1:0/3982518009 >> [v2:172.21.15.1:6800/77295,v1:172.21.15.1:6801/77295] conn(0x7f3184074320 msgr2=0x7f3184076760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.925+0000 7f31a85f9700 1 --2- 172.21.15.1:0/3982518009 >> [v2:172.21.15.1:6800/77295,v1:172.21.15.1:6801/77295] conn(0x7f3184074320 0x7f3184076760 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:10.939 INFO:teuthology.orchestra.run.smithi001.stdout:55834574857 2024-03-18T18:50:10.939 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-18T18:50:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.925+0000 7ff35bc38700 1 -- 172.21.15.1:0/3719720830 shutdown_connections 2024-03-18T18:50:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.925+0000 7ff35bc38700 1 -- 172.21.15.1:0/3719720830 >> 172.21.15.1:0/3719720830 conn(0x7ff35409c560 msgr2=0x7ff35415b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.925+0000 7ff35bc38700 1 -- 172.21.15.1:0/3719720830 shutdown_connections 2024-03-18T18:50:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.925+0000 7ff35bc38700 1 -- 172.21.15.1:0/3719720830 wait complete. 2024-03-18T18:50:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.926+0000 7f31a85f9700 1 -- 172.21.15.1:0/3982518009 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f318406ad60 msgr2=0x7f318406d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.926+0000 7f31a85f9700 1 --2- 172.21.15.1:0/3982518009 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f318406ad60 0x7f318406d220 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f319c000c00 tx=0x7f319c01a000 comp rx=0 tx=0).stop 2024-03-18T18:50:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.926+0000 7f31a85f9700 1 -- 172.21.15.1:0/3982518009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a00ce450 msgr2=0x7f31a014c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:10.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.926+0000 7f31a85f9700 1 --2- 172.21.15.1:0/3982518009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31a00ce450 0x7f31a014c260 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f319800b780 tx=0x7f319800ba90 comp rx=0 tx=0).stop 2024-03-18T18:50:10.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.926+0000 7f31a6b96700 1 -- 172.21.15.1:0/3982518009 reap_dead start 2024-03-18T18:50:10.941 INFO:teuthology.orchestra.run.smithi001.stdout:55834574857 2024-03-18T18:50:10.942 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-18T18:50:10.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.927+0000 7f31a85f9700 1 -- 172.21.15.1:0/3982518009 shutdown_connections 2024-03-18T18:50:10.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.927+0000 7f31a85f9700 1 -- 172.21.15.1:0/3982518009 >> 172.21.15.1:0/3982518009 conn(0x7f31a006a750 msgr2=0x7f31a0157750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:10.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.927+0000 7f31a85f9700 1 -- 172.21.15.1:0/3982518009 shutdown_connections 2024-03-18T18:50:10.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:10.933+0000 7f31a85f9700 1 -- 172.21.15.1:0/3982518009 wait complete. 2024-03-18T18:50:10.943 INFO:teuthology.orchestra.run.smithi001.stdout:55834574857 2024-03-18T18:50:10.943 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-18T18:50:10.944 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-18T18:50:10.945 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-18T18:50:10.945 INFO:teuthology.orchestra.run.smithi001.stdout:55834574857 2024-03-18T18:50:10.946 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-18T18:50:10.947 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-18T18:50:10.947 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-18T18:50:10.948 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-18T18:50:10.948 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-18T18:50:11.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.137+0000 7f68ae49a700 1 Processor -- start 2024-03-18T18:50:11.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.137+0000 7f68ae49a700 1 -- start start 2024-03-18T18:50:11.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.139+0000 7f68ae49a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68a806d1f0 0x7f68a80706d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.139+0000 7f68ae49a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f68a8070c10 0x7f68a80cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.139+0000 7f68ae49a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a80cc6e0 con 0x7f68a814b1b0 2024-03-18T18:50:11.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.139+0000 7f68ae49a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68a814b590 con 0x7f68a8070c10 2024-03-18T18:50:11.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.139+0000 7f68ae49a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a8071a00 con 0x7f68a806d1f0 2024-03-18T18:50:11.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.139+0000 7f68aca37700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f68a8070c10 0x7f68a80cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.139+0000 7f68aca37700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f68a8070c10 0x7f68a80cd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45302/0 (socket says 172.21.15.1:45302) 2024-03-18T18:50:11.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.139+0000 7f68aca37700 1 -- 172.21.15.1:0/3726426016 learned_addr learned my addr 172.21.15.1:0/3726426016 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:11.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.140+0000 7f68a77fe700 1 --2- 172.21.15.1:0/3726426016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68a806d1f0 0x7f68a80706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.140+0000 7f68aca37700 1 -- 172.21.15.1:0/3726426016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68a806d1f0 msgr2=0x7f68a80706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:11.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.140+0000 7f68aca37700 1 --2- 172.21.15.1:0/3726426016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68a806d1f0 0x7f68a80706d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.140+0000 7f68aca37700 1 -- 172.21.15.1:0/3726426016 >> v1:172.21.15.1:6789/0 conn(0x7f68a814b1b0 legacy=0x7f68a806cae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T18:50:11.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.140+0000 7f68aca37700 1 -- 172.21.15.1:0/3726426016 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a8071440 con 0x7f68a8070c10 2024-03-18T18:50:11.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.140+0000 7f68aca37700 1 --2- 172.21.15.1:0/3726426016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a8070c10 0x7f68a80cd180 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f68980098f0 tx=0x7f6898009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faca88f922a71802 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.141+0000 7f68a6ffd700 1 -- 172.21.15.1:0/3726426016 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6898004230 con 0x7f68a8070c10 2024-03-18T18:50:11.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.141+0000 7f68a6ffd700 1 -- 172.21.15.1:0/3726426016 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6898004390 con 0x7f68a8070c10 2024-03-18T18:50:11.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.141+0000 7f68a6ffd700 1 -- 172.21.15.1:0/3726426016 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68980036a0 con 0x7f68a8070c10 2024-03-18T18:50:11.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.142+0000 7f68ae49a700 1 -- 172.21.15.1:0/3726426016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a8070c10 msgr2=0x7f68a80cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.142+0000 7f68ae49a700 1 --2- 172.21.15.1:0/3726426016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a8070c10 0x7f68a80cd180 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f68980098f0 tx=0x7f6898009c00 comp rx=0 tx=0).stop 2024-03-18T18:50:11.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.143+0000 7f68ae49a700 1 -- 172.21.15.1:0/3726426016 shutdown_connections 2024-03-18T18:50:11.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.143+0000 7f68ae49a700 1 --2- 172.21.15.1:0/3726426016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a8070c10 0x7f68a80cd180 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.143+0000 7f68ae49a700 1 --2- 172.21.15.1:0/3726426016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68a806d1f0 0x7f68a80706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.143+0000 7f68ae49a700 1 -- 172.21.15.1:0/3726426016 >> 172.21.15.1:0/3726426016 conn(0x7f68a806a750 msgr2=0x7f68a806ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:11.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.143+0000 7f68ae49a700 1 -- 172.21.15.1:0/3726426016 shutdown_connections 2024-03-18T18:50:11.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.144+0000 7f68ae49a700 1 -- 172.21.15.1:0/3726426016 wait complete. 2024-03-18T18:50:11.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.145+0000 7f68ae49a700 1 Processor -- start 2024-03-18T18:50:11.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.146+0000 7f68ae49a700 1 -- start start 2024-03-18T18:50:11.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.147+0000 7f68ae49a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68a806d1f0 0x7f68a80ad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.147+0000 7f68ae49a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68a8070c10 0x7f68a8163cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.147+0000 7f68ae49a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a814b1b0 0x7f68a8164210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.147+0000 7f68ae49a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a8163940 con 0x7f68a814b1b0 2024-03-18T18:50:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.147+0000 7f68ae49a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a8163660 con 0x7f68a8070c10 2024-03-18T18:50:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.147+0000 7f68ae49a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a80a85d0 con 0x7f68a806d1f0 2024-03-18T18:50:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.148+0000 7f68aca37700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a814b1b0 0x7f68a8164210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.148+0000 7f68aca37700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a814b1b0 0x7f68a8164210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45306/0 (socket says 172.21.15.1:45306) 2024-03-18T18:50:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.148+0000 7f68aca37700 1 -- 172.21.15.1:0/3029651617 learned_addr learned my addr 172.21.15.1:0/3029651617 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.149+0000 7f68aca37700 1 -- 172.21.15.1:0/3029651617 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68a806d1f0 msgr2=0x7f68a80ad0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T18:50:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.149+0000 7f68aca37700 1 --2- 172.21.15.1:0/3029651617 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68a806d1f0 0x7f68a80ad0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.149+0000 7f68aca37700 1 -- 172.21.15.1:0/3029651617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68a8070c10 msgr2=0x7f68a8163cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T18:50:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.149+0000 7f68aca37700 1 --2- 172.21.15.1:0/3029651617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68a8070c10 0x7f68a8163cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.149+0000 7f68aca37700 1 -- 172.21.15.1:0/3029651617 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6898009550 con 0x7f68a814b1b0 2024-03-18T18:50:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.149+0000 7f68aca37700 1 --2- 172.21.15.1:0/3029651617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a814b1b0 0x7f68a8164210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f68a806d660 tx=0x7f6898004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.152+0000 7f68a4ff9700 1 -- 172.21.15.1:0/3029651617 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689800b660 con 0x7f68a814b1b0 2024-03-18T18:50:11.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.152+0000 7f68a4ff9700 1 -- 172.21.15.1:0/3029651617 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689800b7c0 con 0x7f68a814b1b0 2024-03-18T18:50:11.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.152+0000 7f68a4ff9700 1 -- 172.21.15.1:0/3029651617 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689800bb30 con 0x7f68a814b1b0 2024-03-18T18:50:11.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.152+0000 7f68ae49a700 1 -- 172.21.15.1:0/3029651617 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a80a88b0 con 0x7f68a814b1b0 2024-03-18T18:50:11.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.152+0000 7f68ae49a700 1 -- 172.21.15.1:0/3029651617 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a80ad5e0 con 0x7f68a814b1b0 2024-03-18T18:50:11.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.156+0000 7f68a4ff9700 1 -- 172.21.15.1:0/3029651617 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6898022070 con 0x7f68a814b1b0 2024-03-18T18:50:11.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.157+0000 7f68a4ff9700 1 --2- 172.21.15.1:0/3029651617 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f689406ad60 0x7f689406d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.157+0000 7f68a4ff9700 1 -- 172.21.15.1:0/3029651617 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f689808b0f0 con 0x7f68a814b1b0 2024-03-18T18:50:11.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.159+0000 7f68a7fff700 1 --2- 172.21.15.1:0/3029651617 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f689406ad60 0x7f689406d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.159+0000 7f68a7fff700 1 --2- 172.21.15.1:0/3029651617 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f689406ad60 0x7f689406d220 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f689c006fd0 tx=0x7f689c006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.165+0000 7f68ae49a700 1 -- 172.21.15.1:0/3029651617 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68880052f0 con 0x7f68a814b1b0 2024-03-18T18:50:11.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.174+0000 7f68a4ff9700 1 -- 172.21.15.1:0/3029651617 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6898056050 con 0x7f68a814b1b0 2024-03-18T18:50:11.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.204+0000 7f467704a700 1 Processor -- start 2024-03-18T18:50:11.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.204+0000 7f3f47a24700 1 Processor -- start 2024-03-18T18:50:11.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.204+0000 7f3f47a24700 1 -- start start 2024-03-18T18:50:11.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.204+0000 7f467704a700 1 -- start start 2024-03-18T18:50:11.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.206+0000 7ff9796dd700 1 Processor -- start 2024-03-18T18:50:11.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.206+0000 7f467704a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f467015b6c0 0x7f46700b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.206+0000 7f467704a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f46700cd430 0x7f46700c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.206+0000 7f467704a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46700d01f0 con 0x7f46700c8600 2024-03-18T18:50:11.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.206+0000 7f467704a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46700ccfa0 con 0x7f46700cd430 2024-03-18T18:50:11.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.206+0000 7f467704a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46700d1db0 con 0x7f467015b6c0 2024-03-18T18:50:11.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7ff5738b8700 1 Processor -- start 2024-03-18T18:50:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.206+0000 7ff9796dd700 1 -- start start 2024-03-18T18:50:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7ff9796dd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff97414a480 0x7ff97406cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7ff9796dd700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff974070580 con 0x7ff97406d080 2024-03-18T18:50:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7ff9796dd700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9740713e0 con 0x7ff9740b3c60 2024-03-18T18:50:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7ff9796dd700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff97406d510 con 0x7ff97414a480 2024-03-18T18:50:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.206+0000 7f4674de6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f467015b6c0 0x7f46700b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7f466ffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f46700cd430 0x7f46700c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7f466ffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f46700cd430 0x7f46700c80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45310/0 (socket says 172.21.15.1:45310) 2024-03-18T18:50:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7f466ffff700 1 -- 172.21.15.1:0/1908055323 learned_addr learned my addr 172.21.15.1:0/1908055323 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7f466ffff700 1 -- 172.21.15.1:0/1908055323 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f467015b6c0 msgr2=0x7f46700b2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7f466ffff700 1 --2- 172.21.15.1:0/1908055323 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f467015b6c0 0x7f46700b2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7f466ffff700 1 -- 172.21.15.1:0/1908055323 >> v1:172.21.15.165:6789/0 conn(0x7f46700c8600 legacy=0x7f46700cedd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T18:50:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7f466ffff700 1 -- 172.21.15.1:0/1908055323 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46700cff10 con 0x7f46700cd430 2024-03-18T18:50:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.206+0000 7f3f47a24700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3f40161100 0x7f3f400b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.206+0000 7f3f47a24700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3f4008a2d0 0x7f3f4008a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.206+0000 7f3f47a24700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f400b1f30 con 0x7f3f4016daf0 2024-03-18T18:50:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.206+0000 7f3f47a24700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f401614e0 con 0x7f3f40161100 2024-03-18T18:50:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.206+0000 7f3f47a24700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f4008acc0 con 0x7f3f4008a2d0 2024-03-18T18:50:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7f3f457c0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3f40161100 0x7f3f400b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7f3f45fc1700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f3f4016daf0 0x7f3f4016df80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:38336/0 (socket says 172.21.15.1:38336) 2024-03-18T18:50:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7f3f45fc1700 1 -- 172.21.15.1:0/1921409567 learned_addr learned my addr 172.21.15.1:0/1921409567 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7f3f457c0700 1 -- 172.21.15.1:0/1921409567 >> v2:172.21.15.165:3300/0 conn(0x7f3f4008a2d0 msgr2=0x7f3f4008a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7f3f457c0700 1 --2- 172.21.15.1:0/1921409567 >> v2:172.21.15.165:3300/0 conn(0x7f3f4008a2d0 0x7f3f4008a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7f3f457c0700 1 -- 172.21.15.1:0/1921409567 >> v1:172.21.15.165:6789/0 conn(0x7f3f4016daf0 legacy=0x7f3f4016df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7f3f457c0700 1 -- 172.21.15.1:0/1921409567 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4006d730 con 0x7f3f40161100 2024-03-18T18:50:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.208+0000 7f466ffff700 1 --2- 172.21.15.1:0/1908055323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46700cd430 0x7f46700c80c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f46640098f0 tx=0x7f4664009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74a0bdc3f7cc0f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.208+0000 7ff9727fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff97406d080 0x7ff9740b3550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60296/0 (socket says 172.21.15.1:60296) 2024-03-18T18:50:11.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.208+0000 7ff9727fc700 1 -- 172.21.15.1:0/2041434067 learned_addr learned my addr 172.21.15.1:0/2041434067 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:11.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.208+0000 7f3f457c0700 1 --2- 172.21.15.1:0/1921409567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f40161100 0x7f3f400b36f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3f400722c0 tx=0x7f3f3c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3262731b715d0f6e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.208+0000 7ff972ffd700 1 --2- 172.21.15.1:0/2041434067 >> v2:172.21.15.165:3300/0 conn(0x7ff97414a480 0x7ff97406cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.208+0000 7f466f7fe700 1 -- 172.21.15.1:0/1908055323 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4664004230 con 0x7f46700cd430 2024-03-18T18:50:11.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.207+0000 7ff5738b8700 1 -- start start 2024-03-18T18:50:11.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.209+0000 7ff5738b8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff56c0cea60 0x7ff56c159de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.209+0000 7ff5738b8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff56c15a320 0x7ff56c070790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.209+0000 7ff5738b8700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56c0cfee0 con 0x7ff56c150400 2024-03-18T18:50:11.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.209+0000 7ff5738b8700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff56c0cbad0 con 0x7ff56c0cea60 2024-03-18T18:50:11.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.209+0000 7ff5738b8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c15d0c0 con 0x7ff56c15a320 2024-03-18T18:50:11.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.209+0000 7ff972ffd700 1 -- 172.21.15.1:0/2041434067 >> v1:172.21.15.165:6789/0 conn(0x7ff9740b3c60 legacy=0x7ff9740cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.209+0000 7ff972ffd700 1 -- 172.21.15.1:0/2041434067 >> v1:172.21.15.1:6789/0 conn(0x7ff97406d080 legacy=0x7ff9740b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.209+0000 7ff972ffd700 1 -- 172.21.15.1:0/2041434067 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9740707d0 con 0x7ff97414a480 2024-03-18T18:50:11.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.209+0000 7ff972ffd700 1 --2- 172.21.15.1:0/2041434067 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff97414a480 0x7ff97406cb40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff964009840 tx=0x7ff964009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fee506706083459d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.209+0000 7ff971ffb700 1 -- 172.21.15.1:0/2041434067 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff964004230 con 0x7ff97414a480 2024-03-18T18:50:11.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.209+0000 7ff971ffb700 1 -- 172.21.15.1:0/2041434067 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff964004390 con 0x7ff97414a480 2024-03-18T18:50:11.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.209+0000 7ff971ffb700 1 -- 172.21.15.1:0/2041434067 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9640036a0 con 0x7ff97414a480 2024-03-18T18:50:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.208+0000 7f466f7fe700 1 -- 172.21.15.1:0/1908055323 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4664004390 con 0x7f46700cd430 2024-03-18T18:50:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.208+0000 7f466f7fe700 1 -- 172.21.15.1:0/1908055323 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46640036a0 con 0x7f46700cd430 2024-03-18T18:50:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.209+0000 7f467704a700 1 -- 172.21.15.1:0/1908055323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46700cd430 msgr2=0x7f46700c80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.209+0000 7f467704a700 1 --2- 172.21.15.1:0/1908055323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46700cd430 0x7f46700c80c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f46640098f0 tx=0x7f4664009c00 comp rx=0 tx=0).stop 2024-03-18T18:50:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.208+0000 7f3f37fff700 1 -- 172.21.15.1:0/1921409567 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c00f940 con 0x7f3f40161100 2024-03-18T18:50:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.208+0000 7f3f37fff700 1 -- 172.21.15.1:0/1921409567 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3c00faa0 con 0x7f3f40161100 2024-03-18T18:50:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.208+0000 7f3f37fff700 1 -- 172.21.15.1:0/1921409567 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c00fe10 con 0x7f3f40161100 2024-03-18T18:50:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.209+0000 7f3f47a24700 1 -- 172.21.15.1:0/1921409567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f40161100 msgr2=0x7f3f400b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.209+0000 7f3f47a24700 1 --2- 172.21.15.1:0/1921409567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f40161100 0x7f3f400b36f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3f400722c0 tx=0x7f3f3c009b20 comp rx=0 tx=0).stop 2024-03-18T18:50:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7f3f47a24700 1 -- 172.21.15.1:0/1921409567 shutdown_connections 2024-03-18T18:50:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7f3f47a24700 1 --2- 172.21.15.1:0/1921409567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f40161100 0x7f3f400b36f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7f3f47a24700 1 --2- 172.21.15.1:0/1921409567 >> v2:172.21.15.165:3300/0 conn(0x7f3f4008a2d0 0x7f3f4008a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7f3f47a24700 1 -- 172.21.15.1:0/1921409567 >> 172.21.15.1:0/1921409567 conn(0x7f3f4006a750 msgr2=0x7f3f4006ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.209+0000 7fbcd0c86700 1 Processor -- start 2024-03-18T18:50:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.209+0000 7fbcd0c86700 1 -- start start 2024-03-18T18:50:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7f3f47a24700 1 -- 172.21.15.1:0/1921409567 shutdown_connections 2024-03-18T18:50:11.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7ff571654700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7ff56c150400 0x7ff56c0ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:38354/0 (socket says 172.21.15.1:38354) 2024-03-18T18:50:11.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7ff571654700 1 -- 172.21.15.1:0/1762631102 learned_addr learned my addr 172.21.15.1:0/1762631102 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:11.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7ff570e53700 1 --2- 172.21.15.1:0/1762631102 >> v2:172.21.15.1:3300/0 conn(0x7ff56c0cea60 0x7ff56c159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7ff563fff700 1 -- 172.21.15.1:0/1762631102 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98572960 0 0) 0x7ff56c0cfee0 con 0x7ff56c150400 2024-03-18T18:50:11.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7ff563fff700 1 -- 172.21.15.1:0/1762631102 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5580035b0 con 0x7ff56c150400 2024-03-18T18:50:11.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7ff570e53700 1 -- 172.21.15.1:0/1762631102 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff56c15a320 msgr2=0x7ff56c070790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T18:50:11.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7ff570e53700 1 --2- 172.21.15.1:0/1762631102 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff56c15a320 0x7ff56c070790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7ff570e53700 1 -- 172.21.15.1:0/1762631102 >> v1:172.21.15.165:6789/0 conn(0x7ff56c150400 legacy=0x7ff56c0ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7ff570e53700 1 -- 172.21.15.1:0/1762631102 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c15d330 con 0x7ff56c0cea60 2024-03-18T18:50:11.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7ff570e53700 1 --2- 172.21.15.1:0/1762631102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c0cea60 0x7ff56c159de0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff56400bd40 tx=0x7ff564009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f71f5b7c7c8ea8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7fbcd0c86700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbccc14b1b0 0x7fbccc06cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7fbcd0c86700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbccc06d020 0x7fbccc0706d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7fbcd0c86700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbccc0cc6e0 con 0x7fbccc070c10 2024-03-18T18:50:11.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7fbcd0c86700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbccc14b590 con 0x7fbccc06d020 2024-03-18T18:50:11.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7fbcd0c86700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbccc06d4d0 con 0x7fbccc14b1b0 2024-03-18T18:50:11.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7ff9796dd700 1 -- 172.21.15.1:0/2041434067 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff97414a480 msgr2=0x7ff97406cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7ff9796dd700 1 --2- 172.21.15.1:0/2041434067 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff97414a480 0x7ff97406cb40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff964009840 tx=0x7ff964009b50 comp rx=0 tx=0).stop 2024-03-18T18:50:11.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7ff563fff700 1 -- 172.21.15.1:0/1762631102 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff564003c20 con 0x7ff56c0cea60 2024-03-18T18:50:11.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7ff563fff700 1 -- 172.21.15.1:0/1762631102 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff564003d80 con 0x7ff56c0cea60 2024-03-18T18:50:11.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7ff563fff700 1 -- 172.21.15.1:0/1762631102 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff564009d00 con 0x7ff56c0cea60 2024-03-18T18:50:11.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7fbcca59c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbccc14b1b0 0x7fbccc06cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7fbccad9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbccc070c10 0x7fbccc0b1f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60302/0 (socket says 172.21.15.1:60302) 2024-03-18T18:50:11.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7fbccad9d700 1 -- 172.21.15.1:0/2583138841 learned_addr learned my addr 172.21.15.1:0/2583138841 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:11.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7fbcc9d9b700 1 --2- 172.21.15.1:0/2583138841 >> v2:172.21.15.1:3300/0 conn(0x7fbccc06d020 0x7fbccc0706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7fbcc9d9b700 1 -- 172.21.15.1:0/2583138841 >> v2:172.21.15.165:3300/0 conn(0x7fbccc14b1b0 msgr2=0x7fbccc06cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:11.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7fbcc9d9b700 1 --2- 172.21.15.1:0/2583138841 >> v2:172.21.15.165:3300/0 conn(0x7fbccc14b1b0 0x7fbccc06cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7fbcc9d9b700 1 -- 172.21.15.1:0/2583138841 >> v1:172.21.15.1:6789/0 conn(0x7fbccc070c10 legacy=0x7fbccc0b1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7fbcc9d9b700 1 -- 172.21.15.1:0/2583138841 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc071440 con 0x7fbccc06d020 2024-03-18T18:50:11.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7ff9796dd700 1 -- 172.21.15.1:0/2041434067 shutdown_connections 2024-03-18T18:50:11.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7ff9796dd700 1 --2- 172.21.15.1:0/2041434067 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff97414a480 0x7ff97406cb40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7ff9796dd700 1 -- 172.21.15.1:0/2041434067 >> 172.21.15.1:0/2041434067 conn(0x7ff97406ab70 msgr2=0x7ff97406af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:11.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7f467704a700 1 -- 172.21.15.1:0/1908055323 shutdown_connections 2024-03-18T18:50:11.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7f467704a700 1 --2- 172.21.15.1:0/1908055323 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f467015b6c0 0x7f46700b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7f467704a700 1 --2- 172.21.15.1:0/1908055323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46700cd430 0x7f46700c80c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.211+0000 7f467704a700 1 -- 172.21.15.1:0/1908055323 >> 172.21.15.1:0/1908055323 conn(0x7f46700d0ff0 msgr2=0x7f46700d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:11.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.212+0000 7f467704a700 1 -- 172.21.15.1:0/1908055323 shutdown_connections 2024-03-18T18:50:11.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.212+0000 7f467704a700 1 -- 172.21.15.1:0/1908055323 wait complete. 2024-03-18T18:50:11.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.210+0000 7f3f47a24700 1 -- 172.21.15.1:0/1921409567 wait complete. 2024-03-18T18:50:11.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.212+0000 7fbcc9d9b700 1 --2- 172.21.15.1:0/2583138841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbccc06d020 0x7fbccc0706d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbccc06d630 tx=0x7fbcc4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd34ae17266c4a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.212+0000 7f3f47a24700 1 Processor -- start 2024-03-18T18:50:11.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.212+0000 7ff5738b8700 1 -- 172.21.15.1:0/1762631102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c0cea60 msgr2=0x7ff56c159de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.212+0000 7ff5738b8700 1 --2- 172.21.15.1:0/1762631102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c0cea60 0x7ff56c159de0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff56400bd40 tx=0x7ff564009540 comp rx=0 tx=0).stop 2024-03-18T18:50:11.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.212+0000 7fbcc959a700 1 -- 172.21.15.1:0/2583138841 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4004230 con 0x7fbccc06d020 2024-03-18T18:50:11.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.212+0000 7fbcc959a700 1 -- 172.21.15.1:0/2583138841 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc4004390 con 0x7fbccc06d020 2024-03-18T18:50:11.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.212+0000 7fbcc959a700 1 -- 172.21.15.1:0/2583138841 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc40036a0 con 0x7fbccc06d020 2024-03-18T18:50:11.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.212+0000 7f3f47a24700 1 -- start start 2024-03-18T18:50:11.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.213+0000 7ff5738b8700 1 -- 172.21.15.1:0/1762631102 shutdown_connections 2024-03-18T18:50:11.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.213+0000 7ff5738b8700 1 --2- 172.21.15.1:0/1762631102 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff56c15a320 0x7ff56c070790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.213+0000 7ff5738b8700 1 --2- 172.21.15.1:0/1762631102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c0cea60 0x7ff56c159de0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.213+0000 7ff5738b8700 1 -- 172.21.15.1:0/1762631102 >> 172.21.15.1:0/1762631102 conn(0x7ff56c06a750 msgr2=0x7ff56c06ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:11.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.213+0000 7ff5738b8700 1 -- 172.21.15.1:0/1762631102 shutdown_connections 2024-03-18T18:50:11.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.213+0000 7f3f47a24700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f4008a2d0 0x7f3f4016b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.213+0000 7f467704a700 1 Processor -- start 2024-03-18T18:50:11.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.213+0000 7f3f47a24700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f40161100 0x7f3f4015a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.213+0000 7f3f47a24700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4016daf0 0x7f3f4006c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.213+0000 7f3f47a24700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4015b3a0 con 0x7f3f4016daf0 2024-03-18T18:50:11.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.213+0000 7f3f47a24700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f40157870 con 0x7f3f4008a2d0 2024-03-18T18:50:11.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.213+0000 7f3f47a24700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4015ade0 con 0x7f3f40161100 2024-03-18T18:50:11.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.213+0000 7fbcd0c86700 1 -- 172.21.15.1:0/2583138841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbccc06d020 msgr2=0x7fbccc0706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.213+0000 7fbcd0c86700 1 --2- 172.21.15.1:0/2583138841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbccc06d020 0x7fbccc0706d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbccc06d630 tx=0x7fbcc4009b20 comp rx=0 tx=0).stop 2024-03-18T18:50:11.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.214+0000 7fbcd0c86700 1 -- 172.21.15.1:0/2583138841 shutdown_connections 2024-03-18T18:50:11.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.214+0000 7fbcd0c86700 1 --2- 172.21.15.1:0/2583138841 >> v2:172.21.15.165:3300/0 conn(0x7fbccc14b1b0 0x7fbccc06cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.214+0000 7fbcd0c86700 1 --2- 172.21.15.1:0/2583138841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbccc06d020 0x7fbccc0706d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.214+0000 7fbcd0c86700 1 -- 172.21.15.1:0/2583138841 >> 172.21.15.1:0/2583138841 conn(0x7fbccc06a750 msgr2=0x7fbccc06ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:11.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.214+0000 7f3f45fc1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4016daf0 0x7f3f4006c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.214+0000 7f3f45fc1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4016daf0 0x7f3f4006c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45338/0 (socket says 172.21.15.1:45338) 2024-03-18T18:50:11.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.214+0000 7f3f45fc1700 1 -- 172.21.15.1:0/2450722055 learned_addr learned my addr 172.21.15.1:0/2450722055 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:11.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.214+0000 7fbcd0c86700 1 -- 172.21.15.1:0/2583138841 shutdown_connections 2024-03-18T18:50:11.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.214+0000 7f3f457c0700 1 --2- 172.21.15.1:0/2450722055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f4008a2d0 0x7f3f4016b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.214+0000 7f3f45fc1700 1 -- 172.21.15.1:0/2450722055 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f40161100 msgr2=0x7f3f4015a340 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T18:50:11.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.214+0000 7f3f45fc1700 1 --2- 172.21.15.1:0/2450722055 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f40161100 0x7f3f4015a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.214+0000 7f3f45fc1700 1 -- 172.21.15.1:0/2450722055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f4008a2d0 msgr2=0x7f3f4016b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.214+0000 7f3f45fc1700 1 --2- 172.21.15.1:0/2450722055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f4008a2d0 0x7f3f4016b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.214+0000 7f3f45fc1700 1 -- 172.21.15.1:0/2450722055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c0094a0 con 0x7f3f4016daf0 2024-03-18T18:50:11.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.213+0000 7ff5738b8700 1 -- 172.21.15.1:0/1762631102 wait complete. 2024-03-18T18:50:11.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.215+0000 7ff5738b8700 1 Processor -- start 2024-03-18T18:50:11.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.215+0000 7f3f45fc1700 1 --2- 172.21.15.1:0/2450722055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4016daf0 0x7f3f4006c720 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3f3800c9f0 tx=0x7f3f3800cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.215+0000 7f3f35ffb700 1 -- 172.21.15.1:0/2450722055 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3800a9e0 con 0x7f3f4016daf0 2024-03-18T18:50:11.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.215+0000 7f3f35ffb700 1 -- 172.21.15.1:0/2450722055 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3800ab40 con 0x7f3f4016daf0 2024-03-18T18:50:11.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.215+0000 7f3f35ffb700 1 -- 172.21.15.1:0/2450722055 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f38012400 con 0x7f3f4016daf0 2024-03-18T18:50:11.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.214+0000 7fbcd0c86700 1 -- 172.21.15.1:0/2583138841 wait complete. 2024-03-18T18:50:11.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.216+0000 7fbcd0c86700 1 Processor -- start 2024-03-18T18:50:11.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.215+0000 7ff5738b8700 1 -- start start 2024-03-18T18:50:11.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.216+0000 7ff5738b8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56c0cea60 0x7ff56c14f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.216+0000 7ff5738b8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c150400 0x7ff56c154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.216+0000 7ff5738b8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff56c15a320 0x7ff56c0ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.216+0000 7ff5738b8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c150ae0 con 0x7ff56c150400 2024-03-18T18:50:11.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.216+0000 7ff5738b8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c150dc0 con 0x7ff56c0cea60 2024-03-18T18:50:11.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.216+0000 7ff5738b8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c1510a0 con 0x7ff56c15a320 2024-03-18T18:50:11.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.216+0000 7ff571654700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56c0cea60 0x7ff56c14f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.216+0000 7ff570e53700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c150400 0x7ff56c154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.216+0000 7ff570e53700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c150400 0x7ff56c154c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45344/0 (socket says 172.21.15.1:45344) 2024-03-18T18:50:11.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.216+0000 7ff570e53700 1 -- 172.21.15.1:0/2633858282 learned_addr learned my addr 172.21.15.1:0/2633858282 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:11.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.217+0000 7ff571e55700 1 --2- 172.21.15.1:0/2633858282 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff56c15a320 0x7ff56c0ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.215+0000 7f3f47a24700 1 -- 172.21.15.1:0/2450722055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4015b120 con 0x7f3f4016daf0 2024-03-18T18:50:11.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.216+0000 7f3f47a24700 1 -- 172.21.15.1:0/2450722055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f40208a40 con 0x7f3f4016daf0 2024-03-18T18:50:11.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.217+0000 7f3f47a24700 1 -- 172.21.15.1:0/2450722055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f400cec50 con 0x7f3f4016daf0 2024-03-18T18:50:11.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.217+0000 7ff570e53700 1 -- 172.21.15.1:0/2633858282 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff56c15a320 msgr2=0x7ff56c0ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.217+0000 7ff570e53700 1 --2- 172.21.15.1:0/2633858282 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff56c15a320 0x7ff56c0ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.217+0000 7ff570e53700 1 -- 172.21.15.1:0/2633858282 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56c0cea60 msgr2=0x7ff56c14f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.217+0000 7ff570e53700 1 --2- 172.21.15.1:0/2633858282 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56c0cea60 0x7ff56c14f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.217+0000 7ff570e53700 1 -- 172.21.15.1:0/2633858282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56400b980 con 0x7ff56c150400 2024-03-18T18:50:11.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.216+0000 7fbcd0c86700 1 -- start start 2024-03-18T18:50:11.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.217+0000 7fbcd0c86700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbccc06d020 0x7fbccc0a9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.217+0000 7fbcd0c86700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbccc070c10 0x7fbccc164260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.217+0000 7fbcd0c86700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbccc14b1b0 0x7fbccc0ad830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.217+0000 7fbcd0c86700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc071a00 con 0x7fbccc14b1b0 2024-03-18T18:50:11.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.217+0000 7fbcd0c86700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc0aab80 con 0x7fbccc06d020 2024-03-18T18:50:11.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.217+0000 7fbcd0c86700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc0a9f70 con 0x7fbccc070c10 2024-03-18T18:50:11.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.217+0000 7fbcca59c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbccc06d020 0x7fbccc0a9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.217+0000 7fbcca59c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbccc06d020 0x7fbccc0a9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:58152/0 (socket says 172.21.15.1:58152) 2024-03-18T18:50:11.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.217+0000 7fbcca59c700 1 -- 172.21.15.1:0/3639292817 learned_addr learned my addr 172.21.15.1:0/3639292817 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:11.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.218+0000 7f467704a700 1 -- start start 2024-03-18T18:50:11.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.218+0000 7ff570e53700 1 --2- 172.21.15.1:0/2633858282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c150400 0x7ff56c154c80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff564009890 tx=0x7ff564004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.218+0000 7fbccad9d700 1 --2- 172.21.15.1:0/3639292817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbccc14b1b0 0x7fbccc0ad830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.218+0000 7fbcc9d9b700 1 --2- 172.21.15.1:0/3639292817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbccc070c10 0x7fbccc164260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.218+0000 7ff561ffb700 1 -- 172.21.15.1:0/2633858282 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff564004440 con 0x7ff56c150400 2024-03-18T18:50:11.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.218+0000 7fbccad9d700 1 -- 172.21.15.1:0/3639292817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbccc070c10 msgr2=0x7fbccc164260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.218+0000 7fbccad9d700 1 --2- 172.21.15.1:0/3639292817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbccc070c10 0x7fbccc164260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.218+0000 7fbccad9d700 1 -- 172.21.15.1:0/3639292817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbccc06d020 msgr2=0x7fbccc0a9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.218+0000 7fbccad9d700 1 --2- 172.21.15.1:0/3639292817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbccc06d020 0x7fbccc0a9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.218+0000 7fbccad9d700 1 -- 172.21.15.1:0/3639292817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc40094a0 con 0x7fbccc14b1b0 2024-03-18T18:50:11.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.218+0000 7ff5738b8700 1 -- 172.21.15.1:0/2633858282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56c151380 con 0x7ff56c150400 2024-03-18T18:50:11.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.218+0000 7fbccad9d700 1 --2- 172.21.15.1:0/3639292817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbccc14b1b0 0x7fbccc0ad830 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc00cc80 tx=0x7fbcbc00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.218+0000 7ff5738b8700 1 -- 172.21.15.1:0/2633858282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff56c14ff20 con 0x7ff56c150400 2024-03-18T18:50:11.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.219+0000 7f467704a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f46700c8600 0x7f46700cf060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.219+0000 7f467704a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46700cd430 0x7f467014f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.219+0000 7f467704a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f467015b6c0 0x7f467014feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.219+0000 7f467704a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46700c6900 con 0x7f467015b6c0 2024-03-18T18:50:11.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.219+0000 7f467704a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46701503f0 con 0x7f46700c8600 2024-03-18T18:50:11.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.219+0000 7f467704a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46700c5c80 con 0x7f46700cd430 2024-03-18T18:50:11.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.219+0000 7f466ffff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46700cd430 0x7f467014f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.219+0000 7f4674de6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f46700c8600 0x7f46700cf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.220+0000 7ff9796dd700 1 -- 172.21.15.1:0/2041434067 shutdown_connections 2024-03-18T18:50:11.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.220+0000 7ff5738b8700 1 -- 172.21.15.1:0/2633858282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5500052f0 con 0x7ff56c150400 2024-03-18T18:50:11.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.220+0000 7f46755e7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f467015b6c0 0x7f467014feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.220+0000 7f46755e7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f467015b6c0 0x7f467014feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45350/0 (socket says 172.21.15.1:45350) 2024-03-18T18:50:11.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.220+0000 7f46755e7700 1 -- 172.21.15.1:0/3843361470 learned_addr learned my addr 172.21.15.1:0/3843361470 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:11.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.220+0000 7f46755e7700 1 -- 172.21.15.1:0/3843361470 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46700cd430 msgr2=0x7f467014f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.220+0000 7f46755e7700 1 --2- 172.21.15.1:0/3843361470 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46700cd430 0x7f467014f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.220+0000 7f46755e7700 1 -- 172.21.15.1:0/3843361470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f46700c8600 msgr2=0x7f46700cf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.220+0000 7f46755e7700 1 --2- 172.21.15.1:0/3843361470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f46700c8600 0x7f46700cf060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.220+0000 7f46755e7700 1 -- 172.21.15.1:0/3843361470 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4664009550 con 0x7f467015b6c0 2024-03-18T18:50:11.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.220+0000 7f46755e7700 1 --2- 172.21.15.1:0/3843361470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f467015b6c0 0x7f467014feb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f466800b780 tx=0x7f466800ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.221+0000 7f3f35ffb700 1 -- 172.21.15.1:0/2450722055 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3f3801b3b0 con 0x7f3f4016daf0 2024-03-18T18:50:11.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.222+0000 7f3f35ffb700 1 --2- 172.21.15.1:0/2450722055 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3f2406ace0 0x7f3f2406d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.222+0000 7f3f35ffb700 1 -- 172.21.15.1:0/2450722055 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3f380895d0 con 0x7f3f4016daf0 2024-03-18T18:50:11.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.220+0000 7ff9796dd700 1 -- 172.21.15.1:0/2041434067 wait complete. 2024-03-18T18:50:11.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.222+0000 7ff9796dd700 1 Processor -- start 2024-03-18T18:50:11.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.222+0000 7ff9796dd700 1 -- start start 2024-03-18T18:50:11.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.223+0000 7ff9796dd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff97406d080 0x7ff9741f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.223+0000 7ff9796dd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9740b3c60 0x7ff9740c3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.223+0000 7ff9796dd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff97414a480 0x7ff974162e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.223+0000 7ff9796dd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9740c5320 con 0x7ff97414a480 2024-03-18T18:50:11.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.223+0000 7ff9796dd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9741f7870 con 0x7ff97406d080 2024-03-18T18:50:11.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.223+0000 7ff9796dd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9740c47a0 con 0x7ff9740b3c60 2024-03-18T18:50:11.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7ff9737fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff97414a480 0x7ff974162e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7ff9737fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff97414a480 0x7ff974162e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45358/0 (socket says 172.21.15.1:45358) 2024-03-18T18:50:11.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7ff9737fe700 1 -- 172.21.15.1:0/3535219354 learned_addr learned my addr 172.21.15.1:0/3535219354 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:11.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7ff9727fc700 1 --2- 172.21.15.1:0/3535219354 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9740b3c60 0x7ff9740c3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7ff972ffd700 1 --2- 172.21.15.1:0/3535219354 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff97406d080 0x7ff9741f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.223+0000 7f1f1cb55700 1 Processor -- start 2024-03-18T18:50:11.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.223+0000 7f1f1cb55700 1 -- start start 2024-03-18T18:50:11.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7f1f1cb55700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1f1806a750 0x7f1f180cd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7f1f1cb55700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f1806e2b0 0x7f1f1806e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7f1f1cb55700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f18161f60 con 0x7f1f180c3320 2024-03-18T18:50:11.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7f1f1cb55700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f180ce1f0 con 0x7f1f1806a750 2024-03-18T18:50:11.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7f1f1cb55700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f180cf610 con 0x7f1f1806e2b0 2024-03-18T18:50:11.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.223+0000 7f3f457c0700 1 --2- 172.21.15.1:0/2450722055 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3f2406ace0 0x7f3f2406d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7f3f457c0700 1 --2- 172.21.15.1:0/2450722055 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3f2406ace0 0x7f3f2406d1a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c000c00 tx=0x7f3f3c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7f1f15d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1f180c3320 0x7f1f1806ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60312/0 (socket says 172.21.15.1:60312) 2024-03-18T18:50:11.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7f1f15d9b700 1 -- 172.21.15.1:0/4148921047 learned_addr learned my addr 172.21.15.1:0/4148921047 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:11.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7ff972ffd700 1 -- 172.21.15.1:0/3535219354 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9740b3c60 msgr2=0x7ff9740c3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7ff972ffd700 1 --2- 172.21.15.1:0/3535219354 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9740b3c60 0x7ff9740c3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7ff972ffd700 1 -- 172.21.15.1:0/3535219354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff97414a480 msgr2=0x7ff974162e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7ff972ffd700 1 --2- 172.21.15.1:0/3535219354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff97414a480 0x7ff974162e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7ff972ffd700 1 -- 172.21.15.1:0/3535219354 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9640094a0 con 0x7ff97406d080 2024-03-18T18:50:11.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7f1f1659c700 1 --2- 172.21.15.1:0/4148921047 >> v2:172.21.15.165:3300/0 conn(0x7f1f1806a750 0x7f1f180cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7f1f16d9d700 1 --2- 172.21.15.1:0/4148921047 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f1806e2b0 0x7f1f1806e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.224+0000 7ff972ffd700 1 --2- 172.21.15.1:0/3535219354 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff97406d080 0x7ff9741f7330 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff964015000 tx=0x7ff964004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.225+0000 7ff95b7fe700 1 -- 172.21.15.1:0/3535219354 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff964003de0 con 0x7ff97406d080 2024-03-18T18:50:11.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.225+0000 7f1f1659c700 1 -- 172.21.15.1:0/4148921047 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f1806e2b0 msgr2=0x7f1f1806e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:11.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.225+0000 7f1f1659c700 1 --2- 172.21.15.1:0/4148921047 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f1806e2b0 0x7f1f1806e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.225+0000 7f1f1659c700 1 -- 172.21.15.1:0/4148921047 >> v1:172.21.15.1:6789/0 conn(0x7f1f180c3320 legacy=0x7f1f1806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.225+0000 7ff95b7fe700 1 -- 172.21.15.1:0/3535219354 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff964011630 con 0x7ff97406d080 2024-03-18T18:50:11.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.225+0000 7ff95b7fe700 1 -- 172.21.15.1:0/3535219354 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9640119a0 con 0x7ff97406d080 2024-03-18T18:50:11.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.225+0000 7f1f1659c700 1 -- 172.21.15.1:0/4148921047 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f180cdf10 con 0x7f1f1806a750 2024-03-18T18:50:11.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.225+0000 7f1f1659c700 1 --2- 172.21.15.1:0/4148921047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f1806a750 0x7f1f180cd520 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c009840 tx=0x7f1f0c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=588eb8eaa1ecd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.225+0000 7f1f1559a700 1 -- 172.21.15.1:0/4148921047 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c011830 con 0x7f1f1806a750 2024-03-18T18:50:11.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.225+0000 7f1f1559a700 1 -- 172.21.15.1:0/4148921047 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0c011990 con 0x7f1f1806a750 2024-03-18T18:50:11.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.225+0000 7f1f1559a700 1 -- 172.21.15.1:0/4148921047 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c011d20 con 0x7f1f1806a750 2024-03-18T18:50:11.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.226+0000 7f466d7fa700 1 -- 172.21.15.1:0/3843361470 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4668011820 con 0x7f467015b6c0 2024-03-18T18:50:11.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.226+0000 7f3f35ffb700 1 -- 172.21.15.1:0/2450722055 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3f38054670 con 0x7f3f4016daf0 2024-03-18T18:50:11.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.225+0000 7ff9796dd700 1 -- 172.21.15.1:0/3535219354 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9740c4a80 con 0x7ff97406d080 2024-03-18T18:50:11.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.225+0000 7ff9796dd700 1 -- 172.21.15.1:0/3535219354 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9740c5040 con 0x7ff97406d080 2024-03-18T18:50:11.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.226+0000 7ff9796dd700 1 -- 172.21.15.1:0/3535219354 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff974073350 con 0x7ff97406d080 2024-03-18T18:50:11.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.227+0000 7fbcbaffd700 1 -- 172.21.15.1:0/3639292817 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcbc00ada0 con 0x7fbccc14b1b0 2024-03-18T18:50:11.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.227+0000 7fbcbaffd700 1 -- 172.21.15.1:0/3639292817 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcbc00ce80 con 0x7fbccc14b1b0 2024-03-18T18:50:11.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.227+0000 7fbcbaffd700 1 -- 172.21.15.1:0/3639292817 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcbc0072d0 con 0x7fbccc14b1b0 2024-03-18T18:50:11.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.226+0000 7f467704a700 1 -- 172.21.15.1:0/3843361470 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46700c5f60 con 0x7f467015b6c0 2024-03-18T18:50:11.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.226+0000 7f467704a700 1 -- 172.21.15.1:0/3843361470 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46700c6520 con 0x7f467015b6c0 2024-03-18T18:50:11.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.227+0000 7f466d7fa700 1 -- 172.21.15.1:0/3843361470 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466800d040 con 0x7f467015b6c0 2024-03-18T18:50:11.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.227+0000 7f466d7fa700 1 -- 172.21.15.1:0/3843361470 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4668011b80 con 0x7f467015b6c0 2024-03-18T18:50:11.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.226+0000 7f1f1cb55700 1 -- 172.21.15.1:0/4148921047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f1806a750 msgr2=0x7f1f180cd520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.226+0000 7f1f1cb55700 1 --2- 172.21.15.1:0/4148921047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f1806a750 0x7f1f180cd520 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c009840 tx=0x7f1f0c009b50 comp rx=0 tx=0).stop 2024-03-18T18:50:11.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.227+0000 7f1f1cb55700 1 -- 172.21.15.1:0/4148921047 shutdown_connections 2024-03-18T18:50:11.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.227+0000 7f1f1cb55700 1 --2- 172.21.15.1:0/4148921047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f1806a750 0x7f1f180cd520 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.227+0000 7f1f1cb55700 1 --2- 172.21.15.1:0/4148921047 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f1806e2b0 0x7f1f1806e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.227+0000 7f1f1cb55700 1 -- 172.21.15.1:0/4148921047 >> 172.21.15.1:0/4148921047 conn(0x7f1f180cca00 msgr2=0x7f1f180cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:11.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.229+0000 7ff561ffb700 1 -- 172.21.15.1:0/2633858282 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5640045a0 con 0x7ff56c150400 2024-03-18T18:50:11.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.229+0000 7ff561ffb700 1 -- 172.21.15.1:0/2633858282 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff564004910 con 0x7ff56c150400 2024-03-18T18:50:11.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.229+0000 7ff561ffb700 1 -- 172.21.15.1:0/2633858282 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff564015420 con 0x7ff56c150400 2024-03-18T18:50:11.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.227+0000 7fbcd0c86700 1 -- 172.21.15.1:0/3639292817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbccc0aa250 con 0x7fbccc14b1b0 2024-03-18T18:50:11.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.227+0000 7fbcd0c86700 1 -- 172.21.15.1:0/3639292817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbccc0aa810 con 0x7fbccc14b1b0 2024-03-18T18:50:11.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.229+0000 7fbcd0c86700 1 -- 172.21.15.1:0/3639292817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb00052f0 con 0x7fbccc14b1b0 2024-03-18T18:50:11.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.230+0000 7ff561ffb700 1 --2- 172.21.15.1:0/2633858282 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff55806b1c0 0x7ff55806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.230+0000 7ff561ffb700 1 -- 172.21.15.1:0/2633858282 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff56408be00 con 0x7ff56c150400 2024-03-18T18:50:11.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.228+0000 7ff95b7fe700 1 -- 172.21.15.1:0/3535219354 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff964022020 con 0x7ff97406d080 2024-03-18T18:50:11.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.229+0000 7ff95b7fe700 1 --2- 172.21.15.1:0/3535219354 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff96006ac90 0x7ff96006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.229+0000 7ff95b7fe700 1 -- 172.21.15.1:0/3535219354 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff96408b100 con 0x7ff97406d080 2024-03-18T18:50:11.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.231+0000 7ff571654700 1 --2- 172.21.15.1:0/2633858282 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff55806b1c0 0x7ff55806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.231+0000 7f467704a700 1 -- 172.21.15.1:0/3843361470 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46500052f0 con 0x7f467015b6c0 2024-03-18T18:50:11.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.231+0000 7ff9727fc700 1 --2- 172.21.15.1:0/3535219354 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff96006ac90 0x7ff96006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.232+0000 7ff571654700 1 --2- 172.21.15.1:0/2633858282 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff55806b1c0 0x7ff55806d680 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff55c004a60 tx=0x7ff55c0060a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.232+0000 7ff9727fc700 1 --2- 172.21.15.1:0/3535219354 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff96006ac90 0x7ff96006d150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff96c002230 tx=0x7ff96c001e30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.232+0000 7f1f1cb55700 1 -- 172.21.15.1:0/4148921047 shutdown_connections 2024-03-18T18:50:11.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.232+0000 7ff561ffb700 1 -- 172.21.15.1:0/2633858282 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff564056d60 con 0x7ff56c150400 2024-03-18T18:50:11.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.233+0000 7ff95b7fe700 1 -- 172.21.15.1:0/3535219354 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff964056060 con 0x7ff97406d080 2024-03-18T18:50:11.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.235+0000 7f466d7fa700 1 -- 172.21.15.1:0/3843361470 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4668011ce0 con 0x7f467015b6c0 2024-03-18T18:50:11.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.236+0000 7f466d7fa700 1 --2- 172.21.15.1:0/3843361470 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f465c06ac90 0x7f465c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.236+0000 7f466d7fa700 1 -- 172.21.15.1:0/3843361470 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4668089fa0 con 0x7f467015b6c0 2024-03-18T18:50:11.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.238+0000 7f4674de6700 1 --2- 172.21.15.1:0/3843361470 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f465c06ac90 0x7f465c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.239+0000 7f4674de6700 1 --2- 172.21.15.1:0/3843361470 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f465c06ac90 0x7f465c06d150 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4660009b40 tx=0x7f4660009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.238+0000 7f1f1cb55700 1 -- 172.21.15.1:0/4148921047 wait complete. 2024-03-18T18:50:11.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.240+0000 7f466d7fa700 1 -- 172.21.15.1:0/3843361470 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4668054e70 con 0x7f467015b6c0 2024-03-18T18:50:11.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.241+0000 7fbcbaffd700 1 -- 172.21.15.1:0/3639292817 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fbcbc007430 con 0x7fbccc14b1b0 2024-03-18T18:50:11.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.242+0000 7fbcbaffd700 1 --2- 172.21.15.1:0/3639292817 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbcac06ad60 0x7fbcac06d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.242+0000 7fbcbaffd700 1 -- 172.21.15.1:0/3639292817 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbcbc089d40 con 0x7fbccc14b1b0 2024-03-18T18:50:11.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.240+0000 7f1f1cb55700 1 Processor -- start 2024-03-18T18:50:11.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.245+0000 7f1f1cb55700 1 -- start start 2024-03-18T18:50:11.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.246+0000 7f1f1cb55700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f1806a750 0x7f1f18160b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.246+0000 7f1f1cb55700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f1806e2b0 0x7f1f1809d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.246+0000 7f1f1cb55700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f180c3320 0x7f1f180b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.246+0000 7f1f1cb55700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f180b49e0 con 0x7f1f180c3320 2024-03-18T18:50:11.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.246+0000 7f1f1cb55700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f180b3dd0 con 0x7f1f1806a750 2024-03-18T18:50:11.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.246+0000 7f1f1cb55700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f180b40b0 con 0x7f1f1806e2b0 2024-03-18T18:50:11.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.246+0000 7f1f1659c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f1806a750 0x7f1f18160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.246+0000 7f1f15d9b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f1806e2b0 0x7f1f1809d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.247+0000 7f1f1659c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f1806a750 0x7f1f18160b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:58188/0 (socket says 172.21.15.1:58188) 2024-03-18T18:50:11.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.247+0000 7f1f1659c700 1 -- 172.21.15.1:0/410067032 learned_addr learned my addr 172.21.15.1:0/410067032 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:11.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.247+0000 7f1f1659c700 1 -- 172.21.15.1:0/410067032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f1806e2b0 msgr2=0x7f1f1809d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.247+0000 7f1f1659c700 1 --2- 172.21.15.1:0/410067032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f1806e2b0 0x7f1f1809d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.247+0000 7f1f1659c700 1 -- 172.21.15.1:0/410067032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f180c3320 msgr2=0x7f1f180b4e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T18:50:11.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.247+0000 7f1f1659c700 1 --2- 172.21.15.1:0/410067032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f180c3320 0x7f1f180b4e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.247+0000 7f1f1659c700 1 -- 172.21.15.1:0/410067032 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f0c0094a0 con 0x7f1f1806a750 2024-03-18T18:50:11.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.247+0000 7fbcca59c700 1 --2- 172.21.15.1:0/3639292817 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbcac06ad60 0x7fbcac06d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.248+0000 7fbcca59c700 1 --2- 172.21.15.1:0/3639292817 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbcac06ad60 0x7fbcac06d220 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000a9b0 tx=0x7fbcc0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.249+0000 7fbcbaffd700 1 -- 172.21.15.1:0/3639292817 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbcbc054d50 con 0x7fbccc14b1b0 2024-03-18T18:50:11.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.249+0000 7f1f1659c700 1 --2- 172.21.15.1:0/410067032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f1806a750 0x7f1f18160b00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0090b0 tx=0x7f1f0c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.250+0000 7f1f06ffd700 1 -- 172.21.15.1:0/410067032 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c004590 con 0x7f1f1806a750 2024-03-18T18:50:11.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.250+0000 7f1f06ffd700 1 -- 172.21.15.1:0/410067032 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0c01f070 con 0x7f1f1806a750 2024-03-18T18:50:11.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.250+0000 7f1f06ffd700 1 -- 172.21.15.1:0/410067032 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c0038e0 con 0x7f1f1806a750 2024-03-18T18:50:11.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.250+0000 7f1f1cb55700 1 -- 172.21.15.1:0/410067032 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f180b4390 con 0x7f1f1806a750 2024-03-18T18:50:11.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.250+0000 7f1f1cb55700 1 -- 172.21.15.1:0/410067032 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1809d660 con 0x7f1f1806a750 2024-03-18T18:50:11.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.251+0000 7f1f1cb55700 1 -- 172.21.15.1:0/410067032 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f180cdb50 con 0x7f1f1806a750 2024-03-18T18:50:11.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.253+0000 7f1f06ffd700 1 -- 172.21.15.1:0/410067032 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1f0c003a40 con 0x7f1f1806a750 2024-03-18T18:50:11.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.254+0000 7f1f06ffd700 1 --2- 172.21.15.1:0/410067032 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1ef806ad60 0x7f1ef806d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.254+0000 7f1f06ffd700 1 -- 172.21.15.1:0/410067032 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1f0c021070 con 0x7f1f1806a750 2024-03-18T18:50:11.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.256+0000 7f1f15d9b700 1 --2- 172.21.15.1:0/410067032 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1ef806ad60 0x7f1ef806d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.256+0000 7f1f15d9b700 1 --2- 172.21.15.1:0/410067032 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1ef806ad60 0x7f1ef806d220 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1f08000f60 tx=0x7f1f08007f10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.257+0000 7f1f06ffd700 1 -- 172.21.15.1:0/410067032 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1f0c056060 con 0x7f1f1806a750 2024-03-18T18:50:11.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.296+0000 7faf59a24700 1 Processor -- start 2024-03-18T18:50:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.296+0000 7faf59a24700 1 -- start start 2024-03-18T18:50:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.298+0000 7faf59a24700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7faf54150400 0x7faf540ce350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.298+0000 7faf59a24700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf540ce890 0x7faf54159de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.298+0000 7faf59a24700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faf5415a320 0x7faf5406c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.298+0000 7faf59a24700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf540cbad0 con 0x7faf5415a320 2024-03-18T18:50:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.298+0000 7faf59a24700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf540ced10 con 0x7faf54150400 2024-03-18T18:50:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.298+0000 7faf59a24700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf540cfee0 con 0x7faf540ce890 2024-03-18T18:50:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.298+0000 7faf537fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faf5415a320 0x7faf5406c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.298+0000 7faf537fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faf5415a320 0x7faf5406c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45370/0 (socket says 172.21.15.1:45370) 2024-03-18T18:50:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.298+0000 7faf537fe700 1 -- 172.21.15.1:0/497605830 learned_addr learned my addr 172.21.15.1:0/497605830 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:11.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.299+0000 7faf537fe700 1 -- 172.21.15.1:0/497605830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf540ce890 msgr2=0x7faf54159de0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T18:50:11.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.299+0000 7faf537fe700 1 --2- 172.21.15.1:0/497605830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf540ce890 0x7faf54159de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.299+0000 7faf537fe700 1 -- 172.21.15.1:0/497605830 >> v2:172.21.15.165:3300/0 conn(0x7faf54150400 msgr2=0x7faf540ce350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T18:50:11.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.299+0000 7faf537fe700 1 --2- 172.21.15.1:0/497605830 >> v2:172.21.15.165:3300/0 conn(0x7faf54150400 0x7faf540ce350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.299+0000 7faf537fe700 1 -- 172.21.15.1:0/497605830 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf5415d0c0 con 0x7faf5415a320 2024-03-18T18:50:11.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.299+0000 7faf537fe700 1 --2- 172.21.15.1:0/497605830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5415a320 0x7faf5406c720 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faf5406d890 tx=0x7faf44008bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=573dd51a5bed893f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.300+0000 7faf51ffb700 1 -- 172.21.15.1:0/497605830 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4400b820 con 0x7faf5415a320 2024-03-18T18:50:11.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.300+0000 7faf51ffb700 1 -- 172.21.15.1:0/497605830 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf4400b980 con 0x7faf5415a320 2024-03-18T18:50:11.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.300+0000 7faf51ffb700 1 -- 172.21.15.1:0/497605830 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf440126f0 con 0x7faf5415a320 2024-03-18T18:50:11.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.301+0000 7faf59a24700 1 -- 172.21.15.1:0/497605830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5415a320 msgr2=0x7faf5406c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.301+0000 7faf59a24700 1 --2- 172.21.15.1:0/497605830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5415a320 0x7faf5406c720 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faf5406d890 tx=0x7faf44008bd0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.302+0000 7faf59a24700 1 -- 172.21.15.1:0/497605830 shutdown_connections 2024-03-18T18:50:11.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.302+0000 7faf59a24700 1 --2- 172.21.15.1:0/497605830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5415a320 0x7faf5406c720 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.302+0000 7faf59a24700 1 --2- 172.21.15.1:0/497605830 >> v2:172.21.15.165:3300/0 conn(0x7faf54150400 0x7faf540ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.302+0000 7faf59a24700 1 --2- 172.21.15.1:0/497605830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf540ce890 0x7faf54159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.302+0000 7faf59a24700 1 -- 172.21.15.1:0/497605830 >> 172.21.15.1:0/497605830 conn(0x7faf5406a750 msgr2=0x7faf5406ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:11.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.302+0000 7faf59a24700 1 -- 172.21.15.1:0/497605830 shutdown_connections 2024-03-18T18:50:11.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.302+0000 7faf59a24700 1 -- 172.21.15.1:0/497605830 wait complete. 2024-03-18T18:50:11.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.304+0000 7faf59a24700 1 Processor -- start 2024-03-18T18:50:11.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.305+0000 7faf59a24700 1 -- start start 2024-03-18T18:50:11.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.306+0000 7faf59a24700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf540ce890 0x7faf541f57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.306+0000 7faf59a24700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf54150400 0x7faf54154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.306+0000 7faf59a24700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5415a320 0x7faf541545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.306+0000 7faf59a24700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5414fdd0 con 0x7faf5415a320 2024-03-18T18:50:11.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.306+0000 7faf59a24700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf541f5d20 con 0x7faf540ce890 2024-03-18T18:50:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.306+0000 7faf59a24700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf541530e0 con 0x7faf54150400 2024-03-18T18:50:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.306+0000 7faf537fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5415a320 0x7faf541545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.306+0000 7faf537fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5415a320 0x7faf541545a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45374/0 (socket says 172.21.15.1:45374) 2024-03-18T18:50:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.306+0000 7faf537fe700 1 -- 172.21.15.1:0/2034147432 learned_addr learned my addr 172.21.15.1:0/2034147432 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.307+0000 7faf537fe700 1 -- 172.21.15.1:0/2034147432 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf54150400 msgr2=0x7faf54154060 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T18:50:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.307+0000 7faf537fe700 1 --2- 172.21.15.1:0/2034147432 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf54150400 0x7faf54154060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.307+0000 7faf537fe700 1 -- 172.21.15.1:0/2034147432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf540ce890 msgr2=0x7faf541f57e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T18:50:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.307+0000 7faf537fe700 1 --2- 172.21.15.1:0/2034147432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf540ce890 0x7faf541f57e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.307+0000 7faf537fe700 1 -- 172.21.15.1:0/2034147432 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf440085d0 con 0x7faf5415a320 2024-03-18T18:50:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.308+0000 7faf537fe700 1 --2- 172.21.15.1:0/2034147432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5415a320 0x7faf541545a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7faf4400bbd0 tx=0x7faf44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.310+0000 7faf3bfff700 1 -- 172.21.15.1:0/2034147432 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf44004030 con 0x7faf5415a320 2024-03-18T18:50:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.310+0000 7faf3bfff700 1 -- 172.21.15.1:0/2034147432 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf44004190 con 0x7faf5415a320 2024-03-18T18:50:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.310+0000 7faf3bfff700 1 -- 172.21.15.1:0/2034147432 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf44004520 con 0x7faf5415a320 2024-03-18T18:50:11.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.310+0000 7faf59a24700 1 -- 172.21.15.1:0/2034147432 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf541533c0 con 0x7faf5415a320 2024-03-18T18:50:11.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.310+0000 7faf59a24700 1 -- 172.21.15.1:0/2034147432 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf540af680 con 0x7faf5415a320 2024-03-18T18:50:11.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.312+0000 7faf59a24700 1 -- 172.21.15.1:0/2034147432 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf340052f0 con 0x7faf5415a320 2024-03-18T18:50:11.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.323+0000 7faf3bfff700 1 -- 172.21.15.1:0/2034147432 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7faf440036a0 con 0x7faf5415a320 2024-03-18T18:50:11.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.324+0000 7faf3bfff700 1 --2- 172.21.15.1:0/2034147432 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf4006ad60 0x7faf4006d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:11.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.324+0000 7faf3bfff700 1 -- 172.21.15.1:0/2034147432 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7faf4408abd0 con 0x7faf5415a320 2024-03-18T18:50:11.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.324+0000 7faf3bfff700 1 -- 172.21.15.1:0/2034147432 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faf4408e2a0 con 0x7faf5415a320 2024-03-18T18:50:11.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.334+0000 7faf52ffd700 1 --2- 172.21.15.1:0/2034147432 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf4006ad60 0x7faf4006d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:11.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.335+0000 7faf52ffd700 1 --2- 172.21.15.1:0/2034147432 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf4006ad60 0x7faf4006d220 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7faf48009710 tx=0x7faf48006c60 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:11.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.591+0000 7f68ae49a700 1 -- 172.21.15.1:0/3029651617 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f68880055a0 con 0x7f68a814b1b0 2024-03-18T18:50:11.591 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-18T18:50:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.592+0000 7f68a4ff9700 1 -- 172.21.15.1:0/3029651617 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6898059670 con 0x7f68a814b1b0 2024-03-18T18:50:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.595+0000 7f68ae49a700 1 -- 172.21.15.1:0/3029651617 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f689406ad60 msgr2=0x7f689406d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.595+0000 7f68ae49a700 1 --2- 172.21.15.1:0/3029651617 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f689406ad60 0x7f689406d220 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f689c006fd0 tx=0x7f689c006bc0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.595+0000 7f68ae49a700 1 -- 172.21.15.1:0/3029651617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a814b1b0 msgr2=0x7f68a8164210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.595+0000 7f68ae49a700 1 --2- 172.21.15.1:0/3029651617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a814b1b0 0x7f68a8164210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f68a806d660 tx=0x7f6898004750 comp rx=0 tx=0).stop 2024-03-18T18:50:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.596+0000 7f68ae49a700 1 -- 172.21.15.1:0/3029651617 shutdown_connections 2024-03-18T18:50:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.596+0000 7f68ae49a700 1 --2- 172.21.15.1:0/3029651617 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f689406ad60 0x7f689406d220 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.596+0000 7f68ae49a700 1 --2- 172.21.15.1:0/3029651617 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f68a806d1f0 0x7f68a80ad0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.596+0000 7f68ae49a700 1 --2- 172.21.15.1:0/3029651617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68a814b1b0 0x7f68a8164210 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.596+0000 7f68ae49a700 1 --2- 172.21.15.1:0/3029651617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68a8070c10 0x7f68a8163cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.596+0000 7f68ae49a700 1 -- 172.21.15.1:0/3029651617 >> 172.21.15.1:0/3029651617 conn(0x7f68a806a750 msgr2=0x7f68a80cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:11.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.596+0000 7f68ae49a700 1 -- 172.21.15.1:0/3029651617 shutdown_connections 2024-03-18T18:50:11.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.597+0000 7f68ae49a700 1 -- 172.21.15.1:0/3029651617 wait complete. 2024-03-18T18:50:11.611 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.5 2024-03-18T18:50:11.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.635+0000 7f3f47a24700 1 -- 172.21.15.1:0/2450722055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f3f40073350 con 0x7f3f4016daf0 2024-03-18T18:50:11.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.635+0000 7f3f35ffb700 1 -- 172.21.15.1:0/2450722055 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3f38057c90 con 0x7f3f4016daf0 2024-03-18T18:50:11.632 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-18T18:50:11.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.638+0000 7f3f47a24700 1 -- 172.21.15.1:0/2450722055 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3f2406ace0 msgr2=0x7f3f2406d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.638+0000 7f3f47a24700 1 --2- 172.21.15.1:0/2450722055 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3f2406ace0 0x7f3f2406d1a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c000c00 tx=0x7f3f3c008030 comp rx=0 tx=0).stop 2024-03-18T18:50:11.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.638+0000 7f3f47a24700 1 -- 172.21.15.1:0/2450722055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4016daf0 msgr2=0x7f3f4006c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.638+0000 7f3f47a24700 1 --2- 172.21.15.1:0/2450722055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4016daf0 0x7f3f4006c720 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3f3800c9f0 tx=0x7f3f3800cd00 comp rx=0 tx=0).stop 2024-03-18T18:50:11.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.641+0000 7f3f47a24700 1 -- 172.21.15.1:0/2450722055 shutdown_connections 2024-03-18T18:50:11.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.641+0000 7f3f47a24700 1 --2- 172.21.15.1:0/2450722055 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3f2406ace0 0x7f3f2406d1a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.641+0000 7f3f47a24700 1 --2- 172.21.15.1:0/2450722055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4016daf0 0x7f3f4006c720 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.641+0000 7f3f47a24700 1 --2- 172.21.15.1:0/2450722055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f4008a2d0 0x7f3f4016b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.641+0000 7f3f47a24700 1 --2- 172.21.15.1:0/2450722055 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f40161100 0x7f3f4015a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.641+0000 7f3f47a24700 1 -- 172.21.15.1:0/2450722055 >> 172.21.15.1:0/2450722055 conn(0x7f3f4006a750 msgr2=0x7f3f4016c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:11.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.641+0000 7f3f47a24700 1 -- 172.21.15.1:0/2450722055 shutdown_connections 2024-03-18T18:50:11.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.641+0000 7f3f47a24700 1 -- 172.21.15.1:0/2450722055 wait complete. 2024-03-18T18:50:11.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.644+0000 7ff5738b8700 1 -- 172.21.15.1:0/2633858282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff550005870 con 0x7ff56c150400 2024-03-18T18:50:11.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.645+0000 7ff561ffb700 1 -- 172.21.15.1:0/2633858282 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff56405a380 con 0x7ff56c150400 2024-03-18T18:50:11.641 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-18T18:50:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.649+0000 7ff5738b8700 1 -- 172.21.15.1:0/2633858282 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff55806b1c0 msgr2=0x7ff55806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.649+0000 7ff5738b8700 1 --2- 172.21.15.1:0/2633858282 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff55806b1c0 0x7ff55806d680 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff55c004a60 tx=0x7ff55c0060a0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.649+0000 7ff5738b8700 1 -- 172.21.15.1:0/2633858282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c150400 msgr2=0x7ff56c154c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.649+0000 7ff5738b8700 1 --2- 172.21.15.1:0/2633858282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c150400 0x7ff56c154c80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff564009890 tx=0x7ff564004ad0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.649+0000 7ff5738b8700 1 -- 172.21.15.1:0/2633858282 shutdown_connections 2024-03-18T18:50:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.649+0000 7ff5738b8700 1 --2- 172.21.15.1:0/2633858282 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff55806b1c0 0x7ff55806d680 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff55c004a60 tx=0x7ff55c0060a0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.649+0000 7ff5738b8700 1 --2- 172.21.15.1:0/2633858282 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff56c15a320 0x7ff56c0ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.649+0000 7ff5738b8700 1 --2- 172.21.15.1:0/2633858282 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56c0cea60 0x7ff56c14f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.649+0000 7ff5738b8700 1 --2- 172.21.15.1:0/2633858282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff56c150400 0x7ff56c154c80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.649+0000 7ff5738b8700 1 -- 172.21.15.1:0/2633858282 >> 172.21.15.1:0/2633858282 conn(0x7ff56c06a750 msgr2=0x7ff56c06ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:11.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.649+0000 7ff5738b8700 1 -- 172.21.15.1:0/2633858282 shutdown_connections 2024-03-18T18:50:11.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.649+0000 7ff5738b8700 1 -- 172.21.15.1:0/2633858282 wait complete. 2024-03-18T18:50:11.654 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.3 2024-03-18T18:50:11.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.661+0000 7f467704a700 1 -- 172.21.15.1:0/3843361470 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4650005870 con 0x7f467015b6c0 2024-03-18T18:50:11.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.662+0000 7ff9796dd700 1 -- 172.21.15.1:0/3535219354 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7ff974146d70 con 0x7ff97406d080 2024-03-18T18:50:11.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.663+0000 7f466d7fa700 1 -- 172.21.15.1:0/3843361470 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4668015090 con 0x7f467015b6c0 2024-03-18T18:50:11.659 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-18T18:50:11.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.666+0000 7ff95b7fe700 1 -- 172.21.15.1:0/3535219354 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff964059680 con 0x7ff97406d080 2024-03-18T18:50:11.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.666+0000 7f467704a700 1 -- 172.21.15.1:0/3843361470 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f465c06ac90 msgr2=0x7f465c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.666+0000 7f467704a700 1 --2- 172.21.15.1:0/3843361470 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f465c06ac90 0x7f465c06d150 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4660009b40 tx=0x7f4660009210 comp rx=0 tx=0).stop 2024-03-18T18:50:11.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.666+0000 7f467704a700 1 -- 172.21.15.1:0/3843361470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f467015b6c0 msgr2=0x7f467014feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.666+0000 7f467704a700 1 --2- 172.21.15.1:0/3843361470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f467015b6c0 0x7f467014feb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f466800b780 tx=0x7f466800ba90 comp rx=0 tx=0).stop 2024-03-18T18:50:11.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.666+0000 7f467704a700 1 -- 172.21.15.1:0/3843361470 shutdown_connections 2024-03-18T18:50:11.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.666+0000 7f467704a700 1 --2- 172.21.15.1:0/3843361470 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f465c06ac90 0x7f465c06d150 secure :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4660009b40 tx=0x7f4660009210 comp rx=0 tx=0).stop 2024-03-18T18:50:11.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.666+0000 7f467704a700 1 --2- 172.21.15.1:0/3843361470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f467015b6c0 0x7f467014feb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.666+0000 7f467704a700 1 --2- 172.21.15.1:0/3843361470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f46700c8600 0x7f46700cf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.666+0000 7f467704a700 1 --2- 172.21.15.1:0/3843361470 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46700cd430 0x7f467014f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.666+0000 7f467704a700 1 -- 172.21.15.1:0/3843361470 >> 172.21.15.1:0/3843361470 conn(0x7f46700d0ff0 msgr2=0x7f46700ca200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:11.664 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-18T18:50:11.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.669+0000 7f467704a700 1 -- 172.21.15.1:0/3843361470 shutdown_connections 2024-03-18T18:50:11.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.669+0000 7f467704a700 1 -- 172.21.15.1:0/3843361470 wait complete. 2024-03-18T18:50:11.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.670+0000 7ff9796dd700 1 -- 172.21.15.1:0/3535219354 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff96006ac90 msgr2=0x7ff96006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.670+0000 7ff9796dd700 1 --2- 172.21.15.1:0/3535219354 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff96006ac90 0x7ff96006d150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff96c002230 tx=0x7ff96c001e30 comp rx=0 tx=0).stop 2024-03-18T18:50:11.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.670+0000 7ff9796dd700 1 -- 172.21.15.1:0/3535219354 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff97406d080 msgr2=0x7ff9741f7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.670+0000 7ff9796dd700 1 --2- 172.21.15.1:0/3535219354 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff97406d080 0x7ff9741f7330 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff964015000 tx=0x7ff964004750 comp rx=0 tx=0).stop 2024-03-18T18:50:11.668 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.0 2024-03-18T18:50:11.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.670+0000 7ff9796dd700 1 -- 172.21.15.1:0/3535219354 shutdown_connections 2024-03-18T18:50:11.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.670+0000 7ff9796dd700 1 --2- 172.21.15.1:0/3535219354 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff96006ac90 0x7ff96006d150 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.670+0000 7ff9796dd700 1 --2- 172.21.15.1:0/3535219354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff97414a480 0x7ff974162e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.670+0000 7ff9796dd700 1 --2- 172.21.15.1:0/3535219354 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff97406d080 0x7ff9741f7330 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff964015000 tx=0x7ff964004750 comp rx=0 tx=0).stop 2024-03-18T18:50:11.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.670+0000 7ff9796dd700 1 --2- 172.21.15.1:0/3535219354 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9740b3c60 0x7ff9740c3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.670+0000 7ff9796dd700 1 -- 172.21.15.1:0/3535219354 >> 172.21.15.1:0/3535219354 conn(0x7ff97406ab70 msgr2=0x7ff9740cd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:11.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.670+0000 7ff9796dd700 1 -- 172.21.15.1:0/3535219354 shutdown_connections 2024-03-18T18:50:11.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.671+0000 7ff9796dd700 1 -- 172.21.15.1:0/3535219354 wait complete. 2024-03-18T18:50:11.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.682+0000 7f1f1cb55700 1 -- 172.21.15.1:0/410067032 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f1f18072520 con 0x7f1f1806a750 2024-03-18T18:50:11.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.682+0000 7f1f06ffd700 1 -- 172.21.15.1:0/410067032 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1f0c059680 con 0x7f1f1806a750 2024-03-18T18:50:11.679 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-18T18:50:11.679 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.4 2024-03-18T18:50:11.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.685+0000 7f1f1cb55700 1 -- 172.21.15.1:0/410067032 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1ef806ad60 msgr2=0x7f1ef806d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.685+0000 7f1f1cb55700 1 --2- 172.21.15.1:0/410067032 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1ef806ad60 0x7f1ef806d220 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1f08000f60 tx=0x7f1f08007f10 comp rx=0 tx=0).stop 2024-03-18T18:50:11.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.685+0000 7f1f1cb55700 1 -- 172.21.15.1:0/410067032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f1806a750 msgr2=0x7f1f18160b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.685+0000 7f1f1cb55700 1 --2- 172.21.15.1:0/410067032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f1806a750 0x7f1f18160b00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0090b0 tx=0x7f1f0c004750 comp rx=0 tx=0).stop 2024-03-18T18:50:11.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.685+0000 7f1f1cb55700 1 -- 172.21.15.1:0/410067032 shutdown_connections 2024-03-18T18:50:11.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.685+0000 7f1f1cb55700 1 --2- 172.21.15.1:0/410067032 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1ef806ad60 0x7f1ef806d220 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.685+0000 7f1f1cb55700 1 --2- 172.21.15.1:0/410067032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f180c3320 0x7f1f180b4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.685+0000 7f1f1cb55700 1 --2- 172.21.15.1:0/410067032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f1806a750 0x7f1f18160b00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.685+0000 7f1f1cb55700 1 --2- 172.21.15.1:0/410067032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f1806e2b0 0x7f1f1809d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.685+0000 7f1f1cb55700 1 -- 172.21.15.1:0/410067032 >> 172.21.15.1:0/410067032 conn(0x7f1f180cca00 msgr2=0x7f1f180cc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:11.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.685+0000 7f1f1cb55700 1 -- 172.21.15.1:0/410067032 shutdown_connections 2024-03-18T18:50:11.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.685+0000 7f1f1cb55700 1 -- 172.21.15.1:0/410067032 wait complete. 2024-03-18T18:50:11.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.688+0000 7fbcd0c86700 1 -- 172.21.15.1:0/3639292817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fbcb0005870 con 0x7fbccc14b1b0 2024-03-18T18:50:11.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.688+0000 7fbcbaffd700 1 -- 172.21.15.1:0/3639292817 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbcbc022020 con 0x7fbccc14b1b0 2024-03-18T18:50:11.684 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-18T18:50:11.686 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.6 2024-03-18T18:50:11.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.691+0000 7fbcd0c86700 1 -- 172.21.15.1:0/3639292817 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbcac06ad60 msgr2=0x7fbcac06d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.691+0000 7fbcd0c86700 1 --2- 172.21.15.1:0/3639292817 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbcac06ad60 0x7fbcac06d220 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000a9b0 tx=0x7fbcc0008040 comp rx=0 tx=0).stop 2024-03-18T18:50:11.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.691+0000 7fbcd0c86700 1 -- 172.21.15.1:0/3639292817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbccc14b1b0 msgr2=0x7fbccc0ad830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.691+0000 7fbcd0c86700 1 --2- 172.21.15.1:0/3639292817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbccc14b1b0 0x7fbccc0ad830 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc00cc80 tx=0x7fbcbc00a340 comp rx=0 tx=0).stop 2024-03-18T18:50:11.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.691+0000 7fbcd0c86700 1 -- 172.21.15.1:0/3639292817 shutdown_connections 2024-03-18T18:50:11.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.691+0000 7fbcd0c86700 1 --2- 172.21.15.1:0/3639292817 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbcac06ad60 0x7fbcac06d220 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.691+0000 7fbcd0c86700 1 --2- 172.21.15.1:0/3639292817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbccc14b1b0 0x7fbccc0ad830 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.691+0000 7fbcd0c86700 1 --2- 172.21.15.1:0/3639292817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbccc06d020 0x7fbccc0a9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.691+0000 7fbcd0c86700 1 --2- 172.21.15.1:0/3639292817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbccc070c10 0x7fbccc164260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.691+0000 7fbcd0c86700 1 -- 172.21.15.1:0/3639292817 >> 172.21.15.1:0/3639292817 conn(0x7fbccc06a750 msgr2=0x7fbccc0cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:11.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.692+0000 7fbcd0c86700 1 -- 172.21.15.1:0/3639292817 shutdown_connections 2024-03-18T18:50:11.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.692+0000 7fbcd0c86700 1 -- 172.21.15.1:0/3639292817 wait complete. 2024-03-18T18:50:11.695 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.7 2024-03-18T18:50:11.705 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.2 2024-03-18T18:50:11.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.729+0000 7faf59a24700 1 -- 172.21.15.1:0/2034147432 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7faf34005160 con 0x7faf5415a320 2024-03-18T18:50:11.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.729+0000 7faf3bfff700 1 -- 172.21.15.1:0/2034147432 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faf44016090 con 0x7faf5415a320 2024-03-18T18:50:11.726 INFO:teuthology.orchestra.run.smithi001.stdout:55834574855 2024-03-18T18:50:11.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.731+0000 7faf59a24700 1 -- 172.21.15.1:0/2034147432 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf4006ad60 msgr2=0x7faf4006d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.731+0000 7faf59a24700 1 --2- 172.21.15.1:0/2034147432 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf4006ad60 0x7faf4006d220 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7faf48009710 tx=0x7faf48006c60 comp rx=0 tx=0).stop 2024-03-18T18:50:11.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.731+0000 7faf59a24700 1 -- 172.21.15.1:0/2034147432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5415a320 msgr2=0x7faf541545a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:11.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.731+0000 7faf59a24700 1 --2- 172.21.15.1:0/2034147432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5415a320 0x7faf541545a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7faf4400bbd0 tx=0x7faf44004750 comp rx=0 tx=0).stop 2024-03-18T18:50:11.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.731+0000 7faf59a24700 1 -- 172.21.15.1:0/2034147432 shutdown_connections 2024-03-18T18:50:11.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.731+0000 7faf59a24700 1 --2- 172.21.15.1:0/2034147432 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf4006ad60 0x7faf4006d220 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.731+0000 7faf59a24700 1 --2- 172.21.15.1:0/2034147432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf5415a320 0x7faf541545a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.731+0000 7faf59a24700 1 --2- 172.21.15.1:0/2034147432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf540ce890 0x7faf541f57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.731+0000 7faf59a24700 1 --2- 172.21.15.1:0/2034147432 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf54150400 0x7faf54154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:11.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.731+0000 7faf59a24700 1 -- 172.21.15.1:0/2034147432 >> 172.21.15.1:0/2034147432 conn(0x7faf5406a750 msgr2=0x7faf541583e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:11.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.731+0000 7faf59a24700 1 -- 172.21.15.1:0/2034147432 shutdown_connections 2024-03-18T18:50:11.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:11.731+0000 7faf59a24700 1 -- 172.21.15.1:0/2034147432 wait complete. 2024-03-18T18:50:11.740 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.1 2024-03-18T18:50:12.613 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-18T18:50:12.656 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-18T18:50:12.669 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-18T18:50:12.681 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-18T18:50:12.687 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-18T18:50:12.696 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-18T18:50:12.706 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-18T18:50:12.741 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-18T18:50:12.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.787+0000 7fecfb6a0700 1 Processor -- start 2024-03-18T18:50:12.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.787+0000 7fecfb6a0700 1 -- start start 2024-03-18T18:50:12.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.788+0000 7fecfb6a0700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fecf40ce450 0x7fecf40cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.788+0000 7fecfb6a0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fecf40b2210 0x7fecf40b2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.788+0000 7fecfb6a0700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf406d690 con 0x7fecf4071df0 2024-03-18T18:50:12.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.788+0000 7fecfb6a0700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf40ce060 con 0x7fecf40ce450 2024-03-18T18:50:12.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.788+0000 7fecfb6a0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf41597c0 con 0x7fecf40b2210 2024-03-18T18:50:12.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.790+0000 7fecf8c3b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fecf40ce450 0x7fecf40cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.790+0000 7fecf8c3b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fecf40ce450 0x7fecf40cd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:58206/0 (socket says 172.21.15.1:58206) 2024-03-18T18:50:12.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.790+0000 7fecf8c3b700 1 -- 172.21.15.1:0/937864382 learned_addr learned my addr 172.21.15.1:0/937864382 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:12.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.790+0000 7fecf9c3d700 1 --2- 172.21.15.1:0/937864382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fecf40b2210 0x7fecf40b2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.790+0000 7fecf8c3b700 1 -- 172.21.15.1:0/937864382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fecf40b2210 msgr2=0x7fecf40b2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:12.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.790+0000 7fecf8c3b700 1 --2- 172.21.15.1:0/937864382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fecf40b2210 0x7fecf40b2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.790+0000 7fecf8c3b700 1 -- 172.21.15.1:0/937864382 >> v1:172.21.15.165:6789/0 conn(0x7fecf4071df0 legacy=0x7fecf40721d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.790+0000 7fecf8c3b700 1 -- 172.21.15.1:0/937864382 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf406cdf0 con 0x7fecf40ce450 2024-03-18T18:50:12.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.790+0000 7fecf8c3b700 1 --2- 172.21.15.1:0/937864382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecf40ce450 0x7fecf40cd6f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fecf000a990 tx=0x7fecf000a2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e9d95427cee8dc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.792+0000 7fecebfff700 1 -- 172.21.15.1:0/937864382 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf000acf0 con 0x7fecf40ce450 2024-03-18T18:50:12.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.792+0000 7fecebfff700 1 -- 172.21.15.1:0/937864382 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf000ae50 con 0x7fecf40ce450 2024-03-18T18:50:12.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.792+0000 7fecebfff700 1 -- 172.21.15.1:0/937864382 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf000ca90 con 0x7fecf40ce450 2024-03-18T18:50:12.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.793+0000 7fecfb6a0700 1 -- 172.21.15.1:0/937864382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecf40ce450 msgr2=0x7fecf40cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.793+0000 7fecfb6a0700 1 --2- 172.21.15.1:0/937864382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecf40ce450 0x7fecf40cd6f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fecf000a990 tx=0x7fecf000a2d0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.793+0000 7fecfb6a0700 1 -- 172.21.15.1:0/937864382 shutdown_connections 2024-03-18T18:50:12.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.793+0000 7fecfb6a0700 1 --2- 172.21.15.1:0/937864382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fecf40b2210 0x7fecf40b2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.793+0000 7fecfb6a0700 1 --2- 172.21.15.1:0/937864382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecf40ce450 0x7fecf40cd6f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.793+0000 7fecfb6a0700 1 -- 172.21.15.1:0/937864382 >> 172.21.15.1:0/937864382 conn(0x7fecf406a750 msgr2=0x7fecf406ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:12.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.794+0000 7fecfb6a0700 1 -- 172.21.15.1:0/937864382 shutdown_connections 2024-03-18T18:50:12.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.795+0000 7fecfb6a0700 1 -- 172.21.15.1:0/937864382 wait complete. 2024-03-18T18:50:12.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.797+0000 7fecfb6a0700 1 Processor -- start 2024-03-18T18:50:12.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.797+0000 7fecfb6a0700 1 -- start start 2024-03-18T18:50:12.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.798+0000 7fecfb6a0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fecf4071df0 0x7fecf4142470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.798+0000 7fecfb6a0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecf40b2210 0x7fecf40b0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.798+0000 7fecfb6a0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecf40ce450 0x7fecf40b1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.798+0000 7fecfb6a0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40b1990 con 0x7fecf40ce450 2024-03-18T18:50:12.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.798+0000 7fecfb6a0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf41e13a0 con 0x7fecf40b2210 2024-03-18T18:50:12.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.798+0000 7fecfb6a0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf41e1610 con 0x7fecf4071df0 2024-03-18T18:50:12.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.800+0000 7fecf8c3b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecf40b2210 0x7fecf40b0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.800+0000 7fecf8c3b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecf40b2210 0x7fecf40b0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:58222/0 (socket says 172.21.15.1:58222) 2024-03-18T18:50:12.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.800+0000 7fecf8c3b700 1 -- 172.21.15.1:0/5849303 learned_addr learned my addr 172.21.15.1:0/5849303 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:12.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.802+0000 7fecf9c3d700 1 --2- 172.21.15.1:0/5849303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecf40ce450 0x7fecf40b1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.802+0000 7fecf9c3d700 1 -- 172.21.15.1:0/5849303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fecf4071df0 msgr2=0x7fecf4142470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T18:50:12.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.803+0000 7fecf943c700 1 --2- 172.21.15.1:0/5849303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fecf4071df0 0x7fecf4142470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.804+0000 7fecf9c3d700 1 --2- 172.21.15.1:0/5849303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fecf4071df0 0x7fecf4142470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.804+0000 7fecf9c3d700 1 -- 172.21.15.1:0/5849303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecf40b2210 msgr2=0x7fecf40b0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.804+0000 7fecf9c3d700 1 --2- 172.21.15.1:0/5849303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecf40b2210 0x7fecf40b0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.804+0000 7fecf9c3d700 1 -- 172.21.15.1:0/5849303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf000a5f0 con 0x7fecf40ce450 2024-03-18T18:50:12.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.804+0000 7fecf8c3b700 1 --2- 172.21.15.1:0/5849303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecf40b2210 0x7fecf40b0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:12.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.804+0000 7fecf943c700 1 --2- 172.21.15.1:0/5849303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fecf4071df0 0x7fecf4142470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:12.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.805+0000 7fecf9c3d700 1 --2- 172.21.15.1:0/5849303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecf40ce450 0x7fecf40b1450 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fece400ba30 tx=0x7fece400bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.812+0000 7fece9ffb700 1 -- 172.21.15.1:0/5849303 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4008930 con 0x7fecf40ce450 2024-03-18T18:50:12.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.812+0000 7fece9ffb700 1 -- 172.21.15.1:0/5849303 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece4008a90 con 0x7fecf40ce450 2024-03-18T18:50:12.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.812+0000 7fece9ffb700 1 -- 172.21.15.1:0/5849303 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4008e20 con 0x7fecf40ce450 2024-03-18T18:50:12.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.813+0000 7fecfb6a0700 1 -- 172.21.15.1:0/5849303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf41e18f0 con 0x7fecf40ce450 2024-03-18T18:50:12.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.814+0000 7fecfb6a0700 1 -- 172.21.15.1:0/5849303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf4157330 con 0x7fecf40ce450 2024-03-18T18:50:12.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.816+0000 7fece9ffb700 1 -- 172.21.15.1:0/5849303 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fece40153e0 con 0x7fecf40ce450 2024-03-18T18:50:12.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.817+0000 7fece9ffb700 1 --2- 172.21.15.1:0/5849303 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fecd806ad60 0x7fecd806d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.817+0000 7fece9ffb700 1 -- 172.21.15.1:0/5849303 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fece4089410 con 0x7fecf40ce450 2024-03-18T18:50:12.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.818+0000 7fecf943c700 1 --2- 172.21.15.1:0/5849303 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fecd806ad60 0x7fecd806d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.818+0000 7fecf943c700 1 --2- 172.21.15.1:0/5849303 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fecd806ad60 0x7fecd806d220 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fecec0012e0 tx=0x7fecec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.821+0000 7fecfb6a0700 1 -- 172.21.15.1:0/5849303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf4159000 con 0x7fecf40ce450 2024-03-18T18:50:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.833+0000 7fece9ffb700 1 -- 172.21.15.1:0/5849303 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fece4054250 con 0x7fecf40ce450 2024-03-18T18:50:12.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.864+0000 7fce7a288700 1 Processor -- start 2024-03-18T18:50:12.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.865+0000 7fce7a288700 1 -- start start 2024-03-18T18:50:12.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.866+0000 7fce7a288700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fce7414b1b0 0x7fce7406cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.866+0000 7fce7a288700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce74070de0 0x7fce740cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.866+0000 7fce7a288700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce740cc6e0 con 0x7fce7406d020 2024-03-18T18:50:12.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.866+0000 7fce7a288700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce7414b590 con 0x7fce7414b1b0 2024-03-18T18:50:12.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.866+0000 7fce7a288700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce7406d4d0 con 0x7fce74070de0 2024-03-18T18:50:12.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.867+0000 7fce737fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fce7406d020 0x7fce740706d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60328/0 (socket says 172.21.15.1:60328) 2024-03-18T18:50:12.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.867+0000 7fce737fe700 1 -- 172.21.15.1:0/1021862481 learned_addr learned my addr 172.21.15.1:0/1021862481 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:12.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.868+0000 7fce78825700 1 --2- 172.21.15.1:0/1021862481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce74070de0 0x7fce740cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.868+0000 7fce72ffd700 1 -- 172.21.15.1:0/1021862481 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1649062724 0 0) 0x7fce740cc6e0 con 0x7fce7406d020 2024-03-18T18:50:12.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.868+0000 7fce72ffd700 1 -- 172.21.15.1:0/1021862481 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce5c0035b0 con 0x7fce7406d020 2024-03-18T18:50:12.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.868+0000 7fce73fff700 1 --2- 172.21.15.1:0/1021862481 >> v2:172.21.15.165:3300/0 conn(0x7fce7414b1b0 0x7fce7406cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.869+0000 7fce73fff700 1 -- 172.21.15.1:0/1021862481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce74070de0 msgr2=0x7fce740cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:12.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.869+0000 7fce73fff700 1 --2- 172.21.15.1:0/1021862481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce74070de0 0x7fce740cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.869+0000 7fce73fff700 1 -- 172.21.15.1:0/1021862481 >> v1:172.21.15.1:6789/0 conn(0x7fce7406d020 legacy=0x7fce740706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.869+0000 7fce73fff700 1 -- 172.21.15.1:0/1021862481 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce74071440 con 0x7fce7414b1b0 2024-03-18T18:50:12.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.869+0000 7fce73fff700 1 --2- 172.21.15.1:0/1021862481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce7414b1b0 0x7fce7406cae0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fce6c009d90 tx=0x7fce6c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d38398b17b418fc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.869+0000 7fce72ffd700 1 -- 172.21.15.1:0/1021862481 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c00e070 con 0x7fce7414b1b0 2024-03-18T18:50:12.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.870+0000 7fce72ffd700 1 -- 172.21.15.1:0/1021862481 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce6c00bae0 con 0x7fce7414b1b0 2024-03-18T18:50:12.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.870+0000 7fce72ffd700 1 -- 172.21.15.1:0/1021862481 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c00be90 con 0x7fce7414b1b0 2024-03-18T18:50:12.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.871+0000 7fce7a288700 1 -- 172.21.15.1:0/1021862481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce7414b1b0 msgr2=0x7fce7406cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.871+0000 7fce7a288700 1 --2- 172.21.15.1:0/1021862481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce7414b1b0 0x7fce7406cae0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fce6c009d90 tx=0x7fce6c00b330 comp rx=0 tx=0).stop 2024-03-18T18:50:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.879+0000 7fce7a288700 1 -- 172.21.15.1:0/1021862481 shutdown_connections 2024-03-18T18:50:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.879+0000 7fce7a288700 1 --2- 172.21.15.1:0/1021862481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce7414b1b0 0x7fce7406cae0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.879+0000 7fce7a288700 1 --2- 172.21.15.1:0/1021862481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce74070de0 0x7fce740cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.879+0000 7fce7a288700 1 -- 172.21.15.1:0/1021862481 >> 172.21.15.1:0/1021862481 conn(0x7fce7406a750 msgr2=0x7fce7406ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:12.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.881+0000 7fce7a288700 1 -- 172.21.15.1:0/1021862481 shutdown_connections 2024-03-18T18:50:12.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.881+0000 7fce7a288700 1 -- 172.21.15.1:0/1021862481 wait complete. 2024-03-18T18:50:12.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.883+0000 7fce7a288700 1 Processor -- start 2024-03-18T18:50:12.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.883+0000 7fce7a288700 1 -- start start 2024-03-18T18:50:12.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.884+0000 7fce7a288700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce7406d020 0x7fce7415b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.884+0000 7fce7a288700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce74070de0 0x7fce74163fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.884+0000 7fce7a288700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce7414b1b0 0x7fce74164510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.884+0000 7fce7a288700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce74071a00 con 0x7fce74070de0 2024-03-18T18:50:12.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.884+0000 7fce7a288700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce740ac470 con 0x7fce7406d020 2024-03-18T18:50:12.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.884+0000 7fce7a288700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce7415bc50 con 0x7fce7414b1b0 2024-03-18T18:50:12.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.885+0000 7fce78825700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce7414b1b0 0x7fce74164510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.885+0000 7fce78825700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce7414b1b0 0x7fce74164510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:60198/0 (socket says 172.21.15.1:60198) 2024-03-18T18:50:12.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.885+0000 7fce78825700 1 -- 172.21.15.1:0/3091121682 learned_addr learned my addr 172.21.15.1:0/3091121682 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:12.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.885+0000 7fce78825700 1 -- 172.21.15.1:0/3091121682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce7406d020 msgr2=0x7fce7415b710 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T18:50:12.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.885+0000 7fce78825700 1 --2- 172.21.15.1:0/3091121682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce7406d020 0x7fce7415b710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.885+0000 7fce78825700 1 -- 172.21.15.1:0/3091121682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce74070de0 msgr2=0x7fce74163fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.885+0000 7fce78825700 1 --2- 172.21.15.1:0/3091121682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce74070de0 0x7fce74163fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.885+0000 7fce78825700 1 -- 172.21.15.1:0/3091121682 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6c009a70 con 0x7fce7414b1b0 2024-03-18T18:50:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.886+0000 7fce78825700 1 --2- 172.21.15.1:0/3091121682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce7414b1b0 0x7fce74164510 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fce6800e810 tx=0x7fce6800ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.886+0000 7fce70ff9700 1 -- 172.21.15.1:0/3091121682 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce68013610 con 0x7fce7414b1b0 2024-03-18T18:50:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.886+0000 7fce70ff9700 1 -- 172.21.15.1:0/3091121682 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce68013770 con 0x7fce7414b1b0 2024-03-18T18:50:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.886+0000 7fce70ff9700 1 -- 172.21.15.1:0/3091121682 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce68013b20 con 0x7fce7414b1b0 2024-03-18T18:50:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.887+0000 7fce7a288700 1 -- 172.21.15.1:0/3091121682 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce740ab530 con 0x7fce7414b1b0 2024-03-18T18:50:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.887+0000 7fce7a288700 1 -- 172.21.15.1:0/3091121682 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce740abad0 con 0x7fce7414b1b0 2024-03-18T18:50:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.890+0000 7fce70ff9700 1 -- 172.21.15.1:0/3091121682 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fce68013c80 con 0x7fce7414b1b0 2024-03-18T18:50:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.890+0000 7fce70ff9700 1 --2- 172.21.15.1:0/3091121682 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce5c06b210 0x7fce5c06d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.890+0000 7fce70ff9700 1 -- 172.21.15.1:0/3091121682 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fce68089190 con 0x7fce7414b1b0 2024-03-18T18:50:12.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.891+0000 7fce73fff700 1 --2- 172.21.15.1:0/3091121682 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce5c06b210 0x7fce5c06d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.891+0000 7fce7a288700 1 -- 172.21.15.1:0/3091121682 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce540052f0 con 0x7fce7414b1b0 2024-03-18T18:50:12.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.902+0000 7fce73fff700 1 --2- 172.21.15.1:0/3091121682 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce5c06b210 0x7fce5c06d6d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fce6c000f80 tx=0x7fce6c011000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.903+0000 7fce70ff9700 1 -- 172.21.15.1:0/3091121682 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fce680540f0 con 0x7fce7414b1b0 2024-03-18T18:50:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.920+0000 7fa50ec74700 1 Processor -- start 2024-03-18T18:50:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.920+0000 7fa50ec74700 1 -- start start 2024-03-18T18:50:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.921+0000 7fa50ec74700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa50806d1f0 0x7fa5080706d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.921+0000 7fa50ec74700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa508070c10 0x7fa5080cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.921+0000 7fa50ec74700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5080cc6e0 con 0x7fa50814b1b0 2024-03-18T18:50:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.921+0000 7fa50ec74700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa50814b590 con 0x7fa50806d1f0 2024-03-18T18:50:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.921+0000 7fa50ec74700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa508071a00 con 0x7fa508070c10 2024-03-18T18:50:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.922+0000 7fa50d211700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa508070c10 0x7fa5080cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.922+0000 7fa50d211700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa508070c10 0x7fa5080cd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:58238/0 (socket says 172.21.15.1:58238) 2024-03-18T18:50:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.922+0000 7fa50d211700 1 -- 172.21.15.1:0/2428968149 learned_addr learned my addr 172.21.15.1:0/2428968149 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.923+0000 7fa50d211700 1 -- 172.21.15.1:0/2428968149 >> v2:172.21.15.1:3300/0 conn(0x7fa50806d1f0 msgr2=0x7fa5080706d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T18:50:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.923+0000 7fa50d211700 1 --2- 172.21.15.1:0/2428968149 >> v2:172.21.15.1:3300/0 conn(0x7fa50806d1f0 0x7fa5080706d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.923+0000 7fa50d211700 1 -- 172.21.15.1:0/2428968149 >> v1:172.21.15.1:6789/0 conn(0x7fa50814b1b0 legacy=0x7fa50806cae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T18:50:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.923+0000 7fa50d211700 1 -- 172.21.15.1:0/2428968149 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa508071440 con 0x7fa508070c10 2024-03-18T18:50:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.923+0000 7fa50d211700 1 --2- 172.21.15.1:0/2428968149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa508070c10 0x7fa5080cd180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa50806e250 tx=0x7fa4fc009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34cef436aacb34b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.924+0000 7faba1b18700 1 Processor -- start 2024-03-18T18:50:12.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.925+0000 7faba1b18700 1 -- start start 2024-03-18T18:50:12.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.925+0000 7fa5077fe700 1 -- 172.21.15.1:0/2428968149 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc004230 con 0x7fa508070c10 2024-03-18T18:50:12.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.925+0000 7fa5077fe700 1 -- 172.21.15.1:0/2428968149 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4fc004390 con 0x7fa508070c10 2024-03-18T18:50:12.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.925+0000 7fa5077fe700 1 -- 172.21.15.1:0/2428968149 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc0036a0 con 0x7fa508070c10 2024-03-18T18:50:12.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.926+0000 7fa50ec74700 1 -- 172.21.15.1:0/2428968149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa508070c10 msgr2=0x7fa5080cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.926+0000 7fa50ec74700 1 --2- 172.21.15.1:0/2428968149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa508070c10 0x7fa5080cd180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa50806e250 tx=0x7fa4fc009bd0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.926+0000 7faba1b18700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fab9c06d250 0x7fab9c0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.926+0000 7faba1b18700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fab9c0b3a90 0x7fab9c0cc770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.926+0000 7faba1b18700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab9c070580 con 0x7fab9c14a480 2024-03-18T18:50:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.926+0000 7faba1b18700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab9c0713e0 con 0x7fab9c06d250 2024-03-18T18:50:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.926+0000 7faba1b18700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab9c0707d0 con 0x7fab9c0b3a90 2024-03-18T18:50:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.927+0000 7fa50ec74700 1 -- 172.21.15.1:0/2428968149 shutdown_connections 2024-03-18T18:50:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.927+0000 7fa50ec74700 1 --2- 172.21.15.1:0/2428968149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa508070c10 0x7fa5080cd180 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.927+0000 7fa50ec74700 1 --2- 172.21.15.1:0/2428968149 >> v2:172.21.15.1:3300/0 conn(0x7fa50806d1f0 0x7fa5080706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.927+0000 7fa50ec74700 1 -- 172.21.15.1:0/2428968149 >> 172.21.15.1:0/2428968149 conn(0x7fa50806a750 msgr2=0x7fa50806ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.927+0000 7fab9b7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fab9c14a480 0x7fab9c06cb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60344/0 (socket says 172.21.15.1:60344) 2024-03-18T18:50:12.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.927+0000 7fab9b7fe700 1 -- 172.21.15.1:0/3798239346 learned_addr learned my addr 172.21.15.1:0/3798239346 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:12.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.927+0000 7fab9bfff700 1 --2- 172.21.15.1:0/3798239346 >> v2:172.21.15.165:3300/0 conn(0x7fab9c0b3a90 0x7fab9c0cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.927+0000 7fa50ec74700 1 -- 172.21.15.1:0/2428968149 shutdown_connections 2024-03-18T18:50:12.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.927+0000 7fa50ec74700 1 -- 172.21.15.1:0/2428968149 wait complete. 2024-03-18T18:50:12.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.927+0000 7fab9affd700 1 --2- 172.21.15.1:0/3798239346 >> v2:172.21.15.1:3300/0 conn(0x7fab9c06d250 0x7fab9c0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.927+0000 7fab9affd700 1 -- 172.21.15.1:0/3798239346 >> v2:172.21.15.165:3300/0 conn(0x7fab9c0b3a90 msgr2=0x7fab9c0cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:12.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.927+0000 7fab9affd700 1 --2- 172.21.15.1:0/3798239346 >> v2:172.21.15.165:3300/0 conn(0x7fab9c0b3a90 0x7fab9c0cc770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.927+0000 7fab9affd700 1 -- 172.21.15.1:0/3798239346 >> v1:172.21.15.1:6789/0 conn(0x7fab9c14a480 legacy=0x7fab9c06cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.927+0000 7fab9affd700 1 -- 172.21.15.1:0/3798239346 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab9c070ab0 con 0x7fab9c06d250 2024-03-18T18:50:12.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.928+0000 7fab9affd700 1 --2- 172.21.15.1:0/3798239346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab9c06d250 0x7fab9c0b3550 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0b4600 tx=0x7fab88009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501b89b6bf6f8b05 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.928+0000 7fab9a7fc700 1 -- 172.21.15.1:0/3798239346 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab88011830 con 0x7fab9c06d250 2024-03-18T18:50:12.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.928+0000 7fab9a7fc700 1 -- 172.21.15.1:0/3798239346 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab88011990 con 0x7fab9c06d250 2024-03-18T18:50:12.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.928+0000 7fab9a7fc700 1 -- 172.21.15.1:0/3798239346 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab88011d20 con 0x7fab9c06d250 2024-03-18T18:50:12.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.929+0000 7fa50ec74700 1 Processor -- start 2024-03-18T18:50:12.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.929+0000 7fa50ec74700 1 -- start start 2024-03-18T18:50:12.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.929+0000 7faba1b18700 1 -- 172.21.15.1:0/3798239346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab9c06d250 msgr2=0x7fab9c0b3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.929+0000 7faba1b18700 1 --2- 172.21.15.1:0/3798239346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab9c06d250 0x7fab9c0b3550 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0b4600 tx=0x7fab88009b20 comp rx=0 tx=0).stop 2024-03-18T18:50:12.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.930+0000 7fa50ec74700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa50806d1f0 0x7fa5080a84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.930+0000 7fa50ec74700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa508070c10 0x7fa5080a89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.930+0000 7fa50ec74700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa50814b1b0 0x7fa5080a8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.930+0000 7fa50ec74700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5080ae2c0 con 0x7fa508070c10 2024-03-18T18:50:12.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.930+0000 7fa50ec74700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5080ab840 con 0x7fa50806d1f0 2024-03-18T18:50:12.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.930+0000 7fa50ec74700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5080abb20 con 0x7fa50814b1b0 2024-03-18T18:50:12.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.931+0000 7fa50ca10700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa50806d1f0 0x7fa5080a84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.931+0000 7fa507fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa508070c10 0x7fa5080a89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.931+0000 7fa50ca10700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa50806d1f0 0x7fa5080a84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:58256/0 (socket says 172.21.15.1:58256) 2024-03-18T18:50:12.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.931+0000 7fa50ca10700 1 -- 172.21.15.1:0/581034463 learned_addr learned my addr 172.21.15.1:0/581034463 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:12.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.931+0000 7faba1b18700 1 -- 172.21.15.1:0/3798239346 shutdown_connections 2024-03-18T18:50:12.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.931+0000 7faba1b18700 1 --2- 172.21.15.1:0/3798239346 >> v2:172.21.15.165:3300/0 conn(0x7fab9c0b3a90 0x7fab9c0cc770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.931+0000 7faba1b18700 1 --2- 172.21.15.1:0/3798239346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab9c06d250 0x7fab9c0b3550 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.931+0000 7faba1b18700 1 -- 172.21.15.1:0/3798239346 >> 172.21.15.1:0/3798239346 conn(0x7fab9c06ab70 msgr2=0x7fab9c06af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:12.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.931+0000 7fa50d211700 1 --2- 172.21.15.1:0/581034463 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa50814b1b0 0x7fa5080a8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.931+0000 7faba1b18700 1 -- 172.21.15.1:0/3798239346 shutdown_connections 2024-03-18T18:50:12.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.931+0000 7fa507fff700 1 -- 172.21.15.1:0/581034463 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa50814b1b0 msgr2=0x7fa5080a8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.931+0000 7fa507fff700 1 --2- 172.21.15.1:0/581034463 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa50814b1b0 0x7fa5080a8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.931+0000 7fa507fff700 1 -- 172.21.15.1:0/581034463 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa50806d1f0 msgr2=0x7fa5080a84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.931+0000 7fa507fff700 1 --2- 172.21.15.1:0/581034463 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa50806d1f0 0x7fa5080a84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.931+0000 7fa507fff700 1 -- 172.21.15.1:0/581034463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc009550 con 0x7fa508070c10 2024-03-18T18:50:12.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.931+0000 7fa50d211700 1 --2- 172.21.15.1:0/581034463 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa50814b1b0 0x7fa5080a8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:50:12.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.931+0000 7fa50ca10700 1 --2- 172.21.15.1:0/581034463 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa50806d1f0 0x7fa5080a84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:50:12.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.932+0000 7fa507fff700 1 --2- 172.21.15.1:0/581034463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa508070c10 0x7fa5080a89e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa50000bd50 tx=0x7fa50000bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.932+0000 7f0434b79700 1 Processor -- start 2024-03-18T18:50:12.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.932+0000 7fa5057fa700 1 -- 172.21.15.1:0/581034463 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50000cbb0 con 0x7fa508070c10 2024-03-18T18:50:12.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.932+0000 7fa5057fa700 1 -- 172.21.15.1:0/581034463 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50000cd30 con 0x7fa508070c10 2024-03-18T18:50:12.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.932+0000 7fa5057fa700 1 -- 172.21.15.1:0/581034463 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa500014680 con 0x7fa508070c10 2024-03-18T18:50:12.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.933+0000 7f516f7a6700 1 Processor -- start 2024-03-18T18:50:12.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.932+0000 7fa50ec74700 1 -- 172.21.15.1:0/581034463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5080abe00 con 0x7fa508070c10 2024-03-18T18:50:12.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.932+0000 7fa50ec74700 1 -- 172.21.15.1:0/581034463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa508161560 con 0x7fa508070c10 2024-03-18T18:50:12.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.935+0000 7fa5057fa700 1 -- 172.21.15.1:0/581034463 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa50001d3b0 con 0x7fa508070c10 2024-03-18T18:50:12.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.935+0000 7fa50ec74700 1 -- 172.21.15.1:0/581034463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5080ac320 con 0x7fa508070c10 2024-03-18T18:50:12.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.935+0000 7fa5057fa700 1 --2- 172.21.15.1:0/581034463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa4f406ad60 0x7fa4f406d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.936+0000 7fa50ca10700 1 --2- 172.21.15.1:0/581034463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa4f406ad60 0x7fa4f406d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.933+0000 7f0434b79700 1 -- start start 2024-03-18T18:50:12.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.934+0000 7f0434b79700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0430071df0 0x7f04300721d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.934+0000 7f0434b79700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04300b2210 0x7f04300b2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.934+0000 7f0434b79700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f043006d690 con 0x7f04300ce450 2024-03-18T18:50:12.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.934+0000 7f0434b79700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04300ce060 con 0x7f0430071df0 2024-03-18T18:50:12.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.934+0000 7f0434b79700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04301597c0 con 0x7f04300b2210 2024-03-18T18:50:12.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.935+0000 7f042e59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0430071df0 0x7f04300721d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.935+0000 7f042e59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0430071df0 0x7f04300721d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45410/0 (socket says 172.21.15.1:45410) 2024-03-18T18:50:12.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.936+0000 7f042e59c700 1 -- 172.21.15.1:0/28262806 learned_addr learned my addr 172.21.15.1:0/28262806 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:12.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.936+0000 7f042e59c700 1 -- 172.21.15.1:0/28262806 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04300b2210 msgr2=0x7f04300b2680 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T18:50:12.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.936+0000 7f042e59c700 1 --2- 172.21.15.1:0/28262806 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04300b2210 0x7f04300b2680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.936+0000 7f042e59c700 1 -- 172.21.15.1:0/28262806 >> v1:172.21.15.1:6789/0 conn(0x7f04300ce450 legacy=0x7f04300cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:12.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.936+0000 7f042e59c700 1 -- 172.21.15.1:0/28262806 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04300cde00 con 0x7f0430071df0 2024-03-18T18:50:12.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.935+0000 7faba1b18700 1 -- 172.21.15.1:0/3798239346 wait complete. 2024-03-18T18:50:12.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.933+0000 7f516f7a6700 1 -- start start 2024-03-18T18:50:12.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.935+0000 7f516f7a6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5168071df0 0x7f51680721d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.935+0000 7f516f7a6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51680b2210 0x7f51680b2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.935+0000 7f516f7a6700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f516806d690 con 0x7f51680ce450 2024-03-18T18:50:12.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.935+0000 7f516f7a6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51680ce060 con 0x7f51680b2210 2024-03-18T18:50:12.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.935+0000 7f516f7a6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51681597c0 con 0x7f5168071df0 2024-03-18T18:50:12.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.935+0000 7f516cd41700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f51680ce450 0x7f51680cd6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:38364/0 (socket says 172.21.15.1:38364) 2024-03-18T18:50:12.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.935+0000 7f516cd41700 1 -- 172.21.15.1:0/4020383393 learned_addr learned my addr 172.21.15.1:0/4020383393 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:12.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.936+0000 7f042e59c700 1 --2- 172.21.15.1:0/28262806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0430071df0 0x7f04300721d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f042800d370 tx=0x7f042800d680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3400992ad68e0012 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.937+0000 7f516d542700 1 --2- 172.21.15.1:0/4020383393 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5168071df0 0x7f51680721d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.937+0000 7fa5057fa700 1 -- 172.21.15.1:0/581034463 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa5000895e0 con 0x7fa508070c10 2024-03-18T18:50:12.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.937+0000 7fa50ca10700 1 --2- 172.21.15.1:0/581034463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa4f406ad60 0x7fa4f406d220 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009e50 tx=0x7fa4f8009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.937+0000 7f516dd43700 1 --2- 172.21.15.1:0/4020383393 >> v2:172.21.15.1:3300/0 conn(0x7f51680b2210 0x7f51680b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.937+0000 7faba1b18700 1 Processor -- start 2024-03-18T18:50:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.937+0000 7f516dd43700 1 -- 172.21.15.1:0/4020383393 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5168071df0 msgr2=0x7f51680721d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.937+0000 7f516dd43700 1 --2- 172.21.15.1:0/4020383393 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5168071df0 0x7f51680721d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.937+0000 7f516dd43700 1 -- 172.21.15.1:0/4020383393 >> v1:172.21.15.165:6789/0 conn(0x7f51680ce450 legacy=0x7f51680cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.937+0000 7f516dd43700 1 -- 172.21.15.1:0/4020383393 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51680cde00 con 0x7f51680b2210 2024-03-18T18:50:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.938+0000 7f516dd43700 1 --2- 172.21.15.1:0/4020383393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51680b2210 0x7f51680b2660 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f51580091b0 tx=0x7f51580094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4160ce16ef725fa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.938+0000 7faba1b18700 1 -- start start 2024-03-18T18:50:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.939+0000 7faba1b18700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab9c06d250 0x7fab9c2028a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.940+0000 7faba1b18700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fab9c0b3a90 0x7fab9c149010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.940+0000 7faba1b18700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fab9c14a480 0x7fab9c149550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.940+0000 7faba1b18700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c202de0 con 0x7fab9c06d250 2024-03-18T18:50:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.940+0000 7faba1b18700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c141790 con 0x7fab9c0b3a90 2024-03-18T18:50:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.940+0000 7faba1b18700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c13f170 con 0x7fab9c14a480 2024-03-18T18:50:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.940+0000 7fab9b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab9c06d250 0x7fab9c2028a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.940+0000 7fab9b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab9c06d250 0x7fab9c2028a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45434/0 (socket says 172.21.15.1:45434) 2024-03-18T18:50:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.940+0000 7fab9b7fe700 1 -- 172.21.15.1:0/1877649225 learned_addr learned my addr 172.21.15.1:0/1877649225 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.940+0000 7fab9bfff700 1 --2- 172.21.15.1:0/1877649225 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fab9c14a480 0x7fab9c149550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.940+0000 7fab9affd700 1 --2- 172.21.15.1:0/1877649225 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fab9c0b3a90 0x7fab9c149010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.940+0000 7fab9b7fe700 1 -- 172.21.15.1:0/1877649225 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fab9c14a480 msgr2=0x7fab9c149550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.940+0000 7fab9b7fe700 1 --2- 172.21.15.1:0/1877649225 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fab9c14a480 0x7fab9c149550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.940+0000 7fab9b7fe700 1 -- 172.21.15.1:0/1877649225 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fab9c0b3a90 msgr2=0x7fab9c149010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.940+0000 7fab9b7fe700 1 --2- 172.21.15.1:0/1877649225 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fab9c0b3a90 0x7fab9c149010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.940+0000 7fab9b7fe700 1 -- 172.21.15.1:0/1877649225 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab880094a0 con 0x7fab9c06d250 2024-03-18T18:50:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.941+0000 7fab9b7fe700 1 --2- 172.21.15.1:0/1877649225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab9c06d250 0x7fab9c2028a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fab8c00c520 tx=0x7fab8c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.943+0000 7f515ffff700 1 -- 172.21.15.1:0/4020383393 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515800c900 con 0x7f51680b2210 2024-03-18T18:50:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.943+0000 7f515ffff700 1 -- 172.21.15.1:0/4020383393 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515800ca60 con 0x7f51680b2210 2024-03-18T18:50:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.943+0000 7f515ffff700 1 -- 172.21.15.1:0/4020383393 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515800cdd0 con 0x7f51680b2210 2024-03-18T18:50:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.943+0000 7fa5057fa700 1 -- 172.21.15.1:0/581034463 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa5080ac320 con 0x7fa508070c10 2024-03-18T18:50:12.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.943+0000 7f042d59a700 1 -- 172.21.15.1:0/28262806 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0428010070 con 0x7f0430071df0 2024-03-18T18:50:12.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.943+0000 7f042d59a700 1 -- 172.21.15.1:0/28262806 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042800dcd0 con 0x7f0430071df0 2024-03-18T18:50:12.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.943+0000 7f042d59a700 1 -- 172.21.15.1:0/28262806 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0428007d70 con 0x7f0430071df0 2024-03-18T18:50:12.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.944+0000 7f516f7a6700 1 -- 172.21.15.1:0/4020383393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51680b2210 msgr2=0x7f51680b2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.944+0000 7f516f7a6700 1 --2- 172.21.15.1:0/4020383393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51680b2210 0x7f51680b2660 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f51580091b0 tx=0x7f51580094c0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.944+0000 7f0434b79700 1 -- 172.21.15.1:0/28262806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0430071df0 msgr2=0x7f04300721d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.944+0000 7f0434b79700 1 --2- 172.21.15.1:0/28262806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0430071df0 0x7f04300721d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f042800d370 tx=0x7f042800d680 comp rx=0 tx=0).stop 2024-03-18T18:50:12.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.944+0000 7f516f7a6700 1 -- 172.21.15.1:0/4020383393 shutdown_connections 2024-03-18T18:50:12.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.944+0000 7f516f7a6700 1 --2- 172.21.15.1:0/4020383393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51680b2210 0x7f51680b2660 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.944+0000 7f516f7a6700 1 --2- 172.21.15.1:0/4020383393 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5168071df0 0x7f51680721d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.944+0000 7f516f7a6700 1 -- 172.21.15.1:0/4020383393 >> 172.21.15.1:0/4020383393 conn(0x7f516806a750 msgr2=0x7f516806ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.944+0000 7fab93fff700 1 -- 172.21.15.1:0/1877649225 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8c005d70 con 0x7fab9c06d250 2024-03-18T18:50:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.944+0000 7fab93fff700 1 -- 172.21.15.1:0/1877649225 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab8c0070d0 con 0x7fab9c06d250 2024-03-18T18:50:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.944+0000 7fab93fff700 1 -- 172.21.15.1:0/1877649225 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8c007440 con 0x7fab9c06d250 2024-03-18T18:50:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.947+0000 7f516f7a6700 1 -- 172.21.15.1:0/4020383393 shutdown_connections 2024-03-18T18:50:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.947+0000 7faba1b18700 1 -- 172.21.15.1:0/1877649225 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab9c141cc0 con 0x7fab9c06d250 2024-03-18T18:50:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.947+0000 7faba1b18700 1 -- 172.21.15.1:0/1877649225 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab9c13ee00 con 0x7fab9c06d250 2024-03-18T18:50:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.947+0000 7f516f7a6700 1 -- 172.21.15.1:0/4020383393 wait complete. 2024-03-18T18:50:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.949+0000 7f516f7a6700 1 Processor -- start 2024-03-18T18:50:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.949+0000 7f516f7a6700 1 -- start start 2024-03-18T18:50:12.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.950+0000 7f516f7a6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5168071df0 0x7f51680aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.950+0000 7fab93fff700 1 -- 172.21.15.1:0/1877649225 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fab8c0075a0 con 0x7fab9c06d250 2024-03-18T18:50:12.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.950+0000 7f516f7a6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51680b2210 0x7f5168139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.950+0000 7f516d542700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5168071df0 0x7f51680aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.951+0000 7f516d542700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5168071df0 0x7f51680aeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:58272/0 (socket says 172.21.15.1:58272) 2024-03-18T18:50:12.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.951+0000 7f516cd41700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51680b2210 0x7f5168139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.951+0000 7f516cd41700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51680b2210 0x7f5168139fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:60252/0 (socket says 172.21.15.1:60252) 2024-03-18T18:50:12.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.951+0000 7fab93fff700 1 --2- 172.21.15.1:0/1877649225 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fab8406ac90 0x7fab8406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.951+0000 7fab93fff700 1 -- 172.21.15.1:0/1877649225 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fab8c0889e0 con 0x7fab9c06d250 2024-03-18T18:50:12.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.951+0000 7f516f7a6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51680ce450 0x7f516814d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.951+0000 7f516f7a6700 1 -- 172.21.15.1:0/1607412839 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5168138100 con 0x7f51680ce450 2024-03-18T18:50:12.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.951+0000 7f516d542700 1 -- 172.21.15.1:0/1607412839 learned_addr learned my addr 172.21.15.1:0/1607412839 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:12.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.951+0000 7f516f7a6700 1 -- 172.21.15.1:0/1607412839 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516814d900 con 0x7f5168071df0 2024-03-18T18:50:12.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.951+0000 7f516f7a6700 1 -- 172.21.15.1:0/1607412839 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5168137480 con 0x7f51680b2210 2024-03-18T18:50:12.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.951+0000 7f516dd43700 1 --2- 172.21.15.1:0/1607412839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51680ce450 0x7f516814d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.952+0000 7f0434b79700 1 -- 172.21.15.1:0/28262806 shutdown_connections 2024-03-18T18:50:12.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.952+0000 7f0434b79700 1 --2- 172.21.15.1:0/28262806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0430071df0 0x7f04300721d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.952+0000 7f0434b79700 1 --2- 172.21.15.1:0/28262806 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04300b2210 0x7f04300b2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.952+0000 7f0434b79700 1 -- 172.21.15.1:0/28262806 >> 172.21.15.1:0/28262806 conn(0x7f043006a750 msgr2=0x7f043006ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:12.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.952+0000 7f516cd41700 1 -- 172.21.15.1:0/1607412839 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5168071df0 msgr2=0x7f51680aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.952+0000 7f516cd41700 1 --2- 172.21.15.1:0/1607412839 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5168071df0 0x7f51680aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.952+0000 7f516cd41700 1 -- 172.21.15.1:0/1607412839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51680ce450 msgr2=0x7f516814d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.952+0000 7f516cd41700 1 --2- 172.21.15.1:0/1607412839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51680ce450 0x7f516814d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.952+0000 7f516cd41700 1 -- 172.21.15.1:0/1607412839 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5158009d30 con 0x7f51680b2210 2024-03-18T18:50:12.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.952+0000 7f516d542700 1 --2- 172.21.15.1:0/1607412839 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5168071df0 0x7f51680aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:50:12.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.952+0000 7f516cd41700 1 --2- 172.21.15.1:0/1607412839 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51680b2210 0x7f5168139fa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f51680ce870 tx=0x7f516000cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.954+0000 7fab9affd700 1 --2- 172.21.15.1:0/1877649225 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fab8406ac90 0x7fab8406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.954+0000 7fab9affd700 1 --2- 172.21.15.1:0/1877649225 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fab8406ac90 0x7fab8406d150 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fab9c072d30 tx=0x7fab8801c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.954+0000 7faba1b18700 1 -- 172.21.15.1:0/1877649225 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab9c073350 con 0x7fab9c06d250 2024-03-18T18:50:12.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.956+0000 7f515dffb700 1 -- 172.21.15.1:0/1607412839 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516000a9e0 con 0x7f51680b2210 2024-03-18T18:50:12.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.956+0000 7f515dffb700 1 -- 172.21.15.1:0/1607412839 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516000ab40 con 0x7f51680b2210 2024-03-18T18:50:12.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.956+0000 7f515dffb700 1 -- 172.21.15.1:0/1607412839 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516000ce80 con 0x7f51680b2210 2024-03-18T18:50:12.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.956+0000 7f516f7a6700 1 -- 172.21.15.1:0/1607412839 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5168137760 con 0x7f51680b2210 2024-03-18T18:50:12.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.956+0000 7f516f7a6700 1 -- 172.21.15.1:0/1607412839 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5168137d20 con 0x7f51680b2210 2024-03-18T18:50:12.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.957+0000 7f0434b79700 1 -- 172.21.15.1:0/28262806 shutdown_connections 2024-03-18T18:50:12.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.958+0000 7f0434b79700 1 -- 172.21.15.1:0/28262806 wait complete. 2024-03-18T18:50:12.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.960+0000 7f0434b79700 1 Processor -- start 2024-03-18T18:50:12.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.960+0000 7f0434b79700 1 -- start start 2024-03-18T18:50:12.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.961+0000 7f0434b79700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0430071df0 0x7f04301528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.961+0000 7f0434b79700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04300b2210 0x7f04300add50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.961+0000 7f0434b79700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04300ce450 0x7f04301ecfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.961+0000 7f0434b79700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043013d730 con 0x7f04300ce450 2024-03-18T18:50:12.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.961+0000 7f0434b79700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043013cb60 con 0x7f0430071df0 2024-03-18T18:50:12.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.961+0000 7f0434b79700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04300ae290 con 0x7f04300b2210 2024-03-18T18:50:12.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.962+0000 7f042dd9b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04300b2210 0x7f04300add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.962+0000 7f042e59c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0430071df0 0x7f04301528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.962+0000 7f042dd9b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04300b2210 0x7f04300add50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:60262/0 (socket says 172.21.15.1:60262) 2024-03-18T18:50:12.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.962+0000 7f042dd9b700 1 -- 172.21.15.1:0/1741387052 learned_addr learned my addr 172.21.15.1:0/1741387052 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:12.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.962+0000 7f042e59c700 1 --2- 172.21.15.1:0/1741387052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0430071df0 0x7f04301528a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:58286/0 (socket says 172.21.15.1:58286) 2024-03-18T18:50:12.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.962+0000 7f042dd9b700 1 -- 172.21.15.1:0/1741387052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0430071df0 msgr2=0x7f04301528a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.962+0000 7f042dd9b700 1 --2- 172.21.15.1:0/1741387052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0430071df0 0x7f04301528a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.962+0000 7f042dd9b700 1 -- 172.21.15.1:0/1741387052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04300ce450 msgr2=0x7f04301ecfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.962+0000 7f042dd9b700 1 --2- 172.21.15.1:0/1741387052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04300ce450 0x7f04301ecfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.962+0000 7f042dd9b700 1 -- 172.21.15.1:0/1741387052 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042800d050 con 0x7f04300b2210 2024-03-18T18:50:12.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.962+0000 7f042dd9b700 1 --2- 172.21.15.1:0/1741387052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04300b2210 0x7f04300add50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f042400b470 tx=0x7f042400b780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.963+0000 7f041effd700 1 -- 172.21.15.1:0/1741387052 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0424011830 con 0x7f04300b2210 2024-03-18T18:50:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.963+0000 7f0434b79700 1 -- 172.21.15.1:0/1741387052 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f043013ce00 con 0x7f04300b2210 2024-03-18T18:50:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.963+0000 7f0434b79700 1 -- 172.21.15.1:0/1741387052 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f043013d3c0 con 0x7f04300b2210 2024-03-18T18:50:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.964+0000 7f0434b79700 1 -- 172.21.15.1:0/1741387052 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0430073350 con 0x7f04300b2210 2024-03-18T18:50:12.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.965+0000 7fab93fff700 1 -- 172.21.15.1:0/1877649225 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fab8c0538e0 con 0x7fab9c06d250 2024-03-18T18:50:12.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.968+0000 7f041effd700 1 -- 172.21.15.1:0/1741387052 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04240092a0 con 0x7f04300b2210 2024-03-18T18:50:12.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.968+0000 7f041effd700 1 -- 172.21.15.1:0/1741387052 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0424011b90 con 0x7f04300b2210 2024-03-18T18:50:12.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.970+0000 7f041effd700 1 -- 172.21.15.1:0/1741387052 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0424011cf0 con 0x7f04300b2210 2024-03-18T18:50:12.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.970+0000 7f041effd700 1 --2- 172.21.15.1:0/1741387052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f041006ac90 0x7f041006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.970+0000 7f041effd700 1 -- 172.21.15.1:0/1741387052 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0424089c00 con 0x7f04300b2210 2024-03-18T18:50:12.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.962+0000 7f516f7a6700 1 -- 172.21.15.1:0/1607412839 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51480052f0 con 0x7f51680b2210 2024-03-18T18:50:12.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.968+0000 7f515dffb700 1 -- 172.21.15.1:0/1607412839 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f516001b3b0 con 0x7f51680b2210 2024-03-18T18:50:12.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.969+0000 7f515dffb700 1 --2- 172.21.15.1:0/1607412839 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f515406f300 0x7f51540717c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.969+0000 7f515dffb700 1 -- 172.21.15.1:0/1607412839 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5160089500 con 0x7f51680b2210 2024-03-18T18:50:12.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.969+0000 7f515dffb700 1 -- 172.21.15.1:0/1607412839 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5160089980 con 0x7f51680b2210 2024-03-18T18:50:12.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.974+0000 7f042e59c700 1 --2- 172.21.15.1:0/1741387052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f041006ac90 0x7f041006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.975+0000 7f042e59c700 1 --2- 172.21.15.1:0/1741387052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f041006ac90 0x7f041006d150 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f042800d6c0 tx=0x7f0428012000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.973+0000 7f516d542700 1 --2- 172.21.15.1:0/1607412839 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f515406f300 0x7f51540717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.974+0000 7f516d542700 1 --2- 172.21.15.1:0/1607412839 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f515406f300 0x7f51540717c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f516400f240 tx=0x7f5164008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.976+0000 7f9808bf5700 1 Processor -- start 2024-03-18T18:50:12.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.976+0000 7f9808bf5700 1 -- start start 2024-03-18T18:50:12.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.978+0000 7f9808bf5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9804160e90 0x7f980406c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.978+0000 7f9808bf5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f980416b110 0x7f980408a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.978+0000 7f9808bf5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f980408a810 0x7f980416daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.978+0000 7f9808bf5700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f980408aca0 con 0x7f9804160e90 2024-03-18T18:50:12.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.978+0000 7f9808bf5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f980406e0c0 con 0x7f980408a810 2024-03-18T18:50:12.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.978+0000 7f9808bf5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98040b3f40 con 0x7f980416b110 2024-03-18T18:50:12.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.978+0000 7f9801d9b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f980416b110 0x7f980408a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.978+0000 7f9801d9b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f980416b110 0x7f980408a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:60278/0 (socket says 172.21.15.1:60278) 2024-03-18T18:50:12.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.978+0000 7f9801d9b700 1 -- 172.21.15.1:0/880417305 learned_addr learned my addr 172.21.15.1:0/880417305 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:12.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.979+0000 7f9802d9d700 1 --2- 172.21.15.1:0/880417305 >> v2:172.21.15.165:3300/0 conn(0x7f980408a810 0x7f980416daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.980+0000 7f9802d9d700 1 -- 172.21.15.1:0/880417305 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f980416b110 msgr2=0x7f980408a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:12.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.980+0000 7f9802d9d700 1 --2- 172.21.15.1:0/880417305 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f980416b110 0x7f980408a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.980+0000 7f9802d9d700 1 -- 172.21.15.1:0/880417305 >> v2:172.21.15.1:3300/0 conn(0x7f9804160e90 msgr2=0x7f980406c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:12.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.980+0000 7f9802d9d700 1 --2- 172.21.15.1:0/880417305 >> v2:172.21.15.1:3300/0 conn(0x7f9804160e90 0x7f980406c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.980+0000 7f9802d9d700 1 -- 172.21.15.1:0/880417305 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980406d020 con 0x7f980408a810 2024-03-18T18:50:12.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.980+0000 7f9802d9d700 1 --2- 172.21.15.1:0/880417305 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f980408a810 0x7f980416daf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f980416b590 tx=0x7f97f4009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b642a0c0c7aa5fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.980+0000 7f980159a700 1 -- 172.21.15.1:0/880417305 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f4011830 con 0x7f980408a810 2024-03-18T18:50:12.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.980+0000 7f980159a700 1 -- 172.21.15.1:0/880417305 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f4011990 con 0x7f980408a810 2024-03-18T18:50:12.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.980+0000 7f980159a700 1 -- 172.21.15.1:0/880417305 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f4011d20 con 0x7f980408a810 2024-03-18T18:50:12.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.981+0000 7f041effd700 1 -- 172.21.15.1:0/1741387052 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0424054b40 con 0x7f04300b2210 2024-03-18T18:50:12.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.981+0000 7f9808bf5700 1 -- 172.21.15.1:0/880417305 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f980408a810 msgr2=0x7f980416daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.981+0000 7f9808bf5700 1 --2- 172.21.15.1:0/880417305 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f980408a810 0x7f980416daf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f980416b590 tx=0x7f97f4009b20 comp rx=0 tx=0).stop 2024-03-18T18:50:12.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.981+0000 7f9808bf5700 1 -- 172.21.15.1:0/880417305 shutdown_connections 2024-03-18T18:50:12.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.981+0000 7f9808bf5700 1 --2- 172.21.15.1:0/880417305 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f980408a810 0x7f980416daf0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.981+0000 7f9808bf5700 1 --2- 172.21.15.1:0/880417305 >> v2:172.21.15.1:3300/0 conn(0x7f9804160e90 0x7f980406c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.981+0000 7f9808bf5700 1 --2- 172.21.15.1:0/880417305 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f980416b110 0x7f980408a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.981+0000 7f9808bf5700 1 -- 172.21.15.1:0/880417305 >> 172.21.15.1:0/880417305 conn(0x7f980406a750 msgr2=0x7f980406ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:12.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.987+0000 7f9808bf5700 1 -- 172.21.15.1:0/880417305 shutdown_connections 2024-03-18T18:50:12.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.987+0000 7f9808bf5700 1 -- 172.21.15.1:0/880417305 wait complete. 2024-03-18T18:50:12.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.989+0000 7f9808bf5700 1 Processor -- start 2024-03-18T18:50:12.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.990+0000 7f9808bf5700 1 -- start start 2024-03-18T18:50:12.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.991+0000 7f9808bf5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f980408a810 0x7f98040acfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.991+0000 7f9808bf5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9804160e90 0x7f98040aa3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.991+0000 7f9808bf5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f980416b110 0x7f98040aa8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.991+0000 7f9808bf5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9804160360 con 0x7f980408a810 2024-03-18T18:50:12.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.991+0000 7f9808bf5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98040a9800 con 0x7f980416b110 2024-03-18T18:50:12.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.991+0000 7f9808bf5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98040ae2b0 con 0x7f9804160e90 2024-03-18T18:50:12.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.992+0000 7f980259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f980408a810 0x7f98040acfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.992+0000 7f980259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f980408a810 0x7f98040acfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45472/0 (socket says 172.21.15.1:45472) 2024-03-18T18:50:12.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.992+0000 7f980259c700 1 -- 172.21.15.1:0/58078322 learned_addr learned my addr 172.21.15.1:0/58078322 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:12.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.993+0000 7f9802d9d700 1 --2- 172.21.15.1:0/58078322 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f980416b110 0x7f98040aa8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.993+0000 7f9801d9b700 1 --2- 172.21.15.1:0/58078322 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9804160e90 0x7f98040aa3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.993+0000 7f980259c700 1 -- 172.21.15.1:0/58078322 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9804160e90 msgr2=0x7f98040aa3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.993+0000 7f980259c700 1 --2- 172.21.15.1:0/58078322 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9804160e90 0x7f98040aa3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.993+0000 7f980259c700 1 -- 172.21.15.1:0/58078322 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f980416b110 msgr2=0x7f98040aa8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:12.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.993+0000 7f980259c700 1 --2- 172.21.15.1:0/58078322 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f980416b110 0x7f98040aa8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:12.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.993+0000 7f980259c700 1 -- 172.21.15.1:0/58078322 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f40094a0 con 0x7f980408a810 2024-03-18T18:50:12.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.993+0000 7f980259c700 1 --2- 172.21.15.1:0/58078322 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f980408a810 0x7f98040acfc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f97fc00ba50 tx=0x7f97fc00be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:12.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.994+0000 7f97f2ffd700 1 -- 172.21.15.1:0/58078322 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc00cab0 con 0x7f980408a810 2024-03-18T18:50:12.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.995+0000 7f97f2ffd700 1 -- 172.21.15.1:0/58078322 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97fc00cc30 con 0x7f980408a810 2024-03-18T18:50:12.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.995+0000 7f97f2ffd700 1 -- 172.21.15.1:0/58078322 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc014440 con 0x7f980408a810 2024-03-18T18:50:12.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.996+0000 7f9808bf5700 1 -- 172.21.15.1:0/58078322 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98040ad6a0 con 0x7f980408a810 2024-03-18T18:50:12.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.996+0000 7f9808bf5700 1 -- 172.21.15.1:0/58078322 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98040adc60 con 0x7f980408a810 2024-03-18T18:50:12.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.997+0000 7f9808bf5700 1 -- 172.21.15.1:0/58078322 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9804073350 con 0x7f980408a810 2024-03-18T18:50:12.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:12.999+0000 7f97f2ffd700 1 -- 172.21.15.1:0/58078322 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f97fc0145c0 con 0x7f980408a810 2024-03-18T18:50:12.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.000+0000 7f97f2ffd700 1 --2- 172.21.15.1:0/58078322 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97e406ad60 0x7f97e406d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:12.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.000+0000 7f97f2ffd700 1 -- 172.21.15.1:0/58078322 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f97fc006e40 con 0x7f980408a810 2024-03-18T18:50:12.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.001+0000 7f9801d9b700 1 --2- 172.21.15.1:0/58078322 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97e406ad60 0x7f97e406d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:12.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.002+0000 7f9801d9b700 1 --2- 172.21.15.1:0/58078322 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97e406ad60 0x7f97e406d220 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f98040a9bd0 tx=0x7f97f800a040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:13.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.008+0000 7f97f2ffd700 1 -- 172.21.15.1:0/58078322 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f97fc054af0 con 0x7f980408a810 2024-03-18T18:50:13.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.014+0000 7f1e97b2a700 1 Processor -- start 2024-03-18T18:50:13.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.014+0000 7f1e97b2a700 1 -- start start 2024-03-18T18:50:13.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.015+0000 7f1e97b2a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1e9006d250 0x7f1e900b3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:13.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.015+0000 7f1e97b2a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e90070580 con 0x7f1e9014a480 2024-03-18T18:50:13.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.015+0000 7f1e97b2a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e900713e0 con 0x7f1e900b3a90 2024-03-18T18:50:13.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.015+0000 7f1e97b2a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e900707d0 con 0x7f1e9006d250 2024-03-18T18:50:13.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.017+0000 7f1e950c5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1e9006d250 0x7f1e900b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:13.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.017+0000 7f1e950c5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1e9006d250 0x7f1e900b3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45480/0 (socket says 172.21.15.1:45480) 2024-03-18T18:50:13.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.017+0000 7f1e950c5700 1 -- 172.21.15.1:0/1565280817 learned_addr learned my addr 172.21.15.1:0/1565280817 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.031+0000 7f1e948c4700 1 -- 172.21.15.1:0/1565280817 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167437807 0 0) 0x7f1e900713e0 con 0x7f1e900b3a90 2024-03-18T18:50:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.031+0000 7f1e948c4700 1 -- 172.21.15.1:0/1565280817 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e7c0035b0 con 0x7f1e900b3a90 2024-03-18T18:50:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.031+0000 7f1e948c4700 1 -- 172.21.15.1:0/1565280817 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595237063 0 0) 0x7f1e90070580 con 0x7f1e9014a480 2024-03-18T18:50:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.031+0000 7f1e948c4700 1 -- 172.21.15.1:0/1565280817 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e900713e0 con 0x7f1e9014a480 2024-03-18T18:50:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.031+0000 7f1e950c5700 1 -- 172.21.15.1:0/1565280817 >> v1:172.21.15.165:6789/0 conn(0x7f1e900b3a90 legacy=0x7f1e900cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.031+0000 7f1e950c5700 1 -- 172.21.15.1:0/1565280817 >> v1:172.21.15.1:6789/0 conn(0x7f1e9014a480 legacy=0x7f1e9006cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.031+0000 7f1e950c5700 1 -- 172.21.15.1:0/1565280817 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e90070ab0 con 0x7f1e9006d250 2024-03-18T18:50:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.031+0000 7f1e950c5700 1 --2- 172.21.15.1:0/1565280817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e9006d250 0x7f1e900b3550 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1e800098f0 tx=0x7f1e80009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8404c40599b1cd0d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.032+0000 7f1e948c4700 1 -- 172.21.15.1:0/1565280817 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e80011830 con 0x7f1e9006d250 2024-03-18T18:50:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.032+0000 7f1e948c4700 1 -- 172.21.15.1:0/1565280817 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e80011990 con 0x7f1e9006d250 2024-03-18T18:50:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.032+0000 7f1e948c4700 1 -- 172.21.15.1:0/1565280817 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e80011d20 con 0x7f1e9006d250 2024-03-18T18:50:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.033+0000 7f1e97b2a700 1 -- 172.21.15.1:0/1565280817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e9006d250 msgr2=0x7f1e900b3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.033+0000 7f1e97b2a700 1 --2- 172.21.15.1:0/1565280817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e9006d250 0x7f1e900b3550 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1e800098f0 tx=0x7f1e80009c00 comp rx=0 tx=0).stop 2024-03-18T18:50:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.033+0000 7f1e97b2a700 1 -- 172.21.15.1:0/1565280817 shutdown_connections 2024-03-18T18:50:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.033+0000 7f1e97b2a700 1 --2- 172.21.15.1:0/1565280817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e9006d250 0x7f1e900b3550 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1e800098f0 tx=0x7f1e80009c00 comp rx=0 tx=0).stop 2024-03-18T18:50:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.033+0000 7f1e97b2a700 1 -- 172.21.15.1:0/1565280817 >> 172.21.15.1:0/1565280817 conn(0x7f1e9006ab70 msgr2=0x7f1e9006af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.034+0000 7f1e97b2a700 1 -- 172.21.15.1:0/1565280817 shutdown_connections 2024-03-18T18:50:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.034+0000 7f1e97b2a700 1 -- 172.21.15.1:0/1565280817 wait complete. 2024-03-18T18:50:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.036+0000 7f1e97b2a700 1 Processor -- start 2024-03-18T18:50:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.037+0000 7f1e97b2a700 1 -- start start 2024-03-18T18:50:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.038+0000 7f1e97b2a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e900b3a90 0x7f1e901fb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.038+0000 7f1e97b2a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e9014a480 0x7f1e90164070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.038+0000 7f1e97b2a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e901f71d0 0x7f1e900c4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.038+0000 7f1e97b2a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e901f8210 con 0x7f1e9014a480 2024-03-18T18:50:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.038+0000 7f1e97b2a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e901fb6e0 con 0x7f1e900b3a90 2024-03-18T18:50:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.038+0000 7f1e97b2a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e901f7690 con 0x7f1e901f71d0 2024-03-18T18:50:13.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.040+0000 7f1e960c7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e901f71d0 0x7f1e900c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:13.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.040+0000 7f1e958c6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e900b3a90 0x7f1e901fb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:13.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.040+0000 7f1e960c7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e901f71d0 0x7f1e900c4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:60296/0 (socket says 172.21.15.1:60296) 2024-03-18T18:50:13.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.040+0000 7f1e960c7700 1 -- 172.21.15.1:0/1548142473 learned_addr learned my addr 172.21.15.1:0/1548142473 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:13.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.042+0000 7f1e950c5700 1 --2- 172.21.15.1:0/1548142473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e9014a480 0x7f1e90164070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:13.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.044+0000 7f1e958c6700 1 -- 172.21.15.1:0/1548142473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e901f71d0 msgr2=0x7f1e900c4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.044+0000 7f1e958c6700 1 --2- 172.21.15.1:0/1548142473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e901f71d0 0x7f1e900c4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.044+0000 7f1e958c6700 1 -- 172.21.15.1:0/1548142473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e9014a480 msgr2=0x7f1e90164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.044+0000 7f1e958c6700 1 --2- 172.21.15.1:0/1548142473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e9014a480 0x7f1e90164070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.044+0000 7f1e958c6700 1 -- 172.21.15.1:0/1548142473 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e80009550 con 0x7f1e900b3a90 2024-03-18T18:50:13.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.046+0000 7f1e958c6700 1 --2- 172.21.15.1:0/1548142473 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e900b3a90 0x7f1e901fb1a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1e8800cae0 tx=0x7f1e8800cdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:13.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.047+0000 7f1e867fc700 1 -- 172.21.15.1:0/1548142473 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e88003c00 con 0x7f1e900b3a90 2024-03-18T18:50:13.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.047+0000 7f1e867fc700 1 -- 172.21.15.1:0/1548142473 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e88003d80 con 0x7f1e900b3a90 2024-03-18T18:50:13.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.047+0000 7f1e867fc700 1 -- 172.21.15.1:0/1548142473 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8800aae0 con 0x7f1e900b3a90 2024-03-18T18:50:13.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.047+0000 7f1e97b2a700 1 -- 172.21.15.1:0/1548142473 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e901f7970 con 0x7f1e900b3a90 2024-03-18T18:50:13.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.047+0000 7f1e97b2a700 1 -- 172.21.15.1:0/1548142473 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e901f7f30 con 0x7f1e900b3a90 2024-03-18T18:50:13.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.050+0000 7f1e97b2a700 1 -- 172.21.15.1:0/1548142473 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e780052f0 con 0x7f1e900b3a90 2024-03-18T18:50:13.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.051+0000 7f1e867fc700 1 -- 172.21.15.1:0/1548142473 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1e8800ac40 con 0x7f1e900b3a90 2024-03-18T18:50:13.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.052+0000 7f1e867fc700 1 --2- 172.21.15.1:0/1548142473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1e7406ad60 0x7f1e7406d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:13.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.052+0000 7f1e867fc700 1 -- 172.21.15.1:0/1548142473 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1e88089620 con 0x7f1e900b3a90 2024-03-18T18:50:13.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.053+0000 7f1e950c5700 1 --2- 172.21.15.1:0/1548142473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1e7406ad60 0x7f1e7406d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:13.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.054+0000 7f1e950c5700 1 --2- 172.21.15.1:0/1548142473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1e7406ad60 0x7f1e7406d220 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1e90072d30 tx=0x7f1e80008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:13.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.057+0000 7f1e867fc700 1 -- 172.21.15.1:0/1548142473 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1e880544f0 con 0x7f1e900b3a90 2024-03-18T18:50:13.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.214+0000 7fecfb6a0700 1 -- 172.21.15.1:0/5849303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fecf41e1e10 con 0x7fecf40ce450 2024-03-18T18:50:13.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.215+0000 7fece9ffb700 1 -- 172.21.15.1:0/5849303 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fecf41e1e10 con 0x7fecf40ce450 2024-03-18T18:50:13.214 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-18T18:50:13.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.218+0000 7fecfb6a0700 1 -- 172.21.15.1:0/5849303 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fecd806ad60 msgr2=0x7fecd806d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.218+0000 7fecfb6a0700 1 --2- 172.21.15.1:0/5849303 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fecd806ad60 0x7fecd806d220 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fecec0012e0 tx=0x7fecec009040 comp rx=0 tx=0).stop 2024-03-18T18:50:13.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.218+0000 7fecfb6a0700 1 -- 172.21.15.1:0/5849303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecf40ce450 msgr2=0x7fecf40b1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.218+0000 7fecfb6a0700 1 --2- 172.21.15.1:0/5849303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecf40ce450 0x7fecf40b1450 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fece400ba30 tx=0x7fece400bdf0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.219+0000 7fecfb6a0700 1 -- 172.21.15.1:0/5849303 shutdown_connections 2024-03-18T18:50:13.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.219+0000 7fecfb6a0700 1 --2- 172.21.15.1:0/5849303 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fecd806ad60 0x7fecd806d220 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.219+0000 7fecfb6a0700 1 --2- 172.21.15.1:0/5849303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fecf4071df0 0x7fecf4142470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.219+0000 7fecfb6a0700 1 --2- 172.21.15.1:0/5849303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecf40ce450 0x7fecf40b1450 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.219+0000 7fecfb6a0700 1 --2- 172.21.15.1:0/5849303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecf40b2210 0x7fecf40b0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.219+0000 7fecfb6a0700 1 -- 172.21.15.1:0/5849303 >> 172.21.15.1:0/5849303 conn(0x7fecf406a750 msgr2=0x7fecf41dea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:13.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.220+0000 7fecfb6a0700 1 -- 172.21.15.1:0/5849303 shutdown_connections 2024-03-18T18:50:13.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.220+0000 7fecfb6a0700 1 -- 172.21.15.1:0/5849303 wait complete. 2024-03-18T18:50:13.235 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.5 2024-03-18T18:50:13.235 DEBUG:teuthology.parallel:result is None 2024-03-18T18:50:13.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.297+0000 7fce7a288700 1 -- 172.21.15.1:0/3091121682 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fce540055a0 con 0x7fce7414b1b0 2024-03-18T18:50:13.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.298+0000 7fce70ff9700 1 -- 172.21.15.1:0/3091121682 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fce68021090 con 0x7fce7414b1b0 2024-03-18T18:50:13.294 INFO:teuthology.orchestra.run.smithi001.stdout:55834574857 2024-03-18T18:50:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.301+0000 7fce7a288700 1 -- 172.21.15.1:0/3091121682 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce5c06b210 msgr2=0x7fce5c06d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.301+0000 7fce7a288700 1 --2- 172.21.15.1:0/3091121682 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce5c06b210 0x7fce5c06d6d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fce6c000f80 tx=0x7fce6c011000 comp rx=0 tx=0).stop 2024-03-18T18:50:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.301+0000 7fce7a288700 1 -- 172.21.15.1:0/3091121682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce7414b1b0 msgr2=0x7fce74164510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.301+0000 7fce7a288700 1 --2- 172.21.15.1:0/3091121682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce7414b1b0 0x7fce74164510 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fce6800e810 tx=0x7fce6800ebd0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.301+0000 7fce7a288700 1 -- 172.21.15.1:0/3091121682 shutdown_connections 2024-03-18T18:50:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.301+0000 7fce7a288700 1 --2- 172.21.15.1:0/3091121682 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce5c06b210 0x7fce5c06d6d0 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fce6c000f80 tx=0x7fce6c011000 comp rx=0 tx=0).stop 2024-03-18T18:50:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.301+0000 7fce7a288700 1 --2- 172.21.15.1:0/3091121682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce7414b1b0 0x7fce74164510 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.301+0000 7fce7a288700 1 --2- 172.21.15.1:0/3091121682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce7406d020 0x7fce7415b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.301+0000 7fce7a288700 1 --2- 172.21.15.1:0/3091121682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce74070de0 0x7fce74163fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.301+0000 7fce7a288700 1 -- 172.21.15.1:0/3091121682 >> 172.21.15.1:0/3091121682 conn(0x7fce7406a750 msgr2=0x7fce74139570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.302+0000 7fce7a288700 1 -- 172.21.15.1:0/3091121682 shutdown_connections 2024-03-18T18:50:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.302+0000 7fce7a288700 1 -- 172.21.15.1:0/3091121682 wait complete. 2024-03-18T18:50:13.316 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.3 2024-03-18T18:50:13.316 DEBUG:teuthology.parallel:result is None 2024-03-18T18:50:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.360+0000 7f0434b79700 1 -- 172.21.15.1:0/1741387052 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f043013d870 con 0x7f04300b2210 2024-03-18T18:50:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.360+0000 7f1e97b2a700 1 -- 172.21.15.1:0/1548142473 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1e780055a0 con 0x7f1e900b3a90 2024-03-18T18:50:13.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.360+0000 7f041effd700 1 -- 172.21.15.1:0/1741387052 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0424058160 con 0x7f04300b2210 2024-03-18T18:50:13.357 INFO:teuthology.orchestra.run.smithi001.stdout:55834574857 2024-03-18T18:50:13.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.360+0000 7f1e867fc700 1 -- 172.21.15.1:0/1548142473 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1e88057b10 con 0x7f1e900b3a90 2024-03-18T18:50:13.358 INFO:teuthology.orchestra.run.smithi001.stdout:55834574857 2024-03-18T18:50:13.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.362+0000 7f0434b79700 1 -- 172.21.15.1:0/1741387052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f041006ac90 msgr2=0x7f041006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f0434b79700 1 --2- 172.21.15.1:0/1741387052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f041006ac90 0x7f041006d150 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f042800d6c0 tx=0x7f0428012000 comp rx=0 tx=0).stop 2024-03-18T18:50:13.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f0434b79700 1 -- 172.21.15.1:0/1741387052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04300b2210 msgr2=0x7f04300add50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f0434b79700 1 --2- 172.21.15.1:0/1741387052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04300b2210 0x7f04300add50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f042400b470 tx=0x7f042400b780 comp rx=0 tx=0).stop 2024-03-18T18:50:13.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f0434b79700 1 -- 172.21.15.1:0/1741387052 shutdown_connections 2024-03-18T18:50:13.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f0434b79700 1 --2- 172.21.15.1:0/1741387052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f041006ac90 0x7f041006d150 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f042800d6c0 tx=0x7f0428012000 comp rx=0 tx=0).stop 2024-03-18T18:50:13.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f0434b79700 1 --2- 172.21.15.1:0/1741387052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04300ce450 0x7f04301ecfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f0434b79700 1 --2- 172.21.15.1:0/1741387052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0430071df0 0x7f04301528a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f0434b79700 1 --2- 172.21.15.1:0/1741387052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04300b2210 0x7f04300add50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f0434b79700 1 -- 172.21.15.1:0/1741387052 >> 172.21.15.1:0/1741387052 conn(0x7f043006a750 msgr2=0x7f0430158350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:13.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.362+0000 7f1e97b2a700 1 -- 172.21.15.1:0/1548142473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1e7406ad60 msgr2=0x7f1e7406d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f1e97b2a700 1 --2- 172.21.15.1:0/1548142473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1e7406ad60 0x7f1e7406d220 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1e90072d30 tx=0x7f1e80008030 comp rx=0 tx=0).stop 2024-03-18T18:50:13.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f1e97b2a700 1 -- 172.21.15.1:0/1548142473 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e900b3a90 msgr2=0x7f1e901fb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f1e97b2a700 1 --2- 172.21.15.1:0/1548142473 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e900b3a90 0x7f1e901fb1a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1e8800cae0 tx=0x7f1e8800cdf0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f1e97b2a700 1 -- 172.21.15.1:0/1548142473 shutdown_connections 2024-03-18T18:50:13.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f1e97b2a700 1 --2- 172.21.15.1:0/1548142473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1e7406ad60 0x7f1e7406d220 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f1e97b2a700 1 --2- 172.21.15.1:0/1548142473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e901f71d0 0x7f1e900c4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f1e97b2a700 1 --2- 172.21.15.1:0/1548142473 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e900b3a90 0x7f1e901fb1a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f1e97b2a700 1 --2- 172.21.15.1:0/1548142473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e9014a480 0x7f1e90164070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f1e97b2a700 1 -- 172.21.15.1:0/1548142473 >> 172.21.15.1:0/1548142473 conn(0x7f1e9006ab70 msgr2=0x7f1e900cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:13.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f1e97b2a700 1 -- 172.21.15.1:0/1548142473 shutdown_connections 2024-03-18T18:50:13.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f1e97b2a700 1 -- 172.21.15.1:0/1548142473 wait complete. 2024-03-18T18:50:13.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f0434b79700 1 -- 172.21.15.1:0/1741387052 shutdown_connections 2024-03-18T18:50:13.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.363+0000 7f0434b79700 1 -- 172.21.15.1:0/1741387052 wait complete. 2024-03-18T18:50:13.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.364+0000 7fa50ec74700 1 -- 172.21.15.1:0/581034463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa5080ac320 con 0x7fa508070c10 2024-03-18T18:50:13.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.365+0000 7fa5057fa700 1 -- 172.21.15.1:0/581034463 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa500019020 con 0x7fa508070c10 2024-03-18T18:50:13.363 INFO:teuthology.orchestra.run.smithi001.stdout:55834574857 2024-03-18T18:50:13.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.370+0000 7fa50ec74700 1 -- 172.21.15.1:0/581034463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa4f406ad60 msgr2=0x7fa4f406d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.370+0000 7fa50ec74700 1 --2- 172.21.15.1:0/581034463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa4f406ad60 0x7fa4f406d220 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009e50 tx=0x7fa4f8009190 comp rx=0 tx=0).stop 2024-03-18T18:50:13.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.370+0000 7fa50ec74700 1 -- 172.21.15.1:0/581034463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa508070c10 msgr2=0x7fa5080a89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.370+0000 7fa50ec74700 1 --2- 172.21.15.1:0/581034463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa508070c10 0x7fa5080a89e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa50000bd50 tx=0x7fa50000bd80 comp rx=0 tx=0).stop 2024-03-18T18:50:13.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.370+0000 7fa50ec74700 1 -- 172.21.15.1:0/581034463 shutdown_connections 2024-03-18T18:50:13.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.370+0000 7fa50ec74700 1 --2- 172.21.15.1:0/581034463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa4f406ad60 0x7fa4f406d220 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009e50 tx=0x7fa4f8009190 comp rx=0 tx=0).stop 2024-03-18T18:50:13.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.370+0000 7fa50ec74700 1 --2- 172.21.15.1:0/581034463 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa50814b1b0 0x7fa5080a8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.370+0000 7fa50ec74700 1 --2- 172.21.15.1:0/581034463 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa50806d1f0 0x7fa5080a84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.370+0000 7fa50ec74700 1 --2- 172.21.15.1:0/581034463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa508070c10 0x7fa5080a89e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.370+0000 7fa50ec74700 1 -- 172.21.15.1:0/581034463 >> 172.21.15.1:0/581034463 conn(0x7fa50806a750 msgr2=0x7fa5081381b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:13.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.372+0000 7fa50ec74700 1 -- 172.21.15.1:0/581034463 shutdown_connections 2024-03-18T18:50:13.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.372+0000 7fa50ec74700 1 -- 172.21.15.1:0/581034463 wait complete. 2024-03-18T18:50:13.375 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.2 2024-03-18T18:50:13.375 DEBUG:teuthology.parallel:result is None 2024-03-18T18:50:13.377 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.4 2024-03-18T18:50:13.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.380+0000 7f516f7a6700 1 -- 172.21.15.1:0/1607412839 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f5148005870 con 0x7f51680b2210 2024-03-18T18:50:13.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.381+0000 7f515dffb700 1 -- 172.21.15.1:0/1607412839 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5160054460 con 0x7f51680b2210 2024-03-18T18:50:13.377 DEBUG:teuthology.parallel:result is None 2024-03-18T18:50:13.378 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-18T18:50:13.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.383+0000 7f516f7a6700 1 -- 172.21.15.1:0/1607412839 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f515406f300 msgr2=0x7f51540717c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.383+0000 7f516f7a6700 1 --2- 172.21.15.1:0/1607412839 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f515406f300 0x7f51540717c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f516400f240 tx=0x7f5164008040 comp rx=0 tx=0).stop 2024-03-18T18:50:13.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.383+0000 7f516f7a6700 1 -- 172.21.15.1:0/1607412839 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51680b2210 msgr2=0x7f5168139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.383+0000 7f516f7a6700 1 --2- 172.21.15.1:0/1607412839 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51680b2210 0x7f5168139fa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f51680ce870 tx=0x7f516000cd50 comp rx=0 tx=0).stop 2024-03-18T18:50:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.383+0000 7f516f7a6700 1 -- 172.21.15.1:0/1607412839 shutdown_connections 2024-03-18T18:50:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.383+0000 7f516f7a6700 1 --2- 172.21.15.1:0/1607412839 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f515406f300 0x7f51540717c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.383+0000 7f516f7a6700 1 --2- 172.21.15.1:0/1607412839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51680ce450 0x7f516814d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.383+0000 7f516f7a6700 1 --2- 172.21.15.1:0/1607412839 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5168071df0 0x7f51680aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.383+0000 7f516f7a6700 1 --2- 172.21.15.1:0/1607412839 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51680b2210 0x7f5168139fa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.383+0000 7f516f7a6700 1 -- 172.21.15.1:0/1607412839 >> 172.21.15.1:0/1607412839 conn(0x7f516806a750 msgr2=0x7f5168070580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.383+0000 7f516f7a6700 1 -- 172.21.15.1:0/1607412839 shutdown_connections 2024-03-18T18:50:13.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.383+0000 7f516f7a6700 1 -- 172.21.15.1:0/1607412839 wait complete. 2024-03-18T18:50:13.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.391+0000 7faba1b18700 1 -- 172.21.15.1:0/1877649225 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fab9c0cc8b0 con 0x7fab9c06d250 2024-03-18T18:50:13.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.392+0000 7fab93fff700 1 -- 172.21.15.1:0/1877649225 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fab8c017070 con 0x7fab9c06d250 2024-03-18T18:50:13.388 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-18T18:50:13.389 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.0 2024-03-18T18:50:13.389 DEBUG:teuthology.parallel:result is None 2024-03-18T18:50:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.394+0000 7faba1b18700 1 -- 172.21.15.1:0/1877649225 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fab8406ac90 msgr2=0x7fab8406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.394+0000 7faba1b18700 1 --2- 172.21.15.1:0/1877649225 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fab8406ac90 0x7fab8406d150 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fab9c072d30 tx=0x7fab8801c040 comp rx=0 tx=0).stop 2024-03-18T18:50:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.394+0000 7faba1b18700 1 -- 172.21.15.1:0/1877649225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab9c06d250 msgr2=0x7fab9c2028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.394+0000 7faba1b18700 1 --2- 172.21.15.1:0/1877649225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab9c06d250 0x7fab9c2028a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fab8c00c520 tx=0x7fab8c00c830 comp rx=0 tx=0).stop 2024-03-18T18:50:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.394+0000 7faba1b18700 1 -- 172.21.15.1:0/1877649225 shutdown_connections 2024-03-18T18:50:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.394+0000 7faba1b18700 1 --2- 172.21.15.1:0/1877649225 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fab8406ac90 0x7fab8406d150 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.394+0000 7faba1b18700 1 --2- 172.21.15.1:0/1877649225 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fab9c14a480 0x7fab9c149550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.394+0000 7faba1b18700 1 --2- 172.21.15.1:0/1877649225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fab9c06d250 0x7fab9c2028a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.394+0000 7faba1b18700 1 --2- 172.21.15.1:0/1877649225 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fab9c0b3a90 0x7fab9c149010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.394+0000 7faba1b18700 1 -- 172.21.15.1:0/1877649225 >> 172.21.15.1:0/1877649225 conn(0x7fab9c06ab70 msgr2=0x7fab9c164580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:13.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.394+0000 7faba1b18700 1 -- 172.21.15.1:0/1877649225 shutdown_connections 2024-03-18T18:50:13.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.394+0000 7faba1b18700 1 -- 172.21.15.1:0/1877649225 wait complete. 2024-03-18T18:50:13.393 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.7 2024-03-18T18:50:13.393 DEBUG:teuthology.parallel:result is None 2024-03-18T18:50:13.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.406+0000 7f9808bf5700 1 -- 172.21.15.1:0/58078322 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9804071530 con 0x7f980408a810 2024-03-18T18:50:13.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.406+0000 7f97f2ffd700 1 -- 172.21.15.1:0/58078322 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f97fc023020 con 0x7f980408a810 2024-03-18T18:50:13.403 INFO:teuthology.orchestra.run.smithi001.stdout:55834574856 2024-03-18T18:50:13.404 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.6 2024-03-18T18:50:13.404 DEBUG:teuthology.parallel:result is None 2024-03-18T18:50:13.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.408+0000 7f9808bf5700 1 -- 172.21.15.1:0/58078322 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97e406ad60 msgr2=0x7f97e406d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.408+0000 7f9808bf5700 1 --2- 172.21.15.1:0/58078322 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97e406ad60 0x7f97e406d220 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f98040a9bd0 tx=0x7f97f800a040 comp rx=0 tx=0).stop 2024-03-18T18:50:13.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.408+0000 7f9808bf5700 1 -- 172.21.15.1:0/58078322 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f980408a810 msgr2=0x7f98040acfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.408+0000 7f9808bf5700 1 --2- 172.21.15.1:0/58078322 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f980408a810 0x7f98040acfc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f97fc00ba50 tx=0x7f97fc00be10 comp rx=0 tx=0).stop 2024-03-18T18:50:13.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.408+0000 7f9808bf5700 1 -- 172.21.15.1:0/58078322 shutdown_connections 2024-03-18T18:50:13.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.408+0000 7f9808bf5700 1 --2- 172.21.15.1:0/58078322 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97e406ad60 0x7f97e406d220 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.408+0000 7f9808bf5700 1 --2- 172.21.15.1:0/58078322 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f980416b110 0x7f98040aa8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.408+0000 7f9808bf5700 1 --2- 172.21.15.1:0/58078322 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f980408a810 0x7f98040acfc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.408+0000 7f9808bf5700 1 --2- 172.21.15.1:0/58078322 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9804160e90 0x7f98040aa3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.408+0000 7f9808bf5700 1 -- 172.21.15.1:0/58078322 >> 172.21.15.1:0/58078322 conn(0x7f980406a750 msgr2=0x7f98040b0390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:13.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.408+0000 7f9808bf5700 1 -- 172.21.15.1:0/58078322 shutdown_connections 2024-03-18T18:50:13.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.408+0000 7f9808bf5700 1 -- 172.21.15.1:0/58078322 wait complete. 2024-03-18T18:50:13.417 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.1 2024-03-18T18:50:13.417 DEBUG:teuthology.parallel:result is None 2024-03-18T18:50:13.417 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-18T18:50:13.417 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T18:50:13.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.586+0000 7f90ddd1f700 1 Processor -- start 2024-03-18T18:50:13.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.586+0000 7f90ddd1f700 1 -- start start 2024-03-18T18:50:13.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.587+0000 7f90ddd1f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f90d80af610 0x7f90d80cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:13.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.587+0000 7f90ddd1f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f90d80ccf40 0x7f90d8071a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:13.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.587+0000 7f90ddd1f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d80cf090 con 0x7f90d8144b70 2024-03-18T18:50:13.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.587+0000 7f90ddd1f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d8144f50 con 0x7f90d80ccf40 2024-03-18T18:50:13.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.587+0000 7f90ddd1f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d80cd3b0 con 0x7f90d80af610 2024-03-18T18:50:13.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.587+0000 7f90d7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f90d80ccf40 0x7f90d8071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:13.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.587+0000 7f90d7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f90d80ccf40 0x7f90d8071a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45494/0 (socket says 172.21.15.1:45494) 2024-03-18T18:50:13.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.587+0000 7f90d7fff700 1 -- 172.21.15.1:0/2922123964 learned_addr learned my addr 172.21.15.1:0/2922123964 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:13.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.587+0000 7f90d6ffd700 1 --2- 172.21.15.1:0/2922123964 >> v2:172.21.15.165:3300/0 conn(0x7f90d80af610 0x7f90d80cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:13.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.588+0000 7f90d7fff700 1 -- 172.21.15.1:0/2922123964 >> v2:172.21.15.165:3300/0 conn(0x7f90d80af610 msgr2=0x7f90d80cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:13.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.588+0000 7f90d7fff700 1 --2- 172.21.15.1:0/2922123964 >> v2:172.21.15.165:3300/0 conn(0x7f90d80af610 0x7f90d80cca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.588+0000 7f90d7fff700 1 -- 172.21.15.1:0/2922123964 >> v1:172.21.15.165:6789/0 conn(0x7f90d8144b70 legacy=0x7f90d80ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.588+0000 7f90d7fff700 1 -- 172.21.15.1:0/2922123964 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d80cf2d0 con 0x7f90d80ccf40 2024-03-18T18:50:13.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.588+0000 7f90d7fff700 1 --2- 172.21.15.1:0/2922123964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90d80ccf40 0x7f90d8071a10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f90d80b6230 tx=0x7f90d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e93a18f4a594158b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:13.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.588+0000 7f90d67fc700 1 -- 172.21.15.1:0/2922123964 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d000e070 con 0x7f90d80ccf40 2024-03-18T18:50:13.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.588+0000 7f90d67fc700 1 -- 172.21.15.1:0/2922123964 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d000b910 con 0x7f90d80ccf40 2024-03-18T18:50:13.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.588+0000 7f90d67fc700 1 -- 172.21.15.1:0/2922123964 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d000bcc0 con 0x7f90d80ccf40 2024-03-18T18:50:13.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.588+0000 7f90ddd1f700 1 -- 172.21.15.1:0/2922123964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90d80ccf40 msgr2=0x7f90d8071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.588+0000 7f90ddd1f700 1 --2- 172.21.15.1:0/2922123964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90d80ccf40 0x7f90d8071a10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f90d80b6230 tx=0x7f90d000b330 comp rx=0 tx=0).stop 2024-03-18T18:50:13.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.588+0000 7f90ddd1f700 1 -- 172.21.15.1:0/2922123964 shutdown_connections 2024-03-18T18:50:13.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.588+0000 7f90ddd1f700 1 --2- 172.21.15.1:0/2922123964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90d80ccf40 0x7f90d8071a10 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.588+0000 7f90ddd1f700 1 --2- 172.21.15.1:0/2922123964 >> v2:172.21.15.165:3300/0 conn(0x7f90d80af610 0x7f90d80cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.588+0000 7f90ddd1f700 1 -- 172.21.15.1:0/2922123964 >> 172.21.15.1:0/2922123964 conn(0x7f90d80b1d20 msgr2=0x7f90d80b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:13.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.588+0000 7f90ddd1f700 1 -- 172.21.15.1:0/2922123964 shutdown_connections 2024-03-18T18:50:13.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.588+0000 7f90ddd1f700 1 -- 172.21.15.1:0/2922123964 wait complete. 2024-03-18T18:50:13.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.590+0000 7f90ddd1f700 1 Processor -- start 2024-03-18T18:50:13.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.590+0000 7f90ddd1f700 1 -- start start 2024-03-18T18:50:13.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.590+0000 7f90ddd1f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90d80af610 0x7f90d813e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:13.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90ddd1f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90d80ccf40 0x7f90d8138080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:13.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90ddd1f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90d8144b70 0x7f90d81377d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:13.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90ddd1f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d81385c0 con 0x7f90d80af610 2024-03-18T18:50:13.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90ddd1f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d8137d10 con 0x7f90d80ccf40 2024-03-18T18:50:13.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90ddd1f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d813dbf0 con 0x7f90d8144b70 2024-03-18T18:50:13.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90d77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90d80af610 0x7f90d813e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:13.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90d77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90d80af610 0x7f90d813e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45508/0 (socket says 172.21.15.1:45508) 2024-03-18T18:50:13.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90d77fe700 1 -- 172.21.15.1:0/1623485731 learned_addr learned my addr 172.21.15.1:0/1623485731 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:13.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90d6ffd700 1 --2- 172.21.15.1:0/1623485731 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90d80ccf40 0x7f90d8138080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:13.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90d77fe700 1 -- 172.21.15.1:0/1623485731 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90d8144b70 msgr2=0x7f90d81377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90d7fff700 1 --2- 172.21.15.1:0/1623485731 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90d8144b70 0x7f90d81377d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:13.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90d77fe700 1 --2- 172.21.15.1:0/1623485731 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90d8144b70 0x7f90d81377d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90d77fe700 1 -- 172.21.15.1:0/1623485731 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90d80ccf40 msgr2=0x7f90d8138080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90d77fe700 1 --2- 172.21.15.1:0/1623485731 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90d80ccf40 0x7f90d8138080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90d77fe700 1 -- 172.21.15.1:0/1623485731 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d0009b30 con 0x7f90d80af610 2024-03-18T18:50:13.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90d77fe700 1 --2- 172.21.15.1:0/1623485731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90d80af610 0x7f90d813e170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f90c800c430 tx=0x7f90c800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:13.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90cffff700 1 -- 172.21.15.1:0/1623485731 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c80052d0 con 0x7f90d80af610 2024-03-18T18:50:13.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90cffff700 1 -- 172.21.15.1:0/1623485731 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c8005450 con 0x7f90d80af610 2024-03-18T18:50:13.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90ddd1f700 1 -- 172.21.15.1:0/1623485731 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d813de90 con 0x7f90d80af610 2024-03-18T18:50:13.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90cffff700 1 -- 172.21.15.1:0/1623485731 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c8006aa0 con 0x7f90d80af610 2024-03-18T18:50:13.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.591+0000 7f90ddd1f700 1 -- 172.21.15.1:0/1623485731 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d813e7e0 con 0x7f90d80af610 2024-03-18T18:50:13.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.592+0000 7f90ddd1f700 1 -- 172.21.15.1:0/1623485731 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d80cda90 con 0x7f90d80af610 2024-03-18T18:50:13.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.593+0000 7f90cffff700 1 -- 172.21.15.1:0/1623485731 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f90c8003680 con 0x7f90d80af610 2024-03-18T18:50:13.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.593+0000 7f90cffff700 1 --2- 172.21.15.1:0/1623485731 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f90b806ac90 0x7f90b806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:13.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.593+0000 7f90cffff700 1 -- 172.21.15.1:0/1623485731 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f90c8088630 con 0x7f90d80af610 2024-03-18T18:50:13.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.593+0000 7f90d6ffd700 1 --2- 172.21.15.1:0/1623485731 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f90b806ac90 0x7f90b806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:13.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.594+0000 7f90d6ffd700 1 --2- 172.21.15.1:0/1623485731 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f90b806ac90 0x7f90b806d150 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f90c0009ab0 tx=0x7f90c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:13.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.598+0000 7f90cffff700 1 -- 172.21.15.1:0/1623485731 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f90c80535b0 con 0x7f90d80af610 2024-03-18T18:50:13.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.797+0000 7f90ddd1f700 1 -- 172.21.15.1:0/1623485731 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f90d80cfb70 con 0x7f90b806ac90 2024-03-18T18:50:13.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.806+0000 7f90cffff700 1 -- 172.21.15.1:0/1623485731 <== mgr.4104 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345246 (secure 0 0 0) 0x7f90c00106e0 con 0x7f90b806ac90 2024-03-18T18:50:13.807 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:13.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.810+0000 7f90ddd1f700 1 -- 172.21.15.1:0/1623485731 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f90b806ac90 msgr2=0x7f90b806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.810+0000 7f90ddd1f700 1 --2- 172.21.15.1:0/1623485731 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f90b806ac90 0x7f90b806d150 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f90c0009ab0 tx=0x7f90c0008040 comp rx=0 tx=0).stop 2024-03-18T18:50:13.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.810+0000 7f90ddd1f700 1 -- 172.21.15.1:0/1623485731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90d80af610 msgr2=0x7f90d813e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.810+0000 7f90ddd1f700 1 --2- 172.21.15.1:0/1623485731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90d80af610 0x7f90d813e170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f90c800c430 tx=0x7f90c800c7f0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.810+0000 7f90ddd1f700 1 -- 172.21.15.1:0/1623485731 shutdown_connections 2024-03-18T18:50:13.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.810+0000 7f90ddd1f700 1 --2- 172.21.15.1:0/1623485731 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f90b806ac90 0x7f90b806d150 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.810+0000 7f90ddd1f700 1 --2- 172.21.15.1:0/1623485731 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90d8144b70 0x7f90d81377d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.810+0000 7f90ddd1f700 1 --2- 172.21.15.1:0/1623485731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90d80af610 0x7f90d813e170 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.810+0000 7f90ddd1f700 1 --2- 172.21.15.1:0/1623485731 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90d80ccf40 0x7f90d8138080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.810+0000 7f90ddd1f700 1 -- 172.21.15.1:0/1623485731 >> 172.21.15.1:0/1623485731 conn(0x7f90d80b1d20 msgr2=0x7f90d81436d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:13.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.810+0000 7f90ddd1f700 1 -- 172.21.15.1:0/1623485731 shutdown_connections 2024-03-18T18:50:13.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.810+0000 7f90ddd1f700 1 -- 172.21.15.1:0/1623485731 wait complete. 2024-03-18T18:50:13.810 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T18:50:13.818 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-03-18T18:50:13.394519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594961,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219464,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755256,"statfs":{"total":773094113280,"available":772869382144,"internally_reserved":0,"allocated":3383296,"data_stored":2025530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003752"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976876+0000","last_change":"2024-03-18T18:49:44.831395+0000","last_active":"2024-03-18T18:49:48.976876+0000","last_peered":"2024-03-18T18:49:48.976876+0000","last_clean":"2024-03-18T18:49:48.976876+0000","last_became_active":"2024-03-18T18:49:44.830706+0000","last_became_peered":"2024-03-18T18:49:44.830706+0000","last_unstale":"2024-03-18T18:49:48.976876+0000","last_undegraded":"2024-03-18T18:49:48.976876+0000","last_fullsized":"2024-03-18T18:49:48.976876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:20:43.395005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841775+0000","last_change":"2024-03-18T18:49:44.829492+0000","last_active":"2024-03-18T18:49:48.841775+0000","last_peered":"2024-03-18T18:49:48.841775+0000","last_clean":"2024-03-18T18:49:48.841775+0000","last_became_active":"2024-03-18T18:49:44.829223+0000","last_became_peered":"2024-03-18T18:49:44.829223+0000","last_unstale":"2024-03-18T18:49:48.841775+0000","last_undegraded":"2024-03-18T18:49:48.841775+0000","last_fullsized":"2024-03-18T18:49:48.841775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.845869+0000","last_change":"2024-03-18T18:49:44.831909+0000","last_active":"2024-03-18T18:49:48.845869+0000","last_peered":"2024-03-18T18:49:48.845869+0000","last_clean":"2024-03-18T18:49:48.845869+0000","last_became_active":"2024-03-18T18:49:44.831661+0000","last_became_peered":"2024-03-18T18:49:44.831661+0000","last_unstale":"2024-03-18T18:49:48.845869+0000","last_undegraded":"2024-03-18T18:49:48.845869+0000","last_fullsized":"2024-03-18T18:49:48.845869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:14.055199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840094+0000","last_change":"2024-03-18T18:49:44.838298+0000","last_active":"2024-03-18T18:49:48.840094+0000","last_peered":"2024-03-18T18:49:48.840094+0000","last_clean":"2024-03-18T18:49:48.840094+0000","last_became_active":"2024-03-18T18:49:44.837689+0000","last_became_peered":"2024-03-18T18:49:44.837689+0000","last_unstale":"2024-03-18T18:49:48.840094+0000","last_undegraded":"2024-03-18T18:49:48.840094+0000","last_fullsized":"2024-03-18T18:49:48.840094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:50.238592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975645+0000","last_change":"2024-03-18T18:49:44.841081+0000","last_active":"2024-03-18T18:49:48.975645+0000","last_peered":"2024-03-18T18:49:48.975645+0000","last_clean":"2024-03-18T18:49:48.975645+0000","last_became_active":"2024-03-18T18:49:44.840798+0000","last_became_peered":"2024-03-18T18:49:44.840798+0000","last_unstale":"2024-03-18T18:49:48.975645+0000","last_undegraded":"2024-03-18T18:49:48.975645+0000","last_fullsized":"2024-03-18T18:49:48.975645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:27:55.580010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839590+0000","last_change":"2024-03-18T18:49:44.836710+0000","last_active":"2024-03-18T18:49:48.839590+0000","last_peered":"2024-03-18T18:49:48.839590+0000","last_clean":"2024-03-18T18:49:48.839590+0000","last_became_active":"2024-03-18T18:49:44.836473+0000","last_became_peered":"2024-03-18T18:49:44.836473+0000","last_unstale":"2024-03-18T18:49:48.839590+0000","last_undegraded":"2024-03-18T18:49:48.839590+0000","last_fullsized":"2024-03-18T18:49:48.839590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:59:20.616256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976795+0000","last_change":"2024-03-18T18:49:44.829680+0000","last_active":"2024-03-18T18:49:48.976795+0000","last_peered":"2024-03-18T18:49:48.976795+0000","last_clean":"2024-03-18T18:49:48.976795+0000","last_became_active":"2024-03-18T18:49:44.829139+0000","last_became_peered":"2024-03-18T18:49:44.829139+0000","last_unstale":"2024-03-18T18:49:48.976795+0000","last_undegraded":"2024-03-18T18:49:48.976795+0000","last_fullsized":"2024-03-18T18:49:48.976795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:59:20.616256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839889+0000","last_change":"2024-03-18T18:49:44.837035+0000","last_active":"2024-03-18T18:49:48.839889+0000","last_peered":"2024-03-18T18:49:48.839889+0000","last_clean":"2024-03-18T18:49:48.839889+0000","last_became_active":"2024-03-18T18:49:44.836598+0000","last_became_peered":"2024-03-18T18:49:44.836598+0000","last_unstale":"2024-03-18T18:49:48.839889+0000","last_undegraded":"2024-03-18T18:49:48.839889+0000","last_fullsized":"2024-03-18T18:49:48.839889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:50.238592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976827+0000","last_change":"2024-03-18T18:49:44.835014+0000","last_active":"2024-03-18T18:49:48.976827+0000","last_peered":"2024-03-18T18:49:48.976827+0000","last_clean":"2024-03-18T18:49:48.976827+0000","last_became_active":"2024-03-18T18:49:44.834438+0000","last_became_peered":"2024-03-18T18:49:44.834438+0000","last_unstale":"2024-03-18T18:49:48.976827+0000","last_undegraded":"2024-03-18T18:49:48.976827+0000","last_fullsized":"2024-03-18T18:49:48.976827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:07.013090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835727+0000","last_change":"2024-03-18T18:49:44.837039+0000","last_active":"2024-03-18T18:49:48.835727+0000","last_peered":"2024-03-18T18:49:48.835727+0000","last_clean":"2024-03-18T18:49:48.835727+0000","last_became_active":"2024-03-18T18:49:44.836749+0000","last_became_peered":"2024-03-18T18:49:44.836749+0000","last_unstale":"2024-03-18T18:49:48.835727+0000","last_undegraded":"2024-03-18T18:49:48.835727+0000","last_fullsized":"2024-03-18T18:49:48.835727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836415+0000","last_change":"2024-03-18T18:49:44.832546+0000","last_active":"2024-03-18T18:49:48.836415+0000","last_peered":"2024-03-18T18:49:48.836415+0000","last_clean":"2024-03-18T18:49:48.836415+0000","last_became_active":"2024-03-18T18:49:44.832333+0000","last_became_peered":"2024-03-18T18:49:44.832333+0000","last_unstale":"2024-03-18T18:49:48.836415+0000","last_undegraded":"2024-03-18T18:49:48.836415+0000","last_fullsized":"2024-03-18T18:49:48.836415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842011+0000","last_change":"2024-03-18T18:49:44.834366+0000","last_active":"2024-03-18T18:49:48.842011+0000","last_peered":"2024-03-18T18:49:48.842011+0000","last_clean":"2024-03-18T18:49:48.842011+0000","last_became_active":"2024-03-18T18:49:44.833912+0000","last_became_peered":"2024-03-18T18:49:44.833912+0000","last_unstale":"2024-03-18T18:49:48.842011+0000","last_undegraded":"2024-03-18T18:49:48.842011+0000","last_fullsized":"2024-03-18T18:49:48.842011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:09:08.248420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839130+0000","last_change":"2024-03-18T18:49:44.823814+0000","last_active":"2024-03-18T18:49:48.839130+0000","last_peered":"2024-03-18T18:49:48.839130+0000","last_clean":"2024-03-18T18:49:48.839130+0000","last_became_active":"2024-03-18T18:49:44.823554+0000","last_became_peered":"2024-03-18T18:49:44.823554+0000","last_unstale":"2024-03-18T18:49:48.839130+0000","last_undegraded":"2024-03-18T18:49:48.839130+0000","last_fullsized":"2024-03-18T18:49:48.839130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:05.661348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853566+0000","last_change":"2024-03-18T18:49:44.828163+0000","last_active":"2024-03-18T18:49:48.853566+0000","last_peered":"2024-03-18T18:49:48.853566+0000","last_clean":"2024-03-18T18:49:48.853566+0000","last_became_active":"2024-03-18T18:49:44.827352+0000","last_became_peered":"2024-03-18T18:49:44.827352+0000","last_unstale":"2024-03-18T18:49:48.853566+0000","last_undegraded":"2024-03-18T18:49:48.853566+0000","last_fullsized":"2024-03-18T18:49:48.853566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:30.009397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840573+0000","last_change":"2024-03-18T18:49:44.839200+0000","last_active":"2024-03-18T18:49:48.840573+0000","last_peered":"2024-03-18T18:49:48.840573+0000","last_clean":"2024-03-18T18:49:48.840573+0000","last_became_active":"2024-03-18T18:49:44.838967+0000","last_became_peered":"2024-03-18T18:49:44.838967+0000","last_unstale":"2024-03-18T18:49:48.840573+0000","last_undegraded":"2024-03-18T18:49:48.840573+0000","last_fullsized":"2024-03-18T18:49:48.840573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:14.055199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839664+0000","last_change":"2024-03-18T18:49:44.834022+0000","last_active":"2024-03-18T18:49:48.839664+0000","last_peered":"2024-03-18T18:49:48.839664+0000","last_clean":"2024-03-18T18:49:48.839664+0000","last_became_active":"2024-03-18T18:49:44.833758+0000","last_became_peered":"2024-03-18T18:49:44.833758+0000","last_unstale":"2024-03-18T18:49:48.839664+0000","last_undegraded":"2024-03-18T18:49:48.839664+0000","last_fullsized":"2024-03-18T18:49:48.839664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:25:14.491414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840752+0000","last_change":"2024-03-18T18:49:44.842296+0000","last_active":"2024-03-18T18:49:48.840752+0000","last_peered":"2024-03-18T18:49:48.840752+0000","last_clean":"2024-03-18T18:49:48.840752+0000","last_became_active":"2024-03-18T18:49:44.841720+0000","last_became_peered":"2024-03-18T18:49:44.841720+0000","last_unstale":"2024-03-18T18:49:48.840752+0000","last_undegraded":"2024-03-18T18:49:48.840752+0000","last_fullsized":"2024-03-18T18:49:48.840752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836699+0000","last_change":"2024-03-18T18:49:44.828352+0000","last_active":"2024-03-18T18:49:48.836699+0000","last_peered":"2024-03-18T18:49:48.836699+0000","last_clean":"2024-03-18T18:49:48.836699+0000","last_became_active":"2024-03-18T18:49:44.828059+0000","last_became_peered":"2024-03-18T18:49:44.828059+0000","last_unstale":"2024-03-18T18:49:48.836699+0000","last_undegraded":"2024-03-18T18:49:48.836699+0000","last_fullsized":"2024-03-18T18:49:48.836699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:29.829759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976858+0000","last_change":"2024-03-18T18:49:44.841034+0000","last_active":"2024-03-18T18:49:48.976858+0000","last_peered":"2024-03-18T18:49:48.976858+0000","last_clean":"2024-03-18T18:49:48.976858+0000","last_became_active":"2024-03-18T18:49:44.840765+0000","last_became_peered":"2024-03-18T18:49:44.840765+0000","last_unstale":"2024-03-18T18:49:48.976858+0000","last_undegraded":"2024-03-18T18:49:48.976858+0000","last_fullsized":"2024-03-18T18:49:48.976858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:34.102274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.982406+0000","last_change":"2024-03-18T18:49:48.981322+0000","last_active":"2024-03-18T18:49:48.982406+0000","last_peered":"2024-03-18T18:49:48.982406+0000","last_clean":"2024-03-18T18:49:48.982406+0000","last_became_active":"2024-03-18T18:49:48.980927+0000","last_became_peered":"2024-03-18T18:49:48.980927+0000","last_unstale":"2024-03-18T18:49:48.982406+0000","last_undegraded":"2024-03-18T18:49:48.982406+0000","last_fullsized":"2024-03-18T18:49:48.982406+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:57:44.834422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.847587+0000","last_change":"2024-03-18T18:49:44.396090+0000","last_active":"2024-03-18T18:49:48.847587+0000","last_peered":"2024-03-18T18:49:48.847587+0000","last_clean":"2024-03-18T18:49:48.847587+0000","last_became_active":"2024-03-18T18:49:44.395823+0000","last_became_peered":"2024-03-18T18:49:44.395823+0000","last_unstale":"2024-03-18T18:49:48.847587+0000","last_undegraded":"2024-03-18T18:49:48.847587+0000","last_fullsized":"2024-03-18T18:49:48.847587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:00:40.075042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840491+0000","last_change":"2024-03-18T18:49:44.821381+0000","last_active":"2024-03-18T18:49:48.840491+0000","last_peered":"2024-03-18T18:49:48.840491+0000","last_clean":"2024-03-18T18:49:48.840491+0000","last_became_active":"2024-03-18T18:49:44.821129+0000","last_became_peered":"2024-03-18T18:49:44.821129+0000","last_unstale":"2024-03-18T18:49:48.840491+0000","last_undegraded":"2024-03-18T18:49:48.840491+0000","last_fullsized":"2024-03-18T18:49:48.840491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853754+0000","last_change":"2024-03-18T18:49:48.852436+0000","last_active":"2024-03-18T18:49:48.853754+0000","last_peered":"2024-03-18T18:49:48.853754+0000","last_clean":"2024-03-18T18:49:48.853754+0000","last_became_active":"2024-03-18T18:49:48.851959+0000","last_became_peered":"2024-03-18T18:49:48.851959+0000","last_unstale":"2024-03-18T18:49:48.853754+0000","last_undegraded":"2024-03-18T18:49:48.853754+0000","last_fullsized":"2024-03-18T18:49:48.853754+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:31:52.237923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840602+0000","last_change":"2024-03-18T18:49:44.399646+0000","last_active":"2024-03-18T18:49:48.840602+0000","last_peered":"2024-03-18T18:49:48.840602+0000","last_clean":"2024-03-18T18:49:48.840602+0000","last_became_active":"2024-03-18T18:49:44.399368+0000","last_became_peered":"2024-03-18T18:49:44.399368+0000","last_unstale":"2024-03-18T18:49:48.840602+0000","last_undegraded":"2024-03-18T18:49:48.840602+0000","last_fullsized":"2024-03-18T18:49:48.840602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:29.763642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.847482+0000","last_change":"2024-03-18T18:49:44.832977+0000","last_active":"2024-03-18T18:49:48.847482+0000","last_peered":"2024-03-18T18:49:48.847482+0000","last_clean":"2024-03-18T18:49:48.847482+0000","last_became_active":"2024-03-18T18:49:44.832624+0000","last_became_peered":"2024-03-18T18:49:44.832624+0000","last_unstale":"2024-03-18T18:49:48.847482+0000","last_undegraded":"2024-03-18T18:49:48.847482+0000","last_fullsized":"2024-03-18T18:49:48.847482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853697+0000","last_change":"2024-03-18T18:49:48.852585+0000","last_active":"2024-03-18T18:49:48.853697+0000","last_peered":"2024-03-18T18:49:48.853697+0000","last_clean":"2024-03-18T18:49:48.853697+0000","last_became_active":"2024-03-18T18:49:48.852137+0000","last_became_peered":"2024-03-18T18:49:48.852137+0000","last_unstale":"2024-03-18T18:49:48.853697+0000","last_undegraded":"2024-03-18T18:49:48.853697+0000","last_fullsized":"2024-03-18T18:49:48.853697+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:50:12.397097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975894+0000","last_change":"2024-03-18T18:49:44.394010+0000","last_active":"2024-03-18T18:49:48.975894+0000","last_peered":"2024-03-18T18:49:48.975894+0000","last_clean":"2024-03-18T18:49:48.975894+0000","last_became_active":"2024-03-18T18:49:44.393512+0000","last_became_peered":"2024-03-18T18:49:44.393512+0000","last_unstale":"2024-03-18T18:49:48.975894+0000","last_undegraded":"2024-03-18T18:49:48.975894+0000","last_fullsized":"2024-03-18T18:49:48.975894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:10:24.082597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839873+0000","last_change":"2024-03-18T18:49:44.836167+0000","last_active":"2024-03-18T18:49:48.839873+0000","last_peered":"2024-03-18T18:49:48.839873+0000","last_clean":"2024-03-18T18:49:48.839873+0000","last_became_active":"2024-03-18T18:49:44.835887+0000","last_became_peered":"2024-03-18T18:49:44.835887+0000","last_unstale":"2024-03-18T18:49:48.839873+0000","last_undegraded":"2024-03-18T18:49:48.839873+0000","last_fullsized":"2024-03-18T18:49:48.839873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:34.102274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.857961+0000","last_change":"2024-03-18T18:49:48.857276+0000","last_active":"2024-03-18T18:49:48.857961+0000","last_peered":"2024-03-18T18:49:48.857961+0000","last_clean":"2024-03-18T18:49:48.857961+0000","last_became_active":"2024-03-18T18:49:48.856870+0000","last_became_peered":"2024-03-18T18:49:48.856870+0000","last_unstale":"2024-03-18T18:49:48.857961+0000","last_undegraded":"2024-03-18T18:49:48.857961+0000","last_fullsized":"2024-03-18T18:49:48.857961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:31.426438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975669+0000","last_change":"2024-03-18T18:49:44.401760+0000","last_active":"2024-03-18T18:49:48.975669+0000","last_peered":"2024-03-18T18:49:48.975669+0000","last_clean":"2024-03-18T18:49:48.975669+0000","last_became_active":"2024-03-18T18:49:44.401565+0000","last_became_peered":"2024-03-18T18:49:44.401565+0000","last_unstale":"2024-03-18T18:49:48.975669+0000","last_undegraded":"2024-03-18T18:49:48.975669+0000","last_fullsized":"2024-03-18T18:49:48.975669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:09:07.245729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840474+0000","last_change":"2024-03-18T18:49:44.834985+0000","last_active":"2024-03-18T18:49:48.840474+0000","last_peered":"2024-03-18T18:49:48.840474+0000","last_clean":"2024-03-18T18:49:48.840474+0000","last_became_active":"2024-03-18T18:49:44.834719+0000","last_became_peered":"2024-03-18T18:49:44.834719+0000","last_unstale":"2024-03-18T18:49:48.840474+0000","last_undegraded":"2024-03-18T18:49:48.840474+0000","last_fullsized":"2024-03-18T18:49:48.840474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:27:55.580010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.854528+0000","last_change":"2024-03-18T18:49:48.852823+0000","last_active":"2024-03-18T18:49:48.854528+0000","last_peered":"2024-03-18T18:49:48.854528+0000","last_clean":"2024-03-18T18:49:48.854528+0000","last_became_active":"2024-03-18T18:49:48.852431+0000","last_became_peered":"2024-03-18T18:49:48.852431+0000","last_unstale":"2024-03-18T18:49:48.854528+0000","last_undegraded":"2024-03-18T18:49:48.854528+0000","last_fullsized":"2024-03-18T18:49:48.854528+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:10:04.268147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839292+0000","last_change":"2024-03-18T18:49:44.402292+0000","last_active":"2024-03-18T18:49:48.839292+0000","last_peered":"2024-03-18T18:49:48.839292+0000","last_clean":"2024-03-18T18:49:48.839292+0000","last_became_active":"2024-03-18T18:49:44.402028+0000","last_became_peered":"2024-03-18T18:49:44.402028+0000","last_unstale":"2024-03-18T18:49:48.839292+0000","last_undegraded":"2024-03-18T18:49:48.839292+0000","last_fullsized":"2024-03-18T18:49:48.839292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.843349+0000","last_change":"2024-03-18T18:49:44.832023+0000","last_active":"2024-03-18T18:49:48.843349+0000","last_peered":"2024-03-18T18:49:48.843349+0000","last_clean":"2024-03-18T18:49:48.843349+0000","last_became_active":"2024-03-18T18:49:44.831734+0000","last_became_peered":"2024-03-18T18:49:44.831734+0000","last_unstale":"2024-03-18T18:49:48.843349+0000","last_undegraded":"2024-03-18T18:49:48.843349+0000","last_fullsized":"2024-03-18T18:49:48.843349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:10:25.085288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.856842+0000","last_change":"2024-03-18T18:49:48.852370+0000","last_active":"2024-03-18T18:49:48.856842+0000","last_peered":"2024-03-18T18:49:48.856842+0000","last_clean":"2024-03-18T18:49:48.856842+0000","last_became_active":"2024-03-18T18:49:48.851953+0000","last_became_peered":"2024-03-18T18:49:48.851953+0000","last_unstale":"2024-03-18T18:49:48.856842+0000","last_undegraded":"2024-03-18T18:49:48.856842+0000","last_fullsized":"2024-03-18T18:49:48.856842+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:17.506065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836324+0000","last_change":"2024-03-18T18:49:44.394163+0000","last_active":"2024-03-18T18:49:48.836324+0000","last_peered":"2024-03-18T18:49:48.836324+0000","last_clean":"2024-03-18T18:49:48.836324+0000","last_became_active":"2024-03-18T18:49:44.393105+0000","last_became_peered":"2024-03-18T18:49:44.393105+0000","last_unstale":"2024-03-18T18:49:48.836324+0000","last_undegraded":"2024-03-18T18:49:48.836324+0000","last_fullsized":"2024-03-18T18:49:48.836324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:30.124822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.846218+0000","last_change":"2024-03-18T18:49:44.834950+0000","last_active":"2024-03-18T18:49:48.846218+0000","last_peered":"2024-03-18T18:49:48.846218+0000","last_clean":"2024-03-18T18:49:48.846218+0000","last_became_active":"2024-03-18T18:49:44.834765+0000","last_became_peered":"2024-03-18T18:49:44.834765+0000","last_unstale":"2024-03-18T18:49:48.846218+0000","last_undegraded":"2024-03-18T18:49:48.846218+0000","last_fullsized":"2024-03-18T18:49:48.846218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:49.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.857842+0000","last_change":"2024-03-18T18:49:48.857127+0000","last_active":"2024-03-18T18:49:48.857842+0000","last_peered":"2024-03-18T18:49:48.857842+0000","last_clean":"2024-03-18T18:49:48.857842+0000","last_became_active":"2024-03-18T18:49:48.856745+0000","last_became_peered":"2024-03-18T18:49:48.856745+0000","last_unstale":"2024-03-18T18:49:48.857842+0000","last_undegraded":"2024-03-18T18:49:48.857842+0000","last_fullsized":"2024-03-18T18:49:48.857842+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:46:22.763891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840897+0000","last_change":"2024-03-18T18:49:44.396730+0000","last_active":"2024-03-18T18:49:48.840897+0000","last_peered":"2024-03-18T18:49:48.840897+0000","last_clean":"2024-03-18T18:49:48.840897+0000","last_became_active":"2024-03-18T18:49:44.396415+0000","last_became_peered":"2024-03-18T18:49:44.396415+0000","last_unstale":"2024-03-18T18:49:48.840897+0000","last_undegraded":"2024-03-18T18:49:48.840897+0000","last_fullsized":"2024-03-18T18:49:48.840897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:28.338064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839641+0000","last_change":"2024-03-18T18:49:44.836855+0000","last_active":"2024-03-18T18:49:48.839641+0000","last_peered":"2024-03-18T18:49:48.839641+0000","last_clean":"2024-03-18T18:49:48.839641+0000","last_became_active":"2024-03-18T18:49:44.836275+0000","last_became_peered":"2024-03-18T18:49:44.836275+0000","last_unstale":"2024-03-18T18:49:48.839641+0000","last_undegraded":"2024-03-18T18:49:48.839641+0000","last_fullsized":"2024-03-18T18:49:48.839641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:49.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980433+0000","last_change":"2024-03-18T18:49:48.978139+0000","last_active":"2024-03-18T18:49:48.980433+0000","last_peered":"2024-03-18T18:49:48.980433+0000","last_clean":"2024-03-18T18:49:48.980433+0000","last_became_active":"2024-03-18T18:49:48.977755+0000","last_became_peered":"2024-03-18T18:49:48.977755+0000","last_unstale":"2024-03-18T18:49:48.980433+0000","last_undegraded":"2024-03-18T18:49:48.980433+0000","last_fullsized":"2024-03-18T18:49:48.980433+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:43.744272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839770+0000","last_change":"2024-03-18T18:49:44.400836+0000","last_active":"2024-03-18T18:49:48.839770+0000","last_peered":"2024-03-18T18:49:48.839770+0000","last_clean":"2024-03-18T18:49:48.839770+0000","last_became_active":"2024-03-18T18:49:44.400587+0000","last_became_peered":"2024-03-18T18:49:44.400587+0000","last_unstale":"2024-03-18T18:49:48.839770+0000","last_undegraded":"2024-03-18T18:49:48.839770+0000","last_fullsized":"2024-03-18T18:49:48.839770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:30.124822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840775+0000","last_change":"2024-03-18T18:49:44.822283+0000","last_active":"2024-03-18T18:49:48.840775+0000","last_peered":"2024-03-18T18:49:48.840775+0000","last_clean":"2024-03-18T18:49:48.840775+0000","last_became_active":"2024-03-18T18:49:44.822016+0000","last_became_peered":"2024-03-18T18:49:44.822016+0000","last_unstale":"2024-03-18T18:49:48.840775+0000","last_undegraded":"2024-03-18T18:49:48.840775+0000","last_fullsized":"2024-03-18T18:49:48.840775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981750+0000","last_change":"2024-03-18T18:49:48.978066+0000","last_active":"2024-03-18T18:49:48.981750+0000","last_peered":"2024-03-18T18:49:48.981750+0000","last_clean":"2024-03-18T18:49:48.981750+0000","last_became_active":"2024-03-18T18:49:48.977682+0000","last_became_peered":"2024-03-18T18:49:48.977682+0000","last_unstale":"2024-03-18T18:49:48.981750+0000","last_undegraded":"2024-03-18T18:49:48.981750+0000","last_fullsized":"2024-03-18T18:49:48.981750+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:31:52.237923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836616+0000","last_change":"2024-03-18T18:49:44.402381+0000","last_active":"2024-03-18T18:49:48.836616+0000","last_peered":"2024-03-18T18:49:48.836616+0000","last_clean":"2024-03-18T18:49:48.836616+0000","last_became_active":"2024-03-18T18:49:44.402101+0000","last_became_peered":"2024-03-18T18:49:44.402101+0000","last_unstale":"2024-03-18T18:49:48.836616+0000","last_undegraded":"2024-03-18T18:49:48.836616+0000","last_fullsized":"2024-03-18T18:49:48.836616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:39.899537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840754+0000","last_change":"2024-03-18T18:49:44.841434+0000","last_active":"2024-03-18T18:49:48.840754+0000","last_peered":"2024-03-18T18:49:48.840754+0000","last_clean":"2024-03-18T18:49:48.840754+0000","last_became_active":"2024-03-18T18:49:44.841186+0000","last_became_peered":"2024-03-18T18:49:44.841186+0000","last_unstale":"2024-03-18T18:49:48.840754+0000","last_undegraded":"2024-03-18T18:49:48.840754+0000","last_fullsized":"2024-03-18T18:49:48.840754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.850697+0000","last_change":"2024-03-18T18:49:48.849692+0000","last_active":"2024-03-18T18:49:48.850697+0000","last_peered":"2024-03-18T18:49:48.850697+0000","last_clean":"2024-03-18T18:49:48.850697+0000","last_became_active":"2024-03-18T18:49:48.849230+0000","last_became_peered":"2024-03-18T18:49:48.849230+0000","last_unstale":"2024-03-18T18:49:48.850697+0000","last_undegraded":"2024-03-18T18:49:48.850697+0000","last_fullsized":"2024-03-18T18:49:48.850697+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:45:34.390044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840676+0000","last_change":"2024-03-18T18:49:44.400484+0000","last_active":"2024-03-18T18:49:48.840676+0000","last_peered":"2024-03-18T18:49:48.840676+0000","last_clean":"2024-03-18T18:49:48.840676+0000","last_became_active":"2024-03-18T18:49:44.398334+0000","last_became_peered":"2024-03-18T18:49:44.398334+0000","last_unstale":"2024-03-18T18:49:48.840676+0000","last_undegraded":"2024-03-18T18:49:48.840676+0000","last_fullsized":"2024-03-18T18:49:48.840676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:29.763642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.847751+0000","last_change":"2024-03-18T18:49:44.832690+0000","last_active":"2024-03-18T18:49:48.847751+0000","last_peered":"2024-03-18T18:49:48.847751+0000","last_clean":"2024-03-18T18:49:48.847751+0000","last_became_active":"2024-03-18T18:49:44.832317+0000","last_became_peered":"2024-03-18T18:49:44.832317+0000","last_unstale":"2024-03-18T18:49:48.847751+0000","last_undegraded":"2024-03-18T18:49:48.847751+0000","last_fullsized":"2024-03-18T18:49:48.847751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:34.102274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.850171+0000","last_change":"2024-03-18T18:49:48.849391+0000","last_active":"2024-03-18T18:49:48.850171+0000","last_peered":"2024-03-18T18:49:48.850171+0000","last_clean":"2024-03-18T18:49:48.850171+0000","last_became_active":"2024-03-18T18:49:48.849018+0000","last_became_peered":"2024-03-18T18:49:48.849018+0000","last_unstale":"2024-03-18T18:49:48.850171+0000","last_undegraded":"2024-03-18T18:49:48.850171+0000","last_fullsized":"2024-03-18T18:49:48.850171+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:20:46.845872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841507+0000","last_change":"2024-03-18T18:49:44.402300+0000","last_active":"2024-03-18T18:49:48.841507+0000","last_peered":"2024-03-18T18:49:48.841507+0000","last_clean":"2024-03-18T18:49:48.841507+0000","last_became_active":"2024-03-18T18:49:44.401998+0000","last_became_peered":"2024-03-18T18:49:44.401998+0000","last_unstale":"2024-03-18T18:49:48.841507+0000","last_undegraded":"2024-03-18T18:49:48.841507+0000","last_fullsized":"2024-03-18T18:49:48.841507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:09:07.245729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841888+0000","last_change":"2024-03-18T18:49:44.835310+0000","last_active":"2024-03-18T18:49:48.841888+0000","last_peered":"2024-03-18T18:49:48.841888+0000","last_clean":"2024-03-18T18:49:48.841888+0000","last_became_active":"2024-03-18T18:49:44.835035+0000","last_became_peered":"2024-03-18T18:49:44.835035+0000","last_unstale":"2024-03-18T18:49:48.841888+0000","last_undegraded":"2024-03-18T18:49:48.841888+0000","last_fullsized":"2024-03-18T18:49:48.841888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:29.829759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.865513+0000","last_change":"2024-03-18T18:49:48.863905+0000","last_active":"2024-03-18T18:49:48.865513+0000","last_peered":"2024-03-18T18:49:48.865513+0000","last_clean":"2024-03-18T18:49:48.865513+0000","last_became_active":"2024-03-18T18:49:48.863234+0000","last_became_peered":"2024-03-18T18:49:48.863234+0000","last_unstale":"2024-03-18T18:49:48.865513+0000","last_undegraded":"2024-03-18T18:49:48.865513+0000","last_fullsized":"2024-03-18T18:49:48.865513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:16:08.575913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T18:49:54.442358+0000","last_change":"2024-03-18T18:49:44.402443+0000","last_active":"2024-03-18T18:49:54.442358+0000","last_peered":"2024-03-18T18:49:54.442358+0000","last_clean":"2024-03-18T18:49:54.442358+0000","last_became_active":"2024-03-18T18:49:44.402144+0000","last_became_peered":"2024-03-18T18:49:44.402144+0000","last_unstale":"2024-03-18T18:49:54.442358+0000","last_undegraded":"2024-03-18T18:49:54.442358+0000","last_fullsized":"2024-03-18T18:49:54.442358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:06.816608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3795,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976826+0000","last_change":"2024-03-18T18:49:44.829814+0000","last_active":"2024-03-18T18:49:48.976826+0000","last_peered":"2024-03-18T18:49:48.976826+0000","last_clean":"2024-03-18T18:49:48.976826+0000","last_became_active":"2024-03-18T18:49:44.829548+0000","last_became_peered":"2024-03-18T18:49:44.829548+0000","last_unstale":"2024-03-18T18:49:48.976826+0000","last_undegraded":"2024-03-18T18:49:48.976826+0000","last_fullsized":"2024-03-18T18:49:48.976826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:02:06.725321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980345+0000","last_change":"2024-03-18T18:49:48.978265+0000","last_active":"2024-03-18T18:49:48.980345+0000","last_peered":"2024-03-18T18:49:48.980345+0000","last_clean":"2024-03-18T18:49:48.980345+0000","last_became_active":"2024-03-18T18:49:48.977853+0000","last_became_peered":"2024-03-18T18:49:48.977853+0000","last_unstale":"2024-03-18T18:49:48.980345+0000","last_undegraded":"2024-03-18T18:49:48.980345+0000","last_fullsized":"2024-03-18T18:49:48.980345+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:43.744272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.846561+0000","last_change":"2024-03-18T18:49:44.392639+0000","last_active":"2024-03-18T18:49:48.846561+0000","last_peered":"2024-03-18T18:49:48.846561+0000","last_clean":"2024-03-18T18:49:48.846561+0000","last_became_active":"2024-03-18T18:49:44.392339+0000","last_became_peered":"2024-03-18T18:49:44.392339+0000","last_unstale":"2024-03-18T18:49:48.846561+0000","last_undegraded":"2024-03-18T18:49:48.846561+0000","last_fullsized":"2024-03-18T18:49:48.846561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:49.491669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853657+0000","last_change":"2024-03-18T18:49:44.836441+0000","last_active":"2024-03-18T18:49:48.853657+0000","last_peered":"2024-03-18T18:49:48.853657+0000","last_clean":"2024-03-18T18:49:48.853657+0000","last_became_active":"2024-03-18T18:49:44.836156+0000","last_became_peered":"2024-03-18T18:49:44.836156+0000","last_unstale":"2024-03-18T18:49:48.853657+0000","last_undegraded":"2024-03-18T18:49:48.853657+0000","last_fullsized":"2024-03-18T18:49:48.853657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:27.913188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980346+0000","last_change":"2024-03-18T18:49:48.977065+0000","last_active":"2024-03-18T18:49:48.980346+0000","last_peered":"2024-03-18T18:49:48.980346+0000","last_clean":"2024-03-18T18:49:48.980346+0000","last_became_active":"2024-03-18T18:49:48.976568+0000","last_became_peered":"2024-03-18T18:49:48.976568+0000","last_unstale":"2024-03-18T18:49:48.980346+0000","last_undegraded":"2024-03-18T18:49:48.980346+0000","last_fullsized":"2024-03-18T18:49:48.980346+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:21.533140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839322+0000","last_change":"2024-03-18T18:49:44.404269+0000","last_active":"2024-03-18T18:49:48.839322+0000","last_peered":"2024-03-18T18:49:48.839322+0000","last_clean":"2024-03-18T18:49:48.839322+0000","last_became_active":"2024-03-18T18:49:44.403992+0000","last_became_peered":"2024-03-18T18:49:44.403992+0000","last_unstale":"2024-03-18T18:49:48.839322+0000","last_undegraded":"2024-03-18T18:49:48.839322+0000","last_fullsized":"2024-03-18T18:49:48.839322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975684+0000","last_change":"2024-03-18T18:49:44.834354+0000","last_active":"2024-03-18T18:49:48.975684+0000","last_peered":"2024-03-18T18:49:48.975684+0000","last_clean":"2024-03-18T18:49:48.975684+0000","last_became_active":"2024-03-18T18:49:44.834114+0000","last_became_peered":"2024-03-18T18:49:44.834114+0000","last_unstale":"2024-03-18T18:49:48.975684+0000","last_undegraded":"2024-03-18T18:49:48.975684+0000","last_fullsized":"2024-03-18T18:49:48.975684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:09:46.686770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981775+0000","last_change":"2024-03-18T18:49:48.978466+0000","last_active":"2024-03-18T18:49:48.981775+0000","last_peered":"2024-03-18T18:49:48.981775+0000","last_clean":"2024-03-18T18:49:48.981775+0000","last_became_active":"2024-03-18T18:49:48.977877+0000","last_became_peered":"2024-03-18T18:49:48.977877+0000","last_unstale":"2024-03-18T18:49:48.981775+0000","last_undegraded":"2024-03-18T18:49:48.981775+0000","last_fullsized":"2024-03-18T18:49:48.981775+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:56.090012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836197+0000","last_change":"2024-03-18T18:49:44.393777+0000","last_active":"2024-03-18T18:49:48.836197+0000","last_peered":"2024-03-18T18:49:48.836197+0000","last_clean":"2024-03-18T18:49:48.836197+0000","last_became_active":"2024-03-18T18:49:44.392861+0000","last_became_peered":"2024-03-18T18:49:44.392861+0000","last_unstale":"2024-03-18T18:49:48.836197+0000","last_undegraded":"2024-03-18T18:49:48.836197+0000","last_fullsized":"2024-03-18T18:49:48.836197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835596+0000","last_change":"2024-03-18T18:49:44.831121+0000","last_active":"2024-03-18T18:49:48.835596+0000","last_peered":"2024-03-18T18:49:48.835596+0000","last_clean":"2024-03-18T18:49:48.835596+0000","last_became_active":"2024-03-18T18:49:44.830864+0000","last_became_peered":"2024-03-18T18:49:44.830864+0000","last_unstale":"2024-03-18T18:49:48.835596+0000","last_undegraded":"2024-03-18T18:49:48.835596+0000","last_fullsized":"2024-03-18T18:49:48.835596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:23.558014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.860292+0000","last_change":"2024-03-18T18:49:48.858618+0000","last_active":"2024-03-18T18:49:48.860292+0000","last_peered":"2024-03-18T18:49:48.860292+0000","last_clean":"2024-03-18T18:49:48.860292+0000","last_became_active":"2024-03-18T18:49:48.858170+0000","last_became_peered":"2024-03-18T18:49:48.858170+0000","last_unstale":"2024-03-18T18:49:48.860292+0000","last_undegraded":"2024-03-18T18:49:48.860292+0000","last_fullsized":"2024-03-18T18:49:48.860292+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:31.426438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836418+0000","last_change":"2024-03-18T18:49:44.397131+0000","last_active":"2024-03-18T18:49:48.836418+0000","last_peered":"2024-03-18T18:49:48.836418+0000","last_clean":"2024-03-18T18:49:48.836418+0000","last_became_active":"2024-03-18T18:49:44.396856+0000","last_became_peered":"2024-03-18T18:49:44.396856+0000","last_unstale":"2024-03-18T18:49:48.836418+0000","last_undegraded":"2024-03-18T18:49:48.836418+0000","last_fullsized":"2024-03-18T18:49:48.836418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976093+0000","last_change":"2024-03-18T18:49:44.838701+0000","last_active":"2024-03-18T18:49:48.976093+0000","last_peered":"2024-03-18T18:49:48.976093+0000","last_clean":"2024-03-18T18:49:48.976093+0000","last_became_active":"2024-03-18T18:49:44.838424+0000","last_became_peered":"2024-03-18T18:49:44.838424+0000","last_unstale":"2024-03-18T18:49:48.976093+0000","last_undegraded":"2024-03-18T18:49:48.976093+0000","last_fullsized":"2024-03-18T18:49:48.976093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:25:14.491414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853778+0000","last_change":"2024-03-18T18:49:48.852659+0000","last_active":"2024-03-18T18:49:48.853778+0000","last_peered":"2024-03-18T18:49:48.853778+0000","last_clean":"2024-03-18T18:49:48.853778+0000","last_became_active":"2024-03-18T18:49:48.852248+0000","last_became_peered":"2024-03-18T18:49:48.852248+0000","last_unstale":"2024-03-18T18:49:48.853778+0000","last_undegraded":"2024-03-18T18:49:48.853778+0000","last_fullsized":"2024-03-18T18:49:48.853778+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:38.261593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841901+0000","last_change":"2024-03-18T18:49:44.399701+0000","last_active":"2024-03-18T18:49:48.841901+0000","last_peered":"2024-03-18T18:49:48.841901+0000","last_clean":"2024-03-18T18:49:48.841901+0000","last_became_active":"2024-03-18T18:49:44.399371+0000","last_became_peered":"2024-03-18T18:49:44.399371+0000","last_unstale":"2024-03-18T18:49:48.841901+0000","last_undegraded":"2024-03-18T18:49:48.841901+0000","last_fullsized":"2024-03-18T18:49:48.841901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:49.491669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975930+0000","last_change":"2024-03-18T18:49:44.834635+0000","last_active":"2024-03-18T18:49:48.975930+0000","last_peered":"2024-03-18T18:49:48.975930+0000","last_clean":"2024-03-18T18:49:48.975930+0000","last_became_active":"2024-03-18T18:49:44.834321+0000","last_became_peered":"2024-03-18T18:49:44.834321+0000","last_unstale":"2024-03-18T18:49:48.975930+0000","last_undegraded":"2024-03-18T18:49:48.975930+0000","last_fullsized":"2024-03-18T18:49:48.975930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:50.238592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.865503+0000","last_change":"2024-03-18T18:49:48.863585+0000","last_active":"2024-03-18T18:49:48.865503+0000","last_peered":"2024-03-18T18:49:48.865503+0000","last_clean":"2024-03-18T18:49:48.865503+0000","last_became_active":"2024-03-18T18:49:48.862710+0000","last_became_peered":"2024-03-18T18:49:48.862710+0000","last_unstale":"2024-03-18T18:49:48.865503+0000","last_undegraded":"2024-03-18T18:49:48.865503+0000","last_fullsized":"2024-03-18T18:49:48.865503+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:31.426438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842830+0000","last_change":"2024-03-18T18:49:44.399251+0000","last_active":"2024-03-18T18:49:48.842830+0000","last_peered":"2024-03-18T18:49:48.842830+0000","last_clean":"2024-03-18T18:49:48.842830+0000","last_became_active":"2024-03-18T18:49:44.398969+0000","last_became_peered":"2024-03-18T18:49:44.398969+0000","last_unstale":"2024-03-18T18:49:48.842830+0000","last_undegraded":"2024-03-18T18:49:48.842830+0000","last_fullsized":"2024-03-18T18:49:48.842830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:39.899537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840553+0000","last_change":"2024-03-18T18:49:44.841934+0000","last_active":"2024-03-18T18:49:48.840553+0000","last_peered":"2024-03-18T18:49:48.840553+0000","last_clean":"2024-03-18T18:49:48.840553+0000","last_became_active":"2024-03-18T18:49:44.840657+0000","last_became_peered":"2024-03-18T18:49:44.840657+0000","last_unstale":"2024-03-18T18:49:48.840553+0000","last_undegraded":"2024-03-18T18:49:48.840553+0000","last_fullsized":"2024-03-18T18:49:48.840553+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.865367+0000","last_change":"2024-03-18T18:49:48.862242+0000","last_active":"2024-03-18T18:49:48.865367+0000","last_peered":"2024-03-18T18:49:48.865367+0000","last_clean":"2024-03-18T18:49:48.865367+0000","last_became_active":"2024-03-18T18:49:48.861779+0000","last_became_peered":"2024-03-18T18:49:48.861779+0000","last_unstale":"2024-03-18T18:49:48.865367+0000","last_undegraded":"2024-03-18T18:49:48.865367+0000","last_fullsized":"2024-03-18T18:49:48.865367+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:49:52.529190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839229+0000","last_change":"2024-03-18T18:49:44.404171+0000","last_active":"2024-03-18T18:49:48.839229+0000","last_peered":"2024-03-18T18:49:48.839229+0000","last_clean":"2024-03-18T18:49:48.839229+0000","last_became_active":"2024-03-18T18:49:44.403758+0000","last_became_peered":"2024-03-18T18:49:44.403758+0000","last_unstale":"2024-03-18T18:49:48.839229+0000","last_undegraded":"2024-03-18T18:49:48.839229+0000","last_fullsized":"2024-03-18T18:49:48.839229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:55.666762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841896+0000","last_change":"2024-03-18T18:49:44.839405+0000","last_active":"2024-03-18T18:49:48.841896+0000","last_peered":"2024-03-18T18:49:48.841896+0000","last_clean":"2024-03-18T18:49:48.841896+0000","last_became_active":"2024-03-18T18:49:44.838946+0000","last_became_peered":"2024-03-18T18:49:44.838946+0000","last_unstale":"2024-03-18T18:49:48.841896+0000","last_undegraded":"2024-03-18T18:49:48.841896+0000","last_fullsized":"2024-03-18T18:49:48.841896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:07.819299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.865474+0000","last_change":"2024-03-18T18:49:48.863506+0000","last_active":"2024-03-18T18:49:48.865474+0000","last_peered":"2024-03-18T18:49:48.865474+0000","last_clean":"2024-03-18T18:49:48.865474+0000","last_became_active":"2024-03-18T18:49:48.863081+0000","last_became_peered":"2024-03-18T18:49:48.863081+0000","last_unstale":"2024-03-18T18:49:48.865474+0000","last_undegraded":"2024-03-18T18:49:48.865474+0000","last_fullsized":"2024-03-18T18:49:48.865474+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:47.284063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839273+0000","last_change":"2024-03-18T18:49:44.833724+0000","last_active":"2024-03-18T18:49:48.839273+0000","last_peered":"2024-03-18T18:49:48.839273+0000","last_clean":"2024-03-18T18:49:48.839273+0000","last_became_active":"2024-03-18T18:49:44.833231+0000","last_became_peered":"2024-03-18T18:49:44.833231+0000","last_unstale":"2024-03-18T18:49:48.839273+0000","last_undegraded":"2024-03-18T18:49:48.839273+0000","last_fullsized":"2024-03-18T18:49:48.839273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:30.009397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836484+0000","last_change":"2024-03-18T18:49:44.402405+0000","last_active":"2024-03-18T18:49:48.836484+0000","last_peered":"2024-03-18T18:49:48.836484+0000","last_clean":"2024-03-18T18:49:48.836484+0000","last_became_active":"2024-03-18T18:49:44.402133+0000","last_became_peered":"2024-03-18T18:49:44.402133+0000","last_unstale":"2024-03-18T18:49:48.836484+0000","last_undegraded":"2024-03-18T18:49:48.836484+0000","last_fullsized":"2024-03-18T18:49:48.836484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:29.763642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981114+0000","last_change":"2024-03-18T18:49:48.978691+0000","last_active":"2024-03-18T18:49:48.981114+0000","last_peered":"2024-03-18T18:49:48.981114+0000","last_clean":"2024-03-18T18:49:48.981114+0000","last_became_active":"2024-03-18T18:49:48.977741+0000","last_became_peered":"2024-03-18T18:49:48.977741+0000","last_unstale":"2024-03-18T18:49:48.981114+0000","last_undegraded":"2024-03-18T18:49:48.981114+0000","last_fullsized":"2024-03-18T18:49:48.981114+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:07:03.479681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840954+0000","last_change":"2024-03-18T18:49:44.833327+0000","last_active":"2024-03-18T18:49:48.840954+0000","last_peered":"2024-03-18T18:49:48.840954+0000","last_clean":"2024-03-18T18:49:48.840954+0000","last_became_active":"2024-03-18T18:49:44.833073+0000","last_became_peered":"2024-03-18T18:49:44.833073+0000","last_unstale":"2024-03-18T18:49:48.840954+0000","last_undegraded":"2024-03-18T18:49:48.840954+0000","last_fullsized":"2024-03-18T18:49:48.840954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:27:55.580010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839748+0000","last_change":"2024-03-18T18:49:44.401250+0000","last_active":"2024-03-18T18:49:48.839748+0000","last_peered":"2024-03-18T18:49:48.839748+0000","last_clean":"2024-03-18T18:49:48.839748+0000","last_became_active":"2024-03-18T18:49:44.398256+0000","last_became_peered":"2024-03-18T18:49:44.398256+0000","last_unstale":"2024-03-18T18:49:48.839748+0000","last_undegraded":"2024-03-18T18:49:48.839748+0000","last_fullsized":"2024-03-18T18:49:48.839748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:06.816608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":167,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975479+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T18:49:48.975479+0000","last_peered":"2024-03-18T18:49:48.975479+0000","last_clean":"2024-03-18T18:49:48.975479+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T18:49:48.975479+0000","last_undegraded":"2024-03-18T18:49:48.975479+0000","last_fullsized":"2024-03-18T18:49:48.975479+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980453+0000","last_change":"2024-03-18T18:49:48.978188+0000","last_active":"2024-03-18T18:49:48.980453+0000","last_peered":"2024-03-18T18:49:48.980453+0000","last_clean":"2024-03-18T18:49:48.980453+0000","last_became_active":"2024-03-18T18:49:48.977852+0000","last_became_peered":"2024-03-18T18:49:48.977852+0000","last_unstale":"2024-03-18T18:49:48.980453+0000","last_undegraded":"2024-03-18T18:49:48.980453+0000","last_fullsized":"2024-03-18T18:49:48.980453+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:38.261593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.848312+0000","last_change":"2024-03-18T18:49:44.839127+0000","last_active":"2024-03-18T18:49:48.848312+0000","last_peered":"2024-03-18T18:49:48.848312+0000","last_clean":"2024-03-18T18:49:48.848312+0000","last_became_active":"2024-03-18T18:49:44.838948+0000","last_became_peered":"2024-03-18T18:49:44.838948+0000","last_unstale":"2024-03-18T18:49:48.848312+0000","last_undegraded":"2024-03-18T18:49:48.848312+0000","last_fullsized":"2024-03-18T18:49:48.848312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:23.558014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975591+0000","last_change":"2024-03-18T18:49:44.402282+0000","last_active":"2024-03-18T18:49:48.975591+0000","last_peered":"2024-03-18T18:49:48.975591+0000","last_clean":"2024-03-18T18:49:48.975591+0000","last_became_active":"2024-03-18T18:49:44.402003+0000","last_became_peered":"2024-03-18T18:49:44.402003+0000","last_unstale":"2024-03-18T18:49:48.975591+0000","last_undegraded":"2024-03-18T18:49:48.975591+0000","last_fullsized":"2024-03-18T18:49:48.975591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:28.827068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981915+0000","last_change":"2024-03-18T18:49:48.978958+0000","last_active":"2024-03-18T18:49:48.981915+0000","last_peered":"2024-03-18T18:49:48.981915+0000","last_clean":"2024-03-18T18:49:48.981915+0000","last_became_active":"2024-03-18T18:49:48.978608+0000","last_became_peered":"2024-03-18T18:49:48.978608+0000","last_unstale":"2024-03-18T18:49:48.981915+0000","last_undegraded":"2024-03-18T18:49:48.981915+0000","last_fullsized":"2024-03-18T18:49:48.981915+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:08:46.162308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975789+0000","last_change":"2024-03-18T18:49:44.397512+0000","last_active":"2024-03-18T18:49:48.975789+0000","last_peered":"2024-03-18T18:49:48.975789+0000","last_clean":"2024-03-18T18:49:48.975789+0000","last_became_active":"2024-03-18T18:49:44.397271+0000","last_became_peered":"2024-03-18T18:49:44.397271+0000","last_unstale":"2024-03-18T18:49:48.975789+0000","last_undegraded":"2024-03-18T18:49:48.975789+0000","last_fullsized":"2024-03-18T18:49:48.975789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:55.666762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842833+0000","last_change":"2024-03-18T18:49:44.838715+0000","last_active":"2024-03-18T18:49:48.842833+0000","last_peered":"2024-03-18T18:49:48.842833+0000","last_clean":"2024-03-18T18:49:48.842833+0000","last_became_active":"2024-03-18T18:49:44.838452+0000","last_became_peered":"2024-03-18T18:49:44.838452+0000","last_unstale":"2024-03-18T18:49:48.842833+0000","last_undegraded":"2024-03-18T18:49:48.842833+0000","last_fullsized":"2024-03-18T18:49:48.842833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:23.558014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981853+0000","last_change":"2024-03-18T18:49:48.978525+0000","last_active":"2024-03-18T18:49:48.981853+0000","last_peered":"2024-03-18T18:49:48.981853+0000","last_clean":"2024-03-18T18:49:48.981853+0000","last_became_active":"2024-03-18T18:49:48.977651+0000","last_became_peered":"2024-03-18T18:49:48.977651+0000","last_unstale":"2024-03-18T18:49:48.981853+0000","last_undegraded":"2024-03-18T18:49:48.981853+0000","last_fullsized":"2024-03-18T18:49:48.981853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:31:53.057656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841616+0000","last_change":"2024-03-18T18:49:44.397630+0000","last_active":"2024-03-18T18:49:48.841616+0000","last_peered":"2024-03-18T18:49:48.841616+0000","last_clean":"2024-03-18T18:49:48.841616+0000","last_became_active":"2024-03-18T18:49:44.396306+0000","last_became_peered":"2024-03-18T18:49:44.396306+0000","last_unstale":"2024-03-18T18:49:48.841616+0000","last_undegraded":"2024-03-18T18:49:48.841616+0000","last_fullsized":"2024-03-18T18:49:48.841616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:10:24.082597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842664+0000","last_change":"2024-03-18T18:49:44.821273+0000","last_active":"2024-03-18T18:49:48.842664+0000","last_peered":"2024-03-18T18:49:48.842664+0000","last_clean":"2024-03-18T18:49:48.842664+0000","last_became_active":"2024-03-18T18:49:44.821036+0000","last_became_peered":"2024-03-18T18:49:44.821036+0000","last_unstale":"2024-03-18T18:49:48.842664+0000","last_undegraded":"2024-03-18T18:49:48.842664+0000","last_fullsized":"2024-03-18T18:49:48.842664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:30.009397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.982011+0000","last_change":"2024-03-18T18:49:48.978130+0000","last_active":"2024-03-18T18:49:48.982011+0000","last_peered":"2024-03-18T18:49:48.982011+0000","last_clean":"2024-03-18T18:49:48.982011+0000","last_became_active":"2024-03-18T18:49:48.977715+0000","last_became_peered":"2024-03-18T18:49:48.977715+0000","last_unstale":"2024-03-18T18:49:48.982011+0000","last_undegraded":"2024-03-18T18:49:48.982011+0000","last_fullsized":"2024-03-18T18:49:48.982011+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:47.284063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836478+0000","last_change":"2024-03-18T18:49:44.394139+0000","last_active":"2024-03-18T18:49:48.836478+0000","last_peered":"2024-03-18T18:49:48.836478+0000","last_clean":"2024-03-18T18:49:48.836478+0000","last_became_active":"2024-03-18T18:49:44.393405+0000","last_became_peered":"2024-03-18T18:49:44.393405+0000","last_unstale":"2024-03-18T18:49:48.836478+0000","last_undegraded":"2024-03-18T18:49:48.836478+0000","last_fullsized":"2024-03-18T18:49:48.836478+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:55.666762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853791+0000","last_change":"2024-03-18T18:49:44.829271+0000","last_active":"2024-03-18T18:49:48.853791+0000","last_peered":"2024-03-18T18:49:48.853791+0000","last_clean":"2024-03-18T18:49:48.853791+0000","last_became_active":"2024-03-18T18:49:44.829019+0000","last_became_peered":"2024-03-18T18:49:44.829019+0000","last_unstale":"2024-03-18T18:49:48.853791+0000","last_undegraded":"2024-03-18T18:49:48.853791+0000","last_fullsized":"2024-03-18T18:49:48.853791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853721+0000","last_change":"2024-03-18T18:49:48.852590+0000","last_active":"2024-03-18T18:49:48.853721+0000","last_peered":"2024-03-18T18:49:48.853721+0000","last_clean":"2024-03-18T18:49:48.853721+0000","last_became_active":"2024-03-18T18:49:48.852088+0000","last_became_peered":"2024-03-18T18:49:48.852088+0000","last_unstale":"2024-03-18T18:49:48.853721+0000","last_undegraded":"2024-03-18T18:49:48.853721+0000","last_fullsized":"2024-03-18T18:49:48.853721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:16:08.575913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975464+0000","last_change":"2024-03-18T18:49:44.393305+0000","last_active":"2024-03-18T18:49:48.975464+0000","last_peered":"2024-03-18T18:49:48.975464+0000","last_clean":"2024-03-18T18:49:48.975464+0000","last_became_active":"2024-03-18T18:49:44.392494+0000","last_became_peered":"2024-03-18T18:49:44.392494+0000","last_unstale":"2024-03-18T18:49:48.975464+0000","last_undegraded":"2024-03-18T18:49:48.975464+0000","last_fullsized":"2024-03-18T18:49:48.975464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:06.816608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841483+0000","last_change":"2024-03-18T18:49:44.837726+0000","last_active":"2024-03-18T18:49:48.841483+0000","last_peered":"2024-03-18T18:49:48.841483+0000","last_clean":"2024-03-18T18:49:48.841483+0000","last_became_active":"2024-03-18T18:49:44.837364+0000","last_became_peered":"2024-03-18T18:49:44.837364+0000","last_unstale":"2024-03-18T18:49:48.841483+0000","last_undegraded":"2024-03-18T18:49:48.841483+0000","last_fullsized":"2024-03-18T18:49:48.841483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:40.902228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.982312+0000","last_change":"2024-03-18T18:49:48.980106+0000","last_active":"2024-03-18T18:49:48.982312+0000","last_peered":"2024-03-18T18:49:48.982312+0000","last_clean":"2024-03-18T18:49:48.982312+0000","last_became_active":"2024-03-18T18:49:48.979709+0000","last_became_peered":"2024-03-18T18:49:48.979709+0000","last_unstale":"2024-03-18T18:49:48.982312+0000","last_undegraded":"2024-03-18T18:49:48.982312+0000","last_fullsized":"2024-03-18T18:49:48.982312+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:21.533140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840674+0000","last_change":"2024-03-18T18:49:44.398886+0000","last_active":"2024-03-18T18:49:48.840674+0000","last_peered":"2024-03-18T18:49:48.840674+0000","last_clean":"2024-03-18T18:49:48.840674+0000","last_became_active":"2024-03-18T18:49:44.398578+0000","last_became_peered":"2024-03-18T18:49:44.398578+0000","last_unstale":"2024-03-18T18:49:48.840674+0000","last_undegraded":"2024-03-18T18:49:48.840674+0000","last_fullsized":"2024-03-18T18:49:48.840674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839766+0000","last_change":"2024-03-18T18:49:44.826751+0000","last_active":"2024-03-18T18:49:48.839766+0000","last_peered":"2024-03-18T18:49:48.839766+0000","last_clean":"2024-03-18T18:49:48.839766+0000","last_became_active":"2024-03-18T18:49:44.825975+0000","last_became_peered":"2024-03-18T18:49:44.825975+0000","last_unstale":"2024-03-18T18:49:48.839766+0000","last_undegraded":"2024-03-18T18:49:48.839766+0000","last_fullsized":"2024-03-18T18:49:48.839766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:02:06.725321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853727+0000","last_change":"2024-03-18T18:49:48.852554+0000","last_active":"2024-03-18T18:49:48.853727+0000","last_peered":"2024-03-18T18:49:48.853727+0000","last_clean":"2024-03-18T18:49:48.853727+0000","last_became_active":"2024-03-18T18:49:48.852121+0000","last_became_peered":"2024-03-18T18:49:48.852121+0000","last_unstale":"2024-03-18T18:49:48.853727+0000","last_undegraded":"2024-03-18T18:49:48.853727+0000","last_fullsized":"2024-03-18T18:49:48.853727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:41:26.757218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841024+0000","last_change":"2024-03-18T18:49:44.402526+0000","last_active":"2024-03-18T18:49:48.841024+0000","last_peered":"2024-03-18T18:49:48.841024+0000","last_clean":"2024-03-18T18:49:48.841024+0000","last_became_active":"2024-03-18T18:49:44.402277+0000","last_became_peered":"2024-03-18T18:49:44.402277+0000","last_unstale":"2024-03-18T18:49:48.841024+0000","last_undegraded":"2024-03-18T18:49:48.841024+0000","last_fullsized":"2024-03-18T18:49:48.841024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:06.816608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839383+0000","last_change":"2024-03-18T18:49:44.830629+0000","last_active":"2024-03-18T18:49:48.839383+0000","last_peered":"2024-03-18T18:49:48.839383+0000","last_clean":"2024-03-18T18:49:48.839383+0000","last_became_active":"2024-03-18T18:49:44.830313+0000","last_became_peered":"2024-03-18T18:49:44.830313+0000","last_unstale":"2024-03-18T18:49:48.839383+0000","last_undegraded":"2024-03-18T18:49:48.839383+0000","last_fullsized":"2024-03-18T18:49:48.839383+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:27.913188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.860287+0000","last_change":"2024-03-18T18:49:48.859810+0000","last_active":"2024-03-18T18:49:48.860287+0000","last_peered":"2024-03-18T18:49:48.860287+0000","last_clean":"2024-03-18T18:49:48.860287+0000","last_became_active":"2024-03-18T18:49:48.859453+0000","last_became_peered":"2024-03-18T18:49:48.859453+0000","last_unstale":"2024-03-18T18:49:48.860287+0000","last_undegraded":"2024-03-18T18:49:48.860287+0000","last_fullsized":"2024-03-18T18:49:48.860287+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:11:09.238467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975758+0000","last_change":"2024-03-18T18:49:44.396856+0000","last_active":"2024-03-18T18:49:48.975758+0000","last_peered":"2024-03-18T18:49:48.975758+0000","last_clean":"2024-03-18T18:49:48.975758+0000","last_became_active":"2024-03-18T18:49:44.395207+0000","last_became_peered":"2024-03-18T18:49:44.395207+0000","last_unstale":"2024-03-18T18:49:48.975758+0000","last_undegraded":"2024-03-18T18:49:48.975758+0000","last_fullsized":"2024-03-18T18:49:48.975758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:39.899537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836368+0000","last_change":"2024-03-18T18:49:44.836659+0000","last_active":"2024-03-18T18:49:48.836368+0000","last_peered":"2024-03-18T18:49:48.836368+0000","last_clean":"2024-03-18T18:49:48.836368+0000","last_became_active":"2024-03-18T18:49:44.836362+0000","last_became_peered":"2024-03-18T18:49:44.836362+0000","last_unstale":"2024-03-18T18:49:48.836368+0000","last_undegraded":"2024-03-18T18:49:48.836368+0000","last_fullsized":"2024-03-18T18:49:48.836368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:49.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980755+0000","last_change":"2024-03-18T18:49:48.978204+0000","last_active":"2024-03-18T18:49:48.980755+0000","last_peered":"2024-03-18T18:49:48.980755+0000","last_clean":"2024-03-18T18:49:48.980755+0000","last_became_active":"2024-03-18T18:49:48.977745+0000","last_became_peered":"2024-03-18T18:49:48.977745+0000","last_unstale":"2024-03-18T18:49:48.980755+0000","last_undegraded":"2024-03-18T18:49:48.980755+0000","last_fullsized":"2024-03-18T18:49:48.980755+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:08:46.162308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.837352+0000","last_change":"2024-03-18T18:49:44.398731+0000","last_active":"2024-03-18T18:49:48.837352+0000","last_peered":"2024-03-18T18:49:48.837352+0000","last_clean":"2024-03-18T18:49:48.837352+0000","last_became_active":"2024-03-18T18:49:44.398448+0000","last_became_peered":"2024-03-18T18:49:44.398448+0000","last_unstale":"2024-03-18T18:49:48.837352+0000","last_undegraded":"2024-03-18T18:49:48.837352+0000","last_fullsized":"2024-03-18T18:49:48.837352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:30.124822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976056+0000","last_change":"2024-03-18T18:49:44.833300+0000","last_active":"2024-03-18T18:49:48.976056+0000","last_peered":"2024-03-18T18:49:48.976056+0000","last_clean":"2024-03-18T18:49:48.976056+0000","last_became_active":"2024-03-18T18:49:44.833027+0000","last_became_peered":"2024-03-18T18:49:44.833027+0000","last_unstale":"2024-03-18T18:49:48.976056+0000","last_undegraded":"2024-03-18T18:49:48.976056+0000","last_fullsized":"2024-03-18T18:49:48.976056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:11:05.787600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.860344+0000","last_change":"2024-03-18T18:49:48.859259+0000","last_active":"2024-03-18T18:49:48.860344+0000","last_peered":"2024-03-18T18:49:48.860344+0000","last_clean":"2024-03-18T18:49:48.860344+0000","last_became_active":"2024-03-18T18:49:48.857100+0000","last_became_peered":"2024-03-18T18:49:48.857100+0000","last_unstale":"2024-03-18T18:49:48.860344+0000","last_undegraded":"2024-03-18T18:49:48.860344+0000","last_fullsized":"2024-03-18T18:49:48.860344+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:47.284063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841409+0000","last_change":"2024-03-18T18:49:44.395491+0000","last_active":"2024-03-18T18:49:48.841409+0000","last_peered":"2024-03-18T18:49:48.841409+0000","last_clean":"2024-03-18T18:49:48.841409+0000","last_became_active":"2024-03-18T18:49:44.394529+0000","last_became_peered":"2024-03-18T18:49:44.394529+0000","last_unstale":"2024-03-18T18:49:48.841409+0000","last_undegraded":"2024-03-18T18:49:48.841409+0000","last_fullsized":"2024-03-18T18:49:48.841409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:55.666762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839957+0000","last_change":"2024-03-18T18:49:44.838466+0000","last_active":"2024-03-18T18:49:48.839957+0000","last_peered":"2024-03-18T18:49:48.839957+0000","last_clean":"2024-03-18T18:49:48.839957+0000","last_became_active":"2024-03-18T18:49:44.838027+0000","last_became_peered":"2024-03-18T18:49:44.838027+0000","last_unstale":"2024-03-18T18:49:48.839957+0000","last_undegraded":"2024-03-18T18:49:48.839957+0000","last_fullsized":"2024-03-18T18:49:48.839957+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:09:46.686770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981225+0000","last_change":"2024-03-18T18:49:48.979431+0000","last_active":"2024-03-18T18:49:48.981225+0000","last_peered":"2024-03-18T18:49:48.981225+0000","last_clean":"2024-03-18T18:49:48.981225+0000","last_became_active":"2024-03-18T18:49:48.978587+0000","last_became_peered":"2024-03-18T18:49:48.978587+0000","last_unstale":"2024-03-18T18:49:48.981225+0000","last_undegraded":"2024-03-18T18:49:48.981225+0000","last_fullsized":"2024-03-18T18:49:48.981225+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:47.284063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836616+0000","last_change":"2024-03-18T18:49:44.403374+0000","last_active":"2024-03-18T18:49:48.836616+0000","last_peered":"2024-03-18T18:49:48.836616+0000","last_clean":"2024-03-18T18:49:48.836616+0000","last_became_active":"2024-03-18T18:49:44.403082+0000","last_became_peered":"2024-03-18T18:49:44.403082+0000","last_unstale":"2024-03-18T18:49:48.836616+0000","last_undegraded":"2024-03-18T18:49:48.836616+0000","last_fullsized":"2024-03-18T18:49:48.836616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:28.338064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976741+0000","last_change":"2024-03-18T18:49:44.835166+0000","last_active":"2024-03-18T18:49:48.976741+0000","last_peered":"2024-03-18T18:49:48.976741+0000","last_clean":"2024-03-18T18:49:48.976741+0000","last_became_active":"2024-03-18T18:49:44.834808+0000","last_became_peered":"2024-03-18T18:49:44.834808+0000","last_unstale":"2024-03-18T18:49:48.976741+0000","last_undegraded":"2024-03-18T18:49:48.976741+0000","last_fullsized":"2024-03-18T18:49:48.976741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:20.816344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835598+0000","last_change":"2024-03-18T18:49:44.829141+0000","last_active":"2024-03-18T18:49:48.835598+0000","last_peered":"2024-03-18T18:49:48.835598+0000","last_clean":"2024-03-18T18:49:48.835598+0000","last_became_active":"2024-03-18T18:49:44.828863+0000","last_became_peered":"2024-03-18T18:49:44.828863+0000","last_unstale":"2024-03-18T18:49:48.835598+0000","last_undegraded":"2024-03-18T18:49:48.835598+0000","last_fullsized":"2024-03-18T18:49:48.835598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:30.009397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836583+0000","last_change":"2024-03-18T18:49:44.831132+0000","last_active":"2024-03-18T18:49:48.836583+0000","last_peered":"2024-03-18T18:49:48.836583+0000","last_clean":"2024-03-18T18:49:48.836583+0000","last_became_active":"2024-03-18T18:49:44.830873+0000","last_became_peered":"2024-03-18T18:49:44.830873+0000","last_unstale":"2024-03-18T18:49:48.836583+0000","last_undegraded":"2024-03-18T18:49:48.836583+0000","last_fullsized":"2024-03-18T18:49:48.836583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:09:08.248420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839404+0000","last_change":"2024-03-18T18:49:44.841166+0000","last_active":"2024-03-18T18:49:48.839404+0000","last_peered":"2024-03-18T18:49:48.839404+0000","last_clean":"2024-03-18T18:49:48.839404+0000","last_became_active":"2024-03-18T18:49:44.840477+0000","last_became_peered":"2024-03-18T18:49:44.840477+0000","last_unstale":"2024-03-18T18:49:48.839404+0000","last_undegraded":"2024-03-18T18:49:48.839404+0000","last_fullsized":"2024-03-18T18:49:48.839404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:20.816344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.846309+0000","last_change":"2024-03-18T18:49:44.838132+0000","last_active":"2024-03-18T18:49:48.846309+0000","last_peered":"2024-03-18T18:49:48.846309+0000","last_clean":"2024-03-18T18:49:48.846309+0000","last_became_active":"2024-03-18T18:49:44.837914+0000","last_became_peered":"2024-03-18T18:49:44.837914+0000","last_unstale":"2024-03-18T18:49:48.846309+0000","last_undegraded":"2024-03-18T18:49:48.846309+0000","last_fullsized":"2024-03-18T18:49:48.846309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840730+0000","last_change":"2024-03-18T18:49:44.835464+0000","last_active":"2024-03-18T18:49:48.840730+0000","last_peered":"2024-03-18T18:49:48.840730+0000","last_clean":"2024-03-18T18:49:48.840730+0000","last_became_active":"2024-03-18T18:49:44.835190+0000","last_became_peered":"2024-03-18T18:49:44.835190+0000","last_unstale":"2024-03-18T18:49:48.840730+0000","last_undegraded":"2024-03-18T18:49:48.840730+0000","last_fullsized":"2024-03-18T18:49:48.840730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841309+0000","last_change":"2024-03-18T18:49:44.819962+0000","last_active":"2024-03-18T18:49:48.841309+0000","last_peered":"2024-03-18T18:49:48.841309+0000","last_clean":"2024-03-18T18:49:48.841309+0000","last_became_active":"2024-03-18T18:49:44.819698+0000","last_became_peered":"2024-03-18T18:49:44.819698+0000","last_unstale":"2024-03-18T18:49:48.841309+0000","last_undegraded":"2024-03-18T18:49:48.841309+0000","last_fullsized":"2024-03-18T18:49:48.841309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835920+0000","last_change":"2024-03-18T18:49:44.830928+0000","last_active":"2024-03-18T18:49:48.835920+0000","last_peered":"2024-03-18T18:49:48.835920+0000","last_clean":"2024-03-18T18:49:48.835920+0000","last_became_active":"2024-03-18T18:49:44.830658+0000","last_became_peered":"2024-03-18T18:49:44.830658+0000","last_unstale":"2024-03-18T18:49:48.835920+0000","last_undegraded":"2024-03-18T18:49:48.835920+0000","last_fullsized":"2024-03-18T18:49:48.835920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842763+0000","last_change":"2024-03-18T18:49:44.838837+0000","last_active":"2024-03-18T18:49:48.842763+0000","last_peered":"2024-03-18T18:49:48.842763+0000","last_clean":"2024-03-18T18:49:48.842763+0000","last_became_active":"2024-03-18T18:49:44.838564+0000","last_became_peered":"2024-03-18T18:49:44.838564+0000","last_unstale":"2024-03-18T18:49:48.842763+0000","last_undegraded":"2024-03-18T18:49:48.842763+0000","last_fullsized":"2024-03-18T18:49:48.842763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:34.102274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839856+0000","last_change":"2024-03-18T18:49:44.841576+0000","last_active":"2024-03-18T18:49:48.839856+0000","last_peered":"2024-03-18T18:49:48.839856+0000","last_clean":"2024-03-18T18:49:48.839856+0000","last_became_active":"2024-03-18T18:49:44.840814+0000","last_became_peered":"2024-03-18T18:49:44.840814+0000","last_unstale":"2024-03-18T18:49:48.839856+0000","last_undegraded":"2024-03-18T18:49:48.839856+0000","last_fullsized":"2024-03-18T18:49:48.839856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:02:06.725321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840423+0000","last_change":"2024-03-18T18:49:44.840212+0000","last_active":"2024-03-18T18:49:48.840423+0000","last_peered":"2024-03-18T18:49:48.840423+0000","last_clean":"2024-03-18T18:49:48.840423+0000","last_became_active":"2024-03-18T18:49:44.839960+0000","last_became_peered":"2024-03-18T18:49:44.839960+0000","last_unstale":"2024-03-18T18:49:48.840423+0000","last_undegraded":"2024-03-18T18:49:48.840423+0000","last_fullsized":"2024-03-18T18:49:48.840423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839485+0000","last_change":"2024-03-18T18:49:44.841770+0000","last_active":"2024-03-18T18:49:48.839485+0000","last_peered":"2024-03-18T18:49:48.839485+0000","last_clean":"2024-03-18T18:49:48.839485+0000","last_became_active":"2024-03-18T18:49:44.840939+0000","last_became_peered":"2024-03-18T18:49:44.840939+0000","last_unstale":"2024-03-18T18:49:48.839485+0000","last_undegraded":"2024-03-18T18:49:48.839485+0000","last_fullsized":"2024-03-18T18:49:48.839485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:20:43.395005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.846397+0000","last_change":"2024-03-18T18:49:44.841290+0000","last_active":"2024-03-18T18:49:48.846397+0000","last_peered":"2024-03-18T18:49:48.846397+0000","last_clean":"2024-03-18T18:49:48.846397+0000","last_became_active":"2024-03-18T18:49:44.840949+0000","last_became_peered":"2024-03-18T18:49:44.840949+0000","last_unstale":"2024-03-18T18:49:48.846397+0000","last_undegraded":"2024-03-18T18:49:48.846397+0000","last_fullsized":"2024-03-18T18:49:48.846397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:05.661348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835876+0000","last_change":"2024-03-18T18:49:44.837671+0000","last_active":"2024-03-18T18:49:48.835876+0000","last_peered":"2024-03-18T18:49:48.835876+0000","last_clean":"2024-03-18T18:49:48.835876+0000","last_became_active":"2024-03-18T18:49:44.837265+0000","last_became_peered":"2024-03-18T18:49:44.837265+0000","last_unstale":"2024-03-18T18:49:48.835876+0000","last_undegraded":"2024-03-18T18:49:48.835876+0000","last_fullsized":"2024-03-18T18:49:48.835876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:14.055199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4593,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":695401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":104611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":695457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":104451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":104577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":104541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T18:50:13.824 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T18:50:13.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.988+0000 7fae92efd700 1 Processor -- start 2024-03-18T18:50:13.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.989+0000 7fae92efd700 1 -- start start 2024-03-18T18:50:13.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.989+0000 7fae92efd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fae8c15d580 0x7fae8c072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:13.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.989+0000 7fae92efd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae8c06dd50 0x7fae8c0cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:13.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.989+0000 7fae92efd700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c06f490 con 0x7fae8c06e290 2024-03-18T18:50:13.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.989+0000 7fae92efd700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8c071cf0 con 0x7fae8c15d580 2024-03-18T18:50:13.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.989+0000 7fae92efd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c06f1b0 con 0x7fae8c06dd50 2024-03-18T18:50:13.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.989+0000 7fae90c99700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fae8c15d580 0x7fae8c072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:13.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.990+0000 7fae90c99700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fae8c15d580 0x7fae8c072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45514/0 (socket says 172.21.15.1:45514) 2024-03-18T18:50:13.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.990+0000 7fae90c99700 1 -- 172.21.15.1:0/2671221178 learned_addr learned my addr 172.21.15.1:0/2671221178 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:13.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.990+0000 7fae9149a700 1 --2- 172.21.15.1:0/2671221178 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae8c06dd50 0x7fae8c0cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:13.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.990+0000 7fae90c99700 1 -- 172.21.15.1:0/2671221178 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae8c06dd50 msgr2=0x7fae8c0cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:13.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.990+0000 7fae90c99700 1 --2- 172.21.15.1:0/2671221178 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae8c06dd50 0x7fae8c0cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.990+0000 7fae90c99700 1 -- 172.21.15.1:0/2671221178 >> v1:172.21.15.165:6789/0 conn(0x7fae8c06e290 legacy=0x7fae8c06d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.990+0000 7fae90c99700 1 -- 172.21.15.1:0/2671221178 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c0716a0 con 0x7fae8c15d580 2024-03-18T18:50:13.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.990+0000 7fae90c99700 1 --2- 172.21.15.1:0/2671221178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae8c15d580 0x7fae8c072180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fae800098f0 tx=0x7fae80009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989483f21a756201 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:13.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.990+0000 7fae8b7fe700 1 -- 172.21.15.1:0/2671221178 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8000b8a0 con 0x7fae8c15d580 2024-03-18T18:50:13.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.990+0000 7fae8b7fe700 1 -- 172.21.15.1:0/2671221178 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8000ba00 con 0x7fae8c15d580 2024-03-18T18:50:13.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.990+0000 7fae8b7fe700 1 -- 172.21.15.1:0/2671221178 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8000bd90 con 0x7fae8c15d580 2024-03-18T18:50:13.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.991+0000 7fae92efd700 1 -- 172.21.15.1:0/2671221178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae8c15d580 msgr2=0x7fae8c072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.991+0000 7fae92efd700 1 --2- 172.21.15.1:0/2671221178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae8c15d580 0x7fae8c072180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fae800098f0 tx=0x7fae80009c00 comp rx=0 tx=0).stop 2024-03-18T18:50:13.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.991+0000 7fae92efd700 1 -- 172.21.15.1:0/2671221178 shutdown_connections 2024-03-18T18:50:13.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.991+0000 7fae92efd700 1 --2- 172.21.15.1:0/2671221178 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae8c06dd50 0x7fae8c0cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.991+0000 7fae92efd700 1 --2- 172.21.15.1:0/2671221178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae8c15d580 0x7fae8c072180 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.991+0000 7fae92efd700 1 -- 172.21.15.1:0/2671221178 >> 172.21.15.1:0/2671221178 conn(0x7fae8c0cc990 msgr2=0x7fae8c0ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:13.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.991+0000 7fae92efd700 1 -- 172.21.15.1:0/2671221178 shutdown_connections 2024-03-18T18:50:13.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.991+0000 7fae92efd700 1 -- 172.21.15.1:0/2671221178 wait complete. 2024-03-18T18:50:13.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.992+0000 7fae92efd700 1 Processor -- start 2024-03-18T18:50:13.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.992+0000 7fae92efd700 1 -- start start 2024-03-18T18:50:13.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.993+0000 7fae92efd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae8c06dd50 0x7fae8c0b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:13.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.993+0000 7fae92efd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae8c15d580 0x7fae8c15ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:13.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.993+0000 7fae92efd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae8c15fd30 0x7fae8c155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:13.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.993+0000 7fae92efd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c15e8f0 con 0x7fae8c15d580 2024-03-18T18:50:13.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.993+0000 7fae92efd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c15dce0 con 0x7fae8c06dd50 2024-03-18T18:50:13.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.993+0000 7fae92efd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c15dfc0 con 0x7fae8c15fd30 2024-03-18T18:50:13.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.993+0000 7fae8bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae8c15d580 0x7fae8c15ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:13.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.993+0000 7fae8bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae8c15d580 0x7fae8c15ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45528/0 (socket says 172.21.15.1:45528) 2024-03-18T18:50:13.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.993+0000 7fae8bfff700 1 -- 172.21.15.1:0/1695701061 learned_addr learned my addr 172.21.15.1:0/1695701061 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:13.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.993+0000 7fae90c99700 1 --2- 172.21.15.1:0/1695701061 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae8c06dd50 0x7fae8c0b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:13.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.993+0000 7fae9149a700 1 --2- 172.21.15.1:0/1695701061 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae8c15fd30 0x7fae8c155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:13.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.993+0000 7fae8bfff700 1 -- 172.21.15.1:0/1695701061 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae8c15fd30 msgr2=0x7fae8c155770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.993+0000 7fae8bfff700 1 --2- 172.21.15.1:0/1695701061 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae8c15fd30 0x7fae8c155770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.993+0000 7fae8bfff700 1 -- 172.21.15.1:0/1695701061 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae8c06dd50 msgr2=0x7fae8c0b4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:13.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.993+0000 7fae8bfff700 1 --2- 172.21.15.1:0/1695701061 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae8c06dd50 0x7fae8c0b4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:13.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.993+0000 7fae8bfff700 1 -- 172.21.15.1:0/1695701061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae80009550 con 0x7fae8c15d580 2024-03-18T18:50:13.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.993+0000 7fae90c99700 1 --2- 172.21.15.1:0/1695701061 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae8c06dd50 0x7fae8c0b4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:13.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.993+0000 7fae9149a700 1 --2- 172.21.15.1:0/1695701061 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae8c15fd30 0x7fae8c155770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:13.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.993+0000 7fae8bfff700 1 --2- 172.21.15.1:0/1695701061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae8c15d580 0x7fae8c15ed80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fae7800c4e0 tx=0x7fae7800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:13.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.994+0000 7fae897fa700 1 -- 172.21.15.1:0/1695701061 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae780052d0 con 0x7fae8c15d580 2024-03-18T18:50:13.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.994+0000 7fae897fa700 1 -- 172.21.15.1:0/1695701061 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae78005430 con 0x7fae8c15d580 2024-03-18T18:50:13.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.994+0000 7fae92efd700 1 -- 172.21.15.1:0/1695701061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae8c15e2a0 con 0x7fae8c15d580 2024-03-18T18:50:13.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.994+0000 7fae897fa700 1 -- 172.21.15.1:0/1695701061 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae780038c0 con 0x7fae8c15d580 2024-03-18T18:50:13.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.994+0000 7fae92efd700 1 -- 172.21.15.1:0/1695701061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8c0b50c0 con 0x7fae8c15d580 2024-03-18T18:50:13.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.995+0000 7fae92efd700 1 -- 172.21.15.1:0/1695701061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae8c0676a0 con 0x7fae8c15d580 2024-03-18T18:50:13.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.995+0000 7fae897fa700 1 -- 172.21.15.1:0/1695701061 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fae78003a20 con 0x7fae8c15d580 2024-03-18T18:50:13.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.996+0000 7fae897fa700 1 --2- 172.21.15.1:0/1695701061 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae7006ac90 0x7fae7006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:13.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.996+0000 7fae897fa700 1 -- 172.21.15.1:0/1695701061 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fae78088b20 con 0x7fae8c15d580 2024-03-18T18:50:13.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.996+0000 7fae90c99700 1 --2- 172.21.15.1:0/1695701061 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae7006ac90 0x7fae7006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:13.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:13.996+0000 7fae90c99700 1 --2- 172.21.15.1:0/1695701061 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae7006ac90 0x7fae7006d150 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fae80014040 tx=0x7fae80008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:13.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.000+0000 7fae897fa700 1 -- 172.21.15.1:0/1695701061 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fae78053a00 con 0x7fae8c15d580 2024-03-18T18:50:14.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.201+0000 7fae92efd700 1 -- 172.21.15.1:0/1695701061 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fae8c06ee40 con 0x7fae7006ac90 2024-03-18T18:50:14.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.210+0000 7fae897fa700 1 -- 172.21.15.1:0/1695701061 <== mgr.4104 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345246 (secure 0 0 0) 0x7fae8001f070 con 0x7fae7006ac90 2024-03-18T18:50:14.211 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:14.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.214+0000 7fae92efd700 1 -- 172.21.15.1:0/1695701061 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae7006ac90 msgr2=0x7fae7006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:14.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.214+0000 7fae92efd700 1 --2- 172.21.15.1:0/1695701061 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae7006ac90 0x7fae7006d150 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fae80014040 tx=0x7fae80008030 comp rx=0 tx=0).stop 2024-03-18T18:50:14.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.214+0000 7fae92efd700 1 -- 172.21.15.1:0/1695701061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae8c15d580 msgr2=0x7fae8c15ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:14.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.214+0000 7fae92efd700 1 --2- 172.21.15.1:0/1695701061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae8c15d580 0x7fae8c15ed80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fae7800c4e0 tx=0x7fae7800c8a0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.214+0000 7fae92efd700 1 -- 172.21.15.1:0/1695701061 shutdown_connections 2024-03-18T18:50:14.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.214+0000 7fae92efd700 1 --2- 172.21.15.1:0/1695701061 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae7006ac90 0x7fae7006d150 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.214+0000 7fae92efd700 1 --2- 172.21.15.1:0/1695701061 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae8c15fd30 0x7fae8c155770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.214+0000 7fae92efd700 1 --2- 172.21.15.1:0/1695701061 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae8c06dd50 0x7fae8c0b4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.214+0000 7fae92efd700 1 --2- 172.21.15.1:0/1695701061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae8c15d580 0x7fae8c15ed80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.214+0000 7fae92efd700 1 -- 172.21.15.1:0/1695701061 >> 172.21.15.1:0/1695701061 conn(0x7fae8c0cc990 msgr2=0x7fae8c0684d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:14.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.214+0000 7fae92efd700 1 -- 172.21.15.1:0/1695701061 shutdown_connections 2024-03-18T18:50:14.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.214+0000 7fae92efd700 1 -- 172.21.15.1:0/1695701061 wait complete. 2024-03-18T18:50:14.214 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T18:50:14.222 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-03-18T18:50:13.394519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594961,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219464,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755256,"statfs":{"total":773094113280,"available":772869382144,"internally_reserved":0,"allocated":3383296,"data_stored":2025530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003752"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976876+0000","last_change":"2024-03-18T18:49:44.831395+0000","last_active":"2024-03-18T18:49:48.976876+0000","last_peered":"2024-03-18T18:49:48.976876+0000","last_clean":"2024-03-18T18:49:48.976876+0000","last_became_active":"2024-03-18T18:49:44.830706+0000","last_became_peered":"2024-03-18T18:49:44.830706+0000","last_unstale":"2024-03-18T18:49:48.976876+0000","last_undegraded":"2024-03-18T18:49:48.976876+0000","last_fullsized":"2024-03-18T18:49:48.976876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:20:43.395005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841775+0000","last_change":"2024-03-18T18:49:44.829492+0000","last_active":"2024-03-18T18:49:48.841775+0000","last_peered":"2024-03-18T18:49:48.841775+0000","last_clean":"2024-03-18T18:49:48.841775+0000","last_became_active":"2024-03-18T18:49:44.829223+0000","last_became_peered":"2024-03-18T18:49:44.829223+0000","last_unstale":"2024-03-18T18:49:48.841775+0000","last_undegraded":"2024-03-18T18:49:48.841775+0000","last_fullsized":"2024-03-18T18:49:48.841775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.845869+0000","last_change":"2024-03-18T18:49:44.831909+0000","last_active":"2024-03-18T18:49:48.845869+0000","last_peered":"2024-03-18T18:49:48.845869+0000","last_clean":"2024-03-18T18:49:48.845869+0000","last_became_active":"2024-03-18T18:49:44.831661+0000","last_became_peered":"2024-03-18T18:49:44.831661+0000","last_unstale":"2024-03-18T18:49:48.845869+0000","last_undegraded":"2024-03-18T18:49:48.845869+0000","last_fullsized":"2024-03-18T18:49:48.845869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:14.055199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840094+0000","last_change":"2024-03-18T18:49:44.838298+0000","last_active":"2024-03-18T18:49:48.840094+0000","last_peered":"2024-03-18T18:49:48.840094+0000","last_clean":"2024-03-18T18:49:48.840094+0000","last_became_active":"2024-03-18T18:49:44.837689+0000","last_became_peered":"2024-03-18T18:49:44.837689+0000","last_unstale":"2024-03-18T18:49:48.840094+0000","last_undegraded":"2024-03-18T18:49:48.840094+0000","last_fullsized":"2024-03-18T18:49:48.840094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:50.238592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975645+0000","last_change":"2024-03-18T18:49:44.841081+0000","last_active":"2024-03-18T18:49:48.975645+0000","last_peered":"2024-03-18T18:49:48.975645+0000","last_clean":"2024-03-18T18:49:48.975645+0000","last_became_active":"2024-03-18T18:49:44.840798+0000","last_became_peered":"2024-03-18T18:49:44.840798+0000","last_unstale":"2024-03-18T18:49:48.975645+0000","last_undegraded":"2024-03-18T18:49:48.975645+0000","last_fullsized":"2024-03-18T18:49:48.975645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:27:55.580010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839590+0000","last_change":"2024-03-18T18:49:44.836710+0000","last_active":"2024-03-18T18:49:48.839590+0000","last_peered":"2024-03-18T18:49:48.839590+0000","last_clean":"2024-03-18T18:49:48.839590+0000","last_became_active":"2024-03-18T18:49:44.836473+0000","last_became_peered":"2024-03-18T18:49:44.836473+0000","last_unstale":"2024-03-18T18:49:48.839590+0000","last_undegraded":"2024-03-18T18:49:48.839590+0000","last_fullsized":"2024-03-18T18:49:48.839590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:59:20.616256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976795+0000","last_change":"2024-03-18T18:49:44.829680+0000","last_active":"2024-03-18T18:49:48.976795+0000","last_peered":"2024-03-18T18:49:48.976795+0000","last_clean":"2024-03-18T18:49:48.976795+0000","last_became_active":"2024-03-18T18:49:44.829139+0000","last_became_peered":"2024-03-18T18:49:44.829139+0000","last_unstale":"2024-03-18T18:49:48.976795+0000","last_undegraded":"2024-03-18T18:49:48.976795+0000","last_fullsized":"2024-03-18T18:49:48.976795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:59:20.616256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839889+0000","last_change":"2024-03-18T18:49:44.837035+0000","last_active":"2024-03-18T18:49:48.839889+0000","last_peered":"2024-03-18T18:49:48.839889+0000","last_clean":"2024-03-18T18:49:48.839889+0000","last_became_active":"2024-03-18T18:49:44.836598+0000","last_became_peered":"2024-03-18T18:49:44.836598+0000","last_unstale":"2024-03-18T18:49:48.839889+0000","last_undegraded":"2024-03-18T18:49:48.839889+0000","last_fullsized":"2024-03-18T18:49:48.839889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:50.238592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976827+0000","last_change":"2024-03-18T18:49:44.835014+0000","last_active":"2024-03-18T18:49:48.976827+0000","last_peered":"2024-03-18T18:49:48.976827+0000","last_clean":"2024-03-18T18:49:48.976827+0000","last_became_active":"2024-03-18T18:49:44.834438+0000","last_became_peered":"2024-03-18T18:49:44.834438+0000","last_unstale":"2024-03-18T18:49:48.976827+0000","last_undegraded":"2024-03-18T18:49:48.976827+0000","last_fullsized":"2024-03-18T18:49:48.976827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:07.013090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835727+0000","last_change":"2024-03-18T18:49:44.837039+0000","last_active":"2024-03-18T18:49:48.835727+0000","last_peered":"2024-03-18T18:49:48.835727+0000","last_clean":"2024-03-18T18:49:48.835727+0000","last_became_active":"2024-03-18T18:49:44.836749+0000","last_became_peered":"2024-03-18T18:49:44.836749+0000","last_unstale":"2024-03-18T18:49:48.835727+0000","last_undegraded":"2024-03-18T18:49:48.835727+0000","last_fullsized":"2024-03-18T18:49:48.835727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836415+0000","last_change":"2024-03-18T18:49:44.832546+0000","last_active":"2024-03-18T18:49:48.836415+0000","last_peered":"2024-03-18T18:49:48.836415+0000","last_clean":"2024-03-18T18:49:48.836415+0000","last_became_active":"2024-03-18T18:49:44.832333+0000","last_became_peered":"2024-03-18T18:49:44.832333+0000","last_unstale":"2024-03-18T18:49:48.836415+0000","last_undegraded":"2024-03-18T18:49:48.836415+0000","last_fullsized":"2024-03-18T18:49:48.836415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842011+0000","last_change":"2024-03-18T18:49:44.834366+0000","last_active":"2024-03-18T18:49:48.842011+0000","last_peered":"2024-03-18T18:49:48.842011+0000","last_clean":"2024-03-18T18:49:48.842011+0000","last_became_active":"2024-03-18T18:49:44.833912+0000","last_became_peered":"2024-03-18T18:49:44.833912+0000","last_unstale":"2024-03-18T18:49:48.842011+0000","last_undegraded":"2024-03-18T18:49:48.842011+0000","last_fullsized":"2024-03-18T18:49:48.842011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:09:08.248420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839130+0000","last_change":"2024-03-18T18:49:44.823814+0000","last_active":"2024-03-18T18:49:48.839130+0000","last_peered":"2024-03-18T18:49:48.839130+0000","last_clean":"2024-03-18T18:49:48.839130+0000","last_became_active":"2024-03-18T18:49:44.823554+0000","last_became_peered":"2024-03-18T18:49:44.823554+0000","last_unstale":"2024-03-18T18:49:48.839130+0000","last_undegraded":"2024-03-18T18:49:48.839130+0000","last_fullsized":"2024-03-18T18:49:48.839130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:05.661348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853566+0000","last_change":"2024-03-18T18:49:44.828163+0000","last_active":"2024-03-18T18:49:48.853566+0000","last_peered":"2024-03-18T18:49:48.853566+0000","last_clean":"2024-03-18T18:49:48.853566+0000","last_became_active":"2024-03-18T18:49:44.827352+0000","last_became_peered":"2024-03-18T18:49:44.827352+0000","last_unstale":"2024-03-18T18:49:48.853566+0000","last_undegraded":"2024-03-18T18:49:48.853566+0000","last_fullsized":"2024-03-18T18:49:48.853566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:30.009397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840573+0000","last_change":"2024-03-18T18:49:44.839200+0000","last_active":"2024-03-18T18:49:48.840573+0000","last_peered":"2024-03-18T18:49:48.840573+0000","last_clean":"2024-03-18T18:49:48.840573+0000","last_became_active":"2024-03-18T18:49:44.838967+0000","last_became_peered":"2024-03-18T18:49:44.838967+0000","last_unstale":"2024-03-18T18:49:48.840573+0000","last_undegraded":"2024-03-18T18:49:48.840573+0000","last_fullsized":"2024-03-18T18:49:48.840573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:14.055199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839664+0000","last_change":"2024-03-18T18:49:44.834022+0000","last_active":"2024-03-18T18:49:48.839664+0000","last_peered":"2024-03-18T18:49:48.839664+0000","last_clean":"2024-03-18T18:49:48.839664+0000","last_became_active":"2024-03-18T18:49:44.833758+0000","last_became_peered":"2024-03-18T18:49:44.833758+0000","last_unstale":"2024-03-18T18:49:48.839664+0000","last_undegraded":"2024-03-18T18:49:48.839664+0000","last_fullsized":"2024-03-18T18:49:48.839664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:25:14.491414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840752+0000","last_change":"2024-03-18T18:49:44.842296+0000","last_active":"2024-03-18T18:49:48.840752+0000","last_peered":"2024-03-18T18:49:48.840752+0000","last_clean":"2024-03-18T18:49:48.840752+0000","last_became_active":"2024-03-18T18:49:44.841720+0000","last_became_peered":"2024-03-18T18:49:44.841720+0000","last_unstale":"2024-03-18T18:49:48.840752+0000","last_undegraded":"2024-03-18T18:49:48.840752+0000","last_fullsized":"2024-03-18T18:49:48.840752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836699+0000","last_change":"2024-03-18T18:49:44.828352+0000","last_active":"2024-03-18T18:49:48.836699+0000","last_peered":"2024-03-18T18:49:48.836699+0000","last_clean":"2024-03-18T18:49:48.836699+0000","last_became_active":"2024-03-18T18:49:44.828059+0000","last_became_peered":"2024-03-18T18:49:44.828059+0000","last_unstale":"2024-03-18T18:49:48.836699+0000","last_undegraded":"2024-03-18T18:49:48.836699+0000","last_fullsized":"2024-03-18T18:49:48.836699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:29.829759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976858+0000","last_change":"2024-03-18T18:49:44.841034+0000","last_active":"2024-03-18T18:49:48.976858+0000","last_peered":"2024-03-18T18:49:48.976858+0000","last_clean":"2024-03-18T18:49:48.976858+0000","last_became_active":"2024-03-18T18:49:44.840765+0000","last_became_peered":"2024-03-18T18:49:44.840765+0000","last_unstale":"2024-03-18T18:49:48.976858+0000","last_undegraded":"2024-03-18T18:49:48.976858+0000","last_fullsized":"2024-03-18T18:49:48.976858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:34.102274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.982406+0000","last_change":"2024-03-18T18:49:48.981322+0000","last_active":"2024-03-18T18:49:48.982406+0000","last_peered":"2024-03-18T18:49:48.982406+0000","last_clean":"2024-03-18T18:49:48.982406+0000","last_became_active":"2024-03-18T18:49:48.980927+0000","last_became_peered":"2024-03-18T18:49:48.980927+0000","last_unstale":"2024-03-18T18:49:48.982406+0000","last_undegraded":"2024-03-18T18:49:48.982406+0000","last_fullsized":"2024-03-18T18:49:48.982406+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:57:44.834422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.847587+0000","last_change":"2024-03-18T18:49:44.396090+0000","last_active":"2024-03-18T18:49:48.847587+0000","last_peered":"2024-03-18T18:49:48.847587+0000","last_clean":"2024-03-18T18:49:48.847587+0000","last_became_active":"2024-03-18T18:49:44.395823+0000","last_became_peered":"2024-03-18T18:49:44.395823+0000","last_unstale":"2024-03-18T18:49:48.847587+0000","last_undegraded":"2024-03-18T18:49:48.847587+0000","last_fullsized":"2024-03-18T18:49:48.847587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:00:40.075042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840491+0000","last_change":"2024-03-18T18:49:44.821381+0000","last_active":"2024-03-18T18:49:48.840491+0000","last_peered":"2024-03-18T18:49:48.840491+0000","last_clean":"2024-03-18T18:49:48.840491+0000","last_became_active":"2024-03-18T18:49:44.821129+0000","last_became_peered":"2024-03-18T18:49:44.821129+0000","last_unstale":"2024-03-18T18:49:48.840491+0000","last_undegraded":"2024-03-18T18:49:48.840491+0000","last_fullsized":"2024-03-18T18:49:48.840491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853754+0000","last_change":"2024-03-18T18:49:48.852436+0000","last_active":"2024-03-18T18:49:48.853754+0000","last_peered":"2024-03-18T18:49:48.853754+0000","last_clean":"2024-03-18T18:49:48.853754+0000","last_became_active":"2024-03-18T18:49:48.851959+0000","last_became_peered":"2024-03-18T18:49:48.851959+0000","last_unstale":"2024-03-18T18:49:48.853754+0000","last_undegraded":"2024-03-18T18:49:48.853754+0000","last_fullsized":"2024-03-18T18:49:48.853754+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:31:52.237923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840602+0000","last_change":"2024-03-18T18:49:44.399646+0000","last_active":"2024-03-18T18:49:48.840602+0000","last_peered":"2024-03-18T18:49:48.840602+0000","last_clean":"2024-03-18T18:49:48.840602+0000","last_became_active":"2024-03-18T18:49:44.399368+0000","last_became_peered":"2024-03-18T18:49:44.399368+0000","last_unstale":"2024-03-18T18:49:48.840602+0000","last_undegraded":"2024-03-18T18:49:48.840602+0000","last_fullsized":"2024-03-18T18:49:48.840602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:29.763642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.847482+0000","last_change":"2024-03-18T18:49:44.832977+0000","last_active":"2024-03-18T18:49:48.847482+0000","last_peered":"2024-03-18T18:49:48.847482+0000","last_clean":"2024-03-18T18:49:48.847482+0000","last_became_active":"2024-03-18T18:49:44.832624+0000","last_became_peered":"2024-03-18T18:49:44.832624+0000","last_unstale":"2024-03-18T18:49:48.847482+0000","last_undegraded":"2024-03-18T18:49:48.847482+0000","last_fullsized":"2024-03-18T18:49:48.847482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853697+0000","last_change":"2024-03-18T18:49:48.852585+0000","last_active":"2024-03-18T18:49:48.853697+0000","last_peered":"2024-03-18T18:49:48.853697+0000","last_clean":"2024-03-18T18:49:48.853697+0000","last_became_active":"2024-03-18T18:49:48.852137+0000","last_became_peered":"2024-03-18T18:49:48.852137+0000","last_unstale":"2024-03-18T18:49:48.853697+0000","last_undegraded":"2024-03-18T18:49:48.853697+0000","last_fullsized":"2024-03-18T18:49:48.853697+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:50:12.397097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975894+0000","last_change":"2024-03-18T18:49:44.394010+0000","last_active":"2024-03-18T18:49:48.975894+0000","last_peered":"2024-03-18T18:49:48.975894+0000","last_clean":"2024-03-18T18:49:48.975894+0000","last_became_active":"2024-03-18T18:49:44.393512+0000","last_became_peered":"2024-03-18T18:49:44.393512+0000","last_unstale":"2024-03-18T18:49:48.975894+0000","last_undegraded":"2024-03-18T18:49:48.975894+0000","last_fullsized":"2024-03-18T18:49:48.975894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:10:24.082597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839873+0000","last_change":"2024-03-18T18:49:44.836167+0000","last_active":"2024-03-18T18:49:48.839873+0000","last_peered":"2024-03-18T18:49:48.839873+0000","last_clean":"2024-03-18T18:49:48.839873+0000","last_became_active":"2024-03-18T18:49:44.835887+0000","last_became_peered":"2024-03-18T18:49:44.835887+0000","last_unstale":"2024-03-18T18:49:48.839873+0000","last_undegraded":"2024-03-18T18:49:48.839873+0000","last_fullsized":"2024-03-18T18:49:48.839873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:34.102274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.857961+0000","last_change":"2024-03-18T18:49:48.857276+0000","last_active":"2024-03-18T18:49:48.857961+0000","last_peered":"2024-03-18T18:49:48.857961+0000","last_clean":"2024-03-18T18:49:48.857961+0000","last_became_active":"2024-03-18T18:49:48.856870+0000","last_became_peered":"2024-03-18T18:49:48.856870+0000","last_unstale":"2024-03-18T18:49:48.857961+0000","last_undegraded":"2024-03-18T18:49:48.857961+0000","last_fullsized":"2024-03-18T18:49:48.857961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:31.426438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975669+0000","last_change":"2024-03-18T18:49:44.401760+0000","last_active":"2024-03-18T18:49:48.975669+0000","last_peered":"2024-03-18T18:49:48.975669+0000","last_clean":"2024-03-18T18:49:48.975669+0000","last_became_active":"2024-03-18T18:49:44.401565+0000","last_became_peered":"2024-03-18T18:49:44.401565+0000","last_unstale":"2024-03-18T18:49:48.975669+0000","last_undegraded":"2024-03-18T18:49:48.975669+0000","last_fullsized":"2024-03-18T18:49:48.975669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:09:07.245729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840474+0000","last_change":"2024-03-18T18:49:44.834985+0000","last_active":"2024-03-18T18:49:48.840474+0000","last_peered":"2024-03-18T18:49:48.840474+0000","last_clean":"2024-03-18T18:49:48.840474+0000","last_became_active":"2024-03-18T18:49:44.834719+0000","last_became_peered":"2024-03-18T18:49:44.834719+0000","last_unstale":"2024-03-18T18:49:48.840474+0000","last_undegraded":"2024-03-18T18:49:48.840474+0000","last_fullsized":"2024-03-18T18:49:48.840474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:27:55.580010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.854528+0000","last_change":"2024-03-18T18:49:48.852823+0000","last_active":"2024-03-18T18:49:48.854528+0000","last_peered":"2024-03-18T18:49:48.854528+0000","last_clean":"2024-03-18T18:49:48.854528+0000","last_became_active":"2024-03-18T18:49:48.852431+0000","last_became_peered":"2024-03-18T18:49:48.852431+0000","last_unstale":"2024-03-18T18:49:48.854528+0000","last_undegraded":"2024-03-18T18:49:48.854528+0000","last_fullsized":"2024-03-18T18:49:48.854528+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:10:04.268147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839292+0000","last_change":"2024-03-18T18:49:44.402292+0000","last_active":"2024-03-18T18:49:48.839292+0000","last_peered":"2024-03-18T18:49:48.839292+0000","last_clean":"2024-03-18T18:49:48.839292+0000","last_became_active":"2024-03-18T18:49:44.402028+0000","last_became_peered":"2024-03-18T18:49:44.402028+0000","last_unstale":"2024-03-18T18:49:48.839292+0000","last_undegraded":"2024-03-18T18:49:48.839292+0000","last_fullsized":"2024-03-18T18:49:48.839292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.843349+0000","last_change":"2024-03-18T18:49:44.832023+0000","last_active":"2024-03-18T18:49:48.843349+0000","last_peered":"2024-03-18T18:49:48.843349+0000","last_clean":"2024-03-18T18:49:48.843349+0000","last_became_active":"2024-03-18T18:49:44.831734+0000","last_became_peered":"2024-03-18T18:49:44.831734+0000","last_unstale":"2024-03-18T18:49:48.843349+0000","last_undegraded":"2024-03-18T18:49:48.843349+0000","last_fullsized":"2024-03-18T18:49:48.843349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:10:25.085288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.856842+0000","last_change":"2024-03-18T18:49:48.852370+0000","last_active":"2024-03-18T18:49:48.856842+0000","last_peered":"2024-03-18T18:49:48.856842+0000","last_clean":"2024-03-18T18:49:48.856842+0000","last_became_active":"2024-03-18T18:49:48.851953+0000","last_became_peered":"2024-03-18T18:49:48.851953+0000","last_unstale":"2024-03-18T18:49:48.856842+0000","last_undegraded":"2024-03-18T18:49:48.856842+0000","last_fullsized":"2024-03-18T18:49:48.856842+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:17.506065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836324+0000","last_change":"2024-03-18T18:49:44.394163+0000","last_active":"2024-03-18T18:49:48.836324+0000","last_peered":"2024-03-18T18:49:48.836324+0000","last_clean":"2024-03-18T18:49:48.836324+0000","last_became_active":"2024-03-18T18:49:44.393105+0000","last_became_peered":"2024-03-18T18:49:44.393105+0000","last_unstale":"2024-03-18T18:49:48.836324+0000","last_undegraded":"2024-03-18T18:49:48.836324+0000","last_fullsized":"2024-03-18T18:49:48.836324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:30.124822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.846218+0000","last_change":"2024-03-18T18:49:44.834950+0000","last_active":"2024-03-18T18:49:48.846218+0000","last_peered":"2024-03-18T18:49:48.846218+0000","last_clean":"2024-03-18T18:49:48.846218+0000","last_became_active":"2024-03-18T18:49:44.834765+0000","last_became_peered":"2024-03-18T18:49:44.834765+0000","last_unstale":"2024-03-18T18:49:48.846218+0000","last_undegraded":"2024-03-18T18:49:48.846218+0000","last_fullsized":"2024-03-18T18:49:48.846218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:49.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.857842+0000","last_change":"2024-03-18T18:49:48.857127+0000","last_active":"2024-03-18T18:49:48.857842+0000","last_peered":"2024-03-18T18:49:48.857842+0000","last_clean":"2024-03-18T18:49:48.857842+0000","last_became_active":"2024-03-18T18:49:48.856745+0000","last_became_peered":"2024-03-18T18:49:48.856745+0000","last_unstale":"2024-03-18T18:49:48.857842+0000","last_undegraded":"2024-03-18T18:49:48.857842+0000","last_fullsized":"2024-03-18T18:49:48.857842+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:46:22.763891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840897+0000","last_change":"2024-03-18T18:49:44.396730+0000","last_active":"2024-03-18T18:49:48.840897+0000","last_peered":"2024-03-18T18:49:48.840897+0000","last_clean":"2024-03-18T18:49:48.840897+0000","last_became_active":"2024-03-18T18:49:44.396415+0000","last_became_peered":"2024-03-18T18:49:44.396415+0000","last_unstale":"2024-03-18T18:49:48.840897+0000","last_undegraded":"2024-03-18T18:49:48.840897+0000","last_fullsized":"2024-03-18T18:49:48.840897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:28.338064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839641+0000","last_change":"2024-03-18T18:49:44.836855+0000","last_active":"2024-03-18T18:49:48.839641+0000","last_peered":"2024-03-18T18:49:48.839641+0000","last_clean":"2024-03-18T18:49:48.839641+0000","last_became_active":"2024-03-18T18:49:44.836275+0000","last_became_peered":"2024-03-18T18:49:44.836275+0000","last_unstale":"2024-03-18T18:49:48.839641+0000","last_undegraded":"2024-03-18T18:49:48.839641+0000","last_fullsized":"2024-03-18T18:49:48.839641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:49.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980433+0000","last_change":"2024-03-18T18:49:48.978139+0000","last_active":"2024-03-18T18:49:48.980433+0000","last_peered":"2024-03-18T18:49:48.980433+0000","last_clean":"2024-03-18T18:49:48.980433+0000","last_became_active":"2024-03-18T18:49:48.977755+0000","last_became_peered":"2024-03-18T18:49:48.977755+0000","last_unstale":"2024-03-18T18:49:48.980433+0000","last_undegraded":"2024-03-18T18:49:48.980433+0000","last_fullsized":"2024-03-18T18:49:48.980433+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:43.744272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839770+0000","last_change":"2024-03-18T18:49:44.400836+0000","last_active":"2024-03-18T18:49:48.839770+0000","last_peered":"2024-03-18T18:49:48.839770+0000","last_clean":"2024-03-18T18:49:48.839770+0000","last_became_active":"2024-03-18T18:49:44.400587+0000","last_became_peered":"2024-03-18T18:49:44.400587+0000","last_unstale":"2024-03-18T18:49:48.839770+0000","last_undegraded":"2024-03-18T18:49:48.839770+0000","last_fullsized":"2024-03-18T18:49:48.839770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:30.124822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840775+0000","last_change":"2024-03-18T18:49:44.822283+0000","last_active":"2024-03-18T18:49:48.840775+0000","last_peered":"2024-03-18T18:49:48.840775+0000","last_clean":"2024-03-18T18:49:48.840775+0000","last_became_active":"2024-03-18T18:49:44.822016+0000","last_became_peered":"2024-03-18T18:49:44.822016+0000","last_unstale":"2024-03-18T18:49:48.840775+0000","last_undegraded":"2024-03-18T18:49:48.840775+0000","last_fullsized":"2024-03-18T18:49:48.840775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981750+0000","last_change":"2024-03-18T18:49:48.978066+0000","last_active":"2024-03-18T18:49:48.981750+0000","last_peered":"2024-03-18T18:49:48.981750+0000","last_clean":"2024-03-18T18:49:48.981750+0000","last_became_active":"2024-03-18T18:49:48.977682+0000","last_became_peered":"2024-03-18T18:49:48.977682+0000","last_unstale":"2024-03-18T18:49:48.981750+0000","last_undegraded":"2024-03-18T18:49:48.981750+0000","last_fullsized":"2024-03-18T18:49:48.981750+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:31:52.237923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836616+0000","last_change":"2024-03-18T18:49:44.402381+0000","last_active":"2024-03-18T18:49:48.836616+0000","last_peered":"2024-03-18T18:49:48.836616+0000","last_clean":"2024-03-18T18:49:48.836616+0000","last_became_active":"2024-03-18T18:49:44.402101+0000","last_became_peered":"2024-03-18T18:49:44.402101+0000","last_unstale":"2024-03-18T18:49:48.836616+0000","last_undegraded":"2024-03-18T18:49:48.836616+0000","last_fullsized":"2024-03-18T18:49:48.836616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:39.899537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840754+0000","last_change":"2024-03-18T18:49:44.841434+0000","last_active":"2024-03-18T18:49:48.840754+0000","last_peered":"2024-03-18T18:49:48.840754+0000","last_clean":"2024-03-18T18:49:48.840754+0000","last_became_active":"2024-03-18T18:49:44.841186+0000","last_became_peered":"2024-03-18T18:49:44.841186+0000","last_unstale":"2024-03-18T18:49:48.840754+0000","last_undegraded":"2024-03-18T18:49:48.840754+0000","last_fullsized":"2024-03-18T18:49:48.840754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.850697+0000","last_change":"2024-03-18T18:49:48.849692+0000","last_active":"2024-03-18T18:49:48.850697+0000","last_peered":"2024-03-18T18:49:48.850697+0000","last_clean":"2024-03-18T18:49:48.850697+0000","last_became_active":"2024-03-18T18:49:48.849230+0000","last_became_peered":"2024-03-18T18:49:48.849230+0000","last_unstale":"2024-03-18T18:49:48.850697+0000","last_undegraded":"2024-03-18T18:49:48.850697+0000","last_fullsized":"2024-03-18T18:49:48.850697+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:45:34.390044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840676+0000","last_change":"2024-03-18T18:49:44.400484+0000","last_active":"2024-03-18T18:49:48.840676+0000","last_peered":"2024-03-18T18:49:48.840676+0000","last_clean":"2024-03-18T18:49:48.840676+0000","last_became_active":"2024-03-18T18:49:44.398334+0000","last_became_peered":"2024-03-18T18:49:44.398334+0000","last_unstale":"2024-03-18T18:49:48.840676+0000","last_undegraded":"2024-03-18T18:49:48.840676+0000","last_fullsized":"2024-03-18T18:49:48.840676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:29.763642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.847751+0000","last_change":"2024-03-18T18:49:44.832690+0000","last_active":"2024-03-18T18:49:48.847751+0000","last_peered":"2024-03-18T18:49:48.847751+0000","last_clean":"2024-03-18T18:49:48.847751+0000","last_became_active":"2024-03-18T18:49:44.832317+0000","last_became_peered":"2024-03-18T18:49:44.832317+0000","last_unstale":"2024-03-18T18:49:48.847751+0000","last_undegraded":"2024-03-18T18:49:48.847751+0000","last_fullsized":"2024-03-18T18:49:48.847751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:34.102274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.850171+0000","last_change":"2024-03-18T18:49:48.849391+0000","last_active":"2024-03-18T18:49:48.850171+0000","last_peered":"2024-03-18T18:49:48.850171+0000","last_clean":"2024-03-18T18:49:48.850171+0000","last_became_active":"2024-03-18T18:49:48.849018+0000","last_became_peered":"2024-03-18T18:49:48.849018+0000","last_unstale":"2024-03-18T18:49:48.850171+0000","last_undegraded":"2024-03-18T18:49:48.850171+0000","last_fullsized":"2024-03-18T18:49:48.850171+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:20:46.845872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841507+0000","last_change":"2024-03-18T18:49:44.402300+0000","last_active":"2024-03-18T18:49:48.841507+0000","last_peered":"2024-03-18T18:49:48.841507+0000","last_clean":"2024-03-18T18:49:48.841507+0000","last_became_active":"2024-03-18T18:49:44.401998+0000","last_became_peered":"2024-03-18T18:49:44.401998+0000","last_unstale":"2024-03-18T18:49:48.841507+0000","last_undegraded":"2024-03-18T18:49:48.841507+0000","last_fullsized":"2024-03-18T18:49:48.841507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:09:07.245729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841888+0000","last_change":"2024-03-18T18:49:44.835310+0000","last_active":"2024-03-18T18:49:48.841888+0000","last_peered":"2024-03-18T18:49:48.841888+0000","last_clean":"2024-03-18T18:49:48.841888+0000","last_became_active":"2024-03-18T18:49:44.835035+0000","last_became_peered":"2024-03-18T18:49:44.835035+0000","last_unstale":"2024-03-18T18:49:48.841888+0000","last_undegraded":"2024-03-18T18:49:48.841888+0000","last_fullsized":"2024-03-18T18:49:48.841888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:29.829759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.865513+0000","last_change":"2024-03-18T18:49:48.863905+0000","last_active":"2024-03-18T18:49:48.865513+0000","last_peered":"2024-03-18T18:49:48.865513+0000","last_clean":"2024-03-18T18:49:48.865513+0000","last_became_active":"2024-03-18T18:49:48.863234+0000","last_became_peered":"2024-03-18T18:49:48.863234+0000","last_unstale":"2024-03-18T18:49:48.865513+0000","last_undegraded":"2024-03-18T18:49:48.865513+0000","last_fullsized":"2024-03-18T18:49:48.865513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:16:08.575913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-18T18:49:54.442358+0000","last_change":"2024-03-18T18:49:44.402443+0000","last_active":"2024-03-18T18:49:54.442358+0000","last_peered":"2024-03-18T18:49:54.442358+0000","last_clean":"2024-03-18T18:49:54.442358+0000","last_became_active":"2024-03-18T18:49:44.402144+0000","last_became_peered":"2024-03-18T18:49:44.402144+0000","last_unstale":"2024-03-18T18:49:54.442358+0000","last_undegraded":"2024-03-18T18:49:54.442358+0000","last_fullsized":"2024-03-18T18:49:54.442358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:06.816608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3795,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976826+0000","last_change":"2024-03-18T18:49:44.829814+0000","last_active":"2024-03-18T18:49:48.976826+0000","last_peered":"2024-03-18T18:49:48.976826+0000","last_clean":"2024-03-18T18:49:48.976826+0000","last_became_active":"2024-03-18T18:49:44.829548+0000","last_became_peered":"2024-03-18T18:49:44.829548+0000","last_unstale":"2024-03-18T18:49:48.976826+0000","last_undegraded":"2024-03-18T18:49:48.976826+0000","last_fullsized":"2024-03-18T18:49:48.976826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:02:06.725321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980345+0000","last_change":"2024-03-18T18:49:48.978265+0000","last_active":"2024-03-18T18:49:48.980345+0000","last_peered":"2024-03-18T18:49:48.980345+0000","last_clean":"2024-03-18T18:49:48.980345+0000","last_became_active":"2024-03-18T18:49:48.977853+0000","last_became_peered":"2024-03-18T18:49:48.977853+0000","last_unstale":"2024-03-18T18:49:48.980345+0000","last_undegraded":"2024-03-18T18:49:48.980345+0000","last_fullsized":"2024-03-18T18:49:48.980345+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:43.744272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.846561+0000","last_change":"2024-03-18T18:49:44.392639+0000","last_active":"2024-03-18T18:49:48.846561+0000","last_peered":"2024-03-18T18:49:48.846561+0000","last_clean":"2024-03-18T18:49:48.846561+0000","last_became_active":"2024-03-18T18:49:44.392339+0000","last_became_peered":"2024-03-18T18:49:44.392339+0000","last_unstale":"2024-03-18T18:49:48.846561+0000","last_undegraded":"2024-03-18T18:49:48.846561+0000","last_fullsized":"2024-03-18T18:49:48.846561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:49.491669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853657+0000","last_change":"2024-03-18T18:49:44.836441+0000","last_active":"2024-03-18T18:49:48.853657+0000","last_peered":"2024-03-18T18:49:48.853657+0000","last_clean":"2024-03-18T18:49:48.853657+0000","last_became_active":"2024-03-18T18:49:44.836156+0000","last_became_peered":"2024-03-18T18:49:44.836156+0000","last_unstale":"2024-03-18T18:49:48.853657+0000","last_undegraded":"2024-03-18T18:49:48.853657+0000","last_fullsized":"2024-03-18T18:49:48.853657+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:27.913188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980346+0000","last_change":"2024-03-18T18:49:48.977065+0000","last_active":"2024-03-18T18:49:48.980346+0000","last_peered":"2024-03-18T18:49:48.980346+0000","last_clean":"2024-03-18T18:49:48.980346+0000","last_became_active":"2024-03-18T18:49:48.976568+0000","last_became_peered":"2024-03-18T18:49:48.976568+0000","last_unstale":"2024-03-18T18:49:48.980346+0000","last_undegraded":"2024-03-18T18:49:48.980346+0000","last_fullsized":"2024-03-18T18:49:48.980346+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:21.533140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839322+0000","last_change":"2024-03-18T18:49:44.404269+0000","last_active":"2024-03-18T18:49:48.839322+0000","last_peered":"2024-03-18T18:49:48.839322+0000","last_clean":"2024-03-18T18:49:48.839322+0000","last_became_active":"2024-03-18T18:49:44.403992+0000","last_became_peered":"2024-03-18T18:49:44.403992+0000","last_unstale":"2024-03-18T18:49:48.839322+0000","last_undegraded":"2024-03-18T18:49:48.839322+0000","last_fullsized":"2024-03-18T18:49:48.839322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975684+0000","last_change":"2024-03-18T18:49:44.834354+0000","last_active":"2024-03-18T18:49:48.975684+0000","last_peered":"2024-03-18T18:49:48.975684+0000","last_clean":"2024-03-18T18:49:48.975684+0000","last_became_active":"2024-03-18T18:49:44.834114+0000","last_became_peered":"2024-03-18T18:49:44.834114+0000","last_unstale":"2024-03-18T18:49:48.975684+0000","last_undegraded":"2024-03-18T18:49:48.975684+0000","last_fullsized":"2024-03-18T18:49:48.975684+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:09:46.686770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981775+0000","last_change":"2024-03-18T18:49:48.978466+0000","last_active":"2024-03-18T18:49:48.981775+0000","last_peered":"2024-03-18T18:49:48.981775+0000","last_clean":"2024-03-18T18:49:48.981775+0000","last_became_active":"2024-03-18T18:49:48.977877+0000","last_became_peered":"2024-03-18T18:49:48.977877+0000","last_unstale":"2024-03-18T18:49:48.981775+0000","last_undegraded":"2024-03-18T18:49:48.981775+0000","last_fullsized":"2024-03-18T18:49:48.981775+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:56.090012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836197+0000","last_change":"2024-03-18T18:49:44.393777+0000","last_active":"2024-03-18T18:49:48.836197+0000","last_peered":"2024-03-18T18:49:48.836197+0000","last_clean":"2024-03-18T18:49:48.836197+0000","last_became_active":"2024-03-18T18:49:44.392861+0000","last_became_peered":"2024-03-18T18:49:44.392861+0000","last_unstale":"2024-03-18T18:49:48.836197+0000","last_undegraded":"2024-03-18T18:49:48.836197+0000","last_fullsized":"2024-03-18T18:49:48.836197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835596+0000","last_change":"2024-03-18T18:49:44.831121+0000","last_active":"2024-03-18T18:49:48.835596+0000","last_peered":"2024-03-18T18:49:48.835596+0000","last_clean":"2024-03-18T18:49:48.835596+0000","last_became_active":"2024-03-18T18:49:44.830864+0000","last_became_peered":"2024-03-18T18:49:44.830864+0000","last_unstale":"2024-03-18T18:49:48.835596+0000","last_undegraded":"2024-03-18T18:49:48.835596+0000","last_fullsized":"2024-03-18T18:49:48.835596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:23.558014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.860292+0000","last_change":"2024-03-18T18:49:48.858618+0000","last_active":"2024-03-18T18:49:48.860292+0000","last_peered":"2024-03-18T18:49:48.860292+0000","last_clean":"2024-03-18T18:49:48.860292+0000","last_became_active":"2024-03-18T18:49:48.858170+0000","last_became_peered":"2024-03-18T18:49:48.858170+0000","last_unstale":"2024-03-18T18:49:48.860292+0000","last_undegraded":"2024-03-18T18:49:48.860292+0000","last_fullsized":"2024-03-18T18:49:48.860292+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:31.426438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836418+0000","last_change":"2024-03-18T18:49:44.397131+0000","last_active":"2024-03-18T18:49:48.836418+0000","last_peered":"2024-03-18T18:49:48.836418+0000","last_clean":"2024-03-18T18:49:48.836418+0000","last_became_active":"2024-03-18T18:49:44.396856+0000","last_became_peered":"2024-03-18T18:49:44.396856+0000","last_unstale":"2024-03-18T18:49:48.836418+0000","last_undegraded":"2024-03-18T18:49:48.836418+0000","last_fullsized":"2024-03-18T18:49:48.836418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976093+0000","last_change":"2024-03-18T18:49:44.838701+0000","last_active":"2024-03-18T18:49:48.976093+0000","last_peered":"2024-03-18T18:49:48.976093+0000","last_clean":"2024-03-18T18:49:48.976093+0000","last_became_active":"2024-03-18T18:49:44.838424+0000","last_became_peered":"2024-03-18T18:49:44.838424+0000","last_unstale":"2024-03-18T18:49:48.976093+0000","last_undegraded":"2024-03-18T18:49:48.976093+0000","last_fullsized":"2024-03-18T18:49:48.976093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:25:14.491414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853778+0000","last_change":"2024-03-18T18:49:48.852659+0000","last_active":"2024-03-18T18:49:48.853778+0000","last_peered":"2024-03-18T18:49:48.853778+0000","last_clean":"2024-03-18T18:49:48.853778+0000","last_became_active":"2024-03-18T18:49:48.852248+0000","last_became_peered":"2024-03-18T18:49:48.852248+0000","last_unstale":"2024-03-18T18:49:48.853778+0000","last_undegraded":"2024-03-18T18:49:48.853778+0000","last_fullsized":"2024-03-18T18:49:48.853778+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:38.261593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841901+0000","last_change":"2024-03-18T18:49:44.399701+0000","last_active":"2024-03-18T18:49:48.841901+0000","last_peered":"2024-03-18T18:49:48.841901+0000","last_clean":"2024-03-18T18:49:48.841901+0000","last_became_active":"2024-03-18T18:49:44.399371+0000","last_became_peered":"2024-03-18T18:49:44.399371+0000","last_unstale":"2024-03-18T18:49:48.841901+0000","last_undegraded":"2024-03-18T18:49:48.841901+0000","last_fullsized":"2024-03-18T18:49:48.841901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:49.491669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975930+0000","last_change":"2024-03-18T18:49:44.834635+0000","last_active":"2024-03-18T18:49:48.975930+0000","last_peered":"2024-03-18T18:49:48.975930+0000","last_clean":"2024-03-18T18:49:48.975930+0000","last_became_active":"2024-03-18T18:49:44.834321+0000","last_became_peered":"2024-03-18T18:49:44.834321+0000","last_unstale":"2024-03-18T18:49:48.975930+0000","last_undegraded":"2024-03-18T18:49:48.975930+0000","last_fullsized":"2024-03-18T18:49:48.975930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:50.238592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.865503+0000","last_change":"2024-03-18T18:49:48.863585+0000","last_active":"2024-03-18T18:49:48.865503+0000","last_peered":"2024-03-18T18:49:48.865503+0000","last_clean":"2024-03-18T18:49:48.865503+0000","last_became_active":"2024-03-18T18:49:48.862710+0000","last_became_peered":"2024-03-18T18:49:48.862710+0000","last_unstale":"2024-03-18T18:49:48.865503+0000","last_undegraded":"2024-03-18T18:49:48.865503+0000","last_fullsized":"2024-03-18T18:49:48.865503+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:31.426438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842830+0000","last_change":"2024-03-18T18:49:44.399251+0000","last_active":"2024-03-18T18:49:48.842830+0000","last_peered":"2024-03-18T18:49:48.842830+0000","last_clean":"2024-03-18T18:49:48.842830+0000","last_became_active":"2024-03-18T18:49:44.398969+0000","last_became_peered":"2024-03-18T18:49:44.398969+0000","last_unstale":"2024-03-18T18:49:48.842830+0000","last_undegraded":"2024-03-18T18:49:48.842830+0000","last_fullsized":"2024-03-18T18:49:48.842830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:39.899537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840553+0000","last_change":"2024-03-18T18:49:44.841934+0000","last_active":"2024-03-18T18:49:48.840553+0000","last_peered":"2024-03-18T18:49:48.840553+0000","last_clean":"2024-03-18T18:49:48.840553+0000","last_became_active":"2024-03-18T18:49:44.840657+0000","last_became_peered":"2024-03-18T18:49:44.840657+0000","last_unstale":"2024-03-18T18:49:48.840553+0000","last_undegraded":"2024-03-18T18:49:48.840553+0000","last_fullsized":"2024-03-18T18:49:48.840553+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.865367+0000","last_change":"2024-03-18T18:49:48.862242+0000","last_active":"2024-03-18T18:49:48.865367+0000","last_peered":"2024-03-18T18:49:48.865367+0000","last_clean":"2024-03-18T18:49:48.865367+0000","last_became_active":"2024-03-18T18:49:48.861779+0000","last_became_peered":"2024-03-18T18:49:48.861779+0000","last_unstale":"2024-03-18T18:49:48.865367+0000","last_undegraded":"2024-03-18T18:49:48.865367+0000","last_fullsized":"2024-03-18T18:49:48.865367+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:49:52.529190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839229+0000","last_change":"2024-03-18T18:49:44.404171+0000","last_active":"2024-03-18T18:49:48.839229+0000","last_peered":"2024-03-18T18:49:48.839229+0000","last_clean":"2024-03-18T18:49:48.839229+0000","last_became_active":"2024-03-18T18:49:44.403758+0000","last_became_peered":"2024-03-18T18:49:44.403758+0000","last_unstale":"2024-03-18T18:49:48.839229+0000","last_undegraded":"2024-03-18T18:49:48.839229+0000","last_fullsized":"2024-03-18T18:49:48.839229+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:55.666762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841896+0000","last_change":"2024-03-18T18:49:44.839405+0000","last_active":"2024-03-18T18:49:48.841896+0000","last_peered":"2024-03-18T18:49:48.841896+0000","last_clean":"2024-03-18T18:49:48.841896+0000","last_became_active":"2024-03-18T18:49:44.838946+0000","last_became_peered":"2024-03-18T18:49:44.838946+0000","last_unstale":"2024-03-18T18:49:48.841896+0000","last_undegraded":"2024-03-18T18:49:48.841896+0000","last_fullsized":"2024-03-18T18:49:48.841896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:07.819299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.865474+0000","last_change":"2024-03-18T18:49:48.863506+0000","last_active":"2024-03-18T18:49:48.865474+0000","last_peered":"2024-03-18T18:49:48.865474+0000","last_clean":"2024-03-18T18:49:48.865474+0000","last_became_active":"2024-03-18T18:49:48.863081+0000","last_became_peered":"2024-03-18T18:49:48.863081+0000","last_unstale":"2024-03-18T18:49:48.865474+0000","last_undegraded":"2024-03-18T18:49:48.865474+0000","last_fullsized":"2024-03-18T18:49:48.865474+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:47.284063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839273+0000","last_change":"2024-03-18T18:49:44.833724+0000","last_active":"2024-03-18T18:49:48.839273+0000","last_peered":"2024-03-18T18:49:48.839273+0000","last_clean":"2024-03-18T18:49:48.839273+0000","last_became_active":"2024-03-18T18:49:44.833231+0000","last_became_peered":"2024-03-18T18:49:44.833231+0000","last_unstale":"2024-03-18T18:49:48.839273+0000","last_undegraded":"2024-03-18T18:49:48.839273+0000","last_fullsized":"2024-03-18T18:49:48.839273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:30.009397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836484+0000","last_change":"2024-03-18T18:49:44.402405+0000","last_active":"2024-03-18T18:49:48.836484+0000","last_peered":"2024-03-18T18:49:48.836484+0000","last_clean":"2024-03-18T18:49:48.836484+0000","last_became_active":"2024-03-18T18:49:44.402133+0000","last_became_peered":"2024-03-18T18:49:44.402133+0000","last_unstale":"2024-03-18T18:49:48.836484+0000","last_undegraded":"2024-03-18T18:49:48.836484+0000","last_fullsized":"2024-03-18T18:49:48.836484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:29.763642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981114+0000","last_change":"2024-03-18T18:49:48.978691+0000","last_active":"2024-03-18T18:49:48.981114+0000","last_peered":"2024-03-18T18:49:48.981114+0000","last_clean":"2024-03-18T18:49:48.981114+0000","last_became_active":"2024-03-18T18:49:48.977741+0000","last_became_peered":"2024-03-18T18:49:48.977741+0000","last_unstale":"2024-03-18T18:49:48.981114+0000","last_undegraded":"2024-03-18T18:49:48.981114+0000","last_fullsized":"2024-03-18T18:49:48.981114+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:07:03.479681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840954+0000","last_change":"2024-03-18T18:49:44.833327+0000","last_active":"2024-03-18T18:49:48.840954+0000","last_peered":"2024-03-18T18:49:48.840954+0000","last_clean":"2024-03-18T18:49:48.840954+0000","last_became_active":"2024-03-18T18:49:44.833073+0000","last_became_peered":"2024-03-18T18:49:44.833073+0000","last_unstale":"2024-03-18T18:49:48.840954+0000","last_undegraded":"2024-03-18T18:49:48.840954+0000","last_fullsized":"2024-03-18T18:49:48.840954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:27:55.580010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839748+0000","last_change":"2024-03-18T18:49:44.401250+0000","last_active":"2024-03-18T18:49:48.839748+0000","last_peered":"2024-03-18T18:49:48.839748+0000","last_clean":"2024-03-18T18:49:48.839748+0000","last_became_active":"2024-03-18T18:49:44.398256+0000","last_became_peered":"2024-03-18T18:49:44.398256+0000","last_unstale":"2024-03-18T18:49:48.839748+0000","last_undegraded":"2024-03-18T18:49:48.839748+0000","last_fullsized":"2024-03-18T18:49:48.839748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:06.816608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":167,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975479+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T18:49:48.975479+0000","last_peered":"2024-03-18T18:49:48.975479+0000","last_clean":"2024-03-18T18:49:48.975479+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T18:49:48.975479+0000","last_undegraded":"2024-03-18T18:49:48.975479+0000","last_fullsized":"2024-03-18T18:49:48.975479+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980453+0000","last_change":"2024-03-18T18:49:48.978188+0000","last_active":"2024-03-18T18:49:48.980453+0000","last_peered":"2024-03-18T18:49:48.980453+0000","last_clean":"2024-03-18T18:49:48.980453+0000","last_became_active":"2024-03-18T18:49:48.977852+0000","last_became_peered":"2024-03-18T18:49:48.977852+0000","last_unstale":"2024-03-18T18:49:48.980453+0000","last_undegraded":"2024-03-18T18:49:48.980453+0000","last_fullsized":"2024-03-18T18:49:48.980453+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:24:38.261593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.848312+0000","last_change":"2024-03-18T18:49:44.839127+0000","last_active":"2024-03-18T18:49:48.848312+0000","last_peered":"2024-03-18T18:49:48.848312+0000","last_clean":"2024-03-18T18:49:48.848312+0000","last_became_active":"2024-03-18T18:49:44.838948+0000","last_became_peered":"2024-03-18T18:49:44.838948+0000","last_unstale":"2024-03-18T18:49:48.848312+0000","last_undegraded":"2024-03-18T18:49:48.848312+0000","last_fullsized":"2024-03-18T18:49:48.848312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:23.558014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975591+0000","last_change":"2024-03-18T18:49:44.402282+0000","last_active":"2024-03-18T18:49:48.975591+0000","last_peered":"2024-03-18T18:49:48.975591+0000","last_clean":"2024-03-18T18:49:48.975591+0000","last_became_active":"2024-03-18T18:49:44.402003+0000","last_became_peered":"2024-03-18T18:49:44.402003+0000","last_unstale":"2024-03-18T18:49:48.975591+0000","last_undegraded":"2024-03-18T18:49:48.975591+0000","last_fullsized":"2024-03-18T18:49:48.975591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:28.827068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981915+0000","last_change":"2024-03-18T18:49:48.978958+0000","last_active":"2024-03-18T18:49:48.981915+0000","last_peered":"2024-03-18T18:49:48.981915+0000","last_clean":"2024-03-18T18:49:48.981915+0000","last_became_active":"2024-03-18T18:49:48.978608+0000","last_became_peered":"2024-03-18T18:49:48.978608+0000","last_unstale":"2024-03-18T18:49:48.981915+0000","last_undegraded":"2024-03-18T18:49:48.981915+0000","last_fullsized":"2024-03-18T18:49:48.981915+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:08:46.162308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975789+0000","last_change":"2024-03-18T18:49:44.397512+0000","last_active":"2024-03-18T18:49:48.975789+0000","last_peered":"2024-03-18T18:49:48.975789+0000","last_clean":"2024-03-18T18:49:48.975789+0000","last_became_active":"2024-03-18T18:49:44.397271+0000","last_became_peered":"2024-03-18T18:49:44.397271+0000","last_unstale":"2024-03-18T18:49:48.975789+0000","last_undegraded":"2024-03-18T18:49:48.975789+0000","last_fullsized":"2024-03-18T18:49:48.975789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:55.666762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842833+0000","last_change":"2024-03-18T18:49:44.838715+0000","last_active":"2024-03-18T18:49:48.842833+0000","last_peered":"2024-03-18T18:49:48.842833+0000","last_clean":"2024-03-18T18:49:48.842833+0000","last_became_active":"2024-03-18T18:49:44.838452+0000","last_became_peered":"2024-03-18T18:49:44.838452+0000","last_unstale":"2024-03-18T18:49:48.842833+0000","last_undegraded":"2024-03-18T18:49:48.842833+0000","last_fullsized":"2024-03-18T18:49:48.842833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:23.558014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981853+0000","last_change":"2024-03-18T18:49:48.978525+0000","last_active":"2024-03-18T18:49:48.981853+0000","last_peered":"2024-03-18T18:49:48.981853+0000","last_clean":"2024-03-18T18:49:48.981853+0000","last_became_active":"2024-03-18T18:49:48.977651+0000","last_became_peered":"2024-03-18T18:49:48.977651+0000","last_unstale":"2024-03-18T18:49:48.981853+0000","last_undegraded":"2024-03-18T18:49:48.981853+0000","last_fullsized":"2024-03-18T18:49:48.981853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:31:53.057656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841616+0000","last_change":"2024-03-18T18:49:44.397630+0000","last_active":"2024-03-18T18:49:48.841616+0000","last_peered":"2024-03-18T18:49:48.841616+0000","last_clean":"2024-03-18T18:49:48.841616+0000","last_became_active":"2024-03-18T18:49:44.396306+0000","last_became_peered":"2024-03-18T18:49:44.396306+0000","last_unstale":"2024-03-18T18:49:48.841616+0000","last_undegraded":"2024-03-18T18:49:48.841616+0000","last_fullsized":"2024-03-18T18:49:48.841616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:10:24.082597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842664+0000","last_change":"2024-03-18T18:49:44.821273+0000","last_active":"2024-03-18T18:49:48.842664+0000","last_peered":"2024-03-18T18:49:48.842664+0000","last_clean":"2024-03-18T18:49:48.842664+0000","last_became_active":"2024-03-18T18:49:44.821036+0000","last_became_peered":"2024-03-18T18:49:44.821036+0000","last_unstale":"2024-03-18T18:49:48.842664+0000","last_undegraded":"2024-03-18T18:49:48.842664+0000","last_fullsized":"2024-03-18T18:49:48.842664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:30.009397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.982011+0000","last_change":"2024-03-18T18:49:48.978130+0000","last_active":"2024-03-18T18:49:48.982011+0000","last_peered":"2024-03-18T18:49:48.982011+0000","last_clean":"2024-03-18T18:49:48.982011+0000","last_became_active":"2024-03-18T18:49:48.977715+0000","last_became_peered":"2024-03-18T18:49:48.977715+0000","last_unstale":"2024-03-18T18:49:48.982011+0000","last_undegraded":"2024-03-18T18:49:48.982011+0000","last_fullsized":"2024-03-18T18:49:48.982011+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:47.284063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836478+0000","last_change":"2024-03-18T18:49:44.394139+0000","last_active":"2024-03-18T18:49:48.836478+0000","last_peered":"2024-03-18T18:49:48.836478+0000","last_clean":"2024-03-18T18:49:48.836478+0000","last_became_active":"2024-03-18T18:49:44.393405+0000","last_became_peered":"2024-03-18T18:49:44.393405+0000","last_unstale":"2024-03-18T18:49:48.836478+0000","last_undegraded":"2024-03-18T18:49:48.836478+0000","last_fullsized":"2024-03-18T18:49:48.836478+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:55.666762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853791+0000","last_change":"2024-03-18T18:49:44.829271+0000","last_active":"2024-03-18T18:49:48.853791+0000","last_peered":"2024-03-18T18:49:48.853791+0000","last_clean":"2024-03-18T18:49:48.853791+0000","last_became_active":"2024-03-18T18:49:44.829019+0000","last_became_peered":"2024-03-18T18:49:44.829019+0000","last_unstale":"2024-03-18T18:49:48.853791+0000","last_undegraded":"2024-03-18T18:49:48.853791+0000","last_fullsized":"2024-03-18T18:49:48.853791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853721+0000","last_change":"2024-03-18T18:49:48.852590+0000","last_active":"2024-03-18T18:49:48.853721+0000","last_peered":"2024-03-18T18:49:48.853721+0000","last_clean":"2024-03-18T18:49:48.853721+0000","last_became_active":"2024-03-18T18:49:48.852088+0000","last_became_peered":"2024-03-18T18:49:48.852088+0000","last_unstale":"2024-03-18T18:49:48.853721+0000","last_undegraded":"2024-03-18T18:49:48.853721+0000","last_fullsized":"2024-03-18T18:49:48.853721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:16:08.575913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975464+0000","last_change":"2024-03-18T18:49:44.393305+0000","last_active":"2024-03-18T18:49:48.975464+0000","last_peered":"2024-03-18T18:49:48.975464+0000","last_clean":"2024-03-18T18:49:48.975464+0000","last_became_active":"2024-03-18T18:49:44.392494+0000","last_became_peered":"2024-03-18T18:49:44.392494+0000","last_unstale":"2024-03-18T18:49:48.975464+0000","last_undegraded":"2024-03-18T18:49:48.975464+0000","last_fullsized":"2024-03-18T18:49:48.975464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:06.816608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841483+0000","last_change":"2024-03-18T18:49:44.837726+0000","last_active":"2024-03-18T18:49:48.841483+0000","last_peered":"2024-03-18T18:49:48.841483+0000","last_clean":"2024-03-18T18:49:48.841483+0000","last_became_active":"2024-03-18T18:49:44.837364+0000","last_became_peered":"2024-03-18T18:49:44.837364+0000","last_unstale":"2024-03-18T18:49:48.841483+0000","last_undegraded":"2024-03-18T18:49:48.841483+0000","last_fullsized":"2024-03-18T18:49:48.841483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:40.902228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.982312+0000","last_change":"2024-03-18T18:49:48.980106+0000","last_active":"2024-03-18T18:49:48.982312+0000","last_peered":"2024-03-18T18:49:48.982312+0000","last_clean":"2024-03-18T18:49:48.982312+0000","last_became_active":"2024-03-18T18:49:48.979709+0000","last_became_peered":"2024-03-18T18:49:48.979709+0000","last_unstale":"2024-03-18T18:49:48.982312+0000","last_undegraded":"2024-03-18T18:49:48.982312+0000","last_fullsized":"2024-03-18T18:49:48.982312+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:21.533140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840674+0000","last_change":"2024-03-18T18:49:44.398886+0000","last_active":"2024-03-18T18:49:48.840674+0000","last_peered":"2024-03-18T18:49:48.840674+0000","last_clean":"2024-03-18T18:49:48.840674+0000","last_became_active":"2024-03-18T18:49:44.398578+0000","last_became_peered":"2024-03-18T18:49:44.398578+0000","last_unstale":"2024-03-18T18:49:48.840674+0000","last_undegraded":"2024-03-18T18:49:48.840674+0000","last_fullsized":"2024-03-18T18:49:48.840674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:55.354561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839766+0000","last_change":"2024-03-18T18:49:44.826751+0000","last_active":"2024-03-18T18:49:48.839766+0000","last_peered":"2024-03-18T18:49:48.839766+0000","last_clean":"2024-03-18T18:49:48.839766+0000","last_became_active":"2024-03-18T18:49:44.825975+0000","last_became_peered":"2024-03-18T18:49:44.825975+0000","last_unstale":"2024-03-18T18:49:48.839766+0000","last_undegraded":"2024-03-18T18:49:48.839766+0000","last_fullsized":"2024-03-18T18:49:48.839766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:02:06.725321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.853727+0000","last_change":"2024-03-18T18:49:48.852554+0000","last_active":"2024-03-18T18:49:48.853727+0000","last_peered":"2024-03-18T18:49:48.853727+0000","last_clean":"2024-03-18T18:49:48.853727+0000","last_became_active":"2024-03-18T18:49:48.852121+0000","last_became_peered":"2024-03-18T18:49:48.852121+0000","last_unstale":"2024-03-18T18:49:48.853727+0000","last_undegraded":"2024-03-18T18:49:48.853727+0000","last_fullsized":"2024-03-18T18:49:48.853727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:41:26.757218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841024+0000","last_change":"2024-03-18T18:49:44.402526+0000","last_active":"2024-03-18T18:49:48.841024+0000","last_peered":"2024-03-18T18:49:48.841024+0000","last_clean":"2024-03-18T18:49:48.841024+0000","last_became_active":"2024-03-18T18:49:44.402277+0000","last_became_peered":"2024-03-18T18:49:44.402277+0000","last_unstale":"2024-03-18T18:49:48.841024+0000","last_undegraded":"2024-03-18T18:49:48.841024+0000","last_fullsized":"2024-03-18T18:49:48.841024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:06.816608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839383+0000","last_change":"2024-03-18T18:49:44.830629+0000","last_active":"2024-03-18T18:49:48.839383+0000","last_peered":"2024-03-18T18:49:48.839383+0000","last_clean":"2024-03-18T18:49:48.839383+0000","last_became_active":"2024-03-18T18:49:44.830313+0000","last_became_peered":"2024-03-18T18:49:44.830313+0000","last_unstale":"2024-03-18T18:49:48.839383+0000","last_undegraded":"2024-03-18T18:49:48.839383+0000","last_fullsized":"2024-03-18T18:49:48.839383+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:27.913188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.860287+0000","last_change":"2024-03-18T18:49:48.859810+0000","last_active":"2024-03-18T18:49:48.860287+0000","last_peered":"2024-03-18T18:49:48.860287+0000","last_clean":"2024-03-18T18:49:48.860287+0000","last_became_active":"2024-03-18T18:49:48.859453+0000","last_became_peered":"2024-03-18T18:49:48.859453+0000","last_unstale":"2024-03-18T18:49:48.860287+0000","last_undegraded":"2024-03-18T18:49:48.860287+0000","last_fullsized":"2024-03-18T18:49:48.860287+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:11:09.238467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.975758+0000","last_change":"2024-03-18T18:49:44.396856+0000","last_active":"2024-03-18T18:49:48.975758+0000","last_peered":"2024-03-18T18:49:48.975758+0000","last_clean":"2024-03-18T18:49:48.975758+0000","last_became_active":"2024-03-18T18:49:44.395207+0000","last_became_peered":"2024-03-18T18:49:44.395207+0000","last_unstale":"2024-03-18T18:49:48.975758+0000","last_undegraded":"2024-03-18T18:49:48.975758+0000","last_fullsized":"2024-03-18T18:49:48.975758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:39.899537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836368+0000","last_change":"2024-03-18T18:49:44.836659+0000","last_active":"2024-03-18T18:49:48.836368+0000","last_peered":"2024-03-18T18:49:48.836368+0000","last_clean":"2024-03-18T18:49:48.836368+0000","last_became_active":"2024-03-18T18:49:44.836362+0000","last_became_peered":"2024-03-18T18:49:44.836362+0000","last_unstale":"2024-03-18T18:49:48.836368+0000","last_undegraded":"2024-03-18T18:49:48.836368+0000","last_fullsized":"2024-03-18T18:49:48.836368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:49.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.980755+0000","last_change":"2024-03-18T18:49:48.978204+0000","last_active":"2024-03-18T18:49:48.980755+0000","last_peered":"2024-03-18T18:49:48.980755+0000","last_clean":"2024-03-18T18:49:48.980755+0000","last_became_active":"2024-03-18T18:49:48.977745+0000","last_became_peered":"2024-03-18T18:49:48.977745+0000","last_unstale":"2024-03-18T18:49:48.980755+0000","last_undegraded":"2024-03-18T18:49:48.980755+0000","last_fullsized":"2024-03-18T18:49:48.980755+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:08:46.162308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.837352+0000","last_change":"2024-03-18T18:49:44.398731+0000","last_active":"2024-03-18T18:49:48.837352+0000","last_peered":"2024-03-18T18:49:48.837352+0000","last_clean":"2024-03-18T18:49:48.837352+0000","last_became_active":"2024-03-18T18:49:44.398448+0000","last_became_peered":"2024-03-18T18:49:44.398448+0000","last_unstale":"2024-03-18T18:49:48.837352+0000","last_undegraded":"2024-03-18T18:49:48.837352+0000","last_fullsized":"2024-03-18T18:49:48.837352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:30.124822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976056+0000","last_change":"2024-03-18T18:49:44.833300+0000","last_active":"2024-03-18T18:49:48.976056+0000","last_peered":"2024-03-18T18:49:48.976056+0000","last_clean":"2024-03-18T18:49:48.976056+0000","last_became_active":"2024-03-18T18:49:44.833027+0000","last_became_peered":"2024-03-18T18:49:44.833027+0000","last_unstale":"2024-03-18T18:49:48.976056+0000","last_undegraded":"2024-03-18T18:49:48.976056+0000","last_fullsized":"2024-03-18T18:49:48.976056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:11:05.787600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.860344+0000","last_change":"2024-03-18T18:49:48.859259+0000","last_active":"2024-03-18T18:49:48.860344+0000","last_peered":"2024-03-18T18:49:48.860344+0000","last_clean":"2024-03-18T18:49:48.860344+0000","last_became_active":"2024-03-18T18:49:48.857100+0000","last_became_peered":"2024-03-18T18:49:48.857100+0000","last_unstale":"2024-03-18T18:49:48.860344+0000","last_undegraded":"2024-03-18T18:49:48.860344+0000","last_fullsized":"2024-03-18T18:49:48.860344+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:47.284063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841409+0000","last_change":"2024-03-18T18:49:44.395491+0000","last_active":"2024-03-18T18:49:48.841409+0000","last_peered":"2024-03-18T18:49:48.841409+0000","last_clean":"2024-03-18T18:49:48.841409+0000","last_became_active":"2024-03-18T18:49:44.394529+0000","last_became_peered":"2024-03-18T18:49:44.394529+0000","last_unstale":"2024-03-18T18:49:48.841409+0000","last_undegraded":"2024-03-18T18:49:48.841409+0000","last_fullsized":"2024-03-18T18:49:48.841409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:55.666762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839957+0000","last_change":"2024-03-18T18:49:44.838466+0000","last_active":"2024-03-18T18:49:48.839957+0000","last_peered":"2024-03-18T18:49:48.839957+0000","last_clean":"2024-03-18T18:49:48.839957+0000","last_became_active":"2024-03-18T18:49:44.838027+0000","last_became_peered":"2024-03-18T18:49:44.838027+0000","last_unstale":"2024-03-18T18:49:48.839957+0000","last_undegraded":"2024-03-18T18:49:48.839957+0000","last_fullsized":"2024-03-18T18:49:48.839957+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:09:46.686770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.981225+0000","last_change":"2024-03-18T18:49:48.979431+0000","last_active":"2024-03-18T18:49:48.981225+0000","last_peered":"2024-03-18T18:49:48.981225+0000","last_clean":"2024-03-18T18:49:48.981225+0000","last_became_active":"2024-03-18T18:49:48.978587+0000","last_became_peered":"2024-03-18T18:49:48.978587+0000","last_unstale":"2024-03-18T18:49:48.981225+0000","last_undegraded":"2024-03-18T18:49:48.981225+0000","last_fullsized":"2024-03-18T18:49:48.981225+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:47.820365+0000","last_clean_scrub_stamp":"2024-03-18T18:49:47.820365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:47.284063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836616+0000","last_change":"2024-03-18T18:49:44.403374+0000","last_active":"2024-03-18T18:49:48.836616+0000","last_peered":"2024-03-18T18:49:48.836616+0000","last_clean":"2024-03-18T18:49:48.836616+0000","last_became_active":"2024-03-18T18:49:44.403082+0000","last_became_peered":"2024-03-18T18:49:44.403082+0000","last_unstale":"2024-03-18T18:49:48.836616+0000","last_undegraded":"2024-03-18T18:49:48.836616+0000","last_fullsized":"2024-03-18T18:49:48.836616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:43.366807+0000","last_clean_scrub_stamp":"2024-03-18T18:49:43.366807+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:28.338064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.976741+0000","last_change":"2024-03-18T18:49:44.835166+0000","last_active":"2024-03-18T18:49:48.976741+0000","last_peered":"2024-03-18T18:49:48.976741+0000","last_clean":"2024-03-18T18:49:48.976741+0000","last_became_active":"2024-03-18T18:49:44.834808+0000","last_became_peered":"2024-03-18T18:49:44.834808+0000","last_unstale":"2024-03-18T18:49:48.976741+0000","last_undegraded":"2024-03-18T18:49:48.976741+0000","last_fullsized":"2024-03-18T18:49:48.976741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:20.816344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835598+0000","last_change":"2024-03-18T18:49:44.829141+0000","last_active":"2024-03-18T18:49:48.835598+0000","last_peered":"2024-03-18T18:49:48.835598+0000","last_clean":"2024-03-18T18:49:48.835598+0000","last_became_active":"2024-03-18T18:49:44.828863+0000","last_became_peered":"2024-03-18T18:49:44.828863+0000","last_unstale":"2024-03-18T18:49:48.835598+0000","last_undegraded":"2024-03-18T18:49:48.835598+0000","last_fullsized":"2024-03-18T18:49:48.835598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:30.009397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.836583+0000","last_change":"2024-03-18T18:49:44.831132+0000","last_active":"2024-03-18T18:49:48.836583+0000","last_peered":"2024-03-18T18:49:48.836583+0000","last_clean":"2024-03-18T18:49:48.836583+0000","last_became_active":"2024-03-18T18:49:44.830873+0000","last_became_peered":"2024-03-18T18:49:44.830873+0000","last_unstale":"2024-03-18T18:49:48.836583+0000","last_undegraded":"2024-03-18T18:49:48.836583+0000","last_fullsized":"2024-03-18T18:49:48.836583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:09:08.248420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839404+0000","last_change":"2024-03-18T18:49:44.841166+0000","last_active":"2024-03-18T18:49:48.839404+0000","last_peered":"2024-03-18T18:49:48.839404+0000","last_clean":"2024-03-18T18:49:48.839404+0000","last_became_active":"2024-03-18T18:49:44.840477+0000","last_became_peered":"2024-03-18T18:49:44.840477+0000","last_unstale":"2024-03-18T18:49:48.839404+0000","last_undegraded":"2024-03-18T18:49:48.839404+0000","last_fullsized":"2024-03-18T18:49:48.839404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:20.816344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.846309+0000","last_change":"2024-03-18T18:49:44.838132+0000","last_active":"2024-03-18T18:49:48.846309+0000","last_peered":"2024-03-18T18:49:48.846309+0000","last_clean":"2024-03-18T18:49:48.846309+0000","last_became_active":"2024-03-18T18:49:44.837914+0000","last_became_peered":"2024-03-18T18:49:44.837914+0000","last_unstale":"2024-03-18T18:49:48.846309+0000","last_undegraded":"2024-03-18T18:49:48.846309+0000","last_fullsized":"2024-03-18T18:49:48.846309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840730+0000","last_change":"2024-03-18T18:49:44.835464+0000","last_active":"2024-03-18T18:49:48.840730+0000","last_peered":"2024-03-18T18:49:48.840730+0000","last_clean":"2024-03-18T18:49:48.840730+0000","last_became_active":"2024-03-18T18:49:44.835190+0000","last_became_peered":"2024-03-18T18:49:44.835190+0000","last_unstale":"2024-03-18T18:49:48.840730+0000","last_undegraded":"2024-03-18T18:49:48.840730+0000","last_fullsized":"2024-03-18T18:49:48.840730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:28.062413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.841309+0000","last_change":"2024-03-18T18:49:44.819962+0000","last_active":"2024-03-18T18:49:48.841309+0000","last_peered":"2024-03-18T18:49:48.841309+0000","last_clean":"2024-03-18T18:49:48.841309+0000","last_became_active":"2024-03-18T18:49:44.819698+0000","last_became_peered":"2024-03-18T18:49:44.819698+0000","last_unstale":"2024-03-18T18:49:48.841309+0000","last_undegraded":"2024-03-18T18:49:48.841309+0000","last_fullsized":"2024-03-18T18:49:48.841309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:59.165561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835920+0000","last_change":"2024-03-18T18:49:44.830928+0000","last_active":"2024-03-18T18:49:48.835920+0000","last_peered":"2024-03-18T18:49:48.835920+0000","last_clean":"2024-03-18T18:49:48.835920+0000","last_became_active":"2024-03-18T18:49:44.830658+0000","last_became_peered":"2024-03-18T18:49:44.830658+0000","last_unstale":"2024-03-18T18:49:48.835920+0000","last_undegraded":"2024-03-18T18:49:48.835920+0000","last_fullsized":"2024-03-18T18:49:48.835920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.842763+0000","last_change":"2024-03-18T18:49:44.838837+0000","last_active":"2024-03-18T18:49:48.842763+0000","last_peered":"2024-03-18T18:49:48.842763+0000","last_clean":"2024-03-18T18:49:48.842763+0000","last_became_active":"2024-03-18T18:49:44.838564+0000","last_became_peered":"2024-03-18T18:49:44.838564+0000","last_unstale":"2024-03-18T18:49:48.842763+0000","last_undegraded":"2024-03-18T18:49:48.842763+0000","last_fullsized":"2024-03-18T18:49:48.842763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:34.102274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839856+0000","last_change":"2024-03-18T18:49:44.841576+0000","last_active":"2024-03-18T18:49:48.839856+0000","last_peered":"2024-03-18T18:49:48.839856+0000","last_clean":"2024-03-18T18:49:48.839856+0000","last_became_active":"2024-03-18T18:49:44.840814+0000","last_became_peered":"2024-03-18T18:49:44.840814+0000","last_unstale":"2024-03-18T18:49:48.839856+0000","last_undegraded":"2024-03-18T18:49:48.839856+0000","last_fullsized":"2024-03-18T18:49:48.839856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:02:06.725321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.840423+0000","last_change":"2024-03-18T18:49:44.840212+0000","last_active":"2024-03-18T18:49:48.840423+0000","last_peered":"2024-03-18T18:49:48.840423+0000","last_clean":"2024-03-18T18:49:48.840423+0000","last_became_active":"2024-03-18T18:49:44.839960+0000","last_became_peered":"2024-03-18T18:49:44.839960+0000","last_unstale":"2024-03-18T18:49:48.840423+0000","last_undegraded":"2024-03-18T18:49:48.840423+0000","last_fullsized":"2024-03-18T18:49:48.840423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:31:02.175653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.839485+0000","last_change":"2024-03-18T18:49:44.841770+0000","last_active":"2024-03-18T18:49:48.839485+0000","last_peered":"2024-03-18T18:49:48.839485+0000","last_clean":"2024-03-18T18:49:48.839485+0000","last_became_active":"2024-03-18T18:49:44.840939+0000","last_became_peered":"2024-03-18T18:49:44.840939+0000","last_unstale":"2024-03-18T18:49:48.839485+0000","last_undegraded":"2024-03-18T18:49:48.839485+0000","last_fullsized":"2024-03-18T18:49:48.839485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:20:43.395005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.846397+0000","last_change":"2024-03-18T18:49:44.841290+0000","last_active":"2024-03-18T18:49:48.846397+0000","last_peered":"2024-03-18T18:49:48.846397+0000","last_clean":"2024-03-18T18:49:48.846397+0000","last_became_active":"2024-03-18T18:49:44.840949+0000","last_became_peered":"2024-03-18T18:49:44.840949+0000","last_unstale":"2024-03-18T18:49:48.846397+0000","last_undegraded":"2024-03-18T18:49:48.846397+0000","last_fullsized":"2024-03-18T18:49:48.846397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:05.661348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-18T18:49:48.835876+0000","last_change":"2024-03-18T18:49:44.837671+0000","last_active":"2024-03-18T18:49:48.835876+0000","last_peered":"2024-03-18T18:49:48.835876+0000","last_clean":"2024-03-18T18:49:48.835876+0000","last_became_active":"2024-03-18T18:49:44.837265+0000","last_became_peered":"2024-03-18T18:49:44.837265+0000","last_unstale":"2024-03-18T18:49:48.835876+0000","last_undegraded":"2024-03-18T18:49:48.835876+0000","last_fullsized":"2024-03-18T18:49:48.835876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:44.369498+0000","last_clean_scrub_stamp":"2024-03-18T18:49:44.369498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:14.055199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4593,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":695401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":104611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":695457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":104451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":104577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":104541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T18:50:14.227 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-18T18:50:14.227 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-18T18:50:14.228 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-18T18:50:14.228 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-18T18:50:14.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.389+0000 7f2fdf654700 1 Processor -- start 2024-03-18T18:50:14.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.390+0000 7f2fdf654700 1 -- start start 2024-03-18T18:50:14.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.390+0000 7f2fdf654700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2fd80d1490 0x7f2fd80d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:14.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.390+0000 7f2fdf654700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd80b2c60 con 0x7f2fd815d740 2024-03-18T18:50:14.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.390+0000 7f2fdf654700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd80d2280 con 0x7f2fd80d1e50 2024-03-18T18:50:14.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.390+0000 7f2fdf654700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd815db20 con 0x7f2fd80d1490 2024-03-18T18:50:14.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.390+0000 7f2fdd3f0700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2fd815d740 0x7f2fd80d3a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55842/0 (socket says 172.21.15.1:55842) 2024-03-18T18:50:14.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.390+0000 7f2fdd3f0700 1 -- 172.21.15.1:0/3423050477 learned_addr learned my addr 172.21.15.1:0/3423050477 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:14.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fdcbef700 1 --2- 172.21.15.1:0/3423050477 >> v2:172.21.15.165:3300/0 conn(0x7f2fd80d1490 0x7f2fd80d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:14.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fcbfff700 1 -- 172.21.15.1:0/3423050477 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690986406 0 0) 0x7f2fd80b2c60 con 0x7f2fd815d740 2024-03-18T18:50:14.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fcbfff700 1 -- 172.21.15.1:0/3423050477 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc40035b0 con 0x7f2fd815d740 2024-03-18T18:50:14.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fcbfff700 1 -- 172.21.15.1:0/3423050477 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509539143 0 0) 0x7f2fd80d2280 con 0x7f2fd80d1e50 2024-03-18T18:50:14.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fcbfff700 1 -- 172.21.15.1:0/3423050477 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd80b2c60 con 0x7f2fd80d1e50 2024-03-18T18:50:14.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fcbfff700 1 -- 172.21.15.1:0/3423050477 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3678798858 0 0) 0x7f2fc40035b0 con 0x7f2fd815d740 2024-03-18T18:50:14.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fcbfff700 1 -- 172.21.15.1:0/3423050477 >> v2:172.21.15.165:3300/0 conn(0x7f2fd80d1490 msgr2=0x7f2fd80d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:14.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fcbfff700 1 --2- 172.21.15.1:0/3423050477 >> v2:172.21.15.165:3300/0 conn(0x7f2fd80d1490 0x7f2fd80d1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fcbfff700 1 -- 172.21.15.1:0/3423050477 >> v1:172.21.15.165:6789/0 conn(0x7f2fd80d1e50 legacy=0x7f2fd80b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:14.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fcbfff700 1 -- 172.21.15.1:0/3423050477 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd80cea80 con 0x7f2fd815d740 2024-03-18T18:50:14.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fcbfff700 1 -- 172.21.15.1:0/3423050477 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f2fd4001670 con 0x7f2fd815d740 2024-03-18T18:50:14.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fcbfff700 1 -- 172.21.15.1:0/3423050477 >> v1:172.21.15.1:6789/0 conn(0x7f2fd815d740 legacy=0x7f2fd80d3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:14.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fcbfff700 1 --2- 172.21.15.1:0/3423050477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fc4003be0 0x7f2fc4003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:14.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fcbfff700 1 --2- 172.21.15.1:0/3423050477 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fc4004850 0x7f2fc4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:14.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fcbfff700 1 --2- 172.21.15.1:0/3423050477 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fc4005260 0x7f2fc400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:14.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fcbfff700 1 -- 172.21.15.1:0/3423050477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd815db20 con 0x7f2fc4003be0 2024-03-18T18:50:14.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fcbfff700 1 -- 172.21.15.1:0/3423050477 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc400c190 con 0x7f2fc4004850 2024-03-18T18:50:14.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fcbfff700 1 -- 172.21.15.1:0/3423050477 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc400c350 con 0x7f2fc4005260 2024-03-18T18:50:14.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fdd3f0700 1 --2- 172.21.15.1:0/3423050477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fc4003be0 0x7f2fc4003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:14.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fdcbef700 1 --2- 172.21.15.1:0/3423050477 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fc4004850 0x7f2fc4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:14.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fddbf1700 1 --2- 172.21.15.1:0/3423050477 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fc4005260 0x7f2fc400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:14.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fdd3f0700 1 -- 172.21.15.1:0/3423050477 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fc4005260 msgr2=0x7f2fc400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:14.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fdd3f0700 1 --2- 172.21.15.1:0/3423050477 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fc4005260 0x7f2fc400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fdd3f0700 1 -- 172.21.15.1:0/3423050477 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fc4004850 msgr2=0x7f2fc4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:14.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fdd3f0700 1 --2- 172.21.15.1:0/3423050477 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fc4004850 0x7f2fc4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fdd3f0700 1 -- 172.21.15.1:0/3423050477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc40035b0 con 0x7f2fc4003be0 2024-03-18T18:50:14.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.391+0000 7f2fddbf1700 1 -- 172.21.15.1:0/3423050477 reap_dead start 2024-03-18T18:50:14.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.392+0000 7f2fdd3f0700 1 --2- 172.21.15.1:0/3423050477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fc4003be0 0x7f2fc4003fe0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400c160 tx=0x7f2fc4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e79b7e1a847dc2de server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:14.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.392+0000 7f2fcbfff700 1 -- 172.21.15.1:0/3423050477 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd400cca0 con 0x7f2fc4003be0 2024-03-18T18:50:14.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.392+0000 7f2fcbfff700 1 -- 172.21.15.1:0/3423050477 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd400ce20 con 0x7f2fc4003be0 2024-03-18T18:50:14.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.392+0000 7f2fcbfff700 1 -- 172.21.15.1:0/3423050477 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4005ba0 con 0x7f2fc4003be0 2024-03-18T18:50:14.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.392+0000 7f2fdf654700 1 -- 172.21.15.1:0/3423050477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fc4003be0 msgr2=0x7f2fc4003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:14.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.392+0000 7f2fdf654700 1 --2- 172.21.15.1:0/3423050477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fc4003be0 0x7f2fc4003fe0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400c160 tx=0x7f2fc4005880 comp rx=0 tx=0).stop 2024-03-18T18:50:14.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.392+0000 7f2fdf654700 1 -- 172.21.15.1:0/3423050477 shutdown_connections 2024-03-18T18:50:14.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.392+0000 7f2fdf654700 1 --2- 172.21.15.1:0/3423050477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fc4003be0 0x7f2fc4003fe0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.392+0000 7f2fdf654700 1 -- 172.21.15.1:0/3423050477 >> 172.21.15.1:0/3423050477 conn(0x7f2fd809c600 msgr2=0x7f2fd80d3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:14.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.392+0000 7f2fdf654700 1 -- 172.21.15.1:0/3423050477 shutdown_connections 2024-03-18T18:50:14.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.392+0000 7f2fdf654700 1 -- 172.21.15.1:0/3423050477 wait complete. 2024-03-18T18:50:14.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.394+0000 7f2fdf654700 1 Processor -- start 2024-03-18T18:50:14.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.394+0000 7f2fdf654700 1 -- start start 2024-03-18T18:50:14.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.395+0000 7f2fdf654700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fc4003be0 0x7f2fd80b6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:14.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.395+0000 7f2fdf654700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fd80b83d0 0x7f2fd815b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:14.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.395+0000 7f2fdf654700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd80cbe60 0x7f2fd80cc310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:14.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.395+0000 7f2fdf654700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd80cc850 con 0x7f2fd80cbe60 2024-03-18T18:50:14.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.395+0000 7f2fdf654700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd80b7a30 con 0x7f2fd80b83d0 2024-03-18T18:50:14.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.395+0000 7f2fdf654700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd80b7190 con 0x7f2fc4003be0 2024-03-18T18:50:14.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.395+0000 7f2fddbf1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd80cbe60 0x7f2fd80cc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:14.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.395+0000 7f2fddbf1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd80cbe60 0x7f2fd80cc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51830/0 (socket says 172.21.15.1:51830) 2024-03-18T18:50:14.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.395+0000 7f2fddbf1700 1 -- 172.21.15.1:0/3213029191 learned_addr learned my addr 172.21.15.1:0/3213029191 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:14.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.395+0000 7f2fdd3f0700 1 --2- 172.21.15.1:0/3213029191 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fc4003be0 0x7f2fd80b6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:14.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.395+0000 7f2fdcbef700 1 --2- 172.21.15.1:0/3213029191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fd80b83d0 0x7f2fd815b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:14.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.395+0000 7f2fddbf1700 1 -- 172.21.15.1:0/3213029191 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fc4003be0 msgr2=0x7f2fd80b6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:14.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.395+0000 7f2fddbf1700 1 --2- 172.21.15.1:0/3213029191 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fc4003be0 0x7f2fd80b6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.395+0000 7f2fddbf1700 1 -- 172.21.15.1:0/3213029191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fd80b83d0 msgr2=0x7f2fd815b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:14.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.395+0000 7f2fddbf1700 1 --2- 172.21.15.1:0/3213029191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fd80b83d0 0x7f2fd815b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.396+0000 7f2fddbf1700 1 -- 172.21.15.1:0/3213029191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd80b7470 con 0x7f2fd80cbe60 2024-03-18T18:50:14.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.396+0000 7f2fddbf1700 1 --2- 172.21.15.1:0/3213029191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd80cbe60 0x7f2fd80cc310 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2fd80b26e0 tx=0x7f2fd000ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:14.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.396+0000 7f2fc9ffb700 1 -- 172.21.15.1:0/3213029191 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0003c10 con 0x7f2fd80cbe60 2024-03-18T18:50:14.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.396+0000 7f2fc9ffb700 1 -- 172.21.15.1:0/3213029191 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd0003d90 con 0x7f2fd80cbe60 2024-03-18T18:50:14.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.396+0000 7f2fc9ffb700 1 -- 172.21.15.1:0/3213029191 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0005310 con 0x7f2fd80cbe60 2024-03-18T18:50:14.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.396+0000 7f2fdf654700 1 -- 172.21.15.1:0/3213029191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd80b7750 con 0x7f2fd80cbe60 2024-03-18T18:50:14.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.396+0000 7f2fdf654700 1 -- 172.21.15.1:0/3213029191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd80b6ea0 con 0x7f2fd80cbe60 2024-03-18T18:50:14.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.397+0000 7f2fdf654700 1 -- 172.21.15.1:0/3213029191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd81f3750 con 0x7f2fd80cbe60 2024-03-18T18:50:14.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.397+0000 7f2fc9ffb700 1 -- 172.21.15.1:0/3213029191 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2fd0005470 con 0x7f2fd80cbe60 2024-03-18T18:50:14.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.398+0000 7f2fc9ffb700 1 --2- 172.21.15.1:0/3213029191 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2fbc06ac90 0x7f2fbc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:14.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.398+0000 7f2fc9ffb700 1 -- 172.21.15.1:0/3213029191 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2fd0089590 con 0x7f2fd80cbe60 2024-03-18T18:50:14.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.398+0000 7f2fdd3f0700 1 --2- 172.21.15.1:0/3213029191 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2fbc06ac90 0x7f2fbc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:14.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.398+0000 7f2fdd3f0700 1 --2- 172.21.15.1:0/3213029191 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2fbc06ac90 0x7f2fbc06d150 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4000c00 tx=0x7f2fd4007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:14.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.402+0000 7f2fc9ffb700 1 -- 172.21.15.1:0/3213029191 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2fd81f3750 con 0x7f2fd80cbe60 2024-03-18T18:50:14.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.651+0000 7f2fdf654700 1 -- 172.21.15.1:0/3213029191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2fd81f3750 con 0x7f2fd80cbe60 2024-03-18T18:50:14.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.652+0000 7f2fc9ffb700 1 -- 172.21.15.1:0/3213029191 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f2fd81f3750 con 0x7f2fd80cbe60 2024-03-18T18:50:14.648 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:14.648 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-18T18:50:14.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.654+0000 7f2fdf654700 1 -- 172.21.15.1:0/3213029191 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2fbc06ac90 msgr2=0x7f2fbc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:14.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.654+0000 7f2fdf654700 1 --2- 172.21.15.1:0/3213029191 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2fbc06ac90 0x7f2fbc06d150 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4000c00 tx=0x7f2fd4007000 comp rx=0 tx=0).stop 2024-03-18T18:50:14.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.654+0000 7f2fdf654700 1 -- 172.21.15.1:0/3213029191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd80cbe60 msgr2=0x7f2fd80cc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:14.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.654+0000 7f2fdf654700 1 --2- 172.21.15.1:0/3213029191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd80cbe60 0x7f2fd80cc310 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2fd80b26e0 tx=0x7f2fd000ef00 comp rx=0 tx=0).stop 2024-03-18T18:50:14.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.654+0000 7f2fdf654700 1 -- 172.21.15.1:0/3213029191 shutdown_connections 2024-03-18T18:50:14.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.654+0000 7f2fdf654700 1 --2- 172.21.15.1:0/3213029191 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2fbc06ac90 0x7f2fbc06d150 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.654+0000 7f2fdf654700 1 --2- 172.21.15.1:0/3213029191 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2fc4003be0 0x7f2fd80b6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.654+0000 7f2fdf654700 1 --2- 172.21.15.1:0/3213029191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fd80cbe60 0x7f2fd80cc310 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.654+0000 7f2fdf654700 1 --2- 172.21.15.1:0/3213029191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2fd80b83d0 0x7f2fd815b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.654+0000 7f2fdf654700 1 -- 172.21.15.1:0/3213029191 >> 172.21.15.1:0/3213029191 conn(0x7f2fd809c600 msgr2=0x7f2fd815ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:14.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.654+0000 7f2fdf654700 1 -- 172.21.15.1:0/3213029191 shutdown_connections 2024-03-18T18:50:14.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.654+0000 7f2fdf654700 1 -- 172.21.15.1:0/3213029191 wait complete. 2024-03-18T18:50:14.662 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-18T18:50:14.663 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:14.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.791+0000 7f3a490a7700 1 Processor -- start 2024-03-18T18:50:14.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.791+0000 7f3a490a7700 1 -- start start 2024-03-18T18:50:14.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.792+0000 7f3a490a7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a440af610 0x7f3a440cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:14.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.792+0000 7f3a490a7700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3a440ccf40 0x7f3a44071a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:14.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.792+0000 7f3a490a7700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a440cf090 con 0x7f3a44144b70 2024-03-18T18:50:14.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.792+0000 7f3a490a7700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a440cd3d0 con 0x7f3a440ccf40 2024-03-18T18:50:14.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.792+0000 7f3a490a7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a44144f50 con 0x7f3a440af610 2024-03-18T18:50:14.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.792+0000 7f3a4259c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a440af610 0x7f3a440cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:14.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.792+0000 7f3a4359e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3a440ccf40 0x7f3a44071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:14.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.792+0000 7f3a42d9d700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f3a44144b70 0x7f3a440ce980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:60808/0 (socket says 172.21.15.1:60808) 2024-03-18T18:50:14.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.792+0000 7f3a42d9d700 1 -- 172.21.15.1:0/4174444008 learned_addr learned my addr 172.21.15.1:0/4174444008 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:14.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.793+0000 7f3a41d9b700 1 -- 172.21.15.1:0/4174444008 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178615502 0 0) 0x7f3a38001110 con 0x7f3a44144b70 2024-03-18T18:50:14.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.793+0000 7f3a41d9b700 1 -- 172.21.15.1:0/4174444008 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a2c0035b0 con 0x7f3a44144b70 2024-03-18T18:50:14.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.793+0000 7f3a4259c700 1 -- 172.21.15.1:0/4174444008 >> v2:172.21.15.165:3300/0 conn(0x7f3a440ccf40 msgr2=0x7f3a44071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:14.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.793+0000 7f3a4259c700 1 --2- 172.21.15.1:0/4174444008 >> v2:172.21.15.165:3300/0 conn(0x7f3a440ccf40 0x7f3a44071a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.793+0000 7f3a4259c700 1 -- 172.21.15.1:0/4174444008 >> v1:172.21.15.165:6789/0 conn(0x7f3a44144b70 legacy=0x7f3a440ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:14.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.793+0000 7f3a4259c700 1 -- 172.21.15.1:0/4174444008 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a440cf2d0 con 0x7f3a440af610 2024-03-18T18:50:14.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.793+0000 7f3a4359e700 1 --2- 172.21.15.1:0/4174444008 >> v2:172.21.15.165:3300/0 conn(0x7f3a440ccf40 0x7f3a44071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:50:14.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.793+0000 7f3a4259c700 1 --2- 172.21.15.1:0/4174444008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a440af610 0x7f3a440cca00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3a34009840 tx=0x7f3a34009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=927a56a371b5178e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:14.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.793+0000 7f3a41d9b700 1 -- 172.21.15.1:0/4174444008 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a34011830 con 0x7f3a440af610 2024-03-18T18:50:14.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.793+0000 7f3a41d9b700 1 -- 172.21.15.1:0/4174444008 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a34011990 con 0x7f3a440af610 2024-03-18T18:50:14.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.793+0000 7f3a41d9b700 1 -- 172.21.15.1:0/4174444008 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a34011d00 con 0x7f3a440af610 2024-03-18T18:50:14.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.793+0000 7f3a490a7700 1 -- 172.21.15.1:0/4174444008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a440af610 msgr2=0x7f3a440cca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:14.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.793+0000 7f3a490a7700 1 --2- 172.21.15.1:0/4174444008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a440af610 0x7f3a440cca00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3a34009840 tx=0x7f3a34009b50 comp rx=0 tx=0).stop 2024-03-18T18:50:14.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.794+0000 7f3a490a7700 1 -- 172.21.15.1:0/4174444008 shutdown_connections 2024-03-18T18:50:14.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.794+0000 7f3a490a7700 1 --2- 172.21.15.1:0/4174444008 >> v2:172.21.15.165:3300/0 conn(0x7f3a440ccf40 0x7f3a44071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.794+0000 7f3a490a7700 1 --2- 172.21.15.1:0/4174444008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a440af610 0x7f3a440cca00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.794+0000 7f3a490a7700 1 -- 172.21.15.1:0/4174444008 >> 172.21.15.1:0/4174444008 conn(0x7f3a440b1d20 msgr2=0x7f3a440b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:14.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.794+0000 7f3a490a7700 1 -- 172.21.15.1:0/4174444008 shutdown_connections 2024-03-18T18:50:14.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.794+0000 7f3a490a7700 1 -- 172.21.15.1:0/4174444008 wait complete. 2024-03-18T18:50:14.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.795+0000 7f3a490a7700 1 Processor -- start 2024-03-18T18:50:14.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.795+0000 7f3a490a7700 1 -- start start 2024-03-18T18:50:14.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.796+0000 7f3a490a7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a440af610 0x7f3a4415a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:14.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.796+0000 7f3a490a7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a440ccf40 0x7f3a4413a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:14.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.796+0000 7f3a490a7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a44144b70 0x7f3a44138080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:14.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.796+0000 7f3a490a7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4413a740 con 0x7f3a440af610 2024-03-18T18:50:14.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.796+0000 7f3a490a7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4413a9e0 con 0x7f3a440ccf40 2024-03-18T18:50:14.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.796+0000 7f3a490a7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a44139130 con 0x7f3a44144b70 2024-03-18T18:50:14.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.796+0000 7f3a42d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a440af610 0x7f3a4415a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:14.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.796+0000 7f3a42d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a440af610 0x7f3a4415a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51840/0 (socket says 172.21.15.1:51840) 2024-03-18T18:50:14.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.796+0000 7f3a42d9d700 1 -- 172.21.15.1:0/678360085 learned_addr learned my addr 172.21.15.1:0/678360085 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:14.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.796+0000 7f3a4359e700 1 --2- 172.21.15.1:0/678360085 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a44144b70 0x7f3a44138080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:14.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.796+0000 7f3a4259c700 1 --2- 172.21.15.1:0/678360085 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a440ccf40 0x7f3a4413a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:14.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.796+0000 7f3a42d9d700 1 -- 172.21.15.1:0/678360085 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a44144b70 msgr2=0x7f3a44138080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:14.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.796+0000 7f3a42d9d700 1 --2- 172.21.15.1:0/678360085 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a44144b70 0x7f3a44138080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.796+0000 7f3a42d9d700 1 -- 172.21.15.1:0/678360085 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a440ccf40 msgr2=0x7f3a4413a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:14.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.796+0000 7f3a42d9d700 1 --2- 172.21.15.1:0/678360085 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a440ccf40 0x7f3a4413a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:14.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.796+0000 7f3a42d9d700 1 -- 172.21.15.1:0/678360085 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3c009a70 con 0x7f3a440af610 2024-03-18T18:50:14.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.796+0000 7f3a4259c700 1 --2- 172.21.15.1:0/678360085 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a440ccf40 0x7f3a4413a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:14.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.796+0000 7f3a42d9d700 1 --2- 172.21.15.1:0/678360085 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a440af610 0x7f3a4415a520 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3a3800c420 tx=0x7f3a3800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:14.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.797+0000 7f3a2b7fe700 1 -- 172.21.15.1:0/678360085 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a38005d70 con 0x7f3a440af610 2024-03-18T18:50:14.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.797+0000 7f3a2b7fe700 1 -- 172.21.15.1:0/678360085 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a380070d0 con 0x7f3a440af610 2024-03-18T18:50:14.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.797+0000 7f3a490a7700 1 -- 172.21.15.1:0/678360085 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a340094a0 con 0x7f3a440af610 2024-03-18T18:50:14.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.797+0000 7f3a2b7fe700 1 -- 172.21.15.1:0/678360085 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a38007480 con 0x7f3a440af610 2024-03-18T18:50:14.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.797+0000 7f3a490a7700 1 -- 172.21.15.1:0/678360085 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a441396f0 con 0x7f3a440af610 2024-03-18T18:50:14.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.798+0000 7f3a490a7700 1 -- 172.21.15.1:0/678360085 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a440cda90 con 0x7f3a440af610 2024-03-18T18:50:14.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.798+0000 7f3a2b7fe700 1 -- 172.21.15.1:0/678360085 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3a380075e0 con 0x7f3a440af610 2024-03-18T18:50:14.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.798+0000 7f3a2b7fe700 1 --2- 172.21.15.1:0/678360085 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3a3006ac90 0x7f3a3006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:14.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.799+0000 7f3a2b7fe700 1 -- 172.21.15.1:0/678360085 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3a380888e0 con 0x7f3a440af610 2024-03-18T18:50:14.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.799+0000 7f3a4259c700 1 --2- 172.21.15.1:0/678360085 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3a3006ac90 0x7f3a3006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:14.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.799+0000 7f3a4259c700 1 --2- 172.21.15.1:0/678360085 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3a3006ac90 0x7f3a3006d150 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3a34000c00 tx=0x7f3a34008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:14.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:14.803+0000 7f3a2b7fe700 1 -- 172.21.15.1:0/678360085 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3a380537d0 con 0x7f3a440af610 2024-03-18T18:50:15.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.009+0000 7f3a490a7700 1 -- 172.21.15.1:0/678360085 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a440cf090 con 0x7f3a440af610 2024-03-18T18:50:15.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.010+0000 7f3a2b7fe700 1 -- 172.21.15.1:0/678360085 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17830 (secure 0 0 0) 0x7f3a440cf090 con 0x7f3a440af610 2024-03-18T18:50:15.007 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:15.007 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:49.276961+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:15.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.013+0000 7f3a490a7700 1 -- 172.21.15.1:0/678360085 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3a3006ac90 msgr2=0x7f3a3006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:15.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.013+0000 7f3a490a7700 1 --2- 172.21.15.1:0/678360085 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3a3006ac90 0x7f3a3006d150 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3a34000c00 tx=0x7f3a34008030 comp rx=0 tx=0).stop 2024-03-18T18:50:15.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.013+0000 7f3a490a7700 1 -- 172.21.15.1:0/678360085 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a440af610 msgr2=0x7f3a4415a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:15.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.013+0000 7f3a490a7700 1 --2- 172.21.15.1:0/678360085 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a440af610 0x7f3a4415a520 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3a3800c420 tx=0x7f3a3800c7e0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.013+0000 7f3a490a7700 1 -- 172.21.15.1:0/678360085 shutdown_connections 2024-03-18T18:50:15.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.013+0000 7f3a490a7700 1 --2- 172.21.15.1:0/678360085 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3a3006ac90 0x7f3a3006d150 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.013+0000 7f3a490a7700 1 --2- 172.21.15.1:0/678360085 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a44144b70 0x7f3a44138080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.013+0000 7f3a490a7700 1 --2- 172.21.15.1:0/678360085 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a440af610 0x7f3a4415a520 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.013+0000 7f3a490a7700 1 --2- 172.21.15.1:0/678360085 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a440ccf40 0x7f3a4413a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.013+0000 7f3a490a7700 1 -- 172.21.15.1:0/678360085 >> 172.21.15.1:0/678360085 conn(0x7f3a440b1d20 msgr2=0x7f3a440b1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:15.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.013+0000 7f3a490a7700 1 -- 172.21.15.1:0/678360085 shutdown_connections 2024-03-18T18:50:15.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.013+0000 7f3a490a7700 1 -- 172.21.15.1:0/678360085 wait complete. 2024-03-18T18:50:15.022 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T18:49:42.800801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T18:49:43.776684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T18:49:46.854608+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T18:50:15.022 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T18:50:15.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.153+0000 7ff1d3ca1700 1 Processor -- start 2024-03-18T18:50:15.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.153+0000 7ff1d3ca1700 1 -- start start 2024-03-18T18:50:15.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.154+0000 7ff1d3ca1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff1cc15a380 0x7ff1cc15b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:15.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.155+0000 7ff1d3ca1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff1cc0724e0 0x7ff1cc15baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:15.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.155+0000 7ff1d3ca1700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1cc15b6f0 con 0x7ff1cc071490 2024-03-18T18:50:15.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.155+0000 7ff1d3ca1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1cc0cb510 con 0x7ff1cc0724e0 2024-03-18T18:50:15.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.155+0000 7ff1d3ca1700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1cc0ca900 con 0x7ff1cc15a380 2024-03-18T18:50:15.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.155+0000 7ff1d1a3d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff1cc071490 0x7ff1cc071870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55844/0 (socket says 172.21.15.1:55844) 2024-03-18T18:50:15.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.155+0000 7ff1d1a3d700 1 -- 172.21.15.1:0/1220225664 learned_addr learned my addr 172.21.15.1:0/1220225664 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:15.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.155+0000 7ff1d223e700 1 --2- 172.21.15.1:0/1220225664 >> v2:172.21.15.1:3300/0 conn(0x7ff1cc0724e0 0x7ff1cc15baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:15.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.155+0000 7ff1d123c700 1 --2- 172.21.15.1:0/1220225664 >> v2:172.21.15.165:3300/0 conn(0x7ff1cc15a380 0x7ff1cc15b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:15.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.155+0000 7ff1d0a3b700 1 -- 172.21.15.1:0/1220225664 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2763618659 0 0) 0x7ff1bc001110 con 0x7ff1cc071490 2024-03-18T18:50:15.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.155+0000 7ff1d0a3b700 1 -- 172.21.15.1:0/1220225664 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1b80035b0 con 0x7ff1cc071490 2024-03-18T18:50:15.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.155+0000 7ff1d223e700 1 -- 172.21.15.1:0/1220225664 >> v2:172.21.15.165:3300/0 conn(0x7ff1cc15a380 msgr2=0x7ff1cc15b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:15.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.155+0000 7ff1d223e700 1 --2- 172.21.15.1:0/1220225664 >> v2:172.21.15.165:3300/0 conn(0x7ff1cc15a380 0x7ff1cc15b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.155+0000 7ff1d223e700 1 -- 172.21.15.1:0/1220225664 >> v1:172.21.15.1:6789/0 conn(0x7ff1cc071490 legacy=0x7ff1cc071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:15.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.155+0000 7ff1d223e700 1 -- 172.21.15.1:0/1220225664 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1cc0cabe0 con 0x7ff1cc0724e0 2024-03-18T18:50:15.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.155+0000 7ff1d223e700 1 --2- 172.21.15.1:0/1220225664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1cc0724e0 0x7ff1cc15baa0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc06cfc0 tx=0x7ff1c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2107f40c9e57d92 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:15.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.155+0000 7ff1d0a3b700 1 -- 172.21.15.1:0/1220225664 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c400e070 con 0x7ff1cc0724e0 2024-03-18T18:50:15.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.155+0000 7ff1d0a3b700 1 -- 172.21.15.1:0/1220225664 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c400bb20 con 0x7ff1cc0724e0 2024-03-18T18:50:15.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.155+0000 7ff1d0a3b700 1 -- 172.21.15.1:0/1220225664 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c4012430 con 0x7ff1cc0724e0 2024-03-18T18:50:15.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.156+0000 7ff1d3ca1700 1 -- 172.21.15.1:0/1220225664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1cc0724e0 msgr2=0x7ff1cc15baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:15.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.156+0000 7ff1d3ca1700 1 --2- 172.21.15.1:0/1220225664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1cc0724e0 0x7ff1cc15baa0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc06cfc0 tx=0x7ff1c400b330 comp rx=0 tx=0).stop 2024-03-18T18:50:15.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.156+0000 7ff1d3ca1700 1 -- 172.21.15.1:0/1220225664 shutdown_connections 2024-03-18T18:50:15.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.156+0000 7ff1d3ca1700 1 --2- 172.21.15.1:0/1220225664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1cc0724e0 0x7ff1cc15baa0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.156+0000 7ff1d3ca1700 1 --2- 172.21.15.1:0/1220225664 >> v2:172.21.15.165:3300/0 conn(0x7ff1cc15a380 0x7ff1cc15b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.156+0000 7ff1d3ca1700 1 -- 172.21.15.1:0/1220225664 >> 172.21.15.1:0/1220225664 conn(0x7ff1cc0b1aa0 msgr2=0x7ff1cc0b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:15.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.156+0000 7ff1d3ca1700 1 -- 172.21.15.1:0/1220225664 shutdown_connections 2024-03-18T18:50:15.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.156+0000 7ff1d3ca1700 1 -- 172.21.15.1:0/1220225664 wait complete. 2024-03-18T18:50:15.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.157+0000 7ff1d3ca1700 1 Processor -- start 2024-03-18T18:50:15.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.157+0000 7ff1d3ca1700 1 -- start start 2024-03-18T18:50:15.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.158+0000 7ff1d3ca1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1cc071490 0x7ff1cc0b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:15.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.158+0000 7ff1d3ca1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff1cc0724e0 0x7ff1cc0c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:15.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.158+0000 7ff1d3ca1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff1cc15a380 0x7ff1cc0c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:15.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.158+0000 7ff1d3ca1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc0c9b00 con 0x7ff1cc071490 2024-03-18T18:50:15.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.158+0000 7ff1d3ca1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc0c9de0 con 0x7ff1cc15a380 2024-03-18T18:50:15.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.158+0000 7ff1d3ca1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc0c8ae0 con 0x7ff1cc0724e0 2024-03-18T18:50:15.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.158+0000 7ff1d1a3d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1cc071490 0x7ff1cc0b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:15.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.158+0000 7ff1d1a3d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1cc071490 0x7ff1cc0b52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51852/0 (socket says 172.21.15.1:51852) 2024-03-18T18:50:15.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.158+0000 7ff1d1a3d700 1 -- 172.21.15.1:0/3095200689 learned_addr learned my addr 172.21.15.1:0/3095200689 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:15.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.158+0000 7ff1d1a3d700 1 -- 172.21.15.1:0/3095200689 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff1cc0724e0 msgr2=0x7ff1cc0c7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:15.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.158+0000 7ff1d1a3d700 1 --2- 172.21.15.1:0/3095200689 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff1cc0724e0 0x7ff1cc0c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.158+0000 7ff1d223e700 1 --2- 172.21.15.1:0/3095200689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff1cc15a380 0x7ff1cc0c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:15.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.158+0000 7ff1d1a3d700 1 -- 172.21.15.1:0/3095200689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff1cc15a380 msgr2=0x7ff1cc0c7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:15.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.158+0000 7ff1d1a3d700 1 --2- 172.21.15.1:0/3095200689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff1cc15a380 0x7ff1cc0c7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.158+0000 7ff1d1a3d700 1 -- 172.21.15.1:0/3095200689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c4009a70 con 0x7ff1cc071490 2024-03-18T18:50:15.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.158+0000 7ff1d223e700 1 --2- 172.21.15.1:0/3095200689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff1cc15a380 0x7ff1cc0c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:15.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.159+0000 7ff1d1a3d700 1 --2- 172.21.15.1:0/3095200689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1cc071490 0x7ff1cc0b52a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc007fd0 tx=0x7ff1bc00ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:15.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.159+0000 7ff1c27fc700 1 -- 172.21.15.1:0/3095200689 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc005d70 con 0x7ff1cc071490 2024-03-18T18:50:15.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.159+0000 7ff1c27fc700 1 -- 172.21.15.1:0/3095200689 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1bc0070d0 con 0x7ff1cc071490 2024-03-18T18:50:15.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.159+0000 7ff1d3ca1700 1 -- 172.21.15.1:0/3095200689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1cc0cbb20 con 0x7ff1cc071490 2024-03-18T18:50:15.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.159+0000 7ff1c27fc700 1 -- 172.21.15.1:0/3095200689 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc007460 con 0x7ff1cc071490 2024-03-18T18:50:15.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.159+0000 7ff1d3ca1700 1 -- 172.21.15.1:0/3095200689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1cc0c90a0 con 0x7ff1cc071490 2024-03-18T18:50:15.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.160+0000 7ff1d3ca1700 1 -- 172.21.15.1:0/3095200689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1cc0c9230 con 0x7ff1cc071490 2024-03-18T18:50:15.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.160+0000 7ff1c27fc700 1 -- 172.21.15.1:0/3095200689 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff1bc0075c0 con 0x7ff1cc071490 2024-03-18T18:50:15.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.161+0000 7ff1c27fc700 1 --2- 172.21.15.1:0/3095200689 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff1b006ac90 0x7ff1b006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:15.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.161+0000 7ff1d123c700 1 --2- 172.21.15.1:0/3095200689 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff1b006ac90 0x7ff1b006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:15.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.161+0000 7ff1c27fc700 1 -- 172.21.15.1:0/3095200689 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff1bc088a00 con 0x7ff1cc071490 2024-03-18T18:50:15.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.161+0000 7ff1d123c700 1 --2- 172.21.15.1:0/3095200689 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff1b006ac90 0x7ff1b006d150 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc1e7cc0 tx=0x7ff1c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:15.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.165+0000 7ff1c27fc700 1 -- 172.21.15.1:0/3095200689 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff1bc053980 con 0x7ff1cc071490 2024-03-18T18:50:15.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.368+0000 7ff1d3ca1700 1 -- 172.21.15.1:0/3095200689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff1cc15bec0 con 0x7ff1cc071490 2024-03-18T18:50:15.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.369+0000 7ff1c27fc700 1 -- 172.21.15.1:0/3095200689 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff1cc15bec0 con 0x7ff1cc071490 2024-03-18T18:50:15.365 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T18:50:15.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.371+0000 7ff1d3ca1700 1 -- 172.21.15.1:0/3095200689 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff1b006ac90 msgr2=0x7ff1b006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.371+0000 7ff1d3ca1700 1 --2- 172.21.15.1:0/3095200689 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff1b006ac90 0x7ff1b006d150 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc1e7cc0 tx=0x7ff1c8008040 comp rx=0 tx=0).stop 2024-03-18T18:50:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.371+0000 7ff1d3ca1700 1 -- 172.21.15.1:0/3095200689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1cc071490 msgr2=0x7ff1cc0b52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.371+0000 7ff1d3ca1700 1 --2- 172.21.15.1:0/3095200689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1cc071490 0x7ff1cc0b52a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc007fd0 tx=0x7ff1bc00ca90 comp rx=0 tx=0).stop 2024-03-18T18:50:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.371+0000 7ff1d3ca1700 1 -- 172.21.15.1:0/3095200689 shutdown_connections 2024-03-18T18:50:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.371+0000 7ff1d3ca1700 1 --2- 172.21.15.1:0/3095200689 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff1b006ac90 0x7ff1b006d150 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.371+0000 7ff1d3ca1700 1 --2- 172.21.15.1:0/3095200689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff1cc15a380 0x7ff1cc0c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.371+0000 7ff1d3ca1700 1 --2- 172.21.15.1:0/3095200689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1cc071490 0x7ff1cc0b52a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.371+0000 7ff1d3ca1700 1 --2- 172.21.15.1:0/3095200689 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff1cc0724e0 0x7ff1cc0c7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.371+0000 7ff1d3ca1700 1 -- 172.21.15.1:0/3095200689 >> 172.21.15.1:0/3095200689 conn(0x7ff1cc0b1aa0 msgr2=0x7ff1cc0cf090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.371+0000 7ff1d3ca1700 1 -- 172.21.15.1:0/3095200689 shutdown_connections 2024-03-18T18:50:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.371+0000 7ff1d3ca1700 1 -- 172.21.15.1:0/3095200689 wait complete. 2024-03-18T18:50:15.379 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T18:50:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.511+0000 7f278caf5700 1 Processor -- start 2024-03-18T18:50:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.511+0000 7f278caf5700 1 -- start start 2024-03-18T18:50:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.512+0000 7f278caf5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f27880b3c30 0x7f27880b2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.512+0000 7f278caf5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27880cef70 0x7f27880b4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.512+0000 7f278caf5700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27880b3560 con 0x7f278806ce40 2024-03-18T18:50:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.512+0000 7f278caf5700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27880b4550 con 0x7f27880cef70 2024-03-18T18:50:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.512+0000 7f278caf5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2788071a80 con 0x7f27880b3c30 2024-03-18T18:50:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.512+0000 7f2785d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27880cef70 0x7f27880b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.512+0000 7f2785d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27880cef70 0x7f27880b4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51854/0 (socket says 172.21.15.1:51854) 2024-03-18T18:50:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.512+0000 7f2785d9b700 1 -- 172.21.15.1:0/480313094 learned_addr learned my addr 172.21.15.1:0/480313094 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:15.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.512+0000 7f278659c700 1 --2- 172.21.15.1:0/480313094 >> v2:172.21.15.165:3300/0 conn(0x7f27880b3c30 0x7f27880b2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:15.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.512+0000 7f2785d9b700 1 -- 172.21.15.1:0/480313094 >> v2:172.21.15.165:3300/0 conn(0x7f27880b3c30 msgr2=0x7f27880b2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:15.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.512+0000 7f2785d9b700 1 --2- 172.21.15.1:0/480313094 >> v2:172.21.15.165:3300/0 conn(0x7f27880b3c30 0x7f27880b2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.512+0000 7f2785d9b700 1 -- 172.21.15.1:0/480313094 >> v1:172.21.15.165:6789/0 conn(0x7f278806ce40 legacy=0x7f278815da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:15.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.512+0000 7f2785d9b700 1 -- 172.21.15.1:0/480313094 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27880b2cc0 con 0x7f27880cef70 2024-03-18T18:50:15.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.512+0000 7f2785d9b700 1 --2- 172.21.15.1:0/480313094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27880cef70 0x7f27880b4010 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f27880b2120 tx=0x7f2778009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b40497d16306dce7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:15.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.512+0000 7f278559a700 1 -- 172.21.15.1:0/480313094 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2778004230 con 0x7f27880cef70 2024-03-18T18:50:15.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.512+0000 7f278559a700 1 -- 172.21.15.1:0/480313094 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2778004390 con 0x7f27880cef70 2024-03-18T18:50:15.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.512+0000 7f278559a700 1 -- 172.21.15.1:0/480313094 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27780036a0 con 0x7f27880cef70 2024-03-18T18:50:15.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.513+0000 7f278caf5700 1 -- 172.21.15.1:0/480313094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27880cef70 msgr2=0x7f27880b4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:15.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.513+0000 7f278caf5700 1 --2- 172.21.15.1:0/480313094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27880cef70 0x7f27880b4010 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f27880b2120 tx=0x7f2778009bd0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.513+0000 7f278caf5700 1 -- 172.21.15.1:0/480313094 shutdown_connections 2024-03-18T18:50:15.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.513+0000 7f278caf5700 1 --2- 172.21.15.1:0/480313094 >> v2:172.21.15.165:3300/0 conn(0x7f27880b3c30 0x7f27880b2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.513+0000 7f278caf5700 1 --2- 172.21.15.1:0/480313094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27880cef70 0x7f27880b4010 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.513+0000 7f278caf5700 1 -- 172.21.15.1:0/480313094 >> 172.21.15.1:0/480313094 conn(0x7f27880b1350 msgr2=0x7f27880b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:15.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.513+0000 7f278caf5700 1 -- 172.21.15.1:0/480313094 shutdown_connections 2024-03-18T18:50:15.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.513+0000 7f278caf5700 1 -- 172.21.15.1:0/480313094 wait complete. 2024-03-18T18:50:15.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.514+0000 7f278caf5700 1 Processor -- start 2024-03-18T18:50:15.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.515+0000 7f278caf5700 1 -- start start 2024-03-18T18:50:15.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.515+0000 7f278caf5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f278806ce40 0x7f278814d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:15.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.515+0000 7f278caf5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27880b3c30 0x7f2788136930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:15.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.515+0000 7f278caf5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27880cef70 0x7f278814a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:15.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.515+0000 7f278caf5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27880ac8b0 con 0x7f27880b3c30 2024-03-18T18:50:15.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.515+0000 7f278caf5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2788137170 con 0x7f278806ce40 2024-03-18T18:50:15.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.515+0000 7f278caf5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2788137450 con 0x7f27880cef70 2024-03-18T18:50:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.515+0000 7f2785d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27880b3c30 0x7f2788136930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.515+0000 7f2785d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27880b3c30 0x7f2788136930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51862/0 (socket says 172.21.15.1:51862) 2024-03-18T18:50:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.515+0000 7f2785d9b700 1 -- 172.21.15.1:0/1363979913 learned_addr learned my addr 172.21.15.1:0/1363979913 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.516+0000 7f278659c700 1 --2- 172.21.15.1:0/1363979913 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f278806ce40 0x7f278814d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.516+0000 7f2786d9d700 1 --2- 172.21.15.1:0/1363979913 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27880cef70 0x7f278814a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.516+0000 7f2785d9b700 1 -- 172.21.15.1:0/1363979913 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27880cef70 msgr2=0x7f278814a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.516+0000 7f2785d9b700 1 --2- 172.21.15.1:0/1363979913 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27880cef70 0x7f278814a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.516+0000 7f2785d9b700 1 -- 172.21.15.1:0/1363979913 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f278806ce40 msgr2=0x7f278814d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.516+0000 7f2785d9b700 1 --2- 172.21.15.1:0/1363979913 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f278806ce40 0x7f278814d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.516+0000 7f2785d9b700 1 -- 172.21.15.1:0/1363979913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2778009550 con 0x7f27880b3c30 2024-03-18T18:50:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.516+0000 7f278659c700 1 --2- 172.21.15.1:0/1363979913 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f278806ce40 0x7f278814d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.516+0000 7f2785d9b700 1 --2- 172.21.15.1:0/1363979913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27880b3c30 0x7f2788136930 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2778009130 tx=0x7f2778004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.516+0000 7f2776ffd700 1 -- 172.21.15.1:0/1363979913 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2778011630 con 0x7f27880b3c30 2024-03-18T18:50:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.516+0000 7f2776ffd700 1 -- 172.21.15.1:0/1363979913 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2778011790 con 0x7f27880b3c30 2024-03-18T18:50:15.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.516+0000 7f278caf5700 1 -- 172.21.15.1:0/1363979913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2788137730 con 0x7f27880b3c30 2024-03-18T18:50:15.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.516+0000 7f2776ffd700 1 -- 172.21.15.1:0/1363979913 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2778011b40 con 0x7f27880b3c30 2024-03-18T18:50:15.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.516+0000 7f278caf5700 1 -- 172.21.15.1:0/1363979913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2788136e70 con 0x7f27880b3c30 2024-03-18T18:50:15.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.517+0000 7f278caf5700 1 -- 172.21.15.1:0/1363979913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27880acd40 con 0x7f27880b3c30 2024-03-18T18:50:15.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.517+0000 7f2776ffd700 1 -- 172.21.15.1:0/1363979913 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2778011ca0 con 0x7f27880b3c30 2024-03-18T18:50:15.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.518+0000 7f2776ffd700 1 --2- 172.21.15.1:0/1363979913 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f276806ac90 0x7f276806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:15.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.518+0000 7f2776ffd700 1 -- 172.21.15.1:0/1363979913 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f277808ac80 con 0x7f27880b3c30 2024-03-18T18:50:15.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.518+0000 7f278659c700 1 --2- 172.21.15.1:0/1363979913 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f276806ac90 0x7f276806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:15.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.518+0000 7f278659c700 1 --2- 172.21.15.1:0/1363979913 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f276806ac90 0x7f276806d150 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f278815cb90 tx=0x7f277c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:15.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.522+0000 7f2776ffd700 1 -- 172.21.15.1:0/1363979913 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f27880acd40 con 0x7f27880b3c30 2024-03-18T18:50:15.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.724+0000 7f278caf5700 1 -- 172.21.15.1:0/1363979913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f27880acd40 con 0x7f27880b3c30 2024-03-18T18:50:15.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.725+0000 7f2776ffd700 1 -- 172.21.15.1:0/1363979913 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f27880acd40 con 0x7f27880b3c30 2024-03-18T18:50:15.721 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:15.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.727+0000 7f278caf5700 1 -- 172.21.15.1:0/1363979913 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f276806ac90 msgr2=0x7f276806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:15.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.727+0000 7f278caf5700 1 --2- 172.21.15.1:0/1363979913 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f276806ac90 0x7f276806d150 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f278815cb90 tx=0x7f277c008040 comp rx=0 tx=0).stop 2024-03-18T18:50:15.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.727+0000 7f278caf5700 1 -- 172.21.15.1:0/1363979913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27880b3c30 msgr2=0x7f2788136930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:15.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.727+0000 7f278caf5700 1 --2- 172.21.15.1:0/1363979913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27880b3c30 0x7f2788136930 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2778009130 tx=0x7f2778004750 comp rx=0 tx=0).stop 2024-03-18T18:50:15.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.727+0000 7f278caf5700 1 -- 172.21.15.1:0/1363979913 shutdown_connections 2024-03-18T18:50:15.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.727+0000 7f278caf5700 1 --2- 172.21.15.1:0/1363979913 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f276806ac90 0x7f276806d150 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.727+0000 7f278caf5700 1 --2- 172.21.15.1:0/1363979913 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27880cef70 0x7f278814a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.727+0000 7f278caf5700 1 --2- 172.21.15.1:0/1363979913 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f278806ce40 0x7f278814d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.727+0000 7f278caf5700 1 --2- 172.21.15.1:0/1363979913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27880b3c30 0x7f2788136930 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.727+0000 7f278caf5700 1 -- 172.21.15.1:0/1363979913 >> 172.21.15.1:0/1363979913 conn(0x7f27880b1350 msgr2=0x7f278806d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:15.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.727+0000 7f278caf5700 1 -- 172.21.15.1:0/1363979913 shutdown_connections 2024-03-18T18:50:15.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.727+0000 7f278caf5700 1 -- 172.21.15.1:0/1363979913 wait complete. 2024-03-18T18:50:15.736 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T18:50:15.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.868+0000 7f404f92e700 1 Processor -- start 2024-03-18T18:50:15.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.868+0000 7f404f92e700 1 -- start start 2024-03-18T18:50:15.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.869+0000 7f404f92e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f404815a1d0 0x7f40480add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:15.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.869+0000 7f404f92e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4048071620 0x7f4048071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:15.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.869+0000 7f404f92e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40480ce8f0 con 0x7f40480cb9e0 2024-03-18T18:50:15.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.869+0000 7f404f92e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40480cdb00 con 0x7f4048071620 2024-03-18T18:50:15.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.869+0000 7f404f92e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f404815a5b0 con 0x7f404815a1d0 2024-03-18T18:50:15.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.869+0000 7f404cec9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4048071620 0x7f4048071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:15.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.869+0000 7f404decb700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f40480cb9e0 0x7f40480cbe50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55852/0 (socket says 172.21.15.1:55852) 2024-03-18T18:50:15.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.869+0000 7f404decb700 1 -- 172.21.15.1:0/56050959 learned_addr learned my addr 172.21.15.1:0/56050959 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:15.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.869+0000 7f404d6ca700 1 --2- 172.21.15.1:0/56050959 >> v2:172.21.15.165:3300/0 conn(0x7f404815a1d0 0x7f40480add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:15.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.869+0000 7f403ffff700 1 -- 172.21.15.1:0/56050959 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4231379520 0 0) 0x7f40480ce8f0 con 0x7f40480cb9e0 2024-03-18T18:50:15.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.869+0000 7f403ffff700 1 -- 172.21.15.1:0/56050959 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40340035b0 con 0x7f40480cb9e0 2024-03-18T18:50:15.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.869+0000 7f404cec9700 1 -- 172.21.15.1:0/56050959 >> v2:172.21.15.165:3300/0 conn(0x7f404815a1d0 msgr2=0x7f40480add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:15.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.869+0000 7f404cec9700 1 --2- 172.21.15.1:0/56050959 >> v2:172.21.15.165:3300/0 conn(0x7f404815a1d0 0x7f40480add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.869+0000 7f404cec9700 1 -- 172.21.15.1:0/56050959 >> v1:172.21.15.1:6789/0 conn(0x7f40480cb9e0 legacy=0x7f40480cbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:15.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.869+0000 7f404cec9700 1 -- 172.21.15.1:0/56050959 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4048071fc0 con 0x7f4048071620 2024-03-18T18:50:15.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.869+0000 7f404cec9700 1 --2- 172.21.15.1:0/56050959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4048071620 0x7f4048071a80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f40480726b0 tx=0x7f4038009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb25ee5d01f723d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:15.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.870+0000 7f403ffff700 1 -- 172.21.15.1:0/56050959 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4038011830 con 0x7f4048071620 2024-03-18T18:50:15.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.870+0000 7f403ffff700 1 -- 172.21.15.1:0/56050959 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4038011990 con 0x7f4048071620 2024-03-18T18:50:15.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.870+0000 7f403ffff700 1 -- 172.21.15.1:0/56050959 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4038011d20 con 0x7f4048071620 2024-03-18T18:50:15.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.870+0000 7f404f92e700 1 -- 172.21.15.1:0/56050959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4048071620 msgr2=0x7f4048071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:15.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.870+0000 7f404f92e700 1 --2- 172.21.15.1:0/56050959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4048071620 0x7f4048071a80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f40480726b0 tx=0x7f4038009b20 comp rx=0 tx=0).stop 2024-03-18T18:50:15.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.870+0000 7f404f92e700 1 -- 172.21.15.1:0/56050959 shutdown_connections 2024-03-18T18:50:15.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.870+0000 7f404f92e700 1 --2- 172.21.15.1:0/56050959 >> v2:172.21.15.165:3300/0 conn(0x7f404815a1d0 0x7f40480add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.870+0000 7f404f92e700 1 --2- 172.21.15.1:0/56050959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4048071620 0x7f4048071a80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.870+0000 7f404f92e700 1 -- 172.21.15.1:0/56050959 >> 172.21.15.1:0/56050959 conn(0x7f40480cc9b0 msgr2=0x7f40480ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:15.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.870+0000 7f404f92e700 1 -- 172.21.15.1:0/56050959 shutdown_connections 2024-03-18T18:50:15.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.870+0000 7f404f92e700 1 -- 172.21.15.1:0/56050959 wait complete. 2024-03-18T18:50:15.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.871+0000 7f404f92e700 1 Processor -- start 2024-03-18T18:50:15.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.872+0000 7f404f92e700 1 -- start start 2024-03-18T18:50:15.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.872+0000 7f404f92e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4048071620 0x7f4048157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:15.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.872+0000 7f404f92e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40480cb9e0 0x7f40480b4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:15.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.872+0000 7f404f92e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f404815a1d0 0x7f40480ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:15.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.872+0000 7f404f92e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40480b3ce0 con 0x7f40480cb9e0 2024-03-18T18:50:15.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.872+0000 7f404f92e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4048158160 con 0x7f4048071620 2024-03-18T18:50:15.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.872+0000 7f404f92e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40480b30d0 con 0x7f404815a1d0 2024-03-18T18:50:15.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.872+0000 7f404cec9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40480cb9e0 0x7f40480b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:15.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.872+0000 7f404cec9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40480cb9e0 0x7f40480b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51884/0 (socket says 172.21.15.1:51884) 2024-03-18T18:50:15.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.872+0000 7f404cec9700 1 -- 172.21.15.1:0/909083427 learned_addr learned my addr 172.21.15.1:0/909083427 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:15.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.873+0000 7f404d6ca700 1 --2- 172.21.15.1:0/909083427 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4048071620 0x7f4048157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.873+0000 7f404decb700 1 --2- 172.21.15.1:0/909083427 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f404815a1d0 0x7f40480ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.873+0000 7f404cec9700 1 -- 172.21.15.1:0/909083427 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f404815a1d0 msgr2=0x7f40480ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.873+0000 7f404cec9700 1 --2- 172.21.15.1:0/909083427 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f404815a1d0 0x7f40480ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.873+0000 7f404cec9700 1 -- 172.21.15.1:0/909083427 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4048071620 msgr2=0x7f4048157c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.873+0000 7f404cec9700 1 --2- 172.21.15.1:0/909083427 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4048071620 0x7f4048157c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.873+0000 7f404cec9700 1 -- 172.21.15.1:0/909083427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40380094a0 con 0x7f40480cb9e0 2024-03-18T18:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.873+0000 7f404cec9700 1 --2- 172.21.15.1:0/909083427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40480cb9e0 0x7f40480b4660 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4038000c00 tx=0x7f4038004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.873+0000 7f403dffb700 1 -- 172.21.15.1:0/909083427 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40380036a0 con 0x7f40480cb9e0 2024-03-18T18:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.873+0000 7f403dffb700 1 -- 172.21.15.1:0/909083427 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403801e070 con 0x7f40480cb9e0 2024-03-18T18:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.873+0000 7f404f92e700 1 -- 172.21.15.1:0/909083427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40480b33b0 con 0x7f40480cb9e0 2024-03-18T18:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.873+0000 7f403dffb700 1 -- 172.21.15.1:0/909083427 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4038003a20 con 0x7f40480cb9e0 2024-03-18T18:50:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.873+0000 7f404f92e700 1 -- 172.21.15.1:0/909083427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40480b3970 con 0x7f40480cb9e0 2024-03-18T18:50:15.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.874+0000 7f404f92e700 1 -- 172.21.15.1:0/909083427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40480b3e90 con 0x7f40480cb9e0 2024-03-18T18:50:15.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.875+0000 7f403dffb700 1 -- 172.21.15.1:0/909083427 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4038003b80 con 0x7f40480cb9e0 2024-03-18T18:50:15.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.875+0000 7f403dffb700 1 --2- 172.21.15.1:0/909083427 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f402c06ac90 0x7f402c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:15.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.875+0000 7f403dffb700 1 -- 172.21.15.1:0/909083427 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f403808a3a0 con 0x7f40480cb9e0 2024-03-18T18:50:15.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.875+0000 7f404d6ca700 1 --2- 172.21.15.1:0/909083427 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f402c06ac90 0x7f402c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:15.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.876+0000 7f404d6ca700 1 --2- 172.21.15.1:0/909083427 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f402c06ac90 0x7f402c06d150 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f40480c65e0 tx=0x7f40440091c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:15.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:15.880+0000 7f403dffb700 1 -- 172.21.15.1:0/909083427 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f40480b3e90 con 0x7f40480cb9e0 2024-03-18T18:50:16.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.083+0000 7f404f92e700 1 -- 172.21.15.1:0/909083427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f40480676a0 con 0x7f40480cb9e0 2024-03-18T18:50:16.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.083+0000 7f403dffb700 1 -- 172.21.15.1:0/909083427 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4038055320 con 0x7f40480cb9e0 2024-03-18T18:50:16.080 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T18:50:16.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.086+0000 7f404f92e700 1 -- 172.21.15.1:0/909083427 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f402c06ac90 msgr2=0x7f402c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.086+0000 7f404f92e700 1 --2- 172.21.15.1:0/909083427 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f402c06ac90 0x7f402c06d150 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f40480c65e0 tx=0x7f40440091c0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.086+0000 7f404f92e700 1 -- 172.21.15.1:0/909083427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40480cb9e0 msgr2=0x7f40480b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.086+0000 7f404f92e700 1 --2- 172.21.15.1:0/909083427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40480cb9e0 0x7f40480b4660 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4038000c00 tx=0x7f4038004750 comp rx=0 tx=0).stop 2024-03-18T18:50:16.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.086+0000 7f404f92e700 1 -- 172.21.15.1:0/909083427 shutdown_connections 2024-03-18T18:50:16.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.086+0000 7f404f92e700 1 --2- 172.21.15.1:0/909083427 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f402c06ac90 0x7f402c06d150 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.086+0000 7f404f92e700 1 --2- 172.21.15.1:0/909083427 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f404815a1d0 0x7f40480ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.086+0000 7f404f92e700 1 --2- 172.21.15.1:0/909083427 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4048071620 0x7f4048157c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.086+0000 7f404f92e700 1 --2- 172.21.15.1:0/909083427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40480cb9e0 0x7f40480b4660 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.086+0000 7f404f92e700 1 -- 172.21.15.1:0/909083427 >> 172.21.15.1:0/909083427 conn(0x7f40480cc9b0 msgr2=0x7f40480684d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:16.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.086+0000 7f404f92e700 1 -- 172.21.15.1:0/909083427 shutdown_connections 2024-03-18T18:50:16.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.086+0000 7f404f92e700 1 -- 172.21.15.1:0/909083427 wait complete. 2024-03-18T18:50:16.094 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T18:50:16.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.226+0000 7f2abb343700 1 Processor -- start 2024-03-18T18:50:16.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.226+0000 7f2abb343700 1 -- start start 2024-03-18T18:50:16.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.227+0000 7f2abb343700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2ab40ccfe0 0x7f2ab415d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.227+0000 7f2abb343700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab40b3b00 con 0x7f2ab406ca80 2024-03-18T18:50:16.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.227+0000 7f2abb343700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab40cf7f0 con 0x7f2ab415d5b0 2024-03-18T18:50:16.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.227+0000 7f2abb343700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ab40cbbb0 con 0x7f2ab40ccfe0 2024-03-18T18:50:16.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.227+0000 7f2ab88de700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2ab406ca80 0x7f2ab409a150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55866/0 (socket says 172.21.15.1:55866) 2024-03-18T18:50:16.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.227+0000 7f2ab88de700 1 -- 172.21.15.1:0/2129040966 learned_addr learned my addr 172.21.15.1:0/2129040966 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:16.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.227+0000 7f2ab98e0700 1 --2- 172.21.15.1:0/2129040966 >> v2:172.21.15.165:3300/0 conn(0x7f2ab40ccfe0 0x7f2ab415d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.227+0000 7f2aabfff700 1 -- 172.21.15.1:0/2129040966 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867897942 0 0) 0x7f2ab40b3b00 con 0x7f2ab406ca80 2024-03-18T18:50:16.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.227+0000 7f2aabfff700 1 -- 172.21.15.1:0/2129040966 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aa00035b0 con 0x7f2ab406ca80 2024-03-18T18:50:16.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2aabfff700 1 -- 172.21.15.1:0/2129040966 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 377573937 0 0) 0x7f2aa00035b0 con 0x7f2ab406ca80 2024-03-18T18:50:16.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2aabfff700 1 -- 172.21.15.1:0/2129040966 >> v2:172.21.15.165:3300/0 conn(0x7f2ab40ccfe0 msgr2=0x7f2ab415d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:16.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2aabfff700 1 --2- 172.21.15.1:0/2129040966 >> v2:172.21.15.165:3300/0 conn(0x7f2ab40ccfe0 0x7f2ab415d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2aabfff700 1 -- 172.21.15.1:0/2129040966 >> v1:172.21.15.165:6789/0 conn(0x7f2ab415d5b0 legacy=0x7f2ab4099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2aabfff700 1 -- 172.21.15.1:0/2129040966 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab40d04d0 con 0x7f2ab406ca80 2024-03-18T18:50:16.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2aabfff700 1 -- 172.21.15.1:0/2129040966 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f2ab0001670 con 0x7f2ab406ca80 2024-03-18T18:50:16.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2aabfff700 1 -- 172.21.15.1:0/2129040966 >> v1:172.21.15.1:6789/0 conn(0x7f2ab406ca80 legacy=0x7f2ab409a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2aabfff700 1 --2- 172.21.15.1:0/2129040966 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2aa0003a00 0x7f2aa0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2aabfff700 1 --2- 172.21.15.1:0/2129040966 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2aa0004850 0x7f2aa0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2aabfff700 1 --2- 172.21.15.1:0/2129040966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2aa0005210 0x7f2aa000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2aabfff700 1 -- 172.21.15.1:0/2129040966 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab40cbbb0 con 0x7f2aa0005210 2024-03-18T18:50:16.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2aabfff700 1 -- 172.21.15.1:0/2129040966 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa000c1b0 con 0x7f2aa0003a00 2024-03-18T18:50:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2aabfff700 1 -- 172.21.15.1:0/2129040966 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa000c370 con 0x7f2aa0004850 2024-03-18T18:50:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2ab98e0700 1 --2- 172.21.15.1:0/2129040966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2aa0005210 0x7f2aa000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2ab90df700 1 --2- 172.21.15.1:0/2129040966 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2aa0003a00 0x7f2aa0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2ab88de700 1 --2- 172.21.15.1:0/2129040966 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2aa0004850 0x7f2aa0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2ab98e0700 1 -- 172.21.15.1:0/2129040966 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2aa0004850 msgr2=0x7f2aa0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2ab98e0700 1 --2- 172.21.15.1:0/2129040966 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2aa0004850 0x7f2aa0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2ab98e0700 1 -- 172.21.15.1:0/2129040966 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2aa0003a00 msgr2=0x7f2aa0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2ab98e0700 1 --2- 172.21.15.1:0/2129040966 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2aa0003a00 0x7f2aa0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2ab98e0700 1 -- 172.21.15.1:0/2129040966 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa00035b0 con 0x7f2aa0005210 2024-03-18T18:50:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2ab90df700 1 --2- 172.21.15.1:0/2129040966 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2aa0003a00 0x7f2aa0003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2ab98e0700 1 -- 172.21.15.1:0/2129040966 reap_dead start 2024-03-18T18:50:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2ab98e0700 1 --2- 172.21.15.1:0/2129040966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2aa0005210 0x7f2aa000ba10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009fc0 tx=0x7f2aa0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69b9f85db9db5838 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2aabfff700 1 -- 172.21.15.1:0/2129040966 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac003a20 con 0x7f2aa0005210 2024-03-18T18:50:16.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2aabfff700 1 -- 172.21.15.1:0/2129040966 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aac00f040 con 0x7f2aa0005210 2024-03-18T18:50:16.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.228+0000 7f2aabfff700 1 -- 172.21.15.1:0/2129040966 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aac003de0 con 0x7f2aa0005210 2024-03-18T18:50:16.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.229+0000 7f2abb343700 1 -- 172.21.15.1:0/2129040966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2aa0005210 msgr2=0x7f2aa000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.229+0000 7f2abb343700 1 --2- 172.21.15.1:0/2129040966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2aa0005210 0x7f2aa000ba10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009fc0 tx=0x7f2aa0005880 comp rx=0 tx=0).stop 2024-03-18T18:50:16.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.229+0000 7f2abb343700 1 -- 172.21.15.1:0/2129040966 shutdown_connections 2024-03-18T18:50:16.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.229+0000 7f2abb343700 1 --2- 172.21.15.1:0/2129040966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2aa0005210 0x7f2aa000ba10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.229+0000 7f2abb343700 1 -- 172.21.15.1:0/2129040966 >> 172.21.15.1:0/2129040966 conn(0x7f2ab40b2bd0 msgr2=0x7f2ab40b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:16.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.229+0000 7f2abb343700 1 -- 172.21.15.1:0/2129040966 shutdown_connections 2024-03-18T18:50:16.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.229+0000 7f2abb343700 1 -- 172.21.15.1:0/2129040966 wait complete. 2024-03-18T18:50:16.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.230+0000 7f2abb343700 1 Processor -- start 2024-03-18T18:50:16.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.230+0000 7f2abb343700 1 -- start start 2024-03-18T18:50:16.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.231+0000 7f2abb343700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2aa0005210 0x7f2ab40b5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.231+0000 7f2abb343700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ab40b6430 0x7f2ab414e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.231+0000 7f2abb343700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ab41f30b0 0x7f2ab414e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.231+0000 7f2abb343700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab409efa0 con 0x7f2aa0005210 2024-03-18T18:50:16.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.231+0000 7f2abb343700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab409ecc0 con 0x7f2ab41f30b0 2024-03-18T18:50:16.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.231+0000 7f2abb343700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab40c9670 con 0x7f2ab40b6430 2024-03-18T18:50:16.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.231+0000 7f2ab90df700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2aa0005210 0x7f2ab40b5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.231+0000 7f2ab90df700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2aa0005210 0x7f2ab40b5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51894/0 (socket says 172.21.15.1:51894) 2024-03-18T18:50:16.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.231+0000 7f2ab90df700 1 -- 172.21.15.1:0/3847469357 learned_addr learned my addr 172.21.15.1:0/3847469357 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:16.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.231+0000 7f2ab88de700 1 --2- 172.21.15.1:0/3847469357 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ab40b6430 0x7f2ab414e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.231+0000 7f2ab98e0700 1 --2- 172.21.15.1:0/3847469357 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ab41f30b0 0x7f2ab414e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.231+0000 7f2ab90df700 1 -- 172.21.15.1:0/3847469357 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ab40b6430 msgr2=0x7f2ab414e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.231+0000 7f2ab90df700 1 --2- 172.21.15.1:0/3847469357 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ab40b6430 0x7f2ab414e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.231+0000 7f2ab90df700 1 -- 172.21.15.1:0/3847469357 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ab41f30b0 msgr2=0x7f2ab414e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.231+0000 7f2ab90df700 1 --2- 172.21.15.1:0/3847469357 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ab41f30b0 0x7f2ab414e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.231+0000 7f2ab90df700 1 -- 172.21.15.1:0/3847469357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab40c9950 con 0x7f2aa0005210 2024-03-18T18:50:16.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.232+0000 7f2ab90df700 1 --2- 172.21.15.1:0/3847469357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2aa0005210 0x7f2ab40b5ef0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2aa400e550 tx=0x7f2aa400e860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:16.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.232+0000 7f2aa9ffb700 1 -- 172.21.15.1:0/3847469357 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa40052e0 con 0x7f2aa0005210 2024-03-18T18:50:16.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.232+0000 7f2aa9ffb700 1 -- 172.21.15.1:0/3847469357 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa4005440 con 0x7f2aa0005210 2024-03-18T18:50:16.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.232+0000 7f2abb343700 1 -- 172.21.15.1:0/3847469357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab40c9c30 con 0x7f2aa0005210 2024-03-18T18:50:16.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.232+0000 7f2aa9ffb700 1 -- 172.21.15.1:0/3847469357 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa40038a0 con 0x7f2aa0005210 2024-03-18T18:50:16.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.232+0000 7f2abb343700 1 -- 172.21.15.1:0/3847469357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab409e130 con 0x7f2aa0005210 2024-03-18T18:50:16.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.233+0000 7f2abb343700 1 -- 172.21.15.1:0/3847469357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab40c94d0 con 0x7f2aa0005210 2024-03-18T18:50:16.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.233+0000 7f2aa9ffb700 1 -- 172.21.15.1:0/3847469357 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2aa4003a00 con 0x7f2aa0005210 2024-03-18T18:50:16.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.234+0000 7f2aa9ffb700 1 --2- 172.21.15.1:0/3847469357 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2a9806ac90 0x7f2a9806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.234+0000 7f2aa9ffb700 1 -- 172.21.15.1:0/3847469357 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2aa40888c0 con 0x7f2aa0005210 2024-03-18T18:50:16.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.234+0000 7f2ab88de700 1 --2- 172.21.15.1:0/3847469357 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2a9806ac90 0x7f2a9806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.234+0000 7f2ab88de700 1 --2- 172.21.15.1:0/3847469357 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2a9806ac90 0x7f2a9806d150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40b6810 tx=0x7f2ab0007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:16.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.238+0000 7f2aa9ffb700 1 -- 172.21.15.1:0/3847469357 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2ab40c94d0 con 0x7f2aa0005210 2024-03-18T18:50:16.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.441+0000 7f2abb343700 1 -- 172.21.15.1:0/3847469357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f2ab40c94d0 con 0x7f2aa0005210 2024-03-18T18:50:16.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.442+0000 7f2aa9ffb700 1 -- 172.21.15.1:0/3847469357 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2ab40c94d0 con 0x7f2aa0005210 2024-03-18T18:50:16.438 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:16.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.444+0000 7f2abb343700 1 -- 172.21.15.1:0/3847469357 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2a9806ac90 msgr2=0x7f2a9806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.444+0000 7f2abb343700 1 --2- 172.21.15.1:0/3847469357 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2a9806ac90 0x7f2a9806d150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40b6810 tx=0x7f2ab0007040 comp rx=0 tx=0).stop 2024-03-18T18:50:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.444+0000 7f2abb343700 1 -- 172.21.15.1:0/3847469357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2aa0005210 msgr2=0x7f2ab40b5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.444+0000 7f2abb343700 1 --2- 172.21.15.1:0/3847469357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2aa0005210 0x7f2ab40b5ef0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2aa400e550 tx=0x7f2aa400e860 comp rx=0 tx=0).stop 2024-03-18T18:50:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.444+0000 7f2abb343700 1 -- 172.21.15.1:0/3847469357 shutdown_connections 2024-03-18T18:50:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.444+0000 7f2abb343700 1 --2- 172.21.15.1:0/3847469357 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2a9806ac90 0x7f2a9806d150 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.444+0000 7f2abb343700 1 --2- 172.21.15.1:0/3847469357 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ab41f30b0 0x7f2ab414e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.444+0000 7f2abb343700 1 --2- 172.21.15.1:0/3847469357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2aa0005210 0x7f2ab40b5ef0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.444+0000 7f2abb343700 1 --2- 172.21.15.1:0/3847469357 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ab40b6430 0x7f2ab414e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.444+0000 7f2abb343700 1 -- 172.21.15.1:0/3847469357 >> 172.21.15.1:0/3847469357 conn(0x7f2ab40b2bd0 msgr2=0x7f2ab40b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.444+0000 7f2abb343700 1 -- 172.21.15.1:0/3847469357 shutdown_connections 2024-03-18T18:50:16.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.444+0000 7f2abb343700 1 -- 172.21.15.1:0/3847469357 wait complete. 2024-03-18T18:50:16.452 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T18:50:16.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.580+0000 7f69e92f6700 1 Processor -- start 2024-03-18T18:50:16.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.581+0000 7f69e92f6700 1 -- start start 2024-03-18T18:50:16.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.581+0000 7f69e92f6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f69e40af610 0x7f69e40b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.581+0000 7f69e92f6700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e40b34f0 con 0x7f69e40726c0 2024-03-18T18:50:16.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.581+0000 7f69e92f6700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e415b350 con 0x7f69e415af70 2024-03-18T18:50:16.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.581+0000 7f69e92f6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e4072b70 con 0x7f69e40af610 2024-03-18T18:50:16.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e27fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f69e40726c0 0x7f69e40aef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55872/0 (socket says 172.21.15.1:55872) 2024-03-18T18:50:16.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e27fc700 1 -- 172.21.15.1:0/2781738545 learned_addr learned my addr 172.21.15.1:0/2781738545 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:16.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e37fe700 1 --2- 172.21.15.1:0/2781738545 >> v2:172.21.15.165:3300/0 conn(0x7f69e40af610 0x7f69e40b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e1ffb700 1 -- 172.21.15.1:0/2781738545 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573788177 0 0) 0x7f69e40b34f0 con 0x7f69e40726c0 2024-03-18T18:50:16.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e1ffb700 1 -- 172.21.15.1:0/2781738545 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69cc0035b0 con 0x7f69e40726c0 2024-03-18T18:50:16.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e1ffb700 1 -- 172.21.15.1:0/2781738545 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2496284966 0 0) 0x7f69cc0035b0 con 0x7f69e40726c0 2024-03-18T18:50:16.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e1ffb700 1 -- 172.21.15.1:0/2781738545 >> v2:172.21.15.165:3300/0 conn(0x7f69e40af610 msgr2=0x7f69e40b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:16.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e1ffb700 1 --2- 172.21.15.1:0/2781738545 >> v2:172.21.15.165:3300/0 conn(0x7f69e40af610 0x7f69e40b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e1ffb700 1 -- 172.21.15.1:0/2781738545 >> v1:172.21.15.165:6789/0 conn(0x7f69e415af70 legacy=0x7f69e41617b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e1ffb700 1 -- 172.21.15.1:0/2781738545 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e40b3720 con 0x7f69e40726c0 2024-03-18T18:50:16.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e1ffb700 1 -- 172.21.15.1:0/2781738545 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f69d4001670 con 0x7f69e40726c0 2024-03-18T18:50:16.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e1ffb700 1 -- 172.21.15.1:0/2781738545 >> v1:172.21.15.1:6789/0 conn(0x7f69e40726c0 legacy=0x7f69e40aef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e1ffb700 1 --2- 172.21.15.1:0/2781738545 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69cc003a00 0x7f69cc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e1ffb700 1 --2- 172.21.15.1:0/2781738545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69cc004850 0x7f69cc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e1ffb700 1 --2- 172.21.15.1:0/2781738545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69cc005210 0x7f69cc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e1ffb700 1 -- 172.21.15.1:0/2781738545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc00c180 con 0x7f69cc004850 2024-03-18T18:50:16.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e1ffb700 1 -- 172.21.15.1:0/2781738545 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc00c2f0 con 0x7f69cc005210 2024-03-18T18:50:16.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e1ffb700 1 -- 172.21.15.1:0/2781738545 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc00c4b0 con 0x7f69cc003a00 2024-03-18T18:50:16.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e27fc700 1 --2- 172.21.15.1:0/2781738545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69cc004850 0x7f69cc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e37fe700 1 --2- 172.21.15.1:0/2781738545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69cc005210 0x7f69cc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.582+0000 7f69e2ffd700 1 --2- 172.21.15.1:0/2781738545 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69cc003a00 0x7f69cc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.583+0000 7f69e27fc700 1 -- 172.21.15.1:0/2781738545 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69cc003a00 msgr2=0x7f69cc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:16.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.583+0000 7f69e27fc700 1 --2- 172.21.15.1:0/2781738545 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69cc003a00 0x7f69cc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.583+0000 7f69e27fc700 1 -- 172.21.15.1:0/2781738545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69cc005210 msgr2=0x7f69cc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:16.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.583+0000 7f69e27fc700 1 --2- 172.21.15.1:0/2781738545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69cc005210 0x7f69cc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.583+0000 7f69e27fc700 1 -- 172.21.15.1:0/2781738545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc0035b0 con 0x7f69cc004850 2024-03-18T18:50:16.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.583+0000 7f69e37fe700 1 --2- 172.21.15.1:0/2781738545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69cc005210 0x7f69cc00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:50:16.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.583+0000 7f69e37fe700 1 -- 172.21.15.1:0/2781738545 reap_dead start 2024-03-18T18:50:16.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.583+0000 7f69e27fc700 1 --2- 172.21.15.1:0/2781738545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69cc004850 0x7f69cc004cd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f69d400c240 tx=0x7f69cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=924a90d4f1ad3e2c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:16.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.583+0000 7f69e1ffb700 1 -- 172.21.15.1:0/2781738545 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4007c10 con 0x7f69cc004850 2024-03-18T18:50:16.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.583+0000 7f69e1ffb700 1 -- 172.21.15.1:0/2781738545 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d400cde0 con 0x7f69cc004850 2024-03-18T18:50:16.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.583+0000 7f69e1ffb700 1 -- 172.21.15.1:0/2781738545 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4005ba0 con 0x7f69cc004850 2024-03-18T18:50:16.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.583+0000 7f69e92f6700 1 -- 172.21.15.1:0/2781738545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69cc004850 msgr2=0x7f69cc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.583+0000 7f69e92f6700 1 --2- 172.21.15.1:0/2781738545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69cc004850 0x7f69cc004cd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f69d400c240 tx=0x7f69cc005880 comp rx=0 tx=0).stop 2024-03-18T18:50:16.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.583+0000 7f69e92f6700 1 -- 172.21.15.1:0/2781738545 shutdown_connections 2024-03-18T18:50:16.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.584+0000 7f69e92f6700 1 --2- 172.21.15.1:0/2781738545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69cc004850 0x7f69cc004cd0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.584+0000 7f69e92f6700 1 -- 172.21.15.1:0/2781738545 >> 172.21.15.1:0/2781738545 conn(0x7f69e406d8d0 msgr2=0x7f69e406dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:16.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.584+0000 7f69e92f6700 1 -- 172.21.15.1:0/2781738545 shutdown_connections 2024-03-18T18:50:16.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.584+0000 7f69e92f6700 1 -- 172.21.15.1:0/2781738545 wait complete. 2024-03-18T18:50:16.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.585+0000 7f69e92f6700 1 Processor -- start 2024-03-18T18:50:16.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.585+0000 7f69e92f6700 1 -- start start 2024-03-18T18:50:16.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.586+0000 7f69e92f6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69cc004850 0x7f69e4145ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.586+0000 7f69e92f6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69e415e1e0 0x7f69e415b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.586+0000 7f69e92f6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69e415ba60 0x7f69e41483d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.586+0000 7f69e92f6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e415bef0 con 0x7f69e415e1e0 2024-03-18T18:50:16.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.586+0000 7f69e92f6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e40aeee0 con 0x7f69cc004850 2024-03-18T18:50:16.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.586+0000 7f69e92f6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e415d4d0 con 0x7f69e415ba60 2024-03-18T18:50:16.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.586+0000 7f69e27fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69e415e1e0 0x7f69e415b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.586+0000 7f69e27fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69e415e1e0 0x7f69e415b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51916/0 (socket says 172.21.15.1:51916) 2024-03-18T18:50:16.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.586+0000 7f69e27fc700 1 -- 172.21.15.1:0/2149662739 learned_addr learned my addr 172.21.15.1:0/2149662739 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:16.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.587+0000 7f69e2ffd700 1 --2- 172.21.15.1:0/2149662739 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69cc004850 0x7f69e4145ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.587+0000 7f69e37fe700 1 --2- 172.21.15.1:0/2149662739 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69e415ba60 0x7f69e41483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.587+0000 7f69e27fc700 1 -- 172.21.15.1:0/2149662739 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69e415ba60 msgr2=0x7f69e41483d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.587+0000 7f69e27fc700 1 --2- 172.21.15.1:0/2149662739 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69e415ba60 0x7f69e41483d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.587+0000 7f69e27fc700 1 -- 172.21.15.1:0/2149662739 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69cc004850 msgr2=0x7f69e4145ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.587+0000 7f69e27fc700 1 --2- 172.21.15.1:0/2149662739 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69cc004850 0x7f69e4145ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.587+0000 7f69e27fc700 1 -- 172.21.15.1:0/2149662739 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e4149330 con 0x7f69e415e1e0 2024-03-18T18:50:16.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.587+0000 7f69e37fe700 1 --2- 172.21.15.1:0/2149662739 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69e415ba60 0x7f69e41483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:16.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.587+0000 7f69e27fc700 1 --2- 172.21.15.1:0/2149662739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69e415e1e0 0x7f69e415b520 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f69cc00c2c0 tx=0x7f69d4001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:16.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.587+0000 7f69cb7fe700 1 -- 172.21.15.1:0/2149662739 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4001640 con 0x7f69e415e1e0 2024-03-18T18:50:16.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.587+0000 7f69cb7fe700 1 -- 172.21.15.1:0/2149662739 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d40070d0 con 0x7f69e415e1e0 2024-03-18T18:50:16.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.587+0000 7f69e92f6700 1 -- 172.21.15.1:0/2149662739 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e4149050 con 0x7f69e415e1e0 2024-03-18T18:50:16.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.587+0000 7f69cb7fe700 1 -- 172.21.15.1:0/2149662739 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d40074a0 con 0x7f69e415e1e0 2024-03-18T18:50:16.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.587+0000 7f69e92f6700 1 -- 172.21.15.1:0/2149662739 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e415d9e0 con 0x7f69e415e1e0 2024-03-18T18:50:16.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.588+0000 7f69e92f6700 1 -- 172.21.15.1:0/2149662739 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e415be70 con 0x7f69e415e1e0 2024-03-18T18:50:16.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.589+0000 7f69cb7fe700 1 -- 172.21.15.1:0/2149662739 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f69d4007600 con 0x7f69e415e1e0 2024-03-18T18:50:16.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.589+0000 7f69cb7fe700 1 --2- 172.21.15.1:0/2149662739 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f69d006ac90 0x7f69d006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.589+0000 7f69cb7fe700 1 -- 172.21.15.1:0/2149662739 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f69d408aa30 con 0x7f69e415e1e0 2024-03-18T18:50:16.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.589+0000 7f69e2ffd700 1 --2- 172.21.15.1:0/2149662739 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f69d006ac90 0x7f69d006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.589+0000 7f69e2ffd700 1 --2- 172.21.15.1:0/2149662739 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f69d006ac90 0x7f69d006d150 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f69d8000c00 tx=0x7f69d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:16.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.593+0000 7f69cb7fe700 1 -- 172.21.15.1:0/2149662739 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f69d4055ad0 con 0x7f69e415e1e0 2024-03-18T18:50:16.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.847+0000 7f69e92f6700 1 -- 172.21.15.1:0/2149662739 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69e4073350 con 0x7f69e415e1e0 2024-03-18T18:50:16.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.847+0000 7f69cb7fe700 1 -- 172.21.15.1:0/2149662739 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4727 (secure 0 0 0) 0x7f69d4007d70 con 0x7f69e415e1e0 2024-03-18T18:50:16.844 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:16.844 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/883056851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":883056851},{"type":"v1","addr":"172.21.15.165:6834","nonce":883056851}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1388214718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1388214718},{"type":"v1","addr":"172.21.15.165:6835","nonce":1388214718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3020786191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3020786191},{"type":"v1","addr":"172.21.15.1:6836","nonce":3020786191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T18:49:44.809495+0000","modified":"2024-03-18T18:50:07.888953+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2118908012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2118908012},{"type":"v1","addr":"172.21.15.1:6837","nonce":2118908012}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-18T18:50:16.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.849+0000 7f69e92f6700 1 -- 172.21.15.1:0/2149662739 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f69d006ac90 msgr2=0x7f69d006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.849+0000 7f69e92f6700 1 --2- 172.21.15.1:0/2149662739 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f69d006ac90 0x7f69d006d150 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f69d8000c00 tx=0x7f69d8009040 comp rx=0 tx=0).stop 2024-03-18T18:50:16.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.849+0000 7f69e92f6700 1 -- 172.21.15.1:0/2149662739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69e415e1e0 msgr2=0x7f69e415b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.849+0000 7f69e92f6700 1 --2- 172.21.15.1:0/2149662739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69e415e1e0 0x7f69e415b520 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f69cc00c2c0 tx=0x7f69d4001a00 comp rx=0 tx=0).stop 2024-03-18T18:50:16.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.850+0000 7f69e92f6700 1 -- 172.21.15.1:0/2149662739 shutdown_connections 2024-03-18T18:50:16.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.850+0000 7f69e92f6700 1 --2- 172.21.15.1:0/2149662739 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f69d006ac90 0x7f69d006d150 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.850+0000 7f69e92f6700 1 --2- 172.21.15.1:0/2149662739 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69e415ba60 0x7f69e41483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.850+0000 7f69e92f6700 1 --2- 172.21.15.1:0/2149662739 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69cc004850 0x7f69e4145ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.850+0000 7f69e92f6700 1 --2- 172.21.15.1:0/2149662739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69e415e1e0 0x7f69e415b520 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.850+0000 7f69e92f6700 1 -- 172.21.15.1:0/2149662739 >> 172.21.15.1:0/2149662739 conn(0x7f69e406d8d0 msgr2=0x7f69e40b00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:16.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.850+0000 7f69e92f6700 1 -- 172.21.15.1:0/2149662739 shutdown_connections 2024-03-18T18:50:16.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.850+0000 7f69e92f6700 1 -- 172.21.15.1:0/2149662739 wait complete. 2024-03-18T18:50:16.847 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 9 2024-03-18T18:50:16.859 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:16.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.988+0000 7fb3125f7700 1 Processor -- start 2024-03-18T18:50:16.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.988+0000 7fb3125f7700 1 -- start start 2024-03-18T18:50:16.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.989+0000 7fb3125f7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb30c06e460 0x7fb30c0cb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.989+0000 7fb3125f7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb30c06cfc0 con 0x7fb30c15a300 2024-03-18T18:50:16.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.989+0000 7fb3125f7700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb30c0cb730 con 0x7fb30c0ce6a0 2024-03-18T18:50:16.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.989+0000 7fb3125f7700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb30c0cba10 con 0x7fb30c06e460 2024-03-18T18:50:16.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.989+0000 7fb310b94700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb30c06e460 0x7fb30c0cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.989+0000 7fb30bfff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb30c15a300 0x7fb30c0cdf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55876/0 (socket says 172.21.15.1:55876) 2024-03-18T18:50:16.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.989+0000 7fb30bfff700 1 -- 172.21.15.1:0/344650670 learned_addr learned my addr 172.21.15.1:0/344650670 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:16.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.989+0000 7fb310b94700 1 -- 172.21.15.1:0/344650670 >> v1:172.21.15.165:6789/0 conn(0x7fb30c0ce6a0 legacy=0x7fb30c0ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.989+0000 7fb310b94700 1 -- 172.21.15.1:0/344650670 >> v1:172.21.15.1:6789/0 conn(0x7fb30c15a300 legacy=0x7fb30c0cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.989+0000 7fb310b94700 1 -- 172.21.15.1:0/344650670 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb30c0cbcf0 con 0x7fb30c06e460 2024-03-18T18:50:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.989+0000 7fb30affd700 1 -- 172.21.15.1:0/344650670 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084134972 0 0) 0x7fb30c06cfc0 con 0x7fb30c15a300 2024-03-18T18:50:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.990+0000 7fb310b94700 1 --2- 172.21.15.1:0/344650670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb30c06e460 0x7fb30c0cb030 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb304009d90 tx=0x7fb30400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13c4cddb6c3d372 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.990+0000 7fb30affd700 1 -- 172.21.15.1:0/344650670 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30400e070 con 0x7fb30c06e460 2024-03-18T18:50:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.990+0000 7fb30affd700 1 -- 172.21.15.1:0/344650670 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30400bbd0 con 0x7fb30c06e460 2024-03-18T18:50:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.990+0000 7fb30affd700 1 -- 172.21.15.1:0/344650670 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb304012430 con 0x7fb30c06e460 2024-03-18T18:50:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.990+0000 7fb3125f7700 1 -- 172.21.15.1:0/344650670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb30c06e460 msgr2=0x7fb30c0cb030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.990+0000 7fb3125f7700 1 --2- 172.21.15.1:0/344650670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb30c06e460 0x7fb30c0cb030 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb304009d90 tx=0x7fb30400b330 comp rx=0 tx=0).stop 2024-03-18T18:50:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.990+0000 7fb3125f7700 1 -- 172.21.15.1:0/344650670 shutdown_connections 2024-03-18T18:50:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.990+0000 7fb3125f7700 1 --2- 172.21.15.1:0/344650670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb30c06e460 0x7fb30c0cb030 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.990+0000 7fb3125f7700 1 -- 172.21.15.1:0/344650670 >> 172.21.15.1:0/344650670 conn(0x7fb30c0b1270 msgr2=0x7fb30c0b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.990+0000 7fb3125f7700 1 -- 172.21.15.1:0/344650670 shutdown_connections 2024-03-18T18:50:16.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.990+0000 7fb3125f7700 1 -- 172.21.15.1:0/344650670 wait complete. 2024-03-18T18:50:16.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.991+0000 7fb3125f7700 1 Processor -- start 2024-03-18T18:50:16.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.992+0000 7fb3125f7700 1 -- start start 2024-03-18T18:50:16.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.992+0000 7fb3125f7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb30c06e460 0x7fb30c0ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.992+0000 7fb3125f7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb30c0ce6a0 0x7fb30c0c4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.992+0000 7fb3125f7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb30c15a300 0x7fb30c15a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.992+0000 7fb3125f7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30c1575f0 con 0x7fb30c0ce6a0 2024-03-18T18:50:16.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.992+0000 7fb3125f7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30c156a40 con 0x7fb30c06e460 2024-03-18T18:50:16.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.992+0000 7fb3125f7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb30c15d7d0 con 0x7fb30c15a300 2024-03-18T18:50:16.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.992+0000 7fb30b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb30c0ce6a0 0x7fb30c0c4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.993+0000 7fb30b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb30c0ce6a0 0x7fb30c0c4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51944/0 (socket says 172.21.15.1:51944) 2024-03-18T18:50:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.993+0000 7fb30b7fe700 1 -- 172.21.15.1:0/2134665002 learned_addr learned my addr 172.21.15.1:0/2134665002 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.993+0000 7fb30bfff700 1 --2- 172.21.15.1:0/2134665002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb30c06e460 0x7fb30c0ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.993+0000 7fb310b94700 1 --2- 172.21.15.1:0/2134665002 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb30c15a300 0x7fb30c15a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.993+0000 7fb30b7fe700 1 -- 172.21.15.1:0/2134665002 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb30c15a300 msgr2=0x7fb30c15a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.993+0000 7fb30b7fe700 1 --2- 172.21.15.1:0/2134665002 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb30c15a300 0x7fb30c15a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.993+0000 7fb30b7fe700 1 -- 172.21.15.1:0/2134665002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb30c06e460 msgr2=0x7fb30c0ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.993+0000 7fb30b7fe700 1 --2- 172.21.15.1:0/2134665002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb30c06e460 0x7fb30c0ccb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.993+0000 7fb30b7fe700 1 -- 172.21.15.1:0/2134665002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb304009a70 con 0x7fb30c0ce6a0 2024-03-18T18:50:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.993+0000 7fb310b94700 1 --2- 172.21.15.1:0/2134665002 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb30c15a300 0x7fb30c15a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.993+0000 7fb30b7fe700 1 --2- 172.21.15.1:0/2134665002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb30c0ce6a0 0x7fb30c0c4090 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc00c4e0 tx=0x7fb2fc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.993+0000 7fb308ff9700 1 -- 172.21.15.1:0/2134665002 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc0052d0 con 0x7fb30c0ce6a0 2024-03-18T18:50:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.993+0000 7fb308ff9700 1 -- 172.21.15.1:0/2134665002 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2fc005430 con 0x7fb30c0ce6a0 2024-03-18T18:50:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.993+0000 7fb3125f7700 1 -- 172.21.15.1:0/2134665002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb30c156c50 con 0x7fb30c0ce6a0 2024-03-18T18:50:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.993+0000 7fb308ff9700 1 -- 172.21.15.1:0/2134665002 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc0038c0 con 0x7fb30c0ce6a0 2024-03-18T18:50:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.993+0000 7fb3125f7700 1 -- 172.21.15.1:0/2134665002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb30c157210 con 0x7fb30c0ce6a0 2024-03-18T18:50:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.994+0000 7fb3125f7700 1 -- 172.21.15.1:0/2134665002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb30c0676a0 con 0x7fb30c0ce6a0 2024-03-18T18:50:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.995+0000 7fb308ff9700 1 -- 172.21.15.1:0/2134665002 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb2fc003a20 con 0x7fb30c0ce6a0 2024-03-18T18:50:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.995+0000 7fb308ff9700 1 --2- 172.21.15.1:0/2134665002 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb2ec06ac90 0x7fb2ec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.995+0000 7fb308ff9700 1 -- 172.21.15.1:0/2134665002 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb2fc088a40 con 0x7fb30c0ce6a0 2024-03-18T18:50:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.995+0000 7fb30bfff700 1 --2- 172.21.15.1:0/2134665002 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb2ec06ac90 0x7fb2ec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:16.995+0000 7fb30bfff700 1 --2- 172.21.15.1:0/2134665002 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb2ec06ac90 0x7fb2ec06d150 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4007fd0 tx=0x7fb2f4007c20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:16.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.000+0000 7fb308ff9700 1 -- 172.21.15.1:0/2134665002 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb2fc08ae60 con 0x7fb30c0ce6a0 2024-03-18T18:50:17.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.203+0000 7fb3125f7700 1 -- 172.21.15.1:0/2134665002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb30c073f50 con 0x7fb30c0ce6a0 2024-03-18T18:50:17.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.204+0000 7fb308ff9700 1 -- 172.21.15.1:0/2134665002 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17830 (secure 0 0 0) 0x7fb2fc017090 con 0x7fb30c0ce6a0 2024-03-18T18:50:17.202 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:17.202 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:49.276961+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:17.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.206+0000 7fb3125f7700 1 -- 172.21.15.1:0/2134665002 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb2ec06ac90 msgr2=0x7fb2ec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:17.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.206+0000 7fb3125f7700 1 --2- 172.21.15.1:0/2134665002 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb2ec06ac90 0x7fb2ec06d150 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4007fd0 tx=0x7fb2f4007c20 comp rx=0 tx=0).stop 2024-03-18T18:50:17.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.206+0000 7fb3125f7700 1 -- 172.21.15.1:0/2134665002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb30c0ce6a0 msgr2=0x7fb30c0c4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:17.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.206+0000 7fb3125f7700 1 --2- 172.21.15.1:0/2134665002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb30c0ce6a0 0x7fb30c0c4090 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc00c4e0 tx=0x7fb2fc00c8a0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.207+0000 7fb3125f7700 1 -- 172.21.15.1:0/2134665002 shutdown_connections 2024-03-18T18:50:17.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.207+0000 7fb3125f7700 1 --2- 172.21.15.1:0/2134665002 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb2ec06ac90 0x7fb2ec06d150 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.207+0000 7fb3125f7700 1 --2- 172.21.15.1:0/2134665002 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb30c15a300 0x7fb30c15a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.207+0000 7fb3125f7700 1 --2- 172.21.15.1:0/2134665002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb30c06e460 0x7fb30c0ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.207+0000 7fb3125f7700 1 --2- 172.21.15.1:0/2134665002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb30c0ce6a0 0x7fb30c0c4090 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.207+0000 7fb3125f7700 1 -- 172.21.15.1:0/2134665002 >> 172.21.15.1:0/2134665002 conn(0x7fb30c0b1270 msgr2=0x7fb30c0cfb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:17.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.207+0000 7fb3125f7700 1 -- 172.21.15.1:0/2134665002 shutdown_connections 2024-03-18T18:50:17.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.207+0000 7fb3125f7700 1 -- 172.21.15.1:0/2134665002 wait complete. 2024-03-18T18:50:17.216 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T18:49:42.800801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T18:49:43.776684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T18:49:46.854608+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T18:50:17.216 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T18:50:17.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.347+0000 7ff3d31a1700 1 Processor -- start 2024-03-18T18:50:17.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.348+0000 7ff3d31a1700 1 -- start start 2024-03-18T18:50:17.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.348+0000 7ff3d31a1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3cc15d3f0 0x7ff3cc0ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:17.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.348+0000 7ff3d31a1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff3cc06ae60 0x7ff3cc0b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:17.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.348+0000 7ff3d31a1700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3cc0ace20 con 0x7ff3cc0b4d80 2024-03-18T18:50:17.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.348+0000 7ff3d31a1700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3cc0adcd0 con 0x7ff3cc06ae60 2024-03-18T18:50:17.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.348+0000 7ff3d31a1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0af0b0 con 0x7ff3cc15d3f0 2024-03-18T18:50:17.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.349+0000 7ff3d0f3d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3cc15d3f0 0x7ff3cc0ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:17.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.349+0000 7ff3d173e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff3cc06ae60 0x7ff3cc0b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:17.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.349+0000 7ff3cbfff700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7ff3cc0b4d80 0x7ff3cc06a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:60858/0 (socket says 172.21.15.1:60858) 2024-03-18T18:50:17.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.349+0000 7ff3d0f3d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3cc15d3f0 0x7ff3cc0ad790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:51834/0 (socket says 172.21.15.1:51834) 2024-03-18T18:50:17.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.349+0000 7ff3d173e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff3cc06ae60 0x7ff3cc0b37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:55842/0 (socket says 172.21.15.1:55842) 2024-03-18T18:50:17.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.349+0000 7ff3cbfff700 1 -- 172.21.15.1:0/2898030588 learned_addr learned my addr 172.21.15.1:0/2898030588 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:17.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.349+0000 7ff3d0f3d700 1 -- 172.21.15.1:0/2898030588 >> v2:172.21.15.165:3300/0 conn(0x7ff3cc06ae60 msgr2=0x7ff3cc0b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:17.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.349+0000 7ff3d0f3d700 1 --2- 172.21.15.1:0/2898030588 >> v2:172.21.15.165:3300/0 conn(0x7ff3cc06ae60 0x7ff3cc0b37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.349+0000 7ff3d0f3d700 1 -- 172.21.15.1:0/2898030588 >> v1:172.21.15.165:6789/0 conn(0x7ff3cc0b4d80 legacy=0x7ff3cc06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:17.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.349+0000 7ff3d0f3d700 1 -- 172.21.15.1:0/2898030588 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc06c380 con 0x7ff3cc15d3f0 2024-03-18T18:50:17.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.349+0000 7ff3d0f3d700 1 --2- 172.21.15.1:0/2898030588 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3cc15d3f0 0x7ff3cc0ad790 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0009840 tx=0x7ff3c0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d397ac67c72c929 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:17.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.350+0000 7ff3cb7fe700 1 -- 172.21.15.1:0/2898030588 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0011830 con 0x7ff3cc15d3f0 2024-03-18T18:50:17.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.350+0000 7ff3cb7fe700 1 -- 172.21.15.1:0/2898030588 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c0011990 con 0x7ff3cc15d3f0 2024-03-18T18:50:17.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.350+0000 7ff3cb7fe700 1 -- 172.21.15.1:0/2898030588 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0011d20 con 0x7ff3cc15d3f0 2024-03-18T18:50:17.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.350+0000 7ff3d31a1700 1 -- 172.21.15.1:0/2898030588 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3cc15d3f0 msgr2=0x7ff3cc0ad790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:17.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.350+0000 7ff3d31a1700 1 --2- 172.21.15.1:0/2898030588 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3cc15d3f0 0x7ff3cc0ad790 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0009840 tx=0x7ff3c0009b50 comp rx=0 tx=0).stop 2024-03-18T18:50:17.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.350+0000 7ff3d31a1700 1 -- 172.21.15.1:0/2898030588 shutdown_connections 2024-03-18T18:50:17.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.350+0000 7ff3d31a1700 1 --2- 172.21.15.1:0/2898030588 >> v2:172.21.15.165:3300/0 conn(0x7ff3cc06ae60 0x7ff3cc0b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.350+0000 7ff3d31a1700 1 --2- 172.21.15.1:0/2898030588 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3cc15d3f0 0x7ff3cc0ad790 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.350+0000 7ff3d31a1700 1 -- 172.21.15.1:0/2898030588 >> 172.21.15.1:0/2898030588 conn(0x7ff3cc06b860 msgr2=0x7ff3cc06bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:17.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.350+0000 7ff3d31a1700 1 -- 172.21.15.1:0/2898030588 shutdown_connections 2024-03-18T18:50:17.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.350+0000 7ff3d31a1700 1 -- 172.21.15.1:0/2898030588 wait complete. 2024-03-18T18:50:17.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.351+0000 7ff3d31a1700 1 Processor -- start 2024-03-18T18:50:17.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.351+0000 7ff3d31a1700 1 -- start start 2024-03-18T18:50:17.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.352+0000 7ff3d31a1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3cc06ae60 0x7ff3cc0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:17.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.352+0000 7ff3d31a1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3cc0b4d80 0x7ff3cc14ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:17.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.352+0000 7ff3d31a1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3cc15d3f0 0x7ff3cc0b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:17.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.352+0000 7ff3d31a1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc14cf50 con 0x7ff3cc06ae60 2024-03-18T18:50:17.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.352+0000 7ff3d31a1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0ce100 con 0x7ff3cc0b4d80 2024-03-18T18:50:17.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.352+0000 7ff3d31a1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc1e7b90 con 0x7ff3cc15d3f0 2024-03-18T18:50:17.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.352+0000 7ff3d0f3d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3cc06ae60 0x7ff3cc0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:17.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.352+0000 7ff3d0f3d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3cc06ae60 0x7ff3cc0ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51952/0 (socket says 172.21.15.1:51952) 2024-03-18T18:50:17.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.352+0000 7ff3d0f3d700 1 -- 172.21.15.1:0/4101662909 learned_addr learned my addr 172.21.15.1:0/4101662909 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:17.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.353+0000 7ff3d173e700 1 --2- 172.21.15.1:0/4101662909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3cc15d3f0 0x7ff3cc0b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:17.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.353+0000 7ff3cbfff700 1 --2- 172.21.15.1:0/4101662909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3cc0b4d80 0x7ff3cc14ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:17.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.353+0000 7ff3d0f3d700 1 -- 172.21.15.1:0/4101662909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3cc15d3f0 msgr2=0x7ff3cc0b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:17.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.353+0000 7ff3d0f3d700 1 --2- 172.21.15.1:0/4101662909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3cc15d3f0 0x7ff3cc0b0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.353+0000 7ff3d0f3d700 1 -- 172.21.15.1:0/4101662909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3cc0b4d80 msgr2=0x7ff3cc14ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:17.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.353+0000 7ff3d0f3d700 1 --2- 172.21.15.1:0/4101662909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3cc0b4d80 0x7ff3cc14ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.353+0000 7ff3d0f3d700 1 -- 172.21.15.1:0/4101662909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c00094a0 con 0x7ff3cc06ae60 2024-03-18T18:50:17.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.353+0000 7ff3d173e700 1 --2- 172.21.15.1:0/4101662909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3cc15d3f0 0x7ff3cc0b0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:17.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.353+0000 7ff3cbfff700 1 --2- 172.21.15.1:0/4101662909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3cc0b4d80 0x7ff3cc14ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:17.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.353+0000 7ff3d0f3d700 1 --2- 172.21.15.1:0/4101662909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3cc06ae60 0x7ff3cc0ce590 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0014040 tx=0x7ff3c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:17.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.353+0000 7ff3c97fa700 1 -- 172.21.15.1:0/4101662909 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c00036a0 con 0x7ff3cc06ae60 2024-03-18T18:50:17.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.353+0000 7ff3c97fa700 1 -- 172.21.15.1:0/4101662909 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c001f070 con 0x7ff3cc06ae60 2024-03-18T18:50:17.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.353+0000 7ff3d31a1700 1 -- 172.21.15.1:0/4101662909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3cc1e7e70 con 0x7ff3cc06ae60 2024-03-18T18:50:17.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.353+0000 7ff3c97fa700 1 -- 172.21.15.1:0/4101662909 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0003a00 con 0x7ff3cc06ae60 2024-03-18T18:50:17.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.353+0000 7ff3d31a1700 1 -- 172.21.15.1:0/4101662909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3cc0cdd90 con 0x7ff3cc06ae60 2024-03-18T18:50:17.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.354+0000 7ff3d31a1700 1 -- 172.21.15.1:0/4101662909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3cc0676a0 con 0x7ff3cc06ae60 2024-03-18T18:50:17.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.355+0000 7ff3c97fa700 1 -- 172.21.15.1:0/4101662909 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff3c0003b60 con 0x7ff3cc06ae60 2024-03-18T18:50:17.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.355+0000 7ff3c97fa700 1 --2- 172.21.15.1:0/4101662909 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff3b006ac90 0x7ff3b006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:17.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.355+0000 7ff3c97fa700 1 -- 172.21.15.1:0/4101662909 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff3c0021070 con 0x7ff3cc06ae60 2024-03-18T18:50:17.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.355+0000 7ff3cbfff700 1 --2- 172.21.15.1:0/4101662909 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff3b006ac90 0x7ff3b006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:17.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.355+0000 7ff3cbfff700 1 --2- 172.21.15.1:0/4101662909 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff3b006ac90 0x7ff3b006d150 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8007fd0 tx=0x7ff3b8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:17.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.360+0000 7ff3c97fa700 1 -- 172.21.15.1:0/4101662909 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff3c008e030 con 0x7ff3cc06ae60 2024-03-18T18:50:17.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.563+0000 7ff3d31a1700 1 -- 172.21.15.1:0/4101662909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff3cc072850 con 0x7ff3cc06ae60 2024-03-18T18:50:17.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.563+0000 7ff3c97fa700 1 -- 172.21.15.1:0/4101662909 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff3c00150a0 con 0x7ff3cc06ae60 2024-03-18T18:50:17.560 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T18:50:17.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.565+0000 7ff3d31a1700 1 -- 172.21.15.1:0/4101662909 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff3b006ac90 msgr2=0x7ff3b006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:17.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.565+0000 7ff3d31a1700 1 --2- 172.21.15.1:0/4101662909 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff3b006ac90 0x7ff3b006d150 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8007fd0 tx=0x7ff3b8009040 comp rx=0 tx=0).stop 2024-03-18T18:50:17.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.565+0000 7ff3d31a1700 1 -- 172.21.15.1:0/4101662909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3cc06ae60 msgr2=0x7ff3cc0ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:17.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.565+0000 7ff3d31a1700 1 --2- 172.21.15.1:0/4101662909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3cc06ae60 0x7ff3cc0ce590 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0014040 tx=0x7ff3c0004750 comp rx=0 tx=0).stop 2024-03-18T18:50:17.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.565+0000 7ff3d31a1700 1 -- 172.21.15.1:0/4101662909 shutdown_connections 2024-03-18T18:50:17.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.566+0000 7ff3d31a1700 1 --2- 172.21.15.1:0/4101662909 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff3b006ac90 0x7ff3b006d150 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.566+0000 7ff3d31a1700 1 --2- 172.21.15.1:0/4101662909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3cc15d3f0 0x7ff3cc0b0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.566+0000 7ff3d31a1700 1 --2- 172.21.15.1:0/4101662909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3cc06ae60 0x7ff3cc0ce590 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.566+0000 7ff3d31a1700 1 --2- 172.21.15.1:0/4101662909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3cc0b4d80 0x7ff3cc14ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.566+0000 7ff3d31a1700 1 -- 172.21.15.1:0/4101662909 >> 172.21.15.1:0/4101662909 conn(0x7ff3cc06b860 msgr2=0x7ff3cc0cf2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:17.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.566+0000 7ff3d31a1700 1 -- 172.21.15.1:0/4101662909 shutdown_connections 2024-03-18T18:50:17.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.566+0000 7ff3d31a1700 1 -- 172.21.15.1:0/4101662909 wait complete. 2024-03-18T18:50:17.574 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T18:50:17.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.705+0000 7f1e935ea700 1 Processor -- start 2024-03-18T18:50:17.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.706+0000 7f1e935ea700 1 -- start start 2024-03-18T18:50:17.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.706+0000 7f1e935ea700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e8c0cc0f0 0x7f1e8c0add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:17.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.706+0000 7f1e935ea700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8c0ce8f0 con 0x7f1e8c15a1d0 2024-03-18T18:50:17.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.707+0000 7f1e935ea700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8c0ae260 con 0x7f1e8c071d30 2024-03-18T18:50:17.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.707+0000 7f1e935ea700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c0cdb00 con 0x7f1e8c0cc0f0 2024-03-18T18:50:17.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.707+0000 7f1e91386700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1e8c15a1d0 0x7f1e8c071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55892/0 (socket says 172.21.15.1:55892) 2024-03-18T18:50:17.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.707+0000 7f1e91386700 1 -- 172.21.15.1:0/1725821963 learned_addr learned my addr 172.21.15.1:0/1725821963 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:17.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.707+0000 7f1e91b87700 1 --2- 172.21.15.1:0/1725821963 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e8c0cc0f0 0x7f1e8c0add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:17.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.707+0000 7f1e7ffff700 1 -- 172.21.15.1:0/1725821963 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793333218 0 0) 0x7f1e8c0ce8f0 con 0x7f1e8c15a1d0 2024-03-18T18:50:17.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.707+0000 7f1e7ffff700 1 -- 172.21.15.1:0/1725821963 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e780035b0 con 0x7f1e8c15a1d0 2024-03-18T18:50:17.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.707+0000 7f1e7ffff700 1 -- 172.21.15.1:0/1725821963 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175233009 0 0) 0x7f1e8c0ae260 con 0x7f1e8c071d30 2024-03-18T18:50:17.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.707+0000 7f1e91b87700 1 -- 172.21.15.1:0/1725821963 >> v1:172.21.15.165:6789/0 conn(0x7f1e8c071d30 legacy=0x7f1e8c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:17.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.707+0000 7f1e91b87700 1 -- 172.21.15.1:0/1725821963 >> v1:172.21.15.1:6789/0 conn(0x7f1e8c15a1d0 legacy=0x7f1e8c071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:17.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.707+0000 7f1e91b87700 1 -- 172.21.15.1:0/1725821963 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e8c0cdce0 con 0x7f1e8c0cc0f0 2024-03-18T18:50:17.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.707+0000 7f1e91b87700 1 --2- 172.21.15.1:0/1725821963 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e8c0cc0f0 0x7f1e8c0add20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009dc0 tx=0x7f1e8400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a12accc97d336a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:17.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.708+0000 7f1e7ffff700 1 -- 172.21.15.1:0/1725821963 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8400e070 con 0x7f1e8c0cc0f0 2024-03-18T18:50:17.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.708+0000 7f1e7ffff700 1 -- 172.21.15.1:0/1725821963 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8400bb20 con 0x7f1e8c0cc0f0 2024-03-18T18:50:17.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.708+0000 7f1e7ffff700 1 -- 172.21.15.1:0/1725821963 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8400bed0 con 0x7f1e8c0cc0f0 2024-03-18T18:50:17.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.708+0000 7f1e935ea700 1 -- 172.21.15.1:0/1725821963 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e8c0cc0f0 msgr2=0x7f1e8c0add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:17.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.708+0000 7f1e935ea700 1 --2- 172.21.15.1:0/1725821963 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e8c0cc0f0 0x7f1e8c0add20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009dc0 tx=0x7f1e8400b330 comp rx=0 tx=0).stop 2024-03-18T18:50:17.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.708+0000 7f1e935ea700 1 -- 172.21.15.1:0/1725821963 shutdown_connections 2024-03-18T18:50:17.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.708+0000 7f1e935ea700 1 --2- 172.21.15.1:0/1725821963 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e8c0cc0f0 0x7f1e8c0add20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.708+0000 7f1e935ea700 1 -- 172.21.15.1:0/1725821963 >> 172.21.15.1:0/1725821963 conn(0x7f1e8c0cc9b0 msgr2=0x7f1e8c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:17.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.708+0000 7f1e935ea700 1 -- 172.21.15.1:0/1725821963 shutdown_connections 2024-03-18T18:50:17.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.708+0000 7f1e935ea700 1 -- 172.21.15.1:0/1725821963 wait complete. 2024-03-18T18:50:17.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.709+0000 7f1e935ea700 1 Processor -- start 2024-03-18T18:50:17.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.709+0000 7f1e935ea700 1 -- start start 2024-03-18T18:50:17.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.710+0000 7f1e935ea700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e8c071d30 0x7f1e8c0b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:17.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.710+0000 7f1e935ea700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e8c0cc0f0 0x7f1e8c0b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:17.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.710+0000 7f1e935ea700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e8c15a1d0 0x7f1e8c0b5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:17.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.710+0000 7f1e935ea700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c0b3c30 con 0x7f1e8c071d30 2024-03-18T18:50:17.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.710+0000 7f1e935ea700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c0b3f10 con 0x7f1e8c0cc0f0 2024-03-18T18:50:17.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.710+0000 7f1e935ea700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c0b41f0 con 0x7f1e8c15a1d0 2024-03-18T18:50:17.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.710+0000 7f1e91386700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e8c071d30 0x7f1e8c0b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:17.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.710+0000 7f1e91386700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e8c071d30 0x7f1e8c0b35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51956/0 (socket says 172.21.15.1:51956) 2024-03-18T18:50:17.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.710+0000 7f1e91386700 1 -- 172.21.15.1:0/246192808 learned_addr learned my addr 172.21.15.1:0/246192808 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:17.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.710+0000 7f1e90b85700 1 --2- 172.21.15.1:0/246192808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e8c0cc0f0 0x7f1e8c0b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:17.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.710+0000 7f1e91b87700 1 --2- 172.21.15.1:0/246192808 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e8c15a1d0 0x7f1e8c0b5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:17.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.711+0000 7f1e91386700 1 -- 172.21.15.1:0/246192808 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e8c15a1d0 msgr2=0x7f1e8c0b5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:17.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.711+0000 7f1e91386700 1 --2- 172.21.15.1:0/246192808 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e8c15a1d0 0x7f1e8c0b5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.711+0000 7f1e91386700 1 -- 172.21.15.1:0/246192808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e8c0cc0f0 msgr2=0x7f1e8c0b4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:17.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.711+0000 7f1e91386700 1 --2- 172.21.15.1:0/246192808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e8c0cc0f0 0x7f1e8c0b4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.711+0000 7f1e91386700 1 -- 172.21.15.1:0/246192808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e84009a70 con 0x7f1e8c071d30 2024-03-18T18:50:17.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.711+0000 7f1e91386700 1 --2- 172.21.15.1:0/246192808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e8c071d30 0x7f1e8c0b35a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000c4d0 tx=0x7f1e8000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:17.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.711+0000 7f1e7dffb700 1 -- 172.21.15.1:0/246192808 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e800074e0 con 0x7f1e8c071d30 2024-03-18T18:50:17.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.711+0000 7f1e7dffb700 1 -- 172.21.15.1:0/246192808 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e80007660 con 0x7f1e8c071d30 2024-03-18T18:50:17.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.711+0000 7f1e935ea700 1 -- 172.21.15.1:0/246192808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e8c0b44d0 con 0x7f1e8c071d30 2024-03-18T18:50:17.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.711+0000 7f1e7dffb700 1 -- 172.21.15.1:0/246192808 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e80005a70 con 0x7f1e8c071d30 2024-03-18T18:50:17.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.711+0000 7f1e935ea700 1 -- 172.21.15.1:0/246192808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e8c15a7d0 con 0x7f1e8c071d30 2024-03-18T18:50:17.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.712+0000 7f1e935ea700 1 -- 172.21.15.1:0/246192808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e8c15dd90 con 0x7f1e8c071d30 2024-03-18T18:50:17.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.712+0000 7f1e7dffb700 1 -- 172.21.15.1:0/246192808 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1e80005bd0 con 0x7f1e8c071d30 2024-03-18T18:50:17.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.713+0000 7f1e7dffb700 1 --2- 172.21.15.1:0/246192808 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1e7006ac90 0x7f1e7006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:17.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.713+0000 7f1e7dffb700 1 -- 172.21.15.1:0/246192808 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1e800889c0 con 0x7f1e8c071d30 2024-03-18T18:50:17.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.713+0000 7f1e90b85700 1 --2- 172.21.15.1:0/246192808 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1e7006ac90 0x7f1e7006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:17.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.713+0000 7f1e90b85700 1 --2- 172.21.15.1:0/246192808 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1e7006ac90 0x7f1e7006d150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c1f24d0 tx=0x7f1e88009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:17.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.717+0000 7f1e7dffb700 1 -- 172.21.15.1:0/246192808 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1e8c15dd90 con 0x7f1e8c071d30 2024-03-18T18:50:17.888 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:50:17.891 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:50:17.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.919+0000 7f1e935ea700 1 -- 172.21.15.1:0/246192808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1e8c0b49f0 con 0x7f1e8c071d30 2024-03-18T18:50:17.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.919+0000 7f1e7dffb700 1 -- 172.21.15.1:0/246192808 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1e8c0b49f0 con 0x7f1e8c071d30 2024-03-18T18:50:17.916 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:17.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.921+0000 7f1e935ea700 1 -- 172.21.15.1:0/246192808 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1e7006ac90 msgr2=0x7f1e7006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:17.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.921+0000 7f1e935ea700 1 --2- 172.21.15.1:0/246192808 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1e7006ac90 0x7f1e7006d150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c1f24d0 tx=0x7f1e88009040 comp rx=0 tx=0).stop 2024-03-18T18:50:17.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.921+0000 7f1e935ea700 1 -- 172.21.15.1:0/246192808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e8c071d30 msgr2=0x7f1e8c0b35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:17.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.921+0000 7f1e935ea700 1 --2- 172.21.15.1:0/246192808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e8c071d30 0x7f1e8c0b35a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000c4d0 tx=0x7f1e8000c7e0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.922+0000 7f1e935ea700 1 -- 172.21.15.1:0/246192808 shutdown_connections 2024-03-18T18:50:17.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.922+0000 7f1e935ea700 1 --2- 172.21.15.1:0/246192808 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1e7006ac90 0x7f1e7006d150 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.922+0000 7f1e935ea700 1 --2- 172.21.15.1:0/246192808 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e8c15a1d0 0x7f1e8c0b5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.922+0000 7f1e935ea700 1 --2- 172.21.15.1:0/246192808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e8c071d30 0x7f1e8c0b35a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.922+0000 7f1e935ea700 1 --2- 172.21.15.1:0/246192808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e8c0cc0f0 0x7f1e8c0b4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:17.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.922+0000 7f1e935ea700 1 -- 172.21.15.1:0/246192808 >> 172.21.15.1:0/246192808 conn(0x7f1e8c0cc9b0 msgr2=0x7f1e8c159030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:17.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.922+0000 7f1e935ea700 1 -- 172.21.15.1:0/246192808 shutdown_connections 2024-03-18T18:50:17.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:17.922+0000 7f1e935ea700 1 -- 172.21.15.1:0/246192808 wait complete. 2024-03-18T18:50:17.931 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T18:50:18.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.119+0000 7f856215e700 1 Processor -- start 2024-03-18T18:50:18.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.119+0000 7f856215e700 1 -- start start 2024-03-18T18:50:18.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.120+0000 7f856215e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f855c099a40 0x7f855c099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:18.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.120+0000 7f856215e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f855c0d0cd0 0x7f855c09a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:18.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.120+0000 7f856215e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f855c0cf3d0 0x7f855c0cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:18.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.120+0000 7f856215e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f855c06bdf0 con 0x7f855c0d0cd0 2024-03-18T18:50:18.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.120+0000 7f856215e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f855c15d9f0 con 0x7f855c099a40 2024-03-18T18:50:18.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.120+0000 7f856215e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f855c0d0730 con 0x7f855c0cf3d0 2024-03-18T18:50:18.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.120+0000 7f855affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f855c0d0cd0 0x7f855c09a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:18.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.120+0000 7f855affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f855c0d0cd0 0x7f855c09a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51968/0 (socket says 172.21.15.1:51968) 2024-03-18T18:50:18.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.120+0000 7f855affd700 1 -- 172.21.15.1:0/3330862539 learned_addr learned my addr 172.21.15.1:0/3330862539 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:18.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.121+0000 7f855b7fe700 1 --2- 172.21.15.1:0/3330862539 >> v2:172.21.15.165:3300/0 conn(0x7f855c099a40 0x7f855c099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:18.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.121+0000 7f855bfff700 1 --2- 172.21.15.1:0/3330862539 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f855c0cf3d0 0x7f855c0cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:18.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.121+0000 7f855affd700 1 -- 172.21.15.1:0/3330862539 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f855c0cf3d0 msgr2=0x7f855c0cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:18.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.121+0000 7f855affd700 1 --2- 172.21.15.1:0/3330862539 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f855c0cf3d0 0x7f855c0cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.121+0000 7f855affd700 1 -- 172.21.15.1:0/3330862539 >> v2:172.21.15.165:3300/0 conn(0x7f855c099a40 msgr2=0x7f855c099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:18.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.121+0000 7f855affd700 1 --2- 172.21.15.1:0/3330862539 >> v2:172.21.15.165:3300/0 conn(0x7f855c099a40 0x7f855c099e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.121+0000 7f855affd700 1 -- 172.21.15.1:0/3330862539 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f855c06bb10 con 0x7f855c0d0cd0 2024-03-18T18:50:18.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.121+0000 7f855affd700 1 --2- 172.21.15.1:0/3330862539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f855c0d0cd0 0x7f855c09a360 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f855c071c90 tx=0x7f854c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77fa2518af4c478a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:18.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.121+0000 7f855a7fc700 1 -- 172.21.15.1:0/3330862539 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c00b860 con 0x7f855c0d0cd0 2024-03-18T18:50:18.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.121+0000 7f855a7fc700 1 -- 172.21.15.1:0/3330862539 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854c00b9c0 con 0x7f855c0d0cd0 2024-03-18T18:50:18.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.121+0000 7f855a7fc700 1 -- 172.21.15.1:0/3330862539 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c00bd50 con 0x7f855c0d0cd0 2024-03-18T18:50:18.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.122+0000 7f856215e700 1 -- 172.21.15.1:0/3330862539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f855c0d0cd0 msgr2=0x7f855c09a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:18.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.122+0000 7f856215e700 1 --2- 172.21.15.1:0/3330862539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f855c0d0cd0 0x7f855c09a360 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f855c071c90 tx=0x7f854c009c00 comp rx=0 tx=0).stop 2024-03-18T18:50:18.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.122+0000 7f856215e700 1 -- 172.21.15.1:0/3330862539 shutdown_connections 2024-03-18T18:50:18.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.122+0000 7f856215e700 1 --2- 172.21.15.1:0/3330862539 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f855c0cf3d0 0x7f855c0cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.122+0000 7f856215e700 1 --2- 172.21.15.1:0/3330862539 >> v2:172.21.15.165:3300/0 conn(0x7f855c099a40 0x7f855c099e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.122+0000 7f856215e700 1 --2- 172.21.15.1:0/3330862539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f855c0d0cd0 0x7f855c09a360 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.122+0000 7f856215e700 1 -- 172.21.15.1:0/3330862539 >> 172.21.15.1:0/3330862539 conn(0x7f855c0b2bf0 msgr2=0x7f855c0b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:18.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.122+0000 7f856215e700 1 -- 172.21.15.1:0/3330862539 shutdown_connections 2024-03-18T18:50:18.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.122+0000 7f856215e700 1 -- 172.21.15.1:0/3330862539 wait complete. 2024-03-18T18:50:18.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.123+0000 7f856215e700 1 Processor -- start 2024-03-18T18:50:18.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.123+0000 7f856215e700 1 -- start start 2024-03-18T18:50:18.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.124+0000 7f856215e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f855c099a40 0x7f855c0ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:18.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.124+0000 7f856215e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f855c0cf3d0 0x7f855c0ae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:18.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.124+0000 7f856215e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f855c0d0cd0 0x7f855c0afab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:18.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.124+0000 7f856215e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855c0afff0 con 0x7f855c0cf3d0 2024-03-18T18:50:18.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.124+0000 7f856215e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855c0af720 con 0x7f855c0d0cd0 2024-03-18T18:50:18.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.124+0000 7f856215e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f855c0aee80 con 0x7f855c099a40 2024-03-18T18:50:18.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.124+0000 7f855affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f855c0cf3d0 0x7f855c0ae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:18.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.125+0000 7f855affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f855c0cf3d0 0x7f855c0ae7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51982/0 (socket says 172.21.15.1:51982) 2024-03-18T18:50:18.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.125+0000 7f855affd700 1 -- 172.21.15.1:0/4130940028 learned_addr learned my addr 172.21.15.1:0/4130940028 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:18.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.125+0000 7f855b7fe700 1 --2- 172.21.15.1:0/4130940028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f855c099a40 0x7f855c0ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:18.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.125+0000 7f855bfff700 1 --2- 172.21.15.1:0/4130940028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f855c0d0cd0 0x7f855c0afab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:18.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.125+0000 7f855affd700 1 -- 172.21.15.1:0/4130940028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f855c099a40 msgr2=0x7f855c0ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:18.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.125+0000 7f855affd700 1 --2- 172.21.15.1:0/4130940028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f855c099a40 0x7f855c0ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.125+0000 7f855affd700 1 -- 172.21.15.1:0/4130940028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f855c0d0cd0 msgr2=0x7f855c0afab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:18.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.125+0000 7f855affd700 1 --2- 172.21.15.1:0/4130940028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f855c0d0cd0 0x7f855c0afab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.125+0000 7f855affd700 1 -- 172.21.15.1:0/4130940028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c009550 con 0x7f855c0cf3d0 2024-03-18T18:50:18.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.125+0000 7f855affd700 1 --2- 172.21.15.1:0/4130940028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f855c0cf3d0 0x7f855c0ae7a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f854c009840 tx=0x7f854c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:18.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.125+0000 7f8553fff700 1 -- 172.21.15.1:0/4130940028 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c004450 con 0x7f855c0cf3d0 2024-03-18T18:50:18.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.125+0000 7f8553fff700 1 -- 172.21.15.1:0/4130940028 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854c01e070 con 0x7f855c0cf3d0 2024-03-18T18:50:18.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.125+0000 7f856215e700 1 -- 172.21.15.1:0/4130940028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f855c0af160 con 0x7f855c0cf3d0 2024-03-18T18:50:18.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.125+0000 7f8553fff700 1 -- 172.21.15.1:0/4130940028 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c0038a0 con 0x7f855c0cf3d0 2024-03-18T18:50:18.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.125+0000 7f856215e700 1 -- 172.21.15.1:0/4130940028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f855c0c8300 con 0x7f855c0cf3d0 2024-03-18T18:50:18.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.126+0000 7f856215e700 1 -- 172.21.15.1:0/4130940028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f855c0676a0 con 0x7f855c0cf3d0 2024-03-18T18:50:18.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.127+0000 7f8553fff700 1 -- 172.21.15.1:0/4130940028 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f854c003a00 con 0x7f855c0cf3d0 2024-03-18T18:50:18.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.127+0000 7f8553fff700 1 --2- 172.21.15.1:0/4130940028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f853c06ac90 0x7f853c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:18.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.127+0000 7f8553fff700 1 -- 172.21.15.1:0/4130940028 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f854c020070 con 0x7f855c0cf3d0 2024-03-18T18:50:18.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.127+0000 7f855b7fe700 1 --2- 172.21.15.1:0/4130940028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f853c06ac90 0x7f853c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:18.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.128+0000 7f855b7fe700 1 --2- 172.21.15.1:0/4130940028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f853c06ac90 0x7f853c06d150 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f855c0c4b20 tx=0x7f8544008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:18.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.132+0000 7f8553fff700 1 -- 172.21.15.1:0/4130940028 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f854c0557e0 con 0x7f855c0cf3d0 2024-03-18T18:50:18.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.335+0000 7f856215e700 1 -- 172.21.15.1:0/4130940028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f855c15c190 con 0x7f855c0cf3d0 2024-03-18T18:50:18.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.335+0000 7f8553fff700 1 -- 172.21.15.1:0/4130940028 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f854c0150a0 con 0x7f855c0cf3d0 2024-03-18T18:50:18.332 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T18:50:18.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.337+0000 7f856215e700 1 -- 172.21.15.1:0/4130940028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f853c06ac90 msgr2=0x7f853c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:18.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.337+0000 7f856215e700 1 --2- 172.21.15.1:0/4130940028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f853c06ac90 0x7f853c06d150 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f855c0c4b20 tx=0x7f8544008040 comp rx=0 tx=0).stop 2024-03-18T18:50:18.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.337+0000 7f856215e700 1 -- 172.21.15.1:0/4130940028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f855c0cf3d0 msgr2=0x7f855c0ae7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:18.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.337+0000 7f856215e700 1 --2- 172.21.15.1:0/4130940028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f855c0cf3d0 0x7f855c0ae7a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f854c009840 tx=0x7f854c004750 comp rx=0 tx=0).stop 2024-03-18T18:50:18.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.337+0000 7f856215e700 1 -- 172.21.15.1:0/4130940028 shutdown_connections 2024-03-18T18:50:18.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.337+0000 7f856215e700 1 --2- 172.21.15.1:0/4130940028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f853c06ac90 0x7f853c06d150 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.337+0000 7f856215e700 1 --2- 172.21.15.1:0/4130940028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f855c099a40 0x7f855c0ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.337+0000 7f856215e700 1 --2- 172.21.15.1:0/4130940028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f855c0d0cd0 0x7f855c0afab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.337+0000 7f856215e700 1 --2- 172.21.15.1:0/4130940028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f855c0cf3d0 0x7f855c0ae7a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.337+0000 7f856215e700 1 -- 172.21.15.1:0/4130940028 >> 172.21.15.1:0/4130940028 conn(0x7f855c0b2bf0 msgr2=0x7f855c0cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:18.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.337+0000 7f856215e700 1 -- 172.21.15.1:0/4130940028 shutdown_connections 2024-03-18T18:50:18.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.337+0000 7f856215e700 1 -- 172.21.15.1:0/4130940028 wait complete. 2024-03-18T18:50:18.346 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T18:50:18.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.478+0000 7f906dd7e700 1 Processor -- start 2024-03-18T18:50:18.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.478+0000 7f906dd7e700 1 -- start start 2024-03-18T18:50:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.479+0000 7f906dd7e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9068071550 0x7f90681661f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.479+0000 7f906dd7e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f906815b750 con 0x7f90680aecb0 2024-03-18T18:50:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.479+0000 7f906dd7e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90680ccc20 con 0x7f906815b370 2024-03-18T18:50:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.479+0000 7f906dd7e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90680cf510 con 0x7f9068071550 2024-03-18T18:50:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.479+0000 7f9067fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9068071550 0x7f90681661f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.479+0000 7f9066ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f90680aecb0 0x7f9068070e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55894/0 (socket says 172.21.15.1:55894) 2024-03-18T18:50:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.479+0000 7f9066ffd700 1 -- 172.21.15.1:0/3812450400 learned_addr learned my addr 172.21.15.1:0/3812450400 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.480+0000 7f9067fff700 1 -- 172.21.15.1:0/3812450400 >> v1:172.21.15.165:6789/0 conn(0x7f906815b370 legacy=0x7f90680cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.480+0000 7f9067fff700 1 -- 172.21.15.1:0/3812450400 >> v1:172.21.15.1:6789/0 conn(0x7f90680aecb0 legacy=0x7f9068070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.480+0000 7f9067fff700 1 -- 172.21.15.1:0/3812450400 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90680cc2f0 con 0x7f9068071550 2024-03-18T18:50:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.480+0000 7f90667fc700 1 -- 172.21.15.1:0/3812450400 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775399329 0 0) 0x7f906815b750 con 0x7f90680aecb0 2024-03-18T18:50:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.480+0000 7f9067fff700 1 --2- 172.21.15.1:0/3812450400 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9068071550 0x7f90681661f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9060009d90 tx=0x7f906000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1049d4c67d39673d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.480+0000 7f90667fc700 1 -- 172.21.15.1:0/3812450400 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906000e070 con 0x7f9068071550 2024-03-18T18:50:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.480+0000 7f90667fc700 1 -- 172.21.15.1:0/3812450400 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906000bbd0 con 0x7f9068071550 2024-03-18T18:50:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.480+0000 7f90667fc700 1 -- 172.21.15.1:0/3812450400 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9060012430 con 0x7f9068071550 2024-03-18T18:50:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.481+0000 7f906dd7e700 1 -- 172.21.15.1:0/3812450400 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9068071550 msgr2=0x7f90681661f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.481+0000 7f906dd7e700 1 --2- 172.21.15.1:0/3812450400 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9068071550 0x7f90681661f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9060009d90 tx=0x7f906000b330 comp rx=0 tx=0).stop 2024-03-18T18:50:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.481+0000 7f906dd7e700 1 -- 172.21.15.1:0/3812450400 shutdown_connections 2024-03-18T18:50:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.481+0000 7f906dd7e700 1 --2- 172.21.15.1:0/3812450400 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9068071550 0x7f90681661f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.481+0000 7f906dd7e700 1 -- 172.21.15.1:0/3812450400 >> 172.21.15.1:0/3812450400 conn(0x7f90680b1350 msgr2=0x7f90680b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.481+0000 7f906dd7e700 1 -- 172.21.15.1:0/3812450400 shutdown_connections 2024-03-18T18:50:18.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.481+0000 7f906dd7e700 1 -- 172.21.15.1:0/3812450400 wait complete. 2024-03-18T18:50:18.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.482+0000 7f906dd7e700 1 Processor -- start 2024-03-18T18:50:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.482+0000 7f906dd7e700 1 -- start start 2024-03-18T18:50:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.483+0000 7f906dd7e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9068071550 0x7f906815cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.483+0000 7f906dd7e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90680aecb0 0x7f906815a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.483+0000 7f906dd7e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f906815b370 0x7f906814feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.483+0000 7f906dd7e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9068151af0 con 0x7f9068071550 2024-03-18T18:50:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.483+0000 7f906dd7e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9068159de0 con 0x7f906815b370 2024-03-18T18:50:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.483+0000 7f906dd7e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9068150e70 con 0x7f90680aecb0 2024-03-18T18:50:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.483+0000 7f90677fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9068071550 0x7f906815cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.483+0000 7f90677fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9068071550 0x7f906815cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52010/0 (socket says 172.21.15.1:52010) 2024-03-18T18:50:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.483+0000 7f90677fe700 1 -- 172.21.15.1:0/1835333397 learned_addr learned my addr 172.21.15.1:0/1835333397 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.483+0000 7f9067fff700 1 --2- 172.21.15.1:0/1835333397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f906815b370 0x7f906814feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.483+0000 7f9066ffd700 1 --2- 172.21.15.1:0/1835333397 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90680aecb0 0x7f906815a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.483+0000 7f90677fe700 1 -- 172.21.15.1:0/1835333397 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90680aecb0 msgr2=0x7f906815a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.483+0000 7f90677fe700 1 --2- 172.21.15.1:0/1835333397 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90680aecb0 0x7f906815a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.483+0000 7f90677fe700 1 -- 172.21.15.1:0/1835333397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f906815b370 msgr2=0x7f906814feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.483+0000 7f90677fe700 1 --2- 172.21.15.1:0/1835333397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f906815b370 0x7f906814feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.483+0000 7f90677fe700 1 -- 172.21.15.1:0/1835333397 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9060009a70 con 0x7f9068071550 2024-03-18T18:50:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.483+0000 7f90677fe700 1 --2- 172.21.15.1:0/1835333397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9068071550 0x7f906815cfc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f905c00c4e0 tx=0x7f905c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.483+0000 7f904ffff700 1 -- 172.21.15.1:0/1835333397 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c0052d0 con 0x7f9068071550 2024-03-18T18:50:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.484+0000 7f904ffff700 1 -- 172.21.15.1:0/1835333397 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905c005430 con 0x7f9068071550 2024-03-18T18:50:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.484+0000 7f906dd7e700 1 -- 172.21.15.1:0/1835333397 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9068151150 con 0x7f9068071550 2024-03-18T18:50:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.484+0000 7f904ffff700 1 -- 172.21.15.1:0/1835333397 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c0038c0 con 0x7f9068071550 2024-03-18T18:50:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.484+0000 7f906dd7e700 1 -- 172.21.15.1:0/1835333397 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9068151710 con 0x7f9068071550 2024-03-18T18:50:18.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.484+0000 7f906dd7e700 1 -- 172.21.15.1:0/1835333397 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90680676a0 con 0x7f9068071550 2024-03-18T18:50:18.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.485+0000 7f904ffff700 1 -- 172.21.15.1:0/1835333397 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f905c003a20 con 0x7f9068071550 2024-03-18T18:50:18.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.485+0000 7f904ffff700 1 --2- 172.21.15.1:0/1835333397 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f905406ac90 0x7f905406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:18.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.485+0000 7f904ffff700 1 -- 172.21.15.1:0/1835333397 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f905c088af0 con 0x7f9068071550 2024-03-18T18:50:18.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.486+0000 7f9066ffd700 1 --2- 172.21.15.1:0/1835333397 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f905406ac90 0x7f905406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:18.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.486+0000 7f9066ffd700 1 --2- 172.21.15.1:0/1835333397 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f905406ac90 0x7f905406d150 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9058007fd0 tx=0x7f9058007c20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:18.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.490+0000 7f904ffff700 1 -- 172.21.15.1:0/1835333397 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f905c0539d0 con 0x7f9068071550 2024-03-18T18:50:18.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.694+0000 7f906dd7e700 1 -- 172.21.15.1:0/1835333397 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9068073db0 con 0x7f9068071550 2024-03-18T18:50:18.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.694+0000 7f904ffff700 1 -- 172.21.15.1:0/1835333397 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f905c0170a0 con 0x7f9068071550 2024-03-18T18:50:18.691 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:18.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.696+0000 7f906dd7e700 1 -- 172.21.15.1:0/1835333397 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f905406ac90 msgr2=0x7f905406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:18.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.696+0000 7f906dd7e700 1 --2- 172.21.15.1:0/1835333397 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f905406ac90 0x7f905406d150 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9058007fd0 tx=0x7f9058007c20 comp rx=0 tx=0).stop 2024-03-18T18:50:18.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.696+0000 7f906dd7e700 1 -- 172.21.15.1:0/1835333397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9068071550 msgr2=0x7f906815cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:18.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.696+0000 7f906dd7e700 1 --2- 172.21.15.1:0/1835333397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9068071550 0x7f906815cfc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f905c00c4e0 tx=0x7f905c00c8a0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.697+0000 7f906dd7e700 1 -- 172.21.15.1:0/1835333397 shutdown_connections 2024-03-18T18:50:18.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.697+0000 7f906dd7e700 1 --2- 172.21.15.1:0/1835333397 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f905406ac90 0x7f905406d150 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.697+0000 7f906dd7e700 1 --2- 172.21.15.1:0/1835333397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f906815b370 0x7f906814feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.697+0000 7f906dd7e700 1 --2- 172.21.15.1:0/1835333397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9068071550 0x7f906815cfc0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.697+0000 7f906dd7e700 1 --2- 172.21.15.1:0/1835333397 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90680aecb0 0x7f906815a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.697+0000 7f906dd7e700 1 -- 172.21.15.1:0/1835333397 >> 172.21.15.1:0/1835333397 conn(0x7f90680b1350 msgr2=0x7f90680af2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:18.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.697+0000 7f906dd7e700 1 -- 172.21.15.1:0/1835333397 shutdown_connections 2024-03-18T18:50:18.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.697+0000 7f906dd7e700 1 -- 172.21.15.1:0/1835333397 wait complete. 2024-03-18T18:50:18.705 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T18:50:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.834+0000 7f2e57471700 1 Processor -- start 2024-03-18T18:50:18.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.834+0000 7f2e57471700 1 -- start start 2024-03-18T18:50:18.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.835+0000 7f2e57471700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e50140b20 0x7f2e500cfe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:18.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.835+0000 7f2e57471700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e500cdf80 0x7f2e500cba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:18.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.835+0000 7f2e57471700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e500d03d0 con 0x7f2e500cfab0 2024-03-18T18:50:18.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.835+0000 7f2e57471700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e5015fbe0 con 0x7f2e50140b20 2024-03-18T18:50:18.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.835+0000 7f2e57471700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5015f060 con 0x7f2e500cdf80 2024-03-18T18:50:18.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.835+0000 7f2e54a0c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e50140b20 0x7f2e500cfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:18.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.835+0000 7f2e54a0c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e50140b20 0x7f2e500cfe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52014/0 (socket says 172.21.15.1:52014) 2024-03-18T18:50:18.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.835+0000 7f2e54a0c700 1 -- 172.21.15.1:0/293028191 learned_addr learned my addr 172.21.15.1:0/293028191 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:18.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.835+0000 7f2e55a0e700 1 --2- 172.21.15.1:0/293028191 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e500cdf80 0x7f2e500cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:18.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.835+0000 7f2e54a0c700 1 -- 172.21.15.1:0/293028191 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e500cdf80 msgr2=0x7f2e500cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:18.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.835+0000 7f2e54a0c700 1 --2- 172.21.15.1:0/293028191 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e500cdf80 0x7f2e500cba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.835+0000 7f2e54a0c700 1 -- 172.21.15.1:0/293028191 >> v1:172.21.15.165:6789/0 conn(0x7f2e500cfab0 legacy=0x7f2e5015c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:18.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.835+0000 7f2e54a0c700 1 -- 172.21.15.1:0/293028191 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5015f340 con 0x7f2e50140b20 2024-03-18T18:50:18.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.835+0000 7f2e55a0e700 1 --2- 172.21.15.1:0/293028191 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e500cdf80 0x7f2e500cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:18.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.835+0000 7f2e54a0c700 1 --2- 172.21.15.1:0/293028191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e50140b20 0x7f2e500cfe90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2e400098f0 tx=0x7f2e40009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a389e85257c1d2a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:18.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.835+0000 7f2e47fff700 1 -- 172.21.15.1:0/293028191 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e40004230 con 0x7f2e50140b20 2024-03-18T18:50:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.835+0000 7f2e47fff700 1 -- 172.21.15.1:0/293028191 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e40004390 con 0x7f2e50140b20 2024-03-18T18:50:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.835+0000 7f2e47fff700 1 -- 172.21.15.1:0/293028191 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e400036a0 con 0x7f2e50140b20 2024-03-18T18:50:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.836+0000 7f2e57471700 1 -- 172.21.15.1:0/293028191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e50140b20 msgr2=0x7f2e500cfe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.836+0000 7f2e57471700 1 --2- 172.21.15.1:0/293028191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e50140b20 0x7f2e500cfe90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2e400098f0 tx=0x7f2e40009c00 comp rx=0 tx=0).stop 2024-03-18T18:50:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.836+0000 7f2e57471700 1 -- 172.21.15.1:0/293028191 shutdown_connections 2024-03-18T18:50:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.836+0000 7f2e57471700 1 --2- 172.21.15.1:0/293028191 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e500cdf80 0x7f2e500cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.836+0000 7f2e57471700 1 --2- 172.21.15.1:0/293028191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e50140b20 0x7f2e500cfe90 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.836+0000 7f2e57471700 1 -- 172.21.15.1:0/293028191 >> 172.21.15.1:0/293028191 conn(0x7f2e5006ab70 msgr2=0x7f2e5006af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.836+0000 7f2e57471700 1 -- 172.21.15.1:0/293028191 shutdown_connections 2024-03-18T18:50:18.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.836+0000 7f2e57471700 1 -- 172.21.15.1:0/293028191 wait complete. 2024-03-18T18:50:18.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.837+0000 7f2e57471700 1 Processor -- start 2024-03-18T18:50:18.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.837+0000 7f2e57471700 1 -- start start 2024-03-18T18:50:18.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.838+0000 7f2e57471700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e500cdf80 0x7f2e50158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:18.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.838+0000 7f2e57471700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e500cfab0 0x7f2e50157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:18.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.838+0000 7f2e57471700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e5013faa0 0x7f2e501e5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:18.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.838+0000 7f2e57471700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5013cba0 con 0x7f2e500cdf80 2024-03-18T18:50:18.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.838+0000 7f2e57471700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e501581b0 con 0x7f2e500cfab0 2024-03-18T18:50:18.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.838+0000 7f2e57471700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5013fff0 con 0x7f2e5013faa0 2024-03-18T18:50:18.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.838+0000 7f2e5520d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e500cdf80 0x7f2e50158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:18.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.838+0000 7f2e5520d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e500cdf80 0x7f2e50158370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52026/0 (socket says 172.21.15.1:52026) 2024-03-18T18:50:18.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.838+0000 7f2e5520d700 1 -- 172.21.15.1:0/346973429 learned_addr learned my addr 172.21.15.1:0/346973429 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:18.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.838+0000 7f2e54a0c700 1 --2- 172.21.15.1:0/346973429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e500cfab0 0x7f2e50157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:18.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.838+0000 7f2e55a0e700 1 --2- 172.21.15.1:0/346973429 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e5013faa0 0x7f2e501e5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:18.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.838+0000 7f2e5520d700 1 -- 172.21.15.1:0/346973429 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e5013faa0 msgr2=0x7f2e501e5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:18.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.838+0000 7f2e5520d700 1 --2- 172.21.15.1:0/346973429 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e5013faa0 0x7f2e501e5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.838+0000 7f2e5520d700 1 -- 172.21.15.1:0/346973429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e500cfab0 msgr2=0x7f2e50157c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:18.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.838+0000 7f2e5520d700 1 --2- 172.21.15.1:0/346973429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e500cfab0 0x7f2e50157c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:18.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.838+0000 7f2e5520d700 1 -- 172.21.15.1:0/346973429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e40009550 con 0x7f2e500cdf80 2024-03-18T18:50:18.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.838+0000 7f2e5520d700 1 --2- 172.21.15.1:0/346973429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e500cdf80 0x7f2e50158370 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c00c4d0 tx=0x7f2e4c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:18.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.839+0000 7f2e45ffb700 1 -- 172.21.15.1:0/346973429 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c011790 con 0x7f2e500cdf80 2024-03-18T18:50:18.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.839+0000 7f2e45ffb700 1 -- 172.21.15.1:0/346973429 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4c0118f0 con 0x7f2e500cdf80 2024-03-18T18:50:18.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.839+0000 7f2e45ffb700 1 -- 172.21.15.1:0/346973429 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c011ca0 con 0x7f2e500cdf80 2024-03-18T18:50:18.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.839+0000 7f2e57471700 1 -- 172.21.15.1:0/346973429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e501402d0 con 0x7f2e500cdf80 2024-03-18T18:50:18.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.839+0000 7f2e57471700 1 -- 172.21.15.1:0/346973429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e50140890 con 0x7f2e500cdf80 2024-03-18T18:50:18.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.840+0000 7f2e57471700 1 -- 172.21.15.1:0/346973429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e500676a0 con 0x7f2e500cdf80 2024-03-18T18:50:18.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.840+0000 7f2e45ffb700 1 -- 172.21.15.1:0/346973429 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2e4c01c460 con 0x7f2e500cdf80 2024-03-18T18:50:18.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.840+0000 7f2e45ffb700 1 --2- 172.21.15.1:0/346973429 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2e3406ac90 0x7f2e3406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:18.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.841+0000 7f2e45ffb700 1 -- 172.21.15.1:0/346973429 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2e4c0892c0 con 0x7f2e500cdf80 2024-03-18T18:50:18.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.841+0000 7f2e54a0c700 1 --2- 172.21.15.1:0/346973429 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2e3406ac90 0x7f2e3406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:18.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.841+0000 7f2e54a0c700 1 --2- 172.21.15.1:0/346973429 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2e3406ac90 0x7f2e3406d150 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2e40009130 tx=0x7f2e40008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:18.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:18.845+0000 7f2e45ffb700 1 -- 172.21.15.1:0/346973429 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2e4c0541b0 con 0x7f2e500cdf80 2024-03-18T18:50:19.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.097+0000 7f2e57471700 1 -- 172.21.15.1:0/346973429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e5006cf10 con 0x7f2e500cdf80 2024-03-18T18:50:19.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.097+0000 7f2e45ffb700 1 -- 172.21.15.1:0/346973429 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4727 (secure 0 0 0) 0x7f2e4c0577d0 con 0x7f2e500cdf80 2024-03-18T18:50:19.094 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:19.094 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/883056851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":883056851},{"type":"v1","addr":"172.21.15.165:6834","nonce":883056851}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1388214718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1388214718},{"type":"v1","addr":"172.21.15.165:6835","nonce":1388214718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3020786191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3020786191},{"type":"v1","addr":"172.21.15.1:6836","nonce":3020786191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T18:49:44.809495+0000","modified":"2024-03-18T18:50:07.888953+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2118908012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2118908012},{"type":"v1","addr":"172.21.15.1:6837","nonce":2118908012}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-18T18:50:19.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.099+0000 7f2e57471700 1 -- 172.21.15.1:0/346973429 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2e3406ac90 msgr2=0x7f2e3406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:19.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.099+0000 7f2e57471700 1 --2- 172.21.15.1:0/346973429 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2e3406ac90 0x7f2e3406d150 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2e40009130 tx=0x7f2e40008030 comp rx=0 tx=0).stop 2024-03-18T18:50:19.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.099+0000 7f2e57471700 1 -- 172.21.15.1:0/346973429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e500cdf80 msgr2=0x7f2e50158370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:19.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.099+0000 7f2e57471700 1 --2- 172.21.15.1:0/346973429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e500cdf80 0x7f2e50158370 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c00c4d0 tx=0x7f2e4c00c7e0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.100+0000 7f2e57471700 1 -- 172.21.15.1:0/346973429 shutdown_connections 2024-03-18T18:50:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.100+0000 7f2e57471700 1 --2- 172.21.15.1:0/346973429 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2e3406ac90 0x7f2e3406d150 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.100+0000 7f2e57471700 1 --2- 172.21.15.1:0/346973429 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e5013faa0 0x7f2e501e5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.100+0000 7f2e57471700 1 --2- 172.21.15.1:0/346973429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e500cdf80 0x7f2e50158370 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.100+0000 7f2e57471700 1 --2- 172.21.15.1:0/346973429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e500cfab0 0x7f2e50157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.100+0000 7f2e57471700 1 -- 172.21.15.1:0/346973429 >> 172.21.15.1:0/346973429 conn(0x7f2e5006ab70 msgr2=0x7f2e500af2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.100+0000 7f2e57471700 1 -- 172.21.15.1:0/346973429 shutdown_connections 2024-03-18T18:50:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.100+0000 7f2e57471700 1 -- 172.21.15.1:0/346973429 wait complete. 2024-03-18T18:50:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 9 2024-03-18T18:50:19.108 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:19.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.236+0000 7fa653f02700 1 Processor -- start 2024-03-18T18:50:19.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.236+0000 7fa653f02700 1 -- start start 2024-03-18T18:50:19.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.237+0000 7fa653f02700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa64c0ae600 0x7fa64c0ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:19.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.237+0000 7fa653f02700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa64c15a2e0 0x7fa64c0ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:19.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.237+0000 7fa653f02700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c06e270 con 0x7fa64c0cc170 2024-03-18T18:50:19.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.237+0000 7fa653f02700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa64c0ae050 con 0x7fa64c15a2e0 2024-03-18T18:50:19.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.237+0000 7fa653f02700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa64c06f690 con 0x7fa64c0ae600 2024-03-18T18:50:19.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.237+0000 7fa65149d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa64c15a2e0 0x7fa64c0ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:19.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.237+0000 7fa65149d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa64c15a2e0 0x7fa64c0ae9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52034/0 (socket says 172.21.15.1:52034) 2024-03-18T18:50:19.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.237+0000 7fa65149d700 1 -- 172.21.15.1:0/1696163339 learned_addr learned my addr 172.21.15.1:0/1696163339 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:19.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.238+0000 7fa651c9e700 1 --2- 172.21.15.1:0/1696163339 >> v2:172.21.15.165:3300/0 conn(0x7fa64c0ae600 0x7fa64c0ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:19.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.238+0000 7fa65149d700 1 -- 172.21.15.1:0/1696163339 >> v2:172.21.15.165:3300/0 conn(0x7fa64c0ae600 msgr2=0x7fa64c0ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:19.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.238+0000 7fa65149d700 1 --2- 172.21.15.1:0/1696163339 >> v2:172.21.15.165:3300/0 conn(0x7fa64c0ae600 0x7fa64c0ad790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.238+0000 7fa65149d700 1 -- 172.21.15.1:0/1696163339 >> v1:172.21.15.165:6789/0 conn(0x7fa64c0cc170 legacy=0x7fa64c06ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:19.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.238+0000 7fa65149d700 1 -- 172.21.15.1:0/1696163339 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c0aef20 con 0x7fa64c15a2e0 2024-03-18T18:50:19.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.238+0000 7fa65149d700 1 --2- 172.21.15.1:0/1696163339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c15a2e0 0x7fa64c0ae9e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa64c04ebb0 tx=0x7fa64800be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a8ec2371abbde4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:19.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.238+0000 7fa650c9c700 1 -- 172.21.15.1:0/1696163339 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64800dcf0 con 0x7fa64c15a2e0 2024-03-18T18:50:19.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.238+0000 7fa650c9c700 1 -- 172.21.15.1:0/1696163339 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64800de50 con 0x7fa64c15a2e0 2024-03-18T18:50:19.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.238+0000 7fa650c9c700 1 -- 172.21.15.1:0/1696163339 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa648004250 con 0x7fa64c15a2e0 2024-03-18T18:50:19.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.238+0000 7fa653f02700 1 -- 172.21.15.1:0/1696163339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c15a2e0 msgr2=0x7fa64c0ae9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:19.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.238+0000 7fa653f02700 1 --2- 172.21.15.1:0/1696163339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c15a2e0 0x7fa64c0ae9e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa64c04ebb0 tx=0x7fa64800be70 comp rx=0 tx=0).stop 2024-03-18T18:50:19.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.239+0000 7fa653f02700 1 -- 172.21.15.1:0/1696163339 shutdown_connections 2024-03-18T18:50:19.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.239+0000 7fa653f02700 1 --2- 172.21.15.1:0/1696163339 >> v2:172.21.15.165:3300/0 conn(0x7fa64c0ae600 0x7fa64c0ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.239+0000 7fa653f02700 1 --2- 172.21.15.1:0/1696163339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c15a2e0 0x7fa64c0ae9e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.239+0000 7fa653f02700 1 -- 172.21.15.1:0/1696163339 >> 172.21.15.1:0/1696163339 conn(0x7fa64c06b860 msgr2=0x7fa64c06bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:19.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.239+0000 7fa653f02700 1 -- 172.21.15.1:0/1696163339 shutdown_connections 2024-03-18T18:50:19.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.239+0000 7fa653f02700 1 -- 172.21.15.1:0/1696163339 wait complete. 2024-03-18T18:50:19.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.240+0000 7fa653f02700 1 Processor -- start 2024-03-18T18:50:19.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.240+0000 7fa653f02700 1 -- start start 2024-03-18T18:50:19.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.241+0000 7fa653f02700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64c0ae600 0x7fa64c0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:19.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.241+0000 7fa653f02700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c0cc170 0x7fa64c1f6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:19.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.241+0000 7fa653f02700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa64c15a2e0 0x7fa64c1f6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:19.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.241+0000 7fa653f02700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c1572e0 con 0x7fa64c0cc170 2024-03-18T18:50:19.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.241+0000 7fa653f02700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c0cd090 con 0x7fa64c0ae600 2024-03-18T18:50:19.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.241+0000 7fa653f02700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c0cd370 con 0x7fa64c15a2e0 2024-03-18T18:50:19.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.241+0000 7fa65149d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c0cc170 0x7fa64c1f6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:19.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.241+0000 7fa65149d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c0cc170 0x7fa64c1f6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52046/0 (socket says 172.21.15.1:52046) 2024-03-18T18:50:19.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.241+0000 7fa65149d700 1 -- 172.21.15.1:0/3756566784 learned_addr learned my addr 172.21.15.1:0/3756566784 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:19.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.241+0000 7fa651c9e700 1 --2- 172.21.15.1:0/3756566784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64c0ae600 0x7fa64c0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:19.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.241+0000 7fa65249f700 1 --2- 172.21.15.1:0/3756566784 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa64c15a2e0 0x7fa64c1f6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:19.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.242+0000 7fa65149d700 1 -- 172.21.15.1:0/3756566784 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa64c15a2e0 msgr2=0x7fa64c1f6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:19.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.242+0000 7fa65149d700 1 --2- 172.21.15.1:0/3756566784 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa64c15a2e0 0x7fa64c1f6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.242+0000 7fa65149d700 1 -- 172.21.15.1:0/3756566784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64c0ae600 msgr2=0x7fa64c0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:19.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.242+0000 7fa65149d700 1 --2- 172.21.15.1:0/3756566784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64c0ae600 0x7fa64c0ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.242+0000 7fa65149d700 1 -- 172.21.15.1:0/3756566784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64800b7f0 con 0x7fa64c0cc170 2024-03-18T18:50:19.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.242+0000 7fa65249f700 1 --2- 172.21.15.1:0/3756566784 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa64c15a2e0 0x7fa64c1f6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:19.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.242+0000 7fa65149d700 1 --2- 172.21.15.1:0/3756566784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c0cc170 0x7fa64c1f6140 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa64800beb0 tx=0x7fa648004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:19.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.242+0000 7fa6427fc700 1 -- 172.21.15.1:0/3756566784 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa648004230 con 0x7fa64c0cc170 2024-03-18T18:50:19.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.242+0000 7fa6427fc700 1 -- 172.21.15.1:0/3756566784 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa648010470 con 0x7fa64c0cc170 2024-03-18T18:50:19.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.242+0000 7fa653f02700 1 -- 172.21.15.1:0/3756566784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64c0cd650 con 0x7fa64c0cc170 2024-03-18T18:50:19.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.242+0000 7fa6427fc700 1 -- 172.21.15.1:0/3756566784 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa648010820 con 0x7fa64c0cc170 2024-03-18T18:50:19.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.242+0000 7fa653f02700 1 -- 172.21.15.1:0/3756566784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa64c0cdc10 con 0x7fa64c0cc170 2024-03-18T18:50:19.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.243+0000 7fa653f02700 1 -- 172.21.15.1:0/3756566784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64c0676a0 con 0x7fa64c0cc170 2024-03-18T18:50:19.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.244+0000 7fa6427fc700 1 -- 172.21.15.1:0/3756566784 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa6480193f0 con 0x7fa64c0cc170 2024-03-18T18:50:19.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.244+0000 7fa6427fc700 1 --2- 172.21.15.1:0/3756566784 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa63006ac90 0x7fa63006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:19.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.244+0000 7fa651c9e700 1 --2- 172.21.15.1:0/3756566784 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa63006ac90 0x7fa63006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:19.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.244+0000 7fa6427fc700 1 -- 172.21.15.1:0/3756566784 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa64808b480 con 0x7fa64c0cc170 2024-03-18T18:50:19.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.245+0000 7fa651c9e700 1 --2- 172.21.15.1:0/3756566784 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa63006ac90 0x7fa63006d150 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa63c006b20 tx=0x7fa63c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:19.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.249+0000 7fa6427fc700 1 -- 172.21.15.1:0/3756566784 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa648056380 con 0x7fa64c0cc170 2024-03-18T18:50:19.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.452+0000 7fa653f02700 1 -- 172.21.15.1:0/3756566784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa64c06c420 con 0x7fa64c0cc170 2024-03-18T18:50:19.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.453+0000 7fa6427fc700 1 -- 172.21.15.1:0/3756566784 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17830 (secure 0 0 0) 0x7fa6480599a0 con 0x7fa64c0cc170 2024-03-18T18:50:19.449 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:19.450 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:49.276961+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:19.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.455+0000 7fa653f02700 1 -- 172.21.15.1:0/3756566784 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa63006ac90 msgr2=0x7fa63006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:19.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.455+0000 7fa653f02700 1 --2- 172.21.15.1:0/3756566784 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa63006ac90 0x7fa63006d150 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa63c006b20 tx=0x7fa63c008040 comp rx=0 tx=0).stop 2024-03-18T18:50:19.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.455+0000 7fa653f02700 1 -- 172.21.15.1:0/3756566784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c0cc170 msgr2=0x7fa64c1f6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:19.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.455+0000 7fa653f02700 1 --2- 172.21.15.1:0/3756566784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c0cc170 0x7fa64c1f6140 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa64800beb0 tx=0x7fa648004750 comp rx=0 tx=0).stop 2024-03-18T18:50:19.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.455+0000 7fa653f02700 1 -- 172.21.15.1:0/3756566784 shutdown_connections 2024-03-18T18:50:19.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.455+0000 7fa653f02700 1 --2- 172.21.15.1:0/3756566784 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa63006ac90 0x7fa63006d150 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.455+0000 7fa653f02700 1 --2- 172.21.15.1:0/3756566784 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa64c15a2e0 0x7fa64c1f6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.455+0000 7fa653f02700 1 --2- 172.21.15.1:0/3756566784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64c0ae600 0x7fa64c0ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.455+0000 7fa653f02700 1 --2- 172.21.15.1:0/3756566784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c0cc170 0x7fa64c1f6140 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.455+0000 7fa653f02700 1 -- 172.21.15.1:0/3756566784 >> 172.21.15.1:0/3756566784 conn(0x7fa64c06b860 msgr2=0x7fa64c0b3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:19.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.455+0000 7fa653f02700 1 -- 172.21.15.1:0/3756566784 shutdown_connections 2024-03-18T18:50:19.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.455+0000 7fa653f02700 1 -- 172.21.15.1:0/3756566784 wait complete. 2024-03-18T18:50:19.464 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T18:50:19.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.593+0000 7f096b522700 1 Processor -- start 2024-03-18T18:50:19.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.593+0000 7f096b522700 1 -- start start 2024-03-18T18:50:19.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.594+0000 7f096b522700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0964099ea0 0x7f09640cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:19.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.594+0000 7f096b522700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09640d0c80 con 0x7f0964099a40 2024-03-18T18:50:19.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.594+0000 7f096b522700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09640d09a0 con 0x7f096415d680 2024-03-18T18:50:19.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.594+0000 7f096b522700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09640cb830 con 0x7f0964099ea0 2024-03-18T18:50:19.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.594+0000 7f0968abd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0964099a40 0x7f09640cc310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55904/0 (socket says 172.21.15.1:55904) 2024-03-18T18:50:19.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.594+0000 7f0968abd700 1 -- 172.21.15.1:0/3679846300 learned_addr learned my addr 172.21.15.1:0/3679846300 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:19.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.594+0000 7f0969abf700 1 --2- 172.21.15.1:0/3679846300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0964099ea0 0x7f09640cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:19.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.595+0000 7f095bfff700 1 -- 172.21.15.1:0/3679846300 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834462888 0 0) 0x7f09640d0c80 con 0x7f0964099a40 2024-03-18T18:50:19.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.595+0000 7f095bfff700 1 -- 172.21.15.1:0/3679846300 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09500035b0 con 0x7f0964099a40 2024-03-18T18:50:19.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.595+0000 7f095bfff700 1 -- 172.21.15.1:0/3679846300 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933532059 0 0) 0x7f09640d09a0 con 0x7f096415d680 2024-03-18T18:50:19.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.595+0000 7f095bfff700 1 -- 172.21.15.1:0/3679846300 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09640d0c80 con 0x7f096415d680 2024-03-18T18:50:19.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.595+0000 7f0969abf700 1 -- 172.21.15.1:0/3679846300 >> v1:172.21.15.165:6789/0 conn(0x7f096415d680 legacy=0x7f09640cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:19.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.595+0000 7f095bfff700 1 -- 172.21.15.1:0/3679846300 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3746607809 0 0) 0x7f09500035b0 con 0x7f0964099a40 2024-03-18T18:50:19.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.595+0000 7f0969abf700 1 -- 172.21.15.1:0/3679846300 >> v1:172.21.15.1:6789/0 conn(0x7f0964099a40 legacy=0x7f09640cc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:19.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.595+0000 7f0969abf700 1 -- 172.21.15.1:0/3679846300 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09640cbb10 con 0x7f0964099ea0 2024-03-18T18:50:19.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.595+0000 7f0969abf700 1 --2- 172.21.15.1:0/3679846300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0964099ea0 0x7f09640cca20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f095c009dc0 tx=0x7f095c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a43ea9270aae8ff0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:19.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.595+0000 7f095bfff700 1 -- 172.21.15.1:0/3679846300 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095c00e070 con 0x7f0964099ea0 2024-03-18T18:50:19.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.595+0000 7f095bfff700 1 -- 172.21.15.1:0/3679846300 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095c003a20 con 0x7f0964099ea0 2024-03-18T18:50:19.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.595+0000 7f095bfff700 1 -- 172.21.15.1:0/3679846300 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0960001670 con 0x7f0964099ea0 2024-03-18T18:50:19.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.596+0000 7f096b522700 1 -- 172.21.15.1:0/3679846300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0964099ea0 msgr2=0x7f09640cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:19.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.596+0000 7f096b522700 1 --2- 172.21.15.1:0/3679846300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0964099ea0 0x7f09640cca20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f095c009dc0 tx=0x7f095c00b330 comp rx=0 tx=0).stop 2024-03-18T18:50:19.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.596+0000 7f096b522700 1 -- 172.21.15.1:0/3679846300 shutdown_connections 2024-03-18T18:50:19.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.596+0000 7f096b522700 1 --2- 172.21.15.1:0/3679846300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0964099ea0 0x7f09640cca20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.596+0000 7f096b522700 1 -- 172.21.15.1:0/3679846300 >> 172.21.15.1:0/3679846300 conn(0x7f09640b2ba0 msgr2=0x7f09640b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:19.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.596+0000 7f096b522700 1 -- 172.21.15.1:0/3679846300 shutdown_connections 2024-03-18T18:50:19.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.596+0000 7f096b522700 1 -- 172.21.15.1:0/3679846300 wait complete. 2024-03-18T18:50:19.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.597+0000 7f096b522700 1 Processor -- start 2024-03-18T18:50:19.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.597+0000 7f096b522700 1 -- start start 2024-03-18T18:50:19.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.598+0000 7f096b522700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0964099a40 0x7f09640b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:19.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.598+0000 7f096b522700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0964099ea0 0x7f09640b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:19.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.598+0000 7f096b522700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f096415d680 0x7f096415c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:19.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.598+0000 7f096b522700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0964070140 con 0x7f0964099a40 2024-03-18T18:50:19.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.598+0000 7f096b522700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096406f5c0 con 0x7f0964099ea0 2024-03-18T18:50:19.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.598+0000 7f096b522700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f096406f8a0 con 0x7f096415d680 2024-03-18T18:50:19.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.599+0000 7f09692be700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0964099a40 0x7f09640b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:19.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.599+0000 7f09692be700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0964099a40 0x7f09640b5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52062/0 (socket says 172.21.15.1:52062) 2024-03-18T18:50:19.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.599+0000 7f09692be700 1 -- 172.21.15.1:0/3925691071 learned_addr learned my addr 172.21.15.1:0/3925691071 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:19.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.599+0000 7f0969abf700 1 --2- 172.21.15.1:0/3925691071 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f096415d680 0x7f096415c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:19.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.599+0000 7f0968abd700 1 --2- 172.21.15.1:0/3925691071 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0964099ea0 0x7f09640b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:19.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.599+0000 7f09692be700 1 -- 172.21.15.1:0/3925691071 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f096415d680 msgr2=0x7f096415c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:19.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.599+0000 7f09692be700 1 --2- 172.21.15.1:0/3925691071 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f096415d680 0x7f096415c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.599+0000 7f09692be700 1 -- 172.21.15.1:0/3925691071 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0964099ea0 msgr2=0x7f09640b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:19.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.599+0000 7f09692be700 1 --2- 172.21.15.1:0/3925691071 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0964099ea0 0x7f09640b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.599+0000 7f09692be700 1 -- 172.21.15.1:0/3925691071 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095c009a70 con 0x7f0964099a40 2024-03-18T18:50:19.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.599+0000 7f0969abf700 1 --2- 172.21.15.1:0/3925691071 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f096415d680 0x7f096415c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:19.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.599+0000 7f09692be700 1 --2- 172.21.15.1:0/3925691071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0964099a40 0x7f09640b5a60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f095400c500 tx=0x7f095400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:19.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.599+0000 7f0959ffb700 1 -- 172.21.15.1:0/3925691071 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09540052d0 con 0x7f0964099a40 2024-03-18T18:50:19.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.599+0000 7f0959ffb700 1 -- 172.21.15.1:0/3925691071 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0954005450 con 0x7f0964099a40 2024-03-18T18:50:19.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.599+0000 7f096b522700 1 -- 172.21.15.1:0/3925691071 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f096406fb80 con 0x7f0964099a40 2024-03-18T18:50:19.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.599+0000 7f0959ffb700 1 -- 172.21.15.1:0/3925691071 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09540038c0 con 0x7f0964099a40 2024-03-18T18:50:19.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.599+0000 7f096b522700 1 -- 172.21.15.1:0/3925691071 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09640b5fa0 con 0x7f0964099a40 2024-03-18T18:50:19.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.600+0000 7f096b522700 1 -- 172.21.15.1:0/3925691071 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09640b6160 con 0x7f0964099a40 2024-03-18T18:50:19.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.601+0000 7f0959ffb700 1 -- 172.21.15.1:0/3925691071 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0954003a20 con 0x7f0964099a40 2024-03-18T18:50:19.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.601+0000 7f0959ffb700 1 --2- 172.21.15.1:0/3925691071 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f094806ac90 0x7f094806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:19.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.601+0000 7f0959ffb700 1 -- 172.21.15.1:0/3925691071 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0954088af0 con 0x7f0964099a40 2024-03-18T18:50:19.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.601+0000 7f0968abd700 1 --2- 172.21.15.1:0/3925691071 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f094806ac90 0x7f094806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:19.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.602+0000 7f0968abd700 1 --2- 172.21.15.1:0/3925691071 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f094806ac90 0x7f094806d150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f09640c8f20 tx=0x7f0960001c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:19.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.606+0000 7f0959ffb700 1 -- 172.21.15.1:0/3925691071 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f09540539e0 con 0x7f0964099a40 2024-03-18T18:50:19.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.852+0000 7f096b522700 1 -- 172.21.15.1:0/3925691071 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09640676a0 con 0x7f0964099a40 2024-03-18T18:50:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.853+0000 7f0959ffb700 1 -- 172.21.15.1:0/3925691071 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4727 (secure 0 0 0) 0x7f0954017090 con 0x7f0964099a40 2024-03-18T18:50:19.849 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:19.849 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/883056851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":883056851},{"type":"v1","addr":"172.21.15.165:6834","nonce":883056851}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1388214718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1388214718},{"type":"v1","addr":"172.21.15.165:6835","nonce":1388214718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3020786191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3020786191},{"type":"v1","addr":"172.21.15.1:6836","nonce":3020786191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T18:49:44.809495+0000","modified":"2024-03-18T18:50:07.888953+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2118908012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2118908012},{"type":"v1","addr":"172.21.15.1:6837","nonce":2118908012}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-18T18:50:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.855+0000 7f096b522700 1 -- 172.21.15.1:0/3925691071 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f094806ac90 msgr2=0x7f094806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.855+0000 7f096b522700 1 --2- 172.21.15.1:0/3925691071 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f094806ac90 0x7f094806d150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f09640c8f20 tx=0x7f0960001c00 comp rx=0 tx=0).stop 2024-03-18T18:50:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.855+0000 7f096b522700 1 -- 172.21.15.1:0/3925691071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0964099a40 msgr2=0x7f09640b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.855+0000 7f096b522700 1 --2- 172.21.15.1:0/3925691071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0964099a40 0x7f09640b5a60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f095400c500 tx=0x7f095400c8c0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.855+0000 7f096b522700 1 -- 172.21.15.1:0/3925691071 shutdown_connections 2024-03-18T18:50:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.855+0000 7f096b522700 1 --2- 172.21.15.1:0/3925691071 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f094806ac90 0x7f094806d150 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.855+0000 7f096b522700 1 --2- 172.21.15.1:0/3925691071 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f096415d680 0x7f096415c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.855+0000 7f096b522700 1 --2- 172.21.15.1:0/3925691071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0964099a40 0x7f09640b5a60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.855+0000 7f096b522700 1 --2- 172.21.15.1:0/3925691071 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0964099ea0 0x7f09640b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.855+0000 7f096b522700 1 -- 172.21.15.1:0/3925691071 >> 172.21.15.1:0/3925691071 conn(0x7f09640b2ba0 msgr2=0x7f09640af940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.855+0000 7f096b522700 1 -- 172.21.15.1:0/3925691071 shutdown_connections 2024-03-18T18:50:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:19.855+0000 7f096b522700 1 -- 172.21.15.1:0/3925691071 wait complete. 2024-03-18T18:50:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 9 2024-03-18T18:50:20.267 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T18:49:44.809495+0000', 'modified': '2024-03-18T18:50:07.888953+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4273}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4273': {'gid': 4273, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6837/2118908012', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6835', 'nonce': 2118908012}, {'type': 'v1', 'addr': '172.21.15.1:6837', 'nonce': 2118908012}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T18:50:20.267 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T18:50:20.267 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-18T18:50:20.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.408+0000 7f1aa7785700 1 Processor -- start 2024-03-18T18:50:20.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.409+0000 7f1aa7785700 1 -- start start 2024-03-18T18:50:20.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.409+0000 7f1aa7785700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1aa015d5a0 0x7f1aa00ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.409+0000 7f1aa7785700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1aa00ce550 0x7f1aa006a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.409+0000 7f1aa7785700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1aa00cfa90 0x7f1aa006ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.409+0000 7f1aa7785700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa00cf600 con 0x7f1aa00ce550 2024-03-18T18:50:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.409+0000 7f1aa7785700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa00ce9f0 con 0x7f1aa00cfa90 2024-03-18T18:50:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.409+0000 7f1aa7785700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa00cecd0 con 0x7f1aa015d5a0 2024-03-18T18:50:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.409+0000 7f1aa4d20700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1aa00ce550 0x7f1aa006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.409+0000 7f1aa4d20700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1aa00ce550 0x7f1aa006a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52076/0 (socket says 172.21.15.1:52076) 2024-03-18T18:50:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.410+0000 7f1aa4d20700 1 -- 172.21.15.1:0/2610245440 learned_addr learned my addr 172.21.15.1:0/2610245440 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.410+0000 7f1aa5d22700 1 --2- 172.21.15.1:0/2610245440 >> v2:172.21.15.165:3300/0 conn(0x7f1aa00cfa90 0x7f1aa006ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.410+0000 7f1aa5521700 1 --2- 172.21.15.1:0/2610245440 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1aa015d5a0 0x7f1aa00ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.410+0000 7f1aa4d20700 1 -- 172.21.15.1:0/2610245440 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1aa015d5a0 msgr2=0x7f1aa00ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.410+0000 7f1aa4d20700 1 --2- 172.21.15.1:0/2610245440 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1aa015d5a0 0x7f1aa00ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.410+0000 7f1aa4d20700 1 -- 172.21.15.1:0/2610245440 >> v2:172.21.15.165:3300/0 conn(0x7f1aa00cfa90 msgr2=0x7f1aa006ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.410+0000 7f1aa4d20700 1 --2- 172.21.15.1:0/2610245440 >> v2:172.21.15.165:3300/0 conn(0x7f1aa00cfa90 0x7f1aa006ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.410+0000 7f1aa4d20700 1 -- 172.21.15.1:0/2610245440 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa00cefb0 con 0x7f1aa00ce550 2024-03-18T18:50:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.410+0000 7f1aa5d22700 1 --2- 172.21.15.1:0/2610245440 >> v2:172.21.15.165:3300/0 conn(0x7f1aa00cfa90 0x7f1aa006ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.410+0000 7f1aa4d20700 1 --2- 172.21.15.1:0/2610245440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aa00ce550 0x7f1aa006a750 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00cdec0 tx=0x7f1a9c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc0e9c1e76226d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.410+0000 7f1a97fff700 1 -- 172.21.15.1:0/2610245440 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c00b870 con 0x7f1aa00ce550 2024-03-18T18:50:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.410+0000 7f1a97fff700 1 -- 172.21.15.1:0/2610245440 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9c00b9d0 con 0x7f1aa00ce550 2024-03-18T18:50:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.410+0000 7f1a97fff700 1 -- 172.21.15.1:0/2610245440 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c00bd60 con 0x7f1aa00ce550 2024-03-18T18:50:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.411+0000 7f1aa7785700 1 -- 172.21.15.1:0/2610245440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aa00ce550 msgr2=0x7f1aa006a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.411+0000 7f1aa7785700 1 --2- 172.21.15.1:0/2610245440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aa00ce550 0x7f1aa006a750 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00cdec0 tx=0x7f1a9c009bd0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.411+0000 7f1aa7785700 1 -- 172.21.15.1:0/2610245440 shutdown_connections 2024-03-18T18:50:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.411+0000 7f1aa7785700 1 --2- 172.21.15.1:0/2610245440 >> v2:172.21.15.165:3300/0 conn(0x7f1aa00cfa90 0x7f1aa006ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.411+0000 7f1aa7785700 1 --2- 172.21.15.1:0/2610245440 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1aa015d5a0 0x7f1aa00ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.411+0000 7f1aa7785700 1 --2- 172.21.15.1:0/2610245440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aa00ce550 0x7f1aa006a750 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.411+0000 7f1aa7785700 1 -- 172.21.15.1:0/2610245440 >> 172.21.15.1:0/2610245440 conn(0x7f1aa00cc9b0 msgr2=0x7f1aa00ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.411+0000 7f1aa7785700 1 -- 172.21.15.1:0/2610245440 shutdown_connections 2024-03-18T18:50:20.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.411+0000 7f1aa7785700 1 -- 172.21.15.1:0/2610245440 wait complete. 2024-03-18T18:50:20.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.412+0000 7f1aa7785700 1 Processor -- start 2024-03-18T18:50:20.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.412+0000 7f1aa7785700 1 -- start start 2024-03-18T18:50:20.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa7785700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aa00ce550 0x7f1aa014af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa7785700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1aa00cfa90 0x7f1aa00b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa7785700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1aa015d5a0 0x7f1aa014b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa7785700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa015dec0 con 0x7f1aa00ce550 2024-03-18T18:50:20.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa7785700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa015e1a0 con 0x7f1aa00cfa90 2024-03-18T18:50:20.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa7785700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa00b48a0 con 0x7f1aa015d5a0 2024-03-18T18:50:20.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa5521700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aa00ce550 0x7f1aa014af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa5521700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aa00ce550 0x7f1aa014af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52084/0 (socket says 172.21.15.1:52084) 2024-03-18T18:50:20.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa5521700 1 -- 172.21.15.1:0/2979972281 learned_addr learned my addr 172.21.15.1:0/2979972281 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:20.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa5d22700 1 --2- 172.21.15.1:0/2979972281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1aa015d5a0 0x7f1aa014b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa4d20700 1 --2- 172.21.15.1:0/2979972281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1aa00cfa90 0x7f1aa00b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa5521700 1 -- 172.21.15.1:0/2979972281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1aa015d5a0 msgr2=0x7f1aa014b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:20.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa5521700 1 --2- 172.21.15.1:0/2979972281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1aa015d5a0 0x7f1aa014b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa5521700 1 -- 172.21.15.1:0/2979972281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1aa00cfa90 msgr2=0x7f1aa00b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:20.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa5521700 1 --2- 172.21.15.1:0/2979972281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1aa00cfa90 0x7f1aa00b50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa5521700 1 -- 172.21.15.1:0/2979972281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c009550 con 0x7f1aa00ce550 2024-03-18T18:50:20.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa4d20700 1 --2- 172.21.15.1:0/2979972281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1aa00cfa90 0x7f1aa00b50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:20.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa5d22700 1 --2- 172.21.15.1:0/2979972281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1aa015d5a0 0x7f1aa014b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:20.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa5521700 1 --2- 172.21.15.1:0/2979972281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aa00ce550 0x7f1aa014af30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1a9000c7f0 tx=0x7f1a9000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:20.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1a95ffb700 1 -- 172.21.15.1:0/2979972281 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a90011800 con 0x7f1aa00ce550 2024-03-18T18:50:20.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1a95ffb700 1 -- 172.21.15.1:0/2979972281 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a90011960 con 0x7f1aa00ce550 2024-03-18T18:50:20.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1aa7785700 1 -- 172.21.15.1:0/2979972281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa00b4b80 con 0x7f1aa00ce550 2024-03-18T18:50:20.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.413+0000 7f1a95ffb700 1 -- 172.21.15.1:0/2979972281 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a90011d10 con 0x7f1aa00ce550 2024-03-18T18:50:20.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.414+0000 7f1aa7785700 1 -- 172.21.15.1:0/2979972281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa015db00 con 0x7f1aa00ce550 2024-03-18T18:50:20.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.415+0000 7f1a95ffb700 1 -- 172.21.15.1:0/2979972281 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1a90018360 con 0x7f1aa00ce550 2024-03-18T18:50:20.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.415+0000 7f1a95ffb700 1 --2- 172.21.15.1:0/2979972281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1a8406ac90 0x7f1a8406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.416+0000 7f1a95ffb700 1 -- 172.21.15.1:0/2979972281 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1a90089620 con 0x7f1aa00ce550 2024-03-18T18:50:20.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.416+0000 7f1aa4d20700 1 --2- 172.21.15.1:0/2979972281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1a8406ac90 0x7f1a8406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.416+0000 7f1aa4d20700 1 --2- 172.21.15.1:0/2979972281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1a8406ac90 0x7f1a8406d150 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1aa015c480 tx=0x7f1a9c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:20.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.418+0000 7f1ab392eb80 1 Processor -- start 2024-03-18T18:50:20.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.418+0000 7f1ab392eb80 1 -- start start 2024-03-18T18:50:20.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.418+0000 7f1ab392eb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce3567c10 0x556ce3567ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.418+0000 7f1ab392eb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x556ce35390a0 0x556ce3584580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.418+0000 7f1ab392eb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 0x556ce353c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.418+0000 7f1ab392eb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce34b2f20 con 0x556ce3579630 2024-03-18T18:50:20.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.418+0000 7f1ab392eb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce3494c30 con 0x556ce3567c10 2024-03-18T18:50:20.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.418+0000 7f1ab392eb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x556ce3492430 con 0x556ce35390a0 2024-03-18T18:50:20.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.418+0000 7f1aa5d22700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 0x556ce353c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.419+0000 7f1aa5d22700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 0x556ce353c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52098/0 (socket says 172.21.15.1:52098) 2024-03-18T18:50:20.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.419+0000 7f1aa5d22700 1 -- 172.21.15.1:0/185152039 learned_addr learned my addr 172.21.15.1:0/185152039 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:20.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.419+0000 7f1aa4d20700 1 --2- 172.21.15.1:0/185152039 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x556ce35390a0 0x556ce3584580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.419+0000 7f1aa5521700 1 --2- 172.21.15.1:0/185152039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce3567c10 0x556ce3567ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.419+0000 7f1aa5d22700 1 -- 172.21.15.1:0/185152039 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x556ce35390a0 msgr2=0x556ce3584580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:20.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.419+0000 7f1aa5d22700 1 --2- 172.21.15.1:0/185152039 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x556ce35390a0 0x556ce3584580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.419+0000 7f1aa5d22700 1 -- 172.21.15.1:0/185152039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce3567c10 msgr2=0x556ce3567ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:20.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.419+0000 7f1aa5d22700 1 --2- 172.21.15.1:0/185152039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce3567c10 0x556ce3567ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.419+0000 7f1aa5d22700 1 -- 172.21.15.1:0/185152039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ce3281790 con 0x556ce3579630 2024-03-18T18:50:20.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.419+0000 7f1aa5d22700 1 --2- 172.21.15.1:0/185152039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 0x556ce353c160 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1a9800d460 tx=0x7f1a9800d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f9699e17248aacd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:20.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.419+0000 7f1a8affd700 1 -- 172.21.15.1:0/185152039 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9800dd70 con 0x556ce3579630 2024-03-18T18:50:20.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.419+0000 7f1a8affd700 1 -- 172.21.15.1:0/185152039 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a98015410 con 0x556ce3579630 2024-03-18T18:50:20.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.419+0000 7f1a8affd700 1 -- 172.21.15.1:0/185152039 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a98015780 con 0x556ce3579630 2024-03-18T18:50:20.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.420+0000 7f1ab392eb80 1 -- 172.21.15.1:0/185152039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 msgr2=0x556ce353c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:20.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.420+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/185152039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 0x556ce353c160 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1a9800d460 tx=0x7f1a9800d490 comp rx=0 tx=0).stop 2024-03-18T18:50:20.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.420+0000 7f1ab392eb80 1 -- 172.21.15.1:0/185152039 shutdown_connections 2024-03-18T18:50:20.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.420+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/185152039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 0x556ce353c160 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.420+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/185152039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce3567c10 0x556ce3567ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.420+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/185152039 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x556ce35390a0 0x556ce3584580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.420+0000 7f1ab392eb80 1 -- 172.21.15.1:0/185152039 >> 172.21.15.1:0/185152039 conn(0x556ce349bcf0 msgr2=0x556ce353ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:20.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.420+0000 7f1ab392eb80 1 -- 172.21.15.1:0/185152039 shutdown_connections 2024-03-18T18:50:20.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.420+0000 7f1ab392eb80 1 -- 172.21.15.1:0/185152039 wait complete. 2024-03-18T18:50:20.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.420+0000 7f1ab392eb80 1 -- start start 2024-03-18T18:50:20.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.421+0000 7f1ab392eb80 1 Processor -- start 2024-03-18T18:50:20.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1ab392eb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x556ce35390a0 0x556ce357a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1ab392eb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce3567c10 0x556ce353ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1ab392eb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 0x556ce353cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1ab392eb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce348dd20 con 0x556ce3579630 2024-03-18T18:50:20.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1ab392eb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce3429770 con 0x556ce3567c10 2024-03-18T18:50:20.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1ab392eb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x556ce344c6f0 con 0x556ce35390a0 2024-03-18T18:50:20.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1ab392eb80 10 client.0 _ll_register_callbacks cb 0x556ce32955a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T18:50:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1aa5d22700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 0x556ce353cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1aa5d22700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 0x556ce353cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52106/0 (socket says 172.21.15.1:52106) 2024-03-18T18:50:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1aa5d22700 1 -- 172.21.15.1:0/3328665001 learned_addr learned my addr 172.21.15.1:0/3328665001 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1aa5521700 1 --2- 172.21.15.1:0/3328665001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x556ce35390a0 0x556ce357a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1aa4d20700 1 --2- 172.21.15.1:0/3328665001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce3567c10 0x556ce353ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1aa5d22700 1 -- 172.21.15.1:0/3328665001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x556ce35390a0 msgr2=0x556ce357a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1aa5d22700 1 --2- 172.21.15.1:0/3328665001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x556ce35390a0 0x556ce357a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1aa5d22700 1 -- 172.21.15.1:0/3328665001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce3567c10 msgr2=0x556ce353ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1aa5d22700 1 --2- 172.21.15.1:0/3328665001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce3567c10 0x556ce353ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1aa5d22700 1 -- 172.21.15.1:0/3328665001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a98008c50 con 0x556ce3579630 2024-03-18T18:50:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1aa4d20700 1 --2- 172.21.15.1:0/3328665001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce3567c10 0x556ce353ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1aa5d22700 1 --2- 172.21.15.1:0/3328665001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 0x556ce353cf60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1a98018040 tx=0x7f1a98004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a6730ff2fe8ae28 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1a7ffff700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T18:50:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1a7ffff700 1 -- 172.21.15.1:0/3328665001 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9801f810 con 0x556ce3579630 2024-03-18T18:50:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1a7ffff700 1 -- 172.21.15.1:0/3328665001 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1a9801f970 con 0x556ce3579630 2024-03-18T18:50:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3328665001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556ce352fe90 con 0x556ce3579630 2024-03-18T18:50:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1a7ffff700 1 -- 172.21.15.1:0/3328665001 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a98008480 con 0x556ce3579630 2024-03-18T18:50:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1ab392eb80 10 client.4794 fetch_fsmap learned FSMap version 9 2024-03-18T18:50:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.422+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3328665001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x556ce336e0a0 con 0x556ce3579630 2024-03-18T18:50:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.423+0000 7f1a7ffff700 1 -- 172.21.15.1:0/3328665001 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556ce352fe90 con 0x556ce3579630 2024-03-18T18:50:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.423+0000 7f1a7ffff700 1 -- 172.21.15.1:0/3328665001 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 9) v1 ==== 3122+0+0 (secure 0 0 0) 0x7f1a9801fb30 con 0x556ce3579630 2024-03-18T18:50:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.423+0000 7f1ab392eb80 10 client.4794 fetch_fsmap finished waiting for FSMap version 9 2024-03-18T18:50:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.423+0000 7f1ab392eb80 10 client.4794 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-18T18:50:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.423+0000 7f1ab392eb80 20 client.4794 populate_metadata read hostname 'smithi001' 2024-03-18T18:50:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.423+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/3328665001 >> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] conn(0x556ce356a170 0x556ce355b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.423+0000 7f1ab392eb80 4 client.4794 mds_command: new command op to 4273 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T18:50:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.423+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3328665001 --> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556ce350d010 con 0x556ce356a170 2024-03-18T18:50:20.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.423+0000 7f1aa5521700 1 --2- 172.21.15.1:0/3328665001 >> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] conn(0x556ce356a170 0x556ce355b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.423+0000 7f1a7ffff700 10 client.4794 ms_handle_connect on v2:172.21.15.1:6835/2118908012 2024-03-18T18:50:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.423+0000 7f1aa5521700 1 --2- 172.21.15.1:0/3328665001 >> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] conn(0x556ce356a170 0x556ce355b040 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9f0b4f7cc32b824 server_cookie=b869cabef1572290 in_seq=0 out_seq=0 2024-03-18T18:50:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.424+0000 7f1a7ffff700 1 -- 172.21.15.1:0/3328665001 <== mds.0 v2:172.21.15.1:6835/2118908012 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556ce350d010 con 0x556ce356a170 2024-03-18T18:50:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.424+0000 7f1a7ffff700 10 client.4794 handle_command_reply: tid=0 2024-03-18T18:50:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.424+0000 7f1ab392eb80 1 client.4794 shutdown 2024-03-18T18:50:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.424+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3328665001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 msgr2=0x556ce353cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.424+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/3328665001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 0x556ce353cf60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1a98018040 tx=0x7f1a98004b40 comp rx=0 tx=0).stop 2024-03-18T18:50:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.424+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3328665001 shutdown_connections 2024-03-18T18:50:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.424+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/3328665001 >> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] conn(0x556ce356a170 0x556ce355b040 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.424+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/3328665001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x556ce35390a0 0x556ce357a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.424+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/3328665001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 0x556ce353cf60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.424+0000 7f1a7ffff700 0 client.4794 ms_handle_reset on v2:172.21.15.1:6835/2118908012 2024-03-18T18:50:20.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.424+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/3328665001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce3567c10 0x556ce353ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.424+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3328665001 >> 172.21.15.1:0/3328665001 conn(0x556ce349bcf0 msgr2=0x556ce3565470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.424+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3328665001 shutdown_connections 2024-03-18T18:50:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.424+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3328665001 wait complete. 2024-03-18T18:50:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.424+0000 7f1ab392eb80 20 client.4794 trim_cache size 0 max 16384 2024-03-18T18:50:20.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.442+0000 7f1ab392eb80 1 Processor -- start 2024-03-18T18:50:20.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.442+0000 7f1ab392eb80 1 -- start start 2024-03-18T18:50:20.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1ab392eb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x556ce35390a0 0x556ce3568090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1ab392eb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce3567c10 0x556ce3585d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1ab392eb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 0x556ce3586260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1ab392eb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce3282390 con 0x556ce3579630 2024-03-18T18:50:20.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1ab392eb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce355b450 con 0x556ce3567c10 2024-03-18T18:50:20.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1ab392eb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x556ce353ac00 con 0x556ce35390a0 2024-03-18T18:50:20.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1aa5d22700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 0x556ce3586260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1aa5d22700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 0x556ce3586260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52118/0 (socket says 172.21.15.1:52118) 2024-03-18T18:50:20.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1aa5d22700 1 -- 172.21.15.1:0/3205488140 learned_addr learned my addr 172.21.15.1:0/3205488140 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:20.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1aa5d22700 1 -- 172.21.15.1:0/3205488140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x556ce35390a0 msgr2=0x556ce3568090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:20.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1aa5d22700 1 --2- 172.21.15.1:0/3205488140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x556ce35390a0 0x556ce3568090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1aa4d20700 1 --2- 172.21.15.1:0/3205488140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce3567c10 0x556ce3585d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1aa5d22700 1 -- 172.21.15.1:0/3205488140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce3567c10 msgr2=0x556ce3585d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:20.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1aa5d22700 1 --2- 172.21.15.1:0/3205488140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce3567c10 0x556ce3585d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1aa5d22700 1 -- 172.21.15.1:0/3205488140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ce3454010 con 0x556ce3579630 2024-03-18T18:50:20.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1aa4d20700 1 --2- 172.21.15.1:0/3205488140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce3567c10 0x556ce3585d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:20.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1aa5d22700 1 --2- 172.21.15.1:0/3205488140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 0x556ce3586260 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1a98007130 tx=0x7f1a9801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79331e52314cbfea server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:20.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1a8b7fe700 1 -- 172.21.15.1:0/3205488140 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a98028490 con 0x556ce3579630 2024-03-18T18:50:20.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1a8b7fe700 1 -- 172.21.15.1:0/3205488140 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9800d040 con 0x556ce3579630 2024-03-18T18:50:20.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.443+0000 7f1a8b7fe700 1 -- 172.21.15.1:0/3205488140 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9801f610 con 0x556ce3579630 2024-03-18T18:50:20.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.444+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3205488140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 msgr2=0x556ce3586260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:20.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.444+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/3205488140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 0x556ce3586260 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1a98007130 tx=0x7f1a9801e000 comp rx=0 tx=0).stop 2024-03-18T18:50:20.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.444+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3205488140 shutdown_connections 2024-03-18T18:50:20.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.444+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/3205488140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x556ce35390a0 0x556ce3568090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.444+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/3205488140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3579630 0x556ce3586260 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.444+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/3205488140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce3567c10 0x556ce3585d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.444+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3205488140 >> 172.21.15.1:0/3205488140 conn(0x556ce349bcf0 msgr2=0x556ce353ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:20.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.444+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3205488140 shutdown_connections 2024-03-18T18:50:20.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.444+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3205488140 wait complete. 2024-03-18T18:50:20.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.444+0000 7f1ab392eb80 1 -- start start 2024-03-18T18:50:20.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.445+0000 7f1ab392eb80 1 Processor -- start 2024-03-18T18:50:20.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.445+0000 7f1ab392eb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce35390a0 0x556ce35e8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.445+0000 7f1ab392eb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3567c10 0x556ce3579030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.445+0000 7f1ab392eb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x556ce3579630 0x556ce353b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.445+0000 7f1ab392eb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce353c1e0 con 0x556ce3567c10 2024-03-18T18:50:20.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.445+0000 7f1ab392eb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ce353dfd0 con 0x556ce35390a0 2024-03-18T18:50:20.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.445+0000 7f1ab392eb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x556ce353e200 con 0x556ce3579630 2024-03-18T18:50:20.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.446+0000 7f1ab392eb80 10 client.0 _ll_register_callbacks cb 0x7f1a98008e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T18:50:20.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.446+0000 7f1aa4d20700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3567c10 0x556ce3579030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.446+0000 7f1aa4d20700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3567c10 0x556ce3579030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52128/0 (socket says 172.21.15.1:52128) 2024-03-18T18:50:20.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.446+0000 7f1aa4d20700 1 -- 172.21.15.1:0/3329118940 learned_addr learned my addr 172.21.15.1:0/3329118940 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:20.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.446+0000 7f1aa5521700 1 --2- 172.21.15.1:0/3329118940 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce35390a0 0x556ce35e8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.446+0000 7f1aa5d22700 1 --2- 172.21.15.1:0/3329118940 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x556ce3579630 0x556ce353b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.446+0000 7f1aa4d20700 1 -- 172.21.15.1:0/3329118940 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x556ce3579630 msgr2=0x556ce353b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:20.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.446+0000 7f1aa4d20700 1 --2- 172.21.15.1:0/3329118940 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x556ce3579630 0x556ce353b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.446+0000 7f1aa4d20700 1 -- 172.21.15.1:0/3329118940 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce35390a0 msgr2=0x556ce35e8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:20.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.446+0000 7f1aa4d20700 1 --2- 172.21.15.1:0/3329118940 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce35390a0 0x556ce35e8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.446+0000 7f1aa4d20700 1 -- 172.21.15.1:0/3329118940 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556ce336e0a0 con 0x556ce3567c10 2024-03-18T18:50:20.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.446+0000 7f1aa4d20700 1 --2- 172.21.15.1:0/3329118940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3567c10 0x556ce3579030 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009fd0 tx=0x7f1a9c00e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=412022ebc529e3f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:20.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.446+0000 7f1a7ffff700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T18:50:20.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.446+0000 7f1a7ffff700 1 -- 172.21.15.1:0/3329118940 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c02a040 con 0x556ce3567c10 2024-03-18T18:50:20.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.446+0000 7f1a7ffff700 1 -- 172.21.15.1:0/3329118940 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1a9c003ab0 con 0x556ce3567c10 2024-03-18T18:50:20.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.446+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3329118940 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1a98015410 con 0x556ce3567c10 2024-03-18T18:50:20.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.446+0000 7f1a7ffff700 1 -- 172.21.15.1:0/3329118940 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a9c003c90 con 0x556ce3567c10 2024-03-18T18:50:20.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.446+0000 7f1ab392eb80 10 client.4800 fetch_fsmap learned FSMap version 9 2024-03-18T18:50:20.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.446+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3329118940 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x556ce32e21d0 con 0x556ce3567c10 2024-03-18T18:50:20.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.447+0000 7f1a7ffff700 1 -- 172.21.15.1:0/3329118940 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a98015410 con 0x556ce3567c10 2024-03-18T18:50:20.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.447+0000 7f1a7ffff700 1 -- 172.21.15.1:0/3329118940 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 9) v1 ==== 3122+0+0 (secure 0 0 0) 0x7f1a9c02f440 con 0x556ce3567c10 2024-03-18T18:50:20.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.447+0000 7f1ab392eb80 10 client.4800 fetch_fsmap finished waiting for FSMap version 9 2024-03-18T18:50:20.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.447+0000 7f1ab392eb80 10 client.4800 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-18T18:50:20.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.447+0000 7f1ab392eb80 20 client.4800 populate_metadata read hostname 'smithi001' 2024-03-18T18:50:20.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.447+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/3329118940 >> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] conn(0x556ce3585d00 0x556ce35860e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.447+0000 7f1ab392eb80 4 client.4800 mds_command: new command op to 4273 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T18:50:20.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.447+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3329118940 --> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556ce32c7040 con 0x556ce3585d00 2024-03-18T18:50:20.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.447+0000 7f1aa5521700 1 --2- 172.21.15.1:0/3329118940 >> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] conn(0x556ce3585d00 0x556ce35860e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.447+0000 7f1a7ffff700 10 client.4800 ms_handle_connect on v2:172.21.15.1:6835/2118908012 2024-03-18T18:50:20.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.447+0000 7f1aa5521700 1 --2- 172.21.15.1:0/3329118940 >> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] conn(0x556ce3585d00 0x556ce35860e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d651cc814f4ece43 server_cookie=82a4dba0a00cef72 in_seq=0 out_seq=0 2024-03-18T18:50:20.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.447+0000 7f1a7ffff700 1 -- 172.21.15.1:0/3329118940 <== mds.0 v2:172.21.15.1:6835/2118908012 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x556ce32c7040 con 0x556ce3585d00 2024-03-18T18:50:20.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.447+0000 7f1a7ffff700 10 client.4800 handle_command_reply: tid=0 2024-03-18T18:50:20.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.447+0000 7f1ab392eb80 1 client.4800 shutdown 2024-03-18T18:50:20.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.447+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3329118940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3567c10 msgr2=0x556ce3579030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:20.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.447+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/3329118940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3567c10 0x556ce3579030 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009fd0 tx=0x7f1a9c00e000 comp rx=0 tx=0).stop 2024-03-18T18:50:20.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.447+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3329118940 shutdown_connections 2024-03-18T18:50:20.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/3329118940 >> [v2:172.21.15.1:6835/2118908012,v1:172.21.15.1:6837/2118908012] conn(0x556ce3585d00 0x556ce35860e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/3329118940 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x556ce3579630 0x556ce353b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/3329118940 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556ce35390a0 0x556ce35e8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1ab392eb80 1 --2- 172.21.15.1:0/3329118940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556ce3567c10 0x556ce3579030 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3329118940 >> 172.21.15.1:0/3329118940 conn(0x556ce349bcf0 msgr2=0x556ce3568090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:20.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1a7ffff700 0 client.4800 ms_handle_reset on v2:172.21.15.1:6835/2118908012 2024-03-18T18:50:20.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3329118940 shutdown_connections 2024-03-18T18:50:20.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1ab392eb80 1 -- 172.21.15.1:0/3329118940 wait complete. 2024-03-18T18:50:20.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1ab392eb80 20 client.4800 trim_cache size 0 max 16384 2024-03-18T18:50:20.451 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T18:50:20.451 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "9c1c697f-8655-4ef3-8a13-da568d2cbe86", 2024-03-18T18:50:20.451 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-18T18:50:20.451 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 4273, 2024-03-18T18:50:20.451 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-18T18:50:20.451 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-18T18:50:20.451 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-18T18:50:20.451 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 35.634426212999998, 2024-03-18T18:50:20.451 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 9, 2024-03-18T18:50:20.451 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 25, 2024-03-18T18:50:20.452 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 25, 2024-03-18T18:50:20.452 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 42.225485149999997 2024-03-18T18:50:20.452 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T18:50:20.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1aa7785700 1 -- 172.21.15.1:0/2979972281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1a8406ac90 msgr2=0x7f1a8406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:20.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1aa7785700 1 --2- 172.21.15.1:0/2979972281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1a8406ac90 0x7f1a8406d150 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1aa015c480 tx=0x7f1a9c008030 comp rx=0 tx=0).stop 2024-03-18T18:50:20.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1aa7785700 1 -- 172.21.15.1:0/2979972281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aa00ce550 msgr2=0x7f1aa014af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:20.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1aa7785700 1 --2- 172.21.15.1:0/2979972281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aa00ce550 0x7f1aa014af30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1a9000c7f0 tx=0x7f1a9000cbb0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1aa7785700 1 -- 172.21.15.1:0/2979972281 shutdown_connections 2024-03-18T18:50:20.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1aa7785700 1 --2- 172.21.15.1:0/2979972281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1a8406ac90 0x7f1a8406d150 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1aa7785700 1 --2- 172.21.15.1:0/2979972281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1aa015d5a0 0x7f1aa014b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1aa7785700 1 --2- 172.21.15.1:0/2979972281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aa00ce550 0x7f1aa014af30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1aa7785700 1 --2- 172.21.15.1:0/2979972281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1aa00cfa90 0x7f1aa00b50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1aa7785700 1 -- 172.21.15.1:0/2979972281 >> 172.21.15.1:0/2979972281 conn(0x7f1aa00cc9b0 msgr2=0x7f1aa006f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:20.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1aa7785700 1 -- 172.21.15.1:0/2979972281 shutdown_connections 2024-03-18T18:50:20.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.448+0000 7f1aa7785700 1 -- 172.21.15.1:0/2979972281 wait complete. 2024-03-18T18:50:20.457 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-18T18:50:20.468 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-18T18:50:20.468 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-18T18:50:20.468 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-18T18:50:20.468 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T18:50:20.468 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T18:50:20.468 INFO:tasks.ceph_fuse:client.2 config is {} 2024-03-18T18:50:20.469 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T18:50:20.469 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-18T18:50:20.469 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-18T18:50:20.469 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T18:50:20.469 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-18T18:50:20.469 INFO:tasks.ceph_fuse:client.3 config is {} 2024-03-18T18:50:20.469 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T18:50:20.469 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-18T18:50:20.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:20.469 DEBUG:teuthology.orchestra.run.smithi165:> ip netns list 2024-03-18T18:50:20.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:20.493 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip link delete ceph-brx 2024-03-18T18:50:20.567 INFO:teuthology.orchestra.run.smithi165.stderr:Cannot find device "ceph-brx" 2024-03-18T18:50:20.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T18:50:20.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:20.569 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T18:50:20.588 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-18T18:50:20.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:20.589 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T18:50:20.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:20.669 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link delete ceph-brx 2024-03-18T18:50:20.767 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-18T18:50:20.768 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T18:50:20.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.899+0000 7fc505c68700 1 Processor -- start 2024-03-18T18:50:20.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.899+0000 7fc505c68700 1 -- start start 2024-03-18T18:50:20.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.900+0000 7fc505c68700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5000716b0 0x7fc500071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.900+0000 7fc505c68700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc5000ae580 0x7fc50006d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.900+0000 7fc505c68700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc500071fd0 con 0x7fc5000abd70 2024-03-18T18:50:20.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.900+0000 7fc505c68700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5000ac230 con 0x7fc5000ae580 2024-03-18T18:50:20.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.900+0000 7fc505c68700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc50006c620 con 0x7fc5000716b0 2024-03-18T18:50:20.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.900+0000 7fc4feffd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc5000ae580 0x7fc50006d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.900+0000 7fc4feffd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc5000ae580 0x7fc50006d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:56006/0 (socket says 172.21.15.1:56006) 2024-03-18T18:50:20.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.900+0000 7fc4feffd700 1 -- 172.21.15.1:0/3282027572 learned_addr learned my addr 172.21.15.1:0/3282027572 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:20.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.900+0000 7fc4fffff700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fc5000abd70 0x7fc50006d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:60906/0 (socket says 172.21.15.1:60906) 2024-03-18T18:50:20.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.901+0000 7fc4ff7fe700 1 --2- 172.21.15.1:0/3282027572 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5000716b0 0x7fc500071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.901+0000 7fc4feffd700 1 -- 172.21.15.1:0/3282027572 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5000716b0 msgr2=0x7fc500071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:20.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.901+0000 7fc4feffd700 1 --2- 172.21.15.1:0/3282027572 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5000716b0 0x7fc500071a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.901+0000 7fc4feffd700 1 -- 172.21.15.1:0/3282027572 >> v1:172.21.15.165:6789/0 conn(0x7fc5000abd70 legacy=0x7fc50006d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:20.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.901+0000 7fc4feffd700 1 -- 172.21.15.1:0/3282027572 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc50006c900 con 0x7fc5000ae580 2024-03-18T18:50:20.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.901+0000 7fc4fe7fc700 1 -- 172.21.15.1:0/3282027572 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034909507 0 0) 0x7fc500071fd0 con 0x7fc5000abd70 2024-03-18T18:50:20.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.901+0000 7fc4feffd700 1 --2- 172.21.15.1:0/3282027572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5000ae580 0x7fc50006d100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc4e80098f0 tx=0x7fc4e8009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e3a0515ad607d7d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:20.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.901+0000 7fc4fe7fc700 1 -- 172.21.15.1:0/3282027572 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e800b8a0 con 0x7fc5000ae580 2024-03-18T18:50:20.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.901+0000 7fc4fe7fc700 1 -- 172.21.15.1:0/3282027572 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e800ba00 con 0x7fc5000ae580 2024-03-18T18:50:20.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.901+0000 7fc4fe7fc700 1 -- 172.21.15.1:0/3282027572 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e800bd90 con 0x7fc5000ae580 2024-03-18T18:50:20.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.902+0000 7fc505c68700 1 -- 172.21.15.1:0/3282027572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5000ae580 msgr2=0x7fc50006d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:20.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.902+0000 7fc505c68700 1 --2- 172.21.15.1:0/3282027572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5000ae580 0x7fc50006d100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc4e80098f0 tx=0x7fc4e8009c00 comp rx=0 tx=0).stop 2024-03-18T18:50:20.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.902+0000 7fc505c68700 1 -- 172.21.15.1:0/3282027572 shutdown_connections 2024-03-18T18:50:20.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.902+0000 7fc505c68700 1 --2- 172.21.15.1:0/3282027572 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5000716b0 0x7fc500071a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.902+0000 7fc505c68700 1 --2- 172.21.15.1:0/3282027572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5000ae580 0x7fc50006d100 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.902+0000 7fc505c68700 1 -- 172.21.15.1:0/3282027572 >> 172.21.15.1:0/3282027572 conn(0x7fc5000cee10 msgr2=0x7fc5000cf220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:20.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.902+0000 7fc505c68700 1 -- 172.21.15.1:0/3282027572 shutdown_connections 2024-03-18T18:50:20.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.902+0000 7fc505c68700 1 -- 172.21.15.1:0/3282027572 wait complete. 2024-03-18T18:50:20.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.903+0000 7fc505c68700 1 Processor -- start 2024-03-18T18:50:20.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.903+0000 7fc505c68700 1 -- start start 2024-03-18T18:50:20.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.904+0000 7fc505c68700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5000716b0 0x7fc500155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.904+0000 7fc505c68700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5000abd70 0x7fc5000cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.904+0000 7fc505c68700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc50009a980 0x7fc5000ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.904+0000 7fc505c68700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc500155f40 con 0x7fc5000abd70 2024-03-18T18:50:20.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.904+0000 7fc505c68700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50015e390 con 0x7fc50009a980 2024-03-18T18:50:20.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.904+0000 7fc505c68700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc50015ca80 con 0x7fc5000716b0 2024-03-18T18:50:20.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.904+0000 7fc4feffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5000abd70 0x7fc5000cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.904+0000 7fc4feffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5000abd70 0x7fc5000cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52130/0 (socket says 172.21.15.1:52130) 2024-03-18T18:50:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.904+0000 7fc4feffd700 1 -- 172.21.15.1:0/2086540916 learned_addr learned my addr 172.21.15.1:0/2086540916 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.904+0000 7fc4ff7fe700 1 --2- 172.21.15.1:0/2086540916 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5000716b0 0x7fc500155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.904+0000 7fc4fffff700 1 --2- 172.21.15.1:0/2086540916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc50009a980 0x7fc5000ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.904+0000 7fc4feffd700 1 -- 172.21.15.1:0/2086540916 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5000716b0 msgr2=0x7fc500155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.904+0000 7fc4feffd700 1 --2- 172.21.15.1:0/2086540916 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5000716b0 0x7fc500155a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.904+0000 7fc4feffd700 1 -- 172.21.15.1:0/2086540916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc50009a980 msgr2=0x7fc5000ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.904+0000 7fc4feffd700 1 --2- 172.21.15.1:0/2086540916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc50009a980 0x7fc5000ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.904+0000 7fc4feffd700 1 -- 172.21.15.1:0/2086540916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e8009550 con 0x7fc5000abd70 2024-03-18T18:50:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.905+0000 7fc4feffd700 1 --2- 172.21.15.1:0/2086540916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5000abd70 0x7fc5000cc870 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8014040 tx=0x7fc4e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.905+0000 7fc4f7fff700 1 -- 172.21.15.1:0/2086540916 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e80036a0 con 0x7fc5000abd70 2024-03-18T18:50:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.905+0000 7fc4f7fff700 1 -- 172.21.15.1:0/2086540916 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e801e070 con 0x7fc5000abd70 2024-03-18T18:50:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.905+0000 7fc505c68700 1 -- 172.21.15.1:0/2086540916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc500099bc0 con 0x7fc5000abd70 2024-03-18T18:50:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.905+0000 7fc4f7fff700 1 -- 172.21.15.1:0/2086540916 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8003a20 con 0x7fc5000abd70 2024-03-18T18:50:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.905+0000 7fc505c68700 1 -- 172.21.15.1:0/2086540916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc50009a180 con 0x7fc5000abd70 2024-03-18T18:50:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.906+0000 7fc505c68700 1 -- 172.21.15.1:0/2086540916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5000671c0 con 0x7fc5000abd70 2024-03-18T18:50:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.906+0000 7fc4f7fff700 1 -- 172.21.15.1:0/2086540916 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc4e8003b80 con 0x7fc5000abd70 2024-03-18T18:50:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.907+0000 7fc4f7fff700 1 --2- 172.21.15.1:0/2086540916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc4e006ac90 0x7fc4e006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.907+0000 7fc4f7fff700 1 -- 172.21.15.1:0/2086540916 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc4e8020070 con 0x7fc5000abd70 2024-03-18T18:50:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.907+0000 7fc4ff7fe700 1 --2- 172.21.15.1:0/2086540916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc4e006ac90 0x7fc4e006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.907+0000 7fc4ff7fe700 1 --2- 172.21.15.1:0/2086540916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc4e006ac90 0x7fc4e006d150 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8009b00 tx=0x7fc4f8003a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:20.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:20.911+0000 7fc4f7fff700 1 -- 172.21.15.1:0/2086540916 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc4e8055a60 con 0x7fc5000abd70 2024-03-18T18:50:21.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:21.163+0000 7fc505c68700 1 -- 172.21.15.1:0/2086540916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc500073350 con 0x7fc5000abd70 2024-03-18T18:50:21.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:21.163+0000 7fc4f7fff700 1 -- 172.21.15.1:0/2086540916 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7fc4e8015020 con 0x7fc5000abd70 2024-03-18T18:50:21.160 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T18:50:21.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:21.165+0000 7fc505c68700 1 -- 172.21.15.1:0/2086540916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc4e006ac90 msgr2=0x7fc4e006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:21.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:21.165+0000 7fc505c68700 1 --2- 172.21.15.1:0/2086540916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc4e006ac90 0x7fc4e006d150 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8009b00 tx=0x7fc4f8003a20 comp rx=0 tx=0).stop 2024-03-18T18:50:21.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:21.165+0000 7fc505c68700 1 -- 172.21.15.1:0/2086540916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5000abd70 msgr2=0x7fc5000cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:21.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:21.165+0000 7fc505c68700 1 --2- 172.21.15.1:0/2086540916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5000abd70 0x7fc5000cc870 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8014040 tx=0x7fc4e8004750 comp rx=0 tx=0).stop 2024-03-18T18:50:21.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:21.165+0000 7fc505c68700 1 -- 172.21.15.1:0/2086540916 shutdown_connections 2024-03-18T18:50:21.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:21.165+0000 7fc505c68700 1 --2- 172.21.15.1:0/2086540916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc4e006ac90 0x7fc4e006d150 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:21.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:21.165+0000 7fc505c68700 1 --2- 172.21.15.1:0/2086540916 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5000716b0 0x7fc500155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:21.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:21.165+0000 7fc505c68700 1 --2- 172.21.15.1:0/2086540916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc50009a980 0x7fc5000ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:21.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:21.165+0000 7fc505c68700 1 --2- 172.21.15.1:0/2086540916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5000abd70 0x7fc5000cc870 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:21.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:21.165+0000 7fc505c68700 1 -- 172.21.15.1:0/2086540916 >> 172.21.15.1:0/2086540916 conn(0x7fc5000cee10 msgr2=0x7fc500070d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:21.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:21.166+0000 7fc505c68700 1 -- 172.21.15.1:0/2086540916 shutdown_connections 2024-03-18T18:50:21.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:21.166+0000 7fc505c68700 1 -- 172.21.15.1:0/2086540916 wait complete. 2024-03-18T18:50:21.174 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T18:50:21.174 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T18:50:21.174 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-18T18:50:21.174 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T18:50:21.174 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T18:50:21.174 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T18:50:21.174 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T18:50:21.174 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T18:50:21.174 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T18:50:21.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:21.175 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-18T18:50:21.193 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T18:50:21.194 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T18:50:21.194 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T18:50:21.194 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:21.194 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T18:50:21.194 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:21.194 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T18:50:21.194 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-18T18:50:21.194 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-18T18:50:21.194 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T18:50:21.194 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-18T18:50:21.194 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-18T18:50:21.195 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T18:50:21.195 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-18T18:50:21.195 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-18T18:50:21.195 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T18:50:21.195 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 42548sec preferred_lft 42548sec 2024-03-18T18:50:21.195 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link 2024-03-18T18:50:21.195 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:21.195 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T18:50:21.195 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-18T18:50:21.195 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-18T18:50:21.195 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T18:50:21.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:21.196 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T18:50:21.196 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T18:50:21.196 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add name ceph-brx type bridge 2024-03-18T18:50:21.196 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr flush dev ceph-brx 2024-03-18T18:50:21.196 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set ceph-brx up 2024-03-18T18:50:21.196 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T18:50:21.196 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T18:50:21.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:21.311 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T18:50:21.338 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-18T18:50:21.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:21.339 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-18T18:50:21.400 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-18T18:50:21.400 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T18:50:21.400 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T18:50:21.401 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T18:50:21.401 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T18:50:21.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:21.401 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T18:50:21.401 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T18:50:21.401 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T18:50:21.401 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T18:50:21.401 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T18:50:21.401 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T18:50:21.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:21.506 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T18:50:21.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:21.525 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list-id 2024-03-18T18:50:21.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:21.584 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T18:50:21.584 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T18:50:21.584 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T18:50:21.584 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-18T18:50:21.584 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T18:50:21.680 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-18T18:50:21.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:21.680 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T18:50:21.681 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T18:50:21.681 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-18T18:50:21.681 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T18:50:21.681 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-18T18:50:21.681 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-18T18:50:21.681 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-18T18:50:21.681 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T18:50:21.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:21.869 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T18:50:21.869 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T18:50:21.869 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-18T18:50:21.869 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T18:50:21.869 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T18:50:21.967 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T18:50:21.967 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T18:50:21.968 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T18:50:21.986 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T18:50:21.986 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T18:50:21.986 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T18:50:22.044 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T18:50:22.112 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T18:50:22.113 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T18:50:22.180 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T18:50:22.181 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T18:50:22.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:22.181 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T18:50:22.239 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T18:50:22.239 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T18:50:22.282 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T18:50:22.309 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T18:50:22.310 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T18:50:22.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-18T18:50:22.373+0000 7f76f0c685c0 -1 init, newargv = 0x5569893beb10 newargc=15 2024-03-18T18:50:22.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[88596]: starting ceph client 2024-03-18T18:50:22.374 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[88596]: starting fuse 2024-03-18T18:50:22.380 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T18:50:22.381 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T18:50:22.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:22.382 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T18:50:22.440 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T18:50:22.441 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-03-18T18:50:22.441 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T18:50:22.441 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-18T18:50:22.441 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-18T18:50:22.441 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-18T18:50:22.441 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-18T18:50:22.441 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T18:50:22.441 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-18T18:50:22.441 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T18:50:22.441 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-18T18:50:22.441 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T18:50:22.441 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T18:50:22.442 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-18T18:50:22.442 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-18T18:50:22.442 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-18T18:50:22.442 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-18T18:50:22.442 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T18:50:22.442 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-18T18:50:22.442 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T18:50:22.442 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T18:50:22.442 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T18:50:22.442 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-18T18:50:22.442 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-18T18:50:22.442 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-18T18:50:22.442 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T18:50:22.442 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T18:50:22.442 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-18T18:50:22.443 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-18T18:50:22.566 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.88596.asok 2024-03-18T18:50:22.748 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.88596.asok 2024-03-18T18:50:22.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:22.748 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.88596.asok status 2024-03-18T18:50:22.911 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T18:50:22.911 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-18T18:50:22.911 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T18:50:22.911 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T18:50:22.911 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-18T18:50:22.911 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-18T18:50:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T18:50:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "88596", 2024-03-18T18:50:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-18T18:50:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:50:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-18T18:50:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-18T18:50:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 4812, 2024-03-18T18:50:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-18T18:50:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-18T18:50:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-18T18:50:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4812 2024-03-18T18:50:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:50:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T18:50:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T18:50:22.912 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T18:50:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4064220425 2024-03-18T18:50:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:50:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:50:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T18:50:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T18:50:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T18:50:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4064220425 2024-03-18T18:50:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:50:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.4812 192.168.0.1:0/4064220425", 2024-03-18T18:50:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/4064220425", 2024-03-18T18:50:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-18T18:50:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 9, 2024-03-18T18:50:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 25, 2024-03-18T18:50:22.913 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-18T18:50:22.914 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-18T18:50:22.914 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-18T18:50:22.914 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T18:50:22.921 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T18:50:23.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.050+0000 7fae78e4d700 1 Processor -- start 2024-03-18T18:50:23.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.050+0000 7fae78e4d700 1 -- start start 2024-03-18T18:50:23.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.051+0000 7fae78e4d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fae740af610 0x7fae740cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:23.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.051+0000 7fae78e4d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae740ccf40 0x7fae74071a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:23.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.051+0000 7fae78e4d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae740cf090 con 0x7fae74144b70 2024-03-18T18:50:23.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.051+0000 7fae78e4d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae740cd3d0 con 0x7fae740af610 2024-03-18T18:50:23.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.051+0000 7fae78e4d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae74144f50 con 0x7fae740ccf40 2024-03-18T18:50:23.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.051+0000 7fae71d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fae740af610 0x7fae740cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.051+0000 7fae7259c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fae74144b70 0x7fae740ce980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55920/0 (socket says 172.21.15.1:55920) 2024-03-18T18:50:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.051+0000 7fae7259c700 1 -- 172.21.15.1:0/1543474507 learned_addr learned my addr 172.21.15.1:0/1543474507 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.051+0000 7fae72d9d700 1 --2- 172.21.15.1:0/1543474507 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae740ccf40 0x7fae74071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.052+0000 7fae71d9b700 1 -- 172.21.15.1:0/1543474507 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae740ccf40 msgr2=0x7fae74071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.052+0000 7fae71d9b700 1 --2- 172.21.15.1:0/1543474507 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae740ccf40 0x7fae74071a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.052+0000 7fae71d9b700 1 -- 172.21.15.1:0/1543474507 >> v1:172.21.15.1:6789/0 conn(0x7fae74144b70 legacy=0x7fae740ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.052+0000 7fae71d9b700 1 -- 172.21.15.1:0/1543474507 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae740cf2d0 con 0x7fae740af610 2024-03-18T18:50:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.052+0000 7fae7159a700 1 -- 172.21.15.1:0/1543474507 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239854247 0 0) 0x7fae68001110 con 0x7fae74144b70 2024-03-18T18:50:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.052+0000 7fae72d9d700 1 --2- 172.21.15.1:0/1543474507 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae740ccf40 0x7fae74071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.052+0000 7fae71d9b700 1 --2- 172.21.15.1:0/1543474507 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740af610 0x7fae740cca00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fae64009840 tx=0x7fae64009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d939d19163aead server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.052+0000 7fae7159a700 1 -- 172.21.15.1:0/1543474507 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae64011830 con 0x7fae740af610 2024-03-18T18:50:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.052+0000 7fae7159a700 1 -- 172.21.15.1:0/1543474507 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae64011990 con 0x7fae740af610 2024-03-18T18:50:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.052+0000 7fae7159a700 1 -- 172.21.15.1:0/1543474507 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae64011d20 con 0x7fae740af610 2024-03-18T18:50:23.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.052+0000 7fae78e4d700 1 -- 172.21.15.1:0/1543474507 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740af610 msgr2=0x7fae740cca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:23.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.052+0000 7fae78e4d700 1 --2- 172.21.15.1:0/1543474507 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740af610 0x7fae740cca00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fae64009840 tx=0x7fae64009b50 comp rx=0 tx=0).stop 2024-03-18T18:50:23.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.053+0000 7fae78e4d700 1 -- 172.21.15.1:0/1543474507 shutdown_connections 2024-03-18T18:50:23.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.053+0000 7fae78e4d700 1 --2- 172.21.15.1:0/1543474507 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae740ccf40 0x7fae74071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:23.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.053+0000 7fae78e4d700 1 --2- 172.21.15.1:0/1543474507 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740af610 0x7fae740cca00 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:23.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.053+0000 7fae78e4d700 1 -- 172.21.15.1:0/1543474507 >> 172.21.15.1:0/1543474507 conn(0x7fae740b1d20 msgr2=0x7fae740b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:23.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.053+0000 7fae78e4d700 1 -- 172.21.15.1:0/1543474507 shutdown_connections 2024-03-18T18:50:23.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.053+0000 7fae78e4d700 1 -- 172.21.15.1:0/1543474507 wait complete. 2024-03-18T18:50:23.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.054+0000 7fae78e4d700 1 Processor -- start 2024-03-18T18:50:23.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.054+0000 7fae78e4d700 1 -- start start 2024-03-18T18:50:23.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae78e4d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740af610 0x7fae74154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:23.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae78e4d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae740ccf40 0x7fae7413fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:23.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae78e4d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae74144b70 0x7fae7413ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:23.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae78e4d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae741f32a0 con 0x7fae740af610 2024-03-18T18:50:23.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae78e4d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae74140490 con 0x7fae74144b70 2024-03-18T18:50:23.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae78e4d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae74155000 con 0x7fae740ccf40 2024-03-18T18:50:23.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae7259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740af610 0x7fae74154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:23.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae7259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740af610 0x7fae74154ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52150/0 (socket says 172.21.15.1:52150) 2024-03-18T18:50:23.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae7259c700 1 -- 172.21.15.1:0/1396604028 learned_addr learned my addr 172.21.15.1:0/1396604028 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:23.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae72d9d700 1 --2- 172.21.15.1:0/1396604028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae74144b70 0x7fae7413ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:23.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae71d9b700 1 --2- 172.21.15.1:0/1396604028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae740ccf40 0x7fae7413fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:23.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae7259c700 1 -- 172.21.15.1:0/1396604028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae740ccf40 msgr2=0x7fae7413fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:23.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae7259c700 1 --2- 172.21.15.1:0/1396604028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae740ccf40 0x7fae7413fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:23.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae7259c700 1 -- 172.21.15.1:0/1396604028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae74144b70 msgr2=0x7fae7413ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:23.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae7259c700 1 --2- 172.21.15.1:0/1396604028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae74144b70 0x7fae7413ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:23.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae7259c700 1 -- 172.21.15.1:0/1396604028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae640094a0 con 0x7fae740af610 2024-03-18T18:50:23.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae7259c700 1 --2- 172.21.15.1:0/1396604028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740af610 0x7fae74154ac0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fae6800c550 tx=0x7fae6800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:23.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae62ffd700 1 -- 172.21.15.1:0/1396604028 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae68004c80 con 0x7fae740af610 2024-03-18T18:50:23.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae62ffd700 1 -- 172.21.15.1:0/1396604028 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae68004e00 con 0x7fae740af610 2024-03-18T18:50:23.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae78e4d700 1 -- 172.21.15.1:0/1396604028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae74139860 con 0x7fae740af610 2024-03-18T18:50:23.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae62ffd700 1 -- 172.21.15.1:0/1396604028 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae68007310 con 0x7fae740af610 2024-03-18T18:50:23.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.055+0000 7fae78e4d700 1 -- 172.21.15.1:0/1396604028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae741409c0 con 0x7fae740af610 2024-03-18T18:50:23.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.056+0000 7fae78e4d700 1 -- 172.21.15.1:0/1396604028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae740cda90 con 0x7fae740af610 2024-03-18T18:50:23.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.057+0000 7fae62ffd700 1 -- 172.21.15.1:0/1396604028 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fae68007470 con 0x7fae740af610 2024-03-18T18:50:23.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.057+0000 7fae62ffd700 1 --2- 172.21.15.1:0/1396604028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae5406ac90 0x7fae5406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:23.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.057+0000 7fae62ffd700 1 -- 172.21.15.1:0/1396604028 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fae680892a0 con 0x7fae740af610 2024-03-18T18:50:23.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.057+0000 7fae71d9b700 1 --2- 172.21.15.1:0/1396604028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae5406ac90 0x7fae5406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:23.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.058+0000 7fae71d9b700 1 --2- 172.21.15.1:0/1396604028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae5406ac90 0x7fae5406d150 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fae640090b0 tx=0x7fae64008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:23.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.062+0000 7fae62ffd700 1 -- 172.21.15.1:0/1396604028 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fae68054100 con 0x7fae740af610 2024-03-18T18:50:23.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.310+0000 7fae78e4d700 1 -- 172.21.15.1:0/1396604028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fae740cf090 con 0x7fae740af610 2024-03-18T18:50:23.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.310+0000 7fae62ffd700 1 -- 172.21.15.1:0/1396604028 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7fae740cf090 con 0x7fae740af610 2024-03-18T18:50:23.307 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T18:50:23.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.312+0000 7fae78e4d700 1 -- 172.21.15.1:0/1396604028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae5406ac90 msgr2=0x7fae5406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:23.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.312+0000 7fae78e4d700 1 --2- 172.21.15.1:0/1396604028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae5406ac90 0x7fae5406d150 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fae640090b0 tx=0x7fae64008030 comp rx=0 tx=0).stop 2024-03-18T18:50:23.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.312+0000 7fae78e4d700 1 -- 172.21.15.1:0/1396604028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740af610 msgr2=0x7fae74154ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:23.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.312+0000 7fae78e4d700 1 --2- 172.21.15.1:0/1396604028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740af610 0x7fae74154ac0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fae6800c550 tx=0x7fae6800c910 comp rx=0 tx=0).stop 2024-03-18T18:50:23.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.312+0000 7fae78e4d700 1 -- 172.21.15.1:0/1396604028 shutdown_connections 2024-03-18T18:50:23.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.312+0000 7fae78e4d700 1 --2- 172.21.15.1:0/1396604028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae5406ac90 0x7fae5406d150 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:23.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.312+0000 7fae78e4d700 1 --2- 172.21.15.1:0/1396604028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae74144b70 0x7fae7413ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:23.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.312+0000 7fae78e4d700 1 --2- 172.21.15.1:0/1396604028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740af610 0x7fae74154ac0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:23.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.312+0000 7fae78e4d700 1 --2- 172.21.15.1:0/1396604028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae740ccf40 0x7fae7413fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:23.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.312+0000 7fae78e4d700 1 -- 172.21.15.1:0/1396604028 >> 172.21.15.1:0/1396604028 conn(0x7fae740b1d20 msgr2=0x7fae74143a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:23.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.312+0000 7fae78e4d700 1 -- 172.21.15.1:0/1396604028 shutdown_connections 2024-03-18T18:50:23.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:23.312+0000 7fae78e4d700 1 -- 172.21.15.1:0/1396604028 wait complete. 2024-03-18T18:50:23.320 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T18:50:23.320 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T18:50:23.320 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-18T18:50:23.321 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-18T18:50:23.321 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-18T18:50:23.321 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T18:50:23.321 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T18:50:23.321 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T18:50:23.321 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-18T18:50:23.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:23.321 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-18T18:50:23.340 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T18:50:23.340 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T18:50:23.340 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T18:50:23.340 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:23.340 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T18:50:23.340 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:23.342 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T18:50:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-18T18:50:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-18T18:50:23.342 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T18:50:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-18T18:50:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-18T18:50:23.342 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T18:50:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-18T18:50:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-18T18:50:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T18:50:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 42546sec preferred_lft 42546sec 2024-03-18T18:50:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link 2024-03-18T18:50:23.342 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:23.343 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T18:50:23.343 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-18T18:50:23.343 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-18T18:50:23.343 INFO:teuthology.orchestra.run.smithi001.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T18:50:23.343 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff 2024-03-18T18:50:23.343 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T18:50:23.343 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:23.343 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::4ca9:f5ff:feb3:79a3/64 scope link 2024-03-18T18:50:23.343 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:23.343 INFO:teuthology.orchestra.run.smithi001.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T18:50:23.343 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T18:50:23.343 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::282f:a4ff:fe1a:9908/64 scope link tentative 2024-03-18T18:50:23.343 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:23.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:23.344 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T18:50:23.406 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-18T18:50:23.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:23.408 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-18T18:50:23.469 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-18T18:50:23.469 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T18:50:23.469 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T18:50:23.469 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T18:50:23.470 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T18:50:23.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:23.470 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T18:50:23.470 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T18:50:23.470 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T18:50:23.470 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T18:50:23.470 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T18:50:23.470 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T18:50:23.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:23.576 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T18:50:23.594 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T18:50:23.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:23.595 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list-id 2024-03-18T18:50:23.652 INFO:teuthology.orchestra.run.smithi001.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-18T18:50:23.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:23.653 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T18:50:23.653 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T18:50:23.653 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T18:50:23.653 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-18T18:50:23.653 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T18:50:23.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:23.748 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-18T18:50:23.791 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T18:50:23.792 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T18:50:23.792 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T18:50:23.792 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:23.792 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T18:50:23.792 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:23.792 INFO:teuthology.orchestra.run.smithi001.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T18:50:23.792 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 3a:d8:eb:28:e2:14 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-18T18:50:23.792 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-18T18:50:23.792 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:23.792 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::38d8:ebff:fe28:e214/64 scope link 2024-03-18T18:50:23.792 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:23.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:23.801 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-18T18:50:23.878 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T18:50:23.878 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T18:50:23.878 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T18:50:23.878 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:23.878 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T18:50:23.879 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:23.879 INFO:teuthology.orchestra.run.smithi001.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T18:50:23.879 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 3a:d8:eb:28:e2:14 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-18T18:50:23.879 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-18T18:50:23.879 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:23.879 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::38d8:ebff:fe28:e214/64 scope link 2024-03-18T18:50:23.879 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:23.891 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-18T18:50:23.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:23.891 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T18:50:23.891 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T18:50:23.891 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-18T18:50:23.891 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-18T18:50:23.891 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-18T18:50:23.891 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-18T18:50:23.891 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-18T18:50:23.892 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T18:50:24.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:24.129 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T18:50:24.129 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T18:50:24.129 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.1 up 2024-03-18T18:50:24.129 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set dev brx.1 master ceph-brx 2024-03-18T18:50:24.129 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T18:50:24.227 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-03-18T18:50:24.228 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T18:50:24.228 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-18T18:50:24.245 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-18T18:50:24.246 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T18:50:24.246 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-18T18:50:24.304 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T18:50:24.372 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T18:50:24.372 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T18:50:24.440 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T18:50:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T18:50:24.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:24.442 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T18:50:24.500 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T18:50:24.501 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [47] 2024-03-18T18:50:24.501 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-03-18T18:50:24.543 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T18:50:24.571 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T18:50:24.571 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T18:50:24.631 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi001.stderr:2024-03-18T18:50:24.635+0000 7eff9deae5c0 -1 init, newargv = 0x556ad8f68e10 newargc=15 2024-03-18T18:50:24.631 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi001.stderr:ceph-fuse[89320]: starting ceph client 2024-03-18T18:50:24.636 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi001.stderr:ceph-fuse[89320]: starting fuse 2024-03-18T18:50:24.640 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T18:50:24.641 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T18:50:24.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:24.641 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T18:50:24.699 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T18:50:24.699 INFO:teuthology.orchestra.run.smithi001.stdout:48 2024-03-18T18:50:24.700 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47, 48] 2024-03-18T18:50:24.700 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T18:50:24.700 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-18T18:50:24.700 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-18T18:50:24.700 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-18T18:50:24.700 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-18T18:50:24.700 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T18:50:24.700 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-18T18:50:24.700 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-03-18T18:50:24.700 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-18T18:50:24.700 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-03-18T18:50:24.700 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T18:50:24.701 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-18T18:50:24.701 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-18T18:50:24.701 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-18T18:50:24.701 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-18T18:50:24.701 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T18:50:24.701 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-18T18:50:24.701 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T18:50:24.701 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T18:50:24.701 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T18:50:24.701 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-18T18:50:24.701 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-18T18:50:24.701 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-18T18:50:24.701 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T18:50:24.701 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T18:50:24.701 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.2")) 2024-03-18T18:50:24.702 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-18T18:50:24.824 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.2.89320.asok 2024-03-18T18:50:25.007 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.89320.asok 2024-03-18T18:50:25.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:25.007 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.89320.asok status 2024-03-18T18:50:25.169 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T18:50:25.169 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-18T18:50:25.170 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T18:50:25.170 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T18:50:25.170 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "2", 2024-03-18T18:50:25.170 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-18T18:50:25.170 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-03-18T18:50:25.170 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "89320", 2024-03-18T18:50:25.171 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-18T18:50:25.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:50:25.171 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-18T18:50:25.171 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-18T18:50:25.171 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 4830, 2024-03-18T18:50:25.172 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-18T18:50:25.172 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-18T18:50:25.172 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-18T18:50:25.172 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4830 2024-03-18T18:50:25.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:50:25.173 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T18:50:25.173 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T18:50:25.173 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.2:0", 2024-03-18T18:50:25.173 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4155669794 2024-03-18T18:50:25.173 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:50:25.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:50:25.173 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T18:50:25.173 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T18:50:25.173 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.2:0", 2024-03-18T18:50:25.174 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4155669794 2024-03-18T18:50:25.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:50:25.174 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.4830 192.168.0.2:0/4155669794", 2024-03-18T18:50:25.174 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.2:0/4155669794", 2024-03-18T18:50:25.174 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-18T18:50:25.174 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 9, 2024-03-18T18:50:25.174 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 25, 2024-03-18T18:50:25.174 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-18T18:50:25.174 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-18T18:50:25.174 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-18T18:50:25.174 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T18:50:25.179 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T18:50:25.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.307+0000 7f90e9534700 1 Processor -- start 2024-03-18T18:50:25.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.307+0000 7f90e9534700 1 -- start start 2024-03-18T18:50:25.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.308+0000 7f90e9534700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f90e40d1ea0 0x7f90e4168560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:25.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.308+0000 7f90e9534700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e40cddc0 con 0x7f90e40cf090 2024-03-18T18:50:25.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.308+0000 7f90e9534700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e40cdff0 con 0x7f90e4160940 2024-03-18T18:50:25.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.308+0000 7f90e9534700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90e40ce2d0 con 0x7f90e40d1ea0 2024-03-18T18:50:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.308+0000 7f90e27fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f90e40cf090 0x7f90e40d1790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59486/0 (socket says 172.21.15.1:59486) 2024-03-18T18:50:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.308+0000 7f90e27fc700 1 -- 172.21.15.1:0/1561723233 learned_addr learned my addr 172.21.15.1:0/1561723233 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.308+0000 7f90e37fe700 1 --2- 172.21.15.1:0/1561723233 >> v2:172.21.15.165:3300/0 conn(0x7f90e40d1ea0 0x7f90e4168560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.308+0000 7f90e1ffb700 1 -- 172.21.15.1:0/1561723233 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 905109305 0 0) 0x7f90e40cddc0 con 0x7f90e40cf090 2024-03-18T18:50:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.308+0000 7f90e1ffb700 1 -- 172.21.15.1:0/1561723233 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90cc0035b0 con 0x7f90e40cf090 2024-03-18T18:50:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e1ffb700 1 -- 172.21.15.1:0/1561723233 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3460232349 0 0) 0x7f90cc0035b0 con 0x7f90e40cf090 2024-03-18T18:50:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e1ffb700 1 -- 172.21.15.1:0/1561723233 >> v2:172.21.15.165:3300/0 conn(0x7f90e40d1ea0 msgr2=0x7f90e4168560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e1ffb700 1 --2- 172.21.15.1:0/1561723233 >> v2:172.21.15.165:3300/0 conn(0x7f90e40d1ea0 0x7f90e4168560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e1ffb700 1 -- 172.21.15.1:0/1561723233 >> v1:172.21.15.165:6789/0 conn(0x7f90e4160940 legacy=0x7f90e40d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e1ffb700 1 -- 172.21.15.1:0/1561723233 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e40ce5b0 con 0x7f90e40cf090 2024-03-18T18:50:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e1ffb700 1 -- 172.21.15.1:0/1561723233 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f90d4001670 con 0x7f90e40cf090 2024-03-18T18:50:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e1ffb700 1 -- 172.21.15.1:0/1561723233 >> v1:172.21.15.1:6789/0 conn(0x7f90e40cf090 legacy=0x7f90e40d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e1ffb700 1 --2- 172.21.15.1:0/1561723233 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90cc003a00 0x7f90cc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e1ffb700 1 --2- 172.21.15.1:0/1561723233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90cc004850 0x7f90cc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:25.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e1ffb700 1 --2- 172.21.15.1:0/1561723233 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90cc005210 0x7f90cc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:25.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e1ffb700 1 -- 172.21.15.1:0/1561723233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e40ce2d0 con 0x7f90cc004850 2024-03-18T18:50:25.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e1ffb700 1 -- 172.21.15.1:0/1561723233 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc00c0b0 con 0x7f90cc005210 2024-03-18T18:50:25.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e1ffb700 1 -- 172.21.15.1:0/1561723233 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90cc00c270 con 0x7f90cc003a00 2024-03-18T18:50:25.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e27fc700 1 --2- 172.21.15.1:0/1561723233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90cc004850 0x7f90cc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:25.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e37fe700 1 --2- 172.21.15.1:0/1561723233 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90cc005210 0x7f90cc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:25.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e2ffd700 1 --2- 172.21.15.1:0/1561723233 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90cc003a00 0x7f90cc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:25.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e27fc700 1 -- 172.21.15.1:0/1561723233 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90cc003a00 msgr2=0x7f90cc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:25.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e27fc700 1 --2- 172.21.15.1:0/1561723233 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90cc003a00 0x7f90cc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:25.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e27fc700 1 -- 172.21.15.1:0/1561723233 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90cc005210 msgr2=0x7f90cc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:25.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e27fc700 1 --2- 172.21.15.1:0/1561723233 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90cc005210 0x7f90cc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:25.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e37fe700 1 -- 172.21.15.1:0/1561723233 reap_dead start 2024-03-18T18:50:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e27fc700 1 -- 172.21.15.1:0/1561723233 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc0035b0 con 0x7f90cc004850 2024-03-18T18:50:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.309+0000 7f90e27fc700 1 --2- 172.21.15.1:0/1561723233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90cc004850 0x7f90cc004cd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f90d400c210 tx=0x7f90cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67706e646eff8c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.310+0000 7f90e1ffb700 1 -- 172.21.15.1:0/1561723233 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d4007c10 con 0x7f90cc004850 2024-03-18T18:50:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.310+0000 7f90e1ffb700 1 -- 172.21.15.1:0/1561723233 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d400cd50 con 0x7f90cc004850 2024-03-18T18:50:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.310+0000 7f90e1ffb700 1 -- 172.21.15.1:0/1561723233 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d4005ba0 con 0x7f90cc004850 2024-03-18T18:50:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.310+0000 7f90e9534700 1 -- 172.21.15.1:0/1561723233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90cc004850 msgr2=0x7f90cc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.310+0000 7f90e9534700 1 --2- 172.21.15.1:0/1561723233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90cc004850 0x7f90cc004cd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f90d400c210 tx=0x7f90cc005880 comp rx=0 tx=0).stop 2024-03-18T18:50:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.310+0000 7f90e9534700 1 -- 172.21.15.1:0/1561723233 shutdown_connections 2024-03-18T18:50:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.310+0000 7f90e9534700 1 --2- 172.21.15.1:0/1561723233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90cc004850 0x7f90cc004cd0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.310+0000 7f90e9534700 1 -- 172.21.15.1:0/1561723233 >> 172.21.15.1:0/1561723233 conn(0x7f90e409c520 msgr2=0x7f90e40d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.310+0000 7f90e9534700 1 -- 172.21.15.1:0/1561723233 shutdown_connections 2024-03-18T18:50:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.310+0000 7f90e9534700 1 -- 172.21.15.1:0/1561723233 wait complete. 2024-03-18T18:50:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.311+0000 7f90e9534700 1 Processor -- start 2024-03-18T18:50:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.312+0000 7f90e9534700 1 -- start start 2024-03-18T18:50:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.312+0000 7f90e9534700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90cc004850 0x7f90e40b5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.312+0000 7f90e9534700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90e4146d40 0x7f90e40b7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.312+0000 7f90e9534700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90e40b7d90 0x7f90e40b8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.312+0000 7f90e9534700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e40b1490 con 0x7f90cc004850 2024-03-18T18:50:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.312+0000 7f90e9534700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e40b08c0 con 0x7f90e4146d40 2024-03-18T18:50:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.312+0000 7f90e9534700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90e40b8760 con 0x7f90e40b7d90 2024-03-18T18:50:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.312+0000 7f90e2ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90cc004850 0x7f90e40b5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.312+0000 7f90e2ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90cc004850 0x7f90e40b5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47844/0 (socket says 172.21.15.1:47844) 2024-03-18T18:50:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.312+0000 7f90e2ffd700 1 -- 172.21.15.1:0/4020964140 learned_addr learned my addr 172.21.15.1:0/4020964140 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.313+0000 7f90e37fe700 1 --2- 172.21.15.1:0/4020964140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90e40b7d90 0x7f90e40b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.313+0000 7f90e27fc700 1 --2- 172.21.15.1:0/4020964140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90e4146d40 0x7f90e40b7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.313+0000 7f90e2ffd700 1 -- 172.21.15.1:0/4020964140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90e40b7d90 msgr2=0x7f90e40b8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.313+0000 7f90e2ffd700 1 --2- 172.21.15.1:0/4020964140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90e40b7d90 0x7f90e40b8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.313+0000 7f90e2ffd700 1 -- 172.21.15.1:0/4020964140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90e4146d40 msgr2=0x7f90e40b7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.313+0000 7f90e2ffd700 1 --2- 172.21.15.1:0/4020964140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90e4146d40 0x7f90e40b7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.313+0000 7f90e2ffd700 1 -- 172.21.15.1:0/4020964140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e40b0b60 con 0x7f90cc004850 2024-03-18T18:50:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.313+0000 7f90e37fe700 1 --2- 172.21.15.1:0/4020964140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90e40b7d90 0x7f90e40b8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.313+0000 7f90e27fc700 1 --2- 172.21.15.1:0/4020964140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90e4146d40 0x7f90e40b7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.313+0000 7f90e2ffd700 1 --2- 172.21.15.1:0/4020964140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90cc004850 0x7f90e40b5dd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f90d800f820 tx=0x7f90d800fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.313+0000 7f90cb7fe700 1 -- 172.21.15.1:0/4020964140 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d80052e0 con 0x7f90cc004850 2024-03-18T18:50:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.313+0000 7f90cb7fe700 1 -- 172.21.15.1:0/4020964140 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d8005460 con 0x7f90cc004850 2024-03-18T18:50:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.313+0000 7f90e9534700 1 -- 172.21.15.1:0/4020964140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90e40b0e40 con 0x7f90cc004850 2024-03-18T18:50:25.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.313+0000 7f90cb7fe700 1 -- 172.21.15.1:0/4020964140 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d80038c0 con 0x7f90cc004850 2024-03-18T18:50:25.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.313+0000 7f90e9534700 1 -- 172.21.15.1:0/4020964140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e40b6310 con 0x7f90cc004850 2024-03-18T18:50:25.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.314+0000 7f90e9534700 1 -- 172.21.15.1:0/4020964140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90e4073350 con 0x7f90cc004850 2024-03-18T18:50:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.315+0000 7f90cb7fe700 1 -- 172.21.15.1:0/4020964140 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f90d8003a20 con 0x7f90cc004850 2024-03-18T18:50:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.315+0000 7f90cb7fe700 1 --2- 172.21.15.1:0/4020964140 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f90d006ac90 0x7f90d006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.315+0000 7f90cb7fe700 1 -- 172.21.15.1:0/4020964140 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f90d8089330 con 0x7f90cc004850 2024-03-18T18:50:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.315+0000 7f90e27fc700 1 --2- 172.21.15.1:0/4020964140 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f90d006ac90 0x7f90d006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.315+0000 7f90e27fc700 1 --2- 172.21.15.1:0/4020964140 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f90d006ac90 0x7f90d006d150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f90d4015040 tx=0x7f90d4007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.320+0000 7f90cb7fe700 1 -- 172.21.15.1:0/4020964140 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f90d80542b0 con 0x7f90cc004850 2024-03-18T18:50:25.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.567+0000 7f90e9534700 1 -- 172.21.15.1:0/4020964140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f90e40d0980 con 0x7f90cc004850 2024-03-18T18:50:25.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.567+0000 7f90cb7fe700 1 -- 172.21.15.1:0/4020964140 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f90d80578d0 con 0x7f90cc004850 2024-03-18T18:50:25.564 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T18:50:25.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.569+0000 7f90e9534700 1 -- 172.21.15.1:0/4020964140 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f90d006ac90 msgr2=0x7f90d006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:25.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.569+0000 7f90e9534700 1 --2- 172.21.15.1:0/4020964140 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f90d006ac90 0x7f90d006d150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f90d4015040 tx=0x7f90d4007000 comp rx=0 tx=0).stop 2024-03-18T18:50:25.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.569+0000 7f90e9534700 1 -- 172.21.15.1:0/4020964140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90cc004850 msgr2=0x7f90e40b5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:25.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.569+0000 7f90e9534700 1 --2- 172.21.15.1:0/4020964140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90cc004850 0x7f90e40b5dd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f90d800f820 tx=0x7f90d800fbe0 comp rx=0 tx=0).stop 2024-03-18T18:50:25.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.569+0000 7f90e9534700 1 -- 172.21.15.1:0/4020964140 shutdown_connections 2024-03-18T18:50:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.569+0000 7f90e9534700 1 --2- 172.21.15.1:0/4020964140 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f90d006ac90 0x7f90d006d150 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.569+0000 7f90e9534700 1 --2- 172.21.15.1:0/4020964140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90e40b7d90 0x7f90e40b8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.569+0000 7f90e9534700 1 --2- 172.21.15.1:0/4020964140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90cc004850 0x7f90e40b5dd0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.569+0000 7f90e9534700 1 --2- 172.21.15.1:0/4020964140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90e4146d40 0x7f90e40b7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.569+0000 7f90e9534700 1 -- 172.21.15.1:0/4020964140 >> 172.21.15.1:0/4020964140 conn(0x7f90e409c520 msgr2=0x7f90e406bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.570+0000 7f90e9534700 1 -- 172.21.15.1:0/4020964140 shutdown_connections 2024-03-18T18:50:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:25.570+0000 7f90e9534700 1 -- 172.21.15.1:0/4020964140 wait complete. 2024-03-18T18:50:25.578 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T18:50:25.578 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T18:50:25.578 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi165.front.sepia.ceph.com 2024-03-18T18:50:25.578 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-18T18:50:25.578 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-18T18:50:25.578 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T18:50:25.578 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T18:50:25.578 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T18:50:25.578 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-18T18:50:25.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:25.579 DEBUG:teuthology.orchestra.run.smithi165:> ip addr 2024-03-18T18:50:25.602 INFO:teuthology.orchestra.run.smithi165.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T18:50:25.602 INFO:teuthology.orchestra.run.smithi165.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T18:50:25.603 INFO:teuthology.orchestra.run.smithi165.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T18:50:25.605 INFO:teuthology.orchestra.run.smithi165.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: inet6 ::1/128 scope host 2024-03-18T18:50:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:25.606 INFO:teuthology.orchestra.run.smithi165.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T18:50:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: link/ether 0c:c4:7a:d9:3a:94 brd ff:ff:ff:ff:ff:ff 2024-03-18T18:50:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: altname enp7s0f0 2024-03-18T18:50:25.606 INFO:teuthology.orchestra.run.smithi165.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T18:50:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: link/ether 0c:c4:7a:d9:3a:95 brd ff:ff:ff:ff:ff:ff 2024-03-18T18:50:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: altname enp7s0f1 2024-03-18T18:50:25.606 INFO:teuthology.orchestra.run.smithi165.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T18:50:25.606 INFO:teuthology.orchestra.run.smithi165.stdout: link/ether 0c:c4:7a:88:7f:62 brd ff:ff:ff:ff:ff:ff 2024-03-18T18:50:25.606 INFO:teuthology.orchestra.run.smithi165.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T18:50:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: link/ether 0c:c4:7a:88:7f:63 brd ff:ff:ff:ff:ff:ff 2024-03-18T18:50:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: inet 172.21.15.165/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-18T18:50:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: valid_lft 42536sec preferred_lft 42536sec 2024-03-18T18:50:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: inet6 fe80::ec4:7aff:fe88:7f63/64 scope link 2024-03-18T18:50:25.607 INFO:teuthology.orchestra.run.smithi165.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:25.607 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T18:50:25.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:25.607 DEBUG:teuthology.orchestra.run.smithi165:> (cd / && exec bash -c ' 2024-03-18T18:50:25.607 DEBUG:teuthology.orchestra.run.smithi165:> set -e 2024-03-18T18:50:25.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip link add name ceph-brx type bridge 2024-03-18T18:50:25.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip addr flush dev ceph-brx 2024-03-18T18:50:25.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip link set ceph-brx up 2024-03-18T18:50:25.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T18:50:25.608 DEBUG:teuthology.orchestra.run.smithi165:> ') 2024-03-18T18:50:25.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:25.782 DEBUG:teuthology.orchestra.run.smithi165:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T18:50:25.811 INFO:teuthology.orchestra.run.smithi165.stdout:1 2024-03-18T18:50:25.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:25.812 DEBUG:teuthology.orchestra.run.smithi165:> route 2024-03-18T18:50:25.897 INFO:teuthology.orchestra.run.smithi165.stdout:Kernel IP routing table 2024-03-18T18:50:25.898 INFO:teuthology.orchestra.run.smithi165.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T18:50:25.898 INFO:teuthology.orchestra.run.smithi165.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-18T18:50:25.898 INFO:teuthology.orchestra.run.smithi165.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-18T18:50:25.898 INFO:teuthology.orchestra.run.smithi165.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T18:50:25.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:25.899 DEBUG:teuthology.orchestra.run.smithi165:> (cd / && exec bash -c ' 2024-03-18T18:50:25.899 DEBUG:teuthology.orchestra.run.smithi165:> set -e 2024-03-18T18:50:25.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-18T18:50:25.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-18T18:50:25.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-18T18:50:25.899 DEBUG:teuthology.orchestra.run.smithi165:> ') 2024-03-18T18:50:26.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:26.105 DEBUG:teuthology.orchestra.run.smithi165:> ip netns list 2024-03-18T18:50:26.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:26.123 DEBUG:teuthology.orchestra.run.smithi165:> ip netns list-id 2024-03-18T18:50:26.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:26.182 DEBUG:teuthology.orchestra.run.smithi165:> (cd / && exec bash -c ' 2024-03-18T18:50:26.182 DEBUG:teuthology.orchestra.run.smithi165:> set -e 2024-03-18T18:50:26.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-18T18:50:26.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-18T18:50:26.182 DEBUG:teuthology.orchestra.run.smithi165:> ') 2024-03-18T18:50:26.276 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-18T18:50:26.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:26.276 DEBUG:teuthology.orchestra.run.smithi165:> (cd / && exec bash -c ' 2024-03-18T18:50:26.276 DEBUG:teuthology.orchestra.run.smithi165:> set -e 2024-03-18T18:50:26.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-18T18:50:26.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T18:50:26.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-18T18:50:26.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-18T18:50:26.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-18T18:50:26.277 DEBUG:teuthology.orchestra.run.smithi165:> ') 2024-03-18T18:50:26.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:26.472 DEBUG:teuthology.orchestra.run.smithi165:> (cd / && exec bash -c ' 2024-03-18T18:50:26.473 DEBUG:teuthology.orchestra.run.smithi165:> set -e 2024-03-18T18:50:26.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip link set brx.0 up 2024-03-18T18:50:26.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T18:50:26.473 DEBUG:teuthology.orchestra.run.smithi165:> ') 2024-03-18T18:50:26.569 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-18T18:50:26.569 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T18:50:26.569 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-18T18:50:26.587 INFO:teuthology.orchestra.run.smithi165.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-18T18:50:26.587 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T18:50:26.587 DEBUG:teuthology.orchestra.run.smithi165:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-18T18:50:26.645 DEBUG:teuthology.orchestra.run.smithi165:> sudo modprobe fuse 2024-03-18T18:50:26.713 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T18:50:26.713 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T18:50:26.780 INFO:teuthology.orchestra.run.smithi165.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T18:50:26.781 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T18:50:26.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:26.782 DEBUG:teuthology.orchestra.run.smithi165:> ls /sys/fs/fuse/connections 2024-03-18T18:50:26.840 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T18:50:26.840 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-18T18:50:26.883 DEBUG:teuthology.orchestra.run.smithi165:> sudo modprobe fuse 2024-03-18T18:50:26.910 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T18:50:26.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T18:50:26.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi165.stderr:2024-03-18T18:50:26.972+0000 7fc90920d5c0 -1 init, newargv = 0x558f38d06ee0 newargc=15 2024-03-18T18:50:26.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi165.stderr:ceph-fuse[83778]: starting ceph client 2024-03-18T18:50:26.977 INFO:teuthology.orchestra.run.smithi165.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T18:50:26.978 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T18:50:26.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:26.979 DEBUG:teuthology.orchestra.run.smithi165:> ls /sys/fs/fuse/connections 2024-03-18T18:50:26.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi165.stderr:ceph-fuse[83778]: starting fuse 2024-03-18T18:50:26.997 INFO:teuthology.orchestra.run.smithi165.stdout:47 2024-03-18T18:50:26.998 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-03-18T18:50:26.998 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T18:50:26.998 DEBUG:teuthology.orchestra.run.smithi165:> import glob 2024-03-18T18:50:26.998 DEBUG:teuthology.orchestra.run.smithi165:> import re 2024-03-18T18:50:26.998 DEBUG:teuthology.orchestra.run.smithi165:> import os 2024-03-18T18:50:26.998 DEBUG:teuthology.orchestra.run.smithi165:> import subprocess 2024-03-18T18:50:26.998 DEBUG:teuthology.orchestra.run.smithi165:> 2024-03-18T18:50:26.998 DEBUG:teuthology.orchestra.run.smithi165:> def _find_admin_socket(client_name): 2024-03-18T18:50:26.999 DEBUG:teuthology.orchestra.run.smithi165:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-18T18:50:26.999 DEBUG:teuthology.orchestra.run.smithi165:> files = glob.glob(asok_path) 2024-03-18T18:50:26.999 DEBUG:teuthology.orchestra.run.smithi165:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-18T18:50:26.999 DEBUG:teuthology.orchestra.run.smithi165:> 2024-03-18T18:50:26.999 DEBUG:teuthology.orchestra.run.smithi165:> # Given a non-glob path, it better be there 2024-03-18T18:50:26.999 DEBUG:teuthology.orchestra.run.smithi165:> if "*" not in asok_path: 2024-03-18T18:50:26.999 DEBUG:teuthology.orchestra.run.smithi165:> assert(len(files) == 1) 2024-03-18T18:50:26.999 DEBUG:teuthology.orchestra.run.smithi165:> return files[0] 2024-03-18T18:50:26.999 DEBUG:teuthology.orchestra.run.smithi165:> 2024-03-18T18:50:26.999 DEBUG:teuthology.orchestra.run.smithi165:> for f in files: 2024-03-18T18:50:26.999 DEBUG:teuthology.orchestra.run.smithi165:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T18:50:26.999 DEBUG:teuthology.orchestra.run.smithi165:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T18:50:26.999 DEBUG:teuthology.orchestra.run.smithi165:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T18:50:26.999 DEBUG:teuthology.orchestra.run.smithi165:> contents = proc_f.read() 2024-03-18T18:50:26.999 DEBUG:teuthology.orchestra.run.smithi165:> if mountpoint in contents: 2024-03-18T18:50:26.999 DEBUG:teuthology.orchestra.run.smithi165:> return f 2024-03-18T18:50:27.000 DEBUG:teuthology.orchestra.run.smithi165:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T18:50:27.000 DEBUG:teuthology.orchestra.run.smithi165:> 2024-03-18T18:50:27.000 DEBUG:teuthology.orchestra.run.smithi165:> print(_find_admin_socket("client.1")) 2024-03-18T18:50:27.000 DEBUG:teuthology.orchestra.run.smithi165:> ' 2024-03-18T18:50:27.120 INFO:teuthology.orchestra.run.smithi165.stdout:/var/run/ceph/ceph-client.1.83778.asok 2024-03-18T18:50:27.303 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.83778.asok 2024-03-18T18:50:27.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:27.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.83778.asok status 2024-03-18T18:50:27.461 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-03-18T18:50:27.461 INFO:teuthology.orchestra.run.smithi165.stdout: "metadata": { 2024-03-18T18:50:27.461 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T18:50:27.461 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T18:50:27.462 INFO:teuthology.orchestra.run.smithi165.stdout: "entity_id": "1", 2024-03-18T18:50:27.462 INFO:teuthology.orchestra.run.smithi165.stdout: "hostname": "smithi165", 2024-03-18T18:50:27.462 INFO:teuthology.orchestra.run.smithi165.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-18T18:50:27.462 INFO:teuthology.orchestra.run.smithi165.stdout: "pid": "83778", 2024-03-18T18:50:27.462 INFO:teuthology.orchestra.run.smithi165.stdout: "root": "/" 2024-03-18T18:50:27.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-18T18:50:27.462 INFO:teuthology.orchestra.run.smithi165.stdout: "dentry_count": 0, 2024-03-18T18:50:27.462 INFO:teuthology.orchestra.run.smithi165.stdout: "dentry_pinned_count": 0, 2024-03-18T18:50:27.462 INFO:teuthology.orchestra.run.smithi165.stdout: "id": 4804, 2024-03-18T18:50:27.462 INFO:teuthology.orchestra.run.smithi165.stdout: "inst": { 2024-03-18T18:50:27.462 INFO:teuthology.orchestra.run.smithi165.stdout: "name": { 2024-03-18T18:50:27.462 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "client", 2024-03-18T18:50:27.462 INFO:teuthology.orchestra.run.smithi165.stdout: "num": 4804 2024-03-18T18:50:27.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-18T18:50:27.463 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": { 2024-03-18T18:50:27.463 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-03-18T18:50:27.463 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "192.168.0.1:0", 2024-03-18T18:50:27.463 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 1175830858 2024-03-18T18:50:27.463 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-18T18:50:27.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-18T18:50:27.463 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": { 2024-03-18T18:50:27.463 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-03-18T18:50:27.463 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "192.168.0.1:0", 2024-03-18T18:50:27.463 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 1175830858 2024-03-18T18:50:27.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-18T18:50:27.463 INFO:teuthology.orchestra.run.smithi165.stdout: "inst_str": "client.4804 192.168.0.1:0/1175830858", 2024-03-18T18:50:27.463 INFO:teuthology.orchestra.run.smithi165.stdout: "addr_str": "192.168.0.1:0/1175830858", 2024-03-18T18:50:27.464 INFO:teuthology.orchestra.run.smithi165.stdout: "inode_count": 1, 2024-03-18T18:50:27.464 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_epoch": 9, 2024-03-18T18:50:27.464 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_epoch": 25, 2024-03-18T18:50:27.464 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_epoch_barrier": 0, 2024-03-18T18:50:27.464 INFO:teuthology.orchestra.run.smithi165.stdout: "blocklisted": false, 2024-03-18T18:50:27.464 INFO:teuthology.orchestra.run.smithi165.stdout: "fs_name": "cephfs" 2024-03-18T18:50:27.464 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-03-18T18:50:27.471 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T18:50:27.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.614+0000 7f16dcee0700 1 Processor -- start 2024-03-18T18:50:27.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.614+0000 7f16dcee0700 1 -- start start 2024-03-18T18:50:27.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.615+0000 7f16dcee0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f16d806e8f0 0x7f16d8071440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:27.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.615+0000 7f16dcee0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f16d806ce90 0x7f16d806d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.615+0000 7f16dcee0700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d80cd5a0 con 0x7f16d815d620 2024-03-18T18:50:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.615+0000 7f16dcee0700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16d815da00 con 0x7f16d806ce90 2024-03-18T18:50:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.615+0000 7f16dcee0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d815e5f0 con 0x7f16d806e8f0 2024-03-18T18:50:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.615+0000 7f16d6d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f16d806ce90 0x7f16d806d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.615+0000 7f16d659c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f16d815d620 0x7f16d8070d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59500/0 (socket says 172.21.15.1:59500) 2024-03-18T18:50:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.615+0000 7f16d659c700 1 -- 172.21.15.1:0/3804602228 learned_addr learned my addr 172.21.15.1:0/3804602228 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.615+0000 7f16d6d9d700 1 -- 172.21.15.1:0/3804602228 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f16d806e8f0 msgr2=0x7f16d8071440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.615+0000 7f16d6d9d700 1 --2- 172.21.15.1:0/3804602228 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f16d806e8f0 0x7f16d8071440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.615+0000 7f16d6d9d700 1 -- 172.21.15.1:0/3804602228 >> v1:172.21.15.1:6789/0 conn(0x7f16d815d620 legacy=0x7f16d8070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.615+0000 7f16d6d9d700 1 -- 172.21.15.1:0/3804602228 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d815e8d0 con 0x7f16d806ce90 2024-03-18T18:50:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.615+0000 7f16d559a700 1 -- 172.21.15.1:0/3804602228 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887890316 0 0) 0x7f16d80cd5a0 con 0x7f16d815d620 2024-03-18T18:50:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.615+0000 7f16d6d9d700 1 --2- 172.21.15.1:0/3804602228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d806ce90 0x7f16d806d2b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f16d0009d90 tx=0x7f16d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7df662c97dc9817 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:27.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.615+0000 7f16d559a700 1 -- 172.21.15.1:0/3804602228 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d000e070 con 0x7f16d806ce90 2024-03-18T18:50:27.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.615+0000 7f16d559a700 1 -- 172.21.15.1:0/3804602228 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d000bbd0 con 0x7f16d806ce90 2024-03-18T18:50:27.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.615+0000 7f16d559a700 1 -- 172.21.15.1:0/3804602228 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d0012430 con 0x7f16d806ce90 2024-03-18T18:50:27.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.616+0000 7f16dcee0700 1 -- 172.21.15.1:0/3804602228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d806ce90 msgr2=0x7f16d806d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:27.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.616+0000 7f16dcee0700 1 --2- 172.21.15.1:0/3804602228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d806ce90 0x7f16d806d2b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f16d0009d90 tx=0x7f16d000b330 comp rx=0 tx=0).stop 2024-03-18T18:50:27.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.616+0000 7f16dcee0700 1 -- 172.21.15.1:0/3804602228 shutdown_connections 2024-03-18T18:50:27.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.616+0000 7f16dcee0700 1 --2- 172.21.15.1:0/3804602228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d806ce90 0x7f16d806d2b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:27.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.616+0000 7f16dcee0700 1 --2- 172.21.15.1:0/3804602228 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f16d806e8f0 0x7f16d8071440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:27.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.616+0000 7f16dcee0700 1 -- 172.21.15.1:0/3804602228 >> 172.21.15.1:0/3804602228 conn(0x7f16d80b1270 msgr2=0x7f16d80b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:27.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.616+0000 7f16dcee0700 1 -- 172.21.15.1:0/3804602228 shutdown_connections 2024-03-18T18:50:27.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.616+0000 7f16dcee0700 1 -- 172.21.15.1:0/3804602228 wait complete. 2024-03-18T18:50:27.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.617+0000 7f16dcee0700 1 Processor -- start 2024-03-18T18:50:27.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.618+0000 7f16dcee0700 1 -- start start 2024-03-18T18:50:27.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.618+0000 7f16dcee0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f16d806ce90 0x7f16d814fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:27.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.618+0000 7f16dcee0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d806e8f0 0x7f16d815eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:27.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.618+0000 7f16dcee0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16d815d620 0x7f16d8158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:27.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.618+0000 7f16dcee0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d8160100 con 0x7f16d806e8f0 2024-03-18T18:50:27.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.618+0000 7f16dcee0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d8150180 con 0x7f16d815d620 2024-03-18T18:50:27.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.618+0000 7f16dcee0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d815f480 con 0x7f16d806ce90 2024-03-18T18:50:27.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.618+0000 7f16d5d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d806e8f0 0x7f16d815eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:27.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.618+0000 7f16d5d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d806e8f0 0x7f16d815eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47876/0 (socket says 172.21.15.1:47876) 2024-03-18T18:50:27.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.618+0000 7f16d5d9b700 1 -- 172.21.15.1:0/3161601474 learned_addr learned my addr 172.21.15.1:0/3161601474 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:27.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.619+0000 7f16d6d9d700 1 --2- 172.21.15.1:0/3161601474 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16d815d620 0x7f16d8158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:27.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.619+0000 7f16d5d9b700 1 -- 172.21.15.1:0/3161601474 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f16d806ce90 msgr2=0x7f16d814fc40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T18:50:27.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.619+0000 7f16d5d9b700 1 --2- 172.21.15.1:0/3161601474 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f16d806ce90 0x7f16d814fc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:27.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.619+0000 7f16d5d9b700 1 -- 172.21.15.1:0/3161601474 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16d815d620 msgr2=0x7f16d8158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:27.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.619+0000 7f16d5d9b700 1 --2- 172.21.15.1:0/3161601474 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16d815d620 0x7f16d8158d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:27.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.619+0000 7f16d5d9b700 1 -- 172.21.15.1:0/3161601474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d0009a70 con 0x7f16d806e8f0 2024-03-18T18:50:27.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.619+0000 7f16d6d9d700 1 --2- 172.21.15.1:0/3161601474 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16d815d620 0x7f16d8158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:27.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.619+0000 7f16d5d9b700 1 --2- 172.21.15.1:0/3161601474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d806e8f0 0x7f16d815eb10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f16cc00b530 tx=0x7f16cc00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:27.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.619+0000 7f16c6ffd700 1 -- 172.21.15.1:0/3161601474 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16cc011830 con 0x7f16d806e8f0 2024-03-18T18:50:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.619+0000 7f16c6ffd700 1 -- 172.21.15.1:0/3161601474 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16cc0119b0 con 0x7f16d806e8f0 2024-03-18T18:50:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.619+0000 7f16c6ffd700 1 -- 172.21.15.1:0/3161601474 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16cc011d60 con 0x7f16d806e8f0 2024-03-18T18:50:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.619+0000 7f16dcee0700 1 -- 172.21.15.1:0/3161601474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16d815f760 con 0x7f16d806e8f0 2024-03-18T18:50:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.619+0000 7f16dcee0700 1 -- 172.21.15.1:0/3161601474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16d815fd20 con 0x7f16d806e8f0 2024-03-18T18:50:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.620+0000 7f16dcee0700 1 -- 172.21.15.1:0/3161601474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16d80676a0 con 0x7f16d806e8f0 2024-03-18T18:50:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.621+0000 7f16c6ffd700 1 -- 172.21.15.1:0/3161601474 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f16cc01a400 con 0x7f16d806e8f0 2024-03-18T18:50:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.621+0000 7f16c6ffd700 1 --2- 172.21.15.1:0/3161601474 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f16b806ac90 0x7f16b806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.621+0000 7f16c6ffd700 1 -- 172.21.15.1:0/3161601474 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f16cc089670 con 0x7f16d806e8f0 2024-03-18T18:50:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.621+0000 7f16d659c700 1 --2- 172.21.15.1:0/3161601474 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f16b806ac90 0x7f16b806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:27.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.621+0000 7f16d659c700 1 --2- 172.21.15.1:0/3161601474 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f16b806ac90 0x7f16b806d150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f16c8007fd0 tx=0x7f16c8007940 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:27.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.626+0000 7f16c6ffd700 1 -- 172.21.15.1:0/3161601474 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f16cc0545f0 con 0x7f16d806e8f0 2024-03-18T18:50:27.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.874+0000 7f16dcee0700 1 -- 172.21.15.1:0/3161601474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f16d80cce70 con 0x7f16d806e8f0 2024-03-18T18:50:27.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.874+0000 7f16c6ffd700 1 -- 172.21.15.1:0/3161601474 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f16cc057c10 con 0x7f16d806e8f0 2024-03-18T18:50:27.870 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T18:50:27.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.876+0000 7f16dcee0700 1 -- 172.21.15.1:0/3161601474 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f16b806ac90 msgr2=0x7f16b806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:27.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.876+0000 7f16dcee0700 1 --2- 172.21.15.1:0/3161601474 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f16b806ac90 0x7f16b806d150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f16c8007fd0 tx=0x7f16c8007940 comp rx=0 tx=0).stop 2024-03-18T18:50:27.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.876+0000 7f16dcee0700 1 -- 172.21.15.1:0/3161601474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d806e8f0 msgr2=0x7f16d815eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:27.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.876+0000 7f16dcee0700 1 --2- 172.21.15.1:0/3161601474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d806e8f0 0x7f16d815eb10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f16cc00b530 tx=0x7f16cc00b8f0 comp rx=0 tx=0).stop 2024-03-18T18:50:27.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.876+0000 7f16dcee0700 1 -- 172.21.15.1:0/3161601474 shutdown_connections 2024-03-18T18:50:27.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.876+0000 7f16dcee0700 1 --2- 172.21.15.1:0/3161601474 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f16b806ac90 0x7f16b806d150 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:27.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.876+0000 7f16dcee0700 1 --2- 172.21.15.1:0/3161601474 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f16d806ce90 0x7f16d814fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:27.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.876+0000 7f16dcee0700 1 --2- 172.21.15.1:0/3161601474 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16d815d620 0x7f16d8158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:27.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.876+0000 7f16dcee0700 1 --2- 172.21.15.1:0/3161601474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d806e8f0 0x7f16d815eb10 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:27.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.876+0000 7f16dcee0700 1 -- 172.21.15.1:0/3161601474 >> 172.21.15.1:0/3161601474 conn(0x7f16d80b1270 msgr2=0x7f16d806d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:27.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.876+0000 7f16dcee0700 1 -- 172.21.15.1:0/3161601474 shutdown_connections 2024-03-18T18:50:27.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:27.876+0000 7f16dcee0700 1 -- 172.21.15.1:0/3161601474 wait complete. 2024-03-18T18:50:27.884 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T18:50:27.884 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T18:50:27.884 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi165.front.sepia.ceph.com 2024-03-18T18:50:27.884 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-18T18:50:27.884 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-18T18:50:27.884 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T18:50:27.884 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T18:50:27.885 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T18:50:27.885 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-18T18:50:27.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:27.885 DEBUG:teuthology.orchestra.run.smithi165:> ip addr 2024-03-18T18:50:27.906 INFO:teuthology.orchestra.run.smithi165.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T18:50:27.906 INFO:teuthology.orchestra.run.smithi165.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T18:50:27.906 INFO:teuthology.orchestra.run.smithi165.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T18:50:27.906 INFO:teuthology.orchestra.run.smithi165.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:27.906 INFO:teuthology.orchestra.run.smithi165.stdout: inet6 ::1/128 scope host 2024-03-18T18:50:27.906 INFO:teuthology.orchestra.run.smithi165.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:27.907 INFO:teuthology.orchestra.run.smithi165.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T18:50:27.907 INFO:teuthology.orchestra.run.smithi165.stdout: link/ether 0c:c4:7a:d9:3a:94 brd ff:ff:ff:ff:ff:ff 2024-03-18T18:50:27.907 INFO:teuthology.orchestra.run.smithi165.stdout: altname enp7s0f0 2024-03-18T18:50:27.907 INFO:teuthology.orchestra.run.smithi165.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T18:50:27.907 INFO:teuthology.orchestra.run.smithi165.stdout: link/ether 0c:c4:7a:d9:3a:95 brd ff:ff:ff:ff:ff:ff 2024-03-18T18:50:27.907 INFO:teuthology.orchestra.run.smithi165.stdout: altname enp7s0f1 2024-03-18T18:50:27.907 INFO:teuthology.orchestra.run.smithi165.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T18:50:27.907 INFO:teuthology.orchestra.run.smithi165.stdout: link/ether 0c:c4:7a:88:7f:62 brd ff:ff:ff:ff:ff:ff 2024-03-18T18:50:27.907 INFO:teuthology.orchestra.run.smithi165.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T18:50:27.907 INFO:teuthology.orchestra.run.smithi165.stdout: link/ether 0c:c4:7a:88:7f:63 brd ff:ff:ff:ff:ff:ff 2024-03-18T18:50:27.907 INFO:teuthology.orchestra.run.smithi165.stdout: inet 172.21.15.165/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-18T18:50:27.907 INFO:teuthology.orchestra.run.smithi165.stdout: valid_lft 42534sec preferred_lft 42534sec 2024-03-18T18:50:27.907 INFO:teuthology.orchestra.run.smithi165.stdout: inet6 fe80::ec4:7aff:fe88:7f63/64 scope link 2024-03-18T18:50:27.907 INFO:teuthology.orchestra.run.smithi165.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:27.908 INFO:teuthology.orchestra.run.smithi165.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T18:50:27.908 INFO:teuthology.orchestra.run.smithi165.stdout: link/ether 9a:bb:03:23:f0:c7 brd ff:ff:ff:ff:ff:ff 2024-03-18T18:50:27.908 INFO:teuthology.orchestra.run.smithi165.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T18:50:27.908 INFO:teuthology.orchestra.run.smithi165.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:27.908 INFO:teuthology.orchestra.run.smithi165.stdout: inet6 fe80::c1:6eff:fe77:25c0/64 scope link 2024-03-18T18:50:27.908 INFO:teuthology.orchestra.run.smithi165.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:27.908 INFO:teuthology.orchestra.run.smithi165.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T18:50:27.908 INFO:teuthology.orchestra.run.smithi165.stdout: link/ether 9a:bb:03:23:f0:c7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-18T18:50:27.908 INFO:teuthology.orchestra.run.smithi165.stdout: inet6 fe80::98bb:3ff:fe23:f0c7/64 scope link tentative 2024-03-18T18:50:27.908 INFO:teuthology.orchestra.run.smithi165.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:27.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:27.909 DEBUG:teuthology.orchestra.run.smithi165:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T18:50:27.976 INFO:teuthology.orchestra.run.smithi165.stdout:1 2024-03-18T18:50:27.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:27.978 DEBUG:teuthology.orchestra.run.smithi165:> route 2024-03-18T18:50:28.041 INFO:teuthology.orchestra.run.smithi165.stdout:Kernel IP routing table 2024-03-18T18:50:28.041 INFO:teuthology.orchestra.run.smithi165.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T18:50:28.042 INFO:teuthology.orchestra.run.smithi165.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-18T18:50:28.042 INFO:teuthology.orchestra.run.smithi165.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-18T18:50:28.042 INFO:teuthology.orchestra.run.smithi165.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T18:50:28.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:28.042 DEBUG:teuthology.orchestra.run.smithi165:> (cd / && exec bash -c ' 2024-03-18T18:50:28.042 DEBUG:teuthology.orchestra.run.smithi165:> set -e 2024-03-18T18:50:28.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-18T18:50:28.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-18T18:50:28.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-18T18:50:28.043 DEBUG:teuthology.orchestra.run.smithi165:> ') 2024-03-18T18:50:28.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:28.151 DEBUG:teuthology.orchestra.run.smithi165:> ip netns list 2024-03-18T18:50:28.169 INFO:teuthology.orchestra.run.smithi165.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-18T18:50:28.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:28.170 DEBUG:teuthology.orchestra.run.smithi165:> ip netns list-id 2024-03-18T18:50:28.228 INFO:teuthology.orchestra.run.smithi165.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-18T18:50:28.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:28.228 DEBUG:teuthology.orchestra.run.smithi165:> (cd / && exec bash -c ' 2024-03-18T18:50:28.228 DEBUG:teuthology.orchestra.run.smithi165:> set -e 2024-03-18T18:50:28.229 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-18T18:50:28.229 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-18T18:50:28.229 DEBUG:teuthology.orchestra.run.smithi165:> ') 2024-03-18T18:50:28.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:28.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-18T18:50:28.370 INFO:teuthology.orchestra.run.smithi165.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T18:50:28.370 INFO:teuthology.orchestra.run.smithi165.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T18:50:28.370 INFO:teuthology.orchestra.run.smithi165.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T18:50:28.370 INFO:teuthology.orchestra.run.smithi165.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:28.370 INFO:teuthology.orchestra.run.smithi165.stdout: inet6 ::1/128 scope host 2024-03-18T18:50:28.370 INFO:teuthology.orchestra.run.smithi165.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:28.370 INFO:teuthology.orchestra.run.smithi165.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T18:50:28.370 INFO:teuthology.orchestra.run.smithi165.stdout: link/ether 3a:cd:f0:ae:f8:bb brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-18T18:50:28.370 INFO:teuthology.orchestra.run.smithi165.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-18T18:50:28.370 INFO:teuthology.orchestra.run.smithi165.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:28.371 INFO:teuthology.orchestra.run.smithi165.stdout: inet6 fe80::38cd:f0ff:feae:f8bb/64 scope link tentative 2024-03-18T18:50:28.371 INFO:teuthology.orchestra.run.smithi165.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:28.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:28.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-18T18:50:28.424 INFO:teuthology.orchestra.run.smithi165.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T18:50:28.425 INFO:teuthology.orchestra.run.smithi165.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T18:50:28.425 INFO:teuthology.orchestra.run.smithi165.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T18:50:28.425 INFO:teuthology.orchestra.run.smithi165.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:28.425 INFO:teuthology.orchestra.run.smithi165.stdout: inet6 ::1/128 scope host 2024-03-18T18:50:28.425 INFO:teuthology.orchestra.run.smithi165.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:28.425 INFO:teuthology.orchestra.run.smithi165.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T18:50:28.425 INFO:teuthology.orchestra.run.smithi165.stdout: link/ether 3a:cd:f0:ae:f8:bb brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-18T18:50:28.425 INFO:teuthology.orchestra.run.smithi165.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-18T18:50:28.425 INFO:teuthology.orchestra.run.smithi165.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:28.425 INFO:teuthology.orchestra.run.smithi165.stdout: inet6 fe80::38cd:f0ff:feae:f8bb/64 scope link 2024-03-18T18:50:28.425 INFO:teuthology.orchestra.run.smithi165.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:50:28.446 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-18T18:50:28.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:28.446 DEBUG:teuthology.orchestra.run.smithi165:> (cd / && exec bash -c ' 2024-03-18T18:50:28.446 DEBUG:teuthology.orchestra.run.smithi165:> set -e 2024-03-18T18:50:28.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-18T18:50:28.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-18T18:50:28.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-18T18:50:28.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-18T18:50:28.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-18T18:50:28.447 DEBUG:teuthology.orchestra.run.smithi165:> ') 2024-03-18T18:50:28.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:28.684 DEBUG:teuthology.orchestra.run.smithi165:> (cd / && exec bash -c ' 2024-03-18T18:50:28.684 DEBUG:teuthology.orchestra.run.smithi165:> set -e 2024-03-18T18:50:28.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip link set brx.1 up 2024-03-18T18:50:28.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip link set dev brx.1 master ceph-brx 2024-03-18T18:50:28.685 DEBUG:teuthology.orchestra.run.smithi165:> ') 2024-03-18T18:50:28.781 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-03-18T18:50:28.781 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T18:50:28.781 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-18T18:50:28.799 INFO:teuthology.orchestra.run.smithi165.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-18T18:50:28.799 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T18:50:28.799 DEBUG:teuthology.orchestra.run.smithi165:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-18T18:50:28.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo modprobe fuse 2024-03-18T18:50:28.925 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T18:50:28.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T18:50:28.992 INFO:teuthology.orchestra.run.smithi165.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T18:50:28.993 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T18:50:28.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:28.994 DEBUG:teuthology.orchestra.run.smithi165:> ls /sys/fs/fuse/connections 2024-03-18T18:50:29.051 INFO:teuthology.orchestra.run.smithi165.stdout:47 2024-03-18T18:50:29.052 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [47] 2024-03-18T18:50:29.052 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-03-18T18:50:29.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo modprobe fuse 2024-03-18T18:50:29.120 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T18:50:29.120 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T18:50:29.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi165.stderr:2024-03-18T18:50:29.181+0000 7f428d6bd5c0 -1 init, newargv = 0x560ae5e6ac90 newargc=15 2024-03-18T18:50:29.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi165.stderr:ceph-fuse[84453]: starting ceph client 2024-03-18T18:50:29.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi165.stderr:ceph-fuse[84453]: starting fuse 2024-03-18T18:50:29.188 INFO:teuthology.orchestra.run.smithi165.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T18:50:29.188 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T18:50:29.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:29.189 DEBUG:teuthology.orchestra.run.smithi165:> ls /sys/fs/fuse/connections 2024-03-18T18:50:29.247 INFO:teuthology.orchestra.run.smithi165.stdout:47 2024-03-18T18:50:29.247 INFO:teuthology.orchestra.run.smithi165.stdout:48 2024-03-18T18:50:29.247 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47, 48] 2024-03-18T18:50:29.247 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T18:50:29.248 DEBUG:teuthology.orchestra.run.smithi165:> import glob 2024-03-18T18:50:29.248 DEBUG:teuthology.orchestra.run.smithi165:> import re 2024-03-18T18:50:29.248 DEBUG:teuthology.orchestra.run.smithi165:> import os 2024-03-18T18:50:29.248 DEBUG:teuthology.orchestra.run.smithi165:> import subprocess 2024-03-18T18:50:29.248 DEBUG:teuthology.orchestra.run.smithi165:> 2024-03-18T18:50:29.248 DEBUG:teuthology.orchestra.run.smithi165:> def _find_admin_socket(client_name): 2024-03-18T18:50:29.248 DEBUG:teuthology.orchestra.run.smithi165:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-03-18T18:50:29.248 DEBUG:teuthology.orchestra.run.smithi165:> files = glob.glob(asok_path) 2024-03-18T18:50:29.248 DEBUG:teuthology.orchestra.run.smithi165:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-03-18T18:50:29.248 DEBUG:teuthology.orchestra.run.smithi165:> 2024-03-18T18:50:29.248 DEBUG:teuthology.orchestra.run.smithi165:> # Given a non-glob path, it better be there 2024-03-18T18:50:29.248 DEBUG:teuthology.orchestra.run.smithi165:> if "*" not in asok_path: 2024-03-18T18:50:29.249 DEBUG:teuthology.orchestra.run.smithi165:> assert(len(files) == 1) 2024-03-18T18:50:29.249 DEBUG:teuthology.orchestra.run.smithi165:> return files[0] 2024-03-18T18:50:29.249 DEBUG:teuthology.orchestra.run.smithi165:> 2024-03-18T18:50:29.249 DEBUG:teuthology.orchestra.run.smithi165:> for f in files: 2024-03-18T18:50:29.249 DEBUG:teuthology.orchestra.run.smithi165:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T18:50:29.249 DEBUG:teuthology.orchestra.run.smithi165:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T18:50:29.249 DEBUG:teuthology.orchestra.run.smithi165:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T18:50:29.249 DEBUG:teuthology.orchestra.run.smithi165:> contents = proc_f.read() 2024-03-18T18:50:29.249 DEBUG:teuthology.orchestra.run.smithi165:> if mountpoint in contents: 2024-03-18T18:50:29.249 DEBUG:teuthology.orchestra.run.smithi165:> return f 2024-03-18T18:50:29.249 DEBUG:teuthology.orchestra.run.smithi165:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T18:50:29.249 DEBUG:teuthology.orchestra.run.smithi165:> 2024-03-18T18:50:29.249 DEBUG:teuthology.orchestra.run.smithi165:> print(_find_admin_socket("client.3")) 2024-03-18T18:50:29.250 DEBUG:teuthology.orchestra.run.smithi165:> ' 2024-03-18T18:50:29.370 INFO:teuthology.orchestra.run.smithi165.stdout:/var/run/ceph/ceph-client.3.84453.asok 2024-03-18T18:50:29.553 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.84453.asok 2024-03-18T18:50:29.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:29.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.84453.asok status 2024-03-18T18:50:29.709 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-03-18T18:50:29.709 INFO:teuthology.orchestra.run.smithi165.stdout: "metadata": { 2024-03-18T18:50:29.709 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T18:50:29.709 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T18:50:29.709 INFO:teuthology.orchestra.run.smithi165.stdout: "entity_id": "3", 2024-03-18T18:50:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "hostname": "smithi165", 2024-03-18T18:50:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-03-18T18:50:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "pid": "84453", 2024-03-18T18:50:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "root": "/" 2024-03-18T18:50:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-18T18:50:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "dentry_count": 0, 2024-03-18T18:50:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "dentry_pinned_count": 0, 2024-03-18T18:50:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "id": 4810, 2024-03-18T18:50:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "inst": { 2024-03-18T18:50:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "name": { 2024-03-18T18:50:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "client", 2024-03-18T18:50:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "num": 4810 2024-03-18T18:50:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-18T18:50:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": { 2024-03-18T18:50:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-03-18T18:50:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "192.168.0.2:0", 2024-03-18T18:50:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2160180625 2024-03-18T18:50:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-18T18:50:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-18T18:50:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": { 2024-03-18T18:50:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "type": "v1", 2024-03-18T18:50:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "addr": "192.168.0.2:0", 2024-03-18T18:50:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "nonce": 2160180625 2024-03-18T18:50:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-18T18:50:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "inst_str": "client.4810 192.168.0.2:0/2160180625", 2024-03-18T18:50:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "addr_str": "192.168.0.2:0/2160180625", 2024-03-18T18:50:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "inode_count": 1, 2024-03-18T18:50:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_epoch": 9, 2024-03-18T18:50:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_epoch": 25, 2024-03-18T18:50:29.712 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_epoch_barrier": 0, 2024-03-18T18:50:29.712 INFO:teuthology.orchestra.run.smithi165.stdout: "blocklisted": false, 2024-03-18T18:50:29.712 INFO:teuthology.orchestra.run.smithi165.stdout: "fs_name": "cephfs" 2024-03-18T18:50:29.712 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-03-18T18:50:29.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:29.720 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-18T18:50:29.720 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T18:50:29.743 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-18T18:50:29.743 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T18:50:29.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:29.743 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T18:50:29.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:29.817 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-18T18:50:29.817 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.2 2024-03-18T18:50:29.879 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-18T18:50:29.879 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-03-18T18:50:29.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:29.879 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-18T18:50:29.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:29.950 DEBUG:teuthology.orchestra.run.smithi165:> stat --file-system '--printf=%T 2024-03-18T18:50:29.950 DEBUG:teuthology.orchestra.run.smithi165:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-18T18:50:29.968 INFO:teuthology.orchestra.run.smithi165.stdout:fuseblk 2024-03-18T18:50:29.969 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-18T18:50:29.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:29.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-18T18:50:30.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:30.036 DEBUG:teuthology.orchestra.run.smithi165:> stat --file-system '--printf=%T 2024-03-18T18:50:30.036 DEBUG:teuthology.orchestra.run.smithi165:> ' -- /home/ubuntu/cephtest/mnt.3 2024-03-18T18:50:30.094 INFO:teuthology.orchestra.run.smithi165.stdout:fuseblk 2024-03-18T18:50:30.095 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-03-18T18:50:30.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:30.095 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-18T18:50:30.161 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-18T18:50:30.198 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:30.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.337+0000 7faea261e700 1 Processor -- start 2024-03-18T18:50:30.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.337+0000 7faea261e700 1 -- start start 2024-03-18T18:50:30.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.338+0000 7faea261e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fae9c06a750 0x7fae9c06f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:30.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.338+0000 7faea261e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fae9c06ea50 0x7fae9c0b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:30.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.338+0000 7faea261e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c06fb20 con 0x7fae9c0af3e0 2024-03-18T18:50:30.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.338+0000 7faea261e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9c06dd20 con 0x7fae9c06ea50 2024-03-18T18:50:30.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.338+0000 7faea261e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9c06fd50 con 0x7fae9c06a750 2024-03-18T18:50:30.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.338+0000 7faea0bbb700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fae9c06ea50 0x7fae9c0b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:30.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.338+0000 7fae9b7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fae9c0af3e0 0x7fae9c06ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59506/0 (socket says 172.21.15.1:59506) 2024-03-18T18:50:30.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.338+0000 7fae9b7fe700 1 -- 172.21.15.1:0/1135366738 learned_addr learned my addr 172.21.15.1:0/1135366738 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:30.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.338+0000 7fae9bfff700 1 --2- 172.21.15.1:0/1135366738 >> v2:172.21.15.165:3300/0 conn(0x7fae9c06a750 0x7fae9c06f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:30.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.338+0000 7faea0bbb700 1 -- 172.21.15.1:0/1135366738 >> v2:172.21.15.165:3300/0 conn(0x7fae9c06a750 msgr2=0x7fae9c06f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:30.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.338+0000 7faea0bbb700 1 --2- 172.21.15.1:0/1135366738 >> v2:172.21.15.165:3300/0 conn(0x7fae9c06a750 0x7fae9c06f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.338+0000 7faea0bbb700 1 -- 172.21.15.1:0/1135366738 >> v1:172.21.15.1:6789/0 conn(0x7fae9c0af3e0 legacy=0x7fae9c06ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:30.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.338+0000 7faea0bbb700 1 -- 172.21.15.1:0/1135366738 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9c06df70 con 0x7fae9c06ea50 2024-03-18T18:50:30.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.338+0000 7fae9bfff700 1 --2- 172.21.15.1:0/1135366738 >> v2:172.21.15.165:3300/0 conn(0x7fae9c06a750 0x7fae9c06f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:30.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.338+0000 7faea0bbb700 1 --2- 172.21.15.1:0/1135366738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae9c06ea50 0x7fae9c0b2da0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fae9c06c970 tx=0x7fae9400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c878db034b0535fd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:30.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.339+0000 7fae9affd700 1 -- 172.21.15.1:0/1135366738 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9400e070 con 0x7fae9c06ea50 2024-03-18T18:50:30.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.339+0000 7fae9affd700 1 -- 172.21.15.1:0/1135366738 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9400bbd0 con 0x7fae9c06ea50 2024-03-18T18:50:30.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.339+0000 7fae9affd700 1 -- 172.21.15.1:0/1135366738 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae94012430 con 0x7fae9c06ea50 2024-03-18T18:50:30.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.339+0000 7faea261e700 1 -- 172.21.15.1:0/1135366738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae9c06ea50 msgr2=0x7fae9c0b2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:30.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.339+0000 7faea261e700 1 --2- 172.21.15.1:0/1135366738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae9c06ea50 0x7fae9c0b2da0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fae9c06c970 tx=0x7fae9400b330 comp rx=0 tx=0).stop 2024-03-18T18:50:30.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.339+0000 7faea261e700 1 -- 172.21.15.1:0/1135366738 shutdown_connections 2024-03-18T18:50:30.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.339+0000 7faea261e700 1 --2- 172.21.15.1:0/1135366738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae9c06ea50 0x7fae9c0b2da0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.339+0000 7faea261e700 1 --2- 172.21.15.1:0/1135366738 >> v2:172.21.15.165:3300/0 conn(0x7fae9c06a750 0x7fae9c06f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.339+0000 7faea261e700 1 -- 172.21.15.1:0/1135366738 >> 172.21.15.1:0/1135366738 conn(0x7fae9c0cfef0 msgr2=0x7fae9c0d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:30.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.339+0000 7faea261e700 1 -- 172.21.15.1:0/1135366738 shutdown_connections 2024-03-18T18:50:30.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.339+0000 7faea261e700 1 -- 172.21.15.1:0/1135366738 wait complete. 2024-03-18T18:50:30.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.340+0000 7faea261e700 1 Processor -- start 2024-03-18T18:50:30.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.341+0000 7faea261e700 1 -- start start 2024-03-18T18:50:30.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.341+0000 7faea261e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae9c06a750 0x7fae9c14dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:30.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.341+0000 7faea261e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae9c06ea50 0x7fae9c160210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:30.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.341+0000 7faea261e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae9c0af3e0 0x7fae9c144500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:30.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.341+0000 7faea261e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c15b190 con 0x7fae9c0af3e0 2024-03-18T18:50:30.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.341+0000 7faea261e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c160750 con 0x7fae9c06ea50 2024-03-18T18:50:30.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.341+0000 7faea261e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c15a0f0 con 0x7fae9c06a750 2024-03-18T18:50:30.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.342+0000 7faea0bbb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae9c0af3e0 0x7fae9c144500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:30.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.342+0000 7faea0bbb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae9c0af3e0 0x7fae9c144500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47894/0 (socket says 172.21.15.1:47894) 2024-03-18T18:50:30.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.342+0000 7faea0bbb700 1 -- 172.21.15.1:0/2930892626 learned_addr learned my addr 172.21.15.1:0/2930892626 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:30.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.342+0000 7fae9b7fe700 1 --2- 172.21.15.1:0/2930892626 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae9c06ea50 0x7fae9c160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:30.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.342+0000 7fae9bfff700 1 --2- 172.21.15.1:0/2930892626 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae9c06a750 0x7fae9c14dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:30.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.342+0000 7faea0bbb700 1 -- 172.21.15.1:0/2930892626 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae9c06a750 msgr2=0x7fae9c14dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:30.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.342+0000 7faea0bbb700 1 --2- 172.21.15.1:0/2930892626 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae9c06a750 0x7fae9c14dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.342+0000 7faea0bbb700 1 -- 172.21.15.1:0/2930892626 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae9c06ea50 msgr2=0x7fae9c160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:30.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.342+0000 7faea0bbb700 1 --2- 172.21.15.1:0/2930892626 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae9c06ea50 0x7fae9c160210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.342+0000 7faea0bbb700 1 -- 172.21.15.1:0/2930892626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae94009a70 con 0x7fae9c0af3e0 2024-03-18T18:50:30.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.342+0000 7faea0bbb700 1 --2- 172.21.15.1:0/2930892626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae9c0af3e0 0x7fae9c144500 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fae94014040 tx=0x7fae94004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:30.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.342+0000 7fae98ff9700 1 -- 172.21.15.1:0/2930892626 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae94012da0 con 0x7fae9c0af3e0 2024-03-18T18:50:30.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.342+0000 7fae98ff9700 1 -- 172.21.15.1:0/2930892626 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae94020070 con 0x7fae9c0af3e0 2024-03-18T18:50:30.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.342+0000 7faea261e700 1 -- 172.21.15.1:0/2930892626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9c15a3d0 con 0x7fae9c0af3e0 2024-03-18T18:50:30.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.342+0000 7fae98ff9700 1 -- 172.21.15.1:0/2930892626 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae94004280 con 0x7fae9c0af3e0 2024-03-18T18:50:30.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.342+0000 7faea261e700 1 -- 172.21.15.1:0/2930892626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9c15a990 con 0x7fae9c0af3e0 2024-03-18T18:50:30.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.343+0000 7faea261e700 1 -- 172.21.15.1:0/2930892626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9c0ae7d0 con 0x7fae9c0af3e0 2024-03-18T18:50:30.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.344+0000 7fae98ff9700 1 -- 172.21.15.1:0/2930892626 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fae940043e0 con 0x7fae9c0af3e0 2024-03-18T18:50:30.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.344+0000 7fae98ff9700 1 --2- 172.21.15.1:0/2930892626 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae7c06ac90 0x7fae7c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:30.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.344+0000 7fae98ff9700 1 -- 172.21.15.1:0/2930892626 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fae94022070 con 0x7fae9c0af3e0 2024-03-18T18:50:30.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.344+0000 7fae9bfff700 1 --2- 172.21.15.1:0/2930892626 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae7c06ac90 0x7fae7c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:30.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.345+0000 7fae9bfff700 1 --2- 172.21.15.1:0/2930892626 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae7c06ac90 0x7fae7c06d150 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fae84006fd0 tx=0x7fae84008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:30.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.349+0000 7fae98ff9700 1 -- 172.21.15.1:0/2930892626 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fae940562b0 con 0x7fae9c0af3e0 2024-03-18T18:50:30.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.553+0000 7faea261e700 1 -- 172.21.15.1:0/2930892626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae9c073350 con 0x7fae9c0af3e0 2024-03-18T18:50:30.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.554+0000 7fae98ff9700 1 -- 172.21.15.1:0/2930892626 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17830 (secure 0 0 0) 0x7fae940598d0 con 0x7fae9c0af3e0 2024-03-18T18:50:30.551 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:30.551 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:49.276961+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:30.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.556+0000 7faea261e700 1 -- 172.21.15.1:0/2930892626 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae7c06ac90 msgr2=0x7fae7c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:30.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.556+0000 7faea261e700 1 --2- 172.21.15.1:0/2930892626 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae7c06ac90 0x7fae7c06d150 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fae84006fd0 tx=0x7fae84008040 comp rx=0 tx=0).stop 2024-03-18T18:50:30.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.556+0000 7faea261e700 1 -- 172.21.15.1:0/2930892626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae9c0af3e0 msgr2=0x7fae9c144500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:30.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.556+0000 7faea261e700 1 --2- 172.21.15.1:0/2930892626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae9c0af3e0 0x7fae9c144500 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fae94014040 tx=0x7fae94004ad0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.556+0000 7faea261e700 1 -- 172.21.15.1:0/2930892626 shutdown_connections 2024-03-18T18:50:30.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.556+0000 7faea261e700 1 --2- 172.21.15.1:0/2930892626 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae7c06ac90 0x7fae7c06d150 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.556+0000 7faea261e700 1 --2- 172.21.15.1:0/2930892626 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae9c06a750 0x7fae9c14dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.556+0000 7faea261e700 1 --2- 172.21.15.1:0/2930892626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae9c0af3e0 0x7fae9c144500 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.556+0000 7faea261e700 1 --2- 172.21.15.1:0/2930892626 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae9c06ea50 0x7fae9c160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.556+0000 7faea261e700 1 -- 172.21.15.1:0/2930892626 >> 172.21.15.1:0/2930892626 conn(0x7fae9c0cfef0 msgr2=0x7fae9c0b0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:30.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.556+0000 7faea261e700 1 -- 172.21.15.1:0/2930892626 shutdown_connections 2024-03-18T18:50:30.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.556+0000 7faea261e700 1 -- 172.21.15.1:0/2930892626 wait complete. 2024-03-18T18:50:30.565 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T18:49:42.800801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T18:49:43.776684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T18:49:46.854608+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T18:50:30.565 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T18:50:30.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.697+0000 7fd7824fb700 1 Processor -- start 2024-03-18T18:50:30.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.697+0000 7fd7824fb700 1 -- start start 2024-03-18T18:50:30.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.697+0000 7fd7824fb700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd77c15d5a0 0x7fd77c0ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:30.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.697+0000 7fd7824fb700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd77c06ae60 0x7fd77c167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:30.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.697+0000 7fd7824fb700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd77c0cf600 con 0x7fd77c0ce550 2024-03-18T18:50:30.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.697+0000 7fd7824fb700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd77c0ce9f0 con 0x7fd77c06ae60 2024-03-18T18:50:30.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.697+0000 7fd7824fb700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd77c0cecd0 con 0x7fd77c15d5a0 2024-03-18T18:50:30.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.698+0000 7fd780a98700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd77c06ae60 0x7fd77c167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:30.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.698+0000 7fd780a98700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd77c06ae60 0x7fd77c167140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47908/0 (socket says 172.21.15.1:47908) 2024-03-18T18:50:30.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.698+0000 7fd780a98700 1 -- 172.21.15.1:0/2688800445 learned_addr learned my addr 172.21.15.1:0/2688800445 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:30.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.698+0000 7fd77bfff700 1 --2- 172.21.15.1:0/2688800445 >> v2:172.21.15.165:3300/0 conn(0x7fd77c15d5a0 0x7fd77c0ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:30.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.698+0000 7fd780a98700 1 -- 172.21.15.1:0/2688800445 >> v2:172.21.15.165:3300/0 conn(0x7fd77c15d5a0 msgr2=0x7fd77c0ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:30.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.698+0000 7fd780a98700 1 --2- 172.21.15.1:0/2688800445 >> v2:172.21.15.165:3300/0 conn(0x7fd77c15d5a0 0x7fd77c0ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.698+0000 7fd780a98700 1 -- 172.21.15.1:0/2688800445 >> v1:172.21.15.165:6789/0 conn(0x7fd77c0ce550 legacy=0x7fd77c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:30.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.698+0000 7fd780a98700 1 -- 172.21.15.1:0/2688800445 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77c0cefb0 con 0x7fd77c06ae60 2024-03-18T18:50:30.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.698+0000 7fd780a98700 1 --2- 172.21.15.1:0/2688800445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd77c06ae60 0x7fd77c167140 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd77c072f60 tx=0x7fd77400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c34fddbc5390a52c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:30.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.698+0000 7fd77affd700 1 -- 172.21.15.1:0/2688800445 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77400e070 con 0x7fd77c06ae60 2024-03-18T18:50:30.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.698+0000 7fd77affd700 1 -- 172.21.15.1:0/2688800445 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd77400b910 con 0x7fd77c06ae60 2024-03-18T18:50:30.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.698+0000 7fd77affd700 1 -- 172.21.15.1:0/2688800445 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77400bcc0 con 0x7fd77c06ae60 2024-03-18T18:50:30.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.699+0000 7fd7824fb700 1 -- 172.21.15.1:0/2688800445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd77c06ae60 msgr2=0x7fd77c167140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:30.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.699+0000 7fd7824fb700 1 --2- 172.21.15.1:0/2688800445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd77c06ae60 0x7fd77c167140 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd77c072f60 tx=0x7fd77400b330 comp rx=0 tx=0).stop 2024-03-18T18:50:30.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.699+0000 7fd7824fb700 1 -- 172.21.15.1:0/2688800445 shutdown_connections 2024-03-18T18:50:30.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.699+0000 7fd7824fb700 1 --2- 172.21.15.1:0/2688800445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd77c06ae60 0x7fd77c167140 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.699+0000 7fd7824fb700 1 --2- 172.21.15.1:0/2688800445 >> v2:172.21.15.165:3300/0 conn(0x7fd77c15d5a0 0x7fd77c0ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.699+0000 7fd7824fb700 1 -- 172.21.15.1:0/2688800445 >> 172.21.15.1:0/2688800445 conn(0x7fd77c0cc9b0 msgr2=0x7fd77c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:30.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.699+0000 7fd7824fb700 1 -- 172.21.15.1:0/2688800445 shutdown_connections 2024-03-18T18:50:30.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.699+0000 7fd7824fb700 1 -- 172.21.15.1:0/2688800445 wait complete. 2024-03-18T18:50:30.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.700+0000 7fd7824fb700 1 Processor -- start 2024-03-18T18:50:30.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.700+0000 7fd7824fb700 1 -- start start 2024-03-18T18:50:30.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.701+0000 7fd7824fb700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd77c06ae60 0x7fd77c1480e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:30.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.701+0000 7fd7824fb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd77c0ce550 0x7fd77c1f2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:30.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.701+0000 7fd7824fb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd77c15d5a0 0x7fd77c1f25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:30.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.701+0000 7fd7824fb700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c15db00 con 0x7fd77c0ce550 2024-03-18T18:50:30.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.701+0000 7fd7824fb700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c1f2ae0 con 0x7fd77c06ae60 2024-03-18T18:50:30.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.701+0000 7fd7824fb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c15a710 con 0x7fd77c15d5a0 2024-03-18T18:50:30.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.701+0000 7fd77b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd77c0ce550 0x7fd77c1f2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:30.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.701+0000 7fd77b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd77c0ce550 0x7fd77c1f2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47918/0 (socket says 172.21.15.1:47918) 2024-03-18T18:50:30.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.701+0000 7fd77b7fe700 1 -- 172.21.15.1:0/375426617 learned_addr learned my addr 172.21.15.1:0/375426617 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:30.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.701+0000 7fd77bfff700 1 --2- 172.21.15.1:0/375426617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd77c06ae60 0x7fd77c1480e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:30.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.701+0000 7fd780a98700 1 --2- 172.21.15.1:0/375426617 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd77c15d5a0 0x7fd77c1f25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:30.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.701+0000 7fd77b7fe700 1 -- 172.21.15.1:0/375426617 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd77c15d5a0 msgr2=0x7fd77c1f25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:30.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.701+0000 7fd77b7fe700 1 --2- 172.21.15.1:0/375426617 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd77c15d5a0 0x7fd77c1f25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.701+0000 7fd77b7fe700 1 -- 172.21.15.1:0/375426617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd77c06ae60 msgr2=0x7fd77c1480e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:30.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.701+0000 7fd77b7fe700 1 --2- 172.21.15.1:0/375426617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd77c06ae60 0x7fd77c1480e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.701+0000 7fd77b7fe700 1 -- 172.21.15.1:0/375426617 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd774009b30 con 0x7fd77c0ce550 2024-03-18T18:50:30.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.702+0000 7fd77b7fe700 1 --2- 172.21.15.1:0/375426617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd77c0ce550 0x7fd77c1f2060 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd77000c510 tx=0x7fd77000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:30.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.702+0000 7fd778ff9700 1 -- 172.21.15.1:0/375426617 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7700052d0 con 0x7fd77c0ce550 2024-03-18T18:50:30.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.702+0000 7fd778ff9700 1 -- 172.21.15.1:0/375426617 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd770005430 con 0x7fd77c0ce550 2024-03-18T18:50:30.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.702+0000 7fd7824fb700 1 -- 172.21.15.1:0/375426617 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd77c15dce0 con 0x7fd77c0ce550 2024-03-18T18:50:30.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.702+0000 7fd778ff9700 1 -- 172.21.15.1:0/375426617 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7700038c0 con 0x7fd77c0ce550 2024-03-18T18:50:30.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.702+0000 7fd7824fb700 1 -- 172.21.15.1:0/375426617 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd77c15a990 con 0x7fd77c0ce550 2024-03-18T18:50:30.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.703+0000 7fd7824fb700 1 -- 172.21.15.1:0/375426617 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd77c15abb0 con 0x7fd77c0ce550 2024-03-18T18:50:30.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.703+0000 7fd778ff9700 1 -- 172.21.15.1:0/375426617 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd770003a20 con 0x7fd77c0ce550 2024-03-18T18:50:30.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.704+0000 7fd778ff9700 1 --2- 172.21.15.1:0/375426617 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd76806ac90 0x7fd76806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:30.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.704+0000 7fd778ff9700 1 -- 172.21.15.1:0/375426617 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd770088990 con 0x7fd77c0ce550 2024-03-18T18:50:30.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.704+0000 7fd77bfff700 1 --2- 172.21.15.1:0/375426617 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd76806ac90 0x7fd76806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:30.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.704+0000 7fd77bfff700 1 --2- 172.21.15.1:0/375426617 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd76806ac90 0x7fd76806d150 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd76c009b20 tx=0x7fd76c0091c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:30.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.708+0000 7fd778ff9700 1 -- 172.21.15.1:0/375426617 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd7700537e0 con 0x7fd77c0ce550 2024-03-18T18:50:30.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.908+0000 7fd7824fb700 1 -- 172.21.15.1:0/375426617 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd77c0676a0 con 0x7fd77c0ce550 2024-03-18T18:50:30.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.909+0000 7fd778ff9700 1 -- 172.21.15.1:0/375426617 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd770017020 con 0x7fd77c0ce550 2024-03-18T18:50:30.905 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T18:50:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.910+0000 7fd7824fb700 1 -- 172.21.15.1:0/375426617 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd76806ac90 msgr2=0x7fd76806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.911+0000 7fd7824fb700 1 --2- 172.21.15.1:0/375426617 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd76806ac90 0x7fd76806d150 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd76c009b20 tx=0x7fd76c0091c0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.911+0000 7fd7824fb700 1 -- 172.21.15.1:0/375426617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd77c0ce550 msgr2=0x7fd77c1f2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:30.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.911+0000 7fd7824fb700 1 --2- 172.21.15.1:0/375426617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd77c0ce550 0x7fd77c1f2060 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd77000c510 tx=0x7fd77000c820 comp rx=0 tx=0).stop 2024-03-18T18:50:30.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.911+0000 7fd7824fb700 1 -- 172.21.15.1:0/375426617 shutdown_connections 2024-03-18T18:50:30.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.911+0000 7fd7824fb700 1 --2- 172.21.15.1:0/375426617 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd76806ac90 0x7fd76806d150 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.911+0000 7fd7824fb700 1 --2- 172.21.15.1:0/375426617 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd77c15d5a0 0x7fd77c1f25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.911+0000 7fd7824fb700 1 --2- 172.21.15.1:0/375426617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd77c06ae60 0x7fd77c1480e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.911+0000 7fd7824fb700 1 --2- 172.21.15.1:0/375426617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd77c0ce550 0x7fd77c1f2060 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:30.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.911+0000 7fd7824fb700 1 -- 172.21.15.1:0/375426617 >> 172.21.15.1:0/375426617 conn(0x7fd77c0cc9b0 msgr2=0x7fd77c15c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:30.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.911+0000 7fd7824fb700 1 -- 172.21.15.1:0/375426617 shutdown_connections 2024-03-18T18:50:30.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:30.911+0000 7fd7824fb700 1 -- 172.21.15.1:0/375426617 wait complete. 2024-03-18T18:50:30.919 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T18:50:31.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.051+0000 7ff62be48700 1 Processor -- start 2024-03-18T18:50:31.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.051+0000 7ff62be48700 1 -- start start 2024-03-18T18:50:31.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.052+0000 7ff62be48700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff62415d5b0 0x7ff62415d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:31.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.052+0000 7ff62be48700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff62406ca80 0x7ff6240cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:31.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.052+0000 7ff62be48700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6240b3b00 con 0x7ff624099a40 2024-03-18T18:50:31.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.052+0000 7ff62be48700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6240cbbb0 con 0x7ff62415d5b0 2024-03-18T18:50:31.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.052+0000 7ff62be48700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6240cf7f0 con 0x7ff62406ca80 2024-03-18T18:50:31.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.052+0000 7ff62a3e5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff62406ca80 0x7ff6240cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:31.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.052+0000 7ff629be4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff62415d5b0 0x7ff62415d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:31.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.052+0000 7ff6293e3700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7ff624099a40 0x7ff624099ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:33278/0 (socket says 172.21.15.1:33278) 2024-03-18T18:50:31.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.052+0000 7ff6293e3700 1 -- 172.21.15.1:0/3851930056 learned_addr learned my addr 172.21.15.1:0/3851930056 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:31.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.052+0000 7ff628be2700 1 -- 172.21.15.1:0/3851930056 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078272497 0 0) 0x7ff6240b3b00 con 0x7ff624099a40 2024-03-18T18:50:31.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.053+0000 7ff628be2700 1 -- 172.21.15.1:0/3851930056 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6100035b0 con 0x7ff624099a40 2024-03-18T18:50:31.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.053+0000 7ff62a3e5700 1 -- 172.21.15.1:0/3851930056 >> v2:172.21.15.165:3300/0 conn(0x7ff62415d5b0 msgr2=0x7ff62415d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:31.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.053+0000 7ff62a3e5700 1 --2- 172.21.15.1:0/3851930056 >> v2:172.21.15.165:3300/0 conn(0x7ff62415d5b0 0x7ff62415d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.053+0000 7ff62a3e5700 1 -- 172.21.15.1:0/3851930056 >> v1:172.21.15.165:6789/0 conn(0x7ff624099a40 legacy=0x7ff624099ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:31.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.053+0000 7ff62a3e5700 1 -- 172.21.15.1:0/3851930056 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6240d04d0 con 0x7ff62406ca80 2024-03-18T18:50:31.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.053+0000 7ff629be4700 1 --2- 172.21.15.1:0/3851930056 >> v2:172.21.15.165:3300/0 conn(0x7ff62415d5b0 0x7ff62415d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:50:31.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.053+0000 7ff62a3e5700 1 --2- 172.21.15.1:0/3851930056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff62406ca80 0x7ff6240cbfd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff61c009d90 tx=0x7ff61c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=137631889b838092 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:31.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.053+0000 7ff628be2700 1 -- 172.21.15.1:0/3851930056 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c00e070 con 0x7ff62406ca80 2024-03-18T18:50:31.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.053+0000 7ff628be2700 1 -- 172.21.15.1:0/3851930056 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61c00bb20 con 0x7ff62406ca80 2024-03-18T18:50:31.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.053+0000 7ff628be2700 1 -- 172.21.15.1:0/3851930056 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c012430 con 0x7ff62406ca80 2024-03-18T18:50:31.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.053+0000 7ff62be48700 1 -- 172.21.15.1:0/3851930056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff62406ca80 msgr2=0x7ff6240cbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:31.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.053+0000 7ff62be48700 1 --2- 172.21.15.1:0/3851930056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff62406ca80 0x7ff6240cbfd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff61c009d90 tx=0x7ff61c00b330 comp rx=0 tx=0).stop 2024-03-18T18:50:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.054+0000 7ff62be48700 1 -- 172.21.15.1:0/3851930056 shutdown_connections 2024-03-18T18:50:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.054+0000 7ff62be48700 1 --2- 172.21.15.1:0/3851930056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff62406ca80 0x7ff6240cbfd0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.054+0000 7ff62be48700 1 --2- 172.21.15.1:0/3851930056 >> v2:172.21.15.165:3300/0 conn(0x7ff62415d5b0 0x7ff62415d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.054+0000 7ff62be48700 1 -- 172.21.15.1:0/3851930056 >> 172.21.15.1:0/3851930056 conn(0x7ff6240b2bd0 msgr2=0x7ff6240b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.054+0000 7ff62be48700 1 -- 172.21.15.1:0/3851930056 shutdown_connections 2024-03-18T18:50:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.054+0000 7ff62be48700 1 -- 172.21.15.1:0/3851930056 wait complete. 2024-03-18T18:50:31.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.055+0000 7ff62be48700 1 Processor -- start 2024-03-18T18:50:31.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.055+0000 7ff62be48700 1 -- start start 2024-03-18T18:50:31.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff62be48700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff62406ca80 0x7ff62414fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:31.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff62be48700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff624099a40 0x7ff62415bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:31.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff62be48700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff62415d5b0 0x7ff62415c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff62be48700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6241f2c80 con 0x7ff624099a40 2024-03-18T18:50:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff62be48700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6240c5a80 con 0x7ff62406ca80 2024-03-18T18:50:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff62be48700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6240c4cc0 con 0x7ff62415d5b0 2024-03-18T18:50:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff6293e3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff624099a40 0x7ff62415bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff6293e3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff624099a40 0x7ff62415bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47934/0 (socket says 172.21.15.1:47934) 2024-03-18T18:50:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff6293e3700 1 -- 172.21.15.1:0/3547800351 learned_addr learned my addr 172.21.15.1:0/3547800351 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff629be4700 1 --2- 172.21.15.1:0/3547800351 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff62406ca80 0x7ff62414fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff62a3e5700 1 --2- 172.21.15.1:0/3547800351 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff62415d5b0 0x7ff62415c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff6293e3700 1 -- 172.21.15.1:0/3547800351 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff62415d5b0 msgr2=0x7ff62415c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff6293e3700 1 --2- 172.21.15.1:0/3547800351 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff62415d5b0 0x7ff62415c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff6293e3700 1 -- 172.21.15.1:0/3547800351 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff62406ca80 msgr2=0x7ff62414fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff6293e3700 1 --2- 172.21.15.1:0/3547800351 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff62406ca80 0x7ff62414fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff6293e3700 1 -- 172.21.15.1:0/3547800351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6140094a0 con 0x7ff624099a40 2024-03-18T18:50:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff629be4700 1 --2- 172.21.15.1:0/3547800351 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff62406ca80 0x7ff62414fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:31.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff62a3e5700 1 --2- 172.21.15.1:0/3547800351 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff62415d5b0 0x7ff62415c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:31.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff6293e3700 1 --2- 172.21.15.1:0/3547800351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff624099a40 0x7ff62415bb50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff62000c4d0 tx=0x7ff62000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:31.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff61a7fc700 1 -- 172.21.15.1:0/3547800351 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff620005d70 con 0x7ff624099a40 2024-03-18T18:50:31.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff61a7fc700 1 -- 172.21.15.1:0/3547800351 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6200070d0 con 0x7ff624099a40 2024-03-18T18:50:31.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff62be48700 1 -- 172.21.15.1:0/3547800351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff61c009a70 con 0x7ff624099a40 2024-03-18T18:50:31.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff61a7fc700 1 -- 172.21.15.1:0/3547800351 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff620007460 con 0x7ff624099a40 2024-03-18T18:50:31.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.056+0000 7ff62be48700 1 -- 172.21.15.1:0/3547800351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6240c5280 con 0x7ff624099a40 2024-03-18T18:50:31.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.057+0000 7ff62be48700 1 -- 172.21.15.1:0/3547800351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6240676a0 con 0x7ff624099a40 2024-03-18T18:50:31.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.058+0000 7ff61a7fc700 1 -- 172.21.15.1:0/3547800351 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff6200075c0 con 0x7ff624099a40 2024-03-18T18:50:31.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.059+0000 7ff61a7fc700 1 --2- 172.21.15.1:0/3547800351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff60806ac90 0x7ff60806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:31.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.059+0000 7ff61a7fc700 1 -- 172.21.15.1:0/3547800351 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff620088ec0 con 0x7ff624099a40 2024-03-18T18:50:31.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.059+0000 7ff629be4700 1 --2- 172.21.15.1:0/3547800351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff60806ac90 0x7ff60806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:31.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.059+0000 7ff629be4700 1 --2- 172.21.15.1:0/3547800351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff60806ac90 0x7ff60806d150 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff614009d20 tx=0x7ff614008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:31.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.063+0000 7ff61a7fc700 1 -- 172.21.15.1:0/3547800351 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff620053e90 con 0x7ff624099a40 2024-03-18T18:50:31.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.264+0000 7ff62be48700 1 -- 172.21.15.1:0/3547800351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff6240cc9f0 con 0x7ff624099a40 2024-03-18T18:50:31.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.265+0000 7ff61a7fc700 1 -- 172.21.15.1:0/3547800351 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff6200574b0 con 0x7ff624099a40 2024-03-18T18:50:31.261 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:31.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.267+0000 7ff62be48700 1 -- 172.21.15.1:0/3547800351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff60806ac90 msgr2=0x7ff60806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:31.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.267+0000 7ff62be48700 1 --2- 172.21.15.1:0/3547800351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff60806ac90 0x7ff60806d150 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff614009d20 tx=0x7ff614008040 comp rx=0 tx=0).stop 2024-03-18T18:50:31.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.267+0000 7ff62be48700 1 -- 172.21.15.1:0/3547800351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff624099a40 msgr2=0x7ff62415bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:31.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.267+0000 7ff62be48700 1 --2- 172.21.15.1:0/3547800351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff624099a40 0x7ff62415bb50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff62000c4d0 tx=0x7ff62000c890 comp rx=0 tx=0).stop 2024-03-18T18:50:31.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.267+0000 7ff62be48700 1 -- 172.21.15.1:0/3547800351 shutdown_connections 2024-03-18T18:50:31.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.267+0000 7ff62be48700 1 --2- 172.21.15.1:0/3547800351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff60806ac90 0x7ff60806d150 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.267+0000 7ff62be48700 1 --2- 172.21.15.1:0/3547800351 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff62415d5b0 0x7ff62415c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.267+0000 7ff62be48700 1 --2- 172.21.15.1:0/3547800351 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff62406ca80 0x7ff62414fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.267+0000 7ff62be48700 1 --2- 172.21.15.1:0/3547800351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff624099a40 0x7ff62415bb50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.267+0000 7ff62be48700 1 -- 172.21.15.1:0/3547800351 >> 172.21.15.1:0/3547800351 conn(0x7ff6240b2bd0 msgr2=0x7ff62406fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:31.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.267+0000 7ff62be48700 1 -- 172.21.15.1:0/3547800351 shutdown_connections 2024-03-18T18:50:31.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.267+0000 7ff62be48700 1 -- 172.21.15.1:0/3547800351 wait complete. 2024-03-18T18:50:31.275 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T18:50:31.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.408+0000 7f3ffe49c700 1 Processor -- start 2024-03-18T18:50:31.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.408+0000 7f3ffe49c700 1 -- start start 2024-03-18T18:50:31.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.409+0000 7f3ffe49c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3ff8146db0 0x7f3ff80cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:31.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.409+0000 7f3ffe49c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3ff806b1f0 0x7f3ff8160fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:31.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.409+0000 7f3ffe49c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff8147190 con 0x7f3ff80b3430 2024-03-18T18:50:31.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.409+0000 7f3ffe49c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ff8161820 con 0x7f3ff8146db0 2024-03-18T18:50:31.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.409+0000 7f3ffe49c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ff8161b00 con 0x7f3ff806b1f0 2024-03-18T18:50:31.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.409+0000 7f3ff7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3ff8146db0 0x7f3ff80cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:31.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.409+0000 7f3ff7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3ff8146db0 0x7f3ff80cca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47944/0 (socket says 172.21.15.1:47944) 2024-03-18T18:50:31.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.409+0000 7f3ff7fff700 1 -- 172.21.15.1:0/1251817467 learned_addr learned my addr 172.21.15.1:0/1251817467 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:31.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.410+0000 7f3ff7fff700 1 -- 172.21.15.1:0/1251817467 >> v2:172.21.15.165:3300/0 conn(0x7f3ff806b1f0 msgr2=0x7f3ff8160fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:31.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.410+0000 7f3ff77fe700 1 --2- 172.21.15.1:0/1251817467 >> v2:172.21.15.165:3300/0 conn(0x7f3ff806b1f0 0x7f3ff8160fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:31.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.410+0000 7f3ff7fff700 1 --2- 172.21.15.1:0/1251817467 >> v2:172.21.15.165:3300/0 conn(0x7f3ff806b1f0 0x7f3ff8160fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.410+0000 7f3ff7fff700 1 -- 172.21.15.1:0/1251817467 >> v1:172.21.15.165:6789/0 conn(0x7f3ff80b3430 legacy=0x7f3ff806f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:31.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.410+0000 7f3ff7fff700 1 -- 172.21.15.1:0/1251817467 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff8161de0 con 0x7f3ff8146db0 2024-03-18T18:50:31.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.410+0000 7f3ff77fe700 1 --2- 172.21.15.1:0/1251817467 >> v2:172.21.15.165:3300/0 conn(0x7f3ff806b1f0 0x7f3ff8160fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:31.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.410+0000 7f3ff7fff700 1 --2- 172.21.15.1:0/1251817467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ff8146db0 0x7f3ff80cca30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8160f30 tx=0x7f3fe8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f97c70f8cf2947 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:31.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.410+0000 7f3ff6ffd700 1 -- 172.21.15.1:0/1251817467 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe800b870 con 0x7f3ff8146db0 2024-03-18T18:50:31.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.410+0000 7f3ff6ffd700 1 -- 172.21.15.1:0/1251817467 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe800b9d0 con 0x7f3ff8146db0 2024-03-18T18:50:31.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.410+0000 7f3ff6ffd700 1 -- 172.21.15.1:0/1251817467 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe800bd60 con 0x7f3ff8146db0 2024-03-18T18:50:31.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.411+0000 7f3ffe49c700 1 -- 172.21.15.1:0/1251817467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ff8146db0 msgr2=0x7f3ff80cca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:31.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.411+0000 7f3ffe49c700 1 --2- 172.21.15.1:0/1251817467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ff8146db0 0x7f3ff80cca30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8160f30 tx=0x7f3fe8009bd0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.411+0000 7f3ffe49c700 1 -- 172.21.15.1:0/1251817467 shutdown_connections 2024-03-18T18:50:31.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.411+0000 7f3ffe49c700 1 --2- 172.21.15.1:0/1251817467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ff8146db0 0x7f3ff80cca30 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.411+0000 7f3ffe49c700 1 --2- 172.21.15.1:0/1251817467 >> v2:172.21.15.165:3300/0 conn(0x7f3ff806b1f0 0x7f3ff8160fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.411+0000 7f3ffe49c700 1 -- 172.21.15.1:0/1251817467 >> 172.21.15.1:0/1251817467 conn(0x7f3ff80d0070 msgr2=0x7f3ff80d0480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:31.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.411+0000 7f3ffe49c700 1 -- 172.21.15.1:0/1251817467 shutdown_connections 2024-03-18T18:50:31.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.411+0000 7f3ffe49c700 1 -- 172.21.15.1:0/1251817467 wait complete. 2024-03-18T18:50:31.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.412+0000 7f3ffe49c700 1 Processor -- start 2024-03-18T18:50:31.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.412+0000 7f3ffe49c700 1 -- start start 2024-03-18T18:50:31.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.413+0000 7f3ffe49c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ff806b1f0 0x7f3ff815e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:31.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.413+0000 7f3ffe49c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ff80b3430 0x7f3ff815ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:31.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.413+0000 7f3ffe49c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ff8146db0 0x7f3ff80cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:31.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.413+0000 7f3ffe49c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff80ae910 con 0x7f3ff806b1f0 2024-03-18T18:50:31.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.413+0000 7f3ffe49c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff815d3f0 con 0x7f3ff8146db0 2024-03-18T18:50:31.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.413+0000 7f3ffe49c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff80adfb0 con 0x7f3ff80b3430 2024-03-18T18:50:31.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.413+0000 7f3ff7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ff806b1f0 0x7f3ff815e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:31.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.413+0000 7f3ff7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ff806b1f0 0x7f3ff815e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47950/0 (socket says 172.21.15.1:47950) 2024-03-18T18:50:31.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.413+0000 7f3ff7fff700 1 -- 172.21.15.1:0/1669560521 learned_addr learned my addr 172.21.15.1:0/1669560521 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:31.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.413+0000 7f3ff77fe700 1 --2- 172.21.15.1:0/1669560521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ff80b3430 0x7f3ff815ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:31.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.413+0000 7f3ffca39700 1 --2- 172.21.15.1:0/1669560521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ff8146db0 0x7f3ff80cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:31.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.413+0000 7f3ff7fff700 1 -- 172.21.15.1:0/1669560521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ff80b3430 msgr2=0x7f3ff815ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:31.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.413+0000 7f3ff7fff700 1 --2- 172.21.15.1:0/1669560521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ff80b3430 0x7f3ff815ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.413+0000 7f3ff7fff700 1 -- 172.21.15.1:0/1669560521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ff8146db0 msgr2=0x7f3ff80cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:31.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.413+0000 7f3ff7fff700 1 --2- 172.21.15.1:0/1669560521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ff8146db0 0x7f3ff80cd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.413+0000 7f3ff7fff700 1 -- 172.21.15.1:0/1669560521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe8009550 con 0x7f3ff806b1f0 2024-03-18T18:50:31.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.413+0000 7f3ffca39700 1 --2- 172.21.15.1:0/1669560521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ff8146db0 0x7f3ff80cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:31.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.413+0000 7f3ff7fff700 1 --2- 172.21.15.1:0/1669560521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ff806b1f0 0x7f3ff815e300 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8014040 tx=0x7f3fe8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:31.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.414+0000 7f3ff4ff9700 1 -- 172.21.15.1:0/1669560521 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe8004520 con 0x7f3ff806b1f0 2024-03-18T18:50:31.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.414+0000 7f3ff4ff9700 1 -- 172.21.15.1:0/1669560521 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe801e070 con 0x7f3ff806b1f0 2024-03-18T18:50:31.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.414+0000 7f3ffe49c700 1 -- 172.21.15.1:0/1669560521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff80ae250 con 0x7f3ff806b1f0 2024-03-18T18:50:31.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.414+0000 7f3ff4ff9700 1 -- 172.21.15.1:0/1669560521 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe80038c0 con 0x7f3ff806b1f0 2024-03-18T18:50:31.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.414+0000 7f3ffe49c700 1 -- 172.21.15.1:0/1669560521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff815fca0 con 0x7f3ff806b1f0 2024-03-18T18:50:31.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.414+0000 7f3ffe49c700 1 -- 172.21.15.1:0/1669560521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ff80676a0 con 0x7f3ff806b1f0 2024-03-18T18:50:31.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.415+0000 7f3ff4ff9700 1 -- 172.21.15.1:0/1669560521 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3fe8003a20 con 0x7f3ff806b1f0 2024-03-18T18:50:31.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.415+0000 7f3ff4ff9700 1 --2- 172.21.15.1:0/1669560521 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3fd806ac90 0x7f3fd806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:31.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.415+0000 7f3ff4ff9700 1 -- 172.21.15.1:0/1669560521 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3fe8020070 con 0x7f3ff806b1f0 2024-03-18T18:50:31.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.415+0000 7f3ff77fe700 1 --2- 172.21.15.1:0/1669560521 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3fd806ac90 0x7f3fd806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:31.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.416+0000 7f3ff77fe700 1 --2- 172.21.15.1:0/1669560521 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3fd806ac90 0x7f3fd806d150 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8146010 tx=0x7f3fe0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:31.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.420+0000 7f3ff4ff9700 1 -- 172.21.15.1:0/1669560521 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3fe8055c30 con 0x7f3ff806b1f0 2024-03-18T18:50:31.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.623+0000 7f3ffe49c700 1 -- 172.21.15.1:0/1669560521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3ff80b1860 con 0x7f3ff806b1f0 2024-03-18T18:50:31.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.623+0000 7f3ff4ff9700 1 -- 172.21.15.1:0/1669560521 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3fe802a0a0 con 0x7f3ff806b1f0 2024-03-18T18:50:31.620 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T18:50:31.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.626+0000 7f3ffe49c700 1 -- 172.21.15.1:0/1669560521 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3fd806ac90 msgr2=0x7f3fd806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:31.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.626+0000 7f3ffe49c700 1 --2- 172.21.15.1:0/1669560521 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3fd806ac90 0x7f3fd806d150 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8146010 tx=0x7f3fe0008040 comp rx=0 tx=0).stop 2024-03-18T18:50:31.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.626+0000 7f3ffe49c700 1 -- 172.21.15.1:0/1669560521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ff806b1f0 msgr2=0x7f3ff815e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:31.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.626+0000 7f3ffe49c700 1 --2- 172.21.15.1:0/1669560521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ff806b1f0 0x7f3ff815e300 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8014040 tx=0x7f3fe8004750 comp rx=0 tx=0).stop 2024-03-18T18:50:31.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.626+0000 7f3ffe49c700 1 -- 172.21.15.1:0/1669560521 shutdown_connections 2024-03-18T18:50:31.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.626+0000 7f3ffe49c700 1 --2- 172.21.15.1:0/1669560521 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3fd806ac90 0x7f3fd806d150 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.626+0000 7f3ffe49c700 1 --2- 172.21.15.1:0/1669560521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ff8146db0 0x7f3ff80cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.626+0000 7f3ffe49c700 1 --2- 172.21.15.1:0/1669560521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ff806b1f0 0x7f3ff815e300 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.626+0000 7f3ffe49c700 1 --2- 172.21.15.1:0/1669560521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ff80b3430 0x7f3ff815ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.626+0000 7f3ffe49c700 1 -- 172.21.15.1:0/1669560521 >> 172.21.15.1:0/1669560521 conn(0x7f3ff80d0070 msgr2=0x7f3ff80d0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:31.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.626+0000 7f3ffe49c700 1 -- 172.21.15.1:0/1669560521 shutdown_connections 2024-03-18T18:50:31.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.626+0000 7f3ffe49c700 1 -- 172.21.15.1:0/1669560521 wait complete. 2024-03-18T18:50:31.634 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T18:50:31.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.766+0000 7f8668edf700 1 Processor -- start 2024-03-18T18:50:31.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.766+0000 7f8668edf700 1 -- start start 2024-03-18T18:50:31.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.767+0000 7f8668edf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f866415d5a0 0x7f86640ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:31.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.767+0000 7f8668edf700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f866406ae60 0x7f8664167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:31.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.767+0000 7f8668edf700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86640cf600 con 0x7f86640ce550 2024-03-18T18:50:31.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.767+0000 7f8668edf700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86640ce9f0 con 0x7f866406ae60 2024-03-18T18:50:31.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.767+0000 7f8668edf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86640cecd0 con 0x7f866415d5a0 2024-03-18T18:50:31.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.767+0000 7f8662d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f866406ae60 0x7f8664167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:31.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.767+0000 7f8662d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f866406ae60 0x7f8664167140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47956/0 (socket says 172.21.15.1:47956) 2024-03-18T18:50:31.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.767+0000 7f8662d9d700 1 -- 172.21.15.1:0/2975082584 learned_addr learned my addr 172.21.15.1:0/2975082584 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:31.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.767+0000 7f866259c700 1 --2- 172.21.15.1:0/2975082584 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f866415d5a0 0x7f86640ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:31.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.767+0000 7f8662d9d700 1 -- 172.21.15.1:0/2975082584 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f866415d5a0 msgr2=0x7f86640ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:31.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.767+0000 7f8662d9d700 1 --2- 172.21.15.1:0/2975082584 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f866415d5a0 0x7f86640ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.767+0000 7f8662d9d700 1 -- 172.21.15.1:0/2975082584 >> v1:172.21.15.165:6789/0 conn(0x7f86640ce550 legacy=0x7f866406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:31.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.767+0000 7f8662d9d700 1 -- 172.21.15.1:0/2975082584 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86640cefb0 con 0x7f866406ae60 2024-03-18T18:50:31.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.767+0000 7f8662d9d700 1 --2- 172.21.15.1:0/2975082584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866406ae60 0x7f8664167140 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f865c009e50 tx=0x7f865c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=370f68834cf03c4a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:31.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.767+0000 7f866159a700 1 -- 172.21.15.1:0/2975082584 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c00e070 con 0x7f866406ae60 2024-03-18T18:50:31.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.768+0000 7f866159a700 1 -- 172.21.15.1:0/2975082584 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865c003a20 con 0x7f866406ae60 2024-03-18T18:50:31.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.768+0000 7f866159a700 1 -- 172.21.15.1:0/2975082584 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c003dd0 con 0x7f866406ae60 2024-03-18T18:50:31.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.768+0000 7f8668edf700 1 -- 172.21.15.1:0/2975082584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866406ae60 msgr2=0x7f8664167140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:31.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.768+0000 7f8668edf700 1 --2- 172.21.15.1:0/2975082584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866406ae60 0x7f8664167140 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f865c009e50 tx=0x7f865c00b330 comp rx=0 tx=0).stop 2024-03-18T18:50:31.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.768+0000 7f8668edf700 1 -- 172.21.15.1:0/2975082584 shutdown_connections 2024-03-18T18:50:31.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.768+0000 7f8668edf700 1 --2- 172.21.15.1:0/2975082584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866406ae60 0x7f8664167140 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.768+0000 7f8668edf700 1 --2- 172.21.15.1:0/2975082584 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f866415d5a0 0x7f86640ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.768+0000 7f8668edf700 1 -- 172.21.15.1:0/2975082584 >> 172.21.15.1:0/2975082584 conn(0x7f86640cc9b0 msgr2=0x7f86640ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:31.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.768+0000 7f8668edf700 1 -- 172.21.15.1:0/2975082584 shutdown_connections 2024-03-18T18:50:31.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.768+0000 7f8668edf700 1 -- 172.21.15.1:0/2975082584 wait complete. 2024-03-18T18:50:31.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.769+0000 7f8668edf700 1 Processor -- start 2024-03-18T18:50:31.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.770+0000 7f8668edf700 1 -- start start 2024-03-18T18:50:31.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.770+0000 7f8668edf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866406ae60 0x7f8664149a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:31.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.770+0000 7f8668edf700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86640ce550 0x7f866415b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:31.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.770+0000 7f8668edf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f866415d5a0 0x7f866415b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:31.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.770+0000 7f8668edf700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8664149fc0 con 0x7f866406ae60 2024-03-18T18:50:31.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.770+0000 7f8668edf700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866414d170 con 0x7f86640ce550 2024-03-18T18:50:31.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.770+0000 7f8668edf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866414a290 con 0x7f866415d5a0 2024-03-18T18:50:31.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.770+0000 7f866259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866406ae60 0x7f8664149a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:31.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.770+0000 7f866259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866406ae60 0x7f8664149a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47962/0 (socket says 172.21.15.1:47962) 2024-03-18T18:50:31.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.770+0000 7f866259c700 1 -- 172.21.15.1:0/897069889 learned_addr learned my addr 172.21.15.1:0/897069889 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:31.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.770+0000 7f8662d9d700 1 --2- 172.21.15.1:0/897069889 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f866415d5a0 0x7f866415b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:31.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.770+0000 7f8661d9b700 1 --2- 172.21.15.1:0/897069889 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86640ce550 0x7f866415b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:31.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.771+0000 7f866259c700 1 -- 172.21.15.1:0/897069889 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f866415d5a0 msgr2=0x7f866415b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:31.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.771+0000 7f866259c700 1 --2- 172.21.15.1:0/897069889 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f866415d5a0 0x7f866415b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.771+0000 7f866259c700 1 -- 172.21.15.1:0/897069889 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86640ce550 msgr2=0x7f866415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:31.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.771+0000 7f866259c700 1 --2- 172.21.15.1:0/897069889 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86640ce550 0x7f866415b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.771+0000 7f866259c700 1 -- 172.21.15.1:0/897069889 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865c009b30 con 0x7f866406ae60 2024-03-18T18:50:31.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.771+0000 7f8662d9d700 1 --2- 172.21.15.1:0/897069889 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f866415d5a0 0x7f866415b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:31.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.771+0000 7f8661d9b700 1 --2- 172.21.15.1:0/897069889 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86640ce550 0x7f866415b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:31.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.771+0000 7f866259c700 1 --2- 172.21.15.1:0/897069889 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866406ae60 0x7f8664149a80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f865400c7f0 tx=0x7f865400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:31.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.771+0000 7f8652ffd700 1 -- 172.21.15.1:0/897069889 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8654011800 con 0x7f866406ae60 2024-03-18T18:50:31.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.771+0000 7f8652ffd700 1 -- 172.21.15.1:0/897069889 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8654011960 con 0x7f866406ae60 2024-03-18T18:50:31.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.771+0000 7f8668edf700 1 -- 172.21.15.1:0/897069889 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f866414a570 con 0x7f866406ae60 2024-03-18T18:50:31.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.771+0000 7f8652ffd700 1 -- 172.21.15.1:0/897069889 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8654011d10 con 0x7f866406ae60 2024-03-18T18:50:31.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.771+0000 7f8668edf700 1 -- 172.21.15.1:0/897069889 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f866414ab30 con 0x7f866406ae60 2024-03-18T18:50:31.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.772+0000 7f8668edf700 1 -- 172.21.15.1:0/897069889 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f866414ad50 con 0x7f866406ae60 2024-03-18T18:50:31.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.772+0000 7f8652ffd700 1 -- 172.21.15.1:0/897069889 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8654018360 con 0x7f866406ae60 2024-03-18T18:50:31.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.773+0000 7f8652ffd700 1 --2- 172.21.15.1:0/897069889 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f864406ac90 0x7f864406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:31.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.773+0000 7f8652ffd700 1 -- 172.21.15.1:0/897069889 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f8654089580 con 0x7f866406ae60 2024-03-18T18:50:31.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.773+0000 7f8661d9b700 1 --2- 172.21.15.1:0/897069889 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f864406ac90 0x7f864406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:31.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.773+0000 7f8661d9b700 1 --2- 172.21.15.1:0/897069889 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f864406ac90 0x7f864406d150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8658003fc0 tx=0x7f8658009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:31.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.777+0000 7f8652ffd700 1 -- 172.21.15.1:0/897069889 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8654054480 con 0x7f866406ae60 2024-03-18T18:50:31.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.980+0000 7f8668edf700 1 -- 172.21.15.1:0/897069889 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f86640676a0 con 0x7f866406ae60 2024-03-18T18:50:31.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.980+0000 7f8652ffd700 1 -- 172.21.15.1:0/897069889 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f8654057aa0 con 0x7f866406ae60 2024-03-18T18:50:31.977 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:31.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.982+0000 7f8668edf700 1 -- 172.21.15.1:0/897069889 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f864406ac90 msgr2=0x7f864406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:31.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.982+0000 7f8668edf700 1 --2- 172.21.15.1:0/897069889 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f864406ac90 0x7f864406d150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8658003fc0 tx=0x7f8658009040 comp rx=0 tx=0).stop 2024-03-18T18:50:31.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.982+0000 7f8668edf700 1 -- 172.21.15.1:0/897069889 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866406ae60 msgr2=0x7f8664149a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:31.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.982+0000 7f8668edf700 1 --2- 172.21.15.1:0/897069889 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866406ae60 0x7f8664149a80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f865400c7f0 tx=0x7f865400cbb0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.982+0000 7f8668edf700 1 -- 172.21.15.1:0/897069889 shutdown_connections 2024-03-18T18:50:31.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.982+0000 7f8668edf700 1 --2- 172.21.15.1:0/897069889 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f864406ac90 0x7f864406d150 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.982+0000 7f8668edf700 1 --2- 172.21.15.1:0/897069889 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f866415d5a0 0x7f866415b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.982+0000 7f8668edf700 1 --2- 172.21.15.1:0/897069889 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866406ae60 0x7f8664149a80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.982+0000 7f8668edf700 1 --2- 172.21.15.1:0/897069889 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86640ce550 0x7f866415b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:31.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.982+0000 7f8668edf700 1 -- 172.21.15.1:0/897069889 >> 172.21.15.1:0/897069889 conn(0x7f86640cc9b0 msgr2=0x7f8664071520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:31.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.982+0000 7f8668edf700 1 -- 172.21.15.1:0/897069889 shutdown_connections 2024-03-18T18:50:31.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:31.982+0000 7f8668edf700 1 -- 172.21.15.1:0/897069889 wait complete. 2024-03-18T18:50:31.992 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:32.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.120+0000 7f6affd87700 1 Processor -- start 2024-03-18T18:50:32.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.120+0000 7f6affd87700 1 -- start start 2024-03-18T18:50:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.121+0000 7f6affd87700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6af8143c20 0x7f6af80b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.121+0000 7f6affd87700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6af806f240 0x7f6af80cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.121+0000 7f6affd87700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6af8144000 con 0x7f6af80b3f40 2024-03-18T18:50:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.121+0000 7f6affd87700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6af80b43c0 con 0x7f6af806f240 2024-03-18T18:50:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.121+0000 7f6affd87700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af80b51b0 con 0x7f6af8143c20 2024-03-18T18:50:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.121+0000 7f6afe324700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6af806f240 0x7f6af80cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.121+0000 7f6afe324700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6af806f240 0x7f6af80cddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47968/0 (socket says 172.21.15.1:47968) 2024-03-18T18:50:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.121+0000 7f6afe324700 1 -- 172.21.15.1:0/3624901131 learned_addr learned my addr 172.21.15.1:0/3624901131 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.121+0000 7f6afdb23700 1 --2- 172.21.15.1:0/3624901131 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6af8143c20 0x7f6af80b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.122+0000 7f6afe324700 1 -- 172.21.15.1:0/3624901131 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6af8143c20 msgr2=0x7f6af80b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.122+0000 7f6afe324700 1 --2- 172.21.15.1:0/3624901131 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6af8143c20 0x7f6af80b3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.122+0000 7f6afe324700 1 -- 172.21.15.1:0/3624901131 >> v1:172.21.15.165:6789/0 conn(0x7f6af80b3f40 legacy=0x7f6af806eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.122+0000 7f6afe324700 1 -- 172.21.15.1:0/3624901131 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af80cd8d0 con 0x7f6af806f240 2024-03-18T18:50:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.122+0000 7f6afe324700 1 --2- 172.21.15.1:0/3624901131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af806f240 0x7f6af80cddb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6af0009e80 tx=0x7f6af000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb161830e2efa46 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.122+0000 7f6afcb21700 1 -- 172.21.15.1:0/3624901131 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af000bb80 con 0x7f6af806f240 2024-03-18T18:50:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.122+0000 7f6afcb21700 1 -- 172.21.15.1:0/3624901131 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af000bce0 con 0x7f6af806f240 2024-03-18T18:50:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.122+0000 7f6afcb21700 1 -- 172.21.15.1:0/3624901131 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af00116a0 con 0x7f6af806f240 2024-03-18T18:50:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.122+0000 7f6affd87700 1 -- 172.21.15.1:0/3624901131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af806f240 msgr2=0x7f6af80cddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.122+0000 7f6affd87700 1 --2- 172.21.15.1:0/3624901131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af806f240 0x7f6af80cddb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6af0009e80 tx=0x7f6af000b330 comp rx=0 tx=0).stop 2024-03-18T18:50:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.122+0000 7f6affd87700 1 -- 172.21.15.1:0/3624901131 shutdown_connections 2024-03-18T18:50:32.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.122+0000 7f6affd87700 1 --2- 172.21.15.1:0/3624901131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af806f240 0x7f6af80cddb0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.122+0000 7f6affd87700 1 --2- 172.21.15.1:0/3624901131 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6af8143c20 0x7f6af80b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.122+0000 7f6affd87700 1 -- 172.21.15.1:0/3624901131 >> 172.21.15.1:0/3624901131 conn(0x7f6af80b1d20 msgr2=0x7f6af80b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:32.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.123+0000 7f6affd87700 1 -- 172.21.15.1:0/3624901131 shutdown_connections 2024-03-18T18:50:32.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.123+0000 7f6affd87700 1 -- 172.21.15.1:0/3624901131 wait complete. 2024-03-18T18:50:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.124+0000 7f6affd87700 1 Processor -- start 2024-03-18T18:50:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.124+0000 7f6affd87700 1 -- start start 2024-03-18T18:50:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6affd87700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af806f240 0x7f6af8154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6affd87700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6af80b3f40 0x7f6af81554d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6affd87700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6af8143c20 0x7f6af81f1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6affd87700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af81552d0 con 0x7f6af806f240 2024-03-18T18:50:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6affd87700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af81ee830 con 0x7f6af80b3f40 2024-03-18T18:50:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6affd87700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af8156230 con 0x7f6af8143c20 2024-03-18T18:50:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6afdb23700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af806f240 0x7f6af8154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6afdb23700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af806f240 0x7f6af8154c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47970/0 (socket says 172.21.15.1:47970) 2024-03-18T18:50:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6afdb23700 1 -- 172.21.15.1:0/1985800023 learned_addr learned my addr 172.21.15.1:0/1985800023 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6afe324700 1 --2- 172.21.15.1:0/1985800023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6af8143c20 0x7f6af81f1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:32.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6afd322700 1 --2- 172.21.15.1:0/1985800023 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6af80b3f40 0x7f6af81554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:32.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6afdb23700 1 -- 172.21.15.1:0/1985800023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6af8143c20 msgr2=0x7f6af81f1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:32.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6afdb23700 1 --2- 172.21.15.1:0/1985800023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6af8143c20 0x7f6af81f1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6afdb23700 1 -- 172.21.15.1:0/1985800023 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6af80b3f40 msgr2=0x7f6af81554d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:32.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6afdb23700 1 --2- 172.21.15.1:0/1985800023 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6af80b3f40 0x7f6af81554d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6afdb23700 1 -- 172.21.15.1:0/1985800023 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af0009b30 con 0x7f6af806f240 2024-03-18T18:50:32.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6afe324700 1 --2- 172.21.15.1:0/1985800023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6af8143c20 0x7f6af81f1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:32.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6afd322700 1 --2- 172.21.15.1:0/1985800023 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6af80b3f40 0x7f6af81554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:32.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6afdb23700 1 --2- 172.21.15.1:0/1985800023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af806f240 0x7f6af8154c40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800d800 tx=0x7f6ae800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:32.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6aee7fc700 1 -- 172.21.15.1:0/1985800023 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae80125f0 con 0x7f6af806f240 2024-03-18T18:50:32.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6aee7fc700 1 -- 172.21.15.1:0/1985800023 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae8012750 con 0x7f6af806f240 2024-03-18T18:50:32.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6affd87700 1 -- 172.21.15.1:0/1985800023 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af8156510 con 0x7f6af806f240 2024-03-18T18:50:32.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6aee7fc700 1 -- 172.21.15.1:0/1985800023 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8012b00 con 0x7f6af806f240 2024-03-18T18:50:32.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.125+0000 7f6affd87700 1 -- 172.21.15.1:0/1985800023 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af8156ad0 con 0x7f6af806f240 2024-03-18T18:50:32.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.126+0000 7f6affd87700 1 -- 172.21.15.1:0/1985800023 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af80676a0 con 0x7f6af806f240 2024-03-18T18:50:32.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.127+0000 7f6aee7fc700 1 -- 172.21.15.1:0/1985800023 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6ae8012c60 con 0x7f6af806f240 2024-03-18T18:50:32.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.127+0000 7f6aee7fc700 1 --2- 172.21.15.1:0/1985800023 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6adc06ac90 0x7f6adc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:32.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.127+0000 7f6aee7fc700 1 -- 172.21.15.1:0/1985800023 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6ae80899c0 con 0x7f6af806f240 2024-03-18T18:50:32.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.127+0000 7f6afd322700 1 --2- 172.21.15.1:0/1985800023 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6adc06ac90 0x7f6adc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:32.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.128+0000 7f6afd322700 1 --2- 172.21.15.1:0/1985800023 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6adc06ac90 0x7f6adc06d150 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6af8154b70 tx=0x7f6af4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:32.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.132+0000 7f6aee7fc700 1 -- 172.21.15.1:0/1985800023 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6ae80548b0 con 0x7f6af806f240 2024-03-18T18:50:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.331+0000 7f6affd87700 1 -- 172.21.15.1:0/1985800023 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6af80aff10 con 0x7f6af806f240 2024-03-18T18:50:32.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.332+0000 7f6aee7fc700 1 -- 172.21.15.1:0/1985800023 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17830 (secure 0 0 0) 0x7f6ae8057ed0 con 0x7f6af806f240 2024-03-18T18:50:32.330 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:32.330 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:49:49.276961+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:32.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.334+0000 7f6affd87700 1 -- 172.21.15.1:0/1985800023 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6adc06ac90 msgr2=0x7f6adc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:32.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.334+0000 7f6affd87700 1 --2- 172.21.15.1:0/1985800023 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6adc06ac90 0x7f6adc06d150 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6af8154b70 tx=0x7f6af4009040 comp rx=0 tx=0).stop 2024-03-18T18:50:32.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.334+0000 7f6affd87700 1 -- 172.21.15.1:0/1985800023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af806f240 msgr2=0x7f6af8154c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:32.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.334+0000 7f6affd87700 1 --2- 172.21.15.1:0/1985800023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af806f240 0x7f6af8154c40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800d800 tx=0x7f6ae800dbc0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.334+0000 7f6affd87700 1 -- 172.21.15.1:0/1985800023 shutdown_connections 2024-03-18T18:50:32.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.334+0000 7f6affd87700 1 --2- 172.21.15.1:0/1985800023 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6adc06ac90 0x7f6adc06d150 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.334+0000 7f6affd87700 1 --2- 172.21.15.1:0/1985800023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6af8143c20 0x7f6af81f1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.334+0000 7f6affd87700 1 --2- 172.21.15.1:0/1985800023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af806f240 0x7f6af8154c40 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.334+0000 7f6affd87700 1 --2- 172.21.15.1:0/1985800023 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6af80b3f40 0x7f6af81554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.334+0000 7f6affd87700 1 -- 172.21.15.1:0/1985800023 >> 172.21.15.1:0/1985800023 conn(0x7f6af80b1d20 msgr2=0x7f6af80b0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:32.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.335+0000 7f6affd87700 1 -- 172.21.15.1:0/1985800023 shutdown_connections 2024-03-18T18:50:32.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.335+0000 7f6affd87700 1 -- 172.21.15.1:0/1985800023 wait complete. 2024-03-18T18:50:32.344 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T18:49:42.800801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T18:49:43.776684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T18:49:46.854608+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T18:50:32.345 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T18:50:32.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.476+0000 7f2bfd72c700 1 Processor -- start 2024-03-18T18:50:32.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.476+0000 7f2bfd72c700 1 -- start start 2024-03-18T18:50:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.477+0000 7f2bfd72c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2bf80cfb50 0x7f2bf80d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.477+0000 7f2bfd72c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf80b3370 con 0x7f2bf80cee80 2024-03-18T18:50:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.477+0000 7f2bfd72c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf80cf2d0 con 0x7f2bf80cff30 2024-03-18T18:50:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.477+0000 7f2bfd72c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bf80b2760 con 0x7f2bf80cfb50 2024-03-18T18:50:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.477+0000 7f2bf77fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2bf80cee80 0x7f2bf80b3f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59522/0 (socket says 172.21.15.1:59522) 2024-03-18T18:50:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.477+0000 7f2bf77fe700 1 -- 172.21.15.1:0/702784468 learned_addr learned my addr 172.21.15.1:0/702784468 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.477+0000 7f2bf6ffd700 1 --2- 172.21.15.1:0/702784468 >> v2:172.21.15.165:3300/0 conn(0x7f2bf80cfb50 0x7f2bf80d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:32.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.477+0000 7f2bf5ffb700 1 -- 172.21.15.1:0/702784468 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880240043 0 0) 0x7f2bf80b3370 con 0x7f2bf80cee80 2024-03-18T18:50:32.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.477+0000 7f2bf5ffb700 1 -- 172.21.15.1:0/702784468 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be00035b0 con 0x7f2bf80cee80 2024-03-18T18:50:32.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.477+0000 7f2bf5ffb700 1 -- 172.21.15.1:0/702784468 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953570803 0 0) 0x7f2bf80cf2d0 con 0x7f2bf80cff30 2024-03-18T18:50:32.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.477+0000 7f2bf5ffb700 1 -- 172.21.15.1:0/702784468 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bf80b3370 con 0x7f2bf80cff30 2024-03-18T18:50:32.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.477+0000 7f2bf6ffd700 1 -- 172.21.15.1:0/702784468 >> v1:172.21.15.165:6789/0 conn(0x7f2bf80cff30 legacy=0x7f2bf80b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:32.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.477+0000 7f2bf6ffd700 1 -- 172.21.15.1:0/702784468 >> v1:172.21.15.1:6789/0 conn(0x7f2bf80cee80 legacy=0x7f2bf80b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:32.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.477+0000 7f2bf6ffd700 1 -- 172.21.15.1:0/702784468 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf80b2a40 con 0x7f2bf80cfb50 2024-03-18T18:50:32.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.478+0000 7f2bf6ffd700 1 --2- 172.21.15.1:0/702784468 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bf80cfb50 0x7f2bf80d3c60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2bec009820 tx=0x7f2bec009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcea02745ae92225 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:32.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.478+0000 7f2bf5ffb700 1 -- 172.21.15.1:0/702784468 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bec005e30 con 0x7f2bf80cfb50 2024-03-18T18:50:32.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.478+0000 7f2bf5ffb700 1 -- 172.21.15.1:0/702784468 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bec006030 con 0x7f2bf80cfb50 2024-03-18T18:50:32.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.478+0000 7f2bf5ffb700 1 -- 172.21.15.1:0/702784468 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bec0063c0 con 0x7f2bf80cfb50 2024-03-18T18:50:32.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.478+0000 7f2bfd72c700 1 -- 172.21.15.1:0/702784468 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bf80cfb50 msgr2=0x7f2bf80d3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:32.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.478+0000 7f2bfd72c700 1 --2- 172.21.15.1:0/702784468 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bf80cfb50 0x7f2bf80d3c60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2bec009820 tx=0x7f2bec009b30 comp rx=0 tx=0).stop 2024-03-18T18:50:32.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.478+0000 7f2bfd72c700 1 -- 172.21.15.1:0/702784468 shutdown_connections 2024-03-18T18:50:32.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.478+0000 7f2bfd72c700 1 --2- 172.21.15.1:0/702784468 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bf80cfb50 0x7f2bf80d3c60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.478+0000 7f2bfd72c700 1 -- 172.21.15.1:0/702784468 >> 172.21.15.1:0/702784468 conn(0x7f2bf80d3140 msgr2=0x7f2bf80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:32.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.478+0000 7f2bfd72c700 1 -- 172.21.15.1:0/702784468 shutdown_connections 2024-03-18T18:50:32.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.478+0000 7f2bfd72c700 1 -- 172.21.15.1:0/702784468 wait complete. 2024-03-18T18:50:32.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.480+0000 7f2bfd72c700 1 Processor -- start 2024-03-18T18:50:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.480+0000 7f2bfd72c700 1 -- start start 2024-03-18T18:50:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bfd72c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2bf80cee80 0x7f2bf81579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bfd72c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bf80cfb50 0x7f2bf80c7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bfd72c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bf80cff30 0x7f2bf80b7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bfd72c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf8158b20 con 0x7f2bf80cff30 2024-03-18T18:50:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bfd72c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf80c9320 con 0x7f2bf80cfb50 2024-03-18T18:50:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bfd72c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf80c74b0 con 0x7f2bf80cee80 2024-03-18T18:50:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bf77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bf80cff30 0x7f2bf80b7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bf77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bf80cff30 0x7f2bf80b7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47984/0 (socket says 172.21.15.1:47984) 2024-03-18T18:50:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bf77fe700 1 -- 172.21.15.1:0/4006333419 learned_addr learned my addr 172.21.15.1:0/4006333419 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bf6ffd700 1 --2- 172.21.15.1:0/4006333419 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2bf80cee80 0x7f2bf81579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bf77fe700 1 -- 172.21.15.1:0/4006333419 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2bf80cee80 msgr2=0x7f2bf81579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bf77fe700 1 --2- 172.21.15.1:0/4006333419 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2bf80cee80 0x7f2bf81579d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bf77fe700 1 -- 172.21.15.1:0/4006333419 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bf80cfb50 msgr2=0x7f2bf80c7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:32.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bf67fc700 1 --2- 172.21.15.1:0/4006333419 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bf80cfb50 0x7f2bf80c7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:32.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bf77fe700 1 --2- 172.21.15.1:0/4006333419 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bf80cfb50 0x7f2bf80c7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bf77fe700 1 -- 172.21.15.1:0/4006333419 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bec009430 con 0x7f2bf80cff30 2024-03-18T18:50:32.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bf77fe700 1 --2- 172.21.15.1:0/4006333419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bf80cff30 0x7f2bf80b7d10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2bf000cbc0 tx=0x7f2bf000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:32.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bdf7fe700 1 -- 172.21.15.1:0/4006333419 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf000ab50 con 0x7f2bf80cff30 2024-03-18T18:50:32.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bdf7fe700 1 -- 172.21.15.1:0/4006333419 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf000acb0 con 0x7f2bf80cff30 2024-03-18T18:50:32.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bfd72c700 1 -- 172.21.15.1:0/4006333419 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf80c9570 con 0x7f2bf80cff30 2024-03-18T18:50:32.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bdf7fe700 1 -- 172.21.15.1:0/4006333419 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0012600 con 0x7f2bf80cff30 2024-03-18T18:50:32.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.481+0000 7f2bfd72c700 1 -- 172.21.15.1:0/4006333419 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf80c7700 con 0x7f2bf80cff30 2024-03-18T18:50:32.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.482+0000 7f2bfd72c700 1 -- 172.21.15.1:0/4006333419 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf80c7920 con 0x7f2bf80cff30 2024-03-18T18:50:32.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.483+0000 7f2bdf7fe700 1 -- 172.21.15.1:0/4006333419 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2bf001b3b0 con 0x7f2bf80cff30 2024-03-18T18:50:32.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.483+0000 7f2bdf7fe700 1 --2- 172.21.15.1:0/4006333419 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2be406ac90 0x7f2be406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:32.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.484+0000 7f2bf6ffd700 1 --2- 172.21.15.1:0/4006333419 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2be406ac90 0x7f2be406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:32.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.484+0000 7f2bdf7fe700 1 -- 172.21.15.1:0/4006333419 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2bf0089320 con 0x7f2bf80cff30 2024-03-18T18:50:32.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.484+0000 7f2bf6ffd700 1 --2- 172.21.15.1:0/4006333419 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2be406ac90 0x7f2be406d150 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2bec009b70 tx=0x7f2bec008070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:32.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.488+0000 7f2bdf7fe700 1 -- 172.21.15.1:0/4006333419 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2bf00542c0 con 0x7f2bf80cff30 2024-03-18T18:50:32.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.688+0000 7f2bfd72c700 1 -- 172.21.15.1:0/4006333419 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2bf8073350 con 0x7f2bf80cff30 2024-03-18T18:50:32.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.688+0000 7f2bdf7fe700 1 -- 172.21.15.1:0/4006333419 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2bf00578e0 con 0x7f2bf80cff30 2024-03-18T18:50:32.686 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T18:50:32.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.690+0000 7f2bfd72c700 1 -- 172.21.15.1:0/4006333419 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2be406ac90 msgr2=0x7f2be406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:32.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.690+0000 7f2bfd72c700 1 --2- 172.21.15.1:0/4006333419 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2be406ac90 0x7f2be406d150 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2bec009b70 tx=0x7f2bec008070 comp rx=0 tx=0).stop 2024-03-18T18:50:32.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.690+0000 7f2bfd72c700 1 -- 172.21.15.1:0/4006333419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bf80cff30 msgr2=0x7f2bf80b7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:32.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.690+0000 7f2bfd72c700 1 --2- 172.21.15.1:0/4006333419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bf80cff30 0x7f2bf80b7d10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2bf000cbc0 tx=0x7f2bf000cf80 comp rx=0 tx=0).stop 2024-03-18T18:50:32.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.690+0000 7f2bfd72c700 1 -- 172.21.15.1:0/4006333419 shutdown_connections 2024-03-18T18:50:32.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.690+0000 7f2bfd72c700 1 --2- 172.21.15.1:0/4006333419 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2be406ac90 0x7f2be406d150 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.690+0000 7f2bfd72c700 1 --2- 172.21.15.1:0/4006333419 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2bf80cee80 0x7f2bf81579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.690+0000 7f2bfd72c700 1 --2- 172.21.15.1:0/4006333419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bf80cff30 0x7f2bf80b7d10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.690+0000 7f2bfd72c700 1 --2- 172.21.15.1:0/4006333419 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bf80cfb50 0x7f2bf80c7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.690+0000 7f2bfd72c700 1 -- 172.21.15.1:0/4006333419 >> 172.21.15.1:0/4006333419 conn(0x7f2bf80d3140 msgr2=0x7f2bf80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:32.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.690+0000 7f2bfd72c700 1 -- 172.21.15.1:0/4006333419 shutdown_connections 2024-03-18T18:50:32.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.690+0000 7f2bfd72c700 1 -- 172.21.15.1:0/4006333419 wait complete. 2024-03-18T18:50:32.700 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T18:50:32.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.831+0000 7ffbb90fa700 1 Processor -- start 2024-03-18T18:50:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.831+0000 7ffbb90fa700 1 -- start start 2024-03-18T18:50:32.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.832+0000 7ffbb90fa700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbb40cfa90 0x7ffbb406a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:32.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.832+0000 7ffbb90fa700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffbb406ac90 0x7ffbb4167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:32.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.832+0000 7ffbb90fa700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb40cf600 con 0x7ffbb415d5a0 2024-03-18T18:50:32.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.832+0000 7ffbb90fa700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbb406b140 con 0x7ffbb406ac90 2024-03-18T18:50:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.832+0000 7ffbb90fa700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb40ce720 con 0x7ffbb40cfa90 2024-03-18T18:50:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.832+0000 7ffbb359e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffbb406ac90 0x7ffbb4167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.832+0000 7ffbb2d9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ffbb415d5a0 0x7ffbb40ce010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59536/0 (socket says 172.21.15.1:59536) 2024-03-18T18:50:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.832+0000 7ffbb2d9d700 1 -- 172.21.15.1:0/2612785511 learned_addr learned my addr 172.21.15.1:0/2612785511 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.832+0000 7ffbb259c700 1 --2- 172.21.15.1:0/2612785511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbb40cfa90 0x7ffbb406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.833+0000 7ffbb359e700 1 -- 172.21.15.1:0/2612785511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbb40cfa90 msgr2=0x7ffbb406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.833+0000 7ffbb359e700 1 --2- 172.21.15.1:0/2612785511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbb40cfa90 0x7ffbb406a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.833+0000 7ffbb359e700 1 -- 172.21.15.1:0/2612785511 >> v1:172.21.15.1:6789/0 conn(0x7ffbb415d5a0 legacy=0x7ffbb40ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.833+0000 7ffbb359e700 1 -- 172.21.15.1:0/2612785511 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb40ce9f0 con 0x7ffbb406ac90 2024-03-18T18:50:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.833+0000 7ffbb1d9b700 1 -- 172.21.15.1:0/2612785511 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180427558 0 0) 0x7ffbb40cf600 con 0x7ffbb415d5a0 2024-03-18T18:50:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.833+0000 7ffbb359e700 1 --2- 172.21.15.1:0/2612785511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbb406ac90 0x7ffbb4167140 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ffbac009dc0 tx=0x7ffbac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc68be828fc7ae5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.833+0000 7ffbb1d9b700 1 -- 172.21.15.1:0/2612785511 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac00e070 con 0x7ffbb406ac90 2024-03-18T18:50:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.833+0000 7ffbb1d9b700 1 -- 172.21.15.1:0/2612785511 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbac00bae0 con 0x7ffbb406ac90 2024-03-18T18:50:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.833+0000 7ffbb1d9b700 1 -- 172.21.15.1:0/2612785511 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac00be90 con 0x7ffbb406ac90 2024-03-18T18:50:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.834+0000 7ffbb90fa700 1 -- 172.21.15.1:0/2612785511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbb406ac90 msgr2=0x7ffbb4167140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.834+0000 7ffbb90fa700 1 --2- 172.21.15.1:0/2612785511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbb406ac90 0x7ffbb4167140 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ffbac009dc0 tx=0x7ffbac00b330 comp rx=0 tx=0).stop 2024-03-18T18:50:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.834+0000 7ffbb90fa700 1 -- 172.21.15.1:0/2612785511 shutdown_connections 2024-03-18T18:50:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.834+0000 7ffbb90fa700 1 --2- 172.21.15.1:0/2612785511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbb406ac90 0x7ffbb4167140 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.834+0000 7ffbb90fa700 1 --2- 172.21.15.1:0/2612785511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbb40cfa90 0x7ffbb406a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.834+0000 7ffbb90fa700 1 -- 172.21.15.1:0/2612785511 >> 172.21.15.1:0/2612785511 conn(0x7ffbb40cc9b0 msgr2=0x7ffbb40ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.834+0000 7ffbb90fa700 1 -- 172.21.15.1:0/2612785511 shutdown_connections 2024-03-18T18:50:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.834+0000 7ffbb90fa700 1 -- 172.21.15.1:0/2612785511 wait complete. 2024-03-18T18:50:32.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.835+0000 7ffbb90fa700 1 Processor -- start 2024-03-18T18:50:32.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.835+0000 7ffbb90fa700 1 -- start start 2024-03-18T18:50:32.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.836+0000 7ffbb90fa700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbb406ac90 0x7ffbb4160790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:32.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.836+0000 7ffbb90fa700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffbb40cfa90 0x7ffbb415e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:32.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.836+0000 7ffbb90fa700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbb415d5a0 0x7ffbb41f6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:32.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.836+0000 7ffbb90fa700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb415eab0 con 0x7ffbb406ac90 2024-03-18T18:50:32.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.836+0000 7ffbb90fa700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb415ecc0 con 0x7ffbb40cfa90 2024-03-18T18:50:32.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.836+0000 7ffbb90fa700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb415efa0 con 0x7ffbb415d5a0 2024-03-18T18:50:32.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.836+0000 7ffbb2d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbb406ac90 0x7ffbb4160790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:32.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.836+0000 7ffbb2d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbb406ac90 0x7ffbb4160790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48002/0 (socket says 172.21.15.1:48002) 2024-03-18T18:50:32.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.836+0000 7ffbb2d9d700 1 -- 172.21.15.1:0/3289063843 learned_addr learned my addr 172.21.15.1:0/3289063843 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:32.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.836+0000 7ffbb259c700 1 --2- 172.21.15.1:0/3289063843 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffbb40cfa90 0x7ffbb415e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:32.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.837+0000 7ffbb359e700 1 --2- 172.21.15.1:0/3289063843 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbb415d5a0 0x7ffbb41f6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:32.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.837+0000 7ffbb2d9d700 1 -- 172.21.15.1:0/3289063843 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbb415d5a0 msgr2=0x7ffbb41f6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:32.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.837+0000 7ffbb2d9d700 1 --2- 172.21.15.1:0/3289063843 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbb415d5a0 0x7ffbb41f6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.837+0000 7ffbb2d9d700 1 -- 172.21.15.1:0/3289063843 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffbb40cfa90 msgr2=0x7ffbb415e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:32.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.837+0000 7ffbb2d9d700 1 --2- 172.21.15.1:0/3289063843 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffbb40cfa90 0x7ffbb415e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:32.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.837+0000 7ffbb2d9d700 1 -- 172.21.15.1:0/3289063843 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbac009a70 con 0x7ffbb406ac90 2024-03-18T18:50:32.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.837+0000 7ffbb2d9d700 1 --2- 172.21.15.1:0/3289063843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbb406ac90 0x7ffbb4160790 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ffba40010e0 tx=0x7ffba400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:32.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.837+0000 7ffb9b7fe700 1 -- 172.21.15.1:0/3289063843 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba40074e0 con 0x7ffbb406ac90 2024-03-18T18:50:32.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.837+0000 7ffb9b7fe700 1 -- 172.21.15.1:0/3289063843 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba4007640 con 0x7ffbb406ac90 2024-03-18T18:50:32.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.837+0000 7ffbb90fa700 1 -- 172.21.15.1:0/3289063843 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb40c81f0 con 0x7ffbb406ac90 2024-03-18T18:50:32.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.837+0000 7ffb9b7fe700 1 -- 172.21.15.1:0/3289063843 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba4005a70 con 0x7ffbb406ac90 2024-03-18T18:50:32.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.837+0000 7ffbb90fa700 1 -- 172.21.15.1:0/3289063843 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb40c87b0 con 0x7ffbb406ac90 2024-03-18T18:50:32.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.839+0000 7ffb9b7fe700 1 -- 172.21.15.1:0/3289063843 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ffba4005bd0 con 0x7ffbb406ac90 2024-03-18T18:50:32.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.839+0000 7ffbb90fa700 1 -- 172.21.15.1:0/3289063843 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb4073350 con 0x7ffbb406ac90 2024-03-18T18:50:32.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.839+0000 7ffb9b7fe700 1 --2- 172.21.15.1:0/3289063843 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffba006acc0 0x7ffba006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:32.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.839+0000 7ffb9b7fe700 1 -- 172.21.15.1:0/3289063843 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ffba40889a0 con 0x7ffbb406ac90 2024-03-18T18:50:32.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.839+0000 7ffbb259c700 1 --2- 172.21.15.1:0/3289063843 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffba006acc0 0x7ffba006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:32.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.840+0000 7ffbb259c700 1 --2- 172.21.15.1:0/3289063843 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffba006acc0 0x7ffba006d180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ffba8009be0 tx=0x7ffba8009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:32.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:32.844+0000 7ffb9b7fe700 1 -- 172.21.15.1:0/3289063843 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffba4053920 con 0x7ffbb406ac90 2024-03-18T18:50:33.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.048+0000 7ffbb90fa700 1 -- 172.21.15.1:0/3289063843 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffbb40cd4e0 con 0x7ffbb406ac90 2024-03-18T18:50:33.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.048+0000 7ffb9b7fe700 1 -- 172.21.15.1:0/3289063843 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffba4017020 con 0x7ffbb406ac90 2024-03-18T18:50:33.046 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:33.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.050+0000 7ffbb90fa700 1 -- 172.21.15.1:0/3289063843 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffba006acc0 msgr2=0x7ffba006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:33.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.050+0000 7ffbb90fa700 1 --2- 172.21.15.1:0/3289063843 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffba006acc0 0x7ffba006d180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ffba8009be0 tx=0x7ffba8009210 comp rx=0 tx=0).stop 2024-03-18T18:50:33.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.050+0000 7ffbb90fa700 1 -- 172.21.15.1:0/3289063843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbb406ac90 msgr2=0x7ffbb4160790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:33.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.050+0000 7ffbb90fa700 1 --2- 172.21.15.1:0/3289063843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbb406ac90 0x7ffbb4160790 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ffba40010e0 tx=0x7ffba400ca30 comp rx=0 tx=0).stop 2024-03-18T18:50:33.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.050+0000 7ffbb90fa700 1 -- 172.21.15.1:0/3289063843 shutdown_connections 2024-03-18T18:50:33.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.050+0000 7ffbb90fa700 1 --2- 172.21.15.1:0/3289063843 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffba006acc0 0x7ffba006d180 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.050+0000 7ffbb90fa700 1 --2- 172.21.15.1:0/3289063843 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbb415d5a0 0x7ffbb41f6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.050+0000 7ffbb90fa700 1 --2- 172.21.15.1:0/3289063843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbb406ac90 0x7ffbb4160790 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.050+0000 7ffbb90fa700 1 --2- 172.21.15.1:0/3289063843 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffbb40cfa90 0x7ffbb415e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.050+0000 7ffbb90fa700 1 -- 172.21.15.1:0/3289063843 >> 172.21.15.1:0/3289063843 conn(0x7ffbb40cc9b0 msgr2=0x7ffbb415b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:33.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.050+0000 7ffbb90fa700 1 -- 172.21.15.1:0/3289063843 shutdown_connections 2024-03-18T18:50:33.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.050+0000 7ffbb90fa700 1 -- 172.21.15.1:0/3289063843 wait complete. 2024-03-18T18:50:33.059 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T18:50:33.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.190+0000 7f4249c5c700 1 Processor -- start 2024-03-18T18:50:33.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.190+0000 7f4249c5c700 1 -- start start 2024-03-18T18:50:33.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.191+0000 7f4249c5c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4244070580 0x7f4244070a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.191+0000 7f4249c5c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f424406d3d0 0x7f424406d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.191+0000 7f4249c5c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42440b38d0 con 0x7f42440ce8d0 2024-03-18T18:50:33.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.191+0000 7f4249c5c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f424406dda0 con 0x7f424406d3d0 2024-03-18T18:50:33.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.191+0000 7f4249c5c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42440b4470 con 0x7f4244070580 2024-03-18T18:50:33.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.191+0000 7f4243fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f424406d3d0 0x7f424406d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.191+0000 7f4243fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f424406d3d0 0x7f424406d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48008/0 (socket says 172.21.15.1:48008) 2024-03-18T18:50:33.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.191+0000 7f4243fff700 1 -- 172.21.15.1:0/1544237683 learned_addr learned my addr 172.21.15.1:0/1544237683 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:33.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.191+0000 7f4242ffd700 1 --2- 172.21.15.1:0/1544237683 >> v2:172.21.15.165:3300/0 conn(0x7f4244070580 0x7f4244070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.191+0000 7f4243fff700 1 -- 172.21.15.1:0/1544237683 >> v2:172.21.15.165:3300/0 conn(0x7f4244070580 msgr2=0x7f4244070a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:33.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.191+0000 7f4243fff700 1 --2- 172.21.15.1:0/1544237683 >> v2:172.21.15.165:3300/0 conn(0x7f4244070580 0x7f4244070a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.191+0000 7f4243fff700 1 -- 172.21.15.1:0/1544237683 >> v1:172.21.15.165:6789/0 conn(0x7f42440ce8d0 legacy=0x7f42440b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:33.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.191+0000 7f4243fff700 1 -- 172.21.15.1:0/1544237683 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42440cecb0 con 0x7f424406d3d0 2024-03-18T18:50:33.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.191+0000 7f4242ffd700 1 --2- 172.21.15.1:0/1544237683 >> v2:172.21.15.165:3300/0 conn(0x7f4244070580 0x7f4244070a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:33.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.191+0000 7f4243fff700 1 --2- 172.21.15.1:0/1544237683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f424406d3d0 0x7f424406d860 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f424406e750 tx=0x7f423c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=951dd69ddf75482c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:33.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.192+0000 7f42427fc700 1 -- 172.21.15.1:0/1544237683 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c00bbc0 con 0x7f424406d3d0 2024-03-18T18:50:33.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.192+0000 7f42427fc700 1 -- 172.21.15.1:0/1544237683 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423c00bd20 con 0x7f424406d3d0 2024-03-18T18:50:33.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.192+0000 7f42427fc700 1 -- 172.21.15.1:0/1544237683 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c0116a0 con 0x7f424406d3d0 2024-03-18T18:50:33.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.192+0000 7f4249c5c700 1 -- 172.21.15.1:0/1544237683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f424406d3d0 msgr2=0x7f424406d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:33.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.192+0000 7f4249c5c700 1 --2- 172.21.15.1:0/1544237683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f424406d3d0 0x7f424406d860 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f424406e750 tx=0x7f423c00b330 comp rx=0 tx=0).stop 2024-03-18T18:50:33.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.192+0000 7f4249c5c700 1 -- 172.21.15.1:0/1544237683 shutdown_connections 2024-03-18T18:50:33.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.192+0000 7f4249c5c700 1 --2- 172.21.15.1:0/1544237683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f424406d3d0 0x7f424406d860 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.192+0000 7f4249c5c700 1 --2- 172.21.15.1:0/1544237683 >> v2:172.21.15.165:3300/0 conn(0x7f4244070580 0x7f4244070a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.192+0000 7f4249c5c700 1 -- 172.21.15.1:0/1544237683 >> 172.21.15.1:0/1544237683 conn(0x7f424406ab70 msgr2=0x7f424406af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:33.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.192+0000 7f4249c5c700 1 -- 172.21.15.1:0/1544237683 shutdown_connections 2024-03-18T18:50:33.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.192+0000 7f4249c5c700 1 -- 172.21.15.1:0/1544237683 wait complete. 2024-03-18T18:50:33.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.193+0000 7f4249c5c700 1 Processor -- start 2024-03-18T18:50:33.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.194+0000 7f4249c5c700 1 -- start start 2024-03-18T18:50:33.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.194+0000 7f4249c5c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f424406d3d0 0x7f42440ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.194+0000 7f4249c5c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4244070580 0x7f42441f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.194+0000 7f4249c5c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42440ce8d0 0x7f424415ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.194+0000 7f4249c5c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424415afd0 con 0x7f42440ce8d0 2024-03-18T18:50:33.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.194+0000 7f4249c5c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42441f36c0 con 0x7f4244070580 2024-03-18T18:50:33.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.194+0000 7f4249c5c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42440acf20 con 0x7f424406d3d0 2024-03-18T18:50:33.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.194+0000 7f4243fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42440ce8d0 0x7f424415ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.194+0000 7f4243fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42440ce8d0 0x7f424415ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48024/0 (socket says 172.21.15.1:48024) 2024-03-18T18:50:33.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.195+0000 7f4243fff700 1 -- 172.21.15.1:0/3693004724 learned_addr learned my addr 172.21.15.1:0/3693004724 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:33.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.195+0000 7f4242ffd700 1 --2- 172.21.15.1:0/3693004724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4244070580 0x7f42441f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.195+0000 7f42437fe700 1 --2- 172.21.15.1:0/3693004724 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f424406d3d0 0x7f42440ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.195+0000 7f4243fff700 1 -- 172.21.15.1:0/3693004724 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f424406d3d0 msgr2=0x7f42440ac8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:33.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.195+0000 7f4243fff700 1 --2- 172.21.15.1:0/3693004724 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f424406d3d0 0x7f42440ac8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.195+0000 7f4243fff700 1 -- 172.21.15.1:0/3693004724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4244070580 msgr2=0x7f42441f3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:33.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.195+0000 7f4243fff700 1 --2- 172.21.15.1:0/3693004724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4244070580 0x7f42441f3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.195+0000 7f4243fff700 1 -- 172.21.15.1:0/3693004724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423c009b30 con 0x7f42440ce8d0 2024-03-18T18:50:33.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.195+0000 7f4242ffd700 1 --2- 172.21.15.1:0/3693004724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4244070580 0x7f42441f3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:33.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.195+0000 7f4243fff700 1 --2- 172.21.15.1:0/3693004724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42440ce8d0 0x7f424415ce70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f423c00b370 tx=0x7f423c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:33.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.195+0000 7f423bfff700 1 -- 172.21.15.1:0/3693004724 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c00cea0 con 0x7f42440ce8d0 2024-03-18T18:50:33.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.195+0000 7f423bfff700 1 -- 172.21.15.1:0/3693004724 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423c011c60 con 0x7f42440ce8d0 2024-03-18T18:50:33.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.195+0000 7f4249c5c700 1 -- 172.21.15.1:0/3693004724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42440ad200 con 0x7f42440ce8d0 2024-03-18T18:50:33.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.195+0000 7f423bfff700 1 -- 172.21.15.1:0/3693004724 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c0099b0 con 0x7f42440ce8d0 2024-03-18T18:50:33.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.195+0000 7f4249c5c700 1 -- 172.21.15.1:0/3693004724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42440ad7c0 con 0x7f42440ce8d0 2024-03-18T18:50:33.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.196+0000 7f4249c5c700 1 -- 172.21.15.1:0/3693004724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f424415b110 con 0x7f42440ce8d0 2024-03-18T18:50:33.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.197+0000 7f423bfff700 1 -- 172.21.15.1:0/3693004724 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f423c004040 con 0x7f42440ce8d0 2024-03-18T18:50:33.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.197+0000 7f423bfff700 1 --2- 172.21.15.1:0/3693004724 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f422406ac90 0x7f422406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.197+0000 7f423bfff700 1 -- 172.21.15.1:0/3693004724 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f423c08ad70 con 0x7f42440ce8d0 2024-03-18T18:50:33.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.197+0000 7f42437fe700 1 --2- 172.21.15.1:0/3693004724 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f422406ac90 0x7f422406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.197+0000 7f42437fe700 1 --2- 172.21.15.1:0/3693004724 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f422406ac90 0x7f422406d150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f424415e160 tx=0x7f422c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:33.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.201+0000 7f423bfff700 1 -- 172.21.15.1:0/3693004724 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f424415b110 con 0x7f42440ce8d0 2024-03-18T18:50:33.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.403+0000 7f4249c5c700 1 -- 172.21.15.1:0/3693004724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f424415b110 con 0x7f42440ce8d0 2024-03-18T18:50:33.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.403+0000 7f423bfff700 1 -- 172.21.15.1:0/3693004724 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f424415b110 con 0x7f42440ce8d0 2024-03-18T18:50:33.401 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T18:50:33.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.405+0000 7f4249c5c700 1 -- 172.21.15.1:0/3693004724 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f422406ac90 msgr2=0x7f422406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:33.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.405+0000 7f4249c5c700 1 --2- 172.21.15.1:0/3693004724 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f422406ac90 0x7f422406d150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f424415e160 tx=0x7f422c009040 comp rx=0 tx=0).stop 2024-03-18T18:50:33.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.405+0000 7f4249c5c700 1 -- 172.21.15.1:0/3693004724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42440ce8d0 msgr2=0x7f424415ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:33.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.405+0000 7f4249c5c700 1 --2- 172.21.15.1:0/3693004724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42440ce8d0 0x7f424415ce70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f423c00b370 tx=0x7f423c004ad0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.405+0000 7f4249c5c700 1 -- 172.21.15.1:0/3693004724 shutdown_connections 2024-03-18T18:50:33.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.405+0000 7f4249c5c700 1 --2- 172.21.15.1:0/3693004724 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f422406ac90 0x7f422406d150 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.405+0000 7f4249c5c700 1 --2- 172.21.15.1:0/3693004724 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f424406d3d0 0x7f42440ac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.405+0000 7f4249c5c700 1 --2- 172.21.15.1:0/3693004724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42440ce8d0 0x7f424415ce70 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.405+0000 7f4249c5c700 1 --2- 172.21.15.1:0/3693004724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4244070580 0x7f42441f3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.405+0000 7f4249c5c700 1 -- 172.21.15.1:0/3693004724 >> 172.21.15.1:0/3693004724 conn(0x7f424406ab70 msgr2=0x7f424406cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:33.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.405+0000 7f4249c5c700 1 -- 172.21.15.1:0/3693004724 shutdown_connections 2024-03-18T18:50:33.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.405+0000 7f4249c5c700 1 -- 172.21.15.1:0/3693004724 wait complete. 2024-03-18T18:50:33.415 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T18:50:33.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.546+0000 7f6e8ad3a700 1 Processor -- start 2024-03-18T18:50:33.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.546+0000 7f6e8ad3a700 1 -- start start 2024-03-18T18:50:33.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.547+0000 7f6e8ad3a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6e8415a1d0 0x7f6e840add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.547+0000 7f6e8ad3a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e84071620 0x7f6e84071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.547+0000 7f6e8ad3a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6e840cb9e0 0x7f6e840cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.547+0000 7f6e8ad3a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e840cdb00 con 0x7f6e8415a1d0 2024-03-18T18:50:33.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.547+0000 7f6e8ad3a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e8415a5b0 con 0x7f6e840cb9e0 2024-03-18T18:50:33.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.547+0000 7f6e8ad3a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840ce8f0 con 0x7f6e84071620 2024-03-18T18:50:33.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.547+0000 7f6e88ad6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6e8415a1d0 0x7f6e840add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.547+0000 7f6e88ad6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6e8415a1d0 0x7f6e840add20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48028/0 (socket says 172.21.15.1:48028) 2024-03-18T18:50:33.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.547+0000 7f6e88ad6700 1 -- 172.21.15.1:0/1767294222 learned_addr learned my addr 172.21.15.1:0/1767294222 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:33.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.547+0000 7f6e892d7700 1 --2- 172.21.15.1:0/1767294222 >> v2:172.21.15.165:3300/0 conn(0x7f6e840cb9e0 0x7f6e840cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.547+0000 7f6e83fff700 1 --2- 172.21.15.1:0/1767294222 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e84071620 0x7f6e84071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.548+0000 7f6e88ad6700 1 -- 172.21.15.1:0/1767294222 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e84071620 msgr2=0x7f6e84071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:33.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.548+0000 7f6e88ad6700 1 --2- 172.21.15.1:0/1767294222 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e84071620 0x7f6e84071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.548+0000 7f6e88ad6700 1 -- 172.21.15.1:0/1767294222 >> v2:172.21.15.165:3300/0 conn(0x7f6e840cb9e0 msgr2=0x7f6e840cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:33.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.548+0000 7f6e88ad6700 1 --2- 172.21.15.1:0/1767294222 >> v2:172.21.15.165:3300/0 conn(0x7f6e840cb9e0 0x7f6e840cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.548+0000 7f6e88ad6700 1 -- 172.21.15.1:0/1767294222 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e84071fc0 con 0x7f6e8415a1d0 2024-03-18T18:50:33.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.548+0000 7f6e83fff700 1 --2- 172.21.15.1:0/1767294222 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e84071620 0x7f6e84071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:33.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.548+0000 7f6e88ad6700 1 --2- 172.21.15.1:0/1767294222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e8415a1d0 0x7f6e840add20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6e840cfe60 tx=0x7f6e78009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ace411a0aef4f86d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:33.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.548+0000 7f6e837fe700 1 -- 172.21.15.1:0/1767294222 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e78005e30 con 0x7f6e8415a1d0 2024-03-18T18:50:33.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.548+0000 7f6e837fe700 1 -- 172.21.15.1:0/1767294222 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e78004030 con 0x7f6e8415a1d0 2024-03-18T18:50:33.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.548+0000 7f6e837fe700 1 -- 172.21.15.1:0/1767294222 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e780043e0 con 0x7f6e8415a1d0 2024-03-18T18:50:33.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.548+0000 7f6e8ad3a700 1 -- 172.21.15.1:0/1767294222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e8415a1d0 msgr2=0x7f6e840add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:33.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.548+0000 7f6e8ad3a700 1 --2- 172.21.15.1:0/1767294222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e8415a1d0 0x7f6e840add20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6e840cfe60 tx=0x7f6e78009bd0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.548+0000 7f6e8ad3a700 1 -- 172.21.15.1:0/1767294222 shutdown_connections 2024-03-18T18:50:33.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.548+0000 7f6e8ad3a700 1 --2- 172.21.15.1:0/1767294222 >> v2:172.21.15.165:3300/0 conn(0x7f6e840cb9e0 0x7f6e840cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.548+0000 7f6e8ad3a700 1 --2- 172.21.15.1:0/1767294222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e8415a1d0 0x7f6e840add20 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.548+0000 7f6e8ad3a700 1 --2- 172.21.15.1:0/1767294222 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e84071620 0x7f6e84071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.548+0000 7f6e8ad3a700 1 -- 172.21.15.1:0/1767294222 >> 172.21.15.1:0/1767294222 conn(0x7f6e840cc9b0 msgr2=0x7f6e840ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:33.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.548+0000 7f6e8ad3a700 1 -- 172.21.15.1:0/1767294222 shutdown_connections 2024-03-18T18:50:33.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.548+0000 7f6e8ad3a700 1 -- 172.21.15.1:0/1767294222 wait complete. 2024-03-18T18:50:33.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.550+0000 7f6e8ad3a700 1 Processor -- start 2024-03-18T18:50:33.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.550+0000 7f6e8ad3a700 1 -- start start 2024-03-18T18:50:33.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e8ad3a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e84071620 0x7f6e8415c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e8ad3a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e840cb9e0 0x7f6e840b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e8ad3a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e8415a1d0 0x7f6e84150030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e8ad3a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840c4730 con 0x7f6e84071620 2024-03-18T18:50:33.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e8ad3a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8415c6a0 con 0x7f6e8415a1d0 2024-03-18T18:50:33.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e8ad3a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8415c8d0 con 0x7f6e840cb9e0 2024-03-18T18:50:33.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e88ad6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e84071620 0x7f6e8415c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e88ad6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e84071620 0x7f6e8415c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48040/0 (socket says 172.21.15.1:48040) 2024-03-18T18:50:33.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e88ad6700 1 -- 172.21.15.1:0/1061068765 learned_addr learned my addr 172.21.15.1:0/1061068765 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:33.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e83fff700 1 --2- 172.21.15.1:0/1061068765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e840cb9e0 0x7f6e840b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e892d7700 1 --2- 172.21.15.1:0/1061068765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e8415a1d0 0x7f6e84150030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e88ad6700 1 -- 172.21.15.1:0/1061068765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e840cb9e0 msgr2=0x7f6e840b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:33.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e88ad6700 1 --2- 172.21.15.1:0/1061068765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e840cb9e0 0x7f6e840b54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e88ad6700 1 -- 172.21.15.1:0/1061068765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e8415a1d0 msgr2=0x7f6e84150030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:33.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e88ad6700 1 --2- 172.21.15.1:0/1061068765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e8415a1d0 0x7f6e84150030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e88ad6700 1 -- 172.21.15.1:0/1061068765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e78009500 con 0x7f6e84071620 2024-03-18T18:50:33.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e88ad6700 1 --2- 172.21.15.1:0/1061068765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e84071620 0x7f6e8415c160 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009130 tx=0x7f6e78004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:33.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e817fa700 1 -- 172.21.15.1:0/1061068765 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7800f8a0 con 0x7f6e84071620 2024-03-18T18:50:33.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e817fa700 1 -- 172.21.15.1:0/1061068765 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7800fa00 con 0x7f6e84071620 2024-03-18T18:50:33.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e8ad3a700 1 -- 172.21.15.1:0/1061068765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e8415cbb0 con 0x7f6e84071620 2024-03-18T18:50:33.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e817fa700 1 -- 172.21.15.1:0/1061068765 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7800fd90 con 0x7f6e84071620 2024-03-18T18:50:33.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.551+0000 7f6e8ad3a700 1 -- 172.21.15.1:0/1061068765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e840b5a20 con 0x7f6e84071620 2024-03-18T18:50:33.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.552+0000 7f6e8ad3a700 1 -- 172.21.15.1:0/1061068765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e8415d0d0 con 0x7f6e84071620 2024-03-18T18:50:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.553+0000 7f6e817fa700 1 -- 172.21.15.1:0/1061068765 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6e7802d3f0 con 0x7f6e84071620 2024-03-18T18:50:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.553+0000 7f6e817fa700 1 --2- 172.21.15.1:0/1061068765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6e6806ac90 0x7f6e6806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.553+0000 7f6e817fa700 1 -- 172.21.15.1:0/1061068765 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6e7808c510 con 0x7f6e84071620 2024-03-18T18:50:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.553+0000 7f6e83fff700 1 --2- 172.21.15.1:0/1061068765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6e6806ac90 0x7f6e6806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.554+0000 7f6e83fff700 1 --2- 172.21.15.1:0/1061068765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6e6806ac90 0x7f6e6806d150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6e840ce590 tx=0x7f6e70008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:33.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.558+0000 7f6e817fa700 1 -- 172.21.15.1:0/1061068765 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6e780573f0 con 0x7f6e84071620 2024-03-18T18:50:33.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.758+0000 7f6e8ad3a700 1 -- 172.21.15.1:0/1061068765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f6e840c4470 con 0x7f6e84071620 2024-03-18T18:50:33.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.758+0000 7f6e817fa700 1 -- 172.21.15.1:0/1061068765 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6e7805aa10 con 0x7f6e84071620 2024-03-18T18:50:33.756 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:33.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.761+0000 7f6e8ad3a700 1 -- 172.21.15.1:0/1061068765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6e6806ac90 msgr2=0x7f6e6806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:33.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.761+0000 7f6e8ad3a700 1 --2- 172.21.15.1:0/1061068765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6e6806ac90 0x7f6e6806d150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6e840ce590 tx=0x7f6e70008040 comp rx=0 tx=0).stop 2024-03-18T18:50:33.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.761+0000 7f6e8ad3a700 1 -- 172.21.15.1:0/1061068765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e84071620 msgr2=0x7f6e8415c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.761+0000 7f6e8ad3a700 1 --2- 172.21.15.1:0/1061068765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e84071620 0x7f6e8415c160 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009130 tx=0x7f6e78004750 comp rx=0 tx=0).stop 2024-03-18T18:50:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.761+0000 7f6e8ad3a700 1 -- 172.21.15.1:0/1061068765 shutdown_connections 2024-03-18T18:50:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.761+0000 7f6e8ad3a700 1 --2- 172.21.15.1:0/1061068765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6e6806ac90 0x7f6e6806d150 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.761+0000 7f6e8ad3a700 1 --2- 172.21.15.1:0/1061068765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e8415a1d0 0x7f6e84150030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.761+0000 7f6e8ad3a700 1 --2- 172.21.15.1:0/1061068765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e84071620 0x7f6e8415c160 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.761+0000 7f6e8ad3a700 1 --2- 172.21.15.1:0/1061068765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e840cb9e0 0x7f6e840b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.761+0000 7f6e8ad3a700 1 -- 172.21.15.1:0/1061068765 >> 172.21.15.1:0/1061068765 conn(0x7f6e840cc9b0 msgr2=0x7f6e840afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.761+0000 7f6e8ad3a700 1 -- 172.21.15.1:0/1061068765 shutdown_connections 2024-03-18T18:50:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.761+0000 7f6e8ad3a700 1 -- 172.21.15.1:0/1061068765 wait complete. 2024-03-18T18:50:33.770 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:33.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.897+0000 7f65e1171700 1 Processor -- start 2024-03-18T18:50:33.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.898+0000 7f65e1171700 1 -- start start 2024-03-18T18:50:33.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.898+0000 7f65e1171700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f65dc0d05a0 0x7f65dc09a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.898+0000 7f65e1171700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65dc06ba70 0x7f65dc15d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.898+0000 7f65e1171700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65dc0cf7e0 con 0x7f65dc099a40 2024-03-18T18:50:33.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.898+0000 7f65e1171700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65dc06bf00 con 0x7f65dc0d05a0 2024-03-18T18:50:33.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.898+0000 7f65e1171700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc0cfac0 con 0x7f65dc06ba70 2024-03-18T18:50:33.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.898+0000 7f65dad9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f65dc099a40 0x7f65dc099e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59542/0 (socket says 172.21.15.1:59542) 2024-03-18T18:50:33.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.898+0000 7f65dad9d700 1 -- 172.21.15.1:0/1641566375 learned_addr learned my addr 172.21.15.1:0/1641566375 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:33.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65d9d9b700 1 -- 172.21.15.1:0/1641566375 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1570823008 0 0) 0x7f65dc0cf7e0 con 0x7f65dc099a40 2024-03-18T18:50:33.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65da59c700 1 --2- 172.21.15.1:0/1641566375 >> v2:172.21.15.165:3300/0 conn(0x7f65dc0d05a0 0x7f65dc09a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65db59e700 1 --2- 172.21.15.1:0/1641566375 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65dc06ba70 0x7f65dc15d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65d9d9b700 1 -- 172.21.15.1:0/1641566375 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65c80035b0 con 0x7f65dc099a40 2024-03-18T18:50:33.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65d9d9b700 1 -- 172.21.15.1:0/1641566375 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1434582839 0 0) 0x7f65c80035b0 con 0x7f65dc099a40 2024-03-18T18:50:33.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65d9d9b700 1 -- 172.21.15.1:0/1641566375 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65dc06ba70 msgr2=0x7f65dc15d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:33.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65d9d9b700 1 --2- 172.21.15.1:0/1641566375 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65dc06ba70 0x7f65dc15d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65d9d9b700 1 -- 172.21.15.1:0/1641566375 >> v2:172.21.15.165:3300/0 conn(0x7f65dc0d05a0 msgr2=0x7f65dc09a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:33.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65d9d9b700 1 --2- 172.21.15.1:0/1641566375 >> v2:172.21.15.165:3300/0 conn(0x7f65dc0d05a0 0x7f65dc09a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65d9d9b700 1 -- 172.21.15.1:0/1641566375 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65dc0cfda0 con 0x7f65dc099a40 2024-03-18T18:50:33.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65d9d9b700 1 -- 172.21.15.1:0/1641566375 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f65cc001670 con 0x7f65dc099a40 2024-03-18T18:50:33.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65d9d9b700 1 -- 172.21.15.1:0/1641566375 >> v1:172.21.15.1:6789/0 conn(0x7f65dc099a40 legacy=0x7f65dc099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:33.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65d9d9b700 1 --2- 172.21.15.1:0/1641566375 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65c8003a00 0x7f65c8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65d9d9b700 1 --2- 172.21.15.1:0/1641566375 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65dc06ba70 0x7f65c8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65d9d9b700 1 --2- 172.21.15.1:0/1641566375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65c8004d90 0x7f65c8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65d9d9b700 1 -- 172.21.15.1:0/1641566375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc06bf00 con 0x7f65c8004d90 2024-03-18T18:50:33.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65d9d9b700 1 -- 172.21.15.1:0/1641566375 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc0cfac0 con 0x7f65c8003a00 2024-03-18T18:50:33.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65d9d9b700 1 -- 172.21.15.1:0/1641566375 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c800ba40 con 0x7f65dc06ba70 2024-03-18T18:50:33.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65dad9d700 1 --2- 172.21.15.1:0/1641566375 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65c8003a00 0x7f65c8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65db59e700 1 --2- 172.21.15.1:0/1641566375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65c8004d90 0x7f65c8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65da59c700 1 --2- 172.21.15.1:0/1641566375 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65dc06ba70 0x7f65c8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.899+0000 7f65db59e700 1 -- 172.21.15.1:0/1641566375 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65dc06ba70 msgr2=0x7f65c8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:33.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.900+0000 7f65db59e700 1 --2- 172.21.15.1:0/1641566375 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65dc06ba70 0x7f65c8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.900+0000 7f65db59e700 1 -- 172.21.15.1:0/1641566375 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65c8003a00 msgr2=0x7f65c8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:33.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.900+0000 7f65db59e700 1 --2- 172.21.15.1:0/1641566375 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65c8003a00 0x7f65c8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.900+0000 7f65db59e700 1 -- 172.21.15.1:0/1641566375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c80035b0 con 0x7f65c8004d90 2024-03-18T18:50:33.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.900+0000 7f65db59e700 1 --2- 172.21.15.1:0/1641566375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65c8004d90 0x7f65c8005210 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f65d4009fc0 tx=0x7f65c8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d76bdcd477365c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:33.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.900+0000 7f65d9d9b700 1 -- 172.21.15.1:0/1641566375 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d400ec30 con 0x7f65c8004d90 2024-03-18T18:50:33.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.900+0000 7f65d9d9b700 1 -- 172.21.15.1:0/1641566375 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d400f040 con 0x7f65c8004d90 2024-03-18T18:50:33.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.900+0000 7f65d9d9b700 1 -- 172.21.15.1:0/1641566375 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d4014480 con 0x7f65c8004d90 2024-03-18T18:50:33.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.900+0000 7f65e1171700 1 -- 172.21.15.1:0/1641566375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65c8004d90 msgr2=0x7f65c8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:33.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.900+0000 7f65e1171700 1 --2- 172.21.15.1:0/1641566375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65c8004d90 0x7f65c8005210 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f65d4009fc0 tx=0x7f65c8005880 comp rx=0 tx=0).stop 2024-03-18T18:50:33.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.900+0000 7f65db59e700 1 -- 172.21.15.1:0/1641566375 reap_dead start 2024-03-18T18:50:33.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.900+0000 7f65e1171700 1 -- 172.21.15.1:0/1641566375 shutdown_connections 2024-03-18T18:50:33.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.900+0000 7f65e1171700 1 -- 172.21.15.1:0/1641566375 >> 172.21.15.1:0/1641566375 conn(0x7f65dc0b2cb0 msgr2=0x7f65dc0b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:33.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.900+0000 7f65e1171700 1 -- 172.21.15.1:0/1641566375 shutdown_connections 2024-03-18T18:50:33.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.900+0000 7f65e1171700 1 -- 172.21.15.1:0/1641566375 wait complete. 2024-03-18T18:50:33.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.902+0000 7f65e1171700 1 Processor -- start 2024-03-18T18:50:33.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.902+0000 7f65e1171700 1 -- start start 2024-03-18T18:50:33.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65e1171700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65dc15b340 0x7f65dc0b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65e1171700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65dc0ade00 0x7f65dc0b6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65e1171700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65dc15b720 0x7f65dc0ca660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65e1171700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc0b7090 con 0x7f65dc15b340 2024-03-18T18:50:33.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65e1171700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc0c8c30 con 0x7f65dc0ade00 2024-03-18T18:50:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65e1171700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc0b4a10 con 0x7f65dc15b720 2024-03-18T18:50:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65dad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65dc15b340 0x7f65dc0b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65dad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65dc15b340 0x7f65dc0b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48050/0 (socket says 172.21.15.1:48050) 2024-03-18T18:50:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65dad9d700 1 -- 172.21.15.1:0/2952251423 learned_addr learned my addr 172.21.15.1:0/2952251423 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65da59c700 1 --2- 172.21.15.1:0/2952251423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65dc0ade00 0x7f65dc0b6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65db59e700 1 --2- 172.21.15.1:0/2952251423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65dc15b720 0x7f65dc0ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65dad9d700 1 -- 172.21.15.1:0/2952251423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65dc15b720 msgr2=0x7f65dc0ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65dad9d700 1 --2- 172.21.15.1:0/2952251423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65dc15b720 0x7f65dc0ca660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65dad9d700 1 -- 172.21.15.1:0/2952251423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65dc0ade00 msgr2=0x7f65dc0b6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65dad9d700 1 --2- 172.21.15.1:0/2952251423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65dc0ade00 0x7f65dc0b6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65dad9d700 1 -- 172.21.15.1:0/2952251423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65dc0b4450 con 0x7f65dc15b340 2024-03-18T18:50:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65dad9d700 1 --2- 172.21.15.1:0/2952251423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65dc15b340 0x7f65dc0b3de0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f65cc007310 tx=0x7f65cc007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65d37fe700 1 -- 172.21.15.1:0/2952251423 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc00f870 con 0x7f65dc15b340 2024-03-18T18:50:33.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65d37fe700 1 -- 172.21.15.1:0/2952251423 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65cc00f9d0 con 0x7f65dc15b340 2024-03-18T18:50:33.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65d37fe700 1 -- 172.21.15.1:0/2952251423 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc00fd40 con 0x7f65dc15b340 2024-03-18T18:50:33.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65e1171700 1 -- 172.21.15.1:0/2952251423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65dc0b4730 con 0x7f65dc15b340 2024-03-18T18:50:33.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.903+0000 7f65e1171700 1 -- 172.21.15.1:0/2952251423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65dc0caba0 con 0x7f65dc15b340 2024-03-18T18:50:33.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.904+0000 7f65e1171700 1 -- 172.21.15.1:0/2952251423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65dc068240 con 0x7f65dc15b340 2024-03-18T18:50:33.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.905+0000 7f65d37fe700 1 -- 172.21.15.1:0/2952251423 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f65cc01d460 con 0x7f65dc15b340 2024-03-18T18:50:33.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.905+0000 7f65d37fe700 1 --2- 172.21.15.1:0/2952251423 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f65bc06ac90 0x7f65bc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:33.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.905+0000 7f65d37fe700 1 -- 172.21.15.1:0/2952251423 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f65cc089650 con 0x7f65dc15b340 2024-03-18T18:50:33.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.905+0000 7f65da59c700 1 --2- 172.21.15.1:0/2952251423 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f65bc06ac90 0x7f65bc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:33.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.906+0000 7f65da59c700 1 --2- 172.21.15.1:0/2952251423 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f65bc06ac90 0x7f65bc06d150 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f65dc0ae220 tx=0x7f65c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:33.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:33.910+0000 7f65d37fe700 1 -- 172.21.15.1:0/2952251423 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f65cc0545d0 con 0x7f65dc15b340 2024-03-18T18:50:34.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.119+0000 7f65e1171700 1 -- 172.21.15.1:0/2952251423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65dc09aae0 con 0x7f65dc15b340 2024-03-18T18:50:34.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.120+0000 7f65d37fe700 1 -- 172.21.15.1:0/2952251423 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18217 (secure 0 0 0) 0x7f65cc00fea0 con 0x7f65dc15b340 2024-03-18T18:50:34.118 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:34.118 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":26,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:50:33.988945+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:34.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.122+0000 7f65e1171700 1 -- 172.21.15.1:0/2952251423 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f65bc06ac90 msgr2=0x7f65bc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:34.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.122+0000 7f65e1171700 1 --2- 172.21.15.1:0/2952251423 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f65bc06ac90 0x7f65bc06d150 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f65dc0ae220 tx=0x7f65c4008040 comp rx=0 tx=0).stop 2024-03-18T18:50:34.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.122+0000 7f65e1171700 1 -- 172.21.15.1:0/2952251423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65dc15b340 msgr2=0x7f65dc0b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:34.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.122+0000 7f65e1171700 1 --2- 172.21.15.1:0/2952251423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65dc15b340 0x7f65dc0b3de0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f65cc007310 tx=0x7f65cc007620 comp rx=0 tx=0).stop 2024-03-18T18:50:34.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.122+0000 7f65e1171700 1 -- 172.21.15.1:0/2952251423 shutdown_connections 2024-03-18T18:50:34.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.122+0000 7f65e1171700 1 --2- 172.21.15.1:0/2952251423 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f65bc06ac90 0x7f65bc06d150 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.122+0000 7f65e1171700 1 --2- 172.21.15.1:0/2952251423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65dc15b720 0x7f65dc0ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.122+0000 7f65e1171700 1 --2- 172.21.15.1:0/2952251423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65dc15b340 0x7f65dc0b3de0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.122+0000 7f65e1171700 1 --2- 172.21.15.1:0/2952251423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65dc0ade00 0x7f65dc0b6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.122+0000 7f65e1171700 1 -- 172.21.15.1:0/2952251423 >> 172.21.15.1:0/2952251423 conn(0x7f65dc0b2cb0 msgr2=0x7f65dc099e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:34.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.122+0000 7f65e1171700 1 -- 172.21.15.1:0/2952251423 shutdown_connections 2024-03-18T18:50:34.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.122+0000 7f65e1171700 1 -- 172.21.15.1:0/2952251423 wait complete. 2024-03-18T18:50:34.132 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T18:49:42.800801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T18:49:43.776684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T18:49:46.854608+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T18:50:34.133 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T18:50:34.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.263+0000 7f5b2cd56700 1 Processor -- start 2024-03-18T18:50:34.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.263+0000 7f5b2cd56700 1 -- start start 2024-03-18T18:50:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.264+0000 7f5b2cd56700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b280ace80 0x7f5b280ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.264+0000 7f5b2cd56700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5b280ad820 0x7f5b2815ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.264+0000 7f5b2cd56700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2815c340 con 0x7f5b2806a750 2024-03-18T18:50:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.264+0000 7f5b2cd56700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b28070090 con 0x7f5b280ad820 2024-03-18T18:50:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.264+0000 7f5b2cd56700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2815b7a0 con 0x7f5b280ace80 2024-03-18T18:50:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.264+0000 7f5b2659c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5b2806a750 0x7f5b2806ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59548/0 (socket says 172.21.15.1:59548) 2024-03-18T18:50:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.264+0000 7f5b2659c700 1 -- 172.21.15.1:0/1701261040 learned_addr learned my addr 172.21.15.1:0/1701261040 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.264+0000 7f5b26d9d700 1 --2- 172.21.15.1:0/1701261040 >> v2:172.21.15.165:3300/0 conn(0x7f5b280ad820 0x7f5b2815ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b25d9b700 1 --2- 172.21.15.1:0/1701261040 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b280ace80 0x7f5b280ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b2559a700 1 -- 172.21.15.1:0/1701261040 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050246723 0 0) 0x7f5b2815c340 con 0x7f5b2806a750 2024-03-18T18:50:34.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b2559a700 1 -- 172.21.15.1:0/1701261040 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b100035b0 con 0x7f5b2806a750 2024-03-18T18:50:34.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b2559a700 1 -- 172.21.15.1:0/1701261040 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2025721191 0 0) 0x7f5b100035b0 con 0x7f5b2806a750 2024-03-18T18:50:34.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b2559a700 1 -- 172.21.15.1:0/1701261040 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b280ace80 msgr2=0x7f5b280ad2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:34.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b2559a700 1 --2- 172.21.15.1:0/1701261040 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b280ace80 0x7f5b280ad2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b2559a700 1 -- 172.21.15.1:0/1701261040 >> v2:172.21.15.165:3300/0 conn(0x7f5b280ad820 msgr2=0x7f5b2815ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:34.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b2559a700 1 --2- 172.21.15.1:0/1701261040 >> v2:172.21.15.165:3300/0 conn(0x7f5b280ad820 0x7f5b2815ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b2559a700 1 -- 172.21.15.1:0/1701261040 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2815ba10 con 0x7f5b2806a750 2024-03-18T18:50:34.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b2559a700 1 -- 172.21.15.1:0/1701261040 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f5b1c0017f0 con 0x7f5b2806a750 2024-03-18T18:50:34.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b2559a700 1 -- 172.21.15.1:0/1701261040 >> v1:172.21.15.1:6789/0 conn(0x7f5b2806a750 legacy=0x7f5b2806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:34.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b2559a700 1 --2- 172.21.15.1:0/1701261040 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b10003a00 0x7f5b10003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b2559a700 1 --2- 172.21.15.1:0/1701261040 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b280ace80 0x7f5b10004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b2559a700 1 --2- 172.21.15.1:0/1701261040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b10004d90 0x7f5b10005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b2559a700 1 -- 172.21.15.1:0/1701261040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b28070090 con 0x7f5b10004d90 2024-03-18T18:50:34.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b2559a700 1 -- 172.21.15.1:0/1701261040 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2815b7a0 con 0x7f5b10003a00 2024-03-18T18:50:34.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b2559a700 1 -- 172.21.15.1:0/1701261040 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1000ba10 con 0x7f5b280ace80 2024-03-18T18:50:34.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b26d9d700 1 --2- 172.21.15.1:0/1701261040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b10004d90 0x7f5b10005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b2659c700 1 --2- 172.21.15.1:0/1701261040 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b10003a00 0x7f5b10003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b25d9b700 1 --2- 172.21.15.1:0/1701261040 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b280ace80 0x7f5b10004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b26d9d700 1 -- 172.21.15.1:0/1701261040 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b280ace80 msgr2=0x7f5b10004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:34.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b26d9d700 1 --2- 172.21.15.1:0/1701261040 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b280ace80 0x7f5b10004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b26d9d700 1 -- 172.21.15.1:0/1701261040 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b10003a00 msgr2=0x7f5b10003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:34.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b26d9d700 1 --2- 172.21.15.1:0/1701261040 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b10003a00 0x7f5b10003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.265+0000 7f5b26d9d700 1 -- 172.21.15.1:0/1701261040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b100035b0 con 0x7f5b10004d90 2024-03-18T18:50:34.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.266+0000 7f5b26d9d700 1 --2- 172.21.15.1:0/1701261040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b10004d90 0x7f5b10005210 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5b2000e170 tx=0x7f5b10005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cedd6822184ce482 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:34.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.266+0000 7f5b2559a700 1 -- 172.21.15.1:0/1701261040 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2000ec90 con 0x7f5b10004d90 2024-03-18T18:50:34.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.266+0000 7f5b2559a700 1 -- 172.21.15.1:0/1701261040 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2000f040 con 0x7f5b10004d90 2024-03-18T18:50:34.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.266+0000 7f5b2559a700 1 -- 172.21.15.1:0/1701261040 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b20014480 con 0x7f5b10004d90 2024-03-18T18:50:34.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.266+0000 7f5b2cd56700 1 -- 172.21.15.1:0/1701261040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b10004d90 msgr2=0x7f5b10005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:34.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.266+0000 7f5b2cd56700 1 --2- 172.21.15.1:0/1701261040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b10004d90 0x7f5b10005210 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5b2000e170 tx=0x7f5b10005880 comp rx=0 tx=0).stop 2024-03-18T18:50:34.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.266+0000 7f5b26d9d700 1 -- 172.21.15.1:0/1701261040 reap_dead start 2024-03-18T18:50:34.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.266+0000 7f5b2cd56700 1 -- 172.21.15.1:0/1701261040 shutdown_connections 2024-03-18T18:50:34.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.266+0000 7f5b2cd56700 1 -- 172.21.15.1:0/1701261040 >> 172.21.15.1:0/1701261040 conn(0x7f5b280cc9b0 msgr2=0x7f5b280ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:34.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.266+0000 7f5b2cd56700 1 -- 172.21.15.1:0/1701261040 shutdown_connections 2024-03-18T18:50:34.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.266+0000 7f5b2cd56700 1 -- 172.21.15.1:0/1701261040 wait complete. 2024-03-18T18:50:34.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.268+0000 7f5b2cd56700 1 Processor -- start 2024-03-18T18:50:34.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.268+0000 7f5b2cd56700 1 -- start start 2024-03-18T18:50:34.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b2cd56700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b10004d90 0x7f5b280ce510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b2cd56700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b280c7ef0 0x7f5b28158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b2cd56700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b2806e7c0 0x7f5b2806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b2cd56700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2806f1b0 con 0x7f5b10004d90 2024-03-18T18:50:34.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b2cd56700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b280c9750 con 0x7f5b280c7ef0 2024-03-18T18:50:34.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b2cd56700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b280c86b0 con 0x7f5b2806e7c0 2024-03-18T18:50:34.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b2659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b10004d90 0x7f5b280ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b2659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b10004d90 0x7f5b280ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48070/0 (socket says 172.21.15.1:48070) 2024-03-18T18:50:34.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b2659c700 1 -- 172.21.15.1:0/2297982620 learned_addr learned my addr 172.21.15.1:0/2297982620 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:34.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b26d9d700 1 --2- 172.21.15.1:0/2297982620 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b2806e7c0 0x7f5b2806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b25d9b700 1 --2- 172.21.15.1:0/2297982620 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b280c7ef0 0x7f5b28158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b2659c700 1 -- 172.21.15.1:0/2297982620 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b2806e7c0 msgr2=0x7f5b2806ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:34.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b2659c700 1 --2- 172.21.15.1:0/2297982620 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b2806e7c0 0x7f5b2806ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b2659c700 1 -- 172.21.15.1:0/2297982620 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b280c7ef0 msgr2=0x7f5b28158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:34.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b2659c700 1 --2- 172.21.15.1:0/2297982620 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b280c7ef0 0x7f5b28158290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b2659c700 1 -- 172.21.15.1:0/2297982620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b280c8990 con 0x7f5b10004d90 2024-03-18T18:50:34.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b26d9d700 1 --2- 172.21.15.1:0/2297982620 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b2806e7c0 0x7f5b2806ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:34.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b2659c700 1 --2- 172.21.15.1:0/2297982620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b10004d90 0x7f5b280ce510 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c001020 tx=0x7f5b1c007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:34.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b16ffd700 1 -- 172.21.15.1:0/2297982620 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c00f8a0 con 0x7f5b10004d90 2024-03-18T18:50:34.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b16ffd700 1 -- 172.21.15.1:0/2297982620 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1c00fa00 con 0x7f5b10004d90 2024-03-18T18:50:34.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b16ffd700 1 -- 172.21.15.1:0/2297982620 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c00fdb0 con 0x7f5b10004d90 2024-03-18T18:50:34.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b2cd56700 1 -- 172.21.15.1:0/2297982620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b280c8c70 con 0x7f5b10004d90 2024-03-18T18:50:34.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.269+0000 7f5b2cd56700 1 -- 172.21.15.1:0/2297982620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b280c9230 con 0x7f5b10004d90 2024-03-18T18:50:34.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.270+0000 7f5b2cd56700 1 -- 172.21.15.1:0/2297982620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b280676a0 con 0x7f5b10004d90 2024-03-18T18:50:34.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.271+0000 7f5b16ffd700 1 -- 172.21.15.1:0/2297982620 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5b1c01c400 con 0x7f5b10004d90 2024-03-18T18:50:34.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.271+0000 7f5b16ffd700 1 --2- 172.21.15.1:0/2297982620 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b0806ac90 0x7f5b0806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.271+0000 7f5b16ffd700 1 -- 172.21.15.1:0/2297982620 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5b1c089b60 con 0x7f5b10004d90 2024-03-18T18:50:34.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.271+0000 7f5b25d9b700 1 --2- 172.21.15.1:0/2297982620 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b0806ac90 0x7f5b0806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.272+0000 7f5b25d9b700 1 --2- 172.21.15.1:0/2297982620 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b0806ac90 0x7f5b0806d150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5b2815db20 tx=0x7f5b18008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:34.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.276+0000 7f5b16ffd700 1 -- 172.21.15.1:0/2297982620 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5b1c054990 con 0x7f5b10004d90 2024-03-18T18:50:34.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.476+0000 7f5b2cd56700 1 -- 172.21.15.1:0/2297982620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5b280ca3f0 con 0x7f5b10004d90 2024-03-18T18:50:34.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.477+0000 7f5b16ffd700 1 -- 172.21.15.1:0/2297982620 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5b1c017070 con 0x7f5b10004d90 2024-03-18T18:50:34.474 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T18:50:34.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.479+0000 7f5b2cd56700 1 -- 172.21.15.1:0/2297982620 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b0806ac90 msgr2=0x7f5b0806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:34.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.479+0000 7f5b2cd56700 1 --2- 172.21.15.1:0/2297982620 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b0806ac90 0x7f5b0806d150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5b2815db20 tx=0x7f5b18008040 comp rx=0 tx=0).stop 2024-03-18T18:50:34.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.479+0000 7f5b2cd56700 1 -- 172.21.15.1:0/2297982620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b10004d90 msgr2=0x7f5b280ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:34.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.479+0000 7f5b2cd56700 1 --2- 172.21.15.1:0/2297982620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b10004d90 0x7f5b280ce510 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c001020 tx=0x7f5b1c007680 comp rx=0 tx=0).stop 2024-03-18T18:50:34.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.479+0000 7f5b2cd56700 1 -- 172.21.15.1:0/2297982620 shutdown_connections 2024-03-18T18:50:34.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.479+0000 7f5b2cd56700 1 --2- 172.21.15.1:0/2297982620 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b0806ac90 0x7f5b0806d150 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.479+0000 7f5b2cd56700 1 --2- 172.21.15.1:0/2297982620 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b2806e7c0 0x7f5b2806ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.479+0000 7f5b2cd56700 1 --2- 172.21.15.1:0/2297982620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b10004d90 0x7f5b280ce510 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.479+0000 7f5b2cd56700 1 --2- 172.21.15.1:0/2297982620 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b280c7ef0 0x7f5b28158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.479+0000 7f5b2cd56700 1 -- 172.21.15.1:0/2297982620 >> 172.21.15.1:0/2297982620 conn(0x7f5b280cc9b0 msgr2=0x7f5b280b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:34.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.479+0000 7f5b2cd56700 1 -- 172.21.15.1:0/2297982620 shutdown_connections 2024-03-18T18:50:34.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.479+0000 7f5b2cd56700 1 -- 172.21.15.1:0/2297982620 wait complete. 2024-03-18T18:50:34.488 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T18:50:34.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.620+0000 7f6242553700 1 Processor -- start 2024-03-18T18:50:34.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.620+0000 7f6242553700 1 -- start start 2024-03-18T18:50:34.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.620+0000 7f6242553700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f623c073a60 0x7f623c0ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.620+0000 7f6242553700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f623c06d300 0x7f623c0acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.620+0000 7f6242553700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f623c06e160 con 0x7f623c15d340 2024-03-18T18:50:34.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.620+0000 7f6242553700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f623c161150 con 0x7f623c06d300 2024-03-18T18:50:34.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.620+0000 7f6242553700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f623c06d8c0 con 0x7f623c073a60 2024-03-18T18:50:34.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.621+0000 7f6240af0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f623c06d300 0x7f623c0acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.621+0000 7f623bfff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f623c15d340 0x7f623c1615a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37324/0 (socket says 172.21.15.1:37324) 2024-03-18T18:50:34.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.621+0000 7f623bfff700 1 -- 172.21.15.1:0/522516923 learned_addr learned my addr 172.21.15.1:0/522516923 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:34.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.621+0000 7f623b7fe700 1 --2- 172.21.15.1:0/522516923 >> v2:172.21.15.165:3300/0 conn(0x7f623c073a60 0x7f623c0ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.621+0000 7f6240af0700 1 -- 172.21.15.1:0/522516923 >> v2:172.21.15.165:3300/0 conn(0x7f623c073a60 msgr2=0x7f623c0ac8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:34.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.621+0000 7f6240af0700 1 --2- 172.21.15.1:0/522516923 >> v2:172.21.15.165:3300/0 conn(0x7f623c073a60 0x7f623c0ac8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.621+0000 7f6240af0700 1 -- 172.21.15.1:0/522516923 >> v1:172.21.15.1:6789/0 conn(0x7f623c15d340 legacy=0x7f623c1615a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:34.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.621+0000 7f6240af0700 1 -- 172.21.15.1:0/522516923 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623c06dba0 con 0x7f623c06d300 2024-03-18T18:50:34.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.621+0000 7f623affd700 1 -- 172.21.15.1:0/522516923 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396871187 0 0) 0x7f623c06e160 con 0x7f623c15d340 2024-03-18T18:50:34.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.621+0000 7f6240af0700 1 --2- 172.21.15.1:0/522516923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f623c06d300 0x7f623c0acdf0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f623c160ef0 tx=0x7f623400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da348b7bf6d767 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:34.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.621+0000 7f623affd700 1 -- 172.21.15.1:0/522516923 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623400e070 con 0x7f623c06d300 2024-03-18T18:50:34.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.621+0000 7f623affd700 1 -- 172.21.15.1:0/522516923 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f623400bbd0 con 0x7f623c06d300 2024-03-18T18:50:34.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.621+0000 7f623affd700 1 -- 172.21.15.1:0/522516923 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6234012430 con 0x7f623c06d300 2024-03-18T18:50:34.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.622+0000 7f6242553700 1 -- 172.21.15.1:0/522516923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f623c06d300 msgr2=0x7f623c0acdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:34.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.622+0000 7f6242553700 1 --2- 172.21.15.1:0/522516923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f623c06d300 0x7f623c0acdf0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f623c160ef0 tx=0x7f623400b330 comp rx=0 tx=0).stop 2024-03-18T18:50:34.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.622+0000 7f6242553700 1 -- 172.21.15.1:0/522516923 shutdown_connections 2024-03-18T18:50:34.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.622+0000 7f6242553700 1 --2- 172.21.15.1:0/522516923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f623c06d300 0x7f623c0acdf0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.622+0000 7f6242553700 1 --2- 172.21.15.1:0/522516923 >> v2:172.21.15.165:3300/0 conn(0x7f623c073a60 0x7f623c0ac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.622+0000 7f6242553700 1 -- 172.21.15.1:0/522516923 >> 172.21.15.1:0/522516923 conn(0x7f623c066b70 msgr2=0x7f623c06a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:34.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.622+0000 7f6242553700 1 -- 172.21.15.1:0/522516923 shutdown_connections 2024-03-18T18:50:34.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.622+0000 7f6242553700 1 -- 172.21.15.1:0/522516923 wait complete. 2024-03-18T18:50:34.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.623+0000 7f6242553700 1 Processor -- start 2024-03-18T18:50:34.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.623+0000 7f6242553700 1 -- start start 2024-03-18T18:50:34.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.624+0000 7f6242553700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f623c06d300 0x7f623c140290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.624+0000 7f6242553700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f623c073a60 0x7f623c158e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.624+0000 7f6242553700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f623c15d340 0x7f623c15a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.624+0000 7f6242553700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c1407d0 con 0x7f623c15d340 2024-03-18T18:50:34.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.624+0000 7f6242553700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c15a4a0 con 0x7f623c06d300 2024-03-18T18:50:34.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.624+0000 7f6242553700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f623c159890 con 0x7f623c073a60 2024-03-18T18:50:34.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.624+0000 7f6240af0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f623c15d340 0x7f623c15a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.624+0000 7f6240af0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f623c15d340 0x7f623c15a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49602/0 (socket says 172.21.15.1:49602) 2024-03-18T18:50:34.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.624+0000 7f6240af0700 1 -- 172.21.15.1:0/810240991 learned_addr learned my addr 172.21.15.1:0/810240991 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:34.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.624+0000 7f623bfff700 1 --2- 172.21.15.1:0/810240991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f623c06d300 0x7f623c140290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.624+0000 7f623b7fe700 1 --2- 172.21.15.1:0/810240991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f623c073a60 0x7f623c158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.624+0000 7f6240af0700 1 -- 172.21.15.1:0/810240991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f623c073a60 msgr2=0x7f623c158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:34.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.624+0000 7f6240af0700 1 --2- 172.21.15.1:0/810240991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f623c073a60 0x7f623c158e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.624+0000 7f6240af0700 1 -- 172.21.15.1:0/810240991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f623c06d300 msgr2=0x7f623c140290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:34.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.624+0000 7f6240af0700 1 --2- 172.21.15.1:0/810240991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f623c06d300 0x7f623c140290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.624+0000 7f6240af0700 1 -- 172.21.15.1:0/810240991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6234009a70 con 0x7f623c15d340 2024-03-18T18:50:34.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.625+0000 7f6240af0700 1 --2- 172.21.15.1:0/810240991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f623c15d340 0x7f623c15a980 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f623400b730 tx=0x7f6234004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:34.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.625+0000 7f6238ff9700 1 -- 172.21.15.1:0/810240991 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6234012430 con 0x7f623c15d340 2024-03-18T18:50:34.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.625+0000 7f6238ff9700 1 -- 172.21.15.1:0/810240991 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6234020070 con 0x7f623c15d340 2024-03-18T18:50:34.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.625+0000 7f6242553700 1 -- 172.21.15.1:0/810240991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f623c159b70 con 0x7f623c15d340 2024-03-18T18:50:34.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.625+0000 7f6238ff9700 1 -- 172.21.15.1:0/810240991 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6234004280 con 0x7f623c15d340 2024-03-18T18:50:34.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.625+0000 7f6242553700 1 -- 172.21.15.1:0/810240991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f623c15a130 con 0x7f623c15d340 2024-03-18T18:50:34.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.626+0000 7f6242553700 1 -- 172.21.15.1:0/810240991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f623c0676a0 con 0x7f623c15d340 2024-03-18T18:50:34.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.626+0000 7f6238ff9700 1 -- 172.21.15.1:0/810240991 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f62340043e0 con 0x7f623c15d340 2024-03-18T18:50:34.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.627+0000 7f6238ff9700 1 --2- 172.21.15.1:0/810240991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622806ac90 0x7f622806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.627+0000 7f6238ff9700 1 -- 172.21.15.1:0/810240991 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f6234022070 con 0x7f623c15d340 2024-03-18T18:50:34.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.627+0000 7f623bfff700 1 --2- 172.21.15.1:0/810240991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622806ac90 0x7f622806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.627+0000 7f623bfff700 1 --2- 172.21.15.1:0/810240991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622806ac90 0x7f622806d150 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f622c000f60 tx=0x7f622c007e70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:34.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.632+0000 7f6238ff9700 1 -- 172.21.15.1:0/810240991 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f62340561a0 con 0x7f623c15d340 2024-03-18T18:50:34.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.831+0000 7f6242553700 1 -- 172.21.15.1:0/810240991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f623c15a5e0 con 0x7f623c15d340 2024-03-18T18:50:34.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.832+0000 7f6238ff9700 1 -- 172.21.15.1:0/810240991 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f62340597c0 con 0x7f623c15d340 2024-03-18T18:50:34.829 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:34.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.834+0000 7f6242553700 1 -- 172.21.15.1:0/810240991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622806ac90 msgr2=0x7f622806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:34.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.834+0000 7f6242553700 1 --2- 172.21.15.1:0/810240991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622806ac90 0x7f622806d150 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f622c000f60 tx=0x7f622c007e70 comp rx=0 tx=0).stop 2024-03-18T18:50:34.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.834+0000 7f6242553700 1 -- 172.21.15.1:0/810240991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f623c15d340 msgr2=0x7f623c15a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:34.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.834+0000 7f6242553700 1 --2- 172.21.15.1:0/810240991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f623c15d340 0x7f623c15a980 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f623400b730 tx=0x7f6234004ad0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.834+0000 7f6242553700 1 -- 172.21.15.1:0/810240991 shutdown_connections 2024-03-18T18:50:34.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.834+0000 7f6242553700 1 --2- 172.21.15.1:0/810240991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622806ac90 0x7f622806d150 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.834+0000 7f6242553700 1 --2- 172.21.15.1:0/810240991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f623c15d340 0x7f623c15a980 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.834+0000 7f6242553700 1 --2- 172.21.15.1:0/810240991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f623c06d300 0x7f623c140290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.834+0000 7f6242553700 1 --2- 172.21.15.1:0/810240991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f623c073a60 0x7f623c158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.834+0000 7f6242553700 1 -- 172.21.15.1:0/810240991 >> 172.21.15.1:0/810240991 conn(0x7f623c066b70 msgr2=0x7f623c0cbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:34.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.834+0000 7f6242553700 1 -- 172.21.15.1:0/810240991 shutdown_connections 2024-03-18T18:50:34.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.834+0000 7f6242553700 1 -- 172.21.15.1:0/810240991 wait complete. 2024-03-18T18:50:34.843 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T18:50:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.974+0000 7fd414a4e700 1 Processor -- start 2024-03-18T18:50:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.974+0000 7fd414a4e700 1 -- start start 2024-03-18T18:50:34.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd414a4e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd4100727b0 0x7fd410161610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd414a4e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd410161b50 0x7fd41006a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd414a4e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd410160ea0 con 0x7fd41015ad70 2024-03-18T18:50:34.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd414a4e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4100725b0 con 0x7fd4100727b0 2024-03-18T18:50:34.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd414a4e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd41015e090 con 0x7fd410161b50 2024-03-18T18:50:34.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd40e59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd41015ad70 0x7fd41015d8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37336/0 (socket says 172.21.15.1:37336) 2024-03-18T18:50:34.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd40e59c700 1 -- 172.21.15.1:0/669251683 learned_addr learned my addr 172.21.15.1:0/669251683 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:34.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd40dd9b700 1 --2- 172.21.15.1:0/669251683 >> v2:172.21.15.165:3300/0 conn(0x7fd4100727b0 0x7fd410161610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd40ed9d700 1 --2- 172.21.15.1:0/669251683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd410161b50 0x7fd41006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd40d59a700 1 -- 172.21.15.1:0/669251683 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4284358529 0 0) 0x7fd410160ea0 con 0x7fd41015ad70 2024-03-18T18:50:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd40d59a700 1 -- 172.21.15.1:0/669251683 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f80035b0 con 0x7fd41015ad70 2024-03-18T18:50:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd40d59a700 1 -- 172.21.15.1:0/669251683 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4285956676 0 0) 0x7fd3f80035b0 con 0x7fd41015ad70 2024-03-18T18:50:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd40d59a700 1 -- 172.21.15.1:0/669251683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd410161b50 msgr2=0x7fd41006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd40d59a700 1 --2- 172.21.15.1:0/669251683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd410161b50 0x7fd41006a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd40d59a700 1 -- 172.21.15.1:0/669251683 >> v2:172.21.15.165:3300/0 conn(0x7fd4100727b0 msgr2=0x7fd410161610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd40d59a700 1 --2- 172.21.15.1:0/669251683 >> v2:172.21.15.165:3300/0 conn(0x7fd4100727b0 0x7fd410161610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd40d59a700 1 -- 172.21.15.1:0/669251683 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd41015e370 con 0x7fd41015ad70 2024-03-18T18:50:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd40d59a700 1 -- 172.21.15.1:0/669251683 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fd400001670 con 0x7fd41015ad70 2024-03-18T18:50:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd40d59a700 1 -- 172.21.15.1:0/669251683 >> v1:172.21.15.1:6789/0 conn(0x7fd41015ad70 legacy=0x7fd41015d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd40d59a700 1 --2- 172.21.15.1:0/669251683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd410161b50 0x7fd3f8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.975+0000 7fd40d59a700 1 --2- 172.21.15.1:0/669251683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3f8003f80 0x7fd3f8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.976+0000 7fd40d59a700 1 --2- 172.21.15.1:0/669251683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3f8004d90 0x7fd3f8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.976+0000 7fd40d59a700 1 -- 172.21.15.1:0/669251683 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41015e090 con 0x7fd3f8003f80 2024-03-18T18:50:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.976+0000 7fd40d59a700 1 -- 172.21.15.1:0/669251683 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f800ba10 con 0x7fd3f8004d90 2024-03-18T18:50:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.976+0000 7fd40d59a700 1 -- 172.21.15.1:0/669251683 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f800bb50 con 0x7fd410161b50 2024-03-18T18:50:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.976+0000 7fd40dd9b700 1 --2- 172.21.15.1:0/669251683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3f8003f80 0x7fd3f8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.976+0000 7fd40e59c700 1 --2- 172.21.15.1:0/669251683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd410161b50 0x7fd3f8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.976+0000 7fd40ed9d700 1 --2- 172.21.15.1:0/669251683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3f8004d90 0x7fd3f8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.976+0000 7fd40dd9b700 1 -- 172.21.15.1:0/669251683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd410161b50 msgr2=0x7fd3f8003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.976+0000 7fd40dd9b700 1 --2- 172.21.15.1:0/669251683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd410161b50 0x7fd3f8003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.976+0000 7fd40dd9b700 1 -- 172.21.15.1:0/669251683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3f8004d90 msgr2=0x7fd3f8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.976+0000 7fd40dd9b700 1 --2- 172.21.15.1:0/669251683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3f8004d90 0x7fd3f8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.976+0000 7fd40dd9b700 1 -- 172.21.15.1:0/669251683 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f80035b0 con 0x7fd3f8003f80 2024-03-18T18:50:34.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.976+0000 7fd40ed9d700 1 --2- 172.21.15.1:0/669251683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3f8004d90 0x7fd3f8005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:34.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.976+0000 7fd40dd9b700 1 --2- 172.21.15.1:0/669251683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3f8003f80 0x7fd3f8004850 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd404009b60 tx=0x7fd3f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33ea43b3e285eb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:34.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.976+0000 7fd40d59a700 1 -- 172.21.15.1:0/669251683 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40400e660 con 0x7fd3f8003f80 2024-03-18T18:50:34.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.976+0000 7fd40d59a700 1 -- 172.21.15.1:0/669251683 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40400e7c0 con 0x7fd3f8003f80 2024-03-18T18:50:34.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.976+0000 7fd40d59a700 1 -- 172.21.15.1:0/669251683 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40400eb30 con 0x7fd3f8003f80 2024-03-18T18:50:34.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.977+0000 7fd414a4e700 1 -- 172.21.15.1:0/669251683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3f8003f80 msgr2=0x7fd3f8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:34.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.977+0000 7fd414a4e700 1 --2- 172.21.15.1:0/669251683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3f8003f80 0x7fd3f8004850 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd404009b60 tx=0x7fd3f8005880 comp rx=0 tx=0).stop 2024-03-18T18:50:34.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.977+0000 7fd40ed9d700 1 -- 172.21.15.1:0/669251683 reap_dead start 2024-03-18T18:50:34.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.977+0000 7fd414a4e700 1 -- 172.21.15.1:0/669251683 shutdown_connections 2024-03-18T18:50:34.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.977+0000 7fd414a4e700 1 -- 172.21.15.1:0/669251683 >> 172.21.15.1:0/669251683 conn(0x7fd4100b1cb0 msgr2=0x7fd4100b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:34.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.977+0000 7fd414a4e700 1 -- 172.21.15.1:0/669251683 shutdown_connections 2024-03-18T18:50:34.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.977+0000 7fd414a4e700 1 -- 172.21.15.1:0/669251683 wait complete. 2024-03-18T18:50:34.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.978+0000 7fd414a4e700 1 Processor -- start 2024-03-18T18:50:34.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.978+0000 7fd414a4e700 1 -- start start 2024-03-18T18:50:34.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.979+0000 7fd414a4e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3f8003f80 0x7fd410142f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.979+0000 7fd414a4e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4100d02c0 0x7fd410142720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.979+0000 7fd414a4e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4100ae440 0x7fd4100ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.979+0000 7fd414a4e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4100aee00 con 0x7fd4100d02c0 2024-03-18T18:50:34.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.979+0000 7fd414a4e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4101434f0 con 0x7fd3f8003f80 2024-03-18T18:50:34.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.979+0000 7fd414a4e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4101437d0 con 0x7fd4100ae440 2024-03-18T18:50:34.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.979+0000 7fd40dd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4100d02c0 0x7fd410142720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.979+0000 7fd40dd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4100d02c0 0x7fd410142720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49624/0 (socket says 172.21.15.1:49624) 2024-03-18T18:50:34.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.979+0000 7fd40dd9b700 1 -- 172.21.15.1:0/3736436987 learned_addr learned my addr 172.21.15.1:0/3736436987 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:34.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.979+0000 7fd40e59c700 1 --2- 172.21.15.1:0/3736436987 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3f8003f80 0x7fd410142f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.979+0000 7fd40ed9d700 1 --2- 172.21.15.1:0/3736436987 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4100ae440 0x7fd4100ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.979+0000 7fd40dd9b700 1 -- 172.21.15.1:0/3736436987 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4100ae440 msgr2=0x7fd4100ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:34.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.979+0000 7fd40dd9b700 1 --2- 172.21.15.1:0/3736436987 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4100ae440 0x7fd4100ae8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.979+0000 7fd40dd9b700 1 -- 172.21.15.1:0/3736436987 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3f8003f80 msgr2=0x7fd410142f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:34.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.979+0000 7fd40dd9b700 1 --2- 172.21.15.1:0/3736436987 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3f8003f80 0x7fd410142f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:34.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.979+0000 7fd40dd9b700 1 -- 172.21.15.1:0/3736436987 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd410143ab0 con 0x7fd4100d02c0 2024-03-18T18:50:34.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.979+0000 7fd40ed9d700 1 --2- 172.21.15.1:0/3736436987 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4100ae440 0x7fd4100ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:34.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.979+0000 7fd40dd9b700 1 --2- 172.21.15.1:0/3736436987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4100d02c0 0x7fd410142720 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd3f80046f0 tx=0x7fd4040047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:34.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.980+0000 7fd3feffd700 1 -- 172.21.15.1:0/3736436987 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40400e660 con 0x7fd4100d02c0 2024-03-18T18:50:34.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.980+0000 7fd3feffd700 1 -- 172.21.15.1:0/3736436987 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4040038c0 con 0x7fd4100d02c0 2024-03-18T18:50:34.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.980+0000 7fd414a4e700 1 -- 172.21.15.1:0/3736436987 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd410143d90 con 0x7fd4100d02c0 2024-03-18T18:50:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.980+0000 7fd3feffd700 1 -- 172.21.15.1:0/3736436987 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd404003c70 con 0x7fd4100d02c0 2024-03-18T18:50:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.980+0000 7fd414a4e700 1 -- 172.21.15.1:0/3736436987 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd410142c60 con 0x7fd4100d02c0 2024-03-18T18:50:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.981+0000 7fd414a4e700 1 -- 172.21.15.1:0/3736436987 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4100b3c20 con 0x7fd4100d02c0 2024-03-18T18:50:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.981+0000 7fd3feffd700 1 -- 172.21.15.1:0/3736436987 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd404013470 con 0x7fd4100d02c0 2024-03-18T18:50:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.982+0000 7fd3feffd700 1 --2- 172.21.15.1:0/3736436987 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd3f006acc0 0x7fd3f006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.982+0000 7fd40e59c700 1 --2- 172.21.15.1:0/3736436987 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd3f006acc0 0x7fd3f006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.982+0000 7fd3feffd700 1 -- 172.21.15.1:0/3736436987 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd40408ae50 con 0x7fd4100d02c0 2024-03-18T18:50:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.982+0000 7fd40e59c700 1 --2- 172.21.15.1:0/3736436987 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd3f006acc0 0x7fd3f006d180 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd400000c00 tx=0x7fd400007980 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:34.986+0000 7fd3feffd700 1 -- 172.21.15.1:0/3736436987 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd404055e20 con 0x7fd4100d02c0 2024-03-18T18:50:35.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.203+0000 7fd414a4e700 1 -- 172.21.15.1:0/3736436987 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd4100684d0 con 0x7fd4100d02c0 2024-03-18T18:50:35.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.203+0000 7fd3feffd700 1 -- 172.21.15.1:0/3736436987 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd404059440 con 0x7fd4100d02c0 2024-03-18T18:50:35.201 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T18:50:35.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.206+0000 7fd414a4e700 1 -- 172.21.15.1:0/3736436987 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd3f006acc0 msgr2=0x7fd3f006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:35.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.206+0000 7fd414a4e700 1 --2- 172.21.15.1:0/3736436987 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd3f006acc0 0x7fd3f006d180 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd400000c00 tx=0x7fd400007980 comp rx=0 tx=0).stop 2024-03-18T18:50:35.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.206+0000 7fd414a4e700 1 -- 172.21.15.1:0/3736436987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4100d02c0 msgr2=0x7fd410142720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:35.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.206+0000 7fd414a4e700 1 --2- 172.21.15.1:0/3736436987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4100d02c0 0x7fd410142720 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd3f80046f0 tx=0x7fd4040047c0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.206+0000 7fd414a4e700 1 -- 172.21.15.1:0/3736436987 shutdown_connections 2024-03-18T18:50:35.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.206+0000 7fd414a4e700 1 --2- 172.21.15.1:0/3736436987 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd3f006acc0 0x7fd3f006d180 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.206+0000 7fd414a4e700 1 --2- 172.21.15.1:0/3736436987 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4100ae440 0x7fd4100ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.206+0000 7fd414a4e700 1 --2- 172.21.15.1:0/3736436987 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3f8003f80 0x7fd410142f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.206+0000 7fd414a4e700 1 --2- 172.21.15.1:0/3736436987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4100d02c0 0x7fd410142720 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.206+0000 7fd414a4e700 1 -- 172.21.15.1:0/3736436987 >> 172.21.15.1:0/3736436987 conn(0x7fd4100b1cb0 msgr2=0x7fd41015d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:35.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.206+0000 7fd414a4e700 1 -- 172.21.15.1:0/3736436987 shutdown_connections 2024-03-18T18:50:35.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.206+0000 7fd414a4e700 1 -- 172.21.15.1:0/3736436987 wait complete. 2024-03-18T18:50:35.215 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T18:50:35.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.345+0000 7f44667e8700 1 Processor -- start 2024-03-18T18:50:35.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.345+0000 7f44667e8700 1 -- start start 2024-03-18T18:50:35.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.346+0000 7f44667e8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44600727b0 0x7f4460161610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:35.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.346+0000 7f44667e8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4460161b50 0x7f446006a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:35.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.346+0000 7f44667e8700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4460160ea0 con 0x7f446015ad70 2024-03-18T18:50:35.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.346+0000 7f44667e8700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44600725b0 con 0x7f4460161b50 2024-03-18T18:50:35.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.346+0000 7f44667e8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f446015e090 con 0x7f44600727b0 2024-03-18T18:50:35.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.346+0000 7f4464d85700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4460161b50 0x7f446006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:35.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.346+0000 7f4464d85700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4460161b50 0x7f446006a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49630/0 (socket says 172.21.15.1:49630) 2024-03-18T18:50:35.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.346+0000 7f4464d85700 1 -- 172.21.15.1:0/895539120 learned_addr learned my addr 172.21.15.1:0/895539120 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:35.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.346+0000 7f445f7fe700 1 --2- 172.21.15.1:0/895539120 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44600727b0 0x7f4460161610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:35.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.347+0000 7f4464d85700 1 -- 172.21.15.1:0/895539120 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44600727b0 msgr2=0x7f4460161610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:35.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.347+0000 7f4464d85700 1 --2- 172.21.15.1:0/895539120 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44600727b0 0x7f4460161610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.347+0000 7f4464d85700 1 -- 172.21.15.1:0/895539120 >> v1:172.21.15.165:6789/0 conn(0x7f446015ad70 legacy=0x7f446015d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:35.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.347+0000 7f4464d85700 1 -- 172.21.15.1:0/895539120 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446015e370 con 0x7f4460161b50 2024-03-18T18:50:35.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.347+0000 7f4464d85700 1 --2- 172.21.15.1:0/895539120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4460161b50 0x7f446006a750 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4458009e50 tx=0x7f445800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c96810107858ebc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:35.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.347+0000 7f445effd700 1 -- 172.21.15.1:0/895539120 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445800e070 con 0x7f4460161b50 2024-03-18T18:50:35.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.347+0000 7f445effd700 1 -- 172.21.15.1:0/895539120 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445800b9c0 con 0x7f4460161b50 2024-03-18T18:50:35.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.347+0000 7f445effd700 1 -- 172.21.15.1:0/895539120 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445800bd70 con 0x7f4460161b50 2024-03-18T18:50:35.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.347+0000 7f44667e8700 1 -- 172.21.15.1:0/895539120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4460161b50 msgr2=0x7f446006a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:35.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.347+0000 7f44667e8700 1 --2- 172.21.15.1:0/895539120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4460161b50 0x7f446006a750 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4458009e50 tx=0x7f445800b330 comp rx=0 tx=0).stop 2024-03-18T18:50:35.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.347+0000 7f44667e8700 1 -- 172.21.15.1:0/895539120 shutdown_connections 2024-03-18T18:50:35.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.347+0000 7f44667e8700 1 --2- 172.21.15.1:0/895539120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4460161b50 0x7f446006a750 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.347+0000 7f44667e8700 1 --2- 172.21.15.1:0/895539120 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44600727b0 0x7f4460161610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.347+0000 7f44667e8700 1 -- 172.21.15.1:0/895539120 >> 172.21.15.1:0/895539120 conn(0x7f44600b1cb0 msgr2=0x7f44600b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:35.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.348+0000 7f44667e8700 1 -- 172.21.15.1:0/895539120 shutdown_connections 2024-03-18T18:50:35.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.348+0000 7f44667e8700 1 -- 172.21.15.1:0/895539120 wait complete. 2024-03-18T18:50:35.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.349+0000 7f44667e8700 1 Processor -- start 2024-03-18T18:50:35.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.349+0000 7f44667e8700 1 -- start start 2024-03-18T18:50:35.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f44667e8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44600727b0 0x7f4460151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:35.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f44667e8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f446015ad70 0x7f446015c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:35.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f44667e8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4460161b50 0x7f446015c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:35.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f44667e8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44601e8120 con 0x7f44600727b0 2024-03-18T18:50:35.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f44667e8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44600cdb40 con 0x7f4460161b50 2024-03-18T18:50:35.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f44667e8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44600cde20 con 0x7f446015ad70 2024-03-18T18:50:35.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f445ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44600727b0 0x7f4460151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:35.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f445ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44600727b0 0x7f4460151190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49632/0 (socket says 172.21.15.1:49632) 2024-03-18T18:50:35.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f445ffff700 1 -- 172.21.15.1:0/1706489044 learned_addr learned my addr 172.21.15.1:0/1706489044 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:35.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f445f7fe700 1 --2- 172.21.15.1:0/1706489044 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f446015ad70 0x7f446015c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:35.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f4464d85700 1 --2- 172.21.15.1:0/1706489044 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4460161b50 0x7f446015c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:35.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f445ffff700 1 -- 172.21.15.1:0/1706489044 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f446015ad70 msgr2=0x7f446015c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:35.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f445ffff700 1 --2- 172.21.15.1:0/1706489044 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f446015ad70 0x7f446015c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f445ffff700 1 -- 172.21.15.1:0/1706489044 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4460161b50 msgr2=0x7f446015c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:35.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f445ffff700 1 --2- 172.21.15.1:0/1706489044 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4460161b50 0x7f446015c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f445ffff700 1 -- 172.21.15.1:0/1706489044 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4458009b30 con 0x7f44600727b0 2024-03-18T18:50:35.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f445f7fe700 1 --2- 172.21.15.1:0/1706489044 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f446015ad70 0x7f446015c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:35.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f445ffff700 1 --2- 172.21.15.1:0/1706489044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44600727b0 0x7f4460151190 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f445400c4e0 tx=0x7f445400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:35.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f445cff9700 1 -- 172.21.15.1:0/1706489044 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44540052d0 con 0x7f44600727b0 2024-03-18T18:50:35.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f445cff9700 1 -- 172.21.15.1:0/1706489044 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4454005430 con 0x7f44600727b0 2024-03-18T18:50:35.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f445cff9700 1 -- 172.21.15.1:0/1706489044 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44540038c0 con 0x7f44600727b0 2024-03-18T18:50:35.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f44667e8700 1 -- 172.21.15.1:0/1706489044 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44600ce100 con 0x7f44600727b0 2024-03-18T18:50:35.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.350+0000 7f44667e8700 1 -- 172.21.15.1:0/1706489044 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44601516d0 con 0x7f44600727b0 2024-03-18T18:50:35.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.351+0000 7f44667e8700 1 -- 172.21.15.1:0/1706489044 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44601e85b0 con 0x7f44600727b0 2024-03-18T18:50:35.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.352+0000 7f445cff9700 1 -- 172.21.15.1:0/1706489044 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4454003a20 con 0x7f44600727b0 2024-03-18T18:50:35.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.352+0000 7f445cff9700 1 --2- 172.21.15.1:0/1706489044 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f444c06ac90 0x7f444c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:35.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.352+0000 7f445cff9700 1 -- 172.21.15.1:0/1706489044 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f4454088b10 con 0x7f44600727b0 2024-03-18T18:50:35.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.352+0000 7f445f7fe700 1 --2- 172.21.15.1:0/1706489044 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f444c06ac90 0x7f444c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:35.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.353+0000 7f445f7fe700 1 --2- 172.21.15.1:0/1706489044 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f444c06ac90 0x7f444c06d150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4460157910 tx=0x7f4450008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:35.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.357+0000 7f445cff9700 1 -- 172.21.15.1:0/1706489044 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f44601e85b0 con 0x7f44600727b0 2024-03-18T18:50:35.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.556+0000 7f44667e8700 1 -- 172.21.15.1:0/1706489044 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f44601e85b0 con 0x7f44600727b0 2024-03-18T18:50:35.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.557+0000 7f445cff9700 1 -- 172.21.15.1:0/1706489044 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f44601e85b0 con 0x7f44600727b0 2024-03-18T18:50:35.554 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:35.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.559+0000 7f44667e8700 1 -- 172.21.15.1:0/1706489044 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f444c06ac90 msgr2=0x7f444c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:35.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.559+0000 7f44667e8700 1 --2- 172.21.15.1:0/1706489044 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f444c06ac90 0x7f444c06d150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4460157910 tx=0x7f4450008040 comp rx=0 tx=0).stop 2024-03-18T18:50:35.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.559+0000 7f44667e8700 1 -- 172.21.15.1:0/1706489044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44600727b0 msgr2=0x7f4460151190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:35.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.559+0000 7f44667e8700 1 --2- 172.21.15.1:0/1706489044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44600727b0 0x7f4460151190 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f445400c4e0 tx=0x7f445400c8a0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.559+0000 7f44667e8700 1 -- 172.21.15.1:0/1706489044 shutdown_connections 2024-03-18T18:50:35.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.559+0000 7f44667e8700 1 --2- 172.21.15.1:0/1706489044 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f444c06ac90 0x7f444c06d150 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.559+0000 7f44667e8700 1 --2- 172.21.15.1:0/1706489044 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4460161b50 0x7f446015c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.559+0000 7f44667e8700 1 --2- 172.21.15.1:0/1706489044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44600727b0 0x7f4460151190 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.559+0000 7f44667e8700 1 --2- 172.21.15.1:0/1706489044 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f446015ad70 0x7f446015c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.559+0000 7f44667e8700 1 -- 172.21.15.1:0/1706489044 >> 172.21.15.1:0/1706489044 conn(0x7f44600b1cb0 msgr2=0x7f44600b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:35.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.559+0000 7f44667e8700 1 -- 172.21.15.1:0/1706489044 shutdown_connections 2024-03-18T18:50:35.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.559+0000 7f44667e8700 1 -- 172.21.15.1:0/1706489044 wait complete. 2024-03-18T18:50:35.568 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:35.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.695+0000 7fdfabb90700 1 Processor -- start 2024-03-18T18:50:35.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.695+0000 7fdfabb90700 1 -- start start 2024-03-18T18:50:35.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.696+0000 7fdfabb90700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdfa406f240 0x7fdfa4164ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:35.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.696+0000 7fdfabb90700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa4144000 con 0x7fdfa4143c20 2024-03-18T18:50:35.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.696+0000 7fdfabb90700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa40cd8d0 con 0x7fdfa40b4110 2024-03-18T18:50:35.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.696+0000 7fdfabb90700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa40b51b0 con 0x7fdfa406f240 2024-03-18T18:50:35.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.696+0000 7fdfa992c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdfa4143c20 0x7fdfa40b3a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37352/0 (socket says 172.21.15.1:37352) 2024-03-18T18:50:35.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.696+0000 7fdfa992c700 1 -- 172.21.15.1:0/2061413991 learned_addr learned my addr 172.21.15.1:0/2061413991 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:35.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa892a700 1 -- 172.21.15.1:0/2061413991 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235633864 0 0) 0x7fdfa4144000 con 0x7fdfa4143c20 2024-03-18T18:50:35.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa892a700 1 -- 172.21.15.1:0/2061413991 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf900035b0 con 0x7fdfa4143c20 2024-03-18T18:50:35.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfaa12d700 1 --2- 172.21.15.1:0/2061413991 >> v2:172.21.15.165:3300/0 conn(0x7fdfa406f240 0x7fdfa4164ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:35.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa892a700 1 -- 172.21.15.1:0/2061413991 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 566243585 0 0) 0x7fdf900035b0 con 0x7fdfa4143c20 2024-03-18T18:50:35.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa892a700 1 -- 172.21.15.1:0/2061413991 >> v2:172.21.15.165:3300/0 conn(0x7fdfa406f240 msgr2=0x7fdfa4164ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:35.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa892a700 1 --2- 172.21.15.1:0/2061413991 >> v2:172.21.15.165:3300/0 conn(0x7fdfa406f240 0x7fdfa4164ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa892a700 1 -- 172.21.15.1:0/2061413991 >> v1:172.21.15.165:6789/0 conn(0x7fdfa40b4110 legacy=0x7fdfa406eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:35.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa892a700 1 -- 172.21.15.1:0/2061413991 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa406fdc0 con 0x7fdfa4143c20 2024-03-18T18:50:35.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa892a700 1 -- 172.21.15.1:0/2061413991 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fdfa0001670 con 0x7fdfa4143c20 2024-03-18T18:50:35.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa892a700 1 -- 172.21.15.1:0/2061413991 >> v1:172.21.15.1:6789/0 conn(0x7fdfa4143c20 legacy=0x7fdfa40b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:35.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa892a700 1 --2- 172.21.15.1:0/2061413991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf90003a00 0x7fdf90003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:35.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa892a700 1 --2- 172.21.15.1:0/2061413991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf90004850 0x7fdf90004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:35.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa892a700 1 --2- 172.21.15.1:0/2061413991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf90005210 0x7fdf9000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa892a700 1 -- 172.21.15.1:0/2061413991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9000c130 con 0x7fdf90004850 2024-03-18T18:50:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa892a700 1 -- 172.21.15.1:0/2061413991 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9000c2a0 con 0x7fdf90005210 2024-03-18T18:50:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa892a700 1 -- 172.21.15.1:0/2061413991 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9000c460 con 0x7fdf90003a00 2024-03-18T18:50:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa912b700 1 --2- 172.21.15.1:0/2061413991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf90004850 0x7fdf90004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa992c700 1 --2- 172.21.15.1:0/2061413991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf90003a00 0x7fdf90003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfaa12d700 1 --2- 172.21.15.1:0/2061413991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf90005210 0x7fdf9000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa912b700 1 -- 172.21.15.1:0/2061413991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf90003a00 msgr2=0x7fdf90003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa912b700 1 --2- 172.21.15.1:0/2061413991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf90003a00 0x7fdf90003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa912b700 1 -- 172.21.15.1:0/2061413991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf90005210 msgr2=0x7fdf9000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa912b700 1 --2- 172.21.15.1:0/2061413991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf90005210 0x7fdf9000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfa912b700 1 -- 172.21.15.1:0/2061413991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf900035b0 con 0x7fdf90004850 2024-03-18T18:50:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.697+0000 7fdfaa12d700 1 -- 172.21.15.1:0/2061413991 reap_dead start 2024-03-18T18:50:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.698+0000 7fdfa912b700 1 --2- 172.21.15.1:0/2061413991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf90004850 0x7fdf90004cd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4072890 tx=0x7fdf90005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78dacc25e5b83932 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:35.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.698+0000 7fdfa892a700 1 -- 172.21.15.1:0/2061413991 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf980052d0 con 0x7fdf90004850 2024-03-18T18:50:35.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.698+0000 7fdfa892a700 1 -- 172.21.15.1:0/2061413991 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf98005450 con 0x7fdf90004850 2024-03-18T18:50:35.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.698+0000 7fdfa892a700 1 -- 172.21.15.1:0/2061413991 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf980038c0 con 0x7fdf90004850 2024-03-18T18:50:35.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.698+0000 7fdfabb90700 1 -- 172.21.15.1:0/2061413991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf90004850 msgr2=0x7fdf90004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:35.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.698+0000 7fdfabb90700 1 --2- 172.21.15.1:0/2061413991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf90004850 0x7fdf90004cd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4072890 tx=0x7fdf90005880 comp rx=0 tx=0).stop 2024-03-18T18:50:35.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.698+0000 7fdfabb90700 1 -- 172.21.15.1:0/2061413991 shutdown_connections 2024-03-18T18:50:35.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.698+0000 7fdfabb90700 1 --2- 172.21.15.1:0/2061413991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf90004850 0x7fdf90004cd0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.698+0000 7fdfabb90700 1 -- 172.21.15.1:0/2061413991 >> 172.21.15.1:0/2061413991 conn(0x7fdfa40b1d20 msgr2=0x7fdfa40b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:35.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.698+0000 7fdfabb90700 1 -- 172.21.15.1:0/2061413991 shutdown_connections 2024-03-18T18:50:35.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.698+0000 7fdfabb90700 1 -- 172.21.15.1:0/2061413991 wait complete. 2024-03-18T18:50:35.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.699+0000 7fdfabb90700 1 Processor -- start 2024-03-18T18:50:35.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.700+0000 7fdfabb90700 1 -- start start 2024-03-18T18:50:35.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.700+0000 7fdfabb90700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf90004850 0x7fdfa41f2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:35.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.700+0000 7fdfabb90700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfa41593d0 0x7fdfa4138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:35.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.700+0000 7fdfabb90700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdfa4159920 0x7fdfa413d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:35.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.700+0000 7fdfabb90700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa41f33c0 con 0x7fdfa41593d0 2024-03-18T18:50:35.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.700+0000 7fdfabb90700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa413cfe0 con 0x7fdf90004850 2024-03-18T18:50:35.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.700+0000 7fdfabb90700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa413fec0 con 0x7fdfa4159920 2024-03-18T18:50:35.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.700+0000 7fdfa912b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfa41593d0 0x7fdfa4138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:35.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.700+0000 7fdfa912b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfa41593d0 0x7fdfa4138dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49650/0 (socket says 172.21.15.1:49650) 2024-03-18T18:50:35.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.700+0000 7fdfa912b700 1 -- 172.21.15.1:0/2820230993 learned_addr learned my addr 172.21.15.1:0/2820230993 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:35.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.701+0000 7fdfa992c700 1 --2- 172.21.15.1:0/2820230993 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf90004850 0x7fdfa41f2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:35.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.701+0000 7fdfaa12d700 1 --2- 172.21.15.1:0/2820230993 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdfa4159920 0x7fdfa413d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:35.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.701+0000 7fdfa912b700 1 -- 172.21.15.1:0/2820230993 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdfa4159920 msgr2=0x7fdfa413d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:35.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.701+0000 7fdfa912b700 1 --2- 172.21.15.1:0/2820230993 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdfa4159920 0x7fdfa413d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.701+0000 7fdfa912b700 1 -- 172.21.15.1:0/2820230993 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf90004850 msgr2=0x7fdfa41f2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:35.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.701+0000 7fdfa912b700 1 --2- 172.21.15.1:0/2820230993 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf90004850 0x7fdfa41f2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.701+0000 7fdfa912b700 1 -- 172.21.15.1:0/2820230993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa41e3c20 con 0x7fdfa41593d0 2024-03-18T18:50:35.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.701+0000 7fdfaa12d700 1 --2- 172.21.15.1:0/2820230993 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdfa4159920 0x7fdfa413d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:35.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.701+0000 7fdfa912b700 1 --2- 172.21.15.1:0/2820230993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfa41593d0 0x7fdfa4138dc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdf9000c270 tx=0x7fdf98001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:35.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.701+0000 7fdf967fc700 1 -- 172.21.15.1:0/2820230993 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf98006ea0 con 0x7fdfa41593d0 2024-03-18T18:50:35.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.701+0000 7fdf967fc700 1 -- 172.21.15.1:0/2820230993 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf98007450 con 0x7fdfa41593d0 2024-03-18T18:50:35.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.701+0000 7fdfabb90700 1 -- 172.21.15.1:0/2820230993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa413cd00 con 0x7fdfa41593d0 2024-03-18T18:50:35.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.701+0000 7fdf967fc700 1 -- 172.21.15.1:0/2820230993 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf98001290 con 0x7fdfa41593d0 2024-03-18T18:50:35.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.701+0000 7fdfabb90700 1 -- 172.21.15.1:0/2820230993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa41e41a0 con 0x7fdfa41593d0 2024-03-18T18:50:35.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.702+0000 7fdfabb90700 1 -- 172.21.15.1:0/2820230993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa41e3a80 con 0x7fdfa41593d0 2024-03-18T18:50:35.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.702+0000 7fdf967fc700 1 -- 172.21.15.1:0/2820230993 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fdf98003680 con 0x7fdfa41593d0 2024-03-18T18:50:35.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.703+0000 7fdf967fc700 1 --2- 172.21.15.1:0/2820230993 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf8806ac90 0x7fdf8806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:35.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.703+0000 7fdf967fc700 1 -- 172.21.15.1:0/2820230993 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fdf98089a80 con 0x7fdfa41593d0 2024-03-18T18:50:35.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.703+0000 7fdfa992c700 1 --2- 172.21.15.1:0/2820230993 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf8806ac90 0x7fdf8806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:35.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.703+0000 7fdfa992c700 1 --2- 172.21.15.1:0/2820230993 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf8806ac90 0x7fdf8806d150 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0000c00 tx=0x7fdfa0007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:35.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.707+0000 7fdf967fc700 1 -- 172.21.15.1:0/2820230993 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdfa41e3a80 con 0x7fdfa41593d0 2024-03-18T18:50:35.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.913+0000 7fdfabb90700 1 -- 172.21.15.1:0/2820230993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfa41e3a80 con 0x7fdfa41593d0 2024-03-18T18:50:35.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.914+0000 7fdf967fc700 1 -- 172.21.15.1:0/2820230993 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18217 (secure 0 0 0) 0x7fdfa41e3a80 con 0x7fdfa41593d0 2024-03-18T18:50:35.911 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:35.912 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":27,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:50:34.993482+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:35.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.916+0000 7fdfabb90700 1 -- 172.21.15.1:0/2820230993 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf8806ac90 msgr2=0x7fdf8806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:35.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.916+0000 7fdfabb90700 1 --2- 172.21.15.1:0/2820230993 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf8806ac90 0x7fdf8806d150 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0000c00 tx=0x7fdfa0007040 comp rx=0 tx=0).stop 2024-03-18T18:50:35.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.916+0000 7fdfabb90700 1 -- 172.21.15.1:0/2820230993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfa41593d0 msgr2=0x7fdfa4138dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:35.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.916+0000 7fdfabb90700 1 --2- 172.21.15.1:0/2820230993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfa41593d0 0x7fdfa4138dc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdf9000c270 tx=0x7fdf98001120 comp rx=0 tx=0).stop 2024-03-18T18:50:35.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.916+0000 7fdfabb90700 1 -- 172.21.15.1:0/2820230993 shutdown_connections 2024-03-18T18:50:35.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.916+0000 7fdfabb90700 1 --2- 172.21.15.1:0/2820230993 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf8806ac90 0x7fdf8806d150 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.916+0000 7fdfabb90700 1 --2- 172.21.15.1:0/2820230993 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdfa4159920 0x7fdfa413d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.916+0000 7fdfabb90700 1 --2- 172.21.15.1:0/2820230993 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf90004850 0x7fdfa41f2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.916+0000 7fdfabb90700 1 --2- 172.21.15.1:0/2820230993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfa41593d0 0x7fdfa4138dc0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:35.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.916+0000 7fdfabb90700 1 -- 172.21.15.1:0/2820230993 >> 172.21.15.1:0/2820230993 conn(0x7fdfa40b1d20 msgr2=0x7fdfa40b38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:35.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.916+0000 7fdfabb90700 1 -- 172.21.15.1:0/2820230993 shutdown_connections 2024-03-18T18:50:35.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:35.916+0000 7fdfabb90700 1 -- 172.21.15.1:0/2820230993 wait complete. 2024-03-18T18:50:35.926 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T18:49:42.800801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T18:49:43.776684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T18:49:46.854608+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T18:50:35.927 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T18:50:36.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.057+0000 7ffb23978700 1 Processor -- start 2024-03-18T18:50:36.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.057+0000 7ffb23978700 1 -- start start 2024-03-18T18:50:36.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.058+0000 7ffb23978700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffb1c15d5a0 0x7ffb1c0ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:36.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.058+0000 7ffb23978700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb1c0ce550 0x7ffb1c06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:36.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.058+0000 7ffb23978700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ffb1c0cfa90 0x7ffb1c06ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:36.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.058+0000 7ffb23978700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1c0cf600 con 0x7ffb1c15d5a0 2024-03-18T18:50:36.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.058+0000 7ffb23978700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1c0ce9f0 con 0x7ffb1c0cfa90 2024-03-18T18:50:36.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.058+0000 7ffb23978700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c0cecd0 con 0x7ffb1c0ce550 2024-03-18T18:50:36.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.058+0000 7ffb21714700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffb1c15d5a0 0x7ffb1c0ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:36.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.058+0000 7ffb21714700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffb1c15d5a0 0x7ffb1c0ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49662/0 (socket says 172.21.15.1:49662) 2024-03-18T18:50:36.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.058+0000 7ffb21714700 1 -- 172.21.15.1:0/2434497159 learned_addr learned my addr 172.21.15.1:0/2434497159 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:36.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.058+0000 7ffb20f13700 1 --2- 172.21.15.1:0/2434497159 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb1c0ce550 0x7ffb1c06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:36.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.058+0000 7ffb21f15700 1 --2- 172.21.15.1:0/2434497159 >> v2:172.21.15.165:3300/0 conn(0x7ffb1c0cfa90 0x7ffb1c06ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:36.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.058+0000 7ffb21714700 1 -- 172.21.15.1:0/2434497159 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb1c0ce550 msgr2=0x7ffb1c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:36.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.058+0000 7ffb21714700 1 --2- 172.21.15.1:0/2434497159 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb1c0ce550 0x7ffb1c06a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.058+0000 7ffb21714700 1 -- 172.21.15.1:0/2434497159 >> v2:172.21.15.165:3300/0 conn(0x7ffb1c0cfa90 msgr2=0x7ffb1c06ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:36.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.058+0000 7ffb21714700 1 --2- 172.21.15.1:0/2434497159 >> v2:172.21.15.165:3300/0 conn(0x7ffb1c0cfa90 0x7ffb1c06ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.058+0000 7ffb21714700 1 -- 172.21.15.1:0/2434497159 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1c0cefb0 con 0x7ffb1c15d5a0 2024-03-18T18:50:36.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.058+0000 7ffb21714700 1 --2- 172.21.15.1:0/2434497159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb1c15d5a0 0x7ffb1c0ce010 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c0cdec0 tx=0x7ffb0c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ab239e96b321aa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:36.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.059+0000 7ffb13fff700 1 -- 172.21.15.1:0/2434497159 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c005e30 con 0x7ffb1c15d5a0 2024-03-18T18:50:36.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.059+0000 7ffb13fff700 1 -- 172.21.15.1:0/2434497159 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0c004030 con 0x7ffb1c15d5a0 2024-03-18T18:50:36.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.059+0000 7ffb13fff700 1 -- 172.21.15.1:0/2434497159 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c0043e0 con 0x7ffb1c15d5a0 2024-03-18T18:50:36.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.059+0000 7ffb23978700 1 -- 172.21.15.1:0/2434497159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb1c15d5a0 msgr2=0x7ffb1c0ce010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:36.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.059+0000 7ffb23978700 1 --2- 172.21.15.1:0/2434497159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb1c15d5a0 0x7ffb1c0ce010 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c0cdec0 tx=0x7ffb0c009bd0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.059+0000 7ffb23978700 1 -- 172.21.15.1:0/2434497159 shutdown_connections 2024-03-18T18:50:36.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.059+0000 7ffb23978700 1 --2- 172.21.15.1:0/2434497159 >> v2:172.21.15.165:3300/0 conn(0x7ffb1c0cfa90 0x7ffb1c06ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.059+0000 7ffb23978700 1 --2- 172.21.15.1:0/2434497159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb1c15d5a0 0x7ffb1c0ce010 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.059+0000 7ffb23978700 1 --2- 172.21.15.1:0/2434497159 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb1c0ce550 0x7ffb1c06a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.059+0000 7ffb23978700 1 -- 172.21.15.1:0/2434497159 >> 172.21.15.1:0/2434497159 conn(0x7ffb1c0cc9b0 msgr2=0x7ffb1c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:36.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.059+0000 7ffb23978700 1 -- 172.21.15.1:0/2434497159 shutdown_connections 2024-03-18T18:50:36.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.059+0000 7ffb23978700 1 -- 172.21.15.1:0/2434497159 wait complete. 2024-03-18T18:50:36.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.060+0000 7ffb23978700 1 Processor -- start 2024-03-18T18:50:36.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.061+0000 7ffb23978700 1 -- start start 2024-03-18T18:50:36.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.061+0000 7ffb23978700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb1c0ce550 0x7ffb1c0c5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:36.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.061+0000 7ffb23978700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb1c0cfa90 0x7ffb1c144f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:36.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.061+0000 7ffb23978700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb1c15d5a0 0x7ffb1c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:36.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.061+0000 7ffb23978700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c1465e0 con 0x7ffb1c0ce550 2024-03-18T18:50:36.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.061+0000 7ffb23978700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c0c64d0 con 0x7ffb1c0cfa90 2024-03-18T18:50:36.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.061+0000 7ffb23978700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c1459d0 con 0x7ffb1c15d5a0 2024-03-18T18:50:36.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.061+0000 7ffb21714700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb1c0ce550 0x7ffb1c0c5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:36.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.061+0000 7ffb21714700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb1c0ce550 0x7ffb1c0c5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49672/0 (socket says 172.21.15.1:49672) 2024-03-18T18:50:36.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.061+0000 7ffb21714700 1 -- 172.21.15.1:0/3646163947 learned_addr learned my addr 172.21.15.1:0/3646163947 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:36.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.061+0000 7ffb20f13700 1 --2- 172.21.15.1:0/3646163947 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb1c0cfa90 0x7ffb1c144f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:36.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.062+0000 7ffb21f15700 1 --2- 172.21.15.1:0/3646163947 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb1c15d5a0 0x7ffb1c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:36.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.062+0000 7ffb21714700 1 -- 172.21.15.1:0/3646163947 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb1c15d5a0 msgr2=0x7ffb1c06edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:36.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.062+0000 7ffb21714700 1 --2- 172.21.15.1:0/3646163947 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb1c15d5a0 0x7ffb1c06edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.062+0000 7ffb21714700 1 -- 172.21.15.1:0/3646163947 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb1c0cfa90 msgr2=0x7ffb1c144f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:36.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.062+0000 7ffb21714700 1 --2- 172.21.15.1:0/3646163947 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb1c0cfa90 0x7ffb1c144f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.062+0000 7ffb21714700 1 -- 172.21.15.1:0/3646163947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c009500 con 0x7ffb1c0ce550 2024-03-18T18:50:36.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.062+0000 7ffb20f13700 1 --2- 172.21.15.1:0/3646163947 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb1c0cfa90 0x7ffb1c144f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:36.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.062+0000 7ffb21714700 1 --2- 172.21.15.1:0/3646163947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb1c0ce550 0x7ffb1c0c5f90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009130 tx=0x7ffb0c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:36.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.062+0000 7ffb11ffb700 1 -- 172.21.15.1:0/3646163947 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c003dd0 con 0x7ffb1c0ce550 2024-03-18T18:50:36.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.062+0000 7ffb11ffb700 1 -- 172.21.15.1:0/3646163947 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0c011420 con 0x7ffb1c0ce550 2024-03-18T18:50:36.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.062+0000 7ffb23978700 1 -- 172.21.15.1:0/3646163947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1c145cb0 con 0x7ffb1c0ce550 2024-03-18T18:50:36.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.062+0000 7ffb11ffb700 1 -- 172.21.15.1:0/3646163947 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c0117d0 con 0x7ffb1c0ce550 2024-03-18T18:50:36.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.062+0000 7ffb23978700 1 -- 172.21.15.1:0/3646163947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1c146270 con 0x7ffb1c0ce550 2024-03-18T18:50:36.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.063+0000 7ffb23978700 1 -- 172.21.15.1:0/3646163947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb1c0676a0 con 0x7ffb1c0ce550 2024-03-18T18:50:36.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.064+0000 7ffb11ffb700 1 -- 172.21.15.1:0/3646163947 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ffb0c011930 con 0x7ffb1c0ce550 2024-03-18T18:50:36.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.064+0000 7ffb11ffb700 1 --2- 172.21.15.1:0/3646163947 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffb0006acc0 0x7ffb0006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:36.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.064+0000 7ffb11ffb700 1 -- 172.21.15.1:0/3646163947 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ffb0c08a880 con 0x7ffb1c0ce550 2024-03-18T18:50:36.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.064+0000 7ffb20f13700 1 --2- 172.21.15.1:0/3646163947 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffb0006acc0 0x7ffb0006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:36.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.064+0000 7ffb20f13700 1 --2- 172.21.15.1:0/3646163947 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffb0006acc0 0x7ffb0006d180 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c0b5e90 tx=0x7ffb18008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:36.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.068+0000 7ffb11ffb700 1 -- 172.21.15.1:0/3646163947 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffb0c055730 con 0x7ffb1c0ce550 2024-03-18T18:50:36.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.270+0000 7ffb23978700 1 -- 172.21.15.1:0/3646163947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffb1c06b590 con 0x7ffb1c0ce550 2024-03-18T18:50:36.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.270+0000 7ffb11ffb700 1 -- 172.21.15.1:0/3646163947 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffb0c058d50 con 0x7ffb1c0ce550 2024-03-18T18:50:36.268 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T18:50:36.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.272+0000 7ffb23978700 1 -- 172.21.15.1:0/3646163947 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffb0006acc0 msgr2=0x7ffb0006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:36.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.272+0000 7ffb23978700 1 --2- 172.21.15.1:0/3646163947 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffb0006acc0 0x7ffb0006d180 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c0b5e90 tx=0x7ffb18008040 comp rx=0 tx=0).stop 2024-03-18T18:50:36.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.272+0000 7ffb23978700 1 -- 172.21.15.1:0/3646163947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb1c0ce550 msgr2=0x7ffb1c0c5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:36.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.272+0000 7ffb23978700 1 --2- 172.21.15.1:0/3646163947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb1c0ce550 0x7ffb1c0c5f90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009130 tx=0x7ffb0c004750 comp rx=0 tx=0).stop 2024-03-18T18:50:36.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.272+0000 7ffb23978700 1 -- 172.21.15.1:0/3646163947 shutdown_connections 2024-03-18T18:50:36.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.272+0000 7ffb23978700 1 --2- 172.21.15.1:0/3646163947 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffb0006acc0 0x7ffb0006d180 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.272+0000 7ffb23978700 1 --2- 172.21.15.1:0/3646163947 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb1c15d5a0 0x7ffb1c06edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.272+0000 7ffb23978700 1 --2- 172.21.15.1:0/3646163947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb1c0ce550 0x7ffb1c0c5f90 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.272+0000 7ffb23978700 1 --2- 172.21.15.1:0/3646163947 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb1c0cfa90 0x7ffb1c144f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.272+0000 7ffb23978700 1 -- 172.21.15.1:0/3646163947 >> 172.21.15.1:0/3646163947 conn(0x7ffb1c0cc9b0 msgr2=0x7ffb1c15c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:36.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.272+0000 7ffb23978700 1 -- 172.21.15.1:0/3646163947 shutdown_connections 2024-03-18T18:50:36.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.272+0000 7ffb23978700 1 -- 172.21.15.1:0/3646163947 wait complete. 2024-03-18T18:50:36.282 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T18:50:36.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.413+0000 7f701f324700 1 Processor -- start 2024-03-18T18:50:36.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.413+0000 7f701f324700 1 -- start start 2024-03-18T18:50:36.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.414+0000 7f701f324700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7018071d30 0x7f70180add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:36.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.414+0000 7f701f324700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f70180cb9e0 0x7f70180cbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:36.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.414+0000 7f701f324700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70180ce8f0 con 0x7f701815a1d0 2024-03-18T18:50:36.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.414+0000 7f701f324700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70180cdb00 con 0x7f70180cb9e0 2024-03-18T18:50:36.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.414+0000 7f701f324700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f701815a5b0 con 0x7f7018071d30 2024-03-18T18:50:36.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.414+0000 7f701d0c0700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f701815a1d0 0x7f7018071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37368/0 (socket says 172.21.15.1:37368) 2024-03-18T18:50:36.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.414+0000 7f701d0c0700 1 -- 172.21.15.1:0/1050463804 learned_addr learned my addr 172.21.15.1:0/1050463804 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:36.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.414+0000 7f701d8c1700 1 --2- 172.21.15.1:0/1050463804 >> v2:172.21.15.1:3300/0 conn(0x7f70180cb9e0 0x7f70180cbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:36.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.414+0000 7f701c8bf700 1 --2- 172.21.15.1:0/1050463804 >> v2:172.21.15.165:3300/0 conn(0x7f7018071d30 0x7f70180add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:36.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.414+0000 7f700ffff700 1 -- 172.21.15.1:0/1050463804 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145441946 0 0) 0x7f70180ce8f0 con 0x7f701815a1d0 2024-03-18T18:50:36.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.414+0000 7f700ffff700 1 -- 172.21.15.1:0/1050463804 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70040057c0 con 0x7f701815a1d0 2024-03-18T18:50:36.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.414+0000 7f701d8c1700 1 -- 172.21.15.1:0/1050463804 >> v2:172.21.15.165:3300/0 conn(0x7f7018071d30 msgr2=0x7f70180add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:36.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.414+0000 7f701d8c1700 1 --2- 172.21.15.1:0/1050463804 >> v2:172.21.15.165:3300/0 conn(0x7f7018071d30 0x7f70180add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.414+0000 7f701d8c1700 1 -- 172.21.15.1:0/1050463804 >> v1:172.21.15.1:6789/0 conn(0x7f701815a1d0 legacy=0x7f7018071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:36.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.414+0000 7f701d8c1700 1 -- 172.21.15.1:0/1050463804 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70180ae260 con 0x7f70180cb9e0 2024-03-18T18:50:36.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.415+0000 7f701d8c1700 1 --2- 172.21.15.1:0/1050463804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70180cb9e0 0x7f70180cbe30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f70180cebb0 tx=0x7f701000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3709f71e8256c612 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:36.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.415+0000 7f700ffff700 1 -- 172.21.15.1:0/1050463804 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701000e070 con 0x7f70180cb9e0 2024-03-18T18:50:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.415+0000 7f700ffff700 1 -- 172.21.15.1:0/1050463804 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f701000bb20 con 0x7f70180cb9e0 2024-03-18T18:50:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.415+0000 7f700ffff700 1 -- 172.21.15.1:0/1050463804 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7010012430 con 0x7f70180cb9e0 2024-03-18T18:50:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.415+0000 7f701f324700 1 -- 172.21.15.1:0/1050463804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70180cb9e0 msgr2=0x7f70180cbe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.415+0000 7f701f324700 1 --2- 172.21.15.1:0/1050463804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70180cb9e0 0x7f70180cbe30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f70180cebb0 tx=0x7f701000b330 comp rx=0 tx=0).stop 2024-03-18T18:50:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.415+0000 7f701f324700 1 -- 172.21.15.1:0/1050463804 shutdown_connections 2024-03-18T18:50:36.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.415+0000 7f701f324700 1 --2- 172.21.15.1:0/1050463804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70180cb9e0 0x7f70180cbe30 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.415+0000 7f701f324700 1 --2- 172.21.15.1:0/1050463804 >> v2:172.21.15.165:3300/0 conn(0x7f7018071d30 0x7f70180add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.415+0000 7f701f324700 1 -- 172.21.15.1:0/1050463804 >> 172.21.15.1:0/1050463804 conn(0x7f70180cc9b0 msgr2=0x7f70180ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.415+0000 7f701f324700 1 -- 172.21.15.1:0/1050463804 shutdown_connections 2024-03-18T18:50:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.415+0000 7f701f324700 1 -- 172.21.15.1:0/1050463804 wait complete. 2024-03-18T18:50:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.416+0000 7f701f324700 1 Processor -- start 2024-03-18T18:50:36.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.417+0000 7f701f324700 1 -- start start 2024-03-18T18:50:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.417+0000 7f701f324700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7018071d30 0x7f70181f41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.417+0000 7f701f324700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70180cb9e0 0x7f70181f5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.417+0000 7f701f324700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f701815a1d0 0x7f70181548a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.417+0000 7f701f324700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70181f6950 con 0x7f701815a1d0 2024-03-18T18:50:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.417+0000 7f701f324700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70181f6c30 con 0x7f7018071d30 2024-03-18T18:50:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.417+0000 7f701f324700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70181f6f10 con 0x7f70180cb9e0 2024-03-18T18:50:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.417+0000 7f701d8c1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f701815a1d0 0x7f70181548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:36.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.417+0000 7f701d8c1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f701815a1d0 0x7f70181548a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49692/0 (socket says 172.21.15.1:49692) 2024-03-18T18:50:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.417+0000 7f701d8c1700 1 -- 172.21.15.1:0/3788121836 learned_addr learned my addr 172.21.15.1:0/3788121836 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.418+0000 7f701c8bf700 1 --2- 172.21.15.1:0/3788121836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70180cb9e0 0x7f70181f5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.418+0000 7f701d0c0700 1 --2- 172.21.15.1:0/3788121836 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7018071d30 0x7f70181f41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.418+0000 7f701d8c1700 1 -- 172.21.15.1:0/3788121836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70180cb9e0 msgr2=0x7f70181f5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.418+0000 7f701d8c1700 1 --2- 172.21.15.1:0/3788121836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70180cb9e0 0x7f70181f5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.418+0000 7f701d8c1700 1 -- 172.21.15.1:0/3788121836 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7018071d30 msgr2=0x7f70181f41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.418+0000 7f701d8c1700 1 --2- 172.21.15.1:0/3788121836 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7018071d30 0x7f70181f41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.418+0000 7f701d8c1700 1 -- 172.21.15.1:0/3788121836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7010009a70 con 0x7f701815a1d0 2024-03-18T18:50:36.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.418+0000 7f701c8bf700 1 --2- 172.21.15.1:0/3788121836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70180cb9e0 0x7f70181f5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.418+0000 7f701d8c1700 1 --2- 172.21.15.1:0/3788121836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f701815a1d0 0x7f70181548a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7010014040 tx=0x7f7010004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.418+0000 7f700dffb700 1 -- 172.21.15.1:0/3788121836 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7010012430 con 0x7f701815a1d0 2024-03-18T18:50:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.418+0000 7f700dffb700 1 -- 172.21.15.1:0/3788121836 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7010020070 con 0x7f701815a1d0 2024-03-18T18:50:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.418+0000 7f701f324700 1 -- 172.21.15.1:0/3788121836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70181f71f0 con 0x7f701815a1d0 2024-03-18T18:50:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.418+0000 7f700dffb700 1 -- 172.21.15.1:0/3788121836 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7010003ea0 con 0x7f701815a1d0 2024-03-18T18:50:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.418+0000 7f701f324700 1 -- 172.21.15.1:0/3788121836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70181f6300 con 0x7f701815a1d0 2024-03-18T18:50:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.419+0000 7f701f324700 1 -- 172.21.15.1:0/3788121836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70181f3040 con 0x7f701815a1d0 2024-03-18T18:50:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.420+0000 7f700dffb700 1 -- 172.21.15.1:0/3788121836 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7010004280 con 0x7f701815a1d0 2024-03-18T18:50:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.420+0000 7f700dffb700 1 --2- 172.21.15.1:0/3788121836 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ffc06ac90 0x7f6ffc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:36.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.420+0000 7f700dffb700 1 -- 172.21.15.1:0/3788121836 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7010022070 con 0x7f701815a1d0 2024-03-18T18:50:36.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.420+0000 7f701d0c0700 1 --2- 172.21.15.1:0/3788121836 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ffc06ac90 0x7f6ffc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:36.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.420+0000 7f701d0c0700 1 --2- 172.21.15.1:0/3788121836 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ffc06ac90 0x7f6ffc06d150 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f70140010e0 tx=0x7f7014009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:36.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.425+0000 7f700dffb700 1 -- 172.21.15.1:0/3788121836 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7010056460 con 0x7f701815a1d0 2024-03-18T18:50:36.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.627+0000 7f701f324700 1 -- 172.21.15.1:0/3788121836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f70180676a0 con 0x7f701815a1d0 2024-03-18T18:50:36.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.628+0000 7f700dffb700 1 -- 172.21.15.1:0/3788121836 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7010059a80 con 0x7f701815a1d0 2024-03-18T18:50:36.625 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:36.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.630+0000 7f701f324700 1 -- 172.21.15.1:0/3788121836 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ffc06ac90 msgr2=0x7f6ffc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:36.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.630+0000 7f701f324700 1 --2- 172.21.15.1:0/3788121836 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ffc06ac90 0x7f6ffc06d150 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f70140010e0 tx=0x7f7014009040 comp rx=0 tx=0).stop 2024-03-18T18:50:36.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.630+0000 7f701f324700 1 -- 172.21.15.1:0/3788121836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f701815a1d0 msgr2=0x7f70181548a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:36.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.630+0000 7f701f324700 1 --2- 172.21.15.1:0/3788121836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f701815a1d0 0x7f70181548a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7010014040 tx=0x7f7010004ad0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.630+0000 7f701f324700 1 -- 172.21.15.1:0/3788121836 shutdown_connections 2024-03-18T18:50:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.630+0000 7f701f324700 1 --2- 172.21.15.1:0/3788121836 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ffc06ac90 0x7f6ffc06d150 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.630+0000 7f701f324700 1 --2- 172.21.15.1:0/3788121836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f701815a1d0 0x7f70181548a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.630+0000 7f701f324700 1 --2- 172.21.15.1:0/3788121836 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7018071d30 0x7f70181f41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.630+0000 7f701f324700 1 --2- 172.21.15.1:0/3788121836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70180cb9e0 0x7f70181f5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.630+0000 7f701f324700 1 -- 172.21.15.1:0/3788121836 >> 172.21.15.1:0/3788121836 conn(0x7f70180cc9b0 msgr2=0x7f70180aef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.630+0000 7f701f324700 1 -- 172.21.15.1:0/3788121836 shutdown_connections 2024-03-18T18:50:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.630+0000 7f701f324700 1 -- 172.21.15.1:0/3788121836 wait complete. 2024-03-18T18:50:36.640 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T18:50:36.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.771+0000 7fdd10dc6700 1 Processor -- start 2024-03-18T18:50:36.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.771+0000 7fdd10dc6700 1 -- start start 2024-03-18T18:50:36.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.772+0000 7fdd10dc6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdd0c071d30 0x7fdd0c0add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:36.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.772+0000 7fdd10dc6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd0c0cb9e0 0x7fdd0c0cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:36.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.772+0000 7fdd10dc6700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd0c0ce8f0 con 0x7fdd0c15a1d0 2024-03-18T18:50:36.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.772+0000 7fdd10dc6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd0c0cdb00 con 0x7fdd0c071d30 2024-03-18T18:50:36.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.772+0000 7fdd10dc6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c15a5b0 con 0x7fdd0c0cb9e0 2024-03-18T18:50:36.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.772+0000 7fdd09d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdd0c071d30 0x7fdd0c0add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:36.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.772+0000 7fdd0a59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdd0c15a1d0 0x7fdd0c071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37376/0 (socket says 172.21.15.1:37376) 2024-03-18T18:50:36.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.772+0000 7fdd0a59c700 1 -- 172.21.15.1:0/3663324798 learned_addr learned my addr 172.21.15.1:0/3663324798 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:36.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.772+0000 7fdd0ad9d700 1 --2- 172.21.15.1:0/3663324798 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd0c0cb9e0 0x7fdd0c0cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:36.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.772+0000 7fdd09d9b700 1 -- 172.21.15.1:0/3663324798 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd0c0cb9e0 msgr2=0x7fdd0c0cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:36.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.772+0000 7fdd09d9b700 1 --2- 172.21.15.1:0/3663324798 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd0c0cb9e0 0x7fdd0c0cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.772+0000 7fdd09d9b700 1 -- 172.21.15.1:0/3663324798 >> v1:172.21.15.1:6789/0 conn(0x7fdd0c15a1d0 legacy=0x7fdd0c071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:36.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.773+0000 7fdd09d9b700 1 -- 172.21.15.1:0/3663324798 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd0c0ae260 con 0x7fdd0c071d30 2024-03-18T18:50:36.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.773+0000 7fdd0959a700 1 -- 172.21.15.1:0/3663324798 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431603177 0 0) 0x7fdd0c0ce8f0 con 0x7fdd0c15a1d0 2024-03-18T18:50:36.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.773+0000 7fdd09d9b700 1 --2- 172.21.15.1:0/3663324798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd0c071d30 0x7fdd0c0add20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdd0000bae0 tx=0x7fdd0000bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=727a025d867c454 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:36.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.773+0000 7fdd0959a700 1 -- 172.21.15.1:0/3663324798 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0000dd30 con 0x7fdd0c071d30 2024-03-18T18:50:36.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.773+0000 7fdd0959a700 1 -- 172.21.15.1:0/3663324798 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0000de90 con 0x7fdd0c071d30 2024-03-18T18:50:36.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.773+0000 7fdd0959a700 1 -- 172.21.15.1:0/3663324798 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd00010690 con 0x7fdd0c071d30 2024-03-18T18:50:36.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.773+0000 7fdd10dc6700 1 -- 172.21.15.1:0/3663324798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd0c071d30 msgr2=0x7fdd0c0add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:36.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.773+0000 7fdd10dc6700 1 --2- 172.21.15.1:0/3663324798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd0c071d30 0x7fdd0c0add20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdd0000bae0 tx=0x7fdd0000bdf0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.773+0000 7fdd10dc6700 1 -- 172.21.15.1:0/3663324798 shutdown_connections 2024-03-18T18:50:36.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.773+0000 7fdd10dc6700 1 --2- 172.21.15.1:0/3663324798 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd0c0cb9e0 0x7fdd0c0cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.773+0000 7fdd10dc6700 1 --2- 172.21.15.1:0/3663324798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd0c071d30 0x7fdd0c0add20 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.773+0000 7fdd10dc6700 1 -- 172.21.15.1:0/3663324798 >> 172.21.15.1:0/3663324798 conn(0x7fdd0c0cc9b0 msgr2=0x7fdd0c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:36.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.773+0000 7fdd10dc6700 1 -- 172.21.15.1:0/3663324798 shutdown_connections 2024-03-18T18:50:36.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.774+0000 7fdd10dc6700 1 -- 172.21.15.1:0/3663324798 wait complete. 2024-03-18T18:50:36.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.775+0000 7fdd10dc6700 1 Processor -- start 2024-03-18T18:50:36.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.775+0000 7fdd10dc6700 1 -- start start 2024-03-18T18:50:36.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.776+0000 7fdd10dc6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd0c071d30 0x7fdd0c0c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:36.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.776+0000 7fdd10dc6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd0c0cb9e0 0x7fdd0c0c4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:36.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.776+0000 7fdd10dc6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd0c15a1d0 0x7fdd0c0c49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:36.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.776+0000 7fdd10dc6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c15b9c0 con 0x7fdd0c071d30 2024-03-18T18:50:36.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.776+0000 7fdd10dc6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c15bca0 con 0x7fdd0c15a1d0 2024-03-18T18:50:36.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.776+0000 7fdd10dc6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c15bf80 con 0x7fdd0c0cb9e0 2024-03-18T18:50:36.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.776+0000 7fdd0a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd0c071d30 0x7fdd0c0c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:36.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.776+0000 7fdd0a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd0c071d30 0x7fdd0c0c7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49700/0 (socket says 172.21.15.1:49700) 2024-03-18T18:50:36.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.776+0000 7fdd0a59c700 1 -- 172.21.15.1:0/2117819330 learned_addr learned my addr 172.21.15.1:0/2117819330 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:36.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.776+0000 7fdd09d9b700 1 --2- 172.21.15.1:0/2117819330 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd0c0cb9e0 0x7fdd0c0c4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:36.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.776+0000 7fdd0ad9d700 1 --2- 172.21.15.1:0/2117819330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd0c15a1d0 0x7fdd0c0c49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:36.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.777+0000 7fdd0a59c700 1 -- 172.21.15.1:0/2117819330 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd0c0cb9e0 msgr2=0x7fdd0c0c4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:36.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.777+0000 7fdd0a59c700 1 --2- 172.21.15.1:0/2117819330 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd0c0cb9e0 0x7fdd0c0c4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.777+0000 7fdd0a59c700 1 -- 172.21.15.1:0/2117819330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd0c15a1d0 msgr2=0x7fdd0c0c49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:36.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.777+0000 7fdd0a59c700 1 --2- 172.21.15.1:0/2117819330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd0c15a1d0 0x7fdd0c0c49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.777+0000 7fdd0a59c700 1 -- 172.21.15.1:0/2117819330 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd0000b740 con 0x7fdd0c071d30 2024-03-18T18:50:36.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.777+0000 7fdd0ad9d700 1 --2- 172.21.15.1:0/2117819330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd0c15a1d0 0x7fdd0c0c49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:36.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.777+0000 7fdd0a59c700 1 --2- 172.21.15.1:0/2117819330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd0c071d30 0x7fdd0c0c7000 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00c4a0 tx=0x7fdcfc00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:36.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.777+0000 7fdcfaffd700 1 -- 172.21.15.1:0/2117819330 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc0074e0 con 0x7fdd0c071d30 2024-03-18T18:50:36.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.777+0000 7fdcfaffd700 1 -- 172.21.15.1:0/2117819330 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcfc007640 con 0x7fdd0c071d30 2024-03-18T18:50:36.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.777+0000 7fdd10dc6700 1 -- 172.21.15.1:0/2117819330 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd0c15c260 con 0x7fdd0c071d30 2024-03-18T18:50:36.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.777+0000 7fdcfaffd700 1 -- 172.21.15.1:0/2117819330 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc005a70 con 0x7fdd0c071d30 2024-03-18T18:50:36.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.777+0000 7fdd10dc6700 1 -- 172.21.15.1:0/2117819330 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd0c0c3e30 con 0x7fdd0c071d30 2024-03-18T18:50:36.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.778+0000 7fdd10dc6700 1 -- 172.21.15.1:0/2117819330 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd0c15c780 con 0x7fdd0c071d30 2024-03-18T18:50:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.778+0000 7fdcfaffd700 1 -- 172.21.15.1:0/2117819330 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fdcfc005bd0 con 0x7fdd0c071d30 2024-03-18T18:50:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.779+0000 7fdcfaffd700 1 --2- 172.21.15.1:0/2117819330 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdcec06ac90 0x7fdcec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.779+0000 7fdcfaffd700 1 -- 172.21.15.1:0/2117819330 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fdcfc088c20 con 0x7fdd0c071d30 2024-03-18T18:50:36.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.779+0000 7fdd09d9b700 1 --2- 172.21.15.1:0/2117819330 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdcec06ac90 0x7fdcec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:36.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.779+0000 7fdd09d9b700 1 --2- 172.21.15.1:0/2117819330 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdcec06ac90 0x7fdcec06d150 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdd0000b2c0 tx=0x7fdd00013000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:36.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.784+0000 7fdcfaffd700 1 -- 172.21.15.1:0/2117819330 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdd0c15c780 con 0x7fdd0c071d30 2024-03-18T18:50:36.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.990+0000 7fdd10dc6700 1 -- 172.21.15.1:0/2117819330 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdd0c15c780 con 0x7fdd0c071d30 2024-03-18T18:50:36.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.991+0000 7fdcfaffd700 1 -- 172.21.15.1:0/2117819330 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdd0c15c780 con 0x7fdd0c071d30 2024-03-18T18:50:36.989 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T18:50:36.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.993+0000 7fdd10dc6700 1 -- 172.21.15.1:0/2117819330 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdcec06ac90 msgr2=0x7fdcec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:36.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.993+0000 7fdd10dc6700 1 --2- 172.21.15.1:0/2117819330 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdcec06ac90 0x7fdcec06d150 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdd0000b2c0 tx=0x7fdd00013000 comp rx=0 tx=0).stop 2024-03-18T18:50:36.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.993+0000 7fdd10dc6700 1 -- 172.21.15.1:0/2117819330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd0c071d30 msgr2=0x7fdd0c0c7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:36.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.993+0000 7fdd10dc6700 1 --2- 172.21.15.1:0/2117819330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd0c071d30 0x7fdd0c0c7000 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00c4a0 tx=0x7fdcfc00c7b0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.993+0000 7fdd10dc6700 1 -- 172.21.15.1:0/2117819330 shutdown_connections 2024-03-18T18:50:36.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.993+0000 7fdd10dc6700 1 --2- 172.21.15.1:0/2117819330 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdcec06ac90 0x7fdcec06d150 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.993+0000 7fdd10dc6700 1 --2- 172.21.15.1:0/2117819330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd0c15a1d0 0x7fdd0c0c49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.993+0000 7fdd10dc6700 1 --2- 172.21.15.1:0/2117819330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd0c071d30 0x7fdd0c0c7000 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.993+0000 7fdd10dc6700 1 --2- 172.21.15.1:0/2117819330 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd0c0cb9e0 0x7fdd0c0c4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.993+0000 7fdd10dc6700 1 -- 172.21.15.1:0/2117819330 >> 172.21.15.1:0/2117819330 conn(0x7fdd0c0cc9b0 msgr2=0x7fdd0c0afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.993+0000 7fdd10dc6700 1 -- 172.21.15.1:0/2117819330 shutdown_connections 2024-03-18T18:50:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:36.993+0000 7fdd10dc6700 1 -- 172.21.15.1:0/2117819330 wait complete. 2024-03-18T18:50:37.003 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T18:50:37.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.133+0000 7f18855b1700 1 Processor -- start 2024-03-18T18:50:37.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.134+0000 7f18855b1700 1 -- start start 2024-03-18T18:50:37.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.134+0000 7f18855b1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f188006eda0 0x7f1880161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:37.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.134+0000 7f18855b1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f188006f180 0x7f18800ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:37.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.134+0000 7f18855b1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f188006a750 0x7f188006ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:37.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.134+0000 7f18855b1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18800cb320 con 0x7f188006a750 2024-03-18T18:50:37.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.134+0000 7f18855b1700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18800ce010 con 0x7f188006eda0 2024-03-18T18:50:37.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.134+0000 7f18855b1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f188006b0c0 con 0x7f188006f180 2024-03-18T18:50:37.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.134+0000 7f187f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f188006a750 0x7f188006ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:37.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.134+0000 7f187f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f188006a750 0x7f188006ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49706/0 (socket says 172.21.15.1:49706) 2024-03-18T18:50:37.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.134+0000 7f187f7fe700 1 -- 172.21.15.1:0/3607148743 learned_addr learned my addr 172.21.15.1:0/3607148743 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:37.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.135+0000 7f187effd700 1 --2- 172.21.15.1:0/3607148743 >> v2:172.21.15.165:3300/0 conn(0x7f188006eda0 0x7f1880161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:37.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.135+0000 7f187f7fe700 1 -- 172.21.15.1:0/3607148743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f188006f180 msgr2=0x7f18800ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:37.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.135+0000 7f187e7fc700 1 --2- 172.21.15.1:0/3607148743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f188006f180 0x7f18800ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:37.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.135+0000 7f187f7fe700 1 --2- 172.21.15.1:0/3607148743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f188006f180 0x7f18800ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:37.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.135+0000 7f187f7fe700 1 -- 172.21.15.1:0/3607148743 >> v2:172.21.15.165:3300/0 conn(0x7f188006eda0 msgr2=0x7f1880161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:37.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.135+0000 7f187f7fe700 1 --2- 172.21.15.1:0/3607148743 >> v2:172.21.15.165:3300/0 conn(0x7f188006eda0 0x7f1880161c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:37.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.135+0000 7f187f7fe700 1 -- 172.21.15.1:0/3607148743 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18800caac0 con 0x7f188006a750 2024-03-18T18:50:37.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.135+0000 7f187effd700 1 --2- 172.21.15.1:0/3607148743 >> v2:172.21.15.165:3300/0 conn(0x7f188006eda0 0x7f1880161c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:37.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.135+0000 7f187e7fc700 1 --2- 172.21.15.1:0/3607148743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f188006f180 0x7f18800ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:37.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.135+0000 7f187f7fe700 1 --2- 172.21.15.1:0/3607148743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f188006a750 0x7f188006ab80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f188006d3d0 tx=0x7f187800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67e7d27da9e1be3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:37.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.135+0000 7f187dffb700 1 -- 172.21.15.1:0/3607148743 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187800e070 con 0x7f188006a750 2024-03-18T18:50:37.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.135+0000 7f187dffb700 1 -- 172.21.15.1:0/3607148743 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f187800b9c0 con 0x7f188006a750 2024-03-18T18:50:37.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.135+0000 7f187dffb700 1 -- 172.21.15.1:0/3607148743 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187800bd70 con 0x7f188006a750 2024-03-18T18:50:37.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.136+0000 7f18855b1700 1 -- 172.21.15.1:0/3607148743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f188006a750 msgr2=0x7f188006ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:37.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.136+0000 7f18855b1700 1 --2- 172.21.15.1:0/3607148743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f188006a750 0x7f188006ab80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f188006d3d0 tx=0x7f187800b330 comp rx=0 tx=0).stop 2024-03-18T18:50:37.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.136+0000 7f18855b1700 1 -- 172.21.15.1:0/3607148743 shutdown_connections 2024-03-18T18:50:37.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.136+0000 7f18855b1700 1 --2- 172.21.15.1:0/3607148743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f188006a750 0x7f188006ab80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:37.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.136+0000 7f18855b1700 1 --2- 172.21.15.1:0/3607148743 >> v2:172.21.15.165:3300/0 conn(0x7f188006eda0 0x7f1880161c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:37.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.136+0000 7f18855b1700 1 --2- 172.21.15.1:0/3607148743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f188006f180 0x7f18800ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:37.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.136+0000 7f18855b1700 1 -- 172.21.15.1:0/3607148743 >> 172.21.15.1:0/3607148743 conn(0x7f18800cff30 msgr2=0x7f18800d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:37.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.136+0000 7f18855b1700 1 -- 172.21.15.1:0/3607148743 shutdown_connections 2024-03-18T18:50:37.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.136+0000 7f18855b1700 1 -- 172.21.15.1:0/3607148743 wait complete. 2024-03-18T18:50:37.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.137+0000 7f18855b1700 1 Processor -- start 2024-03-18T18:50:37.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.137+0000 7f18855b1700 1 -- start start 2024-03-18T18:50:37.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.138+0000 7f18855b1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f188006a750 0x7f18800ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.138+0000 7f18855b1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f188006eda0 0x7f18801f1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.138+0000 7f18855b1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f188006f180 0x7f18801f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.138+0000 7f18855b1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18801f2a70 con 0x7f188006eda0 2024-03-18T18:50:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.138+0000 7f18855b1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18800c9ca0 con 0x7f188006f180 2024-03-18T18:50:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.138+0000 7f18855b1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18800c9f80 con 0x7f188006a750 2024-03-18T18:50:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.138+0000 7f187e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f188006eda0 0x7f18801f1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.138+0000 7f187e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f188006eda0 0x7f18801f1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49722/0 (socket says 172.21.15.1:49722) 2024-03-18T18:50:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.138+0000 7f187e7fc700 1 -- 172.21.15.1:0/3461356728 learned_addr learned my addr 172.21.15.1:0/3461356728 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.138+0000 7f187effd700 1 --2- 172.21.15.1:0/3461356728 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f188006a750 0x7f18800ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.138+0000 7f187f7fe700 1 --2- 172.21.15.1:0/3461356728 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f188006f180 0x7f18801f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:37.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.138+0000 7f187e7fc700 1 -- 172.21.15.1:0/3461356728 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f188006a750 msgr2=0x7f18800ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:37.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.138+0000 7f187e7fc700 1 --2- 172.21.15.1:0/3461356728 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f188006a750 0x7f18800ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:37.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.138+0000 7f187e7fc700 1 -- 172.21.15.1:0/3461356728 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f188006f180 msgr2=0x7f18801f2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:37.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.138+0000 7f187e7fc700 1 --2- 172.21.15.1:0/3461356728 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f188006f180 0x7f18801f2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:37.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.138+0000 7f187e7fc700 1 -- 172.21.15.1:0/3461356728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1878009b30 con 0x7f188006eda0 2024-03-18T18:50:37.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.138+0000 7f187f7fe700 1 --2- 172.21.15.1:0/3461356728 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f188006f180 0x7f18801f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:37.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.138+0000 7f187e7fc700 1 --2- 172.21.15.1:0/3461356728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f188006eda0 0x7f18801f1ff0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f187000c7f0 tx=0x7f187000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:37.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.139+0000 7f18677fe700 1 -- 172.21.15.1:0/3461356728 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1870011800 con 0x7f188006eda0 2024-03-18T18:50:37.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.139+0000 7f18677fe700 1 -- 172.21.15.1:0/3461356728 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1870011960 con 0x7f188006eda0 2024-03-18T18:50:37.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.139+0000 7f18855b1700 1 -- 172.21.15.1:0/3461356728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18800ca260 con 0x7f188006eda0 2024-03-18T18:50:37.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.139+0000 7f18677fe700 1 -- 172.21.15.1:0/3461356728 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1870011cf0 con 0x7f188006eda0 2024-03-18T18:50:37.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.139+0000 7f18855b1700 1 -- 172.21.15.1:0/3461356728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18800ca820 con 0x7f188006eda0 2024-03-18T18:50:37.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.140+0000 7f18855b1700 1 -- 172.21.15.1:0/3461356728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1880152c10 con 0x7f188006eda0 2024-03-18T18:50:37.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.140+0000 7f18677fe700 1 -- 172.21.15.1:0/3461356728 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1870018360 con 0x7f188006eda0 2024-03-18T18:50:37.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.140+0000 7f18677fe700 1 --2- 172.21.15.1:0/3461356728 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f186c06ac90 0x7f186c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:37.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.140+0000 7f18677fe700 1 -- 172.21.15.1:0/3461356728 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f1870089630 con 0x7f188006eda0 2024-03-18T18:50:37.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.141+0000 7f187effd700 1 --2- 172.21.15.1:0/3461356728 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f186c06ac90 0x7f186c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:37.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.141+0000 7f187effd700 1 --2- 172.21.15.1:0/3461356728 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f186c06ac90 0x7f186c06d150 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1874006ba0 tx=0x7f1874008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:37.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.145+0000 7f18677fe700 1 -- 172.21.15.1:0/3461356728 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1880152c10 con 0x7f188006eda0 2024-03-18T18:50:37.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.344+0000 7f18855b1700 1 -- 172.21.15.1:0/3461356728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1880152c10 con 0x7f188006eda0 2024-03-18T18:50:37.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.345+0000 7f18677fe700 1 -- 172.21.15.1:0/3461356728 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1880152c10 con 0x7f188006eda0 2024-03-18T18:50:37.342 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:37.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.347+0000 7f18855b1700 1 -- 172.21.15.1:0/3461356728 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f186c06ac90 msgr2=0x7f186c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:37.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.347+0000 7f18855b1700 1 --2- 172.21.15.1:0/3461356728 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f186c06ac90 0x7f186c06d150 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1874006ba0 tx=0x7f1874008040 comp rx=0 tx=0).stop 2024-03-18T18:50:37.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.347+0000 7f18855b1700 1 -- 172.21.15.1:0/3461356728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f188006eda0 msgr2=0x7f18801f1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:37.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.347+0000 7f18855b1700 1 --2- 172.21.15.1:0/3461356728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f188006eda0 0x7f18801f1ff0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f187000c7f0 tx=0x7f187000cb00 comp rx=0 tx=0).stop 2024-03-18T18:50:37.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.347+0000 7f18855b1700 1 -- 172.21.15.1:0/3461356728 shutdown_connections 2024-03-18T18:50:37.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.347+0000 7f18855b1700 1 --2- 172.21.15.1:0/3461356728 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f186c06ac90 0x7f186c06d150 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:37.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.347+0000 7f18855b1700 1 --2- 172.21.15.1:0/3461356728 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f188006a750 0x7f18800ad4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:37.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.347+0000 7f18855b1700 1 --2- 172.21.15.1:0/3461356728 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f188006f180 0x7f18801f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:37.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.347+0000 7f18855b1700 1 --2- 172.21.15.1:0/3461356728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f188006eda0 0x7f18801f1ff0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:37.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.347+0000 7f18855b1700 1 -- 172.21.15.1:0/3461356728 >> 172.21.15.1:0/3461356728 conn(0x7f18800cff30 msgr2=0x7f18800aedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:37.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.347+0000 7f18855b1700 1 -- 172.21.15.1:0/3461356728 shutdown_connections 2024-03-18T18:50:37.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.347+0000 7f18855b1700 1 -- 172.21.15.1:0/3461356728 wait complete. 2024-03-18T18:50:37.391 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2024-03-18T18:50:37.391 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-18T18:50:37.391 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2024-03-18T18:50:37.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.519+0000 7f4e88f26700 1 Processor -- start 2024-03-18T18:50:37.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.519+0000 7f4e88f26700 1 -- start start 2024-03-18T18:50:37.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.520+0000 7f4e88f26700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4e840af610 0x7f4e840cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:37.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.520+0000 7f4e88f26700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e840cf090 con 0x7f4e84144b70 2024-03-18T18:50:37.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.520+0000 7f4e88f26700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e840cf2d0 con 0x7f4e840ccf40 2024-03-18T18:50:37.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.520+0000 7f4e88f26700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e84144f50 con 0x7f4e840af610 2024-03-18T18:50:37.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.520+0000 7f4e81d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4e840af610 0x7f4e840cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:37.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.520+0000 7f4e8259c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4e84144b70 0x7f4e840ce980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37384/0 (socket says 172.21.15.1:37384) 2024-03-18T18:50:37.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.520+0000 7f4e8259c700 1 -- 172.21.15.1:0/4293557388 learned_addr learned my addr 172.21.15.1:0/4293557388 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:37.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.520+0000 7f4e81d9b700 1 -- 172.21.15.1:0/4293557388 >> v1:172.21.15.165:6789/0 conn(0x7f4e840ccf40 legacy=0x7f4e84071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:37.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.520+0000 7f4e81d9b700 1 -- 172.21.15.1:0/4293557388 >> v1:172.21.15.1:6789/0 conn(0x7f4e84144b70 legacy=0x7f4e840ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:37.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.520+0000 7f4e81d9b700 1 -- 172.21.15.1:0/4293557388 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e840cf5b0 con 0x7f4e840af610 2024-03-18T18:50:37.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.520+0000 7f4e8159a700 1 -- 172.21.15.1:0/4293557388 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897771178 0 0) 0x7f4e78001110 con 0x7f4e84144b70 2024-03-18T18:50:37.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.521+0000 7f4e81d9b700 1 --2- 172.21.15.1:0/4293557388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e840af610 0x7f4e840cca00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4e84145090 tx=0x7f4e74009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5396e74f61fde7d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:37.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.521+0000 7f4e8159a700 1 -- 172.21.15.1:0/4293557388 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e74004020 con 0x7f4e840af610 2024-03-18T18:50:37.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.521+0000 7f4e8159a700 1 -- 172.21.15.1:0/4293557388 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e74004180 con 0x7f4e840af610 2024-03-18T18:50:37.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.521+0000 7f4e8159a700 1 -- 172.21.15.1:0/4293557388 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e74004530 con 0x7f4e840af610 2024-03-18T18:50:37.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.521+0000 7f4e88f26700 1 -- 172.21.15.1:0/4293557388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e840af610 msgr2=0x7f4e840cca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:37.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.521+0000 7f4e88f26700 1 --2- 172.21.15.1:0/4293557388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e840af610 0x7f4e840cca00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4e84145090 tx=0x7f4e74009b20 comp rx=0 tx=0).stop 2024-03-18T18:50:37.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.521+0000 7f4e88f26700 1 -- 172.21.15.1:0/4293557388 shutdown_connections 2024-03-18T18:50:37.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.521+0000 7f4e88f26700 1 --2- 172.21.15.1:0/4293557388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e840af610 0x7f4e840cca00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:37.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.521+0000 7f4e88f26700 1 -- 172.21.15.1:0/4293557388 >> 172.21.15.1:0/4293557388 conn(0x7f4e840b1d20 msgr2=0x7f4e840b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:37.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.521+0000 7f4e88f26700 1 -- 172.21.15.1:0/4293557388 shutdown_connections 2024-03-18T18:50:37.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.521+0000 7f4e88f26700 1 -- 172.21.15.1:0/4293557388 wait complete. 2024-03-18T18:50:37.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.523+0000 7f4e88f26700 1 Processor -- start 2024-03-18T18:50:37.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.523+0000 7f4e88f26700 1 -- start start 2024-03-18T18:50:37.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.523+0000 7f4e88f26700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e840af610 0x7f4e84140eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:37.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.523+0000 7f4e88f26700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e840ccf40 0x7f4e84138080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:37.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.523+0000 7f4e88f26700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e84144b70 0x7f4e84139b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:37.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.523+0000 7f4e88f26700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e841543b0 con 0x7f4e840af610 2024-03-18T18:50:37.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.523+0000 7f4e88f26700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e84137cd0 con 0x7f4e840ccf40 2024-03-18T18:50:37.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.523+0000 7f4e88f26700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8413a9f0 con 0x7f4e84144b70 2024-03-18T18:50:37.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.523+0000 7f4e8259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e840af610 0x7f4e84140eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:37.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.524+0000 7f4e8259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e840af610 0x7f4e84140eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49736/0 (socket says 172.21.15.1:49736) 2024-03-18T18:50:37.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.524+0000 7f4e8259c700 1 -- 172.21.15.1:0/2600488339 learned_addr learned my addr 172.21.15.1:0/2600488339 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:37.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.524+0000 7f4e81d9b700 1 --2- 172.21.15.1:0/2600488339 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e840ccf40 0x7f4e84138080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:37.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.524+0000 7f4e82d9d700 1 --2- 172.21.15.1:0/2600488339 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e84144b70 0x7f4e84139b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:37.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.524+0000 7f4e8259c700 1 -- 172.21.15.1:0/2600488339 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e84144b70 msgr2=0x7f4e84139b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:37.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.524+0000 7f4e8259c700 1 --2- 172.21.15.1:0/2600488339 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e84144b70 0x7f4e84139b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:37.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.524+0000 7f4e8259c700 1 -- 172.21.15.1:0/2600488339 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e840ccf40 msgr2=0x7f4e84138080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:37.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.524+0000 7f4e8259c700 1 --2- 172.21.15.1:0/2600488339 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e840ccf40 0x7f4e84138080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:37.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.524+0000 7f4e8259c700 1 -- 172.21.15.1:0/2600488339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e740094a0 con 0x7f4e840af610 2024-03-18T18:50:37.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.524+0000 7f4e81d9b700 1 --2- 172.21.15.1:0/2600488339 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e840ccf40 0x7f4e84138080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:37.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.524+0000 7f4e8259c700 1 --2- 172.21.15.1:0/2600488339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e840af610 0x7f4e84140eb0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800c520 tx=0x7f4e7800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:37.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.524+0000 7f4e72ffd700 1 -- 172.21.15.1:0/2600488339 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e78004c80 con 0x7f4e840af610 2024-03-18T18:50:37.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.524+0000 7f4e72ffd700 1 -- 172.21.15.1:0/2600488339 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e78004e20 con 0x7f4e840af610 2024-03-18T18:50:37.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.524+0000 7f4e88f26700 1 -- 172.21.15.1:0/2600488339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e8413acd0 con 0x7f4e840af610 2024-03-18T18:50:37.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.524+0000 7f4e72ffd700 1 -- 172.21.15.1:0/2600488339 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e78007310 con 0x7f4e840af610 2024-03-18T18:50:37.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.524+0000 7f4e88f26700 1 -- 172.21.15.1:0/2600488339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e841413f0 con 0x7f4e840af610 2024-03-18T18:50:37.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.525+0000 7f4e88f26700 1 -- 172.21.15.1:0/2600488339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e840cda90 con 0x7f4e840af610 2024-03-18T18:50:37.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.526+0000 7f4e72ffd700 1 -- 172.21.15.1:0/2600488339 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4e78007470 con 0x7f4e840af610 2024-03-18T18:50:37.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.526+0000 7f4e72ffd700 1 --2- 172.21.15.1:0/2600488339 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4e6406ac90 0x7f4e6406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:37.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.526+0000 7f4e72ffd700 1 -- 172.21.15.1:0/2600488339 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f4e78089170 con 0x7f4e840af610 2024-03-18T18:50:37.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.526+0000 7f4e81d9b700 1 --2- 172.21.15.1:0/2600488339 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4e6406ac90 0x7f4e6406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:37.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.527+0000 7f4e81d9b700 1 --2- 172.21.15.1:0/2600488339 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4e6406ac90 0x7f4e6406d150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4e84145090 tx=0x7f4e74008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:37.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.531+0000 7f4e72ffd700 1 -- 172.21.15.1:0/2600488339 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4e78054020 con 0x7f4e840af610 2024-03-18T18:50:37.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:37.816+0000 7f4e88f26700 1 -- 172.21.15.1:0/2600488339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f4e8413b1f0 con 0x7f4e840af610 2024-03-18T18:50:38.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.009+0000 7f4e72ffd700 1 -- 172.21.15.1:0/2600488339 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v65) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4e8413b1f0 con 0x7f4e840af610 2024-03-18T18:50:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.011+0000 7f4e88f26700 1 -- 172.21.15.1:0/2600488339 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4e6406ac90 msgr2=0x7f4e6406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.011+0000 7f4e88f26700 1 --2- 172.21.15.1:0/2600488339 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4e6406ac90 0x7f4e6406d150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4e84145090 tx=0x7f4e74008030 comp rx=0 tx=0).stop 2024-03-18T18:50:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.011+0000 7f4e88f26700 1 -- 172.21.15.1:0/2600488339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e840af610 msgr2=0x7f4e84140eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.011+0000 7f4e88f26700 1 --2- 172.21.15.1:0/2600488339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e840af610 0x7f4e84140eb0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800c520 tx=0x7f4e7800c8e0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.011+0000 7f4e88f26700 1 -- 172.21.15.1:0/2600488339 shutdown_connections 2024-03-18T18:50:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.011+0000 7f4e88f26700 1 --2- 172.21.15.1:0/2600488339 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4e6406ac90 0x7f4e6406d150 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.011+0000 7f4e88f26700 1 --2- 172.21.15.1:0/2600488339 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e84144b70 0x7f4e84139b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.011+0000 7f4e88f26700 1 --2- 172.21.15.1:0/2600488339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e840af610 0x7f4e84140eb0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.011+0000 7f4e88f26700 1 --2- 172.21.15.1:0/2600488339 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e840ccf40 0x7f4e84138080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.011+0000 7f4e88f26700 1 -- 172.21.15.1:0/2600488339 >> 172.21.15.1:0/2600488339 conn(0x7f4e840b1d20 msgr2=0x7f4e84071910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.011+0000 7f4e88f26700 1 -- 172.21.15.1:0/2600488339 shutdown_connections 2024-03-18T18:50:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.011+0000 7f4e88f26700 1 -- 172.21.15.1:0/2600488339 wait complete. 2024-03-18T18:50:38.021 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-18T18:50:38.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.152+0000 7f8d279b0700 1 Processor -- start 2024-03-18T18:50:38.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.152+0000 7f8d279b0700 1 -- start start 2024-03-18T18:50:38.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d279b0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8d2015d780 0x7f8d200cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:38.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d279b0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8d2009dd40 0x7f8d200cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:38.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d279b0700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8d200ce5a0 0x7f8d200cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:38.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d279b0700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d200ce090 con 0x7f8d2009dd40 2024-03-18T18:50:38.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d279b0700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d2006db30 con 0x7f8d200ce5a0 2024-03-18T18:50:38.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d279b0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2006d520 con 0x7f8d2015d780 2024-03-18T18:50:38.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d24f4b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8d2009dd40 0x7f8d200cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:38.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d24f4b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8d2009dd40 0x7f8d200cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49742/0 (socket says 172.21.15.1:49742) 2024-03-18T18:50:38.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d24f4b700 1 -- 172.21.15.1:0/2766085768 learned_addr learned my addr 172.21.15.1:0/2766085768 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:38.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d25f4d700 1 --2- 172.21.15.1:0/2766085768 >> v2:172.21.15.165:3300/0 conn(0x7f8d200ce5a0 0x7f8d200cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:38.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d2574c700 1 --2- 172.21.15.1:0/2766085768 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8d2015d780 0x7f8d200cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:38.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d24f4b700 1 -- 172.21.15.1:0/2766085768 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8d2015d780 msgr2=0x7f8d200cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:38.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d24f4b700 1 --2- 172.21.15.1:0/2766085768 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8d2015d780 0x7f8d200cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d24f4b700 1 -- 172.21.15.1:0/2766085768 >> v2:172.21.15.165:3300/0 conn(0x7f8d200ce5a0 msgr2=0x7f8d200cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:38.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d24f4b700 1 --2- 172.21.15.1:0/2766085768 >> v2:172.21.15.165:3300/0 conn(0x7f8d200ce5a0 0x7f8d200cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d24f4b700 1 -- 172.21.15.1:0/2766085768 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2006f460 con 0x7f8d2009dd40 2024-03-18T18:50:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d24f4b700 1 --2- 172.21.15.1:0/2766085768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d2009dd40 0x7f8d200cad70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8d200672f0 tx=0x7f8d1c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac68d6297be7e289 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d17fff700 1 -- 172.21.15.1:0/2766085768 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c00b870 con 0x7f8d2009dd40 2024-03-18T18:50:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d17fff700 1 -- 172.21.15.1:0/2766085768 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1c00b9d0 con 0x7f8d2009dd40 2024-03-18T18:50:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.153+0000 7f8d17fff700 1 -- 172.21.15.1:0/2766085768 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c00bd60 con 0x7f8d2009dd40 2024-03-18T18:50:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.154+0000 7f8d279b0700 1 -- 172.21.15.1:0/2766085768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d2009dd40 msgr2=0x7f8d200cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.154+0000 7f8d279b0700 1 --2- 172.21.15.1:0/2766085768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d2009dd40 0x7f8d200cad70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8d200672f0 tx=0x7f8d1c009bd0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.154+0000 7f8d279b0700 1 -- 172.21.15.1:0/2766085768 shutdown_connections 2024-03-18T18:50:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.154+0000 7f8d279b0700 1 --2- 172.21.15.1:0/2766085768 >> v2:172.21.15.165:3300/0 conn(0x7f8d200ce5a0 0x7f8d200cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.154+0000 7f8d279b0700 1 --2- 172.21.15.1:0/2766085768 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8d2015d780 0x7f8d200cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.154+0000 7f8d279b0700 1 --2- 172.21.15.1:0/2766085768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d2009dd40 0x7f8d200cad70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.154+0000 7f8d279b0700 1 -- 172.21.15.1:0/2766085768 >> 172.21.15.1:0/2766085768 conn(0x7f8d200cca00 msgr2=0x7f8d200cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.154+0000 7f8d279b0700 1 -- 172.21.15.1:0/2766085768 shutdown_connections 2024-03-18T18:50:38.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.154+0000 7f8d279b0700 1 -- 172.21.15.1:0/2766085768 wait complete. 2024-03-18T18:50:38.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.155+0000 7f8d279b0700 1 Processor -- start 2024-03-18T18:50:38.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.155+0000 7f8d279b0700 1 -- start start 2024-03-18T18:50:38.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.156+0000 7f8d279b0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8d2009dd40 0x7f8d201542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:38.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.156+0000 7f8d279b0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d200ce5a0 0x7f8d201f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:38.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.156+0000 7f8d279b0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8d2015d780 0x7f8d200c9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:38.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.156+0000 7f8d279b0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d201547e0 con 0x7f8d200ce5a0 2024-03-18T18:50:38.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.156+0000 7f8d279b0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2009a850 con 0x7f8d2009dd40 2024-03-18T18:50:38.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.156+0000 7f8d279b0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2009ab30 con 0x7f8d2015d780 2024-03-18T18:50:38.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.156+0000 7f8d24f4b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d200ce5a0 0x7f8d201f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:38.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.156+0000 7f8d24f4b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d200ce5a0 0x7f8d201f2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49756/0 (socket says 172.21.15.1:49756) 2024-03-18T18:50:38.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.156+0000 7f8d24f4b700 1 -- 172.21.15.1:0/3563746268 learned_addr learned my addr 172.21.15.1:0/3563746268 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:38.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.156+0000 7f8d25f4d700 1 --2- 172.21.15.1:0/3563746268 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8d2015d780 0x7f8d200c9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:38.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.156+0000 7f8d2574c700 1 --2- 172.21.15.1:0/3563746268 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8d2009dd40 0x7f8d201542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:38.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.156+0000 7f8d24f4b700 1 -- 172.21.15.1:0/3563746268 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8d2015d780 msgr2=0x7f8d200c9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:38.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.156+0000 7f8d24f4b700 1 --2- 172.21.15.1:0/3563746268 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8d2015d780 0x7f8d200c9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.156+0000 7f8d24f4b700 1 -- 172.21.15.1:0/3563746268 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8d2009dd40 msgr2=0x7f8d201542a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:38.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.156+0000 7f8d24f4b700 1 --2- 172.21.15.1:0/3563746268 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8d2009dd40 0x7f8d201542a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.156+0000 7f8d24f4b700 1 -- 172.21.15.1:0/3563746268 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1c009550 con 0x7f8d200ce5a0 2024-03-18T18:50:38.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.156+0000 7f8d2574c700 1 --2- 172.21.15.1:0/3563746268 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8d2009dd40 0x7f8d201542a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:38.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.156+0000 7f8d24f4b700 1 --2- 172.21.15.1:0/3563746268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d200ce5a0 0x7f8d201f2c30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c000c00 tx=0x7f8d1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:38.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.157+0000 7f8d15ffb700 1 -- 172.21.15.1:0/3563746268 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c004520 con 0x7f8d200ce5a0 2024-03-18T18:50:38.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.157+0000 7f8d15ffb700 1 -- 172.21.15.1:0/3563746268 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1c01e070 con 0x7f8d200ce5a0 2024-03-18T18:50:38.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.157+0000 7f8d279b0700 1 -- 172.21.15.1:0/3563746268 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2009ae10 con 0x7f8d200ce5a0 2024-03-18T18:50:38.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.157+0000 7f8d15ffb700 1 -- 172.21.15.1:0/3563746268 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c0038c0 con 0x7f8d200ce5a0 2024-03-18T18:50:38.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.157+0000 7f8d279b0700 1 -- 172.21.15.1:0/3563746268 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2009b3d0 con 0x7f8d200ce5a0 2024-03-18T18:50:38.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.158+0000 7f8d279b0700 1 -- 172.21.15.1:0/3563746268 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2009b5c0 con 0x7f8d200ce5a0 2024-03-18T18:50:38.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.158+0000 7f8d15ffb700 1 -- 172.21.15.1:0/3563746268 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8d1c003a20 con 0x7f8d200ce5a0 2024-03-18T18:50:38.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.159+0000 7f8d15ffb700 1 --2- 172.21.15.1:0/3563746268 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8d0406ac90 0x7f8d0406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:38.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.159+0000 7f8d15ffb700 1 -- 172.21.15.1:0/3563746268 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8d1c020070 con 0x7f8d200ce5a0 2024-03-18T18:50:38.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.159+0000 7f8d2574c700 1 --2- 172.21.15.1:0/3563746268 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8d0406ac90 0x7f8d0406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:38.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.159+0000 7f8d2574c700 1 --2- 172.21.15.1:0/3563746268 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8d0406ac90 0x7f8d0406d150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009970 tx=0x7f8d10008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:38.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.163+0000 7f8d15ffb700 1 -- 172.21.15.1:0/3563746268 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8d1c055a50 con 0x7f8d200ce5a0 2024-03-18T18:50:38.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.371+0000 7f8d279b0700 1 -- 172.21.15.1:0/3563746268 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f8d200676a0 con 0x7f8d200ce5a0 2024-03-18T18:50:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.374+0000 7f8d15ffb700 1 -- 172.21.15.1:0/3563746268 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8d1c015020 con 0x7f8d200ce5a0 2024-03-18T18:50:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.376+0000 7f8d279b0700 1 -- 172.21.15.1:0/3563746268 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8d0406ac90 msgr2=0x7f8d0406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.376+0000 7f8d279b0700 1 --2- 172.21.15.1:0/3563746268 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8d0406ac90 0x7f8d0406d150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009970 tx=0x7f8d10008040 comp rx=0 tx=0).stop 2024-03-18T18:50:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.376+0000 7f8d279b0700 1 -- 172.21.15.1:0/3563746268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d200ce5a0 msgr2=0x7f8d201f2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.376+0000 7f8d279b0700 1 --2- 172.21.15.1:0/3563746268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d200ce5a0 0x7f8d201f2c30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c000c00 tx=0x7f8d1c004750 comp rx=0 tx=0).stop 2024-03-18T18:50:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.376+0000 7f8d279b0700 1 -- 172.21.15.1:0/3563746268 shutdown_connections 2024-03-18T18:50:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.376+0000 7f8d279b0700 1 --2- 172.21.15.1:0/3563746268 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8d0406ac90 0x7f8d0406d150 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.376+0000 7f8d279b0700 1 --2- 172.21.15.1:0/3563746268 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8d2015d780 0x7f8d200c9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.376+0000 7f8d279b0700 1 --2- 172.21.15.1:0/3563746268 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8d2009dd40 0x7f8d201542a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.376+0000 7f8d279b0700 1 --2- 172.21.15.1:0/3563746268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d200ce5a0 0x7f8d201f2c30 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.376+0000 7f8d279b0700 1 -- 172.21.15.1:0/3563746268 >> 172.21.15.1:0/3563746268 conn(0x7f8d200cca00 msgr2=0x7f8d200cb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.376+0000 7f8d279b0700 1 -- 172.21.15.1:0/3563746268 shutdown_connections 2024-03-18T18:50:38.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.376+0000 7f8d279b0700 1 -- 172.21.15.1:0/3563746268 wait complete. 2024-03-18T18:50:38.386 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T18:50:38.386 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T18:50:38.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T18:50:38.486 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T18:50:38.486 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:50:38.486 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T18:50:38.507 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:50:38.507 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T18:50:38.566 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.738+0000 7fce76d29700 1 Processor -- start 2024-03-18T18:50:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.738+0000 7fce76d29700 1 -- start start 2024-03-18T18:50:38.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.738+0000 7fce76d29700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce700ce060 0x7fce70071c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:38.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.738+0000 7fce76d29700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce700cfd20 con 0x7fce700cdc00 2024-03-18T18:50:38.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.738+0000 7fce76d29700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce700717c0 con 0x7fce70161360 2024-03-18T18:50:38.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.738+0000 7fce76d29700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce7015da00 con 0x7fce700ce060 2024-03-18T18:50:38.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce6ffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fce700cdc00 0x7fce70161740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37396/0 (socket says 172.21.15.1:37396) 2024-03-18T18:50:38.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce6ffff700 1 -- 172.21.15.1:0/3183071679 learned_addr learned my addr 172.21.15.1:0/3183071679 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:38.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce6f7fe700 1 -- 172.21.15.1:0/3183071679 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852091761 0 0) 0x7fce700cfd20 con 0x7fce700cdc00 2024-03-18T18:50:38.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce6f7fe700 1 -- 172.21.15.1:0/3183071679 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce600035b0 con 0x7fce700cdc00 2024-03-18T18:50:38.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce752c6700 1 --2- 172.21.15.1:0/3183071679 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce700ce060 0x7fce70071c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:38.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce6f7fe700 1 -- 172.21.15.1:0/3183071679 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 815013270 0 0) 0x7fce600035b0 con 0x7fce700cdc00 2024-03-18T18:50:38.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce6f7fe700 1 -- 172.21.15.1:0/3183071679 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce700ce060 msgr2=0x7fce70071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:38.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce6f7fe700 1 --2- 172.21.15.1:0/3183071679 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce700ce060 0x7fce70071c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce6f7fe700 1 -- 172.21.15.1:0/3183071679 >> v1:172.21.15.165:6789/0 conn(0x7fce70161360 legacy=0x7fce7006b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:38.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce6f7fe700 1 -- 172.21.15.1:0/3183071679 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce700cc8b0 con 0x7fce700cdc00 2024-03-18T18:50:38.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce6f7fe700 1 -- 172.21.15.1:0/3183071679 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fce5c001670 con 0x7fce700cdc00 2024-03-18T18:50:38.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce6f7fe700 1 -- 172.21.15.1:0/3183071679 >> v1:172.21.15.1:6789/0 conn(0x7fce700cdc00 legacy=0x7fce70161740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:38.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce6f7fe700 1 --2- 172.21.15.1:0/3183071679 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce60003a30 0x7fce60003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce6f7fe700 1 --2- 172.21.15.1:0/3183071679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce60004850 0x7fce60004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce6f7fe700 1 --2- 172.21.15.1:0/3183071679 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce700ce060 0x7fce600052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce6f7fe700 1 -- 172.21.15.1:0/3183071679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7015da00 con 0x7fce60004850 2024-03-18T18:50:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce6f7fe700 1 -- 172.21.15.1:0/3183071679 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6000bc00 con 0x7fce60003a30 2024-03-18T18:50:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce6f7fe700 1 -- 172.21.15.1:0/3183071679 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce6000bdc0 con 0x7fce700ce060 2024-03-18T18:50:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce6ffff700 1 --2- 172.21.15.1:0/3183071679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce60004850 0x7fce60004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.739+0000 7fce74ac5700 1 --2- 172.21.15.1:0/3183071679 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce60003a30 0x7fce60003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.740+0000 7fce752c6700 1 --2- 172.21.15.1:0/3183071679 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce700ce060 0x7fce600052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.740+0000 7fce6ffff700 1 -- 172.21.15.1:0/3183071679 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce700ce060 msgr2=0x7fce600052e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.740+0000 7fce6ffff700 1 --2- 172.21.15.1:0/3183071679 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce700ce060 0x7fce600052e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.740+0000 7fce6ffff700 1 -- 172.21.15.1:0/3183071679 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce60003a30 msgr2=0x7fce60003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.740+0000 7fce6ffff700 1 --2- 172.21.15.1:0/3183071679 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce60003a30 0x7fce60003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.740+0000 7fce6ffff700 1 -- 172.21.15.1:0/3183071679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce600035b0 con 0x7fce60004850 2024-03-18T18:50:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.740+0000 7fce752c6700 1 --2- 172.21.15.1:0/3183071679 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce700ce060 0x7fce600052e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:38.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.740+0000 7fce6ffff700 1 --2- 172.21.15.1:0/3183071679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce60004850 0x7fce60004cd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fce5c00c240 tx=0x7fce60005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d61aa81707bbc7c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:38.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.740+0000 7fce6f7fe700 1 -- 172.21.15.1:0/3183071679 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c007c10 con 0x7fce60004850 2024-03-18T18:50:38.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.740+0000 7fce6f7fe700 1 -- 172.21.15.1:0/3183071679 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5c00ce00 con 0x7fce60004850 2024-03-18T18:50:38.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.740+0000 7fce6f7fe700 1 -- 172.21.15.1:0/3183071679 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c005ba0 con 0x7fce60004850 2024-03-18T18:50:38.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.740+0000 7fce76d29700 1 -- 172.21.15.1:0/3183071679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce60004850 msgr2=0x7fce60004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:38.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.740+0000 7fce76d29700 1 --2- 172.21.15.1:0/3183071679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce60004850 0x7fce60004cd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fce5c00c240 tx=0x7fce60005880 comp rx=0 tx=0).stop 2024-03-18T18:50:38.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.740+0000 7fce752c6700 1 -- 172.21.15.1:0/3183071679 reap_dead start 2024-03-18T18:50:38.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.741+0000 7fce76d29700 1 -- 172.21.15.1:0/3183071679 shutdown_connections 2024-03-18T18:50:38.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.741+0000 7fce76d29700 1 -- 172.21.15.1:0/3183071679 >> 172.21.15.1:0/3183071679 conn(0x7fce700ccab0 msgr2=0x7fce700ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:38.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.741+0000 7fce76d29700 1 -- 172.21.15.1:0/3183071679 shutdown_connections 2024-03-18T18:50:38.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.741+0000 7fce76d29700 1 -- 172.21.15.1:0/3183071679 wait complete. 2024-03-18T18:50:38.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.742+0000 7fce76d29700 1 Processor -- start 2024-03-18T18:50:38.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.742+0000 7fce76d29700 1 -- start start 2024-03-18T18:50:38.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.743+0000 7fce76d29700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce60004850 0x7fce70152d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:38.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.743+0000 7fce76d29700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce700c9df0 0x7fce700c3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:38.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.743+0000 7fce76d29700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce700cf240 0x7fce700cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:38.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.743+0000 7fce76d29700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce700c6d50 con 0x7fce60004850 2024-03-18T18:50:38.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.743+0000 7fce76d29700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce700c49d0 con 0x7fce700cf240 2024-03-18T18:50:38.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.743+0000 7fce76d29700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce700cfc10 con 0x7fce700c9df0 2024-03-18T18:50:38.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.743+0000 7fce74ac5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce60004850 0x7fce70152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:38.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.743+0000 7fce74ac5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce60004850 0x7fce70152d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49778/0 (socket says 172.21.15.1:49778) 2024-03-18T18:50:38.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.743+0000 7fce74ac5700 1 -- 172.21.15.1:0/625869351 learned_addr learned my addr 172.21.15.1:0/625869351 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:38.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.743+0000 7fce74ac5700 1 -- 172.21.15.1:0/625869351 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce700c9df0 msgr2=0x7fce700c3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:38.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.743+0000 7fce74ac5700 1 --2- 172.21.15.1:0/625869351 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce700c9df0 0x7fce700c3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.743+0000 7fce74ac5700 1 -- 172.21.15.1:0/625869351 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce700cf240 msgr2=0x7fce700cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:38.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.743+0000 7fce74ac5700 1 --2- 172.21.15.1:0/625869351 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce700cf240 0x7fce700cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.743+0000 7fce74ac5700 1 -- 172.21.15.1:0/625869351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce70153280 con 0x7fce60004850 2024-03-18T18:50:38.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.743+0000 7fce74ac5700 1 --2- 172.21.15.1:0/625869351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce60004850 0x7fce70152d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fce6400f820 tx=0x7fce6400fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:38.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.743+0000 7fce6d7fa700 1 -- 172.21.15.1:0/625869351 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce64005320 con 0x7fce60004850 2024-03-18T18:50:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.743+0000 7fce6d7fa700 1 -- 172.21.15.1:0/625869351 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce64005480 con 0x7fce60004850 2024-03-18T18:50:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.743+0000 7fce76d29700 1 -- 172.21.15.1:0/625869351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce700c6790 con 0x7fce60004850 2024-03-18T18:50:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.743+0000 7fce6d7fa700 1 -- 172.21.15.1:0/625869351 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce64006aa0 con 0x7fce60004850 2024-03-18T18:50:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.744+0000 7fce76d29700 1 -- 172.21.15.1:0/625869351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce700c72a0 con 0x7fce60004850 2024-03-18T18:50:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.744+0000 7fce76d29700 1 -- 172.21.15.1:0/625869351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce700ca980 con 0x7fce60004850 2024-03-18T18:50:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.745+0000 7fce6d7fa700 1 -- 172.21.15.1:0/625869351 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fce64006c00 con 0x7fce60004850 2024-03-18T18:50:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.745+0000 7fce6d7fa700 1 --2- 172.21.15.1:0/625869351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce5406ac90 0x7fce5406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.745+0000 7fce6d7fa700 1 -- 172.21.15.1:0/625869351 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fce64088f80 con 0x7fce60004850 2024-03-18T18:50:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.745+0000 7fce6ffff700 1 --2- 172.21.15.1:0/625869351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce5406ac90 0x7fce5406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.746+0000 7fce6ffff700 1 --2- 172.21.15.1:0/625869351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce5406ac90 0x7fce5406d150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fce7015fec0 tx=0x7fce5c00a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:38.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.750+0000 7fce6d7fa700 1 -- 172.21.15.1:0/625869351 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fce640bc580 con 0x7fce60004850 2024-03-18T18:50:38.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.955+0000 7fce76d29700 1 -- 172.21.15.1:0/625869351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce700676a0 con 0x7fce60004850 2024-03-18T18:50:38.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.956+0000 7fce6d7fa700 1 -- 172.21.15.1:0/625869351 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18217 (secure 0 0 0) 0x7fce64053db0 con 0x7fce60004850 2024-03-18T18:50:38.953 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:38.954 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":27,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:50:34.993482+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.958+0000 7fce76d29700 1 -- 172.21.15.1:0/625869351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce5406ac90 msgr2=0x7fce5406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.958+0000 7fce76d29700 1 --2- 172.21.15.1:0/625869351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce5406ac90 0x7fce5406d150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fce7015fec0 tx=0x7fce5c00a000 comp rx=0 tx=0).stop 2024-03-18T18:50:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.958+0000 7fce76d29700 1 -- 172.21.15.1:0/625869351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce60004850 msgr2=0x7fce70152d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.958+0000 7fce76d29700 1 --2- 172.21.15.1:0/625869351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce60004850 0x7fce70152d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fce6400f820 tx=0x7fce6400fbe0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.958+0000 7fce76d29700 1 -- 172.21.15.1:0/625869351 shutdown_connections 2024-03-18T18:50:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.958+0000 7fce76d29700 1 --2- 172.21.15.1:0/625869351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce5406ac90 0x7fce5406d150 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.958+0000 7fce76d29700 1 --2- 172.21.15.1:0/625869351 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce700cf240 0x7fce700cf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.958+0000 7fce76d29700 1 --2- 172.21.15.1:0/625869351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce60004850 0x7fce70152d40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.958+0000 7fce76d29700 1 --2- 172.21.15.1:0/625869351 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce700c9df0 0x7fce700c3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:38.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.958+0000 7fce76d29700 1 -- 172.21.15.1:0/625869351 >> 172.21.15.1:0/625869351 conn(0x7fce700ccab0 msgr2=0x7fce700ce550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:38.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.958+0000 7fce76d29700 1 -- 172.21.15.1:0/625869351 shutdown_connections 2024-03-18T18:50:38.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:38.958+0000 7fce76d29700 1 -- 172.21.15.1:0/625869351 wait complete. 2024-03-18T18:50:38.968 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T18:49:42.800801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T18:49:43.776684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T18:49:46.854608+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T18:50:38.969 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T18:50:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.099+0000 7f1d246a1700 1 Processor -- start 2024-03-18T18:50:39.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.100+0000 7f1d246a1700 1 -- start start 2024-03-18T18:50:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.100+0000 7f1d246a1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1d1c071cc0 0x7f1d1c0ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:39.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.100+0000 7f1d246a1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1d1c06d560 0x7f1d1c06fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.101+0000 7f1d246a1700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d1c070000 con 0x7f1d1c15a240 2024-03-18T18:50:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.101+0000 7f1d246a1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d1c0adff0 con 0x7f1d1c071cc0 2024-03-18T18:50:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.101+0000 7f1d246a1700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d1c0cac80 con 0x7f1d1c06d560 2024-03-18T18:50:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.101+0000 7f1d21c3c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1d1c071cc0 0x7f1d1c0ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.101+0000 7f1d21c3c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1d1c071cc0 0x7f1d1c0ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49790/0 (socket says 172.21.15.1:49790) 2024-03-18T18:50:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.101+0000 7f1d21c3c700 1 -- 172.21.15.1:0/4204247614 learned_addr learned my addr 172.21.15.1:0/4204247614 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.101+0000 7f1d22c3e700 1 --2- 172.21.15.1:0/4204247614 >> v2:172.21.15.165:3300/0 conn(0x7f1d1c06d560 0x7f1d1c06fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.101+0000 7f1d21c3c700 1 -- 172.21.15.1:0/4204247614 >> v2:172.21.15.165:3300/0 conn(0x7f1d1c06d560 msgr2=0x7f1d1c06fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.101+0000 7f1d21c3c700 1 --2- 172.21.15.1:0/4204247614 >> v2:172.21.15.165:3300/0 conn(0x7f1d1c06d560 0x7f1d1c06fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.101+0000 7f1d21c3c700 1 -- 172.21.15.1:0/4204247614 >> v1:172.21.15.165:6789/0 conn(0x7f1d1c15a240 legacy=0x7f1d1c0715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.101+0000 7f1d21c3c700 1 -- 172.21.15.1:0/4204247614 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1c0caf60 con 0x7f1d1c071cc0 2024-03-18T18:50:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.101+0000 7f1d21c3c700 1 --2- 172.21.15.1:0/4204247614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d1c071cc0 0x7f1d1c0ca580 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0adc30 tx=0x7f1d1000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c955dcd7a2d77bcd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:39.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.101+0000 7f1d2143b700 1 -- 172.21.15.1:0/4204247614 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1000dcf0 con 0x7f1d1c071cc0 2024-03-18T18:50:39.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.101+0000 7f1d2143b700 1 -- 172.21.15.1:0/4204247614 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d1000de50 con 0x7f1d1c071cc0 2024-03-18T18:50:39.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.101+0000 7f1d2143b700 1 -- 172.21.15.1:0/4204247614 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d10004250 con 0x7f1d1c071cc0 2024-03-18T18:50:39.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.102+0000 7f1d246a1700 1 -- 172.21.15.1:0/4204247614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d1c071cc0 msgr2=0x7f1d1c0ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:39.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.102+0000 7f1d246a1700 1 --2- 172.21.15.1:0/4204247614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d1c071cc0 0x7f1d1c0ca580 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0adc30 tx=0x7f1d1000be70 comp rx=0 tx=0).stop 2024-03-18T18:50:39.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.102+0000 7f1d246a1700 1 -- 172.21.15.1:0/4204247614 shutdown_connections 2024-03-18T18:50:39.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.102+0000 7f1d246a1700 1 --2- 172.21.15.1:0/4204247614 >> v2:172.21.15.165:3300/0 conn(0x7f1d1c06d560 0x7f1d1c06fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.102+0000 7f1d246a1700 1 --2- 172.21.15.1:0/4204247614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d1c071cc0 0x7f1d1c0ca580 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.102+0000 7f1d246a1700 1 -- 172.21.15.1:0/4204247614 >> 172.21.15.1:0/4204247614 conn(0x7f1d1c0cff30 msgr2=0x7f1d1c0d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:39.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.102+0000 7f1d246a1700 1 -- 172.21.15.1:0/4204247614 shutdown_connections 2024-03-18T18:50:39.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.102+0000 7f1d246a1700 1 -- 172.21.15.1:0/4204247614 wait complete. 2024-03-18T18:50:39.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.103+0000 7f1d246a1700 1 Processor -- start 2024-03-18T18:50:39.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.104+0000 7f1d246a1700 1 -- start start 2024-03-18T18:50:39.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.104+0000 7f1d246a1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d1c06d560 0x7f1d1c0b0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:39.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.104+0000 7f1d246a1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d1c071cc0 0x7f1d1c0b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:39.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.104+0000 7f1d246a1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d1c15a240 0x7f1d1c0c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:39.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.104+0000 7f1d246a1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c0c4940 con 0x7f1d1c06d560 2024-03-18T18:50:39.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.104+0000 7f1d246a1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c0c4c20 con 0x7f1d1c071cc0 2024-03-18T18:50:39.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.104+0000 7f1d246a1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c0c4f00 con 0x7f1d1c15a240 2024-03-18T18:50:39.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.105+0000 7f1d2243d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d1c06d560 0x7f1d1c0b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:39.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.105+0000 7f1d2243d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d1c06d560 0x7f1d1c0b0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49802/0 (socket says 172.21.15.1:49802) 2024-03-18T18:50:39.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.105+0000 7f1d2243d700 1 -- 172.21.15.1:0/1353107667 learned_addr learned my addr 172.21.15.1:0/1353107667 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:39.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.105+0000 7f1d22c3e700 1 --2- 172.21.15.1:0/1353107667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d1c15a240 0x7f1d1c0c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:39.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.105+0000 7f1d21c3c700 1 --2- 172.21.15.1:0/1353107667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d1c071cc0 0x7f1d1c0b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:39.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.105+0000 7f1d2243d700 1 -- 172.21.15.1:0/1353107667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d1c15a240 msgr2=0x7f1d1c0c3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:39.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.105+0000 7f1d2243d700 1 --2- 172.21.15.1:0/1353107667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d1c15a240 0x7f1d1c0c3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.105+0000 7f1d2243d700 1 -- 172.21.15.1:0/1353107667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d1c071cc0 msgr2=0x7f1d1c0b08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:39.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.105+0000 7f1d2243d700 1 --2- 172.21.15.1:0/1353107667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d1c071cc0 0x7f1d1c0b08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.105+0000 7f1d2243d700 1 -- 172.21.15.1:0/1353107667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1000b7f0 con 0x7f1d1c06d560 2024-03-18T18:50:39.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.105+0000 7f1d21c3c700 1 --2- 172.21.15.1:0/1353107667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d1c071cc0 0x7f1d1c0b08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:39.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.105+0000 7f1d22c3e700 1 --2- 172.21.15.1:0/1353107667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d1c15a240 0x7f1d1c0c3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.105+0000 7f1d2243d700 1 --2- 172.21.15.1:0/1353107667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d1c06d560 0x7f1d1c0b0380 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1d1800b470 tx=0x7f1d1800b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.105+0000 7f1d0effd700 1 -- 172.21.15.1:0/1353107667 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d18011830 con 0x7f1d1c06d560 2024-03-18T18:50:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.105+0000 7f1d0effd700 1 -- 172.21.15.1:0/1353107667 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d18011990 con 0x7f1d1c06d560 2024-03-18T18:50:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.105+0000 7f1d0effd700 1 -- 172.21.15.1:0/1353107667 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d18011d40 con 0x7f1d1c06d560 2024-03-18T18:50:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.105+0000 7f1d246a1700 1 -- 172.21.15.1:0/1353107667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d1c0b4170 con 0x7f1d1c06d560 2024-03-18T18:50:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.105+0000 7f1d246a1700 1 -- 172.21.15.1:0/1353107667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1c1526f0 con 0x7f1d1c06d560 2024-03-18T18:50:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.106+0000 7f1d246a1700 1 -- 172.21.15.1:0/1353107667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d1c152970 con 0x7f1d1c06d560 2024-03-18T18:50:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.107+0000 7f1d0effd700 1 -- 172.21.15.1:0/1353107667 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1d1801a400 con 0x7f1d1c06d560 2024-03-18T18:50:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.107+0000 7f1d0effd700 1 --2- 172.21.15.1:0/1353107667 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1d0006ac90 0x7f1d0006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.107+0000 7f1d0effd700 1 -- 172.21.15.1:0/1353107667 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f1d180896a0 con 0x7f1d1c06d560 2024-03-18T18:50:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.107+0000 7f1d21c3c700 1 --2- 172.21.15.1:0/1353107667 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1d0006ac90 0x7f1d0006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:39.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.108+0000 7f1d21c3c700 1 --2- 172.21.15.1:0/1353107667 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1d0006ac90 0x7f1d0006d150 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c06dd00 tx=0x7f1d10012000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:39.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.112+0000 7f1d0effd700 1 -- 172.21.15.1:0/1353107667 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1d18054550 con 0x7f1d1c06d560 2024-03-18T18:50:39.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.314+0000 7f1d246a1700 1 -- 172.21.15.1:0/1353107667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1d1c0676a0 con 0x7f1d1c06d560 2024-03-18T18:50:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.314+0000 7f1d0effd700 1 -- 172.21.15.1:0/1353107667 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1d18057b70 con 0x7f1d1c06d560 2024-03-18T18:50:39.312 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T18:50:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.316+0000 7f1d246a1700 1 -- 172.21.15.1:0/1353107667 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1d0006ac90 msgr2=0x7f1d0006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.316+0000 7f1d246a1700 1 --2- 172.21.15.1:0/1353107667 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1d0006ac90 0x7f1d0006d150 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c06dd00 tx=0x7f1d10012000 comp rx=0 tx=0).stop 2024-03-18T18:50:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.316+0000 7f1d246a1700 1 -- 172.21.15.1:0/1353107667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d1c06d560 msgr2=0x7f1d1c0b0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.316+0000 7f1d246a1700 1 --2- 172.21.15.1:0/1353107667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d1c06d560 0x7f1d1c0b0380 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1d1800b470 tx=0x7f1d1800b830 comp rx=0 tx=0).stop 2024-03-18T18:50:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.316+0000 7f1d246a1700 1 -- 172.21.15.1:0/1353107667 shutdown_connections 2024-03-18T18:50:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.316+0000 7f1d246a1700 1 --2- 172.21.15.1:0/1353107667 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1d0006ac90 0x7f1d0006d150 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.316+0000 7f1d246a1700 1 --2- 172.21.15.1:0/1353107667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d1c15a240 0x7f1d1c0c3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.316+0000 7f1d246a1700 1 --2- 172.21.15.1:0/1353107667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d1c06d560 0x7f1d1c0b0380 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.316+0000 7f1d246a1700 1 --2- 172.21.15.1:0/1353107667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d1c071cc0 0x7f1d1c0b08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.316+0000 7f1d246a1700 1 -- 172.21.15.1:0/1353107667 >> 172.21.15.1:0/1353107667 conn(0x7f1d1c0cff30 msgr2=0x7f1d1c0b11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.316+0000 7f1d246a1700 1 -- 172.21.15.1:0/1353107667 shutdown_connections 2024-03-18T18:50:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.316+0000 7f1d246a1700 1 -- 172.21.15.1:0/1353107667 wait complete. 2024-03-18T18:50:39.326 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T18:50:39.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.457+0000 7f4abdd44700 1 Processor -- start 2024-03-18T18:50:39.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.457+0000 7f4abdd44700 1 -- start start 2024-03-18T18:50:39.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.458+0000 7f4abdd44700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4ab806a750 0x7f4ab806ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:39.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.458+0000 7f4abdd44700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4ab815ec80 0x7f4ab815f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:39.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.458+0000 7f4abdd44700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab815c340 con 0x7f4ab80ace80 2024-03-18T18:50:39.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.458+0000 7f4abdd44700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ab8070090 con 0x7f4ab806a750 2024-03-18T18:50:39.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.458+0000 7f4abdd44700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ab815b7a0 con 0x7f4ab815ec80 2024-03-18T18:50:39.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.458+0000 7f4ab77fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4ab806a750 0x7f4ab806ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:39.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.458+0000 7f4ab77fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4ab806a750 0x7f4ab806ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49814/0 (socket says 172.21.15.1:49814) 2024-03-18T18:50:39.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.458+0000 7f4ab77fe700 1 -- 172.21.15.1:0/3447017362 learned_addr learned my addr 172.21.15.1:0/3447017362 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:39.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.458+0000 7f4ab7fff700 1 --2- 172.21.15.1:0/3447017362 >> v2:172.21.15.165:3300/0 conn(0x7f4ab815ec80 0x7f4ab815f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:39.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.458+0000 7f4ab77fe700 1 -- 172.21.15.1:0/3447017362 >> v2:172.21.15.165:3300/0 conn(0x7f4ab815ec80 msgr2=0x7f4ab815f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:39.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.458+0000 7f4ab77fe700 1 --2- 172.21.15.1:0/3447017362 >> v2:172.21.15.165:3300/0 conn(0x7f4ab815ec80 0x7f4ab815f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.458+0000 7f4ab77fe700 1 -- 172.21.15.1:0/3447017362 >> v1:172.21.15.165:6789/0 conn(0x7f4ab80ace80 legacy=0x7f4ab80ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:39.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.458+0000 7f4ab77fe700 1 -- 172.21.15.1:0/3447017362 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab806b070 con 0x7f4ab806a750 2024-03-18T18:50:39.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.458+0000 7f4ab7fff700 1 --2- 172.21.15.1:0/3447017362 >> v2:172.21.15.165:3300/0 conn(0x7f4ab815ec80 0x7f4ab815f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:39.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.458+0000 7f4ab77fe700 1 --2- 172.21.15.1:0/3447017362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab806a750 0x7f4ab806ab30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4ab815c7d0 tx=0x7f4aa8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3b70b829dd0e759 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:39.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.458+0000 7f4ab67fc700 1 -- 172.21.15.1:0/3447017362 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa8005e30 con 0x7f4ab806a750 2024-03-18T18:50:39.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.458+0000 7f4ab67fc700 1 -- 172.21.15.1:0/3447017362 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa8004030 con 0x7f4ab806a750 2024-03-18T18:50:39.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.458+0000 7f4ab67fc700 1 -- 172.21.15.1:0/3447017362 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa80043e0 con 0x7f4ab806a750 2024-03-18T18:50:39.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.459+0000 7f4abdd44700 1 -- 172.21.15.1:0/3447017362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab806a750 msgr2=0x7f4ab806ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:39.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.459+0000 7f4abdd44700 1 --2- 172.21.15.1:0/3447017362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab806a750 0x7f4ab806ab30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4ab815c7d0 tx=0x7f4aa8009bd0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.459+0000 7f4abdd44700 1 -- 172.21.15.1:0/3447017362 shutdown_connections 2024-03-18T18:50:39.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.459+0000 7f4abdd44700 1 --2- 172.21.15.1:0/3447017362 >> v2:172.21.15.165:3300/0 conn(0x7f4ab815ec80 0x7f4ab815f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.459+0000 7f4abdd44700 1 --2- 172.21.15.1:0/3447017362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab806a750 0x7f4ab806ab30 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.459+0000 7f4abdd44700 1 -- 172.21.15.1:0/3447017362 >> 172.21.15.1:0/3447017362 conn(0x7f4ab80cc9b0 msgr2=0x7f4ab80ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:39.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.459+0000 7f4abdd44700 1 -- 172.21.15.1:0/3447017362 shutdown_connections 2024-03-18T18:50:39.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.459+0000 7f4abdd44700 1 -- 172.21.15.1:0/3447017362 wait complete. 2024-03-18T18:50:39.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.460+0000 7f4abdd44700 1 Processor -- start 2024-03-18T18:50:39.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.460+0000 7f4abdd44700 1 -- start start 2024-03-18T18:50:39.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.461+0000 7f4abdd44700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab806a750 0x7f4ab81583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:39.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.461+0000 7f4abdd44700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ab80ace80 0x7f4ab815d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:39.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.461+0000 7f4abdd44700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ab815ec80 0x7f4ab815d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:39.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.461+0000 7f4abdd44700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab80b44c0 con 0x7f4ab806a750 2024-03-18T18:50:39.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.461+0000 7f4abdd44700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab80b3bb0 con 0x7f4ab815ec80 2024-03-18T18:50:39.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.461+0000 7f4abdd44700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab80c7ef0 con 0x7f4ab80ace80 2024-03-18T18:50:39.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.461+0000 7f4ab77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab806a750 0x7f4ab81583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:39.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.461+0000 7f4ab77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab806a750 0x7f4ab81583e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49816/0 (socket says 172.21.15.1:49816) 2024-03-18T18:50:39.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.461+0000 7f4ab77fe700 1 -- 172.21.15.1:0/341594902 learned_addr learned my addr 172.21.15.1:0/341594902 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:39.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.461+0000 7f4ab7fff700 1 --2- 172.21.15.1:0/341594902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ab815ec80 0x7f4ab815d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:39.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.461+0000 7f4ab6ffd700 1 --2- 172.21.15.1:0/341594902 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ab80ace80 0x7f4ab815d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:39.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.462+0000 7f4ab77fe700 1 -- 172.21.15.1:0/341594902 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ab80ace80 msgr2=0x7f4ab815d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:39.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.462+0000 7f4ab77fe700 1 --2- 172.21.15.1:0/341594902 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ab80ace80 0x7f4ab815d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.462+0000 7f4ab77fe700 1 -- 172.21.15.1:0/341594902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ab815ec80 msgr2=0x7f4ab815d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:39.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.462+0000 7f4ab77fe700 1 --2- 172.21.15.1:0/341594902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ab815ec80 0x7f4ab815d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.462+0000 7f4ab77fe700 1 -- 172.21.15.1:0/341594902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa8009500 con 0x7f4ab806a750 2024-03-18T18:50:39.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.462+0000 7f4ab77fe700 1 --2- 172.21.15.1:0/341594902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab806a750 0x7f4ab81583e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8009c10 tx=0x7f4aa8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:39.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.462+0000 7f4a9ffff700 1 -- 172.21.15.1:0/341594902 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa801d430 con 0x7f4ab806a750 2024-03-18T18:50:39.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.462+0000 7f4a9ffff700 1 -- 172.21.15.1:0/341594902 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa801d590 con 0x7f4ab806a750 2024-03-18T18:50:39.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.462+0000 7f4a9ffff700 1 -- 172.21.15.1:0/341594902 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa801d940 con 0x7f4ab806a750 2024-03-18T18:50:39.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.462+0000 7f4abdd44700 1 -- 172.21.15.1:0/341594902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab80b3e00 con 0x7f4ab806a750 2024-03-18T18:50:39.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.462+0000 7f4abdd44700 1 -- 172.21.15.1:0/341594902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab8158920 con 0x7f4ab806a750 2024-03-18T18:50:39.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.463+0000 7f4abdd44700 1 -- 172.21.15.1:0/341594902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab80676a0 con 0x7f4ab806a750 2024-03-18T18:50:39.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.463+0000 7f4a9ffff700 1 -- 172.21.15.1:0/341594902 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4aa801daa0 con 0x7f4ab806a750 2024-03-18T18:50:39.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.464+0000 7f4a9ffff700 1 --2- 172.21.15.1:0/341594902 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4aa406ac90 0x7f4aa406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:39.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.464+0000 7f4a9ffff700 1 -- 172.21.15.1:0/341594902 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f4aa808aac0 con 0x7f4ab806a750 2024-03-18T18:50:39.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.464+0000 7f4ab6ffd700 1 --2- 172.21.15.1:0/341594902 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4aa406ac90 0x7f4aa406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:39.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.464+0000 7f4ab6ffd700 1 --2- 172.21.15.1:0/341594902 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4aa406ac90 0x7f4aa406d150 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8071950 tx=0x7f4aac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:39.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.468+0000 7f4a9ffff700 1 -- 172.21.15.1:0/341594902 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4aa8055a90 con 0x7f4ab806a750 2024-03-18T18:50:39.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.669+0000 7f4abdd44700 1 -- 172.21.15.1:0/341594902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4ab80cd640 con 0x7f4ab806a750 2024-03-18T18:50:39.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.669+0000 7f4a9ffff700 1 -- 172.21.15.1:0/341594902 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4aa80210a0 con 0x7f4ab806a750 2024-03-18T18:50:39.667 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.671+0000 7f4abdd44700 1 -- 172.21.15.1:0/341594902 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4aa406ac90 msgr2=0x7f4aa406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.671+0000 7f4abdd44700 1 --2- 172.21.15.1:0/341594902 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4aa406ac90 0x7f4aa406d150 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8071950 tx=0x7f4aac009040 comp rx=0 tx=0).stop 2024-03-18T18:50:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.671+0000 7f4abdd44700 1 -- 172.21.15.1:0/341594902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab806a750 msgr2=0x7f4ab81583e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:39.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.671+0000 7f4abdd44700 1 --2- 172.21.15.1:0/341594902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab806a750 0x7f4ab81583e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8009c10 tx=0x7f4aa8004750 comp rx=0 tx=0).stop 2024-03-18T18:50:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.671+0000 7f4abdd44700 1 -- 172.21.15.1:0/341594902 shutdown_connections 2024-03-18T18:50:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.671+0000 7f4abdd44700 1 --2- 172.21.15.1:0/341594902 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4aa406ac90 0x7f4aa406d150 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.671+0000 7f4abdd44700 1 --2- 172.21.15.1:0/341594902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ab815ec80 0x7f4ab815d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.671+0000 7f4abdd44700 1 --2- 172.21.15.1:0/341594902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab806a750 0x7f4ab81583e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.671+0000 7f4abdd44700 1 --2- 172.21.15.1:0/341594902 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ab80ace80 0x7f4ab815d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.671+0000 7f4abdd44700 1 -- 172.21.15.1:0/341594902 >> 172.21.15.1:0/341594902 conn(0x7f4ab80cc9b0 msgr2=0x7f4ab815c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.672+0000 7f4abdd44700 1 -- 172.21.15.1:0/341594902 shutdown_connections 2024-03-18T18:50:39.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.672+0000 7f4abdd44700 1 -- 172.21.15.1:0/341594902 wait complete. 2024-03-18T18:50:39.681 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T18:50:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.812+0000 7f34457d6700 1 Processor -- start 2024-03-18T18:50:39.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.812+0000 7f34457d6700 1 -- start start 2024-03-18T18:50:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.812+0000 7f34457d6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f34400cfb50 0x7f34400d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.812+0000 7f34457d6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34400cff30 0x7f34400b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.812+0000 7f34457d6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f34400b3d40 0x7f34400b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.812+0000 7f34457d6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34400b3370 con 0x7f34400cfb50 2024-03-18T18:50:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.812+0000 7f34457d6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34400b2760 con 0x7f34400b3d40 2024-03-18T18:50:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.812+0000 7f34457d6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34400b2a40 con 0x7f34400cff30 2024-03-18T18:50:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.813+0000 7f343effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f34400cfb50 0x7f34400d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.813+0000 7f343effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f34400cfb50 0x7f34400d3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49830/0 (socket says 172.21.15.1:49830) 2024-03-18T18:50:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.813+0000 7f343effd700 1 -- 172.21.15.1:0/463366994 learned_addr learned my addr 172.21.15.1:0/463366994 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:39.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.813+0000 7f343f7fe700 1 --2- 172.21.15.1:0/463366994 >> v2:172.21.15.165:3300/0 conn(0x7f34400b3d40 0x7f34400b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.813+0000 7f343e7fc700 1 --2- 172.21.15.1:0/463366994 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34400cff30 0x7f34400b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.813+0000 7f343effd700 1 -- 172.21.15.1:0/463366994 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34400cff30 msgr2=0x7f34400b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.813+0000 7f343effd700 1 --2- 172.21.15.1:0/463366994 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34400cff30 0x7f34400b3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.813+0000 7f343effd700 1 -- 172.21.15.1:0/463366994 >> v2:172.21.15.165:3300/0 conn(0x7f34400b3d40 msgr2=0x7f34400b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.813+0000 7f343effd700 1 --2- 172.21.15.1:0/463366994 >> v2:172.21.15.165:3300/0 conn(0x7f34400b3d40 0x7f34400b41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.813+0000 7f343effd700 1 -- 172.21.15.1:0/463366994 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34400b2d20 con 0x7f34400cfb50 2024-03-18T18:50:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.813+0000 7f343effd700 1 --2- 172.21.15.1:0/463366994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34400cfb50 0x7f34400d3c60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f34400b91c0 tx=0x7f3434009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ae10b18cdd0544 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.813+0000 7f343dffb700 1 -- 172.21.15.1:0/463366994 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3434005e30 con 0x7f34400cfb50 2024-03-18T18:50:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.813+0000 7f343dffb700 1 -- 172.21.15.1:0/463366994 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3434004030 con 0x7f34400cfb50 2024-03-18T18:50:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.813+0000 7f343dffb700 1 -- 172.21.15.1:0/463366994 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34340043e0 con 0x7f34400cfb50 2024-03-18T18:50:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.814+0000 7f34457d6700 1 -- 172.21.15.1:0/463366994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34400cfb50 msgr2=0x7f34400d3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.814+0000 7f34457d6700 1 --2- 172.21.15.1:0/463366994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34400cfb50 0x7f34400d3c60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f34400b91c0 tx=0x7f3434009bd0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.814+0000 7f34457d6700 1 -- 172.21.15.1:0/463366994 shutdown_connections 2024-03-18T18:50:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.814+0000 7f34457d6700 1 --2- 172.21.15.1:0/463366994 >> v2:172.21.15.165:3300/0 conn(0x7f34400b3d40 0x7f34400b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.814+0000 7f34457d6700 1 --2- 172.21.15.1:0/463366994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34400cfb50 0x7f34400d3c60 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.814+0000 7f34457d6700 1 --2- 172.21.15.1:0/463366994 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34400cff30 0x7f34400b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.814+0000 7f34457d6700 1 -- 172.21.15.1:0/463366994 >> 172.21.15.1:0/463366994 conn(0x7f34400d3140 msgr2=0x7f34400d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.814+0000 7f34457d6700 1 -- 172.21.15.1:0/463366994 shutdown_connections 2024-03-18T18:50:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.814+0000 7f34457d6700 1 -- 172.21.15.1:0/463366994 wait complete. 2024-03-18T18:50:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.815+0000 7f34457d6700 1 Processor -- start 2024-03-18T18:50:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.815+0000 7f34457d6700 1 -- start start 2024-03-18T18:50:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.816+0000 7f34457d6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34400b3d40 0x7f34400ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.816+0000 7f34457d6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34400cfb50 0x7f34400b8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.816+0000 7f34457d6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34400cff30 0x7f3440147ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.816+0000 7f34457d6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34400cd150 con 0x7f34400cfb50 2024-03-18T18:50:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.816+0000 7f34457d6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34400b78c0 con 0x7f34400cff30 2024-03-18T18:50:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.816+0000 7f34457d6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3440159800 con 0x7f34400b3d40 2024-03-18T18:50:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.816+0000 7f343e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34400cfb50 0x7f34400b8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.816+0000 7f343e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34400cfb50 0x7f34400b8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49842/0 (socket says 172.21.15.1:49842) 2024-03-18T18:50:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.816+0000 7f343e7fc700 1 -- 172.21.15.1:0/86562289 learned_addr learned my addr 172.21.15.1:0/86562289 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.816+0000 7f343effd700 1 --2- 172.21.15.1:0/86562289 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34400b3d40 0x7f34400ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.816+0000 7f343f7fe700 1 --2- 172.21.15.1:0/86562289 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34400cff30 0x7f3440147ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.816+0000 7f343e7fc700 1 -- 172.21.15.1:0/86562289 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34400b3d40 msgr2=0x7f34400ccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.816+0000 7f343e7fc700 1 --2- 172.21.15.1:0/86562289 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34400b3d40 0x7f34400ccc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.816+0000 7f343e7fc700 1 -- 172.21.15.1:0/86562289 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34400cff30 msgr2=0x7f3440147ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.816+0000 7f343e7fc700 1 --2- 172.21.15.1:0/86562289 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34400cff30 0x7f3440147ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.816+0000 7f343e7fc700 1 -- 172.21.15.1:0/86562289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3434009500 con 0x7f34400cfb50 2024-03-18T18:50:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.817+0000 7f343e7fc700 1 --2- 172.21.15.1:0/86562289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34400cfb50 0x7f34400b8080 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f343000d800 tx=0x7f343000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:39.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.817+0000 7f34277fe700 1 -- 172.21.15.1:0/86562289 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34300125f0 con 0x7f34400cfb50 2024-03-18T18:50:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.817+0000 7f34277fe700 1 -- 172.21.15.1:0/86562289 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3430012750 con 0x7f34400cfb50 2024-03-18T18:50:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.817+0000 7f34457d6700 1 -- 172.21.15.1:0/86562289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3440159240 con 0x7f34400cfb50 2024-03-18T18:50:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.817+0000 7f34277fe700 1 -- 172.21.15.1:0/86562289 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3430012b00 con 0x7f34400cfb50 2024-03-18T18:50:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.817+0000 7f34457d6700 1 -- 172.21.15.1:0/86562289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34400b7b60 con 0x7f34400cfb50 2024-03-18T18:50:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.818+0000 7f34457d6700 1 -- 172.21.15.1:0/86562289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34400b7720 con 0x7f34400cfb50 2024-03-18T18:50:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.818+0000 7f34277fe700 1 -- 172.21.15.1:0/86562289 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3430012c60 con 0x7f34400cfb50 2024-03-18T18:50:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.819+0000 7f34277fe700 1 --2- 172.21.15.1:0/86562289 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f342c06ac90 0x7f342c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.819+0000 7f34277fe700 1 -- 172.21.15.1:0/86562289 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3430089710 con 0x7f34400cfb50 2024-03-18T18:50:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.819+0000 7f343effd700 1 --2- 172.21.15.1:0/86562289 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f342c06ac90 0x7f342c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:39.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.819+0000 7f343effd700 1 --2- 172.21.15.1:0/86562289 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f342c06ac90 0x7f342c06d150 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3434009130 tx=0x7f3434008070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:39.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:39.823+0000 7f34277fe700 1 -- 172.21.15.1:0/86562289 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f34400b7720 con 0x7f34400cfb50 2024-03-18T18:50:40.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.028+0000 7f34457d6700 1 -- 172.21.15.1:0/86562289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f34400b7720 con 0x7f34400cfb50 2024-03-18T18:50:40.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.029+0000 7f34277fe700 1 -- 172.21.15.1:0/86562289 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f34400b7720 con 0x7f34400cfb50 2024-03-18T18:50:40.026 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T18:50:40.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.031+0000 7f34457d6700 1 -- 172.21.15.1:0/86562289 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f342c06ac90 msgr2=0x7f342c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.031+0000 7f34457d6700 1 --2- 172.21.15.1:0/86562289 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f342c06ac90 0x7f342c06d150 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3434009130 tx=0x7f3434008070 comp rx=0 tx=0).stop 2024-03-18T18:50:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.031+0000 7f34457d6700 1 -- 172.21.15.1:0/86562289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34400cfb50 msgr2=0x7f34400b8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.031+0000 7f34457d6700 1 --2- 172.21.15.1:0/86562289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34400cfb50 0x7f34400b8080 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f343000d800 tx=0x7f343000db10 comp rx=0 tx=0).stop 2024-03-18T18:50:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.031+0000 7f34457d6700 1 -- 172.21.15.1:0/86562289 shutdown_connections 2024-03-18T18:50:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.031+0000 7f34457d6700 1 --2- 172.21.15.1:0/86562289 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f342c06ac90 0x7f342c06d150 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.031+0000 7f34457d6700 1 --2- 172.21.15.1:0/86562289 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34400b3d40 0x7f34400ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.031+0000 7f34457d6700 1 --2- 172.21.15.1:0/86562289 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34400cff30 0x7f3440147ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.031+0000 7f34457d6700 1 --2- 172.21.15.1:0/86562289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34400cfb50 0x7f34400b8080 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.031+0000 7f34457d6700 1 -- 172.21.15.1:0/86562289 >> 172.21.15.1:0/86562289 conn(0x7f34400d3140 msgr2=0x7f34400b1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.031+0000 7f34457d6700 1 -- 172.21.15.1:0/86562289 shutdown_connections 2024-03-18T18:50:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.031+0000 7f34457d6700 1 -- 172.21.15.1:0/86562289 wait complete. 2024-03-18T18:50:40.041 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T18:50:40.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.173+0000 7ff9ab277700 1 Processor -- start 2024-03-18T18:50:40.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.173+0000 7ff9ab277700 1 -- start start 2024-03-18T18:50:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.174+0000 7ff9ab277700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff9a40cfb50 0x7ff9a40d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.174+0000 7ff9ab277700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9a40cee80 0x7ff9a40b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.174+0000 7ff9ab277700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a40b3370 con 0x7ff9a40cff30 2024-03-18T18:50:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.174+0000 7ff9ab277700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9a40cf310 con 0x7ff9a40cfb50 2024-03-18T18:50:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.174+0000 7ff9ab277700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a40b2760 con 0x7ff9a40cee80 2024-03-18T18:50:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.174+0000 7ff9a9013700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff9a40cfb50 0x7ff9a40d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.174+0000 7ff9a9013700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff9a40cfb50 0x7ff9a40d3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49856/0 (socket says 172.21.15.1:49856) 2024-03-18T18:50:40.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.174+0000 7ff9a9013700 1 -- 172.21.15.1:0/2419039454 learned_addr learned my addr 172.21.15.1:0/2419039454 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.174+0000 7ff9a9013700 1 -- 172.21.15.1:0/2419039454 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9a40cee80 msgr2=0x7ff9a40b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.174+0000 7ff9a9814700 1 --2- 172.21.15.1:0/2419039454 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9a40cee80 0x7ff9a40b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.175+0000 7ff9a9013700 1 --2- 172.21.15.1:0/2419039454 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9a40cee80 0x7ff9a40b3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.175+0000 7ff9a9013700 1 -- 172.21.15.1:0/2419039454 >> v1:172.21.15.165:6789/0 conn(0x7ff9a40cff30 legacy=0x7ff9a40b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.175+0000 7ff9a9013700 1 -- 172.21.15.1:0/2419039454 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a40b2a40 con 0x7ff9a40cfb50 2024-03-18T18:50:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.175+0000 7ff9a9013700 1 --2- 172.21.15.1:0/2419039454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9a40cfb50 0x7ff9a40d3c60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff99c009ea0 tx=0x7ff99c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=433507309003310e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.175+0000 7ff9a3fff700 1 -- 172.21.15.1:0/2419039454 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c00e070 con 0x7ff9a40cfb50 2024-03-18T18:50:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.175+0000 7ff9a3fff700 1 -- 172.21.15.1:0/2419039454 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99c003a20 con 0x7ff9a40cfb50 2024-03-18T18:50:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.175+0000 7ff9a3fff700 1 -- 172.21.15.1:0/2419039454 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c003dd0 con 0x7ff9a40cfb50 2024-03-18T18:50:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.175+0000 7ff9ab277700 1 -- 172.21.15.1:0/2419039454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9a40cfb50 msgr2=0x7ff9a40d3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.175+0000 7ff9ab277700 1 --2- 172.21.15.1:0/2419039454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9a40cfb50 0x7ff9a40d3c60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff99c009ea0 tx=0x7ff99c00b330 comp rx=0 tx=0).stop 2024-03-18T18:50:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.175+0000 7ff9ab277700 1 -- 172.21.15.1:0/2419039454 shutdown_connections 2024-03-18T18:50:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.175+0000 7ff9ab277700 1 --2- 172.21.15.1:0/2419039454 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9a40cee80 0x7ff9a40b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.175+0000 7ff9ab277700 1 --2- 172.21.15.1:0/2419039454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9a40cfb50 0x7ff9a40d3c60 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.175+0000 7ff9ab277700 1 -- 172.21.15.1:0/2419039454 >> 172.21.15.1:0/2419039454 conn(0x7ff9a40d3140 msgr2=0x7ff9a40d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:40.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.176+0000 7ff9ab277700 1 -- 172.21.15.1:0/2419039454 shutdown_connections 2024-03-18T18:50:40.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.176+0000 7ff9ab277700 1 -- 172.21.15.1:0/2419039454 wait complete. 2024-03-18T18:50:40.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.177+0000 7ff9ab277700 1 Processor -- start 2024-03-18T18:50:40.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.177+0000 7ff9ab277700 1 -- start start 2024-03-18T18:50:40.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.178+0000 7ff9ab277700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9a40cee80 0x7ff9a40ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:40.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.178+0000 7ff9ab277700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9a40cfb50 0x7ff9a415ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:40.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.178+0000 7ff9ab277700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9a40cff30 0x7ff9a40cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:40.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.178+0000 7ff9ab277700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a40c80a0 con 0x7ff9a40cee80 2024-03-18T18:50:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.178+0000 7ff9ab277700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a415cf40 con 0x7ff9a40cfb50 2024-03-18T18:50:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.178+0000 7ff9ab277700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a40c7460 con 0x7ff9a40cff30 2024-03-18T18:50:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.178+0000 7ff9a9013700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9a40cee80 0x7ff9a40ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.178+0000 7ff9a9013700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9a40cee80 0x7ff9a40ccd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49860/0 (socket says 172.21.15.1:49860) 2024-03-18T18:50:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.178+0000 7ff9a9013700 1 -- 172.21.15.1:0/3287704309 learned_addr learned my addr 172.21.15.1:0/3287704309 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.178+0000 7ff9a9814700 1 --2- 172.21.15.1:0/3287704309 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9a40cff30 0x7ff9a40cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.178+0000 7ff9a9013700 1 -- 172.21.15.1:0/3287704309 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9a40cff30 msgr2=0x7ff9a40cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.178+0000 7ff9a8812700 1 --2- 172.21.15.1:0/3287704309 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9a40cfb50 0x7ff9a415ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.178+0000 7ff9a9013700 1 --2- 172.21.15.1:0/3287704309 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9a40cff30 0x7ff9a40cc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.178+0000 7ff9a9013700 1 -- 172.21.15.1:0/3287704309 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9a40cfb50 msgr2=0x7ff9a415ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.179+0000 7ff9a9013700 1 --2- 172.21.15.1:0/3287704309 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9a40cfb50 0x7ff9a415ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.179+0000 7ff9a9013700 1 -- 172.21.15.1:0/3287704309 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff99c009b30 con 0x7ff9a40cee80 2024-03-18T18:50:40.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.179+0000 7ff9a9814700 1 --2- 172.21.15.1:0/3287704309 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9a40cff30 0x7ff9a40cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:40.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.179+0000 7ff9a8812700 1 --2- 172.21.15.1:0/3287704309 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9a40cfb50 0x7ff9a415ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:40.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.179+0000 7ff9a9013700 1 --2- 172.21.15.1:0/3287704309 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9a40cee80 0x7ff9a40ccd80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff99c015040 tx=0x7ff99c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:40.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.179+0000 7ff9a1ffb700 1 -- 172.21.15.1:0/3287704309 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c004540 con 0x7ff9a40cee80 2024-03-18T18:50:40.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.179+0000 7ff9a1ffb700 1 -- 172.21.15.1:0/3287704309 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99c0046c0 con 0x7ff9a40cee80 2024-03-18T18:50:40.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.179+0000 7ff9ab277700 1 -- 172.21.15.1:0/3287704309 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a40c7700 con 0x7ff9a40cee80 2024-03-18T18:50:40.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.179+0000 7ff9a1ffb700 1 -- 172.21.15.1:0/3287704309 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c012430 con 0x7ff9a40cee80 2024-03-18T18:50:40.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.179+0000 7ff9ab277700 1 -- 172.21.15.1:0/3287704309 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a40c7cc0 con 0x7ff9a40cee80 2024-03-18T18:50:40.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.180+0000 7ff9ab277700 1 -- 172.21.15.1:0/3287704309 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a40d0390 con 0x7ff9a40cee80 2024-03-18T18:50:40.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.181+0000 7ff9a1ffb700 1 -- 172.21.15.1:0/3287704309 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff99c012590 con 0x7ff9a40cee80 2024-03-18T18:50:40.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.181+0000 7ff9a1ffb700 1 --2- 172.21.15.1:0/3287704309 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff99006ac90 0x7ff99006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:40.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.181+0000 7ff9a1ffb700 1 -- 172.21.15.1:0/3287704309 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff99c08ad40 con 0x7ff9a40cee80 2024-03-18T18:50:40.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.181+0000 7ff9a8812700 1 --2- 172.21.15.1:0/3287704309 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff99006ac90 0x7ff99006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:40.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.182+0000 7ff9a8812700 1 --2- 172.21.15.1:0/3287704309 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff99006ac90 0x7ff99006d150 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff998006fd0 tx=0x7ff998008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:40.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.186+0000 7ff9a1ffb700 1 -- 172.21.15.1:0/3287704309 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff9a40d0390 con 0x7ff9a40cee80 2024-03-18T18:50:40.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.386+0000 7ff9ab277700 1 -- 172.21.15.1:0/3287704309 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff9a40c81e0 con 0x7ff9a40cee80 2024-03-18T18:50:40.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.387+0000 7ff9a1ffb700 1 -- 172.21.15.1:0/3287704309 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff9a40c81e0 con 0x7ff9a40cee80 2024-03-18T18:50:40.385 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:40.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.389+0000 7ff9ab277700 1 -- 172.21.15.1:0/3287704309 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff99006ac90 msgr2=0x7ff99006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.389+0000 7ff9ab277700 1 --2- 172.21.15.1:0/3287704309 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff99006ac90 0x7ff99006d150 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff998006fd0 tx=0x7ff998008040 comp rx=0 tx=0).stop 2024-03-18T18:50:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.389+0000 7ff9ab277700 1 -- 172.21.15.1:0/3287704309 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9a40cee80 msgr2=0x7ff9a40ccd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.389+0000 7ff9ab277700 1 --2- 172.21.15.1:0/3287704309 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9a40cee80 0x7ff9a40ccd80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff99c015040 tx=0x7ff99c004ad0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.389+0000 7ff9ab277700 1 -- 172.21.15.1:0/3287704309 shutdown_connections 2024-03-18T18:50:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.389+0000 7ff9ab277700 1 --2- 172.21.15.1:0/3287704309 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff99006ac90 0x7ff99006d150 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.389+0000 7ff9ab277700 1 --2- 172.21.15.1:0/3287704309 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9a40cff30 0x7ff9a40cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.389+0000 7ff9ab277700 1 --2- 172.21.15.1:0/3287704309 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9a40cee80 0x7ff9a40ccd80 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.389+0000 7ff9ab277700 1 --2- 172.21.15.1:0/3287704309 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9a40cfb50 0x7ff9a415ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.389+0000 7ff9ab277700 1 -- 172.21.15.1:0/3287704309 >> 172.21.15.1:0/3287704309 conn(0x7ff9a40d3140 msgr2=0x7ff9a40b52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.389+0000 7ff9ab277700 1 -- 172.21.15.1:0/3287704309 shutdown_connections 2024-03-18T18:50:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.389+0000 7ff9ab277700 1 -- 172.21.15.1:0/3287704309 wait complete. 2024-03-18T18:50:40.400 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T18:50:40.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.527+0000 7f15683c0700 1 Processor -- start 2024-03-18T18:50:40.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.528+0000 7f15683c0700 1 -- start start 2024-03-18T18:50:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.528+0000 7f15683c0700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f156015d580 0x7f156006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.528+0000 7f15683c0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15600724b0 0x7f15600cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.528+0000 7f15683c0700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f156006f4f0 con 0x7f156006d380 2024-03-18T18:50:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.528+0000 7f15683c0700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f156015d960 con 0x7f156015d580 2024-03-18T18:50:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.528+0000 7f15683c0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f156006ab30 con 0x7f15600724b0 2024-03-18T18:50:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.529+0000 7f156615c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f156015d580 0x7f156006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.529+0000 7f156595b700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f156006d380 0x7f1560071da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:55404/0 (socket says 172.21.15.1:55404) 2024-03-18T18:50:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.529+0000 7f156695d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15600724b0 0x7f15600cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.529+0000 7f156595b700 1 -- 172.21.15.1:0/572703579 learned_addr learned my addr 172.21.15.1:0/572703579 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.529+0000 7f156615c700 1 -- 172.21.15.1:0/572703579 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15600724b0 msgr2=0x7f15600cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.529+0000 7f156615c700 1 --2- 172.21.15.1:0/572703579 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15600724b0 0x7f15600cf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.529+0000 7f156615c700 1 -- 172.21.15.1:0/572703579 >> v1:172.21.15.165:6789/0 conn(0x7f156006d380 legacy=0x7f1560071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.529+0000 7f156615c700 1 -- 172.21.15.1:0/572703579 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156006ae10 con 0x7f156015d580 2024-03-18T18:50:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.529+0000 7f156695d700 1 --2- 172.21.15.1:0/572703579 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15600724b0 0x7f15600cf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:50:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.529+0000 7f156615c700 1 --2- 172.21.15.1:0/572703579 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f156015d580 0x7f156006efb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f155c009840 tx=0x7f155c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fe419b2f116691b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.530+0000 7f156515a700 1 -- 172.21.15.1:0/572703579 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155c011830 con 0x7f156015d580 2024-03-18T18:50:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.530+0000 7f156515a700 1 -- 172.21.15.1:0/572703579 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155c011990 con 0x7f156015d580 2024-03-18T18:50:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.530+0000 7f156515a700 1 -- 172.21.15.1:0/572703579 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155c011d20 con 0x7f156015d580 2024-03-18T18:50:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.530+0000 7f15683c0700 1 -- 172.21.15.1:0/572703579 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f156015d580 msgr2=0x7f156006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.530+0000 7f15683c0700 1 --2- 172.21.15.1:0/572703579 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f156015d580 0x7f156006efb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f155c009840 tx=0x7f155c009b50 comp rx=0 tx=0).stop 2024-03-18T18:50:40.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.530+0000 7f15683c0700 1 -- 172.21.15.1:0/572703579 shutdown_connections 2024-03-18T18:50:40.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.530+0000 7f15683c0700 1 --2- 172.21.15.1:0/572703579 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15600724b0 0x7f15600cf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.530+0000 7f15683c0700 1 --2- 172.21.15.1:0/572703579 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f156015d580 0x7f156006efb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.530+0000 7f15683c0700 1 -- 172.21.15.1:0/572703579 >> 172.21.15.1:0/572703579 conn(0x7f15600cca00 msgr2=0x7f15600cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:40.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.530+0000 7f15683c0700 1 -- 172.21.15.1:0/572703579 shutdown_connections 2024-03-18T18:50:40.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.530+0000 7f15683c0700 1 -- 172.21.15.1:0/572703579 wait complete. 2024-03-18T18:50:40.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.532+0000 7f15683c0700 1 Processor -- start 2024-03-18T18:50:40.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.532+0000 7f15683c0700 1 -- start start 2024-03-18T18:50:40.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.532+0000 7f15683c0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f156006d380 0x7f15600b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:40.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.532+0000 7f15683c0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15600724b0 0x7f1560157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:40.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.532+0000 7f15683c0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f156015d580 0x7f15600ae120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:40.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.532+0000 7f15683c0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15600b5340 con 0x7f156006d380 2024-03-18T18:50:40.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.532+0000 7f15683c0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15600b5600 con 0x7f156015d580 2024-03-18T18:50:40.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.532+0000 7f15683c0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15600b58e0 con 0x7f15600724b0 2024-03-18T18:50:40.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.532+0000 7f156615c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f156006d380 0x7f15600b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:40.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.532+0000 7f156615c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f156006d380 0x7f15600b3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49868/0 (socket says 172.21.15.1:49868) 2024-03-18T18:50:40.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.532+0000 7f156615c700 1 -- 172.21.15.1:0/3863133538 learned_addr learned my addr 172.21.15.1:0/3863133538 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:40.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.533+0000 7f156695d700 1 --2- 172.21.15.1:0/3863133538 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f156015d580 0x7f15600ae120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:40.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.533+0000 7f156595b700 1 --2- 172.21.15.1:0/3863133538 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15600724b0 0x7f1560157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:40.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.533+0000 7f156615c700 1 -- 172.21.15.1:0/3863133538 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15600724b0 msgr2=0x7f1560157040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:40.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.533+0000 7f156615c700 1 --2- 172.21.15.1:0/3863133538 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15600724b0 0x7f1560157040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.533+0000 7f156615c700 1 -- 172.21.15.1:0/3863133538 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f156015d580 msgr2=0x7f15600ae120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:40.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.533+0000 7f156615c700 1 --2- 172.21.15.1:0/3863133538 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f156015d580 0x7f15600ae120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.533+0000 7f156615c700 1 -- 172.21.15.1:0/3863133538 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1558009a70 con 0x7f156006d380 2024-03-18T18:50:40.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.533+0000 7f156595b700 1 --2- 172.21.15.1:0/3863133538 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15600724b0 0x7f1560157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.533+0000 7f156615c700 1 --2- 172.21.15.1:0/3863133538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f156006d380 0x7f15600b3540 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f155c009fd0 tx=0x7f155c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.533+0000 7f1556ffd700 1 -- 172.21.15.1:0/3863133538 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155c0036a0 con 0x7f156006d380 2024-03-18T18:50:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.533+0000 7f1556ffd700 1 -- 172.21.15.1:0/3863133538 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155c01f070 con 0x7f156006d380 2024-03-18T18:50:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.533+0000 7f15683c0700 1 -- 172.21.15.1:0/3863133538 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f155c0094a0 con 0x7f156006d380 2024-03-18T18:50:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.533+0000 7f1556ffd700 1 -- 172.21.15.1:0/3863133538 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155c003a60 con 0x7f156006d380 2024-03-18T18:50:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.533+0000 7f15683c0700 1 -- 172.21.15.1:0/3863133538 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15601581b0 con 0x7f156006d380 2024-03-18T18:50:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.534+0000 7f15683c0700 1 -- 172.21.15.1:0/3863133538 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15600ceea0 con 0x7f156006d380 2024-03-18T18:50:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.534+0000 7f1556ffd700 1 -- 172.21.15.1:0/3863133538 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f155c003bc0 con 0x7f156006d380 2024-03-18T18:50:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.535+0000 7f1556ffd700 1 --2- 172.21.15.1:0/3863133538 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f154406ac90 0x7f154406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.535+0000 7f156595b700 1 --2- 172.21.15.1:0/3863133538 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f154406ac90 0x7f154406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.535+0000 7f1556ffd700 1 -- 172.21.15.1:0/3863133538 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f155c021070 con 0x7f156006d380 2024-03-18T18:50:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.535+0000 7f156595b700 1 --2- 172.21.15.1:0/3863133538 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f154406ac90 0x7f154406d150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1550007fd0 tx=0x7f1550009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.540+0000 7f1556ffd700 1 -- 172.21.15.1:0/3863133538 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f155c055ed0 con 0x7f156006d380 2024-03-18T18:50:40.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.756+0000 7f15683c0700 1 -- 172.21.15.1:0/3863133538 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1560073350 con 0x7f156006d380 2024-03-18T18:50:40.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.756+0000 7f1556ffd700 1 -- 172.21.15.1:0/3863133538 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f155c015020 con 0x7f156006d380 2024-03-18T18:50:40.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.758+0000 7f15683c0700 1 -- 172.21.15.1:0/3863133538 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f154406ac90 msgr2=0x7f154406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:40.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.758+0000 7f15683c0700 1 --2- 172.21.15.1:0/3863133538 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f154406ac90 0x7f154406d150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1550007fd0 tx=0x7f1550009040 comp rx=0 tx=0).stop 2024-03-18T18:50:40.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.758+0000 7f15683c0700 1 -- 172.21.15.1:0/3863133538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f156006d380 msgr2=0x7f15600b3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:40.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.758+0000 7f15683c0700 1 --2- 172.21.15.1:0/3863133538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f156006d380 0x7f15600b3540 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f155c009fd0 tx=0x7f155c004750 comp rx=0 tx=0).stop 2024-03-18T18:50:40.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.758+0000 7f15683c0700 1 -- 172.21.15.1:0/3863133538 shutdown_connections 2024-03-18T18:50:40.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.758+0000 7f15683c0700 1 --2- 172.21.15.1:0/3863133538 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f154406ac90 0x7f154406d150 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.758+0000 7f15683c0700 1 --2- 172.21.15.1:0/3863133538 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f156015d580 0x7f15600ae120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.758+0000 7f15683c0700 1 --2- 172.21.15.1:0/3863133538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f156006d380 0x7f15600b3540 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.758+0000 7f15683c0700 1 --2- 172.21.15.1:0/3863133538 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15600724b0 0x7f1560157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.758+0000 7f15683c0700 1 -- 172.21.15.1:0/3863133538 >> 172.21.15.1:0/3863133538 conn(0x7f15600cca00 msgr2=0x7f15600b0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.758+0000 7f15683c0700 1 -- 172.21.15.1:0/3863133538 shutdown_connections 2024-03-18T18:50:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.758+0000 7f15683c0700 1 -- 172.21.15.1:0/3863133538 wait complete. 2024-03-18T18:50:40.757 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-18T18:50:40.768 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:50:40.768 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T18:50:40.787 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:40.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.955+0000 7f52755e6700 1 Processor -- start 2024-03-18T18:50:40.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.955+0000 7f52755e6700 1 -- start start 2024-03-18T18:50:40.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.956+0000 7f52755e6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f527006e8f0 0x7f5270071440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:40.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.956+0000 7f52755e6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f527006ce90 0x7f527006d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:40.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.956+0000 7f52755e6700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52700cd5a0 con 0x7f527015d620 2024-03-18T18:50:40.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.956+0000 7f52755e6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f527015da00 con 0x7f527006e8f0 2024-03-18T18:50:40.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.956+0000 7f52755e6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f527015e5f0 con 0x7f527006ce90 2024-03-18T18:50:40.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.956+0000 7f5267fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f527006e8f0 0x7f5270071440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:40.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.956+0000 7f5267fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f527006e8f0 0x7f5270071440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49880/0 (socket says 172.21.15.1:49880) 2024-03-18T18:50:40.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.956+0000 7f5267fff700 1 -- 172.21.15.1:0/2164793859 learned_addr learned my addr 172.21.15.1:0/2164793859 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:40.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.956+0000 7f526f7fe700 1 --2- 172.21.15.1:0/2164793859 >> v2:172.21.15.165:3300/0 conn(0x7f527006ce90 0x7f527006d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:40.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.956+0000 7f5267fff700 1 -- 172.21.15.1:0/2164793859 >> v2:172.21.15.165:3300/0 conn(0x7f527006ce90 msgr2=0x7f527006d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:40.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.957+0000 7f5267fff700 1 --2- 172.21.15.1:0/2164793859 >> v2:172.21.15.165:3300/0 conn(0x7f527006ce90 0x7f527006d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.957+0000 7f5267fff700 1 -- 172.21.15.1:0/2164793859 >> v1:172.21.15.165:6789/0 conn(0x7f527015d620 legacy=0x7f5270070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:40.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.957+0000 7f5267fff700 1 -- 172.21.15.1:0/2164793859 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527015e8d0 con 0x7f527006e8f0 2024-03-18T18:50:40.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.957+0000 7f5267fff700 1 --2- 172.21.15.1:0/2164793859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527006e8f0 0x7f5270071440 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5270071f50 tx=0x7f5258009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df1cf350cee43dc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:40.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.957+0000 7f526e7fc700 1 -- 172.21.15.1:0/2164793859 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525800b870 con 0x7f527006e8f0 2024-03-18T18:50:40.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.957+0000 7f526e7fc700 1 -- 172.21.15.1:0/2164793859 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525800b9d0 con 0x7f527006e8f0 2024-03-18T18:50:40.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.957+0000 7f526e7fc700 1 -- 172.21.15.1:0/2164793859 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525800bd60 con 0x7f527006e8f0 2024-03-18T18:50:40.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.957+0000 7f52755e6700 1 -- 172.21.15.1:0/2164793859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527006e8f0 msgr2=0x7f5270071440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:40.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.957+0000 7f52755e6700 1 --2- 172.21.15.1:0/2164793859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527006e8f0 0x7f5270071440 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5270071f50 tx=0x7f5258009bd0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.957+0000 7f52755e6700 1 -- 172.21.15.1:0/2164793859 shutdown_connections 2024-03-18T18:50:40.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.957+0000 7f52755e6700 1 --2- 172.21.15.1:0/2164793859 >> v2:172.21.15.165:3300/0 conn(0x7f527006ce90 0x7f527006d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.957+0000 7f52755e6700 1 --2- 172.21.15.1:0/2164793859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527006e8f0 0x7f5270071440 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.957+0000 7f52755e6700 1 -- 172.21.15.1:0/2164793859 >> 172.21.15.1:0/2164793859 conn(0x7f52700b1270 msgr2=0x7f52700b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:40.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.958+0000 7f52755e6700 1 -- 172.21.15.1:0/2164793859 shutdown_connections 2024-03-18T18:50:40.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.958+0000 7f52755e6700 1 -- 172.21.15.1:0/2164793859 wait complete. 2024-03-18T18:50:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.959+0000 7f52755e6700 1 Processor -- start 2024-03-18T18:50:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.959+0000 7f52755e6700 1 -- start start 2024-03-18T18:50:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.959+0000 7f52755e6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f527006ce90 0x7f52700b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.959+0000 7f52755e6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f527006e8f0 0x7f52700c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.959+0000 7f52755e6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527015d620 0x7f5270152e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.959+0000 7f52755e6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52700b5160 con 0x7f527015d620 2024-03-18T18:50:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.959+0000 7f52755e6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52700c5ca0 con 0x7f527006e8f0 2024-03-18T18:50:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.959+0000 7f52755e6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5270153360 con 0x7f527006ce90 2024-03-18T18:50:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.960+0000 7f526f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527015d620 0x7f5270152e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.960+0000 7f526f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527015d620 0x7f5270152e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49884/0 (socket says 172.21.15.1:49884) 2024-03-18T18:50:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.960+0000 7f526f7fe700 1 -- 172.21.15.1:0/56451078 learned_addr learned my addr 172.21.15.1:0/56451078 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.960+0000 7f5267fff700 1 --2- 172.21.15.1:0/56451078 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f527006e8f0 0x7f52700c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.960+0000 7f526effd700 1 --2- 172.21.15.1:0/56451078 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f527006ce90 0x7f52700b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.960+0000 7f526f7fe700 1 -- 172.21.15.1:0/56451078 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f527006ce90 msgr2=0x7f52700b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.960+0000 7f526f7fe700 1 --2- 172.21.15.1:0/56451078 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f527006ce90 0x7f52700b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.960+0000 7f526f7fe700 1 -- 172.21.15.1:0/56451078 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f527006e8f0 msgr2=0x7f52700c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.960+0000 7f526f7fe700 1 --2- 172.21.15.1:0/56451078 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f527006e8f0 0x7f52700c5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.960+0000 7f526f7fe700 1 -- 172.21.15.1:0/56451078 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5258009550 con 0x7f527015d620 2024-03-18T18:50:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.960+0000 7f5267fff700 1 --2- 172.21.15.1:0/56451078 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f527006e8f0 0x7f52700c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.960+0000 7f526f7fe700 1 --2- 172.21.15.1:0/56451078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527015d620 0x7f5270152e20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5268007e70 tx=0x7f526800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.960+0000 7f52677fe700 1 -- 172.21.15.1:0/56451078 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526800ed90 con 0x7f527015d620 2024-03-18T18:50:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.960+0000 7f52677fe700 1 -- 172.21.15.1:0/56451078 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5268014420 con 0x7f527015d620 2024-03-18T18:50:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.960+0000 7f52755e6700 1 -- 172.21.15.1:0/56451078 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52700b5370 con 0x7f527015d620 2024-03-18T18:50:40.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.960+0000 7f52677fe700 1 -- 172.21.15.1:0/56451078 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52680147d0 con 0x7f527015d620 2024-03-18T18:50:40.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.960+0000 7f52755e6700 1 -- 172.21.15.1:0/56451078 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52700c3bf0 con 0x7f527015d620 2024-03-18T18:50:40.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.961+0000 7f52755e6700 1 -- 172.21.15.1:0/56451078 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52701e8b00 con 0x7f527015d620 2024-03-18T18:50:40.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.962+0000 7f52677fe700 1 -- 172.21.15.1:0/56451078 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5268014930 con 0x7f527015d620 2024-03-18T18:50:40.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.962+0000 7f52677fe700 1 --2- 172.21.15.1:0/56451078 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f525006ac90 0x7f525006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:40.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.962+0000 7f52677fe700 1 -- 172.21.15.1:0/56451078 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5268089eb0 con 0x7f527015d620 2024-03-18T18:50:40.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.962+0000 7f526effd700 1 --2- 172.21.15.1:0/56451078 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f525006ac90 0x7f525006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:40.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.963+0000 7f526effd700 1 --2- 172.21.15.1:0/56451078 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f525006ac90 0x7f525006d150 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5260000f60 tx=0x7f5260009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:40.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:40.967+0000 7f52677fe700 1 -- 172.21.15.1:0/56451078 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5268054c40 con 0x7f527015d620 2024-03-18T18:50:41.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.168+0000 7f52755e6700 1 -- 172.21.15.1:0/56451078 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52700676a0 con 0x7f527015d620 2024-03-18T18:50:41.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.170+0000 7f52677fe700 1 -- 172.21.15.1:0/56451078 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18217 (secure 0 0 0) 0x7f5268058260 con 0x7f527015d620 2024-03-18T18:50:41.167 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:41.167 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":27,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:50:34.993482+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:41.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.172+0000 7f52755e6700 1 -- 172.21.15.1:0/56451078 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f525006ac90 msgr2=0x7f525006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:41.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.172+0000 7f52755e6700 1 --2- 172.21.15.1:0/56451078 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f525006ac90 0x7f525006d150 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5260000f60 tx=0x7f5260009040 comp rx=0 tx=0).stop 2024-03-18T18:50:41.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.172+0000 7f52755e6700 1 -- 172.21.15.1:0/56451078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527015d620 msgr2=0x7f5270152e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:41.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.172+0000 7f52755e6700 1 --2- 172.21.15.1:0/56451078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527015d620 0x7f5270152e20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5268007e70 tx=0x7f526800e3c0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.172+0000 7f52755e6700 1 -- 172.21.15.1:0/56451078 shutdown_connections 2024-03-18T18:50:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.172+0000 7f52755e6700 1 --2- 172.21.15.1:0/56451078 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f525006ac90 0x7f525006d150 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.172+0000 7f52755e6700 1 --2- 172.21.15.1:0/56451078 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f527006ce90 0x7f52700b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.172+0000 7f52755e6700 1 --2- 172.21.15.1:0/56451078 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527015d620 0x7f5270152e20 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.172+0000 7f52755e6700 1 --2- 172.21.15.1:0/56451078 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f527006e8f0 0x7f52700c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.172+0000 7f52755e6700 1 -- 172.21.15.1:0/56451078 >> 172.21.15.1:0/56451078 conn(0x7f52700b1270 msgr2=0x7f527015ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.172+0000 7f52755e6700 1 -- 172.21.15.1:0/56451078 shutdown_connections 2024-03-18T18:50:41.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.172+0000 7f52755e6700 1 -- 172.21.15.1:0/56451078 wait complete. 2024-03-18T18:50:41.182 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T18:49:42.800801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T18:49:43.776684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T18:49:46.854608+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T18:50:41.182 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T18:50:41.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.312+0000 7f2c0dfb9700 1 Processor -- start 2024-03-18T18:50:41.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.312+0000 7f2c0dfb9700 1 -- start start 2024-03-18T18:50:41.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.313+0000 7f2c0dfb9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c0815d3b0 0x7f2c0806cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:41.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.313+0000 7f2c0dfb9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c080b2b80 con 0x7f2c08071550 2024-03-18T18:50:41.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.313+0000 7f2c0dfb9700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c080b2e60 con 0x7f2c0806d430 2024-03-18T18:50:41.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.313+0000 7f2c0dfb9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c080b3800 con 0x7f2c0815d3b0 2024-03-18T18:50:41.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.313+0000 7f2c07fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2c08071550 0x7f2c080b3c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37402/0 (socket says 172.21.15.1:37402) 2024-03-18T18:50:41.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.313+0000 7f2c07fff700 1 -- 172.21.15.1:0/573943053 learned_addr learned my addr 172.21.15.1:0/573943053 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:41.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.313+0000 7f2c067fc700 1 -- 172.21.15.1:0/573943053 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9800035 0 0) 0x7f2c080b2b80 con 0x7f2c08071550 2024-03-18T18:50:41.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.313+0000 7f2c077fe700 1 --2- 172.21.15.1:0/573943053 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c0815d3b0 0x7f2c0806cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:41.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.313+0000 7f2c067fc700 1 -- 172.21.15.1:0/573943053 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bf00035b0 con 0x7f2c08071550 2024-03-18T18:50:41.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.313+0000 7f2c067fc700 1 -- 172.21.15.1:0/573943053 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3832681680 0 0) 0x7f2bf00035b0 con 0x7f2c08071550 2024-03-18T18:50:41.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c067fc700 1 -- 172.21.15.1:0/573943053 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c0815d3b0 msgr2=0x7f2c0806cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:41.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c067fc700 1 --2- 172.21.15.1:0/573943053 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c0815d3b0 0x7f2c0806cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c067fc700 1 -- 172.21.15.1:0/573943053 >> v1:172.21.15.165:6789/0 conn(0x7f2c0806d430 legacy=0x7f2c08070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:41.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c067fc700 1 -- 172.21.15.1:0/573943053 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c080b3140 con 0x7f2c08071550 2024-03-18T18:50:41.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c067fc700 1 -- 172.21.15.1:0/573943053 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f2c000019f0 con 0x7f2c08071550 2024-03-18T18:50:41.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c077fe700 1 --2- 172.21.15.1:0/573943053 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c0815d3b0 0x7f2c0806cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:41.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c067fc700 1 -- 172.21.15.1:0/573943053 >> v1:172.21.15.1:6789/0 conn(0x7f2c08071550 legacy=0x7f2c080b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:41.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c067fc700 1 --2- 172.21.15.1:0/573943053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bf0003a30 0x7f2bf0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:41.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c067fc700 1 --2- 172.21.15.1:0/573943053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bf0004850 0x7f2bf0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c067fc700 1 --2- 172.21.15.1:0/573943053 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c0815d3b0 0x7f2bf0005280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c067fc700 1 -- 172.21.15.1:0/573943053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c080b3800 con 0x7f2bf0004850 2024-03-18T18:50:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c067fc700 1 -- 172.21.15.1:0/573943053 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf000bba0 con 0x7f2bf0003a30 2024-03-18T18:50:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c067fc700 1 -- 172.21.15.1:0/573943053 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf000bd10 con 0x7f2c0815d3b0 2024-03-18T18:50:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c06ffd700 1 --2- 172.21.15.1:0/573943053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bf0004850 0x7f2bf0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c077fe700 1 --2- 172.21.15.1:0/573943053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bf0003a30 0x7f2bf0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c07fff700 1 --2- 172.21.15.1:0/573943053 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c0815d3b0 0x7f2bf0005280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c06ffd700 1 -- 172.21.15.1:0/573943053 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c0815d3b0 msgr2=0x7f2bf0005280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c06ffd700 1 --2- 172.21.15.1:0/573943053 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c0815d3b0 0x7f2bf0005280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c06ffd700 1 -- 172.21.15.1:0/573943053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bf0003a30 msgr2=0x7f2bf0003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c06ffd700 1 --2- 172.21.15.1:0/573943053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bf0003a30 0x7f2bf0003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c06ffd700 1 -- 172.21.15.1:0/573943053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf00035b0 con 0x7f2bf0004850 2024-03-18T18:50:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c06ffd700 1 --2- 172.21.15.1:0/573943053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bf0004850 0x7f2bf0004cd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2c0806d850 tx=0x7f2bf0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4565629a70c4ddf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:41.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c067fc700 1 -- 172.21.15.1:0/573943053 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf80052d0 con 0x7f2bf0004850 2024-03-18T18:50:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c067fc700 1 -- 172.21.15.1:0/573943053 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf8005470 con 0x7f2bf0004850 2024-03-18T18:50:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.314+0000 7f2c067fc700 1 -- 172.21.15.1:0/573943053 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf80038c0 con 0x7f2bf0004850 2024-03-18T18:50:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.315+0000 7f2c0dfb9700 1 -- 172.21.15.1:0/573943053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bf0004850 msgr2=0x7f2bf0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.315+0000 7f2c0dfb9700 1 --2- 172.21.15.1:0/573943053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bf0004850 0x7f2bf0004cd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2c0806d850 tx=0x7f2bf0005880 comp rx=0 tx=0).stop 2024-03-18T18:50:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.315+0000 7f2c07fff700 1 -- 172.21.15.1:0/573943053 reap_dead start 2024-03-18T18:50:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.315+0000 7f2c0dfb9700 1 -- 172.21.15.1:0/573943053 shutdown_connections 2024-03-18T18:50:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.315+0000 7f2c0dfb9700 1 -- 172.21.15.1:0/573943053 >> 172.21.15.1:0/573943053 conn(0x7f2c080b1350 msgr2=0x7f2c080b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.315+0000 7f2c0dfb9700 1 -- 172.21.15.1:0/573943053 shutdown_connections 2024-03-18T18:50:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.315+0000 7f2c0dfb9700 1 -- 172.21.15.1:0/573943053 wait complete. 2024-03-18T18:50:41.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.316+0000 7f2c0dfb9700 1 Processor -- start 2024-03-18T18:50:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.316+0000 7f2c0dfb9700 1 -- start start 2024-03-18T18:50:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.317+0000 7f2c0dfb9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2bf0004850 0x7f2c081f3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.317+0000 7f2c0dfb9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c08137c30 0x7f2c080af990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.317+0000 7f2c0dfb9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c08134f50 0x7f2c081353e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.317+0000 7f2c0dfb9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c08135920 con 0x7f2c08137c30 2024-03-18T18:50:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.317+0000 7f2c0dfb9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c081383a0 con 0x7f2c08134f50 2024-03-18T18:50:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.317+0000 7f2c0dfb9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c08138680 con 0x7f2bf0004850 2024-03-18T18:50:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.317+0000 7f2c06ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c08137c30 0x7f2c080af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.317+0000 7f2c06ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c08137c30 0x7f2c080af990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49904/0 (socket says 172.21.15.1:49904) 2024-03-18T18:50:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.317+0000 7f2c06ffd700 1 -- 172.21.15.1:0/1467199781 learned_addr learned my addr 172.21.15.1:0/1467199781 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.317+0000 7f2c077fe700 1 --2- 172.21.15.1:0/1467199781 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2bf0004850 0x7f2c081f3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:41.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.317+0000 7f2c07fff700 1 --2- 172.21.15.1:0/1467199781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c08134f50 0x7f2c081353e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.317+0000 7f2c06ffd700 1 -- 172.21.15.1:0/1467199781 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2bf0004850 msgr2=0x7f2c081f3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.317+0000 7f2c06ffd700 1 --2- 172.21.15.1:0/1467199781 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2bf0004850 0x7f2c081f3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.317+0000 7f2c06ffd700 1 -- 172.21.15.1:0/1467199781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c08134f50 msgr2=0x7f2c081353e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.318+0000 7f2c06ffd700 1 --2- 172.21.15.1:0/1467199781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c08134f50 0x7f2c081353e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.318+0000 7f2c06ffd700 1 -- 172.21.15.1:0/1467199781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c08138960 con 0x7f2c08137c30 2024-03-18T18:50:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.318+0000 7f2c07fff700 1 --2- 172.21.15.1:0/1467199781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c08134f50 0x7f2c081353e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.318+0000 7f2c06ffd700 1 --2- 172.21.15.1:0/1467199781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c08137c30 0x7f2c080af990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8004000 tx=0x7f2bf8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.318+0000 7f2beffff700 1 -- 172.21.15.1:0/1467199781 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf8007240 con 0x7f2c08137c30 2024-03-18T18:50:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.318+0000 7f2beffff700 1 -- 172.21.15.1:0/1467199781 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf80073a0 con 0x7f2c08137c30 2024-03-18T18:50:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.318+0000 7f2c0dfb9700 1 -- 172.21.15.1:0/1467199781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c08135d20 con 0x7f2c08137c30 2024-03-18T18:50:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.318+0000 7f2beffff700 1 -- 172.21.15.1:0/1467199781 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf8001290 con 0x7f2c08137c30 2024-03-18T18:50:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.318+0000 7f2c0dfb9700 1 -- 172.21.15.1:0/1467199781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c080afed0 con 0x7f2c08137c30 2024-03-18T18:50:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.319+0000 7f2c0dfb9700 1 -- 172.21.15.1:0/1467199781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c08138ba0 con 0x7f2c08137c30 2024-03-18T18:50:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.319+0000 7f2beffff700 1 -- 172.21.15.1:0/1467199781 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2bf8003680 con 0x7f2c08137c30 2024-03-18T18:50:41.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.320+0000 7f2beffff700 1 --2- 172.21.15.1:0/1467199781 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2bf406ac90 0x7f2bf406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.320+0000 7f2beffff700 1 -- 172.21.15.1:0/1467199781 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2bf808a100 con 0x7f2c08137c30 2024-03-18T18:50:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.320+0000 7f2c077fe700 1 --2- 172.21.15.1:0/1467199781 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2bf406ac90 0x7f2bf406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.320+0000 7f2c077fe700 1 --2- 172.21.15.1:0/1467199781 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2bf406ac90 0x7f2bf406d150 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc004520 tx=0x7f2bfc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:41.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.324+0000 7f2beffff700 1 -- 172.21.15.1:0/1467199781 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2bf80550d0 con 0x7f2c08137c30 2024-03-18T18:50:41.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.528+0000 7f2c0dfb9700 1 -- 172.21.15.1:0/1467199781 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2c08135eb0 con 0x7f2c08137c30 2024-03-18T18:50:41.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.529+0000 7f2beffff700 1 -- 172.21.15.1:0/1467199781 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2bf80586f0 con 0x7f2c08137c30 2024-03-18T18:50:41.526 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T18:50:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.531+0000 7f2c0dfb9700 1 -- 172.21.15.1:0/1467199781 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2bf406ac90 msgr2=0x7f2bf406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.531+0000 7f2c0dfb9700 1 --2- 172.21.15.1:0/1467199781 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2bf406ac90 0x7f2bf406d150 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc004520 tx=0x7f2bfc008040 comp rx=0 tx=0).stop 2024-03-18T18:50:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.531+0000 7f2c0dfb9700 1 -- 172.21.15.1:0/1467199781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c08137c30 msgr2=0x7f2c080af990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.531+0000 7f2c0dfb9700 1 --2- 172.21.15.1:0/1467199781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c08137c30 0x7f2c080af990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8004000 tx=0x7f2bf8001120 comp rx=0 tx=0).stop 2024-03-18T18:50:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.531+0000 7f2c0dfb9700 1 -- 172.21.15.1:0/1467199781 shutdown_connections 2024-03-18T18:50:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.531+0000 7f2c0dfb9700 1 --2- 172.21.15.1:0/1467199781 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2bf406ac90 0x7f2bf406d150 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.531+0000 7f2c0dfb9700 1 --2- 172.21.15.1:0/1467199781 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2bf0004850 0x7f2c081f3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.531+0000 7f2c0dfb9700 1 --2- 172.21.15.1:0/1467199781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c08134f50 0x7f2c081353e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.531+0000 7f2c0dfb9700 1 --2- 172.21.15.1:0/1467199781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c08137c30 0x7f2c080af990 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.531+0000 7f2c0dfb9700 1 -- 172.21.15.1:0/1467199781 >> 172.21.15.1:0/1467199781 conn(0x7f2c080b1350 msgr2=0x7f2c08070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:41.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.531+0000 7f2c0dfb9700 1 -- 172.21.15.1:0/1467199781 shutdown_connections 2024-03-18T18:50:41.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.531+0000 7f2c0dfb9700 1 -- 172.21.15.1:0/1467199781 wait complete. 2024-03-18T18:50:41.541 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T18:50:41.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.671+0000 7f4700862700 1 Processor -- start 2024-03-18T18:50:41.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.671+0000 7f4700862700 1 -- start start 2024-03-18T18:50:41.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.672+0000 7f4700862700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46fc143c60 0x7f46fc06eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:41.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.672+0000 7f4700862700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f46fc0b4a50 0x7f46fc0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:41.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.672+0000 7f4700862700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46fc06f430 con 0x7f46fc072330 2024-03-18T18:50:41.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.672+0000 7f4700862700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46fc06f710 con 0x7f46fc0b4a50 2024-03-18T18:50:41.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.672+0000 7f4700862700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc0cda00 con 0x7f46fc143c60 2024-03-18T18:50:41.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.672+0000 7f46f9d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f46fc072330 0x7f46fc0b4340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37412/0 (socket says 172.21.15.1:37412) 2024-03-18T18:50:41.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.672+0000 7f46f9d9b700 1 -- 172.21.15.1:0/2554793253 learned_addr learned my addr 172.21.15.1:0/2554793253 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:41.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.673+0000 7f46fa59c700 1 --2- 172.21.15.1:0/2554793253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46fc143c60 0x7f46fc06eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:41.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.673+0000 7f46fad9d700 1 --2- 172.21.15.1:0/2554793253 >> v2:172.21.15.165:3300/0 conn(0x7f46fc0b4a50 0x7f46fc0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:41.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.673+0000 7f46f959a700 1 -- 172.21.15.1:0/2554793253 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100812556 0 0) 0x7f46fc06f430 con 0x7f46fc072330 2024-03-18T18:50:41.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.673+0000 7f46f959a700 1 -- 172.21.15.1:0/2554793253 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e40035b0 con 0x7f46fc072330 2024-03-18T18:50:41.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.673+0000 7f46fa59c700 1 -- 172.21.15.1:0/2554793253 >> v2:172.21.15.165:3300/0 conn(0x7f46fc0b4a50 msgr2=0x7f46fc0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:41.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.673+0000 7f46f959a700 1 -- 172.21.15.1:0/2554793253 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1832160945 0 0) 0x7f46e40035b0 con 0x7f46fc072330 2024-03-18T18:50:41.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.673+0000 7f46fa59c700 1 --2- 172.21.15.1:0/2554793253 >> v2:172.21.15.165:3300/0 conn(0x7f46fc0b4a50 0x7f46fc0ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.673+0000 7f46fa59c700 1 -- 172.21.15.1:0/2554793253 >> v1:172.21.15.1:6789/0 conn(0x7f46fc072330 legacy=0x7f46fc0b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:41.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.673+0000 7f46fa59c700 1 -- 172.21.15.1:0/2554793253 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc0cdce0 con 0x7f46fc143c60 2024-03-18T18:50:41.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.673+0000 7f46fa59c700 1 --2- 172.21.15.1:0/2554793253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46fc143c60 0x7f46fc06eb50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f46f0009840 tx=0x7f46f0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4aff5e9e5b6ec3c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:41.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.673+0000 7f46f959a700 1 -- 172.21.15.1:0/2554793253 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ec001670 con 0x7f46fc143c60 2024-03-18T18:50:41.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.673+0000 7f46f959a700 1 -- 172.21.15.1:0/2554793253 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f0004030 con 0x7f46fc143c60 2024-03-18T18:50:41.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.673+0000 7f46f959a700 1 -- 172.21.15.1:0/2554793253 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f00043a0 con 0x7f46fc143c60 2024-03-18T18:50:41.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.674+0000 7f4700862700 1 -- 172.21.15.1:0/2554793253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46fc143c60 msgr2=0x7f46fc06eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:41.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.674+0000 7f4700862700 1 --2- 172.21.15.1:0/2554793253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46fc143c60 0x7f46fc06eb50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f46f0009840 tx=0x7f46f0009b50 comp rx=0 tx=0).stop 2024-03-18T18:50:41.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.674+0000 7f4700862700 1 -- 172.21.15.1:0/2554793253 shutdown_connections 2024-03-18T18:50:41.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.674+0000 7f4700862700 1 --2- 172.21.15.1:0/2554793253 >> v2:172.21.15.165:3300/0 conn(0x7f46fc0b4a50 0x7f46fc0ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.674+0000 7f4700862700 1 --2- 172.21.15.1:0/2554793253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46fc143c60 0x7f46fc06eb50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.674+0000 7f4700862700 1 -- 172.21.15.1:0/2554793253 >> 172.21.15.1:0/2554793253 conn(0x7f46fc0b1d20 msgr2=0x7f46fc0b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:41.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.674+0000 7f4700862700 1 -- 172.21.15.1:0/2554793253 shutdown_connections 2024-03-18T18:50:41.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.674+0000 7f4700862700 1 -- 172.21.15.1:0/2554793253 wait complete. 2024-03-18T18:50:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.675+0000 7f4700862700 1 Processor -- start 2024-03-18T18:50:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.675+0000 7f4700862700 1 -- start start 2024-03-18T18:50:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.676+0000 7f4700862700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46fc072330 0x7f46fc1f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.676+0000 7f4700862700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46fc0b4a50 0x7f46fc0aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.676+0000 7f4700862700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f46fc143c60 0x7f46fc1ef130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:41.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.676+0000 7f4700862700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc1efa50 con 0x7f46fc0b4a50 2024-03-18T18:50:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.676+0000 7f4700862700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc1f3c90 con 0x7f46fc143c60 2024-03-18T18:50:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.676+0000 7f4700862700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc0af2e0 con 0x7f46fc072330 2024-03-18T18:50:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.676+0000 7f46f9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46fc0b4a50 0x7f46fc0aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.676+0000 7f46f9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46fc0b4a50 0x7f46fc0aeda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49906/0 (socket says 172.21.15.1:49906) 2024-03-18T18:50:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.676+0000 7f46f9d9b700 1 -- 172.21.15.1:0/3209467209 learned_addr learned my addr 172.21.15.1:0/3209467209 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.676+0000 7f46fa59c700 1 --2- 172.21.15.1:0/3209467209 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46fc072330 0x7f46fc1f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.676+0000 7f46fad9d700 1 --2- 172.21.15.1:0/3209467209 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f46fc143c60 0x7f46fc1ef130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.677+0000 7f46f9d9b700 1 -- 172.21.15.1:0/3209467209 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46fc072330 msgr2=0x7f46fc1f2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.677+0000 7f46f9d9b700 1 --2- 172.21.15.1:0/3209467209 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46fc072330 0x7f46fc1f2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.677+0000 7f46f9d9b700 1 -- 172.21.15.1:0/3209467209 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f46fc143c60 msgr2=0x7f46fc1ef130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.677+0000 7f46f9d9b700 1 --2- 172.21.15.1:0/3209467209 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f46fc143c60 0x7f46fc1ef130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.677+0000 7f46f9d9b700 1 -- 172.21.15.1:0/3209467209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f00094a0 con 0x7f46fc0b4a50 2024-03-18T18:50:41.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.677+0000 7f46fa59c700 1 --2- 172.21.15.1:0/3209467209 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46fc072330 0x7f46fc1f2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:41.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.677+0000 7f46fad9d700 1 --2- 172.21.15.1:0/3209467209 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f46fc143c60 0x7f46fc1ef130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:41.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.677+0000 7f46f9d9b700 1 --2- 172.21.15.1:0/3209467209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46fc0b4a50 0x7f46fc0aeda0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f46ec00e9c0 tx=0x7f46ec00ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:41.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.677+0000 7f46eaffd700 1 -- 172.21.15.1:0/3209467209 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ec00eea0 con 0x7f46fc0b4a50 2024-03-18T18:50:41.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.677+0000 7f46eaffd700 1 -- 172.21.15.1:0/3209467209 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ec004a80 con 0x7f46fc0b4a50 2024-03-18T18:50:41.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.677+0000 7f4700862700 1 -- 172.21.15.1:0/3209467209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46fc1f33f0 con 0x7f46fc0b4a50 2024-03-18T18:50:41.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.677+0000 7f46eaffd700 1 -- 172.21.15.1:0/3209467209 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ec004e30 con 0x7f46fc0b4a50 2024-03-18T18:50:41.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.677+0000 7f4700862700 1 -- 172.21.15.1:0/3209467209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46fc1f39b0 con 0x7f46fc0b4a50 2024-03-18T18:50:41.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.678+0000 7f4700862700 1 -- 172.21.15.1:0/3209467209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46fc1efb90 con 0x7f46fc0b4a50 2024-03-18T18:50:41.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.678+0000 7f46eaffd700 1 -- 172.21.15.1:0/3209467209 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f46ec0070d0 con 0x7f46fc0b4a50 2024-03-18T18:50:41.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.679+0000 7f46eaffd700 1 --2- 172.21.15.1:0/3209467209 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f46dc06ac90 0x7f46dc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:41.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.679+0000 7f46eaffd700 1 -- 172.21.15.1:0/3209467209 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f46ec08bcd0 con 0x7f46fc0b4a50 2024-03-18T18:50:41.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.679+0000 7f46fa59c700 1 --2- 172.21.15.1:0/3209467209 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f46dc06ac90 0x7f46dc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:41.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.679+0000 7f46fa59c700 1 --2- 172.21.15.1:0/3209467209 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f46dc06ac90 0x7f46dc06d150 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f46f00090b0 tx=0x7f46f0013040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:41.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.683+0000 7f46eaffd700 1 -- 172.21.15.1:0/3209467209 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f46fc1efb90 con 0x7f46fc0b4a50 2024-03-18T18:50:41.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.885+0000 7f4700862700 1 -- 172.21.15.1:0/3209467209 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f46fc1efb90 con 0x7f46fc0b4a50 2024-03-18T18:50:41.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.885+0000 7f46eaffd700 1 -- 172.21.15.1:0/3209467209 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f46fc1efb90 con 0x7f46fc0b4a50 2024-03-18T18:50:41.883 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:41.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.888+0000 7f4700862700 1 -- 172.21.15.1:0/3209467209 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f46dc06ac90 msgr2=0x7f46dc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:41.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.888+0000 7f4700862700 1 --2- 172.21.15.1:0/3209467209 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f46dc06ac90 0x7f46dc06d150 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f46f00090b0 tx=0x7f46f0013040 comp rx=0 tx=0).stop 2024-03-18T18:50:41.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.888+0000 7f4700862700 1 -- 172.21.15.1:0/3209467209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46fc0b4a50 msgr2=0x7f46fc0aeda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:41.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.888+0000 7f4700862700 1 --2- 172.21.15.1:0/3209467209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46fc0b4a50 0x7f46fc0aeda0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f46ec00e9c0 tx=0x7f46ec00ed80 comp rx=0 tx=0).stop 2024-03-18T18:50:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.888+0000 7f4700862700 1 -- 172.21.15.1:0/3209467209 shutdown_connections 2024-03-18T18:50:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.888+0000 7f4700862700 1 --2- 172.21.15.1:0/3209467209 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f46dc06ac90 0x7f46dc06d150 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.888+0000 7f4700862700 1 --2- 172.21.15.1:0/3209467209 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46fc072330 0x7f46fc1f2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.888+0000 7f4700862700 1 --2- 172.21.15.1:0/3209467209 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f46fc143c60 0x7f46fc1ef130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.888+0000 7f4700862700 1 --2- 172.21.15.1:0/3209467209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46fc0b4a50 0x7f46fc0aeda0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.888+0000 7f4700862700 1 -- 172.21.15.1:0/3209467209 >> 172.21.15.1:0/3209467209 conn(0x7f46fc0b1d20 msgr2=0x7f46fc06e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.888+0000 7f4700862700 1 -- 172.21.15.1:0/3209467209 shutdown_connections 2024-03-18T18:50:41.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:41.888+0000 7f4700862700 1 -- 172.21.15.1:0/3209467209 wait complete. 2024-03-18T18:50:41.897 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T18:50:42.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.028+0000 7f1abc6ab700 1 Processor -- start 2024-03-18T18:50:42.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.028+0000 7f1abc6ab700 1 -- start start 2024-03-18T18:50:42.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.029+0000 7f1abc6ab700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1ab415d780 0x7f1ab40cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:42.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.029+0000 7f1abc6ab700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ab409dd40 0x7f1ab40cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:42.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.029+0000 7f1abc6ab700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab40ce090 con 0x7f1ab40ce5a0 2024-03-18T18:50:42.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.029+0000 7f1abc6ab700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab406db30 con 0x7f1ab415d780 2024-03-18T18:50:42.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.029+0000 7f1abc6ab700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab406d520 con 0x7f1ab409dd40 2024-03-18T18:50:42.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.029+0000 7f1aba447700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1ab415d780 0x7f1ab40cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:42.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.029+0000 7f1aba447700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1ab415d780 0x7f1ab40cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49912/0 (socket says 172.21.15.1:49912) 2024-03-18T18:50:42.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.029+0000 7f1aba447700 1 -- 172.21.15.1:0/3223699484 learned_addr learned my addr 172.21.15.1:0/3223699484 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:42.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.030+0000 7f1ab9c46700 1 --2- 172.21.15.1:0/3223699484 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ab409dd40 0x7f1ab40cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:42.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.030+0000 7f1aba447700 1 -- 172.21.15.1:0/3223699484 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ab409dd40 msgr2=0x7f1ab40cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:42.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.030+0000 7f1aba447700 1 --2- 172.21.15.1:0/3223699484 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ab409dd40 0x7f1ab40cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.030+0000 7f1aba447700 1 -- 172.21.15.1:0/3223699484 >> v1:172.21.15.165:6789/0 conn(0x7f1ab40ce5a0 legacy=0x7f1ab40cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:42.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.030+0000 7f1aba447700 1 -- 172.21.15.1:0/3223699484 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab406f460 con 0x7f1ab415d780 2024-03-18T18:50:42.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.030+0000 7f1aba447700 1 --2- 172.21.15.1:0/3223699484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab415d780 0x7f1ab40cdb50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1aa40098f0 tx=0x7f1aa4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4141831a63077d0c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:42.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.030+0000 7f1ab9445700 1 -- 172.21.15.1:0/3223699484 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4004230 con 0x7f1ab415d780 2024-03-18T18:50:42.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.030+0000 7f1ab9445700 1 -- 172.21.15.1:0/3223699484 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa4004390 con 0x7f1ab415d780 2024-03-18T18:50:42.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.030+0000 7f1ab9445700 1 -- 172.21.15.1:0/3223699484 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa40036a0 con 0x7f1ab415d780 2024-03-18T18:50:42.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.031+0000 7f1abc6ab700 1 -- 172.21.15.1:0/3223699484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab415d780 msgr2=0x7f1ab40cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:42.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.031+0000 7f1abc6ab700 1 --2- 172.21.15.1:0/3223699484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab415d780 0x7f1ab40cdb50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1aa40098f0 tx=0x7f1aa4009c00 comp rx=0 tx=0).stop 2024-03-18T18:50:42.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.031+0000 7f1abc6ab700 1 -- 172.21.15.1:0/3223699484 shutdown_connections 2024-03-18T18:50:42.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.031+0000 7f1abc6ab700 1 --2- 172.21.15.1:0/3223699484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab415d780 0x7f1ab40cdb50 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.031+0000 7f1abc6ab700 1 --2- 172.21.15.1:0/3223699484 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ab409dd40 0x7f1ab40cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.031+0000 7f1abc6ab700 1 -- 172.21.15.1:0/3223699484 >> 172.21.15.1:0/3223699484 conn(0x7f1ab40cca00 msgr2=0x7f1ab40cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:42.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.031+0000 7f1abc6ab700 1 -- 172.21.15.1:0/3223699484 shutdown_connections 2024-03-18T18:50:42.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.031+0000 7f1abc6ab700 1 -- 172.21.15.1:0/3223699484 wait complete. 2024-03-18T18:50:42.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.032+0000 7f1abc6ab700 1 Processor -- start 2024-03-18T18:50:42.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.032+0000 7f1abc6ab700 1 -- start start 2024-03-18T18:50:42.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1abc6ab700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab409dd40 0x7f1ab409d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:42.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1abc6ab700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab40ce5a0 0x7f1ab406a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:42.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1abc6ab700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ab415d780 0x7f1ab414a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:42.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1abc6ab700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab414abe0 con 0x7f1ab409dd40 2024-03-18T18:50:42.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1abc6ab700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab406ac90 con 0x7f1ab40ce5a0 2024-03-18T18:50:42.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1abc6ab700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab406af30 con 0x7f1ab415d780 2024-03-18T18:50:42.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1aba447700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab409dd40 0x7f1ab409d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:42.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1aba447700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab409dd40 0x7f1ab409d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49916/0 (socket says 172.21.15.1:49916) 2024-03-18T18:50:42.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1aba447700 1 -- 172.21.15.1:0/1576576051 learned_addr learned my addr 172.21.15.1:0/1576576051 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:42.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1aba447700 1 -- 172.21.15.1:0/1576576051 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ab415d780 msgr2=0x7f1ab414a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:42.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1ab9c46700 1 --2- 172.21.15.1:0/1576576051 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab40ce5a0 0x7f1ab406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:42.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1abac48700 1 --2- 172.21.15.1:0/1576576051 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ab415d780 0x7f1ab414a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:42.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1aba447700 1 --2- 172.21.15.1:0/1576576051 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ab415d780 0x7f1ab414a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1aba447700 1 -- 172.21.15.1:0/1576576051 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab40ce5a0 msgr2=0x7f1ab406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:42.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1aba447700 1 --2- 172.21.15.1:0/1576576051 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab40ce5a0 0x7f1ab406a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1aba447700 1 -- 172.21.15.1:0/1576576051 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa4009550 con 0x7f1ab409dd40 2024-03-18T18:50:42.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1ab9c46700 1 --2- 172.21.15.1:0/1576576051 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab40ce5a0 0x7f1ab406a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1abac48700 1 --2- 172.21.15.1:0/1576576051 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ab415d780 0x7f1ab414a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1aba447700 1 --2- 172.21.15.1:0/1576576051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab409dd40 0x7f1ab409d510 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4009130 tx=0x7f1aa4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1aaaffd700 1 -- 172.21.15.1:0/1576576051 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa400b860 con 0x7f1ab409dd40 2024-03-18T18:50:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1aaaffd700 1 -- 172.21.15.1:0/1576576051 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa400b9c0 con 0x7f1ab409dd40 2024-03-18T18:50:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1aaaffd700 1 -- 172.21.15.1:0/1576576051 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa400bd70 con 0x7f1ab409dd40 2024-03-18T18:50:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1abc6ab700 1 -- 172.21.15.1:0/1576576051 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab406b210 con 0x7f1ab409dd40 2024-03-18T18:50:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.033+0000 7f1abc6ab700 1 -- 172.21.15.1:0/1576576051 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab41e4ec0 con 0x7f1ab409dd40 2024-03-18T18:50:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.034+0000 7f1abc6ab700 1 -- 172.21.15.1:0/1576576051 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab415be30 con 0x7f1ab409dd40 2024-03-18T18:50:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.035+0000 7f1aaaffd700 1 -- 172.21.15.1:0/1576576051 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1aa401b440 con 0x7f1ab409dd40 2024-03-18T18:50:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.035+0000 7f1aaaffd700 1 --2- 172.21.15.1:0/1576576051 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1a9806ac90 0x7f1a9806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.035+0000 7f1aaaffd700 1 -- 172.21.15.1:0/1576576051 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f1aa408b0c0 con 0x7f1ab409dd40 2024-03-18T18:50:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.035+0000 7f1ab9c46700 1 --2- 172.21.15.1:0/1576576051 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1a9806ac90 0x7f1a9806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.036+0000 7f1ab9c46700 1 --2- 172.21.15.1:0/1576576051 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1a9806ac90 0x7f1a9806d150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0006fd0 tx=0x7f1ab0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:42.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.040+0000 7f1aaaffd700 1 -- 172.21.15.1:0/1576576051 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1aa4055f70 con 0x7f1ab409dd40 2024-03-18T18:50:42.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.243+0000 7f1abc6ab700 1 -- 172.21.15.1:0/1576576051 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1ab40676a0 con 0x7f1ab409dd40 2024-03-18T18:50:42.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.243+0000 7f1aaaffd700 1 -- 172.21.15.1:0/1576576051 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1aa4059590 con 0x7f1ab409dd40 2024-03-18T18:50:42.241 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T18:50:42.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.245+0000 7f1abc6ab700 1 -- 172.21.15.1:0/1576576051 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1a9806ac90 msgr2=0x7f1a9806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:42.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.245+0000 7f1abc6ab700 1 --2- 172.21.15.1:0/1576576051 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1a9806ac90 0x7f1a9806d150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0006fd0 tx=0x7f1ab0008040 comp rx=0 tx=0).stop 2024-03-18T18:50:42.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.245+0000 7f1abc6ab700 1 -- 172.21.15.1:0/1576576051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab409dd40 msgr2=0x7f1ab409d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:42.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.245+0000 7f1abc6ab700 1 --2- 172.21.15.1:0/1576576051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab409dd40 0x7f1ab409d510 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4009130 tx=0x7f1aa4004750 comp rx=0 tx=0).stop 2024-03-18T18:50:42.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.245+0000 7f1abc6ab700 1 -- 172.21.15.1:0/1576576051 shutdown_connections 2024-03-18T18:50:42.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.245+0000 7f1abc6ab700 1 --2- 172.21.15.1:0/1576576051 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1a9806ac90 0x7f1a9806d150 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.245+0000 7f1abc6ab700 1 --2- 172.21.15.1:0/1576576051 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ab415d780 0x7f1ab414a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.245+0000 7f1abc6ab700 1 --2- 172.21.15.1:0/1576576051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ab409dd40 0x7f1ab409d510 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.245+0000 7f1abc6ab700 1 --2- 172.21.15.1:0/1576576051 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab40ce5a0 0x7f1ab406a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.245+0000 7f1abc6ab700 1 -- 172.21.15.1:0/1576576051 >> 172.21.15.1:0/1576576051 conn(0x7f1ab40cca00 msgr2=0x7f1ab415b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:42.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.246+0000 7f1abc6ab700 1 -- 172.21.15.1:0/1576576051 shutdown_connections 2024-03-18T18:50:42.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.246+0000 7f1abc6ab700 1 -- 172.21.15.1:0/1576576051 wait complete. 2024-03-18T18:50:42.255 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T18:50:42.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.385+0000 7fa87831b700 1 Processor -- start 2024-03-18T18:50:42.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.385+0000 7fa87831b700 1 -- start start 2024-03-18T18:50:42.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.385+0000 7fa87831b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa87006ca80 0x7fa87015d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:42.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.385+0000 7fa87831b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa87009a150 0x7fa8700cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:42.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.385+0000 7fa87831b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8700b3b00 con 0x7fa87015d5b0 2024-03-18T18:50:42.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.385+0000 7fa87831b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8700cbbb0 con 0x7fa87006ca80 2024-03-18T18:50:42.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.385+0000 7fa87831b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8700cf7f0 con 0x7fa87009a150 2024-03-18T18:50:42.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.386+0000 7fa8758b6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa87006ca80 0x7fa87015d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:42.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.386+0000 7fa8760b7700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa87015d5b0 0x7fa870099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37424/0 (socket says 172.21.15.1:37424) 2024-03-18T18:50:42.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.386+0000 7fa8758b6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa87006ca80 0x7fa87015d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49918/0 (socket says 172.21.15.1:49918) 2024-03-18T18:50:42.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.386+0000 7fa8760b7700 1 -- 172.21.15.1:0/4218748608 learned_addr learned my addr 172.21.15.1:0/4218748608 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:42.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.386+0000 7fa8768b8700 1 --2- 172.21.15.1:0/4218748608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa87009a150 0x7fa8700cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:42.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.386+0000 7fa8758b6700 1 -- 172.21.15.1:0/4218748608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa87009a150 msgr2=0x7fa8700cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:42.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.386+0000 7fa8758b6700 1 --2- 172.21.15.1:0/4218748608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa87009a150 0x7fa8700cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.386+0000 7fa8758b6700 1 -- 172.21.15.1:0/4218748608 >> v1:172.21.15.1:6789/0 conn(0x7fa87015d5b0 legacy=0x7fa870099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:42.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.386+0000 7fa8758b6700 1 -- 172.21.15.1:0/4218748608 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8700d04d0 con 0x7fa87006ca80 2024-03-18T18:50:42.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.386+0000 7fa8758b6700 1 --2- 172.21.15.1:0/4218748608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa87006ca80 0x7fa87015d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa86c009840 tx=0x7fa86c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84eb553ed8a7f373 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.386+0000 7fa8750b5700 1 -- 172.21.15.1:0/4218748608 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c011830 con 0x7fa87006ca80 2024-03-18T18:50:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.386+0000 7fa8750b5700 1 -- 172.21.15.1:0/4218748608 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa86c011990 con 0x7fa87006ca80 2024-03-18T18:50:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.386+0000 7fa8750b5700 1 -- 172.21.15.1:0/4218748608 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c011d20 con 0x7fa87006ca80 2024-03-18T18:50:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.387+0000 7fa87831b700 1 -- 172.21.15.1:0/4218748608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa87006ca80 msgr2=0x7fa87015d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.387+0000 7fa87831b700 1 --2- 172.21.15.1:0/4218748608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa87006ca80 0x7fa87015d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa86c009840 tx=0x7fa86c009b50 comp rx=0 tx=0).stop 2024-03-18T18:50:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.387+0000 7fa87831b700 1 -- 172.21.15.1:0/4218748608 shutdown_connections 2024-03-18T18:50:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.387+0000 7fa87831b700 1 --2- 172.21.15.1:0/4218748608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa87009a150 0x7fa8700cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.387+0000 7fa87831b700 1 --2- 172.21.15.1:0/4218748608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa87006ca80 0x7fa87015d990 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.387+0000 7fa87831b700 1 -- 172.21.15.1:0/4218748608 >> 172.21.15.1:0/4218748608 conn(0x7fa8700b2bd0 msgr2=0x7fa8700b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.387+0000 7fa87831b700 1 -- 172.21.15.1:0/4218748608 shutdown_connections 2024-03-18T18:50:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.387+0000 7fa87831b700 1 -- 172.21.15.1:0/4218748608 wait complete. 2024-03-18T18:50:42.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.388+0000 7fa87831b700 1 Processor -- start 2024-03-18T18:50:42.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.388+0000 7fa87831b700 1 -- start start 2024-03-18T18:50:42.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.389+0000 7fa87831b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa87006ca80 0x7fa8700c6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:42.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.389+0000 7fa87831b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87009a150 0x7fa870158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.389+0000 7fa87831b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa87015d5b0 0x7fa8700b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.389+0000 7fa87831b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8700c55f0 con 0x7fa87006ca80 2024-03-18T18:50:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.389+0000 7fa87831b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8700c58d0 con 0x7fa87009a150 2024-03-18T18:50:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.389+0000 7fa87831b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8700c5bb0 con 0x7fa87015d5b0 2024-03-18T18:50:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.389+0000 7fa8760b7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa87006ca80 0x7fa8700c6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.389+0000 7fa8760b7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa87006ca80 0x7fa8700c6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49924/0 (socket says 172.21.15.1:49924) 2024-03-18T18:50:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.389+0000 7fa8760b7700 1 -- 172.21.15.1:0/2020507162 learned_addr learned my addr 172.21.15.1:0/2020507162 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.389+0000 7fa8758b6700 1 --2- 172.21.15.1:0/2020507162 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87009a150 0x7fa870158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.389+0000 7fa8768b8700 1 --2- 172.21.15.1:0/2020507162 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa87015d5b0 0x7fa8700b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.389+0000 7fa8760b7700 1 -- 172.21.15.1:0/2020507162 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa87015d5b0 msgr2=0x7fa8700b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.389+0000 7fa8760b7700 1 --2- 172.21.15.1:0/2020507162 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa87015d5b0 0x7fa8700b57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.389+0000 7fa8760b7700 1 -- 172.21.15.1:0/2020507162 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87009a150 msgr2=0x7fa870158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.389+0000 7fa8760b7700 1 --2- 172.21.15.1:0/2020507162 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87009a150 0x7fa870158a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.389+0000 7fa8760b7700 1 -- 172.21.15.1:0/2020507162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86c0094a0 con 0x7fa87006ca80 2024-03-18T18:50:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.389+0000 7fa8760b7700 1 --2- 172.21.15.1:0/2020507162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa87006ca80 0x7fa8700c6690 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa86000c520 tx=0x7fa86000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.390+0000 7fa866ffd700 1 -- 172.21.15.1:0/2020507162 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa860005d70 con 0x7fa87006ca80 2024-03-18T18:50:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.390+0000 7fa866ffd700 1 -- 172.21.15.1:0/2020507162 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8600070d0 con 0x7fa87006ca80 2024-03-18T18:50:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.390+0000 7fa87831b700 1 -- 172.21.15.1:0/2020507162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8700c5e90 con 0x7fa87006ca80 2024-03-18T18:50:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.390+0000 7fa866ffd700 1 -- 172.21.15.1:0/2020507162 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa860007440 con 0x7fa87006ca80 2024-03-18T18:50:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.390+0000 7fa87831b700 1 -- 172.21.15.1:0/2020507162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa870158f50 con 0x7fa87006ca80 2024-03-18T18:50:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.391+0000 7fa87831b700 1 -- 172.21.15.1:0/2020507162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8700c6bd0 con 0x7fa87006ca80 2024-03-18T18:50:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.391+0000 7fa866ffd700 1 -- 172.21.15.1:0/2020507162 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa8600075a0 con 0x7fa87006ca80 2024-03-18T18:50:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.392+0000 7fa866ffd700 1 --2- 172.21.15.1:0/2020507162 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa85406ac90 0x7fa85406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.392+0000 7fa8758b6700 1 --2- 172.21.15.1:0/2020507162 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa85406ac90 0x7fa85406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.392+0000 7fa866ffd700 1 -- 172.21.15.1:0/2020507162 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa860088ae0 con 0x7fa87006ca80 2024-03-18T18:50:42.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.392+0000 7fa8758b6700 1 --2- 172.21.15.1:0/2020507162 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa85406ac90 0x7fa85406d150 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa86c009840 tx=0x7fa86c01c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:42.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.396+0000 7fa866ffd700 1 -- 172.21.15.1:0/2020507162 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa8600539b0 con 0x7fa87006ca80 2024-03-18T18:50:42.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.596+0000 7fa87831b700 1 -- 172.21.15.1:0/2020507162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa8700676a0 con 0x7fa87006ca80 2024-03-18T18:50:42.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.596+0000 7fa866ffd700 1 -- 172.21.15.1:0/2020507162 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa8600170a0 con 0x7fa87006ca80 2024-03-18T18:50:42.594 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:42.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.598+0000 7fa87831b700 1 -- 172.21.15.1:0/2020507162 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa85406ac90 msgr2=0x7fa85406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:42.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.598+0000 7fa87831b700 1 --2- 172.21.15.1:0/2020507162 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa85406ac90 0x7fa85406d150 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa86c009840 tx=0x7fa86c01c040 comp rx=0 tx=0).stop 2024-03-18T18:50:42.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.598+0000 7fa87831b700 1 -- 172.21.15.1:0/2020507162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa87006ca80 msgr2=0x7fa8700c6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:42.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.598+0000 7fa87831b700 1 --2- 172.21.15.1:0/2020507162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa87006ca80 0x7fa8700c6690 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa86000c520 tx=0x7fa86000c8e0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.598+0000 7fa87831b700 1 -- 172.21.15.1:0/2020507162 shutdown_connections 2024-03-18T18:50:42.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.598+0000 7fa87831b700 1 --2- 172.21.15.1:0/2020507162 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa85406ac90 0x7fa85406d150 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.598+0000 7fa87831b700 1 --2- 172.21.15.1:0/2020507162 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa87015d5b0 0x7fa8700b57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.598+0000 7fa87831b700 1 --2- 172.21.15.1:0/2020507162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa87006ca80 0x7fa8700c6690 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.598+0000 7fa87831b700 1 --2- 172.21.15.1:0/2020507162 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87009a150 0x7fa870158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.598+0000 7fa87831b700 1 -- 172.21.15.1:0/2020507162 >> 172.21.15.1:0/2020507162 conn(0x7fa8700b2bd0 msgr2=0x7fa870070030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:42.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.598+0000 7fa87831b700 1 -- 172.21.15.1:0/2020507162 shutdown_connections 2024-03-18T18:50:42.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.598+0000 7fa87831b700 1 -- 172.21.15.1:0/2020507162 wait complete. 2024-03-18T18:50:42.608 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T18:50:42.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.736+0000 7fae4939e700 1 Processor -- start 2024-03-18T18:50:42.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.737+0000 7fae4939e700 1 -- start start 2024-03-18T18:50:42.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.737+0000 7fae4939e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fae440b1e20 0x7fae440cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:42.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.737+0000 7fae4939e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae440d0df0 con 0x7fae440cbce0 2024-03-18T18:50:42.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.737+0000 7fae4939e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae440d0550 con 0x7fae440cd870 2024-03-18T18:50:42.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.737+0000 7fae4939e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae440d0830 con 0x7fae440b1e20 2024-03-18T18:50:42.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.737+0000 7fae427fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fae440b1e20 0x7fae440cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:42.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.738+0000 7fae427fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fae440b1e20 0x7fae440cff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49934/0 (socket says 172.21.15.1:49934) 2024-03-18T18:50:42.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.738+0000 7fae427fc700 1 -- 172.21.15.1:0/3616168567 learned_addr learned my addr 172.21.15.1:0/3616168567 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:42.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.738+0000 7fae437fe700 1 --1- 172.21.15.1:0/3616168567 >> v1:172.21.15.1:6789/0 conn(0x7fae440cbce0 0x7fae440cc150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37428/0 (socket says 172.21.15.1:37428) 2024-03-18T18:50:42.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.738+0000 7fae427fc700 1 -- 172.21.15.1:0/3616168567 >> v1:172.21.15.165:6789/0 conn(0x7fae440cd870 legacy=0x7fae440cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:42.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.738+0000 7fae427fc700 1 -- 172.21.15.1:0/3616168567 >> v1:172.21.15.1:6789/0 conn(0x7fae440cbce0 legacy=0x7fae440cc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:42.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.738+0000 7fae427fc700 1 -- 172.21.15.1:0/3616168567 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae440d0b10 con 0x7fae440b1e20 2024-03-18T18:50:42.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.738+0000 7fae427fc700 1 --2- 172.21.15.1:0/3616168567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae440b1e20 0x7fae440cff70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fae380098f0 tx=0x7fae38009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a879d9050deb9a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:42.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.738+0000 7fae41ffb700 1 -- 172.21.15.1:0/3616168567 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3800b8a0 con 0x7fae440b1e20 2024-03-18T18:50:42.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.738+0000 7fae41ffb700 1 -- 172.21.15.1:0/3616168567 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3800ba00 con 0x7fae440b1e20 2024-03-18T18:50:42.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.738+0000 7fae41ffb700 1 -- 172.21.15.1:0/3616168567 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3800bd90 con 0x7fae440b1e20 2024-03-18T18:50:42.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.739+0000 7fae4939e700 1 -- 172.21.15.1:0/3616168567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae440b1e20 msgr2=0x7fae440cff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:42.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.739+0000 7fae4939e700 1 --2- 172.21.15.1:0/3616168567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae440b1e20 0x7fae440cff70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fae380098f0 tx=0x7fae38009c00 comp rx=0 tx=0).stop 2024-03-18T18:50:42.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.739+0000 7fae4939e700 1 -- 172.21.15.1:0/3616168567 shutdown_connections 2024-03-18T18:50:42.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.739+0000 7fae4939e700 1 --2- 172.21.15.1:0/3616168567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae440b1e20 0x7fae440cff70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.739+0000 7fae4939e700 1 -- 172.21.15.1:0/3616168567 >> 172.21.15.1:0/3616168567 conn(0x7fae440d0ff0 msgr2=0x7fae440d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:42.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.739+0000 7fae4939e700 1 -- 172.21.15.1:0/3616168567 shutdown_connections 2024-03-18T18:50:42.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.739+0000 7fae4939e700 1 -- 172.21.15.1:0/3616168567 wait complete. 2024-03-18T18:50:42.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.740+0000 7fae4939e700 1 Processor -- start 2024-03-18T18:50:42.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.740+0000 7fae4939e700 1 -- start start 2024-03-18T18:50:42.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.741+0000 7fae4939e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae440b1e20 0x7fae440b6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:42.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.741+0000 7fae4939e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae440cbce0 0x7fae441e9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:42.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.741+0000 7fae4939e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae440cd870 0x7fae4414af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:42.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.741+0000 7fae4939e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae440b4be0 con 0x7fae440cd870 2024-03-18T18:50:42.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.741+0000 7fae4939e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae441f3950 con 0x7fae440b1e20 2024-03-18T18:50:42.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.741+0000 7fae4939e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae440b69b0 con 0x7fae440cbce0 2024-03-18T18:50:42.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.741+0000 7fae437fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae440cd870 0x7fae4414af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:42.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.741+0000 7fae437fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae440cd870 0x7fae4414af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49950/0 (socket says 172.21.15.1:49950) 2024-03-18T18:50:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.741+0000 7fae437fe700 1 -- 172.21.15.1:0/193061895 learned_addr learned my addr 172.21.15.1:0/193061895 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.741+0000 7fae42ffd700 1 --2- 172.21.15.1:0/193061895 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae440b1e20 0x7fae440b6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.741+0000 7fae427fc700 1 --2- 172.21.15.1:0/193061895 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae440cbce0 0x7fae441e9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.741+0000 7fae437fe700 1 -- 172.21.15.1:0/193061895 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae440cbce0 msgr2=0x7fae441e9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.741+0000 7fae437fe700 1 --2- 172.21.15.1:0/193061895 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae440cbce0 0x7fae441e9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.741+0000 7fae437fe700 1 -- 172.21.15.1:0/193061895 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae440b1e20 msgr2=0x7fae440b6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.741+0000 7fae437fe700 1 --2- 172.21.15.1:0/193061895 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae440b1e20 0x7fae440b6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.741+0000 7fae437fe700 1 -- 172.21.15.1:0/193061895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae38009550 con 0x7fae440cd870 2024-03-18T18:50:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.741+0000 7fae437fe700 1 --2- 172.21.15.1:0/193061895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae440cd870 0x7fae4414af00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00cbd0 tx=0x7fae3c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.741+0000 7fae2b7fe700 1 -- 172.21.15.1:0/193061895 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c00abb0 con 0x7fae440cd870 2024-03-18T18:50:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.742+0000 7fae2b7fe700 1 -- 172.21.15.1:0/193061895 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3c00ad10 con 0x7fae440cd870 2024-03-18T18:50:42.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.742+0000 7fae4939e700 1 -- 172.21.15.1:0/193061895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae440b4060 con 0x7fae440cd870 2024-03-18T18:50:42.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.742+0000 7fae2b7fe700 1 -- 172.21.15.1:0/193061895 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c007270 con 0x7fae440cd870 2024-03-18T18:50:42.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.742+0000 7fae4939e700 1 -- 172.21.15.1:0/193061895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae440b4620 con 0x7fae440cd870 2024-03-18T18:50:42.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.743+0000 7fae4939e700 1 -- 172.21.15.1:0/193061895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae441f3ac0 con 0x7fae440cd870 2024-03-18T18:50:42.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.743+0000 7fae2b7fe700 1 -- 172.21.15.1:0/193061895 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fae3c0073d0 con 0x7fae440cd870 2024-03-18T18:50:42.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.743+0000 7fae2b7fe700 1 --2- 172.21.15.1:0/193061895 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae3006ac90 0x7fae3006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:42.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.743+0000 7fae2b7fe700 1 -- 172.21.15.1:0/193061895 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fae3c089390 con 0x7fae440cd870 2024-03-18T18:50:42.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.743+0000 7fae42ffd700 1 --2- 172.21.15.1:0/193061895 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae3006ac90 0x7fae3006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:42.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.744+0000 7fae42ffd700 1 --2- 172.21.15.1:0/193061895 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae3006ac90 0x7fae3006d150 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fae34000f60 tx=0x7fae34009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:42.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.748+0000 7fae2b7fe700 1 -- 172.21.15.1:0/193061895 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fae3c0542f0 con 0x7fae440cd870 2024-03-18T18:50:42.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.964+0000 7fae4939e700 1 -- 172.21.15.1:0/193061895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fae440676a0 con 0x7fae440cd870 2024-03-18T18:50:42.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.964+0000 7fae2b7fe700 1 -- 172.21.15.1:0/193061895 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7fae3c057910 con 0x7fae440cd870 2024-03-18T18:50:42.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.966+0000 7fae4939e700 1 -- 172.21.15.1:0/193061895 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae3006ac90 msgr2=0x7fae3006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:42.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.966+0000 7fae4939e700 1 --2- 172.21.15.1:0/193061895 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae3006ac90 0x7fae3006d150 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fae34000f60 tx=0x7fae34009040 comp rx=0 tx=0).stop 2024-03-18T18:50:42.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.966+0000 7fae4939e700 1 -- 172.21.15.1:0/193061895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae440cd870 msgr2=0x7fae4414af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:42.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.966+0000 7fae4939e700 1 --2- 172.21.15.1:0/193061895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae440cd870 0x7fae4414af00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00cbd0 tx=0x7fae3c00cf90 comp rx=0 tx=0).stop 2024-03-18T18:50:42.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.966+0000 7fae4939e700 1 -- 172.21.15.1:0/193061895 shutdown_connections 2024-03-18T18:50:42.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.966+0000 7fae4939e700 1 --2- 172.21.15.1:0/193061895 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae3006ac90 0x7fae3006d150 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.966+0000 7fae4939e700 1 --2- 172.21.15.1:0/193061895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae440cd870 0x7fae4414af00 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.966+0000 7fae4939e700 1 --2- 172.21.15.1:0/193061895 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae440b1e20 0x7fae440b6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.966+0000 7fae4939e700 1 --2- 172.21.15.1:0/193061895 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae440cbce0 0x7fae441e9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:42.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.966+0000 7fae4939e700 1 -- 172.21.15.1:0/193061895 >> 172.21.15.1:0/193061895 conn(0x7fae440d0ff0 msgr2=0x7fae4406f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:42.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.967+0000 7fae4939e700 1 -- 172.21.15.1:0/193061895 shutdown_connections 2024-03-18T18:50:42.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:42.967+0000 7fae4939e700 1 -- 172.21.15.1:0/193061895 wait complete. 2024-03-18T18:50:42.965 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-18T18:50:42.976 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-18T18:50:42.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:42.976 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T18:50:43.012 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T18:50:43.013 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T18:50:43.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:43.013 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T18:50:43.071 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:50:43.071 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T18:50:43.094 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:50:43.094 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T18:50:43.157 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.286+0000 7f2f65fbf700 1 Processor -- start 2024-03-18T18:50:43.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.286+0000 7f2f65fbf700 1 -- start start 2024-03-18T18:50:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.287+0000 7f2f65fbf700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2f6006f240 0x7f2f60164ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.287+0000 7f2f65fbf700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f60144000 con 0x7f2f60143c20 2024-03-18T18:50:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.287+0000 7f2f65fbf700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f600cd8d0 con 0x7f2f600b4110 2024-03-18T18:50:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.287+0000 7f2f65fbf700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f600b51b0 con 0x7f2f6006f240 2024-03-18T18:50:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.287+0000 7f2f5f7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2f60143c20 0x7f2f600b3a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37432/0 (socket says 172.21.15.1:37432) 2024-03-18T18:50:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.287+0000 7f2f5f7fe700 1 -- 172.21.15.1:0/4204860308 learned_addr learned my addr 172.21.15.1:0/4204860308 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.287+0000 7f2f5ffff700 1 --2- 172.21.15.1:0/4204860308 >> v2:172.21.15.165:3300/0 conn(0x7f2f6006f240 0x7f2f60164ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.287+0000 7f2f5e7fc700 1 -- 172.21.15.1:0/4204860308 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485443270 0 0) 0x7f2f60144000 con 0x7f2f60143c20 2024-03-18T18:50:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.287+0000 7f2f5e7fc700 1 -- 172.21.15.1:0/4204860308 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f4c0035b0 con 0x7f2f60143c20 2024-03-18T18:50:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5e7fc700 1 -- 172.21.15.1:0/4204860308 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1097120442 0 0) 0x7f2f4c0035b0 con 0x7f2f60143c20 2024-03-18T18:50:43.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5e7fc700 1 -- 172.21.15.1:0/4204860308 >> v2:172.21.15.165:3300/0 conn(0x7f2f6006f240 msgr2=0x7f2f60164ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5e7fc700 1 --2- 172.21.15.1:0/4204860308 >> v2:172.21.15.165:3300/0 conn(0x7f2f6006f240 0x7f2f60164ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5e7fc700 1 -- 172.21.15.1:0/4204860308 >> v1:172.21.15.165:6789/0 conn(0x7f2f600b4110 legacy=0x7f2f6006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5e7fc700 1 -- 172.21.15.1:0/4204860308 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6006fdc0 con 0x7f2f60143c20 2024-03-18T18:50:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5e7fc700 1 -- 172.21.15.1:0/4204860308 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f2f50001670 con 0x7f2f60143c20 2024-03-18T18:50:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5e7fc700 1 -- 172.21.15.1:0/4204860308 >> v1:172.21.15.1:6789/0 conn(0x7f2f60143c20 legacy=0x7f2f600b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5e7fc700 1 --2- 172.21.15.1:0/4204860308 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f4c003a00 0x7f2f4c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5e7fc700 1 --2- 172.21.15.1:0/4204860308 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f4c004850 0x7f2f4c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5e7fc700 1 --2- 172.21.15.1:0/4204860308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f4c005210 0x7f2f4c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5e7fc700 1 -- 172.21.15.1:0/4204860308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c00c080 con 0x7f2f4c005210 2024-03-18T18:50:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5e7fc700 1 -- 172.21.15.1:0/4204860308 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c00c1f0 con 0x7f2f4c004850 2024-03-18T18:50:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5e7fc700 1 -- 172.21.15.1:0/4204860308 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c00c3b0 con 0x7f2f4c003a00 2024-03-18T18:50:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5ffff700 1 --2- 172.21.15.1:0/4204860308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f4c005210 0x7f2f4c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5effd700 1 --2- 172.21.15.1:0/4204860308 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f4c004850 0x7f2f4c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5f7fe700 1 --2- 172.21.15.1:0/4204860308 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f4c003a00 0x7f2f4c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5ffff700 1 -- 172.21.15.1:0/4204860308 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f4c003a00 msgr2=0x7f2f4c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5ffff700 1 --2- 172.21.15.1:0/4204860308 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f4c003a00 0x7f2f4c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5ffff700 1 -- 172.21.15.1:0/4204860308 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f4c004850 msgr2=0x7f2f4c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5ffff700 1 --2- 172.21.15.1:0/4204860308 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f4c004850 0x7f2f4c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5ffff700 1 -- 172.21.15.1:0/4204860308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f4c0035b0 con 0x7f2f4c005210 2024-03-18T18:50:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5f7fe700 1 --2- 172.21.15.1:0/4204860308 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f4c003a00 0x7f2f4c003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5ffff700 1 -- 172.21.15.1:0/4204860308 reap_dead start 2024-03-18T18:50:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.288+0000 7f2f5ffff700 1 --2- 172.21.15.1:0/4204860308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f4c005210 0x7f2f4c00ba10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2f58009fc0 tx=0x7f2f4c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ba943369c8a136 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.289+0000 7f2f5e7fc700 1 -- 172.21.15.1:0/4204860308 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f58003a20 con 0x7f2f4c005210 2024-03-18T18:50:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.289+0000 7f2f5e7fc700 1 -- 172.21.15.1:0/4204860308 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5800f040 con 0x7f2f4c005210 2024-03-18T18:50:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.289+0000 7f2f5e7fc700 1 -- 172.21.15.1:0/4204860308 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f58003de0 con 0x7f2f4c005210 2024-03-18T18:50:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.289+0000 7f2f65fbf700 1 -- 172.21.15.1:0/4204860308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f4c005210 msgr2=0x7f2f4c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.289+0000 7f2f65fbf700 1 --2- 172.21.15.1:0/4204860308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f4c005210 0x7f2f4c00ba10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2f58009fc0 tx=0x7f2f4c005880 comp rx=0 tx=0).stop 2024-03-18T18:50:43.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.289+0000 7f2f65fbf700 1 -- 172.21.15.1:0/4204860308 shutdown_connections 2024-03-18T18:50:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.289+0000 7f2f65fbf700 1 --2- 172.21.15.1:0/4204860308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f4c005210 0x7f2f4c00ba10 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.289+0000 7f2f65fbf700 1 -- 172.21.15.1:0/4204860308 >> 172.21.15.1:0/4204860308 conn(0x7f2f600b1d20 msgr2=0x7f2f600b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.289+0000 7f2f65fbf700 1 -- 172.21.15.1:0/4204860308 shutdown_connections 2024-03-18T18:50:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.289+0000 7f2f65fbf700 1 -- 172.21.15.1:0/4204860308 wait complete. 2024-03-18T18:50:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.291+0000 7f2f65fbf700 1 Processor -- start 2024-03-18T18:50:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.291+0000 7f2f65fbf700 1 -- start start 2024-03-18T18:50:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.292+0000 7f2f65fbf700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f4c005210 0x7f2f600c8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.292+0000 7f2f65fbf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f600ca5c0 0x7f2f601f49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.292+0000 7f2f65fbf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f601f4f00 0x7f2f601e3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.292+0000 7f2f65fbf700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f60132330 con 0x7f2f600ca5c0 2024-03-18T18:50:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.292+0000 7f2f65fbf700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f60131720 con 0x7f2f4c005210 2024-03-18T18:50:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.292+0000 7f2f65fbf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f60131a00 con 0x7f2f601f4f00 2024-03-18T18:50:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.292+0000 7f2f5effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f600ca5c0 0x7f2f601f49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.292+0000 7f2f5effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f600ca5c0 0x7f2f601f49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49960/0 (socket says 172.21.15.1:49960) 2024-03-18T18:50:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.292+0000 7f2f5effd700 1 -- 172.21.15.1:0/3317069144 learned_addr learned my addr 172.21.15.1:0/3317069144 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.292+0000 7f2f5f7fe700 1 --2- 172.21.15.1:0/3317069144 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f4c005210 0x7f2f600c8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.292+0000 7f2f5ffff700 1 --2- 172.21.15.1:0/3317069144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f601f4f00 0x7f2f601e3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.292+0000 7f2f5effd700 1 -- 172.21.15.1:0/3317069144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f601f4f00 msgr2=0x7f2f601e3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.292+0000 7f2f5effd700 1 --2- 172.21.15.1:0/3317069144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f601f4f00 0x7f2f601e3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.292+0000 7f2f5effd700 1 -- 172.21.15.1:0/3317069144 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f4c005210 msgr2=0x7f2f600c8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:43.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.292+0000 7f2f5effd700 1 --2- 172.21.15.1:0/3317069144 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f4c005210 0x7f2f600c8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.292+0000 7f2f5effd700 1 -- 172.21.15.1:0/3317069144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f60131ce0 con 0x7f2f600ca5c0 2024-03-18T18:50:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.292+0000 7f2f5effd700 1 --2- 172.21.15.1:0/3317069144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f600ca5c0 0x7f2f601f49c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2f4800f7f0 tx=0x7f2f4800fb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.293+0000 7f2f57fff700 1 -- 172.21.15.1:0/3317069144 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f480052e0 con 0x7f2f600ca5c0 2024-03-18T18:50:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.293+0000 7f2f57fff700 1 -- 172.21.15.1:0/3317069144 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f48005440 con 0x7f2f600ca5c0 2024-03-18T18:50:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.293+0000 7f2f65fbf700 1 -- 172.21.15.1:0/3317069144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f60131fc0 con 0x7f2f600ca5c0 2024-03-18T18:50:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.293+0000 7f2f57fff700 1 -- 172.21.15.1:0/3317069144 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f480038c0 con 0x7f2f600ca5c0 2024-03-18T18:50:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.293+0000 7f2f65fbf700 1 -- 172.21.15.1:0/3317069144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f600c8fb0 con 0x7f2f600ca5c0 2024-03-18T18:50:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.294+0000 7f2f65fbf700 1 -- 172.21.15.1:0/3317069144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f601324e0 con 0x7f2f600ca5c0 2024-03-18T18:50:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.294+0000 7f2f57fff700 1 -- 172.21.15.1:0/3317069144 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2f48003a20 con 0x7f2f600ca5c0 2024-03-18T18:50:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.294+0000 7f2f57fff700 1 --2- 172.21.15.1:0/3317069144 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2f4006ac90 0x7f2f4006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.295+0000 7f2f57fff700 1 -- 172.21.15.1:0/3317069144 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2f480892d0 con 0x7f2f600ca5c0 2024-03-18T18:50:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.295+0000 7f2f5f7fe700 1 --2- 172.21.15.1:0/3317069144 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2f4006ac90 0x7f2f4006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.295+0000 7f2f5f7fe700 1 --2- 172.21.15.1:0/3317069144 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2f4006ac90 0x7f2f4006d150 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2f50005b00 tx=0x7f2f50007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:43.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.299+0000 7f2f57fff700 1 -- 172.21.15.1:0/3317069144 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2f48054180 con 0x7f2f600ca5c0 2024-03-18T18:50:43.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.504+0000 7f2f65fbf700 1 -- 172.21.15.1:0/3317069144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f600676a0 con 0x7f2f600ca5c0 2024-03-18T18:50:43.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.505+0000 7f2f57fff700 1 -- 172.21.15.1:0/3317069144 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18217 (secure 0 0 0) 0x7f2f480577a0 con 0x7f2f600ca5c0 2024-03-18T18:50:43.503 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:43.503 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":27,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:50:34.993482+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:43.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.507+0000 7f2f65fbf700 1 -- 172.21.15.1:0/3317069144 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2f4006ac90 msgr2=0x7f2f4006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:43.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.507+0000 7f2f65fbf700 1 --2- 172.21.15.1:0/3317069144 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2f4006ac90 0x7f2f4006d150 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2f50005b00 tx=0x7f2f50007040 comp rx=0 tx=0).stop 2024-03-18T18:50:43.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.507+0000 7f2f65fbf700 1 -- 172.21.15.1:0/3317069144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f600ca5c0 msgr2=0x7f2f601f49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:43.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.507+0000 7f2f65fbf700 1 --2- 172.21.15.1:0/3317069144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f600ca5c0 0x7f2f601f49c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2f4800f7f0 tx=0x7f2f4800fb00 comp rx=0 tx=0).stop 2024-03-18T18:50:43.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.507+0000 7f2f65fbf700 1 -- 172.21.15.1:0/3317069144 shutdown_connections 2024-03-18T18:50:43.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.507+0000 7f2f65fbf700 1 --2- 172.21.15.1:0/3317069144 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2f4006ac90 0x7f2f4006d150 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.507+0000 7f2f65fbf700 1 --2- 172.21.15.1:0/3317069144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f601f4f00 0x7f2f601e3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.507+0000 7f2f65fbf700 1 --2- 172.21.15.1:0/3317069144 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f4c005210 0x7f2f600c8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.507+0000 7f2f65fbf700 1 --2- 172.21.15.1:0/3317069144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f600ca5c0 0x7f2f601f49c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.507+0000 7f2f65fbf700 1 -- 172.21.15.1:0/3317069144 >> 172.21.15.1:0/3317069144 conn(0x7f2f600b1d20 msgr2=0x7f2f60140850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:43.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.507+0000 7f2f65fbf700 1 -- 172.21.15.1:0/3317069144 shutdown_connections 2024-03-18T18:50:43.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.507+0000 7f2f65fbf700 1 -- 172.21.15.1:0/3317069144 wait complete. 2024-03-18T18:50:43.517 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T18:49:42.800801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T18:49:43.776684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T18:49:46.854608+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T18:50:43.518 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T18:50:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.648+0000 7ff469194700 1 Processor -- start 2024-03-18T18:50:43.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.648+0000 7ff469194700 1 -- start start 2024-03-18T18:50:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.648+0000 7ff469194700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff46415a240 0x7ff4640ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:43.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.648+0000 7ff469194700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff46406d560 0x7ff46406fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.648+0000 7ff469194700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff464070000 con 0x7ff4640715b0 2024-03-18T18:50:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.648+0000 7ff469194700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4640adff0 con 0x7ff46415a240 2024-03-18T18:50:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.648+0000 7ff469194700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4640cac80 con 0x7ff46406d560 2024-03-18T18:50:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.649+0000 7ff462d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff46415a240 0x7ff4640ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.649+0000 7ff462d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff46415a240 0x7ff4640ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49976/0 (socket says 172.21.15.1:49976) 2024-03-18T18:50:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.649+0000 7ff462d9d700 1 -- 172.21.15.1:0/360780155 learned_addr learned my addr 172.21.15.1:0/360780155 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.649+0000 7ff462d9d700 1 -- 172.21.15.1:0/360780155 >> v2:172.21.15.165:3300/0 conn(0x7ff46406d560 msgr2=0x7ff46406fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.649+0000 7ff46259c700 1 --2- 172.21.15.1:0/360780155 >> v2:172.21.15.165:3300/0 conn(0x7ff46406d560 0x7ff46406fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.649+0000 7ff462d9d700 1 --2- 172.21.15.1:0/360780155 >> v2:172.21.15.165:3300/0 conn(0x7ff46406d560 0x7ff46406fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.649+0000 7ff462d9d700 1 -- 172.21.15.1:0/360780155 >> v1:172.21.15.165:6789/0 conn(0x7ff4640715b0 legacy=0x7ff46415de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.649+0000 7ff462d9d700 1 -- 172.21.15.1:0/360780155 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4640caf60 con 0x7ff46415a240 2024-03-18T18:50:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.649+0000 7ff46259c700 1 --2- 172.21.15.1:0/360780155 >> v2:172.21.15.165:3300/0 conn(0x7ff46406d560 0x7ff46406fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.649+0000 7ff462d9d700 1 --2- 172.21.15.1:0/360780155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff46415a240 0x7ff4640ca580 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff45400bb90 tx=0x7ff45400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e05e2bb90bc1ba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:43.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.649+0000 7ff461d9b700 1 -- 172.21.15.1:0/360780155 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45400dd10 con 0x7ff46415a240 2024-03-18T18:50:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.649+0000 7ff461d9b700 1 -- 172.21.15.1:0/360780155 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff45400de70 con 0x7ff46415a240 2024-03-18T18:50:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.649+0000 7ff461d9b700 1 -- 172.21.15.1:0/360780155 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff454004230 con 0x7ff46415a240 2024-03-18T18:50:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.650+0000 7ff469194700 1 -- 172.21.15.1:0/360780155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff46415a240 msgr2=0x7ff4640ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.650+0000 7ff469194700 1 --2- 172.21.15.1:0/360780155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff46415a240 0x7ff4640ca580 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff45400bb90 tx=0x7ff45400bea0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.650+0000 7ff469194700 1 -- 172.21.15.1:0/360780155 shutdown_connections 2024-03-18T18:50:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.650+0000 7ff469194700 1 --2- 172.21.15.1:0/360780155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff46415a240 0x7ff4640ca580 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.650+0000 7ff469194700 1 --2- 172.21.15.1:0/360780155 >> v2:172.21.15.165:3300/0 conn(0x7ff46406d560 0x7ff46406fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.650+0000 7ff469194700 1 -- 172.21.15.1:0/360780155 >> 172.21.15.1:0/360780155 conn(0x7ff4640cff30 msgr2=0x7ff4640d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.650+0000 7ff469194700 1 -- 172.21.15.1:0/360780155 shutdown_connections 2024-03-18T18:50:43.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.650+0000 7ff469194700 1 -- 172.21.15.1:0/360780155 wait complete. 2024-03-18T18:50:43.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.651+0000 7ff469194700 1 Processor -- start 2024-03-18T18:50:43.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.651+0000 7ff469194700 1 -- start start 2024-03-18T18:50:43.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.652+0000 7ff469194700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff46406d560 0x7ff4640c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:43.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.652+0000 7ff469194700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4640715b0 0x7ff46414b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:43.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.652+0000 7ff469194700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff46415a240 0x7ff4640b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:43.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.652+0000 7ff469194700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4640b4110 con 0x7ff4640715b0 2024-03-18T18:50:43.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.652+0000 7ff469194700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4640c43b0 con 0x7ff46406d560 2024-03-18T18:50:43.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.652+0000 7ff469194700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4640c6700 con 0x7ff46415a240 2024-03-18T18:50:43.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.652+0000 7ff46259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4640715b0 0x7ff46414b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:43.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.652+0000 7ff46259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4640715b0 0x7ff46414b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49984/0 (socket says 172.21.15.1:49984) 2024-03-18T18:50:43.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.652+0000 7ff46259c700 1 -- 172.21.15.1:0/1512463733 learned_addr learned my addr 172.21.15.1:0/1512463733 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:43.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.652+0000 7ff462d9d700 1 --2- 172.21.15.1:0/1512463733 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff46406d560 0x7ff4640c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:43.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.652+0000 7ff46359e700 1 --2- 172.21.15.1:0/1512463733 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff46415a240 0x7ff4640b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:43.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.652+0000 7ff46259c700 1 -- 172.21.15.1:0/1512463733 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff46415a240 msgr2=0x7ff4640b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:43.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.652+0000 7ff46259c700 1 --2- 172.21.15.1:0/1512463733 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff46415a240 0x7ff4640b4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.652+0000 7ff46259c700 1 -- 172.21.15.1:0/1512463733 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff46406d560 msgr2=0x7ff4640c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:43.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.652+0000 7ff46259c700 1 --2- 172.21.15.1:0/1512463733 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff46406d560 0x7ff4640c61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.653+0000 7ff46259c700 1 -- 172.21.15.1:0/1512463733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45400b7f0 con 0x7ff4640715b0 2024-03-18T18:50:43.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.653+0000 7ff46259c700 1 --2- 172.21.15.1:0/1512463733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4640715b0 0x7ff46414b250 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff45800c740 tx=0x7ff45800ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:43.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.653+0000 7ff44b7fe700 1 -- 172.21.15.1:0/1512463733 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff458004220 con 0x7ff4640715b0 2024-03-18T18:50:43.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.653+0000 7ff44b7fe700 1 -- 172.21.15.1:0/1512463733 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff458004380 con 0x7ff4640715b0 2024-03-18T18:50:43.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.653+0000 7ff469194700 1 -- 172.21.15.1:0/1512463733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4640c5130 con 0x7ff4640715b0 2024-03-18T18:50:43.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.653+0000 7ff44b7fe700 1 -- 172.21.15.1:0/1512463733 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4580036c0 con 0x7ff4640715b0 2024-03-18T18:50:43.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.653+0000 7ff469194700 1 -- 172.21.15.1:0/1512463733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4640c4930 con 0x7ff4640715b0 2024-03-18T18:50:43.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.654+0000 7ff469194700 1 -- 172.21.15.1:0/1512463733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4640c6b40 con 0x7ff4640715b0 2024-03-18T18:50:43.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.654+0000 7ff44b7fe700 1 -- 172.21.15.1:0/1512463733 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff458003820 con 0x7ff4640715b0 2024-03-18T18:50:43.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.655+0000 7ff44b7fe700 1 --2- 172.21.15.1:0/1512463733 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff45006ac90 0x7ff45006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:43.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.655+0000 7ff44b7fe700 1 -- 172.21.15.1:0/1512463733 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff4580898f0 con 0x7ff4640715b0 2024-03-18T18:50:43.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.655+0000 7ff462d9d700 1 --2- 172.21.15.1:0/1512463733 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff45006ac90 0x7ff45006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:43.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.655+0000 7ff462d9d700 1 --2- 172.21.15.1:0/1512463733 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff45006ac90 0x7ff45006d150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff45400b340 tx=0x7ff45401d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:43.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.659+0000 7ff44b7fe700 1 -- 172.21.15.1:0/1512463733 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff458054710 con 0x7ff4640715b0 2024-03-18T18:50:43.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.861+0000 7ff469194700 1 -- 172.21.15.1:0/1512463733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff4640676a0 con 0x7ff4640715b0 2024-03-18T18:50:43.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.861+0000 7ff44b7fe700 1 -- 172.21.15.1:0/1512463733 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff458057d30 con 0x7ff4640715b0 2024-03-18T18:50:43.859 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T18:50:43.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.864+0000 7ff469194700 1 -- 172.21.15.1:0/1512463733 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff45006ac90 msgr2=0x7ff45006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:43.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.864+0000 7ff469194700 1 --2- 172.21.15.1:0/1512463733 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff45006ac90 0x7ff45006d150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff45400b340 tx=0x7ff45401d040 comp rx=0 tx=0).stop 2024-03-18T18:50:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.864+0000 7ff469194700 1 -- 172.21.15.1:0/1512463733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4640715b0 msgr2=0x7ff46414b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.864+0000 7ff469194700 1 --2- 172.21.15.1:0/1512463733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4640715b0 0x7ff46414b250 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff45800c740 tx=0x7ff45800ca50 comp rx=0 tx=0).stop 2024-03-18T18:50:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.864+0000 7ff469194700 1 -- 172.21.15.1:0/1512463733 shutdown_connections 2024-03-18T18:50:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.864+0000 7ff469194700 1 --2- 172.21.15.1:0/1512463733 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff45006ac90 0x7ff45006d150 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.864+0000 7ff469194700 1 --2- 172.21.15.1:0/1512463733 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff46415a240 0x7ff4640b4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.864+0000 7ff469194700 1 --2- 172.21.15.1:0/1512463733 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff46406d560 0x7ff4640c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.864+0000 7ff469194700 1 --2- 172.21.15.1:0/1512463733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4640715b0 0x7ff46414b250 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.864+0000 7ff469194700 1 -- 172.21.15.1:0/1512463733 >> 172.21.15.1:0/1512463733 conn(0x7ff4640cff30 msgr2=0x7ff4640b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.864+0000 7ff469194700 1 -- 172.21.15.1:0/1512463733 shutdown_connections 2024-03-18T18:50:43.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:43.864+0000 7ff469194700 1 -- 172.21.15.1:0/1512463733 wait complete. 2024-03-18T18:50:43.873 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T18:50:44.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.004+0000 7fb9c7a97700 1 Processor -- start 2024-03-18T18:50:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.004+0000 7fb9c7a97700 1 -- start start 2024-03-18T18:50:44.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.005+0000 7fb9c7a97700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb9c015d3b0 0x7fb9c006cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:44.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.005+0000 7fb9c7a97700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb9c0071550 0x7fb9c00b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:44.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.005+0000 7fb9c7a97700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c00b2b80 con 0x7fb9c006d430 2024-03-18T18:50:44.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.005+0000 7fb9c7a97700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c00b3800 con 0x7fb9c0071550 2024-03-18T18:50:44.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.005+0000 7fb9c7a97700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c00b2e60 con 0x7fb9c015d3b0 2024-03-18T18:50:44.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.005+0000 7fb9c6034700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb9c0071550 0x7fb9c00b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:44.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.005+0000 7fb9c6034700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb9c0071550 0x7fb9c00b3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50000/0 (socket says 172.21.15.1:50000) 2024-03-18T18:50:44.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.005+0000 7fb9c6034700 1 -- 172.21.15.1:0/209622960 learned_addr learned my addr 172.21.15.1:0/209622960 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:44.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.005+0000 7fb9c5833700 1 --2- 172.21.15.1:0/209622960 >> v2:172.21.15.165:3300/0 conn(0x7fb9c015d3b0 0x7fb9c006cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:44.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.005+0000 7fb9c6034700 1 -- 172.21.15.1:0/209622960 >> v2:172.21.15.165:3300/0 conn(0x7fb9c015d3b0 msgr2=0x7fb9c006cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:44.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.005+0000 7fb9c6034700 1 --2- 172.21.15.1:0/209622960 >> v2:172.21.15.165:3300/0 conn(0x7fb9c015d3b0 0x7fb9c006cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.005+0000 7fb9c6034700 1 -- 172.21.15.1:0/209622960 >> v1:172.21.15.165:6789/0 conn(0x7fb9c006d430 legacy=0x7fb9c0070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:44.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.005+0000 7fb9c6034700 1 -- 172.21.15.1:0/209622960 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c00b3140 con 0x7fb9c0071550 2024-03-18T18:50:44.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.005+0000 7fb9c6034700 1 --2- 172.21.15.1:0/209622960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9c0071550 0x7fb9c00b3c20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00d0810 tx=0x7fb9b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f350d07ab00ebea0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:44.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.005+0000 7fb9c4831700 1 -- 172.21.15.1:0/209622960 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b800e070 con 0x7fb9c0071550 2024-03-18T18:50:44.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.005+0000 7fb9c4831700 1 -- 172.21.15.1:0/209622960 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b800b910 con 0x7fb9c0071550 2024-03-18T18:50:44.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.006+0000 7fb9c4831700 1 -- 172.21.15.1:0/209622960 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b800bcc0 con 0x7fb9c0071550 2024-03-18T18:50:44.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.006+0000 7fb9c7a97700 1 -- 172.21.15.1:0/209622960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9c0071550 msgr2=0x7fb9c00b3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:44.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.006+0000 7fb9c7a97700 1 --2- 172.21.15.1:0/209622960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9c0071550 0x7fb9c00b3c20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00d0810 tx=0x7fb9b800b330 comp rx=0 tx=0).stop 2024-03-18T18:50:44.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.006+0000 7fb9c7a97700 1 -- 172.21.15.1:0/209622960 shutdown_connections 2024-03-18T18:50:44.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.006+0000 7fb9c7a97700 1 --2- 172.21.15.1:0/209622960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9c0071550 0x7fb9c00b3c20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.006+0000 7fb9c7a97700 1 --2- 172.21.15.1:0/209622960 >> v2:172.21.15.165:3300/0 conn(0x7fb9c015d3b0 0x7fb9c006cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.006+0000 7fb9c7a97700 1 -- 172.21.15.1:0/209622960 >> 172.21.15.1:0/209622960 conn(0x7fb9c00b1350 msgr2=0x7fb9c00b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:44.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.006+0000 7fb9c7a97700 1 -- 172.21.15.1:0/209622960 shutdown_connections 2024-03-18T18:50:44.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.006+0000 7fb9c7a97700 1 -- 172.21.15.1:0/209622960 wait complete. 2024-03-18T18:50:44.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.007+0000 7fb9c7a97700 1 Processor -- start 2024-03-18T18:50:44.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.007+0000 7fb9c7a97700 1 -- start start 2024-03-18T18:50:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.008+0000 7fb9c7a97700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9c006d430 0x7fb9c01594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.008+0000 7fb9c7a97700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9c0071550 0x7fb9c01357e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.008+0000 7fb9c7a97700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9c015d3b0 0x7fb9c01538e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.008+0000 7fb9c7a97700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c0159a00 con 0x7fb9c0071550 2024-03-18T18:50:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.008+0000 7fb9c7a97700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c01362b0 con 0x7fb9c015d3b0 2024-03-18T18:50:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.008+0000 7fb9c7a97700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c0136590 con 0x7fb9c006d430 2024-03-18T18:50:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.008+0000 7fb9c5032700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9c0071550 0x7fb9c01357e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:44.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.008+0000 7fb9c5032700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9c0071550 0x7fb9c01357e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50014/0 (socket says 172.21.15.1:50014) 2024-03-18T18:50:44.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.008+0000 7fb9c5032700 1 -- 172.21.15.1:0/2645328729 learned_addr learned my addr 172.21.15.1:0/2645328729 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:44.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.008+0000 7fb9c6034700 1 --2- 172.21.15.1:0/2645328729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9c015d3b0 0x7fb9c01538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:44.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.008+0000 7fb9c5833700 1 --2- 172.21.15.1:0/2645328729 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9c006d430 0x7fb9c01594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:44.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.008+0000 7fb9c5032700 1 -- 172.21.15.1:0/2645328729 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9c006d430 msgr2=0x7fb9c01594c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:44.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.008+0000 7fb9c5032700 1 --2- 172.21.15.1:0/2645328729 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9c006d430 0x7fb9c01594c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.008+0000 7fb9c5032700 1 -- 172.21.15.1:0/2645328729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9c015d3b0 msgr2=0x7fb9c01538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:44.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.008+0000 7fb9c5032700 1 --2- 172.21.15.1:0/2645328729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9c015d3b0 0x7fb9c01538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.009+0000 7fb9c5032700 1 -- 172.21.15.1:0/2645328729 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b8009b30 con 0x7fb9c0071550 2024-03-18T18:50:44.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.009+0000 7fb9c5032700 1 --2- 172.21.15.1:0/2645328729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9c0071550 0x7fb9c01357e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb9b000c4e0 tx=0x7fb9b000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:44.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.009+0000 7fb9b67fc700 1 -- 172.21.15.1:0/2645328729 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b00052d0 con 0x7fb9c0071550 2024-03-18T18:50:44.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.009+0000 7fb9b67fc700 1 -- 172.21.15.1:0/2645328729 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b0005430 con 0x7fb9c0071550 2024-03-18T18:50:44.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.009+0000 7fb9c7a97700 1 -- 172.21.15.1:0/2645328729 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c0136870 con 0x7fb9c0071550 2024-03-18T18:50:44.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.009+0000 7fb9b67fc700 1 -- 172.21.15.1:0/2645328729 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b00038c0 con 0x7fb9c0071550 2024-03-18T18:50:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.009+0000 7fb9c7a97700 1 -- 172.21.15.1:0/2645328729 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c0135d20 con 0x7fb9c0071550 2024-03-18T18:50:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.010+0000 7fb9c7a97700 1 -- 172.21.15.1:0/2645328729 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c0136d90 con 0x7fb9c0071550 2024-03-18T18:50:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.010+0000 7fb9b67fc700 1 -- 172.21.15.1:0/2645328729 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb9b0003a20 con 0x7fb9c0071550 2024-03-18T18:50:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.011+0000 7fb9b67fc700 1 --2- 172.21.15.1:0/2645328729 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb9a406ac90 0x7fb9a406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.011+0000 7fb9b67fc700 1 -- 172.21.15.1:0/2645328729 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fb9b00889a0 con 0x7fb9c0071550 2024-03-18T18:50:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.011+0000 7fb9c5833700 1 --2- 172.21.15.1:0/2645328729 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb9a406ac90 0x7fb9a406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:44.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.011+0000 7fb9c5833700 1 --2- 172.21.15.1:0/2645328729 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb9a406ac90 0x7fb9a406d150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc009b40 tx=0x7fb9bc009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:44.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.015+0000 7fb9b67fc700 1 -- 172.21.15.1:0/2645328729 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb9c0136d90 con 0x7fb9c0071550 2024-03-18T18:50:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.215+0000 7fb9c7a97700 1 -- 172.21.15.1:0/2645328729 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb9c0136d90 con 0x7fb9c0071550 2024-03-18T18:50:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.215+0000 7fb9b67fc700 1 -- 172.21.15.1:0/2645328729 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb9c0136d90 con 0x7fb9c0071550 2024-03-18T18:50:44.213 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:44.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.217+0000 7fb9c7a97700 1 -- 172.21.15.1:0/2645328729 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb9a406ac90 msgr2=0x7fb9a406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:44.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.217+0000 7fb9c7a97700 1 --2- 172.21.15.1:0/2645328729 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb9a406ac90 0x7fb9a406d150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc009b40 tx=0x7fb9bc009210 comp rx=0 tx=0).stop 2024-03-18T18:50:44.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.217+0000 7fb9c7a97700 1 -- 172.21.15.1:0/2645328729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9c0071550 msgr2=0x7fb9c01357e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:44.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.217+0000 7fb9c7a97700 1 --2- 172.21.15.1:0/2645328729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9c0071550 0x7fb9c01357e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb9b000c4e0 tx=0x7fb9b000c7f0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.217+0000 7fb9c7a97700 1 -- 172.21.15.1:0/2645328729 shutdown_connections 2024-03-18T18:50:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.217+0000 7fb9c7a97700 1 --2- 172.21.15.1:0/2645328729 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb9a406ac90 0x7fb9a406d150 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.217+0000 7fb9c7a97700 1 --2- 172.21.15.1:0/2645328729 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9c006d430 0x7fb9c01594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.217+0000 7fb9c7a97700 1 --2- 172.21.15.1:0/2645328729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9c015d3b0 0x7fb9c01538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.217+0000 7fb9c7a97700 1 --2- 172.21.15.1:0/2645328729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9c0071550 0x7fb9c01357e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.217+0000 7fb9c7a97700 1 -- 172.21.15.1:0/2645328729 >> 172.21.15.1:0/2645328729 conn(0x7fb9c00b1350 msgr2=0x7fb9c01601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.217+0000 7fb9c7a97700 1 -- 172.21.15.1:0/2645328729 shutdown_connections 2024-03-18T18:50:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.218+0000 7fb9c7a97700 1 -- 172.21.15.1:0/2645328729 wait complete. 2024-03-18T18:50:44.227 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T18:50:44.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.358+0000 7f92e3966700 1 Processor -- start 2024-03-18T18:50:44.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.358+0000 7f92e3966700 1 -- start start 2024-03-18T18:50:44.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.359+0000 7f92e3966700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f92dc071d30 0x7f92dc0add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:44.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.359+0000 7f92e3966700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92dc0cb9e0 0x7f92dc0cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:44.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.359+0000 7f92e3966700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92dc0ce8f0 con 0x7f92dc15a1d0 2024-03-18T18:50:44.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.359+0000 7f92e3966700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92dc0cdb00 con 0x7f92dc071d30 2024-03-18T18:50:44.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.359+0000 7f92e3966700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc15a5b0 con 0x7f92dc0cb9e0 2024-03-18T18:50:44.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.359+0000 7f92e0f01700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f92dc071d30 0x7f92dc0add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:44.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.359+0000 7f92e1f03700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92dc0cb9e0 0x7f92dc0cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:44.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.359+0000 7f92e1702700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f92dc15a1d0 0x7f92dc071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:55470/0 (socket says 172.21.15.1:55470) 2024-03-18T18:50:44.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.359+0000 7f92e1702700 1 -- 172.21.15.1:0/389945045 learned_addr learned my addr 172.21.15.1:0/389945045 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:44.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.360+0000 7f92e0f01700 1 -- 172.21.15.1:0/389945045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92dc0cb9e0 msgr2=0x7f92dc0cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:44.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.360+0000 7f92e0f01700 1 --2- 172.21.15.1:0/389945045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92dc0cb9e0 0x7f92dc0cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.360+0000 7f92e0f01700 1 -- 172.21.15.1:0/389945045 >> v1:172.21.15.165:6789/0 conn(0x7f92dc15a1d0 legacy=0x7f92dc071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:44.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.360+0000 7f92e0f01700 1 -- 172.21.15.1:0/389945045 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92dc0ae260 con 0x7f92dc071d30 2024-03-18T18:50:44.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.360+0000 7f92e1f03700 1 --2- 172.21.15.1:0/389945045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92dc0cb9e0 0x7f92dc0cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:50:44.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.360+0000 7f92e0f01700 1 --2- 172.21.15.1:0/389945045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92dc071d30 0x7f92dc0add20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f92cc009840 tx=0x7f92cc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f778fcb8f24f8875 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:44.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.360+0000 7f92d3fff700 1 -- 172.21.15.1:0/389945045 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc011830 con 0x7f92dc071d30 2024-03-18T18:50:44.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.360+0000 7f92d3fff700 1 -- 172.21.15.1:0/389945045 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92cc011990 con 0x7f92dc071d30 2024-03-18T18:50:44.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.360+0000 7f92d3fff700 1 -- 172.21.15.1:0/389945045 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc011d20 con 0x7f92dc071d30 2024-03-18T18:50:44.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.361+0000 7f92e3966700 1 -- 172.21.15.1:0/389945045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92dc071d30 msgr2=0x7f92dc0add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:44.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.361+0000 7f92e3966700 1 --2- 172.21.15.1:0/389945045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92dc071d30 0x7f92dc0add20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f92cc009840 tx=0x7f92cc009b50 comp rx=0 tx=0).stop 2024-03-18T18:50:44.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.361+0000 7f92e3966700 1 -- 172.21.15.1:0/389945045 shutdown_connections 2024-03-18T18:50:44.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.361+0000 7f92e3966700 1 --2- 172.21.15.1:0/389945045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92dc0cb9e0 0x7f92dc0cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.361+0000 7f92e3966700 1 --2- 172.21.15.1:0/389945045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92dc071d30 0x7f92dc0add20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.361+0000 7f92e3966700 1 -- 172.21.15.1:0/389945045 >> 172.21.15.1:0/389945045 conn(0x7f92dc0cc9b0 msgr2=0x7f92dc0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:44.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.361+0000 7f92e3966700 1 -- 172.21.15.1:0/389945045 shutdown_connections 2024-03-18T18:50:44.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.361+0000 7f92e3966700 1 -- 172.21.15.1:0/389945045 wait complete. 2024-03-18T18:50:44.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.362+0000 7f92e3966700 1 Processor -- start 2024-03-18T18:50:44.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.362+0000 7f92e3966700 1 -- start start 2024-03-18T18:50:44.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.363+0000 7f92e3966700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92dc071d30 0x7f92dc0c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:44.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.363+0000 7f92e3966700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92dc0cb9e0 0x7f92dc153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:44.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.363+0000 7f92e3966700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92dc15a1d0 0x7f92dc0c5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:44.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.363+0000 7f92e3966700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc1537a0 con 0x7f92dc071d30 2024-03-18T18:50:44.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.363+0000 7f92e3966700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc0c4f00 con 0x7f92dc0cb9e0 2024-03-18T18:50:44.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.363+0000 7f92e3966700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc0c51e0 con 0x7f92dc15a1d0 2024-03-18T18:50:44.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.363+0000 7f92e1702700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92dc071d30 0x7f92dc0c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:44.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.363+0000 7f92e1702700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92dc071d30 0x7f92dc0c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59116/0 (socket says 172.21.15.1:59116) 2024-03-18T18:50:44.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.363+0000 7f92e1702700 1 -- 172.21.15.1:0/3982231599 learned_addr learned my addr 172.21.15.1:0/3982231599 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:44.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.363+0000 7f92e0f01700 1 --2- 172.21.15.1:0/3982231599 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92dc0cb9e0 0x7f92dc153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:44.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.363+0000 7f92e1f03700 1 --2- 172.21.15.1:0/3982231599 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92dc15a1d0 0x7f92dc0c5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:44.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.363+0000 7f92e1702700 1 -- 172.21.15.1:0/3982231599 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92dc15a1d0 msgr2=0x7f92dc0c5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:44.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.363+0000 7f92e1702700 1 --2- 172.21.15.1:0/3982231599 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92dc15a1d0 0x7f92dc0c5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.363+0000 7f92e1702700 1 -- 172.21.15.1:0/3982231599 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92dc0cb9e0 msgr2=0x7f92dc153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:44.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.363+0000 7f92e1702700 1 --2- 172.21.15.1:0/3982231599 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92dc0cb9e0 0x7f92dc153260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.363+0000 7f92e1702700 1 -- 172.21.15.1:0/3982231599 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d4009a70 con 0x7f92dc071d30 2024-03-18T18:50:44.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.363+0000 7f92e1f03700 1 --2- 172.21.15.1:0/3982231599 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92dc15a1d0 0x7f92dc0c5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:44.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.363+0000 7f92e1702700 1 --2- 172.21.15.1:0/3982231599 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92dc071d30 0x7f92dc0c8170 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f92d800c500 tx=0x7f92d800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:44.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.363+0000 7f92d1ffb700 1 -- 172.21.15.1:0/3982231599 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d8005dc0 con 0x7f92dc071d30 2024-03-18T18:50:44.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.364+0000 7f92d1ffb700 1 -- 172.21.15.1:0/3982231599 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d80070d0 con 0x7f92dc071d30 2024-03-18T18:50:44.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.364+0000 7f92e3966700 1 -- 172.21.15.1:0/3982231599 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92cc0094a0 con 0x7f92dc071d30 2024-03-18T18:50:44.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.364+0000 7f92d1ffb700 1 -- 172.21.15.1:0/3982231599 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d8007460 con 0x7f92dc071d30 2024-03-18T18:50:44.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.364+0000 7f92e3966700 1 -- 172.21.15.1:0/3982231599 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92dc0c57a0 con 0x7f92dc071d30 2024-03-18T18:50:44.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.365+0000 7f92e3966700 1 -- 172.21.15.1:0/3982231599 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92dc0676a0 con 0x7f92dc071d30 2024-03-18T18:50:44.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.365+0000 7f92d1ffb700 1 -- 172.21.15.1:0/3982231599 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f92d80075c0 con 0x7f92dc071d30 2024-03-18T18:50:44.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.365+0000 7f92d1ffb700 1 --2- 172.21.15.1:0/3982231599 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f92c006acc0 0x7f92c006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:44.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.365+0000 7f92d1ffb700 1 -- 172.21.15.1:0/3982231599 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f92d8088bf0 con 0x7f92dc071d30 2024-03-18T18:50:44.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.365+0000 7f92e0f01700 1 --2- 172.21.15.1:0/3982231599 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f92c006acc0 0x7f92c006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:44.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.366+0000 7f92e0f01700 1 --2- 172.21.15.1:0/3982231599 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f92c006acc0 0x7f92c006d180 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f92cc000c00 tx=0x7f92cc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:44.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.370+0000 7f92d1ffb700 1 -- 172.21.15.1:0/3982231599 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f92d8053a20 con 0x7f92dc071d30 2024-03-18T18:50:44.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.571+0000 7f92e3966700 1 -- 172.21.15.1:0/3982231599 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f92dc0cb590 con 0x7f92dc071d30 2024-03-18T18:50:44.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.571+0000 7f92d1ffb700 1 -- 172.21.15.1:0/3982231599 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f92d8057040 con 0x7f92dc071d30 2024-03-18T18:50:44.569 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T18:50:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.573+0000 7f92e3966700 1 -- 172.21.15.1:0/3982231599 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f92c006acc0 msgr2=0x7f92c006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.573+0000 7f92e3966700 1 --2- 172.21.15.1:0/3982231599 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f92c006acc0 0x7f92c006d180 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f92cc000c00 tx=0x7f92cc008030 comp rx=0 tx=0).stop 2024-03-18T18:50:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.573+0000 7f92e3966700 1 -- 172.21.15.1:0/3982231599 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92dc071d30 msgr2=0x7f92dc0c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.573+0000 7f92e3966700 1 --2- 172.21.15.1:0/3982231599 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92dc071d30 0x7f92dc0c8170 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f92d800c500 tx=0x7f92d800c8c0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.573+0000 7f92e3966700 1 -- 172.21.15.1:0/3982231599 shutdown_connections 2024-03-18T18:50:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.573+0000 7f92e3966700 1 --2- 172.21.15.1:0/3982231599 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f92c006acc0 0x7f92c006d180 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.573+0000 7f92e3966700 1 --2- 172.21.15.1:0/3982231599 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92dc15a1d0 0x7f92dc0c5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.573+0000 7f92e3966700 1 --2- 172.21.15.1:0/3982231599 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92dc071d30 0x7f92dc0c8170 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.573+0000 7f92e3966700 1 --2- 172.21.15.1:0/3982231599 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92dc0cb9e0 0x7f92dc153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.573+0000 7f92e3966700 1 -- 172.21.15.1:0/3982231599 >> 172.21.15.1:0/3982231599 conn(0x7f92dc0cc9b0 msgr2=0x7f92dc0ca560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:44.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.573+0000 7f92e3966700 1 -- 172.21.15.1:0/3982231599 shutdown_connections 2024-03-18T18:50:44.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.574+0000 7f92e3966700 1 -- 172.21.15.1:0/3982231599 wait complete. 2024-03-18T18:50:44.583 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T18:50:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.713+0000 7f0c218cf700 1 Processor -- start 2024-03-18T18:50:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.713+0000 7f0c218cf700 1 -- start start 2024-03-18T18:50:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.714+0000 7f0c218cf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c1c06ae60 0x7f0c1c168150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.714+0000 7f0c218cf700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c1c0cf600 con 0x7f0c1c15d5a0 2024-03-18T18:50:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.714+0000 7f0c218cf700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c1c0ce720 con 0x7f0c1c0cfa90 2024-03-18T18:50:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.714+0000 7f0c218cf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c0ce9f0 con 0x7f0c1c06ae60 2024-03-18T18:50:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.714+0000 7f0c1affd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0c1c15d5a0 0x7f0c1c0ce010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32892/0 (socket says 172.21.15.1:32892) 2024-03-18T18:50:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.714+0000 7f0c1affd700 1 -- 172.21.15.1:0/4198185817 learned_addr learned my addr 172.21.15.1:0/4198185817 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.715+0000 7f0c1b7fe700 1 --2- 172.21.15.1:0/4198185817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c1c06ae60 0x7f0c1c168150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.715+0000 7f0c19ffb700 1 -- 172.21.15.1:0/4198185817 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983102960 0 0) 0x7f0c1c0cf600 con 0x7f0c1c15d5a0 2024-03-18T18:50:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.715+0000 7f0c19ffb700 1 -- 172.21.15.1:0/4198185817 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c080035b0 con 0x7f0c1c15d5a0 2024-03-18T18:50:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.715+0000 7f0c19ffb700 1 -- 172.21.15.1:0/4198185817 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831025244 0 0) 0x7f0c1c0ce720 con 0x7f0c1c0cfa90 2024-03-18T18:50:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.715+0000 7f0c19ffb700 1 -- 172.21.15.1:0/4198185817 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c1c0cf600 con 0x7f0c1c0cfa90 2024-03-18T18:50:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.715+0000 7f0c1b7fe700 1 -- 172.21.15.1:0/4198185817 >> v1:172.21.15.165:6789/0 conn(0x7f0c1c0cfa90 legacy=0x7f0c1c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.715+0000 7f0c1b7fe700 1 -- 172.21.15.1:0/4198185817 >> v1:172.21.15.1:6789/0 conn(0x7f0c1c15d5a0 legacy=0x7f0c1c0ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.715+0000 7f0c1b7fe700 1 -- 172.21.15.1:0/4198185817 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c1c0cecd0 con 0x7f0c1c06ae60 2024-03-18T18:50:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.715+0000 7f0c1b7fe700 1 --2- 172.21.15.1:0/4198185817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c1c06ae60 0x7f0c1c168150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0c14009dc0 tx=0x7f0c1400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca9daa313faeac2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.715+0000 7f0c19ffb700 1 -- 172.21.15.1:0/4198185817 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1400e070 con 0x7f0c1c06ae60 2024-03-18T18:50:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.715+0000 7f0c19ffb700 1 -- 172.21.15.1:0/4198185817 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c1400bb20 con 0x7f0c1c06ae60 2024-03-18T18:50:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.715+0000 7f0c19ffb700 1 -- 172.21.15.1:0/4198185817 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c14012430 con 0x7f0c1c06ae60 2024-03-18T18:50:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.716+0000 7f0c218cf700 1 -- 172.21.15.1:0/4198185817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c1c06ae60 msgr2=0x7f0c1c168150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.716+0000 7f0c218cf700 1 --2- 172.21.15.1:0/4198185817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c1c06ae60 0x7f0c1c168150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0c14009dc0 tx=0x7f0c1400b330 comp rx=0 tx=0).stop 2024-03-18T18:50:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.716+0000 7f0c218cf700 1 -- 172.21.15.1:0/4198185817 shutdown_connections 2024-03-18T18:50:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.716+0000 7f0c218cf700 1 --2- 172.21.15.1:0/4198185817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c1c06ae60 0x7f0c1c168150 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.716+0000 7f0c218cf700 1 -- 172.21.15.1:0/4198185817 >> 172.21.15.1:0/4198185817 conn(0x7f0c1c0cc9b0 msgr2=0x7f0c1c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.716+0000 7f0c218cf700 1 -- 172.21.15.1:0/4198185817 shutdown_connections 2024-03-18T18:50:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.716+0000 7f0c218cf700 1 -- 172.21.15.1:0/4198185817 wait complete. 2024-03-18T18:50:44.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.717+0000 7f0c218cf700 1 Processor -- start 2024-03-18T18:50:44.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.717+0000 7f0c218cf700 1 -- start start 2024-03-18T18:50:44.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.718+0000 7f0c218cf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c1c06ae60 0x7f0c1c0c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:44.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.718+0000 7f0c218cf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c1c0cfa90 0x7f0c1c0c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:44.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.718+0000 7f0c218cf700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c1c15d5a0 0x7f0c1c1f3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:44.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.718+0000 7f0c218cf700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c0c9fa0 con 0x7f0c1c06ae60 2024-03-18T18:50:44.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.718+0000 7f0c218cf700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c0af970 con 0x7f0c1c15d5a0 2024-03-18T18:50:44.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.718+0000 7f0c218cf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c0b0490 con 0x7f0c1c0cfa90 2024-03-18T18:50:44.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.718+0000 7f0c1affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c1c06ae60 0x7f0c1c0c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:44.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.718+0000 7f0c1affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c1c06ae60 0x7f0c1c0c99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59124/0 (socket says 172.21.15.1:59124) 2024-03-18T18:50:44.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.718+0000 7f0c1affd700 1 -- 172.21.15.1:0/1239624759 learned_addr learned my addr 172.21.15.1:0/1239624759 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:44.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.718+0000 7f0c1b7fe700 1 --2- 172.21.15.1:0/1239624759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c1c15d5a0 0x7f0c1c1f3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:44.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.718+0000 7f0c1a7fc700 1 --2- 172.21.15.1:0/1239624759 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c1c0cfa90 0x7f0c1c0c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:44.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.718+0000 7f0c1affd700 1 -- 172.21.15.1:0/1239624759 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c1c0cfa90 msgr2=0x7f0c1c0c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:44.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.718+0000 7f0c1affd700 1 --2- 172.21.15.1:0/1239624759 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c1c0cfa90 0x7f0c1c0c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.718+0000 7f0c1affd700 1 -- 172.21.15.1:0/1239624759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c1c15d5a0 msgr2=0x7f0c1c1f3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:44.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.718+0000 7f0c1affd700 1 --2- 172.21.15.1:0/1239624759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c1c15d5a0 0x7f0c1c1f3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.718+0000 7f0c1affd700 1 -- 172.21.15.1:0/1239624759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c14009a70 con 0x7f0c1c06ae60 2024-03-18T18:50:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.718+0000 7f0c1b7fe700 1 --2- 172.21.15.1:0/1239624759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c1c15d5a0 0x7f0c1c1f3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.719+0000 7f0c1affd700 1 --2- 172.21.15.1:0/1239624759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c1c06ae60 0x7f0c1c0c99c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0c0400c4d0 tx=0x7f0c0400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.719+0000 7f0c13fff700 1 -- 172.21.15.1:0/1239624759 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c040074e0 con 0x7f0c1c06ae60 2024-03-18T18:50:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.719+0000 7f0c13fff700 1 -- 172.21.15.1:0/1239624759 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c04007640 con 0x7f0c1c06ae60 2024-03-18T18:50:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.719+0000 7f0c218cf700 1 -- 172.21.15.1:0/1239624759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c1c0afb60 con 0x7f0c1c06ae60 2024-03-18T18:50:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.719+0000 7f0c13fff700 1 -- 172.21.15.1:0/1239624759 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c04005a30 con 0x7f0c1c06ae60 2024-03-18T18:50:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.719+0000 7f0c218cf700 1 -- 172.21.15.1:0/1239624759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c1c0b0120 con 0x7f0c1c06ae60 2024-03-18T18:50:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.720+0000 7f0c218cf700 1 -- 172.21.15.1:0/1239624759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c1c0676a0 con 0x7f0c1c06ae60 2024-03-18T18:50:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.720+0000 7f0c13fff700 1 -- 172.21.15.1:0/1239624759 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0c04005b90 con 0x7f0c1c06ae60 2024-03-18T18:50:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.721+0000 7f0c13fff700 1 --2- 172.21.15.1:0/1239624759 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0bfc06ac90 0x7f0bfc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.721+0000 7f0c1a7fc700 1 --2- 172.21.15.1:0/1239624759 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0bfc06ac90 0x7f0bfc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.721+0000 7f0c13fff700 1 -- 172.21.15.1:0/1239624759 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0c04088bc0 con 0x7f0c1c06ae60 2024-03-18T18:50:44.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.721+0000 7f0c1a7fc700 1 --2- 172.21.15.1:0/1239624759 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0bfc06ac90 0x7f0bfc06d150 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c0c9820 tx=0x7f0c0c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:44.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.725+0000 7f0c13fff700 1 -- 172.21.15.1:0/1239624759 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0c040539f0 con 0x7f0c1c06ae60 2024-03-18T18:50:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.929+0000 7f0c218cf700 1 -- 172.21.15.1:0/1239624759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0c1c150f90 con 0x7f0c1c06ae60 2024-03-18T18:50:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.929+0000 7f0c13fff700 1 -- 172.21.15.1:0/1239624759 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0c04017020 con 0x7f0c1c06ae60 2024-03-18T18:50:44.926 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:44.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.931+0000 7f0c218cf700 1 -- 172.21.15.1:0/1239624759 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0bfc06ac90 msgr2=0x7f0bfc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:44.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.931+0000 7f0c218cf700 1 --2- 172.21.15.1:0/1239624759 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0bfc06ac90 0x7f0bfc06d150 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c0c9820 tx=0x7f0c0c009040 comp rx=0 tx=0).stop 2024-03-18T18:50:44.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.931+0000 7f0c218cf700 1 -- 172.21.15.1:0/1239624759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c1c06ae60 msgr2=0x7f0c1c0c99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:44.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.931+0000 7f0c218cf700 1 --2- 172.21.15.1:0/1239624759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c1c06ae60 0x7f0c1c0c99c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0c0400c4d0 tx=0x7f0c0400c890 comp rx=0 tx=0).stop 2024-03-18T18:50:44.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.931+0000 7f0c218cf700 1 -- 172.21.15.1:0/1239624759 shutdown_connections 2024-03-18T18:50:44.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.931+0000 7f0c218cf700 1 --2- 172.21.15.1:0/1239624759 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0bfc06ac90 0x7f0bfc06d150 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.931+0000 7f0c218cf700 1 --2- 172.21.15.1:0/1239624759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c1c15d5a0 0x7f0c1c1f3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.931+0000 7f0c218cf700 1 --2- 172.21.15.1:0/1239624759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c1c06ae60 0x7f0c1c0c99c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.931+0000 7f0c218cf700 1 --2- 172.21.15.1:0/1239624759 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c1c0cfa90 0x7f0c1c0c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:44.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.931+0000 7f0c218cf700 1 -- 172.21.15.1:0/1239624759 >> 172.21.15.1:0/1239624759 conn(0x7f0c1c0cc9b0 msgr2=0x7f0c1c0ad870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:44.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.931+0000 7f0c218cf700 1 -- 172.21.15.1:0/1239624759 shutdown_connections 2024-03-18T18:50:44.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:44.931+0000 7f0c218cf700 1 -- 172.21.15.1:0/1239624759 wait complete. 2024-03-18T18:50:44.941 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T18:50:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.071+0000 7f0c73be5700 1 Processor -- start 2024-03-18T18:50:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.071+0000 7f0c73be5700 1 -- start start 2024-03-18T18:50:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.072+0000 7f0c73be5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c6c15a1d0 0x7f0c6c0add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.072+0000 7f0c73be5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0c6c071620 0x7f0c6c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.072+0000 7f0c73be5700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6c0ce8f0 con 0x7f0c6c0cb9e0 2024-03-18T18:50:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.072+0000 7f0c73be5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c6c0cdb00 con 0x7f0c6c071620 2024-03-18T18:50:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.072+0000 7f0c73be5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c15a5b0 con 0x7f0c6c15a1d0 2024-03-18T18:50:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.072+0000 7f0c72182700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0c6c0cb9e0 0x7f0c6c0cbe70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32902/0 (socket says 172.21.15.1:32902) 2024-03-18T18:50:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.072+0000 7f0c72182700 1 -- 172.21.15.1:0/3163514193 learned_addr learned my addr 172.21.15.1:0/3163514193 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.072+0000 7f0c71180700 1 --2- 172.21.15.1:0/3163514193 >> v2:172.21.15.165:3300/0 conn(0x7f0c6c071620 0x7f0c6c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.072+0000 7f0c71981700 1 --2- 172.21.15.1:0/3163514193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c6c15a1d0 0x7f0c6c0add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.072+0000 7f0c7097f700 1 -- 172.21.15.1:0/3163514193 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 201902827 0 0) 0x7f0c6c0ce8f0 con 0x7f0c6c0cb9e0 2024-03-18T18:50:45.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.072+0000 7f0c7097f700 1 -- 172.21.15.1:0/3163514193 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c580035b0 con 0x7f0c6c0cb9e0 2024-03-18T18:50:45.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.072+0000 7f0c7097f700 1 -- 172.21.15.1:0/3163514193 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3535145412 0 0) 0x7f0c580035b0 con 0x7f0c6c0cb9e0 2024-03-18T18:50:45.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.072+0000 7f0c7097f700 1 -- 172.21.15.1:0/3163514193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c6c15a1d0 msgr2=0x7f0c6c0add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:45.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.072+0000 7f0c7097f700 1 --2- 172.21.15.1:0/3163514193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c6c15a1d0 0x7f0c6c0add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.072+0000 7f0c7097f700 1 -- 172.21.15.1:0/3163514193 >> v2:172.21.15.165:3300/0 conn(0x7f0c6c071620 msgr2=0x7f0c6c071aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:45.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.072+0000 7f0c7097f700 1 --2- 172.21.15.1:0/3163514193 >> v2:172.21.15.165:3300/0 conn(0x7f0c6c071620 0x7f0c6c071aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.072+0000 7f0c7097f700 1 -- 172.21.15.1:0/3163514193 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c0ae260 con 0x7f0c6c0cb9e0 2024-03-18T18:50:45.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.072+0000 7f0c7097f700 1 -- 172.21.15.1:0/3163514193 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f0c640019f0 con 0x7f0c6c0cb9e0 2024-03-18T18:50:45.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.072+0000 7f0c7097f700 1 -- 172.21.15.1:0/3163514193 >> v1:172.21.15.1:6789/0 conn(0x7f0c6c0cb9e0 legacy=0x7f0c6c0cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:45.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.073+0000 7f0c7097f700 1 --2- 172.21.15.1:0/3163514193 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c58003a00 0x7f0c58003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.073+0000 7f0c7097f700 1 --2- 172.21.15.1:0/3163514193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c6c15a1d0 0x7f0c58004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.073+0000 7f0c7097f700 1 --2- 172.21.15.1:0/3163514193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c58004d90 0x7f0c58005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.073+0000 7f0c7097f700 1 -- 172.21.15.1:0/3163514193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c0cdb00 con 0x7f0c58004d90 2024-03-18T18:50:45.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.073+0000 7f0c7097f700 1 -- 172.21.15.1:0/3163514193 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5800ba10 con 0x7f0c58003a00 2024-03-18T18:50:45.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.073+0000 7f0c7097f700 1 -- 172.21.15.1:0/3163514193 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5800bb50 con 0x7f0c6c15a1d0 2024-03-18T18:50:45.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.073+0000 7f0c71180700 1 --2- 172.21.15.1:0/3163514193 >> v2:172.21.15.165:3300/0 conn(0x7f0c6c071620 0x7f0c6c071aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:50:45.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.073+0000 7f0c72182700 1 --2- 172.21.15.1:0/3163514193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c58004d90 0x7f0c58005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.073+0000 7f0c71981700 1 --2- 172.21.15.1:0/3163514193 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c58003a00 0x7f0c58003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.073+0000 7f0c71180700 1 --2- 172.21.15.1:0/3163514193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c6c15a1d0 0x7f0c58004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.073+0000 7f0c72182700 1 -- 172.21.15.1:0/3163514193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c6c15a1d0 msgr2=0x7f0c58004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:45.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.073+0000 7f0c72182700 1 --2- 172.21.15.1:0/3163514193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c6c15a1d0 0x7f0c58004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.073+0000 7f0c72182700 1 -- 172.21.15.1:0/3163514193 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c58003a00 msgr2=0x7f0c58003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:45.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.073+0000 7f0c72182700 1 --2- 172.21.15.1:0/3163514193 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c58003a00 0x7f0c58003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.073+0000 7f0c72182700 1 -- 172.21.15.1:0/3163514193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c580035b0 con 0x7f0c58004d90 2024-03-18T18:50:45.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.073+0000 7f0c72182700 1 --2- 172.21.15.1:0/3163514193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c58004d90 0x7f0c58005210 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0c6400a850 tx=0x7f0c58005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5228da9aa7af66a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:45.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.073+0000 7f0c7097f700 1 -- 172.21.15.1:0/3163514193 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c64012600 con 0x7f0c58004d90 2024-03-18T18:50:45.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.073+0000 7f0c7097f700 1 -- 172.21.15.1:0/3163514193 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c64012760 con 0x7f0c58004d90 2024-03-18T18:50:45.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.073+0000 7f0c7097f700 1 -- 172.21.15.1:0/3163514193 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c64012b10 con 0x7f0c58004d90 2024-03-18T18:50:45.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.074+0000 7f0c73be5700 1 -- 172.21.15.1:0/3163514193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c58004d90 msgr2=0x7f0c58005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:45.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.074+0000 7f0c73be5700 1 --2- 172.21.15.1:0/3163514193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c58004d90 0x7f0c58005210 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0c6400a850 tx=0x7f0c58005880 comp rx=0 tx=0).stop 2024-03-18T18:50:45.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.074+0000 7f0c72182700 1 -- 172.21.15.1:0/3163514193 reap_dead start 2024-03-18T18:50:45.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.074+0000 7f0c73be5700 1 -- 172.21.15.1:0/3163514193 shutdown_connections 2024-03-18T18:50:45.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.074+0000 7f0c73be5700 1 -- 172.21.15.1:0/3163514193 >> 172.21.15.1:0/3163514193 conn(0x7f0c6c0cc9b0 msgr2=0x7f0c6c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:45.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.074+0000 7f0c73be5700 1 -- 172.21.15.1:0/3163514193 shutdown_connections 2024-03-18T18:50:45.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.074+0000 7f0c73be5700 1 -- 172.21.15.1:0/3163514193 wait complete. 2024-03-18T18:50:45.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.075+0000 7f0c73be5700 1 Processor -- start 2024-03-18T18:50:45.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.075+0000 7f0c73be5700 1 -- start start 2024-03-18T18:50:45.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.076+0000 7f0c73be5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c58004d90 0x7f0c6c158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.076+0000 7f0c73be5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c6c14efc0 0x7f0c6c14f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.076+0000 7f0c73be5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c6c0bfbd0 0x7f0c6c0c0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.076+0000 7f0c73be5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c0c6430 con 0x7f0c6c14efc0 2024-03-18T18:50:45.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.076+0000 7f0c73be5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c0700a0 con 0x7f0c6c0bfbd0 2024-03-18T18:50:45.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.076+0000 7f0c73be5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c0c05c0 con 0x7f0c58004d90 2024-03-18T18:50:45.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.076+0000 7f0c71180700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c6c14efc0 0x7f0c6c14f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.076+0000 7f0c71180700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c6c14efc0 0x7f0c6c14f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59154/0 (socket says 172.21.15.1:59154) 2024-03-18T18:50:45.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.076+0000 7f0c71180700 1 -- 172.21.15.1:0/1581075594 learned_addr learned my addr 172.21.15.1:0/1581075594 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:45.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.076+0000 7f0c71981700 1 --2- 172.21.15.1:0/1581075594 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c58004d90 0x7f0c6c158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.076+0000 7f0c72182700 1 --2- 172.21.15.1:0/1581075594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c6c0bfbd0 0x7f0c6c0c0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.076+0000 7f0c71180700 1 -- 172.21.15.1:0/1581075594 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c58004d90 msgr2=0x7f0c6c158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:45.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.076+0000 7f0c71180700 1 --2- 172.21.15.1:0/1581075594 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c58004d90 0x7f0c6c158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.076+0000 7f0c71180700 1 -- 172.21.15.1:0/1581075594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c6c0bfbd0 msgr2=0x7f0c6c0c0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.076+0000 7f0c71180700 1 --2- 172.21.15.1:0/1581075594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c6c0bfbd0 0x7f0c6c0c0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.076+0000 7f0c71180700 1 -- 172.21.15.1:0/1581075594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c06f800 con 0x7f0c6c14efc0 2024-03-18T18:50:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.076+0000 7f0c72182700 1 --2- 172.21.15.1:0/1581075594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c6c0bfbd0 0x7f0c6c0c0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.077+0000 7f0c71180700 1 --2- 172.21.15.1:0/1581075594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c6c14efc0 0x7f0c6c14f440 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c011c20 tx=0x7f0c5c00f240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.077+0000 7f0c627fc700 1 -- 172.21.15.1:0/1581075594 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c00fac0 con 0x7f0c6c14efc0 2024-03-18T18:50:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.077+0000 7f0c627fc700 1 -- 172.21.15.1:0/1581075594 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5c00fc20 con 0x7f0c6c14efc0 2024-03-18T18:50:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.077+0000 7f0c627fc700 1 -- 172.21.15.1:0/1581075594 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c004020 con 0x7f0c6c14efc0 2024-03-18T18:50:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.077+0000 7f0c73be5700 1 -- 172.21.15.1:0/1581075594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c6c06fae0 con 0x7f0c6c14efc0 2024-03-18T18:50:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.077+0000 7f0c73be5700 1 -- 172.21.15.1:0/1581075594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c6c167b80 con 0x7f0c6c14efc0 2024-03-18T18:50:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.078+0000 7f0c73be5700 1 -- 172.21.15.1:0/1581075594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c6c166120 con 0x7f0c6c14efc0 2024-03-18T18:50:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.078+0000 7f0c627fc700 1 -- 172.21.15.1:0/1581075594 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0c5c004180 con 0x7f0c6c14efc0 2024-03-18T18:50:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.079+0000 7f0c627fc700 1 --2- 172.21.15.1:0/1581075594 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c5006ac90 0x7f0c5006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.079+0000 7f0c627fc700 1 -- 172.21.15.1:0/1581075594 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0c5c089c60 con 0x7f0c6c14efc0 2024-03-18T18:50:45.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.079+0000 7f0c71981700 1 --2- 172.21.15.1:0/1581075594 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c5006ac90 0x7f0c5006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.079+0000 7f0c71981700 1 --2- 172.21.15.1:0/1581075594 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c5006ac90 0x7f0c5006d150 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c067f80 tx=0x7f0c680091c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:45.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.083+0000 7f0c627fc700 1 -- 172.21.15.1:0/1581075594 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0c5c054b10 con 0x7f0c6c14efc0 2024-03-18T18:50:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.302+0000 7f0c73be5700 1 -- 172.21.15.1:0/1581075594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0c6c0acd80 con 0x7f0c6c14efc0 2024-03-18T18:50:45.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.302+0000 7f0c627fc700 1 -- 172.21.15.1:0/1581075594 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0c5c022070 con 0x7f0c6c14efc0 2024-03-18T18:50:45.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.304+0000 7f0c73be5700 1 -- 172.21.15.1:0/1581075594 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c5006ac90 msgr2=0x7f0c5006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:45.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.304+0000 7f0c73be5700 1 --2- 172.21.15.1:0/1581075594 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c5006ac90 0x7f0c5006d150 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c067f80 tx=0x7f0c680091c0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.304+0000 7f0c73be5700 1 -- 172.21.15.1:0/1581075594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c6c14efc0 msgr2=0x7f0c6c14f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:45.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.304+0000 7f0c73be5700 1 --2- 172.21.15.1:0/1581075594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c6c14efc0 0x7f0c6c14f440 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c011c20 tx=0x7f0c5c00f240 comp rx=0 tx=0).stop 2024-03-18T18:50:45.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.304+0000 7f0c73be5700 1 -- 172.21.15.1:0/1581075594 shutdown_connections 2024-03-18T18:50:45.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.304+0000 7f0c73be5700 1 --2- 172.21.15.1:0/1581075594 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c5006ac90 0x7f0c5006d150 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.304+0000 7f0c73be5700 1 --2- 172.21.15.1:0/1581075594 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0c58004d90 0x7f0c6c158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.304+0000 7f0c73be5700 1 --2- 172.21.15.1:0/1581075594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c6c0bfbd0 0x7f0c6c0c0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.304+0000 7f0c73be5700 1 --2- 172.21.15.1:0/1581075594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0c6c14efc0 0x7f0c6c14f440 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.304+0000 7f0c73be5700 1 -- 172.21.15.1:0/1581075594 >> 172.21.15.1:0/1581075594 conn(0x7f0c6c0cc9b0 msgr2=0x7f0c6c06bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:45.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.304+0000 7f0c73be5700 1 -- 172.21.15.1:0/1581075594 shutdown_connections 2024-03-18T18:50:45.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.304+0000 7f0c73be5700 1 -- 172.21.15.1:0/1581075594 wait complete. 2024-03-18T18:50:45.303 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-18T18:50:45.314 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:50:45.314 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T18:50:45.337 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:45.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.465+0000 7f37b8f21700 1 Processor -- start 2024-03-18T18:50:45.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.465+0000 7f37b8f21700 1 -- start start 2024-03-18T18:50:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.466+0000 7f37b8f21700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f37b4071d10 0x7f37b40cb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.466+0000 7f37b8f21700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f37b406db70 0x7f37b40cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.466+0000 7f37b8f21700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b40cbb10 con 0x7f37b415a310 2024-03-18T18:50:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.466+0000 7f37b8f21700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37b415a6f0 con 0x7f37b4071d10 2024-03-18T18:50:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.466+0000 7f37b8f21700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b40cfe50 con 0x7f37b406db70 2024-03-18T18:50:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.466+0000 7f37b2d9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f37b406db70 0x7f37b40cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.466+0000 7f37b1d9b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f37b4071d10 0x7f37b40cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.466+0000 7f37b2d9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f37b406db70 0x7f37b40cb5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:40008/0 (socket says 172.21.15.1:40008) 2024-03-18T18:50:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.466+0000 7f37b1d9b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f37b4071d10 0x7f37b40cb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:54860/0 (socket says 172.21.15.1:54860) 2024-03-18T18:50:45.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.466+0000 7f37b2d9d700 1 -- 172.21.15.1:0/3955988687 learned_addr learned my addr 172.21.15.1:0/3955988687 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:45.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.466+0000 7f37b259c700 1 --1- 172.21.15.1:0/3955988687 >> v1:172.21.15.165:6789/0 conn(0x7f37b415a310 0x7f37b4071600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:42478/0 (socket says 172.21.15.1:42478) 2024-03-18T18:50:45.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.466+0000 7f37b2d9d700 1 -- 172.21.15.1:0/3955988687 >> v2:172.21.15.165:3300/0 conn(0x7f37b4071d10 msgr2=0x7f37b40cb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:45.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.466+0000 7f37b2d9d700 1 --2- 172.21.15.1:0/3955988687 >> v2:172.21.15.165:3300/0 conn(0x7f37b4071d10 0x7f37b40cb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.466+0000 7f37b2d9d700 1 -- 172.21.15.1:0/3955988687 >> v1:172.21.15.165:6789/0 conn(0x7f37b415a310 legacy=0x7f37b4071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:45.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.466+0000 7f37b2d9d700 1 -- 172.21.15.1:0/3955988687 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b40cdda0 con 0x7f37b406db70 2024-03-18T18:50:45.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.466+0000 7f37b1d9b700 1 --2- 172.21.15.1:0/3955988687 >> v2:172.21.15.165:3300/0 conn(0x7f37b4071d10 0x7f37b40cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:50:45.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.467+0000 7f37b2d9d700 1 --2- 172.21.15.1:0/3955988687 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f37b406db70 0x7f37b40cb5d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f37ac009e50 tx=0x7f37ac00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe3ce5aca725c1ee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:45.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.467+0000 7f37b159a700 1 -- 172.21.15.1:0/3955988687 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac00e070 con 0x7f37b406db70 2024-03-18T18:50:45.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.467+0000 7f37b159a700 1 -- 172.21.15.1:0/3955988687 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ac00b910 con 0x7f37b406db70 2024-03-18T18:50:45.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.467+0000 7f37b159a700 1 -- 172.21.15.1:0/3955988687 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac00bcc0 con 0x7f37b406db70 2024-03-18T18:50:45.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.467+0000 7f37b8f21700 1 -- 172.21.15.1:0/3955988687 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f37b406db70 msgr2=0x7f37b40cb5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:45.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.467+0000 7f37b8f21700 1 --2- 172.21.15.1:0/3955988687 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f37b406db70 0x7f37b40cb5d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f37ac009e50 tx=0x7f37ac00b330 comp rx=0 tx=0).stop 2024-03-18T18:50:45.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.467+0000 7f37b8f21700 1 -- 172.21.15.1:0/3955988687 shutdown_connections 2024-03-18T18:50:45.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.467+0000 7f37b8f21700 1 --2- 172.21.15.1:0/3955988687 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f37b406db70 0x7f37b40cb5d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.467+0000 7f37b8f21700 1 --2- 172.21.15.1:0/3955988687 >> v2:172.21.15.165:3300/0 conn(0x7f37b4071d10 0x7f37b40cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.467+0000 7f37b8f21700 1 -- 172.21.15.1:0/3955988687 >> 172.21.15.1:0/3955988687 conn(0x7f37b40ccab0 msgr2=0x7f37b40ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:45.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.467+0000 7f37b8f21700 1 -- 172.21.15.1:0/3955988687 shutdown_connections 2024-03-18T18:50:45.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.468+0000 7f37b8f21700 1 -- 172.21.15.1:0/3955988687 wait complete. 2024-03-18T18:50:45.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.469+0000 7f37b8f21700 1 Processor -- start 2024-03-18T18:50:45.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.469+0000 7f37b8f21700 1 -- start start 2024-03-18T18:50:45.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37b8f21700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37b406db70 0x7f37b40c6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37b8f21700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37b4071d10 0x7f37b40b51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37b8f21700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f37b415a310 0x7f37b40b5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37b8f21700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b40a1250 con 0x7f37b4071d10 2024-03-18T18:50:45.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37b8f21700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b40b3b60 con 0x7f37b406db70 2024-03-18T18:50:45.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37b8f21700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b40b2c70 con 0x7f37b415a310 2024-03-18T18:50:45.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37b1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37b4071d10 0x7f37b40b51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37b1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37b4071d10 0x7f37b40b51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59162/0 (socket says 172.21.15.1:59162) 2024-03-18T18:50:45.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37b1d9b700 1 -- 172.21.15.1:0/897906220 learned_addr learned my addr 172.21.15.1:0/897906220 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:45.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37b259c700 1 --2- 172.21.15.1:0/897906220 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37b406db70 0x7f37b40c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37b2d9d700 1 --2- 172.21.15.1:0/897906220 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f37b415a310 0x7f37b40b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37b1d9b700 1 -- 172.21.15.1:0/897906220 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f37b415a310 msgr2=0x7f37b40b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:45.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37b1d9b700 1 --2- 172.21.15.1:0/897906220 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f37b415a310 0x7f37b40b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37b1d9b700 1 -- 172.21.15.1:0/897906220 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37b406db70 msgr2=0x7f37b40c6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:45.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37b1d9b700 1 --2- 172.21.15.1:0/897906220 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37b406db70 0x7f37b40c6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37b1d9b700 1 -- 172.21.15.1:0/897906220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a40094a0 con 0x7f37b4071d10 2024-03-18T18:50:45.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37b1d9b700 1 --2- 172.21.15.1:0/897906220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37b4071d10 0x7f37b40b51c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f37a400ebe0 tx=0x7f37a4009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:45.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37a2ffd700 1 -- 172.21.15.1:0/897906220 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a400c8b0 con 0x7f37b4071d10 2024-03-18T18:50:45.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37a2ffd700 1 -- 172.21.15.1:0/897906220 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a400ca10 con 0x7f37b4071d10 2024-03-18T18:50:45.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37b8f21700 1 -- 172.21.15.1:0/897906220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ac009b30 con 0x7f37b4071d10 2024-03-18T18:50:45.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37a2ffd700 1 -- 172.21.15.1:0/897906220 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a400cda0 con 0x7f37b4071d10 2024-03-18T18:50:45.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.470+0000 7f37b8f21700 1 -- 172.21.15.1:0/897906220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b40b3230 con 0x7f37b4071d10 2024-03-18T18:50:45.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.471+0000 7f37b8f21700 1 -- 172.21.15.1:0/897906220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b40676a0 con 0x7f37b4071d10 2024-03-18T18:50:45.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.472+0000 7f37a2ffd700 1 -- 172.21.15.1:0/897906220 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f37a4013470 con 0x7f37b4071d10 2024-03-18T18:50:45.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.472+0000 7f37a2ffd700 1 --2- 172.21.15.1:0/897906220 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f379406ac90 0x7f379406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.472+0000 7f37a2ffd700 1 -- 172.21.15.1:0/897906220 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f37a4089770 con 0x7f37b4071d10 2024-03-18T18:50:45.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.472+0000 7f37b259c700 1 --2- 172.21.15.1:0/897906220 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f379406ac90 0x7f379406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.473+0000 7f37b259c700 1 --2- 172.21.15.1:0/897906220 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f379406ac90 0x7f379406d150 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f37b406d580 tx=0x7f37a8007d70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:45.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.477+0000 7f37a2ffd700 1 -- 172.21.15.1:0/897906220 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f37a4054620 con 0x7f37b4071d10 2024-03-18T18:50:45.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.680+0000 7f37b8f21700 1 -- 172.21.15.1:0/897906220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37b40b5fb0 con 0x7f37b4071d10 2024-03-18T18:50:45.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.682+0000 7f37a2ffd700 1 -- 172.21.15.1:0/897906220 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18217 (secure 0 0 0) 0x7f37a4057c40 con 0x7f37b4071d10 2024-03-18T18:50:45.679 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:45.679 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":27,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:50:34.993482+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:45.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.684+0000 7f37b8f21700 1 -- 172.21.15.1:0/897906220 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f379406ac90 msgr2=0x7f379406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.684+0000 7f37b8f21700 1 --2- 172.21.15.1:0/897906220 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f379406ac90 0x7f379406d150 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f37b406d580 tx=0x7f37a8007d70 comp rx=0 tx=0).stop 2024-03-18T18:50:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.684+0000 7f37b8f21700 1 -- 172.21.15.1:0/897906220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37b4071d10 msgr2=0x7f37b40b51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.684+0000 7f37b8f21700 1 --2- 172.21.15.1:0/897906220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37b4071d10 0x7f37b40b51c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f37a400ebe0 tx=0x7f37a4009e20 comp rx=0 tx=0).stop 2024-03-18T18:50:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.684+0000 7f37b8f21700 1 -- 172.21.15.1:0/897906220 shutdown_connections 2024-03-18T18:50:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.684+0000 7f37b8f21700 1 --2- 172.21.15.1:0/897906220 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f379406ac90 0x7f379406d150 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.684+0000 7f37b8f21700 1 --2- 172.21.15.1:0/897906220 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f37b415a310 0x7f37b40b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.684+0000 7f37b8f21700 1 --2- 172.21.15.1:0/897906220 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37b406db70 0x7f37b40c6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.684+0000 7f37b8f21700 1 --2- 172.21.15.1:0/897906220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37b4071d10 0x7f37b40b51c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.684+0000 7f37b8f21700 1 -- 172.21.15.1:0/897906220 >> 172.21.15.1:0/897906220 conn(0x7f37b40ccab0 msgr2=0x7f37b406fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.684+0000 7f37b8f21700 1 -- 172.21.15.1:0/897906220 shutdown_connections 2024-03-18T18:50:45.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.684+0000 7f37b8f21700 1 -- 172.21.15.1:0/897906220 wait complete. 2024-03-18T18:50:45.694 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T18:49:42.800801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T18:49:43.776684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T18:49:46.854608+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T18:50:45.694 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T18:50:45.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.825+0000 7fd1f25b1700 1 Processor -- start 2024-03-18T18:50:45.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.825+0000 7fd1f25b1700 1 -- start start 2024-03-18T18:50:45.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.825+0000 7fd1f25b1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd1ec0b1e20 0x7fd1ec0cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.825+0000 7fd1f25b1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1ec0cbce0 0x7fd1ec0cc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.825+0000 7fd1f25b1700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1ec0d0df0 con 0x7fd1ec0cd870 2024-03-18T18:50:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.825+0000 7fd1f25b1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1ec0d0550 con 0x7fd1ec0cbce0 2024-03-18T18:50:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.825+0000 7fd1f25b1700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1ec0d0830 con 0x7fd1ec0b1e20 2024-03-18T18:50:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.826+0000 7fd1f0b4e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1ec0cbce0 0x7fd1ec0cc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.826+0000 7fd1f0b4e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1ec0cbce0 0x7fd1ec0cc150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59166/0 (socket says 172.21.15.1:59166) 2024-03-18T18:50:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.826+0000 7fd1f0b4e700 1 -- 172.21.15.1:0/3774352871 learned_addr learned my addr 172.21.15.1:0/3774352871 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.826+0000 7fd1eb7fe700 1 --2- 172.21.15.1:0/3774352871 >> v2:172.21.15.165:3300/0 conn(0x7fd1ec0b1e20 0x7fd1ec0cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.826+0000 7fd1f0b4e700 1 -- 172.21.15.1:0/3774352871 >> v2:172.21.15.165:3300/0 conn(0x7fd1ec0b1e20 msgr2=0x7fd1ec0cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.826+0000 7fd1f0b4e700 1 --2- 172.21.15.1:0/3774352871 >> v2:172.21.15.165:3300/0 conn(0x7fd1ec0b1e20 0x7fd1ec0cff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.826+0000 7fd1f0b4e700 1 -- 172.21.15.1:0/3774352871 >> v1:172.21.15.165:6789/0 conn(0x7fd1ec0cd870 legacy=0x7fd1ec0cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.826+0000 7fd1f0b4e700 1 -- 172.21.15.1:0/3774352871 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1ec0d0b10 con 0x7fd1ec0cbce0 2024-03-18T18:50:45.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.826+0000 7fd1f0b4e700 1 --2- 172.21.15.1:0/3774352871 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec0cbce0 0x7fd1ec0cc150 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec0715c0 tx=0x7fd1e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0a5418c00534f09 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:45.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.826+0000 7fd1eaffd700 1 -- 172.21.15.1:0/3774352871 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e400e070 con 0x7fd1ec0cbce0 2024-03-18T18:50:45.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.826+0000 7fd1eaffd700 1 -- 172.21.15.1:0/3774352871 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e4003a20 con 0x7fd1ec0cbce0 2024-03-18T18:50:45.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.826+0000 7fd1eaffd700 1 -- 172.21.15.1:0/3774352871 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e4003dd0 con 0x7fd1ec0cbce0 2024-03-18T18:50:45.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.827+0000 7fd1f25b1700 1 -- 172.21.15.1:0/3774352871 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec0cbce0 msgr2=0x7fd1ec0cc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:45.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.827+0000 7fd1f25b1700 1 --2- 172.21.15.1:0/3774352871 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec0cbce0 0x7fd1ec0cc150 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec0715c0 tx=0x7fd1e400b330 comp rx=0 tx=0).stop 2024-03-18T18:50:45.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.827+0000 7fd1f25b1700 1 -- 172.21.15.1:0/3774352871 shutdown_connections 2024-03-18T18:50:45.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.827+0000 7fd1f25b1700 1 --2- 172.21.15.1:0/3774352871 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec0cbce0 0x7fd1ec0cc150 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.827+0000 7fd1f25b1700 1 --2- 172.21.15.1:0/3774352871 >> v2:172.21.15.165:3300/0 conn(0x7fd1ec0b1e20 0x7fd1ec0cff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.827+0000 7fd1f25b1700 1 -- 172.21.15.1:0/3774352871 >> 172.21.15.1:0/3774352871 conn(0x7fd1ec0d0ff0 msgr2=0x7fd1ec0d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:45.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.827+0000 7fd1f25b1700 1 -- 172.21.15.1:0/3774352871 shutdown_connections 2024-03-18T18:50:45.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.827+0000 7fd1f25b1700 1 -- 172.21.15.1:0/3774352871 wait complete. 2024-03-18T18:50:45.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.828+0000 7fd1f25b1700 1 Processor -- start 2024-03-18T18:50:45.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.828+0000 7fd1f25b1700 1 -- start start 2024-03-18T18:50:45.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.829+0000 7fd1f25b1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec0b1e20 0x7fd1ec0cb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.829+0000 7fd1f25b1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1ec0cbce0 0x7fd1ec0adc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.829+0000 7fd1f25b1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1ec0cd870 0x7fd1ec152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.829+0000 7fd1f25b1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec0ae420 con 0x7fd1ec0b1e20 2024-03-18T18:50:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.829+0000 7fd1f25b1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec0b0d90 con 0x7fd1ec0cbce0 2024-03-18T18:50:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.829+0000 7fd1f25b1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec0ae180 con 0x7fd1ec0cd870 2024-03-18T18:50:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.829+0000 7fd1ebfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec0b1e20 0x7fd1ec0cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.829+0000 7fd1ebfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec0b1e20 0x7fd1ec0cb250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59178/0 (socket says 172.21.15.1:59178) 2024-03-18T18:50:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.829+0000 7fd1ebfff700 1 -- 172.21.15.1:0/4038270905 learned_addr learned my addr 172.21.15.1:0/4038270905 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.829+0000 7fd1eb7fe700 1 --2- 172.21.15.1:0/4038270905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1ec0cbce0 0x7fd1ec0adc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.829+0000 7fd1f0b4e700 1 --2- 172.21.15.1:0/4038270905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1ec0cd870 0x7fd1ec152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.829+0000 7fd1ebfff700 1 -- 172.21.15.1:0/4038270905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1ec0cd870 msgr2=0x7fd1ec152e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.829+0000 7fd1ebfff700 1 --2- 172.21.15.1:0/4038270905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1ec0cd870 0x7fd1ec152e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.829+0000 7fd1ebfff700 1 -- 172.21.15.1:0/4038270905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1ec0cbce0 msgr2=0x7fd1ec0adc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:45.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.829+0000 7fd1ebfff700 1 --2- 172.21.15.1:0/4038270905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1ec0cbce0 0x7fd1ec0adc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:45.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.829+0000 7fd1ebfff700 1 -- 172.21.15.1:0/4038270905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e4009b30 con 0x7fd1ec0b1e20 2024-03-18T18:50:45.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.830+0000 7fd1ebfff700 1 --2- 172.21.15.1:0/4038270905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec0b1e20 0x7fd1ec0cb250 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00b4a0 tx=0x7fd1dc00b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:45.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.830+0000 7fd1e8ff9700 1 -- 172.21.15.1:0/4038270905 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc011830 con 0x7fd1ec0b1e20 2024-03-18T18:50:45.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.830+0000 7fd1e8ff9700 1 -- 172.21.15.1:0/4038270905 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1dc011990 con 0x7fd1ec0b1e20 2024-03-18T18:50:45.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.830+0000 7fd1e8ff9700 1 -- 172.21.15.1:0/4038270905 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc011d40 con 0x7fd1ec0b1e20 2024-03-18T18:50:45.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.830+0000 7fd1f25b1700 1 -- 172.21.15.1:0/4038270905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1ec0b04f0 con 0x7fd1ec0b1e20 2024-03-18T18:50:45.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.830+0000 7fd1f25b1700 1 -- 172.21.15.1:0/4038270905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1ec0b0ab0 con 0x7fd1ec0b1e20 2024-03-18T18:50:45.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.831+0000 7fd1f25b1700 1 -- 172.21.15.1:0/4038270905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1ec0676a0 con 0x7fd1ec0b1e20 2024-03-18T18:50:45.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.831+0000 7fd1e8ff9700 1 -- 172.21.15.1:0/4038270905 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd1dc01a400 con 0x7fd1ec0b1e20 2024-03-18T18:50:45.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.832+0000 7fd1e8ff9700 1 --2- 172.21.15.1:0/4038270905 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd1d806ac90 0x7fd1d806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:45.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.832+0000 7fd1eb7fe700 1 --2- 172.21.15.1:0/4038270905 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd1d806ac90 0x7fd1d806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:45.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.832+0000 7fd1e8ff9700 1 -- 172.21.15.1:0/4038270905 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd1dc089600 con 0x7fd1ec0b1e20 2024-03-18T18:50:45.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.832+0000 7fd1eb7fe700 1 --2- 172.21.15.1:0/4038270905 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd1d806ac90 0x7fd1d806d150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec0cb790 tx=0x7fd1e0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:45.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:45.837+0000 7fd1e8ff9700 1 -- 172.21.15.1:0/4038270905 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd1dc054420 con 0x7fd1ec0b1e20 2024-03-18T18:50:46.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.036+0000 7fd1f25b1700 1 -- 172.21.15.1:0/4038270905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd1ec0c7970 con 0x7fd1ec0b1e20 2024-03-18T18:50:46.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.036+0000 7fd1e8ff9700 1 -- 172.21.15.1:0/4038270905 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd1dc057a40 con 0x7fd1ec0b1e20 2024-03-18T18:50:46.034 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T18:50:46.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.038+0000 7fd1f25b1700 1 -- 172.21.15.1:0/4038270905 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd1d806ac90 msgr2=0x7fd1d806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:46.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.038+0000 7fd1f25b1700 1 --2- 172.21.15.1:0/4038270905 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd1d806ac90 0x7fd1d806d150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec0cb790 tx=0x7fd1e0008040 comp rx=0 tx=0).stop 2024-03-18T18:50:46.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.038+0000 7fd1f25b1700 1 -- 172.21.15.1:0/4038270905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec0b1e20 msgr2=0x7fd1ec0cb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:46.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.038+0000 7fd1f25b1700 1 --2- 172.21.15.1:0/4038270905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec0b1e20 0x7fd1ec0cb250 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00b4a0 tx=0x7fd1dc00b7b0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.039+0000 7fd1f25b1700 1 -- 172.21.15.1:0/4038270905 shutdown_connections 2024-03-18T18:50:46.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.039+0000 7fd1f25b1700 1 --2- 172.21.15.1:0/4038270905 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd1d806ac90 0x7fd1d806d150 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.039+0000 7fd1f25b1700 1 --2- 172.21.15.1:0/4038270905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1ec0cd870 0x7fd1ec152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.039+0000 7fd1f25b1700 1 --2- 172.21.15.1:0/4038270905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1ec0b1e20 0x7fd1ec0cb250 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.039+0000 7fd1f25b1700 1 --2- 172.21.15.1:0/4038270905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1ec0cbce0 0x7fd1ec0adc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.039+0000 7fd1f25b1700 1 -- 172.21.15.1:0/4038270905 >> 172.21.15.1:0/4038270905 conn(0x7fd1ec0d0ff0 msgr2=0x7fd1ec06f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:46.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.039+0000 7fd1f25b1700 1 -- 172.21.15.1:0/4038270905 shutdown_connections 2024-03-18T18:50:46.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.039+0000 7fd1f25b1700 1 -- 172.21.15.1:0/4038270905 wait complete. 2024-03-18T18:50:46.048 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T18:50:46.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.178+0000 7fe09a042700 1 Processor -- start 2024-03-18T18:50:46.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.178+0000 7fe09a042700 1 -- start start 2024-03-18T18:50:46.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.179+0000 7fe09a042700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0940cfb50 0x7fe0940d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:46.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.179+0000 7fe09a042700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe0940cff30 0x7fe0940b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:46.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.179+0000 7fe09a042700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe0940b3d40 0x7fe0940b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:46.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.179+0000 7fe09a042700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0940b3370 con 0x7fe0940b3d40 2024-03-18T18:50:46.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.179+0000 7fe09a042700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0940b2760 con 0x7fe0940cff30 2024-03-18T18:50:46.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.179+0000 7fe09a042700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940b2a40 con 0x7fe0940cfb50 2024-03-18T18:50:46.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.179+0000 7fe093fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe0940b3d40 0x7fe0940b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:46.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.179+0000 7fe093fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe0940b3d40 0x7fe0940b41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59184/0 (socket says 172.21.15.1:59184) 2024-03-18T18:50:46.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.179+0000 7fe093fff700 1 -- 172.21.15.1:0/677511842 learned_addr learned my addr 172.21.15.1:0/677511842 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:46.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.179+0000 7fe092ffd700 1 --2- 172.21.15.1:0/677511842 >> v2:172.21.15.165:3300/0 conn(0x7fe0940cff30 0x7fe0940b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:46.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.179+0000 7fe0937fe700 1 --2- 172.21.15.1:0/677511842 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0940cfb50 0x7fe0940d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:46.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.179+0000 7fe093fff700 1 -- 172.21.15.1:0/677511842 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0940cfb50 msgr2=0x7fe0940d3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:46.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.179+0000 7fe093fff700 1 --2- 172.21.15.1:0/677511842 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0940cfb50 0x7fe0940d3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.179+0000 7fe093fff700 1 -- 172.21.15.1:0/677511842 >> v2:172.21.15.165:3300/0 conn(0x7fe0940cff30 msgr2=0x7fe0940b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:46.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.179+0000 7fe093fff700 1 --2- 172.21.15.1:0/677511842 >> v2:172.21.15.165:3300/0 conn(0x7fe0940cff30 0x7fe0940b3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.179+0000 7fe093fff700 1 -- 172.21.15.1:0/677511842 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0940b2d20 con 0x7fe0940b3d40 2024-03-18T18:50:46.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.180+0000 7fe093fff700 1 --2- 172.21.15.1:0/677511842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0940b3d40 0x7fe0940b41d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe0940b5cc0 tx=0x7fe08c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ff074a647334e0b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:46.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.180+0000 7fe0927fc700 1 -- 172.21.15.1:0/677511842 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c00e070 con 0x7fe0940b3d40 2024-03-18T18:50:46.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.180+0000 7fe0927fc700 1 -- 172.21.15.1:0/677511842 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c00b910 con 0x7fe0940b3d40 2024-03-18T18:50:46.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.180+0000 7fe0927fc700 1 -- 172.21.15.1:0/677511842 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c00bcc0 con 0x7fe0940b3d40 2024-03-18T18:50:46.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.180+0000 7fe09a042700 1 -- 172.21.15.1:0/677511842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0940b3d40 msgr2=0x7fe0940b41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:46.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.180+0000 7fe09a042700 1 --2- 172.21.15.1:0/677511842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0940b3d40 0x7fe0940b41d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe0940b5cc0 tx=0x7fe08c00b330 comp rx=0 tx=0).stop 2024-03-18T18:50:46.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.180+0000 7fe09a042700 1 -- 172.21.15.1:0/677511842 shutdown_connections 2024-03-18T18:50:46.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.180+0000 7fe09a042700 1 --2- 172.21.15.1:0/677511842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0940b3d40 0x7fe0940b41d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.180+0000 7fe09a042700 1 --2- 172.21.15.1:0/677511842 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0940cfb50 0x7fe0940d3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.180+0000 7fe09a042700 1 --2- 172.21.15.1:0/677511842 >> v2:172.21.15.165:3300/0 conn(0x7fe0940cff30 0x7fe0940b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.180+0000 7fe09a042700 1 -- 172.21.15.1:0/677511842 >> 172.21.15.1:0/677511842 conn(0x7fe0940d3140 msgr2=0x7fe0940d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:46.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.180+0000 7fe09a042700 1 -- 172.21.15.1:0/677511842 shutdown_connections 2024-03-18T18:50:46.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.180+0000 7fe09a042700 1 -- 172.21.15.1:0/677511842 wait complete. 2024-03-18T18:50:46.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.182+0000 7fe09a042700 1 Processor -- start 2024-03-18T18:50:46.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.182+0000 7fe09a042700 1 -- start start 2024-03-18T18:50:46.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.182+0000 7fe09a042700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0940b3d40 0x7fe0940b0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:46.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe09a042700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0940cfb50 0x7fe0940b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:46.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe09a042700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0940cff30 0x7fe0941581e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:46.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe09a042700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940c7cf0 con 0x7fe0940cff30 2024-03-18T18:50:46.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe09a042700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940b1d90 con 0x7fe0940b3d40 2024-03-18T18:50:46.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe09a042700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940b1090 con 0x7fe0940cfb50 2024-03-18T18:50:46.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe093fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0940cff30 0x7fe0941581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:46.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe093fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0940cff30 0x7fe0941581e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59192/0 (socket says 172.21.15.1:59192) 2024-03-18T18:50:46.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe093fff700 1 -- 172.21.15.1:0/3255435906 learned_addr learned my addr 172.21.15.1:0/3255435906 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:46.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe0937fe700 1 --2- 172.21.15.1:0/3255435906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0940b3d40 0x7fe0940b0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:46.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe092ffd700 1 --2- 172.21.15.1:0/3255435906 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0940cfb50 0x7fe0940b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:46.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe093fff700 1 -- 172.21.15.1:0/3255435906 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0940cfb50 msgr2=0x7fe0940b1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:46.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe093fff700 1 --2- 172.21.15.1:0/3255435906 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0940cfb50 0x7fe0940b1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe093fff700 1 -- 172.21.15.1:0/3255435906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0940b3d40 msgr2=0x7fe0940b0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:46.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe093fff700 1 --2- 172.21.15.1:0/3255435906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0940b3d40 0x7fe0940b0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe093fff700 1 -- 172.21.15.1:0/3255435906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08c009b30 con 0x7fe0940cff30 2024-03-18T18:50:46.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe093fff700 1 --2- 172.21.15.1:0/3255435906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0940cff30 0x7fe0941581e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe09408d8e0 tx=0x7fe08c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:46.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe07bfff700 1 -- 172.21.15.1:0/3255435906 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c00ce80 con 0x7fe0940cff30 2024-03-18T18:50:46.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe07bfff700 1 -- 172.21.15.1:0/3255435906 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c004250 con 0x7fe0940cff30 2024-03-18T18:50:46.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe09a042700 1 -- 172.21.15.1:0/3255435906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0940c7630 con 0x7fe0940cff30 2024-03-18T18:50:46.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe07bfff700 1 -- 172.21.15.1:0/3255435906 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c004600 con 0x7fe0940cff30 2024-03-18T18:50:46.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.183+0000 7fe09a042700 1 -- 172.21.15.1:0/3255435906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe094158720 con 0x7fe0940cff30 2024-03-18T18:50:46.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.184+0000 7fe09a042700 1 -- 172.21.15.1:0/3255435906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0940c7490 con 0x7fe0940cff30 2024-03-18T18:50:46.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.185+0000 7fe07bfff700 1 -- 172.21.15.1:0/3255435906 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe08c004760 con 0x7fe0940cff30 2024-03-18T18:50:46.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.185+0000 7fe07bfff700 1 --2- 172.21.15.1:0/3255435906 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe08006ac90 0x7fe08006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:46.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.185+0000 7fe0937fe700 1 --2- 172.21.15.1:0/3255435906 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe08006ac90 0x7fe08006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:46.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.185+0000 7fe07bfff700 1 -- 172.21.15.1:0/3255435906 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe08c08ada0 con 0x7fe0940cff30 2024-03-18T18:50:46.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.186+0000 7fe0937fe700 1 --2- 172.21.15.1:0/3255435906 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe08006ac90 0x7fe08006d150 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe088009ad0 tx=0x7fe088008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:46.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.190+0000 7fe07bfff700 1 -- 172.21.15.1:0/3255435906 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe0940c7490 con 0x7fe0940cff30 2024-03-18T18:50:46.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.393+0000 7fe09a042700 1 -- 172.21.15.1:0/3255435906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe0940c7490 con 0x7fe0940cff30 2024-03-18T18:50:46.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.393+0000 7fe07bfff700 1 -- 172.21.15.1:0/3255435906 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe0940c7490 con 0x7fe0940cff30 2024-03-18T18:50:46.391 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:46.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.395+0000 7fe09a042700 1 -- 172.21.15.1:0/3255435906 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe08006ac90 msgr2=0x7fe08006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:46.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.395+0000 7fe09a042700 1 --2- 172.21.15.1:0/3255435906 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe08006ac90 0x7fe08006d150 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe088009ad0 tx=0x7fe088008040 comp rx=0 tx=0).stop 2024-03-18T18:50:46.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.395+0000 7fe09a042700 1 -- 172.21.15.1:0/3255435906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0940cff30 msgr2=0x7fe0941581e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:46.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.395+0000 7fe09a042700 1 --2- 172.21.15.1:0/3255435906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0940cff30 0x7fe0941581e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe09408d8e0 tx=0x7fe08c004ad0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.395+0000 7fe09a042700 1 -- 172.21.15.1:0/3255435906 shutdown_connections 2024-03-18T18:50:46.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.395+0000 7fe09a042700 1 --2- 172.21.15.1:0/3255435906 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe08006ac90 0x7fe08006d150 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.395+0000 7fe09a042700 1 --2- 172.21.15.1:0/3255435906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0940cff30 0x7fe0941581e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.395+0000 7fe09a042700 1 --2- 172.21.15.1:0/3255435906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0940b3d40 0x7fe0940b0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.395+0000 7fe09a042700 1 --2- 172.21.15.1:0/3255435906 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0940cfb50 0x7fe0940b1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.395+0000 7fe09a042700 1 -- 172.21.15.1:0/3255435906 >> 172.21.15.1:0/3255435906 conn(0x7fe0940d3140 msgr2=0x7fe0940b5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:46.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.395+0000 7fe09a042700 1 -- 172.21.15.1:0/3255435906 shutdown_connections 2024-03-18T18:50:46.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.395+0000 7fe09a042700 1 -- 172.21.15.1:0/3255435906 wait complete. 2024-03-18T18:50:46.405 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T18:50:46.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.535+0000 7fba45eab700 1 Processor -- start 2024-03-18T18:50:46.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.535+0000 7fba45eab700 1 -- start start 2024-03-18T18:50:46.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.536+0000 7fba45eab700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fba4006ca80 0x7fba4015d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:46.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.536+0000 7fba45eab700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fba4009a150 0x7fba400cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:46.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.536+0000 7fba45eab700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba400b3b00 con 0x7fba4015d5b0 2024-03-18T18:50:46.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.536+0000 7fba45eab700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba400cbbb0 con 0x7fba4009a150 2024-03-18T18:50:46.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.536+0000 7fba45eab700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba400cf7f0 con 0x7fba4006ca80 2024-03-18T18:50:46.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.536+0000 7fba3f7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fba4015d5b0 0x7fba40099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32916/0 (socket says 172.21.15.1:32916) 2024-03-18T18:50:46.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.536+0000 7fba3f7fe700 1 -- 172.21.15.1:0/3971367499 learned_addr learned my addr 172.21.15.1:0/3971367499 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:46.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.536+0000 7fba3ffff700 1 --2- 172.21.15.1:0/3971367499 >> v2:172.21.15.1:3300/0 conn(0x7fba4009a150 0x7fba400cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:46.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.536+0000 7fba3effd700 1 --2- 172.21.15.1:0/3971367499 >> v2:172.21.15.165:3300/0 conn(0x7fba4006ca80 0x7fba4015d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:46.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.536+0000 7fba3ffff700 1 -- 172.21.15.1:0/3971367499 >> v2:172.21.15.165:3300/0 conn(0x7fba4006ca80 msgr2=0x7fba4015d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:46.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.536+0000 7fba3ffff700 1 --2- 172.21.15.1:0/3971367499 >> v2:172.21.15.165:3300/0 conn(0x7fba4006ca80 0x7fba4015d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.536+0000 7fba3ffff700 1 -- 172.21.15.1:0/3971367499 >> v1:172.21.15.1:6789/0 conn(0x7fba4015d5b0 legacy=0x7fba40099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:46.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.536+0000 7fba3ffff700 1 -- 172.21.15.1:0/3971367499 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba400d04d0 con 0x7fba4009a150 2024-03-18T18:50:46.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.536+0000 7fba3e7fc700 1 -- 172.21.15.1:0/3971367499 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857417333 0 0) 0x7fba400b3b00 con 0x7fba4015d5b0 2024-03-18T18:50:46.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.536+0000 7fba3ffff700 1 --2- 172.21.15.1:0/3971367499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba4009a150 0x7fba400cbfd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fba40072c10 tx=0x7fba3800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebbf3c648132a681 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:46.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.537+0000 7fba3e7fc700 1 -- 172.21.15.1:0/3971367499 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3800e070 con 0x7fba4009a150 2024-03-18T18:50:46.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.537+0000 7fba3e7fc700 1 -- 172.21.15.1:0/3971367499 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3800bbd0 con 0x7fba4009a150 2024-03-18T18:50:46.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.537+0000 7fba3e7fc700 1 -- 172.21.15.1:0/3971367499 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba38012430 con 0x7fba4009a150 2024-03-18T18:50:46.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.537+0000 7fba45eab700 1 -- 172.21.15.1:0/3971367499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba4009a150 msgr2=0x7fba400cbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:46.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.537+0000 7fba45eab700 1 --2- 172.21.15.1:0/3971367499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba4009a150 0x7fba400cbfd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fba40072c10 tx=0x7fba3800b330 comp rx=0 tx=0).stop 2024-03-18T18:50:46.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.537+0000 7fba45eab700 1 -- 172.21.15.1:0/3971367499 shutdown_connections 2024-03-18T18:50:46.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.537+0000 7fba45eab700 1 --2- 172.21.15.1:0/3971367499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba4009a150 0x7fba400cbfd0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.537+0000 7fba45eab700 1 --2- 172.21.15.1:0/3971367499 >> v2:172.21.15.165:3300/0 conn(0x7fba4006ca80 0x7fba4015d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.537+0000 7fba45eab700 1 -- 172.21.15.1:0/3971367499 >> 172.21.15.1:0/3971367499 conn(0x7fba400b2bd0 msgr2=0x7fba400b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:46.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.537+0000 7fba45eab700 1 -- 172.21.15.1:0/3971367499 shutdown_connections 2024-03-18T18:50:46.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.537+0000 7fba45eab700 1 -- 172.21.15.1:0/3971367499 wait complete. 2024-03-18T18:50:46.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.538+0000 7fba45eab700 1 Processor -- start 2024-03-18T18:50:46.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.539+0000 7fba45eab700 1 -- start start 2024-03-18T18:50:46.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.539+0000 7fba45eab700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba4006ca80 0x7fba4015aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:46.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.539+0000 7fba45eab700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba4009a150 0x7fba400b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:46.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.539+0000 7fba45eab700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba4015d5b0 0x7fba400c6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:46.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.539+0000 7fba45eab700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba400b6180 con 0x7fba4009a150 2024-03-18T18:50:46.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.539+0000 7fba45eab700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba400b6460 con 0x7fba4006ca80 2024-03-18T18:50:46.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.539+0000 7fba45eab700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba400b6740 con 0x7fba4015d5b0 2024-03-18T18:50:46.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.539+0000 7fba3effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba4009a150 0x7fba400b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:46.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.540+0000 7fba3effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba4009a150 0x7fba400b38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59218/0 (socket says 172.21.15.1:59218) 2024-03-18T18:50:46.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.540+0000 7fba3effd700 1 -- 172.21.15.1:0/2941971711 learned_addr learned my addr 172.21.15.1:0/2941971711 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:46.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.540+0000 7fba3f7fe700 1 --2- 172.21.15.1:0/2941971711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba4006ca80 0x7fba4015aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:46.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.540+0000 7fba3ffff700 1 --2- 172.21.15.1:0/2941971711 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba4015d5b0 0x7fba400c6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:46.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.540+0000 7fba3effd700 1 -- 172.21.15.1:0/2941971711 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba4015d5b0 msgr2=0x7fba400c6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:46.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.540+0000 7fba3effd700 1 --2- 172.21.15.1:0/2941971711 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba4015d5b0 0x7fba400c6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.540+0000 7fba3effd700 1 -- 172.21.15.1:0/2941971711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba4006ca80 msgr2=0x7fba4015aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:46.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.540+0000 7fba3effd700 1 --2- 172.21.15.1:0/2941971711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba4006ca80 0x7fba4015aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.540+0000 7fba3effd700 1 -- 172.21.15.1:0/2941971711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba38009a70 con 0x7fba4009a150 2024-03-18T18:50:46.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.540+0000 7fba3ffff700 1 --2- 172.21.15.1:0/2941971711 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba4015d5b0 0x7fba400c6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:46.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.540+0000 7fba3effd700 1 --2- 172.21.15.1:0/2941971711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba4009a150 0x7fba400b38a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fba2800d7d0 tx=0x7fba2800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:46.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.540+0000 7fba37fff700 1 -- 172.21.15.1:0/2941971711 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba280125f0 con 0x7fba4009a150 2024-03-18T18:50:46.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.540+0000 7fba37fff700 1 -- 172.21.15.1:0/2941971711 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba28012750 con 0x7fba4009a150 2024-03-18T18:50:46.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.540+0000 7fba45eab700 1 -- 172.21.15.1:0/2941971711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba400b6a20 con 0x7fba4009a150 2024-03-18T18:50:46.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.540+0000 7fba37fff700 1 -- 172.21.15.1:0/2941971711 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28012ae0 con 0x7fba4009a150 2024-03-18T18:50:46.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.540+0000 7fba45eab700 1 -- 172.21.15.1:0/2941971711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba400b5e60 con 0x7fba4009a150 2024-03-18T18:50:46.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.541+0000 7fba45eab700 1 -- 172.21.15.1:0/2941971711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba400676a0 con 0x7fba4009a150 2024-03-18T18:50:46.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.541+0000 7fba37fff700 1 -- 172.21.15.1:0/2941971711 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fba28012c40 con 0x7fba4009a150 2024-03-18T18:50:46.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.542+0000 7fba37fff700 1 --2- 172.21.15.1:0/2941971711 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2006ac90 0x7fba2006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:46.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.542+0000 7fba37fff700 1 -- 172.21.15.1:0/2941971711 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fba28089780 con 0x7fba4009a150 2024-03-18T18:50:46.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.542+0000 7fba3f7fe700 1 --2- 172.21.15.1:0/2941971711 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2006ac90 0x7fba2006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:46.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.542+0000 7fba3f7fe700 1 --2- 172.21.15.1:0/2941971711 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2006ac90 0x7fba2006d150 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fba30007fd0 tx=0x7fba30007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:46.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.546+0000 7fba37fff700 1 -- 172.21.15.1:0/2941971711 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fba280545a0 con 0x7fba4009a150 2024-03-18T18:50:46.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.750+0000 7fba45eab700 1 -- 172.21.15.1:0/2941971711 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fba400d0e60 con 0x7fba4009a150 2024-03-18T18:50:46.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.750+0000 7fba37fff700 1 -- 172.21.15.1:0/2941971711 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fba28057bc0 con 0x7fba4009a150 2024-03-18T18:50:46.748 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T18:50:46.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.752+0000 7fba45eab700 1 -- 172.21.15.1:0/2941971711 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2006ac90 msgr2=0x7fba2006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:46.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.752+0000 7fba45eab700 1 --2- 172.21.15.1:0/2941971711 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2006ac90 0x7fba2006d150 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fba30007fd0 tx=0x7fba30007de0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.752+0000 7fba45eab700 1 -- 172.21.15.1:0/2941971711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba4009a150 msgr2=0x7fba400b38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:46.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.752+0000 7fba45eab700 1 --2- 172.21.15.1:0/2941971711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba4009a150 0x7fba400b38a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fba2800d7d0 tx=0x7fba2800dae0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.752+0000 7fba45eab700 1 -- 172.21.15.1:0/2941971711 shutdown_connections 2024-03-18T18:50:46.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.752+0000 7fba45eab700 1 --2- 172.21.15.1:0/2941971711 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2006ac90 0x7fba2006d150 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.752+0000 7fba45eab700 1 --2- 172.21.15.1:0/2941971711 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba4015d5b0 0x7fba400c6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.752+0000 7fba45eab700 1 --2- 172.21.15.1:0/2941971711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba4006ca80 0x7fba4015aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.752+0000 7fba45eab700 1 --2- 172.21.15.1:0/2941971711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba4009a150 0x7fba400b38a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.752+0000 7fba45eab700 1 -- 172.21.15.1:0/2941971711 >> 172.21.15.1:0/2941971711 conn(0x7fba400b2bd0 msgr2=0x7fba40099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:46.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.752+0000 7fba45eab700 1 -- 172.21.15.1:0/2941971711 shutdown_connections 2024-03-18T18:50:46.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.752+0000 7fba45eab700 1 -- 172.21.15.1:0/2941971711 wait complete. 2024-03-18T18:50:46.762 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T18:50:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.893+0000 7fe5d3a89700 1 Processor -- start 2024-03-18T18:50:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.893+0000 7fe5d3a89700 1 -- start start 2024-03-18T18:50:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.894+0000 7fe5d3a89700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe5cc06efb0 0x7fe5cc071da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.894+0000 7fe5d3a89700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5cc06da90 con 0x7fe5cc15d580 2024-03-18T18:50:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.894+0000 7fe5d3a89700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5cc06ab30 con 0x7fe5cc0722e0 2024-03-18T18:50:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.894+0000 7fe5d3a89700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5cc072770 con 0x7fe5cc06efb0 2024-03-18T18:50:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.894+0000 7fe5d1825700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe5cc15d580 0x7fe5cc06d380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32928/0 (socket says 172.21.15.1:32928) 2024-03-18T18:50:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.894+0000 7fe5d1825700 1 -- 172.21.15.1:0/2693009417 learned_addr learned my addr 172.21.15.1:0/2693009417 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.894+0000 7fe5d1024700 1 --2- 172.21.15.1:0/2693009417 >> v2:172.21.15.165:3300/0 conn(0x7fe5cc06efb0 0x7fe5cc071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.894+0000 7fe5d0823700 1 -- 172.21.15.1:0/2693009417 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753356505 0 0) 0x7fe5cc06da90 con 0x7fe5cc15d580 2024-03-18T18:50:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.894+0000 7fe5d0823700 1 -- 172.21.15.1:0/2693009417 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5b80035b0 con 0x7fe5cc15d580 2024-03-18T18:50:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.894+0000 7fe5d0823700 1 -- 172.21.15.1:0/2693009417 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3127171996 0 0) 0x7fe5b80035b0 con 0x7fe5cc15d580 2024-03-18T18:50:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.894+0000 7fe5d1024700 1 -- 172.21.15.1:0/2693009417 >> v1:172.21.15.165:6789/0 conn(0x7fe5cc0722e0 legacy=0x7fe5cc0cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.894+0000 7fe5d1024700 1 -- 172.21.15.1:0/2693009417 >> v1:172.21.15.1:6789/0 conn(0x7fe5cc15d580 legacy=0x7fe5cc06d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.894+0000 7fe5d1024700 1 -- 172.21.15.1:0/2693009417 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5cc06ae10 con 0x7fe5cc06efb0 2024-03-18T18:50:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.895+0000 7fe5d1024700 1 --2- 172.21.15.1:0/2693009417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5cc06efb0 0x7fe5cc071da0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8009870 tx=0x7fe5c8009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1718391292355030 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.895+0000 7fe5d0823700 1 -- 172.21.15.1:0/2693009417 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc001670 con 0x7fe5cc06efb0 2024-03-18T18:50:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.895+0000 7fe5d0823700 1 -- 172.21.15.1:0/2693009417 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c8004030 con 0x7fe5cc06efb0 2024-03-18T18:50:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.895+0000 7fe5d0823700 1 -- 172.21.15.1:0/2693009417 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c80043c0 con 0x7fe5cc06efb0 2024-03-18T18:50:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.895+0000 7fe5d3a89700 1 -- 172.21.15.1:0/2693009417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5cc06efb0 msgr2=0x7fe5cc071da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.895+0000 7fe5d3a89700 1 --2- 172.21.15.1:0/2693009417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5cc06efb0 0x7fe5cc071da0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8009870 tx=0x7fe5c8009b80 comp rx=0 tx=0).stop 2024-03-18T18:50:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.895+0000 7fe5d3a89700 1 -- 172.21.15.1:0/2693009417 shutdown_connections 2024-03-18T18:50:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.895+0000 7fe5d3a89700 1 --2- 172.21.15.1:0/2693009417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5cc06efb0 0x7fe5cc071da0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.895+0000 7fe5d3a89700 1 -- 172.21.15.1:0/2693009417 >> 172.21.15.1:0/2693009417 conn(0x7fe5cc0cca00 msgr2=0x7fe5cc0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.895+0000 7fe5d3a89700 1 -- 172.21.15.1:0/2693009417 shutdown_connections 2024-03-18T18:50:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.895+0000 7fe5d3a89700 1 -- 172.21.15.1:0/2693009417 wait complete. 2024-03-18T18:50:46.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.897+0000 7fe5d3a89700 1 Processor -- start 2024-03-18T18:50:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.897+0000 7fe5d3a89700 1 -- start start 2024-03-18T18:50:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.897+0000 7fe5d3a89700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5cc06efb0 0x7fe5cc15bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.897+0000 7fe5d3a89700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5cc0722e0 0x7fe5cc0c9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.897+0000 7fe5d3a89700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5cc15d580 0x7fe5cc0c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.897+0000 7fe5d3a89700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc0c83f0 con 0x7fe5cc0722e0 2024-03-18T18:50:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.897+0000 7fe5d3a89700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc0c7a20 con 0x7fe5cc06efb0 2024-03-18T18:50:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.897+0000 7fe5d3a89700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc0b5ee0 con 0x7fe5cc15d580 2024-03-18T18:50:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.898+0000 7fe5d1024700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5cc0722e0 0x7fe5cc0c9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.898+0000 7fe5d1024700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5cc0722e0 0x7fe5cc0c9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59226/0 (socket says 172.21.15.1:59226) 2024-03-18T18:50:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.898+0000 7fe5d1024700 1 -- 172.21.15.1:0/2481309376 learned_addr learned my addr 172.21.15.1:0/2481309376 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.898+0000 7fe5d1825700 1 --2- 172.21.15.1:0/2481309376 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5cc06efb0 0x7fe5cc15bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:46.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.898+0000 7fe5d2026700 1 --2- 172.21.15.1:0/2481309376 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5cc15d580 0x7fe5cc0c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:46.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.898+0000 7fe5d1024700 1 -- 172.21.15.1:0/2481309376 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5cc15d580 msgr2=0x7fe5cc0c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:46.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.898+0000 7fe5d1024700 1 --2- 172.21.15.1:0/2481309376 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5cc15d580 0x7fe5cc0c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.898+0000 7fe5d1024700 1 -- 172.21.15.1:0/2481309376 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5cc06efb0 msgr2=0x7fe5cc15bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:46.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.898+0000 7fe5d1024700 1 --2- 172.21.15.1:0/2481309376 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5cc06efb0 0x7fe5cc15bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:46.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.898+0000 7fe5d1024700 1 -- 172.21.15.1:0/2481309376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c80094a0 con 0x7fe5cc0722e0 2024-03-18T18:50:46.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.898+0000 7fe5d1825700 1 --2- 172.21.15.1:0/2481309376 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5cc06efb0 0x7fe5cc15bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:46.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.898+0000 7fe5d1024700 1 --2- 172.21.15.1:0/2481309376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5cc0722e0 0x7fe5cc0c9870 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8000c00 tx=0x7fe5c80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:46.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.898+0000 7fe5c27fc700 1 -- 172.21.15.1:0/2481309376 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8011830 con 0x7fe5cc0722e0 2024-03-18T18:50:46.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.898+0000 7fe5c27fc700 1 -- 172.21.15.1:0/2481309376 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c80119b0 con 0x7fe5cc0722e0 2024-03-18T18:50:46.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.898+0000 7fe5c27fc700 1 -- 172.21.15.1:0/2481309376 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8011d60 con 0x7fe5cc0722e0 2024-03-18T18:50:46.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.898+0000 7fe5d3a89700 1 -- 172.21.15.1:0/2481309376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5cc0c70f0 con 0x7fe5cc0722e0 2024-03-18T18:50:46.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.898+0000 7fe5d3a89700 1 -- 172.21.15.1:0/2481309376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5cc0c76b0 con 0x7fe5cc0722e0 2024-03-18T18:50:46.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.899+0000 7fe5d3a89700 1 -- 172.21.15.1:0/2481309376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5cc0cd770 con 0x7fe5cc0722e0 2024-03-18T18:50:46.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.900+0000 7fe5c27fc700 1 -- 172.21.15.1:0/2481309376 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe5c801b440 con 0x7fe5cc0722e0 2024-03-18T18:50:46.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.900+0000 7fe5c27fc700 1 --2- 172.21.15.1:0/2481309376 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe5b006ac90 0x7fe5b006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:46.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.900+0000 7fe5c27fc700 1 -- 172.21.15.1:0/2481309376 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe5c808ad80 con 0x7fe5cc0722e0 2024-03-18T18:50:46.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.900+0000 7fe5d1825700 1 --2- 172.21.15.1:0/2481309376 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe5b006ac90 0x7fe5b006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:46.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.901+0000 7fe5d1825700 1 --2- 172.21.15.1:0/2481309376 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe5b006ac90 0x7fe5b006d150 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc0017a0 tx=0x7fe5bc001590 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:46.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:46.905+0000 7fe5c27fc700 1 -- 172.21.15.1:0/2481309376 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe5cc0cd770 con 0x7fe5cc0722e0 2024-03-18T18:50:47.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.109+0000 7fe5d3a89700 1 -- 172.21.15.1:0/2481309376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe5cc0cd770 con 0x7fe5cc0722e0 2024-03-18T18:50:47.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.109+0000 7fe5c27fc700 1 -- 172.21.15.1:0/2481309376 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe5cc0cd770 con 0x7fe5cc0722e0 2024-03-18T18:50:47.107 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:47.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.111+0000 7fe5d3a89700 1 -- 172.21.15.1:0/2481309376 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe5b006ac90 msgr2=0x7fe5b006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:47.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.111+0000 7fe5d3a89700 1 --2- 172.21.15.1:0/2481309376 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe5b006ac90 0x7fe5b006d150 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc0017a0 tx=0x7fe5bc001590 comp rx=0 tx=0).stop 2024-03-18T18:50:47.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.111+0000 7fe5d3a89700 1 -- 172.21.15.1:0/2481309376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5cc0722e0 msgr2=0x7fe5cc0c9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:47.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.111+0000 7fe5d3a89700 1 --2- 172.21.15.1:0/2481309376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5cc0722e0 0x7fe5cc0c9870 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8000c00 tx=0x7fe5c80047c0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.112+0000 7fe5d3a89700 1 -- 172.21.15.1:0/2481309376 shutdown_connections 2024-03-18T18:50:47.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.112+0000 7fe5d3a89700 1 --2- 172.21.15.1:0/2481309376 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe5b006ac90 0x7fe5b006d150 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.112+0000 7fe5d3a89700 1 --2- 172.21.15.1:0/2481309376 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe5cc15d580 0x7fe5cc0c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.112+0000 7fe5d3a89700 1 --2- 172.21.15.1:0/2481309376 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe5cc06efb0 0x7fe5cc15bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.112+0000 7fe5d3a89700 1 --2- 172.21.15.1:0/2481309376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5cc0722e0 0x7fe5cc0c9870 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.112+0000 7fe5d3a89700 1 -- 172.21.15.1:0/2481309376 >> 172.21.15.1:0/2481309376 conn(0x7fe5cc0cca00 msgr2=0x7fe5cc0b1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:47.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.112+0000 7fe5d3a89700 1 -- 172.21.15.1:0/2481309376 shutdown_connections 2024-03-18T18:50:47.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.112+0000 7fe5d3a89700 1 -- 172.21.15.1:0/2481309376 wait complete. 2024-03-18T18:50:47.121 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T18:50:47.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.249+0000 7f9ba21c9700 1 Processor -- start 2024-03-18T18:50:47.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.249+0000 7f9ba21c9700 1 -- start start 2024-03-18T18:50:47.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.250+0000 7f9ba21c9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9b9c15d5a0 0x7f9b9c0ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:47.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.250+0000 7f9ba21c9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c0ce550 0x7f9b9c06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:47.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.250+0000 7f9ba21c9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9b9c0cfa90 0x7f9b9c06ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:47.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.250+0000 7f9ba21c9700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b9c0cf600 con 0x7f9b9c15d5a0 2024-03-18T18:50:47.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.250+0000 7f9ba21c9700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b9c0ce9f0 con 0x7f9b9c0cfa90 2024-03-18T18:50:47.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.250+0000 7f9ba21c9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c0cecd0 con 0x7f9b9c0ce550 2024-03-18T18:50:47.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.250+0000 7f9b9b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9b9c15d5a0 0x7f9b9c0ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:47.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.250+0000 7f9b9b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9b9c15d5a0 0x7f9b9c0ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59242/0 (socket says 172.21.15.1:59242) 2024-03-18T18:50:47.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.250+0000 7f9b9b7fe700 1 -- 172.21.15.1:0/31384474 learned_addr learned my addr 172.21.15.1:0/31384474 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:47.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.251+0000 7f9b9bfff700 1 --2- 172.21.15.1:0/31384474 >> v2:172.21.15.165:3300/0 conn(0x7f9b9c0cfa90 0x7f9b9c06ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:47.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.251+0000 7f9b9affd700 1 --2- 172.21.15.1:0/31384474 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c0ce550 0x7f9b9c06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:47.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.251+0000 7f9b9b7fe700 1 -- 172.21.15.1:0/31384474 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c0ce550 msgr2=0x7f9b9c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.251+0000 7f9b9b7fe700 1 --2- 172.21.15.1:0/31384474 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c0ce550 0x7f9b9c06a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.251+0000 7f9b9b7fe700 1 -- 172.21.15.1:0/31384474 >> v2:172.21.15.165:3300/0 conn(0x7f9b9c0cfa90 msgr2=0x7f9b9c06ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.251+0000 7f9b9b7fe700 1 --2- 172.21.15.1:0/31384474 >> v2:172.21.15.165:3300/0 conn(0x7f9b9c0cfa90 0x7f9b9c06ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.251+0000 7f9b9b7fe700 1 -- 172.21.15.1:0/31384474 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c0cefb0 con 0x7f9b9c15d5a0 2024-03-18T18:50:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.251+0000 7f9b9b7fe700 1 --2- 172.21.15.1:0/31384474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b9c15d5a0 0x7f9b9c0ce010 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c0cdec0 tx=0x7f9b84009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4e58f4262ad8988 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.251+0000 7f9b9a7fc700 1 -- 172.21.15.1:0/31384474 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b84005e30 con 0x7f9b9c15d5a0 2024-03-18T18:50:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.251+0000 7f9b9a7fc700 1 -- 172.21.15.1:0/31384474 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b84004030 con 0x7f9b9c15d5a0 2024-03-18T18:50:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.251+0000 7f9b9a7fc700 1 -- 172.21.15.1:0/31384474 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b840043e0 con 0x7f9b9c15d5a0 2024-03-18T18:50:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.252+0000 7f9ba21c9700 1 -- 172.21.15.1:0/31384474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b9c15d5a0 msgr2=0x7f9b9c0ce010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:47.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.252+0000 7f9ba21c9700 1 --2- 172.21.15.1:0/31384474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b9c15d5a0 0x7f9b9c0ce010 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c0cdec0 tx=0x7f9b84009bd0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.252+0000 7f9ba21c9700 1 -- 172.21.15.1:0/31384474 shutdown_connections 2024-03-18T18:50:47.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.252+0000 7f9ba21c9700 1 --2- 172.21.15.1:0/31384474 >> v2:172.21.15.165:3300/0 conn(0x7f9b9c0cfa90 0x7f9b9c06ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.252+0000 7f9ba21c9700 1 --2- 172.21.15.1:0/31384474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b9c15d5a0 0x7f9b9c0ce010 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.252+0000 7f9ba21c9700 1 --2- 172.21.15.1:0/31384474 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c0ce550 0x7f9b9c06a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.252+0000 7f9ba21c9700 1 -- 172.21.15.1:0/31384474 >> 172.21.15.1:0/31384474 conn(0x7f9b9c0cc9b0 msgr2=0x7f9b9c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:47.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.252+0000 7f9ba21c9700 1 -- 172.21.15.1:0/31384474 shutdown_connections 2024-03-18T18:50:47.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.252+0000 7f9ba21c9700 1 -- 172.21.15.1:0/31384474 wait complete. 2024-03-18T18:50:47.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.253+0000 7f9ba21c9700 1 Processor -- start 2024-03-18T18:50:47.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.253+0000 7f9ba21c9700 1 -- start start 2024-03-18T18:50:47.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.254+0000 7f9ba21c9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b9c0ce550 0x7f9b9c15db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:47.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.254+0000 7f9ba21c9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c0cfa90 0x7f9b9c0b55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:47.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.254+0000 7f9ba21c9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b9c15d5a0 0x7f9b9c1593d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:47.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.254+0000 7f9ba21c9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c0b5b00 con 0x7f9b9c0ce550 2024-03-18T18:50:47.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.254+0000 7f9ba21c9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c15e190 con 0x7f9b9c15d5a0 2024-03-18T18:50:47.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.254+0000 7f9ba21c9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c15e470 con 0x7f9b9c0cfa90 2024-03-18T18:50:47.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.254+0000 7f9b9b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b9c0ce550 0x7f9b9c15db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:47.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.254+0000 7f9b9b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b9c0ce550 0x7f9b9c15db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59254/0 (socket says 172.21.15.1:59254) 2024-03-18T18:50:47.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.254+0000 7f9b9b7fe700 1 -- 172.21.15.1:0/166389184 learned_addr learned my addr 172.21.15.1:0/166389184 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:47.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.255+0000 7f9b9bfff700 1 --2- 172.21.15.1:0/166389184 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b9c15d5a0 0x7f9b9c1593d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:47.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.255+0000 7f9b9b7fe700 1 -- 172.21.15.1:0/166389184 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c0cfa90 msgr2=0x7f9b9c0b55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:47.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.255+0000 7f9b9b7fe700 1 --2- 172.21.15.1:0/166389184 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c0cfa90 0x7f9b9c0b55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.255+0000 7f9b9b7fe700 1 -- 172.21.15.1:0/166389184 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b9c15d5a0 msgr2=0x7f9b9c1593d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:47.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.255+0000 7f9b9b7fe700 1 --2- 172.21.15.1:0/166389184 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b9c15d5a0 0x7f9b9c1593d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.255+0000 7f9b9b7fe700 1 -- 172.21.15.1:0/166389184 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b84009500 con 0x7f9b9c0ce550 2024-03-18T18:50:47.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.255+0000 7f9b9b7fe700 1 --2- 172.21.15.1:0/166389184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b9c0ce550 0x7f9b9c15db00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9b84009130 tx=0x7f9b84004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:47.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.255+0000 7f9b93fff700 1 -- 172.21.15.1:0/166389184 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b84003dd0 con 0x7f9b9c0ce550 2024-03-18T18:50:47.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.255+0000 7f9b93fff700 1 -- 172.21.15.1:0/166389184 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b84011420 con 0x7f9b9c0ce550 2024-03-18T18:50:47.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.255+0000 7f9b93fff700 1 -- 172.21.15.1:0/166389184 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b840117d0 con 0x7f9b9c0ce550 2024-03-18T18:50:47.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.255+0000 7f9ba21c9700 1 -- 172.21.15.1:0/166389184 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9c15e750 con 0x7f9b9c0ce550 2024-03-18T18:50:47.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.255+0000 7f9ba21c9700 1 -- 172.21.15.1:0/166389184 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b9c15ed10 con 0x7f9b9c0ce550 2024-03-18T18:50:47.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.256+0000 7f9ba21c9700 1 -- 172.21.15.1:0/166389184 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b9c0676a0 con 0x7f9b9c0ce550 2024-03-18T18:50:47.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.257+0000 7f9b93fff700 1 -- 172.21.15.1:0/166389184 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9b84011930 con 0x7f9b9c0ce550 2024-03-18T18:50:47.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.257+0000 7f9b93fff700 1 --2- 172.21.15.1:0/166389184 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9b7c06ac90 0x7f9b7c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:47.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.257+0000 7f9b93fff700 1 -- 172.21.15.1:0/166389184 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f9b8408a900 con 0x7f9b9c0ce550 2024-03-18T18:50:47.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.257+0000 7f9b9affd700 1 --2- 172.21.15.1:0/166389184 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9b7c06ac90 0x7f9b7c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:47.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.257+0000 7f9b9affd700 1 --2- 172.21.15.1:0/166389184 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9b7c06ac90 0x7f9b7c06d150 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c00a810 tx=0x7f9b8c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:47.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.261+0000 7f9b93fff700 1 -- 172.21.15.1:0/166389184 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9b840557b0 con 0x7f9b9c0ce550 2024-03-18T18:50:47.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.477+0000 7f9ba21c9700 1 -- 172.21.15.1:0/166389184 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9b9c0ae9c0 con 0x7f9b9c0ce550 2024-03-18T18:50:47.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.478+0000 7f9b93fff700 1 -- 172.21.15.1:0/166389184 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9b84058dd0 con 0x7f9b9c0ce550 2024-03-18T18:50:47.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.480+0000 7f9ba21c9700 1 -- 172.21.15.1:0/166389184 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9b7c06ac90 msgr2=0x7f9b7c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:47.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.480+0000 7f9ba21c9700 1 --2- 172.21.15.1:0/166389184 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9b7c06ac90 0x7f9b7c06d150 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c00a810 tx=0x7f9b8c008040 comp rx=0 tx=0).stop 2024-03-18T18:50:47.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.480+0000 7f9ba21c9700 1 -- 172.21.15.1:0/166389184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b9c0ce550 msgr2=0x7f9b9c15db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:47.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.480+0000 7f9ba21c9700 1 --2- 172.21.15.1:0/166389184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b9c0ce550 0x7f9b9c15db00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9b84009130 tx=0x7f9b84004750 comp rx=0 tx=0).stop 2024-03-18T18:50:47.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.480+0000 7f9ba21c9700 1 -- 172.21.15.1:0/166389184 shutdown_connections 2024-03-18T18:50:47.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.480+0000 7f9ba21c9700 1 --2- 172.21.15.1:0/166389184 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9b7c06ac90 0x7f9b7c06d150 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.480+0000 7f9ba21c9700 1 --2- 172.21.15.1:0/166389184 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b9c15d5a0 0x7f9b9c1593d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.480+0000 7f9ba21c9700 1 --2- 172.21.15.1:0/166389184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b9c0ce550 0x7f9b9c15db00 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.480+0000 7f9ba21c9700 1 --2- 172.21.15.1:0/166389184 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b9c0cfa90 0x7f9b9c0b55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.480+0000 7f9ba21c9700 1 -- 172.21.15.1:0/166389184 >> 172.21.15.1:0/166389184 conn(0x7f9b9c0cc9b0 msgr2=0x7f9b9c06e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:47.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.480+0000 7f9ba21c9700 1 -- 172.21.15.1:0/166389184 shutdown_connections 2024-03-18T18:50:47.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.480+0000 7f9ba21c9700 1 -- 172.21.15.1:0/166389184 wait complete. 2024-03-18T18:50:47.479 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-18T18:50:47.490 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi165.front.sepia.ceph.com... 2024-03-18T18:50:47.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:47.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-18T18:50:47.543 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T18:50:47.543 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T18:50:47.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:47.543 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-18T18:50:47.581 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:50:47.581 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T18:50:47.600 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:50:47.600 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T18:50:47.659 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:47.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.828+0000 7ffb60ac5700 1 Processor -- start 2024-03-18T18:50:47.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.828+0000 7ffb60ac5700 1 -- start start 2024-03-18T18:50:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.829+0000 7ffb60ac5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb5c143c20 0x7ffb5c0b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.829+0000 7ffb60ac5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ffb5c0b3f40 0x7ffb5c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.829+0000 7ffb60ac5700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb5c144000 con 0x7ffb5c06f070 2024-03-18T18:50:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.829+0000 7ffb60ac5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb5c0b43c0 con 0x7ffb5c0b3f40 2024-03-18T18:50:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.829+0000 7ffb60ac5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c0b51b0 con 0x7ffb5c143c20 2024-03-18T18:50:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.829+0000 7ffb5a59c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb5c143c20 0x7ffb5c0b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.829+0000 7ffb59d9b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ffb5c0b3f40 0x7ffb5c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:47.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.829+0000 7ffb59d9b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ffb5c0b3f40 0x7ffb5c06eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:54922/0 (socket says 172.21.15.1:54922) 2024-03-18T18:50:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.829+0000 7ffb59d9b700 1 -- 172.21.15.1:0/242416092 learned_addr learned my addr 172.21.15.1:0/242416092 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.829+0000 7ffb5ad9d700 1 --1- 172.21.15.1:0/242416092 >> v1:172.21.15.165:6789/0 conn(0x7ffb5c06f070 0x7ffb5c071650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:42514/0 (socket says 172.21.15.1:42514) 2024-03-18T18:50:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.829+0000 7ffb5a59c700 1 -- 172.21.15.1:0/242416092 >> v2:172.21.15.165:3300/0 conn(0x7ffb5c0b3f40 msgr2=0x7ffb5c06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.829+0000 7ffb5a59c700 1 --2- 172.21.15.1:0/242416092 >> v2:172.21.15.165:3300/0 conn(0x7ffb5c0b3f40 0x7ffb5c06eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.829+0000 7ffb5a59c700 1 -- 172.21.15.1:0/242416092 >> v1:172.21.15.165:6789/0 conn(0x7ffb5c06f070 legacy=0x7ffb5c071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.829+0000 7ffb5a59c700 1 -- 172.21.15.1:0/242416092 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5c0cd8d0 con 0x7ffb5c143c20 2024-03-18T18:50:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.830+0000 7ffb59d9b700 1 --2- 172.21.15.1:0/242416092 >> v2:172.21.15.165:3300/0 conn(0x7ffb5c0b3f40 0x7ffb5c06eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:50:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.830+0000 7ffb5a59c700 1 --2- 172.21.15.1:0/242416092 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb5c143c20 0x7ffb5c0b3a00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00bae0 tx=0x7ffb4c00bdf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da0381e9d081bbc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.830+0000 7ffb5959a700 1 -- 172.21.15.1:0/242416092 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c00dc70 con 0x7ffb5c143c20 2024-03-18T18:50:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.830+0000 7ffb5959a700 1 -- 172.21.15.1:0/242416092 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb4c00ddd0 con 0x7ffb5c143c20 2024-03-18T18:50:47.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.830+0000 7ffb5959a700 1 -- 172.21.15.1:0/242416092 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c010690 con 0x7ffb5c143c20 2024-03-18T18:50:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.830+0000 7ffb60ac5700 1 -- 172.21.15.1:0/242416092 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb5c143c20 msgr2=0x7ffb5c0b3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.830+0000 7ffb60ac5700 1 --2- 172.21.15.1:0/242416092 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb5c143c20 0x7ffb5c0b3a00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00bae0 tx=0x7ffb4c00bdf0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.830+0000 7ffb60ac5700 1 -- 172.21.15.1:0/242416092 shutdown_connections 2024-03-18T18:50:47.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.830+0000 7ffb60ac5700 1 --2- 172.21.15.1:0/242416092 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb5c143c20 0x7ffb5c0b3a00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.830+0000 7ffb60ac5700 1 --2- 172.21.15.1:0/242416092 >> v2:172.21.15.165:3300/0 conn(0x7ffb5c0b3f40 0x7ffb5c06eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.830+0000 7ffb60ac5700 1 -- 172.21.15.1:0/242416092 >> 172.21.15.1:0/242416092 conn(0x7ffb5c0b1d20 msgr2=0x7ffb5c0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.831+0000 7ffb60ac5700 1 -- 172.21.15.1:0/242416092 shutdown_connections 2024-03-18T18:50:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.831+0000 7ffb60ac5700 1 -- 172.21.15.1:0/242416092 wait complete. 2024-03-18T18:50:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.832+0000 7ffb60ac5700 1 Processor -- start 2024-03-18T18:50:47.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.832+0000 7ffb60ac5700 1 -- start start 2024-03-18T18:50:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb60ac5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb5c06f070 0x7ffb5c15bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb60ac5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb5c0b3f40 0x7ffb5c13fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb60ac5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb5c143c20 0x7ffb5c1401a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb60ac5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c1493e0 con 0x7ffb5c06f070 2024-03-18T18:50:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb60ac5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c148a60 con 0x7ffb5c0b3f40 2024-03-18T18:50:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb60ac5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c148d20 con 0x7ffb5c143c20 2024-03-18T18:50:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb5a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb5c06f070 0x7ffb5c15bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb5a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb5c06f070 0x7ffb5c15bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59264/0 (socket says 172.21.15.1:59264) 2024-03-18T18:50:47.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb5a59c700 1 -- 172.21.15.1:0/1026608067 learned_addr learned my addr 172.21.15.1:0/1026608067 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb59d9b700 1 --2- 172.21.15.1:0/1026608067 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb5c0b3f40 0x7ffb5c13fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb5a59c700 1 -- 172.21.15.1:0/1026608067 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb5c143c20 msgr2=0x7ffb5c1401a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb5ad9d700 1 --2- 172.21.15.1:0/1026608067 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb5c143c20 0x7ffb5c1401a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb5a59c700 1 --2- 172.21.15.1:0/1026608067 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb5c143c20 0x7ffb5c1401a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb5a59c700 1 -- 172.21.15.1:0/1026608067 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb5c0b3f40 msgr2=0x7ffb5c13fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb5a59c700 1 --2- 172.21.15.1:0/1026608067 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb5c0b3f40 0x7ffb5c13fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb5a59c700 1 -- 172.21.15.1:0/1026608067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb50009550 con 0x7ffb5c06f070 2024-03-18T18:50:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb59d9b700 1 --2- 172.21.15.1:0/1026608067 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb5c0b3f40 0x7ffb5c13fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb5a59c700 1 --2- 172.21.15.1:0/1026608067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb5c06f070 0x7ffb5c15bef0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00b2c0 tx=0x7ffb4c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb4affd700 1 -- 172.21.15.1:0/1026608067 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c01f070 con 0x7ffb5c06f070 2024-03-18T18:50:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb4affd700 1 -- 172.21.15.1:0/1026608067 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb4c0039f0 con 0x7ffb5c06f070 2024-03-18T18:50:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb60ac5700 1 -- 172.21.15.1:0/1026608067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4c00b740 con 0x7ffb5c06f070 2024-03-18T18:50:47.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb4affd700 1 -- 172.21.15.1:0/1026608067 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c003da0 con 0x7ffb5c06f070 2024-03-18T18:50:47.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.833+0000 7ffb60ac5700 1 -- 172.21.15.1:0/1026608067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb5c13d2f0 con 0x7ffb5c06f070 2024-03-18T18:50:47.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.835+0000 7ffb60ac5700 1 -- 172.21.15.1:0/1026608067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5c0676a0 con 0x7ffb5c06f070 2024-03-18T18:50:47.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.835+0000 7ffb4affd700 1 -- 172.21.15.1:0/1026608067 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ffb4c019400 con 0x7ffb5c06f070 2024-03-18T18:50:47.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.835+0000 7ffb4affd700 1 --2- 172.21.15.1:0/1026608067 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffb3c06ac90 0x7ffb3c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:47.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.835+0000 7ffb4affd700 1 -- 172.21.15.1:0/1026608067 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ffb4c08cad0 con 0x7ffb5c06f070 2024-03-18T18:50:47.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.835+0000 7ffb59d9b700 1 --2- 172.21.15.1:0/1026608067 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffb3c06ac90 0x7ffb3c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:47.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.836+0000 7ffb59d9b700 1 --2- 172.21.15.1:0/1026608067 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffb3c06ac90 0x7ffb3c06d150 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c142c90 tx=0x7ffb50008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:47.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:47.840+0000 7ffb4affd700 1 -- 172.21.15.1:0/1026608067 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffb4c057980 con 0x7ffb5c06f070 2024-03-18T18:50:47.949 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:50:47.952 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:50:48.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.046+0000 7ffb60ac5700 1 -- 172.21.15.1:0/1026608067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb5c06c870 con 0x7ffb5c06f070 2024-03-18T18:50:48.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.047+0000 7ffb4affd700 1 -- 172.21.15.1:0/1026608067 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18217 (secure 0 0 0) 0x7ffb4c05afa0 con 0x7ffb5c06f070 2024-03-18T18:50:48.044 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:48.045 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":27,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:50:34.993482+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:48.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.049+0000 7ffb60ac5700 1 -- 172.21.15.1:0/1026608067 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffb3c06ac90 msgr2=0x7ffb3c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:48.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.049+0000 7ffb60ac5700 1 --2- 172.21.15.1:0/1026608067 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffb3c06ac90 0x7ffb3c06d150 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c142c90 tx=0x7ffb50008040 comp rx=0 tx=0).stop 2024-03-18T18:50:48.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.049+0000 7ffb60ac5700 1 -- 172.21.15.1:0/1026608067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb5c06f070 msgr2=0x7ffb5c15bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:48.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.049+0000 7ffb60ac5700 1 --2- 172.21.15.1:0/1026608067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb5c06f070 0x7ffb5c15bef0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00b2c0 tx=0x7ffb4c004750 comp rx=0 tx=0).stop 2024-03-18T18:50:48.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.049+0000 7ffb60ac5700 1 -- 172.21.15.1:0/1026608067 shutdown_connections 2024-03-18T18:50:48.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.049+0000 7ffb60ac5700 1 --2- 172.21.15.1:0/1026608067 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffb3c06ac90 0x7ffb3c06d150 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.049+0000 7ffb60ac5700 1 --2- 172.21.15.1:0/1026608067 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb5c143c20 0x7ffb5c1401a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.049+0000 7ffb60ac5700 1 --2- 172.21.15.1:0/1026608067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb5c06f070 0x7ffb5c15bef0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.049+0000 7ffb60ac5700 1 --2- 172.21.15.1:0/1026608067 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb5c0b3f40 0x7ffb5c13fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.049+0000 7ffb60ac5700 1 -- 172.21.15.1:0/1026608067 >> 172.21.15.1:0/1026608067 conn(0x7ffb5c0b1d20 msgr2=0x7ffb5c071750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:48.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.050+0000 7ffb60ac5700 1 -- 172.21.15.1:0/1026608067 shutdown_connections 2024-03-18T18:50:48.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.050+0000 7ffb60ac5700 1 -- 172.21.15.1:0/1026608067 wait complete. 2024-03-18T18:50:48.059 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T18:49:42.800801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T18:49:43.776684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T18:49:46.854608+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T18:50:48.060 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T18:50:48.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.230+0000 7f691418c700 1 Processor -- start 2024-03-18T18:50:48.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.230+0000 7f691418c700 1 -- start start 2024-03-18T18:50:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.231+0000 7f691418c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f690c15d780 0x7f690c0cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.231+0000 7f691418c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f690c0ce5a0 0x7f690c0cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.231+0000 7f691418c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f690c0ce090 con 0x7f690c09dd40 2024-03-18T18:50:48.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.231+0000 7f691418c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f690c0cb480 con 0x7f690c0ce5a0 2024-03-18T18:50:48.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.231+0000 7f691418c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f690c06db30 con 0x7f690c15d780 2024-03-18T18:50:48.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.231+0000 7f6911727700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f690c09dd40 0x7f690c0cad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32930/0 (socket says 172.21.15.1:32930) 2024-03-18T18:50:48.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.231+0000 7f6911727700 1 -- 172.21.15.1:0/4007485458 learned_addr learned my addr 172.21.15.1:0/4007485458 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:48.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.231+0000 7f6911f28700 1 --2- 172.21.15.1:0/4007485458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f690c15d780 0x7f690c0cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.231+0000 7f6912729700 1 --2- 172.21.15.1:0/4007485458 >> v2:172.21.15.165:3300/0 conn(0x7f690c0ce5a0 0x7f690c0cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.231+0000 7f6910f26700 1 -- 172.21.15.1:0/4007485458 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2833961416 0 0) 0x7f690c0ce090 con 0x7f690c09dd40 2024-03-18T18:50:48.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.231+0000 7f6910f26700 1 -- 172.21.15.1:0/4007485458 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68f80035b0 con 0x7f690c09dd40 2024-03-18T18:50:48.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6910f26700 1 -- 172.21.15.1:0/4007485458 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1382824453 0 0) 0x7f68f80035b0 con 0x7f690c09dd40 2024-03-18T18:50:48.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6910f26700 1 -- 172.21.15.1:0/4007485458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f690c15d780 msgr2=0x7f690c0cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:48.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6910f26700 1 --2- 172.21.15.1:0/4007485458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f690c15d780 0x7f690c0cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6910f26700 1 -- 172.21.15.1:0/4007485458 >> v2:172.21.15.165:3300/0 conn(0x7f690c0ce5a0 msgr2=0x7f690c0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:48.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6910f26700 1 --2- 172.21.15.1:0/4007485458 >> v2:172.21.15.165:3300/0 conn(0x7f690c0ce5a0 0x7f690c0cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6910f26700 1 -- 172.21.15.1:0/4007485458 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f690c06d520 con 0x7f690c09dd40 2024-03-18T18:50:48.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6910f26700 1 -- 172.21.15.1:0/4007485458 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f6908001670 con 0x7f690c09dd40 2024-03-18T18:50:48.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6910f26700 1 -- 172.21.15.1:0/4007485458 >> v1:172.21.15.1:6789/0 conn(0x7f690c09dd40 legacy=0x7f690c0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:48.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6910f26700 1 --2- 172.21.15.1:0/4007485458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f690c15d780 0x7f68f8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6910f26700 1 --2- 172.21.15.1:0/4007485458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68f8003f80 0x7f68f8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6910f26700 1 --2- 172.21.15.1:0/4007485458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f8004d90 0x7f68f8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6910f26700 1 -- 172.21.15.1:0/4007485458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c0cb480 con 0x7f68f8004d90 2024-03-18T18:50:48.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6910f26700 1 -- 172.21.15.1:0/4007485458 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c06db30 con 0x7f68f8003f80 2024-03-18T18:50:48.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6910f26700 1 -- 172.21.15.1:0/4007485458 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f800ba10 con 0x7f690c15d780 2024-03-18T18:50:48.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6912729700 1 --2- 172.21.15.1:0/4007485458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f8004d90 0x7f68f8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6911f28700 1 --2- 172.21.15.1:0/4007485458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f690c15d780 0x7f68f8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6911727700 1 --2- 172.21.15.1:0/4007485458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68f8003f80 0x7f68f8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6912729700 1 -- 172.21.15.1:0/4007485458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f690c15d780 msgr2=0x7f68f8003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:48.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6912729700 1 --2- 172.21.15.1:0/4007485458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f690c15d780 0x7f68f8003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6912729700 1 -- 172.21.15.1:0/4007485458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68f8003f80 msgr2=0x7f68f8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:48.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6912729700 1 --2- 172.21.15.1:0/4007485458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68f8003f80 0x7f68f8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6912729700 1 -- 172.21.15.1:0/4007485458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f80035b0 con 0x7f68f8004d90 2024-03-18T18:50:48.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.232+0000 7f6912729700 1 --2- 172.21.15.1:0/4007485458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f8004d90 0x7f68f8005210 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6904009fc0 tx=0x7f68f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79bcd27ed050f05 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:48.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.233+0000 7f6910f26700 1 -- 172.21.15.1:0/4007485458 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690400ebd0 con 0x7f68f8004d90 2024-03-18T18:50:48.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.233+0000 7f6910f26700 1 -- 172.21.15.1:0/4007485458 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f690400f040 con 0x7f68f8004d90 2024-03-18T18:50:48.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.233+0000 7f6910f26700 1 -- 172.21.15.1:0/4007485458 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6904014480 con 0x7f68f8004d90 2024-03-18T18:50:48.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.233+0000 7f691418c700 1 -- 172.21.15.1:0/4007485458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f8004d90 msgr2=0x7f68f8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:48.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.233+0000 7f691418c700 1 --2- 172.21.15.1:0/4007485458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68f8004d90 0x7f68f8005210 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6904009fc0 tx=0x7f68f8005880 comp rx=0 tx=0).stop 2024-03-18T18:50:48.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.233+0000 7f6912729700 1 -- 172.21.15.1:0/4007485458 reap_dead start 2024-03-18T18:50:48.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.233+0000 7f691418c700 1 -- 172.21.15.1:0/4007485458 shutdown_connections 2024-03-18T18:50:48.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.233+0000 7f691418c700 1 -- 172.21.15.1:0/4007485458 >> 172.21.15.1:0/4007485458 conn(0x7f690c0cca00 msgr2=0x7f690c0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:48.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.233+0000 7f691418c700 1 -- 172.21.15.1:0/4007485458 shutdown_connections 2024-03-18T18:50:48.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.233+0000 7f691418c700 1 -- 172.21.15.1:0/4007485458 wait complete. 2024-03-18T18:50:48.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.235+0000 7f691418c700 1 Processor -- start 2024-03-18T18:50:48.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.235+0000 7f691418c700 1 -- start start 2024-03-18T18:50:48.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.236+0000 7f691418c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f690c155340 0x7f690c0c4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.236+0000 7f691418c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f690c0c3a90 0x7f690c0adf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.236+0000 7f691418c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f690c155720 0x7f690c0ae4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.236+0000 7f691418c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c0c8d30 con 0x7f690c155340 2024-03-18T18:50:48.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.236+0000 7f691418c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c0af8d0 con 0x7f690c155720 2024-03-18T18:50:48.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.236+0000 7f691418c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f690c0c80f0 con 0x7f690c0c3a90 2024-03-18T18:50:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.236+0000 7f6911f28700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f690c155340 0x7f690c0c4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.236+0000 7f6911f28700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f690c155340 0x7f690c0c4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59278/0 (socket says 172.21.15.1:59278) 2024-03-18T18:50:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.236+0000 7f6911f28700 1 -- 172.21.15.1:0/3471915064 learned_addr learned my addr 172.21.15.1:0/3471915064 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.236+0000 7f6912729700 1 --2- 172.21.15.1:0/3471915064 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f690c155720 0x7f690c0ae4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.236+0000 7f6911727700 1 --2- 172.21.15.1:0/3471915064 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f690c0c3a90 0x7f690c0adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.236+0000 7f6911f28700 1 -- 172.21.15.1:0/3471915064 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f690c0c3a90 msgr2=0x7f690c0adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.236+0000 7f6911f28700 1 --2- 172.21.15.1:0/3471915064 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f690c0c3a90 0x7f690c0adf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.236+0000 7f6911f28700 1 -- 172.21.15.1:0/3471915064 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f690c155720 msgr2=0x7f690c0ae4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.236+0000 7f6911f28700 1 --2- 172.21.15.1:0/3471915064 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f690c155720 0x7f690c0ae4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.236+0000 7f6911f28700 1 -- 172.21.15.1:0/3471915064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f690c0c4960 con 0x7f690c155340 2024-03-18T18:50:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.236+0000 7f6912729700 1 --2- 172.21.15.1:0/3471915064 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f690c155720 0x7f690c0ae4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.236+0000 7f6911f28700 1 --2- 172.21.15.1:0/3471915064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f690c155340 0x7f690c0c4420 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f68fc011aa0 tx=0x7f68fc011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.237+0000 7f69027fc700 1 -- 172.21.15.1:0/3471915064 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc00f920 con 0x7f690c155340 2024-03-18T18:50:48.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.237+0000 7f69027fc700 1 -- 172.21.15.1:0/3471915064 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68fc00fa80 con 0x7f690c155340 2024-03-18T18:50:48.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.237+0000 7f691418c700 1 -- 172.21.15.1:0/3471915064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f690c0c8390 con 0x7f690c155340 2024-03-18T18:50:48.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.237+0000 7f69027fc700 1 -- 172.21.15.1:0/3471915064 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc00fe30 con 0x7f690c155340 2024-03-18T18:50:48.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.237+0000 7f691418c700 1 -- 172.21.15.1:0/3471915064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f690c0c8950 con 0x7f690c155340 2024-03-18T18:50:48.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.238+0000 7f691418c700 1 -- 172.21.15.1:0/3471915064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f690c0c8e70 con 0x7f690c155340 2024-03-18T18:50:48.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.238+0000 7f69027fc700 1 -- 172.21.15.1:0/3471915064 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f68fc014440 con 0x7f690c155340 2024-03-18T18:50:48.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.239+0000 7f69027fc700 1 --2- 172.21.15.1:0/3471915064 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f68f006acc0 0x7f68f006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.239+0000 7f6911727700 1 --2- 172.21.15.1:0/3471915064 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f68f006acc0 0x7f68f006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.239+0000 7f69027fc700 1 -- 172.21.15.1:0/3471915064 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f68fc089e80 con 0x7f690c155340 2024-03-18T18:50:48.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.239+0000 7f6911727700 1 --2- 172.21.15.1:0/3471915064 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f68f006acc0 0x7f68f006d180 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f69080048d0 tx=0x7f6908009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:48.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.243+0000 7f69027fc700 1 -- 172.21.15.1:0/3471915064 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f690c0c8e70 con 0x7f690c155340 2024-03-18T18:50:48.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.449+0000 7f691418c700 1 -- 172.21.15.1:0/3471915064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f690c0c8e70 con 0x7f690c155340 2024-03-18T18:50:48.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.449+0000 7f69027fc700 1 -- 172.21.15.1:0/3471915064 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f690c0c8e70 con 0x7f690c155340 2024-03-18T18:50:48.446 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T18:50:48.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.451+0000 7f691418c700 1 -- 172.21.15.1:0/3471915064 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f68f006acc0 msgr2=0x7f68f006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:48.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.451+0000 7f691418c700 1 --2- 172.21.15.1:0/3471915064 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f68f006acc0 0x7f68f006d180 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f69080048d0 tx=0x7f6908009040 comp rx=0 tx=0).stop 2024-03-18T18:50:48.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.451+0000 7f691418c700 1 -- 172.21.15.1:0/3471915064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f690c155340 msgr2=0x7f690c0c4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:48.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.451+0000 7f691418c700 1 --2- 172.21.15.1:0/3471915064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f690c155340 0x7f690c0c4420 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f68fc011aa0 tx=0x7f68fc011e60 comp rx=0 tx=0).stop 2024-03-18T18:50:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.451+0000 7f691418c700 1 -- 172.21.15.1:0/3471915064 shutdown_connections 2024-03-18T18:50:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.451+0000 7f691418c700 1 --2- 172.21.15.1:0/3471915064 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f68f006acc0 0x7f68f006d180 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.451+0000 7f691418c700 1 --2- 172.21.15.1:0/3471915064 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f690c155720 0x7f690c0ae4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.451+0000 7f691418c700 1 --2- 172.21.15.1:0/3471915064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f690c155340 0x7f690c0c4420 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.451+0000 7f691418c700 1 --2- 172.21.15.1:0/3471915064 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f690c0c3a90 0x7f690c0adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.451+0000 7f691418c700 1 -- 172.21.15.1:0/3471915064 >> 172.21.15.1:0/3471915064 conn(0x7f690c0cca00 msgr2=0x7f690c0afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.451+0000 7f691418c700 1 -- 172.21.15.1:0/3471915064 shutdown_connections 2024-03-18T18:50:48.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.451+0000 7f691418c700 1 -- 172.21.15.1:0/3471915064 wait complete. 2024-03-18T18:50:48.461 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T18:50:48.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.592+0000 7f3435330700 1 Processor -- start 2024-03-18T18:50:48.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.592+0000 7f3435330700 1 -- start start 2024-03-18T18:50:48.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.593+0000 7f3435330700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f343006a750 0x7f343006ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.593+0000 7f3435330700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f343015b900 0x7f343015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.593+0000 7f3435330700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343006b070 con 0x7f343006d560 2024-03-18T18:50:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.593+0000 7f3435330700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343015ef00 con 0x7f343015b900 2024-03-18T18:50:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.593+0000 7f3435330700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343015f1e0 con 0x7f343006a750 2024-03-18T18:50:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.593+0000 7f342e7fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f343006d560 0x7f34300aefd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32946/0 (socket says 172.21.15.1:32946) 2024-03-18T18:50:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.593+0000 7f342f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f343015b900 0x7f343015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.593+0000 7f342e7fc700 1 -- 172.21.15.1:0/519769862 learned_addr learned my addr 172.21.15.1:0/519769862 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.593+0000 7f342effd700 1 --2- 172.21.15.1:0/519769862 >> v2:172.21.15.165:3300/0 conn(0x7f343006a750 0x7f343006ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.594+0000 7f342f7fe700 1 -- 172.21.15.1:0/519769862 >> v2:172.21.15.165:3300/0 conn(0x7f343006a750 msgr2=0x7f343006ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.594+0000 7f342f7fe700 1 --2- 172.21.15.1:0/519769862 >> v2:172.21.15.165:3300/0 conn(0x7f343006a750 0x7f343006ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.594+0000 7f342f7fe700 1 -- 172.21.15.1:0/519769862 >> v1:172.21.15.1:6789/0 conn(0x7f343006d560 legacy=0x7f34300aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.594+0000 7f342f7fe700 1 -- 172.21.15.1:0/519769862 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343015f4c0 con 0x7f343015b900 2024-03-18T18:50:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.594+0000 7f342dffb700 1 -- 172.21.15.1:0/519769862 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615471295 0 0) 0x7f343006b070 con 0x7f343006d560 2024-03-18T18:50:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.594+0000 7f342f7fe700 1 --2- 172.21.15.1:0/519769862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f343015b900 0x7f343015bd70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f34300b2a70 tx=0x7f342800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d788e3fd5a400d5a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.594+0000 7f342dffb700 1 -- 172.21.15.1:0/519769862 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342800e070 con 0x7f343015b900 2024-03-18T18:50:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.594+0000 7f342dffb700 1 -- 172.21.15.1:0/519769862 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342800bb20 con 0x7f343015b900 2024-03-18T18:50:48.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.594+0000 7f342dffb700 1 -- 172.21.15.1:0/519769862 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3428012430 con 0x7f343015b900 2024-03-18T18:50:48.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.594+0000 7f3435330700 1 -- 172.21.15.1:0/519769862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f343015b900 msgr2=0x7f343015bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:48.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.594+0000 7f3435330700 1 --2- 172.21.15.1:0/519769862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f343015b900 0x7f343015bd70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f34300b2a70 tx=0x7f342800b330 comp rx=0 tx=0).stop 2024-03-18T18:50:48.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.594+0000 7f3435330700 1 -- 172.21.15.1:0/519769862 shutdown_connections 2024-03-18T18:50:48.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.594+0000 7f3435330700 1 --2- 172.21.15.1:0/519769862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f343015b900 0x7f343015bd70 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.594+0000 7f3435330700 1 --2- 172.21.15.1:0/519769862 >> v2:172.21.15.165:3300/0 conn(0x7f343006a750 0x7f343006ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.594+0000 7f3435330700 1 -- 172.21.15.1:0/519769862 >> 172.21.15.1:0/519769862 conn(0x7f34300cff30 msgr2=0x7f34300d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:48.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.595+0000 7f3435330700 1 -- 172.21.15.1:0/519769862 shutdown_connections 2024-03-18T18:50:48.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.595+0000 7f3435330700 1 -- 172.21.15.1:0/519769862 wait complete. 2024-03-18T18:50:48.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.596+0000 7f3435330700 1 Processor -- start 2024-03-18T18:50:48.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.596+0000 7f3435330700 1 -- start start 2024-03-18T18:50:48.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f3435330700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f343006a750 0x7f34301e4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f3435330700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f343006d560 0x7f34300c94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f3435330700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f343015b900 0x7f3430155650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f3435330700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34300c99e0 con 0x7f343006a750 2024-03-18T18:50:48.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f3435330700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343015c970 con 0x7f343015b900 2024-03-18T18:50:48.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f3435330700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34301e5250 con 0x7f343006d560 2024-03-18T18:50:48.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f342effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f343006a750 0x7f34301e4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f342effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f343006a750 0x7f34301e4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59300/0 (socket says 172.21.15.1:59300) 2024-03-18T18:50:48.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f342effd700 1 -- 172.21.15.1:0/301017218 learned_addr learned my addr 172.21.15.1:0/301017218 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:48.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f342effd700 1 -- 172.21.15.1:0/301017218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f343006d560 msgr2=0x7f34300c94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:48.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f342f7fe700 1 --2- 172.21.15.1:0/301017218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f343015b900 0x7f3430155650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f342e7fc700 1 --2- 172.21.15.1:0/301017218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f343006d560 0x7f34300c94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f342effd700 1 --2- 172.21.15.1:0/301017218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f343006d560 0x7f34300c94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f342effd700 1 -- 172.21.15.1:0/301017218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f343015b900 msgr2=0x7f3430155650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:48.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f342effd700 1 --2- 172.21.15.1:0/301017218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f343015b900 0x7f3430155650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f342effd700 1 -- 172.21.15.1:0/301017218 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3428009a70 con 0x7f343006a750 2024-03-18T18:50:48.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f342f7fe700 1 --2- 172.21.15.1:0/301017218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f343015b900 0x7f3430155650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:48.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f342e7fc700 1 --2- 172.21.15.1:0/301017218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f343006d560 0x7f34300c94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:48.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f342effd700 1 --2- 172.21.15.1:0/301017218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f343006a750 0x7f34301e4b70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f342400e9b0 tx=0x7f342400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:48.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f34177fe700 1 -- 172.21.15.1:0/301017218 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3424009710 con 0x7f343006a750 2024-03-18T18:50:48.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f34177fe700 1 -- 172.21.15.1:0/301017218 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342400c850 con 0x7f343006a750 2024-03-18T18:50:48.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f3435330700 1 -- 172.21.15.1:0/301017218 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34301e5530 con 0x7f343006a750 2024-03-18T18:50:48.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f34177fe700 1 -- 172.21.15.1:0/301017218 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342400cbc0 con 0x7f343006a750 2024-03-18T18:50:48.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.597+0000 7f3435330700 1 -- 172.21.15.1:0/301017218 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34301e5af0 con 0x7f343006a750 2024-03-18T18:50:48.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.598+0000 7f3435330700 1 -- 172.21.15.1:0/301017218 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343015cc70 con 0x7f343006a750 2024-03-18T18:50:48.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.599+0000 7f34177fe700 1 -- 172.21.15.1:0/301017218 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f342400cd20 con 0x7f343006a750 2024-03-18T18:50:48.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.599+0000 7f34177fe700 1 --2- 172.21.15.1:0/301017218 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f341c06ac90 0x7f341c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.599+0000 7f34177fe700 1 -- 172.21.15.1:0/301017218 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3424089750 con 0x7f343006a750 2024-03-18T18:50:48.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.599+0000 7f342e7fc700 1 --2- 172.21.15.1:0/301017218 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f341c06ac90 0x7f341c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.600+0000 7f342e7fc700 1 --2- 172.21.15.1:0/301017218 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f341c06ac90 0x7f341c06d150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3430156740 tx=0x7f3420009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:48.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.604+0000 7f34177fe700 1 -- 172.21.15.1:0/301017218 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3424054600 con 0x7f343006a750 2024-03-18T18:50:48.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.803+0000 7f3435330700 1 -- 172.21.15.1:0/301017218 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f34300676a0 con 0x7f343006a750 2024-03-18T18:50:48.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.803+0000 7f34177fe700 1 -- 172.21.15.1:0/301017218 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3424057c20 con 0x7f343006a750 2024-03-18T18:50:48.801 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:48.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.805+0000 7f3435330700 1 -- 172.21.15.1:0/301017218 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f341c06ac90 msgr2=0x7f341c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:48.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.805+0000 7f3435330700 1 --2- 172.21.15.1:0/301017218 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f341c06ac90 0x7f341c06d150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3430156740 tx=0x7f3420009040 comp rx=0 tx=0).stop 2024-03-18T18:50:48.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.805+0000 7f3435330700 1 -- 172.21.15.1:0/301017218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f343006a750 msgr2=0x7f34301e4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:48.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.805+0000 7f3435330700 1 --2- 172.21.15.1:0/301017218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f343006a750 0x7f34301e4b70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f342400e9b0 tx=0x7f342400ed70 comp rx=0 tx=0).stop 2024-03-18T18:50:48.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.805+0000 7f3435330700 1 -- 172.21.15.1:0/301017218 shutdown_connections 2024-03-18T18:50:48.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.805+0000 7f3435330700 1 --2- 172.21.15.1:0/301017218 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f341c06ac90 0x7f341c06d150 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.805+0000 7f3435330700 1 --2- 172.21.15.1:0/301017218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f343015b900 0x7f3430155650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.805+0000 7f3435330700 1 --2- 172.21.15.1:0/301017218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f343006a750 0x7f34301e4b70 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.805+0000 7f3435330700 1 --2- 172.21.15.1:0/301017218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f343006d560 0x7f34300c94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.805+0000 7f3435330700 1 -- 172.21.15.1:0/301017218 >> 172.21.15.1:0/301017218 conn(0x7f34300cff30 msgr2=0x7f34300cac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:48.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.805+0000 7f3435330700 1 -- 172.21.15.1:0/301017218 shutdown_connections 2024-03-18T18:50:48.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.805+0000 7f3435330700 1 -- 172.21.15.1:0/301017218 wait complete. 2024-03-18T18:50:48.815 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T18:50:48.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.947+0000 7f85c5e43700 1 Processor -- start 2024-03-18T18:50:48.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.947+0000 7f85c5e43700 1 -- start start 2024-03-18T18:50:48.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.947+0000 7f85c5e43700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85c00cc0f0 0x7f85c00add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.947+0000 7f85c5e43700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c00ce8f0 con 0x7f85c0071d30 2024-03-18T18:50:48.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.947+0000 7f85c5e43700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c00ae260 con 0x7f85c015a1d0 2024-03-18T18:50:48.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.947+0000 7f85c5e43700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c00cdb00 con 0x7f85c00cc0f0 2024-03-18T18:50:48.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85beffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f85c0071d30 0x7f85c00cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32962/0 (socket says 172.21.15.1:32962) 2024-03-18T18:50:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85beffd700 1 -- 172.21.15.1:0/1651892292 learned_addr learned my addr 172.21.15.1:0/1651892292 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85bffff700 1 --2- 172.21.15.1:0/1651892292 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85c00cc0f0 0x7f85c00add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85be7fc700 1 -- 172.21.15.1:0/1651892292 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182083130 0 0) 0x7f85c00ce8f0 con 0x7f85c0071d30 2024-03-18T18:50:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85be7fc700 1 -- 172.21.15.1:0/1651892292 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85a80035b0 con 0x7f85c0071d30 2024-03-18T18:50:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85be7fc700 1 -- 172.21.15.1:0/1651892292 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121088291 0 0) 0x7f85c00ae260 con 0x7f85c015a1d0 2024-03-18T18:50:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85be7fc700 1 -- 172.21.15.1:0/1651892292 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85c00ce8f0 con 0x7f85c015a1d0 2024-03-18T18:50:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85be7fc700 1 -- 172.21.15.1:0/1651892292 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2138510152 0 0) 0x7f85a80035b0 con 0x7f85c0071d30 2024-03-18T18:50:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85be7fc700 1 -- 172.21.15.1:0/1651892292 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85c00cc0f0 msgr2=0x7f85c00add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85be7fc700 1 --2- 172.21.15.1:0/1651892292 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85c00cc0f0 0x7f85c00add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85be7fc700 1 -- 172.21.15.1:0/1651892292 >> v1:172.21.15.165:6789/0 conn(0x7f85c015a1d0 legacy=0x7f85c0071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85be7fc700 1 -- 172.21.15.1:0/1651892292 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c00cdce0 con 0x7f85c0071d30 2024-03-18T18:50:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85be7fc700 1 -- 172.21.15.1:0/1651892292 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f85b4001670 con 0x7f85c0071d30 2024-03-18T18:50:48.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85be7fc700 1 -- 172.21.15.1:0/1651892292 >> v1:172.21.15.1:6789/0 conn(0x7f85c0071d30 legacy=0x7f85c00cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85be7fc700 1 --2- 172.21.15.1:0/1651892292 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85a8003bc0 0x7f85a8003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85be7fc700 1 --2- 172.21.15.1:0/1651892292 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85a8004850 0x7f85a8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85be7fc700 1 --2- 172.21.15.1:0/1651892292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85a8005210 0x7f85a800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85be7fc700 1 -- 172.21.15.1:0/1651892292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c00cdb00 con 0x7f85a8005210 2024-03-18T18:50:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85be7fc700 1 -- 172.21.15.1:0/1651892292 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a8014870 con 0x7f85a8004850 2024-03-18T18:50:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85be7fc700 1 -- 172.21.15.1:0/1651892292 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a8014a30 con 0x7f85a8003bc0 2024-03-18T18:50:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85bffff700 1 --2- 172.21.15.1:0/1651892292 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85c00cc0f0 0x7f85c00add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:50:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85bf7fe700 1 --2- 172.21.15.1:0/1651892292 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85a8003bc0 0x7f85a8003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85beffd700 1 --2- 172.21.15.1:0/1651892292 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85a8004850 0x7f85a8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.948+0000 7f85bffff700 1 --2- 172.21.15.1:0/1651892292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85a8005210 0x7f85a800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.949+0000 7f85bffff700 1 -- 172.21.15.1:0/1651892292 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85a8003bc0 msgr2=0x7f85a8003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.949+0000 7f85bffff700 1 --2- 172.21.15.1:0/1651892292 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85a8003bc0 0x7f85a8003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.949+0000 7f85bffff700 1 -- 172.21.15.1:0/1651892292 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85a8004850 msgr2=0x7f85a8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.949+0000 7f85bffff700 1 --2- 172.21.15.1:0/1651892292 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85a8004850 0x7f85a8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.949+0000 7f85bffff700 1 -- 172.21.15.1:0/1651892292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a80035b0 con 0x7f85a8005210 2024-03-18T18:50:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.949+0000 7f85bffff700 1 --2- 172.21.15.1:0/1651892292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85a8005210 0x7f85a800ba10 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f85c00cc4d0 tx=0x7f85a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e3ff1753717fec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.949+0000 7f85be7fc700 1 -- 172.21.15.1:0/1651892292 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b8003c20 con 0x7f85a8005210 2024-03-18T18:50:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.949+0000 7f85be7fc700 1 -- 172.21.15.1:0/1651892292 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b800f040 con 0x7f85a8005210 2024-03-18T18:50:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.949+0000 7f85be7fc700 1 -- 172.21.15.1:0/1651892292 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b800ea30 con 0x7f85a8005210 2024-03-18T18:50:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.949+0000 7f85c5e43700 1 -- 172.21.15.1:0/1651892292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85a8005210 msgr2=0x7f85a800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.949+0000 7f85c5e43700 1 --2- 172.21.15.1:0/1651892292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85a8005210 0x7f85a800ba10 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f85c00cc4d0 tx=0x7f85a8005880 comp rx=0 tx=0).stop 2024-03-18T18:50:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.949+0000 7f85bffff700 1 -- 172.21.15.1:0/1651892292 reap_dead start 2024-03-18T18:50:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.949+0000 7f85c5e43700 1 -- 172.21.15.1:0/1651892292 shutdown_connections 2024-03-18T18:50:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.949+0000 7f85c5e43700 1 -- 172.21.15.1:0/1651892292 >> 172.21.15.1:0/1651892292 conn(0x7f85c00cc9b0 msgr2=0x7f85c00ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.950+0000 7f85c5e43700 1 -- 172.21.15.1:0/1651892292 shutdown_connections 2024-03-18T18:50:48.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.950+0000 7f85c5e43700 1 -- 172.21.15.1:0/1651892292 wait complete. 2024-03-18T18:50:48.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.951+0000 7f85c5e43700 1 Processor -- start 2024-03-18T18:50:48.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.951+0000 7f85c5e43700 1 -- start start 2024-03-18T18:50:48.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85c5e43700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85c01546a0 0x7f85c006e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85c5e43700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85c00c7000 0x7f85c006ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85c5e43700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85c015b4d0 0x7f85c00c55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85c5e43700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c00c4b10 con 0x7f85c015b4d0 2024-03-18T18:50:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85c5e43700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c015d1a0 con 0x7f85c00c7000 2024-03-18T18:50:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85c5e43700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c00c4db0 con 0x7f85c01546a0 2024-03-18T18:50:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85bffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85c015b4d0 0x7f85c00c55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85bffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85c015b4d0 0x7f85c00c55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59314/0 (socket says 172.21.15.1:59314) 2024-03-18T18:50:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85bffff700 1 -- 172.21.15.1:0/513065313 learned_addr learned my addr 172.21.15.1:0/513065313 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85beffd700 1 --2- 172.21.15.1:0/513065313 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85c00c7000 0x7f85c006ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85bffff700 1 -- 172.21.15.1:0/513065313 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85c01546a0 msgr2=0x7f85c006e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85bffff700 1 --2- 172.21.15.1:0/513065313 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85c01546a0 0x7f85c006e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85bffff700 1 -- 172.21.15.1:0/513065313 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85c00c7000 msgr2=0x7f85c006ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:48.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85bffff700 1 --2- 172.21.15.1:0/513065313 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85c00c7000 0x7f85c006ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85bffff700 1 -- 172.21.15.1:0/513065313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c00c5090 con 0x7f85c015b4d0 2024-03-18T18:50:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85bffff700 1 --2- 172.21.15.1:0/513065313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85c015b4d0 0x7f85c00c55b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f85b8015000 tx=0x7f85b8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85a7fff700 1 -- 172.21.15.1:0/513065313 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b8025070 con 0x7f85c015b4d0 2024-03-18T18:50:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85a7fff700 1 -- 172.21.15.1:0/513065313 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b8003c30 con 0x7f85c015b4d0 2024-03-18T18:50:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85c5e43700 1 -- 172.21.15.1:0/513065313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c015d480 con 0x7f85c015b4d0 2024-03-18T18:50:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85a7fff700 1 -- 172.21.15.1:0/513065313 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b8014860 con 0x7f85c015b4d0 2024-03-18T18:50:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.952+0000 7f85c5e43700 1 -- 172.21.15.1:0/513065313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c015cd30 con 0x7f85c015b4d0 2024-03-18T18:50:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.953+0000 7f85c5e43700 1 -- 172.21.15.1:0/513065313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c015cf20 con 0x7f85c015b4d0 2024-03-18T18:50:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.954+0000 7f85a7fff700 1 -- 172.21.15.1:0/513065313 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f85b80149c0 con 0x7f85c015b4d0 2024-03-18T18:50:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.954+0000 7f85a7fff700 1 --2- 172.21.15.1:0/513065313 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f85ac06ac90 0x7f85ac06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.954+0000 7f85a7fff700 1 -- 172.21.15.1:0/513065313 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f85b808a990 con 0x7f85c015b4d0 2024-03-18T18:50:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.954+0000 7f85bf7fe700 1 --2- 172.21.15.1:0/513065313 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f85ac06ac90 0x7f85ac06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:48.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.955+0000 7f85bf7fe700 1 --2- 172.21.15.1:0/513065313 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f85ac06ac90 0x7f85ac06d150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f85b0000c00 tx=0x7f85b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:48.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:48.959+0000 7f85a7fff700 1 -- 172.21.15.1:0/513065313 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f85b8055960 con 0x7f85c015b4d0 2024-03-18T18:50:49.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.161+0000 7f85c5e43700 1 -- 172.21.15.1:0/513065313 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f85c00676a0 con 0x7f85c015b4d0 2024-03-18T18:50:49.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.162+0000 7f85a7fff700 1 -- 172.21.15.1:0/513065313 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f85b8058f80 con 0x7f85c015b4d0 2024-03-18T18:50:49.159 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T18:50:49.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.164+0000 7f85c5e43700 1 -- 172.21.15.1:0/513065313 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f85ac06ac90 msgr2=0x7f85ac06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:49.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.164+0000 7f85c5e43700 1 --2- 172.21.15.1:0/513065313 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f85ac06ac90 0x7f85ac06d150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f85b0000c00 tx=0x7f85b0009040 comp rx=0 tx=0).stop 2024-03-18T18:50:49.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.164+0000 7f85c5e43700 1 -- 172.21.15.1:0/513065313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85c015b4d0 msgr2=0x7f85c00c55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:49.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.164+0000 7f85c5e43700 1 --2- 172.21.15.1:0/513065313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85c015b4d0 0x7f85c00c55b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f85b8015000 tx=0x7f85b8009750 comp rx=0 tx=0).stop 2024-03-18T18:50:49.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.164+0000 7f85c5e43700 1 -- 172.21.15.1:0/513065313 shutdown_connections 2024-03-18T18:50:49.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.164+0000 7f85c5e43700 1 --2- 172.21.15.1:0/513065313 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f85ac06ac90 0x7f85ac06d150 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.164+0000 7f85c5e43700 1 --2- 172.21.15.1:0/513065313 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85c01546a0 0x7f85c006e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.164+0000 7f85c5e43700 1 --2- 172.21.15.1:0/513065313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85c015b4d0 0x7f85c00c55b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.164+0000 7f85c5e43700 1 --2- 172.21.15.1:0/513065313 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85c00c7000 0x7f85c006ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.164+0000 7f85c5e43700 1 -- 172.21.15.1:0/513065313 >> 172.21.15.1:0/513065313 conn(0x7f85c00cc9b0 msgr2=0x7f85c00aef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:49.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.164+0000 7f85c5e43700 1 -- 172.21.15.1:0/513065313 shutdown_connections 2024-03-18T18:50:49.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.164+0000 7f85c5e43700 1 -- 172.21.15.1:0/513065313 wait complete. 2024-03-18T18:50:49.174 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T18:50:49.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.305+0000 7f49163e5700 1 Processor -- start 2024-03-18T18:50:49.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.305+0000 7f49163e5700 1 -- start start 2024-03-18T18:50:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.306+0000 7f49163e5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f491006a750 0x7f491006ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.306+0000 7f49163e5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f49100ace80 0x7f49100ad300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.306+0000 7f49163e5700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491015c340 con 0x7f49100ad840 2024-03-18T18:50:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.306+0000 7f49163e5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f491015fc90 con 0x7f49100ace80 2024-03-18T18:50:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.306+0000 7f49163e5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4910070090 con 0x7f491006a750 2024-03-18T18:50:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.306+0000 7f4914982700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f49100ad840 0x7f49100caa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32968/0 (socket says 172.21.15.1:32968) 2024-03-18T18:50:49.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.306+0000 7f4914982700 1 -- 172.21.15.1:0/2963397382 learned_addr learned my addr 172.21.15.1:0/2963397382 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.306+0000 7f490f7fe700 1 --2- 172.21.15.1:0/2963397382 >> v2:172.21.15.165:3300/0 conn(0x7f49100ace80 0x7f49100ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.306+0000 7f490effd700 1 -- 172.21.15.1:0/2963397382 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36728094 0 0) 0x7f491015c340 con 0x7f49100ad840 2024-03-18T18:50:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.306+0000 7f490ffff700 1 --2- 172.21.15.1:0/2963397382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f491006a750 0x7f491006ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.306+0000 7f490effd700 1 -- 172.21.15.1:0/2963397382 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48f80035b0 con 0x7f49100ad840 2024-03-18T18:50:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.306+0000 7f490effd700 1 -- 172.21.15.1:0/2963397382 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2926486736 0 0) 0x7f48f80035b0 con 0x7f49100ad840 2024-03-18T18:50:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.306+0000 7f490effd700 1 -- 172.21.15.1:0/2963397382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f491006a750 msgr2=0x7f491006ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.306+0000 7f490effd700 1 --2- 172.21.15.1:0/2963397382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f491006a750 0x7f491006ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.306+0000 7f490effd700 1 -- 172.21.15.1:0/2963397382 >> v2:172.21.15.165:3300/0 conn(0x7f49100ace80 msgr2=0x7f49100ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.306+0000 7f490effd700 1 --2- 172.21.15.1:0/2963397382 >> v2:172.21.15.165:3300/0 conn(0x7f49100ace80 0x7f49100ad300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.306+0000 7f490effd700 1 -- 172.21.15.1:0/2963397382 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491015b7a0 con 0x7f49100ad840 2024-03-18T18:50:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.306+0000 7f490effd700 1 -- 172.21.15.1:0/2963397382 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f49080019f0 con 0x7f49100ad840 2024-03-18T18:50:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.307+0000 7f490effd700 1 -- 172.21.15.1:0/2963397382 >> v1:172.21.15.1:6789/0 conn(0x7f49100ad840 legacy=0x7f49100caa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.307+0000 7f490effd700 1 --2- 172.21.15.1:0/2963397382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f491006a750 0x7f48f8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.307+0000 7f490effd700 1 --2- 172.21.15.1:0/2963397382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f8003f80 0x7f48f8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:49.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.307+0000 7f490effd700 1 --2- 172.21.15.1:0/2963397382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48f8004d90 0x7f48f8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:49.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.307+0000 7f490effd700 1 -- 172.21.15.1:0/2963397382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491015fc90 con 0x7f48f8003f80 2024-03-18T18:50:49.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.307+0000 7f490effd700 1 -- 172.21.15.1:0/2963397382 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4910070090 con 0x7f48f8004d90 2024-03-18T18:50:49.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.307+0000 7f490effd700 1 -- 172.21.15.1:0/2963397382 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f800ba10 con 0x7f491006a750 2024-03-18T18:50:49.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.307+0000 7f490f7fe700 1 --2- 172.21.15.1:0/2963397382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f8003f80 0x7f48f8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:49.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.307+0000 7f490ffff700 1 --2- 172.21.15.1:0/2963397382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f491006a750 0x7f48f8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:49.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.307+0000 7f4914982700 1 --2- 172.21.15.1:0/2963397382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48f8004d90 0x7f48f8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:49.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.307+0000 7f490f7fe700 1 -- 172.21.15.1:0/2963397382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f491006a750 msgr2=0x7f48f8003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:49.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.307+0000 7f490f7fe700 1 --2- 172.21.15.1:0/2963397382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f491006a750 0x7f48f8003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.307+0000 7f490f7fe700 1 -- 172.21.15.1:0/2963397382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48f8004d90 msgr2=0x7f48f8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:49.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.307+0000 7f490f7fe700 1 --2- 172.21.15.1:0/2963397382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48f8004d90 0x7f48f8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.307+0000 7f490f7fe700 1 -- 172.21.15.1:0/2963397382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f80035b0 con 0x7f48f8003f80 2024-03-18T18:50:49.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.307+0000 7f490f7fe700 1 --2- 172.21.15.1:0/2963397382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f8003f80 0x7f48f8004850 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4904009b60 tx=0x7f48f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f17e0a674dc5c244 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:49.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.307+0000 7f490effd700 1 -- 172.21.15.1:0/2963397382 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490400e660 con 0x7f48f8003f80 2024-03-18T18:50:49.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.307+0000 7f490effd700 1 -- 172.21.15.1:0/2963397382 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490400e7c0 con 0x7f48f8003f80 2024-03-18T18:50:49.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.307+0000 7f490effd700 1 -- 172.21.15.1:0/2963397382 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490400eb30 con 0x7f48f8003f80 2024-03-18T18:50:49.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.308+0000 7f49163e5700 1 -- 172.21.15.1:0/2963397382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f8003f80 msgr2=0x7f48f8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:49.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.308+0000 7f49163e5700 1 --2- 172.21.15.1:0/2963397382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f8003f80 0x7f48f8004850 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4904009b60 tx=0x7f48f8005880 comp rx=0 tx=0).stop 2024-03-18T18:50:49.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.308+0000 7f4914982700 1 -- 172.21.15.1:0/2963397382 reap_dead start 2024-03-18T18:50:49.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.308+0000 7f49163e5700 1 -- 172.21.15.1:0/2963397382 shutdown_connections 2024-03-18T18:50:49.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.308+0000 7f49163e5700 1 -- 172.21.15.1:0/2963397382 >> 172.21.15.1:0/2963397382 conn(0x7f49100cc9b0 msgr2=0x7f49100ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:49.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.308+0000 7f49163e5700 1 -- 172.21.15.1:0/2963397382 shutdown_connections 2024-03-18T18:50:49.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.308+0000 7f49163e5700 1 -- 172.21.15.1:0/2963397382 wait complete. 2024-03-18T18:50:49.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.309+0000 7f49163e5700 1 Processor -- start 2024-03-18T18:50:49.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.309+0000 7f49163e5700 1 -- start start 2024-03-18T18:50:49.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.310+0000 7f49163e5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f48f8003f80 0x7f49100b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:49.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.310+0000 7f49163e5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4910156c20 0x7f49100c5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:49.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.310+0000 7f49163e5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f491015d880 0x7f491015dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:49.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.310+0000 7f49163e5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491015e240 con 0x7f491015d880 2024-03-18T18:50:49.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.310+0000 7f49163e5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49100b5bb0 con 0x7f4910156c20 2024-03-18T18:50:49.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.310+0000 7f49163e5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49100b3170 con 0x7f48f8003f80 2024-03-18T18:50:49.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.310+0000 7f4914982700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f491015d880 0x7f491015dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:49.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.310+0000 7f4914982700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f491015d880 0x7f491015dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59326/0 (socket says 172.21.15.1:59326) 2024-03-18T18:50:49.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.310+0000 7f4914982700 1 -- 172.21.15.1:0/754222020 learned_addr learned my addr 172.21.15.1:0/754222020 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:49.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.310+0000 7f490ffff700 1 --2- 172.21.15.1:0/754222020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f48f8003f80 0x7f49100b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:49.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.310+0000 7f490f7fe700 1 --2- 172.21.15.1:0/754222020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4910156c20 0x7f49100c5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:49.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.310+0000 7f4914982700 1 -- 172.21.15.1:0/754222020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f48f8003f80 msgr2=0x7f49100b2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:49.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.310+0000 7f4914982700 1 --2- 172.21.15.1:0/754222020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f48f8003f80 0x7f49100b2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.310+0000 7f4914982700 1 -- 172.21.15.1:0/754222020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4910156c20 msgr2=0x7f49100c5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:49.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.310+0000 7f4914982700 1 --2- 172.21.15.1:0/754222020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4910156c20 0x7f49100c5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.310+0000 7f4914982700 1 -- 172.21.15.1:0/754222020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49100b5280 con 0x7f491015d880 2024-03-18T18:50:49.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.311+0000 7f4914982700 1 --2- 172.21.15.1:0/754222020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f491015d880 0x7f491015dd00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f491015c7d0 tx=0x7f490800a600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:49.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.311+0000 7f490cff9700 1 -- 172.21.15.1:0/754222020 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490800add0 con 0x7f491015d880 2024-03-18T18:50:49.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.311+0000 7f490cff9700 1 -- 172.21.15.1:0/754222020 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49080133f0 con 0x7f491015d880 2024-03-18T18:50:49.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.311+0000 7f490cff9700 1 -- 172.21.15.1:0/754222020 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49080137a0 con 0x7f491015d880 2024-03-18T18:50:49.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.311+0000 7f49163e5700 1 -- 172.21.15.1:0/754222020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49100b5560 con 0x7f491015d880 2024-03-18T18:50:49.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.311+0000 7f49163e5700 1 -- 172.21.15.1:0/754222020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49100c60d0 con 0x7f491015d880 2024-03-18T18:50:49.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.312+0000 7f49163e5700 1 -- 172.21.15.1:0/754222020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49100676a0 con 0x7f491015d880 2024-03-18T18:50:49.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.312+0000 7f490cff9700 1 -- 172.21.15.1:0/754222020 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4908013900 con 0x7f491015d880 2024-03-18T18:50:49.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.313+0000 7f490cff9700 1 --2- 172.21.15.1:0/754222020 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f48fc06ac90 0x7f48fc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:49.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.313+0000 7f490ffff700 1 --2- 172.21.15.1:0/754222020 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f48fc06ac90 0x7f48fc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:49.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.313+0000 7f490cff9700 1 -- 172.21.15.1:0/754222020 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f490808a000 con 0x7f491015d880 2024-03-18T18:50:49.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.313+0000 7f490ffff700 1 --2- 172.21.15.1:0/754222020 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f48fc06ac90 0x7f48fc06d150 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4900009aa0 tx=0x7f4900008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:49.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.317+0000 7f490cff9700 1 -- 172.21.15.1:0/754222020 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4908054e20 con 0x7f491015d880 2024-03-18T18:50:49.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.520+0000 7f49163e5700 1 -- 172.21.15.1:0/754222020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f49100cd640 con 0x7f491015d880 2024-03-18T18:50:49.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.520+0000 7f490cff9700 1 -- 172.21.15.1:0/754222020 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4908058440 con 0x7f491015d880 2024-03-18T18:50:49.518 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:49.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.522+0000 7f49163e5700 1 -- 172.21.15.1:0/754222020 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f48fc06ac90 msgr2=0x7f48fc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:49.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.522+0000 7f49163e5700 1 --2- 172.21.15.1:0/754222020 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f48fc06ac90 0x7f48fc06d150 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4900009aa0 tx=0x7f4900008040 comp rx=0 tx=0).stop 2024-03-18T18:50:49.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.522+0000 7f49163e5700 1 -- 172.21.15.1:0/754222020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f491015d880 msgr2=0x7f491015dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:49.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.522+0000 7f49163e5700 1 --2- 172.21.15.1:0/754222020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f491015d880 0x7f491015dd00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f491015c7d0 tx=0x7f490800a600 comp rx=0 tx=0).stop 2024-03-18T18:50:49.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.523+0000 7f49163e5700 1 -- 172.21.15.1:0/754222020 shutdown_connections 2024-03-18T18:50:49.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.523+0000 7f49163e5700 1 --2- 172.21.15.1:0/754222020 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f48fc06ac90 0x7f48fc06d150 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.523+0000 7f49163e5700 1 --2- 172.21.15.1:0/754222020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f48f8003f80 0x7f49100b2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.523+0000 7f49163e5700 1 --2- 172.21.15.1:0/754222020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f491015d880 0x7f491015dd00 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.523+0000 7f49163e5700 1 --2- 172.21.15.1:0/754222020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4910156c20 0x7f49100c5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.523+0000 7f49163e5700 1 -- 172.21.15.1:0/754222020 >> 172.21.15.1:0/754222020 conn(0x7f49100cc9b0 msgr2=0x7f491015ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:49.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.523+0000 7f49163e5700 1 -- 172.21.15.1:0/754222020 shutdown_connections 2024-03-18T18:50:49.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.523+0000 7f49163e5700 1 -- 172.21.15.1:0/754222020 wait complete. 2024-03-18T18:50:49.532 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T18:50:49.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.661+0000 7f3115069700 1 Processor -- start 2024-03-18T18:50:49.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.661+0000 7f3115069700 1 -- start start 2024-03-18T18:50:49.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.661+0000 7f3115069700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3110158e30 0x7f3110071500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:49.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.661+0000 7f3115069700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f311006ae60 0x7f31100cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:49.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.661+0000 7f3115069700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311015c500 con 0x7f31100cbf70 2024-03-18T18:50:49.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.661+0000 7f3115069700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31100af8b0 con 0x7f3110158e30 2024-03-18T18:50:49.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.661+0000 7f3115069700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311015b8f0 con 0x7f311006ae60 2024-03-18T18:50:49.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.661+0000 7f310e59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f31100cbf70 0x7f311006a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32978/0 (socket says 172.21.15.1:32978) 2024-03-18T18:50:49.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.661+0000 7f310e59c700 1 -- 172.21.15.1:0/2078937451 learned_addr learned my addr 172.21.15.1:0/2078937451 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:49.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.661+0000 7f310ed9d700 1 --2- 172.21.15.1:0/2078937451 >> v2:172.21.15.165:3300/0 conn(0x7f3110158e30 0x7f3110071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:49.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.661+0000 7f310f59e700 1 --2- 172.21.15.1:0/2078937451 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f311006ae60 0x7f31100cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:49.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.662+0000 7f310dd9b700 1 -- 172.21.15.1:0/2078937451 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169887583 0 0) 0x7f311015c500 con 0x7f31100cbf70 2024-03-18T18:50:49.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.662+0000 7f310dd9b700 1 -- 172.21.15.1:0/2078937451 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30fc0035b0 con 0x7f31100cbf70 2024-03-18T18:50:49.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.662+0000 7f310ed9d700 1 -- 172.21.15.1:0/2078937451 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f311006ae60 msgr2=0x7f31100cc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:49.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.662+0000 7f310ed9d700 1 --2- 172.21.15.1:0/2078937451 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f311006ae60 0x7f31100cc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.662+0000 7f310ed9d700 1 -- 172.21.15.1:0/2078937451 >> v1:172.21.15.1:6789/0 conn(0x7f31100cbf70 legacy=0x7f311006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:49.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.662+0000 7f310ed9d700 1 -- 172.21.15.1:0/2078937451 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311015bbd0 con 0x7f3110158e30 2024-03-18T18:50:49.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.662+0000 7f310f59e700 1 --2- 172.21.15.1:0/2078937451 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f311006ae60 0x7f31100cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:50:49.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.662+0000 7f310ed9d700 1 --2- 172.21.15.1:0/2078937451 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3110158e30 0x7f3110071500 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3100009840 tx=0x7f3100009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68089b8dfa185401 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:49.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.662+0000 7f310dd9b700 1 -- 172.21.15.1:0/2078937451 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3100011830 con 0x7f3110158e30 2024-03-18T18:50:49.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.662+0000 7f310dd9b700 1 -- 172.21.15.1:0/2078937451 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3100011990 con 0x7f3110158e30 2024-03-18T18:50:49.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.662+0000 7f310dd9b700 1 -- 172.21.15.1:0/2078937451 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3100011d20 con 0x7f3110158e30 2024-03-18T18:50:49.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.663+0000 7f3115069700 1 -- 172.21.15.1:0/2078937451 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3110158e30 msgr2=0x7f3110071500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:49.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.663+0000 7f3115069700 1 --2- 172.21.15.1:0/2078937451 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3110158e30 0x7f3110071500 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3100009840 tx=0x7f3100009b50 comp rx=0 tx=0).stop 2024-03-18T18:50:49.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.663+0000 7f3115069700 1 -- 172.21.15.1:0/2078937451 shutdown_connections 2024-03-18T18:50:49.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.663+0000 7f3115069700 1 --2- 172.21.15.1:0/2078937451 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3110158e30 0x7f3110071500 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.663+0000 7f3115069700 1 --2- 172.21.15.1:0/2078937451 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f311006ae60 0x7f31100cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.663+0000 7f3115069700 1 -- 172.21.15.1:0/2078937451 >> 172.21.15.1:0/2078937451 conn(0x7f31100cca00 msgr2=0x7f31100cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:49.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.663+0000 7f3115069700 1 -- 172.21.15.1:0/2078937451 shutdown_connections 2024-03-18T18:50:49.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.663+0000 7f3115069700 1 -- 172.21.15.1:0/2078937451 wait complete. 2024-03-18T18:50:49.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.665+0000 7f3115069700 1 Processor -- start 2024-03-18T18:50:49.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.665+0000 7f3115069700 1 -- start start 2024-03-18T18:50:49.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f3115069700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f311006ae60 0x7f3110158630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:49.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f3115069700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31100cbf70 0x7f31100ae3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:49.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f3115069700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3110158e30 0x7f31100ae920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:49.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f3115069700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31100c63c0 con 0x7f31100cbf70 2024-03-18T18:50:49.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f3115069700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31100c57b0 con 0x7f3110158e30 2024-03-18T18:50:49.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f3115069700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31100c5a90 con 0x7f311006ae60 2024-03-18T18:50:49.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f310e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31100cbf70 0x7f31100ae3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:49.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f310e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31100cbf70 0x7f31100ae3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59340/0 (socket says 172.21.15.1:59340) 2024-03-18T18:50:49.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f310e59c700 1 -- 172.21.15.1:0/2045573028 learned_addr learned my addr 172.21.15.1:0/2045573028 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:49.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f310f59e700 1 --2- 172.21.15.1:0/2045573028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3110158e30 0x7f31100ae920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f310ed9d700 1 --2- 172.21.15.1:0/2045573028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f311006ae60 0x7f3110158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f310e59c700 1 -- 172.21.15.1:0/2045573028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f311006ae60 msgr2=0x7f3110158630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f310e59c700 1 --2- 172.21.15.1:0/2045573028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f311006ae60 0x7f3110158630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f310e59c700 1 -- 172.21.15.1:0/2045573028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3110158e30 msgr2=0x7f31100ae920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f310e59c700 1 --2- 172.21.15.1:0/2045573028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3110158e30 0x7f31100ae920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f310e59c700 1 -- 172.21.15.1:0/2045573028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3108009a70 con 0x7f31100cbf70 2024-03-18T18:50:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f310f59e700 1 --2- 172.21.15.1:0/2045573028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3110158e30 0x7f31100ae920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f310e59c700 1 --2- 172.21.15.1:0/2045573028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31100cbf70 0x7f31100ae3e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f30f800c4d0 tx=0x7f30f800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f31077fe700 1 -- 172.21.15.1:0/2045573028 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f8005d70 con 0x7f31100cbf70 2024-03-18T18:50:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f31077fe700 1 -- 172.21.15.1:0/2045573028 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f80070d0 con 0x7f31100cbf70 2024-03-18T18:50:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f3115069700 1 -- 172.21.15.1:0/2045573028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31000094a0 con 0x7f31100cbf70 2024-03-18T18:50:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f31077fe700 1 -- 172.21.15.1:0/2045573028 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f8007440 con 0x7f31100cbf70 2024-03-18T18:50:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.666+0000 7f3115069700 1 -- 172.21.15.1:0/2045573028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31100c6050 con 0x7f31100cbf70 2024-03-18T18:50:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.668+0000 7f3115069700 1 -- 172.21.15.1:0/2045573028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3110073940 con 0x7f31100cbf70 2024-03-18T18:50:49.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.668+0000 7f31077fe700 1 -- 172.21.15.1:0/2045573028 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f30f80075a0 con 0x7f31100cbf70 2024-03-18T18:50:49.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.668+0000 7f31077fe700 1 --2- 172.21.15.1:0/2045573028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f30f006ac90 0x7f30f006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:49.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.668+0000 7f31077fe700 1 -- 172.21.15.1:0/2045573028 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f30f8088b10 con 0x7f31100cbf70 2024-03-18T18:50:49.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.668+0000 7f310ed9d700 1 --2- 172.21.15.1:0/2045573028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f30f006ac90 0x7f30f006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:49.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.669+0000 7f310ed9d700 1 --2- 172.21.15.1:0/2045573028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f30f006ac90 0x7f30f006d150 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3100000c00 tx=0x7f310001c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:49.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.673+0000 7f31077fe700 1 -- 172.21.15.1:0/2045573028 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f30f80539e0 con 0x7f31100cbf70 2024-03-18T18:50:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.893+0000 7f3115069700 1 -- 172.21.15.1:0/2045573028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f31100676a0 con 0x7f31100cbf70 2024-03-18T18:50:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.893+0000 7f31077fe700 1 -- 172.21.15.1:0/2045573028 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f30f8017020 con 0x7f31100cbf70 2024-03-18T18:50:49.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.896+0000 7f3115069700 1 -- 172.21.15.1:0/2045573028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f30f006ac90 msgr2=0x7f30f006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:49.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.896+0000 7f3115069700 1 --2- 172.21.15.1:0/2045573028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f30f006ac90 0x7f30f006d150 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3100000c00 tx=0x7f310001c040 comp rx=0 tx=0).stop 2024-03-18T18:50:49.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.896+0000 7f3115069700 1 -- 172.21.15.1:0/2045573028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31100cbf70 msgr2=0x7f31100ae3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:49.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.896+0000 7f3115069700 1 --2- 172.21.15.1:0/2045573028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31100cbf70 0x7f31100ae3e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f30f800c4d0 tx=0x7f30f800c890 comp rx=0 tx=0).stop 2024-03-18T18:50:49.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.896+0000 7f3115069700 1 -- 172.21.15.1:0/2045573028 shutdown_connections 2024-03-18T18:50:49.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.896+0000 7f3115069700 1 --2- 172.21.15.1:0/2045573028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f30f006ac90 0x7f30f006d150 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.896+0000 7f3115069700 1 --2- 172.21.15.1:0/2045573028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f311006ae60 0x7f3110158630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.896+0000 7f3115069700 1 --2- 172.21.15.1:0/2045573028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3110158e30 0x7f31100ae920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.896+0000 7f3115069700 1 --2- 172.21.15.1:0/2045573028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31100cbf70 0x7f31100ae3e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:49.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.896+0000 7f3115069700 1 -- 172.21.15.1:0/2045573028 >> 172.21.15.1:0/2045573028 conn(0x7f31100cca00 msgr2=0x7f311015dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:49.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.896+0000 7f3115069700 1 -- 172.21.15.1:0/2045573028 shutdown_connections 2024-03-18T18:50:49.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:49.896+0000 7f3115069700 1 -- 172.21.15.1:0/2045573028 wait complete. 2024-03-18T18:50:49.895 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-18T18:50:49.905 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T18:50:49.905 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T18:50:49.926 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:50.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.094+0000 7fce9329e700 1 Processor -- start 2024-03-18T18:50:50.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.094+0000 7fce9329e700 1 -- start start 2024-03-18T18:50:50.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.094+0000 7fce9329e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce8c071600 0x7fce8c071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:50.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.094+0000 7fce9329e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8c0acd80 con 0x7fce8c06e2c0 2024-03-18T18:50:50.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.094+0000 7fce9329e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8c0cf220 con 0x7fce8c15a320 2024-03-18T18:50:50.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.094+0000 7fce9329e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c071fc0 con 0x7fce8c071600 2024-03-18T18:50:50.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.095+0000 7fce9183b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fce8c06e2c0 0x7fce8c06e750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32988/0 (socket says 172.21.15.1:32988) 2024-03-18T18:50:50.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.095+0000 7fce9183b700 1 -- 172.21.15.1:0/1967280057 learned_addr learned my addr 172.21.15.1:0/1967280057 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:50.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.095+0000 7fce90839700 1 --2- 172.21.15.1:0/1967280057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce8c071600 0x7fce8c071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:50.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.095+0000 7fce7ffff700 1 -- 172.21.15.1:0/1967280057 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785883143 0 0) 0x7fce8c0acd80 con 0x7fce8c06e2c0 2024-03-18T18:50:50.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.095+0000 7fce7ffff700 1 -- 172.21.15.1:0/1967280057 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce780035b0 con 0x7fce8c06e2c0 2024-03-18T18:50:50.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.095+0000 7fce7ffff700 1 -- 172.21.15.1:0/1967280057 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974418232 0 0) 0x7fce88001110 con 0x7fce8c15a320 2024-03-18T18:50:50.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.095+0000 7fce7ffff700 1 -- 172.21.15.1:0/1967280057 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce8c0acd80 con 0x7fce8c15a320 2024-03-18T18:50:50.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.095+0000 7fce7ffff700 1 -- 172.21.15.1:0/1967280057 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1794890031 0 0) 0x7fce780035b0 con 0x7fce8c06e2c0 2024-03-18T18:50:50.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.095+0000 7fce90839700 1 -- 172.21.15.1:0/1967280057 >> v1:172.21.15.165:6789/0 conn(0x7fce8c15a320 legacy=0x7fce8c0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:50.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.095+0000 7fce90839700 1 -- 172.21.15.1:0/1967280057 >> v1:172.21.15.1:6789/0 conn(0x7fce8c06e2c0 legacy=0x7fce8c06e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:50.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.095+0000 7fce90839700 1 -- 172.21.15.1:0/1967280057 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8c0cef40 con 0x7fce8c071600 2024-03-18T18:50:50.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.095+0000 7fce90839700 1 --2- 172.21.15.1:0/1967280057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce8c071600 0x7fce8c071a80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fce800098a0 tx=0x7fce80009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5eb7198112fc0bb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:50.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.096+0000 7fce7ffff700 1 -- 172.21.15.1:0/1967280057 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce840019f0 con 0x7fce8c071600 2024-03-18T18:50:50.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.096+0000 7fce7ffff700 1 -- 172.21.15.1:0/1967280057 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce80004030 con 0x7fce8c071600 2024-03-18T18:50:50.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.096+0000 7fce7ffff700 1 -- 172.21.15.1:0/1967280057 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce800043c0 con 0x7fce8c071600 2024-03-18T18:50:50.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.096+0000 7fce9329e700 1 -- 172.21.15.1:0/1967280057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce8c071600 msgr2=0x7fce8c071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:50.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.096+0000 7fce9329e700 1 --2- 172.21.15.1:0/1967280057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce8c071600 0x7fce8c071a80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fce800098a0 tx=0x7fce80009bb0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.096+0000 7fce9329e700 1 -- 172.21.15.1:0/1967280057 shutdown_connections 2024-03-18T18:50:50.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.096+0000 7fce9329e700 1 --2- 172.21.15.1:0/1967280057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce8c071600 0x7fce8c071a80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.096+0000 7fce9329e700 1 -- 172.21.15.1:0/1967280057 >> 172.21.15.1:0/1967280057 conn(0x7fce8c0cc990 msgr2=0x7fce8c0ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:50.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.096+0000 7fce9329e700 1 -- 172.21.15.1:0/1967280057 shutdown_connections 2024-03-18T18:50:50.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.096+0000 7fce9329e700 1 -- 172.21.15.1:0/1967280057 wait complete. 2024-03-18T18:50:50.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.098+0000 7fce9329e700 1 Processor -- start 2024-03-18T18:50:50.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.098+0000 7fce9329e700 1 -- start start 2024-03-18T18:50:50.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce9329e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce8c06e2c0 0x7fce8c0c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:50.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce9329e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce8c071600 0x7fce8c154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:50.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce9329e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce8c1e26d0 0x7fce8c0c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:50.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce9329e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c0c9ee0 con 0x7fce8c06e2c0 2024-03-18T18:50:50.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce9329e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c15bac0 con 0x7fce8c1e26d0 2024-03-18T18:50:50.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce9329e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c15bda0 con 0x7fce8c071600 2024-03-18T18:50:50.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce9103a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce8c06e2c0 0x7fce8c0c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:50.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce9103a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce8c06e2c0 0x7fce8c0c99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59356/0 (socket says 172.21.15.1:59356) 2024-03-18T18:50:50.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce9103a700 1 -- 172.21.15.1:0/3830501804 learned_addr learned my addr 172.21.15.1:0/3830501804 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:50.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce90839700 1 --2- 172.21.15.1:0/3830501804 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce8c071600 0x7fce8c154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:50.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce9183b700 1 --2- 172.21.15.1:0/3830501804 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce8c1e26d0 0x7fce8c0c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:50.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce9103a700 1 -- 172.21.15.1:0/3830501804 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce8c071600 msgr2=0x7fce8c154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:50.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce9103a700 1 --2- 172.21.15.1:0/3830501804 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce8c071600 0x7fce8c154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce9103a700 1 -- 172.21.15.1:0/3830501804 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce8c1e26d0 msgr2=0x7fce8c0c79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:50.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce9103a700 1 --2- 172.21.15.1:0/3830501804 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce8c1e26d0 0x7fce8c0c79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce9103a700 1 -- 172.21.15.1:0/3830501804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce800094a0 con 0x7fce8c06e2c0 2024-03-18T18:50:50.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce9103a700 1 --2- 172.21.15.1:0/3830501804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce8c06e2c0 0x7fce8c0c99a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fce8800c700 tx=0x7fce8800cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:50.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce7dffb700 1 -- 172.21.15.1:0/3830501804 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce880052d0 con 0x7fce8c06e2c0 2024-03-18T18:50:50.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce7dffb700 1 -- 172.21.15.1:0/3830501804 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce88005450 con 0x7fce8c06e2c0 2024-03-18T18:50:50.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce9329e700 1 -- 172.21.15.1:0/3830501804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce8c15c080 con 0x7fce8c06e2c0 2024-03-18T18:50:50.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce7dffb700 1 -- 172.21.15.1:0/3830501804 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce880038c0 con 0x7fce8c06e2c0 2024-03-18T18:50:50.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.100+0000 7fce9329e700 1 -- 172.21.15.1:0/3830501804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce8c15c640 con 0x7fce8c06e2c0 2024-03-18T18:50:50.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.101+0000 7fce9329e700 1 -- 172.21.15.1:0/3830501804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce8c0ca600 con 0x7fce8c06e2c0 2024-03-18T18:50:50.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.101+0000 7fce7dffb700 1 -- 172.21.15.1:0/3830501804 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fce88003a20 con 0x7fce8c06e2c0 2024-03-18T18:50:50.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.102+0000 7fce7dffb700 1 --2- 172.21.15.1:0/3830501804 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce7006ac90 0x7fce7006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:50.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.102+0000 7fce7dffb700 1 -- 172.21.15.1:0/3830501804 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fce88088b70 con 0x7fce8c06e2c0 2024-03-18T18:50:50.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.102+0000 7fce90839700 1 --2- 172.21.15.1:0/3830501804 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce7006ac90 0x7fce7006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:50.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.102+0000 7fce90839700 1 --2- 172.21.15.1:0/3830501804 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce7006ac90 0x7fce7006d150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fce800097c0 tx=0x7fce8001c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:50.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.107+0000 7fce7dffb700 1 -- 172.21.15.1:0/3830501804 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fce880539a0 con 0x7fce8c06e2c0 2024-03-18T18:50:50.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.312+0000 7fce9329e700 1 -- 172.21.15.1:0/3830501804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce8c0cf220 con 0x7fce8c06e2c0 2024-03-18T18:50:50.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.313+0000 7fce7dffb700 1 -- 172.21.15.1:0/3830501804 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18217 (secure 0 0 0) 0x7fce8c0cf220 con 0x7fce8c06e2c0 2024-03-18T18:50:50.311 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:50.311 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":27,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:50:34.993482+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:50.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.316+0000 7fce9329e700 1 -- 172.21.15.1:0/3830501804 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce7006ac90 msgr2=0x7fce7006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:50.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.316+0000 7fce9329e700 1 --2- 172.21.15.1:0/3830501804 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce7006ac90 0x7fce7006d150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fce800097c0 tx=0x7fce8001c040 comp rx=0 tx=0).stop 2024-03-18T18:50:50.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.316+0000 7fce9329e700 1 -- 172.21.15.1:0/3830501804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce8c06e2c0 msgr2=0x7fce8c0c99a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:50.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.316+0000 7fce9329e700 1 --2- 172.21.15.1:0/3830501804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce8c06e2c0 0x7fce8c0c99a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fce8800c700 tx=0x7fce8800cac0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.316+0000 7fce9329e700 1 -- 172.21.15.1:0/3830501804 shutdown_connections 2024-03-18T18:50:50.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.316+0000 7fce9329e700 1 --2- 172.21.15.1:0/3830501804 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce7006ac90 0x7fce7006d150 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.316+0000 7fce9329e700 1 --2- 172.21.15.1:0/3830501804 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce8c1e26d0 0x7fce8c0c79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.316+0000 7fce9329e700 1 --2- 172.21.15.1:0/3830501804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce8c06e2c0 0x7fce8c0c99a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.316+0000 7fce9329e700 1 --2- 172.21.15.1:0/3830501804 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce8c071600 0x7fce8c154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.316+0000 7fce9329e700 1 -- 172.21.15.1:0/3830501804 >> 172.21.15.1:0/3830501804 conn(0x7fce8c0cc990 msgr2=0x7fce8c0cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:50.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.316+0000 7fce9329e700 1 -- 172.21.15.1:0/3830501804 shutdown_connections 2024-03-18T18:50:50.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.316+0000 7fce9329e700 1 -- 172.21.15.1:0/3830501804 wait complete. 2024-03-18T18:50:50.326 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T18:49:42.800801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T18:49:43.776684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T18:49:46.854608+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T18:50:50.326 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T18:50:50.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.456+0000 7fa5a7d53700 1 Processor -- start 2024-03-18T18:50:50.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.456+0000 7fa5a7d53700 1 -- start start 2024-03-18T18:50:50.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.457+0000 7fa5a7d53700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa5a015a1d0 0x7fa5a00add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:50.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.457+0000 7fa5a7d53700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5a00ce8f0 con 0x7fa5a0071620 2024-03-18T18:50:50.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.457+0000 7fa5a7d53700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5a00ae260 con 0x7fa5a00cc0f0 2024-03-18T18:50:50.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.457+0000 7fa5a7d53700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5a00cdb00 con 0x7fa5a015a1d0 2024-03-18T18:50:50.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.457+0000 7fa5a5aef700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa5a015a1d0 0x7fa5a00add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:50.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.457+0000 7fa5a52ee700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa5a0071620 0x7fa5a00cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32996/0 (socket says 172.21.15.1:32996) 2024-03-18T18:50:50.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.457+0000 7fa5a5aef700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa5a015a1d0 0x7fa5a00add20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59360/0 (socket says 172.21.15.1:59360) 2024-03-18T18:50:50.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.457+0000 7fa5a52ee700 1 -- 172.21.15.1:0/728454879 learned_addr learned my addr 172.21.15.1:0/728454879 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:50.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.457+0000 7fa5a5aef700 1 -- 172.21.15.1:0/728454879 >> v1:172.21.15.165:6789/0 conn(0x7fa5a00cc0f0 legacy=0x7fa5a015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:50.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.457+0000 7fa5a5aef700 1 -- 172.21.15.1:0/728454879 >> v1:172.21.15.1:6789/0 conn(0x7fa5a0071620 legacy=0x7fa5a00cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:50.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.457+0000 7fa5a5aef700 1 -- 172.21.15.1:0/728454879 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a00cdce0 con 0x7fa5a015a1d0 2024-03-18T18:50:50.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.458+0000 7fa5a5aef700 1 --2- 172.21.15.1:0/728454879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a015a1d0 0x7fa5a00add20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa590009820 tx=0x7fa590009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed35929ffb8807b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:50.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.458+0000 7fa5a4aed700 1 -- 172.21.15.1:0/728454879 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa590005e30 con 0x7fa5a015a1d0 2024-03-18T18:50:50.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.458+0000 7fa5a4aed700 1 -- 172.21.15.1:0/728454879 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa590006030 con 0x7fa5a015a1d0 2024-03-18T18:50:50.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.458+0000 7fa5a4aed700 1 -- 172.21.15.1:0/728454879 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5900063e0 con 0x7fa5a015a1d0 2024-03-18T18:50:50.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.458+0000 7fa5a7d53700 1 -- 172.21.15.1:0/728454879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a015a1d0 msgr2=0x7fa5a00add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:50.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.458+0000 7fa5a7d53700 1 --2- 172.21.15.1:0/728454879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a015a1d0 0x7fa5a00add20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa590009820 tx=0x7fa590009b30 comp rx=0 tx=0).stop 2024-03-18T18:50:50.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.458+0000 7fa5a7d53700 1 -- 172.21.15.1:0/728454879 shutdown_connections 2024-03-18T18:50:50.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.458+0000 7fa5a7d53700 1 --2- 172.21.15.1:0/728454879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a015a1d0 0x7fa5a00add20 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.458+0000 7fa5a7d53700 1 -- 172.21.15.1:0/728454879 >> 172.21.15.1:0/728454879 conn(0x7fa5a00cc9b0 msgr2=0x7fa5a00ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:50.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.458+0000 7fa5a7d53700 1 -- 172.21.15.1:0/728454879 shutdown_connections 2024-03-18T18:50:50.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.458+0000 7fa5a7d53700 1 -- 172.21.15.1:0/728454879 wait complete. 2024-03-18T18:50:50.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.460+0000 7fa5a7d53700 1 Processor -- start 2024-03-18T18:50:50.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.460+0000 7fa5a7d53700 1 -- start start 2024-03-18T18:50:50.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.461+0000 7fa5a7d53700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a0071620 0x7fa5a0145390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:50.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.461+0000 7fa5a7d53700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5a00cc0f0 0x7fa5a015c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:50.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.461+0000 7fa5a7d53700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5a015a1d0 0x7fa5a015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:50.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.461+0000 7fa5a7d53700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a00ca2a0 con 0x7fa5a0071620 2024-03-18T18:50:50.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.461+0000 7fa5a7d53700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a01458d0 con 0x7fa5a00cc0f0 2024-03-18T18:50:50.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.461+0000 7fa5a7d53700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a01f15b0 con 0x7fa5a015a1d0 2024-03-18T18:50:50.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.461+0000 7fa5a5aef700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a0071620 0x7fa5a0145390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:50.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.461+0000 7fa5a5aef700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a0071620 0x7fa5a0145390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59372/0 (socket says 172.21.15.1:59372) 2024-03-18T18:50:50.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.461+0000 7fa5a5aef700 1 -- 172.21.15.1:0/3808159239 learned_addr learned my addr 172.21.15.1:0/3808159239 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:50.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.461+0000 7fa5a5aef700 1 -- 172.21.15.1:0/3808159239 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5a015a1d0 msgr2=0x7fa5a015cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:50.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.461+0000 7fa5a5aef700 1 --2- 172.21.15.1:0/3808159239 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5a015a1d0 0x7fa5a015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.461+0000 7fa5a5aef700 1 -- 172.21.15.1:0/3808159239 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5a00cc0f0 msgr2=0x7fa5a015c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:50.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.461+0000 7fa5a52ee700 1 --2- 172.21.15.1:0/3808159239 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5a00cc0f0 0x7fa5a015c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:50.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.461+0000 7fa5a5aef700 1 --2- 172.21.15.1:0/3808159239 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5a00cc0f0 0x7fa5a015c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.461+0000 7fa5a5aef700 1 -- 172.21.15.1:0/3808159239 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa590009430 con 0x7fa5a0071620 2024-03-18T18:50:50.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.461+0000 7fa5a52ee700 1 --2- 172.21.15.1:0/3808159239 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5a00cc0f0 0x7fa5a015c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:50.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.462+0000 7fa5a5aef700 1 --2- 172.21.15.1:0/3808159239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a0071620 0x7fa5a0145390 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa590000c00 tx=0x7fa590004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:50.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.462+0000 7fa5967fc700 1 -- 172.21.15.1:0/3808159239 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa590004250 con 0x7fa5a0071620 2024-03-18T18:50:50.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.462+0000 7fa5967fc700 1 -- 172.21.15.1:0/3808159239 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5900043d0 con 0x7fa5a0071620 2024-03-18T18:50:50.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.462+0000 7fa5a7d53700 1 -- 172.21.15.1:0/3808159239 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a00c9e10 con 0x7fa5a0071620 2024-03-18T18:50:50.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.462+0000 7fa5967fc700 1 -- 172.21.15.1:0/3808159239 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5900036a0 con 0x7fa5a0071620 2024-03-18T18:50:50.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.462+0000 7fa5a7d53700 1 -- 172.21.15.1:0/3808159239 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a00c9200 con 0x7fa5a0071620 2024-03-18T18:50:50.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.463+0000 7fa5a7d53700 1 -- 172.21.15.1:0/3808159239 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5a00ca0f0 con 0x7fa5a0071620 2024-03-18T18:50:50.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.463+0000 7fa5967fc700 1 -- 172.21.15.1:0/3808159239 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa590022070 con 0x7fa5a0071620 2024-03-18T18:50:50.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.464+0000 7fa5967fc700 1 --2- 172.21.15.1:0/3808159239 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa58406ac90 0x7fa58406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:50.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.464+0000 7fa5967fc700 1 -- 172.21.15.1:0/3808159239 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa59008b0a0 con 0x7fa5a0071620 2024-03-18T18:50:50.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.464+0000 7fa5a52ee700 1 --2- 172.21.15.1:0/3808159239 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa58406ac90 0x7fa58406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:50.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.464+0000 7fa5a52ee700 1 --2- 172.21.15.1:0/3808159239 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa58406ac90 0x7fa58406d150 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa5a00c6d90 tx=0x7fa59c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:50.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.469+0000 7fa5967fc700 1 -- 172.21.15.1:0/3808159239 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa590055f50 con 0x7fa5a0071620 2024-03-18T18:50:50.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.669+0000 7fa5a7d53700 1 -- 172.21.15.1:0/3808159239 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa5a00ad340 con 0x7fa5a0071620 2024-03-18T18:50:50.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.669+0000 7fa5967fc700 1 -- 172.21.15.1:0/3808159239 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa590059570 con 0x7fa5a0071620 2024-03-18T18:50:50.667 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T18:50:50.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.672+0000 7fa5a7d53700 1 -- 172.21.15.1:0/3808159239 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa58406ac90 msgr2=0x7fa58406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.672+0000 7fa5a7d53700 1 --2- 172.21.15.1:0/3808159239 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa58406ac90 0x7fa58406d150 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa5a00c6d90 tx=0x7fa59c009040 comp rx=0 tx=0).stop 2024-03-18T18:50:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.672+0000 7fa5a7d53700 1 -- 172.21.15.1:0/3808159239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a0071620 msgr2=0x7fa5a0145390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.672+0000 7fa5a7d53700 1 --2- 172.21.15.1:0/3808159239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a0071620 0x7fa5a0145390 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa590000c00 tx=0x7fa590004750 comp rx=0 tx=0).stop 2024-03-18T18:50:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.672+0000 7fa5a7d53700 1 -- 172.21.15.1:0/3808159239 shutdown_connections 2024-03-18T18:50:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.672+0000 7fa5a7d53700 1 --2- 172.21.15.1:0/3808159239 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa58406ac90 0x7fa58406d150 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.672+0000 7fa5a7d53700 1 --2- 172.21.15.1:0/3808159239 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5a015a1d0 0x7fa5a015cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.672+0000 7fa5a7d53700 1 --2- 172.21.15.1:0/3808159239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a0071620 0x7fa5a0145390 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.672+0000 7fa5a7d53700 1 --2- 172.21.15.1:0/3808159239 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5a00cc0f0 0x7fa5a015c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.672+0000 7fa5a7d53700 1 -- 172.21.15.1:0/3808159239 >> 172.21.15.1:0/3808159239 conn(0x7fa5a00cc9b0 msgr2=0x7fa5a00b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.672+0000 7fa5a7d53700 1 -- 172.21.15.1:0/3808159239 shutdown_connections 2024-03-18T18:50:50.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.672+0000 7fa5a7d53700 1 -- 172.21.15.1:0/3808159239 wait complete. 2024-03-18T18:50:50.680 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T18:50:50.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.812+0000 7fac2393d700 1 Processor -- start 2024-03-18T18:50:50.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.812+0000 7fac2393d700 1 -- start start 2024-03-18T18:50:50.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.813+0000 7fac2393d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fac1c0b1e20 0x7fac1c0cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:50.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.813+0000 7fac2393d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fac1c0cbce0 0x7fac1c0cc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:50.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.813+0000 7fac2393d700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac1c0d0df0 con 0x7fac1c0cd870 2024-03-18T18:50:50.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.813+0000 7fac2393d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac1c0d0550 con 0x7fac1c0b1e20 2024-03-18T18:50:50.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.813+0000 7fac2393d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac1c0d0830 con 0x7fac1c0cbce0 2024-03-18T18:50:50.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.813+0000 7fac20ed8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fac1c0b1e20 0x7fac1c0cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:50.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.813+0000 7fac20ed8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fac1c0b1e20 0x7fac1c0cff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59382/0 (socket says 172.21.15.1:59382) 2024-03-18T18:50:50.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.813+0000 7fac20ed8700 1 -- 172.21.15.1:0/4091923639 learned_addr learned my addr 172.21.15.1:0/4091923639 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:50.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.814+0000 7fac21eda700 1 --2- 172.21.15.1:0/4091923639 >> v2:172.21.15.165:3300/0 conn(0x7fac1c0cbce0 0x7fac1c0cc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:50.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.814+0000 7fac20ed8700 1 -- 172.21.15.1:0/4091923639 >> v2:172.21.15.165:3300/0 conn(0x7fac1c0cbce0 msgr2=0x7fac1c0cc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:50.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.814+0000 7fac20ed8700 1 --2- 172.21.15.1:0/4091923639 >> v2:172.21.15.165:3300/0 conn(0x7fac1c0cbce0 0x7fac1c0cc150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.814+0000 7fac20ed8700 1 -- 172.21.15.1:0/4091923639 >> v1:172.21.15.165:6789/0 conn(0x7fac1c0cd870 legacy=0x7fac1c0cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:50.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.814+0000 7fac20ed8700 1 -- 172.21.15.1:0/4091923639 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1c0d0b10 con 0x7fac1c0b1e20 2024-03-18T18:50:50.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.814+0000 7fac21eda700 1 --2- 172.21.15.1:0/4091923639 >> v2:172.21.15.165:3300/0 conn(0x7fac1c0cbce0 0x7fac1c0cc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:50.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.814+0000 7fac20ed8700 1 --2- 172.21.15.1:0/4091923639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c0b1e20 0x7fac1c0cff70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0ce550 tx=0x7fac18009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8d3bab9a5c3a07c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:50.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.814+0000 7fac13fff700 1 -- 172.21.15.1:0/4091923639 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18004230 con 0x7fac1c0b1e20 2024-03-18T18:50:50.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.814+0000 7fac13fff700 1 -- 172.21.15.1:0/4091923639 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac18004390 con 0x7fac1c0b1e20 2024-03-18T18:50:50.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.814+0000 7fac13fff700 1 -- 172.21.15.1:0/4091923639 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac180036a0 con 0x7fac1c0b1e20 2024-03-18T18:50:50.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.814+0000 7fac2393d700 1 -- 172.21.15.1:0/4091923639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c0b1e20 msgr2=0x7fac1c0cff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:50.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.814+0000 7fac2393d700 1 --2- 172.21.15.1:0/4091923639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c0b1e20 0x7fac1c0cff70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0ce550 tx=0x7fac18009bd0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.815+0000 7fac2393d700 1 -- 172.21.15.1:0/4091923639 shutdown_connections 2024-03-18T18:50:50.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.815+0000 7fac2393d700 1 --2- 172.21.15.1:0/4091923639 >> v2:172.21.15.165:3300/0 conn(0x7fac1c0cbce0 0x7fac1c0cc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.815+0000 7fac2393d700 1 --2- 172.21.15.1:0/4091923639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c0b1e20 0x7fac1c0cff70 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.815+0000 7fac2393d700 1 -- 172.21.15.1:0/4091923639 >> 172.21.15.1:0/4091923639 conn(0x7fac1c0d0ff0 msgr2=0x7fac1c0d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:50.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.815+0000 7fac2393d700 1 -- 172.21.15.1:0/4091923639 shutdown_connections 2024-03-18T18:50:50.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.815+0000 7fac2393d700 1 -- 172.21.15.1:0/4091923639 wait complete. 2024-03-18T18:50:50.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.816+0000 7fac2393d700 1 Processor -- start 2024-03-18T18:50:50.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.816+0000 7fac2393d700 1 -- start start 2024-03-18T18:50:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac2393d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1c0b1e20 0x7fac1c0b0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac2393d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c0cbce0 0x7fac1c0aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac2393d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac1c0cd870 0x7fac1c0af050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac2393d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c0b02a0 con 0x7fac1c0cbce0 2024-03-18T18:50:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac2393d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c0af690 con 0x7fac1c0b1e20 2024-03-18T18:50:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac2393d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c0af970 con 0x7fac1c0cd870 2024-03-18T18:50:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac20ed8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c0cbce0 0x7fac1c0aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac20ed8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c0cbce0 0x7fac1c0aeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59390/0 (socket says 172.21.15.1:59390) 2024-03-18T18:50:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac20ed8700 1 -- 172.21.15.1:0/2861327164 learned_addr learned my addr 172.21.15.1:0/2861327164 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac20ed8700 1 -- 172.21.15.1:0/2861327164 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac1c0cd870 msgr2=0x7fac1c0af050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac20ed8700 1 --2- 172.21.15.1:0/2861327164 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac1c0cd870 0x7fac1c0af050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac20ed8700 1 -- 172.21.15.1:0/2861327164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1c0b1e20 msgr2=0x7fac1c0b0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:50.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac216d9700 1 --2- 172.21.15.1:0/2861327164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1c0b1e20 0x7fac1c0b0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac20ed8700 1 --2- 172.21.15.1:0/2861327164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1c0b1e20 0x7fac1c0b0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac20ed8700 1 -- 172.21.15.1:0/2861327164 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac18009550 con 0x7fac1c0cbce0 2024-03-18T18:50:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac20ed8700 1 --2- 172.21.15.1:0/2861327164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c0cbce0 0x7fac1c0aeb10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fac1c072000 tx=0x7fac18004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac11ffb700 1 -- 172.21.15.1:0/2861327164 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18003d80 con 0x7fac1c0cbce0 2024-03-18T18:50:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac11ffb700 1 -- 172.21.15.1:0/2861327164 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1801f070 con 0x7fac1c0cbce0 2024-03-18T18:50:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac2393d700 1 -- 172.21.15.1:0/2861327164 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac1c0afc50 con 0x7fac1c0cbce0 2024-03-18T18:50:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.817+0000 7fac11ffb700 1 -- 172.21.15.1:0/2861327164 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18011870 con 0x7fac1c0cbce0 2024-03-18T18:50:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.818+0000 7fac2393d700 1 -- 172.21.15.1:0/2861327164 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1c159cd0 con 0x7fac1c0cbce0 2024-03-18T18:50:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.818+0000 7fac2393d700 1 -- 172.21.15.1:0/2861327164 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1c0676a0 con 0x7fac1c0cbce0 2024-03-18T18:50:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.819+0000 7fac11ffb700 1 -- 172.21.15.1:0/2861327164 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fac18021030 con 0x7fac1c0cbce0 2024-03-18T18:50:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.819+0000 7fac11ffb700 1 --2- 172.21.15.1:0/2861327164 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac0006ac90 0x7fac0006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.820+0000 7fac216d9700 1 --2- 172.21.15.1:0/2861327164 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac0006ac90 0x7fac0006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.820+0000 7fac11ffb700 1 -- 172.21.15.1:0/2861327164 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fac1808ade0 con 0x7fac1c0cbce0 2024-03-18T18:50:50.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.820+0000 7fac216d9700 1 --2- 172.21.15.1:0/2861327164 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac0006ac90 0x7fac0006d150 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fac0c00a7d0 tx=0x7fac0c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:50.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:50.824+0000 7fac11ffb700 1 -- 172.21.15.1:0/2861327164 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fac18055bf0 con 0x7fac1c0cbce0 2024-03-18T18:50:51.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.024+0000 7fac2393d700 1 -- 172.21.15.1:0/2861327164 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fac1c0b03e0 con 0x7fac1c0cbce0 2024-03-18T18:50:51.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.025+0000 7fac11ffb700 1 -- 172.21.15.1:0/2861327164 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fac18059210 con 0x7fac1c0cbce0 2024-03-18T18:50:51.021 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:51.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.027+0000 7fac2393d700 1 -- 172.21.15.1:0/2861327164 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac0006ac90 msgr2=0x7fac0006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:51.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.027+0000 7fac2393d700 1 --2- 172.21.15.1:0/2861327164 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac0006ac90 0x7fac0006d150 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fac0c00a7d0 tx=0x7fac0c008040 comp rx=0 tx=0).stop 2024-03-18T18:50:51.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.027+0000 7fac2393d700 1 -- 172.21.15.1:0/2861327164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c0cbce0 msgr2=0x7fac1c0aeb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:51.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.027+0000 7fac2393d700 1 --2- 172.21.15.1:0/2861327164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c0cbce0 0x7fac1c0aeb10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fac1c072000 tx=0x7fac18004750 comp rx=0 tx=0).stop 2024-03-18T18:50:51.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.027+0000 7fac2393d700 1 -- 172.21.15.1:0/2861327164 shutdown_connections 2024-03-18T18:50:51.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.027+0000 7fac2393d700 1 --2- 172.21.15.1:0/2861327164 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac0006ac90 0x7fac0006d150 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.027+0000 7fac2393d700 1 --2- 172.21.15.1:0/2861327164 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac1c0cd870 0x7fac1c0af050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.027+0000 7fac2393d700 1 --2- 172.21.15.1:0/2861327164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1c0b1e20 0x7fac1c0b0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.027+0000 7fac2393d700 1 --2- 172.21.15.1:0/2861327164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c0cbce0 0x7fac1c0aeb10 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.027+0000 7fac2393d700 1 -- 172.21.15.1:0/2861327164 >> 172.21.15.1:0/2861327164 conn(0x7fac1c0d0ff0 msgr2=0x7fac1c0cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:51.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.027+0000 7fac2393d700 1 -- 172.21.15.1:0/2861327164 shutdown_connections 2024-03-18T18:50:51.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.027+0000 7fac2393d700 1 -- 172.21.15.1:0/2861327164 wait complete. 2024-03-18T18:50:51.036 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T18:50:51.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.168+0000 7f0d19a5d700 1 Processor -- start 2024-03-18T18:50:51.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.168+0000 7f0d19a5d700 1 -- start start 2024-03-18T18:50:51.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.169+0000 7f0d19a5d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0d140b4810 0x7f0d140d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:51.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.169+0000 7f0d19a5d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d140cde70 0x7f0d140ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:51.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.169+0000 7f0d19a5d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d140b3370 con 0x7f0d140cfb50 2024-03-18T18:50:51.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.169+0000 7f0d19a5d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d140b2760 con 0x7f0d140b4810 2024-03-18T18:50:51.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.169+0000 7f0d19a5d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d140b2a40 con 0x7f0d140cde70 2024-03-18T18:50:51.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.169+0000 7f0d127fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0d140b4810 0x7f0d140d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:51.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.169+0000 7f0d12ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0d140cfb50 0x7f0d140cff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33010/0 (socket says 172.21.15.1:33010) 2024-03-18T18:50:51.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.169+0000 7f0d12ffd700 1 -- 172.21.15.1:0/2741170634 learned_addr learned my addr 172.21.15.1:0/2741170634 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:51.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.169+0000 7f0d127fc700 1 -- 172.21.15.1:0/2741170634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d140cde70 msgr2=0x7f0d140ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:51.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.169+0000 7f0d127fc700 1 --2- 172.21.15.1:0/2741170634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d140cde70 0x7f0d140ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.169+0000 7f0d127fc700 1 -- 172.21.15.1:0/2741170634 >> v1:172.21.15.1:6789/0 conn(0x7f0d140cfb50 legacy=0x7f0d140cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:51.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.169+0000 7f0d127fc700 1 -- 172.21.15.1:0/2741170634 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d140b2d20 con 0x7f0d140b4810 2024-03-18T18:50:51.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.169+0000 7f0d11ffb700 1 -- 172.21.15.1:0/2741170634 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692803402 0 0) 0x7f0d140b3370 con 0x7f0d140cfb50 2024-03-18T18:50:51.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.169+0000 7f0d127fc700 1 --2- 172.21.15.1:0/2741170634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d140b4810 0x7f0d140d3c60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0d08009840 tx=0x7f0d08009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d22373a0e27565c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:51.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.169+0000 7f0d11ffb700 1 -- 172.21.15.1:0/2741170634 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d08011830 con 0x7f0d140b4810 2024-03-18T18:50:51.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.169+0000 7f0d11ffb700 1 -- 172.21.15.1:0/2741170634 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d08011990 con 0x7f0d140b4810 2024-03-18T18:50:51.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.169+0000 7f0d11ffb700 1 -- 172.21.15.1:0/2741170634 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d08011d20 con 0x7f0d140b4810 2024-03-18T18:50:51.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.170+0000 7f0d19a5d700 1 -- 172.21.15.1:0/2741170634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d140b4810 msgr2=0x7f0d140d3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:51.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.170+0000 7f0d19a5d700 1 --2- 172.21.15.1:0/2741170634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d140b4810 0x7f0d140d3c60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0d08009840 tx=0x7f0d08009b50 comp rx=0 tx=0).stop 2024-03-18T18:50:51.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.170+0000 7f0d19a5d700 1 -- 172.21.15.1:0/2741170634 shutdown_connections 2024-03-18T18:50:51.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.170+0000 7f0d19a5d700 1 --2- 172.21.15.1:0/2741170634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d140cde70 0x7f0d140ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.170+0000 7f0d19a5d700 1 --2- 172.21.15.1:0/2741170634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d140b4810 0x7f0d140d3c60 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.170+0000 7f0d19a5d700 1 -- 172.21.15.1:0/2741170634 >> 172.21.15.1:0/2741170634 conn(0x7f0d140d3140 msgr2=0x7f0d140d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:51.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.170+0000 7f0d19a5d700 1 -- 172.21.15.1:0/2741170634 shutdown_connections 2024-03-18T18:50:51.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.170+0000 7f0d19a5d700 1 -- 172.21.15.1:0/2741170634 wait complete. 2024-03-18T18:50:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.171+0000 7f0d19a5d700 1 Processor -- start 2024-03-18T18:50:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.172+0000 7f0d19a5d700 1 -- start start 2024-03-18T18:50:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.172+0000 7f0d19a5d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d140b4810 0x7f0d140b5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.172+0000 7f0d19a5d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d140cde70 0x7f0d140b0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.172+0000 7f0d19a5d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d140cfb50 0x7f0d141f3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.172+0000 7f0d19a5d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d140b62d0 con 0x7f0d140b4810 2024-03-18T18:50:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.172+0000 7f0d19a5d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d140cb9b0 con 0x7f0d140cfb50 2024-03-18T18:50:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.172+0000 7f0d19a5d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d140cbc90 con 0x7f0d140cde70 2024-03-18T18:50:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.172+0000 7f0d12ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d140b4810 0x7f0d140b5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.172+0000 7f0d12ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d140b4810 0x7f0d140b5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59414/0 (socket says 172.21.15.1:59414) 2024-03-18T18:50:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.172+0000 7f0d12ffd700 1 -- 172.21.15.1:0/1634350318 learned_addr learned my addr 172.21.15.1:0/1634350318 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.172+0000 7f0d127fc700 1 --2- 172.21.15.1:0/1634350318 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d140cde70 0x7f0d140b0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.173+0000 7f0d137fe700 1 --2- 172.21.15.1:0/1634350318 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d140cfb50 0x7f0d141f3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:51.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.173+0000 7f0d12ffd700 1 -- 172.21.15.1:0/1634350318 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d140cde70 msgr2=0x7f0d140b0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:51.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.173+0000 7f0d12ffd700 1 --2- 172.21.15.1:0/1634350318 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d140cde70 0x7f0d140b0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.173+0000 7f0d12ffd700 1 -- 172.21.15.1:0/1634350318 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d140cfb50 msgr2=0x7f0d141f3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:51.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.173+0000 7f0d12ffd700 1 --2- 172.21.15.1:0/1634350318 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d140cfb50 0x7f0d141f3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.173+0000 7f0d12ffd700 1 -- 172.21.15.1:0/1634350318 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d080094a0 con 0x7f0d140b4810 2024-03-18T18:50:51.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.173+0000 7f0d12ffd700 1 --2- 172.21.15.1:0/1634350318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d140b4810 0x7f0d140b5cf0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0d0400c4a0 tx=0x7f0d0400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:51.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.173+0000 7f0cfbfff700 1 -- 172.21.15.1:0/1634350318 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d040074e0 con 0x7f0d140b4810 2024-03-18T18:50:51.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.173+0000 7f0cfbfff700 1 -- 172.21.15.1:0/1634350318 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d04007640 con 0x7f0d140b4810 2024-03-18T18:50:51.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.173+0000 7f0d19a5d700 1 -- 172.21.15.1:0/1634350318 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d140cbf70 con 0x7f0d140b4810 2024-03-18T18:50:51.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.173+0000 7f0cfbfff700 1 -- 172.21.15.1:0/1634350318 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d04005a70 con 0x7f0d140b4810 2024-03-18T18:50:51.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.173+0000 7f0d19a5d700 1 -- 172.21.15.1:0/1634350318 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1415add0 con 0x7f0d140b4810 2024-03-18T18:50:51.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.174+0000 7f0d19a5d700 1 -- 172.21.15.1:0/1634350318 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d140cc490 con 0x7f0d140b4810 2024-03-18T18:50:51.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.174+0000 7f0cfbfff700 1 -- 172.21.15.1:0/1634350318 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0d04005bd0 con 0x7f0d140b4810 2024-03-18T18:50:51.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.175+0000 7f0cfbfff700 1 --2- 172.21.15.1:0/1634350318 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0d0006ac90 0x7f0d0006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:51.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.175+0000 7f0cfbfff700 1 -- 172.21.15.1:0/1634350318 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0d04088a00 con 0x7f0d140b4810 2024-03-18T18:50:51.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.175+0000 7f0d127fc700 1 --2- 172.21.15.1:0/1634350318 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0d0006ac90 0x7f0d0006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:51.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.175+0000 7f0d127fc700 1 --2- 172.21.15.1:0/1634350318 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0d0006ac90 0x7f0d0006d150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0d080090b0 tx=0x7f0d08008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:51.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.180+0000 7f0cfbfff700 1 -- 172.21.15.1:0/1634350318 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0d140cc490 con 0x7f0d140b4810 2024-03-18T18:50:51.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.382+0000 7f0d19a5d700 1 -- 172.21.15.1:0/1634350318 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0d140cc490 con 0x7f0d140b4810 2024-03-18T18:50:51.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.382+0000 7f0cfbfff700 1 -- 172.21.15.1:0/1634350318 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0d140cc490 con 0x7f0d140b4810 2024-03-18T18:50:51.379 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T18:50:51.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.384+0000 7f0d19a5d700 1 -- 172.21.15.1:0/1634350318 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0d0006ac90 msgr2=0x7f0d0006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:51.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.384+0000 7f0d19a5d700 1 --2- 172.21.15.1:0/1634350318 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0d0006ac90 0x7f0d0006d150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0d080090b0 tx=0x7f0d08008030 comp rx=0 tx=0).stop 2024-03-18T18:50:51.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.384+0000 7f0d19a5d700 1 -- 172.21.15.1:0/1634350318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d140b4810 msgr2=0x7f0d140b5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:51.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.384+0000 7f0d19a5d700 1 --2- 172.21.15.1:0/1634350318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d140b4810 0x7f0d140b5cf0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0d0400c4a0 tx=0x7f0d0400c7b0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.384+0000 7f0d19a5d700 1 -- 172.21.15.1:0/1634350318 shutdown_connections 2024-03-18T18:50:51.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.384+0000 7f0d19a5d700 1 --2- 172.21.15.1:0/1634350318 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0d0006ac90 0x7f0d0006d150 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.384+0000 7f0d19a5d700 1 --2- 172.21.15.1:0/1634350318 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d140cfb50 0x7f0d141f3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.384+0000 7f0d19a5d700 1 --2- 172.21.15.1:0/1634350318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d140b4810 0x7f0d140b5cf0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.384+0000 7f0d19a5d700 1 --2- 172.21.15.1:0/1634350318 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d140cde70 0x7f0d140b0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.384+0000 7f0d19a5d700 1 -- 172.21.15.1:0/1634350318 >> 172.21.15.1:0/1634350318 conn(0x7f0d140d3140 msgr2=0x7f0d1406c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:51.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.384+0000 7f0d19a5d700 1 -- 172.21.15.1:0/1634350318 shutdown_connections 2024-03-18T18:50:51.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.384+0000 7f0d19a5d700 1 -- 172.21.15.1:0/1634350318 wait complete. 2024-03-18T18:50:51.395 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T18:50:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.527+0000 7f4202a02700 1 Processor -- start 2024-03-18T18:50:51.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.527+0000 7f4202a02700 1 -- start start 2024-03-18T18:50:51.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.528+0000 7f4202a02700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f41fc15d500 0x7f41fc06c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:51.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.528+0000 7f4202a02700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc06f490 con 0x7f41fc06c7b0 2024-03-18T18:50:51.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.528+0000 7f4202a02700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc06e880 con 0x7f41fc09a150 2024-03-18T18:50:51.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.528+0000 7f4202a02700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41fc06eb60 con 0x7f41fc15d500 2024-03-18T18:50:51.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.528+0000 7f41fb7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f41fc06c7b0 0x7f41fc099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33014/0 (socket says 172.21.15.1:33014) 2024-03-18T18:50:51.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.528+0000 7f41fb7fe700 1 -- 172.21.15.1:0/1429565479 learned_addr learned my addr 172.21.15.1:0/1429565479 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:51.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.528+0000 7f41fbfff700 1 --2- 172.21.15.1:0/1429565479 >> v2:172.21.15.165:3300/0 conn(0x7f41fc15d500 0x7f41fc06c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:51.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.528+0000 7f41faffd700 1 -- 172.21.15.1:0/1429565479 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038244914 0 0) 0x7f41fc06f490 con 0x7f41fc06c7b0 2024-03-18T18:50:51.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.528+0000 7f41faffd700 1 -- 172.21.15.1:0/1429565479 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41e80035b0 con 0x7f41fc06c7b0 2024-03-18T18:50:51.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.528+0000 7f41fbfff700 1 -- 172.21.15.1:0/1429565479 >> v1:172.21.15.165:6789/0 conn(0x7f41fc09a150 legacy=0x7f41fc0cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:51.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.528+0000 7f41fbfff700 1 -- 172.21.15.1:0/1429565479 >> v1:172.21.15.1:6789/0 conn(0x7f41fc06c7b0 legacy=0x7f41fc099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:51.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.528+0000 7f41faffd700 1 -- 172.21.15.1:0/1429565479 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 143260648 0 0) 0x7f41e80035b0 con 0x7f41fc06c7b0 2024-03-18T18:50:51.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.528+0000 7f41fbfff700 1 -- 172.21.15.1:0/1429565479 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41fc06ee40 con 0x7f41fc15d500 2024-03-18T18:50:51.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.529+0000 7f41fbfff700 1 --2- 172.21.15.1:0/1429565479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41fc15d500 0x7f41fc06c270 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f41ec009840 tx=0x7f41ec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8ae49b2321ac1ca server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:51.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.529+0000 7f41faffd700 1 -- 172.21.15.1:0/1429565479 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e4001670 con 0x7f41fc15d500 2024-03-18T18:50:51.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.529+0000 7f41faffd700 1 -- 172.21.15.1:0/1429565479 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41ec004030 con 0x7f41fc15d500 2024-03-18T18:50:51.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.529+0000 7f41faffd700 1 -- 172.21.15.1:0/1429565479 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ec0043c0 con 0x7f41fc15d500 2024-03-18T18:50:51.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.529+0000 7f4202a02700 1 -- 172.21.15.1:0/1429565479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41fc15d500 msgr2=0x7f41fc06c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:51.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.529+0000 7f4202a02700 1 --2- 172.21.15.1:0/1429565479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41fc15d500 0x7f41fc06c270 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f41ec009840 tx=0x7f41ec009b50 comp rx=0 tx=0).stop 2024-03-18T18:50:51.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.529+0000 7f4202a02700 1 -- 172.21.15.1:0/1429565479 shutdown_connections 2024-03-18T18:50:51.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.530+0000 7f4202a02700 1 --2- 172.21.15.1:0/1429565479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41fc15d500 0x7f41fc06c270 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.530+0000 7f4202a02700 1 -- 172.21.15.1:0/1429565479 >> 172.21.15.1:0/1429565479 conn(0x7f41fc0b2c50 msgr2=0x7f41fc0b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:51.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.530+0000 7f4202a02700 1 -- 172.21.15.1:0/1429565479 shutdown_connections 2024-03-18T18:50:51.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.530+0000 7f4202a02700 1 -- 172.21.15.1:0/1429565479 wait complete. 2024-03-18T18:50:51.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.531+0000 7f4202a02700 1 Processor -- start 2024-03-18T18:50:51.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.531+0000 7f4202a02700 1 -- start start 2024-03-18T18:50:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f4202a02700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41fc06c7b0 0x7f41fc15da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f4202a02700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41fc09a150 0x7f41fc15adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f4202a02700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41fc15d500 0x7f41fc15f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f4202a02700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc15ecc0 con 0x7f41fc09a150 2024-03-18T18:50:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f4202a02700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc15e040 con 0x7f41fc15d500 2024-03-18T18:50:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f4202a02700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc15e320 con 0x7f41fc06c7b0 2024-03-18T18:50:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f41fb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41fc09a150 0x7f41fc15adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f41fb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41fc09a150 0x7f41fc15adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59420/0 (socket says 172.21.15.1:59420) 2024-03-18T18:50:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f41fb7fe700 1 -- 172.21.15.1:0/1610679295 learned_addr learned my addr 172.21.15.1:0/1610679295 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f4200f9f700 1 --2- 172.21.15.1:0/1610679295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41fc15d500 0x7f41fc15f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f41fbfff700 1 --2- 172.21.15.1:0/1610679295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41fc06c7b0 0x7f41fc15da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f41fb7fe700 1 -- 172.21.15.1:0/1610679295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41fc06c7b0 msgr2=0x7f41fc15da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:51.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f41fb7fe700 1 --2- 172.21.15.1:0/1610679295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41fc06c7b0 0x7f41fc15da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f41fb7fe700 1 -- 172.21.15.1:0/1610679295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41fc15d500 msgr2=0x7f41fc15f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f41fb7fe700 1 --2- 172.21.15.1:0/1610679295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41fc15d500 0x7f41fc15f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f41fb7fe700 1 -- 172.21.15.1:0/1610679295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ec0094a0 con 0x7f41fc09a150 2024-03-18T18:50:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f41fb7fe700 1 --2- 172.21.15.1:0/1610679295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41fc09a150 0x7f41fc15adf0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f41e40011e0 tx=0x7f41e400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f41f8ff9700 1 -- 172.21.15.1:0/1610679295 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e4007560 con 0x7f41fc09a150 2024-03-18T18:50:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f41f8ff9700 1 -- 172.21.15.1:0/1610679295 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e40076c0 con 0x7f41fc09a150 2024-03-18T18:50:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f41f8ff9700 1 -- 172.21.15.1:0/1610679295 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e4004a90 con 0x7f41fc09a150 2024-03-18T18:50:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f4202a02700 1 -- 172.21.15.1:0/1610679295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41fc15e600 con 0x7f41fc09a150 2024-03-18T18:50:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.532+0000 7f4202a02700 1 -- 172.21.15.1:0/1610679295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41fc15b330 con 0x7f41fc09a150 2024-03-18T18:50:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.533+0000 7f4202a02700 1 -- 172.21.15.1:0/1610679295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41fc15ee00 con 0x7f41fc09a150 2024-03-18T18:50:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.534+0000 7f41f8ff9700 1 -- 172.21.15.1:0/1610679295 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f41e4004bf0 con 0x7f41fc09a150 2024-03-18T18:50:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.534+0000 7f41f8ff9700 1 --2- 172.21.15.1:0/1610679295 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f41dc06ac90 0x7f41dc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:51.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.534+0000 7f41f8ff9700 1 -- 172.21.15.1:0/1610679295 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f41e40887f0 con 0x7f41fc09a150 2024-03-18T18:50:51.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.534+0000 7f41fbfff700 1 --2- 172.21.15.1:0/1610679295 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f41dc06ac90 0x7f41dc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:51.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.535+0000 7f41fbfff700 1 --2- 172.21.15.1:0/1610679295 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f41dc06ac90 0x7f41dc06d150 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f41ec000c00 tx=0x7f41ec013040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:51.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.539+0000 7f41f8ff9700 1 -- 172.21.15.1:0/1610679295 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f41fc15ee00 con 0x7f41fc09a150 2024-03-18T18:50:51.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.740+0000 7f4202a02700 1 -- 172.21.15.1:0/1610679295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f41fc15ee00 con 0x7f41fc09a150 2024-03-18T18:50:51.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.740+0000 7f41f8ff9700 1 -- 172.21.15.1:0/1610679295 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f41e4018020 con 0x7f41fc09a150 2024-03-18T18:50:51.737 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.742+0000 7f4202a02700 1 -- 172.21.15.1:0/1610679295 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f41dc06ac90 msgr2=0x7f41dc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.742+0000 7f4202a02700 1 --2- 172.21.15.1:0/1610679295 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f41dc06ac90 0x7f41dc06d150 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f41ec000c00 tx=0x7f41ec013040 comp rx=0 tx=0).stop 2024-03-18T18:50:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.742+0000 7f4202a02700 1 -- 172.21.15.1:0/1610679295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41fc09a150 msgr2=0x7f41fc15adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.742+0000 7f4202a02700 1 --2- 172.21.15.1:0/1610679295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41fc09a150 0x7f41fc15adf0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f41e40011e0 tx=0x7f41e400cb00 comp rx=0 tx=0).stop 2024-03-18T18:50:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.742+0000 7f4202a02700 1 -- 172.21.15.1:0/1610679295 shutdown_connections 2024-03-18T18:50:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.742+0000 7f4202a02700 1 --2- 172.21.15.1:0/1610679295 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f41dc06ac90 0x7f41dc06d150 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.742+0000 7f4202a02700 1 --2- 172.21.15.1:0/1610679295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41fc06c7b0 0x7f41fc15da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.742+0000 7f4202a02700 1 --2- 172.21.15.1:0/1610679295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41fc15d500 0x7f41fc15f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.742+0000 7f4202a02700 1 --2- 172.21.15.1:0/1610679295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41fc09a150 0x7f41fc15adf0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.742+0000 7f4202a02700 1 -- 172.21.15.1:0/1610679295 >> 172.21.15.1:0/1610679295 conn(0x7f41fc0b2c50 msgr2=0x7f41fc0b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:51.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.742+0000 7f4202a02700 1 -- 172.21.15.1:0/1610679295 shutdown_connections 2024-03-18T18:50:51.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.742+0000 7f4202a02700 1 -- 172.21.15.1:0/1610679295 wait complete. 2024-03-18T18:50:51.751 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T18:50:51.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.881+0000 7f32a4120700 1 Processor -- start 2024-03-18T18:50:51.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.881+0000 7f32a4120700 1 -- start start 2024-03-18T18:50:51.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.882+0000 7f32a4120700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f329c15b4d0 0x7f329c0cd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:51.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.882+0000 7f32a4120700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f329c0cbf00 0x7f329c0cc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:51.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.882+0000 7f32a4120700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f329c0cf2c0 con 0x7f329c0cf9a0 2024-03-18T18:50:51.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.882+0000 7f32a4120700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f329c0725f0 con 0x7f329c0cbf00 2024-03-18T18:50:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.882+0000 7f32a4120700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f329c0cf750 con 0x7f329c15b4d0 2024-03-18T18:50:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.882+0000 7f32a26bd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f329c0cbf00 0x7f329c0cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.882+0000 7f32a26bd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f329c0cbf00 0x7f329c0cc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59430/0 (socket says 172.21.15.1:59430) 2024-03-18T18:50:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.883+0000 7f32a26bd700 1 -- 172.21.15.1:0/2664787525 learned_addr learned my addr 172.21.15.1:0/2664787525 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.883+0000 7f32a1ebc700 1 --2- 172.21.15.1:0/2664787525 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f329c15b4d0 0x7f329c0cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.883+0000 7f32a26bd700 1 -- 172.21.15.1:0/2664787525 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f329c15b4d0 msgr2=0x7f329c0cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.883+0000 7f32a26bd700 1 --2- 172.21.15.1:0/2664787525 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f329c15b4d0 0x7f329c0cd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.883+0000 7f32a26bd700 1 -- 172.21.15.1:0/2664787525 >> v1:172.21.15.165:6789/0 conn(0x7f329c0cf9a0 legacy=0x7f329c071ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.883+0000 7f32a26bd700 1 -- 172.21.15.1:0/2664787525 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329c0ca5d0 con 0x7f329c0cbf00 2024-03-18T18:50:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.883+0000 7f32a1ebc700 1 --2- 172.21.15.1:0/2664787525 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f329c15b4d0 0x7f329c0cd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.883+0000 7f32a26bd700 1 --2- 172.21.15.1:0/2664787525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0cbf00 0x7f329c0cc390 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3294009e50 tx=0x7f329400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34ae4c60eb787060 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.883+0000 7f32a0eba700 1 -- 172.21.15.1:0/2664787525 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329400e070 con 0x7f329c0cbf00 2024-03-18T18:50:51.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.883+0000 7f32a0eba700 1 -- 172.21.15.1:0/2664787525 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329400b9c0 con 0x7f329c0cbf00 2024-03-18T18:50:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.883+0000 7f32a0eba700 1 -- 172.21.15.1:0/2664787525 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329400bd70 con 0x7f329c0cbf00 2024-03-18T18:50:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.884+0000 7f32a4120700 1 -- 172.21.15.1:0/2664787525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0cbf00 msgr2=0x7f329c0cc390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.884+0000 7f32a4120700 1 --2- 172.21.15.1:0/2664787525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0cbf00 0x7f329c0cc390 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3294009e50 tx=0x7f329400b330 comp rx=0 tx=0).stop 2024-03-18T18:50:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.884+0000 7f32a4120700 1 -- 172.21.15.1:0/2664787525 shutdown_connections 2024-03-18T18:50:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.884+0000 7f32a4120700 1 --2- 172.21.15.1:0/2664787525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0cbf00 0x7f329c0cc390 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.884+0000 7f32a4120700 1 --2- 172.21.15.1:0/2664787525 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f329c15b4d0 0x7f329c0cd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.884+0000 7f32a4120700 1 -- 172.21.15.1:0/2664787525 >> 172.21.15.1:0/2664787525 conn(0x7f329c0cca00 msgr2=0x7f329c0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.884+0000 7f32a4120700 1 -- 172.21.15.1:0/2664787525 shutdown_connections 2024-03-18T18:50:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.884+0000 7f32a4120700 1 -- 172.21.15.1:0/2664787525 wait complete. 2024-03-18T18:50:51.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.885+0000 7f32a4120700 1 Processor -- start 2024-03-18T18:50:51.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.885+0000 7f32a4120700 1 -- start start 2024-03-18T18:50:51.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.886+0000 7f32a4120700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0cbf00 0x7f329c150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:51.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.886+0000 7f32a4120700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f329c0cf9a0 0x7f329c0b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:51.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.886+0000 7f32a4120700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f329c15b4d0 0x7f329c0c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:51.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.886+0000 7f32a4120700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329c1505a0 con 0x7f329c0cbf00 2024-03-18T18:50:51.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.886+0000 7f32a4120700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329c0c83a0 con 0x7f329c0cf9a0 2024-03-18T18:50:51.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.886+0000 7f32a4120700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f329c0c8680 con 0x7f329c15b4d0 2024-03-18T18:50:51.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.886+0000 7f32a1ebc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0cbf00 0x7f329c150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:51.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.886+0000 7f32a1ebc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0cbf00 0x7f329c150060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59444/0 (socket says 172.21.15.1:59444) 2024-03-18T18:50:51.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.886+0000 7f32a1ebc700 1 -- 172.21.15.1:0/2308223513 learned_addr learned my addr 172.21.15.1:0/2308223513 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:51.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.886+0000 7f32a16bb700 1 --2- 172.21.15.1:0/2308223513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f329c0cf9a0 0x7f329c0b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:51.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.886+0000 7f32a26bd700 1 --2- 172.21.15.1:0/2308223513 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f329c15b4d0 0x7f329c0c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:51.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.887+0000 7f32a1ebc700 1 -- 172.21.15.1:0/2308223513 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f329c15b4d0 msgr2=0x7f329c0c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:51.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.887+0000 7f32a1ebc700 1 --2- 172.21.15.1:0/2308223513 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f329c15b4d0 0x7f329c0c9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.887+0000 7f32a1ebc700 1 -- 172.21.15.1:0/2308223513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f329c0cf9a0 msgr2=0x7f329c0b2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:51.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.887+0000 7f32a1ebc700 1 --2- 172.21.15.1:0/2308223513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f329c0cf9a0 0x7f329c0b2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:51.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.887+0000 7f32a1ebc700 1 -- 172.21.15.1:0/2308223513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3294009b30 con 0x7f329c0cbf00 2024-03-18T18:50:51.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.887+0000 7f32a26bd700 1 --2- 172.21.15.1:0/2308223513 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f329c15b4d0 0x7f329c0c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:51.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.887+0000 7f32a1ebc700 1 --2- 172.21.15.1:0/2308223513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0cbf00 0x7f329c150060 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f328c00c750 tx=0x7f328c00ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:51.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.887+0000 7f32927fc700 1 -- 172.21.15.1:0/2308223513 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c011800 con 0x7f329c0cbf00 2024-03-18T18:50:51.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.887+0000 7f32927fc700 1 -- 172.21.15.1:0/2308223513 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328c011960 con 0x7f329c0cbf00 2024-03-18T18:50:51.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.887+0000 7f32927fc700 1 -- 172.21.15.1:0/2308223513 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c011cd0 con 0x7f329c0cbf00 2024-03-18T18:50:51.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.887+0000 7f32a4120700 1 -- 172.21.15.1:0/2308223513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f329c0c8960 con 0x7f329c0cbf00 2024-03-18T18:50:51.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.887+0000 7f32a4120700 1 -- 172.21.15.1:0/2308223513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f329c0c8f20 con 0x7f329c0cbf00 2024-03-18T18:50:51.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.888+0000 7f32a4120700 1 -- 172.21.15.1:0/2308223513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f329c0676a0 con 0x7f329c0cbf00 2024-03-18T18:50:51.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.888+0000 7f32927fc700 1 -- 172.21.15.1:0/2308223513 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f328c021350 con 0x7f329c0cbf00 2024-03-18T18:50:51.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.889+0000 7f32927fc700 1 --2- 172.21.15.1:0/2308223513 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f328006ac90 0x7f328006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:51.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.889+0000 7f32927fc700 1 -- 172.21.15.1:0/2308223513 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f328c089670 con 0x7f329c0cbf00 2024-03-18T18:50:51.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.889+0000 7f32a16bb700 1 --2- 172.21.15.1:0/2308223513 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f328006ac90 0x7f328006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:51.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.889+0000 7f32a16bb700 1 --2- 172.21.15.1:0/2308223513 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f328006ac90 0x7f328006d150 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f329c0b5f40 tx=0x7f3298009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:51.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:51.893+0000 7f32927fc700 1 -- 172.21.15.1:0/2308223513 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f328c0543f0 con 0x7f329c0cbf00 2024-03-18T18:50:52.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.113+0000 7f32a4120700 1 -- 172.21.15.1:0/2308223513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f329c0cd670 con 0x7f329c0cbf00 2024-03-18T18:50:52.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.114+0000 7f32927fc700 1 -- 172.21.15.1:0/2308223513 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f328c057a10 con 0x7f329c0cbf00 2024-03-18T18:50:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.116+0000 7f32a4120700 1 -- 172.21.15.1:0/2308223513 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f328006ac90 msgr2=0x7f328006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.116+0000 7f32a4120700 1 --2- 172.21.15.1:0/2308223513 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f328006ac90 0x7f328006d150 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f329c0b5f40 tx=0x7f3298009040 comp rx=0 tx=0).stop 2024-03-18T18:50:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.116+0000 7f32a4120700 1 -- 172.21.15.1:0/2308223513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0cbf00 msgr2=0x7f329c150060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.116+0000 7f32a4120700 1 --2- 172.21.15.1:0/2308223513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0cbf00 0x7f329c150060 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f328c00c750 tx=0x7f328c00ca60 comp rx=0 tx=0).stop 2024-03-18T18:50:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.116+0000 7f32a4120700 1 -- 172.21.15.1:0/2308223513 shutdown_connections 2024-03-18T18:50:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.116+0000 7f32a4120700 1 --2- 172.21.15.1:0/2308223513 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f328006ac90 0x7f328006d150 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.116+0000 7f32a4120700 1 --2- 172.21.15.1:0/2308223513 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f329c15b4d0 0x7f329c0c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.116+0000 7f32a4120700 1 --2- 172.21.15.1:0/2308223513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f329c0cbf00 0x7f329c150060 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.116+0000 7f32a4120700 1 --2- 172.21.15.1:0/2308223513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f329c0cf9a0 0x7f329c0b2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.116+0000 7f32a4120700 1 -- 172.21.15.1:0/2308223513 >> 172.21.15.1:0/2308223513 conn(0x7f329c0cca00 msgr2=0x7f329c0acdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.116+0000 7f32a4120700 1 -- 172.21.15.1:0/2308223513 shutdown_connections 2024-03-18T18:50:52.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.116+0000 7f32a4120700 1 -- 172.21.15.1:0/2308223513 wait complete. 2024-03-18T18:50:52.114 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-18T18:50:52.124 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-18T18:50:52.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:52.125 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-03-18T18:50:52.162 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T18:50:52.162 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T18:50:52.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:52.163 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-18T18:50:52.220 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:50:52.221 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T18:50:52.243 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:50:52.243 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T18:50:52.307 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:52.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.436+0000 7f282ac6a700 1 Processor -- start 2024-03-18T18:50:52.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.437+0000 7f282ac6a700 1 -- start start 2024-03-18T18:50:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.438+0000 7f282ac6a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2824099a40 0x7f282406ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.438+0000 7f282ac6a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f28240d05a0 0x7f2824099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.438+0000 7f282ac6a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28240cf7e0 con 0x7f282409a360 2024-03-18T18:50:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.438+0000 7f282ac6a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28240cfac0 con 0x7f2824099a40 2024-03-18T18:50:52.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.438+0000 7f282ac6a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28240cfda0 con 0x7f28240d05a0 2024-03-18T18:50:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.438+0000 7f2828a06700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2824099a40 0x7f282406ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.438+0000 7f2828a06700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2824099a40 0x7f282406ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59452/0 (socket says 172.21.15.1:59452) 2024-03-18T18:50:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.438+0000 7f2828a06700 1 -- 172.21.15.1:0/3019495054 learned_addr learned my addr 172.21.15.1:0/3019495054 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.438+0000 7f2828a06700 1 -- 172.21.15.1:0/3019495054 >> v2:172.21.15.165:3300/0 conn(0x7f28240d05a0 msgr2=0x7f2824099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.438+0000 7f2823fff700 1 --2- 172.21.15.1:0/3019495054 >> v2:172.21.15.165:3300/0 conn(0x7f28240d05a0 0x7f2824099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.438+0000 7f2828a06700 1 --2- 172.21.15.1:0/3019495054 >> v2:172.21.15.165:3300/0 conn(0x7f28240d05a0 0x7f2824099e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:52.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.438+0000 7f2828a06700 1 -- 172.21.15.1:0/3019495054 >> v1:172.21.15.165:6789/0 conn(0x7f282409a360 legacy=0x7f282406e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.438+0000 7f2828a06700 1 -- 172.21.15.1:0/3019495054 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28240d0080 con 0x7f2824099a40 2024-03-18T18:50:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.438+0000 7f2828a06700 1 --2- 172.21.15.1:0/3019495054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2824099a40 0x7f282406ba70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2824071d40 tx=0x7f2818009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dcf8184ad449ace server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.438+0000 7f28237fe700 1 -- 172.21.15.1:0/3019495054 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2818004230 con 0x7f2824099a40 2024-03-18T18:50:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.439+0000 7f28237fe700 1 -- 172.21.15.1:0/3019495054 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2818004390 con 0x7f2824099a40 2024-03-18T18:50:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.439+0000 7f28237fe700 1 -- 172.21.15.1:0/3019495054 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28180036a0 con 0x7f2824099a40 2024-03-18T18:50:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.439+0000 7f282ac6a700 1 -- 172.21.15.1:0/3019495054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2824099a40 msgr2=0x7f282406ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.439+0000 7f282ac6a700 1 --2- 172.21.15.1:0/3019495054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2824099a40 0x7f282406ba70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2824071d40 tx=0x7f2818009bd0 comp rx=0 tx=0).stop 2024-03-18T18:50:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.439+0000 7f282ac6a700 1 -- 172.21.15.1:0/3019495054 shutdown_connections 2024-03-18T18:50:52.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.439+0000 7f282ac6a700 1 --2- 172.21.15.1:0/3019495054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2824099a40 0x7f282406ba70 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.439+0000 7f282ac6a700 1 --2- 172.21.15.1:0/3019495054 >> v2:172.21.15.165:3300/0 conn(0x7f28240d05a0 0x7f2824099e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.439+0000 7f282ac6a700 1 -- 172.21.15.1:0/3019495054 >> 172.21.15.1:0/3019495054 conn(0x7f28240b2cb0 msgr2=0x7f28240b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.439+0000 7f282ac6a700 1 -- 172.21.15.1:0/3019495054 shutdown_connections 2024-03-18T18:50:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.439+0000 7f282ac6a700 1 -- 172.21.15.1:0/3019495054 wait complete. 2024-03-18T18:50:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.440+0000 7f282ac6a700 1 Processor -- start 2024-03-18T18:50:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.440+0000 7f282ac6a700 1 -- start start 2024-03-18T18:50:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.441+0000 7f282ac6a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2824099a40 0x7f2824146280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:52.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.441+0000 7f282ac6a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f282409a360 0x7f28240b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.441+0000 7f282ac6a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28240d05a0 0x7f2824148d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.441+0000 7f282ac6a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28240b4210 con 0x7f2824099a40 2024-03-18T18:50:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.441+0000 7f282ac6a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282415ac40 con 0x7f282409a360 2024-03-18T18:50:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.441+0000 7f282ac6a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f282415b780 con 0x7f28240d05a0 2024-03-18T18:50:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.441+0000 7f2828a06700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2824099a40 0x7f2824146280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.441+0000 7f2828a06700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2824099a40 0x7f2824146280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59468/0 (socket says 172.21.15.1:59468) 2024-03-18T18:50:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.441+0000 7f2828a06700 1 -- 172.21.15.1:0/1739245766 learned_addr learned my addr 172.21.15.1:0/1739245766 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:52.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.441+0000 7f2829207700 1 --2- 172.21.15.1:0/1739245766 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28240d05a0 0x7f2824148d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:52.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.441+0000 7f2823fff700 1 --2- 172.21.15.1:0/1739245766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f282409a360 0x7f28240b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:52.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.442+0000 7f2828a06700 1 -- 172.21.15.1:0/1739245766 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28240d05a0 msgr2=0x7f2824148d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:52.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.442+0000 7f2828a06700 1 --2- 172.21.15.1:0/1739245766 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28240d05a0 0x7f2824148d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:52.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.442+0000 7f2828a06700 1 -- 172.21.15.1:0/1739245766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f282409a360 msgr2=0x7f28240b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:52.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.442+0000 7f2828a06700 1 --2- 172.21.15.1:0/1739245766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f282409a360 0x7f28240b39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:52.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.442+0000 7f2828a06700 1 -- 172.21.15.1:0/1739245766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2818009550 con 0x7f2824099a40 2024-03-18T18:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.442+0000 7f2828a06700 1 --2- 172.21.15.1:0/1739245766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2824099a40 0x7f2824146280 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2824072230 tx=0x7f2818004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.442+0000 7f28217fa700 1 -- 172.21.15.1:0/1739245766 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2818003c90 con 0x7f2824099a40 2024-03-18T18:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.442+0000 7f28217fa700 1 -- 172.21.15.1:0/1739245766 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281801f070 con 0x7f2824099a40 2024-03-18T18:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.442+0000 7f282ac6a700 1 -- 172.21.15.1:0/1739245766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f282415ae50 con 0x7f2824099a40 2024-03-18T18:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.442+0000 7f28217fa700 1 -- 172.21.15.1:0/1739245766 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2818011870 con 0x7f2824099a40 2024-03-18T18:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.442+0000 7f282ac6a700 1 -- 172.21.15.1:0/1739245766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f282415b410 con 0x7f2824099a40 2024-03-18T18:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.443+0000 7f282ac6a700 1 -- 172.21.15.1:0/1739245766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f282415b5d0 con 0x7f2824099a40 2024-03-18T18:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.444+0000 7f28217fa700 1 -- 172.21.15.1:0/1739245766 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2818021030 con 0x7f2824099a40 2024-03-18T18:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.444+0000 7f28217fa700 1 --2- 172.21.15.1:0/1739245766 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f280806ac90 0x7f280806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:52.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.444+0000 7f28217fa700 1 -- 172.21.15.1:0/1739245766 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f281808ac60 con 0x7f2824099a40 2024-03-18T18:50:52.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.444+0000 7f2823fff700 1 --2- 172.21.15.1:0/1739245766 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f280806ac90 0x7f280806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:52.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.445+0000 7f2823fff700 1 --2- 172.21.15.1:0/1739245766 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f280806ac90 0x7f280806d150 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f28241467c0 tx=0x7f2810008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:52.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.449+0000 7f28217fa700 1 -- 172.21.15.1:0/1739245766 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2818055b10 con 0x7f2824099a40 2024-03-18T18:50:52.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.652+0000 7f282ac6a700 1 -- 172.21.15.1:0/1739245766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f282409aae0 con 0x7f2824099a40 2024-03-18T18:50:52.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.653+0000 7f28217fa700 1 -- 172.21.15.1:0/1739245766 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18217 (secure 0 0 0) 0x7f2818029090 con 0x7f2824099a40 2024-03-18T18:50:52.650 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:52.650 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":27,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:50:34.993482+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:52.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.655+0000 7f282ac6a700 1 -- 172.21.15.1:0/1739245766 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f280806ac90 msgr2=0x7f280806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:52.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.655+0000 7f282ac6a700 1 --2- 172.21.15.1:0/1739245766 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f280806ac90 0x7f280806d150 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f28241467c0 tx=0x7f2810008040 comp rx=0 tx=0).stop 2024-03-18T18:50:52.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.655+0000 7f282ac6a700 1 -- 172.21.15.1:0/1739245766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2824099a40 msgr2=0x7f2824146280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:52.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.655+0000 7f282ac6a700 1 --2- 172.21.15.1:0/1739245766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2824099a40 0x7f2824146280 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2824072230 tx=0x7f2818004750 comp rx=0 tx=0).stop 2024-03-18T18:50:52.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.655+0000 7f282ac6a700 1 -- 172.21.15.1:0/1739245766 shutdown_connections 2024-03-18T18:50:52.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.655+0000 7f282ac6a700 1 --2- 172.21.15.1:0/1739245766 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f280806ac90 0x7f280806d150 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:52.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.655+0000 7f282ac6a700 1 --2- 172.21.15.1:0/1739245766 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28240d05a0 0x7f2824148d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:52.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.655+0000 7f282ac6a700 1 --2- 172.21.15.1:0/1739245766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2824099a40 0x7f2824146280 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:52.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.655+0000 7f282ac6a700 1 --2- 172.21.15.1:0/1739245766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f282409a360 0x7f28240b39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:52.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.655+0000 7f282ac6a700 1 -- 172.21.15.1:0/1739245766 >> 172.21.15.1:0/1739245766 conn(0x7f28240b2cb0 msgr2=0x7f28240b0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:52.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.656+0000 7f282ac6a700 1 -- 172.21.15.1:0/1739245766 shutdown_connections 2024-03-18T18:50:52.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.656+0000 7f282ac6a700 1 -- 172.21.15.1:0/1739245766 wait complete. 2024-03-18T18:50:52.664 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T18:49:42.800801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T18:49:43.776684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T18:49:46.854608+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T18:50:52.665 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T18:50:52.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.797+0000 7f5ceb6b5700 1 Processor -- start 2024-03-18T18:50:52.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.797+0000 7f5ceb6b5700 1 -- start start 2024-03-18T18:50:52.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.798+0000 7f5ceb6b5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5ce415a5f0 0x7f5ce40cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:52.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.798+0000 7f5ceb6b5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5ce40cc4d0 0x7f5ce406d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:52.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.798+0000 7f5ceb6b5700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce415aad0 con 0x7f5ce415dbe0 2024-03-18T18:50:52.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.798+0000 7f5ceb6b5700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce413f750 con 0x7f5ce40cc4d0 2024-03-18T18:50:52.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.798+0000 7f5ceb6b5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce40cc930 con 0x7f5ce415a5f0 2024-03-18T18:50:52.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.798+0000 7f5ce9c52700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5ce40cc4d0 0x7f5ce406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:52.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.798+0000 7f5ce9451700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5ce415dbe0 0x7f5ce40b2fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33016/0 (socket says 172.21.15.1:33016) 2024-03-18T18:50:52.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.798+0000 7f5ce9451700 1 -- 172.21.15.1:0/872064466 learned_addr learned my addr 172.21.15.1:0/872064466 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:52.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.798+0000 7f5ce8c50700 1 --2- 172.21.15.1:0/872064466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5ce415a5f0 0x7f5ce40cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:52.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.799+0000 7f5ce9c52700 1 -- 172.21.15.1:0/872064466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5ce415a5f0 msgr2=0x7f5ce40cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:52.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.799+0000 7f5ce9c52700 1 --2- 172.21.15.1:0/872064466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5ce415a5f0 0x7f5ce40cbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:52.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.799+0000 7f5ce9c52700 1 -- 172.21.15.1:0/872064466 >> v1:172.21.15.1:6789/0 conn(0x7f5ce415dbe0 legacy=0x7f5ce40b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.799+0000 7f5cdbfff700 1 -- 172.21.15.1:0/872064466 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212612358 0 0) 0x7f5ce415aad0 con 0x7f5ce415dbe0 2024-03-18T18:50:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.799+0000 7f5ce9c52700 1 -- 172.21.15.1:0/872064466 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce415adb0 con 0x7f5ce40cc4d0 2024-03-18T18:50:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.799+0000 7f5ce9c52700 1 --2- 172.21.15.1:0/872064466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ce40cc4d0 0x7f5ce406d500 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc009d90 tx=0x7f5cdc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e58926bd887be8dc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.799+0000 7f5cdbfff700 1 -- 172.21.15.1:0/872064466 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc00e070 con 0x7f5ce40cc4d0 2024-03-18T18:50:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.799+0000 7f5cdbfff700 1 -- 172.21.15.1:0/872064466 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cdc00bbd0 con 0x7f5ce40cc4d0 2024-03-18T18:50:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.799+0000 7f5cdbfff700 1 -- 172.21.15.1:0/872064466 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc012430 con 0x7f5ce40cc4d0 2024-03-18T18:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.799+0000 7f5ceb6b5700 1 -- 172.21.15.1:0/872064466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ce40cc4d0 msgr2=0x7f5ce406d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.799+0000 7f5ceb6b5700 1 --2- 172.21.15.1:0/872064466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ce40cc4d0 0x7f5ce406d500 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc009d90 tx=0x7f5cdc00b330 comp rx=0 tx=0).stop 2024-03-18T18:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.799+0000 7f5ceb6b5700 1 -- 172.21.15.1:0/872064466 shutdown_connections 2024-03-18T18:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.799+0000 7f5ceb6b5700 1 --2- 172.21.15.1:0/872064466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ce40cc4d0 0x7f5ce406d500 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.799+0000 7f5ceb6b5700 1 --2- 172.21.15.1:0/872064466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5ce415a5f0 0x7f5ce40cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.799+0000 7f5ceb6b5700 1 -- 172.21.15.1:0/872064466 >> 172.21.15.1:0/872064466 conn(0x7f5ce40b1cb0 msgr2=0x7f5ce40b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.800+0000 7f5ceb6b5700 1 -- 172.21.15.1:0/872064466 shutdown_connections 2024-03-18T18:50:52.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.800+0000 7f5ceb6b5700 1 -- 172.21.15.1:0/872064466 wait complete. 2024-03-18T18:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.801+0000 7f5ceb6b5700 1 Processor -- start 2024-03-18T18:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.801+0000 7f5ceb6b5700 1 -- start start 2024-03-18T18:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.801+0000 7f5ceb6b5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ce40cc4d0 0x7f5ce41f28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5ceb6b5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5ce415a5f0 0x7f5ce41536d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5ceb6b5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ce415dbe0 0x7f5ce4158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5ceb6b5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce4150dd0 con 0x7f5ce415dbe0 2024-03-18T18:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5ceb6b5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce4153c10 con 0x7f5ce40cc4d0 2024-03-18T18:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5ceb6b5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce41526a0 con 0x7f5ce415a5f0 2024-03-18T18:50:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5ce9c52700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ce415dbe0 0x7f5ce4158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5ce9c52700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ce415dbe0 0x7f5ce4158fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59490/0 (socket says 172.21.15.1:59490) 2024-03-18T18:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5ce9c52700 1 -- 172.21.15.1:0/2734042333 learned_addr learned my addr 172.21.15.1:0/2734042333 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5ce9451700 1 --2- 172.21.15.1:0/2734042333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ce40cc4d0 0x7f5ce41f28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5ce8c50700 1 --2- 172.21.15.1:0/2734042333 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5ce415a5f0 0x7f5ce41536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5ce9c52700 1 -- 172.21.15.1:0/2734042333 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5ce415a5f0 msgr2=0x7f5ce41536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5ce9c52700 1 --2- 172.21.15.1:0/2734042333 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5ce415a5f0 0x7f5ce41536d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5ce9c52700 1 -- 172.21.15.1:0/2734042333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ce40cc4d0 msgr2=0x7f5ce41f28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5ce9c52700 1 --2- 172.21.15.1:0/2734042333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ce40cc4d0 0x7f5ce41f28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5ce9c52700 1 -- 172.21.15.1:0/2734042333 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cdc009a70 con 0x7f5ce415dbe0 2024-03-18T18:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5ce9451700 1 --2- 172.21.15.1:0/2734042333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ce40cc4d0 0x7f5ce41f28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5ce9c52700 1 --2- 172.21.15.1:0/2734042333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ce415dbe0 0x7f5ce4158fe0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40728b0 tx=0x7f5cdc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5cd9ffb700 1 -- 172.21.15.1:0/2734042333 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc012430 con 0x7f5ce415dbe0 2024-03-18T18:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5cd9ffb700 1 -- 172.21.15.1:0/2734042333 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cdc020070 con 0x7f5ce415dbe0 2024-03-18T18:50:52.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5ceb6b5700 1 -- 172.21.15.1:0/2734042333 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce4151b20 con 0x7f5ce415dbe0 2024-03-18T18:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.802+0000 7f5cd9ffb700 1 -- 172.21.15.1:0/2734042333 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc004280 con 0x7f5ce415dbe0 2024-03-18T18:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.803+0000 7f5ceb6b5700 1 -- 172.21.15.1:0/2734042333 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce41520e0 con 0x7f5ce415dbe0 2024-03-18T18:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.804+0000 7f5ceb6b5700 1 -- 172.21.15.1:0/2734042333 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce41f2ea0 con 0x7f5ce415dbe0 2024-03-18T18:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.805+0000 7f5cd9ffb700 1 -- 172.21.15.1:0/2734042333 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5cdc0043e0 con 0x7f5ce415dbe0 2024-03-18T18:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.805+0000 7f5cd9ffb700 1 --2- 172.21.15.1:0/2734042333 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5cc806ac90 0x7f5cc806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.805+0000 7f5cd9ffb700 1 -- 172.21.15.1:0/2734042333 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5cdc022070 con 0x7f5ce415dbe0 2024-03-18T18:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.805+0000 7f5ce9451700 1 --2- 172.21.15.1:0/2734042333 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5cc806ac90 0x7f5cc806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:52.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.806+0000 7f5ce9451700 1 --2- 172.21.15.1:0/2734042333 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5cc806ac90 0x7f5cc806d150 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4007fd0 tx=0x7f5cd4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:52.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:52.809+0000 7f5cd9ffb700 1 -- 172.21.15.1:0/2734042333 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5cdc056b30 con 0x7f5ce415dbe0 2024-03-18T18:50:53.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.011+0000 7f5ceb6b5700 1 -- 172.21.15.1:0/2734042333 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5ce40d0c00 con 0x7f5ce415dbe0 2024-03-18T18:50:53.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.012+0000 7f5cd9ffb700 1 -- 172.21.15.1:0/2734042333 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5cdc05a150 con 0x7f5ce415dbe0 2024-03-18T18:50:53.008 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T18:50:53.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.014+0000 7f5ceb6b5700 1 -- 172.21.15.1:0/2734042333 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5cc806ac90 msgr2=0x7f5cc806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.014+0000 7f5ceb6b5700 1 --2- 172.21.15.1:0/2734042333 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5cc806ac90 0x7f5cc806d150 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4007fd0 tx=0x7f5cd4009040 comp rx=0 tx=0).stop 2024-03-18T18:50:53.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.014+0000 7f5ceb6b5700 1 -- 172.21.15.1:0/2734042333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ce415dbe0 msgr2=0x7f5ce4158fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.014+0000 7f5ceb6b5700 1 --2- 172.21.15.1:0/2734042333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ce415dbe0 0x7f5ce4158fe0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40728b0 tx=0x7f5cdc004ad0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.014+0000 7f5ceb6b5700 1 -- 172.21.15.1:0/2734042333 shutdown_connections 2024-03-18T18:50:53.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.014+0000 7f5ceb6b5700 1 --2- 172.21.15.1:0/2734042333 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5cc806ac90 0x7f5cc806d150 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.014+0000 7f5ceb6b5700 1 --2- 172.21.15.1:0/2734042333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ce415dbe0 0x7f5ce4158fe0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.014+0000 7f5ceb6b5700 1 --2- 172.21.15.1:0/2734042333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ce40cc4d0 0x7f5ce41f28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.014+0000 7f5ceb6b5700 1 --2- 172.21.15.1:0/2734042333 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5ce415a5f0 0x7f5ce41536d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.014+0000 7f5ceb6b5700 1 -- 172.21.15.1:0/2734042333 >> 172.21.15.1:0/2734042333 conn(0x7f5ce40b1cb0 msgr2=0x7f5ce415d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:53.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.014+0000 7f5ceb6b5700 1 -- 172.21.15.1:0/2734042333 shutdown_connections 2024-03-18T18:50:53.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.014+0000 7f5ceb6b5700 1 -- 172.21.15.1:0/2734042333 wait complete. 2024-03-18T18:50:53.023 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T18:50:53.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.154+0000 7f10dcea9700 1 Processor -- start 2024-03-18T18:50:53.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.154+0000 7f10dcea9700 1 -- start start 2024-03-18T18:50:53.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.155+0000 7f10dcea9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f10d80b3a40 0x7f10d81682c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:53.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.155+0000 7f10dcea9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d806ed30 con 0x7f10d8146020 2024-03-18T18:50:53.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.155+0000 7f10dcea9700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d806f010 con 0x7f10d806fdd0 2024-03-18T18:50:53.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.155+0000 7f10dcea9700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d806eb50 con 0x7f10d80b3a40 2024-03-18T18:50:53.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.156+0000 7f10d6d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f10d80b3a40 0x7f10d81682c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:53.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.156+0000 7f10d659c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f10d8146020 0x7f10d81465a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33024/0 (socket says 172.21.15.1:33024) 2024-03-18T18:50:53.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.156+0000 7f10d6d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f10d80b3a40 0x7f10d81682c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59504/0 (socket says 172.21.15.1:59504) 2024-03-18T18:50:53.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.156+0000 7f10d659c700 1 -- 172.21.15.1:0/1047892185 learned_addr learned my addr 172.21.15.1:0/1047892185 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:53.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.156+0000 7f10d6d9d700 1 -- 172.21.15.1:0/1047892185 >> v1:172.21.15.165:6789/0 conn(0x7f10d806fdd0 legacy=0x7f10d80b3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:53.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.156+0000 7f10d6d9d700 1 -- 172.21.15.1:0/1047892185 >> v1:172.21.15.1:6789/0 conn(0x7f10d8146020 legacy=0x7f10d81465a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.156+0000 7f10d559a700 1 -- 172.21.15.1:0/1047892185 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963733612 0 0) 0x7f10d806ed30 con 0x7f10d8146020 2024-03-18T18:50:53.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.156+0000 7f10d6d9d700 1 -- 172.21.15.1:0/1047892185 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d806f2f0 con 0x7f10d80b3a40 2024-03-18T18:50:53.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.156+0000 7f10d6d9d700 1 --2- 172.21.15.1:0/1047892185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10d80b3a40 0x7f10d81682c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f10d0009d90 tx=0x7f10d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6292ef77af050c66 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:53.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.156+0000 7f10d559a700 1 -- 172.21.15.1:0/1047892185 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d000e070 con 0x7f10d80b3a40 2024-03-18T18:50:53.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.156+0000 7f10d559a700 1 -- 172.21.15.1:0/1047892185 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d000bb90 con 0x7f10d80b3a40 2024-03-18T18:50:53.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.156+0000 7f10d559a700 1 -- 172.21.15.1:0/1047892185 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d0003a20 con 0x7f10d80b3a40 2024-03-18T18:50:53.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.157+0000 7f10dcea9700 1 -- 172.21.15.1:0/1047892185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10d80b3a40 msgr2=0x7f10d81682c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.157+0000 7f10dcea9700 1 --2- 172.21.15.1:0/1047892185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10d80b3a40 0x7f10d81682c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f10d0009d90 tx=0x7f10d000b330 comp rx=0 tx=0).stop 2024-03-18T18:50:53.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.157+0000 7f10dcea9700 1 -- 172.21.15.1:0/1047892185 shutdown_connections 2024-03-18T18:50:53.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.157+0000 7f10dcea9700 1 --2- 172.21.15.1:0/1047892185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10d80b3a40 0x7f10d81682c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.157+0000 7f10dcea9700 1 -- 172.21.15.1:0/1047892185 >> 172.21.15.1:0/1047892185 conn(0x7f10d80b1d20 msgr2=0x7f10d80b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:53.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.157+0000 7f10dcea9700 1 -- 172.21.15.1:0/1047892185 shutdown_connections 2024-03-18T18:50:53.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.157+0000 7f10dcea9700 1 -- 172.21.15.1:0/1047892185 wait complete. 2024-03-18T18:50:53.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.158+0000 7f10dcea9700 1 Processor -- start 2024-03-18T18:50:53.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.158+0000 7f10dcea9700 1 -- start start 2024-03-18T18:50:53.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.159+0000 7f10dcea9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10d806fdd0 0x7f10d80cca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:53.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.159+0000 7f10dcea9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10d80b3a40 0x7f10d8137d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:53.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.159+0000 7f10dcea9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10d8146020 0x7f10d80b0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:53.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.159+0000 7f10dcea9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d806f9b0 con 0x7f10d8146020 2024-03-18T18:50:53.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.159+0000 7f10dcea9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d80cddf0 con 0x7f10d80b3a40 2024-03-18T18:50:53.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.159+0000 7f10dcea9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d80ccf40 con 0x7f10d806fdd0 2024-03-18T18:50:53.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.159+0000 7f10d6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10d8146020 0x7f10d80b0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:53.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.159+0000 7f10d6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10d8146020 0x7f10d80b0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59518/0 (socket says 172.21.15.1:59518) 2024-03-18T18:50:53.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.159+0000 7f10d6d9d700 1 -- 172.21.15.1:0/2683738624 learned_addr learned my addr 172.21.15.1:0/2683738624 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:53.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.159+0000 7f10d5d9b700 1 --2- 172.21.15.1:0/2683738624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10d80b3a40 0x7f10d8137d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:53.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.159+0000 7f10d659c700 1 --2- 172.21.15.1:0/2683738624 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10d806fdd0 0x7f10d80cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:53.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.159+0000 7f10d6d9d700 1 -- 172.21.15.1:0/2683738624 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10d806fdd0 msgr2=0x7f10d80cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.159+0000 7f10d6d9d700 1 --2- 172.21.15.1:0/2683738624 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10d806fdd0 0x7f10d80cca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.159+0000 7f10d6d9d700 1 -- 172.21.15.1:0/2683738624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10d80b3a40 msgr2=0x7f10d8137d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.159+0000 7f10d6d9d700 1 --2- 172.21.15.1:0/2683738624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10d80b3a40 0x7f10d8137d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.159+0000 7f10d6d9d700 1 -- 172.21.15.1:0/2683738624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d0009a70 con 0x7f10d8146020 2024-03-18T18:50:53.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.160+0000 7f10d6d9d700 1 --2- 172.21.15.1:0/2683738624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10d8146020 0x7f10d80b0510 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f10d8067530 tx=0x7f10d0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:53.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.160+0000 7f10ceffd700 1 -- 172.21.15.1:0/2683738624 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d0003a20 con 0x7f10d8146020 2024-03-18T18:50:53.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.160+0000 7f10ceffd700 1 -- 172.21.15.1:0/2683738624 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d0012860 con 0x7f10d8146020 2024-03-18T18:50:53.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.160+0000 7f10dcea9700 1 -- 172.21.15.1:0/2683738624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d80cd1e0 con 0x7f10d8146020 2024-03-18T18:50:53.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.160+0000 7f10ceffd700 1 -- 172.21.15.1:0/2683738624 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d0012c10 con 0x7f10d8146020 2024-03-18T18:50:53.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.160+0000 7f10dcea9700 1 -- 172.21.15.1:0/2683738624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d80cd7a0 con 0x7f10d8146020 2024-03-18T18:50:53.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.161+0000 7f10dcea9700 1 -- 172.21.15.1:0/2683738624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d8070f20 con 0x7f10d8146020 2024-03-18T18:50:53.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.161+0000 7f10ceffd700 1 -- 172.21.15.1:0/2683738624 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f10d001d420 con 0x7f10d8146020 2024-03-18T18:50:53.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.161+0000 7f10ceffd700 1 --2- 172.21.15.1:0/2683738624 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f10b806ac90 0x7f10b806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:53.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.162+0000 7f10ceffd700 1 -- 172.21.15.1:0/2683738624 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f10d008ba20 con 0x7f10d8146020 2024-03-18T18:50:53.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.162+0000 7f10d659c700 1 --2- 172.21.15.1:0/2683738624 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f10b806ac90 0x7f10b806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:53.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.162+0000 7f10d659c700 1 --2- 172.21.15.1:0/2683738624 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f10b806ac90 0x7f10b806d150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f10c8000f60 tx=0x7f10c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:53.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.166+0000 7f10ceffd700 1 -- 172.21.15.1:0/2683738624 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f10d0056830 con 0x7f10d8146020 2024-03-18T18:50:53.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.368+0000 7f10dcea9700 1 -- 172.21.15.1:0/2683738624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f10d80ce180 con 0x7f10d8146020 2024-03-18T18:50:53.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.368+0000 7f10ceffd700 1 -- 172.21.15.1:0/2683738624 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f10d0059e50 con 0x7f10d8146020 2024-03-18T18:50:53.365 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:53.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.370+0000 7f10dcea9700 1 -- 172.21.15.1:0/2683738624 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f10b806ac90 msgr2=0x7f10b806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.370+0000 7f10dcea9700 1 --2- 172.21.15.1:0/2683738624 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f10b806ac90 0x7f10b806d150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f10c8000f60 tx=0x7f10c8009040 comp rx=0 tx=0).stop 2024-03-18T18:50:53.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.370+0000 7f10dcea9700 1 -- 172.21.15.1:0/2683738624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10d8146020 msgr2=0x7f10d80b0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.370+0000 7f10dcea9700 1 --2- 172.21.15.1:0/2683738624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10d8146020 0x7f10d80b0510 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f10d8067530 tx=0x7f10d0004ad0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.370+0000 7f10dcea9700 1 -- 172.21.15.1:0/2683738624 shutdown_connections 2024-03-18T18:50:53.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.370+0000 7f10dcea9700 1 --2- 172.21.15.1:0/2683738624 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f10b806ac90 0x7f10b806d150 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.370+0000 7f10dcea9700 1 --2- 172.21.15.1:0/2683738624 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10d806fdd0 0x7f10d80cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.370+0000 7f10dcea9700 1 --2- 172.21.15.1:0/2683738624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10d8146020 0x7f10d80b0510 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.370+0000 7f10dcea9700 1 --2- 172.21.15.1:0/2683738624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10d80b3a40 0x7f10d8137d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.370+0000 7f10dcea9700 1 -- 172.21.15.1:0/2683738624 >> 172.21.15.1:0/2683738624 conn(0x7f10d80b1d20 msgr2=0x7f10d80af1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:53.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.370+0000 7f10dcea9700 1 -- 172.21.15.1:0/2683738624 shutdown_connections 2024-03-18T18:50:53.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.370+0000 7f10dcea9700 1 -- 172.21.15.1:0/2683738624 wait complete. 2024-03-18T18:50:53.379 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T18:50:53.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.511+0000 7fbd8c06f700 1 Processor -- start 2024-03-18T18:50:53.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.511+0000 7fbd8c06f700 1 -- start start 2024-03-18T18:50:53.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd8c06f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbd84143c60 0x7fbd8406eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:53.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd8c06f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8406f430 con 0x7fbd84072330 2024-03-18T18:50:53.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd8c06f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8406f710 con 0x7fbd840b4a50 2024-03-18T18:50:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd8c06f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd840cda00 con 0x7fbd84143c60 2024-03-18T18:50:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd8960a700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbd84072330 0x7fbd840b4340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33038/0 (socket says 172.21.15.1:33038) 2024-03-18T18:50:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd8960a700 1 -- 172.21.15.1:0/2476525012 learned_addr learned my addr 172.21.15.1:0/2476525012 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd89e0b700 1 --2- 172.21.15.1:0/2476525012 >> v2:172.21.15.165:3300/0 conn(0x7fbd84143c60 0x7fbd8406eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd88e09700 1 -- 172.21.15.1:0/2476525012 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337663297 0 0) 0x7fbd8406f430 con 0x7fbd84072330 2024-03-18T18:50:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd88e09700 1 -- 172.21.15.1:0/2476525012 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd700035b0 con 0x7fbd84072330 2024-03-18T18:50:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd88e09700 1 -- 172.21.15.1:0/2476525012 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079718505 0 0) 0x7fbd8406f710 con 0x7fbd840b4a50 2024-03-18T18:50:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd88e09700 1 -- 172.21.15.1:0/2476525012 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd8406f430 con 0x7fbd840b4a50 2024-03-18T18:50:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd88e09700 1 -- 172.21.15.1:0/2476525012 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 964866192 0 0) 0x7fbd700035b0 con 0x7fbd84072330 2024-03-18T18:50:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd88e09700 1 -- 172.21.15.1:0/2476525012 >> v2:172.21.15.165:3300/0 conn(0x7fbd84143c60 msgr2=0x7fbd8406eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd88e09700 1 --2- 172.21.15.1:0/2476525012 >> v2:172.21.15.165:3300/0 conn(0x7fbd84143c60 0x7fbd8406eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd88e09700 1 -- 172.21.15.1:0/2476525012 >> v1:172.21.15.165:6789/0 conn(0x7fbd840b4a50 legacy=0x7fbd840ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd88e09700 1 -- 172.21.15.1:0/2476525012 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd840cdce0 con 0x7fbd84072330 2024-03-18T18:50:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd88e09700 1 -- 172.21.15.1:0/2476525012 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fbd80001670 con 0x7fbd84072330 2024-03-18T18:50:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd88e09700 1 -- 172.21.15.1:0/2476525012 >> v1:172.21.15.1:6789/0 conn(0x7fbd84072330 legacy=0x7fbd840b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd88e09700 1 --2- 172.21.15.1:0/2476525012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd70003b90 0x7fbd70003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd88e09700 1 --2- 172.21.15.1:0/2476525012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd70004850 0x7fbd70004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd88e09700 1 --2- 172.21.15.1:0/2476525012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd70005280 0x7fbd7000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd88e09700 1 -- 172.21.15.1:0/2476525012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd840cda00 con 0x7fbd70005280 2024-03-18T18:50:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd88e09700 1 -- 172.21.15.1:0/2476525012 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7000c1e0 con 0x7fbd70003b90 2024-03-18T18:50:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd88e09700 1 -- 172.21.15.1:0/2476525012 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7000e5b0 con 0x7fbd70004850 2024-03-18T18:50:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd89e0b700 1 --2- 172.21.15.1:0/2476525012 >> v2:172.21.15.165:3300/0 conn(0x7fbd84143c60 0x7fbd8406eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:50:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.512+0000 7fbd8a60c700 1 --2- 172.21.15.1:0/2476525012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd70005280 0x7fbd7000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.513+0000 7fbd8960a700 1 --2- 172.21.15.1:0/2476525012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd70004850 0x7fbd70004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.513+0000 7fbd89e0b700 1 --2- 172.21.15.1:0/2476525012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd70003b90 0x7fbd70003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.513+0000 7fbd8a60c700 1 -- 172.21.15.1:0/2476525012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd70004850 msgr2=0x7fbd70004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.513+0000 7fbd8a60c700 1 --2- 172.21.15.1:0/2476525012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd70004850 0x7fbd70004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.513+0000 7fbd8a60c700 1 -- 172.21.15.1:0/2476525012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd70003b90 msgr2=0x7fbd70003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.513+0000 7fbd8a60c700 1 --2- 172.21.15.1:0/2476525012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd70003b90 0x7fbd70003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.513+0000 7fbd8a60c700 1 -- 172.21.15.1:0/2476525012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd700035b0 con 0x7fbd70005280 2024-03-18T18:50:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.513+0000 7fbd8960a700 1 --2- 172.21.15.1:0/2476525012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd70004850 0x7fbd70004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.513+0000 7fbd8a60c700 1 -- 172.21.15.1:0/2476525012 reap_dead start 2024-03-18T18:50:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.513+0000 7fbd8a60c700 1 --2- 172.21.15.1:0/2476525012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd70005280 0x7fbd7000ba10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c002230 tx=0x7fbd70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d468c54503596ac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.513+0000 7fbd88e09700 1 -- 172.21.15.1:0/2476525012 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c001f30 con 0x7fbd70005280 2024-03-18T18:50:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.513+0000 7fbd88e09700 1 -- 172.21.15.1:0/2476525012 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd7c00ce80 con 0x7fbd70005280 2024-03-18T18:50:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.513+0000 7fbd88e09700 1 -- 172.21.15.1:0/2476525012 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c003c40 con 0x7fbd70005280 2024-03-18T18:50:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.514+0000 7fbd8c06f700 1 -- 172.21.15.1:0/2476525012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd70005280 msgr2=0x7fbd7000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.514+0000 7fbd8c06f700 1 --2- 172.21.15.1:0/2476525012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd70005280 0x7fbd7000ba10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c002230 tx=0x7fbd70005880 comp rx=0 tx=0).stop 2024-03-18T18:50:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.514+0000 7fbd8c06f700 1 -- 172.21.15.1:0/2476525012 shutdown_connections 2024-03-18T18:50:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.514+0000 7fbd8c06f700 1 --2- 172.21.15.1:0/2476525012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd70005280 0x7fbd7000ba10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.514+0000 7fbd8c06f700 1 -- 172.21.15.1:0/2476525012 >> 172.21.15.1:0/2476525012 conn(0x7fbd840b1d20 msgr2=0x7fbd840b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.514+0000 7fbd8c06f700 1 -- 172.21.15.1:0/2476525012 shutdown_connections 2024-03-18T18:50:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.514+0000 7fbd8c06f700 1 -- 172.21.15.1:0/2476525012 wait complete. 2024-03-18T18:50:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.515+0000 7fbd8c06f700 1 Processor -- start 2024-03-18T18:50:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.515+0000 7fbd8c06f700 1 -- start start 2024-03-18T18:50:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd8c06f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd70005280 0x7fbd840c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd8c06f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd840cb330 0x7fbd840cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd8c06f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd840ae3f0 0x7fbd840ae870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd8c06f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd840cbcf0 con 0x7fbd840cb330 2024-03-18T18:50:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd8c06f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd840aedb0 con 0x7fbd840ae3f0 2024-03-18T18:50:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd8c06f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd840cbef0 con 0x7fbd70005280 2024-03-18T18:50:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd8960a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd840cb330 0x7fbd840cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd8960a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd840cb330 0x7fbd840cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59530/0 (socket says 172.21.15.1:59530) 2024-03-18T18:50:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd8960a700 1 -- 172.21.15.1:0/2228753073 learned_addr learned my addr 172.21.15.1:0/2228753073 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd8a60c700 1 --2- 172.21.15.1:0/2228753073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd840ae3f0 0x7fbd840ae870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd89e0b700 1 --2- 172.21.15.1:0/2228753073 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd70005280 0x7fbd840c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd8960a700 1 -- 172.21.15.1:0/2228753073 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd70005280 msgr2=0x7fbd840c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd8960a700 1 --2- 172.21.15.1:0/2228753073 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd70005280 0x7fbd840c9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd8960a700 1 -- 172.21.15.1:0/2228753073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd840ae3f0 msgr2=0x7fbd840ae870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd8960a700 1 --2- 172.21.15.1:0/2228753073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd840ae3f0 0x7fbd840ae870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd8960a700 1 -- 172.21.15.1:0/2228753073 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd840cc1d0 con 0x7fbd840cb330 2024-03-18T18:50:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd8a60c700 1 --2- 172.21.15.1:0/2228753073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd840ae3f0 0x7fbd840ae870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd8960a700 1 --2- 172.21.15.1:0/2228753073 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd840cb330 0x7fbd840cb7b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbd800017a0 tx=0x7fbd8000eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd767fc700 1 -- 172.21.15.1:0/2228753073 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd80012590 con 0x7fbd840cb330 2024-03-18T18:50:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.516+0000 7fbd767fc700 1 -- 172.21.15.1:0/2228753073 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd800126f0 con 0x7fbd840cb330 2024-03-18T18:50:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.517+0000 7fbd767fc700 1 -- 172.21.15.1:0/2228753073 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd80012a60 con 0x7fbd840cb330 2024-03-18T18:50:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.517+0000 7fbd8c06f700 1 -- 172.21.15.1:0/2228753073 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd840cc4b0 con 0x7fbd840cb330 2024-03-18T18:50:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.517+0000 7fbd8c06f700 1 -- 172.21.15.1:0/2228753073 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd840cca70 con 0x7fbd840cb330 2024-03-18T18:50:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.517+0000 7fbd8c06f700 1 -- 172.21.15.1:0/2228753073 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd840676a0 con 0x7fbd840cb330 2024-03-18T18:50:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.518+0000 7fbd767fc700 1 -- 172.21.15.1:0/2228753073 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fbd80012bc0 con 0x7fbd840cb330 2024-03-18T18:50:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.518+0000 7fbd767fc700 1 --2- 172.21.15.1:0/2228753073 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbd6806ac90 0x7fbd6806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.519+0000 7fbd89e0b700 1 --2- 172.21.15.1:0/2228753073 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbd6806ac90 0x7fbd6806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.519+0000 7fbd767fc700 1 -- 172.21.15.1:0/2228753073 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fbd80089390 con 0x7fbd840cb330 2024-03-18T18:50:53.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.519+0000 7fbd89e0b700 1 --2- 172.21.15.1:0/2228753073 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbd6806ac90 0x7fbd6806d150 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbd78004530 tx=0x7fbd78008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:53.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.523+0000 7fbd767fc700 1 -- 172.21.15.1:0/2228753073 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbd800541b0 con 0x7fbd840cb330 2024-03-18T18:50:53.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.723+0000 7fbd8c06f700 1 -- 172.21.15.1:0/2228753073 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbd840c9e70 con 0x7fbd840cb330 2024-03-18T18:50:53.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.723+0000 7fbd767fc700 1 -- 172.21.15.1:0/2228753073 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbd800577d0 con 0x7fbd840cb330 2024-03-18T18:50:53.720 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T18:50:53.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.725+0000 7fbd8c06f700 1 -- 172.21.15.1:0/2228753073 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbd6806ac90 msgr2=0x7fbd6806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.725+0000 7fbd8c06f700 1 --2- 172.21.15.1:0/2228753073 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbd6806ac90 0x7fbd6806d150 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbd78004530 tx=0x7fbd78008040 comp rx=0 tx=0).stop 2024-03-18T18:50:53.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.725+0000 7fbd8c06f700 1 -- 172.21.15.1:0/2228753073 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd840cb330 msgr2=0x7fbd840cb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.725+0000 7fbd8c06f700 1 --2- 172.21.15.1:0/2228753073 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd840cb330 0x7fbd840cb7b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbd800017a0 tx=0x7fbd8000eb10 comp rx=0 tx=0).stop 2024-03-18T18:50:53.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.725+0000 7fbd8c06f700 1 -- 172.21.15.1:0/2228753073 shutdown_connections 2024-03-18T18:50:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.725+0000 7fbd8c06f700 1 --2- 172.21.15.1:0/2228753073 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbd6806ac90 0x7fbd6806d150 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.725+0000 7fbd8c06f700 1 --2- 172.21.15.1:0/2228753073 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd70005280 0x7fbd840c9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.725+0000 7fbd8c06f700 1 --2- 172.21.15.1:0/2228753073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd840ae3f0 0x7fbd840ae870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.725+0000 7fbd8c06f700 1 --2- 172.21.15.1:0/2228753073 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd840cb330 0x7fbd840cb7b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.725+0000 7fbd8c06f700 1 -- 172.21.15.1:0/2228753073 >> 172.21.15.1:0/2228753073 conn(0x7fbd840b1d20 msgr2=0x7fbd840b37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.726+0000 7fbd8c06f700 1 -- 172.21.15.1:0/2228753073 shutdown_connections 2024-03-18T18:50:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.726+0000 7fbd8c06f700 1 -- 172.21.15.1:0/2228753073 wait complete. 2024-03-18T18:50:53.734 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T18:50:53.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.866+0000 7f413a545700 1 Processor -- start 2024-03-18T18:50:53.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.866+0000 7f413a545700 1 -- start start 2024-03-18T18:50:53.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.867+0000 7f413a545700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f41340aede0 0x7f4134159e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:53.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.867+0000 7f413a545700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413415e3c0 con 0x7f413406d950 2024-03-18T18:50:53.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.867+0000 7f413a545700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41340af1c0 con 0x7f41340b3320 2024-03-18T18:50:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.867+0000 7f413a545700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f413415d4d0 con 0x7f41340aede0 2024-03-18T18:50:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.867+0000 7f41337fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f413406d950 0x7f413415dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33044/0 (socket says 172.21.15.1:33044) 2024-03-18T18:50:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.867+0000 7f41337fe700 1 -- 172.21.15.1:0/3772329276 learned_addr learned my addr 172.21.15.1:0/3772329276 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.867+0000 7f4133fff700 1 --2- 172.21.15.1:0/3772329276 >> v2:172.21.15.165:3300/0 conn(0x7f41340aede0 0x7f4134159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.867+0000 7f4132ffd700 1 -- 172.21.15.1:0/3772329276 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511156761 0 0) 0x7f413415e3c0 con 0x7f413406d950 2024-03-18T18:50:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.867+0000 7f4132ffd700 1 -- 172.21.15.1:0/3772329276 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41200035b0 con 0x7f413406d950 2024-03-18T18:50:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.868+0000 7f4132ffd700 1 -- 172.21.15.1:0/3772329276 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167003866 0 0) 0x7f41340af1c0 con 0x7f41340b3320 2024-03-18T18:50:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.868+0000 7f4132ffd700 1 -- 172.21.15.1:0/3772329276 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f413415e3c0 con 0x7f41340b3320 2024-03-18T18:50:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.868+0000 7f4133fff700 1 -- 172.21.15.1:0/3772329276 >> v1:172.21.15.165:6789/0 conn(0x7f41340b3320 legacy=0x7f41340b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.868+0000 7f4132ffd700 1 -- 172.21.15.1:0/3772329276 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1816015324 0 0) 0x7f41200035b0 con 0x7f413406d950 2024-03-18T18:50:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.868+0000 7f4133fff700 1 -- 172.21.15.1:0/3772329276 >> v1:172.21.15.1:6789/0 conn(0x7f413406d950 legacy=0x7f413415dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.868+0000 7f4133fff700 1 -- 172.21.15.1:0/3772329276 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413415a730 con 0x7f41340aede0 2024-03-18T18:50:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.868+0000 7f4133fff700 1 --2- 172.21.15.1:0/3772329276 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41340aede0 0x7f4134159e70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f411c00a9e0 tx=0x7f411c00acf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa11aa7fb35d42fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.868+0000 7f4132ffd700 1 -- 172.21.15.1:0/3772329276 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4124001670 con 0x7f41340aede0 2024-03-18T18:50:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.868+0000 7f4132ffd700 1 -- 172.21.15.1:0/3772329276 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411c00ae80 con 0x7f41340aede0 2024-03-18T18:50:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.868+0000 7f4132ffd700 1 -- 172.21.15.1:0/3772329276 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c004250 con 0x7f41340aede0 2024-03-18T18:50:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.868+0000 7f413a545700 1 -- 172.21.15.1:0/3772329276 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41340aede0 msgr2=0x7f4134159e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.868+0000 7f413a545700 1 --2- 172.21.15.1:0/3772329276 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41340aede0 0x7f4134159e70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f411c00a9e0 tx=0x7f411c00acf0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.869+0000 7f413a545700 1 -- 172.21.15.1:0/3772329276 shutdown_connections 2024-03-18T18:50:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.869+0000 7f413a545700 1 --2- 172.21.15.1:0/3772329276 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41340aede0 0x7f4134159e70 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.869+0000 7f413a545700 1 -- 172.21.15.1:0/3772329276 >> 172.21.15.1:0/3772329276 conn(0x7f4134066b70 msgr2=0x7f413406a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.869+0000 7f413a545700 1 -- 172.21.15.1:0/3772329276 shutdown_connections 2024-03-18T18:50:53.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.869+0000 7f413a545700 1 -- 172.21.15.1:0/3772329276 wait complete. 2024-03-18T18:50:53.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.870+0000 7f413a545700 1 Processor -- start 2024-03-18T18:50:53.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.870+0000 7f413a545700 1 -- start start 2024-03-18T18:50:53.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f413a545700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f413406d950 0x7f4134156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:53.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f413a545700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41340aede0 0x7f41340b1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:53.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f413a545700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41340b3320 0x7f41340b1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:53.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f413a545700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4134148aa0 con 0x7f41340b3320 2024-03-18T18:50:53.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f413a545700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4134156ed0 con 0x7f413406d950 2024-03-18T18:50:53.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f413a545700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4134153cf0 con 0x7f41340aede0 2024-03-18T18:50:53.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f4138ae2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41340b3320 0x7f41340b1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:53.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f4138ae2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41340b3320 0x7f41340b1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59534/0 (socket says 172.21.15.1:59534) 2024-03-18T18:50:53.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f4138ae2700 1 -- 172.21.15.1:0/2041124279 learned_addr learned my addr 172.21.15.1:0/2041124279 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:53.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f41337fe700 1 --2- 172.21.15.1:0/2041124279 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41340aede0 0x7f41340b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:53.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f4133fff700 1 --2- 172.21.15.1:0/2041124279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f413406d950 0x7f4134156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:53.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f4138ae2700 1 -- 172.21.15.1:0/2041124279 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41340aede0 msgr2=0x7f41340b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f4138ae2700 1 --2- 172.21.15.1:0/2041124279 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41340aede0 0x7f41340b1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f4138ae2700 1 -- 172.21.15.1:0/2041124279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f413406d950 msgr2=0x7f4134156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:53.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f4138ae2700 1 --2- 172.21.15.1:0/2041124279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f413406d950 0x7f4134156990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:53.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f4138ae2700 1 -- 172.21.15.1:0/2041124279 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411c00a640 con 0x7f41340b3320 2024-03-18T18:50:53.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f4133fff700 1 --2- 172.21.15.1:0/2041124279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f413406d950 0x7f4134156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:53.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f41337fe700 1 --2- 172.21.15.1:0/2041124279 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41340aede0 0x7f41340b1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:53.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f4138ae2700 1 --2- 172.21.15.1:0/2041124279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41340b3320 0x7f41340b1980 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f412c00cbc0 tx=0x7f412c00cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:53.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.871+0000 7f4130ff9700 1 -- 172.21.15.1:0/2041124279 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c003c00 con 0x7f41340b3320 2024-03-18T18:50:53.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.872+0000 7f4130ff9700 1 -- 172.21.15.1:0/2041124279 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412c003d60 con 0x7f41340b3320 2024-03-18T18:50:53.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.872+0000 7f413a545700 1 -- 172.21.15.1:0/2041124279 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4134153fd0 con 0x7f41340b3320 2024-03-18T18:50:53.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.872+0000 7f4130ff9700 1 -- 172.21.15.1:0/2041124279 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c00abb0 con 0x7f41340b3320 2024-03-18T18:50:53.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.872+0000 7f413a545700 1 -- 172.21.15.1:0/2041124279 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4134154590 con 0x7f41340b3320 2024-03-18T18:50:53.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.872+0000 7f413a545700 1 -- 172.21.15.1:0/2041124279 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41340676a0 con 0x7f41340b3320 2024-03-18T18:50:53.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.873+0000 7f4130ff9700 1 -- 172.21.15.1:0/2041124279 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f412c00ad10 con 0x7f41340b3320 2024-03-18T18:50:53.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.874+0000 7f4130ff9700 1 --2- 172.21.15.1:0/2041124279 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f411406ac90 0x7f411406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:53.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.874+0000 7f4133fff700 1 --2- 172.21.15.1:0/2041124279 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f411406ac90 0x7f411406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:53.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.874+0000 7f4130ff9700 1 -- 172.21.15.1:0/2041124279 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f412c089820 con 0x7f41340b3320 2024-03-18T18:50:53.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.874+0000 7f4133fff700 1 --2- 172.21.15.1:0/2041124279 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f411406ac90 0x7f411406d150 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f411c000c00 tx=0x7f411c012000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:53.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:53.878+0000 7f4130ff9700 1 -- 172.21.15.1:0/2041124279 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f412c054630 con 0x7f41340b3320 2024-03-18T18:50:54.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.081+0000 7f413a545700 1 -- 172.21.15.1:0/2041124279 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f41340b4a60 con 0x7f41340b3320 2024-03-18T18:50:54.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.082+0000 7f4130ff9700 1 -- 172.21.15.1:0/2041124279 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f412c057c50 con 0x7f41340b3320 2024-03-18T18:50:54.078 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:54.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.084+0000 7f413a545700 1 -- 172.21.15.1:0/2041124279 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f411406ac90 msgr2=0x7f411406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:54.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.084+0000 7f413a545700 1 --2- 172.21.15.1:0/2041124279 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f411406ac90 0x7f411406d150 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f411c000c00 tx=0x7f411c012000 comp rx=0 tx=0).stop 2024-03-18T18:50:54.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.084+0000 7f413a545700 1 -- 172.21.15.1:0/2041124279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41340b3320 msgr2=0x7f41340b1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:54.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.084+0000 7f413a545700 1 --2- 172.21.15.1:0/2041124279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41340b3320 0x7f41340b1980 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f412c00cbc0 tx=0x7f412c00cf80 comp rx=0 tx=0).stop 2024-03-18T18:50:54.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.084+0000 7f413a545700 1 -- 172.21.15.1:0/2041124279 shutdown_connections 2024-03-18T18:50:54.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.084+0000 7f413a545700 1 --2- 172.21.15.1:0/2041124279 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f411406ac90 0x7f411406d150 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.084+0000 7f413a545700 1 --2- 172.21.15.1:0/2041124279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41340b3320 0x7f41340b1980 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.084+0000 7f413a545700 1 --2- 172.21.15.1:0/2041124279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f413406d950 0x7f4134156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.084+0000 7f413a545700 1 --2- 172.21.15.1:0/2041124279 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41340aede0 0x7f41340b1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.084+0000 7f413a545700 1 -- 172.21.15.1:0/2041124279 >> 172.21.15.1:0/2041124279 conn(0x7f4134066b70 msgr2=0x7f41340685d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:54.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.084+0000 7f413a545700 1 -- 172.21.15.1:0/2041124279 shutdown_connections 2024-03-18T18:50:54.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.084+0000 7f413a545700 1 -- 172.21.15.1:0/2041124279 wait complete. 2024-03-18T18:50:54.093 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T18:50:54.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.222+0000 7fc5c4338700 1 Processor -- start 2024-03-18T18:50:54.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.222+0000 7fc5c4338700 1 -- start start 2024-03-18T18:50:54.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.224+0000 7fc5c4338700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc5bc15a1d0 0x7fc5bc0add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:54.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.224+0000 7fc5c4338700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5bc0cc0f0 0x7fc5bc071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:54.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.224+0000 7fc5c4338700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5bc0ce8f0 con 0x7fc5bc071620 2024-03-18T18:50:54.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.224+0000 7fc5c4338700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5bc0cdb00 con 0x7fc5bc0cc0f0 2024-03-18T18:50:54.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.224+0000 7fc5c4338700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5bc15a5b0 con 0x7fc5bc15a1d0 2024-03-18T18:50:54.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.224+0000 7fc5c28d5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5bc0cc0f0 0x7fc5bc071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:54.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.224+0000 7fc5c28d5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5bc0cc0f0 0x7fc5bc071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59548/0 (socket says 172.21.15.1:59548) 2024-03-18T18:50:54.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.224+0000 7fc5c28d5700 1 -- 172.21.15.1:0/2013777900 learned_addr learned my addr 172.21.15.1:0/2013777900 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:54.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.224+0000 7fc5c20d4700 1 --2- 172.21.15.1:0/2013777900 >> v2:172.21.15.165:3300/0 conn(0x7fc5bc15a1d0 0x7fc5bc0add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:54.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.224+0000 7fc5c28d5700 1 -- 172.21.15.1:0/2013777900 >> v2:172.21.15.165:3300/0 conn(0x7fc5bc15a1d0 msgr2=0x7fc5bc0add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:54.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.224+0000 7fc5c28d5700 1 --2- 172.21.15.1:0/2013777900 >> v2:172.21.15.165:3300/0 conn(0x7fc5bc15a1d0 0x7fc5bc0add20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.224+0000 7fc5c28d5700 1 -- 172.21.15.1:0/2013777900 >> v1:172.21.15.165:6789/0 conn(0x7fc5bc071620 legacy=0x7fc5bc0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.224+0000 7fc5c28d5700 1 -- 172.21.15.1:0/2013777900 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc071fc0 con 0x7fc5bc0cc0f0 2024-03-18T18:50:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.224+0000 7fc5c28d5700 1 --2- 172.21.15.1:0/2013777900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5bc0cc0f0 0x7fc5bc071a80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc0726b0 tx=0x7fc5b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e0ca709a922c75e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.224+0000 7fc5c10d2700 1 -- 172.21.15.1:0/2013777900 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b400e070 con 0x7fc5bc0cc0f0 2024-03-18T18:50:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.224+0000 7fc5c10d2700 1 -- 172.21.15.1:0/2013777900 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b4003a20 con 0x7fc5bc0cc0f0 2024-03-18T18:50:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.224+0000 7fc5c10d2700 1 -- 172.21.15.1:0/2013777900 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4003dd0 con 0x7fc5bc0cc0f0 2024-03-18T18:50:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.225+0000 7fc5c4338700 1 -- 172.21.15.1:0/2013777900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5bc0cc0f0 msgr2=0x7fc5bc071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.225+0000 7fc5c4338700 1 --2- 172.21.15.1:0/2013777900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5bc0cc0f0 0x7fc5bc071a80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc0726b0 tx=0x7fc5b400b330 comp rx=0 tx=0).stop 2024-03-18T18:50:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.225+0000 7fc5c4338700 1 -- 172.21.15.1:0/2013777900 shutdown_connections 2024-03-18T18:50:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.225+0000 7fc5c4338700 1 --2- 172.21.15.1:0/2013777900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5bc0cc0f0 0x7fc5bc071a80 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.225+0000 7fc5c4338700 1 --2- 172.21.15.1:0/2013777900 >> v2:172.21.15.165:3300/0 conn(0x7fc5bc15a1d0 0x7fc5bc0add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.225+0000 7fc5c4338700 1 -- 172.21.15.1:0/2013777900 >> 172.21.15.1:0/2013777900 conn(0x7fc5bc0cc9b0 msgr2=0x7fc5bc0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.225+0000 7fc5c4338700 1 -- 172.21.15.1:0/2013777900 shutdown_connections 2024-03-18T18:50:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.225+0000 7fc5c4338700 1 -- 172.21.15.1:0/2013777900 wait complete. 2024-03-18T18:50:54.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.226+0000 7fc5c4338700 1 Processor -- start 2024-03-18T18:50:54.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.226+0000 7fc5c4338700 1 -- start start 2024-03-18T18:50:54.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.227+0000 7fc5c4338700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5bc071620 0x7fc5bc1527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:54.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.227+0000 7fc5c4338700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5bc0cc0f0 0x7fc5bc15c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:54.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.227+0000 7fc5c4338700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5bc15a1d0 0x7fc5bc0b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:54.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.227+0000 7fc5c4338700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc152ce0 con 0x7fc5bc071620 2024-03-18T18:50:54.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.227+0000 7fc5c4338700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc15d120 con 0x7fc5bc15a1d0 2024-03-18T18:50:54.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.227+0000 7fc5c4338700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc15d400 con 0x7fc5bc0cc0f0 2024-03-18T18:50:54.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.227+0000 7fc5c20d4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5bc071620 0x7fc5bc1527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:54.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.227+0000 7fc5c20d4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5bc071620 0x7fc5bc1527a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59556/0 (socket says 172.21.15.1:59556) 2024-03-18T18:50:54.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.227+0000 7fc5c20d4700 1 -- 172.21.15.1:0/2894971479 learned_addr learned my addr 172.21.15.1:0/2894971479 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:54.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.227+0000 7fc5c28d5700 1 --2- 172.21.15.1:0/2894971479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5bc15a1d0 0x7fc5bc0b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:54.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.227+0000 7fc5c20d4700 1 -- 172.21.15.1:0/2894971479 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5bc0cc0f0 msgr2=0x7fc5bc15c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:54.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.227+0000 7fc5c20d4700 1 --2- 172.21.15.1:0/2894971479 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5bc0cc0f0 0x7fc5bc15c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.227+0000 7fc5c20d4700 1 -- 172.21.15.1:0/2894971479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5bc15a1d0 msgr2=0x7fc5bc0b0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:54.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.227+0000 7fc5c20d4700 1 --2- 172.21.15.1:0/2894971479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5bc15a1d0 0x7fc5bc0b0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.227+0000 7fc5c20d4700 1 -- 172.21.15.1:0/2894971479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b4009b30 con 0x7fc5bc071620 2024-03-18T18:50:54.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.227+0000 7fc5c28d5700 1 --2- 172.21.15.1:0/2894971479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5bc15a1d0 0x7fc5bc0b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:54.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.227+0000 7fc5c20d4700 1 --2- 172.21.15.1:0/2894971479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5bc071620 0x7fc5bc1527a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800c7f0 tx=0x7fc5b800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:54.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.228+0000 7fc5b2ffd700 1 -- 172.21.15.1:0/2894971479 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8011800 con 0x7fc5bc071620 2024-03-18T18:50:54.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.228+0000 7fc5b2ffd700 1 -- 172.21.15.1:0/2894971479 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b8011960 con 0x7fc5bc071620 2024-03-18T18:50:54.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.228+0000 7fc5c4338700 1 -- 172.21.15.1:0/2894971479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5bc156d00 con 0x7fc5bc071620 2024-03-18T18:50:54.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.228+0000 7fc5b2ffd700 1 -- 172.21.15.1:0/2894971479 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8011d10 con 0x7fc5bc071620 2024-03-18T18:50:54.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.228+0000 7fc5c4338700 1 -- 172.21.15.1:0/2894971479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5bc1572c0 con 0x7fc5bc071620 2024-03-18T18:50:54.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.229+0000 7fc5c4338700 1 -- 172.21.15.1:0/2894971479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5bc1574e0 con 0x7fc5bc071620 2024-03-18T18:50:54.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.229+0000 7fc5b2ffd700 1 -- 172.21.15.1:0/2894971479 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc5b8018360 con 0x7fc5bc071620 2024-03-18T18:50:54.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.230+0000 7fc5b2ffd700 1 --2- 172.21.15.1:0/2894971479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc5a006ac90 0x7fc5a006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:54.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.230+0000 7fc5b2ffd700 1 -- 172.21.15.1:0/2894971479 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc5b8089720 con 0x7fc5bc071620 2024-03-18T18:50:54.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.230+0000 7fc5c18d3700 1 --2- 172.21.15.1:0/2894971479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc5a006ac90 0x7fc5a006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:54.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.230+0000 7fc5c18d3700 1 --2- 172.21.15.1:0/2894971479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc5a006ac90 0x7fc5a006d150 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc068100 tx=0x7fc5ac006c30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:54.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.234+0000 7fc5b2ffd700 1 -- 172.21.15.1:0/2894971479 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc5b8054550 con 0x7fc5bc071620 2024-03-18T18:50:54.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.456+0000 7fc5c4338700 1 -- 172.21.15.1:0/2894971479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc5bc0676a0 con 0x7fc5bc071620 2024-03-18T18:50:54.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.457+0000 7fc5b2ffd700 1 -- 172.21.15.1:0/2894971479 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc5b8057b70 con 0x7fc5bc071620 2024-03-18T18:50:54.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.459+0000 7fc5c4338700 1 -- 172.21.15.1:0/2894971479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc5a006ac90 msgr2=0x7fc5a006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:54.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.459+0000 7fc5c4338700 1 --2- 172.21.15.1:0/2894971479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc5a006ac90 0x7fc5a006d150 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc068100 tx=0x7fc5ac006c30 comp rx=0 tx=0).stop 2024-03-18T18:50:54.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.459+0000 7fc5c4338700 1 -- 172.21.15.1:0/2894971479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5bc071620 msgr2=0x7fc5bc1527a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:54.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.459+0000 7fc5c4338700 1 --2- 172.21.15.1:0/2894971479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5bc071620 0x7fc5bc1527a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800c7f0 tx=0x7fc5b800cbb0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.459+0000 7fc5c4338700 1 -- 172.21.15.1:0/2894971479 shutdown_connections 2024-03-18T18:50:54.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.459+0000 7fc5c4338700 1 --2- 172.21.15.1:0/2894971479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc5a006ac90 0x7fc5a006d150 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.459+0000 7fc5c4338700 1 --2- 172.21.15.1:0/2894971479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5bc15a1d0 0x7fc5bc0b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.459+0000 7fc5c4338700 1 --2- 172.21.15.1:0/2894971479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5bc071620 0x7fc5bc1527a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.459+0000 7fc5c4338700 1 --2- 172.21.15.1:0/2894971479 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5bc0cc0f0 0x7fc5bc15c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.459+0000 7fc5c4338700 1 -- 172.21.15.1:0/2894971479 >> 172.21.15.1:0/2894971479 conn(0x7fc5bc0cc9b0 msgr2=0x7fc5bc0ccd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:54.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.459+0000 7fc5c4338700 1 -- 172.21.15.1:0/2894971479 shutdown_connections 2024-03-18T18:50:54.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.459+0000 7fc5c4338700 1 -- 172.21.15.1:0/2894971479 wait complete. 2024-03-18T18:50:54.457 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-18T18:50:54.471 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T18:50:54.471 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T18:50:54.495 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:54.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.624+0000 7f9d60b5c700 1 Processor -- start 2024-03-18T18:50:54.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.624+0000 7f9d60b5c700 1 -- start start 2024-03-18T18:50:54.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.624+0000 7f9d60b5c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d5c06a750 0x7f9d5c0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:54.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.624+0000 7f9d60b5c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9d5c0ace80 0x7f9d5c0ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:54.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.625+0000 7f9d60b5c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5c06ee90 con 0x7f9d5c06ea10 2024-03-18T18:50:54.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.625+0000 7f9d60b5c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d5c06e1f0 con 0x7f9d5c0ace80 2024-03-18T18:50:54.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.625+0000 7f9d60b5c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c06e4d0 con 0x7f9d5c06a750 2024-03-18T18:50:54.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.625+0000 7f9d5ad9d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9d5c0ace80 0x7f9d5c0ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:54.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.625+0000 7f9d59d9b700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f9d5c06ea10 0x7f9d5c06ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:36010/0 (socket says 172.21.15.1:36010) 2024-03-18T18:50:54.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.625+0000 7f9d59d9b700 1 -- 172.21.15.1:0/1195095186 learned_addr learned my addr 172.21.15.1:0/1195095186 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:54.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.625+0000 7f9d5ad9d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9d5c0ace80 0x7f9d5c0ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:55358/0 (socket says 172.21.15.1:55358) 2024-03-18T18:50:54.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.625+0000 7f9d5a59c700 1 --2- 172.21.15.1:0/1195095186 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d5c06a750 0x7f9d5c0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:54.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.625+0000 7f9d5959a700 1 -- 172.21.15.1:0/1195095186 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964729849 0 0) 0x7f9d5c06ee90 con 0x7f9d5c06ea10 2024-03-18T18:50:54.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.625+0000 7f9d5959a700 1 -- 172.21.15.1:0/1195095186 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d480035b0 con 0x7f9d5c06ea10 2024-03-18T18:50:54.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.625+0000 7f9d5ad9d700 1 -- 172.21.15.1:0/1195095186 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d5c06a750 msgr2=0x7f9d5c0ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:54.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.625+0000 7f9d5ad9d700 1 --2- 172.21.15.1:0/1195095186 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d5c06a750 0x7f9d5c0ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.625+0000 7f9d5ad9d700 1 -- 172.21.15.1:0/1195095186 >> v1:172.21.15.165:6789/0 conn(0x7f9d5c06ea10 legacy=0x7f9d5c06ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:54.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.625+0000 7f9d5ad9d700 1 -- 172.21.15.1:0/1195095186 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5c0cc2a0 con 0x7f9d5c0ace80 2024-03-18T18:50:54.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.625+0000 7f9d5a59c700 1 --2- 172.21.15.1:0/1195095186 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d5c06a750 0x7f9d5c0ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:50:54.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.626+0000 7f9d5ad9d700 1 --2- 172.21.15.1:0/1195095186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d5c0ace80 0x7f9d5c0ad310 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9d54009d90 tx=0x7f9d5400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc014a83c1848851 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:54.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.626+0000 7f9d5959a700 1 -- 172.21.15.1:0/1195095186 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5400e070 con 0x7f9d5c0ace80 2024-03-18T18:50:54.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.626+0000 7f9d5959a700 1 -- 172.21.15.1:0/1195095186 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5400bb20 con 0x7f9d5c0ace80 2024-03-18T18:50:54.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.626+0000 7f9d5959a700 1 -- 172.21.15.1:0/1195095186 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d54012430 con 0x7f9d5c0ace80 2024-03-18T18:50:54.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.626+0000 7f9d60b5c700 1 -- 172.21.15.1:0/1195095186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d5c0ace80 msgr2=0x7f9d5c0ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:54.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.626+0000 7f9d60b5c700 1 --2- 172.21.15.1:0/1195095186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d5c0ace80 0x7f9d5c0ad310 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9d54009d90 tx=0x7f9d5400b330 comp rx=0 tx=0).stop 2024-03-18T18:50:54.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.626+0000 7f9d60b5c700 1 -- 172.21.15.1:0/1195095186 shutdown_connections 2024-03-18T18:50:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.626+0000 7f9d60b5c700 1 --2- 172.21.15.1:0/1195095186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d5c0ace80 0x7f9d5c0ad310 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.626+0000 7f9d60b5c700 1 --2- 172.21.15.1:0/1195095186 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d5c06a750 0x7f9d5c0ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.626+0000 7f9d60b5c700 1 -- 172.21.15.1:0/1195095186 >> 172.21.15.1:0/1195095186 conn(0x7f9d5c0cc990 msgr2=0x7f9d5c0ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.626+0000 7f9d60b5c700 1 -- 172.21.15.1:0/1195095186 shutdown_connections 2024-03-18T18:50:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.627+0000 7f9d60b5c700 1 -- 172.21.15.1:0/1195095186 wait complete. 2024-03-18T18:50:54.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.628+0000 7f9d60b5c700 1 Processor -- start 2024-03-18T18:50:54.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.628+0000 7f9d60b5c700 1 -- start start 2024-03-18T18:50:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d60b5c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d5c06a750 0x7f9d5c0cf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d60b5c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d5c06ea10 0x7f9d5c157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d60b5c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d5c0ace80 0x7f9d5c14bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d60b5c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c0ce770 con 0x7f9d5c06ea10 2024-03-18T18:50:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d60b5c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c0c6850 con 0x7f9d5c0ace80 2024-03-18T18:50:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d60b5c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c0cea10 con 0x7f9d5c06a750 2024-03-18T18:50:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d59d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d5c06ea10 0x7f9d5c157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d59d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d5c06ea10 0x7f9d5c157920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38404/0 (socket says 172.21.15.1:38404) 2024-03-18T18:50:54.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d59d9b700 1 -- 172.21.15.1:0/533421999 learned_addr learned my addr 172.21.15.1:0/533421999 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d5ad9d700 1 --2- 172.21.15.1:0/533421999 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d5c0ace80 0x7f9d5c14bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d5a59c700 1 --2- 172.21.15.1:0/533421999 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d5c06a750 0x7f9d5c0cf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d59d9b700 1 -- 172.21.15.1:0/533421999 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d5c06a750 msgr2=0x7f9d5c0cf4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d59d9b700 1 --2- 172.21.15.1:0/533421999 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d5c06a750 0x7f9d5c0cf4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d59d9b700 1 -- 172.21.15.1:0/533421999 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d5c0ace80 msgr2=0x7f9d5c14bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d59d9b700 1 --2- 172.21.15.1:0/533421999 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d5c0ace80 0x7f9d5c14bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d59d9b700 1 -- 172.21.15.1:0/533421999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d440094a0 con 0x7f9d5c06ea10 2024-03-18T18:50:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d59d9b700 1 --2- 172.21.15.1:0/533421999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d5c06ea10 0x7f9d5c157920 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c00c700 tx=0x7f9d4c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d52ffd700 1 -- 172.21.15.1:0/533421999 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c005d70 con 0x7f9d5c06ea10 2024-03-18T18:50:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d52ffd700 1 -- 172.21.15.1:0/533421999 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d4c0070d0 con 0x7f9d5c06ea10 2024-03-18T18:50:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d60b5c700 1 -- 172.21.15.1:0/533421999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d54009a70 con 0x7f9d5c06ea10 2024-03-18T18:50:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d52ffd700 1 -- 172.21.15.1:0/533421999 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c007460 con 0x7f9d5c06ea10 2024-03-18T18:50:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.629+0000 7f9d60b5c700 1 -- 172.21.15.1:0/533421999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d5c0cefd0 con 0x7f9d5c06ea10 2024-03-18T18:50:54.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.630+0000 7f9d60b5c700 1 -- 172.21.15.1:0/533421999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d5c0676a0 con 0x7f9d5c06ea10 2024-03-18T18:50:54.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.631+0000 7f9d52ffd700 1 -- 172.21.15.1:0/533421999 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9d4c0075c0 con 0x7f9d5c06ea10 2024-03-18T18:50:54.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.631+0000 7f9d52ffd700 1 --2- 172.21.15.1:0/533421999 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9d3c06ac90 0x7f9d3c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:54.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.631+0000 7f9d52ffd700 1 -- 172.21.15.1:0/533421999 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f9d4c088b70 con 0x7f9d5c06ea10 2024-03-18T18:50:54.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.631+0000 7f9d5a59c700 1 --2- 172.21.15.1:0/533421999 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9d3c06ac90 0x7f9d3c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:54.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.632+0000 7f9d5a59c700 1 --2- 172.21.15.1:0/533421999 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9d3c06ac90 0x7f9d3c06d150 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9d44009d00 tx=0x7f9d44008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:54.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.636+0000 7f9d52ffd700 1 -- 172.21.15.1:0/533421999 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9d4c053a20 con 0x7f9d5c06ea10 2024-03-18T18:50:54.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.838+0000 7f9d60b5c700 1 -- 172.21.15.1:0/533421999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d5c067db0 con 0x7f9d5c06ea10 2024-03-18T18:50:54.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.840+0000 7f9d52ffd700 1 -- 172.21.15.1:0/533421999 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18217 (secure 0 0 0) 0x7f9d4c017090 con 0x7f9d5c06ea10 2024-03-18T18:50:54.837 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:54.837 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":27,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:50:34.993482+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.842+0000 7f9d60b5c700 1 -- 172.21.15.1:0/533421999 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9d3c06ac90 msgr2=0x7f9d3c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.842+0000 7f9d60b5c700 1 --2- 172.21.15.1:0/533421999 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9d3c06ac90 0x7f9d3c06d150 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9d44009d00 tx=0x7f9d44008040 comp rx=0 tx=0).stop 2024-03-18T18:50:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.842+0000 7f9d60b5c700 1 -- 172.21.15.1:0/533421999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d5c06ea10 msgr2=0x7f9d5c157920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.842+0000 7f9d60b5c700 1 --2- 172.21.15.1:0/533421999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d5c06ea10 0x7f9d5c157920 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c00c700 tx=0x7f9d4c00ca10 comp rx=0 tx=0).stop 2024-03-18T18:50:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.842+0000 7f9d60b5c700 1 -- 172.21.15.1:0/533421999 shutdown_connections 2024-03-18T18:50:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.842+0000 7f9d60b5c700 1 --2- 172.21.15.1:0/533421999 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9d3c06ac90 0x7f9d3c06d150 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.842+0000 7f9d60b5c700 1 --2- 172.21.15.1:0/533421999 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d5c06a750 0x7f9d5c0cf4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.842+0000 7f9d60b5c700 1 --2- 172.21.15.1:0/533421999 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d5c0ace80 0x7f9d5c14bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.842+0000 7f9d60b5c700 1 --2- 172.21.15.1:0/533421999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d5c06ea10 0x7f9d5c157920 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.842+0000 7f9d60b5c700 1 -- 172.21.15.1:0/533421999 >> 172.21.15.1:0/533421999 conn(0x7f9d5c0cc990 msgr2=0x7f9d5c0af940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:54.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.842+0000 7f9d60b5c700 1 -- 172.21.15.1:0/533421999 shutdown_connections 2024-03-18T18:50:54.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.842+0000 7f9d60b5c700 1 -- 172.21.15.1:0/533421999 wait complete. 2024-03-18T18:50:54.851 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T18:49:42.800801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T18:49:43.776684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T18:49:46.854608+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T18:50:54.852 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T18:50:54.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.983+0000 7fea45a6e700 1 Processor -- start 2024-03-18T18:50:54.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.983+0000 7fea45a6e700 1 -- start start 2024-03-18T18:50:54.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.984+0000 7fea45a6e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fea4015b6c0 0x7fea400b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:54.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.984+0000 7fea45a6e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea400cd430 0x7fea400c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:54.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.984+0000 7fea45a6e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea400d01f0 con 0x7fea400c8600 2024-03-18T18:50:54.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.984+0000 7fea45a6e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea400ccfa0 con 0x7fea4015b6c0 2024-03-18T18:50:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.984+0000 7fea45a6e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea400d1db0 con 0x7fea400cd430 2024-03-18T18:50:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.984+0000 7fea3f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fea4015b6c0 0x7fea400b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.984+0000 7fea3f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fea4015b6c0 0x7fea400b2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38410/0 (socket says 172.21.15.1:38410) 2024-03-18T18:50:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.984+0000 7fea3f7fe700 1 -- 172.21.15.1:0/1502798977 learned_addr learned my addr 172.21.15.1:0/1502798977 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.984+0000 7fea3effd700 1 --2- 172.21.15.1:0/1502798977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea400cd430 0x7fea400c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.985+0000 7fea3f7fe700 1 -- 172.21.15.1:0/1502798977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea400cd430 msgr2=0x7fea400c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.985+0000 7fea3f7fe700 1 --2- 172.21.15.1:0/1502798977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea400cd430 0x7fea400c80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.985+0000 7fea3f7fe700 1 -- 172.21.15.1:0/1502798977 >> v1:172.21.15.165:6789/0 conn(0x7fea400c8600 legacy=0x7fea400cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.985+0000 7fea3f7fe700 1 -- 172.21.15.1:0/1502798977 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea400cff10 con 0x7fea4015b6c0 2024-03-18T18:50:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.985+0000 7fea3f7fe700 1 --2- 172.21.15.1:0/1502798977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea4015b6c0 0x7fea400b2160 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fea28009920 tx=0x7fea28009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b62c48eb52205ea1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.985+0000 7fea3e7fc700 1 -- 172.21.15.1:0/1502798977 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2800ba20 con 0x7fea4015b6c0 2024-03-18T18:50:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.985+0000 7fea3e7fc700 1 -- 172.21.15.1:0/1502798977 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2800bb80 con 0x7fea4015b6c0 2024-03-18T18:50:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.985+0000 7fea3e7fc700 1 -- 172.21.15.1:0/1502798977 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28009e70 con 0x7fea4015b6c0 2024-03-18T18:50:54.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.985+0000 7fea45a6e700 1 -- 172.21.15.1:0/1502798977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea4015b6c0 msgr2=0x7fea400b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.985+0000 7fea45a6e700 1 --2- 172.21.15.1:0/1502798977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea4015b6c0 0x7fea400b2160 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fea28009920 tx=0x7fea28009c30 comp rx=0 tx=0).stop 2024-03-18T18:50:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.985+0000 7fea45a6e700 1 -- 172.21.15.1:0/1502798977 shutdown_connections 2024-03-18T18:50:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.985+0000 7fea45a6e700 1 --2- 172.21.15.1:0/1502798977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea4015b6c0 0x7fea400b2160 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.985+0000 7fea45a6e700 1 --2- 172.21.15.1:0/1502798977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea400cd430 0x7fea400c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.985+0000 7fea45a6e700 1 -- 172.21.15.1:0/1502798977 >> 172.21.15.1:0/1502798977 conn(0x7fea400d0ff0 msgr2=0x7fea400d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.986+0000 7fea45a6e700 1 -- 172.21.15.1:0/1502798977 shutdown_connections 2024-03-18T18:50:54.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.986+0000 7fea45a6e700 1 -- 172.21.15.1:0/1502798977 wait complete. 2024-03-18T18:50:54.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.987+0000 7fea45a6e700 1 Processor -- start 2024-03-18T18:50:54.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.987+0000 7fea45a6e700 1 -- start start 2024-03-18T18:50:54.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea45a6e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea400c8600 0x7fea40156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:54.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea45a6e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea400cd430 0x7fea400b1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:54.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea45a6e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea4015b6c0 0x7fea400c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:54.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea45a6e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea400b0f20 con 0x7fea400c8600 2024-03-18T18:50:54.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea45a6e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea40156280 con 0x7fea4015b6c0 2024-03-18T18:50:54.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea45a6e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea401578b0 con 0x7fea400cd430 2024-03-18T18:50:54.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea3f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea400c8600 0x7fea40156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea3f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea400c8600 0x7fea40156710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38420/0 (socket says 172.21.15.1:38420) 2024-03-18T18:50:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea3f7fe700 1 -- 172.21.15.1:0/3600942242 learned_addr learned my addr 172.21.15.1:0/3600942242 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea3ffff700 1 --2- 172.21.15.1:0/3600942242 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea4015b6c0 0x7fea400c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea3effd700 1 --2- 172.21.15.1:0/3600942242 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea400cd430 0x7fea400b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea3f7fe700 1 -- 172.21.15.1:0/3600942242 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea400cd430 msgr2=0x7fea400b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea3f7fe700 1 --2- 172.21.15.1:0/3600942242 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea400cd430 0x7fea400b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea3f7fe700 1 -- 172.21.15.1:0/3600942242 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea4015b6c0 msgr2=0x7fea400c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea3f7fe700 1 --2- 172.21.15.1:0/3600942242 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea4015b6c0 0x7fea400c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea3f7fe700 1 -- 172.21.15.1:0/3600942242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea28009550 con 0x7fea400c8600 2024-03-18T18:50:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea3effd700 1 --2- 172.21.15.1:0/3600942242 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea400cd430 0x7fea400b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea3f7fe700 1 --2- 172.21.15.1:0/3600942242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea400c8600 0x7fea40156710 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fea2800c010 tx=0x7fea28004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea37fff700 1 -- 172.21.15.1:0/3600942242 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28004400 con 0x7fea400c8600 2024-03-18T18:50:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea37fff700 1 -- 172.21.15.1:0/3600942242 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2801f070 con 0x7fea400c8600 2024-03-18T18:50:54.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea45a6e700 1 -- 172.21.15.1:0/3600942242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea401572f0 con 0x7fea400c8600 2024-03-18T18:50:54.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea37fff700 1 -- 172.21.15.1:0/3600942242 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea280038e0 con 0x7fea400c8600 2024-03-18T18:50:54.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.988+0000 7fea45a6e700 1 -- 172.21.15.1:0/3600942242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea40155f10 con 0x7fea400c8600 2024-03-18T18:50:54.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.989+0000 7fea45a6e700 1 -- 172.21.15.1:0/3600942242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea401560a0 con 0x7fea400c8600 2024-03-18T18:50:54.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.990+0000 7fea37fff700 1 -- 172.21.15.1:0/3600942242 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fea28021030 con 0x7fea400c8600 2024-03-18T18:50:54.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.990+0000 7fea37fff700 1 --2- 172.21.15.1:0/3600942242 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fea2006ac90 0x7fea2006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:54.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.990+0000 7fea3effd700 1 --2- 172.21.15.1:0/3600942242 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fea2006ac90 0x7fea2006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:54.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.990+0000 7fea37fff700 1 -- 172.21.15.1:0/3600942242 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fea2808aad0 con 0x7fea400c8600 2024-03-18T18:50:54.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.991+0000 7fea3effd700 1 --2- 172.21.15.1:0/3600942242 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fea2006ac90 0x7fea2006d150 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fea4015a5f0 tx=0x7fea30008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:54.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:54.995+0000 7fea37fff700 1 -- 172.21.15.1:0/3600942242 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fea28055aa0 con 0x7fea400c8600 2024-03-18T18:50:55.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.198+0000 7fea45a6e700 1 -- 172.21.15.1:0/3600942242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fea40156560 con 0x7fea400c8600 2024-03-18T18:50:55.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.198+0000 7fea37fff700 1 -- 172.21.15.1:0/3600942242 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fea28015020 con 0x7fea400c8600 2024-03-18T18:50:55.195 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T18:50:55.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.200+0000 7fea45a6e700 1 -- 172.21.15.1:0/3600942242 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fea2006ac90 msgr2=0x7fea2006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:55.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.200+0000 7fea45a6e700 1 --2- 172.21.15.1:0/3600942242 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fea2006ac90 0x7fea2006d150 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fea4015a5f0 tx=0x7fea30008040 comp rx=0 tx=0).stop 2024-03-18T18:50:55.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.200+0000 7fea45a6e700 1 -- 172.21.15.1:0/3600942242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea400c8600 msgr2=0x7fea40156710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:55.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.200+0000 7fea45a6e700 1 --2- 172.21.15.1:0/3600942242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea400c8600 0x7fea40156710 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fea2800c010 tx=0x7fea28004750 comp rx=0 tx=0).stop 2024-03-18T18:50:55.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.200+0000 7fea45a6e700 1 -- 172.21.15.1:0/3600942242 shutdown_connections 2024-03-18T18:50:55.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.200+0000 7fea45a6e700 1 --2- 172.21.15.1:0/3600942242 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fea2006ac90 0x7fea2006d150 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.200+0000 7fea45a6e700 1 --2- 172.21.15.1:0/3600942242 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea4015b6c0 0x7fea400c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.200+0000 7fea45a6e700 1 --2- 172.21.15.1:0/3600942242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea400c8600 0x7fea40156710 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.200+0000 7fea45a6e700 1 --2- 172.21.15.1:0/3600942242 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea400cd430 0x7fea400b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.200+0000 7fea45a6e700 1 -- 172.21.15.1:0/3600942242 >> 172.21.15.1:0/3600942242 conn(0x7fea400d0ff0 msgr2=0x7fea400cebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:55.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.201+0000 7fea45a6e700 1 -- 172.21.15.1:0/3600942242 shutdown_connections 2024-03-18T18:50:55.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.201+0000 7fea45a6e700 1 -- 172.21.15.1:0/3600942242 wait complete. 2024-03-18T18:50:55.209 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T18:50:55.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.340+0000 7f3e083ad700 1 Processor -- start 2024-03-18T18:50:55.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.340+0000 7f3e083ad700 1 -- start start 2024-03-18T18:50:55.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.341+0000 7f3e083ad700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e0015eba0 0x7f3e000cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:55.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.341+0000 7f3e083ad700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3e000b0500 0x7f3e000cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:55.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.341+0000 7f3e083ad700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3e000cfdf0 0x7f3e00099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:55.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.341+0000 7f3e083ad700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e0015ef80 con 0x7f3e000cfdf0 2024-03-18T18:50:55.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.341+0000 7f3e083ad700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e000cc9a0 con 0x7f3e000b0500 2024-03-18T18:50:55.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.341+0000 7f3e083ad700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e000ccc80 con 0x7f3e0015eba0 2024-03-18T18:50:55.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.342+0000 7f3e0694a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3e000cfdf0 0x7f3e00099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:55.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.342+0000 7f3e0694a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3e000cfdf0 0x7f3e00099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38436/0 (socket says 172.21.15.1:38436) 2024-03-18T18:50:55.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.342+0000 7f3e0694a700 1 -- 172.21.15.1:0/884422756 learned_addr learned my addr 172.21.15.1:0/884422756 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:55.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.342+0000 7f3e05948700 1 --2- 172.21.15.1:0/884422756 >> v2:172.21.15.165:3300/0 conn(0x7f3e000b0500 0x7f3e000cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:55.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.342+0000 7f3e0694a700 1 -- 172.21.15.1:0/884422756 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e0015eba0 msgr2=0x7f3e000cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:55.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.342+0000 7f3e0694a700 1 --2- 172.21.15.1:0/884422756 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e0015eba0 0x7f3e000cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.342+0000 7f3e0694a700 1 -- 172.21.15.1:0/884422756 >> v2:172.21.15.165:3300/0 conn(0x7f3e000b0500 msgr2=0x7f3e000cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:55.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.342+0000 7f3e0694a700 1 --2- 172.21.15.1:0/884422756 >> v2:172.21.15.165:3300/0 conn(0x7f3e000b0500 0x7f3e000cf8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.342+0000 7f3e0694a700 1 -- 172.21.15.1:0/884422756 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e000ccf60 con 0x7f3e000cfdf0 2024-03-18T18:50:55.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.342+0000 7f3e0694a700 1 --2- 172.21.15.1:0/884422756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e000cfdf0 0x7f3e00099a40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3e000722f0 tx=0x7f3df800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b6f28f9101b09d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:55.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.342+0000 7f3e05147700 1 -- 172.21.15.1:0/884422756 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df800e070 con 0x7f3e000cfdf0 2024-03-18T18:50:55.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.342+0000 7f3e05147700 1 -- 172.21.15.1:0/884422756 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df800b9c0 con 0x7f3e000cfdf0 2024-03-18T18:50:55.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.342+0000 7f3e05147700 1 -- 172.21.15.1:0/884422756 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df800bd70 con 0x7f3e000cfdf0 2024-03-18T18:50:55.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.343+0000 7f3e083ad700 1 -- 172.21.15.1:0/884422756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e000cfdf0 msgr2=0x7f3e00099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:55.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.343+0000 7f3e083ad700 1 --2- 172.21.15.1:0/884422756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e000cfdf0 0x7f3e00099a40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3e000722f0 tx=0x7f3df800b330 comp rx=0 tx=0).stop 2024-03-18T18:50:55.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.343+0000 7f3e083ad700 1 -- 172.21.15.1:0/884422756 shutdown_connections 2024-03-18T18:50:55.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.343+0000 7f3e083ad700 1 --2- 172.21.15.1:0/884422756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e000cfdf0 0x7f3e00099a40 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.343+0000 7f3e083ad700 1 --2- 172.21.15.1:0/884422756 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e0015eba0 0x7f3e000cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.343+0000 7f3e083ad700 1 --2- 172.21.15.1:0/884422756 >> v2:172.21.15.165:3300/0 conn(0x7f3e000b0500 0x7f3e000cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.343+0000 7f3e083ad700 1 -- 172.21.15.1:0/884422756 >> 172.21.15.1:0/884422756 conn(0x7f3e000b2bd0 msgr2=0x7f3e000b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:55.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.343+0000 7f3e083ad700 1 -- 172.21.15.1:0/884422756 shutdown_connections 2024-03-18T18:50:55.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.343+0000 7f3e083ad700 1 -- 172.21.15.1:0/884422756 wait complete. 2024-03-18T18:50:55.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.344+0000 7f3e083ad700 1 Processor -- start 2024-03-18T18:50:55.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.344+0000 7f3e083ad700 1 -- start start 2024-03-18T18:50:55.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.345+0000 7f3e083ad700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e000b0500 0x7f3e000b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:55.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.345+0000 7f3e083ad700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e000cfdf0 0x7f3e000c6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:55.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.345+0000 7f3e083ad700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e0015eba0 0x7f3e000c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:55.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.345+0000 7f3e083ad700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e000c9220 con 0x7f3e0015eba0 2024-03-18T18:50:55.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.345+0000 7f3e083ad700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e000c71e0 con 0x7f3e000cfdf0 2024-03-18T18:50:55.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.345+0000 7f3e083ad700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e000c8610 con 0x7f3e000b0500 2024-03-18T18:50:55.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.345+0000 7f3e0694a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e0015eba0 0x7f3e000c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:55.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.345+0000 7f3e0694a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e0015eba0 0x7f3e000c6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38448/0 (socket says 172.21.15.1:38448) 2024-03-18T18:50:55.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.345+0000 7f3e0694a700 1 -- 172.21.15.1:0/4283003662 learned_addr learned my addr 172.21.15.1:0/4283003662 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:55.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.345+0000 7f3e05948700 1 --2- 172.21.15.1:0/4283003662 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e000cfdf0 0x7f3e000c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:55.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.345+0000 7f3e06149700 1 --2- 172.21.15.1:0/4283003662 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e000b0500 0x7f3e000b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:55.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.345+0000 7f3e0694a700 1 -- 172.21.15.1:0/4283003662 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e000b0500 msgr2=0x7f3e000b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:55.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.345+0000 7f3e0694a700 1 --2- 172.21.15.1:0/4283003662 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e000b0500 0x7f3e000b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.345+0000 7f3e0694a700 1 -- 172.21.15.1:0/4283003662 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e000cfdf0 msgr2=0x7f3e000c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:55.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.345+0000 7f3e0694a700 1 --2- 172.21.15.1:0/4283003662 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e000cfdf0 0x7f3e000c6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.345+0000 7f3e0694a700 1 -- 172.21.15.1:0/4283003662 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df8009b30 con 0x7f3e0015eba0 2024-03-18T18:50:55.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.345+0000 7f3e05948700 1 --2- 172.21.15.1:0/4283003662 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e000cfdf0 0x7f3e000c6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:55.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.345+0000 7f3e0694a700 1 --2- 172.21.15.1:0/4283003662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e0015eba0 0x7f3e000c6ca0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3e00072150 tx=0x7f3df8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:55.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.346+0000 7f3df6ffd700 1 -- 172.21.15.1:0/4283003662 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df8004250 con 0x7f3e0015eba0 2024-03-18T18:50:55.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.346+0000 7f3df6ffd700 1 -- 172.21.15.1:0/4283003662 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df80043b0 con 0x7f3e0015eba0 2024-03-18T18:50:55.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.346+0000 7f3e083ad700 1 -- 172.21.15.1:0/4283003662 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e000c88f0 con 0x7f3e0015eba0 2024-03-18T18:50:55.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.346+0000 7f3df6ffd700 1 -- 172.21.15.1:0/4283003662 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df8004740 con 0x7f3e0015eba0 2024-03-18T18:50:55.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.346+0000 7f3e083ad700 1 -- 172.21.15.1:0/4283003662 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e000c8eb0 con 0x7f3e0015eba0 2024-03-18T18:50:55.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.347+0000 7f3e083ad700 1 -- 172.21.15.1:0/4283003662 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e000c93d0 con 0x7f3e0015eba0 2024-03-18T18:50:55.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.347+0000 7f3df6ffd700 1 -- 172.21.15.1:0/4283003662 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3df801c440 con 0x7f3e0015eba0 2024-03-18T18:50:55.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.347+0000 7f3df6ffd700 1 --2- 172.21.15.1:0/4283003662 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3de406ac90 0x7f3de406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:55.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.348+0000 7f3df6ffd700 1 -- 172.21.15.1:0/4283003662 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3df808aca0 con 0x7f3e0015eba0 2024-03-18T18:50:55.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.348+0000 7f3e06149700 1 --2- 172.21.15.1:0/4283003662 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3de406ac90 0x7f3de406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:55.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.348+0000 7f3e06149700 1 --2- 172.21.15.1:0/4283003662 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3de406ac90 0x7f3de406d150 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3df0006fd0 tx=0x7f3df0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:55.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.352+0000 7f3df6ffd700 1 -- 172.21.15.1:0/4283003662 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3e000c93d0 con 0x7f3e0015eba0 2024-03-18T18:50:55.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.552+0000 7f3e083ad700 1 -- 172.21.15.1:0/4283003662 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3e000c93d0 con 0x7f3e0015eba0 2024-03-18T18:50:55.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.553+0000 7f3df6ffd700 1 -- 172.21.15.1:0/4283003662 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3df802a020 con 0x7f3e0015eba0 2024-03-18T18:50:55.549 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:55.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.555+0000 7f3e083ad700 1 -- 172.21.15.1:0/4283003662 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3de406ac90 msgr2=0x7f3de406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:55.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.555+0000 7f3e083ad700 1 --2- 172.21.15.1:0/4283003662 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3de406ac90 0x7f3de406d150 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3df0006fd0 tx=0x7f3df0008040 comp rx=0 tx=0).stop 2024-03-18T18:50:55.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.555+0000 7f3e083ad700 1 -- 172.21.15.1:0/4283003662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e0015eba0 msgr2=0x7f3e000c6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:55.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.555+0000 7f3e083ad700 1 --2- 172.21.15.1:0/4283003662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e0015eba0 0x7f3e000c6ca0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3e00072150 tx=0x7f3df8004ad0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.555+0000 7f3e083ad700 1 -- 172.21.15.1:0/4283003662 shutdown_connections 2024-03-18T18:50:55.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.555+0000 7f3e083ad700 1 --2- 172.21.15.1:0/4283003662 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3de406ac90 0x7f3de406d150 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.555+0000 7f3e083ad700 1 --2- 172.21.15.1:0/4283003662 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e000b0500 0x7f3e000b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.555+0000 7f3e083ad700 1 --2- 172.21.15.1:0/4283003662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e0015eba0 0x7f3e000c6ca0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.555+0000 7f3e083ad700 1 --2- 172.21.15.1:0/4283003662 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e000cfdf0 0x7f3e000c6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.555+0000 7f3e083ad700 1 -- 172.21.15.1:0/4283003662 >> 172.21.15.1:0/4283003662 conn(0x7f3e000b2bd0 msgr2=0x7f3e0015d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:55.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.555+0000 7f3e083ad700 1 -- 172.21.15.1:0/4283003662 shutdown_connections 2024-03-18T18:50:55.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.555+0000 7f3e083ad700 1 -- 172.21.15.1:0/4283003662 wait complete. 2024-03-18T18:50:55.564 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T18:50:55.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.695+0000 7f762619f700 1 Processor -- start 2024-03-18T18:50:55.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.695+0000 7f762619f700 1 -- start start 2024-03-18T18:50:55.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.696+0000 7f762619f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76200724b0 0x7f76200cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:55.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.696+0000 7f762619f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f762006da90 con 0x7f762006efb0 2024-03-18T18:50:55.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.696+0000 7f762619f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f762006ab30 con 0x7f762015d580 2024-03-18T18:50:55.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.696+0000 7f762619f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762015d960 con 0x7f76200724b0 2024-03-18T18:50:55.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.696+0000 7f761effd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f762006efb0 0x7f7620071da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32982/0 (socket says 172.21.15.1:32982) 2024-03-18T18:50:55.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.696+0000 7f761effd700 1 -- 172.21.15.1:0/3722706103 learned_addr learned my addr 172.21.15.1:0/3722706103 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:55.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.696+0000 7f761ffff700 1 --2- 172.21.15.1:0/3722706103 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76200724b0 0x7f76200cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:55.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.696+0000 7f761e7fc700 1 -- 172.21.15.1:0/3722706103 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316977509 0 0) 0x7f762006da90 con 0x7f762006efb0 2024-03-18T18:50:55.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.696+0000 7f761e7fc700 1 -- 172.21.15.1:0/3722706103 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f760c0035b0 con 0x7f762006efb0 2024-03-18T18:50:55.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761e7fc700 1 -- 172.21.15.1:0/3722706103 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 232087322 0 0) 0x7f760c0035b0 con 0x7f762006efb0 2024-03-18T18:50:55.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761e7fc700 1 -- 172.21.15.1:0/3722706103 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76200724b0 msgr2=0x7f76200cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:55.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761e7fc700 1 --2- 172.21.15.1:0/3722706103 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76200724b0 0x7f76200cf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761e7fc700 1 -- 172.21.15.1:0/3722706103 >> v1:172.21.15.165:6789/0 conn(0x7f762015d580 legacy=0x7f762006d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761e7fc700 1 -- 172.21.15.1:0/3722706103 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762006ae10 con 0x7f762006efb0 2024-03-18T18:50:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761e7fc700 1 -- 172.21.15.1:0/3722706103 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f7610001670 con 0x7f762006efb0 2024-03-18T18:50:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761e7fc700 1 -- 172.21.15.1:0/3722706103 >> v1:172.21.15.1:6789/0 conn(0x7f762006efb0 legacy=0x7f7620071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761e7fc700 1 --2- 172.21.15.1:0/3722706103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f760c003a30 0x7f760c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761e7fc700 1 --2- 172.21.15.1:0/3722706103 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f760c0043e0 0x7f760c0048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:55.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761e7fc700 1 --2- 172.21.15.1:0/3722706103 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76200724b0 0x7f760c005110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:55.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761e7fc700 1 -- 172.21.15.1:0/3722706103 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762015d960 con 0x7f760c003a30 2024-03-18T18:50:55.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761e7fc700 1 -- 172.21.15.1:0/3722706103 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c00ba40 con 0x7f760c0043e0 2024-03-18T18:50:55.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761e7fc700 1 -- 172.21.15.1:0/3722706103 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f760c00bc00 con 0x7f76200724b0 2024-03-18T18:50:55.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761f7fe700 1 --2- 172.21.15.1:0/3722706103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f760c003a30 0x7f760c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:55.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761effd700 1 --2- 172.21.15.1:0/3722706103 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f760c0043e0 0x7f760c0048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:55.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761ffff700 1 --2- 172.21.15.1:0/3722706103 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76200724b0 0x7f760c005110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:55.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761f7fe700 1 -- 172.21.15.1:0/3722706103 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76200724b0 msgr2=0x7f760c005110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:55.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761f7fe700 1 --2- 172.21.15.1:0/3722706103 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76200724b0 0x7f760c005110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761f7fe700 1 -- 172.21.15.1:0/3722706103 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f760c0043e0 msgr2=0x7f760c0048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:55.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761f7fe700 1 --2- 172.21.15.1:0/3722706103 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f760c0043e0 0x7f760c0048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761f7fe700 1 -- 172.21.15.1:0/3722706103 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f760c0035b0 con 0x7f760c003a30 2024-03-18T18:50:55.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761ffff700 1 --2- 172.21.15.1:0/3722706103 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76200724b0 0x7f760c005110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:55.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761f7fe700 1 --2- 172.21.15.1:0/3722706103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f760c003a30 0x7f760c003e10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f76080018d0 tx=0x7f760c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d09187f0c367ea server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:55.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761e7fc700 1 -- 172.21.15.1:0/3722706103 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760800c690 con 0x7f760c003a30 2024-03-18T18:50:55.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761e7fc700 1 -- 172.21.15.1:0/3722706103 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f760800c810 con 0x7f760c003a30 2024-03-18T18:50:55.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.697+0000 7f761e7fc700 1 -- 172.21.15.1:0/3722706103 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760800cbc0 con 0x7f760c003a30 2024-03-18T18:50:55.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.698+0000 7f762619f700 1 -- 172.21.15.1:0/3722706103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f760c003a30 msgr2=0x7f760c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:55.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.698+0000 7f762619f700 1 --2- 172.21.15.1:0/3722706103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f760c003a30 0x7f760c003e10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f76080018d0 tx=0x7f760c005880 comp rx=0 tx=0).stop 2024-03-18T18:50:55.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.698+0000 7f761ffff700 1 -- 172.21.15.1:0/3722706103 reap_dead start 2024-03-18T18:50:55.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.698+0000 7f762619f700 1 -- 172.21.15.1:0/3722706103 shutdown_connections 2024-03-18T18:50:55.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.698+0000 7f762619f700 1 -- 172.21.15.1:0/3722706103 >> 172.21.15.1:0/3722706103 conn(0x7f76200cca00 msgr2=0x7f76200cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:55.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.698+0000 7f762619f700 1 -- 172.21.15.1:0/3722706103 shutdown_connections 2024-03-18T18:50:55.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.698+0000 7f762619f700 1 -- 172.21.15.1:0/3722706103 wait complete. 2024-03-18T18:50:55.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.699+0000 7f762619f700 1 Processor -- start 2024-03-18T18:50:55.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.699+0000 7f762619f700 1 -- start start 2024-03-18T18:50:55.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.700+0000 7f762619f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f760c003a30 0x7f76201568a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:55.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.700+0000 7f762619f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76201570c0 0x7f762015a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:55.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.700+0000 7f762619f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f762015aa70 0x7f76200ad840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:55.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.700+0000 7f762619f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7620156de0 con 0x7f762015aa70 2024-03-18T18:50:55.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.700+0000 7f762619f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762015aff0 con 0x7f760c003a30 2024-03-18T18:50:55.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.700+0000 7f762619f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762015ecc0 con 0x7f76201570c0 2024-03-18T18:50:55.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.700+0000 7f761ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f762015aa70 0x7f76200ad840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:55.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.700+0000 7f761ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f762015aa70 0x7f76200ad840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38458/0 (socket says 172.21.15.1:38458) 2024-03-18T18:50:55.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.700+0000 7f761ffff700 1 -- 172.21.15.1:0/1752898649 learned_addr learned my addr 172.21.15.1:0/1752898649 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:55.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.700+0000 7f761effd700 1 --2- 172.21.15.1:0/1752898649 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76201570c0 0x7f762015a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.700+0000 7f761f7fe700 1 --2- 172.21.15.1:0/1752898649 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f760c003a30 0x7f76201568a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.700+0000 7f761ffff700 1 -- 172.21.15.1:0/1752898649 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76201570c0 msgr2=0x7f762015a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.700+0000 7f761ffff700 1 --2- 172.21.15.1:0/1752898649 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76201570c0 0x7f762015a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.700+0000 7f761ffff700 1 -- 172.21.15.1:0/1752898649 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f760c003a30 msgr2=0x7f76201568a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.700+0000 7f761ffff700 1 --2- 172.21.15.1:0/1752898649 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f760c003a30 0x7f76201568a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.700+0000 7f761ffff700 1 -- 172.21.15.1:0/1752898649 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762015efa0 con 0x7f762015aa70 2024-03-18T18:50:55.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.701+0000 7f761ffff700 1 --2- 172.21.15.1:0/1752898649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f762015aa70 0x7f76200ad840 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f762006f690 tx=0x7f76180104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:55.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.701+0000 7f7617fff700 1 -- 172.21.15.1:0/1752898649 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761801b070 con 0x7f762015aa70 2024-03-18T18:50:55.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.701+0000 7f7617fff700 1 -- 172.21.15.1:0/1752898649 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761800f040 con 0x7f762015aa70 2024-03-18T18:50:55.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.701+0000 7f762619f700 1 -- 172.21.15.1:0/1752898649 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762015f280 con 0x7f762015aa70 2024-03-18T18:50:55.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.701+0000 7f7617fff700 1 -- 172.21.15.1:0/1752898649 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761800e050 con 0x7f762015aa70 2024-03-18T18:50:55.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.701+0000 7f762619f700 1 -- 172.21.15.1:0/1752898649 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76200adf60 con 0x7f762015aa70 2024-03-18T18:50:55.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.702+0000 7f762619f700 1 -- 172.21.15.1:0/1752898649 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76200676a0 con 0x7f762015aa70 2024-03-18T18:50:55.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.702+0000 7f7617fff700 1 -- 172.21.15.1:0/1752898649 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7618015440 con 0x7f762015aa70 2024-03-18T18:50:55.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.703+0000 7f7617fff700 1 --2- 172.21.15.1:0/1752898649 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f760006ac90 0x7f760006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:55.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.703+0000 7f7617fff700 1 -- 172.21.15.1:0/1752898649 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7618089600 con 0x7f762015aa70 2024-03-18T18:50:55.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.703+0000 7f761f7fe700 1 --2- 172.21.15.1:0/1752898649 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f760006ac90 0x7f760006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:55.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.703+0000 7f761f7fe700 1 --2- 172.21.15.1:0/1752898649 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f760006ac90 0x7f760006d150 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7608014040 tx=0x7f7608005040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:55.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.707+0000 7f7617fff700 1 -- 172.21.15.1:0/1752898649 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7618054420 con 0x7f762015aa70 2024-03-18T18:50:55.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.907+0000 7f762619f700 1 -- 172.21.15.1:0/1752898649 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f76200ceb80 con 0x7f762015aa70 2024-03-18T18:50:55.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.908+0000 7f7617fff700 1 -- 172.21.15.1:0/1752898649 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7618057a40 con 0x7f762015aa70 2024-03-18T18:50:55.907 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T18:50:55.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.910+0000 7f762619f700 1 -- 172.21.15.1:0/1752898649 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f760006ac90 msgr2=0x7f760006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:55.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.910+0000 7f762619f700 1 --2- 172.21.15.1:0/1752898649 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f760006ac90 0x7f760006d150 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7608014040 tx=0x7f7608005040 comp rx=0 tx=0).stop 2024-03-18T18:50:55.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.910+0000 7f762619f700 1 -- 172.21.15.1:0/1752898649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f762015aa70 msgr2=0x7f76200ad840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:55.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.910+0000 7f762619f700 1 --2- 172.21.15.1:0/1752898649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f762015aa70 0x7f76200ad840 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f762006f690 tx=0x7f76180104a0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.910+0000 7f762619f700 1 -- 172.21.15.1:0/1752898649 shutdown_connections 2024-03-18T18:50:55.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.910+0000 7f762619f700 1 --2- 172.21.15.1:0/1752898649 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f760006ac90 0x7f760006d150 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.910+0000 7f762619f700 1 --2- 172.21.15.1:0/1752898649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f762015aa70 0x7f76200ad840 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.910+0000 7f762619f700 1 --2- 172.21.15.1:0/1752898649 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f760c003a30 0x7f76201568a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.910+0000 7f762619f700 1 --2- 172.21.15.1:0/1752898649 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76201570c0 0x7f762015a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:55.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.910+0000 7f762619f700 1 -- 172.21.15.1:0/1752898649 >> 172.21.15.1:0/1752898649 conn(0x7f76200cca00 msgr2=0x7f7620071ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:55.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.910+0000 7f762619f700 1 -- 172.21.15.1:0/1752898649 shutdown_connections 2024-03-18T18:50:55.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:55.910+0000 7f762619f700 1 -- 172.21.15.1:0/1752898649 wait complete. 2024-03-18T18:50:55.919 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T18:50:56.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.050+0000 7fee7ec3e700 1 Processor -- start 2024-03-18T18:50:56.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.050+0000 7fee7ec3e700 1 -- start start 2024-03-18T18:50:56.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.051+0000 7fee7ec3e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee780cd430 0x7fee780b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:56.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.051+0000 7fee7ec3e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee780d01f0 con 0x7fee7815b6c0 2024-03-18T18:50:56.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.051+0000 7fee7ec3e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee780ccfa0 con 0x7fee780c87d0 2024-03-18T18:50:56.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.051+0000 7fee7ec3e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee780d1db0 con 0x7fee780cd430 2024-03-18T18:50:56.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.051+0000 7fee7c9da700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fee7815b6c0 0x7fee780c80c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32990/0 (socket says 172.21.15.1:32990) 2024-03-18T18:50:56.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.051+0000 7fee7c9da700 1 -- 172.21.15.1:0/4185561833 learned_addr learned my addr 172.21.15.1:0/4185561833 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:56.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.051+0000 7fee77fff700 1 --2- 172.21.15.1:0/4185561833 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee780cd430 0x7fee780b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:56.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.051+0000 7fee777fe700 1 -- 172.21.15.1:0/4185561833 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473982890 0 0) 0x7fee780d01f0 con 0x7fee7815b6c0 2024-03-18T18:50:56.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.051+0000 7fee777fe700 1 -- 172.21.15.1:0/4185561833 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee680035b0 con 0x7fee7815b6c0 2024-03-18T18:50:56.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.051+0000 7fee777fe700 1 -- 172.21.15.1:0/4185561833 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130684789 0 0) 0x7fee780ccfa0 con 0x7fee780c87d0 2024-03-18T18:50:56.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.051+0000 7fee777fe700 1 -- 172.21.15.1:0/4185561833 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee780d01f0 con 0x7fee780c87d0 2024-03-18T18:50:56.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.051+0000 7fee77fff700 1 -- 172.21.15.1:0/4185561833 >> v1:172.21.15.165:6789/0 conn(0x7fee780c87d0 legacy=0x7fee780cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:56.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.051+0000 7fee77fff700 1 -- 172.21.15.1:0/4185561833 >> v1:172.21.15.1:6789/0 conn(0x7fee7815b6c0 legacy=0x7fee780c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:56.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.051+0000 7fee77fff700 1 -- 172.21.15.1:0/4185561833 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee780cff10 con 0x7fee780cd430 2024-03-18T18:50:56.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.052+0000 7fee77fff700 1 --2- 172.21.15.1:0/4185561833 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee780cd430 0x7fee780b2160 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fee640098a0 tx=0x7fee64009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26682fe56aa0f389 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:56.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.052+0000 7fee777fe700 1 -- 172.21.15.1:0/4185561833 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee64011830 con 0x7fee780cd430 2024-03-18T18:50:56.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.052+0000 7fee777fe700 1 -- 172.21.15.1:0/4185561833 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee64011990 con 0x7fee780cd430 2024-03-18T18:50:56.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.052+0000 7fee777fe700 1 -- 172.21.15.1:0/4185561833 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee64011d00 con 0x7fee780cd430 2024-03-18T18:50:56.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.052+0000 7fee7ec3e700 1 -- 172.21.15.1:0/4185561833 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee780cd430 msgr2=0x7fee780b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:56.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.052+0000 7fee7ec3e700 1 --2- 172.21.15.1:0/4185561833 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee780cd430 0x7fee780b2160 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fee640098a0 tx=0x7fee64009bb0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.052+0000 7fee7ec3e700 1 -- 172.21.15.1:0/4185561833 shutdown_connections 2024-03-18T18:50:56.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.052+0000 7fee7ec3e700 1 --2- 172.21.15.1:0/4185561833 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee780cd430 0x7fee780b2160 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.052+0000 7fee7ec3e700 1 -- 172.21.15.1:0/4185561833 >> 172.21.15.1:0/4185561833 conn(0x7fee780d0ff0 msgr2=0x7fee780d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:56.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.052+0000 7fee7ec3e700 1 -- 172.21.15.1:0/4185561833 shutdown_connections 2024-03-18T18:50:56.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.052+0000 7fee7ec3e700 1 -- 172.21.15.1:0/4185561833 wait complete. 2024-03-18T18:50:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.053+0000 7fee7ec3e700 1 Processor -- start 2024-03-18T18:50:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.054+0000 7fee7ec3e700 1 -- start start 2024-03-18T18:50:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.054+0000 7fee7ec3e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee780c87d0 0x7fee78149da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.054+0000 7fee7ec3e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee780cd430 0x7fee780c7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.054+0000 7fee7ec3e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee7815b6c0 0x7fee780c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:56.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.054+0000 7fee7ec3e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee780b1b90 con 0x7fee780c87d0 2024-03-18T18:50:56.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.054+0000 7fee7ec3e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7814a440 con 0x7fee780cd430 2024-03-18T18:50:56.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.054+0000 7fee7ec3e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee7806d080 con 0x7fee7815b6c0 2024-03-18T18:50:56.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.054+0000 7fee7c9da700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee780c87d0 0x7fee78149da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:56.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.055+0000 7fee7c9da700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee780c87d0 0x7fee78149da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38470/0 (socket says 172.21.15.1:38470) 2024-03-18T18:50:56.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.055+0000 7fee7c9da700 1 -- 172.21.15.1:0/743794451 learned_addr learned my addr 172.21.15.1:0/743794451 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:56.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.055+0000 7fee77fff700 1 --2- 172.21.15.1:0/743794451 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee780cd430 0x7fee780c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:56.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.055+0000 7fee7c9da700 1 -- 172.21.15.1:0/743794451 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee7815b6c0 msgr2=0x7fee780c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:56.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.055+0000 7fee7d1db700 1 --2- 172.21.15.1:0/743794451 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee7815b6c0 0x7fee780c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:56.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.055+0000 7fee7c9da700 1 --2- 172.21.15.1:0/743794451 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee7815b6c0 0x7fee780c7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.055+0000 7fee7c9da700 1 -- 172.21.15.1:0/743794451 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee780cd430 msgr2=0x7fee780c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:56.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.055+0000 7fee7c9da700 1 --2- 172.21.15.1:0/743794451 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee780cd430 0x7fee780c7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.055+0000 7fee7c9da700 1 -- 172.21.15.1:0/743794451 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee640094a0 con 0x7fee780c87d0 2024-03-18T18:50:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.055+0000 7fee77fff700 1 --2- 172.21.15.1:0/743794451 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee780cd430 0x7fee780c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.055+0000 7fee7c9da700 1 --2- 172.21.15.1:0/743794451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee780c87d0 0x7fee78149da0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fee6c00c700 tx=0x7fee6c00cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.055+0000 7fee757fa700 1 -- 172.21.15.1:0/743794451 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6c005d70 con 0x7fee780c87d0 2024-03-18T18:50:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.055+0000 7fee757fa700 1 -- 172.21.15.1:0/743794451 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6c0070d0 con 0x7fee780c87d0 2024-03-18T18:50:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.055+0000 7fee7ec3e700 1 -- 172.21.15.1:0/743794451 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee7814a6e0 con 0x7fee780c87d0 2024-03-18T18:50:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.055+0000 7fee757fa700 1 -- 172.21.15.1:0/743794451 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6c007480 con 0x7fee780c87d0 2024-03-18T18:50:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.055+0000 7fee7ec3e700 1 -- 172.21.15.1:0/743794451 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee7814aca0 con 0x7fee780c87d0 2024-03-18T18:50:56.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.056+0000 7fee7ec3e700 1 -- 172.21.15.1:0/743794451 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee780676a0 con 0x7fee780c87d0 2024-03-18T18:50:56.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.057+0000 7fee757fa700 1 -- 172.21.15.1:0/743794451 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fee6c0075e0 con 0x7fee780c87d0 2024-03-18T18:50:56.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.057+0000 7fee757fa700 1 --2- 172.21.15.1:0/743794451 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee5c06ac90 0x7fee5c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:56.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.057+0000 7fee77fff700 1 --2- 172.21.15.1:0/743794451 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee5c06ac90 0x7fee5c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:56.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.057+0000 7fee757fa700 1 -- 172.21.15.1:0/743794451 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fee6c088d50 con 0x7fee780c87d0 2024-03-18T18:50:56.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.057+0000 7fee77fff700 1 --2- 172.21.15.1:0/743794451 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee5c06ac90 0x7fee5c06d150 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fee640097c0 tx=0x7fee64008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:56.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.062+0000 7fee757fa700 1 -- 172.21.15.1:0/743794451 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fee6c08c030 con 0x7fee780c87d0 2024-03-18T18:50:56.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.264+0000 7fee7ec3e700 1 -- 172.21.15.1:0/743794451 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fee780ce5b0 con 0x7fee780c87d0 2024-03-18T18:50:56.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.265+0000 7fee757fa700 1 -- 172.21.15.1:0/743794451 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fee6c053af0 con 0x7fee780c87d0 2024-03-18T18:50:56.261 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:56.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.266+0000 7fee7ec3e700 1 -- 172.21.15.1:0/743794451 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee5c06ac90 msgr2=0x7fee5c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:56.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.266+0000 7fee7ec3e700 1 --2- 172.21.15.1:0/743794451 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee5c06ac90 0x7fee5c06d150 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fee640097c0 tx=0x7fee64008030 comp rx=0 tx=0).stop 2024-03-18T18:50:56.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.266+0000 7fee7ec3e700 1 -- 172.21.15.1:0/743794451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee780c87d0 msgr2=0x7fee78149da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:56.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.266+0000 7fee7ec3e700 1 --2- 172.21.15.1:0/743794451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee780c87d0 0x7fee78149da0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fee6c00c700 tx=0x7fee6c00cac0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.267+0000 7fee7ec3e700 1 -- 172.21.15.1:0/743794451 shutdown_connections 2024-03-18T18:50:56.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.267+0000 7fee7ec3e700 1 --2- 172.21.15.1:0/743794451 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee5c06ac90 0x7fee5c06d150 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.267+0000 7fee7ec3e700 1 --2- 172.21.15.1:0/743794451 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee7815b6c0 0x7fee780c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.267+0000 7fee7ec3e700 1 --2- 172.21.15.1:0/743794451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee780c87d0 0x7fee78149da0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.267+0000 7fee7ec3e700 1 --2- 172.21.15.1:0/743794451 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee780cd430 0x7fee780c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.267+0000 7fee7ec3e700 1 -- 172.21.15.1:0/743794451 >> 172.21.15.1:0/743794451 conn(0x7fee780d0ff0 msgr2=0x7fee780ca600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:56.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.267+0000 7fee7ec3e700 1 -- 172.21.15.1:0/743794451 shutdown_connections 2024-03-18T18:50:56.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.267+0000 7fee7ec3e700 1 -- 172.21.15.1:0/743794451 wait complete. 2024-03-18T18:50:56.275 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T18:50:56.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.405+0000 7fac39b61700 1 Processor -- start 2024-03-18T18:50:56.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.405+0000 7fac39b61700 1 -- start start 2024-03-18T18:50:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.406+0000 7fac39b61700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fac3415a240 0x7fac340ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.406+0000 7fac39b61700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fac3406d560 0x7fac3406fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.406+0000 7fac39b61700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac34070000 con 0x7fac340715b0 2024-03-18T18:50:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.406+0000 7fac39b61700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac340adff0 con 0x7fac3415a240 2024-03-18T18:50:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.406+0000 7fac39b61700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac340cac80 con 0x7fac3406d560 2024-03-18T18:50:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.406+0000 7fac337fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fac3415a240 0x7fac340ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.406+0000 7fac337fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fac3415a240 0x7fac340ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38472/0 (socket says 172.21.15.1:38472) 2024-03-18T18:50:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.406+0000 7fac33fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fac340715b0 0x7fac3415de10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33000/0 (socket says 172.21.15.1:33000) 2024-03-18T18:50:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.406+0000 7fac337fe700 1 -- 172.21.15.1:0/2344987081 learned_addr learned my addr 172.21.15.1:0/2344987081 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.406+0000 7fac32ffd700 1 --2- 172.21.15.1:0/2344987081 >> v2:172.21.15.165:3300/0 conn(0x7fac3406d560 0x7fac3406fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.406+0000 7fac337fe700 1 -- 172.21.15.1:0/2344987081 >> v2:172.21.15.165:3300/0 conn(0x7fac3406d560 msgr2=0x7fac3406fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.406+0000 7fac337fe700 1 --2- 172.21.15.1:0/2344987081 >> v2:172.21.15.165:3300/0 conn(0x7fac3406d560 0x7fac3406fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.406+0000 7fac337fe700 1 -- 172.21.15.1:0/2344987081 >> v1:172.21.15.1:6789/0 conn(0x7fac340715b0 legacy=0x7fac3415de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:56.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.406+0000 7fac337fe700 1 -- 172.21.15.1:0/2344987081 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac340caf60 con 0x7fac3415a240 2024-03-18T18:50:56.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.407+0000 7fac337fe700 1 --2- 172.21.15.1:0/2344987081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac3415a240 0x7fac340ca580 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fac240098f0 tx=0x7fac24009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b125f13b9f92264 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:56.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.407+0000 7fac327fc700 1 -- 172.21.15.1:0/2344987081 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2400f8a0 con 0x7fac3415a240 2024-03-18T18:50:56.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.407+0000 7fac327fc700 1 -- 172.21.15.1:0/2344987081 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2400fa00 con 0x7fac3415a240 2024-03-18T18:50:56.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.407+0000 7fac327fc700 1 -- 172.21.15.1:0/2344987081 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2400fd90 con 0x7fac3415a240 2024-03-18T18:50:56.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.407+0000 7fac39b61700 1 -- 172.21.15.1:0/2344987081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac3415a240 msgr2=0x7fac340ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:56.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.407+0000 7fac39b61700 1 --2- 172.21.15.1:0/2344987081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac3415a240 0x7fac340ca580 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fac240098f0 tx=0x7fac24009c00 comp rx=0 tx=0).stop 2024-03-18T18:50:56.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.407+0000 7fac39b61700 1 -- 172.21.15.1:0/2344987081 shutdown_connections 2024-03-18T18:50:56.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.407+0000 7fac39b61700 1 --2- 172.21.15.1:0/2344987081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac3415a240 0x7fac340ca580 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.407+0000 7fac39b61700 1 --2- 172.21.15.1:0/2344987081 >> v2:172.21.15.165:3300/0 conn(0x7fac3406d560 0x7fac3406fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.407+0000 7fac39b61700 1 -- 172.21.15.1:0/2344987081 >> 172.21.15.1:0/2344987081 conn(0x7fac340cff30 msgr2=0x7fac340d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:56.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.407+0000 7fac39b61700 1 -- 172.21.15.1:0/2344987081 shutdown_connections 2024-03-18T18:50:56.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.408+0000 7fac39b61700 1 -- 172.21.15.1:0/2344987081 wait complete. 2024-03-18T18:50:56.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.409+0000 7fac39b61700 1 Processor -- start 2024-03-18T18:50:56.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.409+0000 7fac39b61700 1 -- start start 2024-03-18T18:50:56.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac39b61700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac3406d560 0x7fac34149f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:56.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac39b61700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac340715b0 0x7fac340c3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:56.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac39b61700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac3415a240 0x7fac340b0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:56.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac39b61700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac341eff90 con 0x7fac3406d560 2024-03-18T18:50:56.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac39b61700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3406f820 con 0x7fac340715b0 2024-03-18T18:50:56.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac39b61700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac3414a460 con 0x7fac3415a240 2024-03-18T18:50:56.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac337fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac3406d560 0x7fac34149f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:56.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac337fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac3406d560 0x7fac34149f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38476/0 (socket says 172.21.15.1:38476) 2024-03-18T18:50:56.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac337fe700 1 -- 172.21.15.1:0/3119983156 learned_addr learned my addr 172.21.15.1:0/3119983156 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:56.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac337fe700 1 -- 172.21.15.1:0/3119983156 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac3415a240 msgr2=0x7fac340b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:56.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac337fe700 1 --2- 172.21.15.1:0/3119983156 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac3415a240 0x7fac340b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac337fe700 1 -- 172.21.15.1:0/3119983156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac340715b0 msgr2=0x7fac340c3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:56.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac32ffd700 1 --2- 172.21.15.1:0/3119983156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac340715b0 0x7fac340c3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:56.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac337fe700 1 --2- 172.21.15.1:0/3119983156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac340715b0 0x7fac340c3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac337fe700 1 -- 172.21.15.1:0/3119983156 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac24009550 con 0x7fac3406d560 2024-03-18T18:50:56.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac337fe700 1 --2- 172.21.15.1:0/3119983156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac3406d560 0x7fac34149f20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fac24000c00 tx=0x7fac24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:56.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac2bfff700 1 -- 172.21.15.1:0/3119983156 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2401e070 con 0x7fac3406d560 2024-03-18T18:50:56.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac2bfff700 1 -- 172.21.15.1:0/3119983156 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac24004550 con 0x7fac3406d560 2024-03-18T18:50:56.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac39b61700 1 -- 172.21.15.1:0/3119983156 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac341efd50 con 0x7fac3406d560 2024-03-18T18:50:56.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac2bfff700 1 -- 172.21.15.1:0/3119983156 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac24003af0 con 0x7fac3406d560 2024-03-18T18:50:56.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.410+0000 7fac39b61700 1 -- 172.21.15.1:0/3119983156 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac341ef270 con 0x7fac3406d560 2024-03-18T18:50:56.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.411+0000 7fac39b61700 1 -- 172.21.15.1:0/3119983156 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac340676a0 con 0x7fac3406d560 2024-03-18T18:50:56.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.412+0000 7fac2bfff700 1 -- 172.21.15.1:0/3119983156 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fac24003c50 con 0x7fac3406d560 2024-03-18T18:50:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.412+0000 7fac2bfff700 1 --2- 172.21.15.1:0/3119983156 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac1406ac90 0x7fac1406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.412+0000 7fac2bfff700 1 -- 172.21.15.1:0/3119983156 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fac2408c4d0 con 0x7fac3406d560 2024-03-18T18:50:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.412+0000 7fac32ffd700 1 --2- 172.21.15.1:0/3119983156 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac1406ac90 0x7fac1406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.413+0000 7fac32ffd700 1 --2- 172.21.15.1:0/3119983156 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac1406ac90 0x7fac1406d150 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fac3406da10 tx=0x7fac1c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.417+0000 7fac2bfff700 1 -- 172.21.15.1:0/3119983156 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fac24057380 con 0x7fac3406d560 2024-03-18T18:50:56.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.634+0000 7fac39b61700 1 -- 172.21.15.1:0/3119983156 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fac34072860 con 0x7fac3406d560 2024-03-18T18:50:56.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.635+0000 7fac2bfff700 1 -- 172.21.15.1:0/3119983156 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7fac2405a9a0 con 0x7fac3406d560 2024-03-18T18:50:56.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.636+0000 7fac39b61700 1 -- 172.21.15.1:0/3119983156 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac1406ac90 msgr2=0x7fac1406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:56.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.636+0000 7fac39b61700 1 --2- 172.21.15.1:0/3119983156 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac1406ac90 0x7fac1406d150 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fac3406da10 tx=0x7fac1c008040 comp rx=0 tx=0).stop 2024-03-18T18:50:56.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.636+0000 7fac39b61700 1 -- 172.21.15.1:0/3119983156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac3406d560 msgr2=0x7fac34149f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:56.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.636+0000 7fac39b61700 1 --2- 172.21.15.1:0/3119983156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac3406d560 0x7fac34149f20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fac24000c00 tx=0x7fac24004750 comp rx=0 tx=0).stop 2024-03-18T18:50:56.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.636+0000 7fac39b61700 1 -- 172.21.15.1:0/3119983156 shutdown_connections 2024-03-18T18:50:56.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.637+0000 7fac39b61700 1 --2- 172.21.15.1:0/3119983156 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac1406ac90 0x7fac1406d150 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.637+0000 7fac39b61700 1 --2- 172.21.15.1:0/3119983156 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac3415a240 0x7fac340b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.637+0000 7fac39b61700 1 --2- 172.21.15.1:0/3119983156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac3406d560 0x7fac34149f20 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.637+0000 7fac39b61700 1 --2- 172.21.15.1:0/3119983156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac340715b0 0x7fac340c3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.637+0000 7fac39b61700 1 -- 172.21.15.1:0/3119983156 >> 172.21.15.1:0/3119983156 conn(0x7fac340cff30 msgr2=0x7fac340cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:56.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.637+0000 7fac39b61700 1 -- 172.21.15.1:0/3119983156 shutdown_connections 2024-03-18T18:50:56.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.637+0000 7fac39b61700 1 -- 172.21.15.1:0/3119983156 wait complete. 2024-03-18T18:50:56.634 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-03-18T18:50:56.645 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi165.front.sepia.ceph.com... 2024-03-18T18:50:56.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:56.645 DEBUG:teuthology.orchestra.run.smithi165:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-03-18T18:50:56.689 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T18:50:56.689 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T18:50:56.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:50:56.690 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-18T18:50:56.750 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-18T18:50:56.751 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T18:50:56.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.882+0000 7fe852cc3700 1 Processor -- start 2024-03-18T18:50:56.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.882+0000 7fe852cc3700 1 -- start start 2024-03-18T18:50:56.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.883+0000 7fe852cc3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe84c06a750 0x7fe84c0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:56.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.883+0000 7fe852cc3700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe84c06ee90 con 0x7fe84c06ea10 2024-03-18T18:50:56.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.883+0000 7fe852cc3700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe84c06e1f0 con 0x7fe84c0ace80 2024-03-18T18:50:56.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.883+0000 7fe852cc3700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe84c06e4d0 con 0x7fe84c06a750 2024-03-18T18:50:56.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.883+0000 7fe850a5f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe84c06a750 0x7fe84c0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:56.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.883+0000 7fe84bfff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe84c06ea10 0x7fe84c06ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33002/0 (socket says 172.21.15.1:33002) 2024-03-18T18:50:56.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.883+0000 7fe84bfff700 1 -- 172.21.15.1:0/153406183 learned_addr learned my addr 172.21.15.1:0/153406183 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:56.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.883+0000 7fe850a5f700 1 -- 172.21.15.1:0/153406183 >> v1:172.21.15.165:6789/0 conn(0x7fe84c0ace80 legacy=0x7fe84c0ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:56.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.883+0000 7fe850a5f700 1 -- 172.21.15.1:0/153406183 >> v1:172.21.15.1:6789/0 conn(0x7fe84c06ea10 legacy=0x7fe84c06ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:56.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.883+0000 7fe850a5f700 1 -- 172.21.15.1:0/153406183 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84c0cc2a0 con 0x7fe84c06a750 2024-03-18T18:50:56.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.883+0000 7fe84b7fe700 1 -- 172.21.15.1:0/153406183 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034388849 0 0) 0x7fe84c06ee90 con 0x7fe84c06ea10 2024-03-18T18:50:56.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.884+0000 7fe850a5f700 1 --2- 172.21.15.1:0/153406183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe84c06a750 0x7fe84c0ca5f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe840009840 tx=0x7fe840009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d891fb17510109 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:56.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.884+0000 7fe84b7fe700 1 -- 172.21.15.1:0/153406183 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe840011830 con 0x7fe84c06a750 2024-03-18T18:50:56.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.884+0000 7fe84b7fe700 1 -- 172.21.15.1:0/153406183 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe840011990 con 0x7fe84c06a750 2024-03-18T18:50:56.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.884+0000 7fe84b7fe700 1 -- 172.21.15.1:0/153406183 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe840011d20 con 0x7fe84c06a750 2024-03-18T18:50:56.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.884+0000 7fe852cc3700 1 -- 172.21.15.1:0/153406183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe84c06a750 msgr2=0x7fe84c0ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:56.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.884+0000 7fe852cc3700 1 --2- 172.21.15.1:0/153406183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe84c06a750 0x7fe84c0ca5f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe840009840 tx=0x7fe840009b50 comp rx=0 tx=0).stop 2024-03-18T18:50:56.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.884+0000 7fe852cc3700 1 -- 172.21.15.1:0/153406183 shutdown_connections 2024-03-18T18:50:56.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.884+0000 7fe852cc3700 1 --2- 172.21.15.1:0/153406183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe84c06a750 0x7fe84c0ca5f0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.884+0000 7fe852cc3700 1 -- 172.21.15.1:0/153406183 >> 172.21.15.1:0/153406183 conn(0x7fe84c0cc990 msgr2=0x7fe84c0ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:56.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.884+0000 7fe852cc3700 1 -- 172.21.15.1:0/153406183 shutdown_connections 2024-03-18T18:50:56.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.884+0000 7fe852cc3700 1 -- 172.21.15.1:0/153406183 wait complete. 2024-03-18T18:50:56.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.886+0000 7fe852cc3700 1 Processor -- start 2024-03-18T18:50:56.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.886+0000 7fe852cc3700 1 -- start start 2024-03-18T18:50:56.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.887+0000 7fe852cc3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe84c06a750 0x7fe84c1f7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:56.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.887+0000 7fe852cc3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe84c06ea10 0x7fe84c0c2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:56.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.887+0000 7fe852cc3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe84c0ace80 0x7fe84c0c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:56.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.887+0000 7fe852cc3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c0ce030 con 0x7fe84c06ea10 2024-03-18T18:50:56.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.887+0000 7fe852cc3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c0ce310 con 0x7fe84c0ace80 2024-03-18T18:50:56.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.887+0000 7fe852cc3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c0ce5f0 con 0x7fe84c06a750 2024-03-18T18:50:56.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.887+0000 7fe84bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe84c06ea10 0x7fe84c0c2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:56.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.887+0000 7fe84bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe84c06ea10 0x7fe84c0c2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38494/0 (socket says 172.21.15.1:38494) 2024-03-18T18:50:56.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.887+0000 7fe84bfff700 1 -- 172.21.15.1:0/1406643648 learned_addr learned my addr 172.21.15.1:0/1406643648 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:56.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.887+0000 7fe850a5f700 1 --2- 172.21.15.1:0/1406643648 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe84c06a750 0x7fe84c1f7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:56.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.887+0000 7fe851260700 1 --2- 172.21.15.1:0/1406643648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe84c0ace80 0x7fe84c0c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:56.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.887+0000 7fe84bfff700 1 -- 172.21.15.1:0/1406643648 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe84c06a750 msgr2=0x7fe84c1f7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:56.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.888+0000 7fe84bfff700 1 --2- 172.21.15.1:0/1406643648 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe84c06a750 0x7fe84c1f7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.888+0000 7fe84bfff700 1 -- 172.21.15.1:0/1406643648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe84c0ace80 msgr2=0x7fe84c0c5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:56.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.888+0000 7fe84bfff700 1 --2- 172.21.15.1:0/1406643648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe84c0ace80 0x7fe84c0c5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:56.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.888+0000 7fe84bfff700 1 -- 172.21.15.1:0/1406643648 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8400094a0 con 0x7fe84c06ea10 2024-03-18T18:50:56.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.888+0000 7fe84bfff700 1 --2- 172.21.15.1:0/1406643648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe84c06ea10 0x7fe84c0c2c70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe83800c520 tx=0x7fe83800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:56.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.888+0000 7fe8497fa700 1 -- 172.21.15.1:0/1406643648 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe838004c80 con 0x7fe84c06ea10 2024-03-18T18:50:56.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.888+0000 7fe8497fa700 1 -- 172.21.15.1:0/1406643648 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe838004e00 con 0x7fe84c06ea10 2024-03-18T18:50:56.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.888+0000 7fe852cc3700 1 -- 172.21.15.1:0/1406643648 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe84c0ce8d0 con 0x7fe84c06ea10 2024-03-18T18:50:56.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.888+0000 7fe8497fa700 1 -- 172.21.15.1:0/1406643648 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe838007310 con 0x7fe84c06ea10 2024-03-18T18:50:56.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.888+0000 7fe852cc3700 1 -- 172.21.15.1:0/1406643648 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe84c156e10 con 0x7fe84c06ea10 2024-03-18T18:50:56.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.889+0000 7fe852cc3700 1 -- 172.21.15.1:0/1406643648 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe84c1588f0 con 0x7fe84c06ea10 2024-03-18T18:50:56.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.889+0000 7fe8497fa700 1 -- 172.21.15.1:0/1406643648 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe838007470 con 0x7fe84c06ea10 2024-03-18T18:50:56.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.890+0000 7fe8497fa700 1 --2- 172.21.15.1:0/1406643648 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe83006ac90 0x7fe83006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:56.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.890+0000 7fe8497fa700 1 -- 172.21.15.1:0/1406643648 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe838089270 con 0x7fe84c06ea10 2024-03-18T18:50:56.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.890+0000 7fe850a5f700 1 --2- 172.21.15.1:0/1406643648 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe83006ac90 0x7fe83006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:56.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.890+0000 7fe850a5f700 1 --2- 172.21.15.1:0/1406643648 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe83006ac90 0x7fe83006d150 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe8400090b0 tx=0x7fe840008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:56.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:56.894+0000 7fe8497fa700 1 -- 172.21.15.1:0/1406643648 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe84c1588f0 con 0x7fe84c06ea10 2024-03-18T18:50:57.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.139+0000 7fe852cc3700 1 -- 172.21.15.1:0/1406643648 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe84c1588f0 con 0x7fe84c06ea10 2024-03-18T18:50:57.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.140+0000 7fe8497fa700 1 -- 172.21.15.1:0/1406643648 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4727 (secure 0 0 0) 0x7fe84c1588f0 con 0x7fe84c06ea10 2024-03-18T18:50:57.137 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:57.137 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/883056851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":883056851},{"type":"v1","addr":"172.21.15.165:6834","nonce":883056851}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1388214718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1388214718},{"type":"v1","addr":"172.21.15.165:6835","nonce":1388214718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3020786191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3020786191},{"type":"v1","addr":"172.21.15.1:6836","nonce":3020786191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T18:49:44.809495+0000","modified":"2024-03-18T18:50:07.888953+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2118908012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2118908012},{"type":"v1","addr":"172.21.15.1:6837","nonce":2118908012}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-18T18:50:57.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.142+0000 7fe852cc3700 1 -- 172.21.15.1:0/1406643648 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe83006ac90 msgr2=0x7fe83006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:57.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.142+0000 7fe852cc3700 1 --2- 172.21.15.1:0/1406643648 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe83006ac90 0x7fe83006d150 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe8400090b0 tx=0x7fe840008030 comp rx=0 tx=0).stop 2024-03-18T18:50:57.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.142+0000 7fe852cc3700 1 -- 172.21.15.1:0/1406643648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe84c06ea10 msgr2=0x7fe84c0c2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:57.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.142+0000 7fe852cc3700 1 --2- 172.21.15.1:0/1406643648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe84c06ea10 0x7fe84c0c2c70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe83800c520 tx=0x7fe83800c830 comp rx=0 tx=0).stop 2024-03-18T18:50:57.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.142+0000 7fe852cc3700 1 -- 172.21.15.1:0/1406643648 shutdown_connections 2024-03-18T18:50:57.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.142+0000 7fe852cc3700 1 --2- 172.21.15.1:0/1406643648 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe83006ac90 0x7fe83006d150 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.142+0000 7fe852cc3700 1 --2- 172.21.15.1:0/1406643648 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe84c06a750 0x7fe84c1f7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.142+0000 7fe852cc3700 1 --2- 172.21.15.1:0/1406643648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe84c0ace80 0x7fe84c0c5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.142+0000 7fe852cc3700 1 --2- 172.21.15.1:0/1406643648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe84c06ea10 0x7fe84c0c2c70 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.142+0000 7fe852cc3700 1 -- 172.21.15.1:0/1406643648 >> 172.21.15.1:0/1406643648 conn(0x7fe84c0cc990 msgr2=0x7fe84c0c7070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:57.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.142+0000 7fe852cc3700 1 -- 172.21.15.1:0/1406643648 shutdown_connections 2024-03-18T18:50:57.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.142+0000 7fe852cc3700 1 -- 172.21.15.1:0/1406643648 wait complete. 2024-03-18T18:50:57.140 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 9 2024-03-18T18:50:57.151 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:57.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.282+0000 7f5c67481700 1 Processor -- start 2024-03-18T18:50:57.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.282+0000 7f5c67481700 1 -- start start 2024-03-18T18:50:57.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.283+0000 7f5c67481700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6015d580 0x7f5c60072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:57.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.283+0000 7f5c67481700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c6006e290 0x7f5c600cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:57.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.283+0000 7f5c67481700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5c6006d640 0x7f5c6006dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:57.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.283+0000 7f5c67481700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c6006e010 con 0x7f5c6006e290 2024-03-18T18:50:57.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.283+0000 7f5c67481700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c6006f490 con 0x7f5c6006d640 2024-03-18T18:50:57.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.283+0000 7f5c67481700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c60071cf0 con 0x7f5c6015d580 2024-03-18T18:50:57.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.283+0000 7f5c64a1c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c6006e290 0x7f5c600cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.283+0000 7f5c64a1c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c6006e290 0x7f5c600cd4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38502/0 (socket says 172.21.15.1:38502) 2024-03-18T18:50:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.283+0000 7f5c64a1c700 1 -- 172.21.15.1:0/360653579 learned_addr learned my addr 172.21.15.1:0/360653579 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.283+0000 7f5c6521d700 1 --2- 172.21.15.1:0/360653579 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6015d580 0x7f5c60072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.283+0000 7f5c65a1e700 1 --2- 172.21.15.1:0/360653579 >> v2:172.21.15.165:3300/0 conn(0x7f5c6006d640 0x7f5c6006dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.283+0000 7f5c64a1c700 1 -- 172.21.15.1:0/360653579 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6015d580 msgr2=0x7f5c60072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.283+0000 7f5c64a1c700 1 --2- 172.21.15.1:0/360653579 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6015d580 0x7f5c60072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.283+0000 7f5c64a1c700 1 -- 172.21.15.1:0/360653579 >> v2:172.21.15.165:3300/0 conn(0x7f5c6006d640 msgr2=0x7f5c6006dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.283+0000 7f5c64a1c700 1 --2- 172.21.15.1:0/360653579 >> v2:172.21.15.165:3300/0 conn(0x7f5c6006d640 0x7f5c6006dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.283+0000 7f5c64a1c700 1 -- 172.21.15.1:0/360653579 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6006f1b0 con 0x7f5c6006e290 2024-03-18T18:50:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.283+0000 7f5c65a1e700 1 --2- 172.21.15.1:0/360653579 >> v2:172.21.15.165:3300/0 conn(0x7f5c6006d640 0x7f5c6006dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:57.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.283+0000 7f5c64a1c700 1 --2- 172.21.15.1:0/360653579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6006e290 0x7f5c600cd4b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5c60072eb0 tx=0x7f5c50009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf6a6da16ae0ba6e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:57.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.284+0000 7f5c57fff700 1 -- 172.21.15.1:0/360653579 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5000b870 con 0x7f5c6006e290 2024-03-18T18:50:57.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.284+0000 7f5c57fff700 1 -- 172.21.15.1:0/360653579 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c5000b9d0 con 0x7f5c6006e290 2024-03-18T18:50:57.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.284+0000 7f5c57fff700 1 -- 172.21.15.1:0/360653579 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5000bd60 con 0x7f5c6006e290 2024-03-18T18:50:57.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.284+0000 7f5c67481700 1 -- 172.21.15.1:0/360653579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6006e290 msgr2=0x7f5c600cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:57.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.284+0000 7f5c67481700 1 --2- 172.21.15.1:0/360653579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6006e290 0x7f5c600cd4b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5c60072eb0 tx=0x7f5c50009bd0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.284+0000 7f5c67481700 1 -- 172.21.15.1:0/360653579 shutdown_connections 2024-03-18T18:50:57.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.284+0000 7f5c67481700 1 --2- 172.21.15.1:0/360653579 >> v2:172.21.15.165:3300/0 conn(0x7f5c6006d640 0x7f5c6006dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.284+0000 7f5c67481700 1 --2- 172.21.15.1:0/360653579 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6015d580 0x7f5c60072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.284+0000 7f5c67481700 1 --2- 172.21.15.1:0/360653579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6006e290 0x7f5c600cd4b0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.284+0000 7f5c67481700 1 -- 172.21.15.1:0/360653579 >> 172.21.15.1:0/360653579 conn(0x7f5c600cc990 msgr2=0x7f5c600ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:57.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.284+0000 7f5c67481700 1 -- 172.21.15.1:0/360653579 shutdown_connections 2024-03-18T18:50:57.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.284+0000 7f5c67481700 1 -- 172.21.15.1:0/360653579 wait complete. 2024-03-18T18:50:57.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.286+0000 7f5c67481700 1 Processor -- start 2024-03-18T18:50:57.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.286+0000 7f5c67481700 1 -- start start 2024-03-18T18:50:57.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.286+0000 7f5c67481700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6006d640 0x7f5c600c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:57.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.286+0000 7f5c67481700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6015d580 0x7f5c6015f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:57.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.286+0000 7f5c67481700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c600c6fe0 0x7f5c6006a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:57.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.286+0000 7f5c67481700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6015ff40 con 0x7f5c6015d580 2024-03-18T18:50:57.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.286+0000 7f5c67481700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c600c63f0 con 0x7f5c600c6fe0 2024-03-18T18:50:57.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.286+0000 7f5c67481700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6015fa30 con 0x7f5c6006d640 2024-03-18T18:50:57.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.286+0000 7f5c64a1c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6015d580 0x7f5c6015f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:57.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.286+0000 7f5c64a1c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6015d580 0x7f5c6015f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38518/0 (socket says 172.21.15.1:38518) 2024-03-18T18:50:57.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.286+0000 7f5c64a1c700 1 -- 172.21.15.1:0/1013289344 learned_addr learned my addr 172.21.15.1:0/1013289344 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:57.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.287+0000 7f5c6521d700 1 --2- 172.21.15.1:0/1013289344 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6006d640 0x7f5c600c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:57.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.287+0000 7f5c65a1e700 1 --2- 172.21.15.1:0/1013289344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c600c6fe0 0x7f5c6006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:57.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.287+0000 7f5c64a1c700 1 -- 172.21.15.1:0/1013289344 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6006d640 msgr2=0x7f5c600c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:57.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.287+0000 7f5c64a1c700 1 --2- 172.21.15.1:0/1013289344 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6006d640 0x7f5c600c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.287+0000 7f5c64a1c700 1 -- 172.21.15.1:0/1013289344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c600c6fe0 msgr2=0x7f5c6006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:57.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.287+0000 7f5c64a1c700 1 --2- 172.21.15.1:0/1013289344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c600c6fe0 0x7f5c6006a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.287+0000 7f5c64a1c700 1 -- 172.21.15.1:0/1013289344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c50009550 con 0x7f5c6015d580 2024-03-18T18:50:57.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.287+0000 7f5c64a1c700 1 --2- 172.21.15.1:0/1013289344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6015d580 0x7f5c6015f4f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5c60067310 tx=0x7f5c50004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:57.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.287+0000 7f5c55ffb700 1 -- 172.21.15.1:0/1013289344 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c50004480 con 0x7f5c6015d580 2024-03-18T18:50:57.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.287+0000 7f5c55ffb700 1 -- 172.21.15.1:0/1013289344 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c500045e0 con 0x7f5c6015d580 2024-03-18T18:50:57.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.287+0000 7f5c67481700 1 -- 172.21.15.1:0/1013289344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c601f2f70 con 0x7f5c6015d580 2024-03-18T18:50:57.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.287+0000 7f5c55ffb700 1 -- 172.21.15.1:0/1013289344 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c500038e0 con 0x7f5c6015d580 2024-03-18T18:50:57.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.287+0000 7f5c67481700 1 -- 172.21.15.1:0/1013289344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c600c7830 con 0x7f5c6015d580 2024-03-18T18:50:57.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.288+0000 7f5c67481700 1 -- 172.21.15.1:0/1013289344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c601f3100 con 0x7f5c6015d580 2024-03-18T18:50:57.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.289+0000 7f5c55ffb700 1 -- 172.21.15.1:0/1013289344 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5c50021070 con 0x7f5c6015d580 2024-03-18T18:50:57.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.289+0000 7f5c55ffb700 1 --2- 172.21.15.1:0/1013289344 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5c4406ac90 0x7f5c4406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:57.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.289+0000 7f5c55ffb700 1 -- 172.21.15.1:0/1013289344 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5c5008ade0 con 0x7f5c6015d580 2024-03-18T18:50:57.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.289+0000 7f5c6521d700 1 --2- 172.21.15.1:0/1013289344 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5c4406ac90 0x7f5c4406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:57.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.290+0000 7f5c6521d700 1 --2- 172.21.15.1:0/1013289344 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5c4406ac90 0x7f5c4406d150 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c006b20 tx=0x7f5c5c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:57.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.294+0000 7f5c55ffb700 1 -- 172.21.15.1:0/1013289344 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c50055c00 con 0x7f5c6015d580 2024-03-18T18:50:57.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.498+0000 7f5c67481700 1 -- 172.21.15.1:0/1013289344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c600676a0 con 0x7f5c6015d580 2024-03-18T18:50:57.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.499+0000 7f5c55ffb700 1 -- 172.21.15.1:0/1013289344 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18217 (secure 0 0 0) 0x7f5c50015090 con 0x7f5c6015d580 2024-03-18T18:50:57.496 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:57.496 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":27,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:50:34.993482+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:57.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.502+0000 7f5c67481700 1 -- 172.21.15.1:0/1013289344 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5c4406ac90 msgr2=0x7f5c4406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:57.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.502+0000 7f5c67481700 1 --2- 172.21.15.1:0/1013289344 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5c4406ac90 0x7f5c4406d150 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c006b20 tx=0x7f5c5c008040 comp rx=0 tx=0).stop 2024-03-18T18:50:57.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.502+0000 7f5c67481700 1 -- 172.21.15.1:0/1013289344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6015d580 msgr2=0x7f5c6015f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:57.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.502+0000 7f5c67481700 1 --2- 172.21.15.1:0/1013289344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6015d580 0x7f5c6015f4f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5c60067310 tx=0x7f5c50004750 comp rx=0 tx=0).stop 2024-03-18T18:50:57.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.502+0000 7f5c67481700 1 -- 172.21.15.1:0/1013289344 shutdown_connections 2024-03-18T18:50:57.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.502+0000 7f5c67481700 1 --2- 172.21.15.1:0/1013289344 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5c4406ac90 0x7f5c4406d150 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.502+0000 7f5c67481700 1 --2- 172.21.15.1:0/1013289344 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6006d640 0x7f5c600c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.502+0000 7f5c67481700 1 --2- 172.21.15.1:0/1013289344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c600c6fe0 0x7f5c6006a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.502+0000 7f5c67481700 1 --2- 172.21.15.1:0/1013289344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6015d580 0x7f5c6015f4f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.502+0000 7f5c67481700 1 -- 172.21.15.1:0/1013289344 >> 172.21.15.1:0/1013289344 conn(0x7f5c600cc990 msgr2=0x7f5c600cbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:57.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.502+0000 7f5c67481700 1 -- 172.21.15.1:0/1013289344 shutdown_connections 2024-03-18T18:50:57.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.502+0000 7f5c67481700 1 -- 172.21.15.1:0/1013289344 wait complete. 2024-03-18T18:50:57.511 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T18:49:42.800801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T18:49:43.776684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T18:49:46.854608+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T18:50:57.512 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T18:50:57.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.643+0000 7fb7fe832700 1 Processor -- start 2024-03-18T18:50:57.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.643+0000 7fb7fe832700 1 -- start start 2024-03-18T18:50:57.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.644+0000 7fb7fe832700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb7f815d780 0x7fb7f80cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:57.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.644+0000 7fb7fe832700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7f809dd40 0x7fb7f80cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:57.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.644+0000 7fb7fe832700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f80ce090 con 0x7fb7f80ce5a0 2024-03-18T18:50:57.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.644+0000 7fb7fe832700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f806db30 con 0x7fb7f815d780 2024-03-18T18:50:57.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.644+0000 7fb7fe832700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f806d520 con 0x7fb7f809dd40 2024-03-18T18:50:57.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.644+0000 7fb7f7fff700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb7f815d780 0x7fb7f80cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:57.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.644+0000 7fb7f77fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7f809dd40 0x7fb7f80cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:57.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.644+0000 7fb7fcdcf700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fb7f80ce5a0 0x7fb7f80cea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:36052/0 (socket says 172.21.15.1:36052) 2024-03-18T18:50:57.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.644+0000 7fb7f77fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7f809dd40 0x7fb7f80cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:46992/0 (socket says 172.21.15.1:46992) 2024-03-18T18:50:57.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.644+0000 7fb7f7fff700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb7f815d780 0x7fb7f80cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:55482/0 (socket says 172.21.15.1:55482) 2024-03-18T18:50:57.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.644+0000 7fb7fcdcf700 1 -- 172.21.15.1:0/2051065905 learned_addr learned my addr 172.21.15.1:0/2051065905 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:57.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.645+0000 7fb7f77fe700 1 -- 172.21.15.1:0/2051065905 >> v2:172.21.15.165:3300/0 conn(0x7fb7f815d780 msgr2=0x7fb7f80cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:57.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.645+0000 7fb7f77fe700 1 --2- 172.21.15.1:0/2051065905 >> v2:172.21.15.165:3300/0 conn(0x7fb7f815d780 0x7fb7f80cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.645+0000 7fb7f77fe700 1 -- 172.21.15.1:0/2051065905 >> v1:172.21.15.165:6789/0 conn(0x7fb7f80ce5a0 legacy=0x7fb7f80cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:57.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.645+0000 7fb7f77fe700 1 -- 172.21.15.1:0/2051065905 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f806f460 con 0x7fb7f809dd40 2024-03-18T18:50:57.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.645+0000 7fb7f7fff700 1 --2- 172.21.15.1:0/2051065905 >> v2:172.21.15.165:3300/0 conn(0x7fb7f815d780 0x7fb7f80cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:50:57.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.645+0000 7fb7f77fe700 1 --2- 172.21.15.1:0/2051065905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7f809dd40 0x7fb7f80cad70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009840 tx=0x7fb7e8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdfd5f3d92fd11ff server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:57.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.645+0000 7fb7f6ffd700 1 -- 172.21.15.1:0/2051065905 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8011830 con 0x7fb7f809dd40 2024-03-18T18:50:57.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.645+0000 7fb7f6ffd700 1 -- 172.21.15.1:0/2051065905 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e8011990 con 0x7fb7f809dd40 2024-03-18T18:50:57.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.645+0000 7fb7f6ffd700 1 -- 172.21.15.1:0/2051065905 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8011d20 con 0x7fb7f809dd40 2024-03-18T18:50:57.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.646+0000 7fb7fe832700 1 -- 172.21.15.1:0/2051065905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7f809dd40 msgr2=0x7fb7f80cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:57.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.646+0000 7fb7fe832700 1 --2- 172.21.15.1:0/2051065905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7f809dd40 0x7fb7f80cad70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009840 tx=0x7fb7e8009b50 comp rx=0 tx=0).stop 2024-03-18T18:50:57.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.646+0000 7fb7fe832700 1 -- 172.21.15.1:0/2051065905 shutdown_connections 2024-03-18T18:50:57.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.646+0000 7fb7fe832700 1 --2- 172.21.15.1:0/2051065905 >> v2:172.21.15.165:3300/0 conn(0x7fb7f815d780 0x7fb7f80cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.646+0000 7fb7fe832700 1 --2- 172.21.15.1:0/2051065905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7f809dd40 0x7fb7f80cad70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.646+0000 7fb7fe832700 1 -- 172.21.15.1:0/2051065905 >> 172.21.15.1:0/2051065905 conn(0x7fb7f80cca00 msgr2=0x7fb7f80cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:57.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.646+0000 7fb7fe832700 1 -- 172.21.15.1:0/2051065905 shutdown_connections 2024-03-18T18:50:57.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.646+0000 7fb7fe832700 1 -- 172.21.15.1:0/2051065905 wait complete. 2024-03-18T18:50:57.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.647+0000 7fb7fe832700 1 Processor -- start 2024-03-18T18:50:57.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.647+0000 7fb7fe832700 1 -- start start 2024-03-18T18:50:57.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7fe832700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7f809dd40 0x7fb7f8149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:57.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7fe832700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7f80ce5a0 0x7fb7f80c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:57.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7fe832700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7f815d780 0x7fb7f814e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:57.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7fe832700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f80c7ba0 con 0x7fb7f815d780 2024-03-18T18:50:57.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7fe832700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f814dc50 con 0x7fb7f80ce5a0 2024-03-18T18:50:57.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7fe832700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f814df30 con 0x7fb7f809dd40 2024-03-18T18:50:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7fcdcf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7f815d780 0x7fb7f814e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7fcdcf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7f815d780 0x7fb7f814e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38522/0 (socket says 172.21.15.1:38522) 2024-03-18T18:50:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7fcdcf700 1 -- 172.21.15.1:0/1506535281 learned_addr learned my addr 172.21.15.1:0/1506535281 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7f7fff700 1 --2- 172.21.15.1:0/1506535281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7f809dd40 0x7fb7f8149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7f77fe700 1 --2- 172.21.15.1:0/1506535281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7f80ce5a0 0x7fb7f80c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7fcdcf700 1 -- 172.21.15.1:0/1506535281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7f809dd40 msgr2=0x7fb7f8149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7fcdcf700 1 --2- 172.21.15.1:0/1506535281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7f809dd40 0x7fb7f8149f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7fcdcf700 1 -- 172.21.15.1:0/1506535281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7f80ce5a0 msgr2=0x7fb7f80c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7fcdcf700 1 --2- 172.21.15.1:0/1506535281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7f80ce5a0 0x7fb7f80c7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7fcdcf700 1 -- 172.21.15.1:0/1506535281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e00094a0 con 0x7fb7f815d780 2024-03-18T18:50:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7fcdcf700 1 --2- 172.21.15.1:0/1506535281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7f815d780 0x7fb7f814e730 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000cbc0 tx=0x7fb7f000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7f4ff9700 1 -- 172.21.15.1:0/1506535281 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f000a9e0 con 0x7fb7f815d780 2024-03-18T18:50:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7f4ff9700 1 -- 172.21.15.1:0/1506535281 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f000ab40 con 0x7fb7f815d780 2024-03-18T18:50:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7fe832700 1 -- 172.21.15.1:0/1506535281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e80094a0 con 0x7fb7f815d780 2024-03-18T18:50:57.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.648+0000 7fb7f4ff9700 1 -- 172.21.15.1:0/1506535281 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f0012400 con 0x7fb7f815d780 2024-03-18T18:50:57.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.649+0000 7fb7fe832700 1 -- 172.21.15.1:0/1506535281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f80c7e80 con 0x7fb7f815d780 2024-03-18T18:50:57.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.650+0000 7fb7fe832700 1 -- 172.21.15.1:0/1506535281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f814dab0 con 0x7fb7f815d780 2024-03-18T18:50:57.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.650+0000 7fb7f4ff9700 1 -- 172.21.15.1:0/1506535281 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb7f001b3b0 con 0x7fb7f815d780 2024-03-18T18:50:57.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.650+0000 7fb7f4ff9700 1 --2- 172.21.15.1:0/1506535281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb7d806ac90 0x7fb7d806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:57.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.650+0000 7fb7f4ff9700 1 -- 172.21.15.1:0/1506535281 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fb7f00891d0 con 0x7fb7f815d780 2024-03-18T18:50:57.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.650+0000 7fb7f7fff700 1 --2- 172.21.15.1:0/1506535281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb7d806ac90 0x7fb7d806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:57.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.651+0000 7fb7f7fff700 1 --2- 172.21.15.1:0/1506535281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb7d806ac90 0x7fb7d806d150 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0009cd0 tx=0x7fb7e0009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:57.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.655+0000 7fb7f4ff9700 1 -- 172.21.15.1:0/1506535281 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb7f814dab0 con 0x7fb7f815d780 2024-03-18T18:50:57.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.856+0000 7fb7fe832700 1 -- 172.21.15.1:0/1506535281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb7f814dd90 con 0x7fb7f815d780 2024-03-18T18:50:57.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.856+0000 7fb7f4ff9700 1 -- 172.21.15.1:0/1506535281 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb7f814dd90 con 0x7fb7f815d780 2024-03-18T18:50:57.853 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T18:50:57.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.858+0000 7fb7fe832700 1 -- 172.21.15.1:0/1506535281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb7d806ac90 msgr2=0x7fb7d806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:57.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.858+0000 7fb7fe832700 1 --2- 172.21.15.1:0/1506535281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb7d806ac90 0x7fb7d806d150 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0009cd0 tx=0x7fb7e0009210 comp rx=0 tx=0).stop 2024-03-18T18:50:57.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.858+0000 7fb7fe832700 1 -- 172.21.15.1:0/1506535281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7f815d780 msgr2=0x7fb7f814e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:57.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.858+0000 7fb7fe832700 1 --2- 172.21.15.1:0/1506535281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7f815d780 0x7fb7f814e730 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000cbc0 tx=0x7fb7f000ced0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.858+0000 7fb7fe832700 1 -- 172.21.15.1:0/1506535281 shutdown_connections 2024-03-18T18:50:57.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.858+0000 7fb7fe832700 1 --2- 172.21.15.1:0/1506535281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb7d806ac90 0x7fb7d806d150 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.858+0000 7fb7fe832700 1 --2- 172.21.15.1:0/1506535281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7f809dd40 0x7fb7f8149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.858+0000 7fb7fe832700 1 --2- 172.21.15.1:0/1506535281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7f815d780 0x7fb7f814e730 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.858+0000 7fb7fe832700 1 --2- 172.21.15.1:0/1506535281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7f80ce5a0 0x7fb7f80c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:57.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.858+0000 7fb7fe832700 1 -- 172.21.15.1:0/1506535281 >> 172.21.15.1:0/1506535281 conn(0x7fb7f80cca00 msgr2=0x7fb7f80b0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:57.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.859+0000 7fb7fe832700 1 -- 172.21.15.1:0/1506535281 shutdown_connections 2024-03-18T18:50:57.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.859+0000 7fb7fe832700 1 -- 172.21.15.1:0/1506535281 wait complete. 2024-03-18T18:50:57.867 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T18:50:57.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.999+0000 7efcada28700 1 Processor -- start 2024-03-18T18:50:57.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:57.999+0000 7efcada28700 1 -- start start 2024-03-18T18:50:57.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.000+0000 7efcada28700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7efca8071d30 0x7efca80add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.000+0000 7efcada28700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca80ce8f0 con 0x7efca80cb9e0 2024-03-18T18:50:58.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.000+0000 7efcada28700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca80ae260 con 0x7efca815a1d0 2024-03-18T18:50:58.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.000+0000 7efcada28700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca80cdb00 con 0x7efca8071d30 2024-03-18T18:50:58.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.000+0000 7efca77fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efca80cb9e0 0x7efca80cbe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33006/0 (socket says 172.21.15.1:33006) 2024-03-18T18:50:58.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.000+0000 7efca77fe700 1 -- 172.21.15.1:0/3128297112 learned_addr learned my addr 172.21.15.1:0/3128297112 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:58.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca67fc700 1 --2- 172.21.15.1:0/3128297112 >> v2:172.21.15.165:3300/0 conn(0x7efca8071d30 0x7efca80add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca5ffb700 1 -- 172.21.15.1:0/3128297112 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124403507 0 0) 0x7efca80ce8f0 con 0x7efca80cb9e0 2024-03-18T18:50:58.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca5ffb700 1 -- 172.21.15.1:0/3128297112 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc900035b0 con 0x7efca80cb9e0 2024-03-18T18:50:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca5ffb700 1 -- 172.21.15.1:0/3128297112 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3420504898 0 0) 0x7efc900035b0 con 0x7efca80cb9e0 2024-03-18T18:50:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca5ffb700 1 -- 172.21.15.1:0/3128297112 >> v2:172.21.15.165:3300/0 conn(0x7efca8071d30 msgr2=0x7efca80add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca5ffb700 1 --2- 172.21.15.1:0/3128297112 >> v2:172.21.15.165:3300/0 conn(0x7efca8071d30 0x7efca80add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca5ffb700 1 -- 172.21.15.1:0/3128297112 >> v1:172.21.15.165:6789/0 conn(0x7efca815a1d0 legacy=0x7efca8071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca5ffb700 1 -- 172.21.15.1:0/3128297112 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca80cdce0 con 0x7efca80cb9e0 2024-03-18T18:50:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca5ffb700 1 -- 172.21.15.1:0/3128297112 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7efca00019f0 con 0x7efca80cb9e0 2024-03-18T18:50:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca5ffb700 1 -- 172.21.15.1:0/3128297112 >> v1:172.21.15.1:6789/0 conn(0x7efca80cb9e0 legacy=0x7efca80cbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca5ffb700 1 --2- 172.21.15.1:0/3128297112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc90003a50 0x7efc90003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca5ffb700 1 --2- 172.21.15.1:0/3128297112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc90004850 0x7efc90004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca5ffb700 1 --2- 172.21.15.1:0/3128297112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc90005210 0x7efc9000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca5ffb700 1 -- 172.21.15.1:0/3128297112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca80cdb00 con 0x7efc90004850 2024-03-18T18:50:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca5ffb700 1 -- 172.21.15.1:0/3128297112 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9000c110 con 0x7efc90005210 2024-03-18T18:50:58.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca5ffb700 1 -- 172.21.15.1:0/3128297112 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc9000c2d0 con 0x7efc90003a50 2024-03-18T18:50:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca67fc700 1 --2- 172.21.15.1:0/3128297112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc90004850 0x7efc90004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca77fe700 1 --2- 172.21.15.1:0/3128297112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc90005210 0x7efc9000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca6ffd700 1 --2- 172.21.15.1:0/3128297112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc90003a50 0x7efc90003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca67fc700 1 -- 172.21.15.1:0/3128297112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc90003a50 msgr2=0x7efc90003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca67fc700 1 --2- 172.21.15.1:0/3128297112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc90003a50 0x7efc90003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca67fc700 1 -- 172.21.15.1:0/3128297112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc90005210 msgr2=0x7efc9000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca67fc700 1 --2- 172.21.15.1:0/3128297112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc90005210 0x7efc9000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.001+0000 7efca67fc700 1 -- 172.21.15.1:0/3128297112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc900035b0 con 0x7efc90004850 2024-03-18T18:50:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.002+0000 7efca77fe700 1 -- 172.21.15.1:0/3128297112 reap_dead start 2024-03-18T18:50:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.002+0000 7efca67fc700 1 --2- 172.21.15.1:0/3128297112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc90004850 0x7efc90004cd0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009b60 tx=0x7efc90005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aefe7fecd12530f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.002+0000 7efca5ffb700 1 -- 172.21.15.1:0/3128297112 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c00e6c0 con 0x7efc90004850 2024-03-18T18:50:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.002+0000 7efca5ffb700 1 -- 172.21.15.1:0/3128297112 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9c00e820 con 0x7efc90004850 2024-03-18T18:50:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.002+0000 7efca5ffb700 1 -- 172.21.15.1:0/3128297112 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c00eb90 con 0x7efc90004850 2024-03-18T18:50:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.002+0000 7efcada28700 1 -- 172.21.15.1:0/3128297112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc90004850 msgr2=0x7efc90004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.002+0000 7efcada28700 1 --2- 172.21.15.1:0/3128297112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc90004850 0x7efc90004cd0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009b60 tx=0x7efc90005880 comp rx=0 tx=0).stop 2024-03-18T18:50:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.002+0000 7efcada28700 1 -- 172.21.15.1:0/3128297112 shutdown_connections 2024-03-18T18:50:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.002+0000 7efcada28700 1 --2- 172.21.15.1:0/3128297112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc90004850 0x7efc90004cd0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.002+0000 7efcada28700 1 -- 172.21.15.1:0/3128297112 >> 172.21.15.1:0/3128297112 conn(0x7efca80cc9b0 msgr2=0x7efca80ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.002+0000 7efcada28700 1 -- 172.21.15.1:0/3128297112 shutdown_connections 2024-03-18T18:50:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.002+0000 7efcada28700 1 -- 172.21.15.1:0/3128297112 wait complete. 2024-03-18T18:50:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.004+0000 7efcada28700 1 Processor -- start 2024-03-18T18:50:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.004+0000 7efcada28700 1 -- start start 2024-03-18T18:50:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efcada28700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc90004850 0x7efca80c4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efcada28700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efca80c5af0 0x7efca80c6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efcada28700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efca814e340 0x7efca815a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efcada28700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca80c5f70 con 0x7efc90004850 2024-03-18T18:50:58.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efcada28700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca80c4a50 con 0x7efca814e340 2024-03-18T18:50:58.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efcada28700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca80c4d30 con 0x7efca80c5af0 2024-03-18T18:50:58.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efca6ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc90004850 0x7efca80c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efca6ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc90004850 0x7efca80c4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38548/0 (socket says 172.21.15.1:38548) 2024-03-18T18:50:58.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efca6ffd700 1 -- 172.21.15.1:0/1505214270 learned_addr learned my addr 172.21.15.1:0/1505214270 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:58.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efca77fe700 1 --2- 172.21.15.1:0/1505214270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efca814e340 0x7efca815a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efca67fc700 1 --2- 172.21.15.1:0/1505214270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efca80c5af0 0x7efca80c6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efca6ffd700 1 -- 172.21.15.1:0/1505214270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efca80c5af0 msgr2=0x7efca80c6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efca6ffd700 1 --2- 172.21.15.1:0/1505214270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efca80c5af0 0x7efca80c6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efca6ffd700 1 -- 172.21.15.1:0/1505214270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efca814e340 msgr2=0x7efca815a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efca6ffd700 1 --2- 172.21.15.1:0/1505214270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efca814e340 0x7efca815a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efca6ffd700 1 -- 172.21.15.1:0/1505214270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca80c5010 con 0x7efc90004850 2024-03-18T18:50:58.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efca67fc700 1 --2- 172.21.15.1:0/1505214270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efca80c5af0 0x7efca80c6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efca6ffd700 1 --2- 172.21.15.1:0/1505214270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc90004850 0x7efca80c4440 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7efc9800f5e0 tx=0x7efc9800f9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efc8ffff700 1 -- 172.21.15.1:0/1505214270 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc98005320 con 0x7efc90004850 2024-03-18T18:50:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efc8ffff700 1 -- 172.21.15.1:0/1505214270 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc98005480 con 0x7efc90004850 2024-03-18T18:50:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efc8ffff700 1 -- 172.21.15.1:0/1505214270 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc98006af0 con 0x7efc90004850 2024-03-18T18:50:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efcada28700 1 -- 172.21.15.1:0/1505214270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca80c52f0 con 0x7efc90004850 2024-03-18T18:50:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.005+0000 7efcada28700 1 -- 172.21.15.1:0/1505214270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca80c6940 con 0x7efc90004850 2024-03-18T18:50:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.006+0000 7efcada28700 1 -- 172.21.15.1:0/1505214270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca80c5940 con 0x7efc90004850 2024-03-18T18:50:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.007+0000 7efc8ffff700 1 -- 172.21.15.1:0/1505214270 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7efc98006c50 con 0x7efc90004850 2024-03-18T18:50:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.007+0000 7efc8ffff700 1 --2- 172.21.15.1:0/1505214270 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efc9406ac90 0x7efc9406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.007+0000 7efc8ffff700 1 -- 172.21.15.1:0/1505214270 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7efc98088a80 con 0x7efc90004850 2024-03-18T18:50:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.007+0000 7efca67fc700 1 --2- 172.21.15.1:0/1505214270 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efc9406ac90 0x7efc9406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.008+0000 7efca67fc700 1 --2- 172.21.15.1:0/1505214270 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efc9406ac90 0x7efc9406d150 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efc9c0090b0 tx=0x7efc9c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:58.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.012+0000 7efc8ffff700 1 -- 172.21.15.1:0/1505214270 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efc98053930 con 0x7efc90004850 2024-03-18T18:50:58.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.215+0000 7efcada28700 1 -- 172.21.15.1:0/1505214270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efca80676a0 con 0x7efc90004850 2024-03-18T18:50:58.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.216+0000 7efc8ffff700 1 -- 172.21.15.1:0/1505214270 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7efc9801a020 con 0x7efc90004850 2024-03-18T18:50:58.212 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:58.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.218+0000 7efcada28700 1 -- 172.21.15.1:0/1505214270 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efc9406ac90 msgr2=0x7efc9406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.218+0000 7efcada28700 1 --2- 172.21.15.1:0/1505214270 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efc9406ac90 0x7efc9406d150 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efc9c0090b0 tx=0x7efc9c008030 comp rx=0 tx=0).stop 2024-03-18T18:50:58.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.218+0000 7efcada28700 1 -- 172.21.15.1:0/1505214270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc90004850 msgr2=0x7efca80c4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.218+0000 7efcada28700 1 --2- 172.21.15.1:0/1505214270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc90004850 0x7efca80c4440 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7efc9800f5e0 tx=0x7efc9800f9a0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.218+0000 7efcada28700 1 -- 172.21.15.1:0/1505214270 shutdown_connections 2024-03-18T18:50:58.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.218+0000 7efcada28700 1 --2- 172.21.15.1:0/1505214270 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efc9406ac90 0x7efc9406d150 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.218+0000 7efcada28700 1 --2- 172.21.15.1:0/1505214270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efca814e340 0x7efca815a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.218+0000 7efcada28700 1 --2- 172.21.15.1:0/1505214270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc90004850 0x7efca80c4440 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.218+0000 7efcada28700 1 --2- 172.21.15.1:0/1505214270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efca80c5af0 0x7efca80c6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.218+0000 7efcada28700 1 -- 172.21.15.1:0/1505214270 >> 172.21.15.1:0/1505214270 conn(0x7efca80cc9b0 msgr2=0x7efca806f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:58.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.218+0000 7efcada28700 1 -- 172.21.15.1:0/1505214270 shutdown_connections 2024-03-18T18:50:58.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.218+0000 7efcada28700 1 -- 172.21.15.1:0/1505214270 wait complete. 2024-03-18T18:50:58.226 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T18:50:58.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.358+0000 7fa1b3954700 1 Processor -- start 2024-03-18T18:50:58.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.359+0000 7fa1b3954700 1 -- start start 2024-03-18T18:50:58.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.359+0000 7fa1b3954700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa1ac15b6c0 0x7fa1ac0b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.359+0000 7fa1b3954700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa1ac0c87d0 0x7fa1ac0cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.359+0000 7fa1b3954700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ac0d01f0 con 0x7fa1ac0cd430 2024-03-18T18:50:58.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.359+0000 7fa1b3954700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ac0ccfa0 con 0x7fa1ac15b6c0 2024-03-18T18:50:58.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.359+0000 7fa1b3954700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ac0d1db0 con 0x7fa1ac0c87d0 2024-03-18T18:50:58.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.359+0000 7fa1b16f0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa1ac15b6c0 0x7fa1ac0b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.360+0000 7fa1b0eef700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa1ac0cd430 0x7fa1ac0c80c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33012/0 (socket says 172.21.15.1:33012) 2024-03-18T18:50:58.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.360+0000 7fa1b0eef700 1 -- 172.21.15.1:0/432966163 learned_addr learned my addr 172.21.15.1:0/432966163 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:58.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.360+0000 7fa1b1ef1700 1 --2- 172.21.15.1:0/432966163 >> v2:172.21.15.165:3300/0 conn(0x7fa1ac0c87d0 0x7fa1ac0cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.360+0000 7fa1b16f0700 1 -- 172.21.15.1:0/432966163 >> v2:172.21.15.165:3300/0 conn(0x7fa1ac0c87d0 msgr2=0x7fa1ac0cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:58.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.360+0000 7fa1b16f0700 1 --2- 172.21.15.1:0/432966163 >> v2:172.21.15.165:3300/0 conn(0x7fa1ac0c87d0 0x7fa1ac0cd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.360+0000 7fa1b16f0700 1 -- 172.21.15.1:0/432966163 >> v1:172.21.15.1:6789/0 conn(0x7fa1ac0cd430 legacy=0x7fa1ac0c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.360+0000 7fa1b16f0700 1 -- 172.21.15.1:0/432966163 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac0cff10 con 0x7fa1ac15b6c0 2024-03-18T18:50:58.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.360+0000 7fa1a3fff700 1 -- 172.21.15.1:0/432966163 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622814294 0 0) 0x7fa1ac0d01f0 con 0x7fa1ac0cd430 2024-03-18T18:50:58.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.360+0000 7fa1b16f0700 1 --2- 172.21.15.1:0/432966163 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1ac15b6c0 0x7fa1ac0b2160 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac0d0f90 tx=0x7fa19c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beed7d2237b1a482 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:58.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.360+0000 7fa1a3fff700 1 -- 172.21.15.1:0/432966163 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c011830 con 0x7fa1ac15b6c0 2024-03-18T18:50:58.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.360+0000 7fa1a3fff700 1 -- 172.21.15.1:0/432966163 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19c011990 con 0x7fa1ac15b6c0 2024-03-18T18:50:58.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.360+0000 7fa1a3fff700 1 -- 172.21.15.1:0/432966163 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c011d20 con 0x7fa1ac15b6c0 2024-03-18T18:50:58.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.361+0000 7fa1b3954700 1 -- 172.21.15.1:0/432966163 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1ac15b6c0 msgr2=0x7fa1ac0b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.361+0000 7fa1b3954700 1 --2- 172.21.15.1:0/432966163 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1ac15b6c0 0x7fa1ac0b2160 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac0d0f90 tx=0x7fa19c009b20 comp rx=0 tx=0).stop 2024-03-18T18:50:58.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.361+0000 7fa1b3954700 1 -- 172.21.15.1:0/432966163 shutdown_connections 2024-03-18T18:50:58.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.361+0000 7fa1b3954700 1 --2- 172.21.15.1:0/432966163 >> v2:172.21.15.165:3300/0 conn(0x7fa1ac0c87d0 0x7fa1ac0cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.361+0000 7fa1b3954700 1 --2- 172.21.15.1:0/432966163 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1ac15b6c0 0x7fa1ac0b2160 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.361+0000 7fa1b3954700 1 -- 172.21.15.1:0/432966163 >> 172.21.15.1:0/432966163 conn(0x7fa1ac0d0ff0 msgr2=0x7fa1ac0d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:58.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.361+0000 7fa1b3954700 1 -- 172.21.15.1:0/432966163 shutdown_connections 2024-03-18T18:50:58.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.361+0000 7fa1b3954700 1 -- 172.21.15.1:0/432966163 wait complete. 2024-03-18T18:50:58.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.362+0000 7fa1b3954700 1 Processor -- start 2024-03-18T18:50:58.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.362+0000 7fa1b3954700 1 -- start start 2024-03-18T18:50:58.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1b3954700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1ac0c87d0 0x7fa1ac1512d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1b3954700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ac0cd430 0x7fa1ac0afbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1b3954700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1ac15b6c0 0x7fa1ac14dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1b3954700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac14f770 con 0x7fa1ac15b6c0 2024-03-18T18:50:58.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1b3954700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac151810 con 0x7fa1ac0cd430 2024-03-18T18:50:58.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1b3954700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac14eb60 con 0x7fa1ac0c87d0 2024-03-18T18:50:58.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1b1ef1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1ac15b6c0 0x7fa1ac14dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1b1ef1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1ac15b6c0 0x7fa1ac14dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38560/0 (socket says 172.21.15.1:38560) 2024-03-18T18:50:58.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1b1ef1700 1 -- 172.21.15.1:0/1490390901 learned_addr learned my addr 172.21.15.1:0/1490390901 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:58.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1b0eef700 1 --2- 172.21.15.1:0/1490390901 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ac0cd430 0x7fa1ac0afbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1b16f0700 1 --2- 172.21.15.1:0/1490390901 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1ac0c87d0 0x7fa1ac1512d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1b1ef1700 1 -- 172.21.15.1:0/1490390901 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1ac0c87d0 msgr2=0x7fa1ac1512d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1b1ef1700 1 --2- 172.21.15.1:0/1490390901 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1ac0c87d0 0x7fa1ac1512d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1b1ef1700 1 -- 172.21.15.1:0/1490390901 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ac0cd430 msgr2=0x7fa1ac0afbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1b1ef1700 1 --2- 172.21.15.1:0/1490390901 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ac0cd430 0x7fa1ac0afbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1b1ef1700 1 -- 172.21.15.1:0/1490390901 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19c0094a0 con 0x7fa1ac15b6c0 2024-03-18T18:50:58.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1b16f0700 1 --2- 172.21.15.1:0/1490390901 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1ac0c87d0 0x7fa1ac1512d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:58.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1b1ef1700 1 --2- 172.21.15.1:0/1490390901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1ac15b6c0 0x7fa1ac14dda0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4009dd0 tx=0x7fa1a400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:58.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1a1ffb700 1 -- 172.21.15.1:0/1490390901 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a400cd40 con 0x7fa1ac15b6c0 2024-03-18T18:50:58.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1a1ffb700 1 -- 172.21.15.1:0/1490390901 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a400f040 con 0x7fa1ac15b6c0 2024-03-18T18:50:58.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.363+0000 7fa1b3954700 1 -- 172.21.15.1:0/1490390901 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ac14ee40 con 0x7fa1ac15b6c0 2024-03-18T18:50:58.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.364+0000 7fa1a1ffb700 1 -- 172.21.15.1:0/1490390901 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a40146c0 con 0x7fa1ac15b6c0 2024-03-18T18:50:58.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.364+0000 7fa1b3954700 1 -- 172.21.15.1:0/1490390901 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ac14f400 con 0x7fa1ac15b6c0 2024-03-18T18:50:58.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.365+0000 7fa1b3954700 1 -- 172.21.15.1:0/1490390901 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ac0676a0 con 0x7fa1ac15b6c0 2024-03-18T18:50:58.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.365+0000 7fa1a1ffb700 1 -- 172.21.15.1:0/1490390901 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa1a4014820 con 0x7fa1ac15b6c0 2024-03-18T18:50:58.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.365+0000 7fa1a1ffb700 1 --2- 172.21.15.1:0/1490390901 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa19006ac90 0x7fa19006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.365+0000 7fa1a1ffb700 1 -- 172.21.15.1:0/1490390901 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa1a4089fc0 con 0x7fa1ac15b6c0 2024-03-18T18:50:58.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.366+0000 7fa1b16f0700 1 --2- 172.21.15.1:0/1490390901 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa19006ac90 0x7fa19006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.366+0000 7fa1b16f0700 1 --2- 172.21.15.1:0/1490390901 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa19006ac90 0x7fa19006d150 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa19c009b60 tx=0x7fa19c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:58.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.370+0000 7fa1a1ffb700 1 -- 172.21.15.1:0/1490390901 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa1a40bd580 con 0x7fa1ac15b6c0 2024-03-18T18:50:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.572+0000 7fa1b3954700 1 -- 172.21.15.1:0/1490390901 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa1ac14f8b0 con 0x7fa1ac15b6c0 2024-03-18T18:50:58.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.572+0000 7fa1a1ffb700 1 -- 172.21.15.1:0/1490390901 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa1a4018020 con 0x7fa1ac15b6c0 2024-03-18T18:50:58.569 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T18:50:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.575+0000 7fa1b3954700 1 -- 172.21.15.1:0/1490390901 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa19006ac90 msgr2=0x7fa19006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.575+0000 7fa1b3954700 1 --2- 172.21.15.1:0/1490390901 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa19006ac90 0x7fa19006d150 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa19c009b60 tx=0x7fa19c008030 comp rx=0 tx=0).stop 2024-03-18T18:50:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.575+0000 7fa1b3954700 1 -- 172.21.15.1:0/1490390901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1ac15b6c0 msgr2=0x7fa1ac14dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.575+0000 7fa1b3954700 1 --2- 172.21.15.1:0/1490390901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1ac15b6c0 0x7fa1ac14dda0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4009dd0 tx=0x7fa1a400c3a0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.575+0000 7fa1b3954700 1 -- 172.21.15.1:0/1490390901 shutdown_connections 2024-03-18T18:50:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.575+0000 7fa1b3954700 1 --2- 172.21.15.1:0/1490390901 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa19006ac90 0x7fa19006d150 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.575+0000 7fa1b3954700 1 --2- 172.21.15.1:0/1490390901 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1ac0c87d0 0x7fa1ac1512d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.575+0000 7fa1b3954700 1 --2- 172.21.15.1:0/1490390901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1ac15b6c0 0x7fa1ac14dda0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.575+0000 7fa1b3954700 1 --2- 172.21.15.1:0/1490390901 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ac0cd430 0x7fa1ac0afbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.575+0000 7fa1b3954700 1 -- 172.21.15.1:0/1490390901 >> 172.21.15.1:0/1490390901 conn(0x7fa1ac0d0ff0 msgr2=0x7fa1ac0d13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.575+0000 7fa1b3954700 1 -- 172.21.15.1:0/1490390901 shutdown_connections 2024-03-18T18:50:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.575+0000 7fa1b3954700 1 -- 172.21.15.1:0/1490390901 wait complete. 2024-03-18T18:50:58.584 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T18:50:58.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.715+0000 7f56ddcb9700 1 Processor -- start 2024-03-18T18:50:58.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.715+0000 7f56ddcb9700 1 -- start start 2024-03-18T18:50:58.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.716+0000 7f56ddcb9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56d815a5f0 0x7f56d80cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.716+0000 7f56ddcb9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d815aad0 con 0x7f56d80cc4d0 2024-03-18T18:50:58.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.716+0000 7f56ddcb9700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d815adb0 con 0x7f56d815dbe0 2024-03-18T18:50:58.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.716+0000 7f56ddcb9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d813f750 con 0x7f56d815a5f0 2024-03-18T18:50:58.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.716+0000 7f56d7fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f56d80cc4d0 0x7f56d806d500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33018/0 (socket says 172.21.15.1:33018) 2024-03-18T18:50:58.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.716+0000 7f56d7fff700 1 -- 172.21.15.1:0/1057204041 learned_addr learned my addr 172.21.15.1:0/1057204041 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:58.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.716+0000 7f56d67fc700 1 -- 172.21.15.1:0/1057204041 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537562750 0 0) 0x7f56d815aad0 con 0x7f56d80cc4d0 2024-03-18T18:50:58.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.716+0000 7f56d67fc700 1 -- 172.21.15.1:0/1057204041 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56c00035b0 con 0x7f56d80cc4d0 2024-03-18T18:50:58.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.716+0000 7f56d6ffd700 1 --2- 172.21.15.1:0/1057204041 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56d815a5f0 0x7f56d80cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d67fc700 1 -- 172.21.15.1:0/1057204041 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3121461122 0 0) 0x7f56c00035b0 con 0x7f56d80cc4d0 2024-03-18T18:50:58.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d67fc700 1 -- 172.21.15.1:0/1057204041 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56d815a5f0 msgr2=0x7f56d80cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:58.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d67fc700 1 --2- 172.21.15.1:0/1057204041 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56d815a5f0 0x7f56d80cbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d67fc700 1 -- 172.21.15.1:0/1057204041 >> v1:172.21.15.165:6789/0 conn(0x7f56d815dbe0 legacy=0x7f56d80b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d67fc700 1 -- 172.21.15.1:0/1057204041 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d815b090 con 0x7f56d80cc4d0 2024-03-18T18:50:58.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d67fc700 1 -- 172.21.15.1:0/1057204041 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f56d00019f0 con 0x7f56d80cc4d0 2024-03-18T18:50:58.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d67fc700 1 -- 172.21.15.1:0/1057204041 >> v1:172.21.15.1:6789/0 conn(0x7f56d80cc4d0 legacy=0x7f56d806d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d67fc700 1 --2- 172.21.15.1:0/1057204041 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56d815a5f0 0x7f56c0003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d67fc700 1 --2- 172.21.15.1:0/1057204041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c0003f70 0x7f56c0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d67fc700 1 --2- 172.21.15.1:0/1057204041 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56c0004d90 0x7f56c0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d67fc700 1 -- 172.21.15.1:0/1057204041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d813f750 con 0x7f56c0003f70 2024-03-18T18:50:58.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d67fc700 1 -- 172.21.15.1:0/1057204041 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c000dc80 con 0x7f56c0004d90 2024-03-18T18:50:58.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d67fc700 1 -- 172.21.15.1:0/1057204041 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c000de40 con 0x7f56d815a5f0 2024-03-18T18:50:58.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d6ffd700 1 --2- 172.21.15.1:0/1057204041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c0003f70 0x7f56c0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d77fe700 1 --2- 172.21.15.1:0/1057204041 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56d815a5f0 0x7f56c0003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d7fff700 1 --2- 172.21.15.1:0/1057204041 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56c0004d90 0x7f56c0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d6ffd700 1 -- 172.21.15.1:0/1057204041 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56d815a5f0 msgr2=0x7f56c0003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:58.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d6ffd700 1 --2- 172.21.15.1:0/1057204041 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56d815a5f0 0x7f56c0003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d6ffd700 1 -- 172.21.15.1:0/1057204041 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56c0004d90 msgr2=0x7f56c0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:58.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d6ffd700 1 --2- 172.21.15.1:0/1057204041 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56c0004d90 0x7f56c0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d6ffd700 1 -- 172.21.15.1:0/1057204041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c00035b0 con 0x7f56c0003f70 2024-03-18T18:50:58.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d7fff700 1 --2- 172.21.15.1:0/1057204041 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56c0004d90 0x7f56c0005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:58.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d6ffd700 1 --2- 172.21.15.1:0/1057204041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c0003f70 0x7f56c0004850 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f56c8009b60 tx=0x7f56c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f14f07e331e53a46 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:58.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d67fc700 1 -- 172.21.15.1:0/1057204041 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c800e6c0 con 0x7f56c0003f70 2024-03-18T18:50:58.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d67fc700 1 -- 172.21.15.1:0/1057204041 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c800e820 con 0x7f56c0003f70 2024-03-18T18:50:58.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.717+0000 7f56d67fc700 1 -- 172.21.15.1:0/1057204041 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c800eb90 con 0x7f56c0003f70 2024-03-18T18:50:58.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.718+0000 7f56ddcb9700 1 -- 172.21.15.1:0/1057204041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c0003f70 msgr2=0x7f56c0004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.718+0000 7f56ddcb9700 1 --2- 172.21.15.1:0/1057204041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c0003f70 0x7f56c0004850 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f56c8009b60 tx=0x7f56c0005880 comp rx=0 tx=0).stop 2024-03-18T18:50:58.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.718+0000 7f56d7fff700 1 -- 172.21.15.1:0/1057204041 reap_dead start 2024-03-18T18:50:58.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.718+0000 7f56ddcb9700 1 -- 172.21.15.1:0/1057204041 shutdown_connections 2024-03-18T18:50:58.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.718+0000 7f56ddcb9700 1 -- 172.21.15.1:0/1057204041 >> 172.21.15.1:0/1057204041 conn(0x7f56d80b1cb0 msgr2=0x7f56d80b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:58.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.718+0000 7f56ddcb9700 1 -- 172.21.15.1:0/1057204041 shutdown_connections 2024-03-18T18:50:58.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.718+0000 7f56ddcb9700 1 -- 172.21.15.1:0/1057204041 wait complete. 2024-03-18T18:50:58.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.719+0000 7f56ddcb9700 1 Processor -- start 2024-03-18T18:50:58.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.719+0000 7f56ddcb9700 1 -- start start 2024-03-18T18:50:58.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.720+0000 7f56ddcb9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c0003f70 0x7f56d8154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.720+0000 7f56ddcb9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56d80cd720 0x7f56d8138b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.720+0000 7f56ddcb9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56d80b0e40 0x7f56d80b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.720+0000 7f56ddcb9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d80b1830 con 0x7f56c0003f70 2024-03-18T18:50:58.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.720+0000 7f56ddcb9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d8153e00 con 0x7f56d80b0e40 2024-03-18T18:50:58.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.720+0000 7f56ddcb9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d8154580 con 0x7f56d80cd720 2024-03-18T18:50:58.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.720+0000 7f56d77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c0003f70 0x7f56d8154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.720+0000 7f56d77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c0003f70 0x7f56d8154040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38572/0 (socket says 172.21.15.1:38572) 2024-03-18T18:50:58.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.720+0000 7f56d77fe700 1 -- 172.21.15.1:0/3107884743 learned_addr learned my addr 172.21.15.1:0/3107884743 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:58.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.720+0000 7f56d7fff700 1 --2- 172.21.15.1:0/3107884743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56d80b0e40 0x7f56d80b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.720+0000 7f56d6ffd700 1 --2- 172.21.15.1:0/3107884743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56d80cd720 0x7f56d8138b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.721+0000 7f56d77fe700 1 -- 172.21.15.1:0/3107884743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56d80cd720 msgr2=0x7f56d8138b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.721+0000 7f56d77fe700 1 --2- 172.21.15.1:0/3107884743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56d80cd720 0x7f56d8138b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.721+0000 7f56d77fe700 1 -- 172.21.15.1:0/3107884743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56d80b0e40 msgr2=0x7f56d80b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.721+0000 7f56d77fe700 1 --2- 172.21.15.1:0/3107884743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56d80b0e40 0x7f56d80b12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.721+0000 7f56d77fe700 1 -- 172.21.15.1:0/3107884743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d80cdc20 con 0x7f56c0003f70 2024-03-18T18:50:58.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.721+0000 7f56d6ffd700 1 --2- 172.21.15.1:0/3107884743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56d80cd720 0x7f56d8138b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:50:58.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.721+0000 7f56d77fe700 1 --2- 172.21.15.1:0/3107884743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c0003f70 0x7f56d8154040 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00e800 tx=0x7f56cc00eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:58.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.721+0000 7f56bffff700 1 -- 172.21.15.1:0/3107884743 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc0052e0 con 0x7f56c0003f70 2024-03-18T18:50:58.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.721+0000 7f56bffff700 1 -- 172.21.15.1:0/3107884743 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56cc005460 con 0x7f56c0003f70 2024-03-18T18:50:58.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.721+0000 7f56ddcb9700 1 -- 172.21.15.1:0/3107884743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d80cdf00 con 0x7f56c0003f70 2024-03-18T18:50:58.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.721+0000 7f56bffff700 1 -- 172.21.15.1:0/3107884743 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc0038c0 con 0x7f56c0003f70 2024-03-18T18:50:58.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.721+0000 7f56ddcb9700 1 -- 172.21.15.1:0/3107884743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d80ce4c0 con 0x7f56c0003f70 2024-03-18T18:50:58.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.722+0000 7f56ddcb9700 1 -- 172.21.15.1:0/3107884743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d80676a0 con 0x7f56c0003f70 2024-03-18T18:50:58.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.722+0000 7f56bffff700 1 -- 172.21.15.1:0/3107884743 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f56cc003a20 con 0x7f56c0003f70 2024-03-18T18:50:58.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.723+0000 7f56bffff700 1 --2- 172.21.15.1:0/3107884743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56c406ac90 0x7f56c406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:58.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.723+0000 7f56bffff700 1 -- 172.21.15.1:0/3107884743 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f56cc088f60 con 0x7f56c0003f70 2024-03-18T18:50:58.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.723+0000 7f56d6ffd700 1 --2- 172.21.15.1:0/3107884743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56c406ac90 0x7f56c406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:58.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.723+0000 7f56d6ffd700 1 --2- 172.21.15.1:0/3107884743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56c406ac90 0x7f56c406d150 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f56c80090b0 tx=0x7f56c8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:58.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.727+0000 7f56bffff700 1 -- 172.21.15.1:0/3107884743 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f56cc053d80 con 0x7f56c0003f70 2024-03-18T18:50:58.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.928+0000 7f56ddcb9700 1 -- 172.21.15.1:0/3107884743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f56d813f360 con 0x7f56c0003f70 2024-03-18T18:50:58.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.928+0000 7f56bffff700 1 -- 172.21.15.1:0/3107884743 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f56cc019020 con 0x7f56c0003f70 2024-03-18T18:50:58.925 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T18:50:58.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.930+0000 7f56ddcb9700 1 -- 172.21.15.1:0/3107884743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56c406ac90 msgr2=0x7f56c406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.930+0000 7f56ddcb9700 1 --2- 172.21.15.1:0/3107884743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56c406ac90 0x7f56c406d150 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f56c80090b0 tx=0x7f56c8008030 comp rx=0 tx=0).stop 2024-03-18T18:50:58.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.930+0000 7f56ddcb9700 1 -- 172.21.15.1:0/3107884743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c0003f70 msgr2=0x7f56d8154040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:58.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.930+0000 7f56ddcb9700 1 --2- 172.21.15.1:0/3107884743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c0003f70 0x7f56d8154040 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00e800 tx=0x7f56cc00eb10 comp rx=0 tx=0).stop 2024-03-18T18:50:58.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.930+0000 7f56ddcb9700 1 -- 172.21.15.1:0/3107884743 shutdown_connections 2024-03-18T18:50:58.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.930+0000 7f56ddcb9700 1 --2- 172.21.15.1:0/3107884743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56c406ac90 0x7f56c406d150 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.930+0000 7f56ddcb9700 1 --2- 172.21.15.1:0/3107884743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56d80b0e40 0x7f56d80b12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.930+0000 7f56ddcb9700 1 --2- 172.21.15.1:0/3107884743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c0003f70 0x7f56d8154040 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.930+0000 7f56ddcb9700 1 --2- 172.21.15.1:0/3107884743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56d80cd720 0x7f56d8138b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:58.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.930+0000 7f56ddcb9700 1 -- 172.21.15.1:0/3107884743 >> 172.21.15.1:0/3107884743 conn(0x7f56d80b1cb0 msgr2=0x7f56d806f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:58.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.930+0000 7f56ddcb9700 1 -- 172.21.15.1:0/3107884743 shutdown_connections 2024-03-18T18:50:58.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:58.930+0000 7f56ddcb9700 1 -- 172.21.15.1:0/3107884743 wait complete. 2024-03-18T18:50:58.939 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T18:50:59.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.068+0000 7f896e3fa700 1 Processor -- start 2024-03-18T18:50:59.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.068+0000 7f896e3fa700 1 -- start start 2024-03-18T18:50:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.068+0000 7f896e3fa700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8968144b70 0x7f89680ce980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.068+0000 7f896e3fa700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f89680cd110 0x7f8968071a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.068+0000 7f896e3fa700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89680cf2d0 con 0x7f89680af610 2024-03-18T18:50:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.068+0000 7f896e3fa700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8968144f50 con 0x7f8968144b70 2024-03-18T18:50:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.068+0000 7f896e3fa700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89680cf5b0 con 0x7f89680cd110 2024-03-18T18:50:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.069+0000 7f8967fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8968144b70 0x7f89680ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.069+0000 7f8967fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8968144b70 0x7f89680ce980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38574/0 (socket says 172.21.15.1:38574) 2024-03-18T18:50:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.069+0000 7f8967fff700 1 -- 172.21.15.1:0/381691550 learned_addr learned my addr 172.21.15.1:0/381691550 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.069+0000 7f896c997700 1 --2- 172.21.15.1:0/381691550 >> v2:172.21.15.165:3300/0 conn(0x7f89680cd110 0x7f8968071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.069+0000 7f8967fff700 1 -- 172.21.15.1:0/381691550 >> v2:172.21.15.165:3300/0 conn(0x7f89680cd110 msgr2=0x7f8968071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.069+0000 7f8967fff700 1 --2- 172.21.15.1:0/381691550 >> v2:172.21.15.165:3300/0 conn(0x7f89680cd110 0x7f8968071a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.069+0000 7f8967fff700 1 -- 172.21.15.1:0/381691550 >> v1:172.21.15.165:6789/0 conn(0x7f89680af610 legacy=0x7f89680cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.069+0000 7f8967fff700 1 -- 172.21.15.1:0/381691550 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89680cf890 con 0x7f8968144b70 2024-03-18T18:50:59.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.069+0000 7f896c997700 1 --2- 172.21.15.1:0/381691550 >> v2:172.21.15.165:3300/0 conn(0x7f89680cd110 0x7f8968071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:59.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.069+0000 7f8967fff700 1 --2- 172.21.15.1:0/381691550 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8968144b70 0x7f89680ce980 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f89680715e0 tx=0x7f8958009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f011b16072e56fc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:59.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.069+0000 7f8966ffd700 1 -- 172.21.15.1:0/381691550 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8958004230 con 0x7f8968144b70 2024-03-18T18:50:59.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.069+0000 7f8966ffd700 1 -- 172.21.15.1:0/381691550 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8958004390 con 0x7f8968144b70 2024-03-18T18:50:59.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.069+0000 7f8966ffd700 1 -- 172.21.15.1:0/381691550 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89580036a0 con 0x7f8968144b70 2024-03-18T18:50:59.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.070+0000 7f896e3fa700 1 -- 172.21.15.1:0/381691550 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8968144b70 msgr2=0x7f89680ce980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:59.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.070+0000 7f896e3fa700 1 --2- 172.21.15.1:0/381691550 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8968144b70 0x7f89680ce980 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f89680715e0 tx=0x7f8958009bd0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.070+0000 7f896e3fa700 1 -- 172.21.15.1:0/381691550 shutdown_connections 2024-03-18T18:50:59.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.070+0000 7f896e3fa700 1 --2- 172.21.15.1:0/381691550 >> v2:172.21.15.165:3300/0 conn(0x7f89680cd110 0x7f8968071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.070+0000 7f896e3fa700 1 --2- 172.21.15.1:0/381691550 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8968144b70 0x7f89680ce980 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.070+0000 7f896e3fa700 1 -- 172.21.15.1:0/381691550 >> 172.21.15.1:0/381691550 conn(0x7f89680b1d20 msgr2=0x7f89680b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:59.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.070+0000 7f896e3fa700 1 -- 172.21.15.1:0/381691550 shutdown_connections 2024-03-18T18:50:59.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.070+0000 7f896e3fa700 1 -- 172.21.15.1:0/381691550 wait complete. 2024-03-18T18:50:59.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.071+0000 7f896e3fa700 1 Processor -- start 2024-03-18T18:50:59.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.071+0000 7f896e3fa700 1 -- start start 2024-03-18T18:50:59.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.072+0000 7f896e3fa700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89680af610 0x7f896812fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.072+0000 7f896e3fa700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89680cd110 0x7f89681e4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.072+0000 7f896e3fa700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8968144b70 0x7f896813fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.072+0000 7f896e3fa700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89681e6040 con 0x7f8968144b70 2024-03-18T18:50:59.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.072+0000 7f896e3fa700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896812ff90 con 0x7f89680af610 2024-03-18T18:50:59.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.072+0000 7f896e3fa700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89681e53c0 con 0x7f89680cd110 2024-03-18T18:50:59.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.072+0000 7f896c997700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8968144b70 0x7f896813fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.072+0000 7f896c997700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8968144b70 0x7f896813fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38578/0 (socket says 172.21.15.1:38578) 2024-03-18T18:50:59.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.072+0000 7f896c997700 1 -- 172.21.15.1:0/558864357 learned_addr learned my addr 172.21.15.1:0/558864357 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:59.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.072+0000 7f8967fff700 1 --2- 172.21.15.1:0/558864357 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89680af610 0x7f896812fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.072+0000 7f89677fe700 1 --2- 172.21.15.1:0/558864357 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89680cd110 0x7f89681e4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.072+0000 7f896c997700 1 -- 172.21.15.1:0/558864357 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89680cd110 msgr2=0x7f89681e4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:59.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.072+0000 7f896c997700 1 --2- 172.21.15.1:0/558864357 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89680cd110 0x7f89681e4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.072+0000 7f896c997700 1 -- 172.21.15.1:0/558864357 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89680af610 msgr2=0x7f896812fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:59.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.072+0000 7f896c997700 1 --2- 172.21.15.1:0/558864357 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89680af610 0x7f896812fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.072+0000 7f896c997700 1 -- 172.21.15.1:0/558864357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8958009550 con 0x7f8968144b70 2024-03-18T18:50:59.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.073+0000 7f896c997700 1 --2- 172.21.15.1:0/558864357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8968144b70 0x7f896813fa10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f896000ce80 tx=0x7f896000ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:59.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.073+0000 7f8964ff9700 1 -- 172.21.15.1:0/558864357 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896000ad70 con 0x7f8968144b70 2024-03-18T18:50:59.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.073+0000 7f8964ff9700 1 -- 172.21.15.1:0/558864357 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8960015440 con 0x7f8968144b70 2024-03-18T18:50:59.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.073+0000 7f896e3fa700 1 -- 172.21.15.1:0/558864357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f896806ecb0 con 0x7f8968144b70 2024-03-18T18:50:59.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.073+0000 7f8964ff9700 1 -- 172.21.15.1:0/558864357 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89600157f0 con 0x7f8968144b70 2024-03-18T18:50:59.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.073+0000 7f896e3fa700 1 -- 172.21.15.1:0/558864357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89681e5980 con 0x7f8968144b70 2024-03-18T18:50:59.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.074+0000 7f896e3fa700 1 -- 172.21.15.1:0/558864357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89680cda90 con 0x7f8968144b70 2024-03-18T18:50:59.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.074+0000 7f8964ff9700 1 -- 172.21.15.1:0/558864357 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f896000d430 con 0x7f8968144b70 2024-03-18T18:50:59.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.075+0000 7f8964ff9700 1 --2- 172.21.15.1:0/558864357 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f895406ac90 0x7f895406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.075+0000 7f8964ff9700 1 -- 172.21.15.1:0/558864357 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8960089b90 con 0x7f8968144b70 2024-03-18T18:50:59.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.075+0000 7f8967fff700 1 --2- 172.21.15.1:0/558864357 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f895406ac90 0x7f895406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.075+0000 7f8967fff700 1 --2- 172.21.15.1:0/558864357 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f895406ac90 0x7f895406d150 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8958009130 tx=0x7f8958008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:59.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.080+0000 7f8964ff9700 1 -- 172.21.15.1:0/558864357 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8960054a40 con 0x7f8968144b70 2024-03-18T18:50:59.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.329+0000 7f896e3fa700 1 -- 172.21.15.1:0/558864357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89680676a0 con 0x7f8968144b70 2024-03-18T18:50:59.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.329+0000 7f8964ff9700 1 -- 172.21.15.1:0/558864357 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4727 (secure 0 0 0) 0x7f896001a020 con 0x7f8968144b70 2024-03-18T18:50:59.326 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:59.326 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/883056851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":883056851},{"type":"v1","addr":"172.21.15.165:6834","nonce":883056851}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1388214718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1388214718},{"type":"v1","addr":"172.21.15.165:6835","nonce":1388214718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3020786191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3020786191},{"type":"v1","addr":"172.21.15.1:6836","nonce":3020786191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T18:49:44.809495+0000","modified":"2024-03-18T18:50:07.888953+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2118908012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2118908012},{"type":"v1","addr":"172.21.15.1:6837","nonce":2118908012}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-18T18:50:59.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.331+0000 7f896e3fa700 1 -- 172.21.15.1:0/558864357 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f895406ac90 msgr2=0x7f895406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:59.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.331+0000 7f896e3fa700 1 --2- 172.21.15.1:0/558864357 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f895406ac90 0x7f895406d150 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8958009130 tx=0x7f8958008030 comp rx=0 tx=0).stop 2024-03-18T18:50:59.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.331+0000 7f896e3fa700 1 -- 172.21.15.1:0/558864357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8968144b70 msgr2=0x7f896813fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:59.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.331+0000 7f896e3fa700 1 --2- 172.21.15.1:0/558864357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8968144b70 0x7f896813fa10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f896000ce80 tx=0x7f896000ceb0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.331+0000 7f896e3fa700 1 -- 172.21.15.1:0/558864357 shutdown_connections 2024-03-18T18:50:59.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.331+0000 7f896e3fa700 1 --2- 172.21.15.1:0/558864357 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f895406ac90 0x7f895406d150 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.331+0000 7f896e3fa700 1 --2- 172.21.15.1:0/558864357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8968144b70 0x7f896813fa10 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.331+0000 7f896e3fa700 1 --2- 172.21.15.1:0/558864357 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89680af610 0x7f896812fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.331+0000 7f896e3fa700 1 --2- 172.21.15.1:0/558864357 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89680cd110 0x7f89681e4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.331+0000 7f896e3fa700 1 -- 172.21.15.1:0/558864357 >> 172.21.15.1:0/558864357 conn(0x7f89680b1d20 msgr2=0x7f8968141060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:59.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.332+0000 7f896e3fa700 1 -- 172.21.15.1:0/558864357 shutdown_connections 2024-03-18T18:50:59.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.332+0000 7f896e3fa700 1 -- 172.21.15.1:0/558864357 wait complete. 2024-03-18T18:50:59.329 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 9 2024-03-18T18:50:59.340 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:50:59.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.469+0000 7f7ea8ae8700 1 Processor -- start 2024-03-18T18:50:59.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.469+0000 7f7ea8ae8700 1 -- start start 2024-03-18T18:50:59.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.470+0000 7f7ea8ae8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7ea406dcb0 0x7f7ea40ca820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.470+0000 7f7ea8ae8700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea406e160 con 0x7f7ea415cfc0 2024-03-18T18:50:59.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.470+0000 7f7ea8ae8700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea406e9c0 con 0x7f7ea40cc600 2024-03-18T18:50:59.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.470+0000 7f7ea8ae8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea40cd640 con 0x7f7ea406dcb0 2024-03-18T18:50:59.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.470+0000 7f7ea259c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7ea415cfc0 0x7f7ea40cbef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33020/0 (socket says 172.21.15.1:33020) 2024-03-18T18:50:59.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.470+0000 7f7ea259c700 1 -- 172.21.15.1:0/788807116 learned_addr learned my addr 172.21.15.1:0/788807116 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:59.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.470+0000 7f7ea159a700 1 -- 172.21.15.1:0/788807116 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058868386 0 0) 0x7f7ea406e160 con 0x7f7ea415cfc0 2024-03-18T18:50:59.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.470+0000 7f7ea159a700 1 -- 172.21.15.1:0/788807116 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e8c0035b0 con 0x7f7ea415cfc0 2024-03-18T18:50:59.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.470+0000 7f7ea2d9d700 1 --2- 172.21.15.1:0/788807116 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7ea406dcb0 0x7f7ea40ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea159a700 1 -- 172.21.15.1:0/788807116 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1624987016 0 0) 0x7f7e8c0035b0 con 0x7f7ea415cfc0 2024-03-18T18:50:59.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea159a700 1 -- 172.21.15.1:0/788807116 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7ea406dcb0 msgr2=0x7f7ea40ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:59.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea159a700 1 --2- 172.21.15.1:0/788807116 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7ea406dcb0 0x7f7ea40ca820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea159a700 1 -- 172.21.15.1:0/788807116 >> v1:172.21.15.165:6789/0 conn(0x7f7ea40cc600 legacy=0x7f7ea40ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:59.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea159a700 1 -- 172.21.15.1:0/788807116 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea40cd920 con 0x7f7ea415cfc0 2024-03-18T18:50:59.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea159a700 1 -- 172.21.15.1:0/788807116 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f7e98001670 con 0x7f7ea415cfc0 2024-03-18T18:50:59.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea159a700 1 -- 172.21.15.1:0/788807116 >> v1:172.21.15.1:6789/0 conn(0x7f7ea415cfc0 legacy=0x7f7ea40cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:59.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea159a700 1 --2- 172.21.15.1:0/788807116 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7ea406dcb0 0x7f7e8c003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea159a700 1 --2- 172.21.15.1:0/788807116 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e8c003f70 0x7f7e8c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea159a700 1 --2- 172.21.15.1:0/788807116 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e8c004d90 0x7f7e8c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea159a700 1 -- 172.21.15.1:0/788807116 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea40cd640 con 0x7f7e8c003f70 2024-03-18T18:50:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea159a700 1 -- 172.21.15.1:0/788807116 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c00ba70 con 0x7f7e8c004d90 2024-03-18T18:50:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea159a700 1 -- 172.21.15.1:0/788807116 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c00bc30 con 0x7f7ea406dcb0 2024-03-18T18:50:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea1d9b700 1 --2- 172.21.15.1:0/788807116 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e8c003f70 0x7f7e8c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea259c700 1 --2- 172.21.15.1:0/788807116 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7ea406dcb0 0x7f7e8c003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea2d9d700 1 --2- 172.21.15.1:0/788807116 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e8c004d90 0x7f7e8c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea1d9b700 1 -- 172.21.15.1:0/788807116 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7ea406dcb0 msgr2=0x7f7e8c003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea1d9b700 1 --2- 172.21.15.1:0/788807116 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7ea406dcb0 0x7f7e8c003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea1d9b700 1 -- 172.21.15.1:0/788807116 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e8c004d90 msgr2=0x7f7e8c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea1d9b700 1 --2- 172.21.15.1:0/788807116 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e8c004d90 0x7f7e8c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea1d9b700 1 -- 172.21.15.1:0/788807116 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8c0035b0 con 0x7f7e8c003f70 2024-03-18T18:50:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.471+0000 7f7ea1d9b700 1 --2- 172.21.15.1:0/788807116 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e8c003f70 0x7f7e8c004850 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7e940018b0 tx=0x7f7e8c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daeae93dfd4596f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.472+0000 7f7ea159a700 1 -- 172.21.15.1:0/788807116 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e94001370 con 0x7f7e8c003f70 2024-03-18T18:50:59.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.472+0000 7f7ea159a700 1 -- 172.21.15.1:0/788807116 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e94012640 con 0x7f7e8c003f70 2024-03-18T18:50:59.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.472+0000 7f7ea159a700 1 -- 172.21.15.1:0/788807116 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e940129f0 con 0x7f7e8c003f70 2024-03-18T18:50:59.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.472+0000 7f7ea8ae8700 1 -- 172.21.15.1:0/788807116 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e8c003f70 msgr2=0x7f7e8c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:59.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.472+0000 7f7ea8ae8700 1 --2- 172.21.15.1:0/788807116 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e8c003f70 0x7f7e8c004850 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7e940018b0 tx=0x7f7e8c005880 comp rx=0 tx=0).stop 2024-03-18T18:50:59.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.472+0000 7f7ea2d9d700 1 -- 172.21.15.1:0/788807116 reap_dead start 2024-03-18T18:50:59.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.472+0000 7f7ea8ae8700 1 -- 172.21.15.1:0/788807116 shutdown_connections 2024-03-18T18:50:59.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.472+0000 7f7ea8ae8700 1 -- 172.21.15.1:0/788807116 >> 172.21.15.1:0/788807116 conn(0x7f7ea40b1270 msgr2=0x7f7ea40b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:59.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.472+0000 7f7ea8ae8700 1 -- 172.21.15.1:0/788807116 shutdown_connections 2024-03-18T18:50:59.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.472+0000 7f7ea8ae8700 1 -- 172.21.15.1:0/788807116 wait complete. 2024-03-18T18:50:59.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.474+0000 7f7ea8ae8700 1 Processor -- start 2024-03-18T18:50:59.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.474+0000 7f7ea8ae8700 1 -- start start 2024-03-18T18:50:59.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.474+0000 7f7ea8ae8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e8c003f70 0x7f7ea4146cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.474+0000 7f7ea8ae8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7ea4153e40 0x7f7ea40c4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.474+0000 7f7ea8ae8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7ea40c8e90 0x7f7ea40c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.474+0000 7f7ea8ae8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea40c9880 con 0x7f7e8c003f70 2024-03-18T18:50:59.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.474+0000 7f7ea8ae8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea4147200 con 0x7f7ea40c8e90 2024-03-18T18:50:59.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.474+0000 7f7ea8ae8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea41ee310 con 0x7f7ea4153e40 2024-03-18T18:50:59.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.474+0000 7f7ea259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e8c003f70 0x7f7ea4146cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.474+0000 7f7ea259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e8c003f70 0x7f7ea4146cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38604/0 (socket says 172.21.15.1:38604) 2024-03-18T18:50:59.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.474+0000 7f7ea259c700 1 -- 172.21.15.1:0/3268437610 learned_addr learned my addr 172.21.15.1:0/3268437610 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:59.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.475+0000 7f7ea259c700 1 -- 172.21.15.1:0/3268437610 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7ea4153e40 msgr2=0x7f7ea40c4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:59.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.475+0000 7f7ea1d9b700 1 --2- 172.21.15.1:0/3268437610 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7ea4153e40 0x7f7ea40c4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.475+0000 7f7ea2d9d700 1 --2- 172.21.15.1:0/3268437610 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7ea40c8e90 0x7f7ea40c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.475+0000 7f7ea259c700 1 --2- 172.21.15.1:0/3268437610 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7ea4153e40 0x7f7ea40c4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.475+0000 7f7ea259c700 1 -- 172.21.15.1:0/3268437610 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7ea40c8e90 msgr2=0x7f7ea40c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:59.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.475+0000 7f7ea259c700 1 --2- 172.21.15.1:0/3268437610 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7ea40c8e90 0x7f7ea40c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.475+0000 7f7ea259c700 1 -- 172.21.15.1:0/3268437610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea41efae0 con 0x7f7e8c003f70 2024-03-18T18:50:59.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.475+0000 7f7ea2d9d700 1 --2- 172.21.15.1:0/3268437610 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7ea40c8e90 0x7f7ea40c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:50:59.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.475+0000 7f7ea259c700 1 --2- 172.21.15.1:0/3268437610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e8c003f70 0x7f7ea4146cc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7e980017a0 tx=0x7f7e98007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:59.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.475+0000 7f7e92ffd700 1 -- 172.21.15.1:0/3268437610 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9800f660 con 0x7f7e8c003f70 2024-03-18T18:50:59.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.475+0000 7f7e92ffd700 1 -- 172.21.15.1:0/3268437610 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9800f7c0 con 0x7f7e8c003f70 2024-03-18T18:50:59.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.475+0000 7f7ea8ae8700 1 -- 172.21.15.1:0/3268437610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea41ed700 con 0x7f7e8c003f70 2024-03-18T18:50:59.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.475+0000 7f7e92ffd700 1 -- 172.21.15.1:0/3268437610 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9800fb70 con 0x7f7e8c003f70 2024-03-18T18:50:59.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.475+0000 7f7ea8ae8700 1 -- 172.21.15.1:0/3268437610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea41edcc0 con 0x7f7e8c003f70 2024-03-18T18:50:59.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.476+0000 7f7ea8ae8700 1 -- 172.21.15.1:0/3268437610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea40676a0 con 0x7f7e8c003f70 2024-03-18T18:50:59.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.477+0000 7f7e92ffd700 1 -- 172.21.15.1:0/3268437610 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7e9800fcd0 con 0x7f7e8c003f70 2024-03-18T18:50:59.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.477+0000 7f7e92ffd700 1 --2- 172.21.15.1:0/3268437610 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7e8406ac90 0x7f7e8406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.477+0000 7f7e92ffd700 1 -- 172.21.15.1:0/3268437610 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7e98089780 con 0x7f7e8c003f70 2024-03-18T18:50:59.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.477+0000 7f7ea1d9b700 1 --2- 172.21.15.1:0/3268437610 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7e8406ac90 0x7f7e8406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.478+0000 7f7ea1d9b700 1 --2- 172.21.15.1:0/3268437610 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7e8406ac90 0x7f7e8406d150 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7e94014040 tx=0x7f7e94005040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:59.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.482+0000 7f7e92ffd700 1 -- 172.21.15.1:0/3268437610 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7e98054510 con 0x7f7e8c003f70 2024-03-18T18:50:59.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.683+0000 7f7ea8ae8700 1 -- 172.21.15.1:0/3268437610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ea4071dd0 con 0x7f7e8c003f70 2024-03-18T18:50:59.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.684+0000 7f7e92ffd700 1 -- 172.21.15.1:0/3268437610 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18217 (secure 0 0 0) 0x7f7e98057b30 con 0x7f7e8c003f70 2024-03-18T18:50:59.681 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:50:59.681 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":27,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:50:34.993482+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:50:59.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.686+0000 7f7ea8ae8700 1 -- 172.21.15.1:0/3268437610 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7e8406ac90 msgr2=0x7f7e8406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:59.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.686+0000 7f7ea8ae8700 1 --2- 172.21.15.1:0/3268437610 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7e8406ac90 0x7f7e8406d150 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7e94014040 tx=0x7f7e94005040 comp rx=0 tx=0).stop 2024-03-18T18:50:59.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.686+0000 7f7ea8ae8700 1 -- 172.21.15.1:0/3268437610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e8c003f70 msgr2=0x7f7ea4146cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:59.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.686+0000 7f7ea8ae8700 1 --2- 172.21.15.1:0/3268437610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e8c003f70 0x7f7ea4146cc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7e980017a0 tx=0x7f7e98007620 comp rx=0 tx=0).stop 2024-03-18T18:50:59.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.687+0000 7f7ea8ae8700 1 -- 172.21.15.1:0/3268437610 shutdown_connections 2024-03-18T18:50:59.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.687+0000 7f7ea8ae8700 1 --2- 172.21.15.1:0/3268437610 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7e8406ac90 0x7f7e8406d150 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.687+0000 7f7ea8ae8700 1 --2- 172.21.15.1:0/3268437610 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7ea40c8e90 0x7f7ea40c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.687+0000 7f7ea8ae8700 1 --2- 172.21.15.1:0/3268437610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e8c003f70 0x7f7ea4146cc0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.687+0000 7f7ea8ae8700 1 --2- 172.21.15.1:0/3268437610 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7ea4153e40 0x7f7ea40c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.687+0000 7f7ea8ae8700 1 -- 172.21.15.1:0/3268437610 >> 172.21.15.1:0/3268437610 conn(0x7f7ea40b1270 msgr2=0x7f7ea40b0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:59.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.687+0000 7f7ea8ae8700 1 -- 172.21.15.1:0/3268437610 shutdown_connections 2024-03-18T18:50:59.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.687+0000 7f7ea8ae8700 1 -- 172.21.15.1:0/3268437610 wait complete. 2024-03-18T18:50:59.695 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T18:50:59.696 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T18:50:59.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.824+0000 7f03c4c0f700 1 Processor -- start 2024-03-18T18:50:59.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.824+0000 7f03c4c0f700 1 -- start start 2024-03-18T18:50:59.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.825+0000 7f03c4c0f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f03c015a320 0x7f03c00ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.825+0000 7f03c4c0f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03c0071d10 0x7f03c006e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.825+0000 7f03c4c0f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c00acd80 con 0x7f03c00cea30 2024-03-18T18:50:59.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.825+0000 7f03c4c0f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c00cf220 con 0x7f03c015a320 2024-03-18T18:50:59.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.825+0000 7f03c4c0f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c00cef40 con 0x7f03c0071d10 2024-03-18T18:50:59.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.825+0000 7f03be59c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f03c015a320 0x7f03c00ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.825+0000 7f03bed9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03c0071d10 0x7f03c006e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.825+0000 7f03bdd9b700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f03c00cea30 0x7f03c0071600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:36086/0 (socket says 172.21.15.1:36086) 2024-03-18T18:50:59.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.825+0000 7f03bdd9b700 1 -- 172.21.15.1:0/3542285013 learned_addr learned my addr 172.21.15.1:0/3542285013 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:50:59.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.826+0000 7f03be59c700 1 -- 172.21.15.1:0/3542285013 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03c0071d10 msgr2=0x7f03c006e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:50:59.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.826+0000 7f03be59c700 1 --2- 172.21.15.1:0/3542285013 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03c0071d10 0x7f03c006e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.826+0000 7f03be59c700 1 -- 172.21.15.1:0/3542285013 >> v1:172.21.15.165:6789/0 conn(0x7f03c00cea30 legacy=0x7f03c0071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:59.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.826+0000 7f03be59c700 1 -- 172.21.15.1:0/3542285013 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c006d7e0 con 0x7f03c015a320 2024-03-18T18:50:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.826+0000 7f03bd59a700 1 -- 172.21.15.1:0/3542285013 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211535980 0 0) 0x7f03c00acd80 con 0x7f03c00cea30 2024-03-18T18:50:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.826+0000 7f03be59c700 1 --2- 172.21.15.1:0/3542285013 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03c015a320 0x7f03c00ce4f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f03b4009840 tx=0x7f03b4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=238ab528c0ccbef6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.826+0000 7f03bd59a700 1 -- 172.21.15.1:0/3542285013 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b4011830 con 0x7f03c015a320 2024-03-18T18:50:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.826+0000 7f03bd59a700 1 -- 172.21.15.1:0/3542285013 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b4011990 con 0x7f03c015a320 2024-03-18T18:50:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.826+0000 7f03bd59a700 1 -- 172.21.15.1:0/3542285013 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b4011d20 con 0x7f03c015a320 2024-03-18T18:50:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.827+0000 7f03c4c0f700 1 -- 172.21.15.1:0/3542285013 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03c015a320 msgr2=0x7f03c00ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.827+0000 7f03c4c0f700 1 --2- 172.21.15.1:0/3542285013 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03c015a320 0x7f03c00ce4f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f03b4009840 tx=0x7f03b4009b50 comp rx=0 tx=0).stop 2024-03-18T18:50:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.827+0000 7f03c4c0f700 1 -- 172.21.15.1:0/3542285013 shutdown_connections 2024-03-18T18:50:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.827+0000 7f03c4c0f700 1 --2- 172.21.15.1:0/3542285013 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03c0071d10 0x7f03c006e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.827+0000 7f03c4c0f700 1 --2- 172.21.15.1:0/3542285013 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03c015a320 0x7f03c00ce4f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.827+0000 7f03c4c0f700 1 -- 172.21.15.1:0/3542285013 >> 172.21.15.1:0/3542285013 conn(0x7f03c00cc990 msgr2=0x7f03c00ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:50:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.827+0000 7f03c4c0f700 1 -- 172.21.15.1:0/3542285013 shutdown_connections 2024-03-18T18:50:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.827+0000 7f03c4c0f700 1 -- 172.21.15.1:0/3542285013 wait complete. 2024-03-18T18:50:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.828+0000 7f03c4c0f700 1 Processor -- start 2024-03-18T18:50:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.828+0000 7f03c4c0f700 1 -- start start 2024-03-18T18:50:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.829+0000 7f03c4c0f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03c0071d10 0x7f03c00c4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.829+0000 7f03c4c0f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03c00cea30 0x7f03c01534e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.829+0000 7f03c4c0f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c00b3da0 0x7f03c00cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.829+0000 7f03c4c0f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c00b42f0 con 0x7f03c00b3da0 2024-03-18T18:50:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.829+0000 7f03c4c0f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c00b45d0 con 0x7f03c00cea30 2024-03-18T18:50:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.829+0000 7f03c4c0f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c00b48b0 con 0x7f03c0071d10 2024-03-18T18:50:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.829+0000 7f03bed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c00b3da0 0x7f03c00cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.829+0000 7f03bed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c00b3da0 0x7f03c00cacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38612/0 (socket says 172.21.15.1:38612) 2024-03-18T18:50:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.829+0000 7f03bed9d700 1 -- 172.21.15.1:0/4248289883 learned_addr learned my addr 172.21.15.1:0/4248289883 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:50:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.829+0000 7f03bdd9b700 1 --2- 172.21.15.1:0/4248289883 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03c00cea30 0x7f03c01534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.829+0000 7f03be59c700 1 --2- 172.21.15.1:0/4248289883 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03c0071d10 0x7f03c00c4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.829+0000 7f03bed9d700 1 -- 172.21.15.1:0/4248289883 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03c0071d10 msgr2=0x7f03c00c4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.829+0000 7f03bed9d700 1 --2- 172.21.15.1:0/4248289883 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03c0071d10 0x7f03c00c4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.829+0000 7f03bed9d700 1 -- 172.21.15.1:0/4248289883 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03c00cea30 msgr2=0x7f03c01534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:50:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.829+0000 7f03bed9d700 1 --2- 172.21.15.1:0/4248289883 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03c00cea30 0x7f03c01534e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:50:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.829+0000 7f03bed9d700 1 -- 172.21.15.1:0/4248289883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b40094a0 con 0x7f03c00b3da0 2024-03-18T18:50:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.830+0000 7f03bed9d700 1 --2- 172.21.15.1:0/4248289883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c00b3da0 0x7f03c00cacc0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f03b800efc0 tx=0x7f03b800c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.830+0000 7f03aeffd700 1 -- 172.21.15.1:0/4248289883 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b800cad0 con 0x7f03c00b3da0 2024-03-18T18:50:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.830+0000 7f03aeffd700 1 -- 172.21.15.1:0/4248289883 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b800f040 con 0x7f03c00b3da0 2024-03-18T18:50:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.830+0000 7f03c4c0f700 1 -- 172.21.15.1:0/4248289883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c00b4b90 con 0x7f03c00b3da0 2024-03-18T18:50:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.830+0000 7f03aeffd700 1 -- 172.21.15.1:0/4248289883 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b800ce70 con 0x7f03c00b3da0 2024-03-18T18:50:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.830+0000 7f03c4c0f700 1 -- 172.21.15.1:0/4248289883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c00c3d10 con 0x7f03c00b3da0 2024-03-18T18:50:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.831+0000 7f03c4c0f700 1 -- 172.21.15.1:0/4248289883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c00676a0 con 0x7f03c00b3da0 2024-03-18T18:50:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.831+0000 7f03aeffd700 1 -- 172.21.15.1:0/4248289883 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f03b8014480 con 0x7f03c00b3da0 2024-03-18T18:50:59.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.832+0000 7f03aeffd700 1 --2- 172.21.15.1:0/4248289883 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f03a006ac90 0x7f03a006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:50:59.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.832+0000 7f03aeffd700 1 -- 172.21.15.1:0/4248289883 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f03b8089e10 con 0x7f03c00b3da0 2024-03-18T18:50:59.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.832+0000 7f03be59c700 1 --2- 172.21.15.1:0/4248289883 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f03a006ac90 0x7f03a006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:50:59.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.832+0000 7f03be59c700 1 --2- 172.21.15.1:0/4248289883 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f03a006ac90 0x7f03a006d150 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f03b40090b0 tx=0x7f03b4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:50:59.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:50:59.836+0000 7f03aeffd700 1 -- 172.21.15.1:0/4248289883 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f03b8054c40 con 0x7f03c00b3da0 2024-03-18T18:51:00.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.082+0000 7f03c4c0f700 1 -- 172.21.15.1:0/4248289883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03c0068a00 con 0x7f03c00b3da0 2024-03-18T18:51:00.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.083+0000 7f03aeffd700 1 -- 172.21.15.1:0/4248289883 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4727 (secure 0 0 0) 0x7f03b8058260 con 0x7f03c00b3da0 2024-03-18T18:51:00.079 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:51:00.079 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/883056851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":883056851},{"type":"v1","addr":"172.21.15.165:6834","nonce":883056851}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1388214718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1388214718},{"type":"v1","addr":"172.21.15.165:6835","nonce":1388214718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3020786191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3020786191},{"type":"v1","addr":"172.21.15.1:6836","nonce":3020786191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T18:49:44.809495+0000","modified":"2024-03-18T18:50:07.888953+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2118908012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2118908012},{"type":"v1","addr":"172.21.15.1:6837","nonce":2118908012}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-18T18:51:00.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.085+0000 7f03c4c0f700 1 -- 172.21.15.1:0/4248289883 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f03a006ac90 msgr2=0x7f03a006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.086+0000 7f03c4c0f700 1 --2- 172.21.15.1:0/4248289883 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f03a006ac90 0x7f03a006d150 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f03b40090b0 tx=0x7f03b4008030 comp rx=0 tx=0).stop 2024-03-18T18:51:00.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.086+0000 7f03c4c0f700 1 -- 172.21.15.1:0/4248289883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c00b3da0 msgr2=0x7f03c00cacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.086+0000 7f03c4c0f700 1 --2- 172.21.15.1:0/4248289883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c00b3da0 0x7f03c00cacc0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f03b800efc0 tx=0x7f03b800c0c0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.086+0000 7f03c4c0f700 1 -- 172.21.15.1:0/4248289883 shutdown_connections 2024-03-18T18:51:00.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.086+0000 7f03c4c0f700 1 --2- 172.21.15.1:0/4248289883 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f03a006ac90 0x7f03a006d150 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.086+0000 7f03c4c0f700 1 --2- 172.21.15.1:0/4248289883 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03c0071d10 0x7f03c00c4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.086+0000 7f03c4c0f700 1 --2- 172.21.15.1:0/4248289883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03c00b3da0 0x7f03c00cacc0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.086+0000 7f03c4c0f700 1 --2- 172.21.15.1:0/4248289883 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03c00cea30 0x7f03c01534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.086+0000 7f03c4c0f700 1 -- 172.21.15.1:0/4248289883 >> 172.21.15.1:0/4248289883 conn(0x7f03c00cc990 msgr2=0x7f03c00afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:00.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.086+0000 7f03c4c0f700 1 -- 172.21.15.1:0/4248289883 shutdown_connections 2024-03-18T18:51:00.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.086+0000 7f03c4c0f700 1 -- 172.21.15.1:0/4248289883 wait complete. 2024-03-18T18:51:00.083 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 9 2024-03-18T18:51:00.094 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T18:51:00.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.222+0000 7fe9d7a42700 1 Processor -- start 2024-03-18T18:51:00.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.223+0000 7fe9d7a42700 1 -- start start 2024-03-18T18:51:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.223+0000 7fe9d7a42700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe9d015b4d0 0x7fe9d009a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.223+0000 7fe9d7a42700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe9d009a150 0x7fe9d006f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:00.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.223+0000 7fe9d7a42700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d006f070 con 0x7fe9d00cf750 2024-03-18T18:51:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.223+0000 7fe9d7a42700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d00d0260 con 0x7fe9d009a150 2024-03-18T18:51:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.223+0000 7fe9d7a42700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d00cff80 con 0x7fe9d015b4d0 2024-03-18T18:51:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.224+0000 7fe9d5fdf700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe9d009a150 0x7fe9d006f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.224+0000 7fe9d5fdf700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe9d009a150 0x7fe9d006f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38622/0 (socket says 172.21.15.1:38622) 2024-03-18T18:51:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.224+0000 7fe9d5fdf700 1 -- 172.21.15.1:0/9415393 learned_addr learned my addr 172.21.15.1:0/9415393 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.224+0000 7fe9d57de700 1 --2- 172.21.15.1:0/9415393 >> v2:172.21.15.165:3300/0 conn(0x7fe9d015b4d0 0x7fe9d009a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.224+0000 7fe9d5fdf700 1 -- 172.21.15.1:0/9415393 >> v2:172.21.15.165:3300/0 conn(0x7fe9d015b4d0 msgr2=0x7fe9d009a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.224+0000 7fe9d5fdf700 1 --2- 172.21.15.1:0/9415393 >> v2:172.21.15.165:3300/0 conn(0x7fe9d015b4d0 0x7fe9d009a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.224+0000 7fe9d5fdf700 1 -- 172.21.15.1:0/9415393 >> v1:172.21.15.165:6789/0 conn(0x7fe9d00cf750 legacy=0x7fe9d0099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.224+0000 7fe9d5fdf700 1 -- 172.21.15.1:0/9415393 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d006ea20 con 0x7fe9d009a150 2024-03-18T18:51:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.224+0000 7fe9d57de700 1 --2- 172.21.15.1:0/9415393 >> v2:172.21.15.165:3300/0 conn(0x7fe9d015b4d0 0x7fe9d009a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.224+0000 7fe9d5fdf700 1 --2- 172.21.15.1:0/9415393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9d009a150 0x7fe9d006f500 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0071290 tx=0x7fe9c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea70ba40f3e8dc67 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:00.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.224+0000 7fe9c3fff700 1 -- 172.21.15.1:0/9415393 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c800e070 con 0x7fe9d009a150 2024-03-18T18:51:00.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.224+0000 7fe9c3fff700 1 -- 172.21.15.1:0/9415393 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c800b9c0 con 0x7fe9d009a150 2024-03-18T18:51:00.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.224+0000 7fe9c3fff700 1 -- 172.21.15.1:0/9415393 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c800bd70 con 0x7fe9d009a150 2024-03-18T18:51:00.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.225+0000 7fe9d7a42700 1 -- 172.21.15.1:0/9415393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9d009a150 msgr2=0x7fe9d006f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.225+0000 7fe9d7a42700 1 --2- 172.21.15.1:0/9415393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9d009a150 0x7fe9d006f500 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0071290 tx=0x7fe9c800b330 comp rx=0 tx=0).stop 2024-03-18T18:51:00.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.225+0000 7fe9d7a42700 1 -- 172.21.15.1:0/9415393 shutdown_connections 2024-03-18T18:51:00.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.225+0000 7fe9d7a42700 1 --2- 172.21.15.1:0/9415393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9d009a150 0x7fe9d006f500 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.225+0000 7fe9d7a42700 1 --2- 172.21.15.1:0/9415393 >> v2:172.21.15.165:3300/0 conn(0x7fe9d015b4d0 0x7fe9d009a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.225+0000 7fe9d7a42700 1 -- 172.21.15.1:0/9415393 >> 172.21.15.1:0/9415393 conn(0x7fe9d00b2cb0 msgr2=0x7fe9d00b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:00.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.225+0000 7fe9d7a42700 1 -- 172.21.15.1:0/9415393 shutdown_connections 2024-03-18T18:51:00.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.225+0000 7fe9d7a42700 1 -- 172.21.15.1:0/9415393 wait complete. 2024-03-18T18:51:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.227+0000 7fe9d7a42700 1 Processor -- start 2024-03-18T18:51:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.227+0000 7fe9d7a42700 1 -- start start 2024-03-18T18:51:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d7a42700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9d009a150 0x7fe9d015a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d7a42700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe9d00cf750 0x7fe9d00c5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d7a42700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe9d015b4d0 0x7fe9d00c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d7a42700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d00c5fa0 con 0x7fe9d009a150 2024-03-18T18:51:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d7a42700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d01f3260 con 0x7fe9d00cf750 2024-03-18T18:51:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d7a42700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d01f3540 con 0x7fe9d015b4d0 2024-03-18T18:51:00.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d57de700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9d009a150 0x7fe9d015a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:00.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d57de700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9d009a150 0x7fe9d015a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38624/0 (socket says 172.21.15.1:38624) 2024-03-18T18:51:00.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d57de700 1 -- 172.21.15.1:0/3232617101 learned_addr learned my addr 172.21.15.1:0/3232617101 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:00.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d5fdf700 1 --2- 172.21.15.1:0/3232617101 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe9d015b4d0 0x7fe9d00c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:00.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d4fdd700 1 --2- 172.21.15.1:0/3232617101 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe9d00cf750 0x7fe9d00c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:00.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d57de700 1 -- 172.21.15.1:0/3232617101 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe9d015b4d0 msgr2=0x7fe9d00c75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d57de700 1 --2- 172.21.15.1:0/3232617101 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe9d015b4d0 0x7fe9d00c75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d57de700 1 -- 172.21.15.1:0/3232617101 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe9d00cf750 msgr2=0x7fe9d00c5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d57de700 1 --2- 172.21.15.1:0/3232617101 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe9d00cf750 0x7fe9d00c5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d57de700 1 -- 172.21.15.1:0/3232617101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c8009b30 con 0x7fe9d009a150 2024-03-18T18:51:00.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d5fdf700 1 --2- 172.21.15.1:0/3232617101 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe9d015b4d0 0x7fe9d00c75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:00.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d4fdd700 1 --2- 172.21.15.1:0/3232617101 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe9d00cf750 0x7fe9d00c5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:00.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d57de700 1 --2- 172.21.15.1:0/3232617101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9d009a150 0x7fe9d015a530 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc00e810 tx=0x7fe9cc00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:00.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9c1ffb700 1 -- 172.21.15.1:0/3232617101 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc00c020 con 0x7fe9d009a150 2024-03-18T18:51:00.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9c1ffb700 1 -- 172.21.15.1:0/3232617101 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9cc004250 con 0x7fe9d009a150 2024-03-18T18:51:00.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9d7a42700 1 -- 172.21.15.1:0/3232617101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d01f3820 con 0x7fe9d009a150 2024-03-18T18:51:00.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.228+0000 7fe9c1ffb700 1 -- 172.21.15.1:0/3232617101 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc0036a0 con 0x7fe9d009a150 2024-03-18T18:51:00.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.229+0000 7fe9d7a42700 1 -- 172.21.15.1:0/3232617101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d015aa70 con 0x7fe9d009a150 2024-03-18T18:51:00.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.229+0000 7fe9d7a42700 1 -- 172.21.15.1:0/3232617101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d0068240 con 0x7fe9d009a150 2024-03-18T18:51:00.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.230+0000 7fe9c1ffb700 1 -- 172.21.15.1:0/3232617101 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe9cc003800 con 0x7fe9d009a150 2024-03-18T18:51:00.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.230+0000 7fe9c1ffb700 1 --2- 172.21.15.1:0/3232617101 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe9b406ac90 0x7fe9b406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:00.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.231+0000 7fe9c1ffb700 1 -- 172.21.15.1:0/3232617101 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe9cc089270 con 0x7fe9d009a150 2024-03-18T18:51:00.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.231+0000 7fe9d4fdd700 1 --2- 172.21.15.1:0/3232617101 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe9b406ac90 0x7fe9b406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:00.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.231+0000 7fe9d4fdd700 1 --2- 172.21.15.1:0/3232617101 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe9b406ac90 0x7fe9b406d150 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe9d00c8240 tx=0x7fe9c4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:00.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.235+0000 7fe9c1ffb700 1 -- 172.21.15.1:0/3232617101 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe9cc053ff0 con 0x7fe9d009a150 2024-03-18T18:51:00.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.482+0000 7fe9d7a42700 1 -- 172.21.15.1:0/3232617101 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9d01f3b90 con 0x7fe9d009a150 2024-03-18T18:51:00.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.482+0000 7fe9c1ffb700 1 -- 172.21.15.1:0/3232617101 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4727 (secure 0 0 0) 0x7fe9cc057610 con 0x7fe9d009a150 2024-03-18T18:51:00.479 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:51:00.479 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/883056851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":883056851},{"type":"v1","addr":"172.21.15.165:6834","nonce":883056851}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1388214718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1388214718},{"type":"v1","addr":"172.21.15.165:6835","nonce":1388214718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3020786191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3020786191},{"type":"v1","addr":"172.21.15.1:6836","nonce":3020786191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T18:49:44.809495+0000","modified":"2024-03-18T18:50:07.888953+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2118908012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2118908012},{"type":"v1","addr":"172.21.15.1:6837","nonce":2118908012}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-18T18:51:00.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.484+0000 7fe9d7a42700 1 -- 172.21.15.1:0/3232617101 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe9b406ac90 msgr2=0x7fe9b406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.484+0000 7fe9d7a42700 1 --2- 172.21.15.1:0/3232617101 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe9b406ac90 0x7fe9b406d150 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe9d00c8240 tx=0x7fe9c4009040 comp rx=0 tx=0).stop 2024-03-18T18:51:00.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.484+0000 7fe9d7a42700 1 -- 172.21.15.1:0/3232617101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9d009a150 msgr2=0x7fe9d015a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.484+0000 7fe9d7a42700 1 --2- 172.21.15.1:0/3232617101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9d009a150 0x7fe9d015a530 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc00e810 tx=0x7fe9cc00ebd0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.484+0000 7fe9d7a42700 1 -- 172.21.15.1:0/3232617101 shutdown_connections 2024-03-18T18:51:00.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.484+0000 7fe9d7a42700 1 --2- 172.21.15.1:0/3232617101 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe9b406ac90 0x7fe9b406d150 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.484+0000 7fe9d7a42700 1 --2- 172.21.15.1:0/3232617101 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe9d015b4d0 0x7fe9d00c75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.484+0000 7fe9d7a42700 1 --2- 172.21.15.1:0/3232617101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9d009a150 0x7fe9d015a530 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.484+0000 7fe9d7a42700 1 --2- 172.21.15.1:0/3232617101 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe9d00cf750 0x7fe9d00c5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.484+0000 7fe9d7a42700 1 -- 172.21.15.1:0/3232617101 >> 172.21.15.1:0/3232617101 conn(0x7fe9d00b2cb0 msgr2=0x7fe9d006d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:00.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.484+0000 7fe9d7a42700 1 -- 172.21.15.1:0/3232617101 shutdown_connections 2024-03-18T18:51:00.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.485+0000 7fe9d7a42700 1 -- 172.21.15.1:0/3232617101 wait complete. 2024-03-18T18:51:00.483 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 9 2024-03-18T18:51:00.493 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:51:00.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.621+0000 7f89e7213700 1 Processor -- start 2024-03-18T18:51:00.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.621+0000 7f89e7213700 1 -- start start 2024-03-18T18:51:00.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.622+0000 7f89e7213700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f89e00b6b60 0x7f89e00cc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:00.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.622+0000 7f89e7213700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e00cd0d0 con 0x7f89e00cee90 2024-03-18T18:51:00.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.622+0000 7f89e7213700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e00b3f00 con 0x7f89e00cfda0 2024-03-18T18:51:00.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.622+0000 7f89e7213700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89e00b41e0 con 0x7f89e00b6b60 2024-03-18T18:51:00.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.622+0000 7f89dffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f89e00cee90 0x7f89e00cf690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33026/0 (socket says 172.21.15.1:33026) 2024-03-18T18:51:00.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.622+0000 7f89dffff700 1 -- 172.21.15.1:0/34208371 learned_addr learned my addr 172.21.15.1:0/34208371 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:00.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89df7fe700 1 -- 172.21.15.1:0/34208371 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404206960 0 0) 0x7f89e00cd0d0 con 0x7f89e00cee90 2024-03-18T18:51:00.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89df7fe700 1 -- 172.21.15.1:0/34208371 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89d00035b0 con 0x7f89e00cee90 2024-03-18T18:51:00.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89e4faf700 1 --2- 172.21.15.1:0/34208371 >> v2:172.21.15.165:3300/0 conn(0x7f89e00b6b60 0x7f89e00cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:00.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89df7fe700 1 -- 172.21.15.1:0/34208371 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2345870124 0 0) 0x7f89d00035b0 con 0x7f89e00cee90 2024-03-18T18:51:00.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89df7fe700 1 -- 172.21.15.1:0/34208371 >> v2:172.21.15.165:3300/0 conn(0x7f89e00b6b60 msgr2=0x7f89e00cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:00.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89df7fe700 1 --2- 172.21.15.1:0/34208371 >> v2:172.21.15.165:3300/0 conn(0x7f89e00b6b60 0x7f89e00cc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89df7fe700 1 -- 172.21.15.1:0/34208371 >> v1:172.21.15.165:6789/0 conn(0x7f89e00cfda0 legacy=0x7f89e00b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89df7fe700 1 -- 172.21.15.1:0/34208371 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e00b44c0 con 0x7f89e00cee90 2024-03-18T18:51:00.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89df7fe700 1 -- 172.21.15.1:0/34208371 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f89cc001670 con 0x7f89e00cee90 2024-03-18T18:51:00.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89df7fe700 1 -- 172.21.15.1:0/34208371 >> v1:172.21.15.1:6789/0 conn(0x7f89e00cee90 legacy=0x7f89e00cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89df7fe700 1 --2- 172.21.15.1:0/34208371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89d0003a00 0x7f89d0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:00.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89df7fe700 1 --2- 172.21.15.1:0/34208371 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89d0004850 0x7f89d0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:00.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89df7fe700 1 --2- 172.21.15.1:0/34208371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89d0005210 0x7f89d000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:00.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89df7fe700 1 -- 172.21.15.1:0/34208371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e00b41e0 con 0x7f89d0003a00 2024-03-18T18:51:00.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89df7fe700 1 -- 172.21.15.1:0/34208371 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d000c0b0 con 0x7f89d0005210 2024-03-18T18:51:00.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89df7fe700 1 -- 172.21.15.1:0/34208371 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89d000c270 con 0x7f89d0004850 2024-03-18T18:51:00.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89e4faf700 1 --2- 172.21.15.1:0/34208371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89d0003a00 0x7f89d0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:00.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89e57b0700 1 --2- 172.21.15.1:0/34208371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89d0005210 0x7f89d000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:00.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89dffff700 1 --2- 172.21.15.1:0/34208371 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89d0004850 0x7f89d0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:00.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89e4faf700 1 -- 172.21.15.1:0/34208371 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89d0004850 msgr2=0x7f89d0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:00.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89e4faf700 1 --2- 172.21.15.1:0/34208371 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89d0004850 0x7f89d0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89e4faf700 1 -- 172.21.15.1:0/34208371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89d0005210 msgr2=0x7f89d000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:00.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89e4faf700 1 --2- 172.21.15.1:0/34208371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89d0005210 0x7f89d000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89e4faf700 1 -- 172.21.15.1:0/34208371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d00035b0 con 0x7f89d0003a00 2024-03-18T18:51:00.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.623+0000 7f89e57b0700 1 -- 172.21.15.1:0/34208371 reap_dead start 2024-03-18T18:51:00.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.624+0000 7f89e4faf700 1 --2- 172.21.15.1:0/34208371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89d0003a00 0x7f89d0003e20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f89d4009b60 tx=0x7f89d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eafb909ea7c4e14 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:00.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.624+0000 7f89df7fe700 1 -- 172.21.15.1:0/34208371 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d400e6c0 con 0x7f89d0003a00 2024-03-18T18:51:00.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.624+0000 7f89df7fe700 1 -- 172.21.15.1:0/34208371 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d400e820 con 0x7f89d0003a00 2024-03-18T18:51:00.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.624+0000 7f89df7fe700 1 -- 172.21.15.1:0/34208371 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d400eb90 con 0x7f89d0003a00 2024-03-18T18:51:00.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.624+0000 7f89e7213700 1 -- 172.21.15.1:0/34208371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89d0003a00 msgr2=0x7f89d0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.624+0000 7f89e7213700 1 --2- 172.21.15.1:0/34208371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89d0003a00 0x7f89d0003e20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f89d4009b60 tx=0x7f89d0005880 comp rx=0 tx=0).stop 2024-03-18T18:51:00.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.624+0000 7f89e7213700 1 -- 172.21.15.1:0/34208371 shutdown_connections 2024-03-18T18:51:00.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.624+0000 7f89e7213700 1 --2- 172.21.15.1:0/34208371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89d0003a00 0x7f89d0003e20 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.624+0000 7f89e7213700 1 -- 172.21.15.1:0/34208371 >> 172.21.15.1:0/34208371 conn(0x7f89e00b2ba0 msgr2=0x7f89e00b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:00.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.624+0000 7f89e7213700 1 -- 172.21.15.1:0/34208371 shutdown_connections 2024-03-18T18:51:00.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.624+0000 7f89e7213700 1 -- 172.21.15.1:0/34208371 wait complete. 2024-03-18T18:51:00.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.626+0000 7f89e7213700 1 Processor -- start 2024-03-18T18:51:00.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.626+0000 7f89e7213700 1 -- start start 2024-03-18T18:51:00.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.627+0000 7f89e7213700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89d0003a00 0x7f89e009ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:00.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.627+0000 7f89e7213700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e015d830 0x7f89e01f3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:00.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.627+0000 7f89e7213700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e01f6eb0 0x7f89e006f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:00.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.627+0000 7f89e7213700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e015dcb0 con 0x7f89e015d830 2024-03-18T18:51:00.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.627+0000 7f89e7213700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e01f38e0 con 0x7f89e01f6eb0 2024-03-18T18:51:00.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.627+0000 7f89e7213700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e01f3bc0 con 0x7f89d0003a00 2024-03-18T18:51:00.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.627+0000 7f89dffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e015d830 0x7f89e01f3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:00.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.627+0000 7f89dffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e015d830 0x7f89e01f3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38640/0 (socket says 172.21.15.1:38640) 2024-03-18T18:51:00.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.627+0000 7f89dffff700 1 -- 172.21.15.1:0/1364015074 learned_addr learned my addr 172.21.15.1:0/1364015074 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:00.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.627+0000 7f89e57b0700 1 --2- 172.21.15.1:0/1364015074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e01f6eb0 0x7f89e006f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:00.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.627+0000 7f89e4faf700 1 --2- 172.21.15.1:0/1364015074 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89d0003a00 0x7f89e009ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:00.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.627+0000 7f89dffff700 1 -- 172.21.15.1:0/1364015074 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89d0003a00 msgr2=0x7f89e009ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.627+0000 7f89dffff700 1 --2- 172.21.15.1:0/1364015074 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89d0003a00 0x7f89e009ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.627+0000 7f89dffff700 1 -- 172.21.15.1:0/1364015074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e01f6eb0 msgr2=0x7f89e006f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.627+0000 7f89dffff700 1 --2- 172.21.15.1:0/1364015074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e01f6eb0 0x7f89e006f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.627+0000 7f89dffff700 1 -- 172.21.15.1:0/1364015074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e01f66b0 con 0x7f89e015d830 2024-03-18T18:51:00.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.628+0000 7f89dffff700 1 --2- 172.21.15.1:0/1364015074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e015d830 0x7f89e01f3020 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f89cc007290 tx=0x7f89cc0075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:00.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.628+0000 7f89dd7fa700 1 -- 172.21.15.1:0/1364015074 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89cc00f5e0 con 0x7f89e015d830 2024-03-18T18:51:00.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.628+0000 7f89dd7fa700 1 -- 172.21.15.1:0/1364015074 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89cc00f780 con 0x7f89e015d830 2024-03-18T18:51:00.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.628+0000 7f89e7213700 1 -- 172.21.15.1:0/1364015074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e01f6990 con 0x7f89e015d830 2024-03-18T18:51:00.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.628+0000 7f89dd7fa700 1 -- 172.21.15.1:0/1364015074 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89cc00fb30 con 0x7f89e015d830 2024-03-18T18:51:00.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.628+0000 7f89e7213700 1 -- 172.21.15.1:0/1364015074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e01f3560 con 0x7f89e015d830 2024-03-18T18:51:00.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.629+0000 7f89e7213700 1 -- 172.21.15.1:0/1364015074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e00676a0 con 0x7f89e015d830 2024-03-18T18:51:00.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.630+0000 7f89dd7fa700 1 -- 172.21.15.1:0/1364015074 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f89cc00fc90 con 0x7f89e015d830 2024-03-18T18:51:00.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.630+0000 7f89dd7fa700 1 --2- 172.21.15.1:0/1364015074 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f89c406ac90 0x7f89c406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:00.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.630+0000 7f89dd7fa700 1 -- 172.21.15.1:0/1364015074 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f89cc08a060 con 0x7f89e015d830 2024-03-18T18:51:00.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.630+0000 7f89e4faf700 1 --2- 172.21.15.1:0/1364015074 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f89c406ac90 0x7f89c406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:00.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.630+0000 7f89e4faf700 1 --2- 172.21.15.1:0/1364015074 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f89c406ac90 0x7f89c406d150 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f89d40090b0 tx=0x7f89d401e040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:00.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.634+0000 7f89dd7fa700 1 -- 172.21.15.1:0/1364015074 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f89cc054f60 con 0x7f89e015d830 2024-03-18T18:51:00.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.837+0000 7f89e7213700 1 -- 172.21.15.1:0/1364015074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89e00b6380 con 0x7f89e015d830 2024-03-18T18:51:00.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.838+0000 7f89dd7fa700 1 -- 172.21.15.1:0/1364015074 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18217 (secure 0 0 0) 0x7f89cc058580 con 0x7f89e015d830 2024-03-18T18:51:00.835 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:51:00.836 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":27,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:50:34.993482+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:49:42.800801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T18:49:43.776684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T18:49:46.854608+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:51:00.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.840+0000 7f89e7213700 1 -- 172.21.15.1:0/1364015074 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f89c406ac90 msgr2=0x7f89c406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.840+0000 7f89e7213700 1 --2- 172.21.15.1:0/1364015074 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f89c406ac90 0x7f89c406d150 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f89d40090b0 tx=0x7f89d401e040 comp rx=0 tx=0).stop 2024-03-18T18:51:00.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.840+0000 7f89e7213700 1 -- 172.21.15.1:0/1364015074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e015d830 msgr2=0x7f89e01f3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.840+0000 7f89e7213700 1 --2- 172.21.15.1:0/1364015074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e015d830 0x7f89e01f3020 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f89cc007290 tx=0x7f89cc0075a0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.840+0000 7f89e7213700 1 -- 172.21.15.1:0/1364015074 shutdown_connections 2024-03-18T18:51:00.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.840+0000 7f89e7213700 1 --2- 172.21.15.1:0/1364015074 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f89c406ac90 0x7f89c406d150 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.840+0000 7f89e7213700 1 --2- 172.21.15.1:0/1364015074 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89d0003a00 0x7f89e009ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.840+0000 7f89e7213700 1 --2- 172.21.15.1:0/1364015074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e01f6eb0 0x7f89e006f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.840+0000 7f89e7213700 1 --2- 172.21.15.1:0/1364015074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89e015d830 0x7f89e01f3020 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.841+0000 7f89e7213700 1 -- 172.21.15.1:0/1364015074 >> 172.21.15.1:0/1364015074 conn(0x7f89e00b2ba0 msgr2=0x7f89e00ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:00.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.841+0000 7f89e7213700 1 -- 172.21.15.1:0/1364015074 shutdown_connections 2024-03-18T18:51:00.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.841+0000 7f89e7213700 1 -- 172.21.15.1:0/1364015074 wait complete. 2024-03-18T18:51:00.850 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T18:51:00.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.981+0000 7f9bf4698700 1 Processor -- start 2024-03-18T18:51:00.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.981+0000 7f9bf4698700 1 -- start start 2024-03-18T18:51:00.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.982+0000 7f9bf4698700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9bec06ae60 0x7f9bec168150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:00.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.982+0000 7f9bf4698700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bec0cf600 con 0x7f9bec0cfa90 2024-03-18T18:51:00.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.982+0000 7f9bf4698700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bec0ce720 con 0x7f9bec15d5a0 2024-03-18T18:51:00.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.982+0000 7f9bf4698700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bec0ce9f0 con 0x7f9bec06ae60 2024-03-18T18:51:00.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.982+0000 7f9bf2c35700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9bec06ae60 0x7f9bec168150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:00.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.982+0000 7f9bf1c33700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9bec0cfa90 0x7f9bec06a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33036/0 (socket says 172.21.15.1:33036) 2024-03-18T18:51:00.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.982+0000 7f9bf1c33700 1 -- 172.21.15.1:0/2272630896 learned_addr learned my addr 172.21.15.1:0/2272630896 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:00.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.983+0000 7f9bf2c35700 1 -- 172.21.15.1:0/2272630896 >> v1:172.21.15.165:6789/0 conn(0x7f9bec15d5a0 legacy=0x7f9bec0ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.983+0000 7f9bf2c35700 1 -- 172.21.15.1:0/2272630896 >> v1:172.21.15.1:6789/0 conn(0x7f9bec0cfa90 legacy=0x7f9bec06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.983+0000 7f9bf1432700 1 -- 172.21.15.1:0/2272630896 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442218021 0 0) 0x7f9bec0ce720 con 0x7f9bec15d5a0 2024-03-18T18:51:00.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.983+0000 7f9bf2c35700 1 -- 172.21.15.1:0/2272630896 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bec0cecd0 con 0x7f9bec06ae60 2024-03-18T18:51:00.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.983+0000 7f9bf2c35700 1 --2- 172.21.15.1:0/2272630896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bec06ae60 0x7f9bec168150 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009d90 tx=0x7f9be400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e19f54e7c0a81e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:00.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.983+0000 7f9bf1432700 1 -- 172.21.15.1:0/2272630896 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be400e070 con 0x7f9bec06ae60 2024-03-18T18:51:00.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.983+0000 7f9bf1432700 1 -- 172.21.15.1:0/2272630896 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be400bad0 con 0x7f9bec06ae60 2024-03-18T18:51:00.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.983+0000 7f9bf1432700 1 -- 172.21.15.1:0/2272630896 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be400be80 con 0x7f9bec06ae60 2024-03-18T18:51:00.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.983+0000 7f9bf4698700 1 -- 172.21.15.1:0/2272630896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bec06ae60 msgr2=0x7f9bec168150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.983+0000 7f9bf4698700 1 --2- 172.21.15.1:0/2272630896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bec06ae60 0x7f9bec168150 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009d90 tx=0x7f9be400b330 comp rx=0 tx=0).stop 2024-03-18T18:51:00.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.984+0000 7f9bf4698700 1 -- 172.21.15.1:0/2272630896 shutdown_connections 2024-03-18T18:51:00.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.984+0000 7f9bf4698700 1 --2- 172.21.15.1:0/2272630896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bec06ae60 0x7f9bec168150 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.984+0000 7f9bf4698700 1 -- 172.21.15.1:0/2272630896 >> 172.21.15.1:0/2272630896 conn(0x7f9bec0cc9b0 msgr2=0x7f9bec0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:00.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.984+0000 7f9bf4698700 1 -- 172.21.15.1:0/2272630896 shutdown_connections 2024-03-18T18:51:00.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.984+0000 7f9bf4698700 1 -- 172.21.15.1:0/2272630896 wait complete. 2024-03-18T18:51:00.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.985+0000 7f9bf4698700 1 Processor -- start 2024-03-18T18:51:00.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.985+0000 7f9bf4698700 1 -- start start 2024-03-18T18:51:00.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9bf4698700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bec06ae60 0x7f9bec1f6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:00.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9bf4698700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bec0cfa90 0x7f9bec1f3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:00.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9bf4698700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bec15d5a0 0x7f9bec0c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:00.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9bf4698700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec14c510 con 0x7f9bec0cfa90 2024-03-18T18:51:00.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9bf4698700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec1f3e40 con 0x7f9bec06ae60 2024-03-18T18:51:00.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9bf4698700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec14c7b0 con 0x7f9bec15d5a0 2024-03-18T18:51:00.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9bf1c33700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bec0cfa90 0x7f9bec1f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:00.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9bf1c33700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bec0cfa90 0x7f9bec1f3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38670/0 (socket says 172.21.15.1:38670) 2024-03-18T18:51:00.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9bf1c33700 1 -- 172.21.15.1:0/2438792145 learned_addr learned my addr 172.21.15.1:0/2438792145 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:00.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9bf2434700 1 --2- 172.21.15.1:0/2438792145 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bec06ae60 0x7f9bec1f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:00.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9bf1c33700 1 -- 172.21.15.1:0/2438792145 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bec15d5a0 msgr2=0x7f9bec0c6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9bf1c33700 1 --2- 172.21.15.1:0/2438792145 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bec15d5a0 0x7f9bec0c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9bf1c33700 1 -- 172.21.15.1:0/2438792145 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bec06ae60 msgr2=0x7f9bec1f6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:00.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9bf1c33700 1 --2- 172.21.15.1:0/2438792145 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bec06ae60 0x7f9bec1f6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:00.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9bf1c33700 1 -- 172.21.15.1:0/2438792145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be4009a70 con 0x7f9bec0cfa90 2024-03-18T18:51:00.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9bf2434700 1 --2- 172.21.15.1:0/2438792145 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bec06ae60 0x7f9bec1f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:00.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9bf1c33700 1 --2- 172.21.15.1:0/2438792145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bec0cfa90 0x7f9bec1f3900 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc00c7b0 tx=0x7f9bdc00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:00.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9be2ffd700 1 -- 172.21.15.1:0/2438792145 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc004c80 con 0x7f9bec0cfa90 2024-03-18T18:51:00.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9be2ffd700 1 -- 172.21.15.1:0/2438792145 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bdc004de0 con 0x7f9bec0cfa90 2024-03-18T18:51:00.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9bf4698700 1 -- 172.21.15.1:0/2438792145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bec0c6da0 con 0x7f9bec0cfa90 2024-03-18T18:51:00.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9be2ffd700 1 -- 172.21.15.1:0/2438792145 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc007310 con 0x7f9bec0cfa90 2024-03-18T18:51:00.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.986+0000 7f9bf4698700 1 -- 172.21.15.1:0/2438792145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bec1f6e60 con 0x7f9bec0cfa90 2024-03-18T18:51:00.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.987+0000 7f9bf4698700 1 -- 172.21.15.1:0/2438792145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bec15b8f0 con 0x7f9bec0cfa90 2024-03-18T18:51:00.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.988+0000 7f9be2ffd700 1 -- 172.21.15.1:0/2438792145 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9bdc007470 con 0x7f9bec0cfa90 2024-03-18T18:51:00.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.988+0000 7f9be2ffd700 1 --2- 172.21.15.1:0/2438792145 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9bd006ac90 0x7f9bd006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:00.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.988+0000 7f9be2ffd700 1 -- 172.21.15.1:0/2438792145 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f9bdc088bf0 con 0x7f9bec0cfa90 2024-03-18T18:51:00.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.988+0000 7f9bf2434700 1 --2- 172.21.15.1:0/2438792145 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9bd006ac90 0x7f9bd006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:00.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.989+0000 7f9bf2434700 1 --2- 172.21.15.1:0/2438792145 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9bd006ac90 0x7f9bd006d150 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0af250 tx=0x7f9be8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:00.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:00.993+0000 7f9be2ffd700 1 -- 172.21.15.1:0/2438792145 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9bdc053a10 con 0x7f9bec0cfa90 2024-03-18T18:51:01.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.242+0000 7f9bf4698700 1 -- 172.21.15.1:0/2438792145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9bec0676a0 con 0x7f9bec0cfa90 2024-03-18T18:51:01.239 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-18T18:51:01.243+0000 7fbdbeb46700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T18:51:01.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.257+0000 7f9be2ffd700 1 -- 172.21.15.1:0/2438792145 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9bdc018020 con 0x7f9bec0cfa90 2024-03-18T18:51:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.259+0000 7f9bf4698700 1 -- 172.21.15.1:0/2438792145 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9bd006ac90 msgr2=0x7f9bd006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.259+0000 7f9bf4698700 1 --2- 172.21.15.1:0/2438792145 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9bd006ac90 0x7f9bd006d150 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0af250 tx=0x7f9be8009040 comp rx=0 tx=0).stop 2024-03-18T18:51:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.259+0000 7f9bf4698700 1 -- 172.21.15.1:0/2438792145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bec0cfa90 msgr2=0x7f9bec1f3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.259+0000 7f9bf4698700 1 --2- 172.21.15.1:0/2438792145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bec0cfa90 0x7f9bec1f3900 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc00c7b0 tx=0x7f9bdc00cb70 comp rx=0 tx=0).stop 2024-03-18T18:51:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.259+0000 7f9bf4698700 1 -- 172.21.15.1:0/2438792145 shutdown_connections 2024-03-18T18:51:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.259+0000 7f9bf4698700 1 --2- 172.21.15.1:0/2438792145 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9bd006ac90 0x7f9bd006d150 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.259+0000 7f9bf4698700 1 --2- 172.21.15.1:0/2438792145 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bec15d5a0 0x7f9bec0c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.259+0000 7f9bf4698700 1 --2- 172.21.15.1:0/2438792145 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bec06ae60 0x7f9bec1f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:01.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.259+0000 7f9bf4698700 1 --2- 172.21.15.1:0/2438792145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bec0cfa90 0x7f9bec1f3900 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.259+0000 7f9bf4698700 1 -- 172.21.15.1:0/2438792145 >> 172.21.15.1:0/2438792145 conn(0x7f9bec0cc9b0 msgr2=0x7f9bec0aded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.259+0000 7f9bf4698700 1 -- 172.21.15.1:0/2438792145 shutdown_connections 2024-03-18T18:51:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.259+0000 7f9bf4698700 1 -- 172.21.15.1:0/2438792145 wait complete. 2024-03-18T18:51:01.257 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T18:51:01.257 INFO:tasks.ceph.mds.a.smithi001.stderr: -5410> 2024-03-18T18:49:48.243+0000 7f460b344700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T18:51:01.269 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T18:51:01.288 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-18T18:51:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.401+0000 7fcdc0367700 1 Processor -- start 2024-03-18T18:51:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.401+0000 7fcdc0367700 1 -- start start 2024-03-18T18:51:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.402+0000 7fcdc0367700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcdb80ace80 0x7fcdb80ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.402+0000 7fcdc0367700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdb80ad850 0x7fcdb815ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.402+0000 7fcdc0367700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb815c340 con 0x7fcdb806a750 2024-03-18T18:51:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.402+0000 7fcdc0367700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdb8070090 con 0x7fcdb80ace80 2024-03-18T18:51:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.402+0000 7fcdc0367700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb815b7a0 con 0x7fcdb80ad850 2024-03-18T18:51:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.402+0000 7fcdbe103700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcdb806a750 0x7fcdb806ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33068/0 (socket says 172.21.15.1:33068) 2024-03-18T18:51:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.402+0000 7fcdbe103700 1 -- 172.21.15.1:0/1847976023 learned_addr learned my addr 172.21.15.1:0/1847976023 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.402+0000 7fcdbd902700 1 --2- 172.21.15.1:0/1847976023 >> v2:172.21.15.165:3300/0 conn(0x7fcdb80ace80 0x7fcdb80ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.402+0000 7fcdbe904700 1 --2- 172.21.15.1:0/1847976023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdb80ad850 0x7fcdb815ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.402+0000 7fcdbd101700 1 -- 172.21.15.1:0/1847976023 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207019105 0 0) 0x7fcdb815c340 con 0x7fcdb806a750 2024-03-18T18:51:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.402+0000 7fcdbd101700 1 -- 172.21.15.1:0/1847976023 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcda40035b0 con 0x7fcdb806a750 2024-03-18T18:51:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.402+0000 7fcdbd902700 1 -- 172.21.15.1:0/1847976023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdb80ad850 msgr2=0x7fcdb815ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.402+0000 7fcdbd902700 1 --2- 172.21.15.1:0/1847976023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdb80ad850 0x7fcdb815ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.402+0000 7fcdbd902700 1 -- 172.21.15.1:0/1847976023 >> v1:172.21.15.1:6789/0 conn(0x7fcdb806a750 legacy=0x7fcdb806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.402+0000 7fcdbd902700 1 -- 172.21.15.1:0/1847976023 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb815ba10 con 0x7fcdb80ace80 2024-03-18T18:51:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.402+0000 7fcdbd902700 1 --2- 172.21.15.1:0/1847976023 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdb80ace80 0x7fcdb80ad310 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcda8009840 tx=0x7fcda8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acbd416f47337576 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.403+0000 7fcdbd101700 1 -- 172.21.15.1:0/1847976023 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda8011830 con 0x7fcdb80ace80 2024-03-18T18:51:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.403+0000 7fcdbd101700 1 -- 172.21.15.1:0/1847976023 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcda8011990 con 0x7fcdb80ace80 2024-03-18T18:51:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.403+0000 7fcdbd101700 1 -- 172.21.15.1:0/1847976023 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda8011d20 con 0x7fcdb80ace80 2024-03-18T18:51:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.403+0000 7fcdc0367700 1 -- 172.21.15.1:0/1847976023 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdb80ace80 msgr2=0x7fcdb80ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.403+0000 7fcdc0367700 1 --2- 172.21.15.1:0/1847976023 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdb80ace80 0x7fcdb80ad310 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcda8009840 tx=0x7fcda8009b50 comp rx=0 tx=0).stop 2024-03-18T18:51:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.403+0000 7fcdc0367700 1 -- 172.21.15.1:0/1847976023 shutdown_connections 2024-03-18T18:51:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.403+0000 7fcdc0367700 1 --2- 172.21.15.1:0/1847976023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdb80ad850 0x7fcdb815ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:01.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.403+0000 7fcdc0367700 1 --2- 172.21.15.1:0/1847976023 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdb80ace80 0x7fcdb80ad310 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.403+0000 7fcdc0367700 1 -- 172.21.15.1:0/1847976023 >> 172.21.15.1:0/1847976023 conn(0x7fcdb80cc9b0 msgr2=0x7fcdb80ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.403+0000 7fcdc0367700 1 -- 172.21.15.1:0/1847976023 shutdown_connections 2024-03-18T18:51:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.403+0000 7fcdc0367700 1 -- 172.21.15.1:0/1847976023 wait complete. 2024-03-18T18:51:01.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.405+0000 7fcdc0367700 1 Processor -- start 2024-03-18T18:51:01.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.405+0000 7fcdc0367700 1 -- start start 2024-03-18T18:51:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.405+0000 7fcdc0367700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdb806a750 0x7fcdb80c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.405+0000 7fcdc0367700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdb80ace80 0x7fcdb8158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.405+0000 7fcdc0367700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb80ad850 0x7fcdb809ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.405+0000 7fcdc0367700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb80c8380 con 0x7fcdb80ad850 2024-03-18T18:51:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.405+0000 7fcdc0367700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb8158760 con 0x7fcdb80ace80 2024-03-18T18:51:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.405+0000 7fcdc0367700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb80b5bc0 con 0x7fcdb806a750 2024-03-18T18:51:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.406+0000 7fcdbe904700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb80ad850 0x7fcdb809ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.406+0000 7fcdbe904700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb80ad850 0x7fcdb809ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38692/0 (socket says 172.21.15.1:38692) 2024-03-18T18:51:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.406+0000 7fcdbe904700 1 -- 172.21.15.1:0/1576208341 learned_addr learned my addr 172.21.15.1:0/1576208341 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.406+0000 7fcdbd902700 1 --2- 172.21.15.1:0/1576208341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdb80ace80 0x7fcdb8158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.406+0000 7fcdbe103700 1 --2- 172.21.15.1:0/1576208341 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdb806a750 0x7fcdb80c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.406+0000 7fcdbe904700 1 -- 172.21.15.1:0/1576208341 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdb806a750 msgr2=0x7fcdb80c7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:01.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.406+0000 7fcdbe904700 1 --2- 172.21.15.1:0/1576208341 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdb806a750 0x7fcdb80c7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.406+0000 7fcdbe904700 1 -- 172.21.15.1:0/1576208341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdb80ace80 msgr2=0x7fcdb8158220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.406+0000 7fcdbe904700 1 --2- 172.21.15.1:0/1576208341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdb80ace80 0x7fcdb8158220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.406+0000 7fcdbe904700 1 -- 172.21.15.1:0/1576208341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda80094a0 con 0x7fcdb80ad850 2024-03-18T18:51:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.406+0000 7fcdbd902700 1 --2- 172.21.15.1:0/1576208341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdb80ace80 0x7fcdb8158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.406+0000 7fcdbe904700 1 --2- 172.21.15.1:0/1576208341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb80ad850 0x7fcdb809ce60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000efc0 tx=0x7fcdb000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.406+0000 7fcdaeffd700 1 -- 172.21.15.1:0/1576208341 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0009ca0 con 0x7fcdb80ad850 2024-03-18T18:51:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.406+0000 7fcdaeffd700 1 -- 172.21.15.1:0/1576208341 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb000f040 con 0x7fcdb80ad850 2024-03-18T18:51:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.406+0000 7fcdc0367700 1 -- 172.21.15.1:0/1576208341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb80c7700 con 0x7fcdb80ad850 2024-03-18T18:51:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.406+0000 7fcdaeffd700 1 -- 172.21.15.1:0/1576208341 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0014480 con 0x7fcdb80ad850 2024-03-18T18:51:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.406+0000 7fcdc0367700 1 -- 172.21.15.1:0/1576208341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb80c7cc0 con 0x7fcdb80ad850 2024-03-18T18:51:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.407+0000 7fcdc0367700 1 -- 172.21.15.1:0/1576208341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb80c84c0 con 0x7fcdb80ad850 2024-03-18T18:51:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.408+0000 7fcdaeffd700 1 -- 172.21.15.1:0/1576208341 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcdb00145e0 con 0x7fcdb80ad850 2024-03-18T18:51:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.408+0000 7fcdaeffd700 1 --2- 172.21.15.1:0/1576208341 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcd9c06ac90 0x7fcd9c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:01.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.408+0000 7fcdaeffd700 1 -- 172.21.15.1:0/1576208341 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6749+0+0 (secure 0 0 0) 0x7fcdb008a540 con 0x7fcdb80ad850 2024-03-18T18:51:01.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.408+0000 7fcdbe103700 1 --2- 172.21.15.1:0/1576208341 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcd9c06ac90 0x7fcd9c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:01.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.409+0000 7fcdbe103700 1 --2- 172.21.15.1:0/1576208341 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcd9c06ac90 0x7fcd9c06d150 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40010e0 tx=0x7fcdb4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:01.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.413+0000 7fcdaeffd700 1 -- 172.21.15.1:0/1576208341 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcdb80c84c0 con 0x7fcdb80ad850 2024-03-18T18:51:01.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:01.661+0000 7fcdc0367700 1 -- 172.21.15.1:0/1576208341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcdb80c84c0 con 0x7fcdb80ad850 2024-03-18T18:51:02.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.261+0000 7fcdaeffd700 1 -- 172.21.15.1:0/1576208341 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcdb80c84c0 con 0x7fcdb80ad850 2024-03-18T18:51:02.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.264+0000 7fcdc0367700 1 -- 172.21.15.1:0/1576208341 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcd9c06ac90 msgr2=0x7fcd9c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:02.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.264+0000 7fcdc0367700 1 --2- 172.21.15.1:0/1576208341 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcd9c06ac90 0x7fcd9c06d150 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40010e0 tx=0x7fcdb4009040 comp rx=0 tx=0).stop 2024-03-18T18:51:02.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.264+0000 7fcdc0367700 1 -- 172.21.15.1:0/1576208341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb80ad850 msgr2=0x7fcdb809ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:02.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.264+0000 7fcdc0367700 1 --2- 172.21.15.1:0/1576208341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb80ad850 0x7fcdb809ce60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000efc0 tx=0x7fcdb000c3a0 comp rx=0 tx=0).stop 2024-03-18T18:51:02.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.264+0000 7fcdc0367700 1 -- 172.21.15.1:0/1576208341 shutdown_connections 2024-03-18T18:51:02.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.264+0000 7fcdc0367700 1 --2- 172.21.15.1:0/1576208341 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcd9c06ac90 0x7fcd9c06d150 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:02.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.264+0000 7fcdc0367700 1 --2- 172.21.15.1:0/1576208341 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdb806a750 0x7fcdb80c7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:02.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.264+0000 7fcdc0367700 1 --2- 172.21.15.1:0/1576208341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdb80ad850 0x7fcdb809ce60 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:02.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.264+0000 7fcdc0367700 1 --2- 172.21.15.1:0/1576208341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdb80ace80 0x7fcdb8158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:02.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.264+0000 7fcdc0367700 1 -- 172.21.15.1:0/1576208341 >> 172.21.15.1:0/1576208341 conn(0x7fcdb80cc9b0 msgr2=0x7fcdb806edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:02.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.264+0000 7fcdc0367700 1 -- 172.21.15.1:0/1576208341 shutdown_connections 2024-03-18T18:51:02.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.264+0000 7fcdc0367700 1 -- 172.21.15.1:0/1576208341 wait complete. 2024-03-18T18:51:02.276 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T18:51:02.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.457+0000 7f3b5865f700 1 Processor -- start 2024-03-18T18:51:02.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.457+0000 7f3b5865f700 1 -- start start 2024-03-18T18:51:02.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.458+0000 7f3b5865f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3b500d0460 0x7f3b500b8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:02.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.458+0000 7f3b5865f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b500d1280 con 0x7f3b5015d430 2024-03-18T18:51:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.458+0000 7f3b5865f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b500d1560 con 0x7f3b5015d810 2024-03-18T18:51:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.458+0000 7f3b5865f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b500d1f00 con 0x7f3b500d0460 2024-03-18T18:51:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b563fb700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3b5015d430 0x7f3b500cea80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33078/0 (socket says 172.21.15.1:33078) 2024-03-18T18:51:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b563fb700 1 -- 172.21.15.1:0/937350335 learned_addr learned my addr 172.21.15.1:0/937350335 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b553f9700 1 -- 172.21.15.1:0/937350335 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020593870 0 0) 0x7f3b500d1280 con 0x7f3b5015d430 2024-03-18T18:51:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b56bfc700 1 --2- 172.21.15.1:0/937350335 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3b500d0460 0x7f3b500b8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b553f9700 1 -- 172.21.15.1:0/937350335 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b3c0035b0 con 0x7f3b5015d430 2024-03-18T18:51:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b553f9700 1 -- 172.21.15.1:0/937350335 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3713905922 0 0) 0x7f3b3c0035b0 con 0x7f3b5015d430 2024-03-18T18:51:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b553f9700 1 -- 172.21.15.1:0/937350335 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3b500d0460 msgr2=0x7f3b500b8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b553f9700 1 --2- 172.21.15.1:0/937350335 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3b500d0460 0x7f3b500b8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b553f9700 1 -- 172.21.15.1:0/937350335 >> v1:172.21.15.165:6789/0 conn(0x7f3b5015d810 legacy=0x7f3b500cfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:02.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b553f9700 1 -- 172.21.15.1:0/937350335 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b500d1840 con 0x7f3b5015d430 2024-03-18T18:51:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b553f9700 1 -- 172.21.15.1:0/937350335 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f3b40001670 con 0x7f3b5015d430 2024-03-18T18:51:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b553f9700 1 -- 172.21.15.1:0/937350335 >> v1:172.21.15.1:6789/0 conn(0x7f3b5015d430 legacy=0x7f3b500cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b553f9700 1 --2- 172.21.15.1:0/937350335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b3c003a30 0x7f3b3c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b553f9700 1 --2- 172.21.15.1:0/937350335 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3b500d0460 0x7f3b3c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b553f9700 1 --2- 172.21.15.1:0/937350335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b3c004d90 0x7f3b3c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b553f9700 1 -- 172.21.15.1:0/937350335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b500d1f00 con 0x7f3b3c003a30 2024-03-18T18:51:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b553f9700 1 -- 172.21.15.1:0/937350335 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c00baf0 con 0x7f3b3c004d90 2024-03-18T18:51:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b553f9700 1 -- 172.21.15.1:0/937350335 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c00bcb0 con 0x7f3b500d0460 2024-03-18T18:51:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b563fb700 1 --2- 172.21.15.1:0/937350335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b3c003a30 0x7f3b3c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b55bfa700 1 --2- 172.21.15.1:0/937350335 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3b500d0460 0x7f3b3c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.459+0000 7f3b56bfc700 1 --2- 172.21.15.1:0/937350335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b3c004d90 0x7f3b3c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.460+0000 7f3b563fb700 1 -- 172.21.15.1:0/937350335 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3b500d0460 msgr2=0x7f3b3c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.460+0000 7f3b563fb700 1 --2- 172.21.15.1:0/937350335 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3b500d0460 0x7f3b3c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:02.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.460+0000 7f3b563fb700 1 -- 172.21.15.1:0/937350335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b3c004d90 msgr2=0x7f3b3c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:02.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.460+0000 7f3b563fb700 1 --2- 172.21.15.1:0/937350335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b3c004d90 0x7f3b3c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:02.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.460+0000 7f3b563fb700 1 -- 172.21.15.1:0/937350335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3c0035b0 con 0x7f3b3c003a30 2024-03-18T18:51:02.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.460+0000 7f3b563fb700 1 --2- 172.21.15.1:0/937350335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b3c003a30 0x7f3b3c003e50 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3b4000c210 tx=0x7f3b3c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1cf850cf71594df server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:02.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.460+0000 7f3b553f9700 1 -- 172.21.15.1:0/937350335 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b40007c10 con 0x7f3b3c003a30 2024-03-18T18:51:02.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.460+0000 7f3b553f9700 1 -- 172.21.15.1:0/937350335 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4000cdb0 con 0x7f3b3c003a30 2024-03-18T18:51:02.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.460+0000 7f3b553f9700 1 -- 172.21.15.1:0/937350335 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b40005ba0 con 0x7f3b3c003a30 2024-03-18T18:51:02.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.460+0000 7f3b5865f700 1 -- 172.21.15.1:0/937350335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b3c003a30 msgr2=0x7f3b3c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:02.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.460+0000 7f3b5865f700 1 --2- 172.21.15.1:0/937350335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b3c003a30 0x7f3b3c003e50 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3b4000c210 tx=0x7f3b3c005880 comp rx=0 tx=0).stop 2024-03-18T18:51:02.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.461+0000 7f3b56bfc700 1 -- 172.21.15.1:0/937350335 reap_dead start 2024-03-18T18:51:02.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.461+0000 7f3b5865f700 1 -- 172.21.15.1:0/937350335 shutdown_connections 2024-03-18T18:51:02.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.461+0000 7f3b5865f700 1 -- 172.21.15.1:0/937350335 >> 172.21.15.1:0/937350335 conn(0x7f3b5009c630 msgr2=0x7f3b500d3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:02.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.461+0000 7f3b5865f700 1 -- 172.21.15.1:0/937350335 shutdown_connections 2024-03-18T18:51:02.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.461+0000 7f3b5865f700 1 -- 172.21.15.1:0/937350335 wait complete. 2024-03-18T18:51:02.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.462+0000 7f3b5865f700 1 Processor -- start 2024-03-18T18:51:02.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.462+0000 7f3b5865f700 1 -- start start 2024-03-18T18:51:02.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b5865f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b3c003a30 0x7f3b5015e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:02.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b5865f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b5015b440 0x7f3b500a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:02.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b5865f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3b500c9500 0x7f3b500c9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:02.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b5865f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5015f8e0 con 0x7f3b5015b440 2024-03-18T18:51:02.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b5865f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b500c9ec0 con 0x7f3b3c003a30 2024-03-18T18:51:02.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b5865f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5015ed70 con 0x7f3b500c9500 2024-03-18T18:51:02.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b55bfa700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b5015b440 0x7f3b500a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:02.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b55bfa700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b5015b440 0x7f3b500a0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38706/0 (socket says 172.21.15.1:38706) 2024-03-18T18:51:02.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b55bfa700 1 -- 172.21.15.1:0/2858824960 learned_addr learned my addr 172.21.15.1:0/2858824960 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:02.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b56bfc700 1 --2- 172.21.15.1:0/2858824960 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3b500c9500 0x7f3b500c9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:02.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b563fb700 1 --2- 172.21.15.1:0/2858824960 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b3c003a30 0x7f3b5015e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:02.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b55bfa700 1 -- 172.21.15.1:0/2858824960 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3b500c9500 msgr2=0x7f3b500c9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:02.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b55bfa700 1 --2- 172.21.15.1:0/2858824960 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3b500c9500 0x7f3b500c9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:02.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b55bfa700 1 -- 172.21.15.1:0/2858824960 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b3c003a30 msgr2=0x7f3b5015e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:02.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b55bfa700 1 --2- 172.21.15.1:0/2858824960 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b3c003a30 0x7f3b5015e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:02.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b55bfa700 1 -- 172.21.15.1:0/2858824960 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5015f040 con 0x7f3b5015b440 2024-03-18T18:51:02.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b55bfa700 1 --2- 172.21.15.1:0/2858824960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b5015b440 0x7f3b500a0ad0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00f7f0 tx=0x7f3b4c00fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:02.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b46ffd700 1 -- 172.21.15.1:0/2858824960 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c0052e0 con 0x7f3b5015b440 2024-03-18T18:51:02.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b46ffd700 1 -- 172.21.15.1:0/2858824960 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4c005440 con 0x7f3b5015b440 2024-03-18T18:51:02.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b5865f700 1 -- 172.21.15.1:0/2858824960 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b5015f320 con 0x7f3b5015b440 2024-03-18T18:51:02.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.463+0000 7f3b46ffd700 1 -- 172.21.15.1:0/2858824960 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c0038c0 con 0x7f3b5015b440 2024-03-18T18:51:02.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.464+0000 7f3b5865f700 1 -- 172.21.15.1:0/2858824960 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5014c520 con 0x7f3b5015b440 2024-03-18T18:51:02.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.465+0000 7f3b5865f700 1 -- 172.21.15.1:0/2858824960 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b5014c380 con 0x7f3b5015b440 2024-03-18T18:51:02.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.465+0000 7f3b46ffd700 1 -- 172.21.15.1:0/2858824960 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3b4c003a20 con 0x7f3b5015b440 2024-03-18T18:51:02.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.465+0000 7f3b46ffd700 1 --2- 172.21.15.1:0/2858824960 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3b3406ac90 0x7f3b3406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:02.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.465+0000 7f3b46ffd700 1 -- 172.21.15.1:0/2858824960 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f3b4c089530 con 0x7f3b5015b440 2024-03-18T18:51:02.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.466+0000 7f3b563fb700 1 --2- 172.21.15.1:0/2858824960 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3b3406ac90 0x7f3b3406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:02.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.466+0000 7f3b563fb700 1 --2- 172.21.15.1:0/2858824960 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3b3406ac90 0x7f3b3406d150 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3b40015040 tx=0x7f3b40007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:02.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.470+0000 7f3b46ffd700 1 -- 172.21.15.1:0/2858824960 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3b5014c380 con 0x7f3b5015b440 2024-03-18T18:51:02.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:02.673+0000 7f3b5865f700 1 -- 172.21.15.1:0/2858824960 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3b5014c380 con 0x7f3b5015b440 2024-03-18T18:51:03.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.263+0000 7f3b46ffd700 1 -- 172.21.15.1:0/2858824960 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3b5014c380 con 0x7f3b5015b440 2024-03-18T18:51:03.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.267+0000 7f3b5865f700 1 -- 172.21.15.1:0/2858824960 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3b3406ac90 msgr2=0x7f3b3406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:03.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.267+0000 7f3b5865f700 1 --2- 172.21.15.1:0/2858824960 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3b3406ac90 0x7f3b3406d150 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3b40015040 tx=0x7f3b40007000 comp rx=0 tx=0).stop 2024-03-18T18:51:03.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.267+0000 7f3b5865f700 1 -- 172.21.15.1:0/2858824960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b5015b440 msgr2=0x7f3b500a0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:03.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.267+0000 7f3b5865f700 1 --2- 172.21.15.1:0/2858824960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b5015b440 0x7f3b500a0ad0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00f7f0 tx=0x7f3b4c00fbb0 comp rx=0 tx=0).stop 2024-03-18T18:51:03.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.268+0000 7f3b5865f700 1 -- 172.21.15.1:0/2858824960 shutdown_connections 2024-03-18T18:51:03.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.268+0000 7f3b5865f700 1 --2- 172.21.15.1:0/2858824960 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3b3406ac90 0x7f3b3406d150 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:03.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.268+0000 7f3b5865f700 1 --2- 172.21.15.1:0/2858824960 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3b500c9500 0x7f3b500c9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:03.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.268+0000 7f3b5865f700 1 --2- 172.21.15.1:0/2858824960 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b3c003a30 0x7f3b5015e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:03.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.268+0000 7f3b5865f700 1 --2- 172.21.15.1:0/2858824960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b5015b440 0x7f3b500a0ad0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:03.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.268+0000 7f3b5865f700 1 -- 172.21.15.1:0/2858824960 >> 172.21.15.1:0/2858824960 conn(0x7f3b5009c630 msgr2=0x7f3b5006cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:03.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.268+0000 7f3b5865f700 1 -- 172.21.15.1:0/2858824960 shutdown_connections 2024-03-18T18:51:03.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.268+0000 7f3b5865f700 1 -- 172.21.15.1:0/2858824960 wait complete. 2024-03-18T18:51:03.265 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-18T18:51:03.279 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T18:51:03.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.420+0000 7f948d73f700 1 Processor -- start 2024-03-18T18:51:03.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.420+0000 7f948d73f700 1 -- start start 2024-03-18T18:51:03.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.420+0000 7f948d73f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f948815d8b0 0x7f94880d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:03.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.420+0000 7f948d73f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94880d0040 0x7f94880cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:03.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.420+0000 7f948d73f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f948815dc90 con 0x7f94880d1170 2024-03-18T18:51:03.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.420+0000 7f948d73f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f948815ec90 con 0x7f94880d0040 2024-03-18T18:51:03.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.421+0000 7f948d73f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f948815ded0 con 0x7f948815d8b0 2024-03-18T18:51:03.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.421+0000 7f94877fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94880d0040 0x7f94880cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:03.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.421+0000 7f94877fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94880d0040 0x7f94880cde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38718/0 (socket says 172.21.15.1:38718) 2024-03-18T18:51:03.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.421+0000 7f94877fe700 1 -- 172.21.15.1:0/2765791508 learned_addr learned my addr 172.21.15.1:0/2765791508 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:03.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.421+0000 7f9486ffd700 1 --2- 172.21.15.1:0/2765791508 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f948815d8b0 0x7f94880d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:03.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.421+0000 7f94877fe700 1 -- 172.21.15.1:0/2765791508 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f948815d8b0 msgr2=0x7f94880d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:03.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.421+0000 7f94877fe700 1 --2- 172.21.15.1:0/2765791508 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f948815d8b0 0x7f94880d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:03.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.421+0000 7f94877fe700 1 -- 172.21.15.1:0/2765791508 >> v1:172.21.15.165:6789/0 conn(0x7f94880d1170 legacy=0x7f94880cf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:03.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.421+0000 7f94877fe700 1 -- 172.21.15.1:0/2765791508 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948815e1b0 con 0x7f94880d0040 2024-03-18T18:51:03.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.421+0000 7f94877fe700 1 --2- 172.21.15.1:0/2765791508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94880d0040 0x7f94880cde10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9480009e50 tx=0x7f948000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484a7d7303bb6cfe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:03.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.421+0000 7f9485ffb700 1 -- 172.21.15.1:0/2765791508 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948000e070 con 0x7f94880d0040 2024-03-18T18:51:03.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.421+0000 7f9485ffb700 1 -- 172.21.15.1:0/2765791508 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9480003a20 con 0x7f94880d0040 2024-03-18T18:51:03.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.421+0000 7f9485ffb700 1 -- 172.21.15.1:0/2765791508 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9480003dd0 con 0x7f94880d0040 2024-03-18T18:51:03.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.422+0000 7f948d73f700 1 -- 172.21.15.1:0/2765791508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94880d0040 msgr2=0x7f94880cde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:03.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.422+0000 7f948d73f700 1 --2- 172.21.15.1:0/2765791508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94880d0040 0x7f94880cde10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9480009e50 tx=0x7f948000b330 comp rx=0 tx=0).stop 2024-03-18T18:51:03.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.422+0000 7f948d73f700 1 -- 172.21.15.1:0/2765791508 shutdown_connections 2024-03-18T18:51:03.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.422+0000 7f948d73f700 1 --2- 172.21.15.1:0/2765791508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94880d0040 0x7f94880cde10 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:03.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.422+0000 7f948d73f700 1 --2- 172.21.15.1:0/2765791508 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f948815d8b0 0x7f94880d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:03.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.422+0000 7f948d73f700 1 -- 172.21.15.1:0/2765791508 >> 172.21.15.1:0/2765791508 conn(0x7f948809c640 msgr2=0x7f94880d3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:03.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.422+0000 7f948d73f700 1 -- 172.21.15.1:0/2765791508 shutdown_connections 2024-03-18T18:51:03.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.422+0000 7f948d73f700 1 -- 172.21.15.1:0/2765791508 wait complete. 2024-03-18T18:51:03.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.424+0000 7f948d73f700 1 Processor -- start 2024-03-18T18:51:03.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.424+0000 7f948d73f700 1 -- start start 2024-03-18T18:51:03.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f948d73f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94880d0040 0x7f94880ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:03.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f948d73f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94880d1170 0x7f94881578b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:03.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f948d73f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f948815d8b0 0x7f9488158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:03.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f948d73f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948815ac90 con 0x7f94880d1170 2024-03-18T18:51:03.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f948d73f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9488159ed0 con 0x7f948815d8b0 2024-03-18T18:51:03.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f948d73f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9488159bf0 con 0x7f94880d0040 2024-03-18T18:51:03.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f94867fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94880d1170 0x7f94881578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:03.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f94867fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94880d1170 0x7f94881578b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38730/0 (socket says 172.21.15.1:38730) 2024-03-18T18:51:03.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f94867fc700 1 -- 172.21.15.1:0/2036141581 learned_addr learned my addr 172.21.15.1:0/2036141581 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:03.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f9486ffd700 1 --2- 172.21.15.1:0/2036141581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94880d0040 0x7f94880ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:03.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f94877fe700 1 --2- 172.21.15.1:0/2036141581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f948815d8b0 0x7f9488158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:03.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f94867fc700 1 -- 172.21.15.1:0/2036141581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94880d0040 msgr2=0x7f94880ca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:03.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f94867fc700 1 --2- 172.21.15.1:0/2036141581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94880d0040 0x7f94880ca3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:03.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f94867fc700 1 -- 172.21.15.1:0/2036141581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f948815d8b0 msgr2=0x7f9488158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:03.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f94867fc700 1 --2- 172.21.15.1:0/2036141581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f948815d8b0 0x7f9488158fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:03.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f94867fc700 1 -- 172.21.15.1:0/2036141581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9480009b30 con 0x7f94880d1170 2024-03-18T18:51:03.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f94877fe700 1 --2- 172.21.15.1:0/2036141581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f948815d8b0 0x7f9488158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:03.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f94867fc700 1 --2- 172.21.15.1:0/2036141581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94880d1170 0x7f94881578b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f947c00c4d0 tx=0x7f947c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:03.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f946f7fe700 1 -- 172.21.15.1:0/2036141581 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f947c011790 con 0x7f94880d1170 2024-03-18T18:51:03.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.425+0000 7f946f7fe700 1 -- 172.21.15.1:0/2036141581 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f947c0118f0 con 0x7f94880d1170 2024-03-18T18:51:03.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.426+0000 7f948d73f700 1 -- 172.21.15.1:0/2036141581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f948815a640 con 0x7f94880d1170 2024-03-18T18:51:03.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.426+0000 7f946f7fe700 1 -- 172.21.15.1:0/2036141581 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f947c011ca0 con 0x7f94880d1170 2024-03-18T18:51:03.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.426+0000 7f948d73f700 1 -- 172.21.15.1:0/2036141581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9488157df0 con 0x7f94880d1170 2024-03-18T18:51:03.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.427+0000 7f948d73f700 1 -- 172.21.15.1:0/2036141581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94880d15b0 con 0x7f94880d1170 2024-03-18T18:51:03.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.427+0000 7f946f7fe700 1 -- 172.21.15.1:0/2036141581 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f947c01c460 con 0x7f94880d1170 2024-03-18T18:51:03.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.427+0000 7f946f7fe700 1 --2- 172.21.15.1:0/2036141581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f947406ac90 0x7f947406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:03.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.428+0000 7f946f7fe700 1 -- 172.21.15.1:0/2036141581 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6311+0+0 (secure 0 0 0) 0x7f947c089380 con 0x7f94880d1170 2024-03-18T18:51:03.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.428+0000 7f9486ffd700 1 --2- 172.21.15.1:0/2036141581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f947406ac90 0x7f947406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:03.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.428+0000 7f9486ffd700 1 --2- 172.21.15.1:0/2036141581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f947406ac90 0x7f947406d150 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9478006b20 tx=0x7f9478008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:03.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.432+0000 7f946f7fe700 1 -- 172.21.15.1:0/2036141581 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f94880d15b0 con 0x7f94880d1170 2024-03-18T18:51:03.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:03.636+0000 7f948d73f700 1 -- 172.21.15.1:0/2036141581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f94880d15b0 con 0x7f94880d1170 2024-03-18T18:51:04.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.267+0000 7f946f7fe700 1 -- 172.21.15.1:0/2036141581 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) v1 ==== 169+0+0 (secure 0 0 0) 0x7f94880d15b0 con 0x7f94880d1170 2024-03-18T18:51:04.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.270+0000 7f948d73f700 1 -- 172.21.15.1:0/2036141581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f947406ac90 msgr2=0x7f947406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:04.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.270+0000 7f948d73f700 1 --2- 172.21.15.1:0/2036141581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f947406ac90 0x7f947406d150 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9478006b20 tx=0x7f9478008040 comp rx=0 tx=0).stop 2024-03-18T18:51:04.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.270+0000 7f948d73f700 1 -- 172.21.15.1:0/2036141581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94880d1170 msgr2=0x7f94881578b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:04.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.270+0000 7f948d73f700 1 --2- 172.21.15.1:0/2036141581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94880d1170 0x7f94881578b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f947c00c4d0 tx=0x7f947c00c7e0 comp rx=0 tx=0).stop 2024-03-18T18:51:04.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.270+0000 7f948d73f700 1 -- 172.21.15.1:0/2036141581 shutdown_connections 2024-03-18T18:51:04.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.270+0000 7f948d73f700 1 --2- 172.21.15.1:0/2036141581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f947406ac90 0x7f947406d150 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:04.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.270+0000 7f948d73f700 1 --2- 172.21.15.1:0/2036141581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94880d0040 0x7f94880ca3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:04.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.270+0000 7f948d73f700 1 --2- 172.21.15.1:0/2036141581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f948815d8b0 0x7f9488158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:04.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.270+0000 7f948d73f700 1 --2- 172.21.15.1:0/2036141581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94880d1170 0x7f94881578b0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:04.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.270+0000 7f948d73f700 1 -- 172.21.15.1:0/2036141581 >> 172.21.15.1:0/2036141581 conn(0x7f948809c640 msgr2=0x7f94880ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:04.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.270+0000 7f948d73f700 1 -- 172.21.15.1:0/2036141581 shutdown_connections 2024-03-18T18:51:04.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.270+0000 7f948d73f700 1 -- 172.21.15.1:0/2036141581 wait complete. 2024-03-18T18:51:04.268 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-18T18:51:04.283 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-18T18:51:04.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.422+0000 7f3ad8cbe700 1 Processor -- start 2024-03-18T18:51:04.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.422+0000 7f3ad8cbe700 1 -- start start 2024-03-18T18:51:04.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.423+0000 7f3ad8cbe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3ad406fe10 0x7f3ad40b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:04.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.423+0000 7f3ad8cbe700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3ad406ae60 0x7f3ad406b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:04.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.423+0000 7f3ad8cbe700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad415d970 con 0x7f3ad415d590 2024-03-18T18:51:04.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.423+0000 7f3ad8cbe700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ad4072320 con 0x7f3ad406fe10 2024-03-18T18:51:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.423+0000 7f3ad8cbe700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ad4072600 con 0x7f3ad406ae60 2024-03-18T18:51:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.424+0000 7f3ad1d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3ad406fe10 0x7f3ad40b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.424+0000 7f3ad1d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3ad406fe10 0x7f3ad40b3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53528/0 (socket says 172.21.15.1:53528) 2024-03-18T18:51:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.424+0000 7f3ad1d9b700 1 -- 172.21.15.1:0/4065655362 learned_addr learned my addr 172.21.15.1:0/4065655362 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.424+0000 7f3ad2d9d700 1 --2- 172.21.15.1:0/4065655362 >> v2:172.21.15.165:3300/0 conn(0x7f3ad406ae60 0x7f3ad406b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.424+0000 7f3ad1d9b700 1 -- 172.21.15.1:0/4065655362 >> v2:172.21.15.165:3300/0 conn(0x7f3ad406ae60 msgr2=0x7f3ad406b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.424+0000 7f3ad1d9b700 1 --2- 172.21.15.1:0/4065655362 >> v2:172.21.15.165:3300/0 conn(0x7f3ad406ae60 0x7f3ad406b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.424+0000 7f3ad1d9b700 1 -- 172.21.15.1:0/4065655362 >> v1:172.21.15.165:6789/0 conn(0x7f3ad415d590 legacy=0x7f3ad406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.424+0000 7f3ad1d9b700 1 -- 172.21.15.1:0/4065655362 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad40b30b0 con 0x7f3ad406fe10 2024-03-18T18:51:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.424+0000 7f3ad2d9d700 1 --2- 172.21.15.1:0/4065655362 >> v2:172.21.15.165:3300/0 conn(0x7f3ad406ae60 0x7f3ad406b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.424+0000 7f3ad1d9b700 1 --2- 172.21.15.1:0/4065655362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad406fe10 0x7f3ad40b3bc0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3ad40728e0 tx=0x7f3ac4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c851526b84d2c4f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.424+0000 7f3ad159a700 1 -- 172.21.15.1:0/4065655362 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4004230 con 0x7f3ad406fe10 2024-03-18T18:51:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.424+0000 7f3ad159a700 1 -- 172.21.15.1:0/4065655362 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac4004390 con 0x7f3ad406fe10 2024-03-18T18:51:04.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.424+0000 7f3ad159a700 1 -- 172.21.15.1:0/4065655362 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac40036a0 con 0x7f3ad406fe10 2024-03-18T18:51:04.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.425+0000 7f3ad8cbe700 1 -- 172.21.15.1:0/4065655362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad406fe10 msgr2=0x7f3ad40b3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:04.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.425+0000 7f3ad8cbe700 1 --2- 172.21.15.1:0/4065655362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad406fe10 0x7f3ad40b3bc0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3ad40728e0 tx=0x7f3ac4009bd0 comp rx=0 tx=0).stop 2024-03-18T18:51:04.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.425+0000 7f3ad8cbe700 1 -- 172.21.15.1:0/4065655362 shutdown_connections 2024-03-18T18:51:04.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.425+0000 7f3ad8cbe700 1 --2- 172.21.15.1:0/4065655362 >> v2:172.21.15.165:3300/0 conn(0x7f3ad406ae60 0x7f3ad406b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:04.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.425+0000 7f3ad8cbe700 1 --2- 172.21.15.1:0/4065655362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad406fe10 0x7f3ad40b3bc0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:04.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.425+0000 7f3ad8cbe700 1 -- 172.21.15.1:0/4065655362 >> 172.21.15.1:0/4065655362 conn(0x7f3ad40d00c0 msgr2=0x7f3ad40d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:04.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.425+0000 7f3ad8cbe700 1 -- 172.21.15.1:0/4065655362 shutdown_connections 2024-03-18T18:51:04.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.425+0000 7f3ad8cbe700 1 -- 172.21.15.1:0/4065655362 wait complete. 2024-03-18T18:51:04.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.426+0000 7f3ad8cbe700 1 Processor -- start 2024-03-18T18:51:04.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.426+0000 7f3ad8cbe700 1 -- start start 2024-03-18T18:51:04.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.427+0000 7f3ad8cbe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad406ae60 0x7f3ad4150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:04.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.427+0000 7f3ad8cbe700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ad406fe10 0x7f3ad413c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:04.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.427+0000 7f3ad8cbe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ad415d590 0x7f3ad41560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:04.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.427+0000 7f3ad8cbe700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad4150cc0 con 0x7f3ad406ae60 2024-03-18T18:51:04.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.427+0000 7f3ad8cbe700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad4157600 con 0x7f3ad406fe10 2024-03-18T18:51:04.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.427+0000 7f3ad8cbe700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad413cc10 con 0x7f3ad415d590 2024-03-18T18:51:04.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.427+0000 7f3ad259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad406ae60 0x7f3ad4150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:04.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.427+0000 7f3ad259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad406ae60 0x7f3ad4150780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53540/0 (socket says 172.21.15.1:53540) 2024-03-18T18:51:04.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.427+0000 7f3ad259c700 1 -- 172.21.15.1:0/867357087 learned_addr learned my addr 172.21.15.1:0/867357087 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.427+0000 7f3ad2d9d700 1 --2- 172.21.15.1:0/867357087 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ad415d590 0x7f3ad41560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.427+0000 7f3ad1d9b700 1 --2- 172.21.15.1:0/867357087 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ad406fe10 0x7f3ad413c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.427+0000 7f3ad259c700 1 -- 172.21.15.1:0/867357087 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ad415d590 msgr2=0x7f3ad41560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.427+0000 7f3ad259c700 1 --2- 172.21.15.1:0/867357087 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ad415d590 0x7f3ad41560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.427+0000 7f3ad259c700 1 -- 172.21.15.1:0/867357087 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ad406fe10 msgr2=0x7f3ad413c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.427+0000 7f3ad259c700 1 --2- 172.21.15.1:0/867357087 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ad406fe10 0x7f3ad413c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.427+0000 7f3ad259c700 1 -- 172.21.15.1:0/867357087 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac4009550 con 0x7f3ad406ae60 2024-03-18T18:51:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.427+0000 7f3ad1d9b700 1 --2- 172.21.15.1:0/867357087 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ad406fe10 0x7f3ad413c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.428+0000 7f3ad259c700 1 --2- 172.21.15.1:0/867357087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad406ae60 0x7f3ad4150780 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00c4d0 tx=0x7f3abc00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.428+0000 7f3acaffd700 1 -- 172.21.15.1:0/867357087 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc011790 con 0x7f3ad406ae60 2024-03-18T18:51:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.428+0000 7f3acaffd700 1 -- 172.21.15.1:0/867357087 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3abc0118f0 con 0x7f3ad406ae60 2024-03-18T18:51:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.428+0000 7f3ad8cbe700 1 -- 172.21.15.1:0/867357087 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad413cef0 con 0x7f3ad406ae60 2024-03-18T18:51:04.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.428+0000 7f3acaffd700 1 -- 172.21.15.1:0/867357087 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc011ca0 con 0x7f3ad406ae60 2024-03-18T18:51:04.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.428+0000 7f3ad8cbe700 1 -- 172.21.15.1:0/867357087 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad413d4b0 con 0x7f3ad406ae60 2024-03-18T18:51:04.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.429+0000 7f3ad8cbe700 1 -- 172.21.15.1:0/867357087 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad4157800 con 0x7f3ad406ae60 2024-03-18T18:51:04.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.429+0000 7f3acaffd700 1 -- 172.21.15.1:0/867357087 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3abc01c460 con 0x7f3ad406ae60 2024-03-18T18:51:04.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.430+0000 7f3acaffd700 1 --2- 172.21.15.1:0/867357087 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ab406ac90 0x7f3ab406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:04.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.430+0000 7f3acaffd700 1 -- 172.21.15.1:0/867357087 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f3abc0890b0 con 0x7f3ad406ae60 2024-03-18T18:51:04.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.430+0000 7f3ad1d9b700 1 --2- 172.21.15.1:0/867357087 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ab406ac90 0x7f3ab406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:04.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.430+0000 7f3ad1d9b700 1 --2- 172.21.15.1:0/867357087 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ab406ac90 0x7f3ab406d150 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009130 tx=0x7f3ac4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:04.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.434+0000 7f3acaffd700 1 -- 172.21.15.1:0/867357087 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3abc0542d0 con 0x7f3ad406ae60 2024-03-18T18:51:04.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:04.639+0000 7f3ad8cbe700 1 -- 172.21.15.1:0/867357087 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3ad40b0c90 con 0x7f3ad406ae60 2024-03-18T18:51:05.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.269+0000 7f3acaffd700 1 -- 172.21.15.1:0/867357087 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) v1 ==== 178+0+0 (secure 0 0 0) 0x7f3abc0578f0 con 0x7f3ad406ae60 2024-03-18T18:51:05.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.273+0000 7f3ad8cbe700 1 -- 172.21.15.1:0/867357087 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ab406ac90 msgr2=0x7f3ab406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:05.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.273+0000 7f3ad8cbe700 1 --2- 172.21.15.1:0/867357087 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ab406ac90 0x7f3ab406d150 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009130 tx=0x7f3ac4008030 comp rx=0 tx=0).stop 2024-03-18T18:51:05.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.273+0000 7f3ad8cbe700 1 -- 172.21.15.1:0/867357087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad406ae60 msgr2=0x7f3ad4150780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:05.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.273+0000 7f3ad8cbe700 1 --2- 172.21.15.1:0/867357087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad406ae60 0x7f3ad4150780 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00c4d0 tx=0x7f3abc00c7e0 comp rx=0 tx=0).stop 2024-03-18T18:51:05.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.277+0000 7f3ad8cbe700 1 -- 172.21.15.1:0/867357087 shutdown_connections 2024-03-18T18:51:05.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.277+0000 7f3ad8cbe700 1 --2- 172.21.15.1:0/867357087 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ab406ac90 0x7f3ab406d150 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:05.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.277+0000 7f3ad8cbe700 1 --2- 172.21.15.1:0/867357087 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ad415d590 0x7f3ad41560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:05.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.277+0000 7f3ad8cbe700 1 --2- 172.21.15.1:0/867357087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ad406ae60 0x7f3ad4150780 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:05.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.277+0000 7f3ad8cbe700 1 --2- 172.21.15.1:0/867357087 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ad406fe10 0x7f3ad413c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:05.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.277+0000 7f3ad8cbe700 1 -- 172.21.15.1:0/867357087 >> 172.21.15.1:0/867357087 conn(0x7f3ad40d00c0 msgr2=0x7f3ad40afd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:05.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.277+0000 7f3ad8cbe700 1 -- 172.21.15.1:0/867357087 shutdown_connections 2024-03-18T18:51:05.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.277+0000 7f3ad8cbe700 1 -- 172.21.15.1:0/867357087 wait complete. 2024-03-18T18:51:05.275 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data_ec' removed 2024-03-18T18:51:05.294 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T18:51:05.294 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-18T18:51:05.294 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T18:51:05.294 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T18:51:05.294 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T18:51:05.294 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-18T18:51:05.295 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-18T18:51:05.295 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T18:51:05.295 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T18:51:05.295 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-18T18:51:05.295 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-18T18:51:05.296 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T18:51:05.296 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T18:51:05.296 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-18T18:51:05.296 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-18T18:51:05.296 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T18:51:05.307 INFO:tasks.ceph.mds.b:Stopped 2024-03-18T18:51:05.307 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-18T18:51:05.308 INFO:tasks.ceph.mds.d:Stopped 2024-03-18T18:51:05.308 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-18T18:51:05.309 INFO:tasks.ceph.mds.c:Stopped 2024-03-18T18:51:05.309 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-18T18:51:05.310 INFO:tasks.ceph.mds.a:Stopped 2024-03-18T18:51:05.310 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-18T18:51:05.349 INFO:tasks.ceph.mds.b:Started 2024-03-18T18:51:05.350 DEBUG:teuthology.parallel:result is None 2024-03-18T18:51:05.351 INFO:tasks.ceph.mds.d:Started 2024-03-18T18:51:05.351 DEBUG:teuthology.parallel:result is None 2024-03-18T18:51:05.352 INFO:tasks.ceph.mds.c:Started 2024-03-18T18:51:05.352 DEBUG:teuthology.parallel:result is None 2024-03-18T18:51:05.352 INFO:tasks.ceph.mds.a:Started 2024-03-18T18:51:05.352 DEBUG:teuthology.parallel:result is None 2024-03-18T18:51:05.353 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-18T18:51:05.439 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-03-18T18:51:05.439 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-03-18T18:51:05.441 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T18:51:05.441 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-18T18:51:05.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.512+0000 7fe28661d700 1 Processor -- start 2024-03-18T18:51:05.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.512+0000 7fe28661d700 1 -- start start 2024-03-18T18:51:05.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.513+0000 7fe28661d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe2800cfee0 0x7fe2800d3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:05.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.513+0000 7fe28661d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe2800d1160 0x7fe280160780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:05.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.513+0000 7fe28661d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2800d1670 con 0x7fe2800d02c0 2024-03-18T18:51:05.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.513+0000 7fe28661d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2800d22a0 con 0x7fe2800cfee0 2024-03-18T18:51:05.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.513+0000 7fe28661d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2800b25f0 con 0x7fe2800d1160 2024-03-18T18:51:05.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.513+0000 7fe27ffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe2800cfee0 0x7fe2800d3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:05.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.513+0000 7fe284bba700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe2800d02c0 0x7fe2800ce0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57380/0 (socket says 172.21.15.1:57380) 2024-03-18T18:51:05.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.513+0000 7fe284bba700 1 -- 172.21.15.1:0/2039164196 learned_addr learned my addr 172.21.15.1:0/2039164196 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:05.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.513+0000 7fe27f7fe700 1 --2- 172.21.15.1:0/2039164196 >> v2:172.21.15.165:3300/0 conn(0x7fe2800d1160 0x7fe280160780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:05.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.514+0000 7fe27ffff700 1 -- 172.21.15.1:0/2039164196 >> v2:172.21.15.165:3300/0 conn(0x7fe2800d1160 msgr2=0x7fe280160780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:05.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.514+0000 7fe27ffff700 1 --2- 172.21.15.1:0/2039164196 >> v2:172.21.15.165:3300/0 conn(0x7fe2800d1160 0x7fe280160780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:05.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.514+0000 7fe27ffff700 1 -- 172.21.15.1:0/2039164196 >> v1:172.21.15.1:6789/0 conn(0x7fe2800d02c0 legacy=0x7fe2800ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:05.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.514+0000 7fe27ffff700 1 -- 172.21.15.1:0/2039164196 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2800d1970 con 0x7fe2800cfee0 2024-03-18T18:51:05.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.514+0000 7fe27f7fe700 1 --2- 172.21.15.1:0/2039164196 >> v2:172.21.15.165:3300/0 conn(0x7fe2800d1160 0x7fe280160780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:05.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.514+0000 7fe27effd700 1 -- 172.21.15.1:0/2039164196 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952743609 0 0) 0x7fe2800d1670 con 0x7fe2800d02c0 2024-03-18T18:51:05.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.514+0000 7fe27ffff700 1 --2- 172.21.15.1:0/2039164196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2800cfee0 0x7fe2800d3d80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe2800b9310 tx=0x7fe274009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e708c8aec73f0d0f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:05.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.514+0000 7fe27effd700 1 -- 172.21.15.1:0/2039164196 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe274011830 con 0x7fe2800cfee0 2024-03-18T18:51:05.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.514+0000 7fe27effd700 1 -- 172.21.15.1:0/2039164196 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe274011990 con 0x7fe2800cfee0 2024-03-18T18:51:05.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.514+0000 7fe27effd700 1 -- 172.21.15.1:0/2039164196 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe274011d20 con 0x7fe2800cfee0 2024-03-18T18:51:05.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.514+0000 7fe28661d700 1 -- 172.21.15.1:0/2039164196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2800cfee0 msgr2=0x7fe2800d3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:05.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.514+0000 7fe28661d700 1 --2- 172.21.15.1:0/2039164196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2800cfee0 0x7fe2800d3d80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe2800b9310 tx=0x7fe274009b20 comp rx=0 tx=0).stop 2024-03-18T18:51:05.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.514+0000 7fe28661d700 1 -- 172.21.15.1:0/2039164196 shutdown_connections 2024-03-18T18:51:05.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.514+0000 7fe28661d700 1 --2- 172.21.15.1:0/2039164196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2800cfee0 0x7fe2800d3d80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:05.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.514+0000 7fe28661d700 1 --2- 172.21.15.1:0/2039164196 >> v2:172.21.15.165:3300/0 conn(0x7fe2800d1160 0x7fe280160780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:05.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.514+0000 7fe28661d700 1 -- 172.21.15.1:0/2039164196 >> 172.21.15.1:0/2039164196 conn(0x7fe2800d3260 msgr2=0x7fe2800d3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:05.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.515+0000 7fe28661d700 1 -- 172.21.15.1:0/2039164196 shutdown_connections 2024-03-18T18:51:05.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.515+0000 7fe28661d700 1 -- 172.21.15.1:0/2039164196 wait complete. 2024-03-18T18:51:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.516+0000 7fe28661d700 1 Processor -- start 2024-03-18T18:51:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.516+0000 7fe28661d700 1 -- start start 2024-03-18T18:51:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe28661d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2800cfee0 0x7fe280157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe28661d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2800d02c0 0x7fe2800caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe28661d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2800d1160 0x7fe280146100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe28661d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28015a440 con 0x7fe2800d1160 2024-03-18T18:51:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe28661d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2800c9980 con 0x7fe2800d02c0 2024-03-18T18:51:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe28661d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe280157710 con 0x7fe2800cfee0 2024-03-18T18:51:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe284bba700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2800d1160 0x7fe280146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:05.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe284bba700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2800d1160 0x7fe280146100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53594/0 (socket says 172.21.15.1:53594) 2024-03-18T18:51:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe284bba700 1 -- 172.21.15.1:0/3899712712 learned_addr learned my addr 172.21.15.1:0/3899712712 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe27f7fe700 1 --2- 172.21.15.1:0/3899712712 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2800d02c0 0x7fe2800caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe27ffff700 1 --2- 172.21.15.1:0/3899712712 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2800cfee0 0x7fe280157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe284bba700 1 -- 172.21.15.1:0/3899712712 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2800cfee0 msgr2=0x7fe280157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe284bba700 1 --2- 172.21.15.1:0/3899712712 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2800cfee0 0x7fe280157070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe284bba700 1 -- 172.21.15.1:0/3899712712 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2800d02c0 msgr2=0x7fe2800caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe284bba700 1 --2- 172.21.15.1:0/3899712712 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2800d02c0 0x7fe2800caeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe284bba700 1 -- 172.21.15.1:0/3899712712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2740094a0 con 0x7fe2800d1160 2024-03-18T18:51:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe284bba700 1 --2- 172.21.15.1:0/3899712712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2800d1160 0x7fe280146100 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe27800cb60 tx=0x7fe27800ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe27cff9700 1 -- 172.21.15.1:0/3899712712 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27800a9e0 con 0x7fe2800d1160 2024-03-18T18:51:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe27cff9700 1 -- 172.21.15.1:0/3899712712 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27800ab40 con 0x7fe2800d1160 2024-03-18T18:51:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe28661d700 1 -- 172.21.15.1:0/3899712712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe280159e80 con 0x7fe2800d1160 2024-03-18T18:51:05.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe27cff9700 1 -- 172.21.15.1:0/3899712712 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2780070d0 con 0x7fe2800d1160 2024-03-18T18:51:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.517+0000 7fe28661d700 1 -- 172.21.15.1:0/3899712712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2800cb5c0 con 0x7fe2800d1160 2024-03-18T18:51:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.518+0000 7fe28661d700 1 -- 172.21.15.1:0/3899712712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe280067bc0 con 0x7fe2800d1160 2024-03-18T18:51:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.519+0000 7fe27cff9700 1 -- 172.21.15.1:0/3899712712 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe278007230 con 0x7fe2800d1160 2024-03-18T18:51:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.519+0000 7fe27cff9700 1 --2- 172.21.15.1:0/3899712712 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe26c06ac90 0x7fe26c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.519+0000 7fe27cff9700 1 -- 172.21.15.1:0/3899712712 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fe278088d70 con 0x7fe2800d1160 2024-03-18T18:51:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.519+0000 7fe27ffff700 1 --2- 172.21.15.1:0/3899712712 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe26c06ac90 0x7fe26c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:05.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.520+0000 7fe27ffff700 1 --2- 172.21.15.1:0/3899712712 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe26c06ac90 0x7fe26c06d150 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe2740090b0 tx=0x7fe27401c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:05.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.524+0000 7fe27cff9700 1 -- 172.21.15.1:0/3899712712 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe2780541a0 con 0x7fe2800d1160 2024-03-18T18:51:05.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:05.742+0000 7fe28661d700 1 -- 172.21.15.1:0/3899712712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe2800cb3f0 con 0x7fe2800d1160 2024-03-18T18:51:06.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.480+0000 7fe27cff9700 1 -- 172.21.15.1:0/3899712712 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v36) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe278018080 con 0x7fe2800d1160 2024-03-18T18:51:06.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.483+0000 7fe28661d700 1 -- 172.21.15.1:0/3899712712 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe26c06ac90 msgr2=0x7fe26c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:06.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.483+0000 7fe28661d700 1 --2- 172.21.15.1:0/3899712712 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe26c06ac90 0x7fe26c06d150 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe2740090b0 tx=0x7fe27401c040 comp rx=0 tx=0).stop 2024-03-18T18:51:06.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.483+0000 7fe28661d700 1 -- 172.21.15.1:0/3899712712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2800d1160 msgr2=0x7fe280146100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:06.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.483+0000 7fe28661d700 1 --2- 172.21.15.1:0/3899712712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2800d1160 0x7fe280146100 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe27800cb60 tx=0x7fe27800ce70 comp rx=0 tx=0).stop 2024-03-18T18:51:06.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.483+0000 7fe28661d700 1 -- 172.21.15.1:0/3899712712 shutdown_connections 2024-03-18T18:51:06.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.483+0000 7fe28661d700 1 --2- 172.21.15.1:0/3899712712 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe26c06ac90 0x7fe26c06d150 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:06.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.483+0000 7fe28661d700 1 --2- 172.21.15.1:0/3899712712 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2800cfee0 0x7fe280157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:06.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.483+0000 7fe28661d700 1 --2- 172.21.15.1:0/3899712712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2800d1160 0x7fe280146100 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:06.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.483+0000 7fe28661d700 1 --2- 172.21.15.1:0/3899712712 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2800d02c0 0x7fe2800caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:06.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.483+0000 7fe28661d700 1 -- 172.21.15.1:0/3899712712 >> 172.21.15.1:0/3899712712 conn(0x7fe2800d3260 msgr2=0x7fe28006cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:06.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.483+0000 7fe28661d700 1 -- 172.21.15.1:0/3899712712 shutdown_connections 2024-03-18T18:51:06.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.483+0000 7fe28661d700 1 -- 172.21.15.1:0/3899712712 wait complete. 2024-03-18T18:51:06.481 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-18T18:51:06.495 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-18T18:51:06.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.632+0000 7f3e9ea67700 1 Processor -- start 2024-03-18T18:51:06.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.632+0000 7f3e9ea67700 1 -- start start 2024-03-18T18:51:06.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.633+0000 7f3e9ea67700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3e98146190 0x7f3e9806a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:06.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.633+0000 7f3e9ea67700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3e981465a0 0x7f3e9806ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:06.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.633+0000 7f3e9ea67700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e98146af0 0x7f3e98071650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:06.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.633+0000 7f3e9ea67700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e9806f850 con 0x7f3e98146190 2024-03-18T18:51:06.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.633+0000 7f3e9ea67700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e9806ebd0 con 0x7f3e981465a0 2024-03-18T18:51:06.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.633+0000 7f3e9ea67700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9806eeb0 con 0x7f3e98146af0 2024-03-18T18:51:06.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.633+0000 7f3e9c803700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3e98146190 0x7f3e9806a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:06.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.633+0000 7f3e9c803700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3e98146190 0x7f3e9806a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53602/0 (socket says 172.21.15.1:53602) 2024-03-18T18:51:06.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.633+0000 7f3e9c803700 1 -- 172.21.15.1:0/1274232363 learned_addr learned my addr 172.21.15.1:0/1274232363 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:06.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.633+0000 7f3e97fff700 1 --2- 172.21.15.1:0/1274232363 >> v2:172.21.15.165:3300/0 conn(0x7f3e981465a0 0x7f3e9806ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:06.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.633+0000 7f3e9d004700 1 --2- 172.21.15.1:0/1274232363 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e98146af0 0x7f3e98071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:06.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.633+0000 7f3e9c803700 1 -- 172.21.15.1:0/1274232363 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e98146af0 msgr2=0x7f3e98071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:06.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.633+0000 7f3e9c803700 1 --2- 172.21.15.1:0/1274232363 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e98146af0 0x7f3e98071650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:06.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.633+0000 7f3e9c803700 1 -- 172.21.15.1:0/1274232363 >> v2:172.21.15.165:3300/0 conn(0x7f3e981465a0 msgr2=0x7f3e9806ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:06.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.633+0000 7f3e9c803700 1 --2- 172.21.15.1:0/1274232363 >> v2:172.21.15.165:3300/0 conn(0x7f3e981465a0 0x7f3e9806ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:06.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.633+0000 7f3e9c803700 1 -- 172.21.15.1:0/1274232363 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9806f190 con 0x7f3e98146190 2024-03-18T18:51:06.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.633+0000 7f3e9c803700 1 --2- 172.21.15.1:0/1274232363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e98146190 0x7f3e9806a750 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3e980731a0 tx=0x7f3e8c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c572c1944c3e0bcd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:06.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.634+0000 7f3e977fe700 1 -- 172.21.15.1:0/1274232363 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c00b870 con 0x7f3e98146190 2024-03-18T18:51:06.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.634+0000 7f3e977fe700 1 -- 172.21.15.1:0/1274232363 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8c00b9d0 con 0x7f3e98146190 2024-03-18T18:51:06.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.634+0000 7f3e977fe700 1 -- 172.21.15.1:0/1274232363 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c00bd60 con 0x7f3e98146190 2024-03-18T18:51:06.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.634+0000 7f3e9ea67700 1 -- 172.21.15.1:0/1274232363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e98146190 msgr2=0x7f3e9806a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:06.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.634+0000 7f3e9ea67700 1 --2- 172.21.15.1:0/1274232363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e98146190 0x7f3e9806a750 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3e980731a0 tx=0x7f3e8c009bd0 comp rx=0 tx=0).stop 2024-03-18T18:51:06.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.634+0000 7f3e9ea67700 1 -- 172.21.15.1:0/1274232363 shutdown_connections 2024-03-18T18:51:06.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.634+0000 7f3e9ea67700 1 --2- 172.21.15.1:0/1274232363 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e98146af0 0x7f3e98071650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:06.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.634+0000 7f3e9ea67700 1 --2- 172.21.15.1:0/1274232363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e98146190 0x7f3e9806a750 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:06.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.634+0000 7f3e9ea67700 1 --2- 172.21.15.1:0/1274232363 >> v2:172.21.15.165:3300/0 conn(0x7f3e981465a0 0x7f3e9806ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:06.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.634+0000 7f3e9ea67700 1 -- 172.21.15.1:0/1274232363 >> 172.21.15.1:0/1274232363 conn(0x7f3e980b1d20 msgr2=0x7f3e980b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:06.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.634+0000 7f3e9ea67700 1 -- 172.21.15.1:0/1274232363 shutdown_connections 2024-03-18T18:51:06.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.634+0000 7f3e9ea67700 1 -- 172.21.15.1:0/1274232363 wait complete. 2024-03-18T18:51:06.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.636+0000 7f3e9ea67700 1 Processor -- start 2024-03-18T18:51:06.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.636+0000 7f3e9ea67700 1 -- start start 2024-03-18T18:51:06.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.636+0000 7f3e9ea67700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e981465a0 0x7f3e980ce460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:06.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e9ea67700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e98146af0 0x7f3e9813ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:06.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e9ea67700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e98140080 0x7f3e980c50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:06.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e9ea67700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e98142810 con 0x7f3e98140080 2024-03-18T18:51:06.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e9ea67700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e980cf400 con 0x7f3e981465a0 2024-03-18T18:51:06.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e9ea67700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9813f1f0 con 0x7f3e98146af0 2024-03-18T18:51:06.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e9d004700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e98140080 0x7f3e980c50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:06.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e9d004700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e98140080 0x7f3e980c50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53612/0 (socket says 172.21.15.1:53612) 2024-03-18T18:51:06.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e9d004700 1 -- 172.21.15.1:0/1947158388 learned_addr learned my addr 172.21.15.1:0/1947158388 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:06.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e97fff700 1 --2- 172.21.15.1:0/1947158388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e98146af0 0x7f3e9813ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:06.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e9c803700 1 --2- 172.21.15.1:0/1947158388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e981465a0 0x7f3e980ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:06.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e9d004700 1 -- 172.21.15.1:0/1947158388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e98146af0 msgr2=0x7f3e9813ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:06.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e9d004700 1 --2- 172.21.15.1:0/1947158388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e98146af0 0x7f3e9813ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:06.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e9d004700 1 -- 172.21.15.1:0/1947158388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e981465a0 msgr2=0x7f3e980ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:06.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e9d004700 1 --2- 172.21.15.1:0/1947158388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e981465a0 0x7f3e980ce460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:06.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e9d004700 1 -- 172.21.15.1:0/1947158388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8c009550 con 0x7f3e98140080 2024-03-18T18:51:06.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e9d004700 1 --2- 172.21.15.1:0/1947158388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e98140080 0x7f3e980c50b0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3e90007e40 tx=0x7f3e9000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:06.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e957fa700 1 -- 172.21.15.1:0/1947158388 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9000ed30 con 0x7f3e98140080 2024-03-18T18:51:06.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e957fa700 1 -- 172.21.15.1:0/1947158388 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e90014420 con 0x7f3e98140080 2024-03-18T18:51:06.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e9ea67700 1 -- 172.21.15.1:0/1947158388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e980ceb60 con 0x7f3e98140080 2024-03-18T18:51:06.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e957fa700 1 -- 172.21.15.1:0/1947158388 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e900147d0 con 0x7f3e98140080 2024-03-18T18:51:06.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.637+0000 7f3e9ea67700 1 -- 172.21.15.1:0/1947158388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e980cf120 con 0x7f3e98140080 2024-03-18T18:51:06.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.638+0000 7f3e9ea67700 1 -- 172.21.15.1:0/1947158388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e980b4a10 con 0x7f3e98140080 2024-03-18T18:51:06.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.639+0000 7f3e957fa700 1 -- 172.21.15.1:0/1947158388 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3e90014930 con 0x7f3e98140080 2024-03-18T18:51:06.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.639+0000 7f3e957fa700 1 --2- 172.21.15.1:0/1947158388 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3e7c06acc0 0x7f3e7c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:06.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.639+0000 7f3e957fa700 1 -- 172.21.15.1:0/1947158388 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f3e90089710 con 0x7f3e98140080 2024-03-18T18:51:06.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.639+0000 7f3e9c803700 1 --2- 172.21.15.1:0/1947158388 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3e7c06acc0 0x7f3e7c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:06.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.640+0000 7f3e9c803700 1 --2- 172.21.15.1:0/1947158388 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3e7c06acc0 0x7f3e7c06d180 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c014040 tx=0x7f3e8c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:06.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.644+0000 7f3e957fa700 1 -- 172.21.15.1:0/1947158388 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3e90054b90 con 0x7f3e98140080 2024-03-18T18:51:06.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.930+0000 7f3e9ea67700 1 -- 172.21.15.1:0/1947158388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3e98073350 con 0x7f3e98140080 2024-03-18T18:51:06.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.930+0000 7f3e957fa700 1 -- 172.21.15.1:0/1947158388 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v2) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f3e900581b0 con 0x7f3e98140080 2024-03-18T18:51:06.927 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:51:06.927 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T18:51:06.927 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-18T18:51:06.927 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.927 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-18T18:51:06.927 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlR93/CxAAbpe+5n3wHRur9xi8QzdCnQ==", 2024-03-18T18:51:06.927 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.927 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T18:51:06.928 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T18:51:06.928 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T18:51:06.928 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T18:51:06.928 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.928 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.928 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-18T18:51:06.928 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJ3OmGBAA37u227nxrMsh/JHt71HBxQ==", 2024-03-18T18:51:06.928 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.928 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T18:51:06.928 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T18:51:06.928 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T18:51:06.928 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T18:51:06.929 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.929 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-18T18:51:06.929 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJZFoExAAmI16oxwNKbf8dKetPfyk+g==", 2024-03-18T18:51:06.929 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.929 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T18:51:06.929 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T18:51:06.929 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T18:51:06.929 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T18:51:06.929 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.929 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.929 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.929 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-18T18:51:06.930 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlQrcvIBAA1LMv+EgC7S6wspX6QF49Pg==", 2024-03-18T18:51:06.930 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.930 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T18:51:06.930 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T18:51:06.930 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T18:51:06.930 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T18:51:06.930 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.930 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.930 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-18T18:51:06.930 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCUjPhlYOVHChAARFQrksZmMIdTlntd8TgvrQ==", 2024-03-18T18:51:06.930 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.930 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T18:51:06.930 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T18:51:06.931 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T18:51:06.931 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.931 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-18T18:51:06.931 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCXjPhl2zSDCBAA6Al1gkWVB79ZI9djGpVChA==", 2024-03-18T18:51:06.931 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.931 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T18:51:06.931 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T18:51:06.931 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T18:51:06.931 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.931 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-18T18:51:06.932 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCajPhlSoG/BhAAR93USDe6DCQW+hREOsnTlw==", 2024-03-18T18:51:06.932 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.932 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T18:51:06.932 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T18:51:06.932 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T18:51:06.932 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.932 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-18T18:51:06.932 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCdjPhlhJg5BRAAWuQhcnxyjR5UOiM13yKdAQ==", 2024-03-18T18:51:06.932 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.932 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T18:51:06.932 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T18:51:06.932 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T18:51:06.932 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.933 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-18T18:51:06.933 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCjjPhlCFnpCRAAKIyOQyVdjXZUTnAS6Nw/ng==", 2024-03-18T18:51:06.933 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.933 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T18:51:06.933 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T18:51:06.933 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T18:51:06.933 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.936 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-18T18:51:06.936 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCmjPhlAeFkCBAAJcEFpRk/03lQJxj+l2hEPA==", 2024-03-18T18:51:06.936 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.936 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T18:51:06.936 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T18:51:06.936 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T18:51:06.936 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.936 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-18T18:51:06.936 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCpjPhl/xuuBhAA0LgKfXEvN2O11ZlRQJVOkw==", 2024-03-18T18:51:06.937 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.937 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T18:51:06.937 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T18:51:06.937 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T18:51:06.937 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.937 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-18T18:51:06.937 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCsjPhldiR4BRAA571oh6U8PzOv/4LzjYLF/A==", 2024-03-18T18:51:06.937 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.937 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T18:51:06.937 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T18:51:06.938 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T18:51:06.938 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.938 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-18T18:51:06.938 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlu+2xJBAANnXGGjfeqP6S2FTecsfOsg==", 2024-03-18T18:51:06.938 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.939 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T18:51:06.939 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T18:51:06.939 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T18:51:06.939 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T18:51:06.939 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.940 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-18T18:51:06.940 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlhJ9+KxAAEnrQrCrgJR6xI+pucwzAXA==", 2024-03-18T18:51:06.940 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.940 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T18:51:06.940 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T18:51:06.940 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T18:51:06.940 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T18:51:06.940 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.941 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.2", 2024-03-18T18:51:06.941 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlxBUpKBAAIGTwmLWyj+P2Yvl55QVcWA==", 2024-03-18T18:51:06.941 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.941 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T18:51:06.942 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T18:51:06.942 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T18:51:06.942 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T18:51:06.942 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.942 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.943 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.3", 2024-03-18T18:51:06.943 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlmao3MRAAHeiQ6eyBh9QjjKPCtxkC8g==", 2024-03-18T18:51:06.943 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.943 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T18:51:06.943 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T18:51:06.943 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T18:51:06.944 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T18:51:06.944 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.944 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.944 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-18T18:51:06.944 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCRjPhlemOuKRAAVUggmtB7+EiVgyfJZ8TCCA==", 2024-03-18T18:51:06.945 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.945 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T18:51:06.945 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T18:51:06.945 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T18:51:06.945 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T18:51:06.946 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.946 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.946 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.bootstrap-mds", 2024-03-18T18:51:06.946 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQC3jPhlr9mCBxAAKbhaWRVec3bbqhKys3T9fA==", 2024-03-18T18:51:06.946 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.947 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile bootstrap-mds" 2024-03-18T18:51:06.947 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.947 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.947 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.bootstrap-mgr", 2024-03-18T18:51:06.948 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQC3jPhlweWCBxAARquxcqz8fi6eAprEgvMemw==", 2024-03-18T18:51:06.948 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.948 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-18T18:51:06.948 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.949 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.bootstrap-osd", 2024-03-18T18:51:06.949 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQC3jPhlkO6CBxAA08mcrqNdVNLcvUI9wvFMCg==", 2024-03-18T18:51:06.949 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.949 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile bootstrap-osd" 2024-03-18T18:51:06.949 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.950 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.bootstrap-rbd", 2024-03-18T18:51:06.950 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQC3jPhleveCBxAALbI7X3qqhmVIRHAnWaHSKw==", 2024-03-18T18:51:06.950 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-18T18:51:06.951 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.951 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-18T18:51:06.951 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQC3jPhl8v+CBxAA1NuK6et3wgO57yWz2LhQpg==", 2024-03-18T18:51:06.952 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.952 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-18T18:51:06.952 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.952 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.bootstrap-rgw", 2024-03-18T18:51:06.953 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQC3jPhl4AiDBxAABIahaz+wEQ7PlsH2yOQnSA==", 2024-03-18T18:51:06.953 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.953 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-18T18:51:06.953 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.954 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-18T18:51:06.954 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhleIuOCBAA9CH+AHewQbn+gpVr+6lJyw==", 2024-03-18T18:51:06.954 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.954 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T18:51:06.954 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T18:51:06.955 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T18:51:06.955 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:06.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:06.955 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-18T18:51:06.955 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlL0pTBBAAdFiPcdwwz514cuLAvioxYA==", 2024-03-18T18:51:06.956 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T18:51:06.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T18:51:06.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T18:51:06.956 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T18:51:06.956 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.956 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:06.957 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-18T18:51:06.957 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T18:51:06.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.933+0000 7f3e9ea67700 1 -- 172.21.15.1:0/1947158388 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3e7c06acc0 msgr2=0x7f3e7c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:06.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.933+0000 7f3e9ea67700 1 --2- 172.21.15.1:0/1947158388 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3e7c06acc0 0x7f3e7c06d180 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c014040 tx=0x7f3e8c008030 comp rx=0 tx=0).stop 2024-03-18T18:51:06.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.933+0000 7f3e9ea67700 1 -- 172.21.15.1:0/1947158388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e98140080 msgr2=0x7f3e980c50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:06.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.933+0000 7f3e9ea67700 1 --2- 172.21.15.1:0/1947158388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e98140080 0x7f3e980c50b0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3e90007e40 tx=0x7f3e9000e3c0 comp rx=0 tx=0).stop 2024-03-18T18:51:06.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.933+0000 7f3e9ea67700 1 -- 172.21.15.1:0/1947158388 shutdown_connections 2024-03-18T18:51:06.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.933+0000 7f3e9ea67700 1 --2- 172.21.15.1:0/1947158388 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3e7c06acc0 0x7f3e7c06d180 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:06.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.933+0000 7f3e9ea67700 1 --2- 172.21.15.1:0/1947158388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e98140080 0x7f3e980c50b0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:06.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.933+0000 7f3e9ea67700 1 --2- 172.21.15.1:0/1947158388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e981465a0 0x7f3e980ce460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:06.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.933+0000 7f3e9ea67700 1 --2- 172.21.15.1:0/1947158388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e98146af0 0x7f3e9813ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:06.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.933+0000 7f3e9ea67700 1 -- 172.21.15.1:0/1947158388 >> 172.21.15.1:0/1947158388 conn(0x7f3e980b1d20 msgr2=0x7f3e980acab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:06.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.933+0000 7f3e9ea67700 1 -- 172.21.15.1:0/1947158388 shutdown_connections 2024-03-18T18:51:06.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:06.933+0000 7f3e9ea67700 1 -- 172.21.15.1:0/1947158388 wait complete. 2024-03-18T18:51:06.962 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-18T18:51:07.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.093+0000 7f533e471700 1 Processor -- start 2024-03-18T18:51:07.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.093+0000 7f533e471700 1 -- start start 2024-03-18T18:51:07.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.094+0000 7f533e471700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5338071cc0 0x7f53380ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.094+0000 7f533e471700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53380cac80 con 0x7f533806d560 2024-03-18T18:51:07.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.094+0000 7f533e471700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53380caf60 con 0x7f533815a240 2024-03-18T18:51:07.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.094+0000 7f533e471700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53380adff0 con 0x7f5338071cc0 2024-03-18T18:51:07.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.094+0000 7f533ca0e700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f533806d560 0x7f53380ae410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57384/0 (socket says 172.21.15.1:57384) 2024-03-18T18:51:07.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.094+0000 7f533ca0e700 1 -- 172.21.15.1:0/2540511641 learned_addr learned my addr 172.21.15.1:0/2540511641 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:07.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.094+0000 7f53377fe700 1 --2- 172.21.15.1:0/2540511641 >> v2:172.21.15.165:3300/0 conn(0x7f5338071cc0 0x7f53380ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.094+0000 7f5336ffd700 1 -- 172.21.15.1:0/2540511641 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032953569 0 0) 0x7f53380cac80 con 0x7f533806d560 2024-03-18T18:51:07.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.094+0000 7f5336ffd700 1 -- 172.21.15.1:0/2540511641 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53240035b0 con 0x7f533806d560 2024-03-18T18:51:07.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.094+0000 7f5336ffd700 1 -- 172.21.15.1:0/2540511641 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2548639058 0 0) 0x7f53240035b0 con 0x7f533806d560 2024-03-18T18:51:07.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.094+0000 7f5336ffd700 1 -- 172.21.15.1:0/2540511641 >> v2:172.21.15.165:3300/0 conn(0x7f5338071cc0 msgr2=0x7f53380ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:07.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.094+0000 7f5336ffd700 1 --2- 172.21.15.1:0/2540511641 >> v2:172.21.15.165:3300/0 conn(0x7f5338071cc0 0x7f53380ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.094+0000 7f5336ffd700 1 -- 172.21.15.1:0/2540511641 >> v1:172.21.15.165:6789/0 conn(0x7f533815a240 legacy=0x7f53380715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:07.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.094+0000 7f5336ffd700 1 -- 172.21.15.1:0/2540511641 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53380cb240 con 0x7f533806d560 2024-03-18T18:51:07.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.094+0000 7f5336ffd700 1 -- 172.21.15.1:0/2540511641 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f53300019f0 con 0x7f533806d560 2024-03-18T18:51:07.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.094+0000 7f5336ffd700 1 -- 172.21.15.1:0/2540511641 >> v1:172.21.15.1:6789/0 conn(0x7f533806d560 legacy=0x7f53380ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:07.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f5336ffd700 1 --2- 172.21.15.1:0/2540511641 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5324003a00 0x7f5324003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f5336ffd700 1 --2- 172.21.15.1:0/2540511641 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5324004850 0x7f5324004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f5336ffd700 1 --2- 172.21.15.1:0/2540511641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5324005210 0x7f532400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f5336ffd700 1 -- 172.21.15.1:0/2540511641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53380adff0 con 0x7f5324005210 2024-03-18T18:51:07.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f5336ffd700 1 -- 172.21.15.1:0/2540511641 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532400c1b0 con 0x7f5324004850 2024-03-18T18:51:07.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f5336ffd700 1 -- 172.21.15.1:0/2540511641 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532400c370 con 0x7f5324003a00 2024-03-18T18:51:07.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f533ca0e700 1 --2- 172.21.15.1:0/2540511641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5324005210 0x7f532400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f53377fe700 1 --2- 172.21.15.1:0/2540511641 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5324004850 0x7f5324004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f5337fff700 1 --2- 172.21.15.1:0/2540511641 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5324003a00 0x7f5324003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f533ca0e700 1 -- 172.21.15.1:0/2540511641 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5324003a00 msgr2=0x7f5324003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:07.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f533ca0e700 1 --2- 172.21.15.1:0/2540511641 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5324003a00 0x7f5324003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f533ca0e700 1 -- 172.21.15.1:0/2540511641 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5324004850 msgr2=0x7f5324004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:07.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f533ca0e700 1 --2- 172.21.15.1:0/2540511641 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5324004850 0x7f5324004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f533ca0e700 1 -- 172.21.15.1:0/2540511641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53240035b0 con 0x7f5324005210 2024-03-18T18:51:07.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f5337fff700 1 --2- 172.21.15.1:0/2540511641 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5324003a00 0x7f5324003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:07.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f533ca0e700 1 -- 172.21.15.1:0/2540511641 reap_dead start 2024-03-18T18:51:07.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f533ca0e700 1 --2- 172.21.15.1:0/2540511641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5324005210 0x7f532400ba10 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f533000a830 tx=0x7f5324005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f073a970b12754cc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:07.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f5336ffd700 1 -- 172.21.15.1:0/2540511641 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533000a2f0 con 0x7f5324005210 2024-03-18T18:51:07.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f5336ffd700 1 -- 172.21.15.1:0/2540511641 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5330012600 con 0x7f5324005210 2024-03-18T18:51:07.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.095+0000 7f5336ffd700 1 -- 172.21.15.1:0/2540511641 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53300129b0 con 0x7f5324005210 2024-03-18T18:51:07.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.096+0000 7f533e471700 1 -- 172.21.15.1:0/2540511641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5324005210 msgr2=0x7f532400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:07.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.096+0000 7f533e471700 1 --2- 172.21.15.1:0/2540511641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5324005210 0x7f532400ba10 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f533000a830 tx=0x7f5324005880 comp rx=0 tx=0).stop 2024-03-18T18:51:07.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.096+0000 7f533e471700 1 -- 172.21.15.1:0/2540511641 shutdown_connections 2024-03-18T18:51:07.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.096+0000 7f533e471700 1 --2- 172.21.15.1:0/2540511641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5324005210 0x7f532400ba10 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.096+0000 7f533e471700 1 -- 172.21.15.1:0/2540511641 >> 172.21.15.1:0/2540511641 conn(0x7f53380cff30 msgr2=0x7f53380d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:07.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.096+0000 7f533e471700 1 -- 172.21.15.1:0/2540511641 shutdown_connections 2024-03-18T18:51:07.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.096+0000 7f533e471700 1 -- 172.21.15.1:0/2540511641 wait complete. 2024-03-18T18:51:07.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.097+0000 7f533e471700 1 Processor -- start 2024-03-18T18:51:07.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.097+0000 7f533e471700 1 -- start start 2024-03-18T18:51:07.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.098+0000 7f533e471700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5324005210 0x7f5338142c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.098+0000 7f533e471700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5338150f70 0x7f53380c9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.098+0000 7f533e471700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53380c7000 0x7f53380c7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.098+0000 7f533e471700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53380c4b80 con 0x7f5324005210 2024-03-18T18:51:07.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.098+0000 7f533e471700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53380c79c0 con 0x7f53380c7000 2024-03-18T18:51:07.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.098+0000 7f533e471700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53380c9860 con 0x7f5338150f70 2024-03-18T18:51:07.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.098+0000 7f5337fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5324005210 0x7f5338142c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.098+0000 7f5337fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5324005210 0x7f5338142c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53636/0 (socket says 172.21.15.1:53636) 2024-03-18T18:51:07.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.098+0000 7f5337fff700 1 -- 172.21.15.1:0/3278835900 learned_addr learned my addr 172.21.15.1:0/3278835900 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:07.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.098+0000 7f53377fe700 1 --2- 172.21.15.1:0/3278835900 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5338150f70 0x7f53380c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.098+0000 7f533ca0e700 1 --2- 172.21.15.1:0/3278835900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53380c7000 0x7f53380c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.098+0000 7f5337fff700 1 -- 172.21.15.1:0/3278835900 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5338150f70 msgr2=0x7f53380c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:07.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.098+0000 7f5337fff700 1 --2- 172.21.15.1:0/3278835900 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5338150f70 0x7f53380c9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.098+0000 7f5337fff700 1 -- 172.21.15.1:0/3278835900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53380c7000 msgr2=0x7f53380c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:07.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.098+0000 7f5337fff700 1 --2- 172.21.15.1:0/3278835900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53380c7000 0x7f53380c7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.098+0000 7f5337fff700 1 -- 172.21.15.1:0/3278835900 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53380c9b40 con 0x7f5324005210 2024-03-18T18:51:07.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.098+0000 7f533ca0e700 1 --2- 172.21.15.1:0/3278835900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53380c7000 0x7f53380c7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:07.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.099+0000 7f5337fff700 1 --2- 172.21.15.1:0/3278835900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5324005210 0x7f5338142c30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f532000e550 tx=0x7f532000e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:07.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.099+0000 7f5334ff9700 1 -- 172.21.15.1:0/3278835900 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53200052e0 con 0x7f5324005210 2024-03-18T18:51:07.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.099+0000 7f5334ff9700 1 -- 172.21.15.1:0/3278835900 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5320005440 con 0x7f5324005210 2024-03-18T18:51:07.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.099+0000 7f533e471700 1 -- 172.21.15.1:0/3278835900 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53380c9e20 con 0x7f5324005210 2024-03-18T18:51:07.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.099+0000 7f5334ff9700 1 -- 172.21.15.1:0/3278835900 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53200038a0 con 0x7f5324005210 2024-03-18T18:51:07.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.099+0000 7f533e471700 1 -- 172.21.15.1:0/3278835900 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53380ca100 con 0x7f5324005210 2024-03-18T18:51:07.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.100+0000 7f533e471700 1 -- 172.21.15.1:0/3278835900 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53380c4e20 con 0x7f5324005210 2024-03-18T18:51:07.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.100+0000 7f5334ff9700 1 -- 172.21.15.1:0/3278835900 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5320003a00 con 0x7f5324005210 2024-03-18T18:51:07.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.101+0000 7f5334ff9700 1 --2- 172.21.15.1:0/3278835900 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f531806ac90 0x7f531806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.101+0000 7f5334ff9700 1 -- 172.21.15.1:0/3278835900 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f53200886f0 con 0x7f5324005210 2024-03-18T18:51:07.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.101+0000 7f53377fe700 1 --2- 172.21.15.1:0/3278835900 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f531806ac90 0x7f531806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.101+0000 7f53377fe700 1 --2- 172.21.15.1:0/3278835900 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f531806ac90 0x7f531806d150 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f53380b3340 tx=0x7f5328013040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:07.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.105+0000 7f5334ff9700 1 -- 172.21.15.1:0/3278835900 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5320053c50 con 0x7f5324005210 2024-03-18T18:51:07.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.394+0000 7f533e471700 1 -- 172.21.15.1:0/3278835900 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f53380676a0 con 0x7f5324005210 2024-03-18T18:51:07.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.397+0000 7f5334ff9700 1 -- 172.21.15.1:0/3278835900 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v3) v1 ==== 90+0+0 (secure 0 0 0) 0x7f53200190a0 con 0x7f5324005210 2024-03-18T18:51:07.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.399+0000 7f533e471700 1 -- 172.21.15.1:0/3278835900 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f531806ac90 msgr2=0x7f531806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:07.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.399+0000 7f533e471700 1 --2- 172.21.15.1:0/3278835900 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f531806ac90 0x7f531806d150 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f53380b3340 tx=0x7f5328013040 comp rx=0 tx=0).stop 2024-03-18T18:51:07.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.399+0000 7f533e471700 1 -- 172.21.15.1:0/3278835900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5324005210 msgr2=0x7f5338142c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:07.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.399+0000 7f533e471700 1 --2- 172.21.15.1:0/3278835900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5324005210 0x7f5338142c30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f532000e550 tx=0x7f532000e910 comp rx=0 tx=0).stop 2024-03-18T18:51:07.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.399+0000 7f533e471700 1 -- 172.21.15.1:0/3278835900 shutdown_connections 2024-03-18T18:51:07.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.399+0000 7f533e471700 1 --2- 172.21.15.1:0/3278835900 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f531806ac90 0x7f531806d150 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.399+0000 7f533e471700 1 --2- 172.21.15.1:0/3278835900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53380c7000 0x7f53380c7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.399+0000 7f533e471700 1 --2- 172.21.15.1:0/3278835900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5324005210 0x7f5338142c30 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.399+0000 7f533e471700 1 --2- 172.21.15.1:0/3278835900 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5338150f70 0x7f53380c9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.399+0000 7f533e471700 1 -- 172.21.15.1:0/3278835900 >> 172.21.15.1:0/3278835900 conn(0x7f53380cff30 msgr2=0x7f53380af990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:07.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.399+0000 7f533e471700 1 -- 172.21.15.1:0/3278835900 shutdown_connections 2024-03-18T18:51:07.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.399+0000 7f533e471700 1 -- 172.21.15.1:0/3278835900 wait complete. 2024-03-18T18:51:07.408 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-18T18:51:07.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.538+0000 7f98ce29f700 1 Processor -- start 2024-03-18T18:51:07.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.538+0000 7f98ce29f700 1 -- start start 2024-03-18T18:51:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.539+0000 7f98ce29f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f98c8160a70 0x7f98c80d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.539+0000 7f98ce29f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c8160e50 0x7f98c80cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.539+0000 7f98ce29f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c8162170 con 0x7f98c81612d0 2024-03-18T18:51:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.539+0000 7f98ce29f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c80cef30 con 0x7f98c8160a70 2024-03-18T18:51:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.539+0000 7f98ce29f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c80cf210 con 0x7f98c8160e50 2024-03-18T18:51:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.539+0000 7f98c7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f98c8160a70 0x7f98c80d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.539+0000 7f98c7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f98c8160a70 0x7f98c80d3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53648/0 (socket says 172.21.15.1:53648) 2024-03-18T18:51:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.539+0000 7f98c7fff700 1 -- 172.21.15.1:0/797746057 learned_addr learned my addr 172.21.15.1:0/797746057 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.539+0000 7f98c77fe700 1 --2- 172.21.15.1:0/797746057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c8160e50 0x7f98c80cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.539+0000 7f98c7fff700 1 -- 172.21.15.1:0/797746057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c8160e50 msgr2=0x7f98c80cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.539+0000 7f98c7fff700 1 --2- 172.21.15.1:0/797746057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c8160e50 0x7f98c80cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.539+0000 7f98c7fff700 1 -- 172.21.15.1:0/797746057 >> v1:172.21.15.165:6789/0 conn(0x7f98c81612d0 legacy=0x7f98c80c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.539+0000 7f98c7fff700 1 -- 172.21.15.1:0/797746057 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c80cf4f0 con 0x7f98c8160a70 2024-03-18T18:51:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.540+0000 7f98c7fff700 1 --2- 172.21.15.1:0/797746057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c8160a70 0x7f98c80d3b70 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f98b80098f0 tx=0x7f98b8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d81adbbbb6382681 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.540+0000 7f98c6ffd700 1 -- 172.21.15.1:0/797746057 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8011830 con 0x7f98c8160a70 2024-03-18T18:51:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.540+0000 7f98c6ffd700 1 -- 172.21.15.1:0/797746057 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b8011990 con 0x7f98c8160a70 2024-03-18T18:51:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.540+0000 7f98c6ffd700 1 -- 172.21.15.1:0/797746057 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8011d20 con 0x7f98c8160a70 2024-03-18T18:51:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.540+0000 7f98ce29f700 1 -- 172.21.15.1:0/797746057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c8160a70 msgr2=0x7f98c80d3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.540+0000 7f98ce29f700 1 --2- 172.21.15.1:0/797746057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c8160a70 0x7f98c80d3b70 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f98b80098f0 tx=0x7f98b8009c00 comp rx=0 tx=0).stop 2024-03-18T18:51:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.540+0000 7f98ce29f700 1 -- 172.21.15.1:0/797746057 shutdown_connections 2024-03-18T18:51:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.540+0000 7f98ce29f700 1 --2- 172.21.15.1:0/797746057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c8160a70 0x7f98c80d3b70 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.540+0000 7f98ce29f700 1 --2- 172.21.15.1:0/797746057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c8160e50 0x7f98c80cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.540+0000 7f98ce29f700 1 -- 172.21.15.1:0/797746057 >> 172.21.15.1:0/797746057 conn(0x7f98c809c560 msgr2=0x7f98c80d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:07.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.540+0000 7f98ce29f700 1 -- 172.21.15.1:0/797746057 shutdown_connections 2024-03-18T18:51:07.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.540+0000 7f98ce29f700 1 -- 172.21.15.1:0/797746057 wait complete. 2024-03-18T18:51:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.541+0000 7f98ce29f700 1 Processor -- start 2024-03-18T18:51:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.542+0000 7f98ce29f700 1 -- start start 2024-03-18T18:51:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.542+0000 7f98ce29f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c8160a70 0x7f98c80cb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.542+0000 7f98ce29f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c8160e50 0x7f98c81f6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.542+0000 7f98ce29f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98c81612d0 0x7f98c80b87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.542+0000 7f98ce29f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c81f73d0 con 0x7f98c8160a70 2024-03-18T18:51:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.542+0000 7f98ce29f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c80ca790 con 0x7f98c81612d0 2024-03-18T18:51:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.542+0000 7f98ce29f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c80caa70 con 0x7f98c8160e50 2024-03-18T18:51:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.543+0000 7f98c7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c8160a70 0x7f98c80cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.543+0000 7f98c7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c8160a70 0x7f98c80cb810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53658/0 (socket says 172.21.15.1:53658) 2024-03-18T18:51:07.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.543+0000 7f98c7fff700 1 -- 172.21.15.1:0/252638134 learned_addr learned my addr 172.21.15.1:0/252638134 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:07.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.543+0000 7f98c77fe700 1 --2- 172.21.15.1:0/252638134 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c8160e50 0x7f98c81f6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.543+0000 7f98cc83c700 1 --2- 172.21.15.1:0/252638134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98c81612d0 0x7f98c80b87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.543+0000 7f98c7fff700 1 -- 172.21.15.1:0/252638134 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c8160e50 msgr2=0x7f98c81f6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:07.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.543+0000 7f98c7fff700 1 --2- 172.21.15.1:0/252638134 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c8160e50 0x7f98c81f6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.543+0000 7f98c7fff700 1 -- 172.21.15.1:0/252638134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98c81612d0 msgr2=0x7f98c80b87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:07.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.543+0000 7f98c7fff700 1 --2- 172.21.15.1:0/252638134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98c81612d0 0x7f98c80b87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.543+0000 7f98c7fff700 1 -- 172.21.15.1:0/252638134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b8009550 con 0x7f98c8160a70 2024-03-18T18:51:07.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.543+0000 7f98c77fe700 1 --2- 172.21.15.1:0/252638134 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c8160e50 0x7f98c81f6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:07.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.543+0000 7f98c7fff700 1 --2- 172.21.15.1:0/252638134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c8160a70 0x7f98c80cb810 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009c40 tx=0x7f98b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:07.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.543+0000 7f98c4ff9700 1 -- 172.21.15.1:0/252638134 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b80036a0 con 0x7f98c8160a70 2024-03-18T18:51:07.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.543+0000 7f98c4ff9700 1 -- 172.21.15.1:0/252638134 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b801e070 con 0x7f98c8160a70 2024-03-18T18:51:07.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.543+0000 7f98c4ff9700 1 -- 172.21.15.1:0/252638134 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8003a60 con 0x7f98c8160a70 2024-03-18T18:51:07.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.543+0000 7f98ce29f700 1 -- 172.21.15.1:0/252638134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c80cad50 con 0x7f98c8160a70 2024-03-18T18:51:07.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.543+0000 7f98ce29f700 1 -- 172.21.15.1:0/252638134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c80cb310 con 0x7f98c8160a70 2024-03-18T18:51:07.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.544+0000 7f98ce29f700 1 -- 172.21.15.1:0/252638134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c80cb680 con 0x7f98c8160a70 2024-03-18T18:51:07.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.545+0000 7f98c4ff9700 1 -- 172.21.15.1:0/252638134 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f98b8003bc0 con 0x7f98c8160a70 2024-03-18T18:51:07.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.545+0000 7f98c4ff9700 1 --2- 172.21.15.1:0/252638134 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f98a806ac90 0x7f98a806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.545+0000 7f98c4ff9700 1 -- 172.21.15.1:0/252638134 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f98b808a030 con 0x7f98c8160a70 2024-03-18T18:51:07.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.545+0000 7f98c77fe700 1 --2- 172.21.15.1:0/252638134 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f98a806ac90 0x7f98a806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.546+0000 7f98c77fe700 1 --2- 172.21.15.1:0/252638134 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f98a806ac90 0x7f98a806d150 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f98c8161230 tx=0x7f98b0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.550+0000 7f98c4ff9700 1 -- 172.21.15.1:0/252638134 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f98b80554b0 con 0x7f98c8160a70 2024-03-18T18:51:07.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.836+0000 7f98ce29f700 1 -- 172.21.15.1:0/252638134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f98c8067390 con 0x7f98c8160a70 2024-03-18T18:51:07.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.839+0000 7f98c4ff9700 1 -- 172.21.15.1:0/252638134 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v4) v1 ==== 90+0+0 (secure 0 0 0) 0x7f98b8058ad0 con 0x7f98c8160a70 2024-03-18T18:51:07.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.842+0000 7f98ce29f700 1 -- 172.21.15.1:0/252638134 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f98a806ac90 msgr2=0x7f98a806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:07.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.842+0000 7f98ce29f700 1 --2- 172.21.15.1:0/252638134 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f98a806ac90 0x7f98a806d150 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f98c8161230 tx=0x7f98b0008040 comp rx=0 tx=0).stop 2024-03-18T18:51:07.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.842+0000 7f98ce29f700 1 -- 172.21.15.1:0/252638134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c8160a70 msgr2=0x7f98c80cb810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:07.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.842+0000 7f98ce29f700 1 --2- 172.21.15.1:0/252638134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c8160a70 0x7f98c80cb810 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009c40 tx=0x7f98b8004750 comp rx=0 tx=0).stop 2024-03-18T18:51:07.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.842+0000 7f98ce29f700 1 -- 172.21.15.1:0/252638134 shutdown_connections 2024-03-18T18:51:07.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.842+0000 7f98ce29f700 1 --2- 172.21.15.1:0/252638134 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f98a806ac90 0x7f98a806d150 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.842+0000 7f98ce29f700 1 --2- 172.21.15.1:0/252638134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98c81612d0 0x7f98c80b87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.842+0000 7f98ce29f700 1 --2- 172.21.15.1:0/252638134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c8160a70 0x7f98c80cb810 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.842+0000 7f98ce29f700 1 --2- 172.21.15.1:0/252638134 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98c8160e50 0x7f98c81f6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.842+0000 7f98ce29f700 1 -- 172.21.15.1:0/252638134 >> 172.21.15.1:0/252638134 conn(0x7f98c809c560 msgr2=0x7f98c80c6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:07.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.842+0000 7f98ce29f700 1 -- 172.21.15.1:0/252638134 shutdown_connections 2024-03-18T18:51:07.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.842+0000 7f98ce29f700 1 -- 172.21.15.1:0/252638134 wait complete. 2024-03-18T18:51:07.850 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-18T18:51:07.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.980+0000 7f7d7eeb1700 1 Processor -- start 2024-03-18T18:51:07.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.981+0000 7f7d7eeb1700 1 -- start start 2024-03-18T18:51:07.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.981+0000 7f7d7eeb1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7d780b1e20 0x7f7d780cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.981+0000 7f7d7eeb1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d780cbce0 0x7f7d780cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.981+0000 7f7d7eeb1700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d780d0df0 con 0x7f7d780cd870 2024-03-18T18:51:07.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.981+0000 7f7d7eeb1700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d780d0550 con 0x7f7d780b1e20 2024-03-18T18:51:07.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.981+0000 7f7d7eeb1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d780d0830 con 0x7f7d780cbce0 2024-03-18T18:51:07.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.981+0000 7f7d7cc4d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7d780cd870 0x7f7d780cdc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57400/0 (socket says 172.21.15.1:57400) 2024-03-18T18:51:07.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.981+0000 7f7d7cc4d700 1 -- 172.21.15.1:0/3992211704 learned_addr learned my addr 172.21.15.1:0/3992211704 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:07.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.982+0000 7f7d7d44e700 1 --2- 172.21.15.1:0/3992211704 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d780cbce0 0x7f7d780cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.982+0000 7f7d77fff700 1 --2- 172.21.15.1:0/3992211704 >> v2:172.21.15.165:3300/0 conn(0x7f7d780b1e20 0x7f7d780cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.982+0000 7f7d777fe700 1 -- 172.21.15.1:0/3992211704 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086631668 0 0) 0x7f7d780d0df0 con 0x7f7d780cd870 2024-03-18T18:51:07.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.982+0000 7f7d777fe700 1 -- 172.21.15.1:0/3992211704 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d680035b0 con 0x7f7d780cd870 2024-03-18T18:51:07.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.982+0000 7f7d777fe700 1 -- 172.21.15.1:0/3992211704 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 177048954 0 0) 0x7f7d680035b0 con 0x7f7d780cd870 2024-03-18T18:51:07.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.982+0000 7f7d7d44e700 1 -- 172.21.15.1:0/3992211704 >> v2:172.21.15.165:3300/0 conn(0x7f7d780b1e20 msgr2=0x7f7d780cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:07.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.982+0000 7f7d7d44e700 1 --2- 172.21.15.1:0/3992211704 >> v2:172.21.15.165:3300/0 conn(0x7f7d780b1e20 0x7f7d780cff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.982+0000 7f7d7d44e700 1 -- 172.21.15.1:0/3992211704 >> v1:172.21.15.1:6789/0 conn(0x7f7d780cd870 legacy=0x7f7d780cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:07.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.982+0000 7f7d7d44e700 1 -- 172.21.15.1:0/3992211704 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d780d0b10 con 0x7f7d780cbce0 2024-03-18T18:51:07.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.982+0000 7f7d77fff700 1 --2- 172.21.15.1:0/3992211704 >> v2:172.21.15.165:3300/0 conn(0x7f7d780b1e20 0x7f7d780cff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:51:07.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.982+0000 7f7d7d44e700 1 --2- 172.21.15.1:0/3992211704 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d780cbce0 0x7f7d780cc170 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009d90 tx=0x7f7d7000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5cd7bfc93dac3c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:07.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.982+0000 7f7d777fe700 1 -- 172.21.15.1:0/3992211704 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7000e070 con 0x7f7d780cbce0 2024-03-18T18:51:07.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.982+0000 7f7d777fe700 1 -- 172.21.15.1:0/3992211704 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d70003a20 con 0x7f7d780cbce0 2024-03-18T18:51:07.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.982+0000 7f7d777fe700 1 -- 172.21.15.1:0/3992211704 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c001670 con 0x7f7d780cbce0 2024-03-18T18:51:07.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.983+0000 7f7d7eeb1700 1 -- 172.21.15.1:0/3992211704 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d780cbce0 msgr2=0x7f7d780cc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:07.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.983+0000 7f7d7eeb1700 1 --2- 172.21.15.1:0/3992211704 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d780cbce0 0x7f7d780cc170 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009d90 tx=0x7f7d7000b330 comp rx=0 tx=0).stop 2024-03-18T18:51:07.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.983+0000 7f7d7eeb1700 1 -- 172.21.15.1:0/3992211704 shutdown_connections 2024-03-18T18:51:07.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.983+0000 7f7d7eeb1700 1 --2- 172.21.15.1:0/3992211704 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d780cbce0 0x7f7d780cc170 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.983+0000 7f7d7eeb1700 1 --2- 172.21.15.1:0/3992211704 >> v2:172.21.15.165:3300/0 conn(0x7f7d780b1e20 0x7f7d780cff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.983+0000 7f7d7eeb1700 1 -- 172.21.15.1:0/3992211704 >> 172.21.15.1:0/3992211704 conn(0x7f7d780d0ff0 msgr2=0x7f7d780d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:07.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.983+0000 7f7d7eeb1700 1 -- 172.21.15.1:0/3992211704 shutdown_connections 2024-03-18T18:51:07.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.983+0000 7f7d7eeb1700 1 -- 172.21.15.1:0/3992211704 wait complete. 2024-03-18T18:51:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.984+0000 7f7d7eeb1700 1 Processor -- start 2024-03-18T18:51:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.984+0000 7f7d7eeb1700 1 -- start start 2024-03-18T18:51:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.985+0000 7f7d7eeb1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d780b1e20 0x7f7d781e4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.985+0000 7f7d7eeb1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d780cbce0 0x7f7d780b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.985+0000 7f7d7eeb1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d780cd870 0x7f7d780caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.985+0000 7f7d7eeb1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d780b1a10 con 0x7f7d780cd870 2024-03-18T18:51:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.985+0000 7f7d7eeb1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d781e6200 con 0x7f7d780b1e20 2024-03-18T18:51:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.985+0000 7f7d7eeb1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d780cafa0 con 0x7f7d780cbce0 2024-03-18T18:51:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.985+0000 7f7d7d44e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d780cd870 0x7f7d780caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.985+0000 7f7d7d44e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d780cd870 0x7f7d780caa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53672/0 (socket says 172.21.15.1:53672) 2024-03-18T18:51:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.985+0000 7f7d7d44e700 1 -- 172.21.15.1:0/3960322834 learned_addr learned my addr 172.21.15.1:0/3960322834 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.985+0000 7f7d7cc4d700 1 --2- 172.21.15.1:0/3960322834 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d780b1e20 0x7f7d781e4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.985+0000 7f7d77fff700 1 --2- 172.21.15.1:0/3960322834 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d780cbce0 0x7f7d780b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.986+0000 7f7d7d44e700 1 -- 172.21.15.1:0/3960322834 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d780cbce0 msgr2=0x7f7d780b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.986+0000 7f7d7d44e700 1 --2- 172.21.15.1:0/3960322834 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d780cbce0 0x7f7d780b14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.986+0000 7f7d7d44e700 1 -- 172.21.15.1:0/3960322834 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d780b1e20 msgr2=0x7f7d781e4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.986+0000 7f7d7d44e700 1 --2- 172.21.15.1:0/3960322834 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d780b1e20 0x7f7d781e4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.986+0000 7f7d7d44e700 1 -- 172.21.15.1:0/3960322834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d640094a0 con 0x7f7d780cd870 2024-03-18T18:51:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.986+0000 7f7d7d44e700 1 --2- 172.21.15.1:0/3960322834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d780cd870 0x7f7d780caa60 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f7d780d0970 tx=0x7f7d70004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.986+0000 7f7d757fa700 1 -- 172.21.15.1:0/3960322834 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d70009410 con 0x7f7d780cd870 2024-03-18T18:51:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.986+0000 7f7d757fa700 1 -- 172.21.15.1:0/3960322834 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d70004460 con 0x7f7d780cd870 2024-03-18T18:51:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.986+0000 7f7d757fa700 1 -- 172.21.15.1:0/3960322834 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d70004810 con 0x7f7d780cd870 2024-03-18T18:51:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.986+0000 7f7d7eeb1700 1 -- 172.21.15.1:0/3960322834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d70009a70 con 0x7f7d780cd870 2024-03-18T18:51:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.986+0000 7f7d7eeb1700 1 -- 172.21.15.1:0/3960322834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d781e58d0 con 0x7f7d780cd870 2024-03-18T18:51:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.987+0000 7f7d7eeb1700 1 -- 172.21.15.1:0/3960322834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d781e63b0 con 0x7f7d780cd870 2024-03-18T18:51:07.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.987+0000 7f7d757fa700 1 -- 172.21.15.1:0/3960322834 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7d70012430 con 0x7f7d780cd870 2024-03-18T18:51:07.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.988+0000 7f7d757fa700 1 --2- 172.21.15.1:0/3960322834 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7d5c06ac90 0x7f7d5c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:07.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.988+0000 7f7d757fa700 1 -- 172.21.15.1:0/3960322834 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f7d7008af30 con 0x7f7d780cd870 2024-03-18T18:51:07.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.988+0000 7f7d7cc4d700 1 --2- 172.21.15.1:0/3960322834 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7d5c06ac90 0x7f7d5c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:07.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.988+0000 7f7d7cc4d700 1 --2- 172.21.15.1:0/3960322834 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7d5c06ac90 0x7f7d5c06d150 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c000f60 tx=0x7f7d6c001e00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:07.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:07.992+0000 7f7d757fa700 1 -- 172.21.15.1:0/3960322834 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7d781e63b0 con 0x7f7d780cd870 2024-03-18T18:51:08.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.283+0000 7f7d7eeb1700 1 -- 172.21.15.1:0/3960322834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f7d781e6370 con 0x7f7d780cd870 2024-03-18T18:51:08.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.287+0000 7f7d757fa700 1 -- 172.21.15.1:0/3960322834 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f7d70059e40 con 0x7f7d780cd870 2024-03-18T18:51:08.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.290+0000 7f7d7eeb1700 1 -- 172.21.15.1:0/3960322834 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7d5c06ac90 msgr2=0x7f7d5c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:08.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.290+0000 7f7d7eeb1700 1 --2- 172.21.15.1:0/3960322834 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7d5c06ac90 0x7f7d5c06d150 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c000f60 tx=0x7f7d6c001e00 comp rx=0 tx=0).stop 2024-03-18T18:51:08.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.290+0000 7f7d7eeb1700 1 -- 172.21.15.1:0/3960322834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d780cd870 msgr2=0x7f7d780caa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:08.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.290+0000 7f7d7eeb1700 1 --2- 172.21.15.1:0/3960322834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d780cd870 0x7f7d780caa60 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f7d780d0970 tx=0x7f7d70004ad0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.290+0000 7f7d7eeb1700 1 -- 172.21.15.1:0/3960322834 shutdown_connections 2024-03-18T18:51:08.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.290+0000 7f7d7eeb1700 1 --2- 172.21.15.1:0/3960322834 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7d5c06ac90 0x7f7d5c06d150 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.290+0000 7f7d7eeb1700 1 --2- 172.21.15.1:0/3960322834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d780cd870 0x7f7d780caa60 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.290+0000 7f7d7eeb1700 1 --2- 172.21.15.1:0/3960322834 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d780b1e20 0x7f7d781e4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.290+0000 7f7d7eeb1700 1 --2- 172.21.15.1:0/3960322834 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d780cbce0 0x7f7d780b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.290+0000 7f7d7eeb1700 1 -- 172.21.15.1:0/3960322834 >> 172.21.15.1:0/3960322834 conn(0x7f7d780d0ff0 msgr2=0x7f7d7815ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:08.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.290+0000 7f7d7eeb1700 1 -- 172.21.15.1:0/3960322834 shutdown_connections 2024-03-18T18:51:08.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.290+0000 7f7d7eeb1700 1 -- 172.21.15.1:0/3960322834 wait complete. 2024-03-18T18:51:08.298 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-18T18:51:08.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.428+0000 7f5e3004d700 1 Processor -- start 2024-03-18T18:51:08.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.428+0000 7f5e3004d700 1 -- start start 2024-03-18T18:51:08.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.429+0000 7f5e3004d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5e28071d30 0x7f5e280add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:08.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.429+0000 7f5e3004d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e280cb9e0 0x7f5e280cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:08.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.429+0000 7f5e3004d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e280ce8f0 con 0x7f5e2815a1d0 2024-03-18T18:51:08.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.429+0000 7f5e3004d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e280cdb00 con 0x7f5e28071d30 2024-03-18T18:51:08.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.429+0000 7f5e3004d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2815a5b0 con 0x7f5e280cb9e0 2024-03-18T18:51:08.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.429+0000 7f5e2dde9700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5e2815a1d0 0x7f5e28071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57408/0 (socket says 172.21.15.1:57408) 2024-03-18T18:51:08.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.429+0000 7f5e2dde9700 1 -- 172.21.15.1:0/166145850 learned_addr learned my addr 172.21.15.1:0/166145850 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:08.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2cde7700 1 -- 172.21.15.1:0/166145850 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563636011 0 0) 0x7f5e280ce8f0 con 0x7f5e2815a1d0 2024-03-18T18:51:08.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2d5e8700 1 --2- 172.21.15.1:0/166145850 >> v2:172.21.15.165:3300/0 conn(0x7f5e28071d30 0x7f5e280add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:08.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2cde7700 1 -- 172.21.15.1:0/166145850 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e140035b0 con 0x7f5e2815a1d0 2024-03-18T18:51:08.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2e5ea700 1 --2- 172.21.15.1:0/166145850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e280cb9e0 0x7f5e280cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:08.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2cde7700 1 -- 172.21.15.1:0/166145850 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1760376354 0 0) 0x7f5e140035b0 con 0x7f5e2815a1d0 2024-03-18T18:51:08.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2cde7700 1 -- 172.21.15.1:0/166145850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e280cb9e0 msgr2=0x7f5e280cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:08.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2cde7700 1 --2- 172.21.15.1:0/166145850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e280cb9e0 0x7f5e280cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2cde7700 1 -- 172.21.15.1:0/166145850 >> v2:172.21.15.165:3300/0 conn(0x7f5e28071d30 msgr2=0x7f5e280add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:08.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2cde7700 1 --2- 172.21.15.1:0/166145850 >> v2:172.21.15.165:3300/0 conn(0x7f5e28071d30 0x7f5e280add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2cde7700 1 -- 172.21.15.1:0/166145850 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e280ae260 con 0x7f5e2815a1d0 2024-03-18T18:51:08.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2cde7700 1 -- 172.21.15.1:0/166145850 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f5e240017f0 con 0x7f5e2815a1d0 2024-03-18T18:51:08.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2cde7700 1 -- 172.21.15.1:0/166145850 >> v1:172.21.15.1:6789/0 conn(0x7f5e2815a1d0 legacy=0x7f5e28071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:08.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2cde7700 1 --2- 172.21.15.1:0/166145850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e14003a00 0x7f5e14003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2cde7700 1 --2- 172.21.15.1:0/166145850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e280cb9e0 0x7f5e14004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2cde7700 1 --2- 172.21.15.1:0/166145850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e14004d90 0x7f5e14005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2cde7700 1 -- 172.21.15.1:0/166145850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e280cdb00 con 0x7f5e14003a00 2024-03-18T18:51:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2cde7700 1 -- 172.21.15.1:0/166145850 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1400ba10 con 0x7f5e14004d90 2024-03-18T18:51:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2cde7700 1 -- 172.21.15.1:0/166145850 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1400bb50 con 0x7f5e280cb9e0 2024-03-18T18:51:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2dde9700 1 --2- 172.21.15.1:0/166145850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e14003a00 0x7f5e14003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2d5e8700 1 --2- 172.21.15.1:0/166145850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e280cb9e0 0x7f5e14004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2e5ea700 1 --2- 172.21.15.1:0/166145850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e14004d90 0x7f5e14005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2dde9700 1 -- 172.21.15.1:0/166145850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e280cb9e0 msgr2=0x7f5e14004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2dde9700 1 --2- 172.21.15.1:0/166145850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e280cb9e0 0x7f5e14004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2dde9700 1 -- 172.21.15.1:0/166145850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e14004d90 msgr2=0x7f5e14005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2dde9700 1 --2- 172.21.15.1:0/166145850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e14004d90 0x7f5e14005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.430+0000 7f5e2dde9700 1 -- 172.21.15.1:0/166145850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e140035b0 con 0x7f5e14003a00 2024-03-18T18:51:08.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.431+0000 7f5e2dde9700 1 --2- 172.21.15.1:0/166145850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e14003a00 0x7f5e14003e20 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400c240 tx=0x7f5e14005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ed1896fe1e6ce1c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:08.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.431+0000 7f5e2cde7700 1 -- 172.21.15.1:0/166145850 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2400cde0 con 0x7f5e14003a00 2024-03-18T18:51:08.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.431+0000 7f5e2cde7700 1 -- 172.21.15.1:0/166145850 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e24007bd0 con 0x7f5e14003a00 2024-03-18T18:51:08.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.431+0000 7f5e2cde7700 1 -- 172.21.15.1:0/166145850 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e24007310 con 0x7f5e14003a00 2024-03-18T18:51:08.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.431+0000 7f5e3004d700 1 -- 172.21.15.1:0/166145850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e14003a00 msgr2=0x7f5e14003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:08.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.431+0000 7f5e3004d700 1 --2- 172.21.15.1:0/166145850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e14003a00 0x7f5e14003e20 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400c240 tx=0x7f5e14005880 comp rx=0 tx=0).stop 2024-03-18T18:51:08.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.431+0000 7f5e2e5ea700 1 -- 172.21.15.1:0/166145850 reap_dead start 2024-03-18T18:51:08.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.431+0000 7f5e3004d700 1 -- 172.21.15.1:0/166145850 shutdown_connections 2024-03-18T18:51:08.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.431+0000 7f5e3004d700 1 -- 172.21.15.1:0/166145850 >> 172.21.15.1:0/166145850 conn(0x7f5e280cc9b0 msgr2=0x7f5e280ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:08.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.431+0000 7f5e3004d700 1 -- 172.21.15.1:0/166145850 shutdown_connections 2024-03-18T18:51:08.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.432+0000 7f5e3004d700 1 -- 172.21.15.1:0/166145850 wait complete. 2024-03-18T18:51:08.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.433+0000 7f5e3004d700 1 Processor -- start 2024-03-18T18:51:08.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.433+0000 7f5e3004d700 1 -- start start 2024-03-18T18:51:08.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.433+0000 7f5e3004d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e14003a00 0x7f5e281571f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:08.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.433+0000 7f5e3004d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e281578f0 0x7f5e280c8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:08.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.433+0000 7f5e3004d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e280c8960 0x7f5e2815c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:08.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.433+0000 7f5e3004d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e28157730 con 0x7f5e14003a00 2024-03-18T18:51:08.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.433+0000 7f5e3004d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e280c56d0 con 0x7f5e280c8960 2024-03-18T18:51:08.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.433+0000 7f5e3004d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2815cdb0 con 0x7f5e281578f0 2024-03-18T18:51:08.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.434+0000 7f5e2dde9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e14003a00 0x7f5e281571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:08.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.434+0000 7f5e2dde9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e14003a00 0x7f5e281571f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53694/0 (socket says 172.21.15.1:53694) 2024-03-18T18:51:08.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.434+0000 7f5e2dde9700 1 -- 172.21.15.1:0/1258473886 learned_addr learned my addr 172.21.15.1:0/1258473886 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:08.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.434+0000 7f5e2e5ea700 1 --2- 172.21.15.1:0/1258473886 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e280c8960 0x7f5e2815c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:08.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.434+0000 7f5e2d5e8700 1 --2- 172.21.15.1:0/1258473886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e281578f0 0x7f5e280c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:08.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.434+0000 7f5e2dde9700 1 -- 172.21.15.1:0/1258473886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e281578f0 msgr2=0x7f5e280c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:08.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.434+0000 7f5e2dde9700 1 --2- 172.21.15.1:0/1258473886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e281578f0 0x7f5e280c8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.434+0000 7f5e2dde9700 1 -- 172.21.15.1:0/1258473886 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e280c8960 msgr2=0x7f5e2815c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:08.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.434+0000 7f5e2dde9700 1 --2- 172.21.15.1:0/1258473886 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e280c8960 0x7f5e2815c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.434+0000 7f5e2dde9700 1 -- 172.21.15.1:0/1258473886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2815d090 con 0x7f5e14003a00 2024-03-18T18:51:08.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.434+0000 7f5e2dde9700 1 --2- 172.21.15.1:0/1258473886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e14003a00 0x7f5e281571f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5e140046e0 tx=0x7f5e24001160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:08.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.434+0000 7f5e1e7fc700 1 -- 172.21.15.1:0/1258473886 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e24005d40 con 0x7f5e14003a00 2024-03-18T18:51:08.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.434+0000 7f5e1e7fc700 1 -- 172.21.15.1:0/1258473886 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e240070d0 con 0x7f5e14003a00 2024-03-18T18:51:08.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.434+0000 7f5e3004d700 1 -- 172.21.15.1:0/1258473886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e2815d370 con 0x7f5e14003a00 2024-03-18T18:51:08.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.434+0000 7f5e1e7fc700 1 -- 172.21.15.1:0/1258473886 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e24007480 con 0x7f5e14003a00 2024-03-18T18:51:08.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.434+0000 7f5e3004d700 1 -- 172.21.15.1:0/1258473886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e2815c720 con 0x7f5e14003a00 2024-03-18T18:51:08.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.435+0000 7f5e3004d700 1 -- 172.21.15.1:0/1258473886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e280cc660 con 0x7f5e14003a00 2024-03-18T18:51:08.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.436+0000 7f5e1e7fc700 1 -- 172.21.15.1:0/1258473886 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5e24019030 con 0x7f5e14003a00 2024-03-18T18:51:08.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.436+0000 7f5e1e7fc700 1 --2- 172.21.15.1:0/1258473886 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e0c06ac90 0x7f5e0c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:08.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.436+0000 7f5e1e7fc700 1 -- 172.21.15.1:0/1258473886 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f5e2408a2a0 con 0x7f5e14003a00 2024-03-18T18:51:08.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.436+0000 7f5e2d5e8700 1 --2- 172.21.15.1:0/1258473886 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e0c06ac90 0x7f5e0c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:08.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.437+0000 7f5e2d5e8700 1 --2- 172.21.15.1:0/1258473886 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e0c06ac90 0x7f5e0c06d150 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5e280cd2d0 tx=0x7f5e18009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:08.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.441+0000 7f5e1e7fc700 1 -- 172.21.15.1:0/1258473886 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5e24055720 con 0x7f5e14003a00 2024-03-18T18:51:08.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.731+0000 7f5e3004d700 1 -- 172.21.15.1:0/1258473886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f5e280c59b0 con 0x7f5e14003a00 2024-03-18T18:51:08.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.735+0000 7f5e1e7fc700 1 -- 172.21.15.1:0/1258473886 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f5e24058d40 con 0x7f5e14003a00 2024-03-18T18:51:08.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.737+0000 7f5e3004d700 1 -- 172.21.15.1:0/1258473886 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e0c06ac90 msgr2=0x7f5e0c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:08.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.737+0000 7f5e3004d700 1 --2- 172.21.15.1:0/1258473886 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e0c06ac90 0x7f5e0c06d150 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5e280cd2d0 tx=0x7f5e18009210 comp rx=0 tx=0).stop 2024-03-18T18:51:08.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.737+0000 7f5e3004d700 1 -- 172.21.15.1:0/1258473886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e14003a00 msgr2=0x7f5e281571f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:08.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.737+0000 7f5e3004d700 1 --2- 172.21.15.1:0/1258473886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e14003a00 0x7f5e281571f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5e140046e0 tx=0x7f5e24001160 comp rx=0 tx=0).stop 2024-03-18T18:51:08.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.737+0000 7f5e3004d700 1 -- 172.21.15.1:0/1258473886 shutdown_connections 2024-03-18T18:51:08.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.737+0000 7f5e3004d700 1 --2- 172.21.15.1:0/1258473886 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e0c06ac90 0x7f5e0c06d150 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.737+0000 7f5e3004d700 1 --2- 172.21.15.1:0/1258473886 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e280c8960 0x7f5e2815c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.737+0000 7f5e3004d700 1 --2- 172.21.15.1:0/1258473886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e14003a00 0x7f5e281571f0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.737+0000 7f5e3004d700 1 --2- 172.21.15.1:0/1258473886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e281578f0 0x7f5e280c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.737+0000 7f5e3004d700 1 -- 172.21.15.1:0/1258473886 >> 172.21.15.1:0/1258473886 conn(0x7f5e280cc9b0 msgr2=0x7f5e28159480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:08.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.737+0000 7f5e3004d700 1 -- 172.21.15.1:0/1258473886 shutdown_connections 2024-03-18T18:51:08.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.737+0000 7f5e3004d700 1 -- 172.21.15.1:0/1258473886 wait complete. 2024-03-18T18:51:08.745 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-18T18:51:08.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.875+0000 7f703b3b9700 1 Processor -- start 2024-03-18T18:51:08.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.875+0000 7f703b3b9700 1 -- start start 2024-03-18T18:51:08.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.876+0000 7f703b3b9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f703415d5b0 0x7f703415d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:08.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.876+0000 7f703b3b9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7034099a40 0x7f7034099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:08.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.876+0000 7f703b3b9700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703409a830 con 0x7f703409a3e0 2024-03-18T18:51:08.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.876+0000 7f703b3b9700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70340cbbb0 con 0x7f7034099a40 2024-03-18T18:51:08.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.876+0000 7f703b3b9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70340b3b00 con 0x7f703415d5b0 2024-03-18T18:51:08.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.876+0000 7f7038954700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7034099a40 0x7f7034099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:08.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.876+0000 7f7039155700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f703415d5b0 0x7f703415d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:08.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.876+0000 7f7039155700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f703415d5b0 0x7f703415d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:43712/0 (socket says 172.21.15.1:43712) 2024-03-18T18:51:08.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.876+0000 7f7038954700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7034099a40 0x7f7034099ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:44382/0 (socket says 172.21.15.1:44382) 2024-03-18T18:51:08.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.876+0000 7f7039155700 1 -- 172.21.15.1:0/1698031460 learned_addr learned my addr 172.21.15.1:0/1698031460 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:08.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.876+0000 7f7039956700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f703409a3e0 0x7f703406ba70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:51348/0 (socket says 172.21.15.1:51348) 2024-03-18T18:51:08.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.876+0000 7f7039155700 1 -- 172.21.15.1:0/1698031460 >> v2:172.21.15.165:3300/0 conn(0x7f7034099a40 msgr2=0x7f7034099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:08.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.876+0000 7f7039155700 1 --2- 172.21.15.1:0/1698031460 >> v2:172.21.15.165:3300/0 conn(0x7f7034099a40 0x7f7034099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.877+0000 7f7039155700 1 -- 172.21.15.1:0/1698031460 >> v1:172.21.15.165:6789/0 conn(0x7f703409a3e0 legacy=0x7f703406ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:08.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.877+0000 7f7039155700 1 -- 172.21.15.1:0/1698031460 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70340cf7f0 con 0x7f703415d5b0 2024-03-18T18:51:08.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.877+0000 7f7038954700 1 --2- 172.21.15.1:0/1698031460 >> v2:172.21.15.165:3300/0 conn(0x7f7034099a40 0x7f7034099ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:51:08.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.877+0000 7f7039155700 1 --2- 172.21.15.1:0/1698031460 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f703415d5b0 0x7f703415d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f70300098f0 tx=0x7f7030009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40d088ad72395216 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:08.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.877+0000 7f702bfff700 1 -- 172.21.15.1:0/1698031460 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7030011830 con 0x7f703415d5b0 2024-03-18T18:51:08.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.877+0000 7f702bfff700 1 -- 172.21.15.1:0/1698031460 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7030011990 con 0x7f703415d5b0 2024-03-18T18:51:08.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.877+0000 7f702bfff700 1 -- 172.21.15.1:0/1698031460 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7030011d20 con 0x7f703415d5b0 2024-03-18T18:51:08.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.877+0000 7f703b3b9700 1 -- 172.21.15.1:0/1698031460 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f703415d5b0 msgr2=0x7f703415d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:08.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.877+0000 7f703b3b9700 1 --2- 172.21.15.1:0/1698031460 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f703415d5b0 0x7f703415d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f70300098f0 tx=0x7f7030009c00 comp rx=0 tx=0).stop 2024-03-18T18:51:08.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.877+0000 7f703b3b9700 1 -- 172.21.15.1:0/1698031460 shutdown_connections 2024-03-18T18:51:08.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.877+0000 7f703b3b9700 1 --2- 172.21.15.1:0/1698031460 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f703415d5b0 0x7f703415d990 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.877+0000 7f703b3b9700 1 --2- 172.21.15.1:0/1698031460 >> v2:172.21.15.165:3300/0 conn(0x7f7034099a40 0x7f7034099ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.877+0000 7f703b3b9700 1 -- 172.21.15.1:0/1698031460 >> 172.21.15.1:0/1698031460 conn(0x7f70340b2bd0 msgr2=0x7f70340b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:08.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.878+0000 7f703b3b9700 1 -- 172.21.15.1:0/1698031460 shutdown_connections 2024-03-18T18:51:08.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.878+0000 7f703b3b9700 1 -- 172.21.15.1:0/1698031460 wait complete. 2024-03-18T18:51:08.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.879+0000 7f703b3b9700 1 Processor -- start 2024-03-18T18:51:08.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.879+0000 7f703b3b9700 1 -- start start 2024-03-18T18:51:08.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.879+0000 7f703b3b9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7034099a40 0x7f70340add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:08.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.879+0000 7f703b3b9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f703409a3e0 0x7f70340b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:08.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f703b3b9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f703415d5b0 0x7f70340b34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:08.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f703b3b9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70341f7880 con 0x7f7034099a40 2024-03-18T18:51:08.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f703b3b9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7034149650 con 0x7f703409a3e0 2024-03-18T18:51:08.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f703b3b9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f703414a630 con 0x7f703415d5b0 2024-03-18T18:51:08.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f7039155700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7034099a40 0x7f70340add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:08.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f7039155700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7034099a40 0x7f70340add90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53700/0 (socket says 172.21.15.1:53700) 2024-03-18T18:51:08.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f7039155700 1 -- 172.21.15.1:0/56932460 learned_addr learned my addr 172.21.15.1:0/56932460 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:08.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f7038954700 1 --2- 172.21.15.1:0/56932460 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f703409a3e0 0x7f70340b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:08.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f7039956700 1 --2- 172.21.15.1:0/56932460 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f703415d5b0 0x7f70340b34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:08.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f7039155700 1 -- 172.21.15.1:0/56932460 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f703415d5b0 msgr2=0x7f70340b34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:08.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f7039155700 1 --2- 172.21.15.1:0/56932460 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f703415d5b0 0x7f70340b34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f7039155700 1 -- 172.21.15.1:0/56932460 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f703409a3e0 msgr2=0x7f70340b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:08.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f7039155700 1 --2- 172.21.15.1:0/56932460 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f703409a3e0 0x7f70340b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:08.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f7039155700 1 -- 172.21.15.1:0/56932460 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70240094a0 con 0x7f7034099a40 2024-03-18T18:51:08.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f7038954700 1 --2- 172.21.15.1:0/56932460 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f703409a3e0 0x7f70340b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:08.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f7039155700 1 --2- 172.21.15.1:0/56932460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7034099a40 0x7f70340add90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7030009c40 tx=0x7f7030004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:08.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f7029ffb700 1 -- 172.21.15.1:0/56932460 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70300036a0 con 0x7f7034099a40 2024-03-18T18:51:08.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f7029ffb700 1 -- 172.21.15.1:0/56932460 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7030003800 con 0x7f7034099a40 2024-03-18T18:51:08.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f703b3b9700 1 -- 172.21.15.1:0/56932460 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7030009550 con 0x7f7034099a40 2024-03-18T18:51:08.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.880+0000 7f7029ffb700 1 -- 172.21.15.1:0/56932460 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7030003bb0 con 0x7f7034099a40 2024-03-18T18:51:08.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.881+0000 7f703b3b9700 1 -- 172.21.15.1:0/56932460 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7034149870 con 0x7f7034099a40 2024-03-18T18:51:08.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.882+0000 7f703b3b9700 1 -- 172.21.15.1:0/56932460 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f703406b3f0 con 0x7f7034099a40 2024-03-18T18:51:08.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.882+0000 7f7029ffb700 1 -- 172.21.15.1:0/56932460 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7030003d10 con 0x7f7034099a40 2024-03-18T18:51:08.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.882+0000 7f7029ffb700 1 --2- 172.21.15.1:0/56932460 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f701806ac90 0x7f701806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:08.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.883+0000 7f7029ffb700 1 -- 172.21.15.1:0/56932460 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f703008a350 con 0x7f7034099a40 2024-03-18T18:51:08.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.883+0000 7f7038954700 1 --2- 172.21.15.1:0/56932460 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f701806ac90 0x7f701806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:08.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.883+0000 7f7038954700 1 --2- 172.21.15.1:0/56932460 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f701806ac90 0x7f701806d150 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f70341f6860 tx=0x7f7024008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:08.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:08.887+0000 7f7029ffb700 1 -- 172.21.15.1:0/56932460 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f70300557d0 con 0x7f7034099a40 2024-03-18T18:51:09.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.175+0000 7f703b3b9700 1 -- 172.21.15.1:0/56932460 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f70340676a0 con 0x7f7034099a40 2024-03-18T18:51:09.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.178+0000 7f7029ffb700 1 -- 172.21.15.1:0/56932460 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7030058df0 con 0x7f7034099a40 2024-03-18T18:51:09.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.181+0000 7f703b3b9700 1 -- 172.21.15.1:0/56932460 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f701806ac90 msgr2=0x7f701806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:09.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.181+0000 7f703b3b9700 1 --2- 172.21.15.1:0/56932460 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f701806ac90 0x7f701806d150 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f70341f6860 tx=0x7f7024008040 comp rx=0 tx=0).stop 2024-03-18T18:51:09.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.181+0000 7f703b3b9700 1 -- 172.21.15.1:0/56932460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7034099a40 msgr2=0x7f70340add90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:09.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.181+0000 7f703b3b9700 1 --2- 172.21.15.1:0/56932460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7034099a40 0x7f70340add90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7030009c40 tx=0x7f7030004750 comp rx=0 tx=0).stop 2024-03-18T18:51:09.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.181+0000 7f703b3b9700 1 -- 172.21.15.1:0/56932460 shutdown_connections 2024-03-18T18:51:09.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.181+0000 7f703b3b9700 1 --2- 172.21.15.1:0/56932460 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f701806ac90 0x7f701806d150 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.181+0000 7f703b3b9700 1 --2- 172.21.15.1:0/56932460 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f703415d5b0 0x7f70340b34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.181+0000 7f703b3b9700 1 --2- 172.21.15.1:0/56932460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7034099a40 0x7f70340add90 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.181+0000 7f703b3b9700 1 --2- 172.21.15.1:0/56932460 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f703409a3e0 0x7f70340b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.181+0000 7f703b3b9700 1 -- 172.21.15.1:0/56932460 >> 172.21.15.1:0/56932460 conn(0x7f70340b2bd0 msgr2=0x7f703406f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:09.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.181+0000 7f703b3b9700 1 -- 172.21.15.1:0/56932460 shutdown_connections 2024-03-18T18:51:09.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.181+0000 7f703b3b9700 1 -- 172.21.15.1:0/56932460 wait complete. 2024-03-18T18:51:09.189 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-18T18:51:09.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.320+0000 7fb771150700 1 Processor -- start 2024-03-18T18:51:09.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.320+0000 7fb771150700 1 -- start start 2024-03-18T18:51:09.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.321+0000 7fb771150700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb76c158e30 0x7fb76c071500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:09.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.321+0000 7fb771150700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb76c0cbf70 0x7fb76c0cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:09.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.321+0000 7fb771150700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76c15c500 con 0x7fb76c06a750 2024-03-18T18:51:09.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.321+0000 7fb771150700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb76c0af8b0 con 0x7fb76c158e30 2024-03-18T18:51:09.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.321+0000 7fb771150700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb76c15b8f0 con 0x7fb76c0cbf70 2024-03-18T18:51:09.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.321+0000 7fb76ad9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb76c158e30 0x7fb76c071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:09.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.321+0000 7fb76ad9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb76c158e30 0x7fb76c071500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53708/0 (socket says 172.21.15.1:53708) 2024-03-18T18:51:09.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.321+0000 7fb76ad9d700 1 -- 172.21.15.1:0/2140075688 learned_addr learned my addr 172.21.15.1:0/2140075688 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:09.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.321+0000 7fb76a59c700 1 --2- 172.21.15.1:0/2140075688 >> v2:172.21.15.165:3300/0 conn(0x7fb76c0cbf70 0x7fb76c0cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:09.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.321+0000 7fb76ad9d700 1 -- 172.21.15.1:0/2140075688 >> v2:172.21.15.165:3300/0 conn(0x7fb76c0cbf70 msgr2=0x7fb76c0cc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:09.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.321+0000 7fb76ad9d700 1 --2- 172.21.15.1:0/2140075688 >> v2:172.21.15.165:3300/0 conn(0x7fb76c0cbf70 0x7fb76c0cc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.321+0000 7fb76ad9d700 1 -- 172.21.15.1:0/2140075688 >> v1:172.21.15.165:6789/0 conn(0x7fb76c06a750 legacy=0x7fb76c0cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:09.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.321+0000 7fb76ad9d700 1 -- 172.21.15.1:0/2140075688 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c15bbd0 con 0x7fb76c158e30 2024-03-18T18:51:09.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.321+0000 7fb76ad9d700 1 --2- 172.21.15.1:0/2140075688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb76c158e30 0x7fb76c071500 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0cee90 tx=0x7fb75c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563773ef4c5c70c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:09.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.321+0000 7fb769d9b700 1 -- 172.21.15.1:0/2140075688 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c00b870 con 0x7fb76c158e30 2024-03-18T18:51:09.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.321+0000 7fb769d9b700 1 -- 172.21.15.1:0/2140075688 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75c00b9d0 con 0x7fb76c158e30 2024-03-18T18:51:09.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.321+0000 7fb769d9b700 1 -- 172.21.15.1:0/2140075688 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c00bd60 con 0x7fb76c158e30 2024-03-18T18:51:09.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.322+0000 7fb771150700 1 -- 172.21.15.1:0/2140075688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb76c158e30 msgr2=0x7fb76c071500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:09.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.322+0000 7fb771150700 1 --2- 172.21.15.1:0/2140075688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb76c158e30 0x7fb76c071500 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0cee90 tx=0x7fb75c009bd0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.322+0000 7fb771150700 1 -- 172.21.15.1:0/2140075688 shutdown_connections 2024-03-18T18:51:09.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.322+0000 7fb771150700 1 --2- 172.21.15.1:0/2140075688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb76c158e30 0x7fb76c071500 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.322+0000 7fb771150700 1 --2- 172.21.15.1:0/2140075688 >> v2:172.21.15.165:3300/0 conn(0x7fb76c0cbf70 0x7fb76c0cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.322+0000 7fb771150700 1 -- 172.21.15.1:0/2140075688 >> 172.21.15.1:0/2140075688 conn(0x7fb76c0cca00 msgr2=0x7fb76c0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:09.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.322+0000 7fb771150700 1 -- 172.21.15.1:0/2140075688 shutdown_connections 2024-03-18T18:51:09.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.322+0000 7fb771150700 1 -- 172.21.15.1:0/2140075688 wait complete. 2024-03-18T18:51:09.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.323+0000 7fb771150700 1 Processor -- start 2024-03-18T18:51:09.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.324+0000 7fb771150700 1 -- start start 2024-03-18T18:51:09.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.324+0000 7fb771150700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb76c06a750 0x7fb76c158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.324+0000 7fb771150700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb76c0cbf70 0x7fb76c15a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.324+0000 7fb771150700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb76c158e30 0x7fb76c15ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.324+0000 7fb771150700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c06d520 con 0x7fb76c06a750 2024-03-18T18:51:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.324+0000 7fb771150700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c06d800 con 0x7fb76c158e30 2024-03-18T18:51:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.324+0000 7fb771150700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c06dae0 con 0x7fb76c0cbf70 2024-03-18T18:51:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.324+0000 7fb76ad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb76c06a750 0x7fb76c158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.324+0000 7fb76ad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb76c06a750 0x7fb76c158370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53712/0 (socket says 172.21.15.1:53712) 2024-03-18T18:51:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.324+0000 7fb76ad9d700 1 -- 172.21.15.1:0/2618567414 learned_addr learned my addr 172.21.15.1:0/2618567414 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.324+0000 7fb76b59e700 1 --2- 172.21.15.1:0/2618567414 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb76c158e30 0x7fb76c15ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.324+0000 7fb76a59c700 1 --2- 172.21.15.1:0/2618567414 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb76c0cbf70 0x7fb76c15a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.325+0000 7fb76ad9d700 1 -- 172.21.15.1:0/2618567414 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb76c0cbf70 msgr2=0x7fb76c15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:09.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.325+0000 7fb76ad9d700 1 --2- 172.21.15.1:0/2618567414 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb76c0cbf70 0x7fb76c15a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.325+0000 7fb76ad9d700 1 -- 172.21.15.1:0/2618567414 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb76c158e30 msgr2=0x7fb76c15ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:09.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.325+0000 7fb76ad9d700 1 --2- 172.21.15.1:0/2618567414 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb76c158e30 0x7fb76c15ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.325+0000 7fb76ad9d700 1 -- 172.21.15.1:0/2618567414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75c009550 con 0x7fb76c06a750 2024-03-18T18:51:09.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.325+0000 7fb76ad9d700 1 --2- 172.21.15.1:0/2618567414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb76c06a750 0x7fb76c158370 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb75c000c00 tx=0x7fb75c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:09.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.325+0000 7fb7537fe700 1 -- 172.21.15.1:0/2618567414 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c004310 con 0x7fb76c06a750 2024-03-18T18:51:09.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.325+0000 7fb7537fe700 1 -- 172.21.15.1:0/2618567414 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75c01e070 con 0x7fb76c06a750 2024-03-18T18:51:09.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.325+0000 7fb771150700 1 -- 172.21.15.1:0/2618567414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76c06ddc0 con 0x7fb76c06a750 2024-03-18T18:51:09.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.325+0000 7fb7537fe700 1 -- 172.21.15.1:0/2618567414 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c0036a0 con 0x7fb76c06a750 2024-03-18T18:51:09.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.325+0000 7fb771150700 1 -- 172.21.15.1:0/2618567414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb76c1588b0 con 0x7fb76c06a750 2024-03-18T18:51:09.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.326+0000 7fb771150700 1 -- 172.21.15.1:0/2618567414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb76c1f3240 con 0x7fb76c06a750 2024-03-18T18:51:09.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.327+0000 7fb7537fe700 1 -- 172.21.15.1:0/2618567414 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb75c020030 con 0x7fb76c06a750 2024-03-18T18:51:09.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.327+0000 7fb7537fe700 1 --2- 172.21.15.1:0/2618567414 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb75806ac90 0x7fb75806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:09.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.327+0000 7fb7537fe700 1 -- 172.21.15.1:0/2618567414 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fb75c089fd0 con 0x7fb76c06a750 2024-03-18T18:51:09.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.327+0000 7fb76a59c700 1 --2- 172.21.15.1:0/2618567414 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb75806ac90 0x7fb75806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:09.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.328+0000 7fb76a59c700 1 --2- 172.21.15.1:0/2618567414 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb75806ac90 0x7fb75806d150 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0c6a20 tx=0x7fb760008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:09.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.332+0000 7fb7537fe700 1 -- 172.21.15.1:0/2618567414 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb75c055450 con 0x7fb76c06a750 2024-03-18T18:51:09.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.616+0000 7fb771150700 1 -- 172.21.15.1:0/2618567414 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7fb76c073940 con 0x7fb76c06a750 2024-03-18T18:51:09.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.619+0000 7fb7537fe700 1 -- 172.21.15.1:0/2618567414 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7fb75c05de60 con 0x7fb76c06a750 2024-03-18T18:51:09.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.621+0000 7fb771150700 1 -- 172.21.15.1:0/2618567414 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb75806ac90 msgr2=0x7fb75806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:09.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.621+0000 7fb771150700 1 --2- 172.21.15.1:0/2618567414 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb75806ac90 0x7fb75806d150 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0c6a20 tx=0x7fb760008040 comp rx=0 tx=0).stop 2024-03-18T18:51:09.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.621+0000 7fb771150700 1 -- 172.21.15.1:0/2618567414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb76c06a750 msgr2=0x7fb76c158370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:09.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.621+0000 7fb771150700 1 --2- 172.21.15.1:0/2618567414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb76c06a750 0x7fb76c158370 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb75c000c00 tx=0x7fb75c004750 comp rx=0 tx=0).stop 2024-03-18T18:51:09.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.621+0000 7fb771150700 1 -- 172.21.15.1:0/2618567414 shutdown_connections 2024-03-18T18:51:09.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.621+0000 7fb771150700 1 --2- 172.21.15.1:0/2618567414 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb75806ac90 0x7fb75806d150 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.621+0000 7fb771150700 1 --2- 172.21.15.1:0/2618567414 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb76c158e30 0x7fb76c15ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.621+0000 7fb771150700 1 --2- 172.21.15.1:0/2618567414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb76c06a750 0x7fb76c158370 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.621+0000 7fb771150700 1 --2- 172.21.15.1:0/2618567414 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb76c0cbf70 0x7fb76c15a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.621+0000 7fb771150700 1 -- 172.21.15.1:0/2618567414 >> 172.21.15.1:0/2618567414 conn(0x7fb76c0cca00 msgr2=0x7fb76c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:09.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.621+0000 7fb771150700 1 -- 172.21.15.1:0/2618567414 shutdown_connections 2024-03-18T18:51:09.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.622+0000 7fb771150700 1 -- 172.21.15.1:0/2618567414 wait complete. 2024-03-18T18:51:09.631 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:51:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.759+0000 7f9ac73fc700 1 Processor -- start 2024-03-18T18:51:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.760+0000 7f9ac73fc700 1 -- start start 2024-03-18T18:51:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.760+0000 7f9ac73fc700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9ac0146190 0x7f9ac006a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.760+0000 7f9ac73fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9ac0146af0 0x7f9ac006ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.760+0000 7f9ac73fc700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac006ebd0 con 0x7f9ac01465a0 2024-03-18T18:51:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.760+0000 7f9ac73fc700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ac006f850 con 0x7f9ac0146af0 2024-03-18T18:51:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.760+0000 7f9ac73fc700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ac006eeb0 con 0x7f9ac0146190 2024-03-18T18:51:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.760+0000 7f9ac5999700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9ac0146af0 0x7f9ac006ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.761+0000 7f9ac5999700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9ac0146af0 0x7f9ac006ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53722/0 (socket says 172.21.15.1:53722) 2024-03-18T18:51:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.761+0000 7f9ac5999700 1 -- 172.21.15.1:0/2870804017 learned_addr learned my addr 172.21.15.1:0/2870804017 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.761+0000 7f9ac5198700 1 --2- 172.21.15.1:0/2870804017 >> v2:172.21.15.165:3300/0 conn(0x7f9ac0146190 0x7f9ac006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.761+0000 7f9ac5999700 1 -- 172.21.15.1:0/2870804017 >> v2:172.21.15.165:3300/0 conn(0x7f9ac0146190 msgr2=0x7f9ac006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:09.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.761+0000 7f9ac5999700 1 --2- 172.21.15.1:0/2870804017 >> v2:172.21.15.165:3300/0 conn(0x7f9ac0146190 0x7f9ac006a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.761+0000 7f9ac5999700 1 -- 172.21.15.1:0/2870804017 >> v1:172.21.15.165:6789/0 conn(0x7f9ac01465a0 legacy=0x7f9ac0071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.761+0000 7f9ac5999700 1 -- 172.21.15.1:0/2870804017 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac006f190 con 0x7f9ac0146af0 2024-03-18T18:51:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.761+0000 7f9ac5999700 1 --2- 172.21.15.1:0/2870804017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ac0146af0 0x7f9ac006ac90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f9ac006e8f0 tx=0x7f9ab800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=675765f918739739 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.761+0000 7f9ab7fff700 1 -- 172.21.15.1:0/2870804017 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab800e070 con 0x7f9ac0146af0 2024-03-18T18:51:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.761+0000 7f9ab7fff700 1 -- 172.21.15.1:0/2870804017 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab8003a20 con 0x7f9ac0146af0 2024-03-18T18:51:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.761+0000 7f9ab7fff700 1 -- 172.21.15.1:0/2870804017 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8003dd0 con 0x7f9ac0146af0 2024-03-18T18:51:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.762+0000 7f9ac73fc700 1 -- 172.21.15.1:0/2870804017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ac0146af0 msgr2=0x7f9ac006ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.762+0000 7f9ac73fc700 1 --2- 172.21.15.1:0/2870804017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ac0146af0 0x7f9ac006ac90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f9ac006e8f0 tx=0x7f9ab800b330 comp rx=0 tx=0).stop 2024-03-18T18:51:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.762+0000 7f9ac73fc700 1 -- 172.21.15.1:0/2870804017 shutdown_connections 2024-03-18T18:51:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.762+0000 7f9ac73fc700 1 --2- 172.21.15.1:0/2870804017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ac0146af0 0x7f9ac006ac90 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.762+0000 7f9ac73fc700 1 --2- 172.21.15.1:0/2870804017 >> v2:172.21.15.165:3300/0 conn(0x7f9ac0146190 0x7f9ac006a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.762+0000 7f9ac73fc700 1 -- 172.21.15.1:0/2870804017 >> 172.21.15.1:0/2870804017 conn(0x7f9ac00b1d20 msgr2=0x7f9ac00b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:09.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.762+0000 7f9ac73fc700 1 -- 172.21.15.1:0/2870804017 shutdown_connections 2024-03-18T18:51:09.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.762+0000 7f9ac73fc700 1 -- 172.21.15.1:0/2870804017 wait complete. 2024-03-18T18:51:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.763+0000 7f9ac73fc700 1 Processor -- start 2024-03-18T18:51:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.763+0000 7f9ac73fc700 1 -- start start 2024-03-18T18:51:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.764+0000 7f9ac73fc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ac01465a0 0x7f9ac00cba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.764+0000 7f9ac73fc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ac0146af0 0x7f9ac00c4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.764+0000 7f9ac73fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ac00c54d0 0x7f9ac00cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.764+0000 7f9ac73fc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac013f1f0 con 0x7f9ac00c54d0 2024-03-18T18:51:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.764+0000 7f9ac73fc700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00c5960 con 0x7f9ac01465a0 2024-03-18T18:51:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.764+0000 7f9ac73fc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00cbf60 con 0x7f9ac0146af0 2024-03-18T18:51:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.764+0000 7f9ac5999700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ac00c54d0 0x7f9ac00cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.764+0000 7f9ac5999700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ac00c54d0 0x7f9ac00cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53730/0 (socket says 172.21.15.1:53730) 2024-03-18T18:51:09.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.764+0000 7f9ac5999700 1 -- 172.21.15.1:0/2580543826 learned_addr learned my addr 172.21.15.1:0/2580543826 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.764+0000 7f9ac5198700 1 --2- 172.21.15.1:0/2580543826 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ac01465a0 0x7f9ac00cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.764+0000 7f9ac4997700 1 --2- 172.21.15.1:0/2580543826 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ac0146af0 0x7f9ac00c4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.764+0000 7f9ac5999700 1 -- 172.21.15.1:0/2580543826 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ac0146af0 msgr2=0x7f9ac00c4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.764+0000 7f9ac5999700 1 --2- 172.21.15.1:0/2580543826 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ac0146af0 0x7f9ac00c4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.764+0000 7f9ac5999700 1 -- 172.21.15.1:0/2580543826 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ac01465a0 msgr2=0x7f9ac00cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.764+0000 7f9ac5999700 1 --2- 172.21.15.1:0/2580543826 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ac01465a0 0x7f9ac00cba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.764+0000 7f9ac5999700 1 -- 172.21.15.1:0/2580543826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab8009b30 con 0x7f9ac00c54d0 2024-03-18T18:51:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.764+0000 7f9ac4997700 1 --2- 172.21.15.1:0/2580543826 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ac0146af0 0x7f9ac00c4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.764+0000 7f9ac5999700 1 --2- 172.21.15.1:0/2580543826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ac00c54d0 0x7f9ac00cdba0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8015040 tx=0x7f9ab8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.765+0000 7f9ab5ffb700 1 -- 172.21.15.1:0/2580543826 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8004440 con 0x7f9ac00c54d0 2024-03-18T18:51:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.765+0000 7f9ab5ffb700 1 -- 172.21.15.1:0/2580543826 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab80045c0 con 0x7f9ac00c54d0 2024-03-18T18:51:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.765+0000 7f9ac73fc700 1 -- 172.21.15.1:0/2580543826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac00ce8d0 con 0x7f9ac00c54d0 2024-03-18T18:51:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.765+0000 7f9ab5ffb700 1 -- 172.21.15.1:0/2580543826 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8004970 con 0x7f9ac00c54d0 2024-03-18T18:51:09.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.765+0000 7f9ac73fc700 1 -- 172.21.15.1:0/2580543826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac00cee90 con 0x7f9ac00c54d0 2024-03-18T18:51:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.766+0000 7f9ac73fc700 1 -- 172.21.15.1:0/2580543826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac00676a0 con 0x7f9ac00c54d0 2024-03-18T18:51:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.766+0000 7f9ab5ffb700 1 -- 172.21.15.1:0/2580543826 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9ab8012430 con 0x7f9ac00c54d0 2024-03-18T18:51:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.767+0000 7f9ab5ffb700 1 --2- 172.21.15.1:0/2580543826 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9aa406ac90 0x7f9aa406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.767+0000 7f9ab5ffb700 1 -- 172.21.15.1:0/2580543826 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f9ab808a680 con 0x7f9ac00c54d0 2024-03-18T18:51:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.767+0000 7f9ac5198700 1 --2- 172.21.15.1:0/2580543826 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9aa406ac90 0x7f9aa406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.767+0000 7f9ac5198700 1 --2- 172.21.15.1:0/2580543826 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9aa406ac90 0x7f9aa406d150 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000a990 tx=0x7f9ab0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:09.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.771+0000 7f9ab5ffb700 1 -- 172.21.15.1:0/2580543826 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9ab8055b00 con 0x7f9ac00c54d0 2024-03-18T18:51:09.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.975+0000 7f9ac73fc700 1 -- 172.21.15.1:0/2580543826 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ac00ad4d0 con 0x7f9ac00c54d0 2024-03-18T18:51:09.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.976+0000 7f9ab5ffb700 1 -- 172.21.15.1:0/2580543826 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+12546 (secure 0 0 0) 0x7f9ab8059120 con 0x7f9ac00c54d0 2024-03-18T18:51:09.973 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:51:09.973 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":36,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:51:06.476975+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:51:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.979+0000 7f9ac73fc700 1 -- 172.21.15.1:0/2580543826 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9aa406ac90 msgr2=0x7f9aa406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.979+0000 7f9ac73fc700 1 --2- 172.21.15.1:0/2580543826 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9aa406ac90 0x7f9aa406d150 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000a990 tx=0x7f9ab0008040 comp rx=0 tx=0).stop 2024-03-18T18:51:09.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.979+0000 7f9ac73fc700 1 -- 172.21.15.1:0/2580543826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ac00c54d0 msgr2=0x7f9ac00cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:09.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.979+0000 7f9ac73fc700 1 --2- 172.21.15.1:0/2580543826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ac00c54d0 0x7f9ac00cdba0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8015040 tx=0x7f9ab8004ad0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.979+0000 7f9ac73fc700 1 -- 172.21.15.1:0/2580543826 shutdown_connections 2024-03-18T18:51:09.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.979+0000 7f9ac73fc700 1 --2- 172.21.15.1:0/2580543826 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9aa406ac90 0x7f9aa406d150 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.979+0000 7f9ac73fc700 1 --2- 172.21.15.1:0/2580543826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ac00c54d0 0x7f9ac00cdba0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.979+0000 7f9ac73fc700 1 --2- 172.21.15.1:0/2580543826 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ac01465a0 0x7f9ac00cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.979+0000 7f9ac73fc700 1 --2- 172.21.15.1:0/2580543826 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ac0146af0 0x7f9ac00c4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:09.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.979+0000 7f9ac73fc700 1 -- 172.21.15.1:0/2580543826 >> 172.21.15.1:0/2580543826 conn(0x7f9ac00b1d20 msgr2=0x7f9ac0071650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:09.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.979+0000 7f9ac73fc700 1 -- 172.21.15.1:0/2580543826 shutdown_connections 2024-03-18T18:51:09.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:09.979+0000 7f9ac73fc700 1 -- 172.21.15.1:0/2580543826 wait complete. 2024-03-18T18:51:09.987 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T18:51:09.987 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T18:51:10.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.118+0000 7fdd23f93700 1 Processor -- start 2024-03-18T18:51:10.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.118+0000 7fdd23f93700 1 -- start start 2024-03-18T18:51:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.119+0000 7fdd23f93700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdd1c0aecb0 0x7fdd1c070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.119+0000 7fdd23f93700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c15b750 con 0x7fdd1c071380 2024-03-18T18:51:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.119+0000 7fdd23f93700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c0ccc20 con 0x7fdd1c15b370 2024-03-18T18:51:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.119+0000 7fdd23f93700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd1c0cf510 con 0x7fdd1c0aecb0 2024-03-18T18:51:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.119+0000 7fdd22530700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdd1c071380 0x7fdd1c06d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57412/0 (socket says 172.21.15.1:57412) 2024-03-18T18:51:10.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.119+0000 7fdd22530700 1 -- 172.21.15.1:0/733060373 learned_addr learned my addr 172.21.15.1:0/733060373 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.119+0000 7fdd2152e700 1 --2- 172.21.15.1:0/733060373 >> v2:172.21.15.165:3300/0 conn(0x7fdd1c0aecb0 0x7fdd1c070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.119+0000 7fdd20d2d700 1 -- 172.21.15.1:0/733060373 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252846156 0 0) 0x7fdd1c15b750 con 0x7fdd1c071380 2024-03-18T18:51:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.119+0000 7fdd20d2d700 1 -- 172.21.15.1:0/733060373 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd080035b0 con 0x7fdd1c071380 2024-03-18T18:51:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd20d2d700 1 -- 172.21.15.1:0/733060373 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2916637849 0 0) 0x7fdd080035b0 con 0x7fdd1c071380 2024-03-18T18:51:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd20d2d700 1 -- 172.21.15.1:0/733060373 >> v2:172.21.15.165:3300/0 conn(0x7fdd1c0aecb0 msgr2=0x7fdd1c070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd20d2d700 1 --2- 172.21.15.1:0/733060373 >> v2:172.21.15.165:3300/0 conn(0x7fdd1c0aecb0 0x7fdd1c070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd20d2d700 1 -- 172.21.15.1:0/733060373 >> v1:172.21.15.165:6789/0 conn(0x7fdd1c15b370 legacy=0x7fdd1c0cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd20d2d700 1 -- 172.21.15.1:0/733060373 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1c0cc2f0 con 0x7fdd1c071380 2024-03-18T18:51:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd20d2d700 1 -- 172.21.15.1:0/733060373 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fdd140019f0 con 0x7fdd1c071380 2024-03-18T18:51:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd20d2d700 1 -- 172.21.15.1:0/733060373 >> v1:172.21.15.1:6789/0 conn(0x7fdd1c071380 legacy=0x7fdd1c06d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd20d2d700 1 --2- 172.21.15.1:0/733060373 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd08003a00 0x7fdd08003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:10.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd20d2d700 1 --2- 172.21.15.1:0/733060373 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd08004850 0x7fdd08004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd20d2d700 1 --2- 172.21.15.1:0/733060373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd08005210 0x7fdd0800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd20d2d700 1 -- 172.21.15.1:0/733060373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c0cf510 con 0x7fdd08005210 2024-03-18T18:51:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd20d2d700 1 -- 172.21.15.1:0/733060373 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0800c1b0 con 0x7fdd08003a00 2024-03-18T18:51:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd20d2d700 1 -- 172.21.15.1:0/733060373 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0800c370 con 0x7fdd08004850 2024-03-18T18:51:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd22530700 1 --2- 172.21.15.1:0/733060373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd08005210 0x7fdd0800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd21d2f700 1 --2- 172.21.15.1:0/733060373 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd08003a00 0x7fdd08003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd2152e700 1 --2- 172.21.15.1:0/733060373 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd08004850 0x7fdd08004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd22530700 1 -- 172.21.15.1:0/733060373 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd08004850 msgr2=0x7fdd08004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd22530700 1 --2- 172.21.15.1:0/733060373 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd08004850 0x7fdd08004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd22530700 1 -- 172.21.15.1:0/733060373 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd08003a00 msgr2=0x7fdd08003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd22530700 1 --2- 172.21.15.1:0/733060373 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd08003a00 0x7fdd08003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd22530700 1 -- 172.21.15.1:0/733060373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd080035b0 con 0x7fdd08005210 2024-03-18T18:51:10.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd22530700 1 -- 172.21.15.1:0/733060373 reap_dead start 2024-03-18T18:51:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.120+0000 7fdd22530700 1 --2- 172.21.15.1:0/733060373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd08005210 0x7fdd0800ba10 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdd1400a820 tx=0x7fdd08005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f5110bc2900c4c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.121+0000 7fdd20d2d700 1 -- 172.21.15.1:0/733060373 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1400a2f0 con 0x7fdd08005210 2024-03-18T18:51:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.121+0000 7fdd20d2d700 1 -- 172.21.15.1:0/733060373 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd14012620 con 0x7fdd08005210 2024-03-18T18:51:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.121+0000 7fdd20d2d700 1 -- 172.21.15.1:0/733060373 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd140129d0 con 0x7fdd08005210 2024-03-18T18:51:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.121+0000 7fdd23f93700 1 -- 172.21.15.1:0/733060373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd08005210 msgr2=0x7fdd0800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.121+0000 7fdd23f93700 1 --2- 172.21.15.1:0/733060373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd08005210 0x7fdd0800ba10 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdd1400a820 tx=0x7fdd08005880 comp rx=0 tx=0).stop 2024-03-18T18:51:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.121+0000 7fdd23f93700 1 -- 172.21.15.1:0/733060373 shutdown_connections 2024-03-18T18:51:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.121+0000 7fdd23f93700 1 --2- 172.21.15.1:0/733060373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd08005210 0x7fdd0800ba10 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.121+0000 7fdd23f93700 1 -- 172.21.15.1:0/733060373 >> 172.21.15.1:0/733060373 conn(0x7fdd1c0b1350 msgr2=0x7fdd1c0b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.121+0000 7fdd23f93700 1 -- 172.21.15.1:0/733060373 shutdown_connections 2024-03-18T18:51:10.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.121+0000 7fdd23f93700 1 -- 172.21.15.1:0/733060373 wait complete. 2024-03-18T18:51:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.122+0000 7fdd23f93700 1 Processor -- start 2024-03-18T18:51:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.122+0000 7fdd23f93700 1 -- start start 2024-03-18T18:51:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.123+0000 7fdd23f93700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd08005210 0x7fdd1c14c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.123+0000 7fdd23f93700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd1c15edf0 0x7fdd1c15f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.123+0000 7fdd23f93700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd1c0ae0b0 0x7fdd1c0ae560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.123+0000 7fdd23f93700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c071830 con 0x7fdd1c0ae0b0 2024-03-18T18:51:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.123+0000 7fdd23f93700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c0b34c0 con 0x7fdd1c15edf0 2024-03-18T18:51:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.123+0000 7fdd23f93700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c15f7b0 con 0x7fdd08005210 2024-03-18T18:51:10.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.123+0000 7fdd22530700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd1c0ae0b0 0x7fdd1c0ae560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.123+0000 7fdd22530700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd1c0ae0b0 0x7fdd1c0ae560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53746/0 (socket says 172.21.15.1:53746) 2024-03-18T18:51:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.123+0000 7fdd22530700 1 -- 172.21.15.1:0/1135220719 learned_addr learned my addr 172.21.15.1:0/1135220719 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.123+0000 7fdd2152e700 1 --2- 172.21.15.1:0/1135220719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd1c15edf0 0x7fdd1c15f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.123+0000 7fdd21d2f700 1 --2- 172.21.15.1:0/1135220719 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd08005210 0x7fdd1c14c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.124+0000 7fdd22530700 1 -- 172.21.15.1:0/1135220719 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd08005210 msgr2=0x7fdd1c14c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.124+0000 7fdd22530700 1 --2- 172.21.15.1:0/1135220719 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd08005210 0x7fdd1c14c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.124+0000 7fdd22530700 1 -- 172.21.15.1:0/1135220719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd1c15edf0 msgr2=0x7fdd1c15f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.124+0000 7fdd22530700 1 --2- 172.21.15.1:0/1135220719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd1c15edf0 0x7fdd1c15f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.124+0000 7fdd22530700 1 -- 172.21.15.1:0/1135220719 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1c14c700 con 0x7fdd1c0ae0b0 2024-03-18T18:51:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.124+0000 7fdd22530700 1 --2- 172.21.15.1:0/1135220719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd1c0ae0b0 0x7fdd1c0ae560 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c15edc0 tx=0x7fdd1c0ccb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.124+0000 7fdd0e7fc700 1 -- 172.21.15.1:0/1135220719 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd14024070 con 0x7fdd1c0ae0b0 2024-03-18T18:51:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.124+0000 7fdd0e7fc700 1 -- 172.21.15.1:0/1135220719 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd140012d0 con 0x7fdd1c0ae0b0 2024-03-18T18:51:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.124+0000 7fdd23f93700 1 -- 172.21.15.1:0/1135220719 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd1c0b2880 con 0x7fdd1c0ae0b0 2024-03-18T18:51:10.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.124+0000 7fdd0e7fc700 1 -- 172.21.15.1:0/1135220719 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd140017b0 con 0x7fdd1c0ae0b0 2024-03-18T18:51:10.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.124+0000 7fdd23f93700 1 -- 172.21.15.1:0/1135220719 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd1c0b2b20 con 0x7fdd1c0ae0b0 2024-03-18T18:51:10.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.125+0000 7fdd23f93700 1 -- 172.21.15.1:0/1135220719 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd1c0b3600 con 0x7fdd1c0ae0b0 2024-03-18T18:51:10.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.125+0000 7fdd0e7fc700 1 -- 172.21.15.1:0/1135220719 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fdd1401d420 con 0x7fdd1c0ae0b0 2024-03-18T18:51:10.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.126+0000 7fdd0e7fc700 1 --2- 172.21.15.1:0/1135220719 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdd0006ac90 0x7fdd0006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:10.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.126+0000 7fdd0e7fc700 1 -- 172.21.15.1:0/1135220719 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fdd1408abd0 con 0x7fdd1c0ae0b0 2024-03-18T18:51:10.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.126+0000 7fdd21d2f700 1 --2- 172.21.15.1:0/1135220719 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdd0006ac90 0x7fdd0006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:10.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.126+0000 7fdd21d2f700 1 --2- 172.21.15.1:0/1135220719 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdd0006ac90 0x7fdd0006d150 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdd10000c00 tx=0x7fdd10009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:10.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.130+0000 7fdd0e7fc700 1 -- 172.21.15.1:0/1135220719 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdd1c0b3600 con 0x7fdd1c0ae0b0 2024-03-18T18:51:10.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.329+0000 7fdd23f93700 1 -- 172.21.15.1:0/1135220719 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdd1c0b3600 con 0x7fdd1c0ae0b0 2024-03-18T18:51:10.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.329+0000 7fdd0e7fc700 1 -- 172.21.15.1:0/1135220719 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdd1c0b3600 con 0x7fdd1c0ae0b0 2024-03-18T18:51:10.326 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T18:51:10.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.331+0000 7fdd23f93700 1 -- 172.21.15.1:0/1135220719 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdd0006ac90 msgr2=0x7fdd0006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:10.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.331+0000 7fdd23f93700 1 --2- 172.21.15.1:0/1135220719 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdd0006ac90 0x7fdd0006d150 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdd10000c00 tx=0x7fdd10009040 comp rx=0 tx=0).stop 2024-03-18T18:51:10.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.331+0000 7fdd23f93700 1 -- 172.21.15.1:0/1135220719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd1c0ae0b0 msgr2=0x7fdd1c0ae560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:10.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.331+0000 7fdd23f93700 1 --2- 172.21.15.1:0/1135220719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd1c0ae0b0 0x7fdd1c0ae560 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c15edc0 tx=0x7fdd1c0ccb90 comp rx=0 tx=0).stop 2024-03-18T18:51:10.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.331+0000 7fdd23f93700 1 -- 172.21.15.1:0/1135220719 shutdown_connections 2024-03-18T18:51:10.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.331+0000 7fdd23f93700 1 --2- 172.21.15.1:0/1135220719 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdd0006ac90 0x7fdd0006d150 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.331+0000 7fdd23f93700 1 --2- 172.21.15.1:0/1135220719 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd08005210 0x7fdd1c14c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.331+0000 7fdd23f93700 1 --2- 172.21.15.1:0/1135220719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd1c0ae0b0 0x7fdd1c0ae560 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.331+0000 7fdd23f93700 1 --2- 172.21.15.1:0/1135220719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd1c15edf0 0x7fdd1c15f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.331+0000 7fdd23f93700 1 -- 172.21.15.1:0/1135220719 >> 172.21.15.1:0/1135220719 conn(0x7fdd1c0b1350 msgr2=0x7fdd1c06e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:10.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.332+0000 7fdd23f93700 1 -- 172.21.15.1:0/1135220719 shutdown_connections 2024-03-18T18:51:10.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.332+0000 7fdd23f93700 1 -- 172.21.15.1:0/1135220719 wait complete. 2024-03-18T18:51:10.340 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T18:51:10.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.469+0000 7f8722b19700 1 Processor -- start 2024-03-18T18:51:10.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.469+0000 7f8722b19700 1 -- start start 2024-03-18T18:51:10.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.470+0000 7f8722b19700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f871c0cd110 0x7f871c166080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:10.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.470+0000 7f8722b19700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c0cf090 con 0x7f871c144b70 2024-03-18T18:51:10.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.470+0000 7f8722b19700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c072a20 con 0x7f871c0af610 2024-03-18T18:51:10.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.470+0000 7f8722b19700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f871c144f50 con 0x7f871c0cd110 2024-03-18T18:51:10.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.470+0000 7f87210b6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f871c0cd110 0x7f871c166080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:10.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.470+0000 7f87208b5700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f871c144b70 0x7f871c0ce980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57420/0 (socket says 172.21.15.1:57420) 2024-03-18T18:51:10.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.470+0000 7f87208b5700 1 -- 172.21.15.1:0/1595671536 learned_addr learned my addr 172.21.15.1:0/1595671536 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:10.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.470+0000 7f87210b6700 1 -- 172.21.15.1:0/1595671536 >> v1:172.21.15.165:6789/0 conn(0x7f871c0af610 legacy=0x7f871c0cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:10.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.470+0000 7f87210b6700 1 -- 172.21.15.1:0/1595671536 >> v1:172.21.15.1:6789/0 conn(0x7f871c144b70 legacy=0x7f871c0ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:10.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.470+0000 7f871b7fe700 1 -- 172.21.15.1:0/1595671536 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552646021 0 0) 0x7f8710001110 con 0x7f871c144b70 2024-03-18T18:51:10.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.470+0000 7f87210b6700 1 -- 172.21.15.1:0/1595671536 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871c0cf2d0 con 0x7f871c0cd110 2024-03-18T18:51:10.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.470+0000 7f87210b6700 1 --2- 172.21.15.1:0/1595671536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f871c0cd110 0x7f871c166080 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8714009d90 tx=0x7f871400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e87789ea5e29bee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:10.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.470+0000 7f871b7fe700 1 -- 172.21.15.1:0/1595671536 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871400e070 con 0x7f871c0cd110 2024-03-18T18:51:10.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.471+0000 7f871b7fe700 1 -- 172.21.15.1:0/1595671536 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f871400bbd0 con 0x7f871c0cd110 2024-03-18T18:51:10.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.471+0000 7f871b7fe700 1 -- 172.21.15.1:0/1595671536 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8714012430 con 0x7f871c0cd110 2024-03-18T18:51:10.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.471+0000 7f8722b19700 1 -- 172.21.15.1:0/1595671536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f871c0cd110 msgr2=0x7f871c166080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:10.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.471+0000 7f8722b19700 1 --2- 172.21.15.1:0/1595671536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f871c0cd110 0x7f871c166080 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8714009d90 tx=0x7f871400b330 comp rx=0 tx=0).stop 2024-03-18T18:51:10.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.471+0000 7f8722b19700 1 -- 172.21.15.1:0/1595671536 shutdown_connections 2024-03-18T18:51:10.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.471+0000 7f8722b19700 1 --2- 172.21.15.1:0/1595671536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f871c0cd110 0x7f871c166080 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.471+0000 7f8722b19700 1 -- 172.21.15.1:0/1595671536 >> 172.21.15.1:0/1595671536 conn(0x7f871c0b1d20 msgr2=0x7f871c0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:10.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.471+0000 7f8722b19700 1 -- 172.21.15.1:0/1595671536 shutdown_connections 2024-03-18T18:51:10.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.471+0000 7f8722b19700 1 -- 172.21.15.1:0/1595671536 wait complete. 2024-03-18T18:51:10.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.472+0000 7f8722b19700 1 Processor -- start 2024-03-18T18:51:10.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.472+0000 7f8722b19700 1 -- start start 2024-03-18T18:51:10.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.473+0000 7f8722b19700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f871c0af610 0x7f871c137860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:10.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.473+0000 7f8722b19700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f871c0cd110 0x7f871c0ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:10.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.473+0000 7f8722b19700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f871c144b70 0x7f871c1439d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:10.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.473+0000 7f8722b19700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871c142ff0 con 0x7f871c0cd110 2024-03-18T18:51:10.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.473+0000 7f8722b19700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871c1426c0 con 0x7f871c144b70 2024-03-18T18:51:10.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.473+0000 7f8722b19700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871c1429a0 con 0x7f871c0af610 2024-03-18T18:51:10.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.473+0000 7f871bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f871c0cd110 0x7f871c0ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:10.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.473+0000 7f871bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f871c0cd110 0x7f871c0ac8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53766/0 (socket says 172.21.15.1:53766) 2024-03-18T18:51:10.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.473+0000 7f871bfff700 1 -- 172.21.15.1:0/1739263325 learned_addr learned my addr 172.21.15.1:0/1739263325 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:10.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.473+0000 7f87208b5700 1 --2- 172.21.15.1:0/1739263325 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f871c0af610 0x7f871c137860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:10.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.473+0000 7f87210b6700 1 --2- 172.21.15.1:0/1739263325 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f871c144b70 0x7f871c1439d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:10.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.474+0000 7f871bfff700 1 -- 172.21.15.1:0/1739263325 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f871c0af610 msgr2=0x7f871c137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:10.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.474+0000 7f871bfff700 1 --2- 172.21.15.1:0/1739263325 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f871c0af610 0x7f871c137860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.474+0000 7f871bfff700 1 -- 172.21.15.1:0/1739263325 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f871c144b70 msgr2=0x7f871c1439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:10.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.474+0000 7f871bfff700 1 --2- 172.21.15.1:0/1739263325 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f871c144b70 0x7f871c1439d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.474+0000 7f871bfff700 1 -- 172.21.15.1:0/1739263325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8714009a70 con 0x7f871c0cd110 2024-03-18T18:51:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.474+0000 7f871bfff700 1 --2- 172.21.15.1:0/1739263325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f871c0cd110 0x7f871c0ac8b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f870800c4e0 tx=0x7f870800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.474+0000 7f87197fa700 1 -- 172.21.15.1:0/1739263325 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87080052d0 con 0x7f871c0cd110 2024-03-18T18:51:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.474+0000 7f87197fa700 1 -- 172.21.15.1:0/1739263325 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8708005430 con 0x7f871c0cd110 2024-03-18T18:51:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.474+0000 7f8722b19700 1 -- 172.21.15.1:0/1739263325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f871c142c80 con 0x7f871c0cd110 2024-03-18T18:51:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.474+0000 7f87197fa700 1 -- 172.21.15.1:0/1739263325 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87080038c0 con 0x7f871c0cd110 2024-03-18T18:51:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.474+0000 7f8722b19700 1 -- 172.21.15.1:0/1739263325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f871c142220 con 0x7f871c0cd110 2024-03-18T18:51:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.475+0000 7f8722b19700 1 -- 172.21.15.1:0/1739263325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f871c0cda90 con 0x7f871c0cd110 2024-03-18T18:51:10.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.475+0000 7f87197fa700 1 -- 172.21.15.1:0/1739263325 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8708003a20 con 0x7f871c0cd110 2024-03-18T18:51:10.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.476+0000 7f87197fa700 1 --2- 172.21.15.1:0/1739263325 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f870006ac90 0x7f870006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:10.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.476+0000 7f87197fa700 1 -- 172.21.15.1:0/1739263325 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f87080885e0 con 0x7f871c0cd110 2024-03-18T18:51:10.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.476+0000 7f87208b5700 1 --2- 172.21.15.1:0/1739263325 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f870006ac90 0x7f870006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:10.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.476+0000 7f87208b5700 1 --2- 172.21.15.1:0/1739263325 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f870006ac90 0x7f870006d150 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8710000f60 tx=0x7f8710007d90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:10.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.481+0000 7f87197fa700 1 -- 172.21.15.1:0/1739263325 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f87080539d0 con 0x7f871c0cd110 2024-03-18T18:51:10.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.686+0000 7f8722b19700 1 -- 172.21.15.1:0/1739263325 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f871c0b6080 con 0x7f871c0cd110 2024-03-18T18:51:10.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.687+0000 7f87197fa700 1 -- 172.21.15.1:0/1739263325 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v36) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8708017060 con 0x7f871c0cd110 2024-03-18T18:51:10.683 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:51:10.683 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T18:51:10.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:10.684 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-18T18:51:10.684 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-18T18:51:10.684 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:10.684 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T18:51:10.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.689+0000 7f8722b19700 1 -- 172.21.15.1:0/1739263325 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f870006ac90 msgr2=0x7f870006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:10.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.689+0000 7f8722b19700 1 --2- 172.21.15.1:0/1739263325 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f870006ac90 0x7f870006d150 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8710000f60 tx=0x7f8710007d90 comp rx=0 tx=0).stop 2024-03-18T18:51:10.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.689+0000 7f8722b19700 1 -- 172.21.15.1:0/1739263325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f871c0cd110 msgr2=0x7f871c0ac8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:10.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.689+0000 7f8722b19700 1 --2- 172.21.15.1:0/1739263325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f871c0cd110 0x7f871c0ac8b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f870800c4e0 tx=0x7f870800c8a0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.689+0000 7f8722b19700 1 -- 172.21.15.1:0/1739263325 shutdown_connections 2024-03-18T18:51:10.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.689+0000 7f8722b19700 1 --2- 172.21.15.1:0/1739263325 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f870006ac90 0x7f870006d150 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.689+0000 7f8722b19700 1 --2- 172.21.15.1:0/1739263325 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f871c0af610 0x7f871c137860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.689+0000 7f8722b19700 1 --2- 172.21.15.1:0/1739263325 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f871c144b70 0x7f871c1439d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.689+0000 7f8722b19700 1 --2- 172.21.15.1:0/1739263325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f871c0cd110 0x7f871c0ac8b0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.689+0000 7f8722b19700 1 -- 172.21.15.1:0/1739263325 >> 172.21.15.1:0/1739263325 conn(0x7f871c0b1d20 msgr2=0x7f871c150650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:10.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.689+0000 7f8722b19700 1 -- 172.21.15.1:0/1739263325 shutdown_connections 2024-03-18T18:51:10.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.689+0000 7f8722b19700 1 -- 172.21.15.1:0/1739263325 wait complete. 2024-03-18T18:51:10.697 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T18:51:10.698 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T18:51:10.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.830+0000 7f545bde3700 1 Processor -- start 2024-03-18T18:51:10.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.830+0000 7f545bde3700 1 -- start start 2024-03-18T18:51:10.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.831+0000 7f545bde3700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5454154aa0 0x7f54540b3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:10.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.831+0000 7f545bde3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f545415b4b0 0x7f5454070e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:10.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.831+0000 7f545bde3700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54540b2890 con 0x7f54541546c0 2024-03-18T18:51:10.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.831+0000 7f545bde3700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54540b3510 con 0x7f545415b4b0 2024-03-18T18:51:10.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.831+0000 7f545bde3700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54540b2b70 con 0x7f5454154aa0 2024-03-18T18:51:10.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.831+0000 7f545a380700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f545415b4b0 0x7f5454070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:10.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.831+0000 7f545a380700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f545415b4b0 0x7f5454070e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53782/0 (socket says 172.21.15.1:53782) 2024-03-18T18:51:10.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.831+0000 7f545a380700 1 -- 172.21.15.1:0/1414155988 learned_addr learned my addr 172.21.15.1:0/1414155988 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:10.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.832+0000 7f545937e700 1 --2- 172.21.15.1:0/1414155988 >> v2:172.21.15.165:3300/0 conn(0x7f5454154aa0 0x7f54540b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:10.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.832+0000 7f545a380700 1 -- 172.21.15.1:0/1414155988 >> v2:172.21.15.165:3300/0 conn(0x7f5454154aa0 msgr2=0x7f54540b3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:10.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.832+0000 7f545a380700 1 --2- 172.21.15.1:0/1414155988 >> v2:172.21.15.165:3300/0 conn(0x7f5454154aa0 0x7f54540b3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.832+0000 7f545a380700 1 -- 172.21.15.1:0/1414155988 >> v1:172.21.15.165:6789/0 conn(0x7f54541546c0 legacy=0x7f54540aec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:10.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.832+0000 7f545a380700 1 -- 172.21.15.1:0/1414155988 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54540b2e50 con 0x7f545415b4b0 2024-03-18T18:51:10.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.832+0000 7f545a380700 1 --2- 172.21.15.1:0/1414155988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f545415b4b0 0x7f5454070e90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f5454070e20 tx=0x7f544c00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0affbb100ae0cc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:10.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.832+0000 7f5458b7d700 1 -- 172.21.15.1:0/1414155988 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c010070 con 0x7f545415b4b0 2024-03-18T18:51:10.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.832+0000 7f5458b7d700 1 -- 172.21.15.1:0/1414155988 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544c00dd50 con 0x7f545415b4b0 2024-03-18T18:51:10.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.832+0000 7f5458b7d700 1 -- 172.21.15.1:0/1414155988 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c00b310 con 0x7f545415b4b0 2024-03-18T18:51:10.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.833+0000 7f545bde3700 1 -- 172.21.15.1:0/1414155988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f545415b4b0 msgr2=0x7f5454070e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:10.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.833+0000 7f545bde3700 1 --2- 172.21.15.1:0/1414155988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f545415b4b0 0x7f5454070e90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f5454070e20 tx=0x7f544c00d430 comp rx=0 tx=0).stop 2024-03-18T18:51:10.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.833+0000 7f545bde3700 1 -- 172.21.15.1:0/1414155988 shutdown_connections 2024-03-18T18:51:10.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.833+0000 7f545bde3700 1 --2- 172.21.15.1:0/1414155988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f545415b4b0 0x7f5454070e90 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.833+0000 7f545bde3700 1 --2- 172.21.15.1:0/1414155988 >> v2:172.21.15.165:3300/0 conn(0x7f5454154aa0 0x7f54540b3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.833+0000 7f545bde3700 1 -- 172.21.15.1:0/1414155988 >> 172.21.15.1:0/1414155988 conn(0x7f54540b1460 msgr2=0x7f54540b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:10.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.833+0000 7f545bde3700 1 -- 172.21.15.1:0/1414155988 shutdown_connections 2024-03-18T18:51:10.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.833+0000 7f545bde3700 1 -- 172.21.15.1:0/1414155988 wait complete. 2024-03-18T18:51:10.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.834+0000 7f545bde3700 1 Processor -- start 2024-03-18T18:51:10.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.834+0000 7f545bde3700 1 -- start start 2024-03-18T18:51:10.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f545bde3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54541546c0 0x7f54541472c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:10.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f545bde3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5454154aa0 0x7f545414e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:10.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f545bde3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f545415b4b0 0x7f54541466c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:10.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f545bde3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5454146c00 con 0x7f545415b4b0 2024-03-18T18:51:10.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f545bde3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5454149030 con 0x7f5454154aa0 2024-03-18T18:51:10.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f545bde3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54540d04a0 con 0x7f54541546c0 2024-03-18T18:51:10.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f545a380700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f545415b4b0 0x7f54541466c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:10.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f545a380700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f545415b4b0 0x7f54541466c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53798/0 (socket says 172.21.15.1:53798) 2024-03-18T18:51:10.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f545a380700 1 -- 172.21.15.1:0/839069803 learned_addr learned my addr 172.21.15.1:0/839069803 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:10.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f545937e700 1 --2- 172.21.15.1:0/839069803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5454154aa0 0x7f545414e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:10.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f5459b7f700 1 --2- 172.21.15.1:0/839069803 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54541546c0 0x7f54541472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:10.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f545a380700 1 -- 172.21.15.1:0/839069803 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54541546c0 msgr2=0x7f54541472c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:10.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f545a380700 1 --2- 172.21.15.1:0/839069803 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54541546c0 0x7f54541472c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f545a380700 1 -- 172.21.15.1:0/839069803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5454154aa0 msgr2=0x7f545414e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:10.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f545a380700 1 --2- 172.21.15.1:0/839069803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5454154aa0 0x7f545414e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:10.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f545a380700 1 -- 172.21.15.1:0/839069803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f544c00bd40 con 0x7f545415b4b0 2024-03-18T18:51:10.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f5459b7f700 1 --2- 172.21.15.1:0/839069803 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54541546c0 0x7f54541472c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:10.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f545937e700 1 --2- 172.21.15.1:0/839069803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5454154aa0 0x7f545414e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:10.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f545a380700 1 --2- 172.21.15.1:0/839069803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f545415b4b0 0x7f54541466c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5454067530 tx=0x7f544c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:10.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f544a7fc700 1 -- 172.21.15.1:0/839069803 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c009a10 con 0x7f545415b4b0 2024-03-18T18:51:10.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f544a7fc700 1 -- 172.21.15.1:0/839069803 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544c009b90 con 0x7f545415b4b0 2024-03-18T18:51:10.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f545bde3700 1 -- 172.21.15.1:0/839069803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5454149280 con 0x7f545415b4b0 2024-03-18T18:51:10.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.835+0000 7f544a7fc700 1 -- 172.21.15.1:0/839069803 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c004030 con 0x7f545415b4b0 2024-03-18T18:51:10.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.836+0000 7f545bde3700 1 -- 172.21.15.1:0/839069803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5454149840 con 0x7f545415b4b0 2024-03-18T18:51:10.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.836+0000 7f545bde3700 1 -- 172.21.15.1:0/839069803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54540676a0 con 0x7f545415b4b0 2024-03-18T18:51:10.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.837+0000 7f544a7fc700 1 -- 172.21.15.1:0/839069803 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f544c004190 con 0x7f545415b4b0 2024-03-18T18:51:10.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.837+0000 7f544a7fc700 1 --2- 172.21.15.1:0/839069803 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f543806ac90 0x7f543806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:10.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.837+0000 7f544a7fc700 1 -- 172.21.15.1:0/839069803 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f544c08cc90 con 0x7f545415b4b0 2024-03-18T18:51:10.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.837+0000 7f5459b7f700 1 --2- 172.21.15.1:0/839069803 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f543806ac90 0x7f543806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:10.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.838+0000 7f5459b7f700 1 --2- 172.21.15.1:0/839069803 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f543806ac90 0x7f543806d150 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5454151960 tx=0x7f5444009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:10.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:10.842+0000 7f544a7fc700 1 -- 172.21.15.1:0/839069803 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f544c057ff0 con 0x7f545415b4b0 2024-03-18T18:51:11.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.045+0000 7f545bde3700 1 -- 172.21.15.1:0/839069803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5454149a00 con 0x7f545415b4b0 2024-03-18T18:51:11.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.290+0000 7f544a7fc700 1 -- 172.21.15.1:0/839069803 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v37) v1 ==== 138+0+0 (secure 0 0 0) 0x7f544c05b610 con 0x7f545415b4b0 2024-03-18T18:51:11.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.294+0000 7f545bde3700 1 -- 172.21.15.1:0/839069803 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f543806ac90 msgr2=0x7f543806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:11.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.294+0000 7f545bde3700 1 --2- 172.21.15.1:0/839069803 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f543806ac90 0x7f543806d150 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5454151960 tx=0x7f5444009040 comp rx=0 tx=0).stop 2024-03-18T18:51:11.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.294+0000 7f545bde3700 1 -- 172.21.15.1:0/839069803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f545415b4b0 msgr2=0x7f54541466c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:11.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.294+0000 7f545bde3700 1 --2- 172.21.15.1:0/839069803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f545415b4b0 0x7f54541466c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5454067530 tx=0x7f544c004ad0 comp rx=0 tx=0).stop 2024-03-18T18:51:11.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.294+0000 7f545bde3700 1 -- 172.21.15.1:0/839069803 shutdown_connections 2024-03-18T18:51:11.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.294+0000 7f545bde3700 1 --2- 172.21.15.1:0/839069803 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f543806ac90 0x7f543806d150 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:11.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.294+0000 7f545bde3700 1 --2- 172.21.15.1:0/839069803 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54541546c0 0x7f54541472c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:11.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.294+0000 7f545bde3700 1 --2- 172.21.15.1:0/839069803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f545415b4b0 0x7f54541466c0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:11.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.294+0000 7f545bde3700 1 --2- 172.21.15.1:0/839069803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5454154aa0 0x7f545414e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:11.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.294+0000 7f545bde3700 1 -- 172.21.15.1:0/839069803 >> 172.21.15.1:0/839069803 conn(0x7f54540b1460 msgr2=0x7f545406ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:11.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.294+0000 7f545bde3700 1 -- 172.21.15.1:0/839069803 shutdown_connections 2024-03-18T18:51:11.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.294+0000 7f545bde3700 1 -- 172.21.15.1:0/839069803 wait complete. 2024-03-18T18:51:11.293 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-18T18:51:11.305 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T18:51:11.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.442+0000 7f15a4d6b700 1 Processor -- start 2024-03-18T18:51:11.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.442+0000 7f15a4d6b700 1 -- start start 2024-03-18T18:51:11.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.443+0000 7f15a4d6b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f15a006ae60 0x7f15a0168150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:11.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.443+0000 7f15a4d6b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a00cf600 con 0x7f15a015d5a0 2024-03-18T18:51:11.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.443+0000 7f15a4d6b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a00ce720 con 0x7f15a00cfa90 2024-03-18T18:51:11.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.443+0000 7f15a4d6b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15a00ce9f0 con 0x7f15a006ae60 2024-03-18T18:51:11.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.443+0000 7f159ed9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f15a006ae60 0x7f15a0168150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:11.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.443+0000 7f159e59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f15a015d5a0 0x7f15a00ce010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57426/0 (socket says 172.21.15.1:57426) 2024-03-18T18:51:11.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.443+0000 7f159e59c700 1 -- 172.21.15.1:0/2236201693 learned_addr learned my addr 172.21.15.1:0/2236201693 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:11.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.443+0000 7f159ed9d700 1 -- 172.21.15.1:0/2236201693 >> v1:172.21.15.165:6789/0 conn(0x7f15a00cfa90 legacy=0x7f15a006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:11.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.443+0000 7f159ed9d700 1 -- 172.21.15.1:0/2236201693 >> v1:172.21.15.1:6789/0 conn(0x7f15a015d5a0 legacy=0x7f15a00ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:11.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.443+0000 7f159ed9d700 1 -- 172.21.15.1:0/2236201693 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a00cecd0 con 0x7f15a006ae60 2024-03-18T18:51:11.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.443+0000 7f159d59a700 1 -- 172.21.15.1:0/2236201693 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869724413 0 0) 0x7f15a00cf600 con 0x7f15a015d5a0 2024-03-18T18:51:11.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.444+0000 7f159ed9d700 1 --2- 172.21.15.1:0/2236201693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15a006ae60 0x7f15a0168150 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1598009d90 tx=0x7f159800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e57a8dbb8bf7a839 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:11.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.444+0000 7f159d59a700 1 -- 172.21.15.1:0/2236201693 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159800e070 con 0x7f15a006ae60 2024-03-18T18:51:11.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.444+0000 7f159d59a700 1 -- 172.21.15.1:0/2236201693 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159800bb90 con 0x7f15a006ae60 2024-03-18T18:51:11.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.444+0000 7f159d59a700 1 -- 172.21.15.1:0/2236201693 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1598003a20 con 0x7f15a006ae60 2024-03-18T18:51:11.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.444+0000 7f15a4d6b700 1 -- 172.21.15.1:0/2236201693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15a006ae60 msgr2=0x7f15a0168150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:11.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.444+0000 7f15a4d6b700 1 --2- 172.21.15.1:0/2236201693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15a006ae60 0x7f15a0168150 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1598009d90 tx=0x7f159800b330 comp rx=0 tx=0).stop 2024-03-18T18:51:11.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.444+0000 7f15a4d6b700 1 -- 172.21.15.1:0/2236201693 shutdown_connections 2024-03-18T18:51:11.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.444+0000 7f15a4d6b700 1 --2- 172.21.15.1:0/2236201693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15a006ae60 0x7f15a0168150 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:11.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.444+0000 7f15a4d6b700 1 -- 172.21.15.1:0/2236201693 >> 172.21.15.1:0/2236201693 conn(0x7f15a00cc9b0 msgr2=0x7f15a00ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:11.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.444+0000 7f15a4d6b700 1 -- 172.21.15.1:0/2236201693 shutdown_connections 2024-03-18T18:51:11.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.444+0000 7f15a4d6b700 1 -- 172.21.15.1:0/2236201693 wait complete. 2024-03-18T18:51:11.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.446+0000 7f15a4d6b700 1 Processor -- start 2024-03-18T18:51:11.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.446+0000 7f15a4d6b700 1 -- start start 2024-03-18T18:51:11.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.446+0000 7f15a4d6b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15a006ae60 0x7f15a015b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:11.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.446+0000 7f15a4d6b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15a00cfa90 0x7f15a00b3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:11.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.446+0000 7f15a4d6b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15a015d5a0 0x7f15a00b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:11.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f15a4d6b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a00b4290 con 0x7f15a00cfa90 2024-03-18T18:51:11.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f15a4d6b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a01f3da0 con 0x7f15a006ae60 2024-03-18T18:51:11.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f15a4d6b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a015f510 con 0x7f15a015d5a0 2024-03-18T18:51:11.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f159dd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15a00cfa90 0x7f15a00b3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:11.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f159dd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15a00cfa90 0x7f15a00b3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53820/0 (socket says 172.21.15.1:53820) 2024-03-18T18:51:11.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f159dd9b700 1 -- 172.21.15.1:0/2526064743 learned_addr learned my addr 172.21.15.1:0/2526064743 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:11.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f159ed9d700 1 --2- 172.21.15.1:0/2526064743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15a015d5a0 0x7f15a00b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:11.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f159dd9b700 1 -- 172.21.15.1:0/2526064743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15a015d5a0 msgr2=0x7f15a00b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:11.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f159e59c700 1 --2- 172.21.15.1:0/2526064743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15a006ae60 0x7f15a015b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:11.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f159dd9b700 1 --2- 172.21.15.1:0/2526064743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15a015d5a0 0x7f15a00b3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:11.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f159dd9b700 1 -- 172.21.15.1:0/2526064743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15a006ae60 msgr2=0x7f15a015b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:11.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f159dd9b700 1 --2- 172.21.15.1:0/2526064743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15a006ae60 0x7f15a015b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:11.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f159dd9b700 1 -- 172.21.15.1:0/2526064743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1598009a70 con 0x7f15a00cfa90 2024-03-18T18:51:11.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f159ed9d700 1 --2- 172.21.15.1:0/2526064743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15a015d5a0 0x7f15a00b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:11.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f159e59c700 1 --2- 172.21.15.1:0/2526064743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15a006ae60 0x7f15a015b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:11.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f159dd9b700 1 --2- 172.21.15.1:0/2526064743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15a00cfa90 0x7f15a00b3720 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f159000c4d0 tx=0x7f159000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:11.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f158effd700 1 -- 172.21.15.1:0/2526064743 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1590011790 con 0x7f15a00cfa90 2024-03-18T18:51:11.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f158effd700 1 -- 172.21.15.1:0/2526064743 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15900118f0 con 0x7f15a00cfa90 2024-03-18T18:51:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f15a4d6b700 1 -- 172.21.15.1:0/2526064743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a01f2fe0 con 0x7f15a00cfa90 2024-03-18T18:51:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.447+0000 7f158effd700 1 -- 172.21.15.1:0/2526064743 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1590011ca0 con 0x7f15a00cfa90 2024-03-18T18:51:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.448+0000 7f15a4d6b700 1 -- 172.21.15.1:0/2526064743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a01f35a0 con 0x7f15a00cfa90 2024-03-18T18:51:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.448+0000 7f15a4d6b700 1 -- 172.21.15.1:0/2526064743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a00676a0 con 0x7f15a00cfa90 2024-03-18T18:51:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.449+0000 7f158effd700 1 -- 172.21.15.1:0/2526064743 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f159001c460 con 0x7f15a00cfa90 2024-03-18T18:51:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.449+0000 7f158effd700 1 --2- 172.21.15.1:0/2526064743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f158006ac90 0x7f158006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.449+0000 7f158effd700 1 -- 172.21.15.1:0/2526064743 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f1590089160 con 0x7f15a00cfa90 2024-03-18T18:51:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.449+0000 7f159e59c700 1 --2- 172.21.15.1:0/2526064743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f158006ac90 0x7f158006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:11.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.450+0000 7f159e59c700 1 --2- 172.21.15.1:0/2526064743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f158006ac90 0x7f158006d150 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f15940016e0 tx=0x7f1594009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:11.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.454+0000 7f158effd700 1 -- 172.21.15.1:0/2526064743 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1590054340 con 0x7f15a00cfa90 2024-03-18T18:51:11.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:11.656+0000 7f15a4d6b700 1 -- 172.21.15.1:0/2526064743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f15a00ce190 con 0x7f15a00cfa90 2024-03-18T18:51:12.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.297+0000 7f158effd700 1 -- 172.21.15.1:0/2526064743 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v38) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1590057960 con 0x7f15a00cfa90 2024-03-18T18:51:12.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.305+0000 7f15a4d6b700 1 -- 172.21.15.1:0/2526064743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f158006ac90 msgr2=0x7f158006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:12.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.305+0000 7f15a4d6b700 1 --2- 172.21.15.1:0/2526064743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f158006ac90 0x7f158006d150 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f15940016e0 tx=0x7f1594009040 comp rx=0 tx=0).stop 2024-03-18T18:51:12.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.305+0000 7f15a4d6b700 1 -- 172.21.15.1:0/2526064743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15a00cfa90 msgr2=0x7f15a00b3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:12.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.305+0000 7f15a4d6b700 1 --2- 172.21.15.1:0/2526064743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15a00cfa90 0x7f15a00b3720 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f159000c4d0 tx=0x7f159000c890 comp rx=0 tx=0).stop 2024-03-18T18:51:12.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.306+0000 7f15a4d6b700 1 -- 172.21.15.1:0/2526064743 shutdown_connections 2024-03-18T18:51:12.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.306+0000 7f15a4d6b700 1 --2- 172.21.15.1:0/2526064743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f158006ac90 0x7f158006d150 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.306+0000 7f15a4d6b700 1 --2- 172.21.15.1:0/2526064743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15a015d5a0 0x7f15a00b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.306+0000 7f15a4d6b700 1 --2- 172.21.15.1:0/2526064743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15a006ae60 0x7f15a015b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.306+0000 7f15a4d6b700 1 --2- 172.21.15.1:0/2526064743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15a00cfa90 0x7f15a00b3720 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.306+0000 7f15a4d6b700 1 -- 172.21.15.1:0/2526064743 >> 172.21.15.1:0/2526064743 conn(0x7f15a00cc9b0 msgr2=0x7f15a0071520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:12.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.308+0000 7f15a4d6b700 1 -- 172.21.15.1:0/2526064743 shutdown_connections 2024-03-18T18:51:12.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.309+0000 7f15a4d6b700 1 -- 172.21.15.1:0/2526064743 wait complete. 2024-03-18T18:51:12.308 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-18T18:51:12.325 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T18:51:12.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.462+0000 7f007231a700 1 Processor -- start 2024-03-18T18:51:12.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.462+0000 7f007231a700 1 -- start start 2024-03-18T18:51:12.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.463+0000 7f007231a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f006c15d5a0 0x7f006c0ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:12.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.463+0000 7f007231a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f006c0ce550 0x7f006c06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:12.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.463+0000 7f007231a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f006c0cfa90 0x7f006c06ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:12.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.463+0000 7f007231a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f006c0cf600 con 0x7f006c0ce550 2024-03-18T18:51:12.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.463+0000 7f007231a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f006c0ce9f0 con 0x7f006c0cfa90 2024-03-18T18:51:12.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.463+0000 7f007231a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f006c0cecd0 con 0x7f006c15d5a0 2024-03-18T18:51:12.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.463+0000 7f006b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f006c0ce550 0x7f006c06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:12.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.463+0000 7f006b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f006c0ce550 0x7f006c06a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53836/0 (socket says 172.21.15.1:53836) 2024-03-18T18:51:12.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.463+0000 7f006b7fe700 1 -- 172.21.15.1:0/1925766945 learned_addr learned my addr 172.21.15.1:0/1925766945 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:12.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.463+0000 7f00708b7700 1 --2- 172.21.15.1:0/1925766945 >> v2:172.21.15.165:3300/0 conn(0x7f006c0cfa90 0x7f006c06ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:12.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.463+0000 7f006bfff700 1 --2- 172.21.15.1:0/1925766945 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f006c15d5a0 0x7f006c0ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:12.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.463+0000 7f006b7fe700 1 -- 172.21.15.1:0/1925766945 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f006c15d5a0 msgr2=0x7f006c0ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:12.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.463+0000 7f006b7fe700 1 --2- 172.21.15.1:0/1925766945 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f006c15d5a0 0x7f006c0ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.463+0000 7f006b7fe700 1 -- 172.21.15.1:0/1925766945 >> v2:172.21.15.165:3300/0 conn(0x7f006c0cfa90 msgr2=0x7f006c06ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:12.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.463+0000 7f006b7fe700 1 --2- 172.21.15.1:0/1925766945 >> v2:172.21.15.165:3300/0 conn(0x7f006c0cfa90 0x7f006c06ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.463+0000 7f006b7fe700 1 -- 172.21.15.1:0/1925766945 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c0cefb0 con 0x7f006c0ce550 2024-03-18T18:51:12.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.463+0000 7f00708b7700 1 --2- 172.21.15.1:0/1925766945 >> v2:172.21.15.165:3300/0 conn(0x7f006c0cfa90 0x7f006c06ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:12.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.463+0000 7f006b7fe700 1 --2- 172.21.15.1:0/1925766945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f006c0ce550 0x7f006c06a750 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f006c0cdec0 tx=0x7f005c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11692b155972c83a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:12.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.464+0000 7f006affd700 1 -- 172.21.15.1:0/1925766945 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c00b870 con 0x7f006c0ce550 2024-03-18T18:51:12.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.464+0000 7f006affd700 1 -- 172.21.15.1:0/1925766945 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005c00b9d0 con 0x7f006c0ce550 2024-03-18T18:51:12.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.464+0000 7f006affd700 1 -- 172.21.15.1:0/1925766945 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c00bd60 con 0x7f006c0ce550 2024-03-18T18:51:12.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.464+0000 7f007231a700 1 -- 172.21.15.1:0/1925766945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f006c0ce550 msgr2=0x7f006c06a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:12.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.464+0000 7f007231a700 1 --2- 172.21.15.1:0/1925766945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f006c0ce550 0x7f006c06a750 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f006c0cdec0 tx=0x7f005c009bd0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.464+0000 7f007231a700 1 -- 172.21.15.1:0/1925766945 shutdown_connections 2024-03-18T18:51:12.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.464+0000 7f007231a700 1 --2- 172.21.15.1:0/1925766945 >> v2:172.21.15.165:3300/0 conn(0x7f006c0cfa90 0x7f006c06ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.464+0000 7f007231a700 1 --2- 172.21.15.1:0/1925766945 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f006c15d5a0 0x7f006c0ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.464+0000 7f007231a700 1 --2- 172.21.15.1:0/1925766945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f006c0ce550 0x7f006c06a750 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.464+0000 7f007231a700 1 -- 172.21.15.1:0/1925766945 >> 172.21.15.1:0/1925766945 conn(0x7f006c0cc9b0 msgr2=0x7f006c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:12.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.464+0000 7f007231a700 1 -- 172.21.15.1:0/1925766945 shutdown_connections 2024-03-18T18:51:12.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.464+0000 7f007231a700 1 -- 172.21.15.1:0/1925766945 wait complete. 2024-03-18T18:51:12.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.466+0000 7f007231a700 1 Processor -- start 2024-03-18T18:51:12.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.466+0000 7f007231a700 1 -- start start 2024-03-18T18:51:12.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.467+0000 7f007231a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f006c0ce550 0x7f006c14af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:12.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.467+0000 7f007231a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f006c0cfa90 0x7f006c0b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:12.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.467+0000 7f007231a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f006c15d5a0 0x7f006c14b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:12.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.467+0000 7f007231a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c15dec0 con 0x7f006c0ce550 2024-03-18T18:51:12.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.467+0000 7f007231a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c15e1a0 con 0x7f006c15d5a0 2024-03-18T18:51:12.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.467+0000 7f007231a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f006c0b48a0 con 0x7f006c0cfa90 2024-03-18T18:51:12.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.467+0000 7f006bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f006c0ce550 0x7f006c14af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:12.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.467+0000 7f006bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f006c0ce550 0x7f006c14af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53844/0 (socket says 172.21.15.1:53844) 2024-03-18T18:51:12.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.467+0000 7f006bfff700 1 -- 172.21.15.1:0/2943760539 learned_addr learned my addr 172.21.15.1:0/2943760539 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:12.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.467+0000 7f006b7fe700 1 --2- 172.21.15.1:0/2943760539 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f006c0cfa90 0x7f006c0b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:12.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.467+0000 7f00708b7700 1 --2- 172.21.15.1:0/2943760539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f006c15d5a0 0x7f006c14b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:12.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.467+0000 7f006bfff700 1 -- 172.21.15.1:0/2943760539 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f006c0cfa90 msgr2=0x7f006c0b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:12.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.467+0000 7f006bfff700 1 --2- 172.21.15.1:0/2943760539 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f006c0cfa90 0x7f006c0b50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.467+0000 7f006bfff700 1 -- 172.21.15.1:0/2943760539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f006c15d5a0 msgr2=0x7f006c14b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:12.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.467+0000 7f006bfff700 1 --2- 172.21.15.1:0/2943760539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f006c15d5a0 0x7f006c14b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.467+0000 7f006bfff700 1 -- 172.21.15.1:0/2943760539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005c009550 con 0x7f006c0ce550 2024-03-18T18:51:12.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.467+0000 7f00708b7700 1 --2- 172.21.15.1:0/2943760539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f006c15d5a0 0x7f006c14b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:12.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.468+0000 7f006bfff700 1 --2- 172.21.15.1:0/2943760539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f006c0ce550 0x7f006c14af30 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f006000c7f0 tx=0x7f006000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:12.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.468+0000 7f0068ff9700 1 -- 172.21.15.1:0/2943760539 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0060011800 con 0x7f006c0ce550 2024-03-18T18:51:12.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.468+0000 7f0068ff9700 1 -- 172.21.15.1:0/2943760539 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0060011960 con 0x7f006c0ce550 2024-03-18T18:51:12.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.468+0000 7f0068ff9700 1 -- 172.21.15.1:0/2943760539 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0060011d10 con 0x7f006c0ce550 2024-03-18T18:51:12.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.468+0000 7f007231a700 1 -- 172.21.15.1:0/2943760539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f006c0b4b80 con 0x7f006c0ce550 2024-03-18T18:51:12.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.468+0000 7f007231a700 1 -- 172.21.15.1:0/2943760539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f006c15db00 con 0x7f006c0ce550 2024-03-18T18:51:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.469+0000 7f007231a700 1 -- 172.21.15.1:0/2943760539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f006c0676a0 con 0x7f006c0ce550 2024-03-18T18:51:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.469+0000 7f0068ff9700 1 -- 172.21.15.1:0/2943760539 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0060018360 con 0x7f006c0ce550 2024-03-18T18:51:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.470+0000 7f0068ff9700 1 --2- 172.21.15.1:0/2943760539 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f005806ac90 0x7f005806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.470+0000 7f006b7fe700 1 --2- 172.21.15.1:0/2943760539 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f005806ac90 0x7f005806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:12.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.470+0000 7f0068ff9700 1 -- 172.21.15.1:0/2943760539 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f0060089410 con 0x7f006c0ce550 2024-03-18T18:51:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.470+0000 7f006b7fe700 1 --2- 172.21.15.1:0/2943760539 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f005806ac90 0x7f005806d150 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f005c014040 tx=0x7f005c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:12.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.475+0000 7f0068ff9700 1 -- 172.21.15.1:0/2943760539 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0060054520 con 0x7f006c0ce550 2024-03-18T18:51:12.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.720+0000 7f007231a700 1 -- 172.21.15.1:0/2943760539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f006c0aff00 con 0x7f006c0ce550 2024-03-18T18:51:12.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.725+0000 7f0068ff9700 1 -- 172.21.15.1:0/2943760539 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f0060057b40 con 0x7f006c0ce550 2024-03-18T18:51:12.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.728+0000 7f007231a700 1 -- 172.21.15.1:0/2943760539 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f005806ac90 msgr2=0x7f005806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:12.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.728+0000 7f007231a700 1 --2- 172.21.15.1:0/2943760539 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f005806ac90 0x7f005806d150 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f005c014040 tx=0x7f005c008030 comp rx=0 tx=0).stop 2024-03-18T18:51:12.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.728+0000 7f007231a700 1 -- 172.21.15.1:0/2943760539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f006c0ce550 msgr2=0x7f006c14af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:12.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.728+0000 7f007231a700 1 --2- 172.21.15.1:0/2943760539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f006c0ce550 0x7f006c14af30 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f006000c7f0 tx=0x7f006000cbb0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.729+0000 7f007231a700 1 -- 172.21.15.1:0/2943760539 shutdown_connections 2024-03-18T18:51:12.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.729+0000 7f007231a700 1 --2- 172.21.15.1:0/2943760539 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f005806ac90 0x7f005806d150 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.729+0000 7f007231a700 1 --2- 172.21.15.1:0/2943760539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f006c15d5a0 0x7f006c14b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.729+0000 7f007231a700 1 --2- 172.21.15.1:0/2943760539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f006c0ce550 0x7f006c14af30 secure :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f006000c7f0 tx=0x7f006000cbb0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.729+0000 7f007231a700 1 --2- 172.21.15.1:0/2943760539 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f006c0cfa90 0x7f006c0b50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.729+0000 7f007231a700 1 -- 172.21.15.1:0/2943760539 >> 172.21.15.1:0/2943760539 conn(0x7f006c0cc9b0 msgr2=0x7f006c06f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:12.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.731+0000 7f007231a700 1 -- 172.21.15.1:0/2943760539 shutdown_connections 2024-03-18T18:51:12.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.731+0000 7f007231a700 1 -- 172.21.15.1:0/2943760539 wait complete. 2024-03-18T18:51:12.739 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 5 and data pool 6 2024-03-18T18:51:12.750 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:51:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.879+0000 7f912d26a700 1 Processor -- start 2024-03-18T18:51:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.879+0000 7f912d26a700 1 -- start start 2024-03-18T18:51:12.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.880+0000 7f912d26a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91280cf090 0x7f91280d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.880+0000 7f912d26a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9128160d20 0x7f91280d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.880+0000 7f912d26a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91280cddc0 con 0x7f9128160940 2024-03-18T18:51:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.880+0000 7f912d26a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91280cdff0 con 0x7f9128160d20 2024-03-18T18:51:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.880+0000 7f912d26a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91280ce2d0 con 0x7f91280cf090 2024-03-18T18:51:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.880+0000 7f91277fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9128160d20 0x7f91280d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.880+0000 7f91277fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9128160d20 0x7f91280d1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53854/0 (socket says 172.21.15.1:53854) 2024-03-18T18:51:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.880+0000 7f91277fe700 1 -- 172.21.15.1:0/2543022202 learned_addr learned my addr 172.21.15.1:0/2543022202 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.880+0000 7f91267fc700 1 --2- 172.21.15.1:0/2543022202 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91280cf090 0x7f91280d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.880+0000 7f91277fe700 1 -- 172.21.15.1:0/2543022202 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91280cf090 msgr2=0x7f91280d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.880+0000 7f91277fe700 1 --2- 172.21.15.1:0/2543022202 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91280cf090 0x7f91280d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.880+0000 7f91277fe700 1 -- 172.21.15.1:0/2543022202 >> v1:172.21.15.165:6789/0 conn(0x7f9128160940 legacy=0x7f91280d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.880+0000 7f91277fe700 1 -- 172.21.15.1:0/2543022202 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91280ce5b0 con 0x7f9128160d20 2024-03-18T18:51:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.880+0000 7f91277fe700 1 --2- 172.21.15.1:0/2543022202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9128160d20 0x7f91280d1cd0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9120009e50 tx=0x7f912000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85ed12d0072a350b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.880+0000 7f9125ffb700 1 -- 172.21.15.1:0/2543022202 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912000e070 con 0x7f9128160d20 2024-03-18T18:51:12.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.880+0000 7f9125ffb700 1 -- 172.21.15.1:0/2543022202 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912000b9c0 con 0x7f9128160d20 2024-03-18T18:51:12.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.880+0000 7f9125ffb700 1 -- 172.21.15.1:0/2543022202 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912000bd70 con 0x7f9128160d20 2024-03-18T18:51:12.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.881+0000 7f912d26a700 1 -- 172.21.15.1:0/2543022202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9128160d20 msgr2=0x7f91280d1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:12.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.881+0000 7f912d26a700 1 --2- 172.21.15.1:0/2543022202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9128160d20 0x7f91280d1cd0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9120009e50 tx=0x7f912000b330 comp rx=0 tx=0).stop 2024-03-18T18:51:12.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.881+0000 7f912d26a700 1 -- 172.21.15.1:0/2543022202 shutdown_connections 2024-03-18T18:51:12.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.881+0000 7f912d26a700 1 --2- 172.21.15.1:0/2543022202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9128160d20 0x7f91280d1cd0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.881+0000 7f912d26a700 1 --2- 172.21.15.1:0/2543022202 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91280cf090 0x7f91280d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.881+0000 7f912d26a700 1 -- 172.21.15.1:0/2543022202 >> 172.21.15.1:0/2543022202 conn(0x7f912809c520 msgr2=0x7f91280d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:12.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.881+0000 7f912d26a700 1 -- 172.21.15.1:0/2543022202 shutdown_connections 2024-03-18T18:51:12.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.881+0000 7f912d26a700 1 -- 172.21.15.1:0/2543022202 wait complete. 2024-03-18T18:51:12.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.882+0000 7f912d26a700 1 Processor -- start 2024-03-18T18:51:12.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.882+0000 7f912d26a700 1 -- start start 2024-03-18T18:51:12.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.883+0000 7f912d26a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91280cf090 0x7f91281556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:12.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.883+0000 7f912d26a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9128160940 0x7f91280c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:12.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.883+0000 7f912d26a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9128160d20 0x7f91280b6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:12.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.883+0000 7f912d26a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91280cafa0 con 0x7f91280cf090 2024-03-18T18:51:12.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.883+0000 7f912d26a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91280ca320 con 0x7f9128160940 2024-03-18T18:51:12.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.883+0000 7f912d26a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91280ca600 con 0x7f9128160d20 2024-03-18T18:51:12.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.883+0000 7f9126ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91280cf090 0x7f91281556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:12.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.883+0000 7f9126ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91280cf090 0x7f91281556e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53870/0 (socket says 172.21.15.1:53870) 2024-03-18T18:51:12.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.883+0000 7f9126ffd700 1 -- 172.21.15.1:0/52649657 learned_addr learned my addr 172.21.15.1:0/52649657 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:12.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.883+0000 7f91267fc700 1 --2- 172.21.15.1:0/52649657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9128160940 0x7f91280c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:12.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.883+0000 7f91277fe700 1 --2- 172.21.15.1:0/52649657 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9128160d20 0x7f91280b6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:12.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.884+0000 7f9126ffd700 1 -- 172.21.15.1:0/52649657 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9128160d20 msgr2=0x7f91280b6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:12.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.884+0000 7f9126ffd700 1 --2- 172.21.15.1:0/52649657 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9128160d20 0x7f91280b6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.884+0000 7f9126ffd700 1 -- 172.21.15.1:0/52649657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9128160940 msgr2=0x7f91280c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.884+0000 7f9126ffd700 1 --2- 172.21.15.1:0/52649657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9128160940 0x7f91280c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.884+0000 7f9126ffd700 1 -- 172.21.15.1:0/52649657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9120009b30 con 0x7f91280cf090 2024-03-18T18:51:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.884+0000 7f9126ffd700 1 --2- 172.21.15.1:0/52649657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91280cf090 0x7f91281556e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f911800c4e0 tx=0x7f911800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.884+0000 7f910f7fe700 1 -- 172.21.15.1:0/52649657 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91180052d0 con 0x7f91280cf090 2024-03-18T18:51:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.884+0000 7f910f7fe700 1 -- 172.21.15.1:0/52649657 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9118005430 con 0x7f91280cf090 2024-03-18T18:51:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.884+0000 7f912d26a700 1 -- 172.21.15.1:0/52649657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91280ca8e0 con 0x7f91280cf090 2024-03-18T18:51:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.884+0000 7f910f7fe700 1 -- 172.21.15.1:0/52649657 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91180038c0 con 0x7f91280cf090 2024-03-18T18:51:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.884+0000 7f912d26a700 1 -- 172.21.15.1:0/52649657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9128155c20 con 0x7f91280cf090 2024-03-18T18:51:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.885+0000 7f912d26a700 1 -- 172.21.15.1:0/52649657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9128073350 con 0x7f91280cf090 2024-03-18T18:51:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.885+0000 7f910f7fe700 1 -- 172.21.15.1:0/52649657 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9118003a20 con 0x7f91280cf090 2024-03-18T18:51:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.886+0000 7f910f7fe700 1 --2- 172.21.15.1:0/52649657 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f911406ac90 0x7f911406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:12.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.886+0000 7f910f7fe700 1 -- 172.21.15.1:0/52649657 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f91180887b0 con 0x7f91280cf090 2024-03-18T18:51:12.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.886+0000 7f91267fc700 1 --2- 172.21.15.1:0/52649657 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f911406ac90 0x7f911406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:12.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.886+0000 7f91267fc700 1 --2- 172.21.15.1:0/52649657 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f911406ac90 0x7f911406d150 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f91280c8810 tx=0x7f911c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:12.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:12.891+0000 7f910f7fe700 1 -- 172.21.15.1:0/52649657 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9118053850 con 0x7f91280cf090 2024-03-18T18:51:13.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.096+0000 7f912d26a700 1 -- 172.21.15.1:0/52649657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91280d0980 con 0x7f91280cf090 2024-03-18T18:51:13.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.097+0000 7f910f7fe700 1 -- 172.21.15.1:0/52649657 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16271 (secure 0 0 0) 0x7f9118056e70 con 0x7f91280cf090 2024-03-18T18:51:13.093 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:51:13.093 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":39,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:51:12.721499+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:51:11.076221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T18:51:11.687105+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:51:13.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.099+0000 7f912d26a700 1 -- 172.21.15.1:0/52649657 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f911406ac90 msgr2=0x7f911406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:13.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.099+0000 7f912d26a700 1 --2- 172.21.15.1:0/52649657 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f911406ac90 0x7f911406d150 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f91280c8810 tx=0x7f911c008040 comp rx=0 tx=0).stop 2024-03-18T18:51:13.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.099+0000 7f912d26a700 1 -- 172.21.15.1:0/52649657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91280cf090 msgr2=0x7f91281556e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:13.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.099+0000 7f912d26a700 1 --2- 172.21.15.1:0/52649657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91280cf090 0x7f91281556e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f911800c4e0 tx=0x7f911800c7f0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.099+0000 7f912d26a700 1 -- 172.21.15.1:0/52649657 shutdown_connections 2024-03-18T18:51:13.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.099+0000 7f912d26a700 1 --2- 172.21.15.1:0/52649657 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f911406ac90 0x7f911406d150 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.099+0000 7f912d26a700 1 --2- 172.21.15.1:0/52649657 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9128160d20 0x7f91280b6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.099+0000 7f912d26a700 1 --2- 172.21.15.1:0/52649657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91280cf090 0x7f91281556e0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.099+0000 7f912d26a700 1 --2- 172.21.15.1:0/52649657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9128160940 0x7f91280c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.099+0000 7f912d26a700 1 -- 172.21.15.1:0/52649657 >> 172.21.15.1:0/52649657 conn(0x7f912809c520 msgr2=0x7f91280d1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:13.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.099+0000 7f912d26a700 1 -- 172.21.15.1:0/52649657 shutdown_connections 2024-03-18T18:51:13.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.099+0000 7f912d26a700 1 -- 172.21.15.1:0/52649657 wait complete. 2024-03-18T18:51:13.109 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:51:13.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.237+0000 7f0358ead700 1 Processor -- start 2024-03-18T18:51:13.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.237+0000 7f0358ead700 1 -- start start 2024-03-18T18:51:13.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.238+0000 7f0358ead700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0354099a40 0x7f0354099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:13.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.238+0000 7f0358ead700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03540cc310 0x7f03540cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:13.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.238+0000 7f0358ead700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f035409a3e0 con 0x7f035415d680 2024-03-18T18:51:13.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.238+0000 7f0358ead700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03540d0c80 con 0x7f03540cc310 2024-03-18T18:51:13.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.238+0000 7f0358ead700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03540d09a0 con 0x7f0354099a40 2024-03-18T18:51:13.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.238+0000 7f0352d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03540cc310 0x7f03540cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:13.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.238+0000 7f0352d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03540cc310 0x7f03540cc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53882/0 (socket says 172.21.15.1:53882) 2024-03-18T18:51:13.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.238+0000 7f0352d9d700 1 -- 172.21.15.1:0/2184319910 learned_addr learned my addr 172.21.15.1:0/2184319910 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.238+0000 7f0351d9b700 1 --2- 172.21.15.1:0/2184319910 >> v2:172.21.15.165:3300/0 conn(0x7f0354099a40 0x7f0354099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.238+0000 7f0352d9d700 1 -- 172.21.15.1:0/2184319910 >> v2:172.21.15.165:3300/0 conn(0x7f0354099a40 msgr2=0x7f0354099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.238+0000 7f0352d9d700 1 --2- 172.21.15.1:0/2184319910 >> v2:172.21.15.165:3300/0 conn(0x7f0354099a40 0x7f0354099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.238+0000 7f0352d9d700 1 -- 172.21.15.1:0/2184319910 >> v1:172.21.15.165:6789/0 conn(0x7f035415d680 legacy=0x7f03540cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.238+0000 7f0352d9d700 1 -- 172.21.15.1:0/2184319910 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03540cb830 con 0x7f03540cc310 2024-03-18T18:51:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.239+0000 7f0352d9d700 1 --2- 172.21.15.1:0/2184319910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03540cc310 0x7f03540cc7a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0354071cb0 tx=0x7f034c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484386be7179bf4a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.239+0000 7f035159a700 1 -- 172.21.15.1:0/2184319910 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034c00e070 con 0x7f03540cc310 2024-03-18T18:51:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.239+0000 7f035159a700 1 -- 172.21.15.1:0/2184319910 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034c00b910 con 0x7f03540cc310 2024-03-18T18:51:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.239+0000 7f035159a700 1 -- 172.21.15.1:0/2184319910 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034c00bcc0 con 0x7f03540cc310 2024-03-18T18:51:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.239+0000 7f0358ead700 1 -- 172.21.15.1:0/2184319910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03540cc310 msgr2=0x7f03540cc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.239+0000 7f0358ead700 1 --2- 172.21.15.1:0/2184319910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03540cc310 0x7f03540cc7a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0354071cb0 tx=0x7f034c00b330 comp rx=0 tx=0).stop 2024-03-18T18:51:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.239+0000 7f0358ead700 1 -- 172.21.15.1:0/2184319910 shutdown_connections 2024-03-18T18:51:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.239+0000 7f0358ead700 1 --2- 172.21.15.1:0/2184319910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03540cc310 0x7f03540cc7a0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.239+0000 7f0358ead700 1 --2- 172.21.15.1:0/2184319910 >> v2:172.21.15.165:3300/0 conn(0x7f0354099a40 0x7f0354099ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.239+0000 7f0358ead700 1 -- 172.21.15.1:0/2184319910 >> 172.21.15.1:0/2184319910 conn(0x7f03540b2ba0 msgr2=0x7f03540b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:13.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.239+0000 7f0358ead700 1 -- 172.21.15.1:0/2184319910 shutdown_connections 2024-03-18T18:51:13.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.239+0000 7f0358ead700 1 -- 172.21.15.1:0/2184319910 wait complete. 2024-03-18T18:51:13.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.241+0000 7f0358ead700 1 Processor -- start 2024-03-18T18:51:13.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.241+0000 7f0358ead700 1 -- start start 2024-03-18T18:51:13.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.242+0000 7f0358ead700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0354099a40 0x7f035415c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:13.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.242+0000 7f0358ead700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03540cc310 0x7f035415faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:13.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.242+0000 7f0358ead700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f035415d680 0x7f03540ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:13.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.242+0000 7f0358ead700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035415edc0 con 0x7f0354099a40 2024-03-18T18:51:13.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.242+0000 7f0358ead700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035415f660 con 0x7f035415d680 2024-03-18T18:51:13.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.242+0000 7f0358ead700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f035415cc40 con 0x7f03540cc310 2024-03-18T18:51:13.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.242+0000 7f035259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0354099a40 0x7f035415c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:13.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.242+0000 7f035259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0354099a40 0x7f035415c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53898/0 (socket says 172.21.15.1:53898) 2024-03-18T18:51:13.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.242+0000 7f035259c700 1 -- 172.21.15.1:0/227117904 learned_addr learned my addr 172.21.15.1:0/227117904 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:13.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.242+0000 7f0351d9b700 1 --2- 172.21.15.1:0/227117904 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03540cc310 0x7f035415faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:13.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.242+0000 7f0352d9d700 1 --2- 172.21.15.1:0/227117904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f035415d680 0x7f03540ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:13.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.242+0000 7f035259c700 1 -- 172.21.15.1:0/227117904 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03540cc310 msgr2=0x7f035415faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:13.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.242+0000 7f035259c700 1 --2- 172.21.15.1:0/227117904 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03540cc310 0x7f035415faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.242+0000 7f035259c700 1 -- 172.21.15.1:0/227117904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f035415d680 msgr2=0x7f03540ca3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:13.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.242+0000 7f035259c700 1 --2- 172.21.15.1:0/227117904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f035415d680 0x7f03540ca3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.242+0000 7f035259c700 1 -- 172.21.15.1:0/227117904 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034c009b30 con 0x7f0354099a40 2024-03-18T18:51:13.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.243+0000 7f035259c700 1 --2- 172.21.15.1:0/227117904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0354099a40 0x7f035415c700 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f034800c480 tx=0x7f034800c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:13.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.243+0000 7f0342ffd700 1 -- 172.21.15.1:0/227117904 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03480052d0 con 0x7f0354099a40 2024-03-18T18:51:13.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.243+0000 7f0342ffd700 1 -- 172.21.15.1:0/227117904 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0348005430 con 0x7f0354099a40 2024-03-18T18:51:13.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.243+0000 7f0358ead700 1 -- 172.21.15.1:0/227117904 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0354160030 con 0x7f0354099a40 2024-03-18T18:51:13.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.243+0000 7f0342ffd700 1 -- 172.21.15.1:0/227117904 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03480038c0 con 0x7f0354099a40 2024-03-18T18:51:13.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.243+0000 7f0358ead700 1 -- 172.21.15.1:0/227117904 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f035415f010 con 0x7f0354099a40 2024-03-18T18:51:13.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.244+0000 7f0358ead700 1 -- 172.21.15.1:0/227117904 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f035415f940 con 0x7f0354099a40 2024-03-18T18:51:13.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.245+0000 7f0342ffd700 1 -- 172.21.15.1:0/227117904 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0348003a20 con 0x7f0354099a40 2024-03-18T18:51:13.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.245+0000 7f0342ffd700 1 --2- 172.21.15.1:0/227117904 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f033406ac90 0x7f033406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:13.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.245+0000 7f0342ffd700 1 -- 172.21.15.1:0/227117904 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0348088820 con 0x7f0354099a40 2024-03-18T18:51:13.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.245+0000 7f0351d9b700 1 --2- 172.21.15.1:0/227117904 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f033406ac90 0x7f033406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:13.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.245+0000 7f0351d9b700 1 --2- 172.21.15.1:0/227117904 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f033406ac90 0x7f033406d150 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0354160d70 tx=0x7f0344009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:13.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.249+0000 7f0342ffd700 1 -- 172.21.15.1:0/227117904 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0348053a30 con 0x7f0354099a40 2024-03-18T18:51:13.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.454+0000 7f0358ead700 1 -- 172.21.15.1:0/227117904 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03540676a0 con 0x7f0354099a40 2024-03-18T18:51:13.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.455+0000 7f0342ffd700 1 -- 172.21.15.1:0/227117904 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16271 (secure 0 0 0) 0x7f0348057050 con 0x7f0354099a40 2024-03-18T18:51:13.452 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:51:13.452 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":39,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:51:12.721499+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:51:11.076221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T18:51:11.687105+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:51:13.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.457+0000 7f0358ead700 1 -- 172.21.15.1:0/227117904 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f033406ac90 msgr2=0x7f033406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:13.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.457+0000 7f0358ead700 1 --2- 172.21.15.1:0/227117904 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f033406ac90 0x7f033406d150 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0354160d70 tx=0x7f0344009210 comp rx=0 tx=0).stop 2024-03-18T18:51:13.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.457+0000 7f0358ead700 1 -- 172.21.15.1:0/227117904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0354099a40 msgr2=0x7f035415c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:13.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.457+0000 7f0358ead700 1 --2- 172.21.15.1:0/227117904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0354099a40 0x7f035415c700 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f034800c480 tx=0x7f034800c790 comp rx=0 tx=0).stop 2024-03-18T18:51:13.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.457+0000 7f0358ead700 1 -- 172.21.15.1:0/227117904 shutdown_connections 2024-03-18T18:51:13.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.457+0000 7f0358ead700 1 --2- 172.21.15.1:0/227117904 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f033406ac90 0x7f033406d150 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.457+0000 7f0358ead700 1 --2- 172.21.15.1:0/227117904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f035415d680 0x7f03540ca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.457+0000 7f0358ead700 1 --2- 172.21.15.1:0/227117904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0354099a40 0x7f035415c700 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.457+0000 7f0358ead700 1 --2- 172.21.15.1:0/227117904 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03540cc310 0x7f035415faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.457+0000 7f0358ead700 1 -- 172.21.15.1:0/227117904 >> 172.21.15.1:0/227117904 conn(0x7f03540b2ba0 msgr2=0x7f03540b2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:13.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.458+0000 7f0358ead700 1 -- 172.21.15.1:0/227117904 shutdown_connections 2024-03-18T18:51:13.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.458+0000 7f0358ead700 1 -- 172.21.15.1:0/227117904 wait complete. 2024-03-18T18:51:13.467 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T18:51:13.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.598+0000 7f3af9105700 1 Processor -- start 2024-03-18T18:51:13.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.598+0000 7f3af9105700 1 -- start start 2024-03-18T18:51:13.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.599+0000 7f3af9105700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af4071d30 0x7f3af40add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:13.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.599+0000 7f3af9105700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3af40cb9e0 0x7f3af40cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:13.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.599+0000 7f3af9105700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af40ce8f0 con 0x7f3af415a1d0 2024-03-18T18:51:13.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.599+0000 7f3af9105700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af40cdb00 con 0x7f3af40cb9e0 2024-03-18T18:51:13.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.599+0000 7f3af9105700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af415a5b0 con 0x7f3af4071d30 2024-03-18T18:51:13.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.599+0000 7f3af2d9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3af415a1d0 0x7f3af4071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57428/0 (socket says 172.21.15.1:57428) 2024-03-18T18:51:13.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.599+0000 7f3af2d9d700 1 -- 172.21.15.1:0/4258209458 learned_addr learned my addr 172.21.15.1:0/4258209458 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:13.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.599+0000 7f3af259c700 1 --2- 172.21.15.1:0/4258209458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af4071d30 0x7f3af40add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:13.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.599+0000 7f3af359e700 1 --2- 172.21.15.1:0/4258209458 >> v2:172.21.15.165:3300/0 conn(0x7f3af40cb9e0 0x7f3af40cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:13.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.599+0000 7f3af1d9b700 1 -- 172.21.15.1:0/4258209458 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154495550 0 0) 0x7f3af40ce8f0 con 0x7f3af415a1d0 2024-03-18T18:51:13.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.599+0000 7f3af1d9b700 1 -- 172.21.15.1:0/4258209458 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3adc0035b0 con 0x7f3af415a1d0 2024-03-18T18:51:13.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af1d9b700 1 -- 172.21.15.1:0/4258209458 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3799051818 0 0) 0x7f3adc0035b0 con 0x7f3af415a1d0 2024-03-18T18:51:13.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af1d9b700 1 -- 172.21.15.1:0/4258209458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af4071d30 msgr2=0x7f3af40add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:13.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af1d9b700 1 --2- 172.21.15.1:0/4258209458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af4071d30 0x7f3af40add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af1d9b700 1 -- 172.21.15.1:0/4258209458 >> v2:172.21.15.165:3300/0 conn(0x7f3af40cb9e0 msgr2=0x7f3af40cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:13.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af1d9b700 1 --2- 172.21.15.1:0/4258209458 >> v2:172.21.15.165:3300/0 conn(0x7f3af40cb9e0 0x7f3af40cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af1d9b700 1 -- 172.21.15.1:0/4258209458 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af40ae260 con 0x7f3af415a1d0 2024-03-18T18:51:13.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af359e700 1 --2- 172.21.15.1:0/4258209458 >> v2:172.21.15.165:3300/0 conn(0x7f3af40cb9e0 0x7f3af40cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:51:13.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af1d9b700 1 -- 172.21.15.1:0/4258209458 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f3ae80017f0 con 0x7f3af415a1d0 2024-03-18T18:51:13.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af1d9b700 1 -- 172.21.15.1:0/4258209458 >> v1:172.21.15.1:6789/0 conn(0x7f3af415a1d0 legacy=0x7f3af4071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:13.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af1d9b700 1 --2- 172.21.15.1:0/4258209458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3adc003a00 0x7f3adc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:13.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af1d9b700 1 --2- 172.21.15.1:0/4258209458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3adc004850 0x7f3adc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:13.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af1d9b700 1 --2- 172.21.15.1:0/4258209458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3adc005210 0x7f3adc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:13.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af1d9b700 1 -- 172.21.15.1:0/4258209458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af40cdb00 con 0x7f3adc003a00 2024-03-18T18:51:13.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af1d9b700 1 -- 172.21.15.1:0/4258209458 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc00bf50 con 0x7f3adc005210 2024-03-18T18:51:13.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af1d9b700 1 -- 172.21.15.1:0/4258209458 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc00c090 con 0x7f3adc004850 2024-03-18T18:51:13.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af259c700 1 --2- 172.21.15.1:0/4258209458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af4071d30 0x7f3af40add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:51:13.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af2d9d700 1 --2- 172.21.15.1:0/4258209458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3adc003a00 0x7f3adc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:13.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af359e700 1 --2- 172.21.15.1:0/4258209458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3adc005210 0x7f3adc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:13.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af259c700 1 --2- 172.21.15.1:0/4258209458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3adc004850 0x7f3adc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:13.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af2d9d700 1 -- 172.21.15.1:0/4258209458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3adc004850 msgr2=0x7f3adc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:13.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af2d9d700 1 --2- 172.21.15.1:0/4258209458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3adc004850 0x7f3adc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af2d9d700 1 -- 172.21.15.1:0/4258209458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3adc005210 msgr2=0x7f3adc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:13.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af2d9d700 1 --2- 172.21.15.1:0/4258209458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3adc005210 0x7f3adc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af2d9d700 1 -- 172.21.15.1:0/4258209458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3adc0035b0 con 0x7f3adc003a00 2024-03-18T18:51:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af359e700 1 --2- 172.21.15.1:0/4258209458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3adc005210 0x7f3adc00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af259c700 1 --2- 172.21.15.1:0/4258209458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3adc004850 0x7f3adc004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af2d9d700 1 --2- 172.21.15.1:0/4258209458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3adc003a00 0x7f3adc003e20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3ae800c350 tx=0x7f3adc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e0f36aa79617a75 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af1d9b700 1 -- 172.21.15.1:0/4258209458 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae800ce50 con 0x7f3adc003a00 2024-03-18T18:51:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af1d9b700 1 -- 172.21.15.1:0/4258209458 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae8007bf0 con 0x7f3adc003a00 2024-03-18T18:51:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.600+0000 7f3af1d9b700 1 -- 172.21.15.1:0/4258209458 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae8007310 con 0x7f3adc003a00 2024-03-18T18:51:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.601+0000 7f3af9105700 1 -- 172.21.15.1:0/4258209458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3adc003a00 msgr2=0x7f3adc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.601+0000 7f3af9105700 1 --2- 172.21.15.1:0/4258209458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3adc003a00 0x7f3adc003e20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3ae800c350 tx=0x7f3adc005880 comp rx=0 tx=0).stop 2024-03-18T18:51:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.601+0000 7f3af359e700 1 -- 172.21.15.1:0/4258209458 reap_dead start 2024-03-18T18:51:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.601+0000 7f3af9105700 1 -- 172.21.15.1:0/4258209458 shutdown_connections 2024-03-18T18:51:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.601+0000 7f3af9105700 1 -- 172.21.15.1:0/4258209458 >> 172.21.15.1:0/4258209458 conn(0x7f3af40cc9b0 msgr2=0x7f3af40ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.601+0000 7f3af9105700 1 -- 172.21.15.1:0/4258209458 shutdown_connections 2024-03-18T18:51:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.601+0000 7f3af9105700 1 -- 172.21.15.1:0/4258209458 wait complete. 2024-03-18T18:51:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.602+0000 7f3af9105700 1 Processor -- start 2024-03-18T18:51:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.603+0000 7f3af9105700 1 -- start start 2024-03-18T18:51:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.603+0000 7f3af9105700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3adc003a00 0x7f3af41f5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.603+0000 7f3af9105700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3af414ce50 0x7f3af414adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.603+0000 7f3af9105700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af414b300 0x7f3af4158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.603+0000 7f3af9105700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af414ddd0 con 0x7f3adc003a00 2024-03-18T18:51:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.603+0000 7f3af9105700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af414d810 con 0x7f3af414ce50 2024-03-18T18:51:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.603+0000 7f3af9105700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af414daf0 con 0x7f3af414b300 2024-03-18T18:51:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.603+0000 7f3af2d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3adc003a00 0x7f3af41f5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.603+0000 7f3af2d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3adc003a00 0x7f3af41f5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53924/0 (socket says 172.21.15.1:53924) 2024-03-18T18:51:13.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.603+0000 7f3af2d9d700 1 -- 172.21.15.1:0/3046772380 learned_addr learned my addr 172.21.15.1:0/3046772380 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:13.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.604+0000 7f3af359e700 1 --2- 172.21.15.1:0/3046772380 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af414b300 0x7f3af4158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:13.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.604+0000 7f3af259c700 1 --2- 172.21.15.1:0/3046772380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3af414ce50 0x7f3af414adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:13.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.604+0000 7f3af2d9d700 1 -- 172.21.15.1:0/3046772380 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af414b300 msgr2=0x7f3af4158460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:13.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.604+0000 7f3af2d9d700 1 --2- 172.21.15.1:0/3046772380 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af414b300 0x7f3af4158460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.604+0000 7f3af2d9d700 1 -- 172.21.15.1:0/3046772380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3af414ce50 msgr2=0x7f3af414adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:13.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.604+0000 7f3af2d9d700 1 --2- 172.21.15.1:0/3046772380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3af414ce50 0x7f3af414adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:13.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.604+0000 7f3af2d9d700 1 -- 172.21.15.1:0/3046772380 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af40c4be0 con 0x7f3adc003a00 2024-03-18T18:51:13.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.604+0000 7f3af259c700 1 --2- 172.21.15.1:0/3046772380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3af414ce50 0x7f3af414adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:13.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.604+0000 7f3af2d9d700 1 --2- 172.21.15.1:0/3046772380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3adc003a00 0x7f3af41f5920 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8015040 tx=0x7f3ae8001bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:13.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.604+0000 7f3adb7fe700 1 -- 172.21.15.1:0/3046772380 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae8001800 con 0x7f3adc003a00 2024-03-18T18:51:13.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.604+0000 7f3adb7fe700 1 -- 172.21.15.1:0/3046772380 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae80070d0 con 0x7f3adc003a00 2024-03-18T18:51:13.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.604+0000 7f3adb7fe700 1 -- 172.21.15.1:0/3046772380 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae8007480 con 0x7f3adc003a00 2024-03-18T18:51:13.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.604+0000 7f3af9105700 1 -- 172.21.15.1:0/3046772380 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af40c4ec0 con 0x7f3adc003a00 2024-03-18T18:51:13.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.604+0000 7f3af9105700 1 -- 172.21.15.1:0/3046772380 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af4150ad0 con 0x7f3adc003a00 2024-03-18T18:51:13.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.605+0000 7f3af9105700 1 -- 172.21.15.1:0/3046772380 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af40676a0 con 0x7f3adc003a00 2024-03-18T18:51:13.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.606+0000 7f3adb7fe700 1 -- 172.21.15.1:0/3046772380 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3ae80075e0 con 0x7f3adc003a00 2024-03-18T18:51:13.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.606+0000 7f3adb7fe700 1 --2- 172.21.15.1:0/3046772380 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ae006ac90 0x7f3ae006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:13.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.606+0000 7f3adb7fe700 1 -- 172.21.15.1:0/3046772380 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3ae808a960 con 0x7f3adc003a00 2024-03-18T18:51:13.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.606+0000 7f3af259c700 1 --2- 172.21.15.1:0/3046772380 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ae006ac90 0x7f3ae006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:13.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.606+0000 7f3af259c700 1 --2- 172.21.15.1:0/3046772380 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ae006ac90 0x7f3ae006d150 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4006f10 tx=0x7f3ae4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:13.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.611+0000 7f3adb7fe700 1 -- 172.21.15.1:0/3046772380 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3ae8055bb0 con 0x7f3adc003a00 2024-03-18T18:51:13.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:13.864+0000 7f3af9105700 1 -- 172.21.15.1:0/3046772380 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3af40cb590 con 0x7f3adc003a00 2024-03-18T18:51:14.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.730+0000 7f3adb7fe700 1 -- 172.21.15.1:0/3046772380 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3ae808a020 con 0x7f3adc003a00 2024-03-18T18:51:14.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.732+0000 7f3af9105700 1 -- 172.21.15.1:0/3046772380 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ae006ac90 msgr2=0x7f3ae006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:14.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.732+0000 7f3af9105700 1 --2- 172.21.15.1:0/3046772380 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ae006ac90 0x7f3ae006d150 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4006f10 tx=0x7f3ae4008040 comp rx=0 tx=0).stop 2024-03-18T18:51:14.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.732+0000 7f3af9105700 1 -- 172.21.15.1:0/3046772380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3adc003a00 msgr2=0x7f3af41f5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:14.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.732+0000 7f3af9105700 1 --2- 172.21.15.1:0/3046772380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3adc003a00 0x7f3af41f5920 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8015040 tx=0x7f3ae8001bf0 comp rx=0 tx=0).stop 2024-03-18T18:51:14.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.733+0000 7f3af9105700 1 -- 172.21.15.1:0/3046772380 shutdown_connections 2024-03-18T18:51:14.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.733+0000 7f3af9105700 1 --2- 172.21.15.1:0/3046772380 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ae006ac90 0x7f3ae006d150 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:14.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.733+0000 7f3af9105700 1 --2- 172.21.15.1:0/3046772380 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3af414b300 0x7f3af4158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:14.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.733+0000 7f3af9105700 1 --2- 172.21.15.1:0/3046772380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3adc003a00 0x7f3af41f5920 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:14.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.733+0000 7f3af9105700 1 --2- 172.21.15.1:0/3046772380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3af414ce50 0x7f3af414adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:14.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.733+0000 7f3af9105700 1 -- 172.21.15.1:0/3046772380 >> 172.21.15.1:0/3046772380 conn(0x7f3af40cc9b0 msgr2=0x7f3af415c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:14.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.733+0000 7f3af9105700 1 -- 172.21.15.1:0/3046772380 shutdown_connections 2024-03-18T18:51:14.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.733+0000 7f3af9105700 1 -- 172.21.15.1:0/3046772380 wait complete. 2024-03-18T18:51:14.744 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-18T18:51:14.744 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T18:51:14.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.883+0000 7fe1f2a47700 1 Processor -- start 2024-03-18T18:51:14.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.883+0000 7fe1f2a47700 1 -- start start 2024-03-18T18:51:14.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.884+0000 7fe1f2a47700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1ec15d5a0 0x7fe1ec0ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:14.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.884+0000 7fe1f2a47700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe1ec06ae60 0x7fe1ec167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:14.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.884+0000 7fe1f2a47700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1ec0cf600 con 0x7fe1ec0ce550 2024-03-18T18:51:14.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.884+0000 7fe1f2a47700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1ec0ce9f0 con 0x7fe1ec06ae60 2024-03-18T18:51:14.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.884+0000 7fe1f2a47700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec0cecd0 con 0x7fe1ec15d5a0 2024-03-18T18:51:14.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.884+0000 7fe1eb7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe1ec0ce550 0x7fe1ec06a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56392/0 (socket says 172.21.15.1:56392) 2024-03-18T18:51:14.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.884+0000 7fe1eb7fe700 1 -- 172.21.15.1:0/1954960925 learned_addr learned my addr 172.21.15.1:0/1954960925 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:14.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.884+0000 7fe1f0fe4700 1 --2- 172.21.15.1:0/1954960925 >> v2:172.21.15.165:3300/0 conn(0x7fe1ec06ae60 0x7fe1ec167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:14.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.884+0000 7fe1ebfff700 1 --2- 172.21.15.1:0/1954960925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1ec15d5a0 0x7fe1ec0ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:14.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.884+0000 7fe1eaffd700 1 -- 172.21.15.1:0/1954960925 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248533907 0 0) 0x7fe1ec0cf600 con 0x7fe1ec0ce550 2024-03-18T18:51:14.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.884+0000 7fe1eaffd700 1 -- 172.21.15.1:0/1954960925 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d40035b0 con 0x7fe1ec0ce550 2024-03-18T18:51:14.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.884+0000 7fe1f0fe4700 1 -- 172.21.15.1:0/1954960925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1ec15d5a0 msgr2=0x7fe1ec0ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:14.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.884+0000 7fe1f0fe4700 1 --2- 172.21.15.1:0/1954960925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1ec15d5a0 0x7fe1ec0ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:14.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.884+0000 7fe1f0fe4700 1 -- 172.21.15.1:0/1954960925 >> v1:172.21.15.1:6789/0 conn(0x7fe1ec0ce550 legacy=0x7fe1ec06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:14.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.884+0000 7fe1f0fe4700 1 -- 172.21.15.1:0/1954960925 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ec0cefb0 con 0x7fe1ec06ae60 2024-03-18T18:51:14.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.884+0000 7fe1ebfff700 1 --2- 172.21.15.1:0/1954960925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1ec15d5a0 0x7fe1ec0ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:51:14.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.885+0000 7fe1f0fe4700 1 --2- 172.21.15.1:0/1954960925 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1ec06ae60 0x7fe1ec167140 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe1e4009d90 tx=0x7fe1e400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb119d1d96819858 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:14.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.885+0000 7fe1eaffd700 1 -- 172.21.15.1:0/1954960925 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e400e070 con 0x7fe1ec06ae60 2024-03-18T18:51:14.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.885+0000 7fe1eaffd700 1 -- 172.21.15.1:0/1954960925 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e400bb20 con 0x7fe1ec06ae60 2024-03-18T18:51:14.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.885+0000 7fe1eaffd700 1 -- 172.21.15.1:0/1954960925 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e4012430 con 0x7fe1ec06ae60 2024-03-18T18:51:14.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.885+0000 7fe1f2a47700 1 -- 172.21.15.1:0/1954960925 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1ec06ae60 msgr2=0x7fe1ec167140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:14.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.885+0000 7fe1f2a47700 1 --2- 172.21.15.1:0/1954960925 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1ec06ae60 0x7fe1ec167140 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe1e4009d90 tx=0x7fe1e400b330 comp rx=0 tx=0).stop 2024-03-18T18:51:14.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.885+0000 7fe1f2a47700 1 -- 172.21.15.1:0/1954960925 shutdown_connections 2024-03-18T18:51:14.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.885+0000 7fe1f2a47700 1 --2- 172.21.15.1:0/1954960925 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1ec06ae60 0x7fe1ec167140 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:14.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.885+0000 7fe1f2a47700 1 --2- 172.21.15.1:0/1954960925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1ec15d5a0 0x7fe1ec0ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:14.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.885+0000 7fe1f2a47700 1 -- 172.21.15.1:0/1954960925 >> 172.21.15.1:0/1954960925 conn(0x7fe1ec0cc9b0 msgr2=0x7fe1ec0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:14.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.885+0000 7fe1f2a47700 1 -- 172.21.15.1:0/1954960925 shutdown_connections 2024-03-18T18:51:14.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.886+0000 7fe1f2a47700 1 -- 172.21.15.1:0/1954960925 wait complete. 2024-03-18T18:51:14.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.887+0000 7fe1f2a47700 1 Processor -- start 2024-03-18T18:51:14.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.887+0000 7fe1f2a47700 1 -- start start 2024-03-18T18:51:14.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.887+0000 7fe1f2a47700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1ec06ae60 0x7fe1ec156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:14.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.887+0000 7fe1f2a47700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1ec0ce550 0x7fe1ec15ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:14.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.887+0000 7fe1f2a47700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1ec15d5a0 0x7fe1ec160480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.888+0000 7fe1f2a47700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec152f40 con 0x7fe1ec15d5a0 2024-03-18T18:51:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.888+0000 7fe1f2a47700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec153220 con 0x7fe1ec0ce550 2024-03-18T18:51:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.888+0000 7fe1f2a47700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec153500 con 0x7fe1ec06ae60 2024-03-18T18:51:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.888+0000 7fe1f0fe4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1ec15d5a0 0x7fe1ec160480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.888+0000 7fe1f0fe4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1ec15d5a0 0x7fe1ec160480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39702/0 (socket says 172.21.15.1:39702) 2024-03-18T18:51:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.888+0000 7fe1f0fe4700 1 -- 172.21.15.1:0/647824194 learned_addr learned my addr 172.21.15.1:0/647824194 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.888+0000 7fe1f0fe4700 1 -- 172.21.15.1:0/647824194 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1ec06ae60 msgr2=0x7fe1ec156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.888+0000 7fe1f0fe4700 1 --2- 172.21.15.1:0/647824194 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1ec06ae60 0x7fe1ec156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.888+0000 7fe1f0fe4700 1 -- 172.21.15.1:0/647824194 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1ec0ce550 msgr2=0x7fe1ec15ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.888+0000 7fe1eb7fe700 1 --2- 172.21.15.1:0/647824194 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1ec0ce550 0x7fe1ec15ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.888+0000 7fe1f0fe4700 1 --2- 172.21.15.1:0/647824194 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1ec0ce550 0x7fe1ec15ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.888+0000 7fe1f0fe4700 1 -- 172.21.15.1:0/647824194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e0009430 con 0x7fe1ec15d5a0 2024-03-18T18:51:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.888+0000 7fe1f0fe4700 1 --2- 172.21.15.1:0/647824194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1ec15d5a0 0x7fe1ec160480 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe1e4014040 tx=0x7fe1e4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.888+0000 7fe1e8ff9700 1 -- 172.21.15.1:0/647824194 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e4012a70 con 0x7fe1ec15d5a0 2024-03-18T18:51:14.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.888+0000 7fe1e8ff9700 1 -- 172.21.15.1:0/647824194 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e4020070 con 0x7fe1ec15d5a0 2024-03-18T18:51:14.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.888+0000 7fe1f2a47700 1 -- 172.21.15.1:0/647824194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e4009a70 con 0x7fe1ec15d5a0 2024-03-18T18:51:14.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.888+0000 7fe1e8ff9700 1 -- 172.21.15.1:0/647824194 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e4012e30 con 0x7fe1ec15d5a0 2024-03-18T18:51:14.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.888+0000 7fe1f2a47700 1 -- 172.21.15.1:0/647824194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1ec153ac0 con 0x7fe1ec15d5a0 2024-03-18T18:51:14.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.889+0000 7fe1f2a47700 1 -- 172.21.15.1:0/647824194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1ec06b620 con 0x7fe1ec15d5a0 2024-03-18T18:51:14.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.890+0000 7fe1e8ff9700 1 -- 172.21.15.1:0/647824194 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe1e4004040 con 0x7fe1ec15d5a0 2024-03-18T18:51:14.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.890+0000 7fe1e8ff9700 1 --2- 172.21.15.1:0/647824194 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1d806ac90 0x7fe1d806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:14.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.890+0000 7fe1e8ff9700 1 -- 172.21.15.1:0/647824194 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe1e4022070 con 0x7fe1ec15d5a0 2024-03-18T18:51:14.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.890+0000 7fe1ebfff700 1 --2- 172.21.15.1:0/647824194 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1d806ac90 0x7fe1d806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:14.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.891+0000 7fe1ebfff700 1 --2- 172.21.15.1:0/647824194 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1d806ac90 0x7fe1d806d150 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0006eb0 tx=0x7fe1e0006e00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:14.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:14.895+0000 7fe1e8ff9700 1 -- 172.21.15.1:0/647824194 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe1e4059d60 con 0x7fe1ec15d5a0 2024-03-18T18:51:15.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.143+0000 7fe1f2a47700 1 -- 172.21.15.1:0/647824194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe1ec153cb0 con 0x7fe1ec15d5a0 2024-03-18T18:51:15.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.731+0000 7fe1e8ff9700 1 -- 172.21.15.1:0/647824194 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe1e40598f0 con 0x7fe1ec15d5a0 2024-03-18T18:51:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.734+0000 7fe1f2a47700 1 -- 172.21.15.1:0/647824194 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1d806ac90 msgr2=0x7fe1d806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.734+0000 7fe1f2a47700 1 --2- 172.21.15.1:0/647824194 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1d806ac90 0x7fe1d806d150 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0006eb0 tx=0x7fe1e0006e00 comp rx=0 tx=0).stop 2024-03-18T18:51:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.734+0000 7fe1f2a47700 1 -- 172.21.15.1:0/647824194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1ec15d5a0 msgr2=0x7fe1ec160480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.734+0000 7fe1f2a47700 1 --2- 172.21.15.1:0/647824194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1ec15d5a0 0x7fe1ec160480 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe1e4014040 tx=0x7fe1e4004ad0 comp rx=0 tx=0).stop 2024-03-18T18:51:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.734+0000 7fe1f2a47700 1 -- 172.21.15.1:0/647824194 shutdown_connections 2024-03-18T18:51:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.734+0000 7fe1f2a47700 1 --2- 172.21.15.1:0/647824194 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1d806ac90 0x7fe1d806d150 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.734+0000 7fe1f2a47700 1 --2- 172.21.15.1:0/647824194 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1ec06ae60 0x7fe1ec156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:15.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.734+0000 7fe1f2a47700 1 --2- 172.21.15.1:0/647824194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1ec15d5a0 0x7fe1ec160480 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.734+0000 7fe1f2a47700 1 --2- 172.21.15.1:0/647824194 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1ec0ce550 0x7fe1ec15ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.734+0000 7fe1f2a47700 1 -- 172.21.15.1:0/647824194 >> 172.21.15.1:0/647824194 conn(0x7fe1ec0cc9b0 msgr2=0x7fe1ec1e65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.734+0000 7fe1f2a47700 1 -- 172.21.15.1:0/647824194 shutdown_connections 2024-03-18T18:51:15.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.734+0000 7fe1f2a47700 1 -- 172.21.15.1:0/647824194 wait complete. 2024-03-18T18:51:15.745 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T18:51:15.746 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T18:51:15.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.881+0000 7fdf70982700 1 Processor -- start 2024-03-18T18:51:15.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.881+0000 7fdf70982700 1 -- start start 2024-03-18T18:51:15.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.882+0000 7fdf70982700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdf6c141710 0x7fdf6c0cf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:15.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.882+0000 7fdf70982700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdf6c15c150 0x7fdf6c0cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:15.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.882+0000 7fdf70982700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6c141af0 con 0x7fdf6c0cf7c0 2024-03-18T18:51:15.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.882+0000 7fdf70982700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf6c0d01b0 con 0x7fdf6c15c150 2024-03-18T18:51:15.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.882+0000 7fdf70982700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf6c15c5a0 con 0x7fdf6c141710 2024-03-18T18:51:15.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.882+0000 7fdf6ad9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdf6c15c150 0x7fdf6c0cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:15.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.882+0000 7fdf6ad9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdf6c15c150 0x7fdf6c0cfc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39712/0 (socket says 172.21.15.1:39712) 2024-03-18T18:51:15.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.882+0000 7fdf6ad9d700 1 -- 172.21.15.1:0/886161149 learned_addr learned my addr 172.21.15.1:0/886161149 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:15.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.882+0000 7fdf6a59c700 1 --2- 172.21.15.1:0/886161149 >> v2:172.21.15.165:3300/0 conn(0x7fdf6c141710 0x7fdf6c0cf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:15.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.882+0000 7fdf6ad9d700 1 -- 172.21.15.1:0/886161149 >> v2:172.21.15.165:3300/0 conn(0x7fdf6c141710 msgr2=0x7fdf6c0cf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:15.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.882+0000 7fdf6ad9d700 1 --2- 172.21.15.1:0/886161149 >> v2:172.21.15.165:3300/0 conn(0x7fdf6c141710 0x7fdf6c0cf1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:15.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.882+0000 7fdf6ad9d700 1 -- 172.21.15.1:0/886161149 >> v1:172.21.15.165:6789/0 conn(0x7fdf6c0cf7c0 legacy=0x7fdf6c15ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:15.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.882+0000 7fdf6ad9d700 1 -- 172.21.15.1:0/886161149 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c0ce420 con 0x7fdf6c15c150 2024-03-18T18:51:15.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.882+0000 7fdf6ad9d700 1 --2- 172.21.15.1:0/886161149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf6c15c150 0x7fdf6c0cfc70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fdf64009e50 tx=0x7fdf6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b079a8dd44b0d14 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:15.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.883+0000 7fdf6959a700 1 -- 172.21.15.1:0/886161149 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6400e070 con 0x7fdf6c15c150 2024-03-18T18:51:15.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.883+0000 7fdf6959a700 1 -- 172.21.15.1:0/886161149 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6400b980 con 0x7fdf6c15c150 2024-03-18T18:51:15.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.883+0000 7fdf6959a700 1 -- 172.21.15.1:0/886161149 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6400bd30 con 0x7fdf6c15c150 2024-03-18T18:51:15.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.883+0000 7fdf70982700 1 -- 172.21.15.1:0/886161149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf6c15c150 msgr2=0x7fdf6c0cfc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:15.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.883+0000 7fdf70982700 1 --2- 172.21.15.1:0/886161149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf6c15c150 0x7fdf6c0cfc70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fdf64009e50 tx=0x7fdf6400b330 comp rx=0 tx=0).stop 2024-03-18T18:51:15.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.883+0000 7fdf70982700 1 -- 172.21.15.1:0/886161149 shutdown_connections 2024-03-18T18:51:15.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.883+0000 7fdf70982700 1 --2- 172.21.15.1:0/886161149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf6c15c150 0x7fdf6c0cfc70 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:15.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.883+0000 7fdf70982700 1 --2- 172.21.15.1:0/886161149 >> v2:172.21.15.165:3300/0 conn(0x7fdf6c141710 0x7fdf6c0cf1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:15.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.883+0000 7fdf70982700 1 -- 172.21.15.1:0/886161149 >> 172.21.15.1:0/886161149 conn(0x7fdf6c06a750 msgr2=0x7fdf6c06ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:15.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.883+0000 7fdf70982700 1 -- 172.21.15.1:0/886161149 shutdown_connections 2024-03-18T18:51:15.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.883+0000 7fdf70982700 1 -- 172.21.15.1:0/886161149 wait complete. 2024-03-18T18:51:15.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.885+0000 7fdf70982700 1 Processor -- start 2024-03-18T18:51:15.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.885+0000 7fdf70982700 1 -- start start 2024-03-18T18:51:15.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.885+0000 7fdf70982700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf6c0cf7c0 0x7fdf6c13ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:15.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.885+0000 7fdf70982700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf6c141710 0x7fdf6c1f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:15.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.885+0000 7fdf70982700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf6c15c150 0x7fdf6c1f3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:15.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.885+0000 7fdf70982700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c0b12e0 con 0x7fdf6c141710 2024-03-18T18:51:15.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.885+0000 7fdf70982700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c0b0960 con 0x7fdf6c15c150 2024-03-18T18:51:15.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.885+0000 7fdf70982700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c0b0c20 con 0x7fdf6c0cf7c0 2024-03-18T18:51:15.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.885+0000 7fdf69d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf6c141710 0x7fdf6c1f2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:15.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.885+0000 7fdf69d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf6c141710 0x7fdf6c1f2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39728/0 (socket says 172.21.15.1:39728) 2024-03-18T18:51:15.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.885+0000 7fdf69d9b700 1 -- 172.21.15.1:0/1550389621 learned_addr learned my addr 172.21.15.1:0/1550389621 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:15.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.886+0000 7fdf6a59c700 1 --2- 172.21.15.1:0/1550389621 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf6c0cf7c0 0x7fdf6c13ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:15.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.886+0000 7fdf6ad9d700 1 --2- 172.21.15.1:0/1550389621 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf6c15c150 0x7fdf6c1f3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:15.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.886+0000 7fdf69d9b700 1 -- 172.21.15.1:0/1550389621 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf6c0cf7c0 msgr2=0x7fdf6c13ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:15.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.886+0000 7fdf69d9b700 1 --2- 172.21.15.1:0/1550389621 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf6c0cf7c0 0x7fdf6c13ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:15.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.886+0000 7fdf69d9b700 1 -- 172.21.15.1:0/1550389621 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf6c15c150 msgr2=0x7fdf6c1f3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:15.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.886+0000 7fdf69d9b700 1 --2- 172.21.15.1:0/1550389621 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf6c15c150 0x7fdf6c1f3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:15.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.886+0000 7fdf69d9b700 1 -- 172.21.15.1:0/1550389621 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf64009b30 con 0x7fdf6c141710 2024-03-18T18:51:15.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.886+0000 7fdf69d9b700 1 --2- 172.21.15.1:0/1550389621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf6c141710 0x7fdf6c1f2e30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fdf6000c490 tx=0x7fdf6000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:15.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.886+0000 7fdf5affd700 1 -- 172.21.15.1:0/1550389621 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf600052d0 con 0x7fdf6c141710 2024-03-18T18:51:15.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.886+0000 7fdf5affd700 1 -- 172.21.15.1:0/1550389621 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf60005450 con 0x7fdf6c141710 2024-03-18T18:51:15.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.886+0000 7fdf70982700 1 -- 172.21.15.1:0/1550389621 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6c0b0f00 con 0x7fdf6c141710 2024-03-18T18:51:15.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.886+0000 7fdf5affd700 1 -- 172.21.15.1:0/1550389621 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf60006aa0 con 0x7fdf6c141710 2024-03-18T18:51:15.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.886+0000 7fdf70982700 1 -- 172.21.15.1:0/1550389621 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6c0b1700 con 0x7fdf6c141710 2024-03-18T18:51:15.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.887+0000 7fdf70982700 1 -- 172.21.15.1:0/1550389621 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf6c068240 con 0x7fdf6c141710 2024-03-18T18:51:15.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.888+0000 7fdf5affd700 1 -- 172.21.15.1:0/1550389621 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fdf60003680 con 0x7fdf6c141710 2024-03-18T18:51:15.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.888+0000 7fdf5affd700 1 --2- 172.21.15.1:0/1550389621 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf4c06ac90 0x7fdf4c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:15.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.888+0000 7fdf5affd700 1 -- 172.21.15.1:0/1550389621 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fdf600887c0 con 0x7fdf6c141710 2024-03-18T18:51:15.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.888+0000 7fdf6a59c700 1 --2- 172.21.15.1:0/1550389621 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf4c06ac90 0x7fdf4c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:15.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.889+0000 7fdf6a59c700 1 --2- 172.21.15.1:0/1550389621 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf4c06ac90 0x7fdf4c06d150 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009930 tx=0x7fdf5c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:15.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:15.893+0000 7fdf5affd700 1 -- 172.21.15.1:0/1550389621 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdf600537e0 con 0x7fdf6c141710 2024-03-18T18:51:16.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.141+0000 7fdf70982700 1 -- 172.21.15.1:0/1550389621 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf6c0676a0 con 0x7fdf6c141710 2024-03-18T18:51:16.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.141+0000 7fdf5affd700 1 -- 172.21.15.1:0/1550389621 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4734 (secure 0 0 0) 0x7fdf60056e00 con 0x7fdf6c141710 2024-03-18T18:51:16.138 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:51:16.138 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4874,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2002167042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2002167042},{"type":"v1","addr":"172.21.15.165:6835","nonce":2002167042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4880,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3844556325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3844556325},{"type":"v1","addr":"172.21.15.1:6837","nonce":3844556325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5278,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1229938169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1229938169},{"type":"v1","addr":"172.21.15.165:6834","nonce":1229938169}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T18:51:12.721816+0000","modified":"2024-03-18T18:51:15.728690+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5436":{"gid":5436,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6836/3077516107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3077516107},{"type":"v1","addr":"172.21.15.1:6836","nonce":3077516107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-18T18:51:16.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.144+0000 7fdf70982700 1 -- 172.21.15.1:0/1550389621 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf4c06ac90 msgr2=0x7fdf4c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:16.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.144+0000 7fdf70982700 1 --2- 172.21.15.1:0/1550389621 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf4c06ac90 0x7fdf4c06d150 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009930 tx=0x7fdf5c008040 comp rx=0 tx=0).stop 2024-03-18T18:51:16.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.144+0000 7fdf70982700 1 -- 172.21.15.1:0/1550389621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf6c141710 msgr2=0x7fdf6c1f2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:16.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.144+0000 7fdf70982700 1 --2- 172.21.15.1:0/1550389621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf6c141710 0x7fdf6c1f2e30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fdf6000c490 tx=0x7fdf6000c850 comp rx=0 tx=0).stop 2024-03-18T18:51:16.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.144+0000 7fdf70982700 1 -- 172.21.15.1:0/1550389621 shutdown_connections 2024-03-18T18:51:16.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.144+0000 7fdf70982700 1 --2- 172.21.15.1:0/1550389621 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf4c06ac90 0x7fdf4c06d150 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.144+0000 7fdf70982700 1 --2- 172.21.15.1:0/1550389621 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf6c0cf7c0 0x7fdf6c13ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.144+0000 7fdf70982700 1 --2- 172.21.15.1:0/1550389621 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf6c15c150 0x7fdf6c1f3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.144+0000 7fdf70982700 1 --2- 172.21.15.1:0/1550389621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf6c141710 0x7fdf6c1f2e30 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.144+0000 7fdf70982700 1 -- 172.21.15.1:0/1550389621 >> 172.21.15.1:0/1550389621 conn(0x7fdf6c06a750 msgr2=0x7fdf6c0ae6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:16.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.144+0000 7fdf70982700 1 -- 172.21.15.1:0/1550389621 shutdown_connections 2024-03-18T18:51:16.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.144+0000 7fdf70982700 1 -- 172.21.15.1:0/1550389621 wait complete. 2024-03-18T18:51:16.142 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 17 2024-03-18T18:51:16.152 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T18:51:16.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.282+0000 7f74b2e75700 1 Processor -- start 2024-03-18T18:51:16.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.282+0000 7f74b2e75700 1 -- start start 2024-03-18T18:51:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.282+0000 7f74b2e75700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f74ac15a320 0x7f74ac0ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.282+0000 7f74b2e75700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74ac071d10 0x7f74ac06e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.282+0000 7f74b2e75700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74ac0acd80 con 0x7f74ac0cea30 2024-03-18T18:51:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.282+0000 7f74b2e75700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74ac0cf220 con 0x7f74ac15a320 2024-03-18T18:51:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.282+0000 7f74b2e75700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac0cef40 con 0x7f74ac071d10 2024-03-18T18:51:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.283+0000 7f74b1412700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74ac071d10 0x7f74ac06e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.283+0000 7f74b0c11700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f74ac15a320 0x7f74ac0ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.283+0000 7f74b0c11700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f74ac15a320 0x7f74ac0ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:54780/0 (socket says 172.21.15.1:54780) 2024-03-18T18:51:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.283+0000 7f74abfff700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f74ac0cea30 0x7f74ac071600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:47684/0 (socket says 172.21.15.1:47684) 2024-03-18T18:51:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.283+0000 7f74b0c11700 1 -- 172.21.15.1:0/4073783362 learned_addr learned my addr 172.21.15.1:0/4073783362 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.283+0000 7f74b1412700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74ac071d10 0x7f74ac06e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:51238/0 (socket says 172.21.15.1:51238) 2024-03-18T18:51:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.283+0000 7f74b0c11700 1 -- 172.21.15.1:0/4073783362 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74ac071d10 msgr2=0x7f74ac06e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.283+0000 7f74b0c11700 1 --2- 172.21.15.1:0/4073783362 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74ac071d10 0x7f74ac06e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.283+0000 7f74b0c11700 1 -- 172.21.15.1:0/4073783362 >> v1:172.21.15.165:6789/0 conn(0x7f74ac0cea30 legacy=0x7f74ac071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.283+0000 7f74b0c11700 1 -- 172.21.15.1:0/4073783362 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ac06d7e0 con 0x7f74ac15a320 2024-03-18T18:51:16.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.283+0000 7f74b1412700 1 --2- 172.21.15.1:0/4073783362 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74ac071d10 0x7f74ac06e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:51:16.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.283+0000 7f74b0c11700 1 --2- 172.21.15.1:0/4073783362 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74ac15a320 0x7f74ac0ce4f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f74a00098f0 tx=0x7f74a0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a2d2155c91c185e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:16.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.283+0000 7f74ab7fe700 1 -- 172.21.15.1:0/4073783362 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a0011830 con 0x7f74ac15a320 2024-03-18T18:51:16.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.283+0000 7f74ab7fe700 1 -- 172.21.15.1:0/4073783362 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a0011990 con 0x7f74ac15a320 2024-03-18T18:51:16.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.284+0000 7f74ab7fe700 1 -- 172.21.15.1:0/4073783362 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a0011d20 con 0x7f74ac15a320 2024-03-18T18:51:16.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.284+0000 7f74b2e75700 1 -- 172.21.15.1:0/4073783362 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74ac15a320 msgr2=0x7f74ac0ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:16.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.284+0000 7f74b2e75700 1 --2- 172.21.15.1:0/4073783362 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74ac15a320 0x7f74ac0ce4f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f74a00098f0 tx=0x7f74a0009c00 comp rx=0 tx=0).stop 2024-03-18T18:51:16.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.284+0000 7f74b2e75700 1 -- 172.21.15.1:0/4073783362 shutdown_connections 2024-03-18T18:51:16.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.284+0000 7f74b2e75700 1 --2- 172.21.15.1:0/4073783362 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74ac071d10 0x7f74ac06e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.284+0000 7f74b2e75700 1 --2- 172.21.15.1:0/4073783362 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74ac15a320 0x7f74ac0ce4f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.284+0000 7f74b2e75700 1 -- 172.21.15.1:0/4073783362 >> 172.21.15.1:0/4073783362 conn(0x7f74ac0cc990 msgr2=0x7f74ac0ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:16.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.284+0000 7f74b2e75700 1 -- 172.21.15.1:0/4073783362 shutdown_connections 2024-03-18T18:51:16.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.284+0000 7f74b2e75700 1 -- 172.21.15.1:0/4073783362 wait complete. 2024-03-18T18:51:16.283 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-18T18:51:16.285+0000 7f42a852b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T18:51:16.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.286+0000 7f74b2e75700 1 Processor -- start 2024-03-18T18:51:16.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.286+0000 7f74b2e75700 1 -- start start 2024-03-18T18:51:16.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.287+0000 7f74b2e75700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74ac071d10 0x7f74ac0c5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:16.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.287+0000 7f74b2e75700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74ac0cea30 0x7f74ac0c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:16.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.287+0000 7f74b2e75700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74ac0c5ca0 0x7f74ac0b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:16.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.287+0000 7f74b2e75700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac0b2ad0 con 0x7f74ac0cea30 2024-03-18T18:51:16.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.287+0000 7f74b2e75700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac0c6130 con 0x7f74ac071d10 2024-03-18T18:51:16.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.287+0000 7f74b2e75700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac0c6410 con 0x7f74ac0c5ca0 2024-03-18T18:51:16.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.287+0000 7f74abfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74ac0cea30 0x7f74ac0c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:16.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.287+0000 7f74abfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74ac0cea30 0x7f74ac0c5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39742/0 (socket says 172.21.15.1:39742) 2024-03-18T18:51:16.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.287+0000 7f74abfff700 1 -- 172.21.15.1:0/3474780024 learned_addr learned my addr 172.21.15.1:0/3474780024 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:16.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.287+0000 7f74b1412700 1 --2- 172.21.15.1:0/3474780024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74ac0c5ca0 0x7f74ac0b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:16.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.287+0000 7f74b0c11700 1 --2- 172.21.15.1:0/3474780024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74ac071d10 0x7f74ac0c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:16.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.287+0000 7f74abfff700 1 -- 172.21.15.1:0/3474780024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74ac0c5ca0 msgr2=0x7f74ac0b4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:16.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.287+0000 7f74abfff700 1 --2- 172.21.15.1:0/3474780024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74ac0c5ca0 0x7f74ac0b4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.287+0000 7f74abfff700 1 -- 172.21.15.1:0/3474780024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74ac071d10 msgr2=0x7f74ac0c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:16.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.287+0000 7f74abfff700 1 --2- 172.21.15.1:0/3474780024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74ac071d10 0x7f74ac0c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.287+0000 7f74abfff700 1 -- 172.21.15.1:0/3474780024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a4009a70 con 0x7f74ac0cea30 2024-03-18T18:51:16.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.287+0000 7f74b0c11700 1 --2- 172.21.15.1:0/3474780024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74ac071d10 0x7f74ac0c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:16.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.287+0000 7f74b1412700 1 --2- 172.21.15.1:0/3474780024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74ac0c5ca0 0x7f74ac0b4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:16.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.287+0000 7f74abfff700 1 --2- 172.21.15.1:0/3474780024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74ac0cea30 0x7f74ac0c5760 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f749800c4f0 tx=0x7f749800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:16.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.288+0000 7f74a97fa700 1 -- 172.21.15.1:0/3474780024 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7498005cb0 con 0x7f74ac0cea30 2024-03-18T18:51:16.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.288+0000 7f74b2e75700 1 -- 172.21.15.1:0/3474780024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a0009550 con 0x7f74ac0cea30 2024-03-18T18:51:16.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.288+0000 7f74a97fa700 1 -- 172.21.15.1:0/3474780024 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7498005e10 con 0x7f74ac0cea30 2024-03-18T18:51:16.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.288+0000 7f74a97fa700 1 -- 172.21.15.1:0/3474780024 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74980072d0 con 0x7f74ac0cea30 2024-03-18T18:51:16.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.288+0000 7f74b2e75700 1 -- 172.21.15.1:0/3474780024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74ac0c69d0 con 0x7f74ac0cea30 2024-03-18T18:51:16.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.288+0000 7f74b2e75700 1 -- 172.21.15.1:0/3474780024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74ac0676a0 con 0x7f74ac0cea30 2024-03-18T18:51:16.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.289+0000 7f74a97fa700 1 -- 172.21.15.1:0/3474780024 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7498007430 con 0x7f74ac0cea30 2024-03-18T18:51:16.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.289+0000 7f74a97fa700 1 --2- 172.21.15.1:0/3474780024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f749006ac90 0x7f749006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:16.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.290+0000 7f74a97fa700 1 -- 172.21.15.1:0/3474780024 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7498088950 con 0x7f74ac0cea30 2024-03-18T18:51:16.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.290+0000 7f74b0c11700 1 --2- 172.21.15.1:0/3474780024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f749006ac90 0x7f749006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:16.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.290+0000 7f74b0c11700 1 --2- 172.21.15.1:0/3474780024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f749006ac90 0x7f749006d150 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f74a0014040 tx=0x7f74a0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:16.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.294+0000 7f74a97fa700 1 -- 172.21.15.1:0/3474780024 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7498053990 con 0x7f74ac0cea30 2024-03-18T18:51:16.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.499+0000 7f74b2e75700 1 -- 172.21.15.1:0/3474780024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74ac0b2c10 con 0x7f74ac0cea30 2024-03-18T18:51:16.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.500+0000 7f74a97fa700 1 -- 172.21.15.1:0/3474780024 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+16258 (secure 0 0 0) 0x7f7498017090 con 0x7f74ac0cea30 2024-03-18T18:51:16.496 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:51:16.497 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":40,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T18:51:13.723449+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:51:11.076221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T18:51:11.687105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T18:51:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.502+0000 7f74b2e75700 1 -- 172.21.15.1:0/3474780024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f749006ac90 msgr2=0x7f749006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.502+0000 7f74b2e75700 1 --2- 172.21.15.1:0/3474780024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f749006ac90 0x7f749006d150 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f74a0014040 tx=0x7f74a0008030 comp rx=0 tx=0).stop 2024-03-18T18:51:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.502+0000 7f74b2e75700 1 -- 172.21.15.1:0/3474780024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74ac0cea30 msgr2=0x7f74ac0c5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.502+0000 7f74b2e75700 1 --2- 172.21.15.1:0/3474780024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74ac0cea30 0x7f74ac0c5760 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f749800c4f0 tx=0x7f749800c8b0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.502+0000 7f74b2e75700 1 -- 172.21.15.1:0/3474780024 shutdown_connections 2024-03-18T18:51:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.503+0000 7f74b2e75700 1 --2- 172.21.15.1:0/3474780024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f749006ac90 0x7f749006d150 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.503+0000 7f74b2e75700 1 --2- 172.21.15.1:0/3474780024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74ac0c5ca0 0x7f74ac0b4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.503+0000 7f74b2e75700 1 --2- 172.21.15.1:0/3474780024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74ac071d10 0x7f74ac0c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.503+0000 7f74b2e75700 1 --2- 172.21.15.1:0/3474780024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74ac0cea30 0x7f74ac0c5760 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.503+0000 7f74b2e75700 1 -- 172.21.15.1:0/3474780024 >> 172.21.15.1:0/3474780024 conn(0x7f74ac0cc990 msgr2=0x7f74ac0afaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:16.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.503+0000 7f74b2e75700 1 -- 172.21.15.1:0/3474780024 shutdown_connections 2024-03-18T18:51:16.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.503+0000 7f74b2e75700 1 -- 172.21.15.1:0/3474780024 wait complete. 2024-03-18T18:51:16.511 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T18:51:16.512 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T18:51:16.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.640+0000 7f3683249700 1 Processor -- start 2024-03-18T18:51:16.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.640+0000 7f3683249700 1 -- start start 2024-03-18T18:51:16.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.641+0000 7f3683249700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f367c15d6b0 0x7f367c0b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:16.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.641+0000 7f3683249700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f367c0ad790 0x7f367c0b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:16.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.641+0000 7f3683249700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367c0acfb0 con 0x7f367c06a750 2024-03-18T18:51:16.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.641+0000 7f3683249700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367c0af7a0 con 0x7f367c15d6b0 2024-03-18T18:51:16.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.641+0000 7f3683249700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367c0b50a0 con 0x7f367c0ad790 2024-03-18T18:51:16.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.641+0000 7f367bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f367c15d6b0 0x7f367c0b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:16.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.641+0000 7f3680fe5700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f367c06a750 0x7f367c06ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56404/0 (socket says 172.21.15.1:56404) 2024-03-18T18:51:16.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.641+0000 7f3680fe5700 1 -- 172.21.15.1:0/4080276289 learned_addr learned my addr 172.21.15.1:0/4080276289 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:16.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.642+0000 7f36817e6700 1 --2- 172.21.15.1:0/4080276289 >> v2:172.21.15.165:3300/0 conn(0x7f367c0ad790 0x7f367c0b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:16.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.642+0000 7f367b7fe700 1 -- 172.21.15.1:0/4080276289 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846661303 0 0) 0x7f367c0acfb0 con 0x7f367c06a750 2024-03-18T18:51:16.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.642+0000 7f367b7fe700 1 -- 172.21.15.1:0/4080276289 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f366c0035b0 con 0x7f367c06a750 2024-03-18T18:51:16.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.642+0000 7f367bfff700 1 -- 172.21.15.1:0/4080276289 >> v2:172.21.15.165:3300/0 conn(0x7f367c0ad790 msgr2=0x7f367c0b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:16.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.642+0000 7f367bfff700 1 --2- 172.21.15.1:0/4080276289 >> v2:172.21.15.165:3300/0 conn(0x7f367c0ad790 0x7f367c0b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.642+0000 7f367bfff700 1 -- 172.21.15.1:0/4080276289 >> v1:172.21.15.1:6789/0 conn(0x7f367c06a750 legacy=0x7f367c06ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:16.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.642+0000 7f367bfff700 1 -- 172.21.15.1:0/4080276289 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367c0af1e0 con 0x7f367c15d6b0 2024-03-18T18:51:16.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.642+0000 7f367bfff700 1 --2- 172.21.15.1:0/4080276289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f367c15d6b0 0x7f367c0b2da0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f367c0726a0 tx=0x7f3668009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84ed2b5cc9c13221 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:16.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.642+0000 7f367b7fe700 1 -- 172.21.15.1:0/4080276289 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668011830 con 0x7f367c15d6b0 2024-03-18T18:51:16.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.642+0000 7f367b7fe700 1 -- 172.21.15.1:0/4080276289 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3668011990 con 0x7f367c15d6b0 2024-03-18T18:51:16.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.642+0000 7f367b7fe700 1 -- 172.21.15.1:0/4080276289 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668011d20 con 0x7f367c15d6b0 2024-03-18T18:51:16.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.643+0000 7f3683249700 1 -- 172.21.15.1:0/4080276289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f367c15d6b0 msgr2=0x7f367c0b2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:16.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.643+0000 7f3683249700 1 --2- 172.21.15.1:0/4080276289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f367c15d6b0 0x7f367c0b2da0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f367c0726a0 tx=0x7f3668009b20 comp rx=0 tx=0).stop 2024-03-18T18:51:16.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.643+0000 7f3683249700 1 -- 172.21.15.1:0/4080276289 shutdown_connections 2024-03-18T18:51:16.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.643+0000 7f3683249700 1 --2- 172.21.15.1:0/4080276289 >> v2:172.21.15.165:3300/0 conn(0x7f367c0ad790 0x7f367c0b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.643+0000 7f3683249700 1 --2- 172.21.15.1:0/4080276289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f367c15d6b0 0x7f367c0b2da0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.643+0000 7f3683249700 1 -- 172.21.15.1:0/4080276289 >> 172.21.15.1:0/4080276289 conn(0x7f367c06b860 msgr2=0x7f367c06bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:16.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.643+0000 7f3683249700 1 -- 172.21.15.1:0/4080276289 shutdown_connections 2024-03-18T18:51:16.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.643+0000 7f3683249700 1 -- 172.21.15.1:0/4080276289 wait complete. 2024-03-18T18:51:16.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.644+0000 7f3683249700 1 Processor -- start 2024-03-18T18:51:16.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.644+0000 7f3683249700 1 -- start start 2024-03-18T18:51:16.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f3683249700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f367c06a750 0x7f367c146ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:16.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f3683249700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f367c0ad790 0x7f367c15ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:16.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f3683249700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f367c15d6b0 0x7f367c145470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:16.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f3683249700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c145e30 con 0x7f367c0ad790 2024-03-18T18:51:16.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f3683249700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c146110 con 0x7f367c15d6b0 2024-03-18T18:51:16.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f3683249700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367c1463f0 con 0x7f367c06a750 2024-03-18T18:51:16.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f367bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f367c0ad790 0x7f367c15ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:16.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f367bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f367c0ad790 0x7f367c15ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39760/0 (socket says 172.21.15.1:39760) 2024-03-18T18:51:16.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f367bfff700 1 -- 172.21.15.1:0/2376770258 learned_addr learned my addr 172.21.15.1:0/2376770258 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:16.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f367bfff700 1 -- 172.21.15.1:0/2376770258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f367c06a750 msgr2=0x7f367c146ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:16.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f367bfff700 1 --2- 172.21.15.1:0/2376770258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f367c06a750 0x7f367c146ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f367bfff700 1 -- 172.21.15.1:0/2376770258 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f367c15d6b0 msgr2=0x7f367c145470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:16.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f367bfff700 1 --2- 172.21.15.1:0/2376770258 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f367c15d6b0 0x7f367c145470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f367bfff700 1 -- 172.21.15.1:0/2376770258 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36680094a0 con 0x7f367c0ad790 2024-03-18T18:51:16.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f367bfff700 1 --2- 172.21.15.1:0/2376770258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f367c0ad790 0x7f367c15ed70 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3668000c00 tx=0x7f3668004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:16.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f36797fa700 1 -- 172.21.15.1:0/2376770258 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36680036a0 con 0x7f367c0ad790 2024-03-18T18:51:16.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f36797fa700 1 -- 172.21.15.1:0/2376770258 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366801e070 con 0x7f367c0ad790 2024-03-18T18:51:16.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f3683249700 1 -- 172.21.15.1:0/2376770258 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f367c1466d0 con 0x7f367c0ad790 2024-03-18T18:51:16.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f36797fa700 1 -- 172.21.15.1:0/2376770258 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668003a60 con 0x7f367c0ad790 2024-03-18T18:51:16.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.645+0000 7f3683249700 1 -- 172.21.15.1:0/2376770258 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f367c15f2b0 con 0x7f367c0ad790 2024-03-18T18:51:16.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.646+0000 7f3683249700 1 -- 172.21.15.1:0/2376770258 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f367c15f4a0 con 0x7f367c0ad790 2024-03-18T18:51:16.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.647+0000 7f36797fa700 1 -- 172.21.15.1:0/2376770258 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3668003bc0 con 0x7f367c0ad790 2024-03-18T18:51:16.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.647+0000 7f36797fa700 1 --2- 172.21.15.1:0/2376770258 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f366006ac90 0x7f366006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:16.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.647+0000 7f36797fa700 1 -- 172.21.15.1:0/2376770258 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f366808a4c0 con 0x7f367c0ad790 2024-03-18T18:51:16.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.647+0000 7f3680fe5700 1 --2- 172.21.15.1:0/2376770258 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f366006ac90 0x7f366006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:16.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.648+0000 7f3680fe5700 1 --2- 172.21.15.1:0/2376770258 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f366006ac90 0x7f366006d150 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f36700016b0 tx=0x7f36700079c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:16.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.652+0000 7f36797fa700 1 -- 172.21.15.1:0/2376770258 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3668055550 con 0x7f367c0ad790 2024-03-18T18:51:16.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.849+0000 7f3683249700 1 -- 172.21.15.1:0/2376770258 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f367c146280 con 0x7f366006ac90 2024-03-18T18:51:16.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.858+0000 7f36797fa700 1 -- 172.21.15.1:0/2376770258 <== mgr.4104 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289556 (secure 0 0 0) 0x7f367c146280 con 0x7f366006ac90 2024-03-18T18:51:16.858 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:51:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.861+0000 7f3683249700 1 -- 172.21.15.1:0/2376770258 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f366006ac90 msgr2=0x7f366006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.861+0000 7f3683249700 1 --2- 172.21.15.1:0/2376770258 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f366006ac90 0x7f366006d150 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f36700016b0 tx=0x7f36700079c0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.861+0000 7f3683249700 1 -- 172.21.15.1:0/2376770258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f367c0ad790 msgr2=0x7f367c15ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.861+0000 7f3683249700 1 --2- 172.21.15.1:0/2376770258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f367c0ad790 0x7f367c15ed70 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3668000c00 tx=0x7f3668004750 comp rx=0 tx=0).stop 2024-03-18T18:51:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.861+0000 7f3683249700 1 -- 172.21.15.1:0/2376770258 shutdown_connections 2024-03-18T18:51:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.861+0000 7f3683249700 1 --2- 172.21.15.1:0/2376770258 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f366006ac90 0x7f366006d150 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.861+0000 7f3683249700 1 --2- 172.21.15.1:0/2376770258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f367c06a750 0x7f367c146ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.861+0000 7f3683249700 1 --2- 172.21.15.1:0/2376770258 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f367c15d6b0 0x7f367c145470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.861+0000 7f3683249700 1 --2- 172.21.15.1:0/2376770258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f367c0ad790 0x7f367c15ed70 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.861+0000 7f3683249700 1 -- 172.21.15.1:0/2376770258 >> 172.21.15.1:0/2376770258 conn(0x7f367c06b860 msgr2=0x7f367c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.861+0000 7f3683249700 1 -- 172.21.15.1:0/2376770258 shutdown_connections 2024-03-18T18:51:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:16.861+0000 7f3683249700 1 -- 172.21.15.1:0/2376770258 wait complete. 2024-03-18T18:51:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T18:51:16.870 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-03-18T18:51:15.406414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":59,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220908,"kb_used_data":4708,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753812,"statfs":{"total":773094113280,"available":772867903488,"internally_reserved":0,"allocated":4820992,"data_stored":2707482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.116927"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.755338+0000","last_change":"2024-03-18T18:51:12.751054+0000","last_active":"2024-03-18T18:51:12.755338+0000","last_peered":"2024-03-18T18:51:12.755338+0000","last_clean":"2024-03-18T18:51:12.755338+0000","last_became_active":"2024-03-18T18:51:12.748978+0000","last_became_peered":"2024-03-18T18:51:12.748978+0000","last_unstale":"2024-03-18T18:51:12.755338+0000","last_undegraded":"2024-03-18T18:51:12.755338+0000","last_fullsized":"2024-03-18T18:51:12.755338+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:41:36.474089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.751179+0000","last_change":"2024-03-18T18:51:12.747397+0000","last_active":"2024-03-18T18:51:12.751179+0000","last_peered":"2024-03-18T18:51:12.751179+0000","last_clean":"2024-03-18T18:51:12.751179+0000","last_became_active":"2024-03-18T18:51:12.743998+0000","last_became_peered":"2024-03-18T18:51:12.743998+0000","last_unstale":"2024-03-18T18:51:12.751179+0000","last_undegraded":"2024-03-18T18:51:12.751179+0000","last_fullsized":"2024-03-18T18:51:12.751179+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:08.546546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.756599+0000","last_change":"2024-03-18T18:51:12.753720+0000","last_active":"2024-03-18T18:51:12.756599+0000","last_peered":"2024-03-18T18:51:12.756599+0000","last_clean":"2024-03-18T18:51:12.756599+0000","last_became_active":"2024-03-18T18:51:12.753455+0000","last_became_peered":"2024-03-18T18:51:12.753455+0000","last_unstale":"2024-03-18T18:51:12.756599+0000","last_undegraded":"2024-03-18T18:51:12.756599+0000","last_fullsized":"2024-03-18T18:51:12.756599+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:04.301923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.747489+0000","last_change":"2024-03-18T18:51:12.745544+0000","last_active":"2024-03-18T18:51:12.747489+0000","last_peered":"2024-03-18T18:51:12.747489+0000","last_clean":"2024-03-18T18:51:12.747489+0000","last_became_active":"2024-03-18T18:51:12.745160+0000","last_became_peered":"2024-03-18T18:51:12.745160+0000","last_unstale":"2024-03-18T18:51:12.747489+0000","last_undegraded":"2024-03-18T18:51:12.747489+0000","last_fullsized":"2024-03-18T18:51:12.747489+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:15:59.751511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.747170+0000","last_change":"2024-03-18T18:51:12.744928+0000","last_active":"2024-03-18T18:51:12.747170+0000","last_peered":"2024-03-18T18:51:12.747170+0000","last_clean":"2024-03-18T18:51:12.747170+0000","last_became_active":"2024-03-18T18:51:12.744684+0000","last_became_peered":"2024-03-18T18:51:12.744684+0000","last_unstale":"2024-03-18T18:51:12.747170+0000","last_undegraded":"2024-03-18T18:51:12.747170+0000","last_fullsized":"2024-03-18T18:51:12.747170+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:08:54.946392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.745359+0000","last_change":"2024-03-18T18:51:12.744662+0000","last_active":"2024-03-18T18:51:12.745359+0000","last_peered":"2024-03-18T18:51:12.745359+0000","last_clean":"2024-03-18T18:51:12.745359+0000","last_became_active":"2024-03-18T18:51:12.744389+0000","last_became_peered":"2024-03-18T18:51:12.744389+0000","last_unstale":"2024-03-18T18:51:12.745359+0000","last_undegraded":"2024-03-18T18:51:12.745359+0000","last_fullsized":"2024-03-18T18:51:12.745359+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:28:11.886895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754200+0000","last_change":"2024-03-18T18:51:12.749916+0000","last_active":"2024-03-18T18:51:12.754200+0000","last_peered":"2024-03-18T18:51:12.754200+0000","last_clean":"2024-03-18T18:51:12.754200+0000","last_became_active":"2024-03-18T18:51:12.749228+0000","last_became_peered":"2024-03-18T18:51:12.749228+0000","last_unstale":"2024-03-18T18:51:12.754200+0000","last_undegraded":"2024-03-18T18:51:12.754200+0000","last_fullsized":"2024-03-18T18:51:12.754200+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:59.968694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.756985+0000","last_change":"2024-03-18T18:51:12.756578+0000","last_active":"2024-03-18T18:51:12.756985+0000","last_peered":"2024-03-18T18:51:12.756985+0000","last_clean":"2024-03-18T18:51:12.756985+0000","last_became_active":"2024-03-18T18:51:12.756370+0000","last_became_peered":"2024-03-18T18:51:12.756370+0000","last_unstale":"2024-03-18T18:51:12.756985+0000","last_undegraded":"2024-03-18T18:51:12.756985+0000","last_fullsized":"2024-03-18T18:51:12.756985+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:40:33.676867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.757317+0000","last_change":"2024-03-18T18:51:12.754260+0000","last_active":"2024-03-18T18:51:12.757317+0000","last_peered":"2024-03-18T18:51:12.757317+0000","last_clean":"2024-03-18T18:51:12.757317+0000","last_became_active":"2024-03-18T18:51:12.753993+0000","last_became_peered":"2024-03-18T18:51:12.753993+0000","last_unstale":"2024-03-18T18:51:12.757317+0000","last_undegraded":"2024-03-18T18:51:12.757317+0000","last_fullsized":"2024-03-18T18:51:12.757317+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:02:57.116657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.756344+0000","last_change":"2024-03-18T18:51:12.751746+0000","last_active":"2024-03-18T18:51:12.756344+0000","last_peered":"2024-03-18T18:51:12.756344+0000","last_clean":"2024-03-18T18:51:12.756344+0000","last_became_active":"2024-03-18T18:51:12.751211+0000","last_became_peered":"2024-03-18T18:51:12.751211+0000","last_unstale":"2024-03-18T18:51:12.756344+0000","last_undegraded":"2024-03-18T18:51:12.756344+0000","last_fullsized":"2024-03-18T18:51:12.756344+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:10:01.129869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.753830+0000","last_change":"2024-03-18T18:51:12.749255+0000","last_active":"2024-03-18T18:51:12.753830+0000","last_peered":"2024-03-18T18:51:12.753830+0000","last_clean":"2024-03-18T18:51:12.753830+0000","last_became_active":"2024-03-18T18:51:12.748716+0000","last_became_peered":"2024-03-18T18:51:12.748716+0000","last_unstale":"2024-03-18T18:51:12.753830+0000","last_undegraded":"2024-03-18T18:51:12.753830+0000","last_fullsized":"2024-03-18T18:51:12.753830+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:06:07.180382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.747598+0000","last_change":"2024-03-18T18:51:12.746869+0000","last_active":"2024-03-18T18:51:12.747598+0000","last_peered":"2024-03-18T18:51:12.747598+0000","last_clean":"2024-03-18T18:51:12.747598+0000","last_became_active":"2024-03-18T18:51:12.746600+0000","last_became_peered":"2024-03-18T18:51:12.746600+0000","last_unstale":"2024-03-18T18:51:12.747598+0000","last_undegraded":"2024-03-18T18:51:12.747598+0000","last_fullsized":"2024-03-18T18:51:12.747598+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:06.703485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.756605+0000","last_change":"2024-03-18T18:51:12.750508+0000","last_active":"2024-03-18T18:51:12.756605+0000","last_peered":"2024-03-18T18:51:12.756605+0000","last_clean":"2024-03-18T18:51:12.756605+0000","last_became_active":"2024-03-18T18:51:12.750163+0000","last_became_peered":"2024-03-18T18:51:12.750163+0000","last_unstale":"2024-03-18T18:51:12.756605+0000","last_undegraded":"2024-03-18T18:51:12.756605+0000","last_fullsized":"2024-03-18T18:51:12.756605+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:18:51.530853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.751366+0000","last_change":"2024-03-18T18:51:12.748631+0000","last_active":"2024-03-18T18:51:12.751366+0000","last_peered":"2024-03-18T18:51:12.751366+0000","last_clean":"2024-03-18T18:51:12.751366+0000","last_became_active":"2024-03-18T18:51:12.748341+0000","last_became_peered":"2024-03-18T18:51:12.748341+0000","last_unstale":"2024-03-18T18:51:12.751366+0000","last_undegraded":"2024-03-18T18:51:12.751366+0000","last_fullsized":"2024-03-18T18:51:12.751366+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:59.968694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754217+0000","last_change":"2024-03-18T18:51:12.749953+0000","last_active":"2024-03-18T18:51:12.754217+0000","last_peered":"2024-03-18T18:51:12.754217+0000","last_clean":"2024-03-18T18:51:12.754217+0000","last_became_active":"2024-03-18T18:51:12.749623+0000","last_became_peered":"2024-03-18T18:51:12.749623+0000","last_unstale":"2024-03-18T18:51:12.754217+0000","last_undegraded":"2024-03-18T18:51:12.754217+0000","last_fullsized":"2024-03-18T18:51:12.754217+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:06.703485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754299+0000","last_change":"2024-03-18T18:51:12.751305+0000","last_active":"2024-03-18T18:51:12.754299+0000","last_peered":"2024-03-18T18:51:12.754299+0000","last_clean":"2024-03-18T18:51:12.754299+0000","last_became_active":"2024-03-18T18:51:12.751012+0000","last_became_peered":"2024-03-18T18:51:12.751012+0000","last_unstale":"2024-03-18T18:51:12.754299+0000","last_undegraded":"2024-03-18T18:51:12.754299+0000","last_fullsized":"2024-03-18T18:51:12.754299+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:06:07.180382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754423+0000","last_change":"2024-03-18T18:51:12.751407+0000","last_active":"2024-03-18T18:51:12.754423+0000","last_peered":"2024-03-18T18:51:12.754423+0000","last_clean":"2024-03-18T18:51:12.754423+0000","last_became_active":"2024-03-18T18:51:12.751086+0000","last_became_peered":"2024-03-18T18:51:12.751086+0000","last_unstale":"2024-03-18T18:51:12.754423+0000","last_undegraded":"2024-03-18T18:51:12.754423+0000","last_fullsized":"2024-03-18T18:51:12.754423+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:30:54.407177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.750946+0000","last_change":"2024-03-18T18:51:12.746400+0000","last_active":"2024-03-18T18:51:12.750946+0000","last_peered":"2024-03-18T18:51:12.750946+0000","last_clean":"2024-03-18T18:51:12.750946+0000","last_became_active":"2024-03-18T18:51:12.746122+0000","last_became_peered":"2024-03-18T18:51:12.746122+0000","last_unstale":"2024-03-18T18:51:12.750946+0000","last_undegraded":"2024-03-18T18:51:12.750946+0000","last_fullsized":"2024-03-18T18:51:12.750946+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:10.811679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.744254+0000","last_change":"2024-03-18T18:51:12.738346+0000","last_active":"2024-03-18T18:51:12.744254+0000","last_peered":"2024-03-18T18:51:12.744254+0000","last_clean":"2024-03-18T18:51:12.744254+0000","last_became_active":"2024-03-18T18:51:12.738085+0000","last_became_peered":"2024-03-18T18:51:12.738085+0000","last_unstale":"2024-03-18T18:51:12.744254+0000","last_undegraded":"2024-03-18T18:51:12.744254+0000","last_fullsized":"2024-03-18T18:51:12.744254+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:56:23.719374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.750147+0000","last_change":"2024-03-18T18:51:12.744825+0000","last_active":"2024-03-18T18:51:12.750147+0000","last_peered":"2024-03-18T18:51:12.750147+0000","last_clean":"2024-03-18T18:51:12.750147+0000","last_became_active":"2024-03-18T18:51:12.744577+0000","last_became_peered":"2024-03-18T18:51:12.744577+0000","last_unstale":"2024-03-18T18:51:12.750147+0000","last_undegraded":"2024-03-18T18:51:12.750147+0000","last_fullsized":"2024-03-18T18:51:12.750147+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:20:54.202383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.759249+0000","last_change":"2024-03-18T18:51:12.754171+0000","last_active":"2024-03-18T18:51:12.759249+0000","last_peered":"2024-03-18T18:51:12.759249+0000","last_clean":"2024-03-18T18:51:12.759249+0000","last_became_active":"2024-03-18T18:51:12.753535+0000","last_became_peered":"2024-03-18T18:51:12.753535+0000","last_unstale":"2024-03-18T18:51:12.759249+0000","last_undegraded":"2024-03-18T18:51:12.759249+0000","last_fullsized":"2024-03-18T18:51:12.759249+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:43:46.861441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.751386+0000","last_change":"2024-03-18T18:51:12.749000+0000","last_active":"2024-03-18T18:51:12.751386+0000","last_peered":"2024-03-18T18:51:12.751386+0000","last_clean":"2024-03-18T18:51:12.751386+0000","last_became_active":"2024-03-18T18:51:12.748034+0000","last_became_peered":"2024-03-18T18:51:12.748034+0000","last_unstale":"2024-03-18T18:51:12.751386+0000","last_undegraded":"2024-03-18T18:51:12.751386+0000","last_fullsized":"2024-03-18T18:51:12.751386+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:21:13.902213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.744715+0000","last_change":"2024-03-18T18:51:12.741944+0000","last_active":"2024-03-18T18:51:12.744715+0000","last_peered":"2024-03-18T18:51:12.744715+0000","last_clean":"2024-03-18T18:51:12.744715+0000","last_became_active":"2024-03-18T18:51:12.738039+0000","last_became_peered":"2024-03-18T18:51:12.738039+0000","last_unstale":"2024-03-18T18:51:12.744715+0000","last_undegraded":"2024-03-18T18:51:12.744715+0000","last_fullsized":"2024-03-18T18:51:12.744715+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:15.086687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.746198+0000","last_change":"2024-03-18T18:51:12.740508+0000","last_active":"2024-03-18T18:51:12.746198+0000","last_peered":"2024-03-18T18:51:12.746198+0000","last_clean":"2024-03-18T18:51:12.746198+0000","last_became_active":"2024-03-18T18:51:12.740257+0000","last_became_peered":"2024-03-18T18:51:12.740257+0000","last_unstale":"2024-03-18T18:51:12.746198+0000","last_undegraded":"2024-03-18T18:51:12.746198+0000","last_fullsized":"2024-03-18T18:51:12.746198+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:20:54.202383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.750716+0000","last_change":"2024-03-18T18:51:12.745721+0000","last_active":"2024-03-18T18:51:12.750716+0000","last_peered":"2024-03-18T18:51:12.750716+0000","last_clean":"2024-03-18T18:51:12.750716+0000","last_became_active":"2024-03-18T18:51:12.745201+0000","last_became_peered":"2024-03-18T18:51:12.745201+0000","last_unstale":"2024-03-18T18:51:12.750716+0000","last_undegraded":"2024-03-18T18:51:12.750716+0000","last_fullsized":"2024-03-18T18:51:12.750716+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:15.086687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754644+0000","last_change":"2024-03-18T18:51:12.752339+0000","last_active":"2024-03-18T18:51:12.754644+0000","last_peered":"2024-03-18T18:51:12.754644+0000","last_clean":"2024-03-18T18:51:12.754644+0000","last_became_active":"2024-03-18T18:51:12.752053+0000","last_became_peered":"2024-03-18T18:51:12.752053+0000","last_unstale":"2024-03-18T18:51:12.754644+0000","last_undegraded":"2024-03-18T18:51:12.754644+0000","last_fullsized":"2024-03-18T18:51:12.754644+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:06.703485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.746157+0000","last_change":"2024-03-18T18:51:12.737443+0000","last_active":"2024-03-18T18:51:12.746157+0000","last_peered":"2024-03-18T18:51:12.746157+0000","last_clean":"2024-03-18T18:51:12.746157+0000","last_became_active":"2024-03-18T18:51:12.737179+0000","last_became_peered":"2024-03-18T18:51:12.737179+0000","last_unstale":"2024-03-18T18:51:12.746157+0000","last_undegraded":"2024-03-18T18:51:12.746157+0000","last_fullsized":"2024-03-18T18:51:12.746157+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:56:07.670601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754027+0000","last_change":"2024-03-18T18:51:12.748803+0000","last_active":"2024-03-18T18:51:12.754027+0000","last_peered":"2024-03-18T18:51:12.754027+0000","last_clean":"2024-03-18T18:51:12.754027+0000","last_became_active":"2024-03-18T18:51:12.748501+0000","last_became_peered":"2024-03-18T18:51:12.748501+0000","last_unstale":"2024-03-18T18:51:12.754027+0000","last_undegraded":"2024-03-18T18:51:12.754027+0000","last_fullsized":"2024-03-18T18:51:12.754027+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:06:26.805929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.746437+0000","last_change":"2024-03-18T18:51:12.736150+0000","last_active":"2024-03-18T18:51:12.746437+0000","last_peered":"2024-03-18T18:51:12.746437+0000","last_clean":"2024-03-18T18:51:12.746437+0000","last_became_active":"2024-03-18T18:51:12.735896+0000","last_became_peered":"2024-03-18T18:51:12.735896+0000","last_unstale":"2024-03-18T18:51:12.746437+0000","last_undegraded":"2024-03-18T18:51:12.746437+0000","last_fullsized":"2024-03-18T18:51:12.746437+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:56:23.719374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.751558+0000","last_change":"2024-03-18T18:51:12.748729+0000","last_active":"2024-03-18T18:51:12.751558+0000","last_peered":"2024-03-18T18:51:12.751558+0000","last_clean":"2024-03-18T18:51:12.751558+0000","last_became_active":"2024-03-18T18:51:12.748480+0000","last_became_peered":"2024-03-18T18:51:12.748480+0000","last_unstale":"2024-03-18T18:51:12.751558+0000","last_undegraded":"2024-03-18T18:51:12.751558+0000","last_fullsized":"2024-03-18T18:51:12.751558+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:11:17.657760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.744696+0000","last_change":"2024-03-18T18:51:12.738308+0000","last_active":"2024-03-18T18:51:12.744696+0000","last_peered":"2024-03-18T18:51:12.744696+0000","last_clean":"2024-03-18T18:51:12.744696+0000","last_became_active":"2024-03-18T18:51:12.736012+0000","last_became_peered":"2024-03-18T18:51:12.736012+0000","last_unstale":"2024-03-18T18:51:12.744696+0000","last_undegraded":"2024-03-18T18:51:12.744696+0000","last_fullsized":"2024-03-18T18:51:12.744696+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:10.811679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.743418+0000","last_change":"2024-03-18T18:51:12.740850+0000","last_active":"2024-03-18T18:51:12.743418+0000","last_peered":"2024-03-18T18:51:12.743418+0000","last_clean":"2024-03-18T18:51:12.743418+0000","last_became_active":"2024-03-18T18:51:12.740595+0000","last_became_peered":"2024-03-18T18:51:12.740595+0000","last_unstale":"2024-03-18T18:51:12.743418+0000","last_undegraded":"2024-03-18T18:51:12.743418+0000","last_fullsized":"2024-03-18T18:51:12.743418+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:37:58.205820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754405+0000","last_change":"2024-03-18T18:51:12.753087+0000","last_active":"2024-03-18T18:51:12.754405+0000","last_peered":"2024-03-18T18:51:12.754405+0000","last_clean":"2024-03-18T18:51:12.754405+0000","last_became_active":"2024-03-18T18:51:12.752833+0000","last_became_peered":"2024-03-18T18:51:12.752833+0000","last_unstale":"2024-03-18T18:51:12.754405+0000","last_undegraded":"2024-03-18T18:51:12.754405+0000","last_fullsized":"2024-03-18T18:51:12.754405+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:19:35.490918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"39'4","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.742189+0000","last_change":"2024-03-18T18:51:12.307632+0000","last_active":"2024-03-18T18:51:12.742189+0000","last_peered":"2024-03-18T18:51:12.742189+0000","last_clean":"2024-03-18T18:51:12.742189+0000","last_became_active":"2024-03-18T18:51:12.307141+0000","last_became_peered":"2024-03-18T18:51:12.307141+0000","last_unstale":"2024-03-18T18:51:12.742189+0000","last_undegraded":"2024-03-18T18:51:12.742189+0000","last_fullsized":"2024-03-18T18:51:12.742189+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:13:18.483310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754700+0000","last_change":"2024-03-18T18:51:12.752712+0000","last_active":"2024-03-18T18:51:12.754700+0000","last_peered":"2024-03-18T18:51:12.754700+0000","last_clean":"2024-03-18T18:51:12.754700+0000","last_became_active":"2024-03-18T18:51:12.752460+0000","last_became_peered":"2024-03-18T18:51:12.752460+0000","last_unstale":"2024-03-18T18:51:12.754700+0000","last_undegraded":"2024-03-18T18:51:12.754700+0000","last_fullsized":"2024-03-18T18:51:12.754700+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:59.968694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"39'1","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.745121+0000","last_change":"2024-03-18T18:51:12.318280+0000","last_active":"2024-03-18T18:51:12.745121+0000","last_peered":"2024-03-18T18:51:12.745121+0000","last_clean":"2024-03-18T18:51:12.745121+0000","last_became_active":"2024-03-18T18:51:12.317960+0000","last_became_peered":"2024-03-18T18:51:12.317960+0000","last_unstale":"2024-03-18T18:51:12.745121+0000","last_undegraded":"2024-03-18T18:51:12.745121+0000","last_fullsized":"2024-03-18T18:51:12.745121+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:20:21.587872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.749131+0000","last_change":"2024-03-18T18:51:12.746107+0000","last_active":"2024-03-18T18:51:12.749131+0000","last_peered":"2024-03-18T18:51:12.749131+0000","last_clean":"2024-03-18T18:51:12.749131+0000","last_became_active":"2024-03-18T18:51:12.745656+0000","last_became_peered":"2024-03-18T18:51:12.745656+0000","last_unstale":"2024-03-18T18:51:12.749131+0000","last_undegraded":"2024-03-18T18:51:12.749131+0000","last_fullsized":"2024-03-18T18:51:12.749131+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:34:03.497588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"39'2","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.744448+0000","last_change":"2024-03-18T18:51:12.312353+0000","last_active":"2024-03-18T18:51:12.744448+0000","last_peered":"2024-03-18T18:51:12.744448+0000","last_clean":"2024-03-18T18:51:12.744448+0000","last_became_active":"2024-03-18T18:51:12.311888+0000","last_became_peered":"2024-03-18T18:51:12.311888+0000","last_unstale":"2024-03-18T18:51:12.744448+0000","last_undegraded":"2024-03-18T18:51:12.744448+0000","last_fullsized":"2024-03-18T18:51:12.744448+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:11:16.652877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.755328+0000","last_change":"2024-03-18T18:51:12.751027+0000","last_active":"2024-03-18T18:51:12.755328+0000","last_peered":"2024-03-18T18:51:12.755328+0000","last_clean":"2024-03-18T18:51:12.755328+0000","last_became_active":"2024-03-18T18:51:12.748500+0000","last_became_peered":"2024-03-18T18:51:12.748500+0000","last_unstale":"2024-03-18T18:51:12.755328+0000","last_undegraded":"2024-03-18T18:51:12.755328+0000","last_fullsized":"2024-03-18T18:51:12.755328+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:40:07.068740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"39'2","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.743162+0000","last_change":"2024-03-18T18:51:12.311843+0000","last_active":"2024-03-18T18:51:12.743162+0000","last_peered":"2024-03-18T18:51:12.743162+0000","last_clean":"2024-03-18T18:51:12.743162+0000","last_became_active":"2024-03-18T18:51:12.311507+0000","last_became_peered":"2024-03-18T18:51:12.311507+0000","last_unstale":"2024-03-18T18:51:12.743162+0000","last_undegraded":"2024-03-18T18:51:12.743162+0000","last_fullsized":"2024-03-18T18:51:12.743162+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:19:34.486035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.745440+0000","last_change":"2024-03-18T18:51:12.742400+0000","last_active":"2024-03-18T18:51:12.745440+0000","last_peered":"2024-03-18T18:51:12.745440+0000","last_clean":"2024-03-18T18:51:12.745440+0000","last_became_active":"2024-03-18T18:51:12.742159+0000","last_became_peered":"2024-03-18T18:51:12.742159+0000","last_unstale":"2024-03-18T18:51:12.745440+0000","last_undegraded":"2024-03-18T18:51:12.745440+0000","last_fullsized":"2024-03-18T18:51:12.745440+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:20.862960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"39'1","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.752544+0000","last_change":"2024-03-18T18:51:12.314920+0000","last_active":"2024-03-18T18:51:12.752544+0000","last_peered":"2024-03-18T18:51:12.752544+0000","last_clean":"2024-03-18T18:51:12.752544+0000","last_became_active":"2024-03-18T18:51:12.314633+0000","last_became_peered":"2024-03-18T18:51:12.314633+0000","last_unstale":"2024-03-18T18:51:12.752544+0000","last_undegraded":"2024-03-18T18:51:12.752544+0000","last_fullsized":"2024-03-18T18:51:12.752544+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:21:33.793911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754231+0000","last_change":"2024-03-18T18:51:12.750081+0000","last_active":"2024-03-18T18:51:12.754231+0000","last_peered":"2024-03-18T18:51:12.754231+0000","last_clean":"2024-03-18T18:51:12.754231+0000","last_became_active":"2024-03-18T18:51:12.749654+0000","last_became_peered":"2024-03-18T18:51:12.749654+0000","last_unstale":"2024-03-18T18:51:12.754231+0000","last_undegraded":"2024-03-18T18:51:12.754231+0000","last_fullsized":"2024-03-18T18:51:12.754231+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:51.202259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"39'1","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.747326+0000","last_change":"2024-03-18T18:51:12.319821+0000","last_active":"2024-03-18T18:51:12.747326+0000","last_peered":"2024-03-18T18:51:12.747326+0000","last_clean":"2024-03-18T18:51:12.747326+0000","last_became_active":"2024-03-18T18:51:12.319558+0000","last_became_peered":"2024-03-18T18:51:12.319558+0000","last_unstale":"2024-03-18T18:51:12.747326+0000","last_undegraded":"2024-03-18T18:51:12.747326+0000","last_fullsized":"2024-03-18T18:51:12.747326+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:22:06.101192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.759166+0000","last_change":"2024-03-18T18:51:12.752230+0000","last_active":"2024-03-18T18:51:12.759166+0000","last_peered":"2024-03-18T18:51:12.759166+0000","last_clean":"2024-03-18T18:51:12.759166+0000","last_became_active":"2024-03-18T18:51:12.751960+0000","last_became_peered":"2024-03-18T18:51:12.751960+0000","last_unstale":"2024-03-18T18:51:12.759166+0000","last_undegraded":"2024-03-18T18:51:12.759166+0000","last_fullsized":"2024-03-18T18:51:12.759166+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:17:06.269383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.728547+0000","last_change":"2024-03-18T18:51:12.318058+0000","last_active":"2024-03-18T18:51:12.728547+0000","last_peered":"2024-03-18T18:51:12.728547+0000","last_clean":"2024-03-18T18:51:12.728547+0000","last_became_active":"2024-03-18T18:51:12.317737+0000","last_became_peered":"2024-03-18T18:51:12.317737+0000","last_unstale":"2024-03-18T18:51:12.728547+0000","last_undegraded":"2024-03-18T18:51:12.728547+0000","last_fullsized":"2024-03-18T18:51:12.728547+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:09.806796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.755645+0000","last_change":"2024-03-18T18:51:12.751554+0000","last_active":"2024-03-18T18:51:12.755645+0000","last_peered":"2024-03-18T18:51:12.755645+0000","last_clean":"2024-03-18T18:51:12.755645+0000","last_became_active":"2024-03-18T18:51:12.748608+0000","last_became_peered":"2024-03-18T18:51:12.748608+0000","last_unstale":"2024-03-18T18:51:12.755645+0000","last_undegraded":"2024-03-18T18:51:12.755645+0000","last_fullsized":"2024-03-18T18:51:12.755645+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:55:21.585205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.737360+0000","last_change":"2024-03-18T18:51:12.314356+0000","last_active":"2024-03-18T18:51:12.737360+0000","last_peered":"2024-03-18T18:51:12.737360+0000","last_clean":"2024-03-18T18:51:12.737360+0000","last_became_active":"2024-03-18T18:51:12.314068+0000","last_became_peered":"2024-03-18T18:51:12.314068+0000","last_unstale":"2024-03-18T18:51:12.737360+0000","last_undegraded":"2024-03-18T18:51:12.737360+0000","last_fullsized":"2024-03-18T18:51:12.737360+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:06:25.801047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.747470+0000","last_change":"2024-03-18T18:51:12.745394+0000","last_active":"2024-03-18T18:51:12.747470+0000","last_peered":"2024-03-18T18:51:12.747470+0000","last_clean":"2024-03-18T18:51:12.747470+0000","last_became_active":"2024-03-18T18:51:12.744822+0000","last_became_peered":"2024-03-18T18:51:12.744822+0000","last_unstale":"2024-03-18T18:51:12.747470+0000","last_undegraded":"2024-03-18T18:51:12.747470+0000","last_fullsized":"2024-03-18T18:51:12.747470+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:45.519051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.735519+0000","last_change":"2024-03-18T18:51:12.315258+0000","last_active":"2024-03-18T18:51:12.735519+0000","last_peered":"2024-03-18T18:51:12.735519+0000","last_clean":"2024-03-18T18:51:12.735519+0000","last_became_active":"2024-03-18T18:51:12.314955+0000","last_became_peered":"2024-03-18T18:51:12.314955+0000","last_unstale":"2024-03-18T18:51:12.735519+0000","last_undegraded":"2024-03-18T18:51:12.735519+0000","last_fullsized":"2024-03-18T18:51:12.735519+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:37:57.200937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.742125+0000","last_change":"2024-03-18T18:51:12.737828+0000","last_active":"2024-03-18T18:51:12.742125+0000","last_peered":"2024-03-18T18:51:12.742125+0000","last_clean":"2024-03-18T18:51:12.742125+0000","last_became_active":"2024-03-18T18:51:12.737164+0000","last_became_peered":"2024-03-18T18:51:12.737164+0000","last_unstale":"2024-03-18T18:51:12.742125+0000","last_undegraded":"2024-03-18T18:51:12.742125+0000","last_fullsized":"2024-03-18T18:51:12.742125+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:20:22.592755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.744411+0000","last_change":"2024-03-18T18:51:12.315010+0000","last_active":"2024-03-18T18:51:12.744411+0000","last_peered":"2024-03-18T18:51:12.744411+0000","last_clean":"2024-03-18T18:51:12.744411+0000","last_became_active":"2024-03-18T18:51:12.314735+0000","last_became_peered":"2024-03-18T18:51:12.314735+0000","last_unstale":"2024-03-18T18:51:12.744411+0000","last_undegraded":"2024-03-18T18:51:12.744411+0000","last_fullsized":"2024-03-18T18:51:12.744411+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:08:53.941509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.746799+0000","last_change":"2024-03-18T18:51:12.745640+0000","last_active":"2024-03-18T18:51:12.746799+0000","last_peered":"2024-03-18T18:51:12.746799+0000","last_clean":"2024-03-18T18:51:12.746799+0000","last_became_active":"2024-03-18T18:51:12.745389+0000","last_became_peered":"2024-03-18T18:51:12.745389+0000","last_unstale":"2024-03-18T18:51:12.746799+0000","last_undegraded":"2024-03-18T18:51:12.746799+0000","last_fullsized":"2024-03-18T18:51:12.746799+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:21:13.902213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"39'3","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754826+0000","last_change":"2024-03-18T18:51:12.316395+0000","last_active":"2024-03-18T18:51:12.754826+0000","last_peered":"2024-03-18T18:51:12.754826+0000","last_clean":"2024-03-18T18:51:12.754826+0000","last_became_active":"2024-03-18T18:51:12.316109+0000","last_became_peered":"2024-03-18T18:51:12.316109+0000","last_unstale":"2024-03-18T18:51:12.754826+0000","last_undegraded":"2024-03-18T18:51:12.754826+0000","last_fullsized":"2024-03-18T18:51:12.754826+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:15:58.746628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.747263+0000","last_change":"2024-03-18T18:51:12.741630+0000","last_active":"2024-03-18T18:51:12.747263+0000","last_peered":"2024-03-18T18:51:12.747263+0000","last_clean":"2024-03-18T18:51:12.747263+0000","last_became_active":"2024-03-18T18:51:12.741356+0000","last_became_peered":"2024-03-18T18:51:12.741356+0000","last_unstale":"2024-03-18T18:51:12.747263+0000","last_undegraded":"2024-03-18T18:51:12.747263+0000","last_fullsized":"2024-03-18T18:51:12.747263+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:28:36.629231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"39'6","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.762165+0000","last_change":"2024-03-18T18:51:12.307177+0000","last_active":"2024-03-18T18:51:12.762165+0000","last_peered":"2024-03-18T18:51:12.762165+0000","last_clean":"2024-03-18T18:51:12.762165+0000","last_became_active":"2024-03-18T18:51:12.306931+0000","last_became_peered":"2024-03-18T18:51:12.306931+0000","last_unstale":"2024-03-18T18:51:12.762165+0000","last_undegraded":"2024-03-18T18:51:12.762165+0000","last_fullsized":"2024-03-18T18:51:12.762165+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:18:50.525971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.757305+0000","last_change":"2024-03-18T18:51:12.753671+0000","last_active":"2024-03-18T18:51:12.757305+0000","last_peered":"2024-03-18T18:51:12.757305+0000","last_clean":"2024-03-18T18:51:12.757305+0000","last_became_active":"2024-03-18T18:51:12.753379+0000","last_became_peered":"2024-03-18T18:51:12.753379+0000","last_unstale":"2024-03-18T18:51:12.757305+0000","last_undegraded":"2024-03-18T18:51:12.757305+0000","last_fullsized":"2024-03-18T18:51:12.757305+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:13.711567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"39'2","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.751108+0000","last_change":"2024-03-18T18:51:12.313397+0000","last_active":"2024-03-18T18:51:12.751108+0000","last_peered":"2024-03-18T18:51:12.751108+0000","last_clean":"2024-03-18T18:51:12.751108+0000","last_became_active":"2024-03-18T18:51:12.313119+0000","last_became_peered":"2024-03-18T18:51:12.313119+0000","last_unstale":"2024-03-18T18:51:12.751108+0000","last_undegraded":"2024-03-18T18:51:12.751108+0000","last_fullsized":"2024-03-18T18:51:12.751108+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:34:02.492705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.756494+0000","last_change":"2024-03-18T18:51:12.748676+0000","last_active":"2024-03-18T18:51:12.756494+0000","last_peered":"2024-03-18T18:51:12.756494+0000","last_clean":"2024-03-18T18:51:12.756494+0000","last_became_active":"2024-03-18T18:51:12.748406+0000","last_became_peered":"2024-03-18T18:51:12.748406+0000","last_unstale":"2024-03-18T18:51:12.756494+0000","last_undegraded":"2024-03-18T18:51:12.756494+0000","last_fullsized":"2024-03-18T18:51:12.756494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:24:46.941173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.728255+0000","last_change":"2024-03-18T18:51:12.314385+0000","last_active":"2024-03-18T18:51:12.728255+0000","last_peered":"2024-03-18T18:51:12.728255+0000","last_clean":"2024-03-18T18:51:12.728255+0000","last_became_active":"2024-03-18T18:51:12.314115+0000","last_became_peered":"2024-03-18T18:51:12.314115+0000","last_unstale":"2024-03-18T18:51:12.728255+0000","last_undegraded":"2024-03-18T18:51:12.728255+0000","last_fullsized":"2024-03-18T18:51:12.728255+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:20:53.197501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.744606+0000","last_change":"2024-03-18T18:51:12.738187+0000","last_active":"2024-03-18T18:51:12.744606+0000","last_peered":"2024-03-18T18:51:12.744606+0000","last_clean":"2024-03-18T18:51:12.744606+0000","last_became_active":"2024-03-18T18:51:12.735956+0000","last_became_peered":"2024-03-18T18:51:12.735956+0000","last_unstale":"2024-03-18T18:51:12.744606+0000","last_undegraded":"2024-03-18T18:51:12.744606+0000","last_fullsized":"2024-03-18T18:51:12.744606+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:11:17.657760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.727450+0000","last_change":"2024-03-18T18:51:12.307067+0000","last_active":"2024-03-18T18:51:12.727450+0000","last_peered":"2024-03-18T18:51:12.727450+0000","last_clean":"2024-03-18T18:51:12.727450+0000","last_became_active":"2024-03-18T18:51:12.306786+0000","last_became_peered":"2024-03-18T18:51:12.306786+0000","last_unstale":"2024-03-18T18:51:12.727450+0000","last_undegraded":"2024-03-18T18:51:12.727450+0000","last_fullsized":"2024-03-18T18:51:12.727450+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:55:02.518147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.755662+0000","last_change":"2024-03-18T18:51:12.751564+0000","last_active":"2024-03-18T18:51:12.755662+0000","last_peered":"2024-03-18T18:51:12.755662+0000","last_clean":"2024-03-18T18:51:12.755662+0000","last_became_active":"2024-03-18T18:51:12.748991+0000","last_became_peered":"2024-03-18T18:51:12.748991+0000","last_unstale":"2024-03-18T18:51:12.755662+0000","last_undegraded":"2024-03-18T18:51:12.755662+0000","last_fullsized":"2024-03-18T18:51:12.755662+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:49:47.690564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"39'8","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.747311+0000","last_change":"2024-03-18T18:51:12.319791+0000","last_active":"2024-03-18T18:51:12.747311+0000","last_peered":"2024-03-18T18:51:12.747311+0000","last_clean":"2024-03-18T18:51:12.747311+0000","last_became_active":"2024-03-18T18:51:12.319481+0000","last_became_peered":"2024-03-18T18:51:12.319481+0000","last_unstale":"2024-03-18T18:51:12.747311+0000","last_undegraded":"2024-03-18T18:51:12.747311+0000","last_fullsized":"2024-03-18T18:51:12.747311+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:34.624046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.747891+0000","last_change":"2024-03-18T18:51:12.742362+0000","last_active":"2024-03-18T18:51:12.747891+0000","last_peered":"2024-03-18T18:51:12.747891+0000","last_clean":"2024-03-18T18:51:12.747891+0000","last_became_active":"2024-03-18T18:51:12.742051+0000","last_became_peered":"2024-03-18T18:51:12.742051+0000","last_unstale":"2024-03-18T18:51:12.747891+0000","last_undegraded":"2024-03-18T18:51:12.747891+0000","last_fullsized":"2024-03-18T18:51:12.747891+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:29:42.322290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"39'1","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.752937+0000","last_change":"2024-03-18T18:51:12.316428+0000","last_active":"2024-03-18T18:51:12.752937+0000","last_peered":"2024-03-18T18:51:12.752937+0000","last_clean":"2024-03-18T18:51:12.752937+0000","last_became_active":"2024-03-18T18:51:12.316013+0000","last_became_peered":"2024-03-18T18:51:12.316013+0000","last_unstale":"2024-03-18T18:51:12.752937+0000","last_undegraded":"2024-03-18T18:51:12.752937+0000","last_fullsized":"2024-03-18T18:51:12.752937+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:09:54.723546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754308+0000","last_change":"2024-03-18T18:51:12.749472+0000","last_active":"2024-03-18T18:51:12.754308+0000","last_peered":"2024-03-18T18:51:12.754308+0000","last_clean":"2024-03-18T18:51:12.754308+0000","last_became_active":"2024-03-18T18:51:12.748996+0000","last_became_peered":"2024-03-18T18:51:12.748996+0000","last_unstale":"2024-03-18T18:51:12.754308+0000","last_undegraded":"2024-03-18T18:51:12.754308+0000","last_fullsized":"2024-03-18T18:51:12.754308+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:21:13.902213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"39'2","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.752248+0000","last_change":"2024-03-18T18:51:12.316517+0000","last_active":"2024-03-18T18:51:12.752248+0000","last_peered":"2024-03-18T18:51:12.752248+0000","last_clean":"2024-03-18T18:51:12.752248+0000","last_became_active":"2024-03-18T18:51:12.316244+0000","last_became_peered":"2024-03-18T18:51:12.316244+0000","last_unstale":"2024-03-18T18:51:12.752248+0000","last_undegraded":"2024-03-18T18:51:12.752248+0000","last_fullsized":"2024-03-18T18:51:12.752248+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:11:16.652877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.749113+0000","last_change":"2024-03-18T18:51:12.745916+0000","last_active":"2024-03-18T18:51:12.749113+0000","last_peered":"2024-03-18T18:51:12.749113+0000","last_clean":"2024-03-18T18:51:12.749113+0000","last_became_active":"2024-03-18T18:51:12.745199+0000","last_became_peered":"2024-03-18T18:51:12.745199+0000","last_unstale":"2024-03-18T18:51:12.749113+0000","last_undegraded":"2024-03-18T18:51:12.749113+0000","last_fullsized":"2024-03-18T18:51:12.749113+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:10:01.129869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"39'1","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.743105+0000","last_change":"2024-03-18T18:51:12.308635+0000","last_active":"2024-03-18T18:51:12.743105+0000","last_peered":"2024-03-18T18:51:12.743105+0000","last_clean":"2024-03-18T18:51:12.743105+0000","last_became_active":"2024-03-18T18:51:12.308357+0000","last_became_peered":"2024-03-18T18:51:12.308357+0000","last_unstale":"2024-03-18T18:51:12.743105+0000","last_undegraded":"2024-03-18T18:51:12.743105+0000","last_fullsized":"2024-03-18T18:51:12.743105+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:27:08.916844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.747637+0000","last_change":"2024-03-18T18:51:12.743998+0000","last_active":"2024-03-18T18:51:12.747637+0000","last_peered":"2024-03-18T18:51:12.747637+0000","last_clean":"2024-03-18T18:51:12.747637+0000","last_became_active":"2024-03-18T18:51:12.743467+0000","last_became_peered":"2024-03-18T18:51:12.743467+0000","last_unstale":"2024-03-18T18:51:12.747637+0000","last_undegraded":"2024-03-18T18:51:12.747637+0000","last_fullsized":"2024-03-18T18:51:12.747637+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:56:07.670601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.728651+0000","last_change":"2024-03-18T18:51:12.311242+0000","last_active":"2024-03-18T18:51:12.728651+0000","last_peered":"2024-03-18T18:51:12.728651+0000","last_clean":"2024-03-18T18:51:12.728651+0000","last_became_active":"2024-03-18T18:51:12.310956+0000","last_became_peered":"2024-03-18T18:51:12.310956+0000","last_unstale":"2024-03-18T18:51:12.728651+0000","last_undegraded":"2024-03-18T18:51:12.728651+0000","last_fullsized":"2024-03-18T18:51:12.728651+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:14.081804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.744895+0000","last_change":"2024-03-18T18:51:12.742697+0000","last_active":"2024-03-18T18:51:12.744895+0000","last_peered":"2024-03-18T18:51:12.744895+0000","last_clean":"2024-03-18T18:51:12.744895+0000","last_became_active":"2024-03-18T18:51:12.742081+0000","last_became_peered":"2024-03-18T18:51:12.742081+0000","last_unstale":"2024-03-18T18:51:12.744895+0000","last_undegraded":"2024-03-18T18:51:12.744895+0000","last_fullsized":"2024-03-18T18:51:12.744895+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:06:26.805929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"39'2","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.741063+0000","last_change":"2024-03-18T18:51:12.312411+0000","last_active":"2024-03-18T18:51:12.741063+0000","last_peered":"2024-03-18T18:51:12.741063+0000","last_clean":"2024-03-18T18:51:12.741063+0000","last_became_active":"2024-03-18T18:51:12.312111+0000","last_became_peered":"2024-03-18T18:51:12.312111+0000","last_unstale":"2024-03-18T18:51:12.741063+0000","last_undegraded":"2024-03-18T18:51:12.741063+0000","last_fullsized":"2024-03-18T18:51:12.741063+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:49.708013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.757116+0000","last_change":"2024-03-18T18:51:12.753947+0000","last_active":"2024-03-18T18:51:12.757116+0000","last_peered":"2024-03-18T18:51:12.757116+0000","last_clean":"2024-03-18T18:51:12.757116+0000","last_became_active":"2024-03-18T18:51:12.753759+0000","last_became_peered":"2024-03-18T18:51:12.753759+0000","last_unstale":"2024-03-18T18:51:12.757116+0000","last_undegraded":"2024-03-18T18:51:12.757116+0000","last_fullsized":"2024-03-18T18:51:12.757116+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:51.202259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.731489+0000","last_change":"2024-03-18T18:51:12.307869+0000","last_active":"2024-03-18T18:51:12.731489+0000","last_peered":"2024-03-18T18:51:12.731489+0000","last_clean":"2024-03-18T18:51:12.731489+0000","last_became_active":"2024-03-18T18:51:12.307554+0000","last_became_peered":"2024-03-18T18:51:12.307554+0000","last_unstale":"2024-03-18T18:51:12.731489+0000","last_undegraded":"2024-03-18T18:51:12.731489+0000","last_fullsized":"2024-03-18T18:51:12.731489+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:00:31.640644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.752304+0000","last_change":"2024-03-18T18:51:12.747250+0000","last_active":"2024-03-18T18:51:12.752304+0000","last_peered":"2024-03-18T18:51:12.752304+0000","last_clean":"2024-03-18T18:51:12.752304+0000","last_became_active":"2024-03-18T18:51:12.746868+0000","last_became_peered":"2024-03-18T18:51:12.746868+0000","last_unstale":"2024-03-18T18:51:12.752304+0000","last_undegraded":"2024-03-18T18:51:12.752304+0000","last_fullsized":"2024-03-18T18:51:12.752304+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:28.851313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.731021+0000","last_change":"2024-03-18T18:51:12.318504+0000","last_active":"2024-03-18T18:51:12.731021+0000","last_peered":"2024-03-18T18:51:12.731021+0000","last_clean":"2024-03-18T18:51:12.731021+0000","last_became_active":"2024-03-18T18:51:12.318165+0000","last_became_peered":"2024-03-18T18:51:12.318165+0000","last_unstale":"2024-03-18T18:51:12.731021+0000","last_undegraded":"2024-03-18T18:51:12.731021+0000","last_fullsized":"2024-03-18T18:51:12.731021+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:58:16.424096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.755374+0000","last_change":"2024-03-18T18:51:12.751138+0000","last_active":"2024-03-18T18:51:12.755374+0000","last_peered":"2024-03-18T18:51:12.755374+0000","last_clean":"2024-03-18T18:51:12.755374+0000","last_became_active":"2024-03-18T18:51:12.748240+0000","last_became_peered":"2024-03-18T18:51:12.748240+0000","last_unstale":"2024-03-18T18:51:12.755374+0000","last_undegraded":"2024-03-18T18:51:12.755374+0000","last_fullsized":"2024-03-18T18:51:12.755374+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:10.892017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.730758+0000","last_change":"2024-03-18T18:51:12.310709+0000","last_active":"2024-03-18T18:51:12.730758+0000","last_peered":"2024-03-18T18:51:12.730758+0000","last_clean":"2024-03-18T18:51:12.730758+0000","last_became_active":"2024-03-18T18:51:12.310435+0000","last_became_peered":"2024-03-18T18:51:12.310435+0000","last_unstale":"2024-03-18T18:51:12.730758+0000","last_undegraded":"2024-03-18T18:51:12.730758+0000","last_fullsized":"2024-03-18T18:51:12.730758+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:20.375014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754490+0000","last_change":"2024-03-18T18:51:12.752279+0000","last_active":"2024-03-18T18:51:12.754490+0000","last_peered":"2024-03-18T18:51:12.754490+0000","last_clean":"2024-03-18T18:51:12.754490+0000","last_became_active":"2024-03-18T18:51:12.751986+0000","last_became_peered":"2024-03-18T18:51:12.751986+0000","last_unstale":"2024-03-18T18:51:12.754490+0000","last_undegraded":"2024-03-18T18:51:12.754490+0000","last_fullsized":"2024-03-18T18:51:12.754490+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:10:01.129869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"39'2","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.752856+0000","last_change":"2024-03-18T18:51:12.316191+0000","last_active":"2024-03-18T18:51:12.752856+0000","last_peered":"2024-03-18T18:51:12.752856+0000","last_clean":"2024-03-18T18:51:12.752856+0000","last_became_active":"2024-03-18T18:51:12.315716+0000","last_became_peered":"2024-03-18T18:51:12.315716+0000","last_unstale":"2024-03-18T18:51:12.752856+0000","last_undegraded":"2024-03-18T18:51:12.752856+0000","last_fullsized":"2024-03-18T18:51:12.752856+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:44.514169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.742105+0000","last_change":"2024-03-18T18:51:12.738039+0000","last_active":"2024-03-18T18:51:12.742105+0000","last_peered":"2024-03-18T18:51:12.742105+0000","last_clean":"2024-03-18T18:51:12.742105+0000","last_became_active":"2024-03-18T18:51:12.737510+0000","last_became_peered":"2024-03-18T18:51:12.737510+0000","last_unstale":"2024-03-18T18:51:12.742105+0000","last_undegraded":"2024-03-18T18:51:12.742105+0000","last_fullsized":"2024-03-18T18:51:12.742105+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:43:46.861441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.731577+0000","last_change":"2024-03-18T18:51:12.316263+0000","last_active":"2024-03-18T18:51:12.731577+0000","last_peered":"2024-03-18T18:51:12.731577+0000","last_clean":"2024-03-18T18:51:12.731577+0000","last_became_active":"2024-03-18T18:51:12.315977+0000","last_became_peered":"2024-03-18T18:51:12.315977+0000","last_unstale":"2024-03-18T18:51:12.731577+0000","last_undegraded":"2024-03-18T18:51:12.731577+0000","last_fullsized":"2024-03-18T18:51:12.731577+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:22:06.101192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.755749+0000","last_change":"2024-03-18T18:51:12.751989+0000","last_active":"2024-03-18T18:51:12.755749+0000","last_peered":"2024-03-18T18:51:12.755749+0000","last_clean":"2024-03-18T18:51:12.755749+0000","last_became_active":"2024-03-18T18:51:12.751706+0000","last_became_peered":"2024-03-18T18:51:12.751706+0000","last_unstale":"2024-03-18T18:51:12.755749+0000","last_undegraded":"2024-03-18T18:51:12.755749+0000","last_fullsized":"2024-03-18T18:51:12.755749+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:10:51.763969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.736414+0000","last_change":"2024-03-18T18:51:12.310516+0000","last_active":"2024-03-18T18:51:12.736414+0000","last_peered":"2024-03-18T18:51:12.736414+0000","last_clean":"2024-03-18T18:51:12.736414+0000","last_became_active":"2024-03-18T18:51:12.310249+0000","last_became_peered":"2024-03-18T18:51:12.310249+0000","last_unstale":"2024-03-18T18:51:12.736414+0000","last_undegraded":"2024-03-18T18:51:12.736414+0000","last_fullsized":"2024-03-18T18:51:12.736414+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:06:06.175499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.753765+0000","last_change":"2024-03-18T18:51:12.752861+0000","last_active":"2024-03-18T18:51:12.753765+0000","last_peered":"2024-03-18T18:51:12.753765+0000","last_clean":"2024-03-18T18:51:12.753765+0000","last_became_active":"2024-03-18T18:51:12.752562+0000","last_became_peered":"2024-03-18T18:51:12.752562+0000","last_unstale":"2024-03-18T18:51:12.753765+0000","last_undegraded":"2024-03-18T18:51:12.753765+0000","last_fullsized":"2024-03-18T18:51:12.753765+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:45:14.459730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.740993+0000","last_change":"2024-03-18T18:51:12.316093+0000","last_active":"2024-03-18T18:51:12.740993+0000","last_peered":"2024-03-18T18:51:12.740993+0000","last_clean":"2024-03-18T18:51:12.740993+0000","last_became_active":"2024-03-18T18:51:12.315826+0000","last_became_peered":"2024-03-18T18:51:12.315826+0000","last_unstale":"2024-03-18T18:51:12.740993+0000","last_undegraded":"2024-03-18T18:51:12.740993+0000","last_fullsized":"2024-03-18T18:51:12.740993+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:43:45.856558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.741646+0000","last_change":"2024-03-18T18:51:12.735827+0000","last_active":"2024-03-18T18:51:12.741646+0000","last_peered":"2024-03-18T18:51:12.741646+0000","last_clean":"2024-03-18T18:51:12.741646+0000","last_became_active":"2024-03-18T18:51:12.735536+0000","last_became_peered":"2024-03-18T18:51:12.735536+0000","last_unstale":"2024-03-18T18:51:12.741646+0000","last_undegraded":"2024-03-18T18:51:12.741646+0000","last_fullsized":"2024-03-18T18:51:12.741646+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:28:36.629231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.727808+0000","last_change":"2024-03-18T18:51:12.312615+0000","last_active":"2024-03-18T18:51:12.727808+0000","last_peered":"2024-03-18T18:51:12.727808+0000","last_clean":"2024-03-18T18:51:12.727808+0000","last_became_active":"2024-03-18T18:51:12.312310+0000","last_became_peered":"2024-03-18T18:51:12.312310+0000","last_unstale":"2024-03-18T18:51:12.727808+0000","last_undegraded":"2024-03-18T18:51:12.727808+0000","last_fullsized":"2024-03-18T18:51:12.727808+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:58:16.424096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.749096+0000","last_change":"2024-03-18T18:51:12.744495+0000","last_active":"2024-03-18T18:51:12.749096+0000","last_peered":"2024-03-18T18:51:12.749096+0000","last_clean":"2024-03-18T18:51:12.749096+0000","last_became_active":"2024-03-18T18:51:12.744216+0000","last_became_peered":"2024-03-18T18:51:12.744216+0000","last_unstale":"2024-03-18T18:51:12.749096+0000","last_undegraded":"2024-03-18T18:51:12.749096+0000","last_fullsized":"2024-03-18T18:51:12.749096+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:06.703485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.737048+0000","last_change":"2024-03-18T18:51:12.312661+0000","last_active":"2024-03-18T18:51:12.737048+0000","last_peered":"2024-03-18T18:51:12.737048+0000","last_clean":"2024-03-18T18:51:12.737048+0000","last_became_active":"2024-03-18T18:51:12.312430+0000","last_became_peered":"2024-03-18T18:51:12.312430+0000","last_unstale":"2024-03-18T18:51:12.737048+0000","last_undegraded":"2024-03-18T18:51:12.737048+0000","last_fullsized":"2024-03-18T18:51:12.737048+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:14.081804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.742174+0000","last_change":"2024-03-18T18:51:12.736875+0000","last_active":"2024-03-18T18:51:12.742174+0000","last_peered":"2024-03-18T18:51:12.742174+0000","last_clean":"2024-03-18T18:51:12.742174+0000","last_became_active":"2024-03-18T18:51:12.736605+0000","last_became_peered":"2024-03-18T18:51:12.736605+0000","last_unstale":"2024-03-18T18:51:12.742174+0000","last_undegraded":"2024-03-18T18:51:12.742174+0000","last_fullsized":"2024-03-18T18:51:12.742174+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:28.851313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.737561+0000","last_change":"2024-03-18T18:51:12.307027+0000","last_active":"2024-03-18T18:51:12.737561+0000","last_peered":"2024-03-18T18:51:12.737561+0000","last_clean":"2024-03-18T18:51:12.737561+0000","last_became_active":"2024-03-18T18:51:12.306512+0000","last_became_peered":"2024-03-18T18:51:12.306512+0000","last_unstale":"2024-03-18T18:51:12.737561+0000","last_undegraded":"2024-03-18T18:51:12.737561+0000","last_fullsized":"2024-03-18T18:51:12.737561+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:28:10.882012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754273+0000","last_change":"2024-03-18T18:51:12.749733+0000","last_active":"2024-03-18T18:51:12.754273+0000","last_peered":"2024-03-18T18:51:12.754273+0000","last_clean":"2024-03-18T18:51:12.754273+0000","last_became_active":"2024-03-18T18:51:12.749369+0000","last_became_peered":"2024-03-18T18:51:12.749369+0000","last_unstale":"2024-03-18T18:51:12.754273+0000","last_undegraded":"2024-03-18T18:51:12.754273+0000","last_fullsized":"2024-03-18T18:51:12.754273+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:17:06.269383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.731072+0000","last_change":"2024-03-18T18:51:12.318988+0000","last_active":"2024-03-18T18:51:12.731072+0000","last_peered":"2024-03-18T18:51:12.731072+0000","last_clean":"2024-03-18T18:51:12.731072+0000","last_became_active":"2024-03-18T18:51:12.318802+0000","last_became_peered":"2024-03-18T18:51:12.318802+0000","last_unstale":"2024-03-18T18:51:12.731072+0000","last_undegraded":"2024-03-18T18:51:12.731072+0000","last_fullsized":"2024-03-18T18:51:12.731072+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:13:18.483310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":176,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.309865+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T18:51:12.309865+0000","last_peered":"2024-03-18T18:51:12.309865+0000","last_clean":"2024-03-18T18:51:12.309865+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T18:51:12.309865+0000","last_undegraded":"2024-03-18T18:51:12.309865+0000","last_fullsized":"2024-03-18T18:51:12.309865+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28064,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343776,"statfs":{"total":96636764160,"available":96608026624,"internally_reserved":0,"allocated":1060864,"data_stored":784160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.47299999999999998}]},{"osd":1,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.57999999999999996}]},{"osd":2,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.45400000000000001}]},{"osd":3,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.497}]},{"osd":4,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.44700000000000001}]},{"osd":5,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.54300000000000004}]},{"osd":6,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.63400000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":192740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.274,"5min":3.274,"15min":3.274},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.3929999999999998,"5min":3.3929999999999998,"15min":3.3929999999999998},"last":0.5}]},{"osd":1,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.51000000000000001}]},{"osd":2,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.2370000000000001,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":0.55300000000000005}]},{"osd":3,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.36499999999999999}]},{"osd":4,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.8819999999999999,"5min":1.8819999999999999,"15min":1.8819999999999999},"last":0.39800000000000002}]},{"osd":5,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.437}]},{"osd":7,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.978,"5min":1.978,"15min":1.978},"last":0.48799999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":438272,"data_stored":185063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.41999999999999998}]},{"osd":2,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.443},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.377}]},{"osd":3,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.35899999999999999}]},{"osd":4,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.57599999999999996}]},{"osd":5,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.51100000000000001}]},{"osd":6,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.56599999999999995}]},{"osd":7,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.504},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.54100000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1048576,"data_stored":782560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.31900000000000001}]},{"osd":2,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.36199999999999999}]},{"osd":3,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.38200000000000001}]},{"osd":4,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.55900000000000005}]},{"osd":5,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.503}]},{"osd":6,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.54600000000000004}]},{"osd":7,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.51200000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":438272,"data_stored":185063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.1219999999999999,"5min":9.1219999999999999,"15min":9.1219999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":9.1560000000000006,"5min":9.1560000000000006,"15min":9.1560000000000006},"last":0.375}]},{"osd":1,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.2460000000000004,"5min":9.2460000000000004,"15min":9.2460000000000004},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.932,"5min":12.932,"15min":12.932},"last":0.35999999999999999}]},{"osd":3,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.4120000000000008,"5min":9.4120000000000008,"15min":9.4120000000000008},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":9.2680000000000007,"5min":9.2680000000000007,"15min":9.2680000000000007},"last":0.33400000000000002}]},{"osd":4,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3190000000000008,"5min":9.3190000000000008,"15min":9.3190000000000008},"last":0.58499999999999996}]},{"osd":5,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.1940000000000008,"5min":9.1940000000000008,"15min":9.1940000000000008},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.72499999999999998}]},{"osd":6,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":9.2219999999999995,"5min":9.2219999999999995,"15min":9.2219999999999995},"last":0.745}]},{"osd":7,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":9.3829999999999991,"5min":9.3829999999999991,"15min":9.3829999999999991},"last":0.71499999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":462848,"data_stored":192864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 18:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.42399999999999999}]},{"osd":1,"last update":"Mon Mar 18 18:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.44500000000000001}]},{"osd":2,"last update":"Mon Mar 18 18:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.54900000000000004}]},{"osd":4,"last update":"Mon Mar 18 18:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.64000000000000001}]},{"osd":5,"last update":"Mon Mar 18 18:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":0.78000000000000003}]},{"osd":6,"last update":"Mon Mar 18 18:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.61099999999999999}]},{"osd":7,"last update":"Mon Mar 18 18:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.752},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.56699999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":458752,"data_stored":192762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.51600000000000001}]},{"osd":1,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.316},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.442}]},{"osd":2,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.45600000000000002}]},{"osd":3,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.501}]},{"osd":5,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.36299999999999999}]},{"osd":6,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.56299999999999994}]},{"osd":7,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.55200000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":458752,"data_stored":192270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 18:50:40 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.3079999999999998,"5min":6.3079999999999998,"15min":6.3079999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.52200000000000002}]},{"osd":1,"last update":"Mon Mar 18 18:50:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":6.2549999999999999,"5min":6.2549999999999999,"15min":6.2549999999999999},"last":0.50600000000000001}]},{"osd":2,"last update":"Mon Mar 18 18:50:40 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.2999999999999998,"5min":6.2999999999999998,"15min":6.2999999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.387,"5min":17.387,"15min":17.387},"last":0.53700000000000003}]},{"osd":3,"last update":"Mon Mar 18 18:50:40 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":6.3330000000000002,"5min":6.3330000000000002,"15min":6.3330000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.46500000000000002}]},{"osd":4,"last update":"Mon Mar 18 18:50:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54500000000000004}]},{"osd":6,"last update":"Mon Mar 18 18:50:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.622},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.56999999999999995}]},{"osd":7,"last update":"Mon Mar 18 18:50:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.59399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T18:51:16.875 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T18:51:17.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.039+0000 7fd265dde700 1 Processor -- start 2024-03-18T18:51:17.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.039+0000 7fd265dde700 1 -- start start 2024-03-18T18:51:17.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.040+0000 7fd265dde700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd26006a750 0x7fd26006ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.040+0000 7fd265dde700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd2600ad790 0x7fd260161700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.040+0000 7fd265dde700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2600acfb0 con 0x7fd26015d6b0 2024-03-18T18:51:17.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.040+0000 7fd265dde700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2600af7a0 con 0x7fd2600ad790 2024-03-18T18:51:17.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.040+0000 7fd265dde700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd26006b070 con 0x7fd26006a750 2024-03-18T18:51:17.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.040+0000 7fd25ffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd2600ad790 0x7fd260161700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.040+0000 7fd25effd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd26015d6b0 0x7fd260160ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56406/0 (socket says 172.21.15.1:56406) 2024-03-18T18:51:17.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.040+0000 7fd25effd700 1 -- 172.21.15.1:0/4065168400 learned_addr learned my addr 172.21.15.1:0/4065168400 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:17.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.040+0000 7fd25f7fe700 1 --2- 172.21.15.1:0/4065168400 >> v2:172.21.15.165:3300/0 conn(0x7fd26006a750 0x7fd26006ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.040+0000 7fd25ffff700 1 -- 172.21.15.1:0/4065168400 >> v2:172.21.15.165:3300/0 conn(0x7fd26006a750 msgr2=0x7fd26006ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:17.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.040+0000 7fd25ffff700 1 --2- 172.21.15.1:0/4065168400 >> v2:172.21.15.165:3300/0 conn(0x7fd26006a750 0x7fd26006ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.040+0000 7fd25ffff700 1 -- 172.21.15.1:0/4065168400 >> v1:172.21.15.1:6789/0 conn(0x7fd26015d6b0 legacy=0x7fd260160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:17.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.040+0000 7fd25ffff700 1 -- 172.21.15.1:0/4065168400 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2600b50a0 con 0x7fd2600ad790 2024-03-18T18:51:17.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.040+0000 7fd25e7fc700 1 -- 172.21.15.1:0/4065168400 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830860656 0 0) 0x7fd2600acfb0 con 0x7fd26015d6b0 2024-03-18T18:51:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.040+0000 7fd25ffff700 1 --2- 172.21.15.1:0/4065168400 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2600ad790 0x7fd260161700 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd260073a90 tx=0x7fd25800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec0f0397eb43480 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.040+0000 7fd25e7fc700 1 -- 172.21.15.1:0/4065168400 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25800e070 con 0x7fd2600ad790 2024-03-18T18:51:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.040+0000 7fd25e7fc700 1 -- 172.21.15.1:0/4065168400 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25800bbd0 con 0x7fd2600ad790 2024-03-18T18:51:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.040+0000 7fd25e7fc700 1 -- 172.21.15.1:0/4065168400 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd258012430 con 0x7fd2600ad790 2024-03-18T18:51:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.041+0000 7fd265dde700 1 -- 172.21.15.1:0/4065168400 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2600ad790 msgr2=0x7fd260161700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.041+0000 7fd265dde700 1 --2- 172.21.15.1:0/4065168400 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2600ad790 0x7fd260161700 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd260073a90 tx=0x7fd25800b330 comp rx=0 tx=0).stop 2024-03-18T18:51:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.041+0000 7fd265dde700 1 -- 172.21.15.1:0/4065168400 shutdown_connections 2024-03-18T18:51:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.041+0000 7fd265dde700 1 --2- 172.21.15.1:0/4065168400 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2600ad790 0x7fd260161700 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.041+0000 7fd265dde700 1 --2- 172.21.15.1:0/4065168400 >> v2:172.21.15.165:3300/0 conn(0x7fd26006a750 0x7fd26006ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.041+0000 7fd265dde700 1 -- 172.21.15.1:0/4065168400 >> 172.21.15.1:0/4065168400 conn(0x7fd26006b860 msgr2=0x7fd26006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.041+0000 7fd265dde700 1 -- 172.21.15.1:0/4065168400 shutdown_connections 2024-03-18T18:51:17.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.041+0000 7fd265dde700 1 -- 172.21.15.1:0/4065168400 wait complete. 2024-03-18T18:51:17.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.042+0000 7fd265dde700 1 Processor -- start 2024-03-18T18:51:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.042+0000 7fd265dde700 1 -- start start 2024-03-18T18:51:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.043+0000 7fd265dde700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd26006a750 0x7fd2600ce540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.043+0000 7fd265dde700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2600ad790 0x7fd2601f35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.043+0000 7fd265dde700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd26015d6b0 0x7fd2601e6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.043+0000 7fd265dde700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2601f47b0 con 0x7fd2600ad790 2024-03-18T18:51:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.043+0000 7fd265dde700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26015eb40 con 0x7fd26006a750 2024-03-18T18:51:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.043+0000 7fd265dde700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2601f3eb0 con 0x7fd26015d6b0 2024-03-18T18:51:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.043+0000 7fd25effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2600ad790 0x7fd2601f35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.043+0000 7fd25effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2600ad790 0x7fd2601f35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39784/0 (socket says 172.21.15.1:39784) 2024-03-18T18:51:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.043+0000 7fd25effd700 1 -- 172.21.15.1:0/3418372736 learned_addr learned my addr 172.21.15.1:0/3418372736 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.043+0000 7fd25f7fe700 1 --2- 172.21.15.1:0/3418372736 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd26006a750 0x7fd2600ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.043+0000 7fd25ffff700 1 --2- 172.21.15.1:0/3418372736 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd26015d6b0 0x7fd2601e6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.043+0000 7fd25effd700 1 -- 172.21.15.1:0/3418372736 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd26015d6b0 msgr2=0x7fd2601e6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:17.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.043+0000 7fd25effd700 1 --2- 172.21.15.1:0/3418372736 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd26015d6b0 0x7fd2601e6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.043+0000 7fd25effd700 1 -- 172.21.15.1:0/3418372736 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd26006a750 msgr2=0x7fd2600ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:17.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.043+0000 7fd25effd700 1 --2- 172.21.15.1:0/3418372736 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd26006a750 0x7fd2600ce540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.044+0000 7fd25effd700 1 -- 172.21.15.1:0/3418372736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd258009a70 con 0x7fd2600ad790 2024-03-18T18:51:17.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.044+0000 7fd25effd700 1 --2- 172.21.15.1:0/3418372736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2600ad790 0x7fd2601f35f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd25400c550 tx=0x7fd25400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:17.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.044+0000 7fd247fff700 1 -- 172.21.15.1:0/3418372736 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd254004c80 con 0x7fd2600ad790 2024-03-18T18:51:17.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.044+0000 7fd247fff700 1 -- 172.21.15.1:0/3418372736 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd254004e00 con 0x7fd2600ad790 2024-03-18T18:51:17.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.044+0000 7fd247fff700 1 -- 172.21.15.1:0/3418372736 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd254007310 con 0x7fd2600ad790 2024-03-18T18:51:17.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.044+0000 7fd265dde700 1 -- 172.21.15.1:0/3418372736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2601f4190 con 0x7fd2600ad790 2024-03-18T18:51:17.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.044+0000 7fd265dde700 1 -- 172.21.15.1:0/3418372736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd26015ede0 con 0x7fd2600ad790 2024-03-18T18:51:17.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.045+0000 7fd265dde700 1 -- 172.21.15.1:0/3418372736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2600addc0 con 0x7fd2600ad790 2024-03-18T18:51:17.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.045+0000 7fd247fff700 1 -- 172.21.15.1:0/3418372736 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd254007470 con 0x7fd2600ad790 2024-03-18T18:51:17.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.046+0000 7fd247fff700 1 --2- 172.21.15.1:0/3418372736 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd24c06ac90 0x7fd24c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.046+0000 7fd247fff700 1 -- 172.21.15.1:0/3418372736 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd254089060 con 0x7fd2600ad790 2024-03-18T18:51:17.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.046+0000 7fd25f7fe700 1 --2- 172.21.15.1:0/3418372736 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd24c06ac90 0x7fd24c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.046+0000 7fd25f7fe700 1 --2- 172.21.15.1:0/3418372736 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd24c06ac90 0x7fd24c06d150 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd250006b20 tx=0x7fd250008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:17.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.050+0000 7fd247fff700 1 -- 172.21.15.1:0/3418372736 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd254054100 con 0x7fd2600ad790 2024-03-18T18:51:17.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.249+0000 7fd265dde700 1 -- 172.21.15.1:0/3418372736 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd2600ad440 con 0x7fd24c06ac90 2024-03-18T18:51:17.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.258+0000 7fd247fff700 1 -- 172.21.15.1:0/3418372736 <== mgr.4104 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289556 (secure 0 0 0) 0x7fd250009e40 con 0x7fd24c06ac90 2024-03-18T18:51:17.257 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:51:17.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.260+0000 7fd265dde700 1 -- 172.21.15.1:0/3418372736 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd24c06ac90 msgr2=0x7fd24c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:17.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.260+0000 7fd265dde700 1 --2- 172.21.15.1:0/3418372736 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd24c06ac90 0x7fd24c06d150 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd250006b20 tx=0x7fd250008040 comp rx=0 tx=0).stop 2024-03-18T18:51:17.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.260+0000 7fd265dde700 1 -- 172.21.15.1:0/3418372736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2600ad790 msgr2=0x7fd2601f35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:17.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.260+0000 7fd265dde700 1 --2- 172.21.15.1:0/3418372736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2600ad790 0x7fd2601f35f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd25400c550 tx=0x7fd25400c860 comp rx=0 tx=0).stop 2024-03-18T18:51:17.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.261+0000 7fd265dde700 1 -- 172.21.15.1:0/3418372736 shutdown_connections 2024-03-18T18:51:17.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.261+0000 7fd265dde700 1 --2- 172.21.15.1:0/3418372736 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd24c06ac90 0x7fd24c06d150 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.261+0000 7fd265dde700 1 --2- 172.21.15.1:0/3418372736 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd26015d6b0 0x7fd2601e6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.261+0000 7fd265dde700 1 --2- 172.21.15.1:0/3418372736 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd26006a750 0x7fd2600ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.261+0000 7fd265dde700 1 --2- 172.21.15.1:0/3418372736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2600ad790 0x7fd2601f35f0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.261+0000 7fd265dde700 1 -- 172.21.15.1:0/3418372736 >> 172.21.15.1:0/3418372736 conn(0x7fd26006b860 msgr2=0x7fd2600b2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:17.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.261+0000 7fd265dde700 1 -- 172.21.15.1:0/3418372736 shutdown_connections 2024-03-18T18:51:17.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.261+0000 7fd265dde700 1 -- 172.21.15.1:0/3418372736 wait complete. 2024-03-18T18:51:17.261 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T18:51:17.269 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-03-18T18:51:15.406414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":59,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220908,"kb_used_data":4708,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753812,"statfs":{"total":773094113280,"available":772867903488,"internally_reserved":0,"allocated":4820992,"data_stored":2707482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.116927"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.755338+0000","last_change":"2024-03-18T18:51:12.751054+0000","last_active":"2024-03-18T18:51:12.755338+0000","last_peered":"2024-03-18T18:51:12.755338+0000","last_clean":"2024-03-18T18:51:12.755338+0000","last_became_active":"2024-03-18T18:51:12.748978+0000","last_became_peered":"2024-03-18T18:51:12.748978+0000","last_unstale":"2024-03-18T18:51:12.755338+0000","last_undegraded":"2024-03-18T18:51:12.755338+0000","last_fullsized":"2024-03-18T18:51:12.755338+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:41:36.474089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.751179+0000","last_change":"2024-03-18T18:51:12.747397+0000","last_active":"2024-03-18T18:51:12.751179+0000","last_peered":"2024-03-18T18:51:12.751179+0000","last_clean":"2024-03-18T18:51:12.751179+0000","last_became_active":"2024-03-18T18:51:12.743998+0000","last_became_peered":"2024-03-18T18:51:12.743998+0000","last_unstale":"2024-03-18T18:51:12.751179+0000","last_undegraded":"2024-03-18T18:51:12.751179+0000","last_fullsized":"2024-03-18T18:51:12.751179+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:08.546546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.756599+0000","last_change":"2024-03-18T18:51:12.753720+0000","last_active":"2024-03-18T18:51:12.756599+0000","last_peered":"2024-03-18T18:51:12.756599+0000","last_clean":"2024-03-18T18:51:12.756599+0000","last_became_active":"2024-03-18T18:51:12.753455+0000","last_became_peered":"2024-03-18T18:51:12.753455+0000","last_unstale":"2024-03-18T18:51:12.756599+0000","last_undegraded":"2024-03-18T18:51:12.756599+0000","last_fullsized":"2024-03-18T18:51:12.756599+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:04.301923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.747489+0000","last_change":"2024-03-18T18:51:12.745544+0000","last_active":"2024-03-18T18:51:12.747489+0000","last_peered":"2024-03-18T18:51:12.747489+0000","last_clean":"2024-03-18T18:51:12.747489+0000","last_became_active":"2024-03-18T18:51:12.745160+0000","last_became_peered":"2024-03-18T18:51:12.745160+0000","last_unstale":"2024-03-18T18:51:12.747489+0000","last_undegraded":"2024-03-18T18:51:12.747489+0000","last_fullsized":"2024-03-18T18:51:12.747489+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:15:59.751511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.747170+0000","last_change":"2024-03-18T18:51:12.744928+0000","last_active":"2024-03-18T18:51:12.747170+0000","last_peered":"2024-03-18T18:51:12.747170+0000","last_clean":"2024-03-18T18:51:12.747170+0000","last_became_active":"2024-03-18T18:51:12.744684+0000","last_became_peered":"2024-03-18T18:51:12.744684+0000","last_unstale":"2024-03-18T18:51:12.747170+0000","last_undegraded":"2024-03-18T18:51:12.747170+0000","last_fullsized":"2024-03-18T18:51:12.747170+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:08:54.946392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.745359+0000","last_change":"2024-03-18T18:51:12.744662+0000","last_active":"2024-03-18T18:51:12.745359+0000","last_peered":"2024-03-18T18:51:12.745359+0000","last_clean":"2024-03-18T18:51:12.745359+0000","last_became_active":"2024-03-18T18:51:12.744389+0000","last_became_peered":"2024-03-18T18:51:12.744389+0000","last_unstale":"2024-03-18T18:51:12.745359+0000","last_undegraded":"2024-03-18T18:51:12.745359+0000","last_fullsized":"2024-03-18T18:51:12.745359+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:28:11.886895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754200+0000","last_change":"2024-03-18T18:51:12.749916+0000","last_active":"2024-03-18T18:51:12.754200+0000","last_peered":"2024-03-18T18:51:12.754200+0000","last_clean":"2024-03-18T18:51:12.754200+0000","last_became_active":"2024-03-18T18:51:12.749228+0000","last_became_peered":"2024-03-18T18:51:12.749228+0000","last_unstale":"2024-03-18T18:51:12.754200+0000","last_undegraded":"2024-03-18T18:51:12.754200+0000","last_fullsized":"2024-03-18T18:51:12.754200+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:59.968694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.756985+0000","last_change":"2024-03-18T18:51:12.756578+0000","last_active":"2024-03-18T18:51:12.756985+0000","last_peered":"2024-03-18T18:51:12.756985+0000","last_clean":"2024-03-18T18:51:12.756985+0000","last_became_active":"2024-03-18T18:51:12.756370+0000","last_became_peered":"2024-03-18T18:51:12.756370+0000","last_unstale":"2024-03-18T18:51:12.756985+0000","last_undegraded":"2024-03-18T18:51:12.756985+0000","last_fullsized":"2024-03-18T18:51:12.756985+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:40:33.676867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.757317+0000","last_change":"2024-03-18T18:51:12.754260+0000","last_active":"2024-03-18T18:51:12.757317+0000","last_peered":"2024-03-18T18:51:12.757317+0000","last_clean":"2024-03-18T18:51:12.757317+0000","last_became_active":"2024-03-18T18:51:12.753993+0000","last_became_peered":"2024-03-18T18:51:12.753993+0000","last_unstale":"2024-03-18T18:51:12.757317+0000","last_undegraded":"2024-03-18T18:51:12.757317+0000","last_fullsized":"2024-03-18T18:51:12.757317+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:02:57.116657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.756344+0000","last_change":"2024-03-18T18:51:12.751746+0000","last_active":"2024-03-18T18:51:12.756344+0000","last_peered":"2024-03-18T18:51:12.756344+0000","last_clean":"2024-03-18T18:51:12.756344+0000","last_became_active":"2024-03-18T18:51:12.751211+0000","last_became_peered":"2024-03-18T18:51:12.751211+0000","last_unstale":"2024-03-18T18:51:12.756344+0000","last_undegraded":"2024-03-18T18:51:12.756344+0000","last_fullsized":"2024-03-18T18:51:12.756344+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:10:01.129869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.753830+0000","last_change":"2024-03-18T18:51:12.749255+0000","last_active":"2024-03-18T18:51:12.753830+0000","last_peered":"2024-03-18T18:51:12.753830+0000","last_clean":"2024-03-18T18:51:12.753830+0000","last_became_active":"2024-03-18T18:51:12.748716+0000","last_became_peered":"2024-03-18T18:51:12.748716+0000","last_unstale":"2024-03-18T18:51:12.753830+0000","last_undegraded":"2024-03-18T18:51:12.753830+0000","last_fullsized":"2024-03-18T18:51:12.753830+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:06:07.180382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.747598+0000","last_change":"2024-03-18T18:51:12.746869+0000","last_active":"2024-03-18T18:51:12.747598+0000","last_peered":"2024-03-18T18:51:12.747598+0000","last_clean":"2024-03-18T18:51:12.747598+0000","last_became_active":"2024-03-18T18:51:12.746600+0000","last_became_peered":"2024-03-18T18:51:12.746600+0000","last_unstale":"2024-03-18T18:51:12.747598+0000","last_undegraded":"2024-03-18T18:51:12.747598+0000","last_fullsized":"2024-03-18T18:51:12.747598+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:06.703485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.756605+0000","last_change":"2024-03-18T18:51:12.750508+0000","last_active":"2024-03-18T18:51:12.756605+0000","last_peered":"2024-03-18T18:51:12.756605+0000","last_clean":"2024-03-18T18:51:12.756605+0000","last_became_active":"2024-03-18T18:51:12.750163+0000","last_became_peered":"2024-03-18T18:51:12.750163+0000","last_unstale":"2024-03-18T18:51:12.756605+0000","last_undegraded":"2024-03-18T18:51:12.756605+0000","last_fullsized":"2024-03-18T18:51:12.756605+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:18:51.530853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.751366+0000","last_change":"2024-03-18T18:51:12.748631+0000","last_active":"2024-03-18T18:51:12.751366+0000","last_peered":"2024-03-18T18:51:12.751366+0000","last_clean":"2024-03-18T18:51:12.751366+0000","last_became_active":"2024-03-18T18:51:12.748341+0000","last_became_peered":"2024-03-18T18:51:12.748341+0000","last_unstale":"2024-03-18T18:51:12.751366+0000","last_undegraded":"2024-03-18T18:51:12.751366+0000","last_fullsized":"2024-03-18T18:51:12.751366+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:59.968694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754217+0000","last_change":"2024-03-18T18:51:12.749953+0000","last_active":"2024-03-18T18:51:12.754217+0000","last_peered":"2024-03-18T18:51:12.754217+0000","last_clean":"2024-03-18T18:51:12.754217+0000","last_became_active":"2024-03-18T18:51:12.749623+0000","last_became_peered":"2024-03-18T18:51:12.749623+0000","last_unstale":"2024-03-18T18:51:12.754217+0000","last_undegraded":"2024-03-18T18:51:12.754217+0000","last_fullsized":"2024-03-18T18:51:12.754217+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:06.703485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754299+0000","last_change":"2024-03-18T18:51:12.751305+0000","last_active":"2024-03-18T18:51:12.754299+0000","last_peered":"2024-03-18T18:51:12.754299+0000","last_clean":"2024-03-18T18:51:12.754299+0000","last_became_active":"2024-03-18T18:51:12.751012+0000","last_became_peered":"2024-03-18T18:51:12.751012+0000","last_unstale":"2024-03-18T18:51:12.754299+0000","last_undegraded":"2024-03-18T18:51:12.754299+0000","last_fullsized":"2024-03-18T18:51:12.754299+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:06:07.180382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754423+0000","last_change":"2024-03-18T18:51:12.751407+0000","last_active":"2024-03-18T18:51:12.754423+0000","last_peered":"2024-03-18T18:51:12.754423+0000","last_clean":"2024-03-18T18:51:12.754423+0000","last_became_active":"2024-03-18T18:51:12.751086+0000","last_became_peered":"2024-03-18T18:51:12.751086+0000","last_unstale":"2024-03-18T18:51:12.754423+0000","last_undegraded":"2024-03-18T18:51:12.754423+0000","last_fullsized":"2024-03-18T18:51:12.754423+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:30:54.407177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.750946+0000","last_change":"2024-03-18T18:51:12.746400+0000","last_active":"2024-03-18T18:51:12.750946+0000","last_peered":"2024-03-18T18:51:12.750946+0000","last_clean":"2024-03-18T18:51:12.750946+0000","last_became_active":"2024-03-18T18:51:12.746122+0000","last_became_peered":"2024-03-18T18:51:12.746122+0000","last_unstale":"2024-03-18T18:51:12.750946+0000","last_undegraded":"2024-03-18T18:51:12.750946+0000","last_fullsized":"2024-03-18T18:51:12.750946+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:10.811679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.744254+0000","last_change":"2024-03-18T18:51:12.738346+0000","last_active":"2024-03-18T18:51:12.744254+0000","last_peered":"2024-03-18T18:51:12.744254+0000","last_clean":"2024-03-18T18:51:12.744254+0000","last_became_active":"2024-03-18T18:51:12.738085+0000","last_became_peered":"2024-03-18T18:51:12.738085+0000","last_unstale":"2024-03-18T18:51:12.744254+0000","last_undegraded":"2024-03-18T18:51:12.744254+0000","last_fullsized":"2024-03-18T18:51:12.744254+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:56:23.719374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.750147+0000","last_change":"2024-03-18T18:51:12.744825+0000","last_active":"2024-03-18T18:51:12.750147+0000","last_peered":"2024-03-18T18:51:12.750147+0000","last_clean":"2024-03-18T18:51:12.750147+0000","last_became_active":"2024-03-18T18:51:12.744577+0000","last_became_peered":"2024-03-18T18:51:12.744577+0000","last_unstale":"2024-03-18T18:51:12.750147+0000","last_undegraded":"2024-03-18T18:51:12.750147+0000","last_fullsized":"2024-03-18T18:51:12.750147+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:20:54.202383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.759249+0000","last_change":"2024-03-18T18:51:12.754171+0000","last_active":"2024-03-18T18:51:12.759249+0000","last_peered":"2024-03-18T18:51:12.759249+0000","last_clean":"2024-03-18T18:51:12.759249+0000","last_became_active":"2024-03-18T18:51:12.753535+0000","last_became_peered":"2024-03-18T18:51:12.753535+0000","last_unstale":"2024-03-18T18:51:12.759249+0000","last_undegraded":"2024-03-18T18:51:12.759249+0000","last_fullsized":"2024-03-18T18:51:12.759249+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:43:46.861441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.751386+0000","last_change":"2024-03-18T18:51:12.749000+0000","last_active":"2024-03-18T18:51:12.751386+0000","last_peered":"2024-03-18T18:51:12.751386+0000","last_clean":"2024-03-18T18:51:12.751386+0000","last_became_active":"2024-03-18T18:51:12.748034+0000","last_became_peered":"2024-03-18T18:51:12.748034+0000","last_unstale":"2024-03-18T18:51:12.751386+0000","last_undegraded":"2024-03-18T18:51:12.751386+0000","last_fullsized":"2024-03-18T18:51:12.751386+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:21:13.902213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.744715+0000","last_change":"2024-03-18T18:51:12.741944+0000","last_active":"2024-03-18T18:51:12.744715+0000","last_peered":"2024-03-18T18:51:12.744715+0000","last_clean":"2024-03-18T18:51:12.744715+0000","last_became_active":"2024-03-18T18:51:12.738039+0000","last_became_peered":"2024-03-18T18:51:12.738039+0000","last_unstale":"2024-03-18T18:51:12.744715+0000","last_undegraded":"2024-03-18T18:51:12.744715+0000","last_fullsized":"2024-03-18T18:51:12.744715+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:15.086687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.746198+0000","last_change":"2024-03-18T18:51:12.740508+0000","last_active":"2024-03-18T18:51:12.746198+0000","last_peered":"2024-03-18T18:51:12.746198+0000","last_clean":"2024-03-18T18:51:12.746198+0000","last_became_active":"2024-03-18T18:51:12.740257+0000","last_became_peered":"2024-03-18T18:51:12.740257+0000","last_unstale":"2024-03-18T18:51:12.746198+0000","last_undegraded":"2024-03-18T18:51:12.746198+0000","last_fullsized":"2024-03-18T18:51:12.746198+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:20:54.202383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.750716+0000","last_change":"2024-03-18T18:51:12.745721+0000","last_active":"2024-03-18T18:51:12.750716+0000","last_peered":"2024-03-18T18:51:12.750716+0000","last_clean":"2024-03-18T18:51:12.750716+0000","last_became_active":"2024-03-18T18:51:12.745201+0000","last_became_peered":"2024-03-18T18:51:12.745201+0000","last_unstale":"2024-03-18T18:51:12.750716+0000","last_undegraded":"2024-03-18T18:51:12.750716+0000","last_fullsized":"2024-03-18T18:51:12.750716+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:15.086687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754644+0000","last_change":"2024-03-18T18:51:12.752339+0000","last_active":"2024-03-18T18:51:12.754644+0000","last_peered":"2024-03-18T18:51:12.754644+0000","last_clean":"2024-03-18T18:51:12.754644+0000","last_became_active":"2024-03-18T18:51:12.752053+0000","last_became_peered":"2024-03-18T18:51:12.752053+0000","last_unstale":"2024-03-18T18:51:12.754644+0000","last_undegraded":"2024-03-18T18:51:12.754644+0000","last_fullsized":"2024-03-18T18:51:12.754644+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:06.703485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.746157+0000","last_change":"2024-03-18T18:51:12.737443+0000","last_active":"2024-03-18T18:51:12.746157+0000","last_peered":"2024-03-18T18:51:12.746157+0000","last_clean":"2024-03-18T18:51:12.746157+0000","last_became_active":"2024-03-18T18:51:12.737179+0000","last_became_peered":"2024-03-18T18:51:12.737179+0000","last_unstale":"2024-03-18T18:51:12.746157+0000","last_undegraded":"2024-03-18T18:51:12.746157+0000","last_fullsized":"2024-03-18T18:51:12.746157+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:56:07.670601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754027+0000","last_change":"2024-03-18T18:51:12.748803+0000","last_active":"2024-03-18T18:51:12.754027+0000","last_peered":"2024-03-18T18:51:12.754027+0000","last_clean":"2024-03-18T18:51:12.754027+0000","last_became_active":"2024-03-18T18:51:12.748501+0000","last_became_peered":"2024-03-18T18:51:12.748501+0000","last_unstale":"2024-03-18T18:51:12.754027+0000","last_undegraded":"2024-03-18T18:51:12.754027+0000","last_fullsized":"2024-03-18T18:51:12.754027+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:06:26.805929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.746437+0000","last_change":"2024-03-18T18:51:12.736150+0000","last_active":"2024-03-18T18:51:12.746437+0000","last_peered":"2024-03-18T18:51:12.746437+0000","last_clean":"2024-03-18T18:51:12.746437+0000","last_became_active":"2024-03-18T18:51:12.735896+0000","last_became_peered":"2024-03-18T18:51:12.735896+0000","last_unstale":"2024-03-18T18:51:12.746437+0000","last_undegraded":"2024-03-18T18:51:12.746437+0000","last_fullsized":"2024-03-18T18:51:12.746437+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:56:23.719374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.751558+0000","last_change":"2024-03-18T18:51:12.748729+0000","last_active":"2024-03-18T18:51:12.751558+0000","last_peered":"2024-03-18T18:51:12.751558+0000","last_clean":"2024-03-18T18:51:12.751558+0000","last_became_active":"2024-03-18T18:51:12.748480+0000","last_became_peered":"2024-03-18T18:51:12.748480+0000","last_unstale":"2024-03-18T18:51:12.751558+0000","last_undegraded":"2024-03-18T18:51:12.751558+0000","last_fullsized":"2024-03-18T18:51:12.751558+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:11:17.657760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.744696+0000","last_change":"2024-03-18T18:51:12.738308+0000","last_active":"2024-03-18T18:51:12.744696+0000","last_peered":"2024-03-18T18:51:12.744696+0000","last_clean":"2024-03-18T18:51:12.744696+0000","last_became_active":"2024-03-18T18:51:12.736012+0000","last_became_peered":"2024-03-18T18:51:12.736012+0000","last_unstale":"2024-03-18T18:51:12.744696+0000","last_undegraded":"2024-03-18T18:51:12.744696+0000","last_fullsized":"2024-03-18T18:51:12.744696+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:10.811679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.743418+0000","last_change":"2024-03-18T18:51:12.740850+0000","last_active":"2024-03-18T18:51:12.743418+0000","last_peered":"2024-03-18T18:51:12.743418+0000","last_clean":"2024-03-18T18:51:12.743418+0000","last_became_active":"2024-03-18T18:51:12.740595+0000","last_became_peered":"2024-03-18T18:51:12.740595+0000","last_unstale":"2024-03-18T18:51:12.743418+0000","last_undegraded":"2024-03-18T18:51:12.743418+0000","last_fullsized":"2024-03-18T18:51:12.743418+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:37:58.205820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754405+0000","last_change":"2024-03-18T18:51:12.753087+0000","last_active":"2024-03-18T18:51:12.754405+0000","last_peered":"2024-03-18T18:51:12.754405+0000","last_clean":"2024-03-18T18:51:12.754405+0000","last_became_active":"2024-03-18T18:51:12.752833+0000","last_became_peered":"2024-03-18T18:51:12.752833+0000","last_unstale":"2024-03-18T18:51:12.754405+0000","last_undegraded":"2024-03-18T18:51:12.754405+0000","last_fullsized":"2024-03-18T18:51:12.754405+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:19:35.490918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"39'4","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.742189+0000","last_change":"2024-03-18T18:51:12.307632+0000","last_active":"2024-03-18T18:51:12.742189+0000","last_peered":"2024-03-18T18:51:12.742189+0000","last_clean":"2024-03-18T18:51:12.742189+0000","last_became_active":"2024-03-18T18:51:12.307141+0000","last_became_peered":"2024-03-18T18:51:12.307141+0000","last_unstale":"2024-03-18T18:51:12.742189+0000","last_undegraded":"2024-03-18T18:51:12.742189+0000","last_fullsized":"2024-03-18T18:51:12.742189+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:13:18.483310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754700+0000","last_change":"2024-03-18T18:51:12.752712+0000","last_active":"2024-03-18T18:51:12.754700+0000","last_peered":"2024-03-18T18:51:12.754700+0000","last_clean":"2024-03-18T18:51:12.754700+0000","last_became_active":"2024-03-18T18:51:12.752460+0000","last_became_peered":"2024-03-18T18:51:12.752460+0000","last_unstale":"2024-03-18T18:51:12.754700+0000","last_undegraded":"2024-03-18T18:51:12.754700+0000","last_fullsized":"2024-03-18T18:51:12.754700+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:59.968694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"39'1","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.745121+0000","last_change":"2024-03-18T18:51:12.318280+0000","last_active":"2024-03-18T18:51:12.745121+0000","last_peered":"2024-03-18T18:51:12.745121+0000","last_clean":"2024-03-18T18:51:12.745121+0000","last_became_active":"2024-03-18T18:51:12.317960+0000","last_became_peered":"2024-03-18T18:51:12.317960+0000","last_unstale":"2024-03-18T18:51:12.745121+0000","last_undegraded":"2024-03-18T18:51:12.745121+0000","last_fullsized":"2024-03-18T18:51:12.745121+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:20:21.587872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.749131+0000","last_change":"2024-03-18T18:51:12.746107+0000","last_active":"2024-03-18T18:51:12.749131+0000","last_peered":"2024-03-18T18:51:12.749131+0000","last_clean":"2024-03-18T18:51:12.749131+0000","last_became_active":"2024-03-18T18:51:12.745656+0000","last_became_peered":"2024-03-18T18:51:12.745656+0000","last_unstale":"2024-03-18T18:51:12.749131+0000","last_undegraded":"2024-03-18T18:51:12.749131+0000","last_fullsized":"2024-03-18T18:51:12.749131+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:34:03.497588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"39'2","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.744448+0000","last_change":"2024-03-18T18:51:12.312353+0000","last_active":"2024-03-18T18:51:12.744448+0000","last_peered":"2024-03-18T18:51:12.744448+0000","last_clean":"2024-03-18T18:51:12.744448+0000","last_became_active":"2024-03-18T18:51:12.311888+0000","last_became_peered":"2024-03-18T18:51:12.311888+0000","last_unstale":"2024-03-18T18:51:12.744448+0000","last_undegraded":"2024-03-18T18:51:12.744448+0000","last_fullsized":"2024-03-18T18:51:12.744448+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:11:16.652877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.755328+0000","last_change":"2024-03-18T18:51:12.751027+0000","last_active":"2024-03-18T18:51:12.755328+0000","last_peered":"2024-03-18T18:51:12.755328+0000","last_clean":"2024-03-18T18:51:12.755328+0000","last_became_active":"2024-03-18T18:51:12.748500+0000","last_became_peered":"2024-03-18T18:51:12.748500+0000","last_unstale":"2024-03-18T18:51:12.755328+0000","last_undegraded":"2024-03-18T18:51:12.755328+0000","last_fullsized":"2024-03-18T18:51:12.755328+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:40:07.068740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"39'2","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.743162+0000","last_change":"2024-03-18T18:51:12.311843+0000","last_active":"2024-03-18T18:51:12.743162+0000","last_peered":"2024-03-18T18:51:12.743162+0000","last_clean":"2024-03-18T18:51:12.743162+0000","last_became_active":"2024-03-18T18:51:12.311507+0000","last_became_peered":"2024-03-18T18:51:12.311507+0000","last_unstale":"2024-03-18T18:51:12.743162+0000","last_undegraded":"2024-03-18T18:51:12.743162+0000","last_fullsized":"2024-03-18T18:51:12.743162+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:19:34.486035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.745440+0000","last_change":"2024-03-18T18:51:12.742400+0000","last_active":"2024-03-18T18:51:12.745440+0000","last_peered":"2024-03-18T18:51:12.745440+0000","last_clean":"2024-03-18T18:51:12.745440+0000","last_became_active":"2024-03-18T18:51:12.742159+0000","last_became_peered":"2024-03-18T18:51:12.742159+0000","last_unstale":"2024-03-18T18:51:12.745440+0000","last_undegraded":"2024-03-18T18:51:12.745440+0000","last_fullsized":"2024-03-18T18:51:12.745440+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:20.862960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"39'1","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.752544+0000","last_change":"2024-03-18T18:51:12.314920+0000","last_active":"2024-03-18T18:51:12.752544+0000","last_peered":"2024-03-18T18:51:12.752544+0000","last_clean":"2024-03-18T18:51:12.752544+0000","last_became_active":"2024-03-18T18:51:12.314633+0000","last_became_peered":"2024-03-18T18:51:12.314633+0000","last_unstale":"2024-03-18T18:51:12.752544+0000","last_undegraded":"2024-03-18T18:51:12.752544+0000","last_fullsized":"2024-03-18T18:51:12.752544+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:21:33.793911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754231+0000","last_change":"2024-03-18T18:51:12.750081+0000","last_active":"2024-03-18T18:51:12.754231+0000","last_peered":"2024-03-18T18:51:12.754231+0000","last_clean":"2024-03-18T18:51:12.754231+0000","last_became_active":"2024-03-18T18:51:12.749654+0000","last_became_peered":"2024-03-18T18:51:12.749654+0000","last_unstale":"2024-03-18T18:51:12.754231+0000","last_undegraded":"2024-03-18T18:51:12.754231+0000","last_fullsized":"2024-03-18T18:51:12.754231+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:51.202259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"39'1","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.747326+0000","last_change":"2024-03-18T18:51:12.319821+0000","last_active":"2024-03-18T18:51:12.747326+0000","last_peered":"2024-03-18T18:51:12.747326+0000","last_clean":"2024-03-18T18:51:12.747326+0000","last_became_active":"2024-03-18T18:51:12.319558+0000","last_became_peered":"2024-03-18T18:51:12.319558+0000","last_unstale":"2024-03-18T18:51:12.747326+0000","last_undegraded":"2024-03-18T18:51:12.747326+0000","last_fullsized":"2024-03-18T18:51:12.747326+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:22:06.101192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.759166+0000","last_change":"2024-03-18T18:51:12.752230+0000","last_active":"2024-03-18T18:51:12.759166+0000","last_peered":"2024-03-18T18:51:12.759166+0000","last_clean":"2024-03-18T18:51:12.759166+0000","last_became_active":"2024-03-18T18:51:12.751960+0000","last_became_peered":"2024-03-18T18:51:12.751960+0000","last_unstale":"2024-03-18T18:51:12.759166+0000","last_undegraded":"2024-03-18T18:51:12.759166+0000","last_fullsized":"2024-03-18T18:51:12.759166+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:17:06.269383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.728547+0000","last_change":"2024-03-18T18:51:12.318058+0000","last_active":"2024-03-18T18:51:12.728547+0000","last_peered":"2024-03-18T18:51:12.728547+0000","last_clean":"2024-03-18T18:51:12.728547+0000","last_became_active":"2024-03-18T18:51:12.317737+0000","last_became_peered":"2024-03-18T18:51:12.317737+0000","last_unstale":"2024-03-18T18:51:12.728547+0000","last_undegraded":"2024-03-18T18:51:12.728547+0000","last_fullsized":"2024-03-18T18:51:12.728547+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:09.806796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.755645+0000","last_change":"2024-03-18T18:51:12.751554+0000","last_active":"2024-03-18T18:51:12.755645+0000","last_peered":"2024-03-18T18:51:12.755645+0000","last_clean":"2024-03-18T18:51:12.755645+0000","last_became_active":"2024-03-18T18:51:12.748608+0000","last_became_peered":"2024-03-18T18:51:12.748608+0000","last_unstale":"2024-03-18T18:51:12.755645+0000","last_undegraded":"2024-03-18T18:51:12.755645+0000","last_fullsized":"2024-03-18T18:51:12.755645+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:55:21.585205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.737360+0000","last_change":"2024-03-18T18:51:12.314356+0000","last_active":"2024-03-18T18:51:12.737360+0000","last_peered":"2024-03-18T18:51:12.737360+0000","last_clean":"2024-03-18T18:51:12.737360+0000","last_became_active":"2024-03-18T18:51:12.314068+0000","last_became_peered":"2024-03-18T18:51:12.314068+0000","last_unstale":"2024-03-18T18:51:12.737360+0000","last_undegraded":"2024-03-18T18:51:12.737360+0000","last_fullsized":"2024-03-18T18:51:12.737360+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:06:25.801047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.747470+0000","last_change":"2024-03-18T18:51:12.745394+0000","last_active":"2024-03-18T18:51:12.747470+0000","last_peered":"2024-03-18T18:51:12.747470+0000","last_clean":"2024-03-18T18:51:12.747470+0000","last_became_active":"2024-03-18T18:51:12.744822+0000","last_became_peered":"2024-03-18T18:51:12.744822+0000","last_unstale":"2024-03-18T18:51:12.747470+0000","last_undegraded":"2024-03-18T18:51:12.747470+0000","last_fullsized":"2024-03-18T18:51:12.747470+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:45.519051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.735519+0000","last_change":"2024-03-18T18:51:12.315258+0000","last_active":"2024-03-18T18:51:12.735519+0000","last_peered":"2024-03-18T18:51:12.735519+0000","last_clean":"2024-03-18T18:51:12.735519+0000","last_became_active":"2024-03-18T18:51:12.314955+0000","last_became_peered":"2024-03-18T18:51:12.314955+0000","last_unstale":"2024-03-18T18:51:12.735519+0000","last_undegraded":"2024-03-18T18:51:12.735519+0000","last_fullsized":"2024-03-18T18:51:12.735519+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:37:57.200937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.742125+0000","last_change":"2024-03-18T18:51:12.737828+0000","last_active":"2024-03-18T18:51:12.742125+0000","last_peered":"2024-03-18T18:51:12.742125+0000","last_clean":"2024-03-18T18:51:12.742125+0000","last_became_active":"2024-03-18T18:51:12.737164+0000","last_became_peered":"2024-03-18T18:51:12.737164+0000","last_unstale":"2024-03-18T18:51:12.742125+0000","last_undegraded":"2024-03-18T18:51:12.742125+0000","last_fullsized":"2024-03-18T18:51:12.742125+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:20:22.592755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.744411+0000","last_change":"2024-03-18T18:51:12.315010+0000","last_active":"2024-03-18T18:51:12.744411+0000","last_peered":"2024-03-18T18:51:12.744411+0000","last_clean":"2024-03-18T18:51:12.744411+0000","last_became_active":"2024-03-18T18:51:12.314735+0000","last_became_peered":"2024-03-18T18:51:12.314735+0000","last_unstale":"2024-03-18T18:51:12.744411+0000","last_undegraded":"2024-03-18T18:51:12.744411+0000","last_fullsized":"2024-03-18T18:51:12.744411+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:08:53.941509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.746799+0000","last_change":"2024-03-18T18:51:12.745640+0000","last_active":"2024-03-18T18:51:12.746799+0000","last_peered":"2024-03-18T18:51:12.746799+0000","last_clean":"2024-03-18T18:51:12.746799+0000","last_became_active":"2024-03-18T18:51:12.745389+0000","last_became_peered":"2024-03-18T18:51:12.745389+0000","last_unstale":"2024-03-18T18:51:12.746799+0000","last_undegraded":"2024-03-18T18:51:12.746799+0000","last_fullsized":"2024-03-18T18:51:12.746799+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:21:13.902213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"39'3","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754826+0000","last_change":"2024-03-18T18:51:12.316395+0000","last_active":"2024-03-18T18:51:12.754826+0000","last_peered":"2024-03-18T18:51:12.754826+0000","last_clean":"2024-03-18T18:51:12.754826+0000","last_became_active":"2024-03-18T18:51:12.316109+0000","last_became_peered":"2024-03-18T18:51:12.316109+0000","last_unstale":"2024-03-18T18:51:12.754826+0000","last_undegraded":"2024-03-18T18:51:12.754826+0000","last_fullsized":"2024-03-18T18:51:12.754826+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:15:58.746628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.747263+0000","last_change":"2024-03-18T18:51:12.741630+0000","last_active":"2024-03-18T18:51:12.747263+0000","last_peered":"2024-03-18T18:51:12.747263+0000","last_clean":"2024-03-18T18:51:12.747263+0000","last_became_active":"2024-03-18T18:51:12.741356+0000","last_became_peered":"2024-03-18T18:51:12.741356+0000","last_unstale":"2024-03-18T18:51:12.747263+0000","last_undegraded":"2024-03-18T18:51:12.747263+0000","last_fullsized":"2024-03-18T18:51:12.747263+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:28:36.629231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"39'6","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.762165+0000","last_change":"2024-03-18T18:51:12.307177+0000","last_active":"2024-03-18T18:51:12.762165+0000","last_peered":"2024-03-18T18:51:12.762165+0000","last_clean":"2024-03-18T18:51:12.762165+0000","last_became_active":"2024-03-18T18:51:12.306931+0000","last_became_peered":"2024-03-18T18:51:12.306931+0000","last_unstale":"2024-03-18T18:51:12.762165+0000","last_undegraded":"2024-03-18T18:51:12.762165+0000","last_fullsized":"2024-03-18T18:51:12.762165+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:18:50.525971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.757305+0000","last_change":"2024-03-18T18:51:12.753671+0000","last_active":"2024-03-18T18:51:12.757305+0000","last_peered":"2024-03-18T18:51:12.757305+0000","last_clean":"2024-03-18T18:51:12.757305+0000","last_became_active":"2024-03-18T18:51:12.753379+0000","last_became_peered":"2024-03-18T18:51:12.753379+0000","last_unstale":"2024-03-18T18:51:12.757305+0000","last_undegraded":"2024-03-18T18:51:12.757305+0000","last_fullsized":"2024-03-18T18:51:12.757305+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:13.711567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"39'2","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.751108+0000","last_change":"2024-03-18T18:51:12.313397+0000","last_active":"2024-03-18T18:51:12.751108+0000","last_peered":"2024-03-18T18:51:12.751108+0000","last_clean":"2024-03-18T18:51:12.751108+0000","last_became_active":"2024-03-18T18:51:12.313119+0000","last_became_peered":"2024-03-18T18:51:12.313119+0000","last_unstale":"2024-03-18T18:51:12.751108+0000","last_undegraded":"2024-03-18T18:51:12.751108+0000","last_fullsized":"2024-03-18T18:51:12.751108+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:34:02.492705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.756494+0000","last_change":"2024-03-18T18:51:12.748676+0000","last_active":"2024-03-18T18:51:12.756494+0000","last_peered":"2024-03-18T18:51:12.756494+0000","last_clean":"2024-03-18T18:51:12.756494+0000","last_became_active":"2024-03-18T18:51:12.748406+0000","last_became_peered":"2024-03-18T18:51:12.748406+0000","last_unstale":"2024-03-18T18:51:12.756494+0000","last_undegraded":"2024-03-18T18:51:12.756494+0000","last_fullsized":"2024-03-18T18:51:12.756494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:24:46.941173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.728255+0000","last_change":"2024-03-18T18:51:12.314385+0000","last_active":"2024-03-18T18:51:12.728255+0000","last_peered":"2024-03-18T18:51:12.728255+0000","last_clean":"2024-03-18T18:51:12.728255+0000","last_became_active":"2024-03-18T18:51:12.314115+0000","last_became_peered":"2024-03-18T18:51:12.314115+0000","last_unstale":"2024-03-18T18:51:12.728255+0000","last_undegraded":"2024-03-18T18:51:12.728255+0000","last_fullsized":"2024-03-18T18:51:12.728255+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:20:53.197501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.744606+0000","last_change":"2024-03-18T18:51:12.738187+0000","last_active":"2024-03-18T18:51:12.744606+0000","last_peered":"2024-03-18T18:51:12.744606+0000","last_clean":"2024-03-18T18:51:12.744606+0000","last_became_active":"2024-03-18T18:51:12.735956+0000","last_became_peered":"2024-03-18T18:51:12.735956+0000","last_unstale":"2024-03-18T18:51:12.744606+0000","last_undegraded":"2024-03-18T18:51:12.744606+0000","last_fullsized":"2024-03-18T18:51:12.744606+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:11:17.657760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.727450+0000","last_change":"2024-03-18T18:51:12.307067+0000","last_active":"2024-03-18T18:51:12.727450+0000","last_peered":"2024-03-18T18:51:12.727450+0000","last_clean":"2024-03-18T18:51:12.727450+0000","last_became_active":"2024-03-18T18:51:12.306786+0000","last_became_peered":"2024-03-18T18:51:12.306786+0000","last_unstale":"2024-03-18T18:51:12.727450+0000","last_undegraded":"2024-03-18T18:51:12.727450+0000","last_fullsized":"2024-03-18T18:51:12.727450+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:55:02.518147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.755662+0000","last_change":"2024-03-18T18:51:12.751564+0000","last_active":"2024-03-18T18:51:12.755662+0000","last_peered":"2024-03-18T18:51:12.755662+0000","last_clean":"2024-03-18T18:51:12.755662+0000","last_became_active":"2024-03-18T18:51:12.748991+0000","last_became_peered":"2024-03-18T18:51:12.748991+0000","last_unstale":"2024-03-18T18:51:12.755662+0000","last_undegraded":"2024-03-18T18:51:12.755662+0000","last_fullsized":"2024-03-18T18:51:12.755662+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:49:47.690564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"39'8","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.747311+0000","last_change":"2024-03-18T18:51:12.319791+0000","last_active":"2024-03-18T18:51:12.747311+0000","last_peered":"2024-03-18T18:51:12.747311+0000","last_clean":"2024-03-18T18:51:12.747311+0000","last_became_active":"2024-03-18T18:51:12.319481+0000","last_became_peered":"2024-03-18T18:51:12.319481+0000","last_unstale":"2024-03-18T18:51:12.747311+0000","last_undegraded":"2024-03-18T18:51:12.747311+0000","last_fullsized":"2024-03-18T18:51:12.747311+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:34.624046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.747891+0000","last_change":"2024-03-18T18:51:12.742362+0000","last_active":"2024-03-18T18:51:12.747891+0000","last_peered":"2024-03-18T18:51:12.747891+0000","last_clean":"2024-03-18T18:51:12.747891+0000","last_became_active":"2024-03-18T18:51:12.742051+0000","last_became_peered":"2024-03-18T18:51:12.742051+0000","last_unstale":"2024-03-18T18:51:12.747891+0000","last_undegraded":"2024-03-18T18:51:12.747891+0000","last_fullsized":"2024-03-18T18:51:12.747891+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:29:42.322290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"39'1","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.752937+0000","last_change":"2024-03-18T18:51:12.316428+0000","last_active":"2024-03-18T18:51:12.752937+0000","last_peered":"2024-03-18T18:51:12.752937+0000","last_clean":"2024-03-18T18:51:12.752937+0000","last_became_active":"2024-03-18T18:51:12.316013+0000","last_became_peered":"2024-03-18T18:51:12.316013+0000","last_unstale":"2024-03-18T18:51:12.752937+0000","last_undegraded":"2024-03-18T18:51:12.752937+0000","last_fullsized":"2024-03-18T18:51:12.752937+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:09:54.723546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754308+0000","last_change":"2024-03-18T18:51:12.749472+0000","last_active":"2024-03-18T18:51:12.754308+0000","last_peered":"2024-03-18T18:51:12.754308+0000","last_clean":"2024-03-18T18:51:12.754308+0000","last_became_active":"2024-03-18T18:51:12.748996+0000","last_became_peered":"2024-03-18T18:51:12.748996+0000","last_unstale":"2024-03-18T18:51:12.754308+0000","last_undegraded":"2024-03-18T18:51:12.754308+0000","last_fullsized":"2024-03-18T18:51:12.754308+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:21:13.902213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"39'2","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.752248+0000","last_change":"2024-03-18T18:51:12.316517+0000","last_active":"2024-03-18T18:51:12.752248+0000","last_peered":"2024-03-18T18:51:12.752248+0000","last_clean":"2024-03-18T18:51:12.752248+0000","last_became_active":"2024-03-18T18:51:12.316244+0000","last_became_peered":"2024-03-18T18:51:12.316244+0000","last_unstale":"2024-03-18T18:51:12.752248+0000","last_undegraded":"2024-03-18T18:51:12.752248+0000","last_fullsized":"2024-03-18T18:51:12.752248+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:11:16.652877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.749113+0000","last_change":"2024-03-18T18:51:12.745916+0000","last_active":"2024-03-18T18:51:12.749113+0000","last_peered":"2024-03-18T18:51:12.749113+0000","last_clean":"2024-03-18T18:51:12.749113+0000","last_became_active":"2024-03-18T18:51:12.745199+0000","last_became_peered":"2024-03-18T18:51:12.745199+0000","last_unstale":"2024-03-18T18:51:12.749113+0000","last_undegraded":"2024-03-18T18:51:12.749113+0000","last_fullsized":"2024-03-18T18:51:12.749113+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:10:01.129869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"39'1","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.743105+0000","last_change":"2024-03-18T18:51:12.308635+0000","last_active":"2024-03-18T18:51:12.743105+0000","last_peered":"2024-03-18T18:51:12.743105+0000","last_clean":"2024-03-18T18:51:12.743105+0000","last_became_active":"2024-03-18T18:51:12.308357+0000","last_became_peered":"2024-03-18T18:51:12.308357+0000","last_unstale":"2024-03-18T18:51:12.743105+0000","last_undegraded":"2024-03-18T18:51:12.743105+0000","last_fullsized":"2024-03-18T18:51:12.743105+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:27:08.916844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.747637+0000","last_change":"2024-03-18T18:51:12.743998+0000","last_active":"2024-03-18T18:51:12.747637+0000","last_peered":"2024-03-18T18:51:12.747637+0000","last_clean":"2024-03-18T18:51:12.747637+0000","last_became_active":"2024-03-18T18:51:12.743467+0000","last_became_peered":"2024-03-18T18:51:12.743467+0000","last_unstale":"2024-03-18T18:51:12.747637+0000","last_undegraded":"2024-03-18T18:51:12.747637+0000","last_fullsized":"2024-03-18T18:51:12.747637+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:56:07.670601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.728651+0000","last_change":"2024-03-18T18:51:12.311242+0000","last_active":"2024-03-18T18:51:12.728651+0000","last_peered":"2024-03-18T18:51:12.728651+0000","last_clean":"2024-03-18T18:51:12.728651+0000","last_became_active":"2024-03-18T18:51:12.310956+0000","last_became_peered":"2024-03-18T18:51:12.310956+0000","last_unstale":"2024-03-18T18:51:12.728651+0000","last_undegraded":"2024-03-18T18:51:12.728651+0000","last_fullsized":"2024-03-18T18:51:12.728651+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:14.081804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.744895+0000","last_change":"2024-03-18T18:51:12.742697+0000","last_active":"2024-03-18T18:51:12.744895+0000","last_peered":"2024-03-18T18:51:12.744895+0000","last_clean":"2024-03-18T18:51:12.744895+0000","last_became_active":"2024-03-18T18:51:12.742081+0000","last_became_peered":"2024-03-18T18:51:12.742081+0000","last_unstale":"2024-03-18T18:51:12.744895+0000","last_undegraded":"2024-03-18T18:51:12.744895+0000","last_fullsized":"2024-03-18T18:51:12.744895+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:06:26.805929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"39'2","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.741063+0000","last_change":"2024-03-18T18:51:12.312411+0000","last_active":"2024-03-18T18:51:12.741063+0000","last_peered":"2024-03-18T18:51:12.741063+0000","last_clean":"2024-03-18T18:51:12.741063+0000","last_became_active":"2024-03-18T18:51:12.312111+0000","last_became_peered":"2024-03-18T18:51:12.312111+0000","last_unstale":"2024-03-18T18:51:12.741063+0000","last_undegraded":"2024-03-18T18:51:12.741063+0000","last_fullsized":"2024-03-18T18:51:12.741063+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:49.708013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.757116+0000","last_change":"2024-03-18T18:51:12.753947+0000","last_active":"2024-03-18T18:51:12.757116+0000","last_peered":"2024-03-18T18:51:12.757116+0000","last_clean":"2024-03-18T18:51:12.757116+0000","last_became_active":"2024-03-18T18:51:12.753759+0000","last_became_peered":"2024-03-18T18:51:12.753759+0000","last_unstale":"2024-03-18T18:51:12.757116+0000","last_undegraded":"2024-03-18T18:51:12.757116+0000","last_fullsized":"2024-03-18T18:51:12.757116+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:51.202259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.731489+0000","last_change":"2024-03-18T18:51:12.307869+0000","last_active":"2024-03-18T18:51:12.731489+0000","last_peered":"2024-03-18T18:51:12.731489+0000","last_clean":"2024-03-18T18:51:12.731489+0000","last_became_active":"2024-03-18T18:51:12.307554+0000","last_became_peered":"2024-03-18T18:51:12.307554+0000","last_unstale":"2024-03-18T18:51:12.731489+0000","last_undegraded":"2024-03-18T18:51:12.731489+0000","last_fullsized":"2024-03-18T18:51:12.731489+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:00:31.640644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.752304+0000","last_change":"2024-03-18T18:51:12.747250+0000","last_active":"2024-03-18T18:51:12.752304+0000","last_peered":"2024-03-18T18:51:12.752304+0000","last_clean":"2024-03-18T18:51:12.752304+0000","last_became_active":"2024-03-18T18:51:12.746868+0000","last_became_peered":"2024-03-18T18:51:12.746868+0000","last_unstale":"2024-03-18T18:51:12.752304+0000","last_undegraded":"2024-03-18T18:51:12.752304+0000","last_fullsized":"2024-03-18T18:51:12.752304+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:28.851313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.731021+0000","last_change":"2024-03-18T18:51:12.318504+0000","last_active":"2024-03-18T18:51:12.731021+0000","last_peered":"2024-03-18T18:51:12.731021+0000","last_clean":"2024-03-18T18:51:12.731021+0000","last_became_active":"2024-03-18T18:51:12.318165+0000","last_became_peered":"2024-03-18T18:51:12.318165+0000","last_unstale":"2024-03-18T18:51:12.731021+0000","last_undegraded":"2024-03-18T18:51:12.731021+0000","last_fullsized":"2024-03-18T18:51:12.731021+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:58:16.424096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.755374+0000","last_change":"2024-03-18T18:51:12.751138+0000","last_active":"2024-03-18T18:51:12.755374+0000","last_peered":"2024-03-18T18:51:12.755374+0000","last_clean":"2024-03-18T18:51:12.755374+0000","last_became_active":"2024-03-18T18:51:12.748240+0000","last_became_peered":"2024-03-18T18:51:12.748240+0000","last_unstale":"2024-03-18T18:51:12.755374+0000","last_undegraded":"2024-03-18T18:51:12.755374+0000","last_fullsized":"2024-03-18T18:51:12.755374+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:10.892017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.730758+0000","last_change":"2024-03-18T18:51:12.310709+0000","last_active":"2024-03-18T18:51:12.730758+0000","last_peered":"2024-03-18T18:51:12.730758+0000","last_clean":"2024-03-18T18:51:12.730758+0000","last_became_active":"2024-03-18T18:51:12.310435+0000","last_became_peered":"2024-03-18T18:51:12.310435+0000","last_unstale":"2024-03-18T18:51:12.730758+0000","last_undegraded":"2024-03-18T18:51:12.730758+0000","last_fullsized":"2024-03-18T18:51:12.730758+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:20.375014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754490+0000","last_change":"2024-03-18T18:51:12.752279+0000","last_active":"2024-03-18T18:51:12.754490+0000","last_peered":"2024-03-18T18:51:12.754490+0000","last_clean":"2024-03-18T18:51:12.754490+0000","last_became_active":"2024-03-18T18:51:12.751986+0000","last_became_peered":"2024-03-18T18:51:12.751986+0000","last_unstale":"2024-03-18T18:51:12.754490+0000","last_undegraded":"2024-03-18T18:51:12.754490+0000","last_fullsized":"2024-03-18T18:51:12.754490+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:10:01.129869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"39'2","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.752856+0000","last_change":"2024-03-18T18:51:12.316191+0000","last_active":"2024-03-18T18:51:12.752856+0000","last_peered":"2024-03-18T18:51:12.752856+0000","last_clean":"2024-03-18T18:51:12.752856+0000","last_became_active":"2024-03-18T18:51:12.315716+0000","last_became_peered":"2024-03-18T18:51:12.315716+0000","last_unstale":"2024-03-18T18:51:12.752856+0000","last_undegraded":"2024-03-18T18:51:12.752856+0000","last_fullsized":"2024-03-18T18:51:12.752856+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:44.514169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.742105+0000","last_change":"2024-03-18T18:51:12.738039+0000","last_active":"2024-03-18T18:51:12.742105+0000","last_peered":"2024-03-18T18:51:12.742105+0000","last_clean":"2024-03-18T18:51:12.742105+0000","last_became_active":"2024-03-18T18:51:12.737510+0000","last_became_peered":"2024-03-18T18:51:12.737510+0000","last_unstale":"2024-03-18T18:51:12.742105+0000","last_undegraded":"2024-03-18T18:51:12.742105+0000","last_fullsized":"2024-03-18T18:51:12.742105+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:43:46.861441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.731577+0000","last_change":"2024-03-18T18:51:12.316263+0000","last_active":"2024-03-18T18:51:12.731577+0000","last_peered":"2024-03-18T18:51:12.731577+0000","last_clean":"2024-03-18T18:51:12.731577+0000","last_became_active":"2024-03-18T18:51:12.315977+0000","last_became_peered":"2024-03-18T18:51:12.315977+0000","last_unstale":"2024-03-18T18:51:12.731577+0000","last_undegraded":"2024-03-18T18:51:12.731577+0000","last_fullsized":"2024-03-18T18:51:12.731577+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:22:06.101192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.755749+0000","last_change":"2024-03-18T18:51:12.751989+0000","last_active":"2024-03-18T18:51:12.755749+0000","last_peered":"2024-03-18T18:51:12.755749+0000","last_clean":"2024-03-18T18:51:12.755749+0000","last_became_active":"2024-03-18T18:51:12.751706+0000","last_became_peered":"2024-03-18T18:51:12.751706+0000","last_unstale":"2024-03-18T18:51:12.755749+0000","last_undegraded":"2024-03-18T18:51:12.755749+0000","last_fullsized":"2024-03-18T18:51:12.755749+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:10:51.763969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.736414+0000","last_change":"2024-03-18T18:51:12.310516+0000","last_active":"2024-03-18T18:51:12.736414+0000","last_peered":"2024-03-18T18:51:12.736414+0000","last_clean":"2024-03-18T18:51:12.736414+0000","last_became_active":"2024-03-18T18:51:12.310249+0000","last_became_peered":"2024-03-18T18:51:12.310249+0000","last_unstale":"2024-03-18T18:51:12.736414+0000","last_undegraded":"2024-03-18T18:51:12.736414+0000","last_fullsized":"2024-03-18T18:51:12.736414+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:06:06.175499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.753765+0000","last_change":"2024-03-18T18:51:12.752861+0000","last_active":"2024-03-18T18:51:12.753765+0000","last_peered":"2024-03-18T18:51:12.753765+0000","last_clean":"2024-03-18T18:51:12.753765+0000","last_became_active":"2024-03-18T18:51:12.752562+0000","last_became_peered":"2024-03-18T18:51:12.752562+0000","last_unstale":"2024-03-18T18:51:12.753765+0000","last_undegraded":"2024-03-18T18:51:12.753765+0000","last_fullsized":"2024-03-18T18:51:12.753765+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:45:14.459730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.740993+0000","last_change":"2024-03-18T18:51:12.316093+0000","last_active":"2024-03-18T18:51:12.740993+0000","last_peered":"2024-03-18T18:51:12.740993+0000","last_clean":"2024-03-18T18:51:12.740993+0000","last_became_active":"2024-03-18T18:51:12.315826+0000","last_became_peered":"2024-03-18T18:51:12.315826+0000","last_unstale":"2024-03-18T18:51:12.740993+0000","last_undegraded":"2024-03-18T18:51:12.740993+0000","last_fullsized":"2024-03-18T18:51:12.740993+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:43:45.856558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.741646+0000","last_change":"2024-03-18T18:51:12.735827+0000","last_active":"2024-03-18T18:51:12.741646+0000","last_peered":"2024-03-18T18:51:12.741646+0000","last_clean":"2024-03-18T18:51:12.741646+0000","last_became_active":"2024-03-18T18:51:12.735536+0000","last_became_peered":"2024-03-18T18:51:12.735536+0000","last_unstale":"2024-03-18T18:51:12.741646+0000","last_undegraded":"2024-03-18T18:51:12.741646+0000","last_fullsized":"2024-03-18T18:51:12.741646+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:28:36.629231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.727808+0000","last_change":"2024-03-18T18:51:12.312615+0000","last_active":"2024-03-18T18:51:12.727808+0000","last_peered":"2024-03-18T18:51:12.727808+0000","last_clean":"2024-03-18T18:51:12.727808+0000","last_became_active":"2024-03-18T18:51:12.312310+0000","last_became_peered":"2024-03-18T18:51:12.312310+0000","last_unstale":"2024-03-18T18:51:12.727808+0000","last_undegraded":"2024-03-18T18:51:12.727808+0000","last_fullsized":"2024-03-18T18:51:12.727808+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:58:16.424096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.749096+0000","last_change":"2024-03-18T18:51:12.744495+0000","last_active":"2024-03-18T18:51:12.749096+0000","last_peered":"2024-03-18T18:51:12.749096+0000","last_clean":"2024-03-18T18:51:12.749096+0000","last_became_active":"2024-03-18T18:51:12.744216+0000","last_became_peered":"2024-03-18T18:51:12.744216+0000","last_unstale":"2024-03-18T18:51:12.749096+0000","last_undegraded":"2024-03-18T18:51:12.749096+0000","last_fullsized":"2024-03-18T18:51:12.749096+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:06.703485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.737048+0000","last_change":"2024-03-18T18:51:12.312661+0000","last_active":"2024-03-18T18:51:12.737048+0000","last_peered":"2024-03-18T18:51:12.737048+0000","last_clean":"2024-03-18T18:51:12.737048+0000","last_became_active":"2024-03-18T18:51:12.312430+0000","last_became_peered":"2024-03-18T18:51:12.312430+0000","last_unstale":"2024-03-18T18:51:12.737048+0000","last_undegraded":"2024-03-18T18:51:12.737048+0000","last_fullsized":"2024-03-18T18:51:12.737048+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:14.081804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.742174+0000","last_change":"2024-03-18T18:51:12.736875+0000","last_active":"2024-03-18T18:51:12.742174+0000","last_peered":"2024-03-18T18:51:12.742174+0000","last_clean":"2024-03-18T18:51:12.742174+0000","last_became_active":"2024-03-18T18:51:12.736605+0000","last_became_peered":"2024-03-18T18:51:12.736605+0000","last_unstale":"2024-03-18T18:51:12.742174+0000","last_undegraded":"2024-03-18T18:51:12.742174+0000","last_fullsized":"2024-03-18T18:51:12.742174+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:28.851313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.737561+0000","last_change":"2024-03-18T18:51:12.307027+0000","last_active":"2024-03-18T18:51:12.737561+0000","last_peered":"2024-03-18T18:51:12.737561+0000","last_clean":"2024-03-18T18:51:12.737561+0000","last_became_active":"2024-03-18T18:51:12.306512+0000","last_became_peered":"2024-03-18T18:51:12.306512+0000","last_unstale":"2024-03-18T18:51:12.737561+0000","last_undegraded":"2024-03-18T18:51:12.737561+0000","last_fullsized":"2024-03-18T18:51:12.737561+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:28:10.882012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.754273+0000","last_change":"2024-03-18T18:51:12.749733+0000","last_active":"2024-03-18T18:51:12.754273+0000","last_peered":"2024-03-18T18:51:12.754273+0000","last_clean":"2024-03-18T18:51:12.754273+0000","last_became_active":"2024-03-18T18:51:12.749369+0000","last_became_peered":"2024-03-18T18:51:12.749369+0000","last_unstale":"2024-03-18T18:51:12.754273+0000","last_undegraded":"2024-03-18T18:51:12.754273+0000","last_fullsized":"2024-03-18T18:51:12.754273+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:12.289465+0000","last_clean_scrub_stamp":"2024-03-18T18:51:12.289465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:17:06.269383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.731072+0000","last_change":"2024-03-18T18:51:12.318988+0000","last_active":"2024-03-18T18:51:12.731072+0000","last_peered":"2024-03-18T18:51:12.731072+0000","last_clean":"2024-03-18T18:51:12.731072+0000","last_became_active":"2024-03-18T18:51:12.318802+0000","last_became_peered":"2024-03-18T18:51:12.318802+0000","last_unstale":"2024-03-18T18:51:12.731072+0000","last_undegraded":"2024-03-18T18:51:12.731072+0000","last_fullsized":"2024-03-18T18:51:12.731072+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:51:11.284582+0000","last_clean_scrub_stamp":"2024-03-18T18:51:11.284582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:13:18.483310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":176,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-18T18:51:12.309865+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T18:51:12.309865+0000","last_peered":"2024-03-18T18:51:12.309865+0000","last_clean":"2024-03-18T18:51:12.309865+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T18:51:12.309865+0000","last_undegraded":"2024-03-18T18:51:12.309865+0000","last_fullsized":"2024-03-18T18:51:12.309865+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28064,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343776,"statfs":{"total":96636764160,"available":96608026624,"internally_reserved":0,"allocated":1060864,"data_stored":784160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.47299999999999998}]},{"osd":1,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.57999999999999996}]},{"osd":2,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.45400000000000001}]},{"osd":3,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.497}]},{"osd":4,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.44700000000000001}]},{"osd":5,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.54300000000000004}]},{"osd":6,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.63400000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":192740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.274,"5min":3.274,"15min":3.274},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.3929999999999998,"5min":3.3929999999999998,"15min":3.3929999999999998},"last":0.5}]},{"osd":1,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.51000000000000001}]},{"osd":2,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.2370000000000001,"5min":2.2370000000000001,"15min":2.2370000000000001},"last":0.55300000000000005}]},{"osd":3,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.36499999999999999}]},{"osd":4,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.8819999999999999,"5min":1.8819999999999999,"15min":1.8819999999999999},"last":0.39800000000000002}]},{"osd":5,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.437}]},{"osd":7,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.978,"5min":1.978,"15min":1.978},"last":0.48799999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":438272,"data_stored":185063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.41999999999999998}]},{"osd":2,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.443},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.377}]},{"osd":3,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.35899999999999999}]},{"osd":4,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.57599999999999996}]},{"osd":5,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.51100000000000001}]},{"osd":6,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.56599999999999995}]},{"osd":7,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.504},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.54100000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1048576,"data_stored":782560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.31900000000000001}]},{"osd":2,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.36199999999999999}]},{"osd":3,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.38200000000000001}]},{"osd":4,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.55900000000000005}]},{"osd":5,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.503}]},{"osd":6,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.54600000000000004}]},{"osd":7,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.51200000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":438272,"data_stored":185063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.1219999999999999,"5min":9.1219999999999999,"15min":9.1219999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":9.1560000000000006,"5min":9.1560000000000006,"15min":9.1560000000000006},"last":0.375}]},{"osd":1,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.2460000000000004,"5min":9.2460000000000004,"15min":9.2460000000000004},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.932,"5min":12.932,"15min":12.932},"last":0.35999999999999999}]},{"osd":3,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.4120000000000008,"5min":9.4120000000000008,"15min":9.4120000000000008},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":9.2680000000000007,"5min":9.2680000000000007,"15min":9.2680000000000007},"last":0.33400000000000002}]},{"osd":4,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.3190000000000008,"5min":9.3190000000000008,"15min":9.3190000000000008},"last":0.58499999999999996}]},{"osd":5,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":9.1940000000000008,"5min":9.1940000000000008,"15min":9.1940000000000008},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.72499999999999998}]},{"osd":6,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":9.3559999999999999,"5min":9.3559999999999999,"15min":9.3559999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":9.2219999999999995,"5min":9.2219999999999995,"15min":9.2219999999999995},"last":0.745}]},{"osd":7,"last update":"Mon Mar 18 18:50:38 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":9.3829999999999991,"5min":9.3829999999999991,"15min":9.3829999999999991},"last":0.71499999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":462848,"data_stored":192864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 18:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.42399999999999999}]},{"osd":1,"last update":"Mon Mar 18 18:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.44500000000000001}]},{"osd":2,"last update":"Mon Mar 18 18:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.54900000000000004}]},{"osd":4,"last update":"Mon Mar 18 18:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.64000000000000001}]},{"osd":5,"last update":"Mon Mar 18 18:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":0.78000000000000003}]},{"osd":6,"last update":"Mon Mar 18 18:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.61099999999999999}]},{"osd":7,"last update":"Mon Mar 18 18:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.752},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.56699999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":458752,"data_stored":192762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.51600000000000001}]},{"osd":1,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.316},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.442}]},{"osd":2,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.45600000000000002}]},{"osd":3,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.501}]},{"osd":5,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.36299999999999999}]},{"osd":6,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.56299999999999994}]},{"osd":7,"last update":"Mon Mar 18 18:50:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.55200000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":458752,"data_stored":192270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 18:50:40 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.3079999999999998,"5min":6.3079999999999998,"15min":6.3079999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.52200000000000002}]},{"osd":1,"last update":"Mon Mar 18 18:50:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":6.2549999999999999,"5min":6.2549999999999999,"15min":6.2549999999999999},"last":0.50600000000000001}]},{"osd":2,"last update":"Mon Mar 18 18:50:40 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.2999999999999998,"5min":6.2999999999999998,"15min":6.2999999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.387,"5min":17.387,"15min":17.387},"last":0.53700000000000003}]},{"osd":3,"last update":"Mon Mar 18 18:50:40 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":6.3330000000000002,"5min":6.3330000000000002,"15min":6.3330000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.46500000000000002}]},{"osd":4,"last update":"Mon Mar 18 18:50:40 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54500000000000004}]},{"osd":6,"last update":"Mon Mar 18 18:50:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.622},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.56999999999999995}]},{"osd":7,"last update":"Mon Mar 18 18:50:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.59399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T18:51:17.273 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T18:51:17.274 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-18T18:51:17.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.446+0000 7fef71517700 1 Processor -- start 2024-03-18T18:51:17.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.446+0000 7fef71517700 1 -- start start 2024-03-18T18:51:17.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.446+0000 7fef71517700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fef6c15b6c0 0x7fef6c0b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.446+0000 7fef71517700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef6c0cd430 0x7fef6c0c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.446+0000 7fef71517700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef6c0d01f0 con 0x7fef6c0c8600 2024-03-18T18:51:17.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.446+0000 7fef71517700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef6c0ccfa0 con 0x7fef6c15b6c0 2024-03-18T18:51:17.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.446+0000 7fef71517700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6c0d1db0 con 0x7fef6c0cd430 2024-03-18T18:51:17.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.447+0000 7fef6affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fef6c15b6c0 0x7fef6c0b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.447+0000 7fef6affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fef6c15b6c0 0x7fef6c0b2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39794/0 (socket says 172.21.15.1:39794) 2024-03-18T18:51:17.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.447+0000 7fef6affd700 1 -- 172.21.15.1:0/2205386 learned_addr learned my addr 172.21.15.1:0/2205386 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:17.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.447+0000 7fef6a7fc700 1 --2- 172.21.15.1:0/2205386 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef6c0cd430 0x7fef6c0c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.447+0000 7fef6affd700 1 -- 172.21.15.1:0/2205386 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef6c0cd430 msgr2=0x7fef6c0c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:17.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.447+0000 7fef6affd700 1 --2- 172.21.15.1:0/2205386 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef6c0cd430 0x7fef6c0c80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.447+0000 7fef6affd700 1 -- 172.21.15.1:0/2205386 >> v1:172.21.15.165:6789/0 conn(0x7fef6c0c8600 legacy=0x7fef6c0cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:17.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.447+0000 7fef6affd700 1 -- 172.21.15.1:0/2205386 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef6c0cff10 con 0x7fef6c15b6c0 2024-03-18T18:51:17.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.447+0000 7fef6affd700 1 --2- 172.21.15.1:0/2205386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef6c15b6c0 0x7fef6c0b2160 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fef64009e80 tx=0x7fef6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=920673f38336039 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:17.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.447+0000 7fef69ffb700 1 -- 172.21.15.1:0/2205386 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6400e070 con 0x7fef6c15b6c0 2024-03-18T18:51:17.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.447+0000 7fef69ffb700 1 -- 172.21.15.1:0/2205386 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6400b8d0 con 0x7fef6c15b6c0 2024-03-18T18:51:17.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.447+0000 7fef69ffb700 1 -- 172.21.15.1:0/2205386 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6400bc80 con 0x7fef6c15b6c0 2024-03-18T18:51:17.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.448+0000 7fef71517700 1 -- 172.21.15.1:0/2205386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef6c15b6c0 msgr2=0x7fef6c0b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:17.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.448+0000 7fef71517700 1 --2- 172.21.15.1:0/2205386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef6c15b6c0 0x7fef6c0b2160 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fef64009e80 tx=0x7fef6400b330 comp rx=0 tx=0).stop 2024-03-18T18:51:17.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.448+0000 7fef71517700 1 -- 172.21.15.1:0/2205386 shutdown_connections 2024-03-18T18:51:17.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.448+0000 7fef71517700 1 --2- 172.21.15.1:0/2205386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef6c15b6c0 0x7fef6c0b2160 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.448+0000 7fef71517700 1 --2- 172.21.15.1:0/2205386 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef6c0cd430 0x7fef6c0c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.448+0000 7fef71517700 1 -- 172.21.15.1:0/2205386 >> 172.21.15.1:0/2205386 conn(0x7fef6c0d0ff0 msgr2=0x7fef6c0d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:17.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.448+0000 7fef71517700 1 -- 172.21.15.1:0/2205386 shutdown_connections 2024-03-18T18:51:17.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.448+0000 7fef71517700 1 -- 172.21.15.1:0/2205386 wait complete. 2024-03-18T18:51:17.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.449+0000 7fef71517700 1 Processor -- start 2024-03-18T18:51:17.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.449+0000 7fef71517700 1 -- start start 2024-03-18T18:51:17.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.450+0000 7fef71517700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef6c0c8600 0x7fef6c157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.450+0000 7fef71517700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef6c0cd430 0x7fef6c09e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.450+0000 7fef71517700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef6c15b6c0 0x7fef6c154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.450+0000 7fef71517700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6c1f1710 con 0x7fef6c0c8600 2024-03-18T18:51:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.450+0000 7fef71517700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6c09adc0 con 0x7fef6c15b6c0 2024-03-18T18:51:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.450+0000 7fef71517700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6c1f1b30 con 0x7fef6c0cd430 2024-03-18T18:51:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.450+0000 7fef6affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef6c0c8600 0x7fef6c157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.450+0000 7fef6affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef6c0c8600 0x7fef6c157240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39804/0 (socket says 172.21.15.1:39804) 2024-03-18T18:51:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.450+0000 7fef6affd700 1 -- 172.21.15.1:0/3433067266 learned_addr learned my addr 172.21.15.1:0/3433067266 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.450+0000 7fef6a7fc700 1 --2- 172.21.15.1:0/3433067266 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef6c0cd430 0x7fef6c09e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.450+0000 7fef6b7fe700 1 --2- 172.21.15.1:0/3433067266 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef6c15b6c0 0x7fef6c154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.451+0000 7fef6affd700 1 -- 172.21.15.1:0/3433067266 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef6c0cd430 msgr2=0x7fef6c09e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.451+0000 7fef6affd700 1 --2- 172.21.15.1:0/3433067266 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef6c0cd430 0x7fef6c09e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.451+0000 7fef6affd700 1 -- 172.21.15.1:0/3433067266 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef6c15b6c0 msgr2=0x7fef6c154450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:17.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.451+0000 7fef6affd700 1 --2- 172.21.15.1:0/3433067266 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef6c15b6c0 0x7fef6c154450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.451+0000 7fef6affd700 1 -- 172.21.15.1:0/3433067266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef64009b30 con 0x7fef6c0c8600 2024-03-18T18:51:17.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.451+0000 7fef6affd700 1 --2- 172.21.15.1:0/3433067266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef6c0c8600 0x7fef6c157240 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fef6400b370 tx=0x7fef64004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:17.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.451+0000 7fef537fe700 1 -- 172.21.15.1:0/3433067266 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef64004880 con 0x7fef6c0c8600 2024-03-18T18:51:17.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.451+0000 7fef537fe700 1 -- 172.21.15.1:0/3433067266 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef640099b0 con 0x7fef6c0c8600 2024-03-18T18:51:17.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.451+0000 7fef71517700 1 -- 172.21.15.1:0/3433067266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef6c09ebe0 con 0x7fef6c0c8600 2024-03-18T18:51:17.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.451+0000 7fef537fe700 1 -- 172.21.15.1:0/3433067266 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6401c680 con 0x7fef6c0c8600 2024-03-18T18:51:17.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.451+0000 7fef71517700 1 -- 172.21.15.1:0/3433067266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef6c1f1050 con 0x7fef6c0c8600 2024-03-18T18:51:17.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.452+0000 7fef71517700 1 -- 172.21.15.1:0/3433067266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef6c0676a0 con 0x7fef6c0c8600 2024-03-18T18:51:17.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.452+0000 7fef537fe700 1 -- 172.21.15.1:0/3433067266 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fef6401c7e0 con 0x7fef6c0c8600 2024-03-18T18:51:17.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.453+0000 7fef537fe700 1 --2- 172.21.15.1:0/3433067266 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fef5806ac90 0x7fef5806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.453+0000 7fef537fe700 1 -- 172.21.15.1:0/3433067266 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fef6408ad50 con 0x7fef6c0c8600 2024-03-18T18:51:17.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.453+0000 7fef6a7fc700 1 --2- 172.21.15.1:0/3433067266 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fef5806ac90 0x7fef5806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.453+0000 7fef6a7fc700 1 --2- 172.21.15.1:0/3433067266 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fef5806ac90 0x7fef5806d150 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fef6c070230 tx=0x7fef60009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:17.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.457+0000 7fef537fe700 1 -- 172.21.15.1:0/3433067266 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fef6408e030 con 0x7fef6c0c8600 2024-03-18T18:51:17.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.743+0000 7fef71517700 1 -- 172.21.15.1:0/3433067266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fef6c0c8be0 con 0x7fef6c0c8600 2024-03-18T18:51:17.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.747+0000 7fef537fe700 1 -- 172.21.15.1:0/3433067266 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v9) v1 ==== 190+0+0 (secure 0 0 0) 0x7fef64055e80 con 0x7fef6c0c8600 2024-03-18T18:51:17.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.749+0000 7fef71517700 1 -- 172.21.15.1:0/3433067266 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fef5806ac90 msgr2=0x7fef5806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:17.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.749+0000 7fef71517700 1 --2- 172.21.15.1:0/3433067266 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fef5806ac90 0x7fef5806d150 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fef6c070230 tx=0x7fef60009210 comp rx=0 tx=0).stop 2024-03-18T18:51:17.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.749+0000 7fef71517700 1 -- 172.21.15.1:0/3433067266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef6c0c8600 msgr2=0x7fef6c157240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:17.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.749+0000 7fef71517700 1 --2- 172.21.15.1:0/3433067266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef6c0c8600 0x7fef6c157240 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fef6400b370 tx=0x7fef64004ad0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.749+0000 7fef71517700 1 -- 172.21.15.1:0/3433067266 shutdown_connections 2024-03-18T18:51:17.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.749+0000 7fef71517700 1 --2- 172.21.15.1:0/3433067266 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fef5806ac90 0x7fef5806d150 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.749+0000 7fef71517700 1 --2- 172.21.15.1:0/3433067266 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef6c15b6c0 0x7fef6c154450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.749+0000 7fef71517700 1 --2- 172.21.15.1:0/3433067266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef6c0c8600 0x7fef6c157240 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.749+0000 7fef71517700 1 --2- 172.21.15.1:0/3433067266 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef6c0cd430 0x7fef6c09e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.749+0000 7fef71517700 1 -- 172.21.15.1:0/3433067266 >> 172.21.15.1:0/3433067266 conn(0x7fef6c0d0ff0 msgr2=0x7fef6c0cebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:17.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.749+0000 7fef71517700 1 -- 172.21.15.1:0/3433067266 shutdown_connections 2024-03-18T18:51:17.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.749+0000 7fef71517700 1 -- 172.21.15.1:0/3433067266 wait complete. 2024-03-18T18:51:17.747 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-18T18:51:17.758 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T18:51:17.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.887+0000 7fc87082c700 1 Processor -- start 2024-03-18T18:51:17.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.887+0000 7fc87082c700 1 -- start start 2024-03-18T18:51:17.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.888+0000 7fc87082c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc86c1465a0 0x7fc86c168660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.888+0000 7fc87082c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c072660 con 0x7fc86c06ae60 2024-03-18T18:51:17.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.888+0000 7fc87082c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c06ebd0 con 0x7fc86c146190 2024-03-18T18:51:17.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.888+0000 7fc87082c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc86c06f850 con 0x7fc86c1465a0 2024-03-18T18:51:17.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.888+0000 7fc869d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc86c06ae60 0x7fc86c164f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56422/0 (socket says 172.21.15.1:56422) 2024-03-18T18:51:17.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.888+0000 7fc869d9b700 1 -- 172.21.15.1:0/1440032084 learned_addr learned my addr 172.21.15.1:0/1440032084 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:17.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.888+0000 7fc86959a700 1 -- 172.21.15.1:0/1440032084 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725314192 0 0) 0x7fc86c072660 con 0x7fc86c06ae60 2024-03-18T18:51:17.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.888+0000 7fc86959a700 1 -- 172.21.15.1:0/1440032084 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8540035b0 con 0x7fc86c06ae60 2024-03-18T18:51:17.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.888+0000 7fc86ad9d700 1 --2- 172.21.15.1:0/1440032084 >> v2:172.21.15.165:3300/0 conn(0x7fc86c1465a0 0x7fc86c168660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc86959a700 1 -- 172.21.15.1:0/1440032084 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2056758218 0 0) 0x7fc8540035b0 con 0x7fc86c06ae60 2024-03-18T18:51:17.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc86959a700 1 -- 172.21.15.1:0/1440032084 >> v2:172.21.15.165:3300/0 conn(0x7fc86c1465a0 msgr2=0x7fc86c168660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:17.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc86959a700 1 --2- 172.21.15.1:0/1440032084 >> v2:172.21.15.165:3300/0 conn(0x7fc86c1465a0 0x7fc86c168660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc86959a700 1 -- 172.21.15.1:0/1440032084 >> v1:172.21.15.165:6789/0 conn(0x7fc86c146190 legacy=0x7fc86c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:17.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc86959a700 1 -- 172.21.15.1:0/1440032084 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c06eeb0 con 0x7fc86c06ae60 2024-03-18T18:51:17.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc86959a700 1 -- 172.21.15.1:0/1440032084 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fc85c001670 con 0x7fc86c06ae60 2024-03-18T18:51:17.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc86959a700 1 -- 172.21.15.1:0/1440032084 >> v1:172.21.15.1:6789/0 conn(0x7fc86c06ae60 legacy=0x7fc86c164f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:17.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc86959a700 1 --2- 172.21.15.1:0/1440032084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc854003a50 0x7fc854003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc86959a700 1 --2- 172.21.15.1:0/1440032084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc854004850 0x7fc854004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc86959a700 1 --2- 172.21.15.1:0/1440032084 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc854005210 0x7fc85400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc86959a700 1 -- 172.21.15.1:0/1440032084 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c06f850 con 0x7fc854004850 2024-03-18T18:51:17.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc86959a700 1 -- 172.21.15.1:0/1440032084 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85400c110 con 0x7fc854003a50 2024-03-18T18:51:17.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc86959a700 1 -- 172.21.15.1:0/1440032084 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85400c2d0 con 0x7fc854005210 2024-03-18T18:51:17.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc869d9b700 1 --2- 172.21.15.1:0/1440032084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc854004850 0x7fc854004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc86a59c700 1 --2- 172.21.15.1:0/1440032084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc854003a50 0x7fc854003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc86ad9d700 1 --2- 172.21.15.1:0/1440032084 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc854005210 0x7fc85400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc869d9b700 1 -- 172.21.15.1:0/1440032084 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc854005210 msgr2=0x7fc85400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:17.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc869d9b700 1 --2- 172.21.15.1:0/1440032084 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc854005210 0x7fc85400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc869d9b700 1 -- 172.21.15.1:0/1440032084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc854003a50 msgr2=0x7fc854003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:17.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc869d9b700 1 --2- 172.21.15.1:0/1440032084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc854003a50 0x7fc854003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc869d9b700 1 -- 172.21.15.1:0/1440032084 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8540035b0 con 0x7fc854004850 2024-03-18T18:51:17.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc86ad9d700 1 -- 172.21.15.1:0/1440032084 reap_dead start 2024-03-18T18:51:17.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc869d9b700 1 --2- 172.21.15.1:0/1440032084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc854004850 0x7fc854004cd0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc85c00c160 tx=0x7fc854005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c31e0a70c8d10a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:17.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc86959a700 1 -- 172.21.15.1:0/1440032084 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c007b70 con 0x7fc854004850 2024-03-18T18:51:17.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc86959a700 1 -- 172.21.15.1:0/1440032084 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85c00cd20 con 0x7fc854004850 2024-03-18T18:51:17.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.889+0000 7fc86959a700 1 -- 172.21.15.1:0/1440032084 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c005ba0 con 0x7fc854004850 2024-03-18T18:51:17.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.890+0000 7fc87082c700 1 -- 172.21.15.1:0/1440032084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc854004850 msgr2=0x7fc854004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:17.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.890+0000 7fc87082c700 1 --2- 172.21.15.1:0/1440032084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc854004850 0x7fc854004cd0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc85c00c160 tx=0x7fc854005880 comp rx=0 tx=0).stop 2024-03-18T18:51:17.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.890+0000 7fc87082c700 1 -- 172.21.15.1:0/1440032084 shutdown_connections 2024-03-18T18:51:17.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.890+0000 7fc87082c700 1 --2- 172.21.15.1:0/1440032084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc854004850 0x7fc854004cd0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.890+0000 7fc87082c700 1 -- 172.21.15.1:0/1440032084 >> 172.21.15.1:0/1440032084 conn(0x7fc86c0b1d20 msgr2=0x7fc86c0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:17.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.890+0000 7fc87082c700 1 -- 172.21.15.1:0/1440032084 shutdown_connections 2024-03-18T18:51:17.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.890+0000 7fc87082c700 1 -- 172.21.15.1:0/1440032084 wait complete. 2024-03-18T18:51:17.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.891+0000 7fc87082c700 1 Processor -- start 2024-03-18T18:51:17.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.891+0000 7fc87082c700 1 -- start start 2024-03-18T18:51:17.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.892+0000 7fc87082c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc854004850 0x7fc86c142f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.892+0000 7fc87082c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc86c133890 0x7fc86c0c5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.892+0000 7fc87082c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc86c138850 0x7fc86c138cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.892+0000 7fc87082c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c133e00 con 0x7fc86c133890 2024-03-18T18:51:17.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.892+0000 7fc87082c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c1340e0 con 0x7fc854004850 2024-03-18T18:51:17.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.892+0000 7fc87082c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c1343c0 con 0x7fc86c138850 2024-03-18T18:51:17.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.892+0000 7fc869d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc86c133890 0x7fc86c0c5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.892+0000 7fc869d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc86c133890 0x7fc86c0c5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39820/0 (socket says 172.21.15.1:39820) 2024-03-18T18:51:17.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.892+0000 7fc869d9b700 1 -- 172.21.15.1:0/2675661398 learned_addr learned my addr 172.21.15.1:0/2675661398 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:17.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.892+0000 7fc86a59c700 1 --2- 172.21.15.1:0/2675661398 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc854004850 0x7fc86c142f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.892+0000 7fc86ad9d700 1 --2- 172.21.15.1:0/2675661398 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc86c138850 0x7fc86c138cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.893+0000 7fc869d9b700 1 -- 172.21.15.1:0/2675661398 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc86c138850 msgr2=0x7fc86c138cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:17.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.893+0000 7fc869d9b700 1 --2- 172.21.15.1:0/2675661398 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc86c138850 0x7fc86c138cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.893+0000 7fc869d9b700 1 -- 172.21.15.1:0/2675661398 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc854004850 msgr2=0x7fc86c142f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:17.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.893+0000 7fc869d9b700 1 --2- 172.21.15.1:0/2675661398 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc854004850 0x7fc86c142f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:17.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.893+0000 7fc869d9b700 1 -- 172.21.15.1:0/2675661398 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c1346a0 con 0x7fc86c133890 2024-03-18T18:51:17.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.893+0000 7fc86ad9d700 1 --2- 172.21.15.1:0/2675661398 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc86c138850 0x7fc86c138cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:17.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.893+0000 7fc869d9b700 1 --2- 172.21.15.1:0/2675661398 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc86c133890 0x7fc86c0c5200 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc85400c0e0 tx=0x7fc85c001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:17.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.893+0000 7fc85affd700 1 -- 172.21.15.1:0/2675661398 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c004c80 con 0x7fc86c133890 2024-03-18T18:51:17.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.893+0000 7fc85affd700 1 -- 172.21.15.1:0/2675661398 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85c004e20 con 0x7fc86c133890 2024-03-18T18:51:17.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.893+0000 7fc87082c700 1 -- 172.21.15.1:0/2675661398 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc86c134980 con 0x7fc86c133890 2024-03-18T18:51:17.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.893+0000 7fc85affd700 1 -- 172.21.15.1:0/2675661398 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c007310 con 0x7fc86c133890 2024-03-18T18:51:17.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.893+0000 7fc87082c700 1 -- 172.21.15.1:0/2675661398 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc86c143480 con 0x7fc86c133890 2024-03-18T18:51:17.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.894+0000 7fc87082c700 1 -- 172.21.15.1:0/2675661398 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc86c0676a0 con 0x7fc86c133890 2024-03-18T18:51:17.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.894+0000 7fc85affd700 1 -- 172.21.15.1:0/2675661398 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc85c007470 con 0x7fc86c133890 2024-03-18T18:51:17.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.895+0000 7fc85affd700 1 --2- 172.21.15.1:0/2675661398 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc84c06ac90 0x7fc84c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:17.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.895+0000 7fc85affd700 1 -- 172.21.15.1:0/2675661398 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc85c089ca0 con 0x7fc86c133890 2024-03-18T18:51:17.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.895+0000 7fc86a59c700 1 --2- 172.21.15.1:0/2675661398 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc84c06ac90 0x7fc84c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:17.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.895+0000 7fc86a59c700 1 --2- 172.21.15.1:0/2675661398 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc84c06ac90 0x7fc84c06d150 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc860000c00 tx=0x7fc860009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:17.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:17.899+0000 7fc85affd700 1 -- 172.21.15.1:0/2675661398 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc85c054dd0 con 0x7fc86c133890 2024-03-18T18:51:18.017 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:51:18.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:51:18.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.148+0000 7fc87082c700 1 -- 172.21.15.1:0/2675661398 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc86c134cf0 con 0x7fc86c133890 2024-03-18T18:51:18.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.149+0000 7fc85affd700 1 -- 172.21.15.1:0/2675661398 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4734 (secure 0 0 0) 0x7fc85c017090 con 0x7fc86c133890 2024-03-18T18:51:18.145 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T18:51:18.145 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4874,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2002167042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2002167042},{"type":"v1","addr":"172.21.15.165:6835","nonce":2002167042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4880,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3844556325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3844556325},{"type":"v1","addr":"172.21.15.1:6837","nonce":3844556325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5278,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1229938169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1229938169},{"type":"v1","addr":"172.21.15.165:6834","nonce":1229938169}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T18:51:12.721816+0000","modified":"2024-03-18T18:51:15.728690+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5436":{"gid":5436,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6836/3077516107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3077516107},{"type":"v1","addr":"172.21.15.1:6836","nonce":3077516107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-18T18:51:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.151+0000 7fc87082c700 1 -- 172.21.15.1:0/2675661398 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc84c06ac90 msgr2=0x7fc84c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.151+0000 7fc87082c700 1 --2- 172.21.15.1:0/2675661398 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc84c06ac90 0x7fc84c06d150 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc860000c00 tx=0x7fc860009040 comp rx=0 tx=0).stop 2024-03-18T18:51:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.151+0000 7fc87082c700 1 -- 172.21.15.1:0/2675661398 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc86c133890 msgr2=0x7fc86c0c5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.151+0000 7fc87082c700 1 --2- 172.21.15.1:0/2675661398 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc86c133890 0x7fc86c0c5200 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc85400c0e0 tx=0x7fc85c001930 comp rx=0 tx=0).stop 2024-03-18T18:51:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.151+0000 7fc87082c700 1 -- 172.21.15.1:0/2675661398 shutdown_connections 2024-03-18T18:51:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.151+0000 7fc87082c700 1 --2- 172.21.15.1:0/2675661398 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc84c06ac90 0x7fc84c06d150 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.151+0000 7fc87082c700 1 --2- 172.21.15.1:0/2675661398 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc86c138850 0x7fc86c138cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.151+0000 7fc87082c700 1 --2- 172.21.15.1:0/2675661398 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc854004850 0x7fc86c142f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.151+0000 7fc87082c700 1 --2- 172.21.15.1:0/2675661398 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc86c133890 0x7fc86c0c5200 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.151+0000 7fc87082c700 1 -- 172.21.15.1:0/2675661398 >> 172.21.15.1:0/2675661398 conn(0x7fc86c0b1d20 msgr2=0x7fc86c0adf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.151+0000 7fc87082c700 1 -- 172.21.15.1:0/2675661398 shutdown_connections 2024-03-18T18:51:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.151+0000 7fc87082c700 1 -- 172.21.15.1:0/2675661398 wait complete. 2024-03-18T18:51:18.149 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 17 2024-03-18T18:51:18.562 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T18:51:12.721816+0000', 'modified': '2024-03-18T18:51:15.728690+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5436}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5436': {'gid': 5436, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6836/3077516107', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 3077516107}, {'type': 'v1', 'addr': '172.21.15.1:6836', 'nonce': 3077516107}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T18:51:18.563 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T18:51:18.563 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T18:51:18.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.697+0000 7f93a67e5700 1 Processor -- start 2024-03-18T18:51:18.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.697+0000 7f93a67e5700 1 -- start start 2024-03-18T18:51:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.698+0000 7f93a67e5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93a0157ba0 0x7f93a0070db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.698+0000 7f93a67e5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f93a015e5e0 0x7f93a00712f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.698+0000 7f93a67e5700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a015a590 con 0x7f93a006e2d0 2024-03-18T18:51:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.698+0000 7f93a67e5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a015eba0 con 0x7f93a015e5e0 2024-03-18T18:51:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.698+0000 7f93a67e5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a015b590 con 0x7f93a0157ba0 2024-03-18T18:51:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.699+0000 7f939ffff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93a0157ba0 0x7f93a0070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.699+0000 7f93a4d82700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f93a015e5e0 0x7f93a00712f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.699+0000 7f939f7fe700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f93a006e2d0 0x7f93a00b25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:47710/0 (socket says 172.21.15.1:47710) 2024-03-18T18:51:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.699+0000 7f939f7fe700 1 -- 172.21.15.1:0/2734189811 learned_addr learned my addr 172.21.15.1:0/2734189811 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.699+0000 7f939ffff700 1 -- 172.21.15.1:0/2734189811 >> v2:172.21.15.165:3300/0 conn(0x7f93a015e5e0 msgr2=0x7f93a00712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.699+0000 7f939ffff700 1 --2- 172.21.15.1:0/2734189811 >> v2:172.21.15.165:3300/0 conn(0x7f93a015e5e0 0x7f93a00712f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.699+0000 7f939ffff700 1 -- 172.21.15.1:0/2734189811 >> v1:172.21.15.165:6789/0 conn(0x7f93a006e2d0 legacy=0x7f93a00b25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.699+0000 7f939effd700 1 -- 172.21.15.1:0/2734189811 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977099167 0 0) 0x7f9394001110 con 0x7f93a006e2d0 2024-03-18T18:51:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.699+0000 7f939ffff700 1 -- 172.21.15.1:0/2734189811 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a00b3640 con 0x7f93a0157ba0 2024-03-18T18:51:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.699+0000 7f93a4d82700 1 --2- 172.21.15.1:0/2734189811 >> v2:172.21.15.165:3300/0 conn(0x7f93a015e5e0 0x7f93a00712f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:51:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.699+0000 7f939ffff700 1 --2- 172.21.15.1:0/2734189811 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93a0157ba0 0x7f93a0070db0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9390009840 tx=0x7f9390009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8afba5fbf2b048f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.699+0000 7f939effd700 1 -- 172.21.15.1:0/2734189811 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9390011830 con 0x7f93a0157ba0 2024-03-18T18:51:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.699+0000 7f939effd700 1 -- 172.21.15.1:0/2734189811 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9390011990 con 0x7f93a0157ba0 2024-03-18T18:51:18.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.699+0000 7f939effd700 1 -- 172.21.15.1:0/2734189811 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9390011d20 con 0x7f93a0157ba0 2024-03-18T18:51:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.700+0000 7f93a67e5700 1 -- 172.21.15.1:0/2734189811 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93a0157ba0 msgr2=0x7f93a0070db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.700+0000 7f93a67e5700 1 --2- 172.21.15.1:0/2734189811 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93a0157ba0 0x7f93a0070db0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9390009840 tx=0x7f9390009b50 comp rx=0 tx=0).stop 2024-03-18T18:51:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.700+0000 7f93a67e5700 1 -- 172.21.15.1:0/2734189811 shutdown_connections 2024-03-18T18:51:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.700+0000 7f93a67e5700 1 --2- 172.21.15.1:0/2734189811 >> v2:172.21.15.165:3300/0 conn(0x7f93a015e5e0 0x7f93a00712f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.700+0000 7f93a67e5700 1 --2- 172.21.15.1:0/2734189811 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93a0157ba0 0x7f93a0070db0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.700+0000 7f93a67e5700 1 -- 172.21.15.1:0/2734189811 >> 172.21.15.1:0/2734189811 conn(0x7f93a006a750 msgr2=0x7f93a006ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.700+0000 7f93a67e5700 1 -- 172.21.15.1:0/2734189811 shutdown_connections 2024-03-18T18:51:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.700+0000 7f93a67e5700 1 -- 172.21.15.1:0/2734189811 wait complete. 2024-03-18T18:51:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.701+0000 7f93a67e5700 1 Processor -- start 2024-03-18T18:51:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.701+0000 7f93a67e5700 1 -- start start 2024-03-18T18:51:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.702+0000 7f93a67e5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93a006e2d0 0x7f93a014c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.702+0000 7f93a67e5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93a0157ba0 0x7f93a0150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.702+0000 7f93a67e5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93a015e5e0 0x7f93a00afc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.702+0000 7f93a67e5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a00af250 con 0x7f93a015e5e0 2024-03-18T18:51:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.702+0000 7f93a67e5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a00afa60 con 0x7f93a006e2d0 2024-03-18T18:51:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.702+0000 7f93a67e5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a00b1d60 con 0x7f93a0157ba0 2024-03-18T18:51:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.702+0000 7f93a4d82700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93a015e5e0 0x7f93a00afc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.702+0000 7f93a4d82700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93a015e5e0 0x7f93a00afc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39832/0 (socket says 172.21.15.1:39832) 2024-03-18T18:51:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.702+0000 7f93a4d82700 1 -- 172.21.15.1:0/1608362547 learned_addr learned my addr 172.21.15.1:0/1608362547 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:18.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.702+0000 7f939f7fe700 1 --2- 172.21.15.1:0/1608362547 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93a0157ba0 0x7f93a0150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.702+0000 7f939ffff700 1 --2- 172.21.15.1:0/1608362547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93a006e2d0 0x7f93a014c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.703+0000 7f93a4d82700 1 -- 172.21.15.1:0/1608362547 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93a0157ba0 msgr2=0x7f93a0150220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.703+0000 7f93a4d82700 1 --2- 172.21.15.1:0/1608362547 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93a0157ba0 0x7f93a0150220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.703+0000 7f93a4d82700 1 -- 172.21.15.1:0/1608362547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93a006e2d0 msgr2=0x7f93a014c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.703+0000 7f93a4d82700 1 --2- 172.21.15.1:0/1608362547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93a006e2d0 0x7f93a014c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.703+0000 7f93a4d82700 1 -- 172.21.15.1:0/1608362547 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9398009a70 con 0x7f93a015e5e0 2024-03-18T18:51:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.703+0000 7f93a4d82700 1 --2- 172.21.15.1:0/1608362547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93a015e5e0 0x7f93a00afc40 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f9398009f00 tx=0x7f939800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.703+0000 7f939cff9700 1 -- 172.21.15.1:0/1608362547 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9398009d70 con 0x7f93a015e5e0 2024-03-18T18:51:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.703+0000 7f939cff9700 1 -- 172.21.15.1:0/1608362547 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939800f040 con 0x7f93a015e5e0 2024-03-18T18:51:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.703+0000 7f939cff9700 1 -- 172.21.15.1:0/1608362547 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939800cd70 con 0x7f93a015e5e0 2024-03-18T18:51:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.703+0000 7f93a67e5700 1 -- 172.21.15.1:0/1608362547 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93900094a0 con 0x7f93a015e5e0 2024-03-18T18:51:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.703+0000 7f93a67e5700 1 -- 172.21.15.1:0/1608362547 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a00af780 con 0x7f93a015e5e0 2024-03-18T18:51:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.705+0000 7f939cff9700 1 -- 172.21.15.1:0/1608362547 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9398014480 con 0x7f93a015e5e0 2024-03-18T18:51:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.705+0000 7f939cff9700 1 --2- 172.21.15.1:0/1608362547 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f938c06ac90 0x7f938c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.705+0000 7f939cff9700 1 -- 172.21.15.1:0/1608362547 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9398089a50 con 0x7f93a015e5e0 2024-03-18T18:51:18.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.705+0000 7f939ffff700 1 --2- 172.21.15.1:0/1608362547 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f938c06ac90 0x7f938c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.705+0000 7f939ffff700 1 --2- 172.21.15.1:0/1608362547 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f938c06ac90 0x7f938c06d150 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9390000c00 tx=0x7f9390008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.707+0000 7f93b298eb80 1 Processor -- start 2024-03-18T18:51:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.707+0000 7f93b298eb80 1 -- start start 2024-03-18T18:51:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.707+0000 7f93b298eb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616315c10 0x55c616315ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.707+0000 7f93b298eb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c6162e70a0 0x55c616332580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.707+0000 7f93b298eb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c616327630 0x55c6162ea160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.707+0000 7f93b298eb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c616260f20 con 0x55c6162e70a0 2024-03-18T18:51:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.707+0000 7f93b298eb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c616242c30 con 0x55c616327630 2024-03-18T18:51:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.707+0000 7f93b298eb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c616240430 con 0x55c616315c10 2024-03-18T18:51:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.708+0000 7f939f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c6162e70a0 0x55c616332580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.708+0000 7f939f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c6162e70a0 0x55c616332580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39836/0 (socket says 172.21.15.1:39836) 2024-03-18T18:51:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.708+0000 7f939f7fe700 1 -- 172.21.15.1:0/2952727724 learned_addr learned my addr 172.21.15.1:0/2952727724 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.708+0000 7f93a4d82700 1 --2- 172.21.15.1:0/2952727724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c616327630 0x55c6162ea160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.708+0000 7f939ffff700 1 --2- 172.21.15.1:0/2952727724 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616315c10 0x55c616315ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.708+0000 7f939f7fe700 1 -- 172.21.15.1:0/2952727724 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616315c10 msgr2=0x55c616315ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.708+0000 7f939f7fe700 1 --2- 172.21.15.1:0/2952727724 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616315c10 0x55c616315ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.708+0000 7f939f7fe700 1 -- 172.21.15.1:0/2952727724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c616327630 msgr2=0x55c6162ea160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.708+0000 7f939f7fe700 1 --2- 172.21.15.1:0/2952727724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c616327630 0x55c6162ea160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.708+0000 7f939f7fe700 1 -- 172.21.15.1:0/2952727724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c61602f790 con 0x55c6162e70a0 2024-03-18T18:51:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.708+0000 7f939ffff700 1 --2- 172.21.15.1:0/2952727724 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616315c10 0x55c616315ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.708+0000 7f939f7fe700 1 --2- 172.21.15.1:0/2952727724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c6162e70a0 0x55c616332580 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9394004e70 tx=0x7f9394007340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aeca437a070dfee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.708+0000 7f9385ffb700 1 -- 172.21.15.1:0/2952727724 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93940014a0 con 0x55c6162e70a0 2024-03-18T18:51:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.708+0000 7f9385ffb700 1 -- 172.21.15.1:0/2952727724 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9394001600 con 0x55c6162e70a0 2024-03-18T18:51:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.708+0000 7f9385ffb700 1 -- 172.21.15.1:0/2952727724 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9394001990 con 0x55c6162e70a0 2024-03-18T18:51:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.709+0000 7f93b298eb80 1 -- 172.21.15.1:0/2952727724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c6162e70a0 msgr2=0x55c616332580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.709+0000 7f93b298eb80 1 --2- 172.21.15.1:0/2952727724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c6162e70a0 0x55c616332580 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9394004e70 tx=0x7f9394007340 comp rx=0 tx=0).stop 2024-03-18T18:51:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.709+0000 7f93b298eb80 1 -- 172.21.15.1:0/2952727724 shutdown_connections 2024-03-18T18:51:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.709+0000 7f93b298eb80 1 --2- 172.21.15.1:0/2952727724 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616315c10 0x55c616315ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.709+0000 7f93b298eb80 1 --2- 172.21.15.1:0/2952727724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c616327630 0x55c6162ea160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.709+0000 7f93b298eb80 1 --2- 172.21.15.1:0/2952727724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c6162e70a0 0x55c616332580 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.709+0000 7f93b298eb80 1 -- 172.21.15.1:0/2952727724 >> 172.21.15.1:0/2952727724 conn(0x55c616249cf0 msgr2=0x55c6162e9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.709+0000 7f93b298eb80 1 -- 172.21.15.1:0/2952727724 shutdown_connections 2024-03-18T18:51:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.709+0000 7f93b298eb80 1 -- 172.21.15.1:0/2952727724 wait complete. 2024-03-18T18:51:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.710+0000 7f93b298eb80 1 -- start start 2024-03-18T18:51:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.710+0000 7f93b298eb80 1 Processor -- start 2024-03-18T18:51:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.710+0000 7f93b298eb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c6162e70a0 0x55c61630f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.710+0000 7f93b298eb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 0x55c61630dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f93b298eb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616327630 0x55c6163343b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f93b298eb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c61623bd20 con 0x55c616315c10 2024-03-18T18:51:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f93b298eb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6161d7770 con 0x55c6162e70a0 2024-03-18T18:51:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f93b298eb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c6161fa6f0 con 0x55c616327630 2024-03-18T18:51:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f93b298eb80 10 client.0 _ll_register_callbacks cb 0x55c6160435a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T18:51:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f939f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 0x55c61630dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f939f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 0x55c61630dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39840/0 (socket says 172.21.15.1:39840) 2024-03-18T18:51:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f939f7fe700 1 -- 172.21.15.1:0/2582101210 learned_addr learned my addr 172.21.15.1:0/2582101210 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f939ffff700 1 --2- 172.21.15.1:0/2582101210 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c6162e70a0 0x55c61630f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f93a4d82700 1 --2- 172.21.15.1:0/2582101210 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616327630 0x55c6163343b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f939f7fe700 1 -- 172.21.15.1:0/2582101210 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616327630 msgr2=0x55c6163343b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f939f7fe700 1 --2- 172.21.15.1:0/2582101210 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616327630 0x55c6163343b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f939f7fe700 1 -- 172.21.15.1:0/2582101210 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c6162e70a0 msgr2=0x55c61630f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f939f7fe700 1 --2- 172.21.15.1:0/2582101210 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c6162e70a0 0x55c61630f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f939f7fe700 1 -- 172.21.15.1:0/2582101210 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9394004aa0 con 0x55c616315c10 2024-03-18T18:51:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f939ffff700 1 --2- 172.21.15.1:0/2582101210 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c6162e70a0 0x55c61630f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:18.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f939f7fe700 1 --2- 172.21.15.1:0/2582101210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 0x55c61630dd20 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x55c6162e9970 tx=0x7f9394012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d8e5fda2635471a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:18.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f937effd700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T18:51:18.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f937effd700 1 -- 172.21.15.1:0/2582101210 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939401c860 con 0x55c616315c10 2024-03-18T18:51:18.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f937effd700 1 -- 172.21.15.1:0/2582101210 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9394007b30 con 0x55c616315c10 2024-03-18T18:51:18.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f93b298eb80 1 -- 172.21.15.1:0/2582101210 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c6162dde90 con 0x55c616315c10 2024-03-18T18:51:18.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f937effd700 1 -- 172.21.15.1:0/2582101210 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f939401c9c0 con 0x55c616315c10 2024-03-18T18:51:18.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f93b298eb80 10 client.5601 fetch_fsmap learned FSMap version 17 2024-03-18T18:51:18.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.711+0000 7f93b298eb80 1 -- 172.21.15.1:0/2582101210 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55c61611c0a0 con 0x55c616315c10 2024-03-18T18:51:18.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.712+0000 7f937effd700 1 -- 172.21.15.1:0/2582101210 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c6162dde90 con 0x55c616315c10 2024-03-18T18:51:18.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.712+0000 7f937effd700 1 -- 172.21.15.1:0/2582101210 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 17) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9394012070 con 0x55c616315c10 2024-03-18T18:51:18.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.712+0000 7f93b298eb80 10 client.5601 fetch_fsmap finished waiting for FSMap version 17 2024-03-18T18:51:18.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.712+0000 7f93b298eb80 10 client.5601 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T18:51:18.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.712+0000 7f93b298eb80 20 client.5601 populate_metadata read hostname 'smithi001' 2024-03-18T18:51:18.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.712+0000 7f93b298eb80 1 --2- 172.21.15.1:0/2582101210 >> [v2:172.21.15.1:6834/3077516107,v1:172.21.15.1:6836/3077516107] conn(0x55c616332610 0x55c6163329f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.712+0000 7f93b298eb80 4 client.5601 mds_command: new command op to 5436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T18:51:18.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.712+0000 7f93b298eb80 1 -- 172.21.15.1:0/2582101210 --> [v2:172.21.15.1:6834/3077516107,v1:172.21.15.1:6836/3077516107] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c6162bb010 con 0x55c616332610 2024-03-18T18:51:18.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.712+0000 7f939ffff700 1 --2- 172.21.15.1:0/2582101210 >> [v2:172.21.15.1:6834/3077516107,v1:172.21.15.1:6836/3077516107] conn(0x55c616332610 0x55c6163329f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.712+0000 7f937effd700 10 client.5601 ms_handle_connect on v2:172.21.15.1:6834/3077516107 2024-03-18T18:51:18.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.712+0000 7f939ffff700 1 --2- 172.21.15.1:0/2582101210 >> [v2:172.21.15.1:6834/3077516107,v1:172.21.15.1:6836/3077516107] conn(0x55c616332610 0x55c6163329f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9460c040e65b3988 server_cookie=76d689daa2906acd in_seq=0 out_seq=0 2024-03-18T18:51:18.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.713+0000 7f937effd700 1 -- 172.21.15.1:0/2582101210 <== mds.0 v2:172.21.15.1:6834/3077516107 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c6162bb010 con 0x55c616332610 2024-03-18T18:51:18.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.713+0000 7f937effd700 10 client.5601 handle_command_reply: tid=0 2024-03-18T18:51:18.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.713+0000 7f93b298eb80 1 client.5601 shutdown 2024-03-18T18:51:18.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.713+0000 7f93b298eb80 1 -- 172.21.15.1:0/2582101210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 msgr2=0x55c61630dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:18.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.713+0000 7f93b298eb80 1 --2- 172.21.15.1:0/2582101210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 0x55c61630dd20 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x55c6162e9970 tx=0x7f9394012000 comp rx=0 tx=0).stop 2024-03-18T18:51:18.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.713+0000 7f93b298eb80 1 -- 172.21.15.1:0/2582101210 shutdown_connections 2024-03-18T18:51:18.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.713+0000 7f93b298eb80 1 --2- 172.21.15.1:0/2582101210 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616327630 0x55c6163343b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.713+0000 7f93b298eb80 1 --2- 172.21.15.1:0/2582101210 >> [v2:172.21.15.1:6834/3077516107,v1:172.21.15.1:6836/3077516107] conn(0x55c616332610 0x55c6163329f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.713+0000 7f93b298eb80 1 --2- 172.21.15.1:0/2582101210 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c6162e70a0 0x55c61630f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.713+0000 7f93b298eb80 1 --2- 172.21.15.1:0/2582101210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 0x55c61630dd20 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.713+0000 7f937effd700 0 client.5601 ms_handle_reset on v2:172.21.15.1:6834/3077516107 2024-03-18T18:51:18.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.713+0000 7f93b298eb80 1 -- 172.21.15.1:0/2582101210 >> 172.21.15.1:0/2582101210 conn(0x55c616249cf0 msgr2=0x55c6162e5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:18.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.713+0000 7f93b298eb80 1 -- 172.21.15.1:0/2582101210 shutdown_connections 2024-03-18T18:51:18.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.713+0000 7f93b298eb80 1 -- 172.21.15.1:0/2582101210 wait complete. 2024-03-18T18:51:18.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.713+0000 7f93b298eb80 20 client.5601 trim_cache size 0 max 16384 2024-03-18T18:51:18.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.731+0000 7f93b298eb80 1 Processor -- start 2024-03-18T18:51:18.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.731+0000 7f93b298eb80 1 -- start start 2024-03-18T18:51:18.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.732+0000 7f93b298eb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c6162e70a0 0x55c616331140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.732+0000 7f93b298eb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 0x55c6162e8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.732+0000 7f93b298eb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616327630 0x55c616313470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.732+0000 7f93b298eb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c616030390 con 0x55c616315c10 2024-03-18T18:51:18.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.732+0000 7f93b298eb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6163166e0 con 0x55c6162e70a0 2024-03-18T18:51:18.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.732+0000 7f93b298eb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c6162e6f20 con 0x55c616327630 2024-03-18T18:51:18.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.732+0000 7f939f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 0x55c6162e8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.732+0000 7f939f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 0x55c6162e8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39854/0 (socket says 172.21.15.1:39854) 2024-03-18T18:51:18.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.732+0000 7f939f7fe700 1 -- 172.21.15.1:0/1957599055 learned_addr learned my addr 172.21.15.1:0/1957599055 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.732+0000 7f939f7fe700 1 -- 172.21.15.1:0/1957599055 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616327630 msgr2=0x55c616313470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.732+0000 7f939f7fe700 1 --2- 172.21.15.1:0/1957599055 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616327630 0x55c616313470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.732+0000 7f939ffff700 1 --2- 172.21.15.1:0/1957599055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c6162e70a0 0x55c616331140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.732+0000 7f939f7fe700 1 -- 172.21.15.1:0/1957599055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c6162e70a0 msgr2=0x55c616331140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.732+0000 7f939f7fe700 1 --2- 172.21.15.1:0/1957599055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c6162e70a0 0x55c616331140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.732+0000 7f939f7fe700 1 -- 172.21.15.1:0/1957599055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c616202010 con 0x55c616315c10 2024-03-18T18:51:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.732+0000 7f939f7fe700 1 --2- 172.21.15.1:0/1957599055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 0x55c6162e8eb0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f9394007940 tx=0x7f939400a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=637b3f545309f891 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.732+0000 7f93867fc700 1 -- 172.21.15.1:0/1957599055 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9394007b30 con 0x55c616315c10 2024-03-18T18:51:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.732+0000 7f93867fc700 1 -- 172.21.15.1:0/1957599055 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9394023070 con 0x55c616315c10 2024-03-18T18:51:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.732+0000 7f93867fc700 1 -- 172.21.15.1:0/1957599055 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939401c660 con 0x55c616315c10 2024-03-18T18:51:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.733+0000 7f93b298eb80 1 -- 172.21.15.1:0/1957599055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 msgr2=0x55c6162e8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.733+0000 7f93b298eb80 1 --2- 172.21.15.1:0/1957599055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 0x55c6162e8eb0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f9394007940 tx=0x7f939400a000 comp rx=0 tx=0).stop 2024-03-18T18:51:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.733+0000 7f93b298eb80 1 -- 172.21.15.1:0/1957599055 shutdown_connections 2024-03-18T18:51:18.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.733+0000 7f93b298eb80 1 --2- 172.21.15.1:0/1957599055 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616327630 0x55c616313470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.733+0000 7f93b298eb80 1 --2- 172.21.15.1:0/1957599055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c6162e70a0 0x55c616331140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.733+0000 7f93b298eb80 1 --2- 172.21.15.1:0/1957599055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 0x55c6162e8eb0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.733+0000 7f93b298eb80 1 -- 172.21.15.1:0/1957599055 >> 172.21.15.1:0/1957599055 conn(0x55c616249cf0 msgr2=0x55c6162e7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:18.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.733+0000 7f93b298eb80 1 -- 172.21.15.1:0/1957599055 shutdown_connections 2024-03-18T18:51:18.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.733+0000 7f93b298eb80 1 -- 172.21.15.1:0/1957599055 wait complete. 2024-03-18T18:51:18.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.733+0000 7f93b298eb80 1 -- start start 2024-03-18T18:51:18.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.734+0000 7f93b298eb80 1 Processor -- start 2024-03-18T18:51:18.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.734+0000 7f93b298eb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c6162e70a0 0x55c6162e5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.734+0000 7f93b298eb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 0x55c616397170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.734+0000 7f93b298eb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616327630 0x55c6163263b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.734+0000 7f93b298eb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c616329f30 con 0x55c616315c10 2024-03-18T18:51:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.734+0000 7f93b298eb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c616326e30 con 0x55c6162e70a0 2024-03-18T18:51:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.734+0000 7f93b298eb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c61630c500 con 0x55c616327630 2024-03-18T18:51:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f93b298eb80 10 client.0 _ll_register_callbacks cb 0x7f9398054b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T18:51:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f939f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 0x55c616397170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f939f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 0x55c616397170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39856/0 (socket says 172.21.15.1:39856) 2024-03-18T18:51:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f939f7fe700 1 -- 172.21.15.1:0/3224477955 learned_addr learned my addr 172.21.15.1:0/3224477955 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f939ffff700 1 --2- 172.21.15.1:0/3224477955 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c6162e70a0 0x55c6162e5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f939f7fe700 1 -- 172.21.15.1:0/3224477955 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616327630 msgr2=0x55c6163263b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f93a4d82700 1 --2- 172.21.15.1:0/3224477955 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616327630 0x55c6163263b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f939f7fe700 1 --2- 172.21.15.1:0/3224477955 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616327630 0x55c6163263b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f939f7fe700 1 -- 172.21.15.1:0/3224477955 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c6162e70a0 msgr2=0x55c6162e5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:18.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f939f7fe700 1 --2- 172.21.15.1:0/3224477955 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c6162e70a0 0x55c6162e5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f939f7fe700 1 -- 172.21.15.1:0/3224477955 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c61611c0a0 con 0x55c616315c10 2024-03-18T18:51:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f939ffff700 1 --2- 172.21.15.1:0/3224477955 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c6162e70a0 0x55c6162e5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f93a4d82700 1 --2- 172.21.15.1:0/3224477955 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616327630 0x55c6163263b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f939f7fe700 1 --2- 172.21.15.1:0/3224477955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 0x55c616397170 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f939402c040 tx=0x7f9394000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=446c3322a572bc41 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f937effd700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T18:51:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f937effd700 1 -- 172.21.15.1:0/3224477955 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939401ce40 con 0x55c616315c10 2024-03-18T18:51:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f937effd700 1 -- 172.21.15.1:0/3224477955 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f939401c660 con 0x55c616315c10 2024-03-18T18:51:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f93b298eb80 1 -- 172.21.15.1:0/3224477955 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9394001600 con 0x55c616315c10 2024-03-18T18:51:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f937effd700 1 -- 172.21.15.1:0/3224477955 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f939401c820 con 0x55c616315c10 2024-03-18T18:51:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f93b298eb80 10 client.5607 fetch_fsmap learned FSMap version 17 2024-03-18T18:51:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f93b298eb80 1 -- 172.21.15.1:0/3224477955 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55c6160901d0 con 0x55c616315c10 2024-03-18T18:51:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.735+0000 7f937effd700 1 -- 172.21.15.1:0/3224477955 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9394001600 con 0x55c616315c10 2024-03-18T18:51:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f937effd700 1 -- 172.21.15.1:0/3224477955 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 17) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9394012070 con 0x55c616315c10 2024-03-18T18:51:18.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f93b298eb80 10 client.5607 fetch_fsmap finished waiting for FSMap version 17 2024-03-18T18:51:18.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f93b298eb80 10 client.5607 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T18:51:18.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f93b298eb80 20 client.5607 populate_metadata read hostname 'smithi001' 2024-03-18T18:51:18.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f93b298eb80 1 --2- 172.21.15.1:0/3224477955 >> [v2:172.21.15.1:6834/3077516107,v1:172.21.15.1:6836/3077516107] conn(0x55c61632a470 0x55c61630db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f93b298eb80 4 client.5607 mds_command: new command op to 5436 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T18:51:18.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f93b298eb80 1 -- 172.21.15.1:0/3224477955 --> [v2:172.21.15.1:6834/3077516107,v1:172.21.15.1:6836/3077516107] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c616075040 con 0x55c61632a470 2024-03-18T18:51:18.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f939ffff700 1 --2- 172.21.15.1:0/3224477955 >> [v2:172.21.15.1:6834/3077516107,v1:172.21.15.1:6836/3077516107] conn(0x55c61632a470 0x55c61630db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f937effd700 10 client.5607 ms_handle_connect on v2:172.21.15.1:6834/3077516107 2024-03-18T18:51:18.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f939ffff700 1 --2- 172.21.15.1:0/3224477955 >> [v2:172.21.15.1:6834/3077516107,v1:172.21.15.1:6836/3077516107] conn(0x55c61632a470 0x55c61630db30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d845faee53a602c6 server_cookie=6741c6eb01cf7317 in_seq=0 out_seq=0 2024-03-18T18:51:18.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f937effd700 1 -- 172.21.15.1:0/3224477955 <== mds.0 v2:172.21.15.1:6834/3077516107 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55c616075040 con 0x55c61632a470 2024-03-18T18:51:18.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f937effd700 10 client.5607 handle_command_reply: tid=0 2024-03-18T18:51:18.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f93b298eb80 1 client.5607 shutdown 2024-03-18T18:51:18.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f93b298eb80 1 -- 172.21.15.1:0/3224477955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 msgr2=0x55c616397170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:18.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f93b298eb80 1 --2- 172.21.15.1:0/3224477955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 0x55c616397170 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f939402c040 tx=0x7f9394000ed0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f93b298eb80 1 -- 172.21.15.1:0/3224477955 shutdown_connections 2024-03-18T18:51:18.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f93b298eb80 1 --2- 172.21.15.1:0/3224477955 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55c616327630 0x55c6163263b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f93b298eb80 1 --2- 172.21.15.1:0/3224477955 >> [v2:172.21.15.1:6834/3077516107,v1:172.21.15.1:6836/3077516107] conn(0x55c61632a470 0x55c61630db30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f93b298eb80 1 --2- 172.21.15.1:0/3224477955 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c6162e70a0 0x55c6162e5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f93b298eb80 1 --2- 172.21.15.1:0/3224477955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c616315c10 0x55c616397170 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f937effd700 0 client.5607 ms_handle_reset on v2:172.21.15.1:6834/3077516107 2024-03-18T18:51:18.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.736+0000 7f93b298eb80 1 -- 172.21.15.1:0/3224477955 >> 172.21.15.1:0/3224477955 conn(0x55c616249cf0 msgr2=0x55c616312ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:18.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.737+0000 7f93b298eb80 1 -- 172.21.15.1:0/3224477955 shutdown_connections 2024-03-18T18:51:18.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.737+0000 7f93b298eb80 1 -- 172.21.15.1:0/3224477955 wait complete. 2024-03-18T18:51:18.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.737+0000 7f93b298eb80 20 client.5607 trim_cache size 0 max 16384 2024-03-18T18:51:18.739 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T18:51:18.739 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "9c1c697f-8655-4ef3-8a13-da568d2cbe86", 2024-03-18T18:51:18.740 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-18T18:51:18.740 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 5436, 2024-03-18T18:51:18.740 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-18T18:51:18.740 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-18T18:51:18.740 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-18T18:51:18.740 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 6.0139022799999999, 2024-03-18T18:51:18.740 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 17, 2024-03-18T18:51:18.740 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 40, 2024-03-18T18:51:18.740 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 40, 2024-03-18T18:51:18.740 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 13.289127239999999 2024-03-18T18:51:18.740 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T18:51:18.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.737+0000 7f93a67e5700 1 -- 172.21.15.1:0/1608362547 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f938c06ac90 msgr2=0x7f938c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:18.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.737+0000 7f93a67e5700 1 --2- 172.21.15.1:0/1608362547 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f938c06ac90 0x7f938c06d150 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9390000c00 tx=0x7f9390008030 comp rx=0 tx=0).stop 2024-03-18T18:51:18.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.737+0000 7f93a67e5700 1 -- 172.21.15.1:0/1608362547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93a015e5e0 msgr2=0x7f93a00afc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:18.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.737+0000 7f93a67e5700 1 --2- 172.21.15.1:0/1608362547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93a015e5e0 0x7f93a00afc40 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f9398009f00 tx=0x7f939800c3a0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.737+0000 7f93a67e5700 1 -- 172.21.15.1:0/1608362547 shutdown_connections 2024-03-18T18:51:18.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.737+0000 7f93a67e5700 1 --2- 172.21.15.1:0/1608362547 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f938c06ac90 0x7f938c06d150 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.737+0000 7f93a67e5700 1 --2- 172.21.15.1:0/1608362547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93a015e5e0 0x7f93a00afc40 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.737+0000 7f93a67e5700 1 --2- 172.21.15.1:0/1608362547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93a006e2d0 0x7f93a014c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.737+0000 7f93a67e5700 1 --2- 172.21.15.1:0/1608362547 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93a0157ba0 0x7f93a0150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.737+0000 7f93a67e5700 1 -- 172.21.15.1:0/1608362547 >> 172.21.15.1:0/1608362547 conn(0x7f93a006a750 msgr2=0x7f93a015cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:18.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.737+0000 7f93a67e5700 1 -- 172.21.15.1:0/1608362547 shutdown_connections 2024-03-18T18:51:18.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.737+0000 7f93a67e5700 1 -- 172.21.15.1:0/1608362547 wait complete. 2024-03-18T18:51:18.745 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T18:51:18.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.873+0000 7f7274423700 1 Processor -- start 2024-03-18T18:51:18.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.873+0000 7f7274423700 1 -- start start 2024-03-18T18:51:18.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.874+0000 7f7274423700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f726c099ea0 0x7f726c0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.874+0000 7f7274423700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726c0d0c80 con 0x7f726c15d680 2024-03-18T18:51:18.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.874+0000 7f7274423700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726c0d09a0 con 0x7f726c099a40 2024-03-18T18:51:18.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.874+0000 7f7274423700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f726c0cb830 con 0x7f726c099ea0 2024-03-18T18:51:18.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.875+0000 7f72729c0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f726c099ea0 0x7f726c0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.875+0000 7f72721bf700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f726c15d680 0x7f726c0cff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56430/0 (socket says 172.21.15.1:56430) 2024-03-18T18:51:18.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.875+0000 7f72721bf700 1 -- 172.21.15.1:0/292787356 learned_addr learned my addr 172.21.15.1:0/292787356 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:18.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.875+0000 7f72729c0700 1 -- 172.21.15.1:0/292787356 >> v1:172.21.15.165:6789/0 conn(0x7f726c099a40 legacy=0x7f726c0cc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:18.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.875+0000 7f72729c0700 1 -- 172.21.15.1:0/292787356 >> v1:172.21.15.1:6789/0 conn(0x7f726c15d680 legacy=0x7f726c0cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:18.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.875+0000 7f72729c0700 1 -- 172.21.15.1:0/292787356 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726c0cbb10 con 0x7f726c099ea0 2024-03-18T18:51:18.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.875+0000 7f72729c0700 1 --2- 172.21.15.1:0/292787356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f726c099ea0 0x7f726c0cca20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7264009d90 tx=0x7f726400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d29d7b9777c7a8d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:18.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.875+0000 7f72711bd700 1 -- 172.21.15.1:0/292787356 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726400e070 con 0x7f726c099ea0 2024-03-18T18:51:18.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.875+0000 7f72711bd700 1 -- 172.21.15.1:0/292787356 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f726400bb90 con 0x7f726c099ea0 2024-03-18T18:51:18.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.875+0000 7f72711bd700 1 -- 172.21.15.1:0/292787356 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7264003a20 con 0x7f726c099ea0 2024-03-18T18:51:18.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.876+0000 7f7274423700 1 -- 172.21.15.1:0/292787356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f726c099ea0 msgr2=0x7f726c0cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:18.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.876+0000 7f7274423700 1 --2- 172.21.15.1:0/292787356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f726c099ea0 0x7f726c0cca20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7264009d90 tx=0x7f726400b330 comp rx=0 tx=0).stop 2024-03-18T18:51:18.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.876+0000 7f7274423700 1 -- 172.21.15.1:0/292787356 shutdown_connections 2024-03-18T18:51:18.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.876+0000 7f7274423700 1 --2- 172.21.15.1:0/292787356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f726c099ea0 0x7f726c0cca20 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.876+0000 7f7274423700 1 -- 172.21.15.1:0/292787356 >> 172.21.15.1:0/292787356 conn(0x7f726c0b2ba0 msgr2=0x7f726c0b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:18.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.876+0000 7f7274423700 1 -- 172.21.15.1:0/292787356 shutdown_connections 2024-03-18T18:51:18.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.876+0000 7f7274423700 1 -- 172.21.15.1:0/292787356 wait complete. 2024-03-18T18:51:18.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.877+0000 7f7274423700 1 Processor -- start 2024-03-18T18:51:18.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.877+0000 7f7274423700 1 -- start start 2024-03-18T18:51:18.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.878+0000 7f7274423700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f726c099a40 0x7f726c0c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.878+0000 7f7274423700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f726c099ea0 0x7f726c0ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.878+0000 7f7274423700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f726c15d680 0x7f726c0ca630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.878+0000 7f7274423700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726c0cac90 con 0x7f726c099a40 2024-03-18T18:51:18.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.878+0000 7f7274423700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726c0caf70 con 0x7f726c15d680 2024-03-18T18:51:18.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.878+0000 7f7274423700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f726c0cb250 con 0x7f726c099ea0 2024-03-18T18:51:18.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.878+0000 7f72721bf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f726c099a40 0x7f726c0c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.878+0000 7f72721bf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f726c099a40 0x7f726c0c8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39882/0 (socket says 172.21.15.1:39882) 2024-03-18T18:51:18.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.878+0000 7f72721bf700 1 -- 172.21.15.1:0/2048272595 learned_addr learned my addr 172.21.15.1:0/2048272595 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:18.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.878+0000 7f72729c0700 1 --2- 172.21.15.1:0/2048272595 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f726c15d680 0x7f726c0ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.878+0000 7f72719be700 1 --2- 172.21.15.1:0/2048272595 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f726c099ea0 0x7f726c0ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.878+0000 7f72721bf700 1 -- 172.21.15.1:0/2048272595 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f726c099ea0 msgr2=0x7f726c0ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:18.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.878+0000 7f72721bf700 1 --2- 172.21.15.1:0/2048272595 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f726c099ea0 0x7f726c0ca0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.878+0000 7f72721bf700 1 -- 172.21.15.1:0/2048272595 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f726c15d680 msgr2=0x7f726c0ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:18.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.878+0000 7f72721bf700 1 --2- 172.21.15.1:0/2048272595 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f726c15d680 0x7f726c0ca630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:18.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.878+0000 7f72721bf700 1 -- 172.21.15.1:0/2048272595 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7264009a70 con 0x7f726c099a40 2024-03-18T18:51:18.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.878+0000 7f72719be700 1 --2- 172.21.15.1:0/2048272595 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f726c099ea0 0x7f726c0ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:18.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.878+0000 7f72729c0700 1 --2- 172.21.15.1:0/2048272595 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f726c15d680 0x7f726c0ca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:18.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.879+0000 7f72721bf700 1 --2- 172.21.15.1:0/2048272595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f726c099a40 0x7f726c0c8180 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f726800c430 tx=0x7f726800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:18.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.879+0000 7f725effd700 1 -- 172.21.15.1:0/2048272595 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7268005d70 con 0x7f726c099a40 2024-03-18T18:51:18.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.879+0000 7f725effd700 1 -- 172.21.15.1:0/2048272595 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72680070d0 con 0x7f726c099a40 2024-03-18T18:51:18.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.879+0000 7f7274423700 1 -- 172.21.15.1:0/2048272595 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f726c0c86c0 con 0x7f726c099a40 2024-03-18T18:51:18.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.879+0000 7f725effd700 1 -- 172.21.15.1:0/2048272595 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7268007460 con 0x7f726c099a40 2024-03-18T18:51:18.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.879+0000 7f7274423700 1 -- 172.21.15.1:0/2048272595 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f726c157a30 con 0x7f726c099a40 2024-03-18T18:51:18.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.880+0000 7f7274423700 1 -- 172.21.15.1:0/2048272595 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f726c15e130 con 0x7f726c099a40 2024-03-18T18:51:18.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.880+0000 7f725effd700 1 -- 172.21.15.1:0/2048272595 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f72680075c0 con 0x7f726c099a40 2024-03-18T18:51:18.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.881+0000 7f725effd700 1 --2- 172.21.15.1:0/2048272595 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f725006ac90 0x7f725006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:18.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.881+0000 7f725effd700 1 -- 172.21.15.1:0/2048272595 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7268088930 con 0x7f726c099a40 2024-03-18T18:51:18.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.881+0000 7f72719be700 1 --2- 172.21.15.1:0/2048272595 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f725006ac90 0x7f725006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:18.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.881+0000 7f72719be700 1 --2- 172.21.15.1:0/2048272595 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f725006ac90 0x7f725006d150 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f726c15deb0 tx=0x7f7260009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:18.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:18.885+0000 7f725effd700 1 -- 172.21.15.1:0/2048272595 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7268053940 con 0x7f726c099a40 2024-03-18T18:51:19.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:19.135+0000 7f7274423700 1 -- 172.21.15.1:0/2048272595 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f726c09a9f0 con 0x7f726c099a40 2024-03-18T18:51:19.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:19.136+0000 7f725effd700 1 -- 172.21.15.1:0/2048272595 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7268017070 con 0x7f726c099a40 2024-03-18T18:51:19.132 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T18:51:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:19.138+0000 7f7274423700 1 -- 172.21.15.1:0/2048272595 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f725006ac90 msgr2=0x7f725006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:19.138+0000 7f7274423700 1 --2- 172.21.15.1:0/2048272595 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f725006ac90 0x7f725006d150 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f726c15deb0 tx=0x7f7260009040 comp rx=0 tx=0).stop 2024-03-18T18:51:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:19.138+0000 7f7274423700 1 -- 172.21.15.1:0/2048272595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f726c099a40 msgr2=0x7f726c0c8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:19.138+0000 7f7274423700 1 --2- 172.21.15.1:0/2048272595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f726c099a40 0x7f726c0c8180 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f726800c430 tx=0x7f726800c7f0 comp rx=0 tx=0).stop 2024-03-18T18:51:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:19.138+0000 7f7274423700 1 -- 172.21.15.1:0/2048272595 shutdown_connections 2024-03-18T18:51:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:19.138+0000 7f7274423700 1 --2- 172.21.15.1:0/2048272595 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f725006ac90 0x7f725006d150 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:19.138+0000 7f7274423700 1 --2- 172.21.15.1:0/2048272595 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f726c15d680 0x7f726c0ca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:19.138+0000 7f7274423700 1 --2- 172.21.15.1:0/2048272595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f726c099a40 0x7f726c0c8180 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:19.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:19.138+0000 7f7274423700 1 --2- 172.21.15.1:0/2048272595 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f726c099ea0 0x7f726c0ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:19.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:19.138+0000 7f7274423700 1 -- 172.21.15.1:0/2048272595 >> 172.21.15.1:0/2048272595 conn(0x7f726c0b2ba0 msgr2=0x7f726c0af960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:19.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:19.138+0000 7f7274423700 1 -- 172.21.15.1:0/2048272595 shutdown_connections 2024-03-18T18:51:19.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:19.138+0000 7f7274423700 1 -- 172.21.15.1:0/2048272595 wait complete. 2024-03-18T18:51:19.146 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T18:51:19.147 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T18:51:19.147 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-18T18:51:19.147 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T18:51:19.147 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T18:51:19.147 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T18:51:19.147 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T18:51:19.147 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T18:51:19.147 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T18:51:19.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:19.147 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-18T18:51:19.166 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T18:51:19.167 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T18:51:19.167 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T18:51:19.167 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:51:19.167 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T18:51:19.167 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:51:19.167 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T18:51:19.167 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-18T18:51:19.167 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-18T18:51:19.167 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T18:51:19.167 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-18T18:51:19.167 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-18T18:51:19.167 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T18:51:19.167 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-18T18:51:19.167 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-18T18:51:19.168 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T18:51:19.168 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 42490sec preferred_lft 42490sec 2024-03-18T18:51:19.168 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link 2024-03-18T18:51:19.168 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:51:19.168 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T18:51:19.168 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-18T18:51:19.168 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-18T18:51:19.168 INFO:teuthology.orchestra.run.smithi001.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T18:51:19.168 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff 2024-03-18T18:51:19.168 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T18:51:19.168 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:51:19.168 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::4ca9:f5ff:feb3:79a3/64 scope link 2024-03-18T18:51:19.168 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:51:19.168 INFO:teuthology.orchestra.run.smithi001.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T18:51:19.169 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T18:51:19.169 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::282f:a4ff:fe1a:9908/64 scope link 2024-03-18T18:51:19.169 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:51:19.169 INFO:teuthology.orchestra.run.smithi001.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T18:51:19.169 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2e:5c:60:78:b2:8c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T18:51:19.169 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::2c5c:60ff:fe78:b28c/64 scope link 2024-03-18T18:51:19.169 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T18:51:19.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:19.169 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T18:51:19.233 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-18T18:51:19.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:19.238 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-18T18:51:19.298 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-18T18:51:19.298 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T18:51:19.298 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T18:51:19.298 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T18:51:19.298 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T18:51:19.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:19.299 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T18:51:19.299 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T18:51:19.299 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T18:51:19.299 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T18:51:19.299 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T18:51:19.299 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T18:51:19.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:19.405 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T18:51:19.424 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T18:51:19.424 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T18:51:19.425 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T18:51:19.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:19.425 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-18T18:51:19.492 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T18:51:19.492 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T18:51:19.492 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T18:51:19.550 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T18:51:19.551 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T18:51:19.551 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T18:51:19.609 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T18:51:19.678 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T18:51:19.679 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T18:51:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T18:51:19.747 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T18:51:19.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:19.748 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T18:51:19.807 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T18:51:19.807 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T18:51:19.849 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T18:51:19.877 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T18:51:19.877 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T18:51:19.939 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-18T18:51:19.942+0000 7f9fd41cc5c0 -1 init, newargv = 0x56064287c020 newargc=15 2024-03-18T18:51:19.939 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[95919]: starting ceph client 2024-03-18T18:51:19.943 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[95919]: starting fuse 2024-03-18T18:51:19.947 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T18:51:19.948 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T18:51:19.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:19.949 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T18:51:20.008 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T18:51:20.008 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-03-18T18:51:20.008 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T18:51:20.008 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-18T18:51:20.008 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-18T18:51:20.008 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-18T18:51:20.009 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-18T18:51:20.009 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T18:51:20.009 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-18T18:51:20.009 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T18:51:20.009 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-18T18:51:20.009 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T18:51:20.009 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T18:51:20.009 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-18T18:51:20.009 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-18T18:51:20.009 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-18T18:51:20.009 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-18T18:51:20.009 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T18:51:20.009 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-18T18:51:20.009 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T18:51:20.009 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T18:51:20.010 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T18:51:20.010 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-18T18:51:20.010 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-18T18:51:20.010 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-18T18:51:20.010 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T18:51:20.010 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T18:51:20.010 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-18T18:51:20.010 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-18T18:51:20.133 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.95919.asok 2024-03-18T18:51:20.315 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.95919.asok 2024-03-18T18:51:20.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:20.315 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.95919.asok status 2024-03-18T18:51:20.477 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T18:51:20.477 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-18T18:51:20.477 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T18:51:20.477 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T18:51:20.477 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-18T18:51:20.477 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-18T18:51:20.477 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T18:51:20.477 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "95919", 2024-03-18T18:51:20.477 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-18T18:51:20.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:20.478 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-18T18:51:20.478 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-18T18:51:20.478 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 5619, 2024-03-18T18:51:20.478 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-18T18:51:20.478 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-18T18:51:20.478 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-18T18:51:20.478 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5619 2024-03-18T18:51:20.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:20.478 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T18:51:20.478 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T18:51:20.478 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T18:51:20.478 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3889209300 2024-03-18T18:51:20.479 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:20.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:20.479 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T18:51:20.479 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T18:51:20.479 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T18:51:20.479 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3889209300 2024-03-18T18:51:20.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T18:51:20.479 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.5619 192.168.0.1:0/3889209300", 2024-03-18T18:51:20.479 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/3889209300", 2024-03-18T18:51:20.479 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-18T18:51:20.479 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 17, 2024-03-18T18:51:20.479 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 40, 2024-03-18T18:51:20.479 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-18T18:51:20.479 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-18T18:51:20.480 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-18T18:51:20.480 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T18:51:20.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:20.487 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-18T18:51:20.487 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T18:51:20.506 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-18T18:51:20.507 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T18:51:20.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:20.507 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T18:51:20.575 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-18T18:51:20.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.746+0000 7f169ef73700 1 Processor -- start 2024-03-18T18:51:20.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.746+0000 7f169ef73700 1 -- start start 2024-03-18T18:51:20.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.747+0000 7f169ef73700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f169815d580 0x7f169806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:20.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.747+0000 7f169ef73700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f169806f4f0 con 0x7f169806d380 2024-03-18T18:51:20.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.747+0000 7f169ef73700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f169806ab30 con 0x7f16980724b0 2024-03-18T18:51:20.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.747+0000 7f169ef73700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f169815d960 con 0x7f169815d580 2024-03-18T18:51:20.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.747+0000 7f1697fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f169806d380 0x7f1698071da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56446/0 (socket says 172.21.15.1:56446) 2024-03-18T18:51:20.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.747+0000 7f1697fff700 1 -- 172.21.15.1:0/3381907167 learned_addr learned my addr 172.21.15.1:0/3381907167 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:20.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.747+0000 7f169cd0f700 1 --2- 172.21.15.1:0/3381907167 >> v2:172.21.15.165:3300/0 conn(0x7f169815d580 0x7f169806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:20.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.748+0000 7f16977fe700 1 -- 172.21.15.1:0/3381907167 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226997535 0 0) 0x7f1684001110 con 0x7f169806d380 2024-03-18T18:51:20.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.748+0000 7f16977fe700 1 -- 172.21.15.1:0/3381907167 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16880035b0 con 0x7f169806d380 2024-03-18T18:51:20.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.748+0000 7f16977fe700 1 -- 172.21.15.1:0/3381907167 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276656557 0 0) 0x7f169806ab30 con 0x7f16980724b0 2024-03-18T18:51:20.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.748+0000 7f16977fe700 1 -- 172.21.15.1:0/3381907167 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1684001110 con 0x7f16980724b0 2024-03-18T18:51:20.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.748+0000 7f169cd0f700 1 -- 172.21.15.1:0/3381907167 >> v1:172.21.15.165:6789/0 conn(0x7f16980724b0 legacy=0x7f16980cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:20.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.748+0000 7f169cd0f700 1 -- 172.21.15.1:0/3381907167 >> v1:172.21.15.1:6789/0 conn(0x7f169806d380 legacy=0x7f1698071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:20.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.748+0000 7f169cd0f700 1 -- 172.21.15.1:0/3381907167 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169806ae10 con 0x7f169815d580 2024-03-18T18:51:20.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.748+0000 7f169cd0f700 1 --2- 172.21.15.1:0/3381907167 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f169815d580 0x7f169806efb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f168c009840 tx=0x7f168c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7f284d3c615ab28 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:20.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.748+0000 7f16977fe700 1 -- 172.21.15.1:0/3381907167 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c011830 con 0x7f169815d580 2024-03-18T18:51:20.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.749+0000 7f16977fe700 1 -- 172.21.15.1:0/3381907167 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168c011990 con 0x7f169815d580 2024-03-18T18:51:20.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.749+0000 7f16977fe700 1 -- 172.21.15.1:0/3381907167 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c011d20 con 0x7f169815d580 2024-03-18T18:51:20.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.749+0000 7f169ef73700 1 -- 172.21.15.1:0/3381907167 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f169815d580 msgr2=0x7f169806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:20.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.749+0000 7f169ef73700 1 --2- 172.21.15.1:0/3381907167 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f169815d580 0x7f169806efb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f168c009840 tx=0x7f168c009b50 comp rx=0 tx=0).stop 2024-03-18T18:51:20.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.749+0000 7f169ef73700 1 -- 172.21.15.1:0/3381907167 shutdown_connections 2024-03-18T18:51:20.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.749+0000 7f169ef73700 1 --2- 172.21.15.1:0/3381907167 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f169815d580 0x7f169806efb0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:20.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.749+0000 7f169ef73700 1 -- 172.21.15.1:0/3381907167 >> 172.21.15.1:0/3381907167 conn(0x7f16980cca00 msgr2=0x7f16980cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:20.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.749+0000 7f169ef73700 1 -- 172.21.15.1:0/3381907167 shutdown_connections 2024-03-18T18:51:20.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.749+0000 7f169ef73700 1 -- 172.21.15.1:0/3381907167 wait complete. 2024-03-18T18:51:20.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.750+0000 7f169ef73700 1 Processor -- start 2024-03-18T18:51:20.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.750+0000 7f169ef73700 1 -- start start 2024-03-18T18:51:20.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.751+0000 7f169ef73700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f169806d380 0x7f16980c5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:20.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.751+0000 7f169ef73700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16980724b0 0x7f16980c3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:20.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.751+0000 7f169ef73700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f169815d580 0x7f16980c3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:20.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.751+0000 7f169ef73700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16980c4460 con 0x7f169806d380 2024-03-18T18:51:20.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.751+0000 7f169ef73700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16980c4740 con 0x7f16980724b0 2024-03-18T18:51:20.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.751+0000 7f169ef73700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16980c4a20 con 0x7f169815d580 2024-03-18T18:51:20.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.752+0000 7f169cd0f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f169806d380 0x7f16980c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:20.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.752+0000 7f169cd0f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f169806d380 0x7f16980c5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39898/0 (socket says 172.21.15.1:39898) 2024-03-18T18:51:20.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.752+0000 7f169cd0f700 1 -- 172.21.15.1:0/189546143 learned_addr learned my addr 172.21.15.1:0/189546143 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:20.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.752+0000 7f1697fff700 1 --2- 172.21.15.1:0/189546143 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16980724b0 0x7f16980c3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:20.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.752+0000 7f169d510700 1 --2- 172.21.15.1:0/189546143 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f169815d580 0x7f16980c3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:20.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.752+0000 7f169cd0f700 1 -- 172.21.15.1:0/189546143 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f169815d580 msgr2=0x7f16980c3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:20.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.752+0000 7f169cd0f700 1 --2- 172.21.15.1:0/189546143 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f169815d580 0x7f16980c3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:20.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.752+0000 7f169cd0f700 1 -- 172.21.15.1:0/189546143 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16980724b0 msgr2=0x7f16980c3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:20.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.752+0000 7f169cd0f700 1 --2- 172.21.15.1:0/189546143 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16980724b0 0x7f16980c3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:20.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.752+0000 7f169cd0f700 1 -- 172.21.15.1:0/189546143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168c0094a0 con 0x7f169806d380 2024-03-18T18:51:20.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.752+0000 7f169cd0f700 1 --2- 172.21.15.1:0/189546143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f169806d380 0x7f16980c5500 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f168c0090b0 tx=0x7f168c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:20.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.752+0000 7f16957fa700 1 -- 172.21.15.1:0/189546143 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c01a6e0 con 0x7f169806d380 2024-03-18T18:51:20.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.752+0000 7f169ef73700 1 -- 172.21.15.1:0/189546143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16980c4d00 con 0x7f169806d380 2024-03-18T18:51:20.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.752+0000 7f16957fa700 1 -- 172.21.15.1:0/189546143 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168c01a840 con 0x7f169806d380 2024-03-18T18:51:20.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.752+0000 7f16957fa700 1 -- 172.21.15.1:0/189546143 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c01abb0 con 0x7f169806d380 2024-03-18T18:51:20.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.752+0000 7f169ef73700 1 -- 172.21.15.1:0/189546143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16981f3d80 con 0x7f169806d380 2024-03-18T18:51:20.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.753+0000 7f169ef73700 1 -- 172.21.15.1:0/189546143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16980c5220 con 0x7f169806d380 2024-03-18T18:51:20.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.754+0000 7f16957fa700 1 -- 172.21.15.1:0/189546143 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f168c022070 con 0x7f169806d380 2024-03-18T18:51:20.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.754+0000 7f16957fa700 1 --2- 172.21.15.1:0/189546143 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f167c06ac90 0x7f167c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:20.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.754+0000 7f16957fa700 1 -- 172.21.15.1:0/189546143 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f168c08a070 con 0x7f169806d380 2024-03-18T18:51:20.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.754+0000 7f1697fff700 1 --2- 172.21.15.1:0/189546143 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f167c06ac90 0x7f167c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:20.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.755+0000 7f1697fff700 1 --2- 172.21.15.1:0/189546143 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f167c06ac90 0x7f167c06d150 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f16980c6ad0 tx=0x7f1684007e00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:20.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.759+0000 7f16957fa700 1 -- 172.21.15.1:0/189546143 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f168c0bd580 con 0x7f169806d380 2024-03-18T18:51:20.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.991+0000 7f169ef73700 1 -- 172.21.15.1:0/189546143 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f16980cd790 con 0x7f167c06ac90 2024-03-18T18:51:20.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.993+0000 7f16957fa700 1 -- 172.21.15.1:0/189546143 <== mgr.4104 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f16840058e0 con 0x7f167c06ac90 2024-03-18T18:51:20.989 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T18:51:20.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T18:51:20.989 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephfs" 2024-03-18T18:51:20.990 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T18:51:20.990 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T18:51:20.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.995+0000 7f169ef73700 1 -- 172.21.15.1:0/189546143 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f167c06ac90 msgr2=0x7f167c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:20.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.995+0000 7f169ef73700 1 --2- 172.21.15.1:0/189546143 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f167c06ac90 0x7f167c06d150 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f16980c6ad0 tx=0x7f1684007e00 comp rx=0 tx=0).stop 2024-03-18T18:51:20.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.995+0000 7f169ef73700 1 -- 172.21.15.1:0/189546143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f169806d380 msgr2=0x7f16980c5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:20.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.995+0000 7f169ef73700 1 --2- 172.21.15.1:0/189546143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f169806d380 0x7f16980c5500 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f168c0090b0 tx=0x7f168c004750 comp rx=0 tx=0).stop 2024-03-18T18:51:20.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.995+0000 7f169ef73700 1 -- 172.21.15.1:0/189546143 shutdown_connections 2024-03-18T18:51:20.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.995+0000 7f169ef73700 1 --2- 172.21.15.1:0/189546143 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f167c06ac90 0x7f167c06d150 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:20.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.995+0000 7f169ef73700 1 --2- 172.21.15.1:0/189546143 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f169815d580 0x7f16980c3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:20.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.995+0000 7f169ef73700 1 --2- 172.21.15.1:0/189546143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f169806d380 0x7f16980c5500 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:20.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.995+0000 7f169ef73700 1 --2- 172.21.15.1:0/189546143 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16980724b0 0x7f16980c3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:20.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.995+0000 7f169ef73700 1 -- 172.21.15.1:0/189546143 >> 172.21.15.1:0/189546143 conn(0x7f16980cca00 msgr2=0x7f16980b1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:20.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.995+0000 7f169ef73700 1 -- 172.21.15.1:0/189546143 shutdown_connections 2024-03-18T18:51:20.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:20.995+0000 7f169ef73700 1 -- 172.21.15.1:0/189546143 wait complete. 2024-03-18T18:51:21.003 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-18T18:51:21.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.134+0000 7fb53f42b700 1 Processor -- start 2024-03-18T18:51:21.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.134+0000 7fb53f42b700 1 -- start start 2024-03-18T18:51:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.135+0000 7fb53f42b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb53815ad70 0x7fb53815d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.135+0000 7fb53f42b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5380727b0 0x7fb538161610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.135+0000 7fb53f42b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb538160ea0 con 0x7fb538161b50 2024-03-18T18:51:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.135+0000 7fb53f42b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5380725b0 con 0x7fb53815ad70 2024-03-18T18:51:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.135+0000 7fb53f42b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53815ddf0 con 0x7fb5380727b0 2024-03-18T18:51:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.136+0000 7fb53c9c6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5380727b0 0x7fb538161610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.136+0000 7fb53d1c7700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb53815ad70 0x7fb53815d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.136+0000 7fb53d9c8700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fb538161b50 0x7fb53806a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:47734/0 (socket says 172.21.15.1:47734) 2024-03-18T18:51:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.136+0000 7fb53d9c8700 1 -- 172.21.15.1:0/1743947582 learned_addr learned my addr 172.21.15.1:0/1743947582 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:21.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.136+0000 7fb53d1c7700 1 -- 172.21.15.1:0/1743947582 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5380727b0 msgr2=0x7fb538161610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.136+0000 7fb53d1c7700 1 --2- 172.21.15.1:0/1743947582 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5380727b0 0x7fb538161610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.136+0000 7fb53d1c7700 1 -- 172.21.15.1:0/1743947582 >> v1:172.21.15.165:6789/0 conn(0x7fb538161b50 legacy=0x7fb53806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.136+0000 7fb53d1c7700 1 -- 172.21.15.1:0/1743947582 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53815e090 con 0x7fb53815ad70 2024-03-18T18:51:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.136+0000 7fb53c9c6700 1 --2- 172.21.15.1:0/1743947582 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5380727b0 0x7fb538161610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:51:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.136+0000 7fb52ffff700 1 -- 172.21.15.1:0/1743947582 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442805178 0 0) 0x7fb538160ea0 con 0x7fb538161b50 2024-03-18T18:51:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.136+0000 7fb53d1c7700 1 --2- 172.21.15.1:0/1743947582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb53815ad70 0x7fb53815d8b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb528009870 tx=0x7fb528009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1034facd4ba7fa2e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.137+0000 7fb52ffff700 1 -- 172.21.15.1:0/1743947582 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb528011830 con 0x7fb53815ad70 2024-03-18T18:51:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.137+0000 7fb52ffff700 1 -- 172.21.15.1:0/1743947582 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb528011990 con 0x7fb53815ad70 2024-03-18T18:51:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.137+0000 7fb52ffff700 1 -- 172.21.15.1:0/1743947582 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb528011d20 con 0x7fb53815ad70 2024-03-18T18:51:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.137+0000 7fb53f42b700 1 -- 172.21.15.1:0/1743947582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb53815ad70 msgr2=0x7fb53815d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.137+0000 7fb53f42b700 1 --2- 172.21.15.1:0/1743947582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb53815ad70 0x7fb53815d8b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb528009870 tx=0x7fb528009b80 comp rx=0 tx=0).stop 2024-03-18T18:51:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.137+0000 7fb53f42b700 1 -- 172.21.15.1:0/1743947582 shutdown_connections 2024-03-18T18:51:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.137+0000 7fb53f42b700 1 --2- 172.21.15.1:0/1743947582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb53815ad70 0x7fb53815d8b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:21.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.137+0000 7fb53f42b700 1 --2- 172.21.15.1:0/1743947582 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5380727b0 0x7fb538161610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.137+0000 7fb53f42b700 1 -- 172.21.15.1:0/1743947582 >> 172.21.15.1:0/1743947582 conn(0x7fb5380b1cb0 msgr2=0x7fb5380b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.137+0000 7fb53f42b700 1 -- 172.21.15.1:0/1743947582 shutdown_connections 2024-03-18T18:51:21.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.137+0000 7fb53f42b700 1 -- 172.21.15.1:0/1743947582 wait complete. 2024-03-18T18:51:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.139+0000 7fb53f42b700 1 Processor -- start 2024-03-18T18:51:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.139+0000 7fb53f42b700 1 -- start start 2024-03-18T18:51:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.140+0000 7fb53f42b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5380727b0 0x7fb5380ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.140+0000 7fb53f42b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb53815ad70 0x7fb5380ae780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.140+0000 7fb53f42b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb538161b50 0x7fb5380ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.140+0000 7fb53f42b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53815b640 con 0x7fb538161b50 2024-03-18T18:51:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.140+0000 7fb53f42b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380cf210 con 0x7fb53815ad70 2024-03-18T18:51:21.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.140+0000 7fb53f42b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380cf4f0 con 0x7fb5380727b0 2024-03-18T18:51:21.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.140+0000 7fb53d9c8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb538161b50 0x7fb5380ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:21.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.140+0000 7fb53d9c8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb538161b50 0x7fb5380ce4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39906/0 (socket says 172.21.15.1:39906) 2024-03-18T18:51:21.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.140+0000 7fb53d9c8700 1 -- 172.21.15.1:0/2313158759 learned_addr learned my addr 172.21.15.1:0/2313158759 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:21.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.140+0000 7fb53c9c6700 1 --2- 172.21.15.1:0/2313158759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb53815ad70 0x7fb5380ae780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:21.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.140+0000 7fb53d1c7700 1 --2- 172.21.15.1:0/2313158759 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5380727b0 0x7fb5380ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:21.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.140+0000 7fb53d9c8700 1 -- 172.21.15.1:0/2313158759 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5380727b0 msgr2=0x7fb5380ae240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:21.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.140+0000 7fb53d9c8700 1 --2- 172.21.15.1:0/2313158759 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5380727b0 0x7fb5380ae240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:21.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.140+0000 7fb53d9c8700 1 -- 172.21.15.1:0/2313158759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb53815ad70 msgr2=0x7fb5380ae780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:21.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.140+0000 7fb53d9c8700 1 --2- 172.21.15.1:0/2313158759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb53815ad70 0x7fb5380ae780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:21.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.140+0000 7fb53d9c8700 1 -- 172.21.15.1:0/2313158759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5340094a0 con 0x7fb538161b50 2024-03-18T18:51:21.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.140+0000 7fb53c9c6700 1 --2- 172.21.15.1:0/2313158759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb53815ad70 0x7fb5380ae780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:21.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.140+0000 7fb53d9c8700 1 --2- 172.21.15.1:0/2313158759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb538161b50 0x7fb5380ce4b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb53000cb40 tx=0x7fb53000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:21.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.141+0000 7fb52dffb700 1 -- 172.21.15.1:0/2313158759 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53000ab50 con 0x7fb538161b50 2024-03-18T18:51:21.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.141+0000 7fb52dffb700 1 -- 172.21.15.1:0/2313158759 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53000acd0 con 0x7fb538161b50 2024-03-18T18:51:21.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.141+0000 7fb53f42b700 1 -- 172.21.15.1:0/2313158759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5280094a0 con 0x7fb538161b50 2024-03-18T18:51:21.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.141+0000 7fb52dffb700 1 -- 172.21.15.1:0/2313158759 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb530007310 con 0x7fb538161b50 2024-03-18T18:51:21.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.141+0000 7fb53f42b700 1 -- 172.21.15.1:0/2313158759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb53815d900 con 0x7fb538161b50 2024-03-18T18:51:21.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.142+0000 7fb53f42b700 1 -- 172.21.15.1:0/2313158759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb53815b410 con 0x7fb538161b50 2024-03-18T18:51:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.142+0000 7fb52dffb700 1 -- 172.21.15.1:0/2313158759 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb530007470 con 0x7fb538161b50 2024-03-18T18:51:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.142+0000 7fb52dffb700 1 --2- 172.21.15.1:0/2313158759 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb51c06ac90 0x7fb51c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.143+0000 7fb52dffb700 1 -- 172.21.15.1:0/2313158759 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb530089070 con 0x7fb538161b50 2024-03-18T18:51:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.143+0000 7fb53d1c7700 1 --2- 172.21.15.1:0/2313158759 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb51c06ac90 0x7fb51c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:21.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.143+0000 7fb53d1c7700 1 --2- 172.21.15.1:0/2313158759 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb51c06ac90 0x7fb51c06d150 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb53806f140 tx=0x7fb528008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:21.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.147+0000 7fb52dffb700 1 -- 172.21.15.1:0/2313158759 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb5300bc580 con 0x7fb538161b50 2024-03-18T18:51:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.415+0000 7fb53f42b700 1 -- 172.21.15.1:0/2313158759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fb5380cfa10 con 0x7fb538161b50 2024-03-18T18:51:21.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.894+0000 7fb52dffb700 1 -- 172.21.15.1:0/2313158759 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 6) v1 ==== 89154+0+0 (secure 0 0 0) 0x7fb530054450 con 0x7fb538161b50 2024-03-18T18:51:21.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.895+0000 7fb52dffb700 1 -- 172.21.15.1:0/2313158759 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) v1 ==== 92+0+0 (secure 0 0 0) 0x7fb530083dd0 con 0x7fb538161b50 2024-03-18T18:51:21.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.898+0000 7fb53f42b700 1 -- 172.21.15.1:0/2313158759 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb51c06ac90 msgr2=0x7fb51c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:21.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.898+0000 7fb53f42b700 1 --2- 172.21.15.1:0/2313158759 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb51c06ac90 0x7fb51c06d150 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb53806f140 tx=0x7fb528008030 comp rx=0 tx=0).stop 2024-03-18T18:51:21.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.898+0000 7fb53f42b700 1 -- 172.21.15.1:0/2313158759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb538161b50 msgr2=0x7fb5380ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:21.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.898+0000 7fb53f42b700 1 --2- 172.21.15.1:0/2313158759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb538161b50 0x7fb5380ce4b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb53000cb40 tx=0x7fb53000cf00 comp rx=0 tx=0).stop 2024-03-18T18:51:21.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.898+0000 7fb53f42b700 1 -- 172.21.15.1:0/2313158759 shutdown_connections 2024-03-18T18:51:21.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.898+0000 7fb53f42b700 1 --2- 172.21.15.1:0/2313158759 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb51c06ac90 0x7fb51c06d150 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:21.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.898+0000 7fb53f42b700 1 --2- 172.21.15.1:0/2313158759 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb5380727b0 0x7fb5380ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:21.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.898+0000 7fb53f42b700 1 --2- 172.21.15.1:0/2313158759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb538161b50 0x7fb5380ce4b0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:21.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.898+0000 7fb53f42b700 1 --2- 172.21.15.1:0/2313158759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb53815ad70 0x7fb5380ae780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:21.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.898+0000 7fb53f42b700 1 -- 172.21.15.1:0/2313158759 >> 172.21.15.1:0/2313158759 conn(0x7fb5380b1cb0 msgr2=0x7fb5380b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:21.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.898+0000 7fb53f42b700 1 -- 172.21.15.1:0/2313158759 shutdown_connections 2024-03-18T18:51:21.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:21.898+0000 7fb53f42b700 1 -- 172.21.15.1:0/2313158759 wait complete. 2024-03-18T18:51:21.908 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-18T18:51:22.026 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:22.029+0000 7fefd9382240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T18:51:22.034 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:22.037+0000 7f90e0994240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T18:51:22.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.046+0000 7fd88760c700 1 Processor -- start 2024-03-18T18:51:22.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.046+0000 7fd88760c700 1 -- start start 2024-03-18T18:51:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.047+0000 7fd88760c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd88015a240 0x7fd8800ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.047+0000 7fd88760c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd88006d560 0x7fd88006fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.047+0000 7fd88760c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd880070000 con 0x7fd8800715b0 2024-03-18T18:51:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.047+0000 7fd88760c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8800adff0 con 0x7fd88006d560 2024-03-18T18:51:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.047+0000 7fd88760c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8800cac80 con 0x7fd88015a240 2024-03-18T18:51:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.047+0000 7fd884ba7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd88006d560 0x7fd88006fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.047+0000 7fd884ba7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd88006d560 0x7fd88006fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39932/0 (socket says 172.21.15.1:39932) 2024-03-18T18:51:22.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.047+0000 7fd884ba7700 1 -- 172.21.15.1:0/1537988890 learned_addr learned my addr 172.21.15.1:0/1537988890 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:22.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.047+0000 7fd8853a8700 1 --2- 172.21.15.1:0/1537988890 >> v2:172.21.15.165:3300/0 conn(0x7fd88015a240 0x7fd8800ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:22.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.047+0000 7fd884ba7700 1 -- 172.21.15.1:0/1537988890 >> v2:172.21.15.165:3300/0 conn(0x7fd88015a240 msgr2=0x7fd8800ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:22.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.047+0000 7fd884ba7700 1 --2- 172.21.15.1:0/1537988890 >> v2:172.21.15.165:3300/0 conn(0x7fd88015a240 0x7fd8800ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.047+0000 7fd884ba7700 1 -- 172.21.15.1:0/1537988890 >> v1:172.21.15.1:6789/0 conn(0x7fd8800715b0 legacy=0x7fd88015de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:22.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.047+0000 7fd884ba7700 1 -- 172.21.15.1:0/1537988890 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8800caf60 con 0x7fd88006d560 2024-03-18T18:51:22.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.047+0000 7fd884ba7700 1 --2- 172.21.15.1:0/1537988890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd88006d560 0x7fd88006fac0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd8700098f0 tx=0x7fd870009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf1972b0a33f35 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:22.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.047+0000 7fd877fff700 1 -- 172.21.15.1:0/1537988890 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87000f8a0 con 0x7fd88006d560 2024-03-18T18:51:22.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.048+0000 7fd877fff700 1 -- 172.21.15.1:0/1537988890 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87000fa00 con 0x7fd88006d560 2024-03-18T18:51:22.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.048+0000 7fd877fff700 1 -- 172.21.15.1:0/1537988890 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87000fd90 con 0x7fd88006d560 2024-03-18T18:51:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.048+0000 7fd88760c700 1 -- 172.21.15.1:0/1537988890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd88006d560 msgr2=0x7fd88006fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.048+0000 7fd88760c700 1 --2- 172.21.15.1:0/1537988890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd88006d560 0x7fd88006fac0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd8700098f0 tx=0x7fd870009c00 comp rx=0 tx=0).stop 2024-03-18T18:51:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.048+0000 7fd88760c700 1 -- 172.21.15.1:0/1537988890 shutdown_connections 2024-03-18T18:51:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.048+0000 7fd88760c700 1 --2- 172.21.15.1:0/1537988890 >> v2:172.21.15.165:3300/0 conn(0x7fd88015a240 0x7fd8800ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.048+0000 7fd88760c700 1 --2- 172.21.15.1:0/1537988890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd88006d560 0x7fd88006fac0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.048+0000 7fd88760c700 1 -- 172.21.15.1:0/1537988890 >> 172.21.15.1:0/1537988890 conn(0x7fd8800cff30 msgr2=0x7fd8800d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.048+0000 7fd88760c700 1 -- 172.21.15.1:0/1537988890 shutdown_connections 2024-03-18T18:51:22.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.048+0000 7fd88760c700 1 -- 172.21.15.1:0/1537988890 wait complete. 2024-03-18T18:51:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.049+0000 7fd88760c700 1 Processor -- start 2024-03-18T18:51:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.050+0000 7fd88760c700 1 -- start start 2024-03-18T18:51:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.050+0000 7fd88760c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd88006d560 0x7fd8800b4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.050+0000 7fd88760c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8800715b0 0x7fd8800c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.050+0000 7fd88760c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd88015a240 0x7fd8800c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.050+0000 7fd88760c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8800b3d60 con 0x7fd88006d560 2024-03-18T18:51:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.050+0000 7fd88760c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8800b30e0 con 0x7fd8800715b0 2024-03-18T18:51:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.050+0000 7fd88760c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8800b33c0 con 0x7fd88015a240 2024-03-18T18:51:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.050+0000 7fd8853a8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd88006d560 0x7fd8800b4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.050+0000 7fd8853a8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd88006d560 0x7fd8800b4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39948/0 (socket says 172.21.15.1:39948) 2024-03-18T18:51:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.050+0000 7fd8853a8700 1 -- 172.21.15.1:0/2110557473 learned_addr learned my addr 172.21.15.1:0/2110557473 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.050+0000 7fd884ba7700 1 --2- 172.21.15.1:0/2110557473 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8800715b0 0x7fd8800c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.051+0000 7fd885ba9700 1 --2- 172.21.15.1:0/2110557473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd88015a240 0x7fd8800c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.051+0000 7fd8853a8700 1 -- 172.21.15.1:0/2110557473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd88015a240 msgr2=0x7fd8800c7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:22.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.051+0000 7fd8853a8700 1 --2- 172.21.15.1:0/2110557473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd88015a240 0x7fd8800c7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.051+0000 7fd8853a8700 1 -- 172.21.15.1:0/2110557473 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8800715b0 msgr2=0x7fd8800c8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:22.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.051+0000 7fd8853a8700 1 --2- 172.21.15.1:0/2110557473 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8800715b0 0x7fd8800c8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.051+0000 7fd8853a8700 1 -- 172.21.15.1:0/2110557473 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd870009550 con 0x7fd88006d560 2024-03-18T18:51:22.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.051+0000 7fd8853a8700 1 --2- 172.21.15.1:0/2110557473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd88006d560 0x7fd8800b4180 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fd87c00e870 tx=0x7fd87c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:22.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.051+0000 7fd875ffb700 1 -- 172.21.15.1:0/2110557473 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87c009590 con 0x7fd88006d560 2024-03-18T18:51:22.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.051+0000 7fd875ffb700 1 -- 172.21.15.1:0/2110557473 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87c013680 con 0x7fd88006d560 2024-03-18T18:51:22.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.051+0000 7fd875ffb700 1 -- 172.21.15.1:0/2110557473 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87c0139f0 con 0x7fd88006d560 2024-03-18T18:51:22.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.051+0000 7fd88760c700 1 -- 172.21.15.1:0/2110557473 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8800b36a0 con 0x7fd88006d560 2024-03-18T18:51:22.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.051+0000 7fd88760c700 1 -- 172.21.15.1:0/2110557473 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8800b46c0 con 0x7fd88006d560 2024-03-18T18:51:22.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.052+0000 7fd88760c700 1 -- 172.21.15.1:0/2110557473 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8800b3fd0 con 0x7fd88006d560 2024-03-18T18:51:22.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.053+0000 7fd875ffb700 1 -- 172.21.15.1:0/2110557473 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 6) v1 ==== 89154+0+0 (secure 0 0 0) 0x7fd87c013b50 con 0x7fd88006d560 2024-03-18T18:51:22.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.053+0000 7fd875ffb700 1 --2- 172.21.15.1:0/2110557473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd86406ace0 0x7fd86406d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:22.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.053+0000 7fd884ba7700 1 -- 172.21.15.1:0/2110557473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd86406ace0 msgr2=0x7fd86406d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T18:51:22.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.053+0000 7fd875ffb700 1 -- 172.21.15.1:0/2110557473 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd87c089330 con 0x7fd88006d560 2024-03-18T18:51:22.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.053+0000 7fd884ba7700 1 --2- 172.21.15.1:0/2110557473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd86406ace0 0x7fd86406d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T18:51:22.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.062+0000 7fd875ffb700 1 -- 172.21.15.1:0/2110557473 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd87c0543c0 con 0x7fd88006d560 2024-03-18T18:51:22.174 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:22.177+0000 7fefd9382240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T18:51:22.208 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:22.211+0000 7f90e0994240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T18:51:22.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.254+0000 7fd884ba7700 1 -- 172.21.15.1:0/2110557473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd86406ace0 msgr2=0x7fd86406d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T18:51:22.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.254+0000 7fd884ba7700 1 --2- 172.21.15.1:0/2110557473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd86406ace0 0x7fd86406d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T18:51:22.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.314+0000 7fd88760c700 1 -- 172.21.15.1:0/2110557473 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fd8800676a0 con 0x7fd88006d560 2024-03-18T18:51:22.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.318+0000 7fd875ffb700 1 -- 172.21.15.1:0/2110557473 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v11)=0 v11) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd87c0579e0 con 0x7fd88006d560 2024-03-18T18:51:22.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.321+0000 7fd88760c700 1 -- 172.21.15.1:0/2110557473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd86406ace0 msgr2=0x7fd86406d1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T18:51:22.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.321+0000 7fd88760c700 1 --2- 172.21.15.1:0/2110557473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd86406ace0 0x7fd86406d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.321+0000 7fd88760c700 1 -- 172.21.15.1:0/2110557473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd88006d560 msgr2=0x7fd8800b4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:22.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.321+0000 7fd88760c700 1 --2- 172.21.15.1:0/2110557473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd88006d560 0x7fd8800b4180 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fd87c00e870 tx=0x7fd87c00eb80 comp rx=0 tx=0).stop 2024-03-18T18:51:22.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.321+0000 7fd88760c700 1 -- 172.21.15.1:0/2110557473 shutdown_connections 2024-03-18T18:51:22.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.321+0000 7fd88760c700 1 --2- 172.21.15.1:0/2110557473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd86406ace0 0x7fd86406d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.321+0000 7fd88760c700 1 --2- 172.21.15.1:0/2110557473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd88015a240 0x7fd8800c7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.321+0000 7fd88760c700 1 --2- 172.21.15.1:0/2110557473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd88006d560 0x7fd8800b4180 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.321+0000 7fd88760c700 1 --2- 172.21.15.1:0/2110557473 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8800715b0 0x7fd8800c8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.321+0000 7fd88760c700 1 -- 172.21.15.1:0/2110557473 >> 172.21.15.1:0/2110557473 conn(0x7fd8800cff30 msgr2=0x7fd8800cbe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:22.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.321+0000 7fd88760c700 1 -- 172.21.15.1:0/2110557473 shutdown_connections 2024-03-18T18:51:22.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.321+0000 7fd88760c700 1 -- 172.21.15.1:0/2110557473 wait complete. 2024-03-18T18:51:22.330 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-18T18:51:22.440 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:22.443+0000 7fefd9382240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T18:51:22.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.463+0000 7f43eef02700 1 Processor -- start 2024-03-18T18:51:22.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.463+0000 7f43eef02700 1 -- start start 2024-03-18T18:51:22.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.464+0000 7f43eef02700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f43e815d290 0x7f43e80af030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:22.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.464+0000 7f43eef02700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43e815d670 0x7f43e80af570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:22.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.464+0000 7f43eef02700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f43e80b4530 0x7f43e80b49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:22.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.464+0000 7f43eef02700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e806e3e0 con 0x7f43e80b4530 2024-03-18T18:51:22.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.464+0000 7f43eef02700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e80b4f00 con 0x7f43e815d290 2024-03-18T18:51:22.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.464+0000 7f43eef02700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e815db90 con 0x7f43e815d670 2024-03-18T18:51:22.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.464+0000 7f43ed49f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f43e80b4530 0x7f43e80b49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:22.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.464+0000 7f43ed49f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f43e80b4530 0x7f43e80b49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39956/0 (socket says 172.21.15.1:39956) 2024-03-18T18:51:22.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.464+0000 7f43ed49f700 1 -- 172.21.15.1:0/350459969 learned_addr learned my addr 172.21.15.1:0/350459969 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:22.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.464+0000 7f43ecc9e700 1 --2- 172.21.15.1:0/350459969 >> v2:172.21.15.165:3300/0 conn(0x7f43e815d290 0x7f43e80af030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:22.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.464+0000 7f43e7fff700 1 --2- 172.21.15.1:0/350459969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43e815d670 0x7f43e80af570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:22.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.464+0000 7f43ed49f700 1 -- 172.21.15.1:0/350459969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43e815d670 msgr2=0x7f43e80af570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:22.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.464+0000 7f43ed49f700 1 --2- 172.21.15.1:0/350459969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43e815d670 0x7f43e80af570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.464+0000 7f43ed49f700 1 -- 172.21.15.1:0/350459969 >> v2:172.21.15.165:3300/0 conn(0x7f43e815d290 msgr2=0x7f43e80af030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:22.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.464+0000 7f43ed49f700 1 --2- 172.21.15.1:0/350459969 >> v2:172.21.15.165:3300/0 conn(0x7f43e815d290 0x7f43e80af030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.464+0000 7f43ed49f700 1 -- 172.21.15.1:0/350459969 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e815de70 con 0x7f43e80b4530 2024-03-18T18:51:22.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.464+0000 7f43ecc9e700 1 --2- 172.21.15.1:0/350459969 >> v2:172.21.15.165:3300/0 conn(0x7f43e815d290 0x7f43e80af030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:22.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.464+0000 7f43ed49f700 1 --2- 172.21.15.1:0/350459969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e80b4530 0x7f43e80b49c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f43e80b2c60 tx=0x7f43e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c22557653aa0eeb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:22.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.465+0000 7f43e77fe700 1 -- 172.21.15.1:0/350459969 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e000e070 con 0x7f43e80b4530 2024-03-18T18:51:22.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.465+0000 7f43e77fe700 1 -- 172.21.15.1:0/350459969 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e000b9c0 con 0x7f43e80b4530 2024-03-18T18:51:22.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.465+0000 7f43e77fe700 1 -- 172.21.15.1:0/350459969 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e000bd70 con 0x7f43e80b4530 2024-03-18T18:51:22.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.465+0000 7f43eef02700 1 -- 172.21.15.1:0/350459969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e80b4530 msgr2=0x7f43e80b49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:22.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.465+0000 7f43eef02700 1 --2- 172.21.15.1:0/350459969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e80b4530 0x7f43e80b49c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f43e80b2c60 tx=0x7f43e000b330 comp rx=0 tx=0).stop 2024-03-18T18:51:22.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.465+0000 7f43eef02700 1 -- 172.21.15.1:0/350459969 shutdown_connections 2024-03-18T18:51:22.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.465+0000 7f43eef02700 1 --2- 172.21.15.1:0/350459969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e80b4530 0x7f43e80b49c0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.465+0000 7f43eef02700 1 --2- 172.21.15.1:0/350459969 >> v2:172.21.15.165:3300/0 conn(0x7f43e815d290 0x7f43e80af030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.465+0000 7f43eef02700 1 --2- 172.21.15.1:0/350459969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43e815d670 0x7f43e80af570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.465+0000 7f43eef02700 1 -- 172.21.15.1:0/350459969 >> 172.21.15.1:0/350459969 conn(0x7f43e806d500 msgr2=0x7f43e806d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:22.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.465+0000 7f43eef02700 1 -- 172.21.15.1:0/350459969 shutdown_connections 2024-03-18T18:51:22.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.465+0000 7f43eef02700 1 -- 172.21.15.1:0/350459969 wait complete. 2024-03-18T18:51:22.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.467+0000 7f43eef02700 1 Processor -- start 2024-03-18T18:51:22.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.467+0000 7f43eef02700 1 -- start start 2024-03-18T18:51:22.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.468+0000 7f43eef02700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e80b4530 0x7f43e815c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:22.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.468+0000 7f43eef02700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43e815d290 0x7f43e80b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:22.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.468+0000 7f43eef02700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43e815d670 0x7f43e814c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:22.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.468+0000 7f43eef02700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e815cb90 con 0x7f43e80b4530 2024-03-18T18:51:22.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.468+0000 7f43eef02700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e80b0ab0 con 0x7f43e815d290 2024-03-18T18:51:22.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.468+0000 7f43eef02700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e80b0ca0 con 0x7f43e815d670 2024-03-18T18:51:22.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.468+0000 7f43ecc9e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e80b4530 0x7f43e815c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:22.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.468+0000 7f43ecc9e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e80b4530 0x7f43e815c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39970/0 (socket says 172.21.15.1:39970) 2024-03-18T18:51:22.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.468+0000 7f43ecc9e700 1 -- 172.21.15.1:0/2227157523 learned_addr learned my addr 172.21.15.1:0/2227157523 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:22.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.468+0000 7f43e7fff700 1 --2- 172.21.15.1:0/2227157523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43e815d290 0x7f43e80b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:22.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.468+0000 7f43ed49f700 1 --2- 172.21.15.1:0/2227157523 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43e815d670 0x7f43e814c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:22.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.468+0000 7f43ecc9e700 1 -- 172.21.15.1:0/2227157523 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43e815d670 msgr2=0x7f43e814c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:22.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.468+0000 7f43ecc9e700 1 --2- 172.21.15.1:0/2227157523 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43e815d670 0x7f43e814c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.468+0000 7f43ecc9e700 1 -- 172.21.15.1:0/2227157523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43e815d290 msgr2=0x7f43e80b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:22.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.468+0000 7f43ecc9e700 1 --2- 172.21.15.1:0/2227157523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43e815d290 0x7f43e80b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.468+0000 7f43ecc9e700 1 -- 172.21.15.1:0/2227157523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e0009b30 con 0x7f43e80b4530 2024-03-18T18:51:22.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.468+0000 7f43ecc9e700 1 --2- 172.21.15.1:0/2227157523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e80b4530 0x7f43e815c650 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f43dc00e810 tx=0x7f43dc00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:22.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.469+0000 7f43e57fa700 1 -- 172.21.15.1:0/2227157523 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc013610 con 0x7f43e80b4530 2024-03-18T18:51:22.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.469+0000 7f43e57fa700 1 -- 172.21.15.1:0/2227157523 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43dc013770 con 0x7f43e80b4530 2024-03-18T18:51:22.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.469+0000 7f43eef02700 1 -- 172.21.15.1:0/2227157523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e80b0f80 con 0x7f43e80b4530 2024-03-18T18:51:22.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.469+0000 7f43e57fa700 1 -- 172.21.15.1:0/2227157523 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc013b20 con 0x7f43e80b4530 2024-03-18T18:51:22.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.469+0000 7f43eef02700 1 -- 172.21.15.1:0/2227157523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e80b1540 con 0x7f43e80b4530 2024-03-18T18:51:22.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.470+0000 7f43eef02700 1 -- 172.21.15.1:0/2227157523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e80ccf50 con 0x7f43e80b4530 2024-03-18T18:51:22.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.470+0000 7f43e57fa700 1 -- 172.21.15.1:0/2227157523 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 6) v1 ==== 89154+0+0 (secure 0 0 0) 0x7f43dc013c80 con 0x7f43e80b4530 2024-03-18T18:51:22.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.471+0000 7f43e57fa700 1 --2- 172.21.15.1:0/2227157523 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43cc06ace0 0x7f43cc06d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:22.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.471+0000 7f43e7fff700 1 -- 172.21.15.1:0/2227157523 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43cc06ace0 msgr2=0x7f43cc06d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T18:51:22.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.471+0000 7f43e57fa700 1 -- 172.21.15.1:0/2227157523 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f43dc088f40 con 0x7f43e80b4530 2024-03-18T18:51:22.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.471+0000 7f43e7fff700 1 --2- 172.21.15.1:0/2227157523 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43cc06ace0 0x7f43cc06d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T18:51:22.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.475+0000 7f43e57fa700 1 -- 172.21.15.1:0/2227157523 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f43dc054070 con 0x7f43e80b4530 2024-03-18T18:51:22.492 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:22.496+0000 7f90e0994240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T18:51:22.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.672+0000 7f43e7fff700 1 -- 172.21.15.1:0/2227157523 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43cc06ace0 msgr2=0x7f43cc06d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T18:51:22.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.672+0000 7f43e7fff700 1 --2- 172.21.15.1:0/2227157523 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43cc06ace0 0x7f43cc06d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T18:51:22.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.678+0000 7f43eef02700 1 -- 172.21.15.1:0/2227157523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f43e8073350 con 0x7f43e80b4530 2024-03-18T18:51:22.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.682+0000 7f43e57fa700 1 -- 172.21.15.1:0/2227157523 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v12)=0 v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7f43dc057690 con 0x7f43e80b4530 2024-03-18T18:51:22.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.684+0000 7f43eef02700 1 -- 172.21.15.1:0/2227157523 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43cc06ace0 msgr2=0x7f43cc06d1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T18:51:22.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.684+0000 7f43eef02700 1 --2- 172.21.15.1:0/2227157523 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43cc06ace0 0x7f43cc06d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.684+0000 7f43eef02700 1 -- 172.21.15.1:0/2227157523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e80b4530 msgr2=0x7f43e815c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:22.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.684+0000 7f43eef02700 1 --2- 172.21.15.1:0/2227157523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e80b4530 0x7f43e815c650 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f43dc00e810 tx=0x7f43dc00eb20 comp rx=0 tx=0).stop 2024-03-18T18:51:22.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.684+0000 7f43eef02700 1 -- 172.21.15.1:0/2227157523 shutdown_connections 2024-03-18T18:51:22.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.684+0000 7f43eef02700 1 --2- 172.21.15.1:0/2227157523 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43cc06ace0 0x7f43cc06d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.684+0000 7f43eef02700 1 --2- 172.21.15.1:0/2227157523 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43e815d670 0x7f43e814c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.684+0000 7f43eef02700 1 --2- 172.21.15.1:0/2227157523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43e80b4530 0x7f43e815c650 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.684+0000 7f43eef02700 1 --2- 172.21.15.1:0/2227157523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43e815d290 0x7f43e80b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.684+0000 7f43eef02700 1 -- 172.21.15.1:0/2227157523 >> 172.21.15.1:0/2227157523 conn(0x7f43e806d500 msgr2=0x7f43e80af1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:22.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.684+0000 7f43eef02700 1 -- 172.21.15.1:0/2227157523 shutdown_connections 2024-03-18T18:51:22.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.684+0000 7f43eef02700 1 -- 172.21.15.1:0/2227157523 wait complete. 2024-03-18T18:51:22.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:22.693 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-03-18T18:51:22.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:22.717 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2024-03-18T18:51:22.779 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir0 1m 2024-03-18T18:51:22.871 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:22.874+0000 7fefd9382240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T18:51:22.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.953+0000 7fc54a5c0700 1 Processor -- start 2024-03-18T18:51:22.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.953+0000 7fc54a5c0700 1 -- start start 2024-03-18T18:51:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.954+0000 7fc54a5c0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5440b2650 0x7fc544070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.954+0000 7fc54a5c0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5440ac8b0 0x7fc544071380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:22.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.954+0000 7fc54a5c0700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5440b2b80 con 0x7fc54415d3b0 2024-03-18T18:51:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.954+0000 7fc54a5c0700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5440b3800 con 0x7fc5440ac8b0 2024-03-18T18:51:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.954+0000 7fc54a5c0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5440b2e60 con 0x7fc5440b2650 2024-03-18T18:51:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.954+0000 7fc548b5d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5440ac8b0 0x7fc544071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.954+0000 7fc548b5d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5440ac8b0 0x7fc544071380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39978/0 (socket says 172.21.15.1:39978) 2024-03-18T18:51:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.954+0000 7fc548b5d700 1 -- 172.21.15.1:0/1838016418 learned_addr learned my addr 172.21.15.1:0/1838016418 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.954+0000 7fc5437fe700 1 --2- 172.21.15.1:0/1838016418 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5440b2650 0x7fc544070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.955+0000 7fc548b5d700 1 -- 172.21.15.1:0/1838016418 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5440b2650 msgr2=0x7fc544070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.955+0000 7fc548b5d700 1 --2- 172.21.15.1:0/1838016418 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5440b2650 0x7fc544070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.955+0000 7fc548b5d700 1 -- 172.21.15.1:0/1838016418 >> v1:172.21.15.1:6789/0 conn(0x7fc54415d3b0 legacy=0x7fc54406cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.955+0000 7fc548b5d700 1 -- 172.21.15.1:0/1838016418 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5440b3140 con 0x7fc5440ac8b0 2024-03-18T18:51:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.955+0000 7fc542ffd700 1 -- 172.21.15.1:0/1838016418 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2987630757 0 0) 0x7fc5440b2b80 con 0x7fc54415d3b0 2024-03-18T18:51:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.955+0000 7fc548b5d700 1 --2- 172.21.15.1:0/1838016418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5440ac8b0 0x7fc544071380 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc53c009e50 tx=0x7fc53c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c9dcea0e65b72ba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.955+0000 7fc542ffd700 1 -- 172.21.15.1:0/1838016418 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c00e070 con 0x7fc5440ac8b0 2024-03-18T18:51:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.955+0000 7fc542ffd700 1 -- 172.21.15.1:0/1838016418 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc53c00b910 con 0x7fc5440ac8b0 2024-03-18T18:51:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.955+0000 7fc542ffd700 1 -- 172.21.15.1:0/1838016418 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c00bcc0 con 0x7fc5440ac8b0 2024-03-18T18:51:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.955+0000 7fc54a5c0700 1 -- 172.21.15.1:0/1838016418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5440ac8b0 msgr2=0x7fc544071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.956+0000 7fc54a5c0700 1 --2- 172.21.15.1:0/1838016418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5440ac8b0 0x7fc544071380 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc53c009e50 tx=0x7fc53c00b330 comp rx=0 tx=0).stop 2024-03-18T18:51:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.956+0000 7fc54a5c0700 1 -- 172.21.15.1:0/1838016418 shutdown_connections 2024-03-18T18:51:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.956+0000 7fc54a5c0700 1 --2- 172.21.15.1:0/1838016418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5440ac8b0 0x7fc544071380 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.956+0000 7fc54a5c0700 1 --2- 172.21.15.1:0/1838016418 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5440b2650 0x7fc544070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.956+0000 7fc54a5c0700 1 -- 172.21.15.1:0/1838016418 >> 172.21.15.1:0/1838016418 conn(0x7fc5440b1350 msgr2=0x7fc5440b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.956+0000 7fc54a5c0700 1 -- 172.21.15.1:0/1838016418 shutdown_connections 2024-03-18T18:51:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.956+0000 7fc54a5c0700 1 -- 172.21.15.1:0/1838016418 wait complete. 2024-03-18T18:51:22.955 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:22.956+0000 7f90e0994240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T18:51:22.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.957+0000 7fc54a5c0700 1 Processor -- start 2024-03-18T18:51:22.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.957+0000 7fc54a5c0700 1 -- start start 2024-03-18T18:51:22.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.958+0000 7fc54a5c0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5440ac8b0 0x7fc5440c45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:22.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.958+0000 7fc54a5c0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5440b2650 0x7fc5441f6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:22.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.958+0000 7fc54a5c0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc54415d3b0 0x7fc5441f6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:22.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.958+0000 7fc54a5c0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc544156af0 con 0x7fc54415d3b0 2024-03-18T18:51:22.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.958+0000 7fc54a5c0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5441597e0 con 0x7fc5440b2650 2024-03-18T18:51:22.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.958+0000 7fc54a5c0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc544158c60 con 0x7fc5440ac8b0 2024-03-18T18:51:22.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.958+0000 7fc548b5d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc54415d3b0 0x7fc5441f6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:22.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.958+0000 7fc548b5d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc54415d3b0 0x7fc5441f6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39982/0 (socket says 172.21.15.1:39982) 2024-03-18T18:51:22.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.958+0000 7fc5437fe700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5440b2650 0x7fc5441f6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:22.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.958+0000 7fc548b5d700 1 -- 172.21.15.1:0/1302878765 learned_addr learned my addr 172.21.15.1:0/1302878765 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:22.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.958+0000 7fc543fff700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5440ac8b0 0x7fc5440c45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:22.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.958+0000 7fc548b5d700 1 -- 172.21.15.1:0/1302878765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5440ac8b0 msgr2=0x7fc5440c45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:22.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.958+0000 7fc548b5d700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5440ac8b0 0x7fc5440c45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.958+0000 7fc548b5d700 1 -- 172.21.15.1:0/1302878765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5440b2650 msgr2=0x7fc5441f6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:22.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.958+0000 7fc548b5d700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5440b2650 0x7fc5441f6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:22.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.958+0000 7fc548b5d700 1 -- 172.21.15.1:0/1302878765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53c009b30 con 0x7fc54415d3b0 2024-03-18T18:51:22.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.958+0000 7fc5437fe700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5440b2650 0x7fc5441f6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:51:22.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.958+0000 7fc543fff700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5440ac8b0 0x7fc5440c45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:51:22.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.959+0000 7fc548b5d700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc54415d3b0 0x7fc5441f6840 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc53c00b680 tx=0x7fc53c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:22.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.959+0000 7fc540ff9700 1 -- 172.21.15.1:0/1302878765 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c0048b0 con 0x7fc54415d3b0 2024-03-18T18:51:22.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.959+0000 7fc540ff9700 1 -- 172.21.15.1:0/1302878765 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc53c0092e0 con 0x7fc54415d3b0 2024-03-18T18:51:22.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.959+0000 7fc54a5c0700 1 -- 172.21.15.1:0/1302878765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc544158f40 con 0x7fc54415d3b0 2024-03-18T18:51:22.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.959+0000 7fc540ff9700 1 -- 172.21.15.1:0/1302878765 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c01c660 con 0x7fc54415d3b0 2024-03-18T18:51:22.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.959+0000 7fc54a5c0700 1 -- 172.21.15.1:0/1302878765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc544159500 con 0x7fc54415d3b0 2024-03-18T18:51:22.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.960+0000 7fc54a5c0700 1 -- 172.21.15.1:0/1302878765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc544153ee0 con 0x7fc54415d3b0 2024-03-18T18:51:22.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.960+0000 7fc540ff9700 1 -- 172.21.15.1:0/1302878765 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 6) v1 ==== 89154+0+0 (secure 0 0 0) 0x7fc53c01c7c0 con 0x7fc54415d3b0 2024-03-18T18:51:22.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.961+0000 7fc540ff9700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc53006ace0 0x7fc53006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:22.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.961+0000 7fc543fff700 1 -- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc53006ace0 msgr2=0x7fc53006d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T18:51:22.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.961+0000 7fc540ff9700 1 -- 172.21.15.1:0/1302878765 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc53c08af90 con 0x7fc54415d3b0 2024-03-18T18:51:22.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.961+0000 7fc543fff700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc53006ace0 0x7fc53006d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T18:51:22.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:22.965+0000 7fc540ff9700 1 -- 172.21.15.1:0/1302878765 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc53c0be580 con 0x7fc54415d3b0 2024-03-18T18:51:23.081 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:23.084+0000 7fefd9382240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T18:51:23.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:23.162+0000 7fc543fff700 1 -- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc53006ace0 msgr2=0x7fc53006d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T18:51:23.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:23.162+0000 7fc543fff700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc53006ace0 0x7fc53006d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T18:51:23.187 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:23.191+0000 7f90e0994240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T18:51:23.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:23.223+0000 7fc54a5c0700 1 -- 172.21.15.1:0/1302878765 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc54406d600 con 0x7fc53006ace0 2024-03-18T18:51:23.451 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:23.454+0000 7fefd9382240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T18:51:23.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:23.563+0000 7fc543fff700 1 -- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc53006ace0 msgr2=0x7fc53006d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T18:51:23.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:23.563+0000 7fc543fff700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc53006ace0 0x7fc53006d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T18:51:23.571 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:23.575+0000 7f90e0994240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T18:51:24.035 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:24.039+0000 7fefd9382240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T18:51:24.107 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:24.110+0000 7fefd9382240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T18:51:24.184 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:24.187+0000 7f90e0994240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T18:51:24.234 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:24.237+0000 7fefd9382240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T18:51:24.257 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:24.261+0000 7f90e0994240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T18:51:24.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:24.364+0000 7fc543fff700 1 -- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc53006ace0 msgr2=0x7fc53006d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T18:51:24.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:24.364+0000 7fc543fff700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc53006ace0 0x7fc53006d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T18:51:24.390 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:24.393+0000 7f90e0994240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T18:51:24.411 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:24.414+0000 7fefd9382240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T18:51:24.482 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:24.485+0000 7fefd9382240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T18:51:24.560 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:24.563+0000 7fefd9382240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T18:51:24.573 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:24.577+0000 7f90e0994240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T18:51:24.637 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:24.640+0000 7fefd9382240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T18:51:24.647 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:24.651+0000 7f90e0994240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T18:51:24.713 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:24.716+0000 7fefd9382240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T18:51:24.727 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:24.731+0000 7f90e0994240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T18:51:24.809 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:24.811+0000 7f90e0994240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T18:51:24.881 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:24.884+0000 7fefd9382240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T18:51:24.887 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:24.891+0000 7f90e0994240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T18:51:25.062 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:25.066+0000 7f90e0994240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T18:51:25.095 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:25.098+0000 7fefd9382240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T18:51:25.244 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:25.247+0000 7fefd9382240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T18:51:25.281 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:25.285+0000 7f90e0994240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T18:51:25.315 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:25.319+0000 7fefd9382240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T18:51:25.436 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:25.440+0000 7f90e0994240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T18:51:25.511 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:25.515+0000 7f90e0994240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T18:51:25.760 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:25.763+0000 7fefd9382240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T18:51:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:25.965+0000 7fc543fff700 1 -- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc53006ace0 msgr2=0x7fc53006d1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T18:51:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:25.965+0000 7fc543fff700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc53006ace0 0x7fc53006d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T18:51:25.975 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:25.979+0000 7f90e0994240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T18:51:26.094 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:26.097+0000 7fefd9382240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T18:51:26.253 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:26.257+0000 7fefd9382240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T18:51:26.320 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:26.323+0000 7f90e0994240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T18:51:26.486 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:26.490+0000 7f90e0994240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T18:51:26.639 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:26.642+0000 7fefd9382240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T18:51:26.801 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:26.804+0000 7fefd9382240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T18:51:26.889 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:26.892+0000 7f90e0994240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T18:51:27.058 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:27.062+0000 7f90e0994240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T18:51:27.161 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:27.165+0000 7fefd9382240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T18:51:27.249 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T18:51:27.252+0000 7fefd9382240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T18:51:27.428 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:27.432+0000 7f90e0994240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T18:51:27.518 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:27.522+0000 7f90e0994240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T18:51:27.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:27.542+0000 7fc540ff9700 1 -- 172.21.15.1:0/1302878765 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 7) v1 ==== 88865+0+0 (secure 0 0 0) 0x7fc53c056490 con 0x7fc54415d3b0 2024-03-18T18:51:27.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:27.542+0000 7fc540ff9700 1 -- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc53006ace0 msgr2=0x7fc53006d1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T18:51:27.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:27.542+0000 7fc540ff9700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc53006ace0 0x7fc53006d1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:27.700 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:27.703+0000 7f90565b7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:51:27.700 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:27.703+0000 7f90565b7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:51:27.701 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:27.703+0000 7f90565b7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:51:27.701 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:27.703+0000 7f90565b7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:51:27.701 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:27.703+0000 7f90565b7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:51:27.701 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:27.704+0000 7f90535b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:51:27.701 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:27.704+0000 7f90535b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:51:27.701 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:27.704+0000 7f90535b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:51:27.701 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:27.704+0000 7f90535b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:51:27.701 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:51:27.704+0000 7f90535b1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:51:28.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.547+0000 7fc540ff9700 1 -- 172.21.15.1:0/1302878765 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 8) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fc53c02d350 con 0x7fc54415d3b0 2024-03-18T18:51:28.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.548+0000 7fc540ff9700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc53006ace0 0x7fc53006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:28.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.548+0000 7fc540ff9700 1 -- 172.21.15.1:0/1302878765 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc53001d470 con 0x7fc53006ace0 2024-03-18T18:51:28.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.550+0000 7fc543fff700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc53006ace0 0x7fc53006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:28.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.553+0000 7fc543fff700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc53006ace0 0x7fc53006d1a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc534007fd0 tx=0x7fc534009040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:28.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.633+0000 7fc540ff9700 1 -- 172.21.15.1:0/1302878765 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fc53001d470 con 0x7fc53006ace0 2024-03-18T18:51:28.629 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path /snap_test_dir1/dir0 2024-03-18T18:51:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.635+0000 7fc54a5c0700 1 -- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc53006ace0 msgr2=0x7fc53006d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.635+0000 7fc54a5c0700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc53006ace0 0x7fc53006d1a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc534007fd0 tx=0x7fc534009040 comp rx=0 tx=0).stop 2024-03-18T18:51:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.635+0000 7fc54a5c0700 1 -- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc54415d3b0 msgr2=0x7fc5441f6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.635+0000 7fc54a5c0700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc54415d3b0 0x7fc5441f6840 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc53c00b680 tx=0x7fc53c004ad0 comp rx=0 tx=0).stop 2024-03-18T18:51:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.635+0000 7fc54a5c0700 1 -- 172.21.15.1:0/1302878765 shutdown_connections 2024-03-18T18:51:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.635+0000 7fc54a5c0700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc53006ace0 0x7fc53006d1a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.635+0000 7fc54a5c0700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5440ac8b0 0x7fc5440c45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.635+0000 7fc54a5c0700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc54415d3b0 0x7fc5441f6840 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.635+0000 7fc54a5c0700 1 --2- 172.21.15.1:0/1302878765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5440b2650 0x7fc5441f6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.635+0000 7fc54a5c0700 1 -- 172.21.15.1:0/1302878765 >> 172.21.15.1:0/1302878765 conn(0x7fc5440b1350 msgr2=0x7fc544160120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:28.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.635+0000 7fc54a5c0700 1 -- 172.21.15.1:0/1302878765 shutdown_connections 2024-03-18T18:51:28.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.635+0000 7fc54a5c0700 1 -- 172.21.15.1:0/1302878765 wait complete. 2024-03-18T18:51:28.645 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2024-03-18T18:51:28.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:28.645 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2024-03-18T18:51:28.666 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir1 1m 2024-03-18T18:51:28.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.840+0000 7fca081af700 1 Processor -- start 2024-03-18T18:51:28.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.840+0000 7fca081af700 1 -- start start 2024-03-18T18:51:28.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.841+0000 7fca081af700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca000cb7b0 0x7fca00099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:28.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.841+0000 7fca081af700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fca00099f80 0x7fca000cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:28.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.841+0000 7fca081af700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0009a3e0 con 0x7fca0015d500 2024-03-18T18:51:28.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.841+0000 7fca081af700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca0006f490 con 0x7fca00099f80 2024-03-18T18:51:28.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.841+0000 7fca081af700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca0006e880 con 0x7fca000cb7b0 2024-03-18T18:51:28.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.841+0000 7fca0674c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fca00099f80 0x7fca000cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:28.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.841+0000 7fca0674c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fca00099f80 0x7fca000cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41818/0 (socket says 172.21.15.1:41818) 2024-03-18T18:51:28.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.841+0000 7fca0674c700 1 -- 172.21.15.1:0/3732312473 learned_addr learned my addr 172.21.15.1:0/3732312473 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:28.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.842+0000 7fca0574a700 1 --2- 172.21.15.1:0/3732312473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca000cb7b0 0x7fca00099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:28.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.842+0000 7fca0674c700 1 -- 172.21.15.1:0/3732312473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca000cb7b0 msgr2=0x7fca00099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:28.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.842+0000 7fca0674c700 1 --2- 172.21.15.1:0/3732312473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca000cb7b0 0x7fca00099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:28.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.842+0000 7fca0674c700 1 -- 172.21.15.1:0/3732312473 >> v1:172.21.15.1:6789/0 conn(0x7fca0015d500 legacy=0x7fca0006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:28.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.842+0000 7fca04f49700 1 -- 172.21.15.1:0/3732312473 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706562668 0 0) 0x7fca0009a3e0 con 0x7fca0015d500 2024-03-18T18:51:28.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.842+0000 7fca0674c700 1 -- 172.21.15.1:0/3732312473 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0006eb60 con 0x7fca00099f80 2024-03-18T18:51:28.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.842+0000 7fca0574a700 1 --2- 172.21.15.1:0/3732312473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca000cb7b0 0x7fca00099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:28.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.842+0000 7fca0674c700 1 --2- 172.21.15.1:0/3732312473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca00099f80 0x7fca000cccd0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fc9f8009e50 tx=0x7fc9f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f114b60d493d527 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:28.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.842+0000 7fca04f49700 1 -- 172.21.15.1:0/3732312473 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f800e070 con 0x7fca00099f80 2024-03-18T18:51:28.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.842+0000 7fca04f49700 1 -- 172.21.15.1:0/3732312473 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f800b9c0 con 0x7fca00099f80 2024-03-18T18:51:28.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.842+0000 7fca04f49700 1 -- 172.21.15.1:0/3732312473 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f800bd70 con 0x7fca00099f80 2024-03-18T18:51:28.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.843+0000 7fca081af700 1 -- 172.21.15.1:0/3732312473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca00099f80 msgr2=0x7fca000cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:28.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.843+0000 7fca081af700 1 --2- 172.21.15.1:0/3732312473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca00099f80 0x7fca000cccd0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fc9f8009e50 tx=0x7fc9f800b330 comp rx=0 tx=0).stop 2024-03-18T18:51:28.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.843+0000 7fca081af700 1 -- 172.21.15.1:0/3732312473 shutdown_connections 2024-03-18T18:51:28.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.843+0000 7fca081af700 1 --2- 172.21.15.1:0/3732312473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca00099f80 0x7fca000cccd0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:28.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.843+0000 7fca081af700 1 --2- 172.21.15.1:0/3732312473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca000cb7b0 0x7fca00099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:28.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.843+0000 7fca081af700 1 -- 172.21.15.1:0/3732312473 >> 172.21.15.1:0/3732312473 conn(0x7fca000b2c50 msgr2=0x7fca000b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:28.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.843+0000 7fca081af700 1 -- 172.21.15.1:0/3732312473 shutdown_connections 2024-03-18T18:51:28.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.843+0000 7fca081af700 1 -- 172.21.15.1:0/3732312473 wait complete. 2024-03-18T18:51:28.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.844+0000 7fca081af700 1 Processor -- start 2024-03-18T18:51:28.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.844+0000 7fca081af700 1 -- start start 2024-03-18T18:51:28.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fca081af700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca00099f80 0x7fca00159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:28.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fca081af700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca000cb7b0 0x7fca001525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:28.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fca081af700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca0015d500 0x7fca000ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:28.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fca081af700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca000ca800 con 0x7fca0015d500 2024-03-18T18:51:28.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fca081af700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca000cb060 con 0x7fca000cb7b0 2024-03-18T18:51:28.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fca081af700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca000caaa0 con 0x7fca00099f80 2024-03-18T18:51:28.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fca0674c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca0015d500 0x7fca000ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:28.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fca0674c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca0015d500 0x7fca000ccfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41832/0 (socket says 172.21.15.1:41832) 2024-03-18T18:51:28.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fca0674c700 1 -- 172.21.15.1:0/1156960721 learned_addr learned my addr 172.21.15.1:0/1156960721 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:28.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fca05f4b700 1 --2- 172.21.15.1:0/1156960721 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca00099f80 0x7fca00159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:28.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fca0574a700 1 --2- 172.21.15.1:0/1156960721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca000cb7b0 0x7fca001525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:28.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fca0674c700 1 -- 172.21.15.1:0/1156960721 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca00099f80 msgr2=0x7fca00159b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:28.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fca0674c700 1 --2- 172.21.15.1:0/1156960721 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca00099f80 0x7fca00159b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:28.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fca0674c700 1 -- 172.21.15.1:0/1156960721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca000cb7b0 msgr2=0x7fca001525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:28.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fca0674c700 1 --2- 172.21.15.1:0/1156960721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca000cb7b0 0x7fca001525d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:28.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fca0674c700 1 -- 172.21.15.1:0/1156960721 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f8009b30 con 0x7fca0015d500 2024-03-18T18:51:28.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fca0674c700 1 --2- 172.21.15.1:0/1156960721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca0015d500 0x7fca000ccfa0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fca00072230 tx=0x7fc9f8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:28.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fc9f67fc700 1 -- 172.21.15.1:0/1156960721 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f80092e0 con 0x7fca0015d500 2024-03-18T18:51:28.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fc9f67fc700 1 -- 172.21.15.1:0/1156960721 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f80099b0 con 0x7fca0015d500 2024-03-18T18:51:28.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fca081af700 1 -- 172.21.15.1:0/1156960721 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca000cad80 con 0x7fca0015d500 2024-03-18T18:51:28.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fc9f67fc700 1 -- 172.21.15.1:0/1156960721 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f801c660 con 0x7fca0015d500 2024-03-18T18:51:28.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.845+0000 7fca081af700 1 -- 172.21.15.1:0/1156960721 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca000ccaa0 con 0x7fca0015d500 2024-03-18T18:51:28.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.846+0000 7fca081af700 1 -- 172.21.15.1:0/1156960721 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca000cce10 con 0x7fca0015d500 2024-03-18T18:51:28.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.847+0000 7fc9f67fc700 1 -- 172.21.15.1:0/1156960721 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fc9f801c7c0 con 0x7fca0015d500 2024-03-18T18:51:28.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.847+0000 7fc9f67fc700 1 --2- 172.21.15.1:0/1156960721 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc9e406ab30 0x7fc9e406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:28.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.847+0000 7fc9f67fc700 1 -- 172.21.15.1:0/1156960721 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fc9f808af90 con 0x7fca0015d500 2024-03-18T18:51:28.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.847+0000 7fca05f4b700 1 --2- 172.21.15.1:0/1156960721 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc9e406ab30 0x7fc9e406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:28.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.847+0000 7fca05f4b700 1 --2- 172.21.15.1:0/1156960721 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc9e406ab30 0x7fc9e406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0000f60 tx=0x7fc9f0009040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:28.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:28.852+0000 7fc9f67fc700 1 -- 172.21.15.1:0/1156960721 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc9f8056270 con 0x7fca0015d500 2024-03-18T18:51:29.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.086+0000 7fca081af700 1 -- 172.21.15.1:0/1156960721 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fca000cc650 con 0x7fc9e406ab30 2024-03-18T18:51:29.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.144+0000 7fc9f67fc700 1 -- 172.21.15.1:0/1156960721 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fca000cc650 con 0x7fc9e406ab30 2024-03-18T18:51:29.140 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path /snap_test_dir1/dir1 2024-03-18T18:51:29.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.146+0000 7fca081af700 1 -- 172.21.15.1:0/1156960721 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc9e406ab30 msgr2=0x7fc9e406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:29.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.146+0000 7fca081af700 1 --2- 172.21.15.1:0/1156960721 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc9e406ab30 0x7fc9e406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0000f60 tx=0x7fc9f0009040 comp rx=0 tx=0).stop 2024-03-18T18:51:29.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.146+0000 7fca081af700 1 -- 172.21.15.1:0/1156960721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca0015d500 msgr2=0x7fca000ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:29.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.146+0000 7fca081af700 1 --2- 172.21.15.1:0/1156960721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca0015d500 0x7fca000ccfa0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fca00072230 tx=0x7fc9f8004ad0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.146+0000 7fca081af700 1 -- 172.21.15.1:0/1156960721 shutdown_connections 2024-03-18T18:51:29.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.146+0000 7fca081af700 1 --2- 172.21.15.1:0/1156960721 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc9e406ab30 0x7fc9e406cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.146+0000 7fca081af700 1 --2- 172.21.15.1:0/1156960721 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca00099f80 0x7fca00159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.146+0000 7fca081af700 1 --2- 172.21.15.1:0/1156960721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca0015d500 0x7fca000ccfa0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.146+0000 7fca081af700 1 --2- 172.21.15.1:0/1156960721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca000cb7b0 0x7fca001525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.146+0000 7fca081af700 1 -- 172.21.15.1:0/1156960721 >> 172.21.15.1:0/1156960721 conn(0x7fca000b2c50 msgr2=0x7fca0006b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:29.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.146+0000 7fca081af700 1 -- 172.21.15.1:0/1156960721 shutdown_connections 2024-03-18T18:51:29.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.146+0000 7fca081af700 1 -- 172.21.15.1:0/1156960721 wait complete. 2024-03-18T18:51:29.155 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2024-03-18T18:51:29.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:29.155 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2024-03-18T18:51:29.176 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir2 1m 2024-03-18T18:51:29.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.348+0000 7fe1253e6700 1 Processor -- start 2024-03-18T18:51:29.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.348+0000 7fe1253e6700 1 -- start start 2024-03-18T18:51:29.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.349+0000 7fe1253e6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe12015a380 0x7fe12015b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:29.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.349+0000 7fe1253e6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe1200724e0 0x7fe12015baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:29.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.349+0000 7fe1253e6700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe12015b6f0 con 0x7fe120071490 2024-03-18T18:51:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.349+0000 7fe1253e6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1200cb510 con 0x7fe12015a380 2024-03-18T18:51:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.349+0000 7fe1253e6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1200ca900 con 0x7fe1200724e0 2024-03-18T18:51:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.349+0000 7fe11e7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe12015a380 0x7fe12015b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.349+0000 7fe11e7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe12015a380 0x7fe12015b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41844/0 (socket says 172.21.15.1:41844) 2024-03-18T18:51:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.349+0000 7fe11e7fc700 1 -- 172.21.15.1:0/3075200604 learned_addr learned my addr 172.21.15.1:0/3075200604 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.350+0000 7fe11f7fe700 1 --2- 172.21.15.1:0/3075200604 >> v2:172.21.15.165:3300/0 conn(0x7fe1200724e0 0x7fe12015baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.350+0000 7fe11e7fc700 1 -- 172.21.15.1:0/3075200604 >> v2:172.21.15.165:3300/0 conn(0x7fe1200724e0 msgr2=0x7fe12015baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.350+0000 7fe11e7fc700 1 --2- 172.21.15.1:0/3075200604 >> v2:172.21.15.165:3300/0 conn(0x7fe1200724e0 0x7fe12015baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.350+0000 7fe11e7fc700 1 -- 172.21.15.1:0/3075200604 >> v1:172.21.15.165:6789/0 conn(0x7fe120071490 legacy=0x7fe120071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.350+0000 7fe11e7fc700 1 -- 172.21.15.1:0/3075200604 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1200cabe0 con 0x7fe12015a380 2024-03-18T18:51:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.350+0000 7fe11e7fc700 1 --2- 172.21.15.1:0/3075200604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe12015a380 0x7fe12015b1b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe12015ed40 tx=0x7fe110009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8490ceecebc12962 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.350+0000 7fe11dffb700 1 -- 172.21.15.1:0/3075200604 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe110004230 con 0x7fe12015a380 2024-03-18T18:51:29.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.350+0000 7fe11dffb700 1 -- 172.21.15.1:0/3075200604 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe110004390 con 0x7fe12015a380 2024-03-18T18:51:29.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.350+0000 7fe11dffb700 1 -- 172.21.15.1:0/3075200604 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1100036a0 con 0x7fe12015a380 2024-03-18T18:51:29.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.350+0000 7fe1253e6700 1 -- 172.21.15.1:0/3075200604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe12015a380 msgr2=0x7fe12015b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:29.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.350+0000 7fe1253e6700 1 --2- 172.21.15.1:0/3075200604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe12015a380 0x7fe12015b1b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe12015ed40 tx=0x7fe110009bd0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.351+0000 7fe1253e6700 1 -- 172.21.15.1:0/3075200604 shutdown_connections 2024-03-18T18:51:29.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.351+0000 7fe1253e6700 1 --2- 172.21.15.1:0/3075200604 >> v2:172.21.15.165:3300/0 conn(0x7fe1200724e0 0x7fe12015baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.351+0000 7fe1253e6700 1 --2- 172.21.15.1:0/3075200604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe12015a380 0x7fe12015b1b0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.351+0000 7fe1253e6700 1 -- 172.21.15.1:0/3075200604 >> 172.21.15.1:0/3075200604 conn(0x7fe1200b1aa0 msgr2=0x7fe1200b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:29.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.351+0000 7fe1253e6700 1 -- 172.21.15.1:0/3075200604 shutdown_connections 2024-03-18T18:51:29.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.351+0000 7fe1253e6700 1 -- 172.21.15.1:0/3075200604 wait complete. 2024-03-18T18:51:29.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.352+0000 7fe1253e6700 1 Processor -- start 2024-03-18T18:51:29.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.352+0000 7fe1253e6700 1 -- start start 2024-03-18T18:51:29.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.353+0000 7fe1253e6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe120071490 0x7fe1200afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:29.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.353+0000 7fe1253e6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1200724e0 0x7fe1200b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.353+0000 7fe1253e6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe12015a380 0x7fe12006f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.353+0000 7fe1253e6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12006fe70 con 0x7fe1200724e0 2024-03-18T18:51:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.353+0000 7fe1253e6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200b4cb0 con 0x7fe12015a380 2024-03-18T18:51:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.353+0000 7fe1253e6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200b5680 con 0x7fe120071490 2024-03-18T18:51:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.353+0000 7fe11e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1200724e0 0x7fe1200b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.353+0000 7fe11e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1200724e0 0x7fe1200b5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41856/0 (socket says 172.21.15.1:41856) 2024-03-18T18:51:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.353+0000 7fe11e7fc700 1 -- 172.21.15.1:0/2383912224 learned_addr learned my addr 172.21.15.1:0/2383912224 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.353+0000 7fe11f7fe700 1 --2- 172.21.15.1:0/2383912224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe12015a380 0x7fe12006f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.353+0000 7fe11effd700 1 --2- 172.21.15.1:0/2383912224 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe120071490 0x7fe1200afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.353+0000 7fe11e7fc700 1 -- 172.21.15.1:0/2383912224 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe120071490 msgr2=0x7fe1200afc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.353+0000 7fe11e7fc700 1 --2- 172.21.15.1:0/2383912224 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe120071490 0x7fe1200afc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.353+0000 7fe11e7fc700 1 -- 172.21.15.1:0/2383912224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe12015a380 msgr2=0x7fe12006f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.353+0000 7fe11e7fc700 1 --2- 172.21.15.1:0/2383912224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe12015a380 0x7fe12006f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.353+0000 7fe11e7fc700 1 -- 172.21.15.1:0/2383912224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe110009550 con 0x7fe1200724e0 2024-03-18T18:51:29.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.353+0000 7fe11f7fe700 1 --2- 172.21.15.1:0/2383912224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe12015a380 0x7fe12006f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.353+0000 7fe11e7fc700 1 --2- 172.21.15.1:0/2383912224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1200724e0 0x7fe1200b5140 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe110009130 tx=0x7fe110004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.354+0000 7fe1077fe700 1 -- 172.21.15.1:0/2383912224 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe110011630 con 0x7fe1200724e0 2024-03-18T18:51:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.354+0000 7fe1077fe700 1 -- 172.21.15.1:0/2383912224 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe110011790 con 0x7fe1200724e0 2024-03-18T18:51:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.354+0000 7fe1253e6700 1 -- 172.21.15.1:0/2383912224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1200cbb20 con 0x7fe1200724e0 2024-03-18T18:51:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.354+0000 7fe1077fe700 1 -- 172.21.15.1:0/2383912224 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe110011b20 con 0x7fe1200724e0 2024-03-18T18:51:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.354+0000 7fe1253e6700 1 -- 172.21.15.1:0/2383912224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1200c9020 con 0x7fe1200724e0 2024-03-18T18:51:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.355+0000 7fe1253e6700 1 -- 172.21.15.1:0/2383912224 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1200cb980 con 0x7fe1200724e0 2024-03-18T18:51:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.355+0000 7fe1077fe700 1 -- 172.21.15.1:0/2383912224 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fe110011c80 con 0x7fe1200724e0 2024-03-18T18:51:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.355+0000 7fe1077fe700 1 --2- 172.21.15.1:0/2383912224 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe10c06ab60 0x7fe10c06d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.356+0000 7fe1077fe700 1 -- 172.21.15.1:0/2383912224 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fe11008a580 con 0x7fe1200724e0 2024-03-18T18:51:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.356+0000 7fe11effd700 1 --2- 172.21.15.1:0/2383912224 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe10c06ab60 0x7fe10c06d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:29.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.356+0000 7fe11effd700 1 --2- 172.21.15.1:0/2383912224 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe10c06ab60 0x7fe10c06d020 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe114004f30 tx=0x7fe114009040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:29.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.360+0000 7fe1077fe700 1 -- 172.21.15.1:0/2383912224 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe1200cb980 con 0x7fe1200724e0 2024-03-18T18:51:29.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.593+0000 7fe1253e6700 1 -- 172.21.15.1:0/2383912224 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir2", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1200cb1a0 con 0x7fe10c06ab60 2024-03-18T18:51:29.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.657+0000 7fe1077fe700 1 -- 172.21.15.1:0/2383912224 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fe114007e40 con 0x7fe10c06ab60 2024-03-18T18:51:29.653 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path /snap_test_dir1/dir2 2024-03-18T18:51:29.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.659+0000 7fe1253e6700 1 -- 172.21.15.1:0/2383912224 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe10c06ab60 msgr2=0x7fe10c06d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:29.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.659+0000 7fe1253e6700 1 --2- 172.21.15.1:0/2383912224 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe10c06ab60 0x7fe10c06d020 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe114004f30 tx=0x7fe114009040 comp rx=0 tx=0).stop 2024-03-18T18:51:29.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.659+0000 7fe1253e6700 1 -- 172.21.15.1:0/2383912224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1200724e0 msgr2=0x7fe1200b5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:29.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.659+0000 7fe1253e6700 1 --2- 172.21.15.1:0/2383912224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1200724e0 0x7fe1200b5140 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe110009130 tx=0x7fe110004750 comp rx=0 tx=0).stop 2024-03-18T18:51:29.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.659+0000 7fe1253e6700 1 -- 172.21.15.1:0/2383912224 shutdown_connections 2024-03-18T18:51:29.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.659+0000 7fe1253e6700 1 --2- 172.21.15.1:0/2383912224 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe10c06ab60 0x7fe10c06d020 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.659+0000 7fe1253e6700 1 --2- 172.21.15.1:0/2383912224 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe120071490 0x7fe1200afc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.659+0000 7fe1253e6700 1 --2- 172.21.15.1:0/2383912224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe12015a380 0x7fe12006f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.659+0000 7fe1253e6700 1 --2- 172.21.15.1:0/2383912224 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1200724e0 0x7fe1200b5140 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.659+0000 7fe1253e6700 1 -- 172.21.15.1:0/2383912224 >> 172.21.15.1:0/2383912224 conn(0x7fe1200b1aa0 msgr2=0x7fe1200b0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:29.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.659+0000 7fe1253e6700 1 -- 172.21.15.1:0/2383912224 shutdown_connections 2024-03-18T18:51:29.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.659+0000 7fe1253e6700 1 -- 172.21.15.1:0/2383912224 wait complete. 2024-03-18T18:51:29.667 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2024-03-18T18:51:29.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:29.667 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2024-03-18T18:51:29.688 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir3 1m 2024-03-18T18:51:29.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.859+0000 7f1335f1e700 1 Processor -- start 2024-03-18T18:51:29.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.859+0000 7f1335f1e700 1 -- start start 2024-03-18T18:51:29.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.860+0000 7f1335f1e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1330071cc0 0x7f13300ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:29.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.860+0000 7f1335f1e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f133006d560 0x7f133006fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:29.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.860+0000 7f1335f1e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1330070000 con 0x7f133015a240 2024-03-18T18:51:29.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.860+0000 7f1335f1e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13300adff0 con 0x7f133006d560 2024-03-18T18:51:29.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.860+0000 7f1335f1e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13300cac80 con 0x7f1330071cc0 2024-03-18T18:51:29.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.861+0000 7f132effd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1330071cc0 0x7f13300ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:29.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.861+0000 7f132ffff700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f133006d560 0x7f133006fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:29.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.861+0000 7f132effd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1330071cc0 0x7f13300ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:53016/0 (socket says 172.21.15.1:53016) 2024-03-18T18:51:29.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.861+0000 7f132effd700 1 -- 172.21.15.1:0/2747766760 learned_addr learned my addr 172.21.15.1:0/2747766760 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:29.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.861+0000 7f132f7fe700 1 --1- 172.21.15.1:0/2747766760 >> v1:172.21.15.165:6789/0 conn(0x7f133015a240 0x7f13300715b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:41968/0 (socket says 172.21.15.1:41968) 2024-03-18T18:51:29.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.861+0000 7f132ffff700 1 -- 172.21.15.1:0/2747766760 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1330071cc0 msgr2=0x7f13300ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:29.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.861+0000 7f132ffff700 1 --2- 172.21.15.1:0/2747766760 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1330071cc0 0x7f13300ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.861+0000 7f132ffff700 1 -- 172.21.15.1:0/2747766760 >> v1:172.21.15.165:6789/0 conn(0x7f133015a240 legacy=0x7f13300715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:29.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.861+0000 7f132ffff700 1 -- 172.21.15.1:0/2747766760 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13300caf60 con 0x7f133006d560 2024-03-18T18:51:29.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.861+0000 7f132effd700 1 --2- 172.21.15.1:0/2747766760 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1330071cc0 0x7f13300ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:51:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.861+0000 7f132ffff700 1 --2- 172.21.15.1:0/2747766760 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f133006d560 0x7f133006fac0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1328009d90 tx=0x7f132800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43fa9833e253a5b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.861+0000 7f132e7fc700 1 -- 172.21.15.1:0/2747766760 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132800e070 con 0x7f133006d560 2024-03-18T18:51:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.861+0000 7f132e7fc700 1 -- 172.21.15.1:0/2747766760 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132800bb20 con 0x7f133006d560 2024-03-18T18:51:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.861+0000 7f132e7fc700 1 -- 172.21.15.1:0/2747766760 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328012430 con 0x7f133006d560 2024-03-18T18:51:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.862+0000 7f1335f1e700 1 -- 172.21.15.1:0/2747766760 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f133006d560 msgr2=0x7f133006fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.862+0000 7f1335f1e700 1 --2- 172.21.15.1:0/2747766760 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f133006d560 0x7f133006fac0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1328009d90 tx=0x7f132800b330 comp rx=0 tx=0).stop 2024-03-18T18:51:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.862+0000 7f1335f1e700 1 -- 172.21.15.1:0/2747766760 shutdown_connections 2024-03-18T18:51:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.862+0000 7f1335f1e700 1 --2- 172.21.15.1:0/2747766760 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f133006d560 0x7f133006fac0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.862+0000 7f1335f1e700 1 --2- 172.21.15.1:0/2747766760 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1330071cc0 0x7f13300ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.862+0000 7f1335f1e700 1 -- 172.21.15.1:0/2747766760 >> 172.21.15.1:0/2747766760 conn(0x7f13300cff30 msgr2=0x7f13300d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.862+0000 7f1335f1e700 1 -- 172.21.15.1:0/2747766760 shutdown_connections 2024-03-18T18:51:29.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.862+0000 7f1335f1e700 1 -- 172.21.15.1:0/2747766760 wait complete. 2024-03-18T18:51:29.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.863+0000 7f1335f1e700 1 Processor -- start 2024-03-18T18:51:29.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.863+0000 7f1335f1e700 1 -- start start 2024-03-18T18:51:29.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.864+0000 7f1335f1e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f133006d560 0x7f13301e4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:29.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.864+0000 7f1335f1e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1330071cc0 0x7f133006a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:29.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.864+0000 7f1335f1e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f133015a240 0x7f133006ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:29.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.864+0000 7f1335f1e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1330159130 con 0x7f1330071cc0 2024-03-18T18:51:29.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.864+0000 7f1335f1e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133014c2c0 con 0x7f133006d560 2024-03-18T18:51:29.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.864+0000 7f1335f1e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13300b53a0 con 0x7f133015a240 2024-03-18T18:51:29.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.864+0000 7f132effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1330071cc0 0x7f133006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:29.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.864+0000 7f132effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1330071cc0 0x7f133006a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41882/0 (socket says 172.21.15.1:41882) 2024-03-18T18:51:29.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.864+0000 7f132effd700 1 -- 172.21.15.1:0/3163180291 learned_addr learned my addr 172.21.15.1:0/3163180291 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.864+0000 7f132f7fe700 1 --2- 172.21.15.1:0/3163180291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f133006d560 0x7f13301e4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.864+0000 7f132ffff700 1 --2- 172.21.15.1:0/3163180291 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f133015a240 0x7f133006ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.865+0000 7f132effd700 1 -- 172.21.15.1:0/3163180291 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f133015a240 msgr2=0x7f133006ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.865+0000 7f132effd700 1 --2- 172.21.15.1:0/3163180291 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f133015a240 0x7f133006ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.865+0000 7f132effd700 1 -- 172.21.15.1:0/3163180291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f133006d560 msgr2=0x7f13301e4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.865+0000 7f132effd700 1 --2- 172.21.15.1:0/3163180291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f133006d560 0x7f13301e4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.865+0000 7f132effd700 1 -- 172.21.15.1:0/3163180291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1318009550 con 0x7f1330071cc0 2024-03-18T18:51:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.865+0000 7f132effd700 1 --2- 172.21.15.1:0/3163180291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1330071cc0 0x7f133006a750 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f131800ebe0 tx=0x7f131800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.865+0000 7f1327fff700 1 -- 172.21.15.1:0/3163180291 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131800c850 con 0x7f1330071cc0 2024-03-18T18:51:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.865+0000 7f1327fff700 1 -- 172.21.15.1:0/3163180291 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131800c9b0 con 0x7f1330071cc0 2024-03-18T18:51:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.865+0000 7f1327fff700 1 -- 172.21.15.1:0/3163180291 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131800cd40 con 0x7f1330071cc0 2024-03-18T18:51:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.865+0000 7f1335f1e700 1 -- 172.21.15.1:0/3163180291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1328009a70 con 0x7f1330071cc0 2024-03-18T18:51:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.865+0000 7f1335f1e700 1 -- 172.21.15.1:0/3163180291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13300b5960 con 0x7f1330071cc0 2024-03-18T18:51:29.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.866+0000 7f1335f1e700 1 -- 172.21.15.1:0/3163180291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13300676a0 con 0x7f1330071cc0 2024-03-18T18:51:29.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.867+0000 7f1327fff700 1 -- 172.21.15.1:0/3163180291 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f1318013470 con 0x7f1330071cc0 2024-03-18T18:51:29.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.867+0000 7f1327fff700 1 --2- 172.21.15.1:0/3163180291 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f131006ab30 0x7f131006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:29.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.867+0000 7f132f7fe700 1 --2- 172.21.15.1:0/3163180291 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f131006ab30 0x7f131006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:29.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.867+0000 7f1327fff700 1 -- 172.21.15.1:0/3163180291 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f13180892d0 con 0x7f1330071cc0 2024-03-18T18:51:29.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.868+0000 7f132f7fe700 1 --2- 172.21.15.1:0/3163180291 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f131006ab30 0x7f131006cff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1320000f60 tx=0x7f1320007ed0 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:29.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:29.872+0000 7f1327fff700 1 -- 172.21.15.1:0/3163180291 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f13180545b0 con 0x7f1330071cc0 2024-03-18T18:51:30.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.099+0000 7f1335f1e700 1 -- 172.21.15.1:0/3163180291 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir3", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f133015a620 con 0x7f131006ab30 2024-03-18T18:51:30.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.153+0000 7f1327fff700 1 -- 172.21.15.1:0/3163180291 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f1320004af0 con 0x7f131006ab30 2024-03-18T18:51:30.149 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path /snap_test_dir1/dir3 2024-03-18T18:51:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.155+0000 7f1335f1e700 1 -- 172.21.15.1:0/3163180291 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f131006ab30 msgr2=0x7f131006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.155+0000 7f1335f1e700 1 --2- 172.21.15.1:0/3163180291 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f131006ab30 0x7f131006cff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1320000f60 tx=0x7f1320007ed0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.155+0000 7f1335f1e700 1 -- 172.21.15.1:0/3163180291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1330071cc0 msgr2=0x7f133006a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.155+0000 7f1335f1e700 1 --2- 172.21.15.1:0/3163180291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1330071cc0 0x7f133006a750 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f131800ebe0 tx=0x7f131800eef0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.155+0000 7f1335f1e700 1 -- 172.21.15.1:0/3163180291 shutdown_connections 2024-03-18T18:51:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.155+0000 7f1335f1e700 1 --2- 172.21.15.1:0/3163180291 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f131006ab30 0x7f131006cff0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.155+0000 7f1335f1e700 1 --2- 172.21.15.1:0/3163180291 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f133015a240 0x7f133006ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.155+0000 7f1335f1e700 1 --2- 172.21.15.1:0/3163180291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f133006d560 0x7f13301e4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.155+0000 7f1335f1e700 1 --2- 172.21.15.1:0/3163180291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1330071cc0 0x7f133006a750 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.155+0000 7f1335f1e700 1 -- 172.21.15.1:0/3163180291 >> 172.21.15.1:0/3163180291 conn(0x7f13300cff30 msgr2=0x7f13300afb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:30.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.155+0000 7f1335f1e700 1 -- 172.21.15.1:0/3163180291 shutdown_connections 2024-03-18T18:51:30.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.155+0000 7f1335f1e700 1 -- 172.21.15.1:0/3163180291 wait complete. 2024-03-18T18:51:30.164 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2024-03-18T18:51:30.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:30.164 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2024-03-18T18:51:30.184 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir4 1m 2024-03-18T18:51:30.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.355+0000 7ffa334d5700 1 Processor -- start 2024-03-18T18:51:30.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.356+0000 7ffa334d5700 1 -- start start 2024-03-18T18:51:30.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa334d5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa2c0cc0f0 0x7ffa2c0add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:30.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa334d5700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2c0ce8f0 con 0x7ffa2c071d30 2024-03-18T18:51:30.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa334d5700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2c0ae260 con 0x7ffa2c15a1d0 2024-03-18T18:51:30.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa334d5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c0cdb00 con 0x7ffa2c0cc0f0 2024-03-18T18:51:30.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa30a70700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ffa2c071d30 0x7ffa2c0cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52912/0 (socket says 172.21.15.1:52912) 2024-03-18T18:51:30.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa30a70700 1 -- 172.21.15.1:0/1097001229 learned_addr learned my addr 172.21.15.1:0/1097001229 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:30.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa31a72700 1 --2- 172.21.15.1:0/1097001229 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa2c0cc0f0 0x7ffa2c0add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:30.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa1ffff700 1 -- 172.21.15.1:0/1097001229 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184217764 0 0) 0x7ffa2c0ce8f0 con 0x7ffa2c071d30 2024-03-18T18:51:30.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa1ffff700 1 -- 172.21.15.1:0/1097001229 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa180035b0 con 0x7ffa2c071d30 2024-03-18T18:51:30.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa1ffff700 1 -- 172.21.15.1:0/1097001229 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4198946850 0 0) 0x7ffa180035b0 con 0x7ffa2c071d30 2024-03-18T18:51:30.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa1ffff700 1 -- 172.21.15.1:0/1097001229 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa2c0cc0f0 msgr2=0x7ffa2c0add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:30.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa1ffff700 1 --2- 172.21.15.1:0/1097001229 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa2c0cc0f0 0x7ffa2c0add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa1ffff700 1 -- 172.21.15.1:0/1097001229 >> v1:172.21.15.165:6789/0 conn(0x7ffa2c15a1d0 legacy=0x7ffa2c071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:30.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa1ffff700 1 -- 172.21.15.1:0/1097001229 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c0cdce0 con 0x7ffa2c071d30 2024-03-18T18:51:30.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa1ffff700 1 -- 172.21.15.1:0/1097001229 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7ffa28001670 con 0x7ffa2c071d30 2024-03-18T18:51:30.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa1ffff700 1 -- 172.21.15.1:0/1097001229 >> v1:172.21.15.1:6789/0 conn(0x7ffa2c071d30 legacy=0x7ffa2c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:30.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa1ffff700 1 --2- 172.21.15.1:0/1097001229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa18003a30 0x7ffa18003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:30.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa1ffff700 1 --2- 172.21.15.1:0/1097001229 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa2c0cc0f0 0x7ffa18004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:30.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa1ffff700 1 --2- 172.21.15.1:0/1097001229 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa18004d90 0x7ffa18005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:30.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa1ffff700 1 -- 172.21.15.1:0/1097001229 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c0cdb00 con 0x7ffa18003a30 2024-03-18T18:51:30.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa1ffff700 1 -- 172.21.15.1:0/1097001229 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1800bac0 con 0x7ffa18004d90 2024-03-18T18:51:30.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.357+0000 7ffa1ffff700 1 -- 172.21.15.1:0/1097001229 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1800bc80 con 0x7ffa2c0cc0f0 2024-03-18T18:51:30.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.358+0000 7ffa31271700 1 --2- 172.21.15.1:0/1097001229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa18003a30 0x7ffa18003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:30.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.358+0000 7ffa30a70700 1 --2- 172.21.15.1:0/1097001229 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa2c0cc0f0 0x7ffa18004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:30.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.358+0000 7ffa31a72700 1 --2- 172.21.15.1:0/1097001229 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa18004d90 0x7ffa18005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:30.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.358+0000 7ffa31271700 1 -- 172.21.15.1:0/1097001229 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa2c0cc0f0 msgr2=0x7ffa18004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:30.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.358+0000 7ffa31271700 1 --2- 172.21.15.1:0/1097001229 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa2c0cc0f0 0x7ffa18004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.358+0000 7ffa31271700 1 -- 172.21.15.1:0/1097001229 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa18004d90 msgr2=0x7ffa18005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:30.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.358+0000 7ffa31271700 1 --2- 172.21.15.1:0/1097001229 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa18004d90 0x7ffa18005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.358+0000 7ffa31271700 1 -- 172.21.15.1:0/1097001229 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa180035b0 con 0x7ffa18003a30 2024-03-18T18:51:30.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.358+0000 7ffa31a72700 1 --2- 172.21.15.1:0/1097001229 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa18004d90 0x7ffa18005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:30.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.358+0000 7ffa31271700 1 --2- 172.21.15.1:0/1097001229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa18003a30 0x7ffa18003e50 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ffa20001830 tx=0x7ffa18005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1655c00f5611cb8b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:30.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.358+0000 7ffa1ffff700 1 -- 172.21.15.1:0/1097001229 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa20001320 con 0x7ffa18003a30 2024-03-18T18:51:30.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.358+0000 7ffa1ffff700 1 -- 172.21.15.1:0/1097001229 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa200052f0 con 0x7ffa18003a30 2024-03-18T18:51:30.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.358+0000 7ffa1ffff700 1 -- 172.21.15.1:0/1097001229 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa200068b0 con 0x7ffa18003a30 2024-03-18T18:51:30.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.359+0000 7ffa334d5700 1 -- 172.21.15.1:0/1097001229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa18003a30 msgr2=0x7ffa18003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:30.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.359+0000 7ffa334d5700 1 --2- 172.21.15.1:0/1097001229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa18003a30 0x7ffa18003e50 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ffa20001830 tx=0x7ffa18005880 comp rx=0 tx=0).stop 2024-03-18T18:51:30.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.359+0000 7ffa31a72700 1 -- 172.21.15.1:0/1097001229 reap_dead start 2024-03-18T18:51:30.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.359+0000 7ffa334d5700 1 -- 172.21.15.1:0/1097001229 shutdown_connections 2024-03-18T18:51:30.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.359+0000 7ffa334d5700 1 -- 172.21.15.1:0/1097001229 >> 172.21.15.1:0/1097001229 conn(0x7ffa2c0cc9b0 msgr2=0x7ffa2c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:30.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.359+0000 7ffa334d5700 1 -- 172.21.15.1:0/1097001229 shutdown_connections 2024-03-18T18:51:30.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.359+0000 7ffa334d5700 1 -- 172.21.15.1:0/1097001229 wait complete. 2024-03-18T18:51:30.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.360+0000 7ffa334d5700 1 Processor -- start 2024-03-18T18:51:30.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.360+0000 7ffa334d5700 1 -- start start 2024-03-18T18:51:30.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa334d5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa18003a30 0x7ffa2c1534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:30.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa334d5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa2c15c7d0 0x7ffa2c15ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:30.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa334d5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa2c152b10 0x7ffa2c0ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:30.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa334d5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c15cc50 con 0x7ffa2c152b10 2024-03-18T18:51:30.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa334d5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c15ceb0 con 0x7ffa2c15c7d0 2024-03-18T18:51:30.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa334d5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c15d190 con 0x7ffa18003a30 2024-03-18T18:51:30.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa31a72700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa2c152b10 0x7ffa2c0ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:30.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa31a72700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa2c152b10 0x7ffa2c0ca560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41894/0 (socket says 172.21.15.1:41894) 2024-03-18T18:51:30.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa31a72700 1 -- 172.21.15.1:0/3287452171 learned_addr learned my addr 172.21.15.1:0/3287452171 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:30.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa30a70700 1 --2- 172.21.15.1:0/3287452171 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa2c15c7d0 0x7ffa2c15ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:30.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa31271700 1 --2- 172.21.15.1:0/3287452171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa18003a30 0x7ffa2c1534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:30.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa31a72700 1 -- 172.21.15.1:0/3287452171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa18003a30 msgr2=0x7ffa2c1534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:30.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa31a72700 1 --2- 172.21.15.1:0/3287452171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa18003a30 0x7ffa2c1534b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa31a72700 1 -- 172.21.15.1:0/3287452171 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa2c15c7d0 msgr2=0x7ffa2c15ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:30.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa31a72700 1 --2- 172.21.15.1:0/3287452171 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa2c15c7d0 0x7ffa2c15ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa31a72700 1 -- 172.21.15.1:0/3287452171 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c15d470 con 0x7ffa2c152b10 2024-03-18T18:51:30.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa31271700 1 --2- 172.21.15.1:0/3287452171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa18003a30 0x7ffa2c1534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:30.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa31a72700 1 --2- 172.21.15.1:0/3287452171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa2c152b10 0x7ffa2c0ca560 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c09df60 tx=0x7ffa240104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:30.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa1dffb700 1 -- 172.21.15.1:0/3287452171 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2400e050 con 0x7ffa2c152b10 2024-03-18T18:51:30.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa1dffb700 1 -- 172.21.15.1:0/3287452171 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2400f040 con 0x7ffa2c152b10 2024-03-18T18:51:30.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa1dffb700 1 -- 172.21.15.1:0/3287452171 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24010e50 con 0x7ffa2c152b10 2024-03-18T18:51:30.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.361+0000 7ffa334d5700 1 -- 172.21.15.1:0/3287452171 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2c15a650 con 0x7ffa2c152b10 2024-03-18T18:51:30.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.362+0000 7ffa334d5700 1 -- 172.21.15.1:0/3287452171 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2c153c30 con 0x7ffa2c152b10 2024-03-18T18:51:30.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.363+0000 7ffa334d5700 1 -- 172.21.15.1:0/3287452171 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2c15ab70 con 0x7ffa2c152b10 2024-03-18T18:51:30.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.363+0000 7ffa1dffb700 1 -- 172.21.15.1:0/3287452171 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 88992+0+0 (secure 0 0 0) 0x7ffa24003a20 con 0x7ffa2c152b10 2024-03-18T18:51:30.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.363+0000 7ffa1dffb700 1 --2- 172.21.15.1:0/3287452171 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa1006ab30 0x7ffa1006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:30.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.364+0000 7ffa1dffb700 1 -- 172.21.15.1:0/3287452171 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6287+0+0 (secure 0 0 0) 0x7ffa240895f0 con 0x7ffa2c152b10 2024-03-18T18:51:30.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.364+0000 7ffa31271700 1 --2- 172.21.15.1:0/3287452171 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa1006ab30 0x7ffa1006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:30.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.364+0000 7ffa31271700 1 --2- 172.21.15.1:0/3287452171 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa1006ab30 0x7ffa1006cff0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffa20000ef0 tx=0x7ffa20009030 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:30.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.368+0000 7ffa1dffb700 1 -- 172.21.15.1:0/3287452171 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffa2c15ab70 con 0x7ffa2c152b10 2024-03-18T18:51:30.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.552+0000 7ffa1dffb700 1 -- 172.21.15.1:0/3287452171 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 9) v1 ==== 89257+0+0 (secure 0 0 0) 0x7ffa24003cd0 con 0x7ffa2c152b10 2024-03-18T18:51:30.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.604+0000 7ffa334d5700 1 -- 172.21.15.1:0/3287452171 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir4", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa2c0ce580 con 0x7ffa1006ab30 2024-03-18T18:51:30.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.673+0000 7ffa1dffb700 1 -- 172.21.15.1:0/3287452171 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7ffa200054e0 con 0x7ffa1006ab30 2024-03-18T18:51:30.669 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path /snap_test_dir1/dir4 2024-03-18T18:51:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.675+0000 7ffa334d5700 1 -- 172.21.15.1:0/3287452171 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa1006ab30 msgr2=0x7ffa1006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.675+0000 7ffa334d5700 1 --2- 172.21.15.1:0/3287452171 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa1006ab30 0x7ffa1006cff0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffa20000ef0 tx=0x7ffa20009030 comp rx=0 tx=0).stop 2024-03-18T18:51:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.675+0000 7ffa334d5700 1 -- 172.21.15.1:0/3287452171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa2c152b10 msgr2=0x7ffa2c0ca560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.675+0000 7ffa334d5700 1 --2- 172.21.15.1:0/3287452171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa2c152b10 0x7ffa2c0ca560 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c09df60 tx=0x7ffa240104a0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.675+0000 7ffa334d5700 1 -- 172.21.15.1:0/3287452171 shutdown_connections 2024-03-18T18:51:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.675+0000 7ffa334d5700 1 --2- 172.21.15.1:0/3287452171 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa1006ab30 0x7ffa1006cff0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.675+0000 7ffa334d5700 1 --2- 172.21.15.1:0/3287452171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa18003a30 0x7ffa2c1534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.675+0000 7ffa334d5700 1 --2- 172.21.15.1:0/3287452171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa2c152b10 0x7ffa2c0ca560 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.675+0000 7ffa334d5700 1 --2- 172.21.15.1:0/3287452171 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa2c15c7d0 0x7ffa2c15ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.675+0000 7ffa334d5700 1 -- 172.21.15.1:0/3287452171 >> 172.21.15.1:0/3287452171 conn(0x7ffa2c0cc9b0 msgr2=0x7ffa2c06f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.675+0000 7ffa334d5700 1 -- 172.21.15.1:0/3287452171 shutdown_connections 2024-03-18T18:51:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.675+0000 7ffa334d5700 1 -- 172.21.15.1:0/3287452171 wait complete. 2024-03-18T18:51:30.683 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2024-03-18T18:51:30.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:30.684 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2024-03-18T18:51:30.704 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir5 1m 2024-03-18T18:51:30.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.876+0000 7f732b393700 1 Processor -- start 2024-03-18T18:51:30.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.876+0000 7f732b393700 1 -- start start 2024-03-18T18:51:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.877+0000 7f732b393700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f732415a240 0x7f73240ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:30.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.877+0000 7f732b393700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7324071cc0 0x7f732406fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.877+0000 7f732b393700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7324070000 con 0x7f732406d560 2024-03-18T18:51:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.877+0000 7f732b393700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73240adff0 con 0x7f732415a240 2024-03-18T18:51:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.877+0000 7f732b393700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73240cac80 con 0x7f7324071cc0 2024-03-18T18:51:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.877+0000 7f732892e700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f732406d560 0x7f73240715b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52916/0 (socket says 172.21.15.1:52916) 2024-03-18T18:51:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.877+0000 7f732892e700 1 -- 172.21.15.1:0/1994871950 learned_addr learned my addr 172.21.15.1:0/1994871950 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.877+0000 7f732912f700 1 --2- 172.21.15.1:0/1994871950 >> v2:172.21.15.165:3300/0 conn(0x7f732415a240 0x7f73240ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.877+0000 7f7329930700 1 --2- 172.21.15.1:0/1994871950 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7324071cc0 0x7f732406fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.877+0000 7f731bfff700 1 -- 172.21.15.1:0/1994871950 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1007734562 0 0) 0x7f7324070000 con 0x7f732406d560 2024-03-18T18:51:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.877+0000 7f731bfff700 1 -- 172.21.15.1:0/1994871950 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73100057c0 con 0x7f732406d560 2024-03-18T18:51:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.877+0000 7f731bfff700 1 -- 172.21.15.1:0/1994871950 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3057620811 0 0) 0x7f73100057c0 con 0x7f732406d560 2024-03-18T18:51:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.877+0000 7f731bfff700 1 -- 172.21.15.1:0/1994871950 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7324071cc0 msgr2=0x7f732406fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.877+0000 7f731bfff700 1 --2- 172.21.15.1:0/1994871950 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7324071cc0 0x7f732406fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.877+0000 7f731bfff700 1 -- 172.21.15.1:0/1994871950 >> v2:172.21.15.165:3300/0 conn(0x7f732415a240 msgr2=0x7f73240ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:30.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.877+0000 7f731bfff700 1 --2- 172.21.15.1:0/1994871950 >> v2:172.21.15.165:3300/0 conn(0x7f732415a240 0x7f73240ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f731bfff700 1 -- 172.21.15.1:0/1994871950 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73240caf60 con 0x7f732406d560 2024-03-18T18:51:30.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f731bfff700 1 -- 172.21.15.1:0/1994871950 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f7314001670 con 0x7f732406d560 2024-03-18T18:51:30.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f731bfff700 1 -- 172.21.15.1:0/1994871950 >> v1:172.21.15.1:6789/0 conn(0x7f732406d560 legacy=0x7f73240715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:30.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f731bfff700 1 --2- 172.21.15.1:0/1994871950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7310005c10 0x7f7310006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:30.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f731bfff700 1 --2- 172.21.15.1:0/1994871950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7310006a60 0x7f7310006ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:30.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f731bfff700 1 --2- 172.21.15.1:0/1994871950 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7324071cc0 0x7f7310007420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:30.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f731bfff700 1 -- 172.21.15.1:0/1994871950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73240adff0 con 0x7f7310006a60 2024-03-18T18:51:30.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f731bfff700 1 -- 172.21.15.1:0/1994871950 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73240cac80 con 0x7f7310005c10 2024-03-18T18:51:30.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f731bfff700 1 -- 172.21.15.1:0/1994871950 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731000dc20 con 0x7f7324071cc0 2024-03-18T18:51:30.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f732892e700 1 --2- 172.21.15.1:0/1994871950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7310006a60 0x7f7310006ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:30.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f732912f700 1 --2- 172.21.15.1:0/1994871950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7310005c10 0x7f7310006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:30.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f7329930700 1 --2- 172.21.15.1:0/1994871950 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7324071cc0 0x7f7310007420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:30.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f732892e700 1 -- 172.21.15.1:0/1994871950 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7324071cc0 msgr2=0x7f7310007420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:30.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f732892e700 1 --2- 172.21.15.1:0/1994871950 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7324071cc0 0x7f7310007420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f732892e700 1 -- 172.21.15.1:0/1994871950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7310005c10 msgr2=0x7f7310006030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:30.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f732892e700 1 --2- 172.21.15.1:0/1994871950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7310005c10 0x7f7310006030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f732892e700 1 -- 172.21.15.1:0/1994871950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73100057c0 con 0x7f7310006a60 2024-03-18T18:51:30.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f732892e700 1 --2- 172.21.15.1:0/1994871950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7310006a60 0x7f7310006ee0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f731400c210 tx=0x7f7310007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cefcfb5749d407b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:30.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f731bfff700 1 -- 172.21.15.1:0/1994871950 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314007c10 con 0x7f7310006a60 2024-03-18T18:51:30.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f731bfff700 1 -- 172.21.15.1:0/1994871950 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f731400cdb0 con 0x7f7310006a60 2024-03-18T18:51:30.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.878+0000 7f731bfff700 1 -- 172.21.15.1:0/1994871950 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314005ba0 con 0x7f7310006a60 2024-03-18T18:51:30.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.879+0000 7f732b393700 1 -- 172.21.15.1:0/1994871950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7310006a60 msgr2=0x7f7310006ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:30.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.879+0000 7f732b393700 1 --2- 172.21.15.1:0/1994871950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7310006a60 0x7f7310006ee0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f731400c210 tx=0x7f7310007a90 comp rx=0 tx=0).stop 2024-03-18T18:51:30.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.879+0000 7f7329930700 1 -- 172.21.15.1:0/1994871950 reap_dead start 2024-03-18T18:51:30.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.879+0000 7f732b393700 1 -- 172.21.15.1:0/1994871950 shutdown_connections 2024-03-18T18:51:30.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.879+0000 7f732b393700 1 -- 172.21.15.1:0/1994871950 >> 172.21.15.1:0/1994871950 conn(0x7f73240cff30 msgr2=0x7f73240d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:30.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.879+0000 7f732b393700 1 -- 172.21.15.1:0/1994871950 shutdown_connections 2024-03-18T18:51:30.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.879+0000 7f732b393700 1 -- 172.21.15.1:0/1994871950 wait complete. 2024-03-18T18:51:30.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.880+0000 7f732b393700 1 Processor -- start 2024-03-18T18:51:30.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.880+0000 7f732b393700 1 -- start start 2024-03-18T18:51:30.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.881+0000 7f732b393700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7310006a60 0x7f73241ef360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:30.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.881+0000 7f732b393700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73240c5750 0x7f73240c5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:30.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.881+0000 7f732b393700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73241499a0 0x7f7324149e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:30.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.881+0000 7f732b393700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73240c64b0 con 0x7f73240c5750 2024-03-18T18:51:30.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.881+0000 7f732b393700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73240c6790 con 0x7f73241499a0 2024-03-18T18:51:30.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.881+0000 7f732b393700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7324148ec0 con 0x7f7310006a60 2024-03-18T18:51:30.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.881+0000 7f732892e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73240c5750 0x7f73240c5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:30.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.881+0000 7f732892e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73240c5750 0x7f73240c5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41906/0 (socket says 172.21.15.1:41906) 2024-03-18T18:51:30.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.881+0000 7f732892e700 1 -- 172.21.15.1:0/2541556315 learned_addr learned my addr 172.21.15.1:0/2541556315 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:30.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.881+0000 7f732912f700 1 --2- 172.21.15.1:0/2541556315 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7310006a60 0x7f73241ef360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:30.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.881+0000 7f7329930700 1 --2- 172.21.15.1:0/2541556315 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73241499a0 0x7f7324149e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:30.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.881+0000 7f732892e700 1 -- 172.21.15.1:0/2541556315 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7310006a60 msgr2=0x7f73241ef360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:30.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.881+0000 7f732892e700 1 --2- 172.21.15.1:0/2541556315 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7310006a60 0x7f73241ef360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.881+0000 7f732892e700 1 -- 172.21.15.1:0/2541556315 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73241499a0 msgr2=0x7f7324149e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:30.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.881+0000 7f732892e700 1 --2- 172.21.15.1:0/2541556315 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73241499a0 0x7f7324149e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:30.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.881+0000 7f732892e700 1 -- 172.21.15.1:0/2541556315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73241491a0 con 0x7f73240c5750 2024-03-18T18:51:30.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.881+0000 7f7329930700 1 --2- 172.21.15.1:0/2541556315 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73241499a0 0x7f7324149e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:30.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.882+0000 7f732892e700 1 --2- 172.21.15.1:0/2541556315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73240c5750 0x7f73240c5bd0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f731400c800 tx=0x7f7314001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:30.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.882+0000 7f7319ffb700 1 -- 172.21.15.1:0/2541556315 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314001640 con 0x7f73240c5750 2024-03-18T18:51:30.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.882+0000 7f7319ffb700 1 -- 172.21.15.1:0/2541556315 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73140070d0 con 0x7f73240c5750 2024-03-18T18:51:30.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.882+0000 7f732b393700 1 -- 172.21.15.1:0/2541556315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7324149480 con 0x7f73240c5750 2024-03-18T18:51:30.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.882+0000 7f7319ffb700 1 -- 172.21.15.1:0/2541556315 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314007480 con 0x7f73240c5750 2024-03-18T18:51:30.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.882+0000 7f732b393700 1 -- 172.21.15.1:0/2541556315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f732414a360 con 0x7f73240c5750 2024-03-18T18:51:30.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.883+0000 7f732b393700 1 -- 172.21.15.1:0/2541556315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73240c6900 con 0x7f73240c5750 2024-03-18T18:51:30.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.883+0000 7f7319ffb700 1 -- 172.21.15.1:0/2541556315 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 9) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f7314019030 con 0x7f73240c5750 2024-03-18T18:51:30.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.884+0000 7f7319ffb700 1 --2- 172.21.15.1:0/2541556315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f730806ae30 0x7f730806d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:30.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.884+0000 7f7319ffb700 1 -- 172.21.15.1:0/2541556315 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f731408a9d0 con 0x7f73240c5750 2024-03-18T18:51:30.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.884+0000 7f732912f700 1 --2- 172.21.15.1:0/2541556315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f730806ae30 0x7f730806d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:30.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.884+0000 7f732912f700 1 --2- 172.21.15.1:0/2541556315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f730806ae30 0x7f730806d2f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7320009c60 tx=0x7f7320009210 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:30.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:30.888+0000 7f7319ffb700 1 -- 172.21.15.1:0/2541556315 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7314055870 con 0x7f73240c5750 2024-03-18T18:51:31.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.116+0000 7f732b393700 1 -- 172.21.15.1:0/2541556315 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir5", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7324149030 con 0x7f730806ae30 2024-03-18T18:51:31.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.174+0000 7f7319ffb700 1 -- 172.21.15.1:0/2541556315 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f7324149030 con 0x7f730806ae30 2024-03-18T18:51:31.171 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path /snap_test_dir1/dir5 2024-03-18T18:51:31.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.177+0000 7f732b393700 1 -- 172.21.15.1:0/2541556315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f730806ae30 msgr2=0x7f730806d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:31.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.177+0000 7f732b393700 1 --2- 172.21.15.1:0/2541556315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f730806ae30 0x7f730806d2f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7320009c60 tx=0x7f7320009210 comp rx=0 tx=0).stop 2024-03-18T18:51:31.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.177+0000 7f732b393700 1 -- 172.21.15.1:0/2541556315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73240c5750 msgr2=0x7f73240c5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:31.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.177+0000 7f732b393700 1 --2- 172.21.15.1:0/2541556315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73240c5750 0x7f73240c5bd0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f731400c800 tx=0x7f7314001a00 comp rx=0 tx=0).stop 2024-03-18T18:51:31.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.177+0000 7f732b393700 1 -- 172.21.15.1:0/2541556315 shutdown_connections 2024-03-18T18:51:31.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.177+0000 7f732b393700 1 --2- 172.21.15.1:0/2541556315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f730806ae30 0x7f730806d2f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:31.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.177+0000 7f732b393700 1 --2- 172.21.15.1:0/2541556315 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7310006a60 0x7f73241ef360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:31.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.177+0000 7f732b393700 1 --2- 172.21.15.1:0/2541556315 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73241499a0 0x7f7324149e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:31.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.177+0000 7f732b393700 1 --2- 172.21.15.1:0/2541556315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73240c5750 0x7f73240c5bd0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:31.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.177+0000 7f732b393700 1 -- 172.21.15.1:0/2541556315 >> 172.21.15.1:0/2541556315 conn(0x7f73240cff30 msgr2=0x7f73240b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:31.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.177+0000 7f732b393700 1 -- 172.21.15.1:0/2541556315 shutdown_connections 2024-03-18T18:51:31.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.177+0000 7f732b393700 1 -- 172.21.15.1:0/2541556315 wait complete. 2024-03-18T18:51:31.185 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2024-03-18T18:51:31.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:31.185 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2024-03-18T18:51:31.205 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir6 1m 2024-03-18T18:51:31.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.377+0000 7f158bfc6700 1 Processor -- start 2024-03-18T18:51:31.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.377+0000 7f158bfc6700 1 -- start start 2024-03-18T18:51:31.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.378+0000 7f158bfc6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f158415d5a0 0x7f15840ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:31.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.378+0000 7f158bfc6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f15840ce550 0x7f158406a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:31.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.378+0000 7f158bfc6700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15840cf600 con 0x7f15840cfa90 2024-03-18T18:51:31.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.378+0000 7f158bfc6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15840ce9f0 con 0x7f15840ce550 2024-03-18T18:51:31.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.378+0000 7f158bfc6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15840cecd0 con 0x7f158415d5a0 2024-03-18T18:51:31.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.378+0000 7f1589561700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f15840ce550 0x7f158406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:31.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.378+0000 7f158a563700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f15840cfa90 0x7f1584071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52924/0 (socket says 172.21.15.1:52924) 2024-03-18T18:51:31.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.378+0000 7f1589561700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f15840ce550 0x7f158406a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41910/0 (socket says 172.21.15.1:41910) 2024-03-18T18:51:31.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.378+0000 7f158a563700 1 -- 172.21.15.1:0/3917111422 learned_addr learned my addr 172.21.15.1:0/3917111422 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:31.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.378+0000 7f1589d62700 1 --2- 172.21.15.1:0/3917111422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f158415d5a0 0x7f15840ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:31.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.378+0000 7f1589561700 1 -- 172.21.15.1:0/3917111422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f158415d5a0 msgr2=0x7f15840ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:31.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.378+0000 7f1589561700 1 --2- 172.21.15.1:0/3917111422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f158415d5a0 0x7f15840ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:31.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.378+0000 7f1589561700 1 -- 172.21.15.1:0/3917111422 >> v1:172.21.15.1:6789/0 conn(0x7f15840cfa90 legacy=0x7f1584071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:31.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.378+0000 7f1589561700 1 -- 172.21.15.1:0/3917111422 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15840cefb0 con 0x7f15840ce550 2024-03-18T18:51:31.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.378+0000 7f1588d60700 1 -- 172.21.15.1:0/3917111422 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635260808 0 0) 0x7f15840cf600 con 0x7f15840cfa90 2024-03-18T18:51:31.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.378+0000 7f1589561700 1 --2- 172.21.15.1:0/3917111422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15840ce550 0x7f158406a750 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1574009840 tx=0x7f1574009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2dd87a090731a8e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:31.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.378+0000 7f1588d60700 1 -- 172.21.15.1:0/3917111422 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574011830 con 0x7f15840ce550 2024-03-18T18:51:31.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.379+0000 7f1588d60700 1 -- 172.21.15.1:0/3917111422 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1574011990 con 0x7f15840ce550 2024-03-18T18:51:31.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.379+0000 7f1588d60700 1 -- 172.21.15.1:0/3917111422 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574011d20 con 0x7f15840ce550 2024-03-18T18:51:31.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.379+0000 7f158bfc6700 1 -- 172.21.15.1:0/3917111422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15840ce550 msgr2=0x7f158406a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:31.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.379+0000 7f158bfc6700 1 --2- 172.21.15.1:0/3917111422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15840ce550 0x7f158406a750 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1574009840 tx=0x7f1574009b50 comp rx=0 tx=0).stop 2024-03-18T18:51:31.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.379+0000 7f158bfc6700 1 -- 172.21.15.1:0/3917111422 shutdown_connections 2024-03-18T18:51:31.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.379+0000 7f158bfc6700 1 --2- 172.21.15.1:0/3917111422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f158415d5a0 0x7f15840ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:31.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.379+0000 7f158bfc6700 1 --2- 172.21.15.1:0/3917111422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15840ce550 0x7f158406a750 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:31.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.379+0000 7f158bfc6700 1 -- 172.21.15.1:0/3917111422 >> 172.21.15.1:0/3917111422 conn(0x7f15840cc9b0 msgr2=0x7f15840ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:31.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.379+0000 7f158bfc6700 1 -- 172.21.15.1:0/3917111422 shutdown_connections 2024-03-18T18:51:31.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.379+0000 7f158bfc6700 1 -- 172.21.15.1:0/3917111422 wait complete. 2024-03-18T18:51:31.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.380+0000 7f158bfc6700 1 Processor -- start 2024-03-18T18:51:31.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.380+0000 7f158bfc6700 1 -- start start 2024-03-18T18:51:31.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.381+0000 7f158bfc6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15840ce550 0x7f1584160750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:31.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.381+0000 7f158bfc6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15840cfa90 0x7f1584152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:31.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.381+0000 7f158bfc6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158415d5a0 0x7f1584156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:31.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.381+0000 7f158bfc6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1584152570 con 0x7f15840ce550 2024-03-18T18:51:31.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.381+0000 7f158bfc6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1584157180 con 0x7f158415d5a0 2024-03-18T18:51:31.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.381+0000 7f158bfc6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1584157400 con 0x7f15840cfa90 2024-03-18T18:51:31.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.381+0000 7f1589d62700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15840ce550 0x7f1584160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:31.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.381+0000 7f1589d62700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15840ce550 0x7f1584160750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41920/0 (socket says 172.21.15.1:41920) 2024-03-18T18:51:31.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.381+0000 7f1589d62700 1 -- 172.21.15.1:0/154046482 learned_addr learned my addr 172.21.15.1:0/154046482 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:31.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.381+0000 7f1589561700 1 --2- 172.21.15.1:0/154046482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15840cfa90 0x7f1584152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:31.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.381+0000 7f158a563700 1 --2- 172.21.15.1:0/154046482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158415d5a0 0x7f1584156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:31.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.382+0000 7f1589d62700 1 -- 172.21.15.1:0/154046482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15840cfa90 msgr2=0x7f1584152030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:31.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.382+0000 7f1589d62700 1 --2- 172.21.15.1:0/154046482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15840cfa90 0x7f1584152030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:31.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.382+0000 7f1589d62700 1 -- 172.21.15.1:0/154046482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158415d5a0 msgr2=0x7f1584156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:31.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.382+0000 7f1589d62700 1 --2- 172.21.15.1:0/154046482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158415d5a0 0x7f1584156c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:31.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.382+0000 7f1589d62700 1 -- 172.21.15.1:0/154046482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15740094a0 con 0x7f15840ce550 2024-03-18T18:51:31.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.382+0000 7f1589d62700 1 --2- 172.21.15.1:0/154046482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15840ce550 0x7f1584160750 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f158000d7d0 tx=0x7f158000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:31.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.382+0000 7f157a7fc700 1 -- 172.21.15.1:0/154046482 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1580017070 con 0x7f15840ce550 2024-03-18T18:51:31.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.382+0000 7f157a7fc700 1 -- 172.21.15.1:0/154046482 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1580005a20 con 0x7f15840ce550 2024-03-18T18:51:31.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.382+0000 7f158bfc6700 1 -- 172.21.15.1:0/154046482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f158415e850 con 0x7f15840ce550 2024-03-18T18:51:31.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.382+0000 7f157a7fc700 1 -- 172.21.15.1:0/154046482 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1580005bd0 con 0x7f15840ce550 2024-03-18T18:51:31.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.382+0000 7f158bfc6700 1 -- 172.21.15.1:0/154046482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f158415ee10 con 0x7f15840ce550 2024-03-18T18:51:31.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.383+0000 7f158bfc6700 1 -- 172.21.15.1:0/154046482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15840676a0 con 0x7f15840ce550 2024-03-18T18:51:31.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.383+0000 7f157a7fc700 1 -- 172.21.15.1:0/154046482 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 9) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f1580005d30 con 0x7f15840ce550 2024-03-18T18:51:31.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.384+0000 7f157a7fc700 1 --2- 172.21.15.1:0/154046482 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f156806ae00 0x7f156806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:31.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.384+0000 7f157a7fc700 1 -- 172.21.15.1:0/154046482 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f15800890b0 con 0x7f15840ce550 2024-03-18T18:51:31.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.384+0000 7f1589561700 1 --2- 172.21.15.1:0/154046482 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f156806ae00 0x7f156806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:31.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.384+0000 7f1589561700 1 --2- 172.21.15.1:0/154046482 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f156806ae00 0x7f156806d2c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f15740090b0 tx=0x7f1574008030 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:31.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.388+0000 7f157a7fc700 1 -- 172.21.15.1:0/154046482 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1580053fd0 con 0x7f15840ce550 2024-03-18T18:51:31.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.617+0000 7f157a7fc700 1 -- 172.21.15.1:0/154046482 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f15800543a0 con 0x7f15840ce550 2024-03-18T18:51:31.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.619+0000 7f158bfc6700 1 -- 172.21.15.1:0/154046482 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir6", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1584067010 con 0x7f156806ae00 2024-03-18T18:51:31.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.671+0000 7f157a7fc700 1 -- 172.21.15.1:0/154046482 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f1584067010 con 0x7f156806ae00 2024-03-18T18:51:31.668 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path /snap_test_dir1/dir6 2024-03-18T18:51:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.673+0000 7f158bfc6700 1 -- 172.21.15.1:0/154046482 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f156806ae00 msgr2=0x7f156806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.673+0000 7f158bfc6700 1 --2- 172.21.15.1:0/154046482 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f156806ae00 0x7f156806d2c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f15740090b0 tx=0x7f1574008030 comp rx=0 tx=0).stop 2024-03-18T18:51:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.673+0000 7f158bfc6700 1 -- 172.21.15.1:0/154046482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15840ce550 msgr2=0x7f1584160750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.673+0000 7f158bfc6700 1 --2- 172.21.15.1:0/154046482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15840ce550 0x7f1584160750 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f158000d7d0 tx=0x7f158000dae0 comp rx=0 tx=0).stop 2024-03-18T18:51:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.673+0000 7f158bfc6700 1 -- 172.21.15.1:0/154046482 shutdown_connections 2024-03-18T18:51:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.673+0000 7f158bfc6700 1 --2- 172.21.15.1:0/154046482 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f156806ae00 0x7f156806d2c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.673+0000 7f158bfc6700 1 --2- 172.21.15.1:0/154046482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158415d5a0 0x7f1584156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:31.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.673+0000 7f158bfc6700 1 --2- 172.21.15.1:0/154046482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15840ce550 0x7f1584160750 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:31.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.673+0000 7f158bfc6700 1 --2- 172.21.15.1:0/154046482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15840cfa90 0x7f1584152030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:31.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.673+0000 7f158bfc6700 1 -- 172.21.15.1:0/154046482 >> 172.21.15.1:0/154046482 conn(0x7f15840cc9b0 msgr2=0x7f15840b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:31.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.673+0000 7f158bfc6700 1 -- 172.21.15.1:0/154046482 shutdown_connections 2024-03-18T18:51:31.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.673+0000 7f158bfc6700 1 -- 172.21.15.1:0/154046482 wait complete. 2024-03-18T18:51:31.681 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2024-03-18T18:51:31.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:31.682 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2024-03-18T18:51:31.703 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir7 1m 2024-03-18T18:51:31.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.875+0000 7f3ae4897700 1 Processor -- start 2024-03-18T18:51:31.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.875+0000 7f3ae4897700 1 -- start start 2024-03-18T18:51:31.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.876+0000 7f3ae4897700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3ae006ae60 0x7f3ae0168150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:31.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.876+0000 7f3ae4897700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae00cf600 con 0x7f3ae015d5a0 2024-03-18T18:51:31.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.876+0000 7f3ae4897700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae00ce720 con 0x7f3ae00cfa90 2024-03-18T18:51:31.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.876+0000 7f3ae4897700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae00ce9f0 con 0x7f3ae006ae60 2024-03-18T18:51:31.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.876+0000 7f3aded9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3ae006ae60 0x7f3ae0168150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:31.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.876+0000 7f3ade59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3ae015d5a0 0x7f3ae00ce010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52934/0 (socket says 172.21.15.1:52934) 2024-03-18T18:51:31.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.876+0000 7f3ade59c700 1 -- 172.21.15.1:0/3702142280 learned_addr learned my addr 172.21.15.1:0/3702142280 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:31.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.876+0000 7f3aded9d700 1 -- 172.21.15.1:0/3702142280 >> v1:172.21.15.165:6789/0 conn(0x7f3ae00cfa90 legacy=0x7f3ae006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:31.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.876+0000 7f3aded9d700 1 -- 172.21.15.1:0/3702142280 >> v1:172.21.15.1:6789/0 conn(0x7f3ae015d5a0 legacy=0x7f3ae00ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:31.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.876+0000 7f3aded9d700 1 -- 172.21.15.1:0/3702142280 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae00cecd0 con 0x7f3ae006ae60 2024-03-18T18:51:31.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.876+0000 7f3add59a700 1 -- 172.21.15.1:0/3702142280 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038661208 0 0) 0x7f3ae00cf600 con 0x7f3ae015d5a0 2024-03-18T18:51:31.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.877+0000 7f3aded9d700 1 --2- 172.21.15.1:0/3702142280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ae006ae60 0x7f3ae0168150 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009d90 tx=0x7f3ad800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ca067085b4d5b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:31.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.877+0000 7f3add59a700 1 -- 172.21.15.1:0/3702142280 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad800e070 con 0x7f3ae006ae60 2024-03-18T18:51:31.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.877+0000 7f3add59a700 1 -- 172.21.15.1:0/3702142280 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad800bbd0 con 0x7f3ae006ae60 2024-03-18T18:51:31.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.877+0000 7f3add59a700 1 -- 172.21.15.1:0/3702142280 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad8012430 con 0x7f3ae006ae60 2024-03-18T18:51:31.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.877+0000 7f3ae4897700 1 -- 172.21.15.1:0/3702142280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ae006ae60 msgr2=0x7f3ae0168150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:31.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.877+0000 7f3ae4897700 1 --2- 172.21.15.1:0/3702142280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ae006ae60 0x7f3ae0168150 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009d90 tx=0x7f3ad800b330 comp rx=0 tx=0).stop 2024-03-18T18:51:31.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.877+0000 7f3ae4897700 1 -- 172.21.15.1:0/3702142280 shutdown_connections 2024-03-18T18:51:31.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.877+0000 7f3ae4897700 1 --2- 172.21.15.1:0/3702142280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ae006ae60 0x7f3ae0168150 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:31.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.877+0000 7f3ae4897700 1 -- 172.21.15.1:0/3702142280 >> 172.21.15.1:0/3702142280 conn(0x7f3ae00cc9b0 msgr2=0x7f3ae00ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:31.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.877+0000 7f3ae4897700 1 -- 172.21.15.1:0/3702142280 shutdown_connections 2024-03-18T18:51:31.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.877+0000 7f3ae4897700 1 -- 172.21.15.1:0/3702142280 wait complete. 2024-03-18T18:51:31.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.879+0000 7f3ae4897700 1 Processor -- start 2024-03-18T18:51:31.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.879+0000 7f3ae4897700 1 -- start start 2024-03-18T18:51:31.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.879+0000 7f3ae4897700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ae006ae60 0x7f3ae006dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:31.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.879+0000 7f3ae4897700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ae00cfa90 0x7f3ae0160410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:31.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.879+0000 7f3ae4897700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ae015d5a0 0x7f3ae0157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:31.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.879+0000 7f3ae4897700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae0160950 con 0x7f3ae006ae60 2024-03-18T18:51:31.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.879+0000 7f3ae4897700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae006eb70 con 0x7f3ae00cfa90 2024-03-18T18:51:31.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.880+0000 7f3ae4897700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae006e240 con 0x7f3ae015d5a0 2024-03-18T18:51:31.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.880+0000 7f3ade59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ae006ae60 0x7f3ae006dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:31.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.880+0000 7f3ade59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ae006ae60 0x7f3ae006dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41946/0 (socket says 172.21.15.1:41946) 2024-03-18T18:51:31.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.880+0000 7f3ade59c700 1 -- 172.21.15.1:0/4251150844 learned_addr learned my addr 172.21.15.1:0/4251150844 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:31.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.880+0000 7f3addd9b700 1 --2- 172.21.15.1:0/4251150844 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ae00cfa90 0x7f3ae0160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:31.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.880+0000 7f3ade59c700 1 -- 172.21.15.1:0/4251150844 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ae015d5a0 msgr2=0x7f3ae0157d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:31.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.880+0000 7f3aded9d700 1 --2- 172.21.15.1:0/4251150844 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ae015d5a0 0x7f3ae0157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:31.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.880+0000 7f3ade59c700 1 --2- 172.21.15.1:0/4251150844 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ae015d5a0 0x7f3ae0157d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:31.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.880+0000 7f3ade59c700 1 -- 172.21.15.1:0/4251150844 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ae00cfa90 msgr2=0x7f3ae0160410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:31.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.880+0000 7f3ade59c700 1 --2- 172.21.15.1:0/4251150844 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ae00cfa90 0x7f3ae0160410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.880+0000 7f3ade59c700 1 -- 172.21.15.1:0/4251150844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad8009a70 con 0x7f3ae006ae60 2024-03-18T18:51:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.880+0000 7f3addd9b700 1 --2- 172.21.15.1:0/4251150844 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ae00cfa90 0x7f3ae0160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.880+0000 7f3ade59c700 1 --2- 172.21.15.1:0/4251150844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ae006ae60 0x7f3ae006dab0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3ad000c470 tx=0x7f3ad000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.880+0000 7f3aceffd700 1 -- 172.21.15.1:0/4251150844 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad00074e0 con 0x7f3ae006ae60 2024-03-18T18:51:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.880+0000 7f3aceffd700 1 -- 172.21.15.1:0/4251150844 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad0007640 con 0x7f3ae006ae60 2024-03-18T18:51:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.880+0000 7f3ae4897700 1 -- 172.21.15.1:0/4251150844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae006e520 con 0x7f3ae006ae60 2024-03-18T18:51:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.880+0000 7f3aceffd700 1 -- 172.21.15.1:0/4251150844 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad0005a50 con 0x7f3ae006ae60 2024-03-18T18:51:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.880+0000 7f3ae4897700 1 -- 172.21.15.1:0/4251150844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae01582c0 con 0x7f3ae006ae60 2024-03-18T18:51:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.881+0000 7f3ae4897700 1 -- 172.21.15.1:0/4251150844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae0072960 con 0x7f3ae006ae60 2024-03-18T18:51:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.882+0000 7f3aceffd700 1 -- 172.21.15.1:0/4251150844 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f3ad0005bb0 con 0x7f3ae006ae60 2024-03-18T18:51:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.882+0000 7f3aceffd700 1 --2- 172.21.15.1:0/4251150844 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ac006ae90 0x7f3ac006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.882+0000 7f3aceffd700 1 -- 172.21.15.1:0/4251150844 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f3ad0088eb0 con 0x7f3ae006ae60 2024-03-18T18:51:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.882+0000 7f3addd9b700 1 --2- 172.21.15.1:0/4251150844 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ac006ae90 0x7f3ac006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:31.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.883+0000 7f3addd9b700 1 --2- 172.21.15.1:0/4251150844 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ac006ae90 0x7f3ac006d350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3ae01599b0 tx=0x7f3ad4009040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:31.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:31.887+0000 7f3aceffd700 1 -- 172.21.15.1:0/4251150844 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3ad0053ab0 con 0x7f3ae006ae60 2024-03-18T18:51:32.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.121+0000 7f3ae4897700 1 -- 172.21.15.1:0/4251150844 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir7", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ae006e3b0 con 0x7f3ac006ae90 2024-03-18T18:51:32.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.184+0000 7f3aceffd700 1 -- 172.21.15.1:0/4251150844 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f3ae006e3b0 con 0x7f3ac006ae90 2024-03-18T18:51:32.182 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path /snap_test_dir1/dir7 2024-03-18T18:51:32.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.186+0000 7f3ae4897700 1 -- 172.21.15.1:0/4251150844 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ac006ae90 msgr2=0x7f3ac006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:32.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.186+0000 7f3ae4897700 1 --2- 172.21.15.1:0/4251150844 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ac006ae90 0x7f3ac006d350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3ae01599b0 tx=0x7f3ad4009040 comp rx=0 tx=0).stop 2024-03-18T18:51:32.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.186+0000 7f3ae4897700 1 -- 172.21.15.1:0/4251150844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ae006ae60 msgr2=0x7f3ae006dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:32.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.186+0000 7f3ae4897700 1 --2- 172.21.15.1:0/4251150844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ae006ae60 0x7f3ae006dab0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3ad000c470 tx=0x7f3ad000c830 comp rx=0 tx=0).stop 2024-03-18T18:51:32.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.186+0000 7f3ae4897700 1 -- 172.21.15.1:0/4251150844 shutdown_connections 2024-03-18T18:51:32.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.186+0000 7f3ae4897700 1 --2- 172.21.15.1:0/4251150844 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ac006ae90 0x7f3ac006d350 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.186+0000 7f3ae4897700 1 --2- 172.21.15.1:0/4251150844 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ae015d5a0 0x7f3ae0157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.186+0000 7f3ae4897700 1 --2- 172.21.15.1:0/4251150844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ae006ae60 0x7f3ae006dab0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.186+0000 7f3ae4897700 1 --2- 172.21.15.1:0/4251150844 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ae00cfa90 0x7f3ae0160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.186+0000 7f3ae4897700 1 -- 172.21.15.1:0/4251150844 >> 172.21.15.1:0/4251150844 conn(0x7f3ae00cc9b0 msgr2=0x7f3ae00ade50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:32.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.187+0000 7f3ae4897700 1 -- 172.21.15.1:0/4251150844 shutdown_connections 2024-03-18T18:51:32.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.187+0000 7f3ae4897700 1 -- 172.21.15.1:0/4251150844 wait complete. 2024-03-18T18:51:32.195 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2024-03-18T18:51:32.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:32.195 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2024-03-18T18:51:32.217 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir8 1m 2024-03-18T18:51:32.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.390+0000 7f7dc9bc3700 1 Processor -- start 2024-03-18T18:51:32.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.390+0000 7f7dc9bc3700 1 -- start start 2024-03-18T18:51:32.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.391+0000 7f7dc9bc3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7dc415d5b0 0x7f7dc415d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:32.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.391+0000 7f7dc9bc3700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7dc4099a40 0x7f7dc4099ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:32.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.391+0000 7f7dc9bc3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc409a410 0x7f7dc406ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:32.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.391+0000 7f7dc9bc3700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc40cbbb0 con 0x7f7dc415d5b0 2024-03-18T18:51:32.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.391+0000 7f7dc9bc3700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc409a860 con 0x7f7dc4099a40 2024-03-18T18:51:32.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.391+0000 7f7dc9bc3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc40b3b00 con 0x7f7dc409a410 2024-03-18T18:51:32.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.391+0000 7f7dc37fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7dc415d5b0 0x7f7dc415d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:32.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.391+0000 7f7dc37fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7dc415d5b0 0x7f7dc415d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41960/0 (socket says 172.21.15.1:41960) 2024-03-18T18:51:32.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.391+0000 7f7dc37fe700 1 -- 172.21.15.1:0/4189585835 learned_addr learned my addr 172.21.15.1:0/4189585835 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:32.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.391+0000 7f7dc3fff700 1 --2- 172.21.15.1:0/4189585835 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc409a410 0x7f7dc406ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:32.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.391+0000 7f7dc2ffd700 1 --2- 172.21.15.1:0/4189585835 >> v2:172.21.15.165:3300/0 conn(0x7f7dc4099a40 0x7f7dc4099ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:32.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.391+0000 7f7dc37fe700 1 -- 172.21.15.1:0/4189585835 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc409a410 msgr2=0x7f7dc406ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:32.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.391+0000 7f7dc37fe700 1 --2- 172.21.15.1:0/4189585835 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc409a410 0x7f7dc406ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.391+0000 7f7dc37fe700 1 -- 172.21.15.1:0/4189585835 >> v2:172.21.15.165:3300/0 conn(0x7f7dc4099a40 msgr2=0x7f7dc4099ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:32.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.391+0000 7f7dc37fe700 1 --2- 172.21.15.1:0/4189585835 >> v2:172.21.15.165:3300/0 conn(0x7f7dc4099a40 0x7f7dc4099ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.391+0000 7f7dc37fe700 1 -- 172.21.15.1:0/4189585835 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc40cf7f0 con 0x7f7dc415d5b0 2024-03-18T18:51:32.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.391+0000 7f7dc3fff700 1 --2- 172.21.15.1:0/4189585835 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc409a410 0x7f7dc406ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:32.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.391+0000 7f7dc2ffd700 1 --2- 172.21.15.1:0/4189585835 >> v2:172.21.15.165:3300/0 conn(0x7f7dc4099a40 0x7f7dc4099ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:32.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.392+0000 7f7dc37fe700 1 --2- 172.21.15.1:0/4189585835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc415d5b0 0x7f7dc415d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4072c10 tx=0x7f7db4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f55bbd905e898b15 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:32.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.392+0000 7f7dc27fc700 1 -- 172.21.15.1:0/4189585835 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db400b860 con 0x7f7dc415d5b0 2024-03-18T18:51:32.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.392+0000 7f7dc27fc700 1 -- 172.21.15.1:0/4189585835 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db400b9c0 con 0x7f7dc415d5b0 2024-03-18T18:51:32.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.392+0000 7f7dc27fc700 1 -- 172.21.15.1:0/4189585835 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db400bd50 con 0x7f7dc415d5b0 2024-03-18T18:51:32.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.392+0000 7f7dc9bc3700 1 -- 172.21.15.1:0/4189585835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc415d5b0 msgr2=0x7f7dc415d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:32.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.392+0000 7f7dc9bc3700 1 --2- 172.21.15.1:0/4189585835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc415d5b0 0x7f7dc415d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4072c10 tx=0x7f7db4009c00 comp rx=0 tx=0).stop 2024-03-18T18:51:32.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.392+0000 7f7dc9bc3700 1 -- 172.21.15.1:0/4189585835 shutdown_connections 2024-03-18T18:51:32.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.392+0000 7f7dc9bc3700 1 --2- 172.21.15.1:0/4189585835 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc409a410 0x7f7dc406ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.392+0000 7f7dc9bc3700 1 --2- 172.21.15.1:0/4189585835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc415d5b0 0x7f7dc415d990 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.392+0000 7f7dc9bc3700 1 --2- 172.21.15.1:0/4189585835 >> v2:172.21.15.165:3300/0 conn(0x7f7dc4099a40 0x7f7dc4099ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.392+0000 7f7dc9bc3700 1 -- 172.21.15.1:0/4189585835 >> 172.21.15.1:0/4189585835 conn(0x7f7dc40b2bd0 msgr2=0x7f7dc40b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:32.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.392+0000 7f7dc9bc3700 1 -- 172.21.15.1:0/4189585835 shutdown_connections 2024-03-18T18:51:32.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.392+0000 7f7dc9bc3700 1 -- 172.21.15.1:0/4189585835 wait complete. 2024-03-18T18:51:32.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.394+0000 7f7dc9bc3700 1 Processor -- start 2024-03-18T18:51:32.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.394+0000 7f7dc9bc3700 1 -- start start 2024-03-18T18:51:32.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.395+0000 7f7dc9bc3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc4099a40 0x7f7dc41f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:32.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.395+0000 7f7dc9bc3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc409a410 0x7f7dc40c9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:32.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.395+0000 7f7dc9bc3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7dc415d5b0 0x7f7dc40c6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:32.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.395+0000 7f7dc9bc3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc41580b0 con 0x7f7dc409a410 2024-03-18T18:51:32.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.395+0000 7f7dc9bc3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc41574a0 con 0x7f7dc415d5b0 2024-03-18T18:51:32.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.395+0000 7f7dc9bc3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc4157780 con 0x7f7dc4099a40 2024-03-18T18:51:32.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.395+0000 7f7dc2ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc409a410 0x7f7dc40c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:32.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.395+0000 7f7dc2ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc409a410 0x7f7dc40c9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41970/0 (socket says 172.21.15.1:41970) 2024-03-18T18:51:32.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.395+0000 7f7dc2ffd700 1 -- 172.21.15.1:0/759493266 learned_addr learned my addr 172.21.15.1:0/759493266 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:32.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.395+0000 7f7dc3fff700 1 --2- 172.21.15.1:0/759493266 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7dc415d5b0 0x7f7dc40c6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:32.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.395+0000 7f7dc37fe700 1 --2- 172.21.15.1:0/759493266 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc4099a40 0x7f7dc41f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:32.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.396+0000 7f7dc2ffd700 1 -- 172.21.15.1:0/759493266 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc4099a40 msgr2=0x7f7dc41f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:32.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.396+0000 7f7dc2ffd700 1 --2- 172.21.15.1:0/759493266 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc4099a40 0x7f7dc41f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.396+0000 7f7dc2ffd700 1 -- 172.21.15.1:0/759493266 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7dc415d5b0 msgr2=0x7f7dc40c6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:32.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.396+0000 7f7dc2ffd700 1 --2- 172.21.15.1:0/759493266 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7dc415d5b0 0x7f7dc40c6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.396+0000 7f7dc2ffd700 1 -- 172.21.15.1:0/759493266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db4009550 con 0x7f7dc409a410 2024-03-18T18:51:32.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.396+0000 7f7dc2ffd700 1 --2- 172.21.15.1:0/759493266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc409a410 0x7f7dc40c9010 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7dac00c7f0 tx=0x7f7dac00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:32.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.396+0000 7f7dbbfff700 1 -- 172.21.15.1:0/759493266 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dac011800 con 0x7f7dc409a410 2024-03-18T18:51:32.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.396+0000 7f7dbbfff700 1 -- 172.21.15.1:0/759493266 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dac011960 con 0x7f7dc409a410 2024-03-18T18:51:32.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.396+0000 7f7dc9bc3700 1 -- 172.21.15.1:0/759493266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc4157a60 con 0x7f7dc409a410 2024-03-18T18:51:32.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.396+0000 7f7dbbfff700 1 -- 172.21.15.1:0/759493266 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dac011d10 con 0x7f7dc409a410 2024-03-18T18:51:32.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.396+0000 7f7dc9bc3700 1 -- 172.21.15.1:0/759493266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc41f2f10 con 0x7f7dc409a410 2024-03-18T18:51:32.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.397+0000 7f7dc9bc3700 1 -- 172.21.15.1:0/759493266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc41f3130 con 0x7f7dc409a410 2024-03-18T18:51:32.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.397+0000 7f7dbbfff700 1 -- 172.21.15.1:0/759493266 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f7dac018360 con 0x7f7dc409a410 2024-03-18T18:51:32.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.398+0000 7f7dbbfff700 1 --2- 172.21.15.1:0/759493266 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7da406ae90 0x7f7da406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:32.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.398+0000 7f7dbbfff700 1 -- 172.21.15.1:0/759493266 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f7dac089830 con 0x7f7dc409a410 2024-03-18T18:51:32.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.398+0000 7f7dc37fe700 1 --2- 172.21.15.1:0/759493266 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7da406ae90 0x7f7da406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:32.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.398+0000 7f7dc37fe700 1 --2- 172.21.15.1:0/759493266 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7da406ae90 0x7f7da406d350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7db4000c00 tx=0x7f7db4008030 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:32.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.403+0000 7f7dbbfff700 1 -- 172.21.15.1:0/759493266 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7dac054440 con 0x7f7dc409a410 2024-03-18T18:51:32.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.635+0000 7f7dc9bc3700 1 -- 172.21.15.1:0/759493266 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir8", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7dc40d07b0 con 0x7f7da406ae90 2024-03-18T18:51:32.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.695+0000 7f7dbbfff700 1 -- 172.21.15.1:0/759493266 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f7db4006d20 con 0x7f7da406ae90 2024-03-18T18:51:32.692 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path /snap_test_dir1/dir8 2024-03-18T18:51:32.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.697+0000 7f7dc9bc3700 1 -- 172.21.15.1:0/759493266 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7da406ae90 msgr2=0x7f7da406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:32.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.698+0000 7f7dc9bc3700 1 --2- 172.21.15.1:0/759493266 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7da406ae90 0x7f7da406d350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7db4000c00 tx=0x7f7db4008030 comp rx=0 tx=0).stop 2024-03-18T18:51:32.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.698+0000 7f7dc9bc3700 1 -- 172.21.15.1:0/759493266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc409a410 msgr2=0x7f7dc40c9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:32.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.698+0000 7f7dc9bc3700 1 --2- 172.21.15.1:0/759493266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc409a410 0x7f7dc40c9010 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7dac00c7f0 tx=0x7f7dac00cb00 comp rx=0 tx=0).stop 2024-03-18T18:51:32.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.698+0000 7f7dc9bc3700 1 -- 172.21.15.1:0/759493266 shutdown_connections 2024-03-18T18:51:32.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.698+0000 7f7dc9bc3700 1 --2- 172.21.15.1:0/759493266 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7da406ae90 0x7f7da406d350 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.698+0000 7f7dc9bc3700 1 --2- 172.21.15.1:0/759493266 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc4099a40 0x7f7dc41f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.698+0000 7f7dc9bc3700 1 --2- 172.21.15.1:0/759493266 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7dc415d5b0 0x7f7dc40c6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.698+0000 7f7dc9bc3700 1 --2- 172.21.15.1:0/759493266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc409a410 0x7f7dc40c9010 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.698+0000 7f7dc9bc3700 1 -- 172.21.15.1:0/759493266 >> 172.21.15.1:0/759493266 conn(0x7f7dc40b2bd0 msgr2=0x7f7dc406cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:32.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.698+0000 7f7dc9bc3700 1 -- 172.21.15.1:0/759493266 shutdown_connections 2024-03-18T18:51:32.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.698+0000 7f7dc9bc3700 1 -- 172.21.15.1:0/759493266 wait complete. 2024-03-18T18:51:32.706 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2024-03-18T18:51:32.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T18:51:32.706 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2024-03-18T18:51:32.726 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir9 1m 2024-03-18T18:51:32.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.898+0000 7fef4d302700 1 Processor -- start 2024-03-18T18:51:32.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.898+0000 7fef4d302700 1 -- start start 2024-03-18T18:51:32.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.899+0000 7fef4d302700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef4815d3e0 0x7fef480d0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:32.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.899+0000 7fef4d302700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fef4815d7c0 0x7fef480d17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:32.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.899+0000 7fef4d302700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef480ce1e0 con 0x7fef480d1cf0 2024-03-18T18:51:32.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.899+0000 7fef4d302700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef480cee60 con 0x7fef4815d7c0 2024-03-18T18:51:32.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.899+0000 7fef4d302700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef480d2160 con 0x7fef4815d3e0 2024-03-18T18:51:32.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.899+0000 7fef477fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fef480d1cf0 0x7fef480b82d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52938/0 (socket says 172.21.15.1:52938) 2024-03-18T18:51:32.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.899+0000 7fef477fe700 1 -- 172.21.15.1:0/3735575141 learned_addr learned my addr 172.21.15.1:0/3735575141 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:32.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.899+0000 7fef45ffb700 1 -- 172.21.15.1:0/3735575141 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244385896 0 0) 0x7fef480ce1e0 con 0x7fef480d1cf0 2024-03-18T18:51:32.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.899+0000 7fef467fc700 1 --2- 172.21.15.1:0/3735575141 >> v2:172.21.15.165:3300/0 conn(0x7fef4815d7c0 0x7fef480d17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:32.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.899+0000 7fef46ffd700 1 --2- 172.21.15.1:0/3735575141 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef4815d3e0 0x7fef480d0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:32.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.899+0000 7fef45ffb700 1 -- 172.21.15.1:0/3735575141 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef300035b0 con 0x7fef480d1cf0 2024-03-18T18:51:32.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef45ffb700 1 -- 172.21.15.1:0/3735575141 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 763518786 0 0) 0x7fef300035b0 con 0x7fef480d1cf0 2024-03-18T18:51:32.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef45ffb700 1 -- 172.21.15.1:0/3735575141 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef4815d3e0 msgr2=0x7fef480d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:32.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef45ffb700 1 --2- 172.21.15.1:0/3735575141 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef4815d3e0 0x7fef480d0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef45ffb700 1 -- 172.21.15.1:0/3735575141 >> v2:172.21.15.165:3300/0 conn(0x7fef4815d7c0 msgr2=0x7fef480d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:32.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef45ffb700 1 --2- 172.21.15.1:0/3735575141 >> v2:172.21.15.165:3300/0 conn(0x7fef4815d7c0 0x7fef480d17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef45ffb700 1 -- 172.21.15.1:0/3735575141 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef480ce4c0 con 0x7fef480d1cf0 2024-03-18T18:51:32.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef45ffb700 1 -- 172.21.15.1:0/3735575141 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fef400019f0 con 0x7fef480d1cf0 2024-03-18T18:51:32.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef45ffb700 1 -- 172.21.15.1:0/3735575141 >> v1:172.21.15.1:6789/0 conn(0x7fef480d1cf0 legacy=0x7fef480b82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:32.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef45ffb700 1 --2- 172.21.15.1:0/3735575141 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef4815d3e0 0x7fef30003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:32.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef45ffb700 1 --2- 172.21.15.1:0/3735575141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef30003fb0 0x7fef30004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:32.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef45ffb700 1 --2- 172.21.15.1:0/3735575141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef30004d90 0x7fef30005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:32.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef45ffb700 1 -- 172.21.15.1:0/3735575141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef480cee60 con 0x7fef30004d90 2024-03-18T18:51:32.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef45ffb700 1 -- 172.21.15.1:0/3735575141 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef480d2160 con 0x7fef30003fb0 2024-03-18T18:51:32.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef45ffb700 1 -- 172.21.15.1:0/3735575141 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3000ba10 con 0x7fef4815d3e0 2024-03-18T18:51:32.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef477fe700 1 --2- 172.21.15.1:0/3735575141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef30004d90 0x7fef30005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:32.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef467fc700 1 --2- 172.21.15.1:0/3735575141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef30003fb0 0x7fef30004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:32.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef46ffd700 1 --2- 172.21.15.1:0/3735575141 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef4815d3e0 0x7fef30003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:32.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef477fe700 1 -- 172.21.15.1:0/3735575141 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef4815d3e0 msgr2=0x7fef30003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:32.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef477fe700 1 --2- 172.21.15.1:0/3735575141 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef4815d3e0 0x7fef30003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef477fe700 1 -- 172.21.15.1:0/3735575141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef30003fb0 msgr2=0x7fef30004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:32.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef477fe700 1 --2- 172.21.15.1:0/3735575141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef30003fb0 0x7fef30004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef477fe700 1 -- 172.21.15.1:0/3735575141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef300035b0 con 0x7fef30004d90 2024-03-18T18:51:32.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef477fe700 1 --2- 172.21.15.1:0/3735575141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef30004d90 0x7fef30005210 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fef4000a820 tx=0x7fef30005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c069e34fcdf38b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:32.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef45ffb700 1 -- 172.21.15.1:0/3735575141 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4000a2f0 con 0x7fef30004d90 2024-03-18T18:51:32.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef45ffb700 1 -- 172.21.15.1:0/3735575141 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef40012620 con 0x7fef30004d90 2024-03-18T18:51:32.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.900+0000 7fef45ffb700 1 -- 172.21.15.1:0/3735575141 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef400129d0 con 0x7fef30004d90 2024-03-18T18:51:32.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.901+0000 7fef4d302700 1 -- 172.21.15.1:0/3735575141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef30004d90 msgr2=0x7fef30005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:32.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.901+0000 7fef4d302700 1 --2- 172.21.15.1:0/3735575141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef30004d90 0x7fef30005210 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fef4000a820 tx=0x7fef30005880 comp rx=0 tx=0).stop 2024-03-18T18:51:32.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.901+0000 7fef477fe700 1 -- 172.21.15.1:0/3735575141 reap_dead start 2024-03-18T18:51:32.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.901+0000 7fef4d302700 1 -- 172.21.15.1:0/3735575141 shutdown_connections 2024-03-18T18:51:32.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.901+0000 7fef4d302700 1 -- 172.21.15.1:0/3735575141 >> 172.21.15.1:0/3735575141 conn(0x7fef4809c540 msgr2=0x7fef480d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:32.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.901+0000 7fef4d302700 1 -- 172.21.15.1:0/3735575141 shutdown_connections 2024-03-18T18:51:32.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.901+0000 7fef4d302700 1 -- 172.21.15.1:0/3735575141 wait complete. 2024-03-18T18:51:32.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.902+0000 7fef4d302700 1 Processor -- start 2024-03-18T18:51:32.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.902+0000 7fef4d302700 1 -- start start 2024-03-18T18:51:32.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.903+0000 7fef4d302700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef481f2e50 0x7fef4815a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:32.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.903+0000 7fef4d302700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef481f3d20 0x7fef481f4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:32.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.903+0000 7fef4d302700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef480b78c0 0x7fef480b7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:32.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.903+0000 7fef4d302700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef480b8280 con 0x7fef480b78c0 2024-03-18T18:51:32.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.903+0000 7fef4d302700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef481f4ca0 con 0x7fef481f2e50 2024-03-18T18:51:32.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.903+0000 7fef4d302700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef4815a010 con 0x7fef481f3d20 2024-03-18T18:51:32.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.903+0000 7fef477fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef480b78c0 0x7fef480b7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:32.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.903+0000 7fef477fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef480b78c0 0x7fef480b7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41984/0 (socket says 172.21.15.1:41984) 2024-03-18T18:51:32.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.903+0000 7fef477fe700 1 -- 172.21.15.1:0/1572485699 learned_addr learned my addr 172.21.15.1:0/1572485699 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:32.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.903+0000 7fef46ffd700 1 --2- 172.21.15.1:0/1572485699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef481f2e50 0x7fef4815a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:32.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.903+0000 7fef467fc700 1 --2- 172.21.15.1:0/1572485699 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef481f3d20 0x7fef481f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:32.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.903+0000 7fef477fe700 1 -- 172.21.15.1:0/1572485699 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef481f3d20 msgr2=0x7fef481f4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:32.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.903+0000 7fef477fe700 1 --2- 172.21.15.1:0/1572485699 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef481f3d20 0x7fef481f4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.903+0000 7fef477fe700 1 -- 172.21.15.1:0/1572485699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef481f2e50 msgr2=0x7fef4815a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:32.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.903+0000 7fef477fe700 1 --2- 172.21.15.1:0/1572485699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef481f2e50 0x7fef4815a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:32.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.903+0000 7fef477fe700 1 -- 172.21.15.1:0/1572485699 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4815aa70 con 0x7fef480b78c0 2024-03-18T18:51:32.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.904+0000 7fef477fe700 1 --2- 172.21.15.1:0/1572485699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef480b78c0 0x7fef480b7d40 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fef30004720 tx=0x7fef480d3f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:32.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.904+0000 7fef2f7fe700 1 -- 172.21.15.1:0/1572485699 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef40024070 con 0x7fef480b78c0 2024-03-18T18:51:32.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.904+0000 7fef2f7fe700 1 -- 172.21.15.1:0/1572485699 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef400017b0 con 0x7fef480b78c0 2024-03-18T18:51:32.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.904+0000 7fef4d302700 1 -- 172.21.15.1:0/1572485699 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef481f4470 con 0x7fef480b78c0 2024-03-18T18:51:32.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.904+0000 7fef2f7fe700 1 -- 172.21.15.1:0/1572485699 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4001e430 con 0x7fef480b78c0 2024-03-18T18:51:32.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.904+0000 7fef4d302700 1 -- 172.21.15.1:0/1572485699 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef481f41a0 con 0x7fef480b78c0 2024-03-18T18:51:32.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.905+0000 7fef4d302700 1 -- 172.21.15.1:0/1572485699 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef48073350 con 0x7fef480b78c0 2024-03-18T18:51:32.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.905+0000 7fef2f7fe700 1 -- 172.21.15.1:0/1572485699 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fef4001e5b0 con 0x7fef480b78c0 2024-03-18T18:51:32.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.906+0000 7fef2f7fe700 1 --2- 172.21.15.1:0/1572485699 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fef3406ae90 0x7fef3406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:32.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.906+0000 7fef2f7fe700 1 -- 172.21.15.1:0/1572485699 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fef4008abf0 con 0x7fef480b78c0 2024-03-18T18:51:32.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.906+0000 7fef46ffd700 1 --2- 172.21.15.1:0/1572485699 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fef3406ae90 0x7fef3406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:32.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.906+0000 7fef46ffd700 1 --2- 172.21.15.1:0/1572485699 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fef3406ae90 0x7fef3406d350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fef4815ae40 tx=0x7fef3c009210 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:32.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:32.910+0000 7fef2f7fe700 1 -- 172.21.15.1:0/1572485699 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fef40055a60 con 0x7fef480b78c0 2024-03-18T18:51:33.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.140+0000 7fef4d302700 1 -- 172.21.15.1:0/1572485699 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir9", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fef480cf280 con 0x7fef3406ae90 2024-03-18T18:51:33.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.194+0000 7fef2f7fe700 1 -- 172.21.15.1:0/1572485699 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fef3c019070 con 0x7fef3406ae90 2024-03-18T18:51:33.190 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path /snap_test_dir1/dir9 2024-03-18T18:51:33.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.196+0000 7fef4d302700 1 -- 172.21.15.1:0/1572485699 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fef3406ae90 msgr2=0x7fef3406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:33.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.196+0000 7fef4d302700 1 --2- 172.21.15.1:0/1572485699 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fef3406ae90 0x7fef3406d350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fef4815ae40 tx=0x7fef3c009210 comp rx=0 tx=0).stop 2024-03-18T18:51:33.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.196+0000 7fef4d302700 1 -- 172.21.15.1:0/1572485699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef480b78c0 msgr2=0x7fef480b7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:33.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.196+0000 7fef4d302700 1 --2- 172.21.15.1:0/1572485699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef480b78c0 0x7fef480b7d40 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fef30004720 tx=0x7fef480d3f90 comp rx=0 tx=0).stop 2024-03-18T18:51:33.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.196+0000 7fef4d302700 1 -- 172.21.15.1:0/1572485699 shutdown_connections 2024-03-18T18:51:33.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.196+0000 7fef4d302700 1 --2- 172.21.15.1:0/1572485699 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fef3406ae90 0x7fef3406d350 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:33.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.196+0000 7fef4d302700 1 --2- 172.21.15.1:0/1572485699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef480b78c0 0x7fef480b7d40 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:33.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.196+0000 7fef4d302700 1 --2- 172.21.15.1:0/1572485699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef481f2e50 0x7fef4815a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:33.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.196+0000 7fef4d302700 1 --2- 172.21.15.1:0/1572485699 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef481f3d20 0x7fef481f4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:33.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.196+0000 7fef4d302700 1 -- 172.21.15.1:0/1572485699 >> 172.21.15.1:0/1572485699 conn(0x7fef4809c540 msgr2=0x7fef4815ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:33.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.196+0000 7fef4d302700 1 -- 172.21.15.1:0/1572485699 shutdown_connections 2024-03-18T18:51:33.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.196+0000 7fef4d302700 1 -- 172.21.15.1:0/1572485699 wait complete. 2024-03-18T18:51:33.204 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2024-03-18T18:51:33.205 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir0 1m 2024-03-18T18:51:33.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.336+0000 7f2f2f88e700 1 Processor -- start 2024-03-18T18:51:33.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.336+0000 7f2f2f88e700 1 -- start start 2024-03-18T18:51:33.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.337+0000 7f2f2f88e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2f280cfa90 0x7f2f2806a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:33.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.337+0000 7f2f2f88e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f280cf600 con 0x7f2f2815d5a0 2024-03-18T18:51:33.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.337+0000 7f2f2f88e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f280ce720 con 0x7f2f2806ac90 2024-03-18T18:51:33.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.337+0000 7f2f2f88e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f2806b140 con 0x7f2f280cfa90 2024-03-18T18:51:33.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.337+0000 7f2f2ce29700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2f280cfa90 0x7f2f2806a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:33.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.337+0000 7f2f2d62a700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2f2815d5a0 0x7f2f280ce010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52944/0 (socket says 172.21.15.1:52944) 2024-03-18T18:51:33.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.337+0000 7f2f2d62a700 1 -- 172.21.15.1:0/3447102558 learned_addr learned my addr 172.21.15.1:0/3447102558 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:33.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.338+0000 7f2f2ce29700 1 -- 172.21.15.1:0/3447102558 >> v1:172.21.15.165:6789/0 conn(0x7f2f2806ac90 legacy=0x7f2f28167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:33.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.338+0000 7f2f2ce29700 1 -- 172.21.15.1:0/3447102558 >> v1:172.21.15.1:6789/0 conn(0x7f2f2815d5a0 legacy=0x7f2f280ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:33.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.338+0000 7f2f2ce29700 1 -- 172.21.15.1:0/3447102558 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f280ce9f0 con 0x7f2f280cfa90 2024-03-18T18:51:33.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.338+0000 7f2f2ce29700 1 --2- 172.21.15.1:0/3447102558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f280cfa90 0x7f2f2806a750 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f2f24009840 tx=0x7f2f24009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e66a0e2b5febd7e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:33.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.338+0000 7f2f1ffff700 1 -- 172.21.15.1:0/3447102558 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f24011830 con 0x7f2f280cfa90 2024-03-18T18:51:33.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.338+0000 7f2f1ffff700 1 -- 172.21.15.1:0/3447102558 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f24011990 con 0x7f2f280cfa90 2024-03-18T18:51:33.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.338+0000 7f2f1ffff700 1 -- 172.21.15.1:0/3447102558 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f24011d20 con 0x7f2f280cfa90 2024-03-18T18:51:33.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.339+0000 7f2f2f88e700 1 -- 172.21.15.1:0/3447102558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f280cfa90 msgr2=0x7f2f2806a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:33.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.339+0000 7f2f2f88e700 1 --2- 172.21.15.1:0/3447102558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f280cfa90 0x7f2f2806a750 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f2f24009840 tx=0x7f2f24009b50 comp rx=0 tx=0).stop 2024-03-18T18:51:33.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.339+0000 7f2f2f88e700 1 -- 172.21.15.1:0/3447102558 shutdown_connections 2024-03-18T18:51:33.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.339+0000 7f2f2f88e700 1 --2- 172.21.15.1:0/3447102558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f280cfa90 0x7f2f2806a750 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:33.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.339+0000 7f2f2f88e700 1 -- 172.21.15.1:0/3447102558 >> 172.21.15.1:0/3447102558 conn(0x7f2f280cc9b0 msgr2=0x7f2f280ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:33.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.339+0000 7f2f2f88e700 1 -- 172.21.15.1:0/3447102558 shutdown_connections 2024-03-18T18:51:33.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.339+0000 7f2f2f88e700 1 -- 172.21.15.1:0/3447102558 wait complete. 2024-03-18T18:51:33.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.340+0000 7f2f2f88e700 1 Processor -- start 2024-03-18T18:51:33.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.340+0000 7f2f2f88e700 1 -- start start 2024-03-18T18:51:33.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.341+0000 7f2f2f88e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f2806ac90 0x7f2f28159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:33.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.341+0000 7f2f2f88e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f280cfa90 0x7f2f2815fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:33.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.341+0000 7f2f2f88e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f2815d5a0 0x7f2f28071f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:33.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.341+0000 7f2f2f88e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f28160400 con 0x7f2f280cfa90 2024-03-18T18:51:33.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.341+0000 7f2f2f88e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f28160c40 con 0x7f2f2806ac90 2024-03-18T18:51:33.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.341+0000 7f2f2f88e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f28160680 con 0x7f2f2815d5a0 2024-03-18T18:51:33.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.341+0000 7f2f2ce29700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f280cfa90 0x7f2f2815fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:33.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.341+0000 7f2f2ce29700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f280cfa90 0x7f2f2815fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42014/0 (socket says 172.21.15.1:42014) 2024-03-18T18:51:33.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.341+0000 7f2f2ce29700 1 -- 172.21.15.1:0/3672405884 learned_addr learned my addr 172.21.15.1:0/3672405884 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:33.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.341+0000 7f2f2d62a700 1 --2- 172.21.15.1:0/3672405884 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f2806ac90 0x7f2f28159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:33.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.341+0000 7f2f2de2b700 1 --2- 172.21.15.1:0/3672405884 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f2815d5a0 0x7f2f28071f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:33.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.342+0000 7f2f2ce29700 1 -- 172.21.15.1:0/3672405884 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f2815d5a0 msgr2=0x7f2f28071f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:33.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.342+0000 7f2f2ce29700 1 --2- 172.21.15.1:0/3672405884 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f2815d5a0 0x7f2f28071f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:33.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.342+0000 7f2f2ce29700 1 -- 172.21.15.1:0/3672405884 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f2806ac90 msgr2=0x7f2f28159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:33.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.342+0000 7f2f2ce29700 1 --2- 172.21.15.1:0/3672405884 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f2806ac90 0x7f2f28159080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:33.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.342+0000 7f2f2ce29700 1 -- 172.21.15.1:0/3672405884 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f240094a0 con 0x7f2f280cfa90 2024-03-18T18:51:33.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.342+0000 7f2f2ce29700 1 --2- 172.21.15.1:0/3672405884 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f280cfa90 0x7f2f2815fec0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f2f240090b0 tx=0x7f2f24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:33.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.342+0000 7f2f1dffb700 1 -- 172.21.15.1:0/3672405884 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f240036a0 con 0x7f2f280cfa90 2024-03-18T18:51:33.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.342+0000 7f2f1dffb700 1 -- 172.21.15.1:0/3672405884 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2401f070 con 0x7f2f280cfa90 2024-03-18T18:51:33.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.342+0000 7f2f2f88e700 1 -- 172.21.15.1:0/3672405884 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f28160960 con 0x7f2f280cfa90 2024-03-18T18:51:33.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.342+0000 7f2f1dffb700 1 -- 172.21.15.1:0/3672405884 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f24003a60 con 0x7f2f280cfa90 2024-03-18T18:51:33.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.342+0000 7f2f2f88e700 1 -- 172.21.15.1:0/3672405884 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f280c3e70 con 0x7f2f280cfa90 2024-03-18T18:51:33.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.343+0000 7f2f2f88e700 1 -- 172.21.15.1:0/3672405884 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f280676a0 con 0x7f2f280cfa90 2024-03-18T18:51:33.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.344+0000 7f2f1dffb700 1 -- 172.21.15.1:0/3672405884 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f2f24003bc0 con 0x7f2f280cfa90 2024-03-18T18:51:33.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.344+0000 7f2f1dffb700 1 --2- 172.21.15.1:0/3672405884 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2f0c06ae90 0x7f2f0c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:33.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.344+0000 7f2f1dffb700 1 -- 172.21.15.1:0/3672405884 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f2f24021070 con 0x7f2f280cfa90 2024-03-18T18:51:33.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.344+0000 7f2f2d62a700 1 --2- 172.21.15.1:0/3672405884 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2f0c06ae90 0x7f2f0c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:33.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.345+0000 7f2f2d62a700 1 --2- 172.21.15.1:0/3672405884 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2f0c06ae90 0x7f2f0c06d350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2f18007fd0 tx=0x7f2f18009040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:33.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.349+0000 7f2f1dffb700 1 -- 172.21.15.1:0/3672405884 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2f240be580 con 0x7f2f280cfa90 2024-03-18T18:51:33.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.581+0000 7f2f2f88e700 1 -- 172.21.15.1:0/3672405884 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f280cf290 con 0x7f2f0c06ae90 2024-03-18T18:51:33.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.632+0000 7f2f1dffb700 1 -- 172.21.15.1:0/3672405884 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f2f18001500 con 0x7f2f0c06ae90 2024-03-18T18:51:33.629 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule activated for path /snap_test_dir1/dir0 2024-03-18T18:51:33.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.635+0000 7f2f2f88e700 1 -- 172.21.15.1:0/3672405884 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2f0c06ae90 msgr2=0x7f2f0c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:33.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.635+0000 7f2f2f88e700 1 --2- 172.21.15.1:0/3672405884 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2f0c06ae90 0x7f2f0c06d350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2f18007fd0 tx=0x7f2f18009040 comp rx=0 tx=0).stop 2024-03-18T18:51:33.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.635+0000 7f2f2f88e700 1 -- 172.21.15.1:0/3672405884 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f280cfa90 msgr2=0x7f2f2815fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:33.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.635+0000 7f2f2f88e700 1 --2- 172.21.15.1:0/3672405884 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f280cfa90 0x7f2f2815fec0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f2f240090b0 tx=0x7f2f24004750 comp rx=0 tx=0).stop 2024-03-18T18:51:33.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.635+0000 7f2f2f88e700 1 -- 172.21.15.1:0/3672405884 shutdown_connections 2024-03-18T18:51:33.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.635+0000 7f2f2f88e700 1 --2- 172.21.15.1:0/3672405884 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2f0c06ae90 0x7f2f0c06d350 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:33.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.635+0000 7f2f2f88e700 1 --2- 172.21.15.1:0/3672405884 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f2815d5a0 0x7f2f28071f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:33.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.635+0000 7f2f2f88e700 1 --2- 172.21.15.1:0/3672405884 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f2806ac90 0x7f2f28159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:33.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.635+0000 7f2f2f88e700 1 --2- 172.21.15.1:0/3672405884 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f280cfa90 0x7f2f2815fec0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:33.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.635+0000 7f2f2f88e700 1 -- 172.21.15.1:0/3672405884 >> 172.21.15.1:0/3672405884 conn(0x7f2f280cc9b0 msgr2=0x7f2f280adbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:33.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.635+0000 7f2f2f88e700 1 -- 172.21.15.1:0/3672405884 shutdown_connections 2024-03-18T18:51:33.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.635+0000 7f2f2f88e700 1 -- 172.21.15.1:0/3672405884 wait complete. 2024-03-18T18:51:33.643 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2024-03-18T18:51:33.643 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir1 1m 2024-03-18T18:51:33.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.775+0000 7fe66f5cd700 1 Processor -- start 2024-03-18T18:51:33.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.775+0000 7fe66f5cd700 1 -- start start 2024-03-18T18:51:33.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.776+0000 7fe66f5cd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe6680ca5d0 0x7fe6680cd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:33.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.776+0000 7fe66f5cd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe66815b540 0x7fe6680cc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:33.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.776+0000 7fe66f5cd700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6680cf440 con 0x7fe66806a750 2024-03-18T18:51:33.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.776+0000 7fe66f5cd700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6680caa60 con 0x7fe6680ca5d0 2024-03-18T18:51:33.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.776+0000 7fe66f5cd700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe668071fb0 con 0x7fe66815b540 2024-03-18T18:51:33.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.776+0000 7fe66cb68700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe6680ca5d0 0x7fe6680cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:33.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.776+0000 7fe66cb68700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe6680ca5d0 0x7fe6680cd520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42016/0 (socket says 172.21.15.1:42016) 2024-03-18T18:51:33.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.776+0000 7fe66d369700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe66806a750 0x7fe66806ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52956/0 (socket says 172.21.15.1:52956) 2024-03-18T18:51:33.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.776+0000 7fe66cb68700 1 -- 172.21.15.1:0/3060963801 learned_addr learned my addr 172.21.15.1:0/3060963801 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:33.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.776+0000 7fe66db6a700 1 --2- 172.21.15.1:0/3060963801 >> v2:172.21.15.165:3300/0 conn(0x7fe66815b540 0x7fe6680cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:33.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.776+0000 7fe66cb68700 1 -- 172.21.15.1:0/3060963801 >> v2:172.21.15.165:3300/0 conn(0x7fe66815b540 msgr2=0x7fe6680cc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:33.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.776+0000 7fe66cb68700 1 --2- 172.21.15.1:0/3060963801 >> v2:172.21.15.165:3300/0 conn(0x7fe66815b540 0x7fe6680cc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:33.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.776+0000 7fe66cb68700 1 -- 172.21.15.1:0/3060963801 >> v1:172.21.15.1:6789/0 conn(0x7fe66806a750 legacy=0x7fe66806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:33.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.776+0000 7fe66cb68700 1 -- 172.21.15.1:0/3060963801 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe668072290 con 0x7fe6680ca5d0 2024-03-18T18:51:33.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.776+0000 7fe65ffff700 1 -- 172.21.15.1:0/3060963801 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406527890 0 0) 0x7fe6680cf440 con 0x7fe66806a750 2024-03-18T18:51:33.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.776+0000 7fe66cb68700 1 --2- 172.21.15.1:0/3060963801 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6680ca5d0 0x7fe6680cd520 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fe6680b2a50 tx=0x7fe658009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e91c9c9306a481 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:33.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.776+0000 7fe65ffff700 1 -- 172.21.15.1:0/3060963801 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65800b870 con 0x7fe6680ca5d0 2024-03-18T18:51:33.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.776+0000 7fe65ffff700 1 -- 172.21.15.1:0/3060963801 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65800b9d0 con 0x7fe6680ca5d0 2024-03-18T18:51:33.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.776+0000 7fe65ffff700 1 -- 172.21.15.1:0/3060963801 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65800bd60 con 0x7fe6680ca5d0 2024-03-18T18:51:33.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.777+0000 7fe66f5cd700 1 -- 172.21.15.1:0/3060963801 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6680ca5d0 msgr2=0x7fe6680cd520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:33.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.777+0000 7fe66f5cd700 1 --2- 172.21.15.1:0/3060963801 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6680ca5d0 0x7fe6680cd520 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fe6680b2a50 tx=0x7fe658009bd0 comp rx=0 tx=0).stop 2024-03-18T18:51:33.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.777+0000 7fe66f5cd700 1 -- 172.21.15.1:0/3060963801 shutdown_connections 2024-03-18T18:51:33.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.777+0000 7fe66f5cd700 1 --2- 172.21.15.1:0/3060963801 >> v2:172.21.15.165:3300/0 conn(0x7fe66815b540 0x7fe6680cc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:33.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.777+0000 7fe66f5cd700 1 --2- 172.21.15.1:0/3060963801 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6680ca5d0 0x7fe6680cd520 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:33.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.777+0000 7fe66f5cd700 1 -- 172.21.15.1:0/3060963801 >> 172.21.15.1:0/3060963801 conn(0x7fe6680cca00 msgr2=0x7fe6680cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:33.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.777+0000 7fe66f5cd700 1 -- 172.21.15.1:0/3060963801 shutdown_connections 2024-03-18T18:51:33.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.777+0000 7fe66f5cd700 1 -- 172.21.15.1:0/3060963801 wait complete. 2024-03-18T18:51:33.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.778+0000 7fe66f5cd700 1 Processor -- start 2024-03-18T18:51:33.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.778+0000 7fe66f5cd700 1 -- start start 2024-03-18T18:51:33.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.779+0000 7fe66f5cd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe66806a750 0x7fe6680c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:33.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.779+0000 7fe66f5cd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe66815b540 0x7fe6681f69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:33.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.779+0000 7fe66f5cd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6681521b0 0x7fe668152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:33.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.779+0000 7fe66f5cd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6680c8b50 con 0x7fe66815b540 2024-03-18T18:51:33.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.779+0000 7fe66f5cd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6680c6a70 con 0x7fe66806a750 2024-03-18T18:51:33.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.779+0000 7fe66f5cd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6680c8df0 con 0x7fe6681521b0 2024-03-18T18:51:33.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.779+0000 7fe66cb68700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe66815b540 0x7fe6681f69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:33.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.779+0000 7fe66cb68700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe66815b540 0x7fe6681f69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42026/0 (socket says 172.21.15.1:42026) 2024-03-18T18:51:33.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.779+0000 7fe66cb68700 1 -- 172.21.15.1:0/1606014418 learned_addr learned my addr 172.21.15.1:0/1606014418 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:33.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.779+0000 7fe66d369700 1 --2- 172.21.15.1:0/1606014418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe66806a750 0x7fe6680c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:33.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.779+0000 7fe66db6a700 1 --2- 172.21.15.1:0/1606014418 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6681521b0 0x7fe668152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:33.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.779+0000 7fe66cb68700 1 -- 172.21.15.1:0/1606014418 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6681521b0 msgr2=0x7fe668152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:33.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.779+0000 7fe66cb68700 1 --2- 172.21.15.1:0/1606014418 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6681521b0 0x7fe668152600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:33.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.779+0000 7fe66cb68700 1 -- 172.21.15.1:0/1606014418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe66806a750 msgr2=0x7fe6680c95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:33.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.779+0000 7fe66cb68700 1 --2- 172.21.15.1:0/1606014418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe66806a750 0x7fe6680c95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:33.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.779+0000 7fe66cb68700 1 -- 172.21.15.1:0/1606014418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe658009550 con 0x7fe66815b540 2024-03-18T18:51:33.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.780+0000 7fe66db6a700 1 --2- 172.21.15.1:0/1606014418 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6681521b0 0x7fe668152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:33.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.780+0000 7fe66cb68700 1 --2- 172.21.15.1:0/1606014418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe66815b540 0x7fe6681f69f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fe658014040 tx=0x7fe658004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:33.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.780+0000 7fe65dffb700 1 -- 172.21.15.1:0/1606014418 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe658004220 con 0x7fe66815b540 2024-03-18T18:51:33.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.780+0000 7fe65dffb700 1 -- 172.21.15.1:0/1606014418 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65801e070 con 0x7fe66815b540 2024-03-18T18:51:33.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.780+0000 7fe66f5cd700 1 -- 172.21.15.1:0/1606014418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6680c90d0 con 0x7fe66815b540 2024-03-18T18:51:33.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.780+0000 7fe65dffb700 1 -- 172.21.15.1:0/1606014418 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe658004580 con 0x7fe66815b540 2024-03-18T18:51:33.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.780+0000 7fe66f5cd700 1 -- 172.21.15.1:0/1606014418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6680c66d0 con 0x7fe66815b540 2024-03-18T18:51:33.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.781+0000 7fe66f5cd700 1 -- 172.21.15.1:0/1606014418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6680676a0 con 0x7fe66815b540 2024-03-18T18:51:33.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.782+0000 7fe65dffb700 1 -- 172.21.15.1:0/1606014418 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fe658012800 con 0x7fe66815b540 2024-03-18T18:51:33.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.782+0000 7fe65dffb700 1 --2- 172.21.15.1:0/1606014418 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe64c06aec0 0x7fe64c06d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:33.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.782+0000 7fe65dffb700 1 -- 172.21.15.1:0/1606014418 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fe65808aff0 con 0x7fe66815b540 2024-03-18T18:51:33.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.782+0000 7fe66d369700 1 --2- 172.21.15.1:0/1606014418 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe64c06aec0 0x7fe64c06d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:33.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.783+0000 7fe66d369700 1 --2- 172.21.15.1:0/1606014418 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe64c06aec0 0x7fe64c06d380 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe664000f60 tx=0x7fe664007c90 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:33.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:33.786+0000 7fe65dffb700 1 -- 172.21.15.1:0/1606014418 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe658055d50 con 0x7fe66815b540 2024-03-18T18:51:34.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.017+0000 7fe66f5cd700 1 -- 172.21.15.1:0/1606014418 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6680cc590 con 0x7fe64c06aec0 2024-03-18T18:51:34.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.068+0000 7fe65dffb700 1 -- 172.21.15.1:0/1606014418 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fe664013420 con 0x7fe64c06aec0 2024-03-18T18:51:34.065 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule activated for path /snap_test_dir1/dir1 2024-03-18T18:51:34.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.070+0000 7fe66f5cd700 1 -- 172.21.15.1:0/1606014418 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe64c06aec0 msgr2=0x7fe64c06d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:34.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.070+0000 7fe66f5cd700 1 --2- 172.21.15.1:0/1606014418 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe64c06aec0 0x7fe64c06d380 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe664000f60 tx=0x7fe664007c90 comp rx=0 tx=0).stop 2024-03-18T18:51:34.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.070+0000 7fe66f5cd700 1 -- 172.21.15.1:0/1606014418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe66815b540 msgr2=0x7fe6681f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:34.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.070+0000 7fe66f5cd700 1 --2- 172.21.15.1:0/1606014418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe66815b540 0x7fe6681f69f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fe658014040 tx=0x7fe658004750 comp rx=0 tx=0).stop 2024-03-18T18:51:34.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.070+0000 7fe66f5cd700 1 -- 172.21.15.1:0/1606014418 shutdown_connections 2024-03-18T18:51:34.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.070+0000 7fe66f5cd700 1 --2- 172.21.15.1:0/1606014418 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe64c06aec0 0x7fe64c06d380 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.070+0000 7fe66f5cd700 1 --2- 172.21.15.1:0/1606014418 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6681521b0 0x7fe668152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.070+0000 7fe66f5cd700 1 --2- 172.21.15.1:0/1606014418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe66806a750 0x7fe6680c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.070+0000 7fe66f5cd700 1 --2- 172.21.15.1:0/1606014418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe66815b540 0x7fe6681f69f0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.070+0000 7fe66f5cd700 1 -- 172.21.15.1:0/1606014418 >> 172.21.15.1:0/1606014418 conn(0x7fe6680cca00 msgr2=0x7fe6680ce7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:34.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.070+0000 7fe66f5cd700 1 -- 172.21.15.1:0/1606014418 shutdown_connections 2024-03-18T18:51:34.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.070+0000 7fe66f5cd700 1 -- 172.21.15.1:0/1606014418 wait complete. 2024-03-18T18:51:34.079 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2024-03-18T18:51:34.079 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir2 1m 2024-03-18T18:51:34.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.210+0000 7fa374cf7700 1 Processor -- start 2024-03-18T18:51:34.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.210+0000 7fa374cf7700 1 -- start start 2024-03-18T18:51:34.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.211+0000 7fa374cf7700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa3700cfa90 0x7fa37006a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:34.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.211+0000 7fa374cf7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa37006ac90 0x7fa370167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:34.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.211+0000 7fa374cf7700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3700cf600 con 0x7fa37015d5a0 2024-03-18T18:51:34.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.211+0000 7fa374cf7700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa37006b140 con 0x7fa3700cfa90 2024-03-18T18:51:34.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.211+0000 7fa374cf7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3700ce720 con 0x7fa37006ac90 2024-03-18T18:51:34.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.211+0000 7fa36ed9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa37006ac90 0x7fa370167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:34.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.211+0000 7fa36dd9b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa3700cfa90 0x7fa37006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:34.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.211+0000 7fa36e59c700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fa37015d5a0 0x7fa3700ce010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:41994/0 (socket says 172.21.15.1:41994) 2024-03-18T18:51:34.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.211+0000 7fa36e59c700 1 -- 172.21.15.1:0/3974705867 learned_addr learned my addr 172.21.15.1:0/3974705867 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:34.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.212+0000 7fa36ed9d700 1 -- 172.21.15.1:0/3974705867 >> v2:172.21.15.165:3300/0 conn(0x7fa3700cfa90 msgr2=0x7fa37006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:34.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.212+0000 7fa36ed9d700 1 --2- 172.21.15.1:0/3974705867 >> v2:172.21.15.165:3300/0 conn(0x7fa3700cfa90 0x7fa37006a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.212+0000 7fa36ed9d700 1 -- 172.21.15.1:0/3974705867 >> v1:172.21.15.165:6789/0 conn(0x7fa37015d5a0 legacy=0x7fa3700ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:34.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.212+0000 7fa36ed9d700 1 -- 172.21.15.1:0/3974705867 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3700ce9f0 con 0x7fa37006ac90 2024-03-18T18:51:34.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.212+0000 7fa36dd9b700 1 --2- 172.21.15.1:0/3974705867 >> v2:172.21.15.165:3300/0 conn(0x7fa3700cfa90 0x7fa37006a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:51:34.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.212+0000 7fa36ed9d700 1 --2- 172.21.15.1:0/3974705867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa37006ac90 0x7fa370167140 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa368009d90 tx=0x7fa36800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36a8266258403b56 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:34.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.212+0000 7fa36d59a700 1 -- 172.21.15.1:0/3974705867 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36800e070 con 0x7fa37006ac90 2024-03-18T18:51:34.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.212+0000 7fa36d59a700 1 -- 172.21.15.1:0/3974705867 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36800bb20 con 0x7fa37006ac90 2024-03-18T18:51:34.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.212+0000 7fa36d59a700 1 -- 172.21.15.1:0/3974705867 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa368012430 con 0x7fa37006ac90 2024-03-18T18:51:34.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.213+0000 7fa374cf7700 1 -- 172.21.15.1:0/3974705867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa37006ac90 msgr2=0x7fa370167140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:34.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.213+0000 7fa374cf7700 1 --2- 172.21.15.1:0/3974705867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa37006ac90 0x7fa370167140 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa368009d90 tx=0x7fa36800b330 comp rx=0 tx=0).stop 2024-03-18T18:51:34.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.213+0000 7fa374cf7700 1 -- 172.21.15.1:0/3974705867 shutdown_connections 2024-03-18T18:51:34.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.213+0000 7fa374cf7700 1 --2- 172.21.15.1:0/3974705867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa37006ac90 0x7fa370167140 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.213+0000 7fa374cf7700 1 --2- 172.21.15.1:0/3974705867 >> v2:172.21.15.165:3300/0 conn(0x7fa3700cfa90 0x7fa37006a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.213+0000 7fa374cf7700 1 -- 172.21.15.1:0/3974705867 >> 172.21.15.1:0/3974705867 conn(0x7fa3700cc9b0 msgr2=0x7fa3700ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:34.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.213+0000 7fa374cf7700 1 -- 172.21.15.1:0/3974705867 shutdown_connections 2024-03-18T18:51:34.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.213+0000 7fa374cf7700 1 -- 172.21.15.1:0/3974705867 wait complete. 2024-03-18T18:51:34.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.214+0000 7fa374cf7700 1 Processor -- start 2024-03-18T18:51:34.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.214+0000 7fa374cf7700 1 -- start start 2024-03-18T18:51:34.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.215+0000 7fa374cf7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa37006ac90 0x7fa3700c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:34.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.215+0000 7fa374cf7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3700cfa90 0x7fa3701430f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:34.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.215+0000 7fa374cf7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa37015d5a0 0x7fa370160500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:34.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.215+0000 7fa374cf7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa370157f40 con 0x7fa3700cfa90 2024-03-18T18:51:34.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.215+0000 7fa374cf7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3701581b0 con 0x7fa37015d5a0 2024-03-18T18:51:34.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.215+0000 7fa374cf7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa370158490 con 0x7fa37006ac90 2024-03-18T18:51:34.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.215+0000 7fa36dd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3700cfa90 0x7fa3701430f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:34.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.215+0000 7fa36dd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3700cfa90 0x7fa3701430f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42040/0 (socket says 172.21.15.1:42040) 2024-03-18T18:51:34.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.215+0000 7fa36dd9b700 1 -- 172.21.15.1:0/472835766 learned_addr learned my addr 172.21.15.1:0/472835766 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:34.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.215+0000 7fa36ed9d700 1 --2- 172.21.15.1:0/472835766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa37015d5a0 0x7fa370160500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:34.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.215+0000 7fa36e59c700 1 --2- 172.21.15.1:0/472835766 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa37006ac90 0x7fa3700c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:34.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.215+0000 7fa36dd9b700 1 -- 172.21.15.1:0/472835766 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa37006ac90 msgr2=0x7fa3700c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:34.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.215+0000 7fa36dd9b700 1 --2- 172.21.15.1:0/472835766 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa37006ac90 0x7fa3700c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.215+0000 7fa36dd9b700 1 -- 172.21.15.1:0/472835766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa37015d5a0 msgr2=0x7fa370160500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:34.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.215+0000 7fa36dd9b700 1 --2- 172.21.15.1:0/472835766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa37015d5a0 0x7fa370160500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.215+0000 7fa36dd9b700 1 -- 172.21.15.1:0/472835766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3600094a0 con 0x7fa3700cfa90 2024-03-18T18:51:34.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.215+0000 7fa36dd9b700 1 --2- 172.21.15.1:0/472835766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3700cfa90 0x7fa3701430f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa36000ebe0 tx=0x7fa36000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:34.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.216+0000 7fa35effd700 1 -- 172.21.15.1:0/472835766 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36000c850 con 0x7fa3700cfa90 2024-03-18T18:51:34.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.216+0000 7fa35effd700 1 -- 172.21.15.1:0/472835766 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36000c9b0 con 0x7fa3700cfa90 2024-03-18T18:51:34.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.216+0000 7fa374cf7700 1 -- 172.21.15.1:0/472835766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa368009a70 con 0x7fa3700cfa90 2024-03-18T18:51:34.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.216+0000 7fa35effd700 1 -- 172.21.15.1:0/472835766 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36000cd40 con 0x7fa3700cfa90 2024-03-18T18:51:34.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.216+0000 7fa374cf7700 1 -- 172.21.15.1:0/472835766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa370158a50 con 0x7fa3700cfa90 2024-03-18T18:51:34.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.217+0000 7fa374cf7700 1 -- 172.21.15.1:0/472835766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3700676a0 con 0x7fa3700cfa90 2024-03-18T18:51:34.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.217+0000 7fa35effd700 1 -- 172.21.15.1:0/472835766 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fa360013470 con 0x7fa3700cfa90 2024-03-18T18:51:34.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.218+0000 7fa35effd700 1 --2- 172.21.15.1:0/472835766 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa35006ae90 0x7fa35006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:34.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.218+0000 7fa36e59c700 1 --2- 172.21.15.1:0/472835766 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa35006ae90 0x7fa35006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:34.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.218+0000 7fa35effd700 1 -- 172.21.15.1:0/472835766 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fa360089990 con 0x7fa3700cfa90 2024-03-18T18:51:34.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.218+0000 7fa36e59c700 1 --2- 172.21.15.1:0/472835766 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa35006ae90 0x7fa35006d350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa364005960 tx=0x7fa364009040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:34.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.222+0000 7fa35effd700 1 -- 172.21.15.1:0/472835766 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa3600546b0 con 0x7fa3700cfa90 2024-03-18T18:51:34.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.455+0000 7fa374cf7700 1 -- 172.21.15.1:0/472835766 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa370158320 con 0x7fa35006ae90 2024-03-18T18:51:34.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.502+0000 7fa35effd700 1 -- 172.21.15.1:0/472835766 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fa370158320 con 0x7fa35006ae90 2024-03-18T18:51:34.499 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule activated for path /snap_test_dir1/dir2 2024-03-18T18:51:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.504+0000 7fa374cf7700 1 -- 172.21.15.1:0/472835766 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa35006ae90 msgr2=0x7fa35006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.504+0000 7fa374cf7700 1 --2- 172.21.15.1:0/472835766 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa35006ae90 0x7fa35006d350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa364005960 tx=0x7fa364009040 comp rx=0 tx=0).stop 2024-03-18T18:51:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.504+0000 7fa374cf7700 1 -- 172.21.15.1:0/472835766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3700cfa90 msgr2=0x7fa3701430f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.504+0000 7fa374cf7700 1 --2- 172.21.15.1:0/472835766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3700cfa90 0x7fa3701430f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa36000ebe0 tx=0x7fa36000eef0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.504+0000 7fa374cf7700 1 -- 172.21.15.1:0/472835766 shutdown_connections 2024-03-18T18:51:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.504+0000 7fa374cf7700 1 --2- 172.21.15.1:0/472835766 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa35006ae90 0x7fa35006d350 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.504+0000 7fa374cf7700 1 --2- 172.21.15.1:0/472835766 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa37006ac90 0x7fa3700c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.504+0000 7fa374cf7700 1 --2- 172.21.15.1:0/472835766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa37015d5a0 0x7fa370160500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.504+0000 7fa374cf7700 1 --2- 172.21.15.1:0/472835766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3700cfa90 0x7fa3701430f0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.504+0000 7fa374cf7700 1 -- 172.21.15.1:0/472835766 >> 172.21.15.1:0/472835766 conn(0x7fa3700cc9b0 msgr2=0x7fa3700add70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:34.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.504+0000 7fa374cf7700 1 -- 172.21.15.1:0/472835766 shutdown_connections 2024-03-18T18:51:34.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.504+0000 7fa374cf7700 1 -- 172.21.15.1:0/472835766 wait complete. 2024-03-18T18:51:34.513 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2024-03-18T18:51:34.513 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir3 1m 2024-03-18T18:51:34.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.646+0000 7f3a98bc4700 1 Processor -- start 2024-03-18T18:51:34.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.646+0000 7f3a98bc4700 1 -- start start 2024-03-18T18:51:34.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.647+0000 7f3a98bc4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3a9415d3e0 0x7f3a940d0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:34.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.647+0000 7f3a98bc4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a9415d7c0 0x7f3a940d17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:34.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.647+0000 7f3a98bc4700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a940ce1e0 con 0x7f3a940d1cf0 2024-03-18T18:51:34.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.647+0000 7f3a98bc4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a940cee60 con 0x7f3a9415d3e0 2024-03-18T18:51:34.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.647+0000 7f3a98bc4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a940d2160 con 0x7f3a9415d7c0 2024-03-18T18:51:34.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.648+0000 7f3a9259c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3a9415d3e0 0x7f3a940d0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:34.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.648+0000 7f3a9259c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3a9415d3e0 0x7f3a940d0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43638/0 (socket says 172.21.15.1:43638) 2024-03-18T18:51:34.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.648+0000 7f3a9259c700 1 -- 172.21.15.1:0/274010704 learned_addr learned my addr 172.21.15.1:0/274010704 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:34.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.648+0000 7f3a91d9b700 1 --2- 172.21.15.1:0/274010704 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a9415d7c0 0x7f3a940d17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:34.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.648+0000 7f3a9259c700 1 -- 172.21.15.1:0/274010704 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a9415d7c0 msgr2=0x7f3a940d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:34.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.648+0000 7f3a9259c700 1 --2- 172.21.15.1:0/274010704 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a9415d7c0 0x7f3a940d17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.648+0000 7f3a9259c700 1 -- 172.21.15.1:0/274010704 >> v1:172.21.15.165:6789/0 conn(0x7f3a940d1cf0 legacy=0x7f3a940b82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:34.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.648+0000 7f3a9259c700 1 -- 172.21.15.1:0/274010704 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a940ce4c0 con 0x7f3a9415d3e0 2024-03-18T18:51:34.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.648+0000 7f3a9259c700 1 --2- 172.21.15.1:0/274010704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a9415d3e0 0x7f3a940d0fb0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3a88009920 tx=0x7f3a88009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad6335636302184e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:34.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.648+0000 7f3a9159a700 1 -- 172.21.15.1:0/274010704 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a88004230 con 0x7f3a9415d3e0 2024-03-18T18:51:34.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.648+0000 7f3a9159a700 1 -- 172.21.15.1:0/274010704 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a88004390 con 0x7f3a9415d3e0 2024-03-18T18:51:34.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.648+0000 7f3a9159a700 1 -- 172.21.15.1:0/274010704 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a88009e70 con 0x7f3a9415d3e0 2024-03-18T18:51:34.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.649+0000 7f3a98bc4700 1 -- 172.21.15.1:0/274010704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a9415d3e0 msgr2=0x7f3a940d0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:34.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.649+0000 7f3a98bc4700 1 --2- 172.21.15.1:0/274010704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a9415d3e0 0x7f3a940d0fb0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3a88009920 tx=0x7f3a88009c30 comp rx=0 tx=0).stop 2024-03-18T18:51:34.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.649+0000 7f3a98bc4700 1 -- 172.21.15.1:0/274010704 shutdown_connections 2024-03-18T18:51:34.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.649+0000 7f3a98bc4700 1 --2- 172.21.15.1:0/274010704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a9415d3e0 0x7f3a940d0fb0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.649+0000 7f3a98bc4700 1 --2- 172.21.15.1:0/274010704 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a9415d7c0 0x7f3a940d17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.649+0000 7f3a98bc4700 1 -- 172.21.15.1:0/274010704 >> 172.21.15.1:0/274010704 conn(0x7f3a9409c540 msgr2=0x7f3a940d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:34.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.649+0000 7f3a98bc4700 1 -- 172.21.15.1:0/274010704 shutdown_connections 2024-03-18T18:51:34.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.649+0000 7f3a98bc4700 1 -- 172.21.15.1:0/274010704 wait complete. 2024-03-18T18:51:34.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.650+0000 7f3a98bc4700 1 Processor -- start 2024-03-18T18:51:34.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.650+0000 7f3a98bc4700 1 -- start start 2024-03-18T18:51:34.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.651+0000 7f3a98bc4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a940d1cf0 0x7f3a94153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:34.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.651+0000 7f3a98bc4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a9415d3e0 0x7f3a940c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:34.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.651+0000 7f3a98bc4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a94153660 0x7f3a940c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:34.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.651+0000 7f3a98bc4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a94152fb0 con 0x7f3a9415d3e0 2024-03-18T18:51:34.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.651+0000 7f3a98bc4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a940c7ef0 con 0x7f3a940d1cf0 2024-03-18T18:51:34.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.651+0000 7f3a98bc4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a940c81d0 con 0x7f3a94153660 2024-03-18T18:51:34.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.651+0000 7f3a91d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a9415d3e0 0x7f3a940c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:34.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.651+0000 7f3a91d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a9415d3e0 0x7f3a940c8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43648/0 (socket says 172.21.15.1:43648) 2024-03-18T18:51:34.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.651+0000 7f3a91d9b700 1 -- 172.21.15.1:0/1031565212 learned_addr learned my addr 172.21.15.1:0/1031565212 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:34.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.651+0000 7f3a9259c700 1 --2- 172.21.15.1:0/1031565212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a940d1cf0 0x7f3a94153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:34.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.651+0000 7f3a91d9b700 1 -- 172.21.15.1:0/1031565212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a94153660 msgr2=0x7f3a940c94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:34.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.651+0000 7f3a92d9d700 1 --2- 172.21.15.1:0/1031565212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a94153660 0x7f3a940c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:34.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.651+0000 7f3a91d9b700 1 --2- 172.21.15.1:0/1031565212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a94153660 0x7f3a940c94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.651+0000 7f3a91d9b700 1 -- 172.21.15.1:0/1031565212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a940d1cf0 msgr2=0x7f3a94153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:34.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.651+0000 7f3a91d9b700 1 --2- 172.21.15.1:0/1031565212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a940d1cf0 0x7f3a94153120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.651+0000 7f3a91d9b700 1 -- 172.21.15.1:0/1031565212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a88009550 con 0x7f3a9415d3e0 2024-03-18T18:51:34.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.651+0000 7f3a91d9b700 1 --2- 172.21.15.1:0/1031565212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a9415d3e0 0x7f3a940c8f90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3a8400c7f0 tx=0x7f3a8400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:34.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.652+0000 7f3a82ffd700 1 -- 172.21.15.1:0/1031565212 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a84011800 con 0x7f3a9415d3e0 2024-03-18T18:51:34.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.652+0000 7f3a82ffd700 1 -- 172.21.15.1:0/1031565212 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a84011960 con 0x7f3a9415d3e0 2024-03-18T18:51:34.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.652+0000 7f3a98bc4700 1 -- 172.21.15.1:0/1031565212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a940c84b0 con 0x7f3a9415d3e0 2024-03-18T18:51:34.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.652+0000 7f3a82ffd700 1 -- 172.21.15.1:0/1031565212 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a84011cf0 con 0x7f3a9415d3e0 2024-03-18T18:51:34.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.652+0000 7f3a98bc4700 1 -- 172.21.15.1:0/1031565212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a940c8a70 con 0x7f3a9415d3e0 2024-03-18T18:51:34.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.653+0000 7f3a98bc4700 1 -- 172.21.15.1:0/1031565212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a940c8c90 con 0x7f3a9415d3e0 2024-03-18T18:51:34.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.653+0000 7f3a82ffd700 1 -- 172.21.15.1:0/1031565212 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f3a84024480 con 0x7f3a9415d3e0 2024-03-18T18:51:34.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.654+0000 7f3a82ffd700 1 --2- 172.21.15.1:0/1031565212 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3a7406ae90 0x7f3a7406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:34.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.654+0000 7f3a9259c700 1 --2- 172.21.15.1:0/1031565212 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3a7406ae90 0x7f3a7406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:34.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.654+0000 7f3a82ffd700 1 -- 172.21.15.1:0/1031565212 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f3a84089d30 con 0x7f3a9415d3e0 2024-03-18T18:51:34.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.654+0000 7f3a9259c700 1 --2- 172.21.15.1:0/1031565212 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3a7406ae90 0x7f3a7406d350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3a88009c70 tx=0x7f3a88008030 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:34.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.658+0000 7f3a82ffd700 1 -- 172.21.15.1:0/1031565212 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3a840549a0 con 0x7f3a9415d3e0 2024-03-18T18:51:34.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.893+0000 7f3a98bc4700 1 -- 172.21.15.1:0/1031565212 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a940d14f0 con 0x7f3a7406ae90 2024-03-18T18:51:34.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.937+0000 7f3a82ffd700 1 -- 172.21.15.1:0/1031565212 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f3a88020070 con 0x7f3a7406ae90 2024-03-18T18:51:34.934 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule activated for path /snap_test_dir1/dir3 2024-03-18T18:51:34.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.939+0000 7f3a98bc4700 1 -- 172.21.15.1:0/1031565212 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3a7406ae90 msgr2=0x7f3a7406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:34.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.939+0000 7f3a98bc4700 1 --2- 172.21.15.1:0/1031565212 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3a7406ae90 0x7f3a7406d350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3a88009c70 tx=0x7f3a88008030 comp rx=0 tx=0).stop 2024-03-18T18:51:34.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.939+0000 7f3a98bc4700 1 -- 172.21.15.1:0/1031565212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a9415d3e0 msgr2=0x7f3a940c8f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:34.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.939+0000 7f3a98bc4700 1 --2- 172.21.15.1:0/1031565212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a9415d3e0 0x7f3a940c8f90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3a8400c7f0 tx=0x7f3a8400cbb0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.939+0000 7f3a98bc4700 1 -- 172.21.15.1:0/1031565212 shutdown_connections 2024-03-18T18:51:34.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.939+0000 7f3a98bc4700 1 --2- 172.21.15.1:0/1031565212 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3a7406ae90 0x7f3a7406d350 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.939+0000 7f3a98bc4700 1 --2- 172.21.15.1:0/1031565212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a94153660 0x7f3a940c94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.939+0000 7f3a98bc4700 1 --2- 172.21.15.1:0/1031565212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a940d1cf0 0x7f3a94153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.939+0000 7f3a98bc4700 1 --2- 172.21.15.1:0/1031565212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a9415d3e0 0x7f3a940c8f90 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:34.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.939+0000 7f3a98bc4700 1 -- 172.21.15.1:0/1031565212 >> 172.21.15.1:0/1031565212 conn(0x7f3a9409c540 msgr2=0x7f3a9415f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:34.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.940+0000 7f3a98bc4700 1 -- 172.21.15.1:0/1031565212 shutdown_connections 2024-03-18T18:51:34.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:34.940+0000 7f3a98bc4700 1 -- 172.21.15.1:0/1031565212 wait complete. 2024-03-18T18:51:34.948 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2024-03-18T18:51:34.948 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir4 1m 2024-03-18T18:51:35.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.079+0000 7f5c58d3e700 1 Processor -- start 2024-03-18T18:51:35.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.079+0000 7f5c58d3e700 1 -- start start 2024-03-18T18:51:35.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.080+0000 7f5c58d3e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c540b2650 0x7f5c54070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:35.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.080+0000 7f5c58d3e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5c540ac8b0 0x7f5c54071380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:35.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.080+0000 7f5c58d3e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c540b2b80 con 0x7f5c5415d3b0 2024-03-18T18:51:35.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.080+0000 7f5c58d3e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c540b3800 con 0x7f5c540ac8b0 2024-03-18T18:51:35.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.080+0000 7f5c58d3e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c540b2e60 con 0x7f5c540b2650 2024-03-18T18:51:35.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.081+0000 7f5c5259c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5c5415d3b0 0x7f5c5406cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47088/0 (socket says 172.21.15.1:47088) 2024-03-18T18:51:35.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.081+0000 7f5c5259c700 1 -- 172.21.15.1:0/535867972 learned_addr learned my addr 172.21.15.1:0/535867972 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:35.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.081+0000 7f5c52d9d700 1 --2- 172.21.15.1:0/535867972 >> v2:172.21.15.165:3300/0 conn(0x7f5c540ac8b0 0x7f5c54071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:35.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.081+0000 7f5c51d9b700 1 --2- 172.21.15.1:0/535867972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c540b2650 0x7f5c54070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:35.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.081+0000 7f5c5159a700 1 -- 172.21.15.1:0/535867972 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489398377 0 0) 0x7f5c540b2b80 con 0x7f5c5415d3b0 2024-03-18T18:51:35.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.081+0000 7f5c5159a700 1 -- 172.21.15.1:0/535867972 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c3c0035b0 con 0x7f5c5415d3b0 2024-03-18T18:51:35.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.081+0000 7f5c5159a700 1 -- 172.21.15.1:0/535867972 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2001668914 0 0) 0x7f5c3c0035b0 con 0x7f5c5415d3b0 2024-03-18T18:51:35.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.081+0000 7f5c51d9b700 1 -- 172.21.15.1:0/535867972 >> v2:172.21.15.165:3300/0 conn(0x7f5c540ac8b0 msgr2=0x7f5c54071380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:35.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.081+0000 7f5c51d9b700 1 --2- 172.21.15.1:0/535867972 >> v2:172.21.15.165:3300/0 conn(0x7f5c540ac8b0 0x7f5c54071380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.081+0000 7f5c51d9b700 1 -- 172.21.15.1:0/535867972 >> v1:172.21.15.1:6789/0 conn(0x7f5c5415d3b0 legacy=0x7f5c5406cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:35.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.081+0000 7f5c51d9b700 1 -- 172.21.15.1:0/535867972 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c540b3140 con 0x7f5c540b2650 2024-03-18T18:51:35.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.081+0000 7f5c52d9d700 1 --2- 172.21.15.1:0/535867972 >> v2:172.21.15.165:3300/0 conn(0x7f5c540ac8b0 0x7f5c54071380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T18:51:35.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.081+0000 7f5c51d9b700 1 --2- 172.21.15.1:0/535867972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c540b2650 0x7f5c54070e40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009840 tx=0x7f5c44009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e05e19078205caf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:35.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.082+0000 7f5c5159a700 1 -- 172.21.15.1:0/535867972 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c48001670 con 0x7f5c540b2650 2024-03-18T18:51:35.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.082+0000 7f5c5159a700 1 -- 172.21.15.1:0/535867972 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c44004030 con 0x7f5c540b2650 2024-03-18T18:51:35.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.082+0000 7f5c5159a700 1 -- 172.21.15.1:0/535867972 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c440043c0 con 0x7f5c540b2650 2024-03-18T18:51:35.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.082+0000 7f5c58d3e700 1 -- 172.21.15.1:0/535867972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c540b2650 msgr2=0x7f5c54070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:35.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.082+0000 7f5c58d3e700 1 --2- 172.21.15.1:0/535867972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c540b2650 0x7f5c54070e40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009840 tx=0x7f5c44009b50 comp rx=0 tx=0).stop 2024-03-18T18:51:35.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.082+0000 7f5c58d3e700 1 -- 172.21.15.1:0/535867972 shutdown_connections 2024-03-18T18:51:35.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.082+0000 7f5c58d3e700 1 --2- 172.21.15.1:0/535867972 >> v2:172.21.15.165:3300/0 conn(0x7f5c540ac8b0 0x7f5c54071380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.082+0000 7f5c58d3e700 1 --2- 172.21.15.1:0/535867972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c540b2650 0x7f5c54070e40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.082+0000 7f5c58d3e700 1 -- 172.21.15.1:0/535867972 >> 172.21.15.1:0/535867972 conn(0x7f5c540b1350 msgr2=0x7f5c540b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:35.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.082+0000 7f5c58d3e700 1 -- 172.21.15.1:0/535867972 shutdown_connections 2024-03-18T18:51:35.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.082+0000 7f5c58d3e700 1 -- 172.21.15.1:0/535867972 wait complete. 2024-03-18T18:51:35.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.083+0000 7f5c58d3e700 1 Processor -- start 2024-03-18T18:51:35.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.084+0000 7f5c58d3e700 1 -- start start 2024-03-18T18:51:35.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.084+0000 7f5c58d3e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c540ac8b0 0x7f5c54142d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:35.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.084+0000 7f5c58d3e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c540b2650 0x7f5c5415fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:35.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.084+0000 7f5c58d3e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c5415d3b0 0x7f5c54160420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:35.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.084+0000 7f5c58d3e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c54157850 con 0x7f5c540b2650 2024-03-18T18:51:35.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.084+0000 7f5c58d3e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c54143290 con 0x7f5c5415d3b0 2024-03-18T18:51:35.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.084+0000 7f5c58d3e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c54144150 con 0x7f5c540ac8b0 2024-03-18T18:51:35.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.084+0000 7f5c51d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c540b2650 0x7f5c5415fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:35.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.084+0000 7f5c51d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c540b2650 0x7f5c5415fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43656/0 (socket says 172.21.15.1:43656) 2024-03-18T18:51:35.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.084+0000 7f5c51d9b700 1 -- 172.21.15.1:0/125809340 learned_addr learned my addr 172.21.15.1:0/125809340 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:35.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.084+0000 7f5c52d9d700 1 --2- 172.21.15.1:0/125809340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c5415d3b0 0x7f5c54160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:35.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.084+0000 7f5c5259c700 1 --2- 172.21.15.1:0/125809340 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c540ac8b0 0x7f5c54142d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:35.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.085+0000 7f5c51d9b700 1 -- 172.21.15.1:0/125809340 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c540ac8b0 msgr2=0x7f5c54142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:35.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.085+0000 7f5c51d9b700 1 --2- 172.21.15.1:0/125809340 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c540ac8b0 0x7f5c54142d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.085+0000 7f5c51d9b700 1 -- 172.21.15.1:0/125809340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c5415d3b0 msgr2=0x7f5c54160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:35.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.085+0000 7f5c51d9b700 1 --2- 172.21.15.1:0/125809340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c5415d3b0 0x7f5c54160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.085+0000 7f5c51d9b700 1 -- 172.21.15.1:0/125809340 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c009a70 con 0x7f5c540b2650 2024-03-18T18:51:35.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.085+0000 7f5c51d9b700 1 --2- 172.21.15.1:0/125809340 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c540b2650 0x7f5c5415fee0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5c440090b0 tx=0x7f5c44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:35.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.085+0000 7f5c42ffd700 1 -- 172.21.15.1:0/125809340 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c44003cb0 con 0x7f5c540b2650 2024-03-18T18:51:35.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.085+0000 7f5c42ffd700 1 -- 172.21.15.1:0/125809340 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4401f070 con 0x7f5c540b2650 2024-03-18T18:51:35.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.085+0000 7f5c42ffd700 1 -- 172.21.15.1:0/125809340 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c44011870 con 0x7f5c540b2650 2024-03-18T18:51:35.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.085+0000 7f5c58d3e700 1 -- 172.21.15.1:0/125809340 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c440094a0 con 0x7f5c540b2650 2024-03-18T18:51:35.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.085+0000 7f5c58d3e700 1 -- 172.21.15.1:0/125809340 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c54144710 con 0x7f5c540b2650 2024-03-18T18:51:35.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.086+0000 7f5c58d3e700 1 -- 172.21.15.1:0/125809340 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c540676a0 con 0x7f5c540b2650 2024-03-18T18:51:35.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.086+0000 7f5c42ffd700 1 -- 172.21.15.1:0/125809340 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f5c440119d0 con 0x7f5c540b2650 2024-03-18T18:51:35.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.087+0000 7f5c42ffd700 1 --2- 172.21.15.1:0/125809340 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5c3406ae90 0x7f5c3406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:35.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.087+0000 7f5c42ffd700 1 -- 172.21.15.1:0/125809340 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f5c44021070 con 0x7f5c540b2650 2024-03-18T18:51:35.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.087+0000 7f5c5259c700 1 --2- 172.21.15.1:0/125809340 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5c3406ae90 0x7f5c3406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:35.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.087+0000 7f5c5259c700 1 --2- 172.21.15.1:0/125809340 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5c3406ae90 0x7f5c3406d350 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5c5415bb50 tx=0x7f5c48001730 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:35.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.092+0000 7f5c42ffd700 1 -- 172.21.15.1:0/125809340 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c44091080 con 0x7f5c540b2650 2024-03-18T18:51:35.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.322+0000 7f5c58d3e700 1 -- 172.21.15.1:0/125809340 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c5406d600 con 0x7f5c3406ae90 2024-03-18T18:51:35.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.375+0000 7f5c42ffd700 1 -- 172.21.15.1:0/125809340 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f5c4800f690 con 0x7f5c3406ae90 2024-03-18T18:51:35.372 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule activated for path /snap_test_dir1/dir4 2024-03-18T18:51:35.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.377+0000 7f5c58d3e700 1 -- 172.21.15.1:0/125809340 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5c3406ae90 msgr2=0x7f5c3406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:35.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.377+0000 7f5c58d3e700 1 --2- 172.21.15.1:0/125809340 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5c3406ae90 0x7f5c3406d350 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5c5415bb50 tx=0x7f5c48001730 comp rx=0 tx=0).stop 2024-03-18T18:51:35.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.377+0000 7f5c58d3e700 1 -- 172.21.15.1:0/125809340 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c540b2650 msgr2=0x7f5c5415fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:35.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.377+0000 7f5c58d3e700 1 --2- 172.21.15.1:0/125809340 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c540b2650 0x7f5c5415fee0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5c440090b0 tx=0x7f5c44004750 comp rx=0 tx=0).stop 2024-03-18T18:51:35.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.377+0000 7f5c58d3e700 1 -- 172.21.15.1:0/125809340 shutdown_connections 2024-03-18T18:51:35.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.377+0000 7f5c58d3e700 1 --2- 172.21.15.1:0/125809340 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5c3406ae90 0x7f5c3406d350 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.377+0000 7f5c58d3e700 1 --2- 172.21.15.1:0/125809340 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c540ac8b0 0x7f5c54142d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.377+0000 7f5c58d3e700 1 --2- 172.21.15.1:0/125809340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c5415d3b0 0x7f5c54160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.377+0000 7f5c58d3e700 1 --2- 172.21.15.1:0/125809340 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c540b2650 0x7f5c5415fee0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.377+0000 7f5c58d3e700 1 -- 172.21.15.1:0/125809340 >> 172.21.15.1:0/125809340 conn(0x7f5c540b1350 msgr2=0x7f5c5406f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:35.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.377+0000 7f5c58d3e700 1 -- 172.21.15.1:0/125809340 shutdown_connections 2024-03-18T18:51:35.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.377+0000 7f5c58d3e700 1 -- 172.21.15.1:0/125809340 wait complete. 2024-03-18T18:51:35.385 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2024-03-18T18:51:35.386 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir5 1m 2024-03-18T18:51:35.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.518+0000 7f324d3ba700 1 Processor -- start 2024-03-18T18:51:35.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.518+0000 7f324d3ba700 1 -- start start 2024-03-18T18:51:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.519+0000 7f324d3ba700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32480cfb50 0x7f32480d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.519+0000 7f324d3ba700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32480b3370 con 0x7f32480cee80 2024-03-18T18:51:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.519+0000 7f324d3ba700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32480cf2d0 con 0x7f32480cff30 2024-03-18T18:51:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.519+0000 7f324d3ba700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32480b2760 con 0x7f32480cfb50 2024-03-18T18:51:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.519+0000 7f3246ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32480cfb50 0x7f32480d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.519+0000 7f32477fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f32480cee80 0x7f32480b3f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47104/0 (socket says 172.21.15.1:47104) 2024-03-18T18:51:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.519+0000 7f3246ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32480cfb50 0x7f32480d3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43662/0 (socket says 172.21.15.1:43662) 2024-03-18T18:51:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.519+0000 7f32477fe700 1 -- 172.21.15.1:0/964928731 learned_addr learned my addr 172.21.15.1:0/964928731 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T18:51:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.519+0000 7f3246ffd700 1 -- 172.21.15.1:0/964928731 >> v1:172.21.15.165:6789/0 conn(0x7f32480cff30 legacy=0x7f32480b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.519+0000 7f3246ffd700 1 -- 172.21.15.1:0/964928731 >> v1:172.21.15.1:6789/0 conn(0x7f32480cee80 legacy=0x7f32480b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.519+0000 7f3246ffd700 1 -- 172.21.15.1:0/964928731 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32480b2a40 con 0x7f32480cfb50 2024-03-18T18:51:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.520+0000 7f3246ffd700 1 --2- 172.21.15.1:0/964928731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32480cfb50 0x7f32480d3c60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f323c009820 tx=0x7f323c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5132544939af11fc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.520+0000 7f3245ffb700 1 -- 172.21.15.1:0/964928731 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c005e30 con 0x7f32480cfb50 2024-03-18T18:51:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.520+0000 7f3245ffb700 1 -- 172.21.15.1:0/964928731 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f323c004030 con 0x7f32480cfb50 2024-03-18T18:51:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.520+0000 7f3245ffb700 1 -- 172.21.15.1:0/964928731 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c0043c0 con 0x7f32480cfb50 2024-03-18T18:51:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.520+0000 7f324d3ba700 1 -- 172.21.15.1:0/964928731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32480cfb50 msgr2=0x7f32480d3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.520+0000 7f324d3ba700 1 --2- 172.21.15.1:0/964928731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32480cfb50 0x7f32480d3c60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f323c009820 tx=0x7f323c009b30 comp rx=0 tx=0).stop 2024-03-18T18:51:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.520+0000 7f324d3ba700 1 -- 172.21.15.1:0/964928731 shutdown_connections 2024-03-18T18:51:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.520+0000 7f324d3ba700 1 --2- 172.21.15.1:0/964928731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32480cfb50 0x7f32480d3c60 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.520+0000 7f324d3ba700 1 -- 172.21.15.1:0/964928731 >> 172.21.15.1:0/964928731 conn(0x7f32480d3140 msgr2=0x7f32480d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.520+0000 7f324d3ba700 1 -- 172.21.15.1:0/964928731 shutdown_connections 2024-03-18T18:51:35.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.520+0000 7f324d3ba700 1 -- 172.21.15.1:0/964928731 wait complete. 2024-03-18T18:51:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.522+0000 7f324d3ba700 1 Processor -- start 2024-03-18T18:51:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.522+0000 7f324d3ba700 1 -- start start 2024-03-18T18:51:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.522+0000 7f324d3ba700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32480cee80 0x7f32481578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.522+0000 7f324d3ba700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32480cfb50 0x7f32480cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.522+0000 7f324d3ba700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32480cff30 0x7f32481f3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.522+0000 7f324d3ba700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32480cb590 con 0x7f32480cff30 2024-03-18T18:51:35.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.522+0000 7f324d3ba700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32480cb870 con 0x7f32480cee80 2024-03-18T18:51:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.522+0000 7f324d3ba700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32480cbb50 con 0x7f32480cfb50 2024-03-18T18:51:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.522+0000 7f32477fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32480cff30 0x7f32481f3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.522+0000 7f32477fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32480cff30 0x7f32481f3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43668/0 (socket says 172.21.15.1:43668) 2024-03-18T18:51:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.523+0000 7f32477fe700 1 -- 172.21.15.1:0/892673692 learned_addr learned my addr 172.21.15.1:0/892673692 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.523+0000 7f3246ffd700 1 --2- 172.21.15.1:0/892673692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32480cee80 0x7f32481578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.523+0000 7f32467fc700 1 --2- 172.21.15.1:0/892673692 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32480cfb50 0x7f32480cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.523+0000 7f32477fe700 1 -- 172.21.15.1:0/892673692 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32480cfb50 msgr2=0x7f32480cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.523+0000 7f32477fe700 1 --2- 172.21.15.1:0/892673692 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32480cfb50 0x7f32480cac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.523+0000 7f32477fe700 1 -- 172.21.15.1:0/892673692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32480cee80 msgr2=0x7f32481578d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.523+0000 7f32477fe700 1 --2- 172.21.15.1:0/892673692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32480cee80 0x7f32481578d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.523+0000 7f32477fe700 1 -- 172.21.15.1:0/892673692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323c009430 con 0x7f32480cff30 2024-03-18T18:51:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.523+0000 7f32467fc700 1 --2- 172.21.15.1:0/892673692 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32480cfb50 0x7f32480cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.523+0000 7f32477fe700 1 --2- 172.21.15.1:0/892673692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32480cff30 0x7f32481f3570 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f324000cbc0 tx=0x7f324000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.523+0000 7f322f7fe700 1 -- 172.21.15.1:0/892673692 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324000aaa0 con 0x7f32480cff30 2024-03-18T18:51:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.523+0000 7f322f7fe700 1 -- 172.21.15.1:0/892673692 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324000ac00 con 0x7f32480cff30 2024-03-18T18:51:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.523+0000 7f322f7fe700 1 -- 172.21.15.1:0/892673692 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3240012400 con 0x7f32480cff30 2024-03-18T18:51:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.523+0000 7f324d3ba700 1 -- 172.21.15.1:0/892673692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32480cbe30 con 0x7f32480cff30 2024-03-18T18:51:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.523+0000 7f324d3ba700 1 -- 172.21.15.1:0/892673692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32480cb160 con 0x7f32480cff30 2024-03-18T18:51:35.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.524+0000 7f324d3ba700 1 -- 172.21.15.1:0/892673692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32480d0390 con 0x7f32480cff30 2024-03-18T18:51:35.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.524+0000 7f322f7fe700 1 -- 172.21.15.1:0/892673692 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f324001b3b0 con 0x7f32480cff30 2024-03-18T18:51:35.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.525+0000 7f322f7fe700 1 --2- 172.21.15.1:0/892673692 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f323406ae90 0x7f323406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:35.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.525+0000 7f322f7fe700 1 -- 172.21.15.1:0/892673692 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f3240089410 con 0x7f32480cff30 2024-03-18T18:51:35.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.525+0000 7f3246ffd700 1 --2- 172.21.15.1:0/892673692 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f323406ae90 0x7f323406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:35.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.525+0000 7f3246ffd700 1 --2- 172.21.15.1:0/892673692 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f323406ae90 0x7f323406d350 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f323c000c00 tx=0x7f323c008070 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:35.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.529+0000 7f322f7fe700 1 -- 172.21.15.1:0/892673692 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f32480d0390 con 0x7f32480cff30 2024-03-18T18:51:35.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.764+0000 7f324d3ba700 1 -- 172.21.15.1:0/892673692 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f32480cb9e0 con 0x7f323406ae90 2024-03-18T18:51:35.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.817+0000 7f322f7fe700 1 -- 172.21.15.1:0/892673692 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f32480cb9e0 con 0x7f323406ae90 2024-03-18T18:51:35.814 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule activated for path /snap_test_dir1/dir5 2024-03-18T18:51:35.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.819+0000 7f324d3ba700 1 -- 172.21.15.1:0/892673692 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f323406ae90 msgr2=0x7f323406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:35.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.819+0000 7f324d3ba700 1 --2- 172.21.15.1:0/892673692 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f323406ae90 0x7f323406d350 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f323c000c00 tx=0x7f323c008070 comp rx=0 tx=0).stop 2024-03-18T18:51:35.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.819+0000 7f324d3ba700 1 -- 172.21.15.1:0/892673692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32480cff30 msgr2=0x7f32481f3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:35.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.819+0000 7f324d3ba700 1 --2- 172.21.15.1:0/892673692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32480cff30 0x7f32481f3570 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f324000cbc0 tx=0x7f324000ced0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.819+0000 7f324d3ba700 1 -- 172.21.15.1:0/892673692 shutdown_connections 2024-03-18T18:51:35.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.819+0000 7f324d3ba700 1 --2- 172.21.15.1:0/892673692 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f323406ae90 0x7f323406d350 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.819+0000 7f324d3ba700 1 --2- 172.21.15.1:0/892673692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32480cff30 0x7f32481f3570 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.819+0000 7f324d3ba700 1 --2- 172.21.15.1:0/892673692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32480cee80 0x7f32481578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.819+0000 7f324d3ba700 1 --2- 172.21.15.1:0/892673692 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32480cfb50 0x7f32480cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.819+0000 7f324d3ba700 1 -- 172.21.15.1:0/892673692 >> 172.21.15.1:0/892673692 conn(0x7f32480d3140 msgr2=0x7f324815fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:35.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.819+0000 7f324d3ba700 1 -- 172.21.15.1:0/892673692 shutdown_connections 2024-03-18T18:51:35.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.819+0000 7f324d3ba700 1 -- 172.21.15.1:0/892673692 wait complete. 2024-03-18T18:51:35.828 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2024-03-18T18:51:35.828 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir6 1m 2024-03-18T18:51:35.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.960+0000 7f7fe8c8d700 1 Processor -- start 2024-03-18T18:51:35.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.960+0000 7f7fe8c8d700 1 -- start start 2024-03-18T18:51:35.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.961+0000 7f7fe8c8d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe40cfa90 0x7f7fe406a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:35.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.961+0000 7f7fe8c8d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7fe406ac90 0x7f7fe4167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:35.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.961+0000 7f7fe8c8d700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe40cf600 con 0x7f7fe415d5a0 2024-03-18T18:51:35.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.961+0000 7f7fe8c8d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe406b140 con 0x7f7fe406ac90 2024-03-18T18:51:35.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.961+0000 7f7fe8c8d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe40ce720 con 0x7f7fe40cfa90 2024-03-18T18:51:35.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.961+0000 7f7fe2d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7fe406ac90 0x7f7fe4167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:35.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.961+0000 7f7fe2d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7fe406ac90 0x7f7fe4167140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43670/0 (socket says 172.21.15.1:43670) 2024-03-18T18:51:35.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.961+0000 7f7fe2d9d700 1 -- 172.21.15.1:0/1986510558 learned_addr learned my addr 172.21.15.1:0/1986510558 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.961+0000 7f7fe1d9b700 1 --2- 172.21.15.1:0/1986510558 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe40cfa90 0x7f7fe406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.962+0000 7f7fe2d9d700 1 -- 172.21.15.1:0/1986510558 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe40cfa90 msgr2=0x7f7fe406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.962+0000 7f7fe2d9d700 1 --2- 172.21.15.1:0/1986510558 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe40cfa90 0x7f7fe406a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.962+0000 7f7fe2d9d700 1 -- 172.21.15.1:0/1986510558 >> v1:172.21.15.165:6789/0 conn(0x7f7fe415d5a0 legacy=0x7f7fe40ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.962+0000 7f7fe2d9d700 1 -- 172.21.15.1:0/1986510558 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe40ce9f0 con 0x7f7fe406ac90 2024-03-18T18:51:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.962+0000 7f7fe1d9b700 1 --2- 172.21.15.1:0/1986510558 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe40cfa90 0x7f7fe406a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.962+0000 7f7fe2d9d700 1 --2- 172.21.15.1:0/1986510558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe406ac90 0x7f7fe4167140 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009e80 tx=0x7f7fdc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f7332b30a40f17d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.962+0000 7f7fe159a700 1 -- 172.21.15.1:0/1986510558 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc00e070 con 0x7f7fe406ac90 2024-03-18T18:51:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.962+0000 7f7fe159a700 1 -- 172.21.15.1:0/1986510558 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fdc003a20 con 0x7f7fe406ac90 2024-03-18T18:51:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.962+0000 7f7fe159a700 1 -- 172.21.15.1:0/1986510558 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc003dd0 con 0x7f7fe406ac90 2024-03-18T18:51:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.962+0000 7f7fe8c8d700 1 -- 172.21.15.1:0/1986510558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe406ac90 msgr2=0x7f7fe4167140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:35.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.962+0000 7f7fe8c8d700 1 --2- 172.21.15.1:0/1986510558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe406ac90 0x7f7fe4167140 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009e80 tx=0x7f7fdc00b330 comp rx=0 tx=0).stop 2024-03-18T18:51:35.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.962+0000 7f7fe8c8d700 1 -- 172.21.15.1:0/1986510558 shutdown_connections 2024-03-18T18:51:35.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.962+0000 7f7fe8c8d700 1 --2- 172.21.15.1:0/1986510558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe406ac90 0x7f7fe4167140 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.962+0000 7f7fe8c8d700 1 --2- 172.21.15.1:0/1986510558 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe40cfa90 0x7f7fe406a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.962+0000 7f7fe8c8d700 1 -- 172.21.15.1:0/1986510558 >> 172.21.15.1:0/1986510558 conn(0x7f7fe40cc9b0 msgr2=0x7f7fe40ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:35.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.963+0000 7f7fe8c8d700 1 -- 172.21.15.1:0/1986510558 shutdown_connections 2024-03-18T18:51:35.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.963+0000 7f7fe8c8d700 1 -- 172.21.15.1:0/1986510558 wait complete. 2024-03-18T18:51:35.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.964+0000 7f7fe8c8d700 1 Processor -- start 2024-03-18T18:51:35.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.964+0000 7f7fe8c8d700 1 -- start start 2024-03-18T18:51:35.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe8c8d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fe406ac90 0x7f7fe414e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:35.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe8c8d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe40cfa90 0x7f7fe40b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:35.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe8c8d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe415d5a0 0x7f7fe40b5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:35.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe8c8d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe414e120 con 0x7f7fe415d5a0 2024-03-18T18:51:35.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe8c8d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe41f3560 con 0x7f7fe406ac90 2024-03-18T18:51:35.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe8c8d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe41f3840 con 0x7f7fe40cfa90 2024-03-18T18:51:35.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe2d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe415d5a0 0x7f7fe40b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:35.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe2d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe415d5a0 0x7f7fe40b5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43678/0 (socket says 172.21.15.1:43678) 2024-03-18T18:51:35.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe2d9d700 1 -- 172.21.15.1:0/3003376254 learned_addr learned my addr 172.21.15.1:0/3003376254 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:35.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe259c700 1 --2- 172.21.15.1:0/3003376254 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fe406ac90 0x7f7fe414e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:35.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe1d9b700 1 --2- 172.21.15.1:0/3003376254 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe40cfa90 0x7f7fe40b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:35.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe2d9d700 1 -- 172.21.15.1:0/3003376254 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe40cfa90 msgr2=0x7f7fe40b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:35.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe2d9d700 1 --2- 172.21.15.1:0/3003376254 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe40cfa90 0x7f7fe40b4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe2d9d700 1 -- 172.21.15.1:0/3003376254 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fe406ac90 msgr2=0x7f7fe414e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:35.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe2d9d700 1 --2- 172.21.15.1:0/3003376254 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fe406ac90 0x7f7fe414e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:35.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe2d9d700 1 -- 172.21.15.1:0/3003376254 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fdc009b30 con 0x7f7fe415d5a0 2024-03-18T18:51:35.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe1d9b700 1 --2- 172.21.15.1:0/3003376254 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe40cfa90 0x7f7fe40b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:35.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe259c700 1 --2- 172.21.15.1:0/3003376254 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fe406ac90 0x7f7fe414e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T18:51:35.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe2d9d700 1 --2- 172.21.15.1:0/3003376254 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe415d5a0 0x7f7fe40b5270 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f7fe406cfd0 tx=0x7f7fdc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:35.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fdaffd700 1 -- 172.21.15.1:0/3003376254 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc004440 con 0x7f7fe415d5a0 2024-03-18T18:51:35.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fdaffd700 1 -- 172.21.15.1:0/3003376254 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fdc0045e0 con 0x7f7fe415d5a0 2024-03-18T18:51:35.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fe8c8d700 1 -- 172.21.15.1:0/3003376254 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe41f3b20 con 0x7f7fe415d5a0 2024-03-18T18:51:35.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.965+0000 7f7fdaffd700 1 -- 172.21.15.1:0/3003376254 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc0092e0 con 0x7f7fe415d5a0 2024-03-18T18:51:35.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.966+0000 7f7fe8c8d700 1 -- 172.21.15.1:0/3003376254 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe414ea80 con 0x7f7fe415d5a0 2024-03-18T18:51:35.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.967+0000 7f7fe8c8d700 1 -- 172.21.15.1:0/3003376254 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe414e260 con 0x7f7fe415d5a0 2024-03-18T18:51:35.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.967+0000 7f7fdaffd700 1 -- 172.21.15.1:0/3003376254 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f7fdc012430 con 0x7f7fe415d5a0 2024-03-18T18:51:35.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.967+0000 7f7fdaffd700 1 --2- 172.21.15.1:0/3003376254 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7fc406ae90 0x7f7fc406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:35.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.967+0000 7f7fdaffd700 1 -- 172.21.15.1:0/3003376254 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f7fdc08af10 con 0x7f7fe415d5a0 2024-03-18T18:51:35.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.967+0000 7f7fe259c700 1 --2- 172.21.15.1:0/3003376254 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7fc406ae90 0x7f7fc406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:35.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.968+0000 7f7fe259c700 1 --2- 172.21.15.1:0/3003376254 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7fc406ae90 0x7f7fc406d350 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc003f80 tx=0x7f7fcc009040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:35.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:35.972+0000 7f7fdaffd700 1 -- 172.21.15.1:0/3003376254 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7fe414e260 con 0x7f7fe415d5a0 2024-03-18T18:51:36.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.202+0000 7f7fe8c8d700 1 -- 172.21.15.1:0/3003376254 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7fe40cf290 con 0x7f7fc406ae90 2024-03-18T18:51:36.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.257+0000 7f7fdaffd700 1 -- 172.21.15.1:0/3003376254 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f7fcc003b90 con 0x7f7fc406ae90 2024-03-18T18:51:36.254 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule activated for path /snap_test_dir1/dir6 2024-03-18T18:51:36.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.259+0000 7f7fe8c8d700 1 -- 172.21.15.1:0/3003376254 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7fc406ae90 msgr2=0x7f7fc406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:36.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.259+0000 7f7fe8c8d700 1 --2- 172.21.15.1:0/3003376254 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7fc406ae90 0x7f7fc406d350 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc003f80 tx=0x7f7fcc009040 comp rx=0 tx=0).stop 2024-03-18T18:51:36.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.259+0000 7f7fe8c8d700 1 -- 172.21.15.1:0/3003376254 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe415d5a0 msgr2=0x7f7fe40b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:36.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.259+0000 7f7fe8c8d700 1 --2- 172.21.15.1:0/3003376254 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe415d5a0 0x7f7fe40b5270 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f7fe406cfd0 tx=0x7f7fdc004ad0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.259+0000 7f7fe8c8d700 1 -- 172.21.15.1:0/3003376254 shutdown_connections 2024-03-18T18:51:36.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.259+0000 7f7fe8c8d700 1 --2- 172.21.15.1:0/3003376254 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7fc406ae90 0x7f7fc406d350 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.259+0000 7f7fe8c8d700 1 --2- 172.21.15.1:0/3003376254 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe415d5a0 0x7f7fe40b5270 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.259+0000 7f7fe8c8d700 1 --2- 172.21.15.1:0/3003376254 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fe406ac90 0x7f7fe414e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.259+0000 7f7fe8c8d700 1 --2- 172.21.15.1:0/3003376254 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe40cfa90 0x7f7fe40b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.260+0000 7f7fe8c8d700 1 -- 172.21.15.1:0/3003376254 >> 172.21.15.1:0/3003376254 conn(0x7f7fe40cc9b0 msgr2=0x7f7fe40b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:36.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.260+0000 7f7fe8c8d700 1 -- 172.21.15.1:0/3003376254 shutdown_connections 2024-03-18T18:51:36.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.260+0000 7f7fe8c8d700 1 -- 172.21.15.1:0/3003376254 wait complete. 2024-03-18T18:51:36.268 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2024-03-18T18:51:36.268 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir7 1m 2024-03-18T18:51:36.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.401+0000 7f7e4115e700 1 Processor -- start 2024-03-18T18:51:36.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.401+0000 7f7e4115e700 1 -- start start 2024-03-18T18:51:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.402+0000 7f7e4115e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7e3c06a9c0 0x7f7e3c0ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.402+0000 7f7e4115e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e3c06d310 0x7f7e3c06f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.402+0000 7f7e4115e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7e3c0cc3b0 0x7f7e3c06f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.402+0000 7f7e4115e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e3c06d880 con 0x7f7e3c06a9c0 2024-03-18T18:51:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.402+0000 7f7e4115e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e3c071ea0 con 0x7f7e3c0cc3b0 2024-03-18T18:51:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.402+0000 7f7e4115e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c06f010 con 0x7f7e3c06d310 2024-03-18T18:51:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.402+0000 7f7e3ad9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7e3c06a9c0 0x7f7e3c0ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.402+0000 7f7e3ad9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7e3c06a9c0 0x7f7e3c0ca5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43688/0 (socket says 172.21.15.1:43688) 2024-03-18T18:51:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.402+0000 7f7e3ad9d700 1 -- 172.21.15.1:0/407613951 learned_addr learned my addr 172.21.15.1:0/407613951 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.402+0000 7f7e3ad9d700 1 -- 172.21.15.1:0/407613951 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e3c06d310 msgr2=0x7f7e3c06f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:36.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.402+0000 7f7e3ad9d700 1 --2- 172.21.15.1:0/407613951 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e3c06d310 0x7f7e3c06f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.402+0000 7f7e3ad9d700 1 -- 172.21.15.1:0/407613951 >> v2:172.21.15.165:3300/0 conn(0x7f7e3c0cc3b0 msgr2=0x7f7e3c06f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.402+0000 7f7e3ad9d700 1 --2- 172.21.15.1:0/407613951 >> v2:172.21.15.165:3300/0 conn(0x7f7e3c0cc3b0 0x7f7e3c06f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.402+0000 7f7e3ad9d700 1 -- 172.21.15.1:0/407613951 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c06ff20 con 0x7f7e3c06a9c0 2024-03-18T18:51:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.402+0000 7f7e3ad9d700 1 --2- 172.21.15.1:0/407613951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e3c06a9c0 0x7f7e3c0ca5c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c0711b0 tx=0x7f7e30009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1d6bddf2b3fb940 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.402+0000 7f7e39d9b700 1 -- 172.21.15.1:0/407613951 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3000b870 con 0x7f7e3c06a9c0 2024-03-18T18:51:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.402+0000 7f7e39d9b700 1 -- 172.21.15.1:0/407613951 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3000b9d0 con 0x7f7e3c06a9c0 2024-03-18T18:51:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.402+0000 7f7e39d9b700 1 -- 172.21.15.1:0/407613951 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3000bd60 con 0x7f7e3c06a9c0 2024-03-18T18:51:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.403+0000 7f7e4115e700 1 -- 172.21.15.1:0/407613951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e3c06a9c0 msgr2=0x7f7e3c0ca5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.403+0000 7f7e4115e700 1 --2- 172.21.15.1:0/407613951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e3c06a9c0 0x7f7e3c0ca5c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c0711b0 tx=0x7f7e30009bd0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.403+0000 7f7e4115e700 1 -- 172.21.15.1:0/407613951 shutdown_connections 2024-03-18T18:51:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.403+0000 7f7e4115e700 1 --2- 172.21.15.1:0/407613951 >> v2:172.21.15.165:3300/0 conn(0x7f7e3c0cc3b0 0x7f7e3c06f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.403+0000 7f7e4115e700 1 --2- 172.21.15.1:0/407613951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e3c06a9c0 0x7f7e3c0ca5c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.403+0000 7f7e4115e700 1 --2- 172.21.15.1:0/407613951 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e3c06d310 0x7f7e3c06f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.403+0000 7f7e4115e700 1 -- 172.21.15.1:0/407613951 >> 172.21.15.1:0/407613951 conn(0x7f7e3c0b1aa0 msgr2=0x7f7e3c0b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.403+0000 7f7e4115e700 1 -- 172.21.15.1:0/407613951 shutdown_connections 2024-03-18T18:51:36.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.403+0000 7f7e4115e700 1 -- 172.21.15.1:0/407613951 wait complete. 2024-03-18T18:51:36.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.405+0000 7f7e4115e700 1 Processor -- start 2024-03-18T18:51:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.405+0000 7f7e4115e700 1 -- start start 2024-03-18T18:51:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.405+0000 7f7e4115e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e3c06a9c0 0x7f7e3c1e5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.405+0000 7f7e4115e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e3c06d310 0x7f7e3c159c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.405+0000 7f7e4115e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e3c0cc3b0 0x7f7e3c1e4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.405+0000 7f7e4115e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c1e3ff0 con 0x7f7e3c06a9c0 2024-03-18T18:51:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.405+0000 7f7e4115e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c15c650 con 0x7f7e3c06d310 2024-03-18T18:51:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.405+0000 7f7e4115e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c15c930 con 0x7f7e3c0cc3b0 2024-03-18T18:51:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.405+0000 7f7e3ad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e3c06a9c0 0x7f7e3c1e5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.405+0000 7f7e3ad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e3c06a9c0 0x7f7e3c1e5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43702/0 (socket says 172.21.15.1:43702) 2024-03-18T18:51:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.405+0000 7f7e3ad9d700 1 -- 172.21.15.1:0/521652896 learned_addr learned my addr 172.21.15.1:0/521652896 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.406+0000 7f7e3a59c700 1 --2- 172.21.15.1:0/521652896 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e3c06d310 0x7f7e3c159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.406+0000 7f7e3b59e700 1 --2- 172.21.15.1:0/521652896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e3c0cc3b0 0x7f7e3c1e4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.406+0000 7f7e3ad9d700 1 -- 172.21.15.1:0/521652896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e3c0cc3b0 msgr2=0x7f7e3c1e4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:36.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.406+0000 7f7e3ad9d700 1 --2- 172.21.15.1:0/521652896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e3c0cc3b0 0x7f7e3c1e4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.406+0000 7f7e3ad9d700 1 -- 172.21.15.1:0/521652896 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e3c06d310 msgr2=0x7f7e3c159c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.406+0000 7f7e3ad9d700 1 --2- 172.21.15.1:0/521652896 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e3c06d310 0x7f7e3c159c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.406+0000 7f7e3ad9d700 1 -- 172.21.15.1:0/521652896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e30009550 con 0x7f7e3c06a9c0 2024-03-18T18:51:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.406+0000 7f7e3ad9d700 1 --2- 172.21.15.1:0/521652896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e3c06a9c0 0x7f7e3c1e5740 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c067530 tx=0x7f7e30004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.406+0000 7f7e237fe700 1 -- 172.21.15.1:0/521652896 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e30004310 con 0x7f7e3c06a9c0 2024-03-18T18:51:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.406+0000 7f7e237fe700 1 -- 172.21.15.1:0/521652896 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3001f070 con 0x7f7e3c06a9c0 2024-03-18T18:51:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.406+0000 7f7e237fe700 1 -- 172.21.15.1:0/521652896 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e300036a0 con 0x7f7e3c06a9c0 2024-03-18T18:51:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.406+0000 7f7e4115e700 1 -- 172.21.15.1:0/521652896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e3c15cc10 con 0x7f7e3c06a9c0 2024-03-18T18:51:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.406+0000 7f7e4115e700 1 -- 172.21.15.1:0/521652896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e3c15a140 con 0x7f7e3c06a9c0 2024-03-18T18:51:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.407+0000 7f7e4115e700 1 -- 172.21.15.1:0/521652896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e3c0676a0 con 0x7f7e3c06a9c0 2024-03-18T18:51:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.408+0000 7f7e237fe700 1 -- 172.21.15.1:0/521652896 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f7e30021030 con 0x7f7e3c06a9c0 2024-03-18T18:51:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.408+0000 7f7e237fe700 1 --2- 172.21.15.1:0/521652896 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7e2806ae90 0x7f7e2806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:36.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.408+0000 7f7e237fe700 1 -- 172.21.15.1:0/521652896 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f7e3008abf0 con 0x7f7e3c06a9c0 2024-03-18T18:51:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.408+0000 7f7e3a59c700 1 --2- 172.21.15.1:0/521652896 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7e2806ae90 0x7f7e2806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:36.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.408+0000 7f7e3a59c700 1 --2- 172.21.15.1:0/521652896 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7e2806ae90 0x7f7e2806d350 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c0cbd70 tx=0x7f7e2c009190 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:36.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.413+0000 7f7e237fe700 1 -- 172.21.15.1:0/521652896 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7e300559a0 con 0x7f7e3c06a9c0 2024-03-18T18:51:36.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.644+0000 7f7e4115e700 1 -- 172.21.15.1:0/521652896 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e3c15c4d0 con 0x7f7e2806ae90 2024-03-18T18:51:36.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.700+0000 7f7e237fe700 1 -- 172.21.15.1:0/521652896 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f7e2c009660 con 0x7f7e2806ae90 2024-03-18T18:51:36.697 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule activated for path /snap_test_dir1/dir7 2024-03-18T18:51:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.703+0000 7f7e4115e700 1 -- 172.21.15.1:0/521652896 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7e2806ae90 msgr2=0x7f7e2806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.703+0000 7f7e4115e700 1 --2- 172.21.15.1:0/521652896 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7e2806ae90 0x7f7e2806d350 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c0cbd70 tx=0x7f7e2c009190 comp rx=0 tx=0).stop 2024-03-18T18:51:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.703+0000 7f7e4115e700 1 -- 172.21.15.1:0/521652896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e3c06a9c0 msgr2=0x7f7e3c1e5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.703+0000 7f7e4115e700 1 --2- 172.21.15.1:0/521652896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e3c06a9c0 0x7f7e3c1e5740 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c067530 tx=0x7f7e30004750 comp rx=0 tx=0).stop 2024-03-18T18:51:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.703+0000 7f7e4115e700 1 -- 172.21.15.1:0/521652896 shutdown_connections 2024-03-18T18:51:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.703+0000 7f7e4115e700 1 --2- 172.21.15.1:0/521652896 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7e2806ae90 0x7f7e2806d350 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.703+0000 7f7e4115e700 1 --2- 172.21.15.1:0/521652896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e3c0cc3b0 0x7f7e3c1e4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.703+0000 7f7e4115e700 1 --2- 172.21.15.1:0/521652896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e3c06a9c0 0x7f7e3c1e5740 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.703+0000 7f7e4115e700 1 --2- 172.21.15.1:0/521652896 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e3c06d310 0x7f7e3c159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.703+0000 7f7e4115e700 1 -- 172.21.15.1:0/521652896 >> 172.21.15.1:0/521652896 conn(0x7f7e3c0b1aa0 msgr2=0x7f7e3c06f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.703+0000 7f7e4115e700 1 -- 172.21.15.1:0/521652896 shutdown_connections 2024-03-18T18:51:36.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.703+0000 7f7e4115e700 1 -- 172.21.15.1:0/521652896 wait complete. 2024-03-18T18:51:36.711 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2024-03-18T18:51:36.712 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir8 1m 2024-03-18T18:51:36.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.845+0000 7fcb52761700 1 Processor -- start 2024-03-18T18:51:36.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.845+0000 7fcb52761700 1 -- start start 2024-03-18T18:51:36.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.846+0000 7fcb52761700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb4c160e50 0x7fcb4c0cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:36.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.846+0000 7fcb52761700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcb4c1612d0 0x7fcb4c161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:36.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.846+0000 7fcb52761700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4c162170 con 0x7fcb4c160a70 2024-03-18T18:51:36.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.846+0000 7fcb52761700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb4c161c60 con 0x7fcb4c1612d0 2024-03-18T18:51:36.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.846+0000 7fcb52761700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c0cef30 con 0x7fcb4c160e50 2024-03-18T18:51:36.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.846+0000 7fcb50cfe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcb4c1612d0 0x7fcb4c161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.846+0000 7fcb50cfe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcb4c1612d0 0x7fcb4c161720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43706/0 (socket says 172.21.15.1:43706) 2024-03-18T18:51:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.846+0000 7fcb50cfe700 1 -- 172.21.15.1:0/2259513278 learned_addr learned my addr 172.21.15.1:0/2259513278 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.846+0000 7fcb4bfff700 1 --1- 172.21.15.1:0/2259513278 >> v1:172.21.15.1:6789/0 conn(0x7fcb4c160a70 0x7fcb4c0d3b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47114/0 (socket says 172.21.15.1:47114) 2024-03-18T18:51:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.847+0000 7fcb4b7fe700 1 --2- 172.21.15.1:0/2259513278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb4c160e50 0x7fcb4c0cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.847+0000 7fcb50cfe700 1 -- 172.21.15.1:0/2259513278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb4c160e50 msgr2=0x7fcb4c0cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.847+0000 7fcb50cfe700 1 --2- 172.21.15.1:0/2259513278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb4c160e50 0x7fcb4c0cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.847+0000 7fcb50cfe700 1 -- 172.21.15.1:0/2259513278 >> v1:172.21.15.1:6789/0 conn(0x7fcb4c160a70 legacy=0x7fcb4c0d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.847+0000 7fcb50cfe700 1 -- 172.21.15.1:0/2259513278 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c0cf210 con 0x7fcb4c1612d0 2024-03-18T18:51:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.847+0000 7fcb4affd700 1 -- 172.21.15.1:0/2259513278 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777698424 0 0) 0x7fcb3c001110 con 0x7fcb4c160a70 2024-03-18T18:51:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.847+0000 7fcb50cfe700 1 --2- 172.21.15.1:0/2259513278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb4c1612d0 0x7fcb4c161720 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009e50 tx=0x7fcb4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb5eec8edc80ef6f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.847+0000 7fcb4affd700 1 -- 172.21.15.1:0/2259513278 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4400e070 con 0x7fcb4c1612d0 2024-03-18T18:51:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.847+0000 7fcb4affd700 1 -- 172.21.15.1:0/2259513278 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4400b9c0 con 0x7fcb4c1612d0 2024-03-18T18:51:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.847+0000 7fcb4affd700 1 -- 172.21.15.1:0/2259513278 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4400bd70 con 0x7fcb4c1612d0 2024-03-18T18:51:36.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.848+0000 7fcb52761700 1 -- 172.21.15.1:0/2259513278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb4c1612d0 msgr2=0x7fcb4c161720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:36.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.848+0000 7fcb52761700 1 --2- 172.21.15.1:0/2259513278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb4c1612d0 0x7fcb4c161720 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009e50 tx=0x7fcb4400b330 comp rx=0 tx=0).stop 2024-03-18T18:51:36.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.848+0000 7fcb52761700 1 -- 172.21.15.1:0/2259513278 shutdown_connections 2024-03-18T18:51:36.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.848+0000 7fcb52761700 1 --2- 172.21.15.1:0/2259513278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb4c1612d0 0x7fcb4c161720 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.848+0000 7fcb52761700 1 --2- 172.21.15.1:0/2259513278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb4c160e50 0x7fcb4c0cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.848+0000 7fcb52761700 1 -- 172.21.15.1:0/2259513278 >> 172.21.15.1:0/2259513278 conn(0x7fcb4c09c560 msgr2=0x7fcb4c0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:36.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.848+0000 7fcb52761700 1 -- 172.21.15.1:0/2259513278 shutdown_connections 2024-03-18T18:51:36.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.848+0000 7fcb52761700 1 -- 172.21.15.1:0/2259513278 wait complete. 2024-03-18T18:51:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.849+0000 7fcb52761700 1 Processor -- start 2024-03-18T18:51:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.849+0000 7fcb52761700 1 -- start start 2024-03-18T18:51:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.850+0000 7fcb52761700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb4c160a70 0x7fcb4c154820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.850+0000 7fcb52761700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb4c160e50 0x7fcb4c0d17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.850+0000 7fcb52761700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb4c1612d0 0x7fcb4c0d1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.850+0000 7fcb52761700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c1f75f0 con 0x7fcb4c1612d0 2024-03-18T18:51:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.850+0000 7fcb52761700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c1f69e0 con 0x7fcb4c160a70 2024-03-18T18:51:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.850+0000 7fcb52761700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c1f6cc0 con 0x7fcb4c160e50 2024-03-18T18:51:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.850+0000 7fcb50cfe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb4c1612d0 0x7fcb4c0d1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.850+0000 7fcb50cfe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb4c1612d0 0x7fcb4c0d1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43722/0 (socket says 172.21.15.1:43722) 2024-03-18T18:51:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.850+0000 7fcb50cfe700 1 -- 172.21.15.1:0/4018140477 learned_addr learned my addr 172.21.15.1:0/4018140477 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.850+0000 7fcb4b7fe700 1 --2- 172.21.15.1:0/4018140477 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb4c160e50 0x7fcb4c0d17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.851+0000 7fcb4bfff700 1 --2- 172.21.15.1:0/4018140477 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb4c160a70 0x7fcb4c154820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.851+0000 7fcb50cfe700 1 -- 172.21.15.1:0/4018140477 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb4c160e50 msgr2=0x7fcb4c0d17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.851+0000 7fcb50cfe700 1 --2- 172.21.15.1:0/4018140477 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb4c160e50 0x7fcb4c0d17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.851+0000 7fcb50cfe700 1 -- 172.21.15.1:0/4018140477 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb4c160a70 msgr2=0x7fcb4c154820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:36.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.851+0000 7fcb50cfe700 1 --2- 172.21.15.1:0/4018140477 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb4c160a70 0x7fcb4c154820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:36.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.851+0000 7fcb50cfe700 1 -- 172.21.15.1:0/4018140477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb44009b30 con 0x7fcb4c1612d0 2024-03-18T18:51:36.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.851+0000 7fcb4bfff700 1 --2- 172.21.15.1:0/4018140477 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb4c160a70 0x7fcb4c154820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T18:51:36.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.851+0000 7fcb50cfe700 1 --2- 172.21.15.1:0/4018140477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb4c1612d0 0x7fcb4c0d1d00 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c06afe0 tx=0x7fcb44004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:36.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.851+0000 7fcb48ff9700 1 -- 172.21.15.1:0/4018140477 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb44003ea0 con 0x7fcb4c1612d0 2024-03-18T18:51:36.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.851+0000 7fcb48ff9700 1 -- 172.21.15.1:0/4018140477 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb440099b0 con 0x7fcb4c1612d0 2024-03-18T18:51:36.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.851+0000 7fcb52761700 1 -- 172.21.15.1:0/4018140477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4c1f6fa0 con 0x7fcb4c1612d0 2024-03-18T18:51:36.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.851+0000 7fcb48ff9700 1 -- 172.21.15.1:0/4018140477 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4401c680 con 0x7fcb4c1612d0 2024-03-18T18:51:36.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.851+0000 7fcb52761700 1 -- 172.21.15.1:0/4018140477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4c1f6670 con 0x7fcb4c1612d0 2024-03-18T18:51:36.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.852+0000 7fcb52761700 1 -- 172.21.15.1:0/4018140477 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4c0ca320 con 0x7fcb4c1612d0 2024-03-18T18:51:36.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.852+0000 7fcb48ff9700 1 -- 172.21.15.1:0/4018140477 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fcb4401c7e0 con 0x7fcb4c1612d0 2024-03-18T18:51:36.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.853+0000 7fcb48ff9700 1 --2- 172.21.15.1:0/4018140477 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcb2c06ae90 0x7fcb2c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:36.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.853+0000 7fcb48ff9700 1 -- 172.21.15.1:0/4018140477 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fcb4408b620 con 0x7fcb4c1612d0 2024-03-18T18:51:36.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.853+0000 7fcb4bfff700 1 --2- 172.21.15.1:0/4018140477 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcb2c06ae90 0x7fcb2c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:36.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.853+0000 7fcb4bfff700 1 --2- 172.21.15.1:0/4018140477 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcb2c06ae90 0x7fcb2c06d350 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c000f60 tx=0x7fcb3c007d30 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:36.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:36.857+0000 7fcb48ff9700 1 -- 172.21.15.1:0/4018140477 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcb4c0ca320 con 0x7fcb4c1612d0 2024-03-18T18:51:37.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.091+0000 7fcb52761700 1 -- 172.21.15.1:0/4018140477 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb4c0cfab0 con 0x7fcb2c06ae90 2024-03-18T18:51:37.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.145+0000 7fcb48ff9700 1 -- 172.21.15.1:0/4018140477 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fcb3c004870 con 0x7fcb2c06ae90 2024-03-18T18:51:37.142 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule activated for path /snap_test_dir1/dir8 2024-03-18T18:51:37.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.148+0000 7fcb52761700 1 -- 172.21.15.1:0/4018140477 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcb2c06ae90 msgr2=0x7fcb2c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.148+0000 7fcb52761700 1 --2- 172.21.15.1:0/4018140477 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcb2c06ae90 0x7fcb2c06d350 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c000f60 tx=0x7fcb3c007d30 comp rx=0 tx=0).stop 2024-03-18T18:51:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.148+0000 7fcb52761700 1 -- 172.21.15.1:0/4018140477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb4c1612d0 msgr2=0x7fcb4c0d1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.148+0000 7fcb52761700 1 --2- 172.21.15.1:0/4018140477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb4c1612d0 0x7fcb4c0d1d00 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c06afe0 tx=0x7fcb44004ad0 comp rx=0 tx=0).stop 2024-03-18T18:51:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.148+0000 7fcb52761700 1 -- 172.21.15.1:0/4018140477 shutdown_connections 2024-03-18T18:51:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.148+0000 7fcb52761700 1 --2- 172.21.15.1:0/4018140477 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcb2c06ae90 0x7fcb2c06d350 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.148+0000 7fcb52761700 1 --2- 172.21.15.1:0/4018140477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb4c1612d0 0x7fcb4c0d1d00 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.148+0000 7fcb52761700 1 --2- 172.21.15.1:0/4018140477 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb4c160a70 0x7fcb4c154820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.148+0000 7fcb52761700 1 --2- 172.21.15.1:0/4018140477 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb4c160e50 0x7fcb4c0d17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.148+0000 7fcb52761700 1 -- 172.21.15.1:0/4018140477 >> 172.21.15.1:0/4018140477 conn(0x7fcb4c09c560 msgr2=0x7fcb4c06b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.148+0000 7fcb52761700 1 -- 172.21.15.1:0/4018140477 shutdown_connections 2024-03-18T18:51:37.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.148+0000 7fcb52761700 1 -- 172.21.15.1:0/4018140477 wait complete. 2024-03-18T18:51:37.156 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2024-03-18T18:51:37.156 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir9 1m 2024-03-18T18:51:37.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.288+0000 7f9020eba700 1 Processor -- start 2024-03-18T18:51:37.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.289+0000 7f9020eba700 1 -- start start 2024-03-18T18:51:37.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.289+0000 7f9020eba700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f901c0ade00 0x7f901c0cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:37.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.289+0000 7f9020eba700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f901c0ccc20 0x7f901c06fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:37.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.289+0000 7f9020eba700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901c0b3aa0 con 0x7f901c099a40 2024-03-18T18:51:37.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.289+0000 7f9020eba700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f901c0ae280 con 0x7f901c0ccc20 2024-03-18T18:51:37.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.289+0000 7f9020eba700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901c06e9b0 con 0x7f901c0ade00 2024-03-18T18:51:37.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.290+0000 7f901ad9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f901c0ccc20 0x7f901c06fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:37.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.290+0000 7f901ad9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f901c0ccc20 0x7f901c06fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43728/0 (socket says 172.21.15.1:43728) 2024-03-18T18:51:37.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.290+0000 7f901ad9d700 1 -- 172.21.15.1:0/2059240676 learned_addr learned my addr 172.21.15.1:0/2059240676 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:37.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.290+0000 7f9019d9b700 1 --2- 172.21.15.1:0/2059240676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f901c0ade00 0x7f901c0cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:37.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.290+0000 7f901ad9d700 1 -- 172.21.15.1:0/2059240676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f901c0ade00 msgr2=0x7f901c0cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:37.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.290+0000 7f901ad9d700 1 --2- 172.21.15.1:0/2059240676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f901c0ade00 0x7f901c0cc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:37.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.290+0000 7f901ad9d700 1 -- 172.21.15.1:0/2059240676 >> v1:172.21.15.165:6789/0 conn(0x7f901c099a40 legacy=0x7f901c099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T18:51:37.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.290+0000 7f901ad9d700 1 -- 172.21.15.1:0/2059240676 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901c06ec90 con 0x7f901c0ccc20 2024-03-18T18:51:37.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.290+0000 7f901ad9d700 1 --2- 172.21.15.1:0/2059240676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c0ccc20 0x7f901c06fa50 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9014009e50 tx=0x7f901400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89980f6895db53a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:37.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.290+0000 7f901959a700 1 -- 172.21.15.1:0/2059240676 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901400e070 con 0x7f901c0ccc20 2024-03-18T18:51:37.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.290+0000 7f901959a700 1 -- 172.21.15.1:0/2059240676 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f901400b9c0 con 0x7f901c0ccc20 2024-03-18T18:51:37.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.290+0000 7f901959a700 1 -- 172.21.15.1:0/2059240676 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901400bd70 con 0x7f901c0ccc20 2024-03-18T18:51:37.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.291+0000 7f9020eba700 1 -- 172.21.15.1:0/2059240676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c0ccc20 msgr2=0x7f901c06fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:37.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.291+0000 7f9020eba700 1 --2- 172.21.15.1:0/2059240676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c0ccc20 0x7f901c06fa50 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9014009e50 tx=0x7f901400b330 comp rx=0 tx=0).stop 2024-03-18T18:51:37.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.291+0000 7f9020eba700 1 -- 172.21.15.1:0/2059240676 shutdown_connections 2024-03-18T18:51:37.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.291+0000 7f9020eba700 1 --2- 172.21.15.1:0/2059240676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c0ccc20 0x7f901c06fa50 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:37.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.291+0000 7f9020eba700 1 --2- 172.21.15.1:0/2059240676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f901c0ade00 0x7f901c0cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:37.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.291+0000 7f9020eba700 1 -- 172.21.15.1:0/2059240676 >> 172.21.15.1:0/2059240676 conn(0x7f901c0b2c50 msgr2=0x7f901c0b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:37.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.291+0000 7f9020eba700 1 -- 172.21.15.1:0/2059240676 shutdown_connections 2024-03-18T18:51:37.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.291+0000 7f9020eba700 1 -- 172.21.15.1:0/2059240676 wait complete. 2024-03-18T18:51:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.292+0000 7f9020eba700 1 Processor -- start 2024-03-18T18:51:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.292+0000 7f9020eba700 1 -- start start 2024-03-18T18:51:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.293+0000 7f9020eba700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f901c099a40 0x7f901c0b4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.293+0000 7f9020eba700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c0ade00 0x7f901c0c4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.293+0000 7f9020eba700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f901c0ccc20 0x7f901c0c5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.293+0000 7f9020eba700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901c14b290 con 0x7f901c0ade00 2024-03-18T18:51:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.293+0000 7f9020eba700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901c0b51c0 con 0x7f901c0ccc20 2024-03-18T18:51:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.293+0000 7f9020eba700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901c0b5460 con 0x7f901c099a40 2024-03-18T18:51:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.293+0000 7f9019d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c0ade00 0x7f901c0c4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.293+0000 7f9019d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c0ade00 0x7f901c0c4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43744/0 (socket says 172.21.15.1:43744) 2024-03-18T18:51:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.293+0000 7f9019d9b700 1 -- 172.21.15.1:0/283506913 learned_addr learned my addr 172.21.15.1:0/283506913 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T18:51:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.293+0000 7f901ad9d700 1 --2- 172.21.15.1:0/283506913 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f901c0ccc20 0x7f901c0c5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.293+0000 7f901a59c700 1 --2- 172.21.15.1:0/283506913 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f901c099a40 0x7f901c0b4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.294+0000 7f9019d9b700 1 -- 172.21.15.1:0/283506913 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f901c099a40 msgr2=0x7f901c0b4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.294+0000 7f9019d9b700 1 --2- 172.21.15.1:0/283506913 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f901c099a40 0x7f901c0b4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.294+0000 7f9019d9b700 1 -- 172.21.15.1:0/283506913 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f901c0ccc20 msgr2=0x7f901c0c5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.294+0000 7f9019d9b700 1 --2- 172.21.15.1:0/283506913 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f901c0ccc20 0x7f901c0c5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.294+0000 7f9019d9b700 1 -- 172.21.15.1:0/283506913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9014009b30 con 0x7f901c0ade00 2024-03-18T18:51:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.294+0000 7f9019d9b700 1 --2- 172.21.15.1:0/283506913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c0ade00 0x7f901c0c4c50 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f900c00e810 tx=0x7f900c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.294+0000 7f900affd700 1 -- 172.21.15.1:0/283506913 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c00c070 con 0x7f901c0ade00 2024-03-18T18:51:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.294+0000 7f900affd700 1 -- 172.21.15.1:0/283506913 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900c013610 con 0x7f901c0ade00 2024-03-18T18:51:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.294+0000 7f9020eba700 1 -- 172.21.15.1:0/283506913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f901c0b5740 con 0x7f901c0ade00 2024-03-18T18:51:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.294+0000 7f900affd700 1 -- 172.21.15.1:0/283506913 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c0139c0 con 0x7f901c0ade00 2024-03-18T18:51:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.294+0000 7f9020eba700 1 -- 172.21.15.1:0/283506913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f901c0b5d00 con 0x7f901c0ade00 2024-03-18T18:51:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.295+0000 7f9020eba700 1 -- 172.21.15.1:0/283506913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f901c0676a0 con 0x7f901c0ade00 2024-03-18T18:51:37.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.296+0000 7f900affd700 1 -- 172.21.15.1:0/283506913 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 10) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f900c013b20 con 0x7f901c0ade00 2024-03-18T18:51:37.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.296+0000 7f900affd700 1 --2- 172.21.15.1:0/283506913 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8ffc06ae90 0x7f8ffc06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T18:51:37.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.297+0000 7f901a59c700 1 --2- 172.21.15.1:0/283506913 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8ffc06ae90 0x7f8ffc06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T18:51:37.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.297+0000 7f900affd700 1 -- 172.21.15.1:0/283506913 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f900c089cc0 con 0x7f901c0ade00 2024-03-18T18:51:37.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.297+0000 7f901a59c700 1 --2- 172.21.15.1:0/283506913 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8ffc06ae90 0x7f8ffc06d350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9010007fd0 tx=0x7f9010009040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T18:51:37.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.301+0000 7f900affd700 1 -- 172.21.15.1:0/283506913 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f900c054960 con 0x7f901c0ade00 2024-03-18T18:51:37.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.533+0000 7f9020eba700 1 -- 172.21.15.1:0/283506913 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f901c06f530 con 0x7f8ffc06ae90 2024-03-18T18:51:37.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.582+0000 7f900affd700 1 -- 172.21.15.1:0/283506913 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f9010001d60 con 0x7f8ffc06ae90 2024-03-18T18:51:37.579 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule activated for path /snap_test_dir1/dir9 2024-03-18T18:51:37.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.584+0000 7f9020eba700 1 -- 172.21.15.1:0/283506913 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8ffc06ae90 msgr2=0x7f8ffc06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:37.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.584+0000 7f9020eba700 1 --2- 172.21.15.1:0/283506913 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8ffc06ae90 0x7f8ffc06d350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9010007fd0 tx=0x7f9010009040 comp rx=0 tx=0).stop 2024-03-18T18:51:37.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.584+0000 7f9020eba700 1 -- 172.21.15.1:0/283506913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c0ade00 msgr2=0x7f901c0c4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T18:51:37.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.584+0000 7f9020eba700 1 --2- 172.21.15.1:0/283506913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c0ade00 0x7f901c0c4c50 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f900c00e810 tx=0x7f900c00eb20 comp rx=0 tx=0).stop 2024-03-18T18:51:37.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.584+0000 7f9020eba700 1 -- 172.21.15.1:0/283506913 shutdown_connections 2024-03-18T18:51:37.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.584+0000 7f9020eba700 1 --2- 172.21.15.1:0/283506913 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8ffc06ae90 0x7f8ffc06d350 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:37.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.584+0000 7f9020eba700 1 --2- 172.21.15.1:0/283506913 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f901c099a40 0x7f901c0b4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:37.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.584+0000 7f9020eba700 1 --2- 172.21.15.1:0/283506913 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f901c0ccc20 0x7f901c0c5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:37.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.584+0000 7f9020eba700 1 --2- 172.21.15.1:0/283506913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f901c0ade00 0x7f901c0c4c50 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T18:51:37.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.584+0000 7f9020eba700 1 -- 172.21.15.1:0/283506913 >> 172.21.15.1:0/283506913 conn(0x7f901c0b2c50 msgr2=0x7f901c0cdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T18:51:37.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.584+0000 7f9020eba700 1 -- 172.21.15.1:0/283506913 shutdown_connections 2024-03-18T18:51:37.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T18:51:37.584+0000 7f9020eba700 1 -- 172.21.15.1:0/283506913 wait complete. 2024-03-18T18:51:37.593 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2024-03-18T18:51:48.111 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:51:48.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:52:00.075 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:52:00.077+0000 7f9045d96700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:52:00.075 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:52:00.077+0000 7f9045d96700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:52:00.075 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:52:00.077+0000 7f9045d96700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:52:00.075 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:52:00.077+0000 7f9045d96700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:52:00.075 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:52:00.077+0000 7f9045d96700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:52:18.196 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:52:18.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:52:48.291 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:52:48.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:53:18.358 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:53:18.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:53:48.424 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:53:48.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:54:18.487 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:54:18.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:54:48.568 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:54:48.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:55:18.632 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:55:18.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:55:48.713 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:55:48.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:56:00.814 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:56:00.818+0000 7f903e507700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:56:00.815 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:56:00.818+0000 7f903e507700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:56:00.815 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:56:00.818+0000 7f903e507700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:56:00.815 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:56:00.818+0000 7f903e507700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:56:00.815 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T18:56:00.818+0000 7f903e507700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T18:56:18.777 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:56:18.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:56:48.849 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:56:48.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:57:18.921 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:57:18.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:57:48.991 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:57:48.994 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:58:19.062 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:58:19.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:58:49.408 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:58:49.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:59:19.537 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:59:19.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:59:49.683 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T18:59:49.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:00:19.787 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:00:19.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:00:49.916 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:00:49.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:01:20.087 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:01:20.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:01:50.223 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:01:50.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:02:19.594 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir0 1m 2024-03-18T19:02:19.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.741+0000 7f6d41c74700 1 Processor -- start 2024-03-18T19:02:19.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.741+0000 7f6d41c74700 1 -- start start 2024-03-18T19:02:19.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.742+0000 7f6d41c74700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6d3c15b480 0x7f6d3c0cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:19.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.742+0000 7f6d41c74700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6d3c0cfdd0 0x7f6d3c0c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:19.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.742+0000 7f6d41c74700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d3c0c4c50 con 0x7f6d3c0cc7b0 2024-03-18T19:02:19.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.742+0000 7f6d41c74700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d3c0c5210 con 0x7f6d3c0cfdd0 2024-03-18T19:02:19.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.742+0000 7f6d41c74700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d3c0cccb0 con 0x7f6d3c15b480 2024-03-18T19:02:19.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.742+0000 7f6d3bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6d3c0cfdd0 0x7f6d3c0c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:19.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.742+0000 7f6d3affd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6d3c0cc7b0 0x7f6d3c0cf6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32868/0 (socket says 172.21.15.1:32868) 2024-03-18T19:02:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.742+0000 7f6d3affd700 1 -- 172.21.15.1:0/245606072 learned_addr learned my addr 172.21.15.1:0/245606072 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.742+0000 7f6d3b7fe700 1 --2- 172.21.15.1:0/245606072 >> v2:172.21.15.165:3300/0 conn(0x7f6d3c15b480 0x7f6d3c0cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.742+0000 7f6d3bfff700 1 -- 172.21.15.1:0/245606072 >> v2:172.21.15.165:3300/0 conn(0x7f6d3c15b480 msgr2=0x7f6d3c0cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.742+0000 7f6d3bfff700 1 --2- 172.21.15.1:0/245606072 >> v2:172.21.15.165:3300/0 conn(0x7f6d3c15b480 0x7f6d3c0cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.742+0000 7f6d3bfff700 1 -- 172.21.15.1:0/245606072 >> v1:172.21.15.1:6789/0 conn(0x7f6d3c0cc7b0 legacy=0x7f6d3c0cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.742+0000 7f6d3bfff700 1 -- 172.21.15.1:0/245606072 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3c0ccf90 con 0x7f6d3c0cfdd0 2024-03-18T19:02:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.742+0000 7f6d3b7fe700 1 --2- 172.21.15.1:0/245606072 >> v2:172.21.15.165:3300/0 conn(0x7f6d3c15b480 0x7f6d3c0cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.742+0000 7f6d3a7fc700 1 -- 172.21.15.1:0/245606072 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335221968 0 0) 0x7f6d3c0c4c50 con 0x7f6d3c0cc7b0 2024-03-18T19:02:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.742+0000 7f6d3bfff700 1 --2- 172.21.15.1:0/245606072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d3c0cfdd0 0x7f6d3c0c56a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c0d07c0 tx=0x7f6d3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da2a3488a9250a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.742+0000 7f6d3a7fc700 1 -- 172.21.15.1:0/245606072 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3400e070 con 0x7f6d3c0cfdd0 2024-03-18T19:02:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.742+0000 7f6d3a7fc700 1 -- 172.21.15.1:0/245606072 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d3400bbd0 con 0x7f6d3c0cfdd0 2024-03-18T19:02:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.743+0000 7f6d3a7fc700 1 -- 172.21.15.1:0/245606072 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d34012430 con 0x7f6d3c0cfdd0 2024-03-18T19:02:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.743+0000 7f6d41c74700 1 -- 172.21.15.1:0/245606072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d3c0cfdd0 msgr2=0x7f6d3c0c56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.743+0000 7f6d41c74700 1 --2- 172.21.15.1:0/245606072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d3c0cfdd0 0x7f6d3c0c56a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c0d07c0 tx=0x7f6d3400b330 comp rx=0 tx=0).stop 2024-03-18T19:02:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.743+0000 7f6d41c74700 1 -- 172.21.15.1:0/245606072 shutdown_connections 2024-03-18T19:02:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.743+0000 7f6d41c74700 1 --2- 172.21.15.1:0/245606072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d3c0cfdd0 0x7f6d3c0c56a0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.743+0000 7f6d41c74700 1 --2- 172.21.15.1:0/245606072 >> v2:172.21.15.165:3300/0 conn(0x7f6d3c15b480 0x7f6d3c0cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.743+0000 7f6d41c74700 1 -- 172.21.15.1:0/245606072 >> 172.21.15.1:0/245606072 conn(0x7f6d3c0b2bf0 msgr2=0x7f6d3c0b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.743+0000 7f6d41c74700 1 -- 172.21.15.1:0/245606072 shutdown_connections 2024-03-18T19:02:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.743+0000 7f6d41c74700 1 -- 172.21.15.1:0/245606072 wait complete. 2024-03-18T19:02:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.745+0000 7f6d41c74700 1 Processor -- start 2024-03-18T19:02:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.745+0000 7f6d41c74700 1 -- start start 2024-03-18T19:02:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d41c74700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d3c0cc7b0 0x7f6d3c14b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d41c74700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6d3c0cfdd0 0x7f6d3c0c9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d41c74700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d3c15b480 0x7f6d3c0c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d41c74700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c1f3b30 con 0x7f6d3c0cc7b0 2024-03-18T19:02:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d41c74700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c15e8a0 con 0x7f6d3c15b480 2024-03-18T19:02:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d41c74700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c1f3d80 con 0x7f6d3c0cfdd0 2024-03-18T19:02:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d3b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d3c0cc7b0 0x7f6d3c14b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d3b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d3c0cc7b0 0x7f6d3c14b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55390/0 (socket says 172.21.15.1:55390) 2024-03-18T19:02:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d3b7fe700 1 -- 172.21.15.1:0/1311174719 learned_addr learned my addr 172.21.15.1:0/1311174719 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d3bfff700 1 --2- 172.21.15.1:0/1311174719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d3c15b480 0x7f6d3c0c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d3affd700 1 --2- 172.21.15.1:0/1311174719 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6d3c0cfdd0 0x7f6d3c0c9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d3b7fe700 1 -- 172.21.15.1:0/1311174719 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6d3c0cfdd0 msgr2=0x7f6d3c0c9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d3b7fe700 1 --2- 172.21.15.1:0/1311174719 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6d3c0cfdd0 0x7f6d3c0c9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d3b7fe700 1 -- 172.21.15.1:0/1311174719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d3c15b480 msgr2=0x7f6d3c0c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:19.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d3b7fe700 1 --2- 172.21.15.1:0/1311174719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d3c15b480 0x7f6d3c0c70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d3b7fe700 1 -- 172.21.15.1:0/1311174719 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d34009a70 con 0x7f6d3c0cc7b0 2024-03-18T19:02:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d3affd700 1 --2- 172.21.15.1:0/1311174719 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6d3c0cfdd0 0x7f6d3c0c9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:02:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d3b7fe700 1 --2- 172.21.15.1:0/1311174719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d3c0cc7b0 0x7f6d3c14b6d0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c00e810 tx=0x7f6d2c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d23fff700 1 -- 172.21.15.1:0/1311174719 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c004230 con 0x7f6d3c0cc7b0 2024-03-18T19:02:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d23fff700 1 -- 172.21.15.1:0/1311174719 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2c004390 con 0x7f6d3c0cc7b0 2024-03-18T19:02:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d41c74700 1 -- 172.21.15.1:0/1311174719 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d3c1f4060 con 0x7f6d3c0cc7b0 2024-03-18T19:02:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.746+0000 7f6d23fff700 1 -- 172.21.15.1:0/1311174719 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c0036a0 con 0x7f6d3c0cc7b0 2024-03-18T19:02:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.747+0000 7f6d41c74700 1 -- 172.21.15.1:0/1311174719 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d3c1f4620 con 0x7f6d3c0cc7b0 2024-03-18T19:02:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.747+0000 7f6d41c74700 1 -- 172.21.15.1:0/1311174719 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d3c1f4990 con 0x7f6d3c0cc7b0 2024-03-18T19:02:19.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.748+0000 7f6d23fff700 1 -- 172.21.15.1:0/1311174719 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f6d2c003800 con 0x7f6d3c0cc7b0 2024-03-18T19:02:19.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.748+0000 7f6d23fff700 1 --2- 172.21.15.1:0/1311174719 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6d2806ae90 0x7f6d2806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:19.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.749+0000 7f6d3affd700 1 --2- 172.21.15.1:0/1311174719 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6d2806ae90 0x7f6d2806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:19.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.749+0000 7f6d23fff700 1 -- 172.21.15.1:0/1311174719 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f6d2c0894a0 con 0x7f6d3c0cc7b0 2024-03-18T19:02:19.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.749+0000 7f6d3affd700 1 --2- 172.21.15.1:0/1311174719 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6d2806ae90 0x7f6d2806d350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6d30007fd0 tx=0x7f6d30009040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:19.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.753+0000 7f6d23fff700 1 -- 172.21.15.1:0/1311174719 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6d2c054120 con 0x7f6d3c0cc7b0 2024-03-18T19:02:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:19.983+0000 7f6d41c74700 1 -- 172.21.15.1:0/1311174719 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d3c1f3ef0 con 0x7f6d2806ae90 2024-03-18T19:02:20.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.053+0000 7f6d23fff700 1 -- 172.21.15.1:0/1311174719 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f6d3c1f3ef0 con 0x7f6d2806ae90 2024-03-18T19:02:20.050 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2024-03-18T19:02:20.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.055+0000 7f6d41c74700 1 -- 172.21.15.1:0/1311174719 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6d2806ae90 msgr2=0x7f6d2806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:20.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.055+0000 7f6d41c74700 1 --2- 172.21.15.1:0/1311174719 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6d2806ae90 0x7f6d2806d350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6d30007fd0 tx=0x7f6d30009040 comp rx=0 tx=0).stop 2024-03-18T19:02:20.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.055+0000 7f6d41c74700 1 -- 172.21.15.1:0/1311174719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d3c0cc7b0 msgr2=0x7f6d3c14b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:20.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.055+0000 7f6d41c74700 1 --2- 172.21.15.1:0/1311174719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d3c0cc7b0 0x7f6d3c14b6d0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c00e810 tx=0x7f6d2c00ebd0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.055+0000 7f6d41c74700 1 -- 172.21.15.1:0/1311174719 shutdown_connections 2024-03-18T19:02:20.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.055+0000 7f6d41c74700 1 --2- 172.21.15.1:0/1311174719 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6d2806ae90 0x7f6d2806d350 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.055+0000 7f6d41c74700 1 --2- 172.21.15.1:0/1311174719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d3c15b480 0x7f6d3c0c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.055+0000 7f6d41c74700 1 --2- 172.21.15.1:0/1311174719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6d3c0cc7b0 0x7f6d3c14b6d0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.055+0000 7f6d41c74700 1 --2- 172.21.15.1:0/1311174719 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6d3c0cfdd0 0x7f6d3c0c9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.055+0000 7f6d41c74700 1 -- 172.21.15.1:0/1311174719 >> 172.21.15.1:0/1311174719 conn(0x7f6d3c0b2bf0 msgr2=0x7f6d3c0b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:20.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.055+0000 7f6d41c74700 1 -- 172.21.15.1:0/1311174719 shutdown_connections 2024-03-18T19:02:20.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.055+0000 7f6d41c74700 1 -- 172.21.15.1:0/1311174719 wait complete. 2024-03-18T19:02:20.064 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2024-03-18T19:02:20.065 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir1 1m 2024-03-18T19:02:20.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.195+0000 7fe10366a700 1 Processor -- start 2024-03-18T19:02:20.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.196+0000 7fe10366a700 1 -- start start 2024-03-18T19:02:20.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.196+0000 7fe10366a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0fc0cfb50 0x7fe0fc0d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:20.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.196+0000 7fe10366a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe0fc0cff30 0x7fe0fc0b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:20.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.196+0000 7fe10366a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0fc0b3370 con 0x7fe0fc0b3d40 2024-03-18T19:02:20.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.196+0000 7fe10366a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0fc0b2760 con 0x7fe0fc0cff30 2024-03-18T19:02:20.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.196+0000 7fe10366a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc0b2a40 con 0x7fe0fc0cfb50 2024-03-18T19:02:20.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.196+0000 7fe100c05700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe0fc0cff30 0x7fe0fc0b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:20.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.196+0000 7fe100c05700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe0fc0cff30 0x7fe0fc0b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55392/0 (socket says 172.21.15.1:55392) 2024-03-18T19:02:20.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.197+0000 7fe100c05700 1 -- 172.21.15.1:0/3896054895 learned_addr learned my addr 172.21.15.1:0/3896054895 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:20.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.197+0000 7fe101406700 1 --2- 172.21.15.1:0/3896054895 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0fc0cfb50 0x7fe0fc0d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:20.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.197+0000 7fe100c05700 1 -- 172.21.15.1:0/3896054895 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0fc0cfb50 msgr2=0x7fe0fc0d3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:20.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.197+0000 7fe100c05700 1 --2- 172.21.15.1:0/3896054895 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0fc0cfb50 0x7fe0fc0d3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.197+0000 7fe100c05700 1 -- 172.21.15.1:0/3896054895 >> v1:172.21.15.165:6789/0 conn(0x7fe0fc0b3d40 legacy=0x7fe0fc0b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:20.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.197+0000 7fe100c05700 1 -- 172.21.15.1:0/3896054895 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0fc0b2d20 con 0x7fe0fc0cff30 2024-03-18T19:02:20.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.197+0000 7fe101406700 1 --2- 172.21.15.1:0/3896054895 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0fc0cfb50 0x7fe0fc0d3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:20.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.197+0000 7fe100c05700 1 --2- 172.21.15.1:0/3896054895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0fc0cff30 0x7fe0fc0b3800 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe0f80098f0 tx=0x7fe0f8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=729538b0fb26b3f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:20.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.197+0000 7fe0f3fff700 1 -- 172.21.15.1:0/3896054895 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f800b8a0 con 0x7fe0fc0cff30 2024-03-18T19:02:20.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.197+0000 7fe0f3fff700 1 -- 172.21.15.1:0/3896054895 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f800ba00 con 0x7fe0fc0cff30 2024-03-18T19:02:20.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.197+0000 7fe0f3fff700 1 -- 172.21.15.1:0/3896054895 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f800bd90 con 0x7fe0fc0cff30 2024-03-18T19:02:20.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.198+0000 7fe10366a700 1 -- 172.21.15.1:0/3896054895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0fc0cff30 msgr2=0x7fe0fc0b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:20.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.198+0000 7fe10366a700 1 --2- 172.21.15.1:0/3896054895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0fc0cff30 0x7fe0fc0b3800 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe0f80098f0 tx=0x7fe0f8009c00 comp rx=0 tx=0).stop 2024-03-18T19:02:20.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.198+0000 7fe10366a700 1 -- 172.21.15.1:0/3896054895 shutdown_connections 2024-03-18T19:02:20.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.198+0000 7fe10366a700 1 --2- 172.21.15.1:0/3896054895 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0fc0cfb50 0x7fe0fc0d3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.198+0000 7fe10366a700 1 --2- 172.21.15.1:0/3896054895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0fc0cff30 0x7fe0fc0b3800 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.198+0000 7fe10366a700 1 -- 172.21.15.1:0/3896054895 >> 172.21.15.1:0/3896054895 conn(0x7fe0fc0d3140 msgr2=0x7fe0fc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:20.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.198+0000 7fe10366a700 1 -- 172.21.15.1:0/3896054895 shutdown_connections 2024-03-18T19:02:20.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.198+0000 7fe10366a700 1 -- 172.21.15.1:0/3896054895 wait complete. 2024-03-18T19:02:20.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.199+0000 7fe10366a700 1 Processor -- start 2024-03-18T19:02:20.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.199+0000 7fe10366a700 1 -- start start 2024-03-18T19:02:20.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe10366a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0fc0b3d40 0x7fe0fc0c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:20.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe10366a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0fc0cfb50 0x7fe0fc0c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:20.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe10366a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0fc0cff30 0x7fe0fc149f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:20.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe10366a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc0c7360 con 0x7fe0fc0cff30 2024-03-18T19:02:20.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe10366a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc0c7640 con 0x7fe0fc0cfb50 2024-03-18T19:02:20.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe10366a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc0c7920 con 0x7fe0fc0b3d40 2024-03-18T19:02:20.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe101c07700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0fc0cff30 0x7fe0fc149f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:20.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe101c07700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0fc0cff30 0x7fe0fc149f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55398/0 (socket says 172.21.15.1:55398) 2024-03-18T19:02:20.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe101c07700 1 -- 172.21.15.1:0/1043544689 learned_addr learned my addr 172.21.15.1:0/1043544689 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe101406700 1 --2- 172.21.15.1:0/1043544689 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0fc0b3d40 0x7fe0fc0c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe100c05700 1 --2- 172.21.15.1:0/1043544689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0fc0cfb50 0x7fe0fc0c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe101c07700 1 -- 172.21.15.1:0/1043544689 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0fc0b3d40 msgr2=0x7fe0fc0c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe101c07700 1 --2- 172.21.15.1:0/1043544689 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0fc0b3d40 0x7fe0fc0c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe101c07700 1 -- 172.21.15.1:0/1043544689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0fc0cfb50 msgr2=0x7fe0fc0c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe101c07700 1 --2- 172.21.15.1:0/1043544689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0fc0cfb50 0x7fe0fc0c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe101c07700 1 -- 172.21.15.1:0/1043544689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f8009550 con 0x7fe0fc0cff30 2024-03-18T19:02:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe101c07700 1 --2- 172.21.15.1:0/1043544689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0fc0cff30 0x7fe0fc149f90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fe0f400cc50 tx=0x7fe0f400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe0f1ffb700 1 -- 172.21.15.1:0/1043544689 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f4003c00 con 0x7fe0fc0cff30 2024-03-18T19:02:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe0f1ffb700 1 -- 172.21.15.1:0/1043544689 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f4003d60 con 0x7fe0fc0cff30 2024-03-18T19:02:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe10366a700 1 -- 172.21.15.1:0/1043544689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0fc0c7c00 con 0x7fe0fc0cff30 2024-03-18T19:02:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.200+0000 7fe0f1ffb700 1 -- 172.21.15.1:0/1043544689 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f400aab0 con 0x7fe0fc0cff30 2024-03-18T19:02:20.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.201+0000 7fe10366a700 1 -- 172.21.15.1:0/1043544689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0fc0c6fe0 con 0x7fe0fc0cff30 2024-03-18T19:02:20.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.202+0000 7fe10366a700 1 -- 172.21.15.1:0/1043544689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0fc0c8120 con 0x7fe0fc0cff30 2024-03-18T19:02:20.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.202+0000 7fe0f1ffb700 1 -- 172.21.15.1:0/1043544689 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fe0f400ac10 con 0x7fe0fc0cff30 2024-03-18T19:02:20.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.202+0000 7fe0f1ffb700 1 --2- 172.21.15.1:0/1043544689 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe0e006ae90 0x7fe0e006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:20.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.203+0000 7fe0f1ffb700 1 -- 172.21.15.1:0/1043544689 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fe0f40898e0 con 0x7fe0fc0cff30 2024-03-18T19:02:20.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.203+0000 7fe101406700 1 --2- 172.21.15.1:0/1043544689 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe0e006ae90 0x7fe0e006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:20.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.203+0000 7fe101406700 1 --2- 172.21.15.1:0/1043544689 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe0e006ae90 0x7fe0e006d350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0099c0 tx=0x7fe0ec008040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:20.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.207+0000 7fe0f1ffb700 1 -- 172.21.15.1:0/1043544689 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe0fc0c8120 con 0x7fe0fc0cff30 2024-03-18T19:02:20.300 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:02:20.302 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:02:20.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.441+0000 7fe10366a700 1 -- 172.21.15.1:0/1043544689 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0fc0d03b0 con 0x7fe0e006ae90 2024-03-18T19:02:20.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.511+0000 7fe0f1ffb700 1 -- 172.21.15.1:0/1043544689 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fe0ec009e30 con 0x7fe0e006ae90 2024-03-18T19:02:20.508 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2024-03-18T19:02:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.513+0000 7fe10366a700 1 -- 172.21.15.1:0/1043544689 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe0e006ae90 msgr2=0x7fe0e006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.513+0000 7fe10366a700 1 --2- 172.21.15.1:0/1043544689 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe0e006ae90 0x7fe0e006d350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0099c0 tx=0x7fe0ec008040 comp rx=0 tx=0).stop 2024-03-18T19:02:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.513+0000 7fe10366a700 1 -- 172.21.15.1:0/1043544689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0fc0cff30 msgr2=0x7fe0fc149f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.513+0000 7fe10366a700 1 --2- 172.21.15.1:0/1043544689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0fc0cff30 0x7fe0fc149f90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fe0f400cc50 tx=0x7fe0f400cf60 comp rx=0 tx=0).stop 2024-03-18T19:02:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.513+0000 7fe10366a700 1 -- 172.21.15.1:0/1043544689 shutdown_connections 2024-03-18T19:02:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.513+0000 7fe10366a700 1 --2- 172.21.15.1:0/1043544689 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe0e006ae90 0x7fe0e006d350 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.513+0000 7fe10366a700 1 --2- 172.21.15.1:0/1043544689 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0fc0b3d40 0x7fe0fc0c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.513+0000 7fe10366a700 1 --2- 172.21.15.1:0/1043544689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0fc0cff30 0x7fe0fc149f90 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.513+0000 7fe10366a700 1 --2- 172.21.15.1:0/1043544689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0fc0cfb50 0x7fe0fc0c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.513+0000 7fe10366a700 1 -- 172.21.15.1:0/1043544689 >> 172.21.15.1:0/1043544689 conn(0x7fe0fc0d3140 msgr2=0x7fe0fc15ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.513+0000 7fe10366a700 1 -- 172.21.15.1:0/1043544689 shutdown_connections 2024-03-18T19:02:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.513+0000 7fe10366a700 1 -- 172.21.15.1:0/1043544689 wait complete. 2024-03-18T19:02:20.522 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2024-03-18T19:02:20.522 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir2 1m 2024-03-18T19:02:20.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.653+0000 7fdd2ff8f700 1 Processor -- start 2024-03-18T19:02:20.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.653+0000 7fdd2ff8f700 1 -- start start 2024-03-18T19:02:20.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.654+0000 7fdd2ff8f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdd2815d5a0 0x7fdd280ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:20.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.654+0000 7fdd2ff8f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd2806ae60 0x7fdd28167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:20.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.654+0000 7fdd2ff8f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd280cf600 con 0x7fdd280ce550 2024-03-18T19:02:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.654+0000 7fdd2ff8f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd280ce9f0 con 0x7fdd2815d5a0 2024-03-18T19:02:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.654+0000 7fdd2ff8f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd280cecd0 con 0x7fdd2806ae60 2024-03-18T19:02:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.654+0000 7fdd2e52c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd2806ae60 0x7fdd28167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.654+0000 7fdd2dd2b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdd2815d5a0 0x7fdd280ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.654+0000 7fdd2e52c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd2806ae60 0x7fdd28167140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:56852/0 (socket says 172.21.15.1:56852) 2024-03-18T19:02:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.654+0000 7fdd2e52c700 1 -- 172.21.15.1:0/1595414536 learned_addr learned my addr 172.21.15.1:0/1595414536 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.654+0000 7fdd2d52a700 1 --1- 172.21.15.1:0/1595414536 >> v1:172.21.15.165:6789/0 conn(0x7fdd280ce550 0x7fdd2806a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:52868/0 (socket says 172.21.15.1:52868) 2024-03-18T19:02:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.654+0000 7fdd2e52c700 1 -- 172.21.15.1:0/1595414536 >> v2:172.21.15.165:3300/0 conn(0x7fdd2815d5a0 msgr2=0x7fdd280ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.654+0000 7fdd2e52c700 1 --2- 172.21.15.1:0/1595414536 >> v2:172.21.15.165:3300/0 conn(0x7fdd2815d5a0 0x7fdd280ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.654+0000 7fdd2e52c700 1 -- 172.21.15.1:0/1595414536 >> v1:172.21.15.165:6789/0 conn(0x7fdd280ce550 legacy=0x7fdd2806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.654+0000 7fdd2e52c700 1 -- 172.21.15.1:0/1595414536 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd280cefb0 con 0x7fdd2806ae60 2024-03-18T19:02:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.654+0000 7fdd2dd2b700 1 --2- 172.21.15.1:0/1595414536 >> v2:172.21.15.165:3300/0 conn(0x7fdd2815d5a0 0x7fdd280ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:02:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.655+0000 7fdd2e52c700 1 --2- 172.21.15.1:0/1595414536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd2806ae60 0x7fdd28167140 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdd20009e50 tx=0x7fdd2000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a3d3859a98a1da6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.655+0000 7fdd2cd29700 1 -- 172.21.15.1:0/1595414536 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2000e070 con 0x7fdd2806ae60 2024-03-18T19:02:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.655+0000 7fdd2cd29700 1 -- 172.21.15.1:0/1595414536 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd2000b910 con 0x7fdd2806ae60 2024-03-18T19:02:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.655+0000 7fdd2cd29700 1 -- 172.21.15.1:0/1595414536 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2000bcc0 con 0x7fdd2806ae60 2024-03-18T19:02:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.655+0000 7fdd2ff8f700 1 -- 172.21.15.1:0/1595414536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd2806ae60 msgr2=0x7fdd28167140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.655+0000 7fdd2ff8f700 1 --2- 172.21.15.1:0/1595414536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd2806ae60 0x7fdd28167140 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdd20009e50 tx=0x7fdd2000b330 comp rx=0 tx=0).stop 2024-03-18T19:02:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.655+0000 7fdd2ff8f700 1 -- 172.21.15.1:0/1595414536 shutdown_connections 2024-03-18T19:02:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.655+0000 7fdd2ff8f700 1 --2- 172.21.15.1:0/1595414536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd2806ae60 0x7fdd28167140 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.655+0000 7fdd2ff8f700 1 --2- 172.21.15.1:0/1595414536 >> v2:172.21.15.165:3300/0 conn(0x7fdd2815d5a0 0x7fdd280ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.655+0000 7fdd2ff8f700 1 -- 172.21.15.1:0/1595414536 >> 172.21.15.1:0/1595414536 conn(0x7fdd280cc9b0 msgr2=0x7fdd280ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.655+0000 7fdd2ff8f700 1 -- 172.21.15.1:0/1595414536 shutdown_connections 2024-03-18T19:02:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.655+0000 7fdd2ff8f700 1 -- 172.21.15.1:0/1595414536 wait complete. 2024-03-18T19:02:20.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.657+0000 7fdd2ff8f700 1 Processor -- start 2024-03-18T19:02:20.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.657+0000 7fdd2ff8f700 1 -- start start 2024-03-18T19:02:20.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.657+0000 7fdd2ff8f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd2806ae60 0x7fdd280c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:20.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.657+0000 7fdd2ff8f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd280ce550 0x7fdd2814aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:20.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.657+0000 7fdd2ff8f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd2815d5a0 0x7fdd2814afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:20.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.657+0000 7fdd2ff8f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd280ca510 con 0x7fdd2815d5a0 2024-03-18T19:02:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.657+0000 7fdd2ff8f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd280c9900 con 0x7fdd280ce550 2024-03-18T19:02:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.657+0000 7fdd2ff8f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd280c9be0 con 0x7fdd2806ae60 2024-03-18T19:02:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.657+0000 7fdd2e52c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd2815d5a0 0x7fdd2814afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.658+0000 7fdd2e52c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd2815d5a0 0x7fdd2814afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55404/0 (socket says 172.21.15.1:55404) 2024-03-18T19:02:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.658+0000 7fdd2e52c700 1 -- 172.21.15.1:0/1168801098 learned_addr learned my addr 172.21.15.1:0/1168801098 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.658+0000 7fdd2d52a700 1 --2- 172.21.15.1:0/1168801098 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd280ce550 0x7fdd2814aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.658+0000 7fdd2dd2b700 1 --2- 172.21.15.1:0/1168801098 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd2806ae60 0x7fdd280c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.658+0000 7fdd2e52c700 1 -- 172.21.15.1:0/1168801098 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd2806ae60 msgr2=0x7fdd280c7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.658+0000 7fdd2e52c700 1 --2- 172.21.15.1:0/1168801098 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd2806ae60 0x7fdd280c7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.658+0000 7fdd2e52c700 1 -- 172.21.15.1:0/1168801098 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd280ce550 msgr2=0x7fdd2814aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.658+0000 7fdd2e52c700 1 --2- 172.21.15.1:0/1168801098 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd280ce550 0x7fdd2814aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.658+0000 7fdd2e52c700 1 -- 172.21.15.1:0/1168801098 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd24009430 con 0x7fdd2815d5a0 2024-03-18T19:02:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.658+0000 7fdd2d52a700 1 --2- 172.21.15.1:0/1168801098 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd280ce550 0x7fdd2814aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.658+0000 7fdd2e52c700 1 --2- 172.21.15.1:0/1168801098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd2815d5a0 0x7fdd2814afd0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fdd280cee10 tx=0x7fdd20004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.658+0000 7fdd1e7fc700 1 -- 172.21.15.1:0/1168801098 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd200048b0 con 0x7fdd2815d5a0 2024-03-18T19:02:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.658+0000 7fdd1e7fc700 1 -- 172.21.15.1:0/1168801098 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd200099b0 con 0x7fdd2815d5a0 2024-03-18T19:02:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.658+0000 7fdd2ff8f700 1 -- 172.21.15.1:0/1168801098 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd20009b30 con 0x7fdd2815d5a0 2024-03-18T19:02:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.658+0000 7fdd1e7fc700 1 -- 172.21.15.1:0/1168801098 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2001c680 con 0x7fdd2815d5a0 2024-03-18T19:02:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.658+0000 7fdd2ff8f700 1 -- 172.21.15.1:0/1168801098 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd280ca1a0 con 0x7fdd2815d5a0 2024-03-18T19:02:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.659+0000 7fdd2ff8f700 1 -- 172.21.15.1:0/1168801098 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd280676a0 con 0x7fdd2815d5a0 2024-03-18T19:02:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.660+0000 7fdd1e7fc700 1 -- 172.21.15.1:0/1168801098 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fdd2001c7e0 con 0x7fdd2815d5a0 2024-03-18T19:02:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.660+0000 7fdd1e7fc700 1 --2- 172.21.15.1:0/1168801098 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdd0c06ae90 0x7fdd0c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.660+0000 7fdd1e7fc700 1 -- 172.21.15.1:0/1168801098 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fdd2008b620 con 0x7fdd2815d5a0 2024-03-18T19:02:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.660+0000 7fdd2dd2b700 1 --2- 172.21.15.1:0/1168801098 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdd0c06ae90 0x7fdd0c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.660+0000 7fdd2dd2b700 1 --2- 172.21.15.1:0/1168801098 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdd0c06ae90 0x7fdd0c06d350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009cb0 tx=0x7fdd240091c0 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.665+0000 7fdd1e7fc700 1 -- 172.21.15.1:0/1168801098 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdd200562a0 con 0x7fdd2815d5a0 2024-03-18T19:02:20.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.897+0000 7fdd2ff8f700 1 -- 172.21.15.1:0/1168801098 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd2815d980 con 0x7fdd0c06ae90 2024-03-18T19:02:20.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.965+0000 7fdd1e7fc700 1 -- 172.21.15.1:0/1168801098 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fdd24009900 con 0x7fdd0c06ae90 2024-03-18T19:02:20.962 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2024-03-18T19:02:20.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.967+0000 7fdd2ff8f700 1 -- 172.21.15.1:0/1168801098 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdd0c06ae90 msgr2=0x7fdd0c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:20.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.967+0000 7fdd2ff8f700 1 --2- 172.21.15.1:0/1168801098 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdd0c06ae90 0x7fdd0c06d350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009cb0 tx=0x7fdd240091c0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.967+0000 7fdd2ff8f700 1 -- 172.21.15.1:0/1168801098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd2815d5a0 msgr2=0x7fdd2814afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:20.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.967+0000 7fdd2ff8f700 1 --2- 172.21.15.1:0/1168801098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd2815d5a0 0x7fdd2814afd0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fdd280cee10 tx=0x7fdd20004ad0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.967+0000 7fdd2ff8f700 1 -- 172.21.15.1:0/1168801098 shutdown_connections 2024-03-18T19:02:20.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.967+0000 7fdd2ff8f700 1 --2- 172.21.15.1:0/1168801098 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdd0c06ae90 0x7fdd0c06d350 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.967+0000 7fdd2ff8f700 1 --2- 172.21.15.1:0/1168801098 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd2806ae60 0x7fdd280c7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.967+0000 7fdd2ff8f700 1 --2- 172.21.15.1:0/1168801098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd2815d5a0 0x7fdd2814afd0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.967+0000 7fdd2ff8f700 1 --2- 172.21.15.1:0/1168801098 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd280ce550 0x7fdd2814aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:20.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.967+0000 7fdd2ff8f700 1 -- 172.21.15.1:0/1168801098 >> 172.21.15.1:0/1168801098 conn(0x7fdd280cc9b0 msgr2=0x7fdd28071520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:20.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.967+0000 7fdd2ff8f700 1 -- 172.21.15.1:0/1168801098 shutdown_connections 2024-03-18T19:02:20.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:20.967+0000 7fdd2ff8f700 1 -- 172.21.15.1:0/1168801098 wait complete. 2024-03-18T19:02:20.976 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2024-03-18T19:02:20.976 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir3 1m 2024-03-18T19:02:21.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.107+0000 7f6bcbc6a700 1 Processor -- start 2024-03-18T19:02:21.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.108+0000 7f6bcbc6a700 1 -- start start 2024-03-18T19:02:21.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.108+0000 7f6bcbc6a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bc4071550 0x7f6bc40b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:21.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bcbc6a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc40b2b80 con 0x7f6bc40b2650 2024-03-18T19:02:21.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bcbc6a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc40b2e60 con 0x7f6bc415d3b0 2024-03-18T19:02:21.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bcbc6a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc40b3800 con 0x7f6bc4071550 2024-03-18T19:02:21.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc9205700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6bc40b2650 0x7f6bc4070e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32878/0 (socket says 172.21.15.1:32878) 2024-03-18T19:02:21.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc9205700 1 -- 172.21.15.1:0/2166241464 learned_addr learned my addr 172.21.15.1:0/2166241464 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:21.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bca207700 1 --2- 172.21.15.1:0/2166241464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bc4071550 0x7f6bc40b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:21.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc8a04700 1 -- 172.21.15.1:0/2166241464 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142774291 0 0) 0x7f6bc40b2b80 con 0x7f6bc40b2650 2024-03-18T19:02:21.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc8a04700 1 -- 172.21.15.1:0/2166241464 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bb00035b0 con 0x7f6bc40b2650 2024-03-18T19:02:21.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc8a04700 1 -- 172.21.15.1:0/2166241464 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424081781 0 0) 0x7f6bc40b2e60 con 0x7f6bc415d3b0 2024-03-18T19:02:21.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc8a04700 1 -- 172.21.15.1:0/2166241464 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bc40b2b80 con 0x7f6bc415d3b0 2024-03-18T19:02:21.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc8a04700 1 -- 172.21.15.1:0/2166241464 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 850652322 0 0) 0x7f6bb00035b0 con 0x7f6bc40b2650 2024-03-18T19:02:21.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc8a04700 1 -- 172.21.15.1:0/2166241464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bc4071550 msgr2=0x7f6bc40b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:21.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc8a04700 1 --2- 172.21.15.1:0/2166241464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bc4071550 0x7f6bc40b3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:21.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc8a04700 1 -- 172.21.15.1:0/2166241464 >> v1:172.21.15.165:6789/0 conn(0x7f6bc415d3b0 legacy=0x7f6bc406cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:21.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc8a04700 1 -- 172.21.15.1:0/2166241464 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc40b3140 con 0x7f6bc40b2650 2024-03-18T19:02:21.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc8a04700 1 -- 172.21.15.1:0/2166241464 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f6bb8001670 con 0x7f6bc40b2650 2024-03-18T19:02:21.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc8a04700 1 -- 172.21.15.1:0/2166241464 >> v1:172.21.15.1:6789/0 conn(0x7f6bc40b2650 legacy=0x7f6bc4070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:21.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc8a04700 1 --2- 172.21.15.1:0/2166241464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bb0003bc0 0x7f6bb0003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:21.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc8a04700 1 --2- 172.21.15.1:0/2166241464 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bb0004850 0x7f6bb0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:21.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc8a04700 1 --2- 172.21.15.1:0/2166241464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bc4071550 0x7f6bb0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:21.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc8a04700 1 -- 172.21.15.1:0/2166241464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc40b3800 con 0x7f6bb0003bc0 2024-03-18T19:02:21.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc8a04700 1 -- 172.21.15.1:0/2166241464 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb000baf0 con 0x7f6bb0004850 2024-03-18T19:02:21.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc8a04700 1 -- 172.21.15.1:0/2166241464 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb000bcb0 con 0x7f6bc4071550 2024-03-18T19:02:21.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.109+0000 7f6bc9a06700 1 --2- 172.21.15.1:0/2166241464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bb0003bc0 0x7f6bb0003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:21.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.110+0000 7f6bc9205700 1 --2- 172.21.15.1:0/2166241464 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bb0004850 0x7f6bb0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:21.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.110+0000 7f6bca207700 1 --2- 172.21.15.1:0/2166241464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bc4071550 0x7f6bb0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:21.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.110+0000 7f6bc9a06700 1 -- 172.21.15.1:0/2166241464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bc4071550 msgr2=0x7f6bb0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:21.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.110+0000 7f6bc9a06700 1 --2- 172.21.15.1:0/2166241464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bc4071550 0x7f6bb0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:21.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.110+0000 7f6bc9a06700 1 -- 172.21.15.1:0/2166241464 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bb0004850 msgr2=0x7f6bb0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:21.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.110+0000 7f6bc9a06700 1 --2- 172.21.15.1:0/2166241464 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bb0004850 0x7f6bb0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:21.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.110+0000 7f6bc9a06700 1 -- 172.21.15.1:0/2166241464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb00035b0 con 0x7f6bb0003bc0 2024-03-18T19:02:21.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.110+0000 7f6bc9a06700 1 --2- 172.21.15.1:0/2166241464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bb0003bc0 0x7f6bb0003fc0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0001980 tx=0x7f6bb0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6789aeb6623596b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:21.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.110+0000 7f6bc8a04700 1 -- 172.21.15.1:0/2166241464 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc0001470 con 0x7f6bb0003bc0 2024-03-18T19:02:21.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.110+0000 7f6bc8a04700 1 -- 172.21.15.1:0/2166241464 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc00052d0 con 0x7f6bb0003bc0 2024-03-18T19:02:21.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.110+0000 7f6bc8a04700 1 -- 172.21.15.1:0/2166241464 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc0003680 con 0x7f6bb0003bc0 2024-03-18T19:02:21.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.111+0000 7f6bcbc6a700 1 -- 172.21.15.1:0/2166241464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bb0003bc0 msgr2=0x7f6bb0003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:21.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.111+0000 7f6bcbc6a700 1 --2- 172.21.15.1:0/2166241464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bb0003bc0 0x7f6bb0003fc0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0001980 tx=0x7f6bb0005880 comp rx=0 tx=0).stop 2024-03-18T19:02:21.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.111+0000 7f6bca207700 1 -- 172.21.15.1:0/2166241464 reap_dead start 2024-03-18T19:02:21.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.111+0000 7f6bcbc6a700 1 -- 172.21.15.1:0/2166241464 shutdown_connections 2024-03-18T19:02:21.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.111+0000 7f6bcbc6a700 1 -- 172.21.15.1:0/2166241464 >> 172.21.15.1:0/2166241464 conn(0x7f6bc40b1350 msgr2=0x7f6bc40b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:21.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.111+0000 7f6bcbc6a700 1 -- 172.21.15.1:0/2166241464 shutdown_connections 2024-03-18T19:02:21.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.111+0000 7f6bcbc6a700 1 -- 172.21.15.1:0/2166241464 wait complete. 2024-03-18T19:02:21.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.112+0000 7f6bcbc6a700 1 Processor -- start 2024-03-18T19:02:21.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.112+0000 7f6bcbc6a700 1 -- start start 2024-03-18T19:02:21.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bcbc6a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bb0003bc0 0x7f6bc4158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:21.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bcbc6a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bc41517b0 0x7f6bc406ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:21.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bcbc6a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bc4151c30 0x7f6bc406f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:21.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bcbc6a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc4159470 con 0x7f6bc41517b0 2024-03-18T19:02:21.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bcbc6a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc4157a10 con 0x7f6bc4151c30 2024-03-18T19:02:21.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bcbc6a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc4157cf0 con 0x7f6bb0003bc0 2024-03-18T19:02:21.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bc9205700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bc41517b0 0x7f6bc406ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:21.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bc9205700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bc41517b0 0x7f6bc406ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55418/0 (socket says 172.21.15.1:55418) 2024-03-18T19:02:21.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bc9205700 1 -- 172.21.15.1:0/3821417011 learned_addr learned my addr 172.21.15.1:0/3821417011 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bca207700 1 --2- 172.21.15.1:0/3821417011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bc4151c30 0x7f6bc406f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bc9a06700 1 --2- 172.21.15.1:0/3821417011 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bb0003bc0 0x7f6bc4158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bc9205700 1 -- 172.21.15.1:0/3821417011 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bb0003bc0 msgr2=0x7f6bc4158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bc9205700 1 --2- 172.21.15.1:0/3821417011 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bb0003bc0 0x7f6bc4158f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bc9205700 1 -- 172.21.15.1:0/3821417011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bc4151c30 msgr2=0x7f6bc406f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bc9205700 1 --2- 172.21.15.1:0/3821417011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bc4151c30 0x7f6bc406f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bc9205700 1 -- 172.21.15.1:0/3821417011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc4157fd0 con 0x7f6bc41517b0 2024-03-18T19:02:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bc9205700 1 --2- 172.21.15.1:0/3821417011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bc41517b0 0x7f6bc406ec30 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8004f70 tx=0x7f6bb8007340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bb67fc700 1 -- 172.21.15.1:0/3821417011 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8011580 con 0x7f6bc41517b0 2024-03-18T19:02:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bb67fc700 1 -- 172.21.15.1:0/3821417011 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb80116e0 con 0x7f6bc41517b0 2024-03-18T19:02:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bcbc6a700 1 -- 172.21.15.1:0/3821417011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc41582b0 con 0x7f6bc41517b0 2024-03-18T19:02:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bb67fc700 1 -- 172.21.15.1:0/3821417011 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8011a50 con 0x7f6bc41517b0 2024-03-18T19:02:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.113+0000 7f6bcbc6a700 1 -- 172.21.15.1:0/3821417011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc4152830 con 0x7f6bc41517b0 2024-03-18T19:02:21.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.114+0000 7f6bcbc6a700 1 -- 172.21.15.1:0/3821417011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc4153080 con 0x7f6bc41517b0 2024-03-18T19:02:21.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.115+0000 7f6bb67fc700 1 -- 172.21.15.1:0/3821417011 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f6bb8011bb0 con 0x7f6bc41517b0 2024-03-18T19:02:21.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.115+0000 7f6bb67fc700 1 --2- 172.21.15.1:0/3821417011 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ba806ae90 0x7f6ba806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:21.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.115+0000 7f6bc9a06700 1 --2- 172.21.15.1:0/3821417011 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ba806ae90 0x7f6ba806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:21.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.115+0000 7f6bb67fc700 1 -- 172.21.15.1:0/3821417011 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f6bb8089900 con 0x7f6bc41517b0 2024-03-18T19:02:21.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.116+0000 7f6bc9a06700 1 --2- 172.21.15.1:0/3821417011 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ba806ae90 0x7f6ba806d350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0015040 tx=0x7f6bc000a000 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:21.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.120+0000 7f6bb67fc700 1 -- 172.21.15.1:0/3821417011 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6bb8054600 con 0x7f6bc41517b0 2024-03-18T19:02:21.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.350+0000 7f6bcbc6a700 1 -- 172.21.15.1:0/3821417011 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bc406d600 con 0x7f6ba806ae90 2024-03-18T19:02:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.415+0000 7f6bb67fc700 1 -- 172.21.15.1:0/3821417011 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f6bc0005150 con 0x7f6ba806ae90 2024-03-18T19:02:21.412 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2024-03-18T19:02:21.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.417+0000 7f6bcbc6a700 1 -- 172.21.15.1:0/3821417011 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ba806ae90 msgr2=0x7f6ba806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:21.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.417+0000 7f6bcbc6a700 1 --2- 172.21.15.1:0/3821417011 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ba806ae90 0x7f6ba806d350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0015040 tx=0x7f6bc000a000 comp rx=0 tx=0).stop 2024-03-18T19:02:21.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.417+0000 7f6bcbc6a700 1 -- 172.21.15.1:0/3821417011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bc41517b0 msgr2=0x7f6bc406ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:21.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.417+0000 7f6bcbc6a700 1 --2- 172.21.15.1:0/3821417011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bc41517b0 0x7f6bc406ec30 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8004f70 tx=0x7f6bb8007340 comp rx=0 tx=0).stop 2024-03-18T19:02:21.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.417+0000 7f6bcbc6a700 1 -- 172.21.15.1:0/3821417011 shutdown_connections 2024-03-18T19:02:21.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.417+0000 7f6bcbc6a700 1 --2- 172.21.15.1:0/3821417011 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ba806ae90 0x7f6ba806d350 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:21.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.417+0000 7f6bcbc6a700 1 --2- 172.21.15.1:0/3821417011 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bb0003bc0 0x7f6bc4158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:21.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.417+0000 7f6bcbc6a700 1 --2- 172.21.15.1:0/3821417011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bc4151c30 0x7f6bc406f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:21.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.417+0000 7f6bcbc6a700 1 --2- 172.21.15.1:0/3821417011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bc41517b0 0x7f6bc406ec30 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:21.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.417+0000 7f6bcbc6a700 1 -- 172.21.15.1:0/3821417011 >> 172.21.15.1:0/3821417011 conn(0x7f6bc40b1350 msgr2=0x7f6bc415c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:21.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.417+0000 7f6bcbc6a700 1 -- 172.21.15.1:0/3821417011 shutdown_connections 2024-03-18T19:02:21.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.417+0000 7f6bcbc6a700 1 -- 172.21.15.1:0/3821417011 wait complete. 2024-03-18T19:02:21.426 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2024-03-18T19:02:21.426 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir4 1m 2024-03-18T19:02:21.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.557+0000 7f14cae7f700 1 Processor -- start 2024-03-18T19:02:21.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.557+0000 7f14cae7f700 1 -- start start 2024-03-18T19:02:21.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.558+0000 7f14cae7f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f14c40cc7b0 0x7f14c40cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:21.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.558+0000 7f14cae7f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f14c40cfc00 0x7f14c40c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:21.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.558+0000 7f14cae7f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c40c4c50 con 0x7f14c415b480 2024-03-18T19:02:21.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.558+0000 7f14cae7f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c40d00b0 con 0x7f14c40cc7b0 2024-03-18T19:02:21.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.558+0000 7f14cae7f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c40c5210 con 0x7f14c40cfc00 2024-03-18T19:02:21.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.558+0000 7f14c3fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f14c40cc7b0 0x7f14c40cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:21.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.558+0000 7f14c8c1b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f14c415b480 0x7f14c40cedd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32894/0 (socket says 172.21.15.1:32894) 2024-03-18T19:02:21.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.558+0000 7f14c8c1b700 1 -- 172.21.15.1:0/2433554753 learned_addr learned my addr 172.21.15.1:0/2433554753 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:21.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.558+0000 7f14c941c700 1 --2- 172.21.15.1:0/2433554753 >> v2:172.21.15.165:3300/0 conn(0x7f14c40cfc00 0x7f14c40c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:21.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.558+0000 7f14c3fff700 1 -- 172.21.15.1:0/2433554753 >> v2:172.21.15.165:3300/0 conn(0x7f14c40cfc00 msgr2=0x7f14c40c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:21.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.558+0000 7f14c3fff700 1 --2- 172.21.15.1:0/2433554753 >> v2:172.21.15.165:3300/0 conn(0x7f14c40cfc00 0x7f14c40c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:21.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.558+0000 7f14c3fff700 1 -- 172.21.15.1:0/2433554753 >> v1:172.21.15.1:6789/0 conn(0x7f14c415b480 legacy=0x7f14c40cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:21.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.558+0000 7f14c37fe700 1 -- 172.21.15.1:0/2433554753 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694110784 0 0) 0x7f14c40c4c50 con 0x7f14c415b480 2024-03-18T19:02:21.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.558+0000 7f14c3fff700 1 -- 172.21.15.1:0/2433554753 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c40cf4e0 con 0x7f14c40cc7b0 2024-03-18T19:02:21.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.559+0000 7f14c3fff700 1 --2- 172.21.15.1:0/2433554753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14c40cc7b0 0x7f14c40cf6c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f14c4071cc0 tx=0x7f14b0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36ca66d56921b5b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:21.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.559+0000 7f14c37fe700 1 -- 172.21.15.1:0/2433554753 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0011830 con 0x7f14c40cc7b0 2024-03-18T19:02:21.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.559+0000 7f14c37fe700 1 -- 172.21.15.1:0/2433554753 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b0011990 con 0x7f14c40cc7b0 2024-03-18T19:02:21.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.559+0000 7f14c37fe700 1 -- 172.21.15.1:0/2433554753 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0011d20 con 0x7f14c40cc7b0 2024-03-18T19:02:21.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.559+0000 7f14cae7f700 1 -- 172.21.15.1:0/2433554753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14c40cc7b0 msgr2=0x7f14c40cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:21.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.559+0000 7f14cae7f700 1 --2- 172.21.15.1:0/2433554753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14c40cc7b0 0x7f14c40cf6c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f14c4071cc0 tx=0x7f14b0009b20 comp rx=0 tx=0).stop 2024-03-18T19:02:21.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.559+0000 7f14cae7f700 1 -- 172.21.15.1:0/2433554753 shutdown_connections 2024-03-18T19:02:21.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.559+0000 7f14cae7f700 1 --2- 172.21.15.1:0/2433554753 >> v2:172.21.15.165:3300/0 conn(0x7f14c40cfc00 0x7f14c40c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:21.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.559+0000 7f14cae7f700 1 --2- 172.21.15.1:0/2433554753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14c40cc7b0 0x7f14c40cf6c0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:21.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.559+0000 7f14cae7f700 1 -- 172.21.15.1:0/2433554753 >> 172.21.15.1:0/2433554753 conn(0x7f14c40b2bf0 msgr2=0x7f14c40b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:21.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.559+0000 7f14cae7f700 1 -- 172.21.15.1:0/2433554753 shutdown_connections 2024-03-18T19:02:21.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.559+0000 7f14cae7f700 1 -- 172.21.15.1:0/2433554753 wait complete. 2024-03-18T19:02:21.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.560+0000 7f14cae7f700 1 Processor -- start 2024-03-18T19:02:21.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.561+0000 7f14cae7f700 1 -- start start 2024-03-18T19:02:21.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.561+0000 7f14cae7f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14c40cc7b0 0x7f14c40b6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:21.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.561+0000 7f14cae7f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14c40cfc00 0x7f14c4151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:21.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.561+0000 7f14cae7f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14c415b480 0x7f14c40ce620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:21.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.561+0000 7f14cae7f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c40b6e70 con 0x7f14c40cfc00 2024-03-18T19:02:21.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.561+0000 7f14cae7f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c406b860 con 0x7f14c415b480 2024-03-18T19:02:21.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.561+0000 7f14cae7f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c406bb40 con 0x7f14c40cc7b0 2024-03-18T19:02:21.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.561+0000 7f14c3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14c40cfc00 0x7f14c4151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:21.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.562+0000 7f14c3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14c40cfc00 0x7f14c4151730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55424/0 (socket says 172.21.15.1:55424) 2024-03-18T19:02:21.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.562+0000 7f14c3fff700 1 -- 172.21.15.1:0/4194043173 learned_addr learned my addr 172.21.15.1:0/4194043173 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:21.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.562+0000 7f14c941c700 1 --2- 172.21.15.1:0/4194043173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14c415b480 0x7f14c40ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:21.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.562+0000 7f14c8c1b700 1 --2- 172.21.15.1:0/4194043173 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14c40cc7b0 0x7f14c40b6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:21.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.562+0000 7f14c3fff700 1 -- 172.21.15.1:0/4194043173 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14c40cc7b0 msgr2=0x7f14c40b6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:21.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.562+0000 7f14c3fff700 1 --2- 172.21.15.1:0/4194043173 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14c40cc7b0 0x7f14c40b6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:21.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.562+0000 7f14c3fff700 1 -- 172.21.15.1:0/4194043173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14c415b480 msgr2=0x7f14c40ce620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:21.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.562+0000 7f14c3fff700 1 --2- 172.21.15.1:0/4194043173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14c415b480 0x7f14c40ce620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:21.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.562+0000 7f14c3fff700 1 -- 172.21.15.1:0/4194043173 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b00094a0 con 0x7f14c40cfc00 2024-03-18T19:02:21.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.562+0000 7f14c3fff700 1 --2- 172.21.15.1:0/4194043173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14c40cfc00 0x7f14c4151730 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009b60 tx=0x7f14b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:21.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.562+0000 7f14c17fa700 1 -- 172.21.15.1:0/4194043173 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b00036a0 con 0x7f14c40cfc00 2024-03-18T19:02:21.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.562+0000 7f14c17fa700 1 -- 172.21.15.1:0/4194043173 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b001e070 con 0x7f14c40cfc00 2024-03-18T19:02:21.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.562+0000 7f14c17fa700 1 -- 172.21.15.1:0/4194043173 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0003a20 con 0x7f14c40cfc00 2024-03-18T19:02:21.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.562+0000 7f14cae7f700 1 -- 172.21.15.1:0/4194043173 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c406be20 con 0x7f14c40cfc00 2024-03-18T19:02:21.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.562+0000 7f14cae7f700 1 -- 172.21.15.1:0/4194043173 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c40c8390 con 0x7f14c40cfc00 2024-03-18T19:02:21.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.563+0000 7f14cae7f700 1 -- 172.21.15.1:0/4194043173 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c406c340 con 0x7f14c40cfc00 2024-03-18T19:02:21.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.564+0000 7f14c17fa700 1 -- 172.21.15.1:0/4194043173 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f14b0003b80 con 0x7f14c40cfc00 2024-03-18T19:02:21.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.564+0000 7f14c17fa700 1 --2- 172.21.15.1:0/4194043173 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14a806ae90 0x7f14a806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:21.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.564+0000 7f14c17fa700 1 -- 172.21.15.1:0/4194043173 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f14b008a840 con 0x7f14c40cfc00 2024-03-18T19:02:21.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.564+0000 7f14c8c1b700 1 --2- 172.21.15.1:0/4194043173 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14a806ae90 0x7f14a806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:21.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.564+0000 7f14c8c1b700 1 --2- 172.21.15.1:0/4194043173 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14a806ae90 0x7f14a806d350 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f14b8000f60 tx=0x7f14b8007be0 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:21.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.568+0000 7f14c17fa700 1 -- 172.21.15.1:0/4194043173 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f14c406c340 con 0x7f14c40cfc00 2024-03-18T19:02:21.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.801+0000 7f14cae7f700 1 -- 172.21.15.1:0/4194043173 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f14c40c8550 con 0x7f14a806ae90 2024-03-18T19:02:21.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.875+0000 7f14c17fa700 1 -- 172.21.15.1:0/4194043173 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f14c40c8550 con 0x7f14a806ae90 2024-03-18T19:02:21.873 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2024-03-18T19:02:21.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.878+0000 7f14cae7f700 1 -- 172.21.15.1:0/4194043173 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14a806ae90 msgr2=0x7f14a806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:21.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.878+0000 7f14cae7f700 1 --2- 172.21.15.1:0/4194043173 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14a806ae90 0x7f14a806d350 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f14b8000f60 tx=0x7f14b8007be0 comp rx=0 tx=0).stop 2024-03-18T19:02:21.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.878+0000 7f14cae7f700 1 -- 172.21.15.1:0/4194043173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14c40cfc00 msgr2=0x7f14c4151730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:21.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.878+0000 7f14cae7f700 1 --2- 172.21.15.1:0/4194043173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14c40cfc00 0x7f14c4151730 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009b60 tx=0x7f14b0004750 comp rx=0 tx=0).stop 2024-03-18T19:02:21.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.878+0000 7f14cae7f700 1 -- 172.21.15.1:0/4194043173 shutdown_connections 2024-03-18T19:02:21.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.878+0000 7f14cae7f700 1 --2- 172.21.15.1:0/4194043173 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14a806ae90 0x7f14a806d350 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:21.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.878+0000 7f14cae7f700 1 --2- 172.21.15.1:0/4194043173 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14c40cc7b0 0x7f14c40b6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:21.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.878+0000 7f14cae7f700 1 --2- 172.21.15.1:0/4194043173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14c415b480 0x7f14c40ce620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:21.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.878+0000 7f14cae7f700 1 --2- 172.21.15.1:0/4194043173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14c40cfc00 0x7f14c4151730 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:21.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.878+0000 7f14cae7f700 1 -- 172.21.15.1:0/4194043173 >> 172.21.15.1:0/4194043173 conn(0x7f14c40b2bf0 msgr2=0x7f14c415da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:21.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.878+0000 7f14cae7f700 1 -- 172.21.15.1:0/4194043173 shutdown_connections 2024-03-18T19:02:21.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:21.878+0000 7f14cae7f700 1 -- 172.21.15.1:0/4194043173 wait complete. 2024-03-18T19:02:21.887 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2024-03-18T19:02:21.887 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir5 1m 2024-03-18T19:02:22.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.018+0000 7f3dcd292700 1 Processor -- start 2024-03-18T19:02:22.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.018+0000 7f3dcd292700 1 -- start start 2024-03-18T19:02:22.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.019+0000 7f3dcd292700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3dc80b1e80 0x7f3dc80cad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.020+0000 7f3dcd292700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc8142fc0 con 0x7f3dc80cb280 2024-03-18T19:02:22.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.020+0000 7f3dcd292700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc80cfff0 con 0x7f3dc80cfa10 2024-03-18T19:02:22.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.020+0000 7f3dcd292700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc80cb730 con 0x7f3dc80b1e80 2024-03-18T19:02:22.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.020+0000 7f3dc67fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3dc80cb280 0x7f3dc80cf300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32898/0 (socket says 172.21.15.1:32898) 2024-03-18T19:02:22.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.020+0000 7f3dc67fc700 1 -- 172.21.15.1:0/3505434323 learned_addr learned my addr 172.21.15.1:0/3505434323 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:22.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.020+0000 7f3dc6ffd700 1 --2- 172.21.15.1:0/3505434323 >> v2:172.21.15.165:3300/0 conn(0x7f3dc80b1e80 0x7f3dc80cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.020+0000 7f3dc5ffb700 1 -- 172.21.15.1:0/3505434323 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119082980 0 0) 0x7f3dc8142fc0 con 0x7f3dc80cb280 2024-03-18T19:02:22.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.020+0000 7f3dc5ffb700 1 -- 172.21.15.1:0/3505434323 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db00035b0 con 0x7f3dc80cb280 2024-03-18T19:02:22.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.020+0000 7f3dc6ffd700 1 -- 172.21.15.1:0/3505434323 >> v1:172.21.15.165:6789/0 conn(0x7f3dc80cfa10 legacy=0x7f3dc8161310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:22.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.020+0000 7f3dc6ffd700 1 -- 172.21.15.1:0/3505434323 >> v1:172.21.15.1:6789/0 conn(0x7f3dc80cb280 legacy=0x7f3dc80cf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:22.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.020+0000 7f3dc6ffd700 1 -- 172.21.15.1:0/3505434323 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc8143210 con 0x7f3dc80b1e80 2024-03-18T19:02:22.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.020+0000 7f3dc6ffd700 1 --2- 172.21.15.1:0/3505434323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dc80b1e80 0x7f3dc80cad40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009840 tx=0x7f3db8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c80c3e9c179cef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:22.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.023+0000 7f3dc5ffb700 1 -- 172.21.15.1:0/3505434323 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db8011830 con 0x7f3dc80b1e80 2024-03-18T19:02:22.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.023+0000 7f3dc5ffb700 1 -- 172.21.15.1:0/3505434323 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db8011990 con 0x7f3dc80b1e80 2024-03-18T19:02:22.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.023+0000 7f3dc5ffb700 1 -- 172.21.15.1:0/3505434323 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db8011d20 con 0x7f3dc80b1e80 2024-03-18T19:02:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.024+0000 7f3dcd292700 1 -- 172.21.15.1:0/3505434323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dc80b1e80 msgr2=0x7f3dc80cad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.024+0000 7f3dcd292700 1 --2- 172.21.15.1:0/3505434323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dc80b1e80 0x7f3dc80cad40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009840 tx=0x7f3db8009b50 comp rx=0 tx=0).stop 2024-03-18T19:02:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.024+0000 7f3dcd292700 1 -- 172.21.15.1:0/3505434323 shutdown_connections 2024-03-18T19:02:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.024+0000 7f3dcd292700 1 --2- 172.21.15.1:0/3505434323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dc80b1e80 0x7f3dc80cad40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.024+0000 7f3dcd292700 1 -- 172.21.15.1:0/3505434323 >> 172.21.15.1:0/3505434323 conn(0x7f3dc8066c80 msgr2=0x7f3dc80b2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.024+0000 7f3dcd292700 1 -- 172.21.15.1:0/3505434323 shutdown_connections 2024-03-18T19:02:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.024+0000 7f3dcd292700 1 -- 172.21.15.1:0/3505434323 wait complete. 2024-03-18T19:02:22.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.025+0000 7f3dcd292700 1 Processor -- start 2024-03-18T19:02:22.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.025+0000 7f3dcd292700 1 -- start start 2024-03-18T19:02:22.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3dcd292700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3dc80b1e80 0x7f3dc815d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3dcd292700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dc80cb280 0x7f3dc8144f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3dcd292700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dc80cfa10 0x7f3dc80ce1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3dcd292700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc8144ad0 con 0x7f3dc80cfa10 2024-03-18T19:02:22.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3dcd292700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc81441e0 con 0x7f3dc80cb280 2024-03-18T19:02:22.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3dcd292700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc8144480 con 0x7f3dc80b1e80 2024-03-18T19:02:22.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3dc77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dc80cfa10 0x7f3dc80ce1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3dc77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dc80cfa10 0x7f3dc80ce1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55438/0 (socket says 172.21.15.1:55438) 2024-03-18T19:02:22.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3dc77fe700 1 -- 172.21.15.1:0/998874158 learned_addr learned my addr 172.21.15.1:0/998874158 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:22.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3dc67fc700 1 --2- 172.21.15.1:0/998874158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dc80cb280 0x7f3dc8144f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3dc6ffd700 1 --2- 172.21.15.1:0/998874158 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3dc80b1e80 0x7f3dc815d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3dc77fe700 1 -- 172.21.15.1:0/998874158 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3dc80b1e80 msgr2=0x7f3dc815d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:22.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3dc77fe700 1 --2- 172.21.15.1:0/998874158 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3dc80b1e80 0x7f3dc815d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3dc77fe700 1 -- 172.21.15.1:0/998874158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dc80cb280 msgr2=0x7f3dc8144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:22.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3dc77fe700 1 --2- 172.21.15.1:0/998874158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dc80cb280 0x7f3dc8144f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3dc77fe700 1 -- 172.21.15.1:0/998874158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db80094a0 con 0x7f3dc80cfa10 2024-03-18T19:02:22.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3dc67fc700 1 --2- 172.21.15.1:0/998874158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dc80cb280 0x7f3dc8144f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:22.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3dc77fe700 1 --2- 172.21.15.1:0/998874158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dc80cfa10 0x7f3dc80ce1e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f3dc000cb50 tx=0x7f3dc000cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:22.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3daf7fe700 1 -- 172.21.15.1:0/998874158 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0003c00 con 0x7f3dc80cfa10 2024-03-18T19:02:22.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3daf7fe700 1 -- 172.21.15.1:0/998874158 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc0003d60 con 0x7f3dc80cfa10 2024-03-18T19:02:22.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.026+0000 7f3dcd292700 1 -- 172.21.15.1:0/998874158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc8144760 con 0x7f3dc80cfa10 2024-03-18T19:02:22.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.027+0000 7f3daf7fe700 1 -- 172.21.15.1:0/998874158 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc000ab70 con 0x7f3dc80cfa10 2024-03-18T19:02:22.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.027+0000 7f3dcd292700 1 -- 172.21.15.1:0/998874158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc8146e70 con 0x7f3dc80cfa10 2024-03-18T19:02:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.028+0000 7f3dcd292700 1 -- 172.21.15.1:0/998874158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc81448f0 con 0x7f3dc80cfa10 2024-03-18T19:02:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.028+0000 7f3daf7fe700 1 -- 172.21.15.1:0/998874158 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f3dc000acd0 con 0x7f3dc80cfa10 2024-03-18T19:02:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.028+0000 7f3daf7fe700 1 --2- 172.21.15.1:0/998874158 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3db406ae90 0x7f3db406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.028+0000 7f3daf7fe700 1 -- 172.21.15.1:0/998874158 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f3dc0089990 con 0x7f3dc80cfa10 2024-03-18T19:02:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.029+0000 7f3dc6ffd700 1 --2- 172.21.15.1:0/998874158 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3db406ae90 0x7f3db406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.029+0000 7f3dc6ffd700 1 --2- 172.21.15.1:0/998874158 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3db406ae90 0x7f3db406d350 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3db80090b0 tx=0x7f3db8008030 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:22.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.033+0000 7f3daf7fe700 1 -- 172.21.15.1:0/998874158 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3dc0054720 con 0x7f3dc80cfa10 2024-03-18T19:02:22.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.263+0000 7f3dcd292700 1 -- 172.21.15.1:0/998874158 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3dc81437d0 con 0x7f3db406ae90 2024-03-18T19:02:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.330+0000 7f3daf7fe700 1 -- 172.21.15.1:0/998874158 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f3db8011630 con 0x7f3db406ae90 2024-03-18T19:02:22.327 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2024-03-18T19:02:22.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.332+0000 7f3dcd292700 1 -- 172.21.15.1:0/998874158 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3db406ae90 msgr2=0x7f3db406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.332+0000 7f3dcd292700 1 --2- 172.21.15.1:0/998874158 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3db406ae90 0x7f3db406d350 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3db80090b0 tx=0x7f3db8008030 comp rx=0 tx=0).stop 2024-03-18T19:02:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.332+0000 7f3dcd292700 1 -- 172.21.15.1:0/998874158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dc80cfa10 msgr2=0x7f3dc80ce1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.332+0000 7f3dcd292700 1 --2- 172.21.15.1:0/998874158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dc80cfa10 0x7f3dc80ce1e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f3dc000cb50 tx=0x7f3dc000cf10 comp rx=0 tx=0).stop 2024-03-18T19:02:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.332+0000 7f3dcd292700 1 -- 172.21.15.1:0/998874158 shutdown_connections 2024-03-18T19:02:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.332+0000 7f3dcd292700 1 --2- 172.21.15.1:0/998874158 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3db406ae90 0x7f3db406d350 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.332+0000 7f3dcd292700 1 --2- 172.21.15.1:0/998874158 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3dc80b1e80 0x7f3dc815d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.332+0000 7f3dcd292700 1 --2- 172.21.15.1:0/998874158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3dc80cfa10 0x7f3dc80ce1e0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.332+0000 7f3dcd292700 1 --2- 172.21.15.1:0/998874158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3dc80cb280 0x7f3dc8144f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.332+0000 7f3dcd292700 1 -- 172.21.15.1:0/998874158 >> 172.21.15.1:0/998874158 conn(0x7f3dc8066c80 msgr2=0x7f3dc80c4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.332+0000 7f3dcd292700 1 -- 172.21.15.1:0/998874158 shutdown_connections 2024-03-18T19:02:22.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.332+0000 7f3dcd292700 1 -- 172.21.15.1:0/998874158 wait complete. 2024-03-18T19:02:22.341 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2024-03-18T19:02:22.341 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir6 1m 2024-03-18T19:02:22.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.472+0000 7f17b3a32700 1 Processor -- start 2024-03-18T19:02:22.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.472+0000 7f17b3a32700 1 -- start start 2024-03-18T19:02:22.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.473+0000 7f17b3a32700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17ac06f180 0x7f17ac161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.473+0000 7f17b3a32700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ac0cad60 con 0x7f17ac06a750 2024-03-18T19:02:22.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.473+0000 7f17b3a32700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ac0cb040 con 0x7f17ac06eda0 2024-03-18T19:02:22.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.473+0000 7f17b3a32700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac0cb320 con 0x7f17ac06f180 2024-03-18T19:02:22.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.473+0000 7f17b1fcf700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f17ac06a750 0x7f17ac15cbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32904/0 (socket says 172.21.15.1:32904) 2024-03-18T19:02:22.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.473+0000 7f17b1fcf700 1 -- 172.21.15.1:0/3172974667 learned_addr learned my addr 172.21.15.1:0/3172974667 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:22.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.473+0000 7f17b0fcd700 1 --2- 172.21.15.1:0/3172974667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17ac06f180 0x7f17ac161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.473+0000 7f17a3fff700 1 -- 172.21.15.1:0/3172974667 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242481772 0 0) 0x7f17ac0cad60 con 0x7f17ac06a750 2024-03-18T19:02:22.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.473+0000 7f17a3fff700 1 -- 172.21.15.1:0/3172974667 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17980035b0 con 0x7f17ac06a750 2024-03-18T19:02:22.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17a3fff700 1 -- 172.21.15.1:0/3172974667 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691319451 0 0) 0x7f17ac0cb040 con 0x7f17ac06eda0 2024-03-18T19:02:22.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17a3fff700 1 -- 172.21.15.1:0/3172974667 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17ac0cad60 con 0x7f17ac06eda0 2024-03-18T19:02:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17a3fff700 1 -- 172.21.15.1:0/3172974667 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1135096388 0 0) 0x7f17980035b0 con 0x7f17ac06a750 2024-03-18T19:02:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17a3fff700 1 -- 172.21.15.1:0/3172974667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17ac06f180 msgr2=0x7f17ac161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17a3fff700 1 --2- 172.21.15.1:0/3172974667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17ac06f180 0x7f17ac161c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17a3fff700 1 -- 172.21.15.1:0/3172974667 >> v1:172.21.15.165:6789/0 conn(0x7f17ac06eda0 legacy=0x7f17ac0ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17a3fff700 1 -- 172.21.15.1:0/3172974667 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ac0ce220 con 0x7f17ac06a750 2024-03-18T19:02:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17a3fff700 1 -- 172.21.15.1:0/3172974667 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f17a40019f0 con 0x7f17ac06a750 2024-03-18T19:02:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17a3fff700 1 -- 172.21.15.1:0/3172974667 >> v1:172.21.15.1:6789/0 conn(0x7f17ac06a750 legacy=0x7f17ac15cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17a3fff700 1 --2- 172.21.15.1:0/3172974667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17ac06f180 0x7f1798003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17a3fff700 1 --2- 172.21.15.1:0/3172974667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1798004120 0x7f1798004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17a3fff700 1 --2- 172.21.15.1:0/3172974667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1798004d90 0x7f1798005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17a3fff700 1 -- 172.21.15.1:0/3172974667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179800bac0 con 0x7f1798004120 2024-03-18T19:02:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17a3fff700 1 -- 172.21.15.1:0/3172974667 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179800bc30 con 0x7f1798004d90 2024-03-18T19:02:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17a3fff700 1 -- 172.21.15.1:0/3172974667 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f179800bdf0 con 0x7f17ac06f180 2024-03-18T19:02:22.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17b0fcd700 1 --2- 172.21.15.1:0/3172974667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1798004120 0x7f1798004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17b17ce700 1 --2- 172.21.15.1:0/3172974667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17ac06f180 0x7f1798003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17b1fcf700 1 --2- 172.21.15.1:0/3172974667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1798004d90 0x7f1798005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17b0fcd700 1 -- 172.21.15.1:0/3172974667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17ac06f180 msgr2=0x7f1798003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:22.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17b0fcd700 1 --2- 172.21.15.1:0/3172974667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17ac06f180 0x7f1798003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17b0fcd700 1 -- 172.21.15.1:0/3172974667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1798004d90 msgr2=0x7f1798005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:22.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17b0fcd700 1 --2- 172.21.15.1:0/3172974667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1798004d90 0x7f1798005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17b0fcd700 1 -- 172.21.15.1:0/3172974667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17980035b0 con 0x7f1798004120 2024-03-18T19:02:22.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17b0fcd700 1 --2- 172.21.15.1:0/3172974667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1798004120 0x7f1798004850 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f179c009b60 tx=0x7f1798005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e0a48897219c1e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:22.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17a3fff700 1 -- 172.21.15.1:0/3172974667 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c00e6c0 con 0x7f1798004120 2024-03-18T19:02:22.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17a3fff700 1 -- 172.21.15.1:0/3172974667 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f179c00e820 con 0x7f1798004120 2024-03-18T19:02:22.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.474+0000 7f17a3fff700 1 -- 172.21.15.1:0/3172974667 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c00eb90 con 0x7f1798004120 2024-03-18T19:02:22.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.475+0000 7f17b3a32700 1 -- 172.21.15.1:0/3172974667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1798004120 msgr2=0x7f1798004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:22.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.475+0000 7f17b3a32700 1 --2- 172.21.15.1:0/3172974667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1798004120 0x7f1798004850 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f179c009b60 tx=0x7f1798005880 comp rx=0 tx=0).stop 2024-03-18T19:02:22.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.475+0000 7f17b1fcf700 1 -- 172.21.15.1:0/3172974667 reap_dead start 2024-03-18T19:02:22.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.475+0000 7f17b3a32700 1 -- 172.21.15.1:0/3172974667 shutdown_connections 2024-03-18T19:02:22.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.475+0000 7f17b3a32700 1 -- 172.21.15.1:0/3172974667 >> 172.21.15.1:0/3172974667 conn(0x7f17ac0cff30 msgr2=0x7f17ac0d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:22.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.475+0000 7f17b3a32700 1 -- 172.21.15.1:0/3172974667 shutdown_connections 2024-03-18T19:02:22.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.475+0000 7f17b3a32700 1 -- 172.21.15.1:0/3172974667 wait complete. 2024-03-18T19:02:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.477+0000 7f17b3a32700 1 Processor -- start 2024-03-18T19:02:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.477+0000 7f17b3a32700 1 -- start start 2024-03-18T19:02:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17b3a32700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1798004120 0x7f17ac1f6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17b3a32700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17ac071cc0 0x7f17ac0b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17b3a32700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17ac072140 0x7f17ac0c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17b3a32700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac0b5a50 con 0x7f1798004120 2024-03-18T19:02:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17b3a32700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac1f7500 con 0x7f17ac072140 2024-03-18T19:02:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17b3a32700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac0c7470 con 0x7f17ac071cc0 2024-03-18T19:02:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17b17ce700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1798004120 0x7f17ac1f6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17b17ce700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1798004120 0x7f17ac1f6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55456/0 (socket says 172.21.15.1:55456) 2024-03-18T19:02:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17b17ce700 1 -- 172.21.15.1:0/2761612458 learned_addr learned my addr 172.21.15.1:0/2761612458 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17b0fcd700 1 --2- 172.21.15.1:0/2761612458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17ac071cc0 0x7f17ac0b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17b1fcf700 1 --2- 172.21.15.1:0/2761612458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17ac072140 0x7f17ac0c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17b17ce700 1 -- 172.21.15.1:0/2761612458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17ac071cc0 msgr2=0x7f17ac0b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17b17ce700 1 --2- 172.21.15.1:0/2761612458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17ac071cc0 0x7f17ac0b2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17b17ce700 1 -- 172.21.15.1:0/2761612458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17ac072140 msgr2=0x7f17ac0c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17b17ce700 1 --2- 172.21.15.1:0/2761612458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17ac072140 0x7f17ac0c9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17b17ce700 1 -- 172.21.15.1:0/2761612458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ac0c8e40 con 0x7f1798004120 2024-03-18T19:02:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17b17ce700 1 --2- 172.21.15.1:0/2761612458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1798004120 0x7f17ac1f6fc0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f17a800f810 tx=0x7f17a800fbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17a1ffb700 1 -- 172.21.15.1:0/2761612458 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a80052e0 con 0x7f1798004120 2024-03-18T19:02:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17a1ffb700 1 -- 172.21.15.1:0/2761612458 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a8005440 con 0x7f1798004120 2024-03-18T19:02:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17b3a32700 1 -- 172.21.15.1:0/2761612458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17ac0c9120 con 0x7f1798004120 2024-03-18T19:02:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.478+0000 7f17a1ffb700 1 -- 172.21.15.1:0/2761612458 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a80038c0 con 0x7f1798004120 2024-03-18T19:02:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.479+0000 7f17b3a32700 1 -- 172.21.15.1:0/2761612458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17ac0c8af0 con 0x7f1798004120 2024-03-18T19:02:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.480+0000 7f17b3a32700 1 -- 172.21.15.1:0/2761612458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17ac0c8cb0 con 0x7f1798004120 2024-03-18T19:02:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.480+0000 7f17a1ffb700 1 -- 172.21.15.1:0/2761612458 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f17a8003a20 con 0x7f1798004120 2024-03-18T19:02:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.480+0000 7f17a1ffb700 1 --2- 172.21.15.1:0/2761612458 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f179006ae90 0x7f179006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.481+0000 7f17b0fcd700 1 --2- 172.21.15.1:0/2761612458 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f179006ae90 0x7f179006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.481+0000 7f17a1ffb700 1 -- 172.21.15.1:0/2761612458 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f17a80898a0 con 0x7f1798004120 2024-03-18T19:02:22.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.481+0000 7f17b0fcd700 1 --2- 172.21.15.1:0/2761612458 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f179006ae90 0x7f179006d350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f179c0090b0 tx=0x7f179c008030 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:22.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.485+0000 7f17a1ffb700 1 -- 172.21.15.1:0/2761612458 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f17a80545a0 con 0x7f1798004120 2024-03-18T19:02:22.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.715+0000 7f17b3a32700 1 -- 172.21.15.1:0/2761612458 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f17ac161c60 con 0x7f179006ae90 2024-03-18T19:02:22.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.780+0000 7f17a1ffb700 1 -- 172.21.15.1:0/2761612458 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f179c006c60 con 0x7f179006ae90 2024-03-18T19:02:22.778 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2024-03-18T19:02:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.783+0000 7f17b3a32700 1 -- 172.21.15.1:0/2761612458 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f179006ae90 msgr2=0x7f179006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.783+0000 7f17b3a32700 1 --2- 172.21.15.1:0/2761612458 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f179006ae90 0x7f179006d350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f179c0090b0 tx=0x7f179c008030 comp rx=0 tx=0).stop 2024-03-18T19:02:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.783+0000 7f17b3a32700 1 -- 172.21.15.1:0/2761612458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1798004120 msgr2=0x7f17ac1f6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.783+0000 7f17b3a32700 1 --2- 172.21.15.1:0/2761612458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1798004120 0x7f17ac1f6fc0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f17a800f810 tx=0x7f17a800fbd0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.783+0000 7f17b3a32700 1 -- 172.21.15.1:0/2761612458 shutdown_connections 2024-03-18T19:02:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.783+0000 7f17b3a32700 1 --2- 172.21.15.1:0/2761612458 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f179006ae90 0x7f179006d350 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.783+0000 7f17b3a32700 1 --2- 172.21.15.1:0/2761612458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17ac072140 0x7f17ac0c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.783+0000 7f17b3a32700 1 --2- 172.21.15.1:0/2761612458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1798004120 0x7f17ac1f6fc0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.783+0000 7f17b3a32700 1 --2- 172.21.15.1:0/2761612458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17ac071cc0 0x7f17ac0b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.783+0000 7f17b3a32700 1 -- 172.21.15.1:0/2761612458 >> 172.21.15.1:0/2761612458 conn(0x7f17ac0cff30 msgr2=0x7f17ac0afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:22.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.783+0000 7f17b3a32700 1 -- 172.21.15.1:0/2761612458 shutdown_connections 2024-03-18T19:02:22.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.783+0000 7f17b3a32700 1 -- 172.21.15.1:0/2761612458 wait complete. 2024-03-18T19:02:22.792 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2024-03-18T19:02:22.792 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir7 1m 2024-03-18T19:02:22.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.923+0000 7ff57a0c6700 1 Processor -- start 2024-03-18T19:02:22.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.923+0000 7ff57a0c6700 1 -- start start 2024-03-18T19:02:22.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.923+0000 7ff57a0c6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5740cf9d0 0x7ff5740717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.923+0000 7ff57a0c6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff574144500 0x7ff5740ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.923+0000 7ff57a0c6700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff574071d20 con 0x7ff5740cfdb0 2024-03-18T19:02:22.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.923+0000 7ff57a0c6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff57415ed40 con 0x7ff574144500 2024-03-18T19:02:22.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.923+0000 7ff57a0c6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff57415ef90 con 0x7ff5740cf9d0 2024-03-18T19:02:22.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff573fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff5740cfdb0 0x7ff5740cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32916/0 (socket says 172.21.15.1:32916) 2024-03-18T19:02:22.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff573fff700 1 -- 172.21.15.1:0/4275296793 learned_addr learned my addr 172.21.15.1:0/4275296793 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:22.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff572ffd700 1 --2- 172.21.15.1:0/4275296793 >> v2:172.21.15.165:3300/0 conn(0x7ff574144500 0x7ff5740ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff5727fc700 1 -- 172.21.15.1:0/4275296793 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542157392 0 0) 0x7ff56c001490 con 0x7ff5740cfdb0 2024-03-18T19:02:22.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff5737fe700 1 --2- 172.21.15.1:0/4275296793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5740cf9d0 0x7ff5740717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff5727fc700 1 -- 172.21.15.1:0/4275296793 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5600035b0 con 0x7ff5740cfdb0 2024-03-18T19:02:22.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff5727fc700 1 -- 172.21.15.1:0/4275296793 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 810906978 0 0) 0x7ff5600035b0 con 0x7ff5740cfdb0 2024-03-18T19:02:22.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff5727fc700 1 -- 172.21.15.1:0/4275296793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5740cf9d0 msgr2=0x7ff5740717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:22.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff5727fc700 1 --2- 172.21.15.1:0/4275296793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5740cf9d0 0x7ff5740717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff5727fc700 1 -- 172.21.15.1:0/4275296793 >> v2:172.21.15.165:3300/0 conn(0x7ff574144500 msgr2=0x7ff5740ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:22.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff5727fc700 1 --2- 172.21.15.1:0/4275296793 >> v2:172.21.15.165:3300/0 conn(0x7ff574144500 0x7ff5740ce4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff5727fc700 1 -- 172.21.15.1:0/4275296793 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57415f270 con 0x7ff5740cfdb0 2024-03-18T19:02:22.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff5727fc700 1 -- 172.21.15.1:0/4275296793 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7ff56c001b80 con 0x7ff5740cfdb0 2024-03-18T19:02:22.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff5727fc700 1 -- 172.21.15.1:0/4275296793 >> v1:172.21.15.1:6789/0 conn(0x7ff5740cfdb0 legacy=0x7ff5740cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:22.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff5727fc700 1 --2- 172.21.15.1:0/4275296793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff560003a00 0x7ff560003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff5727fc700 1 --2- 172.21.15.1:0/4275296793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff560004850 0x7ff560004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff5727fc700 1 --2- 172.21.15.1:0/4275296793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5740cf9d0 0x7ff560005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff5727fc700 1 -- 172.21.15.1:0/4275296793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57415ed40 con 0x7ff560003a00 2024-03-18T19:02:22.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff5727fc700 1 -- 172.21.15.1:0/4275296793 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57415ef90 con 0x7ff560004850 2024-03-18T19:02:22.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff5727fc700 1 -- 172.21.15.1:0/4275296793 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56000ba10 con 0x7ff5740cf9d0 2024-03-18T19:02:22.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff5737fe700 1 --2- 172.21.15.1:0/4275296793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff560003a00 0x7ff560003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.924+0000 7ff572ffd700 1 --2- 172.21.15.1:0/4275296793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff560004850 0x7ff560004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.925+0000 7ff573fff700 1 --2- 172.21.15.1:0/4275296793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5740cf9d0 0x7ff560005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.925+0000 7ff5737fe700 1 -- 172.21.15.1:0/4275296793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5740cf9d0 msgr2=0x7ff560005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.925+0000 7ff5737fe700 1 --2- 172.21.15.1:0/4275296793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5740cf9d0 0x7ff560005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.925+0000 7ff5737fe700 1 -- 172.21.15.1:0/4275296793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff560004850 msgr2=0x7ff560004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.925+0000 7ff5737fe700 1 --2- 172.21.15.1:0/4275296793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff560004850 0x7ff560004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.925+0000 7ff5737fe700 1 -- 172.21.15.1:0/4275296793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5600035b0 con 0x7ff560003a00 2024-03-18T19:02:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.925+0000 7ff572ffd700 1 --2- 172.21.15.1:0/4275296793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff560004850 0x7ff560004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.925+0000 7ff5737fe700 1 --2- 172.21.15.1:0/4275296793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff560003a00 0x7ff560003e20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff55c009b90 tx=0x7ff560005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=873ab570a1d35a46 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.925+0000 7ff5727fc700 1 -- 172.21.15.1:0/4275296793 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55c00e6c0 con 0x7ff560003a00 2024-03-18T19:02:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.925+0000 7ff5727fc700 1 -- 172.21.15.1:0/4275296793 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff55c00e820 con 0x7ff560003a00 2024-03-18T19:02:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.925+0000 7ff5727fc700 1 -- 172.21.15.1:0/4275296793 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55c00eb90 con 0x7ff560003a00 2024-03-18T19:02:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.925+0000 7ff57a0c6700 1 -- 172.21.15.1:0/4275296793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff560003a00 msgr2=0x7ff560003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.925+0000 7ff57a0c6700 1 --2- 172.21.15.1:0/4275296793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff560003a00 0x7ff560003e20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff55c009b90 tx=0x7ff560005880 comp rx=0 tx=0).stop 2024-03-18T19:02:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.925+0000 7ff573fff700 1 -- 172.21.15.1:0/4275296793 reap_dead start 2024-03-18T19:02:22.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.926+0000 7ff57a0c6700 1 -- 172.21.15.1:0/4275296793 shutdown_connections 2024-03-18T19:02:22.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.926+0000 7ff57a0c6700 1 -- 172.21.15.1:0/4275296793 >> 172.21.15.1:0/4275296793 conn(0x7ff574066b70 msgr2=0x7ff57406a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:22.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.926+0000 7ff57a0c6700 1 -- 172.21.15.1:0/4275296793 shutdown_connections 2024-03-18T19:02:22.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.926+0000 7ff57a0c6700 1 -- 172.21.15.1:0/4275296793 wait complete. 2024-03-18T19:02:22.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.927+0000 7ff57a0c6700 1 Processor -- start 2024-03-18T19:02:22.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.927+0000 7ff57a0c6700 1 -- start start 2024-03-18T19:02:22.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff57a0c6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff560003a00 0x7ff5741418f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff57a0c6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff574159c80 0x7ff57413fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff57a0c6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff57415a100 0x7ff57413cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff57a0c6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff574140850 con 0x7ff574159c80 2024-03-18T19:02:22.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff57a0c6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff574140b30 con 0x7ff57415a100 2024-03-18T19:02:22.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff57a0c6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff574140e10 con 0x7ff560003a00 2024-03-18T19:02:22.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff572ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff574159c80 0x7ff57413fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff572ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff574159c80 0x7ff57413fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55476/0 (socket says 172.21.15.1:55476) 2024-03-18T19:02:22.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff572ffd700 1 -- 172.21.15.1:0/3718874726 learned_addr learned my addr 172.21.15.1:0/3718874726 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:22.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff5737fe700 1 --2- 172.21.15.1:0/3718874726 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff560003a00 0x7ff5741418f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff573fff700 1 --2- 172.21.15.1:0/3718874726 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff57415a100 0x7ff57413cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff572ffd700 1 -- 172.21.15.1:0/3718874726 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff560003a00 msgr2=0x7ff5741418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:22.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff572ffd700 1 --2- 172.21.15.1:0/3718874726 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff560003a00 0x7ff5741418f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff572ffd700 1 -- 172.21.15.1:0/3718874726 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff57415a100 msgr2=0x7ff57413cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff572ffd700 1 --2- 172.21.15.1:0/3718874726 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff57415a100 0x7ff57413cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff572ffd700 1 -- 172.21.15.1:0/3718874726 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5741410f0 con 0x7ff574159c80 2024-03-18T19:02:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff572ffd700 1 --2- 172.21.15.1:0/3718874726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff574159c80 0x7ff57413fdd0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff5640118d0 tx=0x7ff564011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff56bfff700 1 -- 172.21.15.1:0/3718874726 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff564004220 con 0x7ff574159c80 2024-03-18T19:02:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff56bfff700 1 -- 172.21.15.1:0/3718874726 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff564004380 con 0x7ff574159c80 2024-03-18T19:02:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff56bfff700 1 -- 172.21.15.1:0/3718874726 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5640036a0 con 0x7ff574159c80 2024-03-18T19:02:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.928+0000 7ff57a0c6700 1 -- 172.21.15.1:0/3718874726 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5741413d0 con 0x7ff574159c80 2024-03-18T19:02:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.929+0000 7ff57a0c6700 1 -- 172.21.15.1:0/3718874726 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff57415e150 con 0x7ff574159c80 2024-03-18T19:02:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.930+0000 7ff57a0c6700 1 -- 172.21.15.1:0/3718874726 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff574071760 con 0x7ff574159c80 2024-03-18T19:02:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.930+0000 7ff56bfff700 1 -- 172.21.15.1:0/3718874726 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7ff564003800 con 0x7ff574159c80 2024-03-18T19:02:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.931+0000 7ff56bfff700 1 --2- 172.21.15.1:0/3718874726 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff55406aec0 0x7ff55406d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.931+0000 7ff5737fe700 1 --2- 172.21.15.1:0/3718874726 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff55406aec0 0x7ff55406d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.931+0000 7ff56bfff700 1 -- 172.21.15.1:0/3718874726 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7ff564089e20 con 0x7ff574159c80 2024-03-18T19:02:22.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.931+0000 7ff5737fe700 1 --2- 172.21.15.1:0/3718874726 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff55406aec0 0x7ff55406d380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff55c009b30 tx=0x7ff55c008030 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:22.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:22.935+0000 7ff56bfff700 1 -- 172.21.15.1:0/3718874726 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff564054a80 con 0x7ff574159c80 2024-03-18T19:02:23.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.170+0000 7ff57a0c6700 1 -- 172.21.15.1:0/3718874726 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5740cb950 con 0x7ff55406aec0 2024-03-18T19:02:23.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.235+0000 7ff56bfff700 1 -- 172.21.15.1:0/3718874726 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7ff55c00e990 con 0x7ff55406aec0 2024-03-18T19:02:23.232 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2024-03-18T19:02:23.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.238+0000 7ff57a0c6700 1 -- 172.21.15.1:0/3718874726 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff55406aec0 msgr2=0x7ff55406d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:23.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.238+0000 7ff57a0c6700 1 --2- 172.21.15.1:0/3718874726 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff55406aec0 0x7ff55406d380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff55c009b30 tx=0x7ff55c008030 comp rx=0 tx=0).stop 2024-03-18T19:02:23.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.238+0000 7ff57a0c6700 1 -- 172.21.15.1:0/3718874726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff574159c80 msgr2=0x7ff57413fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:23.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.238+0000 7ff57a0c6700 1 --2- 172.21.15.1:0/3718874726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff574159c80 0x7ff57413fdd0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff5640118d0 tx=0x7ff564011c90 comp rx=0 tx=0).stop 2024-03-18T19:02:23.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.238+0000 7ff57a0c6700 1 -- 172.21.15.1:0/3718874726 shutdown_connections 2024-03-18T19:02:23.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.238+0000 7ff57a0c6700 1 --2- 172.21.15.1:0/3718874726 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff55406aec0 0x7ff55406d380 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:23.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.238+0000 7ff57a0c6700 1 --2- 172.21.15.1:0/3718874726 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff560003a00 0x7ff5741418f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:23.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.238+0000 7ff57a0c6700 1 --2- 172.21.15.1:0/3718874726 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff57415a100 0x7ff57413cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:23.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.238+0000 7ff57a0c6700 1 --2- 172.21.15.1:0/3718874726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff574159c80 0x7ff57413fdd0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:23.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.238+0000 7ff57a0c6700 1 -- 172.21.15.1:0/3718874726 >> 172.21.15.1:0/3718874726 conn(0x7ff574066b70 msgr2=0x7ff5740cc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:23.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.238+0000 7ff57a0c6700 1 -- 172.21.15.1:0/3718874726 shutdown_connections 2024-03-18T19:02:23.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.238+0000 7ff57a0c6700 1 -- 172.21.15.1:0/3718874726 wait complete. 2024-03-18T19:02:23.246 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2024-03-18T19:02:23.247 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir8 1m 2024-03-18T19:02:23.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.378+0000 7faec1bc3700 1 Processor -- start 2024-03-18T19:02:23.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.378+0000 7faec1bc3700 1 -- start start 2024-03-18T19:02:23.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.379+0000 7faec1bc3700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7faebc15d5b0 0x7faebc15d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:23.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.379+0000 7faec1bc3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faebc099a40 0x7faebc099ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:23.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.379+0000 7faec1bc3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faebc09a410 0x7faebc06ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:23.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.379+0000 7faec1bc3700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faebc0cbbb0 con 0x7faebc099a40 2024-03-18T19:02:23.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.379+0000 7faec1bc3700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faebc09a860 con 0x7faebc15d5b0 2024-03-18T19:02:23.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.379+0000 7faec1bc3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faebc0b3b00 con 0x7faebc09a410 2024-03-18T19:02:23.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.379+0000 7faebaffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faebc099a40 0x7faebc099ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.379+0000 7faebaffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faebc099a40 0x7faebc099ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55484/0 (socket says 172.21.15.1:55484) 2024-03-18T19:02:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.379+0000 7faebaffd700 1 -- 172.21.15.1:0/3078729902 learned_addr learned my addr 172.21.15.1:0/3078729902 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.379+0000 7faebb7fe700 1 --2- 172.21.15.1:0/3078729902 >> v2:172.21.15.165:3300/0 conn(0x7faebc15d5b0 0x7faebc15d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.379+0000 7faebbfff700 1 --2- 172.21.15.1:0/3078729902 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faebc09a410 0x7faebc06ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.379+0000 7faebaffd700 1 -- 172.21.15.1:0/3078729902 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faebc09a410 msgr2=0x7faebc06ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.379+0000 7faebaffd700 1 --2- 172.21.15.1:0/3078729902 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faebc09a410 0x7faebc06ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.379+0000 7faebaffd700 1 -- 172.21.15.1:0/3078729902 >> v2:172.21.15.165:3300/0 conn(0x7faebc15d5b0 msgr2=0x7faebc15d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.379+0000 7faebaffd700 1 --2- 172.21.15.1:0/3078729902 >> v2:172.21.15.165:3300/0 conn(0x7faebc15d5b0 0x7faebc15d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.379+0000 7faebaffd700 1 -- 172.21.15.1:0/3078729902 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faebc0cf7f0 con 0x7faebc099a40 2024-03-18T19:02:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.379+0000 7faebaffd700 1 --2- 172.21.15.1:0/3078729902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faebc099a40 0x7faebc099ed0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7faebc072080 tx=0x7faeb0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8476b5c9098cc93 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.379+0000 7faeba7fc700 1 -- 172.21.15.1:0/3078729902 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb000b860 con 0x7faebc099a40 2024-03-18T19:02:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.380+0000 7faeba7fc700 1 -- 172.21.15.1:0/3078729902 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb000b9c0 con 0x7faebc099a40 2024-03-18T19:02:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.380+0000 7faeba7fc700 1 -- 172.21.15.1:0/3078729902 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb000bd50 con 0x7faebc099a40 2024-03-18T19:02:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.380+0000 7faec1bc3700 1 -- 172.21.15.1:0/3078729902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faebc099a40 msgr2=0x7faebc099ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.380+0000 7faec1bc3700 1 --2- 172.21.15.1:0/3078729902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faebc099a40 0x7faebc099ed0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7faebc072080 tx=0x7faeb0009c00 comp rx=0 tx=0).stop 2024-03-18T19:02:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.380+0000 7faec1bc3700 1 -- 172.21.15.1:0/3078729902 shutdown_connections 2024-03-18T19:02:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.380+0000 7faec1bc3700 1 --2- 172.21.15.1:0/3078729902 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faebc09a410 0x7faebc06ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.380+0000 7faec1bc3700 1 --2- 172.21.15.1:0/3078729902 >> v2:172.21.15.165:3300/0 conn(0x7faebc15d5b0 0x7faebc15d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.380+0000 7faec1bc3700 1 --2- 172.21.15.1:0/3078729902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faebc099a40 0x7faebc099ed0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.380+0000 7faec1bc3700 1 -- 172.21.15.1:0/3078729902 >> 172.21.15.1:0/3078729902 conn(0x7faebc0b2bd0 msgr2=0x7faebc0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.380+0000 7faec1bc3700 1 -- 172.21.15.1:0/3078729902 shutdown_connections 2024-03-18T19:02:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.380+0000 7faec1bc3700 1 -- 172.21.15.1:0/3078729902 wait complete. 2024-03-18T19:02:23.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.382+0000 7faec1bc3700 1 Processor -- start 2024-03-18T19:02:23.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.382+0000 7faec1bc3700 1 -- start start 2024-03-18T19:02:23.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.383+0000 7faec1bc3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faebc099a40 0x7faebc0c5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:23.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.383+0000 7faec1bc3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faebc09a410 0x7faebc1f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:23.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.383+0000 7faec1bc3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faebc15d5b0 0x7faebc0c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:23.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.383+0000 7faec1bc3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc1f2f50 con 0x7faebc09a410 2024-03-18T19:02:23.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.383+0000 7faec1bc3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc0c6d20 con 0x7faebc099a40 2024-03-18T19:02:23.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.383+0000 7faec1bc3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faebc0c7000 con 0x7faebc15d5b0 2024-03-18T19:02:23.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.383+0000 7faebaffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faebc09a410 0x7faebc1f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:23.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.383+0000 7faebaffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faebc09a410 0x7faebc1f2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55486/0 (socket says 172.21.15.1:55486) 2024-03-18T19:02:23.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.383+0000 7faebaffd700 1 -- 172.21.15.1:0/3285159899 learned_addr learned my addr 172.21.15.1:0/3285159899 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:23.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.383+0000 7faebb7fe700 1 --2- 172.21.15.1:0/3285159899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faebc099a40 0x7faebc0c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:23.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.383+0000 7faebbfff700 1 --2- 172.21.15.1:0/3285159899 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faebc15d5b0 0x7faebc0c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:23.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.383+0000 7faebaffd700 1 -- 172.21.15.1:0/3285159899 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faebc15d5b0 msgr2=0x7faebc0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:23.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.383+0000 7faebaffd700 1 --2- 172.21.15.1:0/3285159899 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faebc15d5b0 0x7faebc0c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:23.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.383+0000 7faebaffd700 1 -- 172.21.15.1:0/3285159899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faebc099a40 msgr2=0x7faebc0c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:23.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.383+0000 7faebaffd700 1 --2- 172.21.15.1:0/3285159899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faebc099a40 0x7faebc0c5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:23.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.383+0000 7faebaffd700 1 -- 172.21.15.1:0/3285159899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb0009550 con 0x7faebc09a410 2024-03-18T19:02:23.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.383+0000 7faebb7fe700 1 --2- 172.21.15.1:0/3285159899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faebc099a40 0x7faebc0c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:23.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.383+0000 7faebaffd700 1 --2- 172.21.15.1:0/3285159899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faebc09a410 0x7faebc1f2a10 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7faeb0009130 tx=0x7faeb0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:23.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.384+0000 7faea3fff700 1 -- 172.21.15.1:0/3285159899 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb00044d0 con 0x7faebc09a410 2024-03-18T19:02:23.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.384+0000 7faea3fff700 1 -- 172.21.15.1:0/3285159899 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb001e070 con 0x7faebc09a410 2024-03-18T19:02:23.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.384+0000 7faec1bc3700 1 -- 172.21.15.1:0/3285159899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faebc0c72e0 con 0x7faebc09a410 2024-03-18T19:02:23.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.384+0000 7faea3fff700 1 -- 172.21.15.1:0/3285159899 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb00038e0 con 0x7faebc09a410 2024-03-18T19:02:23.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.384+0000 7faec1bc3700 1 -- 172.21.15.1:0/3285159899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faebc0c78a0 con 0x7faebc09a410 2024-03-18T19:02:23.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.385+0000 7faec1bc3700 1 -- 172.21.15.1:0/3285159899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faebc0c7a30 con 0x7faebc09a410 2024-03-18T19:02:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.385+0000 7faea3fff700 1 -- 172.21.15.1:0/3285159899 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7faeb0003a40 con 0x7faebc09a410 2024-03-18T19:02:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.385+0000 7faea3fff700 1 --2- 172.21.15.1:0/3285159899 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faea806ae90 0x7faea806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.386+0000 7faea3fff700 1 -- 172.21.15.1:0/3285159899 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7faeb0020070 con 0x7faebc09a410 2024-03-18T19:02:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.386+0000 7faebb7fe700 1 --2- 172.21.15.1:0/3285159899 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faea806ae90 0x7faea806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.386+0000 7faebb7fe700 1 --2- 172.21.15.1:0/3285159899 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faea806ae90 0x7faea806d350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faeac0099f0 tx=0x7faeac008040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:23.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.390+0000 7faea3fff700 1 -- 172.21.15.1:0/3285159899 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faeb0055a20 con 0x7faebc09a410 2024-03-18T19:02:23.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.621+0000 7faec1bc3700 1 -- 172.21.15.1:0/3285159899 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7faebc0d07b0 con 0x7faea806ae90 2024-03-18T19:02:23.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.687+0000 7faea3fff700 1 -- 172.21.15.1:0/3285159899 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7faeac009680 con 0x7faea806ae90 2024-03-18T19:02:23.684 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2024-03-18T19:02:23.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.690+0000 7faec1bc3700 1 -- 172.21.15.1:0/3285159899 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faea806ae90 msgr2=0x7faea806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:23.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.690+0000 7faec1bc3700 1 --2- 172.21.15.1:0/3285159899 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faea806ae90 0x7faea806d350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faeac0099f0 tx=0x7faeac008040 comp rx=0 tx=0).stop 2024-03-18T19:02:23.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.690+0000 7faec1bc3700 1 -- 172.21.15.1:0/3285159899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faebc09a410 msgr2=0x7faebc1f2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:23.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.690+0000 7faec1bc3700 1 --2- 172.21.15.1:0/3285159899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faebc09a410 0x7faebc1f2a10 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7faeb0009130 tx=0x7faeb0004750 comp rx=0 tx=0).stop 2024-03-18T19:02:23.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.690+0000 7faec1bc3700 1 -- 172.21.15.1:0/3285159899 shutdown_connections 2024-03-18T19:02:23.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.690+0000 7faec1bc3700 1 --2- 172.21.15.1:0/3285159899 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faea806ae90 0x7faea806d350 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:23.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.690+0000 7faec1bc3700 1 --2- 172.21.15.1:0/3285159899 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faebc15d5b0 0x7faebc0c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:23.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.690+0000 7faec1bc3700 1 --2- 172.21.15.1:0/3285159899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faebc099a40 0x7faebc0c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:23.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.690+0000 7faec1bc3700 1 --2- 172.21.15.1:0/3285159899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faebc09a410 0x7faebc1f2a10 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:23.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.690+0000 7faec1bc3700 1 -- 172.21.15.1:0/3285159899 >> 172.21.15.1:0/3285159899 conn(0x7faebc0b2bd0 msgr2=0x7faebc06c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:23.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.690+0000 7faec1bc3700 1 -- 172.21.15.1:0/3285159899 shutdown_connections 2024-03-18T19:02:23.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.690+0000 7faec1bc3700 1 -- 172.21.15.1:0/3285159899 wait complete. 2024-03-18T19:02:23.698 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2024-03-18T19:02:23.699 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir9 1m 2024-03-18T19:02:23.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.830+0000 7f5f05f79700 1 Processor -- start 2024-03-18T19:02:23.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.830+0000 7f5f05f79700 1 -- start start 2024-03-18T19:02:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.831+0000 7f5f05f79700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5f00071d30 0x7f5f000add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.831+0000 7f5f05f79700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f000ce8f0 con 0x7f5f000cb9e0 2024-03-18T19:02:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.831+0000 7f5f05f79700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f000ae260 con 0x7f5f0015a1d0 2024-03-18T19:02:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.831+0000 7f5f05f79700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f000cdb00 con 0x7f5f00071d30 2024-03-18T19:02:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.831+0000 7f5efffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5f000cb9e0 0x7f5f000cbe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32924/0 (socket says 172.21.15.1:32924) 2024-03-18T19:02:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.831+0000 7f5efffff700 1 -- 172.21.15.1:0/1225437061 learned_addr learned my addr 172.21.15.1:0/1225437061 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.831+0000 7f5efeffd700 1 --2- 172.21.15.1:0/1225437061 >> v2:172.21.15.165:3300/0 conn(0x7f5f00071d30 0x7f5f000add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.831+0000 7f5efe7fc700 1 -- 172.21.15.1:0/1225437061 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710421942 0 0) 0x7f5f000ce8f0 con 0x7f5f000cb9e0 2024-03-18T19:02:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.831+0000 7f5efe7fc700 1 -- 172.21.15.1:0/1225437061 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee80035b0 con 0x7f5f000cb9e0 2024-03-18T19:02:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.831+0000 7f5efeffd700 1 -- 172.21.15.1:0/1225437061 >> v1:172.21.15.165:6789/0 conn(0x7f5f0015a1d0 legacy=0x7f5f00071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.831+0000 7f5efe7fc700 1 -- 172.21.15.1:0/1225437061 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3048945905 0 0) 0x7f5ee80035b0 con 0x7f5f000cb9e0 2024-03-18T19:02:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.831+0000 7f5efeffd700 1 -- 172.21.15.1:0/1225437061 >> v1:172.21.15.1:6789/0 conn(0x7f5f000cb9e0 legacy=0x7f5f000cbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.831+0000 7f5efeffd700 1 -- 172.21.15.1:0/1225437061 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f000cdce0 con 0x7f5f00071d30 2024-03-18T19:02:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.832+0000 7f5efeffd700 1 --2- 172.21.15.1:0/1225437061 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f00071d30 0x7f5f000add20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4009840 tx=0x7f5ef4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4315019223e1759c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.832+0000 7f5efe7fc700 1 -- 172.21.15.1:0/1225437061 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef80019f0 con 0x7f5f00071d30 2024-03-18T19:02:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.832+0000 7f5efe7fc700 1 -- 172.21.15.1:0/1225437061 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef4004030 con 0x7f5f00071d30 2024-03-18T19:02:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.832+0000 7f5efe7fc700 1 -- 172.21.15.1:0/1225437061 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef40043a0 con 0x7f5f00071d30 2024-03-18T19:02:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.832+0000 7f5f05f79700 1 -- 172.21.15.1:0/1225437061 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f00071d30 msgr2=0x7f5f000add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.832+0000 7f5f05f79700 1 --2- 172.21.15.1:0/1225437061 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f00071d30 0x7f5f000add20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4009840 tx=0x7f5ef4009b50 comp rx=0 tx=0).stop 2024-03-18T19:02:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.832+0000 7f5f05f79700 1 -- 172.21.15.1:0/1225437061 shutdown_connections 2024-03-18T19:02:23.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.832+0000 7f5f05f79700 1 --2- 172.21.15.1:0/1225437061 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f00071d30 0x7f5f000add20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:23.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.832+0000 7f5f05f79700 1 -- 172.21.15.1:0/1225437061 >> 172.21.15.1:0/1225437061 conn(0x7f5f000cc9b0 msgr2=0x7f5f000ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:23.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.832+0000 7f5f05f79700 1 -- 172.21.15.1:0/1225437061 shutdown_connections 2024-03-18T19:02:23.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.832+0000 7f5f05f79700 1 -- 172.21.15.1:0/1225437061 wait complete. 2024-03-18T19:02:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.834+0000 7f5f05f79700 1 Processor -- start 2024-03-18T19:02:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.834+0000 7f5f05f79700 1 -- start start 2024-03-18T19:02:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.834+0000 7f5f05f79700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f00071d30 0x7f5f0006f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.834+0000 7f5f05f79700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5f000cb9e0 0x7f5f000b4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.834+0000 7f5f05f79700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f0015a1d0 0x7f5f000b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.834+0000 7f5f05f79700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f001f14c0 con 0x7f5f00071d30 2024-03-18T19:02:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.834+0000 7f5f05f79700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f001f17a0 con 0x7f5f0015a1d0 2024-03-18T19:02:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.834+0000 7f5f05f79700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f001f1a80 con 0x7f5f000cb9e0 2024-03-18T19:02:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.835+0000 7f5eff7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f00071d30 0x7f5f0006f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.835+0000 7f5eff7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f00071d30 0x7f5f0006f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55502/0 (socket says 172.21.15.1:55502) 2024-03-18T19:02:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.835+0000 7f5eff7fe700 1 -- 172.21.15.1:0/3759355602 learned_addr learned my addr 172.21.15.1:0/3759355602 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.835+0000 7f5efffff700 1 --2- 172.21.15.1:0/3759355602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f0015a1d0 0x7f5f000b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.835+0000 7f5efeffd700 1 --2- 172.21.15.1:0/3759355602 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5f000cb9e0 0x7f5f000b4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.835+0000 7f5eff7fe700 1 -- 172.21.15.1:0/3759355602 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5f000cb9e0 msgr2=0x7f5f000b4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.835+0000 7f5eff7fe700 1 --2- 172.21.15.1:0/3759355602 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5f000cb9e0 0x7f5f000b4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.835+0000 7f5eff7fe700 1 -- 172.21.15.1:0/3759355602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f0015a1d0 msgr2=0x7f5f000b52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.835+0000 7f5eff7fe700 1 --2- 172.21.15.1:0/3759355602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f0015a1d0 0x7f5f000b52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.835+0000 7f5eff7fe700 1 -- 172.21.15.1:0/3759355602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef40094a0 con 0x7f5f00071d30 2024-03-18T19:02:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.835+0000 7f5eff7fe700 1 --2- 172.21.15.1:0/3759355602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f00071d30 0x7f5f0006f860 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5ef000c500 tx=0x7f5ef000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.835+0000 7f5ee7fff700 1 -- 172.21.15.1:0/3759355602 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef00054e0 con 0x7f5f00071d30 2024-03-18T19:02:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.835+0000 7f5ee7fff700 1 -- 172.21.15.1:0/3759355602 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef00036d0 con 0x7f5f00071d30 2024-03-18T19:02:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.835+0000 7f5f05f79700 1 -- 172.21.15.1:0/3759355602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f001f1d60 con 0x7f5f00071d30 2024-03-18T19:02:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.835+0000 7f5ee7fff700 1 -- 172.21.15.1:0/3759355602 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0003a80 con 0x7f5f00071d30 2024-03-18T19:02:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.835+0000 7f5f05f79700 1 -- 172.21.15.1:0/3759355602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f001f22a0 con 0x7f5f00071d30 2024-03-18T19:02:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.836+0000 7f5f05f79700 1 -- 172.21.15.1:0/3759355602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f0015dd90 con 0x7f5f00071d30 2024-03-18T19:02:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.837+0000 7f5ee7fff700 1 -- 172.21.15.1:0/3759355602 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f5ef0003be0 con 0x7f5f00071d30 2024-03-18T19:02:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.837+0000 7f5ee7fff700 1 --2- 172.21.15.1:0/3759355602 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5eec06ae90 0x7f5eec06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.837+0000 7f5ee7fff700 1 -- 172.21.15.1:0/3759355602 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f5ef0088dd0 con 0x7f5f00071d30 2024-03-18T19:02:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.837+0000 7f5efeffd700 1 --2- 172.21.15.1:0/3759355602 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5eec06ae90 0x7f5eec06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.838+0000 7f5efeffd700 1 --2- 172.21.15.1:0/3759355602 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5eec06ae90 0x7f5eec06d350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4000c00 tx=0x7f5ef4008000 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:23.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:23.842+0000 7f5ee7fff700 1 -- 172.21.15.1:0/3759355602 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5f0015dd90 con 0x7f5f00071d30 2024-03-18T19:02:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.074+0000 7f5f05f79700 1 -- 172.21.15.1:0/3759355602 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f001f1910 con 0x7f5eec06ae90 2024-03-18T19:02:24.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.143+0000 7f5ee7fff700 1 -- 172.21.15.1:0/3759355602 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f5f001f1910 con 0x7f5eec06ae90 2024-03-18T19:02:24.140 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2024-03-18T19:02:24.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.145+0000 7f5f05f79700 1 -- 172.21.15.1:0/3759355602 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5eec06ae90 msgr2=0x7f5eec06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:24.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.145+0000 7f5f05f79700 1 --2- 172.21.15.1:0/3759355602 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5eec06ae90 0x7f5eec06d350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4000c00 tx=0x7f5ef4008000 comp rx=0 tx=0).stop 2024-03-18T19:02:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.145+0000 7f5f05f79700 1 -- 172.21.15.1:0/3759355602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f00071d30 msgr2=0x7f5f0006f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.145+0000 7f5f05f79700 1 --2- 172.21.15.1:0/3759355602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f00071d30 0x7f5f0006f860 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5ef000c500 tx=0x7f5ef000c810 comp rx=0 tx=0).stop 2024-03-18T19:02:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.145+0000 7f5f05f79700 1 -- 172.21.15.1:0/3759355602 shutdown_connections 2024-03-18T19:02:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.145+0000 7f5f05f79700 1 --2- 172.21.15.1:0/3759355602 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5eec06ae90 0x7f5eec06d350 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.145+0000 7f5f05f79700 1 --2- 172.21.15.1:0/3759355602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f0015a1d0 0x7f5f000b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.145+0000 7f5f05f79700 1 --2- 172.21.15.1:0/3759355602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f00071d30 0x7f5f0006f860 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.145+0000 7f5f05f79700 1 --2- 172.21.15.1:0/3759355602 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5f000cb9e0 0x7f5f000b4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.145+0000 7f5f05f79700 1 -- 172.21.15.1:0/3759355602 >> 172.21.15.1:0/3759355602 conn(0x7f5f000cc9b0 msgr2=0x7f5f000ca560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.145+0000 7f5f05f79700 1 -- 172.21.15.1:0/3759355602 shutdown_connections 2024-03-18T19:02:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.145+0000 7f5f05f79700 1 -- 172.21.15.1:0/3759355602 wait complete. 2024-03-18T19:02:24.154 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2024-03-18T19:02:24.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:24.155 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2024-03-18T19:02:24.178 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:24.178 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:24.178 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:24.178 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:24.178 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:24.178 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:24.178 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:24.178 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:24.178 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:24.178 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:24.178 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:24.179 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-18-18_52_00_UTC', 'scheduled-2024-03-18-18_53_00_UTC', 'scheduled-2024-03-18-18_54_00_UTC', 'scheduled-2024-03-18-18_55_00_UTC', 'scheduled-2024-03-18-18_56_00_UTC', 'scheduled-2024-03-18-18_57_00_UTC', 'scheduled-2024-03-18-18_58_00_UTC', 'scheduled-2024-03-18-18_59_00_UTC', 'scheduled-2024-03-18-19_00_00_UTC', 'scheduled-2024-03-18-19_01_00_UTC', 'scheduled-2024-03-18-19_02_00_UTC'] 2024-03-18T19:02:24.179 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir0 2024-03-18T19:02:24.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.349+0000 7fb27d7e2700 1 Processor -- start 2024-03-18T19:02:24.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.349+0000 7fb27d7e2700 1 -- start start 2024-03-18T19:02:24.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.349+0000 7fb27d7e2700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb27815d5f0 0x7fb2780d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:24.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.350+0000 7fb27d7e2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2780cfbf0 0x7fb2780d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:24.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.350+0000 7fb27d7e2700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2780d1910 con 0x7fb2780d3a80 2024-03-18T19:02:24.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.350+0000 7fb27d7e2700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2780d05a0 con 0x7fb2780cfbf0 2024-03-18T19:02:24.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.350+0000 7fb27d7e2700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb27815d9d0 con 0x7fb27815d5f0 2024-03-18T19:02:24.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.350+0000 7fb2777fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2780cfbf0 0x7fb2780d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:24.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.350+0000 7fb2777fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2780cfbf0 0x7fb2780d0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55516/0 (socket says 172.21.15.1:55516) 2024-03-18T19:02:24.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.350+0000 7fb2777fe700 1 -- 172.21.15.1:0/380837594 learned_addr learned my addr 172.21.15.1:0/380837594 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:24.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.350+0000 7fb276ffd700 1 --2- 172.21.15.1:0/380837594 >> v2:172.21.15.165:3300/0 conn(0x7fb27815d5f0 0x7fb2780d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:24.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.350+0000 7fb2777fe700 1 -- 172.21.15.1:0/380837594 >> v2:172.21.15.165:3300/0 conn(0x7fb27815d5f0 msgr2=0x7fb2780d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:24.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.350+0000 7fb2777fe700 1 --2- 172.21.15.1:0/380837594 >> v2:172.21.15.165:3300/0 conn(0x7fb27815d5f0 0x7fb2780d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:24.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.350+0000 7fb2777fe700 1 -- 172.21.15.1:0/380837594 >> v1:172.21.15.165:6789/0 conn(0x7fb2780d3a80 legacy=0x7fb2780cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:24.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.350+0000 7fb2777fe700 1 -- 172.21.15.1:0/380837594 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2780cf150 con 0x7fb2780cfbf0 2024-03-18T19:02:24.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.350+0000 7fb2777fe700 1 --2- 172.21.15.1:0/380837594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2780cfbf0 0x7fb2780d0060 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fb278067f80 tx=0x7fb27000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5716e30fe2fb478 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:24.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.350+0000 7fb275ffb700 1 -- 172.21.15.1:0/380837594 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27000e070 con 0x7fb2780cfbf0 2024-03-18T19:02:24.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.350+0000 7fb275ffb700 1 -- 172.21.15.1:0/380837594 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27000b910 con 0x7fb2780cfbf0 2024-03-18T19:02:24.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.351+0000 7fb275ffb700 1 -- 172.21.15.1:0/380837594 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27000bcc0 con 0x7fb2780cfbf0 2024-03-18T19:02:24.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.351+0000 7fb27d7e2700 1 -- 172.21.15.1:0/380837594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2780cfbf0 msgr2=0x7fb2780d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:24.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.351+0000 7fb27d7e2700 1 --2- 172.21.15.1:0/380837594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2780cfbf0 0x7fb2780d0060 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fb278067f80 tx=0x7fb27000b330 comp rx=0 tx=0).stop 2024-03-18T19:02:24.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.351+0000 7fb27d7e2700 1 -- 172.21.15.1:0/380837594 shutdown_connections 2024-03-18T19:02:24.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.351+0000 7fb27d7e2700 1 --2- 172.21.15.1:0/380837594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2780cfbf0 0x7fb2780d0060 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:24.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.351+0000 7fb27d7e2700 1 --2- 172.21.15.1:0/380837594 >> v2:172.21.15.165:3300/0 conn(0x7fb27815d5f0 0x7fb2780d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:24.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.351+0000 7fb27d7e2700 1 -- 172.21.15.1:0/380837594 >> 172.21.15.1:0/380837594 conn(0x7fb27809c540 msgr2=0x7fb2780d3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:24.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.351+0000 7fb27d7e2700 1 -- 172.21.15.1:0/380837594 shutdown_connections 2024-03-18T19:02:24.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.351+0000 7fb27d7e2700 1 -- 172.21.15.1:0/380837594 wait complete. 2024-03-18T19:02:24.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.352+0000 7fb27d7e2700 1 Processor -- start 2024-03-18T19:02:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.352+0000 7fb27d7e2700 1 -- start start 2024-03-18T19:02:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.353+0000 7fb27d7e2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2780cfbf0 0x7fb278159870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.353+0000 7fb27d7e2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2780d3a80 0x7fb2780c7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.353+0000 7fb27d7e2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb27815d5f0 0x7fb2780ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.353+0000 7fb27d7e2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2780c7ad0 con 0x7fb2780d3a80 2024-03-18T19:02:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.353+0000 7fb27d7e2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb278159db0 con 0x7fb27815d5f0 2024-03-18T19:02:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.353+0000 7fb27d7e2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2780c6e50 con 0x7fb2780cfbf0 2024-03-18T19:02:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.353+0000 7fb2767fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2780d3a80 0x7fb2780c7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.353+0000 7fb2767fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2780d3a80 0x7fb2780c7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55530/0 (socket says 172.21.15.1:55530) 2024-03-18T19:02:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.353+0000 7fb2767fc700 1 -- 172.21.15.1:0/2496977874 learned_addr learned my addr 172.21.15.1:0/2496977874 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.353+0000 7fb2777fe700 1 --2- 172.21.15.1:0/2496977874 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb27815d5f0 0x7fb2780ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.353+0000 7fb276ffd700 1 --2- 172.21.15.1:0/2496977874 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2780cfbf0 0x7fb278159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.354+0000 7fb2767fc700 1 -- 172.21.15.1:0/2496977874 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2780cfbf0 msgr2=0x7fb278159870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:24.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.354+0000 7fb2767fc700 1 --2- 172.21.15.1:0/2496977874 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2780cfbf0 0x7fb278159870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:24.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.354+0000 7fb2767fc700 1 -- 172.21.15.1:0/2496977874 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb27815d5f0 msgr2=0x7fb2780ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:24.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.354+0000 7fb2767fc700 1 --2- 172.21.15.1:0/2496977874 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb27815d5f0 0x7fb2780ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:24.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.354+0000 7fb2767fc700 1 -- 172.21.15.1:0/2496977874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb270009b30 con 0x7fb2780d3a80 2024-03-18T19:02:24.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.354+0000 7fb2767fc700 1 --2- 172.21.15.1:0/2496977874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2780d3a80 0x7fb2780c7ef0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fb26c00c450 tx=0x7fb26c00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:24.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.354+0000 7fb25f7fe700 1 -- 172.21.15.1:0/2496977874 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c0052d0 con 0x7fb2780d3a80 2024-03-18T19:02:24.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.354+0000 7fb25f7fe700 1 -- 172.21.15.1:0/2496977874 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26c005430 con 0x7fb2780d3a80 2024-03-18T19:02:24.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.354+0000 7fb27d7e2700 1 -- 172.21.15.1:0/2496977874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2780c7130 con 0x7fb2780d3a80 2024-03-18T19:02:24.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.354+0000 7fb25f7fe700 1 -- 172.21.15.1:0/2496977874 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c0038c0 con 0x7fb2780d3a80 2024-03-18T19:02:24.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.354+0000 7fb27d7e2700 1 -- 172.21.15.1:0/2496977874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2780c76f0 con 0x7fb2780d3a80 2024-03-18T19:02:24.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.355+0000 7fb27d7e2700 1 -- 172.21.15.1:0/2496977874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2780c7c10 con 0x7fb2780d3a80 2024-03-18T19:02:24.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.355+0000 7fb25f7fe700 1 -- 172.21.15.1:0/2496977874 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fb26c003a20 con 0x7fb2780d3a80 2024-03-18T19:02:24.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.356+0000 7fb25f7fe700 1 --2- 172.21.15.1:0/2496977874 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb26406ae90 0x7fb26406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:24.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.356+0000 7fb25f7fe700 1 -- 172.21.15.1:0/2496977874 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fb26c088b30 con 0x7fb2780d3a80 2024-03-18T19:02:24.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.356+0000 7fb276ffd700 1 --2- 172.21.15.1:0/2496977874 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb26406ae90 0x7fb26406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:24.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.356+0000 7fb276ffd700 1 --2- 172.21.15.1:0/2496977874 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb26406ae90 0x7fb26406d350 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb268009be0 tx=0x7fb268009210 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:24.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.360+0000 7fb25f7fe700 1 -- 172.21.15.1:0/2496977874 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb2780c7c10 con 0x7fb2780d3a80 2024-03-18T19:02:24.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.588+0000 7fb27d7e2700 1 -- 172.21.15.1:0/2496977874 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb2780d15a0 con 0x7fb26406ae90 2024-03-18T19:02:24.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.596+0000 7fb25f7fe700 1 -- 172.21.15.1:0/2496977874 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fb268011860 con 0x7fb26406ae90 2024-03-18T19:02:24.592 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:24.593 INFO:teuthology.orchestra.run.smithi001.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:28", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.598+0000 7fb27d7e2700 1 -- 172.21.15.1:0/2496977874 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb26406ae90 msgr2=0x7fb26406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.598+0000 7fb27d7e2700 1 --2- 172.21.15.1:0/2496977874 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb26406ae90 0x7fb26406d350 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb268009be0 tx=0x7fb268009210 comp rx=0 tx=0).stop 2024-03-18T19:02:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.598+0000 7fb27d7e2700 1 -- 172.21.15.1:0/2496977874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2780d3a80 msgr2=0x7fb2780c7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.598+0000 7fb27d7e2700 1 --2- 172.21.15.1:0/2496977874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2780d3a80 0x7fb2780c7ef0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fb26c00c450 tx=0x7fb26c00c760 comp rx=0 tx=0).stop 2024-03-18T19:02:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.598+0000 7fb27d7e2700 1 -- 172.21.15.1:0/2496977874 shutdown_connections 2024-03-18T19:02:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.598+0000 7fb27d7e2700 1 --2- 172.21.15.1:0/2496977874 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb26406ae90 0x7fb26406d350 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.598+0000 7fb27d7e2700 1 --2- 172.21.15.1:0/2496977874 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2780cfbf0 0x7fb278159870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.598+0000 7fb27d7e2700 1 --2- 172.21.15.1:0/2496977874 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb27815d5f0 0x7fb2780ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.598+0000 7fb27d7e2700 1 --2- 172.21.15.1:0/2496977874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2780d3a80 0x7fb2780c7ef0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.598+0000 7fb27d7e2700 1 -- 172.21.15.1:0/2496977874 >> 172.21.15.1:0/2496977874 conn(0x7fb27809c540 msgr2=0x7fb27815ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.598+0000 7fb27d7e2700 1 -- 172.21.15.1:0/2496977874 shutdown_connections 2024-03-18T19:02:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.598+0000 7fb27d7e2700 1 -- 172.21.15.1:0/2496977874 wait complete. 2024-03-18T19:02:24.606 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:28", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:24.607 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1m', 'retention': {}, 'start': '2024-03-18T00:00:00', 'created': '2024-03-18T18:51:28', 'first': '2024-03-18T18:52:00', 'last': '2024-03-18T19:02:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-03-18T19:02:24.607 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-18T19:02:24.607 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-18T19:02:24.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:24.607 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2024-03-18T19:02:24.629 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:24.629 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:24.630 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:24.630 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:24.630 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:24.630 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:24.630 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:24.630 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:24.630 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:24.630 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:24.630 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:24.630 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-18-18_52_00_UTC', 'scheduled-2024-03-18-18_53_00_UTC', 'scheduled-2024-03-18-18_54_00_UTC', 'scheduled-2024-03-18-18_55_00_UTC', 'scheduled-2024-03-18-18_56_00_UTC', 'scheduled-2024-03-18-18_57_00_UTC', 'scheduled-2024-03-18-18_58_00_UTC', 'scheduled-2024-03-18-18_59_00_UTC', 'scheduled-2024-03-18-19_00_00_UTC', 'scheduled-2024-03-18-19_01_00_UTC', 'scheduled-2024-03-18-19_02_00_UTC'] 2024-03-18T19:02:24.630 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir1 2024-03-18T19:02:24.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.802+0000 7f9d71ce3700 1 Processor -- start 2024-03-18T19:02:24.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.802+0000 7f9d71ce3700 1 -- start start 2024-03-18T19:02:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.802+0000 7f9d71ce3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d6c164000 0x7f9d6c1663f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.802+0000 7f9d71ce3700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6c15c340 con 0x7f9d6c0ace80 2024-03-18T19:02:24.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.802+0000 7f9d71ce3700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6c15b7a0 con 0x7f9d6c06a750 2024-03-18T19:02:24.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.802+0000 7f9d71ce3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c15fc90 con 0x7f9d6c164000 2024-03-18T19:02:24.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6affd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9d6c0ace80 0x7f9d6c0ad310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41600/0 (socket says 172.21.15.1:41600) 2024-03-18T19:02:24.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6affd700 1 -- 172.21.15.1:0/3619298311 learned_addr learned my addr 172.21.15.1:0/3619298311 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:24.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6a7fc700 1 -- 172.21.15.1:0/3619298311 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928920079 0 0) 0x7f9d6c15c340 con 0x7f9d6c0ace80 2024-03-18T19:02:24.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6a7fc700 1 -- 172.21.15.1:0/3619298311 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d540035b0 con 0x7f9d6c0ace80 2024-03-18T19:02:24.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6bfff700 1 --2- 172.21.15.1:0/3619298311 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d6c164000 0x7f9d6c1663f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:24.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6a7fc700 1 -- 172.21.15.1:0/3619298311 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1610224150 0 0) 0x7f9d540035b0 con 0x7f9d6c0ace80 2024-03-18T19:02:24.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6a7fc700 1 -- 172.21.15.1:0/3619298311 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d6c164000 msgr2=0x7f9d6c1663f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:24.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6a7fc700 1 --2- 172.21.15.1:0/3619298311 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d6c164000 0x7f9d6c1663f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:24.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6a7fc700 1 -- 172.21.15.1:0/3619298311 >> v1:172.21.15.165:6789/0 conn(0x7f9d6c06a750 legacy=0x7f9d6c06ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:24.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6a7fc700 1 -- 172.21.15.1:0/3619298311 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6c15ba10 con 0x7f9d6c0ace80 2024-03-18T19:02:24.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6a7fc700 1 -- 172.21.15.1:0/3619298311 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f9d60001670 con 0x7f9d6c0ace80 2024-03-18T19:02:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6a7fc700 1 -- 172.21.15.1:0/3619298311 >> v1:172.21.15.1:6789/0 conn(0x7f9d6c0ace80 legacy=0x7f9d6c0ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6a7fc700 1 --2- 172.21.15.1:0/3619298311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d54003a30 0x7f9d54003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6a7fc700 1 --2- 172.21.15.1:0/3619298311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d54004850 0x7f9d54004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6a7fc700 1 --2- 172.21.15.1:0/3619298311 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d6c164000 0x7f9d54005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6a7fc700 1 -- 172.21.15.1:0/3619298311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c15fc90 con 0x7f9d54004850 2024-03-18T19:02:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6a7fc700 1 -- 172.21.15.1:0/3619298311 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5400bb40 con 0x7f9d54003a30 2024-03-18T19:02:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6a7fc700 1 -- 172.21.15.1:0/3619298311 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5400bd00 con 0x7f9d6c164000 2024-03-18T19:02:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6affd700 1 --2- 172.21.15.1:0/3619298311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d54004850 0x7f9d54004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6b7fe700 1 --2- 172.21.15.1:0/3619298311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d54003a30 0x7f9d54003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.803+0000 7f9d6bfff700 1 --2- 172.21.15.1:0/3619298311 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d6c164000 0x7f9d54005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.804+0000 7f9d6affd700 1 -- 172.21.15.1:0/3619298311 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d6c164000 msgr2=0x7f9d54005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.804+0000 7f9d6affd700 1 --2- 172.21.15.1:0/3619298311 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d6c164000 0x7f9d54005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.804+0000 7f9d6affd700 1 -- 172.21.15.1:0/3619298311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d54003a30 msgr2=0x7f9d54003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.804+0000 7f9d6affd700 1 --2- 172.21.15.1:0/3619298311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d54003a30 0x7f9d54003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.804+0000 7f9d6affd700 1 -- 172.21.15.1:0/3619298311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d540035b0 con 0x7f9d54004850 2024-03-18T19:02:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.804+0000 7f9d6affd700 1 --2- 172.21.15.1:0/3619298311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d54004850 0x7f9d54004cd0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f9d6000c2a0 tx=0x7f9d54005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a09ad26b267045dc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.804+0000 7f9d6a7fc700 1 -- 172.21.15.1:0/3619298311 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d60007c10 con 0x7f9d54004850 2024-03-18T19:02:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.804+0000 7f9d6a7fc700 1 -- 172.21.15.1:0/3619298311 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6000cde0 con 0x7f9d54004850 2024-03-18T19:02:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.804+0000 7f9d6a7fc700 1 -- 172.21.15.1:0/3619298311 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d60005ba0 con 0x7f9d54004850 2024-03-18T19:02:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.804+0000 7f9d71ce3700 1 -- 172.21.15.1:0/3619298311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d54004850 msgr2=0x7f9d54004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.804+0000 7f9d71ce3700 1 --2- 172.21.15.1:0/3619298311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d54004850 0x7f9d54004cd0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f9d6000c2a0 tx=0x7f9d54005880 comp rx=0 tx=0).stop 2024-03-18T19:02:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.804+0000 7f9d6bfff700 1 -- 172.21.15.1:0/3619298311 reap_dead start 2024-03-18T19:02:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.804+0000 7f9d71ce3700 1 -- 172.21.15.1:0/3619298311 shutdown_connections 2024-03-18T19:02:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.804+0000 7f9d71ce3700 1 -- 172.21.15.1:0/3619298311 >> 172.21.15.1:0/3619298311 conn(0x7f9d6c0cc9b0 msgr2=0x7f9d6c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.804+0000 7f9d71ce3700 1 -- 172.21.15.1:0/3619298311 shutdown_connections 2024-03-18T19:02:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.805+0000 7f9d71ce3700 1 -- 172.21.15.1:0/3619298311 wait complete. 2024-03-18T19:02:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.806+0000 7f9d71ce3700 1 Processor -- start 2024-03-18T19:02:24.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.806+0000 7f9d71ce3700 1 -- start start 2024-03-18T19:02:24.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d71ce3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d54004850 0x7f9d6c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:24.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d71ce3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d6c15a8c0 0x7f9d6c152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:24.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d71ce3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d6c0ce7e0 0x7f9d6c15ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:24.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d71ce3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c152ac0 con 0x7f9d6c15a8c0 2024-03-18T19:02:24.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d71ce3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c0cdab0 con 0x7f9d6c0ce7e0 2024-03-18T19:02:24.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d71ce3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c0ce350 con 0x7f9d54004850 2024-03-18T19:02:24.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d6affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d6c15a8c0 0x7f9d6c152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:24.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d6affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d6c15a8c0 0x7f9d6c152580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47106/0 (socket says 172.21.15.1:47106) 2024-03-18T19:02:24.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d6affd700 1 -- 172.21.15.1:0/2343841028 learned_addr learned my addr 172.21.15.1:0/2343841028 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:24.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d6b7fe700 1 --2- 172.21.15.1:0/2343841028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d54004850 0x7f9d6c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:24.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d6bfff700 1 --2- 172.21.15.1:0/2343841028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d6c0ce7e0 0x7f9d6c15ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:24.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d6affd700 1 -- 172.21.15.1:0/2343841028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d54004850 msgr2=0x7f9d6c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:24.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d6affd700 1 --2- 172.21.15.1:0/2343841028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d54004850 0x7f9d6c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:24.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d6affd700 1 -- 172.21.15.1:0/2343841028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d6c0ce7e0 msgr2=0x7f9d6c15ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:24.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d6affd700 1 --2- 172.21.15.1:0/2343841028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d6c0ce7e0 0x7f9d6c15ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:24.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d6affd700 1 -- 172.21.15.1:0/2343841028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6c15adc0 con 0x7f9d6c15a8c0 2024-03-18T19:02:24.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d6affd700 1 --2- 172.21.15.1:0/2343841028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d6c15a8c0 0x7f9d6c152580 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f9d6000cf80 tx=0x7f9d6c072a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:24.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d53fff700 1 -- 172.21.15.1:0/2343841028 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d60001430 con 0x7f9d6c15a8c0 2024-03-18T19:02:24.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d53fff700 1 -- 172.21.15.1:0/2343841028 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d60001590 con 0x7f9d6c15a8c0 2024-03-18T19:02:24.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d53fff700 1 -- 172.21.15.1:0/2343841028 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d60007310 con 0x7f9d6c15a8c0 2024-03-18T19:02:24.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.807+0000 7f9d71ce3700 1 -- 172.21.15.1:0/2343841028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d6c0cdd00 con 0x7f9d6c15a8c0 2024-03-18T19:02:24.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.808+0000 7f9d71ce3700 1 -- 172.21.15.1:0/2343841028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d6c0cec90 con 0x7f9d6c15a8c0 2024-03-18T19:02:24.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.808+0000 7f9d71ce3700 1 -- 172.21.15.1:0/2343841028 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6c0cee50 con 0x7f9d6c15a8c0 2024-03-18T19:02:24.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.809+0000 7f9d53fff700 1 -- 172.21.15.1:0/2343841028 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f9d60019030 con 0x7f9d6c15a8c0 2024-03-18T19:02:24.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.809+0000 7f9d53fff700 1 --2- 172.21.15.1:0/2343841028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9d5806ae90 0x7f9d5806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:24.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.809+0000 7f9d53fff700 1 -- 172.21.15.1:0/2343841028 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f9d6008a900 con 0x7f9d6c15a8c0 2024-03-18T19:02:24.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.809+0000 7f9d6b7fe700 1 --2- 172.21.15.1:0/2343841028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9d5806ae90 0x7f9d5806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:24.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.810+0000 7f9d6b7fe700 1 --2- 172.21.15.1:0/2343841028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9d5806ae90 0x7f9d5806d350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0018b0 tx=0x7f9d5c009040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:24.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:24.814+0000 7f9d53fff700 1 -- 172.21.15.1:0/2343841028 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9d6c0cee50 con 0x7f9d6c15a8c0 2024-03-18T19:02:25.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.039+0000 7f9d71ce3700 1 -- 172.21.15.1:0/2343841028 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9d6c0ada20 con 0x7f9d5806ae90 2024-03-18T19:02:25.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.048+0000 7f9d53fff700 1 -- 172.21.15.1:0/2343841028 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f9d5c001630 con 0x7f9d5806ae90 2024-03-18T19:02:25.045 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:25.045 INFO:teuthology.orchestra.run.smithi001.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:29", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:25.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.050+0000 7f9d71ce3700 1 -- 172.21.15.1:0/2343841028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9d5806ae90 msgr2=0x7f9d5806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:25.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.050+0000 7f9d71ce3700 1 --2- 172.21.15.1:0/2343841028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9d5806ae90 0x7f9d5806d350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0018b0 tx=0x7f9d5c009040 comp rx=0 tx=0).stop 2024-03-18T19:02:25.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.050+0000 7f9d71ce3700 1 -- 172.21.15.1:0/2343841028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d6c15a8c0 msgr2=0x7f9d6c152580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:25.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.050+0000 7f9d71ce3700 1 --2- 172.21.15.1:0/2343841028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d6c15a8c0 0x7f9d6c152580 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f9d6000cf80 tx=0x7f9d6c072a60 comp rx=0 tx=0).stop 2024-03-18T19:02:25.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.050+0000 7f9d71ce3700 1 -- 172.21.15.1:0/2343841028 shutdown_connections 2024-03-18T19:02:25.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.050+0000 7f9d71ce3700 1 --2- 172.21.15.1:0/2343841028 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9d5806ae90 0x7f9d5806d350 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.050+0000 7f9d71ce3700 1 --2- 172.21.15.1:0/2343841028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9d54004850 0x7f9d6c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.050+0000 7f9d71ce3700 1 --2- 172.21.15.1:0/2343841028 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d6c0ce7e0 0x7f9d6c15ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.050+0000 7f9d71ce3700 1 --2- 172.21.15.1:0/2343841028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9d6c15a8c0 0x7f9d6c152580 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.050+0000 7f9d71ce3700 1 -- 172.21.15.1:0/2343841028 >> 172.21.15.1:0/2343841028 conn(0x7f9d6c0cc9b0 msgr2=0x7f9d6c0b1370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:25.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.050+0000 7f9d71ce3700 1 -- 172.21.15.1:0/2343841028 shutdown_connections 2024-03-18T19:02:25.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.050+0000 7f9d71ce3700 1 -- 172.21.15.1:0/2343841028 wait complete. 2024-03-18T19:02:25.059 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:29", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:25.059 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1m', 'retention': {}, 'start': '2024-03-18T00:00:00', 'created': '2024-03-18T18:51:29', 'first': '2024-03-18T18:52:00', 'last': '2024-03-18T19:02:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-03-18T19:02:25.059 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-18T19:02:25.059 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-18T19:02:25.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:25.059 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2024-03-18T19:02:25.081 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:25.081 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:25.082 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:25.082 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:25.082 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:25.082 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:25.082 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:25.082 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:25.082 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_01_UTC 2024-03-18T19:02:25.082 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:25.082 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:25.082 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-18-18_52_00_UTC', 'scheduled-2024-03-18-18_53_00_UTC', 'scheduled-2024-03-18-18_54_00_UTC', 'scheduled-2024-03-18-18_55_00_UTC', 'scheduled-2024-03-18-18_56_00_UTC', 'scheduled-2024-03-18-18_57_00_UTC', 'scheduled-2024-03-18-18_58_00_UTC', 'scheduled-2024-03-18-18_59_00_UTC', 'scheduled-2024-03-18-19_00_01_UTC', 'scheduled-2024-03-18-19_01_00_UTC', 'scheduled-2024-03-18-19_02_00_UTC'] 2024-03-18T19:02:25.083 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir2 2024-03-18T19:02:25.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.252+0000 7f199d7b6700 1 Processor -- start 2024-03-18T19:02:25.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.252+0000 7f199d7b6700 1 -- start start 2024-03-18T19:02:25.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.253+0000 7f199d7b6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f19980b4810 0x7f19980d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:25.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.253+0000 7f199d7b6700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19980b3370 con 0x7f19980cfb50 2024-03-18T19:02:25.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.253+0000 7f199d7b6700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19980b2760 con 0x7f19980cde70 2024-03-18T19:02:25.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.253+0000 7f199d7b6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19980b2a40 con 0x7f19980b4810 2024-03-18T19:02:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.253+0000 7f19967fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f19980b4810 0x7f19980d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.253+0000 7f1996ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f19980cfb50 0x7f19980cff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41616/0 (socket says 172.21.15.1:41616) 2024-03-18T19:02:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.253+0000 7f1996ffd700 1 -- 172.21.15.1:0/815876839 learned_addr learned my addr 172.21.15.1:0/815876839 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.253+0000 7f19967fc700 1 -- 172.21.15.1:0/815876839 >> v1:172.21.15.165:6789/0 conn(0x7f19980cde70 legacy=0x7f19980ce2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.253+0000 7f19967fc700 1 -- 172.21.15.1:0/815876839 >> v1:172.21.15.1:6789/0 conn(0x7f19980cfb50 legacy=0x7f19980cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.253+0000 7f19967fc700 1 -- 172.21.15.1:0/815876839 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19980b2d20 con 0x7f19980b4810 2024-03-18T19:02:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.253+0000 7f1995ffb700 1 -- 172.21.15.1:0/815876839 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957637032 0 0) 0x7f19980b3370 con 0x7f19980cfb50 2024-03-18T19:02:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.254+0000 7f19967fc700 1 --2- 172.21.15.1:0/815876839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19980b4810 0x7f19980d3c60 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f1988009840 tx=0x7f1988009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=488338d61c0540d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.254+0000 7f1995ffb700 1 -- 172.21.15.1:0/815876839 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1988011830 con 0x7f19980b4810 2024-03-18T19:02:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.254+0000 7f1995ffb700 1 -- 172.21.15.1:0/815876839 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1988011990 con 0x7f19980b4810 2024-03-18T19:02:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.254+0000 7f1995ffb700 1 -- 172.21.15.1:0/815876839 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1988011d20 con 0x7f19980b4810 2024-03-18T19:02:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.254+0000 7f199d7b6700 1 -- 172.21.15.1:0/815876839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19980b4810 msgr2=0x7f19980d3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:25.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.254+0000 7f199d7b6700 1 --2- 172.21.15.1:0/815876839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19980b4810 0x7f19980d3c60 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f1988009840 tx=0x7f1988009b50 comp rx=0 tx=0).stop 2024-03-18T19:02:25.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.254+0000 7f199d7b6700 1 -- 172.21.15.1:0/815876839 shutdown_connections 2024-03-18T19:02:25.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.254+0000 7f199d7b6700 1 --2- 172.21.15.1:0/815876839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19980b4810 0x7f19980d3c60 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.254+0000 7f199d7b6700 1 -- 172.21.15.1:0/815876839 >> 172.21.15.1:0/815876839 conn(0x7f19980d3140 msgr2=0x7f19980d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:25.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.254+0000 7f199d7b6700 1 -- 172.21.15.1:0/815876839 shutdown_connections 2024-03-18T19:02:25.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.254+0000 7f199d7b6700 1 -- 172.21.15.1:0/815876839 wait complete. 2024-03-18T19:02:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.256+0000 7f199d7b6700 1 Processor -- start 2024-03-18T19:02:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.256+0000 7f199d7b6700 1 -- start start 2024-03-18T19:02:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.256+0000 7f199d7b6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19980b4810 0x7f1998159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.256+0000 7f199d7b6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19980cde70 0x7f199815b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.256+0000 7f199d7b6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19980cfb50 0x7f199815bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.256+0000 7f199d7b6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19980b62a0 con 0x7f19980cfb50 2024-03-18T19:02:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.256+0000 7f199d7b6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199815a9a0 con 0x7f19980b4810 2024-03-18T19:02:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.256+0000 7f199d7b6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f199815ac80 con 0x7f19980cde70 2024-03-18T19:02:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.256+0000 7f19977fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19980cfb50 0x7f199815bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.256+0000 7f19977fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19980cfb50 0x7f199815bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47128/0 (socket says 172.21.15.1:47128) 2024-03-18T19:02:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.256+0000 7f19977fe700 1 -- 172.21.15.1:0/2680074694 learned_addr learned my addr 172.21.15.1:0/2680074694 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.257+0000 7f1996ffd700 1 --2- 172.21.15.1:0/2680074694 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19980b4810 0x7f1998159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.257+0000 7f19967fc700 1 --2- 172.21.15.1:0/2680074694 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19980cde70 0x7f199815b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.257+0000 7f19977fe700 1 -- 172.21.15.1:0/2680074694 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19980cde70 msgr2=0x7f199815b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.257+0000 7f19977fe700 1 --2- 172.21.15.1:0/2680074694 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19980cde70 0x7f199815b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.257+0000 7f19977fe700 1 -- 172.21.15.1:0/2680074694 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19980b4810 msgr2=0x7f1998159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.257+0000 7f19977fe700 1 --2- 172.21.15.1:0/2680074694 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19980b4810 0x7f1998159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.257+0000 7f19977fe700 1 -- 172.21.15.1:0/2680074694 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19880094a0 con 0x7f19980cfb50 2024-03-18T19:02:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.257+0000 7f1996ffd700 1 --2- 172.21.15.1:0/2680074694 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19980b4810 0x7f1998159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.257+0000 7f19977fe700 1 --2- 172.21.15.1:0/2680074694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19980cfb50 0x7f199815bcc0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f199000cc50 tx=0x7f199000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.257+0000 7f197f7fe700 1 -- 172.21.15.1:0/2680074694 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199000ade0 con 0x7f19980cfb50 2024-03-18T19:02:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.257+0000 7f197f7fe700 1 -- 172.21.15.1:0/2680074694 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199000ce50 con 0x7f19980cfb50 2024-03-18T19:02:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.257+0000 7f199d7b6700 1 -- 172.21.15.1:0/2680074694 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f199815af60 con 0x7f19980cfb50 2024-03-18T19:02:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.257+0000 7f197f7fe700 1 -- 172.21.15.1:0/2680074694 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1990003c20 con 0x7f19980cfb50 2024-03-18T19:02:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.257+0000 7f199d7b6700 1 -- 172.21.15.1:0/2680074694 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1998158ca0 con 0x7f19980cfb50 2024-03-18T19:02:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.258+0000 7f199d7b6700 1 -- 172.21.15.1:0/2680074694 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19980b6410 con 0x7f19980cfb50 2024-03-18T19:02:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.259+0000 7f197f7fe700 1 -- 172.21.15.1:0/2680074694 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f19900050d0 con 0x7f19980cfb50 2024-03-18T19:02:25.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.259+0000 7f197f7fe700 1 --2- 172.21.15.1:0/2680074694 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f198406ae90 0x7f198406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:25.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.259+0000 7f197f7fe700 1 -- 172.21.15.1:0/2680074694 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f1990089a40 con 0x7f19980cfb50 2024-03-18T19:02:25.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.259+0000 7f1996ffd700 1 --2- 172.21.15.1:0/2680074694 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f198406ae90 0x7f198406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:25.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.260+0000 7f1996ffd700 1 --2- 172.21.15.1:0/2680074694 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f198406ae90 0x7f198406d350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f198c000c00 tx=0x7f198c009040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:25.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.264+0000 7f197f7fe700 1 -- 172.21.15.1:0/2680074694 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f199008be60 con 0x7f19980cfb50 2024-03-18T19:02:25.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.493+0000 7f199d7b6700 1 -- 172.21.15.1:0/2680074694 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f199815adf0 con 0x7f198406ae90 2024-03-18T19:02:25.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.501+0000 7f197f7fe700 1 -- 172.21.15.1:0/2680074694 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f199815adf0 con 0x7f198406ae90 2024-03-18T19:02:25.498 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:25.498 INFO:teuthology.orchestra.run.smithi001.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:29", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:25.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.503+0000 7f199d7b6700 1 -- 172.21.15.1:0/2680074694 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f198406ae90 msgr2=0x7f198406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:25.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.503+0000 7f199d7b6700 1 --2- 172.21.15.1:0/2680074694 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f198406ae90 0x7f198406d350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f198c000c00 tx=0x7f198c009040 comp rx=0 tx=0).stop 2024-03-18T19:02:25.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.503+0000 7f199d7b6700 1 -- 172.21.15.1:0/2680074694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19980cfb50 msgr2=0x7f199815bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:25.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.503+0000 7f199d7b6700 1 --2- 172.21.15.1:0/2680074694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19980cfb50 0x7f199815bcc0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f199000cc50 tx=0x7f199000a340 comp rx=0 tx=0).stop 2024-03-18T19:02:25.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.503+0000 7f199d7b6700 1 -- 172.21.15.1:0/2680074694 shutdown_connections 2024-03-18T19:02:25.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.503+0000 7f199d7b6700 1 --2- 172.21.15.1:0/2680074694 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f198406ae90 0x7f198406d350 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.503+0000 7f199d7b6700 1 --2- 172.21.15.1:0/2680074694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19980cfb50 0x7f199815bcc0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.503+0000 7f199d7b6700 1 --2- 172.21.15.1:0/2680074694 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19980b4810 0x7f1998159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.503+0000 7f199d7b6700 1 --2- 172.21.15.1:0/2680074694 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19980cde70 0x7f199815b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.503+0000 7f199d7b6700 1 -- 172.21.15.1:0/2680074694 >> 172.21.15.1:0/2680074694 conn(0x7f19980d3140 msgr2=0x7f199806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:25.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.503+0000 7f199d7b6700 1 -- 172.21.15.1:0/2680074694 shutdown_connections 2024-03-18T19:02:25.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.503+0000 7f199d7b6700 1 -- 172.21.15.1:0/2680074694 wait complete. 2024-03-18T19:02:25.512 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:29", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:25.512 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1m', 'retention': {}, 'start': '2024-03-18T00:00:00', 'created': '2024-03-18T18:51:29', 'first': '2024-03-18T18:52:00', 'last': '2024-03-18T19:02:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-03-18T19:02:25.512 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-18T19:02:25.512 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-18T19:02:25.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:25.512 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2024-03-18T19:02:25.535 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:25.535 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_01_UTC 2024-03-18T19:02:25.535 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:25.535 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:25.535 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:25.535 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:25.535 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:25.536 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:25.536 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:25.536 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:25.536 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:25.536 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-18-18_52_00_UTC', 'scheduled-2024-03-18-18_53_01_UTC', 'scheduled-2024-03-18-18_54_00_UTC', 'scheduled-2024-03-18-18_55_00_UTC', 'scheduled-2024-03-18-18_56_00_UTC', 'scheduled-2024-03-18-18_57_00_UTC', 'scheduled-2024-03-18-18_58_00_UTC', 'scheduled-2024-03-18-18_59_00_UTC', 'scheduled-2024-03-18-19_00_00_UTC', 'scheduled-2024-03-18-19_01_00_UTC', 'scheduled-2024-03-18-19_02_00_UTC'] 2024-03-18T19:02:25.536 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir3 2024-03-18T19:02:25.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.706+0000 7fec7eee8700 1 Processor -- start 2024-03-18T19:02:25.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.706+0000 7fec7eee8700 1 -- start start 2024-03-18T19:02:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec7eee8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec78071d30 0x7fec780add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec7eee8700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec780ce8f0 con 0x7fec780cb9e0 2024-03-18T19:02:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec7eee8700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec780ae260 con 0x7fec7815a1d0 2024-03-18T19:02:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec7eee8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec780cdb00 con 0x7fec78071d30 2024-03-18T19:02:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec7d485700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fec780cb9e0 0x7fec780cbe70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41626/0 (socket says 172.21.15.1:41626) 2024-03-18T19:02:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec7d485700 1 -- 172.21.15.1:0/1721386118 learned_addr learned my addr 172.21.15.1:0/1721386118 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec77fff700 1 --2- 172.21.15.1:0/1721386118 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec78071d30 0x7fec780add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec777fe700 1 -- 172.21.15.1:0/1721386118 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075460407 0 0) 0x7fec780ce8f0 con 0x7fec780cb9e0 2024-03-18T19:02:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec777fe700 1 -- 172.21.15.1:0/1721386118 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec680035b0 con 0x7fec780cb9e0 2024-03-18T19:02:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec777fe700 1 -- 172.21.15.1:0/1721386118 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165228856 0 0) 0x7fec780ae260 con 0x7fec7815a1d0 2024-03-18T19:02:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec777fe700 1 -- 172.21.15.1:0/1721386118 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec780ce8f0 con 0x7fec7815a1d0 2024-03-18T19:02:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec777fe700 1 -- 172.21.15.1:0/1721386118 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2517566372 0 0) 0x7fec680035b0 con 0x7fec780cb9e0 2024-03-18T19:02:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec777fe700 1 -- 172.21.15.1:0/1721386118 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec78071d30 msgr2=0x7fec780add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec777fe700 1 --2- 172.21.15.1:0/1721386118 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec78071d30 0x7fec780add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec777fe700 1 -- 172.21.15.1:0/1721386118 >> v1:172.21.15.165:6789/0 conn(0x7fec7815a1d0 legacy=0x7fec78071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec777fe700 1 -- 172.21.15.1:0/1721386118 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec780cdce0 con 0x7fec780cb9e0 2024-03-18T19:02:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec777fe700 1 -- 172.21.15.1:0/1721386118 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fec700019f0 con 0x7fec780cb9e0 2024-03-18T19:02:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec777fe700 1 -- 172.21.15.1:0/1721386118 >> v1:172.21.15.1:6789/0 conn(0x7fec780cb9e0 legacy=0x7fec780cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec777fe700 1 --2- 172.21.15.1:0/1721386118 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec78071d30 0x7fec68003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec777fe700 1 --2- 172.21.15.1:0/1721386118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec68004170 0x7fec68004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec777fe700 1 --2- 172.21.15.1:0/1721386118 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec68004d90 0x7fec68005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec777fe700 1 -- 172.21.15.1:0/1721386118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec780cdb00 con 0x7fec68004170 2024-03-18T19:02:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec777fe700 1 -- 172.21.15.1:0/1721386118 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6800dd60 con 0x7fec68004d90 2024-03-18T19:02:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.707+0000 7fec777fe700 1 -- 172.21.15.1:0/1721386118 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec6800df20 con 0x7fec78071d30 2024-03-18T19:02:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.708+0000 7fec77fff700 1 --2- 172.21.15.1:0/1721386118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec68004170 0x7fec68004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.708+0000 7fec7d485700 1 --2- 172.21.15.1:0/1721386118 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec68004d90 0x7fec68005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.708+0000 7fec7cc84700 1 --2- 172.21.15.1:0/1721386118 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec78071d30 0x7fec68003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.708+0000 7fec77fff700 1 -- 172.21.15.1:0/1721386118 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec78071d30 msgr2=0x7fec68003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.708+0000 7fec77fff700 1 --2- 172.21.15.1:0/1721386118 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec78071d30 0x7fec68003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.708+0000 7fec77fff700 1 -- 172.21.15.1:0/1721386118 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec68004d90 msgr2=0x7fec68005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.708+0000 7fec77fff700 1 --2- 172.21.15.1:0/1721386118 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec68004d90 0x7fec68005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.708+0000 7fec77fff700 1 -- 172.21.15.1:0/1721386118 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec680035b0 con 0x7fec68004170 2024-03-18T19:02:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.708+0000 7fec77fff700 1 --2- 172.21.15.1:0/1721386118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec68004170 0x7fec68004850 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fec64009b60 tx=0x7fec68005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78debcbbeaf88692 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.708+0000 7fec777fe700 1 -- 172.21.15.1:0/1721386118 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6400e6c0 con 0x7fec68004170 2024-03-18T19:02:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.708+0000 7fec777fe700 1 -- 172.21.15.1:0/1721386118 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6400e820 con 0x7fec68004170 2024-03-18T19:02:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.708+0000 7fec777fe700 1 -- 172.21.15.1:0/1721386118 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6400eb90 con 0x7fec68004170 2024-03-18T19:02:25.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.709+0000 7fec7eee8700 1 -- 172.21.15.1:0/1721386118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec68004170 msgr2=0x7fec68004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:25.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.709+0000 7fec7eee8700 1 --2- 172.21.15.1:0/1721386118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec68004170 0x7fec68004850 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fec64009b60 tx=0x7fec68005880 comp rx=0 tx=0).stop 2024-03-18T19:02:25.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.709+0000 7fec7d485700 1 -- 172.21.15.1:0/1721386118 reap_dead start 2024-03-18T19:02:25.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.709+0000 7fec7eee8700 1 -- 172.21.15.1:0/1721386118 shutdown_connections 2024-03-18T19:02:25.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.709+0000 7fec7eee8700 1 -- 172.21.15.1:0/1721386118 >> 172.21.15.1:0/1721386118 conn(0x7fec780cc9b0 msgr2=0x7fec780ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:25.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.709+0000 7fec7eee8700 1 -- 172.21.15.1:0/1721386118 shutdown_connections 2024-03-18T19:02:25.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.709+0000 7fec7eee8700 1 -- 172.21.15.1:0/1721386118 wait complete. 2024-03-18T19:02:25.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.710+0000 7fec7eee8700 1 Processor -- start 2024-03-18T19:02:25.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.710+0000 7fec7eee8700 1 -- start start 2024-03-18T19:02:25.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.711+0000 7fec7eee8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec68004170 0x7fec78157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:25.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.711+0000 7fec7eee8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec780b4770 0x7fec780b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:25.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.711+0000 7fec7eee8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec780b5130 0x7fec780c8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:25.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.711+0000 7fec7eee8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec780b5ab0 con 0x7fec68004170 2024-03-18T19:02:25.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.711+0000 7fec7eee8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec780c6580 con 0x7fec780b4770 2024-03-18T19:02:25.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.711+0000 7fec7eee8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec780c5c50 con 0x7fec780b5130 2024-03-18T19:02:25.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.711+0000 7fec7cc84700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec68004170 0x7fec78157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:25.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.711+0000 7fec7cc84700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec68004170 0x7fec78157080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47140/0 (socket says 172.21.15.1:47140) 2024-03-18T19:02:25.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.711+0000 7fec7cc84700 1 -- 172.21.15.1:0/2103176022 learned_addr learned my addr 172.21.15.1:0/2103176022 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:25.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.711+0000 7fec77fff700 1 --2- 172.21.15.1:0/2103176022 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec780b4770 0x7fec780b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:25.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.711+0000 7fec7d485700 1 --2- 172.21.15.1:0/2103176022 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec780b5130 0x7fec780c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:25.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.711+0000 7fec7cc84700 1 -- 172.21.15.1:0/2103176022 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec780b5130 msgr2=0x7fec780c8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:25.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.711+0000 7fec7cc84700 1 --2- 172.21.15.1:0/2103176022 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec780b5130 0x7fec780c8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.711+0000 7fec7cc84700 1 -- 172.21.15.1:0/2103176022 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec780b4770 msgr2=0x7fec780b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:25.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.711+0000 7fec7cc84700 1 --2- 172.21.15.1:0/2103176022 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec780b4770 0x7fec780b4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.711+0000 7fec7cc84700 1 -- 172.21.15.1:0/2103176022 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec780c5f30 con 0x7fec68004170 2024-03-18T19:02:25.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.712+0000 7fec7cc84700 1 --2- 172.21.15.1:0/2103176022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec68004170 0x7fec78157080 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00f810 tx=0x7fec6c00fbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.712+0000 7fec757fa700 1 -- 172.21.15.1:0/2103176022 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c0052e0 con 0x7fec68004170 2024-03-18T19:02:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.712+0000 7fec757fa700 1 -- 172.21.15.1:0/2103176022 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6c005440 con 0x7fec68004170 2024-03-18T19:02:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.712+0000 7fec7eee8700 1 -- 172.21.15.1:0/2103176022 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec780c6210 con 0x7fec68004170 2024-03-18T19:02:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.712+0000 7fec757fa700 1 -- 172.21.15.1:0/2103176022 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c0055b0 con 0x7fec68004170 2024-03-18T19:02:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.712+0000 7fec7eee8700 1 -- 172.21.15.1:0/2103176022 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec780b55e0 con 0x7fec68004170 2024-03-18T19:02:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.713+0000 7fec7eee8700 1 -- 172.21.15.1:0/2103176022 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec780c6860 con 0x7fec68004170 2024-03-18T19:02:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.713+0000 7fec757fa700 1 -- 172.21.15.1:0/2103176022 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fec6c003910 con 0x7fec68004170 2024-03-18T19:02:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.714+0000 7fec757fa700 1 --2- 172.21.15.1:0/2103176022 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fec5c06ae90 0x7fec5c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.714+0000 7fec757fa700 1 -- 172.21.15.1:0/2103176022 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fec6c089880 con 0x7fec68004170 2024-03-18T19:02:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.714+0000 7fec77fff700 1 --2- 172.21.15.1:0/2103176022 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fec5c06ae90 0x7fec5c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.714+0000 7fec77fff700 1 --2- 172.21.15.1:0/2103176022 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fec5c06ae90 0x7fec5c06d350 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fec640090b0 tx=0x7fec64008030 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:25.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.718+0000 7fec757fa700 1 -- 172.21.15.1:0/2103176022 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fec6c054580 con 0x7fec68004170 2024-03-18T19:02:25.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.944+0000 7fec7eee8700 1 -- 172.21.15.1:0/2103176022 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fec78067010 con 0x7fec5c06ae90 2024-03-18T19:02:25.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.951+0000 7fec757fa700 1 -- 172.21.15.1:0/2103176022 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fec78067010 con 0x7fec5c06ae90 2024-03-18T19:02:25.947 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:25.948 INFO:teuthology.orchestra.run.smithi001.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:30", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.953+0000 7fec7eee8700 1 -- 172.21.15.1:0/2103176022 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fec5c06ae90 msgr2=0x7fec5c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.953+0000 7fec7eee8700 1 --2- 172.21.15.1:0/2103176022 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fec5c06ae90 0x7fec5c06d350 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fec640090b0 tx=0x7fec64008030 comp rx=0 tx=0).stop 2024-03-18T19:02:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.953+0000 7fec7eee8700 1 -- 172.21.15.1:0/2103176022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec68004170 msgr2=0x7fec78157080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.953+0000 7fec7eee8700 1 --2- 172.21.15.1:0/2103176022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec68004170 0x7fec78157080 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00f810 tx=0x7fec6c00fbd0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.953+0000 7fec7eee8700 1 -- 172.21.15.1:0/2103176022 shutdown_connections 2024-03-18T19:02:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.953+0000 7fec7eee8700 1 --2- 172.21.15.1:0/2103176022 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fec5c06ae90 0x7fec5c06d350 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.953+0000 7fec7eee8700 1 --2- 172.21.15.1:0/2103176022 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec780b5130 0x7fec780c8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.953+0000 7fec7eee8700 1 --2- 172.21.15.1:0/2103176022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec68004170 0x7fec78157080 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.953+0000 7fec7eee8700 1 --2- 172.21.15.1:0/2103176022 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec780b4770 0x7fec780b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.953+0000 7fec7eee8700 1 -- 172.21.15.1:0/2103176022 >> 172.21.15.1:0/2103176022 conn(0x7fec780cc9b0 msgr2=0x7fec780afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.953+0000 7fec7eee8700 1 -- 172.21.15.1:0/2103176022 shutdown_connections 2024-03-18T19:02:25.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:25.953+0000 7fec7eee8700 1 -- 172.21.15.1:0/2103176022 wait complete. 2024-03-18T19:02:25.962 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:30", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:25.962 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1m', 'retention': {}, 'start': '2024-03-18T00:00:00', 'created': '2024-03-18T18:51:30', 'first': '2024-03-18T18:52:00', 'last': '2024-03-18T19:02:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-03-18T19:02:25.962 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-18T19:02:25.962 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-18T19:02:25.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:25.962 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2024-03-18T19:02:25.985 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:25.985 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:25.985 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:25.985 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:25.985 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:25.985 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:25.986 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:25.986 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:25.986 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:25.986 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:25.986 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:25.986 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-18-18_52_00_UTC', 'scheduled-2024-03-18-18_53_00_UTC', 'scheduled-2024-03-18-18_54_00_UTC', 'scheduled-2024-03-18-18_55_00_UTC', 'scheduled-2024-03-18-18_56_00_UTC', 'scheduled-2024-03-18-18_57_00_UTC', 'scheduled-2024-03-18-18_58_00_UTC', 'scheduled-2024-03-18-18_59_00_UTC', 'scheduled-2024-03-18-19_00_00_UTC', 'scheduled-2024-03-18-19_01_00_UTC', 'scheduled-2024-03-18-19_02_00_UTC'] 2024-03-18T19:02:25.986 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir4 2024-03-18T19:02:26.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.155+0000 7fd7771eb700 1 Processor -- start 2024-03-18T19:02:26.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.155+0000 7fd7771eb700 1 -- start start 2024-03-18T19:02:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.156+0000 7fd7771eb700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd7700cf9d0 0x7fd7700717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.156+0000 7fd7771eb700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7700cfdb0 0x7fd770141010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.156+0000 7fd7771eb700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd770071d20 con 0x7fd770144500 2024-03-18T19:02:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.156+0000 7fd7771eb700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd77015ed40 con 0x7fd7700cfdb0 2024-03-18T19:02:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.156+0000 7fd7771eb700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd77015ef90 con 0x7fd7700cf9d0 2024-03-18T19:02:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.156+0000 7fd775788700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7700cfdb0 0x7fd770141010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.156+0000 7fd775788700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7700cfdb0 0x7fd770141010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47146/0 (socket says 172.21.15.1:47146) 2024-03-18T19:02:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.156+0000 7fd775788700 1 -- 172.21.15.1:0/2014823019 learned_addr learned my addr 172.21.15.1:0/2014823019 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.157+0000 7fd774f87700 1 --2- 172.21.15.1:0/2014823019 >> v2:172.21.15.165:3300/0 conn(0x7fd7700cf9d0 0x7fd7700717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.157+0000 7fd775788700 1 -- 172.21.15.1:0/2014823019 >> v2:172.21.15.165:3300/0 conn(0x7fd7700cf9d0 msgr2=0x7fd7700717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:26.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.157+0000 7fd775788700 1 --2- 172.21.15.1:0/2014823019 >> v2:172.21.15.165:3300/0 conn(0x7fd7700cf9d0 0x7fd7700717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:26.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.157+0000 7fd775788700 1 -- 172.21.15.1:0/2014823019 >> v1:172.21.15.165:6789/0 conn(0x7fd770144500 legacy=0x7fd7700ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:26.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.157+0000 7fd775788700 1 -- 172.21.15.1:0/2014823019 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77015f270 con 0x7fd7700cfdb0 2024-03-18T19:02:26.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.157+0000 7fd775788700 1 --2- 172.21.15.1:0/2014823019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7700cfdb0 0x7fd770141010 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd768009e80 tx=0x7fd76800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bae406b4b55b5c37 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:26.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.157+0000 7fd76f7fe700 1 -- 172.21.15.1:0/2014823019 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76800e070 con 0x7fd7700cfdb0 2024-03-18T19:02:26.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.157+0000 7fd76f7fe700 1 -- 172.21.15.1:0/2014823019 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd768003a20 con 0x7fd7700cfdb0 2024-03-18T19:02:26.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.157+0000 7fd76f7fe700 1 -- 172.21.15.1:0/2014823019 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd768003dd0 con 0x7fd7700cfdb0 2024-03-18T19:02:26.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.157+0000 7fd7771eb700 1 -- 172.21.15.1:0/2014823019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7700cfdb0 msgr2=0x7fd770141010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:26.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.157+0000 7fd7771eb700 1 --2- 172.21.15.1:0/2014823019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7700cfdb0 0x7fd770141010 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd768009e80 tx=0x7fd76800b330 comp rx=0 tx=0).stop 2024-03-18T19:02:26.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.157+0000 7fd7771eb700 1 -- 172.21.15.1:0/2014823019 shutdown_connections 2024-03-18T19:02:26.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.158+0000 7fd7771eb700 1 --2- 172.21.15.1:0/2014823019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7700cfdb0 0x7fd770141010 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:26.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.158+0000 7fd7771eb700 1 --2- 172.21.15.1:0/2014823019 >> v2:172.21.15.165:3300/0 conn(0x7fd7700cf9d0 0x7fd7700717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:26.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.158+0000 7fd7771eb700 1 -- 172.21.15.1:0/2014823019 >> 172.21.15.1:0/2014823019 conn(0x7fd770066b70 msgr2=0x7fd77006a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:26.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.158+0000 7fd7771eb700 1 -- 172.21.15.1:0/2014823019 shutdown_connections 2024-03-18T19:02:26.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.158+0000 7fd7771eb700 1 -- 172.21.15.1:0/2014823019 wait complete. 2024-03-18T19:02:26.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.159+0000 7fd7771eb700 1 Processor -- start 2024-03-18T19:02:26.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.159+0000 7fd7771eb700 1 -- start start 2024-03-18T19:02:26.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.160+0000 7fd7771eb700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7700cf9d0 0x7fd770157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:26.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.160+0000 7fd7771eb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7700cfdb0 0x7fd77013fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:26.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.160+0000 7fd7771eb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd770144500 0x7fd770140310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:26.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.160+0000 7fd7771eb700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7701f0c40 con 0x7fd7700cfdb0 2024-03-18T19:02:26.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.160+0000 7fd7771eb700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7701f1c10 con 0x7fd7700cf9d0 2024-03-18T19:02:26.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.160+0000 7fd7771eb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd77015e6e0 con 0x7fd770144500 2024-03-18T19:02:26.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.160+0000 7fd76ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7700cfdb0 0x7fd77013fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:26.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.160+0000 7fd76ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7700cfdb0 0x7fd77013fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47152/0 (socket says 172.21.15.1:47152) 2024-03-18T19:02:26.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.160+0000 7fd76ffff700 1 -- 172.21.15.1:0/1940965785 learned_addr learned my addr 172.21.15.1:0/1940965785 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:26.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.161+0000 7fd774f87700 1 --2- 172.21.15.1:0/1940965785 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7700cf9d0 0x7fd770157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:26.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.161+0000 7fd775788700 1 --2- 172.21.15.1:0/1940965785 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd770144500 0x7fd770140310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:26.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.161+0000 7fd76ffff700 1 -- 172.21.15.1:0/1940965785 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd770144500 msgr2=0x7fd770140310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:26.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.161+0000 7fd76ffff700 1 --2- 172.21.15.1:0/1940965785 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd770144500 0x7fd770140310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:26.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.161+0000 7fd76ffff700 1 -- 172.21.15.1:0/1940965785 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7700cf9d0 msgr2=0x7fd770157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:26.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.161+0000 7fd76ffff700 1 --2- 172.21.15.1:0/1940965785 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7700cf9d0 0x7fd770157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:26.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.161+0000 7fd76ffff700 1 -- 172.21.15.1:0/1940965785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd768009b30 con 0x7fd7700cfdb0 2024-03-18T19:02:26.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.161+0000 7fd775788700 1 --2- 172.21.15.1:0/1940965785 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd770144500 0x7fd770140310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:02:26.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.161+0000 7fd76ffff700 1 --2- 172.21.15.1:0/1940965785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7700cfdb0 0x7fd77013fdd0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00b480 tx=0x7fd75c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:26.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.161+0000 7fd76d7fa700 1 -- 172.21.15.1:0/1940965785 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c011830 con 0x7fd7700cfdb0 2024-03-18T19:02:26.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.161+0000 7fd76d7fa700 1 -- 172.21.15.1:0/1940965785 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75c011990 con 0x7fd7700cfdb0 2024-03-18T19:02:26.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.161+0000 7fd7771eb700 1 -- 172.21.15.1:0/1940965785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd770157fd0 con 0x7fd7700cfdb0 2024-03-18T19:02:26.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.161+0000 7fd76d7fa700 1 -- 172.21.15.1:0/1940965785 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c011d40 con 0x7fd7700cfdb0 2024-03-18T19:02:26.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.161+0000 7fd7771eb700 1 -- 172.21.15.1:0/1940965785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7701f0f00 con 0x7fd7700cfdb0 2024-03-18T19:02:26.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.162+0000 7fd7771eb700 1 -- 172.21.15.1:0/1940965785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7700676a0 con 0x7fd7700cfdb0 2024-03-18T19:02:26.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.162+0000 7fd76d7fa700 1 -- 172.21.15.1:0/1940965785 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fd75c01a400 con 0x7fd7700cfdb0 2024-03-18T19:02:26.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.163+0000 7fd76d7fa700 1 --2- 172.21.15.1:0/1940965785 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd75406ae90 0x7fd75406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:26.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.163+0000 7fd76d7fa700 1 -- 172.21.15.1:0/1940965785 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fd75c089560 con 0x7fd7700cfdb0 2024-03-18T19:02:26.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.163+0000 7fd774f87700 1 --2- 172.21.15.1:0/1940965785 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd75406ae90 0x7fd75406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:26.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.163+0000 7fd774f87700 1 --2- 172.21.15.1:0/1940965785 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd75406ae90 0x7fd75406d350 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd764009930 tx=0x7fd764008040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:26.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.167+0000 7fd76d7fa700 1 -- 172.21.15.1:0/1940965785 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd75c0542f0 con 0x7fd7700cfdb0 2024-03-18T19:02:26.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.396+0000 7fd7771eb700 1 -- 172.21.15.1:0/1940965785 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd7700cb950 con 0x7fd75406ae90 2024-03-18T19:02:26.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.404+0000 7fd76d7fa700 1 -- 172.21.15.1:0/1940965785 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fd7640095c0 con 0x7fd75406ae90 2024-03-18T19:02:26.402 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:26.402 INFO:teuthology.orchestra.run.smithi001.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:30", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.406+0000 7fd7771eb700 1 -- 172.21.15.1:0/1940965785 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd75406ae90 msgr2=0x7fd75406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.406+0000 7fd7771eb700 1 --2- 172.21.15.1:0/1940965785 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd75406ae90 0x7fd75406d350 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd764009930 tx=0x7fd764008040 comp rx=0 tx=0).stop 2024-03-18T19:02:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.406+0000 7fd7771eb700 1 -- 172.21.15.1:0/1940965785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7700cfdb0 msgr2=0x7fd77013fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.406+0000 7fd7771eb700 1 --2- 172.21.15.1:0/1940965785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7700cfdb0 0x7fd77013fdd0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00b480 tx=0x7fd75c00b840 comp rx=0 tx=0).stop 2024-03-18T19:02:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.407+0000 7fd7771eb700 1 -- 172.21.15.1:0/1940965785 shutdown_connections 2024-03-18T19:02:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.407+0000 7fd7771eb700 1 --2- 172.21.15.1:0/1940965785 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd75406ae90 0x7fd75406d350 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.407+0000 7fd7771eb700 1 --2- 172.21.15.1:0/1940965785 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd770144500 0x7fd770140310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.407+0000 7fd7771eb700 1 --2- 172.21.15.1:0/1940965785 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7700cf9d0 0x7fd770157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.407+0000 7fd7771eb700 1 --2- 172.21.15.1:0/1940965785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7700cfdb0 0x7fd77013fdd0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.407+0000 7fd7771eb700 1 -- 172.21.15.1:0/1940965785 >> 172.21.15.1:0/1940965785 conn(0x7fd770066b70 msgr2=0x7fd7700ceee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:26.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.407+0000 7fd7771eb700 1 -- 172.21.15.1:0/1940965785 shutdown_connections 2024-03-18T19:02:26.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.407+0000 7fd7771eb700 1 -- 172.21.15.1:0/1940965785 wait complete. 2024-03-18T19:02:26.415 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:30", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:26.415 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1m', 'retention': {}, 'start': '2024-03-18T00:00:00', 'created': '2024-03-18T18:51:30', 'first': '2024-03-18T18:52:00', 'last': '2024-03-18T19:02:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-03-18T19:02:26.416 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-18T19:02:26.416 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-18T19:02:26.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:26.416 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2024-03-18T19:02:26.438 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:26.439 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:26.439 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:26.439 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:26.439 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:26.439 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:26.439 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:26.439 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:26.439 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:26.439 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:26.439 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:26.439 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-18-18_52_00_UTC', 'scheduled-2024-03-18-18_53_00_UTC', 'scheduled-2024-03-18-18_54_00_UTC', 'scheduled-2024-03-18-18_55_00_UTC', 'scheduled-2024-03-18-18_56_00_UTC', 'scheduled-2024-03-18-18_57_00_UTC', 'scheduled-2024-03-18-18_58_00_UTC', 'scheduled-2024-03-18-18_59_00_UTC', 'scheduled-2024-03-18-19_00_00_UTC', 'scheduled-2024-03-18-19_01_00_UTC', 'scheduled-2024-03-18-19_02_00_UTC'] 2024-03-18T19:02:26.440 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir5 2024-03-18T19:02:26.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.609+0000 7f0837f68700 1 Processor -- start 2024-03-18T19:02:26.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.609+0000 7f0837f68700 1 -- start start 2024-03-18T19:02:26.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0837f68700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08300b4810 0x7f08300d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:26.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0837f68700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08300b3370 con 0x7f08300cfb50 2024-03-18T19:02:26.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0837f68700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08300b2760 con 0x7f08300cde70 2024-03-18T19:02:26.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0837f68700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08300b2a40 con 0x7f08300b4810 2024-03-18T19:02:26.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0835d04700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f08300cfb50 0x7f08300cff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41636/0 (socket says 172.21.15.1:41636) 2024-03-18T19:02:26.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0835d04700 1 -- 172.21.15.1:0/411708258 learned_addr learned my addr 172.21.15.1:0/411708258 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:26.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0835503700 1 --2- 172.21.15.1:0/411708258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08300b4810 0x7f08300d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:26.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0834d02700 1 -- 172.21.15.1:0/411708258 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492977024 0 0) 0x7f08300b3370 con 0x7f08300cfb50 2024-03-18T19:02:26.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0834d02700 1 -- 172.21.15.1:0/411708258 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f081c0035b0 con 0x7f08300cfb50 2024-03-18T19:02:26.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0834d02700 1 -- 172.21.15.1:0/411708258 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071863982 0 0) 0x7f08300b2760 con 0x7f08300cde70 2024-03-18T19:02:26.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0834d02700 1 -- 172.21.15.1:0/411708258 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08300b3370 con 0x7f08300cde70 2024-03-18T19:02:26.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0834d02700 1 -- 172.21.15.1:0/411708258 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1455816398 0 0) 0x7f081c0035b0 con 0x7f08300cfb50 2024-03-18T19:02:26.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0834d02700 1 -- 172.21.15.1:0/411708258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08300b4810 msgr2=0x7f08300d3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:26.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0834d02700 1 --2- 172.21.15.1:0/411708258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08300b4810 0x7f08300d3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:26.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0834d02700 1 -- 172.21.15.1:0/411708258 >> v1:172.21.15.165:6789/0 conn(0x7f08300cde70 legacy=0x7f08300ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:26.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0834d02700 1 -- 172.21.15.1:0/411708258 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08300b2d20 con 0x7f08300cfb50 2024-03-18T19:02:26.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0834d02700 1 -- 172.21.15.1:0/411708258 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f08200017f0 con 0x7f08300cfb50 2024-03-18T19:02:26.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0834d02700 1 -- 172.21.15.1:0/411708258 >> v1:172.21.15.1:6789/0 conn(0x7f08300cfb50 legacy=0x7f08300cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:26.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0834d02700 1 --2- 172.21.15.1:0/411708258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08300b4810 0x7f081c003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:26.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0834d02700 1 --2- 172.21.15.1:0/411708258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f081c004170 0x7f081c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:26.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0834d02700 1 --2- 172.21.15.1:0/411708258 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f081c004d90 0x7f081c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:26.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0834d02700 1 -- 172.21.15.1:0/411708258 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08300b2a40 con 0x7f081c004170 2024-03-18T19:02:26.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0834d02700 1 -- 172.21.15.1:0/411708258 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081c00bb50 con 0x7f081c004d90 2024-03-18T19:02:26.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.610+0000 7f0834d02700 1 -- 172.21.15.1:0/411708258 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f081c00bd10 con 0x7f08300b4810 2024-03-18T19:02:26.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.611+0000 7f0835503700 1 --2- 172.21.15.1:0/411708258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f081c004170 0x7f081c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:26.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.611+0000 7f0835d04700 1 --2- 172.21.15.1:0/411708258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08300b4810 0x7f081c003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:26.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.611+0000 7f0836505700 1 --2- 172.21.15.1:0/411708258 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f081c004d90 0x7f081c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.611+0000 7f0835503700 1 -- 172.21.15.1:0/411708258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08300b4810 msgr2=0x7f081c003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.611+0000 7f0835503700 1 --2- 172.21.15.1:0/411708258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08300b4810 0x7f081c003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.611+0000 7f0835503700 1 -- 172.21.15.1:0/411708258 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f081c004d90 msgr2=0x7f081c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.611+0000 7f0835503700 1 --2- 172.21.15.1:0/411708258 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f081c004d90 0x7f081c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.611+0000 7f0835503700 1 -- 172.21.15.1:0/411708258 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081c0035b0 con 0x7f081c004170 2024-03-18T19:02:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.611+0000 7f0835503700 1 --2- 172.21.15.1:0/411708258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f081c004170 0x7f081c004850 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f082c009b60 tx=0x7f081c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2719be5479d6ed55 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.611+0000 7f0834d02700 1 -- 172.21.15.1:0/411708258 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c00e6c0 con 0x7f081c004170 2024-03-18T19:02:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.611+0000 7f0834d02700 1 -- 172.21.15.1:0/411708258 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082c00e820 con 0x7f081c004170 2024-03-18T19:02:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.611+0000 7f0834d02700 1 -- 172.21.15.1:0/411708258 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c00eb90 con 0x7f081c004170 2024-03-18T19:02:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.612+0000 7f0837f68700 1 -- 172.21.15.1:0/411708258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f081c004170 msgr2=0x7f081c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.612+0000 7f0837f68700 1 --2- 172.21.15.1:0/411708258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f081c004170 0x7f081c004850 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f082c009b60 tx=0x7f081c005880 comp rx=0 tx=0).stop 2024-03-18T19:02:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.612+0000 7f0836505700 1 -- 172.21.15.1:0/411708258 reap_dead start 2024-03-18T19:02:26.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.612+0000 7f0837f68700 1 -- 172.21.15.1:0/411708258 shutdown_connections 2024-03-18T19:02:26.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.612+0000 7f0837f68700 1 -- 172.21.15.1:0/411708258 >> 172.21.15.1:0/411708258 conn(0x7f08300d3140 msgr2=0x7f08300d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:26.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.612+0000 7f0837f68700 1 -- 172.21.15.1:0/411708258 shutdown_connections 2024-03-18T19:02:26.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.612+0000 7f0837f68700 1 -- 172.21.15.1:0/411708258 wait complete. 2024-03-18T19:02:26.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.613+0000 7f0837f68700 1 Processor -- start 2024-03-18T19:02:26.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.613+0000 7f0837f68700 1 -- start start 2024-03-18T19:02:26.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.614+0000 7f0837f68700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f081c004170 0x7f083015d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:26.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.614+0000 7f0837f68700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08300c6aa0 0x7f08300c7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:26.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.614+0000 7f0837f68700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08300c8e80 0x7f08300c9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:26.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.614+0000 7f0837f68700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083015da70 con 0x7f08300c6aa0 2024-03-18T19:02:26.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.614+0000 7f0837f68700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08300cb6a0 con 0x7f081c004170 2024-03-18T19:02:26.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.614+0000 7f0837f68700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08300cb0e0 con 0x7f08300c8e80 2024-03-18T19:02:26.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.614+0000 7f0835503700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08300c6aa0 0x7f08300c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:26.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.614+0000 7f0835503700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08300c6aa0 0x7f08300c7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47162/0 (socket says 172.21.15.1:47162) 2024-03-18T19:02:26.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.614+0000 7f0835503700 1 -- 172.21.15.1:0/2916221281 learned_addr learned my addr 172.21.15.1:0/2916221281 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:26.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.615+0000 7f0835d04700 1 --2- 172.21.15.1:0/2916221281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f081c004170 0x7f083015d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:26.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.615+0000 7f0836505700 1 --2- 172.21.15.1:0/2916221281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08300c8e80 0x7f08300c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:26.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.615+0000 7f0835503700 1 -- 172.21.15.1:0/2916221281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08300c8e80 msgr2=0x7f08300c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:26.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.615+0000 7f0835503700 1 --2- 172.21.15.1:0/2916221281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08300c8e80 0x7f08300c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:26.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.615+0000 7f0835503700 1 -- 172.21.15.1:0/2916221281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f081c004170 msgr2=0x7f083015d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.615+0000 7f0835503700 1 --2- 172.21.15.1:0/2916221281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f081c004170 0x7f083015d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.615+0000 7f0835503700 1 -- 172.21.15.1:0/2916221281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08300c9b50 con 0x7f08300c6aa0 2024-03-18T19:02:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.615+0000 7f0835503700 1 --2- 172.21.15.1:0/2916221281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08300c6aa0 0x7f08300c7440 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f081c00bb20 tx=0x7f082c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.615+0000 7f08267fc700 1 -- 172.21.15.1:0/2916221281 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c00e6c0 con 0x7f08300c6aa0 2024-03-18T19:02:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.615+0000 7f08267fc700 1 -- 172.21.15.1:0/2916221281 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082c009ed0 con 0x7f08300c6aa0 2024-03-18T19:02:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.615+0000 7f0837f68700 1 -- 172.21.15.1:0/2916221281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08300c9e30 con 0x7f08300c6aa0 2024-03-18T19:02:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.615+0000 7f08267fc700 1 -- 172.21.15.1:0/2916221281 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c003cf0 con 0x7f08300c6aa0 2024-03-18T19:02:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.615+0000 7f0837f68700 1 -- 172.21.15.1:0/2916221281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08300c9840 con 0x7f08300c6aa0 2024-03-18T19:02:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.616+0000 7f0837f68700 1 -- 172.21.15.1:0/2916221281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0830073350 con 0x7f08300c6aa0 2024-03-18T19:02:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.616+0000 7f08267fc700 1 -- 172.21.15.1:0/2916221281 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f082c013470 con 0x7f08300c6aa0 2024-03-18T19:02:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.617+0000 7f08267fc700 1 --2- 172.21.15.1:0/2916221281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f081406ae90 0x7f081406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.617+0000 7f0835d04700 1 --2- 172.21.15.1:0/2916221281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f081406ae90 0x7f081406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.617+0000 7f08267fc700 1 -- 172.21.15.1:0/2916221281 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f082c08b090 con 0x7f08300c6aa0 2024-03-18T19:02:26.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.617+0000 7f0835d04700 1 --2- 172.21.15.1:0/2916221281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f081406ae90 0x7f081406d350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f08200011d0 tx=0x7f0820009040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:26.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.622+0000 7f08267fc700 1 -- 172.21.15.1:0/2916221281 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f082c055e20 con 0x7f08300c6aa0 2024-03-18T19:02:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.852+0000 7f0837f68700 1 -- 172.21.15.1:0/2916221281 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0830160d90 con 0x7f081406ae90 2024-03-18T19:02:26.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.859+0000 7f08267fc700 1 -- 172.21.15.1:0/2916221281 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f0820000f70 con 0x7f081406ae90 2024-03-18T19:02:26.856 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:26.856 INFO:teuthology.orchestra.run.smithi001.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:31", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:26.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.861+0000 7f0837f68700 1 -- 172.21.15.1:0/2916221281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f081406ae90 msgr2=0x7f081406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:26.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.861+0000 7f0837f68700 1 --2- 172.21.15.1:0/2916221281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f081406ae90 0x7f081406d350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f08200011d0 tx=0x7f0820009040 comp rx=0 tx=0).stop 2024-03-18T19:02:26.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.861+0000 7f0837f68700 1 -- 172.21.15.1:0/2916221281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08300c6aa0 msgr2=0x7f08300c7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:26.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.861+0000 7f0837f68700 1 --2- 172.21.15.1:0/2916221281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08300c6aa0 0x7f08300c7440 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f081c00bb20 tx=0x7f082c0047c0 comp rx=0 tx=0).stop 2024-03-18T19:02:26.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.861+0000 7f0837f68700 1 -- 172.21.15.1:0/2916221281 shutdown_connections 2024-03-18T19:02:26.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.861+0000 7f0837f68700 1 --2- 172.21.15.1:0/2916221281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f081406ae90 0x7f081406d350 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:26.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.861+0000 7f0837f68700 1 --2- 172.21.15.1:0/2916221281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08300c8e80 0x7f08300c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:26.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.861+0000 7f0837f68700 1 --2- 172.21.15.1:0/2916221281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f081c004170 0x7f083015d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:26.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.861+0000 7f0837f68700 1 --2- 172.21.15.1:0/2916221281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08300c6aa0 0x7f08300c7440 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:26.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.861+0000 7f0837f68700 1 -- 172.21.15.1:0/2916221281 >> 172.21.15.1:0/2916221281 conn(0x7f08300d3140 msgr2=0x7f083015e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:26.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.861+0000 7f0837f68700 1 -- 172.21.15.1:0/2916221281 shutdown_connections 2024-03-18T19:02:26.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:26.861+0000 7f0837f68700 1 -- 172.21.15.1:0/2916221281 wait complete. 2024-03-18T19:02:26.870 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:31", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:26.870 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1m', 'retention': {}, 'start': '2024-03-18T00:00:00', 'created': '2024-03-18T18:51:31', 'first': '2024-03-18T18:52:00', 'last': '2024-03-18T19:02:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-03-18T19:02:26.870 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-18T19:02:26.870 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-18T19:02:26.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:26.870 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2024-03-18T19:02:26.893 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:26.893 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:26.893 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:26.893 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:26.894 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:26.894 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:26.894 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:26.894 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:26.894 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:26.894 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:26.894 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:26.894 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-18-18_52_00_UTC', 'scheduled-2024-03-18-18_53_00_UTC', 'scheduled-2024-03-18-18_54_00_UTC', 'scheduled-2024-03-18-18_55_00_UTC', 'scheduled-2024-03-18-18_56_00_UTC', 'scheduled-2024-03-18-18_57_00_UTC', 'scheduled-2024-03-18-18_58_00_UTC', 'scheduled-2024-03-18-18_59_00_UTC', 'scheduled-2024-03-18-19_00_00_UTC', 'scheduled-2024-03-18-19_01_00_UTC', 'scheduled-2024-03-18-19_02_00_UTC'] 2024-03-18T19:02:26.894 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir6 2024-03-18T19:02:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.065+0000 7f682b60b700 1 Processor -- start 2024-03-18T19:02:27.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.065+0000 7f682b60b700 1 -- start start 2024-03-18T19:02:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.066+0000 7f682b60b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f682415d5a0 0x7f68240ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.066+0000 7f682b60b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f68240ce550 0x7f682406a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.066+0000 7f682b60b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68240cf600 con 0x7f68240cfa90 2024-03-18T19:02:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.066+0000 7f682b60b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68240ce9f0 con 0x7f68240ce550 2024-03-18T19:02:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.066+0000 7f682b60b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68240cecd0 con 0x7f682415d5a0 2024-03-18T19:02:27.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.066+0000 7f68293a7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f682415d5a0 0x7f68240ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.066+0000 7f6828ba6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f68240ce550 0x7f682406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.066+0000 7f6828ba6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f68240ce550 0x7f682406a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:32926/0 (socket says 172.21.15.1:32926) 2024-03-18T19:02:27.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.066+0000 7f6828ba6700 1 -- 172.21.15.1:0/1811142704 learned_addr learned my addr 172.21.15.1:0/1811142704 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:27.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.067+0000 7f6828ba6700 1 -- 172.21.15.1:0/1811142704 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f682415d5a0 msgr2=0x7f68240ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:27.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.067+0000 7f6828ba6700 1 --2- 172.21.15.1:0/1811142704 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f682415d5a0 0x7f68240ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.067+0000 7f6828ba6700 1 -- 172.21.15.1:0/1811142704 >> v1:172.21.15.165:6789/0 conn(0x7f68240cfa90 legacy=0x7f6824071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:27.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.067+0000 7f6828ba6700 1 -- 172.21.15.1:0/1811142704 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68240cefb0 con 0x7f68240ce550 2024-03-18T19:02:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.067+0000 7f681bfff700 1 -- 172.21.15.1:0/1811142704 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073312580 0 0) 0x7f68240cf600 con 0x7f68240cfa90 2024-03-18T19:02:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.067+0000 7f68293a7700 1 --2- 172.21.15.1:0/1811142704 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f682415d5a0 0x7f68240ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:02:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.067+0000 7f6828ba6700 1 --2- 172.21.15.1:0/1811142704 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68240ce550 0x7f682406a750 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f68200098f0 tx=0x7f6820009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=480b0d3dfaf9de6b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.067+0000 7f681bfff700 1 -- 172.21.15.1:0/1811142704 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6820011830 con 0x7f68240ce550 2024-03-18T19:02:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.067+0000 7f681bfff700 1 -- 172.21.15.1:0/1811142704 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6820011990 con 0x7f68240ce550 2024-03-18T19:02:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.067+0000 7f681bfff700 1 -- 172.21.15.1:0/1811142704 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6820011d20 con 0x7f68240ce550 2024-03-18T19:02:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.068+0000 7f682b60b700 1 -- 172.21.15.1:0/1811142704 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68240ce550 msgr2=0x7f682406a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.068+0000 7f682b60b700 1 --2- 172.21.15.1:0/1811142704 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68240ce550 0x7f682406a750 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f68200098f0 tx=0x7f6820009c00 comp rx=0 tx=0).stop 2024-03-18T19:02:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.068+0000 7f682b60b700 1 -- 172.21.15.1:0/1811142704 shutdown_connections 2024-03-18T19:02:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.068+0000 7f682b60b700 1 --2- 172.21.15.1:0/1811142704 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f682415d5a0 0x7f68240ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.068+0000 7f682b60b700 1 --2- 172.21.15.1:0/1811142704 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68240ce550 0x7f682406a750 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.068+0000 7f682b60b700 1 -- 172.21.15.1:0/1811142704 >> 172.21.15.1:0/1811142704 conn(0x7f68240cc9b0 msgr2=0x7f68240ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.068+0000 7f682b60b700 1 -- 172.21.15.1:0/1811142704 shutdown_connections 2024-03-18T19:02:27.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.068+0000 7f682b60b700 1 -- 172.21.15.1:0/1811142704 wait complete. 2024-03-18T19:02:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.069+0000 7f682b60b700 1 Processor -- start 2024-03-18T19:02:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.069+0000 7f682b60b700 1 -- start start 2024-03-18T19:02:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.070+0000 7f682b60b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68240ce550 0x7f68240b5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.070+0000 7f682b60b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68240cfa90 0x7f6824152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.070+0000 7f682b60b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f682415d5a0 0x7f6824153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.070+0000 7f682b60b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68241548c0 con 0x7f68240cfa90 2024-03-18T19:02:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.070+0000 7f682b60b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6824154ba0 con 0x7f68240ce550 2024-03-18T19:02:27.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.070+0000 7f682b60b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6824154e80 con 0x7f682415d5a0 2024-03-18T19:02:27.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.070+0000 7f6828ba6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68240cfa90 0x7f6824152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.070+0000 7f6828ba6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68240cfa90 0x7f6824152580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47174/0 (socket says 172.21.15.1:47174) 2024-03-18T19:02:27.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.070+0000 7f6828ba6700 1 -- 172.21.15.1:0/2427274418 learned_addr learned my addr 172.21.15.1:0/2427274418 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:27.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.070+0000 7f68293a7700 1 --2- 172.21.15.1:0/2427274418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68240ce550 0x7f68240b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.070+0000 7f6829ba8700 1 --2- 172.21.15.1:0/2427274418 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f682415d5a0 0x7f6824153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.070+0000 7f6828ba6700 1 -- 172.21.15.1:0/2427274418 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f682415d5a0 msgr2=0x7f6824153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:27.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.070+0000 7f6828ba6700 1 --2- 172.21.15.1:0/2427274418 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f682415d5a0 0x7f6824153fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.070+0000 7f6828ba6700 1 -- 172.21.15.1:0/2427274418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68240ce550 msgr2=0x7f68240b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:27.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.070+0000 7f6828ba6700 1 --2- 172.21.15.1:0/2427274418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68240ce550 0x7f68240b5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.070+0000 7f6828ba6700 1 -- 172.21.15.1:0/2427274418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6814009430 con 0x7f68240cfa90 2024-03-18T19:02:27.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.070+0000 7f6828ba6700 1 --2- 172.21.15.1:0/2427274418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68240cfa90 0x7f6824152580 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f682406cfd0 tx=0x7f6820004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:27.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.071+0000 7f6819ffb700 1 -- 172.21.15.1:0/2427274418 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6820004400 con 0x7f68240cfa90 2024-03-18T19:02:27.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.071+0000 7f6819ffb700 1 -- 172.21.15.1:0/2427274418 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682001f070 con 0x7f68240cfa90 2024-03-18T19:02:27.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.071+0000 7f682b60b700 1 -- 172.21.15.1:0/2427274418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6820009550 con 0x7f68240cfa90 2024-03-18T19:02:27.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.071+0000 7f6819ffb700 1 -- 172.21.15.1:0/2427274418 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68200038e0 con 0x7f68240cfa90 2024-03-18T19:02:27.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.071+0000 7f682b60b700 1 -- 172.21.15.1:0/2427274418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6824155440 con 0x7f68240cfa90 2024-03-18T19:02:27.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.072+0000 7f682b60b700 1 -- 172.21.15.1:0/2427274418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68240676a0 con 0x7f68240cfa90 2024-03-18T19:02:27.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.072+0000 7f6819ffb700 1 -- 172.21.15.1:0/2427274418 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f6820021030 con 0x7f68240cfa90 2024-03-18T19:02:27.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.073+0000 7f6819ffb700 1 --2- 172.21.15.1:0/2427274418 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f680806ae90 0x7f680806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.073+0000 7f6819ffb700 1 -- 172.21.15.1:0/2427274418 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f682008ad40 con 0x7f68240cfa90 2024-03-18T19:02:27.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.073+0000 7f68293a7700 1 --2- 172.21.15.1:0/2427274418 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f680806ae90 0x7f680806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.073+0000 7f68293a7700 1 --2- 172.21.15.1:0/2427274418 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f680806ae90 0x7f680806d350 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6814009cb0 tx=0x7f68140091c0 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:27.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.078+0000 7f6819ffb700 1 -- 172.21.15.1:0/2427274418 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6820055b50 con 0x7f68240cfa90 2024-03-18T19:02:27.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.305+0000 7f682b60b700 1 -- 172.21.15.1:0/2427274418 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f68240b1090 con 0x7f680806ae90 2024-03-18T19:02:27.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.313+0000 7f6819ffb700 1 -- 172.21.15.1:0/2427274418 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f68240b1090 con 0x7f680806ae90 2024-03-18T19:02:27.310 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:27.310 INFO:teuthology.orchestra.run.smithi001.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:31", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:27.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.315+0000 7f682b60b700 1 -- 172.21.15.1:0/2427274418 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f680806ae90 msgr2=0x7f680806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:27.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.315+0000 7f682b60b700 1 --2- 172.21.15.1:0/2427274418 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f680806ae90 0x7f680806d350 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6814009cb0 tx=0x7f68140091c0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.315+0000 7f682b60b700 1 -- 172.21.15.1:0/2427274418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68240cfa90 msgr2=0x7f6824152580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:27.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.315+0000 7f682b60b700 1 --2- 172.21.15.1:0/2427274418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68240cfa90 0x7f6824152580 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f682406cfd0 tx=0x7f6820004750 comp rx=0 tx=0).stop 2024-03-18T19:02:27.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.315+0000 7f682b60b700 1 -- 172.21.15.1:0/2427274418 shutdown_connections 2024-03-18T19:02:27.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.315+0000 7f682b60b700 1 --2- 172.21.15.1:0/2427274418 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f680806ae90 0x7f680806d350 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.315+0000 7f682b60b700 1 --2- 172.21.15.1:0/2427274418 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f682415d5a0 0x7f6824153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.315+0000 7f682b60b700 1 --2- 172.21.15.1:0/2427274418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68240ce550 0x7f68240b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.315+0000 7f682b60b700 1 --2- 172.21.15.1:0/2427274418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68240cfa90 0x7f6824152580 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.315+0000 7f682b60b700 1 -- 172.21.15.1:0/2427274418 >> 172.21.15.1:0/2427274418 conn(0x7f68240cc9b0 msgr2=0x7f68240afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:27.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.315+0000 7f682b60b700 1 -- 172.21.15.1:0/2427274418 shutdown_connections 2024-03-18T19:02:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.315+0000 7f682b60b700 1 -- 172.21.15.1:0/2427274418 wait complete. 2024-03-18T19:02:27.324 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:31", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:27.324 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1m', 'retention': {}, 'start': '2024-03-18T00:00:00', 'created': '2024-03-18T18:51:31', 'first': '2024-03-18T18:52:00', 'last': '2024-03-18T19:02:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-03-18T19:02:27.324 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-18T19:02:27.324 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-18T19:02:27.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:27.325 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2024-03-18T19:02:27.347 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:27.347 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:27.347 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:27.347 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:27.347 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:27.347 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:27.347 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:27.347 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:27.347 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:27.347 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:27.348 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:27.348 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-18-18_52_00_UTC', 'scheduled-2024-03-18-18_53_00_UTC', 'scheduled-2024-03-18-18_54_00_UTC', 'scheduled-2024-03-18-18_55_00_UTC', 'scheduled-2024-03-18-18_56_00_UTC', 'scheduled-2024-03-18-18_57_00_UTC', 'scheduled-2024-03-18-18_58_00_UTC', 'scheduled-2024-03-18-18_59_00_UTC', 'scheduled-2024-03-18-19_00_00_UTC', 'scheduled-2024-03-18-19_01_00_UTC', 'scheduled-2024-03-18-19_02_00_UTC'] 2024-03-18T19:02:27.348 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir7 2024-03-18T19:02:27.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.518+0000 7f73c6cfd700 1 Processor -- start 2024-03-18T19:02:27.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.518+0000 7f73c6cfd700 1 -- start start 2024-03-18T19:02:27.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.519+0000 7f73c6cfd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f73c0071500 0x7f73c006a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.519+0000 7f73c6cfd700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c015c500 con 0x7f73c006ac90 2024-03-18T19:02:27.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.519+0000 7f73c6cfd700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c00af8b0 con 0x7f73c0158e30 2024-03-18T19:02:27.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.519+0000 7f73c6cfd700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c006b120 con 0x7f73c0071500 2024-03-18T19:02:27.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.519+0000 7f73c529a700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f73c006ac90 0x7f73c00cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41638/0 (socket says 172.21.15.1:41638) 2024-03-18T19:02:27.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.519+0000 7f73c529a700 1 -- 172.21.15.1:0/4067389861 learned_addr learned my addr 172.21.15.1:0/4067389861 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:27.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.519+0000 7f73bf7fe700 1 -- 172.21.15.1:0/4067389861 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417385834 0 0) 0x7f73c015c500 con 0x7f73c006ac90 2024-03-18T19:02:27.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.519+0000 7f73bf7fe700 1 -- 172.21.15.1:0/4067389861 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73b00035b0 con 0x7f73c006ac90 2024-03-18T19:02:27.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.519+0000 7f73bffff700 1 --2- 172.21.15.1:0/4067389861 >> v2:172.21.15.165:3300/0 conn(0x7f73c0071500 0x7f73c006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.519+0000 7f73bf7fe700 1 -- 172.21.15.1:0/4067389861 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4164587208 0 0) 0x7f73b00035b0 con 0x7f73c006ac90 2024-03-18T19:02:27.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.519+0000 7f73bf7fe700 1 -- 172.21.15.1:0/4067389861 >> v2:172.21.15.165:3300/0 conn(0x7f73c0071500 msgr2=0x7f73c006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:27.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.519+0000 7f73bf7fe700 1 --2- 172.21.15.1:0/4067389861 >> v2:172.21.15.165:3300/0 conn(0x7f73c0071500 0x7f73c006a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.519+0000 7f73bf7fe700 1 -- 172.21.15.1:0/4067389861 >> v1:172.21.15.165:6789/0 conn(0x7f73c0158e30 legacy=0x7f73c00cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:27.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.519+0000 7f73bf7fe700 1 -- 172.21.15.1:0/4067389861 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c015b8f0 con 0x7f73c006ac90 2024-03-18T19:02:27.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.519+0000 7f73bf7fe700 1 -- 172.21.15.1:0/4067389861 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f73b80019f0 con 0x7f73c006ac90 2024-03-18T19:02:27.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73bf7fe700 1 -- 172.21.15.1:0/4067389861 >> v1:172.21.15.1:6789/0 conn(0x7f73c006ac90 legacy=0x7f73c00cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:27.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73bf7fe700 1 --2- 172.21.15.1:0/4067389861 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73b0003a00 0x7f73b0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73bf7fe700 1 --2- 172.21.15.1:0/4067389861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73b0004400 0x7f73b00048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73bf7fe700 1 --2- 172.21.15.1:0/4067389861 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f73b00050d0 0x7f73b000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73bf7fe700 1 -- 172.21.15.1:0/4067389861 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c006b120 con 0x7f73b0004400 2024-03-18T19:02:27.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73bf7fe700 1 -- 172.21.15.1:0/4067389861 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b000c030 con 0x7f73b0003a00 2024-03-18T19:02:27.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73bf7fe700 1 -- 172.21.15.1:0/4067389861 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b000c1f0 con 0x7f73b00050d0 2024-03-18T19:02:27.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73bffff700 1 --2- 172.21.15.1:0/4067389861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73b0004400 0x7f73b00048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73c4a99700 1 --2- 172.21.15.1:0/4067389861 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73b0003a00 0x7f73b0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73c529a700 1 --2- 172.21.15.1:0/4067389861 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f73b00050d0 0x7f73b000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73bffff700 1 -- 172.21.15.1:0/4067389861 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f73b00050d0 msgr2=0x7f73b000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:27.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73bffff700 1 --2- 172.21.15.1:0/4067389861 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f73b00050d0 0x7f73b000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73bffff700 1 -- 172.21.15.1:0/4067389861 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73b0003a00 msgr2=0x7f73b0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73bffff700 1 --2- 172.21.15.1:0/4067389861 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73b0003a00 0x7f73b0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73bffff700 1 -- 172.21.15.1:0/4067389861 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b00035b0 con 0x7f73b0004400 2024-03-18T19:02:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73c529a700 1 -- 172.21.15.1:0/4067389861 reap_dead start 2024-03-18T19:02:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73bffff700 1 --2- 172.21.15.1:0/4067389861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73b0004400 0x7f73b00048d0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009b60 tx=0x7f73b0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b3b36ec0c3afe94 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73bf7fe700 1 -- 172.21.15.1:0/4067389861 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac00e6c0 con 0x7f73b0004400 2024-03-18T19:02:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73bf7fe700 1 -- 172.21.15.1:0/4067389861 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ac00e820 con 0x7f73b0004400 2024-03-18T19:02:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.520+0000 7f73bf7fe700 1 -- 172.21.15.1:0/4067389861 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac00eb90 con 0x7f73b0004400 2024-03-18T19:02:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.521+0000 7f73c6cfd700 1 -- 172.21.15.1:0/4067389861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73b0004400 msgr2=0x7f73b00048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.521+0000 7f73c6cfd700 1 --2- 172.21.15.1:0/4067389861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73b0004400 0x7f73b00048d0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009b60 tx=0x7f73b0005880 comp rx=0 tx=0).stop 2024-03-18T19:02:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.521+0000 7f73c6cfd700 1 -- 172.21.15.1:0/4067389861 shutdown_connections 2024-03-18T19:02:27.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.521+0000 7f73c6cfd700 1 --2- 172.21.15.1:0/4067389861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73b0004400 0x7f73b00048d0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.521+0000 7f73c6cfd700 1 -- 172.21.15.1:0/4067389861 >> 172.21.15.1:0/4067389861 conn(0x7f73c00cca00 msgr2=0x7f73c00cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:27.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.521+0000 7f73c6cfd700 1 -- 172.21.15.1:0/4067389861 shutdown_connections 2024-03-18T19:02:27.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.521+0000 7f73c6cfd700 1 -- 172.21.15.1:0/4067389861 wait complete. 2024-03-18T19:02:27.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.522+0000 7f73c6cfd700 1 Processor -- start 2024-03-18T19:02:27.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.522+0000 7f73c6cfd700 1 -- start start 2024-03-18T19:02:27.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.523+0000 7f73c6cfd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73b0004400 0x7f73c0156730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.523+0000 7f73c6cfd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73c01f41a0 0x7f73c00b1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.523+0000 7f73c6cfd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f73c00718e0 0x7f73c0071d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.523+0000 7f73c6cfd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c00722b0 con 0x7f73b0004400 2024-03-18T19:02:27.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.523+0000 7f73c6cfd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c00b2b40 con 0x7f73c01f41a0 2024-03-18T19:02:27.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.523+0000 7f73c6cfd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c00c49f0 con 0x7f73c00718e0 2024-03-18T19:02:27.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.523+0000 7f73c4a99700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73b0004400 0x7f73c0156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.523+0000 7f73c4a99700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73b0004400 0x7f73c0156730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47192/0 (socket says 172.21.15.1:47192) 2024-03-18T19:02:27.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.523+0000 7f73c4a99700 1 -- 172.21.15.1:0/1938116393 learned_addr learned my addr 172.21.15.1:0/1938116393 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:27.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.523+0000 7f73bffff700 1 --2- 172.21.15.1:0/1938116393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73c01f41a0 0x7f73c00b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.523+0000 7f73c4a99700 1 -- 172.21.15.1:0/1938116393 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f73c00718e0 msgr2=0x7f73c0071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.523+0000 7f73c529a700 1 --2- 172.21.15.1:0/1938116393 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f73c00718e0 0x7f73c0071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.523+0000 7f73c4a99700 1 --2- 172.21.15.1:0/1938116393 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f73c00718e0 0x7f73c0071d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.523+0000 7f73c4a99700 1 -- 172.21.15.1:0/1938116393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73c01f41a0 msgr2=0x7f73c00b1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.523+0000 7f73c4a99700 1 --2- 172.21.15.1:0/1938116393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73c01f41a0 0x7f73c00b1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.523+0000 7f73c4a99700 1 -- 172.21.15.1:0/1938116393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c01f46a0 con 0x7f73b0004400 2024-03-18T19:02:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.523+0000 7f73c529a700 1 --2- 172.21.15.1:0/1938116393 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f73c00718e0 0x7f73c0071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:02:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.523+0000 7f73bffff700 1 --2- 172.21.15.1:0/1938116393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73c01f41a0 0x7f73c00b1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.523+0000 7f73c4a99700 1 --2- 172.21.15.1:0/1938116393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73b0004400 0x7f73c0156730 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f73b400f860 tx=0x7f73b400fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.524+0000 7f73bd7fa700 1 -- 172.21.15.1:0/1938116393 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b400fe60 con 0x7f73b0004400 2024-03-18T19:02:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.524+0000 7f73bd7fa700 1 -- 172.21.15.1:0/1938116393 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b40052e0 con 0x7f73b0004400 2024-03-18T19:02:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.524+0000 7f73c6cfd700 1 -- 172.21.15.1:0/1938116393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c01f4980 con 0x7f73b0004400 2024-03-18T19:02:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.524+0000 7f73bd7fa700 1 -- 172.21.15.1:0/1938116393 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b4003680 con 0x7f73b0004400 2024-03-18T19:02:27.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.524+0000 7f73c6cfd700 1 -- 172.21.15.1:0/1938116393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c01f4f40 con 0x7f73b0004400 2024-03-18T19:02:27.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.525+0000 7f73c6cfd700 1 -- 172.21.15.1:0/1938116393 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c0073940 con 0x7f73b0004400 2024-03-18T19:02:27.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.525+0000 7f73bd7fa700 1 -- 172.21.15.1:0/1938116393 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f73b4003800 con 0x7f73b0004400 2024-03-18T19:02:27.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.525+0000 7f73bd7fa700 1 --2- 172.21.15.1:0/1938116393 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f73a406ae90 0x7f73a406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.526+0000 7f73bd7fa700 1 -- 172.21.15.1:0/1938116393 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f73b4089820 con 0x7f73b0004400 2024-03-18T19:02:27.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.526+0000 7f73bffff700 1 --2- 172.21.15.1:0/1938116393 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f73a406ae90 0x7f73a406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.526+0000 7f73bffff700 1 --2- 172.21.15.1:0/1938116393 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f73a406ae90 0x7f73a406d350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0090b0 tx=0x7f73ac01e040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:27.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.530+0000 7f73bd7fa700 1 -- 172.21.15.1:0/1938116393 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f73b40545b0 con 0x7f73b0004400 2024-03-18T19:02:27.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.757+0000 7f73c6cfd700 1 -- 172.21.15.1:0/1938116393 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f73c015c190 con 0x7f73a406ae90 2024-03-18T19:02:27.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.764+0000 7f73bd7fa700 1 -- 172.21.15.1:0/1938116393 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f73ac00eb10 con 0x7f73a406ae90 2024-03-18T19:02:27.761 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:27.761 INFO:teuthology.orchestra.run.smithi001.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:32", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.766+0000 7f73c6cfd700 1 -- 172.21.15.1:0/1938116393 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f73a406ae90 msgr2=0x7f73a406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.766+0000 7f73c6cfd700 1 --2- 172.21.15.1:0/1938116393 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f73a406ae90 0x7f73a406d350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0090b0 tx=0x7f73ac01e040 comp rx=0 tx=0).stop 2024-03-18T19:02:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.766+0000 7f73c6cfd700 1 -- 172.21.15.1:0/1938116393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73b0004400 msgr2=0x7f73c0156730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.766+0000 7f73c6cfd700 1 --2- 172.21.15.1:0/1938116393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73b0004400 0x7f73c0156730 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f73b400f860 tx=0x7f73b400fc20 comp rx=0 tx=0).stop 2024-03-18T19:02:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.766+0000 7f73c6cfd700 1 -- 172.21.15.1:0/1938116393 shutdown_connections 2024-03-18T19:02:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.766+0000 7f73c6cfd700 1 --2- 172.21.15.1:0/1938116393 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f73a406ae90 0x7f73a406d350 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.766+0000 7f73c6cfd700 1 --2- 172.21.15.1:0/1938116393 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f73c00718e0 0x7f73c0071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.766+0000 7f73c6cfd700 1 --2- 172.21.15.1:0/1938116393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73b0004400 0x7f73c0156730 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.766+0000 7f73c6cfd700 1 --2- 172.21.15.1:0/1938116393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73c01f41a0 0x7f73c00b1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.766+0000 7f73c6cfd700 1 -- 172.21.15.1:0/1938116393 >> 172.21.15.1:0/1938116393 conn(0x7f73c00cca00 msgr2=0x7f73c015cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.767+0000 7f73c6cfd700 1 -- 172.21.15.1:0/1938116393 shutdown_connections 2024-03-18T19:02:27.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.767+0000 7f73c6cfd700 1 -- 172.21.15.1:0/1938116393 wait complete. 2024-03-18T19:02:27.775 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:32", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:27.775 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1m', 'retention': {}, 'start': '2024-03-18T00:00:00', 'created': '2024-03-18T18:51:32', 'first': '2024-03-18T18:52:00', 'last': '2024-03-18T19:02:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-03-18T19:02:27.775 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-18T19:02:27.775 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-18T19:02:27.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:27.776 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2024-03-18T19:02:27.798 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:27.798 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:27.798 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:27.798 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:27.798 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:27.798 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:27.799 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:27.799 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:27.799 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:27.799 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:27.799 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:27.799 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-18-18_52_00_UTC', 'scheduled-2024-03-18-18_53_00_UTC', 'scheduled-2024-03-18-18_54_00_UTC', 'scheduled-2024-03-18-18_55_00_UTC', 'scheduled-2024-03-18-18_56_00_UTC', 'scheduled-2024-03-18-18_57_00_UTC', 'scheduled-2024-03-18-18_58_00_UTC', 'scheduled-2024-03-18-18_59_00_UTC', 'scheduled-2024-03-18-19_00_00_UTC', 'scheduled-2024-03-18-19_01_00_UTC', 'scheduled-2024-03-18-19_02_00_UTC'] 2024-03-18T19:02:27.799 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir8 2024-03-18T19:02:27.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.969+0000 7f632a98f700 1 Processor -- start 2024-03-18T19:02:27.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.969+0000 7f632a98f700 1 -- start start 2024-03-18T19:02:27.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.970+0000 7f632a98f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f632409dd40 0x7f63240cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.970+0000 7f632a98f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f63240ce5a0 0x7f63240cea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.970+0000 7f632a98f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63240ce090 con 0x7f632415d780 2024-03-18T19:02:27.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.970+0000 7f632a98f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63240cb480 con 0x7f63240ce5a0 2024-03-18T19:02:27.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.970+0000 7f632a98f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632406db30 con 0x7f632409dd40 2024-03-18T19:02:27.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.970+0000 7f63237fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f632409dd40 0x7f63240cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.970+0000 7f6328f2c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f63240ce5a0 0x7f63240cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.970+0000 7f63237fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f632409dd40 0x7f63240cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:36268/0 (socket says 172.21.15.1:36268) 2024-03-18T19:02:27.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.970+0000 7f63237fe700 1 -- 172.21.15.1:0/2831082988 learned_addr learned my addr 172.21.15.1:0/2831082988 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:27.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.970+0000 7f6323fff700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f632415d780 0x7f63240cad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:43634/0 (socket says 172.21.15.1:43634) 2024-03-18T19:02:27.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.971+0000 7f6328f2c700 1 -- 172.21.15.1:0/2831082988 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f632409dd40 msgr2=0x7f63240cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:27.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.971+0000 7f6328f2c700 1 --2- 172.21.15.1:0/2831082988 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f632409dd40 0x7f63240cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.971+0000 7f6328f2c700 1 -- 172.21.15.1:0/2831082988 >> v1:172.21.15.165:6789/0 conn(0x7f632415d780 legacy=0x7f63240cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.971+0000 7f6328f2c700 1 -- 172.21.15.1:0/2831082988 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632406d520 con 0x7f63240ce5a0 2024-03-18T19:02:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.971+0000 7f6322ffd700 1 -- 172.21.15.1:0/2831082988 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601450034 0 0) 0x7f63240ce090 con 0x7f632415d780 2024-03-18T19:02:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.971+0000 7f63237fe700 1 --2- 172.21.15.1:0/2831082988 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f632409dd40 0x7f63240cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:02:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.971+0000 7f6328f2c700 1 --2- 172.21.15.1:0/2831082988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63240ce5a0 0x7f63240cea50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f631c00bfa0 tx=0x7f631c00d330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ac7a5b40781b136 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.971+0000 7f6322ffd700 1 -- 172.21.15.1:0/2831082988 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c010070 con 0x7f63240ce5a0 2024-03-18T19:02:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.971+0000 7f6322ffd700 1 -- 172.21.15.1:0/2831082988 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631c00b2e0 con 0x7f63240ce5a0 2024-03-18T19:02:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.971+0000 7f6322ffd700 1 -- 172.21.15.1:0/2831082988 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c00dc40 con 0x7f63240ce5a0 2024-03-18T19:02:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.972+0000 7f632a98f700 1 -- 172.21.15.1:0/2831082988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63240ce5a0 msgr2=0x7f63240cea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.972+0000 7f632a98f700 1 --2- 172.21.15.1:0/2831082988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63240ce5a0 0x7f63240cea50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f631c00bfa0 tx=0x7f631c00d330 comp rx=0 tx=0).stop 2024-03-18T19:02:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.972+0000 7f632a98f700 1 -- 172.21.15.1:0/2831082988 shutdown_connections 2024-03-18T19:02:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.972+0000 7f632a98f700 1 --2- 172.21.15.1:0/2831082988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63240ce5a0 0x7f63240cea50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.972+0000 7f632a98f700 1 --2- 172.21.15.1:0/2831082988 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f632409dd40 0x7f63240cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.972+0000 7f632a98f700 1 -- 172.21.15.1:0/2831082988 >> 172.21.15.1:0/2831082988 conn(0x7f63240cca00 msgr2=0x7f63240cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.972+0000 7f632a98f700 1 -- 172.21.15.1:0/2831082988 shutdown_connections 2024-03-18T19:02:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.972+0000 7f632a98f700 1 -- 172.21.15.1:0/2831082988 wait complete. 2024-03-18T19:02:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.973+0000 7f632a98f700 1 Processor -- start 2024-03-18T19:02:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.973+0000 7f632a98f700 1 -- start start 2024-03-18T19:02:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f632a98f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f632409dd40 0x7f63241f2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f632a98f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f63240ce5a0 0x7f63240c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f632a98f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f632415d780 0x7f63240c6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f632a98f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63241f2470 con 0x7f632415d780 2024-03-18T19:02:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f632a98f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632414ded0 con 0x7f632409dd40 2024-03-18T19:02:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f632a98f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63241f1e40 con 0x7f63240ce5a0 2024-03-18T19:02:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f6328f2c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f632415d780 0x7f63240c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f6328f2c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f632415d780 0x7f63240c6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47202/0 (socket says 172.21.15.1:47202) 2024-03-18T19:02:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f6328f2c700 1 -- 172.21.15.1:0/2305703918 learned_addr learned my addr 172.21.15.1:0/2305703918 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f63237fe700 1 --2- 172.21.15.1:0/2305703918 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f63240ce5a0 0x7f63240c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f6323fff700 1 --2- 172.21.15.1:0/2305703918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f632409dd40 0x7f63241f2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f6328f2c700 1 -- 172.21.15.1:0/2305703918 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f63240ce5a0 msgr2=0x7f63240c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f6328f2c700 1 --2- 172.21.15.1:0/2305703918 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f63240ce5a0 0x7f63240c8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f6328f2c700 1 -- 172.21.15.1:0/2305703918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f632409dd40 msgr2=0x7f63241f2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:27.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f6328f2c700 1 --2- 172.21.15.1:0/2305703918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f632409dd40 0x7f63241f2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:27.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f6328f2c700 1 -- 172.21.15.1:0/2305703918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6314009550 con 0x7f632415d780 2024-03-18T19:02:27.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f6323fff700 1 --2- 172.21.15.1:0/2305703918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f632409dd40 0x7f63241f2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:02:27.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f63237fe700 1 --2- 172.21.15.1:0/2305703918 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f63240ce5a0 0x7f63240c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:02:27.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f6328f2c700 1 --2- 172.21.15.1:0/2305703918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f632415d780 0x7f63240c6220 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f631c00dc30 tx=0x7f631c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:27.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.974+0000 7f6320ff9700 1 -- 172.21.15.1:0/2305703918 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c004250 con 0x7f632415d780 2024-03-18T19:02:27.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.975+0000 7f6320ff9700 1 -- 172.21.15.1:0/2305703918 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631c0043b0 con 0x7f632415d780 2024-03-18T19:02:27.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.975+0000 7f632a98f700 1 -- 172.21.15.1:0/2305703918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f631c00bc80 con 0x7f632415d780 2024-03-18T19:02:27.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.975+0000 7f6320ff9700 1 -- 172.21.15.1:0/2305703918 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c004760 con 0x7f632415d780 2024-03-18T19:02:27.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.975+0000 7f632a98f700 1 -- 172.21.15.1:0/2305703918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63240c9c50 con 0x7f632415d780 2024-03-18T19:02:27.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.975+0000 7f632a98f700 1 -- 172.21.15.1:0/2305703918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63241f2620 con 0x7f632415d780 2024-03-18T19:02:27.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.976+0000 7f6320ff9700 1 -- 172.21.15.1:0/2305703918 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f631c009460 con 0x7f632415d780 2024-03-18T19:02:27.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.976+0000 7f6320ff9700 1 --2- 172.21.15.1:0/2305703918 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f630406ae90 0x7f630406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:27.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.976+0000 7f6320ff9700 1 -- 172.21.15.1:0/2305703918 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f631c08d490 con 0x7f632415d780 2024-03-18T19:02:27.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.977+0000 7f6323fff700 1 --2- 172.21.15.1:0/2305703918 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f630406ae90 0x7f630406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:27.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.977+0000 7f6323fff700 1 --2- 172.21.15.1:0/2305703918 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f630406ae90 0x7f630406d350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f630c007fd0 tx=0x7f630c009040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:27.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:27.981+0000 7f6320ff9700 1 -- 172.21.15.1:0/2305703918 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f63241f2620 con 0x7f632415d780 2024-03-18T19:02:28.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.209+0000 7f632a98f700 1 -- 172.21.15.1:0/2305703918 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f632406d7c0 con 0x7f630406ae90 2024-03-18T19:02:28.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.217+0000 7f6320ff9700 1 -- 172.21.15.1:0/2305703918 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f630c001370 con 0x7f630406ae90 2024-03-18T19:02:28.213 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:28.214 INFO:teuthology.orchestra.run.smithi001.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:32", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:28.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.219+0000 7f632a98f700 1 -- 172.21.15.1:0/2305703918 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f630406ae90 msgr2=0x7f630406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:28.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.219+0000 7f632a98f700 1 --2- 172.21.15.1:0/2305703918 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f630406ae90 0x7f630406d350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f630c007fd0 tx=0x7f630c009040 comp rx=0 tx=0).stop 2024-03-18T19:02:28.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.219+0000 7f632a98f700 1 -- 172.21.15.1:0/2305703918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f632415d780 msgr2=0x7f63240c6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:28.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.219+0000 7f632a98f700 1 --2- 172.21.15.1:0/2305703918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f632415d780 0x7f63240c6220 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f631c00dc30 tx=0x7f631c004ad0 comp rx=0 tx=0).stop 2024-03-18T19:02:28.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.219+0000 7f632a98f700 1 -- 172.21.15.1:0/2305703918 shutdown_connections 2024-03-18T19:02:28.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.219+0000 7f632a98f700 1 --2- 172.21.15.1:0/2305703918 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f630406ae90 0x7f630406d350 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:28.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.219+0000 7f632a98f700 1 --2- 172.21.15.1:0/2305703918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f632415d780 0x7f63240c6220 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:28.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.219+0000 7f632a98f700 1 --2- 172.21.15.1:0/2305703918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f632409dd40 0x7f63241f2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:28.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.219+0000 7f632a98f700 1 --2- 172.21.15.1:0/2305703918 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f63240ce5a0 0x7f63240c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:28.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.219+0000 7f632a98f700 1 -- 172.21.15.1:0/2305703918 >> 172.21.15.1:0/2305703918 conn(0x7f63240cca00 msgr2=0x7f6324159f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:28.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.219+0000 7f632a98f700 1 -- 172.21.15.1:0/2305703918 shutdown_connections 2024-03-18T19:02:28.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.219+0000 7f632a98f700 1 -- 172.21.15.1:0/2305703918 wait complete. 2024-03-18T19:02:28.228 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:32", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:28.228 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1m', 'retention': {}, 'start': '2024-03-18T00:00:00', 'created': '2024-03-18T18:51:32', 'first': '2024-03-18T18:52:00', 'last': '2024-03-18T19:02:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-03-18T19:02:28.228 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-18T19:02:28.228 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-18T19:02:28.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:28.228 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2024-03-18T19:02:28.251 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:28.251 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:28.251 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:28.251 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:28.252 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:28.252 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:28.252 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:28.252 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:28.252 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:28.252 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:28.252 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:28.252 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-18-18_52_00_UTC', 'scheduled-2024-03-18-18_53_00_UTC', 'scheduled-2024-03-18-18_54_00_UTC', 'scheduled-2024-03-18-18_55_00_UTC', 'scheduled-2024-03-18-18_56_00_UTC', 'scheduled-2024-03-18-18_57_00_UTC', 'scheduled-2024-03-18-18_58_00_UTC', 'scheduled-2024-03-18-18_59_00_UTC', 'scheduled-2024-03-18-19_00_00_UTC', 'scheduled-2024-03-18-19_01_00_UTC', 'scheduled-2024-03-18-19_02_00_UTC'] 2024-03-18T19:02:28.252 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir9 2024-03-18T19:02:28.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.422+0000 7f0900a78700 1 Processor -- start 2024-03-18T19:02:28.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.422+0000 7f0900a78700 1 -- start start 2024-03-18T19:02:28.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.422+0000 7f0900a78700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08fc06a750 0x7f08fc06ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:28.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.422+0000 7f0900a78700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f08fc0ace80 0x7f08fc0ad300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:28.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.423+0000 7f0900a78700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08fc15c340 con 0x7f08fc0ad840 2024-03-18T19:02:28.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.423+0000 7f0900a78700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08fc15fc90 con 0x7f08fc0ace80 2024-03-18T19:02:28.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.423+0000 7f0900a78700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc070090 con 0x7f08fc06a750 2024-03-18T19:02:28.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.423+0000 7f08fa59c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08fc06a750 0x7f08fc06ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:28.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.423+0000 7f08f9d9b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f08fc0ace80 0x7f08fc0ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:28.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.423+0000 7f08fad9d700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f08fc0ad840 0x7f08fc0caa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:43636/0 (socket says 172.21.15.1:43636) 2024-03-18T19:02:28.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.423+0000 7f08fad9d700 1 -- 172.21.15.1:0/1924103254 learned_addr learned my addr 172.21.15.1:0/1924103254 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:28.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.423+0000 7f08f959a700 1 -- 172.21.15.1:0/1924103254 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584877109 0 0) 0x7f08fc15c340 con 0x7f08fc0ad840 2024-03-18T19:02:28.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.423+0000 7f08f959a700 1 -- 172.21.15.1:0/1924103254 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08e40035b0 con 0x7f08fc0ad840 2024-03-18T19:02:28.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.423+0000 7f08fa59c700 1 -- 172.21.15.1:0/1924103254 >> v2:172.21.15.165:3300/0 conn(0x7f08fc0ace80 msgr2=0x7f08fc0ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:28.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.423+0000 7f08fa59c700 1 --2- 172.21.15.1:0/1924103254 >> v2:172.21.15.165:3300/0 conn(0x7f08fc0ace80 0x7f08fc0ad300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:28.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.423+0000 7f08fa59c700 1 -- 172.21.15.1:0/1924103254 >> v1:172.21.15.165:6789/0 conn(0x7f08fc0ad840 legacy=0x7f08fc0caa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:28.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.423+0000 7f08fa59c700 1 -- 172.21.15.1:0/1924103254 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08fc15b7a0 con 0x7f08fc06a750 2024-03-18T19:02:28.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.423+0000 7f08f9d9b700 1 --2- 172.21.15.1:0/1924103254 >> v2:172.21.15.165:3300/0 conn(0x7f08fc0ace80 0x7f08fc0ad300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:02:28.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.424+0000 7f08fa59c700 1 --2- 172.21.15.1:0/1924103254 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08fc06a750 0x7f08fc06ab30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f08ec009820 tx=0x7f08ec009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30e04122622830e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:28.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.424+0000 7f08f959a700 1 -- 172.21.15.1:0/1924103254 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ec005e30 con 0x7f08fc06a750 2024-03-18T19:02:28.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.424+0000 7f08f959a700 1 -- 172.21.15.1:0/1924103254 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ec006030 con 0x7f08fc06a750 2024-03-18T19:02:28.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.424+0000 7f08f959a700 1 -- 172.21.15.1:0/1924103254 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ec0063c0 con 0x7f08fc06a750 2024-03-18T19:02:28.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.424+0000 7f0900a78700 1 -- 172.21.15.1:0/1924103254 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08fc06a750 msgr2=0x7f08fc06ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:28.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.424+0000 7f0900a78700 1 --2- 172.21.15.1:0/1924103254 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08fc06a750 0x7f08fc06ab30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f08ec009820 tx=0x7f08ec009b30 comp rx=0 tx=0).stop 2024-03-18T19:02:28.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.424+0000 7f0900a78700 1 -- 172.21.15.1:0/1924103254 shutdown_connections 2024-03-18T19:02:28.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.424+0000 7f0900a78700 1 --2- 172.21.15.1:0/1924103254 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08fc06a750 0x7f08fc06ab30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:28.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.424+0000 7f0900a78700 1 --2- 172.21.15.1:0/1924103254 >> v2:172.21.15.165:3300/0 conn(0x7f08fc0ace80 0x7f08fc0ad300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:28.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.424+0000 7f0900a78700 1 -- 172.21.15.1:0/1924103254 >> 172.21.15.1:0/1924103254 conn(0x7f08fc0cc9b0 msgr2=0x7f08fc0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:28.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.425+0000 7f0900a78700 1 -- 172.21.15.1:0/1924103254 shutdown_connections 2024-03-18T19:02:28.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.425+0000 7f0900a78700 1 -- 172.21.15.1:0/1924103254 wait complete. 2024-03-18T19:02:28.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.426+0000 7f0900a78700 1 Processor -- start 2024-03-18T19:02:28.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.426+0000 7f0900a78700 1 -- start start 2024-03-18T19:02:28.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f0900a78700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f08fc06a750 0x7f08fc1583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:28.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f0900a78700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08fc0ace80 0x7f08fc0c61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:28.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f0900a78700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08fc0ad840 0x7f08fc0b2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:28.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f0900a78700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc0c7000 con 0x7f08fc0ad840 2024-03-18T19:02:28.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f0900a78700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc0c4970 con 0x7f08fc06a750 2024-03-18T19:02:28.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f0900a78700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc0c53e0 con 0x7f08fc0ace80 2024-03-18T19:02:28.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f08fad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08fc0ad840 0x7f08fc0b2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:28.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f08fad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08fc0ad840 0x7f08fc0b2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47208/0 (socket says 172.21.15.1:47208) 2024-03-18T19:02:28.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f08fad9d700 1 -- 172.21.15.1:0/1950700905 learned_addr learned my addr 172.21.15.1:0/1950700905 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:28.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f08fa59c700 1 --2- 172.21.15.1:0/1950700905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f08fc06a750 0x7f08fc1583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:28.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f08f9d9b700 1 --2- 172.21.15.1:0/1950700905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08fc0ace80 0x7f08fc0c61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:28.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f08fad9d700 1 -- 172.21.15.1:0/1950700905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08fc0ace80 msgr2=0x7f08fc0c61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:28.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f08fad9d700 1 --2- 172.21.15.1:0/1950700905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08fc0ace80 0x7f08fc0c61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:28.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f08fad9d700 1 -- 172.21.15.1:0/1950700905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f08fc06a750 msgr2=0x7f08fc1583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:28.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f08fad9d700 1 --2- 172.21.15.1:0/1950700905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f08fc06a750 0x7f08fc1583c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:28.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f08fad9d700 1 -- 172.21.15.1:0/1950700905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f00094a0 con 0x7f08fc0ad840 2024-03-18T19:02:28.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f08fad9d700 1 --2- 172.21.15.1:0/1950700905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08fc0ad840 0x7f08fc0b2b60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f08f400caa0 tx=0x7f08f400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:28.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f08eaffd700 1 -- 172.21.15.1:0/1950700905 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f400aa80 con 0x7f08fc0ad840 2024-03-18T19:02:28.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f08eaffd700 1 -- 172.21.15.1:0/1950700905 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f400abe0 con 0x7f08fc0ad840 2024-03-18T19:02:28.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.427+0000 7f08eaffd700 1 -- 172.21.15.1:0/1950700905 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f4012400 con 0x7f08fc0ad840 2024-03-18T19:02:28.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.428+0000 7f0900a78700 1 -- 172.21.15.1:0/1950700905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ec009430 con 0x7f08fc0ad840 2024-03-18T19:02:28.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.428+0000 7f0900a78700 1 -- 172.21.15.1:0/1950700905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08fc0c59a0 con 0x7f08fc0ad840 2024-03-18T19:02:28.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.428+0000 7f0900a78700 1 -- 172.21.15.1:0/1950700905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08fc0676a0 con 0x7f08fc0ad840 2024-03-18T19:02:28.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.429+0000 7f08eaffd700 1 -- 172.21.15.1:0/1950700905 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f08f401b3b0 con 0x7f08fc0ad840 2024-03-18T19:02:28.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.429+0000 7f08eaffd700 1 --2- 172.21.15.1:0/1950700905 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f08dc06aec0 0x7f08dc06d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:28.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.429+0000 7f08eaffd700 1 -- 172.21.15.1:0/1950700905 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f08f4089610 con 0x7f08fc0ad840 2024-03-18T19:02:28.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.429+0000 7f08fa59c700 1 --2- 172.21.15.1:0/1950700905 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f08dc06aec0 0x7f08dc06d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:28.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.430+0000 7f08fa59c700 1 --2- 172.21.15.1:0/1950700905 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f08dc06aec0 0x7f08dc06d380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f08ec000c00 tx=0x7f08ec008070 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:28.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.434+0000 7f08eaffd700 1 -- 172.21.15.1:0/1950700905 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f08f4054310 con 0x7f08fc0ad840 2024-03-18T19:02:28.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.662+0000 7f0900a78700 1 -- 172.21.15.1:0/1950700905 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f08fc0c5550 con 0x7f08dc06aec0 2024-03-18T19:02:28.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.669+0000 7f08eaffd700 1 -- 172.21.15.1:0/1950700905 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f08fc0c5550 con 0x7f08dc06aec0 2024-03-18T19:02:28.666 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:28.666 INFO:teuthology.orchestra.run.smithi001.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:33", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:28.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.672+0000 7f0900a78700 1 -- 172.21.15.1:0/1950700905 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f08dc06aec0 msgr2=0x7f08dc06d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:28.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.672+0000 7f0900a78700 1 --2- 172.21.15.1:0/1950700905 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f08dc06aec0 0x7f08dc06d380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f08ec000c00 tx=0x7f08ec008070 comp rx=0 tx=0).stop 2024-03-18T19:02:28.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.672+0000 7f0900a78700 1 -- 172.21.15.1:0/1950700905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08fc0ad840 msgr2=0x7f08fc0b2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:28.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.672+0000 7f0900a78700 1 --2- 172.21.15.1:0/1950700905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08fc0ad840 0x7f08fc0b2b60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f08f400caa0 tx=0x7f08f400ce60 comp rx=0 tx=0).stop 2024-03-18T19:02:28.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.672+0000 7f0900a78700 1 -- 172.21.15.1:0/1950700905 shutdown_connections 2024-03-18T19:02:28.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.672+0000 7f0900a78700 1 --2- 172.21.15.1:0/1950700905 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f08dc06aec0 0x7f08dc06d380 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:28.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.672+0000 7f0900a78700 1 --2- 172.21.15.1:0/1950700905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08fc0ad840 0x7f08fc0b2b60 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:28.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.672+0000 7f0900a78700 1 --2- 172.21.15.1:0/1950700905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f08fc06a750 0x7f08fc1583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:28.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.672+0000 7f0900a78700 1 --2- 172.21.15.1:0/1950700905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f08fc0ace80 0x7f08fc0c61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:28.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.672+0000 7f0900a78700 1 -- 172.21.15.1:0/1950700905 >> 172.21.15.1:0/1950700905 conn(0x7f08fc0cc9b0 msgr2=0x7f08fc0caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:28.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.672+0000 7f0900a78700 1 -- 172.21.15.1:0/1950700905 shutdown_connections 2024-03-18T19:02:28.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.672+0000 7f0900a78700 1 -- 172.21.15.1:0/1950700905 wait complete. 2024-03-18T19:02:28.680 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T18:51:33", "first": "2024-03-18T18:52:00", "last": "2024-03-18T19:02:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-03-18T19:02:28.681 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1m', 'retention': {}, 'start': '2024-03-18T00:00:00', 'created': '2024-03-18T18:51:33', 'first': '2024-03-18T18:52:00', 'last': '2024-03-18T19:02:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-03-18T19:02:28.681 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-18T19:02:28.681 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-18T19:02:28.681 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir0 1m 2024-03-18T19:02:28.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.812+0000 7fdca36c1700 1 Processor -- start 2024-03-18T19:02:28.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.812+0000 7fdca36c1700 1 -- start start 2024-03-18T19:02:28.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.813+0000 7fdca36c1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdc9c06ae60 0x7fdc9c161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:28.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.813+0000 7fdca36c1700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9c0cad60 con 0x7fdc9c06eda0 2024-03-18T19:02:28.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.813+0000 7fdca36c1700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9c0cb040 con 0x7fdc9c06f180 2024-03-18T19:02:28.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.813+0000 7fdca36c1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc9c0cb320 con 0x7fdc9c06ae60 2024-03-18T19:02:28.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.813+0000 7fdca1c5e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdc9c06ae60 0x7fdc9c161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:28.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.813+0000 7fdca145d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdc9c06eda0 0x7fdc9c0ca580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41644/0 (socket says 172.21.15.1:41644) 2024-03-18T19:02:28.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.813+0000 7fdca145d700 1 -- 172.21.15.1:0/1578668420 learned_addr learned my addr 172.21.15.1:0/1578668420 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:28.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.813+0000 7fdca1c5e700 1 -- 172.21.15.1:0/1578668420 >> v1:172.21.15.165:6789/0 conn(0x7fdc9c06f180 legacy=0x7fdc9c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:28.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.813+0000 7fdca1c5e700 1 -- 172.21.15.1:0/1578668420 >> v1:172.21.15.1:6789/0 conn(0x7fdc9c06eda0 legacy=0x7fdc9c0ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:28.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.813+0000 7fdc93fff700 1 -- 172.21.15.1:0/1578668420 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333837641 0 0) 0x7fdc9c0cad60 con 0x7fdc9c06eda0 2024-03-18T19:02:28.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.813+0000 7fdca1c5e700 1 -- 172.21.15.1:0/1578668420 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9c0ce220 con 0x7fdc9c06ae60 2024-03-18T19:02:28.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.814+0000 7fdca1c5e700 1 --2- 172.21.15.1:0/1578668420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c06ae60 0x7fdc9c161c60 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fdc94009d90 tx=0x7fdc9400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7273a47f429c8900 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:28.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.814+0000 7fdc93fff700 1 -- 172.21.15.1:0/1578668420 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9400e070 con 0x7fdc9c06ae60 2024-03-18T19:02:28.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.814+0000 7fdc93fff700 1 -- 172.21.15.1:0/1578668420 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9400bb90 con 0x7fdc9c06ae60 2024-03-18T19:02:28.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.814+0000 7fdc93fff700 1 -- 172.21.15.1:0/1578668420 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94003a20 con 0x7fdc9c06ae60 2024-03-18T19:02:28.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.814+0000 7fdca36c1700 1 -- 172.21.15.1:0/1578668420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c06ae60 msgr2=0x7fdc9c161c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:28.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.814+0000 7fdca36c1700 1 --2- 172.21.15.1:0/1578668420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c06ae60 0x7fdc9c161c60 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fdc94009d90 tx=0x7fdc9400b330 comp rx=0 tx=0).stop 2024-03-18T19:02:28.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.814+0000 7fdca36c1700 1 -- 172.21.15.1:0/1578668420 shutdown_connections 2024-03-18T19:02:28.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.814+0000 7fdca36c1700 1 --2- 172.21.15.1:0/1578668420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c06ae60 0x7fdc9c161c60 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:28.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.814+0000 7fdca36c1700 1 -- 172.21.15.1:0/1578668420 >> 172.21.15.1:0/1578668420 conn(0x7fdc9c0cff30 msgr2=0x7fdc9c0d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:28.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.814+0000 7fdca36c1700 1 -- 172.21.15.1:0/1578668420 shutdown_connections 2024-03-18T19:02:28.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.814+0000 7fdca36c1700 1 -- 172.21.15.1:0/1578668420 wait complete. 2024-03-18T19:02:28.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.816+0000 7fdca36c1700 1 Processor -- start 2024-03-18T19:02:28.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.816+0000 7fdca36c1700 1 -- start start 2024-03-18T19:02:28.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.816+0000 7fdca36c1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c06ae60 0x7fdc9c0c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:28.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.816+0000 7fdca36c1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc9c06eda0 0x7fdc9c0ae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:28.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.816+0000 7fdca36c1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc9c06f180 0x7fdc9c0c3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:28.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.816+0000 7fdca36c1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c0c4540 con 0x7fdc9c06ae60 2024-03-18T19:02:28.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.816+0000 7fdca36c1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c0c4820 con 0x7fdc9c06f180 2024-03-18T19:02:28.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.816+0000 7fdca36c1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c0c4b00 con 0x7fdc9c06eda0 2024-03-18T19:02:28.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.816+0000 7fdca145d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c06ae60 0x7fdc9c0c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:28.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.816+0000 7fdca145d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c06ae60 0x7fdc9c0c55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47228/0 (socket says 172.21.15.1:47228) 2024-03-18T19:02:28.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.816+0000 7fdca145d700 1 -- 172.21.15.1:0/3686436252 learned_addr learned my addr 172.21.15.1:0/3686436252 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:28.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.817+0000 7fdca1c5e700 1 --2- 172.21.15.1:0/3686436252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc9c06f180 0x7fdc9c0c3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:28.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.817+0000 7fdca0c5c700 1 --2- 172.21.15.1:0/3686436252 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc9c06eda0 0x7fdc9c0ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:28.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.817+0000 7fdca145d700 1 -- 172.21.15.1:0/3686436252 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc9c06eda0 msgr2=0x7fdc9c0ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:28.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.817+0000 7fdca145d700 1 --2- 172.21.15.1:0/3686436252 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc9c06eda0 0x7fdc9c0ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:28.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.817+0000 7fdca145d700 1 -- 172.21.15.1:0/3686436252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc9c06f180 msgr2=0x7fdc9c0c3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:28.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.817+0000 7fdca145d700 1 --2- 172.21.15.1:0/3686436252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc9c06f180 0x7fdc9c0c3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:28.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.817+0000 7fdca145d700 1 -- 172.21.15.1:0/3686436252 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc94009a70 con 0x7fdc9c06ae60 2024-03-18T19:02:28.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.817+0000 7fdca1c5e700 1 --2- 172.21.15.1:0/3686436252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc9c06f180 0x7fdc9c0c3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:28.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.817+0000 7fdca0c5c700 1 --2- 172.21.15.1:0/3686436252 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc9c06eda0 0x7fdc9c0ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:28.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.817+0000 7fdca145d700 1 --2- 172.21.15.1:0/3686436252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c06ae60 0x7fdc9c0c55e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fdc980011e0 tx=0x7fdc9800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:28.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.817+0000 7fdc91ffb700 1 -- 172.21.15.1:0/3686436252 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc98004c80 con 0x7fdc9c06ae60 2024-03-18T19:02:28.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.817+0000 7fdc91ffb700 1 -- 172.21.15.1:0/3686436252 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc98004e00 con 0x7fdc9c06ae60 2024-03-18T19:02:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.817+0000 7fdca36c1700 1 -- 172.21.15.1:0/3686436252 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc9c0c4de0 con 0x7fdc9c06ae60 2024-03-18T19:02:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.817+0000 7fdc91ffb700 1 -- 172.21.15.1:0/3686436252 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc98007310 con 0x7fdc9c06ae60 2024-03-18T19:02:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.817+0000 7fdca36c1700 1 -- 172.21.15.1:0/3686436252 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc9c0c5b20 con 0x7fdc9c06ae60 2024-03-18T19:02:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.818+0000 7fdca36c1700 1 -- 172.21.15.1:0/3686436252 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc9c0676a0 con 0x7fdc9c06ae60 2024-03-18T19:02:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.819+0000 7fdc91ffb700 1 -- 172.21.15.1:0/3686436252 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fdc98007470 con 0x7fdc9c06ae60 2024-03-18T19:02:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.819+0000 7fdc91ffb700 1 --2- 172.21.15.1:0/3686436252 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdc8006ae90 0x7fdc8006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.819+0000 7fdc91ffb700 1 -- 172.21.15.1:0/3686436252 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7fdc98089690 con 0x7fdc9c06ae60 2024-03-18T19:02:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.819+0000 7fdca0c5c700 1 --2- 172.21.15.1:0/3686436252 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdc8006ae90 0x7fdc8006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:28.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.819+0000 7fdca0c5c700 1 --2- 172.21.15.1:0/3686436252 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdc8006ae90 0x7fdc8006d350 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c1f6780 tx=0x7fdc8c008040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:28.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:28.824+0000 7fdc91ffb700 1 -- 172.21.15.1:0/3686436252 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdc98054390 con 0x7fdc9c06ae60 2024-03-18T19:02:29.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:29.057+0000 7fdca36c1700 1 -- 172.21.15.1:0/3686436252 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc9c068800 con 0x7fdc8006ae90 2024-03-18T19:02:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:29.141+0000 7fdc91ffb700 1 -- 172.21.15.1:0/3686436252 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fdc9c068800 con 0x7fdc8006ae90 2024-03-18T19:02:29.139 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule removed for path /snap_test_dir1/dir0 2024-03-18T19:02:29.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:29.144+0000 7fdca36c1700 1 -- 172.21.15.1:0/3686436252 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdc8006ae90 msgr2=0x7fdc8006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:29.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:29.144+0000 7fdca36c1700 1 --2- 172.21.15.1:0/3686436252 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdc8006ae90 0x7fdc8006d350 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c1f6780 tx=0x7fdc8c008040 comp rx=0 tx=0).stop 2024-03-18T19:02:29.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:29.144+0000 7fdca36c1700 1 -- 172.21.15.1:0/3686436252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c06ae60 msgr2=0x7fdc9c0c55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:29.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:29.144+0000 7fdca36c1700 1 --2- 172.21.15.1:0/3686436252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c06ae60 0x7fdc9c0c55e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fdc980011e0 tx=0x7fdc9800cae0 comp rx=0 tx=0).stop 2024-03-18T19:02:29.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:29.144+0000 7fdca36c1700 1 -- 172.21.15.1:0/3686436252 shutdown_connections 2024-03-18T19:02:29.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:29.144+0000 7fdca36c1700 1 --2- 172.21.15.1:0/3686436252 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdc8006ae90 0x7fdc8006d350 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:29.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:29.144+0000 7fdca36c1700 1 --2- 172.21.15.1:0/3686436252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc9c06f180 0x7fdc9c0c3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:29.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:29.144+0000 7fdca36c1700 1 --2- 172.21.15.1:0/3686436252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9c06ae60 0x7fdc9c0c55e0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:29.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:29.144+0000 7fdca36c1700 1 --2- 172.21.15.1:0/3686436252 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc9c06eda0 0x7fdc9c0ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:29.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:29.144+0000 7fdca36c1700 1 -- 172.21.15.1:0/3686436252 >> 172.21.15.1:0/3686436252 conn(0x7fdc9c0cff30 msgr2=0x7fdc9c0b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:29.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:29.144+0000 7fdca36c1700 1 -- 172.21.15.1:0/3686436252 shutdown_connections 2024-03-18T19:02:29.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:29.144+0000 7fdca36c1700 1 -- 172.21.15.1:0/3686436252 wait complete. 2024-03-18T19:02:29.152 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2024-03-18T19:02:29.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:29.153 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2024-03-18T19:02:29.175 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:29.175 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:29.175 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:29.175 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:29.175 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:29.175 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:29.175 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:29.175 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:29.176 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:29.176 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:29.176 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:29.176 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:29.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:29.176 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-18-18_52_00_UTC) 2024-03-18T19:02:29.241 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:29.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:29.242 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-18-18_53_00_UTC) 2024-03-18T19:02:29.307 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:29.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:29.308 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-18-18_54_00_UTC) 2024-03-18T19:02:29.373 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:29.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:29.373 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-18-18_55_00_UTC) 2024-03-18T19:02:29.439 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:29.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:29.439 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-18-18_56_00_UTC) 2024-03-18T19:02:29.505 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:29.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:29.505 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-18-18_57_00_UTC) 2024-03-18T19:02:29.570 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:29.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:29.570 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-18-18_58_00_UTC) 2024-03-18T19:02:29.637 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:29.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:29.637 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-18-18_59_00_UTC) 2024-03-18T19:02:29.703 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:29.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:29.703 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-18-19_00_00_UTC) 2024-03-18T19:02:29.769 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:29.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:29.770 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-18-19_01_00_UTC) 2024-03-18T19:02:29.837 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:29.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:29.837 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-03-18-19_02_00_UTC) 2024-03-18T19:02:29.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:29.904 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2024-03-18T19:02:29.968 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir1 1m 2024-03-18T19:02:30.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.140+0000 7f26f6acf700 1 Processor -- start 2024-03-18T19:02:30.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.140+0000 7f26f6acf700 1 -- start start 2024-03-18T19:02:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.141+0000 7f26f6acf700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26f006f180 0x7f26f0161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.141+0000 7f26f6acf700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f26f006a750 0x7f26f006ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.141+0000 7f26f6acf700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f006b0a0 con 0x7f26f006eda0 2024-03-18T19:02:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.141+0000 7f26f6acf700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26f00cb320 con 0x7f26f006f180 2024-03-18T19:02:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.141+0000 7f26f6acf700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26f00ce010 con 0x7f26f006a750 2024-03-18T19:02:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.141+0000 7f26effff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26f006f180 0x7f26f0161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.141+0000 7f26f486b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f26f006eda0 0x7f26f00ca580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41658/0 (socket says 172.21.15.1:41658) 2024-03-18T19:02:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.141+0000 7f26f486b700 1 -- 172.21.15.1:0/3092033898 learned_addr learned my addr 172.21.15.1:0/3092033898 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.141+0000 7f26f506c700 1 --2- 172.21.15.1:0/3092033898 >> v2:172.21.15.165:3300/0 conn(0x7f26f006a750 0x7f26f006ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:30.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.142+0000 7f26effff700 1 -- 172.21.15.1:0/3092033898 >> v2:172.21.15.165:3300/0 conn(0x7f26f006a750 msgr2=0x7f26f006ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.142+0000 7f26effff700 1 --2- 172.21.15.1:0/3092033898 >> v2:172.21.15.165:3300/0 conn(0x7f26f006a750 0x7f26f006ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.142+0000 7f26effff700 1 -- 172.21.15.1:0/3092033898 >> v1:172.21.15.1:6789/0 conn(0x7f26f006eda0 legacy=0x7f26f00ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.142+0000 7f26effff700 1 -- 172.21.15.1:0/3092033898 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f00cad60 con 0x7f26f006f180 2024-03-18T19:02:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.142+0000 7f26ef7fe700 1 -- 172.21.15.1:0/3092033898 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850757829 0 0) 0x7f26f006b0a0 con 0x7f26f006eda0 2024-03-18T19:02:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.142+0000 7f26f506c700 1 --2- 172.21.15.1:0/3092033898 >> v2:172.21.15.165:3300/0 conn(0x7f26f006a750 0x7f26f006ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.142+0000 7f26effff700 1 --2- 172.21.15.1:0/3092033898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26f006f180 0x7f26f0161c60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f26f01621a0 tx=0x7f26dc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=924ff8c4b76d1992 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.142+0000 7f26ef7fe700 1 -- 172.21.15.1:0/3092033898 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc011830 con 0x7f26f006f180 2024-03-18T19:02:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.142+0000 7f26ef7fe700 1 -- 172.21.15.1:0/3092033898 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26dc011990 con 0x7f26f006f180 2024-03-18T19:02:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.142+0000 7f26ef7fe700 1 -- 172.21.15.1:0/3092033898 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc011d20 con 0x7f26f006f180 2024-03-18T19:02:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.143+0000 7f26f6acf700 1 -- 172.21.15.1:0/3092033898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26f006f180 msgr2=0x7f26f0161c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.143+0000 7f26f6acf700 1 --2- 172.21.15.1:0/3092033898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26f006f180 0x7f26f0161c60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f26f01621a0 tx=0x7f26dc009b20 comp rx=0 tx=0).stop 2024-03-18T19:02:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.143+0000 7f26f6acf700 1 -- 172.21.15.1:0/3092033898 shutdown_connections 2024-03-18T19:02:30.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.143+0000 7f26f6acf700 1 --2- 172.21.15.1:0/3092033898 >> v2:172.21.15.165:3300/0 conn(0x7f26f006a750 0x7f26f006ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.143+0000 7f26f6acf700 1 --2- 172.21.15.1:0/3092033898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26f006f180 0x7f26f0161c60 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.143+0000 7f26f6acf700 1 -- 172.21.15.1:0/3092033898 >> 172.21.15.1:0/3092033898 conn(0x7f26f00cff30 msgr2=0x7f26f00d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.143+0000 7f26f6acf700 1 -- 172.21.15.1:0/3092033898 shutdown_connections 2024-03-18T19:02:30.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.143+0000 7f26f6acf700 1 -- 172.21.15.1:0/3092033898 wait complete. 2024-03-18T19:02:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.144+0000 7f26f6acf700 1 Processor -- start 2024-03-18T19:02:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.144+0000 7f26f6acf700 1 -- start start 2024-03-18T19:02:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.145+0000 7f26f6acf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26f006a750 0x7f26f01f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.145+0000 7f26f6acf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f26f006eda0 0x7f26f00c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.145+0000 7f26f6acf700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26f006f180 0x7f26f01f6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.145+0000 7f26f6acf700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f01f74b0 con 0x7f26f006a750 2024-03-18T19:02:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.145+0000 7f26f6acf700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f01f6ef0 con 0x7f26f006f180 2024-03-18T19:02:30.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.145+0000 7f26f6acf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f01f71d0 con 0x7f26f006eda0 2024-03-18T19:02:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.145+0000 7f26f486b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26f006a750 0x7f26f01f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.145+0000 7f26f486b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26f006a750 0x7f26f01f2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47260/0 (socket says 172.21.15.1:47260) 2024-03-18T19:02:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.145+0000 7f26f486b700 1 -- 172.21.15.1:0/386979315 learned_addr learned my addr 172.21.15.1:0/386979315 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.146+0000 7f26f486b700 1 -- 172.21.15.1:0/386979315 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f26f006eda0 msgr2=0x7f26f00c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.146+0000 7f26f486b700 1 --2- 172.21.15.1:0/386979315 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f26f006eda0 0x7f26f00c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.146+0000 7f26f486b700 1 -- 172.21.15.1:0/386979315 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26f006f180 msgr2=0x7f26f01f6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.146+0000 7f26f486b700 1 --2- 172.21.15.1:0/386979315 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26f006f180 0x7f26f01f6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.146+0000 7f26f486b700 1 -- 172.21.15.1:0/386979315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc0094a0 con 0x7f26f006a750 2024-03-18T19:02:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.146+0000 7f26f486b700 1 --2- 172.21.15.1:0/386979315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26f006a750 0x7f26f01f2830 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f26e40011e0 tx=0x7f26e400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.146+0000 7f26ed7fa700 1 -- 172.21.15.1:0/386979315 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e4004c80 con 0x7f26f006a750 2024-03-18T19:02:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.146+0000 7f26ed7fa700 1 -- 172.21.15.1:0/386979315 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e4004e00 con 0x7f26f006a750 2024-03-18T19:02:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.146+0000 7f26f6acf700 1 -- 172.21.15.1:0/386979315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f00c7130 con 0x7f26f006a750 2024-03-18T19:02:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.146+0000 7f26ed7fa700 1 -- 172.21.15.1:0/386979315 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e4007310 con 0x7f26f006a750 2024-03-18T19:02:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.146+0000 7f26f6acf700 1 -- 172.21.15.1:0/386979315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f01f2d70 con 0x7f26f006a750 2024-03-18T19:02:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.147+0000 7f26f6acf700 1 -- 172.21.15.1:0/386979315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f00c7650 con 0x7f26f006a750 2024-03-18T19:02:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.147+0000 7f26ed7fa700 1 -- 172.21.15.1:0/386979315 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f26e4007470 con 0x7f26f006a750 2024-03-18T19:02:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.148+0000 7f26ed7fa700 1 --2- 172.21.15.1:0/386979315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f26d406ae90 0x7f26d406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.148+0000 7f26ed7fa700 1 -- 172.21.15.1:0/386979315 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6316+0+0 (secure 0 0 0) 0x7f26e4089640 con 0x7f26f006a750 2024-03-18T19:02:30.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.148+0000 7f26effff700 1 --2- 172.21.15.1:0/386979315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f26d406ae90 0x7f26d406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:30.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.148+0000 7f26effff700 1 --2- 172.21.15.1:0/386979315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f26d406ae90 0x7f26d406d350 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f26f01621a0 tx=0x7f26dc010000 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:30.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.152+0000 7f26ed7fa700 1 -- 172.21.15.1:0/386979315 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f26f00c7650 con 0x7f26f006a750 2024-03-18T19:02:30.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.381+0000 7f26f6acf700 1 -- 172.21.15.1:0/386979315 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f26f00ce500 con 0x7f26d406ae90 2024-03-18T19:02:30.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.467+0000 7f26ed7fa700 1 -- 172.21.15.1:0/386979315 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f26dc01e070 con 0x7f26d406ae90 2024-03-18T19:02:30.464 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule removed for path /snap_test_dir1/dir1 2024-03-18T19:02:30.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.469+0000 7f26f6acf700 1 -- 172.21.15.1:0/386979315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f26d406ae90 msgr2=0x7f26d406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:30.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.469+0000 7f26f6acf700 1 --2- 172.21.15.1:0/386979315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f26d406ae90 0x7f26d406d350 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f26f01621a0 tx=0x7f26dc010000 comp rx=0 tx=0).stop 2024-03-18T19:02:30.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.469+0000 7f26f6acf700 1 -- 172.21.15.1:0/386979315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26f006a750 msgr2=0x7f26f01f2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:30.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.469+0000 7f26f6acf700 1 --2- 172.21.15.1:0/386979315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26f006a750 0x7f26f01f2830 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f26e40011e0 tx=0x7f26e400cae0 comp rx=0 tx=0).stop 2024-03-18T19:02:30.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.469+0000 7f26f6acf700 1 -- 172.21.15.1:0/386979315 shutdown_connections 2024-03-18T19:02:30.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.469+0000 7f26f6acf700 1 --2- 172.21.15.1:0/386979315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f26d406ae90 0x7f26d406d350 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:30.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.469+0000 7f26f6acf700 1 --2- 172.21.15.1:0/386979315 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26f006f180 0x7f26f01f6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:30.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.469+0000 7f26f6acf700 1 --2- 172.21.15.1:0/386979315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26f006a750 0x7f26f01f2830 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:30.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.469+0000 7f26f6acf700 1 --2- 172.21.15.1:0/386979315 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f26f006eda0 0x7f26f00c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:30.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.469+0000 7f26f6acf700 1 -- 172.21.15.1:0/386979315 >> 172.21.15.1:0/386979315 conn(0x7f26f00cff30 msgr2=0x7f26f00cf210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:30.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.469+0000 7f26f6acf700 1 -- 172.21.15.1:0/386979315 shutdown_connections 2024-03-18T19:02:30.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:30.469+0000 7f26f6acf700 1 -- 172.21.15.1:0/386979315 wait complete. 2024-03-18T19:02:30.478 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2024-03-18T19:02:30.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:30.478 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2024-03-18T19:02:30.502 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:30.502 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:30.502 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:30.502 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:30.502 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:30.502 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:30.502 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:30.502 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:30.503 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:30.503 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:30.503 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:30.503 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:30.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:30.503 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-18-18_52_00_UTC) 2024-03-18T19:02:30.569 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:30.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:30.569 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-18-18_53_00_UTC) 2024-03-18T19:02:30.635 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:30.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:30.635 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-18-18_54_00_UTC) 2024-03-18T19:02:30.701 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:30.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:30.702 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-18-18_55_00_UTC) 2024-03-18T19:02:30.767 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:30.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:30.767 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-18-18_56_00_UTC) 2024-03-18T19:02:30.834 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:30.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:30.834 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-18-18_57_00_UTC) 2024-03-18T19:02:30.900 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:30.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:30.901 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-18-18_58_00_UTC) 2024-03-18T19:02:30.967 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:30.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:30.967 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-18-18_59_00_UTC) 2024-03-18T19:02:31.034 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:31.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:31.034 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-18-19_00_00_UTC) 2024-03-18T19:02:31.116 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:31.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:31.116 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-18-19_01_00_UTC) 2024-03-18T19:02:31.192 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:31.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:31.192 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-03-18-19_02_00_UTC) 2024-03-18T19:02:31.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:31.257 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2024-03-18T19:02:31.319 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir2 1m 2024-03-18T19:02:31.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.503+0000 7f22a3b14700 1 Processor -- start 2024-03-18T19:02:31.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.503+0000 7f22a3b14700 1 -- start start 2024-03-18T19:02:31.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.505+0000 7f22a3b14700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f229c0ce190 0x7f229c070de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:31.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.505+0000 7f22a3b14700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229c0cc6e0 con 0x7f229c14b1b0 2024-03-18T19:02:31.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.505+0000 7f22a3b14700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229c071440 con 0x7f229c06d1f0 2024-03-18T19:02:31.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.505+0000 7f22a3b14700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f229c14b590 con 0x7f229c0ce190 2024-03-18T19:02:31.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.505+0000 7f22a10af700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f229c06d1f0 0x7f229c0706d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:43654/0 (socket says 172.21.15.1:43654) 2024-03-18T19:02:31.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.505+0000 7f22a10af700 1 -- 172.21.15.1:0/3948735608 learned_addr learned my addr 172.21.15.1:0/3948735608 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:31.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.505+0000 7f22a20b1700 1 --2- 172.21.15.1:0/3948735608 >> v2:172.21.15.1:3300/0 conn(0x7f229c0ce190 0x7f229c070de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:31.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.506+0000 7f22a20b1700 1 -- 172.21.15.1:0/3948735608 >> v1:172.21.15.165:6789/0 conn(0x7f229c06d1f0 legacy=0x7f229c0706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:31.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.506+0000 7f22a08ae700 1 -- 172.21.15.1:0/3948735608 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40107087 0 0) 0x7f229c071440 con 0x7f229c06d1f0 2024-03-18T19:02:31.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.506+0000 7f22a20b1700 1 -- 172.21.15.1:0/3948735608 >> v1:172.21.15.1:6789/0 conn(0x7f229c14b1b0 legacy=0x7f229c06cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:31.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.506+0000 7f22a20b1700 1 -- 172.21.15.1:0/3948735608 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229c071720 con 0x7f229c0ce190 2024-03-18T19:02:31.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.506+0000 7f22a20b1700 1 --2- 172.21.15.1:0/3948735608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f229c0ce190 0x7f229c070de0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2294009d90 tx=0x7f229400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6eb5f9291f504ca server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:31.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.506+0000 7f22a08ae700 1 -- 172.21.15.1:0/3948735608 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229400e070 con 0x7f229c0ce190 2024-03-18T19:02:31.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.506+0000 7f22a08ae700 1 -- 172.21.15.1:0/3948735608 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229400bb90 con 0x7f229c0ce190 2024-03-18T19:02:31.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.506+0000 7f22a08ae700 1 -- 172.21.15.1:0/3948735608 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2294003a20 con 0x7f229c0ce190 2024-03-18T19:02:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.507+0000 7f22a3b14700 1 -- 172.21.15.1:0/3948735608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f229c0ce190 msgr2=0x7f229c070de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.507+0000 7f22a3b14700 1 --2- 172.21.15.1:0/3948735608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f229c0ce190 0x7f229c070de0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2294009d90 tx=0x7f229400b330 comp rx=0 tx=0).stop 2024-03-18T19:02:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.507+0000 7f22a3b14700 1 -- 172.21.15.1:0/3948735608 shutdown_connections 2024-03-18T19:02:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.507+0000 7f22a3b14700 1 --2- 172.21.15.1:0/3948735608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f229c0ce190 0x7f229c070de0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.507+0000 7f22a3b14700 1 -- 172.21.15.1:0/3948735608 >> 172.21.15.1:0/3948735608 conn(0x7f229c06a750 msgr2=0x7f229c06ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.508+0000 7f22a3b14700 1 -- 172.21.15.1:0/3948735608 shutdown_connections 2024-03-18T19:02:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.508+0000 7f22a3b14700 1 -- 172.21.15.1:0/3948735608 wait complete. 2024-03-18T19:02:31.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.510+0000 7f22a3b14700 1 Processor -- start 2024-03-18T19:02:31.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.510+0000 7f22a3b14700 1 -- start start 2024-03-18T19:02:31.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.511+0000 7f22a3b14700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f229c06d1f0 0x7f229c164140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:31.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.511+0000 7f22a3b14700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f229c0ce190 0x7f229c160a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:31.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.511+0000 7f22a3b14700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f229c14b1b0 0x7f229c160f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:31.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.512+0000 7f22a3b14700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229c071a00 con 0x7f229c14b1b0 2024-03-18T19:02:31.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.512+0000 7f22a3b14700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229c136920 con 0x7f229c0ce190 2024-03-18T19:02:31.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.512+0000 7f22a3b14700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229c1614a0 con 0x7f229c06d1f0 2024-03-18T19:02:31.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.512+0000 7f22a10af700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f229c0ce190 0x7f229c160a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:31.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.512+0000 7f22a18b0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f229c06d1f0 0x7f229c164140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:31.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.512+0000 7f22a18b0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f229c06d1f0 0x7f229c164140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:36326/0 (socket says 172.21.15.1:36326) 2024-03-18T19:02:31.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.512+0000 7f22a18b0700 1 -- 172.21.15.1:0/3321986832 learned_addr learned my addr 172.21.15.1:0/3321986832 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:31.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.512+0000 7f22a18b0700 1 -- 172.21.15.1:0/3321986832 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f229c0ce190 msgr2=0x7f229c160a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:31.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.512+0000 7f22a18b0700 1 --2- 172.21.15.1:0/3321986832 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f229c0ce190 0x7f229c160a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:31.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.512+0000 7f22a18b0700 1 -- 172.21.15.1:0/3321986832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f229c14b1b0 msgr2=0x7f229c160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:31.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.512+0000 7f22a18b0700 1 --2- 172.21.15.1:0/3321986832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f229c14b1b0 0x7f229c160f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:31.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.512+0000 7f22a18b0700 1 -- 172.21.15.1:0/3321986832 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2294009a70 con 0x7f229c06d1f0 2024-03-18T19:02:31.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.513+0000 7f22a18b0700 1 --2- 172.21.15.1:0/3321986832 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f229c06d1f0 0x7f229c164140 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f229800b4a0 tx=0x7f229800b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:31.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.513+0000 7f22927fc700 1 -- 172.21.15.1:0/3321986832 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298011830 con 0x7f229c06d1f0 2024-03-18T19:02:31.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.513+0000 7f22927fc700 1 -- 172.21.15.1:0/3321986832 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2298011990 con 0x7f229c06d1f0 2024-03-18T19:02:31.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.513+0000 7f22927fc700 1 -- 172.21.15.1:0/3321986832 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298011d40 con 0x7f229c06d1f0 2024-03-18T19:02:31.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.513+0000 7f22a3b14700 1 -- 172.21.15.1:0/3321986832 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f229c138e50 con 0x7f229c06d1f0 2024-03-18T19:02:31.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.513+0000 7f22a3b14700 1 -- 172.21.15.1:0/3321986832 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f229c139410 con 0x7f229c06d1f0 2024-03-18T19:02:31.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.515+0000 7f22a3b14700 1 -- 172.21.15.1:0/3321986832 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f229c139630 con 0x7f229c06d1f0 2024-03-18T19:02:31.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.515+0000 7f22927fc700 1 -- 172.21.15.1:0/3321986832 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f229801a400 con 0x7f229c06d1f0 2024-03-18T19:02:31.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.515+0000 7f22927fc700 1 --2- 172.21.15.1:0/3321986832 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f228006af60 0x7f228006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:31.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.516+0000 7f22927fc700 1 -- 172.21.15.1:0/3321986832 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6692+0+0 (secure 0 0 0) 0x7f2298089ad0 con 0x7f229c06d1f0 2024-03-18T19:02:31.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.516+0000 7f22a10af700 1 --2- 172.21.15.1:0/3321986832 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f228006af60 0x7f228006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:31.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.516+0000 7f22a10af700 1 --2- 172.21.15.1:0/3321986832 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f228006af60 0x7f228006d420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f228c007fd0 tx=0x7f228c007c20 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:31.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.521+0000 7f22927fc700 1 -- 172.21.15.1:0/3321986832 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2298054630 con 0x7f229c06d1f0 2024-03-18T19:02:31.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.758+0000 7f22a3b14700 1 -- 172.21.15.1:0/3321986832 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f229c0ccc60 con 0x7f228006af60 2024-03-18T19:02:31.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.828+0000 7f22927fc700 1 -- 172.21.15.1:0/3321986832 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f228c005a20 con 0x7f228006af60 2024-03-18T19:02:31.825 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule removed for path /snap_test_dir1/dir2 2024-03-18T19:02:31.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.830+0000 7f22a3b14700 1 -- 172.21.15.1:0/3321986832 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f228006af60 msgr2=0x7f228006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:31.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.830+0000 7f22a3b14700 1 --2- 172.21.15.1:0/3321986832 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f228006af60 0x7f228006d420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f228c007fd0 tx=0x7f228c007c20 comp rx=0 tx=0).stop 2024-03-18T19:02:31.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.830+0000 7f22a3b14700 1 -- 172.21.15.1:0/3321986832 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f229c06d1f0 msgr2=0x7f229c164140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:31.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.830+0000 7f22a3b14700 1 --2- 172.21.15.1:0/3321986832 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f229c06d1f0 0x7f229c164140 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f229800b4a0 tx=0x7f229800b860 comp rx=0 tx=0).stop 2024-03-18T19:02:31.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.830+0000 7f22a3b14700 1 -- 172.21.15.1:0/3321986832 shutdown_connections 2024-03-18T19:02:31.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.830+0000 7f22a3b14700 1 --2- 172.21.15.1:0/3321986832 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f228006af60 0x7f228006d420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:31.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.830+0000 7f22a3b14700 1 --2- 172.21.15.1:0/3321986832 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f229c06d1f0 0x7f229c164140 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:31.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.830+0000 7f22a3b14700 1 --2- 172.21.15.1:0/3321986832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f229c14b1b0 0x7f229c160f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:31.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.830+0000 7f22a3b14700 1 --2- 172.21.15.1:0/3321986832 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f229c0ce190 0x7f229c160a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:31.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.830+0000 7f22a3b14700 1 -- 172.21.15.1:0/3321986832 >> 172.21.15.1:0/3321986832 conn(0x7f229c06a750 msgr2=0x7f229c0b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:31.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.830+0000 7f22a3b14700 1 -- 172.21.15.1:0/3321986832 shutdown_connections 2024-03-18T19:02:31.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:31.830+0000 7f22a3b14700 1 -- 172.21.15.1:0/3321986832 wait complete. 2024-03-18T19:02:31.839 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2024-03-18T19:02:31.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:31.839 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2024-03-18T19:02:31.866 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:31.866 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:31.867 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:31.867 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:31.867 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:31.867 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:31.867 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:31.867 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:31.867 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_01_UTC 2024-03-18T19:02:31.867 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:31.867 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:31.867 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:31.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:31.868 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-18-18_52_00_UTC) 2024-03-18T19:02:31.934 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:31.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:31.935 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-18-18_53_00_UTC) 2024-03-18T19:02:32.001 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:32.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:32.001 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-18-18_54_00_UTC) 2024-03-18T19:02:32.081 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:32.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:32.081 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-18-18_55_00_UTC) 2024-03-18T19:02:32.148 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:32.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:32.149 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-18-18_56_00_UTC) 2024-03-18T19:02:32.215 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:32.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:32.216 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-18-18_57_00_UTC) 2024-03-18T19:02:32.281 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:32.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:32.281 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-18-18_58_00_UTC) 2024-03-18T19:02:32.348 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:32.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:32.348 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-18-18_59_00_UTC) 2024-03-18T19:02:32.413 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-18-19_00_01_UTC 2024-03-18T19:02:32.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:32.414 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-18-19_00_01_UTC) 2024-03-18T19:02:32.479 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:32.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:32.479 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-18-19_01_00_UTC) 2024-03-18T19:02:32.545 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:32.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:32.545 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-03-18-19_02_00_UTC) 2024-03-18T19:02:32.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:32.623 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2024-03-18T19:02:32.684 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir3 1m 2024-03-18T19:02:32.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.856+0000 7f8450b89700 1 Processor -- start 2024-03-18T19:02:32.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.856+0000 7f8450b89700 1 -- start start 2024-03-18T19:02:32.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.857+0000 7f8450b89700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f844c071d30 0x7f844c0add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:32.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.857+0000 7f8450b89700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f844c0cb9e0 0x7f844c0cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:32.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.857+0000 7f8450b89700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c0ce8f0 con 0x7f844c15a1d0 2024-03-18T19:02:32.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.857+0000 7f8450b89700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f844c0cdb00 con 0x7f844c071d30 2024-03-18T19:02:32.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.857+0000 7f8450b89700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844c15a5b0 con 0x7f844c0cb9e0 2024-03-18T19:02:32.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.857+0000 7f8449d9b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f844c071d30 0x7f844c0add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:32.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.857+0000 7f844ad9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f844c0cb9e0 0x7f844c0cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:32.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.857+0000 7f844a59c700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f844c15a1d0 0x7f844c071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:43658/0 (socket says 172.21.15.1:43658) 2024-03-18T19:02:32.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.858+0000 7f844a59c700 1 -- 172.21.15.1:0/2838112451 learned_addr learned my addr 172.21.15.1:0/2838112451 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:32.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.858+0000 7f844ad9d700 1 -- 172.21.15.1:0/2838112451 >> v2:172.21.15.165:3300/0 conn(0x7f844c071d30 msgr2=0x7f844c0add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:32.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.858+0000 7f844ad9d700 1 --2- 172.21.15.1:0/2838112451 >> v2:172.21.15.165:3300/0 conn(0x7f844c071d30 0x7f844c0add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:32.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.858+0000 7f844ad9d700 1 -- 172.21.15.1:0/2838112451 >> v1:172.21.15.165:6789/0 conn(0x7f844c15a1d0 legacy=0x7f844c071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:32.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.858+0000 7f844ad9d700 1 -- 172.21.15.1:0/2838112451 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c0ae260 con 0x7f844c0cb9e0 2024-03-18T19:02:32.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.858+0000 7f8449d9b700 1 --2- 172.21.15.1:0/2838112451 >> v2:172.21.15.165:3300/0 conn(0x7f844c071d30 0x7f844c0add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:02:32.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.858+0000 7f844ad9d700 1 --2- 172.21.15.1:0/2838112451 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f844c0cb9e0 0x7f844c0cbe70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8444009d90 tx=0x7f844400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d826d07057edb6a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:32.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.858+0000 7f844959a700 1 -- 172.21.15.1:0/2838112451 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844400e070 con 0x7f844c0cb9e0 2024-03-18T19:02:32.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.858+0000 7f844959a700 1 -- 172.21.15.1:0/2838112451 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844400bb20 con 0x7f844c0cb9e0 2024-03-18T19:02:32.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.858+0000 7f844959a700 1 -- 172.21.15.1:0/2838112451 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444012430 con 0x7f844c0cb9e0 2024-03-18T19:02:32.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.859+0000 7f8450b89700 1 -- 172.21.15.1:0/2838112451 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f844c0cb9e0 msgr2=0x7f844c0cbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:32.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.859+0000 7f8450b89700 1 --2- 172.21.15.1:0/2838112451 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f844c0cb9e0 0x7f844c0cbe70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8444009d90 tx=0x7f844400b330 comp rx=0 tx=0).stop 2024-03-18T19:02:32.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.859+0000 7f8450b89700 1 -- 172.21.15.1:0/2838112451 shutdown_connections 2024-03-18T19:02:32.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.859+0000 7f8450b89700 1 --2- 172.21.15.1:0/2838112451 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f844c0cb9e0 0x7f844c0cbe70 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:32.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.859+0000 7f8450b89700 1 --2- 172.21.15.1:0/2838112451 >> v2:172.21.15.165:3300/0 conn(0x7f844c071d30 0x7f844c0add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:32.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.859+0000 7f8450b89700 1 -- 172.21.15.1:0/2838112451 >> 172.21.15.1:0/2838112451 conn(0x7f844c0cc9b0 msgr2=0x7f844c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:32.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.859+0000 7f8450b89700 1 -- 172.21.15.1:0/2838112451 shutdown_connections 2024-03-18T19:02:32.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.859+0000 7f8450b89700 1 -- 172.21.15.1:0/2838112451 wait complete. 2024-03-18T19:02:32.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.860+0000 7f8450b89700 1 Processor -- start 2024-03-18T19:02:32.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.860+0000 7f8450b89700 1 -- start start 2024-03-18T19:02:32.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.861+0000 7f8450b89700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f844c071d30 0x7f844c06ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:32.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.861+0000 7f8450b89700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f844c0cb9e0 0x7f844c0c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:32.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.861+0000 7f8450b89700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f844c15a1d0 0x7f844c14b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:32.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.861+0000 7f8450b89700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c0c7f50 con 0x7f844c0cb9e0 2024-03-18T19:02:32.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.861+0000 7f8450b89700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c0c8d70 con 0x7f844c071d30 2024-03-18T19:02:32.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.861+0000 7f8450b89700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844c0c8160 con 0x7f844c15a1d0 2024-03-18T19:02:32.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.861+0000 7f8449d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f844c0cb9e0 0x7f844c0c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:32.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.861+0000 7f8449d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f844c0cb9e0 0x7f844c0c7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47278/0 (socket says 172.21.15.1:47278) 2024-03-18T19:02:32.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.861+0000 7f8449d9b700 1 -- 172.21.15.1:0/1400013123 learned_addr learned my addr 172.21.15.1:0/1400013123 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:32.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.861+0000 7f844ad9d700 1 --2- 172.21.15.1:0/1400013123 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f844c15a1d0 0x7f844c14b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:32.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.861+0000 7f844a59c700 1 --2- 172.21.15.1:0/1400013123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f844c071d30 0x7f844c06ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:32.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.861+0000 7f8449d9b700 1 -- 172.21.15.1:0/1400013123 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f844c15a1d0 msgr2=0x7f844c14b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:32.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.861+0000 7f8449d9b700 1 --2- 172.21.15.1:0/1400013123 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f844c15a1d0 0x7f844c14b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:32.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.861+0000 7f8449d9b700 1 -- 172.21.15.1:0/1400013123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f844c071d30 msgr2=0x7f844c06ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:32.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.861+0000 7f8449d9b700 1 --2- 172.21.15.1:0/1400013123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f844c071d30 0x7f844c06ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:32.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.861+0000 7f8449d9b700 1 -- 172.21.15.1:0/1400013123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c0094a0 con 0x7f844c0cb9e0 2024-03-18T19:02:32.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.862+0000 7f8449d9b700 1 --2- 172.21.15.1:0/1400013123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f844c0cb9e0 0x7f844c0c7a10 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f843c00ec10 tx=0x7f843c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:32.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.862+0000 7f843affd700 1 -- 172.21.15.1:0/1400013123 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c00c850 con 0x7f844c0cb9e0 2024-03-18T19:02:32.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.862+0000 7f843affd700 1 -- 172.21.15.1:0/1400013123 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843c00c9b0 con 0x7f844c0cb9e0 2024-03-18T19:02:32.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.862+0000 7f8450b89700 1 -- 172.21.15.1:0/1400013123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8444009a70 con 0x7f844c0cb9e0 2024-03-18T19:02:32.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.862+0000 7f843affd700 1 -- 172.21.15.1:0/1400013123 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c00cd20 con 0x7f844c0cb9e0 2024-03-18T19:02:32.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.862+0000 7f8450b89700 1 -- 172.21.15.1:0/1400013123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c0c8720 con 0x7f844c0cb9e0 2024-03-18T19:02:32.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.863+0000 7f8450b89700 1 -- 172.21.15.1:0/1400013123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844c0cfc20 con 0x7f844c0cb9e0 2024-03-18T19:02:32.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.863+0000 7f843affd700 1 -- 172.21.15.1:0/1400013123 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f843c013470 con 0x7f844c0cb9e0 2024-03-18T19:02:32.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.864+0000 7f843affd700 1 --2- 172.21.15.1:0/1400013123 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f842c06ae90 0x7f842c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:32.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.864+0000 7f843affd700 1 -- 172.21.15.1:0/1400013123 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6756+0+0 (secure 0 0 0) 0x7f843c089ac0 con 0x7f844c0cb9e0 2024-03-18T19:02:32.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.864+0000 7f844a59c700 1 --2- 172.21.15.1:0/1400013123 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f842c06ae90 0x7f842c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:32.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.864+0000 7f844a59c700 1 --2- 172.21.15.1:0/1400013123 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f842c06ae90 0x7f842c06d350 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8440005960 tx=0x7f8440009040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:32.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:32.869+0000 7f843affd700 1 -- 172.21.15.1:0/1400013123 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f843c054690 con 0x7f844c0cb9e0 2024-03-18T19:02:33.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:33.100+0000 7f8450b89700 1 -- 172.21.15.1:0/1400013123 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f844c0cdfc0 con 0x7f842c06ae90 2024-03-18T19:02:33.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:33.188+0000 7f843affd700 1 -- 172.21.15.1:0/1400013123 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f8440001550 con 0x7f842c06ae90 2024-03-18T19:02:33.185 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule removed for path /snap_test_dir1/dir3 2024-03-18T19:02:33.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:33.190+0000 7f8450b89700 1 -- 172.21.15.1:0/1400013123 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f842c06ae90 msgr2=0x7f842c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:33.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:33.190+0000 7f8450b89700 1 --2- 172.21.15.1:0/1400013123 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f842c06ae90 0x7f842c06d350 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8440005960 tx=0x7f8440009040 comp rx=0 tx=0).stop 2024-03-18T19:02:33.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:33.190+0000 7f8450b89700 1 -- 172.21.15.1:0/1400013123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f844c0cb9e0 msgr2=0x7f844c0c7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:33.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:33.190+0000 7f8450b89700 1 --2- 172.21.15.1:0/1400013123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f844c0cb9e0 0x7f844c0c7a10 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f843c00ec10 tx=0x7f843c00ef20 comp rx=0 tx=0).stop 2024-03-18T19:02:33.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:33.190+0000 7f8450b89700 1 -- 172.21.15.1:0/1400013123 shutdown_connections 2024-03-18T19:02:33.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:33.190+0000 7f8450b89700 1 --2- 172.21.15.1:0/1400013123 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f842c06ae90 0x7f842c06d350 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:33.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:33.190+0000 7f8450b89700 1 --2- 172.21.15.1:0/1400013123 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f844c15a1d0 0x7f844c14b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:33.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:33.190+0000 7f8450b89700 1 --2- 172.21.15.1:0/1400013123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f844c071d30 0x7f844c06ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:33.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:33.190+0000 7f8450b89700 1 --2- 172.21.15.1:0/1400013123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f844c0cb9e0 0x7f844c0c7a10 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:33.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:33.190+0000 7f8450b89700 1 -- 172.21.15.1:0/1400013123 >> 172.21.15.1:0/1400013123 conn(0x7f844c0cc9b0 msgr2=0x7f844c06f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:33.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:33.190+0000 7f8450b89700 1 -- 172.21.15.1:0/1400013123 shutdown_connections 2024-03-18T19:02:33.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:33.190+0000 7f8450b89700 1 -- 172.21.15.1:0/1400013123 wait complete. 2024-03-18T19:02:33.199 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2024-03-18T19:02:33.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:33.199 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2024-03-18T19:02:33.230 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:33.230 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_01_UTC 2024-03-18T19:02:33.230 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:33.231 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:33.231 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:33.231 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:33.231 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:33.231 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:33.231 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:33.231 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:33.231 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:33.232 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:33.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:33.232 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-18-18_52_00_UTC) 2024-03-18T19:02:33.298 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-18-18_53_01_UTC 2024-03-18T19:02:33.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:33.298 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-18-18_53_01_UTC) 2024-03-18T19:02:33.364 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:33.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:33.365 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-18-18_54_00_UTC) 2024-03-18T19:02:33.430 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:33.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:33.431 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-18-18_55_00_UTC) 2024-03-18T19:02:33.496 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:33.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:33.496 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-18-18_56_00_UTC) 2024-03-18T19:02:33.562 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:33.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:33.562 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-18-18_57_00_UTC) 2024-03-18T19:02:33.628 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:33.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:33.629 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-18-18_58_00_UTC) 2024-03-18T19:02:33.695 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:33.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:33.695 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-18-18_59_00_UTC) 2024-03-18T19:02:33.761 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:33.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:33.761 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-18-19_00_00_UTC) 2024-03-18T19:02:33.827 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:33.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:33.827 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-18-19_01_00_UTC) 2024-03-18T19:02:33.894 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:33.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:33.894 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-03-18-19_02_00_UTC) 2024-03-18T19:02:33.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:33.961 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2024-03-18T19:02:34.029 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir4 1m 2024-03-18T19:02:34.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.209+0000 7efd08ff4700 1 Processor -- start 2024-03-18T19:02:34.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.209+0000 7efd08ff4700 1 -- start start 2024-03-18T19:02:34.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.210+0000 7efd08ff4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efd040cc0f0 0x7efd040add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:34.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.210+0000 7efd08ff4700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd040ce8f0 con 0x7efd04071d30 2024-03-18T19:02:34.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.210+0000 7efd08ff4700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd040ae260 con 0x7efd0415a1d0 2024-03-18T19:02:34.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.210+0000 7efd08ff4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd040cdb00 con 0x7efd040cc0f0 2024-03-18T19:02:34.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.211+0000 7efd02d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efd040cc0f0 0x7efd040add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:34.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.211+0000 7efd02d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efd040cc0f0 0x7efd040add20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47294/0 (socket says 172.21.15.1:47294) 2024-03-18T19:02:34.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.211+0000 7efd01d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efd04071d30 0x7efd040cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41686/0 (socket says 172.21.15.1:41686) 2024-03-18T19:02:34.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.211+0000 7efd02d9d700 1 -- 172.21.15.1:0/1576419751 learned_addr learned my addr 172.21.15.1:0/1576419751 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:34.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.211+0000 7efd02d9d700 1 -- 172.21.15.1:0/1576419751 >> v1:172.21.15.165:6789/0 conn(0x7efd0415a1d0 legacy=0x7efd04071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:34.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.211+0000 7efd02d9d700 1 -- 172.21.15.1:0/1576419751 >> v1:172.21.15.1:6789/0 conn(0x7efd04071d30 legacy=0x7efd040cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:34.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.211+0000 7efd02d9d700 1 -- 172.21.15.1:0/1576419751 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd040cdce0 con 0x7efd040cc0f0 2024-03-18T19:02:34.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.211+0000 7efd0159a700 1 -- 172.21.15.1:0/1576419751 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923704593 0 0) 0x7efd040ce8f0 con 0x7efd04071d30 2024-03-18T19:02:34.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.211+0000 7efd02d9d700 1 --2- 172.21.15.1:0/1576419751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd040cc0f0 0x7efd040add20 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009e50 tx=0x7efcfc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36acad472a96688d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:34.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.211+0000 7efd0159a700 1 -- 172.21.15.1:0/1576419751 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc00e070 con 0x7efd040cc0f0 2024-03-18T19:02:34.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.211+0000 7efd0159a700 1 -- 172.21.15.1:0/1576419751 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcfc00b9c0 con 0x7efd040cc0f0 2024-03-18T19:02:34.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.211+0000 7efd0159a700 1 -- 172.21.15.1:0/1576419751 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc00bd70 con 0x7efd040cc0f0 2024-03-18T19:02:34.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.212+0000 7efd08ff4700 1 -- 172.21.15.1:0/1576419751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd040cc0f0 msgr2=0x7efd040add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:34.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.212+0000 7efd08ff4700 1 --2- 172.21.15.1:0/1576419751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd040cc0f0 0x7efd040add20 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009e50 tx=0x7efcfc00b330 comp rx=0 tx=0).stop 2024-03-18T19:02:34.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.212+0000 7efd08ff4700 1 -- 172.21.15.1:0/1576419751 shutdown_connections 2024-03-18T19:02:34.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.212+0000 7efd08ff4700 1 --2- 172.21.15.1:0/1576419751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd040cc0f0 0x7efd040add20 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:34.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.212+0000 7efd08ff4700 1 -- 172.21.15.1:0/1576419751 >> 172.21.15.1:0/1576419751 conn(0x7efd040cc9b0 msgr2=0x7efd040ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:34.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.212+0000 7efd08ff4700 1 -- 172.21.15.1:0/1576419751 shutdown_connections 2024-03-18T19:02:34.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.212+0000 7efd08ff4700 1 -- 172.21.15.1:0/1576419751 wait complete. 2024-03-18T19:02:34.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.213+0000 7efd08ff4700 1 Processor -- start 2024-03-18T19:02:34.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.213+0000 7efd08ff4700 1 -- start start 2024-03-18T19:02:34.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.214+0000 7efd08ff4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd04071d30 0x7efd041f73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:34.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.214+0000 7efd08ff4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd040cc0f0 0x7efd0406d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:34.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.214+0000 7efd08ff4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd0415a1d0 0x7efd040c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:34.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.214+0000 7efd08ff4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0406db80 con 0x7efd040cc0f0 2024-03-18T19:02:34.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.214+0000 7efd08ff4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0406e980 con 0x7efd04071d30 2024-03-18T19:02:34.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.214+0000 7efd08ff4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd0406dd70 con 0x7efd0415a1d0 2024-03-18T19:02:34.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.214+0000 7efd01d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd040cc0f0 0x7efd0406d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:34.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.214+0000 7efd01d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd040cc0f0 0x7efd0406d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47302/0 (socket says 172.21.15.1:47302) 2024-03-18T19:02:34.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.214+0000 7efd01d9b700 1 -- 172.21.15.1:0/3440597568 learned_addr learned my addr 172.21.15.1:0/3440597568 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:34.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.214+0000 7efd0259c700 1 --2- 172.21.15.1:0/3440597568 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd04071d30 0x7efd041f73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:34.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.214+0000 7efd02d9d700 1 --2- 172.21.15.1:0/3440597568 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd0415a1d0 0x7efd040c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:34.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.214+0000 7efd01d9b700 1 -- 172.21.15.1:0/3440597568 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd0415a1d0 msgr2=0x7efd040c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:34.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.214+0000 7efd01d9b700 1 --2- 172.21.15.1:0/3440597568 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd0415a1d0 0x7efd040c94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:34.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.214+0000 7efd01d9b700 1 -- 172.21.15.1:0/3440597568 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd04071d30 msgr2=0x7efd041f73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:34.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.214+0000 7efd01d9b700 1 --2- 172.21.15.1:0/3440597568 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd04071d30 0x7efd041f73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:34.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.215+0000 7efd01d9b700 1 -- 172.21.15.1:0/3440597568 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc009b30 con 0x7efd040cc0f0 2024-03-18T19:02:34.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.215+0000 7efd01d9b700 1 --2- 172.21.15.1:0/3440597568 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd040cc0f0 0x7efd0406d640 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7efcf800c520 tx=0x7efcf800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:34.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.215+0000 7efcf2ffd700 1 -- 172.21.15.1:0/3440597568 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf8004c80 con 0x7efd040cc0f0 2024-03-18T19:02:34.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.215+0000 7efcf2ffd700 1 -- 172.21.15.1:0/3440597568 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf8004e00 con 0x7efd040cc0f0 2024-03-18T19:02:34.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.215+0000 7efd08ff4700 1 -- 172.21.15.1:0/3440597568 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd0406e050 con 0x7efd040cc0f0 2024-03-18T19:02:34.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.215+0000 7efcf2ffd700 1 -- 172.21.15.1:0/3440597568 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf8007310 con 0x7efd040cc0f0 2024-03-18T19:02:34.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.215+0000 7efd08ff4700 1 -- 172.21.15.1:0/3440597568 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd0406e610 con 0x7efd040cc0f0 2024-03-18T19:02:34.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.216+0000 7efd08ff4700 1 -- 172.21.15.1:0/3440597568 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd0415dd90 con 0x7efd040cc0f0 2024-03-18T19:02:34.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.216+0000 7efcf2ffd700 1 -- 172.21.15.1:0/3440597568 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7efcf8007470 con 0x7efd040cc0f0 2024-03-18T19:02:34.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.217+0000 7efcf2ffd700 1 --2- 172.21.15.1:0/3440597568 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efce406ae90 0x7efce406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:34.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.217+0000 7efd0259c700 1 --2- 172.21.15.1:0/3440597568 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efce406ae90 0x7efce406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:34.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.217+0000 7efcf2ffd700 1 -- 172.21.15.1:0/3440597568 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6980+0+0 (secure 0 0 0) 0x7efcf8089860 con 0x7efd040cc0f0 2024-03-18T19:02:34.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.217+0000 7efd0259c700 1 --2- 172.21.15.1:0/3440597568 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efce406ae90 0x7efce406d350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efcf4000fb0 tx=0x7efcf4009040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:34.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.222+0000 7efcf2ffd700 1 -- 172.21.15.1:0/3440597568 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efd0415dd90 con 0x7efd040cc0f0 2024-03-18T19:02:34.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.450+0000 7efd08ff4700 1 -- 172.21.15.1:0/3440597568 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7efd040ce580 con 0x7efce406ae90 2024-03-18T19:02:34.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.534+0000 7efcf2ffd700 1 -- 172.21.15.1:0/3440597568 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7efcf4001510 con 0x7efce406ae90 2024-03-18T19:02:34.530 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule removed for path /snap_test_dir1/dir4 2024-03-18T19:02:34.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.536+0000 7efd08ff4700 1 -- 172.21.15.1:0/3440597568 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efce406ae90 msgr2=0x7efce406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:34.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.536+0000 7efd08ff4700 1 --2- 172.21.15.1:0/3440597568 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efce406ae90 0x7efce406d350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efcf4000fb0 tx=0x7efcf4009040 comp rx=0 tx=0).stop 2024-03-18T19:02:34.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.536+0000 7efd08ff4700 1 -- 172.21.15.1:0/3440597568 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd040cc0f0 msgr2=0x7efd0406d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:34.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.536+0000 7efd08ff4700 1 --2- 172.21.15.1:0/3440597568 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd040cc0f0 0x7efd0406d640 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7efcf800c520 tx=0x7efcf800c830 comp rx=0 tx=0).stop 2024-03-18T19:02:34.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.536+0000 7efd08ff4700 1 -- 172.21.15.1:0/3440597568 shutdown_connections 2024-03-18T19:02:34.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.536+0000 7efd08ff4700 1 --2- 172.21.15.1:0/3440597568 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efce406ae90 0x7efce406d350 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:34.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.536+0000 7efd08ff4700 1 --2- 172.21.15.1:0/3440597568 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd0415a1d0 0x7efd040c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:34.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.536+0000 7efd08ff4700 1 --2- 172.21.15.1:0/3440597568 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd04071d30 0x7efd041f73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:34.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.536+0000 7efd08ff4700 1 --2- 172.21.15.1:0/3440597568 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd040cc0f0 0x7efd0406d640 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:34.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.536+0000 7efd08ff4700 1 -- 172.21.15.1:0/3440597568 >> 172.21.15.1:0/3440597568 conn(0x7efd040cc9b0 msgr2=0x7efd0406f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:34.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.536+0000 7efd08ff4700 1 -- 172.21.15.1:0/3440597568 shutdown_connections 2024-03-18T19:02:34.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:34.536+0000 7efd08ff4700 1 -- 172.21.15.1:0/3440597568 wait complete. 2024-03-18T19:02:34.544 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2024-03-18T19:02:34.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:34.544 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2024-03-18T19:02:34.575 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:34.575 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:34.575 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:34.575 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:34.575 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:34.575 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:34.575 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:34.575 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:34.575 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:34.575 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:34.576 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:34.576 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:34.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:34.576 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-18-18_52_00_UTC) 2024-03-18T19:02:34.642 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:34.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:34.643 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-18-18_53_00_UTC) 2024-03-18T19:02:34.708 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:34.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:34.709 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-18-18_54_00_UTC) 2024-03-18T19:02:34.774 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:34.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:34.774 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-18-18_55_00_UTC) 2024-03-18T19:02:34.842 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:34.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:34.843 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-18-18_56_00_UTC) 2024-03-18T19:02:34.908 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:34.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:34.909 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-18-18_57_00_UTC) 2024-03-18T19:02:34.974 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:34.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:34.974 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-18-18_58_00_UTC) 2024-03-18T19:02:35.040 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:35.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:35.040 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-18-18_59_00_UTC) 2024-03-18T19:02:35.134 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:35.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:35.134 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-18-19_00_00_UTC) 2024-03-18T19:02:35.162 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:35.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:35.162 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-18-19_01_00_UTC) 2024-03-18T19:02:35.228 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:35.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:35.229 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-03-18-19_02_00_UTC) 2024-03-18T19:02:35.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:35.309 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2024-03-18T19:02:35.371 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir5 1m 2024-03-18T19:02:35.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.544+0000 7f899f984700 1 Processor -- start 2024-03-18T19:02:35.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.544+0000 7f899f984700 1 -- start start 2024-03-18T19:02:35.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.545+0000 7f899f984700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8998071cc0 0x7f89980ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:35.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.545+0000 7f899f984700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89980cac80 con 0x7f899806d560 2024-03-18T19:02:35.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.545+0000 7f899f984700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89980caf60 con 0x7f899815a240 2024-03-18T19:02:35.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.545+0000 7f899f984700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89980adff0 con 0x7f8998071cc0 2024-03-18T19:02:35.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.545+0000 7f899df21700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f899806d560 0x7f89980ae410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56174/0 (socket says 172.21.15.1:56174) 2024-03-18T19:02:35.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.545+0000 7f899df21700 1 -- 172.21.15.1:0/2953823088 learned_addr learned my addr 172.21.15.1:0/2953823088 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:35.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.546+0000 7f899cf1f700 1 --2- 172.21.15.1:0/2953823088 >> v2:172.21.15.165:3300/0 conn(0x7f8998071cc0 0x7f89980ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:35.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.546+0000 7f898ffff700 1 -- 172.21.15.1:0/2953823088 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870820740 0 0) 0x7f89980cac80 con 0x7f899806d560 2024-03-18T19:02:35.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.546+0000 7f898ffff700 1 -- 172.21.15.1:0/2953823088 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89840035b0 con 0x7f899806d560 2024-03-18T19:02:35.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.546+0000 7f898ffff700 1 -- 172.21.15.1:0/2953823088 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 491148111 0 0) 0x7f89980caf60 con 0x7f899815a240 2024-03-18T19:02:35.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.546+0000 7f898ffff700 1 -- 172.21.15.1:0/2953823088 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89980cac80 con 0x7f899815a240 2024-03-18T19:02:35.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.546+0000 7f899cf1f700 1 -- 172.21.15.1:0/2953823088 >> v1:172.21.15.165:6789/0 conn(0x7f899815a240 legacy=0x7f89980715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:35.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.546+0000 7f899cf1f700 1 -- 172.21.15.1:0/2953823088 >> v1:172.21.15.1:6789/0 conn(0x7f899806d560 legacy=0x7f89980ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:35.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.546+0000 7f899cf1f700 1 -- 172.21.15.1:0/2953823088 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89980cb240 con 0x7f8998071cc0 2024-03-18T19:02:35.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.546+0000 7f899cf1f700 1 --2- 172.21.15.1:0/2953823088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8998071cc0 0x7f89980ca580 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8994009840 tx=0x7f8994009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d71ee5bd7e81ce6d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:35.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.546+0000 7f898ffff700 1 -- 172.21.15.1:0/2953823088 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8994011830 con 0x7f8998071cc0 2024-03-18T19:02:35.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.546+0000 7f898ffff700 1 -- 172.21.15.1:0/2953823088 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8994011990 con 0x7f8998071cc0 2024-03-18T19:02:35.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.546+0000 7f898ffff700 1 -- 172.21.15.1:0/2953823088 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8994011d20 con 0x7f8998071cc0 2024-03-18T19:02:35.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.547+0000 7f899f984700 1 -- 172.21.15.1:0/2953823088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8998071cc0 msgr2=0x7f89980ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:35.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.547+0000 7f899f984700 1 --2- 172.21.15.1:0/2953823088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8998071cc0 0x7f89980ca580 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8994009840 tx=0x7f8994009b50 comp rx=0 tx=0).stop 2024-03-18T19:02:35.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.547+0000 7f899f984700 1 -- 172.21.15.1:0/2953823088 shutdown_connections 2024-03-18T19:02:35.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.547+0000 7f899f984700 1 --2- 172.21.15.1:0/2953823088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8998071cc0 0x7f89980ca580 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:35.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.547+0000 7f899f984700 1 -- 172.21.15.1:0/2953823088 >> 172.21.15.1:0/2953823088 conn(0x7f89980cff30 msgr2=0x7f89980d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:35.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.547+0000 7f899f984700 1 -- 172.21.15.1:0/2953823088 shutdown_connections 2024-03-18T19:02:35.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.547+0000 7f899f984700 1 -- 172.21.15.1:0/2953823088 wait complete. 2024-03-18T19:02:35.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.548+0000 7f899f984700 1 Processor -- start 2024-03-18T19:02:35.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.548+0000 7f899f984700 1 -- start start 2024-03-18T19:02:35.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.549+0000 7f899f984700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f899806d560 0x7f89981575d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:35.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.549+0000 7f899f984700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8998071cc0 0x7f89980b3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:35.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.549+0000 7f899f984700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f899815a240 0x7f89980b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:35.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.549+0000 7f899f984700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89980b44a0 con 0x7f8998071cc0 2024-03-18T19:02:35.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.549+0000 7f899f984700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89980b46d0 con 0x7f899806d560 2024-03-18T19:02:35.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.549+0000 7f899f984700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89980b49b0 con 0x7f899815a240 2024-03-18T19:02:35.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.549+0000 7f899cf1f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8998071cc0 0x7f89980b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:35.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.549+0000 7f899cf1f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8998071cc0 0x7f89980b3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35264/0 (socket says 172.21.15.1:35264) 2024-03-18T19:02:35.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.549+0000 7f899cf1f700 1 -- 172.21.15.1:0/3969735246 learned_addr learned my addr 172.21.15.1:0/3969735246 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:35.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.549+0000 7f899df21700 1 --2- 172.21.15.1:0/3969735246 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f899815a240 0x7f89980b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:35.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.549+0000 7f899d720700 1 --2- 172.21.15.1:0/3969735246 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f899806d560 0x7f89981575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:35.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.549+0000 7f899cf1f700 1 -- 172.21.15.1:0/3969735246 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f899815a240 msgr2=0x7f89980b3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:35.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.549+0000 7f899cf1f700 1 --2- 172.21.15.1:0/3969735246 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f899815a240 0x7f89980b3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:35.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.549+0000 7f899cf1f700 1 -- 172.21.15.1:0/3969735246 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f899806d560 msgr2=0x7f89981575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:35.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.549+0000 7f899cf1f700 1 --2- 172.21.15.1:0/3969735246 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f899806d560 0x7f89981575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:35.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.549+0000 7f899cf1f700 1 -- 172.21.15.1:0/3969735246 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89940094a0 con 0x7f8998071cc0 2024-03-18T19:02:35.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.549+0000 7f899df21700 1 --2- 172.21.15.1:0/3969735246 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f899815a240 0x7f89980b3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:35.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.549+0000 7f899d720700 1 --2- 172.21.15.1:0/3969735246 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f899806d560 0x7f89981575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:35.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.549+0000 7f899cf1f700 1 --2- 172.21.15.1:0/3969735246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8998071cc0 0x7f89980b3730 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f89940090b0 tx=0x7f8994004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:35.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.550+0000 7f898dffb700 1 -- 172.21.15.1:0/3969735246 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899401a860 con 0x7f8998071cc0 2024-03-18T19:02:35.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.550+0000 7f898dffb700 1 -- 172.21.15.1:0/3969735246 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899401a9c0 con 0x7f8998071cc0 2024-03-18T19:02:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.550+0000 7f899f984700 1 -- 172.21.15.1:0/3969735246 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89980b4c90 con 0x7f8998071cc0 2024-03-18T19:02:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.550+0000 7f898dffb700 1 -- 172.21.15.1:0/3969735246 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899401ad30 con 0x7f8998071cc0 2024-03-18T19:02:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.550+0000 7f899f984700 1 -- 172.21.15.1:0/3969735246 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89980b5250 con 0x7f8998071cc0 2024-03-18T19:02:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.551+0000 7f899f984700 1 -- 172.21.15.1:0/3969735246 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8998067db0 con 0x7f8998071cc0 2024-03-18T19:02:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.551+0000 7f898dffb700 1 -- 172.21.15.1:0/3969735246 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f8994022070 con 0x7f8998071cc0 2024-03-18T19:02:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.552+0000 7f898dffb700 1 --2- 172.21.15.1:0/3969735246 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f897c06ae90 0x7f897c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.552+0000 7f898dffb700 1 -- 172.21.15.1:0/3969735246 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f899408acc0 con 0x7f8998071cc0 2024-03-18T19:02:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.552+0000 7f899d720700 1 --2- 172.21.15.1:0/3969735246 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f897c06ae90 0x7f897c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:35.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.552+0000 7f899d720700 1 --2- 172.21.15.1:0/3969735246 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f897c06ae90 0x7f897c06d350 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f89880011e0 tx=0x7f8988009040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:35.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.556+0000 7f898dffb700 1 -- 172.21.15.1:0/3969735246 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f89940558a0 con 0x7f8998071cc0 2024-03-18T19:02:35.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.784+0000 7f899f984700 1 -- 172.21.15.1:0/3969735246 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f899815a620 con 0x7f897c06ae90 2024-03-18T19:02:35.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.851+0000 7f898dffb700 1 -- 172.21.15.1:0/3969735246 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f8988001450 con 0x7f897c06ae90 2024-03-18T19:02:35.848 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule removed for path /snap_test_dir1/dir5 2024-03-18T19:02:35.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.854+0000 7f899f984700 1 -- 172.21.15.1:0/3969735246 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f897c06ae90 msgr2=0x7f897c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:35.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.854+0000 7f899f984700 1 --2- 172.21.15.1:0/3969735246 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f897c06ae90 0x7f897c06d350 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f89880011e0 tx=0x7f8988009040 comp rx=0 tx=0).stop 2024-03-18T19:02:35.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.854+0000 7f899f984700 1 -- 172.21.15.1:0/3969735246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8998071cc0 msgr2=0x7f89980b3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:35.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.854+0000 7f899f984700 1 --2- 172.21.15.1:0/3969735246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8998071cc0 0x7f89980b3730 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f89940090b0 tx=0x7f8994004750 comp rx=0 tx=0).stop 2024-03-18T19:02:35.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.854+0000 7f899f984700 1 -- 172.21.15.1:0/3969735246 shutdown_connections 2024-03-18T19:02:35.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.854+0000 7f899f984700 1 --2- 172.21.15.1:0/3969735246 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f897c06ae90 0x7f897c06d350 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:35.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.854+0000 7f899f984700 1 --2- 172.21.15.1:0/3969735246 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f899815a240 0x7f89980b3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:35.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.854+0000 7f899f984700 1 --2- 172.21.15.1:0/3969735246 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f899806d560 0x7f89981575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:35.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.854+0000 7f899f984700 1 --2- 172.21.15.1:0/3969735246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8998071cc0 0x7f89980b3730 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:35.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.854+0000 7f899f984700 1 -- 172.21.15.1:0/3969735246 >> 172.21.15.1:0/3969735246 conn(0x7f89980cff30 msgr2=0x7f899806e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:35.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.854+0000 7f899f984700 1 -- 172.21.15.1:0/3969735246 shutdown_connections 2024-03-18T19:02:35.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:35.854+0000 7f899f984700 1 -- 172.21.15.1:0/3969735246 wait complete. 2024-03-18T19:02:35.861 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2024-03-18T19:02:35.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:35.862 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2024-03-18T19:02:35.885 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:35.885 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:35.885 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:35.886 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:35.886 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:35.886 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:35.886 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:35.886 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:35.886 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:35.886 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:35.886 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:35.886 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:35.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:35.887 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-18-18_52_00_UTC) 2024-03-18T19:02:35.952 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:35.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:35.952 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-18-18_53_00_UTC) 2024-03-18T19:02:36.017 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:36.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:36.017 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-18-18_54_00_UTC) 2024-03-18T19:02:36.099 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:36.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:36.099 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-18-18_55_00_UTC) 2024-03-18T19:02:36.166 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:36.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:36.167 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-18-18_56_00_UTC) 2024-03-18T19:02:36.232 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:36.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:36.232 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-18-18_57_00_UTC) 2024-03-18T19:02:36.298 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:36.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:36.299 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-18-18_58_00_UTC) 2024-03-18T19:02:36.373 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:36.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:36.373 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-18-18_59_00_UTC) 2024-03-18T19:02:36.441 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:36.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:36.442 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-18-19_00_00_UTC) 2024-03-18T19:02:36.506 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:36.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:36.507 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-18-19_01_00_UTC) 2024-03-18T19:02:36.572 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:36.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:36.572 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-03-18-19_02_00_UTC) 2024-03-18T19:02:36.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:36.638 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2024-03-18T19:02:36.712 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir6 1m 2024-03-18T19:02:36.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.883+0000 7f5e39281700 1 Processor -- start 2024-03-18T19:02:36.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.884+0000 7f5e39281700 1 -- start start 2024-03-18T19:02:36.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.884+0000 7f5e39281700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e340cd870 0x7f5e340cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:36.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.884+0000 7f5e39281700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5e340cbce0 0x7f5e340cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:36.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.884+0000 7f5e39281700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e340d0df0 con 0x7f5e340b1e20 2024-03-18T19:02:36.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.884+0000 7f5e39281700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e340d0550 con 0x7f5e340cbce0 2024-03-18T19:02:36.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.884+0000 7f5e39281700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e340d0830 con 0x7f5e340cd870 2024-03-18T19:02:36.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.884+0000 7f5e327fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5e340b1e20 0x7f5e340cdc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56180/0 (socket says 172.21.15.1:56180) 2024-03-18T19:02:36.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.884+0000 7f5e327fc700 1 -- 172.21.15.1:0/1948104794 learned_addr learned my addr 172.21.15.1:0/1948104794 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:36.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e337fe700 1 --2- 172.21.15.1:0/1948104794 >> v2:172.21.15.165:3300/0 conn(0x7f5e340cbce0 0x7f5e340cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:36.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e32ffd700 1 --2- 172.21.15.1:0/1948104794 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e340cd870 0x7f5e340cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:36.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e31ffb700 1 -- 172.21.15.1:0/1948104794 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232505283 0 0) 0x7f5e340d0df0 con 0x7f5e340b1e20 2024-03-18T19:02:36.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e31ffb700 1 -- 172.21.15.1:0/1948104794 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e1c0035b0 con 0x7f5e340b1e20 2024-03-18T19:02:36.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e31ffb700 1 -- 172.21.15.1:0/1948104794 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2164106239 0 0) 0x7f5e1c0035b0 con 0x7f5e340b1e20 2024-03-18T19:02:36.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e31ffb700 1 -- 172.21.15.1:0/1948104794 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e340cd870 msgr2=0x7f5e340cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:36.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e31ffb700 1 --2- 172.21.15.1:0/1948104794 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e340cd870 0x7f5e340cff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:36.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e31ffb700 1 -- 172.21.15.1:0/1948104794 >> v2:172.21.15.165:3300/0 conn(0x7f5e340cbce0 msgr2=0x7f5e340cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:36.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e31ffb700 1 --2- 172.21.15.1:0/1948104794 >> v2:172.21.15.165:3300/0 conn(0x7f5e340cbce0 0x7f5e340cc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:36.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e31ffb700 1 -- 172.21.15.1:0/1948104794 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e340d0b10 con 0x7f5e340b1e20 2024-03-18T19:02:36.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e31ffb700 1 -- 172.21.15.1:0/1948104794 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f5e24001670 con 0x7f5e340b1e20 2024-03-18T19:02:36.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e337fe700 1 --2- 172.21.15.1:0/1948104794 >> v2:172.21.15.165:3300/0 conn(0x7f5e340cbce0 0x7f5e340cc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:02:36.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e31ffb700 1 -- 172.21.15.1:0/1948104794 >> v1:172.21.15.1:6789/0 conn(0x7f5e340b1e20 legacy=0x7f5e340cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:36.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e31ffb700 1 --2- 172.21.15.1:0/1948104794 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e340cd870 0x7f5e1c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:36.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e31ffb700 1 --2- 172.21.15.1:0/1948104794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e1c003f80 0x7f5e1c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:36.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e31ffb700 1 --2- 172.21.15.1:0/1948104794 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e1c004d90 0x7f5e1c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:36.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e31ffb700 1 -- 172.21.15.1:0/1948104794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e340d0550 con 0x7f5e1c003f80 2024-03-18T19:02:36.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e31ffb700 1 -- 172.21.15.1:0/1948104794 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e340d0830 con 0x7f5e1c004d90 2024-03-18T19:02:36.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e31ffb700 1 -- 172.21.15.1:0/1948104794 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c00ba10 con 0x7f5e340cd870 2024-03-18T19:02:36.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e327fc700 1 --2- 172.21.15.1:0/1948104794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e1c003f80 0x7f5e1c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:36.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e32ffd700 1 --2- 172.21.15.1:0/1948104794 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e340cd870 0x7f5e1c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:36.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.885+0000 7f5e337fe700 1 --2- 172.21.15.1:0/1948104794 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e1c004d90 0x7f5e1c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:36.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.886+0000 7f5e32ffd700 1 -- 172.21.15.1:0/1948104794 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e1c004d90 msgr2=0x7f5e1c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:36.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.886+0000 7f5e32ffd700 1 --2- 172.21.15.1:0/1948104794 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e1c004d90 0x7f5e1c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:36.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.886+0000 7f5e32ffd700 1 -- 172.21.15.1:0/1948104794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e1c003f80 msgr2=0x7f5e1c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:36.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.886+0000 7f5e32ffd700 1 --2- 172.21.15.1:0/1948104794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e1c003f80 0x7f5e1c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:36.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.886+0000 7f5e32ffd700 1 -- 172.21.15.1:0/1948104794 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c0035b0 con 0x7f5e340cd870 2024-03-18T19:02:36.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.886+0000 7f5e327fc700 1 --2- 172.21.15.1:0/1948104794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e1c003f80 0x7f5e1c004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:02:36.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.886+0000 7f5e32ffd700 1 --2- 172.21.15.1:0/1948104794 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e340cd870 0x7f5e1c003a00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009b60 tx=0x7f5e1c005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31c68435790a4e52 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:36.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.886+0000 7f5e31ffb700 1 -- 172.21.15.1:0/1948104794 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2800e6c0 con 0x7f5e340cd870 2024-03-18T19:02:36.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.886+0000 7f5e31ffb700 1 -- 172.21.15.1:0/1948104794 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2800e820 con 0x7f5e340cd870 2024-03-18T19:02:36.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.886+0000 7f5e31ffb700 1 -- 172.21.15.1:0/1948104794 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2800eb90 con 0x7f5e340cd870 2024-03-18T19:02:36.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.886+0000 7f5e39281700 1 -- 172.21.15.1:0/1948104794 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e340cd870 msgr2=0x7f5e1c003a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:36.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.886+0000 7f5e39281700 1 --2- 172.21.15.1:0/1948104794 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e340cd870 0x7f5e1c003a00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009b60 tx=0x7f5e1c005880 comp rx=0 tx=0).stop 2024-03-18T19:02:36.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.887+0000 7f5e337fe700 1 -- 172.21.15.1:0/1948104794 reap_dead start 2024-03-18T19:02:36.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.887+0000 7f5e39281700 1 -- 172.21.15.1:0/1948104794 shutdown_connections 2024-03-18T19:02:36.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.887+0000 7f5e39281700 1 -- 172.21.15.1:0/1948104794 >> 172.21.15.1:0/1948104794 conn(0x7f5e340d0ff0 msgr2=0x7f5e340d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:36.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.887+0000 7f5e39281700 1 -- 172.21.15.1:0/1948104794 shutdown_connections 2024-03-18T19:02:36.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.887+0000 7f5e39281700 1 -- 172.21.15.1:0/1948104794 wait complete. 2024-03-18T19:02:36.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.888+0000 7f5e39281700 1 Processor -- start 2024-03-18T19:02:36.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.888+0000 7f5e39281700 1 -- start start 2024-03-18T19:02:36.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e39281700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e340cd870 0x7f5e34156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:36.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e39281700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e341ed4d0 0x7f5e341f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:36.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e39281700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e34158630 0x7f5e340c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:36.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e39281700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e34157d50 con 0x7f5e341ed4d0 2024-03-18T19:02:36.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e39281700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e341f44c0 con 0x7f5e34158630 2024-03-18T19:02:36.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e39281700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e340c92b0 con 0x7f5e340cd870 2024-03-18T19:02:36.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e327fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e341ed4d0 0x7f5e341f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:36.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e327fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e341ed4d0 0x7f5e341f3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35288/0 (socket says 172.21.15.1:35288) 2024-03-18T19:02:36.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e327fc700 1 -- 172.21.15.1:0/2209434667 learned_addr learned my addr 172.21.15.1:0/2209434667 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:36.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e32ffd700 1 --2- 172.21.15.1:0/2209434667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e340cd870 0x7f5e34156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:36.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e337fe700 1 --2- 172.21.15.1:0/2209434667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e34158630 0x7f5e340c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:36.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e327fc700 1 -- 172.21.15.1:0/2209434667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e340cd870 msgr2=0x7f5e34156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:36.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e327fc700 1 --2- 172.21.15.1:0/2209434667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e340cd870 0x7f5e34156db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:36.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e327fc700 1 -- 172.21.15.1:0/2209434667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e34158630 msgr2=0x7f5e340c9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:36.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e327fc700 1 --2- 172.21.15.1:0/2209434667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e34158630 0x7f5e340c9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:36.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e327fc700 1 -- 172.21.15.1:0/2209434667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e341ed950 con 0x7f5e341ed4d0 2024-03-18T19:02:36.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e327fc700 1 --2- 172.21.15.1:0/2209434667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e341ed4d0 0x7f5e341f3f80 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f5e24001cc0 tx=0x7f5e24007660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:36.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e1b7fe700 1 -- 172.21.15.1:0/2209434667 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e240077f0 con 0x7f5e341ed4d0 2024-03-18T19:02:36.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e1b7fe700 1 -- 172.21.15.1:0/2209434667 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2400f8d0 con 0x7f5e341ed4d0 2024-03-18T19:02:36.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e39281700 1 -- 172.21.15.1:0/2209434667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e341574b0 con 0x7f5e341ed4d0 2024-03-18T19:02:36.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.889+0000 7f5e1b7fe700 1 -- 172.21.15.1:0/2209434667 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2400fc60 con 0x7f5e341ed4d0 2024-03-18T19:02:36.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.890+0000 7f5e39281700 1 -- 172.21.15.1:0/2209434667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e34157a70 con 0x7f5e341ed4d0 2024-03-18T19:02:36.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.890+0000 7f5e39281700 1 -- 172.21.15.1:0/2209434667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e340c9110 con 0x7f5e341ed4d0 2024-03-18T19:02:36.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.891+0000 7f5e1b7fe700 1 -- 172.21.15.1:0/2209434667 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f5e2401d460 con 0x7f5e341ed4d0 2024-03-18T19:02:36.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.891+0000 7f5e1b7fe700 1 --2- 172.21.15.1:0/2209434667 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e2006ae90 0x7f5e2006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:36.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.891+0000 7f5e1b7fe700 1 -- 172.21.15.1:0/2209434667 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7044+0+0 (secure 0 0 0) 0x7f5e24089c90 con 0x7f5e341ed4d0 2024-03-18T19:02:36.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.891+0000 7f5e32ffd700 1 --2- 172.21.15.1:0/2209434667 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e2006ae90 0x7f5e2006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:36.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.892+0000 7f5e32ffd700 1 --2- 172.21.15.1:0/2209434667 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e2006ae90 0x7f5e2006d350 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5e280090b0 tx=0x7f5e2801f040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:36.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:36.896+0000 7f5e1b7fe700 1 -- 172.21.15.1:0/2209434667 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5e340c9110 con 0x7f5e341ed4d0 2024-03-18T19:02:37.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:37.135+0000 7f5e39281700 1 -- 172.21.15.1:0/2209434667 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e340b25a0 con 0x7f5e2006ae90 2024-03-18T19:02:37.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:37.219+0000 7f5e1b7fe700 1 -- 172.21.15.1:0/2209434667 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f5e2800ea90 con 0x7f5e2006ae90 2024-03-18T19:02:37.216 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule removed for path /snap_test_dir1/dir6 2024-03-18T19:02:37.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:37.221+0000 7f5e39281700 1 -- 172.21.15.1:0/2209434667 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e2006ae90 msgr2=0x7f5e2006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:37.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:37.221+0000 7f5e39281700 1 --2- 172.21.15.1:0/2209434667 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e2006ae90 0x7f5e2006d350 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5e280090b0 tx=0x7f5e2801f040 comp rx=0 tx=0).stop 2024-03-18T19:02:37.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:37.221+0000 7f5e39281700 1 -- 172.21.15.1:0/2209434667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e341ed4d0 msgr2=0x7f5e341f3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:37.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:37.221+0000 7f5e39281700 1 --2- 172.21.15.1:0/2209434667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e341ed4d0 0x7f5e341f3f80 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f5e24001cc0 tx=0x7f5e24007660 comp rx=0 tx=0).stop 2024-03-18T19:02:37.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:37.221+0000 7f5e39281700 1 -- 172.21.15.1:0/2209434667 shutdown_connections 2024-03-18T19:02:37.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:37.221+0000 7f5e39281700 1 --2- 172.21.15.1:0/2209434667 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e2006ae90 0x7f5e2006d350 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:37.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:37.221+0000 7f5e39281700 1 --2- 172.21.15.1:0/2209434667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e340cd870 0x7f5e34156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:37.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:37.221+0000 7f5e39281700 1 --2- 172.21.15.1:0/2209434667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e34158630 0x7f5e340c9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:37.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:37.221+0000 7f5e39281700 1 --2- 172.21.15.1:0/2209434667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e341ed4d0 0x7f5e341f3f80 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:37.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:37.221+0000 7f5e39281700 1 -- 172.21.15.1:0/2209434667 >> 172.21.15.1:0/2209434667 conn(0x7f5e340d0ff0 msgr2=0x7f5e34156140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:37.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:37.222+0000 7f5e39281700 1 -- 172.21.15.1:0/2209434667 shutdown_connections 2024-03-18T19:02:37.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:37.222+0000 7f5e39281700 1 -- 172.21.15.1:0/2209434667 wait complete. 2024-03-18T19:02:37.229 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2024-03-18T19:02:37.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:37.229 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2024-03-18T19:02:37.260 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:37.260 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:37.260 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:37.260 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:37.260 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:37.260 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:37.260 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:37.260 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:37.260 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:37.260 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:37.261 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:37.261 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:37.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:37.262 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-18-18_52_00_UTC) 2024-03-18T19:02:37.329 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:37.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:37.329 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-18-18_53_00_UTC) 2024-03-18T19:02:37.395 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:37.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:37.395 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-18-18_54_00_UTC) 2024-03-18T19:02:37.460 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:37.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:37.460 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-18-18_55_00_UTC) 2024-03-18T19:02:37.524 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:37.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:37.525 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-18-18_56_00_UTC) 2024-03-18T19:02:37.590 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:37.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:37.590 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-18-18_57_00_UTC) 2024-03-18T19:02:37.654 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:37.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:37.655 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-18-18_58_00_UTC) 2024-03-18T19:02:37.719 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:37.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:37.720 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-18-18_59_00_UTC) 2024-03-18T19:02:37.784 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:37.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:37.785 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-18-19_00_00_UTC) 2024-03-18T19:02:37.849 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:37.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:37.850 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-18-19_01_00_UTC) 2024-03-18T19:02:37.918 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:37.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:37.919 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-03-18-19_02_00_UTC) 2024-03-18T19:02:37.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:37.984 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2024-03-18T19:02:38.047 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir7 1m 2024-03-18T19:02:38.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.226+0000 7fe32950b700 1 Processor -- start 2024-03-18T19:02:38.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.226+0000 7fe32950b700 1 -- start start 2024-03-18T19:02:38.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.227+0000 7fe32950b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3240cfb50 0x7fe3240d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:38.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.227+0000 7fe32950b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe3240cff30 0x7fe3240b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:38.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.227+0000 7fe32950b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3240b3370 con 0x7fe3240b3d40 2024-03-18T19:02:38.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.227+0000 7fe32950b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3240b2760 con 0x7fe3240cff30 2024-03-18T19:02:38.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.227+0000 7fe32950b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3240b2a40 con 0x7fe3240cfb50 2024-03-18T19:02:38.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.227+0000 7fe3237fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe3240b3d40 0x7fe3240b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56182/0 (socket says 172.21.15.1:56182) 2024-03-18T19:02:38.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.227+0000 7fe3237fe700 1 -- 172.21.15.1:0/1389524581 learned_addr learned my addr 172.21.15.1:0/1389524581 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:38.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.227+0000 7fe322ffd700 1 --2- 172.21.15.1:0/1389524581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3240cfb50 0x7fe3240d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:38.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.227+0000 7fe3227fc700 1 --2- 172.21.15.1:0/1389524581 >> v2:172.21.15.165:3300/0 conn(0x7fe3240cff30 0x7fe3240b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:38.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.227+0000 7fe321ffb700 1 -- 172.21.15.1:0/1389524581 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848126553 0 0) 0x7fe3240b3370 con 0x7fe3240b3d40 2024-03-18T19:02:38.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.227+0000 7fe321ffb700 1 -- 172.21.15.1:0/1389524581 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe30c0035b0 con 0x7fe3240b3d40 2024-03-18T19:02:38.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.227+0000 7fe3227fc700 1 -- 172.21.15.1:0/1389524581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3240cfb50 msgr2=0x7fe3240d3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:38.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.227+0000 7fe3227fc700 1 --2- 172.21.15.1:0/1389524581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3240cfb50 0x7fe3240d3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:38.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.227+0000 7fe3227fc700 1 -- 172.21.15.1:0/1389524581 >> v1:172.21.15.1:6789/0 conn(0x7fe3240b3d40 legacy=0x7fe3240b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:38.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.227+0000 7fe3227fc700 1 -- 172.21.15.1:0/1389524581 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3240b2d20 con 0x7fe3240cff30 2024-03-18T19:02:38.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.227+0000 7fe322ffd700 1 --2- 172.21.15.1:0/1389524581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3240cfb50 0x7fe3240d3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:02:38.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.228+0000 7fe3227fc700 1 --2- 172.21.15.1:0/1389524581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3240cff30 0x7fe3240b3800 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe314009840 tx=0x7fe314009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76154df80910330e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:38.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.228+0000 7fe321ffb700 1 -- 172.21.15.1:0/1389524581 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314011830 con 0x7fe3240cff30 2024-03-18T19:02:38.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.228+0000 7fe321ffb700 1 -- 172.21.15.1:0/1389524581 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe314011990 con 0x7fe3240cff30 2024-03-18T19:02:38.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.228+0000 7fe321ffb700 1 -- 172.21.15.1:0/1389524581 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314011d20 con 0x7fe3240cff30 2024-03-18T19:02:38.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.228+0000 7fe32950b700 1 -- 172.21.15.1:0/1389524581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3240cff30 msgr2=0x7fe3240b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:38.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.228+0000 7fe32950b700 1 --2- 172.21.15.1:0/1389524581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3240cff30 0x7fe3240b3800 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe314009840 tx=0x7fe314009b50 comp rx=0 tx=0).stop 2024-03-18T19:02:38.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.228+0000 7fe32950b700 1 -- 172.21.15.1:0/1389524581 shutdown_connections 2024-03-18T19:02:38.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.228+0000 7fe32950b700 1 --2- 172.21.15.1:0/1389524581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3240cfb50 0x7fe3240d3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:38.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.228+0000 7fe32950b700 1 --2- 172.21.15.1:0/1389524581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3240cff30 0x7fe3240b3800 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:38.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.228+0000 7fe32950b700 1 -- 172.21.15.1:0/1389524581 >> 172.21.15.1:0/1389524581 conn(0x7fe3240d3140 msgr2=0x7fe3240d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:38.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.228+0000 7fe32950b700 1 -- 172.21.15.1:0/1389524581 shutdown_connections 2024-03-18T19:02:38.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.228+0000 7fe32950b700 1 -- 172.21.15.1:0/1389524581 wait complete. 2024-03-18T19:02:38.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.230+0000 7fe32950b700 1 Processor -- start 2024-03-18T19:02:38.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.230+0000 7fe32950b700 1 -- start start 2024-03-18T19:02:38.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.230+0000 7fe32950b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3240b3d40 0x7fe3240cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:38.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.230+0000 7fe32950b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3240cfb50 0x7fe32414d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:38.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.230+0000 7fe32950b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3240cff30 0x7fe32415dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:38.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.230+0000 7fe32950b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3240ca540 con 0x7fe3240cfb50 2024-03-18T19:02:38.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.230+0000 7fe32950b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3240ca820 con 0x7fe3240b3d40 2024-03-18T19:02:38.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.230+0000 7fe32950b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3240cab00 con 0x7fe3240cff30 2024-03-18T19:02:38.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.230+0000 7fe3227fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3240cfb50 0x7fe32414d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:38.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.230+0000 7fe3227fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3240cfb50 0x7fe32414d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35302/0 (socket says 172.21.15.1:35302) 2024-03-18T19:02:38.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.231+0000 7fe3227fc700 1 -- 172.21.15.1:0/3130188442 learned_addr learned my addr 172.21.15.1:0/3130188442 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:38.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.231+0000 7fe322ffd700 1 --2- 172.21.15.1:0/3130188442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3240b3d40 0x7fe3240cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:38.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.231+0000 7fe3237fe700 1 --2- 172.21.15.1:0/3130188442 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3240cff30 0x7fe32415dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:38.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.231+0000 7fe3227fc700 1 -- 172.21.15.1:0/3130188442 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3240cff30 msgr2=0x7fe32415dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:38.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.231+0000 7fe3227fc700 1 --2- 172.21.15.1:0/3130188442 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3240cff30 0x7fe32415dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:38.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.231+0000 7fe3227fc700 1 -- 172.21.15.1:0/3130188442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3240b3d40 msgr2=0x7fe3240cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:38.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.231+0000 7fe3227fc700 1 --2- 172.21.15.1:0/3130188442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3240b3d40 0x7fe3240cc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:38.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.231+0000 7fe3227fc700 1 -- 172.21.15.1:0/3130188442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe318009430 con 0x7fe3240cfb50 2024-03-18T19:02:38.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.231+0000 7fe3237fe700 1 --2- 172.21.15.1:0/3130188442 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3240cff30 0x7fe32415dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:02:38.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.231+0000 7fe3227fc700 1 --2- 172.21.15.1:0/3130188442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3240cfb50 0x7fe32414d580 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe3140090b0 tx=0x7fe314004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:38.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.231+0000 7fe30b7fe700 1 -- 172.21.15.1:0/3130188442 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3140036a0 con 0x7fe3240cfb50 2024-03-18T19:02:38.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.231+0000 7fe30b7fe700 1 -- 172.21.15.1:0/3130188442 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31401e070 con 0x7fe3240cfb50 2024-03-18T19:02:38.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.231+0000 7fe32950b700 1 -- 172.21.15.1:0/3130188442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3140094a0 con 0x7fe3240cfb50 2024-03-18T19:02:38.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.231+0000 7fe30b7fe700 1 -- 172.21.15.1:0/3130188442 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314003a60 con 0x7fe3240cfb50 2024-03-18T19:02:38.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.231+0000 7fe32950b700 1 -- 172.21.15.1:0/3130188442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3240cb0c0 con 0x7fe3240cfb50 2024-03-18T19:02:38.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.232+0000 7fe32950b700 1 -- 172.21.15.1:0/3130188442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3240cb2b0 con 0x7fe3240cfb50 2024-03-18T19:02:38.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.233+0000 7fe30b7fe700 1 -- 172.21.15.1:0/3130188442 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fe314003bc0 con 0x7fe3240cfb50 2024-03-18T19:02:38.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.233+0000 7fe30b7fe700 1 --2- 172.21.15.1:0/3130188442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe31006ae90 0x7fe31006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:38.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.233+0000 7fe30b7fe700 1 -- 172.21.15.1:0/3130188442 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6836+0+0 (secure 0 0 0) 0x7fe31408aa00 con 0x7fe3240cfb50 2024-03-18T19:02:38.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.233+0000 7fe322ffd700 1 --2- 172.21.15.1:0/3130188442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe31006ae90 0x7fe31006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:38.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.233+0000 7fe322ffd700 1 --2- 172.21.15.1:0/3130188442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe31006ae90 0x7fe31006d350 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe318009ce0 tx=0x7fe318008040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:38.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.238+0000 7fe30b7fe700 1 -- 172.21.15.1:0/3130188442 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe314055580 con 0x7fe3240cfb50 2024-03-18T19:02:38.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.464+0000 7fe32950b700 1 -- 172.21.15.1:0/3130188442 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3240d03b0 con 0x7fe31006ae90 2024-03-18T19:02:38.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.578+0000 7fe30b7fe700 1 -- 172.21.15.1:0/3130188442 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fe318009930 con 0x7fe31006ae90 2024-03-18T19:02:38.574 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule removed for path /snap_test_dir1/dir7 2024-03-18T19:02:38.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.581+0000 7fe32950b700 1 -- 172.21.15.1:0/3130188442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe31006ae90 msgr2=0x7fe31006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:38.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.581+0000 7fe32950b700 1 --2- 172.21.15.1:0/3130188442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe31006ae90 0x7fe31006d350 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe318009ce0 tx=0x7fe318008040 comp rx=0 tx=0).stop 2024-03-18T19:02:38.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.581+0000 7fe32950b700 1 -- 172.21.15.1:0/3130188442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3240cfb50 msgr2=0x7fe32414d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:38.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.581+0000 7fe32950b700 1 --2- 172.21.15.1:0/3130188442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3240cfb50 0x7fe32414d580 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe3140090b0 tx=0x7fe314004750 comp rx=0 tx=0).stop 2024-03-18T19:02:38.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.581+0000 7fe32950b700 1 -- 172.21.15.1:0/3130188442 shutdown_connections 2024-03-18T19:02:38.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.581+0000 7fe32950b700 1 --2- 172.21.15.1:0/3130188442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe31006ae90 0x7fe31006d350 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:38.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.581+0000 7fe32950b700 1 --2- 172.21.15.1:0/3130188442 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3240cff30 0x7fe32415dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:38.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.581+0000 7fe32950b700 1 --2- 172.21.15.1:0/3130188442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3240b3d40 0x7fe3240cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:38.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.581+0000 7fe32950b700 1 --2- 172.21.15.1:0/3130188442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3240cfb50 0x7fe32414d580 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:38.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.581+0000 7fe32950b700 1 -- 172.21.15.1:0/3130188442 >> 172.21.15.1:0/3130188442 conn(0x7fe3240d3140 msgr2=0x7fe3240d1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:38.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.581+0000 7fe32950b700 1 -- 172.21.15.1:0/3130188442 shutdown_connections 2024-03-18T19:02:38.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:38.581+0000 7fe32950b700 1 -- 172.21.15.1:0/3130188442 wait complete. 2024-03-18T19:02:38.588 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2024-03-18T19:02:38.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:38.589 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2024-03-18T19:02:38.618 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:38.618 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:38.618 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:38.619 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:38.619 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:38.619 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:38.619 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:38.619 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:38.619 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:38.619 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:38.619 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:38.620 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:38.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:38.620 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-18-18_52_00_UTC) 2024-03-18T19:02:38.686 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:38.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:38.686 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-18-18_53_00_UTC) 2024-03-18T19:02:38.751 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:38.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:38.752 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-18-18_54_00_UTC) 2024-03-18T19:02:38.817 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:38.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:38.818 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-18-18_55_00_UTC) 2024-03-18T19:02:38.883 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:38.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:38.883 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-18-18_56_00_UTC) 2024-03-18T19:02:38.958 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:38.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:38.959 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-18-18_57_00_UTC) 2024-03-18T19:02:39.040 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:39.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:39.040 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-18-18_58_00_UTC) 2024-03-18T19:02:39.105 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:39.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:39.105 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-18-18_59_00_UTC) 2024-03-18T19:02:39.179 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:39.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:39.180 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-18-19_00_00_UTC) 2024-03-18T19:02:39.247 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:39.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:39.248 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-18-19_01_00_UTC) 2024-03-18T19:02:39.315 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:39.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:39.315 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-03-18-19_02_00_UTC) 2024-03-18T19:02:39.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:39.381 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2024-03-18T19:02:39.442 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir8 1m 2024-03-18T19:02:39.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.614+0000 7f6575d60700 1 Processor -- start 2024-03-18T19:02:39.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.615+0000 7f6575d60700 1 -- start start 2024-03-18T19:02:39.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.615+0000 7f6575d60700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65700ccfe0 0x7f657015d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:39.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.615+0000 7f6575d60700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65700b3b00 con 0x7f657006ca80 2024-03-18T19:02:39.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.615+0000 7f6575d60700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65700cf7f0 con 0x7f657015d5b0 2024-03-18T19:02:39.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.615+0000 7f6575d60700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65700cbbb0 con 0x7f65700ccfe0 2024-03-18T19:02:39.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.615+0000 7f656effd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f657006ca80 0x7f657009a150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56190/0 (socket says 172.21.15.1:56190) 2024-03-18T19:02:39.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656effd700 1 -- 172.21.15.1:0/1611379639 learned_addr learned my addr 172.21.15.1:0/1611379639 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:39.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656ffff700 1 --2- 172.21.15.1:0/1611379639 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65700ccfe0 0x7f657015d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:39.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656e7fc700 1 -- 172.21.15.1:0/1611379639 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370721546 0 0) 0x7f65700b3b00 con 0x7f657006ca80 2024-03-18T19:02:39.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656e7fc700 1 -- 172.21.15.1:0/1611379639 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65580035b0 con 0x7f657006ca80 2024-03-18T19:02:39.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656e7fc700 1 -- 172.21.15.1:0/1611379639 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2753110908 0 0) 0x7f65580035b0 con 0x7f657006ca80 2024-03-18T19:02:39.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656e7fc700 1 -- 172.21.15.1:0/1611379639 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65700ccfe0 msgr2=0x7f657015d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:39.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656e7fc700 1 --2- 172.21.15.1:0/1611379639 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65700ccfe0 0x7f657015d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:39.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656e7fc700 1 -- 172.21.15.1:0/1611379639 >> v1:172.21.15.165:6789/0 conn(0x7f657015d5b0 legacy=0x7f6570099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:39.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656e7fc700 1 -- 172.21.15.1:0/1611379639 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65700d04d0 con 0x7f657006ca80 2024-03-18T19:02:39.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656e7fc700 1 -- 172.21.15.1:0/1611379639 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f6564001670 con 0x7f657006ca80 2024-03-18T19:02:39.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656e7fc700 1 -- 172.21.15.1:0/1611379639 >> v1:172.21.15.1:6789/0 conn(0x7f657006ca80 legacy=0x7f657009a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:39.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656e7fc700 1 --2- 172.21.15.1:0/1611379639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6558003a30 0x7f6558003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:39.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656e7fc700 1 --2- 172.21.15.1:0/1611379639 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65580043e0 0x7f65580048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:39.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656e7fc700 1 --2- 172.21.15.1:0/1611379639 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65700ccfe0 0x7f65580051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:39.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656e7fc700 1 -- 172.21.15.1:0/1611379639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65700cbbb0 con 0x7f6558003a30 2024-03-18T19:02:39.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656e7fc700 1 -- 172.21.15.1:0/1611379639 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655800bac0 con 0x7f65580043e0 2024-03-18T19:02:39.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656e7fc700 1 -- 172.21.15.1:0/1611379639 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f655800bc80 con 0x7f65700ccfe0 2024-03-18T19:02:39.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656f7fe700 1 --2- 172.21.15.1:0/1611379639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6558003a30 0x7f6558003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:39.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656effd700 1 --2- 172.21.15.1:0/1611379639 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65580043e0 0x7f65580048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:39.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.616+0000 7f656ffff700 1 --2- 172.21.15.1:0/1611379639 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65700ccfe0 0x7f65580051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:39.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.617+0000 7f656f7fe700 1 -- 172.21.15.1:0/1611379639 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65700ccfe0 msgr2=0x7f65580051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:39.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.617+0000 7f656f7fe700 1 --2- 172.21.15.1:0/1611379639 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65700ccfe0 0x7f65580051b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:39.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.617+0000 7f656f7fe700 1 -- 172.21.15.1:0/1611379639 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65580043e0 msgr2=0x7f65580048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:39.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.617+0000 7f656f7fe700 1 --2- 172.21.15.1:0/1611379639 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65580043e0 0x7f65580048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:39.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.617+0000 7f656f7fe700 1 -- 172.21.15.1:0/1611379639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65580035b0 con 0x7f6558003a30 2024-03-18T19:02:39.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.617+0000 7f656f7fe700 1 --2- 172.21.15.1:0/1611379639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6558003a30 0x7f6558003e10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f65600018d0 tx=0x7f6558005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40c98ad15c4c5c77 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:39.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.617+0000 7f656e7fc700 1 -- 172.21.15.1:0/1611379639 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656000c690 con 0x7f6558003a30 2024-03-18T19:02:39.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.617+0000 7f656e7fc700 1 -- 172.21.15.1:0/1611379639 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656000c810 con 0x7f6558003a30 2024-03-18T19:02:39.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.617+0000 7f656e7fc700 1 -- 172.21.15.1:0/1611379639 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656000cbc0 con 0x7f6558003a30 2024-03-18T19:02:39.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.617+0000 7f6575d60700 1 -- 172.21.15.1:0/1611379639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6558003a30 msgr2=0x7f6558003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:39.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.617+0000 7f6575d60700 1 --2- 172.21.15.1:0/1611379639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6558003a30 0x7f6558003e10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f65600018d0 tx=0x7f6558005880 comp rx=0 tx=0).stop 2024-03-18T19:02:39.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.617+0000 7f656ffff700 1 -- 172.21.15.1:0/1611379639 reap_dead start 2024-03-18T19:02:39.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.617+0000 7f6575d60700 1 -- 172.21.15.1:0/1611379639 shutdown_connections 2024-03-18T19:02:39.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.617+0000 7f6575d60700 1 -- 172.21.15.1:0/1611379639 >> 172.21.15.1:0/1611379639 conn(0x7f65700b2bd0 msgr2=0x7f65700b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:39.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.617+0000 7f6575d60700 1 -- 172.21.15.1:0/1611379639 shutdown_connections 2024-03-18T19:02:39.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.617+0000 7f6575d60700 1 -- 172.21.15.1:0/1611379639 wait complete. 2024-03-18T19:02:39.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.619+0000 7f6575d60700 1 Processor -- start 2024-03-18T19:02:39.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.619+0000 7f6575d60700 1 -- start start 2024-03-18T19:02:39.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.619+0000 7f6575d60700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6558003a30 0x7f65700c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:39.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.619+0000 7f6575d60700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6570148660 0x7f65700c7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:39.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f6575d60700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65700c7640 0x7f6570145ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:39.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f6575d60700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65700c7af0 con 0x7f65700c7640 2024-03-18T19:02:39.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f6575d60700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65700c5110 con 0x7f6570148660 2024-03-18T19:02:39.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f6575d60700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6570146c40 con 0x7f6558003a30 2024-03-18T19:02:39.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f656ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65700c7640 0x7f6570145ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:39.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f656ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65700c7640 0x7f6570145ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35324/0 (socket says 172.21.15.1:35324) 2024-03-18T19:02:39.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f656ffff700 1 -- 172.21.15.1:0/3028648814 learned_addr learned my addr 172.21.15.1:0/3028648814 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:39.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f656effd700 1 --2- 172.21.15.1:0/3028648814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6570148660 0x7f65700c7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:39.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f656f7fe700 1 --2- 172.21.15.1:0/3028648814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6558003a30 0x7f65700c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:39.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f656ffff700 1 -- 172.21.15.1:0/3028648814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6558003a30 msgr2=0x7f65700c4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:39.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f656ffff700 1 --2- 172.21.15.1:0/3028648814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6558003a30 0x7f65700c4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:39.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f656ffff700 1 -- 172.21.15.1:0/3028648814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6570148660 msgr2=0x7f65700c7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:39.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f656ffff700 1 --2- 172.21.15.1:0/3028648814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6570148660 0x7f65700c7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:39.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f656ffff700 1 -- 172.21.15.1:0/3028648814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65700c5390 con 0x7f65700c7640 2024-03-18T19:02:39.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f656ffff700 1 --2- 172.21.15.1:0/3028648814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65700c7640 0x7f6570145ce0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f65700cec60 tx=0x7f656800f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:39.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f6557fff700 1 -- 172.21.15.1:0/3028648814 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656800e040 con 0x7f65700c7640 2024-03-18T19:02:39.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f6557fff700 1 -- 172.21.15.1:0/3028648814 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656800f040 con 0x7f65700c7640 2024-03-18T19:02:39.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f6557fff700 1 -- 172.21.15.1:0/3028648814 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6568015430 con 0x7f65700c7640 2024-03-18T19:02:39.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f6575d60700 1 -- 172.21.15.1:0/3028648814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65700c5670 con 0x7f65700c7640 2024-03-18T19:02:39.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.620+0000 7f6575d60700 1 -- 172.21.15.1:0/3028648814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65700c5c30 con 0x7f65700c7640 2024-03-18T19:02:39.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.621+0000 7f6575d60700 1 -- 172.21.15.1:0/3028648814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65700676a0 con 0x7f65700c7640 2024-03-18T19:02:39.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.622+0000 7f6557fff700 1 -- 172.21.15.1:0/3028648814 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f6568015590 con 0x7f65700c7640 2024-03-18T19:02:39.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.622+0000 7f6557fff700 1 --2- 172.21.15.1:0/3028648814 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f655c06ae90 0x7f655c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:39.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.623+0000 7f6557fff700 1 -- 172.21.15.1:0/3028648814 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6884+0+0 (secure 0 0 0) 0x7f656808a1a0 con 0x7f65700c7640 2024-03-18T19:02:39.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.623+0000 7f656f7fe700 1 --2- 172.21.15.1:0/3028648814 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f655c06ae90 0x7f655c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:39.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.623+0000 7f656f7fe700 1 --2- 172.21.15.1:0/3028648814 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f655c06ae90 0x7f655c06d350 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f65600018d0 tx=0x7f6560005040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:39.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.627+0000 7f6557fff700 1 -- 172.21.15.1:0/3028648814 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6568054b50 con 0x7f65700c7640 2024-03-18T19:02:39.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.857+0000 7f6575d60700 1 -- 172.21.15.1:0/3028648814 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f65700d07b0 con 0x7f655c06ae90 2024-03-18T19:02:39.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.929+0000 7f6557fff700 1 -- 172.21.15.1:0/3028648814 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f6560001330 con 0x7f655c06ae90 2024-03-18T19:02:39.925 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule removed for path /snap_test_dir1/dir8 2024-03-18T19:02:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.931+0000 7f6575d60700 1 -- 172.21.15.1:0/3028648814 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f655c06ae90 msgr2=0x7f655c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.931+0000 7f6575d60700 1 --2- 172.21.15.1:0/3028648814 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f655c06ae90 0x7f655c06d350 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f65600018d0 tx=0x7f6560005040 comp rx=0 tx=0).stop 2024-03-18T19:02:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.931+0000 7f6575d60700 1 -- 172.21.15.1:0/3028648814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65700c7640 msgr2=0x7f6570145ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.931+0000 7f6575d60700 1 --2- 172.21.15.1:0/3028648814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65700c7640 0x7f6570145ce0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f65700cec60 tx=0x7f656800f490 comp rx=0 tx=0).stop 2024-03-18T19:02:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.931+0000 7f6575d60700 1 -- 172.21.15.1:0/3028648814 shutdown_connections 2024-03-18T19:02:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.931+0000 7f6575d60700 1 --2- 172.21.15.1:0/3028648814 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f655c06ae90 0x7f655c06d350 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.931+0000 7f6575d60700 1 --2- 172.21.15.1:0/3028648814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6558003a30 0x7f65700c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.931+0000 7f6575d60700 1 --2- 172.21.15.1:0/3028648814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65700c7640 0x7f6570145ce0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.931+0000 7f6575d60700 1 --2- 172.21.15.1:0/3028648814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6570148660 0x7f65700c7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.931+0000 7f6575d60700 1 -- 172.21.15.1:0/3028648814 >> 172.21.15.1:0/3028648814 conn(0x7f65700b2bd0 msgr2=0x7f657006f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.931+0000 7f6575d60700 1 -- 172.21.15.1:0/3028648814 shutdown_connections 2024-03-18T19:02:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:39.931+0000 7f6575d60700 1 -- 172.21.15.1:0/3028648814 wait complete. 2024-03-18T19:02:39.939 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2024-03-18T19:02:39.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:39.939 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2024-03-18T19:02:39.965 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:39.965 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:39.965 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:39.965 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:39.965 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:39.965 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:39.965 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:39.965 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:39.966 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:39.966 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:39.966 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:39.967 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:39.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:39.967 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-18-18_52_00_UTC) 2024-03-18T19:02:40.048 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:40.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:40.048 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-18-18_53_00_UTC) 2024-03-18T19:02:40.115 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:40.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:40.116 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-18-18_54_00_UTC) 2024-03-18T19:02:40.188 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:40.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:40.188 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-18-18_55_00_UTC) 2024-03-18T19:02:40.256 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:40.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:40.256 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-18-18_56_00_UTC) 2024-03-18T19:02:40.322 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:40.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:40.322 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-18-18_57_00_UTC) 2024-03-18T19:02:40.387 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:40.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:40.387 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-18-18_58_00_UTC) 2024-03-18T19:02:40.455 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:40.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:40.456 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-18-18_59_00_UTC) 2024-03-18T19:02:40.521 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:40.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:40.522 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-18-19_00_00_UTC) 2024-03-18T19:02:40.588 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:40.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:40.588 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-18-19_01_00_UTC) 2024-03-18T19:02:40.655 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:40.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:40.655 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-03-18-19_02_00_UTC) 2024-03-18T19:02:40.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:40.721 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2024-03-18T19:02:40.783 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir9 1m 2024-03-18T19:02:40.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.955+0000 7fe05a927700 1 Processor -- start 2024-03-18T19:02:40.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.955+0000 7fe05a927700 1 -- start start 2024-03-18T19:02:40.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.956+0000 7fe05a927700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe05415a1d0 0x7fe0540add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:40.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.956+0000 7fe05a927700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0540cc0f0 0x7fe054071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:40.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.956+0000 7fe05a927700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0540ce8f0 con 0x7fe054071620 2024-03-18T19:02:40.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.956+0000 7fe05a927700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0540cdb00 con 0x7fe05415a1d0 2024-03-18T19:02:40.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.956+0000 7fe05a927700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe05415a5b0 con 0x7fe0540cc0f0 2024-03-18T19:02:40.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.956+0000 7fe058ec4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0540cc0f0 0x7fe054071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:40.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.956+0000 7fe053fff700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe05415a1d0 0x7fe0540add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:40.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.956+0000 7fe053fff700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe05415a1d0 0x7fe0540add20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:49094/0 (socket says 172.21.15.1:49094) 2024-03-18T19:02:40.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.956+0000 7fe053fff700 1 -- 172.21.15.1:0/4181868790 learned_addr learned my addr 172.21.15.1:0/4181868790 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:40.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.956+0000 7fe053fff700 1 -- 172.21.15.1:0/4181868790 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0540cc0f0 msgr2=0x7fe054071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:40.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.957+0000 7fe053fff700 1 --2- 172.21.15.1:0/4181868790 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0540cc0f0 0x7fe054071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:40.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.957+0000 7fe053fff700 1 -- 172.21.15.1:0/4181868790 >> v1:172.21.15.165:6789/0 conn(0x7fe054071620 legacy=0x7fe0540cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:40.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.957+0000 7fe053fff700 1 -- 172.21.15.1:0/4181868790 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe054071fc0 con 0x7fe05415a1d0 2024-03-18T19:02:40.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.957+0000 7fe052ffd700 1 -- 172.21.15.1:0/4181868790 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975903917 0 0) 0x7fe0540ce8f0 con 0x7fe054071620 2024-03-18T19:02:40.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.957+0000 7fe058ec4700 1 --2- 172.21.15.1:0/4181868790 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0540cc0f0 0x7fe054071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:02:40.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.957+0000 7fe053fff700 1 --2- 172.21.15.1:0/4181868790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe05415a1d0 0x7fe0540add20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe03c006fd0 tx=0x7fe03c009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2707fb2fddd89590 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:40.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.957+0000 7fe052ffd700 1 -- 172.21.15.1:0/4181868790 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c005e30 con 0x7fe05415a1d0 2024-03-18T19:02:40.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.957+0000 7fe052ffd700 1 -- 172.21.15.1:0/4181868790 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe03c004030 con 0x7fe05415a1d0 2024-03-18T19:02:40.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.957+0000 7fe052ffd700 1 -- 172.21.15.1:0/4181868790 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c0043e0 con 0x7fe05415a1d0 2024-03-18T19:02:40.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.957+0000 7fe05a927700 1 -- 172.21.15.1:0/4181868790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe05415a1d0 msgr2=0x7fe0540add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:40.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.957+0000 7fe05a927700 1 --2- 172.21.15.1:0/4181868790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe05415a1d0 0x7fe0540add20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe03c006fd0 tx=0x7fe03c009bd0 comp rx=0 tx=0).stop 2024-03-18T19:02:40.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.957+0000 7fe05a927700 1 -- 172.21.15.1:0/4181868790 shutdown_connections 2024-03-18T19:02:40.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.957+0000 7fe05a927700 1 --2- 172.21.15.1:0/4181868790 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0540cc0f0 0x7fe054071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:40.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.957+0000 7fe05a927700 1 --2- 172.21.15.1:0/4181868790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe05415a1d0 0x7fe0540add20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:40.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.957+0000 7fe05a927700 1 -- 172.21.15.1:0/4181868790 >> 172.21.15.1:0/4181868790 conn(0x7fe0540cc9b0 msgr2=0x7fe0540ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:40.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.957+0000 7fe05a927700 1 -- 172.21.15.1:0/4181868790 shutdown_connections 2024-03-18T19:02:40.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.957+0000 7fe05a927700 1 -- 172.21.15.1:0/4181868790 wait complete. 2024-03-18T19:02:40.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.959+0000 7fe05a927700 1 Processor -- start 2024-03-18T19:02:40.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.959+0000 7fe05a927700 1 -- start start 2024-03-18T19:02:40.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe05a927700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe054071620 0x7fe0540c63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:40.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe05a927700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0540cc0f0 0x7fe05415cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:40.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe05a927700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe05415a1d0 0x7fe0540b4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:40.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe05a927700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0540c6910 con 0x7fe05415a1d0 2024-03-18T19:02:40.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe05a927700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0540c5f40 con 0x7fe0540cc0f0 2024-03-18T19:02:40.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe05a927700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0540c5c40 con 0x7fe054071620 2024-03-18T19:02:40.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe058ec4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe05415a1d0 0x7fe0540b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:40.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe058ec4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe05415a1d0 0x7fe0540b4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35340/0 (socket says 172.21.15.1:35340) 2024-03-18T19:02:40.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe058ec4700 1 -- 172.21.15.1:0/3055616819 learned_addr learned my addr 172.21.15.1:0/3055616819 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:40.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe053fff700 1 --2- 172.21.15.1:0/3055616819 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe054071620 0x7fe0540c63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:40.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe0537fe700 1 --2- 172.21.15.1:0/3055616819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0540cc0f0 0x7fe05415cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:40.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe058ec4700 1 -- 172.21.15.1:0/3055616819 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe054071620 msgr2=0x7fe0540c63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:40.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe058ec4700 1 --2- 172.21.15.1:0/3055616819 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe054071620 0x7fe0540c63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:40.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe058ec4700 1 -- 172.21.15.1:0/3055616819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0540cc0f0 msgr2=0x7fe05415cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:40.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe058ec4700 1 --2- 172.21.15.1:0/3055616819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0540cc0f0 0x7fe05415cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:40.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe058ec4700 1 -- 172.21.15.1:0/3055616819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04c009a70 con 0x7fe05415a1d0 2024-03-18T19:02:40.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe0537fe700 1 --2- 172.21.15.1:0/3055616819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0540cc0f0 0x7fe05415cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:40.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe058ec4700 1 --2- 172.21.15.1:0/3055616819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe05415a1d0 0x7fe0540b4150 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe04c009eb0 tx=0x7fe04c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:40.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe050ff9700 1 -- 172.21.15.1:0/3055616819 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04c009d50 con 0x7fe05415a1d0 2024-03-18T19:02:40.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe050ff9700 1 -- 172.21.15.1:0/3055616819 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe04c00f040 con 0x7fe05415a1d0 2024-03-18T19:02:40.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.960+0000 7fe05a927700 1 -- 172.21.15.1:0/3055616819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe03c009500 con 0x7fe05415a1d0 2024-03-18T19:02:40.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.961+0000 7fe050ff9700 1 -- 172.21.15.1:0/3055616819 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04c014480 con 0x7fe05415a1d0 2024-03-18T19:02:40.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.961+0000 7fe05a927700 1 -- 172.21.15.1:0/3055616819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0540b4bd0 con 0x7fe05415a1d0 2024-03-18T19:02:40.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.962+0000 7fe05a927700 1 -- 172.21.15.1:0/3055616819 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0540c60d0 con 0x7fe05415a1d0 2024-03-18T19:02:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.962+0000 7fe050ff9700 1 -- 172.21.15.1:0/3055616819 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7fe04c0145e0 con 0x7fe05415a1d0 2024-03-18T19:02:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.962+0000 7fe050ff9700 1 --2- 172.21.15.1:0/3055616819 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe03406aec0 0x7fe03406d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.962+0000 7fe050ff9700 1 -- 172.21.15.1:0/3055616819 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6788+0+0 (secure 0 0 0) 0x7fe04c08aa50 con 0x7fe05415a1d0 2024-03-18T19:02:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.962+0000 7fe053fff700 1 --2- 172.21.15.1:0/3055616819 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe03406aec0 0x7fe03406d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.963+0000 7fe053fff700 1 --2- 172.21.15.1:0/3055616819 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe03406aec0 0x7fe03406d380 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe03c006fd0 tx=0x7fe03c008070 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:40.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:40.967+0000 7fe050ff9700 1 -- 172.21.15.1:0/3055616819 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe04c0553d0 con 0x7fe05415a1d0 2024-03-18T19:02:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:41.268+0000 7fe05a927700 1 -- 172.21.15.1:0/3055616819 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0540cdfc0 con 0x7fe03406aec0 2024-03-18T19:02:41.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:41.339+0000 7fe050ff9700 1 -- 172.21.15.1:0/3055616819 <== mgr.4997 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fe03c006c00 con 0x7fe03406aec0 2024-03-18T19:02:41.335 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule removed for path /snap_test_dir1/dir9 2024-03-18T19:02:41.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:41.341+0000 7fe05a927700 1 -- 172.21.15.1:0/3055616819 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe03406aec0 msgr2=0x7fe03406d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:41.341+0000 7fe05a927700 1 --2- 172.21.15.1:0/3055616819 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe03406aec0 0x7fe03406d380 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe03c006fd0 tx=0x7fe03c008070 comp rx=0 tx=0).stop 2024-03-18T19:02:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:41.341+0000 7fe05a927700 1 -- 172.21.15.1:0/3055616819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe05415a1d0 msgr2=0x7fe0540b4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:41.341+0000 7fe05a927700 1 --2- 172.21.15.1:0/3055616819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe05415a1d0 0x7fe0540b4150 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe04c009eb0 tx=0x7fe04c009f90 comp rx=0 tx=0).stop 2024-03-18T19:02:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:41.341+0000 7fe05a927700 1 -- 172.21.15.1:0/3055616819 shutdown_connections 2024-03-18T19:02:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:41.341+0000 7fe05a927700 1 --2- 172.21.15.1:0/3055616819 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe03406aec0 0x7fe03406d380 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:41.341+0000 7fe05a927700 1 --2- 172.21.15.1:0/3055616819 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe054071620 0x7fe0540c63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:41.341+0000 7fe05a927700 1 --2- 172.21.15.1:0/3055616819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe05415a1d0 0x7fe0540b4150 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:41.341+0000 7fe05a927700 1 --2- 172.21.15.1:0/3055616819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0540cc0f0 0x7fe05415cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:41.341+0000 7fe05a927700 1 -- 172.21.15.1:0/3055616819 >> 172.21.15.1:0/3055616819 conn(0x7fe0540cc9b0 msgr2=0x7fe05406f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:41.341+0000 7fe05a927700 1 -- 172.21.15.1:0/3055616819 shutdown_connections 2024-03-18T19:02:41.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:41.341+0000 7fe05a927700 1 -- 172.21.15.1:0/3055616819 wait complete. 2024-03-18T19:02:41.349 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2024-03-18T19:02:41.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:41.349 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2024-03-18T19:02:41.375 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:41.375 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:41.375 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:41.375 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:41.375 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:41.375 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:41.375 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:41.375 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:41.375 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:41.376 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:41.376 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:41.376 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-18-18_52_00_UTC 2024-03-18T19:02:41.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:41.377 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-18-18_52_00_UTC) 2024-03-18T19:02:41.443 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-18-18_53_00_UTC 2024-03-18T19:02:41.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:41.443 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-18-18_53_00_UTC) 2024-03-18T19:02:41.508 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-18-18_54_00_UTC 2024-03-18T19:02:41.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:41.508 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-18-18_54_00_UTC) 2024-03-18T19:02:41.575 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-18-18_55_00_UTC 2024-03-18T19:02:41.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:41.575 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-18-18_55_00_UTC) 2024-03-18T19:02:41.644 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-18-18_56_00_UTC 2024-03-18T19:02:41.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:41.644 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-18-18_56_00_UTC) 2024-03-18T19:02:41.709 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-18-18_57_00_UTC 2024-03-18T19:02:41.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:41.710 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-18-18_57_00_UTC) 2024-03-18T19:02:41.776 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-18-18_58_00_UTC 2024-03-18T19:02:41.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:41.776 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-18-18_58_00_UTC) 2024-03-18T19:02:41.841 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-18-18_59_00_UTC 2024-03-18T19:02:41.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:41.842 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-18-18_59_00_UTC) 2024-03-18T19:02:41.908 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-18-19_00_00_UTC 2024-03-18T19:02:41.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:41.908 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-18-19_00_00_UTC) 2024-03-18T19:02:41.975 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-18-19_01_00_UTC 2024-03-18T19:02:41.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:41.975 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-18-19_01_00_UTC) 2024-03-18T19:02:42.055 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-03-18-19_02_00_UTC 2024-03-18T19:02:42.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:42.055 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-03-18-19_02_00_UTC) 2024-03-18T19:02:42.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:42.120 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2024-03-18T19:02:42.184 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-18T19:02:42.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.368+0000 7f336668a700 1 Processor -- start 2024-03-18T19:02:42.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.368+0000 7f336668a700 1 -- start start 2024-03-18T19:02:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.369+0000 7f336668a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f336015a1d0 0x7f33600add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.369+0000 7f336668a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f33600cc0f0 0x7f3360071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.369+0000 7f336668a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33600ce8f0 con 0x7f3360071620 2024-03-18T19:02:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.369+0000 7f336668a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33600cdb00 con 0x7f336015a1d0 2024-03-18T19:02:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.369+0000 7f336668a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f336015a5b0 con 0x7f33600cc0f0 2024-03-18T19:02:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.369+0000 7f335ffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f336015a1d0 0x7f33600add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.369+0000 7f335ffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f336015a1d0 0x7f33600add20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35344/0 (socket says 172.21.15.1:35344) 2024-03-18T19:02:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.369+0000 7f335ffff700 1 -- 172.21.15.1:0/3966118364 learned_addr learned my addr 172.21.15.1:0/3966118364 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.370+0000 7f3364c27700 1 --2- 172.21.15.1:0/3966118364 >> v2:172.21.15.165:3300/0 conn(0x7f33600cc0f0 0x7f3360071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.370+0000 7f335ffff700 1 -- 172.21.15.1:0/3966118364 >> v2:172.21.15.165:3300/0 conn(0x7f33600cc0f0 msgr2=0x7f3360071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:42.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.370+0000 7f335ffff700 1 --2- 172.21.15.1:0/3966118364 >> v2:172.21.15.165:3300/0 conn(0x7f33600cc0f0 0x7f3360071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:42.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.370+0000 7f335ffff700 1 -- 172.21.15.1:0/3966118364 >> v1:172.21.15.165:6789/0 conn(0x7f3360071620 legacy=0x7f33600cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:42.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.370+0000 7f335ffff700 1 -- 172.21.15.1:0/3966118364 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3360071fc0 con 0x7f336015a1d0 2024-03-18T19:02:42.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.370+0000 7f335ffff700 1 --2- 172.21.15.1:0/3966118364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f336015a1d0 0x7f33600add20 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f3360072200 tx=0x7f3348009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f236eec220384a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:42.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.370+0000 7f335effd700 1 -- 172.21.15.1:0/3966118364 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348005e30 con 0x7f336015a1d0 2024-03-18T19:02:42.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.370+0000 7f335effd700 1 -- 172.21.15.1:0/3966118364 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3348006030 con 0x7f336015a1d0 2024-03-18T19:02:42.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.370+0000 7f335effd700 1 -- 172.21.15.1:0/3966118364 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33480063e0 con 0x7f336015a1d0 2024-03-18T19:02:42.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.370+0000 7f336668a700 1 -- 172.21.15.1:0/3966118364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f336015a1d0 msgr2=0x7f33600add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:42.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.370+0000 7f336668a700 1 --2- 172.21.15.1:0/3966118364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f336015a1d0 0x7f33600add20 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f3360072200 tx=0x7f3348009bd0 comp rx=0 tx=0).stop 2024-03-18T19:02:42.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.370+0000 7f336668a700 1 -- 172.21.15.1:0/3966118364 shutdown_connections 2024-03-18T19:02:42.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.370+0000 7f336668a700 1 --2- 172.21.15.1:0/3966118364 >> v2:172.21.15.165:3300/0 conn(0x7f33600cc0f0 0x7f3360071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:42.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.370+0000 7f336668a700 1 --2- 172.21.15.1:0/3966118364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f336015a1d0 0x7f33600add20 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:42.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.370+0000 7f336668a700 1 -- 172.21.15.1:0/3966118364 >> 172.21.15.1:0/3966118364 conn(0x7f33600cc9b0 msgr2=0x7f33600ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:42.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.371+0000 7f336668a700 1 -- 172.21.15.1:0/3966118364 shutdown_connections 2024-03-18T19:02:42.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.371+0000 7f336668a700 1 -- 172.21.15.1:0/3966118364 wait complete. 2024-03-18T19:02:42.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.372+0000 7f336668a700 1 Processor -- start 2024-03-18T19:02:42.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.372+0000 7f336668a700 1 -- start start 2024-03-18T19:02:42.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f336668a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3360071620 0x7f336006a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:42.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f336668a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f33600cc0f0 0x7f33600c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:42.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f336668a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f336015a1d0 0x7f33600c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:42.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f336668a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33600c7000 con 0x7f3360071620 2024-03-18T19:02:42.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f336668a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33600c5e40 con 0x7f336015a1d0 2024-03-18T19:02:42.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f336668a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33600c6120 con 0x7f33600cc0f0 2024-03-18T19:02:42.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f335ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3360071620 0x7f336006a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:42.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f335ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3360071620 0x7f336006a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35348/0 (socket says 172.21.15.1:35348) 2024-03-18T19:02:42.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f335ffff700 1 -- 172.21.15.1:0/3813532137 learned_addr learned my addr 172.21.15.1:0/3813532137 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:42.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f3364c27700 1 --2- 172.21.15.1:0/3813532137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f336015a1d0 0x7f33600c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:42.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f335f7fe700 1 --2- 172.21.15.1:0/3813532137 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f33600cc0f0 0x7f33600c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:42.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f335ffff700 1 -- 172.21.15.1:0/3813532137 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f33600cc0f0 msgr2=0x7f33600c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:42.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f335ffff700 1 --2- 172.21.15.1:0/3813532137 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f33600cc0f0 0x7f33600c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:42.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f335ffff700 1 -- 172.21.15.1:0/3813532137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f336015a1d0 msgr2=0x7f33600c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:42.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f335ffff700 1 --2- 172.21.15.1:0/3813532137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f336015a1d0 0x7f33600c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:42.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f335ffff700 1 -- 172.21.15.1:0/3813532137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3348009500 con 0x7f3360071620 2024-03-18T19:02:42.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f3364c27700 1 --2- 172.21.15.1:0/3813532137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f336015a1d0 0x7f33600c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:42.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f335ffff700 1 --2- 172.21.15.1:0/3813532137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3360071620 0x7f336006a9c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3348013040 tx=0x7f3348004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:42.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f335cff9700 1 -- 172.21.15.1:0/3813532137 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348025430 con 0x7f3360071620 2024-03-18T19:02:42.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f335cff9700 1 -- 172.21.15.1:0/3813532137 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3348020070 con 0x7f3360071620 2024-03-18T19:02:42.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f336668a700 1 -- 172.21.15.1:0/3813532137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33600c6400 con 0x7f3360071620 2024-03-18T19:02:42.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.373+0000 7f335cff9700 1 -- 172.21.15.1:0/3813532137 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348025810 con 0x7f3360071620 2024-03-18T19:02:42.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.374+0000 7f336668a700 1 -- 172.21.15.1:0/3813532137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336006af00 con 0x7f3360071620 2024-03-18T19:02:42.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.375+0000 7f336668a700 1 -- 172.21.15.1:0/3813532137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33600ad060 con 0x7f3360071620 2024-03-18T19:02:42.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.375+0000 7f335cff9700 1 -- 172.21.15.1:0/3813532137 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 89325+0+0 (secure 0 0 0) 0x7f3348022030 con 0x7f3360071620 2024-03-18T19:02:42.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.376+0000 7f335cff9700 1 --2- 172.21.15.1:0/3813532137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f334006ae90 0x7f334006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:42.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.376+0000 7f335cff9700 1 -- 172.21.15.1:0/3813532137 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7136+0+0 (secure 0 0 0) 0x7f334808b690 con 0x7f3360071620 2024-03-18T19:02:42.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.376+0000 7f335f7fe700 1 --2- 172.21.15.1:0/3813532137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f334006ae90 0x7f334006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:42.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.376+0000 7f335f7fe700 1 --2- 172.21.15.1:0/3813532137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f334006ae90 0x7f334006d350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f336015d9b0 tx=0x7f3350008040 comp rx=0 tx=0).ready entity=mgr.4997 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:42.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.380+0000 7f335cff9700 1 -- 172.21.15.1:0/3813532137 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f33480561f0 con 0x7f3360071620 2024-03-18T19:02:42.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:42.642+0000 7f336668a700 1 -- 172.21.15.1:0/3813532137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f336006b0f0 con 0x7f3360071620 2024-03-18T19:02:43.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.050+0000 7f335cff9700 1 -- 172.21.15.1:0/3813532137 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f33480565c0 con 0x7f3360071620 2024-03-18T19:02:43.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.051+0000 7f335cff9700 1 -- 172.21.15.1:0/3813532137 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v14) v1 ==== 93+0+0 (secure 0 0 0) 0x7f3348059c80 con 0x7f3360071620 2024-03-18T19:02:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.055+0000 7f336668a700 1 -- 172.21.15.1:0/3813532137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f334006ae90 msgr2=0x7f334006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.055+0000 7f336668a700 1 --2- 172.21.15.1:0/3813532137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f334006ae90 0x7f334006d350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f336015d9b0 tx=0x7f3350008040 comp rx=0 tx=0).stop 2024-03-18T19:02:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.055+0000 7f336668a700 1 -- 172.21.15.1:0/3813532137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3360071620 msgr2=0x7f336006a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.055+0000 7f336668a700 1 --2- 172.21.15.1:0/3813532137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3360071620 0x7f336006a9c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3348013040 tx=0x7f3348004750 comp rx=0 tx=0).stop 2024-03-18T19:02:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.055+0000 7f336668a700 1 -- 172.21.15.1:0/3813532137 shutdown_connections 2024-03-18T19:02:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.055+0000 7f336668a700 1 --2- 172.21.15.1:0/3813532137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f334006ae90 0x7f334006d350 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.055+0000 7f336668a700 1 --2- 172.21.15.1:0/3813532137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f336015a1d0 0x7f33600c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:43.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.055+0000 7f336668a700 1 --2- 172.21.15.1:0/3813532137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3360071620 0x7f336006a9c0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.055+0000 7f336668a700 1 --2- 172.21.15.1:0/3813532137 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f33600cc0f0 0x7f33600c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.055+0000 7f336668a700 1 -- 172.21.15.1:0/3813532137 >> 172.21.15.1:0/3813532137 conn(0x7f33600cc9b0 msgr2=0x7f33600ccd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.055+0000 7f336668a700 1 -- 172.21.15.1:0/3813532137 shutdown_connections 2024-03-18T19:02:43.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.055+0000 7f336668a700 1 -- 172.21.15.1:0/3813532137 wait complete. 2024-03-18T19:02:43.067 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T19:02:43.067 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:02:43.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:02:43.155 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T19:02:43.155 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:02:43.155 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:02:43.176 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:02:43.206 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:43.210+0000 7fdb7645b240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:02:43.209 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:43.211+0000 7f5e46aab240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:02:43.354 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:43.358+0000 7f5e46aab240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:02:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.375+0000 7f13206ee700 1 Processor -- start 2024-03-18T19:02:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.375+0000 7f13206ee700 1 -- start start 2024-03-18T19:02:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.377+0000 7f13206ee700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f131808a270 0x7f131816b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.377+0000 7f13206ee700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1318070f30 con 0x7f131808a7c0 2024-03-18T19:02:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.377+0000 7f13206ee700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131816dcc0 con 0x7f131806a750 2024-03-18T19:02:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.377+0000 7f13206ee700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f131816dfa0 con 0x7f131808a270 2024-03-18T19:02:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.378+0000 7f131dc89700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f131808a7c0 0x7f131816a7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56192/0 (socket says 172.21.15.1:56192) 2024-03-18T19:02:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.378+0000 7f131dc89700 1 -- 172.21.15.1:0/2149854049 learned_addr learned my addr 172.21.15.1:0/2149854049 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:43.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.378+0000 7f131d488700 1 -- 172.21.15.1:0/2149854049 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543098244 0 0) 0x7f1318070f30 con 0x7f131808a7c0 2024-03-18T19:02:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.378+0000 7f131d488700 1 -- 172.21.15.1:0/2149854049 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1304003590 con 0x7f131808a7c0 2024-03-18T19:02:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.379+0000 7f131d488700 1 -- 172.21.15.1:0/2149854049 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 684333911 0 0) 0x7f1304003590 con 0x7f131808a7c0 2024-03-18T19:02:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.379+0000 7f131d488700 1 -- 172.21.15.1:0/2149854049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f131808a270 msgr2=0x7f131816b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.379+0000 7f131d488700 1 --2- 172.21.15.1:0/2149854049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f131808a270 0x7f131816b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.379+0000 7f131d488700 1 -- 172.21.15.1:0/2149854049 >> v1:172.21.15.165:6789/0 conn(0x7f131806a750 legacy=0x7f131806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.379+0000 7f131d488700 1 -- 172.21.15.1:0/2149854049 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131816e280 con 0x7f131808a7c0 2024-03-18T19:02:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.379+0000 7f131d488700 1 -- 172.21.15.1:0/2149854049 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f1314001670 con 0x7f131808a7c0 2024-03-18T19:02:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.379+0000 7f131d488700 1 -- 172.21.15.1:0/2149854049 >> v1:172.21.15.1:6789/0 conn(0x7f131808a7c0 legacy=0x7f131816a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.379+0000 7f131d488700 1 --2- 172.21.15.1:0/2149854049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1304004bf0 0x7f1304003920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.379+0000 7f131d488700 1 --2- 172.21.15.1:0/2149854049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1304003e60 0x7f13040042e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.379+0000 7f131d488700 1 --2- 172.21.15.1:0/2149854049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f131808a270 0x7f130400d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.379+0000 7f131d488700 1 -- 172.21.15.1:0/2149854049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131816dfa0 con 0x7f1304003e60 2024-03-18T19:02:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.379+0000 7f131d488700 1 -- 172.21.15.1:0/2149854049 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130400de10 con 0x7f1304004bf0 2024-03-18T19:02:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.379+0000 7f131d488700 1 -- 172.21.15.1:0/2149854049 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130400dfd0 con 0x7f131808a270 2024-03-18T19:02:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.379+0000 7f131e48a700 1 --2- 172.21.15.1:0/2149854049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1304004bf0 0x7f1304003920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.379+0000 7f131dc89700 1 --2- 172.21.15.1:0/2149854049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1304003e60 0x7f13040042e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.380+0000 7f131e48a700 1 -- 172.21.15.1:0/2149854049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f131808a270 msgr2=0x7f130400d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.380+0000 7f131e48a700 1 --2- 172.21.15.1:0/2149854049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f131808a270 0x7f130400d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.380+0000 7f131e48a700 1 -- 172.21.15.1:0/2149854049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1304003e60 msgr2=0x7f13040042e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.380+0000 7f131e48a700 1 --2- 172.21.15.1:0/2149854049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1304003e60 0x7f13040042e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.380+0000 7f131e48a700 1 -- 172.21.15.1:0/2149854049 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1304003590 con 0x7f1304004bf0 2024-03-18T19:02:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.380+0000 7f131e48a700 1 --2- 172.21.15.1:0/2149854049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1304004bf0 0x7f1304003920 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1310001de0 tx=0x7f1304003840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e0d3e8016d14a21 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:43.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.382+0000 7f131d488700 1 -- 172.21.15.1:0/2149854049 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1310001850 con 0x7f1304004bf0 2024-03-18T19:02:43.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.382+0000 7f131d488700 1 -- 172.21.15.1:0/2149854049 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1310003c20 con 0x7f1304004bf0 2024-03-18T19:02:43.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.382+0000 7f131d488700 1 -- 172.21.15.1:0/2149854049 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13100050d0 con 0x7f1304004bf0 2024-03-18T19:02:43.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.383+0000 7f13206ee700 1 -- 172.21.15.1:0/2149854049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1304004bf0 msgr2=0x7f1304003920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.383+0000 7f13206ee700 1 --2- 172.21.15.1:0/2149854049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1304004bf0 0x7f1304003920 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1310001de0 tx=0x7f1304003840 comp rx=0 tx=0).stop 2024-03-18T19:02:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.383+0000 7f131ec8b700 1 -- 172.21.15.1:0/2149854049 reap_dead start 2024-03-18T19:02:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.383+0000 7f13206ee700 1 -- 172.21.15.1:0/2149854049 shutdown_connections 2024-03-18T19:02:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.383+0000 7f13206ee700 1 -- 172.21.15.1:0/2149854049 >> 172.21.15.1:0/2149854049 conn(0x7f13180b0a00 msgr2=0x7f13180b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.384+0000 7f13206ee700 1 -- 172.21.15.1:0/2149854049 shutdown_connections 2024-03-18T19:02:43.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.384+0000 7f13206ee700 1 -- 172.21.15.1:0/2149854049 wait complete. 2024-03-18T19:02:43.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.386+0000 7f13206ee700 1 Processor -- start 2024-03-18T19:02:43.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.388+0000 7f13206ee700 1 -- start start 2024-03-18T19:02:43.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.389+0000 7f13206ee700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1304004bf0 0x7f131809c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.389+0000 7f13206ee700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1318167570 0x7f13180c8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.389+0000 7f13206ee700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1318099a40 0x7f131806c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.389+0000 7f13206ee700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1318167f10 con 0x7f1318099a40 2024-03-18T19:02:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.389+0000 7f13206ee700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13180ace90 con 0x7f1304004bf0 2024-03-18T19:02:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.389+0000 7f13206ee700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13181679f0 con 0x7f1318167570 2024-03-18T19:02:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.390+0000 7f131ec8b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1318099a40 0x7f131806c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.390+0000 7f131ec8b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1318099a40 0x7f131806c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35402/0 (socket says 172.21.15.1:35402) 2024-03-18T19:02:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.390+0000 7f131ec8b700 1 -- 172.21.15.1:0/2218054094 learned_addr learned my addr 172.21.15.1:0/2218054094 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.390+0000 7f131dc89700 1 --2- 172.21.15.1:0/2218054094 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1318167570 0x7f13180c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.390+0000 7f131ec8b700 1 -- 172.21.15.1:0/2218054094 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1318167570 msgr2=0x7f13180c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.390+0000 7f131ec8b700 1 --2- 172.21.15.1:0/2218054094 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1318167570 0x7f13180c8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.390+0000 7f131ec8b700 1 -- 172.21.15.1:0/2218054094 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1304004bf0 msgr2=0x7f131809c550 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T19:02:43.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.390+0000 7f131ec8b700 1 --2- 172.21.15.1:0/2218054094 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1304004bf0 0x7f131809c550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:43.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.390+0000 7f131ec8b700 1 -- 172.21.15.1:0/2218054094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13180ac0f0 con 0x7f1318099a40 2024-03-18T19:02:43.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.391+0000 7f131ec8b700 1 --2- 172.21.15.1:0/2218054094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1318099a40 0x7f131806c4f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f13180cede0 tx=0x7f130800c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:43.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.391+0000 7f130effd700 1 -- 172.21.15.1:0/2218054094 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130800cee0 con 0x7f1318099a40 2024-03-18T19:02:43.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.391+0000 7f13206ee700 1 -- 172.21.15.1:0/2218054094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13180ac3d0 con 0x7f1318099a40 2024-03-18T19:02:43.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.391+0000 7f13206ee700 1 -- 172.21.15.1:0/2218054094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1318099ec0 con 0x7f1318099a40 2024-03-18T19:02:43.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.391+0000 7f130effd700 1 -- 172.21.15.1:0/2218054094 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1308005c50 con 0x7f1318099a40 2024-03-18T19:02:43.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.391+0000 7f130effd700 1 -- 172.21.15.1:0/2218054094 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1308007070 con 0x7f1318099a40 2024-03-18T19:02:43.388 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:43.385+0000 7fdb7645b240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:02:43.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.393+0000 7f130effd700 1 -- 172.21.15.1:0/2218054094 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f13080071d0 con 0x7f1318099a40 2024-03-18T19:02:43.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.394+0000 7f130effd700 1 --2- 172.21.15.1:0/2218054094 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f12fc06ae70 0x7f12fc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:43.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.394+0000 7f131e48a700 1 -- 172.21.15.1:0/2218054094 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f12fc06ae70 msgr2=0x7f12fc06d330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:43.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.394+0000 7f130effd700 1 -- 172.21.15.1:0/2218054094 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6948+0+0 (secure 0 0 0) 0x7f1308088fa0 con 0x7f1318099a40 2024-03-18T19:02:43.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.394+0000 7f131e48a700 1 --2- 172.21.15.1:0/2218054094 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f12fc06ae70 0x7f12fc06d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:02:43.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.394+0000 7f13206ee700 1 -- 172.21.15.1:0/2218054094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13000052f0 con 0x7f1318099a40 2024-03-18T19:02:43.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.404+0000 7f130effd700 1 -- 172.21.15.1:0/2218054094 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f13080538f0 con 0x7f1318099a40 2024-03-18T19:02:43.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.595+0000 7f131e48a700 1 -- 172.21.15.1:0/2218054094 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f12fc06ae70 msgr2=0x7f12fc06d330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:43.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.595+0000 7f131e48a700 1 --2- 172.21.15.1:0/2218054094 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f12fc06ae70 0x7f12fc06d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:02:43.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.639+0000 7f13206ee700 1 -- 172.21.15.1:0/2218054094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1300005870 con 0x7f1318099a40 2024-03-18T19:02:43.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.641+0000 7f130effd700 1 -- 172.21.15.1:0/2218054094 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+17644 (secure 0 0 0) 0x7f1308056f10 con 0x7f1318099a40 2024-03-18T19:02:43.637 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:43.637 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":56,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:02:43.014664+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:51:11.076221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T18:51:11.687105+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":218,"snap_epoch":56,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.26","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/1992330207":"2024-03-19T18:51:27.537808+0000","172.21.15.1:6832/76998":"2024-03-19T18:51:27.537808+0000","172.21.15.1:0/4026407635":"2024-03-19T18:51:27.537808+0000","172.21.15.1:0/2322425425":"2024-03-19T18:51:27.537808+0000","172.21.15.1:6833/76998":"2024-03-19T18:51:27.537808+0000","172.21.15.1:0/72803206":"2024-03-19T18:51:27.537808+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":2},{"begin":7,"length":1},{"begin":9,"length":1},{"begin":13,"length":2},{"begin":17,"length":1},{"begin":19,"length":2},{"begin":24,"length":2},{"begin":28,"length":1},{"begin":30,"length":2},{"begin":34,"length":2},{"begin":38,"length":1},{"begin":40,"length":3},{"begin":45,"length":2},{"begin":49,"length":1},{"begin":51,"length":3},{"begin":56,"length":2},{"begin":60,"length":1},{"begin":63,"length":2},{"begin":67,"length":2},{"begin":71,"length":2},{"begin":74,"length":2},{"begin":78,"length":2},{"begin":84,"length":2},{"begin":88,"length":2},{"begin":95,"length":2},{"begin":99,"length":2},{"begin":105,"length":2},{"begin":109,"length":2},{"begin":168,"length":51}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":7,"length":1},{"begin":17,"length":1},{"begin":28,"length":1},{"begin":38,"length":1},{"begin":49,"length":1},{"begin":60,"length":1},{"begin":71,"length":2},{"begin":109,"length":1},{"begin":210,"length":9}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:02:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.643+0000 7f13206ee700 1 -- 172.21.15.1:0/2218054094 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f12fc06ae70 msgr2=0x7f12fc06d330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:02:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.643+0000 7f13206ee700 1 --2- 172.21.15.1:0/2218054094 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f12fc06ae70 0x7f12fc06d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:43.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.643+0000 7f13206ee700 1 -- 172.21.15.1:0/2218054094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1318099a40 msgr2=0x7f131806c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:43.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.643+0000 7f13206ee700 1 --2- 172.21.15.1:0/2218054094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1318099a40 0x7f131806c4f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f13180cede0 tx=0x7f130800c720 comp rx=0 tx=0).stop 2024-03-18T19:02:43.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.643+0000 7f13206ee700 1 -- 172.21.15.1:0/2218054094 shutdown_connections 2024-03-18T19:02:43.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.643+0000 7f13206ee700 1 --2- 172.21.15.1:0/2218054094 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f12fc06ae70 0x7f12fc06d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:43.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.643+0000 7f13206ee700 1 --2- 172.21.15.1:0/2218054094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1318099a40 0x7f131806c4f0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:43.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.643+0000 7f13206ee700 1 --2- 172.21.15.1:0/2218054094 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1304004bf0 0x7f131809c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:43.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.643+0000 7f13206ee700 1 --2- 172.21.15.1:0/2218054094 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1318167570 0x7f13180c8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:43.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.643+0000 7f13206ee700 1 -- 172.21.15.1:0/2218054094 >> 172.21.15.1:0/2218054094 conn(0x7f13180b0a00 msgr2=0x7f13180aed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:43.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.643+0000 7f13206ee700 1 -- 172.21.15.1:0/2218054094 shutdown_connections 2024-03-18T19:02:43.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.643+0000 7f13206ee700 1 -- 172.21.15.1:0/2218054094 wait complete. 2024-03-18T19:02:43.647 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:43.651+0000 7f5e46aab240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:02:43.652 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T18:51:11.076221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T18:51:11.687105+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 218, 'snap_epoch': 56, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:02:43.652 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:02:43.692 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:43.696+0000 7fdb7645b240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:02:43.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.787+0000 7f12bb520700 1 Processor -- start 2024-03-18T19:02:43.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.787+0000 7f12bb520700 1 -- start start 2024-03-18T19:02:43.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.788+0000 7f12bb520700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f12b40cd870 0x7f12b40cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:43.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.788+0000 7f12bb520700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b40d0df0 con 0x7f12b40b1e20 2024-03-18T19:02:43.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.788+0000 7f12bb520700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b40d0550 con 0x7f12b40cbce0 2024-03-18T19:02:43.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.788+0000 7f12bb520700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b40d0830 con 0x7f12b40cd870 2024-03-18T19:02:43.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.788+0000 7f12b92bc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f12b40cd870 0x7f12b40cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:43.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.788+0000 7f12b8abb700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f12b40b1e20 0x7f12b40cdc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56200/0 (socket says 172.21.15.1:56200) 2024-03-18T19:02:43.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.788+0000 7f12b8abb700 1 -- 172.21.15.1:0/1423558840 learned_addr learned my addr 172.21.15.1:0/1423558840 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:43.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.788+0000 7f12b92bc700 1 --2- 172.21.15.1:0/1423558840 >> v2:172.21.15.1:3300/0 conn(0x7f12b40cd870 0x7f12b40cff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35416/0 (socket says 172.21.15.1:35416) 2024-03-18T19:02:43.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.788+0000 7f12b92bc700 1 -- 172.21.15.1:0/1423558840 >> v1:172.21.15.165:6789/0 conn(0x7f12b40cbce0 legacy=0x7f12b40cc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:43.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.788+0000 7f12b92bc700 1 -- 172.21.15.1:0/1423558840 >> v1:172.21.15.1:6789/0 conn(0x7f12b40b1e20 legacy=0x7f12b40cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:43.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.789+0000 7f12b92bc700 1 -- 172.21.15.1:0/1423558840 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b40d0b10 con 0x7f12b40cd870 2024-03-18T19:02:43.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.788+0000 7f12abfff700 1 -- 172.21.15.1:0/1423558840 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249709018 0 0) 0x7f12b40d0df0 con 0x7f12b40b1e20 2024-03-18T19:02:43.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.789+0000 7f12b92bc700 1 --2- 172.21.15.1:0/1423558840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b40cd870 0x7f12b40cff70 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009840 tx=0x7f12b0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29a107696f5aa13 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:43.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.789+0000 7f12abfff700 1 -- 172.21.15.1:0/1423558840 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b0011830 con 0x7f12b40cd870 2024-03-18T19:02:43.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.789+0000 7f12abfff700 1 -- 172.21.15.1:0/1423558840 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b0011990 con 0x7f12b40cd870 2024-03-18T19:02:43.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.789+0000 7f12abfff700 1 -- 172.21.15.1:0/1423558840 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b0011d20 con 0x7f12b40cd870 2024-03-18T19:02:43.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.789+0000 7f12bb520700 1 -- 172.21.15.1:0/1423558840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b40cd870 msgr2=0x7f12b40cff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:43.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.789+0000 7f12bb520700 1 --2- 172.21.15.1:0/1423558840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b40cd870 0x7f12b40cff70 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009840 tx=0x7f12b0009b50 comp rx=0 tx=0).stop 2024-03-18T19:02:43.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.789+0000 7f12bb520700 1 -- 172.21.15.1:0/1423558840 shutdown_connections 2024-03-18T19:02:43.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.789+0000 7f12bb520700 1 --2- 172.21.15.1:0/1423558840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b40cd870 0x7f12b40cff70 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:43.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.789+0000 7f12bb520700 1 -- 172.21.15.1:0/1423558840 >> 172.21.15.1:0/1423558840 conn(0x7f12b40d0ff0 msgr2=0x7f12b40d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:43.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.790+0000 7f12bb520700 1 -- 172.21.15.1:0/1423558840 shutdown_connections 2024-03-18T19:02:43.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.790+0000 7f12bb520700 1 -- 172.21.15.1:0/1423558840 wait complete. 2024-03-18T19:02:43.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.791+0000 7f12bb520700 1 Processor -- start 2024-03-18T19:02:43.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.791+0000 7f12bb520700 1 -- start start 2024-03-18T19:02:43.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.791+0000 7f12bb520700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b40b1e20 0x7f12b41f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:43.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12bb520700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12b40cbce0 0x7f12b415a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:43.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12bb520700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12b40cd870 0x7f12b415acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:43.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12bb520700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b41f7b20 con 0x7f12b40b1e20 2024-03-18T19:02:43.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12bb520700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b409cf10 con 0x7f12b40cd870 2024-03-18T19:02:43.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12bb520700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b409d1f0 con 0x7f12b40cbce0 2024-03-18T19:02:43.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12b92bc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b40b1e20 0x7f12b41f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:43.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12b92bc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b40b1e20 0x7f12b41f2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35418/0 (socket says 172.21.15.1:35418) 2024-03-18T19:02:43.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12b92bc700 1 -- 172.21.15.1:0/3639924213 learned_addr learned my addr 172.21.15.1:0/3639924213 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:43.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12b8abb700 1 --2- 172.21.15.1:0/3639924213 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12b40cbce0 0x7f12b415a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:43.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12b9abd700 1 --2- 172.21.15.1:0/3639924213 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12b40cd870 0x7f12b415acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:43.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12b92bc700 1 -- 172.21.15.1:0/3639924213 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12b40cbce0 msgr2=0x7f12b415a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:43.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12b92bc700 1 --2- 172.21.15.1:0/3639924213 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12b40cbce0 0x7f12b415a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:43.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12b92bc700 1 -- 172.21.15.1:0/3639924213 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12b40cd870 msgr2=0x7f12b415acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:43.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12b92bc700 1 --2- 172.21.15.1:0/3639924213 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12b40cd870 0x7f12b415acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:43.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12b92bc700 1 -- 172.21.15.1:0/3639924213 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b00094a0 con 0x7f12b40b1e20 2024-03-18T19:02:43.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12b92bc700 1 --2- 172.21.15.1:0/3639924213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b40b1e20 0x7f12b41f2f40 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f12b0014040 tx=0x7f12b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:43.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12a9ffb700 1 -- 172.21.15.1:0/3639924213 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b001a7e0 con 0x7f12b40b1e20 2024-03-18T19:02:43.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12a9ffb700 1 -- 172.21.15.1:0/3639924213 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b001a940 con 0x7f12b40b1e20 2024-03-18T19:02:43.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12a9ffb700 1 -- 172.21.15.1:0/3639924213 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b001acb0 con 0x7f12b40b1e20 2024-03-18T19:02:43.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.792+0000 7f12bb520700 1 -- 172.21.15.1:0/3639924213 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b409d4d0 con 0x7f12b40b1e20 2024-03-18T19:02:43.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.793+0000 7f12bb520700 1 -- 172.21.15.1:0/3639924213 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b41f3480 con 0x7f12b40b1e20 2024-03-18T19:02:43.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.794+0000 7f12bb520700 1 -- 172.21.15.1:0/3639924213 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b40cff70 con 0x7f12b40b1e20 2024-03-18T19:02:43.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.794+0000 7f12a9ffb700 1 -- 172.21.15.1:0/3639924213 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f12b0022070 con 0x7f12b40b1e20 2024-03-18T19:02:43.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.795+0000 7f12a9ffb700 1 --2- 172.21.15.1:0/3639924213 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f129806ae40 0x7f129806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:43.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.795+0000 7f12b8abb700 1 -- 172.21.15.1:0/3639924213 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f129806ae40 msgr2=0x7f129806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:43.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.795+0000 7f12a9ffb700 1 -- 172.21.15.1:0/3639924213 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6948+0+0 (secure 0 0 0) 0x7f12b008a810 con 0x7f12b40b1e20 2024-03-18T19:02:43.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.795+0000 7f12b8abb700 1 --2- 172.21.15.1:0/3639924213 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f129806ae40 0x7f129806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:02:43.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.799+0000 7f12a9ffb700 1 -- 172.21.15.1:0/3639924213 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f12b00554f0 con 0x7f12b40b1e20 2024-03-18T19:02:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.996+0000 7f12b8abb700 1 -- 172.21.15.1:0/3639924213 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f129806ae40 msgr2=0x7f129806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:43.996+0000 7f12b8abb700 1 --2- 172.21.15.1:0/3639924213 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f129806ae40 0x7f129806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:02:43.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.003+0000 7f12bb520700 1 -- 172.21.15.1:0/3639924213 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f12b40676a0 con 0x7f12b40b1e20 2024-03-18T19:02:43.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.003+0000 7f12a9ffb700 1 -- 172.21.15.1:0/3639924213 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7f12b0058b10 con 0x7f12b40b1e20 2024-03-18T19:02:43.999 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:02:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.005+0000 7f12bb520700 1 -- 172.21.15.1:0/3639924213 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f129806ae40 msgr2=0x7f129806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:02:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.005+0000 7f12bb520700 1 --2- 172.21.15.1:0/3639924213 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f129806ae40 0x7f129806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.005+0000 7f12bb520700 1 -- 172.21.15.1:0/3639924213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b40b1e20 msgr2=0x7f12b41f2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.005+0000 7f12bb520700 1 --2- 172.21.15.1:0/3639924213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b40b1e20 0x7f12b41f2f40 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f12b0014040 tx=0x7f12b0004750 comp rx=0 tx=0).stop 2024-03-18T19:02:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.005+0000 7f12bb520700 1 -- 172.21.15.1:0/3639924213 shutdown_connections 2024-03-18T19:02:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.005+0000 7f12bb520700 1 --2- 172.21.15.1:0/3639924213 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f129806ae40 0x7f129806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.005+0000 7f12bb520700 1 --2- 172.21.15.1:0/3639924213 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12b40cd870 0x7f12b415acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.005+0000 7f12bb520700 1 --2- 172.21.15.1:0/3639924213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12b40b1e20 0x7f12b41f2f40 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.005+0000 7f12bb520700 1 --2- 172.21.15.1:0/3639924213 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12b40cbce0 0x7f12b415a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.005+0000 7f12bb520700 1 -- 172.21.15.1:0/3639924213 >> 172.21.15.1:0/3639924213 conn(0x7f12b40d0ff0 msgr2=0x7f12b40c8340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:44.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.006+0000 7f12bb520700 1 -- 172.21.15.1:0/3639924213 shutdown_connections 2024-03-18T19:02:44.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.006+0000 7f12bb520700 1 -- 172.21.15.1:0/3639924213 wait complete. 2024-03-18T19:02:44.015 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T19:02:44.076 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:44.080+0000 7f5e46aab240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:02:44.158 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:44.162+0000 7fdb7645b240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:02:44.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.189+0000 7f046be13700 1 Processor -- start 2024-03-18T19:02:44.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.189+0000 7f046be13700 1 -- start start 2024-03-18T19:02:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.190+0000 7f046be13700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f046415b6c0 0x7f04640b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.190+0000 7f046be13700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f04640c87d0 0x7f04640cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.190+0000 7f046be13700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04640d01f0 con 0x7f04640cd430 2024-03-18T19:02:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.190+0000 7f046be13700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04640ccfa0 con 0x7f046415b6c0 2024-03-18T19:02:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.190+0000 7f046be13700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04640d1db0 con 0x7f04640c87d0 2024-03-18T19:02:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.190+0000 7f0469baf700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f046415b6c0 0x7f04640b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.190+0000 7f0469baf700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f046415b6c0 0x7f04640b2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35426/0 (socket says 172.21.15.1:35426) 2024-03-18T19:02:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.190+0000 7f0469baf700 1 -- 172.21.15.1:0/453558950 learned_addr learned my addr 172.21.15.1:0/453558950 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.190+0000 7f046a3b0700 1 --2- 172.21.15.1:0/453558950 >> v2:172.21.15.165:3300/0 conn(0x7f04640c87d0 0x7f04640cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.190+0000 7f0469baf700 1 -- 172.21.15.1:0/453558950 >> v2:172.21.15.165:3300/0 conn(0x7f04640c87d0 msgr2=0x7f04640cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.190+0000 7f0469baf700 1 --2- 172.21.15.1:0/453558950 >> v2:172.21.15.165:3300/0 conn(0x7f04640c87d0 0x7f04640cd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.190+0000 7f0469baf700 1 -- 172.21.15.1:0/453558950 >> v1:172.21.15.165:6789/0 conn(0x7f04640cd430 legacy=0x7f04640c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.190+0000 7f0469baf700 1 -- 172.21.15.1:0/453558950 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04640cff10 con 0x7f046415b6c0 2024-03-18T19:02:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.191+0000 7f0469baf700 1 --2- 172.21.15.1:0/453558950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f046415b6c0 0x7f04640b2160 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f04640d0f90 tx=0x7f0460009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c569eca044db04c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.191+0000 7f0468bad700 1 -- 172.21.15.1:0/453558950 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0460011830 con 0x7f046415b6c0 2024-03-18T19:02:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.191+0000 7f0468bad700 1 -- 172.21.15.1:0/453558950 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0460011990 con 0x7f046415b6c0 2024-03-18T19:02:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.191+0000 7f0468bad700 1 -- 172.21.15.1:0/453558950 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0460011d20 con 0x7f046415b6c0 2024-03-18T19:02:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.191+0000 7f046be13700 1 -- 172.21.15.1:0/453558950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f046415b6c0 msgr2=0x7f04640b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.191+0000 7f046be13700 1 --2- 172.21.15.1:0/453558950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f046415b6c0 0x7f04640b2160 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f04640d0f90 tx=0x7f0460009bd0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.191+0000 7f046be13700 1 -- 172.21.15.1:0/453558950 shutdown_connections 2024-03-18T19:02:44.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.191+0000 7f046be13700 1 --2- 172.21.15.1:0/453558950 >> v2:172.21.15.165:3300/0 conn(0x7f04640c87d0 0x7f04640cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.191+0000 7f046be13700 1 --2- 172.21.15.1:0/453558950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f046415b6c0 0x7f04640b2160 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.191+0000 7f046be13700 1 -- 172.21.15.1:0/453558950 >> 172.21.15.1:0/453558950 conn(0x7f04640d0ff0 msgr2=0x7f04640d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:44.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.192+0000 7f046be13700 1 -- 172.21.15.1:0/453558950 shutdown_connections 2024-03-18T19:02:44.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.192+0000 7f046be13700 1 -- 172.21.15.1:0/453558950 wait complete. 2024-03-18T19:02:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.193+0000 7f046be13700 1 Processor -- start 2024-03-18T19:02:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.193+0000 7f046be13700 1 -- start start 2024-03-18T19:02:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.193+0000 7f046be13700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04640c87d0 0x7f04640b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.193+0000 7f046be13700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04640cd430 0x7f0464159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.193+0000 7f046be13700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f046415b6c0 0x7f04640b3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.193+0000 7f046be13700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046409e990 con 0x7f04640c87d0 2024-03-18T19:02:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.193+0000 7f046be13700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04640b3000 con 0x7f046415b6c0 2024-03-18T19:02:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.193+0000 7f046be13700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0464159e50 con 0x7f04640cd430 2024-03-18T19:02:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.194+0000 7f0469baf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04640c87d0 0x7f04640b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.194+0000 7f0469baf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04640c87d0 0x7f04640b2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35434/0 (socket says 172.21.15.1:35434) 2024-03-18T19:02:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.194+0000 7f0469baf700 1 -- 172.21.15.1:0/2712162636 learned_addr learned my addr 172.21.15.1:0/2712162636 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.194+0000 7f046a3b0700 1 --2- 172.21.15.1:0/2712162636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f046415b6c0 0x7f04640b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.194+0000 7f04693ae700 1 --2- 172.21.15.1:0/2712162636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04640cd430 0x7f0464159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.194+0000 7f0469baf700 1 -- 172.21.15.1:0/2712162636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04640cd430 msgr2=0x7f0464159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.194+0000 7f0469baf700 1 --2- 172.21.15.1:0/2712162636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04640cd430 0x7f0464159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.194+0000 7f0469baf700 1 -- 172.21.15.1:0/2712162636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f046415b6c0 msgr2=0x7f04640b3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.194+0000 7f0469baf700 1 --2- 172.21.15.1:0/2712162636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f046415b6c0 0x7f04640b3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.194+0000 7f0469baf700 1 -- 172.21.15.1:0/2712162636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0460009550 con 0x7f04640c87d0 2024-03-18T19:02:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.194+0000 7f0469baf700 1 --2- 172.21.15.1:0/2712162636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04640c87d0 0x7f04640b2ac0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f0460009c10 tx=0x7f0460004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.194+0000 7f045a7fc700 1 -- 172.21.15.1:0/2712162636 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0460004450 con 0x7f04640c87d0 2024-03-18T19:02:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.194+0000 7f045a7fc700 1 -- 172.21.15.1:0/2712162636 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046001e070 con 0x7f04640c87d0 2024-03-18T19:02:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.194+0000 7f046be13700 1 -- 172.21.15.1:0/2712162636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f046409dd80 con 0x7f04640c87d0 2024-03-18T19:02:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.194+0000 7f045a7fc700 1 -- 172.21.15.1:0/2712162636 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04600038c0 con 0x7f04640c87d0 2024-03-18T19:02:44.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.194+0000 7f046be13700 1 -- 172.21.15.1:0/2712162636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f046409e340 con 0x7f04640c87d0 2024-03-18T19:02:44.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.195+0000 7f046be13700 1 -- 172.21.15.1:0/2712162636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04640676a0 con 0x7f04640c87d0 2024-03-18T19:02:44.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.196+0000 7f045a7fc700 1 -- 172.21.15.1:0/2712162636 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f0460003a20 con 0x7f04640c87d0 2024-03-18T19:02:44.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.196+0000 7f045a7fc700 1 --2- 172.21.15.1:0/2712162636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f044806ae40 0x7f044806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:44.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.196+0000 7f04693ae700 1 -- 172.21.15.1:0/2712162636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f044806ae40 msgr2=0x7f044806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:44.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.196+0000 7f045a7fc700 1 -- 172.21.15.1:0/2712162636 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6884+0+0 (secure 0 0 0) 0x7f046008a8c0 con 0x7f04640c87d0 2024-03-18T19:02:44.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.196+0000 7f04693ae700 1 --2- 172.21.15.1:0/2712162636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f044806ae40 0x7f044806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:02:44.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.201+0000 7f045a7fc700 1 -- 172.21.15.1:0/2712162636 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0460055390 con 0x7f04640c87d0 2024-03-18T19:02:44.289 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:44.292+0000 7f5e46aab240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:02:44.380 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:44.384+0000 7fdb7645b240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:02:44.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.397+0000 7f04693ae700 1 -- 172.21.15.1:0/2712162636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f044806ae40 msgr2=0x7f044806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:44.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.397+0000 7f04693ae700 1 --2- 172.21.15.1:0/2712162636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f044806ae40 0x7f044806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:02:44.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.406+0000 7f046be13700 1 -- 172.21.15.1:0/2712162636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0464070280 con 0x7f04640c87d0 2024-03-18T19:02:44.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.407+0000 7f045a7fc700 1 -- 172.21.15.1:0/2712162636 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) v1 ==== 104+0+11 (secure 0 0 0) 0x7f04600045b0 con 0x7f04640c87d0 2024-03-18T19:02:44.403 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T19:02:44.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.409+0000 7f046be13700 1 -- 172.21.15.1:0/2712162636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f044806ae40 msgr2=0x7f044806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:02:44.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.409+0000 7f046be13700 1 --2- 172.21.15.1:0/2712162636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f044806ae40 0x7f044806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.409+0000 7f046be13700 1 -- 172.21.15.1:0/2712162636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04640c87d0 msgr2=0x7f04640b2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:44.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.409+0000 7f046be13700 1 --2- 172.21.15.1:0/2712162636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04640c87d0 0x7f04640b2ac0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f0460009c10 tx=0x7f0460004750 comp rx=0 tx=0).stop 2024-03-18T19:02:44.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.409+0000 7f046be13700 1 -- 172.21.15.1:0/2712162636 shutdown_connections 2024-03-18T19:02:44.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.409+0000 7f046be13700 1 --2- 172.21.15.1:0/2712162636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f044806ae40 0x7f044806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.409+0000 7f046be13700 1 --2- 172.21.15.1:0/2712162636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f046415b6c0 0x7f04640b3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.409+0000 7f046be13700 1 --2- 172.21.15.1:0/2712162636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04640c87d0 0x7f04640b2ac0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.409+0000 7f046be13700 1 --2- 172.21.15.1:0/2712162636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04640cd430 0x7f0464159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.409+0000 7f046be13700 1 -- 172.21.15.1:0/2712162636 >> 172.21.15.1:0/2712162636 conn(0x7f04640d0ff0 msgr2=0x7f04640c9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:44.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.409+0000 7f046be13700 1 -- 172.21.15.1:0/2712162636 shutdown_connections 2024-03-18T19:02:44.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.409+0000 7f046be13700 1 -- 172.21.15.1:0/2712162636 wait complete. 2024-03-18T19:02:44.417 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T19:02:44.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.558+0000 7f298a39d700 1 Processor -- start 2024-03-18T19:02:44.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.558+0000 7f298a39d700 1 -- start start 2024-03-18T19:02:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.559+0000 7f298a39d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29840cecb0 0x7f29840cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.559+0000 7f298a39d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29840ce090 con 0x7f298409dd40 2024-03-18T19:02:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.559+0000 7f298a39d700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298406db30 con 0x7f298415d780 2024-03-18T19:02:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.559+0000 7f298a39d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29840cf120 con 0x7f29840cecb0 2024-03-18T19:02:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.559+0000 7f29837fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f298409dd40 0x7f29840ce5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37328/0 (socket says 172.21.15.1:37328) 2024-03-18T19:02:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.559+0000 7f29837fe700 1 -- 172.21.15.1:0/1791024333 learned_addr learned my addr 172.21.15.1:0/1791024333 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.560+0000 7f298893a700 1 --2- 172.21.15.1:0/1791024333 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29840cecb0 0x7f29840cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.560+0000 7f2982ffd700 1 -- 172.21.15.1:0/1791024333 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015541134 0 0) 0x7f29840ce090 con 0x7f298409dd40 2024-03-18T19:02:44.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.560+0000 7f2982ffd700 1 -- 172.21.15.1:0/1791024333 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f296c0035b0 con 0x7f298409dd40 2024-03-18T19:02:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.560+0000 7f2982ffd700 1 -- 172.21.15.1:0/1791024333 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274294217 0 0) 0x7f298406db30 con 0x7f298415d780 2024-03-18T19:02:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.560+0000 7f2982ffd700 1 -- 172.21.15.1:0/1791024333 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29840ce090 con 0x7f298415d780 2024-03-18T19:02:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.560+0000 7f298893a700 1 -- 172.21.15.1:0/1791024333 >> v1:172.21.15.165:6789/0 conn(0x7f298415d780 legacy=0x7f29840cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.560+0000 7f298893a700 1 -- 172.21.15.1:0/1791024333 >> v1:172.21.15.1:6789/0 conn(0x7f298409dd40 legacy=0x7f29840ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.560+0000 7f298893a700 1 -- 172.21.15.1:0/1791024333 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298406d520 con 0x7f29840cecb0 2024-03-18T19:02:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.560+0000 7f298893a700 1 --2- 172.21.15.1:0/1791024333 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29840cecb0 0x7f29840cdb50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f297c009dc0 tx=0x7f297c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a977574b0e53cf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.560+0000 7f2982ffd700 1 -- 172.21.15.1:0/1791024333 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297c00e070 con 0x7f29840cecb0 2024-03-18T19:02:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.560+0000 7f2982ffd700 1 -- 172.21.15.1:0/1791024333 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f297c00bb20 con 0x7f29840cecb0 2024-03-18T19:02:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.560+0000 7f2982ffd700 1 -- 172.21.15.1:0/1791024333 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297c012430 con 0x7f29840cecb0 2024-03-18T19:02:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.561+0000 7f298a39d700 1 -- 172.21.15.1:0/1791024333 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29840cecb0 msgr2=0x7f29840cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.561+0000 7f298a39d700 1 --2- 172.21.15.1:0/1791024333 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29840cecb0 0x7f29840cdb50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f297c009dc0 tx=0x7f297c00b330 comp rx=0 tx=0).stop 2024-03-18T19:02:44.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.561+0000 7f298a39d700 1 -- 172.21.15.1:0/1791024333 shutdown_connections 2024-03-18T19:02:44.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.561+0000 7f298a39d700 1 --2- 172.21.15.1:0/1791024333 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29840cecb0 0x7f29840cdb50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.561+0000 7f298a39d700 1 -- 172.21.15.1:0/1791024333 >> 172.21.15.1:0/1791024333 conn(0x7f29840cca00 msgr2=0x7f29840cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:44.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.561+0000 7f298a39d700 1 -- 172.21.15.1:0/1791024333 shutdown_connections 2024-03-18T19:02:44.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.561+0000 7f298a39d700 1 -- 172.21.15.1:0/1791024333 wait complete. 2024-03-18T19:02:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.562+0000 7f298a39d700 1 Processor -- start 2024-03-18T19:02:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.562+0000 7f298a39d700 1 -- start start 2024-03-18T19:02:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.563+0000 7f298a39d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f298409dd40 0x7f298415b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.563+0000 7f298a39d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29840cecb0 0x7f29840c4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.563+0000 7f298a39d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f298415d780 0x7f2984099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.563+0000 7f298a39d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29840c59a0 con 0x7f298415d780 2024-03-18T19:02:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.563+0000 7f298a39d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29840c4d90 con 0x7f29840cecb0 2024-03-18T19:02:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.563+0000 7f298a39d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29840c5070 con 0x7f298409dd40 2024-03-18T19:02:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.563+0000 7f298893a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f298415d780 0x7f2984099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.563+0000 7f2983fff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f298409dd40 0x7f298415b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.563+0000 7f298893a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f298415d780 0x7f2984099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34414/0 (socket says 172.21.15.1:34414) 2024-03-18T19:02:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.563+0000 7f298893a700 1 -- 172.21.15.1:0/787153345 learned_addr learned my addr 172.21.15.1:0/787153345 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.563+0000 7f29837fe700 1 --2- 172.21.15.1:0/787153345 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29840cecb0 0x7f29840c4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.564+0000 7f2983fff700 1 -- 172.21.15.1:0/787153345 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29840cecb0 msgr2=0x7f29840c4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.564+0000 7f2983fff700 1 --2- 172.21.15.1:0/787153345 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29840cecb0 0x7f29840c4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.564+0000 7f2983fff700 1 -- 172.21.15.1:0/787153345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f298415d780 msgr2=0x7f2984099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.564+0000 7f2983fff700 1 --2- 172.21.15.1:0/787153345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f298415d780 0x7f2984099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.564+0000 7f2983fff700 1 -- 172.21.15.1:0/787153345 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297c009a70 con 0x7f298409dd40 2024-03-18T19:02:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.564+0000 7f2983fff700 1 --2- 172.21.15.1:0/787153345 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f298409dd40 0x7f298415b070 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f297400c3c0 tx=0x7f297400c6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.564+0000 7f2980ff9700 1 -- 172.21.15.1:0/787153345 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29740052d0 con 0x7f298409dd40 2024-03-18T19:02:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.564+0000 7f2980ff9700 1 -- 172.21.15.1:0/787153345 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2974005430 con 0x7f298409dd40 2024-03-18T19:02:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.564+0000 7f298a39d700 1 -- 172.21.15.1:0/787153345 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29840c5350 con 0x7f298409dd40 2024-03-18T19:02:44.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.564+0000 7f2980ff9700 1 -- 172.21.15.1:0/787153345 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29740038c0 con 0x7f298409dd40 2024-03-18T19:02:44.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.564+0000 7f298a39d700 1 -- 172.21.15.1:0/787153345 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29840c4aa0 con 0x7f298409dd40 2024-03-18T19:02:44.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.565+0000 7f298a39d700 1 -- 172.21.15.1:0/787153345 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29840676a0 con 0x7f298409dd40 2024-03-18T19:02:44.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.566+0000 7f2980ff9700 1 -- 172.21.15.1:0/787153345 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f2974003a20 con 0x7f298409dd40 2024-03-18T19:02:44.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.566+0000 7f2980ff9700 1 --2- 172.21.15.1:0/787153345 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f297006ae40 0x7f297006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:44.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.566+0000 7f29837fe700 1 -- 172.21.15.1:0/787153345 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f297006ae40 msgr2=0x7f297006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:44.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.566+0000 7f2980ff9700 1 -- 172.21.15.1:0/787153345 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6884+0+0 (secure 0 0 0) 0x7f2974088dc0 con 0x7f298409dd40 2024-03-18T19:02:44.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.566+0000 7f29837fe700 1 --2- 172.21.15.1:0/787153345 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f297006ae40 0x7f297006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:02:44.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.571+0000 7f2980ff9700 1 -- 172.21.15.1:0/787153345 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2974053940 con 0x7f298409dd40 2024-03-18T19:02:44.668 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:44.671+0000 7f5e46aab240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:02:44.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.767+0000 7f29837fe700 1 -- 172.21.15.1:0/787153345 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f297006ae40 msgr2=0x7f297006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:44.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.767+0000 7f29837fe700 1 --2- 172.21.15.1:0/787153345 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f297006ae40 0x7f297006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:02:44.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.775+0000 7f298a39d700 1 -- 172.21.15.1:0/787153345 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f29840c6b10 con 0x7f298409dd40 2024-03-18T19:02:44.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.775+0000 7f2980ff9700 1 -- 172.21.15.1:0/787153345 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) v1 ==== 100+0+11 (secure 0 0 0) 0x7f29740170a0 con 0x7f298409dd40 2024-03-18T19:02:44.771 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T19:02:44.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.777+0000 7f298a39d700 1 -- 172.21.15.1:0/787153345 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f297006ae40 msgr2=0x7f297006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:02:44.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.777+0000 7f298a39d700 1 --2- 172.21.15.1:0/787153345 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f297006ae40 0x7f297006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.777+0000 7f298a39d700 1 -- 172.21.15.1:0/787153345 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f298409dd40 msgr2=0x7f298415b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:44.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.777+0000 7f298a39d700 1 --2- 172.21.15.1:0/787153345 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f298409dd40 0x7f298415b070 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f297400c3c0 tx=0x7f297400c6d0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.777+0000 7f298a39d700 1 -- 172.21.15.1:0/787153345 shutdown_connections 2024-03-18T19:02:44.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.777+0000 7f298a39d700 1 --2- 172.21.15.1:0/787153345 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f297006ae40 0x7f297006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.777+0000 7f298a39d700 1 --2- 172.21.15.1:0/787153345 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f298409dd40 0x7f298415b070 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.777+0000 7f298a39d700 1 --2- 172.21.15.1:0/787153345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f298415d780 0x7f2984099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.777+0000 7f298a39d700 1 --2- 172.21.15.1:0/787153345 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29840cecb0 0x7f29840c4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.777+0000 7f298a39d700 1 -- 172.21.15.1:0/787153345 >> 172.21.15.1:0/787153345 conn(0x7f29840cca00 msgr2=0x7f2984071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:44.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.778+0000 7f298a39d700 1 -- 172.21.15.1:0/787153345 shutdown_connections 2024-03-18T19:02:44.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.778+0000 7f298a39d700 1 -- 172.21.15.1:0/787153345 wait complete. 2024-03-18T19:02:44.781 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:44.786+0000 7fdb7645b240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:02:44.786 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T19:02:44.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.918+0000 7fc0acf2d700 1 Processor -- start 2024-03-18T19:02:44.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.918+0000 7fc0acf2d700 1 -- start start 2024-03-18T19:02:44.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.919+0000 7fc0acf2d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0a806f180 0x7fc0a8161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:44.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.919+0000 7fc0acf2d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a80cad60 con 0x7fc0a806eda0 2024-03-18T19:02:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.919+0000 7fc0acf2d700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a80cb040 con 0x7fc0a806a750 2024-03-18T19:02:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.919+0000 7fc0acf2d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a80cb320 con 0x7fc0a806f180 2024-03-18T19:02:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.919+0000 7fc0a659c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc0a806eda0 0x7fc0a80ca580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37340/0 (socket says 172.21.15.1:37340) 2024-03-18T19:02:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.919+0000 7fc0a659c700 1 -- 172.21.15.1:0/2900343447 learned_addr learned my addr 172.21.15.1:0/2900343447 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.919+0000 7fc0a5d9b700 1 --2- 172.21.15.1:0/2900343447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0a806f180 0x7fc0a8161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.919+0000 7fc0a559a700 1 -- 172.21.15.1:0/2900343447 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362288399 0 0) 0x7fc0a80cad60 con 0x7fc0a806eda0 2024-03-18T19:02:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.919+0000 7fc0a559a700 1 -- 172.21.15.1:0/2900343447 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0900035b0 con 0x7fc0a806eda0 2024-03-18T19:02:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.919+0000 7fc0a559a700 1 -- 172.21.15.1:0/2900343447 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761269587 0 0) 0x7fc0a80cb040 con 0x7fc0a806a750 2024-03-18T19:02:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.919+0000 7fc0a5d9b700 1 -- 172.21.15.1:0/2900343447 >> v1:172.21.15.165:6789/0 conn(0x7fc0a806a750 legacy=0x7fc0a815cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.919+0000 7fc0a5d9b700 1 -- 172.21.15.1:0/2900343447 >> v1:172.21.15.1:6789/0 conn(0x7fc0a806eda0 legacy=0x7fc0a80ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.919+0000 7fc0a5d9b700 1 -- 172.21.15.1:0/2900343447 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a80ce220 con 0x7fc0a806f180 2024-03-18T19:02:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.920+0000 7fc0a5d9b700 1 --2- 172.21.15.1:0/2900343447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0a806f180 0x7fc0a8161c60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc09c009870 tx=0x7fc09c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23604e96020e20f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.920+0000 7fc0a559a700 1 -- 172.21.15.1:0/2900343447 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c011830 con 0x7fc0a806f180 2024-03-18T19:02:44.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.920+0000 7fc0a559a700 1 -- 172.21.15.1:0/2900343447 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09c011990 con 0x7fc0a806f180 2024-03-18T19:02:44.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.920+0000 7fc0a559a700 1 -- 172.21.15.1:0/2900343447 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c011d20 con 0x7fc0a806f180 2024-03-18T19:02:44.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.920+0000 7fc0acf2d700 1 -- 172.21.15.1:0/2900343447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0a806f180 msgr2=0x7fc0a8161c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:44.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.920+0000 7fc0acf2d700 1 --2- 172.21.15.1:0/2900343447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0a806f180 0x7fc0a8161c60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc09c009870 tx=0x7fc09c009b80 comp rx=0 tx=0).stop 2024-03-18T19:02:44.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.920+0000 7fc0acf2d700 1 -- 172.21.15.1:0/2900343447 shutdown_connections 2024-03-18T19:02:44.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.920+0000 7fc0acf2d700 1 --2- 172.21.15.1:0/2900343447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0a806f180 0x7fc0a8161c60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.920+0000 7fc0acf2d700 1 -- 172.21.15.1:0/2900343447 >> 172.21.15.1:0/2900343447 conn(0x7fc0a80cff30 msgr2=0x7fc0a80d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:44.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.921+0000 7fc0acf2d700 1 -- 172.21.15.1:0/2900343447 shutdown_connections 2024-03-18T19:02:44.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.921+0000 7fc0acf2d700 1 -- 172.21.15.1:0/2900343447 wait complete. 2024-03-18T19:02:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.922+0000 7fc0acf2d700 1 Processor -- start 2024-03-18T19:02:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.922+0000 7fc0acf2d700 1 -- start start 2024-03-18T19:02:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.923+0000 7fc0acf2d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0a806a750 0x7fc0a80c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.923+0000 7fc0acf2d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0a806eda0 0x7fc0a814ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.923+0000 7fc0acf2d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0a806f180 0x7fc0a80b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.923+0000 7fc0acf2d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a80c5e20 con 0x7fc0a806eda0 2024-03-18T19:02:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.923+0000 7fc0acf2d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a8151950 con 0x7fc0a806a750 2024-03-18T19:02:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.923+0000 7fc0acf2d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a8151c30 con 0x7fc0a806f180 2024-03-18T19:02:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.923+0000 7fc0a5d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0a806eda0 0x7fc0a814ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.923+0000 7fc0a5d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0a806eda0 0x7fc0a814ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34416/0 (socket says 172.21.15.1:34416) 2024-03-18T19:02:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.923+0000 7fc0a5d9b700 1 -- 172.21.15.1:0/2508719965 learned_addr learned my addr 172.21.15.1:0/2508719965 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.923+0000 7fc0a659c700 1 --2- 172.21.15.1:0/2508719965 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0a806a750 0x7fc0a80c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.923+0000 7fc0a6d9d700 1 --2- 172.21.15.1:0/2508719965 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0a806f180 0x7fc0a80b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.923+0000 7fc0a5d9b700 1 -- 172.21.15.1:0/2508719965 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0a806f180 msgr2=0x7fc0a80b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.923+0000 7fc0a5d9b700 1 --2- 172.21.15.1:0/2508719965 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0a806f180 0x7fc0a80b42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.923+0000 7fc0a5d9b700 1 -- 172.21.15.1:0/2508719965 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0a806a750 msgr2=0x7fc0a80c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.923+0000 7fc0a5d9b700 1 --2- 172.21.15.1:0/2508719965 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0a806a750 0x7fc0a80c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.923+0000 7fc0a5d9b700 1 -- 172.21.15.1:0/2508719965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c0094a0 con 0x7fc0a806eda0 2024-03-18T19:02:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.923+0000 7fc0a5d9b700 1 --2- 172.21.15.1:0/2508719965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0a806eda0 0x7fc0a814ed00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc09c014040 tx=0x7fc09c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.924+0000 7fc096ffd700 1 -- 172.21.15.1:0/2508719965 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c01a7e0 con 0x7fc0a806eda0 2024-03-18T19:02:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.924+0000 7fc096ffd700 1 -- 172.21.15.1:0/2508719965 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09c01a940 con 0x7fc0a806eda0 2024-03-18T19:02:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.924+0000 7fc0acf2d700 1 -- 172.21.15.1:0/2508719965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a8151f10 con 0x7fc0a806eda0 2024-03-18T19:02:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.924+0000 7fc096ffd700 1 -- 172.21.15.1:0/2508719965 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c01acd0 con 0x7fc0a806eda0 2024-03-18T19:02:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.924+0000 7fc0acf2d700 1 -- 172.21.15.1:0/2508719965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a814f240 con 0x7fc0a806eda0 2024-03-18T19:02:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.925+0000 7fc0acf2d700 1 -- 172.21.15.1:0/2508719965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a80cbff0 con 0x7fc0a806eda0 2024-03-18T19:02:44.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.926+0000 7fc096ffd700 1 -- 172.21.15.1:0/2508719965 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7fc09c022070 con 0x7fc0a806eda0 2024-03-18T19:02:44.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.926+0000 7fc096ffd700 1 --2- 172.21.15.1:0/2508719965 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc08806ae40 0x7fc08806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:44.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.926+0000 7fc0a659c700 1 -- 172.21.15.1:0/2508719965 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc08806ae40 msgr2=0x7fc08806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:44.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.926+0000 7fc096ffd700 1 -- 172.21.15.1:0/2508719965 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6884+0+0 (secure 0 0 0) 0x7fc09c08a9d0 con 0x7fc0a806eda0 2024-03-18T19:02:44.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.926+0000 7fc0a659c700 1 --2- 172.21.15.1:0/2508719965 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc08806ae40 0x7fc08806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:02:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:44.931+0000 7fc096ffd700 1 -- 172.21.15.1:0/2508719965 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc09c055520 con 0x7fc0a806eda0 2024-03-18T19:02:45.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.127+0000 7fc0a659c700 1 -- 172.21.15.1:0/2508719965 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc08806ae40 msgr2=0x7fc08806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:45.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.127+0000 7fc0a659c700 1 --2- 172.21.15.1:0/2508719965 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc08806ae40 0x7fc08806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:02:45.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.150+0000 7fc0acf2d700 1 -- 172.21.15.1:0/2508719965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc0a8161040 con 0x7fc0a806eda0 2024-03-18T19:02:45.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.150+0000 7fc096ffd700 1 -- 172.21.15.1:0/2508719965 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v57) v1 ==== 80+0+336 (secure 0 0 0) 0x7fc09c058b40 con 0x7fc0a806eda0 2024-03-18T19:02:45.146 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1992330207 2024-03-19T18:51:27.537808+0000 2024-03-18T19:02:45.146 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6832/76998 2024-03-19T18:51:27.537808+0000 2024-03-18T19:02:45.146 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/4026407635 2024-03-19T18:51:27.537808+0000 2024-03-18T19:02:45.146 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2322425425 2024-03-19T18:51:27.537808+0000 2024-03-18T19:02:45.146 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6833/76998 2024-03-19T18:51:27.537808+0000 2024-03-18T19:02:45.147 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/72803206 2024-03-19T18:51:27.537808+0000 2024-03-18T19:02:45.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.153+0000 7fc0acf2d700 1 -- 172.21.15.1:0/2508719965 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc08806ae40 msgr2=0x7fc08806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:02:45.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.153+0000 7fc0acf2d700 1 --2- 172.21.15.1:0/2508719965 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc08806ae40 0x7fc08806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:45.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.153+0000 7fc0acf2d700 1 -- 172.21.15.1:0/2508719965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0a806eda0 msgr2=0x7fc0a814ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:45.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.153+0000 7fc0acf2d700 1 --2- 172.21.15.1:0/2508719965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0a806eda0 0x7fc0a814ed00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc09c014040 tx=0x7fc09c004750 comp rx=0 tx=0).stop 2024-03-18T19:02:45.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.153+0000 7fc0acf2d700 1 -- 172.21.15.1:0/2508719965 shutdown_connections 2024-03-18T19:02:45.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.153+0000 7fc0acf2d700 1 --2- 172.21.15.1:0/2508719965 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc08806ae40 0x7fc08806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:45.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.153+0000 7fc0acf2d700 1 --2- 172.21.15.1:0/2508719965 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0a806f180 0x7fc0a80b42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:45.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.153+0000 7fc0acf2d700 1 --2- 172.21.15.1:0/2508719965 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0a806a750 0x7fc0a80c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:45.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.153+0000 7fc0acf2d700 1 --2- 172.21.15.1:0/2508719965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0a806eda0 0x7fc0a814ed00 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:45.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.153+0000 7fc0acf2d700 1 -- 172.21.15.1:0/2508719965 >> 172.21.15.1:0/2508719965 conn(0x7fc0a80cff30 msgr2=0x7fc0a806dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:45.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.153+0000 7fc0acf2d700 1 -- 172.21.15.1:0/2508719965 shutdown_connections 2024-03-18T19:02:45.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.153+0000 7fc0acf2d700 1 -- 172.21.15.1:0/2508719965 wait complete. 2024-03-18T19:02:45.150 INFO:teuthology.orchestra.run.smithi001.stderr:listed 6 entries 2024-03-18T19:02:45.161 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-18T19:02:45.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:45.161 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T19:02:45.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[95919]: fuse finished with error 0 and tester_r 0 2024-03-18T19:02:45.196 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T19:02:45.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:45.196 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T19:02:45.253 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:45.257+0000 7f5e46aab240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:02:45.325 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:45.328+0000 7f5e46aab240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:02:45.369 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:02:45.369 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:02:45.388 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:02:45.388 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T19:02:45.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:45.388 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-18T19:02:45.414 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:45.418+0000 7fdb7645b240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:02:45.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:02:45.446 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-18T19:02:45.446 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:02:45.446 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:02:45.456 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:45.459+0000 7f5e46aab240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:02:45.465 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:02:45.466 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T19:02:45.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:45.466 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-18T19:02:45.489 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:45.494+0000 7fdb7645b240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:02:45.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:02:45.506 INFO:teuthology.orchestra.run.smithi001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-18T19:02:45.507 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:02:45.507 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:02:45.525 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:02:45.525 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T19:02:45.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:02:45.525 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-18T19:02:45.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:02:45.583 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-18T19:02:45.583 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:02:45.628 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:45.632+0000 7fdb7645b240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:02:45.639 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:45.642+0000 7f5e46aab240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:02:45.710 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:45.714+0000 7f5e46aab240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:02:45.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.717+0000 7fb95344f700 1 Processor -- start 2024-03-18T19:02:45.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.717+0000 7fb95344f700 1 -- start start 2024-03-18T19:02:45.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.718+0000 7fb95344f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb94c0716b0 0x7fb94c071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:45.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.718+0000 7fb95344f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb94c0ae580 0x7fb94c06d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:45.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.718+0000 7fb95344f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94c071fd0 con 0x7fb94c0abd70 2024-03-18T19:02:45.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.718+0000 7fb95344f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb94c0ac230 con 0x7fb94c0716b0 2024-03-18T19:02:45.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.718+0000 7fb95344f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c06c620 con 0x7fb94c0ae580 2024-03-18T19:02:45.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.718+0000 7fb9511eb700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb94c0716b0 0x7fb94c071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:45.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.718+0000 7fb9519ec700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb94c0abd70 0x7fb94c06d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37348/0 (socket says 172.21.15.1:37348) 2024-03-18T19:02:45.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.718+0000 7fb9511eb700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb94c0716b0 0x7fb94c071a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34424/0 (socket says 172.21.15.1:34424) 2024-03-18T19:02:45.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.718+0000 7fb9519ec700 1 -- 172.21.15.1:0/2411280903 learned_addr learned my addr 172.21.15.1:0/2411280903 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:45.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.718+0000 7fb9509ea700 1 --2- 172.21.15.1:0/2411280903 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb94c0ae580 0x7fb94c06d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:45.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.718+0000 7fb9511eb700 1 -- 172.21.15.1:0/2411280903 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb94c0ae580 msgr2=0x7fb94c06d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:45.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.718+0000 7fb9511eb700 1 --2- 172.21.15.1:0/2411280903 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb94c0ae580 0x7fb94c06d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:45.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.718+0000 7fb9511eb700 1 -- 172.21.15.1:0/2411280903 >> v1:172.21.15.1:6789/0 conn(0x7fb94c0abd70 legacy=0x7fb94c06d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:45.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.718+0000 7fb9511eb700 1 -- 172.21.15.1:0/2411280903 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c06c900 con 0x7fb94c0716b0 2024-03-18T19:02:45.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.718+0000 7fb943fff700 1 -- 172.21.15.1:0/2411280903 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383308288 0 0) 0x7fb94c071fd0 con 0x7fb94c0abd70 2024-03-18T19:02:45.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.718+0000 7fb9511eb700 1 --2- 172.21.15.1:0/2411280903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb94c0716b0 0x7fb94c071a90 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb948009840 tx=0x7fb948009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=696c9b850cfbb7d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:45.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.718+0000 7fb943fff700 1 -- 172.21.15.1:0/2411280903 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb948011830 con 0x7fb94c0716b0 2024-03-18T19:02:45.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.719+0000 7fb943fff700 1 -- 172.21.15.1:0/2411280903 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb948011990 con 0x7fb94c0716b0 2024-03-18T19:02:45.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.719+0000 7fb943fff700 1 -- 172.21.15.1:0/2411280903 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb948011d20 con 0x7fb94c0716b0 2024-03-18T19:02:45.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.719+0000 7fb95344f700 1 -- 172.21.15.1:0/2411280903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb94c0716b0 msgr2=0x7fb94c071a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:45.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.719+0000 7fb95344f700 1 --2- 172.21.15.1:0/2411280903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb94c0716b0 0x7fb94c071a90 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb948009840 tx=0x7fb948009b50 comp rx=0 tx=0).stop 2024-03-18T19:02:45.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.719+0000 7fb95344f700 1 -- 172.21.15.1:0/2411280903 shutdown_connections 2024-03-18T19:02:45.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.719+0000 7fb95344f700 1 --2- 172.21.15.1:0/2411280903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb94c0716b0 0x7fb94c071a90 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:45.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.719+0000 7fb95344f700 1 --2- 172.21.15.1:0/2411280903 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb94c0ae580 0x7fb94c06d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:45.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.719+0000 7fb95344f700 1 -- 172.21.15.1:0/2411280903 >> 172.21.15.1:0/2411280903 conn(0x7fb94c0cee10 msgr2=0x7fb94c0cf220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:45.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.719+0000 7fb95344f700 1 -- 172.21.15.1:0/2411280903 shutdown_connections 2024-03-18T19:02:45.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.719+0000 7fb95344f700 1 -- 172.21.15.1:0/2411280903 wait complete. 2024-03-18T19:02:45.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.720+0000 7fb95344f700 1 Processor -- start 2024-03-18T19:02:45.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.721+0000 7fb95344f700 1 -- start start 2024-03-18T19:02:45.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.721+0000 7fb95344f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb94c0716b0 0x7fb94c146550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:45.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.721+0000 7fb95344f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb94c0abd70 0x7fb94c09adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:45.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.721+0000 7fb95344f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb94c09bfc0 0x7fb94c09c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:45.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.721+0000 7fb95344f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c146a90 con 0x7fb94c0abd70 2024-03-18T19:02:45.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.721+0000 7fb95344f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c09a930 con 0x7fb94c0716b0 2024-03-18T19:02:45.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.721+0000 7fb95344f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c099a40 con 0x7fb94c09bfc0 2024-03-18T19:02:45.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.721+0000 7fb9509ea700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb94c0abd70 0x7fb94c09adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:45.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.721+0000 7fb9509ea700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb94c0abd70 0x7fb94c09adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34432/0 (socket says 172.21.15.1:34432) 2024-03-18T19:02:45.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.721+0000 7fb9509ea700 1 -- 172.21.15.1:0/697520752 learned_addr learned my addr 172.21.15.1:0/697520752 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:45.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.721+0000 7fb9511eb700 1 --2- 172.21.15.1:0/697520752 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb94c0716b0 0x7fb94c146550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:45.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.721+0000 7fb9519ec700 1 --2- 172.21.15.1:0/697520752 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb94c09bfc0 0x7fb94c09c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:45.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.722+0000 7fb9509ea700 1 -- 172.21.15.1:0/697520752 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb94c09bfc0 msgr2=0x7fb94c09c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:45.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.722+0000 7fb9509ea700 1 --2- 172.21.15.1:0/697520752 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb94c09bfc0 0x7fb94c09c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:45.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.722+0000 7fb9509ea700 1 -- 172.21.15.1:0/697520752 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb94c0716b0 msgr2=0x7fb94c146550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:45.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.722+0000 7fb9509ea700 1 --2- 172.21.15.1:0/697520752 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb94c0716b0 0x7fb94c146550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:45.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.722+0000 7fb9509ea700 1 -- 172.21.15.1:0/697520752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9480094a0 con 0x7fb94c0abd70 2024-03-18T19:02:45.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.722+0000 7fb9509ea700 1 --2- 172.21.15.1:0/697520752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb94c0abd70 0x7fb94c09adc0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00e870 tx=0x7fb93c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:45.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.722+0000 7fb941ffb700 1 -- 172.21.15.1:0/697520752 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c009530 con 0x7fb94c0abd70 2024-03-18T19:02:45.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.722+0000 7fb941ffb700 1 -- 172.21.15.1:0/697520752 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93c013680 con 0x7fb94c0abd70 2024-03-18T19:02:45.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.722+0000 7fb95344f700 1 -- 172.21.15.1:0/697520752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb94c099d20 con 0x7fb94c0abd70 2024-03-18T19:02:45.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.722+0000 7fb941ffb700 1 -- 172.21.15.1:0/697520752 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c013a10 con 0x7fb94c0abd70 2024-03-18T19:02:45.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.722+0000 7fb95344f700 1 -- 172.21.15.1:0/697520752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb94c09a2e0 con 0x7fb94c0abd70 2024-03-18T19:02:45.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.723+0000 7fb95344f700 1 -- 172.21.15.1:0/697520752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94c09aae0 con 0x7fb94c0abd70 2024-03-18T19:02:45.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.724+0000 7fb941ffb700 1 -- 172.21.15.1:0/697520752 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7fb93c013b70 con 0x7fb94c0abd70 2024-03-18T19:02:45.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.724+0000 7fb941ffb700 1 --2- 172.21.15.1:0/697520752 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb93006ae40 0x7fb93006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:45.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.724+0000 7fb9511eb700 1 -- 172.21.15.1:0/697520752 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb93006ae40 msgr2=0x7fb93006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:45.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.725+0000 7fb941ffb700 1 -- 172.21.15.1:0/697520752 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6884+0+0 (secure 0 0 0) 0x7fb93c089f40 con 0x7fb94c0abd70 2024-03-18T19:02:45.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.725+0000 7fb9511eb700 1 --2- 172.21.15.1:0/697520752 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb93006ae40 0x7fb93006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:02:45.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.729+0000 7fb941ffb700 1 -- 172.21.15.1:0/697520752 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb93c0549d0 con 0x7fb94c0abd70 2024-03-18T19:02:45.789 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:45.792+0000 7f5e46aab240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:02:45.815 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:45.820+0000 7fdb7645b240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:02:45.865 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:45.869+0000 7f5e46aab240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:02:45.891 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:45.896+0000 7fdb7645b240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:02:45.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.926+0000 7fb9511eb700 1 -- 172.21.15.1:0/697520752 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb93006ae40 msgr2=0x7fb93006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:45.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.926+0000 7fb9511eb700 1 --2- 172.21.15.1:0/697520752 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb93006ae40 0x7fb93006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:02:45.942 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:45.946+0000 7f5e46aab240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:02:45.974 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:45.978+0000 7fdb7645b240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:02:45.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.982+0000 7fb95344f700 1 -- 172.21.15.1:0/697520752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb94c09aaa0 con 0x7fb94c0abd70 2024-03-18T19:02:45.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.982+0000 7fb941ffb700 1 -- 172.21.15.1:0/697520752 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4734 (secure 0 0 0) 0x7fb93c021090 con 0x7fb94c0abd70 2024-03-18T19:02:45.978 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:45.978 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4874,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2002167042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2002167042},{"type":"v1","addr":"172.21.15.165:6835","nonce":2002167042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4880,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3844556325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3844556325},{"type":"v1","addr":"172.21.15.1:6837","nonce":3844556325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5278,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1229938169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1229938169},{"type":"v1","addr":"172.21.15.165:6834","nonce":1229938169}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T18:51:12.721816+0000","modified":"2024-03-18T18:51:15.728690+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5436":{"gid":5436,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6836/3077516107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3077516107},{"type":"v1","addr":"172.21.15.1:6836","nonce":3077516107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-18T19:02:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.985+0000 7fb95344f700 1 -- 172.21.15.1:0/697520752 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb93006ae40 msgr2=0x7fb93006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:02:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.985+0000 7fb95344f700 1 --2- 172.21.15.1:0/697520752 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb93006ae40 0x7fb93006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.985+0000 7fb95344f700 1 -- 172.21.15.1:0/697520752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb94c0abd70 msgr2=0x7fb94c09adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.985+0000 7fb95344f700 1 --2- 172.21.15.1:0/697520752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb94c0abd70 0x7fb94c09adc0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00e870 tx=0x7fb93c00eb80 comp rx=0 tx=0).stop 2024-03-18T19:02:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.985+0000 7fb95344f700 1 -- 172.21.15.1:0/697520752 shutdown_connections 2024-03-18T19:02:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.985+0000 7fb95344f700 1 --2- 172.21.15.1:0/697520752 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb93006ae40 0x7fb93006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.985+0000 7fb95344f700 1 --2- 172.21.15.1:0/697520752 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb94c09bfc0 0x7fb94c09c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:45.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.985+0000 7fb95344f700 1 --2- 172.21.15.1:0/697520752 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb94c0716b0 0x7fb94c146550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.985+0000 7fb95344f700 1 --2- 172.21.15.1:0/697520752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb94c0abd70 0x7fb94c09adc0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.985+0000 7fb95344f700 1 -- 172.21.15.1:0/697520752 >> 172.21.15.1:0/697520752 conn(0x7fb94c0cee10 msgr2=0x7fb94c0caef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.985+0000 7fb95344f700 1 -- 172.21.15.1:0/697520752 shutdown_connections 2024-03-18T19:02:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:45.985+0000 7fb95344f700 1 -- 172.21.15.1:0/697520752 wait complete. 2024-03-18T19:02:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 17 2024-03-18T19:02:45.993 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:02:46.065 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:46.069+0000 7fdb7645b240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:02:46.116 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:46.119+0000 7f5e46aab240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:02:46.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.129+0000 7f5e39775700 1 Processor -- start 2024-03-18T19:02:46.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.129+0000 7f5e39775700 1 -- start start 2024-03-18T19:02:46.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.129+0000 7f5e39775700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e3415aff0 0x7f5e3415f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:46.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.129+0000 7f5e39775700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5e3415b3d0 0x7f5e340cf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:46.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.130+0000 7f5e39775700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3415fcc0 con 0x7f5e34073b00 2024-03-18T19:02:46.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.130+0000 7f5e39775700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e340ce200 con 0x7f5e3415b3d0 2024-03-18T19:02:46.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.130+0000 7f5e39775700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e340ce4a0 con 0x7f5e3415aff0 2024-03-18T19:02:46.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.130+0000 7f5e337fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5e34073b00 0x7f5e340cfa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37364/0 (socket says 172.21.15.1:37364) 2024-03-18T19:02:46.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.130+0000 7f5e337fe700 1 -- 172.21.15.1:0/3527448003 learned_addr learned my addr 172.21.15.1:0/3527448003 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:46.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.130+0000 7f5e32ffd700 1 --2- 172.21.15.1:0/3527448003 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e3415aff0 0x7f5e3415f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:46.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.130+0000 7f5e327fc700 1 --2- 172.21.15.1:0/3527448003 >> v2:172.21.15.165:3300/0 conn(0x7f5e3415b3d0 0x7f5e340cf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:46.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.130+0000 7f5e31ffb700 1 -- 172.21.15.1:0/3527448003 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936352901 0 0) 0x7f5e3415fcc0 con 0x7f5e34073b00 2024-03-18T19:02:46.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.130+0000 7f5e31ffb700 1 -- 172.21.15.1:0/3527448003 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e1c0035b0 con 0x7f5e34073b00 2024-03-18T19:02:46.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.130+0000 7f5e31ffb700 1 -- 172.21.15.1:0/3527448003 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2487378466 0 0) 0x7f5e1c0035b0 con 0x7f5e34073b00 2024-03-18T19:02:46.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.130+0000 7f5e32ffd700 1 -- 172.21.15.1:0/3527448003 >> v2:172.21.15.165:3300/0 conn(0x7f5e3415b3d0 msgr2=0x7f5e340cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:46.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.130+0000 7f5e32ffd700 1 --2- 172.21.15.1:0/3527448003 >> v2:172.21.15.165:3300/0 conn(0x7f5e3415b3d0 0x7f5e340cf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.130+0000 7f5e32ffd700 1 -- 172.21.15.1:0/3527448003 >> v1:172.21.15.1:6789/0 conn(0x7f5e34073b00 legacy=0x7f5e340cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:46.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.130+0000 7f5e32ffd700 1 -- 172.21.15.1:0/3527448003 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e340ce780 con 0x7f5e3415aff0 2024-03-18T19:02:46.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.130+0000 7f5e327fc700 1 --2- 172.21.15.1:0/3527448003 >> v2:172.21.15.165:3300/0 conn(0x7f5e3415b3d0 0x7f5e340cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:02:46.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.130+0000 7f5e32ffd700 1 --2- 172.21.15.1:0/3527448003 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e3415aff0 0x7f5e3415f780 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009840 tx=0x7f5e24009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c6d99544d24eebd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:46.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.131+0000 7f5e31ffb700 1 -- 172.21.15.1:0/3527448003 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c0019f0 con 0x7f5e3415aff0 2024-03-18T19:02:46.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.131+0000 7f5e31ffb700 1 -- 172.21.15.1:0/3527448003 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e24004030 con 0x7f5e3415aff0 2024-03-18T19:02:46.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.131+0000 7f5e31ffb700 1 -- 172.21.15.1:0/3527448003 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e240043a0 con 0x7f5e3415aff0 2024-03-18T19:02:46.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.131+0000 7f5e39775700 1 -- 172.21.15.1:0/3527448003 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e3415aff0 msgr2=0x7f5e3415f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:46.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.131+0000 7f5e39775700 1 --2- 172.21.15.1:0/3527448003 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e3415aff0 0x7f5e3415f780 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009840 tx=0x7f5e24009b50 comp rx=0 tx=0).stop 2024-03-18T19:02:46.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.131+0000 7f5e39775700 1 -- 172.21.15.1:0/3527448003 shutdown_connections 2024-03-18T19:02:46.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.131+0000 7f5e39775700 1 --2- 172.21.15.1:0/3527448003 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e3415aff0 0x7f5e3415f780 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.131+0000 7f5e39775700 1 --2- 172.21.15.1:0/3527448003 >> v2:172.21.15.165:3300/0 conn(0x7f5e3415b3d0 0x7f5e340cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.131+0000 7f5e39775700 1 -- 172.21.15.1:0/3527448003 >> 172.21.15.1:0/3527448003 conn(0x7f5e3406a750 msgr2=0x7f5e3406ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:46.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.131+0000 7f5e39775700 1 -- 172.21.15.1:0/3527448003 shutdown_connections 2024-03-18T19:02:46.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.131+0000 7f5e39775700 1 -- 172.21.15.1:0/3527448003 wait complete. 2024-03-18T19:02:46.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.133+0000 7f5e39775700 1 Processor -- start 2024-03-18T19:02:46.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.133+0000 7f5e39775700 1 -- start start 2024-03-18T19:02:46.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.133+0000 7f5e39775700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e34073b00 0x7f5e3413d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:46.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.133+0000 7f5e39775700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e3415aff0 0x7f5e3415c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:46.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.133+0000 7f5e39775700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e3415b3d0 0x7f5e341f3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:46.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.133+0000 7f5e39775700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3413dc50 con 0x7f5e34073b00 2024-03-18T19:02:46.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.133+0000 7f5e39775700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3413df30 con 0x7f5e3415b3d0 2024-03-18T19:02:46.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.133+0000 7f5e39775700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3413e210 con 0x7f5e3415aff0 2024-03-18T19:02:46.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.134+0000 7f5e32ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e34073b00 0x7f5e3413d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:46.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.134+0000 7f5e32ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e34073b00 0x7f5e3413d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34440/0 (socket says 172.21.15.1:34440) 2024-03-18T19:02:46.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.134+0000 7f5e32ffd700 1 -- 172.21.15.1:0/3267145095 learned_addr learned my addr 172.21.15.1:0/3267145095 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:46.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.134+0000 7f5e327fc700 1 --2- 172.21.15.1:0/3267145095 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e3415aff0 0x7f5e3415c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:46.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.134+0000 7f5e337fe700 1 --2- 172.21.15.1:0/3267145095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e3415b3d0 0x7f5e341f3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:46.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.134+0000 7f5e32ffd700 1 -- 172.21.15.1:0/3267145095 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e3415aff0 msgr2=0x7f5e3415c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:46.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.134+0000 7f5e32ffd700 1 --2- 172.21.15.1:0/3267145095 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e3415aff0 0x7f5e3415c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.134+0000 7f5e32ffd700 1 -- 172.21.15.1:0/3267145095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e3415b3d0 msgr2=0x7f5e341f3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:46.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.134+0000 7f5e32ffd700 1 --2- 172.21.15.1:0/3267145095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e3415b3d0 0x7f5e341f3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.134+0000 7f5e32ffd700 1 -- 172.21.15.1:0/3267145095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e28009430 con 0x7f5e34073b00 2024-03-18T19:02:46.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.134+0000 7f5e327fc700 1 --2- 172.21.15.1:0/3267145095 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e3415aff0 0x7f5e3415c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:02:46.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.134+0000 7f5e32ffd700 1 --2- 172.21.15.1:0/3267145095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e34073b00 0x7f5e3413d670 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f5e24000c00 tx=0x7f5e24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:46.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.134+0000 7f5e1b7fe700 1 -- 172.21.15.1:0/3267145095 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e24003ea0 con 0x7f5e34073b00 2024-03-18T19:02:46.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.134+0000 7f5e1b7fe700 1 -- 172.21.15.1:0/3267145095 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2401f070 con 0x7f5e34073b00 2024-03-18T19:02:46.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.134+0000 7f5e39775700 1 -- 172.21.15.1:0/3267145095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e240094a0 con 0x7f5e34073b00 2024-03-18T19:02:46.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.134+0000 7f5e1b7fe700 1 -- 172.21.15.1:0/3267145095 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e24011850 con 0x7f5e34073b00 2024-03-18T19:02:46.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.134+0000 7f5e39775700 1 -- 172.21.15.1:0/3267145095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3413e7d0 con 0x7f5e34073b00 2024-03-18T19:02:46.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.135+0000 7f5e39775700 1 -- 172.21.15.1:0/3267145095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e340676a0 con 0x7f5e34073b00 2024-03-18T19:02:46.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.136+0000 7f5e1b7fe700 1 -- 172.21.15.1:0/3267145095 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f5e240119b0 con 0x7f5e34073b00 2024-03-18T19:02:46.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.136+0000 7f5e1b7fe700 1 --2- 172.21.15.1:0/3267145095 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e2006ae40 0x7f5e2006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:46.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.136+0000 7f5e327fc700 1 -- 172.21.15.1:0/3267145095 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e2006ae40 msgr2=0x7f5e2006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:46.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.136+0000 7f5e1b7fe700 1 -- 172.21.15.1:0/3267145095 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6884+0+0 (secure 0 0 0) 0x7f5e24021070 con 0x7f5e34073b00 2024-03-18T19:02:46.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.136+0000 7f5e327fc700 1 --2- 172.21.15.1:0/3267145095 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e2006ae40 0x7f5e2006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:02:46.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.141+0000 7f5e1b7fe700 1 -- 172.21.15.1:0/3267145095 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5e24055bb0 con 0x7f5e34073b00 2024-03-18T19:02:46.151 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:46.155+0000 7fdb7645b240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:02:46.332 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:46.336+0000 7fdb7645b240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:02:46.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.337+0000 7f5e327fc700 1 -- 172.21.15.1:0/3267145095 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e2006ae40 msgr2=0x7f5e2006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:46.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.337+0000 7f5e327fc700 1 --2- 172.21.15.1:0/3267145095 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e2006ae40 0x7f5e2006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:02:46.333 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:46.336+0000 7f5e46aab240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:02:46.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.349+0000 7f5e39775700 1 -- 172.21.15.1:0/3267145095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e340af4d0 con 0x7f5e34073b00 2024-03-18T19:02:46.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.351+0000 7f5e1b7fe700 1 -- 172.21.15.1:0/3267145095 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+17550 (secure 0 0 0) 0x7f5e2402b090 con 0x7f5e34073b00 2024-03-18T19:02:46.347 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:46.347 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":58,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:02:46.048208+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:51:11.076221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T18:51:11.687105+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.26","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/1992330207":"2024-03-19T18:51:27.537808+0000","172.21.15.1:6832/76998":"2024-03-19T18:51:27.537808+0000","172.21.15.1:0/4026407635":"2024-03-19T18:51:27.537808+0000","172.21.15.1:0/2322425425":"2024-03-19T18:51:27.537808+0000","172.21.15.1:6833/76998":"2024-03-19T18:51:27.537808+0000","172.21.15.1:0/72803206":"2024-03-19T18:51:27.537808+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":2},{"begin":7,"length":1},{"begin":9,"length":1},{"begin":13,"length":2},{"begin":17,"length":1},{"begin":19,"length":2},{"begin":24,"length":2},{"begin":28,"length":1},{"begin":30,"length":2},{"begin":34,"length":2},{"begin":38,"length":1},{"begin":40,"length":3},{"begin":45,"length":2},{"begin":49,"length":1},{"begin":51,"length":3},{"begin":56,"length":2},{"begin":60,"length":1},{"begin":63,"length":2},{"begin":67,"length":2},{"begin":71,"length":2},{"begin":74,"length":2},{"begin":78,"length":2},{"begin":82,"length":1},{"begin":84,"length":2},{"begin":88,"length":2},{"begin":93,"length":1},{"begin":95,"length":2},{"begin":99,"length":2},{"begin":103,"length":1},{"begin":105,"length":2},{"begin":109,"length":2},{"begin":168,"length":54}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":103,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:02:46.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.353+0000 7f5e39775700 1 -- 172.21.15.1:0/3267145095 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e2006ae40 msgr2=0x7f5e2006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:02:46.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.353+0000 7f5e39775700 1 --2- 172.21.15.1:0/3267145095 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e2006ae40 0x7f5e2006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.353+0000 7f5e39775700 1 -- 172.21.15.1:0/3267145095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e34073b00 msgr2=0x7f5e3413d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:46.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.353+0000 7f5e39775700 1 --2- 172.21.15.1:0/3267145095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e34073b00 0x7f5e3413d670 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f5e24000c00 tx=0x7f5e24004750 comp rx=0 tx=0).stop 2024-03-18T19:02:46.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.353+0000 7f5e39775700 1 -- 172.21.15.1:0/3267145095 shutdown_connections 2024-03-18T19:02:46.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.353+0000 7f5e39775700 1 --2- 172.21.15.1:0/3267145095 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5e2006ae40 0x7f5e2006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.353+0000 7f5e39775700 1 --2- 172.21.15.1:0/3267145095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e3415b3d0 0x7f5e341f3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.353+0000 7f5e39775700 1 --2- 172.21.15.1:0/3267145095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e34073b00 0x7f5e3413d670 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.353+0000 7f5e39775700 1 --2- 172.21.15.1:0/3267145095 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5e3415aff0 0x7f5e3415c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.353+0000 7f5e39775700 1 -- 172.21.15.1:0/3267145095 >> 172.21.15.1:0/3267145095 conn(0x7f5e3406a750 msgr2=0x7f5e34143780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:46.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.353+0000 7f5e39775700 1 -- 172.21.15.1:0/3267145095 shutdown_connections 2024-03-18T19:02:46.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.353+0000 7f5e39775700 1 -- 172.21.15.1:0/3267145095 wait complete. 2024-03-18T19:02:46.361 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T18:51:11.076221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T18:51:11.687105+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 221, 'snap_epoch': 58, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:02:46.362 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:02:46.481 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:46.485+0000 7f5e46aab240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:02:46.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.500+0000 7f5afd3f7700 1 Processor -- start 2024-03-18T19:02:46.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.500+0000 7f5afd3f7700 1 -- start start 2024-03-18T19:02:46.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.501+0000 7f5afd3f7700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5af8099a40 0x7f5af8099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:46.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.501+0000 7f5afd3f7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5af80d0cd0 0x7f5af809a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:46.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.501+0000 7f5afd3f7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af80d0730 con 0x7f5af80cf3d0 2024-03-18T19:02:46.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.501+0000 7f5afd3f7700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af806bdf0 con 0x7f5af80d0cd0 2024-03-18T19:02:46.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.501+0000 7f5afd3f7700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af815d9f0 con 0x7f5af8099a40 2024-03-18T19:02:46.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.501+0000 7f5af67fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5af80d0cd0 0x7f5af809a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:46.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.501+0000 7f5af67fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5af80d0cd0 0x7f5af809a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34454/0 (socket says 172.21.15.1:34454) 2024-03-18T19:02:46.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.501+0000 7f5af67fc700 1 -- 172.21.15.1:0/3536191209 learned_addr learned my addr 172.21.15.1:0/3536191209 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:46.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.501+0000 7f5af77fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5af80cf3d0 0x7f5af80cf800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37368/0 (socket says 172.21.15.1:37368) 2024-03-18T19:02:46.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.501+0000 7f5af6ffd700 1 --2- 172.21.15.1:0/3536191209 >> v2:172.21.15.165:3300/0 conn(0x7f5af8099a40 0x7f5af8099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:46.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.501+0000 7f5af67fc700 1 -- 172.21.15.1:0/3536191209 >> v2:172.21.15.165:3300/0 conn(0x7f5af8099a40 msgr2=0x7f5af8099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:46.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.501+0000 7f5af67fc700 1 --2- 172.21.15.1:0/3536191209 >> v2:172.21.15.165:3300/0 conn(0x7f5af8099a40 0x7f5af8099e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.501+0000 7f5af67fc700 1 -- 172.21.15.1:0/3536191209 >> v1:172.21.15.1:6789/0 conn(0x7f5af80cf3d0 legacy=0x7f5af80cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:46.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.501+0000 7f5af67fc700 1 -- 172.21.15.1:0/3536191209 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af806bb10 con 0x7f5af80d0cd0 2024-03-18T19:02:46.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.501+0000 7f5af5ffb700 1 -- 172.21.15.1:0/3536191209 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200874888 0 0) 0x7f5af80d0730 con 0x7f5af80cf3d0 2024-03-18T19:02:46.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.501+0000 7f5af6ffd700 1 --2- 172.21.15.1:0/3536191209 >> v2:172.21.15.165:3300/0 conn(0x7f5af8099a40 0x7f5af8099e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:46.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.501+0000 7f5af67fc700 1 --2- 172.21.15.1:0/3536191209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af80d0cd0 0x7f5af809a360 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f5af8071c90 tx=0x7f5aec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30a2a4d5495d9b67 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:46.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.501+0000 7f5af5ffb700 1 -- 172.21.15.1:0/3536191209 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec00b860 con 0x7f5af80d0cd0 2024-03-18T19:02:46.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.502+0000 7f5af5ffb700 1 -- 172.21.15.1:0/3536191209 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aec00b9c0 con 0x7f5af80d0cd0 2024-03-18T19:02:46.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.502+0000 7f5af5ffb700 1 -- 172.21.15.1:0/3536191209 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec00bd50 con 0x7f5af80d0cd0 2024-03-18T19:02:46.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.502+0000 7f5afd3f7700 1 -- 172.21.15.1:0/3536191209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af80d0cd0 msgr2=0x7f5af809a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:46.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.502+0000 7f5afd3f7700 1 --2- 172.21.15.1:0/3536191209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af80d0cd0 0x7f5af809a360 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f5af8071c90 tx=0x7f5aec009c00 comp rx=0 tx=0).stop 2024-03-18T19:02:46.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.502+0000 7f5afd3f7700 1 -- 172.21.15.1:0/3536191209 shutdown_connections 2024-03-18T19:02:46.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.502+0000 7f5afd3f7700 1 --2- 172.21.15.1:0/3536191209 >> v2:172.21.15.165:3300/0 conn(0x7f5af8099a40 0x7f5af8099e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.502+0000 7f5afd3f7700 1 --2- 172.21.15.1:0/3536191209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af80d0cd0 0x7f5af809a360 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.502+0000 7f5afd3f7700 1 -- 172.21.15.1:0/3536191209 >> 172.21.15.1:0/3536191209 conn(0x7f5af80b2bf0 msgr2=0x7f5af80b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:46.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.502+0000 7f5afd3f7700 1 -- 172.21.15.1:0/3536191209 shutdown_connections 2024-03-18T19:02:46.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.502+0000 7f5afd3f7700 1 -- 172.21.15.1:0/3536191209 wait complete. 2024-03-18T19:02:46.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.504+0000 7f5afd3f7700 1 Processor -- start 2024-03-18T19:02:46.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.504+0000 7f5afd3f7700 1 -- start start 2024-03-18T19:02:46.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.504+0000 7f5afd3f7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5af8099a40 0x7f5af815f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:46.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.504+0000 7f5afd3f7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af80cf3d0 0x7f5af8150820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:46.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.504+0000 7f5afd3f7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5af80d0cd0 0x7f5af815db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:46.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.504+0000 7f5afd3f7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af815f9d0 con 0x7f5af80cf3d0 2024-03-18T19:02:46.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.504+0000 7f5afd3f7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af8150d60 con 0x7f5af80d0cd0 2024-03-18T19:02:46.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.504+0000 7f5afd3f7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af815e3f0 con 0x7f5af8099a40 2024-03-18T19:02:46.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.504+0000 7f5af67fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af80cf3d0 0x7f5af8150820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:46.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.504+0000 7f5af67fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af80cf3d0 0x7f5af8150820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34470/0 (socket says 172.21.15.1:34470) 2024-03-18T19:02:46.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.504+0000 7f5af67fc700 1 -- 172.21.15.1:0/571512947 learned_addr learned my addr 172.21.15.1:0/571512947 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:46.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.505+0000 7f5af6ffd700 1 --2- 172.21.15.1:0/571512947 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5af8099a40 0x7f5af815f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:46.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.505+0000 7f5af77fe700 1 --2- 172.21.15.1:0/571512947 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5af80d0cd0 0x7f5af815db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:46.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.505+0000 7f5af67fc700 1 -- 172.21.15.1:0/571512947 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5af8099a40 msgr2=0x7f5af815f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:46.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.505+0000 7f5af67fc700 1 --2- 172.21.15.1:0/571512947 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5af8099a40 0x7f5af815f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.505+0000 7f5af67fc700 1 -- 172.21.15.1:0/571512947 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5af80d0cd0 msgr2=0x7f5af815db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:46.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.505+0000 7f5af67fc700 1 --2- 172.21.15.1:0/571512947 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5af80d0cd0 0x7f5af815db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.505+0000 7f5af67fc700 1 -- 172.21.15.1:0/571512947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec009550 con 0x7f5af80cf3d0 2024-03-18T19:02:46.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.505+0000 7f5af67fc700 1 --2- 172.21.15.1:0/571512947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af80cf3d0 0x7f5af8150820 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5aec000c00 tx=0x7f5aec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:46.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.505+0000 7f5adf7fe700 1 -- 172.21.15.1:0/571512947 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec004510 con 0x7f5af80cf3d0 2024-03-18T19:02:46.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.505+0000 7f5adf7fe700 1 -- 172.21.15.1:0/571512947 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aec01e070 con 0x7f5af80cf3d0 2024-03-18T19:02:46.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.505+0000 7f5afd3f7700 1 -- 172.21.15.1:0/571512947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af815e6d0 con 0x7f5af80cf3d0 2024-03-18T19:02:46.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.505+0000 7f5adf7fe700 1 -- 172.21.15.1:0/571512947 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec0038e0 con 0x7f5af80cf3d0 2024-03-18T19:02:46.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.505+0000 7f5afd3f7700 1 -- 172.21.15.1:0/571512947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af815ec90 con 0x7f5af80cf3d0 2024-03-18T19:02:46.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.506+0000 7f5afd3f7700 1 -- 172.21.15.1:0/571512947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af80676a0 con 0x7f5af80cf3d0 2024-03-18T19:02:46.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.507+0000 7f5adf7fe700 1 -- 172.21.15.1:0/571512947 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f5aec003a40 con 0x7f5af80cf3d0 2024-03-18T19:02:46.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.507+0000 7f5adf7fe700 1 --2- 172.21.15.1:0/571512947 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ae406ae40 0x7f5ae406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:46.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.507+0000 7f5af6ffd700 1 -- 172.21.15.1:0/571512947 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ae406ae40 msgr2=0x7f5ae406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:46.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.507+0000 7f5adf7fe700 1 -- 172.21.15.1:0/571512947 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6884+0+0 (secure 0 0 0) 0x7f5aec020070 con 0x7f5af80cf3d0 2024-03-18T19:02:46.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.507+0000 7f5af6ffd700 1 --2- 172.21.15.1:0/571512947 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ae406ae40 0x7f5ae406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:02:46.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.512+0000 7f5adf7fe700 1 -- 172.21.15.1:0/571512947 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5aec0558b0 con 0x7f5af80cf3d0 2024-03-18T19:02:46.554 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:46.557+0000 7f5e46aab240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:02:46.558 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:46.562+0000 7fdb7645b240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:02:46.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.708+0000 7f5af6ffd700 1 -- 172.21.15.1:0/571512947 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ae406ae40 msgr2=0x7f5ae406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:46.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.708+0000 7f5af6ffd700 1 --2- 172.21.15.1:0/571512947 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ae406ae40 0x7f5ae406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:02:46.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.719+0000 7f5afd3f7700 1 -- 172.21.15.1:0/571512947 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5af8160d90 con 0x7f5af80cf3d0 2024-03-18T19:02:46.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.720+0000 7f5adf7fe700 1 -- 172.21.15.1:0/571512947 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5aec02a020 con 0x7f5af80cf3d0 2024-03-18T19:02:46.716 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:02:46.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.722+0000 7f5afd3f7700 1 -- 172.21.15.1:0/571512947 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ae406ae40 msgr2=0x7f5ae406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:02:46.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.722+0000 7f5afd3f7700 1 --2- 172.21.15.1:0/571512947 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ae406ae40 0x7f5ae406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.722+0000 7f5afd3f7700 1 -- 172.21.15.1:0/571512947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af80cf3d0 msgr2=0x7f5af8150820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.722+0000 7f5afd3f7700 1 --2- 172.21.15.1:0/571512947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af80cf3d0 0x7f5af8150820 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5aec000c00 tx=0x7f5aec004750 comp rx=0 tx=0).stop 2024-03-18T19:02:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.722+0000 7f5afd3f7700 1 -- 172.21.15.1:0/571512947 shutdown_connections 2024-03-18T19:02:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.722+0000 7f5afd3f7700 1 --2- 172.21.15.1:0/571512947 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ae406ae40 0x7f5ae406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.722+0000 7f5afd3f7700 1 --2- 172.21.15.1:0/571512947 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5af8099a40 0x7f5af815f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.722+0000 7f5afd3f7700 1 --2- 172.21.15.1:0/571512947 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5af80d0cd0 0x7f5af815db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.722+0000 7f5afd3f7700 1 --2- 172.21.15.1:0/571512947 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5af80cf3d0 0x7f5af8150820 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.722+0000 7f5afd3f7700 1 -- 172.21.15.1:0/571512947 >> 172.21.15.1:0/571512947 conn(0x7f5af80b2bf0 msgr2=0x7f5af815cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.722+0000 7f5afd3f7700 1 -- 172.21.15.1:0/571512947 shutdown_connections 2024-03-18T19:02:46.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.722+0000 7f5afd3f7700 1 -- 172.21.15.1:0/571512947 wait complete. 2024-03-18T19:02:46.721 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:46.726+0000 7fdb7645b240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:02:46.730 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T19:02:46.803 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:46.807+0000 7fdb7645b240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:02:46.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.864+0000 7f09d728f700 1 Processor -- start 2024-03-18T19:02:46.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.865+0000 7f09d728f700 1 -- start start 2024-03-18T19:02:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.866+0000 7f09d728f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f09d015d780 0x7f09d00cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.866+0000 7f09d728f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f09d009dd40 0x7f09d00cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.866+0000 7f09d728f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f09d00ce5a0 0x7f09d00cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.866+0000 7f09d728f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d00ce090 con 0x7f09d009dd40 2024-03-18T19:02:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.866+0000 7f09d728f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d006db30 con 0x7f09d015d780 2024-03-18T19:02:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.866+0000 7f09d728f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d006d520 con 0x7f09d00ce5a0 2024-03-18T19:02:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.866+0000 7f09d482a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f09d009dd40 0x7f09d00cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.866+0000 7f09d482a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f09d009dd40 0x7f09d00cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34474/0 (socket says 172.21.15.1:34474) 2024-03-18T19:02:46.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.866+0000 7f09d482a700 1 -- 172.21.15.1:0/2514513504 learned_addr learned my addr 172.21.15.1:0/2514513504 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.866+0000 7f09d582c700 1 --2- 172.21.15.1:0/2514513504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f09d00ce5a0 0x7f09d00cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.866+0000 7f09d502b700 1 --2- 172.21.15.1:0/2514513504 >> v2:172.21.15.165:3300/0 conn(0x7f09d015d780 0x7f09d00cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.866+0000 7f09d482a700 1 -- 172.21.15.1:0/2514513504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f09d00ce5a0 msgr2=0x7f09d00cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.866+0000 7f09d482a700 1 --2- 172.21.15.1:0/2514513504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f09d00ce5a0 0x7f09d00cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.866+0000 7f09d482a700 1 -- 172.21.15.1:0/2514513504 >> v2:172.21.15.165:3300/0 conn(0x7f09d015d780 msgr2=0x7f09d00cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.866+0000 7f09d482a700 1 --2- 172.21.15.1:0/2514513504 >> v2:172.21.15.165:3300/0 conn(0x7f09d015d780 0x7f09d00cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.866+0000 7f09d482a700 1 -- 172.21.15.1:0/2514513504 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d006f460 con 0x7f09d009dd40 2024-03-18T19:02:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.866+0000 7f09d482a700 1 --2- 172.21.15.1:0/2514513504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09d009dd40 0x7f09d00cad70 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f09d00674b0 tx=0x7f09c0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62658b0c678302a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.867+0000 7f09c7fff700 1 -- 172.21.15.1:0/2514513504 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0011830 con 0x7f09d009dd40 2024-03-18T19:02:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.867+0000 7f09c7fff700 1 -- 172.21.15.1:0/2514513504 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c0011990 con 0x7f09d009dd40 2024-03-18T19:02:46.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.867+0000 7f09c7fff700 1 -- 172.21.15.1:0/2514513504 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0011d20 con 0x7f09d009dd40 2024-03-18T19:02:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.868+0000 7f09d728f700 1 -- 172.21.15.1:0/2514513504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09d009dd40 msgr2=0x7f09d00cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.868+0000 7f09d728f700 1 --2- 172.21.15.1:0/2514513504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09d009dd40 0x7f09d00cad70 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f09d00674b0 tx=0x7f09c0009bd0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.868+0000 7f09d728f700 1 -- 172.21.15.1:0/2514513504 shutdown_connections 2024-03-18T19:02:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.868+0000 7f09d728f700 1 --2- 172.21.15.1:0/2514513504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f09d00ce5a0 0x7f09d00cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.868+0000 7f09d728f700 1 --2- 172.21.15.1:0/2514513504 >> v2:172.21.15.165:3300/0 conn(0x7f09d015d780 0x7f09d00cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.868+0000 7f09d728f700 1 --2- 172.21.15.1:0/2514513504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09d009dd40 0x7f09d00cad70 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.868+0000 7f09d728f700 1 -- 172.21.15.1:0/2514513504 >> 172.21.15.1:0/2514513504 conn(0x7f09d00cca00 msgr2=0x7f09d00cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.868+0000 7f09d728f700 1 -- 172.21.15.1:0/2514513504 shutdown_connections 2024-03-18T19:02:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.868+0000 7f09d728f700 1 -- 172.21.15.1:0/2514513504 wait complete. 2024-03-18T19:02:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.869+0000 7f09d728f700 1 Processor -- start 2024-03-18T19:02:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.869+0000 7f09d728f700 1 -- start start 2024-03-18T19:02:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.870+0000 7f09d728f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f09d009dd40 0x7f09d015a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.870+0000 7f09d728f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09d00ce5a0 0x7f09d01f4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.870+0000 7f09d728f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09d015d780 0x7f09d0157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.870+0000 7f09d728f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d01f4a20 con 0x7f09d00ce5a0 2024-03-18T19:02:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.870+0000 7f09d728f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d01f4d00 con 0x7f09d015d780 2024-03-18T19:02:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.870+0000 7f09d728f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d01f4fe0 con 0x7f09d009dd40 2024-03-18T19:02:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.870+0000 7f09d482a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09d00ce5a0 0x7f09d01f4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.870+0000 7f09d482a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09d00ce5a0 0x7f09d01f4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34482/0 (socket says 172.21.15.1:34482) 2024-03-18T19:02:46.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.870+0000 7f09d482a700 1 -- 172.21.15.1:0/2189578925 learned_addr learned my addr 172.21.15.1:0/2189578925 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.870+0000 7f09d502b700 1 --2- 172.21.15.1:0/2189578925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f09d009dd40 0x7f09d015a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.870+0000 7f09d582c700 1 --2- 172.21.15.1:0/2189578925 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09d015d780 0x7f09d0157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.870+0000 7f09d482a700 1 -- 172.21.15.1:0/2189578925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f09d009dd40 msgr2=0x7f09d015a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.870+0000 7f09d482a700 1 --2- 172.21.15.1:0/2189578925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f09d009dd40 0x7f09d015a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.870+0000 7f09d482a700 1 -- 172.21.15.1:0/2189578925 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09d015d780 msgr2=0x7f09d0157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.870+0000 7f09d482a700 1 --2- 172.21.15.1:0/2189578925 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09d015d780 0x7f09d0157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.870+0000 7f09d482a700 1 -- 172.21.15.1:0/2189578925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c0009550 con 0x7f09d00ce5a0 2024-03-18T19:02:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.870+0000 7f09d482a700 1 --2- 172.21.15.1:0/2189578925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09d00ce5a0 0x7f09d01f4160 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009c10 tx=0x7f09c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.870+0000 7f09c5ffb700 1 -- 172.21.15.1:0/2189578925 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0004510 con 0x7f09d00ce5a0 2024-03-18T19:02:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.871+0000 7f09c5ffb700 1 -- 172.21.15.1:0/2189578925 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c001e070 con 0x7f09d00ce5a0 2024-03-18T19:02:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.871+0000 7f09d728f700 1 -- 172.21.15.1:0/2189578925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d01f52c0 con 0x7f09d00ce5a0 2024-03-18T19:02:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.871+0000 7f09c5ffb700 1 -- 172.21.15.1:0/2189578925 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c00038e0 con 0x7f09d00ce5a0 2024-03-18T19:02:46.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.871+0000 7f09d728f700 1 -- 172.21.15.1:0/2189578925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d01f46a0 con 0x7f09d00ce5a0 2024-03-18T19:02:46.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.872+0000 7f09d728f700 1 -- 172.21.15.1:0/2189578925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d01f57e0 con 0x7f09d00ce5a0 2024-03-18T19:02:46.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.873+0000 7f09c5ffb700 1 -- 172.21.15.1:0/2189578925 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f09c0003a40 con 0x7f09d00ce5a0 2024-03-18T19:02:46.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.873+0000 7f09c5ffb700 1 --2- 172.21.15.1:0/2189578925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f09b406ae40 0x7f09b406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:46.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.873+0000 7f09d502b700 1 -- 172.21.15.1:0/2189578925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f09b406ae40 msgr2=0x7f09b406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:46.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.873+0000 7f09c5ffb700 1 -- 172.21.15.1:0/2189578925 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6884+0+0 (secure 0 0 0) 0x7f09c0020070 con 0x7f09d00ce5a0 2024-03-18T19:02:46.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.873+0000 7f09d502b700 1 --2- 172.21.15.1:0/2189578925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f09b406ae40 0x7f09b406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:02:46.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:46.882+0000 7f09c5ffb700 1 -- 172.21.15.1:0/2189578925 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f09d01f57e0 con 0x7f09d00ce5a0 2024-03-18T19:02:46.998 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:47.002+0000 7f5e46aab240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:02:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.074+0000 7f09d502b700 1 -- 172.21.15.1:0/2189578925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f09b406ae40 msgr2=0x7f09b406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.074+0000 7f09d502b700 1 --2- 172.21.15.1:0/2189578925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f09b406ae40 0x7f09b406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:02:47.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.085+0000 7f09d728f700 1 -- 172.21.15.1:0/2189578925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f09d01f57e0 con 0x7f09d00ce5a0 2024-03-18T19:02:47.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.085+0000 7f09c5ffb700 1 -- 172.21.15.1:0/2189578925 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7f09d01f57e0 con 0x7f09d00ce5a0 2024-03-18T19:02:47.081 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T19:02:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.088+0000 7f09d728f700 1 -- 172.21.15.1:0/2189578925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f09b406ae40 msgr2=0x7f09b406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:02:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.088+0000 7f09d728f700 1 --2- 172.21.15.1:0/2189578925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f09b406ae40 0x7f09b406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.088+0000 7f09d728f700 1 -- 172.21.15.1:0/2189578925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09d00ce5a0 msgr2=0x7f09d01f4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.088+0000 7f09d728f700 1 --2- 172.21.15.1:0/2189578925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09d00ce5a0 0x7f09d01f4160 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009c10 tx=0x7f09c0004750 comp rx=0 tx=0).stop 2024-03-18T19:02:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.088+0000 7f09d728f700 1 -- 172.21.15.1:0/2189578925 shutdown_connections 2024-03-18T19:02:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.088+0000 7f09d728f700 1 --2- 172.21.15.1:0/2189578925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f09b406ae40 0x7f09b406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.088+0000 7f09d728f700 1 --2- 172.21.15.1:0/2189578925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f09d009dd40 0x7f09d015a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.088+0000 7f09d728f700 1 --2- 172.21.15.1:0/2189578925 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f09d015d780 0x7f09d0157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.088+0000 7f09d728f700 1 --2- 172.21.15.1:0/2189578925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09d00ce5a0 0x7f09d01f4160 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.088+0000 7f09d728f700 1 -- 172.21.15.1:0/2189578925 >> 172.21.15.1:0/2189578925 conn(0x7f09d00cca00 msgr2=0x7f09d00b58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:47.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.088+0000 7f09d728f700 1 -- 172.21.15.1:0/2189578925 shutdown_connections 2024-03-18T19:02:47.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.088+0000 7f09d728f700 1 -- 172.21.15.1:0/2189578925 wait complete. 2024-03-18T19:02:47.096 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T19:02:47.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.230+0000 7fd3f43fa700 1 Processor -- start 2024-03-18T19:02:47.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.230+0000 7fd3f43fa700 1 -- start start 2024-03-18T19:02:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.231+0000 7fd3f43fa700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd3ec15b6c0 0x7fd3ec0b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.231+0000 7fd3f43fa700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ec0d01f0 con 0x7fd3ec0c87d0 2024-03-18T19:02:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.231+0000 7fd3f43fa700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ec0ccfa0 con 0x7fd3ec0cd430 2024-03-18T19:02:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.231+0000 7fd3f43fa700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3ec0d1db0 con 0x7fd3ec15b6c0 2024-03-18T19:02:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.231+0000 7fd3f2997700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd3ec0c87d0 0x7fd3ec0cd9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37370/0 (socket says 172.21.15.1:37370) 2024-03-18T19:02:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.231+0000 7fd3f2997700 1 -- 172.21.15.1:0/1421331932 learned_addr learned my addr 172.21.15.1:0/1421331932 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.231+0000 7fd3f2196700 1 --2- 172.21.15.1:0/1421331932 >> v2:172.21.15.165:3300/0 conn(0x7fd3ec15b6c0 0x7fd3ec0b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.231+0000 7fd3f1194700 1 -- 172.21.15.1:0/1421331932 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538042128 0 0) 0x7fd3ec0d01f0 con 0x7fd3ec0c87d0 2024-03-18T19:02:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.231+0000 7fd3f1194700 1 -- 172.21.15.1:0/1421331932 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3d80035b0 con 0x7fd3ec0c87d0 2024-03-18T19:02:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.231+0000 7fd3f2196700 1 -- 172.21.15.1:0/1421331932 >> v1:172.21.15.165:6789/0 conn(0x7fd3ec0cd430 legacy=0x7fd3ec0c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.231+0000 7fd3f2196700 1 -- 172.21.15.1:0/1421331932 >> v1:172.21.15.1:6789/0 conn(0x7fd3ec0c87d0 legacy=0x7fd3ec0cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.232+0000 7fd3f2196700 1 -- 172.21.15.1:0/1421331932 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ec0cff10 con 0x7fd3ec15b6c0 2024-03-18T19:02:47.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.232+0000 7fd3f2196700 1 --2- 172.21.15.1:0/1421331932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ec15b6c0 0x7fd3ec0b2160 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8009840 tx=0x7fd3e8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fba82c15cd49bb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:47.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.232+0000 7fd3f1194700 1 -- 172.21.15.1:0/1421331932 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8011830 con 0x7fd3ec15b6c0 2024-03-18T19:02:47.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.232+0000 7fd3f1194700 1 -- 172.21.15.1:0/1421331932 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e8011990 con 0x7fd3ec15b6c0 2024-03-18T19:02:47.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.232+0000 7fd3f1194700 1 -- 172.21.15.1:0/1421331932 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8011d00 con 0x7fd3ec15b6c0 2024-03-18T19:02:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.232+0000 7fd3f43fa700 1 -- 172.21.15.1:0/1421331932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ec15b6c0 msgr2=0x7fd3ec0b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.232+0000 7fd3f43fa700 1 --2- 172.21.15.1:0/1421331932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ec15b6c0 0x7fd3ec0b2160 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8009840 tx=0x7fd3e8009b50 comp rx=0 tx=0).stop 2024-03-18T19:02:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.233+0000 7fd3f43fa700 1 -- 172.21.15.1:0/1421331932 shutdown_connections 2024-03-18T19:02:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.233+0000 7fd3f43fa700 1 --2- 172.21.15.1:0/1421331932 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ec15b6c0 0x7fd3ec0b2160 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.233+0000 7fd3f43fa700 1 -- 172.21.15.1:0/1421331932 >> 172.21.15.1:0/1421331932 conn(0x7fd3ec0d0ff0 msgr2=0x7fd3ec0d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.233+0000 7fd3f43fa700 1 -- 172.21.15.1:0/1421331932 shutdown_connections 2024-03-18T19:02:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.233+0000 7fd3f43fa700 1 -- 172.21.15.1:0/1421331932 wait complete. 2024-03-18T19:02:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.234+0000 7fd3f43fa700 1 Processor -- start 2024-03-18T19:02:47.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.234+0000 7fd3f43fa700 1 -- start start 2024-03-18T19:02:47.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3f43fa700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ec0c87d0 0x7fd3ec09e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:47.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3f43fa700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3ec0cd430 0x7fd3ec0b5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3f43fa700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3ec15b6c0 0x7fd3ec0b6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3f43fa700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec1f1170 con 0x7fd3ec0cd430 2024-03-18T19:02:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3f43fa700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec09bcf0 con 0x7fd3ec0c87d0 2024-03-18T19:02:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3f43fa700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec146fe0 con 0x7fd3ec15b6c0 2024-03-18T19:02:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3f1995700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3ec0cd430 0x7fd3ec0b5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3f1995700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3ec0cd430 0x7fd3ec0b5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34492/0 (socket says 172.21.15.1:34492) 2024-03-18T19:02:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3f1995700 1 -- 172.21.15.1:0/1865939608 learned_addr learned my addr 172.21.15.1:0/1865939608 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3f2196700 1 --2- 172.21.15.1:0/1865939608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ec0c87d0 0x7fd3ec09e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3f2997700 1 --2- 172.21.15.1:0/1865939608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3ec15b6c0 0x7fd3ec0b6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3f1995700 1 -- 172.21.15.1:0/1865939608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3ec15b6c0 msgr2=0x7fd3ec0b6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3f1995700 1 --2- 172.21.15.1:0/1865939608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3ec15b6c0 0x7fd3ec0b6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3f1995700 1 -- 172.21.15.1:0/1865939608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ec0c87d0 msgr2=0x7fd3ec09e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3f1995700 1 --2- 172.21.15.1:0/1865939608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ec0c87d0 0x7fd3ec09e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3f1995700 1 -- 172.21.15.1:0/1865939608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e80094a0 con 0x7fd3ec0cd430 2024-03-18T19:02:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3f1995700 1 --2- 172.21.15.1:0/1865939608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3ec0cd430 0x7fd3ec0b5ad0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00c4e0 tx=0x7fd3dc00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3e2ffd700 1 -- 172.21.15.1:0/1865939608 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc0052d0 con 0x7fd3ec0cd430 2024-03-18T19:02:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3e2ffd700 1 -- 172.21.15.1:0/1865939608 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3dc005430 con 0x7fd3ec0cd430 2024-03-18T19:02:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3f43fa700 1 -- 172.21.15.1:0/1865939608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ec1472c0 con 0x7fd3ec0cd430 2024-03-18T19:02:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.235+0000 7fd3e2ffd700 1 -- 172.21.15.1:0/1865939608 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc0038c0 con 0x7fd3ec0cd430 2024-03-18T19:02:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.236+0000 7fd3f43fa700 1 -- 172.21.15.1:0/1865939608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ec147880 con 0x7fd3ec0cd430 2024-03-18T19:02:47.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.236+0000 7fd3f43fa700 1 -- 172.21.15.1:0/1865939608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3ec0676a0 con 0x7fd3ec0cd430 2024-03-18T19:02:47.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.237+0000 7fd3e2ffd700 1 -- 172.21.15.1:0/1865939608 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7fd3dc003a20 con 0x7fd3ec0cd430 2024-03-18T19:02:47.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.238+0000 7fd3e2ffd700 1 --2- 172.21.15.1:0/1865939608 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd3d006ae40 0x7fd3d006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:47.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.238+0000 7fd3f2196700 1 -- 172.21.15.1:0/1865939608 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd3d006ae40 msgr2=0x7fd3d006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:47.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.238+0000 7fd3e2ffd700 1 -- 172.21.15.1:0/1865939608 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6884+0+0 (secure 0 0 0) 0x7fd3dc089280 con 0x7fd3ec0cd430 2024-03-18T19:02:47.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.238+0000 7fd3f2196700 1 --2- 172.21.15.1:0/1865939608 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd3d006ae40 0x7fd3d006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:02:47.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.242+0000 7fd3e2ffd700 1 -- 172.21.15.1:0/1865939608 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd3dc053dd0 con 0x7fd3ec0cd430 2024-03-18T19:02:47.289 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:47.293+0000 7fdb7645b240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:02:47.338 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:47.342+0000 7f5e46aab240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:02:47.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.439+0000 7fd3f2196700 1 -- 172.21.15.1:0/1865939608 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd3d006ae40 msgr2=0x7fd3d006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:47.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.439+0000 7fd3f2196700 1 --2- 172.21.15.1:0/1865939608 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd3d006ae40 0x7fd3d006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:02:47.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.443+0000 7fd3f43fa700 1 -- 172.21.15.1:0/1865939608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd3ec0c9c50 con 0x7fd3ec0cd430 2024-03-18T19:02:47.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.444+0000 7fd3e2ffd700 1 -- 172.21.15.1:0/1865939608 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd3dc017020 con 0x7fd3ec0cd430 2024-03-18T19:02:47.440 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T19:02:47.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.446+0000 7fd3f43fa700 1 -- 172.21.15.1:0/1865939608 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd3d006ae40 msgr2=0x7fd3d006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:02:47.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.446+0000 7fd3f43fa700 1 --2- 172.21.15.1:0/1865939608 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd3d006ae40 0x7fd3d006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.446+0000 7fd3f43fa700 1 -- 172.21.15.1:0/1865939608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3ec0cd430 msgr2=0x7fd3ec0b5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:47.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.446+0000 7fd3f43fa700 1 --2- 172.21.15.1:0/1865939608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3ec0cd430 0x7fd3ec0b5ad0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00c4e0 tx=0x7fd3dc00c7f0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.446+0000 7fd3f43fa700 1 -- 172.21.15.1:0/1865939608 shutdown_connections 2024-03-18T19:02:47.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.446+0000 7fd3f43fa700 1 --2- 172.21.15.1:0/1865939608 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd3d006ae40 0x7fd3d006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.446+0000 7fd3f43fa700 1 --2- 172.21.15.1:0/1865939608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3ec15b6c0 0x7fd3ec0b6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.446+0000 7fd3f43fa700 1 --2- 172.21.15.1:0/1865939608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ec0c87d0 0x7fd3ec09e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.446+0000 7fd3f43fa700 1 --2- 172.21.15.1:0/1865939608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3ec0cd430 0x7fd3ec0b5ad0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.446+0000 7fd3f43fa700 1 -- 172.21.15.1:0/1865939608 >> 172.21.15.1:0/1865939608 conn(0x7fd3ec0d0ff0 msgr2=0x7fd3ec0c8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:47.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.446+0000 7fd3f43fa700 1 -- 172.21.15.1:0/1865939608 shutdown_connections 2024-03-18T19:02:47.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.446+0000 7fd3f43fa700 1 -- 172.21.15.1:0/1865939608 wait complete. 2024-03-18T19:02:47.454 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:02:47.497 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:47.501+0000 7f5e46aab240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:02:47.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.587+0000 7f0a07944700 1 Processor -- start 2024-03-18T19:02:47.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.587+0000 7f0a07944700 1 -- start start 2024-03-18T19:02:47.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.588+0000 7f0a07944700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0a0015d580 0x7f0a00072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:47.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.588+0000 7f0a07944700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a0006dd50 0x7f0a000cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:47.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.588+0000 7f0a07944700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0006f490 con 0x7f0a0006e290 2024-03-18T19:02:47.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.588+0000 7f0a07944700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a00071cf0 con 0x7f0a0015d580 2024-03-18T19:02:47.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.588+0000 7f0a07944700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0006f1b0 con 0x7f0a0006dd50 2024-03-18T19:02:47.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.588+0000 7f0a056e0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0a0015d580 0x7f0a00072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:47.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.588+0000 7f0a04edf700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0a0006e290 0x7f0a0006d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37374/0 (socket says 172.21.15.1:37374) 2024-03-18T19:02:47.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.588+0000 7f0a04edf700 1 -- 172.21.15.1:0/3407092144 learned_addr learned my addr 172.21.15.1:0/3407092144 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:47.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.588+0000 7f0a056e0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0a0015d580 0x7f0a00072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34508/0 (socket says 172.21.15.1:34508) 2024-03-18T19:02:47.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.588+0000 7f0a05ee1700 1 --2- 172.21.15.1:0/3407092144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a0006dd50 0x7f0a000cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:47.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.589+0000 7f0a056e0700 1 -- 172.21.15.1:0/3407092144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a0006dd50 msgr2=0x7f0a000cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:47.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.589+0000 7f0a056e0700 1 --2- 172.21.15.1:0/3407092144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a0006dd50 0x7f0a000cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.589+0000 7f0a056e0700 1 -- 172.21.15.1:0/3407092144 >> v1:172.21.15.1:6789/0 conn(0x7f0a0006e290 legacy=0x7f0a0006d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:47.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.589+0000 7f0a056e0700 1 -- 172.21.15.1:0/3407092144 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a000716a0 con 0x7f0a0015d580 2024-03-18T19:02:47.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.589+0000 7f09f7fff700 1 -- 172.21.15.1:0/3407092144 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496313 0 0) 0x7f0a0006f490 con 0x7f0a0006e290 2024-03-18T19:02:47.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.589+0000 7f0a056e0700 1 --2- 172.21.15.1:0/3407092144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a0015d580 0x7f0a00072180 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f09fc009840 tx=0x7f09fc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=194e5c3869d7ed71 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:47.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.589+0000 7f09f7fff700 1 -- 172.21.15.1:0/3407092144 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc011830 con 0x7f0a0015d580 2024-03-18T19:02:47.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.589+0000 7f09f7fff700 1 -- 172.21.15.1:0/3407092144 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09fc011990 con 0x7f0a0015d580 2024-03-18T19:02:47.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.589+0000 7f09f7fff700 1 -- 172.21.15.1:0/3407092144 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc011d20 con 0x7f0a0015d580 2024-03-18T19:02:47.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.589+0000 7f0a07944700 1 -- 172.21.15.1:0/3407092144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a0015d580 msgr2=0x7f0a00072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:47.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.589+0000 7f0a07944700 1 --2- 172.21.15.1:0/3407092144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a0015d580 0x7f0a00072180 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f09fc009840 tx=0x7f09fc009b50 comp rx=0 tx=0).stop 2024-03-18T19:02:47.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.590+0000 7f0a07944700 1 -- 172.21.15.1:0/3407092144 shutdown_connections 2024-03-18T19:02:47.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.590+0000 7f0a07944700 1 --2- 172.21.15.1:0/3407092144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a0015d580 0x7f0a00072180 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.590+0000 7f0a07944700 1 --2- 172.21.15.1:0/3407092144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a0006dd50 0x7f0a000cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.590+0000 7f0a07944700 1 -- 172.21.15.1:0/3407092144 >> 172.21.15.1:0/3407092144 conn(0x7f0a000cc990 msgr2=0x7f0a000ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:47.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.590+0000 7f0a07944700 1 -- 172.21.15.1:0/3407092144 shutdown_connections 2024-03-18T19:02:47.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.590+0000 7f0a07944700 1 -- 172.21.15.1:0/3407092144 wait complete. 2024-03-18T19:02:47.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.591+0000 7f0a07944700 1 Processor -- start 2024-03-18T19:02:47.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.591+0000 7f0a07944700 1 -- start start 2024-03-18T19:02:47.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.592+0000 7f0a07944700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a0006dd50 0x7f0a001562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:47.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.592+0000 7f0a07944700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a0015d580 0x7f0a000b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:47.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.592+0000 7f0a07944700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a00160350 0x7f0a000c3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:47.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.592+0000 7f0a07944700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000b4b70 con 0x7f0a00160350 2024-03-18T19:02:47.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.592+0000 7f0a07944700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000b3ef0 con 0x7f0a0015d580 2024-03-18T19:02:47.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.592+0000 7f0a07944700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000b41d0 con 0x7f0a0006dd50 2024-03-18T19:02:47.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.592+0000 7f0a05ee1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a00160350 0x7f0a000c3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:47.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.592+0000 7f0a05ee1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a00160350 0x7f0a000c3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34514/0 (socket says 172.21.15.1:34514) 2024-03-18T19:02:47.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.592+0000 7f0a05ee1700 1 -- 172.21.15.1:0/4136159358 learned_addr learned my addr 172.21.15.1:0/4136159358 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:47.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.592+0000 7f0a056e0700 1 --2- 172.21.15.1:0/4136159358 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a0006dd50 0x7f0a001562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:47.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.592+0000 7f0a04edf700 1 --2- 172.21.15.1:0/4136159358 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a0015d580 0x7f0a000b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:47.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.592+0000 7f0a05ee1700 1 -- 172.21.15.1:0/4136159358 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a0006dd50 msgr2=0x7f0a001562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:47.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.592+0000 7f0a05ee1700 1 --2- 172.21.15.1:0/4136159358 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a0006dd50 0x7f0a001562e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.592+0000 7f0a05ee1700 1 -- 172.21.15.1:0/4136159358 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a0015d580 msgr2=0x7f0a000b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:47.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.592+0000 7f0a05ee1700 1 --2- 172.21.15.1:0/4136159358 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a0015d580 0x7f0a000b35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.592+0000 7f0a05ee1700 1 -- 172.21.15.1:0/4136159358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc0094a0 con 0x7f0a00160350 2024-03-18T19:02:47.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.592+0000 7f0a056e0700 1 --2- 172.21.15.1:0/4136159358 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a0006dd50 0x7f0a001562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:02:47.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.592+0000 7f0a05ee1700 1 --2- 172.21.15.1:0/4136159358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a00160350 0x7f0a000c3fe0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f09f800ef90 tx=0x7f09f800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:47.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.592+0000 7f09f5ffb700 1 -- 172.21.15.1:0/4136159358 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f800cc90 con 0x7f0a00160350 2024-03-18T19:02:47.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.593+0000 7f09f5ffb700 1 -- 172.21.15.1:0/4136159358 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f800f040 con 0x7f0a00160350 2024-03-18T19:02:47.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.593+0000 7f0a07944700 1 -- 172.21.15.1:0/4136159358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a000b44b0 con 0x7f0a00160350 2024-03-18T19:02:47.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.593+0000 7f09f5ffb700 1 -- 172.21.15.1:0/4136159358 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f8014480 con 0x7f0a00160350 2024-03-18T19:02:47.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.593+0000 7f0a07944700 1 -- 172.21.15.1:0/4136159358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a00156820 con 0x7f0a00160350 2024-03-18T19:02:47.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.594+0000 7f0a07944700 1 -- 172.21.15.1:0/4136159358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a000b4e90 con 0x7f0a00160350 2024-03-18T19:02:47.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.594+0000 7f09f5ffb700 1 -- 172.21.15.1:0/4136159358 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f09f80145e0 con 0x7f0a00160350 2024-03-18T19:02:47.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.595+0000 7f09f5ffb700 1 --2- 172.21.15.1:0/4136159358 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f09e406ae40 0x7f09e406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:47.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.595+0000 7f0a056e0700 1 -- 172.21.15.1:0/4136159358 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f09e406ae40 msgr2=0x7f09e406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:47.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.595+0000 7f09f5ffb700 1 -- 172.21.15.1:0/4136159358 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6884+0+0 (secure 0 0 0) 0x7f09f808a800 con 0x7f0a00160350 2024-03-18T19:02:47.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.595+0000 7f0a056e0700 1 --2- 172.21.15.1:0/4136159358 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f09e406ae40 0x7f09e406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:02:47.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.599+0000 7f09f5ffb700 1 -- 172.21.15.1:0/4136159358 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f09f8055110 con 0x7f0a00160350 2024-03-18T19:02:47.644 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:47.649+0000 7fdb7645b240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:02:47.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.796+0000 7f0a056e0700 1 -- 172.21.15.1:0/4136159358 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f09e406ae40 msgr2=0x7f09e406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.796+0000 7f0a056e0700 1 --2- 172.21.15.1:0/4136159358 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f09e406ae40 0x7f09e406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:02:47.816 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:47.820+0000 7fdb7645b240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:02:47.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.850+0000 7f0a07944700 1 -- 172.21.15.1:0/4136159358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a000b4cb0 con 0x7f0a00160350 2024-03-18T19:02:47.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.850+0000 7f09f5ffb700 1 -- 172.21.15.1:0/4136159358 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4734 (secure 0 0 0) 0x7f0a000b4cb0 con 0x7f0a00160350 2024-03-18T19:02:47.846 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:47.846 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4874,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2002167042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2002167042},{"type":"v1","addr":"172.21.15.165:6835","nonce":2002167042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4880,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3844556325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3844556325},{"type":"v1","addr":"172.21.15.1:6837","nonce":3844556325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5278,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1229938169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1229938169},{"type":"v1","addr":"172.21.15.165:6834","nonce":1229938169}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T18:51:12.721816+0000","modified":"2024-03-18T18:51:15.728690+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5436":{"gid":5436,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6836/3077516107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3077516107},{"type":"v1","addr":"172.21.15.1:6836","nonce":3077516107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-18T19:02:47.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.852+0000 7f0a07944700 1 -- 172.21.15.1:0/4136159358 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f09e406ae40 msgr2=0x7f09e406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:02:47.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.852+0000 7f0a07944700 1 --2- 172.21.15.1:0/4136159358 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f09e406ae40 0x7f09e406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.852+0000 7f0a07944700 1 -- 172.21.15.1:0/4136159358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a00160350 msgr2=0x7f0a000c3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:47.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.852+0000 7f0a07944700 1 --2- 172.21.15.1:0/4136159358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a00160350 0x7f0a000c3fe0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f09f800ef90 tx=0x7f09f800c3a0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.853+0000 7f0a07944700 1 -- 172.21.15.1:0/4136159358 shutdown_connections 2024-03-18T19:02:47.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.853+0000 7f0a07944700 1 --2- 172.21.15.1:0/4136159358 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f09e406ae40 0x7f09e406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.853+0000 7f0a07944700 1 --2- 172.21.15.1:0/4136159358 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0a0006dd50 0x7f0a001562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.853+0000 7f0a07944700 1 --2- 172.21.15.1:0/4136159358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a00160350 0x7f0a000c3fe0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.853+0000 7f0a07944700 1 --2- 172.21.15.1:0/4136159358 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a0015d580 0x7f0a000b35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.853+0000 7f0a07944700 1 -- 172.21.15.1:0/4136159358 >> 172.21.15.1:0/4136159358 conn(0x7f0a000cc990 msgr2=0x7f0a000cbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:47.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.853+0000 7f0a07944700 1 -- 172.21.15.1:0/4136159358 shutdown_connections 2024-03-18T19:02:47.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.853+0000 7f0a07944700 1 -- 172.21.15.1:0/4136159358 wait complete. 2024-03-18T19:02:47.850 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 17 2024-03-18T19:02:47.861 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:02:47.880 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:47.884+0000 7f5e46aab240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:02:47.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.991+0000 7fb8ad016700 1 Processor -- start 2024-03-18T19:02:47.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.991+0000 7fb8ad016700 1 -- start start 2024-03-18T19:02:47.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.992+0000 7fb8ad016700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8a80ae600 0x7fb8a80ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:47.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.992+0000 7fb8ad016700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a806e270 con 0x7fb8a815a2e0 2024-03-18T19:02:47.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.992+0000 7fb8ad016700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a80ae050 con 0x7fb8a80cc170 2024-03-18T19:02:47.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.992+0000 7fb8ad016700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a806f690 con 0x7fb8a80ae600 2024-03-18T19:02:47.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.992+0000 7fb8a659c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb8a815a2e0 0x7fb8a80ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37378/0 (socket says 172.21.15.1:37378) 2024-03-18T19:02:47.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.992+0000 7fb8a659c700 1 -- 172.21.15.1:0/2423011057 learned_addr learned my addr 172.21.15.1:0/2423011057 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:47.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.992+0000 7fb8a6d9d700 1 --2- 172.21.15.1:0/2423011057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8a80ae600 0x7fb8a80ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.993+0000 7fb8a5d9b700 1 -- 172.21.15.1:0/2423011057 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013459640 0 0) 0x7fb8a806e270 con 0x7fb8a815a2e0 2024-03-18T19:02:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.993+0000 7fb8a5d9b700 1 -- 172.21.15.1:0/2423011057 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8900035b0 con 0x7fb8a815a2e0 2024-03-18T19:02:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.993+0000 7fb8a5d9b700 1 -- 172.21.15.1:0/2423011057 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434494969 0 0) 0x7fb8a80ae050 con 0x7fb8a80cc170 2024-03-18T19:02:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.993+0000 7fb8a5d9b700 1 -- 172.21.15.1:0/2423011057 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a806e270 con 0x7fb8a80cc170 2024-03-18T19:02:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.993+0000 7fb8a6d9d700 1 -- 172.21.15.1:0/2423011057 >> v1:172.21.15.165:6789/0 conn(0x7fb8a80cc170 legacy=0x7fb8a806ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.993+0000 7fb8a6d9d700 1 -- 172.21.15.1:0/2423011057 >> v1:172.21.15.1:6789/0 conn(0x7fb8a815a2e0 legacy=0x7fb8a80ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.993+0000 7fb8a6d9d700 1 -- 172.21.15.1:0/2423011057 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a80add70 con 0x7fb8a80ae600 2024-03-18T19:02:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.993+0000 7fb8a6d9d700 1 --2- 172.21.15.1:0/2423011057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8a80ae600 0x7fb8a80ad790 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb898009870 tx=0x7fb898009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cfce330c95d2525 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.993+0000 7fb8a5d9b700 1 -- 172.21.15.1:0/2423011057 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898011830 con 0x7fb8a80ae600 2024-03-18T19:02:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.993+0000 7fb8a5d9b700 1 -- 172.21.15.1:0/2423011057 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb898011990 con 0x7fb8a80ae600 2024-03-18T19:02:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.993+0000 7fb8a5d9b700 1 -- 172.21.15.1:0/2423011057 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898011d20 con 0x7fb8a80ae600 2024-03-18T19:02:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.994+0000 7fb8ad016700 1 -- 172.21.15.1:0/2423011057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8a80ae600 msgr2=0x7fb8a80ad790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.994+0000 7fb8ad016700 1 --2- 172.21.15.1:0/2423011057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8a80ae600 0x7fb8a80ad790 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb898009870 tx=0x7fb898009b80 comp rx=0 tx=0).stop 2024-03-18T19:02:47.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.994+0000 7fb8ad016700 1 -- 172.21.15.1:0/2423011057 shutdown_connections 2024-03-18T19:02:47.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.994+0000 7fb8ad016700 1 --2- 172.21.15.1:0/2423011057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8a80ae600 0x7fb8a80ad790 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.994+0000 7fb8ad016700 1 -- 172.21.15.1:0/2423011057 >> 172.21.15.1:0/2423011057 conn(0x7fb8a806b860 msgr2=0x7fb8a806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:47.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.994+0000 7fb8ad016700 1 -- 172.21.15.1:0/2423011057 shutdown_connections 2024-03-18T19:02:47.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.994+0000 7fb8ad016700 1 -- 172.21.15.1:0/2423011057 wait complete. 2024-03-18T19:02:47.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.995+0000 7fb8ad016700 1 Processor -- start 2024-03-18T19:02:47.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.995+0000 7fb8ad016700 1 -- start start 2024-03-18T19:02:47.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.996+0000 7fb8ad016700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8a80ae600 0x7fb8a81504e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:47.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.996+0000 7fb8ad016700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a80cc170 0x7fb8a80cb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:47.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.996+0000 7fb8ad016700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a815a2e0 0x7fb8a809bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.996+0000 7fb8ad016700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a809af80 con 0x7fb8a80cc170 2024-03-18T19:02:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.996+0000 7fb8ad016700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a809a9c0 con 0x7fb8a815a2e0 2024-03-18T19:02:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.996+0000 7fb8ad016700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a809aca0 con 0x7fb8a80ae600 2024-03-18T19:02:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.996+0000 7fb8a659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a80cc170 0x7fb8a80cb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.996+0000 7fb8a6d9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8a80ae600 0x7fb8a81504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.996+0000 7fb8a659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a80cc170 0x7fb8a80cb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34516/0 (socket says 172.21.15.1:34516) 2024-03-18T19:02:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.996+0000 7fb8a659c700 1 -- 172.21.15.1:0/203913197 learned_addr learned my addr 172.21.15.1:0/203913197 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.996+0000 7fb8a759e700 1 --2- 172.21.15.1:0/203913197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a815a2e0 0x7fb8a809bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.997+0000 7fb8a659c700 1 -- 172.21.15.1:0/203913197 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8a80ae600 msgr2=0x7fb8a81504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.997+0000 7fb8a659c700 1 --2- 172.21.15.1:0/203913197 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8a80ae600 0x7fb8a81504e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.997+0000 7fb8a659c700 1 -- 172.21.15.1:0/203913197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a815a2e0 msgr2=0x7fb8a809bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:47.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.997+0000 7fb8a659c700 1 --2- 172.21.15.1:0/203913197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a815a2e0 0x7fb8a809bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.997+0000 7fb8a659c700 1 -- 172.21.15.1:0/203913197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8980094a0 con 0x7fb8a80cc170 2024-03-18T19:02:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.997+0000 7fb8a759e700 1 --2- 172.21.15.1:0/203913197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a815a2e0 0x7fb8a809bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:02:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.997+0000 7fb8a6d9d700 1 --2- 172.21.15.1:0/203913197 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8a80ae600 0x7fb8a81504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:02:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.997+0000 7fb8a659c700 1 --2- 172.21.15.1:0/203913197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a80cc170 0x7fb8a80cb700 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fb89c005910 tx=0x7fb89c00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.997+0000 7fb88f7fe700 1 -- 172.21.15.1:0/203913197 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c005d70 con 0x7fb8a80cc170 2024-03-18T19:02:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.997+0000 7fb88f7fe700 1 -- 172.21.15.1:0/203913197 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89c00ce80 con 0x7fb8a80cc170 2024-03-18T19:02:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.997+0000 7fb8ad016700 1 -- 172.21.15.1:0/203913197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a80c9290 con 0x7fb8a80cc170 2024-03-18T19:02:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.997+0000 7fb88f7fe700 1 -- 172.21.15.1:0/203913197 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c0072d0 con 0x7fb8a80cc170 2024-03-18T19:02:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.997+0000 7fb8ad016700 1 -- 172.21.15.1:0/203913197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a81f3700 con 0x7fb8a80cc170 2024-03-18T19:02:47.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.998+0000 7fb8ad016700 1 -- 172.21.15.1:0/203913197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a80676a0 con 0x7fb8a80cc170 2024-03-18T19:02:47.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.999+0000 7fb88f7fe700 1 -- 172.21.15.1:0/203913197 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7fb89c007430 con 0x7fb8a80cc170 2024-03-18T19:02:47.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.999+0000 7fb88f7fe700 1 --2- 172.21.15.1:0/203913197 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb89406ae40 0x7fb89406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:47.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.999+0000 7fb8a6d9d700 1 -- 172.21.15.1:0/203913197 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb89406ae40 msgr2=0x7fb89406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:47.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.999+0000 7fb88f7fe700 1 -- 172.21.15.1:0/203913197 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6884+0+0 (secure 0 0 0) 0x7fb89c0890e0 con 0x7fb8a80cc170 2024-03-18T19:02:47.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:47.999+0000 7fb8a6d9d700 1 --2- 172.21.15.1:0/203913197 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb89406ae40 0x7fb89406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:02:47.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.003+0000 7fb88f7fe700 1 -- 172.21.15.1:0/203913197 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb89c053dc0 con 0x7fb8a80cc170 2024-03-18T19:02:48.041 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:48.045+0000 7f5e46aab240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:02:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.200+0000 7fb8a6d9d700 1 -- 172.21.15.1:0/203913197 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb89406ae40 msgr2=0x7fb89406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:48.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.200+0000 7fb8a6d9d700 1 --2- 172.21.15.1:0/203913197 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb89406ae40 0x7fb89406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:02:48.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.209+0000 7fb8ad016700 1 -- 172.21.15.1:0/203913197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8a8159b00 con 0x7fb8a80cc170 2024-03-18T19:02:48.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.210+0000 7fb88f7fe700 1 -- 172.21.15.1:0/203913197 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+17550 (secure 0 0 0) 0x7fb89c017090 con 0x7fb8a80cc170 2024-03-18T19:02:48.206 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:48.206 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":58,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:02:46.048208+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:51:11.076221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T18:51:11.687105+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.26","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/1992330207":"2024-03-19T18:51:27.537808+0000","172.21.15.1:6832/76998":"2024-03-19T18:51:27.537808+0000","172.21.15.1:0/4026407635":"2024-03-19T18:51:27.537808+0000","172.21.15.1:0/2322425425":"2024-03-19T18:51:27.537808+0000","172.21.15.1:6833/76998":"2024-03-19T18:51:27.537808+0000","172.21.15.1:0/72803206":"2024-03-19T18:51:27.537808+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":2},{"begin":7,"length":1},{"begin":9,"length":1},{"begin":13,"length":2},{"begin":17,"length":1},{"begin":19,"length":2},{"begin":24,"length":2},{"begin":28,"length":1},{"begin":30,"length":2},{"begin":34,"length":2},{"begin":38,"length":1},{"begin":40,"length":3},{"begin":45,"length":2},{"begin":49,"length":1},{"begin":51,"length":3},{"begin":56,"length":2},{"begin":60,"length":1},{"begin":63,"length":2},{"begin":67,"length":2},{"begin":71,"length":2},{"begin":74,"length":2},{"begin":78,"length":2},{"begin":82,"length":1},{"begin":84,"length":2},{"begin":88,"length":2},{"begin":93,"length":1},{"begin":95,"length":2},{"begin":99,"length":2},{"begin":103,"length":1},{"begin":105,"length":2},{"begin":109,"length":2},{"begin":168,"length":54}]}],"new_removed_snaps":[{"pool":6,"snaps":[{"begin":103,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:02:48.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.212+0000 7fb8ad016700 1 -- 172.21.15.1:0/203913197 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb89406ae40 msgr2=0x7fb89406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:02:48.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.212+0000 7fb8ad016700 1 --2- 172.21.15.1:0/203913197 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb89406ae40 0x7fb89406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:48.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.212+0000 7fb8ad016700 1 -- 172.21.15.1:0/203913197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a80cc170 msgr2=0x7fb8a80cb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:48.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.212+0000 7fb8ad016700 1 --2- 172.21.15.1:0/203913197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a80cc170 0x7fb8a80cb700 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fb89c005910 tx=0x7fb89c00c7b0 comp rx=0 tx=0).stop 2024-03-18T19:02:48.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.212+0000 7fb8ad016700 1 -- 172.21.15.1:0/203913197 shutdown_connections 2024-03-18T19:02:48.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.212+0000 7fb8ad016700 1 --2- 172.21.15.1:0/203913197 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb89406ae40 0x7fb89406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:48.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.212+0000 7fb8ad016700 1 --2- 172.21.15.1:0/203913197 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8a80ae600 0x7fb8a81504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:48.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.212+0000 7fb8ad016700 1 --2- 172.21.15.1:0/203913197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a815a2e0 0x7fb8a809bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:48.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.212+0000 7fb8ad016700 1 --2- 172.21.15.1:0/203913197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8a80cc170 0x7fb8a80cb700 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:48.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.212+0000 7fb8ad016700 1 -- 172.21.15.1:0/203913197 >> 172.21.15.1:0/203913197 conn(0x7fb8a806b860 msgr2=0x7fb8a806ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:48.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.212+0000 7fb8ad016700 1 -- 172.21.15.1:0/203913197 shutdown_connections 2024-03-18T19:02:48.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.212+0000 7fb8ad016700 1 -- 172.21.15.1:0/203913197 wait complete. 2024-03-18T19:02:48.221 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T19:02:48.221 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:02:48.237 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:48.242+0000 7fdb7645b240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:02:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.355+0000 7f25768a4700 1 Processor -- start 2024-03-18T19:02:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.355+0000 7f25768a4700 1 -- start start 2024-03-18T19:02:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.356+0000 7f25768a4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f257015b4d0 0x7f257009a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.356+0000 7f25768a4700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257006f070 con 0x7f257009a150 2024-03-18T19:02:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.356+0000 7f25768a4700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25700d0260 con 0x7f25700cf750 2024-03-18T19:02:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.356+0000 7f25768a4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25700cff80 con 0x7f257015b4d0 2024-03-18T19:02:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.356+0000 7f2574e41700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f257009a150 0x7f257006f500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37384/0 (socket says 172.21.15.1:37384) 2024-03-18T19:02:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.356+0000 7f2574e41700 1 -- 172.21.15.1:0/482942464 learned_addr learned my addr 172.21.15.1:0/482942464 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.356+0000 7f256ffff700 1 --2- 172.21.15.1:0/482942464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f257015b4d0 0x7f257009a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.356+0000 7f256effd700 1 -- 172.21.15.1:0/482942464 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605762561 0 0) 0x7f257006f070 con 0x7f257009a150 2024-03-18T19:02:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.356+0000 7f256effd700 1 -- 172.21.15.1:0/482942464 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25580035b0 con 0x7f257009a150 2024-03-18T19:02:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.356+0000 7f256effd700 1 -- 172.21.15.1:0/482942464 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066436465 0 0) 0x7f25700d0260 con 0x7f25700cf750 2024-03-18T19:02:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.356+0000 7f256effd700 1 -- 172.21.15.1:0/482942464 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f257006f070 con 0x7f25700cf750 2024-03-18T19:02:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.356+0000 7f256ffff700 1 -- 172.21.15.1:0/482942464 >> v1:172.21.15.165:6789/0 conn(0x7f25700cf750 legacy=0x7f2570099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.356+0000 7f256effd700 1 -- 172.21.15.1:0/482942464 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3889375484 0 0) 0x7f25580035b0 con 0x7f257009a150 2024-03-18T19:02:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.357+0000 7f256ffff700 1 -- 172.21.15.1:0/482942464 >> v1:172.21.15.1:6789/0 conn(0x7f257009a150 legacy=0x7f257006f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.357+0000 7f256ffff700 1 -- 172.21.15.1:0/482942464 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257006ea20 con 0x7f257015b4d0 2024-03-18T19:02:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.357+0000 7f256ffff700 1 --2- 172.21.15.1:0/482942464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f257015b4d0 0x7f257009a650 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2560009870 tx=0x7f2560009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c34be50f4a2a3c26 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.357+0000 7f256effd700 1 -- 172.21.15.1:0/482942464 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25680019f0 con 0x7f257015b4d0 2024-03-18T19:02:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.357+0000 7f256effd700 1 -- 172.21.15.1:0/482942464 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2560004030 con 0x7f257015b4d0 2024-03-18T19:02:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.357+0000 7f256effd700 1 -- 172.21.15.1:0/482942464 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25600043c0 con 0x7f257015b4d0 2024-03-18T19:02:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.357+0000 7f25768a4700 1 -- 172.21.15.1:0/482942464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f257015b4d0 msgr2=0x7f257009a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.357+0000 7f25768a4700 1 --2- 172.21.15.1:0/482942464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f257015b4d0 0x7f257009a650 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2560009870 tx=0x7f2560009b80 comp rx=0 tx=0).stop 2024-03-18T19:02:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.357+0000 7f25768a4700 1 -- 172.21.15.1:0/482942464 shutdown_connections 2024-03-18T19:02:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.357+0000 7f25768a4700 1 --2- 172.21.15.1:0/482942464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f257015b4d0 0x7f257009a650 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.358+0000 7f25768a4700 1 -- 172.21.15.1:0/482942464 >> 172.21.15.1:0/482942464 conn(0x7f25700b2cb0 msgr2=0x7f25700b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.358+0000 7f25768a4700 1 -- 172.21.15.1:0/482942464 shutdown_connections 2024-03-18T19:02:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.358+0000 7f25768a4700 1 -- 172.21.15.1:0/482942464 wait complete. 2024-03-18T19:02:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.359+0000 7f25768a4700 1 Processor -- start 2024-03-18T19:02:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.359+0000 7f25768a4700 1 -- start start 2024-03-18T19:02:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.360+0000 7f25768a4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f257009a150 0x7f25700cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.360+0000 7f25768a4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25700cf750 0x7f25700c5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.360+0000 7f25768a4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f257015b4d0 0x7f25700c6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.360+0000 7f25768a4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25700cb330 con 0x7f25700cf750 2024-03-18T19:02:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.360+0000 7f25768a4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257006c310 con 0x7f257009a150 2024-03-18T19:02:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.360+0000 7f25768a4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257006c5f0 con 0x7f257015b4d0 2024-03-18T19:02:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.360+0000 7f256f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25700cf750 0x7f25700c5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.360+0000 7f256f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25700cf750 0x7f25700c5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34530/0 (socket says 172.21.15.1:34530) 2024-03-18T19:02:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.360+0000 7f256f7fe700 1 -- 172.21.15.1:0/381613356 learned_addr learned my addr 172.21.15.1:0/381613356 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.360+0000 7f2574e41700 1 --2- 172.21.15.1:0/381613356 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f257015b4d0 0x7f25700c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.360+0000 7f256ffff700 1 --2- 172.21.15.1:0/381613356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f257009a150 0x7f25700cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.360+0000 7f256f7fe700 1 -- 172.21.15.1:0/381613356 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f257015b4d0 msgr2=0x7f25700c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.360+0000 7f256f7fe700 1 --2- 172.21.15.1:0/381613356 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f257015b4d0 0x7f25700c6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.360+0000 7f256f7fe700 1 -- 172.21.15.1:0/381613356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f257009a150 msgr2=0x7f25700cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.360+0000 7f256f7fe700 1 --2- 172.21.15.1:0/381613356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f257009a150 0x7f25700cadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.360+0000 7f256f7fe700 1 -- 172.21.15.1:0/381613356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25600094a0 con 0x7f25700cf750 2024-03-18T19:02:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.360+0000 7f256f7fe700 1 --2- 172.21.15.1:0/381613356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25700cf750 0x7f25700c5ce0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f256400c520 tx=0x7f256400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.360+0000 7f256cff9700 1 -- 172.21.15.1:0/381613356 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25640052d0 con 0x7f25700cf750 2024-03-18T19:02:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.360+0000 7f256cff9700 1 -- 172.21.15.1:0/381613356 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2564005430 con 0x7f25700cf750 2024-03-18T19:02:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.361+0000 7f25768a4700 1 -- 172.21.15.1:0/381613356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25700cfd50 con 0x7f25700cf750 2024-03-18T19:02:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.361+0000 7f256cff9700 1 -- 172.21.15.1:0/381613356 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25640038c0 con 0x7f25700cf750 2024-03-18T19:02:48.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.361+0000 7f25768a4700 1 -- 172.21.15.1:0/381613356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f257015bad0 con 0x7f25700cf750 2024-03-18T19:02:48.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.362+0000 7f25768a4700 1 -- 172.21.15.1:0/381613356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2570068240 con 0x7f25700cf750 2024-03-18T19:02:48.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.362+0000 7f256cff9700 1 -- 172.21.15.1:0/381613356 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f2564003a20 con 0x7f25700cf750 2024-03-18T19:02:48.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.363+0000 7f256cff9700 1 --2- 172.21.15.1:0/381613356 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f255c06ae40 0x7f255c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:48.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.363+0000 7f256ffff700 1 -- 172.21.15.1:0/381613356 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f255c06ae40 msgr2=0x7f255c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:48.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.363+0000 7f256cff9700 1 -- 172.21.15.1:0/381613356 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6884+0+0 (secure 0 0 0) 0x7f2564088eb0 con 0x7f25700cf750 2024-03-18T19:02:48.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.363+0000 7f256ffff700 1 --2- 172.21.15.1:0/381613356 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f255c06ae40 0x7f255c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:02:48.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.367+0000 7f256cff9700 1 -- 172.21.15.1:0/381613356 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2564053a00 con 0x7f25700cf750 2024-03-18T19:02:48.408 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:48.411+0000 7f5e46aab240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:02:48.452 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:48.456+0000 7fdb7645b240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:02:48.496 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:02:48.499+0000 7f5e46aab240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:02:48.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.564+0000 7f256ffff700 1 -- 172.21.15.1:0/381613356 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f255c06ae40 msgr2=0x7f255c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:48.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.564+0000 7f256ffff700 1 --2- 172.21.15.1:0/381613356 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f255c06ae40 0x7f255c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:02:48.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.665+0000 7f25768a4700 1 -- 172.21.15.1:0/381613356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25700676a0 con 0x7f25700cf750 2024-03-18T19:02:48.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.665+0000 7f256cff9700 1 -- 172.21.15.1:0/381613356 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4734 (secure 0 0 0) 0x7f2564017090 con 0x7f25700cf750 2024-03-18T19:02:48.661 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:48.662 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4874,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2002167042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2002167042},{"type":"v1","addr":"172.21.15.165:6835","nonce":2002167042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4880,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3844556325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3844556325},{"type":"v1","addr":"172.21.15.1:6837","nonce":3844556325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5278,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1229938169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1229938169},{"type":"v1","addr":"172.21.15.165:6834","nonce":1229938169}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T18:51:12.721816+0000","modified":"2024-03-18T18:51:15.728690+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5436":{"gid":5436,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6836/3077516107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3077516107},{"type":"v1","addr":"172.21.15.1:6836","nonce":3077516107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-18T19:02:48.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.668+0000 7f25768a4700 1 -- 172.21.15.1:0/381613356 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f255c06ae40 msgr2=0x7f255c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:02:48.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.668+0000 7f25768a4700 1 --2- 172.21.15.1:0/381613356 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f255c06ae40 0x7f255c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:48.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.668+0000 7f25768a4700 1 -- 172.21.15.1:0/381613356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25700cf750 msgr2=0x7f25700c5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:48.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.668+0000 7f25768a4700 1 --2- 172.21.15.1:0/381613356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25700cf750 0x7f25700c5ce0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f256400c520 tx=0x7f256400c830 comp rx=0 tx=0).stop 2024-03-18T19:02:48.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.668+0000 7f25768a4700 1 -- 172.21.15.1:0/381613356 shutdown_connections 2024-03-18T19:02:48.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.668+0000 7f25768a4700 1 --2- 172.21.15.1:0/381613356 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f255c06ae40 0x7f255c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:48.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.668+0000 7f25768a4700 1 --2- 172.21.15.1:0/381613356 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f257015b4d0 0x7f25700c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:48.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.668+0000 7f25768a4700 1 --2- 172.21.15.1:0/381613356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f257009a150 0x7f25700cadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:48.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.668+0000 7f25768a4700 1 --2- 172.21.15.1:0/381613356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25700cf750 0x7f25700c5ce0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:48.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.668+0000 7f25768a4700 1 -- 172.21.15.1:0/381613356 >> 172.21.15.1:0/381613356 conn(0x7f25700b2cb0 msgr2=0x7f25700b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:48.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.668+0000 7f25768a4700 1 -- 172.21.15.1:0/381613356 shutdown_connections 2024-03-18T19:02:48.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.668+0000 7f25768a4700 1 -- 172.21.15.1:0/381613356 wait complete. 2024-03-18T19:02:48.665 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 17 2024-03-18T19:02:48.677 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:02:48.837 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:48.841+0000 7fdb7645b240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:02:48.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.851+0000 7f17d184d700 1 Processor -- start 2024-03-18T19:02:48.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.851+0000 7f17d184d700 1 -- start start 2024-03-18T19:02:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.851+0000 7f17d184d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f17cc15a320 0x7f17cc0ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.851+0000 7f17d184d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f17cc071d10 0x7f17cc06e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.851+0000 7f17d184d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17cc0acd80 con 0x7f17cc0cea30 2024-03-18T19:02:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.851+0000 7f17d184d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17cc0cf220 con 0x7f17cc071d10 2024-03-18T19:02:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.851+0000 7f17d184d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17cc0cef40 con 0x7f17cc15a320 2024-03-18T19:02:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.852+0000 7f17cb7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f17cc071d10 0x7f17cc06e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.852+0000 7f17ca7fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f17cc0cea30 0x7f17cc071600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37400/0 (socket says 172.21.15.1:37400) 2024-03-18T19:02:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.852+0000 7f17ca7fc700 1 -- 172.21.15.1:0/855706817 learned_addr learned my addr 172.21.15.1:0/855706817 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.852+0000 7f17caffd700 1 --2- 172.21.15.1:0/855706817 >> v2:172.21.15.165:3300/0 conn(0x7f17cc15a320 0x7f17cc0ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.852+0000 7f17cb7fe700 1 -- 172.21.15.1:0/855706817 >> v2:172.21.15.165:3300/0 conn(0x7f17cc15a320 msgr2=0x7f17cc0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.852+0000 7f17cb7fe700 1 --2- 172.21.15.1:0/855706817 >> v2:172.21.15.165:3300/0 conn(0x7f17cc15a320 0x7f17cc0ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.852+0000 7f17cb7fe700 1 -- 172.21.15.1:0/855706817 >> v1:172.21.15.1:6789/0 conn(0x7f17cc0cea30 legacy=0x7f17cc071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.852+0000 7f17cb7fe700 1 -- 172.21.15.1:0/855706817 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17cc06d7e0 con 0x7f17cc071d10 2024-03-18T19:02:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.852+0000 7f17c9ffb700 1 -- 172.21.15.1:0/855706817 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157600235 0 0) 0x7f17cc0acd80 con 0x7f17cc0cea30 2024-03-18T19:02:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.852+0000 7f17cb7fe700 1 --2- 172.21.15.1:0/855706817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17cc071d10 0x7f17cc06e2c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f17cc067260 tx=0x7f17c400d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c28d436e865a617e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.852+0000 7f17c9ffb700 1 -- 172.21.15.1:0/855706817 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4010070 con 0x7f17cc071d10 2024-03-18T19:02:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.852+0000 7f17c9ffb700 1 -- 172.21.15.1:0/855706817 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c400b2e0 con 0x7f17cc071d10 2024-03-18T19:02:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.852+0000 7f17c9ffb700 1 -- 172.21.15.1:0/855706817 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c400dc10 con 0x7f17cc071d10 2024-03-18T19:02:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.853+0000 7f17d184d700 1 -- 172.21.15.1:0/855706817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17cc071d10 msgr2=0x7f17cc06e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.853+0000 7f17d184d700 1 --2- 172.21.15.1:0/855706817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17cc071d10 0x7f17cc06e2c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f17cc067260 tx=0x7f17c400d330 comp rx=0 tx=0).stop 2024-03-18T19:02:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.853+0000 7f17d184d700 1 -- 172.21.15.1:0/855706817 shutdown_connections 2024-03-18T19:02:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.853+0000 7f17d184d700 1 --2- 172.21.15.1:0/855706817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17cc071d10 0x7f17cc06e2c0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.853+0000 7f17d184d700 1 --2- 172.21.15.1:0/855706817 >> v2:172.21.15.165:3300/0 conn(0x7f17cc15a320 0x7f17cc0ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.853+0000 7f17d184d700 1 -- 172.21.15.1:0/855706817 >> 172.21.15.1:0/855706817 conn(0x7f17cc0cc990 msgr2=0x7f17cc0ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.853+0000 7f17d184d700 1 -- 172.21.15.1:0/855706817 shutdown_connections 2024-03-18T19:02:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.853+0000 7f17d184d700 1 -- 172.21.15.1:0/855706817 wait complete. 2024-03-18T19:02:48.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.854+0000 7f17d184d700 1 Processor -- start 2024-03-18T19:02:48.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.854+0000 7f17d184d700 1 -- start start 2024-03-18T19:02:48.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.855+0000 7f17d184d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17cc071d10 0x7f17cc1511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:48.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.855+0000 7f17d184d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17cc0cea30 0x7f17cc0b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:48.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.855+0000 7f17d184d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17cc0b2ee0 0x7f17cc0b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:48.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.855+0000 7f17d184d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc0c4440 con 0x7f17cc0b2ee0 2024-03-18T19:02:48.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.855+0000 7f17d184d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc0b33f0 con 0x7f17cc071d10 2024-03-18T19:02:48.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.855+0000 7f17d184d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc0c46b0 con 0x7f17cc0cea30 2024-03-18T19:02:48.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.855+0000 7f17cb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17cc0b2ee0 0x7f17cc0b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:48.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.855+0000 7f17cb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17cc0b2ee0 0x7f17cc0b4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34556/0 (socket says 172.21.15.1:34556) 2024-03-18T19:02:48.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.855+0000 7f17cb7fe700 1 -- 172.21.15.1:0/3999293528 learned_addr learned my addr 172.21.15.1:0/3999293528 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:48.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.855+0000 7f17caffd700 1 --2- 172.21.15.1:0/3999293528 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17cc071d10 0x7f17cc1511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:48.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.856+0000 7f17ca7fc700 1 --2- 172.21.15.1:0/3999293528 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17cc0cea30 0x7f17cc0b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:48.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.856+0000 7f17cb7fe700 1 -- 172.21.15.1:0/3999293528 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17cc0cea30 msgr2=0x7f17cc0b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:48.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.856+0000 7f17cb7fe700 1 --2- 172.21.15.1:0/3999293528 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17cc0cea30 0x7f17cc0b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.856+0000 7f17cb7fe700 1 -- 172.21.15.1:0/3999293528 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17cc071d10 msgr2=0x7f17cc1511e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.856+0000 7f17cb7fe700 1 --2- 172.21.15.1:0/3999293528 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17cc071d10 0x7f17cc1511e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.856+0000 7f17cb7fe700 1 -- 172.21.15.1:0/3999293528 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c400bc80 con 0x7f17cc0b2ee0 2024-03-18T19:02:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.856+0000 7f17cb7fe700 1 --2- 172.21.15.1:0/3999293528 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17cc0b2ee0 0x7f17cc0b4ca0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f17cc067310 tx=0x7f17c4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.856+0000 7f17b3fff700 1 -- 172.21.15.1:0/3999293528 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4004450 con 0x7f17cc0b2ee0 2024-03-18T19:02:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.856+0000 7f17b3fff700 1 -- 172.21.15.1:0/3999293528 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c40045b0 con 0x7f17cc0b2ee0 2024-03-18T19:02:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.856+0000 7f17d184d700 1 -- 172.21.15.1:0/3999293528 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17cc0c4990 con 0x7f17cc0b2ee0 2024-03-18T19:02:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.856+0000 7f17b3fff700 1 -- 172.21.15.1:0/3999293528 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4004960 con 0x7f17cc0b2ee0 2024-03-18T19:02:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.856+0000 7f17d184d700 1 -- 172.21.15.1:0/3999293528 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17cc1573c0 con 0x7f17cc0b2ee0 2024-03-18T19:02:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.857+0000 7f17d184d700 1 -- 172.21.15.1:0/3999293528 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17cc0676a0 con 0x7f17cc0b2ee0 2024-03-18T19:02:48.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.858+0000 7f17b3fff700 1 -- 172.21.15.1:0/3999293528 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 14) v1 ==== 89308+0+0 (secure 0 0 0) 0x7f17c4026450 con 0x7f17cc0b2ee0 2024-03-18T19:02:48.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.858+0000 7f17b3fff700 1 --2- 172.21.15.1:0/3999293528 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f17b806ae40 0x7f17b806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:48.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.858+0000 7f17caffd700 1 -- 172.21.15.1:0/3999293528 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f17b806ae40 msgr2=0x7f17b806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:02:48.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.858+0000 7f17b3fff700 1 -- 172.21.15.1:0/3999293528 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6884+0+0 (secure 0 0 0) 0x7f17c408b8e0 con 0x7f17cc0b2ee0 2024-03-18T19:02:48.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.858+0000 7f17caffd700 1 --2- 172.21.15.1:0/3999293528 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f17b806ae40 0x7f17b806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:02:48.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.863+0000 7f17b3fff700 1 -- 172.21.15.1:0/3999293528 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f17c40562a0 con 0x7f17cc0b2ee0 2024-03-18T19:02:48.932 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:48.936+0000 7fdb7645b240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:02:48.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.956+0000 7f17b3fff700 1 -- 172.21.15.1:0/3999293528 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 15) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f17c4056670 con 0x7f17cc0b2ee0 2024-03-18T19:02:48.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.956+0000 7f17b3fff700 1 -- 172.21.15.1:0/3999293528 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f17b806ae40 msgr2=0x7f17b806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:02:48.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:48.956+0000 7f17b3fff700 1 --2- 172.21.15.1:0/3999293528 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f17b806ae40 0x7f17b806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.015 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:49.019+0000 7fdaf1889700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:02:49.016 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:49.019+0000 7fdaf1889700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:02:49.016 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:49.019+0000 7fdaf1889700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:02:49.016 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:49.019+0000 7fdaf1889700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:02:49.016 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:49.019+0000 7fdaf1889700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:02:49.016 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:49.020+0000 7fdaee082700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:02:49.016 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:49.020+0000 7fdaee082700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:02:49.016 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:49.020+0000 7fdaee082700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:02:49.017 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:49.020+0000 7fdaee082700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:02:49.017 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:02:49.020+0000 7fdaee082700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:02:49.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.134+0000 7f17d184d700 1 -- 172.21.15.1:0/3999293528 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17cc157550 con 0x7f17cc0b2ee0 2024-03-18T19:02:49.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.134+0000 7f17b3fff700 1 -- 172.21.15.1:0/3999293528 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4734 (secure 0 0 0) 0x7f17c405d2c0 con 0x7f17cc0b2ee0 2024-03-18T19:02:49.130 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:49.130 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4874,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2002167042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2002167042},{"type":"v1","addr":"172.21.15.165:6835","nonce":2002167042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4880,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3844556325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3844556325},{"type":"v1","addr":"172.21.15.1:6837","nonce":3844556325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5278,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1229938169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1229938169},{"type":"v1","addr":"172.21.15.165:6834","nonce":1229938169}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T18:51:12.721816+0000","modified":"2024-03-18T18:51:15.728690+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5436":{"gid":5436,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6836/3077516107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3077516107},{"type":"v1","addr":"172.21.15.1:6836","nonce":3077516107}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-18T19:02:49.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.136+0000 7f17d184d700 1 -- 172.21.15.1:0/3999293528 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17cc0b2ee0 msgr2=0x7f17cc0b4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:49.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.136+0000 7f17d184d700 1 --2- 172.21.15.1:0/3999293528 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17cc0b2ee0 0x7f17cc0b4ca0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f17cc067310 tx=0x7f17c4004ad0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.136+0000 7f17d184d700 1 -- 172.21.15.1:0/3999293528 shutdown_connections 2024-03-18T19:02:49.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.136+0000 7f17d184d700 1 --2- 172.21.15.1:0/3999293528 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f17b806ae40 0x7f17b806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.136+0000 7f17d184d700 1 --2- 172.21.15.1:0/3999293528 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17cc0b2ee0 0x7f17cc0b4ca0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.136+0000 7f17d184d700 1 --2- 172.21.15.1:0/3999293528 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17cc071d10 0x7f17cc1511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.136+0000 7f17d184d700 1 --2- 172.21.15.1:0/3999293528 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17cc0cea30 0x7f17cc0b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.136+0000 7f17d184d700 1 -- 172.21.15.1:0/3999293528 >> 172.21.15.1:0/3999293528 conn(0x7f17cc0cc990 msgr2=0x7f17cc072130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:49.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.137+0000 7f17d184d700 1 -- 172.21.15.1:0/3999293528 shutdown_connections 2024-03-18T19:02:49.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.137+0000 7f17d184d700 1 -- 172.21.15.1:0/3999293528 wait complete. 2024-03-18T19:02:49.134 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 17 2024-03-18T19:02:49.144 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:02:49.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.273+0000 7f1af0f42700 1 Processor -- start 2024-03-18T19:02:49.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.273+0000 7f1af0f42700 1 -- start start 2024-03-18T19:02:49.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.274+0000 7f1af0f42700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1aec0cf280 0x7f1aec0ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:49.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.274+0000 7f1af0f42700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aec06d640 con 0x7f1aec0cf960 2024-03-18T19:02:49.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.274+0000 7f1af0f42700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aec06de50 con 0x7f1aec06a750 2024-03-18T19:02:49.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.274+0000 7f1af0f42700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aec0acd80 con 0x7f1aec0cf280 2024-03-18T19:02:49.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.274+0000 7f1aead9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1aec0cf280 0x7f1aec0ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:49.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.274+0000 7f1ae9d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1aec0cf960 0x7f1aec0cdda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37414/0 (socket says 172.21.15.1:37414) 2024-03-18T19:02:49.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.274+0000 7f1ae9d9b700 1 -- 172.21.15.1:0/3162408591 learned_addr learned my addr 172.21.15.1:0/3162408591 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:49.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.274+0000 7f1aea59c700 1 --1- 172.21.15.1:0/3162408591 >> v1:172.21.15.165:6789/0 conn(0x7f1aec06a750 0x7f1aec06ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:38778/0 (socket says 172.21.15.1:38778) 2024-03-18T19:02:49.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.274+0000 7f1aead9d700 1 -- 172.21.15.1:0/3162408591 >> v1:172.21.15.165:6789/0 conn(0x7f1aec06a750 legacy=0x7f1aec06ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:49.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.274+0000 7f1aead9d700 1 -- 172.21.15.1:0/3162408591 >> v1:172.21.15.1:6789/0 conn(0x7f1aec0cf960 legacy=0x7f1aec0cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:49.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.274+0000 7f1aead9d700 1 -- 172.21.15.1:0/3162408591 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aec06d890 con 0x7f1aec0cf280 2024-03-18T19:02:49.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.274+0000 7f1ae959a700 1 -- 172.21.15.1:0/3162408591 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21962224 0 0) 0x7f1aec06d640 con 0x7f1aec0cf960 2024-03-18T19:02:49.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.274+0000 7f1aead9d700 1 --2- 172.21.15.1:0/3162408591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aec0cf280 0x7f1aec0ce4b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4009d90 tx=0x7f1ae400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b223f0dfe2a904ab server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:49.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.275+0000 7f1ae959a700 1 -- 172.21.15.1:0/3162408591 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae400e070 con 0x7f1aec0cf280 2024-03-18T19:02:49.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.275+0000 7f1ae959a700 1 -- 172.21.15.1:0/3162408591 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae400bbd0 con 0x7f1aec0cf280 2024-03-18T19:02:49.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.275+0000 7f1ae959a700 1 -- 172.21.15.1:0/3162408591 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4012430 con 0x7f1aec0cf280 2024-03-18T19:02:49.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.275+0000 7f1af0f42700 1 -- 172.21.15.1:0/3162408591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aec0cf280 msgr2=0x7f1aec0ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:49.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.275+0000 7f1af0f42700 1 --2- 172.21.15.1:0/3162408591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aec0cf280 0x7f1aec0ce4b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4009d90 tx=0x7f1ae400b330 comp rx=0 tx=0).stop 2024-03-18T19:02:49.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.275+0000 7f1af0f42700 1 -- 172.21.15.1:0/3162408591 shutdown_connections 2024-03-18T19:02:49.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.275+0000 7f1af0f42700 1 --2- 172.21.15.1:0/3162408591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aec0cf280 0x7f1aec0ce4b0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.275+0000 7f1af0f42700 1 -- 172.21.15.1:0/3162408591 >> 172.21.15.1:0/3162408591 conn(0x7f1aec06b860 msgr2=0x7f1aec06bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:49.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.275+0000 7f1af0f42700 1 -- 172.21.15.1:0/3162408591 shutdown_connections 2024-03-18T19:02:49.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.275+0000 7f1af0f42700 1 -- 172.21.15.1:0/3162408591 wait complete. 2024-03-18T19:02:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.277+0000 7f1af0f42700 1 Processor -- start 2024-03-18T19:02:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.277+0000 7f1af0f42700 1 -- start start 2024-03-18T19:02:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.277+0000 7f1af0f42700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aec06a750 0x7f1aec151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.277+0000 7f1af0f42700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1aec0cf280 0x7f1aec14ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.277+0000 7f1af0f42700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1aec0cf960 0x7f1aec15dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.277+0000 7f1af0f42700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec13e590 con 0x7f1aec06a750 2024-03-18T19:02:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.277+0000 7f1af0f42700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec14f2b0 con 0x7f1aec0cf960 2024-03-18T19:02:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.277+0000 7f1af0f42700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec159e30 con 0x7f1aec0cf280 2024-03-18T19:02:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.277+0000 7f1aea59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aec06a750 0x7f1aec151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.277+0000 7f1aea59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aec06a750 0x7f1aec151d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34680/0 (socket says 172.21.15.1:34680) 2024-03-18T19:02:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.277+0000 7f1aea59c700 1 -- 172.21.15.1:0/2250734865 learned_addr learned my addr 172.21.15.1:0/2250734865 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.278+0000 7f1aead9d700 1 --2- 172.21.15.1:0/2250734865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1aec0cf960 0x7f1aec15dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.278+0000 7f1ae9d9b700 1 --2- 172.21.15.1:0/2250734865 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1aec0cf280 0x7f1aec14ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.278+0000 7f1aea59c700 1 -- 172.21.15.1:0/2250734865 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1aec0cf280 msgr2=0x7f1aec14ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.278+0000 7f1aea59c700 1 --2- 172.21.15.1:0/2250734865 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1aec0cf280 0x7f1aec14ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.278+0000 7f1aea59c700 1 -- 172.21.15.1:0/2250734865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1aec0cf960 msgr2=0x7f1aec15dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.278+0000 7f1aea59c700 1 --2- 172.21.15.1:0/2250734865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1aec0cf960 0x7f1aec15dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.278+0000 7f1aea59c700 1 -- 172.21.15.1:0/2250734865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae4009a70 con 0x7f1aec06a750 2024-03-18T19:02:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.278+0000 7f1aea59c700 1 --2- 172.21.15.1:0/2250734865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aec06a750 0x7f1aec151d60 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f1adc00c520 tx=0x7f1adc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.278+0000 7f1adaffd700 1 -- 172.21.15.1:0/2250734865 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1adc005cb0 con 0x7f1aec06a750 2024-03-18T19:02:49.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.278+0000 7f1adaffd700 1 -- 172.21.15.1:0/2250734865 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1adc005e10 con 0x7f1aec06a750 2024-03-18T19:02:49.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.278+0000 7f1adaffd700 1 -- 172.21.15.1:0/2250734865 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1adc0072d0 con 0x7f1aec06a750 2024-03-18T19:02:49.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.278+0000 7f1af0f42700 1 -- 172.21.15.1:0/2250734865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aec15a110 con 0x7f1aec06a750 2024-03-18T19:02:49.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.278+0000 7f1af0f42700 1 -- 172.21.15.1:0/2250734865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aec15a6d0 con 0x7f1aec06a750 2024-03-18T19:02:49.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.279+0000 7f1af0f42700 1 -- 172.21.15.1:0/2250734865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aec0cd830 con 0x7f1aec06a750 2024-03-18T19:02:49.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.280+0000 7f1adaffd700 1 -- 172.21.15.1:0/2250734865 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 15) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f1adc007430 con 0x7f1aec06a750 2024-03-18T19:02:49.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.280+0000 7f1adaffd700 1 -- 172.21.15.1:0/2250734865 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7141+0+0 (secure 0 0 0) 0x7f1adc088720 con 0x7f1aec06a750 2024-03-18T19:02:49.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.285+0000 7f1adaffd700 1 -- 172.21.15.1:0/2250734865 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1adc053860 con 0x7f1aec06a750 2024-03-18T19:02:49.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.492+0000 7f1af0f42700 1 -- 172.21.15.1:0/2250734865 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1aec0676a0 con 0x7f1aec06a750 2024-03-18T19:02:49.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.493+0000 7f1adaffd700 1 -- 172.21.15.1:0/2250734865 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+17904 (secure 0 0 0) 0x7f1adc017090 con 0x7f1aec06a750 2024-03-18T19:02:49.490 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:49.490 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":59,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:02:48.950570+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-18T18:51:11.076221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-18T18:51:11.687105+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.26","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/1898862301":"2024-03-19T19:02:48.950531+0000","172.21.15.1:0/24360383":"2024-03-19T19:02:48.950531+0000","172.21.15.1:0/2729435891":"2024-03-19T19:02:48.950531+0000","172.21.15.1:0/2281250976":"2024-03-19T19:02:48.950531+0000","172.21.15.1:0/1120598162":"2024-03-19T19:02:48.950531+0000","172.21.15.1:0/72803206":"2024-03-19T18:51:27.537808+0000","172.21.15.1:0/1326267272":"2024-03-19T19:02:48.950531+0000","172.21.15.1:6833/76998":"2024-03-19T18:51:27.537808+0000","172.21.15.1:0/558567310":"2024-03-19T19:02:48.950531+0000","172.21.15.1:0/4026407635":"2024-03-19T18:51:27.537808+0000","172.21.15.1:0/2322425425":"2024-03-19T18:51:27.537808+0000","172.21.15.1:6832/76998":"2024-03-19T18:51:27.537808+0000","172.21.15.1:0/1992330207":"2024-03-19T18:51:27.537808+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":6,"snaps":[{"begin":3,"length":2},{"begin":7,"length":1},{"begin":9,"length":1},{"begin":13,"length":2},{"begin":17,"length":1},{"begin":19,"length":2},{"begin":24,"length":2},{"begin":28,"length":1},{"begin":30,"length":2},{"begin":34,"length":2},{"begin":38,"length":1},{"begin":40,"length":3},{"begin":45,"length":2},{"begin":49,"length":1},{"begin":51,"length":3},{"begin":56,"length":2},{"begin":60,"length":1},{"begin":63,"length":2},{"begin":67,"length":2},{"begin":71,"length":2},{"begin":74,"length":2},{"begin":78,"length":2},{"begin":82,"length":1},{"begin":84,"length":2},{"begin":88,"length":2},{"begin":93,"length":1},{"begin":95,"length":2},{"begin":99,"length":2},{"begin":103,"length":1},{"begin":105,"length":2},{"begin":109,"length":2},{"begin":168,"length":54}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:02:49.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.496+0000 7f1af0f42700 1 -- 172.21.15.1:0/2250734865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aec06a750 msgr2=0x7f1aec151d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:49.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.496+0000 7f1af0f42700 1 --2- 172.21.15.1:0/2250734865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aec06a750 0x7f1aec151d60 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f1adc00c520 tx=0x7f1adc00c8e0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.496+0000 7f1af0f42700 1 -- 172.21.15.1:0/2250734865 shutdown_connections 2024-03-18T19:02:49.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.496+0000 7f1af0f42700 1 --2- 172.21.15.1:0/2250734865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1aec0cf960 0x7f1aec15dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.496+0000 7f1af0f42700 1 --2- 172.21.15.1:0/2250734865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aec06a750 0x7f1aec151d60 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.496+0000 7f1af0f42700 1 --2- 172.21.15.1:0/2250734865 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1aec0cf280 0x7f1aec14ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.496+0000 7f1af0f42700 1 -- 172.21.15.1:0/2250734865 >> 172.21.15.1:0/2250734865 conn(0x7f1aec06b860 msgr2=0x7f1aec0b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:49.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.496+0000 7f1af0f42700 1 -- 172.21.15.1:0/2250734865 shutdown_connections 2024-03-18T19:02:49.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.496+0000 7f1af0f42700 1 -- 172.21.15.1:0/2250734865 wait complete. 2024-03-18T19:02:49.504 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T19:02:49.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.634+0000 7fe0c4463700 1 Processor -- start 2024-03-18T19:02:49.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.635+0000 7fe0c4463700 1 -- start start 2024-03-18T19:02:49.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.636+0000 7fe0c4463700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe0bc15d5a0 0x7fe0bc0ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:49.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.636+0000 7fe0c4463700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe0bc0ce550 0x7fe0bc06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.636+0000 7fe0c4463700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0bc0cf600 con 0x7fe0bc0cfa90 2024-03-18T19:02:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.636+0000 7fe0c4463700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0bc0ce9f0 con 0x7fe0bc0ce550 2024-03-18T19:02:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.636+0000 7fe0c4463700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0bc0cecd0 con 0x7fe0bc15d5a0 2024-03-18T19:02:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.636+0000 7fe0c19fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe0bc0ce550 0x7fe0bc06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.636+0000 7fe0c19fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe0bc0ce550 0x7fe0bc06a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34692/0 (socket says 172.21.15.1:34692) 2024-03-18T19:02:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.636+0000 7fe0c19fe700 1 -- 172.21.15.1:0/3093370818 learned_addr learned my addr 172.21.15.1:0/3093370818 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.636+0000 7fe0c19fe700 1 -- 172.21.15.1:0/3093370818 >> v2:172.21.15.165:3300/0 conn(0x7fe0bc15d5a0 msgr2=0x7fe0bc0ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.636+0000 7fe0c21ff700 1 --2- 172.21.15.1:0/3093370818 >> v2:172.21.15.165:3300/0 conn(0x7fe0bc15d5a0 0x7fe0bc0ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.636+0000 7fe0c19fe700 1 --2- 172.21.15.1:0/3093370818 >> v2:172.21.15.165:3300/0 conn(0x7fe0bc15d5a0 0x7fe0bc0ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.636+0000 7fe0c19fe700 1 -- 172.21.15.1:0/3093370818 >> v1:172.21.15.165:6789/0 conn(0x7fe0bc0cfa90 legacy=0x7fe0bc071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.636+0000 7fe0c19fe700 1 -- 172.21.15.1:0/3093370818 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0bc0cefb0 con 0x7fe0bc0ce550 2024-03-18T19:02:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.636+0000 7fe0c21ff700 1 --2- 172.21.15.1:0/3093370818 >> v2:172.21.15.165:3300/0 conn(0x7fe0bc15d5a0 0x7fe0bc0ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:02:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.636+0000 7fe0c19fe700 1 --2- 172.21.15.1:0/3093370818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0bc0ce550 0x7fe0bc06a750 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc072f60 tx=0x7fe0ac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d13de3bf28f19c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.636+0000 7fe0c11fd700 1 -- 172.21.15.1:0/3093370818 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac004230 con 0x7fe0bc0ce550 2024-03-18T19:02:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.636+0000 7fe0c11fd700 1 -- 172.21.15.1:0/3093370818 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0ac004390 con 0x7fe0bc0ce550 2024-03-18T19:02:49.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.636+0000 7fe0c11fd700 1 -- 172.21.15.1:0/3093370818 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac0036a0 con 0x7fe0bc0ce550 2024-03-18T19:02:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.637+0000 7fe0c4463700 1 -- 172.21.15.1:0/3093370818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0bc0ce550 msgr2=0x7fe0bc06a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.637+0000 7fe0c4463700 1 --2- 172.21.15.1:0/3093370818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0bc0ce550 0x7fe0bc06a750 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc072f60 tx=0x7fe0ac009bd0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.637+0000 7fe0c4463700 1 -- 172.21.15.1:0/3093370818 shutdown_connections 2024-03-18T19:02:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.637+0000 7fe0c4463700 1 --2- 172.21.15.1:0/3093370818 >> v2:172.21.15.165:3300/0 conn(0x7fe0bc15d5a0 0x7fe0bc0ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.637+0000 7fe0c4463700 1 --2- 172.21.15.1:0/3093370818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0bc0ce550 0x7fe0bc06a750 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.637+0000 7fe0c4463700 1 -- 172.21.15.1:0/3093370818 >> 172.21.15.1:0/3093370818 conn(0x7fe0bc0cc9b0 msgr2=0x7fe0bc0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.637+0000 7fe0c4463700 1 -- 172.21.15.1:0/3093370818 shutdown_connections 2024-03-18T19:02:49.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.637+0000 7fe0c4463700 1 -- 172.21.15.1:0/3093370818 wait complete. 2024-03-18T19:02:49.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.638+0000 7fe0c4463700 1 Processor -- start 2024-03-18T19:02:49.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.638+0000 7fe0c4463700 1 -- start start 2024-03-18T19:02:49.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.639+0000 7fe0c4463700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0bc0ce550 0x7fe0bc0b3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:49.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.639+0000 7fe0c4463700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0bc0cfa90 0x7fe0bc1f6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:49.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.639+0000 7fe0c4463700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0bc15d5a0 0x7fe0bc15ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:49.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.639+0000 7fe0c4463700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc0b3d10 con 0x7fe0bc0cfa90 2024-03-18T19:02:49.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.639+0000 7fe0c4463700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc15e4d0 con 0x7fe0bc15d5a0 2024-03-18T19:02:49.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.639+0000 7fe0c4463700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc15e770 con 0x7fe0bc0ce550 2024-03-18T19:02:49.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.639+0000 7fe0c19fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0bc0cfa90 0x7fe0bc1f6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:49.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.639+0000 7fe0c19fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0bc0cfa90 0x7fe0bc1f6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34700/0 (socket says 172.21.15.1:34700) 2024-03-18T19:02:49.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.639+0000 7fe0c19fe700 1 -- 172.21.15.1:0/2726281303 learned_addr learned my addr 172.21.15.1:0/2726281303 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:49.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.640+0000 7fe0c2a00700 1 --2- 172.21.15.1:0/2726281303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0bc15d5a0 0x7fe0bc15ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:49.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.640+0000 7fe0c21ff700 1 --2- 172.21.15.1:0/2726281303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0bc0ce550 0x7fe0bc0b3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:49.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.640+0000 7fe0c19fe700 1 -- 172.21.15.1:0/2726281303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0bc0ce550 msgr2=0x7fe0bc0b3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:49.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.640+0000 7fe0c19fe700 1 --2- 172.21.15.1:0/2726281303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0bc0ce550 0x7fe0bc0b3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.640+0000 7fe0c19fe700 1 -- 172.21.15.1:0/2726281303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0bc15d5a0 msgr2=0x7fe0bc15ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:49.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.640+0000 7fe0c19fe700 1 --2- 172.21.15.1:0/2726281303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0bc15d5a0 0x7fe0bc15ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.640+0000 7fe0c19fe700 1 -- 172.21.15.1:0/2726281303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac009550 con 0x7fe0bc0cfa90 2024-03-18T19:02:49.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.640+0000 7fe0c21ff700 1 --2- 172.21.15.1:0/2726281303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0bc0ce550 0x7fe0bc0b3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:02:49.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.640+0000 7fe0c19fe700 1 --2- 172.21.15.1:0/2726281303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0bc0cfa90 0x7fe0bc1f6f30 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc06cfd0 tx=0x7fe0ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:49.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.640+0000 7fe0b2ffd700 1 -- 172.21.15.1:0/2726281303 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac003d80 con 0x7fe0bc0cfa90 2024-03-18T19:02:49.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.640+0000 7fe0b2ffd700 1 -- 172.21.15.1:0/2726281303 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0ac01f070 con 0x7fe0bc0cfa90 2024-03-18T19:02:49.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.640+0000 7fe0c4463700 1 -- 172.21.15.1:0/2726281303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0bc0b40b0 con 0x7fe0bc0cfa90 2024-03-18T19:02:49.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.640+0000 7fe0b2ffd700 1 -- 172.21.15.1:0/2726281303 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac011870 con 0x7fe0bc0cfa90 2024-03-18T19:02:49.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.640+0000 7fe0c4463700 1 -- 172.21.15.1:0/2726281303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0bc0c6b30 con 0x7fe0bc0cfa90 2024-03-18T19:02:49.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.641+0000 7fe0c4463700 1 -- 172.21.15.1:0/2726281303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0bc0676a0 con 0x7fe0bc0cfa90 2024-03-18T19:02:49.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.641+0000 7fe0b2ffd700 1 -- 172.21.15.1:0/2726281303 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 15) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fe0ac021030 con 0x7fe0bc0cfa90 2024-03-18T19:02:49.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.642+0000 7fe0b2ffd700 1 -- 172.21.15.1:0/2726281303 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7141+0+0 (secure 0 0 0) 0x7fe0ac08ab10 con 0x7fe0bc0cfa90 2024-03-18T19:02:49.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.646+0000 7fe0b2ffd700 1 -- 172.21.15.1:0/2726281303 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe0ac055ab0 con 0x7fe0bc0cfa90 2024-03-18T19:02:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.893+0000 7fe0c4463700 1 -- 172.21.15.1:0/2726281303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe0bc0676a0 con 0x7fe0bc0cfa90 2024-03-18T19:02:49.889 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-18T19:02:49.893+0000 7fbdbeb46700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T19:02:49.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.901+0000 7fe0b2ffd700 1 -- 172.21.15.1:0/2726281303 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe0ac0590d0 con 0x7fe0bc0cfa90 2024-03-18T19:02:49.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.903+0000 7fe0c4463700 1 -- 172.21.15.1:0/2726281303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0bc0cfa90 msgr2=0x7fe0bc1f6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:49.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.903+0000 7fe0c4463700 1 --2- 172.21.15.1:0/2726281303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0bc0cfa90 0x7fe0bc1f6f30 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc06cfd0 tx=0x7fe0ac004750 comp rx=0 tx=0).stop 2024-03-18T19:02:49.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.903+0000 7fe0c4463700 1 -- 172.21.15.1:0/2726281303 shutdown_connections 2024-03-18T19:02:49.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.903+0000 7fe0c4463700 1 --2- 172.21.15.1:0/2726281303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0bc0ce550 0x7fe0bc0b3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.903+0000 7fe0c4463700 1 --2- 172.21.15.1:0/2726281303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0bc15d5a0 0x7fe0bc15ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.903+0000 7fe0c4463700 1 --2- 172.21.15.1:0/2726281303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0bc0cfa90 0x7fe0bc1f6f30 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:49.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.903+0000 7fe0c4463700 1 -- 172.21.15.1:0/2726281303 >> 172.21.15.1:0/2726281303 conn(0x7fe0bc0cc9b0 msgr2=0x7fe0bc06a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:49.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.903+0000 7fe0c4463700 1 -- 172.21.15.1:0/2726281303 shutdown_connections 2024-03-18T19:02:49.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:49.903+0000 7fe0c4463700 1 -- 172.21.15.1:0/2726281303 wait complete. 2024-03-18T19:02:49.900 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T19:02:49.912 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T19:02:49.935 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T19:02:50.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.049+0000 7f15b1a57700 1 Processor -- start 2024-03-18T19:02:50.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.049+0000 7f15b1a57700 1 -- start start 2024-03-18T19:02:50.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.050+0000 7f15b1a57700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f15ac0b4810 0x7f15ac0d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:50.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.050+0000 7f15b1a57700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ac0b3370 con 0x7f15ac0cde70 2024-03-18T19:02:50.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.050+0000 7f15b1a57700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ac0b2760 con 0x7f15ac0cfb50 2024-03-18T19:02:50.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.050+0000 7f15b1a57700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15ac0b2a40 con 0x7f15ac0b4810 2024-03-18T19:02:50.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.050+0000 7f15ab7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f15ac0cde70 0x7f15ac0ce2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37432/0 (socket says 172.21.15.1:37432) 2024-03-18T19:02:50.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.050+0000 7f15ab7fe700 1 -- 172.21.15.1:0/3392971719 learned_addr learned my addr 172.21.15.1:0/3392971719 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:50.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.050+0000 7f15a9ffb700 1 -- 172.21.15.1:0/3392971719 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777109636 0 0) 0x7f15ac0b3370 con 0x7f15ac0cde70 2024-03-18T19:02:50.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.050+0000 7f15aa7fc700 1 --2- 172.21.15.1:0/3392971719 >> v2:172.21.15.165:3300/0 conn(0x7f15ac0b4810 0x7f15ac0d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:50.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.050+0000 7f15a9ffb700 1 -- 172.21.15.1:0/3392971719 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15940035b0 con 0x7f15ac0cde70 2024-03-18T19:02:50.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.050+0000 7f15a9ffb700 1 -- 172.21.15.1:0/3392971719 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 77670956 0 0) 0x7f15940035b0 con 0x7f15ac0cde70 2024-03-18T19:02:50.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.050+0000 7f15a9ffb700 1 -- 172.21.15.1:0/3392971719 >> v2:172.21.15.165:3300/0 conn(0x7f15ac0b4810 msgr2=0x7f15ac0d3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:50.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.050+0000 7f15a9ffb700 1 --2- 172.21.15.1:0/3392971719 >> v2:172.21.15.165:3300/0 conn(0x7f15ac0b4810 0x7f15ac0d3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:50.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.050+0000 7f15a9ffb700 1 -- 172.21.15.1:0/3392971719 >> v1:172.21.15.165:6789/0 conn(0x7f15ac0cfb50 legacy=0x7f15ac0cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:50.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.050+0000 7f15a9ffb700 1 -- 172.21.15.1:0/3392971719 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ac0b2d20 con 0x7f15ac0cde70 2024-03-18T19:02:50.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.050+0000 7f15a9ffb700 1 -- 172.21.15.1:0/3392971719 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f15a40019f0 con 0x7f15ac0cde70 2024-03-18T19:02:50.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.050+0000 7f15a9ffb700 1 -- 172.21.15.1:0/3392971719 >> v1:172.21.15.1:6789/0 conn(0x7f15ac0cde70 legacy=0x7f15ac0ce2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:50.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.050+0000 7f15a9ffb700 1 --2- 172.21.15.1:0/3392971719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1594003a00 0x7f1594003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:50.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.050+0000 7f15a9ffb700 1 --2- 172.21.15.1:0/3392971719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1594004850 0x7f1594004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:50.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.050+0000 7f15a9ffb700 1 --2- 172.21.15.1:0/3392971719 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1594005210 0x7f159400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:50.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.051+0000 7f15a9ffb700 1 -- 172.21.15.1:0/3392971719 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac0b2a40 con 0x7f1594004850 2024-03-18T19:02:50.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.051+0000 7f15a9ffb700 1 -- 172.21.15.1:0/3392971719 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159400c0b0 con 0x7f1594003a00 2024-03-18T19:02:50.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.051+0000 7f15a9ffb700 1 -- 172.21.15.1:0/3392971719 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f159400c270 con 0x7f1594005210 2024-03-18T19:02:50.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.051+0000 7f15aa7fc700 1 --2- 172.21.15.1:0/3392971719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1594004850 0x7f1594004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:50.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.051+0000 7f15aaffd700 1 --2- 172.21.15.1:0/3392971719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1594003a00 0x7f1594003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:50.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.051+0000 7f15ab7fe700 1 --2- 172.21.15.1:0/3392971719 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1594005210 0x7f159400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:50.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.051+0000 7f15aa7fc700 1 -- 172.21.15.1:0/3392971719 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1594005210 msgr2=0x7f159400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:50.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.051+0000 7f15aa7fc700 1 --2- 172.21.15.1:0/3392971719 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1594005210 0x7f159400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:50.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.051+0000 7f15aa7fc700 1 -- 172.21.15.1:0/3392971719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1594003a00 msgr2=0x7f1594003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:50.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.051+0000 7f15aa7fc700 1 --2- 172.21.15.1:0/3392971719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1594003a00 0x7f1594003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:50.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.051+0000 7f15ab7fe700 1 -- 172.21.15.1:0/3392971719 reap_dead start 2024-03-18T19:02:50.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.051+0000 7f15aa7fc700 1 -- 172.21.15.1:0/3392971719 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15940035b0 con 0x7f1594004850 2024-03-18T19:02:50.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.051+0000 7f15aa7fc700 1 --2- 172.21.15.1:0/3392971719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1594004850 0x7f1594004cd0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f159c009b60 tx=0x7f1594005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56272514ef4bd60a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:50.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.051+0000 7f15a9ffb700 1 -- 172.21.15.1:0/3392971719 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c00e6c0 con 0x7f1594004850 2024-03-18T19:02:50.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.051+0000 7f15a9ffb700 1 -- 172.21.15.1:0/3392971719 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159c00e820 con 0x7f1594004850 2024-03-18T19:02:50.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.051+0000 7f15a9ffb700 1 -- 172.21.15.1:0/3392971719 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c00eb90 con 0x7f1594004850 2024-03-18T19:02:50.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.052+0000 7f15b1a57700 1 -- 172.21.15.1:0/3392971719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1594004850 msgr2=0x7f1594004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:50.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.052+0000 7f15b1a57700 1 --2- 172.21.15.1:0/3392971719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1594004850 0x7f1594004cd0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f159c009b60 tx=0x7f1594005880 comp rx=0 tx=0).stop 2024-03-18T19:02:50.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.052+0000 7f15b1a57700 1 -- 172.21.15.1:0/3392971719 shutdown_connections 2024-03-18T19:02:50.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.052+0000 7f15b1a57700 1 --2- 172.21.15.1:0/3392971719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1594004850 0x7f1594004cd0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:50.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.052+0000 7f15b1a57700 1 -- 172.21.15.1:0/3392971719 >> 172.21.15.1:0/3392971719 conn(0x7f15ac0d3140 msgr2=0x7f15ac0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:50.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.052+0000 7f15b1a57700 1 -- 172.21.15.1:0/3392971719 shutdown_connections 2024-03-18T19:02:50.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.052+0000 7f15b1a57700 1 -- 172.21.15.1:0/3392971719 wait complete. 2024-03-18T19:02:50.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.053+0000 7f15b1a57700 1 Processor -- start 2024-03-18T19:02:50.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.053+0000 7f15b1a57700 1 -- start start 2024-03-18T19:02:50.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.054+0000 7f15b1a57700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1594004850 0x7f15ac14b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:50.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.054+0000 7f15b1a57700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15ac158410 0x7f15ac158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:50.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.054+0000 7f15b1a57700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15ac15a1a0 0x7f15ac15a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:50.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.054+0000 7f15b1a57700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac15bc50 con 0x7f15ac15a1a0 2024-03-18T19:02:50.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.054+0000 7f15b1a57700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac14bb10 con 0x7f15ac158410 2024-03-18T19:02:50.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.054+0000 7f15b1a57700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac0b8c30 con 0x7f1594004850 2024-03-18T19:02:50.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.054+0000 7f15ab7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15ac15a1a0 0x7f15ac15a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:50.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.054+0000 7f15aaffd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1594004850 0x7f15ac14b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:50.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.054+0000 7f15ab7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15ac15a1a0 0x7f15ac15a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34728/0 (socket says 172.21.15.1:34728) 2024-03-18T19:02:50.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.054+0000 7f15aaffd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1594004850 0x7f15ac14b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:46522/0 (socket says 172.21.15.1:46522) 2024-03-18T19:02:50.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.054+0000 7f15ab7fe700 1 -- 172.21.15.1:0/3579457484 learned_addr learned my addr 172.21.15.1:0/3579457484 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:50.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.054+0000 7f15aa7fc700 1 --2- 172.21.15.1:0/3579457484 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15ac158410 0x7f15ac158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:50.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.054+0000 7f15ab7fe700 1 -- 172.21.15.1:0/3579457484 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1594004850 msgr2=0x7f15ac14b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:50.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.054+0000 7f15ab7fe700 1 --2- 172.21.15.1:0/3579457484 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1594004850 0x7f15ac14b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:50.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.054+0000 7f15ab7fe700 1 -- 172.21.15.1:0/3579457484 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15ac158410 msgr2=0x7f15ac158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:50.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.054+0000 7f15ab7fe700 1 --2- 172.21.15.1:0/3579457484 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15ac158410 0x7f15ac158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:50.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.054+0000 7f15ab7fe700 1 -- 172.21.15.1:0/3579457484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ac15b040 con 0x7f15ac15a1a0 2024-03-18T19:02:50.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.055+0000 7f15ab7fe700 1 --2- 172.21.15.1:0/3579457484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15ac15a1a0 0x7f15ac15a650 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0c6a70 tx=0x7f15a400a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:50.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.055+0000 7f1593fff700 1 -- 172.21.15.1:0/3579457484 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a400ac80 con 0x7f15ac15a1a0 2024-03-18T19:02:50.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.055+0000 7f1593fff700 1 -- 172.21.15.1:0/3579457484 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a400ae00 con 0x7f15ac15a1a0 2024-03-18T19:02:50.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.055+0000 7f15b1a57700 1 -- 172.21.15.1:0/3579457484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ac15b320 con 0x7f15ac15a1a0 2024-03-18T19:02:50.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.055+0000 7f1593fff700 1 -- 172.21.15.1:0/3579457484 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a4013630 con 0x7f15ac15a1a0 2024-03-18T19:02:50.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.055+0000 7f15b1a57700 1 -- 172.21.15.1:0/3579457484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ac15b8e0 con 0x7f15ac15a1a0 2024-03-18T19:02:50.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.056+0000 7f15b1a57700 1 -- 172.21.15.1:0/3579457484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15ac073350 con 0x7f15ac15a1a0 2024-03-18T19:02:50.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.056+0000 7f1593fff700 1 -- 172.21.15.1:0/3579457484 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f15a4013790 con 0x7f15ac15a1a0 2024-03-18T19:02:50.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.057+0000 7f1593fff700 1 --2- 172.21.15.1:0/3579457484 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f159806aae0 0x7f159806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:50.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.057+0000 7f1593fff700 1 -- 172.21.15.1:0/3579457484 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7227+0+0 (secure 0 0 0) 0x7f15a4089fd0 con 0x7f15ac15a1a0 2024-03-18T19:02:50.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.057+0000 7f15aaffd700 1 --2- 172.21.15.1:0/3579457484 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f159806aae0 0x7f159806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:50.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.057+0000 7f15aaffd700 1 --2- 172.21.15.1:0/3579457484 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f159806aae0 0x7f159806cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0b1e70 tx=0x7f15a0009040 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:50.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.061+0000 7f1593fff700 1 -- 172.21.15.1:0/3579457484 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f15a4054f00 con 0x7f15ac15a1a0 2024-03-18T19:02:50.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.307+0000 7f15b1a57700 1 -- 172.21.15.1:0/3579457484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f15ac067b30 con 0x7f15ac15a1a0 2024-03-18T19:02:50.440 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:02:50.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:02:50.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.962+0000 7f1593fff700 1 -- 172.21.15.1:0/3579457484 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f15a4017020 con 0x7f15ac15a1a0 2024-03-18T19:02:50.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.965+0000 7f15b1a57700 1 -- 172.21.15.1:0/3579457484 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f159806aae0 msgr2=0x7f159806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:50.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.965+0000 7f15b1a57700 1 --2- 172.21.15.1:0/3579457484 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f159806aae0 0x7f159806cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0b1e70 tx=0x7f15a0009040 comp rx=0 tx=0).stop 2024-03-18T19:02:50.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.965+0000 7f15b1a57700 1 -- 172.21.15.1:0/3579457484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15ac15a1a0 msgr2=0x7f15ac15a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:50.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.965+0000 7f15b1a57700 1 --2- 172.21.15.1:0/3579457484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15ac15a1a0 0x7f15ac15a650 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0c6a70 tx=0x7f15a400a5f0 comp rx=0 tx=0).stop 2024-03-18T19:02:50.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.965+0000 7f15b1a57700 1 -- 172.21.15.1:0/3579457484 shutdown_connections 2024-03-18T19:02:50.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.965+0000 7f15b1a57700 1 --2- 172.21.15.1:0/3579457484 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f159806aae0 0x7f159806cfa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:50.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.965+0000 7f15b1a57700 1 --2- 172.21.15.1:0/3579457484 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1594004850 0x7f15ac14b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:50.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.965+0000 7f15b1a57700 1 --2- 172.21.15.1:0/3579457484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15ac15a1a0 0x7f15ac15a650 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:50.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.965+0000 7f15b1a57700 1 --2- 172.21.15.1:0/3579457484 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15ac158410 0x7f15ac158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:50.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.965+0000 7f15b1a57700 1 -- 172.21.15.1:0/3579457484 >> 172.21.15.1:0/3579457484 conn(0x7f15ac0d3140 msgr2=0x7f15ac15e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:50.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.965+0000 7f15b1a57700 1 -- 172.21.15.1:0/3579457484 shutdown_connections 2024-03-18T19:02:50.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:50.965+0000 7f15b1a57700 1 -- 172.21.15.1:0/3579457484 wait complete. 2024-03-18T19:02:50.976 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T19:02:51.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.120+0000 7fb336304700 1 Processor -- start 2024-03-18T19:02:51.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.120+0000 7fb336304700 1 -- start start 2024-03-18T19:02:51.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb336304700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb33015a160 0x7fb3300b3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:51.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb336304700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3300b4d10 con 0x7fb33006f7d0 2024-03-18T19:02:51.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb336304700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3300b4a30 con 0x7fb33015e460 2024-03-18T19:02:51.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb336304700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33015ec10 con 0x7fb33015a160 2024-03-18T19:02:51.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb32f7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb33006f7d0 0x7fb33015dd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37436/0 (socket says 172.21.15.1:37436) 2024-03-18T19:02:51.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb32f7fe700 1 -- 172.21.15.1:0/3113739608 learned_addr learned my addr 172.21.15.1:0/3113739608 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:51.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb32effd700 1 -- 172.21.15.1:0/3113739608 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412327392 0 0) 0x7fb3300b4d10 con 0x7fb33006f7d0 2024-03-18T19:02:51.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb32ffff700 1 --2- 172.21.15.1:0/3113739608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb33015a160 0x7fb3300b3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:51.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb32effd700 1 -- 172.21.15.1:0/3113739608 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3180035b0 con 0x7fb33006f7d0 2024-03-18T19:02:51.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb32effd700 1 -- 172.21.15.1:0/3113739608 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3081831506 0 0) 0x7fb3180035b0 con 0x7fb33006f7d0 2024-03-18T19:02:51.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb32effd700 1 -- 172.21.15.1:0/3113739608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb33015a160 msgr2=0x7fb3300b3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:51.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb32effd700 1 --2- 172.21.15.1:0/3113739608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb33015a160 0x7fb3300b3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:51.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb32effd700 1 -- 172.21.15.1:0/3113739608 >> v1:172.21.15.165:6789/0 conn(0x7fb33015e460 legacy=0x7fb3300717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:51.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb32effd700 1 -- 172.21.15.1:0/3113739608 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33006ecf0 con 0x7fb33006f7d0 2024-03-18T19:02:51.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb32effd700 1 -- 172.21.15.1:0/3113739608 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fb324001670 con 0x7fb33006f7d0 2024-03-18T19:02:51.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb32effd700 1 -- 172.21.15.1:0/3113739608 >> v1:172.21.15.1:6789/0 conn(0x7fb33006f7d0 legacy=0x7fb33015dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:51.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb32effd700 1 --2- 172.21.15.1:0/3113739608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb318003a30 0x7fb318003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:51.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb32effd700 1 --2- 172.21.15.1:0/3113739608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb33015a160 0x7fb318004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:51.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb32effd700 1 --2- 172.21.15.1:0/3113739608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb318004d90 0x7fb318005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:51.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb32effd700 1 -- 172.21.15.1:0/3113739608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33015ec10 con 0x7fb318004d90 2024-03-18T19:02:51.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb32effd700 1 -- 172.21.15.1:0/3113739608 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31800bac0 con 0x7fb318003a30 2024-03-18T19:02:51.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.121+0000 7fb32effd700 1 -- 172.21.15.1:0/3113739608 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31800bc80 con 0x7fb33015a160 2024-03-18T19:02:51.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.122+0000 7fb3348a1700 1 --2- 172.21.15.1:0/3113739608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb318004d90 0x7fb318005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:51.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.122+0000 7fb32f7fe700 1 --2- 172.21.15.1:0/3113739608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb33015a160 0x7fb318004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:51.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.122+0000 7fb32ffff700 1 --2- 172.21.15.1:0/3113739608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb318003a30 0x7fb318003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:51.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.122+0000 7fb3348a1700 1 -- 172.21.15.1:0/3113739608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb33015a160 msgr2=0x7fb318004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:51.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.122+0000 7fb3348a1700 1 --2- 172.21.15.1:0/3113739608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb33015a160 0x7fb318004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:51.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.122+0000 7fb3348a1700 1 -- 172.21.15.1:0/3113739608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb318003a30 msgr2=0x7fb318003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:51.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.122+0000 7fb3348a1700 1 --2- 172.21.15.1:0/3113739608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb318003a30 0x7fb318003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:51.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.122+0000 7fb3348a1700 1 -- 172.21.15.1:0/3113739608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3180035b0 con 0x7fb318004d90 2024-03-18T19:02:51.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.122+0000 7fb3348a1700 1 --2- 172.21.15.1:0/3113739608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb318004d90 0x7fb318005210 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fb33006c6f0 tx=0x7fb318005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22ddcb439320f313 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:51.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.122+0000 7fb32effd700 1 -- 172.21.15.1:0/3113739608 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb328001db0 con 0x7fb318004d90 2024-03-18T19:02:51.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.122+0000 7fb32effd700 1 -- 172.21.15.1:0/3113739608 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32800cc70 con 0x7fb318004d90 2024-03-18T19:02:51.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.122+0000 7fb32effd700 1 -- 172.21.15.1:0/3113739608 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb328003a00 con 0x7fb318004d90 2024-03-18T19:02:51.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.123+0000 7fb336304700 1 -- 172.21.15.1:0/3113739608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb318004d90 msgr2=0x7fb318005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:51.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.123+0000 7fb336304700 1 --2- 172.21.15.1:0/3113739608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb318004d90 0x7fb318005210 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fb33006c6f0 tx=0x7fb318005880 comp rx=0 tx=0).stop 2024-03-18T19:02:51.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.123+0000 7fb3348a1700 1 -- 172.21.15.1:0/3113739608 reap_dead start 2024-03-18T19:02:51.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.123+0000 7fb336304700 1 -- 172.21.15.1:0/3113739608 shutdown_connections 2024-03-18T19:02:51.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.123+0000 7fb336304700 1 -- 172.21.15.1:0/3113739608 >> 172.21.15.1:0/3113739608 conn(0x7fb3300d00c0 msgr2=0x7fb3300d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:51.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.123+0000 7fb336304700 1 -- 172.21.15.1:0/3113739608 shutdown_connections 2024-03-18T19:02:51.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.123+0000 7fb336304700 1 -- 172.21.15.1:0/3113739608 wait complete. 2024-03-18T19:02:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.124+0000 7fb336304700 1 Processor -- start 2024-03-18T19:02:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.124+0000 7fb336304700 1 -- start start 2024-03-18T19:02:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.125+0000 7fb336304700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3301507a0 0x7fb330156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.125+0000 7fb336304700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb330158770 0x7fb330158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.125+0000 7fb336304700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3300cdae0 0x7fb3300cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.125+0000 7fb336304700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3301596c0 con 0x7fb3301507a0 2024-03-18T19:02:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.125+0000 7fb336304700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb330156ad0 con 0x7fb330158770 2024-03-18T19:02:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.125+0000 7fb336304700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb330156db0 con 0x7fb3300cdae0 2024-03-18T19:02:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.125+0000 7fb32ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3301507a0 0x7fb330156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.125+0000 7fb32ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3301507a0 0x7fb330156110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34762/0 (socket says 172.21.15.1:34762) 2024-03-18T19:02:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.125+0000 7fb32ffff700 1 -- 172.21.15.1:0/859415722 learned_addr learned my addr 172.21.15.1:0/859415722 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.125+0000 7fb3348a1700 1 --2- 172.21.15.1:0/859415722 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3300cdae0 0x7fb3300cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.125+0000 7fb32f7fe700 1 --2- 172.21.15.1:0/859415722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb330158770 0x7fb330158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.125+0000 7fb32ffff700 1 -- 172.21.15.1:0/859415722 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3300cdae0 msgr2=0x7fb3300cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.125+0000 7fb32ffff700 1 --2- 172.21.15.1:0/859415722 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3300cdae0 0x7fb3300cdf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.125+0000 7fb32ffff700 1 -- 172.21.15.1:0/859415722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb330158770 msgr2=0x7fb330158bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.125+0000 7fb32ffff700 1 --2- 172.21.15.1:0/859415722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb330158770 0x7fb330158bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.125+0000 7fb32ffff700 1 -- 172.21.15.1:0/859415722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb330157090 con 0x7fb3301507a0 2024-03-18T19:02:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.125+0000 7fb32f7fe700 1 --2- 172.21.15.1:0/859415722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb330158770 0x7fb330158bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:02:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.125+0000 7fb32ffff700 1 --2- 172.21.15.1:0/859415722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3301507a0 0x7fb330156110 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fb320011aa0 tx=0x7fb320011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:51.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.126+0000 7fb32cff9700 1 -- 172.21.15.1:0/859415722 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32000f980 con 0x7fb3301507a0 2024-03-18T19:02:51.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.126+0000 7fb32cff9700 1 -- 172.21.15.1:0/859415722 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32000fae0 con 0x7fb3301507a0 2024-03-18T19:02:51.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.126+0000 7fb336304700 1 -- 172.21.15.1:0/859415722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3300ced40 con 0x7fb3301507a0 2024-03-18T19:02:51.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.126+0000 7fb32cff9700 1 -- 172.21.15.1:0/859415722 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32000fe90 con 0x7fb3301507a0 2024-03-18T19:02:51.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.126+0000 7fb336304700 1 -- 172.21.15.1:0/859415722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb330159130 con 0x7fb3301507a0 2024-03-18T19:02:51.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.127+0000 7fb336304700 1 -- 172.21.15.1:0/859415722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3300cf390 con 0x7fb3301507a0 2024-03-18T19:02:51.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.127+0000 7fb32cff9700 1 -- 172.21.15.1:0/859415722 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fb320014440 con 0x7fb3301507a0 2024-03-18T19:02:51.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.127+0000 7fb32cff9700 1 --2- 172.21.15.1:0/859415722 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb31c06aae0 0x7fb31c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:51.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.128+0000 7fb32cff9700 1 -- 172.21.15.1:0/859415722 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7227+0+0 (secure 0 0 0) 0x7fb320089e30 con 0x7fb3301507a0 2024-03-18T19:02:51.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.128+0000 7fb32f7fe700 1 --2- 172.21.15.1:0/859415722 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb31c06aae0 0x7fb31c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:51.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.128+0000 7fb32f7fe700 1 --2- 172.21.15.1:0/859415722 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb31c06aae0 0x7fb31c06cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb324000c00 tx=0x7fb324009040 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:51.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.132+0000 7fb32cff9700 1 -- 172.21.15.1:0/859415722 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb320054d50 con 0x7fb3301507a0 2024-03-18T19:02:51.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.333+0000 7fb336304700 1 -- 172.21.15.1:0/859415722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb3300676a0 con 0x7fb3301507a0 2024-03-18T19:02:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:51.974+0000 7fb32cff9700 1 -- 172.21.15.1:0/859415722 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 17) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fb320055120 con 0x7fb3301507a0 2024-03-18T19:02:52.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.019+0000 7fb32cff9700 1 -- 172.21.15.1:0/859415722 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v61) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb3200587e0 con 0x7fb3301507a0 2024-03-18T19:02:52.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.023+0000 7fb336304700 1 -- 172.21.15.1:0/859415722 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb31c06aae0 msgr2=0x7fb31c06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:52.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.023+0000 7fb336304700 1 --2- 172.21.15.1:0/859415722 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb31c06aae0 0x7fb31c06cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb324000c00 tx=0x7fb324009040 comp rx=0 tx=0).stop 2024-03-18T19:02:52.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.023+0000 7fb336304700 1 -- 172.21.15.1:0/859415722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3301507a0 msgr2=0x7fb330156110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:52.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.023+0000 7fb336304700 1 --2- 172.21.15.1:0/859415722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3301507a0 0x7fb330156110 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fb320011aa0 tx=0x7fb320011e60 comp rx=0 tx=0).stop 2024-03-18T19:02:52.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.023+0000 7fb336304700 1 -- 172.21.15.1:0/859415722 shutdown_connections 2024-03-18T19:02:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.023+0000 7fb336304700 1 --2- 172.21.15.1:0/859415722 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb31c06aae0 0x7fb31c06cfa0 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb324000c00 tx=0x7fb324009040 comp rx=0 tx=0).stop 2024-03-18T19:02:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.023+0000 7fb336304700 1 --2- 172.21.15.1:0/859415722 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3300cdae0 0x7fb3300cdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.023+0000 7fb336304700 1 --2- 172.21.15.1:0/859415722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3301507a0 0x7fb330156110 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.023+0000 7fb336304700 1 --2- 172.21.15.1:0/859415722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb330158770 0x7fb330158bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.023+0000 7fb336304700 1 -- 172.21.15.1:0/859415722 >> 172.21.15.1:0/859415722 conn(0x7fb3300d00c0 msgr2=0x7fb33006e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.023+0000 7fb336304700 1 -- 172.21.15.1:0/859415722 shutdown_connections 2024-03-18T19:02:52.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.024+0000 7fb336304700 1 -- 172.21.15.1:0/859415722 wait complete. 2024-03-18T19:02:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-18T19:02:52.037 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T19:02:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.184+0000 7f311f191700 1 Processor -- start 2024-03-18T19:02:52.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.184+0000 7f311f191700 1 -- start start 2024-03-18T19:02:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.185+0000 7f311f191700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f311809a150 0x7f31180cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.185+0000 7f311f191700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31180ae260 con 0x7f311815b550 2024-03-18T19:02:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.185+0000 7f311f191700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31180d0150 con 0x7f311809a600 2024-03-18T19:02:52.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.185+0000 7f311f191700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31180cfb90 con 0x7f311809a150 2024-03-18T19:02:52.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.185+0000 7f311cf2d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f311815b550 0x7f3118099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37442/0 (socket says 172.21.15.1:37442) 2024-03-18T19:02:52.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.185+0000 7f311cf2d700 1 -- 172.21.15.1:0/2092912683 learned_addr learned my addr 172.21.15.1:0/2092912683 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:52.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f31177fe700 1 -- 172.21.15.1:0/2092912683 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134464066 0 0) 0x7f31180ae260 con 0x7f311815b550 2024-03-18T19:02:52.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f3117fff700 1 --2- 172.21.15.1:0/2092912683 >> v2:172.21.15.165:3300/0 conn(0x7f311809a150 0x7f31180cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:52.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f31177fe700 1 -- 172.21.15.1:0/2092912683 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31080035b0 con 0x7f311815b550 2024-03-18T19:02:52.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f31177fe700 1 -- 172.21.15.1:0/2092912683 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3971745483 0 0) 0x7f31080035b0 con 0x7f311815b550 2024-03-18T19:02:52.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f31177fe700 1 -- 172.21.15.1:0/2092912683 >> v2:172.21.15.165:3300/0 conn(0x7f311809a150 msgr2=0x7f31180cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:52.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f31177fe700 1 --2- 172.21.15.1:0/2092912683 >> v2:172.21.15.165:3300/0 conn(0x7f311809a150 0x7f31180cbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:52.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f31177fe700 1 -- 172.21.15.1:0/2092912683 >> v1:172.21.15.165:6789/0 conn(0x7f311809a600 legacy=0x7f31180cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:52.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f31177fe700 1 -- 172.21.15.1:0/2092912683 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31180cfe70 con 0x7f311815b550 2024-03-18T19:02:52.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f31177fe700 1 -- 172.21.15.1:0/2092912683 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f310c001670 con 0x7f311815b550 2024-03-18T19:02:52.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f31177fe700 1 -- 172.21.15.1:0/2092912683 >> v1:172.21.15.1:6789/0 conn(0x7f311815b550 legacy=0x7f3118099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:52.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f31177fe700 1 --2- 172.21.15.1:0/2092912683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3108003a00 0x7f3108003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:52.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f31177fe700 1 --2- 172.21.15.1:0/2092912683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3108004850 0x7f3108004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:52.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f31177fe700 1 --2- 172.21.15.1:0/2092912683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3108005210 0x7f310800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:52.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f31177fe700 1 -- 172.21.15.1:0/2092912683 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31180cfb90 con 0x7f3108004850 2024-03-18T19:02:52.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f31177fe700 1 -- 172.21.15.1:0/2092912683 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310800c0b0 con 0x7f3108005210 2024-03-18T19:02:52.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f31177fe700 1 -- 172.21.15.1:0/2092912683 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310800c270 con 0x7f3108003a00 2024-03-18T19:02:52.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f3117fff700 1 --2- 172.21.15.1:0/2092912683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3108004850 0x7f3108004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:52.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f311cf2d700 1 --2- 172.21.15.1:0/2092912683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3108003a00 0x7f3108003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:52.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f311d72e700 1 --2- 172.21.15.1:0/2092912683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3108005210 0x7f310800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:52.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f3117fff700 1 -- 172.21.15.1:0/2092912683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3108003a00 msgr2=0x7f3108003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:52.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f3117fff700 1 --2- 172.21.15.1:0/2092912683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3108003a00 0x7f3108003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:52.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f3117fff700 1 -- 172.21.15.1:0/2092912683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3108005210 msgr2=0x7f310800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:52.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f3117fff700 1 --2- 172.21.15.1:0/2092912683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3108005210 0x7f310800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:52.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.186+0000 7f3117fff700 1 -- 172.21.15.1:0/2092912683 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31080035b0 con 0x7f3108004850 2024-03-18T19:02:52.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.187+0000 7f311d72e700 1 --2- 172.21.15.1:0/2092912683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3108005210 0x7f310800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:52.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.187+0000 7f311d72e700 1 -- 172.21.15.1:0/2092912683 reap_dead start 2024-03-18T19:02:52.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.187+0000 7f311cf2d700 1 --2- 172.21.15.1:0/2092912683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3108003a00 0x7f3108003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:52.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.187+0000 7f3117fff700 1 --2- 172.21.15.1:0/2092912683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3108004850 0x7f3108004cd0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3104009b60 tx=0x7f3108005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5513f4c7ce338ca server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:52.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.187+0000 7f31177fe700 1 -- 172.21.15.1:0/2092912683 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310400e6c0 con 0x7f3108004850 2024-03-18T19:02:52.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.187+0000 7f31177fe700 1 -- 172.21.15.1:0/2092912683 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310400e820 con 0x7f3108004850 2024-03-18T19:02:52.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.187+0000 7f31177fe700 1 -- 172.21.15.1:0/2092912683 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310400eb90 con 0x7f3108004850 2024-03-18T19:02:52.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.187+0000 7f311f191700 1 -- 172.21.15.1:0/2092912683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3108004850 msgr2=0x7f3108004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:52.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.187+0000 7f311f191700 1 --2- 172.21.15.1:0/2092912683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3108004850 0x7f3108004cd0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3104009b60 tx=0x7f3108005880 comp rx=0 tx=0).stop 2024-03-18T19:02:52.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.187+0000 7f311f191700 1 -- 172.21.15.1:0/2092912683 shutdown_connections 2024-03-18T19:02:52.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.187+0000 7f311f191700 1 --2- 172.21.15.1:0/2092912683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3108004850 0x7f3108004cd0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:52.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.187+0000 7f311f191700 1 -- 172.21.15.1:0/2092912683 >> 172.21.15.1:0/2092912683 conn(0x7f31180b2c30 msgr2=0x7f31180b3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:52.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.187+0000 7f311f191700 1 -- 172.21.15.1:0/2092912683 shutdown_connections 2024-03-18T19:02:52.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.187+0000 7f311f191700 1 -- 172.21.15.1:0/2092912683 wait complete. 2024-03-18T19:02:52.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.189+0000 7f311f191700 1 Processor -- start 2024-03-18T19:02:52.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.189+0000 7f311f191700 1 -- start start 2024-03-18T19:02:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f311f191700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3108004850 0x7f311806cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f311f191700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f31180b6d50 0x7f31181f3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f311f191700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f311815dcb0 0x7f311815e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f311f191700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311815e6a0 con 0x7f3108004850 2024-03-18T19:02:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f311f191700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311806d250 con 0x7f311815dcb0 2024-03-18T19:02:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f311f191700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31181f4400 con 0x7f31180b6d50 2024-03-18T19:02:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f311cf2d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3108004850 0x7f311806cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f311cf2d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3108004850 0x7f311806cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34774/0 (socket says 172.21.15.1:34774) 2024-03-18T19:02:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f311cf2d700 1 -- 172.21.15.1:0/3453654552 learned_addr learned my addr 172.21.15.1:0/3453654552 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f311cf2d700 1 -- 172.21.15.1:0/3453654552 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f31180b6d50 msgr2=0x7f31181f3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f311cf2d700 1 --2- 172.21.15.1:0/3453654552 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f31180b6d50 0x7f31181f3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f311cf2d700 1 -- 172.21.15.1:0/3453654552 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f311815dcb0 msgr2=0x7f311815e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f311cf2d700 1 --2- 172.21.15.1:0/3453654552 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f311815dcb0 0x7f311815e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f311cf2d700 1 -- 172.21.15.1:0/3453654552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31180b5cb0 con 0x7f3108004850 2024-03-18T19:02:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f311cf2d700 1 --2- 172.21.15.1:0/3453654552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3108004850 0x7f311806cd10 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f310c00eaf0 tx=0x7f310c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f31157fa700 1 -- 172.21.15.1:0/3453654552 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c005e00 con 0x7f3108004850 2024-03-18T19:02:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f31157fa700 1 -- 172.21.15.1:0/3453654552 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310c00c7b0 con 0x7f3108004850 2024-03-18T19:02:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f311f191700 1 -- 172.21.15.1:0/3453654552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31180b5f90 con 0x7f3108004850 2024-03-18T19:02:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f31157fa700 1 -- 172.21.15.1:0/3453654552 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c00cb20 con 0x7f3108004850 2024-03-18T19:02:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.190+0000 7f311f191700 1 -- 172.21.15.1:0/3453654552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31180b6550 con 0x7f3108004850 2024-03-18T19:02:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.191+0000 7f311f191700 1 -- 172.21.15.1:0/3453654552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31180676a0 con 0x7f3108004850 2024-03-18T19:02:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.192+0000 7f31157fa700 1 -- 172.21.15.1:0/3453654552 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f310c00cc80 con 0x7f3108004850 2024-03-18T19:02:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.192+0000 7f31157fa700 1 --2- 172.21.15.1:0/3453654552 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f30fc06ad20 0x7f30fc06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.192+0000 7f31157fa700 1 -- 172.21.15.1:0/3453654552 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6789+0+0 (secure 0 0 0) 0x7f310c089620 con 0x7f3108004850 2024-03-18T19:02:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.192+0000 7f3117fff700 1 --2- 172.21.15.1:0/3453654552 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f30fc06ad20 0x7f30fc06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.193+0000 7f3117fff700 1 --2- 172.21.15.1:0/3453654552 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f30fc06ad20 0x7f30fc06d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f31040090b0 tx=0x7f3104013000 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.197+0000 7f31157fa700 1 -- 172.21.15.1:0/3453654552 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f310c0543d0 con 0x7f3108004850 2024-03-18T19:02:52.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:52.405+0000 7f311f191700 1 -- 172.21.15.1:0/3453654552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f311806b510 con 0x7f3108004850 2024-03-18T19:02:53.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.081+0000 7f31157fa700 1 -- 172.21.15.1:0/3453654552 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f310c0547a0 con 0x7f3108004850 2024-03-18T19:02:53.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.083+0000 7f31157fa700 1 -- 172.21.15.1:0/3453654552 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v62) v1 ==== 169+0+0 (secure 0 0 0) 0x7f310c087370 con 0x7f3108004850 2024-03-18T19:02:53.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.086+0000 7f311f191700 1 -- 172.21.15.1:0/3453654552 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f30fc06ad20 msgr2=0x7f30fc06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:53.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.086+0000 7f311f191700 1 --2- 172.21.15.1:0/3453654552 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f30fc06ad20 0x7f30fc06d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f31040090b0 tx=0x7f3104013000 comp rx=0 tx=0).stop 2024-03-18T19:02:53.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.086+0000 7f311f191700 1 -- 172.21.15.1:0/3453654552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3108004850 msgr2=0x7f311806cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:53.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.086+0000 7f311f191700 1 --2- 172.21.15.1:0/3453654552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3108004850 0x7f311806cd10 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f310c00eaf0 tx=0x7f310c00eeb0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.086+0000 7f311f191700 1 -- 172.21.15.1:0/3453654552 shutdown_connections 2024-03-18T19:02:53.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.086+0000 7f311f191700 1 --2- 172.21.15.1:0/3453654552 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f30fc06ad20 0x7f30fc06d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.086+0000 7f311f191700 1 --2- 172.21.15.1:0/3453654552 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f311815dcb0 0x7f311815e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.086+0000 7f311f191700 1 --2- 172.21.15.1:0/3453654552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3108004850 0x7f311806cd10 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.086+0000 7f311f191700 1 --2- 172.21.15.1:0/3453654552 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f31180b6d50 0x7f31181f3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.086+0000 7f311f191700 1 -- 172.21.15.1:0/3453654552 >> 172.21.15.1:0/3453654552 conn(0x7f31180b2c30 msgr2=0x7f311806f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:53.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.086+0000 7f311f191700 1 -- 172.21.15.1:0/3453654552 shutdown_connections 2024-03-18T19:02:53.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.086+0000 7f311f191700 1 -- 172.21.15.1:0/3453654552 wait complete. 2024-03-18T19:02:53.083 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-18T19:02:53.097 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-18T19:02:53.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.234+0000 7f98b1c28700 1 Processor -- start 2024-03-18T19:02:53.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.234+0000 7f98b1c28700 1 -- start start 2024-03-18T19:02:53.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.235+0000 7f98b1c28700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f98ac06efb0 0x7f98ac071da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.235+0000 7f98b1c28700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98ac0722e0 0x7f98ac0cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.235+0000 7f98b1c28700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ac06da90 con 0x7f98ac15d580 2024-03-18T19:02:53.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.235+0000 7f98b1c28700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98ac072770 con 0x7f98ac06efb0 2024-03-18T19:02:53.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.235+0000 7f98b1c28700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac15d960 con 0x7f98ac0722e0 2024-03-18T19:02:53.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.235+0000 7f98aaffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f98ac06efb0 0x7f98ac071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.235+0000 7f98ab7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f98ac15d580 0x7f98ac06d380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37450/0 (socket says 172.21.15.1:37450) 2024-03-18T19:02:53.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.235+0000 7f98aaffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f98ac06efb0 0x7f98ac071da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34786/0 (socket says 172.21.15.1:34786) 2024-03-18T19:02:53.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.235+0000 7f98ab7fe700 1 -- 172.21.15.1:0/818904634 learned_addr learned my addr 172.21.15.1:0/818904634 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:53.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.235+0000 7f98abfff700 1 --2- 172.21.15.1:0/818904634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98ac0722e0 0x7f98ac0cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.235+0000 7f98aaffd700 1 -- 172.21.15.1:0/818904634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98ac0722e0 msgr2=0x7f98ac0cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:53.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.235+0000 7f98aaffd700 1 --2- 172.21.15.1:0/818904634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98ac0722e0 0x7f98ac0cf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.235+0000 7f98aaffd700 1 -- 172.21.15.1:0/818904634 >> v1:172.21.15.1:6789/0 conn(0x7f98ac15d580 legacy=0x7f98ac06d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:53.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.235+0000 7f98aaffd700 1 -- 172.21.15.1:0/818904634 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac06ab30 con 0x7f98ac06efb0 2024-03-18T19:02:53.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.235+0000 7f98aa7fc700 1 -- 172.21.15.1:0/818904634 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297353398 0 0) 0x7f98ac06da90 con 0x7f98ac15d580 2024-03-18T19:02:53.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.236+0000 7f98aaffd700 1 --2- 172.21.15.1:0/818904634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98ac06efb0 0x7f98ac071da0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f989c009dc0 tx=0x7f989c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ae10bf24e45009 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:53.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.236+0000 7f98aa7fc700 1 -- 172.21.15.1:0/818904634 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c00e070 con 0x7f98ac06efb0 2024-03-18T19:02:53.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.236+0000 7f98aa7fc700 1 -- 172.21.15.1:0/818904634 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989c00bb90 con 0x7f98ac06efb0 2024-03-18T19:02:53.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.236+0000 7f98aa7fc700 1 -- 172.21.15.1:0/818904634 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c012430 con 0x7f98ac06efb0 2024-03-18T19:02:53.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.236+0000 7f98b1c28700 1 -- 172.21.15.1:0/818904634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98ac06efb0 msgr2=0x7f98ac071da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:53.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.236+0000 7f98b1c28700 1 --2- 172.21.15.1:0/818904634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98ac06efb0 0x7f98ac071da0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f989c009dc0 tx=0x7f989c00b330 comp rx=0 tx=0).stop 2024-03-18T19:02:53.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.236+0000 7f98b1c28700 1 -- 172.21.15.1:0/818904634 shutdown_connections 2024-03-18T19:02:53.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.236+0000 7f98b1c28700 1 --2- 172.21.15.1:0/818904634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98ac0722e0 0x7f98ac0cf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.236+0000 7f98b1c28700 1 --2- 172.21.15.1:0/818904634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98ac06efb0 0x7f98ac071da0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.236+0000 7f98b1c28700 1 -- 172.21.15.1:0/818904634 >> 172.21.15.1:0/818904634 conn(0x7f98ac0cca00 msgr2=0x7f98ac0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:53.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.236+0000 7f98b1c28700 1 -- 172.21.15.1:0/818904634 shutdown_connections 2024-03-18T19:02:53.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.236+0000 7f98b1c28700 1 -- 172.21.15.1:0/818904634 wait complete. 2024-03-18T19:02:53.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.238+0000 7f98b1c28700 1 Processor -- start 2024-03-18T19:02:53.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.238+0000 7f98b1c28700 1 -- start start 2024-03-18T19:02:53.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98b1c28700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98ac06efb0 0x7f98ac14e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98b1c28700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98ac0722e0 0x7f98ac15faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98b1c28700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98ac15d580 0x7f98ac160030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98b1c28700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac0c79b0 con 0x7f98ac0722e0 2024-03-18T19:02:53.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98b1c28700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac14edc0 con 0x7f98ac06efb0 2024-03-18T19:02:53.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98b1c28700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac0c7050 con 0x7f98ac15d580 2024-03-18T19:02:53.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98aaffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98ac0722e0 0x7f98ac15faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98aaffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98ac0722e0 0x7f98ac15faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34798/0 (socket says 172.21.15.1:34798) 2024-03-18T19:02:53.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98aaffd700 1 -- 172.21.15.1:0/2190707903 learned_addr learned my addr 172.21.15.1:0/2190707903 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:53.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98abfff700 1 --2- 172.21.15.1:0/2190707903 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98ac15d580 0x7f98ac160030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98ab7fe700 1 --2- 172.21.15.1:0/2190707903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98ac06efb0 0x7f98ac14e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98aaffd700 1 -- 172.21.15.1:0/2190707903 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98ac15d580 msgr2=0x7f98ac160030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:53.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98aaffd700 1 --2- 172.21.15.1:0/2190707903 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98ac15d580 0x7f98ac160030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98aaffd700 1 -- 172.21.15.1:0/2190707903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98ac06efb0 msgr2=0x7f98ac14e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:53.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98aaffd700 1 --2- 172.21.15.1:0/2190707903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98ac06efb0 0x7f98ac14e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98aaffd700 1 -- 172.21.15.1:0/2190707903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f989c009a70 con 0x7f98ac0722e0 2024-03-18T19:02:53.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98abfff700 1 --2- 172.21.15.1:0/2190707903 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98ac15d580 0x7f98ac160030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:53.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98ab7fe700 1 --2- 172.21.15.1:0/2190707903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98ac06efb0 0x7f98ac14e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:53.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98aaffd700 1 --2- 172.21.15.1:0/2190707903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98ac0722e0 0x7f98ac15faf0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f989c014040 tx=0x7f989c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:53.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98a3fff700 1 -- 172.21.15.1:0/2190707903 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c01b810 con 0x7f98ac0722e0 2024-03-18T19:02:53.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98a3fff700 1 -- 172.21.15.1:0/2190707903 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989c020070 con 0x7f98ac0722e0 2024-03-18T19:02:53.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98b1c28700 1 -- 172.21.15.1:0/2190707903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98ac15ae20 con 0x7f98ac0722e0 2024-03-18T19:02:53.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98a3fff700 1 -- 172.21.15.1:0/2190707903 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c01bbf0 con 0x7f98ac0722e0 2024-03-18T19:02:53.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.239+0000 7f98b1c28700 1 -- 172.21.15.1:0/2190707903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ac0c75d0 con 0x7f98ac0722e0 2024-03-18T19:02:53.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.240+0000 7f98b1c28700 1 -- 172.21.15.1:0/2190707903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98ac0676a0 con 0x7f98ac0722e0 2024-03-18T19:02:53.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.241+0000 7f98a3fff700 1 -- 172.21.15.1:0/2190707903 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f989c01bd50 con 0x7f98ac0722e0 2024-03-18T19:02:53.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.241+0000 7f98a3fff700 1 --2- 172.21.15.1:0/2190707903 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f989406adb0 0x7f989406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.241+0000 7f98a3fff700 1 -- 172.21.15.1:0/2190707903 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f989c022070 con 0x7f98ac0722e0 2024-03-18T19:02:53.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.241+0000 7f98ab7fe700 1 --2- 172.21.15.1:0/2190707903 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f989406adb0 0x7f989406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.242+0000 7f98ab7fe700 1 --2- 172.21.15.1:0/2190707903 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f989406adb0 0x7f989406d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9898000f60 tx=0x7f9898007ca0 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:53.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.246+0000 7f98a3fff700 1 -- 172.21.15.1:0/2190707903 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f989c05a8d0 con 0x7f98ac0722e0 2024-03-18T19:02:53.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.447+0000 7f98b1c28700 1 -- 172.21.15.1:0/2190707903 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f98ac0cd760 con 0x7f98ac0722e0 2024-03-18T19:02:53.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.451+0000 7f98a3fff700 1 -- 172.21.15.1:0/2190707903 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v13) v1 ==== 115+0+0 (secure 0 0 0) 0x7f989c05a460 con 0x7f98ac0722e0 2024-03-18T19:02:53.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.453+0000 7f98b1c28700 1 -- 172.21.15.1:0/2190707903 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f989406adb0 msgr2=0x7f989406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:53.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.453+0000 7f98b1c28700 1 --2- 172.21.15.1:0/2190707903 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f989406adb0 0x7f989406d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9898000f60 tx=0x7f9898007ca0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.453+0000 7f98b1c28700 1 -- 172.21.15.1:0/2190707903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98ac0722e0 msgr2=0x7f98ac15faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:53.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.453+0000 7f98b1c28700 1 --2- 172.21.15.1:0/2190707903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98ac0722e0 0x7f98ac15faf0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f989c014040 tx=0x7f989c004ad0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.454+0000 7f98b1c28700 1 -- 172.21.15.1:0/2190707903 shutdown_connections 2024-03-18T19:02:53.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.454+0000 7f98b1c28700 1 --2- 172.21.15.1:0/2190707903 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f989406adb0 0x7f989406d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.454+0000 7f98b1c28700 1 --2- 172.21.15.1:0/2190707903 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98ac15d580 0x7f98ac160030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.454+0000 7f98b1c28700 1 --2- 172.21.15.1:0/2190707903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98ac06efb0 0x7f98ac14e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.454+0000 7f98b1c28700 1 --2- 172.21.15.1:0/2190707903 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98ac0722e0 0x7f98ac15faf0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.454+0000 7f98b1c28700 1 -- 172.21.15.1:0/2190707903 >> 172.21.15.1:0/2190707903 conn(0x7f98ac0cca00 msgr2=0x7f98ac0afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:53.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.454+0000 7f98b1c28700 1 -- 172.21.15.1:0/2190707903 shutdown_connections 2024-03-18T19:02:53.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.454+0000 7f98b1c28700 1 -- 172.21.15.1:0/2190707903 wait complete. 2024-03-18T19:02:53.462 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-18T19:02:53.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.593+0000 7f9eda2cc700 1 Processor -- start 2024-03-18T19:02:53.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.593+0000 7f9eda2cc700 1 -- start start 2024-03-18T19:02:53.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.594+0000 7f9eda2cc700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9ed40aede0 0x7f9ed4159e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.594+0000 7f9eda2cc700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ed415e3c0 con 0x7f9ed406d950 2024-03-18T19:02:53.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.594+0000 7f9eda2cc700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ed40af1c0 con 0x7f9ed40b3320 2024-03-18T19:02:53.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.594+0000 7f9eda2cc700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed415d4d0 con 0x7f9ed40aede0 2024-03-18T19:02:53.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.594+0000 7f9ed37fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9ed406d950 0x7f9ed415dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37452/0 (socket says 172.21.15.1:37452) 2024-03-18T19:02:53.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.594+0000 7f9ed37fe700 1 -- 172.21.15.1:0/2577880606 learned_addr learned my addr 172.21.15.1:0/2577880606 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:53.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.594+0000 7f9ed3fff700 1 --2- 172.21.15.1:0/2577880606 >> v2:172.21.15.165:3300/0 conn(0x7f9ed40aede0 0x7f9ed4159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.594+0000 7f9ed2ffd700 1 -- 172.21.15.1:0/2577880606 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714412107 0 0) 0x7f9ed415e3c0 con 0x7f9ed406d950 2024-03-18T19:02:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.594+0000 7f9ed2ffd700 1 -- 172.21.15.1:0/2577880606 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ec00035b0 con 0x7f9ed406d950 2024-03-18T19:02:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed2ffd700 1 -- 172.21.15.1:0/2577880606 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3096824053 0 0) 0x7f9ec00035b0 con 0x7f9ed406d950 2024-03-18T19:02:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed2ffd700 1 -- 172.21.15.1:0/2577880606 >> v2:172.21.15.165:3300/0 conn(0x7f9ed40aede0 msgr2=0x7f9ed4159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed2ffd700 1 --2- 172.21.15.1:0/2577880606 >> v2:172.21.15.165:3300/0 conn(0x7f9ed40aede0 0x7f9ed4159e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed2ffd700 1 -- 172.21.15.1:0/2577880606 >> v1:172.21.15.165:6789/0 conn(0x7f9ed40b3320 legacy=0x7f9ed40b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed2ffd700 1 -- 172.21.15.1:0/2577880606 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed415a730 con 0x7f9ed406d950 2024-03-18T19:02:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed2ffd700 1 -- 172.21.15.1:0/2577880606 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f9ec4001670 con 0x7f9ed406d950 2024-03-18T19:02:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed2ffd700 1 -- 172.21.15.1:0/2577880606 >> v1:172.21.15.1:6789/0 conn(0x7f9ed406d950 legacy=0x7f9ed415dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed2ffd700 1 --2- 172.21.15.1:0/2577880606 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ec0003a00 0x7f9ec0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed2ffd700 1 --2- 172.21.15.1:0/2577880606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ec0004850 0x7f9ec0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed2ffd700 1 --2- 172.21.15.1:0/2577880606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ec0005210 0x7f9ec000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed2ffd700 1 -- 172.21.15.1:0/2577880606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed415d4d0 con 0x7f9ec0005210 2024-03-18T19:02:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed2ffd700 1 -- 172.21.15.1:0/2577880606 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec000c0b0 con 0x7f9ec0004850 2024-03-18T19:02:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed2ffd700 1 -- 172.21.15.1:0/2577880606 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec000d380 con 0x7f9ec0003a00 2024-03-18T19:02:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed8869700 1 --2- 172.21.15.1:0/2577880606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ec0005210 0x7f9ec000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed37fe700 1 --2- 172.21.15.1:0/2577880606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ec0004850 0x7f9ec0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed3fff700 1 --2- 172.21.15.1:0/2577880606 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ec0003a00 0x7f9ec0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed8869700 1 -- 172.21.15.1:0/2577880606 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ec0003a00 msgr2=0x7f9ec0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed8869700 1 --2- 172.21.15.1:0/2577880606 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ec0003a00 0x7f9ec0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed8869700 1 -- 172.21.15.1:0/2577880606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ec0004850 msgr2=0x7f9ec0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed8869700 1 --2- 172.21.15.1:0/2577880606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ec0004850 0x7f9ec0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed8869700 1 -- 172.21.15.1:0/2577880606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec00035b0 con 0x7f9ec0005210 2024-03-18T19:02:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed8869700 1 -- 172.21.15.1:0/2577880606 reap_dead start 2024-03-18T19:02:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed8869700 1 --2- 172.21.15.1:0/2577880606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ec0005210 0x7f9ec000ba10 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc00c190 tx=0x7f9ec0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a773e2943473f3bc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed2ffd700 1 -- 172.21.15.1:0/2577880606 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc001db0 con 0x7f9ec0005210 2024-03-18T19:02:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed2ffd700 1 -- 172.21.15.1:0/2577880606 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ecc00ccd0 con 0x7f9ec0005210 2024-03-18T19:02:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.595+0000 7f9ed2ffd700 1 -- 172.21.15.1:0/2577880606 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc003c40 con 0x7f9ec0005210 2024-03-18T19:02:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.596+0000 7f9eda2cc700 1 -- 172.21.15.1:0/2577880606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ec0005210 msgr2=0x7f9ec000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.596+0000 7f9eda2cc700 1 --2- 172.21.15.1:0/2577880606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ec0005210 0x7f9ec000ba10 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc00c190 tx=0x7f9ec0005880 comp rx=0 tx=0).stop 2024-03-18T19:02:53.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.596+0000 7f9eda2cc700 1 -- 172.21.15.1:0/2577880606 shutdown_connections 2024-03-18T19:02:53.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.596+0000 7f9eda2cc700 1 --2- 172.21.15.1:0/2577880606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ec0005210 0x7f9ec000ba10 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.596+0000 7f9eda2cc700 1 -- 172.21.15.1:0/2577880606 >> 172.21.15.1:0/2577880606 conn(0x7f9ed4066b70 msgr2=0x7f9ed406a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:53.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.596+0000 7f9eda2cc700 1 -- 172.21.15.1:0/2577880606 shutdown_connections 2024-03-18T19:02:53.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.596+0000 7f9eda2cc700 1 -- 172.21.15.1:0/2577880606 wait complete. 2024-03-18T19:02:53.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.597+0000 7f9eda2cc700 1 Processor -- start 2024-03-18T19:02:53.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.597+0000 7f9eda2cc700 1 -- start start 2024-03-18T19:02:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.598+0000 7f9eda2cc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ec0005210 0x7f9ed41f24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.598+0000 7f9eda2cc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ed41edbf0 0x7f9ed40b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.598+0000 7f9eda2cc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ed40b0a70 0x7f9ed4140d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.598+0000 7f9eda2cc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed40b0ef0 con 0x7f9ed41edbf0 2024-03-18T19:02:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.598+0000 7f9eda2cc700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed4141c60 con 0x7f9ed40b0a70 2024-03-18T19:02:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.598+0000 7f9eda2cc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed4141980 con 0x7f9ec0005210 2024-03-18T19:02:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.598+0000 7f9ed37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ed41edbf0 0x7f9ed40b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.598+0000 7f9ed37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ed41edbf0 0x7f9ed40b0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34816/0 (socket says 172.21.15.1:34816) 2024-03-18T19:02:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.598+0000 7f9ed37fe700 1 -- 172.21.15.1:0/2660145879 learned_addr learned my addr 172.21.15.1:0/2660145879 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.598+0000 7f9ed3fff700 1 --2- 172.21.15.1:0/2660145879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ec0005210 0x7f9ed41f24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.599+0000 7f9ed8869700 1 --2- 172.21.15.1:0/2660145879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ed40b0a70 0x7f9ed4140d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.599+0000 7f9ed37fe700 1 -- 172.21.15.1:0/2660145879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ec0005210 msgr2=0x7f9ed41f24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.599+0000 7f9ed37fe700 1 --2- 172.21.15.1:0/2660145879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ec0005210 0x7f9ed41f24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.599+0000 7f9ed37fe700 1 -- 172.21.15.1:0/2660145879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ed40b0a70 msgr2=0x7f9ed4140d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.599+0000 7f9ed37fe700 1 --2- 172.21.15.1:0/2660145879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ed40b0a70 0x7f9ed4140d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.599+0000 7f9ed37fe700 1 -- 172.21.15.1:0/2660145879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed41f19c0 con 0x7f9ed41edbf0 2024-03-18T19:02:53.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.599+0000 7f9ed37fe700 1 --2- 172.21.15.1:0/2660145879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ed41edbf0 0x7f9ed40b0530 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f9ec40072c0 tx=0x7f9ec4007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:53.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.599+0000 7f9ed0ff9700 1 -- 172.21.15.1:0/2660145879 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec40077a0 con 0x7f9ed41edbf0 2024-03-18T19:02:53.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.599+0000 7f9ed0ff9700 1 -- 172.21.15.1:0/2660145879 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec400f8d0 con 0x7f9ed41edbf0 2024-03-18T19:02:53.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.599+0000 7f9eda2cc700 1 -- 172.21.15.1:0/2660145879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed41f1ca0 con 0x7f9ed41edbf0 2024-03-18T19:02:53.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.599+0000 7f9ed0ff9700 1 -- 172.21.15.1:0/2660145879 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec400fc80 con 0x7f9ed41edbf0 2024-03-18T19:02:53.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.599+0000 7f9eda2cc700 1 -- 172.21.15.1:0/2660145879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed41ee070 con 0x7f9ed41edbf0 2024-03-18T19:02:53.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.600+0000 7f9eda2cc700 1 -- 172.21.15.1:0/2660145879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed4145080 con 0x7f9ed41edbf0 2024-03-18T19:02:53.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.601+0000 7f9ed0ff9700 1 -- 172.21.15.1:0/2660145879 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9ec401d460 con 0x7f9ed41edbf0 2024-03-18T19:02:53.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.601+0000 7f9ed0ff9700 1 --2- 172.21.15.1:0/2660145879 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9eb406adb0 0x7f9eb406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.601+0000 7f9ed3fff700 1 --2- 172.21.15.1:0/2660145879 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9eb406adb0 0x7f9eb406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.601+0000 7f9ed0ff9700 1 -- 172.21.15.1:0/2660145879 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f9ec4089510 con 0x7f9ed41edbf0 2024-03-18T19:02:53.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.602+0000 7f9ed3fff700 1 --2- 172.21.15.1:0/2660145879 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9eb406adb0 0x7f9eb406d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc009b20 tx=0x7f9ebc008040 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:53.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.606+0000 7f9ed0ff9700 1 -- 172.21.15.1:0/2660145879 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9ec4054570 con 0x7f9ed41edbf0 2024-03-18T19:02:53.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.810+0000 7f9eda2cc700 1 -- 172.21.15.1:0/2660145879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f9ed41ee200 con 0x7f9ed41edbf0 2024-03-18T19:02:53.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.814+0000 7f9ed0ff9700 1 -- 172.21.15.1:0/2660145879 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v14) v1 ==== 120+0+0 (secure 0 0 0) 0x7f9ed41ee200 con 0x7f9ed41edbf0 2024-03-18T19:02:53.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.816+0000 7f9eda2cc700 1 -- 172.21.15.1:0/2660145879 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9eb406adb0 msgr2=0x7f9eb406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:53.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.816+0000 7f9eda2cc700 1 --2- 172.21.15.1:0/2660145879 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9eb406adb0 0x7f9eb406d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc009b20 tx=0x7f9ebc008040 comp rx=0 tx=0).stop 2024-03-18T19:02:53.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.816+0000 7f9eda2cc700 1 -- 172.21.15.1:0/2660145879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ed41edbf0 msgr2=0x7f9ed40b0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:53.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.816+0000 7f9eda2cc700 1 --2- 172.21.15.1:0/2660145879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ed41edbf0 0x7f9ed40b0530 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f9ec40072c0 tx=0x7f9ec4007680 comp rx=0 tx=0).stop 2024-03-18T19:02:53.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.816+0000 7f9eda2cc700 1 -- 172.21.15.1:0/2660145879 shutdown_connections 2024-03-18T19:02:53.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.816+0000 7f9eda2cc700 1 --2- 172.21.15.1:0/2660145879 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9eb406adb0 0x7f9eb406d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.816+0000 7f9eda2cc700 1 --2- 172.21.15.1:0/2660145879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ec0005210 0x7f9ed41f24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.816+0000 7f9eda2cc700 1 --2- 172.21.15.1:0/2660145879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ed40b0a70 0x7f9ed4140d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.816+0000 7f9eda2cc700 1 --2- 172.21.15.1:0/2660145879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ed41edbf0 0x7f9ed40b0530 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.816+0000 7f9eda2cc700 1 -- 172.21.15.1:0/2660145879 >> 172.21.15.1:0/2660145879 conn(0x7f9ed4066b70 msgr2=0x7f9ed40b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:53.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.816+0000 7f9eda2cc700 1 -- 172.21.15.1:0/2660145879 shutdown_connections 2024-03-18T19:02:53.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.816+0000 7f9eda2cc700 1 -- 172.21.15.1:0/2660145879 wait complete. 2024-03-18T19:02:53.824 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-18T19:02:53.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.955+0000 7f87e7337700 1 Processor -- start 2024-03-18T19:02:53.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.955+0000 7f87e7337700 1 -- start start 2024-03-18T19:02:53.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.956+0000 7f87e7337700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87e015d780 0x7f87e00cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.956+0000 7f87e7337700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87e00ce5a0 0x7f87e00cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.956+0000 7f87e7337700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e00ce090 con 0x7f87e009dd40 2024-03-18T19:02:53.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.956+0000 7f87e7337700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e00cb480 con 0x7f87e00ce5a0 2024-03-18T19:02:53.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.956+0000 7f87e7337700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e006db30 con 0x7f87e015d780 2024-03-18T19:02:53.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.956+0000 7f87e58d4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87e00ce5a0 0x7f87e00cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.956+0000 7f87e58d4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87e00ce5a0 0x7f87e00cea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34818/0 (socket says 172.21.15.1:34818) 2024-03-18T19:02:53.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.956+0000 7f87e58d4700 1 -- 172.21.15.1:0/952063589 learned_addr learned my addr 172.21.15.1:0/952063589 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:53.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.957+0000 7f87e50d3700 1 --2- 172.21.15.1:0/952063589 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87e015d780 0x7f87e00cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.957+0000 7f87e58d4700 1 -- 172.21.15.1:0/952063589 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87e015d780 msgr2=0x7f87e00cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:53.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.957+0000 7f87e58d4700 1 --2- 172.21.15.1:0/952063589 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87e015d780 0x7f87e00cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.957+0000 7f87e58d4700 1 -- 172.21.15.1:0/952063589 >> v1:172.21.15.165:6789/0 conn(0x7f87e009dd40 legacy=0x7f87e00cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:53.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.957+0000 7f87e58d4700 1 -- 172.21.15.1:0/952063589 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e006d520 con 0x7f87e00ce5a0 2024-03-18T19:02:53.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.957+0000 7f87e58d4700 1 --2- 172.21.15.1:0/952063589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87e00ce5a0 0x7f87e00cea10 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f87d800d150 tx=0x7f87d800d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41a6e242e006b8a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:53.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.957+0000 7f87d7fff700 1 -- 172.21.15.1:0/952063589 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8010070 con 0x7f87e00ce5a0 2024-03-18T19:02:53.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.957+0000 7f87d7fff700 1 -- 172.21.15.1:0/952063589 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d800ddc0 con 0x7f87e00ce5a0 2024-03-18T19:02:53.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.957+0000 7f87d7fff700 1 -- 172.21.15.1:0/952063589 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d800b310 con 0x7f87e00ce5a0 2024-03-18T19:02:53.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.957+0000 7f87e7337700 1 -- 172.21.15.1:0/952063589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87e00ce5a0 msgr2=0x7f87e00cea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:53.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.957+0000 7f87e7337700 1 --2- 172.21.15.1:0/952063589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87e00ce5a0 0x7f87e00cea10 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f87d800d150 tx=0x7f87d800d460 comp rx=0 tx=0).stop 2024-03-18T19:02:53.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.958+0000 7f87e7337700 1 -- 172.21.15.1:0/952063589 shutdown_connections 2024-03-18T19:02:53.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.958+0000 7f87e7337700 1 --2- 172.21.15.1:0/952063589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87e00ce5a0 0x7f87e00cea10 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.958+0000 7f87e7337700 1 --2- 172.21.15.1:0/952063589 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87e015d780 0x7f87e00cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.958+0000 7f87e7337700 1 -- 172.21.15.1:0/952063589 >> 172.21.15.1:0/952063589 conn(0x7f87e00cca00 msgr2=0x7f87e00cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:53.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.958+0000 7f87e7337700 1 -- 172.21.15.1:0/952063589 shutdown_connections 2024-03-18T19:02:53.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.958+0000 7f87e7337700 1 -- 172.21.15.1:0/952063589 wait complete. 2024-03-18T19:02:53.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.959+0000 7f87e7337700 1 Processor -- start 2024-03-18T19:02:53.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.959+0000 7f87e7337700 1 -- start start 2024-03-18T19:02:53.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87e7337700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87e009dd40 0x7f87e015ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87e7337700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87e00ce5a0 0x7f87e01518a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87e7337700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87e015d780 0x7f87e0151de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87e7337700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e00c5e60 con 0x7f87e009dd40 2024-03-18T19:02:53.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87e7337700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e0152320 con 0x7f87e015d780 2024-03-18T19:02:53.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87e7337700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e00c54c0 con 0x7f87e00ce5a0 2024-03-18T19:02:53.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87e50d3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87e009dd40 0x7f87e015ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87e50d3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87e009dd40 0x7f87e015ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34826/0 (socket says 172.21.15.1:34826) 2024-03-18T19:02:53.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87e50d3700 1 -- 172.21.15.1:0/1048849740 learned_addr learned my addr 172.21.15.1:0/1048849740 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:53.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87e48d2700 1 --2- 172.21.15.1:0/1048849740 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87e00ce5a0 0x7f87e01518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87e58d4700 1 --2- 172.21.15.1:0/1048849740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87e015d780 0x7f87e0151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87e50d3700 1 -- 172.21.15.1:0/1048849740 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87e00ce5a0 msgr2=0x7f87e01518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:53.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87e50d3700 1 --2- 172.21.15.1:0/1048849740 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87e00ce5a0 0x7f87e01518a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87e50d3700 1 -- 172.21.15.1:0/1048849740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87e015d780 msgr2=0x7f87e0151de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:53.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87e50d3700 1 --2- 172.21.15.1:0/1048849740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87e015d780 0x7f87e0151de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:53.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87e50d3700 1 -- 172.21.15.1:0/1048849740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d800bd40 con 0x7f87e009dd40 2024-03-18T19:02:53.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87e50d3700 1 --2- 172.21.15.1:0/1048849740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87e009dd40 0x7f87e015ad70 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f87d000e840 tx=0x7f87d000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:53.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87d5ffb700 1 -- 172.21.15.1:0/1048849740 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d0013610 con 0x7f87e009dd40 2024-03-18T19:02:53.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87d5ffb700 1 -- 172.21.15.1:0/1048849740 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d0013770 con 0x7f87e009dd40 2024-03-18T19:02:53.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87e7337700 1 -- 172.21.15.1:0/1048849740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e00c57a0 con 0x7f87e009dd40 2024-03-18T19:02:53.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.960+0000 7f87d5ffb700 1 -- 172.21.15.1:0/1048849740 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d0013b20 con 0x7f87e009dd40 2024-03-18T19:02:53.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.961+0000 7f87e7337700 1 -- 172.21.15.1:0/1048849740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e00c7bd0 con 0x7f87e009dd40 2024-03-18T19:02:53.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.961+0000 7f87e7337700 1 -- 172.21.15.1:0/1048849740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e00676a0 con 0x7f87e009dd40 2024-03-18T19:02:53.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.962+0000 7f87d5ffb700 1 -- 172.21.15.1:0/1048849740 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f87d0013c80 con 0x7f87e009dd40 2024-03-18T19:02:53.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.962+0000 7f87d5ffb700 1 --2- 172.21.15.1:0/1048849740 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f87c406adb0 0x7f87c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:53.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.963+0000 7f87d5ffb700 1 -- 172.21.15.1:0/1048849740 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f87d00893b0 con 0x7f87e009dd40 2024-03-18T19:02:53.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.963+0000 7f87e48d2700 1 --2- 172.21.15.1:0/1048849740 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f87c406adb0 0x7f87c406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:53.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.963+0000 7f87e48d2700 1 --2- 172.21.15.1:0/1048849740 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f87c406adb0 0x7f87c406d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f87e00ca3c0 tx=0x7f87dc009040 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:53.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:53.967+0000 7f87d5ffb700 1 -- 172.21.15.1:0/1048849740 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f87d0054270 con 0x7f87e009dd40 2024-03-18T19:02:54.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.183+0000 7f87e7337700 1 -- 172.21.15.1:0/1048849740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f87e0072490 con 0x7f87e009dd40 2024-03-18T19:02:54.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.187+0000 7f87d5ffb700 1 -- 172.21.15.1:0/1048849740 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f87d0057890 con 0x7f87e009dd40 2024-03-18T19:02:54.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.189+0000 7f87e7337700 1 -- 172.21.15.1:0/1048849740 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f87c406adb0 msgr2=0x7f87c406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:54.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.189+0000 7f87e7337700 1 --2- 172.21.15.1:0/1048849740 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f87c406adb0 0x7f87c406d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f87e00ca3c0 tx=0x7f87dc009040 comp rx=0 tx=0).stop 2024-03-18T19:02:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.189+0000 7f87e7337700 1 -- 172.21.15.1:0/1048849740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87e009dd40 msgr2=0x7f87e015ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.189+0000 7f87e7337700 1 --2- 172.21.15.1:0/1048849740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87e009dd40 0x7f87e015ad70 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f87d000e840 tx=0x7f87d000ec00 comp rx=0 tx=0).stop 2024-03-18T19:02:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.189+0000 7f87e7337700 1 -- 172.21.15.1:0/1048849740 shutdown_connections 2024-03-18T19:02:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.189+0000 7f87e7337700 1 --2- 172.21.15.1:0/1048849740 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f87c406adb0 0x7f87c406d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.189+0000 7f87e7337700 1 --2- 172.21.15.1:0/1048849740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87e015d780 0x7f87e0151de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.189+0000 7f87e7337700 1 --2- 172.21.15.1:0/1048849740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87e009dd40 0x7f87e015ad70 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.189+0000 7f87e7337700 1 --2- 172.21.15.1:0/1048849740 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87e00ce5a0 0x7f87e01518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.189+0000 7f87e7337700 1 -- 172.21.15.1:0/1048849740 >> 172.21.15.1:0/1048849740 conn(0x7f87e00cca00 msgr2=0x7f87e00c3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.189+0000 7f87e7337700 1 -- 172.21.15.1:0/1048849740 shutdown_connections 2024-03-18T19:02:54.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.189+0000 7f87e7337700 1 -- 172.21.15.1:0/1048849740 wait complete. 2024-03-18T19:02:54.197 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2024-03-18T19:02:54.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.326+0000 7f4a2111a700 1 Processor -- start 2024-03-18T19:02:54.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.326+0000 7f4a2111a700 1 -- start start 2024-03-18T19:02:54.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.327+0000 7f4a2111a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a1c140b20 0x7f4a1c0cfe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:54.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.327+0000 7f4a2111a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a1c0d03d0 con 0x7f4a1c0cfab0 2024-03-18T19:02:54.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.327+0000 7f4a2111a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a1c15fbe0 con 0x7f4a1c0cdf80 2024-03-18T19:02:54.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.327+0000 7f4a2111a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c15f060 con 0x7f4a1c140b20 2024-03-18T19:02:54.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.327+0000 7f4a1ad9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4a1c0cfab0 0x7f4a1c15c120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37460/0 (socket says 172.21.15.1:37460) 2024-03-18T19:02:54.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.327+0000 7f4a1ad9d700 1 -- 172.21.15.1:0/4280845306 learned_addr learned my addr 172.21.15.1:0/4280845306 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:54.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.327+0000 7f4a1a59c700 1 --2- 172.21.15.1:0/4280845306 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a1c140b20 0x7f4a1c0cfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:54.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.328+0000 7f4a19d9b700 1 -- 172.21.15.1:0/4280845306 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862680795 0 0) 0x7f4a1c0d03d0 con 0x7f4a1c0cfab0 2024-03-18T19:02:54.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.328+0000 7f4a19d9b700 1 -- 172.21.15.1:0/4280845306 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a080035b0 con 0x7f4a1c0cfab0 2024-03-18T19:02:54.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.328+0000 7f4a19d9b700 1 -- 172.21.15.1:0/4280845306 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3543240203 0 0) 0x7f4a1c15fbe0 con 0x7f4a1c0cdf80 2024-03-18T19:02:54.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.328+0000 7f4a19d9b700 1 -- 172.21.15.1:0/4280845306 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a1c0d03d0 con 0x7f4a1c0cdf80 2024-03-18T19:02:54.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.328+0000 7f4a19d9b700 1 -- 172.21.15.1:0/4280845306 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1629862609 0 0) 0x7f4a080035b0 con 0x7f4a1c0cfab0 2024-03-18T19:02:54.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.328+0000 7f4a1a59c700 1 -- 172.21.15.1:0/4280845306 >> v1:172.21.15.165:6789/0 conn(0x7f4a1c0cdf80 legacy=0x7f4a1c0cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:54.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.328+0000 7f4a1a59c700 1 -- 172.21.15.1:0/4280845306 >> v1:172.21.15.1:6789/0 conn(0x7f4a1c0cfab0 legacy=0x7f4a1c15c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:54.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.328+0000 7f4a1a59c700 1 -- 172.21.15.1:0/4280845306 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1c15f340 con 0x7f4a1c140b20 2024-03-18T19:02:54.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.328+0000 7f4a1a59c700 1 --2- 172.21.15.1:0/4280845306 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a1c140b20 0x7f4a1c0cfe90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4a04009870 tx=0x7f4a04009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b9f680cee551585 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:54.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.328+0000 7f4a19d9b700 1 -- 172.21.15.1:0/4280845306 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c001670 con 0x7f4a1c140b20 2024-03-18T19:02:54.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.328+0000 7f4a19d9b700 1 -- 172.21.15.1:0/4280845306 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a04004030 con 0x7f4a1c140b20 2024-03-18T19:02:54.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.328+0000 7f4a19d9b700 1 -- 172.21.15.1:0/4280845306 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a040043c0 con 0x7f4a1c140b20 2024-03-18T19:02:54.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.329+0000 7f4a2111a700 1 -- 172.21.15.1:0/4280845306 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a1c140b20 msgr2=0x7f4a1c0cfe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:54.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.329+0000 7f4a2111a700 1 --2- 172.21.15.1:0/4280845306 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a1c140b20 0x7f4a1c0cfe90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4a04009870 tx=0x7f4a04009b80 comp rx=0 tx=0).stop 2024-03-18T19:02:54.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.329+0000 7f4a2111a700 1 -- 172.21.15.1:0/4280845306 shutdown_connections 2024-03-18T19:02:54.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.329+0000 7f4a2111a700 1 --2- 172.21.15.1:0/4280845306 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a1c140b20 0x7f4a1c0cfe90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:54.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.329+0000 7f4a2111a700 1 -- 172.21.15.1:0/4280845306 >> 172.21.15.1:0/4280845306 conn(0x7f4a1c06ab70 msgr2=0x7f4a1c06af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:54.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.329+0000 7f4a2111a700 1 -- 172.21.15.1:0/4280845306 shutdown_connections 2024-03-18T19:02:54.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.329+0000 7f4a2111a700 1 -- 172.21.15.1:0/4280845306 wait complete. 2024-03-18T19:02:54.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.330+0000 7f4a2111a700 1 Processor -- start 2024-03-18T19:02:54.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.330+0000 7f4a2111a700 1 -- start start 2024-03-18T19:02:54.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.331+0000 7f4a2111a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a1c0cdf80 0x7f4a1c13b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:54.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.331+0000 7f4a2111a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a1c0cfab0 0x7f4a1c159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:54.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.331+0000 7f4a2111a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a1c0ace40 0x7f4a1c15a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:54.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.331+0000 7f4a2111a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c0ae030 con 0x7f4a1c0cfab0 2024-03-18T19:02:54.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.331+0000 7f4a2111a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c13b840 con 0x7f4a1c0cdf80 2024-03-18T19:02:54.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.331+0000 7f4a2111a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c0ad3b0 con 0x7f4a1c0ace40 2024-03-18T19:02:54.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.331+0000 7f4a1a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a1c0cfab0 0x7f4a1c159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:54.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.331+0000 7f4a1a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a1c0cfab0 0x7f4a1c159eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34842/0 (socket says 172.21.15.1:34842) 2024-03-18T19:02:54.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.331+0000 7f4a1a59c700 1 -- 172.21.15.1:0/1470606077 learned_addr learned my addr 172.21.15.1:0/1470606077 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:54.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.331+0000 7f4a1ad9d700 1 --2- 172.21.15.1:0/1470606077 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a1c0cdf80 0x7f4a1c13b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:54.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.331+0000 7f4a1b59e700 1 --2- 172.21.15.1:0/1470606077 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a1c0ace40 0x7f4a1c15a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:54.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.332+0000 7f4a1a59c700 1 -- 172.21.15.1:0/1470606077 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a1c0ace40 msgr2=0x7f4a1c15a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:54.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.332+0000 7f4a1a59c700 1 --2- 172.21.15.1:0/1470606077 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a1c0ace40 0x7f4a1c15a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:54.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.332+0000 7f4a1a59c700 1 -- 172.21.15.1:0/1470606077 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a1c0cdf80 msgr2=0x7f4a1c13b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:54.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.332+0000 7f4a1a59c700 1 --2- 172.21.15.1:0/1470606077 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a1c0cdf80 0x7f4a1c13b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:54.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.332+0000 7f4a1a59c700 1 -- 172.21.15.1:0/1470606077 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a040094a0 con 0x7f4a1c0cfab0 2024-03-18T19:02:54.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.332+0000 7f4a1a59c700 1 --2- 172.21.15.1:0/1470606077 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a1c0cfab0 0x7f4a1c159eb0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f4a04000c00 tx=0x7f4a04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:54.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.332+0000 7f4a137fe700 1 -- 172.21.15.1:0/1470606077 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a04003e30 con 0x7f4a1c0cfab0 2024-03-18T19:02:54.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.332+0000 7f4a137fe700 1 -- 172.21.15.1:0/1470606077 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0401f070 con 0x7f4a1c0cfab0 2024-03-18T19:02:54.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.332+0000 7f4a137fe700 1 -- 172.21.15.1:0/1470606077 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a04011870 con 0x7f4a1c0cfab0 2024-03-18T19:02:54.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.332+0000 7f4a2111a700 1 -- 172.21.15.1:0/1470606077 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a1c0ad690 con 0x7f4a1c0cfab0 2024-03-18T19:02:54.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.332+0000 7f4a2111a700 1 -- 172.21.15.1:0/1470606077 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a1c0adc50 con 0x7f4a1c0cfab0 2024-03-18T19:02:54.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.333+0000 7f4a2111a700 1 -- 172.21.15.1:0/1470606077 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a1c0ae170 con 0x7f4a1c0cfab0 2024-03-18T19:02:54.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.333+0000 7f4a137fe700 1 -- 172.21.15.1:0/1470606077 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4a04021030 con 0x7f4a1c0cfab0 2024-03-18T19:02:54.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.334+0000 7f4a137fe700 1 --2- 172.21.15.1:0/1470606077 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49fc06ade0 0x7f49fc06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:54.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.334+0000 7f4a137fe700 1 -- 172.21.15.1:0/1470606077 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f4a0408a8a0 con 0x7f4a1c0cfab0 2024-03-18T19:02:54.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.334+0000 7f4a1ad9d700 1 --2- 172.21.15.1:0/1470606077 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49fc06ade0 0x7f49fc06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:54.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.334+0000 7f4a1ad9d700 1 --2- 172.21.15.1:0/1470606077 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49fc06ade0 0x7f49fc06d2a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c001380 tx=0x7f4a0c001590 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:54.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.338+0000 7f4a137fe700 1 -- 172.21.15.1:0/1470606077 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4a1c0ae170 con 0x7f4a1c0cfab0 2024-03-18T19:02:54.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:54.628+0000 7f4a2111a700 1 -- 172.21.15.1:0/1470606077 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f4a1c0ae170 con 0x7f4a1c0cfab0 2024-03-18T19:02:55.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.081+0000 7f4a137fe700 1 -- 172.21.15.1:0/1470606077 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v478) v1 ==== 158+0+0 (secure 0 0 0) 0x7f4a1c0ae170 con 0x7f4a1c0cfab0 2024-03-18T19:02:55.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.084+0000 7f4a2111a700 1 -- 172.21.15.1:0/1470606077 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49fc06ade0 msgr2=0x7f49fc06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:55.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.084+0000 7f4a2111a700 1 --2- 172.21.15.1:0/1470606077 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49fc06ade0 0x7f49fc06d2a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c001380 tx=0x7f4a0c001590 comp rx=0 tx=0).stop 2024-03-18T19:02:55.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.084+0000 7f4a2111a700 1 -- 172.21.15.1:0/1470606077 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a1c0cfab0 msgr2=0x7f4a1c159eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:55.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.084+0000 7f4a2111a700 1 --2- 172.21.15.1:0/1470606077 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a1c0cfab0 0x7f4a1c159eb0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f4a04000c00 tx=0x7f4a04004750 comp rx=0 tx=0).stop 2024-03-18T19:02:55.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.084+0000 7f4a2111a700 1 -- 172.21.15.1:0/1470606077 shutdown_connections 2024-03-18T19:02:55.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.084+0000 7f4a2111a700 1 --2- 172.21.15.1:0/1470606077 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49fc06ade0 0x7f49fc06d2a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:55.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.084+0000 7f4a2111a700 1 --2- 172.21.15.1:0/1470606077 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4a1c0ace40 0x7f4a1c15a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:55.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.084+0000 7f4a2111a700 1 --2- 172.21.15.1:0/1470606077 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4a1c0cdf80 0x7f4a1c13b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:55.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.084+0000 7f4a2111a700 1 --2- 172.21.15.1:0/1470606077 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a1c0cfab0 0x7f4a1c159eb0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:55.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.084+0000 7f4a2111a700 1 -- 172.21.15.1:0/1470606077 >> 172.21.15.1:0/1470606077 conn(0x7f4a1c06ab70 msgr2=0x7f4a1c1592f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:55.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.084+0000 7f4a2111a700 1 -- 172.21.15.1:0/1470606077 shutdown_connections 2024-03-18T19:02:55.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.084+0000 7f4a2111a700 1 -- 172.21.15.1:0/1470606077 wait complete. 2024-03-18T19:02:55.095 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2024-03-18T19:02:55.095 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2024-03-18T19:02:55.095 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-18T19:02:55.095 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2024-03-18T19:02:55.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.233+0000 7f8a15a27700 1 Processor -- start 2024-03-18T19:02:55.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.233+0000 7f8a15a27700 1 -- start start 2024-03-18T19:02:55.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.234+0000 7f8a15a27700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8a100d1ea0 0x7f8a10168560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:55.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.234+0000 7f8a15a27700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a100cddc0 con 0x7f8a10160940 2024-03-18T19:02:55.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.234+0000 7f8a15a27700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a100cdff0 con 0x7f8a100cf090 2024-03-18T19:02:55.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.234+0000 7f8a15a27700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a100ce2d0 con 0x7f8a100d1ea0 2024-03-18T19:02:55.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.234+0000 7f8a0effd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8a10160940 0x7f8a100d0190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38522/0 (socket says 172.21.15.1:38522) 2024-03-18T19:02:55.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.234+0000 7f8a0effd700 1 -- 172.21.15.1:0/1888778257 learned_addr learned my addr 172.21.15.1:0/1888778257 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:55.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.234+0000 7f8a0dffb700 1 -- 172.21.15.1:0/1888778257 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218701348 0 0) 0x7f8a100cddc0 con 0x7f8a10160940 2024-03-18T19:02:55.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.234+0000 7f8a0f7fe700 1 --2- 172.21.15.1:0/1888778257 >> v2:172.21.15.165:3300/0 conn(0x7f8a100d1ea0 0x7f8a10168560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:55.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.234+0000 7f8a0dffb700 1 -- 172.21.15.1:0/1888778257 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89f80035b0 con 0x7f8a10160940 2024-03-18T19:02:55.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0dffb700 1 -- 172.21.15.1:0/1888778257 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4138369411 0 0) 0x7f89f80035b0 con 0x7f8a10160940 2024-03-18T19:02:55.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0dffb700 1 -- 172.21.15.1:0/1888778257 >> v2:172.21.15.165:3300/0 conn(0x7f8a100d1ea0 msgr2=0x7f8a10168560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:55.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0dffb700 1 --2- 172.21.15.1:0/1888778257 >> v2:172.21.15.165:3300/0 conn(0x7f8a100d1ea0 0x7f8a10168560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:55.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0dffb700 1 -- 172.21.15.1:0/1888778257 >> v1:172.21.15.165:6789/0 conn(0x7f8a100cf090 legacy=0x7f8a100d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:55.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0dffb700 1 -- 172.21.15.1:0/1888778257 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a100ce5b0 con 0x7f8a10160940 2024-03-18T19:02:55.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0dffb700 1 -- 172.21.15.1:0/1888778257 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f8a04001670 con 0x7f8a10160940 2024-03-18T19:02:55.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0dffb700 1 -- 172.21.15.1:0/1888778257 >> v1:172.21.15.1:6789/0 conn(0x7f8a10160940 legacy=0x7f8a100d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:55.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0dffb700 1 --2- 172.21.15.1:0/1888778257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89f8003a00 0x7f89f8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:55.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0dffb700 1 --2- 172.21.15.1:0/1888778257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89f8004850 0x7f89f8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:55.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0dffb700 1 --2- 172.21.15.1:0/1888778257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89f8005210 0x7f89f800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:55.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0dffb700 1 -- 172.21.15.1:0/1888778257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a100ce2d0 con 0x7f89f8003a00 2024-03-18T19:02:55.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0dffb700 1 -- 172.21.15.1:0/1888778257 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f800c0b0 con 0x7f89f8004850 2024-03-18T19:02:55.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0dffb700 1 -- 172.21.15.1:0/1888778257 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f800c270 con 0x7f89f8005210 2024-03-18T19:02:55.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0effd700 1 --2- 172.21.15.1:0/1888778257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89f8003a00 0x7f89f8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:55.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0e7fc700 1 --2- 172.21.15.1:0/1888778257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89f8004850 0x7f89f8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:55.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0f7fe700 1 --2- 172.21.15.1:0/1888778257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89f8005210 0x7f89f800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:55.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0effd700 1 -- 172.21.15.1:0/1888778257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89f8005210 msgr2=0x7f89f800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:55.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0effd700 1 --2- 172.21.15.1:0/1888778257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89f8005210 0x7f89f800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:55.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0effd700 1 -- 172.21.15.1:0/1888778257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89f8004850 msgr2=0x7f89f8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:55.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0effd700 1 --2- 172.21.15.1:0/1888778257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89f8004850 0x7f89f8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:55.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0effd700 1 -- 172.21.15.1:0/1888778257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f80035b0 con 0x7f89f8003a00 2024-03-18T19:02:55.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0f7fe700 1 -- 172.21.15.1:0/1888778257 reap_dead start 2024-03-18T19:02:55.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0effd700 1 --2- 172.21.15.1:0/1888778257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89f8003a00 0x7f89f8003e20 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8a0400c240 tx=0x7f89f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a5c59a420cfa38 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:55.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.235+0000 7f8a0dffb700 1 -- 172.21.15.1:0/1888778257 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a04007c10 con 0x7f89f8003a00 2024-03-18T19:02:55.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.236+0000 7f8a0dffb700 1 -- 172.21.15.1:0/1888778257 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0400cde0 con 0x7f89f8003a00 2024-03-18T19:02:55.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.236+0000 7f8a0dffb700 1 -- 172.21.15.1:0/1888778257 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a04005ba0 con 0x7f89f8003a00 2024-03-18T19:02:55.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.236+0000 7f8a15a27700 1 -- 172.21.15.1:0/1888778257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89f8003a00 msgr2=0x7f89f8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:55.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.236+0000 7f8a15a27700 1 --2- 172.21.15.1:0/1888778257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89f8003a00 0x7f89f8003e20 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8a0400c240 tx=0x7f89f8005880 comp rx=0 tx=0).stop 2024-03-18T19:02:55.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.236+0000 7f8a15a27700 1 -- 172.21.15.1:0/1888778257 shutdown_connections 2024-03-18T19:02:55.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.236+0000 7f8a15a27700 1 --2- 172.21.15.1:0/1888778257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89f8003a00 0x7f89f8003e20 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:55.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.236+0000 7f8a15a27700 1 -- 172.21.15.1:0/1888778257 >> 172.21.15.1:0/1888778257 conn(0x7f8a1009c520 msgr2=0x7f8a100d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:55.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.236+0000 7f8a15a27700 1 -- 172.21.15.1:0/1888778257 shutdown_connections 2024-03-18T19:02:55.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.236+0000 7f8a15a27700 1 -- 172.21.15.1:0/1888778257 wait complete. 2024-03-18T19:02:55.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.237+0000 7f8a15a27700 1 Processor -- start 2024-03-18T19:02:55.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.237+0000 7f8a15a27700 1 -- start start 2024-03-18T19:02:55.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.238+0000 7f8a15a27700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89f8003a00 0x7f8a100c9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:55.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.238+0000 7f8a15a27700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a100b8160 0x7f8a101f64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:55.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.238+0000 7f8a15a27700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a101f6a20 0x7f8a101f6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:55.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.238+0000 7f8a15a27700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a100b0200 con 0x7f8a101f6a20 2024-03-18T19:02:55.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.238+0000 7f8a15a27700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a100b6240 con 0x7f8a100b8160 2024-03-18T19:02:55.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.238+0000 7f8a15a27700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a101f62b0 con 0x7f89f8003a00 2024-03-18T19:02:55.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.238+0000 7f8a0f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a101f6a20 0x7f8a101f6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:55.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.238+0000 7f8a0f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a101f6a20 0x7f8a101f6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48538/0 (socket says 172.21.15.1:48538) 2024-03-18T19:02:55.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.238+0000 7f8a0f7fe700 1 -- 172.21.15.1:0/723039524 learned_addr learned my addr 172.21.15.1:0/723039524 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:55.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.238+0000 7f8a0effd700 1 --2- 172.21.15.1:0/723039524 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89f8003a00 0x7f8a100c9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:55.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.238+0000 7f8a0e7fc700 1 --2- 172.21.15.1:0/723039524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a100b8160 0x7f8a101f64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:55.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.239+0000 7f8a0f7fe700 1 -- 172.21.15.1:0/723039524 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89f8003a00 msgr2=0x7f8a100c9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:55.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.239+0000 7f8a0f7fe700 1 --2- 172.21.15.1:0/723039524 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89f8003a00 0x7f8a100c9520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:55.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.239+0000 7f8a0f7fe700 1 -- 172.21.15.1:0/723039524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a100b8160 msgr2=0x7f8a101f64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:55.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.239+0000 7f8a0f7fe700 1 --2- 172.21.15.1:0/723039524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a100b8160 0x7f8a101f64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:55.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.239+0000 7f8a0f7fe700 1 -- 172.21.15.1:0/723039524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a100b86b0 con 0x7f8a101f6a20 2024-03-18T19:02:55.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.239+0000 7f8a0f7fe700 1 --2- 172.21.15.1:0/723039524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a101f6a20 0x7f8a101f6ea0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f8a100d29a0 tx=0x7f8a08009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:55.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.239+0000 7f89f7fff700 1 -- 172.21.15.1:0/723039524 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0800e070 con 0x7f8a101f6a20 2024-03-18T19:02:55.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.239+0000 7f89f7fff700 1 -- 172.21.15.1:0/723039524 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a08003cf0 con 0x7f8a101f6a20 2024-03-18T19:02:55.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.239+0000 7f8a15a27700 1 -- 172.21.15.1:0/723039524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a100b8990 con 0x7f8a101f6a20 2024-03-18T19:02:55.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.239+0000 7f89f7fff700 1 -- 172.21.15.1:0/723039524 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a08011730 con 0x7f8a101f6a20 2024-03-18T19:02:55.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.239+0000 7f8a15a27700 1 -- 172.21.15.1:0/723039524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a100afe20 con 0x7f8a101f6a20 2024-03-18T19:02:55.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.240+0000 7f8a15a27700 1 -- 172.21.15.1:0/723039524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a100b0340 con 0x7f8a101f6a20 2024-03-18T19:02:55.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.241+0000 7f89f7fff700 1 -- 172.21.15.1:0/723039524 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8a08011890 con 0x7f8a101f6a20 2024-03-18T19:02:55.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.241+0000 7f89f7fff700 1 --2- 172.21.15.1:0/723039524 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f89fc06adb0 0x7f89fc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:55.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.241+0000 7f89f7fff700 1 -- 172.21.15.1:0/723039524 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f8a080897e0 con 0x7f8a101f6a20 2024-03-18T19:02:55.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.241+0000 7f8a0effd700 1 --2- 172.21.15.1:0/723039524 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f89fc06adb0 0x7f89fc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:55.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.242+0000 7f8a0effd700 1 --2- 172.21.15.1:0/723039524 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f89fc06adb0 0x7f89fc06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8a04000c00 tx=0x7f8a04007000 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:55.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.246+0000 7f89f7fff700 1 -- 172.21.15.1:0/723039524 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8a100b0340 con 0x7f8a101f6a20 2024-03-18T19:02:55.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:55.534+0000 7f8a15a27700 1 -- 172.21.15.1:0/723039524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7f8a100b0340 con 0x7f8a101f6a20 2024-03-18T19:02:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.082+0000 7f89f7fff700 1 -- 172.21.15.1:0/723039524 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v479) v1 ==== 157+0+0 (secure 0 0 0) 0x7f8a100b0340 con 0x7f8a101f6a20 2024-03-18T19:02:56.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.085+0000 7f8a15a27700 1 -- 172.21.15.1:0/723039524 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f89fc06adb0 msgr2=0x7f89fc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:56.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.085+0000 7f8a15a27700 1 --2- 172.21.15.1:0/723039524 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f89fc06adb0 0x7f89fc06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8a04000c00 tx=0x7f8a04007000 comp rx=0 tx=0).stop 2024-03-18T19:02:56.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.085+0000 7f8a15a27700 1 -- 172.21.15.1:0/723039524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a101f6a20 msgr2=0x7f8a101f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:56.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.085+0000 7f8a15a27700 1 --2- 172.21.15.1:0/723039524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a101f6a20 0x7f8a101f6ea0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f8a100d29a0 tx=0x7f8a08009f70 comp rx=0 tx=0).stop 2024-03-18T19:02:56.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.085+0000 7f8a15a27700 1 -- 172.21.15.1:0/723039524 shutdown_connections 2024-03-18T19:02:56.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.085+0000 7f8a15a27700 1 --2- 172.21.15.1:0/723039524 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f89fc06adb0 0x7f89fc06d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.085+0000 7f8a15a27700 1 --2- 172.21.15.1:0/723039524 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89f8003a00 0x7f8a100c9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.085+0000 7f8a15a27700 1 --2- 172.21.15.1:0/723039524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a101f6a20 0x7f8a101f6ea0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.085+0000 7f8a15a27700 1 --2- 172.21.15.1:0/723039524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a100b8160 0x7f8a101f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.086+0000 7f8a15a27700 1 -- 172.21.15.1:0/723039524 >> 172.21.15.1:0/723039524 conn(0x7f8a1009c520 msgr2=0x7f8a100b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:56.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.086+0000 7f8a15a27700 1 -- 172.21.15.1:0/723039524 shutdown_connections 2024-03-18T19:02:56.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.086+0000 7f8a15a27700 1 -- 172.21.15.1:0/723039524 wait complete. 2024-03-18T19:02:56.098 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-18T19:02:56.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.241+0000 7f4766228700 1 Processor -- start 2024-03-18T19:02:56.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.241+0000 7f4766228700 1 -- start start 2024-03-18T19:02:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.242+0000 7f4766228700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f476015d5b0 0x7f476015d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.242+0000 7f4766228700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4760099a40 0x7f4760099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.242+0000 7f4766228700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f476009a3e0 0x7f476006ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.242+0000 7f4766228700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47600cbbb0 con 0x7f476009a3e0 2024-03-18T19:02:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.242+0000 7f4766228700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f476009a830 con 0x7f476015d5b0 2024-03-18T19:02:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.242+0000 7f4766228700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47600b3b00 con 0x7f4760099a40 2024-03-18T19:02:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.242+0000 7f475ffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f476009a3e0 0x7f476006ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.242+0000 7f475ffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f476009a3e0 0x7f476006ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48548/0 (socket says 172.21.15.1:48548) 2024-03-18T19:02:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.242+0000 7f475ffff700 1 -- 172.21.15.1:0/3237740278 learned_addr learned my addr 172.21.15.1:0/3237740278 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.242+0000 7f475f7fe700 1 --2- 172.21.15.1:0/3237740278 >> v2:172.21.15.165:3300/0 conn(0x7f476015d5b0 0x7f476015d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.242+0000 7f475effd700 1 --2- 172.21.15.1:0/3237740278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4760099a40 0x7f4760099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.242+0000 7f475ffff700 1 -- 172.21.15.1:0/3237740278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4760099a40 msgr2=0x7f4760099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.242+0000 7f475ffff700 1 --2- 172.21.15.1:0/3237740278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4760099a40 0x7f4760099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.242+0000 7f475ffff700 1 -- 172.21.15.1:0/3237740278 >> v2:172.21.15.165:3300/0 conn(0x7f476015d5b0 msgr2=0x7f476015d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.242+0000 7f475ffff700 1 --2- 172.21.15.1:0/3237740278 >> v2:172.21.15.165:3300/0 conn(0x7f476015d5b0 0x7f476015d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.242+0000 7f475ffff700 1 -- 172.21.15.1:0/3237740278 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47600cf7f0 con 0x7f476009a3e0 2024-03-18T19:02:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.242+0000 7f475ffff700 1 --2- 172.21.15.1:0/3237740278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f476009a3e0 0x7f476006ba70 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f4760072080 tx=0x7f475800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2976a161b11b755 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.243+0000 7f475e7fc700 1 -- 172.21.15.1:0/3237740278 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475800e070 con 0x7f476009a3e0 2024-03-18T19:02:56.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.243+0000 7f475e7fc700 1 -- 172.21.15.1:0/3237740278 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f475800b9c0 con 0x7f476009a3e0 2024-03-18T19:02:56.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.243+0000 7f475e7fc700 1 -- 172.21.15.1:0/3237740278 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475800bd70 con 0x7f476009a3e0 2024-03-18T19:02:56.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.243+0000 7f4766228700 1 -- 172.21.15.1:0/3237740278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f476009a3e0 msgr2=0x7f476006ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:56.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.243+0000 7f4766228700 1 --2- 172.21.15.1:0/3237740278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f476009a3e0 0x7f476006ba70 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f4760072080 tx=0x7f475800b330 comp rx=0 tx=0).stop 2024-03-18T19:02:56.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.243+0000 7f4766228700 1 -- 172.21.15.1:0/3237740278 shutdown_connections 2024-03-18T19:02:56.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.243+0000 7f4766228700 1 --2- 172.21.15.1:0/3237740278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f476009a3e0 0x7f476006ba70 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.243+0000 7f4766228700 1 --2- 172.21.15.1:0/3237740278 >> v2:172.21.15.165:3300/0 conn(0x7f476015d5b0 0x7f476015d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.243+0000 7f4766228700 1 --2- 172.21.15.1:0/3237740278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4760099a40 0x7f4760099ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.243+0000 7f4766228700 1 -- 172.21.15.1:0/3237740278 >> 172.21.15.1:0/3237740278 conn(0x7f47600b2bd0 msgr2=0x7f47600b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:56.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.243+0000 7f4766228700 1 -- 172.21.15.1:0/3237740278 shutdown_connections 2024-03-18T19:02:56.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.243+0000 7f4766228700 1 -- 172.21.15.1:0/3237740278 wait complete. 2024-03-18T19:02:56.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.244+0000 7f4766228700 1 Processor -- start 2024-03-18T19:02:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.245+0000 7f4766228700 1 -- start start 2024-03-18T19:02:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.245+0000 7f4766228700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4760099a40 0x7f4760156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.245+0000 7f4766228700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f476009a3e0 0x7f47600b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.245+0000 7f4766228700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f476015d5b0 0x7f47600b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.245+0000 7f4766228700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47600b6320 con 0x7f476009a3e0 2024-03-18T19:02:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.245+0000 7f4766228700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47600b65c0 con 0x7f476015d5b0 2024-03-18T19:02:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.245+0000 7f4766228700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47600b68a0 con 0x7f4760099a40 2024-03-18T19:02:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.245+0000 7f475effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f476009a3e0 0x7f47600b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.245+0000 7f475effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f476009a3e0 0x7f47600b47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48550/0 (socket says 172.21.15.1:48550) 2024-03-18T19:02:56.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.245+0000 7f475effd700 1 -- 172.21.15.1:0/565223794 learned_addr learned my addr 172.21.15.1:0/565223794 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:56.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.246+0000 7f475ffff700 1 --2- 172.21.15.1:0/565223794 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f476015d5b0 0x7f47600b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:56.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.246+0000 7f475f7fe700 1 --2- 172.21.15.1:0/565223794 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4760099a40 0x7f4760156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:56.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.246+0000 7f475effd700 1 -- 172.21.15.1:0/565223794 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4760099a40 msgr2=0x7f4760156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:56.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.246+0000 7f475effd700 1 --2- 172.21.15.1:0/565223794 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4760099a40 0x7f4760156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.246+0000 7f475effd700 1 -- 172.21.15.1:0/565223794 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f476015d5b0 msgr2=0x7f47600b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:56.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.246+0000 7f475effd700 1 --2- 172.21.15.1:0/565223794 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f476015d5b0 0x7f47600b4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.246+0000 7f475effd700 1 -- 172.21.15.1:0/565223794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4758009b30 con 0x7f476009a3e0 2024-03-18T19:02:56.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.246+0000 7f475effd700 1 --2- 172.21.15.1:0/565223794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f476009a3e0 0x7f47600b47a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f475400d800 tx=0x7f475400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:56.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.246+0000 7f4747fff700 1 -- 172.21.15.1:0/565223794 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47540125f0 con 0x7f476009a3e0 2024-03-18T19:02:56.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.246+0000 7f4747fff700 1 -- 172.21.15.1:0/565223794 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4754012750 con 0x7f476009a3e0 2024-03-18T19:02:56.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.246+0000 7f4766228700 1 -- 172.21.15.1:0/565223794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47600b6b80 con 0x7f476009a3e0 2024-03-18T19:02:56.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.246+0000 7f4747fff700 1 -- 172.21.15.1:0/565223794 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4754012ae0 con 0x7f476009a3e0 2024-03-18T19:02:56.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.246+0000 7f4766228700 1 -- 172.21.15.1:0/565223794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47600b5d20 con 0x7f476009a3e0 2024-03-18T19:02:56.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.247+0000 7f4766228700 1 -- 172.21.15.1:0/565223794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47600b6d70 con 0x7f476009a3e0 2024-03-18T19:02:56.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.248+0000 7f4747fff700 1 -- 172.21.15.1:0/565223794 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4754012c40 con 0x7f476009a3e0 2024-03-18T19:02:56.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.248+0000 7f4747fff700 1 --2- 172.21.15.1:0/565223794 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f474c06adb0 0x7f474c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:56.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.248+0000 7f4747fff700 1 -- 172.21.15.1:0/565223794 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f4754089810 con 0x7f476009a3e0 2024-03-18T19:02:56.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.248+0000 7f475f7fe700 1 --2- 172.21.15.1:0/565223794 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f474c06adb0 0x7f474c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:56.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.248+0000 7f475f7fe700 1 --2- 172.21.15.1:0/565223794 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f474c06adb0 0x7f474c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4750009930 tx=0x7f4750008040 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:56.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.253+0000 7f4747fff700 1 -- 172.21.15.1:0/565223794 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f47600b6d70 con 0x7f476009a3e0 2024-03-18T19:02:56.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.455+0000 7f4766228700 1 -- 172.21.15.1:0/565223794 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f47600b6d70 con 0x7f476009a3e0 2024-03-18T19:02:56.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.459+0000 7f4747fff700 1 -- 172.21.15.1:0/565223794 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f47600b6d70 con 0x7f476009a3e0 2024-03-18T19:02:56.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.461+0000 7f4766228700 1 -- 172.21.15.1:0/565223794 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f474c06adb0 msgr2=0x7f474c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:56.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.461+0000 7f4766228700 1 --2- 172.21.15.1:0/565223794 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f474c06adb0 0x7f474c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4750009930 tx=0x7f4750008040 comp rx=0 tx=0).stop 2024-03-18T19:02:56.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.461+0000 7f4766228700 1 -- 172.21.15.1:0/565223794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f476009a3e0 msgr2=0x7f47600b47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:56.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.461+0000 7f4766228700 1 --2- 172.21.15.1:0/565223794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f476009a3e0 0x7f47600b47a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f475400d800 tx=0x7f475400dbc0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.461+0000 7f4766228700 1 -- 172.21.15.1:0/565223794 shutdown_connections 2024-03-18T19:02:56.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.461+0000 7f4766228700 1 --2- 172.21.15.1:0/565223794 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f474c06adb0 0x7f474c06d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.461+0000 7f4766228700 1 --2- 172.21.15.1:0/565223794 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4760099a40 0x7f4760156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.461+0000 7f4766228700 1 --2- 172.21.15.1:0/565223794 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f476015d5b0 0x7f47600b4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.461+0000 7f4766228700 1 --2- 172.21.15.1:0/565223794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f476009a3e0 0x7f47600b47a0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.461+0000 7f4766228700 1 -- 172.21.15.1:0/565223794 >> 172.21.15.1:0/565223794 conn(0x7f47600b2bd0 msgr2=0x7f476006fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:56.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.461+0000 7f4766228700 1 -- 172.21.15.1:0/565223794 shutdown_connections 2024-03-18T19:02:56.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.461+0000 7f4766228700 1 -- 172.21.15.1:0/565223794 wait complete. 2024-03-18T19:02:56.470 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T19:02:56.470 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:02:56.507 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:02:56.551 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T19:02:56.552 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:02:56.552 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:02:56.570 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:02:56.570 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:02:56.570 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:02:56.615 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:02:56.615 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:02:56.615 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:02:56.634 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:02:56.634 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:02:56.634 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:02:56.676 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:02:56.676 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-18T19:02:56.676 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:02:56.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.806+0000 7ff0218e8700 1 Processor -- start 2024-03-18T19:02:56.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.806+0000 7ff0218e8700 1 -- start start 2024-03-18T19:02:56.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.806+0000 7ff0218e8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff01c15a320 0x7ff01c0ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:56.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.806+0000 7ff0218e8700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff01c071d10 0x7ff01c06e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:56.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.806+0000 7ff0218e8700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff01c0acd80 con 0x7ff01c0cea30 2024-03-18T19:02:56.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.806+0000 7ff0218e8700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff01c0cf220 con 0x7ff01c15a320 2024-03-18T19:02:56.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.806+0000 7ff0218e8700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff01c0cef40 con 0x7ff01c071d10 2024-03-18T19:02:56.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.807+0000 7ff01affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff01c15a320 0x7ff01c0ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:56.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.807+0000 7ff01a7fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff01c0cea30 0x7ff01c071600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38534/0 (socket says 172.21.15.1:38534) 2024-03-18T19:02:56.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.807+0000 7ff01a7fc700 1 -- 172.21.15.1:0/305064048 learned_addr learned my addr 172.21.15.1:0/305064048 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:56.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.807+0000 7ff01b7fe700 1 --2- 172.21.15.1:0/305064048 >> v2:172.21.15.165:3300/0 conn(0x7ff01c071d10 0x7ff01c06e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:56.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.807+0000 7ff019ffb700 1 -- 172.21.15.1:0/305064048 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185756369 0 0) 0x7ff01c0acd80 con 0x7ff01c0cea30 2024-03-18T19:02:56.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.807+0000 7ff019ffb700 1 -- 172.21.15.1:0/305064048 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0040035b0 con 0x7ff01c0cea30 2024-03-18T19:02:56.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.807+0000 7ff01affd700 1 -- 172.21.15.1:0/305064048 >> v2:172.21.15.165:3300/0 conn(0x7ff01c071d10 msgr2=0x7ff01c06e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:56.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.807+0000 7ff01affd700 1 --2- 172.21.15.1:0/305064048 >> v2:172.21.15.165:3300/0 conn(0x7ff01c071d10 0x7ff01c06e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.807+0000 7ff01affd700 1 -- 172.21.15.1:0/305064048 >> v1:172.21.15.1:6789/0 conn(0x7ff01c0cea30 legacy=0x7ff01c071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:56.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.807+0000 7ff01affd700 1 -- 172.21.15.1:0/305064048 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01c06d7e0 con 0x7ff01c15a320 2024-03-18T19:02:56.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.807+0000 7ff01b7fe700 1 --2- 172.21.15.1:0/305064048 >> v2:172.21.15.165:3300/0 conn(0x7ff01c071d10 0x7ff01c06e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:56.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.807+0000 7ff01affd700 1 --2- 172.21.15.1:0/305064048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff01c15a320 0x7ff01c0ce4f0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff01c0ca4f0 tx=0x7ff010009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929b484cd6e4af2f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:56.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.807+0000 7ff019ffb700 1 -- 172.21.15.1:0/305064048 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff010011830 con 0x7ff01c15a320 2024-03-18T19:02:56.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.807+0000 7ff019ffb700 1 -- 172.21.15.1:0/305064048 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff010011990 con 0x7ff01c15a320 2024-03-18T19:02:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.807+0000 7ff019ffb700 1 -- 172.21.15.1:0/305064048 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff010011d20 con 0x7ff01c15a320 2024-03-18T19:02:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.808+0000 7ff0218e8700 1 -- 172.21.15.1:0/305064048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff01c15a320 msgr2=0x7ff01c0ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.808+0000 7ff0218e8700 1 --2- 172.21.15.1:0/305064048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff01c15a320 0x7ff01c0ce4f0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff01c0ca4f0 tx=0x7ff010009b20 comp rx=0 tx=0).stop 2024-03-18T19:02:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.808+0000 7ff0218e8700 1 -- 172.21.15.1:0/305064048 shutdown_connections 2024-03-18T19:02:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.808+0000 7ff0218e8700 1 --2- 172.21.15.1:0/305064048 >> v2:172.21.15.165:3300/0 conn(0x7ff01c071d10 0x7ff01c06e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.808+0000 7ff0218e8700 1 --2- 172.21.15.1:0/305064048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff01c15a320 0x7ff01c0ce4f0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.808+0000 7ff0218e8700 1 -- 172.21.15.1:0/305064048 >> 172.21.15.1:0/305064048 conn(0x7ff01c0cc990 msgr2=0x7ff01c0ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.808+0000 7ff0218e8700 1 -- 172.21.15.1:0/305064048 shutdown_connections 2024-03-18T19:02:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.808+0000 7ff0218e8700 1 -- 172.21.15.1:0/305064048 wait complete. 2024-03-18T19:02:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.809+0000 7ff0218e8700 1 Processor -- start 2024-03-18T19:02:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.809+0000 7ff0218e8700 1 -- start start 2024-03-18T19:02:56.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.810+0000 7ff0218e8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff01c071d10 0x7ff01c15c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:56.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.810+0000 7ff0218e8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff01c0cea30 0x7ff01c15cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:56.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.810+0000 7ff0218e8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff01c153980 0x7ff01c15b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:56.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.810+0000 7ff0218e8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c0ae810 con 0x7ff01c071d10 2024-03-18T19:02:56.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.810+0000 7ff0218e8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c0af0c0 con 0x7ff01c153980 2024-03-18T19:02:56.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.810+0000 7ff0218e8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c15d260 con 0x7ff01c0cea30 2024-03-18T19:02:56.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.810+0000 7ff01affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff01c071d10 0x7ff01c15c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.810+0000 7ff01affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff01c071d10 0x7ff01c15c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48562/0 (socket says 172.21.15.1:48562) 2024-03-18T19:02:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.810+0000 7ff01affd700 1 -- 172.21.15.1:0/1601011665 learned_addr learned my addr 172.21.15.1:0/1601011665 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.810+0000 7ff01affd700 1 -- 172.21.15.1:0/1601011665 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff01c0cea30 msgr2=0x7ff01c15cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.810+0000 7ff01b7fe700 1 --2- 172.21.15.1:0/1601011665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff01c153980 0x7ff01c15b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.810+0000 7ff01affd700 1 --2- 172.21.15.1:0/1601011665 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff01c0cea30 0x7ff01c15cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.810+0000 7ff01affd700 1 -- 172.21.15.1:0/1601011665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff01c153980 msgr2=0x7ff01c15b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.810+0000 7ff01affd700 1 --2- 172.21.15.1:0/1601011665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff01c153980 0x7ff01c15b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.810+0000 7ff01affd700 1 -- 172.21.15.1:0/1601011665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0100094a0 con 0x7ff01c071d10 2024-03-18T19:02:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.811+0000 7ff01affd700 1 --2- 172.21.15.1:0/1601011665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff01c071d10 0x7ff01c15c6f0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff010000c00 tx=0x7ff010004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.811+0000 7ff003fff700 1 -- 172.21.15.1:0/1601011665 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0100038a0 con 0x7ff01c071d10 2024-03-18T19:02:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.811+0000 7ff003fff700 1 -- 172.21.15.1:0/1601011665 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff01001f070 con 0x7ff01c071d10 2024-03-18T19:02:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.811+0000 7ff0218e8700 1 -- 172.21.15.1:0/1601011665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff01c0ada50 con 0x7ff01c071d10 2024-03-18T19:02:56.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.811+0000 7ff003fff700 1 -- 172.21.15.1:0/1601011665 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff010003c60 con 0x7ff01c071d10 2024-03-18T19:02:56.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.811+0000 7ff0218e8700 1 -- 172.21.15.1:0/1601011665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff01c0ae010 con 0x7ff01c071d10 2024-03-18T19:02:56.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.812+0000 7ff0218e8700 1 -- 172.21.15.1:0/1601011665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff01c0676a0 con 0x7ff01c071d10 2024-03-18T19:02:56.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.812+0000 7ff003fff700 1 -- 172.21.15.1:0/1601011665 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff01001a430 con 0x7ff01c071d10 2024-03-18T19:02:56.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.812+0000 7ff003fff700 1 --2- 172.21.15.1:0/1601011665 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff00806adb0 0x7ff00806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:56.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.813+0000 7ff01a7fc700 1 --2- 172.21.15.1:0/1601011665 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff00806adb0 0x7ff00806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:56.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.813+0000 7ff003fff700 1 -- 172.21.15.1:0/1601011665 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5822+0+0 (secure 0 0 0) 0x7ff010021070 con 0x7ff01c071d10 2024-03-18T19:02:56.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.813+0000 7ff01a7fc700 1 --2- 172.21.15.1:0/1601011665 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff00806adb0 0x7ff00806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff01c0c65d0 tx=0x7ff00c0079c0 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:56.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:56.817+0000 7ff003fff700 1 -- 172.21.15.1:0/1601011665 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff010090020 con 0x7ff01c071d10 2024-03-18T19:02:57.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.065+0000 7ff0218e8700 1 -- 172.21.15.1:0/1601011665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff01c15c4d0 con 0x7ff01c071d10 2024-03-18T19:02:57.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.065+0000 7ff003fff700 1 -- 172.21.15.1:0/1601011665 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3523 (secure 0 0 0) 0x7ff01002b020 con 0x7ff01c071d10 2024-03-18T19:02:57.061 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:57.062 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4874,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2002167042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2002167042},{"type":"v1","addr":"172.21.15.165:6835","nonce":2002167042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4880,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3844556325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3844556325},{"type":"v1","addr":"172.21.15.1:6837","nonce":3844556325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5278,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1229938169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1229938169},{"type":"v1","addr":"172.21.15.165:6834","nonce":1229938169}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":6192,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/12071459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":12071459},{"type":"v1","addr":"172.21.15.1:6836","nonce":12071459}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[]} 2024-03-18T19:02:57.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.067+0000 7ff0218e8700 1 -- 172.21.15.1:0/1601011665 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff00806adb0 msgr2=0x7ff00806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:57.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.067+0000 7ff0218e8700 1 --2- 172.21.15.1:0/1601011665 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff00806adb0 0x7ff00806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff01c0c65d0 tx=0x7ff00c0079c0 comp rx=0 tx=0).stop 2024-03-18T19:02:57.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.067+0000 7ff0218e8700 1 -- 172.21.15.1:0/1601011665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff01c071d10 msgr2=0x7ff01c15c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:57.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.068+0000 7ff0218e8700 1 --2- 172.21.15.1:0/1601011665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff01c071d10 0x7ff01c15c6f0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff010000c00 tx=0x7ff010004750 comp rx=0 tx=0).stop 2024-03-18T19:02:57.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.068+0000 7ff0218e8700 1 -- 172.21.15.1:0/1601011665 shutdown_connections 2024-03-18T19:02:57.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.068+0000 7ff0218e8700 1 --2- 172.21.15.1:0/1601011665 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff00806adb0 0x7ff00806d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:57.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.068+0000 7ff0218e8700 1 --2- 172.21.15.1:0/1601011665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff01c153980 0x7ff01c15b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:57.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.068+0000 7ff0218e8700 1 --2- 172.21.15.1:0/1601011665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff01c071d10 0x7ff01c15c6f0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:57.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.068+0000 7ff0218e8700 1 --2- 172.21.15.1:0/1601011665 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff01c0cea30 0x7ff01c15cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:57.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.068+0000 7ff0218e8700 1 -- 172.21.15.1:0/1601011665 >> 172.21.15.1:0/1601011665 conn(0x7ff01c0cc990 msgr2=0x7ff01c072130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:57.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.068+0000 7ff0218e8700 1 -- 172.21.15.1:0/1601011665 shutdown_connections 2024-03-18T19:02:57.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.068+0000 7ff0218e8700 1 -- 172.21.15.1:0/1601011665 wait complete. 2024-03-18T19:02:57.065 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 19 2024-03-18T19:02:57.076 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T19:02:57.076 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-18T19:02:57.076 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T19:02:57.076 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:02:57.077 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T19:02:57.077 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-18T19:02:57.077 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-18T19:02:57.077 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:02:57.077 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T19:02:57.077 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-18T19:02:57.078 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-18T19:02:57.078 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:02:57.078 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T19:02:57.078 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-18T19:02:57.078 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-18T19:02:57.078 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:02:57.085 INFO:tasks.ceph.mds.a:Stopped 2024-03-18T19:02:57.085 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-18T19:02:57.086 INFO:tasks.ceph.mds.c:Stopped 2024-03-18T19:02:57.086 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-18T19:02:57.088 INFO:tasks.ceph.mds.b:Stopped 2024-03-18T19:02:57.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-18T19:02:57.089 INFO:tasks.ceph.mds.d:Stopped 2024-03-18T19:02:57.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-18T19:02:57.128 INFO:tasks.ceph.mds.a:Started 2024-03-18T19:02:57.128 DEBUG:teuthology.parallel:result is None 2024-03-18T19:02:57.129 INFO:tasks.ceph.mds.c:Started 2024-03-18T19:02:57.129 DEBUG:teuthology.parallel:result is None 2024-03-18T19:02:57.131 INFO:tasks.ceph.mds.b:Started 2024-03-18T19:02:57.131 DEBUG:teuthology.parallel:result is None 2024-03-18T19:02:57.132 INFO:tasks.ceph.mds.d:Started 2024-03-18T19:02:57.133 DEBUG:teuthology.parallel:result is None 2024-03-18T19:02:57.133 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-18T19:02:57.211 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T19:02:57.211 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-18T19:02:57.217 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-03-18T19:02:57.217 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-03-18T19:02:57.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.272+0000 7f5469bce700 1 Processor -- start 2024-03-18T19:02:57.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.272+0000 7f5469bce700 1 -- start start 2024-03-18T19:02:57.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.273+0000 7f5469bce700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54640cd430 0x7f54640b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:57.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.273+0000 7f5469bce700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54640d01f0 con 0x7f546415b6c0 2024-03-18T19:02:57.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.273+0000 7f5469bce700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54640ccfa0 con 0x7f54640c87d0 2024-03-18T19:02:57.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.273+0000 7f5469bce700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54640d1db0 con 0x7f54640cd430 2024-03-18T19:02:57.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.273+0000 7f54637fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f546415b6c0 0x7f54640c80c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38546/0 (socket says 172.21.15.1:38546) 2024-03-18T19:02:57.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.273+0000 7f54637fe700 1 -- 172.21.15.1:0/3981889616 learned_addr learned my addr 172.21.15.1:0/3981889616 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:57.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.273+0000 7f5462ffd700 1 --2- 172.21.15.1:0/3981889616 >> v2:172.21.15.1:3300/0 conn(0x7f54640cd430 0x7f54640b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:57.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.273+0000 7f5462ffd700 1 -- 172.21.15.1:0/3981889616 >> v1:172.21.15.165:6789/0 conn(0x7f54640c87d0 legacy=0x7f54640cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:57.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.273+0000 7f5462ffd700 1 -- 172.21.15.1:0/3981889616 >> v1:172.21.15.1:6789/0 conn(0x7f546415b6c0 legacy=0x7f54640c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:57.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.273+0000 7f5462ffd700 1 -- 172.21.15.1:0/3981889616 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54640cff10 con 0x7f54640cd430 2024-03-18T19:02:57.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.273+0000 7f54627fc700 1 -- 172.21.15.1:0/3981889616 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105218122 0 0) 0x7f54640d01f0 con 0x7f546415b6c0 2024-03-18T19:02:57.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.273+0000 7f5462ffd700 1 --2- 172.21.15.1:0/3981889616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54640cd430 0x7f54640b2160 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f5458009840 tx=0x7f5458009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a255bdc8cd0bc7c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:57.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.274+0000 7f54627fc700 1 -- 172.21.15.1:0/3981889616 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5458004230 con 0x7f54640cd430 2024-03-18T19:02:57.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.274+0000 7f54627fc700 1 -- 172.21.15.1:0/3981889616 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5458004390 con 0x7f54640cd430 2024-03-18T19:02:57.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.274+0000 7f54627fc700 1 -- 172.21.15.1:0/3981889616 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54580036a0 con 0x7f54640cd430 2024-03-18T19:02:57.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.274+0000 7f5469bce700 1 -- 172.21.15.1:0/3981889616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54640cd430 msgr2=0x7f54640b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:57.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.274+0000 7f5469bce700 1 --2- 172.21.15.1:0/3981889616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54640cd430 0x7f54640b2160 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f5458009840 tx=0x7f5458009b50 comp rx=0 tx=0).stop 2024-03-18T19:02:57.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.274+0000 7f5469bce700 1 -- 172.21.15.1:0/3981889616 shutdown_connections 2024-03-18T19:02:57.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.274+0000 7f5469bce700 1 --2- 172.21.15.1:0/3981889616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54640cd430 0x7f54640b2160 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:57.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.274+0000 7f5469bce700 1 -- 172.21.15.1:0/3981889616 >> 172.21.15.1:0/3981889616 conn(0x7f54640d0ff0 msgr2=0x7f54640d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:57.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.274+0000 7f5469bce700 1 -- 172.21.15.1:0/3981889616 shutdown_connections 2024-03-18T19:02:57.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.274+0000 7f5469bce700 1 -- 172.21.15.1:0/3981889616 wait complete. 2024-03-18T19:02:57.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.275+0000 7f5469bce700 1 Processor -- start 2024-03-18T19:02:57.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.276+0000 7f5469bce700 1 -- start start 2024-03-18T19:02:57.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.276+0000 7f5469bce700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54640c87d0 0x7f546414e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:57.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.276+0000 7f5469bce700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54640cd430 0x7f54640cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:57.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.276+0000 7f5469bce700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f546415b6c0 0x7f54640cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:57.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.276+0000 7f5469bce700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54641f5730 con 0x7f54640cd430 2024-03-18T19:02:57.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.276+0000 7f5469bce700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546409be10 con 0x7f54640c87d0 2024-03-18T19:02:57.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.276+0000 7f5469bce700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54641f4b60 con 0x7f546415b6c0 2024-03-18T19:02:57.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.276+0000 7f5462ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54640cd430 0x7f54640cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:57.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.276+0000 7f5462ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54640cd430 0x7f54640cd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48600/0 (socket says 172.21.15.1:48600) 2024-03-18T19:02:57.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.276+0000 7f5462ffd700 1 -- 172.21.15.1:0/2476160283 learned_addr learned my addr 172.21.15.1:0/2476160283 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:57.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.277+0000 7f54637fe700 1 --2- 172.21.15.1:0/2476160283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54640c87d0 0x7f546414e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:57.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.277+0000 7f5463fff700 1 --2- 172.21.15.1:0/2476160283 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f546415b6c0 0x7f54640cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:57.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.277+0000 7f5462ffd700 1 -- 172.21.15.1:0/2476160283 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f546415b6c0 msgr2=0x7f54640cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:57.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.277+0000 7f5462ffd700 1 --2- 172.21.15.1:0/2476160283 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f546415b6c0 0x7f54640cddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:57.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.277+0000 7f5462ffd700 1 -- 172.21.15.1:0/2476160283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54640c87d0 msgr2=0x7f546414e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:57.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.277+0000 7f5462ffd700 1 --2- 172.21.15.1:0/2476160283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54640c87d0 0x7f546414e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:57.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.277+0000 7f5462ffd700 1 -- 172.21.15.1:0/2476160283 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54580094a0 con 0x7f54640cd430 2024-03-18T19:02:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.277+0000 7f5463fff700 1 --2- 172.21.15.1:0/2476160283 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f546415b6c0 0x7f54640cddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.277+0000 7f5462ffd700 1 --2- 172.21.15.1:0/2476160283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54640cd430 0x7f54640cd870 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f54580090b0 tx=0x7f5458004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.277+0000 7f544bfff700 1 -- 172.21.15.1:0/2476160283 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5458011630 con 0x7f54640cd430 2024-03-18T19:02:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.277+0000 7f544bfff700 1 -- 172.21.15.1:0/2476160283 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545801e070 con 0x7f54640cd430 2024-03-18T19:02:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.277+0000 7f5469bce700 1 -- 172.21.15.1:0/2476160283 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54641f4e00 con 0x7f54640cd430 2024-03-18T19:02:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.277+0000 7f544bfff700 1 -- 172.21.15.1:0/2476160283 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54580119f0 con 0x7f54640cd430 2024-03-18T19:02:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.277+0000 7f5469bce700 1 -- 172.21.15.1:0/2476160283 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54641f53c0 con 0x7f54640cd430 2024-03-18T19:02:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.278+0000 7f5469bce700 1 -- 172.21.15.1:0/2476160283 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54640676a0 con 0x7f54640cd430 2024-03-18T19:02:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.279+0000 7f544bfff700 1 -- 172.21.15.1:0/2476160283 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5458011b50 con 0x7f54640cd430 2024-03-18T19:02:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.279+0000 7f544bfff700 1 --2- 172.21.15.1:0/2476160283 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f545006adb0 0x7f545006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.279+0000 7f544bfff700 1 -- 172.21.15.1:0/2476160283 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f545808a7b0 con 0x7f54640cd430 2024-03-18T19:02:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.279+0000 7f54637fe700 1 --2- 172.21.15.1:0/2476160283 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f545006adb0 0x7f545006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.279+0000 7f54637fe700 1 --2- 172.21.15.1:0/2476160283 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f545006adb0 0x7f545006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f54540012e0 tx=0x7f5454001aa0 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:57.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.284+0000 7f544bfff700 1 -- 172.21.15.1:0/2476160283 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f54580558a0 con 0x7f54640cd430 2024-03-18T19:02:57.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:57.501+0000 7f5469bce700 1 -- 172.21.15.1:0/2476160283 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f54640c45b0 con 0x7f54640cd430 2024-03-18T19:02:58.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.462+0000 7f544bfff700 1 -- 172.21.15.1:0/2476160283 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v67) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5458020080 con 0x7f54640cd430 2024-03-18T19:02:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.466+0000 7f5469bce700 1 -- 172.21.15.1:0/2476160283 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f545006adb0 msgr2=0x7f545006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.466+0000 7f5469bce700 1 --2- 172.21.15.1:0/2476160283 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f545006adb0 0x7f545006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f54540012e0 tx=0x7f5454001aa0 comp rx=0 tx=0).stop 2024-03-18T19:02:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.466+0000 7f5469bce700 1 -- 172.21.15.1:0/2476160283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54640cd430 msgr2=0x7f54640cd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.466+0000 7f5469bce700 1 --2- 172.21.15.1:0/2476160283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54640cd430 0x7f54640cd870 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f54580090b0 tx=0x7f5458004750 comp rx=0 tx=0).stop 2024-03-18T19:02:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.466+0000 7f5469bce700 1 -- 172.21.15.1:0/2476160283 shutdown_connections 2024-03-18T19:02:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.466+0000 7f5469bce700 1 --2- 172.21.15.1:0/2476160283 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f545006adb0 0x7f545006d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.466+0000 7f5469bce700 1 --2- 172.21.15.1:0/2476160283 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f546415b6c0 0x7f54640cddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.466+0000 7f5469bce700 1 --2- 172.21.15.1:0/2476160283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54640c87d0 0x7f546414e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.466+0000 7f5469bce700 1 --2- 172.21.15.1:0/2476160283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54640cd430 0x7f54640cd870 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.466+0000 7f5469bce700 1 -- 172.21.15.1:0/2476160283 >> 172.21.15.1:0/2476160283 conn(0x7f54640d0ff0 msgr2=0x7f54640ca250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.466+0000 7f5469bce700 1 -- 172.21.15.1:0/2476160283 shutdown_connections 2024-03-18T19:02:58.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.466+0000 7f5469bce700 1 -- 172.21.15.1:0/2476160283 wait complete. 2024-03-18T19:02:58.463 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-18T19:02:58.477 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-18T19:02:58.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.613+0000 7fa6515d4700 1 Processor -- start 2024-03-18T19:02:58.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.613+0000 7fa6515d4700 1 -- start start 2024-03-18T19:02:58.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.614+0000 7fa6515d4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa64c06eb30 0x7fa64c0720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:58.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.614+0000 7fa6515d4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa64c06efb0 0x7fa64c072630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:58.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.614+0000 7fa6515d4700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c144dc0 con 0x7fa64c06a750 2024-03-18T19:02:58.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.614+0000 7fa6515d4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa64c071650 con 0x7fa64c06efb0 2024-03-18T19:02:58.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.614+0000 7fa6515d4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa64c0cf970 con 0x7fa64c06eb30 2024-03-18T19:02:58.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.614+0000 7fa64b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa64c06efb0 0x7fa64c072630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:58.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.614+0000 7fa64b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa64c06efb0 0x7fa64c072630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48602/0 (socket says 172.21.15.1:48602) 2024-03-18T19:02:58.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.614+0000 7fa64b7fe700 1 -- 172.21.15.1:0/3122457771 learned_addr learned my addr 172.21.15.1:0/3122457771 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:58.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.615+0000 7fa64a7fc700 1 --2- 172.21.15.1:0/3122457771 >> v2:172.21.15.165:3300/0 conn(0x7fa64c06eb30 0x7fa64c0720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:58.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.615+0000 7fa64b7fe700 1 -- 172.21.15.1:0/3122457771 >> v2:172.21.15.165:3300/0 conn(0x7fa64c06eb30 msgr2=0x7fa64c0720f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:58.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.615+0000 7fa64b7fe700 1 --2- 172.21.15.1:0/3122457771 >> v2:172.21.15.165:3300/0 conn(0x7fa64c06eb30 0x7fa64c0720f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:58.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.615+0000 7fa64b7fe700 1 -- 172.21.15.1:0/3122457771 >> v1:172.21.15.165:6789/0 conn(0x7fa64c06a750 legacy=0x7fa64c06ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:58.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.615+0000 7fa64b7fe700 1 -- 172.21.15.1:0/3122457771 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c0cfc50 con 0x7fa64c06efb0 2024-03-18T19:02:58.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.615+0000 7fa64b7fe700 1 --2- 172.21.15.1:0/3122457771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c06efb0 0x7fa64c072630 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fa64c15ec10 tx=0x7fa64400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb795996c9eef824 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:58.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.615+0000 7fa649ffb700 1 -- 172.21.15.1:0/3122457771 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64400e070 con 0x7fa64c06efb0 2024-03-18T19:02:58.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.615+0000 7fa649ffb700 1 -- 172.21.15.1:0/3122457771 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa644003a20 con 0x7fa64c06efb0 2024-03-18T19:02:58.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.615+0000 7fa649ffb700 1 -- 172.21.15.1:0/3122457771 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa644003dd0 con 0x7fa64c06efb0 2024-03-18T19:02:58.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.615+0000 7fa6515d4700 1 -- 172.21.15.1:0/3122457771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c06efb0 msgr2=0x7fa64c072630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:58.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.615+0000 7fa6515d4700 1 --2- 172.21.15.1:0/3122457771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c06efb0 0x7fa64c072630 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fa64c15ec10 tx=0x7fa64400b330 comp rx=0 tx=0).stop 2024-03-18T19:02:58.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.616+0000 7fa6515d4700 1 -- 172.21.15.1:0/3122457771 shutdown_connections 2024-03-18T19:02:58.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.616+0000 7fa6515d4700 1 --2- 172.21.15.1:0/3122457771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c06efb0 0x7fa64c072630 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:58.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.616+0000 7fa6515d4700 1 --2- 172.21.15.1:0/3122457771 >> v2:172.21.15.165:3300/0 conn(0x7fa64c06eb30 0x7fa64c0720f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:58.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.616+0000 7fa6515d4700 1 -- 172.21.15.1:0/3122457771 >> 172.21.15.1:0/3122457771 conn(0x7fa64c0b1d20 msgr2=0x7fa64c0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:58.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.616+0000 7fa6515d4700 1 -- 172.21.15.1:0/3122457771 shutdown_connections 2024-03-18T19:02:58.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.616+0000 7fa6515d4700 1 -- 172.21.15.1:0/3122457771 wait complete. 2024-03-18T19:02:58.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.617+0000 7fa6515d4700 1 Processor -- start 2024-03-18T19:02:58.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.617+0000 7fa6515d4700 1 -- start start 2024-03-18T19:02:58.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa6515d4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c06a750 0x7fa64c13d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:58.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa6515d4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64c06eb30 0x7fa64c15d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:58.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa6515d4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa64c06efb0 0x7fa64c15a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:58.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa6515d4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c0c59a0 con 0x7fa64c06a750 2024-03-18T19:02:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa6515d4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c0c5bf0 con 0x7fa64c06eb30 2024-03-18T19:02:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa6515d4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c0c5ed0 con 0x7fa64c06efb0 2024-03-18T19:02:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa64affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c06a750 0x7fa64c13d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa64affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c06a750 0x7fa64c13d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48604/0 (socket says 172.21.15.1:48604) 2024-03-18T19:02:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa64affd700 1 -- 172.21.15.1:0/3405113714 learned_addr learned my addr 172.21.15.1:0/3405113714 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa64a7fc700 1 --2- 172.21.15.1:0/3405113714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64c06eb30 0x7fa64c15d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa64b7fe700 1 --2- 172.21.15.1:0/3405113714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa64c06efb0 0x7fa64c15a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa64affd700 1 -- 172.21.15.1:0/3405113714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa64c06efb0 msgr2=0x7fa64c15a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa64affd700 1 --2- 172.21.15.1:0/3405113714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa64c06efb0 0x7fa64c15a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa64affd700 1 -- 172.21.15.1:0/3405113714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64c06eb30 msgr2=0x7fa64c15d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa64affd700 1 --2- 172.21.15.1:0/3405113714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64c06eb30 0x7fa64c15d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa64affd700 1 -- 172.21.15.1:0/3405113714 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa644009b30 con 0x7fa64c06a750 2024-03-18T19:02:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa64affd700 1 --2- 172.21.15.1:0/3405113714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c06a750 0x7fa64c13d510 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fa64000c4d0 tx=0x7fa64000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:58.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa6337fe700 1 -- 172.21.15.1:0/3405113714 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa640011790 con 0x7fa64c06a750 2024-03-18T19:02:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa6337fe700 1 -- 172.21.15.1:0/3405113714 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6400118f0 con 0x7fa64c06a750 2024-03-18T19:02:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa6515d4700 1 -- 172.21.15.1:0/3405113714 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64c0c61b0 con 0x7fa64c06a750 2024-03-18T19:02:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa6337fe700 1 -- 172.21.15.1:0/3405113714 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa640011ca0 con 0x7fa64c06a750 2024-03-18T19:02:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.618+0000 7fa6515d4700 1 -- 172.21.15.1:0/3405113714 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa64c13da50 con 0x7fa64c06a750 2024-03-18T19:02:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.620+0000 7fa6515d4700 1 -- 172.21.15.1:0/3405113714 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64c071130 con 0x7fa64c06a750 2024-03-18T19:02:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.620+0000 7fa6337fe700 1 -- 172.21.15.1:0/3405113714 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa64001c460 con 0x7fa64c06a750 2024-03-18T19:02:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.620+0000 7fa6337fe700 1 --2- 172.21.15.1:0/3405113714 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa63806adb0 0x7fa63806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.620+0000 7fa6337fe700 1 -- 172.21.15.1:0/3405113714 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fa6400890d0 con 0x7fa64c06a750 2024-03-18T19:02:58.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.620+0000 7fa64a7fc700 1 --2- 172.21.15.1:0/3405113714 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa63806adb0 0x7fa63806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:58.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.621+0000 7fa64a7fc700 1 --2- 172.21.15.1:0/3405113714 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa63806adb0 0x7fa63806d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa64c0cdcf0 tx=0x7fa63c008040 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:58.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.625+0000 7fa6337fe700 1 -- 172.21.15.1:0/3405113714 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa6400543a0 con 0x7fa64c06a750 2024-03-18T19:02:58.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.909+0000 7fa6515d4700 1 -- 172.21.15.1:0/3405113714 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa64c0676a0 con 0x7fa64c06a750 2024-03-18T19:02:58.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.909+0000 7fa6337fe700 1 -- 172.21.15.1:0/3405113714 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fa6400579c0 con 0x7fa64c06a750 2024-03-18T19:02:58.905 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:58.905 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:02:58.905 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-18T19:02:58.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:58.905 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-18T19:02:58.905 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlR93/CxAAbpe+5n3wHRur9xi8QzdCnQ==", 2024-03-18T19:02:58.906 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:02:58.906 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:02:58.906 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:02:58.906 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:02:58.906 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:02:58.906 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.906 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:02:58.906 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:58.906 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-18T19:02:58.906 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJ3OmGBAA37u227nxrMsh/JHt71HBxQ==", 2024-03-18T19:02:58.906 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:02:58.906 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:02:58.906 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:02:58.907 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:02:58.907 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:02:58.907 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:02:58.907 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:58.907 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-18T19:02:58.907 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJZFoExAAmI16oxwNKbf8dKetPfyk+g==", 2024-03-18T19:02:58.907 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:02:58.907 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:02:58.907 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:02:58.907 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:02:58.907 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:02:58.908 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:02:58.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:58.908 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-18T19:02:58.908 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlQrcvIBAA1LMv+EgC7S6wspX6QF49Pg==", 2024-03-18T19:02:58.908 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:02:58.908 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:02:58.908 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:02:58.908 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:02:58.908 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:02:58.908 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.908 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:02:58.908 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:58.909 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-18T19:02:58.909 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCUjPhlYOVHChAARFQrksZmMIdTlntd8TgvrQ==", 2024-03-18T19:02:58.909 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:02:58.909 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:02:58.909 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:02:58.909 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:02:58.909 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:02:58.909 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:58.909 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-18T19:02:58.909 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCXjPhl2zSDCBAA6Al1gkWVB79ZI9djGpVChA==", 2024-03-18T19:02:58.909 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:02:58.909 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:02:58.909 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:02:58.909 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:02:58.910 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:02:58.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:58.910 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-18T19:02:58.910 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCajPhlSoG/BhAAR93USDe6DCQW+hREOsnTlw==", 2024-03-18T19:02:58.910 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:02:58.910 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:02:58.910 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:02:58.910 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:02:58.910 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:02:58.910 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:58.911 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-18T19:02:58.911 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCdjPhlhJg5BRAAWuQhcnxyjR5UOiM13yKdAQ==", 2024-03-18T19:02:58.911 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:02:58.911 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:02:58.911 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:02:58.911 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:02:58.911 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.911 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:02:58.911 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:58.911 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-18T19:02:58.911 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCjjPhlCFnpCRAAKIyOQyVdjXZUTnAS6Nw/ng==", 2024-03-18T19:02:58.911 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:02:58.911 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:02:58.912 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:02:58.912 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:02:58.912 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:02:58.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:58.912 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-18T19:02:58.912 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCmjPhlAeFkCBAAJcEFpRk/03lQJxj+l2hEPA==", 2024-03-18T19:02:58.912 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:02:58.912 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:02:58.912 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:02:58.912 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:02:58.912 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:02:58.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:58.912 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-18T19:02:58.913 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCpjPhl/xuuBhAA0LgKfXEvN2O11ZlRQJVOkw==", 2024-03-18T19:02:58.913 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:02:58.913 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:02:58.913 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:02:58.913 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:02:58.913 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:02:58.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:58.913 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-18T19:02:58.913 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCsjPhldiR4BRAA571oh6U8PzOv/4LzjYLF/A==", 2024-03-18T19:02:58.913 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:02:58.913 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:02:58.913 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:02:58.913 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:02:58.914 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:02:58.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:58.914 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-18T19:02:58.914 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlu+2xJBAANnXGGjfeqP6S2FTecsfOsg==", 2024-03-18T19:02:58.914 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:02:58.914 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:02:58.914 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-18T19:02:58.914 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-18T19:02:58.914 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:02:58.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:58.914 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-18T19:02:58.914 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlhJ9+KxAAEnrQrCrgJR6xI+pucwzAXA==", 2024-03-18T19:02:58.915 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:02:58.915 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:02:58.915 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:02:58.915 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:02:58.915 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:02:58.915 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:02:58.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:58.915 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.2", 2024-03-18T19:02:58.915 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlxBUpKBAAIGTwmLWyj+P2Yvl55QVcWA==", 2024-03-18T19:02:58.915 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:02:58.915 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:02:58.915 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:02:58.915 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:02:58.916 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:02:58.916 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:02:58.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:58.916 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.3", 2024-03-18T19:02:58.916 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlmao3MRAAHeiQ6eyBh9QjjKPCtxkC8g==", 2024-03-18T19:02:58.916 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:02:58.916 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:02:58.916 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:02:58.916 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:02:58.916 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:02:58.916 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:02:58.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:58.916 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-18T19:02:58.917 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCRjPhlemOuKRAAVUggmtB7+EiVgyfJZ8TCCA==", 2024-03-18T19:02:58.917 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:02:58.917 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:02:58.917 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:02:58.917 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:02:58.917 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:02:58.917 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:02:58.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:58.917 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-18T19:02:58.917 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhleIuOCBAA9CH+AHewQbn+gpVr+6lJyw==", 2024-03-18T19:02:58.917 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:02:58.917 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:02:58.917 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T19:02:58.918 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:02:58.918 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:02:58.918 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:58.918 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-18T19:02:58.918 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlL0pTBBAAdFiPcdwwz514cuLAvioxYA==", 2024-03-18T19:02:58.918 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:02:58.918 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:02:58.918 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T19:02:58.918 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:02:58.918 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.918 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:58.918 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-18T19:02:58.918 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:02:58.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.911+0000 7fa6515d4700 1 -- 172.21.15.1:0/3405113714 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa63806adb0 msgr2=0x7fa63806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:58.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.911+0000 7fa6515d4700 1 --2- 172.21.15.1:0/3405113714 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa63806adb0 0x7fa63806d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa64c0cdcf0 tx=0x7fa63c008040 comp rx=0 tx=0).stop 2024-03-18T19:02:58.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.911+0000 7fa6515d4700 1 -- 172.21.15.1:0/3405113714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c06a750 msgr2=0x7fa64c13d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:58.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.911+0000 7fa6515d4700 1 --2- 172.21.15.1:0/3405113714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c06a750 0x7fa64c13d510 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fa64000c4d0 tx=0x7fa64000c890 comp rx=0 tx=0).stop 2024-03-18T19:02:58.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.911+0000 7fa6515d4700 1 -- 172.21.15.1:0/3405113714 shutdown_connections 2024-03-18T19:02:58.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.911+0000 7fa6515d4700 1 --2- 172.21.15.1:0/3405113714 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa63806adb0 0x7fa63806d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:58.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.911+0000 7fa6515d4700 1 --2- 172.21.15.1:0/3405113714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa64c06efb0 0x7fa64c15a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:58.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.911+0000 7fa6515d4700 1 --2- 172.21.15.1:0/3405113714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa64c06a750 0x7fa64c13d510 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:58.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.911+0000 7fa6515d4700 1 --2- 172.21.15.1:0/3405113714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64c06eb30 0x7fa64c15d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:58.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.911+0000 7fa6515d4700 1 -- 172.21.15.1:0/3405113714 >> 172.21.15.1:0/3405113714 conn(0x7fa64c0b1d20 msgr2=0x7fa64c1423b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:58.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.911+0000 7fa6515d4700 1 -- 172.21.15.1:0/3405113714 shutdown_connections 2024-03-18T19:02:58.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:58.912+0000 7fa6515d4700 1 -- 172.21.15.1:0/3405113714 wait complete. 2024-03-18T19:02:58.921 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:02:59.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.051+0000 7fb68b905700 1 Processor -- start 2024-03-18T19:02:59.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.051+0000 7fb68b905700 1 -- start start 2024-03-18T19:02:59.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.052+0000 7fb68b905700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb68406e290 0x7fb684072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.052+0000 7fb68b905700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68406f490 con 0x7fb68406dd50 2024-03-18T19:02:59.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.052+0000 7fb68b905700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb684071cf0 con 0x7fb68415d580 2024-03-18T19:02:59.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.052+0000 7fb68b905700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb68406f1b0 con 0x7fb68406e290 2024-03-18T19:02:59.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.052+0000 7fb689ea2700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb68406dd50 0x7fb6840ca560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38552/0 (socket says 172.21.15.1:38552) 2024-03-18T19:02:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.052+0000 7fb689ea2700 1 -- 172.21.15.1:0/2198982660 learned_addr learned my addr 172.21.15.1:0/2198982660 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.052+0000 7fb688ea0700 1 --2- 172.21.15.1:0/2198982660 >> v2:172.21.15.165:3300/0 conn(0x7fb68406e290 0x7fb684072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb67bfff700 1 -- 172.21.15.1:0/2198982660 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2857255664 0 0) 0x7fb68406f490 con 0x7fb68406dd50 2024-03-18T19:02:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb67bfff700 1 -- 172.21.15.1:0/2198982660 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6700035b0 con 0x7fb68406dd50 2024-03-18T19:02:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb67bfff700 1 -- 172.21.15.1:0/2198982660 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1512422541 0 0) 0x7fb6700035b0 con 0x7fb68406dd50 2024-03-18T19:02:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb67bfff700 1 -- 172.21.15.1:0/2198982660 >> v2:172.21.15.165:3300/0 conn(0x7fb68406e290 msgr2=0x7fb684072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb67bfff700 1 --2- 172.21.15.1:0/2198982660 >> v2:172.21.15.165:3300/0 conn(0x7fb68406e290 0x7fb684072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb67bfff700 1 -- 172.21.15.1:0/2198982660 >> v1:172.21.15.165:6789/0 conn(0x7fb68415d580 legacy=0x7fb68406d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb67bfff700 1 -- 172.21.15.1:0/2198982660 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6840716a0 con 0x7fb68406dd50 2024-03-18T19:02:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb67bfff700 1 -- 172.21.15.1:0/2198982660 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fb67c0019f0 con 0x7fb68406dd50 2024-03-18T19:02:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb67bfff700 1 -- 172.21.15.1:0/2198982660 >> v1:172.21.15.1:6789/0 conn(0x7fb68406dd50 legacy=0x7fb6840ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb67bfff700 1 --2- 172.21.15.1:0/2198982660 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb670003a00 0x7fb670003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb67bfff700 1 --2- 172.21.15.1:0/2198982660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb670004400 0x7fb6700048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb67bfff700 1 --2- 172.21.15.1:0/2198982660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6700050d0 0x7fb67000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb67bfff700 1 -- 172.21.15.1:0/2198982660 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68406f1b0 con 0x7fb6700050d0 2024-03-18T19:02:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb67bfff700 1 -- 172.21.15.1:0/2198982660 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67000bf80 con 0x7fb670004400 2024-03-18T19:02:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb67bfff700 1 -- 172.21.15.1:0/2198982660 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67000c140 con 0x7fb670003a00 2024-03-18T19:02:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb689ea2700 1 --2- 172.21.15.1:0/2198982660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6700050d0 0x7fb67000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb688ea0700 1 --2- 172.21.15.1:0/2198982660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb670004400 0x7fb6700048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb6896a1700 1 --2- 172.21.15.1:0/2198982660 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb670003a00 0x7fb670003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb689ea2700 1 -- 172.21.15.1:0/2198982660 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb670003a00 msgr2=0x7fb670003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb689ea2700 1 --2- 172.21.15.1:0/2198982660 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb670003a00 0x7fb670003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb689ea2700 1 -- 172.21.15.1:0/2198982660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb670004400 msgr2=0x7fb6700048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb689ea2700 1 --2- 172.21.15.1:0/2198982660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb670004400 0x7fb6700048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb689ea2700 1 -- 172.21.15.1:0/2198982660 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6700035b0 con 0x7fb6700050d0 2024-03-18T19:02:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb6896a1700 1 --2- 172.21.15.1:0/2198982660 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb670003a00 0x7fb670003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:02:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb689ea2700 1 -- 172.21.15.1:0/2198982660 reap_dead start 2024-03-18T19:02:59.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.053+0000 7fb689ea2700 1 --2- 172.21.15.1:0/2198982660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6700050d0 0x7fb67000ba10 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb6840cfe60 tx=0x7fb670005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9941b254eb682d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:59.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.054+0000 7fb67bfff700 1 -- 172.21.15.1:0/2198982660 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c012600 con 0x7fb6700050d0 2024-03-18T19:02:59.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.054+0000 7fb67bfff700 1 -- 172.21.15.1:0/2198982660 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67c012780 con 0x7fb6700050d0 2024-03-18T19:02:59.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.054+0000 7fb67bfff700 1 -- 172.21.15.1:0/2198982660 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c012b30 con 0x7fb6700050d0 2024-03-18T19:02:59.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.054+0000 7fb68b905700 1 -- 172.21.15.1:0/2198982660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6700050d0 msgr2=0x7fb67000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:59.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.054+0000 7fb68b905700 1 --2- 172.21.15.1:0/2198982660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6700050d0 0x7fb67000ba10 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb6840cfe60 tx=0x7fb670005880 comp rx=0 tx=0).stop 2024-03-18T19:02:59.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.054+0000 7fb68b905700 1 -- 172.21.15.1:0/2198982660 shutdown_connections 2024-03-18T19:02:59.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.054+0000 7fb68b905700 1 --2- 172.21.15.1:0/2198982660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6700050d0 0x7fb67000ba10 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.054+0000 7fb68b905700 1 -- 172.21.15.1:0/2198982660 >> 172.21.15.1:0/2198982660 conn(0x7fb6840cc990 msgr2=0x7fb6840ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:59.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.054+0000 7fb68b905700 1 -- 172.21.15.1:0/2198982660 shutdown_connections 2024-03-18T19:02:59.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.054+0000 7fb68b905700 1 -- 172.21.15.1:0/2198982660 wait complete. 2024-03-18T19:02:59.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.055+0000 7fb68b905700 1 Processor -- start 2024-03-18T19:02:59.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.056+0000 7fb68b905700 1 -- start start 2024-03-18T19:02:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.056+0000 7fb68b905700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6700050d0 0x7fb6841f2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.056+0000 7fb68b905700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb684158c80 0x7fb68415f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.056+0000 7fb68b905700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb684159100 0x7fb6841e59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.056+0000 7fb68b905700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6840b38a0 con 0x7fb684158c80 2024-03-18T19:02:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.056+0000 7fb68b905700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6840b3b80 con 0x7fb684159100 2024-03-18T19:02:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.056+0000 7fb68b905700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6840b3e60 con 0x7fb6700050d0 2024-03-18T19:02:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.056+0000 7fb688ea0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb684158c80 0x7fb68415f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.056+0000 7fb688ea0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb684158c80 0x7fb68415f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48630/0 (socket says 172.21.15.1:48630) 2024-03-18T19:02:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.056+0000 7fb688ea0700 1 -- 172.21.15.1:0/3719692291 learned_addr learned my addr 172.21.15.1:0/3719692291 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.056+0000 7fb6896a1700 1 --2- 172.21.15.1:0/3719692291 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6700050d0 0x7fb6841f2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.057+0000 7fb689ea2700 1 --2- 172.21.15.1:0/3719692291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb684159100 0x7fb6841e59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.057+0000 7fb688ea0700 1 -- 172.21.15.1:0/3719692291 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6700050d0 msgr2=0x7fb6841f2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.057+0000 7fb688ea0700 1 --2- 172.21.15.1:0/3719692291 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6700050d0 0x7fb6841f2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.057+0000 7fb688ea0700 1 -- 172.21.15.1:0/3719692291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb684159100 msgr2=0x7fb6841e59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.057+0000 7fb688ea0700 1 --2- 172.21.15.1:0/3719692291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb684159100 0x7fb6841e59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.057+0000 7fb688ea0700 1 -- 172.21.15.1:0/3719692291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6840b4140 con 0x7fb684158c80 2024-03-18T19:02:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.057+0000 7fb688ea0700 1 --2- 172.21.15.1:0/3719692291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb684158c80 0x7fb68415f480 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fb6800119a0 tx=0x7fb680011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.057+0000 7fb679ffb700 1 -- 172.21.15.1:0/3719692291 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68000f940 con 0x7fb684158c80 2024-03-18T19:02:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.057+0000 7fb679ffb700 1 -- 172.21.15.1:0/3719692291 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68000faa0 con 0x7fb684158c80 2024-03-18T19:02:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.057+0000 7fb68b905700 1 -- 172.21.15.1:0/3719692291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6840b4420 con 0x7fb684158c80 2024-03-18T19:02:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.057+0000 7fb679ffb700 1 -- 172.21.15.1:0/3719692291 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68000fe30 con 0x7fb684158c80 2024-03-18T19:02:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.057+0000 7fb68b905700 1 -- 172.21.15.1:0/3719692291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6841f3290 con 0x7fb684158c80 2024-03-18T19:02:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.058+0000 7fb68b905700 1 -- 172.21.15.1:0/3719692291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6840676a0 con 0x7fb684158c80 2024-03-18T19:02:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.059+0000 7fb679ffb700 1 -- 172.21.15.1:0/3719692291 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb68001d400 con 0x7fb684158c80 2024-03-18T19:02:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.059+0000 7fb679ffb700 1 --2- 172.21.15.1:0/3719692291 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb66806adb0 0x7fb66806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.059+0000 7fb679ffb700 1 -- 172.21.15.1:0/3719692291 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fb680089960 con 0x7fb684158c80 2024-03-18T19:02:59.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.059+0000 7fb6896a1700 1 --2- 172.21.15.1:0/3719692291 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb66806adb0 0x7fb66806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.059+0000 7fb6896a1700 1 --2- 172.21.15.1:0/3719692291 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb66806adb0 0x7fb66806d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb674007fd0 tx=0x7fb674009040 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:59.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.063+0000 7fb679ffb700 1 -- 172.21.15.1:0/3719692291 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb680054c40 con 0x7fb684158c80 2024-03-18T19:02:59.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.266+0000 7fb68b905700 1 -- 172.21.15.1:0/3719692291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb68415bc00 con 0x7fb684158c80 2024-03-18T19:02:59.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.267+0000 7fb679ffb700 1 -- 172.21.15.1:0/3719692291 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+12546 (secure 0 0 0) 0x7fb680058260 con 0x7fb684158c80 2024-03-18T19:02:59.263 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:59.263 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":67,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:02:58.459102+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:02:59.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.269+0000 7fb68b905700 1 -- 172.21.15.1:0/3719692291 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb66806adb0 msgr2=0x7fb66806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:59.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.269+0000 7fb68b905700 1 --2- 172.21.15.1:0/3719692291 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb66806adb0 0x7fb66806d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb674007fd0 tx=0x7fb674009040 comp rx=0 tx=0).stop 2024-03-18T19:02:59.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.269+0000 7fb68b905700 1 -- 172.21.15.1:0/3719692291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb684158c80 msgr2=0x7fb68415f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:59.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.269+0000 7fb68b905700 1 --2- 172.21.15.1:0/3719692291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb684158c80 0x7fb68415f480 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fb6800119a0 tx=0x7fb680011cb0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.269+0000 7fb68b905700 1 -- 172.21.15.1:0/3719692291 shutdown_connections 2024-03-18T19:02:59.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.269+0000 7fb68b905700 1 --2- 172.21.15.1:0/3719692291 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb66806adb0 0x7fb66806d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.269+0000 7fb68b905700 1 --2- 172.21.15.1:0/3719692291 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6700050d0 0x7fb6841f2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.269+0000 7fb68b905700 1 --2- 172.21.15.1:0/3719692291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb684159100 0x7fb6841e59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.269+0000 7fb68b905700 1 --2- 172.21.15.1:0/3719692291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb684158c80 0x7fb68415f480 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.269+0000 7fb68b905700 1 -- 172.21.15.1:0/3719692291 >> 172.21.15.1:0/3719692291 conn(0x7fb6840cc990 msgr2=0x7fb6840ae1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:59.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.269+0000 7fb68b905700 1 -- 172.21.15.1:0/3719692291 shutdown_connections 2024-03-18T19:02:59.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.269+0000 7fb68b905700 1 -- 172.21.15.1:0/3719692291 wait complete. 2024-03-18T19:02:59.277 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:02:59.278 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:02:59.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.409+0000 7fe1af957700 1 Processor -- start 2024-03-18T19:02:59.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.409+0000 7fe1af957700 1 -- start start 2024-03-18T19:02:59.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.410+0000 7fe1af957700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1a815d3b0 0x7fe1a806cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.410+0000 7fe1af957700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe1a806d430 0x7fe1a8070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.410+0000 7fe1af957700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a80b2b80 con 0x7fe1a80b2650 2024-03-18T19:02:59.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.410+0000 7fe1af957700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a80b3800 con 0x7fe1a806d430 2024-03-18T19:02:59.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.410+0000 7fe1af957700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a80b2e60 con 0x7fe1a815d3b0 2024-03-18T19:02:59.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.410+0000 7fe1acef2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe1a806d430 0x7fe1a8070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.410+0000 7fe1adef4700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe1a80b2650 0x7fe1a80b3c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38568/0 (socket says 172.21.15.1:38568) 2024-03-18T19:02:59.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.410+0000 7fe1acef2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe1a806d430 0x7fe1a8070e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48644/0 (socket says 172.21.15.1:48644) 2024-03-18T19:02:59.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.410+0000 7fe1adef4700 1 -- 172.21.15.1:0/2096496985 learned_addr learned my addr 172.21.15.1:0/2096496985 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:59.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.410+0000 7fe1ad6f3700 1 --2- 172.21.15.1:0/2096496985 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1a815d3b0 0x7fe1a806cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.410+0000 7fe1acef2700 1 -- 172.21.15.1:0/2096496985 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1a815d3b0 msgr2=0x7fe1a806cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:59.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.410+0000 7fe1acef2700 1 --2- 172.21.15.1:0/2096496985 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1a815d3b0 0x7fe1a806cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.410+0000 7fe1acef2700 1 -- 172.21.15.1:0/2096496985 >> v1:172.21.15.1:6789/0 conn(0x7fe1a80b2650 legacy=0x7fe1a80b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:59.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.410+0000 7fe19ffff700 1 -- 172.21.15.1:0/2096496985 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075857678 0 0) 0x7fe1a80b2b80 con 0x7fe1a80b2650 2024-03-18T19:02:59.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.411+0000 7fe1acef2700 1 -- 172.21.15.1:0/2096496985 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a80b3140 con 0x7fe1a806d430 2024-03-18T19:02:59.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.411+0000 7fe1acef2700 1 --2- 172.21.15.1:0/2096496985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1a806d430 0x7fe1a8070e40 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4009840 tx=0x7fe1a4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610b14fcb8f2f246 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:59.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.411+0000 7fe19ffff700 1 -- 172.21.15.1:0/2096496985 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a4011830 con 0x7fe1a806d430 2024-03-18T19:02:59.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.411+0000 7fe19ffff700 1 -- 172.21.15.1:0/2096496985 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a4011990 con 0x7fe1a806d430 2024-03-18T19:02:59.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.411+0000 7fe19ffff700 1 -- 172.21.15.1:0/2096496985 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a4011d20 con 0x7fe1a806d430 2024-03-18T19:02:59.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.411+0000 7fe1af957700 1 -- 172.21.15.1:0/2096496985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1a806d430 msgr2=0x7fe1a8070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:59.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.411+0000 7fe1af957700 1 --2- 172.21.15.1:0/2096496985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1a806d430 0x7fe1a8070e40 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4009840 tx=0x7fe1a4009b50 comp rx=0 tx=0).stop 2024-03-18T19:02:59.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.411+0000 7fe1af957700 1 -- 172.21.15.1:0/2096496985 shutdown_connections 2024-03-18T19:02:59.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.411+0000 7fe1af957700 1 --2- 172.21.15.1:0/2096496985 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1a815d3b0 0x7fe1a806cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.411+0000 7fe1af957700 1 --2- 172.21.15.1:0/2096496985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1a806d430 0x7fe1a8070e40 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.411+0000 7fe1af957700 1 -- 172.21.15.1:0/2096496985 >> 172.21.15.1:0/2096496985 conn(0x7fe1a80b1350 msgr2=0x7fe1a80b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:59.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.411+0000 7fe1af957700 1 -- 172.21.15.1:0/2096496985 shutdown_connections 2024-03-18T19:02:59.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.412+0000 7fe1af957700 1 -- 172.21.15.1:0/2096496985 wait complete. 2024-03-18T19:02:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.413+0000 7fe1af957700 1 Processor -- start 2024-03-18T19:02:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.413+0000 7fe1af957700 1 -- start start 2024-03-18T19:02:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.413+0000 7fe1af957700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1a806d430 0x7fe1a8149200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.413+0000 7fe1af957700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1a80b2650 0x7fe1a8157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.413+0000 7fe1af957700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1a815d3b0 0x7fe1a81551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.413+0000 7fe1af957700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a8157b30 con 0x7fe1a806d430 2024-03-18T19:02:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.413+0000 7fe1af957700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a8152db0 con 0x7fe1a815d3b0 2024-03-18T19:02:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.413+0000 7fe1af957700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a8153090 con 0x7fe1a80b2650 2024-03-18T19:02:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.413+0000 7fe1ad6f3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1a806d430 0x7fe1a8149200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.414+0000 7fe1ad6f3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1a806d430 0x7fe1a8149200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48660/0 (socket says 172.21.15.1:48660) 2024-03-18T19:02:59.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.414+0000 7fe1ad6f3700 1 -- 172.21.15.1:0/1068933416 learned_addr learned my addr 172.21.15.1:0/1068933416 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:59.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.414+0000 7fe1acef2700 1 --2- 172.21.15.1:0/1068933416 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1a80b2650 0x7fe1a8157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.414+0000 7fe1adef4700 1 --2- 172.21.15.1:0/1068933416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1a815d3b0 0x7fe1a81551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.414+0000 7fe1ad6f3700 1 -- 172.21.15.1:0/1068933416 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1a80b2650 msgr2=0x7fe1a8157220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:59.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.414+0000 7fe1ad6f3700 1 --2- 172.21.15.1:0/1068933416 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1a80b2650 0x7fe1a8157220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.414+0000 7fe1ad6f3700 1 -- 172.21.15.1:0/1068933416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1a815d3b0 msgr2=0x7fe1a81551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:59.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.414+0000 7fe1ad6f3700 1 --2- 172.21.15.1:0/1068933416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1a815d3b0 0x7fe1a81551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.414+0000 7fe1ad6f3700 1 -- 172.21.15.1:0/1068933416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a40094a0 con 0x7fe1a806d430 2024-03-18T19:02:59.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.414+0000 7fe1ad6f3700 1 --2- 172.21.15.1:0/1068933416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1a806d430 0x7fe1a8149200 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe19800d800 tx=0x7fe19800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:59.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.414+0000 7fe19dffb700 1 -- 172.21.15.1:0/1068933416 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1980125f0 con 0x7fe1a806d430 2024-03-18T19:02:59.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.414+0000 7fe19dffb700 1 -- 172.21.15.1:0/1068933416 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe198012750 con 0x7fe1a806d430 2024-03-18T19:02:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.414+0000 7fe1af957700 1 -- 172.21.15.1:0/1068933416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a8153370 con 0x7fe1a806d430 2024-03-18T19:02:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.414+0000 7fe19dffb700 1 -- 172.21.15.1:0/1068933416 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe198012b00 con 0x7fe1a806d430 2024-03-18T19:02:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.414+0000 7fe1af957700 1 -- 172.21.15.1:0/1068933416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a8157760 con 0x7fe1a806d430 2024-03-18T19:02:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.415+0000 7fe1af957700 1 -- 172.21.15.1:0/1068933416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a81539c0 con 0x7fe1a806d430 2024-03-18T19:02:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.416+0000 7fe19dffb700 1 -- 172.21.15.1:0/1068933416 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe198012c60 con 0x7fe1a806d430 2024-03-18T19:02:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.416+0000 7fe19dffb700 1 --2- 172.21.15.1:0/1068933416 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe18c06adb0 0x7fe18c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.416+0000 7fe19dffb700 1 -- 172.21.15.1:0/1068933416 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fe198089610 con 0x7fe1a806d430 2024-03-18T19:02:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.416+0000 7fe1acef2700 1 --2- 172.21.15.1:0/1068933416 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe18c06adb0 0x7fe18c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.416+0000 7fe1acef2700 1 --2- 172.21.15.1:0/1068933416 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe18c06adb0 0x7fe18c06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4009840 tx=0x7fe1a4008030 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:59.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.421+0000 7fe19dffb700 1 -- 172.21.15.1:0/1068933416 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe198054870 con 0x7fe1a806d430 2024-03-18T19:02:59.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.626+0000 7fe1af957700 1 -- 172.21.15.1:0/1068933416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe1a8072d70 con 0x7fe1a806d430 2024-03-18T19:02:59.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.626+0000 7fe19dffb700 1 -- 172.21.15.1:0/1068933416 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe198089070 con 0x7fe1a806d430 2024-03-18T19:02:59.622 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:02:59.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.628+0000 7fe1af957700 1 -- 172.21.15.1:0/1068933416 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe18c06adb0 msgr2=0x7fe18c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:59.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.628+0000 7fe1af957700 1 --2- 172.21.15.1:0/1068933416 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe18c06adb0 0x7fe18c06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4009840 tx=0x7fe1a4008030 comp rx=0 tx=0).stop 2024-03-18T19:02:59.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.628+0000 7fe1af957700 1 -- 172.21.15.1:0/1068933416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1a806d430 msgr2=0x7fe1a8149200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:59.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.628+0000 7fe1af957700 1 --2- 172.21.15.1:0/1068933416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1a806d430 0x7fe1a8149200 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe19800d800 tx=0x7fe19800db10 comp rx=0 tx=0).stop 2024-03-18T19:02:59.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.629+0000 7fe1af957700 1 -- 172.21.15.1:0/1068933416 shutdown_connections 2024-03-18T19:02:59.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.629+0000 7fe1af957700 1 --2- 172.21.15.1:0/1068933416 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe18c06adb0 0x7fe18c06d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.629+0000 7fe1af957700 1 --2- 172.21.15.1:0/1068933416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1a815d3b0 0x7fe1a81551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.629+0000 7fe1af957700 1 --2- 172.21.15.1:0/1068933416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1a806d430 0x7fe1a8149200 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.629+0000 7fe1af957700 1 --2- 172.21.15.1:0/1068933416 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1a80b2650 0x7fe1a8157220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.629+0000 7fe1af957700 1 -- 172.21.15.1:0/1068933416 >> 172.21.15.1:0/1068933416 conn(0x7fe1a80b1350 msgr2=0x7fe1a80b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:59.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.629+0000 7fe1af957700 1 -- 172.21.15.1:0/1068933416 shutdown_connections 2024-03-18T19:02:59.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.629+0000 7fe1af957700 1 -- 172.21.15.1:0/1068933416 wait complete. 2024-03-18T19:02:59.637 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T19:02:59.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.766+0000 7f64c53f8700 1 Processor -- start 2024-03-18T19:02:59.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.766+0000 7f64c53f8700 1 -- start start 2024-03-18T19:02:59.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.767+0000 7f64c53f8700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f64c0160940 0x7f64c00d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.767+0000 7f64c53f8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f64c00d1ea0 0x7f64c00b25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.767+0000 7f64c53f8700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c00cddc0 con 0x7f64c00cf090 2024-03-18T19:02:59.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.767+0000 7f64c53f8700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64c00cdff0 con 0x7f64c0160940 2024-03-18T19:02:59.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.767+0000 7f64c53f8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c00ce2d0 con 0x7f64c00d1ea0 2024-03-18T19:02:59.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.767+0000 7f64be7fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f64c00cf090 0x7f64c00d1790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38570/0 (socket says 172.21.15.1:38570) 2024-03-18T19:02:59.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.767+0000 7f64be7fc700 1 -- 172.21.15.1:0/1575941861 learned_addr learned my addr 172.21.15.1:0/1575941861 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:02:59.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.767+0000 7f64bf7fe700 1 --2- 172.21.15.1:0/1575941861 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f64c00d1ea0 0x7f64c00b25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.767+0000 7f64beffd700 1 --2- 172.21.15.1:0/1575941861 >> v2:172.21.15.165:3300/0 conn(0x7f64c0160940 0x7f64c00d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.767+0000 7f64bdffb700 1 -- 172.21.15.1:0/1575941861 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388260672 0 0) 0x7f64c00cddc0 con 0x7f64c00cf090 2024-03-18T19:02:59.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.767+0000 7f64bdffb700 1 -- 172.21.15.1:0/1575941861 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64ac0035b0 con 0x7f64c00cf090 2024-03-18T19:02:59.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.768+0000 7f64bf7fe700 1 -- 172.21.15.1:0/1575941861 >> v2:172.21.15.165:3300/0 conn(0x7f64c0160940 msgr2=0x7f64c00d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:02:59.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.768+0000 7f64bf7fe700 1 --2- 172.21.15.1:0/1575941861 >> v2:172.21.15.165:3300/0 conn(0x7f64c0160940 0x7f64c00d0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.768+0000 7f64bf7fe700 1 -- 172.21.15.1:0/1575941861 >> v1:172.21.15.1:6789/0 conn(0x7f64c00cf090 legacy=0x7f64c00d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:59.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.768+0000 7f64bf7fe700 1 -- 172.21.15.1:0/1575941861 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c00ce5b0 con 0x7f64c00d1ea0 2024-03-18T19:02:59.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.768+0000 7f64bf7fe700 1 --2- 172.21.15.1:0/1575941861 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f64c00d1ea0 0x7f64c00b25a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f64b8009d90 tx=0x7f64b800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=783ff80afe6d17c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:59.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.768+0000 7f64bdffb700 1 -- 172.21.15.1:0/1575941861 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b800e070 con 0x7f64c00d1ea0 2024-03-18T19:02:59.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.768+0000 7f64bdffb700 1 -- 172.21.15.1:0/1575941861 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b800bb20 con 0x7f64c00d1ea0 2024-03-18T19:02:59.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.768+0000 7f64bdffb700 1 -- 172.21.15.1:0/1575941861 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b800be90 con 0x7f64c00d1ea0 2024-03-18T19:02:59.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.769+0000 7f64c53f8700 1 -- 172.21.15.1:0/1575941861 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f64c00d1ea0 msgr2=0x7f64c00b25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:59.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.769+0000 7f64c53f8700 1 --2- 172.21.15.1:0/1575941861 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f64c00d1ea0 0x7f64c00b25a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f64b8009d90 tx=0x7f64b800b330 comp rx=0 tx=0).stop 2024-03-18T19:02:59.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.769+0000 7f64c53f8700 1 -- 172.21.15.1:0/1575941861 shutdown_connections 2024-03-18T19:02:59.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.769+0000 7f64c53f8700 1 --2- 172.21.15.1:0/1575941861 >> v2:172.21.15.165:3300/0 conn(0x7f64c0160940 0x7f64c00d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.769+0000 7f64c53f8700 1 --2- 172.21.15.1:0/1575941861 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f64c00d1ea0 0x7f64c00b25a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.769+0000 7f64c53f8700 1 -- 172.21.15.1:0/1575941861 >> 172.21.15.1:0/1575941861 conn(0x7f64c009c520 msgr2=0x7f64c00d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:59.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.769+0000 7f64c53f8700 1 -- 172.21.15.1:0/1575941861 shutdown_connections 2024-03-18T19:02:59.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.769+0000 7f64c53f8700 1 -- 172.21.15.1:0/1575941861 wait complete. 2024-03-18T19:02:59.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.770+0000 7f64c53f8700 1 Processor -- start 2024-03-18T19:02:59.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.770+0000 7f64c53f8700 1 -- start start 2024-03-18T19:02:59.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.771+0000 7f64c53f8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f64c00cf090 0x7f64c0154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.771+0000 7f64c53f8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64c00d1ea0 0x7f64c015e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.771+0000 7f64c53f8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64c0160940 0x7f64c00c9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.771+0000 7f64c53f8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c01596e0 con 0x7f64c0160940 2024-03-18T19:02:59.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.771+0000 7f64c53f8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c01f2d20 con 0x7f64c00d1ea0 2024-03-18T19:02:59.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.771+0000 7f64c53f8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c00c84a0 con 0x7f64c00cf090 2024-03-18T19:02:59.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.771+0000 7f64bf7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64c0160940 0x7f64c00c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.771+0000 7f64bf7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64c0160940 0x7f64c00c9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48672/0 (socket says 172.21.15.1:48672) 2024-03-18T19:02:59.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.771+0000 7f64bf7fe700 1 -- 172.21.15.1:0/600201094 learned_addr learned my addr 172.21.15.1:0/600201094 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:02:59.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.771+0000 7f64beffd700 1 --2- 172.21.15.1:0/600201094 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f64c00cf090 0x7f64c0154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.771+0000 7f64be7fc700 1 --2- 172.21.15.1:0/600201094 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64c00d1ea0 0x7f64c015e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.771+0000 7f64bf7fe700 1 -- 172.21.15.1:0/600201094 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f64c00cf090 msgr2=0x7f64c0154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:59.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.771+0000 7f64bf7fe700 1 --2- 172.21.15.1:0/600201094 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f64c00cf090 0x7f64c0154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.771+0000 7f64bf7fe700 1 -- 172.21.15.1:0/600201094 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64c00d1ea0 msgr2=0x7f64c015e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:59.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.771+0000 7f64bf7fe700 1 --2- 172.21.15.1:0/600201094 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64c00d1ea0 0x7f64c015e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.771+0000 7f64bf7fe700 1 -- 172.21.15.1:0/600201094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b8009a70 con 0x7f64c0160940 2024-03-18T19:02:59.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.772+0000 7f64bf7fe700 1 --2- 172.21.15.1:0/600201094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64c0160940 0x7f64c00c9910 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f64b8014040 tx=0x7f64b8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:59.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.772+0000 7f64b77fe700 1 -- 172.21.15.1:0/600201094 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b8007000 con 0x7f64c0160940 2024-03-18T19:02:59.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.772+0000 7f64b77fe700 1 -- 172.21.15.1:0/600201094 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b8012650 con 0x7f64c0160940 2024-03-18T19:02:59.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.772+0000 7f64c53f8700 1 -- 172.21.15.1:0/600201094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c00c9e50 con 0x7f64c0160940 2024-03-18T19:02:59.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.772+0000 7f64b77fe700 1 -- 172.21.15.1:0/600201094 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b8012a00 con 0x7f64c0160940 2024-03-18T19:02:59.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.772+0000 7f64c53f8700 1 -- 172.21.15.1:0/600201094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c015f270 con 0x7f64c0160940 2024-03-18T19:02:59.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.773+0000 7f64c53f8700 1 -- 172.21.15.1:0/600201094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c0073350 con 0x7f64c0160940 2024-03-18T19:02:59.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.773+0000 7f64b77fe700 1 -- 172.21.15.1:0/600201094 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f64b8012b60 con 0x7f64c0160940 2024-03-18T19:02:59.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.774+0000 7f64b77fe700 1 --2- 172.21.15.1:0/600201094 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f64a006adb0 0x7f64a006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:02:59.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.774+0000 7f64beffd700 1 --2- 172.21.15.1:0/600201094 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f64a006adb0 0x7f64a006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:02:59.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.774+0000 7f64b77fe700 1 -- 172.21.15.1:0/600201094 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f64b808b3f0 con 0x7f64c0160940 2024-03-18T19:02:59.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.774+0000 7f64beffd700 1 --2- 172.21.15.1:0/600201094 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f64a006adb0 0x7f64a006d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f64b0009b40 tx=0x7f64b0009210 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:02:59.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.778+0000 7f64b77fe700 1 -- 172.21.15.1:0/600201094 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f64b8056770 con 0x7f64c0160940 2024-03-18T19:02:59.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.980+0000 7f64c53f8700 1 -- 172.21.15.1:0/600201094 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f64c0159820 con 0x7f64c0160940 2024-03-18T19:02:59.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.980+0000 7f64b77fe700 1 -- 172.21.15.1:0/600201094 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v67) v1 ==== 84+0+65 (secure 0 0 0) 0x7f64b8059d90 con 0x7f64c0160940 2024-03-18T19:02:59.977 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:02:59.977 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T19:02:59.977 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:02:59.977 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-18T19:02:59.977 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-18T19:02:59.977 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:02:59.977 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T19:02:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.982+0000 7f64c53f8700 1 -- 172.21.15.1:0/600201094 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f64a006adb0 msgr2=0x7f64a006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.982+0000 7f64c53f8700 1 --2- 172.21.15.1:0/600201094 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f64a006adb0 0x7f64a006d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f64b0009b40 tx=0x7f64b0009210 comp rx=0 tx=0).stop 2024-03-18T19:02:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.982+0000 7f64c53f8700 1 -- 172.21.15.1:0/600201094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64c0160940 msgr2=0x7f64c00c9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:02:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.982+0000 7f64c53f8700 1 --2- 172.21.15.1:0/600201094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64c0160940 0x7f64c00c9910 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f64b8014040 tx=0x7f64b8004ad0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.983+0000 7f64c53f8700 1 -- 172.21.15.1:0/600201094 shutdown_connections 2024-03-18T19:02:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.983+0000 7f64c53f8700 1 --2- 172.21.15.1:0/600201094 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f64a006adb0 0x7f64a006d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.983+0000 7f64c53f8700 1 --2- 172.21.15.1:0/600201094 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f64c00cf090 0x7f64c0154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.983+0000 7f64c53f8700 1 --2- 172.21.15.1:0/600201094 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64c0160940 0x7f64c00c9910 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.983+0000 7f64c53f8700 1 --2- 172.21.15.1:0/600201094 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64c00d1ea0 0x7f64c015e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:02:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.983+0000 7f64c53f8700 1 -- 172.21.15.1:0/600201094 >> 172.21.15.1:0/600201094 conn(0x7f64c009c520 msgr2=0x7f64c00cf4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:02:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.983+0000 7f64c53f8700 1 -- 172.21.15.1:0/600201094 shutdown_connections 2024-03-18T19:02:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:02:59.983+0000 7f64c53f8700 1 -- 172.21.15.1:0/600201094 wait complete. 2024-03-18T19:02:59.991 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T19:02:59.991 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T19:03:00.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.124+0000 7f32fc876700 1 Processor -- start 2024-03-18T19:03:00.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.124+0000 7f32fc876700 1 -- start start 2024-03-18T19:03:00.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.125+0000 7f32fc876700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f806f7d0 0x7f32f815dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:00.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.125+0000 7f32fc876700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f80b41a0 con 0x7f32f806fc50 2024-03-18T19:03:00.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.125+0000 7f32fc876700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f80b4d10 con 0x7f32f815a160 2024-03-18T19:03:00.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.125+0000 7f32fc876700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f815ec10 con 0x7f32f806f7d0 2024-03-18T19:03:00.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f6d9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f32f806fc50 0x7f32f80717e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38574/0 (socket says 172.21.15.1:38574) 2024-03-18T19:03:00.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f6d9d700 1 -- 172.21.15.1:0/3796205699 learned_addr learned my addr 172.21.15.1:0/3796205699 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:03:00.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f5d9b700 1 --2- 172.21.15.1:0/3796205699 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f806f7d0 0x7f32f815dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:00.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f559a700 1 -- 172.21.15.1:0/3796205699 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766092730 0 0) 0x7f32f80b41a0 con 0x7f32f806fc50 2024-03-18T19:03:00.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f559a700 1 -- 172.21.15.1:0/3796205699 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e00035b0 con 0x7f32f806fc50 2024-03-18T19:03:00.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f559a700 1 -- 172.21.15.1:0/3796205699 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4206638404 0 0) 0x7f32f80b4d10 con 0x7f32f815a160 2024-03-18T19:03:00.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f559a700 1 -- 172.21.15.1:0/3796205699 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32f80b41a0 con 0x7f32f815a160 2024-03-18T19:03:00.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f559a700 1 -- 172.21.15.1:0/3796205699 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4091662106 0 0) 0x7f32e00035b0 con 0x7f32f806fc50 2024-03-18T19:03:00.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f559a700 1 -- 172.21.15.1:0/3796205699 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f806f7d0 msgr2=0x7f32f815dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:00.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f559a700 1 --2- 172.21.15.1:0/3796205699 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f806f7d0 0x7f32f815dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:00.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f559a700 1 -- 172.21.15.1:0/3796205699 >> v1:172.21.15.165:6789/0 conn(0x7f32f815a160 legacy=0x7f32f80b3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:00.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f559a700 1 -- 172.21.15.1:0/3796205699 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f80b4a30 con 0x7f32f806fc50 2024-03-18T19:03:00.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f559a700 1 -- 172.21.15.1:0/3796205699 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f32f00019f0 con 0x7f32f806fc50 2024-03-18T19:03:00.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f5d9b700 1 --2- 172.21.15.1:0/3796205699 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f806f7d0 0x7f32f815dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:03:00.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f559a700 1 -- 172.21.15.1:0/3796205699 >> v1:172.21.15.1:6789/0 conn(0x7f32f806fc50 legacy=0x7f32f80717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:00.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f559a700 1 --2- 172.21.15.1:0/3796205699 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f806f7d0 0x7f32e0003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:00.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f559a700 1 --2- 172.21.15.1:0/3796205699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e0004150 0x7f32e0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:00.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f559a700 1 --2- 172.21.15.1:0/3796205699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32e0004d90 0x7f32e0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:00.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f559a700 1 -- 172.21.15.1:0/3796205699 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f815ec10 con 0x7f32e0004150 2024-03-18T19:03:00.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f559a700 1 -- 172.21.15.1:0/3796205699 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e000dd30 con 0x7f32e0004d90 2024-03-18T19:03:00.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f559a700 1 -- 172.21.15.1:0/3796205699 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e000def0 con 0x7f32f806f7d0 2024-03-18T19:03:00.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f5d9b700 1 --2- 172.21.15.1:0/3796205699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e0004150 0x7f32e0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:00.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f659c700 1 --2- 172.21.15.1:0/3796205699 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f806f7d0 0x7f32e0003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:00.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.126+0000 7f32f6d9d700 1 --2- 172.21.15.1:0/3796205699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32e0004d90 0x7f32e0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:00.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.127+0000 7f32f5d9b700 1 -- 172.21.15.1:0/3796205699 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f806f7d0 msgr2=0x7f32e0003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:00.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.127+0000 7f32f5d9b700 1 --2- 172.21.15.1:0/3796205699 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f806f7d0 0x7f32e0003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:00.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.127+0000 7f32f5d9b700 1 -- 172.21.15.1:0/3796205699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32e0004d90 msgr2=0x7f32e0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:00.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.127+0000 7f32f5d9b700 1 --2- 172.21.15.1:0/3796205699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32e0004d90 0x7f32e0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:00.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.127+0000 7f32f5d9b700 1 -- 172.21.15.1:0/3796205699 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e80094a0 con 0x7f32e0004150 2024-03-18T19:03:00.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.127+0000 7f32f5d9b700 1 --2- 172.21.15.1:0/3796205699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e0004150 0x7f32e0004850 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009ce0 tx=0x7f32e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dca3edf9aa32cf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:00.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.127+0000 7f32f559a700 1 -- 172.21.15.1:0/3796205699 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e8004030 con 0x7f32e0004150 2024-03-18T19:03:00.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.127+0000 7f32f559a700 1 -- 172.21.15.1:0/3796205699 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e8004190 con 0x7f32e0004150 2024-03-18T19:03:00.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.127+0000 7f32f559a700 1 -- 172.21.15.1:0/3796205699 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e8004540 con 0x7f32e0004150 2024-03-18T19:03:00.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.127+0000 7f32fc876700 1 -- 172.21.15.1:0/3796205699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e0004150 msgr2=0x7f32e0004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:00.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.127+0000 7f32fc876700 1 --2- 172.21.15.1:0/3796205699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e0004150 0x7f32e0004850 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009ce0 tx=0x7f32e0005880 comp rx=0 tx=0).stop 2024-03-18T19:03:00.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.127+0000 7f32f6d9d700 1 -- 172.21.15.1:0/3796205699 reap_dead start 2024-03-18T19:03:00.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.128+0000 7f32fc876700 1 -- 172.21.15.1:0/3796205699 shutdown_connections 2024-03-18T19:03:00.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.128+0000 7f32fc876700 1 -- 172.21.15.1:0/3796205699 >> 172.21.15.1:0/3796205699 conn(0x7f32f80d00c0 msgr2=0x7f32f80d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:00.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.128+0000 7f32fc876700 1 -- 172.21.15.1:0/3796205699 shutdown_connections 2024-03-18T19:03:00.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.128+0000 7f32fc876700 1 -- 172.21.15.1:0/3796205699 wait complete. 2024-03-18T19:03:00.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.129+0000 7f32fc876700 1 Processor -- start 2024-03-18T19:03:00.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.129+0000 7f32fc876700 1 -- start start 2024-03-18T19:03:00.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32fc876700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e0004150 0x7f32f81f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:00.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32fc876700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32f8149cd0 0x7f32f8153980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:00.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32fc876700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f8153ec0 0x7f32f814be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:00.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32fc876700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f8154350 con 0x7f32e0004150 2024-03-18T19:03:00.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32fc876700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f81f4130 con 0x7f32f8149cd0 2024-03-18T19:03:00.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32fc876700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80cf540 con 0x7f32f8153ec0 2024-03-18T19:03:00.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32f659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e0004150 0x7f32f81f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:00.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32f659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e0004150 0x7f32f81f3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48694/0 (socket says 172.21.15.1:48694) 2024-03-18T19:03:00.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32f659c700 1 -- 172.21.15.1:0/3874797328 learned_addr learned my addr 172.21.15.1:0/3874797328 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:00.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32f659c700 1 -- 172.21.15.1:0/3874797328 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f8153ec0 msgr2=0x7f32f814be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:00.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32f659c700 1 --2- 172.21.15.1:0/3874797328 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f8153ec0 0x7f32f814be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:00.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32f659c700 1 -- 172.21.15.1:0/3874797328 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32f8149cd0 msgr2=0x7f32f8153980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:00.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32f5d9b700 1 --2- 172.21.15.1:0/3874797328 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32f8149cd0 0x7f32f8153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:00.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32f659c700 1 --2- 172.21.15.1:0/3874797328 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32f8149cd0 0x7f32f8153980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:00.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32f659c700 1 -- 172.21.15.1:0/3874797328 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f80cded0 con 0x7f32e0004150 2024-03-18T19:03:00.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32f5d9b700 1 --2- 172.21.15.1:0/3874797328 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32f8149cd0 0x7f32f8153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:03:00.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32f659c700 1 --2- 172.21.15.1:0/3874797328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e0004150 0x7f32f81f3bf0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00f830 tx=0x7f32ec00fbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:00.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32e6ffd700 1 -- 172.21.15.1:0/3874797328 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ec005320 con 0x7f32e0004150 2024-03-18T19:03:00.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32e6ffd700 1 -- 172.21.15.1:0/3874797328 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32ec005480 con 0x7f32e0004150 2024-03-18T19:03:00.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32e6ffd700 1 -- 172.21.15.1:0/3874797328 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ec006aa0 con 0x7f32e0004150 2024-03-18T19:03:00.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.130+0000 7f32fc876700 1 -- 172.21.15.1:0/3874797328 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f80ce1b0 con 0x7f32e0004150 2024-03-18T19:03:00.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.131+0000 7f32fc876700 1 -- 172.21.15.1:0/3874797328 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f80ce770 con 0x7f32e0004150 2024-03-18T19:03:00.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.132+0000 7f32fc876700 1 -- 172.21.15.1:0/3874797328 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f80676a0 con 0x7f32e0004150 2024-03-18T19:03:00.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.132+0000 7f32e6ffd700 1 -- 172.21.15.1:0/3874797328 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f32ec006c00 con 0x7f32e0004150 2024-03-18T19:03:00.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.132+0000 7f32e6ffd700 1 --2- 172.21.15.1:0/3874797328 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f32d806adb0 0x7f32d806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:00.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.132+0000 7f32e6ffd700 1 -- 172.21.15.1:0/3874797328 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f32ec0888a0 con 0x7f32e0004150 2024-03-18T19:03:00.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.132+0000 7f32f5d9b700 1 --2- 172.21.15.1:0/3874797328 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f32d806adb0 0x7f32d806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:00.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.133+0000 7f32f5d9b700 1 --2- 172.21.15.1:0/3874797328 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f32d806adb0 0x7f32d806d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f32f8066b10 tx=0x7f32e8008030 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:00.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.137+0000 7f32e6ffd700 1 -- 172.21.15.1:0/3874797328 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f32ec053b80 con 0x7f32e0004150 2024-03-18T19:03:00.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:00.340+0000 7f32fc876700 1 -- 172.21.15.1:0/3874797328 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f32f8072a90 con 0x7f32e0004150 2024-03-18T19:03:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.100+0000 7f32e6ffd700 1 -- 172.21.15.1:0/3874797328 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v68) v1 ==== 138+0+0 (secure 0 0 0) 0x7f32ec01a0b0 con 0x7f32e0004150 2024-03-18T19:03:01.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.104+0000 7f32fc876700 1 -- 172.21.15.1:0/3874797328 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f32d806adb0 msgr2=0x7f32d806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:01.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.104+0000 7f32fc876700 1 --2- 172.21.15.1:0/3874797328 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f32d806adb0 0x7f32d806d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f32f8066b10 tx=0x7f32e8008030 comp rx=0 tx=0).stop 2024-03-18T19:03:01.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.104+0000 7f32fc876700 1 -- 172.21.15.1:0/3874797328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e0004150 msgr2=0x7f32f81f3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:01.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.104+0000 7f32fc876700 1 --2- 172.21.15.1:0/3874797328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e0004150 0x7f32f81f3bf0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00f830 tx=0x7f32ec00fbf0 comp rx=0 tx=0).stop 2024-03-18T19:03:01.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.106+0000 7f32fc876700 1 -- 172.21.15.1:0/3874797328 shutdown_connections 2024-03-18T19:03:01.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.106+0000 7f32fc876700 1 --2- 172.21.15.1:0/3874797328 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f32d806adb0 0x7f32d806d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:01.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.106+0000 7f32fc876700 1 --2- 172.21.15.1:0/3874797328 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32f8153ec0 0x7f32f814be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:01.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.106+0000 7f32fc876700 1 --2- 172.21.15.1:0/3874797328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32e0004150 0x7f32f81f3bf0 secure :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00f830 tx=0x7f32ec00fbf0 comp rx=0 tx=0).stop 2024-03-18T19:03:01.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.106+0000 7f32fc876700 1 --2- 172.21.15.1:0/3874797328 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32f8149cd0 0x7f32f8153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:01.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.107+0000 7f32fc876700 1 -- 172.21.15.1:0/3874797328 >> 172.21.15.1:0/3874797328 conn(0x7f32f80d00c0 msgr2=0x7f32f80717e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:01.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.108+0000 7f32fc876700 1 -- 172.21.15.1:0/3874797328 shutdown_connections 2024-03-18T19:03:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.108+0000 7f32fc876700 1 -- 172.21.15.1:0/3874797328 wait complete. 2024-03-18T19:03:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-18T19:03:01.119 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T19:03:01.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.263+0000 7f2ee083b700 1 Processor -- start 2024-03-18T19:03:01.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.263+0000 7f2ee083b700 1 -- start start 2024-03-18T19:03:01.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.264+0000 7f2ee083b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2edc15eba0 0x7f2edc0cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:01.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.264+0000 7f2ee083b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2edc0b0500 0x7f2edc0cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:01.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.264+0000 7f2ee083b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2edc15ef80 con 0x7f2edc0cfdf0 2024-03-18T19:03:01.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.264+0000 7f2ee083b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2edc0cc9a0 con 0x7f2edc0b0500 2024-03-18T19:03:01.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.264+0000 7f2ee083b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2edc0ccc80 con 0x7f2edc15eba0 2024-03-18T19:03:01.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.264+0000 7f2ed9d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2edc0b0500 0x7f2edc0cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:01.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.264+0000 7f2ed9d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2edc0b0500 0x7f2edc0cf8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48710/0 (socket says 172.21.15.1:48710) 2024-03-18T19:03:01.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.264+0000 7f2ed9d9b700 1 -- 172.21.15.1:0/3880570874 learned_addr learned my addr 172.21.15.1:0/3880570874 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:01.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.264+0000 7f2eda59c700 1 --2- 172.21.15.1:0/3880570874 >> v2:172.21.15.165:3300/0 conn(0x7f2edc15eba0 0x7f2edc0cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:01.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.264+0000 7f2ed9d9b700 1 -- 172.21.15.1:0/3880570874 >> v2:172.21.15.165:3300/0 conn(0x7f2edc15eba0 msgr2=0x7f2edc0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:01.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.264+0000 7f2ed9d9b700 1 --2- 172.21.15.1:0/3880570874 >> v2:172.21.15.165:3300/0 conn(0x7f2edc15eba0 0x7f2edc0cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:01.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.264+0000 7f2ed9d9b700 1 -- 172.21.15.1:0/3880570874 >> v1:172.21.15.165:6789/0 conn(0x7f2edc0cfdf0 legacy=0x7f2edc099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:01.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.264+0000 7f2ed9d9b700 1 -- 172.21.15.1:0/3880570874 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc0ccf60 con 0x7f2edc0b0500 2024-03-18T19:03:01.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.264+0000 7f2eda59c700 1 --2- 172.21.15.1:0/3880570874 >> v2:172.21.15.165:3300/0 conn(0x7f2edc15eba0 0x7f2edc0cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:03:01.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.265+0000 7f2ed9d9b700 1 --2- 172.21.15.1:0/3880570874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2edc0b0500 0x7f2edc0cf8b0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2edc072fb0 tx=0x7f2ed0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97905e84b01bc875 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:01.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.265+0000 7f2ed959a700 1 -- 172.21.15.1:0/3880570874 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed000b870 con 0x7f2edc0b0500 2024-03-18T19:03:01.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.265+0000 7f2ed959a700 1 -- 172.21.15.1:0/3880570874 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed000b9d0 con 0x7f2edc0b0500 2024-03-18T19:03:01.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.265+0000 7f2ed959a700 1 -- 172.21.15.1:0/3880570874 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed000bd60 con 0x7f2edc0b0500 2024-03-18T19:03:01.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.265+0000 7f2ee083b700 1 -- 172.21.15.1:0/3880570874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2edc0b0500 msgr2=0x7f2edc0cf8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:01.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.265+0000 7f2ee083b700 1 --2- 172.21.15.1:0/3880570874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2edc0b0500 0x7f2edc0cf8b0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2edc072fb0 tx=0x7f2ed0009bd0 comp rx=0 tx=0).stop 2024-03-18T19:03:01.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.265+0000 7f2ee083b700 1 -- 172.21.15.1:0/3880570874 shutdown_connections 2024-03-18T19:03:01.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.265+0000 7f2ee083b700 1 --2- 172.21.15.1:0/3880570874 >> v2:172.21.15.165:3300/0 conn(0x7f2edc15eba0 0x7f2edc0cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:01.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.265+0000 7f2ee083b700 1 --2- 172.21.15.1:0/3880570874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2edc0b0500 0x7f2edc0cf8b0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:01.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.265+0000 7f2ee083b700 1 -- 172.21.15.1:0/3880570874 >> 172.21.15.1:0/3880570874 conn(0x7f2edc0b2bd0 msgr2=0x7f2edc0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:01.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.265+0000 7f2ee083b700 1 -- 172.21.15.1:0/3880570874 shutdown_connections 2024-03-18T19:03:01.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.265+0000 7f2ee083b700 1 -- 172.21.15.1:0/3880570874 wait complete. 2024-03-18T19:03:01.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.267+0000 7f2ee083b700 1 Processor -- start 2024-03-18T19:03:01.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.267+0000 7f2ee083b700 1 -- start start 2024-03-18T19:03:01.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ee083b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2edc0b0500 0x7f2edc0c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:01.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ee083b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2edc0cfdf0 0x7f2edc148580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:01.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ee083b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2edc15eba0 0x7f2edc0ca300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:01.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ee083b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc148ac0 con 0x7f2edc0cfdf0 2024-03-18T19:03:01.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ee083b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc0ca840 con 0x7f2edc0b0500 2024-03-18T19:03:01.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ee083b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc0c9e70 con 0x7f2edc15eba0 2024-03-18T19:03:01.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ed9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2edc0cfdf0 0x7f2edc148580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:01.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ed9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2edc0cfdf0 0x7f2edc148580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48724/0 (socket says 172.21.15.1:48724) 2024-03-18T19:03:01.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ed9d9b700 1 -- 172.21.15.1:0/2229562725 learned_addr learned my addr 172.21.15.1:0/2229562725 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:01.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ed9d9b700 1 -- 172.21.15.1:0/2229562725 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2edc15eba0 msgr2=0x7f2edc0ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:01.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ed9d9b700 1 --2- 172.21.15.1:0/2229562725 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2edc15eba0 0x7f2edc0ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:01.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ed9d9b700 1 -- 172.21.15.1:0/2229562725 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2edc0b0500 msgr2=0x7f2edc0c63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:01.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ed9d9b700 1 --2- 172.21.15.1:0/2229562725 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2edc0b0500 0x7f2edc0c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:01.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ed9d9b700 1 -- 172.21.15.1:0/2229562725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed0009550 con 0x7f2edc0cfdf0 2024-03-18T19:03:01.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ed9d9b700 1 --2- 172.21.15.1:0/2229562725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2edc0cfdf0 0x7f2edc148580 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0014040 tx=0x7f2ed0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:01.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ecaffd700 1 -- 172.21.15.1:0/2229562725 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0004520 con 0x7f2edc0cfdf0 2024-03-18T19:03:01.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ecaffd700 1 -- 172.21.15.1:0/2229562725 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed001e070 con 0x7f2edc0cfdf0 2024-03-18T19:03:01.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ee083b700 1 -- 172.21.15.1:0/2229562725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2edc0c92a0 con 0x7f2edc0cfdf0 2024-03-18T19:03:01.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ecaffd700 1 -- 172.21.15.1:0/2229562725 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed00038a0 con 0x7f2edc0cfdf0 2024-03-18T19:03:01.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.268+0000 7f2ee083b700 1 -- 172.21.15.1:0/2229562725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2edc0c9820 con 0x7f2edc0cfdf0 2024-03-18T19:03:01.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.269+0000 7f2ee083b700 1 -- 172.21.15.1:0/2229562725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2edc0ca150 con 0x7f2edc0cfdf0 2024-03-18T19:03:01.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.270+0000 7f2ecaffd700 1 -- 172.21.15.1:0/2229562725 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2ed0003a00 con 0x7f2edc0cfdf0 2024-03-18T19:03:01.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.270+0000 7f2ecaffd700 1 --2- 172.21.15.1:0/2229562725 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2ebc06adb0 0x7f2ebc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:01.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.270+0000 7f2ecaffd700 1 -- 172.21.15.1:0/2229562725 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f2ed0020070 con 0x7f2edc0cfdf0 2024-03-18T19:03:01.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.270+0000 7f2eda59c700 1 --2- 172.21.15.1:0/2229562725 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2ebc06adb0 0x7f2ebc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:01.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.271+0000 7f2eda59c700 1 --2- 172.21.15.1:0/2229562725 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2ebc06adb0 0x7f2ebc06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc006fd0 tx=0x7f2ecc008040 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:01.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.275+0000 7f2ecaffd700 1 -- 172.21.15.1:0/2229562725 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2ed0055b00 con 0x7f2edc0cfdf0 2024-03-18T19:03:01.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:01.481+0000 7f2ee083b700 1 -- 172.21.15.1:0/2229562725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2edc0c9fb0 con 0x7f2edc0cfdf0 2024-03-18T19:03:02.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.102+0000 7f2ecaffd700 1 -- 172.21.15.1:0/2229562725 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v69) v1 ==== 170+0+0 (secure 0 0 0) 0x7f2edc0c9fb0 con 0x7f2edc0cfdf0 2024-03-18T19:03:02.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.107+0000 7f2ee083b700 1 -- 172.21.15.1:0/2229562725 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2ebc06adb0 msgr2=0x7f2ebc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:02.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.107+0000 7f2ee083b700 1 --2- 172.21.15.1:0/2229562725 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2ebc06adb0 0x7f2ebc06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc006fd0 tx=0x7f2ecc008040 comp rx=0 tx=0).stop 2024-03-18T19:03:02.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.107+0000 7f2ee083b700 1 -- 172.21.15.1:0/2229562725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2edc0cfdf0 msgr2=0x7f2edc148580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:02.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.107+0000 7f2ee083b700 1 --2- 172.21.15.1:0/2229562725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2edc0cfdf0 0x7f2edc148580 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0014040 tx=0x7f2ed0004750 comp rx=0 tx=0).stop 2024-03-18T19:03:02.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.111+0000 7f2ee083b700 1 -- 172.21.15.1:0/2229562725 shutdown_connections 2024-03-18T19:03:02.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.111+0000 7f2ee083b700 1 --2- 172.21.15.1:0/2229562725 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2ebc06adb0 0x7f2ebc06d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.111+0000 7f2ee083b700 1 --2- 172.21.15.1:0/2229562725 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2edc15eba0 0x7f2edc0ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.111+0000 7f2ee083b700 1 --2- 172.21.15.1:0/2229562725 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2edc0b0500 0x7f2edc0c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.111+0000 7f2ee083b700 1 --2- 172.21.15.1:0/2229562725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2edc0cfdf0 0x7f2edc148580 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.111+0000 7f2ee083b700 1 -- 172.21.15.1:0/2229562725 >> 172.21.15.1:0/2229562725 conn(0x7f2edc0b2bd0 msgr2=0x7f2edc15c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:02.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.112+0000 7f2ee083b700 1 -- 172.21.15.1:0/2229562725 shutdown_connections 2024-03-18T19:03:02.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.114+0000 7f2ee083b700 1 -- 172.21.15.1:0/2229562725 wait complete. 2024-03-18T19:03:02.115 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-18T19:03:02.135 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T19:03:02.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.272+0000 7f7687ce4700 1 Processor -- start 2024-03-18T19:03:02.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.272+0000 7f7687ce4700 1 -- start start 2024-03-18T19:03:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.273+0000 7f7687ce4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7680099a40 0x7f76800ade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.273+0000 7f7687ce4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f768006fa50 0x7f768006fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.273+0000 7f7687ce4700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76800b3aa0 con 0x7f7680099e20 2024-03-18T19:03:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.273+0000 7f7687ce4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f768006e9b0 con 0x7f7680099a40 2024-03-18T19:03:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.273+0000 7f7687ce4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f768006ec90 con 0x7f768006fa50 2024-03-18T19:03:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.273+0000 7f7685a80700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7680099a40 0x7f76800ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.273+0000 7f7685a80700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7680099a40 0x7f76800ade00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48732/0 (socket says 172.21.15.1:48732) 2024-03-18T19:03:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.273+0000 7f7685a80700 1 -- 172.21.15.1:0/4148527742 learned_addr learned my addr 172.21.15.1:0/4148527742 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.274+0000 7f7686281700 1 --2- 172.21.15.1:0/4148527742 >> v2:172.21.15.165:3300/0 conn(0x7f768006fa50 0x7f768006fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.274+0000 7f7685a80700 1 -- 172.21.15.1:0/4148527742 >> v2:172.21.15.165:3300/0 conn(0x7f768006fa50 msgr2=0x7f768006fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:02.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.274+0000 7f7685a80700 1 --2- 172.21.15.1:0/4148527742 >> v2:172.21.15.165:3300/0 conn(0x7f768006fa50 0x7f768006fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.274+0000 7f7685a80700 1 -- 172.21.15.1:0/4148527742 >> v1:172.21.15.165:6789/0 conn(0x7f7680099e20 legacy=0x7f76800cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:02.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.274+0000 7f7685a80700 1 -- 172.21.15.1:0/4148527742 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768006ef70 con 0x7f7680099a40 2024-03-18T19:03:02.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.274+0000 7f7686281700 1 --2- 172.21.15.1:0/4148527742 >> v2:172.21.15.165:3300/0 conn(0x7f768006fa50 0x7f768006fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:03:02.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.274+0000 7f7685a80700 1 --2- 172.21.15.1:0/4148527742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7680099a40 0x7f76800ade00 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f7680067140 tx=0x7f7670009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4975f0d29654a64c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:02.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.274+0000 7f7684a7e700 1 -- 172.21.15.1:0/4148527742 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7670004230 con 0x7f7680099a40 2024-03-18T19:03:02.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.274+0000 7f7684a7e700 1 -- 172.21.15.1:0/4148527742 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7670004390 con 0x7f7680099a40 2024-03-18T19:03:02.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.274+0000 7f7684a7e700 1 -- 172.21.15.1:0/4148527742 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76700036a0 con 0x7f7680099a40 2024-03-18T19:03:02.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.275+0000 7f7687ce4700 1 -- 172.21.15.1:0/4148527742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7680099a40 msgr2=0x7f76800ade00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:02.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.275+0000 7f7687ce4700 1 --2- 172.21.15.1:0/4148527742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7680099a40 0x7f76800ade00 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f7680067140 tx=0x7f7670009bd0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.275+0000 7f7687ce4700 1 -- 172.21.15.1:0/4148527742 shutdown_connections 2024-03-18T19:03:02.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.275+0000 7f7687ce4700 1 --2- 172.21.15.1:0/4148527742 >> v2:172.21.15.165:3300/0 conn(0x7f768006fa50 0x7f768006fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.275+0000 7f7687ce4700 1 --2- 172.21.15.1:0/4148527742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7680099a40 0x7f76800ade00 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.275+0000 7f7687ce4700 1 -- 172.21.15.1:0/4148527742 >> 172.21.15.1:0/4148527742 conn(0x7f76800b2c50 msgr2=0x7f76800b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:02.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.275+0000 7f7687ce4700 1 -- 172.21.15.1:0/4148527742 shutdown_connections 2024-03-18T19:03:02.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.275+0000 7f7687ce4700 1 -- 172.21.15.1:0/4148527742 wait complete. 2024-03-18T19:03:02.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.276+0000 7f7687ce4700 1 Processor -- start 2024-03-18T19:03:02.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.276+0000 7f7687ce4700 1 -- start start 2024-03-18T19:03:02.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.277+0000 7f7687ce4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f768006fa50 0x7f76801e3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:02.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.277+0000 7f7687ce4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7680099a40 0x7f76801f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:02.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.277+0000 7f7687ce4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7680099e20 0x7f76801e9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:02.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.277+0000 7f7687ce4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76801f4020 con 0x7f7680099a40 2024-03-18T19:03:02.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.277+0000 7f7687ce4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76801f3eb0 con 0x7f768006fa50 2024-03-18T19:03:02.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.277+0000 7f7687ce4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76801e4de0 con 0x7f7680099e20 2024-03-18T19:03:02.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.277+0000 7f768527f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7680099a40 0x7f76801f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:02.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.277+0000 7f768527f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7680099a40 0x7f76801f3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48744/0 (socket says 172.21.15.1:48744) 2024-03-18T19:03:02.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.277+0000 7f768527f700 1 -- 172.21.15.1:0/3902590113 learned_addr learned my addr 172.21.15.1:0/3902590113 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:02.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.277+0000 7f7685a80700 1 --2- 172.21.15.1:0/3902590113 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f768006fa50 0x7f76801e3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:02.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.277+0000 7f768527f700 1 -- 172.21.15.1:0/3902590113 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7680099e20 msgr2=0x7f76801e9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:02.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.277+0000 7f768527f700 1 --2- 172.21.15.1:0/3902590113 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7680099e20 0x7f76801e9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.277+0000 7f768527f700 1 -- 172.21.15.1:0/3902590113 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f768006fa50 msgr2=0x7f76801e3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:02.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.277+0000 7f768527f700 1 --2- 172.21.15.1:0/3902590113 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f768006fa50 0x7f76801e3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.277+0000 7f768527f700 1 -- 172.21.15.1:0/3902590113 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7670009550 con 0x7f7680099a40 2024-03-18T19:03:02.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.277+0000 7f768527f700 1 --2- 172.21.15.1:0/3902590113 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7680099a40 0x7f76801f3970 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f767c00c4d0 tx=0x7f767c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:02.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.277+0000 7f76767fc700 1 -- 172.21.15.1:0/3902590113 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c011790 con 0x7f7680099a40 2024-03-18T19:03:02.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.277+0000 7f76767fc700 1 -- 172.21.15.1:0/3902590113 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c0118f0 con 0x7f7680099a40 2024-03-18T19:03:02.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.277+0000 7f7687ce4700 1 -- 172.21.15.1:0/3902590113 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76801e4260 con 0x7f7680099a40 2024-03-18T19:03:02.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.278+0000 7f76767fc700 1 -- 172.21.15.1:0/3902590113 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c011ca0 con 0x7f7680099a40 2024-03-18T19:03:02.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.278+0000 7f7687ce4700 1 -- 172.21.15.1:0/3902590113 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76801e4820 con 0x7f7680099a40 2024-03-18T19:03:02.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.278+0000 7f7687ce4700 1 -- 172.21.15.1:0/3902590113 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76801f4160 con 0x7f7680099a40 2024-03-18T19:03:02.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.279+0000 7f76767fc700 1 -- 172.21.15.1:0/3902590113 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f767c006860 con 0x7f7680099a40 2024-03-18T19:03:02.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.280+0000 7f76767fc700 1 --2- 172.21.15.1:0/3902590113 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f766406adb0 0x7f766406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:02.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.280+0000 7f76767fc700 1 -- 172.21.15.1:0/3902590113 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f767c089840 con 0x7f7680099a40 2024-03-18T19:03:02.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.280+0000 7f7685a80700 1 --2- 172.21.15.1:0/3902590113 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f766406adb0 0x7f766406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:02.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.280+0000 7f7685a80700 1 --2- 172.21.15.1:0/3902590113 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f766406adb0 0x7f766406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7670000c00 tx=0x7f7670008030 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:02.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.284+0000 7f76767fc700 1 -- 172.21.15.1:0/3902590113 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f767c0549a0 con 0x7f7680099a40 2024-03-18T19:03:02.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.533+0000 7f7687ce4700 1 -- 172.21.15.1:0/3902590113 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f76801f4160 con 0x7f7680099a40 2024-03-18T19:03:02.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.538+0000 7f76767fc700 1 -- 172.21.15.1:0/3902590113 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v21) v1 ==== 172+0+0 (secure 0 0 0) 0x7f76801f4160 con 0x7f7680099a40 2024-03-18T19:03:02.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.544+0000 7f7687ce4700 1 -- 172.21.15.1:0/3902590113 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f766406adb0 msgr2=0x7f766406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:02.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.544+0000 7f7687ce4700 1 --2- 172.21.15.1:0/3902590113 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f766406adb0 0x7f766406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7670000c00 tx=0x7f7670008030 comp rx=0 tx=0).stop 2024-03-18T19:03:02.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.544+0000 7f7687ce4700 1 -- 172.21.15.1:0/3902590113 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7680099a40 msgr2=0x7f76801f3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:02.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.544+0000 7f7687ce4700 1 --2- 172.21.15.1:0/3902590113 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7680099a40 0x7f76801f3970 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f767c00c4d0 tx=0x7f767c00c890 comp rx=0 tx=0).stop 2024-03-18T19:03:02.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.546+0000 7f7687ce4700 1 -- 172.21.15.1:0/3902590113 shutdown_connections 2024-03-18T19:03:02.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.546+0000 7f7687ce4700 1 --2- 172.21.15.1:0/3902590113 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f766406adb0 0x7f766406d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.546+0000 7f7687ce4700 1 --2- 172.21.15.1:0/3902590113 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7680099e20 0x7f76801e9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.546+0000 7f7687ce4700 1 --2- 172.21.15.1:0/3902590113 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f768006fa50 0x7f76801e3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.546+0000 7f7687ce4700 1 --2- 172.21.15.1:0/3902590113 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7680099a40 0x7f76801f3970 secure :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f767c00c4d0 tx=0x7f767c00c890 comp rx=0 tx=0).stop 2024-03-18T19:03:02.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.546+0000 7f7687ce4700 1 -- 172.21.15.1:0/3902590113 >> 172.21.15.1:0/3902590113 conn(0x7f76800b2c50 msgr2=0x7f76800afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:02.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.549+0000 7f7687ce4700 1 -- 172.21.15.1:0/3902590113 shutdown_connections 2024-03-18T19:03:02.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.549+0000 7f7687ce4700 1 -- 172.21.15.1:0/3902590113 wait complete. 2024-03-18T19:03:02.554 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 7 and data pool 8 2024-03-18T19:03:02.572 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:03:02.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.700+0000 7f3d4887a700 1 Processor -- start 2024-03-18T19:03:02.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.700+0000 7f3d4887a700 1 -- start start 2024-03-18T19:03:02.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.701+0000 7f3d4887a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d4415b4d0 0x7f3d4409a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:02.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.701+0000 7f3d4887a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3d440cf750 0x7f3d44099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:02.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.701+0000 7f3d4887a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d4406f070 con 0x7f3d44099f80 2024-03-18T19:03:02.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.701+0000 7f3d4887a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d4409a410 con 0x7f3d440cf750 2024-03-18T19:03:02.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.701+0000 7f3d4887a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d440d0260 con 0x7f3d4415b4d0 2024-03-18T19:03:02.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.702+0000 7f3d4259c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d4415b4d0 0x7f3d4409a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:02.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.702+0000 7f3d41d9b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3d440cf750 0x7f3d44099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:02.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.702+0000 7f3d42d9d700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f3d44099f80 0x7f3d440cbd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:41694/0 (socket says 172.21.15.1:41694) 2024-03-18T19:03:02.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.702+0000 7f3d42d9d700 1 -- 172.21.15.1:0/2236470391 learned_addr learned my addr 172.21.15.1:0/2236470391 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:03:02.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.702+0000 7f3d41d9b700 1 -- 172.21.15.1:0/2236470391 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d4415b4d0 msgr2=0x7f3d4409a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:02.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.702+0000 7f3d41d9b700 1 --2- 172.21.15.1:0/2236470391 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d4415b4d0 0x7f3d4409a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.702+0000 7f3d41d9b700 1 -- 172.21.15.1:0/2236470391 >> v1:172.21.15.165:6789/0 conn(0x7f3d44099f80 legacy=0x7f3d440cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:02.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.702+0000 7f3d41d9b700 1 -- 172.21.15.1:0/2236470391 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d440cff80 con 0x7f3d440cf750 2024-03-18T19:03:02.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.702+0000 7f3d4259c700 1 --2- 172.21.15.1:0/2236470391 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d4415b4d0 0x7f3d4409a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:03:02.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.702+0000 7f3d41d9b700 1 --2- 172.21.15.1:0/2236470391 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d440cf750 0x7f3d44099a40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3d34009840 tx=0x7f3d34009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbb0d554be4af749 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:02.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.702+0000 7f3d4159a700 1 -- 172.21.15.1:0/2236470391 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d34011830 con 0x7f3d440cf750 2024-03-18T19:03:02.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.702+0000 7f3d4159a700 1 -- 172.21.15.1:0/2236470391 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d34011990 con 0x7f3d440cf750 2024-03-18T19:03:02.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.702+0000 7f3d4159a700 1 -- 172.21.15.1:0/2236470391 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d34011d20 con 0x7f3d440cf750 2024-03-18T19:03:02.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.703+0000 7f3d4887a700 1 -- 172.21.15.1:0/2236470391 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d440cf750 msgr2=0x7f3d44099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:02.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.703+0000 7f3d4887a700 1 --2- 172.21.15.1:0/2236470391 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d440cf750 0x7f3d44099a40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3d34009840 tx=0x7f3d34009b50 comp rx=0 tx=0).stop 2024-03-18T19:03:02.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.703+0000 7f3d4887a700 1 -- 172.21.15.1:0/2236470391 shutdown_connections 2024-03-18T19:03:02.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.703+0000 7f3d4887a700 1 --2- 172.21.15.1:0/2236470391 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d4415b4d0 0x7f3d4409a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.703+0000 7f3d4887a700 1 --2- 172.21.15.1:0/2236470391 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d440cf750 0x7f3d44099a40 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.703+0000 7f3d4887a700 1 -- 172.21.15.1:0/2236470391 >> 172.21.15.1:0/2236470391 conn(0x7f3d440b2cb0 msgr2=0x7f3d440b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:02.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.703+0000 7f3d4887a700 1 -- 172.21.15.1:0/2236470391 shutdown_connections 2024-03-18T19:03:02.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.703+0000 7f3d4887a700 1 -- 172.21.15.1:0/2236470391 wait complete. 2024-03-18T19:03:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.704+0000 7f3d4887a700 1 Processor -- start 2024-03-18T19:03:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.704+0000 7f3d4887a700 1 -- start start 2024-03-18T19:03:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.705+0000 7f3d4887a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d44099f80 0x7f3d44150520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.705+0000 7f3d4887a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d440cf750 0x7f3d440c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.705+0000 7f3d4887a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d4415b4d0 0x7f3d440b0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.705+0000 7f3d4887a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d440c8f50 con 0x7f3d4415b4d0 2024-03-18T19:03:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.705+0000 7f3d4887a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d440c9230 con 0x7f3d440cf750 2024-03-18T19:03:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.705+0000 7f3d4887a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d440c9510 con 0x7f3d44099f80 2024-03-18T19:03:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.705+0000 7f3d42d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d4415b4d0 0x7f3d440b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.705+0000 7f3d4259c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d44099f80 0x7f3d44150520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.705+0000 7f3d41d9b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d440cf750 0x7f3d440c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.705+0000 7f3d42d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d4415b4d0 0x7f3d440b0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48760/0 (socket says 172.21.15.1:48760) 2024-03-18T19:03:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.705+0000 7f3d42d9d700 1 -- 172.21.15.1:0/1750529281 learned_addr learned my addr 172.21.15.1:0/1750529281 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.706+0000 7f3d42d9d700 1 -- 172.21.15.1:0/1750529281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d44099f80 msgr2=0x7f3d44150520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.706+0000 7f3d42d9d700 1 --2- 172.21.15.1:0/1750529281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d44099f80 0x7f3d44150520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.706+0000 7f3d42d9d700 1 -- 172.21.15.1:0/1750529281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d440cf750 msgr2=0x7f3d440c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.706+0000 7f3d42d9d700 1 --2- 172.21.15.1:0/1750529281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d440cf750 0x7f3d440c9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.706+0000 7f3d42d9d700 1 -- 172.21.15.1:0/1750529281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d380094a0 con 0x7f3d4415b4d0 2024-03-18T19:03:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.706+0000 7f3d42d9d700 1 --2- 172.21.15.1:0/1750529281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d4415b4d0 0x7f3d440b0040 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c00cbc0 tx=0x7f3d3c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.706+0000 7f3d32ffd700 1 -- 172.21.15.1:0/1750529281 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c00a9e0 con 0x7f3d4415b4d0 2024-03-18T19:03:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.706+0000 7f3d32ffd700 1 -- 172.21.15.1:0/1750529281 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d3c00ab40 con 0x7f3d4415b4d0 2024-03-18T19:03:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.706+0000 7f3d4887a700 1 -- 172.21.15.1:0/1750529281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d340094a0 con 0x7f3d4415b4d0 2024-03-18T19:03:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.706+0000 7f3d32ffd700 1 -- 172.21.15.1:0/1750529281 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c012400 con 0x7f3d4415b4d0 2024-03-18T19:03:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.706+0000 7f3d4887a700 1 -- 172.21.15.1:0/1750529281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d440c9ad0 con 0x7f3d4415b4d0 2024-03-18T19:03:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.707+0000 7f3d4887a700 1 -- 172.21.15.1:0/1750529281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d44068240 con 0x7f3d4415b4d0 2024-03-18T19:03:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.707+0000 7f3d32ffd700 1 -- 172.21.15.1:0/1750529281 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3d3c01b3b0 con 0x7f3d4415b4d0 2024-03-18T19:03:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.708+0000 7f3d32ffd700 1 --2- 172.21.15.1:0/1750529281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3d2406adb0 0x7f3d2406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.708+0000 7f3d32ffd700 1 -- 172.21.15.1:0/1750529281 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3d3c0890c0 con 0x7f3d4415b4d0 2024-03-18T19:03:02.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.708+0000 7f3d4259c700 1 --2- 172.21.15.1:0/1750529281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3d2406adb0 0x7f3d2406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:02.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.708+0000 7f3d4259c700 1 --2- 172.21.15.1:0/1750529281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3d2406adb0 0x7f3d2406d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3d38009cd0 tx=0x7f3d38009210 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:02.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.712+0000 7f3d32ffd700 1 -- 172.21.15.1:0/1750529281 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3d3c054100 con 0x7f3d4415b4d0 2024-03-18T19:03:02.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.915+0000 7f3d4887a700 1 -- 172.21.15.1:0/1750529281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d440c9cc0 con 0x7f3d4415b4d0 2024-03-18T19:03:02.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.917+0000 7f3d32ffd700 1 -- 172.21.15.1:0/1750529281 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16271 (secure 0 0 0) 0x7f3d3c057720 con 0x7f3d4415b4d0 2024-03-18T19:03:02.913 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:03:02.913 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":70,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:03:02.534612+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:03:00.371114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-18T19:03:01.513069+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:03:02.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.919+0000 7f3d4887a700 1 -- 172.21.15.1:0/1750529281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3d2406adb0 msgr2=0x7f3d2406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:02.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.919+0000 7f3d4887a700 1 --2- 172.21.15.1:0/1750529281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3d2406adb0 0x7f3d2406d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3d38009cd0 tx=0x7f3d38009210 comp rx=0 tx=0).stop 2024-03-18T19:03:02.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.919+0000 7f3d4887a700 1 -- 172.21.15.1:0/1750529281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d4415b4d0 msgr2=0x7f3d440b0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:02.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.919+0000 7f3d4887a700 1 --2- 172.21.15.1:0/1750529281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d4415b4d0 0x7f3d440b0040 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c00cbc0 tx=0x7f3d3c00ced0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.919+0000 7f3d4887a700 1 -- 172.21.15.1:0/1750529281 shutdown_connections 2024-03-18T19:03:02.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.919+0000 7f3d4887a700 1 --2- 172.21.15.1:0/1750529281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3d2406adb0 0x7f3d2406d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.919+0000 7f3d4887a700 1 --2- 172.21.15.1:0/1750529281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d44099f80 0x7f3d44150520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.919+0000 7f3d4887a700 1 --2- 172.21.15.1:0/1750529281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d4415b4d0 0x7f3d440b0040 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.919+0000 7f3d4887a700 1 --2- 172.21.15.1:0/1750529281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d440cf750 0x7f3d440c9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:02.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.919+0000 7f3d4887a700 1 -- 172.21.15.1:0/1750529281 >> 172.21.15.1:0/1750529281 conn(0x7f3d440b2cb0 msgr2=0x7f3d4406fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:02.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.919+0000 7f3d4887a700 1 -- 172.21.15.1:0/1750529281 shutdown_connections 2024-03-18T19:03:02.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:02.919+0000 7f3d4887a700 1 -- 172.21.15.1:0/1750529281 wait complete. 2024-03-18T19:03:02.927 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:03:03.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.056+0000 7f95a65ed700 1 Processor -- start 2024-03-18T19:03:03.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.056+0000 7f95a65ed700 1 -- start start 2024-03-18T19:03:03.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.057+0000 7f95a65ed700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f95a015a2e0 0x7f95a00ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:03.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.057+0000 7f95a65ed700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95a00cc170 0x7f95a006ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:03.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.057+0000 7f95a65ed700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a006e270 con 0x7f95a00ae600 2024-03-18T19:03:03.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.057+0000 7f95a65ed700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a00ae050 con 0x7f95a015a2e0 2024-03-18T19:03:03.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.057+0000 7f95a65ed700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00add70 con 0x7f95a00cc170 2024-03-18T19:03:03.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.057+0000 7f959f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f95a015a2e0 0x7f95a00ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:03.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.057+0000 7f959f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f95a015a2e0 0x7f95a00ad790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48772/0 (socket says 172.21.15.1:48772) 2024-03-18T19:03:03.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.057+0000 7f959ffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f95a00ae600 0x7f95a00ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38582/0 (socket says 172.21.15.1:38582) 2024-03-18T19:03:03.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.057+0000 7f959f7fe700 1 -- 172.21.15.1:0/1319192560 learned_addr learned my addr 172.21.15.1:0/1319192560 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:03.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.058+0000 7f95a4b8a700 1 --2- 172.21.15.1:0/1319192560 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95a00cc170 0x7f95a006ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:03.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.058+0000 7f959f7fe700 1 -- 172.21.15.1:0/1319192560 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95a00cc170 msgr2=0x7f95a006ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:03.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.058+0000 7f959f7fe700 1 --2- 172.21.15.1:0/1319192560 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95a00cc170 0x7f95a006ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:03.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.058+0000 7f959f7fe700 1 -- 172.21.15.1:0/1319192560 >> v1:172.21.15.1:6789/0 conn(0x7f95a00ae600 legacy=0x7f95a00ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:03.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.058+0000 7f959f7fe700 1 -- 172.21.15.1:0/1319192560 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a006e4a0 con 0x7f95a015a2e0 2024-03-18T19:03:03.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.058+0000 7f959effd700 1 -- 172.21.15.1:0/1319192560 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880658603 0 0) 0x7f95a006e270 con 0x7f95a00ae600 2024-03-18T19:03:03.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.058+0000 7f959f7fe700 1 --2- 172.21.15.1:0/1319192560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a015a2e0 0x7f95a00ad790 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f959400bb90 tx=0x7f959400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d0e801091bb8070 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:03.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.058+0000 7f959effd700 1 -- 172.21.15.1:0/1319192560 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959400dd50 con 0x7f95a015a2e0 2024-03-18T19:03:03.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.058+0000 7f959effd700 1 -- 172.21.15.1:0/1319192560 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9594010470 con 0x7f95a015a2e0 2024-03-18T19:03:03.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.058+0000 7f959effd700 1 -- 172.21.15.1:0/1319192560 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9594010800 con 0x7f95a015a2e0 2024-03-18T19:03:03.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.059+0000 7f95a65ed700 1 -- 172.21.15.1:0/1319192560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a015a2e0 msgr2=0x7f95a00ad790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:03.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.059+0000 7f95a65ed700 1 --2- 172.21.15.1:0/1319192560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a015a2e0 0x7f95a00ad790 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f959400bb90 tx=0x7f959400bea0 comp rx=0 tx=0).stop 2024-03-18T19:03:03.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.059+0000 7f95a65ed700 1 -- 172.21.15.1:0/1319192560 shutdown_connections 2024-03-18T19:03:03.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.059+0000 7f95a65ed700 1 --2- 172.21.15.1:0/1319192560 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95a00cc170 0x7f95a006ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:03.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.059+0000 7f95a65ed700 1 --2- 172.21.15.1:0/1319192560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a015a2e0 0x7f95a00ad790 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:03.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.059+0000 7f95a65ed700 1 -- 172.21.15.1:0/1319192560 >> 172.21.15.1:0/1319192560 conn(0x7f95a006b860 msgr2=0x7f95a006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:03.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.059+0000 7f95a65ed700 1 -- 172.21.15.1:0/1319192560 shutdown_connections 2024-03-18T19:03:03.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.059+0000 7f95a65ed700 1 -- 172.21.15.1:0/1319192560 wait complete. 2024-03-18T19:03:03.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.060+0000 7f95a65ed700 1 Processor -- start 2024-03-18T19:03:03.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.060+0000 7f95a65ed700 1 -- start start 2024-03-18T19:03:03.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.061+0000 7f95a65ed700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95a00ae600 0x7f95a0148a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:03.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.061+0000 7f95a65ed700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95a00cc170 0x7f95a015c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:03.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.061+0000 7f95a65ed700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a015a2e0 0x7f95a015c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:03.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.061+0000 7f95a65ed700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00cdd10 con 0x7f95a015a2e0 2024-03-18T19:03:03.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.061+0000 7f95a65ed700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a0148f80 con 0x7f95a00ae600 2024-03-18T19:03:03.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.061+0000 7f95a65ed700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00cd090 con 0x7f95a00cc170 2024-03-18T19:03:03.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.061+0000 7f95a4b8a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a015a2e0 0x7f95a015c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:03.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.061+0000 7f95a4b8a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a015a2e0 0x7f95a015c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48782/0 (socket says 172.21.15.1:48782) 2024-03-18T19:03:03.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.061+0000 7f95a4b8a700 1 -- 172.21.15.1:0/1246582863 learned_addr learned my addr 172.21.15.1:0/1246582863 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:03.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.061+0000 7f959f7fe700 1 --2- 172.21.15.1:0/1246582863 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95a00cc170 0x7f95a015c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:03.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.061+0000 7f959ffff700 1 --2- 172.21.15.1:0/1246582863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95a00ae600 0x7f95a0148a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:03.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.061+0000 7f95a4b8a700 1 -- 172.21.15.1:0/1246582863 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95a00cc170 msgr2=0x7f95a015c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:03.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.061+0000 7f95a4b8a700 1 --2- 172.21.15.1:0/1246582863 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95a00cc170 0x7f95a015c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:03.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.061+0000 7f95a4b8a700 1 -- 172.21.15.1:0/1246582863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95a00ae600 msgr2=0x7f95a0148a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:03.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.061+0000 7f95a4b8a700 1 --2- 172.21.15.1:0/1246582863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95a00ae600 0x7f95a0148a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:03.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.061+0000 7f95a4b8a700 1 -- 172.21.15.1:0/1246582863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f959400b7f0 con 0x7f95a015a2e0 2024-03-18T19:03:03.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.061+0000 7f959f7fe700 1 --2- 172.21.15.1:0/1246582863 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95a00cc170 0x7f95a015c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:03:03.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.061+0000 7f959ffff700 1 --2- 172.21.15.1:0/1246582863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95a00ae600 0x7f95a0148a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:03:03.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.061+0000 7f95a4b8a700 1 --2- 172.21.15.1:0/1246582863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a015a2e0 0x7f95a015c690 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f9598007e70 tx=0x7f959800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:03.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.062+0000 7f959cff9700 1 -- 172.21.15.1:0/1246582863 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959800ee50 con 0x7f95a015a2e0 2024-03-18T19:03:03.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.062+0000 7f959cff9700 1 -- 172.21.15.1:0/1246582863 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9598014420 con 0x7f95a015a2e0 2024-03-18T19:03:03.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.062+0000 7f95a65ed700 1 -- 172.21.15.1:0/1246582863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a00cd370 con 0x7f95a015a2e0 2024-03-18T19:03:03.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.062+0000 7f959cff9700 1 -- 172.21.15.1:0/1246582863 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95980147d0 con 0x7f95a015a2e0 2024-03-18T19:03:03.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.062+0000 7f95a65ed700 1 -- 172.21.15.1:0/1246582863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a00cd930 con 0x7f95a015a2e0 2024-03-18T19:03:03.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.063+0000 7f95a65ed700 1 -- 172.21.15.1:0/1246582863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a00cde50 con 0x7f95a015a2e0 2024-03-18T19:03:03.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.063+0000 7f959cff9700 1 -- 172.21.15.1:0/1246582863 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9598014930 con 0x7f95a015a2e0 2024-03-18T19:03:03.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.064+0000 7f959cff9700 1 --2- 172.21.15.1:0/1246582863 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f958c06ade0 0x7f958c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:03.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.064+0000 7f959ffff700 1 --2- 172.21.15.1:0/1246582863 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f958c06ade0 0x7f958c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:03.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.064+0000 7f959cff9700 1 -- 172.21.15.1:0/1246582863 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9598089e40 con 0x7f95a015a2e0 2024-03-18T19:03:03.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.064+0000 7f959ffff700 1 --2- 172.21.15.1:0/1246582863 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f958c06ade0 0x7f958c06d2a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9590000f60 tx=0x7f9590009040 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:03.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.068+0000 7f959cff9700 1 -- 172.21.15.1:0/1246582863 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f95a00cde50 con 0x7f95a015a2e0 2024-03-18T19:03:03.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.270+0000 7f95a65ed700 1 -- 172.21.15.1:0/1246582863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95a00cde50 con 0x7f95a015a2e0 2024-03-18T19:03:03.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.271+0000 7f959cff9700 1 -- 172.21.15.1:0/1246582863 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16271 (secure 0 0 0) 0x7f95a00cde50 con 0x7f95a015a2e0 2024-03-18T19:03:03.267 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:03:03.267 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":70,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:03:02.534612+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:03:00.371114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-18T19:03:01.513069+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:03:03.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.273+0000 7f95a65ed700 1 -- 172.21.15.1:0/1246582863 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f958c06ade0 msgr2=0x7f958c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:03.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.273+0000 7f95a65ed700 1 --2- 172.21.15.1:0/1246582863 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f958c06ade0 0x7f958c06d2a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9590000f60 tx=0x7f9590009040 comp rx=0 tx=0).stop 2024-03-18T19:03:03.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.273+0000 7f95a65ed700 1 -- 172.21.15.1:0/1246582863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a015a2e0 msgr2=0x7f95a015c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:03.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.273+0000 7f95a65ed700 1 --2- 172.21.15.1:0/1246582863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a015a2e0 0x7f95a015c690 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f9598007e70 tx=0x7f959800e3c0 comp rx=0 tx=0).stop 2024-03-18T19:03:03.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.273+0000 7f95a65ed700 1 -- 172.21.15.1:0/1246582863 shutdown_connections 2024-03-18T19:03:03.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.273+0000 7f95a65ed700 1 --2- 172.21.15.1:0/1246582863 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f958c06ade0 0x7f958c06d2a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:03.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.273+0000 7f95a65ed700 1 --2- 172.21.15.1:0/1246582863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95a015a2e0 0x7f95a015c690 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:03.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.273+0000 7f95a65ed700 1 --2- 172.21.15.1:0/1246582863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95a00ae600 0x7f95a0148a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:03.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.273+0000 7f95a65ed700 1 --2- 172.21.15.1:0/1246582863 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95a00cc170 0x7f95a015c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:03.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.273+0000 7f95a65ed700 1 -- 172.21.15.1:0/1246582863 >> 172.21.15.1:0/1246582863 conn(0x7f95a006b860 msgr2=0x7f95a006a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:03.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.273+0000 7f95a65ed700 1 -- 172.21.15.1:0/1246582863 shutdown_connections 2024-03-18T19:03:03.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.273+0000 7f95a65ed700 1 -- 172.21.15.1:0/1246582863 wait complete. 2024-03-18T19:03:03.282 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T19:03:03.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.413+0000 7f7a1c1f4700 1 Processor -- start 2024-03-18T19:03:03.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.414+0000 7f7a1c1f4700 1 -- start start 2024-03-18T19:03:03.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.414+0000 7f7a1c1f4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a1415d5b0 0x7f7a1415d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:03.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.414+0000 7f7a1c1f4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7a14099a40 0x7f7a14099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:03.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.414+0000 7f7a1c1f4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7a1409a3e0 0x7f7a1406ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:03.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.414+0000 7f7a1c1f4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a140cbbb0 con 0x7f7a1409a3e0 2024-03-18T19:03:03.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.414+0000 7f7a1c1f4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a1409a830 con 0x7f7a14099a40 2024-03-18T19:03:03.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.414+0000 7f7a1c1f4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a140b3b00 con 0x7f7a1415d5b0 2024-03-18T19:03:03.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.415+0000 7f7a1a791700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7a1409a3e0 0x7f7a1406ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:03.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.415+0000 7f7a1a791700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7a1409a3e0 0x7f7a1406ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48796/0 (socket says 172.21.15.1:48796) 2024-03-18T19:03:03.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.415+0000 7f7a1a791700 1 -- 172.21.15.1:0/3146271784 learned_addr learned my addr 172.21.15.1:0/3146271784 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:03.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.415+0000 7f7a19f90700 1 --2- 172.21.15.1:0/3146271784 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a1415d5b0 0x7f7a1415d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.415+0000 7f7a1978f700 1 --2- 172.21.15.1:0/3146271784 >> v2:172.21.15.165:3300/0 conn(0x7f7a14099a40 0x7f7a14099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.415+0000 7f7a1a791700 1 -- 172.21.15.1:0/3146271784 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a1415d5b0 msgr2=0x7f7a1415d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.415+0000 7f7a1a791700 1 --2- 172.21.15.1:0/3146271784 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a1415d5b0 0x7f7a1415d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.415+0000 7f7a1a791700 1 -- 172.21.15.1:0/3146271784 >> v2:172.21.15.165:3300/0 conn(0x7f7a14099a40 msgr2=0x7f7a14099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.415+0000 7f7a1a791700 1 --2- 172.21.15.1:0/3146271784 >> v2:172.21.15.165:3300/0 conn(0x7f7a14099a40 0x7f7a14099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.415+0000 7f7a1a791700 1 -- 172.21.15.1:0/3146271784 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a140cf7f0 con 0x7f7a1409a3e0 2024-03-18T19:03:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.415+0000 7f7a1a791700 1 --2- 172.21.15.1:0/3146271784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a1409a3e0 0x7f7a1406ba70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7a140722f0 tx=0x7f7a0c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1ace48454ec9f3e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.415+0000 7f7a18f8e700 1 -- 172.21.15.1:0/3146271784 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c00e070 con 0x7f7a1409a3e0 2024-03-18T19:03:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.415+0000 7f7a18f8e700 1 -- 172.21.15.1:0/3146271784 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0c00b9c0 con 0x7f7a1409a3e0 2024-03-18T19:03:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.415+0000 7f7a18f8e700 1 -- 172.21.15.1:0/3146271784 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c00bd70 con 0x7f7a1409a3e0 2024-03-18T19:03:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.416+0000 7f7a1c1f4700 1 -- 172.21.15.1:0/3146271784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a1409a3e0 msgr2=0x7f7a1406ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.416+0000 7f7a1c1f4700 1 --2- 172.21.15.1:0/3146271784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a1409a3e0 0x7f7a1406ba70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7a140722f0 tx=0x7f7a0c00b330 comp rx=0 tx=0).stop 2024-03-18T19:03:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.416+0000 7f7a1c1f4700 1 -- 172.21.15.1:0/3146271784 shutdown_connections 2024-03-18T19:03:03.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.416+0000 7f7a1c1f4700 1 --2- 172.21.15.1:0/3146271784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a1409a3e0 0x7f7a1406ba70 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:03.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.416+0000 7f7a1c1f4700 1 --2- 172.21.15.1:0/3146271784 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a1415d5b0 0x7f7a1415d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:03.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.416+0000 7f7a1c1f4700 1 --2- 172.21.15.1:0/3146271784 >> v2:172.21.15.165:3300/0 conn(0x7f7a14099a40 0x7f7a14099ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:03.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.416+0000 7f7a1c1f4700 1 -- 172.21.15.1:0/3146271784 >> 172.21.15.1:0/3146271784 conn(0x7f7a140b2bd0 msgr2=0x7f7a140b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:03.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.416+0000 7f7a1c1f4700 1 -- 172.21.15.1:0/3146271784 shutdown_connections 2024-03-18T19:03:03.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.416+0000 7f7a1c1f4700 1 -- 172.21.15.1:0/3146271784 wait complete. 2024-03-18T19:03:03.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.417+0000 7f7a1c1f4700 1 Processor -- start 2024-03-18T19:03:03.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.417+0000 7f7a1c1f4700 1 -- start start 2024-03-18T19:03:03.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.418+0000 7f7a1c1f4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a14099a40 0x7f7a140b10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:03.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.418+0000 7f7a1c1f4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a1409a3e0 0x7f7a1414e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:03.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.418+0000 7f7a1c1f4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a1415d5b0 0x7f7a140aea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:03.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.418+0000 7f7a1c1f4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1414eb70 con 0x7f7a14099a40 2024-03-18T19:03:03.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.418+0000 7f7a1c1f4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a140b1c50 con 0x7f7a1409a3e0 2024-03-18T19:03:03.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.418+0000 7f7a1c1f4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a140b1690 con 0x7f7a1415d5b0 2024-03-18T19:03:03.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.418+0000 7f7a19f90700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a14099a40 0x7f7a140b10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:03.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.418+0000 7f7a19f90700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a14099a40 0x7f7a140b10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48812/0 (socket says 172.21.15.1:48812) 2024-03-18T19:03:03.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.418+0000 7f7a19f90700 1 -- 172.21.15.1:0/1708636825 learned_addr learned my addr 172.21.15.1:0/1708636825 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:03.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.418+0000 7f7a1978f700 1 --2- 172.21.15.1:0/1708636825 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a1409a3e0 0x7f7a1414e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:03.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.418+0000 7f7a1a791700 1 --2- 172.21.15.1:0/1708636825 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a1415d5b0 0x7f7a140aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:03.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.418+0000 7f7a19f90700 1 -- 172.21.15.1:0/1708636825 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a1415d5b0 msgr2=0x7f7a140aea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:03.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.418+0000 7f7a19f90700 1 --2- 172.21.15.1:0/1708636825 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a1415d5b0 0x7f7a140aea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:03.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.418+0000 7f7a19f90700 1 -- 172.21.15.1:0/1708636825 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a1409a3e0 msgr2=0x7f7a1414e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:03.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.418+0000 7f7a19f90700 1 --2- 172.21.15.1:0/1708636825 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a1409a3e0 0x7f7a1414e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:03.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.418+0000 7f7a19f90700 1 -- 172.21.15.1:0/1708636825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0c009b30 con 0x7f7a14099a40 2024-03-18T19:03:03.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.418+0000 7f7a19f90700 1 --2- 172.21.15.1:0/1708636825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a14099a40 0x7f7a140b10b0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7a1000e810 tx=0x7f7a1000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:03.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.419+0000 7f7a067fc700 1 -- 172.21.15.1:0/1708636825 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1000c070 con 0x7f7a14099a40 2024-03-18T19:03:03.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.419+0000 7f7a067fc700 1 -- 172.21.15.1:0/1708636825 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a10013610 con 0x7f7a14099a40 2024-03-18T19:03:03.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.419+0000 7f7a067fc700 1 -- 172.21.15.1:0/1708636825 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a100139c0 con 0x7f7a14099a40 2024-03-18T19:03:03.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.419+0000 7f7a1c1f4700 1 -- 172.21.15.1:0/1708636825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a140b1970 con 0x7f7a14099a40 2024-03-18T19:03:03.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.419+0000 7f7a1c1f4700 1 -- 172.21.15.1:0/1708636825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a140b5160 con 0x7f7a14099a40 2024-03-18T19:03:03.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.420+0000 7f7a1c1f4700 1 -- 172.21.15.1:0/1708636825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a140b54d0 con 0x7f7a14099a40 2024-03-18T19:03:03.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.420+0000 7f7a067fc700 1 -- 172.21.15.1:0/1708636825 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7a10013b20 con 0x7f7a14099a40 2024-03-18T19:03:03.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.420+0000 7f7a067fc700 1 --2- 172.21.15.1:0/1708636825 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f79f806adb0 0x7f79f806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:03.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.421+0000 7f7a067fc700 1 -- 172.21.15.1:0/1708636825 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7a10089420 con 0x7f7a14099a40 2024-03-18T19:03:03.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.421+0000 7f7a1978f700 1 --2- 172.21.15.1:0/1708636825 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f79f806adb0 0x7f79f806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:03.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.421+0000 7f7a1978f700 1 --2- 172.21.15.1:0/1708636825 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f79f806adb0 0x7f79f806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7a1414ee60 tx=0x7f7a08008040 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:03.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.425+0000 7f7a067fc700 1 -- 172.21.15.1:0/1708636825 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7a100542b0 con 0x7f7a14099a40 2024-03-18T19:03:03.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:03.675+0000 7f7a1c1f4700 1 -- 172.21.15.1:0/1708636825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7a140676a0 con 0x7f7a14099a40 2024-03-18T19:03:04.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.548+0000 7f7a067fc700 1 -- 172.21.15.1:0/1708636825 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7a100578d0 con 0x7f7a14099a40 2024-03-18T19:03:04.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.551+0000 7f7a1c1f4700 1 -- 172.21.15.1:0/1708636825 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f79f806adb0 msgr2=0x7f79f806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:04.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.551+0000 7f7a1c1f4700 1 --2- 172.21.15.1:0/1708636825 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f79f806adb0 0x7f79f806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7a1414ee60 tx=0x7f7a08008040 comp rx=0 tx=0).stop 2024-03-18T19:03:04.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.551+0000 7f7a1c1f4700 1 -- 172.21.15.1:0/1708636825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a14099a40 msgr2=0x7f7a140b10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:04.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.551+0000 7f7a1c1f4700 1 --2- 172.21.15.1:0/1708636825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a14099a40 0x7f7a140b10b0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7a1000e810 tx=0x7f7a1000eb20 comp rx=0 tx=0).stop 2024-03-18T19:03:04.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.551+0000 7f7a1c1f4700 1 -- 172.21.15.1:0/1708636825 shutdown_connections 2024-03-18T19:03:04.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.551+0000 7f7a1c1f4700 1 --2- 172.21.15.1:0/1708636825 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f79f806adb0 0x7f79f806d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:04.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.551+0000 7f7a1c1f4700 1 --2- 172.21.15.1:0/1708636825 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a1415d5b0 0x7f7a140aea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:04.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.551+0000 7f7a1c1f4700 1 --2- 172.21.15.1:0/1708636825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a14099a40 0x7f7a140b10b0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:04.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.551+0000 7f7a1c1f4700 1 --2- 172.21.15.1:0/1708636825 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a1409a3e0 0x7f7a1414e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:04.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.551+0000 7f7a1c1f4700 1 -- 172.21.15.1:0/1708636825 >> 172.21.15.1:0/1708636825 conn(0x7f7a140b2bd0 msgr2=0x7f7a140b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:04.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.551+0000 7f7a1c1f4700 1 -- 172.21.15.1:0/1708636825 shutdown_connections 2024-03-18T19:03:04.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.551+0000 7f7a1c1f4700 1 -- 172.21.15.1:0/1708636825 wait complete. 2024-03-18T19:03:04.562 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-18T19:03:04.562 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T19:03:04.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.702+0000 7f30b8c97700 1 Processor -- start 2024-03-18T19:03:04.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.702+0000 7f30b8c97700 1 -- start start 2024-03-18T19:03:04.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.703+0000 7f30b8c97700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f30b40cfa90 0x7f30b406a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:04.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.703+0000 7f30b8c97700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30b406ac90 0x7f30b4167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:04.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.703+0000 7f30b8c97700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b40cf600 con 0x7f30b415d5a0 2024-03-18T19:03:04.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.703+0000 7f30b8c97700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30b406b140 con 0x7f30b40cfa90 2024-03-18T19:03:04.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.703+0000 7f30b8c97700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b40ce720 con 0x7f30b406ac90 2024-03-18T19:03:04.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.703+0000 7f30b259c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f30b415d5a0 0x7f30b40ce010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60144/0 (socket says 172.21.15.1:60144) 2024-03-18T19:03:04.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.703+0000 7f30b259c700 1 -- 172.21.15.1:0/1905781540 learned_addr learned my addr 172.21.15.1:0/1905781540 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:03:04.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.703+0000 7f30b2d9d700 1 --2- 172.21.15.1:0/1905781540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30b406ac90 0x7f30b4167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:04.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.703+0000 7f30b1d9b700 1 --2- 172.21.15.1:0/1905781540 >> v2:172.21.15.165:3300/0 conn(0x7f30b40cfa90 0x7f30b406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.703+0000 7f30b159a700 1 -- 172.21.15.1:0/1905781540 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367491380 0 0) 0x7f30b40cf600 con 0x7f30b415d5a0 2024-03-18T19:03:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.703+0000 7f30b159a700 1 -- 172.21.15.1:0/1905781540 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f309c0035b0 con 0x7f30b415d5a0 2024-03-18T19:03:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b159a700 1 -- 172.21.15.1:0/1905781540 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1039047179 0 0) 0x7f309c0035b0 con 0x7f30b415d5a0 2024-03-18T19:03:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b159a700 1 -- 172.21.15.1:0/1905781540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30b406ac90 msgr2=0x7f30b4167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b159a700 1 --2- 172.21.15.1:0/1905781540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30b406ac90 0x7f30b4167140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b159a700 1 -- 172.21.15.1:0/1905781540 >> v2:172.21.15.165:3300/0 conn(0x7f30b40cfa90 msgr2=0x7f30b406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b159a700 1 --2- 172.21.15.1:0/1905781540 >> v2:172.21.15.165:3300/0 conn(0x7f30b40cfa90 0x7f30b406a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b159a700 1 -- 172.21.15.1:0/1905781540 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b40ce9f0 con 0x7f30b415d5a0 2024-03-18T19:03:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b159a700 1 -- 172.21.15.1:0/1905781540 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f30a80017f0 con 0x7f30b415d5a0 2024-03-18T19:03:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b159a700 1 -- 172.21.15.1:0/1905781540 >> v1:172.21.15.1:6789/0 conn(0x7f30b415d5a0 legacy=0x7f30b40ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b159a700 1 --2- 172.21.15.1:0/1905781540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30b406ac90 0x7f309c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:04.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b159a700 1 --2- 172.21.15.1:0/1905781540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f309c003f80 0x7f309c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:04.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b159a700 1 --2- 172.21.15.1:0/1905781540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f309c004d90 0x7f309c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:04.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b159a700 1 -- 172.21.15.1:0/1905781540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b406b140 con 0x7f309c004d90 2024-03-18T19:03:04.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b159a700 1 -- 172.21.15.1:0/1905781540 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b40ce720 con 0x7f309c003f80 2024-03-18T19:03:04.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b159a700 1 -- 172.21.15.1:0/1905781540 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309c00ba10 con 0x7f30b406ac90 2024-03-18T19:03:04.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b2d9d700 1 --2- 172.21.15.1:0/1905781540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f309c004d90 0x7f309c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:04.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b259c700 1 --2- 172.21.15.1:0/1905781540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30b406ac90 0x7f309c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:04.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b1d9b700 1 --2- 172.21.15.1:0/1905781540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f309c003f80 0x7f309c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:04.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b2d9d700 1 -- 172.21.15.1:0/1905781540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30b406ac90 msgr2=0x7f309c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:04.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b2d9d700 1 --2- 172.21.15.1:0/1905781540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30b406ac90 0x7f309c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:04.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b2d9d700 1 -- 172.21.15.1:0/1905781540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f309c003f80 msgr2=0x7f309c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:04.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b2d9d700 1 --2- 172.21.15.1:0/1905781540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f309c003f80 0x7f309c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:04.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b2d9d700 1 -- 172.21.15.1:0/1905781540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309c0035b0 con 0x7f309c004d90 2024-03-18T19:03:04.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b2d9d700 1 --2- 172.21.15.1:0/1905781540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f309c004d90 0x7f309c005210 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f30ac009fc0 tx=0x7f309c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3472cbfdea407 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:04.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.704+0000 7f30b159a700 1 -- 172.21.15.1:0/1905781540 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ac00ebd0 con 0x7f309c004d90 2024-03-18T19:03:04.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.705+0000 7f30b159a700 1 -- 172.21.15.1:0/1905781540 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ac00f040 con 0x7f309c004d90 2024-03-18T19:03:04.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.705+0000 7f30b159a700 1 -- 172.21.15.1:0/1905781540 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ac014480 con 0x7f309c004d90 2024-03-18T19:03:04.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.705+0000 7f30b8c97700 1 -- 172.21.15.1:0/1905781540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f309c004d90 msgr2=0x7f309c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:04.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.705+0000 7f30b8c97700 1 --2- 172.21.15.1:0/1905781540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f309c004d90 0x7f309c005210 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f30ac009fc0 tx=0x7f309c005880 comp rx=0 tx=0).stop 2024-03-18T19:03:04.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.705+0000 7f30b2d9d700 1 -- 172.21.15.1:0/1905781540 reap_dead start 2024-03-18T19:03:04.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.705+0000 7f30b8c97700 1 -- 172.21.15.1:0/1905781540 shutdown_connections 2024-03-18T19:03:04.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.705+0000 7f30b8c97700 1 -- 172.21.15.1:0/1905781540 >> 172.21.15.1:0/1905781540 conn(0x7f30b40cc9b0 msgr2=0x7f30b40ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:04.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.705+0000 7f30b8c97700 1 -- 172.21.15.1:0/1905781540 shutdown_connections 2024-03-18T19:03:04.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.705+0000 7f30b8c97700 1 -- 172.21.15.1:0/1905781540 wait complete. 2024-03-18T19:03:04.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.707+0000 7f30b8c97700 1 Processor -- start 2024-03-18T19:03:04.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.707+0000 7f30b8c97700 1 -- start start 2024-03-18T19:03:04.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.708+0000 7f30b8c97700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f309c004d90 0x7f30b41f6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:04.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.708+0000 7f30b8c97700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30b40c7c40 0x7f30b40ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:04.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.708+0000 7f30b8c97700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30b415fd30 0x7f30b41601e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:04.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.708+0000 7f30b8c97700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b4160720 con 0x7f309c004d90 2024-03-18T19:03:04.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.708+0000 7f30b8c97700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b41f62e0 con 0x7f30b415fd30 2024-03-18T19:03:04.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.708+0000 7f30b8c97700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b40ca6d0 con 0x7f30b40c7c40 2024-03-18T19:03:04.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.708+0000 7f30b259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f309c004d90 0x7f30b41f6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:04.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.708+0000 7f30b259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f309c004d90 0x7f30b41f6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53846/0 (socket says 172.21.15.1:53846) 2024-03-18T19:03:04.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.708+0000 7f30b259c700 1 -- 172.21.15.1:0/4252674864 learned_addr learned my addr 172.21.15.1:0/4252674864 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:04.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.708+0000 7f30b259c700 1 -- 172.21.15.1:0/4252674864 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30b40c7c40 msgr2=0x7f30b40ca190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:04.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.708+0000 7f30b1d9b700 1 --2- 172.21.15.1:0/4252674864 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30b40c7c40 0x7f30b40ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:04.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.708+0000 7f30b2d9d700 1 --2- 172.21.15.1:0/4252674864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30b415fd30 0x7f30b41601e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:04.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.708+0000 7f30b259c700 1 --2- 172.21.15.1:0/4252674864 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30b40c7c40 0x7f30b40ca190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:04.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.708+0000 7f30b259c700 1 -- 172.21.15.1:0/4252674864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30b415fd30 msgr2=0x7f30b41601e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:04.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.708+0000 7f30b259c700 1 --2- 172.21.15.1:0/4252674864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30b415fd30 0x7f30b41601e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:04.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.708+0000 7f30b259c700 1 -- 172.21.15.1:0/4252674864 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b40c83a0 con 0x7f309c004d90 2024-03-18T19:03:04.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.708+0000 7f30b1d9b700 1 --2- 172.21.15.1:0/4252674864 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30b40c7c40 0x7f30b40ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:03:04.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.708+0000 7f30b259c700 1 --2- 172.21.15.1:0/4252674864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f309c004d90 0x7f30b41f6d30 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f30a800fb70 tx=0x7f30a800ff30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:04.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.708+0000 7f30a2ffd700 1 -- 172.21.15.1:0/4252674864 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a80072e0 con 0x7f309c004d90 2024-03-18T19:03:04.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.709+0000 7f30a2ffd700 1 -- 172.21.15.1:0/4252674864 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a8007440 con 0x7f309c004d90 2024-03-18T19:03:04.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.709+0000 7f30b8c97700 1 -- 172.21.15.1:0/4252674864 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30b40c8680 con 0x7f309c004d90 2024-03-18T19:03:04.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.709+0000 7f30a2ffd700 1 -- 172.21.15.1:0/4252674864 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a8005830 con 0x7f309c004d90 2024-03-18T19:03:04.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.709+0000 7f30b8c97700 1 -- 172.21.15.1:0/4252674864 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30b41f6810 con 0x7f309c004d90 2024-03-18T19:03:04.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.710+0000 7f30b8c97700 1 -- 172.21.15.1:0/4252674864 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30b40c88c0 con 0x7f309c004d90 2024-03-18T19:03:04.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.710+0000 7f30a2ffd700 1 -- 172.21.15.1:0/4252674864 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f30a8005990 con 0x7f309c004d90 2024-03-18T19:03:04.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.711+0000 7f30a2ffd700 1 --2- 172.21.15.1:0/4252674864 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f309406ade0 0x7f309406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:04.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.711+0000 7f30b1d9b700 1 --2- 172.21.15.1:0/4252674864 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f309406ade0 0x7f309406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:04.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.711+0000 7f30a2ffd700 1 -- 172.21.15.1:0/4252674864 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f30a8089820 con 0x7f309c004d90 2024-03-18T19:03:04.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.711+0000 7f30b1d9b700 1 --2- 172.21.15.1:0/4252674864 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f309406ade0 0x7f309406d2a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f30b41f7370 tx=0x7f30a4008040 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:04.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.715+0000 7f30a2ffd700 1 -- 172.21.15.1:0/4252674864 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f30a8054710 con 0x7f309c004d90 2024-03-18T19:03:04.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:04.967+0000 7f30b8c97700 1 -- 172.21.15.1:0/4252674864 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f30b40676a0 con 0x7f309c004d90 2024-03-18T19:03:05.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.553+0000 7f30a2ffd700 1 -- 172.21.15.1:0/4252674864 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) v1 ==== 122+0+0 (secure 0 0 0) 0x7f30a8057d30 con 0x7f309c004d90 2024-03-18T19:03:05.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.556+0000 7f30b8c97700 1 -- 172.21.15.1:0/4252674864 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f309406ade0 msgr2=0x7f309406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:05.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.556+0000 7f30b8c97700 1 --2- 172.21.15.1:0/4252674864 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f309406ade0 0x7f309406d2a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f30b41f7370 tx=0x7f30a4008040 comp rx=0 tx=0).stop 2024-03-18T19:03:05.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.556+0000 7f30b8c97700 1 -- 172.21.15.1:0/4252674864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f309c004d90 msgr2=0x7f30b41f6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:05.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.556+0000 7f30b8c97700 1 --2- 172.21.15.1:0/4252674864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f309c004d90 0x7f30b41f6d30 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f30a800fb70 tx=0x7f30a800ff30 comp rx=0 tx=0).stop 2024-03-18T19:03:05.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.556+0000 7f30b8c97700 1 -- 172.21.15.1:0/4252674864 shutdown_connections 2024-03-18T19:03:05.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.556+0000 7f30b8c97700 1 --2- 172.21.15.1:0/4252674864 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f309406ade0 0x7f309406d2a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:05.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.556+0000 7f30b8c97700 1 --2- 172.21.15.1:0/4252674864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30b415fd30 0x7f30b41601e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:05.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.556+0000 7f30b8c97700 1 --2- 172.21.15.1:0/4252674864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f309c004d90 0x7f30b41f6d30 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:05.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.556+0000 7f30b8c97700 1 --2- 172.21.15.1:0/4252674864 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30b40c7c40 0x7f30b40ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:05.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.556+0000 7f30b8c97700 1 -- 172.21.15.1:0/4252674864 >> 172.21.15.1:0/4252674864 conn(0x7f30b40cc9b0 msgr2=0x7f30b4071520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:05.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.556+0000 7f30b8c97700 1 -- 172.21.15.1:0/4252674864 shutdown_connections 2024-03-18T19:03:05.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.556+0000 7f30b8c97700 1 -- 172.21.15.1:0/4252674864 wait complete. 2024-03-18T19:03:05.567 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T19:03:05.567 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:03:05.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.703+0000 7f409f4c9700 1 Processor -- start 2024-03-18T19:03:05.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.704+0000 7f409f4c9700 1 -- start start 2024-03-18T19:03:05.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.704+0000 7f409f4c9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f409806dd50 0x7f4098072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:05.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.704+0000 7f409f4c9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f409806f490 con 0x7f409815d580 2024-03-18T19:03:05.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.704+0000 7f409f4c9700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4098071cf0 con 0x7f409806e290 2024-03-18T19:03:05.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.704+0000 7f409f4c9700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f409806f1b0 con 0x7f409806dd50 2024-03-18T19:03:05.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.704+0000 7f409da66700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f409806dd50 0x7f4098072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:05.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.704+0000 7f409d265700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f409815d580 0x7f409806d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60156/0 (socket says 172.21.15.1:60156) 2024-03-18T19:03:05.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.705+0000 7f409d265700 1 -- 172.21.15.1:0/2945752973 learned_addr learned my addr 172.21.15.1:0/2945752973 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:03:05.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.705+0000 7f409da66700 1 -- 172.21.15.1:0/2945752973 >> v1:172.21.15.165:6789/0 conn(0x7f409806e290 legacy=0x7f40980ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:05.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.705+0000 7f408ffff700 1 -- 172.21.15.1:0/2945752973 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848266312 0 0) 0x7f409806f490 con 0x7f409815d580 2024-03-18T19:03:05.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.705+0000 7f409da66700 1 -- 172.21.15.1:0/2945752973 >> v1:172.21.15.1:6789/0 conn(0x7f409815d580 legacy=0x7f409806d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:05.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.705+0000 7f409da66700 1 -- 172.21.15.1:0/2945752973 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40980716a0 con 0x7f409806dd50 2024-03-18T19:03:05.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.705+0000 7f409da66700 1 --2- 172.21.15.1:0/2945752973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f409806dd50 0x7f4098072180 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4090009d90 tx=0x7f409000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a55a1bb0e841fc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:05.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.705+0000 7f408ffff700 1 -- 172.21.15.1:0/2945752973 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409000e070 con 0x7f409806dd50 2024-03-18T19:03:05.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.705+0000 7f408ffff700 1 -- 172.21.15.1:0/2945752973 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409000bbd0 con 0x7f409806dd50 2024-03-18T19:03:05.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.705+0000 7f408ffff700 1 -- 172.21.15.1:0/2945752973 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4090012430 con 0x7f409806dd50 2024-03-18T19:03:05.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.706+0000 7f409f4c9700 1 -- 172.21.15.1:0/2945752973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f409806dd50 msgr2=0x7f4098072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:05.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.706+0000 7f409f4c9700 1 --2- 172.21.15.1:0/2945752973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f409806dd50 0x7f4098072180 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4090009d90 tx=0x7f409000b330 comp rx=0 tx=0).stop 2024-03-18T19:03:05.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.706+0000 7f409f4c9700 1 -- 172.21.15.1:0/2945752973 shutdown_connections 2024-03-18T19:03:05.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.706+0000 7f409f4c9700 1 --2- 172.21.15.1:0/2945752973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f409806dd50 0x7f4098072180 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:05.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.706+0000 7f409f4c9700 1 -- 172.21.15.1:0/2945752973 >> 172.21.15.1:0/2945752973 conn(0x7f40980cc990 msgr2=0x7f40980ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:05.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.706+0000 7f409f4c9700 1 -- 172.21.15.1:0/2945752973 shutdown_connections 2024-03-18T19:03:05.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.706+0000 7f409f4c9700 1 -- 172.21.15.1:0/2945752973 wait complete. 2024-03-18T19:03:05.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.707+0000 7f409f4c9700 1 Processor -- start 2024-03-18T19:03:05.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.707+0000 7f409f4c9700 1 -- start start 2024-03-18T19:03:05.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f409f4c9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409806dd50 0x7f40980b5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:05.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f409f4c9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f409815d580 0x7f40981520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:05.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f409f4c9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f409815b550 0x7f40980b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:05.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f409f4c9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40980c8f40 con 0x7f409815b550 2024-03-18T19:03:05.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f409f4c9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40980c9220 con 0x7f409806dd50 2024-03-18T19:03:05.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f409f4c9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40980c9500 con 0x7f409815d580 2024-03-18T19:03:05.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f409da66700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f409815b550 0x7f40980b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:05.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f409da66700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f409815b550 0x7f40980b0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53864/0 (socket says 172.21.15.1:53864) 2024-03-18T19:03:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f409da66700 1 -- 172.21.15.1:0/3778350112 learned_addr learned my addr 172.21.15.1:0/3778350112 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f409d265700 1 --2- 172.21.15.1:0/3778350112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409806dd50 0x7f40980b5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f409ca64700 1 --2- 172.21.15.1:0/3778350112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f409815d580 0x7f40981520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f409da66700 1 -- 172.21.15.1:0/3778350112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f409815d580 msgr2=0x7f40981520f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f409da66700 1 --2- 172.21.15.1:0/3778350112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f409815d580 0x7f40981520f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f409da66700 1 -- 172.21.15.1:0/3778350112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409806dd50 msgr2=0x7f40980b5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f409da66700 1 --2- 172.21.15.1:0/3778350112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409806dd50 0x7f40980b5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f409da66700 1 -- 172.21.15.1:0/3778350112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4090009a70 con 0x7f409815b550 2024-03-18T19:03:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f409ca64700 1 --2- 172.21.15.1:0/3778350112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f409815d580 0x7f40981520f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:03:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f409d265700 1 --2- 172.21.15.1:0/3778350112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409806dd50 0x7f40980b5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:03:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f409da66700 1 --2- 172.21.15.1:0/3778350112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f409815b550 0x7f40980b0a60 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f4090014040 tx=0x7f4090004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.708+0000 7f408dffb700 1 -- 172.21.15.1:0/3778350112 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4090012430 con 0x7f409815b550 2024-03-18T19:03:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.709+0000 7f408dffb700 1 -- 172.21.15.1:0/3778350112 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4090020070 con 0x7f409815b550 2024-03-18T19:03:05.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.709+0000 7f409f4c9700 1 -- 172.21.15.1:0/3778350112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40980c97e0 con 0x7f409815b550 2024-03-18T19:03:05.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.709+0000 7f408dffb700 1 -- 172.21.15.1:0/3778350112 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4090004280 con 0x7f409815b550 2024-03-18T19:03:05.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.709+0000 7f409f4c9700 1 -- 172.21.15.1:0/3778350112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40980b5a90 con 0x7f409815b550 2024-03-18T19:03:05.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.710+0000 7f409f4c9700 1 -- 172.21.15.1:0/3778350112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40980c9d00 con 0x7f409815b550 2024-03-18T19:03:05.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.710+0000 7f408dffb700 1 -- 172.21.15.1:0/3778350112 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f40900043e0 con 0x7f409815b550 2024-03-18T19:03:05.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.711+0000 7f408dffb700 1 --2- 172.21.15.1:0/3778350112 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f407c06adb0 0x7f407c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:05.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.711+0000 7f408dffb700 1 -- 172.21.15.1:0/3778350112 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4090022070 con 0x7f409815b550 2024-03-18T19:03:05.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.711+0000 7f409d265700 1 --2- 172.21.15.1:0/3778350112 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f407c06adb0 0x7f407c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:05.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.711+0000 7f409d265700 1 --2- 172.21.15.1:0/3778350112 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f407c06adb0 0x7f407c06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4088000f60 tx=0x7f4088009040 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:05.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.715+0000 7f408dffb700 1 -- 172.21.15.1:0/3778350112 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4090056430 con 0x7f409815b550 2024-03-18T19:03:05.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.964+0000 7f409f4c9700 1 -- 172.21.15.1:0/3778350112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40980676a0 con 0x7f409815b550 2024-03-18T19:03:05.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.964+0000 7f408dffb700 1 -- 172.21.15.1:0/3778350112 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4731 (secure 0 0 0) 0x7f4090059a50 con 0x7f409815b550 2024-03-18T19:03:05.960 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:03:05.961 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5420,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3327835778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":3327835778},{"type":"v1","addr":"172.21.15.165:6835","nonce":3327835778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6016,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/450295183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":450295183},{"type":"v1","addr":"172.21.15.1:6837","nonce":450295183}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6022,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1053303526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1053303526},{"type":"v1","addr":"172.21.15.165:6834","nonce":1053303526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:03:02.534936+0000","modified":"2024-03-18T19:03:05.546799+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6270":{"gid":6270,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6836/1826374081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1826374081},{"type":"v1","addr":"172.21.15.1:6836","nonce":1826374081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-18T19:03:05.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.966+0000 7f409f4c9700 1 -- 172.21.15.1:0/3778350112 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f407c06adb0 msgr2=0x7f407c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:05.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.966+0000 7f409f4c9700 1 --2- 172.21.15.1:0/3778350112 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f407c06adb0 0x7f407c06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4088000f60 tx=0x7f4088009040 comp rx=0 tx=0).stop 2024-03-18T19:03:05.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.966+0000 7f409f4c9700 1 -- 172.21.15.1:0/3778350112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f409815b550 msgr2=0x7f40980b0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:05.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.966+0000 7f409f4c9700 1 --2- 172.21.15.1:0/3778350112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f409815b550 0x7f40980b0a60 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f4090014040 tx=0x7f4090004ad0 comp rx=0 tx=0).stop 2024-03-18T19:03:05.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.966+0000 7f409f4c9700 1 -- 172.21.15.1:0/3778350112 shutdown_connections 2024-03-18T19:03:05.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.966+0000 7f409f4c9700 1 --2- 172.21.15.1:0/3778350112 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f407c06adb0 0x7f407c06d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:05.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.966+0000 7f409f4c9700 1 --2- 172.21.15.1:0/3778350112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f409815b550 0x7f40980b0a60 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:05.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.966+0000 7f409f4c9700 1 --2- 172.21.15.1:0/3778350112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409806dd50 0x7f40980b5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:05.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.966+0000 7f409f4c9700 1 --2- 172.21.15.1:0/3778350112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f409815d580 0x7f40981520f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:05.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.966+0000 7f409f4c9700 1 -- 172.21.15.1:0/3778350112 >> 172.21.15.1:0/3778350112 conn(0x7f40980cc990 msgr2=0x7f40980cc240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:05.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.966+0000 7f409f4c9700 1 -- 172.21.15.1:0/3778350112 shutdown_connections 2024-03-18T19:03:05.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:05.966+0000 7f409f4c9700 1 -- 172.21.15.1:0/3778350112 wait complete. 2024-03-18T19:03:05.964 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-03-18T19:03:05.975 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:03:06.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.105+0000 7fc56bff2700 1 Processor -- start 2024-03-18T19:03:06.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.105+0000 7fc56bff2700 1 -- start start 2024-03-18T19:03:06.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.106+0000 7fc56bff2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc56409a150 0x7fc564166070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:06.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.106+0000 7fc56bff2700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc56406f070 con 0x7fc56409a650 2024-03-18T19:03:06.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.106+0000 7fc56bff2700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5640d0260 con 0x7fc56415b4d0 2024-03-18T19:03:06.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.106+0000 7fc56bff2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5640cff80 con 0x7fc56409a150 2024-03-18T19:03:06.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.106+0000 7fc56958d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc56409a650 0x7fc564099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60162/0 (socket says 172.21.15.1:60162) 2024-03-18T19:03:06.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.106+0000 7fc56958d700 1 -- 172.21.15.1:0/4099734344 learned_addr learned my addr 172.21.15.1:0/4099734344 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:03:06.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.106+0000 7fc56a58f700 1 --2- 172.21.15.1:0/4099734344 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc56409a150 0x7fc564166070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:06.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.106+0000 7fc568d8c700 1 -- 172.21.15.1:0/4099734344 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176219949 0 0) 0x7fc56406f070 con 0x7fc56409a650 2024-03-18T19:03:06.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.106+0000 7fc568d8c700 1 -- 172.21.15.1:0/4099734344 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5500035b0 con 0x7fc56409a650 2024-03-18T19:03:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.106+0000 7fc568d8c700 1 -- 172.21.15.1:0/4099734344 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359111094 0 0) 0x7fc5640d0260 con 0x7fc56415b4d0 2024-03-18T19:03:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.107+0000 7fc568d8c700 1 -- 172.21.15.1:0/4099734344 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc56406f070 con 0x7fc56415b4d0 2024-03-18T19:03:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.107+0000 7fc56a58f700 1 -- 172.21.15.1:0/4099734344 >> v1:172.21.15.165:6789/0 conn(0x7fc56415b4d0 legacy=0x7fc5640cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.107+0000 7fc56a58f700 1 -- 172.21.15.1:0/4099734344 >> v1:172.21.15.1:6789/0 conn(0x7fc56409a650 legacy=0x7fc564099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.107+0000 7fc568d8c700 1 -- 172.21.15.1:0/4099734344 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1583725122 0 0) 0x7fc5500035b0 con 0x7fc56409a650 2024-03-18T19:03:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.107+0000 7fc56a58f700 1 -- 172.21.15.1:0/4099734344 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56406ea20 con 0x7fc56409a150 2024-03-18T19:03:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.107+0000 7fc56a58f700 1 --2- 172.21.15.1:0/4099734344 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc56409a150 0x7fc564166070 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc55c009dc0 tx=0x7fc55c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90886325122af77e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.107+0000 7fc568d8c700 1 -- 172.21.15.1:0/4099734344 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c00e070 con 0x7fc56409a150 2024-03-18T19:03:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.107+0000 7fc568d8c700 1 -- 172.21.15.1:0/4099734344 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55c003a20 con 0x7fc56409a150 2024-03-18T19:03:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.107+0000 7fc568d8c700 1 -- 172.21.15.1:0/4099734344 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc560001670 con 0x7fc56409a150 2024-03-18T19:03:06.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.107+0000 7fc56bff2700 1 -- 172.21.15.1:0/4099734344 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc56409a150 msgr2=0x7fc564166070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:06.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.107+0000 7fc56bff2700 1 --2- 172.21.15.1:0/4099734344 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc56409a150 0x7fc564166070 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc55c009dc0 tx=0x7fc55c00b330 comp rx=0 tx=0).stop 2024-03-18T19:03:06.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.108+0000 7fc56bff2700 1 -- 172.21.15.1:0/4099734344 shutdown_connections 2024-03-18T19:03:06.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.108+0000 7fc56bff2700 1 --2- 172.21.15.1:0/4099734344 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc56409a150 0x7fc564166070 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.108+0000 7fc56bff2700 1 -- 172.21.15.1:0/4099734344 >> 172.21.15.1:0/4099734344 conn(0x7fc5640b2cb0 msgr2=0x7fc5640b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:06.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.108+0000 7fc56bff2700 1 -- 172.21.15.1:0/4099734344 shutdown_connections 2024-03-18T19:03:06.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.108+0000 7fc56bff2700 1 -- 172.21.15.1:0/4099734344 wait complete. 2024-03-18T19:03:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.109+0000 7fc56bff2700 1 Processor -- start 2024-03-18T19:03:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.109+0000 7fc56bff2700 1 -- start start 2024-03-18T19:03:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc56bff2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc56409a150 0x7fc5640aeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc56bff2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc56409a650 0x7fc5640c6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc56bff2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc56415b4d0 0x7fc5640cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc56bff2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5640c7760 con 0x7fc56409a150 2024-03-18T19:03:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc56bff2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5640c7bf0 con 0x7fc56409a650 2024-03-18T19:03:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc56bff2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5640c6b50 con 0x7fc56415b4d0 2024-03-18T19:03:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc569d8e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc56409a150 0x7fc5640aeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc569d8e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc56409a150 0x7fc5640aeb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53866/0 (socket says 172.21.15.1:53866) 2024-03-18T19:03:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc569d8e700 1 -- 172.21.15.1:0/3961891138 learned_addr learned my addr 172.21.15.1:0/3961891138 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc56958d700 1 --2- 172.21.15.1:0/3961891138 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc56409a650 0x7fc5640c6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc56a58f700 1 --2- 172.21.15.1:0/3961891138 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc56415b4d0 0x7fc5640cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc569d8e700 1 -- 172.21.15.1:0/3961891138 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc56415b4d0 msgr2=0x7fc5640cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:06.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc569d8e700 1 --2- 172.21.15.1:0/3961891138 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc56415b4d0 0x7fc5640cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc569d8e700 1 -- 172.21.15.1:0/3961891138 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc56409a650 msgr2=0x7fc5640c6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:06.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc569d8e700 1 --2- 172.21.15.1:0/3961891138 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc56409a650 0x7fc5640c6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc569d8e700 1 -- 172.21.15.1:0/3961891138 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c009a70 con 0x7fc56409a150 2024-03-18T19:03:06.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc569d8e700 1 --2- 172.21.15.1:0/3961891138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc56409a150 0x7fc5640aeb50 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fc55400c500 tx=0x7fc55400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:06.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc55a7fc700 1 -- 172.21.15.1:0/3961891138 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5540052d0 con 0x7fc56409a150 2024-03-18T19:03:06.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc55a7fc700 1 -- 172.21.15.1:0/3961891138 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc554005450 con 0x7fc56409a150 2024-03-18T19:03:06.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc56bff2700 1 -- 172.21.15.1:0/3961891138 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5640c6e30 con 0x7fc56409a150 2024-03-18T19:03:06.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc55a7fc700 1 -- 172.21.15.1:0/3961891138 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5540038c0 con 0x7fc56409a150 2024-03-18T19:03:06.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.110+0000 7fc56bff2700 1 -- 172.21.15.1:0/3961891138 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5640c73f0 con 0x7fc56409a150 2024-03-18T19:03:06.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.111+0000 7fc56bff2700 1 -- 172.21.15.1:0/3961891138 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc564068240 con 0x7fc56409a150 2024-03-18T19:03:06.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.112+0000 7fc55a7fc700 1 -- 172.21.15.1:0/3961891138 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc554003a20 con 0x7fc56409a150 2024-03-18T19:03:06.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.112+0000 7fc55a7fc700 1 --2- 172.21.15.1:0/3961891138 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc54806adb0 0x7fc54806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:06.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.112+0000 7fc55a7fc700 1 -- 172.21.15.1:0/3961891138 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc554088a90 con 0x7fc56409a150 2024-03-18T19:03:06.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.112+0000 7fc56958d700 1 --2- 172.21.15.1:0/3961891138 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc54806adb0 0x7fc54806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:06.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.113+0000 7fc56958d700 1 --2- 172.21.15.1:0/3961891138 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc54806adb0 0x7fc54806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc564159ae0 tx=0x7fc560001b80 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:06.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.117+0000 7fc55a7fc700 1 -- 172.21.15.1:0/3961891138 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc554053a30 con 0x7fc56409a150 2024-03-18T19:03:06.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.320+0000 7fc56bff2700 1 -- 172.21.15.1:0/3961891138 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5640676a0 con 0x7fc56409a150 2024-03-18T19:03:06.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.321+0000 7fc55a7fc700 1 -- 172.21.15.1:0/3961891138 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16258 (secure 0 0 0) 0x7fc554057050 con 0x7fc56409a150 2024-03-18T19:03:06.317 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:03:06.317 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":71,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:03:03.536436+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:03:00.371114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-18T19:03:01.513069+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:03:06.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.324+0000 7fc56bff2700 1 -- 172.21.15.1:0/3961891138 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc54806adb0 msgr2=0x7fc54806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:06.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.324+0000 7fc56bff2700 1 --2- 172.21.15.1:0/3961891138 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc54806adb0 0x7fc54806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc564159ae0 tx=0x7fc560001b80 comp rx=0 tx=0).stop 2024-03-18T19:03:06.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.324+0000 7fc56bff2700 1 -- 172.21.15.1:0/3961891138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc56409a150 msgr2=0x7fc5640aeb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:06.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.324+0000 7fc56bff2700 1 --2- 172.21.15.1:0/3961891138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc56409a150 0x7fc5640aeb50 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fc55400c500 tx=0x7fc55400c8c0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.324+0000 7fc56bff2700 1 -- 172.21.15.1:0/3961891138 shutdown_connections 2024-03-18T19:03:06.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.324+0000 7fc56bff2700 1 --2- 172.21.15.1:0/3961891138 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc54806adb0 0x7fc54806d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.324+0000 7fc56bff2700 1 --2- 172.21.15.1:0/3961891138 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc56415b4d0 0x7fc5640cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.324+0000 7fc56bff2700 1 --2- 172.21.15.1:0/3961891138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc56409a150 0x7fc5640aeb50 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.324+0000 7fc56bff2700 1 --2- 172.21.15.1:0/3961891138 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc56409a650 0x7fc5640c6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.324+0000 7fc56bff2700 1 -- 172.21.15.1:0/3961891138 >> 172.21.15.1:0/3961891138 conn(0x7fc5640b2cb0 msgr2=0x7fc56415a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:06.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.324+0000 7fc56bff2700 1 -- 172.21.15.1:0/3961891138 shutdown_connections 2024-03-18T19:03:06.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.324+0000 7fc56bff2700 1 -- 172.21.15.1:0/3961891138 wait complete. 2024-03-18T19:03:06.332 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T19:03:06.332 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T19:03:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.461+0000 7f2b14c7e700 1 Processor -- start 2024-03-18T19:03:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.461+0000 7f2b14c7e700 1 -- start start 2024-03-18T19:03:06.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.462+0000 7f2b14c7e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2b100af610 0x7f2b100cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:06.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.462+0000 7f2b14c7e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b100ccf40 0x7f2b10071a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:06.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.462+0000 7f2b14c7e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b100cf090 con 0x7f2b10144b70 2024-03-18T19:03:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.462+0000 7f2b14c7e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b100cd3d0 con 0x7f2b100af610 2024-03-18T19:03:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.462+0000 7f2b14c7e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b10144f50 con 0x7f2b100ccf40 2024-03-18T19:03:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.462+0000 7f2b0e59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2b10144b70 0x7f2b100ce980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60166/0 (socket says 172.21.15.1:60166) 2024-03-18T19:03:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.462+0000 7f2b0e59c700 1 -- 172.21.15.1:0/4102212260 learned_addr learned my addr 172.21.15.1:0/4102212260 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:03:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.462+0000 7f2b0ed9d700 1 --2- 172.21.15.1:0/4102212260 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b100ccf40 0x7f2b10071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.462+0000 7f2b0d59a700 1 -- 172.21.15.1:0/4102212260 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391756341 0 0) 0x7f2b04001110 con 0x7f2b10144b70 2024-03-18T19:03:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0d59a700 1 -- 172.21.15.1:0/4102212260 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af80035b0 con 0x7f2b10144b70 2024-03-18T19:03:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0dd9b700 1 --2- 172.21.15.1:0/4102212260 >> v2:172.21.15.165:3300/0 conn(0x7f2b100af610 0x7f2b100cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0d59a700 1 -- 172.21.15.1:0/4102212260 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4144741176 0 0) 0x7f2af80035b0 con 0x7f2b10144b70 2024-03-18T19:03:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0d59a700 1 -- 172.21.15.1:0/4102212260 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b100ccf40 msgr2=0x7f2b10071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0d59a700 1 --2- 172.21.15.1:0/4102212260 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b100ccf40 0x7f2b10071a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0d59a700 1 -- 172.21.15.1:0/4102212260 >> v2:172.21.15.165:3300/0 conn(0x7f2b100af610 msgr2=0x7f2b100cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0d59a700 1 --2- 172.21.15.1:0/4102212260 >> v2:172.21.15.165:3300/0 conn(0x7f2b100af610 0x7f2b100cca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0d59a700 1 -- 172.21.15.1:0/4102212260 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b100cf2d0 con 0x7f2b10144b70 2024-03-18T19:03:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0d59a700 1 -- 172.21.15.1:0/4102212260 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f2b04001800 con 0x7f2b10144b70 2024-03-18T19:03:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0d59a700 1 -- 172.21.15.1:0/4102212260 >> v1:172.21.15.1:6789/0 conn(0x7f2b10144b70 legacy=0x7f2b100ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0d59a700 1 --2- 172.21.15.1:0/4102212260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2af8003a00 0x7f2af8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0d59a700 1 --2- 172.21.15.1:0/4102212260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2af8004850 0x7f2af8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0d59a700 1 --2- 172.21.15.1:0/4102212260 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b100ccf40 0x7f2af8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0d59a700 1 -- 172.21.15.1:0/4102212260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b100cd3d0 con 0x7f2af8004850 2024-03-18T19:03:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0d59a700 1 -- 172.21.15.1:0/4102212260 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b10144f50 con 0x7f2af8003a00 2024-03-18T19:03:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0d59a700 1 -- 172.21.15.1:0/4102212260 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af800ba10 con 0x7f2b100ccf40 2024-03-18T19:03:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0dd9b700 1 --2- 172.21.15.1:0/4102212260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2af8004850 0x7f2af8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0ed9d700 1 --2- 172.21.15.1:0/4102212260 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b100ccf40 0x7f2af8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:06.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0e59c700 1 --2- 172.21.15.1:0/4102212260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2af8003a00 0x7f2af8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0dd9b700 1 -- 172.21.15.1:0/4102212260 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b100ccf40 msgr2=0x7f2af8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0dd9b700 1 --2- 172.21.15.1:0/4102212260 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b100ccf40 0x7f2af8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0dd9b700 1 -- 172.21.15.1:0/4102212260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2af8003a00 msgr2=0x7f2af8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0dd9b700 1 --2- 172.21.15.1:0/4102212260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2af8003a00 0x7f2af8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0dd9b700 1 -- 172.21.15.1:0/4102212260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af80035b0 con 0x7f2af8004850 2024-03-18T19:03:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.463+0000 7f2b0dd9b700 1 --2- 172.21.15.1:0/4102212260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2af8004850 0x7f2af8004cd0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f2b00009b60 tx=0x7f2af8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f54801dc3fc82cf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.464+0000 7f2b0d59a700 1 -- 172.21.15.1:0/4102212260 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0000e6c0 con 0x7f2af8004850 2024-03-18T19:03:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.464+0000 7f2b0d59a700 1 -- 172.21.15.1:0/4102212260 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0000e820 con 0x7f2af8004850 2024-03-18T19:03:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.464+0000 7f2b0d59a700 1 -- 172.21.15.1:0/4102212260 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0000eb90 con 0x7f2af8004850 2024-03-18T19:03:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.464+0000 7f2b14c7e700 1 -- 172.21.15.1:0/4102212260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2af8004850 msgr2=0x7f2af8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.464+0000 7f2b14c7e700 1 --2- 172.21.15.1:0/4102212260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2af8004850 0x7f2af8004cd0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f2b00009b60 tx=0x7f2af8005880 comp rx=0 tx=0).stop 2024-03-18T19:03:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.464+0000 7f2b0ed9d700 1 -- 172.21.15.1:0/4102212260 reap_dead start 2024-03-18T19:03:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.464+0000 7f2b14c7e700 1 -- 172.21.15.1:0/4102212260 shutdown_connections 2024-03-18T19:03:06.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.464+0000 7f2b14c7e700 1 -- 172.21.15.1:0/4102212260 >> 172.21.15.1:0/4102212260 conn(0x7f2b100b1d20 msgr2=0x7f2b100b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:06.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.464+0000 7f2b14c7e700 1 -- 172.21.15.1:0/4102212260 shutdown_connections 2024-03-18T19:03:06.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.464+0000 7f2b14c7e700 1 -- 172.21.15.1:0/4102212260 wait complete. 2024-03-18T19:03:06.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.466+0000 7f2b14c7e700 1 Processor -- start 2024-03-18T19:03:06.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.466+0000 7f2b14c7e700 1 -- start start 2024-03-18T19:03:06.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.466+0000 7f2b14c7e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2af8004850 0x7f2b101e4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:06.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.466+0000 7f2b14c7e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b10153c60 0x7f2b100b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:06.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2b14c7e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b10142c30 0x7f2b101430b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:06.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2b14c7e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b101435f0 con 0x7f2b10142c30 2024-03-18T19:03:06.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2b14c7e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b100b4100 con 0x7f2af8004850 2024-03-18T19:03:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2b14c7e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b101e5180 con 0x7f2b10153c60 2024-03-18T19:03:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2b0ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b10142c30 0x7f2b101430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2b0ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b10142c30 0x7f2b101430b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53884/0 (socket says 172.21.15.1:53884) 2024-03-18T19:03:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2b0ed9d700 1 -- 172.21.15.1:0/2925842438 learned_addr learned my addr 172.21.15.1:0/2925842438 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2b0dd9b700 1 --2- 172.21.15.1:0/2925842438 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b10153c60 0x7f2b100b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2b0e59c700 1 --2- 172.21.15.1:0/2925842438 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2af8004850 0x7f2b101e4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2b0ed9d700 1 -- 172.21.15.1:0/2925842438 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b10153c60 msgr2=0x7f2b100b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2b0ed9d700 1 --2- 172.21.15.1:0/2925842438 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b10153c60 0x7f2b100b3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2b0ed9d700 1 -- 172.21.15.1:0/2925842438 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2af8004850 msgr2=0x7f2b101e4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2b0ed9d700 1 --2- 172.21.15.1:0/2925842438 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2af8004850 0x7f2b101e4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2b0ed9d700 1 -- 172.21.15.1:0/2925842438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b101e5460 con 0x7f2b10142c30 2024-03-18T19:03:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2b0ed9d700 1 --2- 172.21.15.1:0/2925842438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b10142c30 0x7f2b101430b0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2b100b62b0 tx=0x7f2b0800f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:06.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2afeffd700 1 -- 172.21.15.1:0/2925842438 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0800e040 con 0x7f2b10142c30 2024-03-18T19:03:06.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2afeffd700 1 -- 172.21.15.1:0/2925842438 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0800f040 con 0x7f2b10142c30 2024-03-18T19:03:06.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2afeffd700 1 -- 172.21.15.1:0/2925842438 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b08015430 con 0x7f2b10142c30 2024-03-18T19:03:06.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2b14c7e700 1 -- 172.21.15.1:0/2925842438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b101e5740 con 0x7f2b10142c30 2024-03-18T19:03:06.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.467+0000 7f2b14c7e700 1 -- 172.21.15.1:0/2925842438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b101e5d00 con 0x7f2b10142c30 2024-03-18T19:03:06.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.468+0000 7f2b14c7e700 1 -- 172.21.15.1:0/2925842438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b100cda90 con 0x7f2b10142c30 2024-03-18T19:03:06.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.469+0000 7f2afeffd700 1 -- 172.21.15.1:0/2925842438 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2b08015590 con 0x7f2b10142c30 2024-03-18T19:03:06.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.469+0000 7f2afeffd700 1 --2- 172.21.15.1:0/2925842438 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2af006adb0 0x7f2af006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:06.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.469+0000 7f2afeffd700 1 -- 172.21.15.1:0/2925842438 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2b08089c60 con 0x7f2b10142c30 2024-03-18T19:03:06.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.469+0000 7f2b0e59c700 1 --2- 172.21.15.1:0/2925842438 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2af006adb0 0x7f2af006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:06.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.470+0000 7f2b0e59c700 1 --2- 172.21.15.1:0/2925842438 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2af006adb0 0x7f2af006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2b04000fb0 tx=0x7f2b04009040 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:06.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.474+0000 7f2afeffd700 1 -- 172.21.15.1:0/2925842438 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2b08054ae0 con 0x7f2b10142c30 2024-03-18T19:03:06.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.672+0000 7f2b14c7e700 1 -- 172.21.15.1:0/2925842438 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2b100cfb70 con 0x7f2af006adb0 2024-03-18T19:03:06.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.681+0000 7f2afeffd700 1 -- 172.21.15.1:0/2925842438 <== mgr.6045 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287876 (secure 0 0 0) 0x7f2b04007030 con 0x7f2af006adb0 2024-03-18T19:03:06.681 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:03:06.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.684+0000 7f2b14c7e700 1 -- 172.21.15.1:0/2925842438 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2af006adb0 msgr2=0x7f2af006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:06.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.684+0000 7f2b14c7e700 1 --2- 172.21.15.1:0/2925842438 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2af006adb0 0x7f2af006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2b04000fb0 tx=0x7f2b04009040 comp rx=0 tx=0).stop 2024-03-18T19:03:06.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.684+0000 7f2b14c7e700 1 -- 172.21.15.1:0/2925842438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b10142c30 msgr2=0x7f2b101430b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:06.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.684+0000 7f2b14c7e700 1 --2- 172.21.15.1:0/2925842438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b10142c30 0x7f2b101430b0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2b100b62b0 tx=0x7f2b0800f490 comp rx=0 tx=0).stop 2024-03-18T19:03:06.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.684+0000 7f2b14c7e700 1 -- 172.21.15.1:0/2925842438 shutdown_connections 2024-03-18T19:03:06.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.684+0000 7f2b14c7e700 1 --2- 172.21.15.1:0/2925842438 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2af006adb0 0x7f2af006d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.684+0000 7f2b14c7e700 1 --2- 172.21.15.1:0/2925842438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b10142c30 0x7f2b101430b0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.684+0000 7f2b14c7e700 1 --2- 172.21.15.1:0/2925842438 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2af8004850 0x7f2b101e4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.684+0000 7f2b14c7e700 1 --2- 172.21.15.1:0/2925842438 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b10153c60 0x7f2b100b3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.684+0000 7f2b14c7e700 1 -- 172.21.15.1:0/2925842438 >> 172.21.15.1:0/2925842438 conn(0x7f2b100b1d20 msgr2=0x7f2b10141060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:06.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.684+0000 7f2b14c7e700 1 -- 172.21.15.1:0/2925842438 shutdown_connections 2024-03-18T19:03:06.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.684+0000 7f2b14c7e700 1 -- 172.21.15.1:0/2925842438 wait complete. 2024-03-18T19:03:06.684 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T19:03:06.692 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-18T19:03:04.966772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":240,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":175,"ondisk_log_size":175,"up":176,"acting":176,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":98,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":309240,"kb_used_data":7796,"kb_used_omap":12,"kb_used_meta":301299,"kb_avail":754665480,"statfs":{"total":773094113280,"available":772777451520,"internally_reserved":0,"allocated":7983104,"data_stored":4498409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":308530745},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000674"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542736+0000","last_change":"2024-03-18T19:03:02.554946+0000","last_active":"2024-03-18T19:03:03.542736+0000","last_peered":"2024-03-18T19:03:03.542736+0000","last_clean":"2024-03-18T19:03:03.542736+0000","last_became_active":"2024-03-18T19:03:02.554666+0000","last_became_peered":"2024-03-18T19:03:02.554666+0000","last_unstale":"2024-03-18T19:03:03.542736+0000","last_undegraded":"2024-03-18T19:03:03.542736+0000","last_fullsized":"2024-03-18T19:03:03.542736+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:17:13.986294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389138+0000","last_change":"2024-03-18T19:03:02.556576+0000","last_active":"2024-03-18T19:03:04.389138+0000","last_peered":"2024-03-18T19:03:04.389138+0000","last_clean":"2024-03-18T19:03:04.389138+0000","last_became_active":"2024-03-18T19:03:02.556284+0000","last_became_peered":"2024-03-18T19:03:02.556284+0000","last_unstale":"2024-03-18T19:03:04.389138+0000","last_undegraded":"2024-03-18T19:03:04.389138+0000","last_fullsized":"2024-03-18T19:03:04.389138+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:50:28.343435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388093+0000","last_change":"2024-03-18T19:03:02.568865+0000","last_active":"2024-03-18T19:03:04.388093+0000","last_peered":"2024-03-18T19:03:04.388093+0000","last_clean":"2024-03-18T19:03:04.388093+0000","last_became_active":"2024-03-18T19:03:02.565363+0000","last_became_peered":"2024-03-18T19:03:02.565363+0000","last_unstale":"2024-03-18T19:03:04.388093+0000","last_undegraded":"2024-03-18T19:03:04.388093+0000","last_fullsized":"2024-03-18T19:03:04.388093+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:04:49.333141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388466+0000","last_change":"2024-03-18T19:03:02.555824+0000","last_active":"2024-03-18T19:03:04.388466+0000","last_peered":"2024-03-18T19:03:04.388466+0000","last_clean":"2024-03-18T19:03:04.388466+0000","last_became_active":"2024-03-18T19:03:02.555556+0000","last_became_peered":"2024-03-18T19:03:02.555556+0000","last_unstale":"2024-03-18T19:03:04.388466+0000","last_undegraded":"2024-03-18T19:03:04.388466+0000","last_fullsized":"2024-03-18T19:03:04.388466+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:44:42.648920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389081+0000","last_change":"2024-03-18T19:03:02.551894+0000","last_active":"2024-03-18T19:03:04.389081+0000","last_peered":"2024-03-18T19:03:04.389081+0000","last_clean":"2024-03-18T19:03:04.389081+0000","last_became_active":"2024-03-18T19:03:02.551618+0000","last_became_peered":"2024-03-18T19:03:02.551618+0000","last_unstale":"2024-03-18T19:03:04.389081+0000","last_undegraded":"2024-03-18T19:03:04.389081+0000","last_fullsized":"2024-03-18T19:03:04.389081+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:55:54.064220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542664+0000","last_change":"2024-03-18T19:03:02.556400+0000","last_active":"2024-03-18T19:03:03.542664+0000","last_peered":"2024-03-18T19:03:03.542664+0000","last_clean":"2024-03-18T19:03:03.542664+0000","last_became_active":"2024-03-18T19:03:02.556006+0000","last_became_peered":"2024-03-18T19:03:02.556006+0000","last_unstale":"2024-03-18T19:03:03.542664+0000","last_undegraded":"2024-03-18T19:03:03.542664+0000","last_fullsized":"2024-03-18T19:03:03.542664+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:09.977954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387695+0000","last_change":"2024-03-18T19:03:02.560247+0000","last_active":"2024-03-18T19:03:04.387695+0000","last_peered":"2024-03-18T19:03:04.387695+0000","last_clean":"2024-03-18T19:03:04.387695+0000","last_became_active":"2024-03-18T19:03:02.556601+0000","last_became_peered":"2024-03-18T19:03:02.556601+0000","last_unstale":"2024-03-18T19:03:04.387695+0000","last_undegraded":"2024-03-18T19:03:04.387695+0000","last_fullsized":"2024-03-18T19:03:04.387695+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:44:42.149005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387836+0000","last_change":"2024-03-18T19:03:02.558337+0000","last_active":"2024-03-18T19:03:04.387836+0000","last_peered":"2024-03-18T19:03:04.387836+0000","last_clean":"2024-03-18T19:03:04.387836+0000","last_became_active":"2024-03-18T19:03:02.558083+0000","last_became_peered":"2024-03-18T19:03:02.558083+0000","last_unstale":"2024-03-18T19:03:04.387836+0000","last_undegraded":"2024-03-18T19:03:04.387836+0000","last_fullsized":"2024-03-18T19:03:04.387836+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:13.001203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388335+0000","last_change":"2024-03-18T19:03:02.568191+0000","last_active":"2024-03-18T19:03:04.388335+0000","last_peered":"2024-03-18T19:03:04.388335+0000","last_clean":"2024-03-18T19:03:04.388335+0000","last_became_active":"2024-03-18T19:03:02.567991+0000","last_became_peered":"2024-03-18T19:03:02.567991+0000","last_unstale":"2024-03-18T19:03:04.388335+0000","last_undegraded":"2024-03-18T19:03:04.388335+0000","last_fullsized":"2024-03-18T19:03:04.388335+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:36:45.414625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.543017+0000","last_change":"2024-03-18T19:03:02.554474+0000","last_active":"2024-03-18T19:03:03.543017+0000","last_peered":"2024-03-18T19:03:03.543017+0000","last_clean":"2024-03-18T19:03:03.543017+0000","last_became_active":"2024-03-18T19:03:02.554214+0000","last_became_peered":"2024-03-18T19:03:02.554214+0000","last_unstale":"2024-03-18T19:03:03.543017+0000","last_undegraded":"2024-03-18T19:03:03.543017+0000","last_fullsized":"2024-03-18T19:03:03.543017+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:38:47.495608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.544033+0000","last_change":"2024-03-18T19:03:02.560070+0000","last_active":"2024-03-18T19:03:03.544033+0000","last_peered":"2024-03-18T19:03:03.544033+0000","last_clean":"2024-03-18T19:03:03.544033+0000","last_became_active":"2024-03-18T19:03:02.559856+0000","last_became_peered":"2024-03-18T19:03:02.559856+0000","last_unstale":"2024-03-18T19:03:03.544033+0000","last_undegraded":"2024-03-18T19:03:03.544033+0000","last_fullsized":"2024-03-18T19:03:03.544033+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:05.040442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388126+0000","last_change":"2024-03-18T19:03:02.551750+0000","last_active":"2024-03-18T19:03:04.388126+0000","last_peered":"2024-03-18T19:03:04.388126+0000","last_clean":"2024-03-18T19:03:04.388126+0000","last_became_active":"2024-03-18T19:03:02.551475+0000","last_became_peered":"2024-03-18T19:03:02.551475+0000","last_unstale":"2024-03-18T19:03:04.388126+0000","last_undegraded":"2024-03-18T19:03:04.388126+0000","last_fullsized":"2024-03-18T19:03:04.388126+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:03.437717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542502+0000","last_change":"2024-03-18T19:03:02.547246+0000","last_active":"2024-03-18T19:03:03.542502+0000","last_peered":"2024-03-18T19:03:03.542502+0000","last_clean":"2024-03-18T19:03:03.542502+0000","last_became_active":"2024-03-18T19:03:02.547042+0000","last_became_peered":"2024-03-18T19:03:02.547042+0000","last_unstale":"2024-03-18T19:03:03.542502+0000","last_undegraded":"2024-03-18T19:03:03.542502+0000","last_fullsized":"2024-03-18T19:03:03.542502+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:16:29.862517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389006+0000","last_change":"2024-03-18T19:03:02.563359+0000","last_active":"2024-03-18T19:03:04.389006+0000","last_peered":"2024-03-18T19:03:04.389006+0000","last_clean":"2024-03-18T19:03:04.389006+0000","last_became_active":"2024-03-18T19:03:02.562885+0000","last_became_peered":"2024-03-18T19:03:02.562885+0000","last_unstale":"2024-03-18T19:03:04.389006+0000","last_undegraded":"2024-03-18T19:03:04.389006+0000","last_fullsized":"2024-03-18T19:03:04.389006+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:51:15.238139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388716+0000","last_change":"2024-03-18T19:03:02.556653+0000","last_active":"2024-03-18T19:03:04.388716+0000","last_peered":"2024-03-18T19:03:04.388716+0000","last_clean":"2024-03-18T19:03:04.388716+0000","last_became_active":"2024-03-18T19:03:02.553397+0000","last_became_peered":"2024-03-18T19:03:02.553397+0000","last_unstale":"2024-03-18T19:03:04.388716+0000","last_undegraded":"2024-03-18T19:03:04.388716+0000","last_fullsized":"2024-03-18T19:03:04.388716+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:11:14.319814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542463+0000","last_change":"2024-03-18T19:03:02.546658+0000","last_active":"2024-03-18T19:03:03.542463+0000","last_peered":"2024-03-18T19:03:03.542463+0000","last_clean":"2024-03-18T19:03:03.542463+0000","last_became_active":"2024-03-18T19:03:02.545109+0000","last_became_peered":"2024-03-18T19:03:02.545109+0000","last_unstale":"2024-03-18T19:03:03.542463+0000","last_undegraded":"2024-03-18T19:03:03.542463+0000","last_fullsized":"2024-03-18T19:03:03.542463+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:57:33.441885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388964+0000","last_change":"2024-03-18T19:03:02.566788+0000","last_active":"2024-03-18T19:03:04.388964+0000","last_peered":"2024-03-18T19:03:04.388964+0000","last_clean":"2024-03-18T19:03:04.388964+0000","last_became_active":"2024-03-18T19:03:02.566360+0000","last_became_peered":"2024-03-18T19:03:02.566360+0000","last_unstale":"2024-03-18T19:03:04.388964+0000","last_undegraded":"2024-03-18T19:03:04.388964+0000","last_fullsized":"2024-03-18T19:03:04.388964+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:11:17.817327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389247+0000","last_change":"2024-03-18T19:03:02.554052+0000","last_active":"2024-03-18T19:03:04.389247+0000","last_peered":"2024-03-18T19:03:04.389247+0000","last_clean":"2024-03-18T19:03:04.389247+0000","last_became_active":"2024-03-18T19:03:02.553814+0000","last_became_peered":"2024-03-18T19:03:02.553814+0000","last_unstale":"2024-03-18T19:03:04.389247+0000","last_undegraded":"2024-03-18T19:03:04.389247+0000","last_fullsized":"2024-03-18T19:03:04.389247+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:09:57.608685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388206+0000","last_change":"2024-03-18T19:03:02.565420+0000","last_active":"2024-03-18T19:03:04.388206+0000","last_peered":"2024-03-18T19:03:04.388206+0000","last_clean":"2024-03-18T19:03:04.388206+0000","last_became_active":"2024-03-18T19:03:02.565101+0000","last_became_peered":"2024-03-18T19:03:02.565101+0000","last_unstale":"2024-03-18T19:03:04.388206+0000","last_undegraded":"2024-03-18T19:03:04.388206+0000","last_fullsized":"2024-03-18T19:03:04.388206+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:23:58.064544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389367+0000","last_change":"2024-03-18T19:03:02.544788+0000","last_active":"2024-03-18T19:03:04.389367+0000","last_peered":"2024-03-18T19:03:04.389367+0000","last_clean":"2024-03-18T19:03:04.389367+0000","last_became_active":"2024-03-18T19:03:02.544374+0000","last_became_peered":"2024-03-18T19:03:02.544374+0000","last_unstale":"2024-03-18T19:03:04.389367+0000","last_undegraded":"2024-03-18T19:03:04.389367+0000","last_fullsized":"2024-03-18T19:03:04.389367+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:28:54.390651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388535+0000","last_change":"2024-03-18T19:03:02.548724+0000","last_active":"2024-03-18T19:03:04.388535+0000","last_peered":"2024-03-18T19:03:04.388535+0000","last_clean":"2024-03-18T19:03:04.388535+0000","last_became_active":"2024-03-18T19:03:02.547985+0000","last_became_peered":"2024-03-18T19:03:02.547985+0000","last_unstale":"2024-03-18T19:03:04.388535+0000","last_undegraded":"2024-03-18T19:03:04.388535+0000","last_fullsized":"2024-03-18T19:03:04.388535+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:04:06.001910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387809+0000","last_change":"2024-03-18T19:03:02.566926+0000","last_active":"2024-03-18T19:03:04.387809+0000","last_peered":"2024-03-18T19:03:04.387809+0000","last_clean":"2024-03-18T19:03:04.387809+0000","last_became_active":"2024-03-18T19:03:02.566664+0000","last_became_peered":"2024-03-18T19:03:02.566664+0000","last_unstale":"2024-03-18T19:03:04.387809+0000","last_undegraded":"2024-03-18T19:03:04.387809+0000","last_fullsized":"2024-03-18T19:03:04.387809+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:11:17.817327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.543798+0000","last_change":"2024-03-18T19:03:02.561108+0000","last_active":"2024-03-18T19:03:03.543798+0000","last_peered":"2024-03-18T19:03:03.543798+0000","last_clean":"2024-03-18T19:03:03.543798+0000","last_became_active":"2024-03-18T19:03:02.560912+0000","last_became_peered":"2024-03-18T19:03:02.560912+0000","last_unstale":"2024-03-18T19:03:03.543798+0000","last_undegraded":"2024-03-18T19:03:03.543798+0000","last_fullsized":"2024-03-18T19:03:03.543798+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:17:59.732292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387554+0000","last_change":"2024-03-18T19:03:02.565651+0000","last_active":"2024-03-18T19:03:04.387554+0000","last_peered":"2024-03-18T19:03:04.387554+0000","last_clean":"2024-03-18T19:03:04.387554+0000","last_became_active":"2024-03-18T19:03:02.565445+0000","last_became_peered":"2024-03-18T19:03:02.565445+0000","last_unstale":"2024-03-18T19:03:04.387554+0000","last_undegraded":"2024-03-18T19:03:04.387554+0000","last_fullsized":"2024-03-18T19:03:04.387554+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:25.648248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388902+0000","last_change":"2024-03-18T19:03:02.566647+0000","last_active":"2024-03-18T19:03:04.388902+0000","last_peered":"2024-03-18T19:03:04.388902+0000","last_clean":"2024-03-18T19:03:04.388902+0000","last_became_active":"2024-03-18T19:03:02.566185+0000","last_became_peered":"2024-03-18T19:03:02.566185+0000","last_unstale":"2024-03-18T19:03:04.388902+0000","last_undegraded":"2024-03-18T19:03:04.388902+0000","last_fullsized":"2024-03-18T19:03:04.388902+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:36:45.414625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388188+0000","last_change":"2024-03-18T19:03:02.553778+0000","last_active":"2024-03-18T19:03:04.388188+0000","last_peered":"2024-03-18T19:03:04.388188+0000","last_clean":"2024-03-18T19:03:04.388188+0000","last_became_active":"2024-03-18T19:03:02.553510+0000","last_became_peered":"2024-03-18T19:03:02.553510+0000","last_unstale":"2024-03-18T19:03:04.388188+0000","last_undegraded":"2024-03-18T19:03:04.388188+0000","last_fullsized":"2024-03-18T19:03:04.388188+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:25:29.625002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542886+0000","last_change":"2024-03-18T19:03:02.570122+0000","last_active":"2024-03-18T19:03:03.542886+0000","last_peered":"2024-03-18T19:03:03.542886+0000","last_clean":"2024-03-18T19:03:03.542886+0000","last_became_active":"2024-03-18T19:03:02.569833+0000","last_became_peered":"2024-03-18T19:03:02.569833+0000","last_unstale":"2024-03-18T19:03:03.542886+0000","last_undegraded":"2024-03-18T19:03:03.542886+0000","last_fullsized":"2024-03-18T19:03:03.542886+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:32:11.619630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389465+0000","last_change":"2024-03-18T19:03:02.549860+0000","last_active":"2024-03-18T19:03:04.389465+0000","last_peered":"2024-03-18T19:03:04.389465+0000","last_clean":"2024-03-18T19:03:04.389465+0000","last_became_active":"2024-03-18T19:03:02.549608+0000","last_became_peered":"2024-03-18T19:03:02.549608+0000","last_unstale":"2024-03-18T19:03:04.389465+0000","last_undegraded":"2024-03-18T19:03:04.389465+0000","last_fullsized":"2024-03-18T19:03:04.389465+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:33:54.944921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387968+0000","last_change":"2024-03-18T19:03:02.566090+0000","last_active":"2024-03-18T19:03:04.387968+0000","last_peered":"2024-03-18T19:03:04.387968+0000","last_clean":"2024-03-18T19:03:04.387968+0000","last_became_active":"2024-03-18T19:03:02.565749+0000","last_became_peered":"2024-03-18T19:03:02.565749+0000","last_unstale":"2024-03-18T19:03:04.387968+0000","last_undegraded":"2024-03-18T19:03:04.387968+0000","last_fullsized":"2024-03-18T19:03:04.387968+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:00:13.013248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388466+0000","last_change":"2024-03-18T19:03:02.558410+0000","last_active":"2024-03-18T19:03:04.388466+0000","last_peered":"2024-03-18T19:03:04.388466+0000","last_clean":"2024-03-18T19:03:04.388466+0000","last_became_active":"2024-03-18T19:03:02.557103+0000","last_became_peered":"2024-03-18T19:03:02.557103+0000","last_unstale":"2024-03-18T19:03:04.388466+0000","last_undegraded":"2024-03-18T19:03:04.388466+0000","last_fullsized":"2024-03-18T19:03:04.388466+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:14.393328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387631+0000","last_change":"2024-03-18T19:03:02.563168+0000","last_active":"2024-03-18T19:03:04.387631+0000","last_peered":"2024-03-18T19:03:04.387631+0000","last_clean":"2024-03-18T19:03:04.387631+0000","last_became_active":"2024-03-18T19:03:02.562892+0000","last_became_peered":"2024-03-18T19:03:02.562892+0000","last_unstale":"2024-03-18T19:03:04.387631+0000","last_undegraded":"2024-03-18T19:03:04.387631+0000","last_fullsized":"2024-03-18T19:03:04.387631+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:25:32.860910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:03:02.098935+0000","last_change":"2024-03-18T19:03:02.098935+0000","last_active":"2024-03-18T19:03:02.098935+0000","last_peered":"2024-03-18T19:03:02.098935+0000","last_clean":"2024-03-18T19:03:02.098935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:03:02.098935+0000","last_undegraded":"2024-03-18T19:03:02.098935+0000","last_fullsized":"2024-03-18T19:03:02.098935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388257+0000","last_change":"2024-03-18T19:03:02.555892+0000","last_active":"2024-03-18T19:03:04.388257+0000","last_peered":"2024-03-18T19:03:04.388257+0000","last_clean":"2024-03-18T19:03:04.388257+0000","last_became_active":"2024-03-18T19:03:02.555689+0000","last_became_peered":"2024-03-18T19:03:02.555689+0000","last_unstale":"2024-03-18T19:03:04.388257+0000","last_undegraded":"2024-03-18T19:03:04.388257+0000","last_fullsized":"2024-03-18T19:03:04.388257+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:55:54.064220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"70'4","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389745+0000","last_change":"2024-03-18T19:03:02.120029+0000","last_active":"2024-03-18T19:03:04.389745+0000","last_peered":"2024-03-18T19:03:04.389745+0000","last_clean":"2024-03-18T19:03:04.389745+0000","last_became_active":"2024-03-18T19:03:02.119726+0000","last_became_peered":"2024-03-18T19:03:02.119726+0000","last_unstale":"2024-03-18T19:03:04.389745+0000","last_undegraded":"2024-03-18T19:03:04.389745+0000","last_fullsized":"2024-03-18T19:03:04.389745+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:57:55.947358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:03:02.098935+0000","last_change":"2024-03-18T19:03:02.098935+0000","last_active":"2024-03-18T19:03:02.098935+0000","last_peered":"2024-03-18T19:03:02.098935+0000","last_clean":"2024-03-18T19:03:02.098935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:03:02.098935+0000","last_undegraded":"2024-03-18T19:03:02.098935+0000","last_fullsized":"2024-03-18T19:03:02.098935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"70'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.543100+0000","last_change":"2024-03-18T19:03:02.118338+0000","last_active":"2024-03-18T19:03:03.543100+0000","last_peered":"2024-03-18T19:03:03.543100+0000","last_clean":"2024-03-18T19:03:03.543100+0000","last_became_active":"2024-03-18T19:03:02.117668+0000","last_became_peered":"2024-03-18T19:03:02.117668+0000","last_unstale":"2024-03-18T19:03:03.543100+0000","last_undegraded":"2024-03-18T19:03:03.543100+0000","last_fullsized":"2024-03-18T19:03:03.543100+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:41.192643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388531+0000","last_change":"2024-03-18T19:03:02.553875+0000","last_active":"2024-03-18T19:03:04.388531+0000","last_peered":"2024-03-18T19:03:04.388531+0000","last_clean":"2024-03-18T19:03:04.388531+0000","last_became_active":"2024-03-18T19:03:02.553604+0000","last_became_peered":"2024-03-18T19:03:02.553604+0000","last_unstale":"2024-03-18T19:03:04.388531+0000","last_undegraded":"2024-03-18T19:03:04.388531+0000","last_fullsized":"2024-03-18T19:03:04.388531+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:05.040442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"70'2","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542976+0000","last_change":"2024-03-18T19:03:02.115656+0000","last_active":"2024-03-18T19:03:03.542976+0000","last_peered":"2024-03-18T19:03:03.542976+0000","last_clean":"2024-03-18T19:03:03.542976+0000","last_became_active":"2024-03-18T19:03:02.115321+0000","last_became_peered":"2024-03-18T19:03:02.115321+0000","last_unstale":"2024-03-18T19:03:03.542976+0000","last_undegraded":"2024-03-18T19:03:03.542976+0000","last_fullsized":"2024-03-18T19:03:03.542976+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:22:42.820030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388431+0000","last_change":"2024-03-18T19:03:02.554279+0000","last_active":"2024-03-18T19:03:04.388431+0000","last_peered":"2024-03-18T19:03:04.388431+0000","last_clean":"2024-03-18T19:03:04.388431+0000","last_became_active":"2024-03-18T19:03:02.554031+0000","last_became_peered":"2024-03-18T19:03:02.554031+0000","last_unstale":"2024-03-18T19:03:04.388431+0000","last_undegraded":"2024-03-18T19:03:04.388431+0000","last_fullsized":"2024-03-18T19:03:04.388431+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:38:57.388604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:03:01.094095+0000","last_change":"2024-03-18T19:03:01.094095+0000","last_active":"2024-03-18T19:03:01.094095+0000","last_peered":"2024-03-18T19:03:01.094095+0000","last_clean":"2024-03-18T19:03:01.094095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:03:01.094095+0000","last_undegraded":"2024-03-18T19:03:01.094095+0000","last_fullsized":"2024-03-18T19:03:01.094095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389240+0000","last_change":"2024-03-18T19:03:02.566201+0000","last_active":"2024-03-18T19:03:04.389240+0000","last_peered":"2024-03-18T19:03:04.389240+0000","last_clean":"2024-03-18T19:03:04.389240+0000","last_became_active":"2024-03-18T19:03:02.565924+0000","last_became_peered":"2024-03-18T19:03:02.565924+0000","last_unstale":"2024-03-18T19:03:04.389240+0000","last_undegraded":"2024-03-18T19:03:04.389240+0000","last_fullsized":"2024-03-18T19:03:04.389240+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:50:28.343435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"70'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388077+0000","last_change":"2024-03-18T19:03:02.126132+0000","last_active":"2024-03-18T19:03:04.388077+0000","last_peered":"2024-03-18T19:03:04.388077+0000","last_clean":"2024-03-18T19:03:04.388077+0000","last_became_active":"2024-03-18T19:03:02.125834+0000","last_became_peered":"2024-03-18T19:03:02.125834+0000","last_unstale":"2024-03-18T19:03:04.388077+0000","last_undegraded":"2024-03-18T19:03:04.388077+0000","last_fullsized":"2024-03-18T19:03:04.388077+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:02.410889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388769+0000","last_change":"2024-03-18T19:03:02.558509+0000","last_active":"2024-03-18T19:03:04.388769+0000","last_peered":"2024-03-18T19:03:04.388769+0000","last_clean":"2024-03-18T19:03:04.388769+0000","last_became_active":"2024-03-18T19:03:02.558194+0000","last_became_peered":"2024-03-18T19:03:02.558194+0000","last_unstale":"2024-03-18T19:03:04.388769+0000","last_undegraded":"2024-03-18T19:03:04.388769+0000","last_fullsized":"2024-03-18T19:03:04.388769+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:30.094824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"70'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388771+0000","last_change":"2024-03-18T19:03:02.119225+0000","last_active":"2024-03-18T19:03:04.388771+0000","last_peered":"2024-03-18T19:03:04.388771+0000","last_clean":"2024-03-18T19:03:04.388771+0000","last_became_active":"2024-03-18T19:03:02.118974+0000","last_became_peered":"2024-03-18T19:03:02.118974+0000","last_unstale":"2024-03-18T19:03:04.388771+0000","last_undegraded":"2024-03-18T19:03:04.388771+0000","last_fullsized":"2024-03-18T19:03:04.388771+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:31:00.598479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388070+0000","last_change":"2024-03-18T19:03:02.557608+0000","last_active":"2024-03-18T19:03:04.388070+0000","last_peered":"2024-03-18T19:03:04.388070+0000","last_clean":"2024-03-18T19:03:04.388070+0000","last_became_active":"2024-03-18T19:03:02.557359+0000","last_became_peered":"2024-03-18T19:03:02.557359+0000","last_unstale":"2024-03-18T19:03:04.388070+0000","last_undegraded":"2024-03-18T19:03:04.388070+0000","last_fullsized":"2024-03-18T19:03:04.388070+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:23:19.809075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389131+0000","last_change":"2024-03-18T19:03:02.115443+0000","last_active":"2024-03-18T19:03:04.389131+0000","last_peered":"2024-03-18T19:03:04.389131+0000","last_clean":"2024-03-18T19:03:04.389131+0000","last_became_active":"2024-03-18T19:03:02.114952+0000","last_became_peered":"2024-03-18T19:03:02.114952+0000","last_unstale":"2024-03-18T19:03:04.389131+0000","last_undegraded":"2024-03-18T19:03:04.389131+0000","last_fullsized":"2024-03-18T19:03:04.389131+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:00:28.607976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389514+0000","last_change":"2024-03-18T19:03:02.569890+0000","last_active":"2024-03-18T19:03:04.389514+0000","last_peered":"2024-03-18T19:03:04.389514+0000","last_clean":"2024-03-18T19:03:04.389514+0000","last_became_active":"2024-03-18T19:03:02.569649+0000","last_became_peered":"2024-03-18T19:03:02.569649+0000","last_unstale":"2024-03-18T19:03:04.389514+0000","last_undegraded":"2024-03-18T19:03:04.389514+0000","last_fullsized":"2024-03-18T19:03:04.389514+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:37.161000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387659+0000","last_change":"2024-03-18T19:03:02.130557+0000","last_active":"2024-03-18T19:03:04.387659+0000","last_peered":"2024-03-18T19:03:04.387659+0000","last_clean":"2024-03-18T19:03:04.387659+0000","last_became_active":"2024-03-18T19:03:02.130205+0000","last_became_peered":"2024-03-18T19:03:02.130205+0000","last_unstale":"2024-03-18T19:03:04.387659+0000","last_undegraded":"2024-03-18T19:03:04.387659+0000","last_fullsized":"2024-03-18T19:03:04.387659+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:03.319006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389653+0000","last_change":"2024-03-18T19:03:02.556659+0000","last_active":"2024-03-18T19:03:04.389653+0000","last_peered":"2024-03-18T19:03:04.389653+0000","last_clean":"2024-03-18T19:03:04.389653+0000","last_became_active":"2024-03-18T19:03:02.556345+0000","last_became_peered":"2024-03-18T19:03:02.556345+0000","last_unstale":"2024-03-18T19:03:04.389653+0000","last_undegraded":"2024-03-18T19:03:04.389653+0000","last_fullsized":"2024-03-18T19:03:04.389653+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:09:57.608685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389575+0000","last_change":"2024-03-18T19:03:02.117375+0000","last_active":"2024-03-18T19:03:04.389575+0000","last_peered":"2024-03-18T19:03:04.389575+0000","last_clean":"2024-03-18T19:03:04.389575+0000","last_became_active":"2024-03-18T19:03:02.117060+0000","last_became_peered":"2024-03-18T19:03:02.117060+0000","last_unstale":"2024-03-18T19:03:04.389575+0000","last_undegraded":"2024-03-18T19:03:04.389575+0000","last_fullsized":"2024-03-18T19:03:04.389575+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:17:58.727452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:03:02.098935+0000","last_change":"2024-03-18T19:03:02.098935+0000","last_active":"2024-03-18T19:03:02.098935+0000","last_peered":"2024-03-18T19:03:02.098935+0000","last_clean":"2024-03-18T19:03:02.098935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:03:02.098935+0000","last_undegraded":"2024-03-18T19:03:02.098935+0000","last_fullsized":"2024-03-18T19:03:02.098935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389551+0000","last_change":"2024-03-18T19:03:02.129254+0000","last_active":"2024-03-18T19:03:04.389551+0000","last_peered":"2024-03-18T19:03:04.389551+0000","last_clean":"2024-03-18T19:03:04.389551+0000","last_became_active":"2024-03-18T19:03:02.128964+0000","last_became_peered":"2024-03-18T19:03:02.128964+0000","last_unstale":"2024-03-18T19:03:04.389551+0000","last_undegraded":"2024-03-18T19:03:04.389551+0000","last_fullsized":"2024-03-18T19:03:04.389551+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:38:32.454783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389063+0000","last_change":"2024-03-18T19:03:02.563166+0000","last_active":"2024-03-18T19:03:04.389063+0000","last_peered":"2024-03-18T19:03:04.389063+0000","last_clean":"2024-03-18T19:03:04.389063+0000","last_became_active":"2024-03-18T19:03:02.562588+0000","last_became_peered":"2024-03-18T19:03:02.562588+0000","last_unstale":"2024-03-18T19:03:04.389063+0000","last_undegraded":"2024-03-18T19:03:04.389063+0000","last_fullsized":"2024-03-18T19:03:04.389063+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:05:54.935319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"70'3","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388870+0000","last_change":"2024-03-18T19:03:02.118817+0000","last_active":"2024-03-18T19:03:04.388870+0000","last_peered":"2024-03-18T19:03:04.388870+0000","last_clean":"2024-03-18T19:03:04.388870+0000","last_became_active":"2024-03-18T19:03:02.118498+0000","last_became_peered":"2024-03-18T19:03:02.118498+0000","last_unstale":"2024-03-18T19:03:04.388870+0000","last_undegraded":"2024-03-18T19:03:04.388870+0000","last_fullsized":"2024-03-18T19:03:04.388870+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:33:53.940081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389185+0000","last_change":"2024-03-18T19:03:02.550093+0000","last_active":"2024-03-18T19:03:04.389185+0000","last_peered":"2024-03-18T19:03:04.389185+0000","last_clean":"2024-03-18T19:03:04.389185+0000","last_became_active":"2024-03-18T19:03:02.549775+0000","last_became_peered":"2024-03-18T19:03:02.549775+0000","last_unstale":"2024-03-18T19:03:04.389185+0000","last_undegraded":"2024-03-18T19:03:04.389185+0000","last_fullsized":"2024-03-18T19:03:04.389185+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:44:42.648920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"70'6","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388311+0000","last_change":"2024-03-18T19:03:02.116621+0000","last_active":"2024-03-18T19:03:04.388311+0000","last_peered":"2024-03-18T19:03:04.388311+0000","last_clean":"2024-03-18T19:03:04.388311+0000","last_became_active":"2024-03-18T19:03:02.116306+0000","last_became_peered":"2024-03-18T19:03:02.116306+0000","last_unstale":"2024-03-18T19:03:04.388311+0000","last_undegraded":"2024-03-18T19:03:04.388311+0000","last_fullsized":"2024-03-18T19:03:04.388311+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:34.043005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:03:02.098935+0000","last_change":"2024-03-18T19:03:02.098935+0000","last_active":"2024-03-18T19:03:02.098935+0000","last_peered":"2024-03-18T19:03:02.098935+0000","last_clean":"2024-03-18T19:03:02.098935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:03:02.098935+0000","last_undegraded":"2024-03-18T19:03:02.098935+0000","last_fullsized":"2024-03-18T19:03:02.098935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"70'2","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388176+0000","last_change":"2024-03-18T19:03:02.126189+0000","last_active":"2024-03-18T19:03:04.388176+0000","last_peered":"2024-03-18T19:03:04.388176+0000","last_clean":"2024-03-18T19:03:04.388176+0000","last_became_active":"2024-03-18T19:03:02.125898+0000","last_became_peered":"2024-03-18T19:03:02.125898+0000","last_unstale":"2024-03-18T19:03:04.388176+0000","last_undegraded":"2024-03-18T19:03:04.388176+0000","last_fullsized":"2024-03-18T19:03:04.388176+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:41.197019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388313+0000","last_change":"2024-03-18T19:03:02.570155+0000","last_active":"2024-03-18T19:03:04.388313+0000","last_peered":"2024-03-18T19:03:04.388313+0000","last_clean":"2024-03-18T19:03:04.388313+0000","last_became_active":"2024-03-18T19:03:02.569886+0000","last_became_peered":"2024-03-18T19:03:02.569886+0000","last_unstale":"2024-03-18T19:03:04.388313+0000","last_undegraded":"2024-03-18T19:03:04.388313+0000","last_fullsized":"2024-03-18T19:03:04.388313+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:33.527267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388624+0000","last_change":"2024-03-18T19:03:02.131089+0000","last_active":"2024-03-18T19:03:04.388624+0000","last_peered":"2024-03-18T19:03:04.388624+0000","last_clean":"2024-03-18T19:03:04.388624+0000","last_became_active":"2024-03-18T19:03:02.130777+0000","last_became_peered":"2024-03-18T19:03:02.130777+0000","last_unstale":"2024-03-18T19:03:04.388624+0000","last_undegraded":"2024-03-18T19:03:04.388624+0000","last_fullsized":"2024-03-18T19:03:04.388624+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:46.788609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387923+0000","last_change":"2024-03-18T19:03:02.561595+0000","last_active":"2024-03-18T19:03:04.387923+0000","last_peered":"2024-03-18T19:03:04.387923+0000","last_clean":"2024-03-18T19:03:04.387923+0000","last_became_active":"2024-03-18T19:03:02.561387+0000","last_became_peered":"2024-03-18T19:03:02.561387+0000","last_unstale":"2024-03-18T19:03:04.387923+0000","last_undegraded":"2024-03-18T19:03:04.387923+0000","last_fullsized":"2024-03-18T19:03:04.387923+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:42.647657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388807+0000","last_change":"2024-03-18T19:03:02.120516+0000","last_active":"2024-03-18T19:03:04.388807+0000","last_peered":"2024-03-18T19:03:04.388807+0000","last_clean":"2024-03-18T19:03:04.388807+0000","last_became_active":"2024-03-18T19:03:02.120202+0000","last_became_peered":"2024-03-18T19:03:02.120202+0000","last_unstale":"2024-03-18T19:03:04.388807+0000","last_undegraded":"2024-03-18T19:03:04.388807+0000","last_fullsized":"2024-03-18T19:03:04.388807+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:04.035601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388843+0000","last_change":"2024-03-18T19:03:02.566622+0000","last_active":"2024-03-18T19:03:04.388843+0000","last_peered":"2024-03-18T19:03:04.388843+0000","last_clean":"2024-03-18T19:03:04.388843+0000","last_became_active":"2024-03-18T19:03:02.566050+0000","last_became_peered":"2024-03-18T19:03:02.566050+0000","last_unstale":"2024-03-18T19:03:04.388843+0000","last_undegraded":"2024-03-18T19:03:04.388843+0000","last_fullsized":"2024-03-18T19:03:04.388843+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:59.912019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"70'8","reported_seq":19,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388087+0000","last_change":"2024-03-18T19:03:02.117083+0000","last_active":"2024-03-18T19:03:04.388087+0000","last_peered":"2024-03-18T19:03:04.388087+0000","last_clean":"2024-03-18T19:03:04.388087+0000","last_became_active":"2024-03-18T19:03:02.116814+0000","last_became_peered":"2024-03-18T19:03:02.116814+0000","last_unstale":"2024-03-18T19:03:04.388087+0000","last_undegraded":"2024-03-18T19:03:04.388087+0000","last_fullsized":"2024-03-18T19:03:04.388087+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:58:02.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387881+0000","last_change":"2024-03-18T19:03:02.555576+0000","last_active":"2024-03-18T19:03:04.387881+0000","last_peered":"2024-03-18T19:03:04.387881+0000","last_clean":"2024-03-18T19:03:04.387881+0000","last_became_active":"2024-03-18T19:03:02.555250+0000","last_became_peered":"2024-03-18T19:03:02.555250+0000","last_unstale":"2024-03-18T19:03:04.387881+0000","last_undegraded":"2024-03-18T19:03:04.387881+0000","last_fullsized":"2024-03-18T19:03:04.387881+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:25.648248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"70'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388736+0000","last_change":"2024-03-18T19:03:02.132934+0000","last_active":"2024-03-18T19:03:04.388736+0000","last_peered":"2024-03-18T19:03:04.388736+0000","last_clean":"2024-03-18T19:03:04.388736+0000","last_became_active":"2024-03-18T19:03:02.132612+0000","last_became_peered":"2024-03-18T19:03:02.132612+0000","last_unstale":"2024-03-18T19:03:04.388736+0000","last_undegraded":"2024-03-18T19:03:04.388736+0000","last_fullsized":"2024-03-18T19:03:04.388736+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:18:31.958318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542540+0000","last_change":"2024-03-18T19:03:02.566715+0000","last_active":"2024-03-18T19:03:03.542540+0000","last_peered":"2024-03-18T19:03:03.542540+0000","last_clean":"2024-03-18T19:03:03.542540+0000","last_became_active":"2024-03-18T19:03:02.566443+0000","last_became_peered":"2024-03-18T19:03:02.566443+0000","last_unstale":"2024-03-18T19:03:03.542540+0000","last_undegraded":"2024-03-18T19:03:03.542540+0000","last_fullsized":"2024-03-18T19:03:03.542540+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:35:22.527659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"70'2","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.544134+0000","last_change":"2024-03-18T19:03:02.118090+0000","last_active":"2024-03-18T19:03:03.544134+0000","last_peered":"2024-03-18T19:03:03.544134+0000","last_clean":"2024-03-18T19:03:03.544134+0000","last_became_active":"2024-03-18T19:03:02.117568+0000","last_became_peered":"2024-03-18T19:03:02.117568+0000","last_unstale":"2024-03-18T19:03:03.544134+0000","last_undegraded":"2024-03-18T19:03:03.544134+0000","last_fullsized":"2024-03-18T19:03:03.544134+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:40:43.461004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388672+0000","last_change":"2024-03-18T19:03:02.569748+0000","last_active":"2024-03-18T19:03:04.388672+0000","last_peered":"2024-03-18T19:03:04.388672+0000","last_clean":"2024-03-18T19:03:04.388672+0000","last_became_active":"2024-03-18T19:03:02.569515+0000","last_became_peered":"2024-03-18T19:03:02.569515+0000","last_unstale":"2024-03-18T19:03:04.388672+0000","last_undegraded":"2024-03-18T19:03:04.388672+0000","last_fullsized":"2024-03-18T19:03:04.388672+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:23:33.326062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"70'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388806+0000","last_change":"2024-03-18T19:03:02.118770+0000","last_active":"2024-03-18T19:03:04.388806+0000","last_peered":"2024-03-18T19:03:04.388806+0000","last_clean":"2024-03-18T19:03:04.388806+0000","last_became_active":"2024-03-18T19:03:02.118488+0000","last_became_peered":"2024-03-18T19:03:02.118488+0000","last_unstale":"2024-03-18T19:03:04.388806+0000","last_undegraded":"2024-03-18T19:03:04.388806+0000","last_fullsized":"2024-03-18T19:03:04.388806+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:57.060127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542604+0000","last_change":"2024-03-18T19:03:02.554925+0000","last_active":"2024-03-18T19:03:03.542604+0000","last_peered":"2024-03-18T19:03:03.542604+0000","last_clean":"2024-03-18T19:03:03.542604+0000","last_became_active":"2024-03-18T19:03:02.554617+0000","last_became_peered":"2024-03-18T19:03:02.554617+0000","last_unstale":"2024-03-18T19:03:03.542604+0000","last_undegraded":"2024-03-18T19:03:03.542604+0000","last_fullsized":"2024-03-18T19:03:03.542604+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:09:39.260374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389641+0000","last_change":"2024-03-18T19:03:02.122086+0000","last_active":"2024-03-18T19:03:04.389641+0000","last_peered":"2024-03-18T19:03:04.389641+0000","last_clean":"2024-03-18T19:03:04.389641+0000","last_became_active":"2024-03-18T19:03:02.121891+0000","last_became_peered":"2024-03-18T19:03:02.121891+0000","last_unstale":"2024-03-18T19:03:04.389641+0000","last_undegraded":"2024-03-18T19:03:04.389641+0000","last_fullsized":"2024-03-18T19:03:04.389641+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:18:15.263661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:03:02.098935+0000","last_change":"2024-03-18T19:03:02.098935+0000","last_active":"2024-03-18T19:03:02.098935+0000","last_peered":"2024-03-18T19:03:02.098935+0000","last_clean":"2024-03-18T19:03:02.098935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:03:02.098935+0000","last_undegraded":"2024-03-18T19:03:02.098935+0000","last_fullsized":"2024-03-18T19:03:02.098935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"70'2","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542698+0000","last_change":"2024-03-18T19:03:02.116810+0000","last_active":"2024-03-18T19:03:03.542698+0000","last_peered":"2024-03-18T19:03:03.542698+0000","last_clean":"2024-03-18T19:03:03.542698+0000","last_became_active":"2024-03-18T19:03:02.116512+0000","last_became_peered":"2024-03-18T19:03:02.116512+0000","last_unstale":"2024-03-18T19:03:03.542698+0000","last_undegraded":"2024-03-18T19:03:03.542698+0000","last_fullsized":"2024-03-18T19:03:03.542698+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:31:00.598479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388606+0000","last_change":"2024-03-18T19:03:02.544303+0000","last_active":"2024-03-18T19:03:04.388606+0000","last_peered":"2024-03-18T19:03:04.388606+0000","last_clean":"2024-03-18T19:03:04.388606+0000","last_became_active":"2024-03-18T19:03:02.543821+0000","last_became_peered":"2024-03-18T19:03:02.543821+0000","last_unstale":"2024-03-18T19:03:04.388606+0000","last_undegraded":"2024-03-18T19:03:04.388606+0000","last_fullsized":"2024-03-18T19:03:04.388606+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:55:45.622276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387922+0000","last_change":"2024-03-18T19:03:02.123481+0000","last_active":"2024-03-18T19:03:04.387922+0000","last_peered":"2024-03-18T19:03:04.387922+0000","last_clean":"2024-03-18T19:03:04.387922+0000","last_became_active":"2024-03-18T19:03:02.123245+0000","last_became_peered":"2024-03-18T19:03:02.123245+0000","last_unstale":"2024-03-18T19:03:04.387922+0000","last_undegraded":"2024-03-18T19:03:04.387922+0000","last_fullsized":"2024-03-18T19:03:04.387922+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:42:17.566346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388519+0000","last_change":"2024-03-18T19:03:02.565460+0000","last_active":"2024-03-18T19:03:04.388519+0000","last_peered":"2024-03-18T19:03:04.388519+0000","last_clean":"2024-03-18T19:03:04.388519+0000","last_became_active":"2024-03-18T19:03:02.565184+0000","last_became_peered":"2024-03-18T19:03:02.565184+0000","last_unstale":"2024-03-18T19:03:04.388519+0000","last_undegraded":"2024-03-18T19:03:04.388519+0000","last_fullsized":"2024-03-18T19:03:04.388519+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:22:11.304687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388409+0000","last_change":"2024-03-18T19:03:02.129792+0000","last_active":"2024-03-18T19:03:04.388409+0000","last_peered":"2024-03-18T19:03:04.388409+0000","last_clean":"2024-03-18T19:03:04.388409+0000","last_became_active":"2024-03-18T19:03:02.129485+0000","last_became_peered":"2024-03-18T19:03:02.129485+0000","last_unstale":"2024-03-18T19:03:04.388409+0000","last_undegraded":"2024-03-18T19:03:04.388409+0000","last_fullsized":"2024-03-18T19:03:04.388409+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:34.043005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542836+0000","last_change":"2024-03-18T19:03:02.556628+0000","last_active":"2024-03-18T19:03:03.542836+0000","last_peered":"2024-03-18T19:03:03.542836+0000","last_clean":"2024-03-18T19:03:03.542836+0000","last_became_active":"2024-03-18T19:03:02.556376+0000","last_became_peered":"2024-03-18T19:03:02.556376+0000","last_unstale":"2024-03-18T19:03:03.542836+0000","last_undegraded":"2024-03-18T19:03:03.542836+0000","last_fullsized":"2024-03-18T19:03:03.542836+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:42.201860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.543725+0000","last_change":"2024-03-18T19:03:02.120610+0000","last_active":"2024-03-18T19:03:03.543725+0000","last_peered":"2024-03-18T19:03:03.543725+0000","last_clean":"2024-03-18T19:03:03.543725+0000","last_became_active":"2024-03-18T19:03:02.120106+0000","last_became_peered":"2024-03-18T19:03:02.120106+0000","last_unstale":"2024-03-18T19:03:03.543725+0000","last_undegraded":"2024-03-18T19:03:03.543725+0000","last_fullsized":"2024-03-18T19:03:03.543725+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:46.762602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:03:02.098935+0000","last_change":"2024-03-18T19:03:02.098935+0000","last_active":"2024-03-18T19:03:02.098935+0000","last_peered":"2024-03-18T19:03:02.098935+0000","last_clean":"2024-03-18T19:03:02.098935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:03:02.098935+0000","last_undegraded":"2024-03-18T19:03:02.098935+0000","last_fullsized":"2024-03-18T19:03:02.098935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"70'2","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389800+0000","last_change":"2024-03-18T19:03:02.132529+0000","last_active":"2024-03-18T19:03:04.389800+0000","last_peered":"2024-03-18T19:03:04.389800+0000","last_clean":"2024-03-18T19:03:04.389800+0000","last_became_active":"2024-03-18T19:03:02.132254+0000","last_became_peered":"2024-03-18T19:03:02.132254+0000","last_unstale":"2024-03-18T19:03:04.389800+0000","last_undegraded":"2024-03-18T19:03:04.389800+0000","last_fullsized":"2024-03-18T19:03:04.389800+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:40.355937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542748+0000","last_change":"2024-03-18T19:03:02.549176+0000","last_active":"2024-03-18T19:03:03.542748+0000","last_peered":"2024-03-18T19:03:03.542748+0000","last_clean":"2024-03-18T19:03:03.542748+0000","last_became_active":"2024-03-18T19:03:02.548876+0000","last_became_peered":"2024-03-18T19:03:02.548876+0000","last_unstale":"2024-03-18T19:03:03.542748+0000","last_undegraded":"2024-03-18T19:03:03.542748+0000","last_fullsized":"2024-03-18T19:03:03.542748+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:46:24.309467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388960+0000","last_change":"2024-03-18T19:03:02.119813+0000","last_active":"2024-03-18T19:03:04.388960+0000","last_peered":"2024-03-18T19:03:04.388960+0000","last_clean":"2024-03-18T19:03:04.388960+0000","last_became_active":"2024-03-18T19:03:02.119573+0000","last_became_peered":"2024-03-18T19:03:02.119573+0000","last_unstale":"2024-03-18T19:03:04.388960+0000","last_undegraded":"2024-03-18T19:03:04.388960+0000","last_fullsized":"2024-03-18T19:03:04.388960+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:04:04.997069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388534+0000","last_change":"2024-03-18T19:03:02.547735+0000","last_active":"2024-03-18T19:03:04.388534+0000","last_peered":"2024-03-18T19:03:04.388534+0000","last_clean":"2024-03-18T19:03:04.388534+0000","last_became_active":"2024-03-18T19:03:02.547480+0000","last_became_peered":"2024-03-18T19:03:02.547480+0000","last_unstale":"2024-03-18T19:03:04.388534+0000","last_undegraded":"2024-03-18T19:03:04.388534+0000","last_fullsized":"2024-03-18T19:03:04.388534+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:33.527267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388337+0000","last_change":"2024-03-18T19:03:02.130993+0000","last_active":"2024-03-18T19:03:04.388337+0000","last_peered":"2024-03-18T19:03:04.388337+0000","last_clean":"2024-03-18T19:03:04.388337+0000","last_became_active":"2024-03-18T19:03:02.130714+0000","last_became_peered":"2024-03-18T19:03:02.130714+0000","last_unstale":"2024-03-18T19:03:04.388337+0000","last_undegraded":"2024-03-18T19:03:04.388337+0000","last_fullsized":"2024-03-18T19:03:04.388337+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:42:17.566346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388398+0000","last_change":"2024-03-18T19:03:02.573329+0000","last_active":"2024-03-18T19:03:04.388398+0000","last_peered":"2024-03-18T19:03:04.388398+0000","last_clean":"2024-03-18T19:03:04.388398+0000","last_became_active":"2024-03-18T19:03:02.573047+0000","last_became_peered":"2024-03-18T19:03:02.573047+0000","last_unstale":"2024-03-18T19:03:04.388398+0000","last_undegraded":"2024-03-18T19:03:04.388398+0000","last_fullsized":"2024-03-18T19:03:04.388398+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:00:56.747749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388619+0000","last_change":"2024-03-18T19:03:02.121344+0000","last_active":"2024-03-18T19:03:04.388619+0000","last_peered":"2024-03-18T19:03:04.388619+0000","last_clean":"2024-03-18T19:03:04.388619+0000","last_became_active":"2024-03-18T19:03:02.121071+0000","last_became_peered":"2024-03-18T19:03:02.121071+0000","last_unstale":"2024-03-18T19:03:04.388619+0000","last_undegraded":"2024-03-18T19:03:04.388619+0000","last_fullsized":"2024-03-18T19:03:04.388619+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:02.410889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:03:02.098935+0000","last_change":"2024-03-18T19:03:02.098935+0000","last_active":"2024-03-18T19:03:02.098935+0000","last_peered":"2024-03-18T19:03:02.098935+0000","last_clean":"2024-03-18T19:03:02.098935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:03:02.098935+0000","last_undegraded":"2024-03-18T19:03:02.098935+0000","last_fullsized":"2024-03-18T19:03:02.098935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388240+0000","last_change":"2024-03-18T19:03:02.119509+0000","last_active":"2024-03-18T19:03:04.388240+0000","last_peered":"2024-03-18T19:03:04.388240+0000","last_clean":"2024-03-18T19:03:04.388240+0000","last_became_active":"2024-03-18T19:03:02.119255+0000","last_became_peered":"2024-03-18T19:03:02.119255+0000","last_unstale":"2024-03-18T19:03:04.388240+0000","last_undegraded":"2024-03-18T19:03:04.388240+0000","last_fullsized":"2024-03-18T19:03:04.388240+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:18:31.958318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388466+0000","last_change":"2024-03-18T19:03:02.565481+0000","last_active":"2024-03-18T19:03:04.388466+0000","last_peered":"2024-03-18T19:03:04.388466+0000","last_clean":"2024-03-18T19:03:04.388466+0000","last_became_active":"2024-03-18T19:03:02.565211+0000","last_became_peered":"2024-03-18T19:03:02.565211+0000","last_unstale":"2024-03-18T19:03:04.388466+0000","last_undegraded":"2024-03-18T19:03:04.388466+0000","last_fullsized":"2024-03-18T19:03:04.388466+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:38:57.388604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389732+0000","last_change":"2024-03-18T19:03:02.125309+0000","last_active":"2024-03-18T19:03:04.389732+0000","last_peered":"2024-03-18T19:03:04.389732+0000","last_clean":"2024-03-18T19:03:04.389732+0000","last_became_active":"2024-03-18T19:03:02.125056+0000","last_became_peered":"2024-03-18T19:03:02.125056+0000","last_unstale":"2024-03-18T19:03:04.389732+0000","last_undegraded":"2024-03-18T19:03:04.389732+0000","last_fullsized":"2024-03-18T19:03:04.389732+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:50:16.410240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388356+0000","last_change":"2024-03-18T19:03:02.553667+0000","last_active":"2024-03-18T19:03:04.388356+0000","last_peered":"2024-03-18T19:03:04.388356+0000","last_clean":"2024-03-18T19:03:04.388356+0000","last_became_active":"2024-03-18T19:03:02.553399+0000","last_became_peered":"2024-03-18T19:03:02.553399+0000","last_unstale":"2024-03-18T19:03:04.388356+0000","last_undegraded":"2024-03-18T19:03:04.388356+0000","last_fullsized":"2024-03-18T19:03:04.388356+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:25.431926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:03:01.094095+0000","last_change":"2024-03-18T19:03:01.094095+0000","last_active":"2024-03-18T19:03:01.094095+0000","last_peered":"2024-03-18T19:03:01.094095+0000","last_clean":"2024-03-18T19:03:01.094095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:03:01.094095+0000","last_undegraded":"2024-03-18T19:03:01.094095+0000","last_fullsized":"2024-03-18T19:03:01.094095+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389036+0000","last_change":"2024-03-18T19:03:02.554918+0000","last_active":"2024-03-18T19:03:04.389036+0000","last_peered":"2024-03-18T19:03:04.389036+0000","last_clean":"2024-03-18T19:03:04.389036+0000","last_became_active":"2024-03-18T19:03:02.554578+0000","last_became_peered":"2024-03-18T19:03:02.554578+0000","last_unstale":"2024-03-18T19:03:04.389036+0000","last_undegraded":"2024-03-18T19:03:04.389036+0000","last_fullsized":"2024-03-18T19:03:04.389036+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:45:08.631461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388594+0000","last_change":"2024-03-18T19:03:02.127347+0000","last_active":"2024-03-18T19:03:04.388594+0000","last_peered":"2024-03-18T19:03:04.388594+0000","last_clean":"2024-03-18T19:03:04.388594+0000","last_became_active":"2024-03-18T19:03:02.127070+0000","last_became_peered":"2024-03-18T19:03:02.127070+0000","last_unstale":"2024-03-18T19:03:04.388594+0000","last_undegraded":"2024-03-18T19:03:04.388594+0000","last_fullsized":"2024-03-18T19:03:04.388594+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:58:02.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"60'139","reported_seq":290,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-18T19:03:02.545768+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T19:03:02.545768+0000","last_peered":"2024-03-18T19:03:02.545768+0000","last_clean":"2024-03-18T19:03:02.545768+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T19:03:02.545768+0000","last_undegraded":"2024-03-18T19:03:02.545768+0000","last_fullsized":"2024-03-18T19:03:02.545768+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36,"ondisk_log_size":36,"up":60,"acting":60,"num_store_stats":7},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":114,"acting":114,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575009,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54672,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317168,"statfs":{"total":96636764160,"available":96580780032,"internally_reserved":0,"allocated":1441792,"data_stored":1003779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.55700000000000005,"15min":0.55800000000000005},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.247},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57999999999999996,"15min":0.57099999999999995},"min":{"1min":0.372,"5min":0.318,"15min":0.26700000000000002},"max":{"1min":0.64700000000000002,"5min":0.749,"15min":0.80800000000000005},"last":0.58799999999999997}]},{"osd":1,"last update":"Mon Mar 18 19:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.32300000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.71299999999999997,"5min":0.755,"15min":1.484},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.59299999999999997,"15min":0.58399999999999996},"min":{"1min":0.40699999999999997,"5min":0.378,"15min":0.36599999999999999},"max":{"1min":0.73799999999999999,"5min":0.80000000000000004,"15min":0.92000000000000004},"last":0.54500000000000004}]},{"osd":2,"last update":"Mon Mar 18 19:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56000000000000005,"15min":0.57299999999999995},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.27800000000000002},"max":{"1min":0.77200000000000002,"5min":0.78500000000000003,"15min":1.5129999999999999},"last":0.498},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57499999999999996,"15min":0.57999999999999996},"min":{"1min":0.39100000000000001,"5min":0.36799999999999999,"15min":0.30199999999999999},"max":{"1min":0.70899999999999996,"5min":0.84199999999999997,"15min":1.536},"last":0.61199999999999999}]},{"osd":3,"last update":"Mon Mar 18 19:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.55900000000000005,"15min":0.57799999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.311},"max":{"1min":0.72099999999999997,"5min":0.79800000000000004,"15min":1.5489999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.59299999999999997,"15min":0.60299999999999998},"min":{"1min":0.41999999999999998,"5min":0.39800000000000002,"15min":0.38200000000000001},"max":{"1min":0.71899999999999997,"5min":0.81399999999999995,"15min":1.5660000000000001},"last":0.57399999999999995}]},{"osd":4,"last update":"Mon Mar 18 19:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.44900000000000001,"15min":0.45800000000000002},"min":{"1min":0.28899999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.67500000000000004,"5min":0.70799999999999996,"15min":0.77400000000000002},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.47899999999999998,"15min":0.48399999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.64300000000000002,"5min":0.70899999999999996,"15min":0.73199999999999998},"last":0.442}]},{"osd":5,"last update":"Mon Mar 18 19:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.48599999999999999},"min":{"1min":0.27100000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.64100000000000001,"5min":0.73899999999999999,"15min":1.2310000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.499,"5min":0.51200000000000001,"15min":0.52800000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.28799999999999998},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":1.1220000000000001},"last":0.37}]},{"osd":6,"last update":"Mon Mar 18 19:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.55000000000000004},"min":{"1min":0.38100000000000001,"5min":0.376,"15min":0.34300000000000003},"max":{"1min":0.752,"5min":0.752,"15min":1.069},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52600000000000002,"15min":0.54400000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.72799999999999998,"5min":0.752,"15min":0.79600000000000004},"last":0.38800000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575009,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31956,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339884,"statfs":{"total":96636764160,"available":96604041216,"internally_reserved":0,"allocated":860160,"data_stored":420559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56799999999999995,"15min":0.58199999999999996},"min":{"1min":0.46500000000000002,"5min":0.28499999999999998,"15min":0.25700000000000001},"max":{"1min":0.73499999999999999,"5min":0.76800000000000002,"15min":3.274},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56799999999999995,"15min":0.59599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36299999999999999,"15min":0.27400000000000002},"max":{"1min":0.72599999999999998,"5min":0.78700000000000003,"15min":3.3929999999999998},"last":0.443}]},{"osd":1,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56599999999999995,"15min":0.55900000000000005},"min":{"1min":0.438,"5min":0.316,"15min":0.30599999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.94099999999999995},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57099999999999995,"15min":0.56399999999999995},"min":{"1min":0.44400000000000001,"5min":0.32800000000000001,"15min":0.311},"max":{"1min":0.66700000000000004,"5min":0.76800000000000002,"15min":0.90700000000000003},"last":0.45700000000000002}]},{"osd":2,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58299999999999996,"15min":0.56299999999999994},"min":{"1min":0.497,"5min":0.32200000000000001,"15min":0.29999999999999999},"max":{"1min":0.80300000000000005,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.496},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57599999999999996,"15min":0.58899999999999997},"min":{"1min":0.40500000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72199999999999998,"5min":0.76300000000000001,"15min":2.2370000000000001},"last":0.52000000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.41099999999999998,"5min":0.38100000000000001,"15min":0.32100000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":1.0369999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57399999999999995,"15min":0.58399999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36499999999999999},"max":{"1min":0.67800000000000005,"5min":0.79000000000000004,"15min":1.595},"last":0.50800000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.439,"15min":0.45600000000000002},"min":{"1min":0.32400000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54900000000000004,"5min":0.68700000000000006,"15min":0.76300000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.46800000000000003,"15min":0.49399999999999999},"min":{"1min":0.34899999999999998,"5min":0.32700000000000001,"15min":0.29299999999999998},"max":{"1min":0.56100000000000005,"5min":0.67800000000000005,"15min":1.8819999999999999},"last":0.38600000000000001}]},{"osd":5,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46999999999999997,"15min":0.48399999999999999},"min":{"1min":0.34200000000000003,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68700000000000006,"5min":0.73699999999999999,"15min":1.022},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.51100000000000001,"15min":0.51800000000000002},"min":{"1min":0.38100000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64700000000000002,"5min":0.70499999999999996,"15min":0.81599999999999995},"last":0.40999999999999998}]},{"osd":7,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51300000000000001,"15min":0.53400000000000003},"min":{"1min":0.376,"5min":0.34100000000000003,"15min":0.313},"max":{"1min":0.67900000000000005,"5min":0.72299999999999998,"15min":1.532},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51900000000000002,"15min":0.54500000000000004},"min":{"1min":0.376,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71299999999999997,"5min":0.79100000000000004,"15min":1.978},"last":0.36899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575009,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":860160,"data_stored":420067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.38600000000000001,"15min":0.36699999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.215},"max":{"1min":0.51500000000000001,"5min":0.60499999999999998,"15min":0.83599999999999997},"last":0.373},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.42499999999999999,"15min":0.40999999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23400000000000001},"max":{"1min":0.60199999999999998,"5min":0.68999999999999995,"15min":0.90400000000000003},"last":0.27600000000000002}]},{"osd":2,"last update":"Mon Mar 18 19:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.41799999999999998,"15min":0.39700000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.23300000000000001},"max":{"1min":0.51100000000000001,"5min":0.64000000000000001,"15min":0.88400000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.45600000000000002,"15min":0.438},"min":{"1min":0.312,"5min":0.312,"15min":0.23499999999999999},"max":{"1min":0.53900000000000003,"5min":0.70799999999999996,"15min":0.94999999999999996},"last":0.33000000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.47999999999999998,"15min":0.45600000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27700000000000002},"max":{"1min":0.63900000000000001,"5min":0.65900000000000003,"15min":0.86799999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.42999999999999999,"15min":0.41499999999999998},"min":{"1min":0.30199999999999999,"5min":0.27700000000000002,"15min":0.25600000000000001},"max":{"1min":0.57099999999999995,"5min":0.63900000000000001,"15min":0.93100000000000005},"last":0.30499999999999999}]},{"osd":4,"last update":"Mon Mar 18 19:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.65100000000000002,"15min":0.63300000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41499999999999998},"max":{"1min":0.80900000000000005,"5min":0.92600000000000005,"15min":1.052},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.65500000000000003,"15min":0.63700000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.34499999999999997},"max":{"1min":0.81599999999999995,"5min":0.96099999999999997,"15min":1.0269999999999999},"last":0.52700000000000002}]},{"osd":5,"last update":"Mon Mar 18 19:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.65600000000000003,"15min":0.64000000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.38600000000000001},"max":{"1min":0.748,"5min":0.93999999999999995,"15min":1.0760000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.67800000000000005,"15min":0.65500000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40999999999999998},"max":{"1min":0.78900000000000003,"5min":0.995,"15min":1.1319999999999999},"last":0.496}]},{"osd":6,"last update":"Mon Mar 18 19:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.65500000000000003,"15min":0.63200000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.40300000000000002},"max":{"1min":0.83999999999999997,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.68799999999999994,"15min":0.66700000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.39100000000000001},"max":{"1min":0.74399999999999999,"5min":0.97499999999999998,"15min":1.0089999999999999},"last":0.58099999999999996}]},{"osd":7,"last update":"Mon Mar 18 19:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.68000000000000005,"15min":0.65700000000000003},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.36699999999999999},"max":{"1min":0.88100000000000001,"5min":0.94799999999999995,"15min":1.0660000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.69099999999999995,"15min":0.65900000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.39900000000000002},"max":{"1min":0.871,"5min":0.94299999999999995,"15min":1.1499999999999999},"last":0.55400000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575010,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1454080,"data_stored":1010331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.39400000000000002,"15min":0.373},"min":{"1min":0.26200000000000001,"5min":0.26100000000000001,"15min":0.192},"max":{"1min":0.57399999999999995,"5min":0.621,"15min":0.621},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.42499999999999999,"15min":0.40100000000000002},"min":{"1min":0.317,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.61699999999999999,"5min":0.63200000000000001,"15min":0.66400000000000003},"last":0.441}]},{"osd":2,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.42699999999999999,"15min":0.40799999999999997},"min":{"1min":0.29499999999999998,"5min":0.22,"15min":0.20999999999999999},"max":{"1min":0.60499999999999998,"5min":0.61699999999999999,"15min":0.627},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.437,"15min":0.41999999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.626,"5min":0.626,"15min":0.64800000000000002},"last":0.39500000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.41999999999999998,"15min":0.41899999999999998},"min":{"1min":0.26900000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.56100000000000005,"5min":0.64100000000000001,"15min":5.7359999999999998},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.33900000000000002,"5min":0.307,"15min":0.27700000000000002},"max":{"1min":0.64100000000000001,"5min":0.69899999999999995,"15min":5.9329999999999998},"last":0.41399999999999998}]},{"osd":4,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65100000000000002,"15min":0.627},"min":{"1min":0.47999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.89100000000000001,"5min":1.0229999999999999,"15min":1.22},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65900000000000003,"15min":0.63100000000000001},"min":{"1min":0.48999999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.88,"5min":0.95399999999999996,"15min":1.1020000000000001},"last":0.64600000000000002}]},{"osd":5,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65800000000000003,"15min":0.64200000000000002},"min":{"1min":0.49399999999999999,"5min":0.374,"15min":0.35799999999999998},"max":{"1min":0.90500000000000003,"5min":0.98099999999999998,"15min":2.9649999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66600000000000004,"15min":0.64300000000000002},"min":{"1min":0.45300000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.84299999999999997,"5min":0.90000000000000002,"15min":1.2609999999999999},"last":0.54300000000000004}]},{"osd":6,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66400000000000003,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85299999999999998,"5min":1.0049999999999999,"15min":1.133},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67600000000000005,"15min":0.66800000000000004},"min":{"1min":0.41199999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90800000000000003,"5min":1.0369999999999999,"15min":2.9300000000000002},"last":0.57799999999999996}]},{"osd":7,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66700000000000004,"15min":0.65200000000000002},"min":{"1min":0.50600000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.88700000000000001,"5min":0.96099999999999997,"15min":1.2849999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.68500000000000005,"15min":0.66300000000000003},"min":{"1min":0.47899999999999998,"5min":0.44,"15min":0.38500000000000001},"max":{"1min":0.79800000000000004,"5min":0.96299999999999997,"15min":1.002},"last":0.60699999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575010,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58200,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94313640,"statfs":{"total":96636764160,"available":96577167360,"internally_reserved":0,"allocated":864256,"data_stored":421471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.378,"15min":0.433},"min":{"1min":0.22500000000000001,"5min":0.20799999999999999,"15min":0.17799999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":9.1219999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.46100000000000002,"15min":0.51300000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.66600000000000004,"5min":0.71799999999999997,"15min":9.1560000000000006},"last":0.64300000000000002}]},{"osd":1,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.47999999999999998,"15min":0.53000000000000003},"min":{"1min":0.29399999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":9.2460000000000004},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40799999999999997,"15min":0.48299999999999998},"min":{"1min":0.245,"5min":0.216,"15min":0.216},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":12.932},"last":0.66700000000000004}]},{"osd":3,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.42099999999999999,"15min":0.46899999999999997},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.22500000000000001},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":9.4120000000000008},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.51100000000000001,"15min":0.55700000000000005},"min":{"1min":0.34200000000000003,"5min":0.29499999999999998,"15min":0.27700000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":9.2680000000000007},"last":0.80100000000000005}]},{"osd":4,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.622,"15min":0.60799999999999998},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.85499999999999998,"5min":1,"15min":1},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.69599999999999995},"min":{"1min":0.35899999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":9.3190000000000008},"last":0.69199999999999995}]},{"osd":5,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.65300000000000002,"15min":0.70699999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.309},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":9.1940000000000008},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.65400000000000003,"15min":0.629},"min":{"1min":0.375,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.948,"5min":1.948,"15min":1.948},"last":0.92200000000000004}]},{"osd":6,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.67500000000000004,"15min":0.71399999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.9470000000000001,"5min":2.9470000000000001,"15min":9.3559999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.69099999999999995,"15min":0.749},"min":{"1min":0.48199999999999998,"5min":0.38100000000000001,"15min":0.37},"max":{"1min":0.98599999999999999,"5min":1.083,"15min":9.2219999999999995},"last":0.98599999999999999}]},{"osd":7,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65100000000000002,"15min":0.64100000000000001},"min":{"1min":0.40999999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.67500000000000004,"15min":0.72199999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.34599999999999997},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":9.3829999999999991},"last":0.76100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575010,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36056,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335784,"statfs":{"total":96636764160,"available":96599842816,"internally_reserved":0,"allocated":856064,"data_stored":414873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.40500000000000003,"15min":0.38600000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.81100000000000005,"5min":1.204,"15min":1.4550000000000001},"last":0.309},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.434,"15min":0.40100000000000002},"min":{"1min":0.29799999999999999,"5min":0.29499999999999998,"15min":0.23000000000000001},"max":{"1min":1.137,"5min":1.137,"15min":1.5169999999999999},"last":0.36399999999999999}]},{"osd":1,"last update":"Mon Mar 18 19:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.42099999999999999,"15min":0.40300000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.221},"max":{"1min":0.999,"5min":1.1220000000000001,"15min":1.2909999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.47099999999999997,"15min":0.44600000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.245},"max":{"1min":1.093,"5min":1.1080000000000001,"15min":1.355},"last":0.42799999999999999}]},{"osd":2,"last update":"Mon Mar 18 19:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.47199999999999998,"15min":0.44800000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":1.0369999999999999,"5min":1.1879999999999999,"15min":1.3160000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.28599999999999998},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.4810000000000001},"last":0.38300000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66400000000000003,"15min":0.624},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.35199999999999998},"max":{"1min":1.4219999999999999,"5min":1.871,"15min":1.871},"last":0.502},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.66300000000000003,"15min":0.63800000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0700000000000001,"5min":1.764,"15min":1.764},"last":0.45700000000000002}]},{"osd":5,"last update":"Mon Mar 18 19:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.70199999999999996,"15min":0.66400000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":2.5329999999999999,"5min":2.5329999999999999,"15min":2.5329999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.71899999999999997,"15min":0.69099999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.437},"max":{"1min":1.5409999999999999,"5min":1.794,"15min":1.794},"last":0.53500000000000003}]},{"osd":6,"last update":"Mon Mar 18 19:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.71799999999999997,"15min":0.66500000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.377},"max":{"1min":2.7389999999999999,"5min":2.7389999999999999,"15min":2.7389999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71799999999999997,"15min":0.68500000000000005},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.45400000000000001},"max":{"1min":1.159,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.56000000000000005}]},{"osd":7,"last update":"Mon Mar 18 19:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.69899999999999995,"15min":0.67800000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.3959999999999999,"5min":1.732,"15min":1.732},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.72299999999999998,"15min":0.66400000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.40500000000000003},"max":{"1min":2.7050000000000001,"5min":2.7050000000000001,"15min":2.7050000000000001},"last":0.54400000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575010,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":843776,"data_stored":413869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.58199999999999996,"15min":0.55100000000000005},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.246},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.33500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68500000000000005,"5min":0.77600000000000002,"15min":1.0860000000000001},"last":0.53400000000000003}]},{"osd":1,"last update":"Mon Mar 18 19:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.58999999999999997,"15min":0.58199999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.316},"max":{"1min":0.749,"5min":0.749,"15min":1.1679999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.57099999999999995,"15min":0.56699999999999995},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66800000000000004,"5min":0.79100000000000004,"15min":1.218},"last":0.60999999999999999}]},{"osd":2,"last update":"Mon Mar 18 19:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57399999999999995,"15min":0.56299999999999994},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.36799999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78100000000000003,"5min":0.81100000000000005,"15min":1.204},"last":0.442}]},{"osd":3,"last update":"Mon Mar 18 19:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59099999999999997,"15min":0.58299999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.755,"5min":0.82099999999999995,"15min":1.1799999999999999},"last":0.5},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.51700000000000002}]},{"osd":5,"last update":"Mon Mar 18 19:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.32000000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.78900000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.47299999999999998,"15min":0.496},"min":{"1min":0.35599999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64400000000000002,"5min":0.69699999999999995,"15min":0.77800000000000002},"last":0.55600000000000005}]},{"osd":6,"last update":"Mon Mar 18 19:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47799999999999998,"15min":0.496},"min":{"1min":0.35299999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65100000000000002,"5min":0.71999999999999997,"15min":0.81799999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.51300000000000001,"15min":0.54000000000000004},"min":{"1min":0.40300000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72499999999999998,"5min":0.77700000000000002,"15min":0.88900000000000001},"last":0.48799999999999999}]},{"osd":7,"last update":"Mon Mar 18 19:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51000000000000001,"15min":0.53100000000000003},"min":{"1min":0.40699999999999997,"5min":0.35199999999999998,"15min":0.35099999999999998},"max":{"1min":0.68799999999999994,"5min":0.73599999999999999,"15min":0.75800000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54400000000000004,"15min":0.56299999999999994},"min":{"1min":0.44,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.73799999999999999,"5min":0.79700000000000004,"15min":0.91900000000000004},"last":0.59499999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31900,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339940,"statfs":{"total":96636764160,"available":96604098560,"internally_reserved":0,"allocated":802816,"data_stored":393460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58099999999999996,"15min":0.623},"min":{"1min":0.45300000000000001,"5min":0.26600000000000001,"15min":0.24399999999999999},"max":{"1min":0.69799999999999995,"5min":1.1679999999999999,"15min":6.3079999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57999999999999996,"15min":0.56399999999999995},"min":{"1min":0.437,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.79800000000000004,"5min":1.1319999999999999,"15min":1.208},"last":0.63700000000000001}]},{"osd":1,"last update":"Mon Mar 18 19:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.56799999999999995},"min":{"1min":0.443,"5min":0.33400000000000002,"15min":0.28399999999999997},"max":{"1min":0.70399999999999996,"5min":1.1140000000000001,"15min":1.196},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.59499999999999997,"15min":0.63600000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.29799999999999999},"max":{"1min":0.73599999999999999,"5min":1.21,"15min":6.2549999999999999},"last":0.50600000000000001}]},{"osd":2,"last update":"Mon Mar 18 19:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59599999999999997,"15min":0.63500000000000001},"min":{"1min":0.38600000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.75800000000000001,"5min":1.1579999999999999,"15min":6.2999999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58299999999999996,"15min":0.76000000000000001},"min":{"1min":0.46400000000000002,"5min":0.33400000000000002,"15min":0.33200000000000002},"max":{"1min":0.72299999999999998,"5min":1.0569999999999999,"15min":17.387},"last":0.43099999999999999}]},{"osd":3,"last update":"Mon Mar 18 19:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60199999999999998,"15min":0.65000000000000002},"min":{"1min":0.45000000000000001,"5min":0.38700000000000001,"15min":0.35899999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":6.3330000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.61099999999999999,"15min":0.60399999999999998},"min":{"1min":0.52000000000000002,"5min":0.41499999999999998,"15min":0.41299999999999998},"max":{"1min":0.747,"5min":1.087,"15min":1.2390000000000001},"last":0.54200000000000004}]},{"osd":4,"last update":"Mon Mar 18 19:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.46000000000000002,"15min":0.46899999999999997},"min":{"1min":0.34000000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60099999999999998,"5min":0.67200000000000004,"15min":0.70899999999999996},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.48399999999999999,"15min":0.496},"min":{"1min":0.35099999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.65400000000000003,"5min":0.71099999999999997,"15min":0.83599999999999997},"last":0.59199999999999997}]},{"osd":6,"last update":"Mon Mar 18 19:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.47899999999999998,"15min":0.48999999999999999},"min":{"1min":0.35499999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.65100000000000002,"5min":0.81699999999999995,"15min":0.88300000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.38500000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64500000000000002,"5min":0.73599999999999999,"15min":0.78300000000000003},"last":0.46999999999999997}]},{"osd":7,"last update":"Mon Mar 18 19:02:03 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52500000000000002,"15min":0.53900000000000003},"min":{"1min":0.38900000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.64300000000000002,"5min":0.75600000000000001,"15min":1.0449999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54200000000000004,"15min":0.56000000000000005},"min":{"1min":0.42199999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.70999999999999996,"5min":0.873,"15min":1.071},"last":0.67900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T19:03:06.697 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T19:03:06.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.862+0000 7fc1699cc700 1 Processor -- start 2024-03-18T19:03:06.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.862+0000 7fc1699cc700 1 -- start start 2024-03-18T19:03:06.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.863+0000 7fc1699cc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc1640cf090 0x7fc1640d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:06.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.863+0000 7fc1699cc700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1640cddc0 con 0x7fc164160940 2024-03-18T19:03:06.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.863+0000 7fc1699cc700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1640cdff0 con 0x7fc164160d20 2024-03-18T19:03:06.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.863+0000 7fc1699cc700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1640ce2d0 con 0x7fc1640cf090 2024-03-18T19:03:06.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.863+0000 7fc1627fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc1640cf090 0x7fc1640d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.863+0000 7fc162ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc164160940 0x7fc1640d0190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60170/0 (socket says 172.21.15.1:60170) 2024-03-18T19:03:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.863+0000 7fc162ffd700 1 -- 172.21.15.1:0/791211470 learned_addr learned my addr 172.21.15.1:0/791211470 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:03:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.863+0000 7fc1627fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc1640cf090 0x7fc1640d1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53894/0 (socket says 172.21.15.1:53894) 2024-03-18T19:03:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.863+0000 7fc1627fc700 1 -- 172.21.15.1:0/791211470 >> v1:172.21.15.165:6789/0 conn(0x7fc164160d20 legacy=0x7fc1640b25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.863+0000 7fc1627fc700 1 -- 172.21.15.1:0/791211470 >> v1:172.21.15.1:6789/0 conn(0x7fc164160940 legacy=0x7fc1640d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.863+0000 7fc1627fc700 1 -- 172.21.15.1:0/791211470 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1640ce5b0 con 0x7fc1640cf090 2024-03-18T19:03:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.863+0000 7fc161ffb700 1 -- 172.21.15.1:0/791211470 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830990226 0 0) 0x7fc1640cddc0 con 0x7fc164160940 2024-03-18T19:03:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.864+0000 7fc1627fc700 1 --2- 172.21.15.1:0/791211470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1640cf090 0x7fc1640d1790 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fc154009840 tx=0x7fc154009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b0315f5c37043 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.864+0000 7fc161ffb700 1 -- 172.21.15.1:0/791211470 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc154004230 con 0x7fc1640cf090 2024-03-18T19:03:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.864+0000 7fc161ffb700 1 -- 172.21.15.1:0/791211470 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc154004390 con 0x7fc1640cf090 2024-03-18T19:03:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.864+0000 7fc161ffb700 1 -- 172.21.15.1:0/791211470 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1540036a0 con 0x7fc1640cf090 2024-03-18T19:03:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.864+0000 7fc1699cc700 1 -- 172.21.15.1:0/791211470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1640cf090 msgr2=0x7fc1640d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:06.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.864+0000 7fc1699cc700 1 --2- 172.21.15.1:0/791211470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1640cf090 0x7fc1640d1790 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fc154009840 tx=0x7fc154009b50 comp rx=0 tx=0).stop 2024-03-18T19:03:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.864+0000 7fc1699cc700 1 -- 172.21.15.1:0/791211470 shutdown_connections 2024-03-18T19:03:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.864+0000 7fc1699cc700 1 --2- 172.21.15.1:0/791211470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1640cf090 0x7fc1640d1790 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.864+0000 7fc1699cc700 1 -- 172.21.15.1:0/791211470 >> 172.21.15.1:0/791211470 conn(0x7fc16409c520 msgr2=0x7fc1640d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.864+0000 7fc1699cc700 1 -- 172.21.15.1:0/791211470 shutdown_connections 2024-03-18T19:03:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.864+0000 7fc1699cc700 1 -- 172.21.15.1:0/791211470 wait complete. 2024-03-18T19:03:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.866+0000 7fc1699cc700 1 Processor -- start 2024-03-18T19:03:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.866+0000 7fc1699cc700 1 -- start start 2024-03-18T19:03:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.867+0000 7fc1699cc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1640cf090 0x7fc164158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.867+0000 7fc1699cc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc164160940 0x7fc1640b5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.867+0000 7fc1699cc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc164160d20 0x7fc16415f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.867+0000 7fc1699cc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1640c7c60 con 0x7fc164160940 2024-03-18T19:03:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.867+0000 7fc1699cc700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1640c7050 con 0x7fc164160d20 2024-03-18T19:03:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.867+0000 7fc1699cc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1640c7330 con 0x7fc1640cf090 2024-03-18T19:03:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.867+0000 7fc1627fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc164160940 0x7fc1640b5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.867+0000 7fc1627fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc164160940 0x7fc1640b5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53904/0 (socket says 172.21.15.1:53904) 2024-03-18T19:03:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.867+0000 7fc1627fc700 1 -- 172.21.15.1:0/2534296850 learned_addr learned my addr 172.21.15.1:0/2534296850 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.867+0000 7fc162ffd700 1 --2- 172.21.15.1:0/2534296850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1640cf090 0x7fc164158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.867+0000 7fc1637fe700 1 --2- 172.21.15.1:0/2534296850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc164160d20 0x7fc16415f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.867+0000 7fc1627fc700 1 -- 172.21.15.1:0/2534296850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1640cf090 msgr2=0x7fc164158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.867+0000 7fc1627fc700 1 --2- 172.21.15.1:0/2534296850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1640cf090 0x7fc164158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.867+0000 7fc1627fc700 1 -- 172.21.15.1:0/2534296850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc164160d20 msgr2=0x7fc16415f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.867+0000 7fc1627fc700 1 --2- 172.21.15.1:0/2534296850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc164160d20 0x7fc16415f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.867+0000 7fc1627fc700 1 -- 172.21.15.1:0/2534296850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1540094a0 con 0x7fc164160940 2024-03-18T19:03:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.868+0000 7fc1627fc700 1 --2- 172.21.15.1:0/2534296850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc164160940 0x7fc1640b5f20 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc154000c00 tx=0x7fc154004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.868+0000 7fc14bfff700 1 -- 172.21.15.1:0/2534296850 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc154011630 con 0x7fc164160940 2024-03-18T19:03:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.868+0000 7fc14bfff700 1 -- 172.21.15.1:0/2534296850 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15401e070 con 0x7fc164160940 2024-03-18T19:03:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.868+0000 7fc1699cc700 1 -- 172.21.15.1:0/2534296850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1640c7610 con 0x7fc164160940 2024-03-18T19:03:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.868+0000 7fc14bfff700 1 -- 172.21.15.1:0/2534296850 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1540119f0 con 0x7fc164160940 2024-03-18T19:03:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.868+0000 7fc1699cc700 1 -- 172.21.15.1:0/2534296850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc164158540 con 0x7fc164160940 2024-03-18T19:03:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.869+0000 7fc1699cc700 1 -- 172.21.15.1:0/2534296850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1640b5900 con 0x7fc164160940 2024-03-18T19:03:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.869+0000 7fc14bfff700 1 -- 172.21.15.1:0/2534296850 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc154011b50 con 0x7fc164160940 2024-03-18T19:03:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.869+0000 7fc14bfff700 1 --2- 172.21.15.1:0/2534296850 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc15006adb0 0x7fc15006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.870+0000 7fc162ffd700 1 --2- 172.21.15.1:0/2534296850 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc15006adb0 0x7fc15006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.870+0000 7fc14bfff700 1 -- 172.21.15.1:0/2534296850 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc15408a9a0 con 0x7fc164160940 2024-03-18T19:03:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.870+0000 7fc162ffd700 1 --2- 172.21.15.1:0/2534296850 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc15006adb0 0x7fc15006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc158000f60 tx=0x7fc158007c40 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:06.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:06.874+0000 7fc14bfff700 1 -- 172.21.15.1:0/2534296850 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc154055920 con 0x7fc164160940 2024-03-18T19:03:07.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.070+0000 7fc1699cc700 1 -- 172.21.15.1:0/2534296850 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc1640ceb70 con 0x7fc15006adb0 2024-03-18T19:03:07.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.078+0000 7fc14bfff700 1 -- 172.21.15.1:0/2534296850 <== mgr.6045 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289266 (secure 0 0 0) 0x7fc158005b50 con 0x7fc15006adb0 2024-03-18T19:03:07.078 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:03:07.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.082+0000 7fc1699cc700 1 -- 172.21.15.1:0/2534296850 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc15006adb0 msgr2=0x7fc15006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:07.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.082+0000 7fc1699cc700 1 --2- 172.21.15.1:0/2534296850 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc15006adb0 0x7fc15006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc158000f60 tx=0x7fc158007c40 comp rx=0 tx=0).stop 2024-03-18T19:03:07.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.082+0000 7fc1699cc700 1 -- 172.21.15.1:0/2534296850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc164160940 msgr2=0x7fc1640b5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:07.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.082+0000 7fc1699cc700 1 --2- 172.21.15.1:0/2534296850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc164160940 0x7fc1640b5f20 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc154000c00 tx=0x7fc154004750 comp rx=0 tx=0).stop 2024-03-18T19:03:07.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.082+0000 7fc1699cc700 1 -- 172.21.15.1:0/2534296850 shutdown_connections 2024-03-18T19:03:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.082+0000 7fc1699cc700 1 --2- 172.21.15.1:0/2534296850 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc15006adb0 0x7fc15006d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.082+0000 7fc1699cc700 1 --2- 172.21.15.1:0/2534296850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1640cf090 0x7fc164158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.082+0000 7fc1699cc700 1 --2- 172.21.15.1:0/2534296850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc164160d20 0x7fc16415f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.082+0000 7fc1699cc700 1 --2- 172.21.15.1:0/2534296850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc164160940 0x7fc1640b5f20 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.082+0000 7fc1699cc700 1 -- 172.21.15.1:0/2534296850 >> 172.21.15.1:0/2534296850 conn(0x7fc16409c520 msgr2=0x7fc16406c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.082+0000 7fc1699cc700 1 -- 172.21.15.1:0/2534296850 shutdown_connections 2024-03-18T19:03:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.082+0000 7fc1699cc700 1 -- 172.21.15.1:0/2534296850 wait complete. 2024-03-18T19:03:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T19:03:07.090 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-18T19:03:06.967143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":309280,"kb_used_data":7832,"kb_used_omap":12,"kb_used_meta":301299,"kb_avail":754665440,"statfs":{"total":773094113280,"available":772777410560,"internally_reserved":0,"allocated":8019968,"data_stored":4518236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":308530745},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.867773"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542736+0000","last_change":"2024-03-18T19:03:02.554946+0000","last_active":"2024-03-18T19:03:03.542736+0000","last_peered":"2024-03-18T19:03:03.542736+0000","last_clean":"2024-03-18T19:03:03.542736+0000","last_became_active":"2024-03-18T19:03:02.554666+0000","last_became_peered":"2024-03-18T19:03:02.554666+0000","last_unstale":"2024-03-18T19:03:03.542736+0000","last_undegraded":"2024-03-18T19:03:03.542736+0000","last_fullsized":"2024-03-18T19:03:03.542736+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:17:13.986294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389138+0000","last_change":"2024-03-18T19:03:02.556576+0000","last_active":"2024-03-18T19:03:04.389138+0000","last_peered":"2024-03-18T19:03:04.389138+0000","last_clean":"2024-03-18T19:03:04.389138+0000","last_became_active":"2024-03-18T19:03:02.556284+0000","last_became_peered":"2024-03-18T19:03:02.556284+0000","last_unstale":"2024-03-18T19:03:04.389138+0000","last_undegraded":"2024-03-18T19:03:04.389138+0000","last_fullsized":"2024-03-18T19:03:04.389138+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:50:28.343435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388093+0000","last_change":"2024-03-18T19:03:02.568865+0000","last_active":"2024-03-18T19:03:04.388093+0000","last_peered":"2024-03-18T19:03:04.388093+0000","last_clean":"2024-03-18T19:03:04.388093+0000","last_became_active":"2024-03-18T19:03:02.565363+0000","last_became_peered":"2024-03-18T19:03:02.565363+0000","last_unstale":"2024-03-18T19:03:04.388093+0000","last_undegraded":"2024-03-18T19:03:04.388093+0000","last_fullsized":"2024-03-18T19:03:04.388093+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:04:49.333141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388466+0000","last_change":"2024-03-18T19:03:02.555824+0000","last_active":"2024-03-18T19:03:04.388466+0000","last_peered":"2024-03-18T19:03:04.388466+0000","last_clean":"2024-03-18T19:03:04.388466+0000","last_became_active":"2024-03-18T19:03:02.555556+0000","last_became_peered":"2024-03-18T19:03:02.555556+0000","last_unstale":"2024-03-18T19:03:04.388466+0000","last_undegraded":"2024-03-18T19:03:04.388466+0000","last_fullsized":"2024-03-18T19:03:04.388466+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:44:42.648920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389081+0000","last_change":"2024-03-18T19:03:02.551894+0000","last_active":"2024-03-18T19:03:04.389081+0000","last_peered":"2024-03-18T19:03:04.389081+0000","last_clean":"2024-03-18T19:03:04.389081+0000","last_became_active":"2024-03-18T19:03:02.551618+0000","last_became_peered":"2024-03-18T19:03:02.551618+0000","last_unstale":"2024-03-18T19:03:04.389081+0000","last_undegraded":"2024-03-18T19:03:04.389081+0000","last_fullsized":"2024-03-18T19:03:04.389081+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:55:54.064220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542664+0000","last_change":"2024-03-18T19:03:02.556400+0000","last_active":"2024-03-18T19:03:03.542664+0000","last_peered":"2024-03-18T19:03:03.542664+0000","last_clean":"2024-03-18T19:03:03.542664+0000","last_became_active":"2024-03-18T19:03:02.556006+0000","last_became_peered":"2024-03-18T19:03:02.556006+0000","last_unstale":"2024-03-18T19:03:03.542664+0000","last_undegraded":"2024-03-18T19:03:03.542664+0000","last_fullsized":"2024-03-18T19:03:03.542664+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:09.977954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387695+0000","last_change":"2024-03-18T19:03:02.560247+0000","last_active":"2024-03-18T19:03:04.387695+0000","last_peered":"2024-03-18T19:03:04.387695+0000","last_clean":"2024-03-18T19:03:04.387695+0000","last_became_active":"2024-03-18T19:03:02.556601+0000","last_became_peered":"2024-03-18T19:03:02.556601+0000","last_unstale":"2024-03-18T19:03:04.387695+0000","last_undegraded":"2024-03-18T19:03:04.387695+0000","last_fullsized":"2024-03-18T19:03:04.387695+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:44:42.149005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387836+0000","last_change":"2024-03-18T19:03:02.558337+0000","last_active":"2024-03-18T19:03:04.387836+0000","last_peered":"2024-03-18T19:03:04.387836+0000","last_clean":"2024-03-18T19:03:04.387836+0000","last_became_active":"2024-03-18T19:03:02.558083+0000","last_became_peered":"2024-03-18T19:03:02.558083+0000","last_unstale":"2024-03-18T19:03:04.387836+0000","last_undegraded":"2024-03-18T19:03:04.387836+0000","last_fullsized":"2024-03-18T19:03:04.387836+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:13.001203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388335+0000","last_change":"2024-03-18T19:03:02.568191+0000","last_active":"2024-03-18T19:03:04.388335+0000","last_peered":"2024-03-18T19:03:04.388335+0000","last_clean":"2024-03-18T19:03:04.388335+0000","last_became_active":"2024-03-18T19:03:02.567991+0000","last_became_peered":"2024-03-18T19:03:02.567991+0000","last_unstale":"2024-03-18T19:03:04.388335+0000","last_undegraded":"2024-03-18T19:03:04.388335+0000","last_fullsized":"2024-03-18T19:03:04.388335+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:36:45.414625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.543017+0000","last_change":"2024-03-18T19:03:02.554474+0000","last_active":"2024-03-18T19:03:03.543017+0000","last_peered":"2024-03-18T19:03:03.543017+0000","last_clean":"2024-03-18T19:03:03.543017+0000","last_became_active":"2024-03-18T19:03:02.554214+0000","last_became_peered":"2024-03-18T19:03:02.554214+0000","last_unstale":"2024-03-18T19:03:03.543017+0000","last_undegraded":"2024-03-18T19:03:03.543017+0000","last_fullsized":"2024-03-18T19:03:03.543017+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:38:47.495608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.544033+0000","last_change":"2024-03-18T19:03:02.560070+0000","last_active":"2024-03-18T19:03:03.544033+0000","last_peered":"2024-03-18T19:03:03.544033+0000","last_clean":"2024-03-18T19:03:03.544033+0000","last_became_active":"2024-03-18T19:03:02.559856+0000","last_became_peered":"2024-03-18T19:03:02.559856+0000","last_unstale":"2024-03-18T19:03:03.544033+0000","last_undegraded":"2024-03-18T19:03:03.544033+0000","last_fullsized":"2024-03-18T19:03:03.544033+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:05.040442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388126+0000","last_change":"2024-03-18T19:03:02.551750+0000","last_active":"2024-03-18T19:03:04.388126+0000","last_peered":"2024-03-18T19:03:04.388126+0000","last_clean":"2024-03-18T19:03:04.388126+0000","last_became_active":"2024-03-18T19:03:02.551475+0000","last_became_peered":"2024-03-18T19:03:02.551475+0000","last_unstale":"2024-03-18T19:03:04.388126+0000","last_undegraded":"2024-03-18T19:03:04.388126+0000","last_fullsized":"2024-03-18T19:03:04.388126+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:03.437717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542502+0000","last_change":"2024-03-18T19:03:02.547246+0000","last_active":"2024-03-18T19:03:03.542502+0000","last_peered":"2024-03-18T19:03:03.542502+0000","last_clean":"2024-03-18T19:03:03.542502+0000","last_became_active":"2024-03-18T19:03:02.547042+0000","last_became_peered":"2024-03-18T19:03:02.547042+0000","last_unstale":"2024-03-18T19:03:03.542502+0000","last_undegraded":"2024-03-18T19:03:03.542502+0000","last_fullsized":"2024-03-18T19:03:03.542502+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:16:29.862517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389006+0000","last_change":"2024-03-18T19:03:02.563359+0000","last_active":"2024-03-18T19:03:04.389006+0000","last_peered":"2024-03-18T19:03:04.389006+0000","last_clean":"2024-03-18T19:03:04.389006+0000","last_became_active":"2024-03-18T19:03:02.562885+0000","last_became_peered":"2024-03-18T19:03:02.562885+0000","last_unstale":"2024-03-18T19:03:04.389006+0000","last_undegraded":"2024-03-18T19:03:04.389006+0000","last_fullsized":"2024-03-18T19:03:04.389006+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:51:15.238139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388716+0000","last_change":"2024-03-18T19:03:02.556653+0000","last_active":"2024-03-18T19:03:04.388716+0000","last_peered":"2024-03-18T19:03:04.388716+0000","last_clean":"2024-03-18T19:03:04.388716+0000","last_became_active":"2024-03-18T19:03:02.553397+0000","last_became_peered":"2024-03-18T19:03:02.553397+0000","last_unstale":"2024-03-18T19:03:04.388716+0000","last_undegraded":"2024-03-18T19:03:04.388716+0000","last_fullsized":"2024-03-18T19:03:04.388716+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:11:14.319814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542463+0000","last_change":"2024-03-18T19:03:02.546658+0000","last_active":"2024-03-18T19:03:03.542463+0000","last_peered":"2024-03-18T19:03:03.542463+0000","last_clean":"2024-03-18T19:03:03.542463+0000","last_became_active":"2024-03-18T19:03:02.545109+0000","last_became_peered":"2024-03-18T19:03:02.545109+0000","last_unstale":"2024-03-18T19:03:03.542463+0000","last_undegraded":"2024-03-18T19:03:03.542463+0000","last_fullsized":"2024-03-18T19:03:03.542463+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:57:33.441885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388964+0000","last_change":"2024-03-18T19:03:02.566788+0000","last_active":"2024-03-18T19:03:04.388964+0000","last_peered":"2024-03-18T19:03:04.388964+0000","last_clean":"2024-03-18T19:03:04.388964+0000","last_became_active":"2024-03-18T19:03:02.566360+0000","last_became_peered":"2024-03-18T19:03:02.566360+0000","last_unstale":"2024-03-18T19:03:04.388964+0000","last_undegraded":"2024-03-18T19:03:04.388964+0000","last_fullsized":"2024-03-18T19:03:04.388964+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:11:17.817327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389247+0000","last_change":"2024-03-18T19:03:02.554052+0000","last_active":"2024-03-18T19:03:04.389247+0000","last_peered":"2024-03-18T19:03:04.389247+0000","last_clean":"2024-03-18T19:03:04.389247+0000","last_became_active":"2024-03-18T19:03:02.553814+0000","last_became_peered":"2024-03-18T19:03:02.553814+0000","last_unstale":"2024-03-18T19:03:04.389247+0000","last_undegraded":"2024-03-18T19:03:04.389247+0000","last_fullsized":"2024-03-18T19:03:04.389247+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:09:57.608685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388206+0000","last_change":"2024-03-18T19:03:02.565420+0000","last_active":"2024-03-18T19:03:04.388206+0000","last_peered":"2024-03-18T19:03:04.388206+0000","last_clean":"2024-03-18T19:03:04.388206+0000","last_became_active":"2024-03-18T19:03:02.565101+0000","last_became_peered":"2024-03-18T19:03:02.565101+0000","last_unstale":"2024-03-18T19:03:04.388206+0000","last_undegraded":"2024-03-18T19:03:04.388206+0000","last_fullsized":"2024-03-18T19:03:04.388206+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:23:58.064544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389367+0000","last_change":"2024-03-18T19:03:02.544788+0000","last_active":"2024-03-18T19:03:04.389367+0000","last_peered":"2024-03-18T19:03:04.389367+0000","last_clean":"2024-03-18T19:03:04.389367+0000","last_became_active":"2024-03-18T19:03:02.544374+0000","last_became_peered":"2024-03-18T19:03:02.544374+0000","last_unstale":"2024-03-18T19:03:04.389367+0000","last_undegraded":"2024-03-18T19:03:04.389367+0000","last_fullsized":"2024-03-18T19:03:04.389367+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:28:54.390651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388535+0000","last_change":"2024-03-18T19:03:02.548724+0000","last_active":"2024-03-18T19:03:04.388535+0000","last_peered":"2024-03-18T19:03:04.388535+0000","last_clean":"2024-03-18T19:03:04.388535+0000","last_became_active":"2024-03-18T19:03:02.547985+0000","last_became_peered":"2024-03-18T19:03:02.547985+0000","last_unstale":"2024-03-18T19:03:04.388535+0000","last_undegraded":"2024-03-18T19:03:04.388535+0000","last_fullsized":"2024-03-18T19:03:04.388535+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:04:06.001910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387809+0000","last_change":"2024-03-18T19:03:02.566926+0000","last_active":"2024-03-18T19:03:04.387809+0000","last_peered":"2024-03-18T19:03:04.387809+0000","last_clean":"2024-03-18T19:03:04.387809+0000","last_became_active":"2024-03-18T19:03:02.566664+0000","last_became_peered":"2024-03-18T19:03:02.566664+0000","last_unstale":"2024-03-18T19:03:04.387809+0000","last_undegraded":"2024-03-18T19:03:04.387809+0000","last_fullsized":"2024-03-18T19:03:04.387809+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:11:17.817327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.543798+0000","last_change":"2024-03-18T19:03:02.561108+0000","last_active":"2024-03-18T19:03:03.543798+0000","last_peered":"2024-03-18T19:03:03.543798+0000","last_clean":"2024-03-18T19:03:03.543798+0000","last_became_active":"2024-03-18T19:03:02.560912+0000","last_became_peered":"2024-03-18T19:03:02.560912+0000","last_unstale":"2024-03-18T19:03:03.543798+0000","last_undegraded":"2024-03-18T19:03:03.543798+0000","last_fullsized":"2024-03-18T19:03:03.543798+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:17:59.732292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387554+0000","last_change":"2024-03-18T19:03:02.565651+0000","last_active":"2024-03-18T19:03:04.387554+0000","last_peered":"2024-03-18T19:03:04.387554+0000","last_clean":"2024-03-18T19:03:04.387554+0000","last_became_active":"2024-03-18T19:03:02.565445+0000","last_became_peered":"2024-03-18T19:03:02.565445+0000","last_unstale":"2024-03-18T19:03:04.387554+0000","last_undegraded":"2024-03-18T19:03:04.387554+0000","last_fullsized":"2024-03-18T19:03:04.387554+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:25.648248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388902+0000","last_change":"2024-03-18T19:03:02.566647+0000","last_active":"2024-03-18T19:03:04.388902+0000","last_peered":"2024-03-18T19:03:04.388902+0000","last_clean":"2024-03-18T19:03:04.388902+0000","last_became_active":"2024-03-18T19:03:02.566185+0000","last_became_peered":"2024-03-18T19:03:02.566185+0000","last_unstale":"2024-03-18T19:03:04.388902+0000","last_undegraded":"2024-03-18T19:03:04.388902+0000","last_fullsized":"2024-03-18T19:03:04.388902+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:36:45.414625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388188+0000","last_change":"2024-03-18T19:03:02.553778+0000","last_active":"2024-03-18T19:03:04.388188+0000","last_peered":"2024-03-18T19:03:04.388188+0000","last_clean":"2024-03-18T19:03:04.388188+0000","last_became_active":"2024-03-18T19:03:02.553510+0000","last_became_peered":"2024-03-18T19:03:02.553510+0000","last_unstale":"2024-03-18T19:03:04.388188+0000","last_undegraded":"2024-03-18T19:03:04.388188+0000","last_fullsized":"2024-03-18T19:03:04.388188+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:25:29.625002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542886+0000","last_change":"2024-03-18T19:03:02.570122+0000","last_active":"2024-03-18T19:03:03.542886+0000","last_peered":"2024-03-18T19:03:03.542886+0000","last_clean":"2024-03-18T19:03:03.542886+0000","last_became_active":"2024-03-18T19:03:02.569833+0000","last_became_peered":"2024-03-18T19:03:02.569833+0000","last_unstale":"2024-03-18T19:03:03.542886+0000","last_undegraded":"2024-03-18T19:03:03.542886+0000","last_fullsized":"2024-03-18T19:03:03.542886+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:32:11.619630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389465+0000","last_change":"2024-03-18T19:03:02.549860+0000","last_active":"2024-03-18T19:03:04.389465+0000","last_peered":"2024-03-18T19:03:04.389465+0000","last_clean":"2024-03-18T19:03:04.389465+0000","last_became_active":"2024-03-18T19:03:02.549608+0000","last_became_peered":"2024-03-18T19:03:02.549608+0000","last_unstale":"2024-03-18T19:03:04.389465+0000","last_undegraded":"2024-03-18T19:03:04.389465+0000","last_fullsized":"2024-03-18T19:03:04.389465+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:33:54.944921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387968+0000","last_change":"2024-03-18T19:03:02.566090+0000","last_active":"2024-03-18T19:03:04.387968+0000","last_peered":"2024-03-18T19:03:04.387968+0000","last_clean":"2024-03-18T19:03:04.387968+0000","last_became_active":"2024-03-18T19:03:02.565749+0000","last_became_peered":"2024-03-18T19:03:02.565749+0000","last_unstale":"2024-03-18T19:03:04.387968+0000","last_undegraded":"2024-03-18T19:03:04.387968+0000","last_fullsized":"2024-03-18T19:03:04.387968+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:00:13.013248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388466+0000","last_change":"2024-03-18T19:03:02.558410+0000","last_active":"2024-03-18T19:03:04.388466+0000","last_peered":"2024-03-18T19:03:04.388466+0000","last_clean":"2024-03-18T19:03:04.388466+0000","last_became_active":"2024-03-18T19:03:02.557103+0000","last_became_peered":"2024-03-18T19:03:02.557103+0000","last_unstale":"2024-03-18T19:03:04.388466+0000","last_undegraded":"2024-03-18T19:03:04.388466+0000","last_fullsized":"2024-03-18T19:03:04.388466+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:21:14.393328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387631+0000","last_change":"2024-03-18T19:03:02.563168+0000","last_active":"2024-03-18T19:03:04.387631+0000","last_peered":"2024-03-18T19:03:04.387631+0000","last_clean":"2024-03-18T19:03:04.387631+0000","last_became_active":"2024-03-18T19:03:02.562892+0000","last_became_peered":"2024-03-18T19:03:02.562892+0000","last_unstale":"2024-03-18T19:03:04.387631+0000","last_undegraded":"2024-03-18T19:03:04.387631+0000","last_fullsized":"2024-03-18T19:03:04.387631+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:25:32.860910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388549+0000","last_change":"2024-03-18T19:03:02.554256+0000","last_active":"2024-03-18T19:03:04.388549+0000","last_peered":"2024-03-18T19:03:04.388549+0000","last_clean":"2024-03-18T19:03:04.388549+0000","last_became_active":"2024-03-18T19:03:02.553925+0000","last_became_peered":"2024-03-18T19:03:02.553925+0000","last_unstale":"2024-03-18T19:03:04.388549+0000","last_undegraded":"2024-03-18T19:03:04.388549+0000","last_fullsized":"2024-03-18T19:03:04.388549+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:04:52.461947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388257+0000","last_change":"2024-03-18T19:03:02.555892+0000","last_active":"2024-03-18T19:03:04.388257+0000","last_peered":"2024-03-18T19:03:04.388257+0000","last_clean":"2024-03-18T19:03:04.388257+0000","last_became_active":"2024-03-18T19:03:02.555689+0000","last_became_peered":"2024-03-18T19:03:02.555689+0000","last_unstale":"2024-03-18T19:03:04.388257+0000","last_undegraded":"2024-03-18T19:03:04.388257+0000","last_fullsized":"2024-03-18T19:03:04.388257+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:55:54.064220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"70'4","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389745+0000","last_change":"2024-03-18T19:03:02.120029+0000","last_active":"2024-03-18T19:03:04.389745+0000","last_peered":"2024-03-18T19:03:04.389745+0000","last_clean":"2024-03-18T19:03:04.389745+0000","last_became_active":"2024-03-18T19:03:02.119726+0000","last_became_peered":"2024-03-18T19:03:02.119726+0000","last_unstale":"2024-03-18T19:03:04.389745+0000","last_undegraded":"2024-03-18T19:03:04.389745+0000","last_fullsized":"2024-03-18T19:03:04.389745+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:57:55.947358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389348+0000","last_change":"2024-03-18T19:03:02.559827+0000","last_active":"2024-03-18T19:03:04.389348+0000","last_peered":"2024-03-18T19:03:04.389348+0000","last_clean":"2024-03-18T19:03:04.389348+0000","last_became_active":"2024-03-18T19:03:02.559578+0000","last_became_peered":"2024-03-18T19:03:02.559578+0000","last_unstale":"2024-03-18T19:03:04.389348+0000","last_undegraded":"2024-03-18T19:03:04.389348+0000","last_fullsized":"2024-03-18T19:03:04.389348+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:55:54.064220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"70'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.543100+0000","last_change":"2024-03-18T19:03:02.118338+0000","last_active":"2024-03-18T19:03:03.543100+0000","last_peered":"2024-03-18T19:03:03.543100+0000","last_clean":"2024-03-18T19:03:03.543100+0000","last_became_active":"2024-03-18T19:03:02.117668+0000","last_became_peered":"2024-03-18T19:03:02.117668+0000","last_unstale":"2024-03-18T19:03:03.543100+0000","last_undegraded":"2024-03-18T19:03:03.543100+0000","last_fullsized":"2024-03-18T19:03:03.543100+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:41.192643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388531+0000","last_change":"2024-03-18T19:03:02.553875+0000","last_active":"2024-03-18T19:03:04.388531+0000","last_peered":"2024-03-18T19:03:04.388531+0000","last_clean":"2024-03-18T19:03:04.388531+0000","last_became_active":"2024-03-18T19:03:02.553604+0000","last_became_peered":"2024-03-18T19:03:02.553604+0000","last_unstale":"2024-03-18T19:03:04.388531+0000","last_undegraded":"2024-03-18T19:03:04.388531+0000","last_fullsized":"2024-03-18T19:03:04.388531+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:05.040442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"70'2","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542976+0000","last_change":"2024-03-18T19:03:02.115656+0000","last_active":"2024-03-18T19:03:03.542976+0000","last_peered":"2024-03-18T19:03:03.542976+0000","last_clean":"2024-03-18T19:03:03.542976+0000","last_became_active":"2024-03-18T19:03:02.115321+0000","last_became_peered":"2024-03-18T19:03:02.115321+0000","last_unstale":"2024-03-18T19:03:03.542976+0000","last_undegraded":"2024-03-18T19:03:03.542976+0000","last_fullsized":"2024-03-18T19:03:03.542976+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:22:42.820030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388431+0000","last_change":"2024-03-18T19:03:02.554279+0000","last_active":"2024-03-18T19:03:04.388431+0000","last_peered":"2024-03-18T19:03:04.388431+0000","last_clean":"2024-03-18T19:03:04.388431+0000","last_became_active":"2024-03-18T19:03:02.554031+0000","last_became_peered":"2024-03-18T19:03:02.554031+0000","last_unstale":"2024-03-18T19:03:04.388431+0000","last_undegraded":"2024-03-18T19:03:04.388431+0000","last_fullsized":"2024-03-18T19:03:04.388431+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:38:57.388604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"70'2","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389396+0000","last_change":"2024-03-18T19:03:02.121738+0000","last_active":"2024-03-18T19:03:04.389396+0000","last_peered":"2024-03-18T19:03:04.389396+0000","last_clean":"2024-03-18T19:03:04.389396+0000","last_became_active":"2024-03-18T19:03:02.121460+0000","last_became_peered":"2024-03-18T19:03:02.121460+0000","last_unstale":"2024-03-18T19:03:04.389396+0000","last_undegraded":"2024-03-18T19:03:04.389396+0000","last_fullsized":"2024-03-18T19:03:04.389396+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:38:32.454783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389240+0000","last_change":"2024-03-18T19:03:02.566201+0000","last_active":"2024-03-18T19:03:04.389240+0000","last_peered":"2024-03-18T19:03:04.389240+0000","last_clean":"2024-03-18T19:03:04.389240+0000","last_became_active":"2024-03-18T19:03:02.565924+0000","last_became_peered":"2024-03-18T19:03:02.565924+0000","last_unstale":"2024-03-18T19:03:04.389240+0000","last_undegraded":"2024-03-18T19:03:04.389240+0000","last_fullsized":"2024-03-18T19:03:04.389240+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:50:28.343435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"70'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388077+0000","last_change":"2024-03-18T19:03:02.126132+0000","last_active":"2024-03-18T19:03:04.388077+0000","last_peered":"2024-03-18T19:03:04.388077+0000","last_clean":"2024-03-18T19:03:04.388077+0000","last_became_active":"2024-03-18T19:03:02.125834+0000","last_became_peered":"2024-03-18T19:03:02.125834+0000","last_unstale":"2024-03-18T19:03:04.388077+0000","last_undegraded":"2024-03-18T19:03:04.388077+0000","last_fullsized":"2024-03-18T19:03:04.388077+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:02.410889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388769+0000","last_change":"2024-03-18T19:03:02.558509+0000","last_active":"2024-03-18T19:03:04.388769+0000","last_peered":"2024-03-18T19:03:04.388769+0000","last_clean":"2024-03-18T19:03:04.388769+0000","last_became_active":"2024-03-18T19:03:02.558194+0000","last_became_peered":"2024-03-18T19:03:02.558194+0000","last_unstale":"2024-03-18T19:03:04.388769+0000","last_undegraded":"2024-03-18T19:03:04.388769+0000","last_fullsized":"2024-03-18T19:03:04.388769+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:30.094824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"70'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388771+0000","last_change":"2024-03-18T19:03:02.119225+0000","last_active":"2024-03-18T19:03:04.388771+0000","last_peered":"2024-03-18T19:03:04.388771+0000","last_clean":"2024-03-18T19:03:04.388771+0000","last_became_active":"2024-03-18T19:03:02.118974+0000","last_became_peered":"2024-03-18T19:03:02.118974+0000","last_unstale":"2024-03-18T19:03:04.388771+0000","last_undegraded":"2024-03-18T19:03:04.388771+0000","last_fullsized":"2024-03-18T19:03:04.388771+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:31:00.598479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388070+0000","last_change":"2024-03-18T19:03:02.557608+0000","last_active":"2024-03-18T19:03:04.388070+0000","last_peered":"2024-03-18T19:03:04.388070+0000","last_clean":"2024-03-18T19:03:04.388070+0000","last_became_active":"2024-03-18T19:03:02.557359+0000","last_became_peered":"2024-03-18T19:03:02.557359+0000","last_unstale":"2024-03-18T19:03:04.388070+0000","last_undegraded":"2024-03-18T19:03:04.388070+0000","last_fullsized":"2024-03-18T19:03:04.388070+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:23:19.809075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389131+0000","last_change":"2024-03-18T19:03:02.115443+0000","last_active":"2024-03-18T19:03:04.389131+0000","last_peered":"2024-03-18T19:03:04.389131+0000","last_clean":"2024-03-18T19:03:04.389131+0000","last_became_active":"2024-03-18T19:03:02.114952+0000","last_became_peered":"2024-03-18T19:03:02.114952+0000","last_unstale":"2024-03-18T19:03:04.389131+0000","last_undegraded":"2024-03-18T19:03:04.389131+0000","last_fullsized":"2024-03-18T19:03:04.389131+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:00:28.607976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389514+0000","last_change":"2024-03-18T19:03:02.569890+0000","last_active":"2024-03-18T19:03:04.389514+0000","last_peered":"2024-03-18T19:03:04.389514+0000","last_clean":"2024-03-18T19:03:04.389514+0000","last_became_active":"2024-03-18T19:03:02.569649+0000","last_became_peered":"2024-03-18T19:03:02.569649+0000","last_unstale":"2024-03-18T19:03:04.389514+0000","last_undegraded":"2024-03-18T19:03:04.389514+0000","last_fullsized":"2024-03-18T19:03:04.389514+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:37.161000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387659+0000","last_change":"2024-03-18T19:03:02.130557+0000","last_active":"2024-03-18T19:03:04.387659+0000","last_peered":"2024-03-18T19:03:04.387659+0000","last_clean":"2024-03-18T19:03:04.387659+0000","last_became_active":"2024-03-18T19:03:02.130205+0000","last_became_peered":"2024-03-18T19:03:02.130205+0000","last_unstale":"2024-03-18T19:03:04.387659+0000","last_undegraded":"2024-03-18T19:03:04.387659+0000","last_fullsized":"2024-03-18T19:03:04.387659+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:03.319006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389653+0000","last_change":"2024-03-18T19:03:02.556659+0000","last_active":"2024-03-18T19:03:04.389653+0000","last_peered":"2024-03-18T19:03:04.389653+0000","last_clean":"2024-03-18T19:03:04.389653+0000","last_became_active":"2024-03-18T19:03:02.556345+0000","last_became_peered":"2024-03-18T19:03:02.556345+0000","last_unstale":"2024-03-18T19:03:04.389653+0000","last_undegraded":"2024-03-18T19:03:04.389653+0000","last_fullsized":"2024-03-18T19:03:04.389653+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:09:57.608685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389575+0000","last_change":"2024-03-18T19:03:02.117375+0000","last_active":"2024-03-18T19:03:04.389575+0000","last_peered":"2024-03-18T19:03:04.389575+0000","last_clean":"2024-03-18T19:03:04.389575+0000","last_became_active":"2024-03-18T19:03:02.117060+0000","last_became_peered":"2024-03-18T19:03:02.117060+0000","last_unstale":"2024-03-18T19:03:04.389575+0000","last_undegraded":"2024-03-18T19:03:04.389575+0000","last_fullsized":"2024-03-18T19:03:04.389575+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:17:58.727452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389242+0000","last_change":"2024-03-18T19:03:02.555257+0000","last_active":"2024-03-18T19:03:04.389242+0000","last_peered":"2024-03-18T19:03:04.389242+0000","last_clean":"2024-03-18T19:03:04.389242+0000","last_became_active":"2024-03-18T19:03:02.554864+0000","last_became_peered":"2024-03-18T19:03:02.554864+0000","last_unstale":"2024-03-18T19:03:04.389242+0000","last_undegraded":"2024-03-18T19:03:04.389242+0000","last_fullsized":"2024-03-18T19:03:04.389242+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:44:42.648920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389551+0000","last_change":"2024-03-18T19:03:02.129254+0000","last_active":"2024-03-18T19:03:04.389551+0000","last_peered":"2024-03-18T19:03:04.389551+0000","last_clean":"2024-03-18T19:03:04.389551+0000","last_became_active":"2024-03-18T19:03:02.128964+0000","last_became_peered":"2024-03-18T19:03:02.128964+0000","last_unstale":"2024-03-18T19:03:04.389551+0000","last_undegraded":"2024-03-18T19:03:04.389551+0000","last_fullsized":"2024-03-18T19:03:04.389551+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:38:32.454783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389063+0000","last_change":"2024-03-18T19:03:02.563166+0000","last_active":"2024-03-18T19:03:04.389063+0000","last_peered":"2024-03-18T19:03:04.389063+0000","last_clean":"2024-03-18T19:03:04.389063+0000","last_became_active":"2024-03-18T19:03:02.562588+0000","last_became_peered":"2024-03-18T19:03:02.562588+0000","last_unstale":"2024-03-18T19:03:04.389063+0000","last_undegraded":"2024-03-18T19:03:04.389063+0000","last_fullsized":"2024-03-18T19:03:04.389063+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:05:54.935319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"70'3","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388870+0000","last_change":"2024-03-18T19:03:02.118817+0000","last_active":"2024-03-18T19:03:04.388870+0000","last_peered":"2024-03-18T19:03:04.388870+0000","last_clean":"2024-03-18T19:03:04.388870+0000","last_became_active":"2024-03-18T19:03:02.118498+0000","last_became_peered":"2024-03-18T19:03:02.118498+0000","last_unstale":"2024-03-18T19:03:04.388870+0000","last_undegraded":"2024-03-18T19:03:04.388870+0000","last_fullsized":"2024-03-18T19:03:04.388870+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:33:53.940081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389185+0000","last_change":"2024-03-18T19:03:02.550093+0000","last_active":"2024-03-18T19:03:04.389185+0000","last_peered":"2024-03-18T19:03:04.389185+0000","last_clean":"2024-03-18T19:03:04.389185+0000","last_became_active":"2024-03-18T19:03:02.549775+0000","last_became_peered":"2024-03-18T19:03:02.549775+0000","last_unstale":"2024-03-18T19:03:04.389185+0000","last_undegraded":"2024-03-18T19:03:04.389185+0000","last_fullsized":"2024-03-18T19:03:04.389185+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:44:42.648920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"70'6","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388311+0000","last_change":"2024-03-18T19:03:02.116621+0000","last_active":"2024-03-18T19:03:04.388311+0000","last_peered":"2024-03-18T19:03:04.388311+0000","last_clean":"2024-03-18T19:03:04.388311+0000","last_became_active":"2024-03-18T19:03:02.116306+0000","last_became_peered":"2024-03-18T19:03:02.116306+0000","last_unstale":"2024-03-18T19:03:04.388311+0000","last_undegraded":"2024-03-18T19:03:04.388311+0000","last_fullsized":"2024-03-18T19:03:04.388311+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:34.043005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389293+0000","last_change":"2024-03-18T19:03:02.558000+0000","last_active":"2024-03-18T19:03:04.389293+0000","last_peered":"2024-03-18T19:03:04.389293+0000","last_clean":"2024-03-18T19:03:04.389293+0000","last_became_active":"2024-03-18T19:03:02.557667+0000","last_became_peered":"2024-03-18T19:03:02.557667+0000","last_unstale":"2024-03-18T19:03:04.389293+0000","last_undegraded":"2024-03-18T19:03:04.389293+0000","last_fullsized":"2024-03-18T19:03:04.389293+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:46:24.309467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"70'2","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388176+0000","last_change":"2024-03-18T19:03:02.126189+0000","last_active":"2024-03-18T19:03:04.388176+0000","last_peered":"2024-03-18T19:03:04.388176+0000","last_clean":"2024-03-18T19:03:04.388176+0000","last_became_active":"2024-03-18T19:03:02.125898+0000","last_became_peered":"2024-03-18T19:03:02.125898+0000","last_unstale":"2024-03-18T19:03:04.388176+0000","last_undegraded":"2024-03-18T19:03:04.388176+0000","last_fullsized":"2024-03-18T19:03:04.388176+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:41.197019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388313+0000","last_change":"2024-03-18T19:03:02.570155+0000","last_active":"2024-03-18T19:03:04.388313+0000","last_peered":"2024-03-18T19:03:04.388313+0000","last_clean":"2024-03-18T19:03:04.388313+0000","last_became_active":"2024-03-18T19:03:02.569886+0000","last_became_peered":"2024-03-18T19:03:02.569886+0000","last_unstale":"2024-03-18T19:03:04.388313+0000","last_undegraded":"2024-03-18T19:03:04.388313+0000","last_fullsized":"2024-03-18T19:03:04.388313+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:33.527267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388624+0000","last_change":"2024-03-18T19:03:02.131089+0000","last_active":"2024-03-18T19:03:04.388624+0000","last_peered":"2024-03-18T19:03:04.388624+0000","last_clean":"2024-03-18T19:03:04.388624+0000","last_became_active":"2024-03-18T19:03:02.130777+0000","last_became_peered":"2024-03-18T19:03:02.130777+0000","last_unstale":"2024-03-18T19:03:04.388624+0000","last_undegraded":"2024-03-18T19:03:04.388624+0000","last_fullsized":"2024-03-18T19:03:04.388624+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:46.788609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387923+0000","last_change":"2024-03-18T19:03:02.561595+0000","last_active":"2024-03-18T19:03:04.387923+0000","last_peered":"2024-03-18T19:03:04.387923+0000","last_clean":"2024-03-18T19:03:04.387923+0000","last_became_active":"2024-03-18T19:03:02.561387+0000","last_became_peered":"2024-03-18T19:03:02.561387+0000","last_unstale":"2024-03-18T19:03:04.387923+0000","last_undegraded":"2024-03-18T19:03:04.387923+0000","last_fullsized":"2024-03-18T19:03:04.387923+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:42.647657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388807+0000","last_change":"2024-03-18T19:03:02.120516+0000","last_active":"2024-03-18T19:03:04.388807+0000","last_peered":"2024-03-18T19:03:04.388807+0000","last_clean":"2024-03-18T19:03:04.388807+0000","last_became_active":"2024-03-18T19:03:02.120202+0000","last_became_peered":"2024-03-18T19:03:02.120202+0000","last_unstale":"2024-03-18T19:03:04.388807+0000","last_undegraded":"2024-03-18T19:03:04.388807+0000","last_fullsized":"2024-03-18T19:03:04.388807+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:04.035601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388843+0000","last_change":"2024-03-18T19:03:02.566622+0000","last_active":"2024-03-18T19:03:04.388843+0000","last_peered":"2024-03-18T19:03:04.388843+0000","last_clean":"2024-03-18T19:03:04.388843+0000","last_became_active":"2024-03-18T19:03:02.566050+0000","last_became_peered":"2024-03-18T19:03:02.566050+0000","last_unstale":"2024-03-18T19:03:04.388843+0000","last_undegraded":"2024-03-18T19:03:04.388843+0000","last_fullsized":"2024-03-18T19:03:04.388843+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:59.912019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"70'8","reported_seq":19,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388087+0000","last_change":"2024-03-18T19:03:02.117083+0000","last_active":"2024-03-18T19:03:04.388087+0000","last_peered":"2024-03-18T19:03:04.388087+0000","last_clean":"2024-03-18T19:03:04.388087+0000","last_became_active":"2024-03-18T19:03:02.116814+0000","last_became_peered":"2024-03-18T19:03:02.116814+0000","last_unstale":"2024-03-18T19:03:04.388087+0000","last_undegraded":"2024-03-18T19:03:04.388087+0000","last_fullsized":"2024-03-18T19:03:04.388087+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:58:02.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387881+0000","last_change":"2024-03-18T19:03:02.555576+0000","last_active":"2024-03-18T19:03:04.387881+0000","last_peered":"2024-03-18T19:03:04.387881+0000","last_clean":"2024-03-18T19:03:04.387881+0000","last_became_active":"2024-03-18T19:03:02.555250+0000","last_became_peered":"2024-03-18T19:03:02.555250+0000","last_unstale":"2024-03-18T19:03:04.387881+0000","last_undegraded":"2024-03-18T19:03:04.387881+0000","last_fullsized":"2024-03-18T19:03:04.387881+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:25.648248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"70'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388736+0000","last_change":"2024-03-18T19:03:02.132934+0000","last_active":"2024-03-18T19:03:04.388736+0000","last_peered":"2024-03-18T19:03:04.388736+0000","last_clean":"2024-03-18T19:03:04.388736+0000","last_became_active":"2024-03-18T19:03:02.132612+0000","last_became_peered":"2024-03-18T19:03:02.132612+0000","last_unstale":"2024-03-18T19:03:04.388736+0000","last_undegraded":"2024-03-18T19:03:04.388736+0000","last_fullsized":"2024-03-18T19:03:04.388736+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:18:31.958318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542540+0000","last_change":"2024-03-18T19:03:02.566715+0000","last_active":"2024-03-18T19:03:03.542540+0000","last_peered":"2024-03-18T19:03:03.542540+0000","last_clean":"2024-03-18T19:03:03.542540+0000","last_became_active":"2024-03-18T19:03:02.566443+0000","last_became_peered":"2024-03-18T19:03:02.566443+0000","last_unstale":"2024-03-18T19:03:03.542540+0000","last_undegraded":"2024-03-18T19:03:03.542540+0000","last_fullsized":"2024-03-18T19:03:03.542540+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:35:22.527659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"70'2","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.544134+0000","last_change":"2024-03-18T19:03:02.118090+0000","last_active":"2024-03-18T19:03:03.544134+0000","last_peered":"2024-03-18T19:03:03.544134+0000","last_clean":"2024-03-18T19:03:03.544134+0000","last_became_active":"2024-03-18T19:03:02.117568+0000","last_became_peered":"2024-03-18T19:03:02.117568+0000","last_unstale":"2024-03-18T19:03:03.544134+0000","last_undegraded":"2024-03-18T19:03:03.544134+0000","last_fullsized":"2024-03-18T19:03:03.544134+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:40:43.461004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388672+0000","last_change":"2024-03-18T19:03:02.569748+0000","last_active":"2024-03-18T19:03:04.388672+0000","last_peered":"2024-03-18T19:03:04.388672+0000","last_clean":"2024-03-18T19:03:04.388672+0000","last_became_active":"2024-03-18T19:03:02.569515+0000","last_became_peered":"2024-03-18T19:03:02.569515+0000","last_unstale":"2024-03-18T19:03:04.388672+0000","last_undegraded":"2024-03-18T19:03:04.388672+0000","last_fullsized":"2024-03-18T19:03:04.388672+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:23:33.326062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"70'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388806+0000","last_change":"2024-03-18T19:03:02.118770+0000","last_active":"2024-03-18T19:03:04.388806+0000","last_peered":"2024-03-18T19:03:04.388806+0000","last_clean":"2024-03-18T19:03:04.388806+0000","last_became_active":"2024-03-18T19:03:02.118488+0000","last_became_peered":"2024-03-18T19:03:02.118488+0000","last_unstale":"2024-03-18T19:03:04.388806+0000","last_undegraded":"2024-03-18T19:03:04.388806+0000","last_fullsized":"2024-03-18T19:03:04.388806+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:57.060127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542604+0000","last_change":"2024-03-18T19:03:02.554925+0000","last_active":"2024-03-18T19:03:03.542604+0000","last_peered":"2024-03-18T19:03:03.542604+0000","last_clean":"2024-03-18T19:03:03.542604+0000","last_became_active":"2024-03-18T19:03:02.554617+0000","last_became_peered":"2024-03-18T19:03:02.554617+0000","last_unstale":"2024-03-18T19:03:03.542604+0000","last_undegraded":"2024-03-18T19:03:03.542604+0000","last_fullsized":"2024-03-18T19:03:03.542604+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:09:39.260374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389641+0000","last_change":"2024-03-18T19:03:02.122086+0000","last_active":"2024-03-18T19:03:04.389641+0000","last_peered":"2024-03-18T19:03:04.389641+0000","last_clean":"2024-03-18T19:03:04.389641+0000","last_became_active":"2024-03-18T19:03:02.121891+0000","last_became_peered":"2024-03-18T19:03:02.121891+0000","last_unstale":"2024-03-18T19:03:04.389641+0000","last_undegraded":"2024-03-18T19:03:04.389641+0000","last_fullsized":"2024-03-18T19:03:04.389641+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:18:15.263661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388769+0000","last_change":"2024-03-18T19:03:02.555287+0000","last_active":"2024-03-18T19:03:04.388769+0000","last_peered":"2024-03-18T19:03:04.388769+0000","last_clean":"2024-03-18T19:03:04.388769+0000","last_became_active":"2024-03-18T19:03:02.554856+0000","last_became_peered":"2024-03-18T19:03:02.554856+0000","last_unstale":"2024-03-18T19:03:04.388769+0000","last_undegraded":"2024-03-18T19:03:04.388769+0000","last_fullsized":"2024-03-18T19:03:04.388769+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:23:33.326062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"70'2","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542698+0000","last_change":"2024-03-18T19:03:02.116810+0000","last_active":"2024-03-18T19:03:03.542698+0000","last_peered":"2024-03-18T19:03:03.542698+0000","last_clean":"2024-03-18T19:03:03.542698+0000","last_became_active":"2024-03-18T19:03:02.116512+0000","last_became_peered":"2024-03-18T19:03:02.116512+0000","last_unstale":"2024-03-18T19:03:03.542698+0000","last_undegraded":"2024-03-18T19:03:03.542698+0000","last_fullsized":"2024-03-18T19:03:03.542698+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:31:00.598479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388606+0000","last_change":"2024-03-18T19:03:02.544303+0000","last_active":"2024-03-18T19:03:04.388606+0000","last_peered":"2024-03-18T19:03:04.388606+0000","last_clean":"2024-03-18T19:03:04.388606+0000","last_became_active":"2024-03-18T19:03:02.543821+0000","last_became_peered":"2024-03-18T19:03:02.543821+0000","last_unstale":"2024-03-18T19:03:04.388606+0000","last_undegraded":"2024-03-18T19:03:04.388606+0000","last_fullsized":"2024-03-18T19:03:04.388606+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:55:45.622276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.387922+0000","last_change":"2024-03-18T19:03:02.123481+0000","last_active":"2024-03-18T19:03:04.387922+0000","last_peered":"2024-03-18T19:03:04.387922+0000","last_clean":"2024-03-18T19:03:04.387922+0000","last_became_active":"2024-03-18T19:03:02.123245+0000","last_became_peered":"2024-03-18T19:03:02.123245+0000","last_unstale":"2024-03-18T19:03:04.387922+0000","last_undegraded":"2024-03-18T19:03:04.387922+0000","last_fullsized":"2024-03-18T19:03:04.387922+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:42:17.566346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388519+0000","last_change":"2024-03-18T19:03:02.565460+0000","last_active":"2024-03-18T19:03:04.388519+0000","last_peered":"2024-03-18T19:03:04.388519+0000","last_clean":"2024-03-18T19:03:04.388519+0000","last_became_active":"2024-03-18T19:03:02.565184+0000","last_became_peered":"2024-03-18T19:03:02.565184+0000","last_unstale":"2024-03-18T19:03:04.388519+0000","last_undegraded":"2024-03-18T19:03:04.388519+0000","last_fullsized":"2024-03-18T19:03:04.388519+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:22:11.304687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388409+0000","last_change":"2024-03-18T19:03:02.129792+0000","last_active":"2024-03-18T19:03:04.388409+0000","last_peered":"2024-03-18T19:03:04.388409+0000","last_clean":"2024-03-18T19:03:04.388409+0000","last_became_active":"2024-03-18T19:03:02.129485+0000","last_became_peered":"2024-03-18T19:03:02.129485+0000","last_unstale":"2024-03-18T19:03:04.388409+0000","last_undegraded":"2024-03-18T19:03:04.388409+0000","last_fullsized":"2024-03-18T19:03:04.388409+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:34.043005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542836+0000","last_change":"2024-03-18T19:03:02.556628+0000","last_active":"2024-03-18T19:03:03.542836+0000","last_peered":"2024-03-18T19:03:03.542836+0000","last_clean":"2024-03-18T19:03:03.542836+0000","last_became_active":"2024-03-18T19:03:02.556376+0000","last_became_peered":"2024-03-18T19:03:02.556376+0000","last_unstale":"2024-03-18T19:03:03.542836+0000","last_undegraded":"2024-03-18T19:03:03.542836+0000","last_fullsized":"2024-03-18T19:03:03.542836+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:42.201860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.543725+0000","last_change":"2024-03-18T19:03:02.120610+0000","last_active":"2024-03-18T19:03:03.543725+0000","last_peered":"2024-03-18T19:03:03.543725+0000","last_clean":"2024-03-18T19:03:03.543725+0000","last_became_active":"2024-03-18T19:03:02.120106+0000","last_became_peered":"2024-03-18T19:03:02.120106+0000","last_unstale":"2024-03-18T19:03:03.543725+0000","last_undegraded":"2024-03-18T19:03:03.543725+0000","last_fullsized":"2024-03-18T19:03:03.543725+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:46.762602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388656+0000","last_change":"2024-03-18T19:03:02.552746+0000","last_active":"2024-03-18T19:03:04.388656+0000","last_peered":"2024-03-18T19:03:04.388656+0000","last_clean":"2024-03-18T19:03:04.388656+0000","last_became_active":"2024-03-18T19:03:02.552501+0000","last_became_peered":"2024-03-18T19:03:02.552501+0000","last_unstale":"2024-03-18T19:03:04.388656+0000","last_undegraded":"2024-03-18T19:03:04.388656+0000","last_fullsized":"2024-03-18T19:03:04.388656+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:50:28.343435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"70'2","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389800+0000","last_change":"2024-03-18T19:03:02.132529+0000","last_active":"2024-03-18T19:03:04.389800+0000","last_peered":"2024-03-18T19:03:04.389800+0000","last_clean":"2024-03-18T19:03:04.389800+0000","last_became_active":"2024-03-18T19:03:02.132254+0000","last_became_peered":"2024-03-18T19:03:02.132254+0000","last_unstale":"2024-03-18T19:03:04.389800+0000","last_undegraded":"2024-03-18T19:03:04.389800+0000","last_fullsized":"2024-03-18T19:03:04.389800+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:40.355937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:03.542748+0000","last_change":"2024-03-18T19:03:02.549176+0000","last_active":"2024-03-18T19:03:03.542748+0000","last_peered":"2024-03-18T19:03:03.542748+0000","last_clean":"2024-03-18T19:03:03.542748+0000","last_became_active":"2024-03-18T19:03:02.548876+0000","last_became_peered":"2024-03-18T19:03:02.548876+0000","last_unstale":"2024-03-18T19:03:03.542748+0000","last_undegraded":"2024-03-18T19:03:03.542748+0000","last_fullsized":"2024-03-18T19:03:03.542748+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:46:24.309467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388960+0000","last_change":"2024-03-18T19:03:02.119813+0000","last_active":"2024-03-18T19:03:04.388960+0000","last_peered":"2024-03-18T19:03:04.388960+0000","last_clean":"2024-03-18T19:03:04.388960+0000","last_became_active":"2024-03-18T19:03:02.119573+0000","last_became_peered":"2024-03-18T19:03:02.119573+0000","last_unstale":"2024-03-18T19:03:04.388960+0000","last_undegraded":"2024-03-18T19:03:04.388960+0000","last_fullsized":"2024-03-18T19:03:04.388960+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:04:04.997069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388534+0000","last_change":"2024-03-18T19:03:02.547735+0000","last_active":"2024-03-18T19:03:04.388534+0000","last_peered":"2024-03-18T19:03:04.388534+0000","last_clean":"2024-03-18T19:03:04.388534+0000","last_became_active":"2024-03-18T19:03:02.547480+0000","last_became_peered":"2024-03-18T19:03:02.547480+0000","last_unstale":"2024-03-18T19:03:04.388534+0000","last_undegraded":"2024-03-18T19:03:04.388534+0000","last_fullsized":"2024-03-18T19:03:04.388534+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:33.527267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388337+0000","last_change":"2024-03-18T19:03:02.130993+0000","last_active":"2024-03-18T19:03:04.388337+0000","last_peered":"2024-03-18T19:03:04.388337+0000","last_clean":"2024-03-18T19:03:04.388337+0000","last_became_active":"2024-03-18T19:03:02.130714+0000","last_became_peered":"2024-03-18T19:03:02.130714+0000","last_unstale":"2024-03-18T19:03:04.388337+0000","last_undegraded":"2024-03-18T19:03:04.388337+0000","last_fullsized":"2024-03-18T19:03:04.388337+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:42:17.566346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388398+0000","last_change":"2024-03-18T19:03:02.573329+0000","last_active":"2024-03-18T19:03:04.388398+0000","last_peered":"2024-03-18T19:03:04.388398+0000","last_clean":"2024-03-18T19:03:04.388398+0000","last_became_active":"2024-03-18T19:03:02.573047+0000","last_became_peered":"2024-03-18T19:03:02.573047+0000","last_unstale":"2024-03-18T19:03:04.388398+0000","last_undegraded":"2024-03-18T19:03:04.388398+0000","last_fullsized":"2024-03-18T19:03:04.388398+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:00:56.747749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388619+0000","last_change":"2024-03-18T19:03:02.121344+0000","last_active":"2024-03-18T19:03:04.388619+0000","last_peered":"2024-03-18T19:03:04.388619+0000","last_clean":"2024-03-18T19:03:04.388619+0000","last_became_active":"2024-03-18T19:03:02.121071+0000","last_became_peered":"2024-03-18T19:03:02.121071+0000","last_unstale":"2024-03-18T19:03:04.388619+0000","last_undegraded":"2024-03-18T19:03:04.388619+0000","last_fullsized":"2024-03-18T19:03:04.388619+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:02.410889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388891+0000","last_change":"2024-03-18T19:03:02.559318+0000","last_active":"2024-03-18T19:03:04.388891+0000","last_peered":"2024-03-18T19:03:04.388891+0000","last_clean":"2024-03-18T19:03:04.388891+0000","last_became_active":"2024-03-18T19:03:02.558963+0000","last_became_peered":"2024-03-18T19:03:02.558963+0000","last_unstale":"2024-03-18T19:03:04.388891+0000","last_undegraded":"2024-03-18T19:03:04.388891+0000","last_fullsized":"2024-03-18T19:03:04.388891+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:09:57.608685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388240+0000","last_change":"2024-03-18T19:03:02.119509+0000","last_active":"2024-03-18T19:03:04.388240+0000","last_peered":"2024-03-18T19:03:04.388240+0000","last_clean":"2024-03-18T19:03:04.388240+0000","last_became_active":"2024-03-18T19:03:02.119255+0000","last_became_peered":"2024-03-18T19:03:02.119255+0000","last_unstale":"2024-03-18T19:03:04.388240+0000","last_undegraded":"2024-03-18T19:03:04.388240+0000","last_fullsized":"2024-03-18T19:03:04.388240+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:18:31.958318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388466+0000","last_change":"2024-03-18T19:03:02.565481+0000","last_active":"2024-03-18T19:03:04.388466+0000","last_peered":"2024-03-18T19:03:04.388466+0000","last_clean":"2024-03-18T19:03:04.388466+0000","last_became_active":"2024-03-18T19:03:02.565211+0000","last_became_peered":"2024-03-18T19:03:02.565211+0000","last_unstale":"2024-03-18T19:03:04.388466+0000","last_undegraded":"2024-03-18T19:03:04.388466+0000","last_fullsized":"2024-03-18T19:03:04.388466+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:38:57.388604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389732+0000","last_change":"2024-03-18T19:03:02.125309+0000","last_active":"2024-03-18T19:03:04.389732+0000","last_peered":"2024-03-18T19:03:04.389732+0000","last_clean":"2024-03-18T19:03:04.389732+0000","last_became_active":"2024-03-18T19:03:02.125056+0000","last_became_peered":"2024-03-18T19:03:02.125056+0000","last_unstale":"2024-03-18T19:03:04.389732+0000","last_undegraded":"2024-03-18T19:03:04.389732+0000","last_fullsized":"2024-03-18T19:03:04.389732+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:50:16.410240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388356+0000","last_change":"2024-03-18T19:03:02.553667+0000","last_active":"2024-03-18T19:03:04.388356+0000","last_peered":"2024-03-18T19:03:04.388356+0000","last_clean":"2024-03-18T19:03:04.388356+0000","last_became_active":"2024-03-18T19:03:02.553399+0000","last_became_peered":"2024-03-18T19:03:02.553399+0000","last_unstale":"2024-03-18T19:03:04.388356+0000","last_undegraded":"2024-03-18T19:03:04.388356+0000","last_fullsized":"2024-03-18T19:03:04.388356+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:25.431926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388745+0000","last_change":"2024-03-18T19:03:02.118715+0000","last_active":"2024-03-18T19:03:04.388745+0000","last_peered":"2024-03-18T19:03:04.388745+0000","last_clean":"2024-03-18T19:03:04.388745+0000","last_became_active":"2024-03-18T19:03:02.118429+0000","last_became_peered":"2024-03-18T19:03:02.118429+0000","last_unstale":"2024-03-18T19:03:04.388745+0000","last_undegraded":"2024-03-18T19:03:04.388745+0000","last_fullsized":"2024-03-18T19:03:04.388745+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:31:00.598479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.389036+0000","last_change":"2024-03-18T19:03:02.554918+0000","last_active":"2024-03-18T19:03:04.389036+0000","last_peered":"2024-03-18T19:03:04.389036+0000","last_clean":"2024-03-18T19:03:04.389036+0000","last_became_active":"2024-03-18T19:03:02.554578+0000","last_became_peered":"2024-03-18T19:03:02.554578+0000","last_unstale":"2024-03-18T19:03:04.389036+0000","last_undegraded":"2024-03-18T19:03:04.389036+0000","last_fullsized":"2024-03-18T19:03:04.389036+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:02.098935+0000","last_clean_scrub_stamp":"2024-03-18T19:03:02.098935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:45:08.631461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-18T19:03:04.388594+0000","last_change":"2024-03-18T19:03:02.127347+0000","last_active":"2024-03-18T19:03:04.388594+0000","last_peered":"2024-03-18T19:03:04.388594+0000","last_clean":"2024-03-18T19:03:04.388594+0000","last_became_active":"2024-03-18T19:03:02.127070+0000","last_became_peered":"2024-03-18T19:03:02.127070+0000","last_unstale":"2024-03-18T19:03:04.388594+0000","last_undegraded":"2024-03-18T19:03:04.388594+0000","last_fullsized":"2024-03-18T19:03:04.388594+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:03:01.094095+0000","last_clean_scrub_stamp":"2024-03-18T19:03:01.094095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:58:02.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"60'139","reported_seq":290,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-18T19:03:02.545768+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T19:03:02.545768+0000","last_peered":"2024-03-18T19:03:02.545768+0000","last_clean":"2024-03-18T19:03:02.545768+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T19:03:02.545768+0000","last_undegraded":"2024-03-18T19:03:02.545768+0000","last_fullsized":"2024-03-18T19:03:02.545768+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575009,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54672,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317168,"statfs":{"total":96636764160,"available":96580780032,"internally_reserved":0,"allocated":1441792,"data_stored":1003779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.55700000000000005,"15min":0.55800000000000005},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.247},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57999999999999996,"15min":0.57099999999999995},"min":{"1min":0.372,"5min":0.318,"15min":0.26700000000000002},"max":{"1min":0.64700000000000002,"5min":0.749,"15min":0.80800000000000005},"last":0.58799999999999997}]},{"osd":1,"last update":"Mon Mar 18 19:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.32300000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.71299999999999997,"5min":0.755,"15min":1.484},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.59299999999999997,"15min":0.58399999999999996},"min":{"1min":0.40699999999999997,"5min":0.378,"15min":0.36599999999999999},"max":{"1min":0.73799999999999999,"5min":0.80000000000000004,"15min":0.92000000000000004},"last":0.54500000000000004}]},{"osd":2,"last update":"Mon Mar 18 19:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56000000000000005,"15min":0.57299999999999995},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.27800000000000002},"max":{"1min":0.77200000000000002,"5min":0.78500000000000003,"15min":1.5129999999999999},"last":0.498},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57499999999999996,"15min":0.57999999999999996},"min":{"1min":0.39100000000000001,"5min":0.36799999999999999,"15min":0.30199999999999999},"max":{"1min":0.70899999999999996,"5min":0.84199999999999997,"15min":1.536},"last":0.61199999999999999}]},{"osd":3,"last update":"Mon Mar 18 19:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.55900000000000005,"15min":0.57799999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.311},"max":{"1min":0.72099999999999997,"5min":0.79800000000000004,"15min":1.5489999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.59299999999999997,"15min":0.60299999999999998},"min":{"1min":0.41999999999999998,"5min":0.39800000000000002,"15min":0.38200000000000001},"max":{"1min":0.71899999999999997,"5min":0.81399999999999995,"15min":1.5660000000000001},"last":0.57399999999999995}]},{"osd":4,"last update":"Mon Mar 18 19:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.44900000000000001,"15min":0.45800000000000002},"min":{"1min":0.28899999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.67500000000000004,"5min":0.70799999999999996,"15min":0.77400000000000002},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.47899999999999998,"15min":0.48399999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.64300000000000002,"5min":0.70899999999999996,"15min":0.73199999999999998},"last":0.442}]},{"osd":5,"last update":"Mon Mar 18 19:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.48599999999999999},"min":{"1min":0.27100000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.64100000000000001,"5min":0.73899999999999999,"15min":1.2310000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.499,"5min":0.51200000000000001,"15min":0.52800000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.28799999999999998},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":1.1220000000000001},"last":0.37}]},{"osd":6,"last update":"Mon Mar 18 19:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.55000000000000004},"min":{"1min":0.38100000000000001,"5min":0.376,"15min":0.34300000000000003},"max":{"1min":0.752,"5min":0.752,"15min":1.069},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52600000000000002,"15min":0.54400000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.72799999999999998,"5min":0.752,"15min":0.79600000000000004},"last":0.38800000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575009,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31956,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339884,"statfs":{"total":96636764160,"available":96604041216,"internally_reserved":0,"allocated":860160,"data_stored":420559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56799999999999995,"15min":0.58199999999999996},"min":{"1min":0.46500000000000002,"5min":0.28499999999999998,"15min":0.25700000000000001},"max":{"1min":0.73499999999999999,"5min":0.76800000000000002,"15min":3.274},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56799999999999995,"15min":0.59599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36299999999999999,"15min":0.27400000000000002},"max":{"1min":0.72599999999999998,"5min":0.78700000000000003,"15min":3.3929999999999998},"last":0.443}]},{"osd":1,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56599999999999995,"15min":0.55900000000000005},"min":{"1min":0.438,"5min":0.316,"15min":0.30599999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.94099999999999995},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57099999999999995,"15min":0.56399999999999995},"min":{"1min":0.44400000000000001,"5min":0.32800000000000001,"15min":0.311},"max":{"1min":0.66700000000000004,"5min":0.76800000000000002,"15min":0.90700000000000003},"last":0.45700000000000002}]},{"osd":2,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58299999999999996,"15min":0.56299999999999994},"min":{"1min":0.497,"5min":0.32200000000000001,"15min":0.29999999999999999},"max":{"1min":0.80300000000000005,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.496},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57599999999999996,"15min":0.58899999999999997},"min":{"1min":0.40500000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72199999999999998,"5min":0.76300000000000001,"15min":2.2370000000000001},"last":0.52000000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.41099999999999998,"5min":0.38100000000000001,"15min":0.32100000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":1.0369999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57399999999999995,"15min":0.58399999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36499999999999999},"max":{"1min":0.67800000000000005,"5min":0.79000000000000004,"15min":1.595},"last":0.50800000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.439,"15min":0.45600000000000002},"min":{"1min":0.32400000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54900000000000004,"5min":0.68700000000000006,"15min":0.76300000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.46800000000000003,"15min":0.49399999999999999},"min":{"1min":0.34899999999999998,"5min":0.32700000000000001,"15min":0.29299999999999998},"max":{"1min":0.56100000000000005,"5min":0.67800000000000005,"15min":1.8819999999999999},"last":0.38600000000000001}]},{"osd":5,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46999999999999997,"15min":0.48399999999999999},"min":{"1min":0.34200000000000003,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68700000000000006,"5min":0.73699999999999999,"15min":1.022},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.51100000000000001,"15min":0.51800000000000002},"min":{"1min":0.38100000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64700000000000002,"5min":0.70499999999999996,"15min":0.81599999999999995},"last":0.40999999999999998}]},{"osd":7,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51300000000000001,"15min":0.53400000000000003},"min":{"1min":0.376,"5min":0.34100000000000003,"15min":0.313},"max":{"1min":0.67900000000000005,"5min":0.72299999999999998,"15min":1.532},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51900000000000002,"15min":0.54500000000000004},"min":{"1min":0.376,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71299999999999997,"5min":0.79100000000000004,"15min":1.978},"last":0.36899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575009,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":860160,"data_stored":420067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.38600000000000001,"15min":0.36699999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.215},"max":{"1min":0.51500000000000001,"5min":0.60499999999999998,"15min":0.83599999999999997},"last":0.373},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.42499999999999999,"15min":0.40999999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23400000000000001},"max":{"1min":0.60199999999999998,"5min":0.68999999999999995,"15min":0.90400000000000003},"last":0.27600000000000002}]},{"osd":2,"last update":"Mon Mar 18 19:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.41799999999999998,"15min":0.39700000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.23300000000000001},"max":{"1min":0.51100000000000001,"5min":0.64000000000000001,"15min":0.88400000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.45600000000000002,"15min":0.438},"min":{"1min":0.312,"5min":0.312,"15min":0.23499999999999999},"max":{"1min":0.53900000000000003,"5min":0.70799999999999996,"15min":0.94999999999999996},"last":0.33000000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.47999999999999998,"15min":0.45600000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27700000000000002},"max":{"1min":0.63900000000000001,"5min":0.65900000000000003,"15min":0.86799999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.42999999999999999,"15min":0.41499999999999998},"min":{"1min":0.30199999999999999,"5min":0.27700000000000002,"15min":0.25600000000000001},"max":{"1min":0.57099999999999995,"5min":0.63900000000000001,"15min":0.93100000000000005},"last":0.30499999999999999}]},{"osd":4,"last update":"Mon Mar 18 19:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.65100000000000002,"15min":0.63300000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.41499999999999998},"max":{"1min":0.80900000000000005,"5min":0.92600000000000005,"15min":1.052},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.65500000000000003,"15min":0.63700000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.34499999999999997},"max":{"1min":0.81599999999999995,"5min":0.96099999999999997,"15min":1.0269999999999999},"last":0.52700000000000002}]},{"osd":5,"last update":"Mon Mar 18 19:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.65600000000000003,"15min":0.64000000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.38600000000000001},"max":{"1min":0.748,"5min":0.93999999999999995,"15min":1.0760000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.67800000000000005,"15min":0.65500000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40999999999999998},"max":{"1min":0.78900000000000003,"5min":0.995,"15min":1.1319999999999999},"last":0.496}]},{"osd":6,"last update":"Mon Mar 18 19:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.65500000000000003,"15min":0.63200000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.40300000000000002},"max":{"1min":0.83999999999999997,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.68799999999999994,"15min":0.66700000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.39100000000000001},"max":{"1min":0.74399999999999999,"5min":0.97499999999999998,"15min":1.0089999999999999},"last":0.58099999999999996}]},{"osd":7,"last update":"Mon Mar 18 19:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.68000000000000005,"15min":0.65700000000000003},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.36699999999999999},"max":{"1min":0.88100000000000001,"5min":0.94799999999999995,"15min":1.0660000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.69099999999999995,"15min":0.65900000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.39900000000000002},"max":{"1min":0.871,"5min":0.94299999999999995,"15min":1.1499999999999999},"last":0.55400000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575010,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1454080,"data_stored":1010331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.39400000000000002,"15min":0.373},"min":{"1min":0.26200000000000001,"5min":0.26100000000000001,"15min":0.192},"max":{"1min":0.57399999999999995,"5min":0.621,"15min":0.621},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.42499999999999999,"15min":0.40100000000000002},"min":{"1min":0.317,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.61699999999999999,"5min":0.63200000000000001,"15min":0.66400000000000003},"last":0.441}]},{"osd":2,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.42699999999999999,"15min":0.40799999999999997},"min":{"1min":0.29499999999999998,"5min":0.22,"15min":0.20999999999999999},"max":{"1min":0.60499999999999998,"5min":0.61699999999999999,"15min":0.627},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.437,"15min":0.41999999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.626,"5min":0.626,"15min":0.64800000000000002},"last":0.39500000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.41999999999999998,"15min":0.41899999999999998},"min":{"1min":0.26900000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.56100000000000005,"5min":0.64100000000000001,"15min":5.7359999999999998},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.33900000000000002,"5min":0.307,"15min":0.27700000000000002},"max":{"1min":0.64100000000000001,"5min":0.69899999999999995,"15min":5.9329999999999998},"last":0.41399999999999998}]},{"osd":4,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65100000000000002,"15min":0.627},"min":{"1min":0.47999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.89100000000000001,"5min":1.0229999999999999,"15min":1.22},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65900000000000003,"15min":0.63100000000000001},"min":{"1min":0.48999999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.88,"5min":0.95399999999999996,"15min":1.1020000000000001},"last":0.64600000000000002}]},{"osd":5,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65800000000000003,"15min":0.64200000000000002},"min":{"1min":0.49399999999999999,"5min":0.374,"15min":0.35799999999999998},"max":{"1min":0.90500000000000003,"5min":0.98099999999999998,"15min":2.9649999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66600000000000004,"15min":0.64300000000000002},"min":{"1min":0.45300000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.84299999999999997,"5min":0.90000000000000002,"15min":1.2609999999999999},"last":0.54300000000000004}]},{"osd":6,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66400000000000003,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85299999999999998,"5min":1.0049999999999999,"15min":1.133},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67600000000000005,"15min":0.66800000000000004},"min":{"1min":0.41199999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90800000000000003,"5min":1.0369999999999999,"15min":2.9300000000000002},"last":0.57799999999999996}]},{"osd":7,"last update":"Mon Mar 18 19:02:07 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66700000000000004,"15min":0.65200000000000002},"min":{"1min":0.50600000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.88700000000000001,"5min":0.96099999999999997,"15min":1.2849999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.68500000000000005,"15min":0.66300000000000003},"min":{"1min":0.47899999999999998,"5min":0.44,"15min":0.38500000000000001},"max":{"1min":0.79800000000000004,"5min":0.96299999999999997,"15min":1.002},"last":0.60699999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575010,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58200,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94313640,"statfs":{"total":96636764160,"available":96577167360,"internally_reserved":0,"allocated":864256,"data_stored":421471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.378,"15min":0.433},"min":{"1min":0.22500000000000001,"5min":0.20799999999999999,"15min":0.17799999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":9.1219999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.46100000000000002,"15min":0.51300000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.66600000000000004,"5min":0.71799999999999997,"15min":9.1560000000000006},"last":0.64300000000000002}]},{"osd":1,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.47999999999999998,"15min":0.53000000000000003},"min":{"1min":0.29399999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":9.2460000000000004},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40799999999999997,"15min":0.48299999999999998},"min":{"1min":0.245,"5min":0.216,"15min":0.216},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":12.932},"last":0.66700000000000004}]},{"osd":3,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.42099999999999999,"15min":0.46899999999999997},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.22500000000000001},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":9.4120000000000008},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.51100000000000001,"15min":0.55700000000000005},"min":{"1min":0.34200000000000003,"5min":0.29499999999999998,"15min":0.27700000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":9.2680000000000007},"last":0.80100000000000005}]},{"osd":4,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.622,"15min":0.60799999999999998},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.85499999999999998,"5min":1,"15min":1},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.69599999999999995},"min":{"1min":0.35899999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":9.3190000000000008},"last":0.69199999999999995}]},{"osd":5,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.65300000000000002,"15min":0.70699999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.309},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":9.1940000000000008},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.65400000000000003,"15min":0.629},"min":{"1min":0.375,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.948,"5min":1.948,"15min":1.948},"last":0.92200000000000004}]},{"osd":6,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.67500000000000004,"15min":0.71399999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.9470000000000001,"5min":2.9470000000000001,"15min":9.3559999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.69099999999999995,"15min":0.749},"min":{"1min":0.48199999999999998,"5min":0.38100000000000001,"15min":0.37},"max":{"1min":0.98599999999999999,"5min":1.083,"15min":9.2219999999999995},"last":0.98599999999999999}]},{"osd":7,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65100000000000002,"15min":0.64100000000000001},"min":{"1min":0.40999999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.67500000000000004,"15min":0.72199999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.34599999999999997},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":9.3829999999999991},"last":0.76100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575010,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36056,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335784,"statfs":{"total":96636764160,"available":96599842816,"internally_reserved":0,"allocated":856064,"data_stored":414873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.40500000000000003,"15min":0.38600000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.81100000000000005,"5min":1.204,"15min":1.4550000000000001},"last":0.309},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.434,"15min":0.40100000000000002},"min":{"1min":0.29799999999999999,"5min":0.29499999999999998,"15min":0.23000000000000001},"max":{"1min":1.137,"5min":1.137,"15min":1.5169999999999999},"last":0.36399999999999999}]},{"osd":1,"last update":"Mon Mar 18 19:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.42099999999999999,"15min":0.40300000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.221},"max":{"1min":0.999,"5min":1.1220000000000001,"15min":1.2909999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.47099999999999997,"15min":0.44600000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.245},"max":{"1min":1.093,"5min":1.1080000000000001,"15min":1.355},"last":0.42799999999999999}]},{"osd":2,"last update":"Mon Mar 18 19:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.47199999999999998,"15min":0.44800000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":1.0369999999999999,"5min":1.1879999999999999,"15min":1.3160000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.28599999999999998},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.4810000000000001},"last":0.38300000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66400000000000003,"15min":0.624},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.35199999999999998},"max":{"1min":1.4219999999999999,"5min":1.871,"15min":1.871},"last":0.502},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.66300000000000003,"15min":0.63800000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0700000000000001,"5min":1.764,"15min":1.764},"last":0.45700000000000002}]},{"osd":5,"last update":"Mon Mar 18 19:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.70199999999999996,"15min":0.66400000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":2.5329999999999999,"5min":2.5329999999999999,"15min":2.5329999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.71899999999999997,"15min":0.69099999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.437},"max":{"1min":1.5409999999999999,"5min":1.794,"15min":1.794},"last":0.53500000000000003}]},{"osd":6,"last update":"Mon Mar 18 19:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.71799999999999997,"15min":0.66500000000000004},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.377},"max":{"1min":2.7389999999999999,"5min":2.7389999999999999,"15min":2.7389999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71799999999999997,"15min":0.68500000000000005},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.45400000000000001},"max":{"1min":1.159,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.56000000000000005}]},{"osd":7,"last update":"Mon Mar 18 19:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.69899999999999995,"15min":0.67800000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.3959999999999999,"5min":1.732,"15min":1.732},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.72299999999999998,"15min":0.66400000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.40500000000000003},"max":{"1min":2.7050000000000001,"5min":2.7050000000000001,"15min":2.7050000000000001},"last":0.54400000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575010,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":843776,"data_stored":413869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.58199999999999996,"15min":0.55100000000000005},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.246},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.33500000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68500000000000005,"5min":0.77600000000000002,"15min":1.0860000000000001},"last":0.53400000000000003}]},{"osd":1,"last update":"Mon Mar 18 19:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.58999999999999997,"15min":0.58199999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.316},"max":{"1min":0.749,"5min":0.749,"15min":1.1679999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.57099999999999995,"15min":0.56699999999999995},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66800000000000004,"5min":0.79100000000000004,"15min":1.218},"last":0.60999999999999999}]},{"osd":2,"last update":"Mon Mar 18 19:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57399999999999995,"15min":0.56299999999999994},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.36799999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78100000000000003,"5min":0.81100000000000005,"15min":1.204},"last":0.442}]},{"osd":3,"last update":"Mon Mar 18 19:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.59099999999999997,"15min":0.58299999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.755,"5min":0.82099999999999995,"15min":1.1799999999999999},"last":0.5},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.51700000000000002}]},{"osd":5,"last update":"Mon Mar 18 19:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.45800000000000002,"15min":0.47599999999999998},"min":{"1min":0.32000000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.78900000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.47299999999999998,"15min":0.496},"min":{"1min":0.35599999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64400000000000002,"5min":0.69699999999999995,"15min":0.77800000000000002},"last":0.55600000000000005}]},{"osd":6,"last update":"Mon Mar 18 19:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47799999999999998,"15min":0.496},"min":{"1min":0.35299999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65100000000000002,"5min":0.71999999999999997,"15min":0.81799999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.51300000000000001,"15min":0.54000000000000004},"min":{"1min":0.40300000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72499999999999998,"5min":0.77700000000000002,"15min":0.88900000000000001},"last":0.48799999999999999}]},{"osd":7,"last update":"Mon Mar 18 19:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51000000000000001,"15min":0.53100000000000003},"min":{"1min":0.40699999999999997,"5min":0.35199999999999998,"15min":0.35099999999999998},"max":{"1min":0.68799999999999994,"5min":0.73599999999999999,"15min":0.75800000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54400000000000004,"15min":0.56299999999999994},"min":{"1min":0.44,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.73799999999999999,"5min":0.79700000000000004,"15min":0.91900000000000004},"last":0.59499999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575009,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31940,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339900,"statfs":{"total":96636764160,"available":96604057600,"internally_reserved":0,"allocated":839680,"data_stored":413287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.55700000000000005,"15min":0.60499999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.24399999999999999},"max":{"1min":0.70799999999999996,"5min":0.76800000000000002,"15min":6.3079999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57199999999999995,"15min":0.56499999999999995},"min":{"1min":0.33700000000000002,"5min":0.32500000000000001,"15min":0.25800000000000001},"max":{"1min":0.67000000000000004,"5min":0.81799999999999995,"15min":1.208},"last":0.56100000000000005}]},{"osd":1,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.57499999999999996,"15min":0.56599999999999995},"min":{"1min":0.36799999999999999,"5min":0.33400000000000002,"15min":0.28399999999999997},"max":{"1min":0.69899999999999995,"5min":0.80600000000000005,"15min":1.196},"last":0.433},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58799999999999997,"15min":0.621},"min":{"1min":0.38400000000000001,"5min":0.36599999999999999,"15min":0.29799999999999999},"max":{"1min":0.749,"5min":0.78400000000000003,"15min":6.2549999999999999},"last":0.58099999999999996}]},{"osd":2,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.57899999999999996,"15min":0.61899999999999999},"min":{"1min":0.34799999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.78700000000000003,"5min":0.79800000000000004,"15min":6.2999999999999998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57399999999999995,"15min":0.71299999999999997},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":0.747,"5min":0.76900000000000002,"15min":17.387},"last":0.54500000000000004}]},{"osd":3,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.59399999999999997,"15min":0.63100000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":6.3330000000000002},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60199999999999998,"15min":0.60299999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":1.2390000000000001},"last":0.60599999999999998}]},{"osd":4,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.30299999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.503},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.497},"min":{"1min":0.36699999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.52300000000000002}]},{"osd":6,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.48399999999999999,"15min":0.48999999999999999},"min":{"1min":0.28499999999999998,"5min":0.245,"15min":0.245},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.39100000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.59499999999999997}]},{"osd":7,"last update":"Mon Mar 18 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.52900000000000003,"15min":0.53800000000000003},"min":{"1min":0.32700000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":1.0449999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.55800000000000005,"15min":0.56100000000000005},"min":{"1min":0.40500000000000003,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.628}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T19:03:07.095 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T19:03:07.095 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-18T19:03:07.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.266+0000 7fd3160d9700 1 Processor -- start 2024-03-18T19:03:07.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.266+0000 7fd3160d9700 1 -- start start 2024-03-18T19:03:07.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.267+0000 7fd3160d9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd3100b0500 0x7fd3100cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:07.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.267+0000 7fd3160d9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3100b0e20 0x7fd3100b12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:07.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.267+0000 7fd3160d9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd31015ef80 con 0x7fd31015eba0 2024-03-18T19:03:07.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.267+0000 7fd3160d9700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3100cc9a0 con 0x7fd3100b0500 2024-03-18T19:03:07.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.267+0000 7fd3160d9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3100ccc80 con 0x7fd3100b0e20 2024-03-18T19:03:07.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.267+0000 7fd30f7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd31015eba0 0x7fd3100cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60186/0 (socket says 172.21.15.1:60186) 2024-03-18T19:03:07.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.267+0000 7fd30f7fe700 1 -- 172.21.15.1:0/921092817 learned_addr learned my addr 172.21.15.1:0/921092817 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:03:07.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.267+0000 7fd30ffff700 1 --2- 172.21.15.1:0/921092817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3100b0e20 0x7fd3100b12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:07.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.267+0000 7fd30effd700 1 --2- 172.21.15.1:0/921092817 >> v2:172.21.15.165:3300/0 conn(0x7fd3100b0500 0x7fd3100cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:07.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.267+0000 7fd30e7fc700 1 -- 172.21.15.1:0/921092817 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571163166 0 0) 0x7fd31015ef80 con 0x7fd31015eba0 2024-03-18T19:03:07.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.267+0000 7fd30e7fc700 1 -- 172.21.15.1:0/921092817 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2fc0035b0 con 0x7fd31015eba0 2024-03-18T19:03:07.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30e7fc700 1 -- 172.21.15.1:0/921092817 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2618843120 0 0) 0x7fd2fc0035b0 con 0x7fd31015eba0 2024-03-18T19:03:07.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30e7fc700 1 -- 172.21.15.1:0/921092817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3100b0e20 msgr2=0x7fd3100b12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:07.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30e7fc700 1 --2- 172.21.15.1:0/921092817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3100b0e20 0x7fd3100b12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30e7fc700 1 -- 172.21.15.1:0/921092817 >> v2:172.21.15.165:3300/0 conn(0x7fd3100b0500 msgr2=0x7fd3100cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:07.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30e7fc700 1 --2- 172.21.15.1:0/921092817 >> v2:172.21.15.165:3300/0 conn(0x7fd3100b0500 0x7fd3100cfa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30e7fc700 1 -- 172.21.15.1:0/921092817 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3100ccf60 con 0x7fd31015eba0 2024-03-18T19:03:07.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30e7fc700 1 -- 172.21.15.1:0/921092817 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fd300001670 con 0x7fd31015eba0 2024-03-18T19:03:07.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30e7fc700 1 -- 172.21.15.1:0/921092817 >> v1:172.21.15.1:6789/0 conn(0x7fd31015eba0 legacy=0x7fd3100cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:07.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30e7fc700 1 --2- 172.21.15.1:0/921092817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3100b0e20 0x7fd2fc003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:07.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30e7fc700 1 --2- 172.21.15.1:0/921092817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2fc003f80 0x7fd2fc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:07.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30e7fc700 1 --2- 172.21.15.1:0/921092817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2fc004d90 0x7fd2fc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:07.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30e7fc700 1 -- 172.21.15.1:0/921092817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3100cc9a0 con 0x7fd2fc003f80 2024-03-18T19:03:07.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30e7fc700 1 -- 172.21.15.1:0/921092817 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3100ccc80 con 0x7fd2fc004d90 2024-03-18T19:03:07.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30e7fc700 1 -- 172.21.15.1:0/921092817 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc00ba60 con 0x7fd3100b0e20 2024-03-18T19:03:07.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30effd700 1 --2- 172.21.15.1:0/921092817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2fc003f80 0x7fd2fc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:07.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30f7fe700 1 --2- 172.21.15.1:0/921092817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3100b0e20 0x7fd2fc003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:07.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30ffff700 1 --2- 172.21.15.1:0/921092817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2fc004d90 0x7fd2fc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:07.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30effd700 1 -- 172.21.15.1:0/921092817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3100b0e20 msgr2=0x7fd2fc003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:07.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30effd700 1 --2- 172.21.15.1:0/921092817 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3100b0e20 0x7fd2fc003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30effd700 1 -- 172.21.15.1:0/921092817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2fc004d90 msgr2=0x7fd2fc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:07.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30effd700 1 --2- 172.21.15.1:0/921092817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2fc004d90 0x7fd2fc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30effd700 1 -- 172.21.15.1:0/921092817 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc0035b0 con 0x7fd2fc003f80 2024-03-18T19:03:07.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30effd700 1 --2- 172.21.15.1:0/921092817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2fc003f80 0x7fd2fc004850 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009b90 tx=0x7fd2fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=924f1de4c72d34d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:07.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.268+0000 7fd30e7fc700 1 -- 172.21.15.1:0/921092817 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f800e6c0 con 0x7fd2fc003f80 2024-03-18T19:03:07.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.269+0000 7fd30e7fc700 1 -- 172.21.15.1:0/921092817 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f800e820 con 0x7fd2fc003f80 2024-03-18T19:03:07.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.269+0000 7fd30e7fc700 1 -- 172.21.15.1:0/921092817 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f800eb90 con 0x7fd2fc003f80 2024-03-18T19:03:07.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.269+0000 7fd3160d9700 1 -- 172.21.15.1:0/921092817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2fc003f80 msgr2=0x7fd2fc004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:07.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.269+0000 7fd3160d9700 1 --2- 172.21.15.1:0/921092817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2fc003f80 0x7fd2fc004850 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009b90 tx=0x7fd2fc005880 comp rx=0 tx=0).stop 2024-03-18T19:03:07.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.269+0000 7fd30ffff700 1 -- 172.21.15.1:0/921092817 reap_dead start 2024-03-18T19:03:07.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.269+0000 7fd3160d9700 1 -- 172.21.15.1:0/921092817 shutdown_connections 2024-03-18T19:03:07.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.269+0000 7fd3160d9700 1 -- 172.21.15.1:0/921092817 >> 172.21.15.1:0/921092817 conn(0x7fd3100b2bd0 msgr2=0x7fd3100b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:07.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.269+0000 7fd3160d9700 1 -- 172.21.15.1:0/921092817 shutdown_connections 2024-03-18T19:03:07.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.269+0000 7fd3160d9700 1 -- 172.21.15.1:0/921092817 wait complete. 2024-03-18T19:03:07.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.270+0000 7fd3160d9700 1 Processor -- start 2024-03-18T19:03:07.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.270+0000 7fd3160d9700 1 -- start start 2024-03-18T19:03:07.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.271+0000 7fd3160d9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2fc003f80 0x7fd310151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:07.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.271+0000 7fd3160d9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3100c81b0 0x7fd31006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:07.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.271+0000 7fd3160d9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3100c8630 0x7fd31006f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:07.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.271+0000 7fd3160d9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3100c4de0 con 0x7fd3100c8630 2024-03-18T19:03:07.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.271+0000 7fd3160d9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3100c8b20 con 0x7fd3100c81b0 2024-03-18T19:03:07.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.271+0000 7fd3160d9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3100c8e00 con 0x7fd2fc003f80 2024-03-18T19:03:07.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.271+0000 7fd30ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3100c8630 0x7fd31006f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:07.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.271+0000 7fd30ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3100c8630 0x7fd31006f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53912/0 (socket says 172.21.15.1:53912) 2024-03-18T19:03:07.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.271+0000 7fd30f7fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2fc003f80 0x7fd310151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:07.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.271+0000 7fd30ffff700 1 -- 172.21.15.1:0/2827547303 learned_addr learned my addr 172.21.15.1:0/2827547303 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:07.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.271+0000 7fd30effd700 1 --2- 172.21.15.1:0/2827547303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3100c81b0 0x7fd31006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:07.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.272+0000 7fd30ffff700 1 -- 172.21.15.1:0/2827547303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2fc003f80 msgr2=0x7fd310151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:07.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.272+0000 7fd30ffff700 1 --2- 172.21.15.1:0/2827547303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2fc003f80 0x7fd310151580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.272+0000 7fd30ffff700 1 -- 172.21.15.1:0/2827547303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3100c81b0 msgr2=0x7fd31006f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:07.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.272+0000 7fd30ffff700 1 --2- 172.21.15.1:0/2827547303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3100c81b0 0x7fd31006f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.272+0000 7fd30ffff700 1 -- 172.21.15.1:0/2827547303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3100c90e0 con 0x7fd3100c8630 2024-03-18T19:03:07.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.272+0000 7fd30ffff700 1 --2- 172.21.15.1:0/2827547303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3100c8630 0x7fd31006f650 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fd3100cd870 tx=0x7fd308011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:07.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.272+0000 7fd307fff700 1 -- 172.21.15.1:0/2827547303 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30800e040 con 0x7fd3100c8630 2024-03-18T19:03:07.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.272+0000 7fd307fff700 1 -- 172.21.15.1:0/2827547303 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30800f040 con 0x7fd3100c8630 2024-03-18T19:03:07.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.272+0000 7fd307fff700 1 -- 172.21.15.1:0/2827547303 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30800fc40 con 0x7fd3100c8630 2024-03-18T19:03:07.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.272+0000 7fd3160d9700 1 -- 172.21.15.1:0/2827547303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3100c4a70 con 0x7fd3100c8630 2024-03-18T19:03:07.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.272+0000 7fd3160d9700 1 -- 172.21.15.1:0/2827547303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd310154a10 con 0x7fd3100c8630 2024-03-18T19:03:07.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.273+0000 7fd3160d9700 1 -- 172.21.15.1:0/2827547303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3100c4f50 con 0x7fd3100c8630 2024-03-18T19:03:07.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.273+0000 7fd307fff700 1 -- 172.21.15.1:0/2827547303 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd308015430 con 0x7fd3100c8630 2024-03-18T19:03:07.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.274+0000 7fd307fff700 1 --2- 172.21.15.1:0/2827547303 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd2f006adb0 0x7fd2f006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:07.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.274+0000 7fd307fff700 1 -- 172.21.15.1:0/2827547303 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd308089a50 con 0x7fd3100c8630 2024-03-18T19:03:07.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.274+0000 7fd30f7fe700 1 --2- 172.21.15.1:0/2827547303 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd2f006adb0 0x7fd2f006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:07.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.274+0000 7fd30f7fe700 1 --2- 172.21.15.1:0/2827547303 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd2f006adb0 0x7fd2f006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd300000c00 tx=0x7fd300007980 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:07.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.278+0000 7fd307fff700 1 -- 172.21.15.1:0/2827547303 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd308054960 con 0x7fd3100c8630 2024-03-18T19:03:07.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.566+0000 7fd3160d9700 1 -- 172.21.15.1:0/2827547303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd3100676a0 con 0x7fd3100c8630 2024-03-18T19:03:07.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.569+0000 7fd307fff700 1 -- 172.21.15.1:0/2827547303 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+0 (secure 0 0 0) 0x7fd308057f80 con 0x7fd3100c8630 2024-03-18T19:03:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.571+0000 7fd3160d9700 1 -- 172.21.15.1:0/2827547303 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd2f006adb0 msgr2=0x7fd2f006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.571+0000 7fd3160d9700 1 --2- 172.21.15.1:0/2827547303 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd2f006adb0 0x7fd2f006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd300000c00 tx=0x7fd300007980 comp rx=0 tx=0).stop 2024-03-18T19:03:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.571+0000 7fd3160d9700 1 -- 172.21.15.1:0/2827547303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3100c8630 msgr2=0x7fd31006f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.571+0000 7fd3160d9700 1 --2- 172.21.15.1:0/2827547303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3100c8630 0x7fd31006f650 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fd3100cd870 tx=0x7fd308011ed0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.571+0000 7fd3160d9700 1 -- 172.21.15.1:0/2827547303 shutdown_connections 2024-03-18T19:03:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.571+0000 7fd3160d9700 1 --2- 172.21.15.1:0/2827547303 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd2f006adb0 0x7fd2f006d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.571+0000 7fd3160d9700 1 --2- 172.21.15.1:0/2827547303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2fc003f80 0x7fd310151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.571+0000 7fd3160d9700 1 --2- 172.21.15.1:0/2827547303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3100c8630 0x7fd31006f650 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.571+0000 7fd3160d9700 1 --2- 172.21.15.1:0/2827547303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3100c81b0 0x7fd31006f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.571+0000 7fd3160d9700 1 -- 172.21.15.1:0/2827547303 >> 172.21.15.1:0/2827547303 conn(0x7fd3100b2bd0 msgr2=0x7fd31015bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.571+0000 7fd3160d9700 1 -- 172.21.15.1:0/2827547303 shutdown_connections 2024-03-18T19:03:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.571+0000 7fd3160d9700 1 -- 172.21.15.1:0/2827547303 wait complete. 2024-03-18T19:03:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-18T19:03:07.579 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:03:07.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.709+0000 7f220a77a700 1 Processor -- start 2024-03-18T19:03:07.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.709+0000 7f220a77a700 1 -- start start 2024-03-18T19:03:07.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f220a77a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2204146190 0x7f220406a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:07.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f220a77a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f22041465a0 0x7f220406ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:07.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f220a77a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2204146af0 0x7f2204071650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:07.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f220a77a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f220406f850 con 0x7f22041465a0 2024-03-18T19:03:07.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f220a77a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f220406ebd0 con 0x7f2204146190 2024-03-18T19:03:07.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f220a77a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220406eeb0 con 0x7f2204146af0 2024-03-18T19:03:07.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f22037fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f22041465a0 0x7f220406ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:07.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f22037fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f22041465a0 0x7f220406ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53926/0 (socket says 172.21.15.1:53926) 2024-03-18T19:03:07.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f22037fe700 1 -- 172.21.15.1:0/1868575032 learned_addr learned my addr 172.21.15.1:0/1868575032 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:07.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f2203fff700 1 --2- 172.21.15.1:0/1868575032 >> v2:172.21.15.165:3300/0 conn(0x7f2204146190 0x7f220406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:07.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f2208d17700 1 --2- 172.21.15.1:0/1868575032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2204146af0 0x7f2204071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:07.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f22037fe700 1 -- 172.21.15.1:0/1868575032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2204146af0 msgr2=0x7f2204071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:07.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f22037fe700 1 --2- 172.21.15.1:0/1868575032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2204146af0 0x7f2204071650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f22037fe700 1 -- 172.21.15.1:0/1868575032 >> v2:172.21.15.165:3300/0 conn(0x7f2204146190 msgr2=0x7f220406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:07.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f22037fe700 1 --2- 172.21.15.1:0/1868575032 >> v2:172.21.15.165:3300/0 conn(0x7f2204146190 0x7f220406a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f22037fe700 1 -- 172.21.15.1:0/1868575032 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220406f190 con 0x7f22041465a0 2024-03-18T19:03:07.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f22037fe700 1 --2- 172.21.15.1:0/1868575032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22041465a0 0x7f220406ac90 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f220406e8f0 tx=0x7f21f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e50d7414984dc883 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f2202ffd700 1 -- 172.21.15.1:0/1868575032 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f800b870 con 0x7f22041465a0 2024-03-18T19:03:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f2202ffd700 1 -- 172.21.15.1:0/1868575032 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f800b9d0 con 0x7f22041465a0 2024-03-18T19:03:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.710+0000 7f2202ffd700 1 -- 172.21.15.1:0/1868575032 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f800bd60 con 0x7f22041465a0 2024-03-18T19:03:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.711+0000 7f220a77a700 1 -- 172.21.15.1:0/1868575032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22041465a0 msgr2=0x7f220406ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.711+0000 7f220a77a700 1 --2- 172.21.15.1:0/1868575032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22041465a0 0x7f220406ac90 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f220406e8f0 tx=0x7f21f8009bd0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.711+0000 7f220a77a700 1 -- 172.21.15.1:0/1868575032 shutdown_connections 2024-03-18T19:03:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.711+0000 7f220a77a700 1 --2- 172.21.15.1:0/1868575032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2204146af0 0x7f2204071650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.711+0000 7f220a77a700 1 --2- 172.21.15.1:0/1868575032 >> v2:172.21.15.165:3300/0 conn(0x7f2204146190 0x7f220406a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.711+0000 7f220a77a700 1 --2- 172.21.15.1:0/1868575032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22041465a0 0x7f220406ac90 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.711+0000 7f220a77a700 1 -- 172.21.15.1:0/1868575032 >> 172.21.15.1:0/1868575032 conn(0x7f22040b1d20 msgr2=0x7f22040b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.711+0000 7f220a77a700 1 -- 172.21.15.1:0/1868575032 shutdown_connections 2024-03-18T19:03:07.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.711+0000 7f220a77a700 1 -- 172.21.15.1:0/1868575032 wait complete. 2024-03-18T19:03:07.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.712+0000 7f220a77a700 1 Processor -- start 2024-03-18T19:03:07.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.712+0000 7f220a77a700 1 -- start start 2024-03-18T19:03:07.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.713+0000 7f220a77a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f22041465a0 0x7f22041f2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:07.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.713+0000 7f220a77a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2204146af0 0x7f22041402c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:07.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.713+0000 7f220a77a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2204140800 0x7f22041f3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:07.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.713+0000 7f220a77a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22041418e0 con 0x7f2204146af0 2024-03-18T19:03:07.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.713+0000 7f220a77a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2204141040 con 0x7f2204140800 2024-03-18T19:03:07.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.713+0000 7f220a77a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2204141320 con 0x7f22041465a0 2024-03-18T19:03:07.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.713+0000 7f22037fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2204146af0 0x7f22041402c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:07.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.713+0000 7f22037fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2204146af0 0x7f22041402c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53934/0 (socket says 172.21.15.1:53934) 2024-03-18T19:03:07.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.713+0000 7f22037fe700 1 -- 172.21.15.1:0/1168019159 learned_addr learned my addr 172.21.15.1:0/1168019159 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:07.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.713+0000 7f2203fff700 1 --2- 172.21.15.1:0/1168019159 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f22041465a0 0x7f22041f2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:07.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.713+0000 7f2208d17700 1 --2- 172.21.15.1:0/1168019159 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2204140800 0x7f22041f3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:07.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.713+0000 7f22037fe700 1 -- 172.21.15.1:0/1168019159 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f22041465a0 msgr2=0x7f22041f2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:07.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.713+0000 7f22037fe700 1 --2- 172.21.15.1:0/1168019159 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f22041465a0 0x7f22041f2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.713+0000 7f22037fe700 1 -- 172.21.15.1:0/1168019159 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2204140800 msgr2=0x7f22041f3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:07.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.713+0000 7f22037fe700 1 --2- 172.21.15.1:0/1168019159 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2204140800 0x7f22041f3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.713+0000 7f22037fe700 1 -- 172.21.15.1:0/1168019159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f8009550 con 0x7f2204146af0 2024-03-18T19:03:07.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.714+0000 7f2208d17700 1 --2- 172.21.15.1:0/1168019159 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2204140800 0x7f22041f3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:03:07.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.714+0000 7f22037fe700 1 --2- 172.21.15.1:0/1168019159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2204146af0 0x7f22041402c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f2204067530 tx=0x7f21f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:07.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.714+0000 7f2200ff9700 1 -- 172.21.15.1:0/1168019159 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f8004310 con 0x7f2204146af0 2024-03-18T19:03:07.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.714+0000 7f2200ff9700 1 -- 172.21.15.1:0/1168019159 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f801f070 con 0x7f2204146af0 2024-03-18T19:03:07.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.714+0000 7f220a77a700 1 -- 172.21.15.1:0/1168019159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2204141600 con 0x7f2204146af0 2024-03-18T19:03:07.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.714+0000 7f2200ff9700 1 -- 172.21.15.1:0/1168019159 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f80036a0 con 0x7f2204146af0 2024-03-18T19:03:07.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.714+0000 7f220a77a700 1 -- 172.21.15.1:0/1168019159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2204140cb0 con 0x7f2204146af0 2024-03-18T19:03:07.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.715+0000 7f220a77a700 1 -- 172.21.15.1:0/1168019159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22041f3660 con 0x7f2204146af0 2024-03-18T19:03:07.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.715+0000 7f2200ff9700 1 -- 172.21.15.1:0/1168019159 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f21f8021030 con 0x7f2204146af0 2024-03-18T19:03:07.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.716+0000 7f2200ff9700 1 --2- 172.21.15.1:0/1168019159 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f21f006adb0 0x7f21f006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:07.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.716+0000 7f2203fff700 1 --2- 172.21.15.1:0/1168019159 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f21f006adb0 0x7f21f006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:07.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.716+0000 7f2200ff9700 1 -- 172.21.15.1:0/1168019159 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f21f808a8f0 con 0x7f2204146af0 2024-03-18T19:03:07.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.716+0000 7f2203fff700 1 --2- 172.21.15.1:0/1168019159 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f21f006adb0 0x7f21f006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f21f4006fd0 tx=0x7f21f4008040 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:07.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.720+0000 7f2200ff9700 1 -- 172.21.15.1:0/1168019159 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f22041f3660 con 0x7f2204146af0 2024-03-18T19:03:07.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.971+0000 7f220a77a700 1 -- 172.21.15.1:0/1168019159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22041f3660 con 0x7f2204146af0 2024-03-18T19:03:07.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.972+0000 7f2200ff9700 1 -- 172.21.15.1:0/1168019159 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4731 (secure 0 0 0) 0x7f22041f3660 con 0x7f2204146af0 2024-03-18T19:03:07.968 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:03:07.968 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5420,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3327835778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":3327835778},{"type":"v1","addr":"172.21.15.165:6835","nonce":3327835778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6016,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/450295183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":450295183},{"type":"v1","addr":"172.21.15.1:6837","nonce":450295183}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6022,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1053303526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1053303526},{"type":"v1","addr":"172.21.15.165:6834","nonce":1053303526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:03:02.534936+0000","modified":"2024-03-18T19:03:05.546799+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6270":{"gid":6270,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6836/1826374081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1826374081},{"type":"v1","addr":"172.21.15.1:6836","nonce":1826374081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-18T19:03:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.974+0000 7f220a77a700 1 -- 172.21.15.1:0/1168019159 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f21f006adb0 msgr2=0x7f21f006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.974+0000 7f220a77a700 1 --2- 172.21.15.1:0/1168019159 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f21f006adb0 0x7f21f006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f21f4006fd0 tx=0x7f21f4008040 comp rx=0 tx=0).stop 2024-03-18T19:03:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.974+0000 7f220a77a700 1 -- 172.21.15.1:0/1168019159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2204146af0 msgr2=0x7f22041402c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:07.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.974+0000 7f220a77a700 1 --2- 172.21.15.1:0/1168019159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2204146af0 0x7f22041402c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f2204067530 tx=0x7f21f8004750 comp rx=0 tx=0).stop 2024-03-18T19:03:07.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.974+0000 7f220a77a700 1 -- 172.21.15.1:0/1168019159 shutdown_connections 2024-03-18T19:03:07.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.974+0000 7f220a77a700 1 --2- 172.21.15.1:0/1168019159 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f21f006adb0 0x7f21f006d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.974+0000 7f220a77a700 1 --2- 172.21.15.1:0/1168019159 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f22041465a0 0x7f22041f2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.974+0000 7f220a77a700 1 --2- 172.21.15.1:0/1168019159 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2204140800 0x7f22041f3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.974+0000 7f220a77a700 1 --2- 172.21.15.1:0/1168019159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2204146af0 0x7f22041402c0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:07.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.974+0000 7f220a77a700 1 -- 172.21.15.1:0/1168019159 >> 172.21.15.1:0/1168019159 conn(0x7f22040b1d20 msgr2=0x7f22040b3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:07.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.974+0000 7f220a77a700 1 -- 172.21.15.1:0/1168019159 shutdown_connections 2024-03-18T19:03:07.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:07.974+0000 7f220a77a700 1 -- 172.21.15.1:0/1168019159 wait complete. 2024-03-18T19:03:07.972 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-03-18T19:03:08.385 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T19:03:02.534936+0000', 'modified': '2024-03-18T19:03:05.546799+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6270': {'gid': 6270, 'name': 'c', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6836/1826374081', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6834', 'nonce': 1826374081}, {'type': 'v1', 'addr': '172.21.15.1:6836', 'nonce': 1826374081}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T19:03:08.386 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T19:03:08.386 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T19:03:08.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.522+0000 7fd870293700 1 Processor -- start 2024-03-18T19:03:08.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.522+0000 7fd870293700 1 -- start start 2024-03-18T19:03:08.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.523+0000 7fd870293700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd86806ca80 0x7fd86815d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.523+0000 7fd870293700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd86809a150 0x7fd8680cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.523+0000 7fd870293700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8680b3b00 con 0x7fd86815d5b0 2024-03-18T19:03:08.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.523+0000 7fd870293700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8680cbbb0 con 0x7fd86806ca80 2024-03-18T19:03:08.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.523+0000 7fd870293700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8680cf7f0 con 0x7fd86809a150 2024-03-18T19:03:08.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.523+0000 7fd86e830700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd86809a150 0x7fd8680cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.523+0000 7fd86e830700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd86809a150 0x7fd8680cbfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:49086/0 (socket says 172.21.15.1:49086) 2024-03-18T19:03:08.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.523+0000 7fd86d82e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd86806ca80 0x7fd86815d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.523+0000 7fd86e02f700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fd86815d5b0 0x7fd868099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:59476/0 (socket says 172.21.15.1:59476) 2024-03-18T19:03:08.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.523+0000 7fd86e830700 1 -- 172.21.15.1:0/2817093734 learned_addr learned my addr 172.21.15.1:0/2817093734 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.524+0000 7fd86d82e700 1 -- 172.21.15.1:0/2817093734 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd86809a150 msgr2=0x7fd8680cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.524+0000 7fd86d82e700 1 --2- 172.21.15.1:0/2817093734 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd86809a150 0x7fd8680cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.524+0000 7fd86d82e700 1 -- 172.21.15.1:0/2817093734 >> v1:172.21.15.165:6789/0 conn(0x7fd86815d5b0 legacy=0x7fd868099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.524+0000 7fd86d82e700 1 -- 172.21.15.1:0/2817093734 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8680d04d0 con 0x7fd86806ca80 2024-03-18T19:03:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.524+0000 7fd86d02d700 1 -- 172.21.15.1:0/2817093734 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751819084 0 0) 0x7fd8680b3b00 con 0x7fd86815d5b0 2024-03-18T19:03:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.524+0000 7fd86e830700 1 --2- 172.21.15.1:0/2817093734 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd86809a150 0x7fd8680cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:03:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.524+0000 7fd86d82e700 1 --2- 172.21.15.1:0/2817093734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd86806ca80 0x7fd86815d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd858009840 tx=0x7fd858009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=360347194a555493 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.524+0000 7fd86d02d700 1 -- 172.21.15.1:0/2817093734 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd858011830 con 0x7fd86806ca80 2024-03-18T19:03:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.524+0000 7fd86d02d700 1 -- 172.21.15.1:0/2817093734 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd858011990 con 0x7fd86806ca80 2024-03-18T19:03:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.524+0000 7fd86d02d700 1 -- 172.21.15.1:0/2817093734 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd858011d20 con 0x7fd86806ca80 2024-03-18T19:03:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.524+0000 7fd870293700 1 -- 172.21.15.1:0/2817093734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd86806ca80 msgr2=0x7fd86815d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.524+0000 7fd870293700 1 --2- 172.21.15.1:0/2817093734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd86806ca80 0x7fd86815d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd858009840 tx=0x7fd858009b50 comp rx=0 tx=0).stop 2024-03-18T19:03:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.525+0000 7fd870293700 1 -- 172.21.15.1:0/2817093734 shutdown_connections 2024-03-18T19:03:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.525+0000 7fd870293700 1 --2- 172.21.15.1:0/2817093734 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd86809a150 0x7fd8680cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.525+0000 7fd870293700 1 --2- 172.21.15.1:0/2817093734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd86806ca80 0x7fd86815d990 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.525+0000 7fd870293700 1 -- 172.21.15.1:0/2817093734 >> 172.21.15.1:0/2817093734 conn(0x7fd8680b2bd0 msgr2=0x7fd8680b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.525+0000 7fd870293700 1 -- 172.21.15.1:0/2817093734 shutdown_connections 2024-03-18T19:03:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.525+0000 7fd870293700 1 -- 172.21.15.1:0/2817093734 wait complete. 2024-03-18T19:03:08.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.526+0000 7fd870293700 1 Processor -- start 2024-03-18T19:03:08.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.526+0000 7fd870293700 1 -- start start 2024-03-18T19:03:08.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd870293700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd86806ca80 0x7fd8680b0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd870293700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd86809a150 0x7fd8680adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd870293700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd86815d5b0 0x7fd86814b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd870293700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8680ae3f0 con 0x7fd86809a150 2024-03-18T19:03:08.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd870293700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8680b0f50 con 0x7fd86806ca80 2024-03-18T19:03:08.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd870293700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8680b1230 con 0x7fd86815d5b0 2024-03-18T19:03:08.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd86d82e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd86809a150 0x7fd8680adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd86d82e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd86809a150 0x7fd8680adeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53942/0 (socket says 172.21.15.1:53942) 2024-03-18T19:03:08.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd86d82e700 1 -- 172.21.15.1:0/426488971 learned_addr learned my addr 172.21.15.1:0/426488971 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:08.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd86e02f700 1 --2- 172.21.15.1:0/426488971 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd86806ca80 0x7fd8680b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd86e830700 1 --2- 172.21.15.1:0/426488971 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd86815d5b0 0x7fd86814b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd86d82e700 1 -- 172.21.15.1:0/426488971 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd86815d5b0 msgr2=0x7fd86814b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:08.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd86d82e700 1 --2- 172.21.15.1:0/426488971 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd86815d5b0 0x7fd86814b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd86d82e700 1 -- 172.21.15.1:0/426488971 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd86806ca80 msgr2=0x7fd8680b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:08.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd86d82e700 1 --2- 172.21.15.1:0/426488971 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd86806ca80 0x7fd8680b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd86d82e700 1 -- 172.21.15.1:0/426488971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd860009b30 con 0x7fd86809a150 2024-03-18T19:03:08.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd86d82e700 1 --2- 172.21.15.1:0/426488971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd86809a150 0x7fd8680adeb0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd858009fd0 tx=0x7fd858004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:08.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd85effd700 1 -- 172.21.15.1:0/426488971 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8580036a0 con 0x7fd86809a150 2024-03-18T19:03:08.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd85effd700 1 -- 172.21.15.1:0/426488971 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85801e070 con 0x7fd86809a150 2024-03-18T19:03:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd85effd700 1 -- 172.21.15.1:0/426488971 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd858003a60 con 0x7fd86809a150 2024-03-18T19:03:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd870293700 1 -- 172.21.15.1:0/426488971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8580094a0 con 0x7fd86809a150 2024-03-18T19:03:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.527+0000 7fd870293700 1 -- 172.21.15.1:0/426488971 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8680b17f0 con 0x7fd86809a150 2024-03-18T19:03:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.529+0000 7fd85effd700 1 -- 172.21.15.1:0/426488971 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd858003bc0 con 0x7fd86809a150 2024-03-18T19:03:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.529+0000 7fd85effd700 1 --2- 172.21.15.1:0/426488971 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd84c06adb0 0x7fd84c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.529+0000 7fd85effd700 1 -- 172.21.15.1:0/426488971 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd85808a4c0 con 0x7fd86809a150 2024-03-18T19:03:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.529+0000 7fd86e02f700 1 --2- 172.21.15.1:0/426488971 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd84c06adb0 0x7fd84c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.530+0000 7fd86e02f700 1 --2- 172.21.15.1:0/426488971 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd84c06adb0 0x7fd84c06d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd864007fd0 tx=0x7fd864007de0 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:08.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.531+0000 7fd87c43cb80 1 Processor -- start 2024-03-18T19:03:08.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.531+0000 7fd87c43cb80 1 -- start start 2024-03-18T19:03:08.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.532+0000 7fd87c43cb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db6cc10 0x55d32db6cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.532+0000 7fd87c43cb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db3e0a0 0x55d32db89580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.532+0000 7fd87c43cb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 0x55d32db41160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.532+0000 7fd87c43cb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d32dab7f20 con 0x55d32db7e630 2024-03-18T19:03:08.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.532+0000 7fd87c43cb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d32da99c30 con 0x55d32db3e0a0 2024-03-18T19:03:08.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.532+0000 7fd87c43cb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d32da97430 con 0x55d32db6cc10 2024-03-18T19:03:08.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.532+0000 7fd86e830700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 0x55d32db41160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.532+0000 7fd86e830700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 0x55d32db41160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53944/0 (socket says 172.21.15.1:53944) 2024-03-18T19:03:08.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.532+0000 7fd86e830700 1 -- 172.21.15.1:0/1760355858 learned_addr learned my addr 172.21.15.1:0/1760355858 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:08.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.532+0000 7fd86e02f700 1 --2- 172.21.15.1:0/1760355858 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db6cc10 0x55d32db6cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.532+0000 7fd86d82e700 1 --2- 172.21.15.1:0/1760355858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db3e0a0 0x55d32db89580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.532+0000 7fd86e830700 1 -- 172.21.15.1:0/1760355858 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db6cc10 msgr2=0x55d32db6cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:08.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.532+0000 7fd86e830700 1 --2- 172.21.15.1:0/1760355858 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db6cc10 0x55d32db6cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.532+0000 7fd86e830700 1 -- 172.21.15.1:0/1760355858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db3e0a0 msgr2=0x55d32db89580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:08.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.532+0000 7fd86e830700 1 --2- 172.21.15.1:0/1760355858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db3e0a0 0x55d32db89580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.532+0000 7fd86e830700 1 -- 172.21.15.1:0/1760355858 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d32d886790 con 0x55d32db7e630 2024-03-18T19:03:08.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.533+0000 7fd86e830700 1 --2- 172.21.15.1:0/1760355858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 0x55d32db41160 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fd860011550 tx=0x7fd860009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3598d980357c5aec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:08.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.533+0000 7fd84bfff700 1 -- 172.21.15.1:0/1760355858 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86000e040 con 0x55d32db7e630 2024-03-18T19:03:08.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.533+0000 7fd84bfff700 1 -- 172.21.15.1:0/1760355858 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86000f040 con 0x55d32db7e630 2024-03-18T19:03:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.533+0000 7fd84bfff700 1 -- 172.21.15.1:0/1760355858 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8600092d0 con 0x55d32db7e630 2024-03-18T19:03:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.533+0000 7fd87c43cb80 1 -- 172.21.15.1:0/1760355858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 msgr2=0x55d32db41160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.533+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/1760355858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 0x55d32db41160 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fd860011550 tx=0x7fd860009810 comp rx=0 tx=0).stop 2024-03-18T19:03:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.533+0000 7fd87c43cb80 1 -- 172.21.15.1:0/1760355858 shutdown_connections 2024-03-18T19:03:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.533+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/1760355858 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db6cc10 0x55d32db6cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.533+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/1760355858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 0x55d32db41160 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.533+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/1760355858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db3e0a0 0x55d32db89580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.533+0000 7fd87c43cb80 1 -- 172.21.15.1:0/1760355858 >> 172.21.15.1:0/1760355858 conn(0x55d32daa0cf0 msgr2=0x55d32db40a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.534+0000 7fd87c43cb80 1 -- 172.21.15.1:0/1760355858 shutdown_connections 2024-03-18T19:03:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.534+0000 7fd87c43cb80 1 -- 172.21.15.1:0/1760355858 wait complete. 2024-03-18T19:03:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.534+0000 7fd87c43cb80 1 -- start start 2024-03-18T19:03:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.534+0000 7fd87c43cb80 1 Processor -- start 2024-03-18T19:03:08.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.535+0000 7fd87c43cb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db3e0a0 0x55d32db630b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.535+0000 7fd87c43cb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db6cc10 0x55d32db815f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.535+0000 7fd87c43cb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 0x55d32db702e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.535+0000 7fd87c43cb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d32da92d20 con 0x55d32db7e630 2024-03-18T19:03:08.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.535+0000 7fd87c43cb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d32da2e770 con 0x55d32db6cc10 2024-03-18T19:03:08.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.535+0000 7fd87c43cb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d32da516f0 con 0x55d32db3e0a0 2024-03-18T19:03:08.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.535+0000 7fd87c43cb80 10 client.0 _ll_register_callbacks cb 0x55d32d89a5a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T19:03:08.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.535+0000 7fd86e830700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 0x55d32db702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.535+0000 7fd86e02f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db3e0a0 0x55d32db630b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.535+0000 7fd86e02f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db3e0a0 0x55d32db630b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:49112/0 (socket says 172.21.15.1:49112) 2024-03-18T19:03:08.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.535+0000 7fd86e02f700 1 -- 172.21.15.1:0/1569138682 learned_addr learned my addr 172.21.15.1:0/1569138682 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:08.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.535+0000 7fd86e830700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 0x55d32db702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53960/0 (socket says 172.21.15.1:53960) 2024-03-18T19:03:08.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.535+0000 7fd86d82e700 1 --2- 172.21.15.1:0/1569138682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db6cc10 0x55d32db815f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd86e830700 1 -- 172.21.15.1:0/1569138682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db3e0a0 msgr2=0x55d32db630b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:08.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd86e830700 1 --2- 172.21.15.1:0/1569138682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db3e0a0 0x55d32db630b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd86e830700 1 -- 172.21.15.1:0/1569138682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db6cc10 msgr2=0x55d32db815f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:08.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd86e830700 1 --2- 172.21.15.1:0/1569138682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db6cc10 0x55d32db815f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd86e830700 1 -- 172.21.15.1:0/1569138682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd860011230 con 0x55d32db7e630 2024-03-18T19:03:08.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd86e02f700 1 --2- 172.21.15.1:0/1569138682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db3e0a0 0x55d32db630b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:03:08.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd86e830700 1 --2- 172.21.15.1:0/1569138682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 0x55d32db702e0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fd860009fd0 tx=0x7fd860004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97678281bbb6a66d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:08.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd8497fa700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T19:03:08.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd8497fa700 1 -- 172.21.15.1:0/1569138682 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860022070 con 0x55d32db7e630 2024-03-18T19:03:08.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd87c43cb80 1 -- 172.21.15.1:0/1569138682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d32db34e90 con 0x55d32db7e630 2024-03-18T19:03:08.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd8497fa700 1 -- 172.21.15.1:0/1569138682 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd860015970 con 0x55d32db7e630 2024-03-18T19:03:08.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd8497fa700 1 -- 172.21.15.1:0/1569138682 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd86000e040 con 0x55d32db7e630 2024-03-18T19:03:08.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd87c43cb80 10 client.6414 fetch_fsmap learned FSMap version 25 2024-03-18T19:03:08.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd87c43cb80 1 -- 172.21.15.1:0/1569138682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x55d32d9730a0 con 0x55d32db7e630 2024-03-18T19:03:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd8497fa700 1 -- 172.21.15.1:0/1569138682 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d32db34e90 con 0x55d32db7e630 2024-03-18T19:03:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd8497fa700 1 -- 172.21.15.1:0/1569138682 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd860015b30 con 0x55d32db7e630 2024-03-18T19:03:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd87c43cb80 10 client.6414 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T19:03:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd87c43cb80 10 client.6414 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T19:03:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd87c43cb80 20 client.6414 populate_metadata read hostname 'smithi001' 2024-03-18T19:03:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/1569138682 >> [v2:172.21.15.1:6834/1826374081,v1:172.21.15.1:6836/1826374081] conn(0x55d32db6e2a0 0x55d32db6e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd87c43cb80 4 client.6414 mds_command: new command op to 6270 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T19:03:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd87c43cb80 1 -- 172.21.15.1:0/1569138682 --> [v2:172.21.15.1:6834/1826374081,v1:172.21.15.1:6836/1826374081] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d32db12010 con 0x55d32db6e2a0 2024-03-18T19:03:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.536+0000 7fd86e02f700 1 --2- 172.21.15.1:0/1569138682 >> [v2:172.21.15.1:6834/1826374081,v1:172.21.15.1:6836/1826374081] conn(0x55d32db6e2a0 0x55d32db6e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.537+0000 7fd8497fa700 10 client.6414 ms_handle_connect on v2:172.21.15.1:6834/1826374081 2024-03-18T19:03:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.537+0000 7fd86e02f700 1 --2- 172.21.15.1:0/1569138682 >> [v2:172.21.15.1:6834/1826374081,v1:172.21.15.1:6836/1826374081] conn(0x55d32db6e2a0 0x55d32db6e680 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a570a5d7f7702f54 server_cookie=9bc408755700fdc9 in_seq=0 out_seq=0 2024-03-18T19:03:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.537+0000 7fd8497fa700 1 -- 172.21.15.1:0/1569138682 <== mds.0 v2:172.21.15.1:6834/1826374081 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d32db12010 con 0x55d32db6e2a0 2024-03-18T19:03:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.537+0000 7fd8497fa700 10 client.6414 handle_command_reply: tid=0 2024-03-18T19:03:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.537+0000 7fd87c43cb80 1 client.6414 shutdown 2024-03-18T19:03:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.537+0000 7fd87c43cb80 1 -- 172.21.15.1:0/1569138682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 msgr2=0x55d32db702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.537+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/1569138682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 0x55d32db702e0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fd860009fd0 tx=0x7fd860004b40 comp rx=0 tx=0).stop 2024-03-18T19:03:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.538+0000 7fd87c43cb80 1 -- 172.21.15.1:0/1569138682 shutdown_connections 2024-03-18T19:03:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.538+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/1569138682 >> [v2:172.21.15.1:6834/1826374081,v1:172.21.15.1:6836/1826374081] conn(0x55d32db6e2a0 0x55d32db6e680 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.538+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/1569138682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db3e0a0 0x55d32db630b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.538+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/1569138682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 0x55d32db702e0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.538+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/1569138682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db6cc10 0x55d32db815f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.538+0000 7fd8497fa700 0 client.6414 ms_handle_reset on v2:172.21.15.1:6834/1826374081 2024-03-18T19:03:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.538+0000 7fd87c43cb80 1 -- 172.21.15.1:0/1569138682 >> 172.21.15.1:0/1569138682 conn(0x55d32daa0cf0 msgr2=0x55d32db3c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.538+0000 7fd87c43cb80 1 -- 172.21.15.1:0/1569138682 shutdown_connections 2024-03-18T19:03:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.538+0000 7fd87c43cb80 1 -- 172.21.15.1:0/1569138682 wait complete. 2024-03-18T19:03:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.538+0000 7fd87c43cb80 20 client.6414 trim_cache size 0 max 16384 2024-03-18T19:03:08.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.555+0000 7fd87c43cb80 1 Processor -- start 2024-03-18T19:03:08.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.555+0000 7fd87c43cb80 1 -- start start 2024-03-18T19:03:08.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.556+0000 7fd87c43cb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db3e0a0 0x55d32db3c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.556+0000 7fd87c43cb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db6cc10 0x55d32db6eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.556+0000 7fd87c43cb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 0x55d32db6d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.556+0000 7fd87c43cb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d32d887390 con 0x55d32db7e630 2024-03-18T19:03:08.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.556+0000 7fd87c43cb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d32db34b50 con 0x55d32db3e0a0 2024-03-18T19:03:08.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.556+0000 7fd87c43cb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d32db7fbf0 con 0x55d32db6cc10 2024-03-18T19:03:08.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.556+0000 7fd86e830700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 0x55d32db6d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.556+0000 7fd86e830700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 0x55d32db6d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53962/0 (socket says 172.21.15.1:53962) 2024-03-18T19:03:08.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.556+0000 7fd86e830700 1 -- 172.21.15.1:0/2345186430 learned_addr learned my addr 172.21.15.1:0/2345186430 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:08.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.556+0000 7fd86d82e700 1 --2- 172.21.15.1:0/2345186430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db6cc10 0x55d32db6eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.556+0000 7fd86e02f700 1 --2- 172.21.15.1:0/2345186430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db3e0a0 0x55d32db3c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.556+0000 7fd86e830700 1 -- 172.21.15.1:0/2345186430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db6cc10 msgr2=0x55d32db6eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:08.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.556+0000 7fd86e830700 1 --2- 172.21.15.1:0/2345186430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db6cc10 0x55d32db6eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.556+0000 7fd86e830700 1 -- 172.21.15.1:0/2345186430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db3e0a0 msgr2=0x55d32db3c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:08.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.556+0000 7fd86e830700 1 --2- 172.21.15.1:0/2345186430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db3e0a0 0x55d32db3c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.557+0000 7fd86e830700 1 -- 172.21.15.1:0/2345186430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d32da59010 con 0x55d32db7e630 2024-03-18T19:03:08.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.557+0000 7fd86d82e700 1 --2- 172.21.15.1:0/2345186430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db6cc10 0x55d32db6eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:03:08.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.557+0000 7fd86e02f700 1 --2- 172.21.15.1:0/2345186430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db3e0a0 0x55d32db3c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:03:08.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.557+0000 7fd86e830700 1 --2- 172.21.15.1:0/2345186430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 0x55d32db6d090 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd86000b010 tx=0x7fd860022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b84b6776dbc097a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:08.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.557+0000 7fd85cda8700 1 -- 172.21.15.1:0/2345186430 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86000e040 con 0x55d32db7e630 2024-03-18T19:03:08.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.557+0000 7fd85cda8700 1 -- 172.21.15.1:0/2345186430 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8600099c0 con 0x55d32db7e630 2024-03-18T19:03:08.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.557+0000 7fd85cda8700 1 -- 172.21.15.1:0/2345186430 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860015bb0 con 0x55d32db7e630 2024-03-18T19:03:08.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.557+0000 7fd87c43cb80 1 -- 172.21.15.1:0/2345186430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 msgr2=0x55d32db6d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:08.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.557+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/2345186430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 0x55d32db6d090 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd86000b010 tx=0x7fd860022000 comp rx=0 tx=0).stop 2024-03-18T19:03:08.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.557+0000 7fd87c43cb80 1 -- 172.21.15.1:0/2345186430 shutdown_connections 2024-03-18T19:03:08.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.557+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/2345186430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db7e630 0x55d32db6d090 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.557+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/2345186430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db3e0a0 0x55d32db3c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.557+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/2345186430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db6cc10 0x55d32db6eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.557+0000 7fd87c43cb80 1 -- 172.21.15.1:0/2345186430 >> 172.21.15.1:0/2345186430 conn(0x55d32daa0cf0 msgr2=0x55d32db80d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:08.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.558+0000 7fd87c43cb80 1 -- 172.21.15.1:0/2345186430 shutdown_connections 2024-03-18T19:03:08.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.558+0000 7fd87c43cb80 1 -- 172.21.15.1:0/2345186430 wait complete. 2024-03-18T19:03:08.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.558+0000 7fd87c43cb80 1 -- start start 2024-03-18T19:03:08.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.558+0000 7fd87c43cb80 1 Processor -- start 2024-03-18T19:03:08.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.559+0000 7fd87c43cb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db3e0a0 0x55d32db87b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.559+0000 7fd87c43cb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db6cc10 0x55d32db880d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.559+0000 7fd87c43cb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db7e630 0x55d32db63650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.559+0000 7fd87c43cb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d32db6c800 con 0x55d32db3e0a0 2024-03-18T19:03:08.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.559+0000 7fd87c43cb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d32dbf1b80 con 0x55d32db7e630 2024-03-18T19:03:08.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.559+0000 7fd87c43cb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d32db80990 con 0x55d32db6cc10 2024-03-18T19:03:08.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.559+0000 7fd87c43cb80 10 client.0 _ll_register_callbacks cb 0x7fd858006cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T19:03:08.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.559+0000 7fd86e02f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db3e0a0 0x55d32db87b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.559+0000 7fd86e02f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db3e0a0 0x55d32db87b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53976/0 (socket says 172.21.15.1:53976) 2024-03-18T19:03:08.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.559+0000 7fd86e02f700 1 -- 172.21.15.1:0/710826001 learned_addr learned my addr 172.21.15.1:0/710826001 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:08.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.559+0000 7fd86d82e700 1 --2- 172.21.15.1:0/710826001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db6cc10 0x55d32db880d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.559+0000 7fd86e830700 1 --2- 172.21.15.1:0/710826001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db7e630 0x55d32db63650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.559+0000 7fd86e02f700 1 -- 172.21.15.1:0/710826001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db6cc10 msgr2=0x55d32db880d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:08.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.559+0000 7fd86e02f700 1 --2- 172.21.15.1:0/710826001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db6cc10 0x55d32db880d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.559+0000 7fd86e02f700 1 -- 172.21.15.1:0/710826001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db7e630 msgr2=0x55d32db63650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:08.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.559+0000 7fd86e02f700 1 --2- 172.21.15.1:0/710826001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db7e630 0x55d32db63650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.559+0000 7fd86e02f700 1 -- 172.21.15.1:0/710826001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d32d9730a0 con 0x55d32db3e0a0 2024-03-18T19:03:08.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.560+0000 7fd86e02f700 1 --2- 172.21.15.1:0/710826001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db3e0a0 0x55d32db87b90 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fd864004fd0 tx=0x7fd864000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a06c33edbf3006 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:08.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.560+0000 7fd8497fa700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T19:03:08.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.560+0000 7fd8497fa700 1 -- 172.21.15.1:0/710826001 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd864028530 con 0x55d32db3e0a0 2024-03-18T19:03:08.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.560+0000 7fd8497fa700 1 -- 172.21.15.1:0/710826001 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd86401b040 con 0x55d32db3e0a0 2024-03-18T19:03:08.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.560+0000 7fd87c43cb80 1 -- 172.21.15.1:0/710826001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd86000f040 con 0x55d32db3e0a0 2024-03-18T19:03:08.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.560+0000 7fd8497fa700 1 -- 172.21.15.1:0/710826001 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8640289b0 con 0x55d32db3e0a0 2024-03-18T19:03:08.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.560+0000 7fd87c43cb80 10 client.6420 fetch_fsmap learned FSMap version 25 2024-03-18T19:03:08.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.560+0000 7fd87c43cb80 1 -- 172.21.15.1:0/710826001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x55d32d8e71d0 con 0x55d32db3e0a0 2024-03-18T19:03:08.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.560+0000 7fd8497fa700 1 -- 172.21.15.1:0/710826001 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86000f040 con 0x55d32db3e0a0 2024-03-18T19:03:08.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.560+0000 7fd8497fa700 1 -- 172.21.15.1:0/710826001 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 25) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd864028b10 con 0x55d32db3e0a0 2024-03-18T19:03:08.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.560+0000 7fd87c43cb80 10 client.6420 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T19:03:08.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.560+0000 7fd87c43cb80 10 client.6420 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T19:03:08.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.560+0000 7fd87c43cb80 20 client.6420 populate_metadata read hostname 'smithi001' 2024-03-18T19:03:08.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.560+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/710826001 >> [v2:172.21.15.1:6834/1826374081,v1:172.21.15.1:6836/1826374081] conn(0x55d32db63b90 0x55d32db409a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.560+0000 7fd87c43cb80 4 client.6420 mds_command: new command op to 6270 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T19:03:08.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.560+0000 7fd87c43cb80 1 -- 172.21.15.1:0/710826001 --> [v2:172.21.15.1:6834/1826374081,v1:172.21.15.1:6836/1826374081] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d32d8cc040 con 0x55d32db63b90 2024-03-18T19:03:08.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.560+0000 7fd86d82e700 1 --2- 172.21.15.1:0/710826001 >> [v2:172.21.15.1:6834/1826374081,v1:172.21.15.1:6836/1826374081] conn(0x55d32db63b90 0x55d32db409a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.561+0000 7fd8497fa700 10 client.6420 ms_handle_connect on v2:172.21.15.1:6834/1826374081 2024-03-18T19:03:08.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.561+0000 7fd86d82e700 1 --2- 172.21.15.1:0/710826001 >> [v2:172.21.15.1:6834/1826374081,v1:172.21.15.1:6836/1826374081] conn(0x55d32db63b90 0x55d32db409a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff487f7264304697 server_cookie=230e05874ee25547 in_seq=0 out_seq=0 2024-03-18T19:03:08.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.561+0000 7fd8497fa700 1 -- 172.21.15.1:0/710826001 <== mds.0 v2:172.21.15.1:6834/1826374081 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55d32d8cc040 con 0x55d32db63b90 2024-03-18T19:03:08.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.561+0000 7fd8497fa700 10 client.6420 handle_command_reply: tid=0 2024-03-18T19:03:08.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.561+0000 7fd87c43cb80 1 client.6420 shutdown 2024-03-18T19:03:08.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.561+0000 7fd87c43cb80 1 -- 172.21.15.1:0/710826001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db3e0a0 msgr2=0x55d32db87b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:08.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.561+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/710826001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db3e0a0 0x55d32db87b90 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fd864004fd0 tx=0x7fd864000ed0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.561+0000 7fd87c43cb80 1 -- 172.21.15.1:0/710826001 shutdown_connections 2024-03-18T19:03:08.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.561+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/710826001 >> [v2:172.21.15.1:6834/1826374081,v1:172.21.15.1:6836/1826374081] conn(0x55d32db63b90 0x55d32db409a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.561+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/710826001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d32db7e630 0x55d32db63650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.561+0000 7fd8497fa700 0 client.6420 ms_handle_reset on v2:172.21.15.1:6834/1826374081 2024-03-18T19:03:08.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.561+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/710826001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d32db3e0a0 0x55d32db87b90 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.561+0000 7fd87c43cb80 1 --2- 172.21.15.1:0/710826001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d32db6cc10 0x55d32db880d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.561+0000 7fd87c43cb80 1 -- 172.21.15.1:0/710826001 >> 172.21.15.1:0/710826001 conn(0x55d32daa0cf0 msgr2=0x55d32db6d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:08.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.561+0000 7fd87c43cb80 1 -- 172.21.15.1:0/710826001 shutdown_connections 2024-03-18T19:03:08.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.561+0000 7fd87c43cb80 1 -- 172.21.15.1:0/710826001 wait complete. 2024-03-18T19:03:08.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.561+0000 7fd87c43cb80 20 client.6420 trim_cache size 0 max 16384 2024-03-18T19:03:08.564 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:03:08.564 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "9c1c697f-8655-4ef3-8a13-da568d2cbe86", 2024-03-18T19:03:08.564 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-18T19:03:08.564 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 6270, 2024-03-18T19:03:08.564 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-18T19:03:08.564 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-18T19:03:08.564 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-18T19:03:08.564 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 6.0254213630000004, 2024-03-18T19:03:08.565 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 25, 2024-03-18T19:03:08.565 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 71, 2024-03-18T19:03:08.565 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 71, 2024-03-18T19:03:08.565 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 11.344791869 2024-03-18T19:03:08.565 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:03:08.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.562+0000 7fd870293700 1 -- 172.21.15.1:0/426488971 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd84c06adb0 msgr2=0x7fd84c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:08.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.562+0000 7fd870293700 1 --2- 172.21.15.1:0/426488971 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd84c06adb0 0x7fd84c06d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd864007fd0 tx=0x7fd864007de0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.562+0000 7fd870293700 1 -- 172.21.15.1:0/426488971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd86809a150 msgr2=0x7fd8680adeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:08.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.562+0000 7fd870293700 1 --2- 172.21.15.1:0/426488971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd86809a150 0x7fd8680adeb0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd858009fd0 tx=0x7fd858004750 comp rx=0 tx=0).stop 2024-03-18T19:03:08.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.562+0000 7fd870293700 1 -- 172.21.15.1:0/426488971 shutdown_connections 2024-03-18T19:03:08.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.562+0000 7fd870293700 1 --2- 172.21.15.1:0/426488971 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd84c06adb0 0x7fd84c06d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.562+0000 7fd870293700 1 --2- 172.21.15.1:0/426488971 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd86815d5b0 0x7fd86814b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.562+0000 7fd870293700 1 --2- 172.21.15.1:0/426488971 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd86806ca80 0x7fd8680b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.562+0000 7fd870293700 1 --2- 172.21.15.1:0/426488971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd86809a150 0x7fd8680adeb0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.562+0000 7fd870293700 1 -- 172.21.15.1:0/426488971 >> 172.21.15.1:0/426488971 conn(0x7fd8680b2bd0 msgr2=0x7fd86806ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:08.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.562+0000 7fd870293700 1 -- 172.21.15.1:0/426488971 shutdown_connections 2024-03-18T19:03:08.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.562+0000 7fd870293700 1 -- 172.21.15.1:0/426488971 wait complete. 2024-03-18T19:03:08.570 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T19:03:08.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.699+0000 7fa2ffb8e700 1 Processor -- start 2024-03-18T19:03:08.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.699+0000 7fa2ffb8e700 1 -- start start 2024-03-18T19:03:08.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.700+0000 7fa2ffb8e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2f815a320 0x7fa2f80ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.700+0000 7fa2ffb8e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2f80cea30 0x7fa2f8071600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.700+0000 7fa2ffb8e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa2f8071b40 0x7fa2f80cad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.700+0000 7fa2ffb8e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f8071ff0 con 0x7fa2f815a320 2024-03-18T19:03:08.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.700+0000 7fa2ffb8e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f80acd80 con 0x7fa2f8071b40 2024-03-18T19:03:08.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.700+0000 7fa2ffb8e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f80cf220 con 0x7fa2f80cea30 2024-03-18T19:03:08.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.700+0000 7fa2fd92a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2f815a320 0x7fa2f80ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.700+0000 7fa2fd92a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2f815a320 0x7fa2f80ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53992/0 (socket says 172.21.15.1:53992) 2024-03-18T19:03:08.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.700+0000 7fa2fd92a700 1 -- 172.21.15.1:0/2013109761 learned_addr learned my addr 172.21.15.1:0/2013109761 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:08.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.700+0000 7fa2fd129700 1 --2- 172.21.15.1:0/2013109761 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2f80cea30 0x7fa2f8071600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.700+0000 7fa2fe12b700 1 --2- 172.21.15.1:0/2013109761 >> v2:172.21.15.165:3300/0 conn(0x7fa2f8071b40 0x7fa2f80cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.700+0000 7fa2fd92a700 1 -- 172.21.15.1:0/2013109761 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2f80cea30 msgr2=0x7fa2f8071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:08.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.700+0000 7fa2fd92a700 1 --2- 172.21.15.1:0/2013109761 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2f80cea30 0x7fa2f8071600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.700+0000 7fa2fd92a700 1 -- 172.21.15.1:0/2013109761 >> v2:172.21.15.165:3300/0 conn(0x7fa2f8071b40 msgr2=0x7fa2f80cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:08.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.700+0000 7fa2fd92a700 1 --2- 172.21.15.1:0/2013109761 >> v2:172.21.15.165:3300/0 conn(0x7fa2f8071b40 0x7fa2f80cad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.700+0000 7fa2fd92a700 1 -- 172.21.15.1:0/2013109761 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f80cef40 con 0x7fa2f815a320 2024-03-18T19:03:08.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.700+0000 7fa2fd92a700 1 --2- 172.21.15.1:0/2013109761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2f815a320 0x7fa2f80ce4f0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fa2e80098f0 tx=0x7fa2e8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d73daf0cc8909b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:08.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.701+0000 7fa2fc928700 1 -- 172.21.15.1:0/2013109761 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e800b860 con 0x7fa2f815a320 2024-03-18T19:03:08.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.701+0000 7fa2fc928700 1 -- 172.21.15.1:0/2013109761 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e800b9c0 con 0x7fa2f815a320 2024-03-18T19:03:08.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.701+0000 7fa2fc928700 1 -- 172.21.15.1:0/2013109761 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e800bd50 con 0x7fa2f815a320 2024-03-18T19:03:08.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.701+0000 7fa2ffb8e700 1 -- 172.21.15.1:0/2013109761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2f815a320 msgr2=0x7fa2f80ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:08.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.701+0000 7fa2ffb8e700 1 --2- 172.21.15.1:0/2013109761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2f815a320 0x7fa2f80ce4f0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fa2e80098f0 tx=0x7fa2e8009c00 comp rx=0 tx=0).stop 2024-03-18T19:03:08.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.701+0000 7fa2ffb8e700 1 -- 172.21.15.1:0/2013109761 shutdown_connections 2024-03-18T19:03:08.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.701+0000 7fa2ffb8e700 1 --2- 172.21.15.1:0/2013109761 >> v2:172.21.15.165:3300/0 conn(0x7fa2f8071b40 0x7fa2f80cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.701+0000 7fa2ffb8e700 1 --2- 172.21.15.1:0/2013109761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2f815a320 0x7fa2f80ce4f0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.701+0000 7fa2ffb8e700 1 --2- 172.21.15.1:0/2013109761 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2f80cea30 0x7fa2f8071600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.701+0000 7fa2ffb8e700 1 -- 172.21.15.1:0/2013109761 >> 172.21.15.1:0/2013109761 conn(0x7fa2f80cc990 msgr2=0x7fa2f80ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:08.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.701+0000 7fa2ffb8e700 1 -- 172.21.15.1:0/2013109761 shutdown_connections 2024-03-18T19:03:08.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.701+0000 7fa2ffb8e700 1 -- 172.21.15.1:0/2013109761 wait complete. 2024-03-18T19:03:08.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.702+0000 7fa2ffb8e700 1 Processor -- start 2024-03-18T19:03:08.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.702+0000 7fa2ffb8e700 1 -- start start 2024-03-18T19:03:08.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.703+0000 7fa2ffb8e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2f8071b40 0x7fa2f81586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.703+0000 7fa2ffb8e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2f80cea30 0x7fa2f80c7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.703+0000 7fa2ffb8e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2f80b2b70 0x7fa2f80b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.703+0000 7fa2ffb8e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f8158c20 con 0x7fa2f8071b40 2024-03-18T19:03:08.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.703+0000 7fa2ffb8e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f80c3a40 con 0x7fa2f80b2b70 2024-03-18T19:03:08.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.703+0000 7fa2ffb8e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f80c7660 con 0x7fa2f80cea30 2024-03-18T19:03:08.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.703+0000 7fa2fd92a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2f8071b40 0x7fa2f81586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.703+0000 7fa2fd92a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2f8071b40 0x7fa2f81586e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54000/0 (socket says 172.21.15.1:54000) 2024-03-18T19:03:08.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.703+0000 7fa2fd92a700 1 -- 172.21.15.1:0/660470139 learned_addr learned my addr 172.21.15.1:0/660470139 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:08.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.703+0000 7fa2fd129700 1 --2- 172.21.15.1:0/660470139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2f80cea30 0x7fa2f80c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.703+0000 7fa2fe12b700 1 --2- 172.21.15.1:0/660470139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2f80b2b70 0x7fa2f80b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.704+0000 7fa2fd92a700 1 -- 172.21.15.1:0/660470139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2f80cea30 msgr2=0x7fa2f80c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:08.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.704+0000 7fa2fd92a700 1 --2- 172.21.15.1:0/660470139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2f80cea30 0x7fa2f80c7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.704+0000 7fa2fd92a700 1 -- 172.21.15.1:0/660470139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2f80b2b70 msgr2=0x7fa2f80b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:08.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.704+0000 7fa2fd92a700 1 --2- 172.21.15.1:0/660470139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2f80b2b70 0x7fa2f80b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.704+0000 7fa2fd92a700 1 -- 172.21.15.1:0/660470139 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e8009550 con 0x7fa2f8071b40 2024-03-18T19:03:08.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.704+0000 7fa2fd129700 1 --2- 172.21.15.1:0/660470139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2f80cea30 0x7fa2f80c7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:03:08.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.704+0000 7fa2fd92a700 1 --2- 172.21.15.1:0/660470139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2f8071b40 0x7fa2f81586e0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fa2e800b0e0 tx=0x7fa2e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:08.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.704+0000 7fa2ee7fc700 1 -- 172.21.15.1:0/660470139 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8004480 con 0x7fa2f8071b40 2024-03-18T19:03:08.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.704+0000 7fa2ee7fc700 1 -- 172.21.15.1:0/660470139 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e801f070 con 0x7fa2f8071b40 2024-03-18T19:03:08.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.704+0000 7fa2ffb8e700 1 -- 172.21.15.1:0/660470139 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f80b3070 con 0x7fa2f8071b40 2024-03-18T19:03:08.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.704+0000 7fa2ee7fc700 1 -- 172.21.15.1:0/660470139 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e80038e0 con 0x7fa2f8071b40 2024-03-18T19:03:08.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.704+0000 7fa2ffb8e700 1 -- 172.21.15.1:0/660470139 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f80b3630 con 0x7fa2f8071b40 2024-03-18T19:03:08.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.705+0000 7fa2ffb8e700 1 -- 172.21.15.1:0/660470139 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f80c3bb0 con 0x7fa2f8071b40 2024-03-18T19:03:08.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.705+0000 7fa2ee7fc700 1 -- 172.21.15.1:0/660470139 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa2e8021030 con 0x7fa2f8071b40 2024-03-18T19:03:08.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.706+0000 7fa2ee7fc700 1 --2- 172.21.15.1:0/660470139 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa2dc06adb0 0x7fa2dc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:08.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.706+0000 7fa2ee7fc700 1 -- 172.21.15.1:0/660470139 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa2e808aa70 con 0x7fa2f8071b40 2024-03-18T19:03:08.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.706+0000 7fa2fd129700 1 --2- 172.21.15.1:0/660470139 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa2dc06adb0 0x7fa2dc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:08.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.706+0000 7fa2fd129700 1 --2- 172.21.15.1:0/660470139 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa2dc06adb0 0x7fa2dc06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa2f80c6c10 tx=0x7fa2f4008040 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:08.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.710+0000 7fa2ee7fc700 1 -- 172.21.15.1:0/660470139 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa2e8055a90 con 0x7fa2f8071b40 2024-03-18T19:03:08.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.962+0000 7fa2ffb8e700 1 -- 172.21.15.1:0/660470139 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa2f80676a0 con 0x7fa2f8071b40 2024-03-18T19:03:08.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.963+0000 7fa2ee7fc700 1 -- 172.21.15.1:0/660470139 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa2e8015070 con 0x7fa2f8071b40 2024-03-18T19:03:08.959 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T19:03:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.965+0000 7fa2ffb8e700 1 -- 172.21.15.1:0/660470139 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa2dc06adb0 msgr2=0x7fa2dc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.965+0000 7fa2ffb8e700 1 --2- 172.21.15.1:0/660470139 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa2dc06adb0 0x7fa2dc06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa2f80c6c10 tx=0x7fa2f4008040 comp rx=0 tx=0).stop 2024-03-18T19:03:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.965+0000 7fa2ffb8e700 1 -- 172.21.15.1:0/660470139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2f8071b40 msgr2=0x7fa2f81586e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.965+0000 7fa2ffb8e700 1 --2- 172.21.15.1:0/660470139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2f8071b40 0x7fa2f81586e0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fa2e800b0e0 tx=0x7fa2e8004750 comp rx=0 tx=0).stop 2024-03-18T19:03:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.965+0000 7fa2ffb8e700 1 -- 172.21.15.1:0/660470139 shutdown_connections 2024-03-18T19:03:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.965+0000 7fa2ffb8e700 1 --2- 172.21.15.1:0/660470139 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa2dc06adb0 0x7fa2dc06d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.965+0000 7fa2ffb8e700 1 --2- 172.21.15.1:0/660470139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2f80b2b70 0x7fa2f80b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.965+0000 7fa2ffb8e700 1 --2- 172.21.15.1:0/660470139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2f8071b40 0x7fa2f81586e0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.965+0000 7fa2ffb8e700 1 --2- 172.21.15.1:0/660470139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2f80cea30 0x7fa2f80c7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:08.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.965+0000 7fa2ffb8e700 1 -- 172.21.15.1:0/660470139 >> 172.21.15.1:0/660470139 conn(0x7fa2f80cc990 msgr2=0x7fa2f80cbd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:08.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.965+0000 7fa2ffb8e700 1 -- 172.21.15.1:0/660470139 shutdown_connections 2024-03-18T19:03:08.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:08.965+0000 7fa2ffb8e700 1 -- 172.21.15.1:0/660470139 wait complete. 2024-03-18T19:03:08.973 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T19:03:08.973 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T19:03:08.973 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-18T19:03:08.973 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T19:03:08.973 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T19:03:08.973 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T19:03:08.973 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T19:03:08.973 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T19:03:08.973 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:03:08.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:08.974 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-18T19:03:08.992 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T19:03:08.992 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T19:03:08.992 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T19:03:08.993 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:03:08.993 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T19:03:08.993 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:03:08.993 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:03:08.993 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-18T19:03:08.993 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-18T19:03:08.993 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:03:08.993 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-18T19:03:08.993 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-18T19:03:08.993 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T19:03:08.993 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:03:08.993 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-18T19:03:08.993 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T19:03:08.994 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 41780sec preferred_lft 41780sec 2024-03-18T19:03:08.994 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link 2024-03-18T19:03:08.994 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:03:08.994 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:03:08.994 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:03:08.994 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-18T19:03:08.994 INFO:teuthology.orchestra.run.smithi001.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T19:03:08.994 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:03:08.994 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T19:03:08.994 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:03:08.994 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::4ca9:f5ff:feb3:79a3/64 scope link 2024-03-18T19:03:08.994 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:03:08.995 INFO:teuthology.orchestra.run.smithi001.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T19:03:08.995 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T19:03:08.995 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::282f:a4ff:fe1a:9908/64 scope link 2024-03-18T19:03:08.995 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:03:08.995 INFO:teuthology.orchestra.run.smithi001.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T19:03:08.995 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2e:5c:60:78:b2:8c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T19:03:08.995 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::2c5c:60ff:fe78:b28c/64 scope link 2024-03-18T19:03:08.995 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:03:08.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:08.995 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T19:03:09.060 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-18T19:03:09.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:09.061 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-18T19:03:09.122 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-18T19:03:09.122 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T19:03:09.123 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T19:03:09.123 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T19:03:09.123 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T19:03:09.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:09.123 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T19:03:09.123 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T19:03:09.123 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T19:03:09.123 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T19:03:09.123 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T19:03:09.123 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T19:03:09.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:09.229 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T19:03:09.246 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T19:03:09.247 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T19:03:09.247 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:03:09.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:09.247 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-18T19:03:09.314 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T19:03:09.314 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T19:03:09.314 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T19:03:09.371 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:03:09.372 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T19:03:09.372 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T19:03:09.429 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T19:03:09.498 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T19:03:09.498 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T19:03:09.565 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T19:03:09.565 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T19:03:09.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:09.566 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T19:03:09.625 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T19:03:09.625 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T19:03:09.668 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T19:03:09.697 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T19:03:09.697 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T19:03:09.758 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-18T19:03:09.762+0000 7f95f70615c0 -1 init, newargv = 0x55d0a24ddf40 newargc=15 2024-03-18T19:03:09.758 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[106451]: starting ceph client 2024-03-18T19:03:09.763 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[106451]: starting fuse 2024-03-18T19:03:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T19:03:09.767 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T19:03:09.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:09.767 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T19:03:09.825 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T19:03:09.826 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-03-18T19:03:09.826 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T19:03:09.826 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-18T19:03:09.826 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-18T19:03:09.826 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-18T19:03:09.826 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-18T19:03:09.826 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:03:09.826 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-18T19:03:09.826 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T19:03:09.826 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-18T19:03:09.826 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T19:03:09.827 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:03:09.827 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-18T19:03:09.827 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-18T19:03:09.827 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-18T19:03:09.827 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-18T19:03:09.827 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:03:09.827 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-18T19:03:09.827 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T19:03:09.827 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T19:03:09.827 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T19:03:09.827 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-18T19:03:09.827 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-18T19:03:09.827 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-18T19:03:09.827 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T19:03:09.827 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:03:09.827 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-18T19:03:09.827 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-18T19:03:09.950 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.106451.asok 2024-03-18T19:03:10.132 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.106451.asok 2024-03-18T19:03:10.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:10.132 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.106451.asok status 2024-03-18T19:03:10.294 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:03:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-18T19:03:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T19:03:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T19:03:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-18T19:03:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-18T19:03:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T19:03:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "106451", 2024-03-18T19:03:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-18T19:03:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:03:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-18T19:03:10.294 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-18T19:03:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 6432, 2024-03-18T19:03:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-18T19:03:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-18T19:03:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-18T19:03:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6432 2024-03-18T19:03:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:03:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T19:03:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T19:03:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T19:03:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2962214600 2024-03-18T19:03:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:03:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:03:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T19:03:10.295 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T19:03:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T19:03:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2962214600 2024-03-18T19:03:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:03:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.6432 192.168.0.1:0/2962214600", 2024-03-18T19:03:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/2962214600", 2024-03-18T19:03:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-18T19:03:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 25, 2024-03-18T19:03:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 71, 2024-03-18T19:03:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-18T19:03:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-18T19:03:10.296 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-18T19:03:10.296 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:03:10.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:10.304 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-18T19:03:10.304 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T19:03:10.322 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-18T19:03:10.323 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T19:03:10.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:10.323 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T19:03:10.391 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-18T19:03:10.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.562+0000 7fe2c529d700 1 Processor -- start 2024-03-18T19:03:10.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.562+0000 7fe2c529d700 1 -- start start 2024-03-18T19:03:10.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.563+0000 7fe2c529d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe2c015dbe0 0x7fe2c00b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:10.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.563+0000 7fe2c529d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c015aad0 con 0x7fe2c015a5f0 2024-03-18T19:03:10.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.563+0000 7fe2c529d700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c015adb0 con 0x7fe2c00cc6a0 2024-03-18T19:03:10.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.563+0000 7fe2c529d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c013f750 con 0x7fe2c015dbe0 2024-03-18T19:03:10.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.563+0000 7fe2beffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe2c015dbe0 0x7fe2c00b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:10.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.563+0000 7fe2be7fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe2c015a5f0 0x7fe2c00cbf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60198/0 (socket says 172.21.15.1:60198) 2024-03-18T19:03:10.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.563+0000 7fe2be7fc700 1 -- 172.21.15.1:0/2456282543 learned_addr learned my addr 172.21.15.1:0/2456282543 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:03:10.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.564+0000 7fe2beffd700 1 -- 172.21.15.1:0/2456282543 >> v1:172.21.15.165:6789/0 conn(0x7fe2c00cc6a0 legacy=0x7fe2c006d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:10.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.564+0000 7fe2beffd700 1 -- 172.21.15.1:0/2456282543 >> v1:172.21.15.1:6789/0 conn(0x7fe2c015a5f0 legacy=0x7fe2c00cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:10.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.564+0000 7fe2bdffb700 1 -- 172.21.15.1:0/2456282543 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2795090022 0 0) 0x7fe2c015aad0 con 0x7fe2c015a5f0 2024-03-18T19:03:10.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.564+0000 7fe2beffd700 1 -- 172.21.15.1:0/2456282543 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c015b090 con 0x7fe2c015dbe0 2024-03-18T19:03:10.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.564+0000 7fe2beffd700 1 --2- 172.21.15.1:0/2456282543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c015dbe0 0x7fe2c00b2fb0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0009840 tx=0x7fe2b0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7baabc421df048c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:10.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.564+0000 7fe2bdffb700 1 -- 172.21.15.1:0/2456282543 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0011830 con 0x7fe2c015dbe0 2024-03-18T19:03:10.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.564+0000 7fe2bdffb700 1 -- 172.21.15.1:0/2456282543 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b0011990 con 0x7fe2c015dbe0 2024-03-18T19:03:10.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.564+0000 7fe2bdffb700 1 -- 172.21.15.1:0/2456282543 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0011d20 con 0x7fe2c015dbe0 2024-03-18T19:03:10.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.565+0000 7fe2c529d700 1 -- 172.21.15.1:0/2456282543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c015dbe0 msgr2=0x7fe2c00b2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:10.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.565+0000 7fe2c529d700 1 --2- 172.21.15.1:0/2456282543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c015dbe0 0x7fe2c00b2fb0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0009840 tx=0x7fe2b0009b50 comp rx=0 tx=0).stop 2024-03-18T19:03:10.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.565+0000 7fe2c529d700 1 -- 172.21.15.1:0/2456282543 shutdown_connections 2024-03-18T19:03:10.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.565+0000 7fe2c529d700 1 --2- 172.21.15.1:0/2456282543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c015dbe0 0x7fe2c00b2fb0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:10.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.565+0000 7fe2c529d700 1 -- 172.21.15.1:0/2456282543 >> 172.21.15.1:0/2456282543 conn(0x7fe2c00b1cb0 msgr2=0x7fe2c00b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:10.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.565+0000 7fe2c529d700 1 -- 172.21.15.1:0/2456282543 shutdown_connections 2024-03-18T19:03:10.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.565+0000 7fe2c529d700 1 -- 172.21.15.1:0/2456282543 wait complete. 2024-03-18T19:03:10.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.566+0000 7fe2c529d700 1 Processor -- start 2024-03-18T19:03:10.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.566+0000 7fe2c529d700 1 -- start start 2024-03-18T19:03:10.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.567+0000 7fe2c529d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2c00cc6a0 0x7fe2c013a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:10.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.567+0000 7fe2c529d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2c015a5f0 0x7fe2c01e3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:10.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.567+0000 7fe2c529d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c015dbe0 0x7fe2c013b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:10.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.567+0000 7fe2c529d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c0153f20 con 0x7fe2c015dbe0 2024-03-18T19:03:10.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.567+0000 7fe2c529d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c00ce8a0 con 0x7fe2c00cc6a0 2024-03-18T19:03:10.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.567+0000 7fe2c529d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c0154130 con 0x7fe2c015a5f0 2024-03-18T19:03:10.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.567+0000 7fe2bf7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c015dbe0 0x7fe2c013b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:10.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.567+0000 7fe2bf7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c015dbe0 0x7fe2c013b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54030/0 (socket says 172.21.15.1:54030) 2024-03-18T19:03:10.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.567+0000 7fe2bf7fe700 1 -- 172.21.15.1:0/35032559 learned_addr learned my addr 172.21.15.1:0/35032559 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:10.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.567+0000 7fe2beffd700 1 --2- 172.21.15.1:0/35032559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2c00cc6a0 0x7fe2c013a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:10.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.567+0000 7fe2be7fc700 1 --2- 172.21.15.1:0/35032559 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2c015a5f0 0x7fe2c01e3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:10.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.568+0000 7fe2bf7fe700 1 -- 172.21.15.1:0/35032559 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2c015a5f0 msgr2=0x7fe2c01e3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:10.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.568+0000 7fe2bf7fe700 1 --2- 172.21.15.1:0/35032559 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2c015a5f0 0x7fe2c01e3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:10.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.568+0000 7fe2bf7fe700 1 -- 172.21.15.1:0/35032559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2c00cc6a0 msgr2=0x7fe2c013a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:10.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.568+0000 7fe2bf7fe700 1 --2- 172.21.15.1:0/35032559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2c00cc6a0 0x7fe2c013a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:10.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.568+0000 7fe2bf7fe700 1 -- 172.21.15.1:0/35032559 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b00094a0 con 0x7fe2c015dbe0 2024-03-18T19:03:10.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.568+0000 7fe2bf7fe700 1 --2- 172.21.15.1:0/35032559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c015dbe0 0x7fe2c013b510 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe2b800cbc0 tx=0x7fe2b800cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:10.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.568+0000 7fe2a77fe700 1 -- 172.21.15.1:0/35032559 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b8003c00 con 0x7fe2c015dbe0 2024-03-18T19:03:10.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.568+0000 7fe2a77fe700 1 -- 172.21.15.1:0/35032559 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b8003d60 con 0x7fe2c015dbe0 2024-03-18T19:03:10.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.568+0000 7fe2c529d700 1 -- 172.21.15.1:0/35032559 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c0154410 con 0x7fe2c015dbe0 2024-03-18T19:03:10.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.568+0000 7fe2a77fe700 1 -- 172.21.15.1:0/35032559 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b800abf0 con 0x7fe2c015dbe0 2024-03-18T19:03:10.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.568+0000 7fe2c529d700 1 -- 172.21.15.1:0/35032559 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c00ce530 con 0x7fe2c015dbe0 2024-03-18T19:03:10.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.569+0000 7fe2c529d700 1 -- 172.21.15.1:0/35032559 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c00cea10 con 0x7fe2c015dbe0 2024-03-18T19:03:10.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.570+0000 7fe2a77fe700 1 -- 172.21.15.1:0/35032559 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe2b800ad50 con 0x7fe2c015dbe0 2024-03-18T19:03:10.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.570+0000 7fe2a77fe700 1 --2- 172.21.15.1:0/35032559 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe2ac06adb0 0x7fe2ac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:10.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.570+0000 7fe2a77fe700 1 -- 172.21.15.1:0/35032559 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe2b80896d0 con 0x7fe2c015dbe0 2024-03-18T19:03:10.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.570+0000 7fe2beffd700 1 --2- 172.21.15.1:0/35032559 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe2ac06adb0 0x7fe2ac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:10.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.571+0000 7fe2beffd700 1 --2- 172.21.15.1:0/35032559 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe2ac06adb0 0x7fe2ac06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00090b0 tx=0x7fe2b001c040 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:10.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.575+0000 7fe2a77fe700 1 -- 172.21.15.1:0/35032559 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe2b8054660 con 0x7fe2c015dbe0 2024-03-18T19:03:10.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.804+0000 7fe2c529d700 1 -- 172.21.15.1:0/35032559 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2c00b4520 con 0x7fe2ac06adb0 2024-03-18T19:03:10.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.805+0000 7fe2a77fe700 1 -- 172.21.15.1:0/35032559 <== mgr.6045 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe2b0011c80 con 0x7fe2ac06adb0 2024-03-18T19:03:10.801 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T19:03:10.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:03:10.802 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephfs" 2024-03-18T19:03:10.802 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:03:10.802 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T19:03:10.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.807+0000 7fe2c529d700 1 -- 172.21.15.1:0/35032559 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe2ac06adb0 msgr2=0x7fe2ac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:10.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.808+0000 7fe2c529d700 1 --2- 172.21.15.1:0/35032559 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe2ac06adb0 0x7fe2ac06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00090b0 tx=0x7fe2b001c040 comp rx=0 tx=0).stop 2024-03-18T19:03:10.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.808+0000 7fe2c529d700 1 -- 172.21.15.1:0/35032559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c015dbe0 msgr2=0x7fe2c013b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:10.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.808+0000 7fe2c529d700 1 --2- 172.21.15.1:0/35032559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c015dbe0 0x7fe2c013b510 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe2b800cbc0 tx=0x7fe2b800cf80 comp rx=0 tx=0).stop 2024-03-18T19:03:10.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.808+0000 7fe2c529d700 1 -- 172.21.15.1:0/35032559 shutdown_connections 2024-03-18T19:03:10.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.808+0000 7fe2c529d700 1 --2- 172.21.15.1:0/35032559 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe2ac06adb0 0x7fe2ac06d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:10.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.808+0000 7fe2c529d700 1 --2- 172.21.15.1:0/35032559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c015dbe0 0x7fe2c013b510 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:10.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.808+0000 7fe2c529d700 1 --2- 172.21.15.1:0/35032559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2c00cc6a0 0x7fe2c013a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:10.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.808+0000 7fe2c529d700 1 --2- 172.21.15.1:0/35032559 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2c015a5f0 0x7fe2c01e3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:10.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.808+0000 7fe2c529d700 1 -- 172.21.15.1:0/35032559 >> 172.21.15.1:0/35032559 conn(0x7fe2c00b1cb0 msgr2=0x7fe2c015e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:10.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.808+0000 7fe2c529d700 1 -- 172.21.15.1:0/35032559 shutdown_connections 2024-03-18T19:03:10.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.808+0000 7fe2c529d700 1 -- 172.21.15.1:0/35032559 wait complete. 2024-03-18T19:03:10.815 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-18T19:03:10.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.945+0000 7fa140a1f700 1 Processor -- start 2024-03-18T19:03:10.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.945+0000 7fa140a1f700 1 -- start start 2024-03-18T19:03:10.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.946+0000 7fa140a1f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa13c15a240 0x7fa13c0ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:10.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.946+0000 7fa140a1f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa13c06d560 0x7fa13c06fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:10.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.946+0000 7fa140a1f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13c070000 con 0x7fa13c0715b0 2024-03-18T19:03:10.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.946+0000 7fa140a1f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa13c0adff0 con 0x7fa13c06d560 2024-03-18T19:03:10.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.946+0000 7fa140a1f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c0cac80 con 0x7fa13c15a240 2024-03-18T19:03:10.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.946+0000 7fa13ad9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa13c0715b0 0x7fa13c15de10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60212/0 (socket says 172.21.15.1:60212) 2024-03-18T19:03:10.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.947+0000 7fa13ad9d700 1 -- 172.21.15.1:0/2496321266 learned_addr learned my addr 172.21.15.1:0/2496321266 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:03:10.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.947+0000 7fa13a59c700 1 --2- 172.21.15.1:0/2496321266 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa13c15a240 0x7fa13c0ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:10.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.947+0000 7fa139d9b700 1 --2- 172.21.15.1:0/2496321266 >> v2:172.21.15.165:3300/0 conn(0x7fa13c06d560 0x7fa13c06fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:10.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.947+0000 7fa13959a700 1 -- 172.21.15.1:0/2496321266 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 505461648 0 0) 0x7fa13c070000 con 0x7fa13c0715b0 2024-03-18T19:03:10.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.947+0000 7fa13959a700 1 -- 172.21.15.1:0/2496321266 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1280035b0 con 0x7fa13c0715b0 2024-03-18T19:03:10.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.947+0000 7fa13a59c700 1 -- 172.21.15.1:0/2496321266 >> v2:172.21.15.165:3300/0 conn(0x7fa13c06d560 msgr2=0x7fa13c06fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:10.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.947+0000 7fa13a59c700 1 --2- 172.21.15.1:0/2496321266 >> v2:172.21.15.165:3300/0 conn(0x7fa13c06d560 0x7fa13c06fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:10.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.947+0000 7fa13a59c700 1 -- 172.21.15.1:0/2496321266 >> v1:172.21.15.1:6789/0 conn(0x7fa13c0715b0 legacy=0x7fa13c15de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:10.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.947+0000 7fa13959a700 1 -- 172.21.15.1:0/2496321266 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2143335796 0 0) 0x7fa1280035b0 con 0x7fa13c0715b0 2024-03-18T19:03:10.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.947+0000 7fa13a59c700 1 -- 172.21.15.1:0/2496321266 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c0caf60 con 0x7fa13c15a240 2024-03-18T19:03:10.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.947+0000 7fa139d9b700 1 --2- 172.21.15.1:0/2496321266 >> v2:172.21.15.165:3300/0 conn(0x7fa13c06d560 0x7fa13c06fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:03:10.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.947+0000 7fa13a59c700 1 --2- 172.21.15.1:0/2496321266 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa13c15a240 0x7fa13c0ca580 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa124009840 tx=0x7fa124009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf275e05e6811100 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:10.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.947+0000 7fa13959a700 1 -- 172.21.15.1:0/2496321266 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1340019f0 con 0x7fa13c15a240 2024-03-18T19:03:10.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.947+0000 7fa13959a700 1 -- 172.21.15.1:0/2496321266 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa124004030 con 0x7fa13c15a240 2024-03-18T19:03:10.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.948+0000 7fa13959a700 1 -- 172.21.15.1:0/2496321266 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1240043a0 con 0x7fa13c15a240 2024-03-18T19:03:10.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.948+0000 7fa140a1f700 1 -- 172.21.15.1:0/2496321266 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa13c15a240 msgr2=0x7fa13c0ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:10.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.948+0000 7fa140a1f700 1 --2- 172.21.15.1:0/2496321266 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa13c15a240 0x7fa13c0ca580 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa124009840 tx=0x7fa124009b50 comp rx=0 tx=0).stop 2024-03-18T19:03:10.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.948+0000 7fa140a1f700 1 -- 172.21.15.1:0/2496321266 shutdown_connections 2024-03-18T19:03:10.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.948+0000 7fa140a1f700 1 --2- 172.21.15.1:0/2496321266 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa13c15a240 0x7fa13c0ca580 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:10.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.948+0000 7fa140a1f700 1 --2- 172.21.15.1:0/2496321266 >> v2:172.21.15.165:3300/0 conn(0x7fa13c06d560 0x7fa13c06fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:10.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.948+0000 7fa140a1f700 1 -- 172.21.15.1:0/2496321266 >> 172.21.15.1:0/2496321266 conn(0x7fa13c0cff30 msgr2=0x7fa13c0d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:10.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.948+0000 7fa140a1f700 1 -- 172.21.15.1:0/2496321266 shutdown_connections 2024-03-18T19:03:10.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.948+0000 7fa140a1f700 1 -- 172.21.15.1:0/2496321266 wait complete. 2024-03-18T19:03:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.949+0000 7fa140a1f700 1 Processor -- start 2024-03-18T19:03:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.950+0000 7fa140a1f700 1 -- start start 2024-03-18T19:03:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.950+0000 7fa140a1f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa13c06d560 0x7fa13c0c38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.950+0000 7fa140a1f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa13c0715b0 0x7fa13c1531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.950+0000 7fa140a1f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa13c15a240 0x7fa13c0c9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.950+0000 7fa140a1f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c0c3e20 con 0x7fa13c06d560 2024-03-18T19:03:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.950+0000 7fa140a1f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c0c4070 con 0x7fa13c0715b0 2024-03-18T19:03:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.950+0000 7fa140a1f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c0c4350 con 0x7fa13c15a240 2024-03-18T19:03:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.950+0000 7fa13a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa13c06d560 0x7fa13c0c38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.950+0000 7fa13a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa13c06d560 0x7fa13c0c38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54044/0 (socket says 172.21.15.1:54044) 2024-03-18T19:03:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.950+0000 7fa13a59c700 1 -- 172.21.15.1:0/3073004822 learned_addr learned my addr 172.21.15.1:0/3073004822 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.951+0000 7fa13ad9d700 1 --2- 172.21.15.1:0/3073004822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa13c15a240 0x7fa13c0c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.951+0000 7fa139d9b700 1 --2- 172.21.15.1:0/3073004822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa13c0715b0 0x7fa13c1531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.951+0000 7fa13a59c700 1 -- 172.21.15.1:0/3073004822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa13c15a240 msgr2=0x7fa13c0c9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.951+0000 7fa13a59c700 1 --2- 172.21.15.1:0/3073004822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa13c15a240 0x7fa13c0c9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.951+0000 7fa13a59c700 1 -- 172.21.15.1:0/3073004822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa13c0715b0 msgr2=0x7fa13c1531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.951+0000 7fa13a59c700 1 --2- 172.21.15.1:0/3073004822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa13c0715b0 0x7fa13c1531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.951+0000 7fa13a59c700 1 -- 172.21.15.1:0/3073004822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa12c0094a0 con 0x7fa13c06d560 2024-03-18T19:03:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.951+0000 7fa139d9b700 1 --2- 172.21.15.1:0/3073004822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa13c0715b0 0x7fa13c1531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:03:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.951+0000 7fa13a59c700 1 --2- 172.21.15.1:0/3073004822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa13c06d560 0x7fa13c0c38e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa1240090b0 tx=0x7fa124004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.951+0000 7fa132ffd700 1 -- 172.21.15.1:0/3073004822 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa124003bf0 con 0x7fa13c06d560 2024-03-18T19:03:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.951+0000 7fa132ffd700 1 -- 172.21.15.1:0/3073004822 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12401f070 con 0x7fa13c06d560 2024-03-18T19:03:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.951+0000 7fa132ffd700 1 -- 172.21.15.1:0/3073004822 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa124011630 con 0x7fa13c06d560 2024-03-18T19:03:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.951+0000 7fa140a1f700 1 -- 172.21.15.1:0/3073004822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1240094a0 con 0x7fa13c06d560 2024-03-18T19:03:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.951+0000 7fa140a1f700 1 -- 172.21.15.1:0/3073004822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa13c0c4910 con 0x7fa13c06d560 2024-03-18T19:03:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.952+0000 7fa140a1f700 1 -- 172.21.15.1:0/3073004822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa13c0676a0 con 0x7fa13c06d560 2024-03-18T19:03:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.953+0000 7fa132ffd700 1 -- 172.21.15.1:0/3073004822 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa124021030 con 0x7fa13c06d560 2024-03-18T19:03:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.953+0000 7fa132ffd700 1 --2- 172.21.15.1:0/3073004822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa11c06adb0 0x7fa11c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.953+0000 7fa132ffd700 1 -- 172.21.15.1:0/3073004822 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa12408a920 con 0x7fa13c06d560 2024-03-18T19:03:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.953+0000 7fa139d9b700 1 --2- 172.21.15.1:0/3073004822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa11c06adb0 0x7fa11c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.954+0000 7fa139d9b700 1 --2- 172.21.15.1:0/3073004822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa11c06adb0 0x7fa11c06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa13c068000 tx=0x7fa12c008040 comp rx=0 tx=0).ready entity=mgr.6045 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:10.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:10.958+0000 7fa132ffd700 1 -- 172.21.15.1:0/3073004822 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa124055a60 con 0x7fa13c06d560 2024-03-18T19:03:11.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:11.222+0000 7fa140a1f700 1 -- 172.21.15.1:0/3073004822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fa13c072860 con 0x7fa13c06d560 2024-03-18T19:03:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.036+0000 7fa132ffd700 1 -- 172.21.15.1:0/3073004822 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 19) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fa124055e30 con 0x7fa13c06d560 2024-03-18T19:03:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.038+0000 7fa132ffd700 1 -- 172.21.15.1:0/3073004822 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7fa124070670 con 0x7fa13c06d560 2024-03-18T19:03:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.042+0000 7fa140a1f700 1 -- 172.21.15.1:0/3073004822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa11c06adb0 msgr2=0x7fa11c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.042+0000 7fa140a1f700 1 --2- 172.21.15.1:0/3073004822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa11c06adb0 0x7fa11c06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa13c068000 tx=0x7fa12c008040 comp rx=0 tx=0).stop 2024-03-18T19:03:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.042+0000 7fa140a1f700 1 -- 172.21.15.1:0/3073004822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa13c06d560 msgr2=0x7fa13c0c38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.042+0000 7fa140a1f700 1 --2- 172.21.15.1:0/3073004822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa13c06d560 0x7fa13c0c38e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa1240090b0 tx=0x7fa124004750 comp rx=0 tx=0).stop 2024-03-18T19:03:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.042+0000 7fa140a1f700 1 -- 172.21.15.1:0/3073004822 shutdown_connections 2024-03-18T19:03:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.042+0000 7fa140a1f700 1 --2- 172.21.15.1:0/3073004822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa11c06adb0 0x7fa11c06d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.042+0000 7fa140a1f700 1 --2- 172.21.15.1:0/3073004822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa13c15a240 0x7fa13c0c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.042+0000 7fa140a1f700 1 --2- 172.21.15.1:0/3073004822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa13c06d560 0x7fa13c0c38e0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.042+0000 7fa140a1f700 1 --2- 172.21.15.1:0/3073004822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa13c0715b0 0x7fa13c1531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.042+0000 7fa140a1f700 1 -- 172.21.15.1:0/3073004822 >> 172.21.15.1:0/3073004822 conn(0x7fa13c0cff30 msgr2=0x7fa13c06ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.042+0000 7fa140a1f700 1 -- 172.21.15.1:0/3073004822 shutdown_connections 2024-03-18T19:03:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.042+0000 7fa140a1f700 1 -- 172.21.15.1:0/3073004822 wait complete. 2024-03-18T19:03:12.057 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-18T19:03:12.174 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:12.178+0000 7f6b5e0dc240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:03:12.192 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:12.197+0000 7fd33f2a6240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:03:12.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.198+0000 7fd8fa2c5700 1 Processor -- start 2024-03-18T19:03:12.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.198+0000 7fd8fa2c5700 1 -- start start 2024-03-18T19:03:12.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.198+0000 7fd8fa2c5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8f4157ba0 0x7fd8f4070db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:12.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.198+0000 7fd8fa2c5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8f415e5e0 0x7fd8f40712f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:12.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.198+0000 7fd8fa2c5700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f415a590 con 0x7fd8f406e2d0 2024-03-18T19:03:12.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.198+0000 7fd8fa2c5700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f415eba0 con 0x7fd8f4157ba0 2024-03-18T19:03:12.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.198+0000 7fd8fa2c5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f415b590 con 0x7fd8f415e5e0 2024-03-18T19:03:12.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.199+0000 7fd8f3fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8f4157ba0 0x7fd8f4070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:12.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.199+0000 7fd8f3fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8f4157ba0 0x7fd8f4070db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54068/0 (socket says 172.21.15.1:54068) 2024-03-18T19:03:12.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.199+0000 7fd8f3fff700 1 -- 172.21.15.1:0/3132389008 learned_addr learned my addr 172.21.15.1:0/3132389008 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:12.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.199+0000 7fd8f37fe700 1 --1- 172.21.15.1:0/3132389008 >> v1:172.21.15.1:6789/0 conn(0x7fd8f406e2d0 0x7fd8f40b25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60216/0 (socket says 172.21.15.1:60216) 2024-03-18T19:03:12.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.199+0000 7fd8f8862700 1 --2- 172.21.15.1:0/3132389008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8f415e5e0 0x7fd8f40712f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:12.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.199+0000 7fd8f3fff700 1 -- 172.21.15.1:0/3132389008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8f415e5e0 msgr2=0x7fd8f40712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:12.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.199+0000 7fd8f3fff700 1 --2- 172.21.15.1:0/3132389008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8f415e5e0 0x7fd8f40712f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.199+0000 7fd8f3fff700 1 -- 172.21.15.1:0/3132389008 >> v1:172.21.15.1:6789/0 conn(0x7fd8f406e2d0 legacy=0x7fd8f40b25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:12.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.199+0000 7fd8f3fff700 1 -- 172.21.15.1:0/3132389008 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f40b3640 con 0x7fd8f4157ba0 2024-03-18T19:03:12.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.199+0000 7fd8f2ffd700 1 -- 172.21.15.1:0/3132389008 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214212483 0 0) 0x7fd8e8001110 con 0x7fd8f406e2d0 2024-03-18T19:03:12.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.199+0000 7fd8f3fff700 1 --2- 172.21.15.1:0/3132389008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8f4157ba0 0x7fd8f4070db0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fd8e40098f0 tx=0x7fd8e4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1b623cdb2fbd917 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:12.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.199+0000 7fd8f2ffd700 1 -- 172.21.15.1:0/3132389008 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4011830 con 0x7fd8f4157ba0 2024-03-18T19:03:12.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.199+0000 7fd8f2ffd700 1 -- 172.21.15.1:0/3132389008 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e4011990 con 0x7fd8f4157ba0 2024-03-18T19:03:12.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.199+0000 7fd8f2ffd700 1 -- 172.21.15.1:0/3132389008 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4011d20 con 0x7fd8f4157ba0 2024-03-18T19:03:12.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.200+0000 7fd8fa2c5700 1 -- 172.21.15.1:0/3132389008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8f4157ba0 msgr2=0x7fd8f4070db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:12.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.200+0000 7fd8fa2c5700 1 --2- 172.21.15.1:0/3132389008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8f4157ba0 0x7fd8f4070db0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fd8e40098f0 tx=0x7fd8e4009c00 comp rx=0 tx=0).stop 2024-03-18T19:03:12.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.200+0000 7fd8fa2c5700 1 -- 172.21.15.1:0/3132389008 shutdown_connections 2024-03-18T19:03:12.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.200+0000 7fd8fa2c5700 1 --2- 172.21.15.1:0/3132389008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8f4157ba0 0x7fd8f4070db0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.200+0000 7fd8fa2c5700 1 --2- 172.21.15.1:0/3132389008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8f415e5e0 0x7fd8f40712f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.200+0000 7fd8fa2c5700 1 -- 172.21.15.1:0/3132389008 >> 172.21.15.1:0/3132389008 conn(0x7fd8f406a750 msgr2=0x7fd8f406ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:12.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.200+0000 7fd8fa2c5700 1 -- 172.21.15.1:0/3132389008 shutdown_connections 2024-03-18T19:03:12.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.200+0000 7fd8fa2c5700 1 -- 172.21.15.1:0/3132389008 wait complete. 2024-03-18T19:03:12.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.201+0000 7fd8fa2c5700 1 Processor -- start 2024-03-18T19:03:12.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.201+0000 7fd8fa2c5700 1 -- start start 2024-03-18T19:03:12.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.202+0000 7fd8fa2c5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8f406e2d0 0x7fd8f40b1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:12.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.202+0000 7fd8fa2c5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8f4157ba0 0x7fd8f40afe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:12.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.202+0000 7fd8fa2c5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8f415e5e0 0x7fd8f41eef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:12.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.202+0000 7fd8fa2c5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f4159170 con 0x7fd8f415e5e0 2024-03-18T19:03:12.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.202+0000 7fd8fa2c5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f41537f0 con 0x7fd8f406e2d0 2024-03-18T19:03:12.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.202+0000 7fd8fa2c5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f4158560 con 0x7fd8f4157ba0 2024-03-18T19:03:12.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.202+0000 7fd8f8862700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8f415e5e0 0x7fd8f41eef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:12.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.202+0000 7fd8f37fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8f4157ba0 0x7fd8f40afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:12.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.202+0000 7fd8f3fff700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8f406e2d0 0x7fd8f40b1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:12.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.202+0000 7fd8f8862700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8f415e5e0 0x7fd8f41eef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54076/0 (socket says 172.21.15.1:54076) 2024-03-18T19:03:12.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.202+0000 7fd8f8862700 1 -- 172.21.15.1:0/3307844998 learned_addr learned my addr 172.21.15.1:0/3307844998 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:12.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.202+0000 7fd8f37fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8f4157ba0 0x7fd8f40afe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:49222/0 (socket says 172.21.15.1:49222) 2024-03-18T19:03:12.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.203+0000 7fd8f8862700 1 -- 172.21.15.1:0/3307844998 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8f4157ba0 msgr2=0x7fd8f40afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:12.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.203+0000 7fd8f8862700 1 --2- 172.21.15.1:0/3307844998 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8f4157ba0 0x7fd8f40afe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.203+0000 7fd8f8862700 1 -- 172.21.15.1:0/3307844998 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8f406e2d0 msgr2=0x7fd8f40b1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:12.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.203+0000 7fd8f8862700 1 --2- 172.21.15.1:0/3307844998 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8f406e2d0 0x7fd8f40b1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.203+0000 7fd8f8862700 1 -- 172.21.15.1:0/3307844998 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e4009550 con 0x7fd8f415e5e0 2024-03-18T19:03:12.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.203+0000 7fd8f8862700 1 --2- 172.21.15.1:0/3307844998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8f415e5e0 0x7fd8f41eef00 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec00efc0 tx=0x7fd8ec00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:12.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.203+0000 7fd8f0ff9700 1 -- 172.21.15.1:0/3307844998 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec009ea0 con 0x7fd8f415e5e0 2024-03-18T19:03:12.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.203+0000 7fd8f0ff9700 1 -- 172.21.15.1:0/3307844998 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec00f040 con 0x7fd8f415e5e0 2024-03-18T19:03:12.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.203+0000 7fd8f0ff9700 1 -- 172.21.15.1:0/3307844998 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec00ced0 con 0x7fd8f415e5e0 2024-03-18T19:03:12.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.203+0000 7fd8fa2c5700 1 -- 172.21.15.1:0/3307844998 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f4158840 con 0x7fd8f415e5e0 2024-03-18T19:03:12.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.203+0000 7fd8fa2c5700 1 -- 172.21.15.1:0/3307844998 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f4158e00 con 0x7fd8f415e5e0 2024-03-18T19:03:12.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.204+0000 7fd8fa2c5700 1 -- 172.21.15.1:0/3307844998 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f4159320 con 0x7fd8f415e5e0 2024-03-18T19:03:12.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.205+0000 7fd8f0ff9700 1 -- 172.21.15.1:0/3307844998 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fd8ec014480 con 0x7fd8f415e5e0 2024-03-18T19:03:12.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.205+0000 7fd8f0ff9700 1 --2- 172.21.15.1:0/3307844998 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd8e006ae30 0x7fd8e006d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:12.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.205+0000 7fd8f3fff700 1 -- 172.21.15.1:0/3307844998 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd8e006ae30 msgr2=0x7fd8e006d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:03:12.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.205+0000 7fd8f0ff9700 1 -- 172.21.15.1:0/3307844998 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd8ec08a140 con 0x7fd8f415e5e0 2024-03-18T19:03:12.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.205+0000 7fd8f3fff700 1 --2- 172.21.15.1:0/3307844998 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd8e006ae30 0x7fd8e006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:03:12.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.210+0000 7fd8f0ff9700 1 -- 172.21.15.1:0/3307844998 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd8f4159320 con 0x7fd8f415e5e0 2024-03-18T19:03:12.319 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:12.323+0000 7f6b5e0dc240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:03:12.362 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:12.366+0000 7fd33f2a6240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:03:12.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.406+0000 7fd8f3fff700 1 -- 172.21.15.1:0/3307844998 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd8e006ae30 msgr2=0x7fd8e006d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:03:12.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.406+0000 7fd8f3fff700 1 --2- 172.21.15.1:0/3307844998 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd8e006ae30 0x7fd8e006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:03:12.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.461+0000 7fd8fa2c5700 1 -- 172.21.15.1:0/3307844998 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fd8f406b6e0 con 0x7fd8f415e5e0 2024-03-18T19:03:12.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.465+0000 7fd8f0ff9700 1 -- 172.21.15.1:0/3307844998 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v17)=0 v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd8ec055050 con 0x7fd8f415e5e0 2024-03-18T19:03:12.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.467+0000 7fd8fa2c5700 1 -- 172.21.15.1:0/3307844998 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd8e006ae30 msgr2=0x7fd8e006d2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:03:12.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.467+0000 7fd8fa2c5700 1 --2- 172.21.15.1:0/3307844998 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd8e006ae30 0x7fd8e006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.467+0000 7fd8fa2c5700 1 -- 172.21.15.1:0/3307844998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8f415e5e0 msgr2=0x7fd8f41eef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:12.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.467+0000 7fd8fa2c5700 1 --2- 172.21.15.1:0/3307844998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8f415e5e0 0x7fd8f41eef00 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec00efc0 tx=0x7fd8ec00c3a0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.467+0000 7fd8fa2c5700 1 -- 172.21.15.1:0/3307844998 shutdown_connections 2024-03-18T19:03:12.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.467+0000 7fd8fa2c5700 1 --2- 172.21.15.1:0/3307844998 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd8e006ae30 0x7fd8e006d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.467+0000 7fd8fa2c5700 1 --2- 172.21.15.1:0/3307844998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8f415e5e0 0x7fd8f41eef00 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.467+0000 7fd8fa2c5700 1 --2- 172.21.15.1:0/3307844998 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8f406e2d0 0x7fd8f40b1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.467+0000 7fd8fa2c5700 1 --2- 172.21.15.1:0/3307844998 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8f4157ba0 0x7fd8f40afe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.467+0000 7fd8fa2c5700 1 -- 172.21.15.1:0/3307844998 >> 172.21.15.1:0/3307844998 conn(0x7fd8f406a750 msgr2=0x7fd8f40b3860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:12.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.467+0000 7fd8fa2c5700 1 -- 172.21.15.1:0/3307844998 shutdown_connections 2024-03-18T19:03:12.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.467+0000 7fd8fa2c5700 1 -- 172.21.15.1:0/3307844998 wait complete. 2024-03-18T19:03:12.475 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-18T19:03:12.583 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:12.587+0000 7f6b5e0dc240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:03:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.609+0000 7f7f1b57c700 1 Processor -- start 2024-03-18T19:03:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.609+0000 7f7f1b57c700 1 -- start start 2024-03-18T19:03:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.610+0000 7f7f1b57c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f1415d780 0x7f7f140cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.610+0000 7f7f1b57c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7f1409dd40 0x7f7f140cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.610+0000 7f7f1b57c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f140ce090 con 0x7f7f140ce5a0 2024-03-18T19:03:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.610+0000 7f7f1b57c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f140cf120 con 0x7f7f1409dd40 2024-03-18T19:03:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.610+0000 7f7f1b57c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1406db30 con 0x7f7f1415d780 2024-03-18T19:03:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.610+0000 7f7f18b17700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7f1409dd40 0x7f7f140cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.610+0000 7f7f18b17700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7f1409dd40 0x7f7f140cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54078/0 (socket says 172.21.15.1:54078) 2024-03-18T19:03:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.610+0000 7f7f18b17700 1 -- 172.21.15.1:0/2390016669 learned_addr learned my addr 172.21.15.1:0/2390016669 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.610+0000 7f7f19318700 1 --2- 172.21.15.1:0/2390016669 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f1415d780 0x7f7f140cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.610+0000 7f7f18b17700 1 -- 172.21.15.1:0/2390016669 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f1415d780 msgr2=0x7f7f140cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.610+0000 7f7f18b17700 1 --2- 172.21.15.1:0/2390016669 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f1415d780 0x7f7f140cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.610+0000 7f7f18b17700 1 -- 172.21.15.1:0/2390016669 >> v1:172.21.15.165:6789/0 conn(0x7f7f140ce5a0 legacy=0x7f7f140cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:12.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.610+0000 7f7f18b17700 1 -- 172.21.15.1:0/2390016669 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f1406d520 con 0x7f7f1409dd40 2024-03-18T19:03:12.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.610+0000 7f7f19318700 1 --2- 172.21.15.1:0/2390016669 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f1415d780 0x7f7f140cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:03:12.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.611+0000 7f7f18b17700 1 --2- 172.21.15.1:0/2390016669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f1409dd40 0x7f7f140cad70 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f7f100098f0 tx=0x7f7f10009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdfccebbe8e15a20 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:12.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.611+0000 7f7f0bfff700 1 -- 172.21.15.1:0/2390016669 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1000b8a0 con 0x7f7f1409dd40 2024-03-18T19:03:12.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.611+0000 7f7f0bfff700 1 -- 172.21.15.1:0/2390016669 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1000ba00 con 0x7f7f1409dd40 2024-03-18T19:03:12.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.611+0000 7f7f0bfff700 1 -- 172.21.15.1:0/2390016669 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1000bd90 con 0x7f7f1409dd40 2024-03-18T19:03:12.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.611+0000 7f7f1b57c700 1 -- 172.21.15.1:0/2390016669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f1409dd40 msgr2=0x7f7f140cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:12.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.611+0000 7f7f1b57c700 1 --2- 172.21.15.1:0/2390016669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f1409dd40 0x7f7f140cad70 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f7f100098f0 tx=0x7f7f10009c00 comp rx=0 tx=0).stop 2024-03-18T19:03:12.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.611+0000 7f7f1b57c700 1 -- 172.21.15.1:0/2390016669 shutdown_connections 2024-03-18T19:03:12.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.611+0000 7f7f1b57c700 1 --2- 172.21.15.1:0/2390016669 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f1415d780 0x7f7f140cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.611+0000 7f7f1b57c700 1 --2- 172.21.15.1:0/2390016669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f1409dd40 0x7f7f140cad70 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.611+0000 7f7f1b57c700 1 -- 172.21.15.1:0/2390016669 >> 172.21.15.1:0/2390016669 conn(0x7f7f140cca00 msgr2=0x7f7f140cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:12.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.611+0000 7f7f1b57c700 1 -- 172.21.15.1:0/2390016669 shutdown_connections 2024-03-18T19:03:12.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.612+0000 7f7f1b57c700 1 -- 172.21.15.1:0/2390016669 wait complete. 2024-03-18T19:03:12.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.613+0000 7f7f1b57c700 1 Processor -- start 2024-03-18T19:03:12.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.613+0000 7f7f1b57c700 1 -- start start 2024-03-18T19:03:12.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.613+0000 7f7f1b57c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f1409dd40 0x7f7f141597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:12.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.613+0000 7f7f1b57c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f140ce5a0 0x7f7f140c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:12.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.613+0000 7f7f1b57c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f1415d780 0x7f7f140b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:12.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.613+0000 7f7f1b57c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f141f4090 con 0x7f7f1415d780 2024-03-18T19:03:12.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.613+0000 7f7f1b57c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f14159d20 con 0x7f7f140ce5a0 2024-03-18T19:03:12.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.613+0000 7f7f1b57c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f140c69c0 con 0x7f7f1409dd40 2024-03-18T19:03:12.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.614+0000 7f7f19b19700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f1415d780 0x7f7f140b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:12.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.614+0000 7f7f19b19700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f1415d780 0x7f7f140b4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54086/0 (socket says 172.21.15.1:54086) 2024-03-18T19:03:12.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.614+0000 7f7f19b19700 1 -- 172.21.15.1:0/4122280702 learned_addr learned my addr 172.21.15.1:0/4122280702 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:12.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.614+0000 7f7f18b17700 1 --2- 172.21.15.1:0/4122280702 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f140ce5a0 0x7f7f140c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:12.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.614+0000 7f7f19318700 1 --2- 172.21.15.1:0/4122280702 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f1409dd40 0x7f7f141597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:12.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.614+0000 7f7f19b19700 1 -- 172.21.15.1:0/4122280702 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f1409dd40 msgr2=0x7f7f141597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:12.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.614+0000 7f7f19b19700 1 --2- 172.21.15.1:0/4122280702 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f1409dd40 0x7f7f141597e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.614+0000 7f7f19b19700 1 -- 172.21.15.1:0/4122280702 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f140ce5a0 msgr2=0x7f7f140c27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.614+0000 7f7f19b19700 1 --2- 172.21.15.1:0/4122280702 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f140ce5a0 0x7f7f140c27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.614+0000 7f7f19b19700 1 -- 172.21.15.1:0/4122280702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f10009550 con 0x7f7f1415d780 2024-03-18T19:03:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.614+0000 7f7f19318700 1 --2- 172.21.15.1:0/4122280702 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f1409dd40 0x7f7f141597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:03:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.614+0000 7f7f19b19700 1 --2- 172.21.15.1:0/4122280702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f1415d780 0x7f7f140b4e60 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00cc50 tx=0x7f7f0c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.614+0000 7f7f09ffb700 1 -- 172.21.15.1:0/4122280702 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c003c00 con 0x7f7f1415d780 2024-03-18T19:03:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.614+0000 7f7f09ffb700 1 -- 172.21.15.1:0/4122280702 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0c003d60 con 0x7f7f1415d780 2024-03-18T19:03:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.614+0000 7f7f09ffb700 1 -- 172.21.15.1:0/4122280702 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c00ab60 con 0x7f7f1415d780 2024-03-18T19:03:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.614+0000 7f7f1b57c700 1 -- 172.21.15.1:0/4122280702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f141f3410 con 0x7f7f1415d780 2024-03-18T19:03:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.615+0000 7f7f1b57c700 1 -- 172.21.15.1:0/4122280702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f141f39d0 con 0x7f7f1415d780 2024-03-18T19:03:12.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.615+0000 7f7f1b57c700 1 -- 172.21.15.1:0/4122280702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f140676a0 con 0x7f7f1415d780 2024-03-18T19:03:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.616+0000 7f7f09ffb700 1 -- 172.21.15.1:0/4122280702 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f7f0c00acc0 con 0x7f7f1415d780 2024-03-18T19:03:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.617+0000 7f7f09ffb700 1 --2- 172.21.15.1:0/4122280702 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7ef806ae00 0x7f7ef806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.617+0000 7f7f19318700 1 -- 172.21.15.1:0/4122280702 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7ef806ae00 msgr2=0x7f7ef806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:03:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.617+0000 7f7f09ffb700 1 -- 172.21.15.1:0/4122280702 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7f0c089680 con 0x7f7f1415d780 2024-03-18T19:03:12.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.617+0000 7f7f19318700 1 --2- 172.21.15.1:0/4122280702 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7ef806ae00 0x7f7ef806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:03:12.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.621+0000 7f7f09ffb700 1 -- 172.21.15.1:0/4122280702 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7f0c054610 con 0x7f7f1415d780 2024-03-18T19:03:12.649 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:12.653+0000 7fd33f2a6240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:03:12.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.818+0000 7f7f19318700 1 -- 172.21.15.1:0/4122280702 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7ef806ae00 msgr2=0x7f7ef806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:03:12.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.818+0000 7f7f19318700 1 --2- 172.21.15.1:0/4122280702 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7ef806ae00 0x7f7ef806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:03:12.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.827+0000 7f7f1b57c700 1 -- 172.21.15.1:0/4122280702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f7f14072490 con 0x7f7f1415d780 2024-03-18T19:03:12.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.832+0000 7f7f09ffb700 1 -- 172.21.15.1:0/4122280702 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v18)=0 v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7f7f0c057c30 con 0x7f7f1415d780 2024-03-18T19:03:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.835+0000 7f7f1b57c700 1 -- 172.21.15.1:0/4122280702 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7ef806ae00 msgr2=0x7f7ef806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:03:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.835+0000 7f7f1b57c700 1 --2- 172.21.15.1:0/4122280702 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7ef806ae00 0x7f7ef806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.835+0000 7f7f1b57c700 1 -- 172.21.15.1:0/4122280702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f1415d780 msgr2=0x7f7f140b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.835+0000 7f7f1b57c700 1 --2- 172.21.15.1:0/4122280702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f1415d780 0x7f7f140b4e60 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00cc50 tx=0x7f7f0c00cf60 comp rx=0 tx=0).stop 2024-03-18T19:03:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.835+0000 7f7f1b57c700 1 -- 172.21.15.1:0/4122280702 shutdown_connections 2024-03-18T19:03:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.835+0000 7f7f1b57c700 1 --2- 172.21.15.1:0/4122280702 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7ef806ae00 0x7f7ef806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.835+0000 7f7f1b57c700 1 --2- 172.21.15.1:0/4122280702 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f1409dd40 0x7f7f141597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.835+0000 7f7f1b57c700 1 --2- 172.21.15.1:0/4122280702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f1415d780 0x7f7f140b4e60 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.835+0000 7f7f1b57c700 1 --2- 172.21.15.1:0/4122280702 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f140ce5a0 0x7f7f140c27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:12.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.835+0000 7f7f1b57c700 1 -- 172.21.15.1:0/4122280702 >> 172.21.15.1:0/4122280702 conn(0x7f7f140cca00 msgr2=0x7f7f140b4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:12.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.835+0000 7f7f1b57c700 1 -- 172.21.15.1:0/4122280702 shutdown_connections 2024-03-18T19:03:12.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:12.835+0000 7f7f1b57c700 1 -- 172.21.15.1:0/4122280702 wait complete. 2024-03-18T19:03:12.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:12.843 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-03-18T19:03:12.865 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-03-18T19:03:13.006 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:13.009+0000 7f6b5e0dc240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:03:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.041+0000 7f2d7a40f700 1 Processor -- start 2024-03-18T19:03:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.041+0000 7f2d7a40f700 1 -- start start 2024-03-18T19:03:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.042+0000 7f2d7a40f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d740b0500 0x7f2d740cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.042+0000 7f2d7a40f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2d740b0e20 0x7f2d740b1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.042+0000 7f2d7a40f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d7415ef80 con 0x7f2d7415eba0 2024-03-18T19:03:13.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.042+0000 7f2d7a40f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d740cc9a0 con 0x7f2d740b0e20 2024-03-18T19:03:13.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.042+0000 7f2d7a40f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d740ccc80 con 0x7f2d740b0500 2024-03-18T19:03:13.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.043+0000 7f2d789ac700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2d740b0e20 0x7f2d740b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:13.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.043+0000 7f2d73fff700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f2d7415eba0 0x7f2d740cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:59502/0 (socket says 172.21.15.1:59502) 2024-03-18T19:03:13.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.043+0000 7f2d73fff700 1 -- 172.21.15.1:0/4068790169 learned_addr learned my addr 172.21.15.1:0/4068790169 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:03:13.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.043+0000 7f2d737fe700 1 --2- 172.21.15.1:0/4068790169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d740b0500 0x7f2d740cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:13.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.043+0000 7f2d789ac700 1 -- 172.21.15.1:0/4068790169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d740b0500 msgr2=0x7f2d740cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:13.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.043+0000 7f2d789ac700 1 --2- 172.21.15.1:0/4068790169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d740b0500 0x7f2d740cfa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:13.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.043+0000 7f2d789ac700 1 -- 172.21.15.1:0/4068790169 >> v1:172.21.15.165:6789/0 conn(0x7f2d7415eba0 legacy=0x7f2d740cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:13.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.043+0000 7f2d789ac700 1 -- 172.21.15.1:0/4068790169 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d740ccf60 con 0x7f2d740b0e20 2024-03-18T19:03:13.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.043+0000 7f2d789ac700 1 --2- 172.21.15.1:0/4068790169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d740b0e20 0x7f2d740b1280 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00f370 tx=0x7f2d6c00f680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38ab07c2e669c4cb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:13.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.043+0000 7f2d72ffd700 1 -- 172.21.15.1:0/4068790169 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c012070 con 0x7f2d740b0e20 2024-03-18T19:03:13.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.043+0000 7f2d72ffd700 1 -- 172.21.15.1:0/4068790169 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6c00fe70 con 0x7f2d740b0e20 2024-03-18T19:03:13.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.043+0000 7f2d72ffd700 1 -- 172.21.15.1:0/4068790169 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c00d2f0 con 0x7f2d740b0e20 2024-03-18T19:03:13.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.044+0000 7f2d7a40f700 1 -- 172.21.15.1:0/4068790169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d740b0e20 msgr2=0x7f2d740b1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:13.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.044+0000 7f2d7a40f700 1 --2- 172.21.15.1:0/4068790169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d740b0e20 0x7f2d740b1280 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00f370 tx=0x7f2d6c00f680 comp rx=0 tx=0).stop 2024-03-18T19:03:13.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.044+0000 7f2d7a40f700 1 -- 172.21.15.1:0/4068790169 shutdown_connections 2024-03-18T19:03:13.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.044+0000 7f2d7a40f700 1 --2- 172.21.15.1:0/4068790169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d740b0e20 0x7f2d740b1280 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:13.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.044+0000 7f2d7a40f700 1 --2- 172.21.15.1:0/4068790169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d740b0500 0x7f2d740cfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:13.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.044+0000 7f2d7a40f700 1 -- 172.21.15.1:0/4068790169 >> 172.21.15.1:0/4068790169 conn(0x7f2d740b2bd0 msgr2=0x7f2d740b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:13.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.044+0000 7f2d7a40f700 1 -- 172.21.15.1:0/4068790169 shutdown_connections 2024-03-18T19:03:13.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.044+0000 7f2d7a40f700 1 -- 172.21.15.1:0/4068790169 wait complete. 2024-03-18T19:03:13.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.046+0000 7f2d7a40f700 1 Processor -- start 2024-03-18T19:03:13.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.046+0000 7f2d7a40f700 1 -- start start 2024-03-18T19:03:13.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.047+0000 7f2d7a40f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d740b0500 0x7f2d740c7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:13.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.047+0000 7f2d7a40f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d740b0e20 0x7f2d741f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:13.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.047+0000 7f2d7a40f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d7415eba0 0x7f2d740c6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:13.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.047+0000 7f2d7a40f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d74152e40 con 0x7f2d740b0500 2024-03-18T19:03:13.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.047+0000 7f2d7a40f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d740c7850 con 0x7f2d7415eba0 2024-03-18T19:03:13.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.047+0000 7f2d7a40f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d741f2ac0 con 0x7f2d740b0e20 2024-03-18T19:03:13.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.047+0000 7f2d73fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d740b0500 0x7f2d740c7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:13.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.047+0000 7f2d73fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d740b0500 0x7f2d740c7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54094/0 (socket says 172.21.15.1:54094) 2024-03-18T19:03:13.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.047+0000 7f2d73fff700 1 -- 172.21.15.1:0/1125205916 learned_addr learned my addr 172.21.15.1:0/1125205916 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:13.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.047+0000 7f2d789ac700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d7415eba0 0x7f2d740c6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:13.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.047+0000 7f2d737fe700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d740b0e20 0x7f2d741f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:13.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.047+0000 7f2d73fff700 1 -- 172.21.15.1:0/1125205916 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d740b0e20 msgr2=0x7f2d741f2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:13.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.047+0000 7f2d73fff700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d740b0e20 0x7f2d741f2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:13.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.047+0000 7f2d73fff700 1 -- 172.21.15.1:0/1125205916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d7415eba0 msgr2=0x7f2d740c6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:13.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.047+0000 7f2d73fff700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d7415eba0 0x7f2d740c6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:13.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.047+0000 7f2d73fff700 1 -- 172.21.15.1:0/1125205916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6c00f050 con 0x7f2d740b0500 2024-03-18T19:03:13.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.048+0000 7f2d73fff700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d740b0500 0x7f2d740c7310 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400c4b0 tx=0x7f2d6400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:13.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.048+0000 7f2d70ff9700 1 -- 172.21.15.1:0/1125205916 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d64005d70 con 0x7f2d740b0500 2024-03-18T19:03:13.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.048+0000 7f2d70ff9700 1 -- 172.21.15.1:0/1125205916 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6400ce80 con 0x7f2d740b0500 2024-03-18T19:03:13.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.048+0000 7f2d70ff9700 1 -- 172.21.15.1:0/1125205916 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d640072d0 con 0x7f2d740b0500 2024-03-18T19:03:13.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.048+0000 7f2d7a40f700 1 -- 172.21.15.1:0/1125205916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d741f2da0 con 0x7f2d740b0500 2024-03-18T19:03:13.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.048+0000 7f2d7a40f700 1 -- 172.21.15.1:0/1125205916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7415c010 con 0x7f2d740b0500 2024-03-18T19:03:13.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.049+0000 7f2d7a40f700 1 -- 172.21.15.1:0/1125205916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d741f2fe0 con 0x7f2d740b0500 2024-03-18T19:03:13.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.050+0000 7f2d70ff9700 1 -- 172.21.15.1:0/1125205916 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f2d64007430 con 0x7f2d740b0500 2024-03-18T19:03:13.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.050+0000 7f2d70ff9700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2d6006ae00 0x7f2d6006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:13.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.050+0000 7f2d737fe700 1 -- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2d6006ae00 msgr2=0x7f2d6006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:03:13.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.050+0000 7f2d70ff9700 1 -- 172.21.15.1:0/1125205916 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2d64088780 con 0x7f2d740b0500 2024-03-18T19:03:13.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.050+0000 7f2d737fe700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2d6006ae00 0x7f2d6006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:03:13.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.055+0000 7f2d70ff9700 1 -- 172.21.15.1:0/1125205916 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2d741f2fe0 con 0x7f2d740b0500 2024-03-18T19:03:13.115 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:13.119+0000 7fd33f2a6240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:03:13.214 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:13.218+0000 7f6b5e0dc240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:03:13.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.251+0000 7f2d737fe700 1 -- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2d6006ae00 msgr2=0x7f2d6006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:03:13.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.251+0000 7f2d737fe700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2d6006ae00 0x7f2d6006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:03:13.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.284+0000 7f2d7a40f700 1 -- 172.21.15.1:0/1125205916 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d740b17c0 con 0x7f2d6006ae00 2024-03-18T19:03:13.333 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:13.338+0000 7fd33f2a6240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:03:13.583 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:13.587+0000 7f6b5e0dc240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:03:13.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.652+0000 7f2d737fe700 1 -- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2d6006ae00 msgr2=0x7f2d6006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:03:13.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:13.652+0000 7f2d737fe700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2d6006ae00 0x7f2d6006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T19:03:13.721 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:13.725+0000 7fd33f2a6240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:03:14.165 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:14.169+0000 7f6b5e0dc240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:03:14.235 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:14.239+0000 7f6b5e0dc240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:03:14.337 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:14.341+0000 7fd33f2a6240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:03:14.360 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:14.364+0000 7f6b5e0dc240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:03:14.412 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:14.417+0000 7fd33f2a6240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:03:14.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:14.453+0000 7f2d737fe700 1 -- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2d6006ae00 msgr2=0x7f2d6006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:03:14.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:14.453+0000 7f2d737fe700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2d6006ae00 0x7f2d6006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T19:03:14.538 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:14.542+0000 7f6b5e0dc240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:03:14.547 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:14.551+0000 7fd33f2a6240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:03:14.608 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:14.612+0000 7f6b5e0dc240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:03:14.686 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:14.689+0000 7f6b5e0dc240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:03:14.732 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:14.736+0000 7fd33f2a6240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:03:14.761 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:14.765+0000 7f6b5e0dc240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:03:14.808 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:14.812+0000 7fd33f2a6240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:03:14.838 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:14.841+0000 7f6b5e0dc240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:03:14.889 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:14.893+0000 7fd33f2a6240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:03:14.970 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:14.974+0000 7fd33f2a6240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:03:15.003 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:15.007+0000 7f6b5e0dc240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:03:15.051 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:15.055+0000 7fd33f2a6240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:03:15.216 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:15.219+0000 7f6b5e0dc240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:03:15.229 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:15.233+0000 7fd33f2a6240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:03:15.364 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:15.367+0000 7f6b5e0dc240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:03:15.434 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:15.438+0000 7f6b5e0dc240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:03:15.449 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:15.453+0000 7fd33f2a6240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:03:15.606 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:15.611+0000 7fd33f2a6240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:03:15.682 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:15.686+0000 7fd33f2a6240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:03:15.870 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:15.875+0000 7f6b5e0dc240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:03:16.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:16.054+0000 7f2d737fe700 1 -- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2d6006ae00 msgr2=0x7f2d6006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:03:16.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:16.054+0000 7f2d737fe700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2d6006ae00 0x7f2d6006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T19:03:16.154 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:16.158+0000 7fd33f2a6240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:03:16.204 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:16.209+0000 7f6b5e0dc240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:03:16.363 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:16.367+0000 7f6b5e0dc240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:03:16.502 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:16.506+0000 7fd33f2a6240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:03:16.672 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:16.676+0000 7fd33f2a6240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:03:16.743 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:16.747+0000 7f6b5e0dc240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:03:16.905 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:16.910+0000 7f6b5e0dc240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:03:17.083 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:17.087+0000 7fd33f2a6240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:03:17.257 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:17.261+0000 7fd33f2a6240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:03:17.265 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:17.270+0000 7f6b5e0dc240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:03:17.351 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:03:17.356+0000 7f6b5e0dc240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:03:17.629 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:17.634+0000 7fd33f2a6240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:03:17.720 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:17.725+0000 7fd33f2a6240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:03:17.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:17.745+0000 7f2d70ff9700 1 -- 172.21.15.1:0/1125205916 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 20) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f2d64053c20 con 0x7f2d740b0500 2024-03-18T19:03:17.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:17.745+0000 7f2d70ff9700 1 -- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2d6006ae00 msgr2=0x7f2d6006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:03:17.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:17.745+0000 7f2d70ff9700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2d6006ae00 0x7f2d6006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:17.896 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:17.899+0000 7fd2b4ec9700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:03:17.896 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:17.899+0000 7fd2b4ec9700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:03:17.896 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:17.899+0000 7fd2b4ec9700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:03:17.896 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:17.899+0000 7fd2b4ec9700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:03:17.896 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:17.899+0000 7fd2b1ec3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:03:17.896 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:17.899+0000 7fd2b1ec3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:03:17.896 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:17.899+0000 7fd2b1ec3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:03:17.896 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:17.899+0000 7fd2b1ec3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:03:17.896 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:17.899+0000 7fd2b1ec3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:03:17.896 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:03:17.899+0000 7fd2b4ec9700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:03:18.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.748+0000 7f2d70ff9700 1 -- 172.21.15.1:0/1125205916 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 21) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f2d64086b30 con 0x7f2d740b0500 2024-03-18T19:03:18.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.749+0000 7f2d70ff9700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2d6006ae00 0x7f2d6006d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:18.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.749+0000 7f2d70ff9700 1 -- 172.21.15.1:0/1125205916 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d60035990 con 0x7f2d6006ae00 2024-03-18T19:03:18.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.751+0000 7f2d737fe700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2d6006ae00 0x7f2d6006d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:18.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.755+0000 7f2d737fe700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2d6006ae00 0x7f2d6006d240 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2d740afe10 tx=0x7f2d68009210 comp rx=0 tx=0).ready entity=mgr.6459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:18.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.819+0000 7f2d70ff9700 1 -- 172.21.15.1:0/1125205916 <== mgr.6459 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2d60035990 con 0x7f2d6006ae00 2024-03-18T19:03:18.815 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path snap_test_dir1 2024-03-18T19:03:18.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.821+0000 7f2d7a40f700 1 -- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2d6006ae00 msgr2=0x7f2d6006d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.821+0000 7f2d7a40f700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2d6006ae00 0x7f2d6006d240 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2d740afe10 tx=0x7f2d68009210 comp rx=0 tx=0).stop 2024-03-18T19:03:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.821+0000 7f2d7a40f700 1 -- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d740b0500 msgr2=0x7f2d740c7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.821+0000 7f2d7a40f700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d740b0500 0x7f2d740c7310 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400c4b0 tx=0x7f2d6400c7c0 comp rx=0 tx=0).stop 2024-03-18T19:03:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.821+0000 7f2d7a40f700 1 -- 172.21.15.1:0/1125205916 shutdown_connections 2024-03-18T19:03:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.821+0000 7f2d7a40f700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2d6006ae00 0x7f2d6006d240 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.821+0000 7f2d7a40f700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d7415eba0 0x7f2d740c6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.821+0000 7f2d7a40f700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d740b0500 0x7f2d740c7310 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.821+0000 7f2d7a40f700 1 --2- 172.21.15.1:0/1125205916 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d740b0e20 0x7f2d741f2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.821+0000 7f2d7a40f700 1 -- 172.21.15.1:0/1125205916 >> 172.21.15.1:0/1125205916 conn(0x7f2d740b2bd0 msgr2=0x7f2d7406e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.821+0000 7f2d7a40f700 1 -- 172.21.15.1:0/1125205916 shutdown_connections 2024-03-18T19:03:18.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.821+0000 7f2d7a40f700 1 -- 172.21.15.1:0/1125205916 wait complete. 2024-03-18T19:03:18.830 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-03-18T19:03:18.830 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 2m 2024-03-18T19:03:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.964+0000 7f8928d8f700 1 Processor -- start 2024-03-18T19:03:18.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.964+0000 7f8928d8f700 1 -- start start 2024-03-18T19:03:18.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.965+0000 7f8928d8f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8924158e30 0x7f8924071500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.965+0000 7f8928d8f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89240cbf70 0x7f89240cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.965+0000 7f8928d8f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892415c500 con 0x7f892406a750 2024-03-18T19:03:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.965+0000 7f8928d8f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89240af8b0 con 0x7f8924158e30 2024-03-18T19:03:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.965+0000 7f8928d8f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892415b8f0 con 0x7f89240cbf70 2024-03-18T19:03:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.965+0000 7f8922d9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f892406a750 0x7f89240cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51718/0 (socket says 172.21.15.1:51718) 2024-03-18T19:03:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.965+0000 7f8922d9d700 1 -- 172.21.15.1:0/3756141192 learned_addr learned my addr 172.21.15.1:0/3756141192 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:03:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.965+0000 7f8921d9b700 1 --2- 172.21.15.1:0/3756141192 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89240cbf70 0x7f89240cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.965+0000 7f892259c700 1 --2- 172.21.15.1:0/3756141192 >> v2:172.21.15.165:3300/0 conn(0x7f8924158e30 0x7f8924071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.965+0000 7f892159a700 1 -- 172.21.15.1:0/3756141192 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101070831 0 0) 0x7f892415c500 con 0x7f892406a750 2024-03-18T19:03:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.965+0000 7f892159a700 1 -- 172.21.15.1:0/3756141192 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f890c0035b0 con 0x7f892406a750 2024-03-18T19:03:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892159a700 1 -- 172.21.15.1:0/3756141192 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4180367001 0 0) 0x7f890c0035b0 con 0x7f892406a750 2024-03-18T19:03:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892159a700 1 -- 172.21.15.1:0/3756141192 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89240cbf70 msgr2=0x7f89240cc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:18.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892159a700 1 --2- 172.21.15.1:0/3756141192 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89240cbf70 0x7f89240cc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892159a700 1 -- 172.21.15.1:0/3756141192 >> v2:172.21.15.165:3300/0 conn(0x7f8924158e30 msgr2=0x7f8924071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892159a700 1 --2- 172.21.15.1:0/3756141192 >> v2:172.21.15.165:3300/0 conn(0x7f8924158e30 0x7f8924071500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892159a700 1 -- 172.21.15.1:0/3756141192 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892415bbd0 con 0x7f892406a750 2024-03-18T19:03:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892159a700 1 -- 172.21.15.1:0/3756141192 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f891c0019f0 con 0x7f892406a750 2024-03-18T19:03:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892159a700 1 -- 172.21.15.1:0/3756141192 >> v1:172.21.15.1:6789/0 conn(0x7f892406a750 legacy=0x7f89240cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892159a700 1 --2- 172.21.15.1:0/3756141192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f890c003a00 0x7f890c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892159a700 1 --2- 172.21.15.1:0/3756141192 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f890c004850 0x7f890c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892159a700 1 --2- 172.21.15.1:0/3756141192 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89240cbf70 0x7f890c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892159a700 1 -- 172.21.15.1:0/3756141192 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89240af8b0 con 0x7f890c003a00 2024-03-18T19:03:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892159a700 1 -- 172.21.15.1:0/3756141192 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892415b8f0 con 0x7f890c004850 2024-03-18T19:03:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892159a700 1 -- 172.21.15.1:0/3756141192 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f890c00ba10 con 0x7f89240cbf70 2024-03-18T19:03:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892259c700 1 --2- 172.21.15.1:0/3756141192 >> v2:172.21.15.165:3300/0 conn(0x7f8924158e30 0x7f8924071500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:03:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892259c700 1 --2- 172.21.15.1:0/3756141192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f890c003a00 0x7f890c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:18.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f8921d9b700 1 --2- 172.21.15.1:0/3756141192 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f890c004850 0x7f890c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:18.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f8922d9d700 1 --2- 172.21.15.1:0/3756141192 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89240cbf70 0x7f890c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:18.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892259c700 1 -- 172.21.15.1:0/3756141192 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89240cbf70 msgr2=0x7f890c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:18.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892259c700 1 --2- 172.21.15.1:0/3756141192 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89240cbf70 0x7f890c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:18.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892259c700 1 -- 172.21.15.1:0/3756141192 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f890c004850 msgr2=0x7f890c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:18.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892259c700 1 --2- 172.21.15.1:0/3756141192 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f890c004850 0x7f890c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:18.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892259c700 1 -- 172.21.15.1:0/3756141192 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890c0035b0 con 0x7f890c003a00 2024-03-18T19:03:18.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892259c700 1 --2- 172.21.15.1:0/3756141192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f890c003a00 0x7f890c003e20 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f8918009d10 tx=0x7f890c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70afc5a404f3c93d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:18.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.966+0000 7f892159a700 1 -- 172.21.15.1:0/3756141192 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918009e60 con 0x7f890c003a00 2024-03-18T19:03:18.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.967+0000 7f892159a700 1 -- 172.21.15.1:0/3756141192 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891800e9a0 con 0x7f890c003a00 2024-03-18T19:03:18.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.967+0000 7f892159a700 1 -- 172.21.15.1:0/3756141192 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891800ed10 con 0x7f890c003a00 2024-03-18T19:03:18.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.967+0000 7f8928d8f700 1 -- 172.21.15.1:0/3756141192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f890c003a00 msgr2=0x7f890c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:18.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.967+0000 7f8928d8f700 1 --2- 172.21.15.1:0/3756141192 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f890c003a00 0x7f890c003e20 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f8918009d10 tx=0x7f890c005880 comp rx=0 tx=0).stop 2024-03-18T19:03:18.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.967+0000 7f8922d9d700 1 -- 172.21.15.1:0/3756141192 reap_dead start 2024-03-18T19:03:18.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.967+0000 7f8928d8f700 1 -- 172.21.15.1:0/3756141192 shutdown_connections 2024-03-18T19:03:18.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.967+0000 7f8928d8f700 1 -- 172.21.15.1:0/3756141192 >> 172.21.15.1:0/3756141192 conn(0x7f89240cca00 msgr2=0x7f89240cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:18.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.967+0000 7f8928d8f700 1 -- 172.21.15.1:0/3756141192 shutdown_connections 2024-03-18T19:03:18.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.967+0000 7f8928d8f700 1 -- 172.21.15.1:0/3756141192 wait complete. 2024-03-18T19:03:18.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.969+0000 7f8928d8f700 1 Processor -- start 2024-03-18T19:03:18.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.969+0000 7f8928d8f700 1 -- start start 2024-03-18T19:03:18.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.969+0000 7f8928d8f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f890c003a00 0x7f8924155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:18.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.969+0000 7f8928d8f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89240c7eb0 0x7f89240ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:18.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.969+0000 7f8928d8f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8924155300 0x7f8924155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:18.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.969+0000 7f8928d8f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8924155cd0 con 0x7f8924155300 2024-03-18T19:03:18.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.969+0000 7f8928d8f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89240c91a0 con 0x7f890c003a00 2024-03-18T19:03:18.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.969+0000 7f8928d8f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89240c8330 con 0x7f89240c7eb0 2024-03-18T19:03:18.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.969+0000 7f8922d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8924155300 0x7f8924155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:18.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.969+0000 7f8922d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8924155300 0x7f8924155790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46490/0 (socket says 172.21.15.1:46490) 2024-03-18T19:03:18.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.969+0000 7f8922d9d700 1 -- 172.21.15.1:0/3320769039 learned_addr learned my addr 172.21.15.1:0/3320769039 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:18.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.969+0000 7f8921d9b700 1 --2- 172.21.15.1:0/3320769039 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89240c7eb0 0x7f89240ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:18.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.970+0000 7f892259c700 1 --2- 172.21.15.1:0/3320769039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f890c003a00 0x7f8924155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:18.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.970+0000 7f8922d9d700 1 -- 172.21.15.1:0/3320769039 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89240c7eb0 msgr2=0x7f89240ae1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:18.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.970+0000 7f8922d9d700 1 --2- 172.21.15.1:0/3320769039 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89240c7eb0 0x7f89240ae1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.970+0000 7f8922d9d700 1 -- 172.21.15.1:0/3320769039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f890c003a00 msgr2=0x7f8924155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.970+0000 7f8922d9d700 1 --2- 172.21.15.1:0/3320769039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f890c003a00 0x7f8924155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.970+0000 7f8922d9d700 1 -- 172.21.15.1:0/3320769039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89240c8590 con 0x7f8924155300 2024-03-18T19:03:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.970+0000 7f8922d9d700 1 --2- 172.21.15.1:0/3320769039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8924155300 0x7f8924155790 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f8924074920 tx=0x7f891c00a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.970+0000 7f8912ffd700 1 -- 172.21.15.1:0/3320769039 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c0019e0 con 0x7f8924155300 2024-03-18T19:03:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.970+0000 7f8912ffd700 1 -- 172.21.15.1:0/3320769039 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c00ad50 con 0x7f8924155300 2024-03-18T19:03:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.970+0000 7f8928d8f700 1 -- 172.21.15.1:0/3320769039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89240c8870 con 0x7f8924155300 2024-03-18T19:03:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.970+0000 7f8912ffd700 1 -- 172.21.15.1:0/3320769039 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c013630 con 0x7f8924155300 2024-03-18T19:03:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.970+0000 7f8928d8f700 1 -- 172.21.15.1:0/3320769039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89240c8e30 con 0x7f8924155300 2024-03-18T19:03:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.971+0000 7f8928d8f700 1 -- 172.21.15.1:0/3320769039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8924073940 con 0x7f8924155300 2024-03-18T19:03:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.972+0000 7f8912ffd700 1 -- 172.21.15.1:0/3320769039 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 21) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f891c013790 con 0x7f8924155300 2024-03-18T19:03:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.972+0000 7f8912ffd700 1 --2- 172.21.15.1:0/3320769039 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f890406ab30 0x7f890406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.972+0000 7f8912ffd700 1 -- 172.21.15.1:0/3320769039 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f891c089a00 con 0x7f8924155300 2024-03-18T19:03:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.972+0000 7f892259c700 1 --2- 172.21.15.1:0/3320769039 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f890406ab30 0x7f890406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:18.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.973+0000 7f892259c700 1 --2- 172.21.15.1:0/3320769039 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f890406ab30 0x7f890406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f89180090b0 tx=0x7f891801d040 comp rx=0 tx=0).ready entity=mgr.6459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:18.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:18.977+0000 7f8912ffd700 1 -- 172.21.15.1:0/3320769039 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f891c054c80 con 0x7f8924155300 2024-03-18T19:03:19.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.204+0000 7f8928d8f700 1 -- 172.21.15.1:0/3320769039 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "2m", "target": ["mon-mgr", ""]}) v1 -- 0x7f89240ca890 con 0x7f890406ab30 2024-03-18T19:03:19.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.254+0000 7f8912ffd700 1 -- 172.21.15.1:0/3320769039 <== mgr.6459 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8918021040 con 0x7f890406ab30 2024-03-18T19:03:19.250 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path snap_test_dir1 2024-03-18T19:03:19.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.256+0000 7f8928d8f700 1 -- 172.21.15.1:0/3320769039 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f890406ab30 msgr2=0x7f890406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:19.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.256+0000 7f8928d8f700 1 --2- 172.21.15.1:0/3320769039 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f890406ab30 0x7f890406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f89180090b0 tx=0x7f891801d040 comp rx=0 tx=0).stop 2024-03-18T19:03:19.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.256+0000 7f8928d8f700 1 -- 172.21.15.1:0/3320769039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8924155300 msgr2=0x7f8924155790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:19.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.256+0000 7f8928d8f700 1 --2- 172.21.15.1:0/3320769039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8924155300 0x7f8924155790 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f8924074920 tx=0x7f891c00a640 comp rx=0 tx=0).stop 2024-03-18T19:03:19.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.256+0000 7f8928d8f700 1 -- 172.21.15.1:0/3320769039 shutdown_connections 2024-03-18T19:03:19.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.256+0000 7f8928d8f700 1 --2- 172.21.15.1:0/3320769039 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f890406ab30 0x7f890406cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:19.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.256+0000 7f8928d8f700 1 --2- 172.21.15.1:0/3320769039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8924155300 0x7f8924155790 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:19.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.256+0000 7f8928d8f700 1 --2- 172.21.15.1:0/3320769039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f890c003a00 0x7f8924155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:19.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.256+0000 7f8928d8f700 1 --2- 172.21.15.1:0/3320769039 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89240c7eb0 0x7f89240ae1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:19.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.256+0000 7f8928d8f700 1 -- 172.21.15.1:0/3320769039 >> 172.21.15.1:0/3320769039 conn(0x7f89240cca00 msgr2=0x7f89241573f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:19.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.256+0000 7f8928d8f700 1 -- 172.21.15.1:0/3320769039 shutdown_connections 2024-03-18T19:03:19.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.256+0000 7f8928d8f700 1 -- 172.21.15.1:0/3320769039 wait complete. 2024-03-18T19:03:19.264 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-03-18T19:03:19.264 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-03-18-19_04_00 in ~41s... 2024-03-18T19:03:19.264 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-03-18-19_05_00 in ~101s... 2024-03-18T19:03:19.264 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m', '2m'], expected_retention: [] 2024-03-18T19:03:19.265 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-03-18T19:03:19.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.396+0000 7fcda2f32700 1 Processor -- start 2024-03-18T19:03:19.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.396+0000 7fcda2f32700 1 -- start start 2024-03-18T19:03:19.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.397+0000 7fcda2f32700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcd9c0b0500 0x7fcd9c0cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:19.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.397+0000 7fcda2f32700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd9c0b0e20 0x7fcd9c0b12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.397+0000 7fcda2f32700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd9c15ef80 con 0x7fcd9c15eba0 2024-03-18T19:03:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.397+0000 7fcda2f32700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd9c0cc9a0 con 0x7fcd9c0b0500 2024-03-18T19:03:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.397+0000 7fcda2f32700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0ccc80 con 0x7fcd9c0b0e20 2024-03-18T19:03:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.397+0000 7fcd9bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcd9c0b0500 0x7fcd9c0cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.397+0000 7fcd9bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcd9c0b0500 0x7fcd9c0cfa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46494/0 (socket says 172.21.15.1:46494) 2024-03-18T19:03:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.397+0000 7fcd9bfff700 1 -- 172.21.15.1:0/13193478 learned_addr learned my addr 172.21.15.1:0/13193478 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.397+0000 7fcda14cf700 1 --2- 172.21.15.1:0/13193478 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd9c0b0e20 0x7fcd9c0b12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.397+0000 7fcd9bfff700 1 -- 172.21.15.1:0/13193478 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd9c0b0e20 msgr2=0x7fcd9c0b12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.397+0000 7fcd9bfff700 1 --2- 172.21.15.1:0/13193478 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd9c0b0e20 0x7fcd9c0b12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.397+0000 7fcd9bfff700 1 -- 172.21.15.1:0/13193478 >> v1:172.21.15.165:6789/0 conn(0x7fcd9c15eba0 legacy=0x7fcd9c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:03:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.397+0000 7fcd9bfff700 1 -- 172.21.15.1:0/13193478 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c0ccf60 con 0x7fcd9c0b0500 2024-03-18T19:03:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.397+0000 7fcda14cf700 1 --2- 172.21.15.1:0/13193478 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd9c0b0e20 0x7fcd9c0b12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:03:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.397+0000 7fcd9bfff700 1 --2- 172.21.15.1:0/13193478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c0b0500 0x7fcd9c0cfa80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fcd880098f0 tx=0x7fcd88009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ceb1e028c8651d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:19.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.398+0000 7fcd9b7fe700 1 -- 172.21.15.1:0/13193478 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd88004230 con 0x7fcd9c0b0500 2024-03-18T19:03:19.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.398+0000 7fcd9b7fe700 1 -- 172.21.15.1:0/13193478 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd88004390 con 0x7fcd9c0b0500 2024-03-18T19:03:19.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.398+0000 7fcd9b7fe700 1 -- 172.21.15.1:0/13193478 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd880036a0 con 0x7fcd9c0b0500 2024-03-18T19:03:19.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.398+0000 7fcda2f32700 1 -- 172.21.15.1:0/13193478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c0b0500 msgr2=0x7fcd9c0cfa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:19.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.398+0000 7fcda2f32700 1 --2- 172.21.15.1:0/13193478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c0b0500 0x7fcd9c0cfa80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fcd880098f0 tx=0x7fcd88009c00 comp rx=0 tx=0).stop 2024-03-18T19:03:19.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.398+0000 7fcda2f32700 1 -- 172.21.15.1:0/13193478 shutdown_connections 2024-03-18T19:03:19.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.398+0000 7fcda2f32700 1 --2- 172.21.15.1:0/13193478 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd9c0b0e20 0x7fcd9c0b12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:19.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.398+0000 7fcda2f32700 1 --2- 172.21.15.1:0/13193478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c0b0500 0x7fcd9c0cfa80 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:19.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.398+0000 7fcda2f32700 1 -- 172.21.15.1:0/13193478 >> 172.21.15.1:0/13193478 conn(0x7fcd9c0b2bd0 msgr2=0x7fcd9c0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:19.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.398+0000 7fcda2f32700 1 -- 172.21.15.1:0/13193478 shutdown_connections 2024-03-18T19:03:19.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.398+0000 7fcda2f32700 1 -- 172.21.15.1:0/13193478 wait complete. 2024-03-18T19:03:19.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.399+0000 7fcda2f32700 1 Processor -- start 2024-03-18T19:03:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.399+0000 7fcda2f32700 1 -- start start 2024-03-18T19:03:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.400+0000 7fcda2f32700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd9c0b0500 0x7fcd9c0c3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.400+0000 7fcda2f32700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd9c0b0e20 0x7fcd9c159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.400+0000 7fcda2f32700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c15eba0 0x7fcd9c159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.400+0000 7fcda2f32700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0c4210 con 0x7fcd9c15eba0 2024-03-18T19:03:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.400+0000 7fcda2f32700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c06d180 con 0x7fcd9c0b0e20 2024-03-18T19:03:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.400+0000 7fcda2f32700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c06c570 con 0x7fcd9c0b0500 2024-03-18T19:03:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.400+0000 7fcda14cf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c15eba0 0x7fcd9c159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.400+0000 7fcda14cf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c15eba0 0x7fcd9c159ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46504/0 (socket says 172.21.15.1:46504) 2024-03-18T19:03:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.400+0000 7fcda14cf700 1 -- 172.21.15.1:0/895579017 learned_addr learned my addr 172.21.15.1:0/895579017 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:03:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.400+0000 7fcda0cce700 1 --2- 172.21.15.1:0/895579017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd9c0b0500 0x7fcd9c0c3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.401+0000 7fcd9bfff700 1 --2- 172.21.15.1:0/895579017 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd9c0b0e20 0x7fcd9c159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:19.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.401+0000 7fcda14cf700 1 -- 172.21.15.1:0/895579017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd9c0b0500 msgr2=0x7fcd9c0c3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.401+0000 7fcda14cf700 1 --2- 172.21.15.1:0/895579017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd9c0b0500 0x7fcd9c0c3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.401+0000 7fcda14cf700 1 -- 172.21.15.1:0/895579017 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd9c0b0e20 msgr2=0x7fcd9c159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.401+0000 7fcda14cf700 1 --2- 172.21.15.1:0/895579017 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd9c0b0e20 0x7fcd9c159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.401+0000 7fcda14cf700 1 -- 172.21.15.1:0/895579017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd88009550 con 0x7fcd9c15eba0 2024-03-18T19:03:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.401+0000 7fcda14cf700 1 --2- 172.21.15.1:0/895579017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c15eba0 0x7fcd9c159ff0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fcd9400cef0 tx=0x7fcd9400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.401+0000 7fcd997fa700 1 -- 172.21.15.1:0/895579017 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9400ad00 con 0x7fcd9c15eba0 2024-03-18T19:03:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.401+0000 7fcd997fa700 1 -- 172.21.15.1:0/895579017 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9400ae60 con 0x7fcd9c15eba0 2024-03-18T19:03:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.401+0000 7fcda2f32700 1 -- 172.21.15.1:0/895579017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9c06c850 con 0x7fcd9c15eba0 2024-03-18T19:03:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.401+0000 7fcd997fa700 1 -- 172.21.15.1:0/895579017 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94015680 con 0x7fcd9c15eba0 2024-03-18T19:03:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.401+0000 7fcda2f32700 1 -- 172.21.15.1:0/895579017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9c06ce10 con 0x7fcd9c15eba0 2024-03-18T19:03:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.402+0000 7fcda2f32700 1 -- 172.21.15.1:0/895579017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9c06d2f0 con 0x7fcd9c15eba0 2024-03-18T19:03:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.402+0000 7fcd997fa700 1 -- 172.21.15.1:0/895579017 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 21) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fcd9400d430 con 0x7fcd9c15eba0 2024-03-18T19:03:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.403+0000 7fcd997fa700 1 --2- 172.21.15.1:0/895579017 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcd8006ab30 0x7fcd8006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:03:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.403+0000 7fcd997fa700 1 -- 172.21.15.1:0/895579017 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fcd94089700 con 0x7fcd9c15eba0 2024-03-18T19:03:19.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.403+0000 7fcda0cce700 1 --2- 172.21.15.1:0/895579017 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcd8006ab30 0x7fcd8006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:03:19.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.403+0000 7fcda0cce700 1 --2- 172.21.15.1:0/895579017 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcd8006ab30 0x7fcd8006cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009960 tx=0x7fcd900090e0 comp rx=0 tx=0).ready entity=mgr.6459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:03:19.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.408+0000 7fcd997fa700 1 -- 172.21.15.1:0/895579017 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcd94054980 con 0x7fcd9c15eba0 2024-03-18T19:03:19.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.634+0000 7fcda2f32700 1 -- 172.21.15.1:0/895579017 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcd9c14ec40 con 0x7fcd8006ab30 2024-03-18T19:03:19.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.644+0000 7fcd997fa700 1 -- 172.21.15.1:0/895579017 <== mgr.6459 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+76 (secure 0 0 0) 0x7fcd9c14ec40 con 0x7fcd8006ab30 2024-03-18T19:03:19.640 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:03:19.640 INFO:teuthology.orchestra.run.smithi001.stdout:{"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2024-03-18T19:03:19.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.646+0000 7fcda2f32700 1 -- 172.21.15.1:0/895579017 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcd8006ab30 msgr2=0x7fcd8006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:19.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.646+0000 7fcda2f32700 1 --2- 172.21.15.1:0/895579017 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcd8006ab30 0x7fcd8006cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009960 tx=0x7fcd900090e0 comp rx=0 tx=0).stop 2024-03-18T19:03:19.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.646+0000 7fcda2f32700 1 -- 172.21.15.1:0/895579017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c15eba0 msgr2=0x7fcd9c159ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:03:19.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.646+0000 7fcda2f32700 1 --2- 172.21.15.1:0/895579017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c15eba0 0x7fcd9c159ff0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fcd9400cef0 tx=0x7fcd9400cf20 comp rx=0 tx=0).stop 2024-03-18T19:03:19.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.646+0000 7fcda2f32700 1 -- 172.21.15.1:0/895579017 shutdown_connections 2024-03-18T19:03:19.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.646+0000 7fcda2f32700 1 --2- 172.21.15.1:0/895579017 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcd8006ab30 0x7fcd8006cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:19.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.646+0000 7fcda2f32700 1 --2- 172.21.15.1:0/895579017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd9c0b0500 0x7fcd9c0c3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:19.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.646+0000 7fcda2f32700 1 --2- 172.21.15.1:0/895579017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c15eba0 0x7fcd9c159ff0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:19.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.646+0000 7fcda2f32700 1 --2- 172.21.15.1:0/895579017 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd9c0b0e20 0x7fcd9c159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:03:19.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.646+0000 7fcda2f32700 1 -- 172.21.15.1:0/895579017 >> 172.21.15.1:0/895579017 conn(0x7fcd9c0b2bd0 msgr2=0x7fcd9c0cbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:03:19.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.646+0000 7fcda2f32700 1 -- 172.21.15.1:0/895579017 shutdown_connections 2024-03-18T19:03:19.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:03:19.646+0000 7fcda2f32700 1 -- 172.21.15.1:0/895579017 wait complete. 2024-03-18T19:03:19.654 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2024-03-18T19:03:19.654 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m', '2m'], 'retention': [{}, {}]} 2024-03-18T19:03:19.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:19.654 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:19.676 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:19.677 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:19.677 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:20.553 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:03:20.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:03:20.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:20.678 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:20.704 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:20.704 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:20.704 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:21.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:21.705 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:21.733 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:21.733 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:21.733 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:22.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:22.734 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:22.763 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:22.763 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:22.763 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:23.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:23.764 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:23.791 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:23.792 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:23.792 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:24.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:24.793 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:24.821 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:24.821 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:24.821 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:25.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:25.822 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:25.850 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:25.850 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:25.850 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:26.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:26.851 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:26.879 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:26.879 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:26.879 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:27.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:27.880 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:27.907 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:27.907 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:27.907 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:28.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:28.909 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:28.937 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:28.937 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:28.937 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:29.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:29.938 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:29.966 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:29.966 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:29.966 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:30.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:30.967 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:30.994 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:30.995 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:30.995 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:31.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:31.996 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:32.024 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:32.024 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:32.024 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:33.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:33.026 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:33.053 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:33.053 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:33.053 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:34.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:34.055 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:34.082 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:34.082 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:34.082 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:35.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:35.084 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:35.111 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:35.111 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:35.111 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:36.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:36.112 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:36.140 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:36.140 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:36.140 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:37.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:37.141 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:37.168 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:37.168 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:37.168 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:38.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:38.170 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:38.197 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:38.197 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:38.197 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:39.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:39.198 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:39.226 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:39.226 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:39.226 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:40.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:40.227 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:40.254 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:40.254 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:40.255 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:41.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:41.256 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:41.283 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:41.283 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:41.283 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:42.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:42.284 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:42.313 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:42.313 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:42.313 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:43.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:43.314 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:43.342 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:43.342 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:43.342 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:44.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:44.344 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:44.371 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:44.371 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:44.371 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:45.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:45.372 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:45.400 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:45.400 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:45.400 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:46.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:46.401 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:46.428 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:46.428 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:46.429 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:47.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:47.430 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:47.457 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:47.457 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:47.458 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:48.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:48.459 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:48.486 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:48.487 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:48.487 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:49.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:49.488 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:49.515 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:49.515 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:49.515 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:50.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:50.516 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:50.544 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:50.544 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:50.544 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:50.692 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:03:50.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:03:51.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:51.546 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:51.574 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:51.574 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:51.575 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:52.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:52.576 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:52.603 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:52.604 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:52.604 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:53.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:53.605 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:53.632 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:53.633 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:53.633 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:54.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:54.633 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:54.661 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:54.661 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:54.661 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:55.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:55.663 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:55.690 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:55.690 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:55.690 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:56.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:56.692 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:56.719 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:56.720 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:56.720 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:57.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:57.721 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:57.748 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:57.748 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:57.749 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:58.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:58.750 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:58.778 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:58.778 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:58.778 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:03:59.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:03:59.780 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:03:59.807 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:03:59.808 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:03:59.808 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:00.260 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:04:00.264+0000 7fd2a3ea7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:04:00.260 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:04:00.264+0000 7fd2a3ea7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:04:00.260 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:04:00.264+0000 7fd2a3ea7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:04:00.260 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:04:00.264+0000 7fd2a3ea7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:04:00.260 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:04:00.264+0000 7fd2a3ea7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:04:00.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:00.809 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:00.835 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:00.836 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:00.836 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:00.837 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:00.837 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-03-18-19_04_00_UTC'] 2024-03-18T19:04:00.837 DEBUG:tasks.cephfs.test_snap_schedules:exec=1710788599.2646003, now = 1710788640.837222, timo = 41 2024-03-18T19:04:01.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:01.838 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:01.866 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:01.866 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:01.866 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:01.866 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:02.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:02.868 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:02.895 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:02.895 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:02.895 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:02.895 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:03.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:03.896 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:03.923 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:03.924 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:03.924 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:03.925 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:04.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:04.926 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:04.953 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:04.953 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:04.954 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:04.954 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:05.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:05.955 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:05.981 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:05.982 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:05.982 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:05.983 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:06.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:06.983 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:07.012 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:07.012 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:07.012 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:07.012 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:08.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:08.013 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:08.041 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:08.041 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:08.041 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:08.041 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:09.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:09.043 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:09.070 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:09.070 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:09.070 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:09.070 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:10.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:10.071 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:10.099 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:10.100 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:10.100 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:10.100 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:11.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:11.101 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:11.128 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:11.128 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:11.128 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:11.128 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:12.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:12.130 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:12.157 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:12.157 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:12.157 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:12.157 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:13.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:13.158 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:13.185 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:13.186 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:13.186 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:13.186 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:14.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:14.187 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:14.215 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:14.215 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:14.215 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:14.215 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:15.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:15.217 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:15.244 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:15.244 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:15.245 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:15.245 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:16.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:16.246 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:16.273 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:16.274 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:16.274 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:16.274 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:17.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:17.275 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:17.303 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:17.303 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:17.303 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:17.304 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:18.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:18.305 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:18.331 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:18.332 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:18.332 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:18.332 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:19.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:19.333 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:19.361 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:19.361 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:19.361 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:19.361 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:20.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:20.362 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:20.390 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:20.390 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:20.390 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:20.390 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:20.778 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:04:20.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:04:21.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:21.391 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:21.418 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:21.418 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:21.418 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:21.418 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:22.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:22.419 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:22.447 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:22.447 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:22.447 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:22.448 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:23.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:23.449 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:23.476 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:23.476 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:23.476 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:23.476 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:24.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:24.478 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:24.505 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:24.505 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:24.505 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:24.506 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:25.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:25.507 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:25.534 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:25.535 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:25.535 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:25.535 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:26.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:26.536 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:26.564 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:26.564 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:26.564 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:26.564 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:27.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:27.566 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:27.593 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:27.593 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:27.593 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:27.593 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:28.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:28.595 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:28.622 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:28.623 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:28.623 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:28.623 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:29.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:29.624 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:29.651 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:29.651 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:29.651 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:29.651 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:30.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:30.652 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:30.680 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:30.680 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:30.680 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:30.680 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:31.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:31.681 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:31.707 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:31.708 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:31.708 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:31.708 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:32.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:32.709 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:32.737 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:32.737 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:32.738 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:32.738 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:33.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:33.739 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:33.768 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:33.768 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:33.768 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:33.769 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:34.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:34.770 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:34.798 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:34.799 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:34.799 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:34.799 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:35.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:35.800 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:35.828 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:35.829 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:35.829 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:35.829 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:36.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:36.830 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:36.858 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:36.858 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:36.858 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:36.859 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:37.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:37.859 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:37.887 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:37.887 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:37.887 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:37.887 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:38.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:38.889 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:38.917 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:38.917 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:38.917 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:38.917 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:39.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:39.919 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:39.945 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:39.947 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:39.947 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:39.947 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:40.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:40.948 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:40.978 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:40.978 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:40.978 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:40.978 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:41.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:41.979 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:42.006 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:42.007 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:42.007 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:42.007 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:43.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:43.008 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:43.034 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:43.035 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:43.035 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:43.035 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:44.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:44.036 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:44.063 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:44.064 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:44.064 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:44.064 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:45.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:45.066 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:45.092 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:45.093 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:45.093 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:45.093 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:46.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:46.094 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:46.121 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:46.122 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:46.123 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:46.123 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:47.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:47.124 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:47.150 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:47.151 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:47.151 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:47.151 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:48.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:48.153 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:48.181 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:48.182 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:48.182 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:48.182 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:49.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:49.182 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:49.208 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:49.208 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:49.209 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:49.209 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:50.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:50.210 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:50.238 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:50.238 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:50.238 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:50.238 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:50.925 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:04:50.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:04:51.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:51.240 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:51.268 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:51.268 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:51.268 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:51.269 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:52.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:52.269 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:52.297 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:52.298 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:52.298 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:52.298 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:53.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:53.299 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:53.326 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:53.327 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:53.327 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:53.327 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:54.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:54.328 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:54.356 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:54.357 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:54.357 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:54.357 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:55.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:55.358 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:55.386 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:55.386 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:55.386 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:55.386 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:56.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:56.388 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:56.415 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:56.415 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:56.415 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:56.415 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:57.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:57.416 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:57.444 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:57.445 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:57.445 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:57.445 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:58.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:58.446 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:58.474 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:58.475 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:58.475 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:58.475 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:04:59.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:04:59.477 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:04:59.505 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:04:59.505 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC'} 2024-03-18T19:04:59.505 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:04:59.505 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:05:00.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:00.507 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:05:00.532 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:05:00.532 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_05_00_UTC 2024-03-18T19:05:00.533 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_04_00_UTC', 'scheduled-2024-03-18-19_05_00_UTC'} 2024-03-18T19:05:00.533 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-03-18-19_05_00_UTC'} 2024-03-18T19:05:00.534 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:05:00.534 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-03-18-19_05_00_UTC'] 2024-03-18T19:05:00.534 DEBUG:tasks.cephfs.test_snap_schedules:exec=1710788599.2646003, now = 1710788700.534209, timo = 101 2024-03-18T19:05:01.535 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-03-18T19:05:01.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.681+0000 7fcf4e0ae700 1 Processor -- start 2024-03-18T19:05:01.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.681+0000 7fcf4e0ae700 1 -- start start 2024-03-18T19:05:01.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.681+0000 7fcf4e0ae700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcf4815a240 0x7fcf480ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:01.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.681+0000 7fcf4e0ae700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcf48071cc0 0x7fcf4806fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:01.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.681+0000 7fcf4e0ae700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf48070000 con 0x7fcf4806d560 2024-03-18T19:05:01.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.681+0000 7fcf4e0ae700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf480adff0 con 0x7fcf48071cc0 2024-03-18T19:05:01.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.681+0000 7fcf4e0ae700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf480cac80 con 0x7fcf4815a240 2024-03-18T19:05:01.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.682+0000 7fcf47fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcf48071cc0 0x7fcf4806fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:01.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.682+0000 7fcf47fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcf48071cc0 0x7fcf4806fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42638/0 (socket says 172.21.15.1:42638) 2024-03-18T19:05:01.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.682+0000 7fcf47fff700 1 -- 172.21.15.1:0/1509872971 learned_addr learned my addr 172.21.15.1:0/1509872971 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.682+0000 7fcf477fe700 1 --2- 172.21.15.1:0/1509872971 >> v2:172.21.15.165:3300/0 conn(0x7fcf4815a240 0x7fcf480ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.682+0000 7fcf47fff700 1 -- 172.21.15.1:0/1509872971 >> v2:172.21.15.165:3300/0 conn(0x7fcf4815a240 msgr2=0x7fcf480ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.682+0000 7fcf47fff700 1 --2- 172.21.15.1:0/1509872971 >> v2:172.21.15.165:3300/0 conn(0x7fcf4815a240 0x7fcf480ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.682+0000 7fcf47fff700 1 -- 172.21.15.1:0/1509872971 >> v1:172.21.15.1:6789/0 conn(0x7fcf4806d560 legacy=0x7fcf480715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.682+0000 7fcf47fff700 1 -- 172.21.15.1:0/1509872971 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf480caf60 con 0x7fcf48071cc0 2024-03-18T19:05:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.682+0000 7fcf47fff700 1 --2- 172.21.15.1:0/1509872971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf48071cc0 0x7fcf4806fac0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fcf40009e50 tx=0x7fcf4000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac9183705888565 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.682+0000 7fcf467fc700 1 -- 172.21.15.1:0/1509872971 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4000e070 con 0x7fcf48071cc0 2024-03-18T19:05:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.682+0000 7fcf467fc700 1 -- 172.21.15.1:0/1509872971 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf4000b980 con 0x7fcf48071cc0 2024-03-18T19:05:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.682+0000 7fcf467fc700 1 -- 172.21.15.1:0/1509872971 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4000bd30 con 0x7fcf48071cc0 2024-03-18T19:05:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.683+0000 7fcf4e0ae700 1 -- 172.21.15.1:0/1509872971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf48071cc0 msgr2=0x7fcf4806fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.683+0000 7fcf4e0ae700 1 --2- 172.21.15.1:0/1509872971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf48071cc0 0x7fcf4806fac0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fcf40009e50 tx=0x7fcf4000b330 comp rx=0 tx=0).stop 2024-03-18T19:05:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.683+0000 7fcf4e0ae700 1 -- 172.21.15.1:0/1509872971 shutdown_connections 2024-03-18T19:05:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.683+0000 7fcf4e0ae700 1 --2- 172.21.15.1:0/1509872971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf48071cc0 0x7fcf4806fac0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.683+0000 7fcf4e0ae700 1 --2- 172.21.15.1:0/1509872971 >> v2:172.21.15.165:3300/0 conn(0x7fcf4815a240 0x7fcf480ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.683+0000 7fcf4e0ae700 1 -- 172.21.15.1:0/1509872971 >> 172.21.15.1:0/1509872971 conn(0x7fcf480cff30 msgr2=0x7fcf480d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.683+0000 7fcf4e0ae700 1 -- 172.21.15.1:0/1509872971 shutdown_connections 2024-03-18T19:05:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.683+0000 7fcf4e0ae700 1 -- 172.21.15.1:0/1509872971 wait complete. 2024-03-18T19:05:01.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.684+0000 7fcf4e0ae700 1 Processor -- start 2024-03-18T19:05:01.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.684+0000 7fcf4e0ae700 1 -- start start 2024-03-18T19:05:01.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.685+0000 7fcf4e0ae700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf4806d560 0x7fcf48158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:01.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.685+0000 7fcf4e0ae700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf48071cc0 0x7fcf480b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:01.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.685+0000 7fcf4e0ae700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf4815a240 0x7fcf480c4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:01.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.685+0000 7fcf4e0ae700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf480b2920 con 0x7fcf4806d560 2024-03-18T19:05:01.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.685+0000 7fcf4e0ae700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf480c8790 con 0x7fcf48071cc0 2024-03-18T19:05:01.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.685+0000 7fcf4e0ae700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf480c8a30 con 0x7fcf4815a240 2024-03-18T19:05:01.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.685+0000 7fcf477fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf4806d560 0x7fcf48158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:01.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.685+0000 7fcf477fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf4806d560 0x7fcf48158a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42642/0 (socket says 172.21.15.1:42642) 2024-03-18T19:05:01.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.685+0000 7fcf477fe700 1 -- 172.21.15.1:0/461314753 learned_addr learned my addr 172.21.15.1:0/461314753 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:01.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.685+0000 7fcf47fff700 1 --2- 172.21.15.1:0/461314753 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf4815a240 0x7fcf480c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:01.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.685+0000 7fcf46ffd700 1 --2- 172.21.15.1:0/461314753 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf48071cc0 0x7fcf480b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:01.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.686+0000 7fcf477fe700 1 -- 172.21.15.1:0/461314753 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf4815a240 msgr2=0x7fcf480c4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:01.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.686+0000 7fcf477fe700 1 --2- 172.21.15.1:0/461314753 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf4815a240 0x7fcf480c4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:01.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.686+0000 7fcf477fe700 1 -- 172.21.15.1:0/461314753 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf48071cc0 msgr2=0x7fcf480b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:01.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.686+0000 7fcf477fe700 1 --2- 172.21.15.1:0/461314753 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf48071cc0 0x7fcf480b2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:01.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.686+0000 7fcf477fe700 1 -- 172.21.15.1:0/461314753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf40009b30 con 0x7fcf4806d560 2024-03-18T19:05:01.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.686+0000 7fcf46ffd700 1 --2- 172.21.15.1:0/461314753 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf48071cc0 0x7fcf480b2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:01.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.686+0000 7fcf477fe700 1 --2- 172.21.15.1:0/461314753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf4806d560 0x7fcf48158a10 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fcf3800d800 tx=0x7fcf3800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:01.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.686+0000 7fcf2ffff700 1 -- 172.21.15.1:0/461314753 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf380125f0 con 0x7fcf4806d560 2024-03-18T19:05:01.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.686+0000 7fcf2ffff700 1 -- 172.21.15.1:0/461314753 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf38012750 con 0x7fcf4806d560 2024-03-18T19:05:01.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.686+0000 7fcf4e0ae700 1 -- 172.21.15.1:0/461314753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf480c8d10 con 0x7fcf4806d560 2024-03-18T19:05:01.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.686+0000 7fcf2ffff700 1 -- 172.21.15.1:0/461314753 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf38012b00 con 0x7fcf4806d560 2024-03-18T19:05:01.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.686+0000 7fcf4e0ae700 1 -- 172.21.15.1:0/461314753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf480b25c0 con 0x7fcf4806d560 2024-03-18T19:05:01.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.687+0000 7fcf4e0ae700 1 -- 172.21.15.1:0/461314753 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf480c9230 con 0x7fcf4806d560 2024-03-18T19:05:01.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.687+0000 7fcf2ffff700 1 -- 172.21.15.1:0/461314753 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 25) v1 ==== 89268+0+0 (secure 0 0 0) 0x7fcf38012c60 con 0x7fcf4806d560 2024-03-18T19:05:01.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.688+0000 7fcf2ffff700 1 --2- 172.21.15.1:0/461314753 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcf3406ae00 0x7fcf3406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:01.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.688+0000 7fcf46ffd700 1 --2- 172.21.15.1:0/461314753 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcf3406ae00 0x7fcf3406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:01.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.688+0000 7fcf2ffff700 1 -- 172.21.15.1:0/461314753 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fcf38089a90 con 0x7fcf4806d560 2024-03-18T19:05:01.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.689+0000 7fcf46ffd700 1 --2- 172.21.15.1:0/461314753 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcf3406ae00 0x7fcf3406d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcf480c6b00 tx=0x7fcf3c009040 comp rx=0 tx=0).ready entity=mgr.6459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:01.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.693+0000 7fcf2ffff700 1 -- 172.21.15.1:0/461314753 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcf38054890 con 0x7fcf4806d560 2024-03-18T19:05:01.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.921+0000 7fcf4e0ae700 1 -- 172.21.15.1:0/461314753 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf4815a620 con 0x7fcf3406ae00 2024-03-18T19:05:01.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.981+0000 7fcf2ffff700 1 -- 172.21.15.1:0/461314753 <== mgr.6459 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fcf3c0051a0 con 0x7fcf3406ae00 2024-03-18T19:05:01.977 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule removed for path snap_test_dir1 2024-03-18T19:05:01.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.983+0000 7fcf4e0ae700 1 -- 172.21.15.1:0/461314753 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcf3406ae00 msgr2=0x7fcf3406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:01.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.983+0000 7fcf4e0ae700 1 --2- 172.21.15.1:0/461314753 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcf3406ae00 0x7fcf3406d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcf480c6b00 tx=0x7fcf3c009040 comp rx=0 tx=0).stop 2024-03-18T19:05:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.983+0000 7fcf4e0ae700 1 -- 172.21.15.1:0/461314753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf4806d560 msgr2=0x7fcf48158a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.983+0000 7fcf4e0ae700 1 --2- 172.21.15.1:0/461314753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf4806d560 0x7fcf48158a10 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fcf3800d800 tx=0x7fcf3800db10 comp rx=0 tx=0).stop 2024-03-18T19:05:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.983+0000 7fcf4e0ae700 1 -- 172.21.15.1:0/461314753 shutdown_connections 2024-03-18T19:05:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.983+0000 7fcf4e0ae700 1 --2- 172.21.15.1:0/461314753 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcf3406ae00 0x7fcf3406d2c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.983+0000 7fcf4e0ae700 1 --2- 172.21.15.1:0/461314753 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf4815a240 0x7fcf480c4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.983+0000 7fcf4e0ae700 1 --2- 172.21.15.1:0/461314753 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf4806d560 0x7fcf48158a10 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.983+0000 7fcf4e0ae700 1 --2- 172.21.15.1:0/461314753 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf48071cc0 0x7fcf480b2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.983+0000 7fcf4e0ae700 1 -- 172.21.15.1:0/461314753 >> 172.21.15.1:0/461314753 conn(0x7fcf480cff30 msgr2=0x7fcf480d0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.983+0000 7fcf4e0ae700 1 -- 172.21.15.1:0/461314753 shutdown_connections 2024-03-18T19:05:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:01.983+0000 7fcf4e0ae700 1 -- 172.21.15.1:0/461314753 wait complete. 2024-03-18T19:05:01.991 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-03-18T19:05:01.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:01.991 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:05:02.012 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:05:02.012 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_05_00_UTC 2024-03-18T19:05:02.013 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_04_00_UTC 2024-03-18T19:05:02.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:02.013 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_04_00_UTC) 2024-03-18T19:05:02.076 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_05_00_UTC 2024-03-18T19:05:02.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:02.076 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_05_00_UTC) 2024-03-18T19:05:02.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:02.141 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-03-18T19:05:02.203 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-18T19:05:02.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.375+0000 7f4b3c016700 1 Processor -- start 2024-03-18T19:05:02.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.375+0000 7f4b3c016700 1 -- start start 2024-03-18T19:05:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.376+0000 7f4b3c016700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4b3406ca80 0x7f4b3415d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.376+0000 7f4b3c016700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4b3409a150 0x7f4b340cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.376+0000 7f4b3c016700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b340b3b00 con 0x7f4b3415d5b0 2024-03-18T19:05:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.376+0000 7f4b3c016700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b340cbbb0 con 0x7f4b3409a150 2024-03-18T19:05:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.376+0000 7f4b3c016700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b340cf7f0 con 0x7f4b3406ca80 2024-03-18T19:05:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.376+0000 7f4b3a5b3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4b3409a150 0x7f4b340cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.376+0000 7f4b3a5b3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4b3409a150 0x7f4b340cbfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42650/0 (socket says 172.21.15.1:42650) 2024-03-18T19:05:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.376+0000 7f4b3a5b3700 1 -- 172.21.15.1:0/2284092391 learned_addr learned my addr 172.21.15.1:0/2284092391 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.376+0000 7f4b3a5b3700 1 -- 172.21.15.1:0/2284092391 >> v2:172.21.15.165:3300/0 conn(0x7f4b3406ca80 msgr2=0x7f4b3415d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.376+0000 7f4b395b1700 1 --2- 172.21.15.1:0/2284092391 >> v2:172.21.15.165:3300/0 conn(0x7f4b3406ca80 0x7f4b3415d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.376+0000 7f4b3a5b3700 1 --2- 172.21.15.1:0/2284092391 >> v2:172.21.15.165:3300/0 conn(0x7f4b3406ca80 0x7f4b3415d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.376+0000 7f4b3a5b3700 1 -- 172.21.15.1:0/2284092391 >> v1:172.21.15.165:6789/0 conn(0x7f4b3415d5b0 legacy=0x7f4b34099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.376+0000 7f4b3a5b3700 1 -- 172.21.15.1:0/2284092391 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b340d04d0 con 0x7f4b3409a150 2024-03-18T19:05:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.376+0000 7f4b395b1700 1 --2- 172.21.15.1:0/2284092391 >> v2:172.21.15.165:3300/0 conn(0x7f4b3406ca80 0x7f4b3415d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.376+0000 7f4b3a5b3700 1 --2- 172.21.15.1:0/2284092391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b3409a150 0x7f4b340cbfd0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f4b34072c10 tx=0x7f4b2c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253665ee17e2ea44 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.377+0000 7f4b38db0700 1 -- 172.21.15.1:0/2284092391 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c00e070 con 0x7f4b3409a150 2024-03-18T19:05:02.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.377+0000 7f4b38db0700 1 -- 172.21.15.1:0/2284092391 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2c003a20 con 0x7f4b3409a150 2024-03-18T19:05:02.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.377+0000 7f4b38db0700 1 -- 172.21.15.1:0/2284092391 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c003dd0 con 0x7f4b3409a150 2024-03-18T19:05:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.377+0000 7f4b3c016700 1 -- 172.21.15.1:0/2284092391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b3409a150 msgr2=0x7f4b340cbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.377+0000 7f4b3c016700 1 --2- 172.21.15.1:0/2284092391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b3409a150 0x7f4b340cbfd0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f4b34072c10 tx=0x7f4b2c00b330 comp rx=0 tx=0).stop 2024-03-18T19:05:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.377+0000 7f4b3c016700 1 -- 172.21.15.1:0/2284092391 shutdown_connections 2024-03-18T19:05:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.377+0000 7f4b3c016700 1 --2- 172.21.15.1:0/2284092391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b3409a150 0x7f4b340cbfd0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.377+0000 7f4b3c016700 1 --2- 172.21.15.1:0/2284092391 >> v2:172.21.15.165:3300/0 conn(0x7f4b3406ca80 0x7f4b3415d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.377+0000 7f4b3c016700 1 -- 172.21.15.1:0/2284092391 >> 172.21.15.1:0/2284092391 conn(0x7f4b340b2bd0 msgr2=0x7f4b340b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.377+0000 7f4b3c016700 1 -- 172.21.15.1:0/2284092391 shutdown_connections 2024-03-18T19:05:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.377+0000 7f4b3c016700 1 -- 172.21.15.1:0/2284092391 wait complete. 2024-03-18T19:05:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.378+0000 7f4b3c016700 1 Processor -- start 2024-03-18T19:05:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.378+0000 7f4b3c016700 1 -- start start 2024-03-18T19:05:02.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.379+0000 7f4b3c016700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b3406ca80 0x7f4b340b6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:02.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.379+0000 7f4b3c016700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b3409a150 0x7f4b340ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:02.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.379+0000 7f4b3c016700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b3415d5b0 0x7f4b341588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:02.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.379+0000 7f4b3c016700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b34156eb0 con 0x7f4b3415d5b0 2024-03-18T19:05:02.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.379+0000 7f4b3c016700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b340cad10 con 0x7f4b3409a150 2024-03-18T19:05:02.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.379+0000 7f4b3c016700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b34158e10 con 0x7f4b3406ca80 2024-03-18T19:05:02.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.379+0000 7f4b3a5b3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b3415d5b0 0x7f4b341588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:02.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.379+0000 7f4b3a5b3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b3415d5b0 0x7f4b341588d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42658/0 (socket says 172.21.15.1:42658) 2024-03-18T19:05:02.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.379+0000 7f4b3a5b3700 1 -- 172.21.15.1:0/1102879245 learned_addr learned my addr 172.21.15.1:0/1102879245 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:02.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.379+0000 7f4b395b1700 1 --2- 172.21.15.1:0/1102879245 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b3409a150 0x7f4b340ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:02.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.379+0000 7f4b39db2700 1 --2- 172.21.15.1:0/1102879245 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b3406ca80 0x7f4b340b6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:02.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.380+0000 7f4b3a5b3700 1 -- 172.21.15.1:0/1102879245 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b3406ca80 msgr2=0x7f4b340b6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:02.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.380+0000 7f4b3a5b3700 1 --2- 172.21.15.1:0/1102879245 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b3406ca80 0x7f4b340b6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:02.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.380+0000 7f4b3a5b3700 1 -- 172.21.15.1:0/1102879245 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b3409a150 msgr2=0x7f4b340ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:02.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.380+0000 7f4b3a5b3700 1 --2- 172.21.15.1:0/1102879245 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b3409a150 0x7f4b340ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:02.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.380+0000 7f4b3a5b3700 1 -- 172.21.15.1:0/1102879245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2c009b30 con 0x7f4b3415d5b0 2024-03-18T19:05:02.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.380+0000 7f4b3a5b3700 1 --2- 172.21.15.1:0/1102879245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b3415d5b0 0x7f4b341588d0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c015040 tx=0x7f4b2c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:02.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.380+0000 7f4b2a7fc700 1 -- 172.21.15.1:0/1102879245 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c004440 con 0x7f4b3415d5b0 2024-03-18T19:05:02.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.380+0000 7f4b2a7fc700 1 -- 172.21.15.1:0/1102879245 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2c0045a0 con 0x7f4b3415d5b0 2024-03-18T19:05:02.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.380+0000 7f4b3c016700 1 -- 172.21.15.1:0/1102879245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b340caf40 con 0x7f4b3415d5b0 2024-03-18T19:05:02.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.380+0000 7f4b2a7fc700 1 -- 172.21.15.1:0/1102879245 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c004950 con 0x7f4b3415d5b0 2024-03-18T19:05:02.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.380+0000 7f4b3c016700 1 -- 172.21.15.1:0/1102879245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b340b4110 con 0x7f4b3415d5b0 2024-03-18T19:05:02.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.381+0000 7f4b3c016700 1 -- 172.21.15.1:0/1102879245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b340b3c90 con 0x7f4b3415d5b0 2024-03-18T19:05:02.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.381+0000 7f4b2a7fc700 1 -- 172.21.15.1:0/1102879245 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 25) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f4b2c012430 con 0x7f4b3415d5b0 2024-03-18T19:05:02.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.382+0000 7f4b2a7fc700 1 --2- 172.21.15.1:0/1102879245 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4b1806ae00 0x7f4b1806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:02.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.382+0000 7f4b39db2700 1 --2- 172.21.15.1:0/1102879245 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4b1806ae00 0x7f4b1806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:02.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.382+0000 7f4b2a7fc700 1 -- 172.21.15.1:0/1102879245 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f4b2c08ae40 con 0x7f4b3415d5b0 2024-03-18T19:05:02.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.382+0000 7f4b39db2700 1 --2- 172.21.15.1:0/1102879245 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4b1806ae00 0x7f4b1806d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4b340cbd60 tx=0x7f4b30009040 comp rx=0 tx=0).ready entity=mgr.6459 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:02.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.387+0000 7f4b2a7fc700 1 -- 172.21.15.1:0/1102879245 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4b340b3c90 con 0x7f4b3415d5b0 2024-03-18T19:05:02.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:02.654+0000 7f4b3c016700 1 -- 172.21.15.1:0/1102879245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f4b340b3c90 con 0x7f4b3415d5b0 2024-03-18T19:05:03.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.117+0000 7f4b2a7fc700 1 -- 172.21.15.1:0/1102879245 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f4b2c089090 con 0x7f4b3415d5b0 2024-03-18T19:05:03.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.117+0000 7f4b2a7fc700 1 -- 172.21.15.1:0/1102879245 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v26) v1 ==== 93+0+0 (secure 0 0 0) 0x7f4b2c051a30 con 0x7f4b3415d5b0 2024-03-18T19:05:03.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.120+0000 7f4b3c016700 1 -- 172.21.15.1:0/1102879245 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4b1806ae00 msgr2=0x7f4b1806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:03.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.120+0000 7f4b3c016700 1 --2- 172.21.15.1:0/1102879245 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4b1806ae00 0x7f4b1806d2c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4b340cbd60 tx=0x7f4b30009040 comp rx=0 tx=0).stop 2024-03-18T19:05:03.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.120+0000 7f4b3c016700 1 -- 172.21.15.1:0/1102879245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b3415d5b0 msgr2=0x7f4b341588d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:03.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.120+0000 7f4b3c016700 1 --2- 172.21.15.1:0/1102879245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b3415d5b0 0x7f4b341588d0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c015040 tx=0x7f4b2c004ad0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.121+0000 7f4b3c016700 1 -- 172.21.15.1:0/1102879245 shutdown_connections 2024-03-18T19:05:03.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.121+0000 7f4b3c016700 1 --2- 172.21.15.1:0/1102879245 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4b1806ae00 0x7f4b1806d2c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.121+0000 7f4b3c016700 1 --2- 172.21.15.1:0/1102879245 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b3406ca80 0x7f4b340b6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.121+0000 7f4b3c016700 1 --2- 172.21.15.1:0/1102879245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b3415d5b0 0x7f4b341588d0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.121+0000 7f4b3c016700 1 --2- 172.21.15.1:0/1102879245 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b3409a150 0x7f4b340ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.121+0000 7f4b3c016700 1 -- 172.21.15.1:0/1102879245 >> 172.21.15.1:0/1102879245 conn(0x7f4b340b2bd0 msgr2=0x7f4b3406ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:03.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.121+0000 7f4b3c016700 1 -- 172.21.15.1:0/1102879245 shutdown_connections 2024-03-18T19:05:03.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.121+0000 7f4b3c016700 1 -- 172.21.15.1:0/1102879245 wait complete. 2024-03-18T19:05:03.131 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T19:05:03.131 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:05:03.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:05:03.225 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T19:05:03.225 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:05:03.225 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:05:03.244 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:05:03.258 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:03.262+0000 7f6d09147240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:05:03.267 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:03.271+0000 7f420e1eb240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:05:03.405 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:03.408+0000 7f6d09147240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:05:03.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.442+0000 7f56d07f8700 1 Processor -- start 2024-03-18T19:05:03.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.442+0000 7f56d07f8700 1 -- start start 2024-03-18T19:05:03.439 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:03.443+0000 7f420e1eb240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:05:03.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.443+0000 7f56d07f8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f56c8143c20 0x7f56c80b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:03.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.443+0000 7f56d07f8700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f56c80b3f40 0x7f56c806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:03.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.443+0000 7f56d07f8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56c806f070 0x7f56c8071650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:03.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.443+0000 7f56d07f8700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c80b43c0 con 0x7f56c8143c20 2024-03-18T19:05:03.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.443+0000 7f56d07f8700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c80b51b0 con 0x7f56c80b3f40 2024-03-18T19:05:03.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.443+0000 7f56d07f8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c806f4f0 con 0x7f56c806f070 2024-03-18T19:05:03.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.443+0000 7f56ce594700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f56c8143c20 0x7f56c80b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:03.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.443+0000 7f56ce594700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f56c8143c20 0x7f56c80b3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42690/0 (socket says 172.21.15.1:42690) 2024-03-18T19:05:03.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.443+0000 7f56ce594700 1 -- 172.21.15.1:0/1919247931 learned_addr learned my addr 172.21.15.1:0/1919247931 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:03.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.443+0000 7f56ced95700 1 --2- 172.21.15.1:0/1919247931 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56c806f070 0x7f56c8071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:03.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.443+0000 7f56cdd93700 1 --2- 172.21.15.1:0/1919247931 >> v2:172.21.15.165:3300/0 conn(0x7f56c80b3f40 0x7f56c806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:03.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.444+0000 7f56ce594700 1 -- 172.21.15.1:0/1919247931 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56c806f070 msgr2=0x7f56c8071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:03.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.444+0000 7f56ce594700 1 --2- 172.21.15.1:0/1919247931 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56c806f070 0x7f56c8071650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.444+0000 7f56ce594700 1 -- 172.21.15.1:0/1919247931 >> v2:172.21.15.165:3300/0 conn(0x7f56c80b3f40 msgr2=0x7f56c806eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:03.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.444+0000 7f56ce594700 1 --2- 172.21.15.1:0/1919247931 >> v2:172.21.15.165:3300/0 conn(0x7f56c80b3f40 0x7f56c806eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.444+0000 7f56ce594700 1 -- 172.21.15.1:0/1919247931 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c8144000 con 0x7f56c8143c20 2024-03-18T19:05:03.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.444+0000 7f56ce594700 1 --2- 172.21.15.1:0/1919247931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c8143c20 0x7f56c80b3a00 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f56c80afae0 tx=0x7f56c4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af3db0e5395d1223 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:03.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.444+0000 7f56cd592700 1 -- 172.21.15.1:0/1919247931 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4011830 con 0x7f56c8143c20 2024-03-18T19:05:03.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.444+0000 7f56cd592700 1 -- 172.21.15.1:0/1919247931 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c4011990 con 0x7f56c8143c20 2024-03-18T19:05:03.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.444+0000 7f56cd592700 1 -- 172.21.15.1:0/1919247931 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4011d20 con 0x7f56c8143c20 2024-03-18T19:05:03.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.445+0000 7f56d07f8700 1 -- 172.21.15.1:0/1919247931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c8143c20 msgr2=0x7f56c80b3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:03.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.445+0000 7f56d07f8700 1 --2- 172.21.15.1:0/1919247931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c8143c20 0x7f56c80b3a00 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f56c80afae0 tx=0x7f56c4009c00 comp rx=0 tx=0).stop 2024-03-18T19:05:03.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.445+0000 7f56d07f8700 1 -- 172.21.15.1:0/1919247931 shutdown_connections 2024-03-18T19:05:03.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.445+0000 7f56d07f8700 1 --2- 172.21.15.1:0/1919247931 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56c806f070 0x7f56c8071650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.445+0000 7f56d07f8700 1 --2- 172.21.15.1:0/1919247931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c8143c20 0x7f56c80b3a00 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.445+0000 7f56d07f8700 1 --2- 172.21.15.1:0/1919247931 >> v2:172.21.15.165:3300/0 conn(0x7f56c80b3f40 0x7f56c806eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.445+0000 7f56d07f8700 1 -- 172.21.15.1:0/1919247931 >> 172.21.15.1:0/1919247931 conn(0x7f56c80b1d20 msgr2=0x7f56c80b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:03.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.445+0000 7f56d07f8700 1 -- 172.21.15.1:0/1919247931 shutdown_connections 2024-03-18T19:05:03.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.445+0000 7f56d07f8700 1 -- 172.21.15.1:0/1919247931 wait complete. 2024-03-18T19:05:03.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.446+0000 7f56d07f8700 1 Processor -- start 2024-03-18T19:05:03.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.446+0000 7f56d07f8700 1 -- start start 2024-03-18T19:05:03.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.447+0000 7f56d07f8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56c806f070 0x7f56c80cb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:03.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.447+0000 7f56d07f8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c80b3f40 0x7f56c80c6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:03.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.447+0000 7f56d07f8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56c8143c20 0x7f56c80c6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:03.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.447+0000 7f56d07f8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c80c5030 con 0x7f56c80b3f40 2024-03-18T19:05:03.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.447+0000 7f56d07f8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c81ef1e0 con 0x7f56c806f070 2024-03-18T19:05:03.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.447+0000 7f56d07f8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c81ef4c0 con 0x7f56c8143c20 2024-03-18T19:05:03.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.448+0000 7f56cdd93700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c80b3f40 0x7f56c80c6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:03.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.448+0000 7f56cdd93700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c80b3f40 0x7f56c80c6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42706/0 (socket says 172.21.15.1:42706) 2024-03-18T19:05:03.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.448+0000 7f56cdd93700 1 -- 172.21.15.1:0/1544798122 learned_addr learned my addr 172.21.15.1:0/1544798122 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:03.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.448+0000 7f56ce594700 1 --2- 172.21.15.1:0/1544798122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56c806f070 0x7f56c80cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:03.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.448+0000 7f56ced95700 1 --2- 172.21.15.1:0/1544798122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56c8143c20 0x7f56c80c6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:03.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.448+0000 7f56cdd93700 1 -- 172.21.15.1:0/1544798122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56c8143c20 msgr2=0x7f56c80c6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:03.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.448+0000 7f56cdd93700 1 --2- 172.21.15.1:0/1544798122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56c8143c20 0x7f56c80c6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.448+0000 7f56cdd93700 1 -- 172.21.15.1:0/1544798122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56c806f070 msgr2=0x7f56c80cb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:03.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.448+0000 7f56cdd93700 1 --2- 172.21.15.1:0/1544798122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56c806f070 0x7f56c80cb820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.448+0000 7f56cdd93700 1 -- 172.21.15.1:0/1544798122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c4009550 con 0x7f56c80b3f40 2024-03-18T19:05:03.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.448+0000 7f56cdd93700 1 --2- 172.21.15.1:0/1544798122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c80b3f40 0x7f56c80c6a40 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f56b800e980 tx=0x7f56b800ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:03.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.448+0000 7f56beffd700 1 -- 172.21.15.1:0/1544798122 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b8009710 con 0x7f56c80b3f40 2024-03-18T19:05:03.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.448+0000 7f56beffd700 1 -- 172.21.15.1:0/1544798122 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b800c690 con 0x7f56c80b3f40 2024-03-18T19:05:03.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.448+0000 7f56d07f8700 1 -- 172.21.15.1:0/1544798122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c81ef7a0 con 0x7f56c80b3f40 2024-03-18T19:05:03.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.448+0000 7f56beffd700 1 -- 172.21.15.1:0/1544798122 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b800ca20 con 0x7f56c80b3f40 2024-03-18T19:05:03.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.448+0000 7f56d07f8700 1 -- 172.21.15.1:0/1544798122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c81efa80 con 0x7f56c80b3f40 2024-03-18T19:05:03.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.449+0000 7f56d07f8700 1 -- 172.21.15.1:0/1544798122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c80676a0 con 0x7f56c80b3f40 2024-03-18T19:05:03.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.450+0000 7f56beffd700 1 -- 172.21.15.1:0/1544798122 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f56b800cb80 con 0x7f56c80b3f40 2024-03-18T19:05:03.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.450+0000 7f56beffd700 1 --2- 172.21.15.1:0/1544798122 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56ac06adb0 0x7f56ac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:03.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.450+0000 7f56ce594700 1 -- 172.21.15.1:0/1544798122 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56ac06adb0 msgr2=0x7f56ac06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:03.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.450+0000 7f56beffd700 1 -- 172.21.15.1:0/1544798122 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6519+0+0 (secure 0 0 0) 0x7f56b8089740 con 0x7f56c80b3f40 2024-03-18T19:05:03.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.450+0000 7f56ce594700 1 --2- 172.21.15.1:0/1544798122 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56ac06adb0 0x7f56ac06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:03.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.459+0000 7f56beffd700 1 -- 172.21.15.1:0/1544798122 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f56b8054450 con 0x7f56c80b3f40 2024-03-18T19:05:03.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.651+0000 7f56ce594700 1 -- 172.21.15.1:0/1544798122 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56ac06adb0 msgr2=0x7f56ac06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:03.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.651+0000 7f56ce594700 1 --2- 172.21.15.1:0/1544798122 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56ac06adb0 0x7f56ac06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:03.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.668+0000 7f56d07f8700 1 -- 172.21.15.1:0/1544798122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56c815cf30 con 0x7f56c80b3f40 2024-03-18T19:05:03.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.669+0000 7f56beffd700 1 -- 172.21.15.1:0/1544798122 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+16988 (secure 0 0 0) 0x7f56b805ce60 con 0x7f56c80b3f40 2024-03-18T19:05:03.665 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:03.665 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":75,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:05:03.104576+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:03:00.371114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-18T19:03:01.513069+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":75,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.1d","mappings":[{"from":0,"to":6}]},{"pgid":"8.2f","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6833/76998":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/275404534":"2024-03-19T19:03:17.738629+0000","172.21.15.1:6832/76998":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/1571059650":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/207797678":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/3342330530":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/953259654":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/3431076653":"2024-03-19T19:03:17.738629+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_removed_snaps":[{"pool":8,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:05:03.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.671+0000 7f56d07f8700 1 -- 172.21.15.1:0/1544798122 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56ac06adb0 msgr2=0x7f56ac06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:03.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.671+0000 7f56d07f8700 1 --2- 172.21.15.1:0/1544798122 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56ac06adb0 0x7f56ac06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.671+0000 7f56d07f8700 1 -- 172.21.15.1:0/1544798122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c80b3f40 msgr2=0x7f56c80c6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:03.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.671+0000 7f56d07f8700 1 --2- 172.21.15.1:0/1544798122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c80b3f40 0x7f56c80c6a40 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f56b800e980 tx=0x7f56b800ec90 comp rx=0 tx=0).stop 2024-03-18T19:05:03.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.671+0000 7f56d07f8700 1 -- 172.21.15.1:0/1544798122 shutdown_connections 2024-03-18T19:05:03.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.671+0000 7f56d07f8700 1 --2- 172.21.15.1:0/1544798122 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56ac06adb0 0x7f56ac06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.671+0000 7f56d07f8700 1 --2- 172.21.15.1:0/1544798122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56c8143c20 0x7f56c80c6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.671+0000 7f56d07f8700 1 --2- 172.21.15.1:0/1544798122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56c806f070 0x7f56c80cb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.671+0000 7f56d07f8700 1 --2- 172.21.15.1:0/1544798122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56c80b3f40 0x7f56c80c6a40 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.671+0000 7f56d07f8700 1 -- 172.21.15.1:0/1544798122 >> 172.21.15.1:0/1544798122 conn(0x7f56c80b1d20 msgr2=0x7f56c806ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:03.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.671+0000 7f56d07f8700 1 -- 172.21.15.1:0/1544798122 shutdown_connections 2024-03-18T19:05:03.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.671+0000 7f56d07f8700 1 -- 172.21.15.1:0/1544798122 wait complete. 2024-03-18T19:05:03.674 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:03.678+0000 7f6d09147240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:05:03.680 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T19:03:00.371114+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T19:03:01.513069+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 75, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:05:03.680 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:05:03.730 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:03.734+0000 7f420e1eb240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:05:03.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.816+0000 7f394bae4700 1 Processor -- start 2024-03-18T19:05:03.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.817+0000 7f394bae4700 1 -- start start 2024-03-18T19:05:03.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.817+0000 7f394bae4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f394406ae60 0x7f39440cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:03.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.817+0000 7f394bae4700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394415c500 con 0x7f3944071500 2024-03-18T19:05:03.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.817+0000 7f394bae4700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39440af8b0 con 0x7f3944158e30 2024-03-18T19:05:03.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.817+0000 7f394bae4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3944071980 con 0x7f394406ae60 2024-03-18T19:05:03.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.818+0000 7f394a081700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f394406ae60 0x7f39440cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:03.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.818+0000 7f394907f700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3944071500 0x7f394406a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39566/0 (socket says 172.21.15.1:39566) 2024-03-18T19:05:03.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.818+0000 7f394907f700 1 -- 172.21.15.1:0/2074017923 learned_addr learned my addr 172.21.15.1:0/2074017923 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:03.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.818+0000 7f394a081700 1 -- 172.21.15.1:0/2074017923 >> v1:172.21.15.165:6789/0 conn(0x7f3944158e30 legacy=0x7f39440cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:03.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.818+0000 7f394a081700 1 -- 172.21.15.1:0/2074017923 >> v1:172.21.15.1:6789/0 conn(0x7f3944071500 legacy=0x7f394406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:03.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.818+0000 7f394a081700 1 -- 172.21.15.1:0/2074017923 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394415b8f0 con 0x7f394406ae60 2024-03-18T19:05:03.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.818+0000 7f394887e700 1 -- 172.21.15.1:0/2074017923 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258991831 0 0) 0x7f394415c500 con 0x7f3944071500 2024-03-18T19:05:03.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.818+0000 7f394a081700 1 --2- 172.21.15.1:0/2074017923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f394406ae60 0x7f39440cf150 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f393c009d90 tx=0x7f393c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7a1987d05c178ce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:03.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.818+0000 7f394887e700 1 -- 172.21.15.1:0/2074017923 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c00e070 con 0x7f394406ae60 2024-03-18T19:05:03.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.818+0000 7f394887e700 1 -- 172.21.15.1:0/2074017923 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393c00bbd0 con 0x7f394406ae60 2024-03-18T19:05:03.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.818+0000 7f394887e700 1 -- 172.21.15.1:0/2074017923 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c012430 con 0x7f394406ae60 2024-03-18T19:05:03.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.819+0000 7f394bae4700 1 -- 172.21.15.1:0/2074017923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f394406ae60 msgr2=0x7f39440cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:03.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.819+0000 7f394bae4700 1 --2- 172.21.15.1:0/2074017923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f394406ae60 0x7f39440cf150 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f393c009d90 tx=0x7f393c00b330 comp rx=0 tx=0).stop 2024-03-18T19:05:03.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.819+0000 7f394bae4700 1 -- 172.21.15.1:0/2074017923 shutdown_connections 2024-03-18T19:05:03.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.819+0000 7f394bae4700 1 --2- 172.21.15.1:0/2074017923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f394406ae60 0x7f39440cf150 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.819+0000 7f394bae4700 1 -- 172.21.15.1:0/2074017923 >> 172.21.15.1:0/2074017923 conn(0x7f39440cca00 msgr2=0x7f39440cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:03.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.819+0000 7f394bae4700 1 -- 172.21.15.1:0/2074017923 shutdown_connections 2024-03-18T19:05:03.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.819+0000 7f394bae4700 1 -- 172.21.15.1:0/2074017923 wait complete. 2024-03-18T19:05:03.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.820+0000 7f394bae4700 1 Processor -- start 2024-03-18T19:05:03.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.820+0000 7f394bae4700 1 -- start start 2024-03-18T19:05:03.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.821+0000 7f394bae4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f394406ae60 0x7f39441555f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:03.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.821+0000 7f394bae4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3944071500 0x7f39440b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:03.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.821+0000 7f394bae4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3944158e30 0x7f39440b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:03.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.821+0000 7f394bae4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3944157dd0 con 0x7f3944071500 2024-03-18T19:05:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.821+0000 7f394bae4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39440b33f0 con 0x7f3944158e30 2024-03-18T19:05:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.821+0000 7f394bae4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3944155b30 con 0x7f394406ae60 2024-03-18T19:05:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.821+0000 7f394907f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3944071500 0x7f39440b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.821+0000 7f394907f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3944071500 0x7f39440b3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42734/0 (socket says 172.21.15.1:42734) 2024-03-18T19:05:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.821+0000 7f394907f700 1 -- 172.21.15.1:0/3034516124 learned_addr learned my addr 172.21.15.1:0/3034516124 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.821+0000 7f394a081700 1 --2- 172.21.15.1:0/3034516124 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3944158e30 0x7f39440b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.821+0000 7f3949880700 1 --2- 172.21.15.1:0/3034516124 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f394406ae60 0x7f39441555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.821+0000 7f394907f700 1 -- 172.21.15.1:0/3034516124 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f394406ae60 msgr2=0x7f39441555f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.821+0000 7f394907f700 1 --2- 172.21.15.1:0/3034516124 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f394406ae60 0x7f39441555f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.821+0000 7f394907f700 1 -- 172.21.15.1:0/3034516124 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3944158e30 msgr2=0x7f39440b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.821+0000 7f394907f700 1 --2- 172.21.15.1:0/3034516124 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3944158e30 0x7f39440b3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.821+0000 7f394907f700 1 -- 172.21.15.1:0/3034516124 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393c009a70 con 0x7f3944071500 2024-03-18T19:05:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.822+0000 7f394907f700 1 --2- 172.21.15.1:0/3034516124 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3944071500 0x7f39440b3810 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f393400c520 tx=0x7f393400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.822+0000 7f393a7fc700 1 -- 172.21.15.1:0/3034516124 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3934004c80 con 0x7f3944071500 2024-03-18T19:05:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.822+0000 7f393a7fc700 1 -- 172.21.15.1:0/3034516124 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3934004e00 con 0x7f3944071500 2024-03-18T19:05:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.822+0000 7f394bae4700 1 -- 172.21.15.1:0/3034516124 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3944157af0 con 0x7f3944071500 2024-03-18T19:05:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.822+0000 7f393a7fc700 1 -- 172.21.15.1:0/3034516124 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3934007310 con 0x7f3944071500 2024-03-18T19:05:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.822+0000 7f394bae4700 1 -- 172.21.15.1:0/3034516124 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39440b3010 con 0x7f3944071500 2024-03-18T19:05:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.823+0000 7f394bae4700 1 -- 172.21.15.1:0/3034516124 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39440b2b90 con 0x7f3944071500 2024-03-18T19:05:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.823+0000 7f393a7fc700 1 -- 172.21.15.1:0/3034516124 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f3934007470 con 0x7f3944071500 2024-03-18T19:05:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.824+0000 7f393a7fc700 1 --2- 172.21.15.1:0/3034516124 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f392806adb0 0x7f392806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.824+0000 7f3949880700 1 -- 172.21.15.1:0/3034516124 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f392806adb0 msgr2=0x7f392806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:03.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.824+0000 7f393a7fc700 1 -- 172.21.15.1:0/3034516124 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6519+0+0 (secure 0 0 0) 0x7f39340893e0 con 0x7f3944071500 2024-03-18T19:05:03.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.824+0000 7f3949880700 1 --2- 172.21.15.1:0/3034516124 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f392806adb0 0x7f392806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:03.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:03.829+0000 7f393a7fc700 1 -- 172.21.15.1:0/3034516124 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f39440b2b90 con 0x7f3944071500 2024-03-18T19:05:04.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.025+0000 7f3949880700 1 -- 172.21.15.1:0/3034516124 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f392806adb0 msgr2=0x7f392806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:04.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.025+0000 7f3949880700 1 --2- 172.21.15.1:0/3034516124 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f392806adb0 0x7f392806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:04.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.030+0000 7f394bae4700 1 -- 172.21.15.1:0/3034516124 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f39440b2b90 con 0x7f3944071500 2024-03-18T19:05:04.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.030+0000 7f393a7fc700 1 -- 172.21.15.1:0/3034516124 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) v1 ==== 93+0+10 (secure 0 0 0) 0x7f39440b2b90 con 0x7f3944071500 2024-03-18T19:05:04.026 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:05:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.032+0000 7f394bae4700 1 -- 172.21.15.1:0/3034516124 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f392806adb0 msgr2=0x7f392806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.033+0000 7f394bae4700 1 --2- 172.21.15.1:0/3034516124 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f392806adb0 0x7f392806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.033+0000 7f394bae4700 1 -- 172.21.15.1:0/3034516124 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3944071500 msgr2=0x7f39440b3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.033+0000 7f394bae4700 1 --2- 172.21.15.1:0/3034516124 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3944071500 0x7f39440b3810 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f393400c520 tx=0x7f393400c830 comp rx=0 tx=0).stop 2024-03-18T19:05:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.033+0000 7f394bae4700 1 -- 172.21.15.1:0/3034516124 shutdown_connections 2024-03-18T19:05:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.033+0000 7f394bae4700 1 --2- 172.21.15.1:0/3034516124 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f392806adb0 0x7f392806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.033+0000 7f394bae4700 1 --2- 172.21.15.1:0/3034516124 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f394406ae60 0x7f39441555f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.033+0000 7f394bae4700 1 --2- 172.21.15.1:0/3034516124 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3944158e30 0x7f39440b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.033+0000 7f394bae4700 1 --2- 172.21.15.1:0/3034516124 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3944071500 0x7f39440b3810 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.033+0000 7f394bae4700 1 -- 172.21.15.1:0/3034516124 >> 172.21.15.1:0/3034516124 conn(0x7f39440cca00 msgr2=0x7f39440722f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:04.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.033+0000 7f394bae4700 1 -- 172.21.15.1:0/3034516124 shutdown_connections 2024-03-18T19:05:04.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.033+0000 7f394bae4700 1 -- 172.21.15.1:0/3034516124 wait complete. 2024-03-18T19:05:04.041 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T19:05:04.103 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:04.107+0000 7f6d09147240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:05:04.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.179+0000 7f3851eb0700 1 Processor -- start 2024-03-18T19:05:04.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.179+0000 7f3851eb0700 1 -- start start 2024-03-18T19:05:04.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.179+0000 7f3851eb0700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f384c06d560 0x7f384c0ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:04.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.179+0000 7f3851eb0700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f384c0cac80 con 0x7f384c071cc0 2024-03-18T19:05:04.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.179+0000 7f3851eb0700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f384c0caf60 con 0x7f384c15a240 2024-03-18T19:05:04.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.179+0000 7f3851eb0700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f384c0adff0 con 0x7f384c06d560 2024-03-18T19:05:04.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.180+0000 7f384affd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f384c071cc0 0x7f384c0ae410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39582/0 (socket says 172.21.15.1:39582) 2024-03-18T19:05:04.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.180+0000 7f384affd700 1 -- 172.21.15.1:0/1095421227 learned_addr learned my addr 172.21.15.1:0/1095421227 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:04.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.180+0000 7f384bfff700 1 --2- 172.21.15.1:0/1095421227 >> v2:172.21.15.165:3300/0 conn(0x7f384c06d560 0x7f384c0ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:04.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.180+0000 7f384a7fc700 1 -- 172.21.15.1:0/1095421227 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186231122 0 0) 0x7f384c0cac80 con 0x7f384c071cc0 2024-03-18T19:05:04.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.180+0000 7f384a7fc700 1 -- 172.21.15.1:0/1095421227 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38340035b0 con 0x7f384c071cc0 2024-03-18T19:05:04.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.180+0000 7f384a7fc700 1 -- 172.21.15.1:0/1095421227 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781732895 0 0) 0x7f384c0caf60 con 0x7f384c15a240 2024-03-18T19:05:04.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.180+0000 7f384a7fc700 1 -- 172.21.15.1:0/1095421227 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f384c0cac80 con 0x7f384c15a240 2024-03-18T19:05:04.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.180+0000 7f384bfff700 1 -- 172.21.15.1:0/1095421227 >> v1:172.21.15.165:6789/0 conn(0x7f384c15a240 legacy=0x7f384c0715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:04.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.180+0000 7f384bfff700 1 -- 172.21.15.1:0/1095421227 >> v1:172.21.15.1:6789/0 conn(0x7f384c071cc0 legacy=0x7f384c0ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:04.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.180+0000 7f384bfff700 1 -- 172.21.15.1:0/1095421227 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f384c0cb240 con 0x7f384c06d560 2024-03-18T19:05:04.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.180+0000 7f384bfff700 1 --2- 172.21.15.1:0/1095421227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f384c06d560 0x7f384c0ca580 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3844009d90 tx=0x7f384400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=966ad9dc2861352 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:04.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.180+0000 7f384a7fc700 1 -- 172.21.15.1:0/1095421227 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384400e070 con 0x7f384c06d560 2024-03-18T19:05:04.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.180+0000 7f384a7fc700 1 -- 172.21.15.1:0/1095421227 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384400bb20 con 0x7f384c06d560 2024-03-18T19:05:04.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.180+0000 7f384a7fc700 1 -- 172.21.15.1:0/1095421227 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3844012430 con 0x7f384c06d560 2024-03-18T19:05:04.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.181+0000 7f3851eb0700 1 -- 172.21.15.1:0/1095421227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f384c06d560 msgr2=0x7f384c0ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:04.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.181+0000 7f3851eb0700 1 --2- 172.21.15.1:0/1095421227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f384c06d560 0x7f384c0ca580 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3844009d90 tx=0x7f384400b330 comp rx=0 tx=0).stop 2024-03-18T19:05:04.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.181+0000 7f3851eb0700 1 -- 172.21.15.1:0/1095421227 shutdown_connections 2024-03-18T19:05:04.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.181+0000 7f3851eb0700 1 --2- 172.21.15.1:0/1095421227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f384c06d560 0x7f384c0ca580 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.181+0000 7f3851eb0700 1 -- 172.21.15.1:0/1095421227 >> 172.21.15.1:0/1095421227 conn(0x7f384c0cff30 msgr2=0x7f384c0d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:04.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.181+0000 7f3851eb0700 1 -- 172.21.15.1:0/1095421227 shutdown_connections 2024-03-18T19:05:04.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.181+0000 7f3851eb0700 1 -- 172.21.15.1:0/1095421227 wait complete. 2024-03-18T19:05:04.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.182+0000 7f3851eb0700 1 Processor -- start 2024-03-18T19:05:04.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.182+0000 7f3851eb0700 1 -- start start 2024-03-18T19:05:04.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.183+0000 7f3851eb0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f384c06d560 0x7f384c155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:04.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.183+0000 7f3851eb0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f384c071cc0 0x7f384c157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:04.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.183+0000 7f3851eb0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f384c15a240 0x7f384c1578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:04.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.183+0000 7f3851eb0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384c156500 con 0x7f384c071cc0 2024-03-18T19:05:04.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.183+0000 7f3851eb0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384c0c7000 con 0x7f384c06d560 2024-03-18T19:05:04.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.183+0000 7f3851eb0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f384c0c72e0 con 0x7f384c15a240 2024-03-18T19:05:04.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.183+0000 7f384affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f384c071cc0 0x7f384c157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:04.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.183+0000 7f384b7fe700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f384c06d560 0x7f384c155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:04.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.183+0000 7f384bfff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f384c15a240 0x7f384c1578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:04.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.183+0000 7f384affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f384c071cc0 0x7f384c157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42736/0 (socket says 172.21.15.1:42736) 2024-03-18T19:05:04.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.183+0000 7f384b7fe700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f384c06d560 0x7f384c155fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:33340/0 (socket says 172.21.15.1:33340) 2024-03-18T19:05:04.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.183+0000 7f384affd700 1 -- 172.21.15.1:0/1359317852 learned_addr learned my addr 172.21.15.1:0/1359317852 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:04.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.183+0000 7f384bfff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f384c15a240 0x7f384c1578d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:54558/0 (socket says 172.21.15.1:54558) 2024-03-18T19:05:04.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.184+0000 7f384affd700 1 -- 172.21.15.1:0/1359317852 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f384c15a240 msgr2=0x7f384c1578d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:04.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.184+0000 7f384affd700 1 --2- 172.21.15.1:0/1359317852 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f384c15a240 0x7f384c1578d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.184+0000 7f384affd700 1 -- 172.21.15.1:0/1359317852 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f384c06d560 msgr2=0x7f384c155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:04.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.184+0000 7f384affd700 1 --2- 172.21.15.1:0/1359317852 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f384c06d560 0x7f384c155fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.184+0000 7f384affd700 1 -- 172.21.15.1:0/1359317852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3844009a70 con 0x7f384c071cc0 2024-03-18T19:05:04.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.184+0000 7f384b7fe700 1 --2- 172.21.15.1:0/1359317852 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f384c06d560 0x7f384c155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:04.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.184+0000 7f384bfff700 1 --2- 172.21.15.1:0/1359317852 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f384c15a240 0x7f384c1578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:04.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.184+0000 7f384affd700 1 --2- 172.21.15.1:0/1359317852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f384c071cc0 0x7f384c157390 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f384000c700 tx=0x7f384000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:04.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.184+0000 7f3833fff700 1 -- 172.21.15.1:0/1359317852 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3840005d70 con 0x7f384c071cc0 2024-03-18T19:05:04.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.184+0000 7f3833fff700 1 -- 172.21.15.1:0/1359317852 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38400070d0 con 0x7f384c071cc0 2024-03-18T19:05:04.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.184+0000 7f3851eb0700 1 -- 172.21.15.1:0/1359317852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f384c0c75c0 con 0x7f384c071cc0 2024-03-18T19:05:04.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.184+0000 7f3833fff700 1 -- 172.21.15.1:0/1359317852 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3840007440 con 0x7f384c071cc0 2024-03-18T19:05:04.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.184+0000 7f3851eb0700 1 -- 172.21.15.1:0/1359317852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f384c0c7b80 con 0x7f384c071cc0 2024-03-18T19:05:04.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.185+0000 7f3851eb0700 1 -- 172.21.15.1:0/1359317852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f384c0cfd30 con 0x7f384c071cc0 2024-03-18T19:05:04.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.186+0000 7f3833fff700 1 -- 172.21.15.1:0/1359317852 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f38400075a0 con 0x7f384c071cc0 2024-03-18T19:05:04.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.186+0000 7f3833fff700 1 --2- 172.21.15.1:0/1359317852 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f383806adb0 0x7f383806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:04.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.186+0000 7f384b7fe700 1 -- 172.21.15.1:0/1359317852 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f383806adb0 msgr2=0x7f383806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:04.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.186+0000 7f3833fff700 1 -- 172.21.15.1:0/1359317852 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f3840088a90 con 0x7f384c071cc0 2024-03-18T19:05:04.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.186+0000 7f384b7fe700 1 --2- 172.21.15.1:0/1359317852 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f383806adb0 0x7f383806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:04.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.191+0000 7f3833fff700 1 -- 172.21.15.1:0/1359317852 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f38400538f0 con 0x7f384c071cc0 2024-03-18T19:05:04.198 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:04.202+0000 7f420e1eb240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:05:04.314 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:04.317+0000 7f6d09147240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:05:04.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.387+0000 7f384b7fe700 1 -- 172.21.15.1:0/1359317852 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f383806adb0 msgr2=0x7f383806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:04.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.387+0000 7f384b7fe700 1 --2- 172.21.15.1:0/1359317852 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f383806adb0 0x7f383806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:04.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.391+0000 7f3851eb0700 1 -- 172.21.15.1:0/1359317852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f384c0c7d70 con 0x7f384c071cc0 2024-03-18T19:05:04.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.391+0000 7f3833fff700 1 -- 172.21.15.1:0/1359317852 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v76) v1 ==== 104+0+11 (secure 0 0 0) 0x7f38400170a0 con 0x7f384c071cc0 2024-03-18T19:05:04.387 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T19:05:04.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.394+0000 7f3851eb0700 1 -- 172.21.15.1:0/1359317852 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f383806adb0 msgr2=0x7f383806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:04.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.394+0000 7f3851eb0700 1 --2- 172.21.15.1:0/1359317852 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f383806adb0 0x7f383806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.394+0000 7f3851eb0700 1 -- 172.21.15.1:0/1359317852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f384c071cc0 msgr2=0x7f384c157390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:04.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.394+0000 7f3851eb0700 1 --2- 172.21.15.1:0/1359317852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f384c071cc0 0x7f384c157390 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f384000c700 tx=0x7f384000ca10 comp rx=0 tx=0).stop 2024-03-18T19:05:04.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.394+0000 7f3851eb0700 1 -- 172.21.15.1:0/1359317852 shutdown_connections 2024-03-18T19:05:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.394+0000 7f3851eb0700 1 --2- 172.21.15.1:0/1359317852 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f383806adb0 0x7f383806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.394+0000 7f3851eb0700 1 --2- 172.21.15.1:0/1359317852 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f384c15a240 0x7f384c1578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.394+0000 7f3851eb0700 1 --2- 172.21.15.1:0/1359317852 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f384c06d560 0x7f384c155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.394+0000 7f3851eb0700 1 --2- 172.21.15.1:0/1359317852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f384c071cc0 0x7f384c157390 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.394+0000 7f3851eb0700 1 -- 172.21.15.1:0/1359317852 >> 172.21.15.1:0/1359317852 conn(0x7f384c0cff30 msgr2=0x7f384c06e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.394+0000 7f3851eb0700 1 -- 172.21.15.1:0/1359317852 shutdown_connections 2024-03-18T19:05:04.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.394+0000 7f3851eb0700 1 -- 172.21.15.1:0/1359317852 wait complete. 2024-03-18T19:05:04.403 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T19:05:04.422 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:04.427+0000 7f420e1eb240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:05:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.536+0000 7f3326613700 1 Processor -- start 2024-03-18T19:05:04.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.536+0000 7f3326613700 1 -- start start 2024-03-18T19:05:04.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.537+0000 7f3326613700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3320071bf0 0x7f332015a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:04.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.537+0000 7f3326613700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f332015de00 con 0x7f33200714e0 2024-03-18T19:05:04.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.537+0000 7f3326613700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f332015e0a0 con 0x7f332015a050 2024-03-18T19:05:04.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.537+0000 7f3326613700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f332015e380 con 0x7f3320071bf0 2024-03-18T19:05:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.537+0000 7f3324bb0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3320071bf0 0x7f332015a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.537+0000 7f331f7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f33200714e0 0x7f33200b27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42310/0 (socket says 172.21.15.1:42310) 2024-03-18T19:05:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.537+0000 7f3324bb0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3320071bf0 0x7f332015a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42138/0 (socket says 172.21.15.1:42138) 2024-03-18T19:05:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.537+0000 7f331f7fe700 1 -- 172.21.15.1:0/629664514 learned_addr learned my addr 172.21.15.1:0/629664514 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.537+0000 7f3324bb0700 1 -- 172.21.15.1:0/629664514 >> v1:172.21.15.165:6789/0 conn(0x7f332015a050 legacy=0x7f3320070dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.537+0000 7f3324bb0700 1 -- 172.21.15.1:0/629664514 >> v1:172.21.15.1:6789/0 conn(0x7f33200714e0 legacy=0x7f33200b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.537+0000 7f3324bb0700 1 -- 172.21.15.1:0/629664514 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332015e660 con 0x7f3320071bf0 2024-03-18T19:05:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.538+0000 7f3324bb0700 1 --2- 172.21.15.1:0/629664514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3320071bf0 0x7f332015a430 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3318009d90 tx=0x7f331800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf46d57689ffb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.538+0000 7f331effd700 1 -- 172.21.15.1:0/629664514 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331800e070 con 0x7f3320071bf0 2024-03-18T19:05:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.538+0000 7f331effd700 1 -- 172.21.15.1:0/629664514 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331800bbd0 con 0x7f3320071bf0 2024-03-18T19:05:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.538+0000 7f331effd700 1 -- 172.21.15.1:0/629664514 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3318012430 con 0x7f3320071bf0 2024-03-18T19:05:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.538+0000 7f3326613700 1 -- 172.21.15.1:0/629664514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3320071bf0 msgr2=0x7f332015a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.538+0000 7f3326613700 1 --2- 172.21.15.1:0/629664514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3320071bf0 0x7f332015a430 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3318009d90 tx=0x7f331800b330 comp rx=0 tx=0).stop 2024-03-18T19:05:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.538+0000 7f3326613700 1 -- 172.21.15.1:0/629664514 shutdown_connections 2024-03-18T19:05:04.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.538+0000 7f3326613700 1 --2- 172.21.15.1:0/629664514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3320071bf0 0x7f332015a430 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.538+0000 7f3326613700 1 -- 172.21.15.1:0/629664514 >> 172.21.15.1:0/629664514 conn(0x7f3320066b70 msgr2=0x7f332006a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:04.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.538+0000 7f3326613700 1 -- 172.21.15.1:0/629664514 shutdown_connections 2024-03-18T19:05:04.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.538+0000 7f3326613700 1 -- 172.21.15.1:0/629664514 wait complete. 2024-03-18T19:05:04.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.540+0000 7f3326613700 1 Processor -- start 2024-03-18T19:05:04.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.540+0000 7f3326613700 1 -- start start 2024-03-18T19:05:04.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.541+0000 7f3326613700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33200714e0 0x7f3320155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:04.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.541+0000 7f3326613700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3320071bf0 0x7f33201e5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:04.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.541+0000 7f3326613700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f332015a050 0x7f332014ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:04.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.541+0000 7f3326613700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3320159310 con 0x7f3320071bf0 2024-03-18T19:05:04.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.541+0000 7f3326613700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3320159800 con 0x7f33200714e0 2024-03-18T19:05:04.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.541+0000 7f3326613700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3320159520 con 0x7f332015a050 2024-03-18T19:05:04.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.541+0000 7f331ffff700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33200714e0 0x7f3320155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:04.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.541+0000 7f331ffff700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33200714e0 0x7f3320155a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:50498/0 (socket says 172.21.15.1:50498) 2024-03-18T19:05:04.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.541+0000 7f331ffff700 1 -- 172.21.15.1:0/472754972 learned_addr learned my addr 172.21.15.1:0/472754972 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:04.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.541+0000 7f331f7fe700 1 --2- 172.21.15.1:0/472754972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3320071bf0 0x7f33201e5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:04.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.541+0000 7f3324bb0700 1 --2- 172.21.15.1:0/472754972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f332015a050 0x7f332014ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.542+0000 7f3324bb0700 1 -- 172.21.15.1:0/472754972 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33200714e0 msgr2=0x7f3320155a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.542+0000 7f3324bb0700 1 --2- 172.21.15.1:0/472754972 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33200714e0 0x7f3320155a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.542+0000 7f3324bb0700 1 -- 172.21.15.1:0/472754972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3320071bf0 msgr2=0x7f33201e5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.542+0000 7f3324bb0700 1 --2- 172.21.15.1:0/472754972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3320071bf0 0x7f33201e5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.542+0000 7f3324bb0700 1 -- 172.21.15.1:0/472754972 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3318009a70 con 0x7f332015a050 2024-03-18T19:05:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.542+0000 7f331f7fe700 1 --2- 172.21.15.1:0/472754972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3320071bf0 0x7f33201e5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.542+0000 7f3324bb0700 1 --2- 172.21.15.1:0/472754972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f332015a050 0x7f332014ee30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3318016000 tx=0x7f3318004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:04.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.543+0000 7f331cff9700 1 -- 172.21.15.1:0/472754972 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3318012da0 con 0x7f332015a050 2024-03-18T19:05:04.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.543+0000 7f331cff9700 1 -- 172.21.15.1:0/472754972 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3318020070 con 0x7f332015a050 2024-03-18T19:05:04.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.543+0000 7f3326613700 1 -- 172.21.15.1:0/472754972 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3320150be0 con 0x7f332015a050 2024-03-18T19:05:04.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.543+0000 7f331cff9700 1 -- 172.21.15.1:0/472754972 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3318004280 con 0x7f332015a050 2024-03-18T19:05:04.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.543+0000 7f3326613700 1 -- 172.21.15.1:0/472754972 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33201511a0 con 0x7f332015a050 2024-03-18T19:05:04.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.544+0000 7f3326613700 1 -- 172.21.15.1:0/472754972 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33200aeca0 con 0x7f332015a050 2024-03-18T19:05:04.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.545+0000 7f331cff9700 1 -- 172.21.15.1:0/472754972 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f33180043e0 con 0x7f332015a050 2024-03-18T19:05:04.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.545+0000 7f331cff9700 1 --2- 172.21.15.1:0/472754972 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f330c06adb0 0x7f330c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:04.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.545+0000 7f331ffff700 1 -- 172.21.15.1:0/472754972 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f330c06adb0 msgr2=0x7f330c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:04.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.545+0000 7f331cff9700 1 -- 172.21.15.1:0/472754972 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f3318022070 con 0x7f332015a050 2024-03-18T19:05:04.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.545+0000 7f331ffff700 1 --2- 172.21.15.1:0/472754972 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f330c06adb0 0x7f330c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:04.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.550+0000 7f331cff9700 1 -- 172.21.15.1:0/472754972 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3318056990 con 0x7f332015a050 2024-03-18T19:05:04.692 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:04.696+0000 7f6d09147240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:05:04.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.746+0000 7f331ffff700 1 -- 172.21.15.1:0/472754972 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f330c06adb0 msgr2=0x7f330c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:04.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.746+0000 7f331ffff700 1 --2- 172.21.15.1:0/472754972 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f330c06adb0 0x7f330c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:04.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.752+0000 7f3326613700 1 -- 172.21.15.1:0/472754972 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f33200676a0 con 0x7f332015a050 2024-03-18T19:05:04.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.753+0000 7f331cff9700 1 -- 172.21.15.1:0/472754972 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v76) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3318059fb0 con 0x7f332015a050 2024-03-18T19:05:04.748 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T19:05:04.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.755+0000 7f3326613700 1 -- 172.21.15.1:0/472754972 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f330c06adb0 msgr2=0x7f330c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:04.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.755+0000 7f3326613700 1 --2- 172.21.15.1:0/472754972 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f330c06adb0 0x7f330c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.755+0000 7f3326613700 1 -- 172.21.15.1:0/472754972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f332015a050 msgr2=0x7f332014ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:04.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.755+0000 7f3326613700 1 --2- 172.21.15.1:0/472754972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f332015a050 0x7f332014ee30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3318016000 tx=0x7f3318004ad0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.755+0000 7f3326613700 1 -- 172.21.15.1:0/472754972 shutdown_connections 2024-03-18T19:05:04.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.755+0000 7f3326613700 1 --2- 172.21.15.1:0/472754972 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f330c06adb0 0x7f330c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.755+0000 7f3326613700 1 --2- 172.21.15.1:0/472754972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f332015a050 0x7f332014ee30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.755+0000 7f3326613700 1 --2- 172.21.15.1:0/472754972 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33200714e0 0x7f3320155a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.755+0000 7f3326613700 1 --2- 172.21.15.1:0/472754972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3320071bf0 0x7f33201e5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.755+0000 7f3326613700 1 -- 172.21.15.1:0/472754972 >> 172.21.15.1:0/472754972 conn(0x7f3320066b70 msgr2=0x7f33200b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:04.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.755+0000 7f3326613700 1 -- 172.21.15.1:0/472754972 shutdown_connections 2024-03-18T19:05:04.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.755+0000 7f3326613700 1 -- 172.21.15.1:0/472754972 wait complete. 2024-03-18T19:05:04.764 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T19:05:04.825 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:04.829+0000 7f420e1eb240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:05:04.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.935+0000 7fc810e27700 1 Processor -- start 2024-03-18T19:05:04.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.935+0000 7fc810e27700 1 -- start start 2024-03-18T19:05:04.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.936+0000 7fc810e27700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc80c13e130 0x7fc80c13e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:04.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.936+0000 7fc810e27700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc80c0714c0 0x7fc80c15bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:04.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.936+0000 7fc810e27700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc80c0cbcc0 con 0x7fc80c15b750 2024-03-18T19:05:04.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.936+0000 7fc810e27700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc80c15c210 con 0x7fc80c0714c0 2024-03-18T19:05:04.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.936+0000 7fc810e27700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc80c15c4f0 con 0x7fc80c13e130 2024-03-18T19:05:04.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.937+0000 7fc80ad9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc80c0714c0 0x7fc80c15bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:04.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.937+0000 7fc80ad9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc80c0714c0 0x7fc80c15bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42152/0 (socket says 172.21.15.1:42152) 2024-03-18T19:05:04.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.937+0000 7fc80a59c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc80c13e130 0x7fc80c13e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:04.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.937+0000 7fc80ad9d700 1 -- 172.21.15.1:0/1410870326 learned_addr learned my addr 172.21.15.1:0/1410870326 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:04.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.937+0000 7fc80ad9d700 1 -- 172.21.15.1:0/1410870326 >> v2:172.21.15.165:3300/0 conn(0x7fc80c13e130 msgr2=0x7fc80c13e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:04.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.937+0000 7fc80ad9d700 1 --2- 172.21.15.1:0/1410870326 >> v2:172.21.15.165:3300/0 conn(0x7fc80c13e130 0x7fc80c13e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.937+0000 7fc80ad9d700 1 -- 172.21.15.1:0/1410870326 >> v1:172.21.15.165:6789/0 conn(0x7fc80c15b750 legacy=0x7fc80c070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:04.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.937+0000 7fc80959a700 1 -- 172.21.15.1:0/1410870326 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826710110 0 0) 0x7fc80c0cbcc0 con 0x7fc80c15b750 2024-03-18T19:05:04.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.937+0000 7fc80ad9d700 1 -- 172.21.15.1:0/1410870326 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc80c0cb950 con 0x7fc80c0714c0 2024-03-18T19:05:04.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.937+0000 7fc80ad9d700 1 --2- 172.21.15.1:0/1410870326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc80c0714c0 0x7fc80c15bbe0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc80c06b730 tx=0x7fc80400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e6990116f3e530f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:04.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.937+0000 7fc80959a700 1 -- 172.21.15.1:0/1410870326 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80400e070 con 0x7fc80c0714c0 2024-03-18T19:05:04.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.937+0000 7fc80959a700 1 -- 172.21.15.1:0/1410870326 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc80400b910 con 0x7fc80c0714c0 2024-03-18T19:05:04.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.937+0000 7fc80959a700 1 -- 172.21.15.1:0/1410870326 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80400bcc0 con 0x7fc80c0714c0 2024-03-18T19:05:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.938+0000 7fc810e27700 1 -- 172.21.15.1:0/1410870326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc80c0714c0 msgr2=0x7fc80c15bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.938+0000 7fc810e27700 1 --2- 172.21.15.1:0/1410870326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc80c0714c0 0x7fc80c15bbe0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc80c06b730 tx=0x7fc80400b330 comp rx=0 tx=0).stop 2024-03-18T19:05:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.938+0000 7fc810e27700 1 -- 172.21.15.1:0/1410870326 shutdown_connections 2024-03-18T19:05:04.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.938+0000 7fc810e27700 1 --2- 172.21.15.1:0/1410870326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc80c0714c0 0x7fc80c15bbe0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.938+0000 7fc810e27700 1 --2- 172.21.15.1:0/1410870326 >> v2:172.21.15.165:3300/0 conn(0x7fc80c13e130 0x7fc80c13e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.938+0000 7fc810e27700 1 -- 172.21.15.1:0/1410870326 >> 172.21.15.1:0/1410870326 conn(0x7fc80c06a750 msgr2=0x7fc80c06ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.938+0000 7fc810e27700 1 -- 172.21.15.1:0/1410870326 shutdown_connections 2024-03-18T19:05:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.938+0000 7fc810e27700 1 -- 172.21.15.1:0/1410870326 wait complete. 2024-03-18T19:05:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.939+0000 7fc810e27700 1 Processor -- start 2024-03-18T19:05:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.939+0000 7fc810e27700 1 -- start start 2024-03-18T19:05:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.940+0000 7fc810e27700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc80c0714c0 0x7fc80c0b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.940+0000 7fc810e27700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc80c13e130 0x7fc80c13c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.940+0000 7fc810e27700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc80c15b750 0x7fc80c06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.940+0000 7fc810e27700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc80c0b3a90 con 0x7fc80c13e130 2024-03-18T19:05:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.940+0000 7fc810e27700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc80c0cfe30 con 0x7fc80c15b750 2024-03-18T19:05:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.940+0000 7fc810e27700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc80c155e10 con 0x7fc80c0714c0 2024-03-18T19:05:04.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.940+0000 7fc809d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc80c13e130 0x7fc80c13c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:04.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.940+0000 7fc809d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc80c13e130 0x7fc80c13c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42156/0 (socket says 172.21.15.1:42156) 2024-03-18T19:05:04.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.940+0000 7fc809d9b700 1 -- 172.21.15.1:0/2068895002 learned_addr learned my addr 172.21.15.1:0/2068895002 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:04.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.940+0000 7fc80ad9d700 1 --2- 172.21.15.1:0/2068895002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc80c15b750 0x7fc80c06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:04.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.940+0000 7fc80a59c700 1 --2- 172.21.15.1:0/2068895002 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc80c0714c0 0x7fc80c0b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:04.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.941+0000 7fc80ad9d700 1 -- 172.21.15.1:0/2068895002 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc80c0714c0 msgr2=0x7fc80c0b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:04.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.941+0000 7fc80ad9d700 1 --2- 172.21.15.1:0/2068895002 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc80c0714c0 0x7fc80c0b4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.941+0000 7fc80ad9d700 1 -- 172.21.15.1:0/2068895002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc80c13e130 msgr2=0x7fc80c13c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:04.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.941+0000 7fc80ad9d700 1 --2- 172.21.15.1:0/2068895002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc80c13e130 0x7fc80c13c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:04.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.941+0000 7fc80ad9d700 1 -- 172.21.15.1:0/2068895002 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc804009b30 con 0x7fc80c15b750 2024-03-18T19:05:04.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.941+0000 7fc809d9b700 1 --2- 172.21.15.1:0/2068895002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc80c13e130 0x7fc80c13c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:04.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.941+0000 7fc80ad9d700 1 --2- 172.21.15.1:0/2068895002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc80c15b750 0x7fc80c06de80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc804015000 tx=0x7fc804004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:04.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.941+0000 7fc7faffd700 1 -- 172.21.15.1:0/2068895002 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc804004840 con 0x7fc80c15b750 2024-03-18T19:05:04.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.941+0000 7fc7faffd700 1 -- 172.21.15.1:0/2068895002 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8040099b0 con 0x7fc80c15b750 2024-03-18T19:05:04.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.941+0000 7fc810e27700 1 -- 172.21.15.1:0/2068895002 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc80c159120 con 0x7fc80c15b750 2024-03-18T19:05:04.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.941+0000 7fc7faffd700 1 -- 172.21.15.1:0/2068895002 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80401c680 con 0x7fc80c15b750 2024-03-18T19:05:04.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.941+0000 7fc810e27700 1 -- 172.21.15.1:0/2068895002 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc80c1593c0 con 0x7fc80c15b750 2024-03-18T19:05:04.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.942+0000 7fc810e27700 1 -- 172.21.15.1:0/2068895002 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc80c0676a0 con 0x7fc80c15b750 2024-03-18T19:05:04.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.942+0000 7fc7faffd700 1 -- 172.21.15.1:0/2068895002 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fc80401c7e0 con 0x7fc80c15b750 2024-03-18T19:05:04.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.943+0000 7fc7faffd700 1 --2- 172.21.15.1:0/2068895002 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc7ec06adb0 0x7fc7ec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:04.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.943+0000 7fc80a59c700 1 -- 172.21.15.1:0/2068895002 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc7ec06adb0 msgr2=0x7fc7ec06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:04.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.943+0000 7fc7faffd700 1 -- 172.21.15.1:0/2068895002 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6503+0+0 (secure 0 0 0) 0x7fc80408b070 con 0x7fc80c15b750 2024-03-18T19:05:04.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.943+0000 7fc80a59c700 1 --2- 172.21.15.1:0/2068895002 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc7ec06adb0 0x7fc7ec06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:04.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:04.947+0000 7fc7faffd700 1 -- 172.21.15.1:0/2068895002 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc804055da0 con 0x7fc80c15b750 2024-03-18T19:05:05.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.144+0000 7fc80a59c700 1 -- 172.21.15.1:0/2068895002 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc7ec06adb0 msgr2=0x7fc7ec06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:05.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.144+0000 7fc80a59c700 1 --2- 172.21.15.1:0/2068895002 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc7ec06adb0 0x7fc7ec06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:05.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.168+0000 7fc810e27700 1 -- 172.21.15.1:0/2068895002 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc80c15b190 con 0x7fc80c15b750 2024-03-18T19:05:05.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.168+0000 7fc7faffd700 1 -- 172.21.15.1:0/2068895002 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v76) v1 ==== 80+0+449 (secure 0 0 0) 0x7fc8040593c0 con 0x7fc80c15b750 2024-03-18T19:05:05.164 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6833/76998 2024-03-19T19:03:17.738629+0000 2024-03-18T19:05:05.164 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/275404534 2024-03-19T19:03:17.738629+0000 2024-03-18T19:05:05.164 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6832/76998 2024-03-19T19:03:17.738629+0000 2024-03-18T19:05:05.164 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1571059650 2024-03-19T19:03:17.738629+0000 2024-03-18T19:05:05.164 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/207797678 2024-03-19T19:03:17.738629+0000 2024-03-18T19:05:05.165 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3342330530 2024-03-19T19:03:17.738629+0000 2024-03-18T19:05:05.165 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/953259654 2024-03-19T19:03:17.738629+0000 2024-03-18T19:05:05.165 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3431076653 2024-03-19T19:03:17.738629+0000 2024-03-18T19:05:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.171+0000 7fc810e27700 1 -- 172.21.15.1:0/2068895002 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc7ec06adb0 msgr2=0x7fc7ec06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.171+0000 7fc810e27700 1 --2- 172.21.15.1:0/2068895002 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc7ec06adb0 0x7fc7ec06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.171+0000 7fc810e27700 1 -- 172.21.15.1:0/2068895002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc80c15b750 msgr2=0x7fc80c06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.171+0000 7fc810e27700 1 --2- 172.21.15.1:0/2068895002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc80c15b750 0x7fc80c06de80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc804015000 tx=0x7fc804004ad0 comp rx=0 tx=0).stop 2024-03-18T19:05:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.171+0000 7fc810e27700 1 -- 172.21.15.1:0/2068895002 shutdown_connections 2024-03-18T19:05:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.171+0000 7fc810e27700 1 --2- 172.21.15.1:0/2068895002 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc7ec06adb0 0x7fc7ec06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.171+0000 7fc810e27700 1 --2- 172.21.15.1:0/2068895002 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc80c0714c0 0x7fc80c0b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.171+0000 7fc810e27700 1 --2- 172.21.15.1:0/2068895002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc80c15b750 0x7fc80c06de80 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.171+0000 7fc810e27700 1 --2- 172.21.15.1:0/2068895002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc80c13e130 0x7fc80c13c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:05.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.171+0000 7fc810e27700 1 -- 172.21.15.1:0/2068895002 >> 172.21.15.1:0/2068895002 conn(0x7fc80c06a750 msgr2=0x7fc80c1551f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.171+0000 7fc810e27700 1 -- 172.21.15.1:0/2068895002 shutdown_connections 2024-03-18T19:05:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.171+0000 7fc810e27700 1 -- 172.21.15.1:0/2068895002 wait complete. 2024-03-18T19:05:05.168 INFO:teuthology.orchestra.run.smithi001.stderr:listed 8 entries 2024-03-18T19:05:05.179 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-18T19:05:05.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:05.180 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T19:05:05.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[106451]: fuse finished with error 0 and tester_r 0 2024-03-18T19:05:05.259 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T19:05:05.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:05.259 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T19:05:05.279 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:05.283+0000 7f6d09147240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:05:05.318 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:05:05.318 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:05:05.337 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:05:05.337 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T19:05:05.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:05.337 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-18T19:05:05.353 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:05.356+0000 7f6d09147240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:05:05.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:05:05.369 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-18T19:05:05.370 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:05:05.370 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:05:05.389 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:05:05.389 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T19:05:05.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:05.389 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-18T19:05:05.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:05:05.447 INFO:teuthology.orchestra.run.smithi001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-18T19:05:05.447 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:05:05.447 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:05:05.456 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:05.460+0000 7f420e1eb240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:05:05.466 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:05:05.466 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T19:05:05.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:05.466 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-18T19:05:05.484 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:05.487+0000 7f6d09147240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:05:05.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:05:05.501 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-18T19:05:05.501 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:05:05.534 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:05.538+0000 7f420e1eb240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:05:05.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.632+0000 7f315f316700 1 Processor -- start 2024-03-18T19:05:05.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.633+0000 7f315f316700 1 -- start start 2024-03-18T19:05:05.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.633+0000 7f315f316700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f315806a750 0x7f31580cf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:05.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.633+0000 7f315f316700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f31580cf960 0x7f315806ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:05.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.633+0000 7f315f316700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f315806d640 con 0x7f31580cc6d0 2024-03-18T19:05:05.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.633+0000 7f315f316700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f315806de50 con 0x7f31580cf960 2024-03-18T19:05:05.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.633+0000 7f315f316700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31580acd80 con 0x7f315806a750 2024-03-18T19:05:05.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.634+0000 7f315c8b1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f31580cf960 0x7f315806ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:05.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.634+0000 7f315c8b1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f31580cf960 0x7f315806ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42160/0 (socket says 172.21.15.1:42160) 2024-03-18T19:05:05.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.634+0000 7f315c8b1700 1 -- 172.21.15.1:0/181018657 learned_addr learned my addr 172.21.15.1:0/181018657 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:05.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.634+0000 7f315d8b3700 1 --1- 172.21.15.1:0/181018657 >> v1:172.21.15.1:6789/0 conn(0x7f31580cc6d0 0x7f31580ccb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42322/0 (socket says 172.21.15.1:42322) 2024-03-18T19:05:05.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.634+0000 7f315d0b2700 1 --2- 172.21.15.1:0/181018657 >> v2:172.21.15.165:3300/0 conn(0x7f315806a750 0x7f31580cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:05.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.634+0000 7f315c8b1700 1 -- 172.21.15.1:0/181018657 >> v2:172.21.15.165:3300/0 conn(0x7f315806a750 msgr2=0x7f31580cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:05.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.634+0000 7f315c8b1700 1 --2- 172.21.15.1:0/181018657 >> v2:172.21.15.165:3300/0 conn(0x7f315806a750 0x7f31580cf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:05.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.634+0000 7f315c8b1700 1 -- 172.21.15.1:0/181018657 >> v1:172.21.15.1:6789/0 conn(0x7f31580cc6d0 legacy=0x7f31580ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:05.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.634+0000 7f315c8b1700 1 -- 172.21.15.1:0/181018657 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315806b070 con 0x7f31580cf960 2024-03-18T19:05:05.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.634+0000 7f315c8b1700 1 --2- 172.21.15.1:0/181018657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31580cf960 0x7f315806ab30 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f315806c800 tx=0x7f3148009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3156bd4ba62f60d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:05.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.634+0000 7f314ffff700 1 -- 172.21.15.1:0/181018657 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3148011830 con 0x7f31580cf960 2024-03-18T19:05:05.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.634+0000 7f314ffff700 1 -- 172.21.15.1:0/181018657 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3148011990 con 0x7f31580cf960 2024-03-18T19:05:05.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.634+0000 7f314ffff700 1 -- 172.21.15.1:0/181018657 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3148011d20 con 0x7f31580cf960 2024-03-18T19:05:05.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.635+0000 7f315f316700 1 -- 172.21.15.1:0/181018657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31580cf960 msgr2=0x7f315806ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:05.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.635+0000 7f315f316700 1 --2- 172.21.15.1:0/181018657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31580cf960 0x7f315806ab30 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f315806c800 tx=0x7f3148009bd0 comp rx=0 tx=0).stop 2024-03-18T19:05:05.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.635+0000 7f315f316700 1 -- 172.21.15.1:0/181018657 shutdown_connections 2024-03-18T19:05:05.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.635+0000 7f315f316700 1 --2- 172.21.15.1:0/181018657 >> v2:172.21.15.165:3300/0 conn(0x7f315806a750 0x7f31580cf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:05.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.635+0000 7f315f316700 1 --2- 172.21.15.1:0/181018657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31580cf960 0x7f315806ab30 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:05.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.635+0000 7f315f316700 1 -- 172.21.15.1:0/181018657 >> 172.21.15.1:0/181018657 conn(0x7f315806b860 msgr2=0x7f315806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:05.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.635+0000 7f315f316700 1 -- 172.21.15.1:0/181018657 shutdown_connections 2024-03-18T19:05:05.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.635+0000 7f315f316700 1 -- 172.21.15.1:0/181018657 wait complete. 2024-03-18T19:05:05.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.636+0000 7f315f316700 1 Processor -- start 2024-03-18T19:05:05.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.636+0000 7f315f316700 1 -- start start 2024-03-18T19:05:05.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.637+0000 7f315f316700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f315806a750 0x7f315814a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:05.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.637+0000 7f315f316700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31580cc6d0 0x7f31581465f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:05.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.637+0000 7f315f316700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31580cf960 0x7f3158146b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:05.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.637+0000 7f315f316700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3158145ee0 con 0x7f31580cf960 2024-03-18T19:05:05.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.637+0000 7f315f316700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3158146370 con 0x7f31580cc6d0 2024-03-18T19:05:05.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.637+0000 7f315f316700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31581452d0 con 0x7f315806a750 2024-03-18T19:05:05.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.637+0000 7f315d8b3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31580cf960 0x7f3158146b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:05.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.637+0000 7f315d8b3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31580cf960 0x7f3158146b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42170/0 (socket says 172.21.15.1:42170) 2024-03-18T19:05:05.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.637+0000 7f315d8b3700 1 -- 172.21.15.1:0/289022444 learned_addr learned my addr 172.21.15.1:0/289022444 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:05.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.637+0000 7f315d0b2700 1 --2- 172.21.15.1:0/289022444 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f315806a750 0x7f315814a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:05.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.637+0000 7f315c8b1700 1 --2- 172.21.15.1:0/289022444 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31580cc6d0 0x7f31581465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:05.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.638+0000 7f315d8b3700 1 -- 172.21.15.1:0/289022444 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f315806a750 msgr2=0x7f315814a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:05.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.638+0000 7f315d8b3700 1 --2- 172.21.15.1:0/289022444 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f315806a750 0x7f315814a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:05.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.638+0000 7f315d8b3700 1 -- 172.21.15.1:0/289022444 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31580cc6d0 msgr2=0x7f31581465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:05.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.638+0000 7f315d8b3700 1 --2- 172.21.15.1:0/289022444 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31580cc6d0 0x7f31581465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:05.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.638+0000 7f315d8b3700 1 -- 172.21.15.1:0/289022444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3148009550 con 0x7f31580cf960 2024-03-18T19:05:05.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.638+0000 7f315c8b1700 1 --2- 172.21.15.1:0/289022444 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31580cc6d0 0x7f31581465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:05.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.638+0000 7f315d8b3700 1 --2- 172.21.15.1:0/289022444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31580cf960 0x7f3158146b30 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f315000cbd0 tx=0x7f315000cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:05.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.638+0000 7f314dffb700 1 -- 172.21.15.1:0/289022444 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315000aaa0 con 0x7f31580cf960 2024-03-18T19:05:05.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.638+0000 7f314dffb700 1 -- 172.21.15.1:0/289022444 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315000ac00 con 0x7f31580cf960 2024-03-18T19:05:05.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.638+0000 7f315f316700 1 -- 172.21.15.1:0/289022444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31581455b0 con 0x7f31580cf960 2024-03-18T19:05:05.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.638+0000 7f314dffb700 1 -- 172.21.15.1:0/289022444 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3150007070 con 0x7f31580cf960 2024-03-18T19:05:05.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.638+0000 7f315f316700 1 -- 172.21.15.1:0/289022444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3158145b70 con 0x7f31580cf960 2024-03-18T19:05:05.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.639+0000 7f315f316700 1 -- 172.21.15.1:0/289022444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31580676a0 con 0x7f31580cf960 2024-03-18T19:05:05.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.640+0000 7f314dffb700 1 -- 172.21.15.1:0/289022444 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f31500071d0 con 0x7f31580cf960 2024-03-18T19:05:05.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.641+0000 7f314dffb700 1 --2- 172.21.15.1:0/289022444 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f313c06adb0 0x7f313c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:05.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.641+0000 7f315d0b2700 1 -- 172.21.15.1:0/289022444 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f313c06adb0 msgr2=0x7f313c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:05.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.641+0000 7f314dffb700 1 -- 172.21.15.1:0/289022444 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f31500897b0 con 0x7f31580cf960 2024-03-18T19:05:05.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.641+0000 7f315d0b2700 1 --2- 172.21.15.1:0/289022444 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f313c06adb0 0x7f313c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:05.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.645+0000 7f314dffb700 1 -- 172.21.15.1:0/289022444 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f31500546f0 con 0x7f31580cf960 2024-03-18T19:05:05.664 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:05.668+0000 7f6d09147240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:05:05.674 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:05.678+0000 7f420e1eb240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:05:05.736 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:05.739+0000 7f6d09147240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:05:05.814 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:05.817+0000 7f6d09147240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:05:05.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.842+0000 7f315d0b2700 1 -- 172.21.15.1:0/289022444 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f313c06adb0 msgr2=0x7f313c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:05.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.842+0000 7f315d0b2700 1 --2- 172.21.15.1:0/289022444 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f313c06adb0 0x7f313c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:05.861 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:05.866+0000 7f420e1eb240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:05:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.894+0000 7f315f316700 1 -- 172.21.15.1:0/289022444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f315809dcb0 con 0x7f31580cf960 2024-03-18T19:05:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.894+0000 7f314dffb700 1 -- 172.21.15.1:0/289022444 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4731 (secure 0 0 0) 0x7f3150057d10 con 0x7f31580cf960 2024-03-18T19:05:05.890 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:05.890 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5420,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3327835778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":3327835778},{"type":"v1","addr":"172.21.15.165:6835","nonce":3327835778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6016,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/450295183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":450295183},{"type":"v1","addr":"172.21.15.1:6837","nonce":450295183}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6022,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1053303526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1053303526},{"type":"v1","addr":"172.21.15.165:6834","nonce":1053303526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:03:02.534936+0000","modified":"2024-03-18T19:03:05.546799+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6270":{"gid":6270,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6836/1826374081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1826374081},{"type":"v1","addr":"172.21.15.1:6836","nonce":1826374081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-18T19:05:05.891 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:05.894+0000 7f6d09147240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:05:05.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.897+0000 7f315f316700 1 -- 172.21.15.1:0/289022444 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f313c06adb0 msgr2=0x7f313c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:05.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.897+0000 7f315f316700 1 --2- 172.21.15.1:0/289022444 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f313c06adb0 0x7f313c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:05.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.897+0000 7f315f316700 1 -- 172.21.15.1:0/289022444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31580cf960 msgr2=0x7f3158146b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.897+0000 7f315f316700 1 --2- 172.21.15.1:0/289022444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31580cf960 0x7f3158146b30 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f315000cbd0 tx=0x7f315000cee0 comp rx=0 tx=0).stop 2024-03-18T19:05:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.897+0000 7f315f316700 1 -- 172.21.15.1:0/289022444 shutdown_connections 2024-03-18T19:05:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.897+0000 7f315f316700 1 --2- 172.21.15.1:0/289022444 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f313c06adb0 0x7f313c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.897+0000 7f315f316700 1 --2- 172.21.15.1:0/289022444 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f315806a750 0x7f315814a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.897+0000 7f315f316700 1 --2- 172.21.15.1:0/289022444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31580cf960 0x7f3158146b30 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.897+0000 7f315f316700 1 --2- 172.21.15.1:0/289022444 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31580cc6d0 0x7f31581465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.897+0000 7f315f316700 1 -- 172.21.15.1:0/289022444 >> 172.21.15.1:0/289022444 conn(0x7f315806b860 msgr2=0x7f3158158490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.897+0000 7f315f316700 1 -- 172.21.15.1:0/289022444 shutdown_connections 2024-03-18T19:05:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:05.897+0000 7f315f316700 1 -- 172.21.15.1:0/289022444 wait complete. 2024-03-18T19:05:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-03-18T19:05:05.906 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:05:05.939 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:05.943+0000 7f420e1eb240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:05:05.966 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:05.970+0000 7f6d09147240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:05:06.024 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:06.028+0000 7f420e1eb240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:05:06.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.070+0000 7f868ab00700 1 Processor -- start 2024-03-18T19:05:06.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.070+0000 7f868ab00700 1 -- start start 2024-03-18T19:05:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.071+0000 7f868ab00700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f868409a150 0x7f8684166070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.071+0000 7f868ab00700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f868406f070 con 0x7f868415b4d0 2024-03-18T19:05:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.071+0000 7f868ab00700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86840d0260 con 0x7f868409a650 2024-03-18T19:05:06.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.071+0000 7f868ab00700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86840cff80 con 0x7f868409a150 2024-03-18T19:05:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.071+0000 7f868889c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f868415b4d0 0x7f86840cf750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42330/0 (socket says 172.21.15.1:42330) 2024-03-18T19:05:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.071+0000 7f868889c700 1 -- 172.21.15.1:0/2107723697 learned_addr learned my addr 172.21.15.1:0/2107723697 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.071+0000 7f868909d700 1 --2- 172.21.15.1:0/2107723697 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f868409a150 0x7f8684166070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.071+0000 7f86837fe700 1 -- 172.21.15.1:0/2107723697 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484558005 0 0) 0x7f868406f070 con 0x7f868415b4d0 2024-03-18T19:05:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.071+0000 7f86837fe700 1 -- 172.21.15.1:0/2107723697 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86740035b0 con 0x7f868415b4d0 2024-03-18T19:05:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.071+0000 7f86837fe700 1 -- 172.21.15.1:0/2107723697 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761911123 0 0) 0x7f86840d0260 con 0x7f868409a650 2024-03-18T19:05:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.071+0000 7f86837fe700 1 -- 172.21.15.1:0/2107723697 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f868406f070 con 0x7f868409a650 2024-03-18T19:05:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.071+0000 7f868909d700 1 -- 172.21.15.1:0/2107723697 >> v1:172.21.15.165:6789/0 conn(0x7f868409a650 legacy=0x7f8684099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.071+0000 7f868909d700 1 -- 172.21.15.1:0/2107723697 >> v1:172.21.15.1:6789/0 conn(0x7f868415b4d0 legacy=0x7f86840cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.071+0000 7f868909d700 1 -- 172.21.15.1:0/2107723697 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868406ea20 con 0x7f868409a150 2024-03-18T19:05:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.071+0000 7f868909d700 1 --2- 172.21.15.1:0/2107723697 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f868409a150 0x7f8684166070 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f867c009dc0 tx=0x7f867c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a627e863f3771088 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.072+0000 7f86837fe700 1 -- 172.21.15.1:0/2107723697 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c00e070 con 0x7f868409a150 2024-03-18T19:05:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.072+0000 7f86837fe700 1 -- 172.21.15.1:0/2107723697 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867c00bb20 con 0x7f868409a150 2024-03-18T19:05:06.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.072+0000 7f86837fe700 1 -- 172.21.15.1:0/2107723697 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c012430 con 0x7f868409a150 2024-03-18T19:05:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.072+0000 7f868ab00700 1 -- 172.21.15.1:0/2107723697 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f868409a150 msgr2=0x7f8684166070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.072+0000 7f868ab00700 1 --2- 172.21.15.1:0/2107723697 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f868409a150 0x7f8684166070 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f867c009dc0 tx=0x7f867c00b330 comp rx=0 tx=0).stop 2024-03-18T19:05:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.072+0000 7f868ab00700 1 -- 172.21.15.1:0/2107723697 shutdown_connections 2024-03-18T19:05:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.072+0000 7f868ab00700 1 --2- 172.21.15.1:0/2107723697 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f868409a150 0x7f8684166070 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.072+0000 7f868ab00700 1 -- 172.21.15.1:0/2107723697 >> 172.21.15.1:0/2107723697 conn(0x7f86840b2cb0 msgr2=0x7f86840b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.072+0000 7f868ab00700 1 -- 172.21.15.1:0/2107723697 shutdown_connections 2024-03-18T19:05:06.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.072+0000 7f868ab00700 1 -- 172.21.15.1:0/2107723697 wait complete. 2024-03-18T19:05:06.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.073+0000 7f868ab00700 1 Processor -- start 2024-03-18T19:05:06.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.074+0000 7f868ab00700 1 -- start start 2024-03-18T19:05:06.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.074+0000 7f868ab00700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f868409a150 0x7f86840c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:06.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.074+0000 7f868ab00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f868409a650 0x7f86840c6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:06.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.074+0000 7f868ab00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f868415b4d0 0x7f86840c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:06.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.074+0000 7f868ab00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86840c7d70 con 0x7f868415b4d0 2024-03-18T19:05:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.074+0000 7f868ab00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86840b65d0 con 0x7f868409a650 2024-03-18T19:05:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.074+0000 7f868ab00700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86840b59c0 con 0x7f868409a150 2024-03-18T19:05:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.075+0000 7f868909d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f868415b4d0 0x7f86840c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.075+0000 7f868909d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f868415b4d0 0x7f86840c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42180/0 (socket says 172.21.15.1:42180) 2024-03-18T19:05:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.075+0000 7f868909d700 1 -- 172.21.15.1:0/2324072365 learned_addr learned my addr 172.21.15.1:0/2324072365 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.075+0000 7f868889c700 1 --2- 172.21.15.1:0/2324072365 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f868409a150 0x7f86840c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.075+0000 7f8683fff700 1 --2- 172.21.15.1:0/2324072365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f868409a650 0x7f86840c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.075+0000 7f868909d700 1 -- 172.21.15.1:0/2324072365 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f868409a150 msgr2=0x7f86840c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.075+0000 7f868909d700 1 --2- 172.21.15.1:0/2324072365 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f868409a150 0x7f86840c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.075+0000 7f868909d700 1 -- 172.21.15.1:0/2324072365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f868409a650 msgr2=0x7f86840c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.075+0000 7f868909d700 1 --2- 172.21.15.1:0/2324072365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f868409a650 0x7f86840c6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.075+0000 7f868909d700 1 -- 172.21.15.1:0/2324072365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c009a70 con 0x7f868415b4d0 2024-03-18T19:05:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.075+0000 7f868909d700 1 --2- 172.21.15.1:0/2324072365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f868415b4d0 0x7f86840c6dd0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f867c00b370 tx=0x7f867c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.075+0000 7f86817fa700 1 -- 172.21.15.1:0/2324072365 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c01b810 con 0x7f868415b4d0 2024-03-18T19:05:06.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.075+0000 7f86817fa700 1 -- 172.21.15.1:0/2324072365 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867c020070 con 0x7f868415b4d0 2024-03-18T19:05:06.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.075+0000 7f86817fa700 1 -- 172.21.15.1:0/2324072365 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c01bbf0 con 0x7f868415b4d0 2024-03-18T19:05:06.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.075+0000 7f868ab00700 1 -- 172.21.15.1:0/2324072365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86840b5ca0 con 0x7f868415b4d0 2024-03-18T19:05:06.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.075+0000 7f868ab00700 1 -- 172.21.15.1:0/2324072365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86840b6260 con 0x7f868415b4d0 2024-03-18T19:05:06.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.076+0000 7f868ab00700 1 -- 172.21.15.1:0/2324072365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8684068240 con 0x7f868415b4d0 2024-03-18T19:05:06.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.077+0000 7f86817fa700 1 -- 172.21.15.1:0/2324072365 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f867c01bd50 con 0x7f868415b4d0 2024-03-18T19:05:06.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.077+0000 7f86817fa700 1 --2- 172.21.15.1:0/2324072365 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f866806adb0 0x7f866806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:06.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.077+0000 7f868889c700 1 -- 172.21.15.1:0/2324072365 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f866806adb0 msgr2=0x7f866806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:06.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.077+0000 7f86817fa700 1 -- 172.21.15.1:0/2324072365 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f867c022070 con 0x7f868415b4d0 2024-03-18T19:05:06.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.077+0000 7f868889c700 1 --2- 172.21.15.1:0/2324072365 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f866806adb0 0x7f866806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:06.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.082+0000 7f86817fa700 1 -- 172.21.15.1:0/2324072365 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f867c0566a0 con 0x7f868415b4d0 2024-03-18T19:05:06.110 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:06.114+0000 7f420e1eb240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:05:06.137 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:06.141+0000 7f6d09147240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:05:06.193 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:06.198+0000 7f420e1eb240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:05:06.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.278+0000 7f868889c700 1 -- 172.21.15.1:0/2324072365 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f866806adb0 msgr2=0x7f866806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:06.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.278+0000 7f868889c700 1 --2- 172.21.15.1:0/2324072365 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f866806adb0 0x7f866806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:06.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.284+0000 7f868ab00700 1 -- 172.21.15.1:0/2324072365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f868415be50 con 0x7f868415b4d0 2024-03-18T19:05:06.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.286+0000 7f86817fa700 1 -- 172.21.15.1:0/2324072365 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16965 (secure 0 0 0) 0x7f868415be50 con 0x7f868415b4d0 2024-03-18T19:05:06.282 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:06.282 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":76,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:05:04.110472+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:03:00.371114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-18T19:03:01.513069+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":76,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.1d","mappings":[{"from":0,"to":6}]},{"pgid":"8.2f","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6833/76998":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/275404534":"2024-03-19T19:03:17.738629+0000","172.21.15.1:6832/76998":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/1571059650":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/207797678":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/3342330530":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/953259654":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/3431076653":"2024-03-19T19:03:17.738629+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":8,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:05:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.288+0000 7f868ab00700 1 -- 172.21.15.1:0/2324072365 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f866806adb0 msgr2=0x7f866806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.288+0000 7f868ab00700 1 --2- 172.21.15.1:0/2324072365 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f866806adb0 0x7f866806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.288+0000 7f868ab00700 1 -- 172.21.15.1:0/2324072365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f868415b4d0 msgr2=0x7f86840c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.288+0000 7f868ab00700 1 --2- 172.21.15.1:0/2324072365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f868415b4d0 0x7f86840c6dd0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f867c00b370 tx=0x7f867c004ad0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.288+0000 7f868ab00700 1 -- 172.21.15.1:0/2324072365 shutdown_connections 2024-03-18T19:05:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.288+0000 7f868ab00700 1 --2- 172.21.15.1:0/2324072365 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f866806adb0 0x7f866806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.288+0000 7f868ab00700 1 --2- 172.21.15.1:0/2324072365 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f868409a150 0x7f86840c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.288+0000 7f868ab00700 1 --2- 172.21.15.1:0/2324072365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f868415b4d0 0x7f86840c6dd0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.288+0000 7f868ab00700 1 --2- 172.21.15.1:0/2324072365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f868409a650 0x7f86840c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.288+0000 7f868ab00700 1 -- 172.21.15.1:0/2324072365 >> 172.21.15.1:0/2324072365 conn(0x7f86840b2cb0 msgr2=0x7f868415a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:06.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.288+0000 7f868ab00700 1 -- 172.21.15.1:0/2324072365 shutdown_connections 2024-03-18T19:05:06.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.288+0000 7f868ab00700 1 -- 172.21.15.1:0/2324072365 wait complete. 2024-03-18T19:05:06.296 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T19:03:00.371114+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T19:03:01.513069+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 76, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:05:06.297 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:05:06.351 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:06.355+0000 7f6d09147240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:05:06.377 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:06.381+0000 7f420e1eb240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:05:06.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.431+0000 7ffa21367700 1 Processor -- start 2024-03-18T19:05:06.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.431+0000 7ffa21367700 1 -- start start 2024-03-18T19:05:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.432+0000 7ffa21367700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffa1c0ade00 0x7ffa1c0cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.432+0000 7ffa21367700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ffa1c0ccc20 0x7ffa1c06fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.432+0000 7ffa21367700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c0b3aa0 con 0x7ffa1c099a40 2024-03-18T19:05:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.432+0000 7ffa21367700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa1c0ae280 con 0x7ffa1c0ade00 2024-03-18T19:05:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.432+0000 7ffa21367700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa1c06e9b0 con 0x7ffa1c0ccc20 2024-03-18T19:05:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.432+0000 7ffa1a7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffa1c0ade00 0x7ffa1c0cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.432+0000 7ffa1a7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffa1c0ade00 0x7ffa1c0cc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42188/0 (socket says 172.21.15.1:42188) 2024-03-18T19:05:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.432+0000 7ffa1a7fc700 1 -- 172.21.15.1:0/1172628503 learned_addr learned my addr 172.21.15.1:0/1172628503 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.433+0000 7ffa1b7fe700 1 --2- 172.21.15.1:0/1172628503 >> v2:172.21.15.165:3300/0 conn(0x7ffa1c0ccc20 0x7ffa1c06fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.433+0000 7ffa1a7fc700 1 -- 172.21.15.1:0/1172628503 >> v2:172.21.15.165:3300/0 conn(0x7ffa1c0ccc20 msgr2=0x7ffa1c06fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.433+0000 7ffa1a7fc700 1 --2- 172.21.15.1:0/1172628503 >> v2:172.21.15.165:3300/0 conn(0x7ffa1c0ccc20 0x7ffa1c06fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.433+0000 7ffa1a7fc700 1 -- 172.21.15.1:0/1172628503 >> v1:172.21.15.1:6789/0 conn(0x7ffa1c099a40 legacy=0x7ffa1c099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.433+0000 7ffa1a7fc700 1 -- 172.21.15.1:0/1172628503 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1c06ec90 con 0x7ffa1c0ade00 2024-03-18T19:05:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.433+0000 7ffa1b7fe700 1 --2- 172.21.15.1:0/1172628503 >> v2:172.21.15.165:3300/0 conn(0x7ffa1c0ccc20 0x7ffa1c06fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.433+0000 7ffa1a7fc700 1 --2- 172.21.15.1:0/1172628503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa1c0ade00 0x7ffa1c0cc6e0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c067140 tx=0x7ffa10009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906722f004b3490a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.433+0000 7ffa19ffb700 1 -- 172.21.15.1:0/1172628503 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1000b870 con 0x7ffa1c0ade00 2024-03-18T19:05:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.433+0000 7ffa19ffb700 1 -- 172.21.15.1:0/1172628503 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1000b9d0 con 0x7ffa1c0ade00 2024-03-18T19:05:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.433+0000 7ffa19ffb700 1 -- 172.21.15.1:0/1172628503 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1000bd60 con 0x7ffa1c0ade00 2024-03-18T19:05:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.434+0000 7ffa21367700 1 -- 172.21.15.1:0/1172628503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa1c0ade00 msgr2=0x7ffa1c0cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.434+0000 7ffa21367700 1 --2- 172.21.15.1:0/1172628503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa1c0ade00 0x7ffa1c0cc6e0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c067140 tx=0x7ffa10009bd0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.434+0000 7ffa21367700 1 -- 172.21.15.1:0/1172628503 shutdown_connections 2024-03-18T19:05:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.434+0000 7ffa21367700 1 --2- 172.21.15.1:0/1172628503 >> v2:172.21.15.165:3300/0 conn(0x7ffa1c0ccc20 0x7ffa1c06fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.434+0000 7ffa21367700 1 --2- 172.21.15.1:0/1172628503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa1c0ade00 0x7ffa1c0cc6e0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.434+0000 7ffa21367700 1 -- 172.21.15.1:0/1172628503 >> 172.21.15.1:0/1172628503 conn(0x7ffa1c0b2c50 msgr2=0x7ffa1c0b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.434+0000 7ffa21367700 1 -- 172.21.15.1:0/1172628503 shutdown_connections 2024-03-18T19:05:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.434+0000 7ffa21367700 1 -- 172.21.15.1:0/1172628503 wait complete. 2024-03-18T19:05:06.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.435+0000 7ffa21367700 1 Processor -- start 2024-03-18T19:05:06.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.435+0000 7ffa21367700 1 -- start start 2024-03-18T19:05:06.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.436+0000 7ffa21367700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa1c099a40 0x7ffa1c0c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:06.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.436+0000 7ffa21367700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa1c0ade00 0x7ffa1c1f29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:06.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.436+0000 7ffa21367700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa1c0ccc20 0x7ffa1c1f2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:06.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.436+0000 7ffa21367700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c152f60 con 0x7ffa1c0ade00 2024-03-18T19:05:06.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.436+0000 7ffa21367700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c0c9320 con 0x7ffa1c099a40 2024-03-18T19:05:06.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.436+0000 7ffa21367700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c06c980 con 0x7ffa1c0ccc20 2024-03-18T19:05:06.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.436+0000 7ffa1a7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa1c0ade00 0x7ffa1c1f29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:06.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.436+0000 7ffa1a7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa1c0ade00 0x7ffa1c1f29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42194/0 (socket says 172.21.15.1:42194) 2024-03-18T19:05:06.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.436+0000 7ffa1a7fc700 1 -- 172.21.15.1:0/576258493 learned_addr learned my addr 172.21.15.1:0/576258493 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:06.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.436+0000 7ffa1affd700 1 --2- 172.21.15.1:0/576258493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa1c099a40 0x7ffa1c0c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:06.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.436+0000 7ffa1b7fe700 1 --2- 172.21.15.1:0/576258493 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa1c0ccc20 0x7ffa1c1f2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:06.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.437+0000 7ffa1a7fc700 1 -- 172.21.15.1:0/576258493 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa1c0ccc20 msgr2=0x7ffa1c1f2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:06.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.437+0000 7ffa1a7fc700 1 --2- 172.21.15.1:0/576258493 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa1c0ccc20 0x7ffa1c1f2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.437+0000 7ffa1a7fc700 1 -- 172.21.15.1:0/576258493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa1c099a40 msgr2=0x7ffa1c0c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:06.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.437+0000 7ffa1a7fc700 1 --2- 172.21.15.1:0/576258493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa1c099a40 0x7ffa1c0c6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.437+0000 7ffa1a7fc700 1 -- 172.21.15.1:0/576258493 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa10009550 con 0x7ffa1c0ade00 2024-03-18T19:05:06.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.437+0000 7ffa1a7fc700 1 --2- 172.21.15.1:0/576258493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa1c0ade00 0x7ffa1c1f29a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7ffa10000c00 tx=0x7ffa10004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:06.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.437+0000 7ffa037fe700 1 -- 172.21.15.1:0/576258493 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa10004520 con 0x7ffa1c0ade00 2024-03-18T19:05:06.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.437+0000 7ffa037fe700 1 -- 172.21.15.1:0/576258493 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1001e070 con 0x7ffa1c0ade00 2024-03-18T19:05:06.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.437+0000 7ffa037fe700 1 -- 172.21.15.1:0/576258493 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa100038e0 con 0x7ffa1c0ade00 2024-03-18T19:05:06.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.437+0000 7ffa21367700 1 -- 172.21.15.1:0/576258493 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa1c06cc60 con 0x7ffa1c0ade00 2024-03-18T19:05:06.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.437+0000 7ffa21367700 1 -- 172.21.15.1:0/576258493 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1c06d220 con 0x7ffa1c0ade00 2024-03-18T19:05:06.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.438+0000 7ffa21367700 1 -- 172.21.15.1:0/576258493 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa1c0c9520 con 0x7ffa1c0ade00 2024-03-18T19:05:06.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.439+0000 7ffa037fe700 1 -- 172.21.15.1:0/576258493 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7ffa10003a40 con 0x7ffa1c0ade00 2024-03-18T19:05:06.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.439+0000 7ffa037fe700 1 --2- 172.21.15.1:0/576258493 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa0806adb0 0x7ffa0806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:06.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.439+0000 7ffa1affd700 1 -- 172.21.15.1:0/576258493 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa0806adb0 msgr2=0x7ffa0806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:06.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.439+0000 7ffa037fe700 1 -- 172.21.15.1:0/576258493 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6503+0+0 (secure 0 0 0) 0x7ffa10020070 con 0x7ffa1c0ade00 2024-03-18T19:05:06.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.439+0000 7ffa1affd700 1 --2- 172.21.15.1:0/576258493 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa0806adb0 0x7ffa0806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:06.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.444+0000 7ffa037fe700 1 -- 172.21.15.1:0/576258493 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffa1c0c9520 con 0x7ffa1c0ade00 2024-03-18T19:05:06.502 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:06.506+0000 7f6d09147240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:05:06.575 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:06.578+0000 7f6d09147240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:05:06.602 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:06.606+0000 7f420e1eb240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:05:06.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.640+0000 7ffa1affd700 1 -- 172.21.15.1:0/576258493 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa0806adb0 msgr2=0x7ffa0806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:06.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.640+0000 7ffa1affd700 1 --2- 172.21.15.1:0/576258493 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa0806adb0 0x7ffa0806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:06.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.648+0000 7ffa21367700 1 -- 172.21.15.1:0/576258493 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffa1c0c94e0 con 0x7ffa1c0ade00 2024-03-18T19:05:06.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.648+0000 7ffa037fe700 1 -- 172.21.15.1:0/576258493 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffa1002a020 con 0x7ffa1c0ade00 2024-03-18T19:05:06.644 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:05:06.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.650+0000 7ffa21367700 1 -- 172.21.15.1:0/576258493 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa0806adb0 msgr2=0x7ffa0806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:06.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.650+0000 7ffa21367700 1 --2- 172.21.15.1:0/576258493 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa0806adb0 0x7ffa0806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.650+0000 7ffa21367700 1 -- 172.21.15.1:0/576258493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa1c0ade00 msgr2=0x7ffa1c1f29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:06.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.650+0000 7ffa21367700 1 --2- 172.21.15.1:0/576258493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa1c0ade00 0x7ffa1c1f29a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7ffa10000c00 tx=0x7ffa10004750 comp rx=0 tx=0).stop 2024-03-18T19:05:06.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.650+0000 7ffa21367700 1 -- 172.21.15.1:0/576258493 shutdown_connections 2024-03-18T19:05:06.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.650+0000 7ffa21367700 1 --2- 172.21.15.1:0/576258493 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa0806adb0 0x7ffa0806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.650+0000 7ffa21367700 1 --2- 172.21.15.1:0/576258493 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa1c0ccc20 0x7ffa1c1f2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.650+0000 7ffa21367700 1 --2- 172.21.15.1:0/576258493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa1c099a40 0x7ffa1c0c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.650+0000 7ffa21367700 1 --2- 172.21.15.1:0/576258493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa1c0ade00 0x7ffa1c1f29a0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.650+0000 7ffa21367700 1 -- 172.21.15.1:0/576258493 >> 172.21.15.1:0/576258493 conn(0x7ffa1c0b2c50 msgr2=0x7ffa1c15a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:06.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.651+0000 7ffa21367700 1 -- 172.21.15.1:0/576258493 shutdown_connections 2024-03-18T19:05:06.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.651+0000 7ffa21367700 1 -- 172.21.15.1:0/576258493 wait complete. 2024-03-18T19:05:06.660 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T19:05:06.767 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:06.771+0000 7f420e1eb240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:05:06.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.793+0000 7fac7d5d1700 1 Processor -- start 2024-03-18T19:05:06.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.793+0000 7fac7d5d1700 1 -- start start 2024-03-18T19:05:06.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.794+0000 7fac7d5d1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fac7815b6c0 0x7fac780b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:06.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.794+0000 7fac7d5d1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac780c87d0 0x7fac780cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:06.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.794+0000 7fac7d5d1700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac780d01f0 con 0x7fac780cd430 2024-03-18T19:05:06.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.794+0000 7fac7d5d1700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac780ccfa0 con 0x7fac7815b6c0 2024-03-18T19:05:06.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.794+0000 7fac7d5d1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac780d1db0 con 0x7fac780c87d0 2024-03-18T19:05:06.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.794+0000 7fac777fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac780c87d0 0x7fac780cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:06.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.794+0000 7fac76ffd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fac7815b6c0 0x7fac780b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:06.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.794+0000 7fac767fc700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fac780cd430 0x7fac780c80c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:35774/0 (socket says 172.21.15.1:35774) 2024-03-18T19:05:06.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.794+0000 7fac767fc700 1 -- 172.21.15.1:0/3981766256 learned_addr learned my addr 172.21.15.1:0/3981766256 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:06.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.795+0000 7fac76ffd700 1 -- 172.21.15.1:0/3981766256 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac780c87d0 msgr2=0x7fac780cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:06.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.795+0000 7fac76ffd700 1 --2- 172.21.15.1:0/3981766256 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac780c87d0 0x7fac780cd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.795+0000 7fac76ffd700 1 -- 172.21.15.1:0/3981766256 >> v1:172.21.15.165:6789/0 conn(0x7fac780cd430 legacy=0x7fac780c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:06.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.795+0000 7fac76ffd700 1 -- 172.21.15.1:0/3981766256 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac780cff10 con 0x7fac7815b6c0 2024-03-18T19:05:06.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.795+0000 7fac75ffb700 1 -- 172.21.15.1:0/3981766256 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295899326 0 0) 0x7fac780d01f0 con 0x7fac780cd430 2024-03-18T19:05:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.795+0000 7fac777fe700 1 --2- 172.21.15.1:0/3981766256 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac780c87d0 0x7fac780cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.795+0000 7fac76ffd700 1 --2- 172.21.15.1:0/3981766256 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac7815b6c0 0x7fac780b2160 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fac68009840 tx=0x7fac68009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c34cf420d52e8bb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.795+0000 7fac75ffb700 1 -- 172.21.15.1:0/3981766256 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac68011830 con 0x7fac7815b6c0 2024-03-18T19:05:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.795+0000 7fac75ffb700 1 -- 172.21.15.1:0/3981766256 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac68011990 con 0x7fac7815b6c0 2024-03-18T19:05:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.795+0000 7fac75ffb700 1 -- 172.21.15.1:0/3981766256 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac68011d20 con 0x7fac7815b6c0 2024-03-18T19:05:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.796+0000 7fac7d5d1700 1 -- 172.21.15.1:0/3981766256 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac7815b6c0 msgr2=0x7fac780b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.796+0000 7fac7d5d1700 1 --2- 172.21.15.1:0/3981766256 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac7815b6c0 0x7fac780b2160 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fac68009840 tx=0x7fac68009b50 comp rx=0 tx=0).stop 2024-03-18T19:05:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.796+0000 7fac7d5d1700 1 -- 172.21.15.1:0/3981766256 shutdown_connections 2024-03-18T19:05:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.796+0000 7fac7d5d1700 1 --2- 172.21.15.1:0/3981766256 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac780c87d0 0x7fac780cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.796+0000 7fac7d5d1700 1 --2- 172.21.15.1:0/3981766256 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac7815b6c0 0x7fac780b2160 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.796+0000 7fac7d5d1700 1 -- 172.21.15.1:0/3981766256 >> 172.21.15.1:0/3981766256 conn(0x7fac780d0ff0 msgr2=0x7fac780d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.796+0000 7fac7d5d1700 1 -- 172.21.15.1:0/3981766256 shutdown_connections 2024-03-18T19:05:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.796+0000 7fac7d5d1700 1 -- 172.21.15.1:0/3981766256 wait complete. 2024-03-18T19:05:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.797+0000 7fac7d5d1700 1 Processor -- start 2024-03-18T19:05:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.797+0000 7fac7d5d1700 1 -- start start 2024-03-18T19:05:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.798+0000 7fac7d5d1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac780c87d0 0x7fac780b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.798+0000 7fac7d5d1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac780cd430 0x7fac780ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.798+0000 7fac7d5d1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac7815b6c0 0x7fac781545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.798+0000 7fac7d5d1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac780aebb0 con 0x7fac780c87d0 2024-03-18T19:05:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.798+0000 7fac7d5d1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac780b1230 con 0x7fac780cd430 2024-03-18T19:05:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.798+0000 7fac7d5d1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac780b1510 con 0x7fac7815b6c0 2024-03-18T19:05:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.798+0000 7fac76ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac780c87d0 0x7fac780b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.798+0000 7fac76ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac780c87d0 0x7fac780b0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42206/0 (socket says 172.21.15.1:42206) 2024-03-18T19:05:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.798+0000 7fac76ffd700 1 -- 172.21.15.1:0/85897214 learned_addr learned my addr 172.21.15.1:0/85897214 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:06.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.798+0000 7fac767fc700 1 --2- 172.21.15.1:0/85897214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac780cd430 0x7fac780ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:06.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.798+0000 7fac777fe700 1 --2- 172.21.15.1:0/85897214 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac7815b6c0 0x7fac781545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:06.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.798+0000 7fac76ffd700 1 -- 172.21.15.1:0/85897214 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac7815b6c0 msgr2=0x7fac781545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:06.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.798+0000 7fac76ffd700 1 --2- 172.21.15.1:0/85897214 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac7815b6c0 0x7fac781545c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.798+0000 7fac76ffd700 1 -- 172.21.15.1:0/85897214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac780cd430 msgr2=0x7fac780ae670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:06.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.798+0000 7fac76ffd700 1 --2- 172.21.15.1:0/85897214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac780cd430 0x7fac780ae670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:06.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.798+0000 7fac76ffd700 1 -- 172.21.15.1:0/85897214 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac70009a70 con 0x7fac780c87d0 2024-03-18T19:05:06.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.798+0000 7fac76ffd700 1 --2- 172.21.15.1:0/85897214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac780c87d0 0x7fac780b0b30 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fac68009fd0 tx=0x7fac68004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:06.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.799+0000 7fac5f7fe700 1 -- 172.21.15.1:0/85897214 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac680045d0 con 0x7fac780c87d0 2024-03-18T19:05:06.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.799+0000 7fac5f7fe700 1 -- 172.21.15.1:0/85897214 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6801f070 con 0x7fac780c87d0 2024-03-18T19:05:06.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.799+0000 7fac7d5d1700 1 -- 172.21.15.1:0/85897214 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac680094a0 con 0x7fac780c87d0 2024-03-18T19:05:06.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.799+0000 7fac5f7fe700 1 -- 172.21.15.1:0/85897214 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac680038a0 con 0x7fac780c87d0 2024-03-18T19:05:06.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.799+0000 7fac7d5d1700 1 -- 172.21.15.1:0/85897214 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac780b1ad0 con 0x7fac780c87d0 2024-03-18T19:05:06.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.800+0000 7fac7d5d1700 1 -- 172.21.15.1:0/85897214 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac780676a0 con 0x7fac780c87d0 2024-03-18T19:05:06.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.800+0000 7fac5f7fe700 1 -- 172.21.15.1:0/85897214 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fac68003a00 con 0x7fac780c87d0 2024-03-18T19:05:06.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.800+0000 7fac5f7fe700 1 --2- 172.21.15.1:0/85897214 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac6406adb0 0x7fac6406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:06.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.801+0000 7fac767fc700 1 -- 172.21.15.1:0/85897214 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac6406adb0 msgr2=0x7fac6406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:06.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.801+0000 7fac5f7fe700 1 -- 172.21.15.1:0/85897214 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6503+0+0 (secure 0 0 0) 0x7fac68021070 con 0x7fac780c87d0 2024-03-18T19:05:06.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.801+0000 7fac767fc700 1 --2- 172.21.15.1:0/85897214 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac6406adb0 0x7fac6406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:06.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:06.805+0000 7fac5f7fe700 1 -- 172.21.15.1:0/85897214 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fac68055e30 con 0x7fac780c87d0 2024-03-18T19:05:06.857 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:06.862+0000 7f420e1eb240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:05:06.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.002+0000 7fac767fc700 1 -- 172.21.15.1:0/85897214 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac6406adb0 msgr2=0x7fac6406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:06.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.002+0000 7fac767fc700 1 --2- 172.21.15.1:0/85897214 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac6406adb0 0x7fac6406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:07.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.017+0000 7fac7d5d1700 1 -- 172.21.15.1:0/85897214 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fac780c5330 con 0x7fac780c87d0 2024-03-18T19:05:07.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.017+0000 7fac5f7fe700 1 -- 172.21.15.1:0/85897214 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v76) v1 ==== 104+0+11 (secure 0 0 0) 0x7fac68015020 con 0x7fac780c87d0 2024-03-18T19:05:07.013 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T19:05:07.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.020+0000 7fac7d5d1700 1 -- 172.21.15.1:0/85897214 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac6406adb0 msgr2=0x7fac6406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:07.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.020+0000 7fac7d5d1700 1 --2- 172.21.15.1:0/85897214 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac6406adb0 0x7fac6406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.020+0000 7fac7d5d1700 1 -- 172.21.15.1:0/85897214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac780c87d0 msgr2=0x7fac780b0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:07.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.020+0000 7fac7d5d1700 1 --2- 172.21.15.1:0/85897214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac780c87d0 0x7fac780b0b30 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fac68009fd0 tx=0x7fac68004750 comp rx=0 tx=0).stop 2024-03-18T19:05:07.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.020+0000 7fac7d5d1700 1 -- 172.21.15.1:0/85897214 shutdown_connections 2024-03-18T19:05:07.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.020+0000 7fac7d5d1700 1 --2- 172.21.15.1:0/85897214 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac6406adb0 0x7fac6406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.020+0000 7fac7d5d1700 1 --2- 172.21.15.1:0/85897214 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac7815b6c0 0x7fac781545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.020+0000 7fac7d5d1700 1 --2- 172.21.15.1:0/85897214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac780c87d0 0x7fac780b0b30 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.020+0000 7fac7d5d1700 1 --2- 172.21.15.1:0/85897214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac780cd430 0x7fac780ae670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.020+0000 7fac7d5d1700 1 -- 172.21.15.1:0/85897214 >> 172.21.15.1:0/85897214 conn(0x7fac780d0ff0 msgr2=0x7fac780c8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:07.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.020+0000 7fac7d5d1700 1 -- 172.21.15.1:0/85897214 shutdown_connections 2024-03-18T19:05:07.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.020+0000 7fac7d5d1700 1 -- 172.21.15.1:0/85897214 wait complete. 2024-03-18T19:05:07.020 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:07.024+0000 7f6d09147240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:05:07.028 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T19:05:07.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.163+0000 7fc170cc1700 1 Processor -- start 2024-03-18T19:05:07.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.163+0000 7fc170cc1700 1 -- start start 2024-03-18T19:05:07.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.164+0000 7fc170cc1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc16c15d3b0 0x7fc16c06cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:07.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.164+0000 7fc170cc1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc16c071550 0x7fc16c0b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:07.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.164+0000 7fc170cc1700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc16c0b2b80 con 0x7fc16c06d430 2024-03-18T19:05:07.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.164+0000 7fc170cc1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc16c0b3800 con 0x7fc16c15d3b0 2024-03-18T19:05:07.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.164+0000 7fc170cc1700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc16c0b2e60 con 0x7fc16c071550 2024-03-18T19:05:07.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.164+0000 7fc16a59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc16c15d3b0 0x7fc16c06cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:07.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.164+0000 7fc16a59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc16c15d3b0 0x7fc16c06cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42216/0 (socket says 172.21.15.1:42216) 2024-03-18T19:05:07.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.164+0000 7fc16a59c700 1 -- 172.21.15.1:0/1499581683 learned_addr learned my addr 172.21.15.1:0/1499581683 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:07.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.164+0000 7fc16ad9d700 1 --2- 172.21.15.1:0/1499581683 >> v2:172.21.15.165:3300/0 conn(0x7fc16c071550 0x7fc16c0b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:07.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.164+0000 7fc16a59c700 1 -- 172.21.15.1:0/1499581683 >> v2:172.21.15.165:3300/0 conn(0x7fc16c071550 msgr2=0x7fc16c0b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:07.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.164+0000 7fc16a59c700 1 --2- 172.21.15.1:0/1499581683 >> v2:172.21.15.165:3300/0 conn(0x7fc16c071550 0x7fc16c0b3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.164+0000 7fc16a59c700 1 -- 172.21.15.1:0/1499581683 >> v1:172.21.15.1:6789/0 conn(0x7fc16c06d430 legacy=0x7fc16c070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:07.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.164+0000 7fc16a59c700 1 -- 172.21.15.1:0/1499581683 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16c0b3140 con 0x7fc16c15d3b0 2024-03-18T19:05:07.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.165+0000 7fc16a59c700 1 --2- 172.21.15.1:0/1499581683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc16c15d3b0 0x7fc16c06cef0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fc16c0ae1b0 tx=0x7fc15c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9c659bc335fa22 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:07.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.165+0000 7fc16959a700 1 -- 172.21.15.1:0/1499581683 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c011830 con 0x7fc16c15d3b0 2024-03-18T19:05:07.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.165+0000 7fc16959a700 1 -- 172.21.15.1:0/1499581683 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15c011990 con 0x7fc16c15d3b0 2024-03-18T19:05:07.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.165+0000 7fc16959a700 1 -- 172.21.15.1:0/1499581683 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c011d20 con 0x7fc16c15d3b0 2024-03-18T19:05:07.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.165+0000 7fc170cc1700 1 -- 172.21.15.1:0/1499581683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc16c15d3b0 msgr2=0x7fc16c06cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:07.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.165+0000 7fc170cc1700 1 --2- 172.21.15.1:0/1499581683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc16c15d3b0 0x7fc16c06cef0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fc16c0ae1b0 tx=0x7fc15c009bd0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.166+0000 7fc170cc1700 1 -- 172.21.15.1:0/1499581683 shutdown_connections 2024-03-18T19:05:07.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.166+0000 7fc170cc1700 1 --2- 172.21.15.1:0/1499581683 >> v2:172.21.15.165:3300/0 conn(0x7fc16c071550 0x7fc16c0b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.166+0000 7fc170cc1700 1 --2- 172.21.15.1:0/1499581683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc16c15d3b0 0x7fc16c06cef0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.166+0000 7fc170cc1700 1 -- 172.21.15.1:0/1499581683 >> 172.21.15.1:0/1499581683 conn(0x7fc16c0b1350 msgr2=0x7fc16c0b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:07.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.166+0000 7fc170cc1700 1 -- 172.21.15.1:0/1499581683 shutdown_connections 2024-03-18T19:05:07.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.166+0000 7fc170cc1700 1 -- 172.21.15.1:0/1499581683 wait complete. 2024-03-18T19:05:07.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.167+0000 7fc170cc1700 1 Processor -- start 2024-03-18T19:05:07.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.167+0000 7fc170cc1700 1 -- start start 2024-03-18T19:05:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.168+0000 7fc170cc1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc16c06d430 0x7fc16c1334e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.168+0000 7fc170cc1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc16c071550 0x7fc16c1f2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.168+0000 7fc170cc1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc16c15d3b0 0x7fc16c135370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.168+0000 7fc170cc1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c1330c0 con 0x7fc16c15d3b0 2024-03-18T19:05:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.168+0000 7fc170cc1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c1452a0 con 0x7fc16c071550 2024-03-18T19:05:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.168+0000 7fc170cc1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c132440 con 0x7fc16c06d430 2024-03-18T19:05:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.168+0000 7fc16ad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc16c15d3b0 0x7fc16c135370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.168+0000 7fc16ad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc16c15d3b0 0x7fc16c135370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42228/0 (socket says 172.21.15.1:42228) 2024-03-18T19:05:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.168+0000 7fc16ad9d700 1 -- 172.21.15.1:0/3575386257 learned_addr learned my addr 172.21.15.1:0/3575386257 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.169+0000 7fc16a59c700 1 --2- 172.21.15.1:0/3575386257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc16c06d430 0x7fc16c1334e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.169+0000 7fc169d9b700 1 --2- 172.21.15.1:0/3575386257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc16c071550 0x7fc16c1f2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.169+0000 7fc16ad9d700 1 -- 172.21.15.1:0/3575386257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc16c06d430 msgr2=0x7fc16c1334e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.169+0000 7fc16ad9d700 1 --2- 172.21.15.1:0/3575386257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc16c06d430 0x7fc16c1334e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.169+0000 7fc16ad9d700 1 -- 172.21.15.1:0/3575386257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc16c071550 msgr2=0x7fc16c1f2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.169+0000 7fc16ad9d700 1 --2- 172.21.15.1:0/3575386257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc16c071550 0x7fc16c1f2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.169+0000 7fc16ad9d700 1 -- 172.21.15.1:0/3575386257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15c009550 con 0x7fc16c15d3b0 2024-03-18T19:05:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.169+0000 7fc16ad9d700 1 --2- 172.21.15.1:0/3575386257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc16c15d3b0 0x7fc16c135370 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fc16400ef90 tx=0x7fc16400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.169+0000 7fc15affd700 1 -- 172.21.15.1:0/3575386257 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16400cc90 con 0x7fc16c15d3b0 2024-03-18T19:05:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.169+0000 7fc15affd700 1 -- 172.21.15.1:0/3575386257 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc16400f040 con 0x7fc16c15d3b0 2024-03-18T19:05:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.169+0000 7fc170cc1700 1 -- 172.21.15.1:0/3575386257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc16c132720 con 0x7fc16c15d3b0 2024-03-18T19:05:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.169+0000 7fc15affd700 1 -- 172.21.15.1:0/3575386257 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc164014480 con 0x7fc16c15d3b0 2024-03-18T19:05:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.169+0000 7fc170cc1700 1 -- 172.21.15.1:0/3575386257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc16c132ce0 con 0x7fc16c15d3b0 2024-03-18T19:05:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.170+0000 7fc170cc1700 1 -- 172.21.15.1:0/3575386257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc16c133200 con 0x7fc16c15d3b0 2024-03-18T19:05:07.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.171+0000 7fc15affd700 1 -- 172.21.15.1:0/3575386257 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fc1640145e0 con 0x7fc16c15d3b0 2024-03-18T19:05:07.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.171+0000 7fc15affd700 1 --2- 172.21.15.1:0/3575386257 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc14c06adb0 0x7fc14c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:07.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.171+0000 7fc16a59c700 1 -- 172.21.15.1:0/3575386257 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc14c06adb0 msgr2=0x7fc14c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:07.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.171+0000 7fc15affd700 1 -- 172.21.15.1:0/3575386257 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6503+0+0 (secure 0 0 0) 0x7fc16408a400 con 0x7fc16c15d3b0 2024-03-18T19:05:07.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.171+0000 7fc16a59c700 1 --2- 172.21.15.1:0/3575386257 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc14c06adb0 0x7fc14c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:07.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.176+0000 7fc15affd700 1 -- 172.21.15.1:0/3575386257 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc164055110 con 0x7fc16c15d3b0 2024-03-18T19:05:07.357 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:07.361+0000 7f6d09147240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:05:07.361 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:07.366+0000 7f420e1eb240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:05:07.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.372+0000 7fc16a59c700 1 -- 172.21.15.1:0/3575386257 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc14c06adb0 msgr2=0x7fc14c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:07.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.372+0000 7fc16a59c700 1 --2- 172.21.15.1:0/3575386257 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc14c06adb0 0x7fc14c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:07.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.399+0000 7fc170cc1700 1 -- 172.21.15.1:0/3575386257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc16c0676a0 con 0x7fc16c15d3b0 2024-03-18T19:05:07.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.399+0000 7fc15affd700 1 -- 172.21.15.1:0/3575386257 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v76) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc164009ca0 con 0x7fc16c15d3b0 2024-03-18T19:05:07.395 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T19:05:07.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.402+0000 7fc170cc1700 1 -- 172.21.15.1:0/3575386257 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc14c06adb0 msgr2=0x7fc14c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:07.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.402+0000 7fc170cc1700 1 --2- 172.21.15.1:0/3575386257 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc14c06adb0 0x7fc14c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.402+0000 7fc170cc1700 1 -- 172.21.15.1:0/3575386257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc16c15d3b0 msgr2=0x7fc16c135370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:07.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.402+0000 7fc170cc1700 1 --2- 172.21.15.1:0/3575386257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc16c15d3b0 0x7fc16c135370 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fc16400ef90 tx=0x7fc16400c3a0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.402+0000 7fc170cc1700 1 -- 172.21.15.1:0/3575386257 shutdown_connections 2024-03-18T19:05:07.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.402+0000 7fc170cc1700 1 --2- 172.21.15.1:0/3575386257 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc14c06adb0 0x7fc14c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.402+0000 7fc170cc1700 1 --2- 172.21.15.1:0/3575386257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc16c06d430 0x7fc16c1334e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.402+0000 7fc170cc1700 1 --2- 172.21.15.1:0/3575386257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc16c15d3b0 0x7fc16c135370 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.402+0000 7fc170cc1700 1 --2- 172.21.15.1:0/3575386257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc16c071550 0x7fc16c1f2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.402+0000 7fc170cc1700 1 -- 172.21.15.1:0/3575386257 >> 172.21.15.1:0/3575386257 conn(0x7fc16c0b1350 msgr2=0x7fc16c070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:07.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.402+0000 7fc170cc1700 1 -- 172.21.15.1:0/3575386257 shutdown_connections 2024-03-18T19:05:07.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.402+0000 7fc170cc1700 1 -- 172.21.15.1:0/3575386257 wait complete. 2024-03-18T19:05:07.410 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:05:07.517 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:07.521+0000 7f6d09147240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:05:07.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.545+0000 7fd194030700 1 Processor -- start 2024-03-18T19:05:07.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.545+0000 7fd194030700 1 -- start start 2024-03-18T19:05:07.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.546+0000 7fd194030700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd18c160940 0x7fd18c0d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.546+0000 7fd194030700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd18c0cf090 0x7fd18c0d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.546+0000 7fd194030700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd18c160d20 0x7fd18c0d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.546+0000 7fd194030700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd18c0cddc0 con 0x7fd18c0cf090 2024-03-18T19:05:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.546+0000 7fd194030700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd18c0cdff0 con 0x7fd18c160940 2024-03-18T19:05:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.546+0000 7fd194030700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c0ce2d0 con 0x7fd18c160d20 2024-03-18T19:05:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.546+0000 7fd1915cb700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd18c0cf090 0x7fd18c0d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.546+0000 7fd1915cb700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd18c0cf090 0x7fd18c0d1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42238/0 (socket says 172.21.15.1:42238) 2024-03-18T19:05:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.546+0000 7fd1915cb700 1 -- 172.21.15.1:0/1479248076 learned_addr learned my addr 172.21.15.1:0/1479248076 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.546+0000 7fd1925cd700 1 --2- 172.21.15.1:0/1479248076 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd18c160d20 0x7fd18c0d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.546+0000 7fd191dcc700 1 --2- 172.21.15.1:0/1479248076 >> v2:172.21.15.165:3300/0 conn(0x7fd18c160940 0x7fd18c0d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.547+0000 7fd1915cb700 1 -- 172.21.15.1:0/1479248076 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd18c160d20 msgr2=0x7fd18c0d1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:07.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.547+0000 7fd1915cb700 1 --2- 172.21.15.1:0/1479248076 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd18c160d20 0x7fd18c0d1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.547+0000 7fd1915cb700 1 -- 172.21.15.1:0/1479248076 >> v2:172.21.15.165:3300/0 conn(0x7fd18c160940 msgr2=0x7fd18c0d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:07.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.547+0000 7fd1915cb700 1 --2- 172.21.15.1:0/1479248076 >> v2:172.21.15.165:3300/0 conn(0x7fd18c160940 0x7fd18c0d0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.547+0000 7fd1915cb700 1 -- 172.21.15.1:0/1479248076 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18c0ce5b0 con 0x7fd18c0cf090 2024-03-18T19:05:07.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.547+0000 7fd1915cb700 1 --2- 172.21.15.1:0/1479248076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd18c0cf090 0x7fd18c0d1790 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fd18c06b0e0 tx=0x7fd17c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e6ba629e125e6ff server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:07.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.547+0000 7fd190dca700 1 -- 172.21.15.1:0/1479248076 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c011830 con 0x7fd18c0cf090 2024-03-18T19:05:07.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.547+0000 7fd190dca700 1 -- 172.21.15.1:0/1479248076 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17c011990 con 0x7fd18c0cf090 2024-03-18T19:05:07.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.547+0000 7fd190dca700 1 -- 172.21.15.1:0/1479248076 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c011d20 con 0x7fd18c0cf090 2024-03-18T19:05:07.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.547+0000 7fd194030700 1 -- 172.21.15.1:0/1479248076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd18c0cf090 msgr2=0x7fd18c0d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:07.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.547+0000 7fd194030700 1 --2- 172.21.15.1:0/1479248076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd18c0cf090 0x7fd18c0d1790 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fd18c06b0e0 tx=0x7fd17c009bd0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.548+0000 7fd194030700 1 -- 172.21.15.1:0/1479248076 shutdown_connections 2024-03-18T19:05:07.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.548+0000 7fd194030700 1 --2- 172.21.15.1:0/1479248076 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd18c160d20 0x7fd18c0d1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.548+0000 7fd194030700 1 --2- 172.21.15.1:0/1479248076 >> v2:172.21.15.165:3300/0 conn(0x7fd18c160940 0x7fd18c0d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.548+0000 7fd194030700 1 --2- 172.21.15.1:0/1479248076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd18c0cf090 0x7fd18c0d1790 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.548+0000 7fd194030700 1 -- 172.21.15.1:0/1479248076 >> 172.21.15.1:0/1479248076 conn(0x7fd18c09c520 msgr2=0x7fd18c0d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:07.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.548+0000 7fd194030700 1 -- 172.21.15.1:0/1479248076 shutdown_connections 2024-03-18T19:05:07.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.548+0000 7fd194030700 1 -- 172.21.15.1:0/1479248076 wait complete. 2024-03-18T19:05:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.549+0000 7fd194030700 1 Processor -- start 2024-03-18T19:05:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.549+0000 7fd194030700 1 -- start start 2024-03-18T19:05:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.550+0000 7fd194030700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd18c0cf090 0x7fd18c0b5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.550+0000 7fd194030700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd18c160940 0x7fd18c0b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.550+0000 7fd194030700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd18c160d20 0x7fd18c151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.550+0000 7fd194030700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c0ca280 con 0x7fd18c160940 2024-03-18T19:05:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.550+0000 7fd194030700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c0b64f0 con 0x7fd18c160d20 2024-03-18T19:05:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.550+0000 7fd194030700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c0ca4d0 con 0x7fd18c0cf090 2024-03-18T19:05:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.550+0000 7fd1915cb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd18c160940 0x7fd18c0b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.550+0000 7fd1915cb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd18c160940 0x7fd18c0b07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42250/0 (socket says 172.21.15.1:42250) 2024-03-18T19:05:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.550+0000 7fd1915cb700 1 -- 172.21.15.1:0/3653262925 learned_addr learned my addr 172.21.15.1:0/3653262925 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.550+0000 7fd191dcc700 1 --2- 172.21.15.1:0/3653262925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd18c0cf090 0x7fd18c0b5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.550+0000 7fd1925cd700 1 --2- 172.21.15.1:0/3653262925 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd18c160d20 0x7fd18c151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.551+0000 7fd1915cb700 1 -- 172.21.15.1:0/3653262925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd18c0cf090 msgr2=0x7fd18c0b5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.551+0000 7fd1915cb700 1 --2- 172.21.15.1:0/3653262925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd18c0cf090 0x7fd18c0b5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.551+0000 7fd1915cb700 1 -- 172.21.15.1:0/3653262925 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd18c160d20 msgr2=0x7fd18c151d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.551+0000 7fd1915cb700 1 --2- 172.21.15.1:0/3653262925 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd18c160d20 0x7fd18c151d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.551+0000 7fd1915cb700 1 -- 172.21.15.1:0/3653262925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c009550 con 0x7fd18c160940 2024-03-18T19:05:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.551+0000 7fd1915cb700 1 --2- 172.21.15.1:0/3653262925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd18c160940 0x7fd18c0b07e0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fd17c000c00 tx=0x7fd17c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.551+0000 7fd1827fc700 1 -- 172.21.15.1:0/3653262925 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c004240 con 0x7fd18c160940 2024-03-18T19:05:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.551+0000 7fd1827fc700 1 -- 172.21.15.1:0/3653262925 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17c01f070 con 0x7fd18c160940 2024-03-18T19:05:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.551+0000 7fd1827fc700 1 -- 172.21.15.1:0/3653262925 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c0036a0 con 0x7fd18c160940 2024-03-18T19:05:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.551+0000 7fd194030700 1 -- 172.21.15.1:0/3653262925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18c0ca7b0 con 0x7fd18c160940 2024-03-18T19:05:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.551+0000 7fd194030700 1 -- 172.21.15.1:0/3653262925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18c0cad70 con 0x7fd18c160940 2024-03-18T19:05:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.552+0000 7fd194030700 1 -- 172.21.15.1:0/3653262925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd18c073350 con 0x7fd18c160940 2024-03-18T19:05:07.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.553+0000 7fd1827fc700 1 -- 172.21.15.1:0/3653262925 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fd17c021030 con 0x7fd18c160940 2024-03-18T19:05:07.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.553+0000 7fd1827fc700 1 --2- 172.21.15.1:0/3653262925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd17006ade0 0x7fd17006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:07.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.553+0000 7fd191dcc700 1 -- 172.21.15.1:0/3653262925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd17006ade0 msgr2=0x7fd17006d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:07.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.553+0000 7fd1827fc700 1 -- 172.21.15.1:0/3653262925 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6503+0+0 (secure 0 0 0) 0x7fd17c08a830 con 0x7fd18c160940 2024-03-18T19:05:07.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.553+0000 7fd191dcc700 1 --2- 172.21.15.1:0/3653262925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd17006ade0 0x7fd17006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:07.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.558+0000 7fd1827fc700 1 -- 172.21.15.1:0/3653262925 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd17c055670 con 0x7fd18c160940 2024-03-18T19:05:07.720 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:07.724+0000 7f420e1eb240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:05:07.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.754+0000 7fd191dcc700 1 -- 172.21.15.1:0/3653262925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd17006ade0 msgr2=0x7fd17006d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:07.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.754+0000 7fd191dcc700 1 --2- 172.21.15.1:0/3653262925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd17006ade0 0x7fd17006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:07.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.806+0000 7fd194030700 1 -- 172.21.15.1:0/3653262925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd18c0d0980 con 0x7fd18c160940 2024-03-18T19:05:07.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.807+0000 7fd1827fc700 1 -- 172.21.15.1:0/3653262925 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4731 (secure 0 0 0) 0x7fd17c088e60 con 0x7fd18c160940 2024-03-18T19:05:07.803 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:07.803 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5420,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3327835778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":3327835778},{"type":"v1","addr":"172.21.15.165:6835","nonce":3327835778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6016,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/450295183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":450295183},{"type":"v1","addr":"172.21.15.1:6837","nonce":450295183}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6022,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1053303526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1053303526},{"type":"v1","addr":"172.21.15.165:6834","nonce":1053303526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:03:02.534936+0000","modified":"2024-03-18T19:03:05.546799+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6270":{"gid":6270,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6836/1826374081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1826374081},{"type":"v1","addr":"172.21.15.1:6836","nonce":1826374081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-18T19:05:07.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.809+0000 7fd194030700 1 -- 172.21.15.1:0/3653262925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd17006ade0 msgr2=0x7fd17006d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:07.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.809+0000 7fd194030700 1 --2- 172.21.15.1:0/3653262925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd17006ade0 0x7fd17006d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.809+0000 7fd194030700 1 -- 172.21.15.1:0/3653262925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd18c160940 msgr2=0x7fd18c0b07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:07.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.809+0000 7fd194030700 1 --2- 172.21.15.1:0/3653262925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd18c160940 0x7fd18c0b07e0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fd17c000c00 tx=0x7fd17c004750 comp rx=0 tx=0).stop 2024-03-18T19:05:07.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.809+0000 7fd194030700 1 -- 172.21.15.1:0/3653262925 shutdown_connections 2024-03-18T19:05:07.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.809+0000 7fd194030700 1 --2- 172.21.15.1:0/3653262925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd17006ade0 0x7fd17006d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.809+0000 7fd194030700 1 --2- 172.21.15.1:0/3653262925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd18c0cf090 0x7fd18c0b5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.809+0000 7fd194030700 1 --2- 172.21.15.1:0/3653262925 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd18c160d20 0x7fd18c151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.809+0000 7fd194030700 1 --2- 172.21.15.1:0/3653262925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd18c160940 0x7fd18c0b07e0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.809+0000 7fd194030700 1 -- 172.21.15.1:0/3653262925 >> 172.21.15.1:0/3653262925 conn(0x7fd18c09c520 msgr2=0x7fd18c15c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:07.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.810+0000 7fd194030700 1 -- 172.21.15.1:0/3653262925 shutdown_connections 2024-03-18T19:05:07.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.810+0000 7fd194030700 1 -- 172.21.15.1:0/3653262925 wait complete. 2024-03-18T19:05:07.807 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-03-18T19:05:07.818 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:05:07.896 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:07.900+0000 7f420e1eb240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:05:07.902 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:07.906+0000 7f6d09147240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:05:07.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.948+0000 7f87f3fd1700 1 Processor -- start 2024-03-18T19:05:07.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.949+0000 7f87f3fd1700 1 -- start start 2024-03-18T19:05:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.949+0000 7f87f3fd1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f87ec15b4d0 0x7f87ec09a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.949+0000 7f87f3fd1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87ec09a150 0x7f87ec06f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.949+0000 7f87f3fd1700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87ec06f070 con 0x7f87ec0cf750 2024-03-18T19:05:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.949+0000 7f87f3fd1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87ec0d0260 con 0x7f87ec09a150 2024-03-18T19:05:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.949+0000 7f87f3fd1700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87ec0cff80 con 0x7f87ec15b4d0 2024-03-18T19:05:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.950+0000 7f87f256e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87ec09a150 0x7f87ec06f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:07.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.950+0000 7f87f256e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87ec09a150 0x7f87ec06f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42258/0 (socket says 172.21.15.1:42258) 2024-03-18T19:05:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.950+0000 7f87f256e700 1 -- 172.21.15.1:0/67145425 learned_addr learned my addr 172.21.15.1:0/67145425 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.950+0000 7f87f1d6d700 1 --2- 172.21.15.1:0/67145425 >> v2:172.21.15.165:3300/0 conn(0x7f87ec15b4d0 0x7f87ec09a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.950+0000 7f87f256e700 1 -- 172.21.15.1:0/67145425 >> v2:172.21.15.165:3300/0 conn(0x7f87ec15b4d0 msgr2=0x7f87ec09a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.950+0000 7f87f256e700 1 --2- 172.21.15.1:0/67145425 >> v2:172.21.15.165:3300/0 conn(0x7f87ec15b4d0 0x7f87ec09a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.950+0000 7f87f256e700 1 -- 172.21.15.1:0/67145425 >> v1:172.21.15.1:6789/0 conn(0x7f87ec0cf750 legacy=0x7f87ec099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.950+0000 7f87f256e700 1 -- 172.21.15.1:0/67145425 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ec06ea20 con 0x7f87ec09a150 2024-03-18T19:05:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.950+0000 7f87f256e700 1 --2- 172.21.15.1:0/67145425 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec09a150 0x7f87ec06f500 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f87ec071290 tx=0x7f87e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9fc4bad3dab2927 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.950+0000 7f87f0d6b700 1 -- 172.21.15.1:0/67145425 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e400e070 con 0x7f87ec09a150 2024-03-18T19:05:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.950+0000 7f87f0d6b700 1 -- 172.21.15.1:0/67145425 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e400b910 con 0x7f87ec09a150 2024-03-18T19:05:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.950+0000 7f87f0d6b700 1 -- 172.21.15.1:0/67145425 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e400bcc0 con 0x7f87ec09a150 2024-03-18T19:05:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.951+0000 7f87f3fd1700 1 -- 172.21.15.1:0/67145425 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec09a150 msgr2=0x7f87ec06f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.951+0000 7f87f3fd1700 1 --2- 172.21.15.1:0/67145425 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec09a150 0x7f87ec06f500 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f87ec071290 tx=0x7f87e400b330 comp rx=0 tx=0).stop 2024-03-18T19:05:07.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.951+0000 7f87f3fd1700 1 -- 172.21.15.1:0/67145425 shutdown_connections 2024-03-18T19:05:07.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.951+0000 7f87f3fd1700 1 --2- 172.21.15.1:0/67145425 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec09a150 0x7f87ec06f500 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.951+0000 7f87f3fd1700 1 --2- 172.21.15.1:0/67145425 >> v2:172.21.15.165:3300/0 conn(0x7f87ec15b4d0 0x7f87ec09a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.951+0000 7f87f3fd1700 1 -- 172.21.15.1:0/67145425 >> 172.21.15.1:0/67145425 conn(0x7f87ec0b2cb0 msgr2=0x7f87ec0b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:07.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.951+0000 7f87f3fd1700 1 -- 172.21.15.1:0/67145425 shutdown_connections 2024-03-18T19:05:07.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.951+0000 7f87f3fd1700 1 -- 172.21.15.1:0/67145425 wait complete. 2024-03-18T19:05:07.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.952+0000 7f87f3fd1700 1 Processor -- start 2024-03-18T19:05:07.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.952+0000 7f87f3fd1700 1 -- start start 2024-03-18T19:05:07.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.953+0000 7f87f3fd1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87ec09a150 0x7f87ec157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:07.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.953+0000 7f87f3fd1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87ec0cf750 0x7f87ec15e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:07.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.953+0000 7f87f3fd1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec15b4d0 0x7f87ec15c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:07.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.953+0000 7f87f3fd1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec158ce0 con 0x7f87ec15b4d0 2024-03-18T19:05:07.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.953+0000 7f87f3fd1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec157e30 con 0x7f87ec0cf750 2024-03-18T19:05:07.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.953+0000 7f87f3fd1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec15e930 con 0x7f87ec09a150 2024-03-18T19:05:07.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.953+0000 7f87f256e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec15b4d0 0x7f87ec15c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:07.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.953+0000 7f87f256e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec15b4d0 0x7f87ec15c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42270/0 (socket says 172.21.15.1:42270) 2024-03-18T19:05:07.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.953+0000 7f87f256e700 1 -- 172.21.15.1:0/700685300 learned_addr learned my addr 172.21.15.1:0/700685300 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:07.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.953+0000 7f87f156c700 1 --2- 172.21.15.1:0/700685300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87ec0cf750 0x7f87ec15e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:07.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.953+0000 7f87f1d6d700 1 --2- 172.21.15.1:0/700685300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87ec09a150 0x7f87ec157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:07.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.953+0000 7f87f256e700 1 -- 172.21.15.1:0/700685300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87ec09a150 msgr2=0x7f87ec157040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:07.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.954+0000 7f87f256e700 1 --2- 172.21.15.1:0/700685300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87ec09a150 0x7f87ec157040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.954+0000 7f87f256e700 1 -- 172.21.15.1:0/700685300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87ec0cf750 msgr2=0x7f87ec15e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:07.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.954+0000 7f87f256e700 1 --2- 172.21.15.1:0/700685300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87ec0cf750 0x7f87ec15e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.954+0000 7f87f256e700 1 -- 172.21.15.1:0/700685300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e4009b30 con 0x7f87ec15b4d0 2024-03-18T19:05:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.954+0000 7f87f256e700 1 --2- 172.21.15.1:0/700685300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec15b4d0 0x7f87ec15c120 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f87e400b370 tx=0x7f87e4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.954+0000 7f87e27fc700 1 -- 172.21.15.1:0/700685300 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e4004880 con 0x7f87ec15b4d0 2024-03-18T19:05:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.954+0000 7f87e27fc700 1 -- 172.21.15.1:0/700685300 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e40099b0 con 0x7f87ec15b4d0 2024-03-18T19:05:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.954+0000 7f87f3fd1700 1 -- 172.21.15.1:0/700685300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87ec1580d0 con 0x7f87ec15b4d0 2024-03-18T19:05:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.954+0000 7f87e27fc700 1 -- 172.21.15.1:0/700685300 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e401c680 con 0x7f87ec15b4d0 2024-03-18T19:05:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.954+0000 7f87f3fd1700 1 -- 172.21.15.1:0/700685300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87ec158690 con 0x7f87ec15b4d0 2024-03-18T19:05:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.955+0000 7f87f3fd1700 1 -- 172.21.15.1:0/700685300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87ec068240 con 0x7f87ec15b4d0 2024-03-18T19:05:07.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.956+0000 7f87e27fc700 1 -- 172.21.15.1:0/700685300 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f87e401c7e0 con 0x7f87ec15b4d0 2024-03-18T19:05:07.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.956+0000 7f87e27fc700 1 --2- 172.21.15.1:0/700685300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f87d006adb0 0x7f87d006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:07.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.956+0000 7f87f1d6d700 1 -- 172.21.15.1:0/700685300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f87d006adb0 msgr2=0x7f87d006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:07.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.956+0000 7f87e27fc700 1 -- 172.21.15.1:0/700685300 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f87e408b0f0 con 0x7f87ec15b4d0 2024-03-18T19:05:07.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.956+0000 7f87f1d6d700 1 --2- 172.21.15.1:0/700685300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f87d006adb0 0x7f87d006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:07.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:07.962+0000 7f87e27fc700 1 -- 172.21.15.1:0/700685300 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f87e4055e20 con 0x7f87ec15b4d0 2024-03-18T19:05:08.064 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:08.068+0000 7f6d09147240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:05:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.157+0000 7f87f1d6d700 1 -- 172.21.15.1:0/700685300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f87d006adb0 msgr2=0x7f87d006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:08.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.157+0000 7f87f1d6d700 1 --2- 172.21.15.1:0/700685300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f87d006adb0 0x7f87d006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:08.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.167+0000 7f87f3fd1700 1 -- 172.21.15.1:0/700685300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87ec06d8a0 con 0x7f87ec15b4d0 2024-03-18T19:05:08.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.168+0000 7f87e27fc700 1 -- 172.21.15.1:0/700685300 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16965 (secure 0 0 0) 0x7f87e4059440 con 0x7f87ec15b4d0 2024-03-18T19:05:08.164 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:08.164 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":76,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:05:04.110472+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:03:00.371114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-18T19:03:01.513069+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":76,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.1d","mappings":[{"from":0,"to":6}]},{"pgid":"8.2f","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6833/76998":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/275404534":"2024-03-19T19:03:17.738629+0000","172.21.15.1:6832/76998":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/1571059650":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/207797678":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/3342330530":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/953259654":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/3431076653":"2024-03-19T19:03:17.738629+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":8,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:05:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.170+0000 7f87f3fd1700 1 -- 172.21.15.1:0/700685300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f87d006adb0 msgr2=0x7f87d006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.170+0000 7f87f3fd1700 1 --2- 172.21.15.1:0/700685300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f87d006adb0 0x7f87d006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.170+0000 7f87f3fd1700 1 -- 172.21.15.1:0/700685300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec15b4d0 msgr2=0x7f87ec15c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.170+0000 7f87f3fd1700 1 --2- 172.21.15.1:0/700685300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec15b4d0 0x7f87ec15c120 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f87e400b370 tx=0x7f87e4004ad0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.170+0000 7f87f3fd1700 1 -- 172.21.15.1:0/700685300 shutdown_connections 2024-03-18T19:05:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.170+0000 7f87f3fd1700 1 --2- 172.21.15.1:0/700685300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f87d006adb0 0x7f87d006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.170+0000 7f87f3fd1700 1 --2- 172.21.15.1:0/700685300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87ec09a150 0x7f87ec157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.170+0000 7f87f3fd1700 1 --2- 172.21.15.1:0/700685300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87ec15b4d0 0x7f87ec15c120 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.170+0000 7f87f3fd1700 1 --2- 172.21.15.1:0/700685300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87ec0cf750 0x7f87ec15e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.170+0000 7f87f3fd1700 1 -- 172.21.15.1:0/700685300 >> 172.21.15.1:0/700685300 conn(0x7f87ec0b2cb0 msgr2=0x7f87ec06fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.170+0000 7f87f3fd1700 1 -- 172.21.15.1:0/700685300 shutdown_connections 2024-03-18T19:05:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.170+0000 7f87f3fd1700 1 -- 172.21.15.1:0/700685300 wait complete. 2024-03-18T19:05:08.179 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T19:05:08.179 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:05:08.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.309+0000 7fc53ad64700 1 Processor -- start 2024-03-18T19:05:08.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.310+0000 7fc53ad64700 1 -- start start 2024-03-18T19:05:08.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.310+0000 7fc53ad64700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5340ae600 0x7fc5340ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:08.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.310+0000 7fc53ad64700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc53415a2e0 0x7fc5340ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:08.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.310+0000 7fc53ad64700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5340cc170 0x7fc53406ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:08.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.310+0000 7fc53ad64700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc53406e270 con 0x7fc5340cc170 2024-03-18T19:05:08.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.310+0000 7fc53ad64700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5340ae050 con 0x7fc53415a2e0 2024-03-18T19:05:08.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.310+0000 7fc53ad64700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5340aef20 con 0x7fc5340ae600 2024-03-18T19:05:08.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.311+0000 7fc539301700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5340cc170 0x7fc53406ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:08.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.311+0000 7fc539301700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5340cc170 0x7fc53406ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42282/0 (socket says 172.21.15.1:42282) 2024-03-18T19:05:08.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.311+0000 7fc539301700 1 -- 172.21.15.1:0/2477441057 learned_addr learned my addr 172.21.15.1:0/2477441057 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:08.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.311+0000 7fc533fff700 1 --2- 172.21.15.1:0/2477441057 >> v2:172.21.15.165:3300/0 conn(0x7fc53415a2e0 0x7fc5340ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:08.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.311+0000 7fc538b00700 1 --2- 172.21.15.1:0/2477441057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5340ae600 0x7fc5340ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:08.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.311+0000 7fc539301700 1 -- 172.21.15.1:0/2477441057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5340ae600 msgr2=0x7fc5340ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:08.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.311+0000 7fc539301700 1 --2- 172.21.15.1:0/2477441057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5340ae600 0x7fc5340ad790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.311+0000 7fc539301700 1 -- 172.21.15.1:0/2477441057 >> v2:172.21.15.165:3300/0 conn(0x7fc53415a2e0 msgr2=0x7fc5340ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:08.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.311+0000 7fc539301700 1 --2- 172.21.15.1:0/2477441057 >> v2:172.21.15.165:3300/0 conn(0x7fc53415a2e0 0x7fc5340ae9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.311+0000 7fc539301700 1 -- 172.21.15.1:0/2477441057 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5340add70 con 0x7fc5340cc170 2024-03-18T19:05:08.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.311+0000 7fc539301700 1 --2- 172.21.15.1:0/2477441057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5340cc170 0x7fc53406ef80 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fc534067040 tx=0x7fc52c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae8cc7cc7ef226d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:08.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.311+0000 7fc5337fe700 1 -- 172.21.15.1:0/2477441057 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c00e070 con 0x7fc5340cc170 2024-03-18T19:05:08.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.311+0000 7fc5337fe700 1 -- 172.21.15.1:0/2477441057 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52c00b910 con 0x7fc5340cc170 2024-03-18T19:05:08.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.311+0000 7fc5337fe700 1 -- 172.21.15.1:0/2477441057 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c00bcc0 con 0x7fc5340cc170 2024-03-18T19:05:08.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.312+0000 7fc53ad64700 1 -- 172.21.15.1:0/2477441057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5340cc170 msgr2=0x7fc53406ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:08.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.312+0000 7fc53ad64700 1 --2- 172.21.15.1:0/2477441057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5340cc170 0x7fc53406ef80 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fc534067040 tx=0x7fc52c00b330 comp rx=0 tx=0).stop 2024-03-18T19:05:08.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.312+0000 7fc53ad64700 1 -- 172.21.15.1:0/2477441057 shutdown_connections 2024-03-18T19:05:08.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.312+0000 7fc53ad64700 1 --2- 172.21.15.1:0/2477441057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5340cc170 0x7fc53406ef80 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.312+0000 7fc53ad64700 1 --2- 172.21.15.1:0/2477441057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5340ae600 0x7fc5340ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.312+0000 7fc53ad64700 1 --2- 172.21.15.1:0/2477441057 >> v2:172.21.15.165:3300/0 conn(0x7fc53415a2e0 0x7fc5340ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.312+0000 7fc53ad64700 1 -- 172.21.15.1:0/2477441057 >> 172.21.15.1:0/2477441057 conn(0x7fc53406b860 msgr2=0x7fc53406bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:08.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.312+0000 7fc53ad64700 1 -- 172.21.15.1:0/2477441057 shutdown_connections 2024-03-18T19:05:08.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.312+0000 7fc53ad64700 1 -- 172.21.15.1:0/2477441057 wait complete. 2024-03-18T19:05:08.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.314+0000 7fc53ad64700 1 Processor -- start 2024-03-18T19:05:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.314+0000 7fc53ad64700 1 -- start start 2024-03-18T19:05:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.315+0000 7fc53ad64700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5340ae600 0x7fc53413fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.315+0000 7fc53ad64700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5340cc170 0x7fc5341426d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.315+0000 7fc53ad64700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc53415a2e0 0x7fc534142c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.315+0000 7fc53ad64700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc534153a10 con 0x7fc53415a2e0 2024-03-18T19:05:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.315+0000 7fc53ad64700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc534140340 con 0x7fc5340cc170 2024-03-18T19:05:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.315+0000 7fc53ad64700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc534141220 con 0x7fc5340ae600 2024-03-18T19:05:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.315+0000 7fc539301700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc53415a2e0 0x7fc534142c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.315+0000 7fc539301700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc53415a2e0 0x7fc534142c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42284/0 (socket says 172.21.15.1:42284) 2024-03-18T19:05:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.315+0000 7fc539301700 1 -- 172.21.15.1:0/1631247588 learned_addr learned my addr 172.21.15.1:0/1631247588 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.315+0000 7fc538b00700 1 --2- 172.21.15.1:0/1631247588 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5340ae600 0x7fc53413fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.315+0000 7fc533fff700 1 --2- 172.21.15.1:0/1631247588 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5340cc170 0x7fc5341426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.315+0000 7fc539301700 1 -- 172.21.15.1:0/1631247588 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5340ae600 msgr2=0x7fc53413fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:08.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.315+0000 7fc539301700 1 --2- 172.21.15.1:0/1631247588 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5340ae600 0x7fc53413fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.315+0000 7fc539301700 1 -- 172.21.15.1:0/1631247588 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5340cc170 msgr2=0x7fc5341426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:08.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.315+0000 7fc539301700 1 --2- 172.21.15.1:0/1631247588 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5340cc170 0x7fc5341426d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.315+0000 7fc539301700 1 -- 172.21.15.1:0/1631247588 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c009b30 con 0x7fc53415a2e0 2024-03-18T19:05:08.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.315+0000 7fc539301700 1 --2- 172.21.15.1:0/1631247588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc53415a2e0 0x7fc534142c10 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fc5340672c0 tx=0x7fc52c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:08.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.316+0000 7fc5317fa700 1 -- 172.21.15.1:0/1631247588 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c0099b0 con 0x7fc53415a2e0 2024-03-18T19:05:08.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.316+0000 7fc5317fa700 1 -- 172.21.15.1:0/1631247588 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52c00ce80 con 0x7fc53415a2e0 2024-03-18T19:05:08.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.316+0000 7fc53ad64700 1 -- 172.21.15.1:0/1631247588 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc534141500 con 0x7fc53415a2e0 2024-03-18T19:05:08.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.316+0000 7fc5317fa700 1 -- 172.21.15.1:0/1631247588 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c004490 con 0x7fc53415a2e0 2024-03-18T19:05:08.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.316+0000 7fc53ad64700 1 -- 172.21.15.1:0/1631247588 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc534141ac0 con 0x7fc53415a2e0 2024-03-18T19:05:08.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.317+0000 7fc53ad64700 1 -- 172.21.15.1:0/1631247588 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc53409dc90 con 0x7fc53415a2e0 2024-03-18T19:05:08.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.317+0000 7fc5317fa700 1 -- 172.21.15.1:0/1631247588 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fc52c0045f0 con 0x7fc53415a2e0 2024-03-18T19:05:08.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.318+0000 7fc5317fa700 1 --2- 172.21.15.1:0/1631247588 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc51806adb0 0x7fc51806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:08.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.318+0000 7fc538b00700 1 -- 172.21.15.1:0/1631247588 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc51806adb0 msgr2=0x7fc51806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:08.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.318+0000 7fc5317fa700 1 -- 172.21.15.1:0/1631247588 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6503+0+0 (secure 0 0 0) 0x7fc52c08ab20 con 0x7fc53415a2e0 2024-03-18T19:05:08.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.318+0000 7fc538b00700 1 --2- 172.21.15.1:0/1631247588 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc51806adb0 0x7fc51806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:08.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.322+0000 7fc5317fa700 1 -- 172.21.15.1:0/1631247588 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc52c055850 con 0x7fc53415a2e0 2024-03-18T19:05:08.324 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:08.329+0000 7f420e1eb240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:05:08.427 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:08.431+0000 7f6d09147240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:05:08.501 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:08.505+0000 7f420e1eb240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:05:08.513 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:08.517+0000 7f6d09147240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:05:08.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.519+0000 7fc538b00700 1 -- 172.21.15.1:0/1631247588 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc51806adb0 msgr2=0x7fc51806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:08.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.519+0000 7fc538b00700 1 --2- 172.21.15.1:0/1631247588 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc51806adb0 0x7fc51806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:08.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.574+0000 7fc53ad64700 1 -- 172.21.15.1:0/1631247588 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5340676a0 con 0x7fc53415a2e0 2024-03-18T19:05:08.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.574+0000 7fc5317fa700 1 -- 172.21.15.1:0/1631247588 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4731 (secure 0 0 0) 0x7fc52c020020 con 0x7fc53415a2e0 2024-03-18T19:05:08.571 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:08.571 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5420,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3327835778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":3327835778},{"type":"v1","addr":"172.21.15.165:6835","nonce":3327835778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6016,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/450295183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":450295183},{"type":"v1","addr":"172.21.15.1:6837","nonce":450295183}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6022,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1053303526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1053303526},{"type":"v1","addr":"172.21.15.165:6834","nonce":1053303526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:03:02.534936+0000","modified":"2024-03-18T19:03:05.546799+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6270":{"gid":6270,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6836/1826374081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1826374081},{"type":"v1","addr":"172.21.15.1:6836","nonce":1826374081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-18T19:05:08.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.577+0000 7fc53ad64700 1 -- 172.21.15.1:0/1631247588 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc51806adb0 msgr2=0x7fc51806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:08.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.577+0000 7fc53ad64700 1 --2- 172.21.15.1:0/1631247588 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc51806adb0 0x7fc51806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.577+0000 7fc53ad64700 1 -- 172.21.15.1:0/1631247588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc53415a2e0 msgr2=0x7fc534142c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:08.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.577+0000 7fc53ad64700 1 --2- 172.21.15.1:0/1631247588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc53415a2e0 0x7fc534142c10 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fc5340672c0 tx=0x7fc52c004ad0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.577+0000 7fc53ad64700 1 -- 172.21.15.1:0/1631247588 shutdown_connections 2024-03-18T19:05:08.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.577+0000 7fc53ad64700 1 --2- 172.21.15.1:0/1631247588 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc51806adb0 0x7fc51806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.577+0000 7fc53ad64700 1 --2- 172.21.15.1:0/1631247588 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5340ae600 0x7fc53413fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.577+0000 7fc53ad64700 1 --2- 172.21.15.1:0/1631247588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc53415a2e0 0x7fc534142c10 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.577+0000 7fc53ad64700 1 --2- 172.21.15.1:0/1631247588 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5340cc170 0x7fc5341426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.577+0000 7fc53ad64700 1 -- 172.21.15.1:0/1631247588 >> 172.21.15.1:0/1631247588 conn(0x7fc53406b860 msgr2=0x7fc5340cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:08.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.577+0000 7fc53ad64700 1 -- 172.21.15.1:0/1631247588 shutdown_connections 2024-03-18T19:05:08.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.577+0000 7fc53ad64700 1 -- 172.21.15.1:0/1631247588 wait complete. 2024-03-18T19:05:08.574 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-03-18T19:05:08.585 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:05:08.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.720+0000 7fd0ebc70700 1 Processor -- start 2024-03-18T19:05:08.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.720+0000 7fd0ebc70700 1 -- start start 2024-03-18T19:05:08.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.721+0000 7fd0ebc70700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd0e40d1c60 0x7fd0e40b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.721+0000 7fd0ebc70700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e40cddc0 con 0x7fd0e40d17b0 2024-03-18T19:05:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.721+0000 7fd0ebc70700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e40d3dc0 con 0x7fd0e415d500 2024-03-18T19:05:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.721+0000 7fd0ebc70700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0e40ce060 con 0x7fd0e40d1c60 2024-03-18T19:05:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.721+0000 7fd0e920b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd0e40d17b0 0x7fd0e40b83f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42362/0 (socket says 172.21.15.1:42362) 2024-03-18T19:05:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.721+0000 7fd0e920b700 1 -- 172.21.15.1:0/2400429113 learned_addr learned my addr 172.21.15.1:0/2400429113 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.721+0000 7fd0ea20d700 1 --2- 172.21.15.1:0/2400429113 >> v2:172.21.15.165:3300/0 conn(0x7fd0e40d1c60 0x7fd0e40b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.721+0000 7fd0e8a0a700 1 -- 172.21.15.1:0/2400429113 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169281507 0 0) 0x7fd0e40cddc0 con 0x7fd0e40d17b0 2024-03-18T19:05:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.721+0000 7fd0e8a0a700 1 -- 172.21.15.1:0/2400429113 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0d00035b0 con 0x7fd0e40d17b0 2024-03-18T19:05:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.721+0000 7fd0e8a0a700 1 -- 172.21.15.1:0/2400429113 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281978175 0 0) 0x7fd0e40d3dc0 con 0x7fd0e415d500 2024-03-18T19:05:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.721+0000 7fd0e8a0a700 1 -- 172.21.15.1:0/2400429113 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0e40cddc0 con 0x7fd0e415d500 2024-03-18T19:05:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.722+0000 7fd0ea20d700 1 -- 172.21.15.1:0/2400429113 >> v1:172.21.15.165:6789/0 conn(0x7fd0e415d500 legacy=0x7fd0e40d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.722+0000 7fd0ea20d700 1 -- 172.21.15.1:0/2400429113 >> v1:172.21.15.1:6789/0 conn(0x7fd0e40d17b0 legacy=0x7fd0e40b83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.722+0000 7fd0ea20d700 1 -- 172.21.15.1:0/2400429113 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e40ce340 con 0x7fd0e40d1c60 2024-03-18T19:05:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.722+0000 7fd0ea20d700 1 --2- 172.21.15.1:0/2400429113 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e40d1c60 0x7fd0e40b8b00 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc009d90 tx=0x7fd0dc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c05b72956614b3de server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.722+0000 7fd0e8a0a700 1 -- 172.21.15.1:0/2400429113 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc00e070 con 0x7fd0e40d1c60 2024-03-18T19:05:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.722+0000 7fd0e8a0a700 1 -- 172.21.15.1:0/2400429113 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0dc00bb20 con 0x7fd0e40d1c60 2024-03-18T19:05:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.722+0000 7fd0e8a0a700 1 -- 172.21.15.1:0/2400429113 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc012430 con 0x7fd0e40d1c60 2024-03-18T19:05:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.723+0000 7fd0ebc70700 1 -- 172.21.15.1:0/2400429113 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e40d1c60 msgr2=0x7fd0e40b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.723+0000 7fd0ebc70700 1 --2- 172.21.15.1:0/2400429113 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e40d1c60 0x7fd0e40b8b00 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc009d90 tx=0x7fd0dc00b330 comp rx=0 tx=0).stop 2024-03-18T19:05:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.723+0000 7fd0ebc70700 1 -- 172.21.15.1:0/2400429113 shutdown_connections 2024-03-18T19:05:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.723+0000 7fd0ebc70700 1 --2- 172.21.15.1:0/2400429113 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e40d1c60 0x7fd0e40b8b00 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.723+0000 7fd0ebc70700 1 -- 172.21.15.1:0/2400429113 >> 172.21.15.1:0/2400429113 conn(0x7fd0e409c530 msgr2=0x7fd0e40d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.723+0000 7fd0ebc70700 1 -- 172.21.15.1:0/2400429113 shutdown_connections 2024-03-18T19:05:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.723+0000 7fd0ebc70700 1 -- 172.21.15.1:0/2400429113 wait complete. 2024-03-18T19:05:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.724+0000 7fd0ebc70700 1 Processor -- start 2024-03-18T19:05:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.724+0000 7fd0ebc70700 1 -- start start 2024-03-18T19:05:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.725+0000 7fd0ebc70700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e40d17b0 0x7fd0e41529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.725+0000 7fd0ebc70700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd0e40d1c60 0x7fd0e40b6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.725+0000 7fd0ebc70700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e415d500 0x7fd0e4156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.725+0000 7fd0ebc70700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e4152ee0 con 0x7fd0e415d500 2024-03-18T19:05:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.725+0000 7fd0ebc70700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e40c8c50 con 0x7fd0e40d17b0 2024-03-18T19:05:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.725+0000 7fd0ebc70700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e40b6da0 con 0x7fd0e40d1c60 2024-03-18T19:05:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.726+0000 7fd0ea20d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e415d500 0x7fd0e4156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.726+0000 7fd0ea20d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e415d500 0x7fd0e4156990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42296/0 (socket says 172.21.15.1:42296) 2024-03-18T19:05:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.726+0000 7fd0ea20d700 1 -- 172.21.15.1:0/1337869 learned_addr learned my addr 172.21.15.1:0/1337869 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.726+0000 7fd0e9a0c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e40d17b0 0x7fd0e41529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.726+0000 7fd0e920b700 1 --2- 172.21.15.1:0/1337869 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd0e40d1c60 0x7fd0e40b6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.726+0000 7fd0ea20d700 1 -- 172.21.15.1:0/1337869 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd0e40d1c60 msgr2=0x7fd0e40b6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.726+0000 7fd0ea20d700 1 --2- 172.21.15.1:0/1337869 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd0e40d1c60 0x7fd0e40b6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.726+0000 7fd0ea20d700 1 -- 172.21.15.1:0/1337869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e40d17b0 msgr2=0x7fd0e41529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.726+0000 7fd0ea20d700 1 --2- 172.21.15.1:0/1337869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e40d17b0 0x7fd0e41529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.726+0000 7fd0ea20d700 1 -- 172.21.15.1:0/1337869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc009a70 con 0x7fd0e415d500 2024-03-18T19:05:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.726+0000 7fd0ea20d700 1 --2- 172.21.15.1:0/1337869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e415d500 0x7fd0e4156990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fd0e408da30 tx=0x7fd0dc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.726+0000 7fd0da7fc700 1 -- 172.21.15.1:0/1337869 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc012430 con 0x7fd0e415d500 2024-03-18T19:05:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.726+0000 7fd0da7fc700 1 -- 172.21.15.1:0/1337869 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0dc020070 con 0x7fd0e415d500 2024-03-18T19:05:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.726+0000 7fd0ebc70700 1 -- 172.21.15.1:0/1337869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e40c8600 con 0x7fd0e415d500 2024-03-18T19:05:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.726+0000 7fd0da7fc700 1 -- 172.21.15.1:0/1337869 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc01ba50 con 0x7fd0e415d500 2024-03-18T19:05:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.726+0000 7fd0ebc70700 1 -- 172.21.15.1:0/1337869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e40b6ab0 con 0x7fd0e415d500 2024-03-18T19:05:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.728+0000 7fd0ebc70700 1 -- 172.21.15.1:0/1337869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e40c8dc0 con 0x7fd0e415d500 2024-03-18T19:05:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.728+0000 7fd0da7fc700 1 -- 172.21.15.1:0/1337869 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fd0dc01bbb0 con 0x7fd0e415d500 2024-03-18T19:05:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.728+0000 7fd0da7fc700 1 --2- 172.21.15.1:0/1337869 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd0c806adb0 0x7fd0c806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.728+0000 7fd0e9a0c700 1 -- 172.21.15.1:0/1337869 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd0c806adb0 msgr2=0x7fd0c806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.728+0000 7fd0da7fc700 1 -- 172.21.15.1:0/1337869 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6503+0+0 (secure 0 0 0) 0x7fd0dc022070 con 0x7fd0e415d500 2024-03-18T19:05:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.728+0000 7fd0e9a0c700 1 --2- 172.21.15.1:0/1337869 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd0c806adb0 0x7fd0c806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:08.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.733+0000 7fd0da7fc700 1 -- 172.21.15.1:0/1337869 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd0dc056620 con 0x7fd0e415d500 2024-03-18T19:05:08.886 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:08.890+0000 7f420e1eb240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:05:08.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.929+0000 7fd0e9a0c700 1 -- 172.21.15.1:0/1337869 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd0c806adb0 msgr2=0x7fd0c806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:08.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.929+0000 7fd0e9a0c700 1 --2- 172.21.15.1:0/1337869 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd0c806adb0 0x7fd0c806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:08.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.982+0000 7fd0ebc70700 1 -- 172.21.15.1:0/1337869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0e40676a0 con 0x7fd0e415d500 2024-03-18T19:05:08.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.983+0000 7fd0da7fc700 1 -- 172.21.15.1:0/1337869 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4731 (secure 0 0 0) 0x7fd0dc01be60 con 0x7fd0e415d500 2024-03-18T19:05:08.983 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:08.983 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5420,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3327835778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":3327835778},{"type":"v1","addr":"172.21.15.165:6835","nonce":3327835778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6016,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/450295183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":450295183},{"type":"v1","addr":"172.21.15.1:6837","nonce":450295183}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6022,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1053303526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1053303526},{"type":"v1","addr":"172.21.15.165:6834","nonce":1053303526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:03:02.534936+0000","modified":"2024-03-18T19:03:05.546799+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6270":{"gid":6270,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6836/1826374081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1826374081},{"type":"v1","addr":"172.21.15.1:6836","nonce":1826374081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-18T19:05:08.984 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:08.984+0000 7f420e1eb240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:05:08.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.985+0000 7fd0ebc70700 1 -- 172.21.15.1:0/1337869 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd0c806adb0 msgr2=0x7fd0c806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:08.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.985+0000 7fd0ebc70700 1 --2- 172.21.15.1:0/1337869 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd0c806adb0 0x7fd0c806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.985+0000 7fd0ebc70700 1 -- 172.21.15.1:0/1337869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e415d500 msgr2=0x7fd0e4156990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:08.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.985+0000 7fd0ebc70700 1 --2- 172.21.15.1:0/1337869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e415d500 0x7fd0e4156990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fd0e408da30 tx=0x7fd0dc004ad0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.985+0000 7fd0ebc70700 1 -- 172.21.15.1:0/1337869 shutdown_connections 2024-03-18T19:05:08.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.985+0000 7fd0ebc70700 1 --2- 172.21.15.1:0/1337869 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd0c806adb0 0x7fd0c806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.985+0000 7fd0ebc70700 1 --2- 172.21.15.1:0/1337869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0e415d500 0x7fd0e4156990 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.985+0000 7fd0ebc70700 1 --2- 172.21.15.1:0/1337869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e40d17b0 0x7fd0e41529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.985+0000 7fd0ebc70700 1 --2- 172.21.15.1:0/1337869 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd0e40d1c60 0x7fd0e40b6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:08.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.985+0000 7fd0ebc70700 1 -- 172.21.15.1:0/1337869 >> 172.21.15.1:0/1337869 conn(0x7fd0e409c530 msgr2=0x7fd0e415d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:08.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.985+0000 7fd0ebc70700 1 -- 172.21.15.1:0/1337869 shutdown_connections 2024-03-18T19:05:08.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:08.985+0000 7fd0ebc70700 1 -- 172.21.15.1:0/1337869 wait complete. 2024-03-18T19:05:08.986 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 25 2024-03-18T19:05:08.993 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:05:09.069 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:09.073+0000 7f4186613700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:09.069 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:09.073+0000 7f4186613700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:09.069 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:09.073+0000 7f4186613700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:09.069 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:09.073+0000 7f4186613700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:09.069 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:09.073+0000 7f4186613700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:09.069 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:09.073+0000 7f4189619700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:09.069 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:09.073+0000 7f4189619700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:09.069 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:09.073+0000 7f4189619700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:09.070 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:09.073+0000 7f4189619700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:09.070 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:09.073+0000 7f4189619700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:09.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.138+0000 7fd8e399a700 1 Processor -- start 2024-03-18T19:05:09.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.138+0000 7fd8e399a700 1 -- start start 2024-03-18T19:05:09.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.139+0000 7fd8e399a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8dc099a40 0x7fd8dc099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:09.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.139+0000 7fd8e399a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8dc09a3e0 con 0x7fd8dc0cc310 2024-03-18T19:05:09.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.139+0000 7fd8e399a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8dc0d0c80 con 0x7fd8dc15d680 2024-03-18T19:05:09.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.139+0000 7fd8e399a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8dc0d09a0 con 0x7fd8dc099a40 2024-03-18T19:05:09.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.139+0000 7fd8e0f35700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8dc099a40 0x7fd8dc099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:09.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.139+0000 7fd8e1f37700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd8dc0cc310 0x7fd8dc0cc7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42376/0 (socket says 172.21.15.1:42376) 2024-03-18T19:05:09.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.139+0000 7fd8e1f37700 1 -- 172.21.15.1:0/3648411075 learned_addr learned my addr 172.21.15.1:0/3648411075 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:09.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.139+0000 7fd8e0f35700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8dc099a40 0x7fd8dc099ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42374/0 (socket says 172.21.15.1:42374) 2024-03-18T19:05:09.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.140+0000 7fd8e0f35700 1 -- 172.21.15.1:0/3648411075 >> v1:172.21.15.165:6789/0 conn(0x7fd8dc15d680 legacy=0x7fd8dc0cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:09.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.140+0000 7fd8e0f35700 1 -- 172.21.15.1:0/3648411075 >> v1:172.21.15.1:6789/0 conn(0x7fd8dc0cc310 legacy=0x7fd8dc0cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:09.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.140+0000 7fd8e0f35700 1 -- 172.21.15.1:0/3648411075 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc0cb830 con 0x7fd8dc099a40 2024-03-18T19:05:09.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.140+0000 7fd8d3fff700 1 -- 172.21.15.1:0/3648411075 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439577608 0 0) 0x7fd8dc09a3e0 con 0x7fd8dc0cc310 2024-03-18T19:05:09.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.140+0000 7fd8e0f35700 1 --2- 172.21.15.1:0/3648411075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8dc099a40 0x7fd8dc099ea0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8009840 tx=0x7fd8d8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf02a41890bf0b59 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:09.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.140+0000 7fd8d3fff700 1 -- 172.21.15.1:0/3648411075 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8004230 con 0x7fd8dc099a40 2024-03-18T19:05:09.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.140+0000 7fd8d3fff700 1 -- 172.21.15.1:0/3648411075 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d8004390 con 0x7fd8dc099a40 2024-03-18T19:05:09.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.140+0000 7fd8d3fff700 1 -- 172.21.15.1:0/3648411075 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d80036a0 con 0x7fd8dc099a40 2024-03-18T19:05:09.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.140+0000 7fd8e399a700 1 -- 172.21.15.1:0/3648411075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8dc099a40 msgr2=0x7fd8dc099ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:09.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.141+0000 7fd8e399a700 1 --2- 172.21.15.1:0/3648411075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8dc099a40 0x7fd8dc099ea0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8009840 tx=0x7fd8d8009b50 comp rx=0 tx=0).stop 2024-03-18T19:05:09.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.141+0000 7fd8e399a700 1 -- 172.21.15.1:0/3648411075 shutdown_connections 2024-03-18T19:05:09.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.141+0000 7fd8e399a700 1 --2- 172.21.15.1:0/3648411075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8dc099a40 0x7fd8dc099ea0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.141+0000 7fd8e399a700 1 -- 172.21.15.1:0/3648411075 >> 172.21.15.1:0/3648411075 conn(0x7fd8dc0b2ba0 msgr2=0x7fd8dc0b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:09.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.141+0000 7fd8e399a700 1 -- 172.21.15.1:0/3648411075 shutdown_connections 2024-03-18T19:05:09.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.141+0000 7fd8e399a700 1 -- 172.21.15.1:0/3648411075 wait complete. 2024-03-18T19:05:09.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.142+0000 7fd8e399a700 1 Processor -- start 2024-03-18T19:05:09.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.142+0000 7fd8e399a700 1 -- start start 2024-03-18T19:05:09.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8e399a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8dc099a40 0x7fd8dc0b3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:09.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8e399a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8dc0cc310 0x7fd8dc15b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:09.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8e399a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8dc15d680 0x7fd8dc15f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:09.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8e399a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc15bca0 con 0x7fd8dc0cc310 2024-03-18T19:05:09.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8e399a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc0b3ee0 con 0x7fd8dc099a40 2024-03-18T19:05:09.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8e399a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc0cb350 con 0x7fd8dc15d680 2024-03-18T19:05:09.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8e0f35700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8dc0cc310 0x7fd8dc15b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:09.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8e0f35700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8dc0cc310 0x7fd8dc15b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42382/0 (socket says 172.21.15.1:42382) 2024-03-18T19:05:09.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8e0f35700 1 -- 172.21.15.1:0/3727209945 learned_addr learned my addr 172.21.15.1:0/3727209945 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:09.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8e1f37700 1 --2- 172.21.15.1:0/3727209945 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8dc15d680 0x7fd8dc15f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:09.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8e1736700 1 --2- 172.21.15.1:0/3727209945 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8dc099a40 0x7fd8dc0b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:09.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8e0f35700 1 -- 172.21.15.1:0/3727209945 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8dc15d680 msgr2=0x7fd8dc15f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:09.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8e0f35700 1 --2- 172.21.15.1:0/3727209945 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8dc15d680 0x7fd8dc15f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8e0f35700 1 -- 172.21.15.1:0/3727209945 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8dc099a40 msgr2=0x7fd8dc0b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:09.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8e0f35700 1 --2- 172.21.15.1:0/3727209945 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8dc099a40 0x7fd8dc0b3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8e0f35700 1 -- 172.21.15.1:0/3727209945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d80094a0 con 0x7fd8dc0cc310 2024-03-18T19:05:09.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8e0f35700 1 --2- 172.21.15.1:0/3727209945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8dc0cc310 0x7fd8dc15b760 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fd8d80090b0 tx=0x7fd8d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:09.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8d1ffb700 1 -- 172.21.15.1:0/3727209945 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8011830 con 0x7fd8dc0cc310 2024-03-18T19:05:09.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8d1ffb700 1 -- 172.21.15.1:0/3727209945 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d801e070 con 0x7fd8dc0cc310 2024-03-18T19:05:09.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8e399a700 1 -- 172.21.15.1:0/3727209945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8dc0c8220 con 0x7fd8dc0cc310 2024-03-18T19:05:09.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.143+0000 7fd8d1ffb700 1 -- 172.21.15.1:0/3727209945 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8011bf0 con 0x7fd8dc0cc310 2024-03-18T19:05:09.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.144+0000 7fd8e399a700 1 -- 172.21.15.1:0/3727209945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8dc0c87e0 con 0x7fd8dc0cc310 2024-03-18T19:05:09.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.144+0000 7fd8e399a700 1 -- 172.21.15.1:0/3727209945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8dc06cd50 con 0x7fd8dc0cc310 2024-03-18T19:05:09.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.145+0000 7fd8d1ffb700 1 -- 172.21.15.1:0/3727209945 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 27) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fd8d8011d50 con 0x7fd8dc0cc310 2024-03-18T19:05:09.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.146+0000 7fd8d1ffb700 1 -- 172.21.15.1:0/3727209945 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6717+0+0 (secure 0 0 0) 0x7fd8d808a530 con 0x7fd8dc0cc310 2024-03-18T19:05:09.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.150+0000 7fd8d1ffb700 1 -- 172.21.15.1:0/3727209945 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd8d8055770 con 0x7fd8dc0cc310 2024-03-18T19:05:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.352+0000 7fd8e399a700 1 -- 172.21.15.1:0/3727209945 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8dc0676a0 con 0x7fd8dc0cc310 2024-03-18T19:05:09.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.353+0000 7fd8d1ffb700 1 -- 172.21.15.1:0/3727209945 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+17265 (secure 0 0 0) 0x7fd8d8020090 con 0x7fd8dc0cc310 2024-03-18T19:05:09.350 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:09.351 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":77,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:05:08.999375+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:03:00.371114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-18T19:03:01.513069+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":76,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.1d","mappings":[{"from":0,"to":6}]},{"pgid":"8.2f","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3395896573":"2024-03-19T19:05:08.999330+0000","172.21.15.1:0/1605238486":"2024-03-19T19:05:08.999330+0000","172.21.15.1:0/1830599105":"2024-03-19T19:05:08.999330+0000","172.21.15.1:0/1718873699":"2024-03-19T19:05:08.999330+0000","172.21.15.1:6833/76998":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/2693010811":"2024-03-19T19:05:08.999330+0000","172.21.15.1:0/3008320004":"2024-03-19T19:05:08.999330+0000","172.21.15.1:0/275404534":"2024-03-19T19:03:17.738629+0000","172.21.15.1:6832/76998":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/1571059650":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/207797678":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/3342330530":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/953259654":"2024-03-19T19:03:17.738629+0000","172.21.15.1:0/3431076653":"2024-03-19T19:03:17.738629+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:05:09.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.355+0000 7fd8e399a700 1 -- 172.21.15.1:0/3727209945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8dc0cc310 msgr2=0x7fd8dc15b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.355+0000 7fd8e399a700 1 --2- 172.21.15.1:0/3727209945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8dc0cc310 0x7fd8dc15b760 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fd8d80090b0 tx=0x7fd8d8004750 comp rx=0 tx=0).stop 2024-03-18T19:05:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.355+0000 7fd8e399a700 1 -- 172.21.15.1:0/3727209945 shutdown_connections 2024-03-18T19:05:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.356+0000 7fd8e399a700 1 --2- 172.21.15.1:0/3727209945 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8dc15d680 0x7fd8dc15f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.356+0000 7fd8e399a700 1 --2- 172.21.15.1:0/3727209945 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8dc099a40 0x7fd8dc0b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.356+0000 7fd8e399a700 1 --2- 172.21.15.1:0/3727209945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8dc0cc310 0x7fd8dc15b760 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.356+0000 7fd8e399a700 1 -- 172.21.15.1:0/3727209945 >> 172.21.15.1:0/3727209945 conn(0x7fd8dc0b2ba0 msgr2=0x7fd8dc15c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.356+0000 7fd8e399a700 1 -- 172.21.15.1:0/3727209945 shutdown_connections 2024-03-18T19:05:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.356+0000 7fd8e399a700 1 -- 172.21.15.1:0/3727209945 wait complete. 2024-03-18T19:05:09.364 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T19:05:09.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.494+0000 7f3879c1b700 1 Processor -- start 2024-03-18T19:05:09.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.494+0000 7f3879c1b700 1 -- start start 2024-03-18T19:05:09.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.495+0000 7f3879c1b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f38740ade00 0x7f38740cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:09.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.495+0000 7f3879c1b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38740ccc20 0x7f387406fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:09.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.495+0000 7f3879c1b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38740b3aa0 con 0x7f3874099a40 2024-03-18T19:05:09.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.495+0000 7f3879c1b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38740ae280 con 0x7f38740ade00 2024-03-18T19:05:09.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.495+0000 7f3879c1b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387406e9b0 con 0x7f38740ccc20 2024-03-18T19:05:09.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.495+0000 7f3873fff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38740ccc20 0x7f387406fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:09.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.495+0000 7f3872ffd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f38740ade00 0x7f38740cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:09.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.495+0000 7f38737fe700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f3874099a40 0x7f3874099e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:35818/0 (socket says 172.21.15.1:35818) 2024-03-18T19:05:09.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.495+0000 7f3872ffd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f38740ade00 0x7f38740cc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:50788/0 (socket says 172.21.15.1:50788) 2024-03-18T19:05:09.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.495+0000 7f3873fff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38740ccc20 0x7f387406fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:56034/0 (socket says 172.21.15.1:56034) 2024-03-18T19:05:09.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.495+0000 7f38737fe700 1 -- 172.21.15.1:0/844852128 learned_addr learned my addr 172.21.15.1:0/844852128 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:09.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.495+0000 7f3872ffd700 1 -- 172.21.15.1:0/844852128 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38740ccc20 msgr2=0x7f387406fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:09.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.495+0000 7f3872ffd700 1 --2- 172.21.15.1:0/844852128 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38740ccc20 0x7f387406fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.495+0000 7f3872ffd700 1 -- 172.21.15.1:0/844852128 >> v1:172.21.15.165:6789/0 conn(0x7f3874099a40 legacy=0x7f3874099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:09.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.495+0000 7f3872ffd700 1 -- 172.21.15.1:0/844852128 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387406ec90 con 0x7f38740ade00 2024-03-18T19:05:09.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.495+0000 7f3873fff700 1 --2- 172.21.15.1:0/844852128 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38740ccc20 0x7f387406fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:09.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.495+0000 7f38727fc700 1 -- 172.21.15.1:0/844852128 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355592799 0 0) 0x7f38740b3aa0 con 0x7f3874099a40 2024-03-18T19:05:09.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.496+0000 7f3872ffd700 1 --2- 172.21.15.1:0/844852128 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38740ade00 0x7f38740cc6e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f385c009840 tx=0x7f385c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9940b7dc6acd63fb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:09.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.496+0000 7f38727fc700 1 -- 172.21.15.1:0/844852128 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385c011830 con 0x7f38740ade00 2024-03-18T19:05:09.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.496+0000 7f38727fc700 1 -- 172.21.15.1:0/844852128 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385c011990 con 0x7f38740ade00 2024-03-18T19:05:09.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.496+0000 7f38727fc700 1 -- 172.21.15.1:0/844852128 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385c011d20 con 0x7f38740ade00 2024-03-18T19:05:09.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.496+0000 7f3879c1b700 1 -- 172.21.15.1:0/844852128 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38740ade00 msgr2=0x7f38740cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:09.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.496+0000 7f3879c1b700 1 --2- 172.21.15.1:0/844852128 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38740ade00 0x7f38740cc6e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f385c009840 tx=0x7f385c009b50 comp rx=0 tx=0).stop 2024-03-18T19:05:09.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.496+0000 7f3879c1b700 1 -- 172.21.15.1:0/844852128 shutdown_connections 2024-03-18T19:05:09.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.496+0000 7f3879c1b700 1 --2- 172.21.15.1:0/844852128 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38740ccc20 0x7f387406fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.496+0000 7f3879c1b700 1 --2- 172.21.15.1:0/844852128 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38740ade00 0x7f38740cc6e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.496+0000 7f3879c1b700 1 -- 172.21.15.1:0/844852128 >> 172.21.15.1:0/844852128 conn(0x7f38740b2c50 msgr2=0x7f38740b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:09.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.496+0000 7f3879c1b700 1 -- 172.21.15.1:0/844852128 shutdown_connections 2024-03-18T19:05:09.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.496+0000 7f3879c1b700 1 -- 172.21.15.1:0/844852128 wait complete. 2024-03-18T19:05:09.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.498+0000 7f3879c1b700 1 Processor -- start 2024-03-18T19:05:09.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.498+0000 7f3879c1b700 1 -- start start 2024-03-18T19:05:09.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.498+0000 7f3879c1b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3874099a40 0x7f3874147170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:09.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.498+0000 7f3879c1b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38740ade00 0x7f38740c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:09.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.498+0000 7f3879c1b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38740ccc20 0x7f38740c8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:09.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.499+0000 7f3879c1b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38740c66e0 con 0x7f3874099a40 2024-03-18T19:05:09.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.499+0000 7f3879c1b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38740c5db0 con 0x7f38740ccc20 2024-03-18T19:05:09.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.499+0000 7f3879c1b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38740c6090 con 0x7f38740ade00 2024-03-18T19:05:09.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.499+0000 7f38737fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3874099a40 0x7f3874147170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:09.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.499+0000 7f38737fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3874099a40 0x7f3874147170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42390/0 (socket says 172.21.15.1:42390) 2024-03-18T19:05:09.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.499+0000 7f38737fe700 1 -- 172.21.15.1:0/156639422 learned_addr learned my addr 172.21.15.1:0/156639422 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:09.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.499+0000 7f3872ffd700 1 --2- 172.21.15.1:0/156639422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38740ade00 0x7f38740c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:09.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.499+0000 7f3873fff700 1 --2- 172.21.15.1:0/156639422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38740ccc20 0x7f38740c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:09.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.499+0000 7f38737fe700 1 -- 172.21.15.1:0/156639422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38740ade00 msgr2=0x7f38740c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:09.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.499+0000 7f38737fe700 1 --2- 172.21.15.1:0/156639422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38740ade00 0x7f38740c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.499+0000 7f38737fe700 1 -- 172.21.15.1:0/156639422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38740ccc20 msgr2=0x7f38740c8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:09.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.499+0000 7f38737fe700 1 --2- 172.21.15.1:0/156639422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38740ccc20 0x7f38740c8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.499+0000 7f38737fe700 1 -- 172.21.15.1:0/156639422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f386c009a70 con 0x7f3874099a40 2024-03-18T19:05:09.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.499+0000 7f38737fe700 1 --2- 172.21.15.1:0/156639422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3874099a40 0x7f3874147170 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f38640011e0 tx=0x7f386400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:09.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.499+0000 7f386bfff700 1 -- 172.21.15.1:0/156639422 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3864004c80 con 0x7f3874099a40 2024-03-18T19:05:09.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.499+0000 7f386bfff700 1 -- 172.21.15.1:0/156639422 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3864004e00 con 0x7f3874099a40 2024-03-18T19:05:09.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.499+0000 7f3879c1b700 1 -- 172.21.15.1:0/156639422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385c0094a0 con 0x7f3874099a40 2024-03-18T19:05:09.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.499+0000 7f386bfff700 1 -- 172.21.15.1:0/156639422 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3864007310 con 0x7f3874099a40 2024-03-18T19:05:09.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.499+0000 7f3879c1b700 1 -- 172.21.15.1:0/156639422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f387415aa80 con 0x7f3874099a40 2024-03-18T19:05:09.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.500+0000 7f3879c1b700 1 -- 172.21.15.1:0/156639422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38740676a0 con 0x7f3874099a40 2024-03-18T19:05:09.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.501+0000 7f386bfff700 1 -- 172.21.15.1:0/156639422 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 27) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f3864007470 con 0x7f3874099a40 2024-03-18T19:05:09.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.501+0000 7f386bfff700 1 -- 172.21.15.1:0/156639422 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6717+0+0 (secure 0 0 0) 0x7f3864088f60 con 0x7f3874099a40 2024-03-18T19:05:09.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.505+0000 7f386bfff700 1 -- 172.21.15.1:0/156639422 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f38640bc580 con 0x7f3874099a40 2024-03-18T19:05:09.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.754+0000 7f3879c1b700 1 -- 172.21.15.1:0/156639422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f38740c7e20 con 0x7f3874099a40 2024-03-18T19:05:09.750 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-18T19:05:09.755+0000 7fbdbeb46700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T19:05:09.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.766+0000 7f386bfff700 1 -- 172.21.15.1:0/156639422 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) v1 ==== 161+0+0 (secure 0 0 0) 0x7f38640541a0 con 0x7f3874099a40 2024-03-18T19:05:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.770+0000 7f3879c1b700 1 -- 172.21.15.1:0/156639422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3874099a40 msgr2=0x7f3874147170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.770+0000 7f3879c1b700 1 --2- 172.21.15.1:0/156639422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3874099a40 0x7f3874147170 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f38640011e0 tx=0x7f386400cae0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.770+0000 7f3879c1b700 1 -- 172.21.15.1:0/156639422 shutdown_connections 2024-03-18T19:05:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.770+0000 7f3879c1b700 1 --2- 172.21.15.1:0/156639422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38740ccc20 0x7f38740c8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.770+0000 7f3879c1b700 1 --2- 172.21.15.1:0/156639422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3874099a40 0x7f3874147170 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.770+0000 7f3879c1b700 1 --2- 172.21.15.1:0/156639422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38740ade00 0x7f38740c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.770+0000 7f3879c1b700 1 -- 172.21.15.1:0/156639422 >> 172.21.15.1:0/156639422 conn(0x7f38740b2c50 msgr2=0x7f38740aec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.770+0000 7f3879c1b700 1 -- 172.21.15.1:0/156639422 shutdown_connections 2024-03-18T19:05:09.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.770+0000 7f3879c1b700 1 -- 172.21.15.1:0/156639422 wait complete. 2024-03-18T19:05:09.767 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T19:05:09.782 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T19:05:09.807 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T19:05:09.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.914+0000 7fac6f621700 1 Processor -- start 2024-03-18T19:05:09.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.914+0000 7fac6f621700 1 -- start start 2024-03-18T19:05:09.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.915+0000 7fac6f621700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fac6815eba0 0x7fac680cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:09.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.915+0000 7fac6f621700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac680b0500 0x7fac680cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:09.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.915+0000 7fac6f621700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6815ef80 con 0x7fac680cfdf0 2024-03-18T19:05:09.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.915+0000 7fac6f621700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac680cc9a0 con 0x7fac6815eba0 2024-03-18T19:05:09.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.915+0000 7fac6f621700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac680ccc80 con 0x7fac680b0500 2024-03-18T19:05:09.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.915+0000 7fac6dbbe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fac680cfdf0 0x7fac68099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42406/0 (socket says 172.21.15.1:42406) 2024-03-18T19:05:09.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.915+0000 7fac6dbbe700 1 -- 172.21.15.1:0/3966346401 learned_addr learned my addr 172.21.15.1:0/3966346401 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:09.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.915+0000 7fac6cbbc700 1 --2- 172.21.15.1:0/3966346401 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac680b0500 0x7fac680cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:09.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.915+0000 7fac6d3bd700 1 --2- 172.21.15.1:0/3966346401 >> v2:172.21.15.165:3300/0 conn(0x7fac6815eba0 0x7fac680cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:09.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.915+0000 7fac5bfff700 1 -- 172.21.15.1:0/3966346401 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557203133 0 0) 0x7fac6815ef80 con 0x7fac680cfdf0 2024-03-18T19:05:09.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.915+0000 7fac5bfff700 1 -- 172.21.15.1:0/3966346401 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac540035b0 con 0x7fac680cfdf0 2024-03-18T19:05:09.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.915+0000 7fac5bfff700 1 -- 172.21.15.1:0/3966346401 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3317778280 0 0) 0x7fac540035b0 con 0x7fac680cfdf0 2024-03-18T19:05:09.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.915+0000 7fac6cbbc700 1 -- 172.21.15.1:0/3966346401 >> v2:172.21.15.165:3300/0 conn(0x7fac6815eba0 msgr2=0x7fac680cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:09.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.915+0000 7fac6cbbc700 1 --2- 172.21.15.1:0/3966346401 >> v2:172.21.15.165:3300/0 conn(0x7fac6815eba0 0x7fac680cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.915+0000 7fac6cbbc700 1 -- 172.21.15.1:0/3966346401 >> v1:172.21.15.1:6789/0 conn(0x7fac680cfdf0 legacy=0x7fac68099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:09.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.915+0000 7fac6cbbc700 1 -- 172.21.15.1:0/3966346401 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac680ccf60 con 0x7fac680b0500 2024-03-18T19:05:09.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.915+0000 7fac6d3bd700 1 --2- 172.21.15.1:0/3966346401 >> v2:172.21.15.165:3300/0 conn(0x7fac6815eba0 0x7fac680cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:09.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.916+0000 7fac6cbbc700 1 --2- 172.21.15.1:0/3966346401 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac680b0500 0x7fac680cf8b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fac64009840 tx=0x7fac64009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50f5a0d89850a1b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:09.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.916+0000 7fac5bfff700 1 -- 172.21.15.1:0/3966346401 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac600019f0 con 0x7fac680b0500 2024-03-18T19:05:09.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.916+0000 7fac5bfff700 1 -- 172.21.15.1:0/3966346401 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac64004030 con 0x7fac680b0500 2024-03-18T19:05:09.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.916+0000 7fac5bfff700 1 -- 172.21.15.1:0/3966346401 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac640043c0 con 0x7fac680b0500 2024-03-18T19:05:09.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.916+0000 7fac6f621700 1 -- 172.21.15.1:0/3966346401 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac680b0500 msgr2=0x7fac680cf8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:09.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.916+0000 7fac6f621700 1 --2- 172.21.15.1:0/3966346401 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac680b0500 0x7fac680cf8b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fac64009840 tx=0x7fac64009b50 comp rx=0 tx=0).stop 2024-03-18T19:05:09.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.916+0000 7fac6f621700 1 -- 172.21.15.1:0/3966346401 shutdown_connections 2024-03-18T19:05:09.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.916+0000 7fac6f621700 1 --2- 172.21.15.1:0/3966346401 >> v2:172.21.15.165:3300/0 conn(0x7fac6815eba0 0x7fac680cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.916+0000 7fac6f621700 1 --2- 172.21.15.1:0/3966346401 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac680b0500 0x7fac680cf8b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.916+0000 7fac6f621700 1 -- 172.21.15.1:0/3966346401 >> 172.21.15.1:0/3966346401 conn(0x7fac680b2bd0 msgr2=0x7fac680b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:09.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.916+0000 7fac6f621700 1 -- 172.21.15.1:0/3966346401 shutdown_connections 2024-03-18T19:05:09.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.916+0000 7fac6f621700 1 -- 172.21.15.1:0/3966346401 wait complete. 2024-03-18T19:05:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.917+0000 7fac6f621700 1 Processor -- start 2024-03-18T19:05:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.918+0000 7fac6f621700 1 -- start start 2024-03-18T19:05:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.918+0000 7fac6f621700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac680b0500 0x7fac6815a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.918+0000 7fac6f621700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac680cfdf0 0x7fac6806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.918+0000 7fac6f621700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac6815eba0 0x7fac680c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.918+0000 7fac6f621700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6815a970 con 0x7fac680b0500 2024-03-18T19:05:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.918+0000 7fac6f621700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6806c2c0 con 0x7fac680cfdf0 2024-03-18T19:05:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.918+0000 7fac6f621700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac680c5770 con 0x7fac6815eba0 2024-03-18T19:05:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.918+0000 7fac6d3bd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac680b0500 0x7fac6815a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.918+0000 7fac6d3bd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac680b0500 0x7fac6815a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42418/0 (socket says 172.21.15.1:42418) 2024-03-18T19:05:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.918+0000 7fac6d3bd700 1 -- 172.21.15.1:0/335424370 learned_addr learned my addr 172.21.15.1:0/335424370 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.918+0000 7fac6cbbc700 1 --2- 172.21.15.1:0/335424370 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac680cfdf0 0x7fac6806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:09.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.919+0000 7fac6dbbe700 1 --2- 172.21.15.1:0/335424370 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac6815eba0 0x7fac680c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:09.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.919+0000 7fac6d3bd700 1 -- 172.21.15.1:0/335424370 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac6815eba0 msgr2=0x7fac680c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:09.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.919+0000 7fac6d3bd700 1 --2- 172.21.15.1:0/335424370 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac6815eba0 0x7fac680c9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.919+0000 7fac6d3bd700 1 -- 172.21.15.1:0/335424370 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac680cfdf0 msgr2=0x7fac6806cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:09.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.919+0000 7fac6d3bd700 1 --2- 172.21.15.1:0/335424370 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac680cfdf0 0x7fac6806cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:09.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.919+0000 7fac6d3bd700 1 -- 172.21.15.1:0/335424370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c0094a0 con 0x7fac680b0500 2024-03-18T19:05:09.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.919+0000 7fac6d3bd700 1 --2- 172.21.15.1:0/335424370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac680b0500 0x7fac6815a430 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fac5c00ebe0 tx=0x7fac5c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:09.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.919+0000 7fac59ffb700 1 -- 172.21.15.1:0/335424370 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c009760 con 0x7fac680b0500 2024-03-18T19:05:09.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.919+0000 7fac59ffb700 1 -- 172.21.15.1:0/335424370 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5c00c850 con 0x7fac680b0500 2024-03-18T19:05:09.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.919+0000 7fac6f621700 1 -- 172.21.15.1:0/335424370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac640094a0 con 0x7fac680b0500 2024-03-18T19:05:09.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.919+0000 7fac59ffb700 1 -- 172.21.15.1:0/335424370 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c00cbe0 con 0x7fac680b0500 2024-03-18T19:05:09.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.919+0000 7fac6f621700 1 -- 172.21.15.1:0/335424370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6806c7f0 con 0x7fac680b0500 2024-03-18T19:05:09.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.920+0000 7fac6f621700 1 -- 172.21.15.1:0/335424370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac680676a0 con 0x7fac680b0500 2024-03-18T19:05:09.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.921+0000 7fac59ffb700 1 -- 172.21.15.1:0/335424370 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 27) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fac5c00cd40 con 0x7fac680b0500 2024-03-18T19:05:09.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.921+0000 7fac59ffb700 1 -- 172.21.15.1:0/335424370 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6803+0+0 (secure 0 0 0) 0x7fac5c0891a0 con 0x7fac680b0500 2024-03-18T19:05:09.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:09.926+0000 7fac59ffb700 1 -- 172.21.15.1:0/335424370 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fac5c054380 con 0x7fac680b0500 2024-03-18T19:05:10.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.010+0000 7fac59ffb700 1 -- 172.21.15.1:0/335424370 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 28) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fac5c053e30 con 0x7fac680b0500 2024-03-18T19:05:10.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.010+0000 7fac59ffb700 1 --2- 172.21.15.1:0/335424370 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac4c036150 0x7fac4c072a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:10.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.011+0000 7fac6cbbc700 1 --2- 172.21.15.1:0/335424370 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac4c036150 0x7fac4c072a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:10.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.013+0000 7fac6cbbc700 1 --2- 172.21.15.1:0/335424370 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac4c036150 0x7fac4c072a50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fac64009790 tx=0x7fac64008030 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.179+0000 7fac6f621700 1 -- 172.21.15.1:0/335424370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fac680705e0 con 0x7fac680b0500 2024-03-18T19:05:10.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.769+0000 7fac59ffb700 1 -- 172.21.15.1:0/335424370 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7fac5c087320 con 0x7fac680b0500 2024-03-18T19:05:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.771+0000 7fac6f621700 1 -- 172.21.15.1:0/335424370 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac4c036150 msgr2=0x7fac4c072a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.771+0000 7fac6f621700 1 --2- 172.21.15.1:0/335424370 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac4c036150 0x7fac4c072a50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fac64009790 tx=0x7fac64008030 comp rx=0 tx=0).stop 2024-03-18T19:05:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.772+0000 7fac6f621700 1 -- 172.21.15.1:0/335424370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac680b0500 msgr2=0x7fac6815a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.772+0000 7fac6f621700 1 --2- 172.21.15.1:0/335424370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac680b0500 0x7fac6815a430 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fac5c00ebe0 tx=0x7fac5c00eef0 comp rx=0 tx=0).stop 2024-03-18T19:05:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.772+0000 7fac6f621700 1 -- 172.21.15.1:0/335424370 shutdown_connections 2024-03-18T19:05:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.772+0000 7fac6f621700 1 --2- 172.21.15.1:0/335424370 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac4c036150 0x7fac4c072a50 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.772+0000 7fac6f621700 1 --2- 172.21.15.1:0/335424370 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac6815eba0 0x7fac680c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.772+0000 7fac6f621700 1 --2- 172.21.15.1:0/335424370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac680b0500 0x7fac6815a430 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:10.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.772+0000 7fac6f621700 1 --2- 172.21.15.1:0/335424370 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac680cfdf0 0x7fac6806cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:10.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.772+0000 7fac6f621700 1 -- 172.21.15.1:0/335424370 >> 172.21.15.1:0/335424370 conn(0x7fac680b2bd0 msgr2=0x7fac6815c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:10.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.772+0000 7fac6f621700 1 -- 172.21.15.1:0/335424370 shutdown_connections 2024-03-18T19:05:10.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.772+0000 7fac6f621700 1 -- 172.21.15.1:0/335424370 wait complete. 2024-03-18T19:05:10.783 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T19:05:10.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.965+0000 7f7dcee99700 1 Processor -- start 2024-03-18T19:05:10.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.965+0000 7f7dcee99700 1 -- start start 2024-03-18T19:05:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.966+0000 7f7dcee99700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7dc815d640 0x7f7dc80d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.966+0000 7f7dcee99700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7dc80d0510 0x7f7dc80d0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.966+0000 7f7dcee99700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc8070210 0x7f7dc8070680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.966+0000 7f7dcee99700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc80d0ed0 con 0x7f7dc815d640 2024-03-18T19:05:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.966+0000 7f7dcee99700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc806cc20 con 0x7f7dc80d0510 2024-03-18T19:05:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.966+0000 7f7dcee99700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc815da20 con 0x7f7dc8070210 2024-03-18T19:05:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.966+0000 7f7dccc35700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7dc815d640 0x7f7dc80d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.966+0000 7f7dccc35700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7dc815d640 0x7f7dc80d1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42430/0 (socket says 172.21.15.1:42430) 2024-03-18T19:05:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.966+0000 7f7dccc35700 1 -- 172.21.15.1:0/2754111409 learned_addr learned my addr 172.21.15.1:0/2754111409 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.966+0000 7f7dc7fff700 1 --2- 172.21.15.1:0/2754111409 >> v2:172.21.15.165:3300/0 conn(0x7f7dc80d0510 0x7f7dc80d0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.966+0000 7f7dcd436700 1 --2- 172.21.15.1:0/2754111409 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc8070210 0x7f7dc8070680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.966+0000 7f7dccc35700 1 -- 172.21.15.1:0/2754111409 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc8070210 msgr2=0x7f7dc8070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:10.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.966+0000 7f7dccc35700 1 --2- 172.21.15.1:0/2754111409 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc8070210 0x7f7dc8070680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:10.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.966+0000 7f7dccc35700 1 -- 172.21.15.1:0/2754111409 >> v2:172.21.15.165:3300/0 conn(0x7f7dc80d0510 msgr2=0x7f7dc80d0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:10.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.966+0000 7f7dccc35700 1 --2- 172.21.15.1:0/2754111409 >> v2:172.21.15.165:3300/0 conn(0x7f7dc80d0510 0x7f7dc80d0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:10.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.966+0000 7f7dccc35700 1 -- 172.21.15.1:0/2754111409 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc80d2040 con 0x7f7dc815d640 2024-03-18T19:05:10.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.966+0000 7f7dccc35700 1 --2- 172.21.15.1:0/2754111409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc815d640 0x7f7dc80d1b00 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f7dc806d5d0 tx=0x7f7dbc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba910717248fdfdb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:10.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.967+0000 7f7dc77fe700 1 -- 172.21.15.1:0/2754111409 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc00b870 con 0x7f7dc815d640 2024-03-18T19:05:10.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.967+0000 7f7dc77fe700 1 -- 172.21.15.1:0/2754111409 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dbc00b9d0 con 0x7f7dc815d640 2024-03-18T19:05:10.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.967+0000 7f7dc77fe700 1 -- 172.21.15.1:0/2754111409 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc00bd60 con 0x7f7dc815d640 2024-03-18T19:05:10.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.967+0000 7f7dcee99700 1 -- 172.21.15.1:0/2754111409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc815d640 msgr2=0x7f7dc80d1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:10.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.967+0000 7f7dcee99700 1 --2- 172.21.15.1:0/2754111409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc815d640 0x7f7dc80d1b00 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f7dc806d5d0 tx=0x7f7dbc009bd0 comp rx=0 tx=0).stop 2024-03-18T19:05:10.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.967+0000 7f7dcee99700 1 -- 172.21.15.1:0/2754111409 shutdown_connections 2024-03-18T19:05:10.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.967+0000 7f7dcee99700 1 --2- 172.21.15.1:0/2754111409 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc8070210 0x7f7dc8070680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:10.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.967+0000 7f7dcee99700 1 --2- 172.21.15.1:0/2754111409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc815d640 0x7f7dc80d1b00 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:10.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.967+0000 7f7dcee99700 1 --2- 172.21.15.1:0/2754111409 >> v2:172.21.15.165:3300/0 conn(0x7f7dc80d0510 0x7f7dc80d0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:10.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.967+0000 7f7dcee99700 1 -- 172.21.15.1:0/2754111409 >> 172.21.15.1:0/2754111409 conn(0x7f7dc80b2e80 msgr2=0x7f7dc80b3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:10.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.967+0000 7f7dcee99700 1 -- 172.21.15.1:0/2754111409 shutdown_connections 2024-03-18T19:05:10.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.967+0000 7f7dcee99700 1 -- 172.21.15.1:0/2754111409 wait complete. 2024-03-18T19:05:10.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.968+0000 7f7dcee99700 1 Processor -- start 2024-03-18T19:05:10.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.968+0000 7f7dcee99700 1 -- start start 2024-03-18T19:05:10.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.969+0000 7f7dcee99700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7dc8070210 0x7f7dc81f3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:10.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.969+0000 7f7dcee99700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc80d0510 0x7f7dc80cebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:10.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.969+0000 7f7dcee99700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc815d640 0x7f7dc81f6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:10.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.969+0000 7f7dcee99700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc815f6e0 con 0x7f7dc80d0510 2024-03-18T19:05:10.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.969+0000 7f7dcee99700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc80ccc80 con 0x7f7dc8070210 2024-03-18T19:05:10.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.969+0000 7f7dcee99700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc80ccf60 con 0x7f7dc815d640 2024-03-18T19:05:10.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.969+0000 7f7dc7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc80d0510 0x7f7dc80cebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:10.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.969+0000 7f7dc7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc80d0510 0x7f7dc80cebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42446/0 (socket says 172.21.15.1:42446) 2024-03-18T19:05:10.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.969+0000 7f7dc7fff700 1 -- 172.21.15.1:0/2773788597 learned_addr learned my addr 172.21.15.1:0/2773788597 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:10.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.970+0000 7f7dcd436700 1 --2- 172.21.15.1:0/2773788597 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc815d640 0x7f7dc81f6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:10.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.970+0000 7f7dccc35700 1 --2- 172.21.15.1:0/2773788597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7dc8070210 0x7f7dc81f3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:10.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.970+0000 7f7dc7fff700 1 -- 172.21.15.1:0/2773788597 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc815d640 msgr2=0x7f7dc81f6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:10.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.970+0000 7f7dc7fff700 1 --2- 172.21.15.1:0/2773788597 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc815d640 0x7f7dc81f6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:10.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.970+0000 7f7dc7fff700 1 -- 172.21.15.1:0/2773788597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7dc8070210 msgr2=0x7f7dc81f3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:10.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.970+0000 7f7dc7fff700 1 --2- 172.21.15.1:0/2773788597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7dc8070210 0x7f7dc81f3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:10.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.970+0000 7f7dc7fff700 1 -- 172.21.15.1:0/2773788597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dbc009550 con 0x7f7dc80d0510 2024-03-18T19:05:10.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.970+0000 7f7dccc35700 1 --2- 172.21.15.1:0/2773788597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7dc8070210 0x7f7dc81f3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:10.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.970+0000 7f7dc7fff700 1 --2- 172.21.15.1:0/2773788597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc80d0510 0x7f7dc80cebf0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f7db400e810 tx=0x7f7db400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:10.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.970+0000 7f7dc57fa700 1 -- 172.21.15.1:0/2773788597 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db4013610 con 0x7f7dc80d0510 2024-03-18T19:05:10.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.970+0000 7f7dc57fa700 1 -- 172.21.15.1:0/2773788597 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db4013770 con 0x7f7dc80d0510 2024-03-18T19:05:10.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.970+0000 7f7dcee99700 1 -- 172.21.15.1:0/2773788597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc80cd240 con 0x7f7dc80d0510 2024-03-18T19:05:10.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.970+0000 7f7dc57fa700 1 -- 172.21.15.1:0/2773788597 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db4013b20 con 0x7f7dc80d0510 2024-03-18T19:05:10.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.970+0000 7f7dcee99700 1 -- 172.21.15.1:0/2773788597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc815f200 con 0x7f7dc80d0510 2024-03-18T19:05:10.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.971+0000 7f7dcee99700 1 -- 172.21.15.1:0/2773788597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc80676a0 con 0x7f7dc80d0510 2024-03-18T19:05:10.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.972+0000 7f7dc57fa700 1 -- 172.21.15.1:0/2773788597 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 28) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f7db4013c80 con 0x7f7dc80d0510 2024-03-18T19:05:10.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.972+0000 7f7dc57fa700 1 --2- 172.21.15.1:0/2773788597 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7dac06aae0 0x7f7dac06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:10.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.972+0000 7f7dc57fa700 1 -- 172.21.15.1:0/2773788597 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6803+0+0 (secure 0 0 0) 0x7f7db4089070 con 0x7f7dc80d0510 2024-03-18T19:05:10.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.972+0000 7f7dccc35700 1 --2- 172.21.15.1:0/2773788597 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7dac06aae0 0x7f7dac06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:10.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.972+0000 7f7dccc35700 1 --2- 172.21.15.1:0/2773788597 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7dac06aae0 0x7f7dac06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc014040 tx=0x7f7dbc008030 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:10.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:10.977+0000 7f7dc57fa700 1 -- 172.21.15.1:0/2773788597 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7db40540c0 con 0x7f7dc80d0510 2024-03-18T19:05:11.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:11.196+0000 7f7dcee99700 1 -- 172.21.15.1:0/2773788597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7dc80b4da0 con 0x7f7dc80d0510 2024-03-18T19:05:12.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.018+0000 7f7dc57fa700 1 -- 172.21.15.1:0/2773788597 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 29) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f7db4054490 con 0x7f7dc80d0510 2024-03-18T19:05:12.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.050+0000 7f7dc57fa700 1 -- 172.21.15.1:0/2773788597 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v79) v1 ==== 181+0+0 (secure 0 0 0) 0x7f7db4081390 con 0x7f7dc80d0510 2024-03-18T19:05:12.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.056+0000 7f7dcee99700 1 -- 172.21.15.1:0/2773788597 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7dac06aae0 msgr2=0x7f7dac06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:12.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.056+0000 7f7dcee99700 1 --2- 172.21.15.1:0/2773788597 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7dac06aae0 0x7f7dac06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc014040 tx=0x7f7dbc008030 comp rx=0 tx=0).stop 2024-03-18T19:05:12.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.056+0000 7f7dcee99700 1 -- 172.21.15.1:0/2773788597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc80d0510 msgr2=0x7f7dc80cebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:12.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.056+0000 7f7dcee99700 1 --2- 172.21.15.1:0/2773788597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc80d0510 0x7f7dc80cebf0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f7db400e810 tx=0x7f7db400eb20 comp rx=0 tx=0).stop 2024-03-18T19:05:12.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.056+0000 7f7dcee99700 1 -- 172.21.15.1:0/2773788597 shutdown_connections 2024-03-18T19:05:12.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.056+0000 7f7dcee99700 1 --2- 172.21.15.1:0/2773788597 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7dac06aae0 0x7f7dac06cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:12.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.056+0000 7f7dcee99700 1 --2- 172.21.15.1:0/2773788597 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7dc815d640 0x7f7dc81f6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:12.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.056+0000 7f7dcee99700 1 --2- 172.21.15.1:0/2773788597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7dc8070210 0x7f7dc81f3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:12.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.056+0000 7f7dcee99700 1 --2- 172.21.15.1:0/2773788597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7dc80d0510 0x7f7dc80cebf0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:12.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.056+0000 7f7dcee99700 1 -- 172.21.15.1:0/2773788597 >> 172.21.15.1:0/2773788597 conn(0x7f7dc80b2e80 msgr2=0x7f7dc80b17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:12.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.056+0000 7f7dcee99700 1 -- 172.21.15.1:0/2773788597 shutdown_connections 2024-03-18T19:05:12.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.057+0000 7f7dcee99700 1 -- 172.21.15.1:0/2773788597 wait complete. 2024-03-18T19:05:12.054 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-18T19:05:12.070 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T19:05:12.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.209+0000 7f4391e0e700 1 Processor -- start 2024-03-18T19:05:12.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.209+0000 7f4391e0e700 1 -- start start 2024-03-18T19:05:12.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.210+0000 7f4391e0e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f438c0cf9e0 0x7f438c06db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:12.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.210+0000 7f4391e0e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f438c161780 0x7f438c15cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:12.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.210+0000 7f4391e0e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f438c0b2bb0 con 0x7f438c0cfdc0 2024-03-18T19:05:12.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.210+0000 7f4391e0e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f438c06d090 con 0x7f438c0cf9e0 2024-03-18T19:05:12.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.210+0000 7f4391e0e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f438c161c30 con 0x7f438c161780 2024-03-18T19:05:12.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.210+0000 7f438b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f438c0cf9e0 0x7f438c06db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:12.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.210+0000 7f438b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f438c0cf9e0 0x7f438c06db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42482/0 (socket says 172.21.15.1:42482) 2024-03-18T19:05:12.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.210+0000 7f438b7fe700 1 -- 172.21.15.1:0/2740464922 learned_addr learned my addr 172.21.15.1:0/2740464922 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:12.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.210+0000 7f438bfff700 1 --2- 172.21.15.1:0/2740464922 >> v2:172.21.15.165:3300/0 conn(0x7f438c161780 0x7f438c15cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:12.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.210+0000 7f438b7fe700 1 -- 172.21.15.1:0/2740464922 >> v2:172.21.15.165:3300/0 conn(0x7f438c161780 msgr2=0x7f438c15cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:12.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.210+0000 7f438b7fe700 1 --2- 172.21.15.1:0/2740464922 >> v2:172.21.15.165:3300/0 conn(0x7f438c161780 0x7f438c15cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:12.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.210+0000 7f438b7fe700 1 -- 172.21.15.1:0/2740464922 >> v1:172.21.15.1:6789/0 conn(0x7f438c0cfdc0 legacy=0x7f438c161070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:12.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.210+0000 7f438b7fe700 1 -- 172.21.15.1:0/2740464922 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438c06d300 con 0x7f438c0cf9e0 2024-03-18T19:05:12.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.210+0000 7f438b7fe700 1 --2- 172.21.15.1:0/2740464922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f438c0cf9e0 0x7f438c06db00 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f438c0b43d0 tx=0x7f437c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2366d8cedcacfecd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:12.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.211+0000 7f438a7fc700 1 -- 172.21.15.1:0/2740464922 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c00b870 con 0x7f438c0cf9e0 2024-03-18T19:05:12.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.211+0000 7f438a7fc700 1 -- 172.21.15.1:0/2740464922 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437c00b9d0 con 0x7f438c0cf9e0 2024-03-18T19:05:12.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.211+0000 7f438a7fc700 1 -- 172.21.15.1:0/2740464922 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c00bd60 con 0x7f438c0cf9e0 2024-03-18T19:05:12.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.211+0000 7f4391e0e700 1 -- 172.21.15.1:0/2740464922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f438c0cf9e0 msgr2=0x7f438c06db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:12.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.211+0000 7f4391e0e700 1 --2- 172.21.15.1:0/2740464922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f438c0cf9e0 0x7f438c06db00 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f438c0b43d0 tx=0x7f437c009bd0 comp rx=0 tx=0).stop 2024-03-18T19:05:12.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.211+0000 7f4391e0e700 1 -- 172.21.15.1:0/2740464922 shutdown_connections 2024-03-18T19:05:12.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.211+0000 7f4391e0e700 1 --2- 172.21.15.1:0/2740464922 >> v2:172.21.15.165:3300/0 conn(0x7f438c161780 0x7f438c15cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:12.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.211+0000 7f4391e0e700 1 --2- 172.21.15.1:0/2740464922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f438c0cf9e0 0x7f438c06db00 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:12.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.211+0000 7f4391e0e700 1 -- 172.21.15.1:0/2740464922 >> 172.21.15.1:0/2740464922 conn(0x7f438c06a750 msgr2=0x7f438c06ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:12.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.211+0000 7f4391e0e700 1 -- 172.21.15.1:0/2740464922 shutdown_connections 2024-03-18T19:05:12.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.211+0000 7f4391e0e700 1 -- 172.21.15.1:0/2740464922 wait complete. 2024-03-18T19:05:12.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.212+0000 7f4391e0e700 1 Processor -- start 2024-03-18T19:05:12.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.213+0000 7f4391e0e700 1 -- start start 2024-03-18T19:05:12.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.213+0000 7f4391e0e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f438c0cf9e0 0x7f438c0c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:12.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.213+0000 7f4391e0e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f438c0cfdc0 0x7f438c15b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:12.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.213+0000 7f4391e0e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f438c161780 0x7f438c14bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:12.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.213+0000 7f4391e0e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c14fe30 con 0x7f438c161780 2024-03-18T19:05:12.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.213+0000 7f4391e0e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c0afa00 con 0x7f438c0cfdc0 2024-03-18T19:05:12.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.213+0000 7f4391e0e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438c0aedf0 con 0x7f438c0cf9e0 2024-03-18T19:05:12.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.213+0000 7f438bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f438c161780 0x7f438c14bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:12.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.213+0000 7f438bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f438c161780 0x7f438c14bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42490/0 (socket says 172.21.15.1:42490) 2024-03-18T19:05:12.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.213+0000 7f438bfff700 1 -- 172.21.15.1:0/2330407682 learned_addr learned my addr 172.21.15.1:0/2330407682 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:12.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.214+0000 7f438b7fe700 1 --2- 172.21.15.1:0/2330407682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f438c0cf9e0 0x7f438c0c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:12.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.214+0000 7f438affd700 1 --2- 172.21.15.1:0/2330407682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f438c0cfdc0 0x7f438c15b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:12.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.214+0000 7f438bfff700 1 -- 172.21.15.1:0/2330407682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f438c0cf9e0 msgr2=0x7f438c0c47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:12.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.214+0000 7f438bfff700 1 --2- 172.21.15.1:0/2330407682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f438c0cf9e0 0x7f438c0c47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:12.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.214+0000 7f438bfff700 1 -- 172.21.15.1:0/2330407682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f438c0cfdc0 msgr2=0x7f438c15b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:12.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.214+0000 7f438bfff700 1 --2- 172.21.15.1:0/2330407682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f438c0cfdc0 0x7f438c15b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:12.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.214+0000 7f438bfff700 1 -- 172.21.15.1:0/2330407682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c009550 con 0x7f438c161780 2024-03-18T19:05:12.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.214+0000 7f438affd700 1 --2- 172.21.15.1:0/2330407682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f438c0cfdc0 0x7f438c15b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:12.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.214+0000 7f438bfff700 1 --2- 172.21.15.1:0/2330407682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f438c161780 0x7f438c14bfc0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f4384007e60 tx=0x7f438400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:12.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.214+0000 7f4373fff700 1 -- 172.21.15.1:0/2330407682 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438400ede0 con 0x7f438c161780 2024-03-18T19:05:12.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.214+0000 7f4373fff700 1 -- 172.21.15.1:0/2330407682 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4384014420 con 0x7f438c161780 2024-03-18T19:05:12.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.214+0000 7f4373fff700 1 -- 172.21.15.1:0/2330407682 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43840147d0 con 0x7f438c161780 2024-03-18T19:05:12.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.214+0000 7f4391e0e700 1 -- 172.21.15.1:0/2330407682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f438c0af0d0 con 0x7f438c161780 2024-03-18T19:05:12.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.214+0000 7f4391e0e700 1 -- 172.21.15.1:0/2330407682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438c0af690 con 0x7f438c161780 2024-03-18T19:05:12.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.215+0000 7f4391e0e700 1 -- 172.21.15.1:0/2330407682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f438c0676a0 con 0x7f438c161780 2024-03-18T19:05:12.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.216+0000 7f4373fff700 1 -- 172.21.15.1:0/2330407682 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 29) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f4384014930 con 0x7f438c161780 2024-03-18T19:05:12.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.216+0000 7f4373fff700 1 --2- 172.21.15.1:0/2330407682 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f437806ad20 0x7f437806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:12.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.216+0000 7f4373fff700 1 -- 172.21.15.1:0/2330407682 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6365+0+0 (secure 0 0 0) 0x7f4384089d10 con 0x7f438c161780 2024-03-18T19:05:12.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.216+0000 7f438b7fe700 1 --2- 172.21.15.1:0/2330407682 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f437806ad20 0x7f437806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:12.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.216+0000 7f438b7fe700 1 --2- 172.21.15.1:0/2330407682 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f437806ad20 0x7f437806d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f437c000c00 tx=0x7f437c008030 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:12.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.221+0000 7f4373fff700 1 -- 172.21.15.1:0/2330407682 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4384054c70 con 0x7f438c161780 2024-03-18T19:05:12.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:12.424+0000 7f4391e0e700 1 -- 172.21.15.1:0/2330407682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f438c073b80 con 0x7f438c161780 2024-03-18T19:05:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.140+0000 7f4373fff700 1 -- 172.21.15.1:0/2330407682 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f438408f020 con 0x7f438c161780 2024-03-18T19:05:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.140+0000 7f4373fff700 1 -- 172.21.15.1:0/2330407682 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v80) v1 ==== 169+0+0 (secure 0 0 0) 0x7f43840510f0 con 0x7f438c161780 2024-03-18T19:05:13.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.144+0000 7f4391e0e700 1 -- 172.21.15.1:0/2330407682 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f437806ad20 msgr2=0x7f437806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:13.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.144+0000 7f4391e0e700 1 --2- 172.21.15.1:0/2330407682 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f437806ad20 0x7f437806d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f437c000c00 tx=0x7f437c008030 comp rx=0 tx=0).stop 2024-03-18T19:05:13.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.144+0000 7f4391e0e700 1 -- 172.21.15.1:0/2330407682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f438c161780 msgr2=0x7f438c14bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:13.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.144+0000 7f4391e0e700 1 --2- 172.21.15.1:0/2330407682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f438c161780 0x7f438c14bfc0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f4384007e60 tx=0x7f438400e3c0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.145+0000 7f4391e0e700 1 -- 172.21.15.1:0/2330407682 shutdown_connections 2024-03-18T19:05:13.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.145+0000 7f4391e0e700 1 --2- 172.21.15.1:0/2330407682 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f437806ad20 0x7f437806d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.145+0000 7f4391e0e700 1 --2- 172.21.15.1:0/2330407682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f438c0cf9e0 0x7f438c0c47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.145+0000 7f4391e0e700 1 --2- 172.21.15.1:0/2330407682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f438c161780 0x7f438c14bfc0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.145+0000 7f4391e0e700 1 --2- 172.21.15.1:0/2330407682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f438c0cfdc0 0x7f438c15b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.145+0000 7f4391e0e700 1 -- 172.21.15.1:0/2330407682 >> 172.21.15.1:0/2330407682 conn(0x7f438c06a750 msgr2=0x7f438c15dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:13.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.145+0000 7f4391e0e700 1 -- 172.21.15.1:0/2330407682 shutdown_connections 2024-03-18T19:05:13.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.145+0000 7f4391e0e700 1 -- 172.21.15.1:0/2330407682 wait complete. 2024-03-18T19:05:13.142 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-18T19:05:13.156 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-18T19:05:13.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.294+0000 7f5b0ba4c700 1 Processor -- start 2024-03-18T19:05:13.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.294+0000 7f5b0ba4c700 1 -- start start 2024-03-18T19:05:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.295+0000 7f5b0ba4c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b0415d5b0 0x7f5b0415d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.295+0000 7f5b0ba4c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5b04099a40 0x7f5b04099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.295+0000 7f5b0ba4c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b0409a830 con 0x7f5b0409a3e0 2024-03-18T19:05:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.295+0000 7f5b0ba4c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b040cbbb0 con 0x7f5b04099a40 2024-03-18T19:05:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.295+0000 7f5b0ba4c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b040b3b00 con 0x7f5b0415d5b0 2024-03-18T19:05:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.295+0000 7f5b08fe7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5b04099a40 0x7f5b04099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.295+0000 7f5b08fe7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5b04099a40 0x7f5b04099ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42494/0 (socket says 172.21.15.1:42494) 2024-03-18T19:05:13.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.295+0000 7f5b08fe7700 1 -- 172.21.15.1:0/3869325590 learned_addr learned my addr 172.21.15.1:0/3869325590 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:13.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.295+0000 7f5b097e8700 1 --2- 172.21.15.1:0/3869325590 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b0415d5b0 0x7f5b0415d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:13.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.295+0000 7f5b08fe7700 1 -- 172.21.15.1:0/3869325590 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b0415d5b0 msgr2=0x7f5b0415d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:13.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.295+0000 7f5b08fe7700 1 --2- 172.21.15.1:0/3869325590 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b0415d5b0 0x7f5b0415d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.295+0000 7f5b08fe7700 1 -- 172.21.15.1:0/3869325590 >> v1:172.21.15.1:6789/0 conn(0x7f5b0409a3e0 legacy=0x7f5b0406ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:13.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.295+0000 7f5b08fe7700 1 -- 172.21.15.1:0/3869325590 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b040cf7f0 con 0x7f5b04099a40 2024-03-18T19:05:13.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.295+0000 7f5b08fe7700 1 --2- 172.21.15.1:0/3869325590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b04099a40 0x7f5b04099ea0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f5af40098f0 tx=0x7f5af4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5d752d91fa5b507 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:13.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.296+0000 7f5afbfff700 1 -- 172.21.15.1:0/3869325590 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af400b8a0 con 0x7f5b04099a40 2024-03-18T19:05:13.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.296+0000 7f5afbfff700 1 -- 172.21.15.1:0/3869325590 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af400ba00 con 0x7f5b04099a40 2024-03-18T19:05:13.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.296+0000 7f5afbfff700 1 -- 172.21.15.1:0/3869325590 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af400bd90 con 0x7f5b04099a40 2024-03-18T19:05:13.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.296+0000 7f5b0ba4c700 1 -- 172.21.15.1:0/3869325590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b04099a40 msgr2=0x7f5b04099ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:13.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.296+0000 7f5b0ba4c700 1 --2- 172.21.15.1:0/3869325590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b04099a40 0x7f5b04099ea0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f5af40098f0 tx=0x7f5af4009c00 comp rx=0 tx=0).stop 2024-03-18T19:05:13.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.296+0000 7f5b0ba4c700 1 -- 172.21.15.1:0/3869325590 shutdown_connections 2024-03-18T19:05:13.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.296+0000 7f5b0ba4c700 1 --2- 172.21.15.1:0/3869325590 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b0415d5b0 0x7f5b0415d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.296+0000 7f5b0ba4c700 1 --2- 172.21.15.1:0/3869325590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b04099a40 0x7f5b04099ea0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.296+0000 7f5b0ba4c700 1 -- 172.21.15.1:0/3869325590 >> 172.21.15.1:0/3869325590 conn(0x7f5b040b2bd0 msgr2=0x7f5b040b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:13.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.296+0000 7f5b0ba4c700 1 -- 172.21.15.1:0/3869325590 shutdown_connections 2024-03-18T19:05:13.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.296+0000 7f5b0ba4c700 1 -- 172.21.15.1:0/3869325590 wait complete. 2024-03-18T19:05:13.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.297+0000 7f5b0ba4c700 1 Processor -- start 2024-03-18T19:05:13.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.298+0000 7f5b0ba4c700 1 -- start start 2024-03-18T19:05:13.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.298+0000 7f5b0ba4c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b04099a40 0x7f5b040c5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:13.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.298+0000 7f5b0ba4c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b0409a3e0 0x7f5b040b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:13.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.298+0000 7f5b0ba4c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b0415d5b0 0x7f5b040b6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:13.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.298+0000 7f5b0ba4c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b040c81b0 con 0x7f5b0409a3e0 2024-03-18T19:05:13.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.298+0000 7f5b0ba4c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b040c8490 con 0x7f5b0415d5b0 2024-03-18T19:05:13.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.298+0000 7f5b0ba4c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b040c8770 con 0x7f5b04099a40 2024-03-18T19:05:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.298+0000 7f5b08fe7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b0409a3e0 0x7f5b040b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.298+0000 7f5b08fe7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b0409a3e0 0x7f5b040b5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42504/0 (socket says 172.21.15.1:42504) 2024-03-18T19:05:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.298+0000 7f5b08fe7700 1 -- 172.21.15.1:0/3812550570 learned_addr learned my addr 172.21.15.1:0/3812550570 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.299+0000 7f5b097e8700 1 --2- 172.21.15.1:0/3812550570 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b04099a40 0x7f5b040c5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.299+0000 7f5b09fe9700 1 --2- 172.21.15.1:0/3812550570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b0415d5b0 0x7f5b040b6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.299+0000 7f5b08fe7700 1 -- 172.21.15.1:0/3812550570 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b04099a40 msgr2=0x7f5b040c5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.299+0000 7f5b08fe7700 1 --2- 172.21.15.1:0/3812550570 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b04099a40 0x7f5b040c5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.299+0000 7f5b08fe7700 1 -- 172.21.15.1:0/3812550570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b0415d5b0 msgr2=0x7f5b040b6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.299+0000 7f5b08fe7700 1 --2- 172.21.15.1:0/3812550570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b0415d5b0 0x7f5b040b6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.299+0000 7f5b08fe7700 1 -- 172.21.15.1:0/3812550570 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af4009550 con 0x7f5b0409a3e0 2024-03-18T19:05:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.299+0000 7f5b097e8700 1 --2- 172.21.15.1:0/3812550570 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b04099a40 0x7f5b040c5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.299+0000 7f5b09fe9700 1 --2- 172.21.15.1:0/3812550570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b0415d5b0 0x7f5b040b6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.299+0000 7f5b08fe7700 1 --2- 172.21.15.1:0/3812550570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b0409a3e0 0x7f5b040b5ed0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f5af4009c40 tx=0x7f5af4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.299+0000 7f5af9ffb700 1 -- 172.21.15.1:0/3812550570 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af40045d0 con 0x7f5b0409a3e0 2024-03-18T19:05:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.299+0000 7f5af9ffb700 1 -- 172.21.15.1:0/3812550570 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af401e070 con 0x7f5b0409a3e0 2024-03-18T19:05:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.299+0000 7f5b0ba4c700 1 -- 172.21.15.1:0/3812550570 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b040c8a50 con 0x7f5b0409a3e0 2024-03-18T19:05:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.299+0000 7f5af9ffb700 1 -- 172.21.15.1:0/3812550570 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af40038e0 con 0x7f5b0409a3e0 2024-03-18T19:05:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.299+0000 7f5b0ba4c700 1 -- 172.21.15.1:0/3812550570 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b040c9010 con 0x7f5b0409a3e0 2024-03-18T19:05:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.300+0000 7f5b0ba4c700 1 -- 172.21.15.1:0/3812550570 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b040c9200 con 0x7f5b0409a3e0 2024-03-18T19:05:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.301+0000 7f5af9ffb700 1 -- 172.21.15.1:0/3812550570 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5af4003a40 con 0x7f5b0409a3e0 2024-03-18T19:05:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.301+0000 7f5af9ffb700 1 --2- 172.21.15.1:0/3812550570 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ae806adb0 0x7f5ae806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.301+0000 7f5af9ffb700 1 -- 172.21.15.1:0/3812550570 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f5af4020070 con 0x7f5b0409a3e0 2024-03-18T19:05:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.301+0000 7f5b097e8700 1 --2- 172.21.15.1:0/3812550570 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ae806adb0 0x7f5ae806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.301+0000 7f5b097e8700 1 --2- 172.21.15.1:0/3812550570 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ae806adb0 0x7f5ae806d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009a00 tx=0x7f5b00008040 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:13.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.306+0000 7f5af9ffb700 1 -- 172.21.15.1:0/3812550570 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5af4055a50 con 0x7f5b0409a3e0 2024-03-18T19:05:13.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.516+0000 7f5b0ba4c700 1 -- 172.21.15.1:0/3812550570 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f5b040676a0 con 0x7f5b0409a3e0 2024-03-18T19:05:13.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.520+0000 7f5af9ffb700 1 -- 172.21.15.1:0/3812550570 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v19) v1 ==== 115+0+0 (secure 0 0 0) 0x7f5af402ee80 con 0x7f5b0409a3e0 2024-03-18T19:05:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.522+0000 7f5b0ba4c700 1 -- 172.21.15.1:0/3812550570 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ae806adb0 msgr2=0x7f5ae806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.522+0000 7f5b0ba4c700 1 --2- 172.21.15.1:0/3812550570 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ae806adb0 0x7f5ae806d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009a00 tx=0x7f5b00008040 comp rx=0 tx=0).stop 2024-03-18T19:05:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.522+0000 7f5b0ba4c700 1 -- 172.21.15.1:0/3812550570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b0409a3e0 msgr2=0x7f5b040b5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.522+0000 7f5b0ba4c700 1 --2- 172.21.15.1:0/3812550570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b0409a3e0 0x7f5b040b5ed0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f5af4009c40 tx=0x7f5af4004750 comp rx=0 tx=0).stop 2024-03-18T19:05:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.522+0000 7f5b0ba4c700 1 -- 172.21.15.1:0/3812550570 shutdown_connections 2024-03-18T19:05:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.522+0000 7f5b0ba4c700 1 --2- 172.21.15.1:0/3812550570 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ae806adb0 0x7f5ae806d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.522+0000 7f5b0ba4c700 1 --2- 172.21.15.1:0/3812550570 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b04099a40 0x7f5b040c5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.522+0000 7f5b0ba4c700 1 --2- 172.21.15.1:0/3812550570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b0415d5b0 0x7f5b040b6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.522+0000 7f5b0ba4c700 1 --2- 172.21.15.1:0/3812550570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b0409a3e0 0x7f5b040b5ed0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.522+0000 7f5b0ba4c700 1 -- 172.21.15.1:0/3812550570 >> 172.21.15.1:0/3812550570 conn(0x7f5b040b2bd0 msgr2=0x7f5b040cbed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:13.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.523+0000 7f5b0ba4c700 1 -- 172.21.15.1:0/3812550570 shutdown_connections 2024-03-18T19:05:13.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.523+0000 7f5b0ba4c700 1 -- 172.21.15.1:0/3812550570 wait complete. 2024-03-18T19:05:13.531 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-18T19:05:13.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.661+0000 7f6bc3b7a700 1 Processor -- start 2024-03-18T19:05:13.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.662+0000 7f6bc3b7a700 1 -- start start 2024-03-18T19:05:13.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.663+0000 7f6bc3b7a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6bbc071d30 0x7f6bbc0add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:13.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.663+0000 7f6bc3b7a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6bbc0cb9e0 0x7f6bbc0cbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:13.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.663+0000 7f6bc3b7a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bbc0ce8f0 con 0x7f6bbc15a1d0 2024-03-18T19:05:13.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.663+0000 7f6bc3b7a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bbc0cdb00 con 0x7f6bbc0cb9e0 2024-03-18T19:05:13.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.663+0000 7f6bc3b7a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bbc15a5b0 con 0x7f6bbc071d30 2024-03-18T19:05:13.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.663+0000 7f6bc2117700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6bbc0cb9e0 0x7f6bbc0cbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:13.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.663+0000 7f6bc2117700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6bbc0cb9e0 0x7f6bbc0cbe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42514/0 (socket says 172.21.15.1:42514) 2024-03-18T19:05:13.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.663+0000 7f6bc2117700 1 -- 172.21.15.1:0/2159127973 learned_addr learned my addr 172.21.15.1:0/2159127973 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:13.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.663+0000 7f6bc1115700 1 --2- 172.21.15.1:0/2159127973 >> v2:172.21.15.165:3300/0 conn(0x7f6bbc071d30 0x7f6bbc0add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:13.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.663+0000 7f6bc2117700 1 -- 172.21.15.1:0/2159127973 >> v2:172.21.15.165:3300/0 conn(0x7f6bbc071d30 msgr2=0x7f6bbc0add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:13.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.663+0000 7f6bc2117700 1 --2- 172.21.15.1:0/2159127973 >> v2:172.21.15.165:3300/0 conn(0x7f6bbc071d30 0x7f6bbc0add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.663+0000 7f6bc2117700 1 -- 172.21.15.1:0/2159127973 >> v1:172.21.15.1:6789/0 conn(0x7f6bbc15a1d0 legacy=0x7f6bbc071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:13.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.663+0000 7f6bc2117700 1 -- 172.21.15.1:0/2159127973 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bbc0ae260 con 0x7f6bbc0cb9e0 2024-03-18T19:05:13.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.663+0000 7f6bc2117700 1 --2- 172.21.15.1:0/2159127973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bbc0cb9e0 0x7f6bbc0cbe30 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc0cebb0 tx=0x7f6bb400d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8765074dd61eb39 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:13.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.664+0000 7f6bc0914700 1 -- 172.21.15.1:0/2159127973 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb4010070 con 0x7f6bbc0cb9e0 2024-03-18T19:05:13.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.664+0000 7f6bc0914700 1 -- 172.21.15.1:0/2159127973 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb400dd90 con 0x7f6bbc0cb9e0 2024-03-18T19:05:13.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.664+0000 7f6bc0914700 1 -- 172.21.15.1:0/2159127973 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb400b310 con 0x7f6bbc0cb9e0 2024-03-18T19:05:13.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.664+0000 7f6bc3b7a700 1 -- 172.21.15.1:0/2159127973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bbc0cb9e0 msgr2=0x7f6bbc0cbe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:13.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.664+0000 7f6bc3b7a700 1 --2- 172.21.15.1:0/2159127973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bbc0cb9e0 0x7f6bbc0cbe30 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc0cebb0 tx=0x7f6bb400d430 comp rx=0 tx=0).stop 2024-03-18T19:05:13.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.664+0000 7f6bc3b7a700 1 -- 172.21.15.1:0/2159127973 shutdown_connections 2024-03-18T19:05:13.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.664+0000 7f6bc3b7a700 1 --2- 172.21.15.1:0/2159127973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bbc0cb9e0 0x7f6bbc0cbe30 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.664+0000 7f6bc3b7a700 1 --2- 172.21.15.1:0/2159127973 >> v2:172.21.15.165:3300/0 conn(0x7f6bbc071d30 0x7f6bbc0add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.664+0000 7f6bc3b7a700 1 -- 172.21.15.1:0/2159127973 >> 172.21.15.1:0/2159127973 conn(0x7f6bbc0cc9b0 msgr2=0x7f6bbc0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:13.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.664+0000 7f6bc3b7a700 1 -- 172.21.15.1:0/2159127973 shutdown_connections 2024-03-18T19:05:13.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.664+0000 7f6bc3b7a700 1 -- 172.21.15.1:0/2159127973 wait complete. 2024-03-18T19:05:13.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.666+0000 7f6bc3b7a700 1 Processor -- start 2024-03-18T19:05:13.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.666+0000 7f6bc3b7a700 1 -- start start 2024-03-18T19:05:13.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bc3b7a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bbc071d30 0x7f6bbc157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:13.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bc3b7a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bbc0cb9e0 0x7f6bbc0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:13.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bc3b7a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bbc15a1d0 0x7f6bbc0b10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:13.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bc3b7a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc0c85c0 con 0x7f6bbc0cb9e0 2024-03-18T19:05:13.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bc3b7a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc0c50c0 con 0x7f6bbc15a1d0 2024-03-18T19:05:13.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bc3b7a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc0c4820 con 0x7f6bbc071d30 2024-03-18T19:05:13.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bc1115700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bbc0cb9e0 0x7f6bbc0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:13.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bc1115700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bbc0cb9e0 0x7f6bbc0b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42516/0 (socket says 172.21.15.1:42516) 2024-03-18T19:05:13.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bc1115700 1 -- 172.21.15.1:0/831753615 learned_addr learned my addr 172.21.15.1:0/831753615 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:13.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bc2117700 1 --2- 172.21.15.1:0/831753615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bbc15a1d0 0x7f6bbc0b10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:13.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bc1115700 1 -- 172.21.15.1:0/831753615 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bbc071d30 msgr2=0x7f6bbc157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:13.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bc1916700 1 --2- 172.21.15.1:0/831753615 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bbc071d30 0x7f6bbc157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:13.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bc1115700 1 --2- 172.21.15.1:0/831753615 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bbc071d30 0x7f6bbc157640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bc1115700 1 -- 172.21.15.1:0/831753615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bbc15a1d0 msgr2=0x7f6bbc0b10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:13.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bc1115700 1 --2- 172.21.15.1:0/831753615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bbc15a1d0 0x7f6bbc0b10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bc1115700 1 -- 172.21.15.1:0/831753615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb400bd40 con 0x7f6bbc0cb9e0 2024-03-18T19:05:13.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bc2117700 1 --2- 172.21.15.1:0/831753615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bbc15a1d0 0x7f6bbc0b10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:13.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bc1115700 1 --2- 172.21.15.1:0/831753615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bbc0cb9e0 0x7f6bbc0b0b90 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000d800 tx=0x7f6bb000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:13.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bae7fc700 1 -- 172.21.15.1:0/831753615 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb00125f0 con 0x7f6bbc0cb9e0 2024-03-18T19:05:13.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.667+0000 7f6bae7fc700 1 -- 172.21.15.1:0/831753615 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb0012750 con 0x7f6bbc0cb9e0 2024-03-18T19:05:13.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.668+0000 7f6bc3b7a700 1 -- 172.21.15.1:0/831753615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bbc0c4b00 con 0x7f6bbc0cb9e0 2024-03-18T19:05:13.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.668+0000 7f6bae7fc700 1 -- 172.21.15.1:0/831753615 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0012b00 con 0x7f6bbc0cb9e0 2024-03-18T19:05:13.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.668+0000 7f6bc3b7a700 1 -- 172.21.15.1:0/831753615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bbc0c3ad0 con 0x7f6bbc0cb9e0 2024-03-18T19:05:13.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.668+0000 7f6bc3b7a700 1 -- 172.21.15.1:0/831753615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bbc0c83e0 con 0x7f6bbc0cb9e0 2024-03-18T19:05:13.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.669+0000 7f6bae7fc700 1 -- 172.21.15.1:0/831753615 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6bb0012c60 con 0x7f6bbc0cb9e0 2024-03-18T19:05:13.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.669+0000 7f6bae7fc700 1 --2- 172.21.15.1:0/831753615 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ba006adb0 0x7f6ba006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:13.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.669+0000 7f6bae7fc700 1 -- 172.21.15.1:0/831753615 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f6bb00898e0 con 0x7f6bbc0cb9e0 2024-03-18T19:05:13.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.669+0000 7f6bc1916700 1 --2- 172.21.15.1:0/831753615 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ba006adb0 0x7f6ba006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:13.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.670+0000 7f6bc1916700 1 --2- 172.21.15.1:0/831753615 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ba006adb0 0x7f6ba006d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8000fd0 tx=0x7f6bb8009040 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:13.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.674+0000 7f6bae7fc700 1 -- 172.21.15.1:0/831753615 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6bb0054910 con 0x7f6bbc0cb9e0 2024-03-18T19:05:13.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.876+0000 7f6bc3b7a700 1 -- 172.21.15.1:0/831753615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f6bbc0676a0 con 0x7f6bbc0cb9e0 2024-03-18T19:05:13.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.880+0000 7f6bae7fc700 1 -- 172.21.15.1:0/831753615 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v20) v1 ==== 120+0+0 (secure 0 0 0) 0x7f6bb0057f30 con 0x7f6bbc0cb9e0 2024-03-18T19:05:13.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.882+0000 7f6bc3b7a700 1 -- 172.21.15.1:0/831753615 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ba006adb0 msgr2=0x7f6ba006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.882+0000 7f6bc3b7a700 1 --2- 172.21.15.1:0/831753615 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ba006adb0 0x7f6ba006d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8000fd0 tx=0x7f6bb8009040 comp rx=0 tx=0).stop 2024-03-18T19:05:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.882+0000 7f6bc3b7a700 1 -- 172.21.15.1:0/831753615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bbc0cb9e0 msgr2=0x7f6bbc0b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.882+0000 7f6bc3b7a700 1 --2- 172.21.15.1:0/831753615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bbc0cb9e0 0x7f6bbc0b0b90 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000d800 tx=0x7f6bb000dbc0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.883+0000 7f6bc3b7a700 1 -- 172.21.15.1:0/831753615 shutdown_connections 2024-03-18T19:05:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.883+0000 7f6bc3b7a700 1 --2- 172.21.15.1:0/831753615 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ba006adb0 0x7f6ba006d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.883+0000 7f6bc3b7a700 1 --2- 172.21.15.1:0/831753615 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6bbc071d30 0x7f6bbc157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.883+0000 7f6bc3b7a700 1 --2- 172.21.15.1:0/831753615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6bbc15a1d0 0x7f6bbc0b10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.883+0000 7f6bc3b7a700 1 --2- 172.21.15.1:0/831753615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bbc0cb9e0 0x7f6bbc0b0b90 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.883+0000 7f6bc3b7a700 1 -- 172.21.15.1:0/831753615 >> 172.21.15.1:0/831753615 conn(0x7f6bbc0cc9b0 msgr2=0x7f6bbc06bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.883+0000 7f6bc3b7a700 1 -- 172.21.15.1:0/831753615 shutdown_connections 2024-03-18T19:05:13.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:13.883+0000 7f6bc3b7a700 1 -- 172.21.15.1:0/831753615 wait complete. 2024-03-18T19:05:13.891 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-18T19:05:14.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.022+0000 7fdfdb9d7700 1 Processor -- start 2024-03-18T19:05:14.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.022+0000 7fdfdb9d7700 1 -- start start 2024-03-18T19:05:14.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.023+0000 7fdfdb9d7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdfd415d5a0 0x7fdfd40ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:14.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.023+0000 7fdfdb9d7700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdfd40ce550 0x7fdfd406a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:14.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.023+0000 7fdfdb9d7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd40cf600 con 0x7fdfd40cfa90 2024-03-18T19:05:14.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.023+0000 7fdfdb9d7700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfd40ce9f0 con 0x7fdfd415d5a0 2024-03-18T19:05:14.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.023+0000 7fdfdb9d7700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfd40cecd0 con 0x7fdfd40ce550 2024-03-18T19:05:14.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.023+0000 7fdfd9773700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdfd415d5a0 0x7fdfd40ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:14.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.023+0000 7fdfd9773700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdfd415d5a0 0x7fdfd40ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42524/0 (socket says 172.21.15.1:42524) 2024-03-18T19:05:14.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.023+0000 7fdfd9f74700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdfd40cfa90 0x7fdfd4071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42440/0 (socket says 172.21.15.1:42440) 2024-03-18T19:05:14.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.023+0000 7fdfd9773700 1 -- 172.21.15.1:0/3277605225 learned_addr learned my addr 172.21.15.1:0/3277605225 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:14.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.023+0000 7fdfd8f72700 1 --2- 172.21.15.1:0/3277605225 >> v2:172.21.15.165:3300/0 conn(0x7fdfd40ce550 0x7fdfd406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:14.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.024+0000 7fdfd9773700 1 -- 172.21.15.1:0/3277605225 >> v2:172.21.15.165:3300/0 conn(0x7fdfd40ce550 msgr2=0x7fdfd406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:14.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.024+0000 7fdfd9773700 1 --2- 172.21.15.1:0/3277605225 >> v2:172.21.15.165:3300/0 conn(0x7fdfd40ce550 0x7fdfd406a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:14.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.024+0000 7fdfd9773700 1 -- 172.21.15.1:0/3277605225 >> v1:172.21.15.1:6789/0 conn(0x7fdfd40cfa90 legacy=0x7fdfd4071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:14.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.024+0000 7fdfd9773700 1 -- 172.21.15.1:0/3277605225 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd40cefb0 con 0x7fdfd415d5a0 2024-03-18T19:05:14.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.024+0000 7fdfcbfff700 1 -- 172.21.15.1:0/3277605225 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107814132 0 0) 0x7fdfd40cf600 con 0x7fdfd40cfa90 2024-03-18T19:05:14.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.024+0000 7fdfd9773700 1 --2- 172.21.15.1:0/3277605225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfd415d5a0 0x7fdfd40ce010 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fdfd40af730 tx=0x7fdfc4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67edd9d6c13b6e5a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:14.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.024+0000 7fdfcbfff700 1 -- 172.21.15.1:0/3277605225 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4005e30 con 0x7fdfd415d5a0 2024-03-18T19:05:14.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.024+0000 7fdfcbfff700 1 -- 172.21.15.1:0/3277605225 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc4004030 con 0x7fdfd415d5a0 2024-03-18T19:05:14.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.024+0000 7fdfcbfff700 1 -- 172.21.15.1:0/3277605225 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc40043e0 con 0x7fdfd415d5a0 2024-03-18T19:05:14.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.025+0000 7fdfdb9d7700 1 -- 172.21.15.1:0/3277605225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfd415d5a0 msgr2=0x7fdfd40ce010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:14.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.025+0000 7fdfdb9d7700 1 --2- 172.21.15.1:0/3277605225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfd415d5a0 0x7fdfd40ce010 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fdfd40af730 tx=0x7fdfc4009bd0 comp rx=0 tx=0).stop 2024-03-18T19:05:14.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.025+0000 7fdfdb9d7700 1 -- 172.21.15.1:0/3277605225 shutdown_connections 2024-03-18T19:05:14.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.025+0000 7fdfdb9d7700 1 --2- 172.21.15.1:0/3277605225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfd415d5a0 0x7fdfd40ce010 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:14.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.025+0000 7fdfdb9d7700 1 --2- 172.21.15.1:0/3277605225 >> v2:172.21.15.165:3300/0 conn(0x7fdfd40ce550 0x7fdfd406a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:14.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.025+0000 7fdfdb9d7700 1 -- 172.21.15.1:0/3277605225 >> 172.21.15.1:0/3277605225 conn(0x7fdfd40cc9b0 msgr2=0x7fdfd40ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:14.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.025+0000 7fdfdb9d7700 1 -- 172.21.15.1:0/3277605225 shutdown_connections 2024-03-18T19:05:14.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.025+0000 7fdfdb9d7700 1 -- 172.21.15.1:0/3277605225 wait complete. 2024-03-18T19:05:14.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.026+0000 7fdfdb9d7700 1 Processor -- start 2024-03-18T19:05:14.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.026+0000 7fdfdb9d7700 1 -- start start 2024-03-18T19:05:14.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfdb9d7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdfd40ce550 0x7fdfd4152480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:14.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfdb9d7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdfd40cfa90 0x7fdfd4153c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:14.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfdb9d7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfd415d5a0 0x7fdfd40b3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:14.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfdb9d7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd415e540 con 0x7fdfd415d5a0 2024-03-18T19:05:14.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfdb9d7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd40b5990 con 0x7fdfd40cfa90 2024-03-18T19:05:14.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfdb9d7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd41529c0 con 0x7fdfd40ce550 2024-03-18T19:05:14.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfd9f74700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfd415d5a0 0x7fdfd40b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:14.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfd9f74700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfd415d5a0 0x7fdfd40b3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42540/0 (socket says 172.21.15.1:42540) 2024-03-18T19:05:14.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfd9f74700 1 -- 172.21.15.1:0/4288493725 learned_addr learned my addr 172.21.15.1:0/4288493725 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:14.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfd8f72700 1 --2- 172.21.15.1:0/4288493725 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdfd40cfa90 0x7fdfd4153c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:14.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfd9773700 1 --2- 172.21.15.1:0/4288493725 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdfd40ce550 0x7fdfd4152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:14.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfd9f74700 1 -- 172.21.15.1:0/4288493725 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdfd40ce550 msgr2=0x7fdfd4152480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:14.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfd9f74700 1 --2- 172.21.15.1:0/4288493725 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdfd40ce550 0x7fdfd4152480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:14.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfd9f74700 1 -- 172.21.15.1:0/4288493725 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdfd40cfa90 msgr2=0x7fdfd4153c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:14.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfd9f74700 1 --2- 172.21.15.1:0/4288493725 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdfd40cfa90 0x7fdfd4153c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfd9f74700 1 -- 172.21.15.1:0/4288493725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc4009500 con 0x7fdfd415d5a0 2024-03-18T19:05:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfd9773700 1 --2- 172.21.15.1:0/4288493725 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdfd40ce550 0x7fdfd4152480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfd9f74700 1 --2- 172.21.15.1:0/4288493725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfd415d5a0 0x7fdfd40b3530 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00cc70 tx=0x7fdfcc00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfc9ffb700 1 -- 172.21.15.1:0/4288493725 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc00ad80 con 0x7fdfd415d5a0 2024-03-18T19:05:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfc9ffb700 1 -- 172.21.15.1:0/4288493725 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfcc00ce70 con 0x7fdfd415d5a0 2024-03-18T19:05:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfdb9d7700 1 -- 172.21.15.1:0/4288493725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd415dba0 con 0x7fdfd415d5a0 2024-03-18T19:05:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.027+0000 7fdfc9ffb700 1 -- 172.21.15.1:0/4288493725 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc0072f0 con 0x7fdfd415d5a0 2024-03-18T19:05:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.028+0000 7fdfdb9d7700 1 -- 172.21.15.1:0/4288493725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd40b4010 con 0x7fdfd415d5a0 2024-03-18T19:05:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.029+0000 7fdfdb9d7700 1 -- 172.21.15.1:0/4288493725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd415dd90 con 0x7fdfd415d5a0 2024-03-18T19:05:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.029+0000 7fdfc9ffb700 1 -- 172.21.15.1:0/4288493725 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fdfcc007450 con 0x7fdfd415d5a0 2024-03-18T19:05:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.029+0000 7fdfc9ffb700 1 --2- 172.21.15.1:0/4288493725 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdfb806adb0 0x7fdfb806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.029+0000 7fdfc9ffb700 1 -- 172.21.15.1:0/4288493725 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5865+0+0 (secure 0 0 0) 0x7fdfcc089330 con 0x7fdfd415d5a0 2024-03-18T19:05:14.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.029+0000 7fdfd9773700 1 --2- 172.21.15.1:0/4288493725 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdfb806adb0 0x7fdfb806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:14.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.030+0000 7fdfd9773700 1 --2- 172.21.15.1:0/4288493725 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdfb806adb0 0x7fdfb806d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4014040 tx=0x7fdfc4008070 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:14.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.034+0000 7fdfc9ffb700 1 -- 172.21.15.1:0/4288493725 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdfcc0543f0 con 0x7fdfd415d5a0 2024-03-18T19:05:14.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.239+0000 7fdfdb9d7700 1 -- 172.21.15.1:0/4288493725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fdfd415df90 con 0x7fdfd415d5a0 2024-03-18T19:05:14.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.242+0000 7fdfc9ffb700 1 -- 172.21.15.1:0/4288493725 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdfcc057a10 con 0x7fdfd415d5a0 2024-03-18T19:05:14.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.244+0000 7fdfdb9d7700 1 -- 172.21.15.1:0/4288493725 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdfb806adb0 msgr2=0x7fdfb806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:14.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.244+0000 7fdfdb9d7700 1 --2- 172.21.15.1:0/4288493725 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdfb806adb0 0x7fdfb806d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4014040 tx=0x7fdfc4008070 comp rx=0 tx=0).stop 2024-03-18T19:05:14.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.245+0000 7fdfdb9d7700 1 -- 172.21.15.1:0/4288493725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfd415d5a0 msgr2=0x7fdfd40b3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:14.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.245+0000 7fdfdb9d7700 1 --2- 172.21.15.1:0/4288493725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfd415d5a0 0x7fdfd40b3530 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00cc70 tx=0x7fdfcc00a340 comp rx=0 tx=0).stop 2024-03-18T19:05:14.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.245+0000 7fdfdb9d7700 1 -- 172.21.15.1:0/4288493725 shutdown_connections 2024-03-18T19:05:14.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.245+0000 7fdfdb9d7700 1 --2- 172.21.15.1:0/4288493725 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdfb806adb0 0x7fdfb806d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:14.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.245+0000 7fdfdb9d7700 1 --2- 172.21.15.1:0/4288493725 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdfd40ce550 0x7fdfd4152480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:14.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.245+0000 7fdfdb9d7700 1 --2- 172.21.15.1:0/4288493725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdfd415d5a0 0x7fdfd40b3530 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:14.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.245+0000 7fdfdb9d7700 1 --2- 172.21.15.1:0/4288493725 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdfd40cfa90 0x7fdfd4153c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:14.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.245+0000 7fdfdb9d7700 1 -- 172.21.15.1:0/4288493725 >> 172.21.15.1:0/4288493725 conn(0x7fdfd40cc9b0 msgr2=0x7fdfd40b0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:14.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.245+0000 7fdfdb9d7700 1 -- 172.21.15.1:0/4288493725 shutdown_connections 2024-03-18T19:05:14.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.245+0000 7fdfdb9d7700 1 -- 172.21.15.1:0/4288493725 wait complete. 2024-03-18T19:05:14.253 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2024-03-18T19:05:14.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.382+0000 7fa4c82cd700 1 Processor -- start 2024-03-18T19:05:14.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.382+0000 7fa4c82cd700 1 -- start start 2024-03-18T19:05:14.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.383+0000 7fa4c82cd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa4c006ae60 0x7fa4c0164f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:14.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.383+0000 7fa4c82cd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa4c01465a0 0x7fa4c0167480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:14.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.383+0000 7fa4c82cd700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c0072660 con 0x7fa4c0146190 2024-03-18T19:05:14.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.383+0000 7fa4c82cd700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c006f850 con 0x7fa4c006ae60 2024-03-18T19:05:14.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.383+0000 7fa4c82cd700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c006ebd0 con 0x7fa4c01465a0 2024-03-18T19:05:14.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.383+0000 7fa4c5868700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa4c006ae60 0x7fa4c0164f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:14.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.383+0000 7fa4c5868700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa4c006ae60 0x7fa4c0164f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33660/0 (socket says 172.21.15.1:33660) 2024-03-18T19:05:14.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.383+0000 7fa4c5868700 1 -- 172.21.15.1:0/3211114865 learned_addr learned my addr 172.21.15.1:0/3211114865 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:14.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.383+0000 7fa4c686a700 1 --2- 172.21.15.1:0/3211114865 >> v2:172.21.15.165:3300/0 conn(0x7fa4c01465a0 0x7fa4c0167480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:14.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.383+0000 7fa4c5868700 1 -- 172.21.15.1:0/3211114865 >> v2:172.21.15.165:3300/0 conn(0x7fa4c01465a0 msgr2=0x7fa4c0167480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:14.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.383+0000 7fa4c5868700 1 --2- 172.21.15.1:0/3211114865 >> v2:172.21.15.165:3300/0 conn(0x7fa4c01465a0 0x7fa4c0167480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:14.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.383+0000 7fa4c5868700 1 -- 172.21.15.1:0/3211114865 >> v1:172.21.15.1:6789/0 conn(0x7fa4c0146190 legacy=0x7fa4c006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:14.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.383+0000 7fa4c5067700 1 -- 172.21.15.1:0/3211114865 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2362488715 0 0) 0x7fa4c0072660 con 0x7fa4c0146190 2024-03-18T19:05:14.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.383+0000 7fa4c5868700 1 -- 172.21.15.1:0/3211114865 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c006eeb0 con 0x7fa4c006ae60 2024-03-18T19:05:14.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.383+0000 7fa4c686a700 1 --2- 172.21.15.1:0/3211114865 >> v2:172.21.15.165:3300/0 conn(0x7fa4c01465a0 0x7fa4c0167480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:14.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.383+0000 7fa4c5868700 1 --2- 172.21.15.1:0/3211114865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4c006ae60 0x7fa4c0164f30 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00715e0 tx=0x7fa4b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40eb28291987e00a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:14.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.384+0000 7fa4c5067700 1 -- 172.21.15.1:0/3211114865 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b000b870 con 0x7fa4c006ae60 2024-03-18T19:05:14.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.384+0000 7fa4c5067700 1 -- 172.21.15.1:0/3211114865 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b000b9d0 con 0x7fa4c006ae60 2024-03-18T19:05:14.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.384+0000 7fa4c5067700 1 -- 172.21.15.1:0/3211114865 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b000bd60 con 0x7fa4c006ae60 2024-03-18T19:05:14.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.384+0000 7fa4c82cd700 1 -- 172.21.15.1:0/3211114865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4c006ae60 msgr2=0x7fa4c0164f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:14.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.384+0000 7fa4c82cd700 1 --2- 172.21.15.1:0/3211114865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4c006ae60 0x7fa4c0164f30 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00715e0 tx=0x7fa4b0009bd0 comp rx=0 tx=0).stop 2024-03-18T19:05:14.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.384+0000 7fa4c82cd700 1 -- 172.21.15.1:0/3211114865 shutdown_connections 2024-03-18T19:05:14.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.384+0000 7fa4c82cd700 1 --2- 172.21.15.1:0/3211114865 >> v2:172.21.15.165:3300/0 conn(0x7fa4c01465a0 0x7fa4c0167480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:14.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.384+0000 7fa4c82cd700 1 --2- 172.21.15.1:0/3211114865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4c006ae60 0x7fa4c0164f30 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:14.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.384+0000 7fa4c82cd700 1 -- 172.21.15.1:0/3211114865 >> 172.21.15.1:0/3211114865 conn(0x7fa4c00b1d20 msgr2=0x7fa4c00b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:14.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.384+0000 7fa4c82cd700 1 -- 172.21.15.1:0/3211114865 shutdown_connections 2024-03-18T19:05:14.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.384+0000 7fa4c82cd700 1 -- 172.21.15.1:0/3211114865 wait complete. 2024-03-18T19:05:14.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.385+0000 7fa4c82cd700 1 Processor -- start 2024-03-18T19:05:14.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.385+0000 7fa4c82cd700 1 -- start start 2024-03-18T19:05:14.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.386+0000 7fa4c82cd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4c006ae60 0x7fa4c00cbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:14.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.386+0000 7fa4c82cd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4c01465a0 0x7fa4c01f2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:14.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.386+0000 7fa4c82cd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4c013f600 0x7fa4c0140f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:14.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.386+0000 7fa4c82cd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c0142400 con 0x7fa4c006ae60 2024-03-18T19:05:14.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.386+0000 7fa4c82cd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c01f3310 con 0x7fa4c013f600 2024-03-18T19:05:14.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.386+0000 7fa4c82cd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c0141780 con 0x7fa4c01465a0 2024-03-18T19:05:14.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.386+0000 7fa4c6069700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4c006ae60 0x7fa4c00cbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:14.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.386+0000 7fa4c6069700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4c006ae60 0x7fa4c00cbc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33664/0 (socket says 172.21.15.1:33664) 2024-03-18T19:05:14.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.386+0000 7fa4c6069700 1 -- 172.21.15.1:0/1422439442 learned_addr learned my addr 172.21.15.1:0/1422439442 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:14.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.386+0000 7fa4c5868700 1 --2- 172.21.15.1:0/1422439442 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4c01465a0 0x7fa4c01f2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:14.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.386+0000 7fa4c686a700 1 --2- 172.21.15.1:0/1422439442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4c013f600 0x7fa4c0140f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:14.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.387+0000 7fa4c6069700 1 -- 172.21.15.1:0/1422439442 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4c01465a0 msgr2=0x7fa4c01f2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:14.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.387+0000 7fa4c6069700 1 --2- 172.21.15.1:0/1422439442 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4c01465a0 0x7fa4c01f2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:14.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.387+0000 7fa4c6069700 1 -- 172.21.15.1:0/1422439442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4c013f600 msgr2=0x7fa4c0140f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:14.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.387+0000 7fa4c6069700 1 --2- 172.21.15.1:0/1422439442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4c013f600 0x7fa4c0140f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:14.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.387+0000 7fa4c6069700 1 -- 172.21.15.1:0/1422439442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b0009550 con 0x7fa4c006ae60 2024-03-18T19:05:14.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.387+0000 7fa4c6069700 1 --2- 172.21.15.1:0/1422439442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4c006ae60 0x7fa4c00cbc90 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc0011e0 tx=0x7fa4bc00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:14.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.387+0000 7fa4b6ffd700 1 -- 172.21.15.1:0/1422439442 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc0054e0 con 0x7fa4c006ae60 2024-03-18T19:05:14.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.387+0000 7fa4b6ffd700 1 -- 172.21.15.1:0/1422439442 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4bc003680 con 0x7fa4c006ae60 2024-03-18T19:05:14.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.387+0000 7fa4c82cd700 1 -- 172.21.15.1:0/1422439442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c0141a60 con 0x7fa4c006ae60 2024-03-18T19:05:14.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.387+0000 7fa4b6ffd700 1 -- 172.21.15.1:0/1422439442 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc003a30 con 0x7fa4c006ae60 2024-03-18T19:05:14.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.387+0000 7fa4c82cd700 1 -- 172.21.15.1:0/1422439442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c0142020 con 0x7fa4c006ae60 2024-03-18T19:05:14.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.388+0000 7fa4c82cd700 1 -- 172.21.15.1:0/1422439442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c00676a0 con 0x7fa4c006ae60 2024-03-18T19:05:14.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.388+0000 7fa4b6ffd700 1 -- 172.21.15.1:0/1422439442 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa4bc003b90 con 0x7fa4c006ae60 2024-03-18T19:05:14.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.389+0000 7fa4b6ffd700 1 --2- 172.21.15.1:0/1422439442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa4a406adb0 0x7fa4a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:14.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.389+0000 7fa4b6ffd700 1 -- 172.21.15.1:0/1422439442 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5865+0+0 (secure 0 0 0) 0x7fa4bc088aa0 con 0x7fa4c006ae60 2024-03-18T19:05:14.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.389+0000 7fa4c5868700 1 --2- 172.21.15.1:0/1422439442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa4a406adb0 0x7fa4a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:14.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.389+0000 7fa4c5868700 1 --2- 172.21.15.1:0/1422439442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa4a406adb0 0x7fa4a406d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0014040 tx=0x7fa4b0008030 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:14.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.393+0000 7fa4b6ffd700 1 -- 172.21.15.1:0/1422439442 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa4bc053ad0 con 0x7fa4c006ae60 2024-03-18T19:05:14.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:14.677+0000 7fa4c82cd700 1 -- 172.21.15.1:0/1422439442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7fa4c0146c20 con 0x7fa4c006ae60 2024-03-18T19:05:15.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.135+0000 7fa4b6ffd700 1 -- 172.21.15.1:0/1422439442 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v564) v1 ==== 154+0+0 (secure 0 0 0) 0x7fa4bc017070 con 0x7fa4c006ae60 2024-03-18T19:05:15.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.137+0000 7fa4c82cd700 1 -- 172.21.15.1:0/1422439442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa4a406adb0 msgr2=0x7fa4a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:15.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.137+0000 7fa4c82cd700 1 --2- 172.21.15.1:0/1422439442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa4a406adb0 0x7fa4a406d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0014040 tx=0x7fa4b0008030 comp rx=0 tx=0).stop 2024-03-18T19:05:15.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.137+0000 7fa4c82cd700 1 -- 172.21.15.1:0/1422439442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4c006ae60 msgr2=0x7fa4c00cbc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:15.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.137+0000 7fa4c82cd700 1 --2- 172.21.15.1:0/1422439442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4c006ae60 0x7fa4c00cbc90 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc0011e0 tx=0x7fa4bc00cae0 comp rx=0 tx=0).stop 2024-03-18T19:05:15.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.137+0000 7fa4c82cd700 1 -- 172.21.15.1:0/1422439442 shutdown_connections 2024-03-18T19:05:15.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.137+0000 7fa4c82cd700 1 --2- 172.21.15.1:0/1422439442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa4a406adb0 0x7fa4a406d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:15.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.137+0000 7fa4c82cd700 1 --2- 172.21.15.1:0/1422439442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4c013f600 0x7fa4c0140f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:15.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.137+0000 7fa4c82cd700 1 --2- 172.21.15.1:0/1422439442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4c006ae60 0x7fa4c00cbc90 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:15.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.137+0000 7fa4c82cd700 1 --2- 172.21.15.1:0/1422439442 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4c01465a0 0x7fa4c01f2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:15.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.137+0000 7fa4c82cd700 1 -- 172.21.15.1:0/1422439442 >> 172.21.15.1:0/1422439442 conn(0x7fa4c00b1d20 msgr2=0x7fa4c00ac8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:15.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.137+0000 7fa4c82cd700 1 -- 172.21.15.1:0/1422439442 shutdown_connections 2024-03-18T19:05:15.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.137+0000 7fa4c82cd700 1 -- 172.21.15.1:0/1422439442 wait complete. 2024-03-18T19:05:15.146 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2024-03-18T19:05:15.146 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2024-03-18T19:05:15.147 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-18T19:05:15.147 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2024-03-18T19:05:15.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.279+0000 7fb90cfbe700 1 Processor -- start 2024-03-18T19:05:15.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.279+0000 7fb90cfbe700 1 -- start start 2024-03-18T19:05:15.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.280+0000 7fb90cfbe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb9080af610 0x7fb9080cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:15.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.280+0000 7fb90cfbe700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9080cf090 con 0x7fb9080ccf40 2024-03-18T19:05:15.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.280+0000 7fb90cfbe700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9080cf2d0 con 0x7fb908144b70 2024-03-18T19:05:15.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.280+0000 7fb90cfbe700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb908144f50 con 0x7fb9080af610 2024-03-18T19:05:15.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.280+0000 7fb905d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb9080af610 0x7fb9080cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:15.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.280+0000 7fb905d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb9080af610 0x7fb9080cca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33676/0 (socket says 172.21.15.1:33676) 2024-03-18T19:05:15.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.280+0000 7fb905d9b700 1 -- 172.21.15.1:0/3690662038 learned_addr learned my addr 172.21.15.1:0/3690662038 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:15.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.280+0000 7fb906d9d700 1 --1- 172.21.15.1:0/3690662038 >> v1:172.21.15.1:6789/0 conn(0x7fb9080ccf40 0x7fb908071a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44272/0 (socket says 172.21.15.1:44272) 2024-03-18T19:05:15.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.280+0000 7fb905d9b700 1 -- 172.21.15.1:0/3690662038 >> v1:172.21.15.165:6789/0 conn(0x7fb908144b70 legacy=0x7fb9080ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:15.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.280+0000 7fb905d9b700 1 -- 172.21.15.1:0/3690662038 >> v1:172.21.15.1:6789/0 conn(0x7fb9080ccf40 legacy=0x7fb908071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:15.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.280+0000 7fb905d9b700 1 -- 172.21.15.1:0/3690662038 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9080cf5b0 con 0x7fb9080af610 2024-03-18T19:05:15.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.280+0000 7fb905d9b700 1 --2- 172.21.15.1:0/3690662038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9080af610 0x7fb9080cca00 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fb908145090 tx=0x7fb8fc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38558b590f098fcf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:15.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.280+0000 7fb90559a700 1 -- 172.21.15.1:0/3690662038 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc004230 con 0x7fb9080af610 2024-03-18T19:05:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.280+0000 7fb90559a700 1 -- 172.21.15.1:0/3690662038 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8fc004390 con 0x7fb9080af610 2024-03-18T19:05:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.280+0000 7fb90559a700 1 -- 172.21.15.1:0/3690662038 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc0036a0 con 0x7fb9080af610 2024-03-18T19:05:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.281+0000 7fb90cfbe700 1 -- 172.21.15.1:0/3690662038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9080af610 msgr2=0x7fb9080cca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.281+0000 7fb90cfbe700 1 --2- 172.21.15.1:0/3690662038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9080af610 0x7fb9080cca00 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fb908145090 tx=0x7fb8fc009bd0 comp rx=0 tx=0).stop 2024-03-18T19:05:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.281+0000 7fb90cfbe700 1 -- 172.21.15.1:0/3690662038 shutdown_connections 2024-03-18T19:05:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.281+0000 7fb90cfbe700 1 --2- 172.21.15.1:0/3690662038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9080af610 0x7fb9080cca00 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.281+0000 7fb90cfbe700 1 -- 172.21.15.1:0/3690662038 >> 172.21.15.1:0/3690662038 conn(0x7fb9080b1d20 msgr2=0x7fb9080b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.281+0000 7fb90cfbe700 1 -- 172.21.15.1:0/3690662038 shutdown_connections 2024-03-18T19:05:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.281+0000 7fb90cfbe700 1 -- 172.21.15.1:0/3690662038 wait complete. 2024-03-18T19:05:15.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.282+0000 7fb90cfbe700 1 Processor -- start 2024-03-18T19:05:15.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.282+0000 7fb90cfbe700 1 -- start start 2024-03-18T19:05:15.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.283+0000 7fb90cfbe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9080af610 0x7fb908140700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:15.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.283+0000 7fb90cfbe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9080ccf40 0x7fb9081f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:15.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.283+0000 7fb90cfbe700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb908144b70 0x7fb9081f2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:15.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.283+0000 7fb90cfbe700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90813c0b0 con 0x7fb9080af610 2024-03-18T19:05:15.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.283+0000 7fb90cfbe700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90813b780 con 0x7fb908144b70 2024-03-18T19:05:15.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.283+0000 7fb90cfbe700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb90813ba60 con 0x7fb9080ccf40 2024-03-18T19:05:15.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.283+0000 7fb90659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9080af610 0x7fb908140700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:15.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.283+0000 7fb90659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9080af610 0x7fb908140700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33688/0 (socket says 172.21.15.1:33688) 2024-03-18T19:05:15.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.283+0000 7fb90659c700 1 -- 172.21.15.1:0/2820820011 learned_addr learned my addr 172.21.15.1:0/2820820011 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:15.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.283+0000 7fb906d9d700 1 --2- 172.21.15.1:0/2820820011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb908144b70 0x7fb9081f2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:15.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.283+0000 7fb905d9b700 1 --2- 172.21.15.1:0/2820820011 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9080ccf40 0x7fb9081f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:15.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.283+0000 7fb90659c700 1 -- 172.21.15.1:0/2820820011 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9080ccf40 msgr2=0x7fb9081f21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:15.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.284+0000 7fb90659c700 1 --2- 172.21.15.1:0/2820820011 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9080ccf40 0x7fb9081f21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:15.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.284+0000 7fb90659c700 1 -- 172.21.15.1:0/2820820011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb908144b70 msgr2=0x7fb9081f2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:15.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.284+0000 7fb90659c700 1 --2- 172.21.15.1:0/2820820011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb908144b70 0x7fb9081f2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.284+0000 7fb90659c700 1 -- 172.21.15.1:0/2820820011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8fc009550 con 0x7fb9080af610 2024-03-18T19:05:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.284+0000 7fb90659c700 1 --2- 172.21.15.1:0/2820820011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9080af610 0x7fb908140700 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fb8f800c450 tx=0x7fb8f800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.284+0000 7fb8f6ffd700 1 -- 172.21.15.1:0/2820820011 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f80052d0 con 0x7fb9080af610 2024-03-18T19:05:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.284+0000 7fb8f6ffd700 1 -- 172.21.15.1:0/2820820011 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f8005430 con 0x7fb9080af610 2024-03-18T19:05:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.284+0000 7fb90cfbe700 1 -- 172.21.15.1:0/2820820011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90813bd40 con 0x7fb9080af610 2024-03-18T19:05:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.284+0000 7fb8f6ffd700 1 -- 172.21.15.1:0/2820820011 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f80038c0 con 0x7fb9080af610 2024-03-18T19:05:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.284+0000 7fb90cfbe700 1 -- 172.21.15.1:0/2820820011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb908140c40 con 0x7fb9080af610 2024-03-18T19:05:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.285+0000 7fb90cfbe700 1 -- 172.21.15.1:0/2820820011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9080cda90 con 0x7fb9080af610 2024-03-18T19:05:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.285+0000 7fb8f6ffd700 1 -- 172.21.15.1:0/2820820011 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb8f8003a20 con 0x7fb9080af610 2024-03-18T19:05:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.286+0000 7fb8f6ffd700 1 --2- 172.21.15.1:0/2820820011 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb8e806adb0 0x7fb8e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.286+0000 7fb8f6ffd700 1 -- 172.21.15.1:0/2820820011 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5865+0+0 (secure 0 0 0) 0x7fb8f8088900 con 0x7fb9080af610 2024-03-18T19:05:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.286+0000 7fb905d9b700 1 --2- 172.21.15.1:0/2820820011 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb8e806adb0 0x7fb8e806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:15.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.286+0000 7fb905d9b700 1 --2- 172.21.15.1:0/2820820011 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb8e806adb0 0x7fb8e806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb908144520 tx=0x7fb8fc008030 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:15.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.290+0000 7fb8f6ffd700 1 -- 172.21.15.1:0/2820820011 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb8f80538a0 con 0x7fb9080af610 2024-03-18T19:05:15.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:15.574+0000 7fb90cfbe700 1 -- 172.21.15.1:0/2820820011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7fb9080b6080 con 0x7fb9080af610 2024-03-18T19:05:16.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.136+0000 7fb8f6ffd700 1 -- 172.21.15.1:0/2820820011 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v565) v1 ==== 159+0+0 (secure 0 0 0) 0x7fb8f8056ec0 con 0x7fb9080af610 2024-03-18T19:05:16.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.139+0000 7fb90cfbe700 1 -- 172.21.15.1:0/2820820011 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb8e806adb0 msgr2=0x7fb8e806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:16.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.139+0000 7fb90cfbe700 1 --2- 172.21.15.1:0/2820820011 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb8e806adb0 0x7fb8e806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb908144520 tx=0x7fb8fc008030 comp rx=0 tx=0).stop 2024-03-18T19:05:16.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.139+0000 7fb90cfbe700 1 -- 172.21.15.1:0/2820820011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9080af610 msgr2=0x7fb908140700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:16.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.139+0000 7fb90cfbe700 1 --2- 172.21.15.1:0/2820820011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9080af610 0x7fb908140700 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fb8f800c450 tx=0x7fb8f800c810 comp rx=0 tx=0).stop 2024-03-18T19:05:16.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.139+0000 7fb90cfbe700 1 -- 172.21.15.1:0/2820820011 shutdown_connections 2024-03-18T19:05:16.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.139+0000 7fb90cfbe700 1 --2- 172.21.15.1:0/2820820011 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb8e806adb0 0x7fb8e806d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:16.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.139+0000 7fb90cfbe700 1 --2- 172.21.15.1:0/2820820011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb908144b70 0x7fb9081f2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:16.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.139+0000 7fb90cfbe700 1 --2- 172.21.15.1:0/2820820011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9080af610 0x7fb908140700 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:16.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.139+0000 7fb90cfbe700 1 --2- 172.21.15.1:0/2820820011 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9080ccf40 0x7fb9081f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:16.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.139+0000 7fb90cfbe700 1 -- 172.21.15.1:0/2820820011 >> 172.21.15.1:0/2820820011 conn(0x7fb9080b1d20 msgr2=0x7fb908153df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:16.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.139+0000 7fb90cfbe700 1 -- 172.21.15.1:0/2820820011 shutdown_connections 2024-03-18T19:05:16.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.139+0000 7fb90cfbe700 1 -- 172.21.15.1:0/2820820011 wait complete. 2024-03-18T19:05:16.150 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-18T19:05:16.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.290+0000 7f69732c3700 1 Processor -- start 2024-03-18T19:05:16.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.290+0000 7f69732c3700 1 -- start start 2024-03-18T19:05:16.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.291+0000 7f69732c3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f696c15d5b0 0x7f696c15d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:16.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.291+0000 7f69732c3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f696c099a40 0x7f696c099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:16.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.291+0000 7f69732c3700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c09a830 con 0x7f696c09a3e0 2024-03-18T19:05:16.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.291+0000 7f69732c3700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696c0cbbb0 con 0x7f696c099a40 2024-03-18T19:05:16.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.291+0000 7f69732c3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696c0b3b00 con 0x7f696c15d5b0 2024-03-18T19:05:16.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.291+0000 7f697085e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f696c099a40 0x7f696c099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:16.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.291+0000 7f697085e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f696c099a40 0x7f696c099ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33692/0 (socket says 172.21.15.1:33692) 2024-03-18T19:05:16.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.291+0000 7f697085e700 1 -- 172.21.15.1:0/1766793952 learned_addr learned my addr 172.21.15.1:0/1766793952 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:16.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.291+0000 7f697085e700 1 -- 172.21.15.1:0/1766793952 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f696c15d5b0 msgr2=0x7f696c15d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:16.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.291+0000 7f697105f700 1 --2- 172.21.15.1:0/1766793952 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f696c15d5b0 0x7f696c15d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:16.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.291+0000 7f697085e700 1 --2- 172.21.15.1:0/1766793952 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f696c15d5b0 0x7f696c15d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:16.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.291+0000 7f697085e700 1 -- 172.21.15.1:0/1766793952 >> v1:172.21.15.165:6789/0 conn(0x7f696c09a3e0 legacy=0x7f696c06ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:16.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.291+0000 7f697085e700 1 -- 172.21.15.1:0/1766793952 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c0cf7f0 con 0x7f696c099a40 2024-03-18T19:05:16.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.291+0000 7f697085e700 1 --2- 172.21.15.1:0/1766793952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f696c099a40 0x7f696c099ea0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f695c0098f0 tx=0x7f695c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d93f05de0c44739 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:16.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.292+0000 7f6963fff700 1 -- 172.21.15.1:0/1766793952 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c004230 con 0x7f696c099a40 2024-03-18T19:05:16.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.292+0000 7f6963fff700 1 -- 172.21.15.1:0/1766793952 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695c004390 con 0x7f696c099a40 2024-03-18T19:05:16.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.292+0000 7f6963fff700 1 -- 172.21.15.1:0/1766793952 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c0036a0 con 0x7f696c099a40 2024-03-18T19:05:16.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.292+0000 7f69732c3700 1 -- 172.21.15.1:0/1766793952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f696c099a40 msgr2=0x7f696c099ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:16.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.292+0000 7f69732c3700 1 --2- 172.21.15.1:0/1766793952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f696c099a40 0x7f696c099ea0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f695c0098f0 tx=0x7f695c009c00 comp rx=0 tx=0).stop 2024-03-18T19:05:16.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.292+0000 7f69732c3700 1 -- 172.21.15.1:0/1766793952 shutdown_connections 2024-03-18T19:05:16.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.292+0000 7f69732c3700 1 --2- 172.21.15.1:0/1766793952 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f696c15d5b0 0x7f696c15d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:16.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.292+0000 7f69732c3700 1 --2- 172.21.15.1:0/1766793952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f696c099a40 0x7f696c099ea0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:16.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.292+0000 7f69732c3700 1 -- 172.21.15.1:0/1766793952 >> 172.21.15.1:0/1766793952 conn(0x7f696c0b2bd0 msgr2=0x7f696c0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:16.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.292+0000 7f69732c3700 1 -- 172.21.15.1:0/1766793952 shutdown_connections 2024-03-18T19:05:16.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.292+0000 7f69732c3700 1 -- 172.21.15.1:0/1766793952 wait complete. 2024-03-18T19:05:16.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.293+0000 7f69732c3700 1 Processor -- start 2024-03-18T19:05:16.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.293+0000 7f69732c3700 1 -- start start 2024-03-18T19:05:16.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.294+0000 7f69732c3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f696c099a40 0x7f696c14c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:16.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.294+0000 7f69732c3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f696c09a3e0 0x7f696c06e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:16.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.294+0000 7f69732c3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f696c15d5b0 0x7f696c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:16.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.294+0000 7f69732c3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c1f72f0 con 0x7f696c15d5b0 2024-03-18T19:05:16.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.294+0000 7f69732c3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c14cb70 con 0x7f696c09a3e0 2024-03-18T19:05:16.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.294+0000 7f69732c3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696c1f7710 con 0x7f696c099a40 2024-03-18T19:05:16.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.294+0000 7f6971860700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f696c15d5b0 0x7f696c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:16.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.294+0000 7f6971860700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f696c15d5b0 0x7f696c06ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33698/0 (socket says 172.21.15.1:33698) 2024-03-18T19:05:16.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.294+0000 7f6971860700 1 -- 172.21.15.1:0/2188535457 learned_addr learned my addr 172.21.15.1:0/2188535457 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:16.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.294+0000 7f697105f700 1 --2- 172.21.15.1:0/2188535457 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f696c099a40 0x7f696c14c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:16.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.294+0000 7f697085e700 1 --2- 172.21.15.1:0/2188535457 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f696c09a3e0 0x7f696c06e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:16.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.295+0000 7f6971860700 1 -- 172.21.15.1:0/2188535457 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f696c099a40 msgr2=0x7f696c14c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:16.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.295+0000 7f6971860700 1 --2- 172.21.15.1:0/2188535457 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f696c099a40 0x7f696c14c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:16.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.295+0000 7f6971860700 1 -- 172.21.15.1:0/2188535457 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f696c09a3e0 msgr2=0x7f696c06e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:16.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.295+0000 7f6971860700 1 --2- 172.21.15.1:0/2188535457 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f696c09a3e0 0x7f696c06e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:16.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.295+0000 7f6971860700 1 -- 172.21.15.1:0/2188535457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c009550 con 0x7f696c15d5b0 2024-03-18T19:05:16.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.295+0000 7f6971860700 1 --2- 172.21.15.1:0/2188535457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f696c15d5b0 0x7f696c06ed20 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f696400b970 tx=0x7f696400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:16.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.295+0000 7f6961ffb700 1 -- 172.21.15.1:0/2188535457 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696400d860 con 0x7f696c15d5b0 2024-03-18T19:05:16.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.295+0000 7f6961ffb700 1 -- 172.21.15.1:0/2188535457 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696400d9c0 con 0x7f696c15d5b0 2024-03-18T19:05:16.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.295+0000 7f69732c3700 1 -- 172.21.15.1:0/2188535457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696c0b60a0 con 0x7f696c15d5b0 2024-03-18T19:05:16.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.295+0000 7f6961ffb700 1 -- 172.21.15.1:0/2188535457 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696400dd70 con 0x7f696c15d5b0 2024-03-18T19:05:16.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.295+0000 7f69732c3700 1 -- 172.21.15.1:0/2188535457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696c1f6c30 con 0x7f696c15d5b0 2024-03-18T19:05:16.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.296+0000 7f69732c3700 1 -- 172.21.15.1:0/2188535457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696c1f7430 con 0x7f696c15d5b0 2024-03-18T19:05:16.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.296+0000 7f6961ffb700 1 -- 172.21.15.1:0/2188535457 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f696400e430 con 0x7f696c15d5b0 2024-03-18T19:05:16.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.297+0000 7f6961ffb700 1 --2- 172.21.15.1:0/2188535457 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f695006adb0 0x7f695006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:16.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.297+0000 7f6961ffb700 1 -- 172.21.15.1:0/2188535457 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f69640896b0 con 0x7f696c15d5b0 2024-03-18T19:05:16.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.297+0000 7f697105f700 1 --2- 172.21.15.1:0/2188535457 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f695006adb0 0x7f695006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:16.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.297+0000 7f697105f700 1 --2- 172.21.15.1:0/2188535457 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f695006adb0 0x7f695006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f696800ab60 tx=0x7f6968008040 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:16.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.301+0000 7f6961ffb700 1 -- 172.21.15.1:0/2188535457 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6964054660 con 0x7f696c15d5b0 2024-03-18T19:05:16.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.504+0000 7f69732c3700 1 -- 172.21.15.1:0/2188535457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f696c0676a0 con 0x7f696c15d5b0 2024-03-18T19:05:16.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.508+0000 7f6961ffb700 1 -- 172.21.15.1:0/2188535457 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f696400ded0 con 0x7f696c15d5b0 2024-03-18T19:05:16.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.510+0000 7f69732c3700 1 -- 172.21.15.1:0/2188535457 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f695006adb0 msgr2=0x7f695006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:16.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.510+0000 7f69732c3700 1 --2- 172.21.15.1:0/2188535457 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f695006adb0 0x7f695006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f696800ab60 tx=0x7f6968008040 comp rx=0 tx=0).stop 2024-03-18T19:05:16.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.510+0000 7f69732c3700 1 -- 172.21.15.1:0/2188535457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f696c15d5b0 msgr2=0x7f696c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:16.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.510+0000 7f69732c3700 1 --2- 172.21.15.1:0/2188535457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f696c15d5b0 0x7f696c06ed20 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f696400b970 tx=0x7f696400bc80 comp rx=0 tx=0).stop 2024-03-18T19:05:16.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.510+0000 7f69732c3700 1 -- 172.21.15.1:0/2188535457 shutdown_connections 2024-03-18T19:05:16.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.510+0000 7f69732c3700 1 --2- 172.21.15.1:0/2188535457 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f695006adb0 0x7f695006d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:16.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.510+0000 7f69732c3700 1 --2- 172.21.15.1:0/2188535457 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f696c099a40 0x7f696c14c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:16.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.510+0000 7f69732c3700 1 --2- 172.21.15.1:0/2188535457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f696c15d5b0 0x7f696c06ed20 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:16.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.510+0000 7f69732c3700 1 --2- 172.21.15.1:0/2188535457 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f696c09a3e0 0x7f696c06e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:16.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.510+0000 7f69732c3700 1 -- 172.21.15.1:0/2188535457 >> 172.21.15.1:0/2188535457 conn(0x7f696c0b2bd0 msgr2=0x7f696c0c6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:16.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.510+0000 7f69732c3700 1 -- 172.21.15.1:0/2188535457 shutdown_connections 2024-03-18T19:05:16.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.511+0000 7f69732c3700 1 -- 172.21.15.1:0/2188535457 wait complete. 2024-03-18T19:05:16.519 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T19:05:16.520 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:05:16.561 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:05:16.612 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T19:05:16.612 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:05:16.612 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:05:16.631 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:05:16.631 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:05:16.631 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:05:16.651 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:05:16.651 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:05:16.651 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:05:16.688 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:05:16.689 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:05:16.689 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:05:16.710 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:05:16.710 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-18T19:05:16.710 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:05:16.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.857+0000 7fcd52f6b700 1 Processor -- start 2024-03-18T19:05:16.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.857+0000 7fcd52f6b700 1 -- start start 2024-03-18T19:05:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.858+0000 7fcd52f6b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd4c0af680 0x7fcd4c0cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.858+0000 7fcd52f6b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcd4c0ccf40 0x7fcd4c071ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.858+0000 7fcd52f6b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd4c071650 con 0x7fcd4c144c30 2024-03-18T19:05:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.858+0000 7fcd52f6b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd4c0cd3f0 con 0x7fcd4c0ccf40 2024-03-18T19:05:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.858+0000 7fcd52f6b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4c0cf380 con 0x7fcd4c0af680 2024-03-18T19:05:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.858+0000 7fcd51508700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcd4c0ccf40 0x7fcd4c071ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.858+0000 7fcd51508700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcd4c0ccf40 0x7fcd4c071ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33706/0 (socket says 172.21.15.1:33706) 2024-03-18T19:05:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.858+0000 7fcd51508700 1 -- 172.21.15.1:0/3503252675 learned_addr learned my addr 172.21.15.1:0/3503252675 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.859+0000 7fcd51508700 1 -- 172.21.15.1:0/3503252675 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd4c0af680 msgr2=0x7fcd4c0cca00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T19:05:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.859+0000 7fcd51508700 1 --2- 172.21.15.1:0/3503252675 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd4c0af680 0x7fcd4c0cca00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.859+0000 7fcd51508700 1 -- 172.21.15.1:0/3503252675 >> v1:172.21.15.1:6789/0 conn(0x7fcd4c144c30 legacy=0x7fcd4c0ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:16.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.859+0000 7fcd51508700 1 -- 172.21.15.1:0/3503252675 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd4c0cf660 con 0x7fcd4c0ccf40 2024-03-18T19:05:16.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.859+0000 7fcd51508700 1 --2- 172.21.15.1:0/3503252675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd4c0ccf40 0x7fcd4c071ae0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009e50 tx=0x7fcd4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14aa8df4bff1beac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:16.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.859+0000 7fcd4b7fe700 1 -- 172.21.15.1:0/3503252675 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4400e070 con 0x7fcd4c0ccf40 2024-03-18T19:05:16.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.859+0000 7fcd4b7fe700 1 -- 172.21.15.1:0/3503252675 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd4400b9c0 con 0x7fcd4c0ccf40 2024-03-18T19:05:16.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.859+0000 7fcd4b7fe700 1 -- 172.21.15.1:0/3503252675 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4400bd70 con 0x7fcd4c0ccf40 2024-03-18T19:05:16.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.859+0000 7fcd52f6b700 1 -- 172.21.15.1:0/3503252675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd4c0ccf40 msgr2=0x7fcd4c071ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:16.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.859+0000 7fcd52f6b700 1 --2- 172.21.15.1:0/3503252675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd4c0ccf40 0x7fcd4c071ae0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009e50 tx=0x7fcd4400b330 comp rx=0 tx=0).stop 2024-03-18T19:05:16.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.859+0000 7fcd52f6b700 1 -- 172.21.15.1:0/3503252675 shutdown_connections 2024-03-18T19:05:16.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.859+0000 7fcd52f6b700 1 --2- 172.21.15.1:0/3503252675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd4c0ccf40 0x7fcd4c071ae0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:16.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.859+0000 7fcd52f6b700 1 --2- 172.21.15.1:0/3503252675 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd4c0af680 0x7fcd4c0cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:16.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.859+0000 7fcd52f6b700 1 -- 172.21.15.1:0/3503252675 >> 172.21.15.1:0/3503252675 conn(0x7fcd4c0b1d20 msgr2=0x7fcd4c0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:16.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.859+0000 7fcd52f6b700 1 -- 172.21.15.1:0/3503252675 shutdown_connections 2024-03-18T19:05:16.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.860+0000 7fcd52f6b700 1 -- 172.21.15.1:0/3503252675 wait complete. 2024-03-18T19:05:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.861+0000 7fcd52f6b700 1 Processor -- start 2024-03-18T19:05:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.861+0000 7fcd52f6b700 1 -- start start 2024-03-18T19:05:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd52f6b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd4c0af680 0x7fcd4c142040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd52f6b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd4c0ccf40 0x7fcd4c0c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd52f6b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd4c144c30 0x7fcd4c15ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd52f6b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4c0c9a70 con 0x7fcd4c144c30 2024-03-18T19:05:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd52f6b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4c15b040 con 0x7fcd4c0af680 2024-03-18T19:05:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd52f6b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4c142740 con 0x7fcd4c0ccf40 2024-03-18T19:05:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd51508700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd4c144c30 0x7fcd4c15ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd51508700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd4c144c30 0x7fcd4c15ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33708/0 (socket says 172.21.15.1:33708) 2024-03-18T19:05:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd51508700 1 -- 172.21.15.1:0/3771200954 learned_addr learned my addr 172.21.15.1:0/3771200954 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd50d07700 1 --2- 172.21.15.1:0/3771200954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd4c0af680 0x7fcd4c142040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd4bfff700 1 --2- 172.21.15.1:0/3771200954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd4c0ccf40 0x7fcd4c0c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd51508700 1 -- 172.21.15.1:0/3771200954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd4c0ccf40 msgr2=0x7fcd4c0c9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd51508700 1 --2- 172.21.15.1:0/3771200954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd4c0ccf40 0x7fcd4c0c9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd51508700 1 -- 172.21.15.1:0/3771200954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd4c0af680 msgr2=0x7fcd4c142040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd51508700 1 --2- 172.21.15.1:0/3771200954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd4c0af680 0x7fcd4c142040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd51508700 1 -- 172.21.15.1:0/3771200954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd44009b30 con 0x7fcd4c144c30 2024-03-18T19:05:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd50d07700 1 --2- 172.21.15.1:0/3771200954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd4c0af680 0x7fcd4c142040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:16.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd51508700 1 --2- 172.21.15.1:0/3771200954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd4c144c30 0x7fcd4c15ab00 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fcd4400b730 tx=0x7fcd44004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:16.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd497fa700 1 -- 172.21.15.1:0/3771200954 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd44003ea0 con 0x7fcd4c144c30 2024-03-18T19:05:16.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd497fa700 1 -- 172.21.15.1:0/3771200954 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd440092e0 con 0x7fcd4c144c30 2024-03-18T19:05:16.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd52f6b700 1 -- 172.21.15.1:0/3771200954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd4c142a20 con 0x7fcd4c144c30 2024-03-18T19:05:16.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.862+0000 7fcd497fa700 1 -- 172.21.15.1:0/3771200954 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4401c640 con 0x7fcd4c144c30 2024-03-18T19:05:16.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.863+0000 7fcd52f6b700 1 -- 172.21.15.1:0/3771200954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd4c142fe0 con 0x7fcd4c144c30 2024-03-18T19:05:16.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.863+0000 7fcd52f6b700 1 -- 172.21.15.1:0/3771200954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd4c0b6310 con 0x7fcd4c144c30 2024-03-18T19:05:16.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.864+0000 7fcd497fa700 1 -- 172.21.15.1:0/3771200954 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcd4401c7a0 con 0x7fcd4c144c30 2024-03-18T19:05:16.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.864+0000 7fcd497fa700 1 --2- 172.21.15.1:0/3771200954 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcd3006adb0 0x7fcd3006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:16.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.864+0000 7fcd497fa700 1 -- 172.21.15.1:0/3771200954 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5865+0+0 (secure 0 0 0) 0x7fcd4408b0f0 con 0x7fcd4c144c30 2024-03-18T19:05:16.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.864+0000 7fcd50d07700 1 --2- 172.21.15.1:0/3771200954 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcd3006adb0 0x7fcd3006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:16.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.865+0000 7fcd50d07700 1 --2- 172.21.15.1:0/3771200954 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcd3006adb0 0x7fcd3006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcd40000f60 tx=0x7fcd40009040 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:16.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:16.869+0000 7fcd497fa700 1 -- 172.21.15.1:0/3771200954 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcd440560a0 con 0x7fcd4c144c30 2024-03-18T19:05:17.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.113+0000 7fcd52f6b700 1 -- 172.21.15.1:0/3771200954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd4c0676a0 con 0x7fcd4c144c30 2024-03-18T19:05:17.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.113+0000 7fcd497fa700 1 -- 172.21.15.1:0/3771200954 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+3526 (secure 0 0 0) 0x7fcd440596c0 con 0x7fcd4c144c30 2024-03-18T19:05:17.109 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:17.110 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5420,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3327835778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":3327835778},{"type":"v1","addr":"172.21.15.165:6835","nonce":3327835778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6016,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/450295183","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":450295183},{"type":"v1","addr":"172.21.15.1:6837","nonce":450295183}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6022,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1053303526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1053303526},{"type":"v1","addr":"172.21.15.165:6834","nonce":1053303526}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6708,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/1317127138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1317127138},{"type":"v1","addr":"172.21.15.1:6836","nonce":1317127138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[]} 2024-03-18T19:05:17.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.115+0000 7fcd52f6b700 1 -- 172.21.15.1:0/3771200954 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcd3006adb0 msgr2=0x7fcd3006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:17.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.115+0000 7fcd52f6b700 1 --2- 172.21.15.1:0/3771200954 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcd3006adb0 0x7fcd3006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcd40000f60 tx=0x7fcd40009040 comp rx=0 tx=0).stop 2024-03-18T19:05:17.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.115+0000 7fcd52f6b700 1 -- 172.21.15.1:0/3771200954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd4c144c30 msgr2=0x7fcd4c15ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:17.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.115+0000 7fcd52f6b700 1 --2- 172.21.15.1:0/3771200954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd4c144c30 0x7fcd4c15ab00 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fcd4400b730 tx=0x7fcd44004ad0 comp rx=0 tx=0).stop 2024-03-18T19:05:17.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.115+0000 7fcd52f6b700 1 -- 172.21.15.1:0/3771200954 shutdown_connections 2024-03-18T19:05:17.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.115+0000 7fcd52f6b700 1 --2- 172.21.15.1:0/3771200954 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcd3006adb0 0x7fcd3006d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:17.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.115+0000 7fcd52f6b700 1 --2- 172.21.15.1:0/3771200954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd4c144c30 0x7fcd4c15ab00 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:17.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.115+0000 7fcd52f6b700 1 --2- 172.21.15.1:0/3771200954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd4c0af680 0x7fcd4c142040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:17.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.115+0000 7fcd52f6b700 1 --2- 172.21.15.1:0/3771200954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd4c0ccf40 0x7fcd4c0c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:17.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.115+0000 7fcd52f6b700 1 -- 172.21.15.1:0/3771200954 >> 172.21.15.1:0/3771200954 conn(0x7fcd4c0b1d20 msgr2=0x7fcd4c0b0480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:17.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.115+0000 7fcd52f6b700 1 -- 172.21.15.1:0/3771200954 shutdown_connections 2024-03-18T19:05:17.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.115+0000 7fcd52f6b700 1 -- 172.21.15.1:0/3771200954 wait complete. 2024-03-18T19:05:17.113 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 27 2024-03-18T19:05:17.124 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T19:05:17.124 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-18T19:05:17.124 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T19:05:17.124 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:05:17.124 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T19:05:17.124 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-18T19:05:17.125 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-18T19:05:17.125 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:05:17.125 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T19:05:17.125 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-18T19:05:17.125 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-18T19:05:17.125 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:05:17.125 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T19:05:17.125 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-18T19:05:17.126 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-18T19:05:17.126 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:05:17.133 INFO:tasks.ceph.mds.a:Stopped 2024-03-18T19:05:17.133 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-18T19:05:17.134 INFO:tasks.ceph.mds.c:Stopped 2024-03-18T19:05:17.134 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-18T19:05:17.136 INFO:tasks.ceph.mds.b:Stopped 2024-03-18T19:05:17.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-18T19:05:17.137 INFO:tasks.ceph.mds.d:Stopped 2024-03-18T19:05:17.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-18T19:05:17.175 INFO:tasks.ceph.mds.a:Started 2024-03-18T19:05:17.175 DEBUG:teuthology.parallel:result is None 2024-03-18T19:05:17.176 INFO:tasks.ceph.mds.c:Started 2024-03-18T19:05:17.176 DEBUG:teuthology.parallel:result is None 2024-03-18T19:05:17.179 INFO:tasks.ceph.mds.b:Started 2024-03-18T19:05:17.179 DEBUG:teuthology.parallel:result is None 2024-03-18T19:05:17.180 INFO:tasks.ceph.mds.d:Started 2024-03-18T19:05:17.180 DEBUG:teuthology.parallel:result is None 2024-03-18T19:05:17.180 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-18T19:05:17.257 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-18T19:05:17.258 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T19:05:17.268 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-03-18T19:05:17.268 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-03-18T19:05:17.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.322+0000 7fc5f8385700 1 Processor -- start 2024-03-18T19:05:17.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.322+0000 7fc5f8385700 1 -- start start 2024-03-18T19:05:17.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.323+0000 7fc5f8385700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5f015a240 0x7fc5f00ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:17.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.323+0000 7fc5f8385700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5f006d560 0x7fc5f006fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:17.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.323+0000 7fc5f8385700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5f0070000 con 0x7fc5f00715b0 2024-03-18T19:05:17.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.323+0000 7fc5f8385700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5f00adff0 con 0x7fc5f015a240 2024-03-18T19:05:17.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.323+0000 7fc5f8385700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f00cac80 con 0x7fc5f006d560 2024-03-18T19:05:17.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.323+0000 7fc5f6121700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5f015a240 0x7fc5f00ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:17.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.323+0000 7fc5f6121700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5f015a240 0x7fc5f00ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33754/0 (socket says 172.21.15.1:33754) 2024-03-18T19:05:17.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.323+0000 7fc5f6121700 1 -- 172.21.15.1:0/3987369097 learned_addr learned my addr 172.21.15.1:0/3987369097 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:17.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.324+0000 7fc5f5920700 1 --2- 172.21.15.1:0/3987369097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5f006d560 0x7fc5f006fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:17.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.324+0000 7fc5f6121700 1 -- 172.21.15.1:0/3987369097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5f006d560 msgr2=0x7fc5f006fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:17.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.324+0000 7fc5f6121700 1 --2- 172.21.15.1:0/3987369097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5f006d560 0x7fc5f006fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:17.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.324+0000 7fc5f6121700 1 -- 172.21.15.1:0/3987369097 >> v1:172.21.15.165:6789/0 conn(0x7fc5f00715b0 legacy=0x7fc5f015de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:17.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.324+0000 7fc5f6121700 1 -- 172.21.15.1:0/3987369097 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f00caf60 con 0x7fc5f015a240 2024-03-18T19:05:17.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.324+0000 7fc5f6121700 1 --2- 172.21.15.1:0/3987369097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f015a240 0x7fc5f00ca580 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec0098f0 tx=0x7fc5ec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e10f2355cd8cb0d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:17.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.324+0000 7fc5f511f700 1 -- 172.21.15.1:0/3987369097 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec011830 con 0x7fc5f015a240 2024-03-18T19:05:17.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.324+0000 7fc5f511f700 1 -- 172.21.15.1:0/3987369097 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ec011990 con 0x7fc5f015a240 2024-03-18T19:05:17.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.324+0000 7fc5f511f700 1 -- 172.21.15.1:0/3987369097 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec011d20 con 0x7fc5f015a240 2024-03-18T19:05:17.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.325+0000 7fc5f8385700 1 -- 172.21.15.1:0/3987369097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f015a240 msgr2=0x7fc5f00ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:17.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.325+0000 7fc5f8385700 1 --2- 172.21.15.1:0/3987369097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f015a240 0x7fc5f00ca580 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec0098f0 tx=0x7fc5ec009c00 comp rx=0 tx=0).stop 2024-03-18T19:05:17.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.325+0000 7fc5f8385700 1 -- 172.21.15.1:0/3987369097 shutdown_connections 2024-03-18T19:05:17.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.325+0000 7fc5f8385700 1 --2- 172.21.15.1:0/3987369097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f015a240 0x7fc5f00ca580 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:17.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.325+0000 7fc5f8385700 1 --2- 172.21.15.1:0/3987369097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5f006d560 0x7fc5f006fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:17.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.325+0000 7fc5f8385700 1 -- 172.21.15.1:0/3987369097 >> 172.21.15.1:0/3987369097 conn(0x7fc5f00cff30 msgr2=0x7fc5f00d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:17.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.325+0000 7fc5f8385700 1 -- 172.21.15.1:0/3987369097 shutdown_connections 2024-03-18T19:05:17.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.325+0000 7fc5f8385700 1 -- 172.21.15.1:0/3987369097 wait complete. 2024-03-18T19:05:17.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.326+0000 7fc5f8385700 1 Processor -- start 2024-03-18T19:05:17.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.326+0000 7fc5f8385700 1 -- start start 2024-03-18T19:05:17.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.327+0000 7fc5f8385700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5f006d560 0x7fc5f0149520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:17.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.327+0000 7fc5f8385700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5f00715b0 0x7fc5f00cbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:17.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.327+0000 7fc5f8385700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f015a240 0x7fc5f00cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:17.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.327+0000 7fc5f8385700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f014a8a0 con 0x7fc5f015a240 2024-03-18T19:05:17.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.327+0000 7fc5f8385700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f00c71c0 con 0x7fc5f006d560 2024-03-18T19:05:17.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.327+0000 7fc5f8385700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f014a000 con 0x7fc5f00715b0 2024-03-18T19:05:17.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.327+0000 7fc5f6922700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f015a240 0x7fc5f00cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:17.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.327+0000 7fc5f6922700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f015a240 0x7fc5f00cc0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33760/0 (socket says 172.21.15.1:33760) 2024-03-18T19:05:17.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.327+0000 7fc5f6922700 1 -- 172.21.15.1:0/2013362886 learned_addr learned my addr 172.21.15.1:0/2013362886 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:17.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.327+0000 7fc5f5920700 1 --2- 172.21.15.1:0/2013362886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5f00715b0 0x7fc5f00cbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:17.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.327+0000 7fc5f6121700 1 --2- 172.21.15.1:0/2013362886 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5f006d560 0x7fc5f0149520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:17.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.327+0000 7fc5f6922700 1 -- 172.21.15.1:0/2013362886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5f00715b0 msgr2=0x7fc5f00cbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:17.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.327+0000 7fc5f6922700 1 --2- 172.21.15.1:0/2013362886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5f00715b0 0x7fc5f00cbbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:17.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.327+0000 7fc5f6922700 1 -- 172.21.15.1:0/2013362886 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5f006d560 msgr2=0x7fc5f0149520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:17.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.327+0000 7fc5f6922700 1 --2- 172.21.15.1:0/2013362886 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5f006d560 0x7fc5f0149520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:17.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.327+0000 7fc5f6922700 1 -- 172.21.15.1:0/2013362886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec009550 con 0x7fc5f015a240 2024-03-18T19:05:17.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.327+0000 7fc5f6922700 1 --2- 172.21.15.1:0/2013362886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f015a240 0x7fc5f00cc0f0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fc5e800cc50 tx=0x7fc5e800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:17.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.328+0000 7fc5e6ffd700 1 -- 172.21.15.1:0/2013362886 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e8003c00 con 0x7fc5f015a240 2024-03-18T19:05:17.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.328+0000 7fc5e6ffd700 1 -- 172.21.15.1:0/2013362886 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e8003d60 con 0x7fc5f015a240 2024-03-18T19:05:17.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.328+0000 7fc5e6ffd700 1 -- 172.21.15.1:0/2013362886 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e800aab0 con 0x7fc5f015a240 2024-03-18T19:05:17.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.328+0000 7fc5f8385700 1 -- 172.21.15.1:0/2013362886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f014a2e0 con 0x7fc5f015a240 2024-03-18T19:05:17.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.328+0000 7fc5f8385700 1 -- 172.21.15.1:0/2013362886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f0152040 con 0x7fc5f015a240 2024-03-18T19:05:17.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.329+0000 7fc5f8385700 1 -- 172.21.15.1:0/2013362886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f00c7330 con 0x7fc5f015a240 2024-03-18T19:05:17.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.329+0000 7fc5e6ffd700 1 -- 172.21.15.1:0/2013362886 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc5e800ac10 con 0x7fc5f015a240 2024-03-18T19:05:17.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.330+0000 7fc5e6ffd700 1 --2- 172.21.15.1:0/2013362886 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc5d406adb0 0x7fc5d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:17.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.330+0000 7fc5e6ffd700 1 -- 172.21.15.1:0/2013362886 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5865+0+0 (secure 0 0 0) 0x7fc5e8089480 con 0x7fc5f015a240 2024-03-18T19:05:17.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.330+0000 7fc5f6121700 1 --2- 172.21.15.1:0/2013362886 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc5d406adb0 0x7fc5d406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:17.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.330+0000 7fc5f6121700 1 --2- 172.21.15.1:0/2013362886 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc5d406adb0 0x7fc5d406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec014040 tx=0x7fc5ec008030 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:17.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.334+0000 7fc5e6ffd700 1 -- 172.21.15.1:0/2013362886 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc5e8054540 con 0x7fc5f015a240 2024-03-18T19:05:17.339 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-03-18T19:05:17.339 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-18T19:05:17.339 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-03-18T19:05:17.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:17.552+0000 7fc5f8385700 1 -- 172.21.15.1:0/2013362886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc5f00676a0 con 0x7fc5f015a240 2024-03-18T19:05:18.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.511+0000 7fc5e6ffd700 1 -- 172.21.15.1:0/2013362886 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v85) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc5e8057b60 con 0x7fc5f015a240 2024-03-18T19:05:18.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.514+0000 7fc5f8385700 1 -- 172.21.15.1:0/2013362886 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc5d406adb0 msgr2=0x7fc5d406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:18.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.514+0000 7fc5f8385700 1 --2- 172.21.15.1:0/2013362886 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc5d406adb0 0x7fc5d406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec014040 tx=0x7fc5ec008030 comp rx=0 tx=0).stop 2024-03-18T19:05:18.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.514+0000 7fc5f8385700 1 -- 172.21.15.1:0/2013362886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f015a240 msgr2=0x7fc5f00cc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:18.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.514+0000 7fc5f8385700 1 --2- 172.21.15.1:0/2013362886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f015a240 0x7fc5f00cc0f0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fc5e800cc50 tx=0x7fc5e800cf60 comp rx=0 tx=0).stop 2024-03-18T19:05:18.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.514+0000 7fc5f8385700 1 -- 172.21.15.1:0/2013362886 shutdown_connections 2024-03-18T19:05:18.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.514+0000 7fc5f8385700 1 --2- 172.21.15.1:0/2013362886 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc5d406adb0 0x7fc5d406d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:18.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.514+0000 7fc5f8385700 1 --2- 172.21.15.1:0/2013362886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5f015a240 0x7fc5f00cc0f0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:18.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.514+0000 7fc5f8385700 1 --2- 172.21.15.1:0/2013362886 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5f006d560 0x7fc5f0149520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:18.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.514+0000 7fc5f8385700 1 --2- 172.21.15.1:0/2013362886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5f00715b0 0x7fc5f00cbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:18.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.514+0000 7fc5f8385700 1 -- 172.21.15.1:0/2013362886 >> 172.21.15.1:0/2013362886 conn(0x7fc5f00cff30 msgr2=0x7fc5f00afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:18.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.515+0000 7fc5f8385700 1 -- 172.21.15.1:0/2013362886 shutdown_connections 2024-03-18T19:05:18.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.515+0000 7fc5f8385700 1 -- 172.21.15.1:0/2013362886 wait complete. 2024-03-18T19:05:18.512 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-18T19:05:18.526 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-18T19:05:18.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.662+0000 7f60608c3700 1 Processor -- start 2024-03-18T19:05:18.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.663+0000 7f60608c3700 1 -- start start 2024-03-18T19:05:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f60608c3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f605c15b4d0 0x7f605c09a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f60608c3700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f605c0cf750 0x7f605c099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f60608c3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f605c099f80 0x7f605c0cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f60608c3700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f605c09a410 con 0x7f605c15b4d0 2024-03-18T19:05:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f60608c3700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f605c06f070 con 0x7f605c0cf750 2024-03-18T19:05:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f60608c3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605c0d0260 con 0x7f605c099f80 2024-03-18T19:05:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f605a59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f605c15b4d0 0x7f605c09a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:18.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f605a59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f605c15b4d0 0x7f605c09a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33776/0 (socket says 172.21.15.1:33776) 2024-03-18T19:05:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f605a59c700 1 -- 172.21.15.1:0/1223905273 learned_addr learned my addr 172.21.15.1:0/1223905273 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f605ad9d700 1 --2- 172.21.15.1:0/1223905273 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f605c099f80 0x7f605c0cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f6059d9b700 1 --2- 172.21.15.1:0/1223905273 >> v2:172.21.15.165:3300/0 conn(0x7f605c0cf750 0x7f605c099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f605a59c700 1 -- 172.21.15.1:0/1223905273 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f605c099f80 msgr2=0x7f605c0cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f605a59c700 1 --2- 172.21.15.1:0/1223905273 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f605c099f80 0x7f605c0cbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f605a59c700 1 -- 172.21.15.1:0/1223905273 >> v2:172.21.15.165:3300/0 conn(0x7f605c0cf750 msgr2=0x7f605c099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f605a59c700 1 --2- 172.21.15.1:0/1223905273 >> v2:172.21.15.165:3300/0 conn(0x7f605c0cf750 0x7f605c099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f605a59c700 1 -- 172.21.15.1:0/1223905273 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605c0cff80 con 0x7f605c15b4d0 2024-03-18T19:05:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f605a59c700 1 --2- 172.21.15.1:0/1223905273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c15b4d0 0x7f605c09a650 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f605c071d40 tx=0x7f604c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c2f212ea3c5c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f605959a700 1 -- 172.21.15.1:0/1223905273 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604c00b870 con 0x7f605c15b4d0 2024-03-18T19:05:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f605959a700 1 -- 172.21.15.1:0/1223905273 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604c00b9d0 con 0x7f605c15b4d0 2024-03-18T19:05:18.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.664+0000 7f605959a700 1 -- 172.21.15.1:0/1223905273 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604c00bd60 con 0x7f605c15b4d0 2024-03-18T19:05:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.665+0000 7f60608c3700 1 -- 172.21.15.1:0/1223905273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c15b4d0 msgr2=0x7f605c09a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.665+0000 7f60608c3700 1 --2- 172.21.15.1:0/1223905273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c15b4d0 0x7f605c09a650 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f605c071d40 tx=0x7f604c009bd0 comp rx=0 tx=0).stop 2024-03-18T19:05:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.665+0000 7f60608c3700 1 -- 172.21.15.1:0/1223905273 shutdown_connections 2024-03-18T19:05:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.665+0000 7f60608c3700 1 --2- 172.21.15.1:0/1223905273 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f605c099f80 0x7f605c0cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.665+0000 7f60608c3700 1 --2- 172.21.15.1:0/1223905273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c15b4d0 0x7f605c09a650 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.665+0000 7f60608c3700 1 --2- 172.21.15.1:0/1223905273 >> v2:172.21.15.165:3300/0 conn(0x7f605c0cf750 0x7f605c099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.665+0000 7f60608c3700 1 -- 172.21.15.1:0/1223905273 >> 172.21.15.1:0/1223905273 conn(0x7f605c0b2cb0 msgr2=0x7f605c0b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.665+0000 7f60608c3700 1 -- 172.21.15.1:0/1223905273 shutdown_connections 2024-03-18T19:05:18.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.665+0000 7f60608c3700 1 -- 172.21.15.1:0/1223905273 wait complete. 2024-03-18T19:05:18.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.666+0000 7f60608c3700 1 Processor -- start 2024-03-18T19:05:18.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.666+0000 7f60608c3700 1 -- start start 2024-03-18T19:05:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.667+0000 7f60608c3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f605c099f80 0x7f605c0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.667+0000 7f60608c3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c0cf750 0x7f605c156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.667+0000 7f60608c3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f605c15b4d0 0x7f605c0af290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.667+0000 7f60608c3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c0b5660 con 0x7f605c0cf750 2024-03-18T19:05:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.667+0000 7f60608c3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c0b45c0 con 0x7f605c15b4d0 2024-03-18T19:05:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.667+0000 7f60608c3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605c0b48a0 con 0x7f605c099f80 2024-03-18T19:05:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.667+0000 7f6059d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c0cf750 0x7f605c156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.667+0000 7f6059d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c0cf750 0x7f605c156620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33792/0 (socket says 172.21.15.1:33792) 2024-03-18T19:05:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.667+0000 7f6059d9b700 1 -- 172.21.15.1:0/2879597765 learned_addr learned my addr 172.21.15.1:0/2879597765 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.667+0000 7f605a59c700 1 --2- 172.21.15.1:0/2879597765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f605c099f80 0x7f605c0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.668+0000 7f605ad9d700 1 --2- 172.21.15.1:0/2879597765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f605c15b4d0 0x7f605c0af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.668+0000 7f6059d9b700 1 -- 172.21.15.1:0/2879597765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f605c099f80 msgr2=0x7f605c0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:18.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.668+0000 7f6059d9b700 1 --2- 172.21.15.1:0/2879597765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f605c099f80 0x7f605c0c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.668+0000 7f6059d9b700 1 -- 172.21.15.1:0/2879597765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f605c15b4d0 msgr2=0x7f605c0af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.668+0000 7f6059d9b700 1 --2- 172.21.15.1:0/2879597765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f605c15b4d0 0x7f605c0af290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.668+0000 7f6059d9b700 1 -- 172.21.15.1:0/2879597765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c009550 con 0x7f605c0cf750 2024-03-18T19:05:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.668+0000 7f605ad9d700 1 --2- 172.21.15.1:0/2879597765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f605c15b4d0 0x7f605c0af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.668+0000 7f6059d9b700 1 --2- 172.21.15.1:0/2879597765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c0cf750 0x7f605c156620 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f605000e810 tx=0x7f605000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.668+0000 7f604affd700 1 -- 172.21.15.1:0/2879597765 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6050013610 con 0x7f605c0cf750 2024-03-18T19:05:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.668+0000 7f604affd700 1 -- 172.21.15.1:0/2879597765 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6050013770 con 0x7f605c0cf750 2024-03-18T19:05:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.668+0000 7f60608c3700 1 -- 172.21.15.1:0/2879597765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f605c0b4b80 con 0x7f605c0cf750 2024-03-18T19:05:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.668+0000 7f604affd700 1 -- 172.21.15.1:0/2879597765 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6050013b20 con 0x7f605c0cf750 2024-03-18T19:05:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.668+0000 7f60608c3700 1 -- 172.21.15.1:0/2879597765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f605c0b5140 con 0x7f605c0cf750 2024-03-18T19:05:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.669+0000 7f60608c3700 1 -- 172.21.15.1:0/2879597765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605c068240 con 0x7f605c0cf750 2024-03-18T19:05:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.669+0000 7f604affd700 1 -- 172.21.15.1:0/2879597765 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6050013c80 con 0x7f605c0cf750 2024-03-18T19:05:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.670+0000 7f604affd700 1 --2- 172.21.15.1:0/2879597765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f603c06adb0 0x7f603c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:18.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.670+0000 7f604affd700 1 -- 172.21.15.1:0/2879597765 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f6050089130 con 0x7f605c0cf750 2024-03-18T19:05:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.670+0000 7f605a59c700 1 --2- 172.21.15.1:0/2879597765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f603c06adb0 0x7f603c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:18.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.670+0000 7f605a59c700 1 --2- 172.21.15.1:0/2879597765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f603c06adb0 0x7f603c06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f604c009840 tx=0x7f604c008030 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:18.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.674+0000 7f604affd700 1 -- 172.21.15.1:0/2879597765 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6050054310 con 0x7f605c0cf750 2024-03-18T19:05:18.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.957+0000 7f60608c3700 1 -- 172.21.15.1:0/2879597765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f605c0b5360 con 0x7f605c0cf750 2024-03-18T19:05:18.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.958+0000 7f604affd700 1 -- 172.21.15.1:0/2879597765 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f6050057930 con 0x7f605c0cf750 2024-03-18T19:05:18.954 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:18.954 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:05:18.954 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-18T19:05:18.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:18.954 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-18T19:05:18.954 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlR93/CxAAbpe+5n3wHRur9xi8QzdCnQ==", 2024-03-18T19:05:18.954 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:05:18.954 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:05:18.955 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:05:18.955 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:05:18.955 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:05:18.955 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:18.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:18.955 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-18T19:05:18.955 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJ3OmGBAA37u227nxrMsh/JHt71HBxQ==", 2024-03-18T19:05:18.955 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:05:18.955 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:05:18.955 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:05:18.955 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:05:18.955 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:05:18.956 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:18.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:18.956 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-18T19:05:18.956 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJZFoExAAmI16oxwNKbf8dKetPfyk+g==", 2024-03-18T19:05:18.956 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:05:18.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:05:18.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:05:18.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:05:18.956 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:05:18.956 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:18.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:18.956 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-18T19:05:18.956 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlQrcvIBAA1LMv+EgC7S6wspX6QF49Pg==", 2024-03-18T19:05:18.957 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:05:18.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:05:18.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:05:18.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:05:18.957 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:05:18.957 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:18.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:18.957 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-18T19:05:18.957 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCUjPhlYOVHChAARFQrksZmMIdTlntd8TgvrQ==", 2024-03-18T19:05:18.957 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:05:18.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:05:18.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:05:18.957 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:05:18.957 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:18.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:18.958 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-18T19:05:18.958 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCXjPhl2zSDCBAA6Al1gkWVB79ZI9djGpVChA==", 2024-03-18T19:05:18.958 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:05:18.958 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:05:18.958 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:05:18.958 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:05:18.958 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:18.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:18.959 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-18T19:05:18.959 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCajPhlSoG/BhAAR93USDe6DCQW+hREOsnTlw==", 2024-03-18T19:05:18.959 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:05:18.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:05:18.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:05:18.959 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:05:18.959 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:18.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:18.960 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-18T19:05:18.960 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCdjPhlhJg5BRAAWuQhcnxyjR5UOiM13yKdAQ==", 2024-03-18T19:05:18.960 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:05:18.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:05:18.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:05:18.960 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:05:18.960 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:18.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:18.961 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-18T19:05:18.961 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCjjPhlCFnpCRAAKIyOQyVdjXZUTnAS6Nw/ng==", 2024-03-18T19:05:18.961 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:05:18.961 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:05:18.961 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:05:18.961 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:05:18.961 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:18.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:18.961 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-18T19:05:18.961 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCmjPhlAeFkCBAAJcEFpRk/03lQJxj+l2hEPA==", 2024-03-18T19:05:18.962 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:05:18.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:05:18.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:05:18.962 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:05:18.962 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:18.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:18.962 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-18T19:05:18.962 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCpjPhl/xuuBhAA0LgKfXEvN2O11ZlRQJVOkw==", 2024-03-18T19:05:18.962 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:05:18.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:05:18.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:05:18.962 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:05:18.963 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:18.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:18.963 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-18T19:05:18.963 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCsjPhldiR4BRAA571oh6U8PzOv/4LzjYLF/A==", 2024-03-18T19:05:18.963 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:05:18.963 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:05:18.963 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:05:18.963 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:05:18.963 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:18.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:18.963 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-18T19:05:18.964 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlu+2xJBAANnXGGjfeqP6S2FTecsfOsg==", 2024-03-18T19:05:18.964 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:05:18.964 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:05:18.964 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-18T19:05:18.964 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-18T19:05:18.964 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:18.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:18.964 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-18T19:05:18.964 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlhJ9+KxAAEnrQrCrgJR6xI+pucwzAXA==", 2024-03-18T19:05:18.964 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:05:18.964 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:05:18.965 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:05:18.965 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:05:18.965 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:05:18.965 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:18.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:18.965 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.2", 2024-03-18T19:05:18.965 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlxBUpKBAAIGTwmLWyj+P2Yvl55QVcWA==", 2024-03-18T19:05:18.965 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:05:18.965 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:05:18.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:05:18.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:05:18.966 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:05:18.966 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:18.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:18.966 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.3", 2024-03-18T19:05:18.966 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlmao3MRAAHeiQ6eyBh9QjjKPCtxkC8g==", 2024-03-18T19:05:18.966 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:05:18.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:05:18.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:05:18.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:05:18.966 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:05:18.967 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:18.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:18.967 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-18T19:05:18.967 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCRjPhlemOuKRAAVUggmtB7+EiVgyfJZ8TCCA==", 2024-03-18T19:05:18.967 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:05:18.967 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:05:18.967 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:05:18.967 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:05:18.967 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:05:18.967 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:18.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:18.967 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-18T19:05:18.968 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhleIuOCBAA9CH+AHewQbn+gpVr+6lJyw==", 2024-03-18T19:05:18.968 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:05:18.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:05:18.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T19:05:18.968 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:05:18.968 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:18.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:18.968 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-18T19:05:18.968 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlL0pTBBAAdFiPcdwwz514cuLAvioxYA==", 2024-03-18T19:05:18.968 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:05:18.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:05:18.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T19:05:18.968 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:05:18.969 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.969 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:18.969 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-18T19:05:18.969 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:05:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.960+0000 7f60608c3700 1 -- 172.21.15.1:0/2879597765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f603c06adb0 msgr2=0x7f603c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.960+0000 7f60608c3700 1 --2- 172.21.15.1:0/2879597765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f603c06adb0 0x7f603c06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f604c009840 tx=0x7f604c008030 comp rx=0 tx=0).stop 2024-03-18T19:05:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.960+0000 7f60608c3700 1 -- 172.21.15.1:0/2879597765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c0cf750 msgr2=0x7f605c156620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.960+0000 7f60608c3700 1 --2- 172.21.15.1:0/2879597765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c0cf750 0x7f605c156620 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f605000e810 tx=0x7f605000ebd0 comp rx=0 tx=0).stop 2024-03-18T19:05:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.960+0000 7f60608c3700 1 -- 172.21.15.1:0/2879597765 shutdown_connections 2024-03-18T19:05:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.960+0000 7f60608c3700 1 --2- 172.21.15.1:0/2879597765 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f603c06adb0 0x7f603c06d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.960+0000 7f60608c3700 1 --2- 172.21.15.1:0/2879597765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f605c099f80 0x7f605c0c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.960+0000 7f60608c3700 1 --2- 172.21.15.1:0/2879597765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f605c15b4d0 0x7f605c0af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.960+0000 7f60608c3700 1 --2- 172.21.15.1:0/2879597765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f605c0cf750 0x7f605c156620 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:18.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.960+0000 7f60608c3700 1 -- 172.21.15.1:0/2879597765 >> 172.21.15.1:0/2879597765 conn(0x7f605c0b2cb0 msgr2=0x7f605c0cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:18.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.960+0000 7f60608c3700 1 -- 172.21.15.1:0/2879597765 shutdown_connections 2024-03-18T19:05:18.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:18.960+0000 7f60608c3700 1 -- 172.21.15.1:0/2879597765 wait complete. 2024-03-18T19:05:18.972 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:05:19.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.102+0000 7f01c1c2a700 1 Processor -- start 2024-03-18T19:05:19.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.102+0000 7f01c1c2a700 1 -- start start 2024-03-18T19:05:19.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.103+0000 7f01c1c2a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01bc06a750 0x7f01bc0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:19.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.103+0000 7f01c1c2a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01bc06ee90 con 0x7f01bc06ea10 2024-03-18T19:05:19.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.103+0000 7f01c1c2a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01bc06e1f0 con 0x7f01bc0ace80 2024-03-18T19:05:19.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.103+0000 7f01c1c2a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc06e4d0 con 0x7f01bc06a750 2024-03-18T19:05:19.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.103+0000 7f01baffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f01bc06ea10 0x7f01bc06ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44308/0 (socket says 172.21.15.1:44308) 2024-03-18T19:05:19.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.103+0000 7f01baffd700 1 -- 172.21.15.1:0/3396462532 learned_addr learned my addr 172.21.15.1:0/3396462532 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:19.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.103+0000 7f01ba7fc700 1 -- 172.21.15.1:0/3396462532 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809870689 0 0) 0x7f01bc06ee90 con 0x7f01bc06ea10 2024-03-18T19:05:19.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.103+0000 7f01ba7fc700 1 -- 172.21.15.1:0/3396462532 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a80035b0 con 0x7f01bc06ea10 2024-03-18T19:05:19.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.103+0000 7f01bb7fe700 1 --2- 172.21.15.1:0/3396462532 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01bc06a750 0x7f01bc0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:19.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.103+0000 7f01ba7fc700 1 -- 172.21.15.1:0/3396462532 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 486679456 0 0) 0x7f01a80035b0 con 0x7f01bc06ea10 2024-03-18T19:05:19.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.103+0000 7f01ba7fc700 1 -- 172.21.15.1:0/3396462532 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01bc06a750 msgr2=0x7f01bc0ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.103+0000 7f01ba7fc700 1 --2- 172.21.15.1:0/3396462532 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01bc06a750 0x7f01bc0ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.103+0000 7f01ba7fc700 1 -- 172.21.15.1:0/3396462532 >> v1:172.21.15.165:6789/0 conn(0x7f01bc0ace80 legacy=0x7f01bc0ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.103+0000 7f01ba7fc700 1 -- 172.21.15.1:0/3396462532 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01bc0cc2a0 con 0x7f01bc06ea10 2024-03-18T19:05:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.103+0000 7f01ba7fc700 1 -- 172.21.15.1:0/3396462532 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f01a4001670 con 0x7f01bc06ea10 2024-03-18T19:05:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.104+0000 7f01ba7fc700 1 -- 172.21.15.1:0/3396462532 >> v1:172.21.15.1:6789/0 conn(0x7f01bc06ea10 legacy=0x7f01bc06ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.104+0000 7f01ba7fc700 1 --2- 172.21.15.1:0/3396462532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01a8003a30 0x7f01a8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.104+0000 7f01ba7fc700 1 --2- 172.21.15.1:0/3396462532 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01bc06a750 0x7f01a8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.104+0000 7f01ba7fc700 1 --2- 172.21.15.1:0/3396462532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a8004d90 0x7f01a8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:19.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.104+0000 7f01ba7fc700 1 -- 172.21.15.1:0/3396462532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc06e4d0 con 0x7f01a8004d90 2024-03-18T19:05:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.104+0000 7f01ba7fc700 1 -- 172.21.15.1:0/3396462532 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a800bac0 con 0x7f01a8003a30 2024-03-18T19:05:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.104+0000 7f01ba7fc700 1 -- 172.21.15.1:0/3396462532 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a800bc80 con 0x7f01bc06a750 2024-03-18T19:05:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.104+0000 7f01bbfff700 1 --2- 172.21.15.1:0/3396462532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a8004d90 0x7f01a8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.104+0000 7f01bb7fe700 1 --2- 172.21.15.1:0/3396462532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01a8003a30 0x7f01a8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.104+0000 7f01baffd700 1 --2- 172.21.15.1:0/3396462532 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01bc06a750 0x7f01a8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.104+0000 7f01bbfff700 1 -- 172.21.15.1:0/3396462532 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01bc06a750 msgr2=0x7f01a8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.104+0000 7f01bbfff700 1 --2- 172.21.15.1:0/3396462532 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01bc06a750 0x7f01a8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.104+0000 7f01bbfff700 1 -- 172.21.15.1:0/3396462532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01a8003a30 msgr2=0x7f01a8003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.104+0000 7f01bbfff700 1 --2- 172.21.15.1:0/3396462532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01a8003a30 0x7f01a8003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.104+0000 7f01bbfff700 1 -- 172.21.15.1:0/3396462532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a80035b0 con 0x7f01a8004d90 2024-03-18T19:05:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.104+0000 7f01bbfff700 1 --2- 172.21.15.1:0/3396462532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a8004d90 0x7f01a8005210 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f01b400c160 tx=0x7f01a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9a4304fae31b2bd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:19.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.104+0000 7f01ba7fc700 1 -- 172.21.15.1:0/3396462532 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b4001db0 con 0x7f01a8004d90 2024-03-18T19:05:19.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.104+0000 7f01ba7fc700 1 -- 172.21.15.1:0/3396462532 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b400cca0 con 0x7f01a8004d90 2024-03-18T19:05:19.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.104+0000 7f01ba7fc700 1 -- 172.21.15.1:0/3396462532 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b4003c40 con 0x7f01a8004d90 2024-03-18T19:05:19.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.105+0000 7f01c1c2a700 1 -- 172.21.15.1:0/3396462532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a8004d90 msgr2=0x7f01a8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:19.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.105+0000 7f01c1c2a700 1 --2- 172.21.15.1:0/3396462532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a8004d90 0x7f01a8005210 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f01b400c160 tx=0x7f01a8005880 comp rx=0 tx=0).stop 2024-03-18T19:05:19.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.105+0000 7f01bbfff700 1 -- 172.21.15.1:0/3396462532 reap_dead start 2024-03-18T19:05:19.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.105+0000 7f01c1c2a700 1 -- 172.21.15.1:0/3396462532 shutdown_connections 2024-03-18T19:05:19.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.105+0000 7f01c1c2a700 1 -- 172.21.15.1:0/3396462532 >> 172.21.15.1:0/3396462532 conn(0x7f01bc0cc990 msgr2=0x7f01bc0ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:19.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.105+0000 7f01c1c2a700 1 -- 172.21.15.1:0/3396462532 shutdown_connections 2024-03-18T19:05:19.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.105+0000 7f01c1c2a700 1 -- 172.21.15.1:0/3396462532 wait complete. 2024-03-18T19:05:19.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.106+0000 7f01c1c2a700 1 Processor -- start 2024-03-18T19:05:19.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.106+0000 7f01c1c2a700 1 -- start start 2024-03-18T19:05:19.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.107+0000 7f01c1c2a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01bc1e4670 0x7f01bc14fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:19.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.107+0000 7f01c1c2a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01bc1e4a50 0x7f01bc1502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:19.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.107+0000 7f01c1c2a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01bc0cf7e0 0x7f01bc1f3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:19.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.107+0000 7f01c1c2a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc0ce4f0 con 0x7f01bc0cf7e0 2024-03-18T19:05:19.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.107+0000 7f01c1c2a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc0cf350 con 0x7f01bc1e4a50 2024-03-18T19:05:19.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.107+0000 7f01c1c2a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc150800 con 0x7f01bc1e4670 2024-03-18T19:05:19.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.107+0000 7f01bbfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01bc0cf7e0 0x7f01bc1f3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:19.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.107+0000 7f01bbfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01bc0cf7e0 0x7f01bc1f3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33816/0 (socket says 172.21.15.1:33816) 2024-03-18T19:05:19.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.107+0000 7f01bbfff700 1 -- 172.21.15.1:0/556552893 learned_addr learned my addr 172.21.15.1:0/556552893 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:19.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.107+0000 7f01bb7fe700 1 --2- 172.21.15.1:0/556552893 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01bc1e4670 0x7f01bc14fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:19.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.107+0000 7f01baffd700 1 --2- 172.21.15.1:0/556552893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01bc1e4a50 0x7f01bc1502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:19.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.107+0000 7f01bbfff700 1 -- 172.21.15.1:0/556552893 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01bc1e4670 msgr2=0x7f01bc14fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:19.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.107+0000 7f01bbfff700 1 --2- 172.21.15.1:0/556552893 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01bc1e4670 0x7f01bc14fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.107+0000 7f01bbfff700 1 -- 172.21.15.1:0/556552893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01bc1e4a50 msgr2=0x7f01bc1502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:19.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.107+0000 7f01bbfff700 1 --2- 172.21.15.1:0/556552893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01bc1e4a50 0x7f01bc1502c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.107+0000 7f01bbfff700 1 -- 172.21.15.1:0/556552893 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01bc0ce740 con 0x7f01bc0cf7e0 2024-03-18T19:05:19.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.108+0000 7f01bbfff700 1 --2- 172.21.15.1:0/556552893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01bc0cf7e0 0x7f01bc1f3330 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f01b4004000 tx=0x7f01b40014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:19.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.108+0000 7f01b3fff700 1 -- 172.21.15.1:0/556552893 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b4003c10 con 0x7f01bc0cf7e0 2024-03-18T19:05:19.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.108+0000 7f01b3fff700 1 -- 172.21.15.1:0/556552893 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b4003d70 con 0x7f01bc0cf7e0 2024-03-18T19:05:19.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.108+0000 7f01c1c2a700 1 -- 172.21.15.1:0/556552893 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01bc0cea20 con 0x7f01bc0cf7e0 2024-03-18T19:05:19.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.108+0000 7f01b3fff700 1 -- 172.21.15.1:0/556552893 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b4005310 con 0x7f01bc0cf7e0 2024-03-18T19:05:19.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.108+0000 7f01c1c2a700 1 -- 172.21.15.1:0/556552893 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01bc0cefe0 con 0x7f01bc0cf7e0 2024-03-18T19:05:19.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.109+0000 7f01c1c2a700 1 -- 172.21.15.1:0/556552893 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01bc0676a0 con 0x7f01bc0cf7e0 2024-03-18T19:05:19.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.109+0000 7f01b3fff700 1 -- 172.21.15.1:0/556552893 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f01b4019040 con 0x7f01bc0cf7e0 2024-03-18T19:05:19.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.109+0000 7f01b3fff700 1 --2- 172.21.15.1:0/556552893 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f019c06adb0 0x7f019c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:19.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.110+0000 7f01b3fff700 1 -- 172.21.15.1:0/556552893 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f01b4089db0 con 0x7f01bc0cf7e0 2024-03-18T19:05:19.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.110+0000 7f01bb7fe700 1 --2- 172.21.15.1:0/556552893 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f019c06adb0 0x7f019c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:19.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.110+0000 7f01bb7fe700 1 --2- 172.21.15.1:0/556552893 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f019c06adb0 0x7f019c06d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f01ac009c60 tx=0x7f01ac009210 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:19.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.114+0000 7f01b3fff700 1 -- 172.21.15.1:0/556552893 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f01b40551e0 con 0x7f01bc0cf7e0 2024-03-18T19:05:19.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.319+0000 7f01c1c2a700 1 -- 172.21.15.1:0/556552893 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01bc0c8590 con 0x7f01bc0cf7e0 2024-03-18T19:05:19.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.320+0000 7f01b3fff700 1 -- 172.21.15.1:0/556552893 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+12546 (secure 0 0 0) 0x7f01b4017020 con 0x7f01bc0cf7e0 2024-03-18T19:05:19.317 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:19.317 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":85,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:05:18.508087+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:05:19.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.322+0000 7f01c1c2a700 1 -- 172.21.15.1:0/556552893 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f019c06adb0 msgr2=0x7f019c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:19.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.322+0000 7f01c1c2a700 1 --2- 172.21.15.1:0/556552893 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f019c06adb0 0x7f019c06d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f01ac009c60 tx=0x7f01ac009210 comp rx=0 tx=0).stop 2024-03-18T19:05:19.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.322+0000 7f01c1c2a700 1 -- 172.21.15.1:0/556552893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01bc0cf7e0 msgr2=0x7f01bc1f3330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:19.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.322+0000 7f01c1c2a700 1 --2- 172.21.15.1:0/556552893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01bc0cf7e0 0x7f01bc1f3330 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f01b4004000 tx=0x7f01b40014a0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.322+0000 7f01c1c2a700 1 -- 172.21.15.1:0/556552893 shutdown_connections 2024-03-18T19:05:19.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.322+0000 7f01c1c2a700 1 --2- 172.21.15.1:0/556552893 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f019c06adb0 0x7f019c06d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.322+0000 7f01c1c2a700 1 --2- 172.21.15.1:0/556552893 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01bc1e4670 0x7f01bc14fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.322+0000 7f01c1c2a700 1 --2- 172.21.15.1:0/556552893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01bc0cf7e0 0x7f01bc1f3330 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.322+0000 7f01c1c2a700 1 --2- 172.21.15.1:0/556552893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01bc1e4a50 0x7f01bc1502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.322+0000 7f01c1c2a700 1 -- 172.21.15.1:0/556552893 >> 172.21.15.1:0/556552893 conn(0x7f01bc0cc990 msgr2=0x7f01bc15d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:19.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.323+0000 7f01c1c2a700 1 -- 172.21.15.1:0/556552893 shutdown_connections 2024-03-18T19:05:19.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.323+0000 7f01c1c2a700 1 -- 172.21.15.1:0/556552893 wait complete. 2024-03-18T19:05:19.331 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:05:19.332 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:05:19.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.464+0000 7f2e2d834700 1 Processor -- start 2024-03-18T19:05:19.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.464+0000 7f2e2d834700 1 -- start start 2024-03-18T19:05:19.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.464+0000 7f2e2d834700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e28151150 0x7f2e2815b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:19.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.464+0000 7f2e2d834700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e2806e2c0 0x7f2e2806e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:19.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.464+0000 7f2e2d834700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e2806f160 con 0x7f2e2806ed30 2024-03-18T19:05:19.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.464+0000 7f2e2d834700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e2815b670 con 0x7f2e28151150 2024-03-18T19:05:19.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.464+0000 7f2e2d834700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2806d220 con 0x7f2e2806e2c0 2024-03-18T19:05:19.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.465+0000 7f2e26ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e28151150 0x7f2e2815b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:19.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.465+0000 7f2e26ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e28151150 0x7f2e2815b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33820/0 (socket says 172.21.15.1:33820) 2024-03-18T19:05:19.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.465+0000 7f2e277fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2e2806ed30 0x7f2e2815dcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44320/0 (socket says 172.21.15.1:44320) 2024-03-18T19:05:19.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.465+0000 7f2e26ffd700 1 -- 172.21.15.1:0/333653522 learned_addr learned my addr 172.21.15.1:0/333653522 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.465+0000 7f2e267fc700 1 --2- 172.21.15.1:0/333653522 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e2806e2c0 0x7f2e2806e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.465+0000 7f2e26ffd700 1 -- 172.21.15.1:0/333653522 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e2806e2c0 msgr2=0x7f2e2806e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.465+0000 7f2e26ffd700 1 --2- 172.21.15.1:0/333653522 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e2806e2c0 0x7f2e2806e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.465+0000 7f2e26ffd700 1 -- 172.21.15.1:0/333653522 >> v1:172.21.15.1:6789/0 conn(0x7f2e2806ed30 legacy=0x7f2e2815dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.465+0000 7f2e26ffd700 1 -- 172.21.15.1:0/333653522 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2806d500 con 0x7f2e28151150 2024-03-18T19:05:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.465+0000 7f2e26ffd700 1 --2- 172.21.15.1:0/333653522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e28151150 0x7f2e2815b130 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f2e180098f0 tx=0x7f2e18009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd895ee2d5470a59 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.465+0000 7f2e25ffb700 1 -- 172.21.15.1:0/333653522 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1800b8a0 con 0x7f2e28151150 2024-03-18T19:05:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.465+0000 7f2e25ffb700 1 -- 172.21.15.1:0/333653522 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1800ba00 con 0x7f2e28151150 2024-03-18T19:05:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.465+0000 7f2e25ffb700 1 -- 172.21.15.1:0/333653522 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1800bd90 con 0x7f2e28151150 2024-03-18T19:05:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.466+0000 7f2e2d834700 1 -- 172.21.15.1:0/333653522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e28151150 msgr2=0x7f2e2815b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.466+0000 7f2e2d834700 1 --2- 172.21.15.1:0/333653522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e28151150 0x7f2e2815b130 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f2e180098f0 tx=0x7f2e18009c00 comp rx=0 tx=0).stop 2024-03-18T19:05:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.466+0000 7f2e2d834700 1 -- 172.21.15.1:0/333653522 shutdown_connections 2024-03-18T19:05:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.466+0000 7f2e2d834700 1 --2- 172.21.15.1:0/333653522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e28151150 0x7f2e2815b130 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.466+0000 7f2e2d834700 1 --2- 172.21.15.1:0/333653522 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e2806e2c0 0x7f2e2806e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.466+0000 7f2e2d834700 1 -- 172.21.15.1:0/333653522 >> 172.21.15.1:0/333653522 conn(0x7f2e280b1350 msgr2=0x7f2e280b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:19.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.466+0000 7f2e2d834700 1 -- 172.21.15.1:0/333653522 shutdown_connections 2024-03-18T19:05:19.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.466+0000 7f2e2d834700 1 -- 172.21.15.1:0/333653522 wait complete. 2024-03-18T19:05:19.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.467+0000 7f2e2d834700 1 Processor -- start 2024-03-18T19:05:19.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.467+0000 7f2e2d834700 1 -- start start 2024-03-18T19:05:19.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.468+0000 7f2e2d834700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e2806e2c0 0x7f2e281460a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:19.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.468+0000 7f2e2d834700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e2806ed30 0x7f2e281543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:19.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.468+0000 7f2e2d834700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e28151150 0x7f2e281491c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:19.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.468+0000 7f2e2d834700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e28148da0 con 0x7f2e2806e2c0 2024-03-18T19:05:19.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.468+0000 7f2e2d834700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e28148120 con 0x7f2e28151150 2024-03-18T19:05:19.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.468+0000 7f2e2d834700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e28148400 con 0x7f2e2806ed30 2024-03-18T19:05:19.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.468+0000 7f2e26ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e2806e2c0 0x7f2e281460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:19.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.468+0000 7f2e26ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e2806e2c0 0x7f2e281460a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33826/0 (socket says 172.21.15.1:33826) 2024-03-18T19:05:19.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.468+0000 7f2e26ffd700 1 -- 172.21.15.1:0/3180652997 learned_addr learned my addr 172.21.15.1:0/3180652997 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:19.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.468+0000 7f2e26ffd700 1 -- 172.21.15.1:0/3180652997 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e2806ed30 msgr2=0x7f2e281543f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:19.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.468+0000 7f2e267fc700 1 --2- 172.21.15.1:0/3180652997 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e2806ed30 0x7f2e281543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:19.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.468+0000 7f2e26ffd700 1 --2- 172.21.15.1:0/3180652997 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e2806ed30 0x7f2e281543f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.468+0000 7f2e277fe700 1 --2- 172.21.15.1:0/3180652997 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e28151150 0x7f2e281491c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:19.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.468+0000 7f2e26ffd700 1 -- 172.21.15.1:0/3180652997 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e28151150 msgr2=0x7f2e281491c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:19.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.468+0000 7f2e26ffd700 1 --2- 172.21.15.1:0/3180652997 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e28151150 0x7f2e281491c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.468+0000 7f2e26ffd700 1 -- 172.21.15.1:0/3180652997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e18009550 con 0x7f2e2806e2c0 2024-03-18T19:05:19.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.468+0000 7f2e26ffd700 1 --2- 172.21.15.1:0/3180652997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e2806e2c0 0x7f2e281460a0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f2e18000c00 tx=0x7f2e18004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:19.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.469+0000 7f2e1ffff700 1 -- 172.21.15.1:0/3180652997 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18004480 con 0x7f2e2806e2c0 2024-03-18T19:05:19.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.469+0000 7f2e1ffff700 1 -- 172.21.15.1:0/3180652997 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1801f070 con 0x7f2e2806e2c0 2024-03-18T19:05:19.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.469+0000 7f2e2d834700 1 -- 172.21.15.1:0/3180652997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e281486e0 con 0x7f2e2806e2c0 2024-03-18T19:05:19.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.469+0000 7f2e1ffff700 1 -- 172.21.15.1:0/3180652997 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e180038a0 con 0x7f2e2806e2c0 2024-03-18T19:05:19.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.469+0000 7f2e2d834700 1 -- 172.21.15.1:0/3180652997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e281465e0 con 0x7f2e2806e2c0 2024-03-18T19:05:19.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.470+0000 7f2e2d834700 1 -- 172.21.15.1:0/3180652997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e280676a0 con 0x7f2e2806e2c0 2024-03-18T19:05:19.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.470+0000 7f2e1ffff700 1 -- 172.21.15.1:0/3180652997 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2e18021030 con 0x7f2e2806e2c0 2024-03-18T19:05:19.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.470+0000 7f2e1ffff700 1 --2- 172.21.15.1:0/3180652997 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2e0806adb0 0x7f2e0806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:19.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.470+0000 7f2e1ffff700 1 -- 172.21.15.1:0/3180652997 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f2e1808a640 con 0x7f2e2806e2c0 2024-03-18T19:05:19.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.471+0000 7f2e267fc700 1 --2- 172.21.15.1:0/3180652997 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2e0806adb0 0x7f2e0806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:19.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.471+0000 7f2e267fc700 1 --2- 172.21.15.1:0/3180652997 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2e0806adb0 0x7f2e0806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2e281f6bc0 tx=0x7f2e10008040 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:19.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.475+0000 7f2e1ffff700 1 -- 172.21.15.1:0/3180652997 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2e18055a40 con 0x7f2e2806e2c0 2024-03-18T19:05:19.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.676+0000 7f2e2d834700 1 -- 172.21.15.1:0/3180652997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2e28159d60 con 0x7f2e2806e2c0 2024-03-18T19:05:19.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.677+0000 7f2e1ffff700 1 -- 172.21.15.1:0/3180652997 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2e18059060 con 0x7f2e2806e2c0 2024-03-18T19:05:19.673 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:05:19.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.679+0000 7f2e2d834700 1 -- 172.21.15.1:0/3180652997 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2e0806adb0 msgr2=0x7f2e0806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:19.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.679+0000 7f2e2d834700 1 --2- 172.21.15.1:0/3180652997 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2e0806adb0 0x7f2e0806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2e281f6bc0 tx=0x7f2e10008040 comp rx=0 tx=0).stop 2024-03-18T19:05:19.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.679+0000 7f2e2d834700 1 -- 172.21.15.1:0/3180652997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e2806e2c0 msgr2=0x7f2e281460a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:19.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.679+0000 7f2e2d834700 1 --2- 172.21.15.1:0/3180652997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e2806e2c0 0x7f2e281460a0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f2e18000c00 tx=0x7f2e18004750 comp rx=0 tx=0).stop 2024-03-18T19:05:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.679+0000 7f2e2d834700 1 -- 172.21.15.1:0/3180652997 shutdown_connections 2024-03-18T19:05:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.679+0000 7f2e2d834700 1 --2- 172.21.15.1:0/3180652997 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2e0806adb0 0x7f2e0806d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.679+0000 7f2e2d834700 1 --2- 172.21.15.1:0/3180652997 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e28151150 0x7f2e281491c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.679+0000 7f2e2d834700 1 --2- 172.21.15.1:0/3180652997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e2806e2c0 0x7f2e281460a0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.679+0000 7f2e2d834700 1 --2- 172.21.15.1:0/3180652997 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e2806ed30 0x7f2e281543f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.679+0000 7f2e2d834700 1 -- 172.21.15.1:0/3180652997 >> 172.21.15.1:0/3180652997 conn(0x7f2e280b1350 msgr2=0x7f2e280affd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.679+0000 7f2e2d834700 1 -- 172.21.15.1:0/3180652997 shutdown_connections 2024-03-18T19:05:19.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.679+0000 7f2e2d834700 1 -- 172.21.15.1:0/3180652997 wait complete. 2024-03-18T19:05:19.687 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T19:05:19.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.817+0000 7fed02050700 1 Processor -- start 2024-03-18T19:05:19.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.817+0000 7fed02050700 1 -- start start 2024-03-18T19:05:19.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.818+0000 7fed02050700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fecfc0cfda0 0x7fecfc166120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:19.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.818+0000 7fed02050700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecfc0cd0d0 con 0x7fecfc0cee90 2024-03-18T19:05:19.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.818+0000 7fed02050700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecfc0b3f00 con 0x7fecfc0b6b60 2024-03-18T19:05:19.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.818+0000 7fed02050700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecfc0b41e0 con 0x7fecfc0cfda0 2024-03-18T19:05:19.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.818+0000 7fecfbfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fecfc0cfda0 0x7fecfc166120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:19.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.818+0000 7fecfaffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fecfc0cee90 0x7fecfc0cf690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44326/0 (socket says 172.21.15.1:44326) 2024-03-18T19:05:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.818+0000 7fecfbfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fecfc0cfda0 0x7fecfc166120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33836/0 (socket says 172.21.15.1:33836) 2024-03-18T19:05:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.818+0000 7fecfaffd700 1 -- 172.21.15.1:0/62354376 learned_addr learned my addr 172.21.15.1:0/62354376 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.818+0000 7fecfbfff700 1 -- 172.21.15.1:0/62354376 >> v1:172.21.15.165:6789/0 conn(0x7fecfc0b6b60 legacy=0x7fecfc0cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.818+0000 7fecfbfff700 1 -- 172.21.15.1:0/62354376 >> v1:172.21.15.1:6789/0 conn(0x7fecfc0cee90 legacy=0x7fecfc0cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.818+0000 7fecfa7fc700 1 -- 172.21.15.1:0/62354376 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997635300 0 0) 0x7fecfc0cd0d0 con 0x7fecfc0cee90 2024-03-18T19:05:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.818+0000 7fecfbfff700 1 -- 172.21.15.1:0/62354376 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecfc0b44c0 con 0x7fecfc0cfda0 2024-03-18T19:05:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.819+0000 7fecfbfff700 1 --2- 172.21.15.1:0/62354376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecfc0cfda0 0x7fecfc166120 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fecf4009d90 tx=0x7fecf400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b32c073d5275b15 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.819+0000 7fecfa7fc700 1 -- 172.21.15.1:0/62354376 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf400e070 con 0x7fecfc0cfda0 2024-03-18T19:05:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.819+0000 7fecfa7fc700 1 -- 172.21.15.1:0/62354376 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf400bbd0 con 0x7fecfc0cfda0 2024-03-18T19:05:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.819+0000 7fecfa7fc700 1 -- 172.21.15.1:0/62354376 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4012430 con 0x7fecfc0cfda0 2024-03-18T19:05:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.819+0000 7fed02050700 1 -- 172.21.15.1:0/62354376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecfc0cfda0 msgr2=0x7fecfc166120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.819+0000 7fed02050700 1 --2- 172.21.15.1:0/62354376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecfc0cfda0 0x7fecfc166120 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fecf4009d90 tx=0x7fecf400b330 comp rx=0 tx=0).stop 2024-03-18T19:05:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.819+0000 7fed02050700 1 -- 172.21.15.1:0/62354376 shutdown_connections 2024-03-18T19:05:19.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.819+0000 7fed02050700 1 --2- 172.21.15.1:0/62354376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecfc0cfda0 0x7fecfc166120 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.819+0000 7fed02050700 1 -- 172.21.15.1:0/62354376 >> 172.21.15.1:0/62354376 conn(0x7fecfc0b2ba0 msgr2=0x7fecfc0b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:19.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.819+0000 7fed02050700 1 -- 172.21.15.1:0/62354376 shutdown_connections 2024-03-18T19:05:19.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.819+0000 7fed02050700 1 -- 172.21.15.1:0/62354376 wait complete. 2024-03-18T19:05:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.820+0000 7fed02050700 1 Processor -- start 2024-03-18T19:05:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.821+0000 7fed02050700 1 -- start start 2024-03-18T19:05:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.821+0000 7fed02050700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecfc0b6b60 0x7fecfc1516c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.821+0000 7fed02050700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecfc0cee90 0x7fecfc0c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:19.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.821+0000 7fed02050700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fecfc0cfda0 0x7fecfc14a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.821+0000 7fed02050700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc0c5d60 con 0x7fecfc0b6b60 2024-03-18T19:05:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.821+0000 7fed02050700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc0c4e20 con 0x7fecfc0cee90 2024-03-18T19:05:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.821+0000 7fed02050700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc0c50e0 con 0x7fecfc0cfda0 2024-03-18T19:05:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.821+0000 7fecfb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecfc0b6b60 0x7fecfc1516c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.821+0000 7fecfb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecfc0b6b60 0x7fecfc1516c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33850/0 (socket says 172.21.15.1:33850) 2024-03-18T19:05:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.822+0000 7fecfb7fe700 1 -- 172.21.15.1:0/3385605033 learned_addr learned my addr 172.21.15.1:0/3385605033 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.822+0000 7fecfaffd700 1 --2- 172.21.15.1:0/3385605033 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecfc0cee90 0x7fecfc0c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.822+0000 7fecfbfff700 1 --2- 172.21.15.1:0/3385605033 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fecfc0cfda0 0x7fecfc14a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.822+0000 7fecfb7fe700 1 -- 172.21.15.1:0/3385605033 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fecfc0cfda0 msgr2=0x7fecfc14a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.822+0000 7fecfb7fe700 1 --2- 172.21.15.1:0/3385605033 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fecfc0cfda0 0x7fecfc14a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.822+0000 7fecfb7fe700 1 -- 172.21.15.1:0/3385605033 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecfc0cee90 msgr2=0x7fecfc0c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.822+0000 7fecfb7fe700 1 --2- 172.21.15.1:0/3385605033 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecfc0cee90 0x7fecfc0c6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.822+0000 7fecfb7fe700 1 -- 172.21.15.1:0/3385605033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf4009a70 con 0x7fecfc0b6b60 2024-03-18T19:05:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.822+0000 7fecfbfff700 1 --2- 172.21.15.1:0/3385605033 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fecfc0cfda0 0x7fecfc14a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.822+0000 7fecfb7fe700 1 --2- 172.21.15.1:0/3385605033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecfc0b6b60 0x7fecfc1516c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fecec00c450 tx=0x7fecec00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.822+0000 7fecf3fff700 1 -- 172.21.15.1:0/3385605033 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec0052d0 con 0x7fecfc0b6b60 2024-03-18T19:05:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.822+0000 7fecf3fff700 1 -- 172.21.15.1:0/3385605033 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecec005430 con 0x7fecfc0b6b60 2024-03-18T19:05:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.822+0000 7fecf3fff700 1 -- 172.21.15.1:0/3385605033 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec0038c0 con 0x7fecfc0b6b60 2024-03-18T19:05:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.822+0000 7fed02050700 1 -- 172.21.15.1:0/3385605033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecfc0c53c0 con 0x7fecfc0b6b60 2024-03-18T19:05:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.822+0000 7fed02050700 1 -- 172.21.15.1:0/3385605033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecfc0c5980 con 0x7fecfc0b6b60 2024-03-18T19:05:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.823+0000 7fed02050700 1 -- 172.21.15.1:0/3385605033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecfc0b3d20 con 0x7fecfc0b6b60 2024-03-18T19:05:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.824+0000 7fecf3fff700 1 -- 172.21.15.1:0/3385605033 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fecec003a20 con 0x7fecfc0b6b60 2024-03-18T19:05:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.824+0000 7fecf3fff700 1 --2- 172.21.15.1:0/3385605033 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fecdc06adb0 0x7fecdc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.824+0000 7fecfaffd700 1 --2- 172.21.15.1:0/3385605033 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fecdc06adb0 0x7fecdc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.824+0000 7fecf3fff700 1 -- 172.21.15.1:0/3385605033 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fecec088630 con 0x7fecfc0b6b60 2024-03-18T19:05:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.825+0000 7fecfaffd700 1 --2- 172.21.15.1:0/3385605033 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fecdc06adb0 0x7fecdc06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fece4007fd0 tx=0x7fece4007c20 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:19.829+0000 7fecf3fff700 1 -- 172.21.15.1:0/3385605033 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fecec053890 con 0x7fecfc0b6b60 2024-03-18T19:05:20.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.033+0000 7fed02050700 1 -- 172.21.15.1:0/3385605033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fecfc0c5b10 con 0x7fecfc0b6b60 2024-03-18T19:05:20.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.034+0000 7fecf3fff700 1 -- 172.21.15.1:0/3385605033 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v85) v1 ==== 84+0+65 (secure 0 0 0) 0x7fecec056eb0 con 0x7fecfc0b6b60 2024-03-18T19:05:20.030 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:20.030 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T19:05:20.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:20.030 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-18T19:05:20.030 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-18T19:05:20.030 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:20.030 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T19:05:20.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.036+0000 7fed02050700 1 -- 172.21.15.1:0/3385605033 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fecdc06adb0 msgr2=0x7fecdc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:20.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.036+0000 7fed02050700 1 --2- 172.21.15.1:0/3385605033 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fecdc06adb0 0x7fecdc06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fece4007fd0 tx=0x7fece4007c20 comp rx=0 tx=0).stop 2024-03-18T19:05:20.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.036+0000 7fed02050700 1 -- 172.21.15.1:0/3385605033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecfc0b6b60 msgr2=0x7fecfc1516c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:20.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.036+0000 7fed02050700 1 --2- 172.21.15.1:0/3385605033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecfc0b6b60 0x7fecfc1516c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fecec00c450 tx=0x7fecec00c810 comp rx=0 tx=0).stop 2024-03-18T19:05:20.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.036+0000 7fed02050700 1 -- 172.21.15.1:0/3385605033 shutdown_connections 2024-03-18T19:05:20.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.036+0000 7fed02050700 1 --2- 172.21.15.1:0/3385605033 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fecdc06adb0 0x7fecdc06d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:20.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.036+0000 7fed02050700 1 --2- 172.21.15.1:0/3385605033 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fecfc0cfda0 0x7fecfc14a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:20.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.036+0000 7fed02050700 1 --2- 172.21.15.1:0/3385605033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fecfc0b6b60 0x7fecfc1516c0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:20.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.036+0000 7fed02050700 1 --2- 172.21.15.1:0/3385605033 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecfc0cee90 0x7fecfc0c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:20.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.036+0000 7fed02050700 1 -- 172.21.15.1:0/3385605033 >> 172.21.15.1:0/3385605033 conn(0x7fecfc0b2ba0 msgr2=0x7fecfc0b2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:20.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.036+0000 7fed02050700 1 -- 172.21.15.1:0/3385605033 shutdown_connections 2024-03-18T19:05:20.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.036+0000 7fed02050700 1 -- 172.21.15.1:0/3385605033 wait complete. 2024-03-18T19:05:20.044 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T19:05:20.045 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T19:05:20.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.178+0000 7f42c8cda700 1 Processor -- start 2024-03-18T19:05:20.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.178+0000 7f42c8cda700 1 -- start start 2024-03-18T19:05:20.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.179+0000 7f42c8cda700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42c406fe10 0x7f42c40b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:20.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.179+0000 7f42c8cda700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c415d970 con 0x7f42c415d590 2024-03-18T19:05:20.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.179+0000 7f42c8cda700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c4072320 con 0x7f42c406ae60 2024-03-18T19:05:20.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.179+0000 7f42c8cda700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c4072600 con 0x7f42c406fe10 2024-03-18T19:05:20.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.179+0000 7f42c259c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f42c415d590 0x7f42c406a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44330/0 (socket says 172.21.15.1:44330) 2024-03-18T19:05:20.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.179+0000 7f42c259c700 1 -- 172.21.15.1:0/3277282830 learned_addr learned my addr 172.21.15.1:0/3277282830 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:20.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c159a700 1 -- 172.21.15.1:0/3277282830 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307719123 0 0) 0x7f42c415d970 con 0x7f42c415d590 2024-03-18T19:05:20.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c159a700 1 -- 172.21.15.1:0/3277282830 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42ac0035b0 con 0x7f42c415d590 2024-03-18T19:05:20.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c1d9b700 1 --2- 172.21.15.1:0/3277282830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42c406fe10 0x7f42c40b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:20.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c159a700 1 -- 172.21.15.1:0/3277282830 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 114707183 0 0) 0x7f42ac0035b0 con 0x7f42c415d590 2024-03-18T19:05:20.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c159a700 1 -- 172.21.15.1:0/3277282830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42c406fe10 msgr2=0x7f42c40b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:20.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c159a700 1 --2- 172.21.15.1:0/3277282830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42c406fe10 0x7f42c40b3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:20.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c159a700 1 -- 172.21.15.1:0/3277282830 >> v1:172.21.15.165:6789/0 conn(0x7f42c406ae60 legacy=0x7f42c4167100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:20.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c159a700 1 -- 172.21.15.1:0/3277282830 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c40b30b0 con 0x7f42c415d590 2024-03-18T19:05:20.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c159a700 1 -- 172.21.15.1:0/3277282830 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f42b8001670 con 0x7f42c415d590 2024-03-18T19:05:20.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c159a700 1 -- 172.21.15.1:0/3277282830 >> v1:172.21.15.1:6789/0 conn(0x7f42c415d590 legacy=0x7f42c406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:20.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c159a700 1 --2- 172.21.15.1:0/3277282830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42ac003a30 0x7f42ac003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:20.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c159a700 1 --2- 172.21.15.1:0/3277282830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42c406fe10 0x7f42ac004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:20.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c159a700 1 --2- 172.21.15.1:0/3277282830 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42ac004d90 0x7f42ac005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:20.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c159a700 1 -- 172.21.15.1:0/3277282830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c4072600 con 0x7f42ac003a30 2024-03-18T19:05:20.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c159a700 1 -- 172.21.15.1:0/3277282830 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac00bb40 con 0x7f42ac004d90 2024-03-18T19:05:20.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c159a700 1 -- 172.21.15.1:0/3277282830 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac00bd00 con 0x7f42c406fe10 2024-03-18T19:05:20.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c259c700 1 --2- 172.21.15.1:0/3277282830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42ac003a30 0x7f42ac003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:20.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c1d9b700 1 --2- 172.21.15.1:0/3277282830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42c406fe10 0x7f42ac004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:20.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c2d9d700 1 --2- 172.21.15.1:0/3277282830 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42ac004d90 0x7f42ac005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:20.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c259c700 1 -- 172.21.15.1:0/3277282830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42c406fe10 msgr2=0x7f42ac004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:20.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c259c700 1 --2- 172.21.15.1:0/3277282830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42c406fe10 0x7f42ac004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:20.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c259c700 1 -- 172.21.15.1:0/3277282830 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42ac004d90 msgr2=0x7f42ac005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:20.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c259c700 1 --2- 172.21.15.1:0/3277282830 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42ac004d90 0x7f42ac005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:20.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c259c700 1 -- 172.21.15.1:0/3277282830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ac0035b0 con 0x7f42ac003a30 2024-03-18T19:05:20.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.180+0000 7f42c259c700 1 --2- 172.21.15.1:0/3277282830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42ac003a30 0x7f42ac003e50 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f42b800c240 tx=0x7f42ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49ee8089eab91853 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:20.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.181+0000 7f42c159a700 1 -- 172.21.15.1:0/3277282830 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b8007c10 con 0x7f42ac003a30 2024-03-18T19:05:20.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.181+0000 7f42c159a700 1 -- 172.21.15.1:0/3277282830 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b800cd20 con 0x7f42ac003a30 2024-03-18T19:05:20.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.181+0000 7f42c159a700 1 -- 172.21.15.1:0/3277282830 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b8005ba0 con 0x7f42ac003a30 2024-03-18T19:05:20.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.181+0000 7f42c8cda700 1 -- 172.21.15.1:0/3277282830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42ac003a30 msgr2=0x7f42ac003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:20.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.181+0000 7f42c8cda700 1 --2- 172.21.15.1:0/3277282830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42ac003a30 0x7f42ac003e50 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f42b800c240 tx=0x7f42ac005880 comp rx=0 tx=0).stop 2024-03-18T19:05:20.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.181+0000 7f42c2d9d700 1 -- 172.21.15.1:0/3277282830 reap_dead start 2024-03-18T19:05:20.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.181+0000 7f42c8cda700 1 -- 172.21.15.1:0/3277282830 shutdown_connections 2024-03-18T19:05:20.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.181+0000 7f42c8cda700 1 -- 172.21.15.1:0/3277282830 >> 172.21.15.1:0/3277282830 conn(0x7f42c40d00c0 msgr2=0x7f42c40d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:20.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.181+0000 7f42c8cda700 1 -- 172.21.15.1:0/3277282830 shutdown_connections 2024-03-18T19:05:20.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.181+0000 7f42c8cda700 1 -- 172.21.15.1:0/3277282830 wait complete. 2024-03-18T19:05:20.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.183+0000 7f42c8cda700 1 Processor -- start 2024-03-18T19:05:20.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.183+0000 7f42c8cda700 1 -- start start 2024-03-18T19:05:20.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.183+0000 7f42c8cda700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42ac003a30 0x7f42c4148d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:20.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.183+0000 7f42c8cda700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42c40ccf50 0x7f42c41e8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:20.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.183+0000 7f42c8cda700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42c415b3f0 0x7f42c415b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:20.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.183+0000 7f42c8cda700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c41f47c0 con 0x7f42c40ccf50 2024-03-18T19:05:20.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.183+0000 7f42c8cda700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c415bdc0 con 0x7f42ac003a30 2024-03-18T19:05:20.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.183+0000 7f42c8cda700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c40cd3d0 con 0x7f42c415b3f0 2024-03-18T19:05:20.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.183+0000 7f42c1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42c40ccf50 0x7f42c41e8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:20.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.184+0000 7f42c1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42c40ccf50 0x7f42c41e8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33878/0 (socket says 172.21.15.1:33878) 2024-03-18T19:05:20.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.184+0000 7f42c1d9b700 1 -- 172.21.15.1:0/2952235469 learned_addr learned my addr 172.21.15.1:0/2952235469 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:20.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.184+0000 7f42c2d9d700 1 --2- 172.21.15.1:0/2952235469 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42c415b3f0 0x7f42c415b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:20.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.184+0000 7f42c259c700 1 --2- 172.21.15.1:0/2952235469 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42ac003a30 0x7f42c4148d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:20.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.184+0000 7f42c1d9b700 1 -- 172.21.15.1:0/2952235469 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42c415b3f0 msgr2=0x7f42c415b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:20.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.184+0000 7f42c1d9b700 1 --2- 172.21.15.1:0/2952235469 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42c415b3f0 0x7f42c415b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:20.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.184+0000 7f42c1d9b700 1 -- 172.21.15.1:0/2952235469 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42ac003a30 msgr2=0x7f42c4148d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:20.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.184+0000 7f42c1d9b700 1 --2- 172.21.15.1:0/2952235469 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42ac003a30 0x7f42c4148d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:20.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.184+0000 7f42c1d9b700 1 -- 172.21.15.1:0/2952235469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c41e8880 con 0x7f42c40ccf50 2024-03-18T19:05:20.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.184+0000 7f42c259c700 1 --2- 172.21.15.1:0/2952235469 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42ac003a30 0x7f42c4148d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:20.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.184+0000 7f42c1d9b700 1 --2- 172.21.15.1:0/2952235469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42c40ccf50 0x7f42c41e8340 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f42b4011aa0 tx=0x7f42b4011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:20.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.184+0000 7f42b2ffd700 1 -- 172.21.15.1:0/2952235469 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b400f940 con 0x7f42c40ccf50 2024-03-18T19:05:20.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.184+0000 7f42b2ffd700 1 -- 172.21.15.1:0/2952235469 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b400faa0 con 0x7f42c40ccf50 2024-03-18T19:05:20.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.184+0000 7f42c8cda700 1 -- 172.21.15.1:0/2952235469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c41f4420 con 0x7f42c40ccf50 2024-03-18T19:05:20.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.184+0000 7f42b2ffd700 1 -- 172.21.15.1:0/2952235469 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b400fe50 con 0x7f42c40ccf50 2024-03-18T19:05:20.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.184+0000 7f42c8cda700 1 -- 172.21.15.1:0/2952235469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c40cd5e0 con 0x7f42c40ccf50 2024-03-18T19:05:20.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.185+0000 7f42c8cda700 1 -- 172.21.15.1:0/2952235469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c406ac80 con 0x7f42c40ccf50 2024-03-18T19:05:20.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.186+0000 7f42b2ffd700 1 -- 172.21.15.1:0/2952235469 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f42b4014440 con 0x7f42c40ccf50 2024-03-18T19:05:20.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.186+0000 7f42b2ffd700 1 --2- 172.21.15.1:0/2952235469 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f42a406adb0 0x7f42a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:20.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.186+0000 7f42b2ffd700 1 -- 172.21.15.1:0/2952235469 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f42b40898c0 con 0x7f42c40ccf50 2024-03-18T19:05:20.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.186+0000 7f42c259c700 1 --2- 172.21.15.1:0/2952235469 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f42a406adb0 0x7f42a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:20.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.187+0000 7f42c259c700 1 --2- 172.21.15.1:0/2952235469 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f42a406adb0 0x7f42a406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f42b8001930 tx=0x7f42b8007000 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:20.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.191+0000 7f42b2ffd700 1 -- 172.21.15.1:0/2952235469 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f42b4054da0 con 0x7f42c40ccf50 2024-03-18T19:05:20.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:20.392+0000 7f42c8cda700 1 -- 172.21.15.1:0/2952235469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f42c40676a0 con 0x7f42c40ccf50 2024-03-18T19:05:21.062 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:05:21.065 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:05:21.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.150+0000 7f42b2ffd700 1 -- 172.21.15.1:0/2952235469 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v86) v1 ==== 138+0+0 (secure 0 0 0) 0x7f42b40583c0 con 0x7f42c40ccf50 2024-03-18T19:05:21.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.153+0000 7f42c8cda700 1 -- 172.21.15.1:0/2952235469 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f42a406adb0 msgr2=0x7f42a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:21.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.153+0000 7f42c8cda700 1 --2- 172.21.15.1:0/2952235469 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f42a406adb0 0x7f42a406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f42b8001930 tx=0x7f42b8007000 comp rx=0 tx=0).stop 2024-03-18T19:05:21.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.153+0000 7f42c8cda700 1 -- 172.21.15.1:0/2952235469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42c40ccf50 msgr2=0x7f42c41e8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:21.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.153+0000 7f42c8cda700 1 --2- 172.21.15.1:0/2952235469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42c40ccf50 0x7f42c41e8340 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f42b4011aa0 tx=0x7f42b4011db0 comp rx=0 tx=0).stop 2024-03-18T19:05:21.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.153+0000 7f42c8cda700 1 -- 172.21.15.1:0/2952235469 shutdown_connections 2024-03-18T19:05:21.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.153+0000 7f42c8cda700 1 --2- 172.21.15.1:0/2952235469 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f42a406adb0 0x7f42a406d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:21.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.153+0000 7f42c8cda700 1 --2- 172.21.15.1:0/2952235469 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42c415b3f0 0x7f42c415b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:21.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.153+0000 7f42c8cda700 1 --2- 172.21.15.1:0/2952235469 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42ac003a30 0x7f42c4148d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:21.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.153+0000 7f42c8cda700 1 --2- 172.21.15.1:0/2952235469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42c40ccf50 0x7f42c41e8340 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:21.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.153+0000 7f42c8cda700 1 -- 172.21.15.1:0/2952235469 >> 172.21.15.1:0/2952235469 conn(0x7f42c40d00c0 msgr2=0x7f42c40d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:21.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.154+0000 7f42c8cda700 1 -- 172.21.15.1:0/2952235469 shutdown_connections 2024-03-18T19:05:21.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.154+0000 7f42c8cda700 1 -- 172.21.15.1:0/2952235469 wait complete. 2024-03-18T19:05:21.152 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-18T19:05:21.168 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T19:05:21.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.352+0000 7f90a5137700 1 Processor -- start 2024-03-18T19:05:21.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.352+0000 7f90a5137700 1 -- start start 2024-03-18T19:05:21.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.353+0000 7f90a5137700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f90a015d580 0x7f90a006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:21.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.353+0000 7f90a5137700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f90a006d380 0x7f90a0071da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:21.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.353+0000 7f90a5137700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a006f4f0 con 0x7f90a006d7e0 2024-03-18T19:05:21.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.353+0000 7f90a5137700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a015d960 con 0x7f90a015d580 2024-03-18T19:05:21.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.353+0000 7f90a5137700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a006ab30 con 0x7f90a006d380 2024-03-18T19:05:21.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.353+0000 7f909ed9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f90a015d580 0x7f90a006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.353+0000 7f909ed9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f90a015d580 0x7f90a006efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33880/0 (socket says 172.21.15.1:33880) 2024-03-18T19:05:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.353+0000 7f909f59e700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f90a006d7e0 0x7f90a00cf2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44336/0 (socket says 172.21.15.1:44336) 2024-03-18T19:05:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.353+0000 7f909ed9d700 1 -- 172.21.15.1:0/2712203826 learned_addr learned my addr 172.21.15.1:0/2712203826 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.353+0000 7f909e59c700 1 --2- 172.21.15.1:0/2712203826 >> v2:172.21.15.165:3300/0 conn(0x7f90a006d380 0x7f90a0071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.353+0000 7f909ed9d700 1 -- 172.21.15.1:0/2712203826 >> v2:172.21.15.165:3300/0 conn(0x7f90a006d380 msgr2=0x7f90a0071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.353+0000 7f909ed9d700 1 --2- 172.21.15.1:0/2712203826 >> v2:172.21.15.165:3300/0 conn(0x7f90a006d380 0x7f90a0071da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.353+0000 7f909ed9d700 1 -- 172.21.15.1:0/2712203826 >> v1:172.21.15.1:6789/0 conn(0x7f90a006d7e0 legacy=0x7f90a00cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.353+0000 7f909ed9d700 1 -- 172.21.15.1:0/2712203826 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a006ae10 con 0x7f90a015d580 2024-03-18T19:05:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.353+0000 7f909ed9d700 1 --2- 172.21.15.1:0/2712203826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a015d580 0x7f90a006efb0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f90900098f0 tx=0x7f9090009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5818012f3f8d0731 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.353+0000 7f909dd9b700 1 -- 172.21.15.1:0/2712203826 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909000b860 con 0x7f90a015d580 2024-03-18T19:05:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.354+0000 7f909dd9b700 1 -- 172.21.15.1:0/2712203826 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909000b9c0 con 0x7f90a015d580 2024-03-18T19:05:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.354+0000 7f909dd9b700 1 -- 172.21.15.1:0/2712203826 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909000bd50 con 0x7f90a015d580 2024-03-18T19:05:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.354+0000 7f90a5137700 1 -- 172.21.15.1:0/2712203826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a015d580 msgr2=0x7f90a006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.354+0000 7f90a5137700 1 --2- 172.21.15.1:0/2712203826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a015d580 0x7f90a006efb0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f90900098f0 tx=0x7f9090009c00 comp rx=0 tx=0).stop 2024-03-18T19:05:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.354+0000 7f90a5137700 1 -- 172.21.15.1:0/2712203826 shutdown_connections 2024-03-18T19:05:21.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.354+0000 7f90a5137700 1 --2- 172.21.15.1:0/2712203826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a015d580 0x7f90a006efb0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:21.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.354+0000 7f90a5137700 1 --2- 172.21.15.1:0/2712203826 >> v2:172.21.15.165:3300/0 conn(0x7f90a006d380 0x7f90a0071da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:21.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.354+0000 7f90a5137700 1 -- 172.21.15.1:0/2712203826 >> 172.21.15.1:0/2712203826 conn(0x7f90a00cca00 msgr2=0x7f90a00cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:21.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.354+0000 7f90a5137700 1 -- 172.21.15.1:0/2712203826 shutdown_connections 2024-03-18T19:05:21.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.354+0000 7f90a5137700 1 -- 172.21.15.1:0/2712203826 wait complete. 2024-03-18T19:05:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.356+0000 7f90a5137700 1 Processor -- start 2024-03-18T19:05:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.356+0000 7f90a5137700 1 -- start start 2024-03-18T19:05:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.357+0000 7f90a5137700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90a006d380 0x7f90a01572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.357+0000 7f90a5137700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a006d7e0 0x7f90a015fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.357+0000 7f90a5137700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90a015d580 0x7f90a0160290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.357+0000 7f90a5137700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a0158d20 con 0x7f90a006d7e0 2024-03-18T19:05:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.357+0000 7f90a5137700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a0158110 con 0x7f90a015d580 2024-03-18T19:05:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.357+0000 7f90a5137700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a01583f0 con 0x7f90a006d380 2024-03-18T19:05:21.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.357+0000 7f909e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a006d7e0 0x7f90a015fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.357+0000 7f909e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a006d7e0 0x7f90a015fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33884/0 (socket says 172.21.15.1:33884) 2024-03-18T19:05:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.357+0000 7f909e59c700 1 -- 172.21.15.1:0/3618997732 learned_addr learned my addr 172.21.15.1:0/3618997732 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.357+0000 7f909ed9d700 1 --2- 172.21.15.1:0/3618997732 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90a006d380 0x7f90a01572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.357+0000 7f909f59e700 1 --2- 172.21.15.1:0/3618997732 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90a015d580 0x7f90a0160290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.357+0000 7f909e59c700 1 -- 172.21.15.1:0/3618997732 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90a006d380 msgr2=0x7f90a01572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.357+0000 7f909e59c700 1 --2- 172.21.15.1:0/3618997732 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90a006d380 0x7f90a01572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.357+0000 7f909e59c700 1 -- 172.21.15.1:0/3618997732 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90a015d580 msgr2=0x7f90a0160290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.357+0000 7f909e59c700 1 --2- 172.21.15.1:0/3618997732 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90a015d580 0x7f90a0160290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.357+0000 7f909e59c700 1 -- 172.21.15.1:0/3618997732 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9090009550 con 0x7f90a006d7e0 2024-03-18T19:05:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.358+0000 7f909e59c700 1 --2- 172.21.15.1:0/3618997732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a006d7e0 0x7f90a015fd50 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f908800d800 tx=0x7f908800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.358+0000 7f90977fe700 1 -- 172.21.15.1:0/3618997732 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90880125f0 con 0x7f90a006d7e0 2024-03-18T19:05:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.358+0000 7f90977fe700 1 -- 172.21.15.1:0/3618997732 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9088012750 con 0x7f90a006d7e0 2024-03-18T19:05:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.358+0000 7f90a5137700 1 -- 172.21.15.1:0/3618997732 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a01586d0 con 0x7f90a006d7e0 2024-03-18T19:05:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.358+0000 7f90977fe700 1 -- 172.21.15.1:0/3618997732 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9088012b00 con 0x7f90a006d7e0 2024-03-18T19:05:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.358+0000 7f90a5137700 1 -- 172.21.15.1:0/3618997732 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a01591b0 con 0x7f90a006d7e0 2024-03-18T19:05:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.359+0000 7f90a5137700 1 -- 172.21.15.1:0/3618997732 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a0158ed0 con 0x7f90a006d7e0 2024-03-18T19:05:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.359+0000 7f90977fe700 1 -- 172.21.15.1:0/3618997732 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9088012c60 con 0x7f90a006d7e0 2024-03-18T19:05:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.360+0000 7f90977fe700 1 --2- 172.21.15.1:0/3618997732 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f908006adb0 0x7f908006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.360+0000 7f90977fe700 1 -- 172.21.15.1:0/3618997732 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f9088089340 con 0x7f90a006d7e0 2024-03-18T19:05:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.360+0000 7f909ed9d700 1 --2- 172.21.15.1:0/3618997732 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f908006adb0 0x7f908006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.360+0000 7f909ed9d700 1 --2- 172.21.15.1:0/3618997732 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f908006adb0 0x7f908006d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9090009c40 tx=0x7f9090008030 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:21.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.365+0000 7f90977fe700 1 -- 172.21.15.1:0/3618997732 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f90a0158ed0 con 0x7f90a006d7e0 2024-03-18T19:05:21.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:21.569+0000 7f90a5137700 1 -- 172.21.15.1:0/3618997732 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f90a00676a0 con 0x7f90a006d7e0 2024-03-18T19:05:22.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.153+0000 7f90977fe700 1 -- 172.21.15.1:0/3618997732 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v87) v1 ==== 170+0+0 (secure 0 0 0) 0x7f9088054540 con 0x7f90a006d7e0 2024-03-18T19:05:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.158+0000 7f90a5137700 1 -- 172.21.15.1:0/3618997732 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f908006adb0 msgr2=0x7f908006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.158+0000 7f90a5137700 1 --2- 172.21.15.1:0/3618997732 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f908006adb0 0x7f908006d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9090009c40 tx=0x7f9090008030 comp rx=0 tx=0).stop 2024-03-18T19:05:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.158+0000 7f90a5137700 1 -- 172.21.15.1:0/3618997732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a006d7e0 msgr2=0x7f90a015fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:22.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.158+0000 7f90a5137700 1 --2- 172.21.15.1:0/3618997732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a006d7e0 0x7f90a015fd50 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f908800d800 tx=0x7f908800db10 comp rx=0 tx=0).stop 2024-03-18T19:05:22.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.158+0000 7f90a5137700 1 -- 172.21.15.1:0/3618997732 shutdown_connections 2024-03-18T19:05:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.158+0000 7f90a5137700 1 --2- 172.21.15.1:0/3618997732 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f908006adb0 0x7f908006d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.158+0000 7f90a5137700 1 --2- 172.21.15.1:0/3618997732 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f90a006d380 0x7f90a01572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.158+0000 7f90a5137700 1 --2- 172.21.15.1:0/3618997732 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90a015d580 0x7f90a0160290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.158+0000 7f90a5137700 1 --2- 172.21.15.1:0/3618997732 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90a006d7e0 0x7f90a015fd50 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.158+0000 7f90a5137700 1 -- 172.21.15.1:0/3618997732 >> 172.21.15.1:0/3618997732 conn(0x7f90a00cca00 msgr2=0x7f90a00acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.160+0000 7f90a5137700 1 -- 172.21.15.1:0/3618997732 shutdown_connections 2024-03-18T19:05:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.162+0000 7f90a5137700 1 -- 172.21.15.1:0/3618997732 wait complete. 2024-03-18T19:05:22.166 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-18T19:05:22.180 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T19:05:22.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.318+0000 7f0d7d5ba700 1 Processor -- start 2024-03-18T19:05:22.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.318+0000 7f0d7d5ba700 1 -- start start 2024-03-18T19:05:22.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.319+0000 7f0d7d5ba700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0d78158e50 0x7f0d780cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:22.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.319+0000 7f0d7d5ba700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d780cbf00 0x7f0d78070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:22.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.319+0000 7f0d7d5ba700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0d78071380 0x7f0d7815ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:22.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.319+0000 7f0d7d5ba700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d7806d140 con 0x7f0d78071380 2024-03-18T19:05:22.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.319+0000 7f0d7d5ba700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d78141ac0 con 0x7f0d78158e50 2024-03-18T19:05:22.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.319+0000 7f0d7d5ba700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d780717d0 con 0x7f0d780cbf00 2024-03-18T19:05:22.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.319+0000 7f0d777fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0d78071380 0x7f0d7815ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:22.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.320+0000 7f0d777fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0d78071380 0x7f0d7815ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33892/0 (socket says 172.21.15.1:33892) 2024-03-18T19:05:22.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.320+0000 7f0d777fe700 1 -- 172.21.15.1:0/2467363551 learned_addr learned my addr 172.21.15.1:0/2467363551 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:22.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.320+0000 7f0d76ffd700 1 --2- 172.21.15.1:0/2467363551 >> v2:172.21.15.165:3300/0 conn(0x7f0d78158e50 0x7f0d780cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:22.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.320+0000 7f0d767fc700 1 --2- 172.21.15.1:0/2467363551 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d780cbf00 0x7f0d78070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:22.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.320+0000 7f0d777fe700 1 -- 172.21.15.1:0/2467363551 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d780cbf00 msgr2=0x7f0d78070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:22.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.320+0000 7f0d777fe700 1 --2- 172.21.15.1:0/2467363551 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d780cbf00 0x7f0d78070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.320+0000 7f0d777fe700 1 -- 172.21.15.1:0/2467363551 >> v2:172.21.15.165:3300/0 conn(0x7f0d78158e50 msgr2=0x7f0d780cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:22.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.320+0000 7f0d777fe700 1 --2- 172.21.15.1:0/2467363551 >> v2:172.21.15.165:3300/0 conn(0x7f0d78158e50 0x7f0d780cb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.320+0000 7f0d777fe700 1 -- 172.21.15.1:0/2467363551 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d78159230 con 0x7f0d78071380 2024-03-18T19:05:22.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.320+0000 7f0d76ffd700 1 --2- 172.21.15.1:0/2467363551 >> v2:172.21.15.165:3300/0 conn(0x7f0d78158e50 0x7f0d780cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:22.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.320+0000 7f0d767fc700 1 --2- 172.21.15.1:0/2467363551 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d780cbf00 0x7f0d78070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:22.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.320+0000 7f0d777fe700 1 --2- 172.21.15.1:0/2467363551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d78071380 0x7f0d7815ee60 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f0d780d0500 tx=0x7f0d7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd6bb6a97fc084b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:22.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.320+0000 7f0d75ffb700 1 -- 172.21.15.1:0/2467363551 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7000e070 con 0x7f0d78071380 2024-03-18T19:05:22.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.320+0000 7f0d75ffb700 1 -- 172.21.15.1:0/2467363551 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7000b9c0 con 0x7f0d78071380 2024-03-18T19:05:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.320+0000 7f0d75ffb700 1 -- 172.21.15.1:0/2467363551 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7000bd70 con 0x7f0d78071380 2024-03-18T19:05:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.321+0000 7f0d7d5ba700 1 -- 172.21.15.1:0/2467363551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d78071380 msgr2=0x7f0d7815ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.321+0000 7f0d7d5ba700 1 --2- 172.21.15.1:0/2467363551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d78071380 0x7f0d7815ee60 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f0d780d0500 tx=0x7f0d7000b330 comp rx=0 tx=0).stop 2024-03-18T19:05:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.321+0000 7f0d7d5ba700 1 -- 172.21.15.1:0/2467363551 shutdown_connections 2024-03-18T19:05:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.321+0000 7f0d7d5ba700 1 --2- 172.21.15.1:0/2467363551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d78071380 0x7f0d7815ee60 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.321+0000 7f0d7d5ba700 1 --2- 172.21.15.1:0/2467363551 >> v2:172.21.15.165:3300/0 conn(0x7f0d78158e50 0x7f0d780cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.321+0000 7f0d7d5ba700 1 --2- 172.21.15.1:0/2467363551 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d780cbf00 0x7f0d78070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.321+0000 7f0d7d5ba700 1 -- 172.21.15.1:0/2467363551 >> 172.21.15.1:0/2467363551 conn(0x7f0d7806a750 msgr2=0x7f0d7806ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.321+0000 7f0d7d5ba700 1 -- 172.21.15.1:0/2467363551 shutdown_connections 2024-03-18T19:05:22.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.321+0000 7f0d7d5ba700 1 -- 172.21.15.1:0/2467363551 wait complete. 2024-03-18T19:05:22.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.322+0000 7f0d7d5ba700 1 Processor -- start 2024-03-18T19:05:22.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.322+0000 7f0d7d5ba700 1 -- start start 2024-03-18T19:05:22.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.323+0000 7f0d7d5ba700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d78071380 0x7f0d780c46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:22.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.323+0000 7f0d7d5ba700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d780cbf00 0x7f0d780ab140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.323+0000 7f0d7d5ba700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d78158e50 0x7f0d78140d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.323+0000 7f0d7d5ba700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d780ab9b0 con 0x7f0d78158e50 2024-03-18T19:05:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.323+0000 7f0d7d5ba700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d780abc90 con 0x7f0d780cbf00 2024-03-18T19:05:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.323+0000 7f0d7d5ba700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d780abf70 con 0x7f0d78071380 2024-03-18T19:05:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.323+0000 7f0d777fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d78158e50 0x7f0d78140d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.323+0000 7f0d777fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d78158e50 0x7f0d78140d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33898/0 (socket says 172.21.15.1:33898) 2024-03-18T19:05:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.323+0000 7f0d777fe700 1 -- 172.21.15.1:0/3465990844 learned_addr learned my addr 172.21.15.1:0/3465990844 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:22.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.323+0000 7f0d76ffd700 1 --2- 172.21.15.1:0/3465990844 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d78071380 0x7f0d780c46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.323+0000 7f0d767fc700 1 --2- 172.21.15.1:0/3465990844 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d780cbf00 0x7f0d780ab140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.323+0000 7f0d777fe700 1 -- 172.21.15.1:0/3465990844 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d78071380 msgr2=0x7f0d780c46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.323+0000 7f0d777fe700 1 --2- 172.21.15.1:0/3465990844 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d78071380 0x7f0d780c46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.323+0000 7f0d777fe700 1 -- 172.21.15.1:0/3465990844 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d780cbf00 msgr2=0x7f0d780ab140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.323+0000 7f0d777fe700 1 --2- 172.21.15.1:0/3465990844 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d780cbf00 0x7f0d780ab140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.323+0000 7f0d777fe700 1 -- 172.21.15.1:0/3465990844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d70009b30 con 0x7f0d78158e50 2024-03-18T19:05:22.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.323+0000 7f0d767fc700 1 --2- 172.21.15.1:0/3465990844 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d780cbf00 0x7f0d780ab140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.324+0000 7f0d777fe700 1 --2- 172.21.15.1:0/3465990844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d78158e50 0x7f0d78140d10 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f0d78071f90 tx=0x7f0d70004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.324+0000 7f0d5f7fe700 1 -- 172.21.15.1:0/3465990844 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d700092e0 con 0x7f0d78158e50 2024-03-18T19:05:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.324+0000 7f0d5f7fe700 1 -- 172.21.15.1:0/3465990844 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d70004250 con 0x7f0d78158e50 2024-03-18T19:05:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.324+0000 7f0d7d5ba700 1 -- 172.21.15.1:0/3465990844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d780ac250 con 0x7f0d78158e50 2024-03-18T19:05:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.324+0000 7f0d5f7fe700 1 -- 172.21.15.1:0/3465990844 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d70004620 con 0x7f0d78158e50 2024-03-18T19:05:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.324+0000 7f0d7d5ba700 1 -- 172.21.15.1:0/3465990844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d780ab680 con 0x7f0d78158e50 2024-03-18T19:05:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.325+0000 7f0d7d5ba700 1 -- 172.21.15.1:0/3465990844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d780676a0 con 0x7f0d78158e50 2024-03-18T19:05:22.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.326+0000 7f0d5f7fe700 1 -- 172.21.15.1:0/3465990844 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0d70004780 con 0x7f0d78158e50 2024-03-18T19:05:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.326+0000 7f0d5f7fe700 1 --2- 172.21.15.1:0/3465990844 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0d6406adb0 0x7f0d6406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.326+0000 7f0d5f7fe700 1 -- 172.21.15.1:0/3465990844 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f0d7008ab80 con 0x7f0d78158e50 2024-03-18T19:05:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.326+0000 7f0d76ffd700 1 --2- 172.21.15.1:0/3465990844 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0d6406adb0 0x7f0d6406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:22.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.327+0000 7f0d76ffd700 1 --2- 172.21.15.1:0/3465990844 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0d6406adb0 0x7f0d6406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c0099d0 tx=0x7f0d6c008040 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:22.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.331+0000 7f0d5f7fe700 1 -- 172.21.15.1:0/3465990844 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0d70055af0 con 0x7f0d78158e50 2024-03-18T19:05:22.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.575+0000 7f0d7d5ba700 1 -- 172.21.15.1:0/3465990844 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0d7815c760 con 0x7f0d78158e50 2024-03-18T19:05:22.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.581+0000 7f0d5f7fe700 1 -- 172.21.15.1:0/3465990844 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v29) v1 ==== 173+0+0 (secure 0 0 0) 0x7f0d700290c0 con 0x7f0d78158e50 2024-03-18T19:05:22.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.585+0000 7f0d7d5ba700 1 -- 172.21.15.1:0/3465990844 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0d6406adb0 msgr2=0x7f0d6406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:22.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.585+0000 7f0d7d5ba700 1 --2- 172.21.15.1:0/3465990844 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0d6406adb0 0x7f0d6406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c0099d0 tx=0x7f0d6c008040 comp rx=0 tx=0).stop 2024-03-18T19:05:22.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.585+0000 7f0d7d5ba700 1 -- 172.21.15.1:0/3465990844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d78158e50 msgr2=0x7f0d78140d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:22.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.585+0000 7f0d7d5ba700 1 --2- 172.21.15.1:0/3465990844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d78158e50 0x7f0d78140d10 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f0d78071f90 tx=0x7f0d70004ad0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.586+0000 7f0d7d5ba700 1 -- 172.21.15.1:0/3465990844 shutdown_connections 2024-03-18T19:05:22.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.586+0000 7f0d7d5ba700 1 --2- 172.21.15.1:0/3465990844 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0d6406adb0 0x7f0d6406d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.586+0000 7f0d7d5ba700 1 --2- 172.21.15.1:0/3465990844 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d78071380 0x7f0d780c46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.586+0000 7f0d7d5ba700 1 --2- 172.21.15.1:0/3465990844 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d78158e50 0x7f0d78140d10 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.586+0000 7f0d7d5ba700 1 --2- 172.21.15.1:0/3465990844 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d780cbf00 0x7f0d780ab140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.586+0000 7f0d7d5ba700 1 -- 172.21.15.1:0/3465990844 >> 172.21.15.1:0/3465990844 conn(0x7f0d7806a750 msgr2=0x7f0d78159660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:22.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.587+0000 7f0d7d5ba700 1 -- 172.21.15.1:0/3465990844 shutdown_connections 2024-03-18T19:05:22.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.590+0000 7f0d7d5ba700 1 -- 172.21.15.1:0/3465990844 wait complete. 2024-03-18T19:05:22.592 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 9 and data pool 10 2024-03-18T19:05:22.606 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:05:22.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.735+0000 7febeb9c3700 1 Processor -- start 2024-03-18T19:05:22.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.736+0000 7febeb9c3700 1 -- start start 2024-03-18T19:05:22.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.736+0000 7febeb9c3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7febe415d580 0x7febe4072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:22.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.736+0000 7febeb9c3700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7febe406e290 0x7febe40cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:22.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.736+0000 7febeb9c3700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe406f490 con 0x7febe406d640 2024-03-18T19:05:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.736+0000 7febeb9c3700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe4071cf0 con 0x7febe406e290 2024-03-18T19:05:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.736+0000 7febeb9c3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe406f1b0 con 0x7febe415d580 2024-03-18T19:05:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.737+0000 7febe975f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7febe415d580 0x7febe4072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.737+0000 7febe8f5e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7febe406e290 0x7febe40cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.737+0000 7febe8f5e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7febe406e290 0x7febe40cd4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:53020/0 (socket says 172.21.15.1:53020) 2024-03-18T19:05:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.737+0000 7febe8f5e700 1 -- 172.21.15.1:0/2376131179 learned_addr learned my addr 172.21.15.1:0/2376131179 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.737+0000 7febe975f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7febe415d580 0x7febe4072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:41742/0 (socket says 172.21.15.1:41742) 2024-03-18T19:05:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.737+0000 7febe975f700 1 -- 172.21.15.1:0/2376131179 >> v2:172.21.15.165:3300/0 conn(0x7febe406e290 msgr2=0x7febe40cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.737+0000 7febe975f700 1 --2- 172.21.15.1:0/2376131179 >> v2:172.21.15.165:3300/0 conn(0x7febe406e290 0x7febe40cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.737+0000 7febe975f700 1 -- 172.21.15.1:0/2376131179 >> v1:172.21.15.165:6789/0 conn(0x7febe406d640 legacy=0x7febe40ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.737+0000 7febe975f700 1 -- 172.21.15.1:0/2376131179 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe40716a0 con 0x7febe415d580 2024-03-18T19:05:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.737+0000 7febe8f5e700 1 --2- 172.21.15.1:0/2376131179 >> v2:172.21.15.165:3300/0 conn(0x7febe406e290 0x7febe40cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.737+0000 7febdbfff700 1 -- 172.21.15.1:0/2376131179 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951399375 0 0) 0x7febe406f490 con 0x7febe406d640 2024-03-18T19:05:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.737+0000 7febe975f700 1 --2- 172.21.15.1:0/2376131179 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7febe415d580 0x7febe4072180 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7febd4009840 tx=0x7febd4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8dd9fcdb8da1ae6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.738+0000 7febdbfff700 1 -- 172.21.15.1:0/2376131179 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd4011830 con 0x7febe415d580 2024-03-18T19:05:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.738+0000 7febdbfff700 1 -- 172.21.15.1:0/2376131179 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd4011990 con 0x7febe415d580 2024-03-18T19:05:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.738+0000 7febdbfff700 1 -- 172.21.15.1:0/2376131179 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd4011d20 con 0x7febe415d580 2024-03-18T19:05:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.738+0000 7febeb9c3700 1 -- 172.21.15.1:0/2376131179 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7febe415d580 msgr2=0x7febe4072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.738+0000 7febeb9c3700 1 --2- 172.21.15.1:0/2376131179 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7febe415d580 0x7febe4072180 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7febd4009840 tx=0x7febd4009b50 comp rx=0 tx=0).stop 2024-03-18T19:05:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.738+0000 7febeb9c3700 1 -- 172.21.15.1:0/2376131179 shutdown_connections 2024-03-18T19:05:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.738+0000 7febeb9c3700 1 --2- 172.21.15.1:0/2376131179 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7febe415d580 0x7febe4072180 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.738+0000 7febeb9c3700 1 --2- 172.21.15.1:0/2376131179 >> v2:172.21.15.165:3300/0 conn(0x7febe406e290 0x7febe40cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.738+0000 7febeb9c3700 1 -- 172.21.15.1:0/2376131179 >> 172.21.15.1:0/2376131179 conn(0x7febe40cc990 msgr2=0x7febe40ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.738+0000 7febeb9c3700 1 -- 172.21.15.1:0/2376131179 shutdown_connections 2024-03-18T19:05:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.738+0000 7febeb9c3700 1 -- 172.21.15.1:0/2376131179 wait complete. 2024-03-18T19:05:22.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.740+0000 7febeb9c3700 1 Processor -- start 2024-03-18T19:05:22.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.740+0000 7febeb9c3700 1 -- start start 2024-03-18T19:05:22.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.740+0000 7febeb9c3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7febe406d640 0x7febe40b4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:22.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.740+0000 7febeb9c3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7febe415d580 0x7febe41f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:22.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.740+0000 7febeb9c3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe41f2cc0 0x7febe40c64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:22.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.740+0000 7febeb9c3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe415ef40 con 0x7febe41f2cc0 2024-03-18T19:05:22.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.740+0000 7febeb9c3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe40b48c0 con 0x7febe415d580 2024-03-18T19:05:22.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.740+0000 7febeb9c3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe41f43c0 con 0x7febe406d640 2024-03-18T19:05:22.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.740+0000 7febe9f60700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe41f2cc0 0x7febe40c64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:22.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.740+0000 7febe9f60700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe41f2cc0 0x7febe40c64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33910/0 (socket says 172.21.15.1:33910) 2024-03-18T19:05:22.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.741+0000 7febe9f60700 1 -- 172.21.15.1:0/3285448021 learned_addr learned my addr 172.21.15.1:0/3285448021 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:22.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.741+0000 7febe975f700 1 --2- 172.21.15.1:0/3285448021 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7febe406d640 0x7febe40b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:22.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.741+0000 7febe8f5e700 1 --2- 172.21.15.1:0/3285448021 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7febe415d580 0x7febe41f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:22.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.741+0000 7febe9f60700 1 -- 172.21.15.1:0/3285448021 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7febe406d640 msgr2=0x7febe40b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:22.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.741+0000 7febe9f60700 1 --2- 172.21.15.1:0/3285448021 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7febe406d640 0x7febe40b4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.741+0000 7febe9f60700 1 -- 172.21.15.1:0/3285448021 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7febe415d580 msgr2=0x7febe41f3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:22.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.741+0000 7febe9f60700 1 --2- 172.21.15.1:0/3285448021 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7febe415d580 0x7febe41f3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.741+0000 7febe9f60700 1 -- 172.21.15.1:0/3285448021 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe0009550 con 0x7febe41f2cc0 2024-03-18T19:05:22.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.741+0000 7febe8f5e700 1 --2- 172.21.15.1:0/3285448021 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7febe415d580 0x7febe41f3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:22.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.741+0000 7febe9f60700 1 --2- 172.21.15.1:0/3285448021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe41f2cc0 0x7febe40c64e0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7febdc001560 tx=0x7febdc00cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:22.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.741+0000 7febd9ffb700 1 -- 172.21.15.1:0/3285448021 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc003e30 con 0x7febe41f2cc0 2024-03-18T19:05:22.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.741+0000 7febd9ffb700 1 -- 172.21.15.1:0/3285448021 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febdc005070 con 0x7febe41f2cc0 2024-03-18T19:05:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.741+0000 7febeb9c3700 1 -- 172.21.15.1:0/3285448021 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd40094a0 con 0x7febe41f2cc0 2024-03-18T19:05:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.741+0000 7febd9ffb700 1 -- 172.21.15.1:0/3285448021 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc005420 con 0x7febe41f2cc0 2024-03-18T19:05:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.741+0000 7febeb9c3700 1 -- 172.21.15.1:0/3285448021 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe41f3230 con 0x7febe41f2cc0 2024-03-18T19:05:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.742+0000 7febeb9c3700 1 -- 172.21.15.1:0/3285448021 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe415f0b0 con 0x7febe41f2cc0 2024-03-18T19:05:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.743+0000 7febd9ffb700 1 -- 172.21.15.1:0/3285448021 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7febdc005580 con 0x7febe41f2cc0 2024-03-18T19:05:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.743+0000 7febd9ffb700 1 --2- 172.21.15.1:0/3285448021 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7febc806adb0 0x7febc806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.743+0000 7febd9ffb700 1 -- 172.21.15.1:0/3285448021 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7febdc088d40 con 0x7febe41f2cc0 2024-03-18T19:05:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.743+0000 7febe975f700 1 --2- 172.21.15.1:0/3285448021 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7febc806adb0 0x7febc806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.744+0000 7febe975f700 1 --2- 172.21.15.1:0/3285448021 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7febc806adb0 0x7febc806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7febd4009fd0 tx=0x7febd401c040 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.748+0000 7febd9ffb700 1 -- 172.21.15.1:0/3285448021 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7febe415f0b0 con 0x7febe41f2cc0 2024-03-18T19:05:22.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.954+0000 7febeb9c3700 1 -- 172.21.15.1:0/3285448021 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febe415f0b0 con 0x7febe41f2cc0 2024-03-18T19:05:22.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.955+0000 7febd9ffb700 1 -- 172.21.15.1:0/3285448021 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16213 (secure 0 0 0) 0x7febe415f0b0 con 0x7febe41f2cc0 2024-03-18T19:05:22.951 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:22.951 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":88,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:05:22.577164+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:05:20.423723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-18T19:05:21.600285+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:05:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.957+0000 7febeb9c3700 1 -- 172.21.15.1:0/3285448021 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7febc806adb0 msgr2=0x7febc806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.957+0000 7febeb9c3700 1 --2- 172.21.15.1:0/3285448021 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7febc806adb0 0x7febc806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7febd4009fd0 tx=0x7febd401c040 comp rx=0 tx=0).stop 2024-03-18T19:05:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.957+0000 7febeb9c3700 1 -- 172.21.15.1:0/3285448021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe41f2cc0 msgr2=0x7febe40c64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.957+0000 7febeb9c3700 1 --2- 172.21.15.1:0/3285448021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe41f2cc0 0x7febe40c64e0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7febdc001560 tx=0x7febdc00cb50 comp rx=0 tx=0).stop 2024-03-18T19:05:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.957+0000 7febeb9c3700 1 -- 172.21.15.1:0/3285448021 shutdown_connections 2024-03-18T19:05:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.957+0000 7febeb9c3700 1 --2- 172.21.15.1:0/3285448021 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7febc806adb0 0x7febc806d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.957+0000 7febeb9c3700 1 --2- 172.21.15.1:0/3285448021 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7febe406d640 0x7febe40b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.957+0000 7febeb9c3700 1 --2- 172.21.15.1:0/3285448021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febe41f2cc0 0x7febe40c64e0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.957+0000 7febeb9c3700 1 --2- 172.21.15.1:0/3285448021 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7febe415d580 0x7febe41f3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.957+0000 7febeb9c3700 1 -- 172.21.15.1:0/3285448021 >> 172.21.15.1:0/3285448021 conn(0x7febe40cc990 msgr2=0x7febe40ccd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.957+0000 7febeb9c3700 1 -- 172.21.15.1:0/3285448021 shutdown_connections 2024-03-18T19:05:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:22.957+0000 7febeb9c3700 1 -- 172.21.15.1:0/3285448021 wait complete. 2024-03-18T19:05:22.966 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:05:23.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.094+0000 7f9859f72700 1 Processor -- start 2024-03-18T19:05:23.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.095+0000 7f9859f72700 1 -- start start 2024-03-18T19:05:23.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.095+0000 7f9859f72700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f985415a2e0 0x7f98540ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:23.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.095+0000 7f9859f72700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f98540cc170 0x7f985406ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:23.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.095+0000 7f9859f72700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985406e270 con 0x7f98540ae600 2024-03-18T19:05:23.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.095+0000 7f9859f72700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98540ae050 con 0x7f98540cc170 2024-03-18T19:05:23.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.095+0000 7f9859f72700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98540add70 con 0x7f985415a2e0 2024-03-18T19:05:23.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.096+0000 7f9852ffd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f985415a2e0 0x7f98540ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:23.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.096+0000 7f9853fff700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f98540cc170 0x7f985406ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:23.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.096+0000 7f9852ffd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f985415a2e0 0x7f98540ad790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:41764/0 (socket says 172.21.15.1:41764) 2024-03-18T19:05:23.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.096+0000 7f9852ffd700 1 -- 172.21.15.1:0/941045895 learned_addr learned my addr 172.21.15.1:0/941045895 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:23.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.096+0000 7f9852ffd700 1 -- 172.21.15.1:0/941045895 >> v2:172.21.15.165:3300/0 conn(0x7f98540cc170 msgr2=0x7f985406ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:23.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.096+0000 7f9852ffd700 1 --2- 172.21.15.1:0/941045895 >> v2:172.21.15.165:3300/0 conn(0x7f98540cc170 0x7f985406ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:23.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.096+0000 7f9852ffd700 1 -- 172.21.15.1:0/941045895 >> v1:172.21.15.165:6789/0 conn(0x7f98540ae600 legacy=0x7f98540ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:23.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.096+0000 7f9852ffd700 1 -- 172.21.15.1:0/941045895 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985406e4a0 con 0x7f985415a2e0 2024-03-18T19:05:23.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.096+0000 7f9853fff700 1 --2- 172.21.15.1:0/941045895 >> v2:172.21.15.165:3300/0 conn(0x7f98540cc170 0x7f985406ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:23.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.096+0000 7f9852ffd700 1 --2- 172.21.15.1:0/941045895 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f985415a2e0 0x7f98540ad790 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f98440098f0 tx=0x7f9844009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f7f3e4d17aac166 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:23.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.096+0000 7f98527fc700 1 -- 172.21.15.1:0/941045895 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9844011830 con 0x7f985415a2e0 2024-03-18T19:05:23.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.097+0000 7f98527fc700 1 -- 172.21.15.1:0/941045895 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9844011990 con 0x7f985415a2e0 2024-03-18T19:05:23.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.097+0000 7f98527fc700 1 -- 172.21.15.1:0/941045895 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9844011d20 con 0x7f985415a2e0 2024-03-18T19:05:23.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.097+0000 7f9859f72700 1 -- 172.21.15.1:0/941045895 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f985415a2e0 msgr2=0x7f98540ad790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:23.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.097+0000 7f9859f72700 1 --2- 172.21.15.1:0/941045895 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f985415a2e0 0x7f98540ad790 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f98440098f0 tx=0x7f9844009c00 comp rx=0 tx=0).stop 2024-03-18T19:05:23.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.097+0000 7f9859f72700 1 -- 172.21.15.1:0/941045895 shutdown_connections 2024-03-18T19:05:23.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.097+0000 7f9859f72700 1 --2- 172.21.15.1:0/941045895 >> v2:172.21.15.165:3300/0 conn(0x7f98540cc170 0x7f985406ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:23.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.097+0000 7f9859f72700 1 --2- 172.21.15.1:0/941045895 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f985415a2e0 0x7f98540ad790 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:23.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.097+0000 7f9859f72700 1 -- 172.21.15.1:0/941045895 >> 172.21.15.1:0/941045895 conn(0x7f985406b860 msgr2=0x7f985406bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:23.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.097+0000 7f9859f72700 1 -- 172.21.15.1:0/941045895 shutdown_connections 2024-03-18T19:05:23.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.097+0000 7f9859f72700 1 -- 172.21.15.1:0/941045895 wait complete. 2024-03-18T19:05:23.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.098+0000 7f9859f72700 1 Processor -- start 2024-03-18T19:05:23.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.098+0000 7f9859f72700 1 -- start start 2024-03-18T19:05:23.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.099+0000 7f9859f72700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98540ae600 0x7f985409d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:23.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.099+0000 7f9859f72700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98540cc170 0x7f985415e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:23.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.099+0000 7f9859f72700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f985415a2e0 0x7f98541f4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:23.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.099+0000 7f9859f72700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9854099fb0 con 0x7f985415a2e0 2024-03-18T19:05:23.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.099+0000 7f9859f72700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985409a290 con 0x7f98540cc170 2024-03-18T19:05:23.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.099+0000 7f9859f72700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985409a570 con 0x7f98540ae600 2024-03-18T19:05:23.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.099+0000 7f9853fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f985415a2e0 0x7f98541f4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:23.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.099+0000 7f9853fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f985415a2e0 0x7f98541f4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33918/0 (socket says 172.21.15.1:33918) 2024-03-18T19:05:23.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.099+0000 7f9853fff700 1 -- 172.21.15.1:0/291489106 learned_addr learned my addr 172.21.15.1:0/291489106 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:23.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.099+0000 7f98537fe700 1 --2- 172.21.15.1:0/291489106 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98540ae600 0x7f985409d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:23.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.100+0000 7f9852ffd700 1 --2- 172.21.15.1:0/291489106 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98540cc170 0x7f985415e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:23.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.100+0000 7f9853fff700 1 -- 172.21.15.1:0/291489106 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98540ae600 msgr2=0x7f985409d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:23.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.100+0000 7f9853fff700 1 --2- 172.21.15.1:0/291489106 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98540ae600 0x7f985409d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:23.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.100+0000 7f9853fff700 1 -- 172.21.15.1:0/291489106 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98540cc170 msgr2=0x7f985415e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:23.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.100+0000 7f9853fff700 1 --2- 172.21.15.1:0/291489106 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98540cc170 0x7f985415e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:23.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.100+0000 7f9853fff700 1 -- 172.21.15.1:0/291489106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984c009a70 con 0x7f985415a2e0 2024-03-18T19:05:23.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.100+0000 7f9853fff700 1 --2- 172.21.15.1:0/291489106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f985415a2e0 0x7f98541f4ac0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f984c00ef90 tx=0x7f984c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:23.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.100+0000 7f984bfff700 1 -- 172.21.15.1:0/291489106 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984c009d70 con 0x7f985415a2e0 2024-03-18T19:05:23.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.100+0000 7f984bfff700 1 -- 172.21.15.1:0/291489106 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984c00f040 con 0x7f985415a2e0 2024-03-18T19:05:23.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.100+0000 7f984bfff700 1 -- 172.21.15.1:0/291489106 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984c014480 con 0x7f985415a2e0 2024-03-18T19:05:23.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.100+0000 7f9859f72700 1 -- 172.21.15.1:0/291489106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9844009550 con 0x7f985415a2e0 2024-03-18T19:05:23.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.100+0000 7f9859f72700 1 -- 172.21.15.1:0/291489106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f985409ab30 con 0x7f985415a2e0 2024-03-18T19:05:23.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.101+0000 7f9859f72700 1 -- 172.21.15.1:0/291489106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f985409ad50 con 0x7f985415a2e0 2024-03-18T19:05:23.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.102+0000 7f984bfff700 1 -- 172.21.15.1:0/291489106 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f984c0145e0 con 0x7f985415a2e0 2024-03-18T19:05:23.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.102+0000 7f984bfff700 1 --2- 172.21.15.1:0/291489106 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f983406adb0 0x7f983406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:23.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.102+0000 7f984bfff700 1 -- 172.21.15.1:0/291489106 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f984c08a3a0 con 0x7f985415a2e0 2024-03-18T19:05:23.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.102+0000 7f98537fe700 1 --2- 172.21.15.1:0/291489106 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f983406adb0 0x7f983406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:23.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.102+0000 7f98537fe700 1 --2- 172.21.15.1:0/291489106 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f983406adb0 0x7f983406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f983c000f60 tx=0x7f983c009040 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:23.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.106+0000 7f984bfff700 1 -- 172.21.15.1:0/291489106 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f984c0552b0 con 0x7f985415a2e0 2024-03-18T19:05:23.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.311+0000 7f9859f72700 1 -- 172.21.15.1:0/291489106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98540af4a0 con 0x7f985415a2e0 2024-03-18T19:05:23.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.312+0000 7f984bfff700 1 -- 172.21.15.1:0/291489106 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16213 (secure 0 0 0) 0x7f984c0588d0 con 0x7f985415a2e0 2024-03-18T19:05:23.310 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:23.310 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":88,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:05:22.577164+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:05:20.423723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-18T19:05:21.600285+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:05:23.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.314+0000 7f9859f72700 1 -- 172.21.15.1:0/291489106 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f983406adb0 msgr2=0x7f983406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:23.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.314+0000 7f9859f72700 1 --2- 172.21.15.1:0/291489106 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f983406adb0 0x7f983406d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f983c000f60 tx=0x7f983c009040 comp rx=0 tx=0).stop 2024-03-18T19:05:23.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.314+0000 7f9859f72700 1 -- 172.21.15.1:0/291489106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f985415a2e0 msgr2=0x7f98541f4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:23.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.314+0000 7f9859f72700 1 --2- 172.21.15.1:0/291489106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f985415a2e0 0x7f98541f4ac0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f984c00ef90 tx=0x7f984c00c3a0 comp rx=0 tx=0).stop 2024-03-18T19:05:23.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.315+0000 7f9859f72700 1 -- 172.21.15.1:0/291489106 shutdown_connections 2024-03-18T19:05:23.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.315+0000 7f9859f72700 1 --2- 172.21.15.1:0/291489106 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f983406adb0 0x7f983406d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:23.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.315+0000 7f9859f72700 1 --2- 172.21.15.1:0/291489106 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98540ae600 0x7f985409d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:23.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.315+0000 7f9859f72700 1 --2- 172.21.15.1:0/291489106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f985415a2e0 0x7f98541f4ac0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:23.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.315+0000 7f9859f72700 1 --2- 172.21.15.1:0/291489106 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98540cc170 0x7f985415e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:23.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.315+0000 7f9859f72700 1 -- 172.21.15.1:0/291489106 >> 172.21.15.1:0/291489106 conn(0x7f985406b860 msgr2=0x7f9854155bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:23.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.315+0000 7f9859f72700 1 -- 172.21.15.1:0/291489106 shutdown_connections 2024-03-18T19:05:23.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.315+0000 7f9859f72700 1 -- 172.21.15.1:0/291489106 wait complete. 2024-03-18T19:05:23.323 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T19:05:23.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.455+0000 7f6b0016f700 1 Processor -- start 2024-03-18T19:05:23.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.456+0000 7f6b0016f700 1 -- start start 2024-03-18T19:05:23.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.456+0000 7f6b0016f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6af815b480 0x7f6af80cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:23.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.456+0000 7f6b0016f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6af80cc7b0 0x7f6af80cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:23.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.456+0000 7f6b0016f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6af80c4c50 con 0x7f6af80cfc00 2024-03-18T19:05:23.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.456+0000 7f6b0016f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6af80d00b0 con 0x7f6af80cc7b0 2024-03-18T19:05:23.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.456+0000 7f6b0016f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6af80c5210 con 0x7f6af815b480 2024-03-18T19:05:23.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.457+0000 7f6afd70a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6af80cc7b0 0x7f6af80cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:23.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.457+0000 7f6afd70a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6af80cc7b0 0x7f6af80cf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33930/0 (socket says 172.21.15.1:33930) 2024-03-18T19:05:23.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.457+0000 7f6afd70a700 1 -- 172.21.15.1:0/3263893825 learned_addr learned my addr 172.21.15.1:0/3263893825 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:23.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.457+0000 7f6afdf0b700 1 --2- 172.21.15.1:0/3263893825 >> v2:172.21.15.165:3300/0 conn(0x7f6af815b480 0x7f6af80cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:23.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.457+0000 7f6afd70a700 1 -- 172.21.15.1:0/3263893825 >> v2:172.21.15.165:3300/0 conn(0x7f6af815b480 msgr2=0x7f6af80cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:23.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.457+0000 7f6afd70a700 1 --2- 172.21.15.1:0/3263893825 >> v2:172.21.15.165:3300/0 conn(0x7f6af815b480 0x7f6af80cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:23.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.457+0000 7f6afd70a700 1 -- 172.21.15.1:0/3263893825 >> v1:172.21.15.165:6789/0 conn(0x7f6af80cfc00 legacy=0x7f6af8099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:23.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.457+0000 7f6afd70a700 1 -- 172.21.15.1:0/3263893825 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af80cccb0 con 0x7f6af80cc7b0 2024-03-18T19:05:23.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.457+0000 7f6afd70a700 1 --2- 172.21.15.1:0/3263893825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af80cc7b0 0x7f6af80cf6c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f6af80d07c0 tx=0x7f6aec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec65a3be582f9530 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:23.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.457+0000 7f6afcf09700 1 -- 172.21.15.1:0/3263893825 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec011830 con 0x7f6af80cc7b0 2024-03-18T19:05:23.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.457+0000 7f6afcf09700 1 -- 172.21.15.1:0/3263893825 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aec011990 con 0x7f6af80cc7b0 2024-03-18T19:05:23.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.457+0000 7f6afcf09700 1 -- 172.21.15.1:0/3263893825 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec011d20 con 0x7f6af80cc7b0 2024-03-18T19:05:23.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.458+0000 7f6b0016f700 1 -- 172.21.15.1:0/3263893825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af80cc7b0 msgr2=0x7f6af80cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:23.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.458+0000 7f6b0016f700 1 --2- 172.21.15.1:0/3263893825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af80cc7b0 0x7f6af80cf6c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f6af80d07c0 tx=0x7f6aec009bd0 comp rx=0 tx=0).stop 2024-03-18T19:05:23.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.458+0000 7f6b0016f700 1 -- 172.21.15.1:0/3263893825 shutdown_connections 2024-03-18T19:05:23.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.458+0000 7f6b0016f700 1 --2- 172.21.15.1:0/3263893825 >> v2:172.21.15.165:3300/0 conn(0x7f6af815b480 0x7f6af80cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:23.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.458+0000 7f6b0016f700 1 --2- 172.21.15.1:0/3263893825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af80cc7b0 0x7f6af80cf6c0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:23.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.458+0000 7f6b0016f700 1 -- 172.21.15.1:0/3263893825 >> 172.21.15.1:0/3263893825 conn(0x7f6af80b2bf0 msgr2=0x7f6af80b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:23.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.458+0000 7f6b0016f700 1 -- 172.21.15.1:0/3263893825 shutdown_connections 2024-03-18T19:05:23.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.458+0000 7f6b0016f700 1 -- 172.21.15.1:0/3263893825 wait complete. 2024-03-18T19:05:23.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.459+0000 7f6b0016f700 1 Processor -- start 2024-03-18T19:05:23.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.459+0000 7f6b0016f700 1 -- start start 2024-03-18T19:05:23.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.460+0000 7f6b0016f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af80cc7b0 0x7f6af815e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:23.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.460+0000 7f6b0016f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6af80cfc00 0x7f6af81f0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:23.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.460+0000 7f6b0016f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6af815b480 0x7f6af81f1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:23.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.460+0000 7f6b0016f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af815d860 con 0x7f6af80cc7b0 2024-03-18T19:05:23.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.460+0000 7f6b0016f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af81ee610 con 0x7f6af815b480 2024-03-18T19:05:23.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.460+0000 7f6b0016f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af815cce0 con 0x7f6af80cfc00 2024-03-18T19:05:23.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.460+0000 7f6afdf0b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af80cc7b0 0x7f6af815e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:23.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.460+0000 7f6afdf0b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af80cc7b0 0x7f6af815e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33940/0 (socket says 172.21.15.1:33940) 2024-03-18T19:05:23.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.460+0000 7f6afdf0b700 1 -- 172.21.15.1:0/1489175058 learned_addr learned my addr 172.21.15.1:0/1489175058 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:23.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.460+0000 7f6afd70a700 1 --2- 172.21.15.1:0/1489175058 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6af80cfc00 0x7f6af81f0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:23.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.460+0000 7f6afe70c700 1 --2- 172.21.15.1:0/1489175058 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6af815b480 0x7f6af81f1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:23.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.460+0000 7f6afdf0b700 1 -- 172.21.15.1:0/1489175058 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6af80cfc00 msgr2=0x7f6af81f0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:23.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.460+0000 7f6afdf0b700 1 --2- 172.21.15.1:0/1489175058 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6af80cfc00 0x7f6af81f0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:23.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.460+0000 7f6afdf0b700 1 -- 172.21.15.1:0/1489175058 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6af815b480 msgr2=0x7f6af81f1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:23.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.460+0000 7f6afdf0b700 1 --2- 172.21.15.1:0/1489175058 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6af815b480 0x7f6af81f1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:23.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.460+0000 7f6afdf0b700 1 -- 172.21.15.1:0/1489175058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec009550 con 0x7f6af80cc7b0 2024-03-18T19:05:23.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.460+0000 7f6afe70c700 1 --2- 172.21.15.1:0/1489175058 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6af815b480 0x7f6af81f1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:23.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.461+0000 7f6afdf0b700 1 --2- 172.21.15.1:0/1489175058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af80cc7b0 0x7f6af815e780 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6af400e9d0 tx=0x7f6af400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:23.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.461+0000 7f6aea7fc700 1 -- 172.21.15.1:0/1489175058 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af4009730 con 0x7f6af80cc7b0 2024-03-18T19:05:23.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.461+0000 7f6aea7fc700 1 -- 172.21.15.1:0/1489175058 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af400c850 con 0x7f6af80cc7b0 2024-03-18T19:05:23.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.461+0000 7f6aea7fc700 1 -- 172.21.15.1:0/1489175058 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af400cbe0 con 0x7f6af80cc7b0 2024-03-18T19:05:23.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.461+0000 7f6b0016f700 1 -- 172.21.15.1:0/1489175058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af815cfc0 con 0x7f6af80cc7b0 2024-03-18T19:05:23.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.461+0000 7f6b0016f700 1 -- 172.21.15.1:0/1489175058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af815d580 con 0x7f6af80cc7b0 2024-03-18T19:05:23.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.462+0000 7f6b0016f700 1 -- 172.21.15.1:0/1489175058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af80676a0 con 0x7f6af80cc7b0 2024-03-18T19:05:23.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.462+0000 7f6aea7fc700 1 -- 172.21.15.1:0/1489175058 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6af400cd40 con 0x7f6af80cc7b0 2024-03-18T19:05:23.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.463+0000 7f6aea7fc700 1 --2- 172.21.15.1:0/1489175058 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6adc06adb0 0x7f6adc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:23.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.463+0000 7f6aea7fc700 1 -- 172.21.15.1:0/1489175058 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6af4089740 con 0x7f6af80cc7b0 2024-03-18T19:05:23.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.463+0000 7f6afd70a700 1 --2- 172.21.15.1:0/1489175058 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6adc06adb0 0x7f6adc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:23.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.463+0000 7f6afd70a700 1 --2- 172.21.15.1:0/1489175058 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6adc06adb0 0x7f6adc06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6aec000c00 tx=0x7f6aec008030 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:23.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.467+0000 7f6aea7fc700 1 -- 172.21.15.1:0/1489175058 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6af4054630 con 0x7f6af80cc7b0 2024-03-18T19:05:23.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:23.720+0000 7f6b0016f700 1 -- 172.21.15.1:0/1489175058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6af80c75c0 con 0x7f6af80cc7b0 2024-03-18T19:05:24.052 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-03-18T19:05:24.052 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-18T19:05:24.052 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-03-18T19:05:24.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.595+0000 7f6aea7fc700 1 -- 172.21.15.1:0/1489175058 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v32) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6af4057c50 con 0x7f6af80cc7b0 2024-03-18T19:05:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.598+0000 7f6b0016f700 1 -- 172.21.15.1:0/1489175058 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6adc06adb0 msgr2=0x7f6adc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.598+0000 7f6b0016f700 1 --2- 172.21.15.1:0/1489175058 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6adc06adb0 0x7f6adc06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6aec000c00 tx=0x7f6aec008030 comp rx=0 tx=0).stop 2024-03-18T19:05:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.598+0000 7f6b0016f700 1 -- 172.21.15.1:0/1489175058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af80cc7b0 msgr2=0x7f6af815e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.598+0000 7f6b0016f700 1 --2- 172.21.15.1:0/1489175058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af80cc7b0 0x7f6af815e780 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6af400e9d0 tx=0x7f6af400ece0 comp rx=0 tx=0).stop 2024-03-18T19:05:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.598+0000 7f6b0016f700 1 -- 172.21.15.1:0/1489175058 shutdown_connections 2024-03-18T19:05:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.598+0000 7f6b0016f700 1 --2- 172.21.15.1:0/1489175058 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6adc06adb0 0x7f6adc06d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.598+0000 7f6b0016f700 1 --2- 172.21.15.1:0/1489175058 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6af815b480 0x7f6af81f1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.598+0000 7f6b0016f700 1 --2- 172.21.15.1:0/1489175058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6af80cc7b0 0x7f6af815e780 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.598+0000 7f6b0016f700 1 --2- 172.21.15.1:0/1489175058 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6af80cfc00 0x7f6af81f0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.598+0000 7f6b0016f700 1 -- 172.21.15.1:0/1489175058 >> 172.21.15.1:0/1489175058 conn(0x7f6af80b2bf0 msgr2=0x7f6af80c55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.599+0000 7f6b0016f700 1 -- 172.21.15.1:0/1489175058 shutdown_connections 2024-03-18T19:05:24.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.599+0000 7f6b0016f700 1 -- 172.21.15.1:0/1489175058 wait complete. 2024-03-18T19:05:24.609 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-18T19:05:24.610 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T19:05:24.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.749+0000 7fcc9bb0c700 1 Processor -- start 2024-03-18T19:05:24.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.749+0000 7fcc9bb0c700 1 -- start start 2024-03-18T19:05:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.750+0000 7fcc9bb0c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcc9415eba0 0x7fcc940cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.750+0000 7fcc9bb0c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc940b0500 0x7fcc940cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.750+0000 7fcc9bb0c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcc940cfdf0 0x7fcc94099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.750+0000 7fcc9bb0c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc9415ef80 con 0x7fcc940cfdf0 2024-03-18T19:05:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.750+0000 7fcc9bb0c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc940cc9a0 con 0x7fcc9415eba0 2024-03-18T19:05:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.750+0000 7fcc9bb0c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc940ccc80 con 0x7fcc940b0500 2024-03-18T19:05:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.750+0000 7fcc9a0a9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcc940cfdf0 0x7fcc94099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.750+0000 7fcc9a0a9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcc940cfdf0 0x7fcc94099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33618/0 (socket says 172.21.15.1:33618) 2024-03-18T19:05:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.750+0000 7fcc9a0a9700 1 -- 172.21.15.1:0/1086912828 learned_addr learned my addr 172.21.15.1:0/1086912828 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.750+0000 7fcc990a7700 1 --2- 172.21.15.1:0/1086912828 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc940b0500 0x7fcc940cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.751+0000 7fcc998a8700 1 --2- 172.21.15.1:0/1086912828 >> v2:172.21.15.165:3300/0 conn(0x7fcc9415eba0 0x7fcc940cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.751+0000 7fcc9a0a9700 1 -- 172.21.15.1:0/1086912828 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc940b0500 msgr2=0x7fcc940cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.751+0000 7fcc9a0a9700 1 --2- 172.21.15.1:0/1086912828 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc940b0500 0x7fcc940cf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.751+0000 7fcc9a0a9700 1 -- 172.21.15.1:0/1086912828 >> v2:172.21.15.165:3300/0 conn(0x7fcc9415eba0 msgr2=0x7fcc940cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.751+0000 7fcc9a0a9700 1 --2- 172.21.15.1:0/1086912828 >> v2:172.21.15.165:3300/0 conn(0x7fcc9415eba0 0x7fcc940cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.751+0000 7fcc9a0a9700 1 -- 172.21.15.1:0/1086912828 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc940ccf60 con 0x7fcc940cfdf0 2024-03-18T19:05:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.751+0000 7fcc998a8700 1 --2- 172.21.15.1:0/1086912828 >> v2:172.21.15.165:3300/0 conn(0x7fcc9415eba0 0x7fcc940cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.751+0000 7fcc9a0a9700 1 --2- 172.21.15.1:0/1086912828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc940cfdf0 0x7fcc94099a40 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fcc94072fb0 tx=0x7fcc8c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e59422192a4f506 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.751+0000 7fcc988a6700 1 -- 172.21.15.1:0/1086912828 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c00e070 con 0x7fcc940cfdf0 2024-03-18T19:05:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.751+0000 7fcc988a6700 1 -- 172.21.15.1:0/1086912828 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc8c00b9c0 con 0x7fcc940cfdf0 2024-03-18T19:05:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.751+0000 7fcc988a6700 1 -- 172.21.15.1:0/1086912828 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c00bd70 con 0x7fcc940cfdf0 2024-03-18T19:05:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.751+0000 7fcc9bb0c700 1 -- 172.21.15.1:0/1086912828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc940cfdf0 msgr2=0x7fcc94099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.751+0000 7fcc9bb0c700 1 --2- 172.21.15.1:0/1086912828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc940cfdf0 0x7fcc94099a40 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fcc94072fb0 tx=0x7fcc8c00b330 comp rx=0 tx=0).stop 2024-03-18T19:05:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.752+0000 7fcc9bb0c700 1 -- 172.21.15.1:0/1086912828 shutdown_connections 2024-03-18T19:05:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.752+0000 7fcc9bb0c700 1 --2- 172.21.15.1:0/1086912828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc940cfdf0 0x7fcc94099a40 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.752+0000 7fcc9bb0c700 1 --2- 172.21.15.1:0/1086912828 >> v2:172.21.15.165:3300/0 conn(0x7fcc9415eba0 0x7fcc940cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.752+0000 7fcc9bb0c700 1 --2- 172.21.15.1:0/1086912828 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc940b0500 0x7fcc940cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.752+0000 7fcc9bb0c700 1 -- 172.21.15.1:0/1086912828 >> 172.21.15.1:0/1086912828 conn(0x7fcc940b2bd0 msgr2=0x7fcc940b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.752+0000 7fcc9bb0c700 1 -- 172.21.15.1:0/1086912828 shutdown_connections 2024-03-18T19:05:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.752+0000 7fcc9bb0c700 1 -- 172.21.15.1:0/1086912828 wait complete. 2024-03-18T19:05:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.753+0000 7fcc9bb0c700 1 Processor -- start 2024-03-18T19:05:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.753+0000 7fcc9bb0c700 1 -- start start 2024-03-18T19:05:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.754+0000 7fcc9bb0c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc940b0500 0x7fcc9414dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.754+0000 7fcc9bb0c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc940cfdf0 0x7fcc940aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.754+0000 7fcc9bb0c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc9415eba0 0x7fcc940af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.754+0000 7fcc9bb0c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc940c58c0 con 0x7fcc940b0500 2024-03-18T19:05:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.754+0000 7fcc9bb0c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc940c9360 con 0x7fcc9415eba0 2024-03-18T19:05:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.754+0000 7fcc9bb0c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc940c9640 con 0x7fcc940cfdf0 2024-03-18T19:05:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.754+0000 7fcc998a8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc940b0500 0x7fcc9414dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.754+0000 7fcc998a8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc940b0500 0x7fcc9414dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33632/0 (socket says 172.21.15.1:33632) 2024-03-18T19:05:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.754+0000 7fcc998a8700 1 -- 172.21.15.1:0/91915553 learned_addr learned my addr 172.21.15.1:0/91915553 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.755+0000 7fcc998a8700 1 -- 172.21.15.1:0/91915553 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc940cfdf0 msgr2=0x7fcc940aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.755+0000 7fcc9a0a9700 1 --2- 172.21.15.1:0/91915553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc9415eba0 0x7fcc940af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.755+0000 7fcc990a7700 1 --2- 172.21.15.1:0/91915553 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc940cfdf0 0x7fcc940aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.755+0000 7fcc998a8700 1 --2- 172.21.15.1:0/91915553 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc940cfdf0 0x7fcc940aede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.755+0000 7fcc998a8700 1 -- 172.21.15.1:0/91915553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc9415eba0 msgr2=0x7fcc940af320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.755+0000 7fcc998a8700 1 --2- 172.21.15.1:0/91915553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc9415eba0 0x7fcc940af320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.755+0000 7fcc998a8700 1 -- 172.21.15.1:0/91915553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8c009b30 con 0x7fcc940b0500 2024-03-18T19:05:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.755+0000 7fcc9a0a9700 1 --2- 172.21.15.1:0/91915553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc9415eba0 0x7fcc940af320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.755+0000 7fcc998a8700 1 --2- 172.21.15.1:0/91915553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc940b0500 0x7fcc9414dbd0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fcc8400e810 tx=0x7fcc8400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.755+0000 7fcc8a7fc700 1 -- 172.21.15.1:0/91915553 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8400c020 con 0x7fcc940b0500 2024-03-18T19:05:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.755+0000 7fcc8a7fc700 1 -- 172.21.15.1:0/91915553 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc84004250 con 0x7fcc940b0500 2024-03-18T19:05:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.755+0000 7fcc9bb0c700 1 -- 172.21.15.1:0/91915553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc940c9920 con 0x7fcc940b0500 2024-03-18T19:05:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.755+0000 7fcc8a7fc700 1 -- 172.21.15.1:0/91915553 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc840036a0 con 0x7fcc940b0500 2024-03-18T19:05:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.755+0000 7fcc9bb0c700 1 -- 172.21.15.1:0/91915553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc9414e110 con 0x7fcc940b0500 2024-03-18T19:05:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.756+0000 7fcc9bb0c700 1 -- 172.21.15.1:0/91915553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc9414e2a0 con 0x7fcc940b0500 2024-03-18T19:05:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.756+0000 7fcc8a7fc700 1 -- 172.21.15.1:0/91915553 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcc84003800 con 0x7fcc940b0500 2024-03-18T19:05:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.757+0000 7fcc8a7fc700 1 --2- 172.21.15.1:0/91915553 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcc7806adb0 0x7fcc7806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.757+0000 7fcc8a7fc700 1 -- 172.21.15.1:0/91915553 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fcc84089490 con 0x7fcc940b0500 2024-03-18T19:05:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.757+0000 7fcc990a7700 1 --2- 172.21.15.1:0/91915553 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcc7806adb0 0x7fcc7806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.757+0000 7fcc990a7700 1 --2- 172.21.15.1:0/91915553 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcc7806adb0 0x7fcc7806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcc94159d90 tx=0x7fcc90008040 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:24.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:24.761+0000 7fcc8a7fc700 1 -- 172.21.15.1:0/91915553 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcc84054310 con 0x7fcc940b0500 2024-03-18T19:05:25.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.009+0000 7fcc9bb0c700 1 -- 172.21.15.1:0/91915553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fcc940c9f70 con 0x7fcc940b0500 2024-03-18T19:05:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.603+0000 7fcc8a7fc700 1 -- 172.21.15.1:0/91915553 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v33) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcc84057930 con 0x7fcc940b0500 2024-03-18T19:05:25.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.605+0000 7fcc9bb0c700 1 -- 172.21.15.1:0/91915553 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcc7806adb0 msgr2=0x7fcc7806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.605+0000 7fcc9bb0c700 1 --2- 172.21.15.1:0/91915553 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcc7806adb0 0x7fcc7806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcc94159d90 tx=0x7fcc90008040 comp rx=0 tx=0).stop 2024-03-18T19:05:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.605+0000 7fcc9bb0c700 1 -- 172.21.15.1:0/91915553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc940b0500 msgr2=0x7fcc9414dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.605+0000 7fcc9bb0c700 1 --2- 172.21.15.1:0/91915553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc940b0500 0x7fcc9414dbd0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fcc8400e810 tx=0x7fcc8400ebd0 comp rx=0 tx=0).stop 2024-03-18T19:05:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.605+0000 7fcc9bb0c700 1 -- 172.21.15.1:0/91915553 shutdown_connections 2024-03-18T19:05:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.605+0000 7fcc9bb0c700 1 --2- 172.21.15.1:0/91915553 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcc7806adb0 0x7fcc7806d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.605+0000 7fcc9bb0c700 1 --2- 172.21.15.1:0/91915553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc9415eba0 0x7fcc940af320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.605+0000 7fcc9bb0c700 1 --2- 172.21.15.1:0/91915553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc940b0500 0x7fcc9414dbd0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.605+0000 7fcc9bb0c700 1 --2- 172.21.15.1:0/91915553 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc940cfdf0 0x7fcc940aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.605+0000 7fcc9bb0c700 1 -- 172.21.15.1:0/91915553 >> 172.21.15.1:0/91915553 conn(0x7fcc940b2bd0 msgr2=0x7fcc9415b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.606+0000 7fcc9bb0c700 1 -- 172.21.15.1:0/91915553 shutdown_connections 2024-03-18T19:05:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.606+0000 7fcc9bb0c700 1 -- 172.21.15.1:0/91915553 wait complete. 2024-03-18T19:05:25.616 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T19:05:25.616 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:05:25.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.751+0000 7f0153a7e700 1 Processor -- start 2024-03-18T19:05:25.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.752+0000 7f0153a7e700 1 -- start start 2024-03-18T19:05:25.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.752+0000 7f0153a7e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f014c143c20 0x7f014c0b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:25.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.752+0000 7f0153a7e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f014c06f240 0x7f014c0cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:25.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.752+0000 7f0153a7e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014c144000 con 0x7f014c0b3f40 2024-03-18T19:05:25.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.752+0000 7f0153a7e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f014c0b43c0 con 0x7f014c143c20 2024-03-18T19:05:25.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.752+0000 7f0153a7e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014c0b51b0 con 0x7f014c06f240 2024-03-18T19:05:25.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.752+0000 7f0151019700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f014c0b3f40 0x7f014c06eb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57670/0 (socket says 172.21.15.1:57670) 2024-03-18T19:05:25.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.752+0000 7f0151019700 1 -- 172.21.15.1:0/4149666727 learned_addr learned my addr 172.21.15.1:0/4149666727 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:25.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f0150818700 1 -- 172.21.15.1:0/4149666727 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452153774 0 0) 0x7f014c144000 con 0x7f014c0b3f40 2024-03-18T19:05:25.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f015201b700 1 --2- 172.21.15.1:0/4149666727 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f014c06f240 0x7f014c0cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:25.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f0150818700 1 -- 172.21.15.1:0/4149666727 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01380035b0 con 0x7f014c0b3f40 2024-03-18T19:05:25.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f015181a700 1 --2- 172.21.15.1:0/4149666727 >> v2:172.21.15.165:3300/0 conn(0x7f014c143c20 0x7f014c0b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:25.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f0150818700 1 -- 172.21.15.1:0/4149666727 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1227958579 0 0) 0x7f01380035b0 con 0x7f014c0b3f40 2024-03-18T19:05:25.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f0150818700 1 -- 172.21.15.1:0/4149666727 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f014c06f240 msgr2=0x7f014c0cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:25.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f0150818700 1 --2- 172.21.15.1:0/4149666727 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f014c06f240 0x7f014c0cddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:25.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f0150818700 1 -- 172.21.15.1:0/4149666727 >> v2:172.21.15.165:3300/0 conn(0x7f014c143c20 msgr2=0x7f014c0b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:25.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f0150818700 1 --2- 172.21.15.1:0/4149666727 >> v2:172.21.15.165:3300/0 conn(0x7f014c143c20 0x7f014c0b3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:25.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f0150818700 1 -- 172.21.15.1:0/4149666727 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c0cd8d0 con 0x7f014c0b3f40 2024-03-18T19:05:25.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f0150818700 1 -- 172.21.15.1:0/4149666727 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f0148001670 con 0x7f014c0b3f40 2024-03-18T19:05:25.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f0150818700 1 -- 172.21.15.1:0/4149666727 >> v1:172.21.15.1:6789/0 conn(0x7f014c0b3f40 legacy=0x7f014c06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:25.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f0150818700 1 --2- 172.21.15.1:0/4149666727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0138003a30 0x7f0138003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:25.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f0150818700 1 --2- 172.21.15.1:0/4149666727 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f014c06f240 0x7f0138004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:25.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f0150818700 1 --2- 172.21.15.1:0/4149666727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0138004d90 0x7f0138005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:25.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f0150818700 1 -- 172.21.15.1:0/4149666727 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014c0b43c0 con 0x7f0138004d90 2024-03-18T19:05:25.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f0150818700 1 -- 172.21.15.1:0/4149666727 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013800ba10 con 0x7f0138003a30 2024-03-18T19:05:25.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f0150818700 1 -- 172.21.15.1:0/4149666727 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f013800bb50 con 0x7f014c06f240 2024-03-18T19:05:25.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f015201b700 1 --2- 172.21.15.1:0/4149666727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0138004d90 0x7f0138005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:25.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f015181a700 1 --2- 172.21.15.1:0/4149666727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0138003a30 0x7f0138003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:25.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.753+0000 7f0151019700 1 --2- 172.21.15.1:0/4149666727 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f014c06f240 0x7f0138004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:25.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.754+0000 7f015201b700 1 -- 172.21.15.1:0/4149666727 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f014c06f240 msgr2=0x7f0138004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:25.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.754+0000 7f015201b700 1 --2- 172.21.15.1:0/4149666727 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f014c06f240 0x7f0138004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:25.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.754+0000 7f015201b700 1 -- 172.21.15.1:0/4149666727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0138003a30 msgr2=0x7f0138003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:25.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.754+0000 7f015201b700 1 --2- 172.21.15.1:0/4149666727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0138003a30 0x7f0138003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:25.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.754+0000 7f015201b700 1 -- 172.21.15.1:0/4149666727 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01380035b0 con 0x7f0138004d90 2024-03-18T19:05:25.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.754+0000 7f015201b700 1 --2- 172.21.15.1:0/4149666727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0138004d90 0x7f0138005210 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f0144009bd0 tx=0x7f0138005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d59646b3a3a1b5f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:25.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.754+0000 7f0150818700 1 -- 172.21.15.1:0/4149666727 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014400ec60 con 0x7f0138004d90 2024-03-18T19:05:25.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.754+0000 7f0150818700 1 -- 172.21.15.1:0/4149666727 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014400f040 con 0x7f0138004d90 2024-03-18T19:05:25.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.754+0000 7f0150818700 1 -- 172.21.15.1:0/4149666727 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0144012430 con 0x7f0138004d90 2024-03-18T19:05:25.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.754+0000 7f0153a7e700 1 -- 172.21.15.1:0/4149666727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0138004d90 msgr2=0x7f0138005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:25.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.754+0000 7f0153a7e700 1 --2- 172.21.15.1:0/4149666727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0138004d90 0x7f0138005210 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f0144009bd0 tx=0x7f0138005880 comp rx=0 tx=0).stop 2024-03-18T19:05:25.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.754+0000 7f015201b700 1 -- 172.21.15.1:0/4149666727 reap_dead start 2024-03-18T19:05:25.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.754+0000 7f0153a7e700 1 -- 172.21.15.1:0/4149666727 shutdown_connections 2024-03-18T19:05:25.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.754+0000 7f0153a7e700 1 -- 172.21.15.1:0/4149666727 >> 172.21.15.1:0/4149666727 conn(0x7f014c0b1d20 msgr2=0x7f014c0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:25.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.755+0000 7f0153a7e700 1 -- 172.21.15.1:0/4149666727 shutdown_connections 2024-03-18T19:05:25.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.755+0000 7f0153a7e700 1 -- 172.21.15.1:0/4149666727 wait complete. 2024-03-18T19:05:25.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.756+0000 7f0153a7e700 1 Processor -- start 2024-03-18T19:05:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.756+0000 7f0153a7e700 1 -- start start 2024-03-18T19:05:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.756+0000 7f0153a7e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0138004d90 0x7f014c159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.756+0000 7f0153a7e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f014c0caa80 0x7f014c0a8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.756+0000 7f0153a7e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f014c0aaa30 0x7f014c0aaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.756+0000 7f0153a7e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014c0aa430 con 0x7f014c0caa80 2024-03-18T19:05:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.756+0000 7f0153a7e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014c0ab420 con 0x7f0138004d90 2024-03-18T19:05:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.756+0000 7f0153a7e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014c0a97b0 con 0x7f014c0aaa30 2024-03-18T19:05:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.757+0000 7f0151019700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f014c0caa80 0x7f014c0a8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.757+0000 7f0151019700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f014c0caa80 0x7f014c0a8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33658/0 (socket says 172.21.15.1:33658) 2024-03-18T19:05:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.757+0000 7f0151019700 1 -- 172.21.15.1:0/2126068996 learned_addr learned my addr 172.21.15.1:0/2126068996 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.757+0000 7f015181a700 1 --2- 172.21.15.1:0/2126068996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0138004d90 0x7f014c159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.757+0000 7f015201b700 1 --2- 172.21.15.1:0/2126068996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f014c0aaa30 0x7f014c0aaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.757+0000 7f0151019700 1 -- 172.21.15.1:0/2126068996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f014c0aaa30 msgr2=0x7f014c0aaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.757+0000 7f0151019700 1 --2- 172.21.15.1:0/2126068996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f014c0aaa30 0x7f014c0aaee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.757+0000 7f0151019700 1 -- 172.21.15.1:0/2126068996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0138004d90 msgr2=0x7f014c159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.757+0000 7f0151019700 1 --2- 172.21.15.1:0/2126068996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0138004d90 0x7f014c159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.757+0000 7f0151019700 1 -- 172.21.15.1:0/2126068996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c0a9a90 con 0x7f014c0caa80 2024-03-18T19:05:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.757+0000 7f015181a700 1 --2- 172.21.15.1:0/2126068996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0138004d90 0x7f014c159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.757+0000 7f015201b700 1 --2- 172.21.15.1:0/2126068996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f014c0aaa30 0x7f014c0aaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.757+0000 7f0151019700 1 --2- 172.21.15.1:0/2126068996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f014c0caa80 0x7f014c0a8e70 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f0148007340 tx=0x7f0148007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.757+0000 7f01427fc700 1 -- 172.21.15.1:0/2126068996 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014800f6c0 con 0x7f014c0caa80 2024-03-18T19:05:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.757+0000 7f01427fc700 1 -- 172.21.15.1:0/2126068996 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014800f840 con 0x7f014c0caa80 2024-03-18T19:05:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.757+0000 7f0153a7e700 1 -- 172.21.15.1:0/2126068996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f014c0a9d70 con 0x7f014c0caa80 2024-03-18T19:05:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.757+0000 7f01427fc700 1 -- 172.21.15.1:0/2126068996 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014800fbf0 con 0x7f014c0caa80 2024-03-18T19:05:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.757+0000 7f0153a7e700 1 -- 172.21.15.1:0/2126068996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f014c159e50 con 0x7f014c0caa80 2024-03-18T19:05:25.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.758+0000 7f0153a7e700 1 -- 172.21.15.1:0/2126068996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f014c0aa570 con 0x7f014c0caa80 2024-03-18T19:05:25.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.759+0000 7f01427fc700 1 -- 172.21.15.1:0/2126068996 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f014800fd50 con 0x7f014c0caa80 2024-03-18T19:05:25.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.759+0000 7f01427fc700 1 --2- 172.21.15.1:0/2126068996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f013006adb0 0x7f013006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:25.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.759+0000 7f01427fc700 1 -- 172.21.15.1:0/2126068996 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0148089570 con 0x7f014c0caa80 2024-03-18T19:05:25.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.759+0000 7f015181a700 1 --2- 172.21.15.1:0/2126068996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f013006adb0 0x7f013006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:25.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.760+0000 7f015181a700 1 --2- 172.21.15.1:0/2126068996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f013006adb0 0x7f013006d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f013c009bd0 tx=0x7f013c008040 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:25.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:25.764+0000 7f01427fc700 1 -- 172.21.15.1:0/2126068996 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f014c0aa570 con 0x7f014c0caa80 2024-03-18T19:05:26.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.011+0000 7f0153a7e700 1 -- 172.21.15.1:0/2126068996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f014c0aa570 con 0x7f014c0caa80 2024-03-18T19:05:26.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.011+0000 7f01427fc700 1 -- 172.21.15.1:0/2126068996 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4732 (secure 0 0 0) 0x7f014c0aa570 con 0x7f014c0caa80 2024-03-18T19:05:26.007 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:26.007 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5744,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2437290126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2437290126},{"type":"v1","addr":"172.21.15.1:6836","nonce":2437290126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5747,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2832642785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2832642785},{"type":"v1","addr":"172.21.15.1:6837","nonce":2832642785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5756,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3765527492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3765527492},{"type":"v1","addr":"172.21.15.165:6834","nonce":3765527492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:05:22.577555+0000","modified":"2024-03-18T19:05:25.594597+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5759},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5759":{"gid":5759,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/119895951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":119895951},{"type":"v1","addr":"172.21.15.165:6835","nonce":119895951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-18T19:05:26.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.013+0000 7f0153a7e700 1 -- 172.21.15.1:0/2126068996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f013006adb0 msgr2=0x7f013006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:26.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.013+0000 7f0153a7e700 1 --2- 172.21.15.1:0/2126068996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f013006adb0 0x7f013006d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f013c009bd0 tx=0x7f013c008040 comp rx=0 tx=0).stop 2024-03-18T19:05:26.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.013+0000 7f0153a7e700 1 -- 172.21.15.1:0/2126068996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f014c0caa80 msgr2=0x7f014c0a8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:26.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.013+0000 7f0153a7e700 1 --2- 172.21.15.1:0/2126068996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f014c0caa80 0x7f014c0a8e70 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f0148007340 tx=0x7f0148007650 comp rx=0 tx=0).stop 2024-03-18T19:05:26.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.013+0000 7f0153a7e700 1 -- 172.21.15.1:0/2126068996 shutdown_connections 2024-03-18T19:05:26.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.013+0000 7f0153a7e700 1 --2- 172.21.15.1:0/2126068996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f013006adb0 0x7f013006d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.013+0000 7f0153a7e700 1 --2- 172.21.15.1:0/2126068996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f014c0aaa30 0x7f014c0aaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.013+0000 7f0153a7e700 1 --2- 172.21.15.1:0/2126068996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0138004d90 0x7f014c159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.013+0000 7f0153a7e700 1 --2- 172.21.15.1:0/2126068996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f014c0caa80 0x7f014c0a8e70 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.013+0000 7f0153a7e700 1 -- 172.21.15.1:0/2126068996 >> 172.21.15.1:0/2126068996 conn(0x7f014c0b1d20 msgr2=0x7f014c1428d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:26.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.013+0000 7f0153a7e700 1 -- 172.21.15.1:0/2126068996 shutdown_connections 2024-03-18T19:05:26.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.013+0000 7f0153a7e700 1 -- 172.21.15.1:0/2126068996 wait complete. 2024-03-18T19:05:26.010 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 33 2024-03-18T19:05:26.021 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:05:26.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.151+0000 7f9fd9b98700 1 Processor -- start 2024-03-18T19:05:26.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.151+0000 7f9fd9b98700 1 -- start start 2024-03-18T19:05:26.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.151+0000 7f9fd9b98700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9fd40cc5e0 0x7f9fd406e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:26.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.151+0000 7f9fd9b98700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9fd406ed50 0x7f9fd4099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:26.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.151+0000 7f9fd9b98700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd415b6e0 con 0x7f9fd415b300 2024-03-18T19:05:26.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.151+0000 7f9fd9b98700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fd40cce60 con 0x7f9fd40cc5e0 2024-03-18T19:05:26.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.151+0000 7f9fd9b98700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fd40cecd0 con 0x7f9fd406ed50 2024-03-18T19:05:26.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.152+0000 7f9fd2ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9fd40cc5e0 0x7f9fd406e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:26.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.152+0000 7f9fd2ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9fd40cc5e0 0x7f9fd406e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33674/0 (socket says 172.21.15.1:33674) 2024-03-18T19:05:26.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.152+0000 7f9fd2ffd700 1 -- 172.21.15.1:0/3320635441 learned_addr learned my addr 172.21.15.1:0/3320635441 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:26.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.152+0000 7f9fd3fff700 1 --2- 172.21.15.1:0/3320635441 >> v2:172.21.15.165:3300/0 conn(0x7f9fd406ed50 0x7f9fd4099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:26.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.152+0000 7f9fd2ffd700 1 -- 172.21.15.1:0/3320635441 >> v2:172.21.15.165:3300/0 conn(0x7f9fd406ed50 msgr2=0x7f9fd4099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:26.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.152+0000 7f9fd2ffd700 1 --2- 172.21.15.1:0/3320635441 >> v2:172.21.15.165:3300/0 conn(0x7f9fd406ed50 0x7f9fd4099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.152+0000 7f9fd2ffd700 1 -- 172.21.15.1:0/3320635441 >> v1:172.21.15.165:6789/0 conn(0x7f9fd415b300 legacy=0x7f9fd40cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:26.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.152+0000 7f9fd2ffd700 1 -- 172.21.15.1:0/3320635441 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd40cefb0 con 0x7f9fd40cc5e0 2024-03-18T19:05:26.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.152+0000 7f9fd2ffd700 1 --2- 172.21.15.1:0/3320635441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fd40cc5e0 0x7f9fd406e810 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4067160 tx=0x7f9fc8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=934f93024d3d28df server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:26.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.152+0000 7f9fd27fc700 1 -- 172.21.15.1:0/3320635441 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8004230 con 0x7f9fd40cc5e0 2024-03-18T19:05:26.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.152+0000 7f9fd27fc700 1 -- 172.21.15.1:0/3320635441 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc8004390 con 0x7f9fd40cc5e0 2024-03-18T19:05:26.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.152+0000 7f9fd27fc700 1 -- 172.21.15.1:0/3320635441 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc80036a0 con 0x7f9fd40cc5e0 2024-03-18T19:05:26.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.153+0000 7f9fd9b98700 1 -- 172.21.15.1:0/3320635441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fd40cc5e0 msgr2=0x7f9fd406e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:26.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.153+0000 7f9fd9b98700 1 --2- 172.21.15.1:0/3320635441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fd40cc5e0 0x7f9fd406e810 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4067160 tx=0x7f9fc8009bd0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.153+0000 7f9fd9b98700 1 -- 172.21.15.1:0/3320635441 shutdown_connections 2024-03-18T19:05:26.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.153+0000 7f9fd9b98700 1 --2- 172.21.15.1:0/3320635441 >> v2:172.21.15.165:3300/0 conn(0x7f9fd406ed50 0x7f9fd4099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.153+0000 7f9fd9b98700 1 --2- 172.21.15.1:0/3320635441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fd40cc5e0 0x7f9fd406e810 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.153+0000 7f9fd9b98700 1 -- 172.21.15.1:0/3320635441 >> 172.21.15.1:0/3320635441 conn(0x7f9fd40b2b50 msgr2=0x7f9fd40b2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:26.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.153+0000 7f9fd9b98700 1 -- 172.21.15.1:0/3320635441 shutdown_connections 2024-03-18T19:05:26.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.153+0000 7f9fd9b98700 1 -- 172.21.15.1:0/3320635441 wait complete. 2024-03-18T19:05:26.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.154+0000 7f9fd9b98700 1 Processor -- start 2024-03-18T19:05:26.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.155+0000 7f9fd9b98700 1 -- start start 2024-03-18T19:05:26.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.155+0000 7f9fd9b98700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fd406ed50 0x7f9fd4147250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:26.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.155+0000 7f9fd9b98700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9fd40cc5e0 0x7f9fd40b6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:26.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.155+0000 7f9fd9b98700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9fd415b300 0x7f9fd415d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:26.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.155+0000 7f9fd9b98700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd415c920 con 0x7f9fd406ed50 2024-03-18T19:05:26.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.155+0000 7f9fd9b98700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd4147790 con 0x7f9fd40cc5e0 2024-03-18T19:05:26.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.155+0000 7f9fd9b98700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd40b3ba0 con 0x7f9fd415b300 2024-03-18T19:05:26.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.156+0000 7f9fd37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fd406ed50 0x7f9fd4147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:26.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.156+0000 7f9fd37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fd406ed50 0x7f9fd4147250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33686/0 (socket says 172.21.15.1:33686) 2024-03-18T19:05:26.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.156+0000 7f9fd37fe700 1 -- 172.21.15.1:0/1622250636 learned_addr learned my addr 172.21.15.1:0/1622250636 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:26.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.156+0000 7f9fd2ffd700 1 --2- 172.21.15.1:0/1622250636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9fd40cc5e0 0x7f9fd40b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:26.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.156+0000 7f9fd3fff700 1 --2- 172.21.15.1:0/1622250636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9fd415b300 0x7f9fd415d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:26.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.156+0000 7f9fd37fe700 1 -- 172.21.15.1:0/1622250636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9fd415b300 msgr2=0x7f9fd415d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:26.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.156+0000 7f9fd37fe700 1 --2- 172.21.15.1:0/1622250636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9fd415b300 0x7f9fd415d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.156+0000 7f9fd37fe700 1 -- 172.21.15.1:0/1622250636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9fd40cc5e0 msgr2=0x7f9fd40b6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:26.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.156+0000 7f9fd37fe700 1 --2- 172.21.15.1:0/1622250636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9fd40cc5e0 0x7f9fd40b6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.156+0000 7f9fd37fe700 1 -- 172.21.15.1:0/1622250636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc8009550 con 0x7f9fd406ed50 2024-03-18T19:05:26.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.156+0000 7f9fd2ffd700 1 --2- 172.21.15.1:0/1622250636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9fd40cc5e0 0x7f9fd40b6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:26.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.156+0000 7f9fd37fe700 1 --2- 172.21.15.1:0/1622250636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fd406ed50 0x7f9fd4147250 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f9fc400b4a0 tx=0x7f9fc400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:26.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.156+0000 7f9fbbfff700 1 -- 172.21.15.1:0/1622250636 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4011830 con 0x7f9fd406ed50 2024-03-18T19:05:26.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.156+0000 7f9fbbfff700 1 -- 172.21.15.1:0/1622250636 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc40119b0 con 0x7f9fd406ed50 2024-03-18T19:05:26.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.156+0000 7f9fd9b98700 1 -- 172.21.15.1:0/1622250636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fd415cb70 con 0x7f9fd406ed50 2024-03-18T19:05:26.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.156+0000 7f9fbbfff700 1 -- 172.21.15.1:0/1622250636 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4011d60 con 0x7f9fd406ed50 2024-03-18T19:05:26.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.156+0000 7f9fd9b98700 1 -- 172.21.15.1:0/1622250636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd415d130 con 0x7f9fd406ed50 2024-03-18T19:05:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.157+0000 7f9fd9b98700 1 -- 172.21.15.1:0/1622250636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fd40676a0 con 0x7f9fd406ed50 2024-03-18T19:05:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.158+0000 7f9fbbfff700 1 -- 172.21.15.1:0/1622250636 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9fc401a400 con 0x7f9fd406ed50 2024-03-18T19:05:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.158+0000 7f9fbbfff700 1 --2- 172.21.15.1:0/1622250636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9fc006adb0 0x7f9fc006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.158+0000 7f9fbbfff700 1 -- 172.21.15.1:0/1622250636 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9fc4089600 con 0x7f9fd406ed50 2024-03-18T19:05:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.158+0000 7f9fd2ffd700 1 --2- 172.21.15.1:0/1622250636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9fc006adb0 0x7f9fc006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.159+0000 7f9fd2ffd700 1 --2- 172.21.15.1:0/1622250636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9fc006adb0 0x7f9fc006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8009130 tx=0x7f9fc8008030 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:26.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.163+0000 7f9fbbfff700 1 -- 172.21.15.1:0/1622250636 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9fc40545a0 con 0x7f9fd406ed50 2024-03-18T19:05:26.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.366+0000 7f9fd9b98700 1 -- 172.21.15.1:0/1622250636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fd40ae350 con 0x7f9fd406ed50 2024-03-18T19:05:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.367+0000 7f9fbbfff700 1 -- 172.21.15.1:0/1622250636 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16200 (secure 0 0 0) 0x7f9fc4057bc0 con 0x7f9fd406ed50 2024-03-18T19:05:26.363 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:26.364 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":89,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:05:23.579362+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:05:20.423723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-18T19:05:21.600285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:05:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.369+0000 7f9fd9b98700 1 -- 172.21.15.1:0/1622250636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9fc006adb0 msgr2=0x7f9fc006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.369+0000 7f9fd9b98700 1 --2- 172.21.15.1:0/1622250636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9fc006adb0 0x7f9fc006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8009130 tx=0x7f9fc8008030 comp rx=0 tx=0).stop 2024-03-18T19:05:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.369+0000 7f9fd9b98700 1 -- 172.21.15.1:0/1622250636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fd406ed50 msgr2=0x7f9fd4147250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.369+0000 7f9fd9b98700 1 --2- 172.21.15.1:0/1622250636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fd406ed50 0x7f9fd4147250 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f9fc400b4a0 tx=0x7f9fc400b860 comp rx=0 tx=0).stop 2024-03-18T19:05:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.369+0000 7f9fd9b98700 1 -- 172.21.15.1:0/1622250636 shutdown_connections 2024-03-18T19:05:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.370+0000 7f9fd9b98700 1 --2- 172.21.15.1:0/1622250636 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9fc006adb0 0x7f9fc006d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.370+0000 7f9fd9b98700 1 --2- 172.21.15.1:0/1622250636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9fd415b300 0x7f9fd415d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.370+0000 7f9fd9b98700 1 --2- 172.21.15.1:0/1622250636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fd406ed50 0x7f9fd4147250 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.370+0000 7f9fd9b98700 1 --2- 172.21.15.1:0/1622250636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9fd40cc5e0 0x7f9fd40b6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.370+0000 7f9fd9b98700 1 -- 172.21.15.1:0/1622250636 >> 172.21.15.1:0/1622250636 conn(0x7f9fd40b2b50 msgr2=0x7f9fd40ca140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.370+0000 7f9fd9b98700 1 -- 172.21.15.1:0/1622250636 shutdown_connections 2024-03-18T19:05:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.370+0000 7f9fd9b98700 1 -- 172.21.15.1:0/1622250636 wait complete. 2024-03-18T19:05:26.378 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T19:05:26.379 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T19:05:26.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.508+0000 7fdf19015700 1 Processor -- start 2024-03-18T19:05:26.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.508+0000 7fdf19015700 1 -- start start 2024-03-18T19:05:26.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.509+0000 7fdf19015700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf1415d580 0x7fdf14072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:26.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.509+0000 7fdf19015700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdf1406e290 0x7fdf140cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:26.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.509+0000 7fdf19015700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf1406f490 con 0x7fdf1406d640 2024-03-18T19:05:26.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.509+0000 7fdf19015700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf14071cf0 con 0x7fdf1406e290 2024-03-18T19:05:26.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.509+0000 7fdf19015700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1406f1b0 con 0x7fdf1415d580 2024-03-18T19:05:26.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.509+0000 7fdf12d9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf1415d580 0x7fdf14072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:26.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.509+0000 7fdf1259c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdf1406e290 0x7fdf140cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.509+0000 7fdf12d9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf1415d580 0x7fdf14072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:39942/0 (socket says 172.21.15.1:39942) 2024-03-18T19:05:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.509+0000 7fdf12d9d700 1 -- 172.21.15.1:0/2730674818 learned_addr learned my addr 172.21.15.1:0/2730674818 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.510+0000 7fdf12d9d700 1 -- 172.21.15.1:0/2730674818 >> v2:172.21.15.165:3300/0 conn(0x7fdf1406e290 msgr2=0x7fdf140cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.510+0000 7fdf12d9d700 1 --2- 172.21.15.1:0/2730674818 >> v2:172.21.15.165:3300/0 conn(0x7fdf1406e290 0x7fdf140cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.510+0000 7fdf12d9d700 1 -- 172.21.15.1:0/2730674818 >> v1:172.21.15.165:6789/0 conn(0x7fdf1406d640 legacy=0x7fdf140ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.510+0000 7fdf12d9d700 1 -- 172.21.15.1:0/2730674818 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf140716a0 con 0x7fdf1415d580 2024-03-18T19:05:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.510+0000 7fdf1259c700 1 --2- 172.21.15.1:0/2730674818 >> v2:172.21.15.165:3300/0 conn(0x7fdf1406e290 0x7fdf140cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.510+0000 7fdf12d9d700 1 --2- 172.21.15.1:0/2730674818 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf1415d580 0x7fdf14072180 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdf080098f0 tx=0x7fdf08009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd3ccab9ef532ac7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.510+0000 7fdf11d9b700 1 -- 172.21.15.1:0/2730674818 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf08011830 con 0x7fdf1415d580 2024-03-18T19:05:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.510+0000 7fdf11d9b700 1 -- 172.21.15.1:0/2730674818 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf08011990 con 0x7fdf1415d580 2024-03-18T19:05:26.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.510+0000 7fdf11d9b700 1 -- 172.21.15.1:0/2730674818 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf08011d20 con 0x7fdf1415d580 2024-03-18T19:05:26.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.510+0000 7fdf19015700 1 -- 172.21.15.1:0/2730674818 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf1415d580 msgr2=0x7fdf14072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:26.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.510+0000 7fdf19015700 1 --2- 172.21.15.1:0/2730674818 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf1415d580 0x7fdf14072180 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdf080098f0 tx=0x7fdf08009c00 comp rx=0 tx=0).stop 2024-03-18T19:05:26.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.510+0000 7fdf19015700 1 -- 172.21.15.1:0/2730674818 shutdown_connections 2024-03-18T19:05:26.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.510+0000 7fdf19015700 1 --2- 172.21.15.1:0/2730674818 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf1415d580 0x7fdf14072180 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.510+0000 7fdf19015700 1 --2- 172.21.15.1:0/2730674818 >> v2:172.21.15.165:3300/0 conn(0x7fdf1406e290 0x7fdf140cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.511+0000 7fdf19015700 1 -- 172.21.15.1:0/2730674818 >> 172.21.15.1:0/2730674818 conn(0x7fdf140cc990 msgr2=0x7fdf140ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.511+0000 7fdf19015700 1 -- 172.21.15.1:0/2730674818 shutdown_connections 2024-03-18T19:05:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.511+0000 7fdf19015700 1 -- 172.21.15.1:0/2730674818 wait complete. 2024-03-18T19:05:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.512+0000 7fdf19015700 1 Processor -- start 2024-03-18T19:05:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.512+0000 7fdf19015700 1 -- start start 2024-03-18T19:05:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.513+0000 7fdf19015700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf1406d640 0x7fdf141e6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.513+0000 7fdf19015700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf1415d580 0x7fdf141e8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.513+0000 7fdf19015700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf141f65b0 0x7fdf141f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.513+0000 7fdf19015700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf140c7610 con 0x7fdf1406d640 2024-03-18T19:05:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.513+0000 7fdf19015700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf140c6fe0 con 0x7fdf1415d580 2024-03-18T19:05:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.513+0000 7fdf19015700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf141f7050 con 0x7fdf141f65b0 2024-03-18T19:05:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.513+0000 7fdf12d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf1406d640 0x7fdf141e6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.513+0000 7fdf12d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf1406d640 0x7fdf141e6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33702/0 (socket says 172.21.15.1:33702) 2024-03-18T19:05:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.513+0000 7fdf12d9d700 1 -- 172.21.15.1:0/3002048472 learned_addr learned my addr 172.21.15.1:0/3002048472 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.513+0000 7fdf1259c700 1 --2- 172.21.15.1:0/3002048472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf1415d580 0x7fdf141e8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.513+0000 7fdf1359e700 1 --2- 172.21.15.1:0/3002048472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf141f65b0 0x7fdf141f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.513+0000 7fdf12d9d700 1 -- 172.21.15.1:0/3002048472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf141f65b0 msgr2=0x7fdf141f6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.513+0000 7fdf12d9d700 1 --2- 172.21.15.1:0/3002048472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf141f65b0 0x7fdf141f6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.513+0000 7fdf12d9d700 1 -- 172.21.15.1:0/3002048472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf1415d580 msgr2=0x7fdf141e8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.513+0000 7fdf12d9d700 1 --2- 172.21.15.1:0/3002048472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf1415d580 0x7fdf141e8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.513+0000 7fdf12d9d700 1 -- 172.21.15.1:0/3002048472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf040094a0 con 0x7fdf1406d640 2024-03-18T19:05:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.513+0000 7fdf12d9d700 1 --2- 172.21.15.1:0/3002048472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf1406d640 0x7fdf141e6110 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fdf08014040 tx=0x7fdf08004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.514+0000 7fdefb7fe700 1 -- 172.21.15.1:0/3002048472 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf08004510 con 0x7fdf1406d640 2024-03-18T19:05:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.514+0000 7fdefb7fe700 1 -- 172.21.15.1:0/3002048472 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0801e070 con 0x7fdf1406d640 2024-03-18T19:05:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.514+0000 7fdf19015700 1 -- 172.21.15.1:0/3002048472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf08009550 con 0x7fdf1406d640 2024-03-18T19:05:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.514+0000 7fdefb7fe700 1 -- 172.21.15.1:0/3002048472 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf080038e0 con 0x7fdf1406d640 2024-03-18T19:05:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.514+0000 7fdf19015700 1 -- 172.21.15.1:0/3002048472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf140c7230 con 0x7fdf1406d640 2024-03-18T19:05:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.515+0000 7fdf19015700 1 -- 172.21.15.1:0/3002048472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf140676a0 con 0x7fdf1406d640 2024-03-18T19:05:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.515+0000 7fdefb7fe700 1 -- 172.21.15.1:0/3002048472 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fdf08003a40 con 0x7fdf1406d640 2024-03-18T19:05:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.515+0000 7fdefb7fe700 1 --2- 172.21.15.1:0/3002048472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf0006adb0 0x7fdf0006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:26.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.516+0000 7fdefb7fe700 1 -- 172.21.15.1:0/3002048472 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fdf0808a4d0 con 0x7fdf1406d640 2024-03-18T19:05:26.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.516+0000 7fdf1259c700 1 --2- 172.21.15.1:0/3002048472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf0006adb0 0x7fdf0006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:26.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.516+0000 7fdf1259c700 1 --2- 172.21.15.1:0/3002048472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf0006adb0 0x7fdf0006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdf141e5b20 tx=0x7fdf04009210 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:26.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.520+0000 7fdefb7fe700 1 -- 172.21.15.1:0/3002048472 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdf080553d0 con 0x7fdf1406d640 2024-03-18T19:05:26.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.717+0000 7fdf19015700 1 -- 172.21.15.1:0/3002048472 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdf1406ee40 con 0x7fdf0006adb0 2024-03-18T19:05:26.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.726+0000 7fdefb7fe700 1 -- 172.21.15.1:0/3002048472 <== mgr.6238 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289631 (secure 0 0 0) 0x7fdf040099c0 con 0x7fdf0006adb0 2024-03-18T19:05:26.725 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:26.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.729+0000 7fdf19015700 1 -- 172.21.15.1:0/3002048472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf0006adb0 msgr2=0x7fdf0006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:26.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.729+0000 7fdf19015700 1 --2- 172.21.15.1:0/3002048472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf0006adb0 0x7fdf0006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdf141e5b20 tx=0x7fdf04009210 comp rx=0 tx=0).stop 2024-03-18T19:05:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.729+0000 7fdf19015700 1 -- 172.21.15.1:0/3002048472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf1406d640 msgr2=0x7fdf141e6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.729+0000 7fdf19015700 1 --2- 172.21.15.1:0/3002048472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf1406d640 0x7fdf141e6110 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fdf08014040 tx=0x7fdf08004750 comp rx=0 tx=0).stop 2024-03-18T19:05:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.729+0000 7fdf19015700 1 -- 172.21.15.1:0/3002048472 shutdown_connections 2024-03-18T19:05:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.729+0000 7fdf19015700 1 --2- 172.21.15.1:0/3002048472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf0006adb0 0x7fdf0006d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.729+0000 7fdf19015700 1 --2- 172.21.15.1:0/3002048472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf141f65b0 0x7fdf141f6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.729+0000 7fdf19015700 1 --2- 172.21.15.1:0/3002048472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf1406d640 0x7fdf141e6110 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.729+0000 7fdf19015700 1 --2- 172.21.15.1:0/3002048472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf1415d580 0x7fdf141e8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.729+0000 7fdf19015700 1 -- 172.21.15.1:0/3002048472 >> 172.21.15.1:0/3002048472 conn(0x7fdf140cc990 msgr2=0x7fdf140b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.729+0000 7fdf19015700 1 -- 172.21.15.1:0/3002048472 shutdown_connections 2024-03-18T19:05:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.729+0000 7fdf19015700 1 -- 172.21.15.1:0/3002048472 wait complete. 2024-03-18T19:05:26.728 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T19:05:26.737 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-18T19:05:25.020990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":131,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":337296,"kb_used_data":9612,"kb_used_omap":12,"kb_used_meta":327539,"kb_avail":754637424,"statfs":{"total":773094113280,"available":772748722176,"internally_reserved":0,"allocated":9842688,"data_stored":5464280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":335400505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000746"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.599445+0000","last_change":"2024-03-18T19:05:22.593122+0000","last_active":"2024-03-18T19:05:22.599445+0000","last_peered":"2024-03-18T19:05:22.599445+0000","last_clean":"2024-03-18T19:05:22.599445+0000","last_became_active":"2024-03-18T19:05:22.592878+0000","last_became_peered":"2024-03-18T19:05:22.592878+0000","last_unstale":"2024-03-18T19:05:22.599445+0000","last_undegraded":"2024-03-18T19:05:22.599445+0000","last_fullsized":"2024-03-18T19:05:22.599445+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:50.436797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.598613+0000","last_change":"2024-03-18T19:05:22.592083+0000","last_active":"2024-03-18T19:05:22.598613+0000","last_peered":"2024-03-18T19:05:22.598613+0000","last_clean":"2024-03-18T19:05:22.598613+0000","last_became_active":"2024-03-18T19:05:22.591681+0000","last_became_peered":"2024-03-18T19:05:22.591681+0000","last_unstale":"2024-03-18T19:05:22.598613+0000","last_undegraded":"2024-03-18T19:05:22.598613+0000","last_fullsized":"2024-03-18T19:05:22.598613+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:52.820873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.608848+0000","last_change":"2024-03-18T19:05:22.605702+0000","last_active":"2024-03-18T19:05:22.608848+0000","last_peered":"2024-03-18T19:05:22.608848+0000","last_clean":"2024-03-18T19:05:22.608848+0000","last_became_active":"2024-03-18T19:05:22.605450+0000","last_became_peered":"2024-03-18T19:05:22.605450+0000","last_unstale":"2024-03-18T19:05:22.608848+0000","last_undegraded":"2024-03-18T19:05:22.608848+0000","last_fullsized":"2024-03-18T19:05:22.608848+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:52.107864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966527+0000","last_change":"2024-03-18T19:05:22.597848+0000","last_active":"2024-03-18T19:05:24.966527+0000","last_peered":"2024-03-18T19:05:24.966527+0000","last_clean":"2024-03-18T19:05:24.966527+0000","last_became_active":"2024-03-18T19:05:22.597589+0000","last_became_peered":"2024-03-18T19:05:22.597589+0000","last_unstale":"2024-03-18T19:05:24.966527+0000","last_undegraded":"2024-03-18T19:05:24.966527+0000","last_fullsized":"2024-03-18T19:05:24.966527+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:21:54.604175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590513+0000","last_change":"2024-03-18T19:05:22.599635+0000","last_active":"2024-03-18T19:05:23.590513+0000","last_peered":"2024-03-18T19:05:23.590513+0000","last_clean":"2024-03-18T19:05:23.590513+0000","last_became_active":"2024-03-18T19:05:22.597583+0000","last_became_peered":"2024-03-18T19:05:22.597583+0000","last_unstale":"2024-03-18T19:05:23.590513+0000","last_undegraded":"2024-03-18T19:05:23.590513+0000","last_fullsized":"2024-03-18T19:05:23.590513+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:28.026254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.608151+0000","last_change":"2024-03-18T19:05:22.607718+0000","last_active":"2024-03-18T19:05:22.608151+0000","last_peered":"2024-03-18T19:05:22.608151+0000","last_clean":"2024-03-18T19:05:22.608151+0000","last_became_active":"2024-03-18T19:05:22.607454+0000","last_became_peered":"2024-03-18T19:05:22.607454+0000","last_unstale":"2024-03-18T19:05:22.608151+0000","last_undegraded":"2024-03-18T19:05:22.608151+0000","last_fullsized":"2024-03-18T19:05:22.608151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:04.088770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.591187+0000","last_change":"2024-03-18T19:05:22.603594+0000","last_active":"2024-03-18T19:05:23.591187+0000","last_peered":"2024-03-18T19:05:23.591187+0000","last_clean":"2024-03-18T19:05:23.591187+0000","last_became_active":"2024-03-18T19:05:22.603392+0000","last_became_peered":"2024-03-18T19:05:22.603392+0000","last_unstale":"2024-03-18T19:05:23.591187+0000","last_undegraded":"2024-03-18T19:05:23.591187+0000","last_fullsized":"2024-03-18T19:05:23.591187+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:18:57.616746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597201+0000","last_change":"2024-03-18T19:05:22.604349+0000","last_active":"2024-03-18T19:05:24.597201+0000","last_peered":"2024-03-18T19:05:24.597201+0000","last_clean":"2024-03-18T19:05:24.597201+0000","last_became_active":"2024-03-18T19:05:22.603810+0000","last_became_peered":"2024-03-18T19:05:22.603810+0000","last_unstale":"2024-03-18T19:05:24.597201+0000","last_undegraded":"2024-03-18T19:05:24.597201+0000","last_fullsized":"2024-03-18T19:05:24.597201+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:52.820873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.609474+0000","last_change":"2024-03-18T19:05:22.608030+0000","last_active":"2024-03-18T19:05:22.609474+0000","last_peered":"2024-03-18T19:05:22.609474+0000","last_clean":"2024-03-18T19:05:22.609474+0000","last_became_active":"2024-03-18T19:05:22.607736+0000","last_became_peered":"2024-03-18T19:05:22.607736+0000","last_unstale":"2024-03-18T19:05:22.609474+0000","last_undegraded":"2024-03-18T19:05:22.609474+0000","last_fullsized":"2024-03-18T19:05:22.609474+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:18:55.815832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.606143+0000","last_change":"2024-03-18T19:05:22.604209+0000","last_active":"2024-03-18T19:05:22.606143+0000","last_peered":"2024-03-18T19:05:22.606143+0000","last_clean":"2024-03-18T19:05:22.606143+0000","last_became_active":"2024-03-18T19:05:22.603909+0000","last_became_peered":"2024-03-18T19:05:22.603909+0000","last_unstale":"2024-03-18T19:05:22.606143+0000","last_undegraded":"2024-03-18T19:05:22.606143+0000","last_fullsized":"2024-03-18T19:05:22.606143+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:50.436797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966686+0000","last_change":"2024-03-18T19:05:22.610141+0000","last_active":"2024-03-18T19:05:24.966686+0000","last_peered":"2024-03-18T19:05:24.966686+0000","last_clean":"2024-03-18T19:05:24.966686+0000","last_became_active":"2024-03-18T19:05:22.609866+0000","last_became_peered":"2024-03-18T19:05:22.609866+0000","last_unstale":"2024-03-18T19:05:24.966686+0000","last_undegraded":"2024-03-18T19:05:24.966686+0000","last_fullsized":"2024-03-18T19:05:24.966686+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:06:42.914898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597558+0000","last_change":"2024-03-18T19:05:22.604073+0000","last_active":"2024-03-18T19:05:24.597558+0000","last_peered":"2024-03-18T19:05:24.597558+0000","last_clean":"2024-03-18T19:05:24.597558+0000","last_became_active":"2024-03-18T19:05:22.603786+0000","last_became_peered":"2024-03-18T19:05:22.603786+0000","last_unstale":"2024-03-18T19:05:24.597558+0000","last_undegraded":"2024-03-18T19:05:24.597558+0000","last_fullsized":"2024-03-18T19:05:24.597558+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:21.799481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597667+0000","last_change":"2024-03-18T19:05:22.611538+0000","last_active":"2024-03-18T19:05:24.597667+0000","last_peered":"2024-03-18T19:05:24.597667+0000","last_clean":"2024-03-18T19:05:24.597667+0000","last_became_active":"2024-03-18T19:05:22.611279+0000","last_became_peered":"2024-03-18T19:05:22.611279+0000","last_unstale":"2024-03-18T19:05:24.597667+0000","last_undegraded":"2024-03-18T19:05:24.597667+0000","last_fullsized":"2024-03-18T19:05:24.597667+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:50.436797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.611080+0000","last_change":"2024-03-18T19:05:22.607481+0000","last_active":"2024-03-18T19:05:22.611080+0000","last_peered":"2024-03-18T19:05:22.611080+0000","last_clean":"2024-03-18T19:05:22.611080+0000","last_became_active":"2024-03-18T19:05:22.607184+0000","last_became_peered":"2024-03-18T19:05:22.607184+0000","last_unstale":"2024-03-18T19:05:22.611080+0000","last_undegraded":"2024-03-18T19:05:22.611080+0000","last_fullsized":"2024-03-18T19:05:22.611080+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:21.799481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.610223+0000","last_change":"2024-03-18T19:05:22.606780+0000","last_active":"2024-03-18T19:05:22.610223+0000","last_peered":"2024-03-18T19:05:22.610223+0000","last_clean":"2024-03-18T19:05:22.610223+0000","last_became_active":"2024-03-18T19:05:22.606483+0000","last_became_peered":"2024-03-18T19:05:22.606483+0000","last_unstale":"2024-03-18T19:05:22.610223+0000","last_undegraded":"2024-03-18T19:05:22.610223+0000","last_fullsized":"2024-03-18T19:05:22.610223+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:49:41.091622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.591067+0000","last_change":"2024-03-18T19:05:22.593988+0000","last_active":"2024-03-18T19:05:23.591067+0000","last_peered":"2024-03-18T19:05:23.591067+0000","last_clean":"2024-03-18T19:05:23.591067+0000","last_became_active":"2024-03-18T19:05:22.593698+0000","last_became_peered":"2024-03-18T19:05:22.593698+0000","last_unstale":"2024-03-18T19:05:23.591067+0000","last_undegraded":"2024-03-18T19:05:23.591067+0000","last_fullsized":"2024-03-18T19:05:23.591067+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:42:21.238001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.608007+0000","last_change":"2024-03-18T19:05:22.607110+0000","last_active":"2024-03-18T19:05:22.608007+0000","last_peered":"2024-03-18T19:05:22.608007+0000","last_clean":"2024-03-18T19:05:22.608007+0000","last_became_active":"2024-03-18T19:05:22.606777+0000","last_became_peered":"2024-03-18T19:05:22.606777+0000","last_unstale":"2024-03-18T19:05:22.608007+0000","last_undegraded":"2024-03-18T19:05:22.608007+0000","last_fullsized":"2024-03-18T19:05:22.608007+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:56:09.533605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966624+0000","last_change":"2024-03-18T19:05:22.598497+0000","last_active":"2024-03-18T19:05:24.966624+0000","last_peered":"2024-03-18T19:05:24.966624+0000","last_clean":"2024-03-18T19:05:24.966624+0000","last_became_active":"2024-03-18T19:05:22.598256+0000","last_became_peered":"2024-03-18T19:05:22.598256+0000","last_unstale":"2024-03-18T19:05:24.966624+0000","last_undegraded":"2024-03-18T19:05:24.966624+0000","last_fullsized":"2024-03-18T19:05:24.966624+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:49:03.493777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590714+0000","last_change":"2024-03-18T19:05:22.589776+0000","last_active":"2024-03-18T19:05:23.590714+0000","last_peered":"2024-03-18T19:05:23.590714+0000","last_clean":"2024-03-18T19:05:23.590714+0000","last_became_active":"2024-03-18T19:05:22.589482+0000","last_became_peered":"2024-03-18T19:05:22.589482+0000","last_unstale":"2024-03-18T19:05:23.590714+0000","last_undegraded":"2024-03-18T19:05:23.590714+0000","last_fullsized":"2024-03-18T19:05:23.590714+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:04:08.801897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.114111+0000","last_change":"2024-03-18T19:05:22.595248+0000","last_active":"2024-03-18T19:05:24.114111+0000","last_peered":"2024-03-18T19:05:24.114111+0000","last_clean":"2024-03-18T19:05:24.114111+0000","last_became_active":"2024-03-18T19:05:22.594972+0000","last_became_peered":"2024-03-18T19:05:22.594972+0000","last_unstale":"2024-03-18T19:05:24.114111+0000","last_undegraded":"2024-03-18T19:05:24.114111+0000","last_fullsized":"2024-03-18T19:05:24.114111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:46.862070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597728+0000","last_change":"2024-03-18T19:05:22.593273+0000","last_active":"2024-03-18T19:05:24.597728+0000","last_peered":"2024-03-18T19:05:24.597728+0000","last_clean":"2024-03-18T19:05:24.597728+0000","last_became_active":"2024-03-18T19:05:22.592982+0000","last_became_peered":"2024-03-18T19:05:22.592982+0000","last_unstale":"2024-03-18T19:05:24.597728+0000","last_undegraded":"2024-03-18T19:05:24.597728+0000","last_fullsized":"2024-03-18T19:05:24.597728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:18:55.815832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.113960+0000","last_change":"2024-03-18T19:05:22.607654+0000","last_active":"2024-03-18T19:05:24.113960+0000","last_peered":"2024-03-18T19:05:24.113960+0000","last_clean":"2024-03-18T19:05:24.113960+0000","last_became_active":"2024-03-18T19:05:22.607397+0000","last_became_peered":"2024-03-18T19:05:22.607397+0000","last_unstale":"2024-03-18T19:05:24.113960+0000","last_undegraded":"2024-03-18T19:05:24.113960+0000","last_fullsized":"2024-03-18T19:05:24.113960+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:09:57.526166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.113905+0000","last_change":"2024-03-18T19:05:22.606436+0000","last_active":"2024-03-18T19:05:24.113905+0000","last_peered":"2024-03-18T19:05:24.113905+0000","last_clean":"2024-03-18T19:05:24.113905+0000","last_became_active":"2024-03-18T19:05:22.606176+0000","last_became_peered":"2024-03-18T19:05:22.606176+0000","last_unstale":"2024-03-18T19:05:24.113905+0000","last_undegraded":"2024-03-18T19:05:24.113905+0000","last_fullsized":"2024-03-18T19:05:24.113905+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:26:33.270251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.608773+0000","last_change":"2024-03-18T19:05:22.605839+0000","last_active":"2024-03-18T19:05:22.608773+0000","last_peered":"2024-03-18T19:05:22.608773+0000","last_clean":"2024-03-18T19:05:22.608773+0000","last_became_active":"2024-03-18T19:05:22.605564+0000","last_became_peered":"2024-03-18T19:05:22.605564+0000","last_unstale":"2024-03-18T19:05:22.608773+0000","last_undegraded":"2024-03-18T19:05:22.608773+0000","last_fullsized":"2024-03-18T19:05:22.608773+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:52.820873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597548+0000","last_change":"2024-03-18T19:05:22.597398+0000","last_active":"2024-03-18T19:05:24.597548+0000","last_peered":"2024-03-18T19:05:24.597548+0000","last_clean":"2024-03-18T19:05:24.597548+0000","last_became_active":"2024-03-18T19:05:22.597115+0000","last_became_peered":"2024-03-18T19:05:22.597115+0000","last_unstale":"2024-03-18T19:05:24.597548+0000","last_undegraded":"2024-03-18T19:05:24.597548+0000","last_fullsized":"2024-03-18T19:05:24.597548+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:01:35.139881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597394+0000","last_change":"2024-03-18T19:05:22.594988+0000","last_active":"2024-03-18T19:05:24.597394+0000","last_peered":"2024-03-18T19:05:24.597394+0000","last_clean":"2024-03-18T19:05:24.597394+0000","last_became_active":"2024-03-18T19:05:22.594707+0000","last_became_peered":"2024-03-18T19:05:22.594707+0000","last_unstale":"2024-03-18T19:05:24.597394+0000","last_undegraded":"2024-03-18T19:05:24.597394+0000","last_fullsized":"2024-03-18T19:05:24.597394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:52.107864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966801+0000","last_change":"2024-03-18T19:05:22.596850+0000","last_active":"2024-03-18T19:05:24.966801+0000","last_peered":"2024-03-18T19:05:24.966801+0000","last_clean":"2024-03-18T19:05:24.966801+0000","last_became_active":"2024-03-18T19:05:22.596535+0000","last_became_peered":"2024-03-18T19:05:22.596535+0000","last_unstale":"2024-03-18T19:05:24.966801+0000","last_undegraded":"2024-03-18T19:05:24.966801+0000","last_fullsized":"2024-03-18T19:05:24.966801+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:52:53.706285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.592514+0000","last_change":"2024-03-18T19:05:22.589101+0000","last_active":"2024-03-18T19:05:22.592514+0000","last_peered":"2024-03-18T19:05:22.592514+0000","last_clean":"2024-03-18T19:05:22.592514+0000","last_became_active":"2024-03-18T19:05:22.588790+0000","last_became_peered":"2024-03-18T19:05:22.588790+0000","last_unstale":"2024-03-18T19:05:22.592514+0000","last_undegraded":"2024-03-18T19:05:22.592514+0000","last_fullsized":"2024-03-18T19:05:22.592514+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:52.107864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.613960+0000","last_change":"2024-03-18T19:05:22.613071+0000","last_active":"2024-03-18T19:05:22.613960+0000","last_peered":"2024-03-18T19:05:22.613960+0000","last_clean":"2024-03-18T19:05:22.613960+0000","last_became_active":"2024-03-18T19:05:22.612753+0000","last_became_peered":"2024-03-18T19:05:22.612753+0000","last_unstale":"2024-03-18T19:05:22.613960+0000","last_undegraded":"2024-03-18T19:05:22.613960+0000","last_fullsized":"2024-03-18T19:05:22.613960+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:01:35.139881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597274+0000","last_change":"2024-03-18T19:05:22.608027+0000","last_active":"2024-03-18T19:05:24.597274+0000","last_peered":"2024-03-18T19:05:24.597274+0000","last_clean":"2024-03-18T19:05:24.597274+0000","last_became_active":"2024-03-18T19:05:22.607725+0000","last_became_peered":"2024-03-18T19:05:22.607725+0000","last_unstale":"2024-03-18T19:05:24.597274+0000","last_undegraded":"2024-03-18T19:05:24.597274+0000","last_fullsized":"2024-03-18T19:05:24.597274+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:49:41.091622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966357+0000","last_change":"2024-03-18T19:05:22.609587+0000","last_active":"2024-03-18T19:05:24.966357+0000","last_peered":"2024-03-18T19:05:24.966357+0000","last_clean":"2024-03-18T19:05:24.966357+0000","last_became_active":"2024-03-18T19:05:22.609324+0000","last_became_peered":"2024-03-18T19:05:22.609324+0000","last_unstale":"2024-03-18T19:05:24.966357+0000","last_undegraded":"2024-03-18T19:05:24.966357+0000","last_fullsized":"2024-03-18T19:05:24.966357+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:21.799481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.601380+0000","last_change":"2024-03-18T19:05:22.592257+0000","last_active":"2024-03-18T19:05:22.601380+0000","last_peered":"2024-03-18T19:05:22.601380+0000","last_clean":"2024-03-18T19:05:22.601380+0000","last_became_active":"2024-03-18T19:05:22.591402+0000","last_became_peered":"2024-03-18T19:05:22.591402+0000","last_unstale":"2024-03-18T19:05:22.601380+0000","last_undegraded":"2024-03-18T19:05:22.601380+0000","last_fullsized":"2024-03-18T19:05:22.601380+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:31:20.342831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590621+0000","last_change":"2024-03-18T19:05:22.589035+0000","last_active":"2024-03-18T19:05:23.590621+0000","last_peered":"2024-03-18T19:05:23.590621+0000","last_clean":"2024-03-18T19:05:23.590621+0000","last_became_active":"2024-03-18T19:05:22.588710+0000","last_became_peered":"2024-03-18T19:05:22.588710+0000","last_unstale":"2024-03-18T19:05:23.590621+0000","last_undegraded":"2024-03-18T19:05:23.590621+0000","last_fullsized":"2024-03-18T19:05:23.590621+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:56:45.230445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"88'4","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.607703+0000","last_change":"2024-03-18T19:05:22.170977+0000","last_active":"2024-03-18T19:05:22.607703+0000","last_peered":"2024-03-18T19:05:22.607703+0000","last_clean":"2024-03-18T19:05:22.607703+0000","last_became_active":"2024-03-18T19:05:22.170471+0000","last_became_peered":"2024-03-18T19:05:22.170471+0000","last_unstale":"2024-03-18T19:05:22.607703+0000","last_undegraded":"2024-03-18T19:05:22.607703+0000","last_fullsized":"2024-03-18T19:05:22.607703+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:56:44.225942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966804+0000","last_change":"2024-03-18T19:05:22.598141+0000","last_active":"2024-03-18T19:05:24.966804+0000","last_peered":"2024-03-18T19:05:24.966804+0000","last_clean":"2024-03-18T19:05:24.966804+0000","last_became_active":"2024-03-18T19:05:22.597872+0000","last_became_peered":"2024-03-18T19:05:22.597872+0000","last_unstale":"2024-03-18T19:05:24.966804+0000","last_undegraded":"2024-03-18T19:05:24.966804+0000","last_fullsized":"2024-03-18T19:05:24.966804+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:03:13.170587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"88'1","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.606332+0000","last_change":"2024-03-18T19:05:22.170924+0000","last_active":"2024-03-18T19:05:22.606332+0000","last_peered":"2024-03-18T19:05:22.606332+0000","last_clean":"2024-03-18T19:05:22.606332+0000","last_became_active":"2024-03-18T19:05:22.170658+0000","last_became_peered":"2024-03-18T19:05:22.170658+0000","last_unstale":"2024-03-18T19:05:22.606332+0000","last_undegraded":"2024-03-18T19:05:22.606332+0000","last_fullsized":"2024-03-18T19:05:22.606332+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:10.178077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.594683+0000","last_change":"2024-03-18T19:05:22.592145+0000","last_active":"2024-03-18T19:05:22.594683+0000","last_peered":"2024-03-18T19:05:22.594683+0000","last_clean":"2024-03-18T19:05:22.594683+0000","last_became_active":"2024-03-18T19:05:22.591881+0000","last_became_peered":"2024-03-18T19:05:22.591881+0000","last_unstale":"2024-03-18T19:05:22.594683+0000","last_undegraded":"2024-03-18T19:05:22.594683+0000","last_fullsized":"2024-03-18T19:05:22.594683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:05:30.670583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"88'2","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.606902+0000","last_change":"2024-03-18T19:05:22.169943+0000","last_active":"2024-03-18T19:05:22.606902+0000","last_peered":"2024-03-18T19:05:22.606902+0000","last_clean":"2024-03-18T19:05:22.606902+0000","last_became_active":"2024-03-18T19:05:22.169557+0000","last_became_peered":"2024-03-18T19:05:22.169557+0000","last_unstale":"2024-03-18T19:05:22.606902+0000","last_undegraded":"2024-03-18T19:05:22.606902+0000","last_fullsized":"2024-03-18T19:05:22.606902+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:19:09.137260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.609310+0000","last_change":"2024-03-18T19:05:22.606259+0000","last_active":"2024-03-18T19:05:22.609310+0000","last_peered":"2024-03-18T19:05:22.609310+0000","last_clean":"2024-03-18T19:05:22.609310+0000","last_became_active":"2024-03-18T19:05:22.605873+0000","last_became_peered":"2024-03-18T19:05:22.605873+0000","last_unstale":"2024-03-18T19:05:22.609310+0000","last_undegraded":"2024-03-18T19:05:22.609310+0000","last_fullsized":"2024-03-18T19:05:22.609310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:59:06.740471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"88'2","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.601896+0000","last_change":"2024-03-18T19:05:22.168058+0000","last_active":"2024-03-18T19:05:22.601896+0000","last_peered":"2024-03-18T19:05:22.601896+0000","last_clean":"2024-03-18T19:05:22.601896+0000","last_became_active":"2024-03-18T19:05:22.167785+0000","last_became_peered":"2024-03-18T19:05:22.167785+0000","last_unstale":"2024-03-18T19:05:22.601896+0000","last_undegraded":"2024-03-18T19:05:22.601896+0000","last_fullsized":"2024-03-18T19:05:22.601896+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:20:34.274975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.591155+0000","last_change":"2024-03-18T19:05:22.589947+0000","last_active":"2024-03-18T19:05:23.591155+0000","last_peered":"2024-03-18T19:05:23.591155+0000","last_clean":"2024-03-18T19:05:23.591155+0000","last_became_active":"2024-03-18T19:05:22.589147+0000","last_became_peered":"2024-03-18T19:05:22.589147+0000","last_unstale":"2024-03-18T19:05:23.591155+0000","last_undegraded":"2024-03-18T19:05:23.591155+0000","last_fullsized":"2024-03-18T19:05:23.591155+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:21.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"88'1","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.114061+0000","last_change":"2024-03-18T19:05:22.171498+0000","last_active":"2024-03-18T19:05:24.114061+0000","last_peered":"2024-03-18T19:05:24.114061+0000","last_clean":"2024-03-18T19:05:24.114061+0000","last_became_active":"2024-03-18T19:05:22.171174+0000","last_became_peered":"2024-03-18T19:05:22.171174+0000","last_unstale":"2024-03-18T19:05:24.114061+0000","last_undegraded":"2024-03-18T19:05:24.114061+0000","last_fullsized":"2024-03-18T19:05:24.114061+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:36:30.649705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.611062+0000","last_change":"2024-03-18T19:05:22.610629+0000","last_active":"2024-03-18T19:05:22.611062+0000","last_peered":"2024-03-18T19:05:22.611062+0000","last_clean":"2024-03-18T19:05:22.611062+0000","last_became_active":"2024-03-18T19:05:22.610450+0000","last_became_peered":"2024-03-18T19:05:22.610450+0000","last_unstale":"2024-03-18T19:05:22.611062+0000","last_undegraded":"2024-03-18T19:05:22.611062+0000","last_fullsized":"2024-03-18T19:05:22.611062+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:18:55.815832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"88'1","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.113909+0000","last_change":"2024-03-18T19:05:22.166498+0000","last_active":"2024-03-18T19:05:24.113909+0000","last_peered":"2024-03-18T19:05:24.113909+0000","last_clean":"2024-03-18T19:05:24.113909+0000","last_became_active":"2024-03-18T19:05:22.166236+0000","last_became_peered":"2024-03-18T19:05:22.166236+0000","last_unstale":"2024-03-18T19:05:24.113909+0000","last_undegraded":"2024-03-18T19:05:24.113909+0000","last_fullsized":"2024-03-18T19:05:24.113909+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:10.178077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590437+0000","last_change":"2024-03-18T19:05:22.593776+0000","last_active":"2024-03-18T19:05:23.590437+0000","last_peered":"2024-03-18T19:05:23.590437+0000","last_clean":"2024-03-18T19:05:23.590437+0000","last_became_active":"2024-03-18T19:05:22.593516+0000","last_became_peered":"2024-03-18T19:05:22.593516+0000","last_unstale":"2024-03-18T19:05:23.590437+0000","last_undegraded":"2024-03-18T19:05:23.590437+0000","last_fullsized":"2024-03-18T19:05:23.590437+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:21:54.604175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.114037+0000","last_change":"2024-03-18T19:05:22.165942+0000","last_active":"2024-03-18T19:05:24.114037+0000","last_peered":"2024-03-18T19:05:24.114037+0000","last_clean":"2024-03-18T19:05:24.114037+0000","last_became_active":"2024-03-18T19:05:22.165456+0000","last_became_peered":"2024-03-18T19:05:22.165456+0000","last_unstale":"2024-03-18T19:05:24.114037+0000","last_undegraded":"2024-03-18T19:05:24.114037+0000","last_fullsized":"2024-03-18T19:05:24.114037+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:55:38.621035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.114099+0000","last_change":"2024-03-18T19:05:22.607465+0000","last_active":"2024-03-18T19:05:24.114099+0000","last_peered":"2024-03-18T19:05:24.114099+0000","last_clean":"2024-03-18T19:05:24.114099+0000","last_became_active":"2024-03-18T19:05:22.607217+0000","last_became_peered":"2024-03-18T19:05:22.607217+0000","last_unstale":"2024-03-18T19:05:24.114099+0000","last_undegraded":"2024-03-18T19:05:24.114099+0000","last_fullsized":"2024-03-18T19:05:24.114099+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:28.449312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590369+0000","last_change":"2024-03-18T19:05:22.170069+0000","last_active":"2024-03-18T19:05:23.590369+0000","last_peered":"2024-03-18T19:05:23.590369+0000","last_clean":"2024-03-18T19:05:23.590369+0000","last_became_active":"2024-03-18T19:05:22.169480+0000","last_became_peered":"2024-03-18T19:05:22.169480+0000","last_unstale":"2024-03-18T19:05:23.590369+0000","last_undegraded":"2024-03-18T19:05:23.590369+0000","last_fullsized":"2024-03-18T19:05:23.590369+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:05:33.245506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.588034+0000","last_change":"2024-03-18T19:05:22.587407+0000","last_active":"2024-03-18T19:05:22.588034+0000","last_peered":"2024-03-18T19:05:22.588034+0000","last_clean":"2024-03-18T19:05:22.588034+0000","last_became_active":"2024-03-18T19:05:22.587142+0000","last_became_peered":"2024-03-18T19:05:22.587142+0000","last_unstale":"2024-03-18T19:05:22.588034+0000","last_undegraded":"2024-03-18T19:05:22.588034+0000","last_fullsized":"2024-03-18T19:05:22.588034+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:49:41.091622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.597935+0000","last_change":"2024-03-18T19:05:22.168515+0000","last_active":"2024-03-18T19:05:22.597935+0000","last_peered":"2024-03-18T19:05:22.597935+0000","last_clean":"2024-03-18T19:05:22.597935+0000","last_became_active":"2024-03-18T19:05:22.168231+0000","last_became_peered":"2024-03-18T19:05:22.168231+0000","last_unstale":"2024-03-18T19:05:22.597935+0000","last_undegraded":"2024-03-18T19:05:22.597935+0000","last_fullsized":"2024-03-18T19:05:22.597935+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:03:12.166084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.114182+0000","last_change":"2024-03-18T19:05:22.595294+0000","last_active":"2024-03-18T19:05:24.114182+0000","last_peered":"2024-03-18T19:05:24.114182+0000","last_clean":"2024-03-18T19:05:24.114182+0000","last_became_active":"2024-03-18T19:05:22.595068+0000","last_became_peered":"2024-03-18T19:05:22.595068+0000","last_unstale":"2024-03-18T19:05:24.114182+0000","last_undegraded":"2024-03-18T19:05:24.114182+0000","last_fullsized":"2024-03-18T19:05:24.114182+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:29:34.389598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966715+0000","last_change":"2024-03-18T19:05:22.175943+0000","last_active":"2024-03-18T19:05:24.966715+0000","last_peered":"2024-03-18T19:05:24.966715+0000","last_clean":"2024-03-18T19:05:24.966715+0000","last_became_active":"2024-03-18T19:05:22.175678+0000","last_became_peered":"2024-03-18T19:05:22.175678+0000","last_unstale":"2024-03-18T19:05:24.966715+0000","last_undegraded":"2024-03-18T19:05:24.966715+0000","last_fullsized":"2024-03-18T19:05:24.966715+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:26:03.949433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590745+0000","last_change":"2024-03-18T19:05:22.595869+0000","last_active":"2024-03-18T19:05:23.590745+0000","last_peered":"2024-03-18T19:05:23.590745+0000","last_clean":"2024-03-18T19:05:23.590745+0000","last_became_active":"2024-03-18T19:05:22.595625+0000","last_became_peered":"2024-03-18T19:05:22.595625+0000","last_unstale":"2024-03-18T19:05:23.590745+0000","last_undegraded":"2024-03-18T19:05:23.590745+0000","last_fullsized":"2024-03-18T19:05:23.590745+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:43.837454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"88'3","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.607281+0000","last_change":"2024-03-18T19:05:22.168466+0000","last_active":"2024-03-18T19:05:22.607281+0000","last_peered":"2024-03-18T19:05:22.607281+0000","last_clean":"2024-03-18T19:05:22.607281+0000","last_became_active":"2024-03-18T19:05:22.168178+0000","last_became_peered":"2024-03-18T19:05:22.168178+0000","last_unstale":"2024-03-18T19:05:22.607281+0000","last_undegraded":"2024-03-18T19:05:22.607281+0000","last_fullsized":"2024-03-18T19:05:22.607281+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:25:20.727978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966790+0000","last_change":"2024-03-18T19:05:22.606106+0000","last_active":"2024-03-18T19:05:24.966790+0000","last_peered":"2024-03-18T19:05:24.966790+0000","last_clean":"2024-03-18T19:05:24.966790+0000","last_became_active":"2024-03-18T19:05:22.605868+0000","last_became_peered":"2024-03-18T19:05:22.605868+0000","last_unstale":"2024-03-18T19:05:24.966790+0000","last_undegraded":"2024-03-18T19:05:24.966790+0000","last_fullsized":"2024-03-18T19:05:24.966790+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:25:21.732481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"88'6","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.615880+0000","last_change":"2024-03-18T19:05:22.176269+0000","last_active":"2024-03-18T19:05:22.615880+0000","last_peered":"2024-03-18T19:05:22.615880+0000","last_clean":"2024-03-18T19:05:22.615880+0000","last_became_active":"2024-03-18T19:05:22.175938+0000","last_became_peered":"2024-03-18T19:05:22.175938+0000","last_unstale":"2024-03-18T19:05:22.615880+0000","last_undegraded":"2024-03-18T19:05:22.615880+0000","last_fullsized":"2024-03-18T19:05:22.615880+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:33:27.744462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966328+0000","last_change":"2024-03-18T19:05:22.606043+0000","last_active":"2024-03-18T19:05:24.966328+0000","last_peered":"2024-03-18T19:05:24.966328+0000","last_clean":"2024-03-18T19:05:24.966328+0000","last_became_active":"2024-03-18T19:05:22.605607+0000","last_became_peered":"2024-03-18T19:05:22.605607+0000","last_unstale":"2024-03-18T19:05:24.966328+0000","last_undegraded":"2024-03-18T19:05:24.966328+0000","last_fullsized":"2024-03-18T19:05:24.966328+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:15:54.617487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"88'2","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.606059+0000","last_change":"2024-03-18T19:05:22.165089+0000","last_active":"2024-03-18T19:05:22.606059+0000","last_peered":"2024-03-18T19:05:22.606059+0000","last_clean":"2024-03-18T19:05:22.606059+0000","last_became_active":"2024-03-18T19:05:22.164835+0000","last_became_peered":"2024-03-18T19:05:22.164835+0000","last_unstale":"2024-03-18T19:05:22.606059+0000","last_undegraded":"2024-03-18T19:05:22.606059+0000","last_fullsized":"2024-03-18T19:05:22.606059+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:21:40.562313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597194+0000","last_change":"2024-03-18T19:05:22.594119+0000","last_active":"2024-03-18T19:05:24.597194+0000","last_peered":"2024-03-18T19:05:24.597194+0000","last_clean":"2024-03-18T19:05:24.597194+0000","last_became_active":"2024-03-18T19:05:22.593866+0000","last_became_peered":"2024-03-18T19:05:22.593866+0000","last_unstale":"2024-03-18T19:05:24.597194+0000","last_undegraded":"2024-03-18T19:05:24.597194+0000","last_fullsized":"2024-03-18T19:05:24.597194+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:08:31.593720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597469+0000","last_change":"2024-03-18T19:05:22.173033+0000","last_active":"2024-03-18T19:05:24.597469+0000","last_peered":"2024-03-18T19:05:24.597469+0000","last_clean":"2024-03-18T19:05:24.597469+0000","last_became_active":"2024-03-18T19:05:22.172832+0000","last_became_peered":"2024-03-18T19:05:22.172832+0000","last_unstale":"2024-03-18T19:05:24.597469+0000","last_undegraded":"2024-03-18T19:05:24.597469+0000","last_fullsized":"2024-03-18T19:05:24.597469+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:25:20.727978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590544+0000","last_change":"2024-03-18T19:05:22.590595+0000","last_active":"2024-03-18T19:05:23.590544+0000","last_peered":"2024-03-18T19:05:23.590544+0000","last_clean":"2024-03-18T19:05:23.590544+0000","last_became_active":"2024-03-18T19:05:22.590149+0000","last_became_peered":"2024-03-18T19:05:22.590149+0000","last_unstale":"2024-03-18T19:05:23.590544+0000","last_undegraded":"2024-03-18T19:05:23.590544+0000","last_fullsized":"2024-03-18T19:05:23.590544+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:03:13.170587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.599394+0000","last_change":"2024-03-18T19:05:22.167212+0000","last_active":"2024-03-18T19:05:22.599394+0000","last_peered":"2024-03-18T19:05:22.599394+0000","last_clean":"2024-03-18T19:05:22.599394+0000","last_became_active":"2024-03-18T19:05:22.166166+0000","last_became_peered":"2024-03-18T19:05:22.166166+0000","last_unstale":"2024-03-18T19:05:22.599394+0000","last_undegraded":"2024-03-18T19:05:22.599394+0000","last_fullsized":"2024-03-18T19:05:22.599394+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:26:58.281923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.604410+0000","last_change":"2024-03-18T19:05:22.603847+0000","last_active":"2024-03-18T19:05:22.604410+0000","last_peered":"2024-03-18T19:05:22.604410+0000","last_clean":"2024-03-18T19:05:22.604410+0000","last_became_active":"2024-03-18T19:05:22.603294+0000","last_became_peered":"2024-03-18T19:05:22.603294+0000","last_unstale":"2024-03-18T19:05:22.604410+0000","last_undegraded":"2024-03-18T19:05:22.604410+0000","last_fullsized":"2024-03-18T19:05:22.604410+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:59:06.740471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"88'8","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966329+0000","last_change":"2024-03-18T19:05:22.176028+0000","last_active":"2024-03-18T19:05:24.966329+0000","last_peered":"2024-03-18T19:05:24.966329+0000","last_clean":"2024-03-18T19:05:24.966329+0000","last_became_active":"2024-03-18T19:05:22.175785+0000","last_became_peered":"2024-03-18T19:05:22.175785+0000","last_unstale":"2024-03-18T19:05:24.966329+0000","last_undegraded":"2024-03-18T19:05:24.966329+0000","last_fullsized":"2024-03-18T19:05:24.966329+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:40.241549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.613652+0000","last_change":"2024-03-18T19:05:22.612982+0000","last_active":"2024-03-18T19:05:22.613652+0000","last_peered":"2024-03-18T19:05:22.613652+0000","last_clean":"2024-03-18T19:05:22.613652+0000","last_became_active":"2024-03-18T19:05:22.612719+0000","last_became_peered":"2024-03-18T19:05:22.612719+0000","last_unstale":"2024-03-18T19:05:22.613652+0000","last_undegraded":"2024-03-18T19:05:22.613652+0000","last_fullsized":"2024-03-18T19:05:22.613652+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:56:09.533605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"88'1","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597558+0000","last_change":"2024-03-18T19:05:22.174068+0000","last_active":"2024-03-18T19:05:24.597558+0000","last_peered":"2024-03-18T19:05:24.597558+0000","last_clean":"2024-03-18T19:05:24.597558+0000","last_became_active":"2024-03-18T19:05:22.173815+0000","last_became_peered":"2024-03-18T19:05:22.173815+0000","last_unstale":"2024-03-18T19:05:24.597558+0000","last_undegraded":"2024-03-18T19:05:24.597558+0000","last_fullsized":"2024-03-18T19:05:24.597558+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:26:58.281923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590810+0000","last_change":"2024-03-18T19:05:22.610966+0000","last_active":"2024-03-18T19:05:23.590810+0000","last_peered":"2024-03-18T19:05:23.590810+0000","last_clean":"2024-03-18T19:05:23.590810+0000","last_became_active":"2024-03-18T19:05:22.610709+0000","last_became_peered":"2024-03-18T19:05:22.610709+0000","last_unstale":"2024-03-18T19:05:23.590810+0000","last_undegraded":"2024-03-18T19:05:23.590810+0000","last_fullsized":"2024-03-18T19:05:23.590810+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:49:03.493777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"88'2","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.613651+0000","last_change":"2024-03-18T19:05:22.174968+0000","last_active":"2024-03-18T19:05:22.613651+0000","last_peered":"2024-03-18T19:05:22.613651+0000","last_clean":"2024-03-18T19:05:22.613651+0000","last_became_active":"2024-03-18T19:05:22.174099+0000","last_became_peered":"2024-03-18T19:05:22.174099+0000","last_unstale":"2024-03-18T19:05:22.613651+0000","last_undegraded":"2024-03-18T19:05:22.613651+0000","last_fullsized":"2024-03-18T19:05:22.613651+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:52:03.908127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966686+0000","last_change":"2024-03-18T19:05:22.606297+0000","last_active":"2024-03-18T19:05:24.966686+0000","last_peered":"2024-03-18T19:05:24.966686+0000","last_clean":"2024-03-18T19:05:24.966686+0000","last_became_active":"2024-03-18T19:05:22.600182+0000","last_became_peered":"2024-03-18T19:05:22.600182+0000","last_unstale":"2024-03-18T19:05:24.966686+0000","last_undegraded":"2024-03-18T19:05:24.966686+0000","last_fullsized":"2024-03-18T19:05:24.966686+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:28.026254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"88'1","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.607596+0000","last_change":"2024-03-18T19:05:22.172066+0000","last_active":"2024-03-18T19:05:22.607596+0000","last_peered":"2024-03-18T19:05:22.607596+0000","last_clean":"2024-03-18T19:05:22.607596+0000","last_became_active":"2024-03-18T19:05:22.171775+0000","last_became_peered":"2024-03-18T19:05:22.171775+0000","last_unstale":"2024-03-18T19:05:22.607596+0000","last_undegraded":"2024-03-18T19:05:22.607596+0000","last_fullsized":"2024-03-18T19:05:22.607596+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:56:44.225942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590783+0000","last_change":"2024-03-18T19:05:22.594727+0000","last_active":"2024-03-18T19:05:23.590783+0000","last_peered":"2024-03-18T19:05:23.590783+0000","last_clean":"2024-03-18T19:05:23.590783+0000","last_became_active":"2024-03-18T19:05:22.594490+0000","last_became_peered":"2024-03-18T19:05:22.594490+0000","last_unstale":"2024-03-18T19:05:23.590783+0000","last_undegraded":"2024-03-18T19:05:23.590783+0000","last_fullsized":"2024-03-18T19:05:23.590783+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:26:59.286426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.114156+0000","last_change":"2024-03-18T19:05:22.172962+0000","last_active":"2024-03-18T19:05:24.114156+0000","last_peered":"2024-03-18T19:05:24.114156+0000","last_clean":"2024-03-18T19:05:24.114156+0000","last_became_active":"2024-03-18T19:05:22.172666+0000","last_became_peered":"2024-03-18T19:05:22.172666+0000","last_unstale":"2024-03-18T19:05:24.114156+0000","last_undegraded":"2024-03-18T19:05:24.114156+0000","last_fullsized":"2024-03-18T19:05:24.114156+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:43:39.238232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.601402+0000","last_change":"2024-03-18T19:05:22.593614+0000","last_active":"2024-03-18T19:05:22.601402+0000","last_peered":"2024-03-18T19:05:22.601402+0000","last_clean":"2024-03-18T19:05:22.601402+0000","last_became_active":"2024-03-18T19:05:22.593316+0000","last_became_peered":"2024-03-18T19:05:22.593316+0000","last_unstale":"2024-03-18T19:05:22.601402+0000","last_undegraded":"2024-03-18T19:05:22.601402+0000","last_fullsized":"2024-03-18T19:05:22.601402+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:01:06.391760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"88'2","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.604723+0000","last_change":"2024-03-18T19:05:22.165965+0000","last_active":"2024-03-18T19:05:22.604723+0000","last_peered":"2024-03-18T19:05:22.604723+0000","last_clean":"2024-03-18T19:05:22.604723+0000","last_became_active":"2024-03-18T19:05:22.165682+0000","last_became_peered":"2024-03-18T19:05:22.165682+0000","last_unstale":"2024-03-18T19:05:22.604723+0000","last_undegraded":"2024-03-18T19:05:22.604723+0000","last_fullsized":"2024-03-18T19:05:22.604723+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:13:23.479719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597337+0000","last_change":"2024-03-18T19:05:22.594957+0000","last_active":"2024-03-18T19:05:24.597337+0000","last_peered":"2024-03-18T19:05:24.597337+0000","last_clean":"2024-03-18T19:05:24.597337+0000","last_became_active":"2024-03-18T19:05:22.594718+0000","last_became_peered":"2024-03-18T19:05:22.594718+0000","last_unstale":"2024-03-18T19:05:24.597337+0000","last_undegraded":"2024-03-18T19:05:24.597337+0000","last_fullsized":"2024-03-18T19:05:24.597337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:59:06.740471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597427+0000","last_change":"2024-03-18T19:05:22.174003+0000","last_active":"2024-03-18T19:05:24.597427+0000","last_peered":"2024-03-18T19:05:24.597427+0000","last_clean":"2024-03-18T19:05:24.597427+0000","last_became_active":"2024-03-18T19:05:22.173175+0000","last_became_peered":"2024-03-18T19:05:22.173175+0000","last_unstale":"2024-03-18T19:05:24.597427+0000","last_undegraded":"2024-03-18T19:05:24.597427+0000","last_fullsized":"2024-03-18T19:05:24.597427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:39:26.510910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590649+0000","last_change":"2024-03-18T19:05:22.590787+0000","last_active":"2024-03-18T19:05:23.590649+0000","last_peered":"2024-03-18T19:05:23.590649+0000","last_clean":"2024-03-18T19:05:23.590649+0000","last_became_active":"2024-03-18T19:05:22.590521+0000","last_became_peered":"2024-03-18T19:05:22.590521+0000","last_unstale":"2024-03-18T19:05:23.590649+0000","last_undegraded":"2024-03-18T19:05:23.590649+0000","last_fullsized":"2024-03-18T19:05:23.590649+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:33:28.748965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.589412+0000","last_change":"2024-03-18T19:05:22.170398+0000","last_active":"2024-03-18T19:05:22.589412+0000","last_peered":"2024-03-18T19:05:22.589412+0000","last_clean":"2024-03-18T19:05:22.589412+0000","last_became_active":"2024-03-18T19:05:22.170144+0000","last_became_peered":"2024-03-18T19:05:22.170144+0000","last_unstale":"2024-03-18T19:05:22.589412+0000","last_undegraded":"2024-03-18T19:05:22.589412+0000","last_fullsized":"2024-03-18T19:05:22.589412+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:09:36.470377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.612646+0000","last_change":"2024-03-18T19:05:22.612364+0000","last_active":"2024-03-18T19:05:22.612646+0000","last_peered":"2024-03-18T19:05:22.612646+0000","last_clean":"2024-03-18T19:05:22.612646+0000","last_became_active":"2024-03-18T19:05:22.612180+0000","last_became_peered":"2024-03-18T19:05:22.612180+0000","last_unstale":"2024-03-18T19:05:22.612646+0000","last_undegraded":"2024-03-18T19:05:22.612646+0000","last_fullsized":"2024-03-18T19:05:22.612646+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:09:35.355553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"88'2","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597644+0000","last_change":"2024-03-18T19:05:22.170451+0000","last_active":"2024-03-18T19:05:24.597644+0000","last_peered":"2024-03-18T19:05:24.597644+0000","last_clean":"2024-03-18T19:05:24.597644+0000","last_became_active":"2024-03-18T19:05:22.169918+0000","last_became_peered":"2024-03-18T19:05:22.169918+0000","last_unstale":"2024-03-18T19:05:24.597644+0000","last_undegraded":"2024-03-18T19:05:24.597644+0000","last_fullsized":"2024-03-18T19:05:24.597644+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:21:40.562313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.591224+0000","last_change":"2024-03-18T19:05:22.590697+0000","last_active":"2024-03-18T19:05:23.591224+0000","last_peered":"2024-03-18T19:05:23.591224+0000","last_clean":"2024-03-18T19:05:23.591224+0000","last_became_active":"2024-03-18T19:05:22.590058+0000","last_became_peered":"2024-03-18T19:05:22.590058+0000","last_unstale":"2024-03-18T19:05:23.591224+0000","last_undegraded":"2024-03-18T19:05:23.591224+0000","last_fullsized":"2024-03-18T19:05:23.591224+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:12:45.172990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597566+0000","last_change":"2024-03-18T19:05:22.172938+0000","last_active":"2024-03-18T19:05:24.597566+0000","last_peered":"2024-03-18T19:05:24.597566+0000","last_clean":"2024-03-18T19:05:24.597566+0000","last_became_active":"2024-03-18T19:05:22.172638+0000","last_became_peered":"2024-03-18T19:05:22.172638+0000","last_unstale":"2024-03-18T19:05:24.597566+0000","last_undegraded":"2024-03-18T19:05:24.597566+0000","last_fullsized":"2024-03-18T19:05:24.597566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:52:03.908127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597530+0000","last_change":"2024-03-18T19:05:22.595426+0000","last_active":"2024-03-18T19:05:24.597530+0000","last_peered":"2024-03-18T19:05:24.597530+0000","last_clean":"2024-03-18T19:05:24.597530+0000","last_became_active":"2024-03-18T19:05:22.595167+0000","last_became_peered":"2024-03-18T19:05:22.595167+0000","last_unstale":"2024-03-18T19:05:24.597530+0000","last_undegraded":"2024-03-18T19:05:24.597530+0000","last_fullsized":"2024-03-18T19:05:24.597530+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:52:53.706285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.587723+0000","last_change":"2024-03-18T19:05:22.169731+0000","last_active":"2024-03-18T19:05:22.587723+0000","last_peered":"2024-03-18T19:05:22.587723+0000","last_clean":"2024-03-18T19:05:22.587723+0000","last_became_active":"2024-03-18T19:05:22.169456+0000","last_became_peered":"2024-03-18T19:05:22.169456+0000","last_unstale":"2024-03-18T19:05:22.587723+0000","last_undegraded":"2024-03-18T19:05:22.587723+0000","last_fullsized":"2024-03-18T19:05:22.587723+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:07:00.608366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590850+0000","last_change":"2024-03-18T19:05:22.591088+0000","last_active":"2024-03-18T19:05:23.590850+0000","last_peered":"2024-03-18T19:05:23.590850+0000","last_clean":"2024-03-18T19:05:23.590850+0000","last_became_active":"2024-03-18T19:05:22.588674+0000","last_became_peered":"2024-03-18T19:05:22.588674+0000","last_unstale":"2024-03-18T19:05:23.590850+0000","last_undegraded":"2024-03-18T19:05:23.590850+0000","last_fullsized":"2024-03-18T19:05:23.590850+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:52:04.912630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.584908+0000","last_change":"2024-03-18T19:05:22.174819+0000","last_active":"2024-03-18T19:05:22.584908+0000","last_peered":"2024-03-18T19:05:22.584908+0000","last_clean":"2024-03-18T19:05:22.584908+0000","last_became_active":"2024-03-18T19:05:22.174531+0000","last_became_peered":"2024-03-18T19:05:22.174531+0000","last_unstale":"2024-03-18T19:05:22.584908+0000","last_undegraded":"2024-03-18T19:05:22.584908+0000","last_fullsized":"2024-03-18T19:05:22.584908+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:04:07.797394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590313+0000","last_change":"2024-03-18T19:05:22.588932+0000","last_active":"2024-03-18T19:05:23.590313+0000","last_peered":"2024-03-18T19:05:23.590313+0000","last_clean":"2024-03-18T19:05:23.590313+0000","last_became_active":"2024-03-18T19:05:22.588624+0000","last_became_peered":"2024-03-18T19:05:22.588624+0000","last_unstale":"2024-03-18T19:05:23.590313+0000","last_undegraded":"2024-03-18T19:05:23.590313+0000","last_fullsized":"2024-03-18T19:05:23.590313+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:25:21.732481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.595892+0000","last_change":"2024-03-18T19:05:22.166252+0000","last_active":"2024-03-18T19:05:22.595892+0000","last_peered":"2024-03-18T19:05:22.595892+0000","last_clean":"2024-03-18T19:05:22.595892+0000","last_became_active":"2024-03-18T19:05:22.165915+0000","last_became_peered":"2024-03-18T19:05:22.165915+0000","last_unstale":"2024-03-18T19:05:22.595892+0000","last_undegraded":"2024-03-18T19:05:22.595892+0000","last_fullsized":"2024-03-18T19:05:22.595892+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:04:07.797394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966440+0000","last_change":"2024-03-18T19:05:22.610861+0000","last_active":"2024-03-18T19:05:24.966440+0000","last_peered":"2024-03-18T19:05:24.966440+0000","last_clean":"2024-03-18T19:05:24.966440+0000","last_became_active":"2024-03-18T19:05:22.609566+0000","last_became_peered":"2024-03-18T19:05:22.609566+0000","last_unstale":"2024-03-18T19:05:24.966440+0000","last_undegraded":"2024-03-18T19:05:24.966440+0000","last_fullsized":"2024-03-18T19:05:24.966440+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:28.990836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.114023+0000","last_change":"2024-03-18T19:05:22.167040+0000","last_active":"2024-03-18T19:05:24.114023+0000","last_peered":"2024-03-18T19:05:24.114023+0000","last_clean":"2024-03-18T19:05:24.114023+0000","last_became_active":"2024-03-18T19:05:22.166772+0000","last_became_peered":"2024-03-18T19:05:22.166772+0000","last_unstale":"2024-03-18T19:05:24.114023+0000","last_undegraded":"2024-03-18T19:05:24.114023+0000","last_fullsized":"2024-03-18T19:05:24.114023+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:52:11.707326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.611040+0000","last_change":"2024-03-18T19:05:22.610497+0000","last_active":"2024-03-18T19:05:22.611040+0000","last_peered":"2024-03-18T19:05:22.611040+0000","last_clean":"2024-03-18T19:05:22.611040+0000","last_became_active":"2024-03-18T19:05:22.610210+0000","last_became_peered":"2024-03-18T19:05:22.610210+0000","last_unstale":"2024-03-18T19:05:22.611040+0000","last_undegraded":"2024-03-18T19:05:22.611040+0000","last_fullsized":"2024-03-18T19:05:22.611040+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:01:35.139881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.114053+0000","last_change":"2024-03-18T19:05:22.165747+0000","last_active":"2024-03-18T19:05:24.114053+0000","last_peered":"2024-03-18T19:05:24.114053+0000","last_clean":"2024-03-18T19:05:24.114053+0000","last_became_active":"2024-03-18T19:05:22.165072+0000","last_became_peered":"2024-03-18T19:05:22.165072+0000","last_unstale":"2024-03-18T19:05:24.114053+0000","last_undegraded":"2024-03-18T19:05:24.114053+0000","last_fullsized":"2024-03-18T19:05:24.114053+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:08:04.865734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.591066+0000","last_change":"2024-03-18T19:05:22.594091+0000","last_active":"2024-03-18T19:05:23.591066+0000","last_peered":"2024-03-18T19:05:23.591066+0000","last_clean":"2024-03-18T19:05:23.591066+0000","last_became_active":"2024-03-18T19:05:22.593715+0000","last_became_peered":"2024-03-18T19:05:22.593715+0000","last_unstale":"2024-03-18T19:05:23.591066+0000","last_undegraded":"2024-03-18T19:05:23.591066+0000","last_fullsized":"2024-03-18T19:05:23.591066+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:05:47.646282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.584317+0000","last_change":"2024-03-18T19:05:22.166661+0000","last_active":"2024-03-18T19:05:22.584317+0000","last_peered":"2024-03-18T19:05:22.584317+0000","last_clean":"2024-03-18T19:05:22.584317+0000","last_became_active":"2024-03-18T19:05:22.166368+0000","last_became_peered":"2024-03-18T19:05:22.166368+0000","last_unstale":"2024-03-18T19:05:22.584317+0000","last_undegraded":"2024-03-18T19:05:22.584317+0000","last_fullsized":"2024-03-18T19:05:22.584317+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:21:53.599672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590607+0000","last_change":"2024-03-18T19:05:22.589062+0000","last_active":"2024-03-18T19:05:23.590607+0000","last_peered":"2024-03-18T19:05:23.590607+0000","last_clean":"2024-03-18T19:05:23.590607+0000","last_became_active":"2024-03-18T19:05:22.588764+0000","last_became_peered":"2024-03-18T19:05:22.588764+0000","last_unstale":"2024-03-18T19:05:23.590607+0000","last_undegraded":"2024-03-18T19:05:23.590607+0000","last_fullsized":"2024-03-18T19:05:23.590607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:08:31.593720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.113904+0000","last_change":"2024-03-18T19:05:22.163773+0000","last_active":"2024-03-18T19:05:24.113904+0000","last_peered":"2024-03-18T19:05:24.113904+0000","last_clean":"2024-03-18T19:05:24.113904+0000","last_became_active":"2024-03-18T19:05:22.163466+0000","last_became_peered":"2024-03-18T19:05:22.163466+0000","last_unstale":"2024-03-18T19:05:24.113904+0000","last_undegraded":"2024-03-18T19:05:24.113904+0000","last_fullsized":"2024-03-18T19:05:24.113904+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:19:38.888470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"78'161","reported_seq":384,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.153788+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T19:05:22.153788+0000","last_peered":"2024-03-18T19:05:22.153788+0000","last_clean":"2024-03-18T19:05:22.153788+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T19:05:22.153788+0000","last_undegraded":"2024-03-18T19:05:22.153788+0000","last_fullsized":"2024-03-18T19:05:22.153788+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58996,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312844,"statfs":{"total":96636764160,"available":96576352256,"internally_reserved":0,"allocated":1679360,"data_stored":1123514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.57699999999999996,"15min":0.56999999999999995},"min":{"1min":0.44800000000000001,"5min":0.27800000000000002,"15min":0.247},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.57899999999999996,"15min":0.58199999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.26700000000000002},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.58499999999999996}]},{"osd":1,"last update":"Mon Mar 18 19:05:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.56299999999999994,"15min":0.57299999999999995},"min":{"1min":0.46500000000000002,"5min":0.32300000000000001,"15min":0.307},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":1.484},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41499999999999998,"5min":0.378,"15min":0.36599999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.92000000000000004},"last":0.56100000000000005}]},{"osd":2,"last update":"Mon Mar 18 19:05:04 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.56499999999999995,"15min":0.57799999999999996},"min":{"1min":0.46000000000000002,"5min":0.34499999999999997,"15min":0.27800000000000002},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.5129999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.30199999999999999},"max":{"1min":1.327,"5min":1.327,"15min":1.536},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Mar 18 19:05:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.56200000000000006,"15min":0.58199999999999996},"min":{"1min":0.371,"5min":0.318,"15min":0.311},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":1.5489999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.59099999999999997,"15min":0.60399999999999998},"min":{"1min":0.498,"5min":0.38800000000000001,"15min":0.38200000000000001},"max":{"1min":0.78200000000000003,"5min":0.78600000000000003,"15min":1.5660000000000001},"last":0.60899999999999999}]},{"osd":4,"last update":"Mon Mar 18 19:05:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.44900000000000001,"15min":0.45600000000000002},"min":{"1min":0.32800000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.47299999999999998,"15min":0.47999999999999998},"min":{"1min":0.34799999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.70899999999999996,"15min":0.73199999999999998},"last":0.46500000000000002}]},{"osd":5,"last update":"Mon Mar 18 19:05:04 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47799999999999998,"15min":0.48099999999999998},"min":{"1min":0.35399999999999998,"5min":0.27100000000000002,"15min":0.26000000000000001},"max":{"1min":0.751,"5min":0.751,"15min":1.2310000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.51600000000000001,"15min":0.52200000000000002},"min":{"1min":0.376,"5min":0.30299999999999999,"15min":0.28799999999999998},"max":{"1min":0.76900000000000002,"5min":0.78200000000000003,"15min":1.1220000000000001},"last":0.53200000000000003}]},{"osd":6,"last update":"Mon Mar 18 19:05:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54800000000000004,"15min":0.54900000000000004},"min":{"1min":0.41599999999999998,"5min":0.374,"15min":0.34300000000000003},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":1.069},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53000000000000003,"15min":0.53800000000000003},"min":{"1min":0.44500000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.71999999999999997,"5min":0.752,"15min":0.79600000000000004},"last":0.65500000000000003}]}]},{"osd":6,"up_from":13,"seq":55834575035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1085440,"data_stored":539024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56899999999999995,"15min":0.57199999999999995},"min":{"1min":0.33000000000000002,"5min":0.28499999999999998,"15min":0.25700000000000001},"max":{"1min":0.69799999999999995,"5min":0.75600000000000001,"15min":3.274},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56299999999999994,"15min":0.58099999999999996},"min":{"1min":0.37,"5min":0.34699999999999998,"15min":0.27400000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":3.3929999999999998},"last":0.39400000000000002}]},{"osd":1,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57699999999999996,"15min":0.56799999999999995},"min":{"1min":0.38100000000000001,"5min":0.316,"15min":0.30599999999999999},"max":{"1min":0.73899999999999999,"5min":0.77400000000000002,"15min":0.94099999999999995},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56200000000000006,"15min":0.56999999999999995},"min":{"1min":0.42699999999999999,"5min":0.32000000000000001,"15min":0.311},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.90700000000000003},"last":0.36899999999999999}]},{"osd":2,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58399999999999996,"15min":0.57599999999999996},"min":{"1min":0.42799999999999999,"5min":0.32700000000000001,"15min":0.29999999999999999},"max":{"1min":0.75700000000000001,"5min":0.80300000000000005,"15min":0.81299999999999994},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57199999999999995,"15min":0.58299999999999996},"min":{"1min":0.36699999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.74199999999999999,"5min":0.83199999999999996,"15min":2.2370000000000001},"last":0.53300000000000003}]},{"osd":3,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59199999999999997,"15min":0.58499999999999996},"min":{"1min":0.41199999999999998,"5min":0.40000000000000002,"15min":0.32100000000000001},"max":{"1min":0.75600000000000001,"5min":0.89600000000000002,"15min":1.0369999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.373,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":1.595},"last":0.54500000000000004}]},{"osd":4,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.34000000000000002,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.76300000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47599999999999998,"15min":0.48099999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":1.8819999999999999},"last":0.503}]},{"osd":5,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.46899999999999997,"15min":0.47399999999999998},"min":{"1min":0.33600000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":1.022},"last":0.433},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51200000000000001,"15min":0.51600000000000001},"min":{"1min":0.35699999999999998,"5min":0.35499999999999998,"15min":0.32200000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.81599999999999995},"last":0.60299999999999998}]},{"osd":7,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.52700000000000002,"15min":0.52600000000000002},"min":{"1min":0.34300000000000003,"5min":0.34100000000000003,"15min":0.313},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":1.532},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53000000000000003,"15min":0.53300000000000003},"min":{"1min":0.39500000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":1.978},"last":0.58899999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575035,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32196,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339644,"statfs":{"total":96636764160,"available":96603795456,"internally_reserved":0,"allocated":1097728,"data_stored":534698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.375,"15min":0.373},"min":{"1min":0.23200000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.83599999999999997},"last":0.253},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42499999999999999,"15min":0.41499999999999998},"min":{"1min":0.25700000000000001,"5min":0.23799999999999999,"15min":0.23400000000000001},"max":{"1min":0.65200000000000002,"5min":0.73699999999999999,"15min":0.90400000000000003},"last":0.33100000000000002}]},{"osd":2,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41599999999999998,"15min":0.40400000000000003},"min":{"1min":0.27800000000000002,"5min":0.26500000000000001,"15min":0.23300000000000001},"max":{"1min":0.56699999999999995,"5min":0.65400000000000003,"15min":0.88400000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.45300000000000001,"15min":0.44500000000000001},"min":{"1min":0.34300000000000003,"5min":0.29699999999999999,"15min":0.23499999999999999},"max":{"1min":0.60699999999999998,"5min":0.68700000000000006,"15min":0.94999999999999996},"last":0.39900000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.27700000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.86799999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.435,"5min":0.42799999999999999,"15min":0.42099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28100000000000003,"15min":0.25600000000000001},"max":{"1min":0.65100000000000002,"5min":0.77000000000000002,"15min":0.93100000000000005},"last":0.27800000000000002}]},{"osd":4,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63800000000000001,"15min":0.63300000000000001},"min":{"1min":0.33400000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.92300000000000004,"5min":0.93999999999999995,"15min":1.052},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.63800000000000001,"15min":0.64400000000000002},"min":{"1min":0.46800000000000003,"5min":0.36899999999999999,"15min":0.34499999999999997},"max":{"1min":0.95799999999999996,"5min":1.018,"15min":1.0269999999999999},"last":0.65400000000000003}]},{"osd":5,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63300000000000001,"15min":0.64200000000000002},"min":{"1min":0.434,"5min":0.40100000000000002,"15min":0.38600000000000001},"max":{"1min":0.877,"5min":0.88500000000000001,"15min":1.0760000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.65600000000000003,"15min":0.65700000000000003},"min":{"1min":0.47099999999999997,"5min":0.41099999999999998,"15min":0.40999999999999998},"max":{"1min":0.80900000000000005,"5min":1.006,"15min":1.1319999999999999},"last":0.64100000000000001}]},{"osd":6,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.93500000000000005,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67500000000000004,"15min":0.66900000000000004},"min":{"1min":0.434,"5min":0.42599999999999999,"15min":0.39100000000000001},"max":{"1min":0.93799999999999994,"5min":0.96799999999999997,"15min":1.0089999999999999},"last":0.66800000000000004}]},{"osd":7,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66000000000000003},"min":{"1min":0.48599999999999999,"5min":0.44900000000000001,"15min":0.36699999999999999},"max":{"1min":0.90600000000000003,"5min":0.92900000000000005,"15min":1.0660000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66700000000000004,"15min":0.66400000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.39900000000000002},"max":{"1min":0.88100000000000001,"5min":0.94299999999999995,"15min":1.1499999999999999},"last":0.59799999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575036,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36844,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334996,"statfs":{"total":96636764160,"available":96599035904,"internally_reserved":0,"allocated":1662976,"data_stored":1122738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.377,"15min":0.377},"min":{"1min":0.23100000000000001,"5min":0.19400000000000001,"15min":0.192},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.621},"last":0.254},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.30299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.55400000000000005,"5min":0.61699999999999999,"15min":0.66400000000000003},"last":0.35799999999999998}]},{"osd":2,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41599999999999998,"15min":0.41699999999999998},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.58999999999999997,"5min":0.60499999999999998,"15min":0.627},"last":0.373},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.30499999999999999,"5min":0.25,"15min":0.25},"max":{"1min":0.627,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.31}]},{"osd":3,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.40699999999999997,"15min":0.42999999999999999},"min":{"1min":0.30299999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.57499999999999996,"5min":0.61299999999999999,"15min":5.7359999999999998},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46500000000000002,"15min":0.48799999999999999},"min":{"1min":0.32000000000000001,"5min":0.28799999999999998,"15min":0.27700000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":5.9329999999999998},"last":0.34000000000000002}]},{"osd":4,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.63800000000000001,"15min":0.63900000000000001},"min":{"1min":0.49099999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.91800000000000004,"5min":1.0229999999999999,"15min":1.22},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.45900000000000002,"5min":0.315,"15min":0.315},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.1020000000000001},"last":0.502}]},{"osd":5,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.47999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.96199999999999997,"5min":0.98099999999999998,"15min":2.9649999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.65600000000000003,"15min":0.65800000000000003},"min":{"1min":0.38,"5min":0.31,"15min":0.31},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.2609999999999999},"last":0.59099999999999997}]},{"osd":6,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.64900000000000002,"15min":0.65300000000000002},"min":{"1min":0.40500000000000003,"5min":0.39400000000000002,"15min":0.38100000000000001},"max":{"1min":0.90100000000000002,"5min":0.94599999999999995,"15min":1.133},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.66500000000000004,"15min":0.67600000000000005},"min":{"1min":0.42699999999999999,"5min":0.41199999999999998,"15min":0.40999999999999998},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":2.9300000000000002},"last":0.59899999999999998}]},{"osd":7,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.66600000000000004,"15min":0.66700000000000004},"min":{"1min":0.51900000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.2849999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.66500000000000004,"15min":0.67000000000000004},"min":{"1min":0.44,"5min":0.437,"15min":0.38500000000000001},"max":{"1min":0.94299999999999995,"5min":0.96299999999999997,"15min":1.002},"last":0.58099999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575036,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58420,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94313420,"statfs":{"total":96636764160,"available":96576942080,"internally_reserved":0,"allocated":1081344,"data_stored":538990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38200000000000001,"15min":0.39500000000000002},"min":{"1min":0.249,"5min":0.216,"15min":0.17799999999999999},"max":{"1min":0.66600000000000004,"5min":0.78300000000000003,"15min":9.1219999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.46500000000000002,"15min":0.47599999999999998},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.68899999999999995,"5min":0.77200000000000002,"15min":9.1560000000000006},"last":0.311}]},{"osd":1,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47999999999999998,"15min":0.49299999999999999},"min":{"1min":0.376,"5min":0.29399999999999998,"15min":0.27100000000000002},"max":{"1min":0.69499999999999995,"5min":0.90500000000000003,"15min":9.2460000000000004},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41299999999999998,"15min":0.42499999999999999},"min":{"1min":0.23899999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.61399999999999999,"5min":0.66700000000000004,"15min":12.932},"last":0.32500000000000001}]},{"osd":3,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.42199999999999999,"15min":0.42799999999999999},"min":{"1min":0.29799999999999999,"5min":0.24199999999999999,"15min":0.22500000000000001},"max":{"1min":0.58599999999999997,"5min":1.4019999999999999,"15min":9.4120000000000008},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51300000000000001,"15min":0.52000000000000002},"min":{"1min":0.33800000000000002,"5min":0.29499999999999998,"15min":0.27700000000000002},"max":{"1min":0.67100000000000004,"5min":0.80100000000000005,"15min":9.2680000000000007},"last":0.36499999999999999}]},{"osd":4,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63,"15min":0.61799999999999999},"min":{"1min":0.39400000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.98199999999999998,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63400000000000001,"15min":0.65200000000000002},"min":{"1min":0.441,"5min":0.35899999999999999,"15min":0.32100000000000001},"max":{"1min":0.874,"5min":1.4730000000000001,"15min":9.3190000000000008},"last":0.437}]},{"osd":5,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65300000000000002,"15min":0.66800000000000004},"min":{"1min":0.45300000000000001,"5min":0.34699999999999998,"15min":0.309},"max":{"1min":0.89600000000000002,"5min":1.8109999999999999,"15min":9.1940000000000008},"last":0.497},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65800000000000003,"15min":0.64000000000000001},"min":{"1min":0.40000000000000002,"5min":0.36799999999999999,"15min":0.33400000000000002},"max":{"1min":0.90300000000000002,"5min":1.948,"15min":1.948},"last":0.46300000000000002}]},{"osd":6,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.67700000000000005,"15min":0.67900000000000005},"min":{"1min":0.40999999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.92300000000000004,"5min":2.9470000000000001,"15min":9.3559999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69299999999999995,"15min":0.70899999999999996},"min":{"1min":0.48899999999999999,"5min":0.38100000000000001,"15min":0.37},"max":{"1min":1.0509999999999999,"5min":1.083,"15min":9.2219999999999995},"last":0.58299999999999996}]},{"osd":7,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.66300000000000003,"15min":0.65200000000000002},"min":{"1min":0.42599999999999999,"5min":0.40999999999999998,"15min":0.38200000000000001},"max":{"1min":0.93600000000000005,"5min":1.083,"15min":1.083},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66800000000000004,"15min":0.68400000000000005},"min":{"1min":0.51800000000000002,"5min":0.371,"15min":0.34599999999999997},"max":{"1min":1.032,"5min":1.4350000000000001,"15min":9.3829999999999991},"last":0.45300000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575036,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36276,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335564,"statfs":{"total":96636764160,"available":96599617536,"internally_reserved":0,"allocated":1081344,"data_stored":538990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.38400000000000001,"15min":0.38300000000000001},"min":{"1min":0.249,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63900000000000001,"5min":1.204,"15min":1.4550000000000001},"last":0.312},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42199999999999999,"15min":0.39900000000000002},"min":{"1min":0.29099999999999998,"5min":0.28000000000000003,"15min":0.23000000000000001},"max":{"1min":0.57599999999999996,"5min":1.137,"15min":1.5169999999999999},"last":0.40699999999999997}]},{"osd":1,"last update":"Mon Mar 18 19:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.41199999999999998,"15min":0.40100000000000002},"min":{"1min":0.27800000000000002,"5min":0.248,"15min":0.221},"max":{"1min":0.61699999999999999,"5min":1.1220000000000001,"15min":1.2909999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.45800000000000002,"15min":0.44700000000000001},"min":{"1min":0.34899999999999998,"5min":0.28499999999999998,"15min":0.245},"max":{"1min":0.628,"5min":1.1080000000000001,"15min":1.355},"last":0.42799999999999999}]},{"osd":2,"last update":"Mon Mar 18 19:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.46300000000000002,"15min":0.45000000000000001},"min":{"1min":0.34000000000000002,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.65100000000000002,"5min":1.1879999999999999,"15min":1.3160000000000001},"last":0.504},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.35499999999999998,"5min":0.33300000000000002,"15min":0.28599999999999998},"max":{"1min":0.71499999999999997,"5min":1.2290000000000001,"15min":1.4810000000000001},"last":0.49099999999999999}]},{"osd":4,"last update":"Mon Mar 18 19:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63200000000000001,"15min":0.625},"min":{"1min":0.40600000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81399999999999995,"5min":1.871,"15min":1.871},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63800000000000001,"15min":0.63500000000000001},"min":{"1min":0.441,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.90600000000000003,"5min":1.764,"15min":1.764},"last":0.60399999999999998}]},{"osd":5,"last update":"Mon Mar 18 19:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.68600000000000005,"15min":0.66200000000000003},"min":{"1min":0.44800000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.0169999999999999,"5min":2.5329999999999999,"15min":2.5329999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.68799999999999994},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.437},"max":{"1min":0.94399999999999995,"5min":1.794,"15min":1.794},"last":0.72899999999999998}]},{"osd":6,"last update":"Mon Mar 18 19:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.68999999999999995,"15min":0.67100000000000004},"min":{"1min":0.51300000000000001,"5min":0.41499999999999998,"15min":0.377},"max":{"1min":0.86799999999999999,"5min":2.7389999999999999,"15min":2.7389999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.69199999999999995,"15min":0.68100000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.98299999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.81999999999999995}]},{"osd":7,"last update":"Mon Mar 18 19:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.52600000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91200000000000003,"5min":1.732,"15min":1.732},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.69299999999999995,"15min":0.66900000000000004},"min":{"1min":0.51400000000000001,"5min":0.439,"15min":0.40500000000000003},"max":{"1min":0.95999999999999996,"5min":2.7050000000000001,"15min":2.7050000000000001},"last":0.64500000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575036,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1081344,"data_stored":533934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.246},"max":{"1min":0.73299999999999998,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56399999999999995,"15min":0.58099999999999996},"min":{"1min":0.40300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.76100000000000001,"5min":0.77600000000000002,"15min":1.0860000000000001},"last":0.50800000000000001}]},{"osd":1,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57399999999999995,"15min":0.58399999999999996},"min":{"1min":0.439,"5min":0.34499999999999997,"15min":0.316},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":1.1679999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55700000000000005,"15min":0.57499999999999996},"min":{"1min":0.41599999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":1.218},"last":0.437}]},{"osd":2,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56299999999999994,"15min":0.57299999999999995},"min":{"1min":0.45000000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72699999999999998,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57399999999999995,"15min":0.59899999999999998},"min":{"1min":0.40999999999999998,"5min":0.36799999999999999,"15min":0.32600000000000001},"max":{"1min":0.77200000000000002,"5min":0.78100000000000003,"15min":1.204},"last":0.64600000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56899999999999995,"15min":0.58599999999999997},"min":{"1min":0.439,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.74099999999999999,"5min":0.75600000000000001,"15min":1.1799999999999999},"last":0.495},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57799999999999996,"15min":0.59599999999999997},"min":{"1min":0.41799999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67700000000000005,"5min":1.121,"15min":1.121},"last":0.57999999999999996}]},{"osd":5,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.44600000000000001,"15min":0.46500000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.245},"max":{"1min":0.55900000000000005,"5min":0.73499999999999999,"15min":0.78900000000000003},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.435,"5min":0.46100000000000002,"15min":0.48299999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27200000000000002},"max":{"1min":0.626,"5min":0.64400000000000002,"15min":0.77800000000000002},"last":0.66500000000000004}]},{"osd":6,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.46800000000000003,"15min":0.48499999999999999},"min":{"1min":0.33900000000000002,"5min":0.32700000000000001,"15min":0.29499999999999998},"max":{"1min":0.66500000000000004,"5min":0.68600000000000005,"15min":0.81799999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.497,"15min":0.52300000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.28999999999999998},"max":{"1min":0.59799999999999998,"5min":0.72999999999999998,"15min":0.88900000000000001},"last":0.71099999999999997}]},{"osd":7,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.498,"15min":0.52100000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.628,"5min":0.70899999999999996,"15min":0.75800000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52600000000000002,"15min":0.55300000000000005},"min":{"1min":0.38300000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70599999999999996,"5min":0.75900000000000001,"15min":0.91900000000000004},"last":0.621}]}]},{"osd":5,"up_from":13,"seq":55834575035,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50212,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321628,"statfs":{"total":96636764160,"available":96585347072,"internally_reserved":0,"allocated":1073152,"data_stored":532392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56899999999999995,"15min":0.58599999999999997},"min":{"1min":0.38300000000000001,"5min":0.26800000000000002,"15min":0.24399999999999999},"max":{"1min":0.77000000000000002,"5min":0.78400000000000003,"15min":6.3079999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57899999999999996,"15min":0.57299999999999995},"min":{"1min":0.373,"5min":0.33700000000000002,"15min":0.25800000000000001},"max":{"1min":0.78800000000000003,"5min":0.79800000000000004,"15min":1.208},"last":0.55800000000000005}]},{"osd":1,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.32900000000000001,"5min":0.32600000000000001,"15min":0.28399999999999997},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.29799999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":6.2549999999999999},"last":0.51400000000000001}]},{"osd":2,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.58599999999999997,"15min":0.59699999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.28399999999999997},"max":{"1min":1.153,"5min":1.153,"15min":6.2999999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56499999999999995,"15min":0.623},"min":{"1min":0.39200000000000002,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":0.73999999999999999,"5min":0.748,"15min":17.387},"last":0.54100000000000004}]},{"osd":3,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.35799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":6.3330000000000002},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.212,"5min":1.212,"15min":1.2390000000000001},"last":0.59199999999999997}]},{"osd":4,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.47099999999999997,"15min":0.46800000000000003},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.497,"15min":0.495},"min":{"1min":0.32700000000000001,"5min":0.29499999999999998,"15min":0.23100000000000001},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.42299999999999999}]},{"osd":6,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.46700000000000003,"15min":0.47899999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.70999999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.504,"15min":0.51000000000000001},"min":{"1min":0.32100000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66300000000000003,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.48799999999999999}]},{"osd":7,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.53100000000000003},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55300000000000005,"15min":0.55200000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.28499999999999998},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.67700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T19:05:26.742 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T19:05:26.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.906+0000 7fdabf1a2700 1 Processor -- start 2024-03-18T19:05:26.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.906+0000 7fdabf1a2700 1 -- start start 2024-03-18T19:05:26.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.907+0000 7fdabf1a2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdab80cef80 0x7fdab806b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:26.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.907+0000 7fdabf1a2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdab80cdc00 0x7fdab80cf380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:26.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.907+0000 7fdabf1a2700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdab80ce080 0x7fdab80cf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:26.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.907+0000 7fdabf1a2700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdab80717c0 con 0x7fdab80cdc00 2024-03-18T19:05:26.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.907+0000 7fdabf1a2700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdab815da00 con 0x7fdab80ce080 2024-03-18T19:05:26.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.907+0000 7fdabf1a2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab806b670 con 0x7fdab80cef80 2024-03-18T19:05:26.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.907+0000 7fdab7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdab80cdc00 0x7fdab80cf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:26.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.907+0000 7fdab7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdab80cdc00 0x7fdab80cf380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33710/0 (socket says 172.21.15.1:33710) 2024-03-18T19:05:26.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.907+0000 7fdab7fff700 1 -- 172.21.15.1:0/3334251591 learned_addr learned my addr 172.21.15.1:0/3334251591 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:26.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.907+0000 7fdabcf3e700 1 --2- 172.21.15.1:0/3334251591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdab80cef80 0x7fdab806b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:26.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.907+0000 7fdabd73f700 1 --2- 172.21.15.1:0/3334251591 >> v2:172.21.15.165:3300/0 conn(0x7fdab80ce080 0x7fdab80cf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:26.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.908+0000 7fdab7fff700 1 -- 172.21.15.1:0/3334251591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdab80cef80 msgr2=0x7fdab806b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:26.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.908+0000 7fdab7fff700 1 --2- 172.21.15.1:0/3334251591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdab80cef80 0x7fdab806b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.908+0000 7fdab7fff700 1 -- 172.21.15.1:0/3334251591 >> v2:172.21.15.165:3300/0 conn(0x7fdab80ce080 msgr2=0x7fdab80cf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:26.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.908+0000 7fdab7fff700 1 --2- 172.21.15.1:0/3334251591 >> v2:172.21.15.165:3300/0 conn(0x7fdab80ce080 0x7fdab80cf8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.908+0000 7fdab7fff700 1 -- 172.21.15.1:0/3334251591 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab80cc8b0 con 0x7fdab80cdc00 2024-03-18T19:05:26.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.908+0000 7fdabcf3e700 1 --2- 172.21.15.1:0/3334251591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdab80cef80 0x7fdab806b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:26.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.908+0000 7fdabd73f700 1 --2- 172.21.15.1:0/3334251591 >> v2:172.21.15.165:3300/0 conn(0x7fdab80ce080 0x7fdab80cf8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:26.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.908+0000 7fdab7fff700 1 --2- 172.21.15.1:0/3334251591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdab80cdc00 0x7fdab80cf380 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fdab8073a90 tx=0x7fdaa4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a05d0f02f9974fe7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:26.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.908+0000 7fdab77fe700 1 -- 172.21.15.1:0/3334251591 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa400b870 con 0x7fdab80cdc00 2024-03-18T19:05:26.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.908+0000 7fdab77fe700 1 -- 172.21.15.1:0/3334251591 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa400b9d0 con 0x7fdab80cdc00 2024-03-18T19:05:26.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.908+0000 7fdab77fe700 1 -- 172.21.15.1:0/3334251591 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa400bd60 con 0x7fdab80cdc00 2024-03-18T19:05:26.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.908+0000 7fdabf1a2700 1 -- 172.21.15.1:0/3334251591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdab80cdc00 msgr2=0x7fdab80cf380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:26.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.908+0000 7fdabf1a2700 1 --2- 172.21.15.1:0/3334251591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdab80cdc00 0x7fdab80cf380 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fdab8073a90 tx=0x7fdaa4009bd0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.908+0000 7fdabf1a2700 1 -- 172.21.15.1:0/3334251591 shutdown_connections 2024-03-18T19:05:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.908+0000 7fdabf1a2700 1 --2- 172.21.15.1:0/3334251591 >> v2:172.21.15.165:3300/0 conn(0x7fdab80ce080 0x7fdab80cf8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.908+0000 7fdabf1a2700 1 --2- 172.21.15.1:0/3334251591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdab80cef80 0x7fdab806b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.908+0000 7fdabf1a2700 1 --2- 172.21.15.1:0/3334251591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdab80cdc00 0x7fdab80cf380 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.908+0000 7fdabf1a2700 1 -- 172.21.15.1:0/3334251591 >> 172.21.15.1:0/3334251591 conn(0x7fdab80ccab0 msgr2=0x7fdab80ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.908+0000 7fdabf1a2700 1 -- 172.21.15.1:0/3334251591 shutdown_connections 2024-03-18T19:05:26.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.909+0000 7fdabf1a2700 1 -- 172.21.15.1:0/3334251591 wait complete. 2024-03-18T19:05:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.910+0000 7fdabf1a2700 1 Processor -- start 2024-03-18T19:05:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.910+0000 7fdabf1a2700 1 -- start start 2024-03-18T19:05:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.910+0000 7fdabf1a2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdab80cdc00 0x7fdab80c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdabf1a2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdab80ce080 0x7fdab81f2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdabf1a2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdab80cef80 0x7fdab80a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdabf1a2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab80c6b80 con 0x7fdab80ce080 2024-03-18T19:05:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdabf1a2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab80a07b0 con 0x7fdab80cef80 2024-03-18T19:05:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdabf1a2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab8161f20 con 0x7fdab80cdc00 2024-03-18T19:05:26.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdab7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdab80ce080 0x7fdab81f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdab7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdab80ce080 0x7fdab81f2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33712/0 (socket says 172.21.15.1:33712) 2024-03-18T19:05:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdab7fff700 1 -- 172.21.15.1:0/3817359478 learned_addr learned my addr 172.21.15.1:0/3817359478 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdabcf3e700 1 --2- 172.21.15.1:0/3817359478 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdab80cdc00 0x7fdab80c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdabd73f700 1 --2- 172.21.15.1:0/3817359478 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdab80cef80 0x7fdab80a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdab7fff700 1 -- 172.21.15.1:0/3817359478 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdab80cdc00 msgr2=0x7fdab80c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdab7fff700 1 --2- 172.21.15.1:0/3817359478 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdab80cdc00 0x7fdab80c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdab7fff700 1 -- 172.21.15.1:0/3817359478 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdab80cef80 msgr2=0x7fdab80a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdab7fff700 1 --2- 172.21.15.1:0/3817359478 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdab80cef80 0x7fdab80a0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdab7fff700 1 -- 172.21.15.1:0/3817359478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa4009550 con 0x7fdab80ce080 2024-03-18T19:05:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdab7fff700 1 --2- 172.21.15.1:0/3817359478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdab80ce080 0x7fdab81f2640 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4014040 tx=0x7fdaa4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdab57fa700 1 -- 172.21.15.1:0/3817359478 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4004520 con 0x7fdab80ce080 2024-03-18T19:05:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdab57fa700 1 -- 172.21.15.1:0/3817359478 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa401e070 con 0x7fdab80ce080 2024-03-18T19:05:26.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdabf1a2700 1 -- 172.21.15.1:0/3817359478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab809fe80 con 0x7fdab80ce080 2024-03-18T19:05:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdab57fa700 1 -- 172.21.15.1:0/3817359478 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa40038e0 con 0x7fdab80ce080 2024-03-18T19:05:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.911+0000 7fdabf1a2700 1 -- 172.21.15.1:0/3817359478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab80a0440 con 0x7fdab80ce080 2024-03-18T19:05:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.912+0000 7fdabf1a2700 1 -- 172.21.15.1:0/3817359478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab80676a0 con 0x7fdab80ce080 2024-03-18T19:05:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.913+0000 7fdab57fa700 1 -- 172.21.15.1:0/3817359478 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fdaa4003a40 con 0x7fdab80ce080 2024-03-18T19:05:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.913+0000 7fdab57fa700 1 --2- 172.21.15.1:0/3817359478 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fda9c06adb0 0x7fda9c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.913+0000 7fdab57fa700 1 -- 172.21.15.1:0/3817359478 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fdaa4020070 con 0x7fdab80ce080 2024-03-18T19:05:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.913+0000 7fdabcf3e700 1 --2- 172.21.15.1:0/3817359478 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fda9c06adb0 0x7fda9c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.914+0000 7fdabcf3e700 1 --2- 172.21.15.1:0/3817359478 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fda9c06adb0 0x7fda9c06d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdaac006fd0 tx=0x7fdaac008040 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:26.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:26.918+0000 7fdab57fa700 1 -- 172.21.15.1:0/3817359478 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdaa4055b00 con 0x7fdab80ce080 2024-03-18T19:05:27.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.118+0000 7fdabf1a2700 1 -- 172.21.15.1:0/3817359478 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdab80cc5d0 con 0x7fda9c06adb0 2024-03-18T19:05:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.126+0000 7fdab57fa700 1 -- 172.21.15.1:0/3817359478 <== mgr.6238 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289631 (secure 0 0 0) 0x7fdaac00c050 con 0x7fda9c06adb0 2024-03-18T19:05:27.125 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:27.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.129+0000 7fdabf1a2700 1 -- 172.21.15.1:0/3817359478 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fda9c06adb0 msgr2=0x7fda9c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:27.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.129+0000 7fdabf1a2700 1 --2- 172.21.15.1:0/3817359478 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fda9c06adb0 0x7fda9c06d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdaac006fd0 tx=0x7fdaac008040 comp rx=0 tx=0).stop 2024-03-18T19:05:27.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.129+0000 7fdabf1a2700 1 -- 172.21.15.1:0/3817359478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdab80ce080 msgr2=0x7fdab81f2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:27.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.129+0000 7fdabf1a2700 1 --2- 172.21.15.1:0/3817359478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdab80ce080 0x7fdab81f2640 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4014040 tx=0x7fdaa4004750 comp rx=0 tx=0).stop 2024-03-18T19:05:27.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.129+0000 7fdabf1a2700 1 -- 172.21.15.1:0/3817359478 shutdown_connections 2024-03-18T19:05:27.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.129+0000 7fdabf1a2700 1 --2- 172.21.15.1:0/3817359478 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fda9c06adb0 0x7fda9c06d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:27.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.129+0000 7fdabf1a2700 1 --2- 172.21.15.1:0/3817359478 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdab80cdc00 0x7fdab80c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:27.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.129+0000 7fdabf1a2700 1 --2- 172.21.15.1:0/3817359478 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdab80cef80 0x7fdab80a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:27.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.129+0000 7fdabf1a2700 1 --2- 172.21.15.1:0/3817359478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdab80ce080 0x7fdab81f2640 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:27.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.129+0000 7fdabf1a2700 1 -- 172.21.15.1:0/3817359478 >> 172.21.15.1:0/3817359478 conn(0x7fdab80ccab0 msgr2=0x7fdab80b0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:27.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.129+0000 7fdabf1a2700 1 -- 172.21.15.1:0/3817359478 shutdown_connections 2024-03-18T19:05:27.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.129+0000 7fdabf1a2700 1 -- 172.21.15.1:0/3817359478 wait complete. 2024-03-18T19:05:27.129 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T19:05:27.137 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-18T19:05:27.021538+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":131,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":337296,"kb_used_data":9612,"kb_used_omap":12,"kb_used_meta":327539,"kb_avail":754637424,"statfs":{"total":773094113280,"available":772748722176,"internally_reserved":0,"allocated":9842688,"data_stored":5464280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":335400505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.871901"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.599445+0000","last_change":"2024-03-18T19:05:22.593122+0000","last_active":"2024-03-18T19:05:22.599445+0000","last_peered":"2024-03-18T19:05:22.599445+0000","last_clean":"2024-03-18T19:05:22.599445+0000","last_became_active":"2024-03-18T19:05:22.592878+0000","last_became_peered":"2024-03-18T19:05:22.592878+0000","last_unstale":"2024-03-18T19:05:22.599445+0000","last_undegraded":"2024-03-18T19:05:22.599445+0000","last_fullsized":"2024-03-18T19:05:22.599445+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:50.436797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.598613+0000","last_change":"2024-03-18T19:05:22.592083+0000","last_active":"2024-03-18T19:05:22.598613+0000","last_peered":"2024-03-18T19:05:22.598613+0000","last_clean":"2024-03-18T19:05:22.598613+0000","last_became_active":"2024-03-18T19:05:22.591681+0000","last_became_peered":"2024-03-18T19:05:22.591681+0000","last_unstale":"2024-03-18T19:05:22.598613+0000","last_undegraded":"2024-03-18T19:05:22.598613+0000","last_fullsized":"2024-03-18T19:05:22.598613+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:52.820873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.608848+0000","last_change":"2024-03-18T19:05:22.605702+0000","last_active":"2024-03-18T19:05:22.608848+0000","last_peered":"2024-03-18T19:05:22.608848+0000","last_clean":"2024-03-18T19:05:22.608848+0000","last_became_active":"2024-03-18T19:05:22.605450+0000","last_became_peered":"2024-03-18T19:05:22.605450+0000","last_unstale":"2024-03-18T19:05:22.608848+0000","last_undegraded":"2024-03-18T19:05:22.608848+0000","last_fullsized":"2024-03-18T19:05:22.608848+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:52.107864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966527+0000","last_change":"2024-03-18T19:05:22.597848+0000","last_active":"2024-03-18T19:05:24.966527+0000","last_peered":"2024-03-18T19:05:24.966527+0000","last_clean":"2024-03-18T19:05:24.966527+0000","last_became_active":"2024-03-18T19:05:22.597589+0000","last_became_peered":"2024-03-18T19:05:22.597589+0000","last_unstale":"2024-03-18T19:05:24.966527+0000","last_undegraded":"2024-03-18T19:05:24.966527+0000","last_fullsized":"2024-03-18T19:05:24.966527+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:21:54.604175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590513+0000","last_change":"2024-03-18T19:05:22.599635+0000","last_active":"2024-03-18T19:05:23.590513+0000","last_peered":"2024-03-18T19:05:23.590513+0000","last_clean":"2024-03-18T19:05:23.590513+0000","last_became_active":"2024-03-18T19:05:22.597583+0000","last_became_peered":"2024-03-18T19:05:22.597583+0000","last_unstale":"2024-03-18T19:05:23.590513+0000","last_undegraded":"2024-03-18T19:05:23.590513+0000","last_fullsized":"2024-03-18T19:05:23.590513+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:28.026254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.608151+0000","last_change":"2024-03-18T19:05:22.607718+0000","last_active":"2024-03-18T19:05:22.608151+0000","last_peered":"2024-03-18T19:05:22.608151+0000","last_clean":"2024-03-18T19:05:22.608151+0000","last_became_active":"2024-03-18T19:05:22.607454+0000","last_became_peered":"2024-03-18T19:05:22.607454+0000","last_unstale":"2024-03-18T19:05:22.608151+0000","last_undegraded":"2024-03-18T19:05:22.608151+0000","last_fullsized":"2024-03-18T19:05:22.608151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:04.088770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.591187+0000","last_change":"2024-03-18T19:05:22.603594+0000","last_active":"2024-03-18T19:05:23.591187+0000","last_peered":"2024-03-18T19:05:23.591187+0000","last_clean":"2024-03-18T19:05:23.591187+0000","last_became_active":"2024-03-18T19:05:22.603392+0000","last_became_peered":"2024-03-18T19:05:22.603392+0000","last_unstale":"2024-03-18T19:05:23.591187+0000","last_undegraded":"2024-03-18T19:05:23.591187+0000","last_fullsized":"2024-03-18T19:05:23.591187+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:18:57.616746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597201+0000","last_change":"2024-03-18T19:05:22.604349+0000","last_active":"2024-03-18T19:05:24.597201+0000","last_peered":"2024-03-18T19:05:24.597201+0000","last_clean":"2024-03-18T19:05:24.597201+0000","last_became_active":"2024-03-18T19:05:22.603810+0000","last_became_peered":"2024-03-18T19:05:22.603810+0000","last_unstale":"2024-03-18T19:05:24.597201+0000","last_undegraded":"2024-03-18T19:05:24.597201+0000","last_fullsized":"2024-03-18T19:05:24.597201+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:52.820873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.609474+0000","last_change":"2024-03-18T19:05:22.608030+0000","last_active":"2024-03-18T19:05:22.609474+0000","last_peered":"2024-03-18T19:05:22.609474+0000","last_clean":"2024-03-18T19:05:22.609474+0000","last_became_active":"2024-03-18T19:05:22.607736+0000","last_became_peered":"2024-03-18T19:05:22.607736+0000","last_unstale":"2024-03-18T19:05:22.609474+0000","last_undegraded":"2024-03-18T19:05:22.609474+0000","last_fullsized":"2024-03-18T19:05:22.609474+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:18:55.815832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.606143+0000","last_change":"2024-03-18T19:05:22.604209+0000","last_active":"2024-03-18T19:05:22.606143+0000","last_peered":"2024-03-18T19:05:22.606143+0000","last_clean":"2024-03-18T19:05:22.606143+0000","last_became_active":"2024-03-18T19:05:22.603909+0000","last_became_peered":"2024-03-18T19:05:22.603909+0000","last_unstale":"2024-03-18T19:05:22.606143+0000","last_undegraded":"2024-03-18T19:05:22.606143+0000","last_fullsized":"2024-03-18T19:05:22.606143+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:50.436797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966686+0000","last_change":"2024-03-18T19:05:22.610141+0000","last_active":"2024-03-18T19:05:24.966686+0000","last_peered":"2024-03-18T19:05:24.966686+0000","last_clean":"2024-03-18T19:05:24.966686+0000","last_became_active":"2024-03-18T19:05:22.609866+0000","last_became_peered":"2024-03-18T19:05:22.609866+0000","last_unstale":"2024-03-18T19:05:24.966686+0000","last_undegraded":"2024-03-18T19:05:24.966686+0000","last_fullsized":"2024-03-18T19:05:24.966686+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:06:42.914898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597558+0000","last_change":"2024-03-18T19:05:22.604073+0000","last_active":"2024-03-18T19:05:24.597558+0000","last_peered":"2024-03-18T19:05:24.597558+0000","last_clean":"2024-03-18T19:05:24.597558+0000","last_became_active":"2024-03-18T19:05:22.603786+0000","last_became_peered":"2024-03-18T19:05:22.603786+0000","last_unstale":"2024-03-18T19:05:24.597558+0000","last_undegraded":"2024-03-18T19:05:24.597558+0000","last_fullsized":"2024-03-18T19:05:24.597558+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:21.799481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597667+0000","last_change":"2024-03-18T19:05:22.611538+0000","last_active":"2024-03-18T19:05:24.597667+0000","last_peered":"2024-03-18T19:05:24.597667+0000","last_clean":"2024-03-18T19:05:24.597667+0000","last_became_active":"2024-03-18T19:05:22.611279+0000","last_became_peered":"2024-03-18T19:05:22.611279+0000","last_unstale":"2024-03-18T19:05:24.597667+0000","last_undegraded":"2024-03-18T19:05:24.597667+0000","last_fullsized":"2024-03-18T19:05:24.597667+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:50.436797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.611080+0000","last_change":"2024-03-18T19:05:22.607481+0000","last_active":"2024-03-18T19:05:22.611080+0000","last_peered":"2024-03-18T19:05:22.611080+0000","last_clean":"2024-03-18T19:05:22.611080+0000","last_became_active":"2024-03-18T19:05:22.607184+0000","last_became_peered":"2024-03-18T19:05:22.607184+0000","last_unstale":"2024-03-18T19:05:22.611080+0000","last_undegraded":"2024-03-18T19:05:22.611080+0000","last_fullsized":"2024-03-18T19:05:22.611080+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:21.799481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.610223+0000","last_change":"2024-03-18T19:05:22.606780+0000","last_active":"2024-03-18T19:05:22.610223+0000","last_peered":"2024-03-18T19:05:22.610223+0000","last_clean":"2024-03-18T19:05:22.610223+0000","last_became_active":"2024-03-18T19:05:22.606483+0000","last_became_peered":"2024-03-18T19:05:22.606483+0000","last_unstale":"2024-03-18T19:05:22.610223+0000","last_undegraded":"2024-03-18T19:05:22.610223+0000","last_fullsized":"2024-03-18T19:05:22.610223+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:49:41.091622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.591067+0000","last_change":"2024-03-18T19:05:22.593988+0000","last_active":"2024-03-18T19:05:23.591067+0000","last_peered":"2024-03-18T19:05:23.591067+0000","last_clean":"2024-03-18T19:05:23.591067+0000","last_became_active":"2024-03-18T19:05:22.593698+0000","last_became_peered":"2024-03-18T19:05:22.593698+0000","last_unstale":"2024-03-18T19:05:23.591067+0000","last_undegraded":"2024-03-18T19:05:23.591067+0000","last_fullsized":"2024-03-18T19:05:23.591067+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:42:21.238001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.608007+0000","last_change":"2024-03-18T19:05:22.607110+0000","last_active":"2024-03-18T19:05:22.608007+0000","last_peered":"2024-03-18T19:05:22.608007+0000","last_clean":"2024-03-18T19:05:22.608007+0000","last_became_active":"2024-03-18T19:05:22.606777+0000","last_became_peered":"2024-03-18T19:05:22.606777+0000","last_unstale":"2024-03-18T19:05:22.608007+0000","last_undegraded":"2024-03-18T19:05:22.608007+0000","last_fullsized":"2024-03-18T19:05:22.608007+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:56:09.533605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966624+0000","last_change":"2024-03-18T19:05:22.598497+0000","last_active":"2024-03-18T19:05:24.966624+0000","last_peered":"2024-03-18T19:05:24.966624+0000","last_clean":"2024-03-18T19:05:24.966624+0000","last_became_active":"2024-03-18T19:05:22.598256+0000","last_became_peered":"2024-03-18T19:05:22.598256+0000","last_unstale":"2024-03-18T19:05:24.966624+0000","last_undegraded":"2024-03-18T19:05:24.966624+0000","last_fullsized":"2024-03-18T19:05:24.966624+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:49:03.493777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590714+0000","last_change":"2024-03-18T19:05:22.589776+0000","last_active":"2024-03-18T19:05:23.590714+0000","last_peered":"2024-03-18T19:05:23.590714+0000","last_clean":"2024-03-18T19:05:23.590714+0000","last_became_active":"2024-03-18T19:05:22.589482+0000","last_became_peered":"2024-03-18T19:05:22.589482+0000","last_unstale":"2024-03-18T19:05:23.590714+0000","last_undegraded":"2024-03-18T19:05:23.590714+0000","last_fullsized":"2024-03-18T19:05:23.590714+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:04:08.801897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.114111+0000","last_change":"2024-03-18T19:05:22.595248+0000","last_active":"2024-03-18T19:05:24.114111+0000","last_peered":"2024-03-18T19:05:24.114111+0000","last_clean":"2024-03-18T19:05:24.114111+0000","last_became_active":"2024-03-18T19:05:22.594972+0000","last_became_peered":"2024-03-18T19:05:22.594972+0000","last_unstale":"2024-03-18T19:05:24.114111+0000","last_undegraded":"2024-03-18T19:05:24.114111+0000","last_fullsized":"2024-03-18T19:05:24.114111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:46.862070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597728+0000","last_change":"2024-03-18T19:05:22.593273+0000","last_active":"2024-03-18T19:05:24.597728+0000","last_peered":"2024-03-18T19:05:24.597728+0000","last_clean":"2024-03-18T19:05:24.597728+0000","last_became_active":"2024-03-18T19:05:22.592982+0000","last_became_peered":"2024-03-18T19:05:22.592982+0000","last_unstale":"2024-03-18T19:05:24.597728+0000","last_undegraded":"2024-03-18T19:05:24.597728+0000","last_fullsized":"2024-03-18T19:05:24.597728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:18:55.815832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.113960+0000","last_change":"2024-03-18T19:05:22.607654+0000","last_active":"2024-03-18T19:05:24.113960+0000","last_peered":"2024-03-18T19:05:24.113960+0000","last_clean":"2024-03-18T19:05:24.113960+0000","last_became_active":"2024-03-18T19:05:22.607397+0000","last_became_peered":"2024-03-18T19:05:22.607397+0000","last_unstale":"2024-03-18T19:05:24.113960+0000","last_undegraded":"2024-03-18T19:05:24.113960+0000","last_fullsized":"2024-03-18T19:05:24.113960+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:09:57.526166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.113905+0000","last_change":"2024-03-18T19:05:22.606436+0000","last_active":"2024-03-18T19:05:24.113905+0000","last_peered":"2024-03-18T19:05:24.113905+0000","last_clean":"2024-03-18T19:05:24.113905+0000","last_became_active":"2024-03-18T19:05:22.606176+0000","last_became_peered":"2024-03-18T19:05:22.606176+0000","last_unstale":"2024-03-18T19:05:24.113905+0000","last_undegraded":"2024-03-18T19:05:24.113905+0000","last_fullsized":"2024-03-18T19:05:24.113905+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:26:33.270251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.608773+0000","last_change":"2024-03-18T19:05:22.605839+0000","last_active":"2024-03-18T19:05:22.608773+0000","last_peered":"2024-03-18T19:05:22.608773+0000","last_clean":"2024-03-18T19:05:22.608773+0000","last_became_active":"2024-03-18T19:05:22.605564+0000","last_became_peered":"2024-03-18T19:05:22.605564+0000","last_unstale":"2024-03-18T19:05:22.608773+0000","last_undegraded":"2024-03-18T19:05:22.608773+0000","last_fullsized":"2024-03-18T19:05:22.608773+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:59:52.820873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597548+0000","last_change":"2024-03-18T19:05:22.597398+0000","last_active":"2024-03-18T19:05:24.597548+0000","last_peered":"2024-03-18T19:05:24.597548+0000","last_clean":"2024-03-18T19:05:24.597548+0000","last_became_active":"2024-03-18T19:05:22.597115+0000","last_became_peered":"2024-03-18T19:05:22.597115+0000","last_unstale":"2024-03-18T19:05:24.597548+0000","last_undegraded":"2024-03-18T19:05:24.597548+0000","last_fullsized":"2024-03-18T19:05:24.597548+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:01:35.139881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597394+0000","last_change":"2024-03-18T19:05:22.594988+0000","last_active":"2024-03-18T19:05:24.597394+0000","last_peered":"2024-03-18T19:05:24.597394+0000","last_clean":"2024-03-18T19:05:24.597394+0000","last_became_active":"2024-03-18T19:05:22.594707+0000","last_became_peered":"2024-03-18T19:05:22.594707+0000","last_unstale":"2024-03-18T19:05:24.597394+0000","last_undegraded":"2024-03-18T19:05:24.597394+0000","last_fullsized":"2024-03-18T19:05:24.597394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:52.107864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966801+0000","last_change":"2024-03-18T19:05:22.596850+0000","last_active":"2024-03-18T19:05:24.966801+0000","last_peered":"2024-03-18T19:05:24.966801+0000","last_clean":"2024-03-18T19:05:24.966801+0000","last_became_active":"2024-03-18T19:05:22.596535+0000","last_became_peered":"2024-03-18T19:05:22.596535+0000","last_unstale":"2024-03-18T19:05:24.966801+0000","last_undegraded":"2024-03-18T19:05:24.966801+0000","last_fullsized":"2024-03-18T19:05:24.966801+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:52:53.706285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.592514+0000","last_change":"2024-03-18T19:05:22.589101+0000","last_active":"2024-03-18T19:05:22.592514+0000","last_peered":"2024-03-18T19:05:22.592514+0000","last_clean":"2024-03-18T19:05:22.592514+0000","last_became_active":"2024-03-18T19:05:22.588790+0000","last_became_peered":"2024-03-18T19:05:22.588790+0000","last_unstale":"2024-03-18T19:05:22.592514+0000","last_undegraded":"2024-03-18T19:05:22.592514+0000","last_fullsized":"2024-03-18T19:05:22.592514+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:52.107864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.613960+0000","last_change":"2024-03-18T19:05:22.613071+0000","last_active":"2024-03-18T19:05:22.613960+0000","last_peered":"2024-03-18T19:05:22.613960+0000","last_clean":"2024-03-18T19:05:22.613960+0000","last_became_active":"2024-03-18T19:05:22.612753+0000","last_became_peered":"2024-03-18T19:05:22.612753+0000","last_unstale":"2024-03-18T19:05:22.613960+0000","last_undegraded":"2024-03-18T19:05:22.613960+0000","last_fullsized":"2024-03-18T19:05:22.613960+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:01:35.139881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597274+0000","last_change":"2024-03-18T19:05:22.608027+0000","last_active":"2024-03-18T19:05:24.597274+0000","last_peered":"2024-03-18T19:05:24.597274+0000","last_clean":"2024-03-18T19:05:24.597274+0000","last_became_active":"2024-03-18T19:05:22.607725+0000","last_became_peered":"2024-03-18T19:05:22.607725+0000","last_unstale":"2024-03-18T19:05:24.597274+0000","last_undegraded":"2024-03-18T19:05:24.597274+0000","last_fullsized":"2024-03-18T19:05:24.597274+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:49:41.091622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966357+0000","last_change":"2024-03-18T19:05:22.609587+0000","last_active":"2024-03-18T19:05:24.966357+0000","last_peered":"2024-03-18T19:05:24.966357+0000","last_clean":"2024-03-18T19:05:24.966357+0000","last_became_active":"2024-03-18T19:05:22.609324+0000","last_became_peered":"2024-03-18T19:05:22.609324+0000","last_unstale":"2024-03-18T19:05:24.966357+0000","last_undegraded":"2024-03-18T19:05:24.966357+0000","last_fullsized":"2024-03-18T19:05:24.966357+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:21.799481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.601380+0000","last_change":"2024-03-18T19:05:22.592257+0000","last_active":"2024-03-18T19:05:22.601380+0000","last_peered":"2024-03-18T19:05:22.601380+0000","last_clean":"2024-03-18T19:05:22.601380+0000","last_became_active":"2024-03-18T19:05:22.591402+0000","last_became_peered":"2024-03-18T19:05:22.591402+0000","last_unstale":"2024-03-18T19:05:22.601380+0000","last_undegraded":"2024-03-18T19:05:22.601380+0000","last_fullsized":"2024-03-18T19:05:22.601380+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:31:20.342831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590621+0000","last_change":"2024-03-18T19:05:22.589035+0000","last_active":"2024-03-18T19:05:23.590621+0000","last_peered":"2024-03-18T19:05:23.590621+0000","last_clean":"2024-03-18T19:05:23.590621+0000","last_became_active":"2024-03-18T19:05:22.588710+0000","last_became_peered":"2024-03-18T19:05:22.588710+0000","last_unstale":"2024-03-18T19:05:23.590621+0000","last_undegraded":"2024-03-18T19:05:23.590621+0000","last_fullsized":"2024-03-18T19:05:23.590621+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:56:45.230445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"88'4","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.607703+0000","last_change":"2024-03-18T19:05:22.170977+0000","last_active":"2024-03-18T19:05:22.607703+0000","last_peered":"2024-03-18T19:05:22.607703+0000","last_clean":"2024-03-18T19:05:22.607703+0000","last_became_active":"2024-03-18T19:05:22.170471+0000","last_became_peered":"2024-03-18T19:05:22.170471+0000","last_unstale":"2024-03-18T19:05:22.607703+0000","last_undegraded":"2024-03-18T19:05:22.607703+0000","last_fullsized":"2024-03-18T19:05:22.607703+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:56:44.225942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966804+0000","last_change":"2024-03-18T19:05:22.598141+0000","last_active":"2024-03-18T19:05:24.966804+0000","last_peered":"2024-03-18T19:05:24.966804+0000","last_clean":"2024-03-18T19:05:24.966804+0000","last_became_active":"2024-03-18T19:05:22.597872+0000","last_became_peered":"2024-03-18T19:05:22.597872+0000","last_unstale":"2024-03-18T19:05:24.966804+0000","last_undegraded":"2024-03-18T19:05:24.966804+0000","last_fullsized":"2024-03-18T19:05:24.966804+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:03:13.170587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"88'1","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.606332+0000","last_change":"2024-03-18T19:05:22.170924+0000","last_active":"2024-03-18T19:05:22.606332+0000","last_peered":"2024-03-18T19:05:22.606332+0000","last_clean":"2024-03-18T19:05:22.606332+0000","last_became_active":"2024-03-18T19:05:22.170658+0000","last_became_peered":"2024-03-18T19:05:22.170658+0000","last_unstale":"2024-03-18T19:05:22.606332+0000","last_undegraded":"2024-03-18T19:05:22.606332+0000","last_fullsized":"2024-03-18T19:05:22.606332+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:10.178077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.594683+0000","last_change":"2024-03-18T19:05:22.592145+0000","last_active":"2024-03-18T19:05:22.594683+0000","last_peered":"2024-03-18T19:05:22.594683+0000","last_clean":"2024-03-18T19:05:22.594683+0000","last_became_active":"2024-03-18T19:05:22.591881+0000","last_became_peered":"2024-03-18T19:05:22.591881+0000","last_unstale":"2024-03-18T19:05:22.594683+0000","last_undegraded":"2024-03-18T19:05:22.594683+0000","last_fullsized":"2024-03-18T19:05:22.594683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:05:30.670583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"88'2","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.606902+0000","last_change":"2024-03-18T19:05:22.169943+0000","last_active":"2024-03-18T19:05:22.606902+0000","last_peered":"2024-03-18T19:05:22.606902+0000","last_clean":"2024-03-18T19:05:22.606902+0000","last_became_active":"2024-03-18T19:05:22.169557+0000","last_became_peered":"2024-03-18T19:05:22.169557+0000","last_unstale":"2024-03-18T19:05:22.606902+0000","last_undegraded":"2024-03-18T19:05:22.606902+0000","last_fullsized":"2024-03-18T19:05:22.606902+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:19:09.137260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.609310+0000","last_change":"2024-03-18T19:05:22.606259+0000","last_active":"2024-03-18T19:05:22.609310+0000","last_peered":"2024-03-18T19:05:22.609310+0000","last_clean":"2024-03-18T19:05:22.609310+0000","last_became_active":"2024-03-18T19:05:22.605873+0000","last_became_peered":"2024-03-18T19:05:22.605873+0000","last_unstale":"2024-03-18T19:05:22.609310+0000","last_undegraded":"2024-03-18T19:05:22.609310+0000","last_fullsized":"2024-03-18T19:05:22.609310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:59:06.740471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"88'2","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.601896+0000","last_change":"2024-03-18T19:05:22.168058+0000","last_active":"2024-03-18T19:05:22.601896+0000","last_peered":"2024-03-18T19:05:22.601896+0000","last_clean":"2024-03-18T19:05:22.601896+0000","last_became_active":"2024-03-18T19:05:22.167785+0000","last_became_peered":"2024-03-18T19:05:22.167785+0000","last_unstale":"2024-03-18T19:05:22.601896+0000","last_undegraded":"2024-03-18T19:05:22.601896+0000","last_fullsized":"2024-03-18T19:05:22.601896+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:20:34.274975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.591155+0000","last_change":"2024-03-18T19:05:22.589947+0000","last_active":"2024-03-18T19:05:23.591155+0000","last_peered":"2024-03-18T19:05:23.591155+0000","last_clean":"2024-03-18T19:05:23.591155+0000","last_became_active":"2024-03-18T19:05:22.589147+0000","last_became_peered":"2024-03-18T19:05:22.589147+0000","last_unstale":"2024-03-18T19:05:23.591155+0000","last_undegraded":"2024-03-18T19:05:23.591155+0000","last_fullsized":"2024-03-18T19:05:23.591155+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:21.226687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"88'1","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.114061+0000","last_change":"2024-03-18T19:05:22.171498+0000","last_active":"2024-03-18T19:05:24.114061+0000","last_peered":"2024-03-18T19:05:24.114061+0000","last_clean":"2024-03-18T19:05:24.114061+0000","last_became_active":"2024-03-18T19:05:22.171174+0000","last_became_peered":"2024-03-18T19:05:22.171174+0000","last_unstale":"2024-03-18T19:05:24.114061+0000","last_undegraded":"2024-03-18T19:05:24.114061+0000","last_fullsized":"2024-03-18T19:05:24.114061+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:36:30.649705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.611062+0000","last_change":"2024-03-18T19:05:22.610629+0000","last_active":"2024-03-18T19:05:22.611062+0000","last_peered":"2024-03-18T19:05:22.611062+0000","last_clean":"2024-03-18T19:05:22.611062+0000","last_became_active":"2024-03-18T19:05:22.610450+0000","last_became_peered":"2024-03-18T19:05:22.610450+0000","last_unstale":"2024-03-18T19:05:22.611062+0000","last_undegraded":"2024-03-18T19:05:22.611062+0000","last_fullsized":"2024-03-18T19:05:22.611062+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:18:55.815832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"88'1","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.113909+0000","last_change":"2024-03-18T19:05:22.166498+0000","last_active":"2024-03-18T19:05:24.113909+0000","last_peered":"2024-03-18T19:05:24.113909+0000","last_clean":"2024-03-18T19:05:24.113909+0000","last_became_active":"2024-03-18T19:05:22.166236+0000","last_became_peered":"2024-03-18T19:05:22.166236+0000","last_unstale":"2024-03-18T19:05:24.113909+0000","last_undegraded":"2024-03-18T19:05:24.113909+0000","last_fullsized":"2024-03-18T19:05:24.113909+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:10.178077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590437+0000","last_change":"2024-03-18T19:05:22.593776+0000","last_active":"2024-03-18T19:05:23.590437+0000","last_peered":"2024-03-18T19:05:23.590437+0000","last_clean":"2024-03-18T19:05:23.590437+0000","last_became_active":"2024-03-18T19:05:22.593516+0000","last_became_peered":"2024-03-18T19:05:22.593516+0000","last_unstale":"2024-03-18T19:05:23.590437+0000","last_undegraded":"2024-03-18T19:05:23.590437+0000","last_fullsized":"2024-03-18T19:05:23.590437+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:21:54.604175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.114037+0000","last_change":"2024-03-18T19:05:22.165942+0000","last_active":"2024-03-18T19:05:24.114037+0000","last_peered":"2024-03-18T19:05:24.114037+0000","last_clean":"2024-03-18T19:05:24.114037+0000","last_became_active":"2024-03-18T19:05:22.165456+0000","last_became_peered":"2024-03-18T19:05:22.165456+0000","last_unstale":"2024-03-18T19:05:24.114037+0000","last_undegraded":"2024-03-18T19:05:24.114037+0000","last_fullsized":"2024-03-18T19:05:24.114037+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:55:38.621035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.114099+0000","last_change":"2024-03-18T19:05:22.607465+0000","last_active":"2024-03-18T19:05:24.114099+0000","last_peered":"2024-03-18T19:05:24.114099+0000","last_clean":"2024-03-18T19:05:24.114099+0000","last_became_active":"2024-03-18T19:05:22.607217+0000","last_became_peered":"2024-03-18T19:05:22.607217+0000","last_unstale":"2024-03-18T19:05:24.114099+0000","last_undegraded":"2024-03-18T19:05:24.114099+0000","last_fullsized":"2024-03-18T19:05:24.114099+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:28.449312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590369+0000","last_change":"2024-03-18T19:05:22.170069+0000","last_active":"2024-03-18T19:05:23.590369+0000","last_peered":"2024-03-18T19:05:23.590369+0000","last_clean":"2024-03-18T19:05:23.590369+0000","last_became_active":"2024-03-18T19:05:22.169480+0000","last_became_peered":"2024-03-18T19:05:22.169480+0000","last_unstale":"2024-03-18T19:05:23.590369+0000","last_undegraded":"2024-03-18T19:05:23.590369+0000","last_fullsized":"2024-03-18T19:05:23.590369+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:05:33.245506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.588034+0000","last_change":"2024-03-18T19:05:22.587407+0000","last_active":"2024-03-18T19:05:22.588034+0000","last_peered":"2024-03-18T19:05:22.588034+0000","last_clean":"2024-03-18T19:05:22.588034+0000","last_became_active":"2024-03-18T19:05:22.587142+0000","last_became_peered":"2024-03-18T19:05:22.587142+0000","last_unstale":"2024-03-18T19:05:22.588034+0000","last_undegraded":"2024-03-18T19:05:22.588034+0000","last_fullsized":"2024-03-18T19:05:22.588034+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:49:41.091622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.597935+0000","last_change":"2024-03-18T19:05:22.168515+0000","last_active":"2024-03-18T19:05:22.597935+0000","last_peered":"2024-03-18T19:05:22.597935+0000","last_clean":"2024-03-18T19:05:22.597935+0000","last_became_active":"2024-03-18T19:05:22.168231+0000","last_became_peered":"2024-03-18T19:05:22.168231+0000","last_unstale":"2024-03-18T19:05:22.597935+0000","last_undegraded":"2024-03-18T19:05:22.597935+0000","last_fullsized":"2024-03-18T19:05:22.597935+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:03:12.166084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.114182+0000","last_change":"2024-03-18T19:05:22.595294+0000","last_active":"2024-03-18T19:05:24.114182+0000","last_peered":"2024-03-18T19:05:24.114182+0000","last_clean":"2024-03-18T19:05:24.114182+0000","last_became_active":"2024-03-18T19:05:22.595068+0000","last_became_peered":"2024-03-18T19:05:22.595068+0000","last_unstale":"2024-03-18T19:05:24.114182+0000","last_undegraded":"2024-03-18T19:05:24.114182+0000","last_fullsized":"2024-03-18T19:05:24.114182+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:29:34.389598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966715+0000","last_change":"2024-03-18T19:05:22.175943+0000","last_active":"2024-03-18T19:05:24.966715+0000","last_peered":"2024-03-18T19:05:24.966715+0000","last_clean":"2024-03-18T19:05:24.966715+0000","last_became_active":"2024-03-18T19:05:22.175678+0000","last_became_peered":"2024-03-18T19:05:22.175678+0000","last_unstale":"2024-03-18T19:05:24.966715+0000","last_undegraded":"2024-03-18T19:05:24.966715+0000","last_fullsized":"2024-03-18T19:05:24.966715+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:26:03.949433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590745+0000","last_change":"2024-03-18T19:05:22.595869+0000","last_active":"2024-03-18T19:05:23.590745+0000","last_peered":"2024-03-18T19:05:23.590745+0000","last_clean":"2024-03-18T19:05:23.590745+0000","last_became_active":"2024-03-18T19:05:22.595625+0000","last_became_peered":"2024-03-18T19:05:22.595625+0000","last_unstale":"2024-03-18T19:05:23.590745+0000","last_undegraded":"2024-03-18T19:05:23.590745+0000","last_fullsized":"2024-03-18T19:05:23.590745+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:43.837454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"88'3","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.607281+0000","last_change":"2024-03-18T19:05:22.168466+0000","last_active":"2024-03-18T19:05:22.607281+0000","last_peered":"2024-03-18T19:05:22.607281+0000","last_clean":"2024-03-18T19:05:22.607281+0000","last_became_active":"2024-03-18T19:05:22.168178+0000","last_became_peered":"2024-03-18T19:05:22.168178+0000","last_unstale":"2024-03-18T19:05:22.607281+0000","last_undegraded":"2024-03-18T19:05:22.607281+0000","last_fullsized":"2024-03-18T19:05:22.607281+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:25:20.727978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966790+0000","last_change":"2024-03-18T19:05:22.606106+0000","last_active":"2024-03-18T19:05:24.966790+0000","last_peered":"2024-03-18T19:05:24.966790+0000","last_clean":"2024-03-18T19:05:24.966790+0000","last_became_active":"2024-03-18T19:05:22.605868+0000","last_became_peered":"2024-03-18T19:05:22.605868+0000","last_unstale":"2024-03-18T19:05:24.966790+0000","last_undegraded":"2024-03-18T19:05:24.966790+0000","last_fullsized":"2024-03-18T19:05:24.966790+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:25:21.732481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"88'6","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.615880+0000","last_change":"2024-03-18T19:05:22.176269+0000","last_active":"2024-03-18T19:05:22.615880+0000","last_peered":"2024-03-18T19:05:22.615880+0000","last_clean":"2024-03-18T19:05:22.615880+0000","last_became_active":"2024-03-18T19:05:22.175938+0000","last_became_peered":"2024-03-18T19:05:22.175938+0000","last_unstale":"2024-03-18T19:05:22.615880+0000","last_undegraded":"2024-03-18T19:05:22.615880+0000","last_fullsized":"2024-03-18T19:05:22.615880+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:33:27.744462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966328+0000","last_change":"2024-03-18T19:05:22.606043+0000","last_active":"2024-03-18T19:05:24.966328+0000","last_peered":"2024-03-18T19:05:24.966328+0000","last_clean":"2024-03-18T19:05:24.966328+0000","last_became_active":"2024-03-18T19:05:22.605607+0000","last_became_peered":"2024-03-18T19:05:22.605607+0000","last_unstale":"2024-03-18T19:05:24.966328+0000","last_undegraded":"2024-03-18T19:05:24.966328+0000","last_fullsized":"2024-03-18T19:05:24.966328+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:15:54.617487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"88'2","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.606059+0000","last_change":"2024-03-18T19:05:22.165089+0000","last_active":"2024-03-18T19:05:22.606059+0000","last_peered":"2024-03-18T19:05:22.606059+0000","last_clean":"2024-03-18T19:05:22.606059+0000","last_became_active":"2024-03-18T19:05:22.164835+0000","last_became_peered":"2024-03-18T19:05:22.164835+0000","last_unstale":"2024-03-18T19:05:22.606059+0000","last_undegraded":"2024-03-18T19:05:22.606059+0000","last_fullsized":"2024-03-18T19:05:22.606059+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:21:40.562313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597194+0000","last_change":"2024-03-18T19:05:22.594119+0000","last_active":"2024-03-18T19:05:24.597194+0000","last_peered":"2024-03-18T19:05:24.597194+0000","last_clean":"2024-03-18T19:05:24.597194+0000","last_became_active":"2024-03-18T19:05:22.593866+0000","last_became_peered":"2024-03-18T19:05:22.593866+0000","last_unstale":"2024-03-18T19:05:24.597194+0000","last_undegraded":"2024-03-18T19:05:24.597194+0000","last_fullsized":"2024-03-18T19:05:24.597194+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:08:31.593720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597469+0000","last_change":"2024-03-18T19:05:22.173033+0000","last_active":"2024-03-18T19:05:24.597469+0000","last_peered":"2024-03-18T19:05:24.597469+0000","last_clean":"2024-03-18T19:05:24.597469+0000","last_became_active":"2024-03-18T19:05:22.172832+0000","last_became_peered":"2024-03-18T19:05:22.172832+0000","last_unstale":"2024-03-18T19:05:24.597469+0000","last_undegraded":"2024-03-18T19:05:24.597469+0000","last_fullsized":"2024-03-18T19:05:24.597469+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:25:20.727978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590544+0000","last_change":"2024-03-18T19:05:22.590595+0000","last_active":"2024-03-18T19:05:23.590544+0000","last_peered":"2024-03-18T19:05:23.590544+0000","last_clean":"2024-03-18T19:05:23.590544+0000","last_became_active":"2024-03-18T19:05:22.590149+0000","last_became_peered":"2024-03-18T19:05:22.590149+0000","last_unstale":"2024-03-18T19:05:23.590544+0000","last_undegraded":"2024-03-18T19:05:23.590544+0000","last_fullsized":"2024-03-18T19:05:23.590544+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:03:13.170587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.599394+0000","last_change":"2024-03-18T19:05:22.167212+0000","last_active":"2024-03-18T19:05:22.599394+0000","last_peered":"2024-03-18T19:05:22.599394+0000","last_clean":"2024-03-18T19:05:22.599394+0000","last_became_active":"2024-03-18T19:05:22.166166+0000","last_became_peered":"2024-03-18T19:05:22.166166+0000","last_unstale":"2024-03-18T19:05:22.599394+0000","last_undegraded":"2024-03-18T19:05:22.599394+0000","last_fullsized":"2024-03-18T19:05:22.599394+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:26:58.281923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.604410+0000","last_change":"2024-03-18T19:05:22.603847+0000","last_active":"2024-03-18T19:05:22.604410+0000","last_peered":"2024-03-18T19:05:22.604410+0000","last_clean":"2024-03-18T19:05:22.604410+0000","last_became_active":"2024-03-18T19:05:22.603294+0000","last_became_peered":"2024-03-18T19:05:22.603294+0000","last_unstale":"2024-03-18T19:05:22.604410+0000","last_undegraded":"2024-03-18T19:05:22.604410+0000","last_fullsized":"2024-03-18T19:05:22.604410+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:59:06.740471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"88'8","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966329+0000","last_change":"2024-03-18T19:05:22.176028+0000","last_active":"2024-03-18T19:05:24.966329+0000","last_peered":"2024-03-18T19:05:24.966329+0000","last_clean":"2024-03-18T19:05:24.966329+0000","last_became_active":"2024-03-18T19:05:22.175785+0000","last_became_peered":"2024-03-18T19:05:22.175785+0000","last_unstale":"2024-03-18T19:05:24.966329+0000","last_undegraded":"2024-03-18T19:05:24.966329+0000","last_fullsized":"2024-03-18T19:05:24.966329+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:40.241549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.613652+0000","last_change":"2024-03-18T19:05:22.612982+0000","last_active":"2024-03-18T19:05:22.613652+0000","last_peered":"2024-03-18T19:05:22.613652+0000","last_clean":"2024-03-18T19:05:22.613652+0000","last_became_active":"2024-03-18T19:05:22.612719+0000","last_became_peered":"2024-03-18T19:05:22.612719+0000","last_unstale":"2024-03-18T19:05:22.613652+0000","last_undegraded":"2024-03-18T19:05:22.613652+0000","last_fullsized":"2024-03-18T19:05:22.613652+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:56:09.533605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"88'1","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597558+0000","last_change":"2024-03-18T19:05:22.174068+0000","last_active":"2024-03-18T19:05:24.597558+0000","last_peered":"2024-03-18T19:05:24.597558+0000","last_clean":"2024-03-18T19:05:24.597558+0000","last_became_active":"2024-03-18T19:05:22.173815+0000","last_became_peered":"2024-03-18T19:05:22.173815+0000","last_unstale":"2024-03-18T19:05:24.597558+0000","last_undegraded":"2024-03-18T19:05:24.597558+0000","last_fullsized":"2024-03-18T19:05:24.597558+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:26:58.281923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590810+0000","last_change":"2024-03-18T19:05:22.610966+0000","last_active":"2024-03-18T19:05:23.590810+0000","last_peered":"2024-03-18T19:05:23.590810+0000","last_clean":"2024-03-18T19:05:23.590810+0000","last_became_active":"2024-03-18T19:05:22.610709+0000","last_became_peered":"2024-03-18T19:05:22.610709+0000","last_unstale":"2024-03-18T19:05:23.590810+0000","last_undegraded":"2024-03-18T19:05:23.590810+0000","last_fullsized":"2024-03-18T19:05:23.590810+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:49:03.493777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"88'2","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.613651+0000","last_change":"2024-03-18T19:05:22.174968+0000","last_active":"2024-03-18T19:05:22.613651+0000","last_peered":"2024-03-18T19:05:22.613651+0000","last_clean":"2024-03-18T19:05:22.613651+0000","last_became_active":"2024-03-18T19:05:22.174099+0000","last_became_peered":"2024-03-18T19:05:22.174099+0000","last_unstale":"2024-03-18T19:05:22.613651+0000","last_undegraded":"2024-03-18T19:05:22.613651+0000","last_fullsized":"2024-03-18T19:05:22.613651+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:52:03.908127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966686+0000","last_change":"2024-03-18T19:05:22.606297+0000","last_active":"2024-03-18T19:05:24.966686+0000","last_peered":"2024-03-18T19:05:24.966686+0000","last_clean":"2024-03-18T19:05:24.966686+0000","last_became_active":"2024-03-18T19:05:22.600182+0000","last_became_peered":"2024-03-18T19:05:22.600182+0000","last_unstale":"2024-03-18T19:05:24.966686+0000","last_undegraded":"2024-03-18T19:05:24.966686+0000","last_fullsized":"2024-03-18T19:05:24.966686+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:28.026254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"88'1","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.607596+0000","last_change":"2024-03-18T19:05:22.172066+0000","last_active":"2024-03-18T19:05:22.607596+0000","last_peered":"2024-03-18T19:05:22.607596+0000","last_clean":"2024-03-18T19:05:22.607596+0000","last_became_active":"2024-03-18T19:05:22.171775+0000","last_became_peered":"2024-03-18T19:05:22.171775+0000","last_unstale":"2024-03-18T19:05:22.607596+0000","last_undegraded":"2024-03-18T19:05:22.607596+0000","last_fullsized":"2024-03-18T19:05:22.607596+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:56:44.225942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590783+0000","last_change":"2024-03-18T19:05:22.594727+0000","last_active":"2024-03-18T19:05:23.590783+0000","last_peered":"2024-03-18T19:05:23.590783+0000","last_clean":"2024-03-18T19:05:23.590783+0000","last_became_active":"2024-03-18T19:05:22.594490+0000","last_became_peered":"2024-03-18T19:05:22.594490+0000","last_unstale":"2024-03-18T19:05:23.590783+0000","last_undegraded":"2024-03-18T19:05:23.590783+0000","last_fullsized":"2024-03-18T19:05:23.590783+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:26:59.286426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.114156+0000","last_change":"2024-03-18T19:05:22.172962+0000","last_active":"2024-03-18T19:05:24.114156+0000","last_peered":"2024-03-18T19:05:24.114156+0000","last_clean":"2024-03-18T19:05:24.114156+0000","last_became_active":"2024-03-18T19:05:22.172666+0000","last_became_peered":"2024-03-18T19:05:22.172666+0000","last_unstale":"2024-03-18T19:05:24.114156+0000","last_undegraded":"2024-03-18T19:05:24.114156+0000","last_fullsized":"2024-03-18T19:05:24.114156+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:43:39.238232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.601402+0000","last_change":"2024-03-18T19:05:22.593614+0000","last_active":"2024-03-18T19:05:22.601402+0000","last_peered":"2024-03-18T19:05:22.601402+0000","last_clean":"2024-03-18T19:05:22.601402+0000","last_became_active":"2024-03-18T19:05:22.593316+0000","last_became_peered":"2024-03-18T19:05:22.593316+0000","last_unstale":"2024-03-18T19:05:22.601402+0000","last_undegraded":"2024-03-18T19:05:22.601402+0000","last_fullsized":"2024-03-18T19:05:22.601402+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:01:06.391760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"88'2","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.604723+0000","last_change":"2024-03-18T19:05:22.165965+0000","last_active":"2024-03-18T19:05:22.604723+0000","last_peered":"2024-03-18T19:05:22.604723+0000","last_clean":"2024-03-18T19:05:22.604723+0000","last_became_active":"2024-03-18T19:05:22.165682+0000","last_became_peered":"2024-03-18T19:05:22.165682+0000","last_unstale":"2024-03-18T19:05:22.604723+0000","last_undegraded":"2024-03-18T19:05:22.604723+0000","last_fullsized":"2024-03-18T19:05:22.604723+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:13:23.479719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597337+0000","last_change":"2024-03-18T19:05:22.594957+0000","last_active":"2024-03-18T19:05:24.597337+0000","last_peered":"2024-03-18T19:05:24.597337+0000","last_clean":"2024-03-18T19:05:24.597337+0000","last_became_active":"2024-03-18T19:05:22.594718+0000","last_became_peered":"2024-03-18T19:05:22.594718+0000","last_unstale":"2024-03-18T19:05:24.597337+0000","last_undegraded":"2024-03-18T19:05:24.597337+0000","last_fullsized":"2024-03-18T19:05:24.597337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:59:06.740471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597427+0000","last_change":"2024-03-18T19:05:22.174003+0000","last_active":"2024-03-18T19:05:24.597427+0000","last_peered":"2024-03-18T19:05:24.597427+0000","last_clean":"2024-03-18T19:05:24.597427+0000","last_became_active":"2024-03-18T19:05:22.173175+0000","last_became_peered":"2024-03-18T19:05:22.173175+0000","last_unstale":"2024-03-18T19:05:24.597427+0000","last_undegraded":"2024-03-18T19:05:24.597427+0000","last_fullsized":"2024-03-18T19:05:24.597427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:39:26.510910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590649+0000","last_change":"2024-03-18T19:05:22.590787+0000","last_active":"2024-03-18T19:05:23.590649+0000","last_peered":"2024-03-18T19:05:23.590649+0000","last_clean":"2024-03-18T19:05:23.590649+0000","last_became_active":"2024-03-18T19:05:22.590521+0000","last_became_peered":"2024-03-18T19:05:22.590521+0000","last_unstale":"2024-03-18T19:05:23.590649+0000","last_undegraded":"2024-03-18T19:05:23.590649+0000","last_fullsized":"2024-03-18T19:05:23.590649+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:33:28.748965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.589412+0000","last_change":"2024-03-18T19:05:22.170398+0000","last_active":"2024-03-18T19:05:22.589412+0000","last_peered":"2024-03-18T19:05:22.589412+0000","last_clean":"2024-03-18T19:05:22.589412+0000","last_became_active":"2024-03-18T19:05:22.170144+0000","last_became_peered":"2024-03-18T19:05:22.170144+0000","last_unstale":"2024-03-18T19:05:22.589412+0000","last_undegraded":"2024-03-18T19:05:22.589412+0000","last_fullsized":"2024-03-18T19:05:22.589412+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:09:36.470377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.612646+0000","last_change":"2024-03-18T19:05:22.612364+0000","last_active":"2024-03-18T19:05:22.612646+0000","last_peered":"2024-03-18T19:05:22.612646+0000","last_clean":"2024-03-18T19:05:22.612646+0000","last_became_active":"2024-03-18T19:05:22.612180+0000","last_became_peered":"2024-03-18T19:05:22.612180+0000","last_unstale":"2024-03-18T19:05:22.612646+0000","last_undegraded":"2024-03-18T19:05:22.612646+0000","last_fullsized":"2024-03-18T19:05:22.612646+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:09:35.355553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"88'2","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597644+0000","last_change":"2024-03-18T19:05:22.170451+0000","last_active":"2024-03-18T19:05:24.597644+0000","last_peered":"2024-03-18T19:05:24.597644+0000","last_clean":"2024-03-18T19:05:24.597644+0000","last_became_active":"2024-03-18T19:05:22.169918+0000","last_became_peered":"2024-03-18T19:05:22.169918+0000","last_unstale":"2024-03-18T19:05:24.597644+0000","last_undegraded":"2024-03-18T19:05:24.597644+0000","last_fullsized":"2024-03-18T19:05:24.597644+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:21:40.562313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.591224+0000","last_change":"2024-03-18T19:05:22.590697+0000","last_active":"2024-03-18T19:05:23.591224+0000","last_peered":"2024-03-18T19:05:23.591224+0000","last_clean":"2024-03-18T19:05:23.591224+0000","last_became_active":"2024-03-18T19:05:22.590058+0000","last_became_peered":"2024-03-18T19:05:22.590058+0000","last_unstale":"2024-03-18T19:05:23.591224+0000","last_undegraded":"2024-03-18T19:05:23.591224+0000","last_fullsized":"2024-03-18T19:05:23.591224+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:12:45.172990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597566+0000","last_change":"2024-03-18T19:05:22.172938+0000","last_active":"2024-03-18T19:05:24.597566+0000","last_peered":"2024-03-18T19:05:24.597566+0000","last_clean":"2024-03-18T19:05:24.597566+0000","last_became_active":"2024-03-18T19:05:22.172638+0000","last_became_peered":"2024-03-18T19:05:22.172638+0000","last_unstale":"2024-03-18T19:05:24.597566+0000","last_undegraded":"2024-03-18T19:05:24.597566+0000","last_fullsized":"2024-03-18T19:05:24.597566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:52:03.908127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.597530+0000","last_change":"2024-03-18T19:05:22.595426+0000","last_active":"2024-03-18T19:05:24.597530+0000","last_peered":"2024-03-18T19:05:24.597530+0000","last_clean":"2024-03-18T19:05:24.597530+0000","last_became_active":"2024-03-18T19:05:22.595167+0000","last_became_peered":"2024-03-18T19:05:22.595167+0000","last_unstale":"2024-03-18T19:05:24.597530+0000","last_undegraded":"2024-03-18T19:05:24.597530+0000","last_fullsized":"2024-03-18T19:05:24.597530+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:52:53.706285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.587723+0000","last_change":"2024-03-18T19:05:22.169731+0000","last_active":"2024-03-18T19:05:22.587723+0000","last_peered":"2024-03-18T19:05:22.587723+0000","last_clean":"2024-03-18T19:05:22.587723+0000","last_became_active":"2024-03-18T19:05:22.169456+0000","last_became_peered":"2024-03-18T19:05:22.169456+0000","last_unstale":"2024-03-18T19:05:22.587723+0000","last_undegraded":"2024-03-18T19:05:22.587723+0000","last_fullsized":"2024-03-18T19:05:22.587723+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:07:00.608366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590850+0000","last_change":"2024-03-18T19:05:22.591088+0000","last_active":"2024-03-18T19:05:23.590850+0000","last_peered":"2024-03-18T19:05:23.590850+0000","last_clean":"2024-03-18T19:05:23.590850+0000","last_became_active":"2024-03-18T19:05:22.588674+0000","last_became_peered":"2024-03-18T19:05:22.588674+0000","last_unstale":"2024-03-18T19:05:23.590850+0000","last_undegraded":"2024-03-18T19:05:23.590850+0000","last_fullsized":"2024-03-18T19:05:23.590850+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:52:04.912630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.584908+0000","last_change":"2024-03-18T19:05:22.174819+0000","last_active":"2024-03-18T19:05:22.584908+0000","last_peered":"2024-03-18T19:05:22.584908+0000","last_clean":"2024-03-18T19:05:22.584908+0000","last_became_active":"2024-03-18T19:05:22.174531+0000","last_became_peered":"2024-03-18T19:05:22.174531+0000","last_unstale":"2024-03-18T19:05:22.584908+0000","last_undegraded":"2024-03-18T19:05:22.584908+0000","last_fullsized":"2024-03-18T19:05:22.584908+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:04:07.797394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590313+0000","last_change":"2024-03-18T19:05:22.588932+0000","last_active":"2024-03-18T19:05:23.590313+0000","last_peered":"2024-03-18T19:05:23.590313+0000","last_clean":"2024-03-18T19:05:23.590313+0000","last_became_active":"2024-03-18T19:05:22.588624+0000","last_became_peered":"2024-03-18T19:05:22.588624+0000","last_unstale":"2024-03-18T19:05:23.590313+0000","last_undegraded":"2024-03-18T19:05:23.590313+0000","last_fullsized":"2024-03-18T19:05:23.590313+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:25:21.732481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.595892+0000","last_change":"2024-03-18T19:05:22.166252+0000","last_active":"2024-03-18T19:05:22.595892+0000","last_peered":"2024-03-18T19:05:22.595892+0000","last_clean":"2024-03-18T19:05:22.595892+0000","last_became_active":"2024-03-18T19:05:22.165915+0000","last_became_peered":"2024-03-18T19:05:22.165915+0000","last_unstale":"2024-03-18T19:05:22.595892+0000","last_undegraded":"2024-03-18T19:05:22.595892+0000","last_fullsized":"2024-03-18T19:05:22.595892+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:04:07.797394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.966440+0000","last_change":"2024-03-18T19:05:22.610861+0000","last_active":"2024-03-18T19:05:24.966440+0000","last_peered":"2024-03-18T19:05:24.966440+0000","last_clean":"2024-03-18T19:05:24.966440+0000","last_became_active":"2024-03-18T19:05:22.609566+0000","last_became_peered":"2024-03-18T19:05:22.609566+0000","last_unstale":"2024-03-18T19:05:24.966440+0000","last_undegraded":"2024-03-18T19:05:24.966440+0000","last_fullsized":"2024-03-18T19:05:24.966440+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:28.990836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.114023+0000","last_change":"2024-03-18T19:05:22.167040+0000","last_active":"2024-03-18T19:05:24.114023+0000","last_peered":"2024-03-18T19:05:24.114023+0000","last_clean":"2024-03-18T19:05:24.114023+0000","last_became_active":"2024-03-18T19:05:22.166772+0000","last_became_peered":"2024-03-18T19:05:22.166772+0000","last_unstale":"2024-03-18T19:05:24.114023+0000","last_undegraded":"2024-03-18T19:05:24.114023+0000","last_fullsized":"2024-03-18T19:05:24.114023+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:52:11.707326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.611040+0000","last_change":"2024-03-18T19:05:22.610497+0000","last_active":"2024-03-18T19:05:22.611040+0000","last_peered":"2024-03-18T19:05:22.611040+0000","last_clean":"2024-03-18T19:05:22.611040+0000","last_became_active":"2024-03-18T19:05:22.610210+0000","last_became_peered":"2024-03-18T19:05:22.610210+0000","last_unstale":"2024-03-18T19:05:22.611040+0000","last_undegraded":"2024-03-18T19:05:22.611040+0000","last_fullsized":"2024-03-18T19:05:22.611040+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:01:35.139881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.114053+0000","last_change":"2024-03-18T19:05:22.165747+0000","last_active":"2024-03-18T19:05:24.114053+0000","last_peered":"2024-03-18T19:05:24.114053+0000","last_clean":"2024-03-18T19:05:24.114053+0000","last_became_active":"2024-03-18T19:05:22.165072+0000","last_became_peered":"2024-03-18T19:05:22.165072+0000","last_unstale":"2024-03-18T19:05:24.114053+0000","last_undegraded":"2024-03-18T19:05:24.114053+0000","last_fullsized":"2024-03-18T19:05:24.114053+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:08:04.865734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.591066+0000","last_change":"2024-03-18T19:05:22.594091+0000","last_active":"2024-03-18T19:05:23.591066+0000","last_peered":"2024-03-18T19:05:23.591066+0000","last_clean":"2024-03-18T19:05:23.591066+0000","last_became_active":"2024-03-18T19:05:22.593715+0000","last_became_peered":"2024-03-18T19:05:22.593715+0000","last_unstale":"2024-03-18T19:05:23.591066+0000","last_undegraded":"2024-03-18T19:05:23.591066+0000","last_fullsized":"2024-03-18T19:05:23.591066+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:05:47.646282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.584317+0000","last_change":"2024-03-18T19:05:22.166661+0000","last_active":"2024-03-18T19:05:22.584317+0000","last_peered":"2024-03-18T19:05:22.584317+0000","last_clean":"2024-03-18T19:05:22.584317+0000","last_became_active":"2024-03-18T19:05:22.166368+0000","last_became_peered":"2024-03-18T19:05:22.166368+0000","last_unstale":"2024-03-18T19:05:22.584317+0000","last_undegraded":"2024-03-18T19:05:22.584317+0000","last_fullsized":"2024-03-18T19:05:22.584317+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:21:53.599672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":10,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:23.590607+0000","last_change":"2024-03-18T19:05:22.589062+0000","last_active":"2024-03-18T19:05:23.590607+0000","last_peered":"2024-03-18T19:05:23.590607+0000","last_clean":"2024-03-18T19:05:23.590607+0000","last_became_active":"2024-03-18T19:05:22.588764+0000","last_became_peered":"2024-03-18T19:05:22.588764+0000","last_unstale":"2024-03-18T19:05:23.590607+0000","last_undegraded":"2024-03-18T19:05:23.590607+0000","last_fullsized":"2024-03-18T19:05:23.590607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:22.149223+0000","last_clean_scrub_stamp":"2024-03-18T19:05:22.149223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:08:31.593720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T19:05:24.113904+0000","last_change":"2024-03-18T19:05:22.163773+0000","last_active":"2024-03-18T19:05:24.113904+0000","last_peered":"2024-03-18T19:05:24.113904+0000","last_clean":"2024-03-18T19:05:24.113904+0000","last_became_active":"2024-03-18T19:05:22.163466+0000","last_became_peered":"2024-03-18T19:05:22.163466+0000","last_unstale":"2024-03-18T19:05:24.113904+0000","last_undegraded":"2024-03-18T19:05:24.113904+0000","last_fullsized":"2024-03-18T19:05:24.113904+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:05:21.144720+0000","last_clean_scrub_stamp":"2024-03-18T19:05:21.144720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:19:38.888470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"78'161","reported_seq":384,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-18T19:05:22.153788+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T19:05:22.153788+0000","last_peered":"2024-03-18T19:05:22.153788+0000","last_clean":"2024-03-18T19:05:22.153788+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T19:05:22.153788+0000","last_undegraded":"2024-03-18T19:05:22.153788+0000","last_fullsized":"2024-03-18T19:05:22.153788+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58996,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312844,"statfs":{"total":96636764160,"available":96576352256,"internally_reserved":0,"allocated":1679360,"data_stored":1123514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.57699999999999996,"15min":0.56999999999999995},"min":{"1min":0.44800000000000001,"5min":0.27800000000000002,"15min":0.247},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.57899999999999996,"15min":0.58199999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.26700000000000002},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.58499999999999996}]},{"osd":1,"last update":"Mon Mar 18 19:05:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.56299999999999994,"15min":0.57299999999999995},"min":{"1min":0.46500000000000002,"5min":0.32300000000000001,"15min":0.307},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":1.484},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41499999999999998,"5min":0.378,"15min":0.36599999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.92000000000000004},"last":0.56100000000000005}]},{"osd":2,"last update":"Mon Mar 18 19:05:04 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.56499999999999995,"15min":0.57799999999999996},"min":{"1min":0.46000000000000002,"5min":0.34499999999999997,"15min":0.27800000000000002},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.5129999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.30199999999999999},"max":{"1min":1.327,"5min":1.327,"15min":1.536},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Mar 18 19:05:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.56200000000000006,"15min":0.58199999999999996},"min":{"1min":0.371,"5min":0.318,"15min":0.311},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":1.5489999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.59099999999999997,"15min":0.60399999999999998},"min":{"1min":0.498,"5min":0.38800000000000001,"15min":0.38200000000000001},"max":{"1min":0.78200000000000003,"5min":0.78600000000000003,"15min":1.5660000000000001},"last":0.60899999999999999}]},{"osd":4,"last update":"Mon Mar 18 19:05:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.44900000000000001,"15min":0.45600000000000002},"min":{"1min":0.32800000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.47299999999999998,"15min":0.47999999999999998},"min":{"1min":0.34799999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.70899999999999996,"15min":0.73199999999999998},"last":0.46500000000000002}]},{"osd":5,"last update":"Mon Mar 18 19:05:04 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47799999999999998,"15min":0.48099999999999998},"min":{"1min":0.35399999999999998,"5min":0.27100000000000002,"15min":0.26000000000000001},"max":{"1min":0.751,"5min":0.751,"15min":1.2310000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.51600000000000001,"15min":0.52200000000000002},"min":{"1min":0.376,"5min":0.30299999999999999,"15min":0.28799999999999998},"max":{"1min":0.76900000000000002,"5min":0.78200000000000003,"15min":1.1220000000000001},"last":0.53200000000000003}]},{"osd":6,"last update":"Mon Mar 18 19:05:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54800000000000004,"15min":0.54900000000000004},"min":{"1min":0.41599999999999998,"5min":0.374,"15min":0.34300000000000003},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":1.069},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53000000000000003,"15min":0.53800000000000003},"min":{"1min":0.44500000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.71999999999999997,"5min":0.752,"15min":0.79600000000000004},"last":0.65500000000000003}]}]},{"osd":6,"up_from":13,"seq":55834575035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1085440,"data_stored":539024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56899999999999995,"15min":0.57199999999999995},"min":{"1min":0.33000000000000002,"5min":0.28499999999999998,"15min":0.25700000000000001},"max":{"1min":0.69799999999999995,"5min":0.75600000000000001,"15min":3.274},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56299999999999994,"15min":0.58099999999999996},"min":{"1min":0.37,"5min":0.34699999999999998,"15min":0.27400000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":3.3929999999999998},"last":0.39400000000000002}]},{"osd":1,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57699999999999996,"15min":0.56799999999999995},"min":{"1min":0.38100000000000001,"5min":0.316,"15min":0.30599999999999999},"max":{"1min":0.73899999999999999,"5min":0.77400000000000002,"15min":0.94099999999999995},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56200000000000006,"15min":0.56999999999999995},"min":{"1min":0.42699999999999999,"5min":0.32000000000000001,"15min":0.311},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.90700000000000003},"last":0.36899999999999999}]},{"osd":2,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58399999999999996,"15min":0.57599999999999996},"min":{"1min":0.42799999999999999,"5min":0.32700000000000001,"15min":0.29999999999999999},"max":{"1min":0.75700000000000001,"5min":0.80300000000000005,"15min":0.81299999999999994},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57199999999999995,"15min":0.58299999999999996},"min":{"1min":0.36699999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.74199999999999999,"5min":0.83199999999999996,"15min":2.2370000000000001},"last":0.53300000000000003}]},{"osd":3,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59199999999999997,"15min":0.58499999999999996},"min":{"1min":0.41199999999999998,"5min":0.40000000000000002,"15min":0.32100000000000001},"max":{"1min":0.75600000000000001,"5min":0.89600000000000002,"15min":1.0369999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.373,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":1.595},"last":0.54500000000000004}]},{"osd":4,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.34000000000000002,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.76300000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47599999999999998,"15min":0.48099999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":1.8819999999999999},"last":0.503}]},{"osd":5,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.46899999999999997,"15min":0.47399999999999998},"min":{"1min":0.33600000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":1.022},"last":0.433},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51200000000000001,"15min":0.51600000000000001},"min":{"1min":0.35699999999999998,"5min":0.35499999999999998,"15min":0.32200000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.81599999999999995},"last":0.60299999999999998}]},{"osd":7,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.52700000000000002,"15min":0.52600000000000002},"min":{"1min":0.34300000000000003,"5min":0.34100000000000003,"15min":0.313},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":1.532},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53000000000000003,"15min":0.53300000000000003},"min":{"1min":0.39500000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":1.978},"last":0.58899999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575035,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32196,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339644,"statfs":{"total":96636764160,"available":96603795456,"internally_reserved":0,"allocated":1097728,"data_stored":534698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.375,"15min":0.373},"min":{"1min":0.23200000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.83599999999999997},"last":0.253},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42499999999999999,"15min":0.41499999999999998},"min":{"1min":0.25700000000000001,"5min":0.23799999999999999,"15min":0.23400000000000001},"max":{"1min":0.65200000000000002,"5min":0.73699999999999999,"15min":0.90400000000000003},"last":0.33100000000000002}]},{"osd":2,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41599999999999998,"15min":0.40400000000000003},"min":{"1min":0.27800000000000002,"5min":0.26500000000000001,"15min":0.23300000000000001},"max":{"1min":0.56699999999999995,"5min":0.65400000000000003,"15min":0.88400000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.45300000000000001,"15min":0.44500000000000001},"min":{"1min":0.34300000000000003,"5min":0.29699999999999999,"15min":0.23499999999999999},"max":{"1min":0.60699999999999998,"5min":0.68700000000000006,"15min":0.94999999999999996},"last":0.39900000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.27700000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.86799999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.435,"5min":0.42799999999999999,"15min":0.42099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28100000000000003,"15min":0.25600000000000001},"max":{"1min":0.65100000000000002,"5min":0.77000000000000002,"15min":0.93100000000000005},"last":0.27800000000000002}]},{"osd":4,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63800000000000001,"15min":0.63300000000000001},"min":{"1min":0.33400000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.92300000000000004,"5min":0.93999999999999995,"15min":1.052},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.63800000000000001,"15min":0.64400000000000002},"min":{"1min":0.46800000000000003,"5min":0.36899999999999999,"15min":0.34499999999999997},"max":{"1min":0.95799999999999996,"5min":1.018,"15min":1.0269999999999999},"last":0.65400000000000003}]},{"osd":5,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63300000000000001,"15min":0.64200000000000002},"min":{"1min":0.434,"5min":0.40100000000000002,"15min":0.38600000000000001},"max":{"1min":0.877,"5min":0.88500000000000001,"15min":1.0760000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.65600000000000003,"15min":0.65700000000000003},"min":{"1min":0.47099999999999997,"5min":0.41099999999999998,"15min":0.40999999999999998},"max":{"1min":0.80900000000000005,"5min":1.006,"15min":1.1319999999999999},"last":0.64100000000000001}]},{"osd":6,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.93500000000000005,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67500000000000004,"15min":0.66900000000000004},"min":{"1min":0.434,"5min":0.42599999999999999,"15min":0.39100000000000001},"max":{"1min":0.93799999999999994,"5min":0.96799999999999997,"15min":1.0089999999999999},"last":0.66800000000000004}]},{"osd":7,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66000000000000003},"min":{"1min":0.48599999999999999,"5min":0.44900000000000001,"15min":0.36699999999999999},"max":{"1min":0.90600000000000003,"5min":0.92900000000000005,"15min":1.0660000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66700000000000004,"15min":0.66400000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.39900000000000002},"max":{"1min":0.88100000000000001,"5min":0.94299999999999995,"15min":1.1499999999999999},"last":0.59799999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575036,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36844,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334996,"statfs":{"total":96636764160,"available":96599035904,"internally_reserved":0,"allocated":1662976,"data_stored":1122738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.377,"15min":0.377},"min":{"1min":0.23100000000000001,"5min":0.19400000000000001,"15min":0.192},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.621},"last":0.254},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.30299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.55400000000000005,"5min":0.61699999999999999,"15min":0.66400000000000003},"last":0.35799999999999998}]},{"osd":2,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41599999999999998,"15min":0.41699999999999998},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.58999999999999997,"5min":0.60499999999999998,"15min":0.627},"last":0.373},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.30499999999999999,"5min":0.25,"15min":0.25},"max":{"1min":0.627,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.31}]},{"osd":3,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.40699999999999997,"15min":0.42999999999999999},"min":{"1min":0.30299999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.57499999999999996,"5min":0.61299999999999999,"15min":5.7359999999999998},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46500000000000002,"15min":0.48799999999999999},"min":{"1min":0.32000000000000001,"5min":0.28799999999999998,"15min":0.27700000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":5.9329999999999998},"last":0.34000000000000002}]},{"osd":4,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.63800000000000001,"15min":0.63900000000000001},"min":{"1min":0.49099999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.91800000000000004,"5min":1.0229999999999999,"15min":1.22},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.45900000000000002,"5min":0.315,"15min":0.315},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.1020000000000001},"last":0.502}]},{"osd":5,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.47999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.96199999999999997,"5min":0.98099999999999998,"15min":2.9649999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.65600000000000003,"15min":0.65800000000000003},"min":{"1min":0.38,"5min":0.31,"15min":0.31},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.2609999999999999},"last":0.59099999999999997}]},{"osd":6,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.64900000000000002,"15min":0.65300000000000002},"min":{"1min":0.40500000000000003,"5min":0.39400000000000002,"15min":0.38100000000000001},"max":{"1min":0.90100000000000002,"5min":0.94599999999999995,"15min":1.133},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.66500000000000004,"15min":0.67600000000000005},"min":{"1min":0.42699999999999999,"5min":0.41199999999999998,"15min":0.40999999999999998},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":2.9300000000000002},"last":0.59899999999999998}]},{"osd":7,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.66600000000000004,"15min":0.66700000000000004},"min":{"1min":0.51900000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.2849999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.66500000000000004,"15min":0.67000000000000004},"min":{"1min":0.44,"5min":0.437,"15min":0.38500000000000001},"max":{"1min":0.94299999999999995,"5min":0.96299999999999997,"15min":1.002},"last":0.58099999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575036,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58420,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94313420,"statfs":{"total":96636764160,"available":96576942080,"internally_reserved":0,"allocated":1081344,"data_stored":538990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38200000000000001,"15min":0.39500000000000002},"min":{"1min":0.249,"5min":0.216,"15min":0.17799999999999999},"max":{"1min":0.66600000000000004,"5min":0.78300000000000003,"15min":9.1219999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.46500000000000002,"15min":0.47599999999999998},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.68899999999999995,"5min":0.77200000000000002,"15min":9.1560000000000006},"last":0.311}]},{"osd":1,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47999999999999998,"15min":0.49299999999999999},"min":{"1min":0.376,"5min":0.29399999999999998,"15min":0.27100000000000002},"max":{"1min":0.69499999999999995,"5min":0.90500000000000003,"15min":9.2460000000000004},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41299999999999998,"15min":0.42499999999999999},"min":{"1min":0.23899999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.61399999999999999,"5min":0.66700000000000004,"15min":12.932},"last":0.32500000000000001}]},{"osd":3,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.42199999999999999,"15min":0.42799999999999999},"min":{"1min":0.29799999999999999,"5min":0.24199999999999999,"15min":0.22500000000000001},"max":{"1min":0.58599999999999997,"5min":1.4019999999999999,"15min":9.4120000000000008},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51300000000000001,"15min":0.52000000000000002},"min":{"1min":0.33800000000000002,"5min":0.29499999999999998,"15min":0.27700000000000002},"max":{"1min":0.67100000000000004,"5min":0.80100000000000005,"15min":9.2680000000000007},"last":0.36499999999999999}]},{"osd":4,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63,"15min":0.61799999999999999},"min":{"1min":0.39400000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.98199999999999998,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63400000000000001,"15min":0.65200000000000002},"min":{"1min":0.441,"5min":0.35899999999999999,"15min":0.32100000000000001},"max":{"1min":0.874,"5min":1.4730000000000001,"15min":9.3190000000000008},"last":0.437}]},{"osd":5,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65300000000000002,"15min":0.66800000000000004},"min":{"1min":0.45300000000000001,"5min":0.34699999999999998,"15min":0.309},"max":{"1min":0.89600000000000002,"5min":1.8109999999999999,"15min":9.1940000000000008},"last":0.497},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65800000000000003,"15min":0.64000000000000001},"min":{"1min":0.40000000000000002,"5min":0.36799999999999999,"15min":0.33400000000000002},"max":{"1min":0.90300000000000002,"5min":1.948,"15min":1.948},"last":0.46300000000000002}]},{"osd":6,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.67700000000000005,"15min":0.67900000000000005},"min":{"1min":0.40999999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.92300000000000004,"5min":2.9470000000000001,"15min":9.3559999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69299999999999995,"15min":0.70899999999999996},"min":{"1min":0.48899999999999999,"5min":0.38100000000000001,"15min":0.37},"max":{"1min":1.0509999999999999,"5min":1.083,"15min":9.2219999999999995},"last":0.58299999999999996}]},{"osd":7,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.66300000000000003,"15min":0.65200000000000002},"min":{"1min":0.42599999999999999,"5min":0.40999999999999998,"15min":0.38200000000000001},"max":{"1min":0.93600000000000005,"5min":1.083,"15min":1.083},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66800000000000004,"15min":0.68400000000000005},"min":{"1min":0.51800000000000002,"5min":0.371,"15min":0.34599999999999997},"max":{"1min":1.032,"5min":1.4350000000000001,"15min":9.3829999999999991},"last":0.45300000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575036,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36276,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335564,"statfs":{"total":96636764160,"available":96599617536,"internally_reserved":0,"allocated":1081344,"data_stored":538990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.38400000000000001,"15min":0.38300000000000001},"min":{"1min":0.249,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63900000000000001,"5min":1.204,"15min":1.4550000000000001},"last":0.312},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.42199999999999999,"15min":0.39900000000000002},"min":{"1min":0.29099999999999998,"5min":0.28000000000000003,"15min":0.23000000000000001},"max":{"1min":0.57599999999999996,"5min":1.137,"15min":1.5169999999999999},"last":0.40699999999999997}]},{"osd":1,"last update":"Mon Mar 18 19:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.41199999999999998,"15min":0.40100000000000002},"min":{"1min":0.27800000000000002,"5min":0.248,"15min":0.221},"max":{"1min":0.61699999999999999,"5min":1.1220000000000001,"15min":1.2909999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.45800000000000002,"15min":0.44700000000000001},"min":{"1min":0.34899999999999998,"5min":0.28499999999999998,"15min":0.245},"max":{"1min":0.628,"5min":1.1080000000000001,"15min":1.355},"last":0.42799999999999999}]},{"osd":2,"last update":"Mon Mar 18 19:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.46300000000000002,"15min":0.45000000000000001},"min":{"1min":0.34000000000000002,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.65100000000000002,"5min":1.1879999999999999,"15min":1.3160000000000001},"last":0.504},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.35499999999999998,"5min":0.33300000000000002,"15min":0.28599999999999998},"max":{"1min":0.71499999999999997,"5min":1.2290000000000001,"15min":1.4810000000000001},"last":0.49099999999999999}]},{"osd":4,"last update":"Mon Mar 18 19:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63200000000000001,"15min":0.625},"min":{"1min":0.40600000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81399999999999995,"5min":1.871,"15min":1.871},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63800000000000001,"15min":0.63500000000000001},"min":{"1min":0.441,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.90600000000000003,"5min":1.764,"15min":1.764},"last":0.60399999999999998}]},{"osd":5,"last update":"Mon Mar 18 19:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.68600000000000005,"15min":0.66200000000000003},"min":{"1min":0.44800000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.0169999999999999,"5min":2.5329999999999999,"15min":2.5329999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.68799999999999994},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.437},"max":{"1min":0.94399999999999995,"5min":1.794,"15min":1.794},"last":0.72899999999999998}]},{"osd":6,"last update":"Mon Mar 18 19:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.68999999999999995,"15min":0.67100000000000004},"min":{"1min":0.51300000000000001,"5min":0.41499999999999998,"15min":0.377},"max":{"1min":0.86799999999999999,"5min":2.7389999999999999,"15min":2.7389999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.69199999999999995,"15min":0.68100000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.98299999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.81999999999999995}]},{"osd":7,"last update":"Mon Mar 18 19:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.52600000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91200000000000003,"5min":1.732,"15min":1.732},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.69299999999999995,"15min":0.66900000000000004},"min":{"1min":0.51400000000000001,"5min":0.439,"15min":0.40500000000000003},"max":{"1min":0.95999999999999996,"5min":2.7050000000000001,"15min":2.7050000000000001},"last":0.64500000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575036,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1081344,"data_stored":533934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.246},"max":{"1min":0.73299999999999998,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56399999999999995,"15min":0.58099999999999996},"min":{"1min":0.40300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.76100000000000001,"5min":0.77600000000000002,"15min":1.0860000000000001},"last":0.50800000000000001}]},{"osd":1,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57399999999999995,"15min":0.58399999999999996},"min":{"1min":0.439,"5min":0.34499999999999997,"15min":0.316},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":1.1679999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55700000000000005,"15min":0.57499999999999996},"min":{"1min":0.41599999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":1.218},"last":0.437}]},{"osd":2,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56299999999999994,"15min":0.57299999999999995},"min":{"1min":0.45000000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72699999999999998,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57399999999999995,"15min":0.59899999999999998},"min":{"1min":0.40999999999999998,"5min":0.36799999999999999,"15min":0.32600000000000001},"max":{"1min":0.77200000000000002,"5min":0.78100000000000003,"15min":1.204},"last":0.64600000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56899999999999995,"15min":0.58599999999999997},"min":{"1min":0.439,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.74099999999999999,"5min":0.75600000000000001,"15min":1.1799999999999999},"last":0.495},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57799999999999996,"15min":0.59599999999999997},"min":{"1min":0.41799999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67700000000000005,"5min":1.121,"15min":1.121},"last":0.57999999999999996}]},{"osd":5,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.44600000000000001,"15min":0.46500000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.245},"max":{"1min":0.55900000000000005,"5min":0.73499999999999999,"15min":0.78900000000000003},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.435,"5min":0.46100000000000002,"15min":0.48299999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27200000000000002},"max":{"1min":0.626,"5min":0.64400000000000002,"15min":0.77800000000000002},"last":0.66500000000000004}]},{"osd":6,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.46800000000000003,"15min":0.48499999999999999},"min":{"1min":0.33900000000000002,"5min":0.32700000000000001,"15min":0.29499999999999998},"max":{"1min":0.66500000000000004,"5min":0.68600000000000005,"15min":0.81799999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.497,"15min":0.52300000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.28999999999999998},"max":{"1min":0.59799999999999998,"5min":0.72999999999999998,"15min":0.88900000000000001},"last":0.71099999999999997}]},{"osd":7,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.498,"15min":0.52100000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.628,"5min":0.70899999999999996,"15min":0.75800000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52600000000000002,"15min":0.55300000000000005},"min":{"1min":0.38300000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70599999999999996,"5min":0.75900000000000001,"15min":0.91900000000000004},"last":0.621}]}]},{"osd":5,"up_from":13,"seq":55834575035,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50212,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321628,"statfs":{"total":96636764160,"available":96585347072,"internally_reserved":0,"allocated":1073152,"data_stored":532392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56899999999999995,"15min":0.58599999999999997},"min":{"1min":0.38300000000000001,"5min":0.26800000000000002,"15min":0.24399999999999999},"max":{"1min":0.77000000000000002,"5min":0.78400000000000003,"15min":6.3079999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57899999999999996,"15min":0.57299999999999995},"min":{"1min":0.373,"5min":0.33700000000000002,"15min":0.25800000000000001},"max":{"1min":0.78800000000000003,"5min":0.79800000000000004,"15min":1.208},"last":0.55800000000000005}]},{"osd":1,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.32900000000000001,"5min":0.32600000000000001,"15min":0.28399999999999997},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.29799999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":6.2549999999999999},"last":0.51400000000000001}]},{"osd":2,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.58599999999999997,"15min":0.59699999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.28399999999999997},"max":{"1min":1.153,"5min":1.153,"15min":6.2999999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56499999999999995,"15min":0.623},"min":{"1min":0.39200000000000002,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":0.73999999999999999,"5min":0.748,"15min":17.387},"last":0.54100000000000004}]},{"osd":3,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.35799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":6.3330000000000002},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.212,"5min":1.212,"15min":1.2390000000000001},"last":0.59199999999999997}]},{"osd":4,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.47099999999999997,"15min":0.46800000000000003},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.497,"15min":0.495},"min":{"1min":0.32700000000000001,"5min":0.29499999999999998,"15min":0.23100000000000001},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.42299999999999999}]},{"osd":6,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.46700000000000003,"15min":0.47899999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.70999999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.504,"15min":0.51000000000000001},"min":{"1min":0.32100000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66300000000000003,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.48799999999999999}]},{"osd":7,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.53100000000000003},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55300000000000005,"15min":0.55200000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.28499999999999998},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.67700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T19:05:27.142 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T19:05:27.142 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-18T19:05:27.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.315+0000 7f63743ed700 1 Processor -- start 2024-03-18T19:05:27.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.315+0000 7f63743ed700 1 -- start start 2024-03-18T19:05:27.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.316+0000 7f63743ed700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f636c06ae60 0x7f636c161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:27.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.316+0000 7f63743ed700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636c0cad60 con 0x7f636c06eda0 2024-03-18T19:05:27.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.316+0000 7f63743ed700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636c0cb040 con 0x7f636c06f180 2024-03-18T19:05:27.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.316+0000 7f63743ed700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f636c0cb320 con 0x7f636c06ae60 2024-03-18T19:05:27.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.316+0000 7f6372189700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f636c06eda0 0x7f636c0ca580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57676/0 (socket says 172.21.15.1:57676) 2024-03-18T19:05:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.316+0000 7f6372189700 1 -- 172.21.15.1:0/955360297 learned_addr learned my addr 172.21.15.1:0/955360297 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.317+0000 7f637298a700 1 --2- 172.21.15.1:0/955360297 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f636c06ae60 0x7f636c161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.317+0000 7f6371187700 1 -- 172.21.15.1:0/955360297 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761106435 0 0) 0x7f636c0cad60 con 0x7f636c06eda0 2024-03-18T19:05:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.317+0000 7f6371187700 1 -- 172.21.15.1:0/955360297 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63580035b0 con 0x7f636c06eda0 2024-03-18T19:05:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.317+0000 7f6371187700 1 -- 172.21.15.1:0/955360297 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804657645 0 0) 0x7f636c0cb040 con 0x7f636c06f180 2024-03-18T19:05:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.317+0000 7f637298a700 1 -- 172.21.15.1:0/955360297 >> v1:172.21.15.165:6789/0 conn(0x7f636c06f180 legacy=0x7f636c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.317+0000 7f637298a700 1 -- 172.21.15.1:0/955360297 >> v1:172.21.15.1:6789/0 conn(0x7f636c06eda0 legacy=0x7f636c0ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.317+0000 7f637298a700 1 -- 172.21.15.1:0/955360297 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636c0ce220 con 0x7f636c06ae60 2024-03-18T19:05:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.317+0000 7f637298a700 1 --2- 172.21.15.1:0/955360297 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f636c06ae60 0x7f636c161c60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6364009dc0 tx=0x7f636400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=567b4c2d97a0acf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.317+0000 7f6371187700 1 -- 172.21.15.1:0/955360297 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636400e070 con 0x7f636c06ae60 2024-03-18T19:05:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.317+0000 7f6371187700 1 -- 172.21.15.1:0/955360297 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6364003a20 con 0x7f636c06ae60 2024-03-18T19:05:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.317+0000 7f6371187700 1 -- 172.21.15.1:0/955360297 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6368001670 con 0x7f636c06ae60 2024-03-18T19:05:27.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.318+0000 7f63743ed700 1 -- 172.21.15.1:0/955360297 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f636c06ae60 msgr2=0x7f636c161c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:27.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.318+0000 7f63743ed700 1 --2- 172.21.15.1:0/955360297 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f636c06ae60 0x7f636c161c60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6364009dc0 tx=0x7f636400b330 comp rx=0 tx=0).stop 2024-03-18T19:05:27.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.318+0000 7f63743ed700 1 -- 172.21.15.1:0/955360297 shutdown_connections 2024-03-18T19:05:27.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.318+0000 7f63743ed700 1 --2- 172.21.15.1:0/955360297 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f636c06ae60 0x7f636c161c60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:27.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.318+0000 7f63743ed700 1 -- 172.21.15.1:0/955360297 >> 172.21.15.1:0/955360297 conn(0x7f636c0cff30 msgr2=0x7f636c0d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:27.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.318+0000 7f63743ed700 1 -- 172.21.15.1:0/955360297 shutdown_connections 2024-03-18T19:05:27.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.318+0000 7f63743ed700 1 -- 172.21.15.1:0/955360297 wait complete. 2024-03-18T19:05:27.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.319+0000 7f63743ed700 1 Processor -- start 2024-03-18T19:05:27.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.319+0000 7f63743ed700 1 -- start start 2024-03-18T19:05:27.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.320+0000 7f63743ed700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f636c06ae60 0x7f636c0b3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:27.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.320+0000 7f63743ed700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f636c06eda0 0x7f636c1f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:27.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.320+0000 7f63743ed700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f636c06f180 0x7f636c0b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:27.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.320+0000 7f63743ed700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c1f2d70 con 0x7f636c06eda0 2024-03-18T19:05:27.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.320+0000 7f63743ed700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c0b3b10 con 0x7f636c06f180 2024-03-18T19:05:27.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.320+0000 7f63743ed700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f636c0b3df0 con 0x7f636c06ae60 2024-03-18T19:05:27.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.320+0000 7f6371988700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f636c06eda0 0x7f636c1f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:27.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.320+0000 7f6371988700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f636c06eda0 0x7f636c1f2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33716/0 (socket says 172.21.15.1:33716) 2024-03-18T19:05:27.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.320+0000 7f6371988700 1 -- 172.21.15.1:0/963260511 learned_addr learned my addr 172.21.15.1:0/963260511 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:27.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.320+0000 7f6372189700 1 --2- 172.21.15.1:0/963260511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f636c06ae60 0x7f636c0b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:27.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.320+0000 7f637298a700 1 --2- 172.21.15.1:0/963260511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f636c06f180 0x7f636c0b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:27.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.320+0000 7f6371988700 1 -- 172.21.15.1:0/963260511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f636c06ae60 msgr2=0x7f636c0b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:27.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.320+0000 7f6371988700 1 --2- 172.21.15.1:0/963260511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f636c06ae60 0x7f636c0b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:27.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.320+0000 7f6371988700 1 -- 172.21.15.1:0/963260511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f636c06f180 msgr2=0x7f636c0b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:27.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.320+0000 7f6371988700 1 --2- 172.21.15.1:0/963260511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f636c06f180 0x7f636c0b50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:27.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.320+0000 7f6371988700 1 -- 172.21.15.1:0/963260511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6364009a70 con 0x7f636c06eda0 2024-03-18T19:05:27.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.321+0000 7f6371988700 1 --2- 172.21.15.1:0/963260511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f636c06eda0 0x7f636c1f2830 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f635c00c450 tx=0x7f635c00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:27.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.321+0000 7f6362ffd700 1 -- 172.21.15.1:0/963260511 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c0054e0 con 0x7f636c06eda0 2024-03-18T19:05:27.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.321+0000 7f6362ffd700 1 -- 172.21.15.1:0/963260511 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635c003680 con 0x7f636c06eda0 2024-03-18T19:05:27.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.321+0000 7f6362ffd700 1 -- 172.21.15.1:0/963260511 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c003a10 con 0x7f636c06eda0 2024-03-18T19:05:27.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.321+0000 7f63743ed700 1 -- 172.21.15.1:0/963260511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636c0b40d0 con 0x7f636c06eda0 2024-03-18T19:05:27.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.321+0000 7f63743ed700 1 -- 172.21.15.1:0/963260511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f636c0b5620 con 0x7f636c06eda0 2024-03-18T19:05:27.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.322+0000 7f63743ed700 1 -- 172.21.15.1:0/963260511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f636c0676a0 con 0x7f636c06eda0 2024-03-18T19:05:27.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.322+0000 7f6362ffd700 1 -- 172.21.15.1:0/963260511 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f635c003b70 con 0x7f636c06eda0 2024-03-18T19:05:27.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.323+0000 7f6362ffd700 1 --2- 172.21.15.1:0/963260511 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f635006adb0 0x7f635006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:27.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.323+0000 7f6362ffd700 1 -- 172.21.15.1:0/963260511 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f635c088a00 con 0x7f636c06eda0 2024-03-18T19:05:27.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.323+0000 7f6372189700 1 --2- 172.21.15.1:0/963260511 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f635006adb0 0x7f635006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:27.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.323+0000 7f6372189700 1 --2- 172.21.15.1:0/963260511 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f635006adb0 0x7f635006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6368001b70 tx=0x7f6368001b00 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:27.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.328+0000 7f6362ffd700 1 -- 172.21.15.1:0/963260511 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f635c053990 con 0x7f636c06eda0 2024-03-18T19:05:27.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.616+0000 7f63743ed700 1 -- 172.21.15.1:0/963260511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f636c160bf0 con 0x7f636c06eda0 2024-03-18T19:05:27.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.619+0000 7f6362ffd700 1 -- 172.21.15.1:0/963260511 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+0 (secure 0 0 0) 0x7f635c017020 con 0x7f636c06eda0 2024-03-18T19:05:27.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.621+0000 7f63743ed700 1 -- 172.21.15.1:0/963260511 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f635006adb0 msgr2=0x7f635006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:27.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.621+0000 7f63743ed700 1 --2- 172.21.15.1:0/963260511 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f635006adb0 0x7f635006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6368001b70 tx=0x7f6368001b00 comp rx=0 tx=0).stop 2024-03-18T19:05:27.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.621+0000 7f63743ed700 1 -- 172.21.15.1:0/963260511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f636c06eda0 msgr2=0x7f636c1f2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:27.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.621+0000 7f63743ed700 1 --2- 172.21.15.1:0/963260511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f636c06eda0 0x7f636c1f2830 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f635c00c450 tx=0x7f635c00c760 comp rx=0 tx=0).stop 2024-03-18T19:05:27.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.621+0000 7f63743ed700 1 -- 172.21.15.1:0/963260511 shutdown_connections 2024-03-18T19:05:27.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.621+0000 7f63743ed700 1 --2- 172.21.15.1:0/963260511 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f635006adb0 0x7f635006d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:27.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.621+0000 7f63743ed700 1 --2- 172.21.15.1:0/963260511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f636c06ae60 0x7f636c0b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:27.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.621+0000 7f63743ed700 1 --2- 172.21.15.1:0/963260511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f636c06f180 0x7f636c0b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:27.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.621+0000 7f63743ed700 1 --2- 172.21.15.1:0/963260511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f636c06eda0 0x7f636c1f2830 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:27.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.621+0000 7f63743ed700 1 -- 172.21.15.1:0/963260511 >> 172.21.15.1:0/963260511 conn(0x7f636c0cff30 msgr2=0x7f636c0b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:27.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.621+0000 7f63743ed700 1 -- 172.21.15.1:0/963260511 shutdown_connections 2024-03-18T19:05:27.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.621+0000 7f63743ed700 1 -- 172.21.15.1:0/963260511 wait complete. 2024-03-18T19:05:27.618 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-18T19:05:27.629 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:05:27.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.757+0000 7f9cbb5f2700 1 Processor -- start 2024-03-18T19:05:27.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.758+0000 7f9cbb5f2700 1 -- start start 2024-03-18T19:05:27.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.759+0000 7f9cbb5f2700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9cb415b300 0x7f9cb40cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:27.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.759+0000 7f9cbb5f2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cb40cc5e0 0x7f9cb406e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:27.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.759+0000 7f9cbb5f2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9cb406ed50 0x7f9cb415edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:27.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.759+0000 7f9cbb5f2700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb415b6e0 con 0x7f9cb406ed50 2024-03-18T19:05:27.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.759+0000 7f9cbb5f2700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb40cffd0 con 0x7f9cb415b300 2024-03-18T19:05:27.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.759+0000 7f9cbb5f2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb40cce60 con 0x7f9cb40cc5e0 2024-03-18T19:05:27.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.759+0000 7f9cb9b8f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9cb406ed50 0x7f9cb415edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:27.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.759+0000 7f9cb9b8f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9cb406ed50 0x7f9cb415edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33720/0 (socket says 172.21.15.1:33720) 2024-03-18T19:05:27.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.759+0000 7f9cb9b8f700 1 -- 172.21.15.1:0/3968323137 learned_addr learned my addr 172.21.15.1:0/3968323137 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:27.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.759+0000 7f9cb938e700 1 --2- 172.21.15.1:0/3968323137 >> v2:172.21.15.165:3300/0 conn(0x7f9cb415b300 0x7f9cb40cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:27.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.759+0000 7f9cb8b8d700 1 --2- 172.21.15.1:0/3968323137 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cb40cc5e0 0x7f9cb406e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:27.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.759+0000 7f9cb9b8f700 1 -- 172.21.15.1:0/3968323137 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cb40cc5e0 msgr2=0x7f9cb406e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:27.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.759+0000 7f9cb9b8f700 1 --2- 172.21.15.1:0/3968323137 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cb40cc5e0 0x7f9cb406e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:27.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.759+0000 7f9cb9b8f700 1 -- 172.21.15.1:0/3968323137 >> v2:172.21.15.165:3300/0 conn(0x7f9cb415b300 msgr2=0x7f9cb40cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:27.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.759+0000 7f9cb9b8f700 1 --2- 172.21.15.1:0/3968323137 >> v2:172.21.15.165:3300/0 conn(0x7f9cb415b300 0x7f9cb40cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:27.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.759+0000 7f9cb9b8f700 1 -- 172.21.15.1:0/3968323137 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb40cecd0 con 0x7f9cb406ed50 2024-03-18T19:05:27.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.759+0000 7f9cb8b8d700 1 --2- 172.21.15.1:0/3968323137 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cb40cc5e0 0x7f9cb406e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.759+0000 7f9cb9b8f700 1 --2- 172.21.15.1:0/3968323137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cb406ed50 0x7f9cb415edd0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4071a60 tx=0x7f9cac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7030de3f6fdc782e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.759+0000 7f9cabfff700 1 -- 172.21.15.1:0/3968323137 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cac00e070 con 0x7f9cb406ed50 2024-03-18T19:05:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.760+0000 7f9cabfff700 1 -- 172.21.15.1:0/3968323137 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cac00b9c0 con 0x7f9cb406ed50 2024-03-18T19:05:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.760+0000 7f9cabfff700 1 -- 172.21.15.1:0/3968323137 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cac00bd70 con 0x7f9cb406ed50 2024-03-18T19:05:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.760+0000 7f9cbb5f2700 1 -- 172.21.15.1:0/3968323137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cb406ed50 msgr2=0x7f9cb415edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.760+0000 7f9cbb5f2700 1 --2- 172.21.15.1:0/3968323137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cb406ed50 0x7f9cb415edd0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4071a60 tx=0x7f9cac00b330 comp rx=0 tx=0).stop 2024-03-18T19:05:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.760+0000 7f9cbb5f2700 1 -- 172.21.15.1:0/3968323137 shutdown_connections 2024-03-18T19:05:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.760+0000 7f9cbb5f2700 1 --2- 172.21.15.1:0/3968323137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cb406ed50 0x7f9cb415edd0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.760+0000 7f9cbb5f2700 1 --2- 172.21.15.1:0/3968323137 >> v2:172.21.15.165:3300/0 conn(0x7f9cb415b300 0x7f9cb40cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.760+0000 7f9cbb5f2700 1 --2- 172.21.15.1:0/3968323137 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cb40cc5e0 0x7f9cb406e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.760+0000 7f9cbb5f2700 1 -- 172.21.15.1:0/3968323137 >> 172.21.15.1:0/3968323137 conn(0x7f9cb40b2b50 msgr2=0x7f9cb40b2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.760+0000 7f9cbb5f2700 1 -- 172.21.15.1:0/3968323137 shutdown_connections 2024-03-18T19:05:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.760+0000 7f9cbb5f2700 1 -- 172.21.15.1:0/3968323137 wait complete. 2024-03-18T19:05:27.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.761+0000 7f9cbb5f2700 1 Processor -- start 2024-03-18T19:05:27.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.761+0000 7f9cbb5f2700 1 -- start start 2024-03-18T19:05:27.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.762+0000 7f9cbb5f2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cb406ed50 0x7f9cb4155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:27.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.762+0000 7f9cbb5f2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cb40cc5e0 0x7f9cb40b5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:27.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.762+0000 7f9cbb5f2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9cb415b300 0x7f9cb40c8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:27.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.762+0000 7f9cbb5f2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb40c9500 con 0x7f9cb40cc5e0 2024-03-18T19:05:27.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.762+0000 7f9cbb5f2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb40c8670 con 0x7f9cb415b300 2024-03-18T19:05:27.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.762+0000 7f9cbb5f2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb40c8880 con 0x7f9cb406ed50 2024-03-18T19:05:27.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.762+0000 7f9cb8b8d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cb40cc5e0 0x7f9cb40b5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:27.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.762+0000 7f9cb8b8d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cb40cc5e0 0x7f9cb40b5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33732/0 (socket says 172.21.15.1:33732) 2024-03-18T19:05:27.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.762+0000 7f9cb8b8d700 1 -- 172.21.15.1:0/371106982 learned_addr learned my addr 172.21.15.1:0/371106982 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:27.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.762+0000 7f9cb938e700 1 --2- 172.21.15.1:0/371106982 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cb406ed50 0x7f9cb4155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:27.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.762+0000 7f9cb9b8f700 1 --2- 172.21.15.1:0/371106982 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9cb415b300 0x7f9cb40c8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.763+0000 7f9cb8b8d700 1 -- 172.21.15.1:0/371106982 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cb406ed50 msgr2=0x7f9cb4155760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.763+0000 7f9cb8b8d700 1 --2- 172.21.15.1:0/371106982 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cb406ed50 0x7f9cb4155760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.763+0000 7f9cb8b8d700 1 -- 172.21.15.1:0/371106982 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9cb415b300 msgr2=0x7f9cb40c8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.763+0000 7f9cb8b8d700 1 --2- 172.21.15.1:0/371106982 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9cb415b300 0x7f9cb40c8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.763+0000 7f9cb8b8d700 1 -- 172.21.15.1:0/371106982 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cac009b30 con 0x7f9cb40cc5e0 2024-03-18T19:05:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.763+0000 7f9cb9b8f700 1 --2- 172.21.15.1:0/371106982 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9cb415b300 0x7f9cb40c8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.763+0000 7f9cb8b8d700 1 --2- 172.21.15.1:0/371106982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cb40cc5e0 0x7f9cb40b5c10 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f9cb000c740 tx=0x7f9cb000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.763+0000 7f9ca9ffb700 1 -- 172.21.15.1:0/371106982 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0011800 con 0x7f9cb40cc5e0 2024-03-18T19:05:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.763+0000 7f9ca9ffb700 1 -- 172.21.15.1:0/371106982 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb0011960 con 0x7f9cb40cc5e0 2024-03-18T19:05:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.763+0000 7f9cbb5f2700 1 -- 172.21.15.1:0/371106982 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb40c8b60 con 0x7f9cb40cc5e0 2024-03-18T19:05:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.763+0000 7f9ca9ffb700 1 -- 172.21.15.1:0/371106982 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0011cd0 con 0x7f9cb40cc5e0 2024-03-18T19:05:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.763+0000 7f9cbb5f2700 1 -- 172.21.15.1:0/371106982 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb40c9120 con 0x7f9cb40cc5e0 2024-03-18T19:05:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.764+0000 7f9cbb5f2700 1 -- 172.21.15.1:0/371106982 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb40c9640 con 0x7f9cb40cc5e0 2024-03-18T19:05:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.765+0000 7f9ca9ffb700 1 -- 172.21.15.1:0/371106982 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9cb0021350 con 0x7f9cb40cc5e0 2024-03-18T19:05:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.765+0000 7f9ca9ffb700 1 --2- 172.21.15.1:0/371106982 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9c9806adb0 0x7f9c9806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:27.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.765+0000 7f9ca9ffb700 1 -- 172.21.15.1:0/371106982 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9cb0089660 con 0x7f9cb40cc5e0 2024-03-18T19:05:27.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.765+0000 7f9cb938e700 1 --2- 172.21.15.1:0/371106982 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9c9806adb0 0x7f9c9806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:27.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.765+0000 7f9cb938e700 1 --2- 172.21.15.1:0/371106982 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9c9806adb0 0x7f9c9806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4155ca0 tx=0x7f9ca4008040 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:27.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:27.770+0000 7f9ca9ffb700 1 -- 172.21.15.1:0/371106982 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9cb40c9640 con 0x7f9cb40cc5e0 2024-03-18T19:05:28.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.014+0000 7f9cbb5f2700 1 -- 172.21.15.1:0/371106982 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cb40c9640 con 0x7f9cb40cc5e0 2024-03-18T19:05:28.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.015+0000 7f9ca9ffb700 1 -- 172.21.15.1:0/371106982 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4732 (secure 0 0 0) 0x7f9cb40c9640 con 0x7f9cb40cc5e0 2024-03-18T19:05:28.011 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:28.011 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5744,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2437290126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2437290126},{"type":"v1","addr":"172.21.15.1:6836","nonce":2437290126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5747,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2832642785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2832642785},{"type":"v1","addr":"172.21.15.1:6837","nonce":2832642785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5756,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3765527492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3765527492},{"type":"v1","addr":"172.21.15.165:6834","nonce":3765527492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:05:22.577555+0000","modified":"2024-03-18T19:05:25.594597+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5759},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5759":{"gid":5759,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/119895951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":119895951},{"type":"v1","addr":"172.21.15.165:6835","nonce":119895951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-18T19:05:28.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.017+0000 7f9cbb5f2700 1 -- 172.21.15.1:0/371106982 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9c9806adb0 msgr2=0x7f9c9806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:28.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.017+0000 7f9cbb5f2700 1 --2- 172.21.15.1:0/371106982 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9c9806adb0 0x7f9c9806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4155ca0 tx=0x7f9ca4008040 comp rx=0 tx=0).stop 2024-03-18T19:05:28.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.017+0000 7f9cbb5f2700 1 -- 172.21.15.1:0/371106982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cb40cc5e0 msgr2=0x7f9cb40b5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:28.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.017+0000 7f9cbb5f2700 1 --2- 172.21.15.1:0/371106982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cb40cc5e0 0x7f9cb40b5c10 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f9cb000c740 tx=0x7f9cb000cb00 comp rx=0 tx=0).stop 2024-03-18T19:05:28.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.017+0000 7f9cbb5f2700 1 -- 172.21.15.1:0/371106982 shutdown_connections 2024-03-18T19:05:28.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.017+0000 7f9cbb5f2700 1 --2- 172.21.15.1:0/371106982 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9c9806adb0 0x7f9c9806d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.017+0000 7f9cbb5f2700 1 --2- 172.21.15.1:0/371106982 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cb406ed50 0x7f9cb4155760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.017+0000 7f9cbb5f2700 1 --2- 172.21.15.1:0/371106982 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9cb415b300 0x7f9cb40c8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.017+0000 7f9cbb5f2700 1 --2- 172.21.15.1:0/371106982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cb40cc5e0 0x7f9cb40b5c10 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.017+0000 7f9cbb5f2700 1 -- 172.21.15.1:0/371106982 >> 172.21.15.1:0/371106982 conn(0x7f9cb40b2b50 msgr2=0x7f9cb40cdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:28.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.017+0000 7f9cbb5f2700 1 -- 172.21.15.1:0/371106982 shutdown_connections 2024-03-18T19:05:28.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.017+0000 7f9cbb5f2700 1 -- 172.21.15.1:0/371106982 wait complete. 2024-03-18T19:05:28.014 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 33 2024-03-18T19:05:28.144 INFO:tasks.ceph.mds.b.smithi165.stderr:2024-03-18T19:05:28.149+0000 7f8e59d4d700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T19:05:28.429 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 33, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T19:05:22.577555+0000', 'modified': '2024-03-18T19:05:25.594597+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5759}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5759': {'gid': 5759, 'name': 'b', 'rank': 0, 'incarnation': 29, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.165:6835/119895951', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6833', 'nonce': 119895951}, {'type': 'v1', 'addr': '172.21.15.165:6835', 'nonce': 119895951}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T19:05:28.429 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T19:05:28.429 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T19:05:28.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.564+0000 7fb478e93700 1 Processor -- start 2024-03-18T19:05:28.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.564+0000 7fb478e93700 1 -- start start 2024-03-18T19:05:28.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.564+0000 7fb478e93700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb4740ace80 0x7fb4740ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.565+0000 7fb478e93700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47415c340 con 0x7fb47406a750 2024-03-18T19:05:28.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.565+0000 7fb478e93700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47415b7a0 con 0x7fb4740ad850 2024-03-18T19:05:28.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.565+0000 7fb478e93700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb474070090 con 0x7fb4740ace80 2024-03-18T19:05:28.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.565+0000 7fb471d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb4740ace80 0x7fb4740ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.565+0000 7fb471d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb4740ace80 0x7fb4740ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33734/0 (socket says 172.21.15.1:33734) 2024-03-18T19:05:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.565+0000 7fb47259c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb47406a750 0x7fb47406ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57680/0 (socket says 172.21.15.1:57680) 2024-03-18T19:05:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.565+0000 7fb471d9b700 1 -- 172.21.15.1:0/2814448857 learned_addr learned my addr 172.21.15.1:0/2814448857 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.565+0000 7fb471d9b700 1 -- 172.21.15.1:0/2814448857 >> v1:172.21.15.165:6789/0 conn(0x7fb4740ad850 legacy=0x7fb47415ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.565+0000 7fb471d9b700 1 -- 172.21.15.1:0/2814448857 >> v1:172.21.15.1:6789/0 conn(0x7fb47406a750 legacy=0x7fb47406ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.565+0000 7fb471d9b700 1 -- 172.21.15.1:0/2814448857 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47415ba10 con 0x7fb4740ace80 2024-03-18T19:05:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.565+0000 7fb47159a700 1 -- 172.21.15.1:0/2814448857 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733613537 0 0) 0x7fb47415c340 con 0x7fb47406a750 2024-03-18T19:05:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.565+0000 7fb471d9b700 1 --2- 172.21.15.1:0/2814448857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4740ace80 0x7fb4740ad310 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fb45c0098f0 tx=0x7fb45c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c78d4e77fe0373de server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.565+0000 7fb47159a700 1 -- 172.21.15.1:0/2814448857 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c004230 con 0x7fb4740ace80 2024-03-18T19:05:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.565+0000 7fb47159a700 1 -- 172.21.15.1:0/2814448857 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45c004390 con 0x7fb4740ace80 2024-03-18T19:05:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.565+0000 7fb47159a700 1 -- 172.21.15.1:0/2814448857 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c0036a0 con 0x7fb4740ace80 2024-03-18T19:05:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.566+0000 7fb478e93700 1 -- 172.21.15.1:0/2814448857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4740ace80 msgr2=0x7fb4740ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.566+0000 7fb478e93700 1 --2- 172.21.15.1:0/2814448857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4740ace80 0x7fb4740ad310 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fb45c0098f0 tx=0x7fb45c009c00 comp rx=0 tx=0).stop 2024-03-18T19:05:28.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.566+0000 7fb478e93700 1 -- 172.21.15.1:0/2814448857 shutdown_connections 2024-03-18T19:05:28.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.566+0000 7fb478e93700 1 --2- 172.21.15.1:0/2814448857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4740ace80 0x7fb4740ad310 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.566+0000 7fb478e93700 1 -- 172.21.15.1:0/2814448857 >> 172.21.15.1:0/2814448857 conn(0x7fb4740cc9b0 msgr2=0x7fb4740ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:28.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.566+0000 7fb478e93700 1 -- 172.21.15.1:0/2814448857 shutdown_connections 2024-03-18T19:05:28.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.566+0000 7fb478e93700 1 -- 172.21.15.1:0/2814448857 wait complete. 2024-03-18T19:05:28.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.568+0000 7fb478e93700 1 Processor -- start 2024-03-18T19:05:28.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.568+0000 7fb478e93700 1 -- start start 2024-03-18T19:05:28.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb478e93700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb47406a750 0x7fb47414fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb478e93700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4740ace80 0x7fb4740c98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb478e93700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4740ad850 0x7fb474157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb478e93700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb474154190 con 0x7fb4740ace80 2024-03-18T19:05:28.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb478e93700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4740b3c60 con 0x7fb4740ad850 2024-03-18T19:05:28.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb478e93700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4740c7d40 con 0x7fb47406a750 2024-03-18T19:05:28.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb471d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4740ace80 0x7fb4740c98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb471d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4740ace80 0x7fb4740c98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33746/0 (socket says 172.21.15.1:33746) 2024-03-18T19:05:28.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb471d9b700 1 -- 172.21.15.1:0/2559221331 learned_addr learned my addr 172.21.15.1:0/2559221331 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:28.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb47259c700 1 --2- 172.21.15.1:0/2559221331 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb47406a750 0x7fb47414fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb472d9d700 1 --2- 172.21.15.1:0/2559221331 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4740ad850 0x7fb474157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb471d9b700 1 -- 172.21.15.1:0/2559221331 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb47406a750 msgr2=0x7fb47414fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:28.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb471d9b700 1 --2- 172.21.15.1:0/2559221331 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb47406a750 0x7fb47414fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb471d9b700 1 -- 172.21.15.1:0/2559221331 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4740ad850 msgr2=0x7fb474157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:28.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb471d9b700 1 --2- 172.21.15.1:0/2559221331 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4740ad850 0x7fb474157ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb471d9b700 1 -- 172.21.15.1:0/2559221331 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c009550 con 0x7fb4740ace80 2024-03-18T19:05:28.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb472d9d700 1 --2- 172.21.15.1:0/2559221331 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4740ad850 0x7fb474157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:28.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb47259c700 1 --2- 172.21.15.1:0/2559221331 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb47406a750 0x7fb47414fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:28.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb471d9b700 1 --2- 172.21.15.1:0/2559221331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4740ace80 0x7fb4740c98f0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fb474072d50 tx=0x7fb45c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:28.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb46affd700 1 -- 172.21.15.1:0/2559221331 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c003dc0 con 0x7fb4740ace80 2024-03-18T19:05:28.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb46affd700 1 -- 172.21.15.1:0/2559221331 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45c01f070 con 0x7fb4740ace80 2024-03-18T19:05:28.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb478e93700 1 -- 172.21.15.1:0/2559221331 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4740c8020 con 0x7fb4740ace80 2024-03-18T19:05:28.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb46affd700 1 -- 172.21.15.1:0/2559221331 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c00b8a0 con 0x7fb4740ace80 2024-03-18T19:05:28.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.569+0000 7fb478e93700 1 -- 172.21.15.1:0/2559221331 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4740c7a10 con 0x7fb4740ace80 2024-03-18T19:05:28.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.571+0000 7fb46affd700 1 -- 172.21.15.1:0/2559221331 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb45c021030 con 0x7fb4740ace80 2024-03-18T19:05:28.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.571+0000 7fb46affd700 1 --2- 172.21.15.1:0/2559221331 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb45406adb0 0x7fb45406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.571+0000 7fb46affd700 1 -- 172.21.15.1:0/2559221331 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb45c08ac80 con 0x7fb4740ace80 2024-03-18T19:05:28.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.571+0000 7fb47259c700 1 --2- 172.21.15.1:0/2559221331 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb45406adb0 0x7fb45406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.572+0000 7fb47259c700 1 --2- 172.21.15.1:0/2559221331 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb45406adb0 0x7fb45406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb464001020 tx=0x7fb464009040 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:28.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.573+0000 7fb48503cb80 1 Processor -- start 2024-03-18T19:05:28.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.574+0000 7fb48503cb80 1 -- start start 2024-03-18T19:05:28.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.574+0000 7fb48503cb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebc15c10 0x5624ebc15ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.574+0000 7fb48503cb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebbe70a0 0x5624ebc32580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.574+0000 7fb48503cb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebc27630 0x5624ebbea160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.574+0000 7fb48503cb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ebb60f20 con 0x5624ebbe70a0 2024-03-18T19:05:28.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.574+0000 7fb48503cb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ebb42c30 con 0x5624ebc27630 2024-03-18T19:05:28.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.574+0000 7fb48503cb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624ebb40430 con 0x5624ebc15c10 2024-03-18T19:05:28.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.574+0000 7fb471d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebbe70a0 0x5624ebc32580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.574+0000 7fb471d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebbe70a0 0x5624ebc32580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33758/0 (socket says 172.21.15.1:33758) 2024-03-18T19:05:28.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.574+0000 7fb471d9b700 1 -- 172.21.15.1:0/4180650909 learned_addr learned my addr 172.21.15.1:0/4180650909 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:28.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.575+0000 7fb471d9b700 1 -- 172.21.15.1:0/4180650909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebc15c10 msgr2=0x5624ebc15ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:28.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.575+0000 7fb47259c700 1 --2- 172.21.15.1:0/4180650909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebc15c10 0x5624ebc15ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.575+0000 7fb471d9b700 1 --2- 172.21.15.1:0/4180650909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebc15c10 0x5624ebc15ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.575+0000 7fb471d9b700 1 -- 172.21.15.1:0/4180650909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebc27630 msgr2=0x5624ebbea160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:28.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.575+0000 7fb472d9d700 1 --2- 172.21.15.1:0/4180650909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebc27630 0x5624ebbea160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.575+0000 7fb471d9b700 1 --2- 172.21.15.1:0/4180650909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebc27630 0x5624ebbea160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.575+0000 7fb471d9b700 1 -- 172.21.15.1:0/4180650909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624eb92f790 con 0x5624ebbe70a0 2024-03-18T19:05:28.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.575+0000 7fb471d9b700 1 --2- 172.21.15.1:0/4180650909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebbe70a0 0x5624ebc32580 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fb45c055e60 tx=0x7fb45c0593b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac6246fa1ed3c16 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:28.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.575+0000 7fb453fff700 1 -- 172.21.15.1:0/4180650909 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c059960 con 0x5624ebbe70a0 2024-03-18T19:05:28.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.575+0000 7fb453fff700 1 -- 172.21.15.1:0/4180650909 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45c05d790 con 0x5624ebbe70a0 2024-03-18T19:05:28.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.575+0000 7fb453fff700 1 -- 172.21.15.1:0/4180650909 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c061d50 con 0x5624ebbe70a0 2024-03-18T19:05:28.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.576+0000 7fb48503cb80 1 -- 172.21.15.1:0/4180650909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebbe70a0 msgr2=0x5624ebc32580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:28.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.576+0000 7fb48503cb80 1 --2- 172.21.15.1:0/4180650909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebbe70a0 0x5624ebc32580 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fb45c055e60 tx=0x7fb45c0593b0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.576+0000 7fb48503cb80 1 -- 172.21.15.1:0/4180650909 shutdown_connections 2024-03-18T19:05:28.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.576+0000 7fb48503cb80 1 --2- 172.21.15.1:0/4180650909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebc15c10 0x5624ebc15ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.576+0000 7fb48503cb80 1 --2- 172.21.15.1:0/4180650909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebc27630 0x5624ebbea160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.576+0000 7fb48503cb80 1 --2- 172.21.15.1:0/4180650909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebbe70a0 0x5624ebc32580 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.576+0000 7fb48503cb80 1 -- 172.21.15.1:0/4180650909 >> 172.21.15.1:0/4180650909 conn(0x5624ebb49cf0 msgr2=0x5624ebbe9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:28.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.576+0000 7fb48503cb80 1 -- 172.21.15.1:0/4180650909 shutdown_connections 2024-03-18T19:05:28.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.576+0000 7fb48503cb80 1 -- 172.21.15.1:0/4180650909 wait complete. 2024-03-18T19:05:28.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.576+0000 7fb48503cb80 1 -- start start 2024-03-18T19:05:28.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.576+0000 7fb48503cb80 1 Processor -- start 2024-03-18T19:05:28.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.577+0000 7fb48503cb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebbe70a0 0x5624ebc122f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.577+0000 7fb48503cb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebc15c10 0x5624ebc12830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.577+0000 7fb48503cb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebc27630 0x5624ebc0ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.577+0000 7fb48503cb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ebb3bd20 con 0x5624ebc15c10 2024-03-18T19:05:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.577+0000 7fb48503cb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ebad7770 con 0x5624ebc27630 2024-03-18T19:05:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.577+0000 7fb48503cb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624ebafa6f0 con 0x5624ebbe70a0 2024-03-18T19:05:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.577+0000 7fb48503cb80 10 client.0 _ll_register_callbacks cb 0x5624eb9435a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T19:05:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.577+0000 7fb471d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebc15c10 0x5624ebc12830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb471d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebc15c10 0x5624ebc12830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33762/0 (socket says 172.21.15.1:33762) 2024-03-18T19:05:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb471d9b700 1 -- 172.21.15.1:0/3797277220 learned_addr learned my addr 172.21.15.1:0/3797277220 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb47259c700 1 --2- 172.21.15.1:0/3797277220 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebbe70a0 0x5624ebc122f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb472d9d700 1 --2- 172.21.15.1:0/3797277220 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebc27630 0x5624ebc0ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb471d9b700 1 -- 172.21.15.1:0/3797277220 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebbe70a0 msgr2=0x5624ebc122f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb471d9b700 1 --2- 172.21.15.1:0/3797277220 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebbe70a0 0x5624ebc122f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb471d9b700 1 -- 172.21.15.1:0/3797277220 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebc27630 msgr2=0x5624ebc0ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb471d9b700 1 --2- 172.21.15.1:0/3797277220 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebc27630 0x5624ebc0ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb471d9b700 1 -- 172.21.15.1:0/3797277220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4740c7a10 con 0x5624ebc15c10 2024-03-18T19:05:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb472d9d700 1 --2- 172.21.15.1:0/3797277220 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebc27630 0x5624ebc0ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb471d9b700 1 --2- 172.21.15.1:0/3797277220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebc15c10 0x5624ebc12830 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fb45c003fd0 tx=0x7fb45c006c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab79b926daf1375b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb4517fa700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T19:05:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb4517fa700 1 -- 172.21.15.1:0/3797277220 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c075070 con 0x5624ebc15c10 2024-03-18T19:05:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb4517fa700 1 -- 172.21.15.1:0/3797277220 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb45c061d50 con 0x5624ebc15c10 2024-03-18T19:05:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb48503cb80 1 -- 172.21.15.1:0/3797277220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5624ebbdde90 con 0x5624ebc15c10 2024-03-18T19:05:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb4517fa700 1 -- 172.21.15.1:0/3797277220 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb45c0711c0 con 0x5624ebc15c10 2024-03-18T19:05:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb48503cb80 10 client.6936 fetch_fsmap learned FSMap version 33 2024-03-18T19:05:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb48503cb80 1 -- 172.21.15.1:0/3797277220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x5624eba1c0a0 con 0x5624ebc15c10 2024-03-18T19:05:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb4517fa700 1 -- 172.21.15.1:0/3797277220 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5624ebbdde90 con 0x5624ebc15c10 2024-03-18T19:05:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb4517fa700 1 -- 172.21.15.1:0/3797277220 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb45c089030 con 0x5624ebc15c10 2024-03-18T19:05:28.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb48503cb80 10 client.6936 fetch_fsmap finished waiting for FSMap version 33 2024-03-18T19:05:28.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb48503cb80 10 client.6936 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T19:05:28.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb48503cb80 20 client.6936 populate_metadata read hostname 'smithi001' 2024-03-18T19:05:28.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.578+0000 7fb48503cb80 1 --2- 172.21.15.1:0/3797277220 >> [v2:172.21.15.165:6833/119895951,v1:172.21.15.165:6835/119895951] conn(0x5624ebc29750 0x5624ebc29b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.579+0000 7fb48503cb80 4 client.6936 mds_command: new command op to 5759 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T19:05:28.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.579+0000 7fb48503cb80 1 -- 172.21.15.1:0/3797277220 --> [v2:172.21.15.165:6833/119895951,v1:172.21.15.165:6835/119895951] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5624ebbbb010 con 0x5624ebc29750 2024-03-18T19:05:28.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.579+0000 7fb47259c700 1 --2- 172.21.15.1:0/3797277220 >> [v2:172.21.15.165:6833/119895951,v1:172.21.15.165:6835/119895951] conn(0x5624ebc29750 0x5624ebc29b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.579+0000 7fb4517fa700 10 client.6936 ms_handle_connect on v2:172.21.15.165:6833/119895951 2024-03-18T19:05:28.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.579+0000 7fb47259c700 1 --2- 172.21.15.1:0/3797277220 >> [v2:172.21.15.165:6833/119895951,v1:172.21.15.165:6835/119895951] conn(0x5624ebc29750 0x5624ebc29b30 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c4e8047da564b80 server_cookie=fd943b21a71f5413 in_seq=0 out_seq=0 2024-03-18T19:05:28.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.580+0000 7fb4517fa700 1 -- 172.21.15.1:0/3797277220 <== mds.0 v2:172.21.15.165:6833/119895951 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5624ebbbb010 con 0x5624ebc29750 2024-03-18T19:05:28.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.580+0000 7fb4517fa700 10 client.6936 handle_command_reply: tid=0 2024-03-18T19:05:28.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.580+0000 7fb48503cb80 1 client.6936 shutdown 2024-03-18T19:05:28.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.580+0000 7fb48503cb80 1 -- 172.21.15.1:0/3797277220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebc15c10 msgr2=0x5624ebc12830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:28.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.580+0000 7fb48503cb80 1 --2- 172.21.15.1:0/3797277220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebc15c10 0x5624ebc12830 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fb45c003fd0 tx=0x7fb45c006c60 comp rx=0 tx=0).stop 2024-03-18T19:05:28.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.580+0000 7fb48503cb80 1 -- 172.21.15.1:0/3797277220 shutdown_connections 2024-03-18T19:05:28.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.580+0000 7fb48503cb80 1 --2- 172.21.15.1:0/3797277220 >> [v2:172.21.15.165:6833/119895951,v1:172.21.15.165:6835/119895951] conn(0x5624ebc29750 0x5624ebc29b30 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.580+0000 7fb48503cb80 1 --2- 172.21.15.1:0/3797277220 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebbe70a0 0x5624ebc122f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.580+0000 7fb48503cb80 1 --2- 172.21.15.1:0/3797277220 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebc27630 0x5624ebc0ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.580+0000 7fb48503cb80 1 --2- 172.21.15.1:0/3797277220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebc15c10 0x5624ebc12830 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.580+0000 7fb4517fa700 0 client.6936 ms_handle_reset on v2:172.21.15.165:6833/119895951 2024-03-18T19:05:28.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.580+0000 7fb48503cb80 1 -- 172.21.15.1:0/3797277220 >> 172.21.15.1:0/3797277220 conn(0x5624ebb49cf0 msgr2=0x5624ebc13470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:28.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.580+0000 7fb48503cb80 1 -- 172.21.15.1:0/3797277220 shutdown_connections 2024-03-18T19:05:28.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.580+0000 7fb48503cb80 1 -- 172.21.15.1:0/3797277220 wait complete. 2024-03-18T19:05:28.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.580+0000 7fb48503cb80 20 client.6936 trim_cache size 0 max 16384 2024-03-18T19:05:28.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.599+0000 7fb48503cb80 1 Processor -- start 2024-03-18T19:05:28.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.599+0000 7fb48503cb80 1 -- start start 2024-03-18T19:05:28.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.599+0000 7fb48503cb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebbe70a0 0x5624ebc30ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.599+0000 7fb48503cb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebc15c10 0x5624ebc12280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.599+0000 7fb48503cb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebc27630 0x5624ebc127c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.599+0000 7fb48503cb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624eb930390 con 0x5624ebbe70a0 2024-03-18T19:05:28.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.600+0000 7fb48503cb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ebbec920 con 0x5624ebc27630 2024-03-18T19:05:28.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.600+0000 7fb48503cb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624ebbe6350 con 0x5624ebc15c10 2024-03-18T19:05:28.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.600+0000 7fb47259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebbe70a0 0x5624ebc30ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.600+0000 7fb47259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebbe70a0 0x5624ebc30ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33774/0 (socket says 172.21.15.1:33774) 2024-03-18T19:05:28.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.600+0000 7fb47259c700 1 -- 172.21.15.1:0/2035557320 learned_addr learned my addr 172.21.15.1:0/2035557320 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:28.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.600+0000 7fb47259c700 1 -- 172.21.15.1:0/2035557320 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebc15c10 msgr2=0x5624ebc12280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:28.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.600+0000 7fb47259c700 1 --2- 172.21.15.1:0/2035557320 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebc15c10 0x5624ebc12280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.600+0000 7fb47259c700 1 -- 172.21.15.1:0/2035557320 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebc27630 msgr2=0x5624ebc127c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:28.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.600+0000 7fb47259c700 1 --2- 172.21.15.1:0/2035557320 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebc27630 0x5624ebc127c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.600+0000 7fb47259c700 1 -- 172.21.15.1:0/2035557320 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624ebb02010 con 0x5624ebbe70a0 2024-03-18T19:05:28.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.600+0000 7fb47259c700 1 --2- 172.21.15.1:0/2035557320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebbe70a0 0x5624ebc30ca0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fb464007a50 tx=0x7fb464012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8777b9426c6aeb39 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:28.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.600+0000 7fb468ff9700 1 -- 172.21.15.1:0/2035557320 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb464007ea0 con 0x5624ebbe70a0 2024-03-18T19:05:28.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.600+0000 7fb468ff9700 1 -- 172.21.15.1:0/2035557320 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46401b040 con 0x5624ebbe70a0 2024-03-18T19:05:28.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.600+0000 7fb468ff9700 1 -- 172.21.15.1:0/2035557320 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb464025a80 con 0x5624ebbe70a0 2024-03-18T19:05:28.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.601+0000 7fb48503cb80 1 -- 172.21.15.1:0/2035557320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebbe70a0 msgr2=0x5624ebc30ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:28.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.601+0000 7fb48503cb80 1 --2- 172.21.15.1:0/2035557320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebbe70a0 0x5624ebc30ca0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fb464007a50 tx=0x7fb464012000 comp rx=0 tx=0).stop 2024-03-18T19:05:28.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.601+0000 7fb48503cb80 1 -- 172.21.15.1:0/2035557320 shutdown_connections 2024-03-18T19:05:28.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.601+0000 7fb48503cb80 1 --2- 172.21.15.1:0/2035557320 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebc27630 0x5624ebc127c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.601+0000 7fb48503cb80 1 --2- 172.21.15.1:0/2035557320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebbe70a0 0x5624ebc30ca0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.601+0000 7fb48503cb80 1 --2- 172.21.15.1:0/2035557320 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebc15c10 0x5624ebc12280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.601+0000 7fb48503cb80 1 -- 172.21.15.1:0/2035557320 >> 172.21.15.1:0/2035557320 conn(0x5624ebb49cf0 msgr2=0x5624ebc0eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:28.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.601+0000 7fb48503cb80 1 -- 172.21.15.1:0/2035557320 shutdown_connections 2024-03-18T19:05:28.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.601+0000 7fb48503cb80 1 -- 172.21.15.1:0/2035557320 wait complete. 2024-03-18T19:05:28.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.601+0000 7fb48503cb80 1 -- start start 2024-03-18T19:05:28.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.601+0000 7fb48503cb80 1 Processor -- start 2024-03-18T19:05:28.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb48503cb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebbe70a0 0x5624ebc263b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb48503cb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebc15c10 0x5624ebc96d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb48503cb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebc27630 0x5624ebc14260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb48503cb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ebc268f0 con 0x5624ebc27630 2024-03-18T19:05:28.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb48503cb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ebc972b0 con 0x5624ebbe70a0 2024-03-18T19:05:28.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb48503cb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624ebc26a90 con 0x5624ebc15c10 2024-03-18T19:05:28.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb48503cb80 10 client.0 _ll_register_callbacks cb 0x7fb45c059590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T19:05:28.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb472d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebc27630 0x5624ebc14260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb472d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebc27630 0x5624ebc14260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33780/0 (socket says 172.21.15.1:33780) 2024-03-18T19:05:28.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb472d9d700 1 -- 172.21.15.1:0/148187404 learned_addr learned my addr 172.21.15.1:0/148187404 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:28.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb471d9b700 1 --2- 172.21.15.1:0/148187404 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebc15c10 0x5624ebc96d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb47259c700 1 --2- 172.21.15.1:0/148187404 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebbe70a0 0x5624ebc263b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb472d9d700 1 -- 172.21.15.1:0/148187404 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebc15c10 msgr2=0x5624ebc96d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:28.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb472d9d700 1 --2- 172.21.15.1:0/148187404 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebc15c10 0x5624ebc96d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb472d9d700 1 -- 172.21.15.1:0/148187404 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebbe70a0 msgr2=0x5624ebc263b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:28.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb472d9d700 1 --2- 172.21.15.1:0/148187404 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebbe70a0 0x5624ebc263b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb472d9d700 1 -- 172.21.15.1:0/148187404 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46400f040 con 0x5624ebc27630 2024-03-18T19:05:28.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb471d9b700 1 --2- 172.21.15.1:0/148187404 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebc15c10 0x5624ebc96d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:28.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb472d9d700 1 --2- 172.21.15.1:0/148187404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebc27630 0x5624ebc14260 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fb46c0118c0 tx=0x7fb46c011c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c0546fe4cdd2ca6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:28.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.603+0000 7fb4517fa700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T19:05:28.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.604+0000 7fb4517fa700 1 -- 172.21.15.1:0/148187404 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c00fa10 con 0x5624ebc27630 2024-03-18T19:05:28.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.604+0000 7fb4517fa700 1 -- 172.21.15.1:0/148187404 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb46c011e60 con 0x5624ebc27630 2024-03-18T19:05:28.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.604+0000 7fb48503cb80 1 -- 172.21.15.1:0/148187404 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb45c05d790 con 0x5624ebc27630 2024-03-18T19:05:28.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.604+0000 7fb4517fa700 1 -- 172.21.15.1:0/148187404 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb46c00fb70 con 0x5624ebc27630 2024-03-18T19:05:28.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.604+0000 7fb48503cb80 10 client.6942 fetch_fsmap learned FSMap version 33 2024-03-18T19:05:28.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.604+0000 7fb48503cb80 1 -- 172.21.15.1:0/148187404 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x5624eb9901d0 con 0x5624ebc27630 2024-03-18T19:05:28.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.604+0000 7fb4517fa700 1 -- 172.21.15.1:0/148187404 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45c05d790 con 0x5624ebc27630 2024-03-18T19:05:28.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.604+0000 7fb4517fa700 1 -- 172.21.15.1:0/148187404 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb46c00fcd0 con 0x5624ebc27630 2024-03-18T19:05:28.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.604+0000 7fb48503cb80 10 client.6942 fetch_fsmap finished waiting for FSMap version 33 2024-03-18T19:05:28.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.604+0000 7fb48503cb80 10 client.6942 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T19:05:28.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.604+0000 7fb48503cb80 20 client.6942 populate_metadata read hostname 'smithi001' 2024-03-18T19:05:28.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.604+0000 7fb48503cb80 1 --2- 172.21.15.1:0/148187404 >> [v2:172.21.15.165:6833/119895951,v1:172.21.15.165:6835/119895951] conn(0x5624ebc34290 0x5624ebc34670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.604+0000 7fb48503cb80 4 client.6942 mds_command: new command op to 5759 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T19:05:28.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.604+0000 7fb48503cb80 1 -- 172.21.15.1:0/148187404 --> [v2:172.21.15.165:6833/119895951,v1:172.21.15.165:6835/119895951] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5624eb975040 con 0x5624ebc34290 2024-03-18T19:05:28.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.604+0000 7fb47259c700 1 --2- 172.21.15.1:0/148187404 >> [v2:172.21.15.165:6833/119895951,v1:172.21.15.165:6835/119895951] conn(0x5624ebc34290 0x5624ebc34670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.605+0000 7fb4517fa700 10 client.6942 ms_handle_connect on v2:172.21.15.165:6833/119895951 2024-03-18T19:05:28.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.605+0000 7fb47259c700 1 --2- 172.21.15.1:0/148187404 >> [v2:172.21.15.165:6833/119895951,v1:172.21.15.165:6835/119895951] conn(0x5624ebc34290 0x5624ebc34670 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6cf57fdf1fa38b9 server_cookie=a33c3b2f01b4dd37 in_seq=0 out_seq=0 2024-03-18T19:05:28.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.605+0000 7fb4517fa700 1 -- 172.21.15.1:0/148187404 <== mds.0 v2:172.21.15.165:6833/119895951 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5624eb975040 con 0x5624ebc34290 2024-03-18T19:05:28.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.605+0000 7fb4517fa700 10 client.6942 handle_command_reply: tid=0 2024-03-18T19:05:28.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.605+0000 7fb48503cb80 1 client.6942 shutdown 2024-03-18T19:05:28.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.605+0000 7fb48503cb80 1 -- 172.21.15.1:0/148187404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebc27630 msgr2=0x5624ebc14260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:28.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.605+0000 7fb48503cb80 1 --2- 172.21.15.1:0/148187404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebc27630 0x5624ebc14260 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fb46c0118c0 tx=0x7fb46c011c80 comp rx=0 tx=0).stop 2024-03-18T19:05:28.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.605+0000 7fb48503cb80 1 -- 172.21.15.1:0/148187404 shutdown_connections 2024-03-18T19:05:28.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.605+0000 7fb48503cb80 1 --2- 172.21.15.1:0/148187404 >> [v2:172.21.15.165:6833/119895951,v1:172.21.15.165:6835/119895951] conn(0x5624ebc34290 0x5624ebc34670 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.605+0000 7fb48503cb80 1 --2- 172.21.15.1:0/148187404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5624ebc27630 0x5624ebc14260 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.605+0000 7fb48503cb80 1 --2- 172.21.15.1:0/148187404 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5624ebbe70a0 0x5624ebc263b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.605+0000 7fb4517fa700 0 client.6942 ms_handle_reset on v2:172.21.15.165:6833/119895951 2024-03-18T19:05:28.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.605+0000 7fb48503cb80 1 --2- 172.21.15.1:0/148187404 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5624ebc15c10 0x5624ebc96d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.605+0000 7fb48503cb80 1 -- 172.21.15.1:0/148187404 >> 172.21.15.1:0/148187404 conn(0x5624ebb49cf0 msgr2=0x5624ebc13470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:28.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.605+0000 7fb48503cb80 1 -- 172.21.15.1:0/148187404 shutdown_connections 2024-03-18T19:05:28.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.605+0000 7fb48503cb80 1 -- 172.21.15.1:0/148187404 wait complete. 2024-03-18T19:05:28.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.605+0000 7fb48503cb80 20 client.6942 trim_cache size 0 max 16384 2024-03-18T19:05:28.608 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:05:28.608 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "9c1c697f-8655-4ef3-8a13-da568d2cbe86", 2024-03-18T19:05:28.608 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-18T19:05:28.608 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 5759, 2024-03-18T19:05:28.608 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-18T19:05:28.608 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-18T19:05:28.608 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-18T19:05:28.608 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 6.0257819479999997, 2024-03-18T19:05:28.608 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 33, 2024-03-18T19:05:28.608 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 89, 2024-03-18T19:05:28.608 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T19:05:28.608 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 11.330117035000001 2024-03-18T19:05:28.609 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:05:28.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.606+0000 7fb478e93700 1 -- 172.21.15.1:0/2559221331 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb45406adb0 msgr2=0x7fb45406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:28.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.606+0000 7fb478e93700 1 --2- 172.21.15.1:0/2559221331 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb45406adb0 0x7fb45406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb464001020 tx=0x7fb464009040 comp rx=0 tx=0).stop 2024-03-18T19:05:28.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.606+0000 7fb478e93700 1 -- 172.21.15.1:0/2559221331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4740ace80 msgr2=0x7fb4740c98f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:28.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.606+0000 7fb478e93700 1 --2- 172.21.15.1:0/2559221331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4740ace80 0x7fb4740c98f0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fb474072d50 tx=0x7fb45c004750 comp rx=0 tx=0).stop 2024-03-18T19:05:28.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.606+0000 7fb478e93700 1 -- 172.21.15.1:0/2559221331 shutdown_connections 2024-03-18T19:05:28.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.606+0000 7fb478e93700 1 --2- 172.21.15.1:0/2559221331 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb45406adb0 0x7fb45406d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.606+0000 7fb478e93700 1 --2- 172.21.15.1:0/2559221331 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb47406a750 0x7fb47414fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.606+0000 7fb478e93700 1 --2- 172.21.15.1:0/2559221331 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4740ad850 0x7fb474157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.606+0000 7fb478e93700 1 --2- 172.21.15.1:0/2559221331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4740ace80 0x7fb4740c98f0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.606+0000 7fb478e93700 1 -- 172.21.15.1:0/2559221331 >> 172.21.15.1:0/2559221331 conn(0x7fb4740cc9b0 msgr2=0x7fb4740718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:28.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.606+0000 7fb478e93700 1 -- 172.21.15.1:0/2559221331 shutdown_connections 2024-03-18T19:05:28.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.606+0000 7fb478e93700 1 -- 172.21.15.1:0/2559221331 wait complete. 2024-03-18T19:05:28.614 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T19:05:28.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.743+0000 7f34a2b3a700 1 Processor -- start 2024-03-18T19:05:28.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.743+0000 7f34a2b3a700 1 -- start start 2024-03-18T19:05:28.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.744+0000 7f34a2b3a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f349c146190 0x7f349c06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.744+0000 7f34a2b3a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f349c1465a0 0x7f349c06ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.744+0000 7f34a2b3a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f349c06ebd0 con 0x7f349c146af0 2024-03-18T19:05:28.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.744+0000 7f34a2b3a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f349c06f850 con 0x7f349c146190 2024-03-18T19:05:28.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.744+0000 7f34a2b3a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349c06eeb0 con 0x7f349c1465a0 2024-03-18T19:05:28.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.744+0000 7f349bfff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f349c1465a0 0x7f349c06ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.744+0000 7f34a08d6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f349c146190 0x7f349c06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.744+0000 7f349bfff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f349c1465a0 0x7f349c06ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:40056/0 (socket says 172.21.15.1:40056) 2024-03-18T19:05:28.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.744+0000 7f349bfff700 1 -- 172.21.15.1:0/459929558 learned_addr learned my addr 172.21.15.1:0/459929558 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.745+0000 7f349bfff700 1 -- 172.21.15.1:0/459929558 >> v2:172.21.15.165:3300/0 conn(0x7f349c146190 msgr2=0x7f349c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.745+0000 7f349bfff700 1 --2- 172.21.15.1:0/459929558 >> v2:172.21.15.165:3300/0 conn(0x7f349c146190 0x7f349c06a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.745+0000 7f349bfff700 1 -- 172.21.15.1:0/459929558 >> v1:172.21.15.165:6789/0 conn(0x7f349c146af0 legacy=0x7f349c071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.745+0000 7f349bfff700 1 -- 172.21.15.1:0/459929558 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c06f190 con 0x7f349c1465a0 2024-03-18T19:05:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.745+0000 7f349bfff700 1 --2- 172.21.15.1:0/459929558 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f349c1465a0 0x7f349c06ac90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f34880098f0 tx=0x7f3488009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4931bbe99b8a1fac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.745+0000 7f349b7fe700 1 -- 172.21.15.1:0/459929558 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3488011830 con 0x7f349c1465a0 2024-03-18T19:05:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.745+0000 7f349b7fe700 1 -- 172.21.15.1:0/459929558 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3488011990 con 0x7f349c1465a0 2024-03-18T19:05:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.745+0000 7f349b7fe700 1 -- 172.21.15.1:0/459929558 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3488011d20 con 0x7f349c1465a0 2024-03-18T19:05:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.746+0000 7f34a2b3a700 1 -- 172.21.15.1:0/459929558 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f349c1465a0 msgr2=0x7f349c06ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.746+0000 7f34a2b3a700 1 --2- 172.21.15.1:0/459929558 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f349c1465a0 0x7f349c06ac90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f34880098f0 tx=0x7f3488009c00 comp rx=0 tx=0).stop 2024-03-18T19:05:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.746+0000 7f34a2b3a700 1 -- 172.21.15.1:0/459929558 shutdown_connections 2024-03-18T19:05:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.746+0000 7f34a2b3a700 1 --2- 172.21.15.1:0/459929558 >> v2:172.21.15.165:3300/0 conn(0x7f349c146190 0x7f349c06a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.746+0000 7f34a2b3a700 1 --2- 172.21.15.1:0/459929558 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f349c1465a0 0x7f349c06ac90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.746+0000 7f34a2b3a700 1 -- 172.21.15.1:0/459929558 >> 172.21.15.1:0/459929558 conn(0x7f349c0b1d20 msgr2=0x7f349c0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:28.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.746+0000 7f34a2b3a700 1 -- 172.21.15.1:0/459929558 shutdown_connections 2024-03-18T19:05:28.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.746+0000 7f34a2b3a700 1 -- 172.21.15.1:0/459929558 wait complete. 2024-03-18T19:05:28.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.747+0000 7f34a2b3a700 1 Processor -- start 2024-03-18T19:05:28.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.747+0000 7f34a2b3a700 1 -- start start 2024-03-18T19:05:28.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.748+0000 7f34a2b3a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f349c1465a0 0x7f349c144aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.748+0000 7f34a2b3a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f349c146af0 0x7f349c0c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.748+0000 7f34a2b3a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f349c0c50e0 0x7f349c1e6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.748+0000 7f34a2b3a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c144fe0 con 0x7f349c146af0 2024-03-18T19:05:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.748+0000 7f34a2b3a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c1e5a60 con 0x7f349c1465a0 2024-03-18T19:05:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.748+0000 7f34a2b3a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349c1e4e50 con 0x7f349c0c50e0 2024-03-18T19:05:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.748+0000 7f349bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f349c146af0 0x7f349c0c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.748+0000 7f349bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f349c146af0 0x7f349c0c5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33782/0 (socket says 172.21.15.1:33782) 2024-03-18T19:05:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.748+0000 7f349bfff700 1 -- 172.21.15.1:0/2805870260 learned_addr learned my addr 172.21.15.1:0/2805870260 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.748+0000 7f34a10d7700 1 --2- 172.21.15.1:0/2805870260 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f349c0c50e0 0x7f349c1e6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.748+0000 7f34a08d6700 1 --2- 172.21.15.1:0/2805870260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f349c1465a0 0x7f349c144aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.748+0000 7f349bfff700 1 -- 172.21.15.1:0/2805870260 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f349c0c50e0 msgr2=0x7f349c1e6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.748+0000 7f349bfff700 1 --2- 172.21.15.1:0/2805870260 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f349c0c50e0 0x7f349c1e6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.748+0000 7f349bfff700 1 -- 172.21.15.1:0/2805870260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f349c1465a0 msgr2=0x7f349c144aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:28.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.748+0000 7f349bfff700 1 --2- 172.21.15.1:0/2805870260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f349c1465a0 0x7f349c144aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:28.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.748+0000 7f349bfff700 1 -- 172.21.15.1:0/2805870260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3488009550 con 0x7f349c146af0 2024-03-18T19:05:28.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.748+0000 7f349bfff700 1 --2- 172.21.15.1:0/2805870260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f349c146af0 0x7f349c0c5f70 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f3488009c40 tx=0x7f3488004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:28.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.749+0000 7f34997fa700 1 -- 172.21.15.1:0/2805870260 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34880045d0 con 0x7f349c146af0 2024-03-18T19:05:28.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.749+0000 7f34997fa700 1 -- 172.21.15.1:0/2805870260 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f348801f070 con 0x7f349c146af0 2024-03-18T19:05:28.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.749+0000 7f34a2b3a700 1 -- 172.21.15.1:0/2805870260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f349c1e5130 con 0x7f349c146af0 2024-03-18T19:05:28.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.749+0000 7f34997fa700 1 -- 172.21.15.1:0/2805870260 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34880038a0 con 0x7f349c146af0 2024-03-18T19:05:28.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.749+0000 7f34a2b3a700 1 -- 172.21.15.1:0/2805870260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f349c1e56f0 con 0x7f349c146af0 2024-03-18T19:05:28.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.750+0000 7f34a2b3a700 1 -- 172.21.15.1:0/2805870260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f349c1e5880 con 0x7f349c146af0 2024-03-18T19:05:28.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.750+0000 7f34997fa700 1 -- 172.21.15.1:0/2805870260 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3488003a00 con 0x7f349c146af0 2024-03-18T19:05:28.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.751+0000 7f34997fa700 1 --2- 172.21.15.1:0/2805870260 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f348006adb0 0x7f348006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:28.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.751+0000 7f34997fa700 1 -- 172.21.15.1:0/2805870260 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3488021070 con 0x7f349c146af0 2024-03-18T19:05:28.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.751+0000 7f34a08d6700 1 --2- 172.21.15.1:0/2805870260 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f348006adb0 0x7f348006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:28.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.751+0000 7f34a08d6700 1 --2- 172.21.15.1:0/2805870260 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f348006adb0 0x7f348006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f349c0726e0 tx=0x7f3490009210 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:28.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:28.755+0000 7f34997fa700 1 -- 172.21.15.1:0/2805870260 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3488055df0 con 0x7f349c146af0 2024-03-18T19:05:29.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:29.004+0000 7f34a2b3a700 1 -- 172.21.15.1:0/2805870260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f349c1e5bd0 con 0x7f349c146af0 2024-03-18T19:05:29.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:29.004+0000 7f34997fa700 1 -- 172.21.15.1:0/2805870260 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v33) v1 ==== 53+0+73 (secure 0 0 0) 0x7f348802b070 con 0x7f349c146af0 2024-03-18T19:05:29.001 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T19:05:29.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:29.007+0000 7f34a2b3a700 1 -- 172.21.15.1:0/2805870260 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f348006adb0 msgr2=0x7f348006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:29.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:29.007+0000 7f34a2b3a700 1 --2- 172.21.15.1:0/2805870260 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f348006adb0 0x7f348006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f349c0726e0 tx=0x7f3490009210 comp rx=0 tx=0).stop 2024-03-18T19:05:29.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:29.007+0000 7f34a2b3a700 1 -- 172.21.15.1:0/2805870260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f349c146af0 msgr2=0x7f349c0c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:29.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:29.007+0000 7f34a2b3a700 1 --2- 172.21.15.1:0/2805870260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f349c146af0 0x7f349c0c5f70 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f3488009c40 tx=0x7f3488004750 comp rx=0 tx=0).stop 2024-03-18T19:05:29.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:29.007+0000 7f34a2b3a700 1 -- 172.21.15.1:0/2805870260 shutdown_connections 2024-03-18T19:05:29.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:29.007+0000 7f34a2b3a700 1 --2- 172.21.15.1:0/2805870260 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f348006adb0 0x7f348006d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:29.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:29.007+0000 7f34a2b3a700 1 --2- 172.21.15.1:0/2805870260 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f349c0c50e0 0x7f349c1e6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:29.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:29.007+0000 7f34a2b3a700 1 --2- 172.21.15.1:0/2805870260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f349c1465a0 0x7f349c144aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:29.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:29.007+0000 7f34a2b3a700 1 --2- 172.21.15.1:0/2805870260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f349c146af0 0x7f349c0c5f70 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:29.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:29.007+0000 7f34a2b3a700 1 -- 172.21.15.1:0/2805870260 >> 172.21.15.1:0/2805870260 conn(0x7f349c0b1d20 msgr2=0x7f349c1452c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:29.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:29.007+0000 7f34a2b3a700 1 -- 172.21.15.1:0/2805870260 shutdown_connections 2024-03-18T19:05:29.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:29.007+0000 7f34a2b3a700 1 -- 172.21.15.1:0/2805870260 wait complete. 2024-03-18T19:05:29.015 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T19:05:29.015 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T19:05:29.015 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-18T19:05:29.015 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T19:05:29.015 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T19:05:29.015 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T19:05:29.015 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T19:05:29.015 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T19:05:29.015 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:05:29.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:29.016 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-18T19:05:29.034 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T19:05:29.034 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T19:05:29.034 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T19:05:29.034 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:05:29.034 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T19:05:29.034 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:05:29.034 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:05:29.034 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-18T19:05:29.034 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-18T19:05:29.035 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:05:29.035 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-18T19:05:29.035 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-18T19:05:29.035 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T19:05:29.035 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:05:29.035 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-18T19:05:29.035 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T19:05:29.035 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 41640sec preferred_lft 41640sec 2024-03-18T19:05:29.035 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link 2024-03-18T19:05:29.035 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:05:29.035 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:05:29.035 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:05:29.035 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-18T19:05:29.036 INFO:teuthology.orchestra.run.smithi001.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T19:05:29.036 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:05:29.036 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T19:05:29.036 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:05:29.036 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::4ca9:f5ff:feb3:79a3/64 scope link 2024-03-18T19:05:29.036 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:05:29.036 INFO:teuthology.orchestra.run.smithi001.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T19:05:29.036 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T19:05:29.036 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::282f:a4ff:fe1a:9908/64 scope link 2024-03-18T19:05:29.036 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:05:29.036 INFO:teuthology.orchestra.run.smithi001.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T19:05:29.036 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2e:5c:60:78:b2:8c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T19:05:29.036 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::2c5c:60ff:fe78:b28c/64 scope link 2024-03-18T19:05:29.036 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:05:29.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:29.037 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T19:05:29.101 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-18T19:05:29.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:29.103 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-18T19:05:29.162 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-18T19:05:29.163 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T19:05:29.163 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T19:05:29.163 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T19:05:29.163 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T19:05:29.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:29.163 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T19:05:29.163 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T19:05:29.163 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T19:05:29.164 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T19:05:29.164 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T19:05:29.164 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T19:05:29.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:29.269 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T19:05:29.287 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T19:05:29.287 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T19:05:29.287 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:05:29.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:29.287 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-18T19:05:29.355 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T19:05:29.356 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T19:05:29.356 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T19:05:29.414 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:05:29.414 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T19:05:29.414 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T19:05:29.472 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T19:05:29.540 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T19:05:29.540 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T19:05:29.607 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T19:05:29.608 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T19:05:29.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:29.609 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T19:05:29.668 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T19:05:29.668 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T19:05:29.710 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T19:05:29.739 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T19:05:29.739 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T19:05:29.798 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-18T19:05:29.803+0000 7f9d016045c0 -1 init, newargv = 0x5618ef79eea0 newargc=15 2024-03-18T19:05:29.799 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[112967]: starting ceph client 2024-03-18T19:05:29.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[112967]: starting fuse 2024-03-18T19:05:29.808 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T19:05:29.809 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T19:05:29.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:29.809 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T19:05:29.868 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T19:05:29.868 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-03-18T19:05:29.868 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T19:05:29.868 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-18T19:05:29.868 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-18T19:05:29.868 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-18T19:05:29.869 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-18T19:05:29.869 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:05:29.869 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-18T19:05:29.869 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T19:05:29.869 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-18T19:05:29.869 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T19:05:29.869 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:05:29.869 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-18T19:05:29.869 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-18T19:05:29.869 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-18T19:05:29.869 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-18T19:05:29.869 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:05:29.869 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-18T19:05:29.869 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T19:05:29.869 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T19:05:29.869 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T19:05:29.870 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-18T19:05:29.870 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-18T19:05:29.870 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-18T19:05:29.870 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T19:05:29.870 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:05:29.870 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-18T19:05:29.870 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-18T19:05:29.991 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.112967.asok 2024-03-18T19:05:30.173 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.112967.asok 2024-03-18T19:05:30.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:30.174 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.112967.asok status 2024-03-18T19:05:30.335 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:05:30.335 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-18T19:05:30.335 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T19:05:30.335 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T19:05:30.335 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-18T19:05:30.335 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-18T19:05:30.335 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T19:05:30.335 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "112967", 2024-03-18T19:05:30.335 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-18T19:05:30.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:30.336 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-18T19:05:30.336 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-18T19:05:30.336 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 6951, 2024-03-18T19:05:30.336 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-18T19:05:30.336 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-18T19:05:30.336 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-18T19:05:30.336 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6951 2024-03-18T19:05:30.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:30.336 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T19:05:30.336 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T19:05:30.336 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T19:05:30.336 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2525962391 2024-03-18T19:05:30.336 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:30.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:30.336 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T19:05:30.337 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T19:05:30.337 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T19:05:30.337 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2525962391 2024-03-18T19:05:30.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:05:30.337 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.6951 192.168.0.1:0/2525962391", 2024-03-18T19:05:30.337 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/2525962391", 2024-03-18T19:05:30.337 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-18T19:05:30.337 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 33, 2024-03-18T19:05:30.337 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 89, 2024-03-18T19:05:30.337 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-18T19:05:30.337 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-18T19:05:30.337 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-18T19:05:30.337 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:05:30.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:30.345 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-18T19:05:30.345 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T19:05:30.363 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-18T19:05:30.364 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T19:05:30.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:30.364 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T19:05:30.433 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-18T19:05:30.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.604+0000 7fa00d21a700 1 Processor -- start 2024-03-18T19:05:30.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.604+0000 7fa00d21a700 1 -- start start 2024-03-18T19:05:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.604+0000 7fa00d21a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa008160a70 0x7fa0080d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.604+0000 7fa00d21a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa008160e50 0x7fa0080cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.604+0000 7fa00d21a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa008162170 con 0x7fa0081612d0 2024-03-18T19:05:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.604+0000 7fa00d21a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0080cef30 con 0x7fa008160e50 2024-03-18T19:05:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.604+0000 7fa00d21a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0080cf210 con 0x7fa008160a70 2024-03-18T19:05:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.605+0000 7fa00659c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa008160e50 0x7fa0080cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.605+0000 7fa00659c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa008160e50 0x7fa0080cffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33784/0 (socket says 172.21.15.1:33784) 2024-03-18T19:05:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.605+0000 7fa00659c700 1 -- 172.21.15.1:0/1085693975 learned_addr learned my addr 172.21.15.1:0/1085693975 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.605+0000 7fa006d9d700 1 --2- 172.21.15.1:0/1085693975 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa008160a70 0x7fa0080d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:30.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.605+0000 7fa00659c700 1 -- 172.21.15.1:0/1085693975 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa008160a70 msgr2=0x7fa0080d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.605+0000 7fa00659c700 1 --2- 172.21.15.1:0/1085693975 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa008160a70 0x7fa0080d3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.605+0000 7fa00659c700 1 -- 172.21.15.1:0/1085693975 >> v1:172.21.15.165:6789/0 conn(0x7fa0081612d0 legacy=0x7fa0080c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.605+0000 7fa00659c700 1 -- 172.21.15.1:0/1085693975 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0080cf4f0 con 0x7fa008160e50 2024-03-18T19:05:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.605+0000 7fa00659c700 1 --2- 172.21.15.1:0/1085693975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008160e50 0x7fa0080cffd0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc0098f0 tx=0x7f9ffc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=687b6852da1012c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.605+0000 7fa005d9b700 1 -- 172.21.15.1:0/1085693975 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc011830 con 0x7fa008160e50 2024-03-18T19:05:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.605+0000 7fa005d9b700 1 -- 172.21.15.1:0/1085693975 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ffc011990 con 0x7fa008160e50 2024-03-18T19:05:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.605+0000 7fa005d9b700 1 -- 172.21.15.1:0/1085693975 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc011d20 con 0x7fa008160e50 2024-03-18T19:05:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.606+0000 7fa00d21a700 1 -- 172.21.15.1:0/1085693975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008160e50 msgr2=0x7fa0080cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.606+0000 7fa00d21a700 1 --2- 172.21.15.1:0/1085693975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008160e50 0x7fa0080cffd0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc0098f0 tx=0x7f9ffc009c00 comp rx=0 tx=0).stop 2024-03-18T19:05:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.606+0000 7fa00d21a700 1 -- 172.21.15.1:0/1085693975 shutdown_connections 2024-03-18T19:05:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.606+0000 7fa00d21a700 1 --2- 172.21.15.1:0/1085693975 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa008160a70 0x7fa0080d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.606+0000 7fa00d21a700 1 --2- 172.21.15.1:0/1085693975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008160e50 0x7fa0080cffd0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.606+0000 7fa00d21a700 1 -- 172.21.15.1:0/1085693975 >> 172.21.15.1:0/1085693975 conn(0x7fa00809c560 msgr2=0x7fa0080d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:30.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.606+0000 7fa00d21a700 1 -- 172.21.15.1:0/1085693975 shutdown_connections 2024-03-18T19:05:30.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.606+0000 7fa00d21a700 1 -- 172.21.15.1:0/1085693975 wait complete. 2024-03-18T19:05:30.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.608+0000 7fa00d21a700 1 Processor -- start 2024-03-18T19:05:30.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.608+0000 7fa00d21a700 1 -- start start 2024-03-18T19:05:30.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.608+0000 7fa00d21a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa008160a70 0x7fa0081e91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:30.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.609+0000 7fa00d21a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008160e50 0x7fa008158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:30.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.609+0000 7fa00d21a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa0081612d0 0x7fa0081e7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:30.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.609+0000 7fa00d21a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0081593d0 con 0x7fa008160e50 2024-03-18T19:05:30.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.609+0000 7fa00d21a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0080b66b0 con 0x7fa008160a70 2024-03-18T19:05:30.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.609+0000 7fa00d21a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0081e98b0 con 0x7fa0081612d0 2024-03-18T19:05:30.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.609+0000 7fa00659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008160e50 0x7fa008158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:30.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.609+0000 7fa00659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008160e50 0x7fa008158e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33794/0 (socket says 172.21.15.1:33794) 2024-03-18T19:05:30.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.609+0000 7fa00659c700 1 -- 172.21.15.1:0/63299300 learned_addr learned my addr 172.21.15.1:0/63299300 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:30.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.609+0000 7fa006d9d700 1 --2- 172.21.15.1:0/63299300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa008160a70 0x7fa0081e91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:30.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.609+0000 7fa00759e700 1 --2- 172.21.15.1:0/63299300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa0081612d0 0x7fa0081e7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:30.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.609+0000 7fa00659c700 1 -- 172.21.15.1:0/63299300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa0081612d0 msgr2=0x7fa0081e7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:30.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.609+0000 7fa00659c700 1 --2- 172.21.15.1:0/63299300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa0081612d0 0x7fa0081e7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:30.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.609+0000 7fa00659c700 1 -- 172.21.15.1:0/63299300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa008160a70 msgr2=0x7fa0081e91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:30.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.609+0000 7fa00659c700 1 --2- 172.21.15.1:0/63299300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa008160a70 0x7fa0081e91d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:30.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.609+0000 7fa00659c700 1 -- 172.21.15.1:0/63299300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ffc009550 con 0x7fa008160e50 2024-03-18T19:05:30.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.609+0000 7fa00659c700 1 --2- 172.21.15.1:0/63299300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008160e50 0x7fa008158e90 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc000c00 tx=0x7f9ffc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:30.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.609+0000 7f9fef7fe700 1 -- 172.21.15.1:0/63299300 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc004400 con 0x7fa008160e50 2024-03-18T19:05:30.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.609+0000 7f9fef7fe700 1 -- 172.21.15.1:0/63299300 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ffc01f070 con 0x7fa008160e50 2024-03-18T19:05:30.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.609+0000 7f9fef7fe700 1 -- 172.21.15.1:0/63299300 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc0038e0 con 0x7fa008160e50 2024-03-18T19:05:30.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.610+0000 7fa00d21a700 1 -- 172.21.15.1:0/63299300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0081e9b90 con 0x7fa008160e50 2024-03-18T19:05:30.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.610+0000 7fa00d21a700 1 -- 172.21.15.1:0/63299300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0080b68c0 con 0x7fa008160e50 2024-03-18T19:05:30.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.611+0000 7fa00d21a700 1 -- 172.21.15.1:0/63299300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa008067390 con 0x7fa008160e50 2024-03-18T19:05:30.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.611+0000 7f9fef7fe700 1 -- 172.21.15.1:0/63299300 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9ffc021030 con 0x7fa008160e50 2024-03-18T19:05:30.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.612+0000 7f9fef7fe700 1 --2- 172.21.15.1:0/63299300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9ff406adb0 0x7f9ff406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:30.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.612+0000 7fa006d9d700 1 --2- 172.21.15.1:0/63299300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9ff406adb0 0x7f9ff406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:30.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.612+0000 7f9fef7fe700 1 -- 172.21.15.1:0/63299300 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9ffc08a980 con 0x7fa008160e50 2024-03-18T19:05:30.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.612+0000 7fa006d9d700 1 --2- 172.21.15.1:0/63299300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9ff406adb0 0x7f9ff406d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8009b40 tx=0x7f9ff8009210 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:30.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.616+0000 7f9fef7fe700 1 -- 172.21.15.1:0/63299300 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9ffc055920 con 0x7fa008160e50 2024-03-18T19:05:30.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.845+0000 7fa00d21a700 1 -- 172.21.15.1:0/63299300 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0080d40b0 con 0x7f9ff406adb0 2024-03-18T19:05:30.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.846+0000 7f9fef7fe700 1 -- 172.21.15.1:0/63299300 <== mgr.6238 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f9ff8004040 con 0x7f9ff406adb0 2024-03-18T19:05:30.842 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T19:05:30.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:05:30.842 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephfs" 2024-03-18T19:05:30.843 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:05:30.843 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T19:05:30.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.848+0000 7fa00d21a700 1 -- 172.21.15.1:0/63299300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9ff406adb0 msgr2=0x7f9ff406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:30.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.848+0000 7fa00d21a700 1 --2- 172.21.15.1:0/63299300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9ff406adb0 0x7f9ff406d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8009b40 tx=0x7f9ff8009210 comp rx=0 tx=0).stop 2024-03-18T19:05:30.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.848+0000 7fa00d21a700 1 -- 172.21.15.1:0/63299300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008160e50 msgr2=0x7fa008158e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:30.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.848+0000 7fa00d21a700 1 --2- 172.21.15.1:0/63299300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008160e50 0x7fa008158e90 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc000c00 tx=0x7f9ffc004750 comp rx=0 tx=0).stop 2024-03-18T19:05:30.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.849+0000 7fa00d21a700 1 -- 172.21.15.1:0/63299300 shutdown_connections 2024-03-18T19:05:30.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.849+0000 7fa00d21a700 1 --2- 172.21.15.1:0/63299300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9ff406adb0 0x7f9ff406d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:30.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.849+0000 7fa00d21a700 1 --2- 172.21.15.1:0/63299300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa0081612d0 0x7fa0081e7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:30.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.849+0000 7fa00d21a700 1 --2- 172.21.15.1:0/63299300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa008160a70 0x7fa0081e91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:30.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.849+0000 7fa00d21a700 1 --2- 172.21.15.1:0/63299300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa008160e50 0x7fa008158e90 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:30.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.849+0000 7fa00d21a700 1 -- 172.21.15.1:0/63299300 >> 172.21.15.1:0/63299300 conn(0x7fa00809c560 msgr2=0x7fa00806b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:30.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.849+0000 7fa00d21a700 1 -- 172.21.15.1:0/63299300 shutdown_connections 2024-03-18T19:05:30.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.849+0000 7fa00d21a700 1 -- 172.21.15.1:0/63299300 wait complete. 2024-03-18T19:05:30.856 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-18T19:05:30.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.987+0000 7f9be0cc8700 1 Processor -- start 2024-03-18T19:05:30.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.987+0000 7f9be0cc8700 1 -- start start 2024-03-18T19:05:30.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.988+0000 7f9be0cc8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc15d5b0 0x7f9bdc15d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:30.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.988+0000 7f9be0cc8700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bdc0b3b00 con 0x7f9bdc06ca80 2024-03-18T19:05:30.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.988+0000 7f9be0cc8700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bdc0cf7f0 con 0x7f9bdc099a40 2024-03-18T19:05:30.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.988+0000 7f9be0cc8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc0cbbb0 con 0x7f9bdc15d5b0 2024-03-18T19:05:30.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.988+0000 7f9bdad9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9bdc06ca80 0x7f9bdc0cbfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57684/0 (socket says 172.21.15.1:57684) 2024-03-18T19:05:30.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.988+0000 7f9bdad9d700 1 -- 172.21.15.1:0/2738408025 learned_addr learned my addr 172.21.15.1:0/2738408025 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:30.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.988+0000 7f9bda59c700 1 --2- 172.21.15.1:0/2738408025 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc15d5b0 0x7f9bdc15d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:30.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.988+0000 7f9bd959a700 1 -- 172.21.15.1:0/2738408025 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102732286 0 0) 0x7f9bdc0b3b00 con 0x7f9bdc06ca80 2024-03-18T19:05:30.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.988+0000 7f9bd959a700 1 -- 172.21.15.1:0/2738408025 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bc40035b0 con 0x7f9bdc06ca80 2024-03-18T19:05:30.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bd959a700 1 -- 172.21.15.1:0/2738408025 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 80835686 0 0) 0x7f9bc40035b0 con 0x7f9bdc06ca80 2024-03-18T19:05:30.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bd959a700 1 -- 172.21.15.1:0/2738408025 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc15d5b0 msgr2=0x7f9bdc15d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:30.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bd959a700 1 --2- 172.21.15.1:0/2738408025 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc15d5b0 0x7f9bdc15d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:30.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bd959a700 1 -- 172.21.15.1:0/2738408025 >> v1:172.21.15.165:6789/0 conn(0x7f9bdc099a40 legacy=0x7f9bdc099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:30.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bd959a700 1 -- 172.21.15.1:0/2738408025 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bdc0d04d0 con 0x7f9bdc06ca80 2024-03-18T19:05:30.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bd959a700 1 -- 172.21.15.1:0/2738408025 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f9bd40019f0 con 0x7f9bdc06ca80 2024-03-18T19:05:30.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bd959a700 1 -- 172.21.15.1:0/2738408025 >> v1:172.21.15.1:6789/0 conn(0x7f9bdc06ca80 legacy=0x7f9bdc0cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:30.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bd959a700 1 --2- 172.21.15.1:0/2738408025 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc15d5b0 0x7f9bc4003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:30.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bd959a700 1 --2- 172.21.15.1:0/2738408025 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bc4003f70 0x7f9bc4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:30.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bd959a700 1 --2- 172.21.15.1:0/2738408025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bc4004d90 0x7f9bc4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:30.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bd959a700 1 -- 172.21.15.1:0/2738408025 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc0cbbb0 con 0x7f9bc4004d90 2024-03-18T19:05:30.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bd959a700 1 -- 172.21.15.1:0/2738408025 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc400bb20 con 0x7f9bc4003f70 2024-03-18T19:05:30.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bd959a700 1 -- 172.21.15.1:0/2738408025 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc400bce0 con 0x7f9bdc15d5b0 2024-03-18T19:05:30.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bdad9d700 1 --2- 172.21.15.1:0/2738408025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bc4004d90 0x7f9bc4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:30.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bda59c700 1 --2- 172.21.15.1:0/2738408025 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc15d5b0 0x7f9bc4003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:30.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bd9d9b700 1 --2- 172.21.15.1:0/2738408025 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bc4003f70 0x7f9bc4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:30.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bdad9d700 1 -- 172.21.15.1:0/2738408025 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc15d5b0 msgr2=0x7f9bc4003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:30.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bdad9d700 1 --2- 172.21.15.1:0/2738408025 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bdc15d5b0 0x7f9bc4003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:30.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bdad9d700 1 -- 172.21.15.1:0/2738408025 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bc4003f70 msgr2=0x7f9bc4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:30.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bdad9d700 1 --2- 172.21.15.1:0/2738408025 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bc4003f70 0x7f9bc4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:30.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bdad9d700 1 -- 172.21.15.1:0/2738408025 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc40035b0 con 0x7f9bc4004d90 2024-03-18T19:05:30.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.989+0000 7f9bdad9d700 1 --2- 172.21.15.1:0/2738408025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bc4004d90 0x7f9bc4005210 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400a830 tx=0x7f9bc4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc7047f12d6028f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:30.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.990+0000 7f9bd959a700 1 -- 172.21.15.1:0/2738408025 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4012600 con 0x7f9bc4004d90 2024-03-18T19:05:30.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.990+0000 7f9bd959a700 1 -- 172.21.15.1:0/2738408025 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd4012780 con 0x7f9bc4004d90 2024-03-18T19:05:30.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.990+0000 7f9bd959a700 1 -- 172.21.15.1:0/2738408025 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4012b30 con 0x7f9bc4004d90 2024-03-18T19:05:30.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.990+0000 7f9be0cc8700 1 -- 172.21.15.1:0/2738408025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bc4004d90 msgr2=0x7f9bc4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:30.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.990+0000 7f9be0cc8700 1 --2- 172.21.15.1:0/2738408025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bc4004d90 0x7f9bc4005210 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400a830 tx=0x7f9bc4005880 comp rx=0 tx=0).stop 2024-03-18T19:05:30.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.990+0000 7f9bdad9d700 1 -- 172.21.15.1:0/2738408025 reap_dead start 2024-03-18T19:05:30.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.990+0000 7f9be0cc8700 1 -- 172.21.15.1:0/2738408025 shutdown_connections 2024-03-18T19:05:30.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.990+0000 7f9be0cc8700 1 -- 172.21.15.1:0/2738408025 >> 172.21.15.1:0/2738408025 conn(0x7f9bdc0b2bd0 msgr2=0x7f9bdc0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:30.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.990+0000 7f9be0cc8700 1 -- 172.21.15.1:0/2738408025 shutdown_connections 2024-03-18T19:05:30.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.990+0000 7f9be0cc8700 1 -- 172.21.15.1:0/2738408025 wait complete. 2024-03-18T19:05:30.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.992+0000 7f9be0cc8700 1 Processor -- start 2024-03-18T19:05:30.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.992+0000 7f9be0cc8700 1 -- start start 2024-03-18T19:05:30.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.992+0000 7f9be0cc8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bc4004d90 0x7f9bdc14e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:30.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.992+0000 7f9be0cc8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc0ca1c0 0x7f9bdc0ca640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:30.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.992+0000 7f9be0cc8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bdc09e5d0 0x7f9bdc09ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:30.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.992+0000 7f9be0cc8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc0b6940 con 0x7f9bdc09e5d0 2024-03-18T19:05:30.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.992+0000 7f9be0cc8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc0cab80 con 0x7f9bdc0ca1c0 2024-03-18T19:05:30.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.992+0000 7f9be0cc8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc09ef90 con 0x7f9bc4004d90 2024-03-18T19:05:30.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.992+0000 7f9bdad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bdc09e5d0 0x7f9bdc09ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:30.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.992+0000 7f9bdad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bdc09e5d0 0x7f9bdc09ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33804/0 (socket says 172.21.15.1:33804) 2024-03-18T19:05:30.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.992+0000 7f9bdad9d700 1 -- 172.21.15.1:0/848376011 learned_addr learned my addr 172.21.15.1:0/848376011 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:30.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.993+0000 7f9bd9d9b700 1 --2- 172.21.15.1:0/848376011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc0ca1c0 0x7f9bdc0ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:30.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.993+0000 7f9bda59c700 1 --2- 172.21.15.1:0/848376011 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bc4004d90 0x7f9bdc14e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:30.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.993+0000 7f9bdad9d700 1 -- 172.21.15.1:0/848376011 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bc4004d90 msgr2=0x7f9bdc14e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:30.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.993+0000 7f9bdad9d700 1 --2- 172.21.15.1:0/848376011 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bc4004d90 0x7f9bdc14e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:30.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.993+0000 7f9bdad9d700 1 -- 172.21.15.1:0/848376011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc0ca1c0 msgr2=0x7f9bdc0ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:30.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.993+0000 7f9bdad9d700 1 --2- 172.21.15.1:0/848376011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc0ca1c0 0x7f9bdc0ca640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:30.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.993+0000 7f9bdad9d700 1 -- 172.21.15.1:0/848376011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bdc0b5a70 con 0x7f9bdc09e5d0 2024-03-18T19:05:30.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.993+0000 7f9bd9d9b700 1 --2- 172.21.15.1:0/848376011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc0ca1c0 0x7f9bdc0ca640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:30.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.993+0000 7f9bdad9d700 1 --2- 172.21.15.1:0/848376011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bdc09e5d0 0x7f9bdc09ea50 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400b010 tx=0x7f9bd4001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:30.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.993+0000 7f9bcaffd700 1 -- 172.21.15.1:0/848376011 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4012600 con 0x7f9bdc09e5d0 2024-03-18T19:05:30.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.993+0000 7f9bcaffd700 1 -- 172.21.15.1:0/848376011 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd401e430 con 0x7f9bdc09e5d0 2024-03-18T19:05:30.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.993+0000 7f9be0cc8700 1 -- 172.21.15.1:0/848376011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bdc0b5cc0 con 0x7f9bdc09e5d0 2024-03-18T19:05:30.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.993+0000 7f9bcaffd700 1 -- 172.21.15.1:0/848376011 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd401e7a0 con 0x7f9bdc09e5d0 2024-03-18T19:05:30.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.993+0000 7f9be0cc8700 1 -- 172.21.15.1:0/848376011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bdc0b6280 con 0x7f9bdc09e5d0 2024-03-18T19:05:30.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.994+0000 7f9be0cc8700 1 -- 172.21.15.1:0/848376011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bdc073350 con 0x7f9bdc09e5d0 2024-03-18T19:05:30.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.995+0000 7f9bcaffd700 1 -- 172.21.15.1:0/848376011 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 30) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9bd4024020 con 0x7f9bdc09e5d0 2024-03-18T19:05:30.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.995+0000 7f9bcaffd700 1 --2- 172.21.15.1:0/848376011 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9bbc06adb0 0x7f9bbc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:30.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.995+0000 7f9bcaffd700 1 -- 172.21.15.1:0/848376011 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9bd408b950 con 0x7f9bdc09e5d0 2024-03-18T19:05:30.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.995+0000 7f9bda59c700 1 --2- 172.21.15.1:0/848376011 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9bbc06adb0 0x7f9bbc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:30.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.995+0000 7f9bda59c700 1 --2- 172.21.15.1:0/848376011 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9bbc06adb0 0x7f9bbc06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0009aa0 tx=0x7f9bd0008040 comp rx=0 tx=0).ready entity=mgr.6238 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:30.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:30.999+0000 7f9bcaffd700 1 -- 172.21.15.1:0/848376011 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9bd4056a90 con 0x7f9bdc09e5d0 2024-03-18T19:05:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:31.260+0000 7f9be0cc8700 1 -- 172.21.15.1:0/848376011 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f9bdc06b3f0 con 0x7f9bdc09e5d0 2024-03-18T19:05:32.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.070+0000 7f9bcaffd700 1 -- 172.21.15.1:0/848376011 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 31) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f9bd4056e60 con 0x7f9bdc09e5d0 2024-03-18T19:05:32.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.072+0000 7f9bcaffd700 1 -- 172.21.15.1:0/848376011 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v31) v1 ==== 92+0+0 (secure 0 0 0) 0x7f9bd406f6e0 con 0x7f9bdc09e5d0 2024-03-18T19:05:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.075+0000 7f9be0cc8700 1 -- 172.21.15.1:0/848376011 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9bbc06adb0 msgr2=0x7f9bbc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.075+0000 7f9be0cc8700 1 --2- 172.21.15.1:0/848376011 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9bbc06adb0 0x7f9bbc06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0009aa0 tx=0x7f9bd0008040 comp rx=0 tx=0).stop 2024-03-18T19:05:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.075+0000 7f9be0cc8700 1 -- 172.21.15.1:0/848376011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bdc09e5d0 msgr2=0x7f9bdc09ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.075+0000 7f9be0cc8700 1 --2- 172.21.15.1:0/848376011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bdc09e5d0 0x7f9bdc09ea50 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400b010 tx=0x7f9bd4001d80 comp rx=0 tx=0).stop 2024-03-18T19:05:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.075+0000 7f9be0cc8700 1 -- 172.21.15.1:0/848376011 shutdown_connections 2024-03-18T19:05:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.075+0000 7f9be0cc8700 1 --2- 172.21.15.1:0/848376011 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9bbc06adb0 0x7f9bbc06d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.075+0000 7f9be0cc8700 1 --2- 172.21.15.1:0/848376011 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9bc4004d90 0x7f9bdc14e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.075+0000 7f9be0cc8700 1 --2- 172.21.15.1:0/848376011 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9bdc09e5d0 0x7f9bdc09ea50 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.075+0000 7f9be0cc8700 1 --2- 172.21.15.1:0/848376011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc0ca1c0 0x7f9bdc0ca640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.075+0000 7f9be0cc8700 1 -- 172.21.15.1:0/848376011 >> 172.21.15.1:0/848376011 conn(0x7f9bdc0b2bd0 msgr2=0x7f9bdc06f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:32.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.075+0000 7f9be0cc8700 1 -- 172.21.15.1:0/848376011 shutdown_connections 2024-03-18T19:05:32.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.075+0000 7f9be0cc8700 1 -- 172.21.15.1:0/848376011 wait complete. 2024-03-18T19:05:32.087 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-18T19:05:32.208 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:32.211+0000 7f77842e3240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:05:32.222 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:32.226+0000 7f23cf2df240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:05:32.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.229+0000 7f24aa6b1700 1 Processor -- start 2024-03-18T19:05:32.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.229+0000 7f24aa6b1700 1 -- start start 2024-03-18T19:05:32.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.230+0000 7f24aa6b1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f24a4160e50 0x7f24a40cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:32.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.230+0000 7f24aa6b1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f24a41612d0 0x7f24a41616f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:32.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.230+0000 7f24aa6b1700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a4162170 con 0x7f24a4160a70 2024-03-18T19:05:32.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.230+0000 7f24aa6b1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a4161c30 con 0x7f24a41612d0 2024-03-18T19:05:32.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.230+0000 7f24aa6b1700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a40cef30 con 0x7f24a4160e50 2024-03-18T19:05:32.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.230+0000 7f24a8c4e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f24a41612d0 0x7f24a41616f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:32.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.230+0000 7f24a8c4e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f24a41612d0 0x7f24a41616f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33830/0 (socket says 172.21.15.1:33830) 2024-03-18T19:05:32.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.230+0000 7f24a8c4e700 1 -- 172.21.15.1:0/2345514494 learned_addr learned my addr 172.21.15.1:0/2345514494 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:32.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.230+0000 7f24a37fe700 1 --2- 172.21.15.1:0/2345514494 >> v2:172.21.15.165:3300/0 conn(0x7f24a4160e50 0x7f24a40cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:32.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.230+0000 7f24a2ffd700 1 -- 172.21.15.1:0/2345514494 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1079409616 0 0) 0x7f2494001110 con 0x7f24a4160a70 2024-03-18T19:05:32.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.230+0000 7f24a8c4e700 1 -- 172.21.15.1:0/2345514494 >> v2:172.21.15.165:3300/0 conn(0x7f24a4160e50 msgr2=0x7f24a40cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:32.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.230+0000 7f24a8c4e700 1 --2- 172.21.15.1:0/2345514494 >> v2:172.21.15.165:3300/0 conn(0x7f24a4160e50 0x7f24a40cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.231+0000 7f24a8c4e700 1 -- 172.21.15.1:0/2345514494 >> v1:172.21.15.165:6789/0 conn(0x7f24a4160a70 legacy=0x7f24a40d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:32.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.231+0000 7f24a8c4e700 1 -- 172.21.15.1:0/2345514494 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a40cf210 con 0x7f24a41612d0 2024-03-18T19:05:32.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.231+0000 7f24a8c4e700 1 --2- 172.21.15.1:0/2345514494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a41612d0 0x7f24a41616f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f24a40c9d70 tx=0x7f249c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e15453db4952ed8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:32.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.231+0000 7f24a2ffd700 1 -- 172.21.15.1:0/2345514494 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c00e070 con 0x7f24a41612d0 2024-03-18T19:05:32.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.231+0000 7f24a2ffd700 1 -- 172.21.15.1:0/2345514494 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249c00b910 con 0x7f24a41612d0 2024-03-18T19:05:32.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.231+0000 7f24a2ffd700 1 -- 172.21.15.1:0/2345514494 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c00bcc0 con 0x7f24a41612d0 2024-03-18T19:05:32.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.231+0000 7f24aa6b1700 1 -- 172.21.15.1:0/2345514494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a41612d0 msgr2=0x7f24a41616f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:32.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.231+0000 7f24aa6b1700 1 --2- 172.21.15.1:0/2345514494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a41612d0 0x7f24a41616f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f24a40c9d70 tx=0x7f249c00b330 comp rx=0 tx=0).stop 2024-03-18T19:05:32.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.232+0000 7f24aa6b1700 1 -- 172.21.15.1:0/2345514494 shutdown_connections 2024-03-18T19:05:32.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.232+0000 7f24aa6b1700 1 --2- 172.21.15.1:0/2345514494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a41612d0 0x7f24a41616f0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.232+0000 7f24aa6b1700 1 --2- 172.21.15.1:0/2345514494 >> v2:172.21.15.165:3300/0 conn(0x7f24a4160e50 0x7f24a40cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.232+0000 7f24aa6b1700 1 -- 172.21.15.1:0/2345514494 >> 172.21.15.1:0/2345514494 conn(0x7f24a409c560 msgr2=0x7f24a40d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:32.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.232+0000 7f24aa6b1700 1 -- 172.21.15.1:0/2345514494 shutdown_connections 2024-03-18T19:05:32.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.232+0000 7f24aa6b1700 1 -- 172.21.15.1:0/2345514494 wait complete. 2024-03-18T19:05:32.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.233+0000 7f24aa6b1700 1 Processor -- start 2024-03-18T19:05:32.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.233+0000 7f24aa6b1700 1 -- start start 2024-03-18T19:05:32.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.234+0000 7f24aa6b1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24a4160a70 0x7f24a4151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:32.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.234+0000 7f24aa6b1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24a4160e50 0x7f24a40b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:32.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.234+0000 7f24aa6b1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a41612d0 0x7f24a40cb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:32.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.234+0000 7f24aa6b1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a41f4300 con 0x7f24a41612d0 2024-03-18T19:05:32.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.234+0000 7f24aa6b1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a41f3780 con 0x7f24a4160a70 2024-03-18T19:05:32.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.234+0000 7f24aa6b1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a41f3a60 con 0x7f24a4160e50 2024-03-18T19:05:32.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.234+0000 7f24a8c4e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a41612d0 0x7f24a40cb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:32.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.234+0000 7f24a3fff700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24a4160a70 0x7f24a4151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:32.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.234+0000 7f24a8c4e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a41612d0 0x7f24a40cb5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33838/0 (socket says 172.21.15.1:33838) 2024-03-18T19:05:32.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.235+0000 7f24a8c4e700 1 -- 172.21.15.1:0/4189546442 learned_addr learned my addr 172.21.15.1:0/4189546442 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:32.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.235+0000 7f24a3fff700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24a4160a70 0x7f24a4151860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:42384/0 (socket says 172.21.15.1:42384) 2024-03-18T19:05:32.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.235+0000 7f24a37fe700 1 --2- 172.21.15.1:0/4189546442 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24a4160e50 0x7f24a40b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:32.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.235+0000 7f24a8c4e700 1 -- 172.21.15.1:0/4189546442 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24a4160e50 msgr2=0x7f24a40b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:32.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.235+0000 7f24a8c4e700 1 --2- 172.21.15.1:0/4189546442 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24a4160e50 0x7f24a40b74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.235+0000 7f24a8c4e700 1 -- 172.21.15.1:0/4189546442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24a4160a70 msgr2=0x7f24a4151860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:32.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.235+0000 7f24a8c4e700 1 --2- 172.21.15.1:0/4189546442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24a4160a70 0x7f24a4151860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.235+0000 7f24a8c4e700 1 -- 172.21.15.1:0/4189546442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249c009b30 con 0x7f24a41612d0 2024-03-18T19:05:32.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.235+0000 7f24a3fff700 1 --2- 172.21.15.1:0/4189546442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24a4160a70 0x7f24a4151860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:32.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.235+0000 7f24a8c4e700 1 --2- 172.21.15.1:0/4189546442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a41612d0 0x7f24a40cb5b0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f24a406afe0 tx=0x7f249c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:32.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.235+0000 7f24a0ff9700 1 -- 172.21.15.1:0/4189546442 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c004840 con 0x7f24a41612d0 2024-03-18T19:05:32.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.235+0000 7f24a0ff9700 1 -- 172.21.15.1:0/4189546442 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249c0099b0 con 0x7f24a41612d0 2024-03-18T19:05:32.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.235+0000 7f24aa6b1700 1 -- 172.21.15.1:0/4189546442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a41f3d40 con 0x7f24a41612d0 2024-03-18T19:05:32.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.235+0000 7f24a0ff9700 1 -- 172.21.15.1:0/4189546442 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c01c680 con 0x7f24a41612d0 2024-03-18T19:05:32.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.235+0000 7f24aa6b1700 1 -- 172.21.15.1:0/4189546442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a40b7a30 con 0x7f24a41612d0 2024-03-18T19:05:32.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.236+0000 7f24aa6b1700 1 -- 172.21.15.1:0/4189546442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a4067390 con 0x7f24a41612d0 2024-03-18T19:05:32.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.238+0000 7f24a0ff9700 1 -- 172.21.15.1:0/4189546442 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 31) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f249c01c7e0 con 0x7f24a41612d0 2024-03-18T19:05:32.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.238+0000 7f24a0ff9700 1 --2- 172.21.15.1:0/4189546442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f249006ae00 0x7f249006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:32.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.238+0000 7f24a3fff700 1 -- 172.21.15.1:0/4189546442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f249006ae00 msgr2=0x7f249006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:32.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.238+0000 7f24a0ff9700 1 -- 172.21.15.1:0/4189546442 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f249c08b7b0 con 0x7f24a41612d0 2024-03-18T19:05:32.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.239+0000 7f24a3fff700 1 --2- 172.21.15.1:0/4189546442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f249006ae00 0x7f249006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:32.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.246+0000 7f24a0ff9700 1 -- 172.21.15.1:0/4189546442 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f249c056620 con 0x7f24a41612d0 2024-03-18T19:05:32.354 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:32.358+0000 7f77842e3240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:05:32.395 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:32.399+0000 7f23cf2df240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:05:32.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.440+0000 7f24a3fff700 1 -- 172.21.15.1:0/4189546442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f249006ae00 msgr2=0x7f249006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.440+0000 7f24a3fff700 1 --2- 172.21.15.1:0/4189546442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f249006ae00 0x7f249006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:32.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.498+0000 7f24aa6b1700 1 -- 172.21.15.1:0/4189546442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f24a40676a0 con 0x7f24a41612d0 2024-03-18T19:05:32.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.501+0000 7f24a0ff9700 1 -- 172.21.15.1:0/4189546442 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7f249c059c40 con 0x7f24a41612d0 2024-03-18T19:05:32.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.504+0000 7f24aa6b1700 1 -- 172.21.15.1:0/4189546442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f249006ae00 msgr2=0x7f249006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:32.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.504+0000 7f24aa6b1700 1 --2- 172.21.15.1:0/4189546442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f249006ae00 0x7f249006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.504+0000 7f24aa6b1700 1 -- 172.21.15.1:0/4189546442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a41612d0 msgr2=0x7f24a40cb5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:32.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.504+0000 7f24aa6b1700 1 --2- 172.21.15.1:0/4189546442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a41612d0 0x7f24a40cb5b0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f24a406afe0 tx=0x7f249c004ad0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.504+0000 7f24aa6b1700 1 -- 172.21.15.1:0/4189546442 shutdown_connections 2024-03-18T19:05:32.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.504+0000 7f24aa6b1700 1 --2- 172.21.15.1:0/4189546442 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f249006ae00 0x7f249006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.504+0000 7f24aa6b1700 1 --2- 172.21.15.1:0/4189546442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a41612d0 0x7f24a40cb5b0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.504+0000 7f24aa6b1700 1 --2- 172.21.15.1:0/4189546442 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24a4160a70 0x7f24a4151860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.504+0000 7f24aa6b1700 1 --2- 172.21.15.1:0/4189546442 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24a4160e50 0x7f24a40b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.504+0000 7f24aa6b1700 1 -- 172.21.15.1:0/4189546442 >> 172.21.15.1:0/4189546442 conn(0x7f24a409c560 msgr2=0x7f24a40cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:32.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.504+0000 7f24aa6b1700 1 -- 172.21.15.1:0/4189546442 shutdown_connections 2024-03-18T19:05:32.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.504+0000 7f24aa6b1700 1 -- 172.21.15.1:0/4189546442 wait complete. 2024-03-18T19:05:32.512 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-18T19:05:32.618 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:32.622+0000 7f77842e3240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:05:32.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.647+0000 7f4d29e99700 1 Processor -- start 2024-03-18T19:05:32.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.647+0000 7f4d29e99700 1 -- start start 2024-03-18T19:05:32.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.647+0000 7f4d29e99700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d2406eda0 0x7f4d24161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:32.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.647+0000 7f4d29e99700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4d2406f180 0x7f4d240ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:32.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.647+0000 7f4d29e99700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4d2406a750 0x7f4d2406ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:32.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.647+0000 7f4d29e99700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d240cb320 con 0x7f4d2406f180 2024-03-18T19:05:32.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.647+0000 7f4d29e99700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d240ce010 con 0x7f4d2406a750 2024-03-18T19:05:32.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.647+0000 7f4d29e99700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2406b0a0 con 0x7f4d2406eda0 2024-03-18T19:05:32.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.648+0000 7f4d22ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4d2406f180 0x7f4d240ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:32.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.648+0000 7f4d22ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4d2406f180 0x7f4d240ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33844/0 (socket says 172.21.15.1:33844) 2024-03-18T19:05:32.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.648+0000 7f4d22ffd700 1 -- 172.21.15.1:0/3517866768 learned_addr learned my addr 172.21.15.1:0/3517866768 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:32.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.648+0000 7f4d237fe700 1 --2- 172.21.15.1:0/3517866768 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d2406eda0 0x7f4d24161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:32.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.648+0000 7f4d23fff700 1 --2- 172.21.15.1:0/3517866768 >> v2:172.21.15.165:3300/0 conn(0x7f4d2406a750 0x7f4d2406ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:32.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.648+0000 7f4d22ffd700 1 -- 172.21.15.1:0/3517866768 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d2406eda0 msgr2=0x7f4d24161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:32.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.648+0000 7f4d22ffd700 1 --2- 172.21.15.1:0/3517866768 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d2406eda0 0x7f4d24161c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.648+0000 7f4d22ffd700 1 -- 172.21.15.1:0/3517866768 >> v2:172.21.15.165:3300/0 conn(0x7f4d2406a750 msgr2=0x7f4d2406ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:32.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.648+0000 7f4d22ffd700 1 --2- 172.21.15.1:0/3517866768 >> v2:172.21.15.165:3300/0 conn(0x7f4d2406a750 0x7f4d2406ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.648+0000 7f4d22ffd700 1 -- 172.21.15.1:0/3517866768 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d240caac0 con 0x7f4d2406f180 2024-03-18T19:05:32.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.648+0000 7f4d22ffd700 1 --2- 172.21.15.1:0/3517866768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d2406f180 0x7f4d240ca580 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f4d240c3a10 tx=0x7f4d18009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee00d52eb9583135 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:32.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.648+0000 7f4d227fc700 1 -- 172.21.15.1:0/3517866768 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d18011830 con 0x7f4d2406f180 2024-03-18T19:05:32.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.648+0000 7f4d227fc700 1 -- 172.21.15.1:0/3517866768 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d18011990 con 0x7f4d2406f180 2024-03-18T19:05:32.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.648+0000 7f4d227fc700 1 -- 172.21.15.1:0/3517866768 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d18011d20 con 0x7f4d2406f180 2024-03-18T19:05:32.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.649+0000 7f4d29e99700 1 -- 172.21.15.1:0/3517866768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d2406f180 msgr2=0x7f4d240ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:32.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.649+0000 7f4d29e99700 1 --2- 172.21.15.1:0/3517866768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d2406f180 0x7f4d240ca580 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f4d240c3a10 tx=0x7f4d18009bd0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.649+0000 7f4d29e99700 1 -- 172.21.15.1:0/3517866768 shutdown_connections 2024-03-18T19:05:32.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.649+0000 7f4d29e99700 1 --2- 172.21.15.1:0/3517866768 >> v2:172.21.15.165:3300/0 conn(0x7f4d2406a750 0x7f4d2406ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.649+0000 7f4d29e99700 1 --2- 172.21.15.1:0/3517866768 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d2406eda0 0x7f4d24161c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.649+0000 7f4d29e99700 1 --2- 172.21.15.1:0/3517866768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d2406f180 0x7f4d240ca580 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.649+0000 7f4d29e99700 1 -- 172.21.15.1:0/3517866768 >> 172.21.15.1:0/3517866768 conn(0x7f4d240cff30 msgr2=0x7f4d240d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:32.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.649+0000 7f4d29e99700 1 -- 172.21.15.1:0/3517866768 shutdown_connections 2024-03-18T19:05:32.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.649+0000 7f4d29e99700 1 -- 172.21.15.1:0/3517866768 wait complete. 2024-03-18T19:05:32.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.650+0000 7f4d29e99700 1 Processor -- start 2024-03-18T19:05:32.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.651+0000 7f4d29e99700 1 -- start start 2024-03-18T19:05:32.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.651+0000 7f4d29e99700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d2406a750 0x7f4d2415fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:32.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.651+0000 7f4d29e99700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d2406eda0 0x7f4d241f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:32.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.651+0000 7f4d29e99700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d2406f180 0x7f4d241e8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:32.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.651+0000 7f4d29e99700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d240c45a0 con 0x7f4d2406eda0 2024-03-18T19:05:32.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.651+0000 7f4d29e99700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d240c6300 con 0x7f4d2406f180 2024-03-18T19:05:32.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.651+0000 7f4d29e99700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d241e7cc0 con 0x7f4d2406a750 2024-03-18T19:05:32.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.651+0000 7f4d22ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d2406eda0 0x7f4d241f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:32.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.651+0000 7f4d22ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d2406eda0 0x7f4d241f31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33848/0 (socket says 172.21.15.1:33848) 2024-03-18T19:05:32.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.651+0000 7f4d22ffd700 1 -- 172.21.15.1:0/1350762123 learned_addr learned my addr 172.21.15.1:0/1350762123 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:32.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.651+0000 7f4d23fff700 1 --2- 172.21.15.1:0/1350762123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d2406f180 0x7f4d241e8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:32.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.651+0000 7f4d237fe700 1 --2- 172.21.15.1:0/1350762123 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d2406a750 0x7f4d2415fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:32.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.652+0000 7f4d22ffd700 1 -- 172.21.15.1:0/1350762123 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d2406a750 msgr2=0x7f4d2415fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:32.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.652+0000 7f4d22ffd700 1 --2- 172.21.15.1:0/1350762123 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d2406a750 0x7f4d2415fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.652+0000 7f4d22ffd700 1 -- 172.21.15.1:0/1350762123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d2406f180 msgr2=0x7f4d241e8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:32.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.652+0000 7f4d22ffd700 1 --2- 172.21.15.1:0/1350762123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d2406f180 0x7f4d241e8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.652+0000 7f4d22ffd700 1 -- 172.21.15.1:0/1350762123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d18009550 con 0x7f4d2406eda0 2024-03-18T19:05:32.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.652+0000 7f4d22ffd700 1 --2- 172.21.15.1:0/1350762123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d2406eda0 0x7f4d241f31e0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009c10 tx=0x7f4d18004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:32.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.652+0000 7f4d0bfff700 1 -- 172.21.15.1:0/1350762123 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d18004450 con 0x7f4d2406eda0 2024-03-18T19:05:32.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.652+0000 7f4d0bfff700 1 -- 172.21.15.1:0/1350762123 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1801e070 con 0x7f4d2406eda0 2024-03-18T19:05:32.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.652+0000 7f4d29e99700 1 -- 172.21.15.1:0/1350762123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d241e7fa0 con 0x7f4d2406eda0 2024-03-18T19:05:32.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.652+0000 7f4d0bfff700 1 -- 172.21.15.1:0/1350762123 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d180038e0 con 0x7f4d2406eda0 2024-03-18T19:05:32.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.652+0000 7f4d29e99700 1 -- 172.21.15.1:0/1350762123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d240c5f60 con 0x7f4d2406eda0 2024-03-18T19:05:32.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.653+0000 7f4d29e99700 1 -- 172.21.15.1:0/1350762123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d240c4750 con 0x7f4d2406eda0 2024-03-18T19:05:32.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.654+0000 7f4d0bfff700 1 -- 172.21.15.1:0/1350762123 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 31) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f4d18003a40 con 0x7f4d2406eda0 2024-03-18T19:05:32.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.655+0000 7f4d0bfff700 1 --2- 172.21.15.1:0/1350762123 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4d1006ae00 0x7f4d1006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:32.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.655+0000 7f4d237fe700 1 -- 172.21.15.1:0/1350762123 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4d1006ae00 msgr2=0x7f4d1006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:32.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.655+0000 7f4d0bfff700 1 -- 172.21.15.1:0/1350762123 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4d1808a570 con 0x7f4d2406eda0 2024-03-18T19:05:32.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.655+0000 7f4d237fe700 1 --2- 172.21.15.1:0/1350762123 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4d1006ae00 0x7f4d1006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:32.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.659+0000 7f4d0bfff700 1 -- 172.21.15.1:0/1350762123 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4d18055570 con 0x7f4d2406eda0 2024-03-18T19:05:32.677 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:32.682+0000 7f23cf2df240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:05:32.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.856+0000 7f4d237fe700 1 -- 172.21.15.1:0/1350762123 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4d1006ae00 msgr2=0x7f4d1006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:32.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.856+0000 7f4d237fe700 1 --2- 172.21.15.1:0/1350762123 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4d1006ae00 0x7f4d1006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:32.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.865+0000 7f4d29e99700 1 -- 172.21.15.1:0/1350762123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f4d240b0540 con 0x7f4d2406eda0 2024-03-18T19:05:32.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.868+0000 7f4d0bfff700 1 -- 172.21.15.1:0/1350762123 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7f4d18058b90 con 0x7f4d2406eda0 2024-03-18T19:05:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.870+0000 7f4d29e99700 1 -- 172.21.15.1:0/1350762123 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4d1006ae00 msgr2=0x7f4d1006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.870+0000 7f4d29e99700 1 --2- 172.21.15.1:0/1350762123 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4d1006ae00 0x7f4d1006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.870+0000 7f4d29e99700 1 -- 172.21.15.1:0/1350762123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d2406eda0 msgr2=0x7f4d241f31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.870+0000 7f4d29e99700 1 --2- 172.21.15.1:0/1350762123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d2406eda0 0x7f4d241f31e0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009c10 tx=0x7f4d18004750 comp rx=0 tx=0).stop 2024-03-18T19:05:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.870+0000 7f4d29e99700 1 -- 172.21.15.1:0/1350762123 shutdown_connections 2024-03-18T19:05:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.870+0000 7f4d29e99700 1 --2- 172.21.15.1:0/1350762123 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4d1006ae00 0x7f4d1006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.870+0000 7f4d29e99700 1 --2- 172.21.15.1:0/1350762123 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d2406a750 0x7f4d2415fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.870+0000 7f4d29e99700 1 --2- 172.21.15.1:0/1350762123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d2406f180 0x7f4d241e8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.870+0000 7f4d29e99700 1 --2- 172.21.15.1:0/1350762123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d2406eda0 0x7f4d241f31e0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.870+0000 7f4d29e99700 1 -- 172.21.15.1:0/1350762123 >> 172.21.15.1:0/1350762123 conn(0x7f4d240cff30 msgr2=0x7f4d240afe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.870+0000 7f4d29e99700 1 -- 172.21.15.1:0/1350762123 shutdown_connections 2024-03-18T19:05:32.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:32.870+0000 7f4d29e99700 1 -- 172.21.15.1:0/1350762123 wait complete. 2024-03-18T19:05:32.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:32.879 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-03-18T19:05:32.901 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-03-18T19:05:33.041 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:33.044+0000 7f77842e3240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:05:33.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.075+0000 7f1f3220d700 1 Processor -- start 2024-03-18T19:05:33.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.075+0000 7f1f3220d700 1 -- start start 2024-03-18T19:05:33.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.076+0000 7f1f3220d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1f2c160a70 0x7f1f2c0d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:33.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.076+0000 7f1f3220d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1f2c160e50 0x7f1f2c0cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:33.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.076+0000 7f1f3220d700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c162170 con 0x7f1f2c1612d0 2024-03-18T19:05:33.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.076+0000 7f1f3220d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2c0cef30 con 0x7f1f2c160e50 2024-03-18T19:05:33.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.076+0000 7f1f3220d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2c0cf210 con 0x7f1f2c160a70 2024-03-18T19:05:33.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.076+0000 7f1f23fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1f2c160e50 0x7f1f2c0cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:33.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.076+0000 7f1f2b7fe700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1f2c160a70 0x7f1f2c0d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:33.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.076+0000 7f1f2bfff700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f1f2c1612d0 0x7f1f2c1616f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:39444/0 (socket says 172.21.15.1:39444) 2024-03-18T19:05:33.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.076+0000 7f1f2bfff700 1 -- 172.21.15.1:0/2684986945 learned_addr learned my addr 172.21.15.1:0/2684986945 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:33.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.076+0000 7f1f23fff700 1 -- 172.21.15.1:0/2684986945 >> v2:172.21.15.165:3300/0 conn(0x7f1f2c160a70 msgr2=0x7f1f2c0d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:33.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.077+0000 7f1f23fff700 1 --2- 172.21.15.1:0/2684986945 >> v2:172.21.15.165:3300/0 conn(0x7f1f2c160a70 0x7f1f2c0d3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:33.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.077+0000 7f1f23fff700 1 -- 172.21.15.1:0/2684986945 >> v1:172.21.15.165:6789/0 conn(0x7f1f2c1612d0 legacy=0x7f1f2c1616f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:33.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.077+0000 7f1f23fff700 1 -- 172.21.15.1:0/2684986945 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c0cf4f0 con 0x7f1f2c160e50 2024-03-18T19:05:33.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.077+0000 7f1f2b7fe700 1 --2- 172.21.15.1:0/2684986945 >> v2:172.21.15.165:3300/0 conn(0x7f1f2c160a70 0x7f1f2c0d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:33.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.077+0000 7f1f23fff700 1 --2- 172.21.15.1:0/2684986945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f2c160e50 0x7f1f2c0cffd0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c08d8f0 tx=0x7f1f14009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb12878a8ed02a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:33.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.077+0000 7f1f2affd700 1 -- 172.21.15.1:0/2684986945 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f14011830 con 0x7f1f2c160e50 2024-03-18T19:05:33.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.077+0000 7f1f2affd700 1 -- 172.21.15.1:0/2684986945 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f14011990 con 0x7f1f2c160e50 2024-03-18T19:05:33.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.077+0000 7f1f2affd700 1 -- 172.21.15.1:0/2684986945 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f14011d20 con 0x7f1f2c160e50 2024-03-18T19:05:33.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.077+0000 7f1f3220d700 1 -- 172.21.15.1:0/2684986945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f2c160e50 msgr2=0x7f1f2c0cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:33.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.077+0000 7f1f3220d700 1 --2- 172.21.15.1:0/2684986945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f2c160e50 0x7f1f2c0cffd0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c08d8f0 tx=0x7f1f14009b20 comp rx=0 tx=0).stop 2024-03-18T19:05:33.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.078+0000 7f1f3220d700 1 -- 172.21.15.1:0/2684986945 shutdown_connections 2024-03-18T19:05:33.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.078+0000 7f1f3220d700 1 --2- 172.21.15.1:0/2684986945 >> v2:172.21.15.165:3300/0 conn(0x7f1f2c160a70 0x7f1f2c0d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:33.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.078+0000 7f1f3220d700 1 --2- 172.21.15.1:0/2684986945 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f2c160e50 0x7f1f2c0cffd0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:33.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.078+0000 7f1f3220d700 1 -- 172.21.15.1:0/2684986945 >> 172.21.15.1:0/2684986945 conn(0x7f1f2c09c560 msgr2=0x7f1f2c0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:33.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.078+0000 7f1f3220d700 1 -- 172.21.15.1:0/2684986945 shutdown_connections 2024-03-18T19:05:33.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.078+0000 7f1f3220d700 1 -- 172.21.15.1:0/2684986945 wait complete. 2024-03-18T19:05:33.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.079+0000 7f1f3220d700 1 Processor -- start 2024-03-18T19:05:33.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.079+0000 7f1f3220d700 1 -- start start 2024-03-18T19:05:33.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f3220d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f2c160a70 0x7f1f2c157730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:33.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f3220d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f2c160e50 0x7f1f2c0b5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:33.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f3220d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f2c1612d0 0x7f1f2c0c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:33.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f3220d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c0ca320 con 0x7f1f2c160e50 2024-03-18T19:05:33.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f3220d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c0cb180 con 0x7f1f2c1612d0 2024-03-18T19:05:33.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f3220d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c0c9150 con 0x7f1f2c160a70 2024-03-18T19:05:33.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f23fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f2c160e50 0x7f1f2c0b5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:33.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f2b7fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f2c160a70 0x7f1f2c157730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:33.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f23fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f2c160e50 0x7f1f2c0b5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33872/0 (socket says 172.21.15.1:33872) 2024-03-18T19:05:33.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f23fff700 1 -- 172.21.15.1:0/4108677476 learned_addr learned my addr 172.21.15.1:0/4108677476 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:33.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f2bfff700 1 --2- 172.21.15.1:0/4108677476 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f2c1612d0 0x7f1f2c0c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:33.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f23fff700 1 -- 172.21.15.1:0/4108677476 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f2c160a70 msgr2=0x7f1f2c157730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:33.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f23fff700 1 --2- 172.21.15.1:0/4108677476 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f2c160a70 0x7f1f2c157730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:33.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f23fff700 1 -- 172.21.15.1:0/4108677476 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f2c1612d0 msgr2=0x7f1f2c0c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:33.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f23fff700 1 --2- 172.21.15.1:0/4108677476 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f2c1612d0 0x7f1f2c0c84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:33.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f23fff700 1 -- 172.21.15.1:0/4108677476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c0094a0 con 0x7f1f2c160e50 2024-03-18T19:05:33.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f23fff700 1 --2- 172.21.15.1:0/4108677476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f2c160e50 0x7f1f2c0b5f70 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f1f140090b0 tx=0x7f1f14004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:33.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f28ff9700 1 -- 172.21.15.1:0/4108677476 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f140036a0 con 0x7f1f2c160e50 2024-03-18T19:05:33.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f28ff9700 1 -- 172.21.15.1:0/4108677476 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1401f070 con 0x7f1f2c160e50 2024-03-18T19:05:33.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f3220d700 1 -- 172.21.15.1:0/4108677476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f140094a0 con 0x7f1f2c160e50 2024-03-18T19:05:33.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.080+0000 7f1f28ff9700 1 -- 172.21.15.1:0/4108677476 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f14003a60 con 0x7f1f2c160e50 2024-03-18T19:05:33.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.081+0000 7f1f3220d700 1 -- 172.21.15.1:0/4108677476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2c0ca850 con 0x7f1f2c160e50 2024-03-18T19:05:33.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.082+0000 7f1f3220d700 1 -- 172.21.15.1:0/4108677476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f2c067390 con 0x7f1f2c160e50 2024-03-18T19:05:33.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.082+0000 7f1f28ff9700 1 -- 172.21.15.1:0/4108677476 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 31) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f1f14003bc0 con 0x7f1f2c160e50 2024-03-18T19:05:33.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.083+0000 7f1f28ff9700 1 --2- 172.21.15.1:0/4108677476 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1f0c06ae00 0x7f1f0c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:33.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.083+0000 7f1f2b7fe700 1 -- 172.21.15.1:0/4108677476 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1f0c06ae00 msgr2=0x7f1f0c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:33.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.083+0000 7f1f28ff9700 1 -- 172.21.15.1:0/4108677476 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1f14021070 con 0x7f1f2c160e50 2024-03-18T19:05:33.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.083+0000 7f1f2b7fe700 1 --2- 172.21.15.1:0/4108677476 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1f0c06ae00 0x7f1f0c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:33.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.087+0000 7f1f28ff9700 1 -- 172.21.15.1:0/4108677476 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1f14055db0 con 0x7f1f2c160e50 2024-03-18T19:05:33.135 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:33.140+0000 7f23cf2df240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:05:33.248 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:33.252+0000 7f77842e3240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:05:33.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.284+0000 7f1f2b7fe700 1 -- 172.21.15.1:0/4108677476 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1f0c06ae00 msgr2=0x7f1f0c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:33.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.284+0000 7f1f2b7fe700 1 --2- 172.21.15.1:0/4108677476 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1f0c06ae00 0x7f1f0c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:33.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.315+0000 7f1f3220d700 1 -- 172.21.15.1:0/4108677476 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f2c0cfab0 con 0x7f1f0c06ae00 2024-03-18T19:05:33.355 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:33.359+0000 7f23cf2df240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:05:33.617 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:33.621+0000 7f77842e3240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:05:33.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.685+0000 7f1f2b7fe700 1 -- 172.21.15.1:0/4108677476 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1f0c06ae00 msgr2=0x7f1f0c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:33.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:33.685+0000 7f1f2b7fe700 1 --2- 172.21.15.1:0/4108677476 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1f0c06ae00 0x7f1f0c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T19:05:33.744 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:33.748+0000 7f23cf2df240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:05:34.194 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:34.198+0000 7f77842e3240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:05:34.265 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:34.269+0000 7f77842e3240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:05:34.356 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:34.361+0000 7f23cf2df240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:05:34.390 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:34.394+0000 7f77842e3240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:05:34.432 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:34.436+0000 7f23cf2df240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:05:34.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:34.486+0000 7f1f2b7fe700 1 -- 172.21.15.1:0/4108677476 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1f0c06ae00 msgr2=0x7f1f0c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:34.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:34.486+0000 7f1f2b7fe700 1 --2- 172.21.15.1:0/4108677476 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1f0c06ae00 0x7f1f0c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T19:05:34.566 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:34.570+0000 7f23cf2df240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:05:34.567 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:34.571+0000 7f77842e3240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:05:34.638 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:34.642+0000 7f77842e3240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:05:34.716 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:34.719+0000 7f77842e3240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:05:34.751 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:34.755+0000 7f23cf2df240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:05:34.792 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:34.796+0000 7f77842e3240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:05:34.827 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:34.831+0000 7f23cf2df240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:05:34.869 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:34.873+0000 7f77842e3240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:05:34.909 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:34.914+0000 7f23cf2df240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:05:34.991 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:34.995+0000 7f23cf2df240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:05:35.038 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:35.041+0000 7f77842e3240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:05:35.072 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:35.077+0000 7f23cf2df240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:05:35.251 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:35.255+0000 7f23cf2df240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:05:35.251 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:35.254+0000 7f77842e3240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:05:35.399 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:35.403+0000 7f77842e3240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:05:35.470 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:35.474+0000 7f77842e3240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:05:35.472 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:35.477+0000 7f23cf2df240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:05:35.631 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:35.636+0000 7f23cf2df240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:05:35.708 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:35.712+0000 7f23cf2df240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:05:35.910 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:35.914+0000 7f77842e3240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:05:36.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:36.087+0000 7f1f2b7fe700 1 -- 172.21.15.1:0/4108677476 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1f0c06ae00 msgr2=0x7f1f0c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:36.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:36.087+0000 7f1f2b7fe700 1 --2- 172.21.15.1:0/4108677476 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1f0c06ae00 0x7f1f0c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T19:05:36.178 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:36.182+0000 7f23cf2df240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:05:36.245 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:36.249+0000 7f77842e3240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:05:36.405 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:36.408+0000 7f77842e3240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:05:36.529 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:36.533+0000 7f23cf2df240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:05:36.702 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:36.706+0000 7f23cf2df240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:05:36.787 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:36.791+0000 7f77842e3240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:05:36.948 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:36.952+0000 7f77842e3240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:05:37.112 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:37.116+0000 7f23cf2df240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:05:37.287 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:37.291+0000 7f23cf2df240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:05:37.310 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:37.314+0000 7f77842e3240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:05:37.396 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:37.400+0000 7f77842e3240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:05:37.666 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:37.671+0000 7f23cf2df240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:05:37.760 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:37.764+0000 7f23cf2df240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:05:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:37.784+0000 7f1f28ff9700 1 -- 172.21.15.1:0/4108677476 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 32) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f1f140be580 con 0x7f1f2c160e50 2024-03-18T19:05:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:37.784+0000 7f1f28ff9700 1 -- 172.21.15.1:0/4108677476 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1f0c06ae00 msgr2=0x7f1f0c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:37.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:37.784+0000 7f1f28ff9700 1 --2- 172.21.15.1:0/4108677476 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1f0c06ae00 0x7f1f0c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:37.925 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:37.929+0000 7f2341efc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:37.926 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:37.929+0000 7f2341efc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:37.926 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:37.929+0000 7f2341efc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:37.926 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:37.929+0000 7f2341efc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:37.926 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:37.929+0000 7f2341efc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:37.926 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:37.930+0000 7f2344701700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:37.926 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:37.930+0000 7f2344701700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:37.926 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:37.930+0000 7f2344701700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:37.926 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:37.930+0000 7f2344701700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:37.926 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:37.930+0000 7f2344701700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:38.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:38.788+0000 7f1f28ff9700 1 -- 172.21.15.1:0/4108677476 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 33) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f1f14089200 con 0x7f1f2c160e50 2024-03-18T19:05:38.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:38.789+0000 7f1f28ff9700 1 --2- 172.21.15.1:0/4108677476 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1f0c06ae00 0x7f1f0c06d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:38.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:38.789+0000 7f1f28ff9700 1 -- 172.21.15.1:0/4108677476 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f0c035990 con 0x7f1f0c06ae00 2024-03-18T19:05:38.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:38.793+0000 7f1f2b7fe700 1 --2- 172.21.15.1:0/4108677476 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1f0c06ae00 0x7f1f0c06d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:38.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:38.794+0000 7f1f2b7fe700 1 --2- 172.21.15.1:0/4108677476 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1f0c06ae00 0x7f1f0c06d240 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c1f6a80 tx=0x7f1f1c009210 comp rx=0 tx=0).ready entity=mgr.6562 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:38.798 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:38.803+0000 7f234bf10700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-03-18T19:05:38.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:38.803+0000 7f1f28ff9700 1 -- 172.21.15.1:0/4108677476 <== mgr.6562 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f1f0c035990 con 0x7f1f0c06ae00 2024-03-18T19:05:38.801 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-03-18T19:05:38.804 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-18T19:05:38.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:38.805 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-03-18T19:05:38.831 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-18T19:05:39.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.006+0000 7fa5a8ef9700 1 Processor -- start 2024-03-18T19:05:39.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.006+0000 7fa5a8ef9700 1 -- start start 2024-03-18T19:05:39.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.007+0000 7fa5a8ef9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa5a415d5b0 0x7fa5a415d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:39.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.007+0000 7fa5a8ef9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa5a406ca80 0x7fa5a40cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:39.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.007+0000 7fa5a8ef9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5a40b3b00 con 0x7fa5a4099a40 2024-03-18T19:05:39.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.007+0000 7fa5a8ef9700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5a40cbbb0 con 0x7fa5a406ca80 2024-03-18T19:05:39.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.007+0000 7fa5a8ef9700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5a40cf7f0 con 0x7fa5a415d5b0 2024-03-18T19:05:39.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.007+0000 7fa5a1d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa5a4099a40 0x7fa5a4099ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56358/0 (socket says 172.21.15.1:56358) 2024-03-18T19:05:39.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.007+0000 7fa5a1d9b700 1 -- 172.21.15.1:0/3167140275 learned_addr learned my addr 172.21.15.1:0/3167140275 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:39.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.007+0000 7fa5a2d9d700 1 --2- 172.21.15.1:0/3167140275 >> v2:172.21.15.1:3300/0 conn(0x7fa5a406ca80 0x7fa5a40cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:39.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.007+0000 7fa5a259c700 1 --2- 172.21.15.1:0/3167140275 >> v2:172.21.15.165:3300/0 conn(0x7fa5a415d5b0 0x7fa5a415d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:39.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.007+0000 7fa5a2d9d700 1 -- 172.21.15.1:0/3167140275 >> v2:172.21.15.165:3300/0 conn(0x7fa5a415d5b0 msgr2=0x7fa5a415d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:39.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.007+0000 7fa5a2d9d700 1 --2- 172.21.15.1:0/3167140275 >> v2:172.21.15.165:3300/0 conn(0x7fa5a415d5b0 0x7fa5a415d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:39.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.007+0000 7fa5a2d9d700 1 -- 172.21.15.1:0/3167140275 >> v1:172.21.15.1:6789/0 conn(0x7fa5a4099a40 legacy=0x7fa5a4099ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:39.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.007+0000 7fa5a2d9d700 1 -- 172.21.15.1:0/3167140275 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a40d04d0 con 0x7fa5a406ca80 2024-03-18T19:05:39.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.007+0000 7fa5a2d9d700 1 --2- 172.21.15.1:0/3167140275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a406ca80 0x7fa5a40cbfd0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4072080 tx=0x7fa59c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25507092f30fb6d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:39.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.008+0000 7fa5a159a700 1 -- 172.21.15.1:0/3167140275 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c00e070 con 0x7fa5a406ca80 2024-03-18T19:05:39.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.008+0000 7fa5a159a700 1 -- 172.21.15.1:0/3167140275 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa59c00bbd0 con 0x7fa5a406ca80 2024-03-18T19:05:39.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.008+0000 7fa5a159a700 1 -- 172.21.15.1:0/3167140275 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c012430 con 0x7fa5a406ca80 2024-03-18T19:05:39.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.008+0000 7fa5a8ef9700 1 -- 172.21.15.1:0/3167140275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a406ca80 msgr2=0x7fa5a40cbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:39.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.008+0000 7fa5a8ef9700 1 --2- 172.21.15.1:0/3167140275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a406ca80 0x7fa5a40cbfd0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4072080 tx=0x7fa59c00b330 comp rx=0 tx=0).stop 2024-03-18T19:05:39.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.008+0000 7fa5a8ef9700 1 -- 172.21.15.1:0/3167140275 shutdown_connections 2024-03-18T19:05:39.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.008+0000 7fa5a8ef9700 1 --2- 172.21.15.1:0/3167140275 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a406ca80 0x7fa5a40cbfd0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:39.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.008+0000 7fa5a8ef9700 1 --2- 172.21.15.1:0/3167140275 >> v2:172.21.15.165:3300/0 conn(0x7fa5a415d5b0 0x7fa5a415d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:39.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.008+0000 7fa5a8ef9700 1 -- 172.21.15.1:0/3167140275 >> 172.21.15.1:0/3167140275 conn(0x7fa5a40b2bd0 msgr2=0x7fa5a40b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:39.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.008+0000 7fa5a8ef9700 1 -- 172.21.15.1:0/3167140275 shutdown_connections 2024-03-18T19:05:39.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.008+0000 7fa5a8ef9700 1 -- 172.21.15.1:0/3167140275 wait complete. 2024-03-18T19:05:39.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.010+0000 7fa5a8ef9700 1 Processor -- start 2024-03-18T19:05:39.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.010+0000 7fa5a8ef9700 1 -- start start 2024-03-18T19:05:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.010+0000 7fa5a8ef9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5a406ca80 0x7fa5a409b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.010+0000 7fa5a8ef9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5a4099a40 0x7fa5a409d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.010+0000 7fa5a8ef9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a415d5b0 0x7fa5a409dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.010+0000 7fa5a8ef9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a409e0e0 con 0x7fa5a415d5b0 2024-03-18T19:05:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.010+0000 7fa5a8ef9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a40b4c20 con 0x7fa5a4099a40 2024-03-18T19:05:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.010+0000 7fa5a8ef9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a409e2e0 con 0x7fa5a406ca80 2024-03-18T19:05:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.010+0000 7fa5a2d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a415d5b0 0x7fa5a409dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.010+0000 7fa5a2d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a415d5b0 0x7fa5a409dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41304/0 (socket says 172.21.15.1:41304) 2024-03-18T19:05:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.010+0000 7fa5a2d9d700 1 -- 172.21.15.1:0/3471584693 learned_addr learned my addr 172.21.15.1:0/3471584693 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.011+0000 7fa5a1d9b700 1 --2- 172.21.15.1:0/3471584693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5a4099a40 0x7fa5a409d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:39.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.011+0000 7fa5a259c700 1 --2- 172.21.15.1:0/3471584693 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5a406ca80 0x7fa5a409b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:39.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.011+0000 7fa5a2d9d700 1 -- 172.21.15.1:0/3471584693 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5a406ca80 msgr2=0x7fa5a409b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:39.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.011+0000 7fa5a2d9d700 1 --2- 172.21.15.1:0/3471584693 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5a406ca80 0x7fa5a409b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:39.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.011+0000 7fa5a2d9d700 1 -- 172.21.15.1:0/3471584693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5a4099a40 msgr2=0x7fa5a409d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:39.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.011+0000 7fa5a2d9d700 1 --2- 172.21.15.1:0/3471584693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5a4099a40 0x7fa5a409d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:39.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.011+0000 7fa5a2d9d700 1 -- 172.21.15.1:0/3471584693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa59c009a70 con 0x7fa5a415d5b0 2024-03-18T19:05:39.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.011+0000 7fa5a1d9b700 1 --2- 172.21.15.1:0/3471584693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5a4099a40 0x7fa5a409d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:39.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.011+0000 7fa5a2d9d700 1 --2- 172.21.15.1:0/3471584693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a415d5b0 0x7fa5a409dba0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fa59c014040 tx=0x7fa59c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:39.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.011+0000 7fa592ffd700 1 -- 172.21.15.1:0/3471584693 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c012430 con 0x7fa5a415d5b0 2024-03-18T19:05:39.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.011+0000 7fa592ffd700 1 -- 172.21.15.1:0/3471584693 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa59c020070 con 0x7fa5a415d5b0 2024-03-18T19:05:39.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.011+0000 7fa592ffd700 1 -- 172.21.15.1:0/3471584693 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c004280 con 0x7fa5a415d5b0 2024-03-18T19:05:39.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.011+0000 7fa5a8ef9700 1 -- 172.21.15.1:0/3471584693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a409e5c0 con 0x7fa5a415d5b0 2024-03-18T19:05:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.011+0000 7fa5a8ef9700 1 -- 172.21.15.1:0/3471584693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a409eb80 con 0x7fa5a415d5b0 2024-03-18T19:05:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.012+0000 7fa5a8ef9700 1 -- 172.21.15.1:0/3471584693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5a409ed10 con 0x7fa5a415d5b0 2024-03-18T19:05:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.013+0000 7fa592ffd700 1 -- 172.21.15.1:0/3471584693 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 33) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fa59c0043e0 con 0x7fa5a415d5b0 2024-03-18T19:05:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.013+0000 7fa592ffd700 1 --2- 172.21.15.1:0/3471584693 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa58406ab30 0x7fa58406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.013+0000 7fa592ffd700 1 -- 172.21.15.1:0/3471584693 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fa59c022070 con 0x7fa5a415d5b0 2024-03-18T19:05:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.013+0000 7fa5a259c700 1 --2- 172.21.15.1:0/3471584693 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa58406ab30 0x7fa58406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:39.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.014+0000 7fa5a259c700 1 --2- 172.21.15.1:0/3471584693 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa58406ab30 0x7fa58406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa594009930 tx=0x7fa594008040 comp rx=0 tx=0).ready entity=mgr.6562 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:39.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.018+0000 7fa592ffd700 1 -- 172.21.15.1:0/3471584693 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa59c056280 con 0x7fa5a415d5b0 2024-03-18T19:05:39.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.278+0000 7fa5a8ef9700 1 -- 172.21.15.1:0/3471584693 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fa5a40b4d90 con 0x7fa5a415d5b0 2024-03-18T19:05:39.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.794+0000 7fa592ffd700 1 -- 172.21.15.1:0/3471584693 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 34) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fa59c056650 con 0x7fa5a415d5b0 2024-03-18T19:05:39.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.796+0000 7fa592ffd700 1 -- 172.21.15.1:0/3471584693 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v34) v1 ==== 93+0+0 (secure 0 0 0) 0x7fa59c06eed0 con 0x7fa5a415d5b0 2024-03-18T19:05:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.799+0000 7fa5a8ef9700 1 -- 172.21.15.1:0/3471584693 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa58406ab30 msgr2=0x7fa58406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.799+0000 7fa5a8ef9700 1 --2- 172.21.15.1:0/3471584693 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa58406ab30 0x7fa58406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa594009930 tx=0x7fa594008040 comp rx=0 tx=0).stop 2024-03-18T19:05:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.799+0000 7fa5a8ef9700 1 -- 172.21.15.1:0/3471584693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a415d5b0 msgr2=0x7fa5a409dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.799+0000 7fa5a8ef9700 1 --2- 172.21.15.1:0/3471584693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a415d5b0 0x7fa5a409dba0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fa59c014040 tx=0x7fa59c004ad0 comp rx=0 tx=0).stop 2024-03-18T19:05:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.799+0000 7fa5a8ef9700 1 -- 172.21.15.1:0/3471584693 shutdown_connections 2024-03-18T19:05:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.799+0000 7fa5a8ef9700 1 --2- 172.21.15.1:0/3471584693 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa58406ab30 0x7fa58406cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.799+0000 7fa5a8ef9700 1 --2- 172.21.15.1:0/3471584693 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5a406ca80 0x7fa5a409b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.799+0000 7fa5a8ef9700 1 --2- 172.21.15.1:0/3471584693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5a415d5b0 0x7fa5a409dba0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:39.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.799+0000 7fa5a8ef9700 1 --2- 172.21.15.1:0/3471584693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5a4099a40 0x7fa5a409d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:39.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.799+0000 7fa5a8ef9700 1 -- 172.21.15.1:0/3471584693 >> 172.21.15.1:0/3471584693 conn(0x7fa5a40b2bd0 msgr2=0x7fa5a40ccd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:39.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.799+0000 7fa5a8ef9700 1 -- 172.21.15.1:0/3471584693 shutdown_connections 2024-03-18T19:05:39.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:39.800+0000 7fa5a8ef9700 1 -- 172.21.15.1:0/3471584693 wait complete. 2024-03-18T19:05:39.812 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T19:05:39.812 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:05:39.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:05:39.898 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T19:05:39.899 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:05:39.899 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:05:39.919 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:05:39.936 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:39.940+0000 7f16d3057240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:05:39.944 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:39.948+0000 7fddc73eb240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:05:40.081 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:40.085+0000 7f16d3057240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:05:40.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.112+0000 7fdf32b58700 1 Processor -- start 2024-03-18T19:05:40.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.113+0000 7fdf32b58700 1 -- start start 2024-03-18T19:05:40.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.114+0000 7fdf32b58700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf2c169f70 0x7fdf2c0704d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:40.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.115+0000 7fdf32b58700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2c16e9e0 con 0x7fdf2c0725a0 2024-03-18T19:05:40.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.115+0000 7fdf32b58700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2c0b20a0 con 0x7fdf2c06d420 2024-03-18T19:05:40.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.115+0000 7fdf32b58700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c06d8e0 con 0x7fdf2c169f70 2024-03-18T19:05:40.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.115+0000 7fdf31b56700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf2c169f70 0x7fdf2c0704d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:40.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.115+0000 7fdf31b56700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf2c169f70 0x7fdf2c0704d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:60110/0 (socket says 172.21.15.1:60110) 2024-03-18T19:05:40.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.115+0000 7fdf31b56700 1 -- 172.21.15.1:0/2058587471 learned_addr learned my addr 172.21.15.1:0/2058587471 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:40.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.116+0000 7fdf31b56700 1 -- 172.21.15.1:0/2058587471 >> v1:172.21.15.165:6789/0 conn(0x7fdf2c06d420 legacy=0x7fdf2c16e2d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T19:05:40.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.116+0000 7fdf31b56700 1 -- 172.21.15.1:0/2058587471 >> v1:172.21.15.1:6789/0 conn(0x7fdf2c0725a0 legacy=0x7fdf2c16dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:40.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.116+0000 7fdf31b56700 1 -- 172.21.15.1:0/2058587471 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c06dbc0 con 0x7fdf2c169f70 2024-03-18T19:05:40.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.116+0000 7fdf31b56700 1 --2- 172.21.15.1:0/2058587471 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf2c169f70 0x7fdf2c0704d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdf24009f10 tx=0x7fdf2400ccd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c429c5461411ea0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:40.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.116+0000 7fdf30b54700 1 -- 172.21.15.1:0/2058587471 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf24003a20 con 0x7fdf2c169f70 2024-03-18T19:05:40.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.116+0000 7fdf30b54700 1 -- 172.21.15.1:0/2058587471 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf24003b80 con 0x7fdf2c169f70 2024-03-18T19:05:40.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.116+0000 7fdf30b54700 1 -- 172.21.15.1:0/2058587471 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf240090d0 con 0x7fdf2c169f70 2024-03-18T19:05:40.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.117+0000 7fdf32b58700 1 -- 172.21.15.1:0/2058587471 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf2c169f70 msgr2=0x7fdf2c0704d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:40.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.117+0000 7fdf32b58700 1 --2- 172.21.15.1:0/2058587471 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf2c169f70 0x7fdf2c0704d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdf24009f10 tx=0x7fdf2400ccd0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.115 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:40.117+0000 7fddc73eb240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:05:40.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.117+0000 7fdf32b58700 1 -- 172.21.15.1:0/2058587471 shutdown_connections 2024-03-18T19:05:40.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.117+0000 7fdf32b58700 1 --2- 172.21.15.1:0/2058587471 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf2c169f70 0x7fdf2c0704d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.117+0000 7fdf32b58700 1 -- 172.21.15.1:0/2058587471 >> 172.21.15.1:0/2058587471 conn(0x7fdf2c0ceda0 msgr2=0x7fdf2c0cf1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:40.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.118+0000 7fdf32b58700 1 -- 172.21.15.1:0/2058587471 shutdown_connections 2024-03-18T19:05:40.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.118+0000 7fdf32b58700 1 -- 172.21.15.1:0/2058587471 wait complete. 2024-03-18T19:05:40.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.119+0000 7fdf32b58700 1 Processor -- start 2024-03-18T19:05:40.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.119+0000 7fdf32b58700 1 -- start start 2024-03-18T19:05:40.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.120+0000 7fdf32b58700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf2c06d420 0x7fdf2c16c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:40.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.120+0000 7fdf32b58700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf2c0725a0 0x7fdf2c09be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:40.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.120+0000 7fdf32b58700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf2c169f70 0x7fdf2c167410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:40.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.120+0000 7fdf32b58700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c16d8a0 con 0x7fdf2c169f70 2024-03-18T19:05:40.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.120+0000 7fdf32b58700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c16cc20 con 0x7fdf2c06d420 2024-03-18T19:05:40.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.120+0000 7fdf32b58700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c16cf00 con 0x7fdf2c0725a0 2024-03-18T19:05:40.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.120+0000 7fdf31b56700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf2c06d420 0x7fdf2c16c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:40.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.120+0000 7fdf31b56700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf2c06d420 0x7fdf2c16c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:38824/0 (socket says 172.21.15.1:38824) 2024-03-18T19:05:40.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.120+0000 7fdf31b56700 1 -- 172.21.15.1:0/2152586854 learned_addr learned my addr 172.21.15.1:0/2152586854 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:40.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.120+0000 7fdf32357700 1 --2- 172.21.15.1:0/2152586854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf2c169f70 0x7fdf2c167410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:40.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.120+0000 7fdf31355700 1 --2- 172.21.15.1:0/2152586854 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf2c0725a0 0x7fdf2c09be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:40.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.120+0000 7fdf31b56700 1 -- 172.21.15.1:0/2152586854 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf2c0725a0 msgr2=0x7fdf2c09be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:40.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.120+0000 7fdf31b56700 1 --2- 172.21.15.1:0/2152586854 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf2c0725a0 0x7fdf2c09be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.120+0000 7fdf31b56700 1 -- 172.21.15.1:0/2152586854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf2c169f70 msgr2=0x7fdf2c167410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:40.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.120+0000 7fdf31b56700 1 --2- 172.21.15.1:0/2152586854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf2c169f70 0x7fdf2c167410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.120+0000 7fdf31b56700 1 -- 172.21.15.1:0/2152586854 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf24009b40 con 0x7fdf2c06d420 2024-03-18T19:05:40.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.120+0000 7fdf31b56700 1 --2- 172.21.15.1:0/2152586854 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf2c06d420 0x7fdf2c16c320 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c068260 tx=0x7fdf24004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:40.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.121+0000 7fdf227fc700 1 -- 172.21.15.1:0/2152586854 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf24003a20 con 0x7fdf2c06d420 2024-03-18T19:05:40.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.121+0000 7fdf227fc700 1 -- 172.21.15.1:0/2152586854 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf24009720 con 0x7fdf2c06d420 2024-03-18T19:05:40.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.121+0000 7fdf227fc700 1 -- 172.21.15.1:0/2152586854 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf24008e10 con 0x7fdf2c06d420 2024-03-18T19:05:40.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.121+0000 7fdf32b58700 1 -- 172.21.15.1:0/2152586854 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf2c16d1e0 con 0x7fdf2c06d420 2024-03-18T19:05:40.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.121+0000 7fdf32b58700 1 -- 172.21.15.1:0/2152586854 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf2c16c860 con 0x7fdf2c06d420 2024-03-18T19:05:40.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.122+0000 7fdf32b58700 1 -- 172.21.15.1:0/2152586854 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf2c0747a0 con 0x7fdf2c06d420 2024-03-18T19:05:40.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.122+0000 7fdf227fc700 1 -- 172.21.15.1:0/2152586854 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 34) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fdf2400a780 con 0x7fdf2c06d420 2024-03-18T19:05:40.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.123+0000 7fdf227fc700 1 --2- 172.21.15.1:0/2152586854 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf1006aae0 0x7fdf1006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:40.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.123+0000 7fdf31355700 1 -- 172.21.15.1:0/2152586854 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf1006aae0 msgr2=0x7fdf1006cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:40.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.123+0000 7fdf227fc700 1 -- 172.21.15.1:0/2152586854 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fdf2408a710 con 0x7fdf2c06d420 2024-03-18T19:05:40.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.123+0000 7fdf31355700 1 --2- 172.21.15.1:0/2152586854 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf1006aae0 0x7fdf1006cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:40.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.127+0000 7fdf227fc700 1 -- 172.21.15.1:0/2152586854 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdf240558c0 con 0x7fdf2c06d420 2024-03-18T19:05:40.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.324+0000 7fdf31355700 1 -- 172.21.15.1:0/2152586854 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf1006aae0 msgr2=0x7fdf1006cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:40.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.324+0000 7fdf31355700 1 --2- 172.21.15.1:0/2152586854 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf1006aae0 0x7fdf1006cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:40.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.332+0000 7fdf32b58700 1 -- 172.21.15.1:0/2152586854 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf2c0b3050 con 0x7fdf2c06d420 2024-03-18T19:05:40.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.333+0000 7fdf227fc700 1 -- 172.21.15.1:0/2152586854 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+16681 (secure 0 0 0) 0x7fdf24058ee0 con 0x7fdf2c06d420 2024-03-18T19:05:40.330 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:40.330 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":90,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:05:37.779828+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:05:20.423723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-18T19:05:21.600285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2546947071":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/3332271581":"2024-03-19T19:05:37.779786+0000","172.21.15.1:6833/76998":"2024-03-19T19:05:37.779786+0000","172.21.15.1:6832/76998":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/1890712448":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/300431708":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/4106216506":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/569963265":"2024-03-19T19:05:37.779786+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:05:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.335+0000 7fdf32b58700 1 -- 172.21.15.1:0/2152586854 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf1006aae0 msgr2=0x7fdf1006cfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.335+0000 7fdf32b58700 1 --2- 172.21.15.1:0/2152586854 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf1006aae0 0x7fdf1006cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.335+0000 7fdf32b58700 1 -- 172.21.15.1:0/2152586854 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf2c06d420 msgr2=0x7fdf2c16c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.335+0000 7fdf32b58700 1 --2- 172.21.15.1:0/2152586854 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf2c06d420 0x7fdf2c16c320 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c068260 tx=0x7fdf24004ad0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.335+0000 7fdf32b58700 1 -- 172.21.15.1:0/2152586854 shutdown_connections 2024-03-18T19:05:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.335+0000 7fdf32b58700 1 --2- 172.21.15.1:0/2152586854 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdf1006aae0 0x7fdf1006cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.335+0000 7fdf32b58700 1 --2- 172.21.15.1:0/2152586854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf2c169f70 0x7fdf2c167410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.335+0000 7fdf32b58700 1 --2- 172.21.15.1:0/2152586854 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf2c06d420 0x7fdf2c16c320 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.336+0000 7fdf32b58700 1 --2- 172.21.15.1:0/2152586854 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf2c0725a0 0x7fdf2c09be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.336+0000 7fdf32b58700 1 -- 172.21.15.1:0/2152586854 >> 172.21.15.1:0/2152586854 conn(0x7fdf2c0ceda0 msgr2=0x7fdf2c0b2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.336+0000 7fdf32b58700 1 -- 172.21.15.1:0/2152586854 shutdown_connections 2024-03-18T19:05:40.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.336+0000 7fdf32b58700 1 -- 172.21.15.1:0/2152586854 wait complete. 2024-03-18T19:05:40.345 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T19:05:20.423723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T19:05:21.600285+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:05:40.346 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:05:40.346 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:40.349+0000 7f16d3057240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:05:40.396 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:40.399+0000 7fddc73eb240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:05:40.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.482+0000 7fc7b9bf4700 1 Processor -- start 2024-03-18T19:05:40.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.482+0000 7fc7b9bf4700 1 -- start start 2024-03-18T19:05:40.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.483+0000 7fc7b9bf4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7b415d4b0 0x7fc7b40ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:40.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.483+0000 7fc7b9bf4700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b4161020 con 0x7fc7b40acdf0 2024-03-18T19:05:40.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.483+0000 7fc7b9bf4700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b406d770 con 0x7fc7b406e920 2024-03-18T19:05:40.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.483+0000 7fc7b9bf4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b415d890 con 0x7fc7b415d4b0 2024-03-18T19:05:40.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.483+0000 7fc7b3fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc7b40acdf0 0x7fc7b41614b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56370/0 (socket says 172.21.15.1:56370) 2024-03-18T19:05:40.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.484+0000 7fc7b3fff700 1 -- 172.21.15.1:0/2270540596 learned_addr learned my addr 172.21.15.1:0/2270540596 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:40.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.484+0000 7fc7b37fe700 1 --2- 172.21.15.1:0/2270540596 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7b415d4b0 0x7fc7b40ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:40.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.484+0000 7fc7b27fc700 1 -- 172.21.15.1:0/2270540596 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916424156 0 0) 0x7fc7b4161020 con 0x7fc7b40acdf0 2024-03-18T19:05:40.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.484+0000 7fc7b27fc700 1 -- 172.21.15.1:0/2270540596 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc79c0035b0 con 0x7fc7b40acdf0 2024-03-18T19:05:40.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.484+0000 7fc7b27fc700 1 -- 172.21.15.1:0/2270540596 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814610362 0 0) 0x7fc7b406d770 con 0x7fc7b406e920 2024-03-18T19:05:40.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.484+0000 7fc7b27fc700 1 -- 172.21.15.1:0/2270540596 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7b4161020 con 0x7fc7b406e920 2024-03-18T19:05:40.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.484+0000 7fc7b37fe700 1 -- 172.21.15.1:0/2270540596 >> v1:172.21.15.165:6789/0 conn(0x7fc7b406e920 legacy=0x7fc7b40cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:40.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.484+0000 7fc7b37fe700 1 -- 172.21.15.1:0/2270540596 >> v1:172.21.15.1:6789/0 conn(0x7fc7b40acdf0 legacy=0x7fc7b41614b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:40.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.484+0000 7fc7b37fe700 1 -- 172.21.15.1:0/2270540596 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b406da50 con 0x7fc7b415d4b0 2024-03-18T19:05:40.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.484+0000 7fc7b37fe700 1 --2- 172.21.15.1:0/2270540596 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7b415d4b0 0x7fc7b40ac8b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4009870 tx=0x7fc7a4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5986f99b72c01be1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:40.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.484+0000 7fc7b27fc700 1 -- 172.21.15.1:0/2270540596 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a4011830 con 0x7fc7b415d4b0 2024-03-18T19:05:40.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.484+0000 7fc7b27fc700 1 -- 172.21.15.1:0/2270540596 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a4011990 con 0x7fc7b415d4b0 2024-03-18T19:05:40.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.484+0000 7fc7b27fc700 1 -- 172.21.15.1:0/2270540596 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a4011d00 con 0x7fc7b415d4b0 2024-03-18T19:05:40.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.485+0000 7fc7b9bf4700 1 -- 172.21.15.1:0/2270540596 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7b415d4b0 msgr2=0x7fc7b40ac8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:40.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.485+0000 7fc7b9bf4700 1 --2- 172.21.15.1:0/2270540596 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7b415d4b0 0x7fc7b40ac8b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4009870 tx=0x7fc7a4009b80 comp rx=0 tx=0).stop 2024-03-18T19:05:40.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.485+0000 7fc7b9bf4700 1 -- 172.21.15.1:0/2270540596 shutdown_connections 2024-03-18T19:05:40.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.485+0000 7fc7b9bf4700 1 --2- 172.21.15.1:0/2270540596 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7b415d4b0 0x7fc7b40ac8b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.485+0000 7fc7b9bf4700 1 -- 172.21.15.1:0/2270540596 >> 172.21.15.1:0/2270540596 conn(0x7fc7b4066b70 msgr2=0x7fc7b406a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:40.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.485+0000 7fc7b9bf4700 1 -- 172.21.15.1:0/2270540596 shutdown_connections 2024-03-18T19:05:40.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.485+0000 7fc7b9bf4700 1 -- 172.21.15.1:0/2270540596 wait complete. 2024-03-18T19:05:40.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.487+0000 7fc7b9bf4700 1 Processor -- start 2024-03-18T19:05:40.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.487+0000 7fc7b9bf4700 1 -- start start 2024-03-18T19:05:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.488+0000 7fc7b9bf4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7b406e920 0x7fc7b415c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.488+0000 7fc7b9bf4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7b40acdf0 0x7fc7b413fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.488+0000 7fc7b9bf4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7b415d4b0 0x7fc7b40cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.488+0000 7fc7b9bf4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b40ce310 con 0x7fc7b415d4b0 2024-03-18T19:05:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.488+0000 7fc7b9bf4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b415b8f0 con 0x7fc7b406e920 2024-03-18T19:05:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.488+0000 7fc7b9bf4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b4143650 con 0x7fc7b40acdf0 2024-03-18T19:05:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.488+0000 7fc7b3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7b415d4b0 0x7fc7b40cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.488+0000 7fc7b3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7b415d4b0 0x7fc7b40cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41328/0 (socket says 172.21.15.1:41328) 2024-03-18T19:05:40.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.488+0000 7fc7b3fff700 1 -- 172.21.15.1:0/40819939 learned_addr learned my addr 172.21.15.1:0/40819939 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:40.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.488+0000 7fc7b37fe700 1 --2- 172.21.15.1:0/40819939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7b406e920 0x7fc7b415c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:40.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.488+0000 7fc7b2ffd700 1 --2- 172.21.15.1:0/40819939 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7b40acdf0 0x7fc7b413fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:40.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.488+0000 7fc7b3fff700 1 -- 172.21.15.1:0/40819939 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7b40acdf0 msgr2=0x7fc7b413fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:40.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.488+0000 7fc7b3fff700 1 --2- 172.21.15.1:0/40819939 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7b40acdf0 0x7fc7b413fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.488+0000 7fc7b3fff700 1 -- 172.21.15.1:0/40819939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7b406e920 msgr2=0x7fc7b415c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:40.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.488+0000 7fc7b3fff700 1 --2- 172.21.15.1:0/40819939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7b406e920 0x7fc7b415c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.488+0000 7fc7b3fff700 1 -- 172.21.15.1:0/40819939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a40094a0 con 0x7fc7b415d4b0 2024-03-18T19:05:40.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.488+0000 7fc7b37fe700 1 --2- 172.21.15.1:0/40819939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7b406e920 0x7fc7b415c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:40.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.488+0000 7fc7b3fff700 1 --2- 172.21.15.1:0/40819939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7b415d4b0 0x7fc7b40cd7a0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00cb10 tx=0x7fc7ac00ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:40.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.489+0000 7fc79bfff700 1 -- 172.21.15.1:0/40819939 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac00a9e0 con 0x7fc7b415d4b0 2024-03-18T19:05:40.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.489+0000 7fc79bfff700 1 -- 172.21.15.1:0/40819939 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ac00ab60 con 0x7fc7b415d4b0 2024-03-18T19:05:40.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.489+0000 7fc7b9bf4700 1 -- 172.21.15.1:0/40819939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b4140360 con 0x7fc7b415d4b0 2024-03-18T19:05:40.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.489+0000 7fc79bfff700 1 -- 172.21.15.1:0/40819939 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac012400 con 0x7fc7b415d4b0 2024-03-18T19:05:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.489+0000 7fc7b9bf4700 1 -- 172.21.15.1:0/40819939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b415be50 con 0x7fc7b415d4b0 2024-03-18T19:05:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.490+0000 7fc7b9bf4700 1 -- 172.21.15.1:0/40819939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b40676a0 con 0x7fc7b415d4b0 2024-03-18T19:05:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.490+0000 7fc79bfff700 1 -- 172.21.15.1:0/40819939 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 34) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fc7ac01b3b0 con 0x7fc7b415d4b0 2024-03-18T19:05:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.491+0000 7fc79bfff700 1 --2- 172.21.15.1:0/40819939 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc7a006aae0 0x7fc7a006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.491+0000 7fc7b37fe700 1 -- 172.21.15.1:0/40819939 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc7a006aae0 msgr2=0x7fc7a006cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.491+0000 7fc79bfff700 1 -- 172.21.15.1:0/40819939 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fc7ac088eb0 con 0x7fc7b415d4b0 2024-03-18T19:05:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.491+0000 7fc7b37fe700 1 --2- 172.21.15.1:0/40819939 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc7a006aae0 0x7fc7a006cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:40.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.496+0000 7fc79bfff700 1 -- 172.21.15.1:0/40819939 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc7ac0540b0 con 0x7fc7b415d4b0 2024-03-18T19:05:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.692+0000 7fc7b37fe700 1 -- 172.21.15.1:0/40819939 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc7a006aae0 msgr2=0x7fc7a006cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:40.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.692+0000 7fc7b37fe700 1 --2- 172.21.15.1:0/40819939 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc7a006aae0 0x7fc7a006cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:40.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.696+0000 7fc7b9bf4700 1 -- 172.21.15.1:0/40819939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc7b41591a0 con 0x7fc7b415d4b0 2024-03-18T19:05:40.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.696+0000 7fc79bfff700 1 -- 172.21.15.1:0/40819939 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc7ac017020 con 0x7fc7b415d4b0 2024-03-18T19:05:40.693 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:05:40.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.698+0000 7fc7b9bf4700 1 -- 172.21.15.1:0/40819939 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc7a006aae0 msgr2=0x7fc7a006cfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:40.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.698+0000 7fc7b9bf4700 1 --2- 172.21.15.1:0/40819939 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc7a006aae0 0x7fc7a006cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.698+0000 7fc7b9bf4700 1 -- 172.21.15.1:0/40819939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7b415d4b0 msgr2=0x7fc7b40cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:40.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.698+0000 7fc7b9bf4700 1 --2- 172.21.15.1:0/40819939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7b415d4b0 0x7fc7b40cd7a0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00cb10 tx=0x7fc7ac00ce20 comp rx=0 tx=0).stop 2024-03-18T19:05:40.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.698+0000 7fc7b9bf4700 1 -- 172.21.15.1:0/40819939 shutdown_connections 2024-03-18T19:05:40.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.698+0000 7fc7b9bf4700 1 --2- 172.21.15.1:0/40819939 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc7a006aae0 0x7fc7a006cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.698+0000 7fc7b9bf4700 1 --2- 172.21.15.1:0/40819939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7b415d4b0 0x7fc7b40cd7a0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.698+0000 7fc7b9bf4700 1 --2- 172.21.15.1:0/40819939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7b406e920 0x7fc7b415c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.698+0000 7fc7b9bf4700 1 --2- 172.21.15.1:0/40819939 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc7b40acdf0 0x7fc7b413fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.698+0000 7fc7b9bf4700 1 -- 172.21.15.1:0/40819939 >> 172.21.15.1:0/40819939 conn(0x7fc7b4066b70 msgr2=0x7fc7b40b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:40.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.699+0000 7fc7b9bf4700 1 -- 172.21.15.1:0/40819939 shutdown_connections 2024-03-18T19:05:40.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.699+0000 7fc7b9bf4700 1 -- 172.21.15.1:0/40819939 wait complete. 2024-03-18T19:05:40.708 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T19:05:40.768 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:40.772+0000 7f16d3057240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:05:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.843+0000 7fa7a571e700 1 Processor -- start 2024-03-18T19:05:40.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.843+0000 7fa7a571e700 1 -- start start 2024-03-18T19:05:40.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.844+0000 7fa7a571e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7a00b1e20 0x7fa7a00cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.844+0000 7fa7a571e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa7a00cbce0 0x7fa7a00cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.844+0000 7fa7a571e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a00d0df0 con 0x7fa7a00cd870 2024-03-18T19:05:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.844+0000 7fa7a571e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7a00d0550 con 0x7fa7a00cbce0 2024-03-18T19:05:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.844+0000 7fa7a571e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a00d0830 con 0x7fa7a00b1e20 2024-03-18T19:05:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.844+0000 7fa79f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa7a00cbce0 0x7fa7a00cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.844+0000 7fa79f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa7a00cbce0 0x7fa7a00cc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41344/0 (socket says 172.21.15.1:41344) 2024-03-18T19:05:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.844+0000 7fa79f7fe700 1 -- 172.21.15.1:0/3431715239 learned_addr learned my addr 172.21.15.1:0/3431715239 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.844+0000 7fa79e7fc700 1 --2- 172.21.15.1:0/3431715239 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7a00b1e20 0x7fa7a00cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.844+0000 7fa79f7fe700 1 -- 172.21.15.1:0/3431715239 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7a00b1e20 msgr2=0x7fa7a00cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.844+0000 7fa79f7fe700 1 --2- 172.21.15.1:0/3431715239 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7a00b1e20 0x7fa7a00cff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.844+0000 7fa79f7fe700 1 -- 172.21.15.1:0/3431715239 >> v1:172.21.15.1:6789/0 conn(0x7fa7a00cd870 legacy=0x7fa7a00cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.844+0000 7fa79f7fe700 1 -- 172.21.15.1:0/3431715239 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a00d0b10 con 0x7fa7a00cbce0 2024-03-18T19:05:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.844+0000 7fa79e7fc700 1 --2- 172.21.15.1:0/3431715239 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7a00b1e20 0x7fa7a00cff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.845+0000 7fa79f7fe700 1 --2- 172.21.15.1:0/3431715239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7a00cbce0 0x7fa7a00cc170 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fa798009e50 tx=0x7fa79800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aece67a226b49b53 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.845+0000 7fa79dffb700 1 -- 172.21.15.1:0/3431715239 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79800e070 con 0x7fa7a00cbce0 2024-03-18T19:05:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.845+0000 7fa79dffb700 1 -- 172.21.15.1:0/3431715239 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa79800b9c0 con 0x7fa7a00cbce0 2024-03-18T19:05:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.845+0000 7fa79dffb700 1 -- 172.21.15.1:0/3431715239 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79800bd70 con 0x7fa7a00cbce0 2024-03-18T19:05:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.845+0000 7fa7a571e700 1 -- 172.21.15.1:0/3431715239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7a00cbce0 msgr2=0x7fa7a00cc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.845+0000 7fa7a571e700 1 --2- 172.21.15.1:0/3431715239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7a00cbce0 0x7fa7a00cc170 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fa798009e50 tx=0x7fa79800b330 comp rx=0 tx=0).stop 2024-03-18T19:05:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.845+0000 7fa7a571e700 1 -- 172.21.15.1:0/3431715239 shutdown_connections 2024-03-18T19:05:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.845+0000 7fa7a571e700 1 --2- 172.21.15.1:0/3431715239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7a00cbce0 0x7fa7a00cc170 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.845+0000 7fa7a571e700 1 --2- 172.21.15.1:0/3431715239 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7a00b1e20 0x7fa7a00cff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.845+0000 7fa7a571e700 1 -- 172.21.15.1:0/3431715239 >> 172.21.15.1:0/3431715239 conn(0x7fa7a00d0ff0 msgr2=0x7fa7a00d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.845+0000 7fa7a571e700 1 -- 172.21.15.1:0/3431715239 shutdown_connections 2024-03-18T19:05:40.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.845+0000 7fa7a571e700 1 -- 172.21.15.1:0/3431715239 wait complete. 2024-03-18T19:05:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.847+0000 7fa7a571e700 1 Processor -- start 2024-03-18T19:05:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.847+0000 7fa7a571e700 1 -- start start 2024-03-18T19:05:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.847+0000 7fa7a571e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a00b1e20 0x7fa7a009e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa7a571e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7a00cbce0 0x7fa7a014a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa7a571e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7a00cd870 0x7fa7a009cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa7a571e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a009e2b0 con 0x7fa7a00cbce0 2024-03-18T19:05:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa7a571e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a009d6a0 con 0x7fa7a00b1e20 2024-03-18T19:05:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa7a571e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a009d980 con 0x7fa7a00cd870 2024-03-18T19:05:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa79e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7a00cbce0 0x7fa7a014a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa79e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7a00cbce0 0x7fa7a014a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41348/0 (socket says 172.21.15.1:41348) 2024-03-18T19:05:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa79e7fc700 1 -- 172.21.15.1:0/1244218722 learned_addr learned my addr 172.21.15.1:0/1244218722 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa79f7fe700 1 --2- 172.21.15.1:0/1244218722 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7a00cd870 0x7fa7a009cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa79effd700 1 --2- 172.21.15.1:0/1244218722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a00b1e20 0x7fa7a009e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa79e7fc700 1 -- 172.21.15.1:0/1244218722 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7a00cd870 msgr2=0x7fa7a009cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa79e7fc700 1 --2- 172.21.15.1:0/1244218722 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7a00cd870 0x7fa7a009cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa79e7fc700 1 -- 172.21.15.1:0/1244218722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a00b1e20 msgr2=0x7fa7a009e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa79e7fc700 1 --2- 172.21.15.1:0/1244218722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a00b1e20 0x7fa7a009e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa79e7fc700 1 -- 172.21.15.1:0/1244218722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa798009b30 con 0x7fa7a00cbce0 2024-03-18T19:05:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa79e7fc700 1 --2- 172.21.15.1:0/1244218722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7a00cbce0 0x7fa7a014a2f0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fa79400e810 tx=0x7fa79400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa7877fe700 1 -- 172.21.15.1:0/1244218722 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79400c070 con 0x7fa7a00cbce0 2024-03-18T19:05:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa7877fe700 1 -- 172.21.15.1:0/1244218722 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa794004230 con 0x7fa7a00cbce0 2024-03-18T19:05:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa7a571e700 1 -- 172.21.15.1:0/1244218722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a009dc60 con 0x7fa7a00cbce0 2024-03-18T19:05:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa7877fe700 1 -- 172.21.15.1:0/1244218722 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7940036a0 con 0x7fa7a00cbce0 2024-03-18T19:05:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.848+0000 7fa7a571e700 1 -- 172.21.15.1:0/1244218722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a014a830 con 0x7fa7a00cbce0 2024-03-18T19:05:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.850+0000 7fa7a571e700 1 -- 172.21.15.1:0/1244218722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a00cff70 con 0x7fa7a00cbce0 2024-03-18T19:05:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.850+0000 7fa7877fe700 1 -- 172.21.15.1:0/1244218722 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 35) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fa794003800 con 0x7fa7a00cbce0 2024-03-18T19:05:40.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.851+0000 7fa7877fe700 1 --2- 172.21.15.1:0/1244218722 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa78c06adb0 0x7fa78c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:40.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.851+0000 7fa79effd700 1 -- 172.21.15.1:0/1244218722 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa78c06adb0 msgr2=0x7fa78c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:40.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.851+0000 7fa7877fe700 1 -- 172.21.15.1:0/1244218722 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fa794089420 con 0x7fa7a00cbce0 2024-03-18T19:05:40.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.851+0000 7fa79effd700 1 --2- 172.21.15.1:0/1244218722 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa78c06adb0 0x7fa78c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:40.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:40.856+0000 7fa7877fe700 1 -- 172.21.15.1:0/1244218722 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa7940541d0 con 0x7fa7a00cbce0 2024-03-18T19:05:40.858 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:40.861+0000 7fddc73eb240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:05:40.981 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:40.985+0000 7f16d3057240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:05:41.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.052+0000 7fa79effd700 1 -- 172.21.15.1:0/1244218722 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa78c06adb0 msgr2=0x7fa78c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:41.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.052+0000 7fa79effd700 1 --2- 172.21.15.1:0/1244218722 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa78c06adb0 0x7fa78c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:41.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.065+0000 7fa7a571e700 1 -- 172.21.15.1:0/1244218722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa7a00676a0 con 0x7fa7a00cbce0 2024-03-18T19:05:41.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.065+0000 7fa7877fe700 1 -- 172.21.15.1:0/1244218722 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa7940577f0 con 0x7fa7a00cbce0 2024-03-18T19:05:41.062 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T19:05:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.067+0000 7fa7a571e700 1 -- 172.21.15.1:0/1244218722 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa78c06adb0 msgr2=0x7fa78c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.067+0000 7fa7a571e700 1 --2- 172.21.15.1:0/1244218722 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa78c06adb0 0x7fa78c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.067+0000 7fa7a571e700 1 -- 172.21.15.1:0/1244218722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7a00cbce0 msgr2=0x7fa7a014a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.067+0000 7fa7a571e700 1 --2- 172.21.15.1:0/1244218722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7a00cbce0 0x7fa7a014a2f0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fa79400e810 tx=0x7fa79400eb20 comp rx=0 tx=0).stop 2024-03-18T19:05:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.068+0000 7fa7a571e700 1 -- 172.21.15.1:0/1244218722 shutdown_connections 2024-03-18T19:05:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.068+0000 7fa7a571e700 1 --2- 172.21.15.1:0/1244218722 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa78c06adb0 0x7fa78c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.068+0000 7fa7a571e700 1 --2- 172.21.15.1:0/1244218722 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7a00cd870 0x7fa7a009cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.068+0000 7fa7a571e700 1 --2- 172.21.15.1:0/1244218722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a00b1e20 0x7fa7a009e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.068+0000 7fa7a571e700 1 --2- 172.21.15.1:0/1244218722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7a00cbce0 0x7fa7a014a2f0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.068+0000 7fa7a571e700 1 -- 172.21.15.1:0/1244218722 >> 172.21.15.1:0/1244218722 conn(0x7fa7a00d0ff0 msgr2=0x7fa7a006e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:41.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.068+0000 7fa7a571e700 1 -- 172.21.15.1:0/1244218722 shutdown_connections 2024-03-18T19:05:41.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.068+0000 7fa7a571e700 1 -- 172.21.15.1:0/1244218722 wait complete. 2024-03-18T19:05:41.077 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T19:05:41.080 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:41.084+0000 7fddc73eb240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:05:41.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.210+0000 7fd819961700 1 Processor -- start 2024-03-18T19:05:41.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.210+0000 7fd819961700 1 -- start start 2024-03-18T19:05:41.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.211+0000 7fd819961700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd81415d5a0 0x7fd8140ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:41.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.211+0000 7fd819961700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8140ce550 0x7fd81406a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:41.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.211+0000 7fd819961700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd8140cfa90 0x7fd81406ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:41.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.211+0000 7fd819961700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8140cf600 con 0x7fd8140ce550 2024-03-18T19:05:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.211+0000 7fd819961700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8140ce9f0 con 0x7fd8140cfa90 2024-03-18T19:05:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.211+0000 7fd819961700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8140cecd0 con 0x7fd81415d5a0 2024-03-18T19:05:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.211+0000 7fd8127fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8140ce550 0x7fd81406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.211+0000 7fd8127fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8140ce550 0x7fd81406a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41352/0 (socket says 172.21.15.1:41352) 2024-03-18T19:05:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.211+0000 7fd8127fc700 1 -- 172.21.15.1:0/1360968892 learned_addr learned my addr 172.21.15.1:0/1360968892 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.211+0000 7fd8137fe700 1 --2- 172.21.15.1:0/1360968892 >> v2:172.21.15.165:3300/0 conn(0x7fd8140cfa90 0x7fd81406ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.211+0000 7fd812ffd700 1 --2- 172.21.15.1:0/1360968892 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd81415d5a0 0x7fd8140ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.211+0000 7fd8127fc700 1 -- 172.21.15.1:0/1360968892 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd81415d5a0 msgr2=0x7fd8140ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.211+0000 7fd8127fc700 1 --2- 172.21.15.1:0/1360968892 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd81415d5a0 0x7fd8140ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.211+0000 7fd8127fc700 1 -- 172.21.15.1:0/1360968892 >> v2:172.21.15.165:3300/0 conn(0x7fd8140cfa90 msgr2=0x7fd81406ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.211+0000 7fd8127fc700 1 --2- 172.21.15.1:0/1360968892 >> v2:172.21.15.165:3300/0 conn(0x7fd8140cfa90 0x7fd81406ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.211+0000 7fd8127fc700 1 -- 172.21.15.1:0/1360968892 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8140cefb0 con 0x7fd8140ce550 2024-03-18T19:05:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.212+0000 7fd8127fc700 1 --2- 172.21.15.1:0/1360968892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8140ce550 0x7fd81406a750 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fd8140cdec0 tx=0x7fd808009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc29688254766f8b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:41.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.212+0000 7fd811ffb700 1 -- 172.21.15.1:0/1360968892 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd808011830 con 0x7fd8140ce550 2024-03-18T19:05:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.212+0000 7fd811ffb700 1 -- 172.21.15.1:0/1360968892 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd808011990 con 0x7fd8140ce550 2024-03-18T19:05:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.212+0000 7fd811ffb700 1 -- 172.21.15.1:0/1360968892 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd808011d20 con 0x7fd8140ce550 2024-03-18T19:05:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.212+0000 7fd819961700 1 -- 172.21.15.1:0/1360968892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8140ce550 msgr2=0x7fd81406a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.212+0000 7fd819961700 1 --2- 172.21.15.1:0/1360968892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8140ce550 0x7fd81406a750 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fd8140cdec0 tx=0x7fd808009bd0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.212+0000 7fd819961700 1 -- 172.21.15.1:0/1360968892 shutdown_connections 2024-03-18T19:05:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.212+0000 7fd819961700 1 --2- 172.21.15.1:0/1360968892 >> v2:172.21.15.165:3300/0 conn(0x7fd8140cfa90 0x7fd81406ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.212+0000 7fd819961700 1 --2- 172.21.15.1:0/1360968892 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd81415d5a0 0x7fd8140ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.212+0000 7fd819961700 1 --2- 172.21.15.1:0/1360968892 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8140ce550 0x7fd81406a750 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.212+0000 7fd819961700 1 -- 172.21.15.1:0/1360968892 >> 172.21.15.1:0/1360968892 conn(0x7fd8140cc9b0 msgr2=0x7fd8140ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.212+0000 7fd819961700 1 -- 172.21.15.1:0/1360968892 shutdown_connections 2024-03-18T19:05:41.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.212+0000 7fd819961700 1 -- 172.21.15.1:0/1360968892 wait complete. 2024-03-18T19:05:41.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.214+0000 7fd819961700 1 Processor -- start 2024-03-18T19:05:41.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.215+0000 7fd819961700 1 -- start start 2024-03-18T19:05:41.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.216+0000 7fd819961700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8140ce550 0x7fd8141f6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:41.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.216+0000 7fd819961700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8140cfa90 0x7fd814160230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:41.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.216+0000 7fd819961700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd81415d5a0 0x7fd81414f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:41.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.216+0000 7fd819961700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd814160770 con 0x7fd8140cfa90 2024-03-18T19:05:41.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.216+0000 7fd819961700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81414f970 con 0x7fd8140ce550 2024-03-18T19:05:41.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.216+0000 7fd819961700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8141f74b0 con 0x7fd81415d5a0 2024-03-18T19:05:41.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.216+0000 7fd8127fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8140cfa90 0x7fd814160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:41.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.216+0000 7fd8127fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8140cfa90 0x7fd814160230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41358/0 (socket says 172.21.15.1:41358) 2024-03-18T19:05:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.216+0000 7fd8127fc700 1 -- 172.21.15.1:0/3968206463 learned_addr learned my addr 172.21.15.1:0/3968206463 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.216+0000 7fd812ffd700 1 --2- 172.21.15.1:0/3968206463 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8140ce550 0x7fd8141f6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.216+0000 7fd8137fe700 1 --2- 172.21.15.1:0/3968206463 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd81415d5a0 0x7fd81414f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.216+0000 7fd8127fc700 1 -- 172.21.15.1:0/3968206463 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd81415d5a0 msgr2=0x7fd81414f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.216+0000 7fd8127fc700 1 --2- 172.21.15.1:0/3968206463 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd81415d5a0 0x7fd81414f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.216+0000 7fd8127fc700 1 -- 172.21.15.1:0/3968206463 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8140ce550 msgr2=0x7fd8141f6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.216+0000 7fd8127fc700 1 --2- 172.21.15.1:0/3968206463 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8140ce550 0x7fd8141f6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.216+0000 7fd8127fc700 1 -- 172.21.15.1:0/3968206463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd808009550 con 0x7fd8140cfa90 2024-03-18T19:05:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.216+0000 7fd8127fc700 1 --2- 172.21.15.1:0/3968206463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8140cfa90 0x7fd814160230 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fd808009c10 tx=0x7fd808004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.217+0000 7fd7fbfff700 1 -- 172.21.15.1:0/3968206463 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd808004450 con 0x7fd8140cfa90 2024-03-18T19:05:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.217+0000 7fd7fbfff700 1 -- 172.21.15.1:0/3968206463 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd80801e070 con 0x7fd8140cfa90 2024-03-18T19:05:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.217+0000 7fd819961700 1 -- 172.21.15.1:0/3968206463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8141f6c10 con 0x7fd8140cfa90 2024-03-18T19:05:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.217+0000 7fd7fbfff700 1 -- 172.21.15.1:0/3968206463 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8080038e0 con 0x7fd8140cfa90 2024-03-18T19:05:41.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.217+0000 7fd819961700 1 -- 172.21.15.1:0/3968206463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8141f71d0 con 0x7fd8140cfa90 2024-03-18T19:05:41.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.218+0000 7fd819961700 1 -- 172.21.15.1:0/3968206463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8140676a0 con 0x7fd8140cfa90 2024-03-18T19:05:41.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.218+0000 7fd7fbfff700 1 -- 172.21.15.1:0/3968206463 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 35) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fd808003a40 con 0x7fd8140cfa90 2024-03-18T19:05:41.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.219+0000 7fd7fbfff700 1 --2- 172.21.15.1:0/3968206463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd80006adb0 0x7fd80006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:41.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.219+0000 7fd812ffd700 1 -- 172.21.15.1:0/3968206463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd80006adb0 msgr2=0x7fd80006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:41.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.219+0000 7fd7fbfff700 1 -- 172.21.15.1:0/3968206463 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fd80808a3b0 con 0x7fd8140cfa90 2024-03-18T19:05:41.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.219+0000 7fd812ffd700 1 --2- 172.21.15.1:0/3968206463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd80006adb0 0x7fd80006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:41.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.223+0000 7fd7fbfff700 1 -- 172.21.15.1:0/3968206463 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd808055270 con 0x7fd8140cfa90 2024-03-18T19:05:41.348 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:41.352+0000 7f16d3057240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:05:41.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.420+0000 7fd812ffd700 1 -- 172.21.15.1:0/3968206463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd80006adb0 msgr2=0x7fd80006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:41.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.420+0000 7fd812ffd700 1 --2- 172.21.15.1:0/3968206463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd80006adb0 0x7fd80006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:41.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.426+0000 7fd819961700 1 -- 172.21.15.1:0/3968206463 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd8140c6da0 con 0x7fd8140cfa90 2024-03-18T19:05:41.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.427+0000 7fd7fbfff700 1 -- 172.21.15.1:0/3968206463 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd8080045b0 con 0x7fd8140cfa90 2024-03-18T19:05:41.424 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T19:05:41.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.429+0000 7fd819961700 1 -- 172.21.15.1:0/3968206463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd80006adb0 msgr2=0x7fd80006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.429+0000 7fd819961700 1 --2- 172.21.15.1:0/3968206463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd80006adb0 0x7fd80006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.429+0000 7fd819961700 1 -- 172.21.15.1:0/3968206463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8140cfa90 msgr2=0x7fd814160230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.429+0000 7fd819961700 1 --2- 172.21.15.1:0/3968206463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8140cfa90 0x7fd814160230 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fd808009c10 tx=0x7fd808004750 comp rx=0 tx=0).stop 2024-03-18T19:05:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.429+0000 7fd819961700 1 -- 172.21.15.1:0/3968206463 shutdown_connections 2024-03-18T19:05:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.429+0000 7fd819961700 1 --2- 172.21.15.1:0/3968206463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd80006adb0 0x7fd80006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.429+0000 7fd819961700 1 --2- 172.21.15.1:0/3968206463 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd81415d5a0 0x7fd81414f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.429+0000 7fd819961700 1 --2- 172.21.15.1:0/3968206463 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8140ce550 0x7fd8141f6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.429+0000 7fd819961700 1 --2- 172.21.15.1:0/3968206463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8140cfa90 0x7fd814160230 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.429+0000 7fd819961700 1 -- 172.21.15.1:0/3968206463 >> 172.21.15.1:0/3968206463 conn(0x7fd8140cc9b0 msgr2=0x7fd8140ad620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.430+0000 7fd819961700 1 -- 172.21.15.1:0/3968206463 shutdown_connections 2024-03-18T19:05:41.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.430+0000 7fd819961700 1 -- 172.21.15.1:0/3968206463 wait complete. 2024-03-18T19:05:41.439 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T19:05:41.478 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:41.481+0000 7fddc73eb240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:05:41.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.608+0000 7f35fba3a700 1 Processor -- start 2024-03-18T19:05:41.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.608+0000 7f35fba3a700 1 -- start start 2024-03-18T19:05:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.609+0000 7f35fba3a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35f40cd870 0x7f35f40cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.609+0000 7f35fba3a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f35f40cbce0 0x7f35f40cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.609+0000 7f35fba3a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f40d0df0 con 0x7f35f40b1e20 2024-03-18T19:05:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.609+0000 7f35fba3a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f40d0550 con 0x7f35f40cd870 2024-03-18T19:05:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.609+0000 7f35fba3a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f40d0830 con 0x7f35f40cbce0 2024-03-18T19:05:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.609+0000 7f35f97d6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35f40cd870 0x7f35f40cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.610+0000 7f35f97d6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35f40cd870 0x7f35f40cff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41374/0 (socket says 172.21.15.1:41374) 2024-03-18T19:05:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.610+0000 7f35f97d6700 1 -- 172.21.15.1:0/2656309739 learned_addr learned my addr 172.21.15.1:0/2656309739 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.610+0000 7f35f9fd7700 1 --2- 172.21.15.1:0/2656309739 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f35f40cbce0 0x7f35f40cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:41.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.610+0000 7f35f97d6700 1 -- 172.21.15.1:0/2656309739 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f35f40cbce0 msgr2=0x7f35f40cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.610+0000 7f35f97d6700 1 --2- 172.21.15.1:0/2656309739 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f35f40cbce0 0x7f35f40cc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.610+0000 7f35f97d6700 1 -- 172.21.15.1:0/2656309739 >> v1:172.21.15.165:6789/0 conn(0x7f35f40b1e20 legacy=0x7f35f40cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.610+0000 7f35f97d6700 1 -- 172.21.15.1:0/2656309739 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f40d0b10 con 0x7f35f40cd870 2024-03-18T19:05:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.610+0000 7f35f97d6700 1 --2- 172.21.15.1:0/2656309739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35f40cd870 0x7f35f40cff70 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f35f00098f0 tx=0x7f35f0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff7cce14b7b6fc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.610+0000 7f35ebfff700 1 -- 172.21.15.1:0/2656309739 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0011830 con 0x7f35f40cd870 2024-03-18T19:05:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.610+0000 7f35ebfff700 1 -- 172.21.15.1:0/2656309739 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f0011990 con 0x7f35f40cd870 2024-03-18T19:05:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.610+0000 7f35ebfff700 1 -- 172.21.15.1:0/2656309739 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0011d20 con 0x7f35f40cd870 2024-03-18T19:05:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.611+0000 7f35fba3a700 1 -- 172.21.15.1:0/2656309739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35f40cd870 msgr2=0x7f35f40cff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.611+0000 7f35fba3a700 1 --2- 172.21.15.1:0/2656309739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35f40cd870 0x7f35f40cff70 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f35f00098f0 tx=0x7f35f0009c00 comp rx=0 tx=0).stop 2024-03-18T19:05:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.611+0000 7f35fba3a700 1 -- 172.21.15.1:0/2656309739 shutdown_connections 2024-03-18T19:05:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.611+0000 7f35fba3a700 1 --2- 172.21.15.1:0/2656309739 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f35f40cbce0 0x7f35f40cc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.611+0000 7f35fba3a700 1 --2- 172.21.15.1:0/2656309739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35f40cd870 0x7f35f40cff70 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.611+0000 7f35fba3a700 1 -- 172.21.15.1:0/2656309739 >> 172.21.15.1:0/2656309739 conn(0x7f35f40d0ff0 msgr2=0x7f35f40d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:41.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.611+0000 7f35fba3a700 1 -- 172.21.15.1:0/2656309739 shutdown_connections 2024-03-18T19:05:41.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.611+0000 7f35fba3a700 1 -- 172.21.15.1:0/2656309739 wait complete. 2024-03-18T19:05:41.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.612+0000 7f35fba3a700 1 Processor -- start 2024-03-18T19:05:41.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.612+0000 7f35fba3a700 1 -- start start 2024-03-18T19:05:41.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.613+0000 7f35fba3a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f35f40b1e20 0x7f35f41f3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:41.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.613+0000 7f35fba3a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35f40cbce0 0x7f35f4158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.613+0000 7f35fba3a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f40cd870 0x7f35f4159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.613+0000 7f35fba3a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f40af710 con 0x7f35f40cbce0 2024-03-18T19:05:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.613+0000 7f35fba3a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f409eba0 con 0x7f35f40cd870 2024-03-18T19:05:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.613+0000 7f35fba3a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f409c020 con 0x7f35f40b1e20 2024-03-18T19:05:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.613+0000 7f35f8fd5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35f40cbce0 0x7f35f4158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.613+0000 7f35f97d6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f35f40b1e20 0x7f35f41f3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.613+0000 7f35f8fd5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35f40cbce0 0x7f35f4158920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41378/0 (socket says 172.21.15.1:41378) 2024-03-18T19:05:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.613+0000 7f35f8fd5700 1 -- 172.21.15.1:0/4123348247 learned_addr learned my addr 172.21.15.1:0/4123348247 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.613+0000 7f35f9fd7700 1 --2- 172.21.15.1:0/4123348247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f40cd870 0x7f35f4159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.614+0000 7f35f8fd5700 1 -- 172.21.15.1:0/4123348247 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f35f40b1e20 msgr2=0x7f35f41f3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.614+0000 7f35f8fd5700 1 --2- 172.21.15.1:0/4123348247 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f35f40b1e20 0x7f35f41f3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.614+0000 7f35f8fd5700 1 -- 172.21.15.1:0/4123348247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f40cd870 msgr2=0x7f35f4159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.614+0000 7f35f8fd5700 1 --2- 172.21.15.1:0/4123348247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f40cd870 0x7f35f4159130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.614+0000 7f35f8fd5700 1 -- 172.21.15.1:0/4123348247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f0009550 con 0x7f35f40cbce0 2024-03-18T19:05:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.614+0000 7f35f97d6700 1 --2- 172.21.15.1:0/4123348247 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f35f40b1e20 0x7f35f41f3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.614+0000 7f35f8fd5700 1 --2- 172.21.15.1:0/4123348247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35f40cbce0 0x7f35f4158920 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f35e400c4e0 tx=0x7f35e400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.614+0000 7f35e9ffb700 1 -- 172.21.15.1:0/4123348247 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e40052d0 con 0x7f35f40cbce0 2024-03-18T19:05:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.614+0000 7f35e9ffb700 1 -- 172.21.15.1:0/4123348247 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e4005430 con 0x7f35f40cbce0 2024-03-18T19:05:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.614+0000 7f35e9ffb700 1 -- 172.21.15.1:0/4123348247 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e40038c0 con 0x7f35f40cbce0 2024-03-18T19:05:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.614+0000 7f35fba3a700 1 -- 172.21.15.1:0/4123348247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f409c300 con 0x7f35f40cbce0 2024-03-18T19:05:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.614+0000 7f35fba3a700 1 -- 172.21.15.1:0/4123348247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f4158e60 con 0x7f35f40cbce0 2024-03-18T19:05:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.615+0000 7f35fba3a700 1 -- 172.21.15.1:0/4123348247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f40af850 con 0x7f35f40cbce0 2024-03-18T19:05:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.616+0000 7f35e9ffb700 1 -- 172.21.15.1:0/4123348247 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 35) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f35e4003a20 con 0x7f35f40cbce0 2024-03-18T19:05:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.616+0000 7f35e9ffb700 1 --2- 172.21.15.1:0/4123348247 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f35d806adb0 0x7f35d806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.616+0000 7f35f97d6700 1 -- 172.21.15.1:0/4123348247 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f35d806adb0 msgr2=0x7f35d806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.616+0000 7f35e9ffb700 1 -- 172.21.15.1:0/4123348247 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f35e4088a40 con 0x7f35f40cbce0 2024-03-18T19:05:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.616+0000 7f35f97d6700 1 --2- 172.21.15.1:0/4123348247 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f35d806adb0 0x7f35d806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.621+0000 7f35e9ffb700 1 -- 172.21.15.1:0/4123348247 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f35f40af850 con 0x7f35f40cbce0 2024-03-18T19:05:41.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.817+0000 7f35f97d6700 1 -- 172.21.15.1:0/4123348247 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f35d806adb0 msgr2=0x7f35d806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:41.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.817+0000 7f35f97d6700 1 --2- 172.21.15.1:0/4123348247 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f35d806adb0 0x7f35d806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:41.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.842+0000 7f35fba3a700 1 -- 172.21.15.1:0/4123348247 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f35f40af850 con 0x7f35f40cbce0 2024-03-18T19:05:41.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.842+0000 7f35e9ffb700 1 -- 172.21.15.1:0/4123348247 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v90) v1 ==== 80+0+450 (secure 0 0 0) 0x7f35f40af850 con 0x7f35f40cbce0 2024-03-18T19:05:41.839 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2546947071 2024-03-19T19:05:37.779786+0000 2024-03-18T19:05:41.839 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3332271581 2024-03-19T19:05:37.779786+0000 2024-03-18T19:05:41.840 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6833/76998 2024-03-19T19:05:37.779786+0000 2024-03-18T19:05:41.840 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6832/76998 2024-03-19T19:05:37.779786+0000 2024-03-18T19:05:41.840 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1890712448 2024-03-19T19:05:37.779786+0000 2024-03-18T19:05:41.840 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/300431708 2024-03-19T19:05:37.779786+0000 2024-03-18T19:05:41.840 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/4106216506 2024-03-19T19:05:37.779786+0000 2024-03-18T19:05:41.840 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/569963265 2024-03-19T19:05:37.779786+0000 2024-03-18T19:05:41.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.845+0000 7f35fba3a700 1 -- 172.21.15.1:0/4123348247 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f35d806adb0 msgr2=0x7f35d806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:41.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.845+0000 7f35fba3a700 1 --2- 172.21.15.1:0/4123348247 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f35d806adb0 0x7f35d806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.845+0000 7f35fba3a700 1 -- 172.21.15.1:0/4123348247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35f40cbce0 msgr2=0x7f35f4158920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:41.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.845+0000 7f35fba3a700 1 --2- 172.21.15.1:0/4123348247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35f40cbce0 0x7f35f4158920 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f35e400c4e0 tx=0x7f35e400c7f0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.845+0000 7f35fba3a700 1 -- 172.21.15.1:0/4123348247 shutdown_connections 2024-03-18T19:05:41.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.845+0000 7f35fba3a700 1 --2- 172.21.15.1:0/4123348247 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f35d806adb0 0x7f35d806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.845+0000 7f35fba3a700 1 --2- 172.21.15.1:0/4123348247 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f35f40b1e20 0x7f35f41f3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.845+0000 7f35fba3a700 1 --2- 172.21.15.1:0/4123348247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f40cd870 0x7f35f4159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.845+0000 7f35fba3a700 1 --2- 172.21.15.1:0/4123348247 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35f40cbce0 0x7f35f4158920 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:41.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.845+0000 7f35fba3a700 1 -- 172.21.15.1:0/4123348247 >> 172.21.15.1:0/4123348247 conn(0x7f35f40d0ff0 msgr2=0x7f35f40cc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:41.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.845+0000 7f35fba3a700 1 -- 172.21.15.1:0/4123348247 shutdown_connections 2024-03-18T19:05:41.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:41.845+0000 7f35fba3a700 1 -- 172.21.15.1:0/4123348247 wait complete. 2024-03-18T19:05:41.843 INFO:teuthology.orchestra.run.smithi001.stderr:listed 8 entries 2024-03-18T19:05:41.854 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-18T19:05:41.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:41.854 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T19:05:41.897 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[112967]: fuse finished with error 0 and tester_r 0 2024-03-18T19:05:41.899 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T19:05:41.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:41.899 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T19:05:41.930 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:41.934+0000 7f16d3057240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:05:42.001 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:05:42.001 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:05:42.002 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:42.005+0000 7f16d3057240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:05:42.019 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:05:42.020 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T19:05:42.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:42.020 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-18T19:05:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:05:42.077 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-18T19:05:42.078 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:05:42.078 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:05:42.097 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:05:42.097 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T19:05:42.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:42.097 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-18T19:05:42.111 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:42.114+0000 7fddc73eb240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:05:42.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:05:42.130 INFO:teuthology.orchestra.run.smithi001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-18T19:05:42.130 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:05:42.131 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:05:42.139 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:42.143+0000 7f16d3057240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:05:42.149 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:05:42.149 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T19:05:42.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:05:42.150 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-18T19:05:42.189 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:42.193+0000 7fddc73eb240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:05:42.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:05:42.207 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-18T19:05:42.207 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:05:42.323 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:42.327+0000 7f16d3057240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:05:42.329 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:42.332+0000 7fddc73eb240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:05:42.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.339+0000 7f97f1008700 1 Processor -- start 2024-03-18T19:05:42.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.339+0000 7f97f1008700 1 -- start start 2024-03-18T19:05:42.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.340+0000 7f97f1008700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f97ec09a650 0x7f97ec099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:42.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.340+0000 7f97f1008700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f97ec099f80 0x7f97ec06f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:42.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.340+0000 7f97f1008700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97ec06f070 con 0x7f97ec15b4d0 2024-03-18T19:05:42.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.340+0000 7f97f1008700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97ec09a3f0 con 0x7f97ec099f80 2024-03-18T19:05:42.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.340+0000 7f97f1008700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97ec0d0260 con 0x7f97ec09a650 2024-03-18T19:05:42.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.340+0000 7f97ead9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f97ec15b4d0 0x7f97ec0cf750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56388/0 (socket says 172.21.15.1:56388) 2024-03-18T19:05:42.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.340+0000 7f97eb59e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f97ec099f80 0x7f97ec06f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:42.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.340+0000 7f97ead9d700 1 -- 172.21.15.1:0/458453117 learned_addr learned my addr 172.21.15.1:0/458453117 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:42.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.340+0000 7f97ea59c700 1 --2- 172.21.15.1:0/458453117 >> v2:172.21.15.165:3300/0 conn(0x7f97ec09a650 0x7f97ec099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:42.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.341+0000 7f97e9d9b700 1 -- 172.21.15.1:0/458453117 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935737517 0 0) 0x7f97ec06f070 con 0x7f97ec15b4d0 2024-03-18T19:05:42.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.341+0000 7f97eb59e700 1 -- 172.21.15.1:0/458453117 >> v2:172.21.15.165:3300/0 conn(0x7f97ec09a650 msgr2=0x7f97ec099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:42.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.341+0000 7f97eb59e700 1 --2- 172.21.15.1:0/458453117 >> v2:172.21.15.165:3300/0 conn(0x7f97ec09a650 0x7f97ec099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.341+0000 7f97eb59e700 1 -- 172.21.15.1:0/458453117 >> v1:172.21.15.1:6789/0 conn(0x7f97ec15b4d0 legacy=0x7f97ec0cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:42.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.341+0000 7f97eb59e700 1 -- 172.21.15.1:0/458453117 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec0cff80 con 0x7f97ec099f80 2024-03-18T19:05:42.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.341+0000 7f97eb59e700 1 --2- 172.21.15.1:0/458453117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97ec099f80 0x7f97ec06f500 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f97ec071d40 tx=0x7f97e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d23cb537d66493e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:42.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.341+0000 7f97e9d9b700 1 -- 172.21.15.1:0/458453117 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e400e070 con 0x7f97ec099f80 2024-03-18T19:05:42.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.341+0000 7f97e9d9b700 1 -- 172.21.15.1:0/458453117 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e400bb20 con 0x7f97ec099f80 2024-03-18T19:05:42.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.341+0000 7f97e9d9b700 1 -- 172.21.15.1:0/458453117 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e4012430 con 0x7f97ec099f80 2024-03-18T19:05:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.342+0000 7f97f1008700 1 -- 172.21.15.1:0/458453117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97ec099f80 msgr2=0x7f97ec06f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.342+0000 7f97f1008700 1 --2- 172.21.15.1:0/458453117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97ec099f80 0x7f97ec06f500 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f97ec071d40 tx=0x7f97e400b330 comp rx=0 tx=0).stop 2024-03-18T19:05:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.342+0000 7f97f1008700 1 -- 172.21.15.1:0/458453117 shutdown_connections 2024-03-18T19:05:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.342+0000 7f97f1008700 1 --2- 172.21.15.1:0/458453117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97ec099f80 0x7f97ec06f500 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.342+0000 7f97f1008700 1 --2- 172.21.15.1:0/458453117 >> v2:172.21.15.165:3300/0 conn(0x7f97ec09a650 0x7f97ec099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.342+0000 7f97f1008700 1 -- 172.21.15.1:0/458453117 >> 172.21.15.1:0/458453117 conn(0x7f97ec0b2cb0 msgr2=0x7f97ec0b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:42.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.342+0000 7f97f1008700 1 -- 172.21.15.1:0/458453117 shutdown_connections 2024-03-18T19:05:42.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.342+0000 7f97f1008700 1 -- 172.21.15.1:0/458453117 wait complete. 2024-03-18T19:05:42.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.343+0000 7f97f1008700 1 Processor -- start 2024-03-18T19:05:42.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.343+0000 7f97f1008700 1 -- start start 2024-03-18T19:05:42.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.344+0000 7f97f1008700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97ec099f80 0x7f97ec15bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:42.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.344+0000 7f97f1008700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97ec09a650 0x7f97ec1f5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:42.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.344+0000 7f97f1008700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97ec15b4d0 0x7f97ec14bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:42.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.344+0000 7f97f1008700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec1f6390 con 0x7f97ec099f80 2024-03-18T19:05:42.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.344+0000 7f97f1008700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec14c290 con 0x7f97ec15b4d0 2024-03-18T19:05:42.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.344+0000 7f97f1008700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec0c7760 con 0x7f97ec09a650 2024-03-18T19:05:42.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.344+0000 7f97ead9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97ec099f80 0x7f97ec15bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:42.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.344+0000 7f97ead9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97ec099f80 0x7f97ec15bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41394/0 (socket says 172.21.15.1:41394) 2024-03-18T19:05:42.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.344+0000 7f97ead9d700 1 -- 172.21.15.1:0/3540063083 learned_addr learned my addr 172.21.15.1:0/3540063083 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:42.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.344+0000 7f97eb59e700 1 --2- 172.21.15.1:0/3540063083 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97ec15b4d0 0x7f97ec14bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:42.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.344+0000 7f97ea59c700 1 --2- 172.21.15.1:0/3540063083 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97ec09a650 0x7f97ec1f5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:42.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.345+0000 7f97ead9d700 1 -- 172.21.15.1:0/3540063083 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97ec09a650 msgr2=0x7f97ec1f5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:42.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.345+0000 7f97ead9d700 1 --2- 172.21.15.1:0/3540063083 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97ec09a650 0x7f97ec1f5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.345+0000 7f97ead9d700 1 -- 172.21.15.1:0/3540063083 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97ec15b4d0 msgr2=0x7f97ec14bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:42.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.345+0000 7f97ead9d700 1 --2- 172.21.15.1:0/3540063083 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97ec15b4d0 0x7f97ec14bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.345+0000 7f97ead9d700 1 -- 172.21.15.1:0/3540063083 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e4009a70 con 0x7f97ec099f80 2024-03-18T19:05:42.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.345+0000 7f97ead9d700 1 --2- 172.21.15.1:0/3540063083 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97ec099f80 0x7f97ec15bac0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f97dc00c550 tx=0x7f97dc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:42.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.345+0000 7f97d37fe700 1 -- 172.21.15.1:0/3540063083 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc004c80 con 0x7f97ec099f80 2024-03-18T19:05:42.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.345+0000 7f97d37fe700 1 -- 172.21.15.1:0/3540063083 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97dc004e00 con 0x7f97ec099f80 2024-03-18T19:05:42.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.345+0000 7f97f1008700 1 -- 172.21.15.1:0/3540063083 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ec15d4a0 con 0x7f97ec099f80 2024-03-18T19:05:42.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.345+0000 7f97d37fe700 1 -- 172.21.15.1:0/3540063083 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc007310 con 0x7f97ec099f80 2024-03-18T19:05:42.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.345+0000 7f97f1008700 1 -- 172.21.15.1:0/3540063083 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ec15da60 con 0x7f97ec099f80 2024-03-18T19:05:42.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.346+0000 7f97f1008700 1 -- 172.21.15.1:0/3540063083 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ec068240 con 0x7f97ec099f80 2024-03-18T19:05:42.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.347+0000 7f97d37fe700 1 -- 172.21.15.1:0/3540063083 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 35) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f97dc007470 con 0x7f97ec099f80 2024-03-18T19:05:42.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.347+0000 7f97d37fe700 1 --2- 172.21.15.1:0/3540063083 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97d806ade0 0x7f97d806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:42.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.347+0000 7f97ea59c700 1 -- 172.21.15.1:0/3540063083 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97d806ade0 msgr2=0x7f97d806d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:42.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.347+0000 7f97d37fe700 1 -- 172.21.15.1:0/3540063083 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f97dc0892d0 con 0x7f97ec099f80 2024-03-18T19:05:42.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.347+0000 7f97ea59c700 1 --2- 172.21.15.1:0/3540063083 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97d806ade0 0x7f97d806d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:42.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.352+0000 7f97d37fe700 1 -- 172.21.15.1:0/3540063083 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f97dc054100 con 0x7f97ec099f80 2024-03-18T19:05:42.396 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:42.400+0000 7f16d3057240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:05:42.473 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:42.477+0000 7f16d3057240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:05:42.518 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:42.521+0000 7fddc73eb240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:05:42.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.548+0000 7f97ea59c700 1 -- 172.21.15.1:0/3540063083 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97d806ade0 msgr2=0x7f97d806d2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:42.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.548+0000 7f97ea59c700 1 --2- 172.21.15.1:0/3540063083 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97d806ade0 0x7f97d806d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:42.550 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:42.554+0000 7f16d3057240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:05:42.595 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:42.598+0000 7fddc73eb240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:05:42.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.604+0000 7f97f1008700 1 -- 172.21.15.1:0/3540063083 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97ec15d300 con 0x7f97ec099f80 2024-03-18T19:05:42.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.604+0000 7f97d37fe700 1 -- 172.21.15.1:0/3540063083 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4732 (secure 0 0 0) 0x7f97ec15d300 con 0x7f97ec099f80 2024-03-18T19:05:42.601 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:42.601 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5744,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2437290126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2437290126},{"type":"v1","addr":"172.21.15.1:6836","nonce":2437290126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5747,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2832642785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2832642785},{"type":"v1","addr":"172.21.15.1:6837","nonce":2832642785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5756,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3765527492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3765527492},{"type":"v1","addr":"172.21.15.165:6834","nonce":3765527492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:05:22.577555+0000","modified":"2024-03-18T19:05:25.594597+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5759},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5759":{"gid":5759,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/119895951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":119895951},{"type":"v1","addr":"172.21.15.165:6835","nonce":119895951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-18T19:05:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.606+0000 7f97f1008700 1 -- 172.21.15.1:0/3540063083 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97d806ade0 msgr2=0x7f97d806d2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.606+0000 7f97f1008700 1 --2- 172.21.15.1:0/3540063083 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97d806ade0 0x7f97d806d2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.606+0000 7f97f1008700 1 -- 172.21.15.1:0/3540063083 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97ec099f80 msgr2=0x7f97ec15bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.606+0000 7f97f1008700 1 --2- 172.21.15.1:0/3540063083 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97ec099f80 0x7f97ec15bac0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f97dc00c550 tx=0x7f97dc00c860 comp rx=0 tx=0).stop 2024-03-18T19:05:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.606+0000 7f97f1008700 1 -- 172.21.15.1:0/3540063083 shutdown_connections 2024-03-18T19:05:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.606+0000 7f97f1008700 1 --2- 172.21.15.1:0/3540063083 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97d806ade0 0x7f97d806d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.606+0000 7f97f1008700 1 --2- 172.21.15.1:0/3540063083 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97ec15b4d0 0x7f97ec14bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.607+0000 7f97f1008700 1 --2- 172.21.15.1:0/3540063083 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97ec099f80 0x7f97ec15bac0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.607+0000 7f97f1008700 1 --2- 172.21.15.1:0/3540063083 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97ec09a650 0x7f97ec1f5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.607+0000 7f97f1008700 1 -- 172.21.15.1:0/3540063083 >> 172.21.15.1:0/3540063083 conn(0x7f97ec0b2cb0 msgr2=0x7f97ec0b1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.607+0000 7f97f1008700 1 -- 172.21.15.1:0/3540063083 shutdown_connections 2024-03-18T19:05:42.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.607+0000 7f97f1008700 1 -- 172.21.15.1:0/3540063083 wait complete. 2024-03-18T19:05:42.605 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 33 2024-03-18T19:05:42.616 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:05:42.628 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:42.632+0000 7f16d3057240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:05:42.681 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:42.684+0000 7fddc73eb240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:05:42.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.746+0000 7fb8466cc700 1 Processor -- start 2024-03-18T19:05:42.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.746+0000 7fb8466cc700 1 -- start start 2024-03-18T19:05:42.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.747+0000 7fb8466cc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb84006a750 0x7fb84006ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:42.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.747+0000 7fb8466cc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb8400ad790 0x7fb840161700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:42.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.747+0000 7fb8466cc700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8400acfb0 con 0x7fb84015d6b0 2024-03-18T19:05:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.747+0000 7fb8466cc700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8400af7a0 con 0x7fb8400ad790 2024-03-18T19:05:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.747+0000 7fb8466cc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84006b070 con 0x7fb84006a750 2024-03-18T19:05:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.747+0000 7fb844c69700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb8400ad790 0x7fb840161700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.747+0000 7fb83f7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb84015d6b0 0x7fb840160ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56402/0 (socket says 172.21.15.1:56402) 2024-03-18T19:05:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.747+0000 7fb844c69700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb8400ad790 0x7fb840161700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41404/0 (socket says 172.21.15.1:41404) 2024-03-18T19:05:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.747+0000 7fb83f7fe700 1 -- 172.21.15.1:0/2852741546 learned_addr learned my addr 172.21.15.1:0/2852741546 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.747+0000 7fb83ffff700 1 --2- 172.21.15.1:0/2852741546 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb84006a750 0x7fb84006ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.747+0000 7fb844c69700 1 -- 172.21.15.1:0/2852741546 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb84006a750 msgr2=0x7fb84006ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.747+0000 7fb844c69700 1 --2- 172.21.15.1:0/2852741546 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb84006a750 0x7fb84006ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.747+0000 7fb844c69700 1 -- 172.21.15.1:0/2852741546 >> v1:172.21.15.1:6789/0 conn(0x7fb84015d6b0 legacy=0x7fb840160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.747+0000 7fb844c69700 1 -- 172.21.15.1:0/2852741546 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8400b50a0 con 0x7fb8400ad790 2024-03-18T19:05:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.747+0000 7fb83effd700 1 -- 172.21.15.1:0/2852741546 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708192258 0 0) 0x7fb8400acfb0 con 0x7fb84015d6b0 2024-03-18T19:05:42.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.747+0000 7fb844c69700 1 --2- 172.21.15.1:0/2852741546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8400ad790 0x7fb840161700 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fb838009d90 tx=0x7fb83800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf743b5a71fbdcf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:42.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.748+0000 7fb83effd700 1 -- 172.21.15.1:0/2852741546 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83800e070 con 0x7fb8400ad790 2024-03-18T19:05:42.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.748+0000 7fb83effd700 1 -- 172.21.15.1:0/2852741546 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83800bb20 con 0x7fb8400ad790 2024-03-18T19:05:42.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.748+0000 7fb83effd700 1 -- 172.21.15.1:0/2852741546 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb838012430 con 0x7fb8400ad790 2024-03-18T19:05:42.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.748+0000 7fb8466cc700 1 -- 172.21.15.1:0/2852741546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8400ad790 msgr2=0x7fb840161700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:42.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.748+0000 7fb8466cc700 1 --2- 172.21.15.1:0/2852741546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8400ad790 0x7fb840161700 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fb838009d90 tx=0x7fb83800b330 comp rx=0 tx=0).stop 2024-03-18T19:05:42.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.748+0000 7fb8466cc700 1 -- 172.21.15.1:0/2852741546 shutdown_connections 2024-03-18T19:05:42.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.749+0000 7fb8466cc700 1 --2- 172.21.15.1:0/2852741546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8400ad790 0x7fb840161700 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.749+0000 7fb8466cc700 1 --2- 172.21.15.1:0/2852741546 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb84006a750 0x7fb84006ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.749+0000 7fb8466cc700 1 -- 172.21.15.1:0/2852741546 >> 172.21.15.1:0/2852741546 conn(0x7fb84006b860 msgr2=0x7fb84006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:42.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.749+0000 7fb8466cc700 1 -- 172.21.15.1:0/2852741546 shutdown_connections 2024-03-18T19:05:42.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.749+0000 7fb8466cc700 1 -- 172.21.15.1:0/2852741546 wait complete. 2024-03-18T19:05:42.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.750+0000 7fb8466cc700 1 Processor -- start 2024-03-18T19:05:42.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.750+0000 7fb8466cc700 1 -- start start 2024-03-18T19:05:42.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.751+0000 7fb8466cc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb84006a750 0x7fb8400cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:42.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.751+0000 7fb8466cc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8400ad790 0x7fb8400af8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:42.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.751+0000 7fb8466cc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb84015d6b0 0x7fb8400afe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:42.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.751+0000 7fb8466cc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8400cd260 con 0x7fb84015d6b0 2024-03-18T19:05:42.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.751+0000 7fb8466cc700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8400cc970 con 0x7fb8400ad790 2024-03-18T19:05:42.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.751+0000 7fb8466cc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8400ccc10 con 0x7fb84006a750 2024-03-18T19:05:42.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.751+0000 7fb844c69700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb84015d6b0 0x7fb8400afe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:42.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.751+0000 7fb844c69700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb84015d6b0 0x7fb8400afe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41418/0 (socket says 172.21.15.1:41418) 2024-03-18T19:05:42.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.751+0000 7fb844c69700 1 -- 172.21.15.1:0/4256454140 learned_addr learned my addr 172.21.15.1:0/4256454140 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:42.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.751+0000 7fb83ffff700 1 --2- 172.21.15.1:0/4256454140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb84006a750 0x7fb8400cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:42.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.751+0000 7fb83f7fe700 1 --2- 172.21.15.1:0/4256454140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8400ad790 0x7fb8400af8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:42.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.751+0000 7fb844c69700 1 -- 172.21.15.1:0/4256454140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb84006a750 msgr2=0x7fb8400cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:42.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.751+0000 7fb844c69700 1 --2- 172.21.15.1:0/4256454140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb84006a750 0x7fb8400cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.751+0000 7fb844c69700 1 -- 172.21.15.1:0/4256454140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8400ad790 msgr2=0x7fb8400af8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:42.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.751+0000 7fb844c69700 1 --2- 172.21.15.1:0/4256454140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8400ad790 0x7fb8400af8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.751+0000 7fb844c69700 1 -- 172.21.15.1:0/4256454140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb838009a70 con 0x7fb84015d6b0 2024-03-18T19:05:42.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.751+0000 7fb844c69700 1 --2- 172.21.15.1:0/4256454140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb84015d6b0 0x7fb8400afe20 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fb8400672c0 tx=0x7fb838004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:42.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.752+0000 7fb83cff9700 1 -- 172.21.15.1:0/4256454140 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb838012430 con 0x7fb84015d6b0 2024-03-18T19:05:42.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.752+0000 7fb83cff9700 1 -- 172.21.15.1:0/4256454140 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb838020070 con 0x7fb84015d6b0 2024-03-18T19:05:42.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.752+0000 7fb8466cc700 1 -- 172.21.15.1:0/4256454140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8400ccef0 con 0x7fb84015d6b0 2024-03-18T19:05:42.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.752+0000 7fb83cff9700 1 -- 172.21.15.1:0/4256454140 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb838004280 con 0x7fb84015d6b0 2024-03-18T19:05:42.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.752+0000 7fb8466cc700 1 -- 172.21.15.1:0/4256454140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8400cdc30 con 0x7fb84015d6b0 2024-03-18T19:05:42.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.753+0000 7fb8466cc700 1 -- 172.21.15.1:0/4256454140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8400cd410 con 0x7fb84015d6b0 2024-03-18T19:05:42.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.753+0000 7fb83cff9700 1 -- 172.21.15.1:0/4256454140 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 35) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fb8380043e0 con 0x7fb84015d6b0 2024-03-18T19:05:42.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.753+0000 7fb83cff9700 1 --2- 172.21.15.1:0/4256454140 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb82c06adb0 0x7fb82c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:42.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.754+0000 7fb83ffff700 1 -- 172.21.15.1:0/4256454140 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb82c06adb0 msgr2=0x7fb82c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:42.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.754+0000 7fb83cff9700 1 -- 172.21.15.1:0/4256454140 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fb838022070 con 0x7fb84015d6b0 2024-03-18T19:05:42.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.754+0000 7fb83ffff700 1 --2- 172.21.15.1:0/4256454140 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb82c06adb0 0x7fb82c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:42.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.758+0000 7fb83cff9700 1 -- 172.21.15.1:0/4256454140 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb8400cd410 con 0x7fb84015d6b0 2024-03-18T19:05:42.767 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:42.770+0000 7fddc73eb240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:05:42.797 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:42.801+0000 7f16d3057240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:05:42.850 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:42.853+0000 7fddc73eb240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:05:42.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.955+0000 7fb83ffff700 1 -- 172.21.15.1:0/4256454140 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb82c06adb0 msgr2=0x7fb82c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:42.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.955+0000 7fb83ffff700 1 --2- 172.21.15.1:0/4256454140 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb82c06adb0 0x7fb82c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:42.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.961+0000 7fb8466cc700 1 -- 172.21.15.1:0/4256454140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8400addc0 con 0x7fb84015d6b0 2024-03-18T19:05:42.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.963+0000 7fb83cff9700 1 -- 172.21.15.1:0/4256454140 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+16681 (secure 0 0 0) 0x7fb838056580 con 0x7fb84015d6b0 2024-03-18T19:05:42.960 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:42.960 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":90,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:05:37.779828+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:05:20.423723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-18T19:05:21.600285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2546947071":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/3332271581":"2024-03-19T19:05:37.779786+0000","172.21.15.1:6833/76998":"2024-03-19T19:05:37.779786+0000","172.21.15.1:6832/76998":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/1890712448":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/300431708":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/4106216506":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/569963265":"2024-03-19T19:05:37.779786+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:05:42.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.965+0000 7fb8466cc700 1 -- 172.21.15.1:0/4256454140 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb82c06adb0 msgr2=0x7fb82c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:42.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.965+0000 7fb8466cc700 1 --2- 172.21.15.1:0/4256454140 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb82c06adb0 0x7fb82c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.965+0000 7fb8466cc700 1 -- 172.21.15.1:0/4256454140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb84015d6b0 msgr2=0x7fb8400afe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.965+0000 7fb8466cc700 1 --2- 172.21.15.1:0/4256454140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb84015d6b0 0x7fb8400afe20 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fb8400672c0 tx=0x7fb838004ad0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.965+0000 7fb8466cc700 1 -- 172.21.15.1:0/4256454140 shutdown_connections 2024-03-18T19:05:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.965+0000 7fb8466cc700 1 --2- 172.21.15.1:0/4256454140 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb82c06adb0 0x7fb82c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.965+0000 7fb8466cc700 1 --2- 172.21.15.1:0/4256454140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb84006a750 0x7fb8400cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.965+0000 7fb8466cc700 1 --2- 172.21.15.1:0/4256454140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb84015d6b0 0x7fb8400afe20 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.965+0000 7fb8466cc700 1 --2- 172.21.15.1:0/4256454140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8400ad790 0x7fb8400af8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.965+0000 7fb8466cc700 1 -- 172.21.15.1:0/4256454140 >> 172.21.15.1:0/4256454140 conn(0x7fb84006b860 msgr2=0x7fb840160280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.965+0000 7fb8466cc700 1 -- 172.21.15.1:0/4256454140 shutdown_connections 2024-03-18T19:05:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:42.965+0000 7fb8466cc700 1 -- 172.21.15.1:0/4256454140 wait complete. 2024-03-18T19:05:42.975 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T19:05:20.423723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T19:05:21.600285+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:05:42.976 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:05:43.011 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:43.015+0000 7f16d3057240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:05:43.032 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:43.035+0000 7fddc73eb240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:05:43.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.108+0000 7ff362c4a700 1 Processor -- start 2024-03-18T19:05:43.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.108+0000 7ff362c4a700 1 -- start start 2024-03-18T19:05:43.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.109+0000 7ff362c4a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff35c1612d0 0x7ff35c1616f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:43.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.109+0000 7ff362c4a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35c162170 con 0x7ff35c160e50 2024-03-18T19:05:43.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.109+0000 7ff362c4a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35c161c30 con 0x7ff35c160a70 2024-03-18T19:05:43.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.109+0000 7ff362c4a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff35c0d06e0 con 0x7ff35c1612d0 2024-03-18T19:05:43.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.109+0000 7ff35bfff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff35c160e50 0x7ff35c0cffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56410/0 (socket says 172.21.15.1:56410) 2024-03-18T19:05:43.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.109+0000 7ff35bfff700 1 -- 172.21.15.1:0/719682095 learned_addr learned my addr 172.21.15.1:0/719682095 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:43.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.109+0000 7ff3611e7700 1 --2- 172.21.15.1:0/719682095 >> v2:172.21.15.165:3300/0 conn(0x7ff35c1612d0 0x7ff35c1616f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:43.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.110+0000 7ff35b7fe700 1 -- 172.21.15.1:0/719682095 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148674954 0 0) 0x7ff348001110 con 0x7ff35c160e50 2024-03-18T19:05:43.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.110+0000 7ff35b7fe700 1 -- 172.21.15.1:0/719682095 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff34c0035b0 con 0x7ff35c160e50 2024-03-18T19:05:43.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.110+0000 7ff35b7fe700 1 -- 172.21.15.1:0/719682095 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544899391 0 0) 0x7ff35c161c30 con 0x7ff35c160a70 2024-03-18T19:05:43.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.110+0000 7ff35b7fe700 1 -- 172.21.15.1:0/719682095 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff348001110 con 0x7ff35c160a70 2024-03-18T19:05:43.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.110+0000 7ff3611e7700 1 -- 172.21.15.1:0/719682095 >> v1:172.21.15.165:6789/0 conn(0x7ff35c160a70 legacy=0x7ff35c0d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:43.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.110+0000 7ff3611e7700 1 -- 172.21.15.1:0/719682095 >> v1:172.21.15.1:6789/0 conn(0x7ff35c160e50 legacy=0x7ff35c0cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:43.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.110+0000 7ff3611e7700 1 -- 172.21.15.1:0/719682095 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c0cef30 con 0x7ff35c1612d0 2024-03-18T19:05:43.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.110+0000 7ff3611e7700 1 --2- 172.21.15.1:0/719682095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff35c1612d0 0x7ff35c1616f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff354009d90 tx=0x7ff35400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa22e090870196fa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:43.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.110+0000 7ff35b7fe700 1 -- 172.21.15.1:0/719682095 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35400e070 con 0x7ff35c1612d0 2024-03-18T19:05:43.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.110+0000 7ff35b7fe700 1 -- 172.21.15.1:0/719682095 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35400bb20 con 0x7ff35c1612d0 2024-03-18T19:05:43.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.110+0000 7ff35b7fe700 1 -- 172.21.15.1:0/719682095 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354012430 con 0x7ff35c1612d0 2024-03-18T19:05:43.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.111+0000 7ff362c4a700 1 -- 172.21.15.1:0/719682095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff35c1612d0 msgr2=0x7ff35c1616f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:43.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.111+0000 7ff362c4a700 1 --2- 172.21.15.1:0/719682095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff35c1612d0 0x7ff35c1616f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff354009d90 tx=0x7ff35400b330 comp rx=0 tx=0).stop 2024-03-18T19:05:43.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.111+0000 7ff362c4a700 1 -- 172.21.15.1:0/719682095 shutdown_connections 2024-03-18T19:05:43.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.111+0000 7ff362c4a700 1 --2- 172.21.15.1:0/719682095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff35c1612d0 0x7ff35c1616f0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.111+0000 7ff362c4a700 1 -- 172.21.15.1:0/719682095 >> 172.21.15.1:0/719682095 conn(0x7ff35c09c560 msgr2=0x7ff35c0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:43.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.111+0000 7ff362c4a700 1 -- 172.21.15.1:0/719682095 shutdown_connections 2024-03-18T19:05:43.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.111+0000 7ff362c4a700 1 -- 172.21.15.1:0/719682095 wait complete. 2024-03-18T19:05:43.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.112+0000 7ff362c4a700 1 Processor -- start 2024-03-18T19:05:43.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.112+0000 7ff362c4a700 1 -- start start 2024-03-18T19:05:43.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.113+0000 7ff362c4a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff35c160a70 0x7ff35c14d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:43.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.113+0000 7ff362c4a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff35c160e50 0x7ff35c0ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:43.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.113+0000 7ff362c4a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c1612d0 0x7ff35c159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:43.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.113+0000 7ff362c4a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0c6bd0 con 0x7ff35c1612d0 2024-03-18T19:05:43.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.113+0000 7ff362c4a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0c7750 con 0x7ff35c160e50 2024-03-18T19:05:43.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.113+0000 7ff362c4a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0c6e20 con 0x7ff35c160a70 2024-03-18T19:05:43.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.113+0000 7ff3611e7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c1612d0 0x7ff35c159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:43.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.113+0000 7ff3611e7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c1612d0 0x7ff35c159420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41424/0 (socket says 172.21.15.1:41424) 2024-03-18T19:05:43.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.113+0000 7ff3611e7700 1 -- 172.21.15.1:0/3777983949 learned_addr learned my addr 172.21.15.1:0/3777983949 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:43.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.113+0000 7ff35bfff700 1 --2- 172.21.15.1:0/3777983949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff35c160e50 0x7ff35c0ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:43.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.113+0000 7ff3609e6700 1 --2- 172.21.15.1:0/3777983949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff35c160a70 0x7ff35c14d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:43.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.114+0000 7ff3611e7700 1 -- 172.21.15.1:0/3777983949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff35c160a70 msgr2=0x7ff35c14d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:43.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.114+0000 7ff3611e7700 1 --2- 172.21.15.1:0/3777983949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff35c160a70 0x7ff35c14d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.114+0000 7ff3611e7700 1 -- 172.21.15.1:0/3777983949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff35c160e50 msgr2=0x7ff35c0ca320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:43.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.114+0000 7ff3611e7700 1 --2- 172.21.15.1:0/3777983949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff35c160e50 0x7ff35c0ca320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.114+0000 7ff3611e7700 1 -- 172.21.15.1:0/3777983949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff354009a70 con 0x7ff35c1612d0 2024-03-18T19:05:43.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.114+0000 7ff3611e7700 1 --2- 172.21.15.1:0/3777983949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c1612d0 0x7ff35c159420 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ff35c06afe0 tx=0x7ff354004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:43.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.114+0000 7ff3597fa700 1 -- 172.21.15.1:0/3777983949 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354012430 con 0x7ff35c1612d0 2024-03-18T19:05:43.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.114+0000 7ff3597fa700 1 -- 172.21.15.1:0/3777983949 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff354020070 con 0x7ff35c1612d0 2024-03-18T19:05:43.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.114+0000 7ff3597fa700 1 -- 172.21.15.1:0/3777983949 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35401ba50 con 0x7ff35c1612d0 2024-03-18T19:05:43.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.114+0000 7ff362c4a700 1 -- 172.21.15.1:0/3777983949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff35c0c7100 con 0x7ff35c1612d0 2024-03-18T19:05:43.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.114+0000 7ff362c4a700 1 -- 172.21.15.1:0/3777983949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35c14db40 con 0x7ff35c1612d0 2024-03-18T19:05:43.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.115+0000 7ff362c4a700 1 -- 172.21.15.1:0/3777983949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff35c067390 con 0x7ff35c1612d0 2024-03-18T19:05:43.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.116+0000 7ff3597fa700 1 -- 172.21.15.1:0/3777983949 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 35) v1 ==== 89240+0+0 (secure 0 0 0) 0x7ff35401bbb0 con 0x7ff35c1612d0 2024-03-18T19:05:43.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.117+0000 7ff3597fa700 1 --2- 172.21.15.1:0/3777983949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff34006adb0 0x7ff34006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:43.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.117+0000 7ff3609e6700 1 -- 172.21.15.1:0/3777983949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff34006adb0 msgr2=0x7ff34006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:43.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.117+0000 7ff3597fa700 1 -- 172.21.15.1:0/3777983949 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6373+0+0 (secure 0 0 0) 0x7ff354022070 con 0x7ff35c1612d0 2024-03-18T19:05:43.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.117+0000 7ff3609e6700 1 --2- 172.21.15.1:0/3777983949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff34006adb0 0x7ff34006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:43.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.122+0000 7ff3597fa700 1 -- 172.21.15.1:0/3777983949 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff354056470 con 0x7ff35c1612d0 2024-03-18T19:05:43.159 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:43.163+0000 7f16d3057240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:05:43.231 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:43.234+0000 7f16d3057240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:05:43.257 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:43.260+0000 7fddc73eb240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:05:43.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.318+0000 7ff3609e6700 1 -- 172.21.15.1:0/3777983949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff34006adb0 msgr2=0x7ff34006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:43.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.318+0000 7ff3609e6700 1 --2- 172.21.15.1:0/3777983949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff34006adb0 0x7ff34006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:43.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.323+0000 7ff362c4a700 1 -- 172.21.15.1:0/3777983949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff35c0c78c0 con 0x7ff35c1612d0 2024-03-18T19:05:43.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.323+0000 7ff3597fa700 1 -- 172.21.15.1:0/3777983949 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff35401be60 con 0x7ff35c1612d0 2024-03-18T19:05:43.320 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:05:43.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.325+0000 7ff362c4a700 1 -- 172.21.15.1:0/3777983949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff34006adb0 msgr2=0x7ff34006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:43.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.325+0000 7ff362c4a700 1 --2- 172.21.15.1:0/3777983949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff34006adb0 0x7ff34006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.325+0000 7ff362c4a700 1 -- 172.21.15.1:0/3777983949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c1612d0 msgr2=0x7ff35c159420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:43.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.325+0000 7ff362c4a700 1 --2- 172.21.15.1:0/3777983949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c1612d0 0x7ff35c159420 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ff35c06afe0 tx=0x7ff354004ad0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.325+0000 7ff362c4a700 1 -- 172.21.15.1:0/3777983949 shutdown_connections 2024-03-18T19:05:43.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.325+0000 7ff362c4a700 1 --2- 172.21.15.1:0/3777983949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff34006adb0 0x7ff34006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.325+0000 7ff362c4a700 1 --2- 172.21.15.1:0/3777983949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff35c160a70 0x7ff35c14d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.325+0000 7ff362c4a700 1 --2- 172.21.15.1:0/3777983949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c1612d0 0x7ff35c159420 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.325+0000 7ff362c4a700 1 --2- 172.21.15.1:0/3777983949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff35c160e50 0x7ff35c0ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.325+0000 7ff362c4a700 1 -- 172.21.15.1:0/3777983949 >> 172.21.15.1:0/3777983949 conn(0x7ff35c09c560 msgr2=0x7ff35c15c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:43.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.325+0000 7ff362c4a700 1 -- 172.21.15.1:0/3777983949 shutdown_connections 2024-03-18T19:05:43.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.325+0000 7ff362c4a700 1 -- 172.21.15.1:0/3777983949 wait complete. 2024-03-18T19:05:43.335 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T19:05:43.421 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:43.424+0000 7fddc73eb240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:05:43.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.466+0000 7f13b8376700 1 Processor -- start 2024-03-18T19:05:43.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.466+0000 7f13b8376700 1 -- start start 2024-03-18T19:05:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.467+0000 7f13b8376700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f13b006f180 0x7f13b0161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.467+0000 7f13b8376700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f13b006a750 0x7f13b006aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.467+0000 7f13b8376700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b006b0e0 con 0x7f13b006eda0 2024-03-18T19:05:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.467+0000 7f13b8376700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b00cb320 con 0x7f13b006f180 2024-03-18T19:05:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.467+0000 7f13b8376700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b00ce010 con 0x7f13b006a750 2024-03-18T19:05:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.467+0000 7f13b6913700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f13b006a750 0x7f13b006aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.467+0000 7f13b5911700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f13b006f180 0x7f13b0161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.467+0000 7f13b6112700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f13b006eda0 0x7f13b00ca580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:34116/0 (socket says 172.21.15.1:34116) 2024-03-18T19:05:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.467+0000 7f13b6112700 1 -- 172.21.15.1:0/808258650 learned_addr learned my addr 172.21.15.1:0/808258650 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.468+0000 7f13b5110700 1 -- 172.21.15.1:0/808258650 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676500692 0 0) 0x7f13ac001110 con 0x7f13b006eda0 2024-03-18T19:05:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.468+0000 7f13b5110700 1 -- 172.21.15.1:0/808258650 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f139c0035b0 con 0x7f13b006eda0 2024-03-18T19:05:43.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.468+0000 7f13b5911700 1 -- 172.21.15.1:0/808258650 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f13b006a750 msgr2=0x7f13b006aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.468+0000 7f13b5911700 1 --2- 172.21.15.1:0/808258650 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f13b006a750 0x7f13b006aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.468+0000 7f13b5911700 1 -- 172.21.15.1:0/808258650 >> v1:172.21.15.165:6789/0 conn(0x7f13b006eda0 legacy=0x7f13b00ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.468+0000 7f13b5911700 1 -- 172.21.15.1:0/808258650 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b00cad60 con 0x7f13b006f180 2024-03-18T19:05:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.468+0000 7f13b6913700 1 --2- 172.21.15.1:0/808258650 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f13b006a750 0x7f13b006aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.468+0000 7f13b5911700 1 --2- 172.21.15.1:0/808258650 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13b006f180 0x7f13b0161c60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009840 tx=0x7f13a0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=402c8e6a59c2b110 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.468+0000 7f13b5110700 1 -- 172.21.15.1:0/808258650 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a0011830 con 0x7f13b006f180 2024-03-18T19:05:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.468+0000 7f13b5110700 1 -- 172.21.15.1:0/808258650 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a0011990 con 0x7f13b006f180 2024-03-18T19:05:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.468+0000 7f13b5110700 1 -- 172.21.15.1:0/808258650 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a0011d20 con 0x7f13b006f180 2024-03-18T19:05:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.469+0000 7f13b8376700 1 -- 172.21.15.1:0/808258650 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13b006f180 msgr2=0x7f13b0161c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.469+0000 7f13b8376700 1 --2- 172.21.15.1:0/808258650 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13b006f180 0x7f13b0161c60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009840 tx=0x7f13a0009b50 comp rx=0 tx=0).stop 2024-03-18T19:05:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.469+0000 7f13b8376700 1 -- 172.21.15.1:0/808258650 shutdown_connections 2024-03-18T19:05:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.469+0000 7f13b8376700 1 --2- 172.21.15.1:0/808258650 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f13b006a750 0x7f13b006aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.469+0000 7f13b8376700 1 --2- 172.21.15.1:0/808258650 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13b006f180 0x7f13b0161c60 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.469+0000 7f13b8376700 1 -- 172.21.15.1:0/808258650 >> 172.21.15.1:0/808258650 conn(0x7f13b00cff30 msgr2=0x7f13b00d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:43.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.469+0000 7f13b8376700 1 -- 172.21.15.1:0/808258650 shutdown_connections 2024-03-18T19:05:43.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.469+0000 7f13b8376700 1 -- 172.21.15.1:0/808258650 wait complete. 2024-03-18T19:05:43.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.470+0000 7f13b8376700 1 Processor -- start 2024-03-18T19:05:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.470+0000 7f13b8376700 1 -- start start 2024-03-18T19:05:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.471+0000 7f13b8376700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13b006a750 0x7f13b015c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.471+0000 7f13b8376700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f13b006eda0 0x7f13b01f3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.471+0000 7f13b8376700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13b006f180 0x7f13b00b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.471+0000 7f13b8376700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b00b2ee0 con 0x7f13b006a750 2024-03-18T19:05:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.471+0000 7f13b8376700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b00b31c0 con 0x7f13b006f180 2024-03-18T19:05:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.471+0000 7f13b8376700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b00b34a0 con 0x7f13b006eda0 2024-03-18T19:05:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.471+0000 7f13b6112700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13b006a750 0x7f13b015c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.471+0000 7f13b6112700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13b006a750 0x7f13b015c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41440/0 (socket says 172.21.15.1:41440) 2024-03-18T19:05:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.471+0000 7f13b6112700 1 -- 172.21.15.1:0/815418139 learned_addr learned my addr 172.21.15.1:0/815418139 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.471+0000 7f13b5911700 1 --2- 172.21.15.1:0/815418139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f13b006eda0 0x7f13b01f3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.471+0000 7f13b6913700 1 --2- 172.21.15.1:0/815418139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13b006f180 0x7f13b00b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:43.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.471+0000 7f13b6112700 1 -- 172.21.15.1:0/815418139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f13b006eda0 msgr2=0x7f13b01f3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.471+0000 7f13b6112700 1 --2- 172.21.15.1:0/815418139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f13b006eda0 0x7f13b01f3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.471+0000 7f13b6112700 1 -- 172.21.15.1:0/815418139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13b006f180 msgr2=0x7f13b00b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.471+0000 7f13b6112700 1 --2- 172.21.15.1:0/815418139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13b006f180 0x7f13b00b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.471+0000 7f13b6112700 1 -- 172.21.15.1:0/815418139 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a8009a70 con 0x7f13b006a750 2024-03-18T19:05:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.471+0000 7f13b6112700 1 --2- 172.21.15.1:0/815418139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13b006a750 0x7f13b015c420 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f13ac007fd0 tx=0x7f13ac00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.472+0000 7f13a6ffd700 1 -- 172.21.15.1:0/815418139 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac005d70 con 0x7f13b006a750 2024-03-18T19:05:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.472+0000 7f13a6ffd700 1 -- 172.21.15.1:0/815418139 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13ac0070d0 con 0x7f13b006a750 2024-03-18T19:05:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.472+0000 7f13b8376700 1 -- 172.21.15.1:0/815418139 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a00094a0 con 0x7f13b006a750 2024-03-18T19:05:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.472+0000 7f13a6ffd700 1 -- 172.21.15.1:0/815418139 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac007460 con 0x7f13b006a750 2024-03-18T19:05:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.472+0000 7f13b8376700 1 -- 172.21.15.1:0/815418139 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b00b3a60 con 0x7f13b006a750 2024-03-18T19:05:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.473+0000 7f13b8376700 1 -- 172.21.15.1:0/815418139 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b00676a0 con 0x7f13b006a750 2024-03-18T19:05:43.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.473+0000 7f13a6ffd700 1 -- 172.21.15.1:0/815418139 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 35) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f13ac0075c0 con 0x7f13b006a750 2024-03-18T19:05:43.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.474+0000 7f13a6ffd700 1 --2- 172.21.15.1:0/815418139 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f139406adb0 0x7f139406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:43.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.474+0000 7f13b5911700 1 -- 172.21.15.1:0/815418139 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f139406adb0 msgr2=0x7f139406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:43.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.474+0000 7f13a6ffd700 1 -- 172.21.15.1:0/815418139 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f13ac0889e0 con 0x7f13b006a750 2024-03-18T19:05:43.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.474+0000 7f13b5911700 1 --2- 172.21.15.1:0/815418139 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f139406adb0 0x7f139406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:43.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.478+0000 7f13a6ffd700 1 -- 172.21.15.1:0/815418139 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f13ac053820 con 0x7f13b006a750 2024-03-18T19:05:43.501 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:43.504+0000 7fddc73eb240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:05:43.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.675+0000 7f13b5911700 1 -- 172.21.15.1:0/815418139 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f139406adb0 msgr2=0x7f139406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:43.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.675+0000 7f13b5911700 1 --2- 172.21.15.1:0/815418139 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f139406adb0 0x7f139406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:43.672 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:43.675+0000 7f16d3057240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:05:43.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.681+0000 7f13b8376700 1 -- 172.21.15.1:0/815418139 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f13b006b0e0 con 0x7f13b006a750 2024-03-18T19:05:43.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.681+0000 7f13a6ffd700 1 -- 172.21.15.1:0/815418139 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f13b006b0e0 con 0x7f13b006a750 2024-03-18T19:05:43.678 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T19:05:43.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.683+0000 7f13b8376700 1 -- 172.21.15.1:0/815418139 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f139406adb0 msgr2=0x7f139406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:43.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.683+0000 7f13b8376700 1 --2- 172.21.15.1:0/815418139 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f139406adb0 0x7f139406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.683+0000 7f13b8376700 1 -- 172.21.15.1:0/815418139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13b006a750 msgr2=0x7f13b015c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:43.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.683+0000 7f13b8376700 1 --2- 172.21.15.1:0/815418139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13b006a750 0x7f13b015c420 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f13ac007fd0 tx=0x7f13ac00c9e0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.683+0000 7f13b8376700 1 -- 172.21.15.1:0/815418139 shutdown_connections 2024-03-18T19:05:43.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.683+0000 7f13b8376700 1 --2- 172.21.15.1:0/815418139 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f139406adb0 0x7f139406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.683+0000 7f13b8376700 1 --2- 172.21.15.1:0/815418139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13b006f180 0x7f13b00b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.683+0000 7f13b8376700 1 --2- 172.21.15.1:0/815418139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13b006a750 0x7f13b015c420 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.683+0000 7f13b8376700 1 --2- 172.21.15.1:0/815418139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f13b006eda0 0x7f13b01f3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.683+0000 7f13b8376700 1 -- 172.21.15.1:0/815418139 >> 172.21.15.1:0/815418139 conn(0x7f13b00cff30 msgr2=0x7f13b00b0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:43.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.683+0000 7f13b8376700 1 -- 172.21.15.1:0/815418139 shutdown_connections 2024-03-18T19:05:43.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.684+0000 7f13b8376700 1 -- 172.21.15.1:0/815418139 wait complete. 2024-03-18T19:05:43.693 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T19:05:43.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.825+0000 7f542d834700 1 Processor -- start 2024-03-18T19:05:43.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.825+0000 7f542d834700 1 -- start start 2024-03-18T19:05:43.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.826+0000 7f542d834700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54280ade00 0x7f54280cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:43.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.826+0000 7f542d834700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54280b3aa0 con 0x7f54280ccc20 2024-03-18T19:05:43.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.826+0000 7f542d834700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54280ae280 con 0x7f5428099a40 2024-03-18T19:05:43.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.826+0000 7f542d834700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f542806e9b0 con 0x7f54280ade00 2024-03-18T19:05:43.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.826+0000 7f54277fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f54280ccc20 0x7f542806fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56426/0 (socket says 172.21.15.1:56426) 2024-03-18T19:05:43.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.826+0000 7f54277fe700 1 -- 172.21.15.1:0/2198885661 learned_addr learned my addr 172.21.15.1:0/2198885661 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:43.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.826+0000 7f54267fc700 1 --2- 172.21.15.1:0/2198885661 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54280ade00 0x7f54280cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:43.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.827+0000 7f5425ffb700 1 -- 172.21.15.1:0/2198885661 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185843093 0 0) 0x7f54280b3aa0 con 0x7f54280ccc20 2024-03-18T19:05:43.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.827+0000 7f5425ffb700 1 -- 172.21.15.1:0/2198885661 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54100035b0 con 0x7f54280ccc20 2024-03-18T19:05:43.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.827+0000 7f5425ffb700 1 -- 172.21.15.1:0/2198885661 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060269090 0 0) 0x7f54280ae280 con 0x7f5428099a40 2024-03-18T19:05:43.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.827+0000 7f5425ffb700 1 -- 172.21.15.1:0/2198885661 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54280b3aa0 con 0x7f5428099a40 2024-03-18T19:05:43.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.827+0000 7f54267fc700 1 -- 172.21.15.1:0/2198885661 >> v1:172.21.15.165:6789/0 conn(0x7f5428099a40 legacy=0x7f5428099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:43.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.827+0000 7f54267fc700 1 -- 172.21.15.1:0/2198885661 >> v1:172.21.15.1:6789/0 conn(0x7f54280ccc20 legacy=0x7f542806fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:43.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.827+0000 7f54267fc700 1 -- 172.21.15.1:0/2198885661 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542806ec90 con 0x7f54280ade00 2024-03-18T19:05:43.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.827+0000 7f54267fc700 1 --2- 172.21.15.1:0/2198885661 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54280ade00 0x7f54280cc6e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5418009870 tx=0x7f5418009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12e604462aad6345 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:43.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.827+0000 7f5425ffb700 1 -- 172.21.15.1:0/2198885661 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5418011830 con 0x7f54280ade00 2024-03-18T19:05:43.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.827+0000 7f5425ffb700 1 -- 172.21.15.1:0/2198885661 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5418011990 con 0x7f54280ade00 2024-03-18T19:05:43.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.827+0000 7f5425ffb700 1 -- 172.21.15.1:0/2198885661 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5418011d20 con 0x7f54280ade00 2024-03-18T19:05:43.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.828+0000 7f542d834700 1 -- 172.21.15.1:0/2198885661 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54280ade00 msgr2=0x7f54280cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:43.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.828+0000 7f542d834700 1 --2- 172.21.15.1:0/2198885661 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54280ade00 0x7f54280cc6e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5418009870 tx=0x7f5418009b80 comp rx=0 tx=0).stop 2024-03-18T19:05:43.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.828+0000 7f542d834700 1 -- 172.21.15.1:0/2198885661 shutdown_connections 2024-03-18T19:05:43.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.828+0000 7f542d834700 1 --2- 172.21.15.1:0/2198885661 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54280ade00 0x7f54280cc6e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.828+0000 7f542d834700 1 -- 172.21.15.1:0/2198885661 >> 172.21.15.1:0/2198885661 conn(0x7f54280b2c50 msgr2=0x7f54280b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:43.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.828+0000 7f542d834700 1 -- 172.21.15.1:0/2198885661 shutdown_connections 2024-03-18T19:05:43.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.828+0000 7f542d834700 1 -- 172.21.15.1:0/2198885661 wait complete. 2024-03-18T19:05:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.829+0000 7f542d834700 1 Processor -- start 2024-03-18T19:05:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.829+0000 7f542d834700 1 -- start start 2024-03-18T19:05:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.830+0000 7f542d834700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5428099a40 0x7f54281491e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.830+0000 7f542d834700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54280ade00 0x7f54281f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.830+0000 7f542d834700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54280ccc20 0x7f54280c9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.830+0000 7f542d834700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54280b5840 con 0x7f54280ade00 2024-03-18T19:05:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.830+0000 7f542d834700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54280c8360 con 0x7f5428099a40 2024-03-18T19:05:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.830+0000 7f542d834700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54280c8600 con 0x7f54280ccc20 2024-03-18T19:05:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.830+0000 7f54267fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54280ade00 0x7f54281f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.830+0000 7f54267fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54280ade00 0x7f54281f2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41442/0 (socket says 172.21.15.1:41442) 2024-03-18T19:05:43.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.830+0000 7f54267fc700 1 -- 172.21.15.1:0/3001967255 learned_addr learned my addr 172.21.15.1:0/3001967255 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.830+0000 7f54277fe700 1 --2- 172.21.15.1:0/3001967255 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54280ccc20 0x7f54280c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.830+0000 7f5426ffd700 1 --2- 172.21.15.1:0/3001967255 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5428099a40 0x7f54281491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.830+0000 7f54267fc700 1 -- 172.21.15.1:0/3001967255 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54280ccc20 msgr2=0x7f54280c9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.830+0000 7f54267fc700 1 --2- 172.21.15.1:0/3001967255 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54280ccc20 0x7f54280c9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.830+0000 7f54267fc700 1 -- 172.21.15.1:0/3001967255 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5428099a40 msgr2=0x7f54281491e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.830+0000 7f54267fc700 1 --2- 172.21.15.1:0/3001967255 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5428099a40 0x7f54281491e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.830+0000 7f54267fc700 1 -- 172.21.15.1:0/3001967255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54180094a0 con 0x7f54280ade00 2024-03-18T19:05:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.831+0000 7f54267fc700 1 --2- 172.21.15.1:0/3001967255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54280ade00 0x7f54281f2c10 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f5418014040 tx=0x7f5418004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.831+0000 7f540ffff700 1 -- 172.21.15.1:0/3001967255 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541801a6e0 con 0x7f54280ade00 2024-03-18T19:05:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.831+0000 7f540ffff700 1 -- 172.21.15.1:0/3001967255 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541801a840 con 0x7f54280ade00 2024-03-18T19:05:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.831+0000 7f542d834700 1 -- 172.21.15.1:0/3001967255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54280c88e0 con 0x7f54280ade00 2024-03-18T19:05:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.831+0000 7f540ffff700 1 -- 172.21.15.1:0/3001967255 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541801abb0 con 0x7f54280ade00 2024-03-18T19:05:43.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.831+0000 7f542d834700 1 -- 172.21.15.1:0/3001967255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54280c8ea0 con 0x7f54280ade00 2024-03-18T19:05:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.832+0000 7f542d834700 1 -- 172.21.15.1:0/3001967255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54280c9030 con 0x7f54280ade00 2024-03-18T19:05:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.832+0000 7f540ffff700 1 -- 172.21.15.1:0/3001967255 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 35) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f5418022070 con 0x7f54280ade00 2024-03-18T19:05:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.833+0000 7f540ffff700 1 --2- 172.21.15.1:0/3001967255 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f541406adb0 0x7f541406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.833+0000 7f5426ffd700 1 -- 172.21.15.1:0/3001967255 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f541406adb0 msgr2=0x7f541406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.833+0000 7f540ffff700 1 -- 172.21.15.1:0/3001967255 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f541808a5c0 con 0x7f54280ade00 2024-03-18T19:05:43.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.833+0000 7f5426ffd700 1 --2- 172.21.15.1:0/3001967255 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f541406adb0 0x7f541406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:43.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:43.837+0000 7f540ffff700 1 -- 172.21.15.1:0/3001967255 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f54180555a0 con 0x7f54280ade00 2024-03-18T19:05:43.988 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:43.991+0000 7fddc73eb240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:05:44.006 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:44.010+0000 7f16d3057240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:05:44.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.034+0000 7f5426ffd700 1 -- 172.21.15.1:0/3001967255 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f541406adb0 msgr2=0x7f541406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:44.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.034+0000 7f5426ffd700 1 --2- 172.21.15.1:0/3001967255 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f541406adb0 0x7f541406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:44.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.042+0000 7f542d834700 1 -- 172.21.15.1:0/3001967255 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f54280676a0 con 0x7f54280ade00 2024-03-18T19:05:44.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.042+0000 7f540ffff700 1 -- 172.21.15.1:0/3001967255 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5418029020 con 0x7f54280ade00 2024-03-18T19:05:44.039 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T19:05:44.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.044+0000 7f542d834700 1 -- 172.21.15.1:0/3001967255 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f541406adb0 msgr2=0x7f541406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:44.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.044+0000 7f542d834700 1 --2- 172.21.15.1:0/3001967255 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f541406adb0 0x7f541406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.045+0000 7f542d834700 1 -- 172.21.15.1:0/3001967255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54280ade00 msgr2=0x7f54281f2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:44.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.045+0000 7f542d834700 1 --2- 172.21.15.1:0/3001967255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54280ade00 0x7f54281f2c10 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f5418014040 tx=0x7f5418004750 comp rx=0 tx=0).stop 2024-03-18T19:05:44.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.045+0000 7f542d834700 1 -- 172.21.15.1:0/3001967255 shutdown_connections 2024-03-18T19:05:44.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.045+0000 7f542d834700 1 --2- 172.21.15.1:0/3001967255 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f541406adb0 0x7f541406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.045+0000 7f542d834700 1 --2- 172.21.15.1:0/3001967255 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54280ccc20 0x7f54280c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.045+0000 7f542d834700 1 --2- 172.21.15.1:0/3001967255 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5428099a40 0x7f54281491e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.045+0000 7f542d834700 1 --2- 172.21.15.1:0/3001967255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54280ade00 0x7f54281f2c10 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.045+0000 7f542d834700 1 -- 172.21.15.1:0/3001967255 >> 172.21.15.1:0/3001967255 conn(0x7f54280b2c50 msgr2=0x7f542806cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:44.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.045+0000 7f542d834700 1 -- 172.21.15.1:0/3001967255 shutdown_connections 2024-03-18T19:05:44.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.045+0000 7f542d834700 1 -- 172.21.15.1:0/3001967255 wait complete. 2024-03-18T19:05:44.055 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:05:44.166 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:44.169+0000 7f16d3057240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:05:44.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.188+0000 7f07eb034700 1 Processor -- start 2024-03-18T19:05:44.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.188+0000 7f07eb034700 1 -- start start 2024-03-18T19:05:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.189+0000 7f07eb034700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e409a650 0x7f07e4099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.189+0000 7f07eb034700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f07e4099f80 0x7f07e406f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.189+0000 7f07eb034700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e406f070 con 0x7f07e415b4d0 2024-03-18T19:05:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.189+0000 7f07eb034700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07e409a410 con 0x7f07e4099f80 2024-03-18T19:05:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.189+0000 7f07eb034700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e40d0260 con 0x7f07e409a650 2024-03-18T19:05:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.189+0000 7f07e3fff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e409a650 0x7f07e4099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.189+0000 7f07e95d1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f07e4099f80 0x7f07e406f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.189+0000 7f07e8dd0700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f07e415b4d0 0x7f07e40cf750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:34132/0 (socket says 172.21.15.1:34132) 2024-03-18T19:05:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.189+0000 7f07e95d1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f07e4099f80 0x7f07e406f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:38922/0 (socket says 172.21.15.1:38922) 2024-03-18T19:05:44.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.189+0000 7f07e8dd0700 1 -- 172.21.15.1:0/3191139658 learned_addr learned my addr 172.21.15.1:0/3191139658 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.189+0000 7f07e3fff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e409a650 0x7f07e4099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:60284/0 (socket says 172.21.15.1:60284) 2024-03-18T19:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.190+0000 7f07e3fff700 1 -- 172.21.15.1:0/3191139658 >> v2:172.21.15.165:3300/0 conn(0x7f07e4099f80 msgr2=0x7f07e406f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.190+0000 7f07e3fff700 1 --2- 172.21.15.1:0/3191139658 >> v2:172.21.15.165:3300/0 conn(0x7f07e4099f80 0x7f07e406f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.190+0000 7f07e3fff700 1 -- 172.21.15.1:0/3191139658 >> v1:172.21.15.165:6789/0 conn(0x7f07e415b4d0 legacy=0x7f07e40cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.190+0000 7f07e3fff700 1 -- 172.21.15.1:0/3191139658 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e40cff80 con 0x7f07e409a650 2024-03-18T19:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.190+0000 7f07e95d1700 1 --2- 172.21.15.1:0/3191139658 >> v2:172.21.15.165:3300/0 conn(0x7f07e4099f80 0x7f07e406f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.190+0000 7f07e3fff700 1 --2- 172.21.15.1:0/3191139658 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e409a650 0x7f07e4099a40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f07d0009840 tx=0x7f07d0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a33f4f98960ac039 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.190+0000 7f07e37fe700 1 -- 172.21.15.1:0/3191139658 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d0011830 con 0x7f07e409a650 2024-03-18T19:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.190+0000 7f07e37fe700 1 -- 172.21.15.1:0/3191139658 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d0011990 con 0x7f07e409a650 2024-03-18T19:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.190+0000 7f07e37fe700 1 -- 172.21.15.1:0/3191139658 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d0011d20 con 0x7f07e409a650 2024-03-18T19:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.191+0000 7f07eb034700 1 -- 172.21.15.1:0/3191139658 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e409a650 msgr2=0x7f07e4099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:44.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.191+0000 7f07eb034700 1 --2- 172.21.15.1:0/3191139658 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e409a650 0x7f07e4099a40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f07d0009840 tx=0x7f07d0009b50 comp rx=0 tx=0).stop 2024-03-18T19:05:44.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.191+0000 7f07eb034700 1 -- 172.21.15.1:0/3191139658 shutdown_connections 2024-03-18T19:05:44.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.191+0000 7f07eb034700 1 --2- 172.21.15.1:0/3191139658 >> v2:172.21.15.165:3300/0 conn(0x7f07e4099f80 0x7f07e406f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.191+0000 7f07eb034700 1 --2- 172.21.15.1:0/3191139658 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e409a650 0x7f07e4099a40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.191+0000 7f07eb034700 1 -- 172.21.15.1:0/3191139658 >> 172.21.15.1:0/3191139658 conn(0x7f07e40b2cb0 msgr2=0x7f07e40b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:44.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.191+0000 7f07eb034700 1 -- 172.21.15.1:0/3191139658 shutdown_connections 2024-03-18T19:05:44.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.191+0000 7f07eb034700 1 -- 172.21.15.1:0/3191139658 wait complete. 2024-03-18T19:05:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.192+0000 7f07eb034700 1 Processor -- start 2024-03-18T19:05:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.192+0000 7f07eb034700 1 -- start start 2024-03-18T19:05:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.193+0000 7f07eb034700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e4099f80 0x7f07e40c3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.193+0000 7f07eb034700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e409a650 0x7f07e40b5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.193+0000 7f07eb034700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07e415b4d0 0x7f07e4154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:44.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.193+0000 7f07eb034700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e406bce0 con 0x7f07e4099f80 2024-03-18T19:05:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.193+0000 7f07eb034700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e40b6650 con 0x7f07e415b4d0 2024-03-18T19:05:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.193+0000 7f07eb034700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e406bf80 con 0x7f07e409a650 2024-03-18T19:05:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.193+0000 7f07e8dd0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e4099f80 0x7f07e40c3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.193+0000 7f07e8dd0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e4099f80 0x7f07e40c3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41444/0 (socket says 172.21.15.1:41444) 2024-03-18T19:05:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.193+0000 7f07e8dd0700 1 -- 172.21.15.1:0/875304816 learned_addr learned my addr 172.21.15.1:0/875304816 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.193+0000 7f07e95d1700 1 --2- 172.21.15.1:0/875304816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07e415b4d0 0x7f07e4154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.193+0000 7f07e3fff700 1 --2- 172.21.15.1:0/875304816 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e409a650 0x7f07e40b5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.193+0000 7f07e8dd0700 1 -- 172.21.15.1:0/875304816 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e409a650 msgr2=0x7f07e40b5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.193+0000 7f07e8dd0700 1 --2- 172.21.15.1:0/875304816 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e409a650 0x7f07e40b5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.193+0000 7f07e8dd0700 1 -- 172.21.15.1:0/875304816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07e415b4d0 msgr2=0x7f07e4154420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.193+0000 7f07e8dd0700 1 --2- 172.21.15.1:0/875304816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07e415b4d0 0x7f07e4154420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.193+0000 7f07e8dd0700 1 -- 172.21.15.1:0/875304816 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07dc009a70 con 0x7f07e4099f80 2024-03-18T19:05:44.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.193+0000 7f07e8dd0700 1 --2- 172.21.15.1:0/875304816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e4099f80 0x7f07e40c3cd0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f07d800c430 tx=0x7f07d800c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:44.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.194+0000 7f07e17fa700 1 -- 172.21.15.1:0/875304816 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d8005b40 con 0x7f07e4099f80 2024-03-18T19:05:44.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.194+0000 7f07e17fa700 1 -- 172.21.15.1:0/875304816 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d8005ca0 con 0x7f07e4099f80 2024-03-18T19:05:44.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.194+0000 7f07e17fa700 1 -- 172.21.15.1:0/875304816 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d8007310 con 0x7f07e4099f80 2024-03-18T19:05:44.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.194+0000 7f07eb034700 1 -- 172.21.15.1:0/875304816 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d00094a0 con 0x7f07e4099f80 2024-03-18T19:05:44.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.194+0000 7f07eb034700 1 -- 172.21.15.1:0/875304816 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e406c540 con 0x7f07e4099f80 2024-03-18T19:05:44.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.195+0000 7f07eb034700 1 -- 172.21.15.1:0/875304816 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07e4068240 con 0x7f07e4099f80 2024-03-18T19:05:44.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.195+0000 7f07e17fa700 1 -- 172.21.15.1:0/875304816 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 35) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f07d8007470 con 0x7f07e4099f80 2024-03-18T19:05:44.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.195+0000 7f07e17fa700 1 --2- 172.21.15.1:0/875304816 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f07c806adb0 0x7f07c806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:44.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.196+0000 7f07e3fff700 1 -- 172.21.15.1:0/875304816 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f07c806adb0 msgr2=0x7f07c806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:44.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.196+0000 7f07e17fa700 1 -- 172.21.15.1:0/875304816 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f07d8088aa0 con 0x7f07e4099f80 2024-03-18T19:05:44.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.196+0000 7f07e3fff700 1 --2- 172.21.15.1:0/875304816 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f07c806adb0 0x7f07c806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:44.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.204+0000 7f07e17fa700 1 -- 172.21.15.1:0/875304816 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f07d8053850 con 0x7f07e4099f80 2024-03-18T19:05:44.349 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:44.352+0000 7fddc73eb240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:05:44.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.397+0000 7f07e3fff700 1 -- 172.21.15.1:0/875304816 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f07c806adb0 msgr2=0x7f07c806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:44.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.397+0000 7f07e3fff700 1 --2- 172.21.15.1:0/875304816 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f07c806adb0 0x7f07c806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:44.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.454+0000 7f07eb034700 1 -- 172.21.15.1:0/875304816 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07e40676a0 con 0x7f07e4099f80 2024-03-18T19:05:44.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.455+0000 7f07e17fa700 1 -- 172.21.15.1:0/875304816 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4732 (secure 0 0 0) 0x7f07d8056e70 con 0x7f07e4099f80 2024-03-18T19:05:44.452 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:44.452 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5744,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2437290126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2437290126},{"type":"v1","addr":"172.21.15.1:6836","nonce":2437290126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5747,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2832642785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2832642785},{"type":"v1","addr":"172.21.15.1:6837","nonce":2832642785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5756,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3765527492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3765527492},{"type":"v1","addr":"172.21.15.165:6834","nonce":3765527492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:05:22.577555+0000","modified":"2024-03-18T19:05:25.594597+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5759},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5759":{"gid":5759,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/119895951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":119895951},{"type":"v1","addr":"172.21.15.165:6835","nonce":119895951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-18T19:05:44.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.457+0000 7f07eb034700 1 -- 172.21.15.1:0/875304816 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f07c806adb0 msgr2=0x7f07c806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:44.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.457+0000 7f07eb034700 1 --2- 172.21.15.1:0/875304816 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f07c806adb0 0x7f07c806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.457+0000 7f07eb034700 1 -- 172.21.15.1:0/875304816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e4099f80 msgr2=0x7f07e40c3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:44.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.457+0000 7f07eb034700 1 --2- 172.21.15.1:0/875304816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e4099f80 0x7f07e40c3cd0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f07d800c430 tx=0x7f07d800c740 comp rx=0 tx=0).stop 2024-03-18T19:05:44.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.457+0000 7f07eb034700 1 -- 172.21.15.1:0/875304816 shutdown_connections 2024-03-18T19:05:44.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.457+0000 7f07eb034700 1 --2- 172.21.15.1:0/875304816 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f07c806adb0 0x7f07c806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.457+0000 7f07eb034700 1 --2- 172.21.15.1:0/875304816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07e415b4d0 0x7f07e4154420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.457+0000 7f07eb034700 1 --2- 172.21.15.1:0/875304816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e4099f80 0x7f07e40c3cd0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.457+0000 7f07eb034700 1 --2- 172.21.15.1:0/875304816 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07e409a650 0x7f07e40b5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.457+0000 7f07eb034700 1 -- 172.21.15.1:0/875304816 >> 172.21.15.1:0/875304816 conn(0x7f07e40b2cb0 msgr2=0x7f07e40afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:44.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.458+0000 7f07eb034700 1 -- 172.21.15.1:0/875304816 shutdown_connections 2024-03-18T19:05:44.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.458+0000 7f07eb034700 1 -- 172.21.15.1:0/875304816 wait complete. 2024-03-18T19:05:44.456 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 33 2024-03-18T19:05:44.467 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:05:44.526 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:44.529+0000 7fddc73eb240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:05:44.548 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:44.551+0000 7f16d3057240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:05:44.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.597+0000 7f941d869700 1 Processor -- start 2024-03-18T19:05:44.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.597+0000 7f941d869700 1 -- start start 2024-03-18T19:05:44.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.599+0000 7f941d869700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f941815a320 0x7f94180ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:44.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.599+0000 7f941d869700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94180cea30 0x7f9418071600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:44.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.599+0000 7f941d869700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94180acd80 con 0x7f9418071b40 2024-03-18T19:05:44.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.599+0000 7f941d869700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9418071ff0 con 0x7f941815a320 2024-03-18T19:05:44.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.599+0000 7f941d869700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94180cf220 con 0x7f94180cea30 2024-03-18T19:05:44.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.599+0000 7f9416ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f941815a320 0x7f94180ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:44.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.599+0000 7f9416ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f941815a320 0x7f94180ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36780/0 (socket says 172.21.15.1:36780) 2024-03-18T19:05:44.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.599+0000 7f9416ffd700 1 -- 172.21.15.1:0/3245659699 learned_addr learned my addr 172.21.15.1:0/3245659699 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:44.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.599+0000 7f94177fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9418071b40 0x7f94180cad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55082/0 (socket says 172.21.15.1:55082) 2024-03-18T19:05:44.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.599+0000 7f94167fc700 1 --2- 172.21.15.1:0/3245659699 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94180cea30 0x7f9418071600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:44.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.599+0000 7f9416ffd700 1 -- 172.21.15.1:0/3245659699 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94180cea30 msgr2=0x7f9418071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:44.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.599+0000 7f9416ffd700 1 --2- 172.21.15.1:0/3245659699 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94180cea30 0x7f9418071600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.599+0000 7f9416ffd700 1 -- 172.21.15.1:0/3245659699 >> v1:172.21.15.1:6789/0 conn(0x7f9418071b40 legacy=0x7f94180cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:44.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.599+0000 7f9416ffd700 1 -- 172.21.15.1:0/3245659699 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94180cef40 con 0x7f941815a320 2024-03-18T19:05:44.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.599+0000 7f9415ffb700 1 -- 172.21.15.1:0/3245659699 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423060452 0 0) 0x7f94180acd80 con 0x7f9418071b40 2024-03-18T19:05:44.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.600+0000 7f9416ffd700 1 --2- 172.21.15.1:0/3245659699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f941815a320 0x7f94180ce4f0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f9408009920 tx=0x7f9408009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34b173a18977ca88 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:44.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.600+0000 7f9415ffb700 1 -- 172.21.15.1:0/3245659699 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408011830 con 0x7f941815a320 2024-03-18T19:05:44.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.600+0000 7f9415ffb700 1 -- 172.21.15.1:0/3245659699 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9408011990 con 0x7f941815a320 2024-03-18T19:05:44.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.600+0000 7f9415ffb700 1 -- 172.21.15.1:0/3245659699 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408011d20 con 0x7f941815a320 2024-03-18T19:05:44.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.601+0000 7f941d869700 1 -- 172.21.15.1:0/3245659699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f941815a320 msgr2=0x7f94180ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:44.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.601+0000 7f941d869700 1 --2- 172.21.15.1:0/3245659699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f941815a320 0x7f94180ce4f0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f9408009920 tx=0x7f9408009c30 comp rx=0 tx=0).stop 2024-03-18T19:05:44.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.601+0000 7f941d869700 1 -- 172.21.15.1:0/3245659699 shutdown_connections 2024-03-18T19:05:44.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.601+0000 7f941d869700 1 --2- 172.21.15.1:0/3245659699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f941815a320 0x7f94180ce4f0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.601+0000 7f941d869700 1 --2- 172.21.15.1:0/3245659699 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94180cea30 0x7f9418071600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.601+0000 7f941d869700 1 -- 172.21.15.1:0/3245659699 >> 172.21.15.1:0/3245659699 conn(0x7f94180cc990 msgr2=0x7f94180ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:44.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.601+0000 7f941d869700 1 -- 172.21.15.1:0/3245659699 shutdown_connections 2024-03-18T19:05:44.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.601+0000 7f941d869700 1 -- 172.21.15.1:0/3245659699 wait complete. 2024-03-18T19:05:44.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.602+0000 7f941d869700 1 Processor -- start 2024-03-18T19:05:44.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.602+0000 7f941d869700 1 -- start start 2024-03-18T19:05:44.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.603+0000 7f941d869700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9418071b40 0x7f94180c66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:44.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.603+0000 7f941d869700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94180cea30 0x7f941814a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.603+0000 7f941d869700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f941814f470 0x7f941815c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.603+0000 7f941d869700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94180b5ca0 con 0x7f9418071b40 2024-03-18T19:05:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.603+0000 7f941d869700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941814acb0 con 0x7f94180cea30 2024-03-18T19:05:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.603+0000 7f941d869700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94180b5810 con 0x7f941814f470 2024-03-18T19:05:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.603+0000 7f9416ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9418071b40 0x7f94180c66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.603+0000 7f9416ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9418071b40 0x7f94180c66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36790/0 (socket says 172.21.15.1:36790) 2024-03-18T19:05:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.603+0000 7f9416ffd700 1 -- 172.21.15.1:0/138937109 learned_addr learned my addr 172.21.15.1:0/138937109 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.603+0000 7f94167fc700 1 --2- 172.21.15.1:0/138937109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94180cea30 0x7f941814a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.603+0000 7f94177fe700 1 --2- 172.21.15.1:0/138937109 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f941814f470 0x7f941815c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.603+0000 7f9416ffd700 1 -- 172.21.15.1:0/138937109 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f941814f470 msgr2=0x7f941815c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.603+0000 7f9416ffd700 1 --2- 172.21.15.1:0/138937109 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f941814f470 0x7f941815c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.603+0000 7f9416ffd700 1 -- 172.21.15.1:0/138937109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94180cea30 msgr2=0x7f941814a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.603+0000 7f9416ffd700 1 --2- 172.21.15.1:0/138937109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94180cea30 0x7f941814a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.603+0000 7f9416ffd700 1 -- 172.21.15.1:0/138937109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9408009550 con 0x7f9418071b40 2024-03-18T19:05:44.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.603+0000 7f9416ffd700 1 --2- 172.21.15.1:0/138937109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9418071b40 0x7f94180c66d0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f9408009c70 tx=0x7f9408004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:44.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.604+0000 7f93fffff700 1 -- 172.21.15.1:0/138937109 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408004300 con 0x7f9418071b40 2024-03-18T19:05:44.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.604+0000 7f93fffff700 1 -- 172.21.15.1:0/138937109 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940801f070 con 0x7f9418071b40 2024-03-18T19:05:44.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.604+0000 7f941d869700 1 -- 172.21.15.1:0/138937109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94180b4f50 con 0x7f9418071b40 2024-03-18T19:05:44.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.604+0000 7f93fffff700 1 -- 172.21.15.1:0/138937109 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94080036a0 con 0x7f9418071b40 2024-03-18T19:05:44.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.604+0000 7f941d869700 1 -- 172.21.15.1:0/138937109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94180b54a0 con 0x7f9418071b40 2024-03-18T19:05:44.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.605+0000 7f941d869700 1 -- 172.21.15.1:0/138937109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94180b5660 con 0x7f9418071b40 2024-03-18T19:05:44.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.605+0000 7f93fffff700 1 -- 172.21.15.1:0/138937109 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 35) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f9408021030 con 0x7f9418071b40 2024-03-18T19:05:44.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.606+0000 7f93fffff700 1 --2- 172.21.15.1:0/138937109 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f940406adb0 0x7f940406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:44.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.606+0000 7f94167fc700 1 -- 172.21.15.1:0/138937109 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f940406adb0 msgr2=0x7f940406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:44.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.606+0000 7f93fffff700 1 -- 172.21.15.1:0/138937109 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f940808a880 con 0x7f9418071b40 2024-03-18T19:05:44.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.606+0000 7f94167fc700 1 --2- 172.21.15.1:0/138937109 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f940406adb0 0x7f940406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:44.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.610+0000 7f93fffff700 1 -- 172.21.15.1:0/138937109 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9408055860 con 0x7f9418071b40 2024-03-18T19:05:44.710 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:44.714+0000 7f16d3057240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:05:44.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.807+0000 7f94167fc700 1 -- 172.21.15.1:0/138937109 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f940406adb0 msgr2=0x7f940406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:44.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.807+0000 7f94167fc700 1 --2- 172.21.15.1:0/138937109 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f940406adb0 0x7f940406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:44.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.818+0000 7f941d869700 1 -- 172.21.15.1:0/138937109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94180b5af0 con 0x7f9418071b40 2024-03-18T19:05:44.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.819+0000 7f93fffff700 1 -- 172.21.15.1:0/138937109 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+16681 (secure 0 0 0) 0x7f9408015090 con 0x7f9418071b40 2024-03-18T19:05:44.816 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:44.816 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":90,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:05:37.779828+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:05:20.423723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-18T19:05:21.600285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2546947071":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/3332271581":"2024-03-19T19:05:37.779786+0000","172.21.15.1:6833/76998":"2024-03-19T19:05:37.779786+0000","172.21.15.1:6832/76998":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/1890712448":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/300431708":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/4106216506":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/569963265":"2024-03-19T19:05:37.779786+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:05:44.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.821+0000 7f941d869700 1 -- 172.21.15.1:0/138937109 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f940406adb0 msgr2=0x7f940406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:44.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.821+0000 7f941d869700 1 --2- 172.21.15.1:0/138937109 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f940406adb0 0x7f940406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.821+0000 7f941d869700 1 -- 172.21.15.1:0/138937109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9418071b40 msgr2=0x7f94180c66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:44.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.821+0000 7f941d869700 1 --2- 172.21.15.1:0/138937109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9418071b40 0x7f94180c66d0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f9408009c70 tx=0x7f9408004750 comp rx=0 tx=0).stop 2024-03-18T19:05:44.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.821+0000 7f941d869700 1 -- 172.21.15.1:0/138937109 shutdown_connections 2024-03-18T19:05:44.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.821+0000 7f941d869700 1 --2- 172.21.15.1:0/138937109 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f940406adb0 0x7f940406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.821+0000 7f941d869700 1 --2- 172.21.15.1:0/138937109 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f941814f470 0x7f941815c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.821+0000 7f941d869700 1 --2- 172.21.15.1:0/138937109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9418071b40 0x7f94180c66d0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.821+0000 7f941d869700 1 --2- 172.21.15.1:0/138937109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94180cea30 0x7f941814a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.821+0000 7f941d869700 1 -- 172.21.15.1:0/138937109 >> 172.21.15.1:0/138937109 conn(0x7f94180cc990 msgr2=0x7f941806aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:44.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.822+0000 7f941d869700 1 -- 172.21.15.1:0/138937109 shutdown_connections 2024-03-18T19:05:44.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.822+0000 7f941d869700 1 -- 172.21.15.1:0/138937109 wait complete. 2024-03-18T19:05:44.831 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T19:05:44.832 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:05:44.950 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:44.954+0000 7fddc73eb240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:05:44.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.960+0000 7f02f9f1a700 1 Processor -- start 2024-03-18T19:05:44.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.960+0000 7f02f9f1a700 1 -- start start 2024-03-18T19:05:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.961+0000 7f02f9f1a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02f40d17b0 0x7f02f40d1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.961+0000 7f02f9f1a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f02f40b83f0 0x7f02f40b8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.961+0000 7f02f9f1a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f40cddc0 con 0x7f02f415d500 2024-03-18T19:05:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.961+0000 7f02f9f1a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02f40d21a0 con 0x7f02f40b83f0 2024-03-18T19:05:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.961+0000 7f02f9f1a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f40d3dc0 con 0x7f02f40d17b0 2024-03-18T19:05:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.961+0000 7f02f37fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f02f415d500 0x7f02f40d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55092/0 (socket says 172.21.15.1:55092) 2024-03-18T19:05:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.961+0000 7f02f37fe700 1 -- 172.21.15.1:0/762857611 learned_addr learned my addr 172.21.15.1:0/762857611 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.961+0000 7f02f3fff700 1 --2- 172.21.15.1:0/762857611 >> v2:172.21.15.165:3300/0 conn(0x7f02f40b83f0 0x7f02f40b8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.961+0000 7f02f2ffd700 1 --2- 172.21.15.1:0/762857611 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02f40d17b0 0x7f02f40d1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:44.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.961+0000 7f02f27fc700 1 -- 172.21.15.1:0/762857611 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956369985 0 0) 0x7f02f40cddc0 con 0x7f02f415d500 2024-03-18T19:05:44.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.961+0000 7f02f27fc700 1 -- 172.21.15.1:0/762857611 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e00035b0 con 0x7f02f415d500 2024-03-18T19:05:44.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.962+0000 7f02f2ffd700 1 -- 172.21.15.1:0/762857611 >> v2:172.21.15.165:3300/0 conn(0x7f02f40b83f0 msgr2=0x7f02f40b8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:44.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.962+0000 7f02f2ffd700 1 --2- 172.21.15.1:0/762857611 >> v2:172.21.15.165:3300/0 conn(0x7f02f40b83f0 0x7f02f40b8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.962+0000 7f02f2ffd700 1 -- 172.21.15.1:0/762857611 >> v1:172.21.15.1:6789/0 conn(0x7f02f415d500 legacy=0x7f02f40d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:44.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.962+0000 7f02f2ffd700 1 -- 172.21.15.1:0/762857611 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f40ce060 con 0x7f02f40d17b0 2024-03-18T19:05:44.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.962+0000 7f02f2ffd700 1 --2- 172.21.15.1:0/762857611 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02f40d17b0 0x7f02f40d1c60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009840 tx=0x7f02dc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ee3b009b0e7cb4b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:44.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.962+0000 7f02f27fc700 1 -- 172.21.15.1:0/762857611 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02dc011830 con 0x7f02f40d17b0 2024-03-18T19:05:44.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.962+0000 7f02f27fc700 1 -- 172.21.15.1:0/762857611 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02dc011990 con 0x7f02f40d17b0 2024-03-18T19:05:44.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.962+0000 7f02f27fc700 1 -- 172.21.15.1:0/762857611 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02dc011d20 con 0x7f02f40d17b0 2024-03-18T19:05:44.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.963+0000 7f02f9f1a700 1 -- 172.21.15.1:0/762857611 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02f40d17b0 msgr2=0x7f02f40d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:44.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.963+0000 7f02f9f1a700 1 --2- 172.21.15.1:0/762857611 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02f40d17b0 0x7f02f40d1c60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009840 tx=0x7f02dc009b50 comp rx=0 tx=0).stop 2024-03-18T19:05:44.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.963+0000 7f02f9f1a700 1 -- 172.21.15.1:0/762857611 shutdown_connections 2024-03-18T19:05:44.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.963+0000 7f02f9f1a700 1 --2- 172.21.15.1:0/762857611 >> v2:172.21.15.165:3300/0 conn(0x7f02f40b83f0 0x7f02f40b8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.963+0000 7f02f9f1a700 1 --2- 172.21.15.1:0/762857611 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02f40d17b0 0x7f02f40d1c60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.963+0000 7f02f9f1a700 1 -- 172.21.15.1:0/762857611 >> 172.21.15.1:0/762857611 conn(0x7f02f409c530 msgr2=0x7f02f40d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:44.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.963+0000 7f02f9f1a700 1 -- 172.21.15.1:0/762857611 shutdown_connections 2024-03-18T19:05:44.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.963+0000 7f02f9f1a700 1 -- 172.21.15.1:0/762857611 wait complete. 2024-03-18T19:05:44.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.964+0000 7f02f9f1a700 1 Processor -- start 2024-03-18T19:05:44.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.964+0000 7f02f9f1a700 1 -- start start 2024-03-18T19:05:44.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.965+0000 7f02f9f1a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02f40b83f0 0x7f02f4154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:44.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.965+0000 7f02f9f1a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02f40d17b0 0x7f02f40cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:44.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.965+0000 7f02f9f1a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02f415d500 0x7f02f40b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:44.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.965+0000 7f02f9f1a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f4154c50 con 0x7f02f415d500 2024-03-18T19:05:44.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.965+0000 7f02f9f1a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f40cb790 con 0x7f02f40b83f0 2024-03-18T19:05:44.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.965+0000 7f02f9f1a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f40cba70 con 0x7f02f40d17b0 2024-03-18T19:05:44.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.965+0000 7f02f3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02f415d500 0x7f02f40b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:44.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.965+0000 7f02f3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02f415d500 0x7f02f40b6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36804/0 (socket says 172.21.15.1:36804) 2024-03-18T19:05:44.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.965+0000 7f02f3fff700 1 -- 172.21.15.1:0/3038595724 learned_addr learned my addr 172.21.15.1:0/3038595724 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:44.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.965+0000 7f02f37fe700 1 --2- 172.21.15.1:0/3038595724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02f40b83f0 0x7f02f4154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:44.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.965+0000 7f02f2ffd700 1 --2- 172.21.15.1:0/3038595724 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02f40d17b0 0x7f02f40cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:44.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.965+0000 7f02f3fff700 1 -- 172.21.15.1:0/3038595724 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02f40d17b0 msgr2=0x7f02f40cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:44.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.965+0000 7f02f3fff700 1 --2- 172.21.15.1:0/3038595724 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02f40d17b0 0x7f02f40cacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.965+0000 7f02f3fff700 1 -- 172.21.15.1:0/3038595724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02f40b83f0 msgr2=0x7f02f4154710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:44.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.965+0000 7f02f3fff700 1 --2- 172.21.15.1:0/3038595724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02f40b83f0 0x7f02f4154710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:44.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.965+0000 7f02f3fff700 1 -- 172.21.15.1:0/3038595724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02dc0094a0 con 0x7f02f415d500 2024-03-18T19:05:44.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.965+0000 7f02f3fff700 1 --2- 172.21.15.1:0/3038595724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02f415d500 0x7f02f40b6f90 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f02e400efc0 tx=0x7f02e400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:44.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.966+0000 7f02ebfff700 1 -- 172.21.15.1:0/3038595724 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e4009ea0 con 0x7f02f415d500 2024-03-18T19:05:44.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.966+0000 7f02ebfff700 1 -- 172.21.15.1:0/3038595724 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e400f040 con 0x7f02f415d500 2024-03-18T19:05:44.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.966+0000 7f02f9f1a700 1 -- 172.21.15.1:0/3038595724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f40cbd50 con 0x7f02f415d500 2024-03-18T19:05:44.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.966+0000 7f02ebfff700 1 -- 172.21.15.1:0/3038595724 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e400ced0 con 0x7f02f415d500 2024-03-18T19:05:44.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.966+0000 7f02f9f1a700 1 -- 172.21.15.1:0/3038595724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f41e8900 con 0x7f02f415d500 2024-03-18T19:05:44.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.967+0000 7f02f9f1a700 1 -- 172.21.15.1:0/3038595724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f40cbf90 con 0x7f02f415d500 2024-03-18T19:05:44.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.968+0000 7f02ebfff700 1 -- 172.21.15.1:0/3038595724 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 35) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f02e4014480 con 0x7f02f415d500 2024-03-18T19:05:44.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.968+0000 7f02ebfff700 1 --2- 172.21.15.1:0/3038595724 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f02d406adb0 0x7f02d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:44.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.968+0000 7f02f37fe700 1 -- 172.21.15.1:0/3038595724 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f02d406adb0 msgr2=0x7f02d406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:44.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.968+0000 7f02ebfff700 1 -- 172.21.15.1:0/3038595724 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f02e408a040 con 0x7f02f415d500 2024-03-18T19:05:44.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.968+0000 7f02f37fe700 1 --2- 172.21.15.1:0/3038595724 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f02d406adb0 0x7f02d406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:44.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:44.973+0000 7f02ebfff700 1 -- 172.21.15.1:0/3038595724 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f02f40cbf90 con 0x7f02f415d500 2024-03-18T19:05:45.071 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:45.074+0000 7f16d3057240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:05:45.128 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:45.131+0000 7fddc73eb240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:05:45.157 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:05:45.161+0000 7f16d3057240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:05:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.169+0000 7f02f37fe700 1 -- 172.21.15.1:0/3038595724 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f02d406adb0 msgr2=0x7f02d406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:45.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.169+0000 7f02f37fe700 1 --2- 172.21.15.1:0/3038595724 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f02d406adb0 0x7f02d406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:45.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.225+0000 7f02f9f1a700 1 -- 172.21.15.1:0/3038595724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02f40cbf90 con 0x7f02f415d500 2024-03-18T19:05:45.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.225+0000 7f02ebfff700 1 -- 172.21.15.1:0/3038595724 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4732 (secure 0 0 0) 0x7f02f40cbf90 con 0x7f02f415d500 2024-03-18T19:05:45.223 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:45.223 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5744,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2437290126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2437290126},{"type":"v1","addr":"172.21.15.1:6836","nonce":2437290126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5747,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2832642785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2832642785},{"type":"v1","addr":"172.21.15.1:6837","nonce":2832642785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5756,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3765527492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3765527492},{"type":"v1","addr":"172.21.15.165:6834","nonce":3765527492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:05:22.577555+0000","modified":"2024-03-18T19:05:25.594597+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5759},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5759":{"gid":5759,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/119895951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":119895951},{"type":"v1","addr":"172.21.15.165:6835","nonce":119895951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-18T19:05:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.228+0000 7f02f9f1a700 1 -- 172.21.15.1:0/3038595724 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f02d406adb0 msgr2=0x7f02d406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.228+0000 7f02f9f1a700 1 --2- 172.21.15.1:0/3038595724 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f02d406adb0 0x7f02d406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.228+0000 7f02f9f1a700 1 -- 172.21.15.1:0/3038595724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02f415d500 msgr2=0x7f02f40b6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.228+0000 7f02f9f1a700 1 --2- 172.21.15.1:0/3038595724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02f415d500 0x7f02f40b6f90 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f02e400efc0 tx=0x7f02e400c3a0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.228+0000 7f02f9f1a700 1 -- 172.21.15.1:0/3038595724 shutdown_connections 2024-03-18T19:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.228+0000 7f02f9f1a700 1 --2- 172.21.15.1:0/3038595724 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f02d406adb0 0x7f02d406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.228+0000 7f02f9f1a700 1 --2- 172.21.15.1:0/3038595724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02f415d500 0x7f02f40b6f90 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.228+0000 7f02f9f1a700 1 --2- 172.21.15.1:0/3038595724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02f40b83f0 0x7f02f4154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.228+0000 7f02f9f1a700 1 --2- 172.21.15.1:0/3038595724 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02f40d17b0 0x7f02f40cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.228+0000 7f02f9f1a700 1 -- 172.21.15.1:0/3038595724 >> 172.21.15.1:0/3038595724 conn(0x7f02f409c530 msgr2=0x7f02f40cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.228+0000 7f02f9f1a700 1 -- 172.21.15.1:0/3038595724 shutdown_connections 2024-03-18T19:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.228+0000 7f02f9f1a700 1 -- 172.21.15.1:0/3038595724 wait complete. 2024-03-18T19:05:45.226 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 33 2024-03-18T19:05:45.238 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:05:45.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.373+0000 7effeed59700 1 Processor -- start 2024-03-18T19:05:45.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.373+0000 7effeed59700 1 -- start start 2024-03-18T19:05:45.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.373+0000 7effeed59700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7effe815b120 0x7effe8072010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:45.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.373+0000 7effeed59700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effe815b500 0x7effe806a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:45.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.374+0000 7effeed59700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe806b0e0 con 0x7effe806ac90 2024-03-18T19:05:45.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.374+0000 7effeed59700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effe806ffc0 con 0x7effe815b120 2024-03-18T19:05:45.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.374+0000 7effeed59700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe806b520 con 0x7effe815b500 2024-03-18T19:05:45.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.374+0000 7effed2f6700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7effe806ac90 0x7effe815eee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55100/0 (socket says 172.21.15.1:55100) 2024-03-18T19:05:45.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.374+0000 7effed2f6700 1 -- 172.21.15.1:0/908536424 learned_addr learned my addr 172.21.15.1:0/908536424 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:45.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.374+0000 7effecaf5700 1 --2- 172.21.15.1:0/908536424 >> v2:172.21.15.165:3300/0 conn(0x7effe815b120 0x7effe8072010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:45.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.374+0000 7effe7fff700 1 --2- 172.21.15.1:0/908536424 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effe815b500 0x7effe806a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:45.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.374+0000 7effe77fe700 1 -- 172.21.15.1:0/908536424 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252264521 0 0) 0x7effe0001490 con 0x7effe806ac90 2024-03-18T19:05:45.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.374+0000 7effe77fe700 1 -- 172.21.15.1:0/908536424 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effd80035b0 con 0x7effe806ac90 2024-03-18T19:05:45.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.374+0000 7effecaf5700 1 -- 172.21.15.1:0/908536424 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effe815b500 msgr2=0x7effe806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:45.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.374+0000 7effecaf5700 1 --2- 172.21.15.1:0/908536424 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effe815b500 0x7effe806a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.374+0000 7effecaf5700 1 -- 172.21.15.1:0/908536424 >> v1:172.21.15.1:6789/0 conn(0x7effe806ac90 legacy=0x7effe815eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:45.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.374+0000 7effecaf5700 1 -- 172.21.15.1:0/908536424 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe806ef20 con 0x7effe815b120 2024-03-18T19:05:45.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.374+0000 7effecaf5700 1 --2- 172.21.15.1:0/908536424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effe815b120 0x7effe8072010 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7effdc009840 tx=0x7effdc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=722c56be31360a47 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:45.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.375+0000 7effe77fe700 1 -- 172.21.15.1:0/908536424 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effdc011830 con 0x7effe815b120 2024-03-18T19:05:45.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.375+0000 7effe77fe700 1 -- 172.21.15.1:0/908536424 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effdc011990 con 0x7effe815b120 2024-03-18T19:05:45.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.375+0000 7effe77fe700 1 -- 172.21.15.1:0/908536424 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effdc011d20 con 0x7effe815b120 2024-03-18T19:05:45.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.375+0000 7effeed59700 1 -- 172.21.15.1:0/908536424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effe815b120 msgr2=0x7effe8072010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:45.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.375+0000 7effeed59700 1 --2- 172.21.15.1:0/908536424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effe815b120 0x7effe8072010 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7effdc009840 tx=0x7effdc009b50 comp rx=0 tx=0).stop 2024-03-18T19:05:45.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.375+0000 7effeed59700 1 -- 172.21.15.1:0/908536424 shutdown_connections 2024-03-18T19:05:45.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.375+0000 7effeed59700 1 --2- 172.21.15.1:0/908536424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effe815b120 0x7effe8072010 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.375+0000 7effeed59700 1 --2- 172.21.15.1:0/908536424 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effe815b500 0x7effe806a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.375+0000 7effeed59700 1 -- 172.21.15.1:0/908536424 >> 172.21.15.1:0/908536424 conn(0x7effe80ccab0 msgr2=0x7effe80ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:45.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.375+0000 7effeed59700 1 -- 172.21.15.1:0/908536424 shutdown_connections 2024-03-18T19:05:45.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.375+0000 7effeed59700 1 -- 172.21.15.1:0/908536424 wait complete. 2024-03-18T19:05:45.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.377+0000 7effeed59700 1 Processor -- start 2024-03-18T19:05:45.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.377+0000 7effeed59700 1 -- start start 2024-03-18T19:05:45.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.378+0000 7effeed59700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effe806ac90 0x7effe80cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:45.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.378+0000 7effeed59700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effe815b120 0x7effe81570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:45.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.378+0000 7effeed59700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effe815b500 0x7effe80ca660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:45.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.378+0000 7effeed59700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe81e6370 con 0x7effe815b120 2024-03-18T19:05:45.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.378+0000 7effeed59700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe81e71d0 con 0x7effe815b500 2024-03-18T19:05:45.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.378+0000 7effeed59700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe81e65c0 con 0x7effe806ac90 2024-03-18T19:05:45.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.378+0000 7effe7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effe815b120 0x7effe81570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:45.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.378+0000 7effe7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effe815b120 0x7effe81570e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36814/0 (socket says 172.21.15.1:36814) 2024-03-18T19:05:45.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.378+0000 7effe7fff700 1 -- 172.21.15.1:0/248307857 learned_addr learned my addr 172.21.15.1:0/248307857 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:45.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.378+0000 7effecaf5700 1 --2- 172.21.15.1:0/248307857 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effe806ac90 0x7effe80cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:45.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.378+0000 7effed2f6700 1 --2- 172.21.15.1:0/248307857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effe815b500 0x7effe80ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:45.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.379+0000 7effe7fff700 1 -- 172.21.15.1:0/248307857 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effe806ac90 msgr2=0x7effe80cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:45.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.379+0000 7effe7fff700 1 --2- 172.21.15.1:0/248307857 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effe806ac90 0x7effe80cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.379+0000 7effe7fff700 1 -- 172.21.15.1:0/248307857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effe815b500 msgr2=0x7effe80ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:45.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.379+0000 7effe7fff700 1 --2- 172.21.15.1:0/248307857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effe815b500 0x7effe80ca660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.379+0000 7effe7fff700 1 -- 172.21.15.1:0/248307857 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc0094a0 con 0x7effe815b120 2024-03-18T19:05:45.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.379+0000 7effe7fff700 1 --2- 172.21.15.1:0/248307857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effe815b120 0x7effe81570e0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7effd400e870 tx=0x7effd400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:45.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.379+0000 7effe57fa700 1 -- 172.21.15.1:0/248307857 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4009590 con 0x7effe815b120 2024-03-18T19:05:45.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.379+0000 7effe57fa700 1 -- 172.21.15.1:0/248307857 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd4013680 con 0x7effe815b120 2024-03-18T19:05:45.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.379+0000 7effeed59700 1 -- 172.21.15.1:0/248307857 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe81e68a0 con 0x7effe815b120 2024-03-18T19:05:45.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.379+0000 7effe57fa700 1 -- 172.21.15.1:0/248307857 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd40139f0 con 0x7effe815b120 2024-03-18T19:05:45.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.379+0000 7effeed59700 1 -- 172.21.15.1:0/248307857 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe81e6e60 con 0x7effe815b120 2024-03-18T19:05:45.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.380+0000 7effeed59700 1 -- 172.21.15.1:0/248307857 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe81e7340 con 0x7effe815b120 2024-03-18T19:05:45.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.381+0000 7effe57fa700 1 -- 172.21.15.1:0/248307857 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 35) v1 ==== 89240+0+0 (secure 0 0 0) 0x7effd4013b50 con 0x7effe815b120 2024-03-18T19:05:45.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.381+0000 7effe57fa700 1 --2- 172.21.15.1:0/248307857 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effcc06adb0 0x7effcc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:45.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.381+0000 7effecaf5700 1 -- 172.21.15.1:0/248307857 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effcc06adb0 msgr2=0x7effcc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:45.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.381+0000 7effe57fa700 1 -- 172.21.15.1:0/248307857 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6373+0+0 (secure 0 0 0) 0x7effd4089560 con 0x7effe815b120 2024-03-18T19:05:45.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.381+0000 7effecaf5700 1 --2- 172.21.15.1:0/248307857 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effcc06adb0 0x7effcc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:05:45.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.386+0000 7effe57fa700 1 -- 172.21.15.1:0/248307857 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7effd4054390 con 0x7effe815b120 2024-03-18T19:05:45.512 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:45.515+0000 7fddc73eb240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:05:45.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.582+0000 7effecaf5700 1 -- 172.21.15.1:0/248307857 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effcc06adb0 msgr2=0x7effcc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:05:45.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.582+0000 7effecaf5700 1 --2- 172.21.15.1:0/248307857 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effcc06adb0 0x7effcc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:05:45.606 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:45.609+0000 7fddc73eb240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:05:45.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.629+0000 7effe57fa700 1 -- 172.21.15.1:0/248307857 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 36) v1 ==== 88848+0+0 (secure 0 0 0) 0x7effd40877e0 con 0x7effe815b120 2024-03-18T19:05:45.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.629+0000 7effe57fa700 1 -- 172.21.15.1:0/248307857 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effcc06adb0 msgr2=0x7effcc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:05:45.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.629+0000 7effe57fa700 1 --2- 172.21.15.1:0/248307857 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effcc06adb0 0x7effcc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.641+0000 7effeed59700 1 -- 172.21.15.1:0/248307857 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effe80676a0 con 0x7effe815b120 2024-03-18T19:05:45.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.642+0000 7effe57fa700 1 -- 172.21.15.1:0/248307857 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4732 (secure 0 0 0) 0x7effd4086de0 con 0x7effe815b120 2024-03-18T19:05:45.639 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:45.639 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5744,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2437290126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2437290126},{"type":"v1","addr":"172.21.15.1:6836","nonce":2437290126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5747,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2832642785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2832642785},{"type":"v1","addr":"172.21.15.1:6837","nonce":2832642785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5756,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3765527492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3765527492},{"type":"v1","addr":"172.21.15.165:6834","nonce":3765527492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:05:22.577555+0000","modified":"2024-03-18T19:05:25.594597+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5759},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5759":{"gid":5759,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/119895951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":119895951},{"type":"v1","addr":"172.21.15.165:6835","nonce":119895951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-18T19:05:45.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.644+0000 7effeed59700 1 -- 172.21.15.1:0/248307857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effe815b120 msgr2=0x7effe81570e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:45.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.644+0000 7effeed59700 1 --2- 172.21.15.1:0/248307857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effe815b120 0x7effe81570e0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7effd400e870 tx=0x7effd400eb80 comp rx=0 tx=0).stop 2024-03-18T19:05:45.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.645+0000 7effeed59700 1 -- 172.21.15.1:0/248307857 shutdown_connections 2024-03-18T19:05:45.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.645+0000 7effeed59700 1 --2- 172.21.15.1:0/248307857 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effcc06adb0 0x7effcc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.645+0000 7effeed59700 1 --2- 172.21.15.1:0/248307857 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effe806ac90 0x7effe80cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.645+0000 7effeed59700 1 --2- 172.21.15.1:0/248307857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effe815b500 0x7effe80ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.645+0000 7effeed59700 1 --2- 172.21.15.1:0/248307857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effe815b120 0x7effe81570e0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.645+0000 7effeed59700 1 -- 172.21.15.1:0/248307857 >> 172.21.15.1:0/248307857 conn(0x7effe80ccab0 msgr2=0x7effe8159fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:45.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.645+0000 7effeed59700 1 -- 172.21.15.1:0/248307857 shutdown_connections 2024-03-18T19:05:45.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.645+0000 7effeed59700 1 -- 172.21.15.1:0/248307857 wait complete. 2024-03-18T19:05:45.643 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 33 2024-03-18T19:05:45.654 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:05:45.696 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:45.699+0000 7fdd3e010700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:45.696 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:45.699+0000 7fdd3e010700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:45.696 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:45.699+0000 7fdd3e010700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:45.696 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:45.699+0000 7fdd3e010700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:45.696 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:45.699+0000 7fdd3e010700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:45.696 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:45.699+0000 7fdd42018700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:45.696 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:45.699+0000 7fdd42018700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:45.696 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:45.699+0000 7fdd42018700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:45.696 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:45.699+0000 7fdd42018700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:45.696 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:05:45.699+0000 7fdd42018700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:05:45.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.787+0000 7f5cba327700 1 Processor -- start 2024-03-18T19:05:45.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.787+0000 7f5cba327700 1 -- start start 2024-03-18T19:05:45.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.788+0000 7f5cba327700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5cb40cf090 0x7f5cb40d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:45.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.788+0000 7f5cba327700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5cb4160d20 0x7f5cb40d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:45.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.788+0000 7f5cba327700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb40cddc0 con 0x7f5cb4160940 2024-03-18T19:05:45.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.788+0000 7f5cba327700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cb40cdff0 con 0x7f5cb40cf090 2024-03-18T19:05:45.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.788+0000 7f5cba327700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb40ce2d0 con 0x7f5cb4160d20 2024-03-18T19:05:45.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.788+0000 7f5cb37fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5cb40cf090 0x7f5cb40d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:45.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.788+0000 7f5cb3fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5cb4160940 0x7f5cb40d0190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55104/0 (socket says 172.21.15.1:55104) 2024-03-18T19:05:45.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.788+0000 7f5cb3fff700 1 -- 172.21.15.1:0/669553958 learned_addr learned my addr 172.21.15.1:0/669553958 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:45.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.789+0000 7f5cb88c4700 1 --2- 172.21.15.1:0/669553958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5cb4160d20 0x7f5cb40d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:45.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.789+0000 7f5cb37fe700 1 -- 172.21.15.1:0/669553958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5cb4160d20 msgr2=0x7f5cb40d1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:45.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.789+0000 7f5cb37fe700 1 --2- 172.21.15.1:0/669553958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5cb4160d20 0x7f5cb40d1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.789+0000 7f5cb37fe700 1 -- 172.21.15.1:0/669553958 >> v1:172.21.15.1:6789/0 conn(0x7f5cb4160940 legacy=0x7f5cb40d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:45.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.789+0000 7f5cb37fe700 1 -- 172.21.15.1:0/669553958 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb40ce5b0 con 0x7f5cb40cf090 2024-03-18T19:05:45.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.789+0000 7f5cb37fe700 1 --2- 172.21.15.1:0/669553958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cb40cf090 0x7f5cb40d1790 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8009840 tx=0x7f5ca8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8378f46f660dc77c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:45.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.789+0000 7f5cb2ffd700 1 -- 172.21.15.1:0/669553958 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8011830 con 0x7f5cb40cf090 2024-03-18T19:05:45.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.789+0000 7f5cb2ffd700 1 -- 172.21.15.1:0/669553958 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca8011990 con 0x7f5cb40cf090 2024-03-18T19:05:45.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.789+0000 7f5cb2ffd700 1 -- 172.21.15.1:0/669553958 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8011d20 con 0x7f5cb40cf090 2024-03-18T19:05:45.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.790+0000 7f5cba327700 1 -- 172.21.15.1:0/669553958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cb40cf090 msgr2=0x7f5cb40d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:45.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.790+0000 7f5cba327700 1 --2- 172.21.15.1:0/669553958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cb40cf090 0x7f5cb40d1790 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8009840 tx=0x7f5ca8009b50 comp rx=0 tx=0).stop 2024-03-18T19:05:45.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.790+0000 7f5cba327700 1 -- 172.21.15.1:0/669553958 shutdown_connections 2024-03-18T19:05:45.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.790+0000 7f5cba327700 1 --2- 172.21.15.1:0/669553958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5cb4160d20 0x7f5cb40d1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.790+0000 7f5cba327700 1 --2- 172.21.15.1:0/669553958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cb40cf090 0x7f5cb40d1790 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.790+0000 7f5cba327700 1 -- 172.21.15.1:0/669553958 >> 172.21.15.1:0/669553958 conn(0x7f5cb409c520 msgr2=0x7f5cb40d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:45.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.790+0000 7f5cba327700 1 -- 172.21.15.1:0/669553958 shutdown_connections 2024-03-18T19:05:45.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.790+0000 7f5cba327700 1 -- 172.21.15.1:0/669553958 wait complete. 2024-03-18T19:05:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.791+0000 7f5cba327700 1 Processor -- start 2024-03-18T19:05:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.791+0000 7f5cba327700 1 -- start start 2024-03-18T19:05:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cba327700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5cb40cf090 0x7f5cb415df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cba327700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cb4160940 0x7f5cb414d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cba327700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5cb4160d20 0x7f5cb414a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cba327700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb415f120 con 0x7f5cb4160940 2024-03-18T19:05:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cba327700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb414d9f0 con 0x7f5cb40cf090 2024-03-18T19:05:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cba327700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb415e510 con 0x7f5cb4160d20 2024-03-18T19:05:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cb37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cb4160940 0x7f5cb414d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cb37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cb4160940 0x7f5cb414d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36898/0 (socket says 172.21.15.1:36898) 2024-03-18T19:05:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cb37fe700 1 -- 172.21.15.1:0/1396592586 learned_addr learned my addr 172.21.15.1:0/1396592586 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cb88c4700 1 --2- 172.21.15.1:0/1396592586 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5cb4160d20 0x7f5cb414a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cb3fff700 1 --2- 172.21.15.1:0/1396592586 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5cb40cf090 0x7f5cb415df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cb37fe700 1 -- 172.21.15.1:0/1396592586 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5cb4160d20 msgr2=0x7f5cb414a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cb37fe700 1 --2- 172.21.15.1:0/1396592586 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5cb4160d20 0x7f5cb414a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cb37fe700 1 -- 172.21.15.1:0/1396592586 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5cb40cf090 msgr2=0x7f5cb415df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cb37fe700 1 --2- 172.21.15.1:0/1396592586 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5cb40cf090 0x7f5cb415df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cb37fe700 1 -- 172.21.15.1:0/1396592586 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca80094a0 con 0x7f5cb4160940 2024-03-18T19:05:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cb3fff700 1 --2- 172.21.15.1:0/1396592586 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5cb40cf090 0x7f5cb415df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cb37fe700 1 --2- 172.21.15.1:0/1396592586 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cb4160940 0x7f5cb414d4b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f5ca80090b0 tx=0x7f5ca8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cb0ff9700 1 -- 172.21.15.1:0/1396592586 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca80038a0 con 0x7f5cb4160940 2024-03-18T19:05:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cb0ff9700 1 -- 172.21.15.1:0/1396592586 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca801f070 con 0x7f5cb4160940 2024-03-18T19:05:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cba327700 1 -- 172.21.15.1:0/1396592586 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb415e7f0 con 0x7f5cb4160940 2024-03-18T19:05:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.792+0000 7f5cb0ff9700 1 -- 172.21.15.1:0/1396592586 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8003c60 con 0x7f5cb4160940 2024-03-18T19:05:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.793+0000 7f5cba327700 1 -- 172.21.15.1:0/1396592586 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb415edb0 con 0x7f5cb4160940 2024-03-18T19:05:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.794+0000 7f5cba327700 1 -- 172.21.15.1:0/1396592586 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb4073350 con 0x7f5cb4160940 2024-03-18T19:05:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.794+0000 7f5cb0ff9700 1 -- 172.21.15.1:0/1396592586 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 36) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f5ca801a430 con 0x7f5cb4160940 2024-03-18T19:05:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.795+0000 7f5cb0ff9700 1 -- 172.21.15.1:0/1396592586 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6631+0+0 (secure 0 0 0) 0x7f5ca8021070 con 0x7f5cb4160940 2024-03-18T19:05:45.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:45.799+0000 7f5cb0ff9700 1 -- 172.21.15.1:0/1396592586 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5ca8055e70 con 0x7f5cb4160940 2024-03-18T19:05:46.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.003+0000 7f5cba327700 1 -- 172.21.15.1:0/1396592586 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5cb40d0980 con 0x7f5cb4160940 2024-03-18T19:05:46.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.004+0000 7f5cb0ff9700 1 -- 172.21.15.1:0/1396592586 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+17046 (secure 0 0 0) 0x7f5ca8015020 con 0x7f5cb4160940 2024-03-18T19:05:46.001 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:46.002 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":91,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:05:45.625603+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:05:20.423723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-18T19:05:21.600285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3047863491":"2024-03-19T19:05:45.625549+0000","172.21.15.1:0/2733252713":"2024-03-19T19:05:45.625549+0000","172.21.15.1:0/581120352":"2024-03-19T19:05:45.625549+0000","172.21.15.1:0/2547159815":"2024-03-19T19:05:45.625549+0000","172.21.15.1:0/2546947071":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/3332271581":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/2616773698":"2024-03-19T19:05:45.625549+0000","172.21.15.1:6833/76998":"2024-03-19T19:05:37.779786+0000","172.21.15.1:6832/76998":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/1258734719":"2024-03-19T19:05:45.625549+0000","172.21.15.1:0/1890712448":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/300431708":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/4106216506":"2024-03-19T19:05:37.779786+0000","172.21.15.1:0/569963265":"2024-03-19T19:05:37.779786+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:05:46.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.007+0000 7f5cba327700 1 -- 172.21.15.1:0/1396592586 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cb4160940 msgr2=0x7f5cb414d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:46.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.007+0000 7f5cba327700 1 --2- 172.21.15.1:0/1396592586 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cb4160940 0x7f5cb414d4b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f5ca80090b0 tx=0x7f5ca8004750 comp rx=0 tx=0).stop 2024-03-18T19:05:46.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.007+0000 7f5cba327700 1 -- 172.21.15.1:0/1396592586 shutdown_connections 2024-03-18T19:05:46.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.007+0000 7f5cba327700 1 --2- 172.21.15.1:0/1396592586 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5cb4160d20 0x7f5cb414a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:46.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.007+0000 7f5cba327700 1 --2- 172.21.15.1:0/1396592586 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5cb40cf090 0x7f5cb415df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:46.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.007+0000 7f5cba327700 1 --2- 172.21.15.1:0/1396592586 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cb4160940 0x7f5cb414d4b0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:46.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.007+0000 7f5cba327700 1 -- 172.21.15.1:0/1396592586 >> 172.21.15.1:0/1396592586 conn(0x7f5cb409c520 msgr2=0x7f5cb40d1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:46.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.007+0000 7f5cba327700 1 -- 172.21.15.1:0/1396592586 shutdown_connections 2024-03-18T19:05:46.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.007+0000 7f5cba327700 1 -- 172.21.15.1:0/1396592586 wait complete. 2024-03-18T19:05:46.016 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T19:05:46.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.185+0000 7f4e2c3ba700 1 Processor -- start 2024-03-18T19:05:46.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.185+0000 7f4e2c3ba700 1 -- start start 2024-03-18T19:05:46.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.186+0000 7f4e2c3ba700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4e240aede0 0x7f4e24159e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:46.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.186+0000 7f4e2c3ba700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4e240b3320 0x7f4e240b3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:46.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.186+0000 7f4e2c3ba700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2415e3c0 con 0x7f4e2406d950 2024-03-18T19:05:46.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.186+0000 7f4e2c3ba700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e240b3c70 con 0x7f4e240b3320 2024-03-18T19:05:46.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.186+0000 7f4e2c3ba700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e240af1c0 con 0x7f4e240aede0 2024-03-18T19:05:46.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.186+0000 7f4e2a957700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4e240b3320 0x7f4e240b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:46.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.186+0000 7f4e29955700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4e2406d950 0x7f4e2415dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55118/0 (socket says 172.21.15.1:55118) 2024-03-18T19:05:46.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.186+0000 7f4e29955700 1 -- 172.21.15.1:0/647884910 learned_addr learned my addr 172.21.15.1:0/647884910 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:46.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.186+0000 7f4e2a156700 1 --2- 172.21.15.1:0/647884910 >> v2:172.21.15.165:3300/0 conn(0x7f4e240aede0 0x7f4e24159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:46.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.186+0000 7f4e2a957700 1 -- 172.21.15.1:0/647884910 >> v2:172.21.15.165:3300/0 conn(0x7f4e240aede0 msgr2=0x7f4e24159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:46.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.187+0000 7f4e2a957700 1 --2- 172.21.15.1:0/647884910 >> v2:172.21.15.165:3300/0 conn(0x7f4e240aede0 0x7f4e24159e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:46.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.187+0000 7f4e2a957700 1 -- 172.21.15.1:0/647884910 >> v1:172.21.15.1:6789/0 conn(0x7f4e2406d950 legacy=0x7f4e2415dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:46.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.187+0000 7f4e2a957700 1 -- 172.21.15.1:0/647884910 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2415d4d0 con 0x7f4e240b3320 2024-03-18T19:05:46.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.187+0000 7f4e29154700 1 -- 172.21.15.1:0/647884910 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765884408 0 0) 0x7f4e2415e3c0 con 0x7f4e2406d950 2024-03-18T19:05:46.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.187+0000 7f4e2a156700 1 --2- 172.21.15.1:0/647884910 >> v2:172.21.15.165:3300/0 conn(0x7f4e240aede0 0x7f4e24159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:46.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.187+0000 7f4e2a957700 1 --2- 172.21.15.1:0/647884910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e240b3320 0x7f4e240b3730 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f4e24067160 tx=0x7f4e1c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1982b0154018a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:46.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.187+0000 7f4e29154700 1 -- 172.21.15.1:0/647884910 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c00f070 con 0x7f4e240b3320 2024-03-18T19:05:46.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.187+0000 7f4e29154700 1 -- 172.21.15.1:0/647884910 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e1c00a2e0 con 0x7f4e240b3320 2024-03-18T19:05:46.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.187+0000 7f4e29154700 1 -- 172.21.15.1:0/647884910 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c00cc00 con 0x7f4e240b3320 2024-03-18T19:05:46.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.187+0000 7f4e2c3ba700 1 -- 172.21.15.1:0/647884910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e240b3320 msgr2=0x7f4e240b3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:46.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.187+0000 7f4e2c3ba700 1 --2- 172.21.15.1:0/647884910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e240b3320 0x7f4e240b3730 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f4e24067160 tx=0x7f4e1c00c330 comp rx=0 tx=0).stop 2024-03-18T19:05:46.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.187+0000 7f4e2c3ba700 1 -- 172.21.15.1:0/647884910 shutdown_connections 2024-03-18T19:05:46.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.187+0000 7f4e2c3ba700 1 --2- 172.21.15.1:0/647884910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e240b3320 0x7f4e240b3730 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:46.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.187+0000 7f4e2c3ba700 1 --2- 172.21.15.1:0/647884910 >> v2:172.21.15.165:3300/0 conn(0x7f4e240aede0 0x7f4e24159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:46.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.188+0000 7f4e2c3ba700 1 -- 172.21.15.1:0/647884910 >> 172.21.15.1:0/647884910 conn(0x7f4e24066b70 msgr2=0x7f4e2406a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:46.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.188+0000 7f4e2c3ba700 1 -- 172.21.15.1:0/647884910 shutdown_connections 2024-03-18T19:05:46.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.188+0000 7f4e2c3ba700 1 -- 172.21.15.1:0/647884910 wait complete. 2024-03-18T19:05:46.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.189+0000 7f4e2c3ba700 1 Processor -- start 2024-03-18T19:05:46.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.189+0000 7f4e2c3ba700 1 -- start start 2024-03-18T19:05:46.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.189+0000 7f4e2c3ba700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e2406d950 0x7f4e241f0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:46.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.190+0000 7f4e2c3ba700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e240aede0 0x7f4e24156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:46.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.190+0000 7f4e2c3ba700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e240b3320 0x7f4e241574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:46.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.190+0000 7f4e2c3ba700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e241f0ed0 con 0x7f4e2406d950 2024-03-18T19:05:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.190+0000 7f4e2c3ba700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e24155640 con 0x7f4e240aede0 2024-03-18T19:05:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.190+0000 7f4e2c3ba700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e241e4890 con 0x7f4e240b3320 2024-03-18T19:05:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.190+0000 7f4e2a156700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e2406d950 0x7f4e241f0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.190+0000 7f4e2a156700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e2406d950 0x7f4e241f0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36928/0 (socket says 172.21.15.1:36928) 2024-03-18T19:05:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.190+0000 7f4e2a156700 1 -- 172.21.15.1:0/1405827542 learned_addr learned my addr 172.21.15.1:0/1405827542 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.190+0000 7f4e2a156700 1 -- 172.21.15.1:0/1405827542 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e240b3320 msgr2=0x7f4e241574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.190+0000 7f4e2a156700 1 --2- 172.21.15.1:0/1405827542 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e240b3320 0x7f4e241574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.190+0000 7f4e2a156700 1 -- 172.21.15.1:0/1405827542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e240aede0 msgr2=0x7f4e24156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.190+0000 7f4e2a156700 1 --2- 172.21.15.1:0/1405827542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e240aede0 0x7f4e24156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.190+0000 7f4e2a156700 1 -- 172.21.15.1:0/1405827542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1c00ab80 con 0x7f4e2406d950 2024-03-18T19:05:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.190+0000 7f4e2a156700 1 --2- 172.21.15.1:0/1405827542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e2406d950 0x7f4e241f0990 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f4e1400e810 tx=0x7f4e1400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.190+0000 7f4e1affd700 1 -- 172.21.15.1:0/1405827542 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e14004230 con 0x7f4e2406d950 2024-03-18T19:05:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.190+0000 7f4e1affd700 1 -- 172.21.15.1:0/1405827542 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e14004390 con 0x7f4e2406d950 2024-03-18T19:05:46.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.190+0000 7f4e2c3ba700 1 -- 172.21.15.1:0/1405827542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e241e4b70 con 0x7f4e2406d950 2024-03-18T19:05:46.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.190+0000 7f4e1affd700 1 -- 172.21.15.1:0/1405827542 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e140036a0 con 0x7f4e2406d950 2024-03-18T19:05:46.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.190+0000 7f4e2c3ba700 1 -- 172.21.15.1:0/1405827542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2414eb10 con 0x7f4e2406d950 2024-03-18T19:05:46.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.191+0000 7f4e2c3ba700 1 -- 172.21.15.1:0/1405827542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e241e46f0 con 0x7f4e2406d950 2024-03-18T19:05:46.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.192+0000 7f4e1affd700 1 -- 172.21.15.1:0/1405827542 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 36) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f4e14003800 con 0x7f4e2406d950 2024-03-18T19:05:46.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.192+0000 7f4e1affd700 1 -- 172.21.15.1:0/1405827542 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6631+0+0 (secure 0 0 0) 0x7f4e14088b00 con 0x7f4e2406d950 2024-03-18T19:05:46.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.197+0000 7f4e1affd700 1 -- 172.21.15.1:0/1405827542 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4e241e46f0 con 0x7f4e2406d950 2024-03-18T19:05:46.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.451+0000 7f4e2c3ba700 1 -- 172.21.15.1:0/1405827542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4e241e46f0 con 0x7f4e2406d950 2024-03-18T19:05:46.449 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-18T19:05:46.452+0000 7fbdbeb46700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T19:05:46.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.458+0000 7f4e1affd700 1 -- 172.21.15.1:0/1405827542 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v34) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4e241e46f0 con 0x7f4e2406d950 2024-03-18T19:05:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.461+0000 7f4e2c3ba700 1 -- 172.21.15.1:0/1405827542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e2406d950 msgr2=0x7f4e241f0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.461+0000 7f4e2c3ba700 1 --2- 172.21.15.1:0/1405827542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e2406d950 0x7f4e241f0990 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f4e1400e810 tx=0x7f4e1400ebd0 comp rx=0 tx=0).stop 2024-03-18T19:05:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.461+0000 7f4e2c3ba700 1 -- 172.21.15.1:0/1405827542 shutdown_connections 2024-03-18T19:05:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.461+0000 7f4e2c3ba700 1 --2- 172.21.15.1:0/1405827542 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e240b3320 0x7f4e241574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.461+0000 7f4e2c3ba700 1 --2- 172.21.15.1:0/1405827542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e2406d950 0x7f4e241f0990 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.461+0000 7f4e2c3ba700 1 --2- 172.21.15.1:0/1405827542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e240aede0 0x7f4e24156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.461+0000 7f4e2c3ba700 1 -- 172.21.15.1:0/1405827542 >> 172.21.15.1:0/1405827542 conn(0x7f4e24066b70 msgr2=0x7f4e2415dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:46.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.461+0000 7f4e2c3ba700 1 -- 172.21.15.1:0/1405827542 shutdown_connections 2024-03-18T19:05:46.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.461+0000 7f4e2c3ba700 1 -- 172.21.15.1:0/1405827542 wait complete. 2024-03-18T19:05:46.459 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T19:05:46.459 INFO:tasks.ceph.mds.b.smithi165.stderr: -2660> 2024-03-18T19:05:28.149+0000 7f8e59d4d700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T19:05:46.470 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T19:05:46.484 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-03-18T19:05:46.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.601+0000 7faf42f4b700 1 Processor -- start 2024-03-18T19:05:46.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.601+0000 7faf42f4b700 1 -- start start 2024-03-18T19:05:46.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.602+0000 7faf42f4b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c0cfb50 0x7faf3c0d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:46.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.602+0000 7faf42f4b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faf3c0cff30 0x7faf3c0b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:46.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.602+0000 7faf42f4b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf3c0b3370 con 0x7faf3c0b3d40 2024-03-18T19:05:46.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.602+0000 7faf42f4b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf3c0b2760 con 0x7faf3c0cff30 2024-03-18T19:05:46.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.602+0000 7faf42f4b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c0b2a40 con 0x7faf3c0cfb50 2024-03-18T19:05:46.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.602+0000 7faf3bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faf3c0cff30 0x7faf3c0b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:46.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.602+0000 7faf414e8700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7faf3c0b3d40 0x7faf3c0b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55122/0 (socket says 172.21.15.1:55122) 2024-03-18T19:05:46.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.602+0000 7faf414e8700 1 -- 172.21.15.1:0/2998528696 learned_addr learned my addr 172.21.15.1:0/2998528696 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:46.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.602+0000 7faf40ce7700 1 --2- 172.21.15.1:0/2998528696 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c0cfb50 0x7faf3c0d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:46.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.602+0000 7faf3bfff700 1 -- 172.21.15.1:0/2998528696 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c0cfb50 msgr2=0x7faf3c0d3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:46.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.602+0000 7faf3bfff700 1 --2- 172.21.15.1:0/2998528696 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c0cfb50 0x7faf3c0d3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:46.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.602+0000 7faf3bfff700 1 -- 172.21.15.1:0/2998528696 >> v1:172.21.15.1:6789/0 conn(0x7faf3c0b3d40 legacy=0x7faf3c0b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:46.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.602+0000 7faf3bfff700 1 -- 172.21.15.1:0/2998528696 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3c0b2d20 con 0x7faf3c0cff30 2024-03-18T19:05:46.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.602+0000 7faf3b7fe700 1 -- 172.21.15.1:0/2998528696 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567753034 0 0) 0x7faf3c0b3370 con 0x7faf3c0b3d40 2024-03-18T19:05:46.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.602+0000 7faf40ce7700 1 --2- 172.21.15.1:0/2998528696 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c0cfb50 0x7faf3c0d3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:46.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.602+0000 7faf3bfff700 1 --2- 172.21.15.1:0/2998528696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf3c0cff30 0x7faf3c0b3800 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7faf28009840 tx=0x7faf28009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bbf5f679328136 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:46.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.602+0000 7faf3b7fe700 1 -- 172.21.15.1:0/2998528696 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf28011830 con 0x7faf3c0cff30 2024-03-18T19:05:46.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.603+0000 7faf3b7fe700 1 -- 172.21.15.1:0/2998528696 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf28011990 con 0x7faf3c0cff30 2024-03-18T19:05:46.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.603+0000 7faf3b7fe700 1 -- 172.21.15.1:0/2998528696 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf28011d20 con 0x7faf3c0cff30 2024-03-18T19:05:46.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.603+0000 7faf42f4b700 1 -- 172.21.15.1:0/2998528696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf3c0cff30 msgr2=0x7faf3c0b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:46.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.603+0000 7faf42f4b700 1 --2- 172.21.15.1:0/2998528696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf3c0cff30 0x7faf3c0b3800 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7faf28009840 tx=0x7faf28009b50 comp rx=0 tx=0).stop 2024-03-18T19:05:46.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.603+0000 7faf42f4b700 1 -- 172.21.15.1:0/2998528696 shutdown_connections 2024-03-18T19:05:46.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.603+0000 7faf42f4b700 1 --2- 172.21.15.1:0/2998528696 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c0cfb50 0x7faf3c0d3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:46.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.603+0000 7faf42f4b700 1 --2- 172.21.15.1:0/2998528696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf3c0cff30 0x7faf3c0b3800 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:46.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.603+0000 7faf42f4b700 1 -- 172.21.15.1:0/2998528696 >> 172.21.15.1:0/2998528696 conn(0x7faf3c0d3140 msgr2=0x7faf3c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:46.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.603+0000 7faf42f4b700 1 -- 172.21.15.1:0/2998528696 shutdown_connections 2024-03-18T19:05:46.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.603+0000 7faf42f4b700 1 -- 172.21.15.1:0/2998528696 wait complete. 2024-03-18T19:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.604+0000 7faf42f4b700 1 Processor -- start 2024-03-18T19:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.605+0000 7faf42f4b700 1 -- start start 2024-03-18T19:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.605+0000 7faf42f4b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf3c0b3d40 0x7faf3c0b8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf42f4b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf3c0cfb50 0x7faf3c15c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf42f4b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c0cff30 0x7faf3c0a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf42f4b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c0a0f10 con 0x7faf3c0cfb50 2024-03-18T19:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf42f4b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c1f6dd0 con 0x7faf3c0b3d40 2024-03-18T19:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf42f4b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c15cc20 con 0x7faf3c0cff30 2024-03-18T19:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf3bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf3c0cfb50 0x7faf3c15c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf3bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf3c0cfb50 0x7faf3c15c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36950/0 (socket says 172.21.15.1:36950) 2024-03-18T19:05:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf3bfff700 1 -- 172.21.15.1:0/4023650630 learned_addr learned my addr 172.21.15.1:0/4023650630 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf40ce7700 1 --2- 172.21.15.1:0/4023650630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf3c0b3d40 0x7faf3c0b8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf414e8700 1 --2- 172.21.15.1:0/4023650630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c0cff30 0x7faf3c0a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf3bfff700 1 -- 172.21.15.1:0/4023650630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c0cff30 msgr2=0x7faf3c0a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf3bfff700 1 --2- 172.21.15.1:0/4023650630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c0cff30 0x7faf3c0a0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf3bfff700 1 -- 172.21.15.1:0/4023650630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf3c0b3d40 msgr2=0x7faf3c0b8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf3bfff700 1 --2- 172.21.15.1:0/4023650630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf3c0b3d40 0x7faf3c0b8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf3bfff700 1 -- 172.21.15.1:0/4023650630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf280094a0 con 0x7faf3c0cfb50 2024-03-18T19:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf3bfff700 1 --2- 172.21.15.1:0/4023650630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf3c0cfb50 0x7faf3c15c6e0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7faf280090b0 tx=0x7faf28004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf397fa700 1 -- 172.21.15.1:0/4023650630 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf280036a0 con 0x7faf3c0cfb50 2024-03-18T19:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf397fa700 1 -- 172.21.15.1:0/4023650630 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2801e070 con 0x7faf3c0cfb50 2024-03-18T19:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf42f4b700 1 -- 172.21.15.1:0/4023650630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf3c0a0690 con 0x7faf3c0cfb50 2024-03-18T19:05:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf397fa700 1 -- 172.21.15.1:0/4023650630 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf28003a20 con 0x7faf3c0cfb50 2024-03-18T19:05:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.606+0000 7faf42f4b700 1 -- 172.21.15.1:0/4023650630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf3c0a0c30 con 0x7faf3c0cfb50 2024-03-18T19:05:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.607+0000 7faf42f4b700 1 -- 172.21.15.1:0/4023650630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf3c073350 con 0x7faf3c0cfb50 2024-03-18T19:05:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.608+0000 7faf397fa700 1 -- 172.21.15.1:0/4023650630 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 36) v1 ==== 88848+0+0 (secure 0 0 0) 0x7faf28003b80 con 0x7faf3c0cfb50 2024-03-18T19:05:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.608+0000 7faf397fa700 1 -- 172.21.15.1:0/4023650630 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6717+0+0 (secure 0 0 0) 0x7faf2808a150 con 0x7faf3c0cfb50 2024-03-18T19:05:46.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.613+0000 7faf397fa700 1 -- 172.21.15.1:0/4023650630 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faf28055310 con 0x7faf3c0cfb50 2024-03-18T19:05:46.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.635+0000 7faf397fa700 1 -- 172.21.15.1:0/4023650630 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 37) v1 ==== 88975+0+0 (secure 0 0 0) 0x7faf28054dc0 con 0x7faf3c0cfb50 2024-03-18T19:05:46.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.635+0000 7faf397fa700 1 --2- 172.21.15.1:0/4023650630 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf20036150 0x7faf200728f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:46.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.636+0000 7faf40ce7700 1 --2- 172.21.15.1:0/4023650630 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf20036150 0x7faf200728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:46.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.639+0000 7faf40ce7700 1 --2- 172.21.15.1:0/4023650630 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf20036150 0x7faf200728f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7faf300099c0 tx=0x7faf30008040 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:46.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:46.876+0000 7faf42f4b700 1 -- 172.21.15.1:0/4023650630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7faf3c067b30 con 0x7faf3c0cfb50 2024-03-18T19:05:47.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.460+0000 7faf397fa700 1 -- 172.21.15.1:0/4023650630 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7faf2802ed30 con 0x7faf3c0cfb50 2024-03-18T19:05:47.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.464+0000 7faf42f4b700 1 -- 172.21.15.1:0/4023650630 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf20036150 msgr2=0x7faf200728f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:47.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.464+0000 7faf42f4b700 1 --2- 172.21.15.1:0/4023650630 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf20036150 0x7faf200728f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7faf300099c0 tx=0x7faf30008040 comp rx=0 tx=0).stop 2024-03-18T19:05:47.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.464+0000 7faf42f4b700 1 -- 172.21.15.1:0/4023650630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf3c0cfb50 msgr2=0x7faf3c15c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:47.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.464+0000 7faf42f4b700 1 --2- 172.21.15.1:0/4023650630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf3c0cfb50 0x7faf3c15c6e0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7faf280090b0 tx=0x7faf28004750 comp rx=0 tx=0).stop 2024-03-18T19:05:47.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.464+0000 7faf42f4b700 1 -- 172.21.15.1:0/4023650630 shutdown_connections 2024-03-18T19:05:47.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.464+0000 7faf42f4b700 1 --2- 172.21.15.1:0/4023650630 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf20036150 0x7faf200728f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:47.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.464+0000 7faf42f4b700 1 --2- 172.21.15.1:0/4023650630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c0cff30 0x7faf3c0a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:47.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.464+0000 7faf42f4b700 1 --2- 172.21.15.1:0/4023650630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf3c0b3d40 0x7faf3c0b8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:47.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.464+0000 7faf42f4b700 1 --2- 172.21.15.1:0/4023650630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf3c0cfb50 0x7faf3c15c6e0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:47.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.464+0000 7faf42f4b700 1 -- 172.21.15.1:0/4023650630 >> 172.21.15.1:0/4023650630 conn(0x7faf3c0d3140 msgr2=0x7faf3c0b1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:47.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.464+0000 7faf42f4b700 1 -- 172.21.15.1:0/4023650630 shutdown_connections 2024-03-18T19:05:47.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.464+0000 7faf42f4b700 1 -- 172.21.15.1:0/4023650630 wait complete. 2024-03-18T19:05:47.476 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T19:05:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.615+0000 7f3233f38700 1 Processor -- start 2024-03-18T19:05:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.615+0000 7f3233f38700 1 -- start start 2024-03-18T19:05:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.616+0000 7f3233f38700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f322c0af6b0 0x7f322c15dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.616+0000 7f3233f38700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322c15e860 con 0x7f322c15af20 2024-03-18T19:05:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.616+0000 7f3233f38700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322c0719a0 con 0x7f322c15b300 2024-03-18T19:05:47.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.616+0000 7f3233f38700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322c071c80 con 0x7f322c0af6b0 2024-03-18T19:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.616+0000 7f32324d5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f322c0af6b0 0x7f322c15dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.616+0000 7f3231cd4700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f322c15af20 0x7f322c072480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55130/0 (socket says 172.21.15.1:55130) 2024-03-18T19:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.616+0000 7f3231cd4700 1 -- 172.21.15.1:0/2162023906 learned_addr learned my addr 172.21.15.1:0/2162023906 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.616+0000 7f32324d5700 1 -- 172.21.15.1:0/2162023906 >> v1:172.21.15.165:6789/0 conn(0x7f322c15b300 legacy=0x7f322c0aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.616+0000 7f32324d5700 1 -- 172.21.15.1:0/2162023906 >> v1:172.21.15.1:6789/0 conn(0x7f322c15af20 legacy=0x7f322c072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.616+0000 7f32324d5700 1 -- 172.21.15.1:0/2162023906 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c071f60 con 0x7f322c0af6b0 2024-03-18T19:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.616+0000 7f3230cd2700 1 -- 172.21.15.1:0/2162023906 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818224456 0 0) 0x7f322c15e860 con 0x7f322c15af20 2024-03-18T19:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.616+0000 7f32324d5700 1 --2- 172.21.15.1:0/2162023906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f322c0af6b0 0x7f322c15dac0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f3224009dc0 tx=0x7f322400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=838f3184cdbd704a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.616+0000 7f3230cd2700 1 -- 172.21.15.1:0/2162023906 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322400e070 con 0x7f322c0af6b0 2024-03-18T19:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.616+0000 7f3230cd2700 1 -- 172.21.15.1:0/2162023906 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322400bb90 con 0x7f322c0af6b0 2024-03-18T19:05:47.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.616+0000 7f3230cd2700 1 -- 172.21.15.1:0/2162023906 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3224012430 con 0x7f322c0af6b0 2024-03-18T19:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.617+0000 7f3233f38700 1 -- 172.21.15.1:0/2162023906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f322c0af6b0 msgr2=0x7f322c15dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.617+0000 7f3233f38700 1 --2- 172.21.15.1:0/2162023906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f322c0af6b0 0x7f322c15dac0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f3224009dc0 tx=0x7f322400b330 comp rx=0 tx=0).stop 2024-03-18T19:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.617+0000 7f3233f38700 1 -- 172.21.15.1:0/2162023906 shutdown_connections 2024-03-18T19:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.617+0000 7f3233f38700 1 --2- 172.21.15.1:0/2162023906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f322c0af6b0 0x7f322c15dac0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.617+0000 7f3233f38700 1 -- 172.21.15.1:0/2162023906 >> 172.21.15.1:0/2162023906 conn(0x7f322c0d00a0 msgr2=0x7f322c0d04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.617+0000 7f3233f38700 1 -- 172.21.15.1:0/2162023906 shutdown_connections 2024-03-18T19:05:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.617+0000 7f3233f38700 1 -- 172.21.15.1:0/2162023906 wait complete. 2024-03-18T19:05:47.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.618+0000 7f3233f38700 1 Processor -- start 2024-03-18T19:05:47.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.618+0000 7f3233f38700 1 -- start start 2024-03-18T19:05:47.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.619+0000 7f3233f38700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f322c0af6b0 0x7f322c15edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:47.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.619+0000 7f3233f38700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f322c15af20 0x7f322c13b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:47.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.619+0000 7f3233f38700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f322c15b300 0x7f322c15f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:47.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.619+0000 7f3233f38700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c152e30 con 0x7f322c0af6b0 2024-03-18T19:05:47.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.619+0000 7f3233f38700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c155850 con 0x7f322c15af20 2024-03-18T19:05:47.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.619+0000 7f3233f38700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322c155b30 con 0x7f322c15b300 2024-03-18T19:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.619+0000 7f3231cd4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f322c0af6b0 0x7f322c15edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.619+0000 7f3231cd4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f322c0af6b0 0x7f322c15edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36966/0 (socket says 172.21.15.1:36966) 2024-03-18T19:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.619+0000 7f3231cd4700 1 -- 172.21.15.1:0/3811430627 learned_addr learned my addr 172.21.15.1:0/3811430627 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.619+0000 7f3231cd4700 1 -- 172.21.15.1:0/3811430627 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f322c15b300 msgr2=0x7f322c15f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.619+0000 7f3231cd4700 1 --2- 172.21.15.1:0/3811430627 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f322c15b300 0x7f322c15f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.619+0000 7f3231cd4700 1 -- 172.21.15.1:0/3811430627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f322c15af20 msgr2=0x7f322c13b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.619+0000 7f32314d3700 1 --2- 172.21.15.1:0/3811430627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f322c15af20 0x7f322c13b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.619+0000 7f3231cd4700 1 --2- 172.21.15.1:0/3811430627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f322c15af20 0x7f322c13b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.619+0000 7f3231cd4700 1 -- 172.21.15.1:0/3811430627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3224009a70 con 0x7f322c0af6b0 2024-03-18T19:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.619+0000 7f32314d3700 1 --2- 172.21.15.1:0/3811430627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f322c15af20 0x7f322c13b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.620+0000 7f3231cd4700 1 --2- 172.21.15.1:0/3811430627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f322c0af6b0 0x7f322c15edf0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f322800c520 tx=0x7f322800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.620+0000 7f32227fc700 1 -- 172.21.15.1:0/3811430627 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228004c80 con 0x7f322c0af6b0 2024-03-18T19:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.620+0000 7f32227fc700 1 -- 172.21.15.1:0/3811430627 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3228004e00 con 0x7f322c0af6b0 2024-03-18T19:05:47.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.620+0000 7f3233f38700 1 -- 172.21.15.1:0/3811430627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f322c14f020 con 0x7f322c0af6b0 2024-03-18T19:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.620+0000 7f32227fc700 1 -- 172.21.15.1:0/3811430627 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228007310 con 0x7f322c0af6b0 2024-03-18T19:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.620+0000 7f3233f38700 1 -- 172.21.15.1:0/3811430627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322c13bd40 con 0x7f322c0af6b0 2024-03-18T19:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.621+0000 7f3233f38700 1 -- 172.21.15.1:0/3811430627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322c155cf0 con 0x7f322c0af6b0 2024-03-18T19:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.621+0000 7f32227fc700 1 -- 172.21.15.1:0/3811430627 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 37) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f3228007470 con 0x7f322c0af6b0 2024-03-18T19:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.622+0000 7f32227fc700 1 --2- 172.21.15.1:0/3811430627 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f321006aae0 0x7f321006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.622+0000 7f32227fc700 1 -- 172.21.15.1:0/3811430627 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6717+0+0 (secure 0 0 0) 0x7f32280890e0 con 0x7f322c0af6b0 2024-03-18T19:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.622+0000 7f32314d3700 1 --2- 172.21.15.1:0/3811430627 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f321006aae0 0x7f321006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:47.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.622+0000 7f32314d3700 1 --2- 172.21.15.1:0/3811430627 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f321006aae0 0x7f321006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f322c1f6760 tx=0x7f321c009040 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:47.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.626+0000 7f32227fc700 1 -- 172.21.15.1:0/3811430627 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3228054210 con 0x7f322c0af6b0 2024-03-18T19:05:47.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:47.837+0000 7f3233f38700 1 -- 172.21.15.1:0/3811430627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f322c0676a0 con 0x7f322c0af6b0 2024-03-18T19:05:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.640+0000 7f32227fc700 1 -- 172.21.15.1:0/3811430627 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 38) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f32280545e0 con 0x7f322c0af6b0 2024-03-18T19:05:48.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.812+0000 7f32227fc700 1 -- 172.21.15.1:0/3811430627 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v93) v1 ==== 181+0+0 (secure 0 0 0) 0x7f32280744e0 con 0x7f322c0af6b0 2024-03-18T19:05:48.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.817+0000 7f3233f38700 1 -- 172.21.15.1:0/3811430627 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f321006aae0 msgr2=0x7f321006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:48.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.817+0000 7f3233f38700 1 --2- 172.21.15.1:0/3811430627 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f321006aae0 0x7f321006cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f322c1f6760 tx=0x7f321c009040 comp rx=0 tx=0).stop 2024-03-18T19:05:48.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.817+0000 7f3233f38700 1 -- 172.21.15.1:0/3811430627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f322c0af6b0 msgr2=0x7f322c15edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:48.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.817+0000 7f3233f38700 1 --2- 172.21.15.1:0/3811430627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f322c0af6b0 0x7f322c15edf0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f322800c520 tx=0x7f322800c8e0 comp rx=0 tx=0).stop 2024-03-18T19:05:48.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.818+0000 7f3233f38700 1 -- 172.21.15.1:0/3811430627 shutdown_connections 2024-03-18T19:05:48.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.818+0000 7f3233f38700 1 --2- 172.21.15.1:0/3811430627 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f321006aae0 0x7f321006cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:48.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.818+0000 7f3233f38700 1 --2- 172.21.15.1:0/3811430627 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f322c15b300 0x7f322c15f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:48.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.818+0000 7f3233f38700 1 --2- 172.21.15.1:0/3811430627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f322c0af6b0 0x7f322c15edf0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:48.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.818+0000 7f3233f38700 1 --2- 172.21.15.1:0/3811430627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f322c15af20 0x7f322c13b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:48.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.818+0000 7f3233f38700 1 -- 172.21.15.1:0/3811430627 >> 172.21.15.1:0/3811430627 conn(0x7f322c0d00a0 msgr2=0x7f322c06e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:48.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.818+0000 7f3233f38700 1 -- 172.21.15.1:0/3811430627 shutdown_connections 2024-03-18T19:05:48.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.818+0000 7f3233f38700 1 -- 172.21.15.1:0/3811430627 wait complete. 2024-03-18T19:05:48.817 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-18T19:05:48.831 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T19:05:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.970+0000 7f3ef918f700 1 Processor -- start 2024-03-18T19:05:48.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.970+0000 7f3ef918f700 1 -- start start 2024-03-18T19:05:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.971+0000 7f3ef918f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3ef415d590 0x7f3ef40b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.971+0000 7f3ef918f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ef406a750 0x7f3ef406abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.971+0000 7f3ef918f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ef415d970 con 0x7f3ef406b110 2024-03-18T19:05:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.971+0000 7f3ef918f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ef4072320 con 0x7f3ef415d590 2024-03-18T19:05:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.971+0000 7f3ef918f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef4072600 con 0x7f3ef406a750 2024-03-18T19:05:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.971+0000 7f3ef2d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3ef415d590 0x7f3ef40b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.971+0000 7f3ef2d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3ef415d590 0x7f3ef40b3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36982/0 (socket says 172.21.15.1:36982) 2024-03-18T19:05:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.971+0000 7f3ef2d9d700 1 -- 172.21.15.1:0/283755537 learned_addr learned my addr 172.21.15.1:0/283755537 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:48.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.971+0000 7f3ef259c700 1 --2- 172.21.15.1:0/283755537 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ef406a750 0x7f3ef406abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.971+0000 7f3ef2d9d700 1 -- 172.21.15.1:0/283755537 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ef406a750 msgr2=0x7f3ef406abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.971+0000 7f3ef2d9d700 1 --2- 172.21.15.1:0/283755537 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ef406a750 0x7f3ef406abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.971+0000 7f3ef2d9d700 1 -- 172.21.15.1:0/283755537 >> v1:172.21.15.165:6789/0 conn(0x7f3ef406b110 legacy=0x7f3ef406ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.971+0000 7f3ef2d9d700 1 -- 172.21.15.1:0/283755537 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef40b30b0 con 0x7f3ef415d590 2024-03-18T19:05:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.971+0000 7f3ef2d9d700 1 --2- 172.21.15.1:0/283755537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ef415d590 0x7f3ef40b3bc0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f3ee40098f0 tx=0x7f3ee4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a59ae94ca42c4cac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.971+0000 7f3ef1d9b700 1 -- 172.21.15.1:0/283755537 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4004230 con 0x7f3ef415d590 2024-03-18T19:05:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.971+0000 7f3ef1d9b700 1 -- 172.21.15.1:0/283755537 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee4004390 con 0x7f3ef415d590 2024-03-18T19:05:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.972+0000 7f3ef1d9b700 1 -- 172.21.15.1:0/283755537 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee40036a0 con 0x7f3ef415d590 2024-03-18T19:05:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.972+0000 7f3ef918f700 1 -- 172.21.15.1:0/283755537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ef415d590 msgr2=0x7f3ef40b3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.972+0000 7f3ef918f700 1 --2- 172.21.15.1:0/283755537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ef415d590 0x7f3ef40b3bc0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f3ee40098f0 tx=0x7f3ee4009c00 comp rx=0 tx=0).stop 2024-03-18T19:05:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.972+0000 7f3ef918f700 1 -- 172.21.15.1:0/283755537 shutdown_connections 2024-03-18T19:05:48.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.972+0000 7f3ef918f700 1 --2- 172.21.15.1:0/283755537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ef415d590 0x7f3ef40b3bc0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:48.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.972+0000 7f3ef918f700 1 --2- 172.21.15.1:0/283755537 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ef406a750 0x7f3ef406abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:48.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.972+0000 7f3ef918f700 1 -- 172.21.15.1:0/283755537 >> 172.21.15.1:0/283755537 conn(0x7f3ef40d00c0 msgr2=0x7f3ef40d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:48.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.972+0000 7f3ef918f700 1 -- 172.21.15.1:0/283755537 shutdown_connections 2024-03-18T19:05:48.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.972+0000 7f3ef918f700 1 -- 172.21.15.1:0/283755537 wait complete. 2024-03-18T19:05:48.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.974+0000 7f3ef918f700 1 Processor -- start 2024-03-18T19:05:48.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.974+0000 7f3ef918f700 1 -- start start 2024-03-18T19:05:48.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.975+0000 7f3ef918f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ef406a750 0x7f3ef415bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:48.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.975+0000 7f3ef918f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ef406b110 0x7f3ef41e3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:48.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.975+0000 7f3ef918f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ef415d590 0x7f3ef41e2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:48.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.975+0000 7f3ef918f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef41e5020 con 0x7f3ef415d590 2024-03-18T19:05:48.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.975+0000 7f3ef918f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef41e5440 con 0x7f3ef406b110 2024-03-18T19:05:48.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.975+0000 7f3ef918f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef41e43a0 con 0x7f3ef406a750 2024-03-18T19:05:48.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.975+0000 7f3ef359e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ef415d590 0x7f3ef41e2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:48.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.975+0000 7f3ef359e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ef415d590 0x7f3ef41e2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36996/0 (socket says 172.21.15.1:36996) 2024-03-18T19:05:48.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.975+0000 7f3ef359e700 1 -- 172.21.15.1:0/2440916866 learned_addr learned my addr 172.21.15.1:0/2440916866 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:48.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.975+0000 7f3ef2d9d700 1 --2- 172.21.15.1:0/2440916866 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ef406a750 0x7f3ef415bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:48.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.975+0000 7f3ef259c700 1 --2- 172.21.15.1:0/2440916866 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ef406b110 0x7f3ef41e3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:48.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.975+0000 7f3ef359e700 1 -- 172.21.15.1:0/2440916866 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ef406a750 msgr2=0x7f3ef415bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:48.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.975+0000 7f3ef359e700 1 --2- 172.21.15.1:0/2440916866 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ef406a750 0x7f3ef415bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:48.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.975+0000 7f3ef359e700 1 -- 172.21.15.1:0/2440916866 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ef406b110 msgr2=0x7f3ef41e3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:48.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.975+0000 7f3ef359e700 1 --2- 172.21.15.1:0/2440916866 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ef406b110 0x7f3ef41e3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:48.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.975+0000 7f3ef359e700 1 -- 172.21.15.1:0/2440916866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee4009550 con 0x7f3ef415d590 2024-03-18T19:05:48.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.975+0000 7f3ef359e700 1 --2- 172.21.15.1:0/2440916866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ef415d590 0x7f3ef41e2860 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f3eec00cc10 tx=0x7f3eec00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:48.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.976+0000 7f3edb7fe700 1 -- 172.21.15.1:0/2440916866 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eec00aa60 con 0x7f3ef415d590 2024-03-18T19:05:48.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.976+0000 7f3edb7fe700 1 -- 172.21.15.1:0/2440916866 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eec00abc0 con 0x7f3ef415d590 2024-03-18T19:05:48.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.976+0000 7f3ef918f700 1 -- 172.21.15.1:0/2440916866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ef41e4680 con 0x7f3ef415d590 2024-03-18T19:05:48.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.976+0000 7f3edb7fe700 1 -- 172.21.15.1:0/2440916866 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eec012420 con 0x7f3ef415d590 2024-03-18T19:05:48.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.976+0000 7f3ef918f700 1 -- 172.21.15.1:0/2440916866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ef41e4c40 con 0x7f3ef415d590 2024-03-18T19:05:48.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.977+0000 7f3ef918f700 1 -- 172.21.15.1:0/2440916866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ef41e5160 con 0x7f3ef415d590 2024-03-18T19:05:48.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.977+0000 7f3edb7fe700 1 -- 172.21.15.1:0/2440916866 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 38) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f3eec012580 con 0x7f3ef415d590 2024-03-18T19:05:48.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.977+0000 7f3edb7fe700 1 --2- 172.21.15.1:0/2440916866 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ee006ad20 0x7f3ee006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:48.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.978+0000 7f3edb7fe700 1 -- 172.21.15.1:0/2440916866 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6279+0+0 (secure 0 0 0) 0x7f3eec0897d0 con 0x7f3ef415d590 2024-03-18T19:05:48.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.978+0000 7f3ef2d9d700 1 --2- 172.21.15.1:0/2440916866 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ee006ad20 0x7f3ee006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:48.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.978+0000 7f3ef2d9d700 1 --2- 172.21.15.1:0/2440916866 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ee006ad20 0x7f3ee006d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4009130 tx=0x7f3ee4008030 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:48.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:48.982+0000 7f3edb7fe700 1 -- 172.21.15.1:0/2440916866 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3ef41e5160 con 0x7f3ef415d590 2024-03-18T19:05:49.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.185+0000 7f3ef918f700 1 -- 172.21.15.1:0/2440916866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3ef41e5160 con 0x7f3ef415d590 2024-03-18T19:05:49.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.821+0000 7f3edb7fe700 1 -- 172.21.15.1:0/2440916866 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3eec054be0 con 0x7f3ef415d590 2024-03-18T19:05:49.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.822+0000 7f3edb7fe700 1 -- 172.21.15.1:0/2440916866 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v94) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3eec0582a0 con 0x7f3ef415d590 2024-03-18T19:05:49.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.826+0000 7f3ef918f700 1 -- 172.21.15.1:0/2440916866 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ee006ad20 msgr2=0x7f3ee006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:49.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.826+0000 7f3ef918f700 1 --2- 172.21.15.1:0/2440916866 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ee006ad20 0x7f3ee006d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4009130 tx=0x7f3ee4008030 comp rx=0 tx=0).stop 2024-03-18T19:05:49.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.826+0000 7f3ef918f700 1 -- 172.21.15.1:0/2440916866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ef415d590 msgr2=0x7f3ef41e2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:49.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.826+0000 7f3ef918f700 1 --2- 172.21.15.1:0/2440916866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ef415d590 0x7f3ef41e2860 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f3eec00cc10 tx=0x7f3eec00cf20 comp rx=0 tx=0).stop 2024-03-18T19:05:49.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.826+0000 7f3ef918f700 1 -- 172.21.15.1:0/2440916866 shutdown_connections 2024-03-18T19:05:49.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.826+0000 7f3ef918f700 1 --2- 172.21.15.1:0/2440916866 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3ee006ad20 0x7f3ee006d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:49.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.826+0000 7f3ef918f700 1 --2- 172.21.15.1:0/2440916866 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ef406a750 0x7f3ef415bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:49.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.826+0000 7f3ef918f700 1 --2- 172.21.15.1:0/2440916866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ef415d590 0x7f3ef41e2860 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:49.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.826+0000 7f3ef918f700 1 --2- 172.21.15.1:0/2440916866 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ef406b110 0x7f3ef41e3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:49.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.826+0000 7f3ef918f700 1 -- 172.21.15.1:0/2440916866 >> 172.21.15.1:0/2440916866 conn(0x7f3ef40d00c0 msgr2=0x7f3ef40afd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:49.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.827+0000 7f3ef918f700 1 -- 172.21.15.1:0/2440916866 shutdown_connections 2024-03-18T19:05:49.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.827+0000 7f3ef918f700 1 -- 172.21.15.1:0/2440916866 wait complete. 2024-03-18T19:05:49.825 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-18T19:05:49.839 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-18T19:05:49.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.975+0000 7f7c7fe9f700 1 Processor -- start 2024-03-18T19:05:49.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.976+0000 7f7c7fe9f700 1 -- start start 2024-03-18T19:05:49.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.976+0000 7f7c7fe9f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7c78071d30 0x7f7c780add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:49.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.976+0000 7f7c7fe9f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7c780cb9e0 0x7f7c780cbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:49.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.976+0000 7f7c7fe9f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c780ce8f0 con 0x7f7c7815a1d0 2024-03-18T19:05:49.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.976+0000 7f7c7fe9f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c780cdb00 con 0x7f7c78071d30 2024-03-18T19:05:49.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.976+0000 7f7c7fe9f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c7815a5b0 con 0x7f7c780cb9e0 2024-03-18T19:05:49.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.976+0000 7f7c7d43a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7c78071d30 0x7f7c780add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:49.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.977+0000 7f7c7d43a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7c78071d30 0x7f7c780add20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37008/0 (socket says 172.21.15.1:37008) 2024-03-18T19:05:49.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.977+0000 7f7c7d43a700 1 -- 172.21.15.1:0/3149127546 learned_addr learned my addr 172.21.15.1:0/3149127546 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:49.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.977+0000 7f7c7e43c700 1 --2- 172.21.15.1:0/3149127546 >> v2:172.21.15.165:3300/0 conn(0x7f7c780cb9e0 0x7f7c780cbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:49.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.977+0000 7f7c7d43a700 1 -- 172.21.15.1:0/3149127546 >> v2:172.21.15.165:3300/0 conn(0x7f7c780cb9e0 msgr2=0x7f7c780cbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:49.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.977+0000 7f7c7d43a700 1 --2- 172.21.15.1:0/3149127546 >> v2:172.21.15.165:3300/0 conn(0x7f7c780cb9e0 0x7f7c780cbe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:49.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.977+0000 7f7c7d43a700 1 -- 172.21.15.1:0/3149127546 >> v1:172.21.15.165:6789/0 conn(0x7f7c7815a1d0 legacy=0x7f7c78071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:49.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.977+0000 7f7c7d43a700 1 -- 172.21.15.1:0/3149127546 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c780ae260 con 0x7f7c78071d30 2024-03-18T19:05:49.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.977+0000 7f7c7d43a700 1 --2- 172.21.15.1:0/3149127546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c78071d30 0x7f7c780add20 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f7c780725e0 tx=0x7f7c68009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd8829f776d897a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:49.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.977+0000 7f7c7cc39700 1 -- 172.21.15.1:0/3149127546 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c68004230 con 0x7f7c78071d30 2024-03-18T19:05:49.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.977+0000 7f7c7cc39700 1 -- 172.21.15.1:0/3149127546 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c68004390 con 0x7f7c78071d30 2024-03-18T19:05:49.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.977+0000 7f7c7cc39700 1 -- 172.21.15.1:0/3149127546 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c680036a0 con 0x7f7c78071d30 2024-03-18T19:05:49.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.978+0000 7f7c7fe9f700 1 -- 172.21.15.1:0/3149127546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c78071d30 msgr2=0x7f7c780add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:49.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.978+0000 7f7c7fe9f700 1 --2- 172.21.15.1:0/3149127546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c78071d30 0x7f7c780add20 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f7c780725e0 tx=0x7f7c68009bd0 comp rx=0 tx=0).stop 2024-03-18T19:05:49.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.978+0000 7f7c7fe9f700 1 -- 172.21.15.1:0/3149127546 shutdown_connections 2024-03-18T19:05:49.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.978+0000 7f7c7fe9f700 1 --2- 172.21.15.1:0/3149127546 >> v2:172.21.15.165:3300/0 conn(0x7f7c780cb9e0 0x7f7c780cbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:49.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.978+0000 7f7c7fe9f700 1 --2- 172.21.15.1:0/3149127546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c78071d30 0x7f7c780add20 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:49.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.978+0000 7f7c7fe9f700 1 -- 172.21.15.1:0/3149127546 >> 172.21.15.1:0/3149127546 conn(0x7f7c780cc9b0 msgr2=0x7f7c780ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:49.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.978+0000 7f7c7fe9f700 1 -- 172.21.15.1:0/3149127546 shutdown_connections 2024-03-18T19:05:49.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.978+0000 7f7c7fe9f700 1 -- 172.21.15.1:0/3149127546 wait complete. 2024-03-18T19:05:49.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.979+0000 7f7c7fe9f700 1 Processor -- start 2024-03-18T19:05:49.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.979+0000 7f7c7fe9f700 1 -- start start 2024-03-18T19:05:49.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.980+0000 7f7c7fe9f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c78071d30 0x7f7c780c4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:49.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.980+0000 7f7c7fe9f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c780cb9e0 0x7f7c7815b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:49.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.980+0000 7f7c7fe9f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c7815a1d0 0x7f7c7814e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:49.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.980+0000 7f7c7fe9f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c780c5330 con 0x7f7c78071d30 2024-03-18T19:05:49.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.980+0000 7f7c7fe9f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7815bef0 con 0x7f7c780cb9e0 2024-03-18T19:05:49.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.980+0000 7f7c7fe9f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7814e9f0 con 0x7f7c7815a1d0 2024-03-18T19:05:49.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.980+0000 7f7c7dc3b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c78071d30 0x7f7c780c4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:49.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.980+0000 7f7c7dc3b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c78071d30 0x7f7c780c4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37012/0 (socket says 172.21.15.1:37012) 2024-03-18T19:05:49.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.980+0000 7f7c7dc3b700 1 -- 172.21.15.1:0/3783368736 learned_addr learned my addr 172.21.15.1:0/3783368736 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:49.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.980+0000 7f7c7d43a700 1 --2- 172.21.15.1:0/3783368736 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c780cb9e0 0x7f7c7815b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:49.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.980+0000 7f7c7e43c700 1 --2- 172.21.15.1:0/3783368736 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c7815a1d0 0x7f7c7814e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:49.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.980+0000 7f7c7dc3b700 1 -- 172.21.15.1:0/3783368736 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c7815a1d0 msgr2=0x7f7c7814e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:49.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.980+0000 7f7c7dc3b700 1 --2- 172.21.15.1:0/3783368736 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c7815a1d0 0x7f7c7814e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:49.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.980+0000 7f7c7dc3b700 1 -- 172.21.15.1:0/3783368736 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c780cb9e0 msgr2=0x7f7c7815b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:49.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.980+0000 7f7c7dc3b700 1 --2- 172.21.15.1:0/3783368736 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c780cb9e0 0x7f7c7815b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:49.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.980+0000 7f7c7dc3b700 1 -- 172.21.15.1:0/3783368736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c68009550 con 0x7f7c78071d30 2024-03-18T19:05:49.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.980+0000 7f7c7d43a700 1 --2- 172.21.15.1:0/3783368736 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c780cb9e0 0x7f7c7815b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:49.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.980+0000 7f7c7dc3b700 1 --2- 172.21.15.1:0/3783368736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c78071d30 0x7f7c780c4df0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f7c7400c4e0 tx=0x7f7c7400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:49.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.980+0000 7f7c6e7fc700 1 -- 172.21.15.1:0/3783368736 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c74011790 con 0x7f7c78071d30 2024-03-18T19:05:49.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.981+0000 7f7c6e7fc700 1 -- 172.21.15.1:0/3783368736 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c740118f0 con 0x7f7c78071d30 2024-03-18T19:05:49.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.981+0000 7f7c7fe9f700 1 -- 172.21.15.1:0/3783368736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7815d1c0 con 0x7f7c78071d30 2024-03-18T19:05:49.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.981+0000 7f7c6e7fc700 1 -- 172.21.15.1:0/3783368736 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c74011ca0 con 0x7f7c78071d30 2024-03-18T19:05:49.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.981+0000 7f7c7fe9f700 1 -- 172.21.15.1:0/3783368736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7815c400 con 0x7f7c78071d30 2024-03-18T19:05:49.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.981+0000 7f7c7fe9f700 1 -- 172.21.15.1:0/3783368736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c78073350 con 0x7f7c78071d30 2024-03-18T19:05:49.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.982+0000 7f7c6e7fc700 1 -- 172.21.15.1:0/3783368736 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7c74012420 con 0x7f7c78071d30 2024-03-18T19:05:49.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.982+0000 7f7c6e7fc700 1 --2- 172.21.15.1:0/3783368736 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7c5c06adb0 0x7f7c5c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:49.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.982+0000 7f7c6e7fc700 1 -- 172.21.15.1:0/3783368736 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f7c74089060 con 0x7f7c78071d30 2024-03-18T19:05:49.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.982+0000 7f7c7d43a700 1 --2- 172.21.15.1:0/3783368736 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7c5c06adb0 0x7f7c5c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:49.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.983+0000 7f7c7d43a700 1 --2- 172.21.15.1:0/3783368736 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7c5c06adb0 0x7f7c5c06d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7c68009130 tx=0x7f7c68008030 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:49.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:49.987+0000 7f7c6e7fc700 1 -- 172.21.15.1:0/3783368736 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7c74054120 con 0x7f7c78071d30 2024-03-18T19:05:50.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.188+0000 7f7c7fe9f700 1 -- 172.21.15.1:0/3783368736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f7c780676a0 con 0x7f7c78071d30 2024-03-18T19:05:50.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.192+0000 7f7c6e7fc700 1 -- 172.21.15.1:0/3783368736 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v25) v1 ==== 115+0+0 (secure 0 0 0) 0x7f7c74057740 con 0x7f7c78071d30 2024-03-18T19:05:50.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.194+0000 7f7c7fe9f700 1 -- 172.21.15.1:0/3783368736 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7c5c06adb0 msgr2=0x7f7c5c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:50.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.194+0000 7f7c7fe9f700 1 --2- 172.21.15.1:0/3783368736 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7c5c06adb0 0x7f7c5c06d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7c68009130 tx=0x7f7c68008030 comp rx=0 tx=0).stop 2024-03-18T19:05:50.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.194+0000 7f7c7fe9f700 1 -- 172.21.15.1:0/3783368736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c78071d30 msgr2=0x7f7c780c4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:50.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.194+0000 7f7c7fe9f700 1 --2- 172.21.15.1:0/3783368736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c78071d30 0x7f7c780c4df0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f7c7400c4e0 tx=0x7f7c7400c8a0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.194+0000 7f7c7fe9f700 1 -- 172.21.15.1:0/3783368736 shutdown_connections 2024-03-18T19:05:50.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.194+0000 7f7c7fe9f700 1 --2- 172.21.15.1:0/3783368736 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7c5c06adb0 0x7f7c5c06d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.194+0000 7f7c7fe9f700 1 --2- 172.21.15.1:0/3783368736 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c7815a1d0 0x7f7c7814e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.194+0000 7f7c7fe9f700 1 --2- 172.21.15.1:0/3783368736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c78071d30 0x7f7c780c4df0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.194+0000 7f7c7fe9f700 1 --2- 172.21.15.1:0/3783368736 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c780cb9e0 0x7f7c7815b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.194+0000 7f7c7fe9f700 1 -- 172.21.15.1:0/3783368736 >> 172.21.15.1:0/3783368736 conn(0x7f7c780cc9b0 msgr2=0x7f7c78157e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:50.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.195+0000 7f7c7fe9f700 1 -- 172.21.15.1:0/3783368736 shutdown_connections 2024-03-18T19:05:50.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.195+0000 7f7c7fe9f700 1 -- 172.21.15.1:0/3783368736 wait complete. 2024-03-18T19:05:50.204 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-18T19:05:50.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.339+0000 7f800cb19700 1 Processor -- start 2024-03-18T19:05:50.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.339+0000 7f800cb19700 1 -- start start 2024-03-18T19:05:50.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.340+0000 7f800cb19700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f80080ace80 0x7f80080ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:50.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.340+0000 7f800cb19700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f80080ad850 0x7f800815ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:50.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.340+0000 7f800cb19700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800815c340 con 0x7f800806a750 2024-03-18T19:05:50.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.340+0000 7f800cb19700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8008070090 con 0x7f80080ace80 2024-03-18T19:05:50.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.340+0000 7f800cb19700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f800815b7a0 con 0x7f80080ad850 2024-03-18T19:05:50.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.340+0000 7f8005d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f80080ace80 0x7f80080ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:50.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.340+0000 7f8005d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f80080ace80 0x7f80080ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37018/0 (socket says 172.21.15.1:37018) 2024-03-18T19:05:50.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.340+0000 7f8005d9b700 1 -- 172.21.15.1:0/3749052702 learned_addr learned my addr 172.21.15.1:0/3749052702 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:50.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.340+0000 7f800659c700 1 --1- 172.21.15.1:0/3749052702 >> v1:172.21.15.1:6789/0 conn(0x7f800806a750 0x7f800806ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55144/0 (socket says 172.21.15.1:55144) 2024-03-18T19:05:50.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.340+0000 7f8006d9d700 1 --2- 172.21.15.1:0/3749052702 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f80080ad850 0x7f800815ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:50.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.340+0000 7f8005d9b700 1 -- 172.21.15.1:0/3749052702 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f80080ad850 msgr2=0x7f800815ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:50.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.340+0000 7f8005d9b700 1 --2- 172.21.15.1:0/3749052702 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f80080ad850 0x7f800815ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.340+0000 7f8005d9b700 1 -- 172.21.15.1:0/3749052702 >> v1:172.21.15.1:6789/0 conn(0x7f800806a750 legacy=0x7f800806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:50.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.340+0000 7f8005d9b700 1 -- 172.21.15.1:0/3749052702 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800815ba10 con 0x7f80080ace80 2024-03-18T19:05:50.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.340+0000 7f8005d9b700 1 --2- 172.21.15.1:0/3749052702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80080ace80 0x7f80080ad310 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0098f0 tx=0x7f7ffc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4437cb7c45439a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:50.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.341+0000 7f800559a700 1 -- 172.21.15.1:0/3749052702 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc00b8a0 con 0x7f80080ace80 2024-03-18T19:05:50.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.341+0000 7f800559a700 1 -- 172.21.15.1:0/3749052702 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ffc00ba00 con 0x7f80080ace80 2024-03-18T19:05:50.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.341+0000 7f800559a700 1 -- 172.21.15.1:0/3749052702 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc00bd90 con 0x7f80080ace80 2024-03-18T19:05:50.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.341+0000 7f800cb19700 1 -- 172.21.15.1:0/3749052702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80080ace80 msgr2=0x7f80080ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:50.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.341+0000 7f800cb19700 1 --2- 172.21.15.1:0/3749052702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80080ace80 0x7f80080ad310 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0098f0 tx=0x7f7ffc009c00 comp rx=0 tx=0).stop 2024-03-18T19:05:50.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.341+0000 7f800cb19700 1 -- 172.21.15.1:0/3749052702 shutdown_connections 2024-03-18T19:05:50.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.341+0000 7f800cb19700 1 --2- 172.21.15.1:0/3749052702 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f80080ad850 0x7f800815ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.341+0000 7f800cb19700 1 --2- 172.21.15.1:0/3749052702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80080ace80 0x7f80080ad310 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.341+0000 7f800cb19700 1 -- 172.21.15.1:0/3749052702 >> 172.21.15.1:0/3749052702 conn(0x7f80080cc9b0 msgr2=0x7f80080ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:50.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.341+0000 7f800cb19700 1 -- 172.21.15.1:0/3749052702 shutdown_connections 2024-03-18T19:05:50.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.341+0000 7f800cb19700 1 -- 172.21.15.1:0/3749052702 wait complete. 2024-03-18T19:05:50.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.343+0000 7f800cb19700 1 Processor -- start 2024-03-18T19:05:50.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.343+0000 7f800cb19700 1 -- start start 2024-03-18T19:05:50.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.343+0000 7f800cb19700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f800806a750 0x7f800809cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:50.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.343+0000 7f800cb19700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80080ace80 0x7f80080b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:50.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.343+0000 7f800cb19700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f80080ad850 0x7f80080b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:50.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.343+0000 7f800cb19700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800809d4e0 con 0x7f80080ace80 2024-03-18T19:05:50.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.343+0000 7f800cb19700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80080c8480 con 0x7f800806a750 2024-03-18T19:05:50.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.343+0000 7f800cb19700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80080c7870 con 0x7f80080ad850 2024-03-18T19:05:50.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.343+0000 7f8005d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80080ace80 0x7f80080b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:50.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.343+0000 7f8005d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80080ace80 0x7f80080b2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37020/0 (socket says 172.21.15.1:37020) 2024-03-18T19:05:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.343+0000 7f8005d9b700 1 -- 172.21.15.1:0/1093523044 learned_addr learned my addr 172.21.15.1:0/1093523044 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.344+0000 7f8006d9d700 1 --2- 172.21.15.1:0/1093523044 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f80080ad850 0x7f80080b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.344+0000 7f8005d9b700 1 -- 172.21.15.1:0/1093523044 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f80080ad850 msgr2=0x7f80080b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.344+0000 7f800659c700 1 --2- 172.21.15.1:0/1093523044 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f800806a750 0x7f800809cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.344+0000 7f8005d9b700 1 --2- 172.21.15.1:0/1093523044 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f80080ad850 0x7f80080b30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.344+0000 7f8005d9b700 1 -- 172.21.15.1:0/1093523044 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f800806a750 msgr2=0x7f800809cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.344+0000 7f8005d9b700 1 --2- 172.21.15.1:0/1093523044 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f800806a750 0x7f800809cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.344+0000 7f8005d9b700 1 -- 172.21.15.1:0/1093523044 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc009550 con 0x7f80080ace80 2024-03-18T19:05:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.344+0000 7f8005d9b700 1 --2- 172.21.15.1:0/1093523044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80080ace80 0x7f80080b2b90 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc009c40 tx=0x7f7ffc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.344+0000 7f7ff6ffd700 1 -- 172.21.15.1:0/1093523044 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc004530 con 0x7f80080ace80 2024-03-18T19:05:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.344+0000 7f7ff6ffd700 1 -- 172.21.15.1:0/1093523044 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ffc01f070 con 0x7f80080ace80 2024-03-18T19:05:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.344+0000 7f7ff6ffd700 1 -- 172.21.15.1:0/1093523044 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc0038e0 con 0x7f80080ace80 2024-03-18T19:05:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.344+0000 7f800cb19700 1 -- 172.21.15.1:0/1093523044 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80080c7b50 con 0x7f80080ace80 2024-03-18T19:05:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.344+0000 7f800cb19700 1 -- 172.21.15.1:0/1093523044 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80080c8110 con 0x7f80080ace80 2024-03-18T19:05:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.345+0000 7f800cb19700 1 -- 172.21.15.1:0/1093523044 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80080676a0 con 0x7f80080ace80 2024-03-18T19:05:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.346+0000 7f7ff6ffd700 1 -- 172.21.15.1:0/1093523044 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7ffc021030 con 0x7f80080ace80 2024-03-18T19:05:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.346+0000 7f7ff6ffd700 1 --2- 172.21.15.1:0/1093523044 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7fe806adb0 0x7f7fe806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.346+0000 7f800659c700 1 --2- 172.21.15.1:0/1093523044 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7fe806adb0 0x7f7fe806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.346+0000 7f7ff6ffd700 1 -- 172.21.15.1:0/1093523044 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f7ffc08a9e0 con 0x7f80080ace80 2024-03-18T19:05:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.347+0000 7f800659c700 1 --2- 172.21.15.1:0/1093523044 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7fe806adb0 0x7f7fe806d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8001020 tx=0x7f7ff8009040 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:50.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.351+0000 7f7ff6ffd700 1 -- 172.21.15.1:0/1093523044 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7ffc055bc0 con 0x7f80080ace80 2024-03-18T19:05:50.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.556+0000 7f800cb19700 1 -- 172.21.15.1:0/1093523044 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f80080cd640 con 0x7f80080ace80 2024-03-18T19:05:50.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.560+0000 7f7ff6ffd700 1 -- 172.21.15.1:0/1093523044 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v26) v1 ==== 120+0+0 (secure 0 0 0) 0x7f7ffc0150b0 con 0x7f80080ace80 2024-03-18T19:05:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.562+0000 7f800cb19700 1 -- 172.21.15.1:0/1093523044 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7fe806adb0 msgr2=0x7f7fe806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.562+0000 7f800cb19700 1 --2- 172.21.15.1:0/1093523044 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7fe806adb0 0x7f7fe806d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8001020 tx=0x7f7ff8009040 comp rx=0 tx=0).stop 2024-03-18T19:05:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.562+0000 7f800cb19700 1 -- 172.21.15.1:0/1093523044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80080ace80 msgr2=0x7f80080b2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.562+0000 7f800cb19700 1 --2- 172.21.15.1:0/1093523044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80080ace80 0x7f80080b2b90 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc009c40 tx=0x7f7ffc004750 comp rx=0 tx=0).stop 2024-03-18T19:05:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.562+0000 7f800cb19700 1 -- 172.21.15.1:0/1093523044 shutdown_connections 2024-03-18T19:05:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.562+0000 7f800cb19700 1 --2- 172.21.15.1:0/1093523044 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7fe806adb0 0x7f7fe806d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.562+0000 7f800cb19700 1 --2- 172.21.15.1:0/1093523044 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f80080ad850 0x7f80080b30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.562+0000 7f800cb19700 1 --2- 172.21.15.1:0/1093523044 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f800806a750 0x7f800809cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.562+0000 7f800cb19700 1 --2- 172.21.15.1:0/1093523044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80080ace80 0x7f80080b2b90 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.562+0000 7f800cb19700 1 -- 172.21.15.1:0/1093523044 >> 172.21.15.1:0/1093523044 conn(0x7f80080cc9b0 msgr2=0x7f80080b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.562+0000 7f800cb19700 1 -- 172.21.15.1:0/1093523044 shutdown_connections 2024-03-18T19:05:50.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.562+0000 7f800cb19700 1 -- 172.21.15.1:0/1093523044 wait complete. 2024-03-18T19:05:50.572 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-18T19:05:50.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.702+0000 7f5720c61700 1 Processor -- start 2024-03-18T19:05:50.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.702+0000 7f5720c61700 1 -- start start 2024-03-18T19:05:50.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.703+0000 7f5720c61700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f571c0cd870 0x7f571c0cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:50.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.703+0000 7f5720c61700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f571c0b1e20 0x7f571c0cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:50.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.703+0000 7f5720c61700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571c0d0df0 con 0x7f571c06bc20 2024-03-18T19:05:50.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.703+0000 7f5720c61700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f571c0ce190 con 0x7f571c0b1e20 2024-03-18T19:05:50.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.703+0000 7f5720c61700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571c0d0550 con 0x7f571c0cd870 2024-03-18T19:05:50.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.703+0000 7f5719d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f571c0b1e20 0x7f571c0cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:50.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.703+0000 7f5719d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f571c0b1e20 0x7f571c0cdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37030/0 (socket says 172.21.15.1:37030) 2024-03-18T19:05:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.703+0000 7f5719d9b700 1 -- 172.21.15.1:0/1565883650 learned_addr learned my addr 172.21.15.1:0/1565883650 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.703+0000 7f571ad9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f571c06bc20 0x7f571c06c0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55152/0 (socket says 172.21.15.1:55152) 2024-03-18T19:05:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.703+0000 7f571a59c700 1 --2- 172.21.15.1:0/1565883650 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f571c0cd870 0x7f571c0cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.703+0000 7f5719d9b700 1 -- 172.21.15.1:0/1565883650 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f571c0cd870 msgr2=0x7f571c0cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.703+0000 7f5719d9b700 1 --2- 172.21.15.1:0/1565883650 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f571c0cd870 0x7f571c0cff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.703+0000 7f5719d9b700 1 -- 172.21.15.1:0/1565883650 >> v1:172.21.15.1:6789/0 conn(0x7f571c06bc20 legacy=0x7f571c06c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.703+0000 7f5719d9b700 1 -- 172.21.15.1:0/1565883650 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c0d0830 con 0x7f571c0b1e20 2024-03-18T19:05:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.703+0000 7f5719d9b700 1 --2- 172.21.15.1:0/1565883650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c0b1e20 0x7f571c0cdc50 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f57100098f0 tx=0x7f5710009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a95bf56e832f385a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.703+0000 7f571959a700 1 -- 172.21.15.1:0/1565883650 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571000b8a0 con 0x7f571c0b1e20 2024-03-18T19:05:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.703+0000 7f571959a700 1 -- 172.21.15.1:0/1565883650 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571000ba00 con 0x7f571c0b1e20 2024-03-18T19:05:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.703+0000 7f571959a700 1 -- 172.21.15.1:0/1565883650 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571000bd90 con 0x7f571c0b1e20 2024-03-18T19:05:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.704+0000 7f5720c61700 1 -- 172.21.15.1:0/1565883650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c0b1e20 msgr2=0x7f571c0cdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.704+0000 7f5720c61700 1 --2- 172.21.15.1:0/1565883650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c0b1e20 0x7f571c0cdc50 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f57100098f0 tx=0x7f5710009c00 comp rx=0 tx=0).stop 2024-03-18T19:05:50.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.704+0000 7f5720c61700 1 -- 172.21.15.1:0/1565883650 shutdown_connections 2024-03-18T19:05:50.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.704+0000 7f5720c61700 1 --2- 172.21.15.1:0/1565883650 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f571c0cd870 0x7f571c0cff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.704+0000 7f5720c61700 1 --2- 172.21.15.1:0/1565883650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c0b1e20 0x7f571c0cdc50 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.704+0000 7f5720c61700 1 -- 172.21.15.1:0/1565883650 >> 172.21.15.1:0/1565883650 conn(0x7f571c0d0ff0 msgr2=0x7f571c0d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:50.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.704+0000 7f5720c61700 1 -- 172.21.15.1:0/1565883650 shutdown_connections 2024-03-18T19:05:50.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.704+0000 7f5720c61700 1 -- 172.21.15.1:0/1565883650 wait complete. 2024-03-18T19:05:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.706+0000 7f5720c61700 1 Processor -- start 2024-03-18T19:05:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.706+0000 7f5720c61700 1 -- start start 2024-03-18T19:05:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.706+0000 7f5720c61700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c06bc20 0x7f571c1590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.706+0000 7f5720c61700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f571c0b1e20 0x7f571c09b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.706+0000 7f5720c61700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f571c0cd870 0x7f571c09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.706+0000 7f5720c61700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571c0b6830 con 0x7f571c06bc20 2024-03-18T19:05:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.706+0000 7f5720c61700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571c0b5f90 con 0x7f571c0b1e20 2024-03-18T19:05:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.706+0000 7f5720c61700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571c0b6270 con 0x7f571c0cd870 2024-03-18T19:05:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.706+0000 7f571a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c06bc20 0x7f571c1590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.706+0000 7f571a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c06bc20 0x7f571c1590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37038/0 (socket says 172.21.15.1:37038) 2024-03-18T19:05:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.706+0000 7f571a59c700 1 -- 172.21.15.1:0/985575783 learned_addr learned my addr 172.21.15.1:0/985575783 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.707+0000 7f571a59c700 1 -- 172.21.15.1:0/985575783 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f571c0cd870 msgr2=0x7f571c09bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.707+0000 7f5719d9b700 1 --2- 172.21.15.1:0/985575783 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f571c0b1e20 0x7f571c09b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.707+0000 7f571a59c700 1 --2- 172.21.15.1:0/985575783 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f571c0cd870 0x7f571c09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.707+0000 7f571a59c700 1 -- 172.21.15.1:0/985575783 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f571c0b1e20 msgr2=0x7f571c09b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.707+0000 7f571a59c700 1 --2- 172.21.15.1:0/985575783 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f571c0b1e20 0x7f571c09b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.707+0000 7f571a59c700 1 -- 172.21.15.1:0/985575783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5710009550 con 0x7f571c06bc20 2024-03-18T19:05:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.707+0000 7f571a59c700 1 --2- 172.21.15.1:0/985575783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c06bc20 0x7f571c1590a0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f570c00d800 tx=0x7f570c00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.707+0000 7f570affd700 1 -- 172.21.15.1:0/985575783 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c0125f0 con 0x7f571c06bc20 2024-03-18T19:05:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.707+0000 7f570affd700 1 -- 172.21.15.1:0/985575783 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570c012750 con 0x7f571c06bc20 2024-03-18T19:05:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.707+0000 7f5720c61700 1 -- 172.21.15.1:0/985575783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571c0b6550 con 0x7f571c06bc20 2024-03-18T19:05:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.707+0000 7f570affd700 1 -- 172.21.15.1:0/985575783 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c012b00 con 0x7f571c06bc20 2024-03-18T19:05:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.707+0000 7f5720c61700 1 -- 172.21.15.1:0/985575783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f571c1595e0 con 0x7f571c06bc20 2024-03-18T19:05:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.708+0000 7f5720c61700 1 -- 172.21.15.1:0/985575783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f571c0676a0 con 0x7f571c06bc20 2024-03-18T19:05:50.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.709+0000 7f570affd700 1 -- 172.21.15.1:0/985575783 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f570c012c60 con 0x7f571c06bc20 2024-03-18T19:05:50.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.709+0000 7f570affd700 1 --2- 172.21.15.1:0/985575783 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56fc06adb0 0x7f56fc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:50.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.709+0000 7f5719d9b700 1 --2- 172.21.15.1:0/985575783 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56fc06adb0 0x7f56fc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:50.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.709+0000 7f570affd700 1 -- 172.21.15.1:0/985575783 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f570c0898e0 con 0x7f571c06bc20 2024-03-18T19:05:50.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.710+0000 7f5719d9b700 1 --2- 172.21.15.1:0/985575783 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56fc06adb0 0x7f56fc06d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f571c0b6a80 tx=0x7f5710008030 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:50.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.714+0000 7f570affd700 1 -- 172.21.15.1:0/985575783 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f570c054910 con 0x7f571c06bc20 2024-03-18T19:05:50.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.916+0000 7f5720c61700 1 -- 172.21.15.1:0/985575783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f571c06b470 con 0x7f571c06bc20 2024-03-18T19:05:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.919+0000 7f570affd700 1 -- 172.21.15.1:0/985575783 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f570c057f30 con 0x7f571c06bc20 2024-03-18T19:05:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.921+0000 7f5720c61700 1 -- 172.21.15.1:0/985575783 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56fc06adb0 msgr2=0x7f56fc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.921+0000 7f5720c61700 1 --2- 172.21.15.1:0/985575783 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56fc06adb0 0x7f56fc06d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f571c0b6a80 tx=0x7f5710008030 comp rx=0 tx=0).stop 2024-03-18T19:05:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.921+0000 7f5720c61700 1 -- 172.21.15.1:0/985575783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c06bc20 msgr2=0x7f571c1590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.921+0000 7f5720c61700 1 --2- 172.21.15.1:0/985575783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c06bc20 0x7f571c1590a0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f570c00d800 tx=0x7f570c00dbc0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.921+0000 7f5720c61700 1 -- 172.21.15.1:0/985575783 shutdown_connections 2024-03-18T19:05:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.921+0000 7f5720c61700 1 --2- 172.21.15.1:0/985575783 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56fc06adb0 0x7f56fc06d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.921+0000 7f5720c61700 1 --2- 172.21.15.1:0/985575783 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f571c0cd870 0x7f571c09bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.921+0000 7f5720c61700 1 --2- 172.21.15.1:0/985575783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c06bc20 0x7f571c1590a0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.921+0000 7f5720c61700 1 --2- 172.21.15.1:0/985575783 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f571c0b1e20 0x7f571c09b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:50.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.921+0000 7f5720c61700 1 -- 172.21.15.1:0/985575783 >> 172.21.15.1:0/985575783 conn(0x7f571c0d0ff0 msgr2=0x7f571c06c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:50.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.921+0000 7f5720c61700 1 -- 172.21.15.1:0/985575783 shutdown_connections 2024-03-18T19:05:50.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:50.921+0000 7f5720c61700 1 -- 172.21.15.1:0/985575783 wait complete. 2024-03-18T19:05:50.930 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2024-03-18T19:05:51.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.059+0000 7f4579dc7700 1 Processor -- start 2024-03-18T19:05:51.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.059+0000 7f4579dc7700 1 -- start start 2024-03-18T19:05:51.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f4579dc7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f457406ae60 0x7f4574164f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:51.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f4579dc7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4574072660 con 0x7f4574146190 2024-03-18T19:05:51.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f4579dc7700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457406ebd0 con 0x7f45741465a0 2024-03-18T19:05:51.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f4579dc7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f457406f850 con 0x7f457406ae60 2024-03-18T19:05:51.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f45737fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4574146190 0x7f457406a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55166/0 (socket says 172.21.15.1:55166) 2024-03-18T19:05:51.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f45737fe700 1 -- 172.21.15.1:0/1539476715 learned_addr learned my addr 172.21.15.1:0/1539476715 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:51.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f4572ffd700 1 --2- 172.21.15.1:0/1539476715 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f457406ae60 0x7f4574164f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:51.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f45727fc700 1 -- 172.21.15.1:0/1539476715 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743783463 0 0) 0x7f4574072660 con 0x7f4574146190 2024-03-18T19:05:51.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f45727fc700 1 -- 172.21.15.1:0/1539476715 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f455c0035b0 con 0x7f4574146190 2024-03-18T19:05:51.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f45727fc700 1 -- 172.21.15.1:0/1539476715 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056002490 0 0) 0x7f457406ebd0 con 0x7f45741465a0 2024-03-18T19:05:51.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f45727fc700 1 -- 172.21.15.1:0/1539476715 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4574072660 con 0x7f45741465a0 2024-03-18T19:05:51.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f45727fc700 1 -- 172.21.15.1:0/1539476715 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 19563848 0 0) 0x7f455c0035b0 con 0x7f4574146190 2024-03-18T19:05:51.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f45727fc700 1 -- 172.21.15.1:0/1539476715 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f457406ae60 msgr2=0x7f4574164f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:51.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f45727fc700 1 --2- 172.21.15.1:0/1539476715 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f457406ae60 0x7f4574164f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:51.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f45727fc700 1 -- 172.21.15.1:0/1539476715 >> v1:172.21.15.165:6789/0 conn(0x7f45741465a0 legacy=0x7f4574167480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:51.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f45727fc700 1 -- 172.21.15.1:0/1539476715 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457406eeb0 con 0x7f4574146190 2024-03-18T19:05:51.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f45727fc700 1 -- 172.21.15.1:0/1539476715 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f4568001670 con 0x7f4574146190 2024-03-18T19:05:51.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f4572ffd700 1 --2- 172.21.15.1:0/1539476715 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f457406ae60 0x7f4574164f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:51.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f45727fc700 1 -- 172.21.15.1:0/1539476715 >> v1:172.21.15.1:6789/0 conn(0x7f4574146190 legacy=0x7f457406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:51.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f45727fc700 1 --2- 172.21.15.1:0/1539476715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c003bc0 0x7f455c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:51.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f45727fc700 1 --2- 172.21.15.1:0/1539476715 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f455c004850 0x7f455c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:51.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f45727fc700 1 --2- 172.21.15.1:0/1539476715 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f457406ae60 0x7f455c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:51.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f45727fc700 1 -- 172.21.15.1:0/1539476715 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457406f850 con 0x7f455c003bc0 2024-03-18T19:05:51.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f45727fc700 1 -- 172.21.15.1:0/1539476715 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455c00ba40 con 0x7f455c004850 2024-03-18T19:05:51.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.060+0000 7f45727fc700 1 -- 172.21.15.1:0/1539476715 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f455c00bc00 con 0x7f457406ae60 2024-03-18T19:05:51.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.061+0000 7f45737fe700 1 --2- 172.21.15.1:0/1539476715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c003bc0 0x7f455c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:51.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.061+0000 7f4572ffd700 1 --2- 172.21.15.1:0/1539476715 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f455c004850 0x7f455c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:51.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.061+0000 7f4573fff700 1 --2- 172.21.15.1:0/1539476715 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f457406ae60 0x7f455c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:51.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.061+0000 7f45737fe700 1 -- 172.21.15.1:0/1539476715 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f457406ae60 msgr2=0x7f455c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:51.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.061+0000 7f45737fe700 1 --2- 172.21.15.1:0/1539476715 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f457406ae60 0x7f455c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:51.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.061+0000 7f45737fe700 1 -- 172.21.15.1:0/1539476715 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f455c004850 msgr2=0x7f455c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:51.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.061+0000 7f45737fe700 1 --2- 172.21.15.1:0/1539476715 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f455c004850 0x7f455c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:51.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.061+0000 7f45737fe700 1 -- 172.21.15.1:0/1539476715 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45640094a0 con 0x7f455c003bc0 2024-03-18T19:05:51.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.061+0000 7f4572ffd700 1 --2- 172.21.15.1:0/1539476715 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f455c004850 0x7f455c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:51.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.061+0000 7f45737fe700 1 --2- 172.21.15.1:0/1539476715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c003bc0 0x7f455c003fc0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f456800c210 tx=0x7f455c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe08ec74ff980f88 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:51.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.061+0000 7f45727fc700 1 -- 172.21.15.1:0/1539476715 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456800cd50 con 0x7f455c003bc0 2024-03-18T19:05:51.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.061+0000 7f45727fc700 1 -- 172.21.15.1:0/1539476715 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4568005960 con 0x7f455c003bc0 2024-03-18T19:05:51.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.061+0000 7f45727fc700 1 -- 172.21.15.1:0/1539476715 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4568005d10 con 0x7f455c003bc0 2024-03-18T19:05:51.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.062+0000 7f4579dc7700 1 -- 172.21.15.1:0/1539476715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c003bc0 msgr2=0x7f455c003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:51.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.062+0000 7f4579dc7700 1 --2- 172.21.15.1:0/1539476715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c003bc0 0x7f455c003fc0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f456800c210 tx=0x7f455c005880 comp rx=0 tx=0).stop 2024-03-18T19:05:51.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.062+0000 7f4573fff700 1 -- 172.21.15.1:0/1539476715 reap_dead start 2024-03-18T19:05:51.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.062+0000 7f4579dc7700 1 -- 172.21.15.1:0/1539476715 shutdown_connections 2024-03-18T19:05:51.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.062+0000 7f4579dc7700 1 -- 172.21.15.1:0/1539476715 >> 172.21.15.1:0/1539476715 conn(0x7f45740b1d20 msgr2=0x7f45740b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:51.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.062+0000 7f4579dc7700 1 -- 172.21.15.1:0/1539476715 shutdown_connections 2024-03-18T19:05:51.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.062+0000 7f4579dc7700 1 -- 172.21.15.1:0/1539476715 wait complete. 2024-03-18T19:05:51.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.063+0000 7f4579dc7700 1 Processor -- start 2024-03-18T19:05:51.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.063+0000 7f4579dc7700 1 -- start start 2024-03-18T19:05:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f4579dc7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c003bc0 0x7f457413f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f4579dc7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45741316b0 0x7f4574141020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f4579dc7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4574141560 0x7f45741e7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f4579dc7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45741e7370 con 0x7f455c003bc0 2024-03-18T19:05:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f4579dc7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457413fac0 con 0x7f45741316b0 2024-03-18T19:05:51.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f4579dc7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45740cf0f0 con 0x7f4574141560 2024-03-18T19:05:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f45737fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c003bc0 0x7f457413f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f45737fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c003bc0 0x7f457413f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37042/0 (socket says 172.21.15.1:37042) 2024-03-18T19:05:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f45737fe700 1 -- 172.21.15.1:0/2540986403 learned_addr learned my addr 172.21.15.1:0/2540986403 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f4572ffd700 1 --2- 172.21.15.1:0/2540986403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45741316b0 0x7f4574141020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f45737fe700 1 -- 172.21.15.1:0/2540986403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4574141560 msgr2=0x7f45741e7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f4573fff700 1 --2- 172.21.15.1:0/2540986403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4574141560 0x7f45741e7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f45737fe700 1 --2- 172.21.15.1:0/2540986403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4574141560 0x7f45741e7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f45737fe700 1 -- 172.21.15.1:0/2540986403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45741316b0 msgr2=0x7f4574141020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f45737fe700 1 --2- 172.21.15.1:0/2540986403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45741316b0 0x7f4574141020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f45737fe700 1 -- 172.21.15.1:0/2540986403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45740cf3d0 con 0x7f455c003bc0 2024-03-18T19:05:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f4572ffd700 1 --2- 172.21.15.1:0/2540986403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45741316b0 0x7f4574141020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f4573fff700 1 --2- 172.21.15.1:0/2540986403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4574141560 0x7f45741e7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:51.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f45737fe700 1 --2- 172.21.15.1:0/2540986403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c003bc0 0x7f457413f580 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f455c00ba10 tx=0x7f457406f750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.064+0000 7f455bfff700 1 -- 172.21.15.1:0/2540986403 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4568004c80 con 0x7f455c003bc0 2024-03-18T19:05:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.065+0000 7f455bfff700 1 -- 172.21.15.1:0/2540986403 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4568004de0 con 0x7f455c003bc0 2024-03-18T19:05:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.065+0000 7f4579dc7700 1 -- 172.21.15.1:0/2540986403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45741e7370 con 0x7f455c003bc0 2024-03-18T19:05:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.065+0000 7f455bfff700 1 -- 172.21.15.1:0/2540986403 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45680072f0 con 0x7f455c003bc0 2024-03-18T19:05:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.065+0000 7f4579dc7700 1 -- 172.21.15.1:0/2540986403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45741419e0 con 0x7f455c003bc0 2024-03-18T19:05:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.066+0000 7f4579dc7700 1 -- 172.21.15.1:0/2540986403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45741e6e60 con 0x7f455c003bc0 2024-03-18T19:05:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.066+0000 7f455bfff700 1 -- 172.21.15.1:0/2540986403 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4568007450 con 0x7f455c003bc0 2024-03-18T19:05:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.066+0000 7f455bfff700 1 --2- 172.21.15.1:0/2540986403 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f456006adb0 0x7f456006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.067+0000 7f455bfff700 1 -- 172.21.15.1:0/2540986403 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f456808a7a0 con 0x7f455c003bc0 2024-03-18T19:05:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.067+0000 7f4572ffd700 1 --2- 172.21.15.1:0/2540986403 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f456006adb0 0x7f456006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.067+0000 7f4572ffd700 1 --2- 172.21.15.1:0/2540986403 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f456006adb0 0x7f456006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f457413f3f0 tx=0x7f4564008040 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:51.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.071+0000 7f455bfff700 1 -- 172.21.15.1:0/2540986403 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f45741e6e60 con 0x7f455c003bc0 2024-03-18T19:05:51.192 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:05:51.195 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:05:51.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.357+0000 7f4579dc7700 1 -- 172.21.15.1:0/2540986403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f45741e6e60 con 0x7f455c003bc0 2024-03-18T19:05:51.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.819+0000 7f455bfff700 1 -- 172.21.15.1:0/2540986403 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v594) v1 ==== 156+0+0 (secure 0 0 0) 0x7f45741e6e60 con 0x7f455c003bc0 2024-03-18T19:05:51.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.821+0000 7f4579dc7700 1 -- 172.21.15.1:0/2540986403 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f456006adb0 msgr2=0x7f456006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:51.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.821+0000 7f4579dc7700 1 --2- 172.21.15.1:0/2540986403 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f456006adb0 0x7f456006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f457413f3f0 tx=0x7f4564008040 comp rx=0 tx=0).stop 2024-03-18T19:05:51.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.821+0000 7f4579dc7700 1 -- 172.21.15.1:0/2540986403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c003bc0 msgr2=0x7f457413f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:51.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.821+0000 7f4579dc7700 1 --2- 172.21.15.1:0/2540986403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c003bc0 0x7f457413f580 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f455c00ba10 tx=0x7f457406f750 comp rx=0 tx=0).stop 2024-03-18T19:05:51.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.821+0000 7f4579dc7700 1 -- 172.21.15.1:0/2540986403 shutdown_connections 2024-03-18T19:05:51.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.821+0000 7f4579dc7700 1 --2- 172.21.15.1:0/2540986403 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f456006adb0 0x7f456006d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:51.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.821+0000 7f4579dc7700 1 --2- 172.21.15.1:0/2540986403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4574141560 0x7f45741e7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:51.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.821+0000 7f4579dc7700 1 --2- 172.21.15.1:0/2540986403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c003bc0 0x7f457413f580 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:51.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.821+0000 7f4579dc7700 1 --2- 172.21.15.1:0/2540986403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45741316b0 0x7f4574141020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:51.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.821+0000 7f4579dc7700 1 -- 172.21.15.1:0/2540986403 >> 172.21.15.1:0/2540986403 conn(0x7f45740b1d20 msgr2=0x7f45740ac940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:51.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.822+0000 7f4579dc7700 1 -- 172.21.15.1:0/2540986403 shutdown_connections 2024-03-18T19:05:51.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.822+0000 7f4579dc7700 1 -- 172.21.15.1:0/2540986403 wait complete. 2024-03-18T19:05:51.832 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2024-03-18T19:05:51.832 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2024-03-18T19:05:51.832 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-18T19:05:51.832 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2024-03-18T19:05:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.964+0000 7f88a6882700 1 Processor -- start 2024-03-18T19:05:51.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.964+0000 7f88a6882700 1 -- start start 2024-03-18T19:05:51.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.965+0000 7f88a6882700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a006a750 0x7f88a006ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:51.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.965+0000 7f88a6882700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f88a015d6b0 0x7f88a0160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:51.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.965+0000 7f88a6882700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a00acfb0 con 0x7f88a00ad790 2024-03-18T19:05:51.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.965+0000 7f88a6882700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88a00af7a0 con 0x7f88a015d6b0 2024-03-18T19:05:51.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.965+0000 7f88a6882700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a006b070 con 0x7f88a006a750 2024-03-18T19:05:51.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.965+0000 7f889ffff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a006a750 0x7f88a006ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:51.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.965+0000 7f889f7fe700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f88a015d6b0 0x7f88a0160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:51.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.965+0000 7f88a4e1f700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f88a00ad790 0x7f88a0161530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:52946/0 (socket says 172.21.15.1:52946) 2024-03-18T19:05:51.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.965+0000 7f889ffff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a006a750 0x7f88a006ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:49564/0 (socket says 172.21.15.1:49564) 2024-03-18T19:05:51.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.965+0000 7f88a4e1f700 1 -- 172.21.15.1:0/482883313 learned_addr learned my addr 172.21.15.1:0/482883313 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.965+0000 7f889ffff700 1 -- 172.21.15.1:0/482883313 >> v2:172.21.15.165:3300/0 conn(0x7f88a015d6b0 msgr2=0x7f88a0160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.965+0000 7f889ffff700 1 --2- 172.21.15.1:0/482883313 >> v2:172.21.15.165:3300/0 conn(0x7f88a015d6b0 0x7f88a0160ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.965+0000 7f889ffff700 1 -- 172.21.15.1:0/482883313 >> v1:172.21.15.165:6789/0 conn(0x7f88a00ad790 legacy=0x7f88a0161530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.965+0000 7f889ffff700 1 -- 172.21.15.1:0/482883313 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a00b50a0 con 0x7f88a006a750 2024-03-18T19:05:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.965+0000 7f889effd700 1 -- 172.21.15.1:0/482883313 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617441530 0 0) 0x7f88a00acfb0 con 0x7f88a00ad790 2024-03-18T19:05:51.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.965+0000 7f889f7fe700 1 --2- 172.21.15.1:0/482883313 >> v2:172.21.15.165:3300/0 conn(0x7f88a015d6b0 0x7f88a0160ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:51.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.966+0000 7f889ffff700 1 --2- 172.21.15.1:0/482883313 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a006a750 0x7f88a006ab30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8890009840 tx=0x7f8890009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2ccb07bad7d9eea server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:51.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.966+0000 7f889effd700 1 -- 172.21.15.1:0/482883313 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890011830 con 0x7f88a006a750 2024-03-18T19:05:51.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.966+0000 7f889effd700 1 -- 172.21.15.1:0/482883313 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8890011990 con 0x7f88a006a750 2024-03-18T19:05:51.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.966+0000 7f889effd700 1 -- 172.21.15.1:0/482883313 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890011d20 con 0x7f88a006a750 2024-03-18T19:05:51.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.966+0000 7f88a6882700 1 -- 172.21.15.1:0/482883313 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a006a750 msgr2=0x7f88a006ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:51.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.966+0000 7f88a6882700 1 --2- 172.21.15.1:0/482883313 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a006a750 0x7f88a006ab30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8890009840 tx=0x7f8890009b50 comp rx=0 tx=0).stop 2024-03-18T19:05:51.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.966+0000 7f88a6882700 1 -- 172.21.15.1:0/482883313 shutdown_connections 2024-03-18T19:05:51.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.966+0000 7f88a6882700 1 --2- 172.21.15.1:0/482883313 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a006a750 0x7f88a006ab30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:51.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.966+0000 7f88a6882700 1 --2- 172.21.15.1:0/482883313 >> v2:172.21.15.165:3300/0 conn(0x7f88a015d6b0 0x7f88a0160ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:51.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.966+0000 7f88a6882700 1 -- 172.21.15.1:0/482883313 >> 172.21.15.1:0/482883313 conn(0x7f88a006b860 msgr2=0x7f88a006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:51.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.967+0000 7f88a6882700 1 -- 172.21.15.1:0/482883313 shutdown_connections 2024-03-18T19:05:51.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.967+0000 7f88a6882700 1 -- 172.21.15.1:0/482883313 wait complete. 2024-03-18T19:05:51.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.968+0000 7f88a6882700 1 Processor -- start 2024-03-18T19:05:51.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.968+0000 7f88a6882700 1 -- start start 2024-03-18T19:05:51.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f88a6882700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88a006a750 0x7f88a00cc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:51.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f88a6882700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a00ad790 0x7f88a015c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:51.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f88a6882700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a015d6b0 0x7f88a00cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:51.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f88a6882700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a00cf1e0 con 0x7f88a015d6b0 2024-03-18T19:05:51.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f88a6882700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a01488f0 con 0x7f88a006a750 2024-03-18T19:05:51.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f88a6882700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a0148bd0 con 0x7f88a00ad790 2024-03-18T19:05:51.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f88a4e1f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a015d6b0 0x7f88a00cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:51.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f88a4e1f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a015d6b0 0x7f88a00cb5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37046/0 (socket says 172.21.15.1:37046) 2024-03-18T19:05:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f88a4e1f700 1 -- 172.21.15.1:0/3797477249 learned_addr learned my addr 172.21.15.1:0/3797477249 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f889ffff700 1 --2- 172.21.15.1:0/3797477249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88a006a750 0x7f88a00cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f889f7fe700 1 --2- 172.21.15.1:0/3797477249 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a00ad790 0x7f88a015c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f88a4e1f700 1 -- 172.21.15.1:0/3797477249 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a00ad790 msgr2=0x7f88a015c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f88a4e1f700 1 --2- 172.21.15.1:0/3797477249 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a00ad790 0x7f88a015c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f88a4e1f700 1 -- 172.21.15.1:0/3797477249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88a006a750 msgr2=0x7f88a00cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f88a4e1f700 1 --2- 172.21.15.1:0/3797477249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88a006a750 0x7f88a00cc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f88a4e1f700 1 -- 172.21.15.1:0/3797477249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88940094a0 con 0x7f88a015d6b0 2024-03-18T19:05:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f889ffff700 1 --2- 172.21.15.1:0/3797477249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88a006a750 0x7f88a00cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f88a4e1f700 1 --2- 172.21.15.1:0/3797477249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a015d6b0 0x7f88a00cb5f0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f8898002230 tx=0x7f889800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f889cff9700 1 -- 172.21.15.1:0/3797477249 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889800ac20 con 0x7f88a015d6b0 2024-03-18T19:05:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f889cff9700 1 -- 172.21.15.1:0/3797477249 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889800ada0 con 0x7f88a015d6b0 2024-03-18T19:05:51.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f88a6882700 1 -- 172.21.15.1:0/3797477249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88900094a0 con 0x7f88a015d6b0 2024-03-18T19:05:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f889cff9700 1 -- 172.21.15.1:0/3797477249 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8898007310 con 0x7f88a015d6b0 2024-03-18T19:05:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.969+0000 7f88a6882700 1 -- 172.21.15.1:0/3797477249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a0149190 con 0x7f88a015d6b0 2024-03-18T19:05:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.970+0000 7f88a6882700 1 -- 172.21.15.1:0/3797477249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a00ad420 con 0x7f88a015d6b0 2024-03-18T19:05:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.971+0000 7f889cff9700 1 -- 172.21.15.1:0/3797477249 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8898007470 con 0x7f88a015d6b0 2024-03-18T19:05:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.971+0000 7f889cff9700 1 --2- 172.21.15.1:0/3797477249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f888c06adb0 0x7f888c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.971+0000 7f889cff9700 1 -- 172.21.15.1:0/3797477249 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f88980890a0 con 0x7f88a015d6b0 2024-03-18T19:05:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.971+0000 7f889ffff700 1 --2- 172.21.15.1:0/3797477249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f888c06adb0 0x7f888c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.972+0000 7f889ffff700 1 --2- 172.21.15.1:0/3797477249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f888c06adb0 0x7f888c06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f88900090b0 tx=0x7f889001c040 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:51.976+0000 7f889cff9700 1 -- 172.21.15.1:0/3797477249 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f88a00ad420 con 0x7f88a015d6b0 2024-03-18T19:05:52.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.263+0000 7f88a6882700 1 -- 172.21.15.1:0/3797477249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7f88a00ad420 con 0x7f88a015d6b0 2024-03-18T19:05:52.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.821+0000 7f889cff9700 1 -- 172.21.15.1:0/3797477249 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v595) v1 ==== 169+0+0 (secure 0 0 0) 0x7f88a00ad420 con 0x7f88a015d6b0 2024-03-18T19:05:52.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.823+0000 7f88a6882700 1 -- 172.21.15.1:0/3797477249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f888c06adb0 msgr2=0x7f888c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:52.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.823+0000 7f88a6882700 1 --2- 172.21.15.1:0/3797477249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f888c06adb0 0x7f888c06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f88900090b0 tx=0x7f889001c040 comp rx=0 tx=0).stop 2024-03-18T19:05:52.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.824+0000 7f88a6882700 1 -- 172.21.15.1:0/3797477249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a015d6b0 msgr2=0x7f88a00cb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:52.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.824+0000 7f88a6882700 1 --2- 172.21.15.1:0/3797477249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a015d6b0 0x7f88a00cb5f0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f8898002230 tx=0x7f889800a060 comp rx=0 tx=0).stop 2024-03-18T19:05:52.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.824+0000 7f88a6882700 1 -- 172.21.15.1:0/3797477249 shutdown_connections 2024-03-18T19:05:52.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.824+0000 7f88a6882700 1 --2- 172.21.15.1:0/3797477249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f888c06adb0 0x7f888c06d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:52.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.824+0000 7f88a6882700 1 --2- 172.21.15.1:0/3797477249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a015d6b0 0x7f88a00cb5f0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:52.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.824+0000 7f88a6882700 1 --2- 172.21.15.1:0/3797477249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88a006a750 0x7f88a00cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:52.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.824+0000 7f88a6882700 1 --2- 172.21.15.1:0/3797477249 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a00ad790 0x7f88a015c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:52.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.824+0000 7f88a6882700 1 -- 172.21.15.1:0/3797477249 >> 172.21.15.1:0/3797477249 conn(0x7f88a006b860 msgr2=0x7f88a015fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:52.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.824+0000 7f88a6882700 1 -- 172.21.15.1:0/3797477249 shutdown_connections 2024-03-18T19:05:52.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.824+0000 7f88a6882700 1 -- 172.21.15.1:0/3797477249 wait complete. 2024-03-18T19:05:52.836 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-18T19:05:52.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.974+0000 7f1022893700 1 Processor -- start 2024-03-18T19:05:52.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.974+0000 7f1022893700 1 -- start start 2024-03-18T19:05:52.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.974+0000 7f1022893700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f101c15aee0 0x7f101c141390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:52.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.974+0000 7f1022893700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f101c15b2c0 0x7f101c0cf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:52.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.974+0000 7f1022893700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f101c073d70 0x7f101c0cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:52.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.974+0000 7f1022893700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f101c1418d0 con 0x7f101c15b2c0 2024-03-18T19:05:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.974+0000 7f1022893700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f101c15ec90 con 0x7f101c073d70 2024-03-18T19:05:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.974+0000 7f1022893700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f101c15ef70 con 0x7f101c15aee0 2024-03-18T19:05:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.975+0000 7f101b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f101c15b2c0 0x7f101c0cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.975+0000 7f101b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f101c15b2c0 0x7f101c0cf620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37048/0 (socket says 172.21.15.1:37048) 2024-03-18T19:05:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.975+0000 7f101b7fe700 1 -- 172.21.15.1:0/1352485816 learned_addr learned my addr 172.21.15.1:0/1352485816 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.975+0000 7f101b7fe700 1 -- 172.21.15.1:0/1352485816 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f101c15aee0 msgr2=0x7f101c141390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.975+0000 7f1020e30700 1 --2- 172.21.15.1:0/1352485816 >> v2:172.21.15.165:3300/0 conn(0x7f101c073d70 0x7f101c0cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.975+0000 7f101bfff700 1 --2- 172.21.15.1:0/1352485816 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f101c15aee0 0x7f101c141390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.975+0000 7f101b7fe700 1 --2- 172.21.15.1:0/1352485816 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f101c15aee0 0x7f101c141390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.975+0000 7f101b7fe700 1 -- 172.21.15.1:0/1352485816 >> v2:172.21.15.165:3300/0 conn(0x7f101c073d70 msgr2=0x7f101c0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.975+0000 7f101b7fe700 1 --2- 172.21.15.1:0/1352485816 >> v2:172.21.15.165:3300/0 conn(0x7f101c073d70 0x7f101c0cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:52.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.975+0000 7f101b7fe700 1 -- 172.21.15.1:0/1352485816 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c15f250 con 0x7f101c15b2c0 2024-03-18T19:05:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.975+0000 7f1020e30700 1 --2- 172.21.15.1:0/1352485816 >> v2:172.21.15.165:3300/0 conn(0x7f101c073d70 0x7f101c0cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:05:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.975+0000 7f101b7fe700 1 --2- 172.21.15.1:0/1352485816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f101c15b2c0 0x7f101c0cf620 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f101c0b26e0 tx=0x7f1010009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b63db24cc015b962 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.975+0000 7f101affd700 1 -- 172.21.15.1:0/1352485816 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101000b870 con 0x7f101c15b2c0 2024-03-18T19:05:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.975+0000 7f101affd700 1 -- 172.21.15.1:0/1352485816 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101000b9d0 con 0x7f101c15b2c0 2024-03-18T19:05:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.975+0000 7f101affd700 1 -- 172.21.15.1:0/1352485816 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101000bd60 con 0x7f101c15b2c0 2024-03-18T19:05:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.976+0000 7f1022893700 1 -- 172.21.15.1:0/1352485816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f101c15b2c0 msgr2=0x7f101c0cf620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.976+0000 7f1022893700 1 --2- 172.21.15.1:0/1352485816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f101c15b2c0 0x7f101c0cf620 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f101c0b26e0 tx=0x7f1010009bd0 comp rx=0 tx=0).stop 2024-03-18T19:05:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.976+0000 7f1022893700 1 -- 172.21.15.1:0/1352485816 shutdown_connections 2024-03-18T19:05:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.976+0000 7f1022893700 1 --2- 172.21.15.1:0/1352485816 >> v2:172.21.15.165:3300/0 conn(0x7f101c073d70 0x7f101c0cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.976+0000 7f1022893700 1 --2- 172.21.15.1:0/1352485816 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f101c15aee0 0x7f101c141390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.976+0000 7f1022893700 1 --2- 172.21.15.1:0/1352485816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f101c15b2c0 0x7f101c0cf620 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:52.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.976+0000 7f1022893700 1 -- 172.21.15.1:0/1352485816 >> 172.21.15.1:0/1352485816 conn(0x7f101c066930 msgr2=0x7f101c06a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:52.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.976+0000 7f1022893700 1 -- 172.21.15.1:0/1352485816 shutdown_connections 2024-03-18T19:05:52.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.976+0000 7f1022893700 1 -- 172.21.15.1:0/1352485816 wait complete. 2024-03-18T19:05:52.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.977+0000 7f1022893700 1 Processor -- start 2024-03-18T19:05:52.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.977+0000 7f1022893700 1 -- start start 2024-03-18T19:05:52.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.978+0000 7f1022893700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f101c073d70 0x7f101c143a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:52.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.978+0000 7f1022893700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f101c15aee0 0x7f101c1f0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.978+0000 7f1022893700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f101c15b2c0 0x7f101c13ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.978+0000 7f1022893700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101c15b710 con 0x7f101c15b2c0 2024-03-18T19:05:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.978+0000 7f1022893700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101c1f1180 con 0x7f101c15aee0 2024-03-18T19:05:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.978+0000 7f1022893700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f101c13b0d0 con 0x7f101c073d70 2024-03-18T19:05:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.978+0000 7f1020e30700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f101c15b2c0 0x7f101c13ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.978+0000 7f1020e30700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f101c15b2c0 0x7f101c13ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37054/0 (socket says 172.21.15.1:37054) 2024-03-18T19:05:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.978+0000 7f1020e30700 1 -- 172.21.15.1:0/2687966167 learned_addr learned my addr 172.21.15.1:0/2687966167 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.978+0000 7f101b7fe700 1 --2- 172.21.15.1:0/2687966167 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f101c15aee0 0x7f101c1f0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.978+0000 7f101bfff700 1 --2- 172.21.15.1:0/2687966167 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f101c073d70 0x7f101c143a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.978+0000 7f1020e30700 1 -- 172.21.15.1:0/2687966167 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f101c073d70 msgr2=0x7f101c143a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.978+0000 7f1020e30700 1 --2- 172.21.15.1:0/2687966167 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f101c073d70 0x7f101c143a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.978+0000 7f1020e30700 1 -- 172.21.15.1:0/2687966167 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f101c15aee0 msgr2=0x7f101c1f0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.978+0000 7f1020e30700 1 --2- 172.21.15.1:0/2687966167 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f101c15aee0 0x7f101c1f0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:52.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.979+0000 7f1020e30700 1 -- 172.21.15.1:0/2687966167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1010009550 con 0x7f101c15b2c0 2024-03-18T19:05:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.979+0000 7f1020e30700 1 --2- 172.21.15.1:0/2687966167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f101c15b2c0 0x7f101c13ab90 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f101400ce80 tx=0x7f101400ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.979+0000 7f1018ff9700 1 -- 172.21.15.1:0/2687966167 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101400ad40 con 0x7f101c15b2c0 2024-03-18T19:05:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.979+0000 7f1018ff9700 1 -- 172.21.15.1:0/2687966167 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1014015440 con 0x7f101c15b2c0 2024-03-18T19:05:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.979+0000 7f1022893700 1 -- 172.21.15.1:0/2687966167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f101c1f1420 con 0x7f101c15b2c0 2024-03-18T19:05:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.979+0000 7f1018ff9700 1 -- 172.21.15.1:0/2687966167 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101400aea0 con 0x7f101c15b2c0 2024-03-18T19:05:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.979+0000 7f1022893700 1 -- 172.21.15.1:0/2687966167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f101c1f19e0 con 0x7f101c15b2c0 2024-03-18T19:05:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.980+0000 7f1022893700 1 -- 172.21.15.1:0/2687966167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f101c0676a0 con 0x7f101c15b2c0 2024-03-18T19:05:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.980+0000 7f1018ff9700 1 -- 172.21.15.1:0/2687966167 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f101400d430 con 0x7f101c15b2c0 2024-03-18T19:05:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.981+0000 7f1018ff9700 1 --2- 172.21.15.1:0/2687966167 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f100806adb0 0x7f100806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.981+0000 7f1018ff9700 1 -- 172.21.15.1:0/2687966167 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f10140899e0 con 0x7f101c15b2c0 2024-03-18T19:05:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.981+0000 7f101bfff700 1 --2- 172.21.15.1:0/2687966167 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f100806adb0 0x7f100806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:52.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.981+0000 7f101bfff700 1 --2- 172.21.15.1:0/2687966167 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f100806adb0 0x7f100806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f101c143800 tx=0x7f100c008040 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:52.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:52.985+0000 7f1018ff9700 1 -- 172.21.15.1:0/2687966167 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1014054a00 con 0x7f101c15b2c0 2024-03-18T19:05:53.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.188+0000 7f1022893700 1 -- 172.21.15.1:0/2687966167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f101c0cf020 con 0x7f101c15b2c0 2024-03-18T19:05:53.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.192+0000 7f1018ff9700 1 -- 172.21.15.1:0/2687966167 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f101401a020 con 0x7f101c15b2c0 2024-03-18T19:05:53.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.194+0000 7f1022893700 1 -- 172.21.15.1:0/2687966167 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f100806adb0 msgr2=0x7f100806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:53.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.194+0000 7f1022893700 1 --2- 172.21.15.1:0/2687966167 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f100806adb0 0x7f100806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f101c143800 tx=0x7f100c008040 comp rx=0 tx=0).stop 2024-03-18T19:05:53.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.194+0000 7f1022893700 1 -- 172.21.15.1:0/2687966167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f101c15b2c0 msgr2=0x7f101c13ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:53.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.194+0000 7f1022893700 1 --2- 172.21.15.1:0/2687966167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f101c15b2c0 0x7f101c13ab90 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f101400ce80 tx=0x7f101400ceb0 comp rx=0 tx=0).stop 2024-03-18T19:05:53.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.194+0000 7f1022893700 1 -- 172.21.15.1:0/2687966167 shutdown_connections 2024-03-18T19:05:53.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.194+0000 7f1022893700 1 --2- 172.21.15.1:0/2687966167 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f100806adb0 0x7f100806d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:53.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.194+0000 7f1022893700 1 --2- 172.21.15.1:0/2687966167 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f101c073d70 0x7f101c143a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:53.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.194+0000 7f1022893700 1 --2- 172.21.15.1:0/2687966167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f101c15b2c0 0x7f101c13ab90 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:53.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.194+0000 7f1022893700 1 --2- 172.21.15.1:0/2687966167 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f101c15aee0 0x7f101c1f0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:53.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.194+0000 7f1022893700 1 -- 172.21.15.1:0/2687966167 >> 172.21.15.1:0/2687966167 conn(0x7f101c066930 msgr2=0x7f101c0cfaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:53.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.194+0000 7f1022893700 1 -- 172.21.15.1:0/2687966167 shutdown_connections 2024-03-18T19:05:53.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.194+0000 7f1022893700 1 -- 172.21.15.1:0/2687966167 wait complete. 2024-03-18T19:05:53.203 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T19:05:53.203 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:05:53.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:05:53.292 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T19:05:53.292 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:05:53.292 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:05:53.311 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:05:53.311 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:05:53.312 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:05:53.334 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:05:53.334 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:05:53.334 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:05:53.370 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:05:53.370 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:05:53.370 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:05:53.394 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:05:53.394 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-18T19:05:53.395 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:05:53.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.538+0000 7f79a5073700 1 Processor -- start 2024-03-18T19:05:53.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.538+0000 7f79a5073700 1 -- start start 2024-03-18T19:05:53.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f79a5073700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f79a01465a0 0x7f79a0168660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:53.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f79a5073700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79a0072660 con 0x7f79a0146190 2024-03-18T19:05:53.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f79a5073700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79a006ebd0 con 0x7f79a006ae60 2024-03-18T19:05:53.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f79a5073700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79a006f850 con 0x7f79a01465a0 2024-03-18T19:05:53.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f799ed9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f79a0146190 0x7f79a006a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55168/0 (socket says 172.21.15.1:55168) 2024-03-18T19:05:53.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f799ed9d700 1 -- 172.21.15.1:0/1475457526 learned_addr learned my addr 172.21.15.1:0/1475457526 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:53.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f799dd9b700 1 -- 172.21.15.1:0/1475457526 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860377471 0 0) 0x7f79a0072660 con 0x7f79a0146190 2024-03-18T19:05:53.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f799dd9b700 1 -- 172.21.15.1:0/1475457526 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79880035b0 con 0x7f79a0146190 2024-03-18T19:05:53.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f799f59e700 1 --2- 172.21.15.1:0/1475457526 >> v2:172.21.15.165:3300/0 conn(0x7f79a01465a0 0x7f79a0168660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:53.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f799dd9b700 1 -- 172.21.15.1:0/1475457526 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1891393904 0 0) 0x7f79880035b0 con 0x7f79a0146190 2024-03-18T19:05:53.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f799dd9b700 1 -- 172.21.15.1:0/1475457526 >> v2:172.21.15.165:3300/0 conn(0x7f79a01465a0 msgr2=0x7f79a0168660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:53.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f799dd9b700 1 --2- 172.21.15.1:0/1475457526 >> v2:172.21.15.165:3300/0 conn(0x7f79a01465a0 0x7f79a0168660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:53.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f799dd9b700 1 -- 172.21.15.1:0/1475457526 >> v1:172.21.15.165:6789/0 conn(0x7f79a006ae60 legacy=0x7f79a0164f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:53.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f799dd9b700 1 -- 172.21.15.1:0/1475457526 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a006eeb0 con 0x7f79a0146190 2024-03-18T19:05:53.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f799dd9b700 1 -- 172.21.15.1:0/1475457526 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f7990001670 con 0x7f79a0146190 2024-03-18T19:05:53.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f799dd9b700 1 -- 172.21.15.1:0/1475457526 >> v1:172.21.15.1:6789/0 conn(0x7f79a0146190 legacy=0x7f79a006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:53.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f799dd9b700 1 --2- 172.21.15.1:0/1475457526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7988003a50 0x7f7988003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:53.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f799dd9b700 1 --2- 172.21.15.1:0/1475457526 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7988004850 0x7f7988004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:53.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f799dd9b700 1 --2- 172.21.15.1:0/1475457526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7988005260 0x7f798800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:53.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f799dd9b700 1 -- 172.21.15.1:0/1475457526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a006f850 con 0x7f7988005260 2024-03-18T19:05:53.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f799dd9b700 1 -- 172.21.15.1:0/1475457526 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798800c1b0 con 0x7f7988003a50 2024-03-18T19:05:53.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.539+0000 7f799dd9b700 1 -- 172.21.15.1:0/1475457526 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798800c370 con 0x7f7988004850 2024-03-18T19:05:53.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.540+0000 7f799f59e700 1 --2- 172.21.15.1:0/1475457526 >> v2:172.21.15.165:3300/0 conn(0x7f79a01465a0 0x7f79a0168660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:53.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.540+0000 7f799f59e700 1 --2- 172.21.15.1:0/1475457526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7988005260 0x7f798800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:53.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.540+0000 7f799e59c700 1 --2- 172.21.15.1:0/1475457526 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7988004850 0x7f7988004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:53.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.540+0000 7f799ed9d700 1 --2- 172.21.15.1:0/1475457526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7988003a50 0x7f7988003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:53.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.540+0000 7f799f59e700 1 -- 172.21.15.1:0/1475457526 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7988004850 msgr2=0x7f7988004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:53.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.540+0000 7f799f59e700 1 --2- 172.21.15.1:0/1475457526 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7988004850 0x7f7988004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:53.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.540+0000 7f799f59e700 1 -- 172.21.15.1:0/1475457526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7988003a50 msgr2=0x7f7988003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:53.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.540+0000 7f799f59e700 1 --2- 172.21.15.1:0/1475457526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7988003a50 0x7f7988003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:53.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.540+0000 7f799f59e700 1 -- 172.21.15.1:0/1475457526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79880035b0 con 0x7f7988005260 2024-03-18T19:05:53.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.540+0000 7f799e59c700 1 --2- 172.21.15.1:0/1475457526 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7988004850 0x7f7988004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:05:53.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.540+0000 7f799f59e700 1 -- 172.21.15.1:0/1475457526 reap_dead start 2024-03-18T19:05:53.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.540+0000 7f799f59e700 1 --2- 172.21.15.1:0/1475457526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7988005260 0x7f798800ba10 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f79980073e0 tx=0x7f7988005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ad6a8ae9a60b9f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:53.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.540+0000 7f799dd9b700 1 -- 172.21.15.1:0/1475457526 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7998009490 con 0x7f7988005260 2024-03-18T19:05:53.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.540+0000 7f799dd9b700 1 -- 172.21.15.1:0/1475457526 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7998014400 con 0x7f7988005260 2024-03-18T19:05:53.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.540+0000 7f799dd9b700 1 -- 172.21.15.1:0/1475457526 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7998014770 con 0x7f7988005260 2024-03-18T19:05:53.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.541+0000 7f79a5073700 1 -- 172.21.15.1:0/1475457526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7988005260 msgr2=0x7f798800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:53.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.541+0000 7f79a5073700 1 --2- 172.21.15.1:0/1475457526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7988005260 0x7f798800ba10 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f79980073e0 tx=0x7f7988005880 comp rx=0 tx=0).stop 2024-03-18T19:05:53.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.541+0000 7f79a5073700 1 -- 172.21.15.1:0/1475457526 shutdown_connections 2024-03-18T19:05:53.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.541+0000 7f79a5073700 1 --2- 172.21.15.1:0/1475457526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7988005260 0x7f798800ba10 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:53.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.541+0000 7f79a5073700 1 -- 172.21.15.1:0/1475457526 >> 172.21.15.1:0/1475457526 conn(0x7f79a00b1d20 msgr2=0x7f79a00b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:53.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.541+0000 7f79a5073700 1 -- 172.21.15.1:0/1475457526 shutdown_connections 2024-03-18T19:05:53.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.541+0000 7f79a5073700 1 -- 172.21.15.1:0/1475457526 wait complete. 2024-03-18T19:05:53.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.542+0000 7f79a5073700 1 Processor -- start 2024-03-18T19:05:53.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.542+0000 7f79a5073700 1 -- start start 2024-03-18T19:05:53.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.543+0000 7f79a5073700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7988005260 0x7f79a00cf260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:53.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.543+0000 7f79a5073700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79a013e560 0x7f79a00cca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:53.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.543+0000 7f79a5073700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79a01410e0 0x7f79a0141570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:53.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.543+0000 7f79a5073700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a00cf840 con 0x7f79a013e560 2024-03-18T19:05:53.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.543+0000 7f79a5073700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a0141ab0 con 0x7f79a01410e0 2024-03-18T19:05:53.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.543+0000 7f79a5073700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79a01f4060 con 0x7f7988005260 2024-03-18T19:05:53.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.543+0000 7f799e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79a013e560 0x7f79a00cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.543+0000 7f799e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79a013e560 0x7f79a00cca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37072/0 (socket says 172.21.15.1:37072) 2024-03-18T19:05:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.543+0000 7f799e59c700 1 -- 172.21.15.1:0/1160263951 learned_addr learned my addr 172.21.15.1:0/1160263951 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.543+0000 7f799ed9d700 1 --2- 172.21.15.1:0/1160263951 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7988005260 0x7f79a00cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.543+0000 7f799f59e700 1 --2- 172.21.15.1:0/1160263951 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79a01410e0 0x7f79a0141570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.543+0000 7f799e59c700 1 -- 172.21.15.1:0/1160263951 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7988005260 msgr2=0x7f79a00cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.543+0000 7f799e59c700 1 --2- 172.21.15.1:0/1160263951 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7988005260 0x7f79a00cf260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.543+0000 7f799e59c700 1 -- 172.21.15.1:0/1160263951 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79a01410e0 msgr2=0x7f79a0141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.543+0000 7f799e59c700 1 --2- 172.21.15.1:0/1160263951 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79a01410e0 0x7f79a0141570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.543+0000 7f799e59c700 1 -- 172.21.15.1:0/1160263951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a01f3aa0 con 0x7f79a013e560 2024-03-18T19:05:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.543+0000 7f799e59c700 1 --2- 172.21.15.1:0/1160263951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79a013e560 0x7f79a00cca00 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f799400e550 tx=0x7f799400e860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.544+0000 7f79877fe700 1 -- 172.21.15.1:0/1160263951 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799400eed0 con 0x7f79a013e560 2024-03-18T19:05:53.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.544+0000 7f79877fe700 1 -- 172.21.15.1:0/1160263951 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79940052e0 con 0x7f79a013e560 2024-03-18T19:05:53.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.544+0000 7f79a5073700 1 -- 172.21.15.1:0/1160263951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79a01f3d80 con 0x7f79a013e560 2024-03-18T19:05:53.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.544+0000 7f79877fe700 1 -- 172.21.15.1:0/1160263951 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7994003680 con 0x7f79a013e560 2024-03-18T19:05:53.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.544+0000 7f79a5073700 1 -- 172.21.15.1:0/1160263951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a013e2a0 con 0x7f79a013e560 2024-03-18T19:05:53.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.545+0000 7f79a5073700 1 -- 172.21.15.1:0/1160263951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79a013f4b0 con 0x7f79a013e560 2024-03-18T19:05:53.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.545+0000 7f79877fe700 1 -- 172.21.15.1:0/1160263951 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f79940037e0 con 0x7f79a013e560 2024-03-18T19:05:53.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.546+0000 7f79877fe700 1 --2- 172.21.15.1:0/1160263951 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f798c06adb0 0x7f798c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:53.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.546+0000 7f79877fe700 1 -- 172.21.15.1:0/1160263951 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f7994088ad0 con 0x7f79a013e560 2024-03-18T19:05:53.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.546+0000 7f799ed9d700 1 --2- 172.21.15.1:0/1160263951 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f798c06adb0 0x7f798c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:53.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.546+0000 7f799ed9d700 1 --2- 172.21.15.1:0/1160263951 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f798c06adb0 0x7f798c06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7990000c00 tx=0x7f7990000ed0 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:53.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.550+0000 7f79877fe700 1 -- 172.21.15.1:0/1160263951 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f79a013f4b0 con 0x7f79a013e560 2024-03-18T19:05:53.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.796+0000 7f79a5073700 1 -- 172.21.15.1:0/1160263951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79a013f4b0 con 0x7f79a013e560 2024-03-18T19:05:53.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.797+0000 7f79877fe700 1 -- 172.21.15.1:0/1160263951 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+3526 (secure 0 0 0) 0x7f79a013f4b0 con 0x7f79a013e560 2024-03-18T19:05:53.794 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:05:53.794 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":35,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5744,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2437290126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2437290126},{"type":"v1","addr":"172.21.15.1:6836","nonce":2437290126}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5747,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2832642785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2832642785},{"type":"v1","addr":"172.21.15.1:6837","nonce":2832642785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5756,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3765527492","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3765527492},{"type":"v1","addr":"172.21.15.165:6834","nonce":3765527492}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6742,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/644402620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":644402620},{"type":"v1","addr":"172.21.15.165:6835","nonce":644402620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[]} 2024-03-18T19:05:53.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.799+0000 7f79a5073700 1 -- 172.21.15.1:0/1160263951 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f798c06adb0 msgr2=0x7f798c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:53.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.799+0000 7f79a5073700 1 --2- 172.21.15.1:0/1160263951 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f798c06adb0 0x7f798c06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7990000c00 tx=0x7f7990000ed0 comp rx=0 tx=0).stop 2024-03-18T19:05:53.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.799+0000 7f79a5073700 1 -- 172.21.15.1:0/1160263951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79a013e560 msgr2=0x7f79a00cca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:53.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.799+0000 7f79a5073700 1 --2- 172.21.15.1:0/1160263951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79a013e560 0x7f79a00cca00 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f799400e550 tx=0x7f799400e860 comp rx=0 tx=0).stop 2024-03-18T19:05:53.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.799+0000 7f79a5073700 1 -- 172.21.15.1:0/1160263951 shutdown_connections 2024-03-18T19:05:53.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.799+0000 7f79a5073700 1 --2- 172.21.15.1:0/1160263951 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f798c06adb0 0x7f798c06d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:53.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.799+0000 7f79a5073700 1 --2- 172.21.15.1:0/1160263951 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7988005260 0x7f79a00cf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:53.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.799+0000 7f79a5073700 1 --2- 172.21.15.1:0/1160263951 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79a01410e0 0x7f79a0141570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:53.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.799+0000 7f79a5073700 1 --2- 172.21.15.1:0/1160263951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79a013e560 0x7f79a00cca00 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:53.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.799+0000 7f79a5073700 1 -- 172.21.15.1:0/1160263951 >> 172.21.15.1:0/1160263951 conn(0x7f79a00b1d20 msgr2=0x7f79a00afc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:53.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.799+0000 7f79a5073700 1 -- 172.21.15.1:0/1160263951 shutdown_connections 2024-03-18T19:05:53.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:53.799+0000 7f79a5073700 1 -- 172.21.15.1:0/1160263951 wait complete. 2024-03-18T19:05:53.797 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 35 2024-03-18T19:05:53.808 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T19:05:53.808 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-18T19:05:53.809 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T19:05:53.809 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:05:53.809 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T19:05:53.809 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-18T19:05:53.809 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-18T19:05:53.809 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:05:53.810 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T19:05:53.810 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-18T19:05:53.810 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-18T19:05:53.810 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:05:53.810 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T19:05:53.810 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-18T19:05:53.811 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-18T19:05:53.811 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:05:53.818 INFO:tasks.ceph.mds.c:Stopped 2024-03-18T19:05:53.818 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-18T19:05:53.822 INFO:tasks.ceph.mds.d:Stopped 2024-03-18T19:05:53.822 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-18T19:05:53.861 INFO:tasks.ceph.mds.c:Started 2024-03-18T19:05:53.861 DEBUG:teuthology.parallel:result is None 2024-03-18T19:05:53.865 INFO:tasks.ceph.mds.d:Started 2024-03-18T19:05:53.865 DEBUG:teuthology.parallel:result is None 2024-03-18T19:05:53.939 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T19:05:53.950 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-03-18T19:05:57.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-03-18T19:05:57.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-18T19:05:59.818 INFO:tasks.ceph.mds.a:Stopped 2024-03-18T19:05:59.818 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-18T19:05:59.821 INFO:tasks.ceph.mds.a:Started 2024-03-18T19:05:59.822 DEBUG:teuthology.parallel:result is None 2024-03-18T19:05:59.822 INFO:tasks.ceph.mds.b:Stopped 2024-03-18T19:05:59.822 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-18T19:05:59.825 INFO:tasks.ceph.mds.b:Started 2024-03-18T19:05:59.825 DEBUG:teuthology.parallel:result is None 2024-03-18T19:05:59.825 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-18T19:05:59.912 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-03-18T19:05:59.914 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-18T19:05:59.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.972+0000 7fec0d829700 1 Processor -- start 2024-03-18T19:05:59.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.972+0000 7fec0d829700 1 -- start start 2024-03-18T19:05:59.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.973+0000 7fec0d829700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fec080b1e20 0x7fec080cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.973+0000 7fec0d829700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec080d0df0 con 0x7fec080cd870 2024-03-18T19:05:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.973+0000 7fec0d829700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec080d0550 con 0x7fec080cbce0 2024-03-18T19:05:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.973+0000 7fec0d829700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec080d0830 con 0x7fec080b1e20 2024-03-18T19:05:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.973+0000 7fec06ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fec080cd870 0x7fec080cdc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47016/0 (socket says 172.21.15.1:47016) 2024-03-18T19:05:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.973+0000 7fec06ffd700 1 -- 172.21.15.1:0/1439833283 learned_addr learned my addr 172.21.15.1:0/1439833283 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:05:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.973+0000 7fec067fc700 1 --2- 172.21.15.1:0/1439833283 >> v2:172.21.15.165:3300/0 conn(0x7fec080b1e20 0x7fec080cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.973+0000 7fec05ffb700 1 -- 172.21.15.1:0/1439833283 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994256438 0 0) 0x7fec080d0df0 con 0x7fec080cd870 2024-03-18T19:05:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.973+0000 7fec05ffb700 1 -- 172.21.15.1:0/1439833283 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febf00035b0 con 0x7fec080cd870 2024-03-18T19:05:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.973+0000 7fec05ffb700 1 -- 172.21.15.1:0/1439833283 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715563903 0 0) 0x7fec080d0550 con 0x7fec080cbce0 2024-03-18T19:05:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.973+0000 7fec05ffb700 1 -- 172.21.15.1:0/1439833283 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec080d0df0 con 0x7fec080cbce0 2024-03-18T19:05:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.973+0000 7fec05ffb700 1 -- 172.21.15.1:0/1439833283 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2596130569 0 0) 0x7febf00035b0 con 0x7fec080cd870 2024-03-18T19:05:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.973+0000 7fec05ffb700 1 -- 172.21.15.1:0/1439833283 >> v2:172.21.15.165:3300/0 conn(0x7fec080b1e20 msgr2=0x7fec080cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec05ffb700 1 --2- 172.21.15.1:0/1439833283 >> v2:172.21.15.165:3300/0 conn(0x7fec080b1e20 0x7fec080cff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:59.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec05ffb700 1 -- 172.21.15.1:0/1439833283 >> v1:172.21.15.165:6789/0 conn(0x7fec080cbce0 legacy=0x7fec080cc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:59.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec05ffb700 1 -- 172.21.15.1:0/1439833283 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec080d0b10 con 0x7fec080cd870 2024-03-18T19:05:59.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec05ffb700 1 -- 172.21.15.1:0/1439833283 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7febf8001670 con 0x7fec080cd870 2024-03-18T19:05:59.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec05ffb700 1 -- 172.21.15.1:0/1439833283 >> v1:172.21.15.1:6789/0 conn(0x7fec080cd870 legacy=0x7fec080cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:59.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec05ffb700 1 --2- 172.21.15.1:0/1439833283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febf0003b90 0x7febf0003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:59.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec05ffb700 1 --2- 172.21.15.1:0/1439833283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7febf0004850 0x7febf0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:59.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec05ffb700 1 --2- 172.21.15.1:0/1439833283 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7febf0005210 0x7febf000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:59.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec05ffb700 1 -- 172.21.15.1:0/1439833283 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec080d0830 con 0x7febf0003b90 2024-03-18T19:05:59.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec05ffb700 1 -- 172.21.15.1:0/1439833283 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf000c090 con 0x7febf0004850 2024-03-18T19:05:59.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec05ffb700 1 -- 172.21.15.1:0/1439833283 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf000c250 con 0x7febf0005210 2024-03-18T19:05:59.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec067fc700 1 --2- 172.21.15.1:0/1439833283 >> v2:172.21.15.165:3300/0 conn(0x7fec080b1e20 0x7fec080cff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:05:59.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec06ffd700 1 --2- 172.21.15.1:0/1439833283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febf0003b90 0x7febf0003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:59.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec077fe700 1 --2- 172.21.15.1:0/1439833283 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7febf0005210 0x7febf000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:59.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec067fc700 1 --2- 172.21.15.1:0/1439833283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7febf0004850 0x7febf0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:59.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec06ffd700 1 -- 172.21.15.1:0/1439833283 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7febf0005210 msgr2=0x7febf000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:59.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec06ffd700 1 --2- 172.21.15.1:0/1439833283 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7febf0005210 0x7febf000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:59.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec06ffd700 1 -- 172.21.15.1:0/1439833283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7febf0004850 msgr2=0x7febf0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:05:59.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec06ffd700 1 --2- 172.21.15.1:0/1439833283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7febf0004850 0x7febf0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:59.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec06ffd700 1 -- 172.21.15.1:0/1439833283 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf00035b0 con 0x7febf0003b90 2024-03-18T19:05:59.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec077fe700 1 -- 172.21.15.1:0/1439833283 reap_dead start 2024-03-18T19:05:59.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec06ffd700 1 --2- 172.21.15.1:0/1439833283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febf0003b90 0x7febf0003f90 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7febf800c210 tx=0x7febf0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acb0d6ab68fa8e67 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:59.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.974+0000 7fec05ffb700 1 -- 172.21.15.1:0/1439833283 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf800cd50 con 0x7febf0003b90 2024-03-18T19:05:59.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.975+0000 7fec05ffb700 1 -- 172.21.15.1:0/1439833283 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf8005960 con 0x7febf0003b90 2024-03-18T19:05:59.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.975+0000 7fec05ffb700 1 -- 172.21.15.1:0/1439833283 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8005d10 con 0x7febf0003b90 2024-03-18T19:05:59.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.975+0000 7fec0d829700 1 -- 172.21.15.1:0/1439833283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febf0003b90 msgr2=0x7febf0003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:59.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.975+0000 7fec0d829700 1 --2- 172.21.15.1:0/1439833283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febf0003b90 0x7febf0003f90 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7febf800c210 tx=0x7febf0005880 comp rx=0 tx=0).stop 2024-03-18T19:05:59.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.975+0000 7fec0d829700 1 -- 172.21.15.1:0/1439833283 shutdown_connections 2024-03-18T19:05:59.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.975+0000 7fec0d829700 1 --2- 172.21.15.1:0/1439833283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febf0003b90 0x7febf0003f90 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:59.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.975+0000 7fec0d829700 1 -- 172.21.15.1:0/1439833283 >> 172.21.15.1:0/1439833283 conn(0x7fec080d0ff0 msgr2=0x7fec080d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:05:59.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.975+0000 7fec0d829700 1 -- 172.21.15.1:0/1439833283 shutdown_connections 2024-03-18T19:05:59.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.975+0000 7fec0d829700 1 -- 172.21.15.1:0/1439833283 wait complete. 2024-03-18T19:05:59.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.976+0000 7fec0d829700 1 Processor -- start 2024-03-18T19:05:59.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.976+0000 7fec0d829700 1 -- start start 2024-03-18T19:05:59.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.977+0000 7fec0d829700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febf0003b90 0x7fec08153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:59.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.977+0000 7fec0d829700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec080c80c0 0x7fec080c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:59.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.977+0000 7fec0d829700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec080c9f80 0x7fec080ca430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:59.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.977+0000 7fec0d829700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec080ca970 con 0x7febf0003b90 2024-03-18T19:05:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.977+0000 7fec0d829700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec080c5630 con 0x7fec080c80c0 2024-03-18T19:05:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.977+0000 7fec0d829700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec080c8c30 con 0x7fec080c9f80 2024-03-18T19:05:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.977+0000 7fec06ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febf0003b90 0x7fec08153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.977+0000 7fec06ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febf0003b90 0x7fec08153e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48684/0 (socket says 172.21.15.1:48684) 2024-03-18T19:05:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.977+0000 7fec06ffd700 1 -- 172.21.15.1:0/2999366614 learned_addr learned my addr 172.21.15.1:0/2999366614 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:05:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.977+0000 7fec067fc700 1 --2- 172.21.15.1:0/2999366614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec080c80c0 0x7fec080c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.978+0000 7fec077fe700 1 --2- 172.21.15.1:0/2999366614 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec080c9f80 0x7fec080ca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.978+0000 7fec06ffd700 1 -- 172.21.15.1:0/2999366614 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec080c9f80 msgr2=0x7fec080ca430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.978+0000 7fec06ffd700 1 --2- 172.21.15.1:0/2999366614 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec080c9f80 0x7fec080ca430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.978+0000 7fec06ffd700 1 -- 172.21.15.1:0/2999366614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec080c80c0 msgr2=0x7fec080c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:05:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.978+0000 7fec06ffd700 1 --2- 172.21.15.1:0/2999366614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec080c80c0 0x7fec080c8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:05:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.978+0000 7fec06ffd700 1 -- 172.21.15.1:0/2999366614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec080c8f10 con 0x7febf0003b90 2024-03-18T19:05:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.978+0000 7fec06ffd700 1 --2- 172.21.15.1:0/2999366614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febf0003b90 0x7fec08153e40 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7febf000c060 tx=0x7febf8001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:59.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.978+0000 7febeffff700 1 -- 172.21.15.1:0/2999366614 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8004c80 con 0x7febf0003b90 2024-03-18T19:05:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.978+0000 7febeffff700 1 -- 172.21.15.1:0/2999366614 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf8004de0 con 0x7febf0003b90 2024-03-18T19:05:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.978+0000 7fec0d829700 1 -- 172.21.15.1:0/2999366614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec080c91f0 con 0x7febf0003b90 2024-03-18T19:05:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.978+0000 7febeffff700 1 -- 172.21.15.1:0/2999366614 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf80072f0 con 0x7febf0003b90 2024-03-18T19:05:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.978+0000 7fec0d829700 1 -- 172.21.15.1:0/2999366614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec080c97b0 con 0x7febf0003b90 2024-03-18T19:05:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.979+0000 7fec0d829700 1 -- 172.21.15.1:0/2999366614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec080c9940 con 0x7febf0003b90 2024-03-18T19:05:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.979+0000 7febeffff700 1 -- 172.21.15.1:0/2999366614 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7febf8007450 con 0x7febf0003b90 2024-03-18T19:05:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.980+0000 7febeffff700 1 --2- 172.21.15.1:0/2999366614 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7febf406adb0 0x7febf406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:05:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.980+0000 7febeffff700 1 -- 172.21.15.1:0/2999366614 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5865+0+0 (secure 0 0 0) 0x7febf808a480 con 0x7febf0003b90 2024-03-18T19:05:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.980+0000 7fec067fc700 1 --2- 172.21.15.1:0/2999366614 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7febf406adb0 0x7febf406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:05:59.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.980+0000 7fec067fc700 1 --2- 172.21.15.1:0/2999366614 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7febf406adb0 0x7febf406d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fec080cd5a0 tx=0x7febfc008040 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:05:59.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:05:59.985+0000 7febeffff700 1 -- 172.21.15.1:0/2999366614 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7febf80554c0 con 0x7febf0003b90 2024-03-18T19:06:00.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:00.199+0000 7fec0d829700 1 -- 172.21.15.1:0/2999366614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fec080676a0 con 0x7febf0003b90 2024-03-18T19:06:00.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:00.944+0000 7febeffff700 1 -- 172.21.15.1:0/2999366614 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v99) v1 ==== 97+0+0 (secure 0 0 0) 0x7febf8058ae0 con 0x7febf0003b90 2024-03-18T19:06:00.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:00.947+0000 7fec0d829700 1 -- 172.21.15.1:0/2999366614 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7febf406adb0 msgr2=0x7febf406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:00.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:00.947+0000 7fec0d829700 1 --2- 172.21.15.1:0/2999366614 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7febf406adb0 0x7febf406d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fec080cd5a0 tx=0x7febfc008040 comp rx=0 tx=0).stop 2024-03-18T19:06:00.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:00.947+0000 7fec0d829700 1 -- 172.21.15.1:0/2999366614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febf0003b90 msgr2=0x7fec08153e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:00.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:00.947+0000 7fec0d829700 1 --2- 172.21.15.1:0/2999366614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febf0003b90 0x7fec08153e40 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7febf000c060 tx=0x7febf8001980 comp rx=0 tx=0).stop 2024-03-18T19:06:00.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:00.947+0000 7fec0d829700 1 -- 172.21.15.1:0/2999366614 shutdown_connections 2024-03-18T19:06:00.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:00.947+0000 7fec0d829700 1 --2- 172.21.15.1:0/2999366614 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7febf406adb0 0x7febf406d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:00.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:00.947+0000 7fec0d829700 1 --2- 172.21.15.1:0/2999366614 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec080c9f80 0x7fec080ca430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:00.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:00.947+0000 7fec0d829700 1 --2- 172.21.15.1:0/2999366614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7febf0003b90 0x7fec08153e40 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:00.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:00.947+0000 7fec0d829700 1 --2- 172.21.15.1:0/2999366614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec080c80c0 0x7fec080c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:00.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:00.947+0000 7fec0d829700 1 -- 172.21.15.1:0/2999366614 >> 172.21.15.1:0/2999366614 conn(0x7fec080d0ff0 msgr2=0x7fec080cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:00.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:00.947+0000 7fec0d829700 1 -- 172.21.15.1:0/2999366614 shutdown_connections 2024-03-18T19:06:00.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:00.947+0000 7fec0d829700 1 -- 172.21.15.1:0/2999366614 wait complete. 2024-03-18T19:06:00.945 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-18T19:06:00.958 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-18T19:06:01.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.094+0000 7f00f25ae700 1 Processor -- start 2024-03-18T19:06:01.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.094+0000 7f00f25ae700 1 -- start start 2024-03-18T19:06:01.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.095+0000 7f00f25ae700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f00ec15d580 0x7f00ec072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.095+0000 7f00f25ae700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f00ec06e290 0x7f00ec0cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.095+0000 7f00f25ae700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00ec06d640 0x7f00ec06dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.095+0000 7f00f25ae700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00ec06e010 con 0x7f00ec15d580 2024-03-18T19:06:01.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.095+0000 7f00f25ae700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00ec06f490 con 0x7f00ec06e290 2024-03-18T19:06:01.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.095+0000 7f00f25ae700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec071cf0 con 0x7f00ec06d640 2024-03-18T19:06:01.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.095+0000 7f00ebfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f00ec15d580 0x7f00ec072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.095+0000 7f00ebfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f00ec15d580 0x7f00ec072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48698/0 (socket says 172.21.15.1:48698) 2024-03-18T19:06:01.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.095+0000 7f00ebfff700 1 -- 172.21.15.1:0/624719724 learned_addr learned my addr 172.21.15.1:0/624719724 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:01.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.096+0000 7f00f0b4b700 1 --2- 172.21.15.1:0/624719724 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00ec06d640 0x7f00ec06dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.096+0000 7f00eb7fe700 1 --2- 172.21.15.1:0/624719724 >> v2:172.21.15.165:3300/0 conn(0x7f00ec06e290 0x7f00ec0cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.096+0000 7f00ebfff700 1 -- 172.21.15.1:0/624719724 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00ec06d640 msgr2=0x7f00ec06dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.096+0000 7f00ebfff700 1 --2- 172.21.15.1:0/624719724 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00ec06d640 0x7f00ec06dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.096+0000 7f00ebfff700 1 -- 172.21.15.1:0/624719724 >> v2:172.21.15.165:3300/0 conn(0x7f00ec06e290 msgr2=0x7f00ec0cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.096+0000 7f00ebfff700 1 --2- 172.21.15.1:0/624719724 >> v2:172.21.15.165:3300/0 conn(0x7f00ec06e290 0x7f00ec0cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.096+0000 7f00ebfff700 1 -- 172.21.15.1:0/624719724 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec06f1b0 con 0x7f00ec15d580 2024-03-18T19:06:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.096+0000 7f00ebfff700 1 --2- 172.21.15.1:0/624719724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00ec15d580 0x7f00ec072180 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0712f0 tx=0x7f00d4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1661ebe894321b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.096+0000 7f00eaffd700 1 -- 172.21.15.1:0/624719724 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d400b870 con 0x7f00ec15d580 2024-03-18T19:06:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.096+0000 7f00eaffd700 1 -- 172.21.15.1:0/624719724 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d400b9d0 con 0x7f00ec15d580 2024-03-18T19:06:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.096+0000 7f00eaffd700 1 -- 172.21.15.1:0/624719724 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d400bd60 con 0x7f00ec15d580 2024-03-18T19:06:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.097+0000 7f00f25ae700 1 -- 172.21.15.1:0/624719724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00ec15d580 msgr2=0x7f00ec072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.097+0000 7f00f25ae700 1 --2- 172.21.15.1:0/624719724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00ec15d580 0x7f00ec072180 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0712f0 tx=0x7f00d4009bd0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.097+0000 7f00f25ae700 1 -- 172.21.15.1:0/624719724 shutdown_connections 2024-03-18T19:06:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.097+0000 7f00f25ae700 1 --2- 172.21.15.1:0/624719724 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00ec06d640 0x7f00ec06dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.097+0000 7f00f25ae700 1 --2- 172.21.15.1:0/624719724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00ec15d580 0x7f00ec072180 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.097+0000 7f00f25ae700 1 --2- 172.21.15.1:0/624719724 >> v2:172.21.15.165:3300/0 conn(0x7f00ec06e290 0x7f00ec0cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.097+0000 7f00f25ae700 1 -- 172.21.15.1:0/624719724 >> 172.21.15.1:0/624719724 conn(0x7f00ec0cc990 msgr2=0x7f00ec0ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.097+0000 7f00f25ae700 1 -- 172.21.15.1:0/624719724 shutdown_connections 2024-03-18T19:06:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.097+0000 7f00f25ae700 1 -- 172.21.15.1:0/624719724 wait complete. 2024-03-18T19:06:01.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.098+0000 7f00f25ae700 1 Processor -- start 2024-03-18T19:06:01.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.098+0000 7f00f25ae700 1 -- start start 2024-03-18T19:06:01.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.099+0000 7f00f25ae700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00ec06d640 0x7f00ec14e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.099+0000 7f00f25ae700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00ec15d580 0x7f00ec1f2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.099+0000 7f00f25ae700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00ec1f3360 0x7f00ec0b3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.099+0000 7f00f25ae700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec0b1880 con 0x7f00ec15d580 2024-03-18T19:06:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.099+0000 7f00f25ae700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec15ee80 con 0x7f00ec1f3360 2024-03-18T19:06:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.099+0000 7f00f25ae700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec15f160 con 0x7f00ec06d640 2024-03-18T19:06:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.099+0000 7f00eb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00ec15d580 0x7f00ec1f2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.099+0000 7f00eb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00ec15d580 0x7f00ec1f2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48712/0 (socket says 172.21.15.1:48712) 2024-03-18T19:06:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.099+0000 7f00eb7fe700 1 -- 172.21.15.1:0/1623793402 learned_addr learned my addr 172.21.15.1:0/1623793402 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.100+0000 7f00ebfff700 1 --2- 172.21.15.1:0/1623793402 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00ec06d640 0x7f00ec14e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.100+0000 7f00eb7fe700 1 -- 172.21.15.1:0/1623793402 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00ec06d640 msgr2=0x7f00ec14e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.100+0000 7f00eb7fe700 1 --2- 172.21.15.1:0/1623793402 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00ec06d640 0x7f00ec14e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.100+0000 7f00eb7fe700 1 -- 172.21.15.1:0/1623793402 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00ec1f3360 msgr2=0x7f00ec0b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.100+0000 7f00f0b4b700 1 --2- 172.21.15.1:0/1623793402 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00ec1f3360 0x7f00ec0b3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.100+0000 7f00eb7fe700 1 --2- 172.21.15.1:0/1623793402 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00ec1f3360 0x7f00ec0b3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.100+0000 7f00eb7fe700 1 -- 172.21.15.1:0/1623793402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d4009550 con 0x7f00ec15d580 2024-03-18T19:06:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.100+0000 7f00eb7fe700 1 --2- 172.21.15.1:0/1623793402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00ec15d580 0x7f00ec1f2e20 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f00dc00e810 tx=0x7f00dc00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.100+0000 7f00e8ff9700 1 -- 172.21.15.1:0/1623793402 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00dc013610 con 0x7f00ec15d580 2024-03-18T19:06:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.100+0000 7f00e8ff9700 1 -- 172.21.15.1:0/1623793402 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00dc013770 con 0x7f00ec15d580 2024-03-18T19:06:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.100+0000 7f00f25ae700 1 -- 172.21.15.1:0/1623793402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ec15f440 con 0x7f00ec15d580 2024-03-18T19:06:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.100+0000 7f00e8ff9700 1 -- 172.21.15.1:0/1623793402 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00dc013b20 con 0x7f00ec15d580 2024-03-18T19:06:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.100+0000 7f00f25ae700 1 -- 172.21.15.1:0/1623793402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ec15fa00 con 0x7f00ec15d580 2024-03-18T19:06:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.101+0000 7f00f25ae700 1 -- 172.21.15.1:0/1623793402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00ec0676a0 con 0x7f00ec15d580 2024-03-18T19:06:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.102+0000 7f00e8ff9700 1 -- 172.21.15.1:0/1623793402 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f00dc013c80 con 0x7f00ec15d580 2024-03-18T19:06:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.102+0000 7f00e8ff9700 1 --2- 172.21.15.1:0/1623793402 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f00cc06adb0 0x7f00cc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.102+0000 7f00e8ff9700 1 -- 172.21.15.1:0/1623793402 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f00dc088fc0 con 0x7f00ec15d580 2024-03-18T19:06:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.102+0000 7f00ebfff700 1 --2- 172.21.15.1:0/1623793402 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f00cc06adb0 0x7f00cc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.103+0000 7f00ebfff700 1 --2- 172.21.15.1:0/1623793402 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f00cc06adb0 0x7f00cc06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f00d4009c10 tx=0x7f00d4008030 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:01.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.107+0000 7f00e8ff9700 1 -- 172.21.15.1:0/1623793402 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f00dc0541a0 con 0x7f00ec15d580 2024-03-18T19:06:01.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.390+0000 7f00f25ae700 1 -- 172.21.15.1:0/1623793402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f00ec0cd080 con 0x7f00ec15d580 2024-03-18T19:06:01.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.390+0000 7f00e8ff9700 1 -- 172.21.15.1:0/1623793402 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f00dc0577c0 con 0x7f00ec15d580 2024-03-18T19:06:01.387 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:01.387 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:06:01.387 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-18T19:06:01.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:01.387 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-18T19:06:01.387 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlR93/CxAAbpe+5n3wHRur9xi8QzdCnQ==", 2024-03-18T19:06:01.387 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:01.387 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:06:01.387 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:06:01.387 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:06:01.387 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:01.388 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:01.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:01.388 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-18T19:06:01.388 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJ3OmGBAA37u227nxrMsh/JHt71HBxQ==", 2024-03-18T19:06:01.388 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:01.388 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:06:01.388 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:06:01.388 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:06:01.388 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:01.388 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:01.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:01.388 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-18T19:06:01.389 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJZFoExAAmI16oxwNKbf8dKetPfyk+g==", 2024-03-18T19:06:01.389 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:01.389 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:06:01.389 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:06:01.389 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:06:01.389 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:01.389 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:01.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:01.389 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-18T19:06:01.389 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlQrcvIBAA1LMv+EgC7S6wspX6QF49Pg==", 2024-03-18T19:06:01.389 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:01.389 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:06:01.390 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:06:01.390 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:06:01.390 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:01.390 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:01.390 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:01.390 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-18T19:06:01.390 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCUjPhlYOVHChAARFQrksZmMIdTlntd8TgvrQ==", 2024-03-18T19:06:01.390 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:01.390 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:06:01.390 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:06:01.390 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:01.390 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:01.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:01.391 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-18T19:06:01.391 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCXjPhl2zSDCBAA6Al1gkWVB79ZI9djGpVChA==", 2024-03-18T19:06:01.391 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:01.391 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:06:01.391 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:06:01.391 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:01.391 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:01.391 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:01.391 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-18T19:06:01.391 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCajPhlSoG/BhAAR93USDe6DCQW+hREOsnTlw==", 2024-03-18T19:06:01.392 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:01.392 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:06:01.392 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:06:01.392 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:01.392 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:01.392 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:01.392 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-18T19:06:01.392 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCdjPhlhJg5BRAAWuQhcnxyjR5UOiM13yKdAQ==", 2024-03-18T19:06:01.392 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:01.392 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:06:01.392 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:06:01.392 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:01.392 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:01.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:01.393 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-18T19:06:01.393 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCjjPhlCFnpCRAAKIyOQyVdjXZUTnAS6Nw/ng==", 2024-03-18T19:06:01.393 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:01.393 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:06:01.393 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:06:01.393 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:01.393 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:01.393 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:01.393 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-18T19:06:01.393 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCmjPhlAeFkCBAAJcEFpRk/03lQJxj+l2hEPA==", 2024-03-18T19:06:01.393 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:01.393 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:06:01.393 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:06:01.394 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:01.394 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:01.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:01.394 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-18T19:06:01.394 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCpjPhl/xuuBhAA0LgKfXEvN2O11ZlRQJVOkw==", 2024-03-18T19:06:01.394 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:01.394 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:06:01.394 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:06:01.394 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:01.394 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:01.394 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:01.395 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-18T19:06:01.395 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCsjPhldiR4BRAA571oh6U8PzOv/4LzjYLF/A==", 2024-03-18T19:06:01.395 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:01.395 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:06:01.395 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:06:01.395 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:01.395 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:01.395 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:01.395 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-18T19:06:01.395 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlu+2xJBAANnXGGjfeqP6S2FTecsfOsg==", 2024-03-18T19:06:01.395 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:01.395 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:06:01.395 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-18T19:06:01.396 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-18T19:06:01.396 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:01.396 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:01.396 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-18T19:06:01.396 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlhJ9+KxAAEnrQrCrgJR6xI+pucwzAXA==", 2024-03-18T19:06:01.396 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:01.396 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:06:01.396 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:06:01.396 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:06:01.396 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:06:01.396 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:01.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:01.397 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.2", 2024-03-18T19:06:01.397 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlxBUpKBAAIGTwmLWyj+P2Yvl55QVcWA==", 2024-03-18T19:06:01.397 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:01.397 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:06:01.397 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:06:01.397 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:06:01.397 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:06:01.397 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.397 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:01.397 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:01.397 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.3", 2024-03-18T19:06:01.398 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlmao3MRAAHeiQ6eyBh9QjjKPCtxkC8g==", 2024-03-18T19:06:01.398 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:01.398 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:06:01.398 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:06:01.398 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:06:01.398 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:06:01.398 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:01.398 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:01.398 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-18T19:06:01.398 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCRjPhlemOuKRAAVUggmtB7+EiVgyfJZ8TCCA==", 2024-03-18T19:06:01.398 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:01.398 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:06:01.398 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:06:01.399 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:06:01.399 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:01.399 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:01.399 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:01.399 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-18T19:06:01.399 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhleIuOCBAA9CH+AHewQbn+gpVr+6lJyw==", 2024-03-18T19:06:01.399 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:01.399 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:06:01.399 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T19:06:01.399 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:01.399 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:01.400 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:01.400 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-18T19:06:01.400 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlL0pTBBAAdFiPcdwwz514cuLAvioxYA==", 2024-03-18T19:06:01.400 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:01.400 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:06:01.400 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T19:06:01.400 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:01.400 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.400 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:01.400 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-18T19:06:01.400 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:06:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.392+0000 7f00f25ae700 1 -- 172.21.15.1:0/1623793402 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f00cc06adb0 msgr2=0x7f00cc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.392+0000 7f00f25ae700 1 --2- 172.21.15.1:0/1623793402 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f00cc06adb0 0x7f00cc06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f00d4009c10 tx=0x7f00d4008030 comp rx=0 tx=0).stop 2024-03-18T19:06:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.392+0000 7f00f25ae700 1 -- 172.21.15.1:0/1623793402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00ec15d580 msgr2=0x7f00ec1f2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.392+0000 7f00f25ae700 1 --2- 172.21.15.1:0/1623793402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00ec15d580 0x7f00ec1f2e20 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f00dc00e810 tx=0x7f00dc00ebd0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.392+0000 7f00f25ae700 1 -- 172.21.15.1:0/1623793402 shutdown_connections 2024-03-18T19:06:01.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.392+0000 7f00f25ae700 1 --2- 172.21.15.1:0/1623793402 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f00cc06adb0 0x7f00cc06d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.392+0000 7f00f25ae700 1 --2- 172.21.15.1:0/1623793402 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00ec06d640 0x7f00ec14e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.392+0000 7f00f25ae700 1 --2- 172.21.15.1:0/1623793402 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00ec1f3360 0x7f00ec0b3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.392+0000 7f00f25ae700 1 --2- 172.21.15.1:0/1623793402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00ec15d580 0x7f00ec1f2e20 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.392+0000 7f00f25ae700 1 -- 172.21.15.1:0/1623793402 >> 172.21.15.1:0/1623793402 conn(0x7f00ec0cc990 msgr2=0x7f00ec0aed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.392+0000 7f00f25ae700 1 -- 172.21.15.1:0/1623793402 shutdown_connections 2024-03-18T19:06:01.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.392+0000 7f00f25ae700 1 -- 172.21.15.1:0/1623793402 wait complete. 2024-03-18T19:06:01.403 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:06:01.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.535+0000 7fd1d28c1700 1 Processor -- start 2024-03-18T19:06:01.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.535+0000 7fd1d28c1700 1 -- start start 2024-03-18T19:06:01.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.535+0000 7fd1d28c1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1cc15d6b0 0x7fd1cc0b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.535+0000 7fd1d28c1700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc0acfb0 con 0x7fd1cc06a750 2024-03-18T19:06:01.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.536+0000 7fd1d28c1700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc0b50a0 con 0x7fd1cc0ad790 2024-03-18T19:06:01.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.536+0000 7fd1d28c1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1cc0af7a0 con 0x7fd1cc15d6b0 2024-03-18T19:06:01.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.536+0000 7fd1cb7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1cc15d6b0 0x7fd1cc0b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.536+0000 7fd1cbfff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd1cc06a750 0x7fd1cc06ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47026/0 (socket says 172.21.15.1:47026) 2024-03-18T19:06:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.536+0000 7fd1cbfff700 1 -- 172.21.15.1:0/3481526779 learned_addr learned my addr 172.21.15.1:0/3481526779 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.536+0000 7fd1cb7fe700 1 -- 172.21.15.1:0/3481526779 >> v1:172.21.15.165:6789/0 conn(0x7fd1cc0ad790 legacy=0x7fd1cc0b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.536+0000 7fd1cb7fe700 1 -- 172.21.15.1:0/3481526779 >> v1:172.21.15.1:6789/0 conn(0x7fd1cc06a750 legacy=0x7fd1cc06ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.536+0000 7fd1caffd700 1 -- 172.21.15.1:0/3481526779 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202032209 0 0) 0x7fd1cc0acfb0 con 0x7fd1cc06a750 2024-03-18T19:06:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.536+0000 7fd1cb7fe700 1 -- 172.21.15.1:0/3481526779 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1cc0af1e0 con 0x7fd1cc15d6b0 2024-03-18T19:06:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.536+0000 7fd1cb7fe700 1 --2- 172.21.15.1:0/3481526779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1cc15d6b0 0x7fd1cc0b2da0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0009840 tx=0x7fd1c0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddcd9c93d49272c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.536+0000 7fd1caffd700 1 -- 172.21.15.1:0/3481526779 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0004230 con 0x7fd1cc15d6b0 2024-03-18T19:06:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.536+0000 7fd1caffd700 1 -- 172.21.15.1:0/3481526779 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c0004390 con 0x7fd1cc15d6b0 2024-03-18T19:06:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.536+0000 7fd1caffd700 1 -- 172.21.15.1:0/3481526779 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c00036a0 con 0x7fd1cc15d6b0 2024-03-18T19:06:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.537+0000 7fd1d28c1700 1 -- 172.21.15.1:0/3481526779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1cc15d6b0 msgr2=0x7fd1cc0b2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.537+0000 7fd1d28c1700 1 --2- 172.21.15.1:0/3481526779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1cc15d6b0 0x7fd1cc0b2da0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0009840 tx=0x7fd1c0009b50 comp rx=0 tx=0).stop 2024-03-18T19:06:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.537+0000 7fd1d28c1700 1 -- 172.21.15.1:0/3481526779 shutdown_connections 2024-03-18T19:06:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.537+0000 7fd1d28c1700 1 --2- 172.21.15.1:0/3481526779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1cc15d6b0 0x7fd1cc0b2da0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.537+0000 7fd1d28c1700 1 -- 172.21.15.1:0/3481526779 >> 172.21.15.1:0/3481526779 conn(0x7fd1cc06b860 msgr2=0x7fd1cc06bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.537+0000 7fd1d28c1700 1 -- 172.21.15.1:0/3481526779 shutdown_connections 2024-03-18T19:06:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.537+0000 7fd1d28c1700 1 -- 172.21.15.1:0/3481526779 wait complete. 2024-03-18T19:06:01.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.538+0000 7fd1d28c1700 1 Processor -- start 2024-03-18T19:06:01.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.539+0000 7fd1d28c1700 1 -- start start 2024-03-18T19:06:01.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.539+0000 7fd1d28c1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1cc06a750 0x7fd1cc145430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1d28c1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1cc0ad790 0x7fd1cc142080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1d28c1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1cc15d6b0 0x7fd1cc155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1d28c1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc145970 con 0x7fd1cc06a750 2024-03-18T19:06:01.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1d28c1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc155bd0 con 0x7fd1cc15d6b0 2024-03-18T19:06:01.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1d28c1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc155eb0 con 0x7fd1cc0ad790 2024-03-18T19:06:01.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1cbfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1cc06a750 0x7fd1cc145430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1cbfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1cc06a750 0x7fd1cc145430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48722/0 (socket says 172.21.15.1:48722) 2024-03-18T19:06:01.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1cbfff700 1 -- 172.21.15.1:0/3193836663 learned_addr learned my addr 172.21.15.1:0/3193836663 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:01.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1d0e5e700 1 --2- 172.21.15.1:0/3193836663 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1cc15d6b0 0x7fd1cc155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1cb7fe700 1 --2- 172.21.15.1:0/3193836663 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1cc0ad790 0x7fd1cc142080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1cbfff700 1 -- 172.21.15.1:0/3193836663 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1cc0ad790 msgr2=0x7fd1cc142080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:01.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1cbfff700 1 --2- 172.21.15.1:0/3193836663 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1cc0ad790 0x7fd1cc142080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1cbfff700 1 -- 172.21.15.1:0/3193836663 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1cc15d6b0 msgr2=0x7fd1cc155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:01.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1cbfff700 1 --2- 172.21.15.1:0/3193836663 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1cc15d6b0 0x7fd1cc155330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1cbfff700 1 -- 172.21.15.1:0/3193836663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c00094a0 con 0x7fd1cc06a750 2024-03-18T19:06:01.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1d0e5e700 1 --2- 172.21.15.1:0/3193836663 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1cc15d6b0 0x7fd1cc155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:01.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1cbfff700 1 --2- 172.21.15.1:0/3193836663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1cc06a750 0x7fd1cc145430 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0011e0 tx=0x7fd1bc00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:01.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1c8ff9700 1 -- 172.21.15.1:0/3193836663 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc004c80 con 0x7fd1cc06a750 2024-03-18T19:06:01.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1c8ff9700 1 -- 172.21.15.1:0/3193836663 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1bc004e00 con 0x7fd1cc06a750 2024-03-18T19:06:01.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1c8ff9700 1 -- 172.21.15.1:0/3193836663 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc007310 con 0x7fd1cc06a750 2024-03-18T19:06:01.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1d28c1700 1 -- 172.21.15.1:0/3193836663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1cc156190 con 0x7fd1cc06a750 2024-03-18T19:06:01.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.540+0000 7fd1d28c1700 1 -- 172.21.15.1:0/3193836663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1cc1425c0 con 0x7fd1cc06a750 2024-03-18T19:06:01.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.541+0000 7fd1d28c1700 1 -- 172.21.15.1:0/3193836663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1cc0ad420 con 0x7fd1cc06a750 2024-03-18T19:06:01.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.542+0000 7fd1c8ff9700 1 -- 172.21.15.1:0/3193836663 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd1bc007470 con 0x7fd1cc06a750 2024-03-18T19:06:01.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.542+0000 7fd1c8ff9700 1 --2- 172.21.15.1:0/3193836663 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd1b806adb0 0x7fd1b806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.543+0000 7fd1c8ff9700 1 -- 172.21.15.1:0/3193836663 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fd1bc089570 con 0x7fd1cc06a750 2024-03-18T19:06:01.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.543+0000 7fd1cb7fe700 1 --2- 172.21.15.1:0/3193836663 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd1b806adb0 0x7fd1b806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.543+0000 7fd1cb7fe700 1 --2- 172.21.15.1:0/3193836663 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd1b806adb0 0x7fd1b806d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd1c00090b0 tx=0x7fd1c0008030 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:01.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.546+0000 7fd1c8ff9700 1 -- 172.21.15.1:0/3193836663 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd1bc054840 con 0x7fd1cc06a750 2024-03-18T19:06:01.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.750+0000 7fd1d28c1700 1 -- 172.21.15.1:0/3193836663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1cc0ad420 con 0x7fd1cc06a750 2024-03-18T19:06:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.751+0000 7fd1c8ff9700 1 -- 172.21.15.1:0/3193836663 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+12547 (secure 0 0 0) 0x7fd1cc0ad420 con 0x7fd1cc06a750 2024-03-18T19:06:01.747 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:01.747 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":99,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:06:00.941384+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:06:01.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.753+0000 7fd1d28c1700 1 -- 172.21.15.1:0/3193836663 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd1b806adb0 msgr2=0x7fd1b806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:01.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.753+0000 7fd1d28c1700 1 --2- 172.21.15.1:0/3193836663 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd1b806adb0 0x7fd1b806d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd1c00090b0 tx=0x7fd1c0008030 comp rx=0 tx=0).stop 2024-03-18T19:06:01.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.753+0000 7fd1d28c1700 1 -- 172.21.15.1:0/3193836663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1cc06a750 msgr2=0x7fd1cc145430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:01.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.753+0000 7fd1d28c1700 1 --2- 172.21.15.1:0/3193836663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1cc06a750 0x7fd1cc145430 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0011e0 tx=0x7fd1bc00cae0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.753+0000 7fd1d28c1700 1 -- 172.21.15.1:0/3193836663 shutdown_connections 2024-03-18T19:06:01.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.753+0000 7fd1d28c1700 1 --2- 172.21.15.1:0/3193836663 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd1b806adb0 0x7fd1b806d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.753+0000 7fd1d28c1700 1 --2- 172.21.15.1:0/3193836663 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1cc15d6b0 0x7fd1cc155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.753+0000 7fd1d28c1700 1 --2- 172.21.15.1:0/3193836663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1cc06a750 0x7fd1cc145430 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.753+0000 7fd1d28c1700 1 --2- 172.21.15.1:0/3193836663 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1cc0ad790 0x7fd1cc142080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.753+0000 7fd1d28c1700 1 -- 172.21.15.1:0/3193836663 >> 172.21.15.1:0/3193836663 conn(0x7fd1cc06b860 msgr2=0x7fd1cc0b0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:01.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.753+0000 7fd1d28c1700 1 -- 172.21.15.1:0/3193836663 shutdown_connections 2024-03-18T19:06:01.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.753+0000 7fd1d28c1700 1 -- 172.21.15.1:0/3193836663 wait complete. 2024-03-18T19:06:01.762 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:06:01.763 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:06:01.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.895+0000 7f8ccb795700 1 Processor -- start 2024-03-18T19:06:01.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.895+0000 7f8ccb795700 1 -- start start 2024-03-18T19:06:01.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.896+0000 7f8ccb795700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cc415a1d0 0x7f8cc40add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.896+0000 7f8ccb795700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8cc4071620 0x7f8cc4071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.896+0000 7f8ccb795700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc40ce8f0 con 0x7f8cc40cb9e0 2024-03-18T19:06:01.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.896+0000 7f8ccb795700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc40cdb00 con 0x7f8cc4071620 2024-03-18T19:06:01.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.896+0000 7f8ccb795700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc415a5b0 con 0x7f8cc415a1d0 2024-03-18T19:06:01.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.896+0000 7f8cc9d32700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8cc40cb9e0 0x7f8cc40cbe70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47036/0 (socket says 172.21.15.1:47036) 2024-03-18T19:06:01.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.896+0000 7f8cc9d32700 1 -- 172.21.15.1:0/3419062859 learned_addr learned my addr 172.21.15.1:0/3419062859 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:01.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.896+0000 7f8cc9531700 1 --2- 172.21.15.1:0/3419062859 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cc415a1d0 0x7f8cc40add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.896+0000 7f8cc8d30700 1 --2- 172.21.15.1:0/3419062859 >> v2:172.21.15.165:3300/0 conn(0x7f8cc4071620 0x7f8cc4071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.896+0000 7f8cbbfff700 1 -- 172.21.15.1:0/3419062859 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787912130 0 0) 0x7f8cc40ce8f0 con 0x7f8cc40cb9e0 2024-03-18T19:06:01.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.896+0000 7f8cbbfff700 1 -- 172.21.15.1:0/3419062859 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb00035b0 con 0x7f8cc40cb9e0 2024-03-18T19:06:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cbbfff700 1 -- 172.21.15.1:0/3419062859 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2184156719 0 0) 0x7f8cb00035b0 con 0x7f8cc40cb9e0 2024-03-18T19:06:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cbbfff700 1 -- 172.21.15.1:0/3419062859 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cc415a1d0 msgr2=0x7f8cc40add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cbbfff700 1 --2- 172.21.15.1:0/3419062859 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cc415a1d0 0x7f8cc40add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cbbfff700 1 -- 172.21.15.1:0/3419062859 >> v2:172.21.15.165:3300/0 conn(0x7f8cc4071620 msgr2=0x7f8cc4071aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cbbfff700 1 --2- 172.21.15.1:0/3419062859 >> v2:172.21.15.165:3300/0 conn(0x7f8cc4071620 0x7f8cc4071aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cbbfff700 1 -- 172.21.15.1:0/3419062859 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc40ae260 con 0x7f8cc40cb9e0 2024-03-18T19:06:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cbbfff700 1 -- 172.21.15.1:0/3419062859 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f8cbc0019f0 con 0x7f8cc40cb9e0 2024-03-18T19:06:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cbbfff700 1 -- 172.21.15.1:0/3419062859 >> v1:172.21.15.1:6789/0 conn(0x7f8cc40cb9e0 legacy=0x7f8cc40cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cbbfff700 1 --2- 172.21.15.1:0/3419062859 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cc415a1d0 0x7f8cb0003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cbbfff700 1 --2- 172.21.15.1:0/3419062859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cb0003f80 0x7f8cb0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cbbfff700 1 --2- 172.21.15.1:0/3419062859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb0004d90 0x7f8cb0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cbbfff700 1 -- 172.21.15.1:0/3419062859 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40cdb00 con 0x7f8cb0004d90 2024-03-18T19:06:01.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cbbfff700 1 -- 172.21.15.1:0/3419062859 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb000ba10 con 0x7f8cb0003f80 2024-03-18T19:06:01.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cbbfff700 1 -- 172.21.15.1:0/3419062859 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb000bb50 con 0x7f8cc415a1d0 2024-03-18T19:06:01.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cc9d32700 1 --2- 172.21.15.1:0/3419062859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb0004d90 0x7f8cb0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cc8d30700 1 --2- 172.21.15.1:0/3419062859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cb0003f80 0x7f8cb0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cc9531700 1 --2- 172.21.15.1:0/3419062859 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cc415a1d0 0x7f8cb0003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cc9d32700 1 -- 172.21.15.1:0/3419062859 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cc415a1d0 msgr2=0x7f8cb0003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:01.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cc9d32700 1 --2- 172.21.15.1:0/3419062859 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cc415a1d0 0x7f8cb0003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cc9d32700 1 -- 172.21.15.1:0/3419062859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cb0003f80 msgr2=0x7f8cb0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:01.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cc9d32700 1 --2- 172.21.15.1:0/3419062859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cb0003f80 0x7f8cb0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cc9d32700 1 -- 172.21.15.1:0/3419062859 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb00035b0 con 0x7f8cb0004d90 2024-03-18T19:06:01.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cc8d30700 1 --2- 172.21.15.1:0/3419062859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cb0003f80 0x7f8cb0004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:01.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.897+0000 7f8cc9d32700 1 --2- 172.21.15.1:0/3419062859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb0004d90 0x7f8cb0005210 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc00a460 tx=0x7f8cb0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da1754d4fb4499a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:01.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.898+0000 7f8cbbfff700 1 -- 172.21.15.1:0/3419062859 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc010690 con 0x7f8cb0004d90 2024-03-18T19:06:01.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.898+0000 7f8cbbfff700 1 -- 172.21.15.1:0/3419062859 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cbc0107f0 con 0x7f8cb0004d90 2024-03-18T19:06:01.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.898+0000 7f8cbbfff700 1 -- 172.21.15.1:0/3419062859 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc010ba0 con 0x7f8cb0004d90 2024-03-18T19:06:01.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.898+0000 7f8ccb795700 1 -- 172.21.15.1:0/3419062859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb0004d90 msgr2=0x7f8cb0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:01.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.898+0000 7f8ccb795700 1 --2- 172.21.15.1:0/3419062859 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb0004d90 0x7f8cb0005210 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc00a460 tx=0x7f8cb0005880 comp rx=0 tx=0).stop 2024-03-18T19:06:01.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.898+0000 7f8cc9d32700 1 -- 172.21.15.1:0/3419062859 reap_dead start 2024-03-18T19:06:01.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.898+0000 7f8ccb795700 1 -- 172.21.15.1:0/3419062859 shutdown_connections 2024-03-18T19:06:01.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.898+0000 7f8ccb795700 1 -- 172.21.15.1:0/3419062859 >> 172.21.15.1:0/3419062859 conn(0x7f8cc40cc9b0 msgr2=0x7f8cc40ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:01.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.898+0000 7f8ccb795700 1 -- 172.21.15.1:0/3419062859 shutdown_connections 2024-03-18T19:06:01.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.898+0000 7f8ccb795700 1 -- 172.21.15.1:0/3419062859 wait complete. 2024-03-18T19:06:01.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.900+0000 7f8ccb795700 1 Processor -- start 2024-03-18T19:06:01.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.900+0000 7f8ccb795700 1 -- start start 2024-03-18T19:06:01.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.901+0000 7f8ccb795700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cb0004d90 0x7f8cc40b4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.901+0000 7f8ccb795700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cc40b2b00 0x7f8cc40b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.901+0000 7f8ccb795700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cc40c9b10 0x7f8cc40b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.901+0000 7f8ccb795700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40b54d0 con 0x7f8cc40b2b00 2024-03-18T19:06:01.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.901+0000 7f8ccb795700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc41f1ee0 con 0x7f8cc40c9b10 2024-03-18T19:06:01.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.901+0000 7f8ccb795700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc406ea00 con 0x7f8cb0004d90 2024-03-18T19:06:01.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.901+0000 7f8cc8d30700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cc40b2b00 0x7f8cc40b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.901+0000 7f8cc8d30700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cc40b2b00 0x7f8cc40b4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48742/0 (socket says 172.21.15.1:48742) 2024-03-18T19:06:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.901+0000 7f8cc8d30700 1 -- 172.21.15.1:0/2389081136 learned_addr learned my addr 172.21.15.1:0/2389081136 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.901+0000 7f8cc9531700 1 --2- 172.21.15.1:0/2389081136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cb0004d90 0x7f8cc40b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.901+0000 7f8cc9d32700 1 --2- 172.21.15.1:0/2389081136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cc40c9b10 0x7f8cc40b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.901+0000 7f8cc8d30700 1 -- 172.21.15.1:0/2389081136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cb0004d90 msgr2=0x7f8cc40b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.901+0000 7f8cc8d30700 1 --2- 172.21.15.1:0/2389081136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cb0004d90 0x7f8cc40b4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.901+0000 7f8cc8d30700 1 -- 172.21.15.1:0/2389081136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cc40c9b10 msgr2=0x7f8cc40b4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.901+0000 7f8cc8d30700 1 --2- 172.21.15.1:0/2389081136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cc40c9b10 0x7f8cc40b4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.901+0000 7f8cc8d30700 1 -- 172.21.15.1:0/2389081136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc406e0f0 con 0x7f8cc40b2b00 2024-03-18T19:06:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.901+0000 7f8cc8d30700 1 --2- 172.21.15.1:0/2389081136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cc40b2b00 0x7f8cc40b4a50 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0010930 tx=0x7f8cc0010c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.902+0000 7f8cb9ffb700 1 -- 172.21.15.1:0/2389081136 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0004020 con 0x7f8cc40b2b00 2024-03-18T19:06:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.902+0000 7f8cb9ffb700 1 -- 172.21.15.1:0/2389081136 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc0004180 con 0x7f8cc40b2b00 2024-03-18T19:06:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.902+0000 7f8ccb795700 1 -- 172.21.15.1:0/2389081136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc406e3b0 con 0x7f8cc40b2b00 2024-03-18T19:06:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.902+0000 7f8cb9ffb700 1 -- 172.21.15.1:0/2389081136 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0004530 con 0x7f8cc40b2b00 2024-03-18T19:06:01.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.902+0000 7f8ccb795700 1 -- 172.21.15.1:0/2389081136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc4143f40 con 0x7f8cc40b2b00 2024-03-18T19:06:01.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.903+0000 7f8ccb795700 1 -- 172.21.15.1:0/2389081136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc406ebb0 con 0x7f8cc40b2b00 2024-03-18T19:06:01.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.903+0000 7f8cb9ffb700 1 -- 172.21.15.1:0/2389081136 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8cc00036a0 con 0x7f8cc40b2b00 2024-03-18T19:06:01.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.904+0000 7f8cb9ffb700 1 --2- 172.21.15.1:0/2389081136 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8ca806adb0 0x7f8ca806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:01.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.904+0000 7f8cb9ffb700 1 -- 172.21.15.1:0/2389081136 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f8cc0089610 con 0x7f8cc40b2b00 2024-03-18T19:06:01.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.904+0000 7f8cc9531700 1 --2- 172.21.15.1:0/2389081136 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8ca806adb0 0x7f8ca806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:01.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.904+0000 7f8cc9531700 1 --2- 172.21.15.1:0/2389081136 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8ca806adb0 0x7f8ca806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4009c40 tx=0x7f8cb40091c0 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:01.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:01.908+0000 7f8cb9ffb700 1 -- 172.21.15.1:0/2389081136 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8cc0054a60 con 0x7f8cc40b2b00 2024-03-18T19:06:02.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.110+0000 7f8ccb795700 1 -- 172.21.15.1:0/2389081136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8cc406eb70 con 0x7f8cc40b2b00 2024-03-18T19:06:02.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.110+0000 7f8cb9ffb700 1 -- 172.21.15.1:0/2389081136 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8cc0058080 con 0x7f8cc40b2b00 2024-03-18T19:06:02.106 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:06:02.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.112+0000 7f8ccb795700 1 -- 172.21.15.1:0/2389081136 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8ca806adb0 msgr2=0x7f8ca806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:02.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.113+0000 7f8ccb795700 1 --2- 172.21.15.1:0/2389081136 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8ca806adb0 0x7f8ca806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4009c40 tx=0x7f8cb40091c0 comp rx=0 tx=0).stop 2024-03-18T19:06:02.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.113+0000 7f8ccb795700 1 -- 172.21.15.1:0/2389081136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cc40b2b00 msgr2=0x7f8cc40b4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:02.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.113+0000 7f8ccb795700 1 --2- 172.21.15.1:0/2389081136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cc40b2b00 0x7f8cc40b4a50 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0010930 tx=0x7f8cc0010c40 comp rx=0 tx=0).stop 2024-03-18T19:06:02.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.113+0000 7f8ccb795700 1 -- 172.21.15.1:0/2389081136 shutdown_connections 2024-03-18T19:06:02.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.113+0000 7f8ccb795700 1 --2- 172.21.15.1:0/2389081136 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8ca806adb0 0x7f8ca806d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:02.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.113+0000 7f8ccb795700 1 --2- 172.21.15.1:0/2389081136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cb0004d90 0x7f8cc40b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:02.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.113+0000 7f8ccb795700 1 --2- 172.21.15.1:0/2389081136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cc40c9b10 0x7f8cc40b4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:02.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.113+0000 7f8ccb795700 1 --2- 172.21.15.1:0/2389081136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cc40b2b00 0x7f8cc40b4a50 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:02.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.113+0000 7f8ccb795700 1 -- 172.21.15.1:0/2389081136 >> 172.21.15.1:0/2389081136 conn(0x7f8cc40cc9b0 msgr2=0x7f8cc4157d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:02.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.113+0000 7f8ccb795700 1 -- 172.21.15.1:0/2389081136 shutdown_connections 2024-03-18T19:06:02.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.113+0000 7f8ccb795700 1 -- 172.21.15.1:0/2389081136 wait complete. 2024-03-18T19:06:02.120 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T19:06:02.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.249+0000 7f6a30df2700 1 Processor -- start 2024-03-18T19:06:02.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.249+0000 7f6a30df2700 1 -- start start 2024-03-18T19:06:02.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.250+0000 7f6a30df2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a2c15aff0 0x7f6a2c15f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:02.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.250+0000 7f6a30df2700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6a2c073b00 0x7f6a2c0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:02.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.250+0000 7f6a30df2700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2c15fcc0 con 0x7f6a2c15b3d0 2024-03-18T19:06:02.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.250+0000 7f6a30df2700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a2c0ce200 con 0x7f6a2c073b00 2024-03-18T19:06:02.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.250+0000 7f6a30df2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c0ce4a0 con 0x7f6a2c15aff0 2024-03-18T19:06:02.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.251+0000 7f6a2ad9d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6a2c073b00 0x7f6a2c0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:02.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.251+0000 7f6a2a59c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a2c15aff0 0x7f6a2c15f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:02.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.251+0000 7f6a29d9b700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f6a2c15b3d0 0x7f6a2c0cf540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:50634/0 (socket says 172.21.15.1:50634) 2024-03-18T19:06:02.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.251+0000 7f6a29d9b700 1 -- 172.21.15.1:0/3814643621 learned_addr learned my addr 172.21.15.1:0/3814643621 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:02.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.251+0000 7f6a2a59c700 1 -- 172.21.15.1:0/3814643621 >> v2:172.21.15.165:3300/0 conn(0x7f6a2c073b00 msgr2=0x7f6a2c0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:02.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.251+0000 7f6a2a59c700 1 --2- 172.21.15.1:0/3814643621 >> v2:172.21.15.165:3300/0 conn(0x7f6a2c073b00 0x7f6a2c0cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:02.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.251+0000 7f6a2a59c700 1 -- 172.21.15.1:0/3814643621 >> v1:172.21.15.165:6789/0 conn(0x7f6a2c15b3d0 legacy=0x7f6a2c0cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:02.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.251+0000 7f6a2a59c700 1 -- 172.21.15.1:0/3814643621 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2c0ce780 con 0x7f6a2c15aff0 2024-03-18T19:06:02.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.251+0000 7f6a2a59c700 1 --2- 172.21.15.1:0/3814643621 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a2c15aff0 0x7f6a2c15f780 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6a14009840 tx=0x7f6a14009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e5af6447251f6be server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:02.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.251+0000 7f6a2959a700 1 -- 172.21.15.1:0/3814643621 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a14011830 con 0x7f6a2c15aff0 2024-03-18T19:06:02.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.252+0000 7f6a2959a700 1 -- 172.21.15.1:0/3814643621 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a14011990 con 0x7f6a2c15aff0 2024-03-18T19:06:02.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.252+0000 7f6a2959a700 1 -- 172.21.15.1:0/3814643621 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a14011d20 con 0x7f6a2c15aff0 2024-03-18T19:06:02.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.252+0000 7f6a30df2700 1 -- 172.21.15.1:0/3814643621 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a2c15aff0 msgr2=0x7f6a2c15f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:02.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.252+0000 7f6a30df2700 1 --2- 172.21.15.1:0/3814643621 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a2c15aff0 0x7f6a2c15f780 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6a14009840 tx=0x7f6a14009b50 comp rx=0 tx=0).stop 2024-03-18T19:06:02.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.252+0000 7f6a30df2700 1 -- 172.21.15.1:0/3814643621 shutdown_connections 2024-03-18T19:06:02.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.252+0000 7f6a30df2700 1 --2- 172.21.15.1:0/3814643621 >> v2:172.21.15.165:3300/0 conn(0x7f6a2c073b00 0x7f6a2c0cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:02.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.252+0000 7f6a30df2700 1 --2- 172.21.15.1:0/3814643621 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a2c15aff0 0x7f6a2c15f780 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:02.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.252+0000 7f6a30df2700 1 -- 172.21.15.1:0/3814643621 >> 172.21.15.1:0/3814643621 conn(0x7f6a2c06a750 msgr2=0x7f6a2c06ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:02.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.252+0000 7f6a30df2700 1 -- 172.21.15.1:0/3814643621 shutdown_connections 2024-03-18T19:06:02.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.252+0000 7f6a30df2700 1 -- 172.21.15.1:0/3814643621 wait complete. 2024-03-18T19:06:02.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.253+0000 7f6a30df2700 1 Processor -- start 2024-03-18T19:06:02.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.254+0000 7f6a30df2700 1 -- start start 2024-03-18T19:06:02.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.254+0000 7f6a30df2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a2c073b00 0x7f6a2c0ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:02.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.254+0000 7f6a30df2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a2c15aff0 0x7f6a2c0cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:02.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.254+0000 7f6a30df2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a2c15b3d0 0x7f6a2c0cc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:02.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.254+0000 7f6a30df2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c0adeb0 con 0x7f6a2c15b3d0 2024-03-18T19:06:02.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.254+0000 7f6a30df2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c13b7a0 con 0x7f6a2c15aff0 2024-03-18T19:06:02.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.254+0000 7f6a30df2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c0b3270 con 0x7f6a2c073b00 2024-03-18T19:06:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.254+0000 7f6a2ad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a2c15b3d0 0x7f6a2c0cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.254+0000 7f6a2ad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a2c15b3d0 0x7f6a2c0cc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48754/0 (socket says 172.21.15.1:48754) 2024-03-18T19:06:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.254+0000 7f6a2ad9d700 1 -- 172.21.15.1:0/4232375422 learned_addr learned my addr 172.21.15.1:0/4232375422 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.254+0000 7f6a2a59c700 1 --2- 172.21.15.1:0/4232375422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a2c073b00 0x7f6a2c0ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.255+0000 7f6a29d9b700 1 --2- 172.21.15.1:0/4232375422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a2c15aff0 0x7f6a2c0cbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.255+0000 7f6a2ad9d700 1 -- 172.21.15.1:0/4232375422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a2c073b00 msgr2=0x7f6a2c0ae3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.255+0000 7f6a2ad9d700 1 --2- 172.21.15.1:0/4232375422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a2c073b00 0x7f6a2c0ae3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.255+0000 7f6a2ad9d700 1 -- 172.21.15.1:0/4232375422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a2c15aff0 msgr2=0x7f6a2c0cbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.255+0000 7f6a2ad9d700 1 --2- 172.21.15.1:0/4232375422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a2c15aff0 0x7f6a2c0cbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.255+0000 7f6a2ad9d700 1 -- 172.21.15.1:0/4232375422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a140094a0 con 0x7f6a2c15b3d0 2024-03-18T19:06:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.255+0000 7f6a2ad9d700 1 --2- 172.21.15.1:0/4232375422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a2c15b3d0 0x7f6a2c0cc260 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f6a2400e930 tx=0x7f6a2400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.255+0000 7f6a22ffd700 1 -- 172.21.15.1:0/4232375422 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a240096d0 con 0x7f6a2c15b3d0 2024-03-18T19:06:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.255+0000 7f6a22ffd700 1 -- 172.21.15.1:0/4232375422 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a24013680 con 0x7f6a2c15b3d0 2024-03-18T19:06:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.255+0000 7f6a30df2700 1 -- 172.21.15.1:0/4232375422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2c0acd20 con 0x7f6a2c15b3d0 2024-03-18T19:06:02.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.255+0000 7f6a22ffd700 1 -- 172.21.15.1:0/4232375422 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a24013a10 con 0x7f6a2c15b3d0 2024-03-18T19:06:02.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.255+0000 7f6a30df2700 1 -- 172.21.15.1:0/4232375422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a2c0b37a0 con 0x7f6a2c15b3d0 2024-03-18T19:06:02.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.256+0000 7f6a30df2700 1 -- 172.21.15.1:0/4232375422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a2c13b910 con 0x7f6a2c15b3d0 2024-03-18T19:06:02.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.257+0000 7f6a22ffd700 1 -- 172.21.15.1:0/4232375422 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6a24013b70 con 0x7f6a2c15b3d0 2024-03-18T19:06:02.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.257+0000 7f6a22ffd700 1 --2- 172.21.15.1:0/4232375422 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6a0c06adb0 0x7f6a0c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:02.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.257+0000 7f6a22ffd700 1 -- 172.21.15.1:0/4232375422 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f6a240891b0 con 0x7f6a2c15b3d0 2024-03-18T19:06:02.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.257+0000 7f6a2a59c700 1 --2- 172.21.15.1:0/4232375422 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6a0c06adb0 0x7f6a0c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:02.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.258+0000 7f6a2a59c700 1 --2- 172.21.15.1:0/4232375422 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6a0c06adb0 0x7f6a0c06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6a14000c00 tx=0x7f6a14008030 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:02.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.262+0000 7f6a22ffd700 1 -- 172.21.15.1:0/4232375422 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6a2c13b910 con 0x7f6a2c15b3d0 2024-03-18T19:06:02.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.465+0000 7f6a30df2700 1 -- 172.21.15.1:0/4232375422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6a2c13b910 con 0x7f6a2c15b3d0 2024-03-18T19:06:02.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.465+0000 7f6a22ffd700 1 -- 172.21.15.1:0/4232375422 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v99) v1 ==== 84+0+65 (secure 0 0 0) 0x7f6a2c13b910 con 0x7f6a2c15b3d0 2024-03-18T19:06:02.461 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:02.461 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T19:06:02.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:02.461 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-18T19:06:02.461 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-18T19:06:02.462 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:02.462 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T19:06:02.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.467+0000 7f6a30df2700 1 -- 172.21.15.1:0/4232375422 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6a0c06adb0 msgr2=0x7f6a0c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:02.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.467+0000 7f6a30df2700 1 --2- 172.21.15.1:0/4232375422 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6a0c06adb0 0x7f6a0c06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6a14000c00 tx=0x7f6a14008030 comp rx=0 tx=0).stop 2024-03-18T19:06:02.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.467+0000 7f6a30df2700 1 -- 172.21.15.1:0/4232375422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a2c15b3d0 msgr2=0x7f6a2c0cc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:02.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.467+0000 7f6a30df2700 1 --2- 172.21.15.1:0/4232375422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a2c15b3d0 0x7f6a2c0cc260 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f6a2400e930 tx=0x7f6a2400ec40 comp rx=0 tx=0).stop 2024-03-18T19:06:02.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.467+0000 7f6a30df2700 1 -- 172.21.15.1:0/4232375422 shutdown_connections 2024-03-18T19:06:02.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.467+0000 7f6a30df2700 1 --2- 172.21.15.1:0/4232375422 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6a0c06adb0 0x7f6a0c06d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:02.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.467+0000 7f6a30df2700 1 --2- 172.21.15.1:0/4232375422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a2c073b00 0x7f6a2c0ae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:02.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.467+0000 7f6a30df2700 1 --2- 172.21.15.1:0/4232375422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a2c15b3d0 0x7f6a2c0cc260 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:02.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.467+0000 7f6a30df2700 1 --2- 172.21.15.1:0/4232375422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a2c15aff0 0x7f6a2c0cbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:02.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.467+0000 7f6a30df2700 1 -- 172.21.15.1:0/4232375422 >> 172.21.15.1:0/4232375422 conn(0x7f6a2c06a750 msgr2=0x7f6a2c0b0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:02.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.467+0000 7f6a30df2700 1 -- 172.21.15.1:0/4232375422 shutdown_connections 2024-03-18T19:06:02.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.468+0000 7f6a30df2700 1 -- 172.21.15.1:0/4232375422 wait complete. 2024-03-18T19:06:02.475 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T19:06:02.475 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T19:06:02.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.608+0000 7f7fbd37f700 1 Processor -- start 2024-03-18T19:06:02.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.608+0000 7f7fbd37f700 1 -- start start 2024-03-18T19:06:02.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.609+0000 7f7fbd37f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7fb815d590 0x7f7fb80b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:02.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.609+0000 7f7fbd37f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb815d970 con 0x7f7fb806fe10 2024-03-18T19:06:02.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.609+0000 7f7fbd37f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb8072320 con 0x7f7fb806a750 2024-03-18T19:06:02.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.609+0000 7f7fbd37f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb8072600 con 0x7f7fb815d590 2024-03-18T19:06:02.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.609+0000 7f7fb6ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7fb815d590 0x7f7fb80b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:02.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.609+0000 7f7fb77fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7fb806fe10 0x7f7fb8167100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47038/0 (socket says 172.21.15.1:47038) 2024-03-18T19:06:02.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.609+0000 7f7fb77fe700 1 -- 172.21.15.1:0/1312942205 learned_addr learned my addr 172.21.15.1:0/1312942205 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:02.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.610+0000 7f7fb5ffb700 1 -- 172.21.15.1:0/1312942205 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408333829 0 0) 0x7f7fb815d970 con 0x7f7fb806fe10 2024-03-18T19:06:02.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.610+0000 7f7fb5ffb700 1 -- 172.21.15.1:0/1312942205 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa00035b0 con 0x7f7fb806fe10 2024-03-18T19:06:02.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.610+0000 7f7fb6ffd700 1 -- 172.21.15.1:0/1312942205 >> v1:172.21.15.165:6789/0 conn(0x7f7fb806a750 legacy=0x7f7fb806abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:02.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.610+0000 7f7fb6ffd700 1 -- 172.21.15.1:0/1312942205 >> v1:172.21.15.1:6789/0 conn(0x7f7fb806fe10 legacy=0x7f7fb8167100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:02.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.610+0000 7f7fb6ffd700 1 -- 172.21.15.1:0/1312942205 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb80b30b0 con 0x7f7fb815d590 2024-03-18T19:06:02.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.610+0000 7f7fb6ffd700 1 --2- 172.21.15.1:0/1312942205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fb815d590 0x7f7fb80b3bc0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009840 tx=0x7f7fac009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d69a316ef3affc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:02.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.610+0000 7f7fb5ffb700 1 -- 172.21.15.1:0/1312942205 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac004230 con 0x7f7fb815d590 2024-03-18T19:06:02.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.610+0000 7f7fb5ffb700 1 -- 172.21.15.1:0/1312942205 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fac004390 con 0x7f7fb815d590 2024-03-18T19:06:02.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.610+0000 7f7fb5ffb700 1 -- 172.21.15.1:0/1312942205 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac0036c0 con 0x7f7fb815d590 2024-03-18T19:06:02.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.611+0000 7f7fbd37f700 1 -- 172.21.15.1:0/1312942205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fb815d590 msgr2=0x7f7fb80b3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:02.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.611+0000 7f7fbd37f700 1 --2- 172.21.15.1:0/1312942205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fb815d590 0x7f7fb80b3bc0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009840 tx=0x7f7fac009b50 comp rx=0 tx=0).stop 2024-03-18T19:06:02.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.611+0000 7f7fbd37f700 1 -- 172.21.15.1:0/1312942205 shutdown_connections 2024-03-18T19:06:02.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.611+0000 7f7fbd37f700 1 --2- 172.21.15.1:0/1312942205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fb815d590 0x7f7fb80b3bc0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:02.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.611+0000 7f7fbd37f700 1 -- 172.21.15.1:0/1312942205 >> 172.21.15.1:0/1312942205 conn(0x7f7fb80d00c0 msgr2=0x7f7fb80d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:02.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.611+0000 7f7fbd37f700 1 -- 172.21.15.1:0/1312942205 shutdown_connections 2024-03-18T19:06:02.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.611+0000 7f7fbd37f700 1 -- 172.21.15.1:0/1312942205 wait complete. 2024-03-18T19:06:02.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.612+0000 7f7fbd37f700 1 Processor -- start 2024-03-18T19:06:02.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.612+0000 7f7fbd37f700 1 -- start start 2024-03-18T19:06:02.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.613+0000 7f7fbd37f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fb806a750 0x7f7fb815a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.613+0000 7f7fbd37f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fb806fe10 0x7f7fb8142910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.613+0000 7f7fbd37f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fb815d590 0x7f7fb81e2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.613+0000 7f7fbd37f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb815a990 con 0x7f7fb806a750 2024-03-18T19:06:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.613+0000 7f7fbd37f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb815abe0 con 0x7f7fb806fe10 2024-03-18T19:06:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.613+0000 7f7fbd37f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb815aec0 con 0x7f7fb815d590 2024-03-18T19:06:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.613+0000 7f7fb6ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fb806a750 0x7f7fb815a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.613+0000 7f7fb6ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fb806a750 0x7f7fb815a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48776/0 (socket says 172.21.15.1:48776) 2024-03-18T19:06:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.613+0000 7f7fb6ffd700 1 -- 172.21.15.1:0/3362242991 learned_addr learned my addr 172.21.15.1:0/3362242991 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.613+0000 7f7fb77fe700 1 --2- 172.21.15.1:0/3362242991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fb815d590 0x7f7fb81e2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.613+0000 7f7fb67fc700 1 --2- 172.21.15.1:0/3362242991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fb806fe10 0x7f7fb8142910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.613+0000 7f7fb6ffd700 1 -- 172.21.15.1:0/3362242991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fb815d590 msgr2=0x7f7fb81e2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.613+0000 7f7fb6ffd700 1 --2- 172.21.15.1:0/3362242991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fb815d590 0x7f7fb81e2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.613+0000 7f7fb6ffd700 1 -- 172.21.15.1:0/3362242991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fb806fe10 msgr2=0x7f7fb8142910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.613+0000 7f7fb6ffd700 1 --2- 172.21.15.1:0/3362242991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fb806fe10 0x7f7fb8142910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:02.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.613+0000 7f7fb6ffd700 1 -- 172.21.15.1:0/3362242991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac0094a0 con 0x7f7fb806a750 2024-03-18T19:06:02.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.613+0000 7f7fb6ffd700 1 --2- 172.21.15.1:0/3362242991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fb806a750 0x7f7fb815a000 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0090b0 tx=0x7f7fac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:02.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.613+0000 7f7f9f7fe700 1 -- 172.21.15.1:0/3362242991 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac011630 con 0x7f7fb806a750 2024-03-18T19:06:02.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.613+0000 7f7f9f7fe700 1 -- 172.21.15.1:0/3362242991 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fac011790 con 0x7f7fb806a750 2024-03-18T19:06:02.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.614+0000 7f7fbd37f700 1 -- 172.21.15.1:0/3362242991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb815b1a0 con 0x7f7fb806a750 2024-03-18T19:06:02.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.614+0000 7f7f9f7fe700 1 -- 172.21.15.1:0/3362242991 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac011b00 con 0x7f7fb806a750 2024-03-18T19:06:02.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.614+0000 7f7fbd37f700 1 -- 172.21.15.1:0/3362242991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb81e2d80 con 0x7f7fb806a750 2024-03-18T19:06:02.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.615+0000 7f7fbd37f700 1 -- 172.21.15.1:0/3362242991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb815b6c0 con 0x7f7fb806a750 2024-03-18T19:06:02.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.615+0000 7f7f9f7fe700 1 -- 172.21.15.1:0/3362242991 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7fac018040 con 0x7f7fb806a750 2024-03-18T19:06:02.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.615+0000 7f7f9f7fe700 1 --2- 172.21.15.1:0/3362242991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7fa406adb0 0x7f7fa406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:02.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.616+0000 7f7f9f7fe700 1 -- 172.21.15.1:0/3362242991 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f7fac089b50 con 0x7f7fb806a750 2024-03-18T19:06:02.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.616+0000 7f7fb67fc700 1 --2- 172.21.15.1:0/3362242991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7fa406adb0 0x7f7fa406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:02.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.616+0000 7f7fb67fc700 1 --2- 172.21.15.1:0/3362242991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7fa406adb0 0x7f7fa406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7fb80cf3a0 tx=0x7f7fa8009040 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:02.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.620+0000 7f7f9f7fe700 1 -- 172.21.15.1:0/3362242991 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7fb815b6c0 con 0x7f7fb806a750 2024-03-18T19:06:02.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:02.824+0000 7f7fbd37f700 1 -- 172.21.15.1:0/3362242991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7fb815b6c0 con 0x7f7fb806a750 2024-03-18T19:06:03.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.711+0000 7f7f9f7fe700 1 -- 172.21.15.1:0/3362242991 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v100) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7fb815b6c0 con 0x7f7fb806a750 2024-03-18T19:06:03.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.719+0000 7f7fbd37f700 1 -- 172.21.15.1:0/3362242991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7fa406adb0 msgr2=0x7f7fa406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.719+0000 7f7fbd37f700 1 --2- 172.21.15.1:0/3362242991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7fa406adb0 0x7f7fa406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7fb80cf3a0 tx=0x7f7fa8009040 comp rx=0 tx=0).stop 2024-03-18T19:06:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.719+0000 7f7fbd37f700 1 -- 172.21.15.1:0/3362242991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fb806a750 msgr2=0x7f7fb815a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.719+0000 7f7fbd37f700 1 --2- 172.21.15.1:0/3362242991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fb806a750 0x7f7fb815a000 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0090b0 tx=0x7f7fac004750 comp rx=0 tx=0).stop 2024-03-18T19:06:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.719+0000 7f7fbd37f700 1 -- 172.21.15.1:0/3362242991 shutdown_connections 2024-03-18T19:06:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.719+0000 7f7fbd37f700 1 --2- 172.21.15.1:0/3362242991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7fa406adb0 0x7f7fa406d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.719+0000 7f7fbd37f700 1 --2- 172.21.15.1:0/3362242991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fb815d590 0x7f7fb81e2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.719+0000 7f7fbd37f700 1 --2- 172.21.15.1:0/3362242991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fb806a750 0x7f7fb815a000 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.719+0000 7f7fbd37f700 1 --2- 172.21.15.1:0/3362242991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fb806fe10 0x7f7fb8142910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.720+0000 7f7fbd37f700 1 -- 172.21.15.1:0/3362242991 >> 172.21.15.1:0/3362242991 conn(0x7f7fb80d00c0 msgr2=0x7f7fb80ac9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.720+0000 7f7fbd37f700 1 -- 172.21.15.1:0/3362242991 shutdown_connections 2024-03-18T19:06:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.720+0000 7f7fbd37f700 1 -- 172.21.15.1:0/3362242991 wait complete. 2024-03-18T19:06:03.717 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-18T19:06:03.730 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T19:06:03.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.872+0000 7f8ad98bd700 1 Processor -- start 2024-03-18T19:06:03.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.872+0000 7f8ad98bd700 1 -- start start 2024-03-18T19:06:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.873+0000 7f8ad98bd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ad415a5f0 0x7f8ad40cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.873+0000 7f8ad98bd700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad415aad0 con 0x7f8ad40cc4d0 2024-03-18T19:06:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.873+0000 7f8ad98bd700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad415adb0 con 0x7f8ad415dbe0 2024-03-18T19:06:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.873+0000 7f8ad98bd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad413f750 con 0x7f8ad415a5f0 2024-03-18T19:06:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.873+0000 7f8ad37fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8ad40cc4d0 0x7f8ad406d500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47046/0 (socket says 172.21.15.1:47046) 2024-03-18T19:06:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.873+0000 7f8ad37fe700 1 -- 172.21.15.1:0/1264901505 learned_addr learned my addr 172.21.15.1:0/1264901505 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad1ffb700 1 -- 172.21.15.1:0/1264901505 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261213406 0 0) 0x7f8ad415aad0 con 0x7f8ad40cc4d0 2024-03-18T19:06:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad1ffb700 1 -- 172.21.15.1:0/1264901505 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8abc0035b0 con 0x7f8ad40cc4d0 2024-03-18T19:06:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad27fc700 1 --2- 172.21.15.1:0/1264901505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ad415a5f0 0x7f8ad40cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad1ffb700 1 -- 172.21.15.1:0/1264901505 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3939361259 0 0) 0x7f8abc0035b0 con 0x7f8ad40cc4d0 2024-03-18T19:06:03.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad1ffb700 1 -- 172.21.15.1:0/1264901505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ad415a5f0 msgr2=0x7f8ad40cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:03.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad1ffb700 1 --2- 172.21.15.1:0/1264901505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ad415a5f0 0x7f8ad40cbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:03.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad1ffb700 1 -- 172.21.15.1:0/1264901505 >> v1:172.21.15.165:6789/0 conn(0x7f8ad415dbe0 legacy=0x7f8ad40b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:03.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad1ffb700 1 -- 172.21.15.1:0/1264901505 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad415b090 con 0x7f8ad40cc4d0 2024-03-18T19:06:03.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad1ffb700 1 -- 172.21.15.1:0/1264901505 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f8acc0019f0 con 0x7f8ad40cc4d0 2024-03-18T19:06:03.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad1ffb700 1 -- 172.21.15.1:0/1264901505 >> v1:172.21.15.1:6789/0 conn(0x7f8ad40cc4d0 legacy=0x7f8ad406d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:03.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad1ffb700 1 --2- 172.21.15.1:0/1264901505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8abc003a30 0x7f8abc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:03.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad1ffb700 1 --2- 172.21.15.1:0/1264901505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ad415a5f0 0x7f8abc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:03.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad1ffb700 1 --2- 172.21.15.1:0/1264901505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8abc004d90 0x7f8abc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:03.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad1ffb700 1 -- 172.21.15.1:0/1264901505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad413f750 con 0x7f8abc004d90 2024-03-18T19:06:03.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad1ffb700 1 -- 172.21.15.1:0/1264901505 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8abc00bac0 con 0x7f8abc003a30 2024-03-18T19:06:03.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad1ffb700 1 -- 172.21.15.1:0/1264901505 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8abc00bc80 con 0x7f8ad415a5f0 2024-03-18T19:06:03.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad37fe700 1 --2- 172.21.15.1:0/1264901505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8abc004d90 0x7f8abc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:03.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad2ffd700 1 --2- 172.21.15.1:0/1264901505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8abc003a30 0x7f8abc003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:03.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad27fc700 1 --2- 172.21.15.1:0/1264901505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ad415a5f0 0x7f8abc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad37fe700 1 -- 172.21.15.1:0/1264901505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ad415a5f0 msgr2=0x7f8abc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad37fe700 1 --2- 172.21.15.1:0/1264901505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ad415a5f0 0x7f8abc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad37fe700 1 -- 172.21.15.1:0/1264901505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8abc003a30 msgr2=0x7f8abc003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad37fe700 1 --2- 172.21.15.1:0/1264901505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8abc003a30 0x7f8abc003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.874+0000 7f8ad37fe700 1 -- 172.21.15.1:0/1264901505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8abc0035b0 con 0x7f8abc004d90 2024-03-18T19:06:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.875+0000 7f8ad37fe700 1 --2- 172.21.15.1:0/1264901505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8abc004d90 0x7f8abc005210 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f8acc00a800 tx=0x7f8abc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f1ed83467c5b61 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.875+0000 7f8ad1ffb700 1 -- 172.21.15.1:0/1264901505 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc00a2f0 con 0x7f8abc004d90 2024-03-18T19:06:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.875+0000 7f8ad1ffb700 1 -- 172.21.15.1:0/1264901505 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8acc012620 con 0x7f8abc004d90 2024-03-18T19:06:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.875+0000 7f8ad1ffb700 1 -- 172.21.15.1:0/1264901505 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc0129d0 con 0x7f8abc004d90 2024-03-18T19:06:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.875+0000 7f8ad98bd700 1 -- 172.21.15.1:0/1264901505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8abc004d90 msgr2=0x7f8abc005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.875+0000 7f8ad98bd700 1 --2- 172.21.15.1:0/1264901505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8abc004d90 0x7f8abc005210 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f8acc00a800 tx=0x7f8abc005880 comp rx=0 tx=0).stop 2024-03-18T19:06:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.875+0000 7f8ad37fe700 1 -- 172.21.15.1:0/1264901505 reap_dead start 2024-03-18T19:06:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.875+0000 7f8ad98bd700 1 -- 172.21.15.1:0/1264901505 shutdown_connections 2024-03-18T19:06:03.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.875+0000 7f8ad98bd700 1 -- 172.21.15.1:0/1264901505 >> 172.21.15.1:0/1264901505 conn(0x7f8ad40b1cb0 msgr2=0x7f8ad40b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:03.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.875+0000 7f8ad98bd700 1 -- 172.21.15.1:0/1264901505 shutdown_connections 2024-03-18T19:06:03.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.875+0000 7f8ad98bd700 1 -- 172.21.15.1:0/1264901505 wait complete. 2024-03-18T19:06:03.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.877+0000 7f8ad98bd700 1 Processor -- start 2024-03-18T19:06:03.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.877+0000 7f8ad98bd700 1 -- start start 2024-03-18T19:06:03.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.877+0000 7f8ad98bd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ad4139f50 0x7f8ad41541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.877+0000 7f8ad98bd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ad413a6f0 0x7f8ad41f1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.877+0000 7f8ad98bd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ad41f1880 0x7f8ad413db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.877+0000 7f8ad98bd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad40cf8b0 con 0x7f8ad41f1880 2024-03-18T19:06:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.877+0000 7f8ad98bd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad40cebd0 con 0x7f8ad413a6f0 2024-03-18T19:06:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.877+0000 7f8ad98bd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad40ceeb0 con 0x7f8ad4139f50 2024-03-18T19:06:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.878+0000 7f8ad37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ad41f1880 0x7f8ad413db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.878+0000 7f8ad37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ad41f1880 0x7f8ad413db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48788/0 (socket says 172.21.15.1:48788) 2024-03-18T19:06:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.878+0000 7f8ad37fe700 1 -- 172.21.15.1:0/1055211335 learned_addr learned my addr 172.21.15.1:0/1055211335 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.878+0000 7f8ad2ffd700 1 --2- 172.21.15.1:0/1055211335 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ad4139f50 0x7f8ad41541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.878+0000 7f8ad27fc700 1 --2- 172.21.15.1:0/1055211335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ad413a6f0 0x7f8ad41f1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.878+0000 7f8ad37fe700 1 -- 172.21.15.1:0/1055211335 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ad4139f50 msgr2=0x7f8ad41541b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.878+0000 7f8ad37fe700 1 --2- 172.21.15.1:0/1055211335 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ad4139f50 0x7f8ad41541b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.878+0000 7f8ad37fe700 1 -- 172.21.15.1:0/1055211335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ad413a6f0 msgr2=0x7f8ad41f1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.878+0000 7f8ad37fe700 1 --2- 172.21.15.1:0/1055211335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ad413a6f0 0x7f8ad41f1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.878+0000 7f8ad37fe700 1 -- 172.21.15.1:0/1055211335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad40cf540 con 0x7f8ad41f1880 2024-03-18T19:06:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.878+0000 7f8ad37fe700 1 --2- 172.21.15.1:0/1055211335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ad41f1880 0x7f8ad413db70 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f8acc000f80 tx=0x7f8acc001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.878+0000 7f8abbfff700 1 -- 172.21.15.1:0/1055211335 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc0011c0 con 0x7f8ad41f1880 2024-03-18T19:06:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.878+0000 7f8abbfff700 1 -- 172.21.15.1:0/1055211335 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8acc0017b0 con 0x7f8ad41f1880 2024-03-18T19:06:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.878+0000 7f8ad98bd700 1 -- 172.21.15.1:0/1055211335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad41546f0 con 0x7f8ad41f1880 2024-03-18T19:06:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.878+0000 7f8abbfff700 1 -- 172.21.15.1:0/1055211335 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc01e430 con 0x7f8ad41f1880 2024-03-18T19:06:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.878+0000 7f8ad98bd700 1 -- 172.21.15.1:0/1055211335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad413a510 con 0x7f8ad41f1880 2024-03-18T19:06:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.879+0000 7f8ad98bd700 1 -- 172.21.15.1:0/1055211335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad40cfa60 con 0x7f8ad41f1880 2024-03-18T19:06:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.880+0000 7f8abbfff700 1 -- 172.21.15.1:0/1055211335 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8acc01e7c0 con 0x7f8ad41f1880 2024-03-18T19:06:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.880+0000 7f8abbfff700 1 --2- 172.21.15.1:0/1055211335 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8ac006adb0 0x7f8ac006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.881+0000 7f8abbfff700 1 -- 172.21.15.1:0/1055211335 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f8acc08bea0 con 0x7f8ad41f1880 2024-03-18T19:06:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.881+0000 7f8ad2ffd700 1 --2- 172.21.15.1:0/1055211335 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8ac006adb0 0x7f8ac006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.881+0000 7f8ad2ffd700 1 --2- 172.21.15.1:0/1055211335 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8ac006adb0 0x7f8ac006d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8007fd0 tx=0x7f8ac8009040 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:03.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:03.885+0000 7f8abbfff700 1 -- 172.21.15.1:0/1055211335 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8acc0570c0 con 0x7f8ad41f1880 2024-03-18T19:06:04.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.089+0000 7f8ad98bd700 1 -- 172.21.15.1:0/1055211335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8ad40cfa60 con 0x7f8ad41f1880 2024-03-18T19:06:04.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-03-18T19:06:04.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-18T19:06:04.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.717+0000 7f8abbfff700 1 -- 172.21.15.1:0/1055211335 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v101) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8ad40cfa60 con 0x7f8ad41f1880 2024-03-18T19:06:04.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.725+0000 7f8ad98bd700 1 -- 172.21.15.1:0/1055211335 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8ac006adb0 msgr2=0x7f8ac006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:04.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.725+0000 7f8ad98bd700 1 --2- 172.21.15.1:0/1055211335 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8ac006adb0 0x7f8ac006d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8007fd0 tx=0x7f8ac8009040 comp rx=0 tx=0).stop 2024-03-18T19:06:04.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.725+0000 7f8ad98bd700 1 -- 172.21.15.1:0/1055211335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ad41f1880 msgr2=0x7f8ad413db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:04.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.725+0000 7f8ad98bd700 1 --2- 172.21.15.1:0/1055211335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ad41f1880 0x7f8ad413db70 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f8acc000f80 tx=0x7f8acc001d80 comp rx=0 tx=0).stop 2024-03-18T19:06:04.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.725+0000 7f8ad98bd700 1 -- 172.21.15.1:0/1055211335 shutdown_connections 2024-03-18T19:06:04.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.725+0000 7f8ad98bd700 1 --2- 172.21.15.1:0/1055211335 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8ac006adb0 0x7f8ac006d270 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8007fd0 tx=0x7f8ac8009040 comp rx=0 tx=0).stop 2024-03-18T19:06:04.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.725+0000 7f8ad98bd700 1 --2- 172.21.15.1:0/1055211335 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ad4139f50 0x7f8ad41541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:04.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.725+0000 7f8ad98bd700 1 --2- 172.21.15.1:0/1055211335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ad41f1880 0x7f8ad413db70 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:04.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.725+0000 7f8ad98bd700 1 --2- 172.21.15.1:0/1055211335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ad413a6f0 0x7f8ad41f1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:04.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.725+0000 7f8ad98bd700 1 -- 172.21.15.1:0/1055211335 >> 172.21.15.1:0/1055211335 conn(0x7f8ad40b1cb0 msgr2=0x7f8ad413c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:04.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.728+0000 7f8ad98bd700 1 -- 172.21.15.1:0/1055211335 shutdown_connections 2024-03-18T19:06:04.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.732+0000 7f8ad98bd700 1 -- 172.21.15.1:0/1055211335 wait complete. 2024-03-18T19:06:04.733 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-18T19:06:04.746 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T19:06:04.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.883+0000 7f1b778f2700 1 Processor -- start 2024-03-18T19:06:04.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.884+0000 7f1b778f2700 1 -- start start 2024-03-18T19:06:04.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.884+0000 7f1b778f2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1b7015d500 0x7f1b7006c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:04.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.884+0000 7f1b778f2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b7009a150 0x7f1b700cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:04.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.884+0000 7f1b778f2700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7006f490 con 0x7f1b7006c7b0 2024-03-18T19:06:04.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.884+0000 7f1b778f2700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b7006e880 con 0x7f1b7015d500 2024-03-18T19:06:04.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.884+0000 7f1b778f2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7006eb60 con 0x7f1b7009a150 2024-03-18T19:06:04.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.884+0000 7f1b7568e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1b7015d500 0x7f1b7006c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:04.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.885+0000 7f1b74e8d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1b7006c7b0 0x7f1b70099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50926/0 (socket says 172.21.15.1:50926) 2024-03-18T19:06:04.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.885+0000 7f1b7568e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1b7015d500 0x7f1b7006c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40782/0 (socket says 172.21.15.1:40782) 2024-03-18T19:06:04.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.885+0000 7f1b74e8d700 1 -- 172.21.15.1:0/3695431849 learned_addr learned my addr 172.21.15.1:0/3695431849 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:04.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.885+0000 7f1b75e8f700 1 --2- 172.21.15.1:0/3695431849 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b7009a150 0x7f1b700cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:04.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.885+0000 7f1b7568e700 1 -- 172.21.15.1:0/3695431849 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b7009a150 msgr2=0x7f1b700cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:04.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.885+0000 7f1b7568e700 1 --2- 172.21.15.1:0/3695431849 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b7009a150 0x7f1b700cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:04.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.885+0000 7f1b7568e700 1 -- 172.21.15.1:0/3695431849 >> v1:172.21.15.1:6789/0 conn(0x7f1b7006c7b0 legacy=0x7f1b70099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:04.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.885+0000 7f1b67fff700 1 -- 172.21.15.1:0/3695431849 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365469837 0 0) 0x7f1b7006f490 con 0x7f1b7006c7b0 2024-03-18T19:06:04.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.885+0000 7f1b7568e700 1 -- 172.21.15.1:0/3695431849 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7006ee40 con 0x7f1b7015d500 2024-03-18T19:06:04.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.885+0000 7f1b75e8f700 1 --2- 172.21.15.1:0/3695431849 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b7009a150 0x7f1b700cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:04.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.885+0000 7f1b7568e700 1 --2- 172.21.15.1:0/3695431849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b7015d500 0x7f1b7006c270 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f1b60009840 tx=0x7f1b60009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa623b48a4e0e91 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:04.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.885+0000 7f1b67fff700 1 -- 172.21.15.1:0/3695431849 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60011830 con 0x7f1b7015d500 2024-03-18T19:06:04.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.885+0000 7f1b67fff700 1 -- 172.21.15.1:0/3695431849 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b60011990 con 0x7f1b7015d500 2024-03-18T19:06:04.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.885+0000 7f1b67fff700 1 -- 172.21.15.1:0/3695431849 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60011d20 con 0x7f1b7015d500 2024-03-18T19:06:04.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.886+0000 7f1b778f2700 1 -- 172.21.15.1:0/3695431849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b7015d500 msgr2=0x7f1b7006c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:04.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.886+0000 7f1b778f2700 1 --2- 172.21.15.1:0/3695431849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b7015d500 0x7f1b7006c270 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f1b60009840 tx=0x7f1b60009b50 comp rx=0 tx=0).stop 2024-03-18T19:06:04.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.886+0000 7f1b778f2700 1 -- 172.21.15.1:0/3695431849 shutdown_connections 2024-03-18T19:06:04.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.886+0000 7f1b778f2700 1 --2- 172.21.15.1:0/3695431849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b7015d500 0x7f1b7006c270 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:04.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.886+0000 7f1b778f2700 1 --2- 172.21.15.1:0/3695431849 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b7009a150 0x7f1b700cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:04.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.886+0000 7f1b778f2700 1 -- 172.21.15.1:0/3695431849 >> 172.21.15.1:0/3695431849 conn(0x7f1b700b2c50 msgr2=0x7f1b700b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:04.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.886+0000 7f1b778f2700 1 -- 172.21.15.1:0/3695431849 shutdown_connections 2024-03-18T19:06:04.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.886+0000 7f1b778f2700 1 -- 172.21.15.1:0/3695431849 wait complete. 2024-03-18T19:06:04.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.887+0000 7f1b778f2700 1 Processor -- start 2024-03-18T19:06:04.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.887+0000 7f1b778f2700 1 -- start start 2024-03-18T19:06:04.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.888+0000 7f1b778f2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b7006c7b0 0x7f1b7014e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:04.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.888+0000 7f1b778f2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b7009a150 0x7f1b70150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:04.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.888+0000 7f1b778f2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b7015d500 0x7f1b700c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:04.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.888+0000 7f1b778f2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7014f7c0 con 0x7f1b7009a150 2024-03-18T19:06:04.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.888+0000 7f1b778f2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b700c9c80 con 0x7f1b7006c7b0 2024-03-18T19:06:04.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.888+0000 7f1b778f2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7014f9a0 con 0x7f1b7015d500 2024-03-18T19:06:04.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.888+0000 7f1b74e8d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b7009a150 0x7f1b70150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:04.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.888+0000 7f1b74e8d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b7009a150 0x7f1b70150760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40798/0 (socket says 172.21.15.1:40798) 2024-03-18T19:06:04.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.888+0000 7f1b74e8d700 1 -- 172.21.15.1:0/3236910311 learned_addr learned my addr 172.21.15.1:0/3236910311 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:04.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.888+0000 7f1b75e8f700 1 --2- 172.21.15.1:0/3236910311 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b7015d500 0x7f1b700c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:04.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.888+0000 7f1b7568e700 1 --2- 172.21.15.1:0/3236910311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b7006c7b0 0x7f1b7014e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:04.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.888+0000 7f1b74e8d700 1 -- 172.21.15.1:0/3236910311 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b7015d500 msgr2=0x7f1b700c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:04.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.888+0000 7f1b74e8d700 1 --2- 172.21.15.1:0/3236910311 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b7015d500 0x7f1b700c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:04.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.888+0000 7f1b74e8d700 1 -- 172.21.15.1:0/3236910311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b7006c7b0 msgr2=0x7f1b7014e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:04.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.888+0000 7f1b74e8d700 1 --2- 172.21.15.1:0/3236910311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b7006c7b0 0x7f1b7014e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:04.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.888+0000 7f1b74e8d700 1 -- 172.21.15.1:0/3236910311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b600094a0 con 0x7f1b7009a150 2024-03-18T19:06:04.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.888+0000 7f1b74e8d700 1 --2- 172.21.15.1:0/3236910311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b7009a150 0x7f1b70150760 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0011e0 tx=0x7f1b6c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:04.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.888+0000 7f1b65ffb700 1 -- 172.21.15.1:0/3236910311 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c004c80 con 0x7f1b7009a150 2024-03-18T19:06:04.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.888+0000 7f1b65ffb700 1 -- 172.21.15.1:0/3236910311 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6c004e00 con 0x7f1b7009a150 2024-03-18T19:06:04.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.889+0000 7f1b778f2700 1 -- 172.21.15.1:0/3236910311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7014fc80 con 0x7f1b7009a150 2024-03-18T19:06:04.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.889+0000 7f1b65ffb700 1 -- 172.21.15.1:0/3236910311 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c007310 con 0x7f1b7009a150 2024-03-18T19:06:04.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.889+0000 7f1b778f2700 1 -- 172.21.15.1:0/3236910311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b70150240 con 0x7f1b7009a150 2024-03-18T19:06:04.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.890+0000 7f1b778f2700 1 -- 172.21.15.1:0/3236910311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b70150460 con 0x7f1b7009a150 2024-03-18T19:06:04.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.890+0000 7f1b65ffb700 1 -- 172.21.15.1:0/3236910311 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1b6c007470 con 0x7f1b7009a150 2024-03-18T19:06:04.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.891+0000 7f1b65ffb700 1 --2- 172.21.15.1:0/3236910311 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1b5406adb0 0x7f1b5406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:04.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.891+0000 7f1b7568e700 1 --2- 172.21.15.1:0/3236910311 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1b5406adb0 0x7f1b5406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:04.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.891+0000 7f1b65ffb700 1 -- 172.21.15.1:0/3236910311 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f1b6c0892d0 con 0x7f1b7009a150 2024-03-18T19:06:04.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.891+0000 7f1b7568e700 1 --2- 172.21.15.1:0/3236910311 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1b5406adb0 0x7f1b5406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1b600090b0 tx=0x7f1b6001c040 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:04.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:04.896+0000 7f1b65ffb700 1 -- 172.21.15.1:0/3236910311 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1b6c0542c0 con 0x7f1b7009a150 2024-03-18T19:06:05.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.146+0000 7f1b778f2700 1 -- 172.21.15.1:0/3236910311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1b700676a0 con 0x7f1b7009a150 2024-03-18T19:06:05.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.150+0000 7f1b65ffb700 1 -- 172.21.15.1:0/3236910311 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v38) v1 ==== 174+0+0 (secure 0 0 0) 0x7f1b6c0578e0 con 0x7f1b7009a150 2024-03-18T19:06:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.156+0000 7f1b778f2700 1 -- 172.21.15.1:0/3236910311 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1b5406adb0 msgr2=0x7f1b5406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.156+0000 7f1b778f2700 1 --2- 172.21.15.1:0/3236910311 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1b5406adb0 0x7f1b5406d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1b600090b0 tx=0x7f1b6001c040 comp rx=0 tx=0).stop 2024-03-18T19:06:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.157+0000 7f1b778f2700 1 -- 172.21.15.1:0/3236910311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b7009a150 msgr2=0x7f1b70150760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.157+0000 7f1b778f2700 1 --2- 172.21.15.1:0/3236910311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b7009a150 0x7f1b70150760 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0011e0 tx=0x7f1b6c00cae0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.157+0000 7f1b778f2700 1 -- 172.21.15.1:0/3236910311 shutdown_connections 2024-03-18T19:06:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.157+0000 7f1b778f2700 1 --2- 172.21.15.1:0/3236910311 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1b5406adb0 0x7f1b5406d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.157+0000 7f1b778f2700 1 --2- 172.21.15.1:0/3236910311 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b7015d500 0x7f1b700c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.157+0000 7f1b778f2700 1 --2- 172.21.15.1:0/3236910311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b7006c7b0 0x7f1b7014e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.157+0000 7f1b778f2700 1 --2- 172.21.15.1:0/3236910311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b7009a150 0x7f1b70150760 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.157+0000 7f1b778f2700 1 -- 172.21.15.1:0/3236910311 >> 172.21.15.1:0/3236910311 conn(0x7f1b700b2c50 msgr2=0x7f1b700ade00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:05.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.159+0000 7f1b778f2700 1 -- 172.21.15.1:0/3236910311 shutdown_connections 2024-03-18T19:06:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.164+0000 7f1b778f2700 1 -- 172.21.15.1:0/3236910311 wait complete. 2024-03-18T19:06:05.171 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 11 and data pool 12 2024-03-18T19:06:05.186 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:06:05.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.315+0000 7f247753f700 1 Processor -- start 2024-03-18T19:06:05.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.316+0000 7f247753f700 1 -- start start 2024-03-18T19:06:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.316+0000 7f247753f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f247006ae60 0x7f2470164f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.316+0000 7f247753f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2470072660 con 0x7f24701465a0 2024-03-18T19:06:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.316+0000 7f247753f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247006ebd0 con 0x7f2470146190 2024-03-18T19:06:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.316+0000 7f247753f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f247006f850 con 0x7f247006ae60 2024-03-18T19:06:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.317+0000 7f2475adc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f24701465a0 0x7f2470167480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50938/0 (socket says 172.21.15.1:50938) 2024-03-18T19:06:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.317+0000 7f2475adc700 1 -- 172.21.15.1:0/1472923566 learned_addr learned my addr 172.21.15.1:0/1472923566 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:05.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.317+0000 7f2474ada700 1 --2- 172.21.15.1:0/1472923566 >> v2:172.21.15.165:3300/0 conn(0x7f247006ae60 0x7f2470164f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.317+0000 7f2463fff700 1 -- 172.21.15.1:0/1472923566 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990176296 0 0) 0x7f2470072660 con 0x7f24701465a0 2024-03-18T19:06:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.317+0000 7f2463fff700 1 -- 172.21.15.1:0/1472923566 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f245c0035b0 con 0x7f24701465a0 2024-03-18T19:06:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.317+0000 7f2463fff700 1 -- 172.21.15.1:0/1472923566 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591747415 0 0) 0x7f247006ebd0 con 0x7f2470146190 2024-03-18T19:06:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.317+0000 7f2463fff700 1 -- 172.21.15.1:0/1472923566 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2470072660 con 0x7f2470146190 2024-03-18T19:06:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.317+0000 7f2474ada700 1 -- 172.21.15.1:0/1472923566 >> v1:172.21.15.165:6789/0 conn(0x7f2470146190 legacy=0x7f247006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:05.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.317+0000 7f2463fff700 1 -- 172.21.15.1:0/1472923566 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2700256219 0 0) 0x7f245c0035b0 con 0x7f24701465a0 2024-03-18T19:06:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.317+0000 7f2474ada700 1 -- 172.21.15.1:0/1472923566 >> v1:172.21.15.1:6789/0 conn(0x7f24701465a0 legacy=0x7f2470167480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.317+0000 7f2474ada700 1 -- 172.21.15.1:0/1472923566 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247006eeb0 con 0x7f247006ae60 2024-03-18T19:06:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.317+0000 7f2474ada700 1 --2- 172.21.15.1:0/1472923566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f247006ae60 0x7f2470164f30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2464009840 tx=0x7f2464009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5780d9203a5bc540 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.317+0000 7f2463fff700 1 -- 172.21.15.1:0/1472923566 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24680019f0 con 0x7f247006ae60 2024-03-18T19:06:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.317+0000 7f2463fff700 1 -- 172.21.15.1:0/1472923566 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2464004030 con 0x7f247006ae60 2024-03-18T19:06:05.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.318+0000 7f2463fff700 1 -- 172.21.15.1:0/1472923566 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24640043c0 con 0x7f247006ae60 2024-03-18T19:06:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.318+0000 7f247753f700 1 -- 172.21.15.1:0/1472923566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f247006ae60 msgr2=0x7f2470164f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.318+0000 7f247753f700 1 --2- 172.21.15.1:0/1472923566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f247006ae60 0x7f2470164f30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2464009840 tx=0x7f2464009b50 comp rx=0 tx=0).stop 2024-03-18T19:06:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.318+0000 7f247753f700 1 -- 172.21.15.1:0/1472923566 shutdown_connections 2024-03-18T19:06:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.318+0000 7f247753f700 1 --2- 172.21.15.1:0/1472923566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f247006ae60 0x7f2470164f30 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.318+0000 7f247753f700 1 -- 172.21.15.1:0/1472923566 >> 172.21.15.1:0/1472923566 conn(0x7f24700b1d20 msgr2=0x7f24700b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:05.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.318+0000 7f247753f700 1 -- 172.21.15.1:0/1472923566 shutdown_connections 2024-03-18T19:06:05.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.318+0000 7f247753f700 1 -- 172.21.15.1:0/1472923566 wait complete. 2024-03-18T19:06:05.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.319+0000 7f247753f700 1 Processor -- start 2024-03-18T19:06:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.319+0000 7f247753f700 1 -- start start 2024-03-18T19:06:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.320+0000 7f247753f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f247006ae60 0x7f2470140830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.320+0000 7f247753f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24701465a0 0x7f24700c53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.320+0000 7f247753f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24700cb6a0 0x7f247012f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.320+0000 7f247753f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2470141b40 con 0x7f24701465a0 2024-03-18T19:06:05.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.320+0000 7f247753f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24700c5920 con 0x7f24700cb6a0 2024-03-18T19:06:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.320+0000 7f247753f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2470140f30 con 0x7f247006ae60 2024-03-18T19:06:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.320+0000 7f2474ada700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24701465a0 0x7f24700c53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.320+0000 7f2474ada700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24701465a0 0x7f24700c53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40808/0 (socket says 172.21.15.1:40808) 2024-03-18T19:06:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.320+0000 7f2474ada700 1 -- 172.21.15.1:0/2458603887 learned_addr learned my addr 172.21.15.1:0/2458603887 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.320+0000 7f24752db700 1 --2- 172.21.15.1:0/2458603887 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f247006ae60 0x7f2470140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.320+0000 7f2475adc700 1 --2- 172.21.15.1:0/2458603887 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24700cb6a0 0x7f247012f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:05.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.320+0000 7f2474ada700 1 -- 172.21.15.1:0/2458603887 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f247006ae60 msgr2=0x7f2470140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.320+0000 7f2474ada700 1 --2- 172.21.15.1:0/2458603887 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f247006ae60 0x7f2470140830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.320+0000 7f2474ada700 1 -- 172.21.15.1:0/2458603887 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24700cb6a0 msgr2=0x7f247012f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.320+0000 7f2474ada700 1 --2- 172.21.15.1:0/2458603887 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24700cb6a0 0x7f247012f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.320+0000 7f2474ada700 1 -- 172.21.15.1:0/2458603887 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24640094a0 con 0x7f24701465a0 2024-03-18T19:06:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.321+0000 7f2474ada700 1 --2- 172.21.15.1:0/2458603887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24701465a0 0x7f24700c53e0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f2464009b90 tx=0x7f2464004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:05.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.321+0000 7f2461ffb700 1 -- 172.21.15.1:0/2458603887 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2464004570 con 0x7f24701465a0 2024-03-18T19:06:05.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.321+0000 7f2461ffb700 1 -- 172.21.15.1:0/2458603887 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246401f070 con 0x7f24701465a0 2024-03-18T19:06:05.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.321+0000 7f247753f700 1 -- 172.21.15.1:0/2458603887 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2470141210 con 0x7f24701465a0 2024-03-18T19:06:05.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.321+0000 7f2461ffb700 1 -- 172.21.15.1:0/2458603887 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2464011870 con 0x7f24701465a0 2024-03-18T19:06:05.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.321+0000 7f247753f700 1 -- 172.21.15.1:0/2458603887 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24701417d0 con 0x7f24701465a0 2024-03-18T19:06:05.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.322+0000 7f247753f700 1 -- 172.21.15.1:0/2458603887 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2470141cb0 con 0x7f24701465a0 2024-03-18T19:06:05.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.322+0000 7f2461ffb700 1 -- 172.21.15.1:0/2458603887 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f24640119d0 con 0x7f24701465a0 2024-03-18T19:06:05.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.323+0000 7f2461ffb700 1 --2- 172.21.15.1:0/2458603887 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f245406ade0 0x7f245406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:05.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.323+0000 7f2461ffb700 1 -- 172.21.15.1:0/2458603887 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2464021070 con 0x7f24701465a0 2024-03-18T19:06:05.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.323+0000 7f24752db700 1 --2- 172.21.15.1:0/2458603887 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f245406ade0 0x7f245406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:05.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.323+0000 7f24752db700 1 --2- 172.21.15.1:0/2458603887 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f245406ade0 0x7f245406d2a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f246c0011e0 tx=0x7f246c009040 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:05.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.327+0000 7f2461ffb700 1 -- 172.21.15.1:0/2458603887 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2464055d70 con 0x7f24701465a0 2024-03-18T19:06:05.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.530+0000 7f247753f700 1 -- 172.21.15.1:0/2458603887 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24700676a0 con 0x7f24701465a0 2024-03-18T19:06:05.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.531+0000 7f2461ffb700 1 -- 172.21.15.1:0/2458603887 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+16285 (secure 0 0 0) 0x7f246402b090 con 0x7f24701465a0 2024-03-18T19:06:05.527 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:05.527 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":102,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:06:05.147292+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:06:02.855989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-18T19:06:04.121199+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:06:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.534+0000 7f247753f700 1 -- 172.21.15.1:0/2458603887 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f245406ade0 msgr2=0x7f245406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.534+0000 7f247753f700 1 --2- 172.21.15.1:0/2458603887 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f245406ade0 0x7f245406d2a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f246c0011e0 tx=0x7f246c009040 comp rx=0 tx=0).stop 2024-03-18T19:06:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.534+0000 7f247753f700 1 -- 172.21.15.1:0/2458603887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24701465a0 msgr2=0x7f24700c53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.534+0000 7f247753f700 1 --2- 172.21.15.1:0/2458603887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24701465a0 0x7f24700c53e0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f2464009b90 tx=0x7f2464004750 comp rx=0 tx=0).stop 2024-03-18T19:06:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.534+0000 7f247753f700 1 -- 172.21.15.1:0/2458603887 shutdown_connections 2024-03-18T19:06:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.534+0000 7f247753f700 1 --2- 172.21.15.1:0/2458603887 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f245406ade0 0x7f245406d2a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.534+0000 7f247753f700 1 --2- 172.21.15.1:0/2458603887 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f247006ae60 0x7f2470140830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.534+0000 7f247753f700 1 --2- 172.21.15.1:0/2458603887 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24700cb6a0 0x7f247012f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.534+0000 7f247753f700 1 --2- 172.21.15.1:0/2458603887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24701465a0 0x7f24700c53e0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.534+0000 7f247753f700 1 -- 172.21.15.1:0/2458603887 >> 172.21.15.1:0/2458603887 conn(0x7f24700b1d20 msgr2=0x7f24700b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:05.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.534+0000 7f247753f700 1 -- 172.21.15.1:0/2458603887 shutdown_connections 2024-03-18T19:06:05.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.534+0000 7f247753f700 1 -- 172.21.15.1:0/2458603887 wait complete. 2024-03-18T19:06:05.542 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:06:05.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.671+0000 7f9c234f5700 1 Processor -- start 2024-03-18T19:06:05.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.671+0000 7f9c234f5700 1 -- start start 2024-03-18T19:06:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.672+0000 7f9c234f5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c1c146190 0x7f9c1c06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.672+0000 7f9c234f5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9c1c1465a0 0x7f9c1c06ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.672+0000 7f9c234f5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9c1c146af0 0x7f9c1c071650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.672+0000 7f9c234f5700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c1c06f850 con 0x7f9c1c146af0 2024-03-18T19:06:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.672+0000 7f9c234f5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c1c06ebd0 con 0x7f9c1c1465a0 2024-03-18T19:06:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.672+0000 7f9c234f5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c06eeb0 con 0x7f9c1c146190 2024-03-18T19:06:05.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.672+0000 7f9c21a92700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9c1c146af0 0x7f9c1c071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.672+0000 7f9c21a92700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9c1c146af0 0x7f9c1c071650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40824/0 (socket says 172.21.15.1:40824) 2024-03-18T19:06:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.672+0000 7f9c21a92700 1 -- 172.21.15.1:0/1738699610 learned_addr learned my addr 172.21.15.1:0/1738699610 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.672+0000 7f9c21291700 1 --2- 172.21.15.1:0/1738699610 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c1c146190 0x7f9c1c06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.673+0000 7f9c20a90700 1 --2- 172.21.15.1:0/1738699610 >> v2:172.21.15.165:3300/0 conn(0x7f9c1c1465a0 0x7f9c1c06ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.673+0000 7f9c21a92700 1 -- 172.21.15.1:0/1738699610 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c1c146190 msgr2=0x7f9c1c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.673+0000 7f9c21a92700 1 --2- 172.21.15.1:0/1738699610 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c1c146190 0x7f9c1c06a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.673+0000 7f9c21a92700 1 -- 172.21.15.1:0/1738699610 >> v2:172.21.15.165:3300/0 conn(0x7f9c1c1465a0 msgr2=0x7f9c1c06ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.673+0000 7f9c21a92700 1 --2- 172.21.15.1:0/1738699610 >> v2:172.21.15.165:3300/0 conn(0x7f9c1c1465a0 0x7f9c1c06ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.673+0000 7f9c21a92700 1 -- 172.21.15.1:0/1738699610 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1c06f190 con 0x7f9c1c146af0 2024-03-18T19:06:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.673+0000 7f9c21a92700 1 --2- 172.21.15.1:0/1738699610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c146af0 0x7f9c1c071650 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c0b62e0 tx=0x7f9c1400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b756916ecb74a056 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:05.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.673+0000 7f9c13fff700 1 -- 172.21.15.1:0/1738699610 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1400e070 con 0x7f9c1c146af0 2024-03-18T19:06:05.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.673+0000 7f9c13fff700 1 -- 172.21.15.1:0/1738699610 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1400b9c0 con 0x7f9c1c146af0 2024-03-18T19:06:05.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.673+0000 7f9c13fff700 1 -- 172.21.15.1:0/1738699610 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1400bd70 con 0x7f9c1c146af0 2024-03-18T19:06:05.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.673+0000 7f9c234f5700 1 -- 172.21.15.1:0/1738699610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c146af0 msgr2=0x7f9c1c071650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:05.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.673+0000 7f9c234f5700 1 --2- 172.21.15.1:0/1738699610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c146af0 0x7f9c1c071650 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c0b62e0 tx=0x7f9c1400b330 comp rx=0 tx=0).stop 2024-03-18T19:06:05.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.673+0000 7f9c234f5700 1 -- 172.21.15.1:0/1738699610 shutdown_connections 2024-03-18T19:06:05.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.673+0000 7f9c234f5700 1 --2- 172.21.15.1:0/1738699610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c146af0 0x7f9c1c071650 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.673+0000 7f9c234f5700 1 --2- 172.21.15.1:0/1738699610 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c1c146190 0x7f9c1c06a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.673+0000 7f9c234f5700 1 --2- 172.21.15.1:0/1738699610 >> v2:172.21.15.165:3300/0 conn(0x7f9c1c1465a0 0x7f9c1c06ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.673+0000 7f9c234f5700 1 -- 172.21.15.1:0/1738699610 >> 172.21.15.1:0/1738699610 conn(0x7f9c1c0b1d20 msgr2=0x7f9c1c0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:05.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.674+0000 7f9c234f5700 1 -- 172.21.15.1:0/1738699610 shutdown_connections 2024-03-18T19:06:05.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.674+0000 7f9c234f5700 1 -- 172.21.15.1:0/1738699610 wait complete. 2024-03-18T19:06:05.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.675+0000 7f9c234f5700 1 Processor -- start 2024-03-18T19:06:05.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.675+0000 7f9c234f5700 1 -- start start 2024-03-18T19:06:05.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c234f5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c1c1465a0 0x7f9c1c138690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:05.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c234f5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c146af0 0x7f9c1c0c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:05.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c234f5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c1c1447f0 0x7f9c1c0c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:05.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c234f5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c0c67b0 con 0x7f9c1c146af0 2024-03-18T19:06:05.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c234f5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c0c5ba0 con 0x7f9c1c1465a0 2024-03-18T19:06:05.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c234f5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c0c5e80 con 0x7f9c1c1447f0 2024-03-18T19:06:05.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c20a90700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c146af0 0x7f9c1c0c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:05.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c20a90700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c146af0 0x7f9c1c0c5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40826/0 (socket says 172.21.15.1:40826) 2024-03-18T19:06:05.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c20a90700 1 -- 172.21.15.1:0/3179986676 learned_addr learned my addr 172.21.15.1:0/3179986676 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:05.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c21291700 1 --2- 172.21.15.1:0/3179986676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c1c1465a0 0x7f9c1c138690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:05.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c21a92700 1 --2- 172.21.15.1:0/3179986676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c1c1447f0 0x7f9c1c0c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:05.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c20a90700 1 -- 172.21.15.1:0/3179986676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c1c1447f0 msgr2=0x7f9c1c0c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:05.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c20a90700 1 --2- 172.21.15.1:0/3179986676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c1c1447f0 0x7f9c1c0c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c20a90700 1 -- 172.21.15.1:0/3179986676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c1c1465a0 msgr2=0x7f9c1c138690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c20a90700 1 --2- 172.21.15.1:0/3179986676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c1c1465a0 0x7f9c1c138690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c20a90700 1 -- 172.21.15.1:0/3179986676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c14009b30 con 0x7f9c1c146af0 2024-03-18T19:06:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c21a92700 1 --2- 172.21.15.1:0/3179986676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c1c1447f0 0x7f9c1c0c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c20a90700 1 --2- 172.21.15.1:0/3179986676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c146af0 0x7f9c1c0c5230 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f9c1800e810 tx=0x7f9c1800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c11ffb700 1 -- 172.21.15.1:0/3179986676 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c18013610 con 0x7f9c1c146af0 2024-03-18T19:06:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c11ffb700 1 -- 172.21.15.1:0/3179986676 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c18013770 con 0x7f9c1c146af0 2024-03-18T19:06:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c11ffb700 1 -- 172.21.15.1:0/3179986676 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c18013b20 con 0x7f9c1c146af0 2024-03-18T19:06:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.676+0000 7f9c234f5700 1 -- 172.21.15.1:0/3179986676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c1c0c6160 con 0x7f9c1c146af0 2024-03-18T19:06:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.677+0000 7f9c234f5700 1 -- 172.21.15.1:0/3179986676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c1c138bd0 con 0x7f9c1c146af0 2024-03-18T19:06:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.677+0000 7f9c234f5700 1 -- 172.21.15.1:0/3179986676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c1c073350 con 0x7f9c1c146af0 2024-03-18T19:06:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.678+0000 7f9c11ffb700 1 -- 172.21.15.1:0/3179986676 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9c18013c80 con 0x7f9c1c146af0 2024-03-18T19:06:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.678+0000 7f9c11ffb700 1 --2- 172.21.15.1:0/3179986676 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9c0006adb0 0x7f9c0006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.678+0000 7f9c11ffb700 1 -- 172.21.15.1:0/3179986676 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9c18089310 con 0x7f9c1c146af0 2024-03-18T19:06:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.678+0000 7f9c21291700 1 --2- 172.21.15.1:0/3179986676 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9c0006adb0 0x7f9c0006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.679+0000 7f9c21291700 1 --2- 172.21.15.1:0/3179986676 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9c0006adb0 0x7f9c0006d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c00c900 tx=0x7f9c0c008040 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:05.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.683+0000 7f9c11ffb700 1 -- 172.21.15.1:0/3179986676 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9c180541a0 con 0x7f9c1c146af0 2024-03-18T19:06:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.887+0000 7f9c234f5700 1 -- 172.21.15.1:0/3179986676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c1c0676a0 con 0x7f9c1c146af0 2024-03-18T19:06:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.888+0000 7f9c11ffb700 1 -- 172.21.15.1:0/3179986676 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+16285 (secure 0 0 0) 0x7f9c180577c0 con 0x7f9c1c146af0 2024-03-18T19:06:05.884 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:05.884 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":102,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:06:05.147292+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:06:02.855989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-18T19:06:04.121199+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:06:05.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.890+0000 7f9c234f5700 1 -- 172.21.15.1:0/3179986676 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9c0006adb0 msgr2=0x7f9c0006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.890+0000 7f9c234f5700 1 --2- 172.21.15.1:0/3179986676 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9c0006adb0 0x7f9c0006d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c00c900 tx=0x7f9c0c008040 comp rx=0 tx=0).stop 2024-03-18T19:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.890+0000 7f9c234f5700 1 -- 172.21.15.1:0/3179986676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c146af0 msgr2=0x7f9c1c0c5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.890+0000 7f9c234f5700 1 --2- 172.21.15.1:0/3179986676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c146af0 0x7f9c1c0c5230 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f9c1800e810 tx=0x7f9c1800ebd0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.890+0000 7f9c234f5700 1 -- 172.21.15.1:0/3179986676 shutdown_connections 2024-03-18T19:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.890+0000 7f9c234f5700 1 --2- 172.21.15.1:0/3179986676 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9c0006adb0 0x7f9c0006d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.890+0000 7f9c234f5700 1 --2- 172.21.15.1:0/3179986676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c1c1447f0 0x7f9c1c0c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.890+0000 7f9c234f5700 1 --2- 172.21.15.1:0/3179986676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c1c1465a0 0x7f9c1c138690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.890+0000 7f9c234f5700 1 --2- 172.21.15.1:0/3179986676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c146af0 0x7f9c1c0c5230 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.890+0000 7f9c234f5700 1 -- 172.21.15.1:0/3179986676 >> 172.21.15.1:0/3179986676 conn(0x7f9c1c0b1d20 msgr2=0x7f9c1c0aef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.890+0000 7f9c234f5700 1 -- 172.21.15.1:0/3179986676 shutdown_connections 2024-03-18T19:06:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:05.890+0000 7f9c234f5700 1 -- 172.21.15.1:0/3179986676 wait complete. 2024-03-18T19:06:05.898 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T19:06:06.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.031+0000 7fc2c2aaf700 1 Processor -- start 2024-03-18T19:06:06.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.031+0000 7fc2c2aaf700 1 -- start start 2024-03-18T19:06:06.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.031+0000 7fc2c2aaf700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc2bc071490 0x7fc2bc15b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:06.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.031+0000 7fc2c2aaf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc2bc15a380 0x7fc2bc071870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:06.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.032+0000 7fc2c2aaf700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2bc15b6f0 con 0x7fc2bc0724e0 2024-03-18T19:06:06.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.032+0000 7fc2c2aaf700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2bc0cb510 con 0x7fc2bc071490 2024-03-18T19:06:06.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.032+0000 7fc2c2aaf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc071db0 con 0x7fc2bc15a380 2024-03-18T19:06:06.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.032+0000 7fc2c084b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc2bc071490 0x7fc2bc15b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:06.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.032+0000 7fc2c084b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc2bc071490 0x7fc2bc15b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40840/0 (socket says 172.21.15.1:40840) 2024-03-18T19:06:06.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.032+0000 7fc2c084b700 1 -- 172.21.15.1:0/1018878603 learned_addr learned my addr 172.21.15.1:0/1018878603 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:06.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.032+0000 7fc2bbfff700 1 --2- 172.21.15.1:0/1018878603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc2bc15a380 0x7fc2bc071870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:06.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.032+0000 7fc2c084b700 1 -- 172.21.15.1:0/1018878603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc2bc15a380 msgr2=0x7fc2bc071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:06.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.032+0000 7fc2c084b700 1 --2- 172.21.15.1:0/1018878603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc2bc15a380 0x7fc2bc071870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:06.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.032+0000 7fc2c084b700 1 -- 172.21.15.1:0/1018878603 >> v1:172.21.15.165:6789/0 conn(0x7fc2bc0724e0 legacy=0x7fc2bc15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.032+0000 7fc2c084b700 1 -- 172.21.15.1:0/1018878603 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc0ca900 con 0x7fc2bc071490 2024-03-18T19:06:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.032+0000 7fc2bbfff700 1 --2- 172.21.15.1:0/1018878603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc2bc15a380 0x7fc2bc071870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.032+0000 7fc2c084b700 1 --2- 172.21.15.1:0/1018878603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2bc071490 0x7fc2bc15b1b0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fc2b00098f0 tx=0x7fc2b0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c08c329f671367 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.032+0000 7fc2bb7fe700 1 -- 172.21.15.1:0/1018878603 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0004230 con 0x7fc2bc071490 2024-03-18T19:06:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.032+0000 7fc2bb7fe700 1 -- 172.21.15.1:0/1018878603 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b0004390 con 0x7fc2bc071490 2024-03-18T19:06:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.032+0000 7fc2bb7fe700 1 -- 172.21.15.1:0/1018878603 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b00036a0 con 0x7fc2bc071490 2024-03-18T19:06:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.033+0000 7fc2c2aaf700 1 -- 172.21.15.1:0/1018878603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2bc071490 msgr2=0x7fc2bc15b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.033+0000 7fc2c2aaf700 1 --2- 172.21.15.1:0/1018878603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2bc071490 0x7fc2bc15b1b0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fc2b00098f0 tx=0x7fc2b0009c00 comp rx=0 tx=0).stop 2024-03-18T19:06:06.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.033+0000 7fc2c2aaf700 1 -- 172.21.15.1:0/1018878603 shutdown_connections 2024-03-18T19:06:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.033+0000 7fc2c2aaf700 1 --2- 172.21.15.1:0/1018878603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2bc071490 0x7fc2bc15b1b0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.033+0000 7fc2c2aaf700 1 --2- 172.21.15.1:0/1018878603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc2bc15a380 0x7fc2bc071870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.033+0000 7fc2c2aaf700 1 -- 172.21.15.1:0/1018878603 >> 172.21.15.1:0/1018878603 conn(0x7fc2bc0b1aa0 msgr2=0x7fc2bc0b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.033+0000 7fc2c2aaf700 1 -- 172.21.15.1:0/1018878603 shutdown_connections 2024-03-18T19:06:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.033+0000 7fc2c2aaf700 1 -- 172.21.15.1:0/1018878603 wait complete. 2024-03-18T19:06:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.034+0000 7fc2c2aaf700 1 Processor -- start 2024-03-18T19:06:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.034+0000 7fc2c2aaf700 1 -- start start 2024-03-18T19:06:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.035+0000 7fc2c2aaf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2bc071490 0x7fc2bc0a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:06.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.035+0000 7fc2c2aaf700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2bc0724e0 0x7fc2bc09ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.035+0000 7fc2c2aaf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc2bc15a380 0x7fc2bc15be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.035+0000 7fc2c2aaf700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc0aee30 con 0x7fc2bc071490 2024-03-18T19:06:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.035+0000 7fc2c2aaf700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc1f3590 con 0x7fc2bc0724e0 2024-03-18T19:06:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.035+0000 7fc2c2aaf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc09e880 con 0x7fc2bc15a380 2024-03-18T19:06:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.035+0000 7fc2c084b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2bc071490 0x7fc2bc0a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.035+0000 7fc2c084b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2bc071490 0x7fc2bc0a0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40842/0 (socket says 172.21.15.1:40842) 2024-03-18T19:06:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.035+0000 7fc2c084b700 1 -- 172.21.15.1:0/3406402356 learned_addr learned my addr 172.21.15.1:0/3406402356 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.035+0000 7fc2bbfff700 1 --2- 172.21.15.1:0/3406402356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2bc0724e0 0x7fc2bc09ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.035+0000 7fc2c104c700 1 --2- 172.21.15.1:0/3406402356 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc2bc15a380 0x7fc2bc15be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.035+0000 7fc2c084b700 1 -- 172.21.15.1:0/3406402356 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc2bc15a380 msgr2=0x7fc2bc15be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:06.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.035+0000 7fc2c084b700 1 --2- 172.21.15.1:0/3406402356 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc2bc15a380 0x7fc2bc15be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:06.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.035+0000 7fc2c084b700 1 -- 172.21.15.1:0/3406402356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2bc0724e0 msgr2=0x7fc2bc09ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:06.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.035+0000 7fc2c084b700 1 --2- 172.21.15.1:0/3406402356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2bc0724e0 0x7fc2bc09ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:06.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.035+0000 7fc2c084b700 1 -- 172.21.15.1:0/3406402356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b0009550 con 0x7fc2bc071490 2024-03-18T19:06:06.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.035+0000 7fc2bbfff700 1 --2- 172.21.15.1:0/3406402356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2bc0724e0 0x7fc2bc09ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:06.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.036+0000 7fc2c084b700 1 --2- 172.21.15.1:0/3406402356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2bc071490 0x7fc2bc0a0e60 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc067530 tx=0x7fc2b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:06.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.036+0000 7fc2b97fa700 1 -- 172.21.15.1:0/3406402356 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b000b860 con 0x7fc2bc071490 2024-03-18T19:06:06.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.036+0000 7fc2b97fa700 1 -- 172.21.15.1:0/3406402356 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b000b9c0 con 0x7fc2bc071490 2024-03-18T19:06:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.036+0000 7fc2b97fa700 1 -- 172.21.15.1:0/3406402356 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b000bd70 con 0x7fc2bc071490 2024-03-18T19:06:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.036+0000 7fc2c2aaf700 1 -- 172.21.15.1:0/3406402356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2bc1f37e0 con 0x7fc2bc071490 2024-03-18T19:06:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.036+0000 7fc2c2aaf700 1 -- 172.21.15.1:0/3406402356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2bc1f3da0 con 0x7fc2bc071490 2024-03-18T19:06:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.037+0000 7fc2c2aaf700 1 -- 172.21.15.1:0/3406402356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2bc0676a0 con 0x7fc2bc071490 2024-03-18T19:06:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.037+0000 7fc2b97fa700 1 -- 172.21.15.1:0/3406402356 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc2b001b440 con 0x7fc2bc071490 2024-03-18T19:06:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.038+0000 7fc2b97fa700 1 --2- 172.21.15.1:0/3406402356 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc2a006adb0 0x7fc2a006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.038+0000 7fc2b97fa700 1 -- 172.21.15.1:0/3406402356 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc2b008ad80 con 0x7fc2bc071490 2024-03-18T19:06:06.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.038+0000 7fc2bbfff700 1 --2- 172.21.15.1:0/3406402356 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc2a006adb0 0x7fc2a006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.038+0000 7fc2bbfff700 1 --2- 172.21.15.1:0/3406402356 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc2a006adb0 0x7fc2a006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc0a13a0 tx=0x7fc2a8008040 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:06.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.042+0000 7fc2b97fa700 1 -- 172.21.15.1:0/3406402356 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc2b0055da0 con 0x7fc2bc071490 2024-03-18T19:06:06.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:06.293+0000 7fc2c2aaf700 1 -- 172.21.15.1:0/3406402356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc2bc15b6f0 con 0x7fc2bc071490 2024-03-18T19:06:07.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.161+0000 7fc2b97fa700 1 -- 172.21.15.1:0/3406402356 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v41) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc2bc15b6f0 con 0x7fc2bc071490 2024-03-18T19:06:07.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.164+0000 7fc2c2aaf700 1 -- 172.21.15.1:0/3406402356 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc2a006adb0 msgr2=0x7fc2a006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:07.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.164+0000 7fc2c2aaf700 1 --2- 172.21.15.1:0/3406402356 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc2a006adb0 0x7fc2a006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc0a13a0 tx=0x7fc2a8008040 comp rx=0 tx=0).stop 2024-03-18T19:06:07.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.164+0000 7fc2c2aaf700 1 -- 172.21.15.1:0/3406402356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2bc071490 msgr2=0x7fc2bc0a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:07.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.164+0000 7fc2c2aaf700 1 --2- 172.21.15.1:0/3406402356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2bc071490 0x7fc2bc0a0e60 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc067530 tx=0x7fc2b0004750 comp rx=0 tx=0).stop 2024-03-18T19:06:07.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.164+0000 7fc2c2aaf700 1 -- 172.21.15.1:0/3406402356 shutdown_connections 2024-03-18T19:06:07.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.164+0000 7fc2c2aaf700 1 --2- 172.21.15.1:0/3406402356 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc2a006adb0 0x7fc2a006d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:07.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.164+0000 7fc2c2aaf700 1 --2- 172.21.15.1:0/3406402356 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc2bc15a380 0x7fc2bc15be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:07.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.164+0000 7fc2c2aaf700 1 --2- 172.21.15.1:0/3406402356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2bc071490 0x7fc2bc0a0e60 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:07.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.164+0000 7fc2c2aaf700 1 --2- 172.21.15.1:0/3406402356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2bc0724e0 0x7fc2bc09ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:07.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.164+0000 7fc2c2aaf700 1 -- 172.21.15.1:0/3406402356 >> 172.21.15.1:0/3406402356 conn(0x7fc2bc0b1aa0 msgr2=0x7fc2bc06f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:07.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.165+0000 7fc2c2aaf700 1 -- 172.21.15.1:0/3406402356 shutdown_connections 2024-03-18T19:06:07.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.165+0000 7fc2c2aaf700 1 -- 172.21.15.1:0/3406402356 wait complete. 2024-03-18T19:06:07.175 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-18T19:06:07.175 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T19:06:07.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.315+0000 7fc024e22700 1 Processor -- start 2024-03-18T19:06:07.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.315+0000 7fc024e22700 1 -- start start 2024-03-18T19:06:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.316+0000 7fc024e22700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0200cd870 0x7fc0200cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.316+0000 7fc024e22700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc0200cbce0 0x7fc0200cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.316+0000 7fc024e22700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0200d0df0 con 0x7fc0200b1e20 2024-03-18T19:06:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.316+0000 7fc024e22700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0200d0550 con 0x7fc0200cbce0 2024-03-18T19:06:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.316+0000 7fc024e22700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0200d0830 con 0x7fc0200cd870 2024-03-18T19:06:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.316+0000 7fc01dd9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc0200b1e20 0x7fc0200cdc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50952/0 (socket says 172.21.15.1:50952) 2024-03-18T19:06:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.316+0000 7fc01dd9b700 1 -- 172.21.15.1:0/1804130632 learned_addr learned my addr 172.21.15.1:0/1804130632 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.316+0000 7fc01ed9d700 1 --2- 172.21.15.1:0/1804130632 >> v2:172.21.15.165:3300/0 conn(0x7fc0200cbce0 0x7fc0200cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.316+0000 7fc01e59c700 1 --2- 172.21.15.1:0/1804130632 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0200cd870 0x7fc0200cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.316+0000 7fc01d59a700 1 -- 172.21.15.1:0/1804130632 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371128025 0 0) 0x7fc0200d0df0 con 0x7fc0200b1e20 2024-03-18T19:06:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.316+0000 7fc01d59a700 1 -- 172.21.15.1:0/1804130632 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0080035b0 con 0x7fc0200b1e20 2024-03-18T19:06:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.316+0000 7fc01d59a700 1 -- 172.21.15.1:0/1804130632 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2669015568 0 0) 0x7fc0080035b0 con 0x7fc0200b1e20 2024-03-18T19:06:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.316+0000 7fc01d59a700 1 -- 172.21.15.1:0/1804130632 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0200cd870 msgr2=0x7fc0200cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.316+0000 7fc01d59a700 1 --2- 172.21.15.1:0/1804130632 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0200cd870 0x7fc0200cff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.316+0000 7fc01d59a700 1 -- 172.21.15.1:0/1804130632 >> v2:172.21.15.165:3300/0 conn(0x7fc0200cbce0 msgr2=0x7fc0200cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.316+0000 7fc01d59a700 1 --2- 172.21.15.1:0/1804130632 >> v2:172.21.15.165:3300/0 conn(0x7fc0200cbce0 0x7fc0200cc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.316+0000 7fc01d59a700 1 -- 172.21.15.1:0/1804130632 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0200d0b10 con 0x7fc0200b1e20 2024-03-18T19:06:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01d59a700 1 -- 172.21.15.1:0/1804130632 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fc014001670 con 0x7fc0200b1e20 2024-03-18T19:06:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01d59a700 1 -- 172.21.15.1:0/1804130632 >> v1:172.21.15.1:6789/0 conn(0x7fc0200b1e20 legacy=0x7fc0200cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01d59a700 1 --2- 172.21.15.1:0/1804130632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc008003a00 0x7fc008003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01d59a700 1 --2- 172.21.15.1:0/1804130632 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0200cd870 0x7fc008004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01d59a700 1 --2- 172.21.15.1:0/1804130632 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc008004d90 0x7fc008005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01d59a700 1 -- 172.21.15.1:0/1804130632 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0200d0550 con 0x7fc008003a00 2024-03-18T19:06:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01d59a700 1 -- 172.21.15.1:0/1804130632 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0200d0830 con 0x7fc008004d90 2024-03-18T19:06:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01d59a700 1 -- 172.21.15.1:0/1804130632 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00800ba10 con 0x7fc0200cd870 2024-03-18T19:06:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01e59c700 1 --2- 172.21.15.1:0/1804130632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc008003a00 0x7fc008003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01dd9b700 1 --2- 172.21.15.1:0/1804130632 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0200cd870 0x7fc008004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01ed9d700 1 --2- 172.21.15.1:0/1804130632 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc008004d90 0x7fc008005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01e59c700 1 -- 172.21.15.1:0/1804130632 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0200cd870 msgr2=0x7fc008004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01e59c700 1 --2- 172.21.15.1:0/1804130632 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0200cd870 0x7fc008004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01e59c700 1 -- 172.21.15.1:0/1804130632 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc008004d90 msgr2=0x7fc008005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01e59c700 1 --2- 172.21.15.1:0/1804130632 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc008004d90 0x7fc008005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01e59c700 1 -- 172.21.15.1:0/1804130632 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0080035b0 con 0x7fc008003a00 2024-03-18T19:06:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01ed9d700 1 --2- 172.21.15.1:0/1804130632 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc008004d90 0x7fc008005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01e59c700 1 --2- 172.21.15.1:0/1804130632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc008003a00 0x7fc008003e20 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fc010009b60 tx=0x7fc008005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de32af112a4003f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01d59a700 1 -- 172.21.15.1:0/1804130632 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01000e6c0 con 0x7fc008003a00 2024-03-18T19:06:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01d59a700 1 -- 172.21.15.1:0/1804130632 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01000e820 con 0x7fc008003a00 2024-03-18T19:06:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.317+0000 7fc01d59a700 1 -- 172.21.15.1:0/1804130632 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01000eb90 con 0x7fc008003a00 2024-03-18T19:06:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.318+0000 7fc024e22700 1 -- 172.21.15.1:0/1804130632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc008003a00 msgr2=0x7fc008003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.318+0000 7fc024e22700 1 --2- 172.21.15.1:0/1804130632 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc008003a00 0x7fc008003e20 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fc010009b60 tx=0x7fc008005880 comp rx=0 tx=0).stop 2024-03-18T19:06:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.318+0000 7fc01ed9d700 1 -- 172.21.15.1:0/1804130632 reap_dead start 2024-03-18T19:06:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.318+0000 7fc024e22700 1 -- 172.21.15.1:0/1804130632 shutdown_connections 2024-03-18T19:06:07.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.318+0000 7fc024e22700 1 -- 172.21.15.1:0/1804130632 >> 172.21.15.1:0/1804130632 conn(0x7fc0200d0ff0 msgr2=0x7fc0200d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:07.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.318+0000 7fc024e22700 1 -- 172.21.15.1:0/1804130632 shutdown_connections 2024-03-18T19:06:07.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.318+0000 7fc024e22700 1 -- 172.21.15.1:0/1804130632 wait complete. 2024-03-18T19:06:07.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.319+0000 7fc024e22700 1 Processor -- start 2024-03-18T19:06:07.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.319+0000 7fc024e22700 1 -- start start 2024-03-18T19:06:07.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.320+0000 7fc024e22700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc008003a00 0x7fc0200ca960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:07.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.320+0000 7fc024e22700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc020155a40 0x7fc020158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:07.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.320+0000 7fc024e22700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0200c9330 0x7fc0201567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:07.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.320+0000 7fc024e22700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0200c8b70 con 0x7fc0200c9330 2024-03-18T19:06:07.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.320+0000 7fc024e22700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0200b14b0 con 0x7fc008003a00 2024-03-18T19:06:07.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.320+0000 7fc024e22700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0200b1790 con 0x7fc020155a40 2024-03-18T19:06:07.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.320+0000 7fc01ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0200c9330 0x7fc0201567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:07.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.320+0000 7fc01ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0200c9330 0x7fc0201567b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40866/0 (socket says 172.21.15.1:40866) 2024-03-18T19:06:07.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.320+0000 7fc01ed9d700 1 -- 172.21.15.1:0/1169750403 learned_addr learned my addr 172.21.15.1:0/1169750403 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:07.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.320+0000 7fc01e59c700 1 --2- 172.21.15.1:0/1169750403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc008003a00 0x7fc0200ca960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.320+0000 7fc01dd9b700 1 --2- 172.21.15.1:0/1169750403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc020155a40 0x7fc020158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.320+0000 7fc01ed9d700 1 -- 172.21.15.1:0/1169750403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc020155a40 msgr2=0x7fc020158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.320+0000 7fc01ed9d700 1 --2- 172.21.15.1:0/1169750403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc020155a40 0x7fc020158c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.320+0000 7fc01ed9d700 1 -- 172.21.15.1:0/1169750403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc008003a00 msgr2=0x7fc0200ca960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.320+0000 7fc01ed9d700 1 --2- 172.21.15.1:0/1169750403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc008003a00 0x7fc0200ca960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.320+0000 7fc01ed9d700 1 -- 172.21.15.1:0/1169750403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0200c8e10 con 0x7fc0200c9330 2024-03-18T19:06:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.321+0000 7fc01ed9d700 1 --2- 172.21.15.1:0/1169750403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0200c9330 0x7fc0201567b0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fc020071890 tx=0x7fc0180104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.321+0000 7fc00effd700 1 -- 172.21.15.1:0/1169750403 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01801b070 con 0x7fc0200c9330 2024-03-18T19:06:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.321+0000 7fc00effd700 1 -- 172.21.15.1:0/1169750403 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01800f040 con 0x7fc0200c9330 2024-03-18T19:06:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.321+0000 7fc00effd700 1 -- 172.21.15.1:0/1169750403 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01800e050 con 0x7fc0200c9330 2024-03-18T19:06:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.321+0000 7fc024e22700 1 -- 172.21.15.1:0/1169750403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0200b1a70 con 0x7fc0200c9330 2024-03-18T19:06:07.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.321+0000 7fc024e22700 1 -- 172.21.15.1:0/1169750403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc020159160 con 0x7fc0200c9330 2024-03-18T19:06:07.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.322+0000 7fc024e22700 1 -- 172.21.15.1:0/1169750403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc020159350 con 0x7fc0200c9330 2024-03-18T19:06:07.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.322+0000 7fc00effd700 1 -- 172.21.15.1:0/1169750403 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc018010d00 con 0x7fc0200c9330 2024-03-18T19:06:07.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.323+0000 7fc00effd700 1 --2- 172.21.15.1:0/1169750403 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc00006adb0 0x7fc00006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:07.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.323+0000 7fc00effd700 1 -- 172.21.15.1:0/1169750403 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc018089680 con 0x7fc0200c9330 2024-03-18T19:06:07.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.323+0000 7fc01e59c700 1 --2- 172.21.15.1:0/1169750403 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc00006adb0 0x7fc00006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:07.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.323+0000 7fc01e59c700 1 --2- 172.21.15.1:0/1169750403 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc00006adb0 0x7fc00006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc0100090b0 tx=0x7fc01001f040 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:07.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.327+0000 7fc00effd700 1 -- 172.21.15.1:0/1169750403 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc018054590 con 0x7fc0200c9330 2024-03-18T19:06:07.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:07.576+0000 7fc024e22700 1 -- 172.21.15.1:0/1169750403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc0200c9000 con 0x7fc0200c9330 2024-03-18T19:06:08.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.167+0000 7fc00effd700 1 -- 172.21.15.1:0/1169750403 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc018057bb0 con 0x7fc0200c9330 2024-03-18T19:06:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.170+0000 7fc024e22700 1 -- 172.21.15.1:0/1169750403 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc00006adb0 msgr2=0x7fc00006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.170+0000 7fc024e22700 1 --2- 172.21.15.1:0/1169750403 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc00006adb0 0x7fc00006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc0100090b0 tx=0x7fc01001f040 comp rx=0 tx=0).stop 2024-03-18T19:06:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.170+0000 7fc024e22700 1 -- 172.21.15.1:0/1169750403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0200c9330 msgr2=0x7fc0201567b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.170+0000 7fc024e22700 1 --2- 172.21.15.1:0/1169750403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0200c9330 0x7fc0201567b0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fc020071890 tx=0x7fc0180104a0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.170+0000 7fc024e22700 1 -- 172.21.15.1:0/1169750403 shutdown_connections 2024-03-18T19:06:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.170+0000 7fc024e22700 1 --2- 172.21.15.1:0/1169750403 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc00006adb0 0x7fc00006d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.170+0000 7fc024e22700 1 --2- 172.21.15.1:0/1169750403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0200c9330 0x7fc0201567b0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.170+0000 7fc024e22700 1 --2- 172.21.15.1:0/1169750403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc008003a00 0x7fc0200ca960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.171+0000 7fc024e22700 1 --2- 172.21.15.1:0/1169750403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc020155a40 0x7fc020158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.171+0000 7fc024e22700 1 -- 172.21.15.1:0/1169750403 >> 172.21.15.1:0/1169750403 conn(0x7fc0200d0ff0 msgr2=0x7fc0200c4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:08.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.171+0000 7fc024e22700 1 -- 172.21.15.1:0/1169750403 shutdown_connections 2024-03-18T19:06:08.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.171+0000 7fc024e22700 1 -- 172.21.15.1:0/1169750403 wait complete. 2024-03-18T19:06:08.181 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T19:06:08.182 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:06:08.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.318+0000 7fb6bd60c700 1 Processor -- start 2024-03-18T19:06:08.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.318+0000 7fb6bd60c700 1 -- start start 2024-03-18T19:06:08.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.319+0000 7fb6bd60c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb6b80b6b60 0x7fb6b80cc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:08.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.319+0000 7fb6bd60c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb6b80cee90 0x7fb6b80cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:08.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.319+0000 7fb6bd60c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b80cd0d0 con 0x7fb6b80cfbd0 2024-03-18T19:06:08.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.319+0000 7fb6bd60c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6b80d0040 con 0x7fb6b80cee90 2024-03-18T19:06:08.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.319+0000 7fb6bd60c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6b80b3f00 con 0x7fb6b80b6b60 2024-03-18T19:06:08.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.319+0000 7fb6b67fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb6b80cee90 0x7fb6b80cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:08.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.319+0000 7fb6b67fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb6b80cee90 0x7fb6b80cf690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40868/0 (socket says 172.21.15.1:40868) 2024-03-18T19:06:08.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.319+0000 7fb6b67fc700 1 -- 172.21.15.1:0/2885866018 learned_addr learned my addr 172.21.15.1:0/2885866018 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:08.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.320+0000 7fb6b6ffd700 1 --2- 172.21.15.1:0/2885866018 >> v2:172.21.15.165:3300/0 conn(0x7fb6b80b6b60 0x7fb6b80cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:08.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.320+0000 7fb6b67fc700 1 -- 172.21.15.1:0/2885866018 >> v2:172.21.15.165:3300/0 conn(0x7fb6b80b6b60 msgr2=0x7fb6b80cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:08.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.320+0000 7fb6b67fc700 1 --2- 172.21.15.1:0/2885866018 >> v2:172.21.15.165:3300/0 conn(0x7fb6b80b6b60 0x7fb6b80cc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.320+0000 7fb6b67fc700 1 -- 172.21.15.1:0/2885866018 >> v1:172.21.15.165:6789/0 conn(0x7fb6b80cfbd0 legacy=0x7fb6b8099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:08.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.320+0000 7fb6b67fc700 1 -- 172.21.15.1:0/2885866018 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b80b41e0 con 0x7fb6b80cee90 2024-03-18T19:06:08.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.320+0000 7fb6b67fc700 1 --2- 172.21.15.1:0/2885866018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6b80cee90 0x7fb6b80cf690 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fb6b806e730 tx=0x7fb6a0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd0cf5aac7a0131 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:08.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.320+0000 7fb6b5ffb700 1 -- 172.21.15.1:0/2885866018 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a000b870 con 0x7fb6b80cee90 2024-03-18T19:06:08.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.320+0000 7fb6b5ffb700 1 -- 172.21.15.1:0/2885866018 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a000b9d0 con 0x7fb6b80cee90 2024-03-18T19:06:08.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.320+0000 7fb6b5ffb700 1 -- 172.21.15.1:0/2885866018 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a000bd60 con 0x7fb6b80cee90 2024-03-18T19:06:08.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.321+0000 7fb6bd60c700 1 -- 172.21.15.1:0/2885866018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6b80cee90 msgr2=0x7fb6b80cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:08.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.321+0000 7fb6bd60c700 1 --2- 172.21.15.1:0/2885866018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6b80cee90 0x7fb6b80cf690 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fb6b806e730 tx=0x7fb6a0009bd0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.321+0000 7fb6bd60c700 1 -- 172.21.15.1:0/2885866018 shutdown_connections 2024-03-18T19:06:08.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.321+0000 7fb6bd60c700 1 --2- 172.21.15.1:0/2885866018 >> v2:172.21.15.165:3300/0 conn(0x7fb6b80b6b60 0x7fb6b80cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.321+0000 7fb6bd60c700 1 --2- 172.21.15.1:0/2885866018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6b80cee90 0x7fb6b80cf690 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.321+0000 7fb6bd60c700 1 -- 172.21.15.1:0/2885866018 >> 172.21.15.1:0/2885866018 conn(0x7fb6b80b2ba0 msgr2=0x7fb6b80b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:08.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.321+0000 7fb6bd60c700 1 -- 172.21.15.1:0/2885866018 shutdown_connections 2024-03-18T19:06:08.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.321+0000 7fb6bd60c700 1 -- 172.21.15.1:0/2885866018 wait complete. 2024-03-18T19:06:08.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.322+0000 7fb6bd60c700 1 Processor -- start 2024-03-18T19:06:08.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.322+0000 7fb6bd60c700 1 -- start start 2024-03-18T19:06:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6bd60c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6b80b6b60 0x7fb6b81f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6bd60c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6b80cee90 0x7fb6b80cb040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6bd60c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6b80cfbd0 0x7fb6b80c6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6bd60c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b80ca0c0 con 0x7fb6b80cee90 2024-03-18T19:06:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6bd60c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b80c6280 con 0x7fb6b80b6b60 2024-03-18T19:06:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6bd60c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b80c91d0 con 0x7fb6b80cfbd0 2024-03-18T19:06:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6b67fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6b80cee90 0x7fb6b80cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6b67fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6b80cee90 0x7fb6b80cb040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40878/0 (socket says 172.21.15.1:40878) 2024-03-18T19:06:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6b67fc700 1 -- 172.21.15.1:0/1737218979 learned_addr learned my addr 172.21.15.1:0/1737218979 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6b6ffd700 1 --2- 172.21.15.1:0/1737218979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6b80b6b60 0x7fb6b81f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6b77fe700 1 --2- 172.21.15.1:0/1737218979 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6b80cfbd0 0x7fb6b80c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6b67fc700 1 -- 172.21.15.1:0/1737218979 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6b80cfbd0 msgr2=0x7fb6b80c6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6b67fc700 1 --2- 172.21.15.1:0/1737218979 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6b80cfbd0 0x7fb6b80c6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6b67fc700 1 -- 172.21.15.1:0/1737218979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6b80b6b60 msgr2=0x7fb6b81f1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6b67fc700 1 --2- 172.21.15.1:0/1737218979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6b80b6b60 0x7fb6b81f1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6b67fc700 1 -- 172.21.15.1:0/1737218979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a0009550 con 0x7fb6b80cee90 2024-03-18T19:06:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6b6ffd700 1 --2- 172.21.15.1:0/1737218979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6b80b6b60 0x7fb6b81f1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:08.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6b77fe700 1 --2- 172.21.15.1:0/1737218979 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6b80cfbd0 0x7fb6b80c6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:08.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6b67fc700 1 --2- 172.21.15.1:0/1737218979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6b80cee90 0x7fb6b80cb040 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0000c00 tx=0x7fb6a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:08.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6af7fe700 1 -- 172.21.15.1:0/1737218979 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a0004540 con 0x7fb6b80cee90 2024-03-18T19:06:08.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6af7fe700 1 -- 172.21.15.1:0/1737218979 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a00036a0 con 0x7fb6b80cee90 2024-03-18T19:06:08.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.323+0000 7fb6bd60c700 1 -- 172.21.15.1:0/1737218979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b80c94b0 con 0x7fb6b80cee90 2024-03-18T19:06:08.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.324+0000 7fb6af7fe700 1 -- 172.21.15.1:0/1737218979 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a0003a30 con 0x7fb6b80cee90 2024-03-18T19:06:08.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.324+0000 7fb6bd60c700 1 -- 172.21.15.1:0/1737218979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b80c9a70 con 0x7fb6b80cee90 2024-03-18T19:06:08.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.324+0000 7fb6bd60c700 1 -- 172.21.15.1:0/1737218979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6b80676a0 con 0x7fb6b80cee90 2024-03-18T19:06:08.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.325+0000 7fb6af7fe700 1 -- 172.21.15.1:0/1737218979 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb6a0021070 con 0x7fb6b80cee90 2024-03-18T19:06:08.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.325+0000 7fb6af7fe700 1 --2- 172.21.15.1:0/1737218979 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb69806adb0 0x7fb69806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:08.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.326+0000 7fb6b6ffd700 1 --2- 172.21.15.1:0/1737218979 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb69806adb0 0x7fb69806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:08.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.326+0000 7fb6af7fe700 1 -- 172.21.15.1:0/1737218979 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb6a008ad30 con 0x7fb6b80cee90 2024-03-18T19:06:08.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.326+0000 7fb6b6ffd700 1 --2- 172.21.15.1:0/1737218979 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb69806adb0 0x7fb69806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb6b81f41e0 tx=0x7fb6a8008040 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:08.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.330+0000 7fb6af7fe700 1 -- 172.21.15.1:0/1737218979 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb6a0055d50 con 0x7fb6b80cee90 2024-03-18T19:06:08.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.577+0000 7fb6bd60c700 1 -- 172.21.15.1:0/1737218979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6b815e770 con 0x7fb6b80cee90 2024-03-18T19:06:08.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.578+0000 7fb6af7fe700 1 -- 172.21.15.1:0/1737218979 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4736 (secure 0 0 0) 0x7fb6a0015020 con 0x7fb6b80cee90 2024-03-18T19:06:08.574 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:08.574 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3926877640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3926877640},{"type":"v1","addr":"172.21.15.165:6833","nonce":3926877640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/4293662065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":4293662065},{"type":"v1","addr":"172.21.15.165:6835","nonce":4293662065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2334043797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2334043797},{"type":"v1","addr":"172.21.15.1:6835","nonce":2334043797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:06:05.147601+0000","modified":"2024-03-18T19:06:08.160700+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2526016488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":2526016488},{"type":"v1","addr":"172.21.15.1:6837","nonce":2526016488}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-18T19:06:08.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.580+0000 7fb6bd60c700 1 -- 172.21.15.1:0/1737218979 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb69806adb0 msgr2=0x7fb69806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:08.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.580+0000 7fb6bd60c700 1 --2- 172.21.15.1:0/1737218979 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb69806adb0 0x7fb69806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb6b81f41e0 tx=0x7fb6a8008040 comp rx=0 tx=0).stop 2024-03-18T19:06:08.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.580+0000 7fb6bd60c700 1 -- 172.21.15.1:0/1737218979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6b80cee90 msgr2=0x7fb6b80cb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:08.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.580+0000 7fb6bd60c700 1 --2- 172.21.15.1:0/1737218979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6b80cee90 0x7fb6b80cb040 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0000c00 tx=0x7fb6a0004750 comp rx=0 tx=0).stop 2024-03-18T19:06:08.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.580+0000 7fb6bd60c700 1 -- 172.21.15.1:0/1737218979 shutdown_connections 2024-03-18T19:06:08.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.580+0000 7fb6bd60c700 1 --2- 172.21.15.1:0/1737218979 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb69806adb0 0x7fb69806d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.580+0000 7fb6bd60c700 1 --2- 172.21.15.1:0/1737218979 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6b80cfbd0 0x7fb6b80c6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.580+0000 7fb6bd60c700 1 --2- 172.21.15.1:0/1737218979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6b80b6b60 0x7fb6b81f1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.580+0000 7fb6bd60c700 1 --2- 172.21.15.1:0/1737218979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6b80cee90 0x7fb6b80cb040 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.580+0000 7fb6bd60c700 1 -- 172.21.15.1:0/1737218979 >> 172.21.15.1:0/1737218979 conn(0x7fb6b80b2ba0 msgr2=0x7fb6b80ce3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:08.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.580+0000 7fb6bd60c700 1 -- 172.21.15.1:0/1737218979 shutdown_connections 2024-03-18T19:06:08.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.580+0000 7fb6bd60c700 1 -- 172.21.15.1:0/1737218979 wait complete. 2024-03-18T19:06:08.577 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 42 2024-03-18T19:06:08.588 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:06:08.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.718+0000 7f4cdd256700 1 Processor -- start 2024-03-18T19:06:08.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.718+0000 7f4cdd256700 1 -- start start 2024-03-18T19:06:08.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cdd256700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4cd815a320 0x7f4cd80ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:08.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cdd256700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd80acd80 con 0x7f4cd80cea30 2024-03-18T19:06:08.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cdd256700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd80cf220 con 0x7f4cd8071d10 2024-03-18T19:06:08.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cdd256700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd80cef40 con 0x7f4cd815a320 2024-03-18T19:06:08.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cd659c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4cd80cea30 0x7f4cd8071600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50956/0 (socket says 172.21.15.1:50956) 2024-03-18T19:06:08.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cd659c700 1 -- 172.21.15.1:0/730552943 learned_addr learned my addr 172.21.15.1:0/730552943 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:08.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cd6d9d700 1 --2- 172.21.15.1:0/730552943 >> v2:172.21.15.165:3300/0 conn(0x7f4cd815a320 0x7f4cd80ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:08.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cd5d9b700 1 -- 172.21.15.1:0/730552943 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670208304 0 0) 0x7f4cd80acd80 con 0x7f4cd80cea30 2024-03-18T19:06:08.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cd5d9b700 1 -- 172.21.15.1:0/730552943 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cc00035b0 con 0x7f4cd80cea30 2024-03-18T19:06:08.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cd5d9b700 1 -- 172.21.15.1:0/730552943 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 895181446 0 0) 0x7f4cc00035b0 con 0x7f4cd80cea30 2024-03-18T19:06:08.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cd5d9b700 1 -- 172.21.15.1:0/730552943 >> v2:172.21.15.165:3300/0 conn(0x7f4cd815a320 msgr2=0x7f4cd80ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:08.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cd5d9b700 1 --2- 172.21.15.1:0/730552943 >> v2:172.21.15.165:3300/0 conn(0x7f4cd815a320 0x7f4cd80ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cd5d9b700 1 -- 172.21.15.1:0/730552943 >> v1:172.21.15.165:6789/0 conn(0x7f4cd8071d10 legacy=0x7f4cd806e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:08.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cd5d9b700 1 -- 172.21.15.1:0/730552943 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd806d7e0 con 0x7f4cd80cea30 2024-03-18T19:06:08.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cd5d9b700 1 -- 172.21.15.1:0/730552943 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f4cc8001670 con 0x7f4cd80cea30 2024-03-18T19:06:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cd5d9b700 1 -- 172.21.15.1:0/730552943 >> v1:172.21.15.1:6789/0 conn(0x7f4cd80cea30 legacy=0x7f4cd8071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cd5d9b700 1 --2- 172.21.15.1:0/730552943 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cc0003a00 0x7f4cc0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cd5d9b700 1 --2- 172.21.15.1:0/730552943 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cc0004400 0x7f4cc00048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cd5d9b700 1 --2- 172.21.15.1:0/730552943 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4cc00050d0 0x7f4cc000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cd5d9b700 1 -- 172.21.15.1:0/730552943 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd80cef40 con 0x7f4cc0003a00 2024-03-18T19:06:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cd5d9b700 1 -- 172.21.15.1:0/730552943 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc000c030 con 0x7f4cc0004400 2024-03-18T19:06:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.719+0000 7f4cd5d9b700 1 -- 172.21.15.1:0/730552943 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc000c1f0 con 0x7f4cc00050d0 2024-03-18T19:06:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.720+0000 7f4cd6d9d700 1 --2- 172.21.15.1:0/730552943 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cc0003a00 0x7f4cc0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.720+0000 7f4cd659c700 1 --2- 172.21.15.1:0/730552943 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cc0004400 0x7f4cc00048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.720+0000 7f4cd759e700 1 --2- 172.21.15.1:0/730552943 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4cc00050d0 0x7f4cc000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.720+0000 7f4cd6d9d700 1 -- 172.21.15.1:0/730552943 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4cc00050d0 msgr2=0x7f4cc000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.720+0000 7f4cd6d9d700 1 --2- 172.21.15.1:0/730552943 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4cc00050d0 0x7f4cc000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.720+0000 7f4cd6d9d700 1 -- 172.21.15.1:0/730552943 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cc0004400 msgr2=0x7f4cc00048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:08.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.720+0000 7f4cd6d9d700 1 --2- 172.21.15.1:0/730552943 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cc0004400 0x7f4cc00048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.720+0000 7f4cd6d9d700 1 -- 172.21.15.1:0/730552943 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc00035b0 con 0x7f4cc0003a00 2024-03-18T19:06:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.720+0000 7f4cd759e700 1 --2- 172.21.15.1:0/730552943 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4cc00050d0 0x7f4cc000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.720+0000 7f4cd759e700 1 -- 172.21.15.1:0/730552943 reap_dead start 2024-03-18T19:06:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.720+0000 7f4cd6d9d700 1 --2- 172.21.15.1:0/730552943 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cc0003a00 0x7f4cc0003de0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc009b60 tx=0x7f4cc0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f158596b794fa545 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.720+0000 7f4cd5d9b700 1 -- 172.21.15.1:0/730552943 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc00e6c0 con 0x7f4cc0003a00 2024-03-18T19:06:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.720+0000 7f4cd5d9b700 1 -- 172.21.15.1:0/730552943 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ccc00e820 con 0x7f4cc0003a00 2024-03-18T19:06:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.720+0000 7f4cd5d9b700 1 -- 172.21.15.1:0/730552943 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc00eb90 con 0x7f4cc0003a00 2024-03-18T19:06:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.721+0000 7f4cdd256700 1 -- 172.21.15.1:0/730552943 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cc0003a00 msgr2=0x7f4cc0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.721+0000 7f4cdd256700 1 --2- 172.21.15.1:0/730552943 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cc0003a00 0x7f4cc0003de0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc009b60 tx=0x7f4cc0005880 comp rx=0 tx=0).stop 2024-03-18T19:06:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.721+0000 7f4cdd256700 1 -- 172.21.15.1:0/730552943 shutdown_connections 2024-03-18T19:06:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.721+0000 7f4cdd256700 1 --2- 172.21.15.1:0/730552943 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cc0003a00 0x7f4cc0003de0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.721+0000 7f4cdd256700 1 -- 172.21.15.1:0/730552943 >> 172.21.15.1:0/730552943 conn(0x7f4cd80cc990 msgr2=0x7f4cd80ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.721+0000 7f4cdd256700 1 -- 172.21.15.1:0/730552943 shutdown_connections 2024-03-18T19:06:08.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.721+0000 7f4cdd256700 1 -- 172.21.15.1:0/730552943 wait complete. 2024-03-18T19:06:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.722+0000 7f4cdd256700 1 Processor -- start 2024-03-18T19:06:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.722+0000 7f4cdd256700 1 -- start start 2024-03-18T19:06:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.723+0000 7f4cdd256700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4cc0003a00 0x7f4cd8156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.723+0000 7f4cdd256700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cd815c040 0x7f4cd81e6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.723+0000 7f4cdd256700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cd815c4c0 0x7f4cd81e7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.723+0000 7f4cdd256700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd80c81b0 con 0x7f4cd815c040 2024-03-18T19:06:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.723+0000 7f4cdd256700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd80c8490 con 0x7f4cd815c4c0 2024-03-18T19:06:08.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.723+0000 7f4cdd256700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd80c8770 con 0x7f4cc0003a00 2024-03-18T19:06:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.723+0000 7f4cd659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cd815c040 0x7f4cd81e6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.723+0000 7f4cd659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cd815c040 0x7f4cd81e6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40898/0 (socket says 172.21.15.1:40898) 2024-03-18T19:06:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.723+0000 7f4cd659c700 1 -- 172.21.15.1:0/162342988 learned_addr learned my addr 172.21.15.1:0/162342988 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.723+0000 7f4cd759e700 1 --2- 172.21.15.1:0/162342988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cd815c4c0 0x7f4cd81e7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.723+0000 7f4cd6d9d700 1 --2- 172.21.15.1:0/162342988 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4cc0003a00 0x7f4cd8156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.723+0000 7f4cd659c700 1 -- 172.21.15.1:0/162342988 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4cc0003a00 msgr2=0x7f4cd8156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.723+0000 7f4cd659c700 1 --2- 172.21.15.1:0/162342988 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4cc0003a00 0x7f4cd8156cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.723+0000 7f4cd659c700 1 -- 172.21.15.1:0/162342988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cd815c4c0 msgr2=0x7f4cd81e7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.723+0000 7f4cd659c700 1 --2- 172.21.15.1:0/162342988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cd815c4c0 0x7f4cd81e7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.723+0000 7f4cd659c700 1 -- 172.21.15.1:0/162342988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd80c8a50 con 0x7f4cd815c040 2024-03-18T19:06:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.724+0000 7f4cd659c700 1 --2- 172.21.15.1:0/162342988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cd815c040 0x7f4cd81e6ae0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f4cc80072c0 tx=0x7f4cc80075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.724+0000 7f4cbf7fe700 1 -- 172.21.15.1:0/162342988 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8011580 con 0x7f4cd815c040 2024-03-18T19:06:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.724+0000 7f4cbf7fe700 1 -- 172.21.15.1:0/162342988 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc8011720 con 0x7f4cd815c040 2024-03-18T19:06:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.724+0000 7f4cdd256700 1 -- 172.21.15.1:0/162342988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd80c8d30 con 0x7f4cd815c040 2024-03-18T19:06:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.724+0000 7f4cbf7fe700 1 -- 172.21.15.1:0/162342988 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8011ad0 con 0x7f4cd815c040 2024-03-18T19:06:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.724+0000 7f4cdd256700 1 -- 172.21.15.1:0/162342988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd8157230 con 0x7f4cd815c040 2024-03-18T19:06:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.725+0000 7f4cdd256700 1 -- 172.21.15.1:0/162342988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd80c8f70 con 0x7f4cd815c040 2024-03-18T19:06:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.725+0000 7f4cbf7fe700 1 -- 172.21.15.1:0/162342988 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4cc8011c30 con 0x7f4cd815c040 2024-03-18T19:06:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.726+0000 7f4cbf7fe700 1 --2- 172.21.15.1:0/162342988 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4cc406adb0 0x7f4cc406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.726+0000 7f4cbf7fe700 1 -- 172.21.15.1:0/162342988 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4cc8089510 con 0x7f4cd815c040 2024-03-18T19:06:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.726+0000 7f4cd6d9d700 1 --2- 172.21.15.1:0/162342988 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4cc406adb0 0x7f4cc406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.726+0000 7f4cd6d9d700 1 --2- 172.21.15.1:0/162342988 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4cc406adb0 0x7f4cc406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0090b0 tx=0x7f4ccc01e040 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.730+0000 7f4cbf7fe700 1 -- 172.21.15.1:0/162342988 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4cd80c8f70 con 0x7f4cd815c040 2024-03-18T19:06:08.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.933+0000 7f4cdd256700 1 -- 172.21.15.1:0/162342988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4cd80c8f70 con 0x7f4cd815c040 2024-03-18T19:06:08.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.934+0000 7f4cbf7fe700 1 -- 172.21.15.1:0/162342988 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+16272 (secure 0 0 0) 0x7f4cd80c8f70 con 0x7f4cd815c040 2024-03-18T19:06:08.930 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:08.930 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":103,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:06:06.149482+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:06:02.855989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-18T19:06:04.121199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:06:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.936+0000 7f4cdd256700 1 -- 172.21.15.1:0/162342988 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4cc406adb0 msgr2=0x7f4cc406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.936+0000 7f4cdd256700 1 --2- 172.21.15.1:0/162342988 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4cc406adb0 0x7f4cc406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0090b0 tx=0x7f4ccc01e040 comp rx=0 tx=0).stop 2024-03-18T19:06:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.936+0000 7f4cdd256700 1 -- 172.21.15.1:0/162342988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cd815c040 msgr2=0x7f4cd81e6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.936+0000 7f4cdd256700 1 --2- 172.21.15.1:0/162342988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cd815c040 0x7f4cd81e6ae0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f4cc80072c0 tx=0x7f4cc80075d0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.936+0000 7f4cdd256700 1 -- 172.21.15.1:0/162342988 shutdown_connections 2024-03-18T19:06:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.936+0000 7f4cdd256700 1 --2- 172.21.15.1:0/162342988 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4cc406adb0 0x7f4cc406d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.936+0000 7f4cdd256700 1 --2- 172.21.15.1:0/162342988 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4cc0003a00 0x7f4cd8156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.936+0000 7f4cdd256700 1 --2- 172.21.15.1:0/162342988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cd815c4c0 0x7f4cd81e7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.936+0000 7f4cdd256700 1 --2- 172.21.15.1:0/162342988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cd815c040 0x7f4cd81e6ae0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.936+0000 7f4cdd256700 1 -- 172.21.15.1:0/162342988 >> 172.21.15.1:0/162342988 conn(0x7f4cd80cc990 msgr2=0x7f4cd806e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:08.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.937+0000 7f4cdd256700 1 -- 172.21.15.1:0/162342988 shutdown_connections 2024-03-18T19:06:08.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:08.937+0000 7f4cdd256700 1 -- 172.21.15.1:0/162342988 wait complete. 2024-03-18T19:06:08.945 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T19:06:08.945 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T19:06:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.075+0000 7f38fe5d0700 1 Processor -- start 2024-03-18T19:06:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.075+0000 7f38fe5d0700 1 -- start start 2024-03-18T19:06:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.075+0000 7f38fe5d0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38f815bb90 0x7f38f8099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.075+0000 7f38fe5d0700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f38f815b610 0x7f38f8099f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.075+0000 7f38fe5d0700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f80cc6e0 con 0x7f38f815b230 2024-03-18T19:06:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.075+0000 7f38fe5d0700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38f80ced80 con 0x7f38f815b610 2024-03-18T19:06:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.075+0000 7f38fe5d0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f80d01b0 con 0x7f38f815bb90 2024-03-18T19:06:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.076+0000 7f38fcb6d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f38f815b610 0x7f38f8099f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.076+0000 7f38f77fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38f815bb90 0x7f38f8099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.076+0000 7f38f7fff700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f38f815b230 0x7f38f80cbce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:43442/0 (socket says 172.21.15.1:43442) 2024-03-18T19:06:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.076+0000 7f38f7fff700 1 -- 172.21.15.1:0/2459766482 learned_addr learned my addr 172.21.15.1:0/2459766482 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.076+0000 7f38f6ffd700 1 -- 172.21.15.1:0/2459766482 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080615968 0 0) 0x7f38f80cc6e0 con 0x7f38f815b230 2024-03-18T19:06:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.076+0000 7f38f77fe700 1 -- 172.21.15.1:0/2459766482 >> v2:172.21.15.165:3300/0 conn(0x7f38f815b610 msgr2=0x7f38f8099f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.076+0000 7f38f77fe700 1 --2- 172.21.15.1:0/2459766482 >> v2:172.21.15.165:3300/0 conn(0x7f38f815b610 0x7f38f8099f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.076+0000 7f38f77fe700 1 -- 172.21.15.1:0/2459766482 >> v1:172.21.15.165:6789/0 conn(0x7f38f815b230 legacy=0x7f38f80cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.076+0000 7f38f77fe700 1 -- 172.21.15.1:0/2459766482 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f80cc9c0 con 0x7f38f815bb90 2024-03-18T19:06:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.076+0000 7f38f77fe700 1 --2- 172.21.15.1:0/2459766482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38f815bb90 0x7f38f8099a40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f38ec009840 tx=0x7f38ec009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe902394d7e35f53 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.077+0000 7f38f6ffd700 1 -- 172.21.15.1:0/2459766482 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec011830 con 0x7f38f815bb90 2024-03-18T19:06:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.077+0000 7f38f6ffd700 1 -- 172.21.15.1:0/2459766482 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ec011990 con 0x7f38f815bb90 2024-03-18T19:06:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.077+0000 7f38f6ffd700 1 -- 172.21.15.1:0/2459766482 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec011d20 con 0x7f38f815bb90 2024-03-18T19:06:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.077+0000 7f38fe5d0700 1 -- 172.21.15.1:0/2459766482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38f815bb90 msgr2=0x7f38f8099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.077+0000 7f38fe5d0700 1 --2- 172.21.15.1:0/2459766482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38f815bb90 0x7f38f8099a40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f38ec009840 tx=0x7f38ec009b50 comp rx=0 tx=0).stop 2024-03-18T19:06:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.077+0000 7f38fe5d0700 1 -- 172.21.15.1:0/2459766482 shutdown_connections 2024-03-18T19:06:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.077+0000 7f38fe5d0700 1 --2- 172.21.15.1:0/2459766482 >> v2:172.21.15.165:3300/0 conn(0x7f38f815b610 0x7f38f8099f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.077+0000 7f38fe5d0700 1 --2- 172.21.15.1:0/2459766482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38f815bb90 0x7f38f8099a40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.077+0000 7f38fe5d0700 1 -- 172.21.15.1:0/2459766482 >> 172.21.15.1:0/2459766482 conn(0x7f38f80b2c30 msgr2=0x7f38f80b3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.077+0000 7f38fe5d0700 1 -- 172.21.15.1:0/2459766482 shutdown_connections 2024-03-18T19:06:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.077+0000 7f38fe5d0700 1 -- 172.21.15.1:0/2459766482 wait complete. 2024-03-18T19:06:09.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.079+0000 7f38fe5d0700 1 Processor -- start 2024-03-18T19:06:09.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.079+0000 7f38fe5d0700 1 -- start start 2024-03-18T19:06:09.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.080+0000 7f38fe5d0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38f815b230 0x7f38f80b4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:09.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.080+0000 7f38fe5d0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38f815b610 0x7f38f8157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:09.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.080+0000 7f38fe5d0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38f815bb90 0x7f38f8157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:09.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.080+0000 7f38fe5d0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f80b4fa0 con 0x7f38f815bb90 2024-03-18T19:06:09.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.080+0000 7f38fe5d0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f80b5280 con 0x7f38f815b610 2024-03-18T19:06:09.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.080+0000 7f38fe5d0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f80b5560 con 0x7f38f815b230 2024-03-18T19:06:09.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.080+0000 7f38fcb6d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38f815bb90 0x7f38f8157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:09.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.080+0000 7f38fcb6d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38f815bb90 0x7f38f8157570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40906/0 (socket says 172.21.15.1:40906) 2024-03-18T19:06:09.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.080+0000 7f38fcb6d700 1 -- 172.21.15.1:0/1765389543 learned_addr learned my addr 172.21.15.1:0/1765389543 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:09.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.080+0000 7f38f7fff700 1 --2- 172.21.15.1:0/1765389543 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38f815b230 0x7f38f80b4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:09.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.080+0000 7f38f77fe700 1 --2- 172.21.15.1:0/1765389543 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38f815b610 0x7f38f8157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:09.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.080+0000 7f38fcb6d700 1 -- 172.21.15.1:0/1765389543 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38f815b230 msgr2=0x7f38f80b4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:09.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.080+0000 7f38fcb6d700 1 --2- 172.21.15.1:0/1765389543 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38f815b230 0x7f38f80b4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.080+0000 7f38fcb6d700 1 -- 172.21.15.1:0/1765389543 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38f815b610 msgr2=0x7f38f8157030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:09.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.080+0000 7f38fcb6d700 1 --2- 172.21.15.1:0/1765389543 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38f815b610 0x7f38f8157030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.080+0000 7f38fcb6d700 1 -- 172.21.15.1:0/1765389543 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ec0094a0 con 0x7f38f815bb90 2024-03-18T19:06:09.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.081+0000 7f38fcb6d700 1 --2- 172.21.15.1:0/1765389543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38f815bb90 0x7f38f8157570 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f38f000efc0 tx=0x7f38f000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:09.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.081+0000 7f38f4ff9700 1 -- 172.21.15.1:0/1765389543 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f0009ea0 con 0x7f38f815bb90 2024-03-18T19:06:09.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.081+0000 7f38f4ff9700 1 -- 172.21.15.1:0/1765389543 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f000f040 con 0x7f38f815bb90 2024-03-18T19:06:09.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.081+0000 7f38fe5d0700 1 -- 172.21.15.1:0/1765389543 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f80c4c70 con 0x7f38f815bb90 2024-03-18T19:06:09.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.081+0000 7f38f4ff9700 1 -- 172.21.15.1:0/1765389543 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f000ced0 con 0x7f38f815bb90 2024-03-18T19:06:09.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.081+0000 7f38fe5d0700 1 -- 172.21.15.1:0/1765389543 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f80b4cb0 con 0x7f38f815bb90 2024-03-18T19:06:09.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.082+0000 7f38fe5d0700 1 -- 172.21.15.1:0/1765389543 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38f80c5190 con 0x7f38f815bb90 2024-03-18T19:06:09.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.082+0000 7f38f4ff9700 1 -- 172.21.15.1:0/1765389543 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f38f0014480 con 0x7f38f815bb90 2024-03-18T19:06:09.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.083+0000 7f38f4ff9700 1 --2- 172.21.15.1:0/1765389543 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f38e406adb0 0x7f38e406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:09.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.083+0000 7f38f4ff9700 1 -- 172.21.15.1:0/1765389543 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f38f0089f30 con 0x7f38f815bb90 2024-03-18T19:06:09.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.083+0000 7f38f7fff700 1 --2- 172.21.15.1:0/1765389543 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f38e406adb0 0x7f38e406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:09.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.083+0000 7f38f7fff700 1 --2- 172.21.15.1:0/1765389543 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f38e406adb0 0x7f38e406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f38e8007fd0 tx=0x7f38e8007de0 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.087+0000 7f38f4ff9700 1 -- 172.21.15.1:0/1765389543 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f38f0054db0 con 0x7f38f815bb90 2024-03-18T19:06:09.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.283+0000 7f38fe5d0700 1 -- 172.21.15.1:0/1765389543 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f38f80cc3f0 con 0x7f38e406adb0 2024-03-18T19:06:09.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.292+0000 7f38f4ff9700 1 -- 172.21.15.1:0/1765389543 <== mgr.6631 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290090 (secure 0 0 0) 0x7f38e8001370 con 0x7f38e406adb0 2024-03-18T19:06:09.292 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:09.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.295+0000 7f38fe5d0700 1 -- 172.21.15.1:0/1765389543 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f38e406adb0 msgr2=0x7f38e406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:09.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.295+0000 7f38fe5d0700 1 --2- 172.21.15.1:0/1765389543 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f38e406adb0 0x7f38e406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f38e8007fd0 tx=0x7f38e8007de0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.295+0000 7f38fe5d0700 1 -- 172.21.15.1:0/1765389543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38f815bb90 msgr2=0x7f38f8157570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:09.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.295+0000 7f38fe5d0700 1 --2- 172.21.15.1:0/1765389543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38f815bb90 0x7f38f8157570 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f38f000efc0 tx=0x7f38f000c3a0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.295+0000 7f38fe5d0700 1 -- 172.21.15.1:0/1765389543 shutdown_connections 2024-03-18T19:06:09.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.295+0000 7f38fe5d0700 1 --2- 172.21.15.1:0/1765389543 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f38e406adb0 0x7f38e406d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.295+0000 7f38fe5d0700 1 --2- 172.21.15.1:0/1765389543 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f38f815b230 0x7f38f80b4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.295+0000 7f38fe5d0700 1 --2- 172.21.15.1:0/1765389543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38f815bb90 0x7f38f8157570 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.295+0000 7f38fe5d0700 1 --2- 172.21.15.1:0/1765389543 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38f815b610 0x7f38f8157030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.295+0000 7f38fe5d0700 1 -- 172.21.15.1:0/1765389543 >> 172.21.15.1:0/1765389543 conn(0x7f38f80b2c30 msgr2=0x7f38f80cbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:09.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.296+0000 7f38fe5d0700 1 -- 172.21.15.1:0/1765389543 shutdown_connections 2024-03-18T19:06:09.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.296+0000 7f38fe5d0700 1 -- 172.21.15.1:0/1765389543 wait complete. 2024-03-18T19:06:09.296 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T19:06:09.304 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-18T19:06:07.644654+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":303,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":98,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":342792,"kb_used_data":10996,"kb_used_omap":12,"kb_used_meta":331635,"kb_avail":754631928,"statfs":{"total":773094113280,"available":772743094272,"internally_reserved":0,"allocated":11259904,"data_stored":6180248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":339594809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000803"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.171835+0000","last_change":"2024-03-18T19:06:05.170868+0000","last_active":"2024-03-18T19:06:05.171835+0000","last_peered":"2024-03-18T19:06:05.171835+0000","last_clean":"2024-03-18T19:06:05.171835+0000","last_became_active":"2024-03-18T19:06:05.170592+0000","last_became_peered":"2024-03-18T19:06:05.170592+0000","last_unstale":"2024-03-18T19:06:05.171835+0000","last_undegraded":"2024-03-18T19:06:05.171835+0000","last_fullsized":"2024-03-18T19:06:05.171835+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:59:50.028248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157143+0000","last_change":"2024-03-18T19:06:05.168652+0000","last_active":"2024-03-18T19:06:06.157143+0000","last_peered":"2024-03-18T19:06:06.157143+0000","last_clean":"2024-03-18T19:06:06.157143+0000","last_became_active":"2024-03-18T19:06:05.168384+0000","last_became_peered":"2024-03-18T19:06:05.168384+0000","last_unstale":"2024-03-18T19:06:06.157143+0000","last_undegraded":"2024-03-18T19:06:06.157143+0000","last_fullsized":"2024-03-18T19:06:06.157143+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:10.292804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.175497+0000","last_change":"2024-03-18T19:06:05.172281+0000","last_active":"2024-03-18T19:06:05.175497+0000","last_peered":"2024-03-18T19:06:05.175497+0000","last_clean":"2024-03-18T19:06:05.175497+0000","last_became_active":"2024-03-18T19:06:05.172042+0000","last_became_peered":"2024-03-18T19:06:05.172042+0000","last_unstale":"2024-03-18T19:06:05.175497+0000","last_undegraded":"2024-03-18T19:06:05.175497+0000","last_fullsized":"2024-03-18T19:06:05.175497+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:09:48.834328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.176656+0000","last_change":"2024-03-18T19:06:05.173122+0000","last_active":"2024-03-18T19:06:05.176656+0000","last_peered":"2024-03-18T19:06:05.176656+0000","last_clean":"2024-03-18T19:06:05.176656+0000","last_became_active":"2024-03-18T19:06:05.172838+0000","last_became_peered":"2024-03-18T19:06:05.172838+0000","last_unstale":"2024-03-18T19:06:05.176656+0000","last_undegraded":"2024-03-18T19:06:05.176656+0000","last_fullsized":"2024-03-18T19:06:05.176656+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:11:36.194047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174488+0000","last_change":"2024-03-18T19:06:05.170972+0000","last_active":"2024-03-18T19:06:05.174488+0000","last_peered":"2024-03-18T19:06:05.174488+0000","last_clean":"2024-03-18T19:06:05.174488+0000","last_became_active":"2024-03-18T19:06:05.170709+0000","last_became_peered":"2024-03-18T19:06:05.170709+0000","last_unstale":"2024-03-18T19:06:05.174488+0000","last_undegraded":"2024-03-18T19:06:05.174488+0000","last_fullsized":"2024-03-18T19:06:05.174488+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:03:46.372144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.180909+0000","last_change":"2024-03-18T19:06:05.179927+0000","last_active":"2024-03-18T19:06:05.180909+0000","last_peered":"2024-03-18T19:06:05.180909+0000","last_clean":"2024-03-18T19:06:05.180909+0000","last_became_active":"2024-03-18T19:06:05.179631+0000","last_became_peered":"2024-03-18T19:06:05.179631+0000","last_unstale":"2024-03-18T19:06:05.180909+0000","last_undegraded":"2024-03-18T19:06:05.180909+0000","last_fullsized":"2024-03-18T19:06:05.180909+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:03:43.145212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.156750+0000","last_change":"2024-03-18T19:06:05.168731+0000","last_active":"2024-03-18T19:06:06.156750+0000","last_peered":"2024-03-18T19:06:06.156750+0000","last_clean":"2024-03-18T19:06:06.156750+0000","last_became_active":"2024-03-18T19:06:05.167788+0000","last_became_peered":"2024-03-18T19:06:05.167788+0000","last_unstale":"2024-03-18T19:06:06.156750+0000","last_undegraded":"2024-03-18T19:06:06.156750+0000","last_fullsized":"2024-03-18T19:06:06.156750+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:09:48.834328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174289+0000","last_change":"2024-03-18T19:06:05.167420+0000","last_active":"2024-03-18T19:06:05.174289+0000","last_peered":"2024-03-18T19:06:05.174289+0000","last_clean":"2024-03-18T19:06:05.174289+0000","last_became_active":"2024-03-18T19:06:05.165913+0000","last_became_peered":"2024-03-18T19:06:05.165913+0000","last_unstale":"2024-03-18T19:06:05.174289+0000","last_undegraded":"2024-03-18T19:06:05.174289+0000","last_fullsized":"2024-03-18T19:06:05.174289+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:29.423402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.182553+0000","last_change":"2024-03-18T19:06:05.181568+0000","last_active":"2024-03-18T19:06:05.182553+0000","last_peered":"2024-03-18T19:06:05.182553+0000","last_clean":"2024-03-18T19:06:05.182553+0000","last_became_active":"2024-03-18T19:06:05.181258+0000","last_became_peered":"2024-03-18T19:06:05.181258+0000","last_unstale":"2024-03-18T19:06:05.182553+0000","last_undegraded":"2024-03-18T19:06:05.182553+0000","last_fullsized":"2024-03-18T19:06:05.182553+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:09:40.312969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174368+0000","last_change":"2024-03-18T19:06:05.171798+0000","last_active":"2024-03-18T19:06:05.174368+0000","last_peered":"2024-03-18T19:06:05.174368+0000","last_clean":"2024-03-18T19:06:05.174368+0000","last_became_active":"2024-03-18T19:06:05.171537+0000","last_became_peered":"2024-03-18T19:06:05.171537+0000","last_unstale":"2024-03-18T19:06:05.174368+0000","last_undegraded":"2024-03-18T19:06:05.174368+0000","last_fullsized":"2024-03-18T19:06:05.174368+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:07:30.773855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157226+0000","last_change":"2024-03-18T19:06:05.169086+0000","last_active":"2024-03-18T19:06:06.157226+0000","last_peered":"2024-03-18T19:06:06.157226+0000","last_clean":"2024-03-18T19:06:06.157226+0000","last_became_active":"2024-03-18T19:06:05.168472+0000","last_became_peered":"2024-03-18T19:06:05.168472+0000","last_unstale":"2024-03-18T19:06:06.157226+0000","last_undegraded":"2024-03-18T19:06:06.157226+0000","last_fullsized":"2024-03-18T19:06:06.157226+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:57.274617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.178858+0000","last_change":"2024-03-18T19:06:05.176413+0000","last_active":"2024-03-18T19:06:05.178858+0000","last_peered":"2024-03-18T19:06:05.178858+0000","last_clean":"2024-03-18T19:06:05.178858+0000","last_became_active":"2024-03-18T19:06:05.175933+0000","last_became_peered":"2024-03-18T19:06:05.175933+0000","last_unstale":"2024-03-18T19:06:05.178858+0000","last_undegraded":"2024-03-18T19:06:05.178858+0000","last_fullsized":"2024-03-18T19:06:05.178858+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:30:47.424183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.184117+0000","last_change":"2024-03-18T19:06:05.182075+0000","last_active":"2024-03-18T19:06:05.184117+0000","last_peered":"2024-03-18T19:06:05.184117+0000","last_clean":"2024-03-18T19:06:05.184117+0000","last_became_active":"2024-03-18T19:06:05.181805+0000","last_became_peered":"2024-03-18T19:06:05.181805+0000","last_unstale":"2024-03-18T19:06:05.184117+0000","last_undegraded":"2024-03-18T19:06:05.184117+0000","last_fullsized":"2024-03-18T19:06:05.184117+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:46.650103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.183569+0000","last_change":"2024-03-18T19:06:05.181352+0000","last_active":"2024-03-18T19:06:05.183569+0000","last_peered":"2024-03-18T19:06:05.183569+0000","last_clean":"2024-03-18T19:06:05.183569+0000","last_became_active":"2024-03-18T19:06:05.181008+0000","last_became_peered":"2024-03-18T19:06:05.181008+0000","last_unstale":"2024-03-18T19:06:05.183569+0000","last_undegraded":"2024-03-18T19:06:05.183569+0000","last_fullsized":"2024-03-18T19:06:05.183569+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:11:30.136013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.156921+0000","last_change":"2024-03-18T19:06:05.175508+0000","last_active":"2024-03-18T19:06:06.156921+0000","last_peered":"2024-03-18T19:06:06.156921+0000","last_clean":"2024-03-18T19:06:06.156921+0000","last_became_active":"2024-03-18T19:06:05.175053+0000","last_became_peered":"2024-03-18T19:06:05.175053+0000","last_unstale":"2024-03-18T19:06:06.156921+0000","last_undegraded":"2024-03-18T19:06:06.156921+0000","last_fullsized":"2024-03-18T19:06:06.156921+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:51.812857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.169069+0000","last_change":"2024-03-18T19:06:05.167015+0000","last_active":"2024-03-18T19:06:05.169069+0000","last_peered":"2024-03-18T19:06:05.169069+0000","last_clean":"2024-03-18T19:06:05.169069+0000","last_became_active":"2024-03-18T19:06:05.166748+0000","last_became_peered":"2024-03-18T19:06:05.166748+0000","last_unstale":"2024-03-18T19:06:05.169069+0000","last_undegraded":"2024-03-18T19:06:05.169069+0000","last_fullsized":"2024-03-18T19:06:05.169069+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:39.218334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.175060+0000","last_change":"2024-03-18T19:06:05.171233+0000","last_active":"2024-03-18T19:06:05.175060+0000","last_peered":"2024-03-18T19:06:05.175060+0000","last_clean":"2024-03-18T19:06:05.175060+0000","last_became_active":"2024-03-18T19:06:05.170894+0000","last_became_peered":"2024-03-18T19:06:05.170894+0000","last_unstale":"2024-03-18T19:06:05.175060+0000","last_undegraded":"2024-03-18T19:06:05.175060+0000","last_fullsized":"2024-03-18T19:06:05.175060+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:01:48.953093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.175414+0000","last_change":"2024-03-18T19:06:05.173544+0000","last_active":"2024-03-18T19:06:05.175414+0000","last_peered":"2024-03-18T19:06:05.175414+0000","last_clean":"2024-03-18T19:06:05.175414+0000","last_became_active":"2024-03-18T19:06:05.172319+0000","last_became_peered":"2024-03-18T19:06:05.172319+0000","last_unstale":"2024-03-18T19:06:05.175414+0000","last_undegraded":"2024-03-18T19:06:05.175414+0000","last_fullsized":"2024-03-18T19:06:05.175414+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:55:27.854722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.180211+0000","last_change":"2024-03-18T19:06:05.176492+0000","last_active":"2024-03-18T19:06:05.180211+0000","last_peered":"2024-03-18T19:06:05.180211+0000","last_clean":"2024-03-18T19:06:05.180211+0000","last_became_active":"2024-03-18T19:06:05.176223+0000","last_became_peered":"2024-03-18T19:06:05.176223+0000","last_unstale":"2024-03-18T19:06:05.180211+0000","last_undegraded":"2024-03-18T19:06:05.180211+0000","last_fullsized":"2024-03-18T19:06:05.180211+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:40.596183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.176577+0000","last_change":"2024-03-18T19:06:05.175850+0000","last_active":"2024-03-18T19:06:05.176577+0000","last_peered":"2024-03-18T19:06:05.176577+0000","last_clean":"2024-03-18T19:06:05.176577+0000","last_became_active":"2024-03-18T19:06:05.175213+0000","last_became_peered":"2024-03-18T19:06:05.175213+0000","last_unstale":"2024-03-18T19:06:05.176577+0000","last_undegraded":"2024-03-18T19:06:05.176577+0000","last_fullsized":"2024-03-18T19:06:05.176577+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:13.231915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.179044+0000","last_change":"2024-03-18T19:06:05.175698+0000","last_active":"2024-03-18T19:06:05.179044+0000","last_peered":"2024-03-18T19:06:05.179044+0000","last_clean":"2024-03-18T19:06:05.179044+0000","last_became_active":"2024-03-18T19:06:05.175502+0000","last_became_peered":"2024-03-18T19:06:05.175502+0000","last_unstale":"2024-03-18T19:06:05.179044+0000","last_undegraded":"2024-03-18T19:06:05.179044+0000","last_fullsized":"2024-03-18T19:06:05.179044+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:38:46.621286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.184140+0000","last_change":"2024-03-18T19:06:05.183425+0000","last_active":"2024-03-18T19:06:05.184140+0000","last_peered":"2024-03-18T19:06:05.184140+0000","last_clean":"2024-03-18T19:06:05.184140+0000","last_became_active":"2024-03-18T19:06:05.183155+0000","last_became_peered":"2024-03-18T19:06:05.183155+0000","last_unstale":"2024-03-18T19:06:05.184140+0000","last_undegraded":"2024-03-18T19:06:05.184140+0000","last_fullsized":"2024-03-18T19:06:05.184140+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:17.916885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174059+0000","last_change":"2024-03-18T19:06:05.171509+0000","last_active":"2024-03-18T19:06:05.174059+0000","last_peered":"2024-03-18T19:06:05.174059+0000","last_clean":"2024-03-18T19:06:05.174059+0000","last_became_active":"2024-03-18T19:06:05.171248+0000","last_became_peered":"2024-03-18T19:06:05.171248+0000","last_unstale":"2024-03-18T19:06:05.174059+0000","last_undegraded":"2024-03-18T19:06:05.174059+0000","last_fullsized":"2024-03-18T19:06:05.174059+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:45:43.675246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.156878+0000","last_change":"2024-03-18T19:06:05.173083+0000","last_active":"2024-03-18T19:06:06.156878+0000","last_peered":"2024-03-18T19:06:06.156878+0000","last_clean":"2024-03-18T19:06:06.156878+0000","last_became_active":"2024-03-18T19:06:05.172856+0000","last_became_peered":"2024-03-18T19:06:05.172856+0000","last_unstale":"2024-03-18T19:06:06.156878+0000","last_undegraded":"2024-03-18T19:06:06.156878+0000","last_fullsized":"2024-03-18T19:06:06.156878+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:50.007982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157778+0000","last_change":"2024-03-18T19:06:05.173791+0000","last_active":"2024-03-18T19:06:06.157778+0000","last_peered":"2024-03-18T19:06:06.157778+0000","last_clean":"2024-03-18T19:06:06.157778+0000","last_became_active":"2024-03-18T19:06:05.173538+0000","last_became_peered":"2024-03-18T19:06:05.173538+0000","last_unstale":"2024-03-18T19:06:06.157778+0000","last_undegraded":"2024-03-18T19:06:06.157778+0000","last_fullsized":"2024-03-18T19:06:06.157778+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:36:48.269178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.178673+0000","last_change":"2024-03-18T19:06:05.175304+0000","last_active":"2024-03-18T19:06:05.178673+0000","last_peered":"2024-03-18T19:06:05.178673+0000","last_clean":"2024-03-18T19:06:05.178673+0000","last_became_active":"2024-03-18T19:06:05.175037+0000","last_became_peered":"2024-03-18T19:06:05.175037+0000","last_unstale":"2024-03-18T19:06:05.178673+0000","last_undegraded":"2024-03-18T19:06:05.178673+0000","last_fullsized":"2024-03-18T19:06:05.178673+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:55:27.854722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.173923+0000","last_change":"2024-03-18T19:06:05.172130+0000","last_active":"2024-03-18T19:06:05.173923+0000","last_peered":"2024-03-18T19:06:05.173923+0000","last_clean":"2024-03-18T19:06:05.173923+0000","last_became_active":"2024-03-18T19:06:05.171841+0000","last_became_peered":"2024-03-18T19:06:05.171841+0000","last_unstale":"2024-03-18T19:06:05.173923+0000","last_undegraded":"2024-03-18T19:06:05.173923+0000","last_fullsized":"2024-03-18T19:06:05.173923+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:14.564659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.175085+0000","last_change":"2024-03-18T19:06:05.173695+0000","last_active":"2024-03-18T19:06:05.175085+0000","last_peered":"2024-03-18T19:06:05.175085+0000","last_clean":"2024-03-18T19:06:05.175085+0000","last_became_active":"2024-03-18T19:06:05.173416+0000","last_became_peered":"2024-03-18T19:06:05.173416+0000","last_unstale":"2024-03-18T19:06:05.175085+0000","last_undegraded":"2024-03-18T19:06:05.175085+0000","last_fullsized":"2024-03-18T19:06:05.175085+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:59:06.088817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157130+0000","last_change":"2024-03-18T19:06:05.163843+0000","last_active":"2024-03-18T19:06:06.157130+0000","last_peered":"2024-03-18T19:06:06.157130+0000","last_clean":"2024-03-18T19:06:06.157130+0000","last_became_active":"2024-03-18T19:06:05.163515+0000","last_became_peered":"2024-03-18T19:06:05.163515+0000","last_unstale":"2024-03-18T19:06:06.157130+0000","last_undegraded":"2024-03-18T19:06:06.157130+0000","last_fullsized":"2024-03-18T19:06:06.157130+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:22.181026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.168698+0000","last_change":"2024-03-18T19:06:05.165309+0000","last_active":"2024-03-18T19:06:05.168698+0000","last_peered":"2024-03-18T19:06:05.168698+0000","last_clean":"2024-03-18T19:06:05.168698+0000","last_became_active":"2024-03-18T19:06:05.165002+0000","last_became_peered":"2024-03-18T19:06:05.165002+0000","last_unstale":"2024-03-18T19:06:05.168698+0000","last_undegraded":"2024-03-18T19:06:05.168698+0000","last_fullsized":"2024-03-18T19:06:05.168698+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:04.872630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174130+0000","last_change":"2024-03-18T19:06:05.162716+0000","last_active":"2024-03-18T19:06:05.174130+0000","last_peered":"2024-03-18T19:06:05.174130+0000","last_clean":"2024-03-18T19:06:05.174130+0000","last_became_active":"2024-03-18T19:06:05.162474+0000","last_became_peered":"2024-03-18T19:06:05.162474+0000","last_unstale":"2024-03-18T19:06:05.174130+0000","last_undegraded":"2024-03-18T19:06:05.174130+0000","last_fullsized":"2024-03-18T19:06:05.174130+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:30:47.424183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174497+0000","last_change":"2024-03-18T19:06:05.172172+0000","last_active":"2024-03-18T19:06:05.174497+0000","last_peered":"2024-03-18T19:06:05.174497+0000","last_clean":"2024-03-18T19:06:05.174497+0000","last_became_active":"2024-03-18T19:06:05.171556+0000","last_became_peered":"2024-03-18T19:06:05.171556+0000","last_unstale":"2024-03-18T19:06:05.174497+0000","last_undegraded":"2024-03-18T19:06:05.174497+0000","last_fullsized":"2024-03-18T19:06:05.174497+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:32:09.971963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.175265+0000","last_change":"2024-03-18T19:06:05.171715+0000","last_active":"2024-03-18T19:06:05.175265+0000","last_peered":"2024-03-18T19:06:05.175265+0000","last_clean":"2024-03-18T19:06:05.175265+0000","last_became_active":"2024-03-18T19:06:05.171456+0000","last_became_peered":"2024-03-18T19:06:05.171456+0000","last_unstale":"2024-03-18T19:06:05.175265+0000","last_undegraded":"2024-03-18T19:06:05.175265+0000","last_fullsized":"2024-03-18T19:06:05.175265+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:54:09.212832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"102'4","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.181733+0000","last_change":"2024-03-18T19:06:04.734193+0000","last_active":"2024-03-18T19:06:05.181733+0000","last_peered":"2024-03-18T19:06:05.181733+0000","last_clean":"2024-03-18T19:06:05.181733+0000","last_became_active":"2024-03-18T19:06:04.733935+0000","last_became_peered":"2024-03-18T19:06:04.733935+0000","last_unstale":"2024-03-18T19:06:05.181733+0000","last_undegraded":"2024-03-18T19:06:05.181733+0000","last_fullsized":"2024-03-18T19:06:05.181733+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:21.798183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.156918+0000","last_change":"2024-03-18T19:06:05.175305+0000","last_active":"2024-03-18T19:06:06.156918+0000","last_peered":"2024-03-18T19:06:06.156918+0000","last_clean":"2024-03-18T19:06:06.156918+0000","last_became_active":"2024-03-18T19:06:05.174712+0000","last_became_peered":"2024-03-18T19:06:05.174712+0000","last_unstale":"2024-03-18T19:06:06.156918+0000","last_undegraded":"2024-03-18T19:06:06.156918+0000","last_fullsized":"2024-03-18T19:06:06.156918+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:51:29.990239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.158467+0000","last_change":"2024-03-18T19:06:04.727730+0000","last_active":"2024-03-18T19:06:05.158467+0000","last_peered":"2024-03-18T19:06:05.158467+0000","last_clean":"2024-03-18T19:06:05.158467+0000","last_became_active":"2024-03-18T19:06:04.727444+0000","last_became_peered":"2024-03-18T19:06:04.727444+0000","last_unstale":"2024-03-18T19:06:05.158467+0000","last_undegraded":"2024-03-18T19:06:05.158467+0000","last_fullsized":"2024-03-18T19:06:05.158467+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:24:31.548334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.184227+0000","last_change":"2024-03-18T19:06:05.183264+0000","last_active":"2024-03-18T19:06:05.184227+0000","last_peered":"2024-03-18T19:06:05.184227+0000","last_clean":"2024-03-18T19:06:05.184227+0000","last_became_active":"2024-03-18T19:06:05.182888+0000","last_became_peered":"2024-03-18T19:06:05.182888+0000","last_unstale":"2024-03-18T19:06:05.184227+0000","last_undegraded":"2024-03-18T19:06:05.184227+0000","last_fullsized":"2024-03-18T19:06:05.184227+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:09:48.834328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.170868+0000","last_change":"2024-03-18T19:06:04.728484+0000","last_active":"2024-03-18T19:06:05.170868+0000","last_peered":"2024-03-18T19:06:05.170868+0000","last_clean":"2024-03-18T19:06:05.170868+0000","last_became_active":"2024-03-18T19:06:04.728151+0000","last_became_peered":"2024-03-18T19:06:04.728151+0000","last_unstale":"2024-03-18T19:06:05.170868+0000","last_undegraded":"2024-03-18T19:06:05.170868+0000","last_fullsized":"2024-03-18T19:06:05.170868+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:36:27.277009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.179028+0000","last_change":"2024-03-18T19:06:05.175617+0000","last_active":"2024-03-18T19:06:05.179028+0000","last_peered":"2024-03-18T19:06:05.179028+0000","last_clean":"2024-03-18T19:06:05.179028+0000","last_became_active":"2024-03-18T19:06:05.175412+0000","last_became_peered":"2024-03-18T19:06:05.175412+0000","last_unstale":"2024-03-18T19:06:05.179028+0000","last_undegraded":"2024-03-18T19:06:05.179028+0000","last_fullsized":"2024-03-18T19:06:05.179028+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:45:28.072614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"102'2","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157317+0000","last_change":"2024-03-18T19:06:04.729562+0000","last_active":"2024-03-18T19:06:06.157317+0000","last_peered":"2024-03-18T19:06:06.157317+0000","last_clean":"2024-03-18T19:06:06.157317+0000","last_became_active":"2024-03-18T19:06:04.729275+0000","last_became_peered":"2024-03-18T19:06:04.729275+0000","last_unstale":"2024-03-18T19:06:06.157317+0000","last_undegraded":"2024-03-18T19:06:06.157317+0000","last_fullsized":"2024-03-18T19:06:06.157317+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:26:37.482587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157090+0000","last_change":"2024-03-18T19:06:05.170212+0000","last_active":"2024-03-18T19:06:06.157090+0000","last_peered":"2024-03-18T19:06:06.157090+0000","last_clean":"2024-03-18T19:06:06.157090+0000","last_became_active":"2024-03-18T19:06:05.169953+0000","last_became_peered":"2024-03-18T19:06:05.169953+0000","last_unstale":"2024-03-18T19:06:06.157090+0000","last_undegraded":"2024-03-18T19:06:06.157090+0000","last_fullsized":"2024-03-18T19:06:06.157090+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:46.451963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.170935+0000","last_change":"2024-03-18T19:06:04.734707+0000","last_active":"2024-03-18T19:06:05.170935+0000","last_peered":"2024-03-18T19:06:05.170935+0000","last_clean":"2024-03-18T19:06:05.170935+0000","last_became_active":"2024-03-18T19:06:04.734461+0000","last_became_peered":"2024-03-18T19:06:04.734461+0000","last_unstale":"2024-03-18T19:06:05.170935+0000","last_undegraded":"2024-03-18T19:06:05.170935+0000","last_fullsized":"2024-03-18T19:06:05.170935+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:56.143928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157119+0000","last_change":"2024-03-18T19:06:05.169179+0000","last_active":"2024-03-18T19:06:06.157119+0000","last_peered":"2024-03-18T19:06:06.157119+0000","last_clean":"2024-03-18T19:06:06.157119+0000","last_became_active":"2024-03-18T19:06:05.168843+0000","last_became_peered":"2024-03-18T19:06:05.168843+0000","last_unstale":"2024-03-18T19:06:06.157119+0000","last_undegraded":"2024-03-18T19:06:06.157119+0000","last_fullsized":"2024-03-18T19:06:06.157119+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:48:58.435719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.181115+0000","last_change":"2024-03-18T19:06:04.734668+0000","last_active":"2024-03-18T19:06:05.181115+0000","last_peered":"2024-03-18T19:06:05.181115+0000","last_clean":"2024-03-18T19:06:05.181115+0000","last_became_active":"2024-03-18T19:06:04.734409+0000","last_became_peered":"2024-03-18T19:06:04.734409+0000","last_unstale":"2024-03-18T19:06:05.181115+0000","last_undegraded":"2024-03-18T19:06:05.181115+0000","last_fullsized":"2024-03-18T19:06:05.181115+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:49:20.902554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.182557+0000","last_change":"2024-03-18T19:06:05.181651+0000","last_active":"2024-03-18T19:06:05.182557+0000","last_peered":"2024-03-18T19:06:05.182557+0000","last_clean":"2024-03-18T19:06:05.182557+0000","last_became_active":"2024-03-18T19:06:05.181328+0000","last_became_peered":"2024-03-18T19:06:05.181328+0000","last_unstale":"2024-03-18T19:06:05.182557+0000","last_undegraded":"2024-03-18T19:06:05.182557+0000","last_fullsized":"2024-03-18T19:06:05.182557+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:36:28.282894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157027+0000","last_change":"2024-03-18T19:06:04.727277+0000","last_active":"2024-03-18T19:06:06.157027+0000","last_peered":"2024-03-18T19:06:06.157027+0000","last_clean":"2024-03-18T19:06:06.157027+0000","last_became_active":"2024-03-18T19:06:04.727001+0000","last_became_peered":"2024-03-18T19:06:04.727001+0000","last_unstale":"2024-03-18T19:06:06.157027+0000","last_undegraded":"2024-03-18T19:06:06.157027+0000","last_fullsized":"2024-03-18T19:06:06.157027+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:14.933050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157252+0000","last_change":"2024-03-18T19:06:05.168878+0000","last_active":"2024-03-18T19:06:06.157252+0000","last_peered":"2024-03-18T19:06:06.157252+0000","last_clean":"2024-03-18T19:06:06.157252+0000","last_became_active":"2024-03-18T19:06:05.168567+0000","last_became_peered":"2024-03-18T19:06:05.168567+0000","last_unstale":"2024-03-18T19:06:06.157252+0000","last_undegraded":"2024-03-18T19:06:06.157252+0000","last_fullsized":"2024-03-18T19:06:06.157252+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:09:40.312969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.161685+0000","last_change":"2024-03-18T19:06:04.735666+0000","last_active":"2024-03-18T19:06:05.161685+0000","last_peered":"2024-03-18T19:06:05.161685+0000","last_clean":"2024-03-18T19:06:05.161685+0000","last_became_active":"2024-03-18T19:06:04.735407+0000","last_became_peered":"2024-03-18T19:06:04.735407+0000","last_unstale":"2024-03-18T19:06:05.161685+0000","last_undegraded":"2024-03-18T19:06:05.161685+0000","last_fullsized":"2024-03-18T19:06:05.161685+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:52.738028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.156879+0000","last_change":"2024-03-18T19:06:05.166564+0000","last_active":"2024-03-18T19:06:06.156879+0000","last_peered":"2024-03-18T19:06:06.156879+0000","last_clean":"2024-03-18T19:06:06.156879+0000","last_became_active":"2024-03-18T19:06:05.166311+0000","last_became_peered":"2024-03-18T19:06:05.166311+0000","last_unstale":"2024-03-18T19:06:06.156879+0000","last_undegraded":"2024-03-18T19:06:06.156879+0000","last_fullsized":"2024-03-18T19:06:06.156879+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:36:34.321207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.155624+0000","last_change":"2024-03-18T19:06:04.727189+0000","last_active":"2024-03-18T19:06:05.155624+0000","last_peered":"2024-03-18T19:06:05.155624+0000","last_clean":"2024-03-18T19:06:05.155624+0000","last_became_active":"2024-03-18T19:06:04.726902+0000","last_became_peered":"2024-03-18T19:06:04.726902+0000","last_unstale":"2024-03-18T19:06:05.155624+0000","last_undegraded":"2024-03-18T19:06:05.155624+0000","last_fullsized":"2024-03-18T19:06:05.155624+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:45:42.669362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.178648+0000","last_change":"2024-03-18T19:06:05.174889+0000","last_active":"2024-03-18T19:06:05.178648+0000","last_peered":"2024-03-18T19:06:05.178648+0000","last_clean":"2024-03-18T19:06:05.178648+0000","last_became_active":"2024-03-18T19:06:05.174628+0000","last_became_peered":"2024-03-18T19:06:05.174628+0000","last_unstale":"2024-03-18T19:06:05.178648+0000","last_undegraded":"2024-03-18T19:06:05.178648+0000","last_fullsized":"2024-03-18T19:06:05.178648+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:34.217833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.164960+0000","last_change":"2024-03-18T19:06:04.731599+0000","last_active":"2024-03-18T19:06:05.164960+0000","last_peered":"2024-03-18T19:06:05.164960+0000","last_clean":"2024-03-18T19:06:05.164960+0000","last_became_active":"2024-03-18T19:06:04.731332+0000","last_became_peered":"2024-03-18T19:06:04.731332+0000","last_unstale":"2024-03-18T19:06:05.164960+0000","last_undegraded":"2024-03-18T19:06:05.164960+0000","last_fullsized":"2024-03-18T19:06:05.164960+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:24:31.548334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.183593+0000","last_change":"2024-03-18T19:06:05.181341+0000","last_active":"2024-03-18T19:06:05.183593+0000","last_peered":"2024-03-18T19:06:05.183593+0000","last_clean":"2024-03-18T19:06:05.183593+0000","last_became_active":"2024-03-18T19:06:05.181028+0000","last_became_peered":"2024-03-18T19:06:05.181028+0000","last_unstale":"2024-03-18T19:06:05.183593+0000","last_undegraded":"2024-03-18T19:06:05.183593+0000","last_fullsized":"2024-03-18T19:06:05.183593+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:06.088405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"102'3","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.175438+0000","last_change":"2024-03-18T19:06:04.729893+0000","last_active":"2024-03-18T19:06:05.175438+0000","last_peered":"2024-03-18T19:06:05.175438+0000","last_clean":"2024-03-18T19:06:05.175438+0000","last_became_active":"2024-03-18T19:06:04.729615+0000","last_became_peered":"2024-03-18T19:06:04.729615+0000","last_unstale":"2024-03-18T19:06:05.175438+0000","last_undegraded":"2024-03-18T19:06:05.175438+0000","last_fullsized":"2024-03-18T19:06:05.175438+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:21.448421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174311+0000","last_change":"2024-03-18T19:06:05.161117+0000","last_active":"2024-03-18T19:06:05.174311+0000","last_peered":"2024-03-18T19:06:05.174311+0000","last_clean":"2024-03-18T19:06:05.174311+0000","last_became_active":"2024-03-18T19:06:05.160450+0000","last_became_peered":"2024-03-18T19:06:05.160450+0000","last_unstale":"2024-03-18T19:06:05.174311+0000","last_undegraded":"2024-03-18T19:06:05.174311+0000","last_fullsized":"2024-03-18T19:06:05.174311+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:02:26.476481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"102'6","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157412+0000","last_change":"2024-03-18T19:06:04.735749+0000","last_active":"2024-03-18T19:06:06.157412+0000","last_peered":"2024-03-18T19:06:06.157412+0000","last_clean":"2024-03-18T19:06:06.157412+0000","last_became_active":"2024-03-18T19:06:04.735436+0000","last_became_peered":"2024-03-18T19:06:04.735436+0000","last_unstale":"2024-03-18T19:06:06.157412+0000","last_undegraded":"2024-03-18T19:06:06.157412+0000","last_fullsized":"2024-03-18T19:06:06.157412+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:14:28.241082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.178875+0000","last_change":"2024-03-18T19:06:05.178346+0000","last_active":"2024-03-18T19:06:05.178875+0000","last_peered":"2024-03-18T19:06:05.178875+0000","last_clean":"2024-03-18T19:06:05.178875+0000","last_became_active":"2024-03-18T19:06:05.178118+0000","last_became_peered":"2024-03-18T19:06:05.178118+0000","last_unstale":"2024-03-18T19:06:05.178875+0000","last_undegraded":"2024-03-18T19:06:05.178875+0000","last_fullsized":"2024-03-18T19:06:05.178875+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:54:09.212832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.172713+0000","last_change":"2024-03-18T19:06:04.730583+0000","last_active":"2024-03-18T19:06:05.172713+0000","last_peered":"2024-03-18T19:06:05.172713+0000","last_clean":"2024-03-18T19:06:05.172713+0000","last_became_active":"2024-03-18T19:06:04.730263+0000","last_became_peered":"2024-03-18T19:06:04.730263+0000","last_unstale":"2024-03-18T19:06:05.172713+0000","last_undegraded":"2024-03-18T19:06:05.172713+0000","last_fullsized":"2024-03-18T19:06:05.172713+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:45:27.066729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157115+0000","last_change":"2024-03-18T19:06:05.166464+0000","last_active":"2024-03-18T19:06:06.157115+0000","last_peered":"2024-03-18T19:06:06.157115+0000","last_clean":"2024-03-18T19:06:06.157115+0000","last_became_active":"2024-03-18T19:06:05.166188+0000","last_became_peered":"2024-03-18T19:06:05.166188+0000","last_unstale":"2024-03-18T19:06:06.157115+0000","last_undegraded":"2024-03-18T19:06:06.157115+0000","last_fullsized":"2024-03-18T19:06:06.157115+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:36:28.282894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.153576+0000","last_change":"2024-03-18T19:06:04.725490+0000","last_active":"2024-03-18T19:06:05.153576+0000","last_peered":"2024-03-18T19:06:05.153576+0000","last_clean":"2024-03-18T19:06:05.153576+0000","last_became_active":"2024-03-18T19:06:04.725239+0000","last_became_peered":"2024-03-18T19:06:04.725239+0000","last_unstale":"2024-03-18T19:06:05.153576+0000","last_undegraded":"2024-03-18T19:06:05.153576+0000","last_fullsized":"2024-03-18T19:06:05.153576+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:30:48.969431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157257+0000","last_change":"2024-03-18T19:06:05.170967+0000","last_active":"2024-03-18T19:06:06.157257+0000","last_peered":"2024-03-18T19:06:06.157257+0000","last_clean":"2024-03-18T19:06:06.157257+0000","last_became_active":"2024-03-18T19:06:05.170700+0000","last_became_peered":"2024-03-18T19:06:05.170700+0000","last_unstale":"2024-03-18T19:06:06.157257+0000","last_undegraded":"2024-03-18T19:06:06.157257+0000","last_fullsized":"2024-03-18T19:06:06.157257+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:30:49.975316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.160933+0000","last_change":"2024-03-18T19:06:04.732660+0000","last_active":"2024-03-18T19:06:05.160933+0000","last_peered":"2024-03-18T19:06:05.160933+0000","last_clean":"2024-03-18T19:06:05.160933+0000","last_became_active":"2024-03-18T19:06:04.732378+0000","last_became_peered":"2024-03-18T19:06:04.732378+0000","last_unstale":"2024-03-18T19:06:05.160933+0000","last_undegraded":"2024-03-18T19:06:05.160933+0000","last_fullsized":"2024-03-18T19:06:05.160933+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:21.180986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157287+0000","last_change":"2024-03-18T19:06:05.176448+0000","last_active":"2024-03-18T19:06:06.157287+0000","last_peered":"2024-03-18T19:06:06.157287+0000","last_clean":"2024-03-18T19:06:06.157287+0000","last_became_active":"2024-03-18T19:06:05.176177+0000","last_became_peered":"2024-03-18T19:06:05.176177+0000","last_unstale":"2024-03-18T19:06:06.157287+0000","last_undegraded":"2024-03-18T19:06:06.157287+0000","last_fullsized":"2024-03-18T19:06:06.157287+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:45:28.072614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"102'8","reported_seq":19,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.156997+0000","last_change":"2024-03-18T19:06:04.737096+0000","last_active":"2024-03-18T19:06:06.156997+0000","last_peered":"2024-03-18T19:06:06.156997+0000","last_clean":"2024-03-18T19:06:06.156997+0000","last_became_active":"2024-03-18T19:06:04.736607+0000","last_became_peered":"2024-03-18T19:06:04.736607+0000","last_unstale":"2024-03-18T19:06:06.156997+0000","last_undegraded":"2024-03-18T19:06:06.156997+0000","last_fullsized":"2024-03-18T19:06:06.156997+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:11.573751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.180235+0000","last_change":"2024-03-18T19:06:05.176988+0000","last_active":"2024-03-18T19:06:05.180235+0000","last_peered":"2024-03-18T19:06:05.180235+0000","last_clean":"2024-03-18T19:06:05.180235+0000","last_became_active":"2024-03-18T19:06:05.176751+0000","last_became_peered":"2024-03-18T19:06:05.176751+0000","last_unstale":"2024-03-18T19:06:05.180235+0000","last_undegraded":"2024-03-18T19:06:05.180235+0000","last_fullsized":"2024-03-18T19:06:05.180235+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:37:35.397401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.167391+0000","last_change":"2024-03-18T19:06:04.729993+0000","last_active":"2024-03-18T19:06:05.167391+0000","last_peered":"2024-03-18T19:06:05.167391+0000","last_clean":"2024-03-18T19:06:05.167391+0000","last_became_active":"2024-03-18T19:06:04.729697+0000","last_became_peered":"2024-03-18T19:06:04.729697+0000","last_unstale":"2024-03-18T19:06:05.167391+0000","last_undegraded":"2024-03-18T19:06:05.167391+0000","last_fullsized":"2024-03-18T19:06:05.167391+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:38:45.615401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174395+0000","last_change":"2024-03-18T19:06:05.169243+0000","last_active":"2024-03-18T19:06:05.174395+0000","last_peered":"2024-03-18T19:06:05.174395+0000","last_clean":"2024-03-18T19:06:05.174395+0000","last_became_active":"2024-03-18T19:06:05.168996+0000","last_became_peered":"2024-03-18T19:06:05.168996+0000","last_unstale":"2024-03-18T19:06:05.174395+0000","last_undegraded":"2024-03-18T19:06:05.174395+0000","last_fullsized":"2024-03-18T19:06:05.174395+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:02.904163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.175112+0000","last_change":"2024-03-18T19:06:04.733348+0000","last_active":"2024-03-18T19:06:05.175112+0000","last_peered":"2024-03-18T19:06:05.175112+0000","last_clean":"2024-03-18T19:06:05.175112+0000","last_became_active":"2024-03-18T19:06:04.733087+0000","last_became_peered":"2024-03-18T19:06:04.733087+0000","last_unstale":"2024-03-18T19:06:05.175112+0000","last_undegraded":"2024-03-18T19:06:05.175112+0000","last_fullsized":"2024-03-18T19:06:05.175112+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:16:54.958882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174519+0000","last_change":"2024-03-18T19:06:05.167597+0000","last_active":"2024-03-18T19:06:05.174519+0000","last_peered":"2024-03-18T19:06:05.174519+0000","last_clean":"2024-03-18T19:06:05.174519+0000","last_became_active":"2024-03-18T19:06:05.167373+0000","last_became_peered":"2024-03-18T19:06:05.167373+0000","last_unstale":"2024-03-18T19:06:05.174519+0000","last_undegraded":"2024-03-18T19:06:05.174519+0000","last_fullsized":"2024-03-18T19:06:05.174519+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:51:11.010645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.172218+0000","last_change":"2024-03-18T19:06:04.733453+0000","last_active":"2024-03-18T19:06:05.172218+0000","last_peered":"2024-03-18T19:06:05.172218+0000","last_clean":"2024-03-18T19:06:05.172218+0000","last_became_active":"2024-03-18T19:06:04.732952+0000","last_became_peered":"2024-03-18T19:06:04.732952+0000","last_unstale":"2024-03-18T19:06:05.172218+0000","last_undegraded":"2024-03-18T19:06:05.172218+0000","last_fullsized":"2024-03-18T19:06:05.172218+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:52:54.267278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.180800+0000","last_change":"2024-03-18T19:06:05.176390+0000","last_active":"2024-03-18T19:06:05.180800+0000","last_peered":"2024-03-18T19:06:05.180800+0000","last_clean":"2024-03-18T19:06:05.180800+0000","last_became_active":"2024-03-18T19:06:05.176107+0000","last_became_peered":"2024-03-18T19:06:05.176107+0000","last_unstale":"2024-03-18T19:06:05.180800+0000","last_undegraded":"2024-03-18T19:06:05.180800+0000","last_fullsized":"2024-03-18T19:06:05.180800+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:57:28.462884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.157269+0000","last_change":"2024-03-18T19:06:04.729070+0000","last_active":"2024-03-18T19:06:05.157269+0000","last_peered":"2024-03-18T19:06:05.157269+0000","last_clean":"2024-03-18T19:06:05.157269+0000","last_became_active":"2024-03-18T19:06:04.728830+0000","last_became_peered":"2024-03-18T19:06:04.728830+0000","last_unstale":"2024-03-18T19:06:05.157269+0000","last_undegraded":"2024-03-18T19:06:05.157269+0000","last_fullsized":"2024-03-18T19:06:05.157269+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:40:51.689786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157235+0000","last_change":"2024-03-18T19:06:05.173107+0000","last_active":"2024-03-18T19:06:06.157235+0000","last_peered":"2024-03-18T19:06:06.157235+0000","last_clean":"2024-03-18T19:06:06.157235+0000","last_became_active":"2024-03-18T19:06:05.172838+0000","last_became_peered":"2024-03-18T19:06:05.172838+0000","last_unstale":"2024-03-18T19:06:06.157235+0000","last_undegraded":"2024-03-18T19:06:06.157235+0000","last_fullsized":"2024-03-18T19:06:06.157235+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:06.313685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.180996+0000","last_change":"2024-03-18T19:06:04.735737+0000","last_active":"2024-03-18T19:06:05.180996+0000","last_peered":"2024-03-18T19:06:05.180996+0000","last_clean":"2024-03-18T19:06:05.180996+0000","last_became_active":"2024-03-18T19:06:04.735358+0000","last_became_peered":"2024-03-18T19:06:04.735358+0000","last_unstale":"2024-03-18T19:06:05.180996+0000","last_undegraded":"2024-03-18T19:06:05.180996+0000","last_fullsized":"2024-03-18T19:06:05.180996+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:20:45.425752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157176+0000","last_change":"2024-03-18T19:06:05.171908+0000","last_active":"2024-03-18T19:06:06.157176+0000","last_peered":"2024-03-18T19:06:06.157176+0000","last_clean":"2024-03-18T19:06:06.157176+0000","last_became_active":"2024-03-18T19:06:05.171669+0000","last_became_peered":"2024-03-18T19:06:05.171669+0000","last_unstale":"2024-03-18T19:06:06.157176+0000","last_undegraded":"2024-03-18T19:06:06.157176+0000","last_fullsized":"2024-03-18T19:06:06.157176+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:45.511948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.161418+0000","last_change":"2024-03-18T19:06:04.736438+0000","last_active":"2024-03-18T19:06:05.161418+0000","last_peered":"2024-03-18T19:06:05.161418+0000","last_clean":"2024-03-18T19:06:05.161418+0000","last_became_active":"2024-03-18T19:06:04.736111+0000","last_became_peered":"2024-03-18T19:06:04.736111+0000","last_unstale":"2024-03-18T19:06:05.161418+0000","last_undegraded":"2024-03-18T19:06:05.161418+0000","last_fullsized":"2024-03-18T19:06:05.161418+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:56.438036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157371+0000","last_change":"2024-03-18T19:06:05.173451+0000","last_active":"2024-03-18T19:06:06.157371+0000","last_peered":"2024-03-18T19:06:06.157371+0000","last_clean":"2024-03-18T19:06:06.157371+0000","last_became_active":"2024-03-18T19:06:05.173171+0000","last_became_peered":"2024-03-18T19:06:05.173171+0000","last_unstale":"2024-03-18T19:06:06.157371+0000","last_undegraded":"2024-03-18T19:06:06.157371+0000","last_fullsized":"2024-03-18T19:06:06.157371+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:40:52.695671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.160706+0000","last_change":"2024-03-18T19:06:04.737213+0000","last_active":"2024-03-18T19:06:05.160706+0000","last_peered":"2024-03-18T19:06:05.160706+0000","last_clean":"2024-03-18T19:06:05.160706+0000","last_became_active":"2024-03-18T19:06:04.736846+0000","last_became_peered":"2024-03-18T19:06:04.736846+0000","last_unstale":"2024-03-18T19:06:05.160706+0000","last_undegraded":"2024-03-18T19:06:05.160706+0000","last_fullsized":"2024-03-18T19:06:05.160706+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:00.161012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.173870+0000","last_change":"2024-03-18T19:06:05.170880+0000","last_active":"2024-03-18T19:06:05.173870+0000","last_peered":"2024-03-18T19:06:05.173870+0000","last_clean":"2024-03-18T19:06:05.173870+0000","last_became_active":"2024-03-18T19:06:05.170552+0000","last_became_peered":"2024-03-18T19:06:05.170552+0000","last_unstale":"2024-03-18T19:06:05.173870+0000","last_undegraded":"2024-03-18T19:06:05.173870+0000","last_fullsized":"2024-03-18T19:06:05.173870+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:49:21.908439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174733+0000","last_change":"2024-03-18T19:06:04.733203+0000","last_active":"2024-03-18T19:06:05.174733+0000","last_peered":"2024-03-18T19:06:05.174733+0000","last_clean":"2024-03-18T19:06:05.174733+0000","last_became_active":"2024-03-18T19:06:04.732922+0000","last_became_peered":"2024-03-18T19:06:04.732922+0000","last_unstale":"2024-03-18T19:06:05.174733+0000","last_undegraded":"2024-03-18T19:06:05.174733+0000","last_fullsized":"2024-03-18T19:06:05.174733+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:40:51.689786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.156881+0000","last_change":"2024-03-18T19:06:05.165633+0000","last_active":"2024-03-18T19:06:06.156881+0000","last_peered":"2024-03-18T19:06:06.156881+0000","last_clean":"2024-03-18T19:06:06.156881+0000","last_became_active":"2024-03-18T19:06:05.165174+0000","last_became_peered":"2024-03-18T19:06:05.165174+0000","last_unstale":"2024-03-18T19:06:06.156881+0000","last_undegraded":"2024-03-18T19:06:06.156881+0000","last_fullsized":"2024-03-18T19:06:06.156881+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:49:21.908439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157203+0000","last_change":"2024-03-18T19:06:04.727295+0000","last_active":"2024-03-18T19:06:06.157203+0000","last_peered":"2024-03-18T19:06:06.157203+0000","last_clean":"2024-03-18T19:06:06.157203+0000","last_became_active":"2024-03-18T19:06:04.726972+0000","last_became_peered":"2024-03-18T19:06:04.726972+0000","last_unstale":"2024-03-18T19:06:06.157203+0000","last_undegraded":"2024-03-18T19:06:06.157203+0000","last_fullsized":"2024-03-18T19:06:06.157203+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:41.776279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.184217+0000","last_change":"2024-03-18T19:06:05.183179+0000","last_active":"2024-03-18T19:06:05.184217+0000","last_peered":"2024-03-18T19:06:05.184217+0000","last_clean":"2024-03-18T19:06:05.184217+0000","last_became_active":"2024-03-18T19:06:05.182880+0000","last_became_peered":"2024-03-18T19:06:05.182880+0000","last_unstale":"2024-03-18T19:06:05.184217+0000","last_undegraded":"2024-03-18T19:06:05.184217+0000","last_fullsized":"2024-03-18T19:06:05.184217+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:30:49.975316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.154414+0000","last_change":"2024-03-18T19:06:04.732118+0000","last_active":"2024-03-18T19:06:05.154414+0000","last_peered":"2024-03-18T19:06:05.154414+0000","last_clean":"2024-03-18T19:06:05.154414+0000","last_became_active":"2024-03-18T19:06:04.731849+0000","last_became_peered":"2024-03-18T19:06:04.731849+0000","last_unstale":"2024-03-18T19:06:05.154414+0000","last_undegraded":"2024-03-18T19:06:05.154414+0000","last_fullsized":"2024-03-18T19:06:05.154414+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:09:39.307084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157240+0000","last_change":"2024-03-18T19:06:05.163264+0000","last_active":"2024-03-18T19:06:06.157240+0000","last_peered":"2024-03-18T19:06:06.157240+0000","last_clean":"2024-03-18T19:06:06.157240+0000","last_became_active":"2024-03-18T19:06:05.162967+0000","last_became_peered":"2024-03-18T19:06:05.162967+0000","last_unstale":"2024-03-18T19:06:06.157240+0000","last_undegraded":"2024-03-18T19:06:06.157240+0000","last_fullsized":"2024-03-18T19:06:06.157240+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:40:08.716187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.153804+0000","last_change":"2024-03-18T19:06:04.721220+0000","last_active":"2024-03-18T19:06:05.153804+0000","last_peered":"2024-03-18T19:06:05.153804+0000","last_clean":"2024-03-18T19:06:05.153804+0000","last_became_active":"2024-03-18T19:06:04.720956+0000","last_became_peered":"2024-03-18T19:06:04.720956+0000","last_unstale":"2024-03-18T19:06:05.153804+0000","last_undegraded":"2024-03-18T19:06:05.153804+0000","last_fullsized":"2024-03-18T19:06:05.153804+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:40:51.689786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157118+0000","last_change":"2024-03-18T19:06:05.174203+0000","last_active":"2024-03-18T19:06:06.157118+0000","last_peered":"2024-03-18T19:06:06.157118+0000","last_clean":"2024-03-18T19:06:06.157118+0000","last_became_active":"2024-03-18T19:06:05.173938+0000","last_became_peered":"2024-03-18T19:06:05.173938+0000","last_unstale":"2024-03-18T19:06:06.157118+0000","last_undegraded":"2024-03-18T19:06:06.157118+0000","last_fullsized":"2024-03-18T19:06:06.157118+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:24:32.554218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.156912+0000","last_change":"2024-03-18T19:06:04.729239+0000","last_active":"2024-03-18T19:06:06.156912+0000","last_peered":"2024-03-18T19:06:06.156912+0000","last_clean":"2024-03-18T19:06:06.156912+0000","last_became_active":"2024-03-18T19:06:04.726857+0000","last_became_peered":"2024-03-18T19:06:04.726857+0000","last_unstale":"2024-03-18T19:06:06.156912+0000","last_undegraded":"2024-03-18T19:06:06.156912+0000","last_fullsized":"2024-03-18T19:06:06.156912+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:11:44.520088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157156+0000","last_change":"2024-03-18T19:06:05.168513+0000","last_active":"2024-03-18T19:06:06.157156+0000","last_peered":"2024-03-18T19:06:06.157156+0000","last_clean":"2024-03-18T19:06:06.157156+0000","last_became_active":"2024-03-18T19:06:05.168222+0000","last_became_peered":"2024-03-18T19:06:05.168222+0000","last_unstale":"2024-03-18T19:06:06.157156+0000","last_undegraded":"2024-03-18T19:06:06.157156+0000","last_fullsized":"2024-03-18T19:06:06.157156+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:38:46.621286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.161325+0000","last_change":"2024-03-18T19:06:04.736364+0000","last_active":"2024-03-18T19:06:05.161325+0000","last_peered":"2024-03-18T19:06:05.161325+0000","last_clean":"2024-03-18T19:06:05.161325+0000","last_became_active":"2024-03-18T19:06:04.736105+0000","last_became_peered":"2024-03-18T19:06:04.736105+0000","last_unstale":"2024-03-18T19:06:05.161325+0000","last_undegraded":"2024-03-18T19:06:05.161325+0000","last_fullsized":"2024-03-18T19:06:05.161325+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:08:47.425685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.175431+0000","last_change":"2024-03-18T19:06:05.173562+0000","last_active":"2024-03-18T19:06:05.175431+0000","last_peered":"2024-03-18T19:06:05.175431+0000","last_clean":"2024-03-18T19:06:05.175431+0000","last_became_active":"2024-03-18T19:06:05.171955+0000","last_became_peered":"2024-03-18T19:06:05.171955+0000","last_unstale":"2024-03-18T19:06:05.175431+0000","last_undegraded":"2024-03-18T19:06:05.175431+0000","last_fullsized":"2024-03-18T19:06:05.175431+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:34.217833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.156959+0000","last_change":"2024-03-18T19:06:04.732342+0000","last_active":"2024-03-18T19:06:06.156959+0000","last_peered":"2024-03-18T19:06:06.156959+0000","last_clean":"2024-03-18T19:06:06.156959+0000","last_became_active":"2024-03-18T19:06:04.732038+0000","last_became_peered":"2024-03-18T19:06:04.732038+0000","last_unstale":"2024-03-18T19:06:06.156959+0000","last_undegraded":"2024-03-18T19:06:06.156959+0000","last_fullsized":"2024-03-18T19:06:06.156959+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:38.050395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.173981+0000","last_change":"2024-03-18T19:06:05.164553+0000","last_active":"2024-03-18T19:06:05.173981+0000","last_peered":"2024-03-18T19:06:05.173981+0000","last_clean":"2024-03-18T19:06:05.173981+0000","last_became_active":"2024-03-18T19:06:05.164350+0000","last_became_peered":"2024-03-18T19:06:05.164350+0000","last_unstale":"2024-03-18T19:06:05.173981+0000","last_undegraded":"2024-03-18T19:06:05.173981+0000","last_fullsized":"2024-03-18T19:06:05.173981+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:07:30.773855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.157251+0000","last_change":"2024-03-18T19:06:04.736220+0000","last_active":"2024-03-18T19:06:05.157251+0000","last_peered":"2024-03-18T19:06:05.157251+0000","last_clean":"2024-03-18T19:06:05.157251+0000","last_became_active":"2024-03-18T19:06:04.735922+0000","last_became_peered":"2024-03-18T19:06:04.735922+0000","last_unstale":"2024-03-18T19:06:05.157251+0000","last_undegraded":"2024-03-18T19:06:05.157251+0000","last_fullsized":"2024-03-18T19:06:05.157251+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:34:07.772319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.168878+0000","last_change":"2024-03-18T19:06:05.167624+0000","last_active":"2024-03-18T19:06:05.168878+0000","last_peered":"2024-03-18T19:06:05.168878+0000","last_clean":"2024-03-18T19:06:05.168878+0000","last_became_active":"2024-03-18T19:06:05.167327+0000","last_became_peered":"2024-03-18T19:06:05.167327+0000","last_unstale":"2024-03-18T19:06:05.168878+0000","last_undegraded":"2024-03-18T19:06:05.168878+0000","last_fullsized":"2024-03-18T19:06:05.168878+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:11:36.194047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.170373+0000","last_change":"2024-03-18T19:06:04.737285+0000","last_active":"2024-03-18T19:06:05.170373+0000","last_peered":"2024-03-18T19:06:05.170373+0000","last_clean":"2024-03-18T19:06:05.170373+0000","last_became_active":"2024-03-18T19:06:04.736911+0000","last_became_peered":"2024-03-18T19:06:04.736911+0000","last_unstale":"2024-03-18T19:06:05.170373+0000","last_undegraded":"2024-03-18T19:06:05.170373+0000","last_fullsized":"2024-03-18T19:06:05.170373+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:27.449998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"92'172","reported_seq":436,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-18T19:06:04.715296+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T19:06:04.715296+0000","last_peered":"2024-03-18T19:06:04.715296+0000","last_clean":"2024-03-18T19:06:04.715296+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T19:06:04.715296+0000","last_undegraded":"2024-03-18T19:06:04.715296+0000","last_fullsized":"2024-03-18T19:06:04.715296+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172,"ondisk_log_size":172,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575042,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59168,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312672,"statfs":{"total":96636764160,"available":96576176128,"internally_reserved":0,"allocated":1847296,"data_stored":1213911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.55900000000000005,"15min":0.57099999999999995},"min":{"1min":0.39400000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.64100000000000001,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.56000000000000005,"15min":0.58199999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68700000000000006,"5min":1.387,"15min":1.387},"last":0.47099999999999997}]},{"osd":1,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.54000000000000004,"15min":0.57499999999999996},"min":{"1min":0.36499999999999999,"5min":0.32300000000000001,"15min":0.307},"max":{"1min":0.66200000000000003,"5min":0.81699999999999995,"15min":1.484},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.56499999999999995,"15min":0.58899999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.65600000000000003,"5min":0.80700000000000005,"15min":0.92000000000000004},"last":0.53100000000000003}]},{"osd":2,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55900000000000005,"15min":0.57799999999999996},"min":{"1min":0.42199999999999999,"5min":0.34499999999999997,"15min":0.27800000000000002},"max":{"1min":0.69799999999999995,"5min":1.3600000000000001,"15min":1.5129999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.59099999999999997},"min":{"1min":0.38300000000000001,"5min":0.36899999999999999,"15min":0.30199999999999999},"max":{"1min":0.73699999999999999,"5min":1.327,"15min":1.536},"last":0.44900000000000001}]},{"osd":3,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55200000000000005,"15min":0.57999999999999996},"min":{"1min":0.35599999999999998,"5min":0.318,"15min":0.311},"max":{"1min":0.61699999999999999,"5min":0.79100000000000004,"15min":1.5489999999999999},"last":0.5},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.57599999999999996,"15min":0.60099999999999998},"min":{"1min":0.40600000000000003,"5min":0.38800000000000001,"15min":0.38200000000000001},"max":{"1min":0.64900000000000002,"5min":0.78200000000000003,"15min":1.5660000000000001},"last":0.52000000000000002}]},{"osd":4,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.45400000000000001},"min":{"1min":0.311,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.59299999999999997,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.46999999999999997,"15min":0.47899999999999998},"min":{"1min":0.316,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.73199999999999998},"last":0.41699999999999998}]},{"osd":5,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.47799999999999998,"15min":0.47999999999999998},"min":{"1min":0.34200000000000003,"5min":0.27100000000000002,"15min":0.26000000000000001},"max":{"1min":0.71499999999999997,"5min":0.751,"15min":0.78300000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.504,"15min":0.51800000000000002},"min":{"1min":0.35699999999999998,"5min":0.30299999999999999,"15min":0.28799999999999998},"max":{"1min":0.751,"5min":0.78200000000000003,"15min":0.80300000000000005},"last":0.379}]},{"osd":6,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53500000000000003,"15min":0.54600000000000004},"min":{"1min":0.38600000000000001,"5min":0.374,"15min":0.34300000000000003},"max":{"1min":0.67700000000000005,"5min":0.81799999999999995,"15min":0.84499999999999997},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.35899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68700000000000006,"5min":0.752,"15min":0.77500000000000002},"last":0.46300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1273856,"data_stored":631705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56899999999999995,"15min":0.57199999999999995},"min":{"1min":0.33000000000000002,"5min":0.28499999999999998,"15min":0.25700000000000001},"max":{"1min":0.69799999999999995,"5min":0.75600000000000001,"15min":3.274},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56299999999999994,"15min":0.58099999999999996},"min":{"1min":0.37,"5min":0.34699999999999998,"15min":0.27400000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":3.3929999999999998},"last":0.47799999999999998}]},{"osd":1,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57699999999999996,"15min":0.56799999999999995},"min":{"1min":0.38100000000000001,"5min":0.316,"15min":0.30599999999999999},"max":{"1min":0.73899999999999999,"5min":0.77400000000000002,"15min":0.94099999999999995},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56200000000000006,"15min":0.56999999999999995},"min":{"1min":0.42699999999999999,"5min":0.32000000000000001,"15min":0.311},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.90700000000000003},"last":0.66200000000000003}]},{"osd":2,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58399999999999996,"15min":0.57599999999999996},"min":{"1min":0.42799999999999999,"5min":0.32700000000000001,"15min":0.29999999999999999},"max":{"1min":0.75700000000000001,"5min":0.80300000000000005,"15min":0.81299999999999994},"last":0.625},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57199999999999995,"15min":0.58299999999999996},"min":{"1min":0.36699999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.74199999999999999,"5min":0.83199999999999996,"15min":2.2370000000000001},"last":0.60999999999999999}]},{"osd":3,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59199999999999997,"15min":0.58499999999999996},"min":{"1min":0.41199999999999998,"5min":0.40000000000000002,"15min":0.32100000000000001},"max":{"1min":0.75600000000000001,"5min":0.89600000000000002,"15min":1.0369999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.373,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":1.595},"last":0.63900000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.34000000000000002,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.76300000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47599999999999998,"15min":0.48099999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":1.8819999999999999},"last":0.67500000000000004}]},{"osd":5,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.46899999999999997,"15min":0.47399999999999998},"min":{"1min":0.33600000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":1.022},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51200000000000001,"15min":0.51600000000000001},"min":{"1min":0.35699999999999998,"5min":0.35499999999999998,"15min":0.32200000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.81599999999999995},"last":0.72099999999999997}]},{"osd":7,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.52700000000000002,"15min":0.52600000000000002},"min":{"1min":0.34300000000000003,"5min":0.34100000000000003,"15min":0.313},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":1.532},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53000000000000003,"15min":0.53300000000000003},"min":{"1min":0.39500000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":1.978},"last":0.73399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36448,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335392,"statfs":{"total":96636764160,"available":96599441408,"internally_reserved":0,"allocated":1257472,"data_stored":623633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.375,"15min":0.373},"min":{"1min":0.23200000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.83599999999999997},"last":0.217},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42499999999999999,"15min":0.41499999999999998},"min":{"1min":0.25700000000000001,"5min":0.23799999999999999,"15min":0.23400000000000001},"max":{"1min":0.65200000000000002,"5min":0.73699999999999999,"15min":0.90400000000000003},"last":0.27800000000000002}]},{"osd":2,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41599999999999998,"15min":0.40400000000000003},"min":{"1min":0.27800000000000002,"5min":0.26500000000000001,"15min":0.23300000000000001},"max":{"1min":0.56699999999999995,"5min":0.65400000000000003,"15min":0.88400000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.45300000000000001,"15min":0.44500000000000001},"min":{"1min":0.34300000000000003,"5min":0.29699999999999999,"15min":0.23499999999999999},"max":{"1min":0.60699999999999998,"5min":0.68700000000000006,"15min":0.94999999999999996},"last":0.317}]},{"osd":3,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.27700000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.86799999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.42799999999999999,"15min":0.42099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28100000000000003,"15min":0.25600000000000001},"max":{"1min":0.65100000000000002,"5min":0.77000000000000002,"15min":0.93100000000000005},"last":0.23699999999999999}]},{"osd":4,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63800000000000001,"15min":0.63300000000000001},"min":{"1min":0.33400000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.92300000000000004,"5min":0.93999999999999995,"15min":1.052},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.63800000000000001,"15min":0.64400000000000002},"min":{"1min":0.46800000000000003,"5min":0.36899999999999999,"15min":0.34499999999999997},"max":{"1min":0.95799999999999996,"5min":1.018,"15min":1.0269999999999999},"last":0.52400000000000002}]},{"osd":5,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63300000000000001,"15min":0.64200000000000002},"min":{"1min":0.434,"5min":0.40100000000000002,"15min":0.38600000000000001},"max":{"1min":0.877,"5min":0.88500000000000001,"15min":1.0760000000000001},"last":0.504},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.65600000000000003,"15min":0.65700000000000003},"min":{"1min":0.47099999999999997,"5min":0.41099999999999998,"15min":0.40999999999999998},"max":{"1min":0.80900000000000005,"5min":1.006,"15min":1.1319999999999999},"last":0.59899999999999998}]},{"osd":6,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.93500000000000005,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67500000000000004,"15min":0.66900000000000004},"min":{"1min":0.434,"5min":0.42599999999999999,"15min":0.39100000000000001},"max":{"1min":0.93799999999999994,"5min":0.96799999999999997,"15min":1.0089999999999999},"last":0.628}]},{"osd":7,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66000000000000003},"min":{"1min":0.48599999999999999,"5min":0.44900000000000001,"15min":0.36699999999999999},"max":{"1min":0.90600000000000003,"5min":0.92900000000000005,"15min":1.0660000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66700000000000004,"15min":0.66400000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.39900000000000002},"max":{"1min":0.88100000000000001,"5min":0.94299999999999995,"15min":1.1499999999999999},"last":0.55100000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575043,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37020,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334820,"statfs":{"total":96636764160,"available":96598855680,"internally_reserved":0,"allocated":1843200,"data_stored":1213889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.377,"15min":0.377},"min":{"1min":0.23100000000000001,"5min":0.19400000000000001,"15min":0.192},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.621},"last":0.31},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.30299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.55400000000000005,"5min":0.61699999999999999,"15min":0.66400000000000003},"last":0.26900000000000002}]},{"osd":2,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41599999999999998,"15min":0.41699999999999998},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.58999999999999997,"5min":0.60499999999999998,"15min":0.627},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.30499999999999999,"5min":0.25,"15min":0.25},"max":{"1min":0.627,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.32600000000000001}]},{"osd":3,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.40699999999999997,"15min":0.42999999999999999},"min":{"1min":0.30299999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.57499999999999996,"5min":0.61299999999999999,"15min":5.7359999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46500000000000002,"15min":0.48799999999999999},"min":{"1min":0.32000000000000001,"5min":0.28799999999999998,"15min":0.27700000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":5.9329999999999998},"last":0.33900000000000002}]},{"osd":4,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.63800000000000001,"15min":0.63900000000000001},"min":{"1min":0.49099999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.91800000000000004,"5min":1.0229999999999999,"15min":1.22},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.45900000000000002,"5min":0.315,"15min":0.315},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.1020000000000001},"last":0.68200000000000005}]},{"osd":5,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.47999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.96199999999999997,"5min":0.98099999999999998,"15min":2.9649999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.65600000000000003,"15min":0.65800000000000003},"min":{"1min":0.38,"5min":0.31,"15min":0.31},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.2609999999999999},"last":0.70699999999999996}]},{"osd":6,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.64900000000000002,"15min":0.65300000000000002},"min":{"1min":0.40500000000000003,"5min":0.39400000000000002,"15min":0.38100000000000001},"max":{"1min":0.90100000000000002,"5min":0.94599999999999995,"15min":1.133},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.66500000000000004,"15min":0.67600000000000005},"min":{"1min":0.42699999999999999,"5min":0.41199999999999998,"15min":0.40999999999999998},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":2.9300000000000002},"last":0.64000000000000001}]},{"osd":7,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.66600000000000004,"15min":0.66700000000000004},"min":{"1min":0.51900000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.2849999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.66500000000000004,"15min":0.67000000000000004},"min":{"1min":0.44,"5min":0.437,"15min":0.38500000000000001},"max":{"1min":0.94299999999999995,"5min":0.96299999999999997,"15min":1.002},"last":0.53300000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575043,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58596,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94313244,"statfs":{"total":96636764160,"available":96576761856,"internally_reserved":0,"allocated":1261568,"data_stored":624263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38200000000000001,"15min":0.39500000000000002},"min":{"1min":0.249,"5min":0.216,"15min":0.17799999999999999},"max":{"1min":0.66600000000000004,"5min":0.78300000000000003,"15min":9.1219999999999999},"last":0.252},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.46500000000000002,"15min":0.47599999999999998},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.68899999999999995,"5min":0.77200000000000002,"15min":9.1560000000000006},"last":0.41599999999999998}]},{"osd":1,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47999999999999998,"15min":0.49299999999999999},"min":{"1min":0.376,"5min":0.29399999999999998,"15min":0.27100000000000002},"max":{"1min":0.69499999999999995,"5min":0.90500000000000003,"15min":9.2460000000000004},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41299999999999998,"15min":0.42499999999999999},"min":{"1min":0.23899999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.61399999999999999,"5min":0.66700000000000004,"15min":12.932},"last":0.33100000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.42199999999999999,"15min":0.42799999999999999},"min":{"1min":0.29799999999999999,"5min":0.24199999999999999,"15min":0.22500000000000001},"max":{"1min":0.58599999999999997,"5min":1.4019999999999999,"15min":9.4120000000000008},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51300000000000001,"15min":0.52000000000000002},"min":{"1min":0.33800000000000002,"5min":0.29499999999999998,"15min":0.27700000000000002},"max":{"1min":0.67100000000000004,"5min":0.80100000000000005,"15min":9.2680000000000007},"last":0.38300000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63,"15min":0.61799999999999999},"min":{"1min":0.39400000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.98199999999999998,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63400000000000001,"15min":0.65200000000000002},"min":{"1min":0.441,"5min":0.35899999999999999,"15min":0.32100000000000001},"max":{"1min":0.874,"5min":1.4730000000000001,"15min":9.3190000000000008},"last":0.51400000000000001}]},{"osd":5,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65300000000000002,"15min":0.66800000000000004},"min":{"1min":0.45300000000000001,"5min":0.34699999999999998,"15min":0.309},"max":{"1min":0.89600000000000002,"5min":1.8109999999999999,"15min":9.1940000000000008},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65800000000000003,"15min":0.64000000000000001},"min":{"1min":0.40000000000000002,"5min":0.36799999999999999,"15min":0.33400000000000002},"max":{"1min":0.90300000000000002,"5min":1.948,"15min":1.948},"last":0.48999999999999999}]},{"osd":6,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.67700000000000005,"15min":0.67900000000000005},"min":{"1min":0.40999999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.92300000000000004,"5min":2.9470000000000001,"15min":9.3559999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69299999999999995,"15min":0.70899999999999996},"min":{"1min":0.48899999999999999,"5min":0.38100000000000001,"15min":0.37},"max":{"1min":1.0509999999999999,"5min":1.083,"15min":9.2219999999999995},"last":0.624}]},{"osd":7,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.66300000000000003,"15min":0.65200000000000002},"min":{"1min":0.42599999999999999,"5min":0.40999999999999998,"15min":0.38200000000000001},"max":{"1min":0.93600000000000005,"5min":1.083,"15min":1.083},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66800000000000004,"15min":0.68400000000000005},"min":{"1min":0.51800000000000002,"5min":0.371,"15min":0.34599999999999997},"max":{"1min":1.032,"5min":1.4350000000000001,"15min":9.3829999999999991},"last":0.61199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575043,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36444,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335396,"statfs":{"total":96636764160,"available":96599445504,"internally_reserved":0,"allocated":1253376,"data_stored":624103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.36899999999999999,"15min":0.379},"min":{"1min":0.23699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.51200000000000001,"5min":0.81100000000000005,"15min":1.204},"last":0.221},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.40400000000000003,"15min":0.39600000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.23000000000000001},"max":{"1min":0.57499999999999996,"5min":1.137,"15min":1.137},"last":0.32500000000000001}]},{"osd":1,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40000000000000002,"15min":0.39700000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.221},"max":{"1min":0.621,"5min":0.999,"15min":1.1220000000000001},"last":0.248},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.437,"15min":0.443},"min":{"1min":0.29999999999999999,"5min":0.28499999999999998,"15min":0.26500000000000001},"max":{"1min":0.51900000000000002,"5min":1.093,"15min":1.1080000000000001},"last":0.26300000000000001}]},{"osd":2,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.318,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.67500000000000004,"5min":1.0369999999999999,"15min":1.1879999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.46300000000000002,"15min":0.47499999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.28599999999999998},"max":{"1min":0.55600000000000005,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.34100000000000003}]},{"osd":4,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.052,"5min":1.4219999999999999,"15min":1.871},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.61199999999999999,"15min":0.63},"min":{"1min":0.42699999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78300000000000003,"5min":1.0700000000000001,"15min":1.764},"last":0.433}]},{"osd":5,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.65600000000000003,"15min":0.65900000000000003},"min":{"1min":0.44400000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.0249999999999999,"5min":2.5329999999999999,"15min":2.5329999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67000000000000004,"15min":0.68400000000000005},"min":{"1min":0.47099999999999997,"5min":0.45500000000000002,"15min":0.437},"max":{"1min":0.94599999999999995,"5min":1.5409999999999999,"15min":1.794},"last":0.55000000000000004}]},{"osd":6,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.66700000000000004,"15min":0.67200000000000004},"min":{"1min":0.44400000000000001,"5min":0.41499999999999998,"15min":0.40300000000000002},"max":{"1min":1.1060000000000001,"5min":2.7389999999999999,"15min":2.7389999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.66100000000000003,"15min":0.67500000000000004},"min":{"1min":0.47799999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96699999999999997,"5min":1.159,"15min":1.8069999999999999},"last":0.54100000000000004}]},{"osd":7,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.64400000000000002,"15min":0.66900000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86499999999999999,"5min":1.3959999999999999,"15min":1.732},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.66800000000000004,"15min":0.67000000000000004},"min":{"1min":0.47399999999999998,"5min":0.439,"15min":0.40500000000000003},"max":{"1min":1.085,"5min":2.7050000000000001,"15min":2.7050000000000001},"last":0.51700000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575043,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32364,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339476,"statfs":{"total":96636764160,"available":96603623424,"internally_reserved":0,"allocated":1269760,"data_stored":625189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.246},"max":{"1min":0.73299999999999998,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56399999999999995,"15min":0.58099999999999996},"min":{"1min":0.40300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.76100000000000001,"5min":0.77600000000000002,"15min":1.0860000000000001},"last":0.42599999999999999}]},{"osd":1,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57399999999999995,"15min":0.58399999999999996},"min":{"1min":0.439,"5min":0.34499999999999997,"15min":0.316},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":1.1679999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55700000000000005,"15min":0.57499999999999996},"min":{"1min":0.41599999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":1.218},"last":0.34899999999999998}]},{"osd":2,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56299999999999994,"15min":0.57299999999999995},"min":{"1min":0.45000000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72699999999999998,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57399999999999995,"15min":0.59899999999999998},"min":{"1min":0.40999999999999998,"5min":0.36799999999999999,"15min":0.32600000000000001},"max":{"1min":0.77200000000000002,"5min":0.78100000000000003,"15min":1.204},"last":0.51100000000000001}]},{"osd":3,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56899999999999995,"15min":0.58599999999999997},"min":{"1min":0.439,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.74099999999999999,"5min":0.75600000000000001,"15min":1.1799999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57799999999999996,"15min":0.59599999999999997},"min":{"1min":0.41799999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67700000000000005,"5min":1.121,"15min":1.121},"last":0.52200000000000002}]},{"osd":5,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.44600000000000001,"15min":0.46500000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.245},"max":{"1min":0.55900000000000005,"5min":0.73499999999999999,"15min":0.78900000000000003},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.435,"5min":0.46100000000000002,"15min":0.48299999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27200000000000002},"max":{"1min":0.626,"5min":0.64400000000000002,"15min":0.77800000000000002},"last":0.499}]},{"osd":6,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.46800000000000003,"15min":0.48499999999999999},"min":{"1min":0.33900000000000002,"5min":0.32700000000000001,"15min":0.29499999999999998},"max":{"1min":0.66500000000000004,"5min":0.68600000000000005,"15min":0.81799999999999995},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.497,"15min":0.52300000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.28999999999999998},"max":{"1min":0.59799999999999998,"5min":0.72999999999999998,"15min":0.88900000000000001},"last":0.53700000000000003}]},{"osd":7,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.498,"15min":0.52100000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.628,"5min":0.70899999999999996,"15min":0.75800000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52600000000000002,"15min":0.55300000000000005},"min":{"1min":0.38300000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70599999999999996,"5min":0.75900000000000001,"15min":0.91900000000000004},"last":0.48599999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50392,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321448,"statfs":{"total":96636764160,"available":96585162752,"internally_reserved":0,"allocated":1253376,"data_stored":623555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56899999999999995,"15min":0.58599999999999997},"min":{"1min":0.38300000000000001,"5min":0.26800000000000002,"15min":0.24399999999999999},"max":{"1min":0.77000000000000002,"5min":0.78400000000000003,"15min":6.3079999999999998},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57899999999999996,"15min":0.57299999999999995},"min":{"1min":0.373,"5min":0.33700000000000002,"15min":0.25800000000000001},"max":{"1min":0.78800000000000003,"5min":0.79800000000000004,"15min":1.208},"last":0.57299999999999995}]},{"osd":1,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.32900000000000001,"5min":0.32600000000000001,"15min":0.28399999999999997},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.29799999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":6.2549999999999999},"last":0.70499999999999996}]},{"osd":2,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.58599999999999997,"15min":0.59699999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.28399999999999997},"max":{"1min":1.153,"5min":1.153,"15min":6.2999999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56499999999999995,"15min":0.623},"min":{"1min":0.39200000000000002,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":0.73999999999999999,"5min":0.748,"15min":17.387},"last":0.56000000000000005}]},{"osd":3,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.35799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":6.3330000000000002},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.212,"5min":1.212,"15min":1.2390000000000001},"last":0.54600000000000004}]},{"osd":4,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.47099999999999997,"15min":0.46800000000000003},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.441},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.497,"15min":0.495},"min":{"1min":0.32700000000000001,"5min":0.29499999999999998,"15min":0.23100000000000001},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.49299999999999999}]},{"osd":6,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.46700000000000003,"15min":0.47899999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.70999999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.504,"15min":0.51000000000000001},"min":{"1min":0.32100000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66300000000000003,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.50900000000000001}]},{"osd":7,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.53100000000000003},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55300000000000005,"15min":0.55200000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.28499999999999998},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.47599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T19:06:09.309 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T19:06:09.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.472+0000 7f978b79a700 1 Processor -- start 2024-03-18T19:06:09.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.472+0000 7f978b79a700 1 -- start start 2024-03-18T19:06:09.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.473+0000 7f978b79a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97840b2f10 0x7f97840cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:09.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.473+0000 7f978b79a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f97840b3370 0x7f97840cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:09.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.473+0000 7f978b79a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978415a480 con 0x7f978415a0a0 2024-03-18T19:06:09.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.473+0000 7f978b79a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978415eb40 con 0x7f97840b3370 2024-03-18T19:06:09.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.473+0000 7f978b79a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978415e5a0 con 0x7f97840b2f10 2024-03-18T19:06:09.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.474+0000 7f9789536700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f978415a0a0 0x7f978415dbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:43458/0 (socket says 172.21.15.1:43458) 2024-03-18T19:06:09.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.474+0000 7f9788d35700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97840b2f10 0x7f97840cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:09.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.474+0000 7f9789d37700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f97840b3370 0x7f97840cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:09.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.474+0000 7f9789536700 1 -- 172.21.15.1:0/2987950855 learned_addr learned my addr 172.21.15.1:0/2987950855 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:09.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.474+0000 7f977bfff700 1 -- 172.21.15.1:0/2987950855 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597131660 0 0) 0x7f978415a480 con 0x7f978415a0a0 2024-03-18T19:06:09.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.474+0000 7f977bfff700 1 -- 172.21.15.1:0/2987950855 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97700035b0 con 0x7f978415a0a0 2024-03-18T19:06:09.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.474+0000 7f9789d37700 1 -- 172.21.15.1:0/2987950855 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97840b2f10 msgr2=0x7f97840cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:09.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.474+0000 7f9789d37700 1 --2- 172.21.15.1:0/2987950855 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97840b2f10 0x7f97840cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.474+0000 7f9789d37700 1 -- 172.21.15.1:0/2987950855 >> v1:172.21.15.165:6789/0 conn(0x7f978415a0a0 legacy=0x7f978415dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:09.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.474+0000 7f9789d37700 1 -- 172.21.15.1:0/2987950855 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978415e860 con 0x7f97840b3370 2024-03-18T19:06:09.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.474+0000 7f9788d35700 1 --2- 172.21.15.1:0/2987950855 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97840b2f10 0x7f97840cfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:06:09.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.474+0000 7f9789d37700 1 --2- 172.21.15.1:0/2987950855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97840b3370 0x7f97840cfff0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f977c009d90 tx=0x7f977c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=434bfbd2d5d4cdf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:09.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.474+0000 7f977bfff700 1 -- 172.21.15.1:0/2987950855 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c00e070 con 0x7f97840b3370 2024-03-18T19:06:09.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.474+0000 7f977bfff700 1 -- 172.21.15.1:0/2987950855 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f977c00bb20 con 0x7f97840b3370 2024-03-18T19:06:09.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.474+0000 7f977bfff700 1 -- 172.21.15.1:0/2987950855 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c012430 con 0x7f97840b3370 2024-03-18T19:06:09.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.475+0000 7f978b79a700 1 -- 172.21.15.1:0/2987950855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97840b3370 msgr2=0x7f97840cfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:09.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.475+0000 7f978b79a700 1 --2- 172.21.15.1:0/2987950855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97840b3370 0x7f97840cfff0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f977c009d90 tx=0x7f977c00b330 comp rx=0 tx=0).stop 2024-03-18T19:06:09.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.475+0000 7f978b79a700 1 -- 172.21.15.1:0/2987950855 shutdown_connections 2024-03-18T19:06:09.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.475+0000 7f978b79a700 1 --2- 172.21.15.1:0/2987950855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97840b3370 0x7f97840cfff0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.475+0000 7f978b79a700 1 --2- 172.21.15.1:0/2987950855 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97840b2f10 0x7f97840cfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.475+0000 7f978b79a700 1 -- 172.21.15.1:0/2987950855 >> 172.21.15.1:0/2987950855 conn(0x7f978406ab70 msgr2=0x7f978406af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:09.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.475+0000 7f978b79a700 1 -- 172.21.15.1:0/2987950855 shutdown_connections 2024-03-18T19:06:09.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.475+0000 7f978b79a700 1 -- 172.21.15.1:0/2987950855 wait complete. 2024-03-18T19:06:09.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.476+0000 7f978b79a700 1 Processor -- start 2024-03-18T19:06:09.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.476+0000 7f978b79a700 1 -- start start 2024-03-18T19:06:09.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.477+0000 7f978b79a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97840b2f10 0x7f9784071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:09.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.477+0000 7f978b79a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97840b3370 0x7f9784152890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:09.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.477+0000 7f978b79a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f978415a0a0 0x7f97841f0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:09.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.477+0000 7f978b79a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97841506c0 con 0x7f97840b2f10 2024-03-18T19:06:09.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.477+0000 7f978b79a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978414fab0 con 0x7f978415a0a0 2024-03-18T19:06:09.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.477+0000 7f978b79a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978414fd90 con 0x7f97840b3370 2024-03-18T19:06:09.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.477+0000 7f9789536700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97840b2f10 0x7f9784071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:09.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.477+0000 7f9789536700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97840b2f10 0x7f9784071020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40920/0 (socket says 172.21.15.1:40920) 2024-03-18T19:06:09.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.477+0000 7f9789536700 1 -- 172.21.15.1:0/346002950 learned_addr learned my addr 172.21.15.1:0/346002950 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:09.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.477+0000 7f9789d37700 1 --2- 172.21.15.1:0/346002950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f978415a0a0 0x7f97841f0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:09.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.477+0000 7f9788d35700 1 --2- 172.21.15.1:0/346002950 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97840b3370 0x7f9784152890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:09.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.478+0000 7f9789536700 1 -- 172.21.15.1:0/346002950 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97840b3370 msgr2=0x7f9784152890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:09.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.478+0000 7f9789536700 1 --2- 172.21.15.1:0/346002950 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97840b3370 0x7f9784152890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.478+0000 7f9789536700 1 -- 172.21.15.1:0/346002950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f978415a0a0 msgr2=0x7f97841f0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:09.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.478+0000 7f9789536700 1 --2- 172.21.15.1:0/346002950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f978415a0a0 0x7f97841f0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.478+0000 7f9789536700 1 -- 172.21.15.1:0/346002950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97800094a0 con 0x7f97840b2f10 2024-03-18T19:06:09.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.478+0000 7f9789536700 1 --2- 172.21.15.1:0/346002950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97840b2f10 0x7f9784071020 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f9774001eb0 tx=0x7f977400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:09.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.478+0000 7f9779ffb700 1 -- 172.21.15.1:0/346002950 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9774005d70 con 0x7f97840b2f10 2024-03-18T19:06:09.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.478+0000 7f9779ffb700 1 -- 172.21.15.1:0/346002950 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97740070d0 con 0x7f97840b2f10 2024-03-18T19:06:09.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.478+0000 7f978b79a700 1 -- 172.21.15.1:0/346002950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977c009a70 con 0x7f97840b2f10 2024-03-18T19:06:09.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.478+0000 7f9779ffb700 1 -- 172.21.15.1:0/346002950 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9774007460 con 0x7f97840b2f10 2024-03-18T19:06:09.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.478+0000 7f978b79a700 1 -- 172.21.15.1:0/346002950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9784150350 con 0x7f97840b2f10 2024-03-18T19:06:09.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.479+0000 7f978b79a700 1 -- 172.21.15.1:0/346002950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97840676a0 con 0x7f97840b2f10 2024-03-18T19:06:09.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.480+0000 7f9779ffb700 1 -- 172.21.15.1:0/346002950 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f97740075c0 con 0x7f97840b2f10 2024-03-18T19:06:09.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.480+0000 7f9779ffb700 1 --2- 172.21.15.1:0/346002950 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f976806adb0 0x7f976806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:09.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.480+0000 7f9779ffb700 1 -- 172.21.15.1:0/346002950 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9774088930 con 0x7f97840b2f10 2024-03-18T19:06:09.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.480+0000 7f9788d35700 1 --2- 172.21.15.1:0/346002950 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f976806adb0 0x7f976806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:09.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.480+0000 7f9788d35700 1 --2- 172.21.15.1:0/346002950 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f976806adb0 0x7f976806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9784151e70 tx=0x7f9780009210 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:09.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.484+0000 7f9779ffb700 1 -- 172.21.15.1:0/346002950 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9774053950 con 0x7f97840b2f10 2024-03-18T19:06:09.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.684+0000 7f978b79a700 1 -- 172.21.15.1:0/346002950 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f978415e2c0 con 0x7f976806adb0 2024-03-18T19:06:09.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.692+0000 7f9779ffb700 1 -- 172.21.15.1:0/346002950 <== mgr.6631 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290090 (secure 0 0 0) 0x7f97800099c0 con 0x7f976806adb0 2024-03-18T19:06:09.691 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:09.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.695+0000 7f978b79a700 1 -- 172.21.15.1:0/346002950 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f976806adb0 msgr2=0x7f976806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:09.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.695+0000 7f978b79a700 1 --2- 172.21.15.1:0/346002950 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f976806adb0 0x7f976806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9784151e70 tx=0x7f9780009210 comp rx=0 tx=0).stop 2024-03-18T19:06:09.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.696+0000 7f978b79a700 1 -- 172.21.15.1:0/346002950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97840b2f10 msgr2=0x7f9784071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:09.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.696+0000 7f978b79a700 1 --2- 172.21.15.1:0/346002950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97840b2f10 0x7f9784071020 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f9774001eb0 tx=0x7f977400c7b0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.696+0000 7f978b79a700 1 -- 172.21.15.1:0/346002950 shutdown_connections 2024-03-18T19:06:09.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.696+0000 7f978b79a700 1 --2- 172.21.15.1:0/346002950 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f976806adb0 0x7f976806d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.696+0000 7f978b79a700 1 --2- 172.21.15.1:0/346002950 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f978415a0a0 0x7f97841f0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.696+0000 7f978b79a700 1 --2- 172.21.15.1:0/346002950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97840b2f10 0x7f9784071020 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.696+0000 7f978b79a700 1 --2- 172.21.15.1:0/346002950 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97840b3370 0x7f9784152890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.696+0000 7f978b79a700 1 -- 172.21.15.1:0/346002950 >> 172.21.15.1:0/346002950 conn(0x7f978406ab70 msgr2=0x7f97840cd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:09.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.696+0000 7f978b79a700 1 -- 172.21.15.1:0/346002950 shutdown_connections 2024-03-18T19:06:09.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.696+0000 7f978b79a700 1 -- 172.21.15.1:0/346002950 wait complete. 2024-03-18T19:06:09.694 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T19:06:09.704 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-03-18T19:06:09.645092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":303,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":98,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":342792,"kb_used_data":10996,"kb_used_omap":12,"kb_used_meta":331635,"kb_avail":754631928,"statfs":{"total":773094113280,"available":772743094272,"internally_reserved":0,"allocated":11259904,"data_stored":6180248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":339594809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.932915"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.171835+0000","last_change":"2024-03-18T19:06:05.170868+0000","last_active":"2024-03-18T19:06:05.171835+0000","last_peered":"2024-03-18T19:06:05.171835+0000","last_clean":"2024-03-18T19:06:05.171835+0000","last_became_active":"2024-03-18T19:06:05.170592+0000","last_became_peered":"2024-03-18T19:06:05.170592+0000","last_unstale":"2024-03-18T19:06:05.171835+0000","last_undegraded":"2024-03-18T19:06:05.171835+0000","last_fullsized":"2024-03-18T19:06:05.171835+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:59:50.028248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157143+0000","last_change":"2024-03-18T19:06:05.168652+0000","last_active":"2024-03-18T19:06:06.157143+0000","last_peered":"2024-03-18T19:06:06.157143+0000","last_clean":"2024-03-18T19:06:06.157143+0000","last_became_active":"2024-03-18T19:06:05.168384+0000","last_became_peered":"2024-03-18T19:06:05.168384+0000","last_unstale":"2024-03-18T19:06:06.157143+0000","last_undegraded":"2024-03-18T19:06:06.157143+0000","last_fullsized":"2024-03-18T19:06:06.157143+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:10.292804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.175497+0000","last_change":"2024-03-18T19:06:05.172281+0000","last_active":"2024-03-18T19:06:05.175497+0000","last_peered":"2024-03-18T19:06:05.175497+0000","last_clean":"2024-03-18T19:06:05.175497+0000","last_became_active":"2024-03-18T19:06:05.172042+0000","last_became_peered":"2024-03-18T19:06:05.172042+0000","last_unstale":"2024-03-18T19:06:05.175497+0000","last_undegraded":"2024-03-18T19:06:05.175497+0000","last_fullsized":"2024-03-18T19:06:05.175497+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:09:48.834328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.176656+0000","last_change":"2024-03-18T19:06:05.173122+0000","last_active":"2024-03-18T19:06:05.176656+0000","last_peered":"2024-03-18T19:06:05.176656+0000","last_clean":"2024-03-18T19:06:05.176656+0000","last_became_active":"2024-03-18T19:06:05.172838+0000","last_became_peered":"2024-03-18T19:06:05.172838+0000","last_unstale":"2024-03-18T19:06:05.176656+0000","last_undegraded":"2024-03-18T19:06:05.176656+0000","last_fullsized":"2024-03-18T19:06:05.176656+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:11:36.194047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174488+0000","last_change":"2024-03-18T19:06:05.170972+0000","last_active":"2024-03-18T19:06:05.174488+0000","last_peered":"2024-03-18T19:06:05.174488+0000","last_clean":"2024-03-18T19:06:05.174488+0000","last_became_active":"2024-03-18T19:06:05.170709+0000","last_became_peered":"2024-03-18T19:06:05.170709+0000","last_unstale":"2024-03-18T19:06:05.174488+0000","last_undegraded":"2024-03-18T19:06:05.174488+0000","last_fullsized":"2024-03-18T19:06:05.174488+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:03:46.372144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.180909+0000","last_change":"2024-03-18T19:06:05.179927+0000","last_active":"2024-03-18T19:06:05.180909+0000","last_peered":"2024-03-18T19:06:05.180909+0000","last_clean":"2024-03-18T19:06:05.180909+0000","last_became_active":"2024-03-18T19:06:05.179631+0000","last_became_peered":"2024-03-18T19:06:05.179631+0000","last_unstale":"2024-03-18T19:06:05.180909+0000","last_undegraded":"2024-03-18T19:06:05.180909+0000","last_fullsized":"2024-03-18T19:06:05.180909+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:03:43.145212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.156750+0000","last_change":"2024-03-18T19:06:05.168731+0000","last_active":"2024-03-18T19:06:06.156750+0000","last_peered":"2024-03-18T19:06:06.156750+0000","last_clean":"2024-03-18T19:06:06.156750+0000","last_became_active":"2024-03-18T19:06:05.167788+0000","last_became_peered":"2024-03-18T19:06:05.167788+0000","last_unstale":"2024-03-18T19:06:06.156750+0000","last_undegraded":"2024-03-18T19:06:06.156750+0000","last_fullsized":"2024-03-18T19:06:06.156750+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:09:48.834328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174289+0000","last_change":"2024-03-18T19:06:05.167420+0000","last_active":"2024-03-18T19:06:05.174289+0000","last_peered":"2024-03-18T19:06:05.174289+0000","last_clean":"2024-03-18T19:06:05.174289+0000","last_became_active":"2024-03-18T19:06:05.165913+0000","last_became_peered":"2024-03-18T19:06:05.165913+0000","last_unstale":"2024-03-18T19:06:05.174289+0000","last_undegraded":"2024-03-18T19:06:05.174289+0000","last_fullsized":"2024-03-18T19:06:05.174289+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:29.423402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.182553+0000","last_change":"2024-03-18T19:06:05.181568+0000","last_active":"2024-03-18T19:06:05.182553+0000","last_peered":"2024-03-18T19:06:05.182553+0000","last_clean":"2024-03-18T19:06:05.182553+0000","last_became_active":"2024-03-18T19:06:05.181258+0000","last_became_peered":"2024-03-18T19:06:05.181258+0000","last_unstale":"2024-03-18T19:06:05.182553+0000","last_undegraded":"2024-03-18T19:06:05.182553+0000","last_fullsized":"2024-03-18T19:06:05.182553+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:09:40.312969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174368+0000","last_change":"2024-03-18T19:06:05.171798+0000","last_active":"2024-03-18T19:06:05.174368+0000","last_peered":"2024-03-18T19:06:05.174368+0000","last_clean":"2024-03-18T19:06:05.174368+0000","last_became_active":"2024-03-18T19:06:05.171537+0000","last_became_peered":"2024-03-18T19:06:05.171537+0000","last_unstale":"2024-03-18T19:06:05.174368+0000","last_undegraded":"2024-03-18T19:06:05.174368+0000","last_fullsized":"2024-03-18T19:06:05.174368+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:07:30.773855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157226+0000","last_change":"2024-03-18T19:06:05.169086+0000","last_active":"2024-03-18T19:06:06.157226+0000","last_peered":"2024-03-18T19:06:06.157226+0000","last_clean":"2024-03-18T19:06:06.157226+0000","last_became_active":"2024-03-18T19:06:05.168472+0000","last_became_peered":"2024-03-18T19:06:05.168472+0000","last_unstale":"2024-03-18T19:06:06.157226+0000","last_undegraded":"2024-03-18T19:06:06.157226+0000","last_fullsized":"2024-03-18T19:06:06.157226+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:57.274617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.178858+0000","last_change":"2024-03-18T19:06:05.176413+0000","last_active":"2024-03-18T19:06:05.178858+0000","last_peered":"2024-03-18T19:06:05.178858+0000","last_clean":"2024-03-18T19:06:05.178858+0000","last_became_active":"2024-03-18T19:06:05.175933+0000","last_became_peered":"2024-03-18T19:06:05.175933+0000","last_unstale":"2024-03-18T19:06:05.178858+0000","last_undegraded":"2024-03-18T19:06:05.178858+0000","last_fullsized":"2024-03-18T19:06:05.178858+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:30:47.424183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.184117+0000","last_change":"2024-03-18T19:06:05.182075+0000","last_active":"2024-03-18T19:06:05.184117+0000","last_peered":"2024-03-18T19:06:05.184117+0000","last_clean":"2024-03-18T19:06:05.184117+0000","last_became_active":"2024-03-18T19:06:05.181805+0000","last_became_peered":"2024-03-18T19:06:05.181805+0000","last_unstale":"2024-03-18T19:06:05.184117+0000","last_undegraded":"2024-03-18T19:06:05.184117+0000","last_fullsized":"2024-03-18T19:06:05.184117+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:46.650103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.183569+0000","last_change":"2024-03-18T19:06:05.181352+0000","last_active":"2024-03-18T19:06:05.183569+0000","last_peered":"2024-03-18T19:06:05.183569+0000","last_clean":"2024-03-18T19:06:05.183569+0000","last_became_active":"2024-03-18T19:06:05.181008+0000","last_became_peered":"2024-03-18T19:06:05.181008+0000","last_unstale":"2024-03-18T19:06:05.183569+0000","last_undegraded":"2024-03-18T19:06:05.183569+0000","last_fullsized":"2024-03-18T19:06:05.183569+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:11:30.136013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.156921+0000","last_change":"2024-03-18T19:06:05.175508+0000","last_active":"2024-03-18T19:06:06.156921+0000","last_peered":"2024-03-18T19:06:06.156921+0000","last_clean":"2024-03-18T19:06:06.156921+0000","last_became_active":"2024-03-18T19:06:05.175053+0000","last_became_peered":"2024-03-18T19:06:05.175053+0000","last_unstale":"2024-03-18T19:06:06.156921+0000","last_undegraded":"2024-03-18T19:06:06.156921+0000","last_fullsized":"2024-03-18T19:06:06.156921+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:51.812857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.169069+0000","last_change":"2024-03-18T19:06:05.167015+0000","last_active":"2024-03-18T19:06:05.169069+0000","last_peered":"2024-03-18T19:06:05.169069+0000","last_clean":"2024-03-18T19:06:05.169069+0000","last_became_active":"2024-03-18T19:06:05.166748+0000","last_became_peered":"2024-03-18T19:06:05.166748+0000","last_unstale":"2024-03-18T19:06:05.169069+0000","last_undegraded":"2024-03-18T19:06:05.169069+0000","last_fullsized":"2024-03-18T19:06:05.169069+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:39.218334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.175060+0000","last_change":"2024-03-18T19:06:05.171233+0000","last_active":"2024-03-18T19:06:05.175060+0000","last_peered":"2024-03-18T19:06:05.175060+0000","last_clean":"2024-03-18T19:06:05.175060+0000","last_became_active":"2024-03-18T19:06:05.170894+0000","last_became_peered":"2024-03-18T19:06:05.170894+0000","last_unstale":"2024-03-18T19:06:05.175060+0000","last_undegraded":"2024-03-18T19:06:05.175060+0000","last_fullsized":"2024-03-18T19:06:05.175060+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:01:48.953093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.175414+0000","last_change":"2024-03-18T19:06:05.173544+0000","last_active":"2024-03-18T19:06:05.175414+0000","last_peered":"2024-03-18T19:06:05.175414+0000","last_clean":"2024-03-18T19:06:05.175414+0000","last_became_active":"2024-03-18T19:06:05.172319+0000","last_became_peered":"2024-03-18T19:06:05.172319+0000","last_unstale":"2024-03-18T19:06:05.175414+0000","last_undegraded":"2024-03-18T19:06:05.175414+0000","last_fullsized":"2024-03-18T19:06:05.175414+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:55:27.854722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.180211+0000","last_change":"2024-03-18T19:06:05.176492+0000","last_active":"2024-03-18T19:06:05.180211+0000","last_peered":"2024-03-18T19:06:05.180211+0000","last_clean":"2024-03-18T19:06:05.180211+0000","last_became_active":"2024-03-18T19:06:05.176223+0000","last_became_peered":"2024-03-18T19:06:05.176223+0000","last_unstale":"2024-03-18T19:06:05.180211+0000","last_undegraded":"2024-03-18T19:06:05.180211+0000","last_fullsized":"2024-03-18T19:06:05.180211+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:40.596183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.176577+0000","last_change":"2024-03-18T19:06:05.175850+0000","last_active":"2024-03-18T19:06:05.176577+0000","last_peered":"2024-03-18T19:06:05.176577+0000","last_clean":"2024-03-18T19:06:05.176577+0000","last_became_active":"2024-03-18T19:06:05.175213+0000","last_became_peered":"2024-03-18T19:06:05.175213+0000","last_unstale":"2024-03-18T19:06:05.176577+0000","last_undegraded":"2024-03-18T19:06:05.176577+0000","last_fullsized":"2024-03-18T19:06:05.176577+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:13.231915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.179044+0000","last_change":"2024-03-18T19:06:05.175698+0000","last_active":"2024-03-18T19:06:05.179044+0000","last_peered":"2024-03-18T19:06:05.179044+0000","last_clean":"2024-03-18T19:06:05.179044+0000","last_became_active":"2024-03-18T19:06:05.175502+0000","last_became_peered":"2024-03-18T19:06:05.175502+0000","last_unstale":"2024-03-18T19:06:05.179044+0000","last_undegraded":"2024-03-18T19:06:05.179044+0000","last_fullsized":"2024-03-18T19:06:05.179044+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:38:46.621286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.184140+0000","last_change":"2024-03-18T19:06:05.183425+0000","last_active":"2024-03-18T19:06:05.184140+0000","last_peered":"2024-03-18T19:06:05.184140+0000","last_clean":"2024-03-18T19:06:05.184140+0000","last_became_active":"2024-03-18T19:06:05.183155+0000","last_became_peered":"2024-03-18T19:06:05.183155+0000","last_unstale":"2024-03-18T19:06:05.184140+0000","last_undegraded":"2024-03-18T19:06:05.184140+0000","last_fullsized":"2024-03-18T19:06:05.184140+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:17.916885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174059+0000","last_change":"2024-03-18T19:06:05.171509+0000","last_active":"2024-03-18T19:06:05.174059+0000","last_peered":"2024-03-18T19:06:05.174059+0000","last_clean":"2024-03-18T19:06:05.174059+0000","last_became_active":"2024-03-18T19:06:05.171248+0000","last_became_peered":"2024-03-18T19:06:05.171248+0000","last_unstale":"2024-03-18T19:06:05.174059+0000","last_undegraded":"2024-03-18T19:06:05.174059+0000","last_fullsized":"2024-03-18T19:06:05.174059+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:45:43.675246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.156878+0000","last_change":"2024-03-18T19:06:05.173083+0000","last_active":"2024-03-18T19:06:06.156878+0000","last_peered":"2024-03-18T19:06:06.156878+0000","last_clean":"2024-03-18T19:06:06.156878+0000","last_became_active":"2024-03-18T19:06:05.172856+0000","last_became_peered":"2024-03-18T19:06:05.172856+0000","last_unstale":"2024-03-18T19:06:06.156878+0000","last_undegraded":"2024-03-18T19:06:06.156878+0000","last_fullsized":"2024-03-18T19:06:06.156878+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:50.007982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157778+0000","last_change":"2024-03-18T19:06:05.173791+0000","last_active":"2024-03-18T19:06:06.157778+0000","last_peered":"2024-03-18T19:06:06.157778+0000","last_clean":"2024-03-18T19:06:06.157778+0000","last_became_active":"2024-03-18T19:06:05.173538+0000","last_became_peered":"2024-03-18T19:06:05.173538+0000","last_unstale":"2024-03-18T19:06:06.157778+0000","last_undegraded":"2024-03-18T19:06:06.157778+0000","last_fullsized":"2024-03-18T19:06:06.157778+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:36:48.269178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.178673+0000","last_change":"2024-03-18T19:06:05.175304+0000","last_active":"2024-03-18T19:06:05.178673+0000","last_peered":"2024-03-18T19:06:05.178673+0000","last_clean":"2024-03-18T19:06:05.178673+0000","last_became_active":"2024-03-18T19:06:05.175037+0000","last_became_peered":"2024-03-18T19:06:05.175037+0000","last_unstale":"2024-03-18T19:06:05.178673+0000","last_undegraded":"2024-03-18T19:06:05.178673+0000","last_fullsized":"2024-03-18T19:06:05.178673+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:55:27.854722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.173923+0000","last_change":"2024-03-18T19:06:05.172130+0000","last_active":"2024-03-18T19:06:05.173923+0000","last_peered":"2024-03-18T19:06:05.173923+0000","last_clean":"2024-03-18T19:06:05.173923+0000","last_became_active":"2024-03-18T19:06:05.171841+0000","last_became_peered":"2024-03-18T19:06:05.171841+0000","last_unstale":"2024-03-18T19:06:05.173923+0000","last_undegraded":"2024-03-18T19:06:05.173923+0000","last_fullsized":"2024-03-18T19:06:05.173923+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:14.564659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.175085+0000","last_change":"2024-03-18T19:06:05.173695+0000","last_active":"2024-03-18T19:06:05.175085+0000","last_peered":"2024-03-18T19:06:05.175085+0000","last_clean":"2024-03-18T19:06:05.175085+0000","last_became_active":"2024-03-18T19:06:05.173416+0000","last_became_peered":"2024-03-18T19:06:05.173416+0000","last_unstale":"2024-03-18T19:06:05.175085+0000","last_undegraded":"2024-03-18T19:06:05.175085+0000","last_fullsized":"2024-03-18T19:06:05.175085+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:59:06.088817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157130+0000","last_change":"2024-03-18T19:06:05.163843+0000","last_active":"2024-03-18T19:06:06.157130+0000","last_peered":"2024-03-18T19:06:06.157130+0000","last_clean":"2024-03-18T19:06:06.157130+0000","last_became_active":"2024-03-18T19:06:05.163515+0000","last_became_peered":"2024-03-18T19:06:05.163515+0000","last_unstale":"2024-03-18T19:06:06.157130+0000","last_undegraded":"2024-03-18T19:06:06.157130+0000","last_fullsized":"2024-03-18T19:06:06.157130+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:22.181026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.168698+0000","last_change":"2024-03-18T19:06:05.165309+0000","last_active":"2024-03-18T19:06:05.168698+0000","last_peered":"2024-03-18T19:06:05.168698+0000","last_clean":"2024-03-18T19:06:05.168698+0000","last_became_active":"2024-03-18T19:06:05.165002+0000","last_became_peered":"2024-03-18T19:06:05.165002+0000","last_unstale":"2024-03-18T19:06:05.168698+0000","last_undegraded":"2024-03-18T19:06:05.168698+0000","last_fullsized":"2024-03-18T19:06:05.168698+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:04.872630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174130+0000","last_change":"2024-03-18T19:06:05.162716+0000","last_active":"2024-03-18T19:06:05.174130+0000","last_peered":"2024-03-18T19:06:05.174130+0000","last_clean":"2024-03-18T19:06:05.174130+0000","last_became_active":"2024-03-18T19:06:05.162474+0000","last_became_peered":"2024-03-18T19:06:05.162474+0000","last_unstale":"2024-03-18T19:06:05.174130+0000","last_undegraded":"2024-03-18T19:06:05.174130+0000","last_fullsized":"2024-03-18T19:06:05.174130+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:30:47.424183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174497+0000","last_change":"2024-03-18T19:06:05.172172+0000","last_active":"2024-03-18T19:06:05.174497+0000","last_peered":"2024-03-18T19:06:05.174497+0000","last_clean":"2024-03-18T19:06:05.174497+0000","last_became_active":"2024-03-18T19:06:05.171556+0000","last_became_peered":"2024-03-18T19:06:05.171556+0000","last_unstale":"2024-03-18T19:06:05.174497+0000","last_undegraded":"2024-03-18T19:06:05.174497+0000","last_fullsized":"2024-03-18T19:06:05.174497+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:32:09.971963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.175265+0000","last_change":"2024-03-18T19:06:05.171715+0000","last_active":"2024-03-18T19:06:05.175265+0000","last_peered":"2024-03-18T19:06:05.175265+0000","last_clean":"2024-03-18T19:06:05.175265+0000","last_became_active":"2024-03-18T19:06:05.171456+0000","last_became_peered":"2024-03-18T19:06:05.171456+0000","last_unstale":"2024-03-18T19:06:05.175265+0000","last_undegraded":"2024-03-18T19:06:05.175265+0000","last_fullsized":"2024-03-18T19:06:05.175265+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:54:09.212832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"102'4","reported_seq":14,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.181733+0000","last_change":"2024-03-18T19:06:04.734193+0000","last_active":"2024-03-18T19:06:05.181733+0000","last_peered":"2024-03-18T19:06:05.181733+0000","last_clean":"2024-03-18T19:06:05.181733+0000","last_became_active":"2024-03-18T19:06:04.733935+0000","last_became_peered":"2024-03-18T19:06:04.733935+0000","last_unstale":"2024-03-18T19:06:05.181733+0000","last_undegraded":"2024-03-18T19:06:05.181733+0000","last_fullsized":"2024-03-18T19:06:05.181733+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:21.798183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.156918+0000","last_change":"2024-03-18T19:06:05.175305+0000","last_active":"2024-03-18T19:06:06.156918+0000","last_peered":"2024-03-18T19:06:06.156918+0000","last_clean":"2024-03-18T19:06:06.156918+0000","last_became_active":"2024-03-18T19:06:05.174712+0000","last_became_peered":"2024-03-18T19:06:05.174712+0000","last_unstale":"2024-03-18T19:06:06.156918+0000","last_undegraded":"2024-03-18T19:06:06.156918+0000","last_fullsized":"2024-03-18T19:06:06.156918+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:51:29.990239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.158467+0000","last_change":"2024-03-18T19:06:04.727730+0000","last_active":"2024-03-18T19:06:05.158467+0000","last_peered":"2024-03-18T19:06:05.158467+0000","last_clean":"2024-03-18T19:06:05.158467+0000","last_became_active":"2024-03-18T19:06:04.727444+0000","last_became_peered":"2024-03-18T19:06:04.727444+0000","last_unstale":"2024-03-18T19:06:05.158467+0000","last_undegraded":"2024-03-18T19:06:05.158467+0000","last_fullsized":"2024-03-18T19:06:05.158467+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:24:31.548334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.184227+0000","last_change":"2024-03-18T19:06:05.183264+0000","last_active":"2024-03-18T19:06:05.184227+0000","last_peered":"2024-03-18T19:06:05.184227+0000","last_clean":"2024-03-18T19:06:05.184227+0000","last_became_active":"2024-03-18T19:06:05.182888+0000","last_became_peered":"2024-03-18T19:06:05.182888+0000","last_unstale":"2024-03-18T19:06:05.184227+0000","last_undegraded":"2024-03-18T19:06:05.184227+0000","last_fullsized":"2024-03-18T19:06:05.184227+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:09:48.834328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.170868+0000","last_change":"2024-03-18T19:06:04.728484+0000","last_active":"2024-03-18T19:06:05.170868+0000","last_peered":"2024-03-18T19:06:05.170868+0000","last_clean":"2024-03-18T19:06:05.170868+0000","last_became_active":"2024-03-18T19:06:04.728151+0000","last_became_peered":"2024-03-18T19:06:04.728151+0000","last_unstale":"2024-03-18T19:06:05.170868+0000","last_undegraded":"2024-03-18T19:06:05.170868+0000","last_fullsized":"2024-03-18T19:06:05.170868+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:36:27.277009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.179028+0000","last_change":"2024-03-18T19:06:05.175617+0000","last_active":"2024-03-18T19:06:05.179028+0000","last_peered":"2024-03-18T19:06:05.179028+0000","last_clean":"2024-03-18T19:06:05.179028+0000","last_became_active":"2024-03-18T19:06:05.175412+0000","last_became_peered":"2024-03-18T19:06:05.175412+0000","last_unstale":"2024-03-18T19:06:05.179028+0000","last_undegraded":"2024-03-18T19:06:05.179028+0000","last_fullsized":"2024-03-18T19:06:05.179028+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:45:28.072614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"102'2","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157317+0000","last_change":"2024-03-18T19:06:04.729562+0000","last_active":"2024-03-18T19:06:06.157317+0000","last_peered":"2024-03-18T19:06:06.157317+0000","last_clean":"2024-03-18T19:06:06.157317+0000","last_became_active":"2024-03-18T19:06:04.729275+0000","last_became_peered":"2024-03-18T19:06:04.729275+0000","last_unstale":"2024-03-18T19:06:06.157317+0000","last_undegraded":"2024-03-18T19:06:06.157317+0000","last_fullsized":"2024-03-18T19:06:06.157317+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:26:37.482587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157090+0000","last_change":"2024-03-18T19:06:05.170212+0000","last_active":"2024-03-18T19:06:06.157090+0000","last_peered":"2024-03-18T19:06:06.157090+0000","last_clean":"2024-03-18T19:06:06.157090+0000","last_became_active":"2024-03-18T19:06:05.169953+0000","last_became_peered":"2024-03-18T19:06:05.169953+0000","last_unstale":"2024-03-18T19:06:06.157090+0000","last_undegraded":"2024-03-18T19:06:06.157090+0000","last_fullsized":"2024-03-18T19:06:06.157090+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:46.451963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.170935+0000","last_change":"2024-03-18T19:06:04.734707+0000","last_active":"2024-03-18T19:06:05.170935+0000","last_peered":"2024-03-18T19:06:05.170935+0000","last_clean":"2024-03-18T19:06:05.170935+0000","last_became_active":"2024-03-18T19:06:04.734461+0000","last_became_peered":"2024-03-18T19:06:04.734461+0000","last_unstale":"2024-03-18T19:06:05.170935+0000","last_undegraded":"2024-03-18T19:06:05.170935+0000","last_fullsized":"2024-03-18T19:06:05.170935+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:56.143928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157119+0000","last_change":"2024-03-18T19:06:05.169179+0000","last_active":"2024-03-18T19:06:06.157119+0000","last_peered":"2024-03-18T19:06:06.157119+0000","last_clean":"2024-03-18T19:06:06.157119+0000","last_became_active":"2024-03-18T19:06:05.168843+0000","last_became_peered":"2024-03-18T19:06:05.168843+0000","last_unstale":"2024-03-18T19:06:06.157119+0000","last_undegraded":"2024-03-18T19:06:06.157119+0000","last_fullsized":"2024-03-18T19:06:06.157119+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:48:58.435719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.181115+0000","last_change":"2024-03-18T19:06:04.734668+0000","last_active":"2024-03-18T19:06:05.181115+0000","last_peered":"2024-03-18T19:06:05.181115+0000","last_clean":"2024-03-18T19:06:05.181115+0000","last_became_active":"2024-03-18T19:06:04.734409+0000","last_became_peered":"2024-03-18T19:06:04.734409+0000","last_unstale":"2024-03-18T19:06:05.181115+0000","last_undegraded":"2024-03-18T19:06:05.181115+0000","last_fullsized":"2024-03-18T19:06:05.181115+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:49:20.902554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.182557+0000","last_change":"2024-03-18T19:06:05.181651+0000","last_active":"2024-03-18T19:06:05.182557+0000","last_peered":"2024-03-18T19:06:05.182557+0000","last_clean":"2024-03-18T19:06:05.182557+0000","last_became_active":"2024-03-18T19:06:05.181328+0000","last_became_peered":"2024-03-18T19:06:05.181328+0000","last_unstale":"2024-03-18T19:06:05.182557+0000","last_undegraded":"2024-03-18T19:06:05.182557+0000","last_fullsized":"2024-03-18T19:06:05.182557+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:36:28.282894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157027+0000","last_change":"2024-03-18T19:06:04.727277+0000","last_active":"2024-03-18T19:06:06.157027+0000","last_peered":"2024-03-18T19:06:06.157027+0000","last_clean":"2024-03-18T19:06:06.157027+0000","last_became_active":"2024-03-18T19:06:04.727001+0000","last_became_peered":"2024-03-18T19:06:04.727001+0000","last_unstale":"2024-03-18T19:06:06.157027+0000","last_undegraded":"2024-03-18T19:06:06.157027+0000","last_fullsized":"2024-03-18T19:06:06.157027+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:14.933050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157252+0000","last_change":"2024-03-18T19:06:05.168878+0000","last_active":"2024-03-18T19:06:06.157252+0000","last_peered":"2024-03-18T19:06:06.157252+0000","last_clean":"2024-03-18T19:06:06.157252+0000","last_became_active":"2024-03-18T19:06:05.168567+0000","last_became_peered":"2024-03-18T19:06:05.168567+0000","last_unstale":"2024-03-18T19:06:06.157252+0000","last_undegraded":"2024-03-18T19:06:06.157252+0000","last_fullsized":"2024-03-18T19:06:06.157252+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:09:40.312969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.161685+0000","last_change":"2024-03-18T19:06:04.735666+0000","last_active":"2024-03-18T19:06:05.161685+0000","last_peered":"2024-03-18T19:06:05.161685+0000","last_clean":"2024-03-18T19:06:05.161685+0000","last_became_active":"2024-03-18T19:06:04.735407+0000","last_became_peered":"2024-03-18T19:06:04.735407+0000","last_unstale":"2024-03-18T19:06:05.161685+0000","last_undegraded":"2024-03-18T19:06:05.161685+0000","last_fullsized":"2024-03-18T19:06:05.161685+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:52.738028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.156879+0000","last_change":"2024-03-18T19:06:05.166564+0000","last_active":"2024-03-18T19:06:06.156879+0000","last_peered":"2024-03-18T19:06:06.156879+0000","last_clean":"2024-03-18T19:06:06.156879+0000","last_became_active":"2024-03-18T19:06:05.166311+0000","last_became_peered":"2024-03-18T19:06:05.166311+0000","last_unstale":"2024-03-18T19:06:06.156879+0000","last_undegraded":"2024-03-18T19:06:06.156879+0000","last_fullsized":"2024-03-18T19:06:06.156879+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:36:34.321207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.155624+0000","last_change":"2024-03-18T19:06:04.727189+0000","last_active":"2024-03-18T19:06:05.155624+0000","last_peered":"2024-03-18T19:06:05.155624+0000","last_clean":"2024-03-18T19:06:05.155624+0000","last_became_active":"2024-03-18T19:06:04.726902+0000","last_became_peered":"2024-03-18T19:06:04.726902+0000","last_unstale":"2024-03-18T19:06:05.155624+0000","last_undegraded":"2024-03-18T19:06:05.155624+0000","last_fullsized":"2024-03-18T19:06:05.155624+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:45:42.669362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.178648+0000","last_change":"2024-03-18T19:06:05.174889+0000","last_active":"2024-03-18T19:06:05.178648+0000","last_peered":"2024-03-18T19:06:05.178648+0000","last_clean":"2024-03-18T19:06:05.178648+0000","last_became_active":"2024-03-18T19:06:05.174628+0000","last_became_peered":"2024-03-18T19:06:05.174628+0000","last_unstale":"2024-03-18T19:06:05.178648+0000","last_undegraded":"2024-03-18T19:06:05.178648+0000","last_fullsized":"2024-03-18T19:06:05.178648+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:34.217833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.164960+0000","last_change":"2024-03-18T19:06:04.731599+0000","last_active":"2024-03-18T19:06:05.164960+0000","last_peered":"2024-03-18T19:06:05.164960+0000","last_clean":"2024-03-18T19:06:05.164960+0000","last_became_active":"2024-03-18T19:06:04.731332+0000","last_became_peered":"2024-03-18T19:06:04.731332+0000","last_unstale":"2024-03-18T19:06:05.164960+0000","last_undegraded":"2024-03-18T19:06:05.164960+0000","last_fullsized":"2024-03-18T19:06:05.164960+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:24:31.548334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.183593+0000","last_change":"2024-03-18T19:06:05.181341+0000","last_active":"2024-03-18T19:06:05.183593+0000","last_peered":"2024-03-18T19:06:05.183593+0000","last_clean":"2024-03-18T19:06:05.183593+0000","last_became_active":"2024-03-18T19:06:05.181028+0000","last_became_peered":"2024-03-18T19:06:05.181028+0000","last_unstale":"2024-03-18T19:06:05.183593+0000","last_undegraded":"2024-03-18T19:06:05.183593+0000","last_fullsized":"2024-03-18T19:06:05.183593+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:06.088405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"102'3","reported_seq":13,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.175438+0000","last_change":"2024-03-18T19:06:04.729893+0000","last_active":"2024-03-18T19:06:05.175438+0000","last_peered":"2024-03-18T19:06:05.175438+0000","last_clean":"2024-03-18T19:06:05.175438+0000","last_became_active":"2024-03-18T19:06:04.729615+0000","last_became_peered":"2024-03-18T19:06:04.729615+0000","last_unstale":"2024-03-18T19:06:05.175438+0000","last_undegraded":"2024-03-18T19:06:05.175438+0000","last_fullsized":"2024-03-18T19:06:05.175438+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:21.448421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174311+0000","last_change":"2024-03-18T19:06:05.161117+0000","last_active":"2024-03-18T19:06:05.174311+0000","last_peered":"2024-03-18T19:06:05.174311+0000","last_clean":"2024-03-18T19:06:05.174311+0000","last_became_active":"2024-03-18T19:06:05.160450+0000","last_became_peered":"2024-03-18T19:06:05.160450+0000","last_unstale":"2024-03-18T19:06:05.174311+0000","last_undegraded":"2024-03-18T19:06:05.174311+0000","last_fullsized":"2024-03-18T19:06:05.174311+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:02:26.476481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"102'6","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157412+0000","last_change":"2024-03-18T19:06:04.735749+0000","last_active":"2024-03-18T19:06:06.157412+0000","last_peered":"2024-03-18T19:06:06.157412+0000","last_clean":"2024-03-18T19:06:06.157412+0000","last_became_active":"2024-03-18T19:06:04.735436+0000","last_became_peered":"2024-03-18T19:06:04.735436+0000","last_unstale":"2024-03-18T19:06:06.157412+0000","last_undegraded":"2024-03-18T19:06:06.157412+0000","last_fullsized":"2024-03-18T19:06:06.157412+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:14:28.241082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.178875+0000","last_change":"2024-03-18T19:06:05.178346+0000","last_active":"2024-03-18T19:06:05.178875+0000","last_peered":"2024-03-18T19:06:05.178875+0000","last_clean":"2024-03-18T19:06:05.178875+0000","last_became_active":"2024-03-18T19:06:05.178118+0000","last_became_peered":"2024-03-18T19:06:05.178118+0000","last_unstale":"2024-03-18T19:06:05.178875+0000","last_undegraded":"2024-03-18T19:06:05.178875+0000","last_fullsized":"2024-03-18T19:06:05.178875+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:54:09.212832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.172713+0000","last_change":"2024-03-18T19:06:04.730583+0000","last_active":"2024-03-18T19:06:05.172713+0000","last_peered":"2024-03-18T19:06:05.172713+0000","last_clean":"2024-03-18T19:06:05.172713+0000","last_became_active":"2024-03-18T19:06:04.730263+0000","last_became_peered":"2024-03-18T19:06:04.730263+0000","last_unstale":"2024-03-18T19:06:05.172713+0000","last_undegraded":"2024-03-18T19:06:05.172713+0000","last_fullsized":"2024-03-18T19:06:05.172713+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:45:27.066729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157115+0000","last_change":"2024-03-18T19:06:05.166464+0000","last_active":"2024-03-18T19:06:06.157115+0000","last_peered":"2024-03-18T19:06:06.157115+0000","last_clean":"2024-03-18T19:06:06.157115+0000","last_became_active":"2024-03-18T19:06:05.166188+0000","last_became_peered":"2024-03-18T19:06:05.166188+0000","last_unstale":"2024-03-18T19:06:06.157115+0000","last_undegraded":"2024-03-18T19:06:06.157115+0000","last_fullsized":"2024-03-18T19:06:06.157115+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:36:28.282894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.153576+0000","last_change":"2024-03-18T19:06:04.725490+0000","last_active":"2024-03-18T19:06:05.153576+0000","last_peered":"2024-03-18T19:06:05.153576+0000","last_clean":"2024-03-18T19:06:05.153576+0000","last_became_active":"2024-03-18T19:06:04.725239+0000","last_became_peered":"2024-03-18T19:06:04.725239+0000","last_unstale":"2024-03-18T19:06:05.153576+0000","last_undegraded":"2024-03-18T19:06:05.153576+0000","last_fullsized":"2024-03-18T19:06:05.153576+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:30:48.969431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157257+0000","last_change":"2024-03-18T19:06:05.170967+0000","last_active":"2024-03-18T19:06:06.157257+0000","last_peered":"2024-03-18T19:06:06.157257+0000","last_clean":"2024-03-18T19:06:06.157257+0000","last_became_active":"2024-03-18T19:06:05.170700+0000","last_became_peered":"2024-03-18T19:06:05.170700+0000","last_unstale":"2024-03-18T19:06:06.157257+0000","last_undegraded":"2024-03-18T19:06:06.157257+0000","last_fullsized":"2024-03-18T19:06:06.157257+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:30:49.975316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.160933+0000","last_change":"2024-03-18T19:06:04.732660+0000","last_active":"2024-03-18T19:06:05.160933+0000","last_peered":"2024-03-18T19:06:05.160933+0000","last_clean":"2024-03-18T19:06:05.160933+0000","last_became_active":"2024-03-18T19:06:04.732378+0000","last_became_peered":"2024-03-18T19:06:04.732378+0000","last_unstale":"2024-03-18T19:06:05.160933+0000","last_undegraded":"2024-03-18T19:06:05.160933+0000","last_fullsized":"2024-03-18T19:06:05.160933+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:21.180986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157287+0000","last_change":"2024-03-18T19:06:05.176448+0000","last_active":"2024-03-18T19:06:06.157287+0000","last_peered":"2024-03-18T19:06:06.157287+0000","last_clean":"2024-03-18T19:06:06.157287+0000","last_became_active":"2024-03-18T19:06:05.176177+0000","last_became_peered":"2024-03-18T19:06:05.176177+0000","last_unstale":"2024-03-18T19:06:06.157287+0000","last_undegraded":"2024-03-18T19:06:06.157287+0000","last_fullsized":"2024-03-18T19:06:06.157287+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:45:28.072614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"102'8","reported_seq":19,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.156997+0000","last_change":"2024-03-18T19:06:04.737096+0000","last_active":"2024-03-18T19:06:06.156997+0000","last_peered":"2024-03-18T19:06:06.156997+0000","last_clean":"2024-03-18T19:06:06.156997+0000","last_became_active":"2024-03-18T19:06:04.736607+0000","last_became_peered":"2024-03-18T19:06:04.736607+0000","last_unstale":"2024-03-18T19:06:06.156997+0000","last_undegraded":"2024-03-18T19:06:06.156997+0000","last_fullsized":"2024-03-18T19:06:06.156997+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:11.573751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.180235+0000","last_change":"2024-03-18T19:06:05.176988+0000","last_active":"2024-03-18T19:06:05.180235+0000","last_peered":"2024-03-18T19:06:05.180235+0000","last_clean":"2024-03-18T19:06:05.180235+0000","last_became_active":"2024-03-18T19:06:05.176751+0000","last_became_peered":"2024-03-18T19:06:05.176751+0000","last_unstale":"2024-03-18T19:06:05.180235+0000","last_undegraded":"2024-03-18T19:06:05.180235+0000","last_fullsized":"2024-03-18T19:06:05.180235+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:37:35.397401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.167391+0000","last_change":"2024-03-18T19:06:04.729993+0000","last_active":"2024-03-18T19:06:05.167391+0000","last_peered":"2024-03-18T19:06:05.167391+0000","last_clean":"2024-03-18T19:06:05.167391+0000","last_became_active":"2024-03-18T19:06:04.729697+0000","last_became_peered":"2024-03-18T19:06:04.729697+0000","last_unstale":"2024-03-18T19:06:05.167391+0000","last_undegraded":"2024-03-18T19:06:05.167391+0000","last_fullsized":"2024-03-18T19:06:05.167391+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:38:45.615401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174395+0000","last_change":"2024-03-18T19:06:05.169243+0000","last_active":"2024-03-18T19:06:05.174395+0000","last_peered":"2024-03-18T19:06:05.174395+0000","last_clean":"2024-03-18T19:06:05.174395+0000","last_became_active":"2024-03-18T19:06:05.168996+0000","last_became_peered":"2024-03-18T19:06:05.168996+0000","last_unstale":"2024-03-18T19:06:05.174395+0000","last_undegraded":"2024-03-18T19:06:05.174395+0000","last_fullsized":"2024-03-18T19:06:05.174395+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:02.904163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.175112+0000","last_change":"2024-03-18T19:06:04.733348+0000","last_active":"2024-03-18T19:06:05.175112+0000","last_peered":"2024-03-18T19:06:05.175112+0000","last_clean":"2024-03-18T19:06:05.175112+0000","last_became_active":"2024-03-18T19:06:04.733087+0000","last_became_peered":"2024-03-18T19:06:04.733087+0000","last_unstale":"2024-03-18T19:06:05.175112+0000","last_undegraded":"2024-03-18T19:06:05.175112+0000","last_fullsized":"2024-03-18T19:06:05.175112+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:16:54.958882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174519+0000","last_change":"2024-03-18T19:06:05.167597+0000","last_active":"2024-03-18T19:06:05.174519+0000","last_peered":"2024-03-18T19:06:05.174519+0000","last_clean":"2024-03-18T19:06:05.174519+0000","last_became_active":"2024-03-18T19:06:05.167373+0000","last_became_peered":"2024-03-18T19:06:05.167373+0000","last_unstale":"2024-03-18T19:06:05.174519+0000","last_undegraded":"2024-03-18T19:06:05.174519+0000","last_fullsized":"2024-03-18T19:06:05.174519+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:51:11.010645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"102'1","reported_seq":11,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.172218+0000","last_change":"2024-03-18T19:06:04.733453+0000","last_active":"2024-03-18T19:06:05.172218+0000","last_peered":"2024-03-18T19:06:05.172218+0000","last_clean":"2024-03-18T19:06:05.172218+0000","last_became_active":"2024-03-18T19:06:04.732952+0000","last_became_peered":"2024-03-18T19:06:04.732952+0000","last_unstale":"2024-03-18T19:06:05.172218+0000","last_undegraded":"2024-03-18T19:06:05.172218+0000","last_fullsized":"2024-03-18T19:06:05.172218+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:52:54.267278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.180800+0000","last_change":"2024-03-18T19:06:05.176390+0000","last_active":"2024-03-18T19:06:05.180800+0000","last_peered":"2024-03-18T19:06:05.180800+0000","last_clean":"2024-03-18T19:06:05.180800+0000","last_became_active":"2024-03-18T19:06:05.176107+0000","last_became_peered":"2024-03-18T19:06:05.176107+0000","last_unstale":"2024-03-18T19:06:05.180800+0000","last_undegraded":"2024-03-18T19:06:05.180800+0000","last_fullsized":"2024-03-18T19:06:05.180800+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:57:28.462884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.157269+0000","last_change":"2024-03-18T19:06:04.729070+0000","last_active":"2024-03-18T19:06:05.157269+0000","last_peered":"2024-03-18T19:06:05.157269+0000","last_clean":"2024-03-18T19:06:05.157269+0000","last_became_active":"2024-03-18T19:06:04.728830+0000","last_became_peered":"2024-03-18T19:06:04.728830+0000","last_unstale":"2024-03-18T19:06:05.157269+0000","last_undegraded":"2024-03-18T19:06:05.157269+0000","last_fullsized":"2024-03-18T19:06:05.157269+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:40:51.689786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157235+0000","last_change":"2024-03-18T19:06:05.173107+0000","last_active":"2024-03-18T19:06:06.157235+0000","last_peered":"2024-03-18T19:06:06.157235+0000","last_clean":"2024-03-18T19:06:06.157235+0000","last_became_active":"2024-03-18T19:06:05.172838+0000","last_became_peered":"2024-03-18T19:06:05.172838+0000","last_unstale":"2024-03-18T19:06:06.157235+0000","last_undegraded":"2024-03-18T19:06:06.157235+0000","last_fullsized":"2024-03-18T19:06:06.157235+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:06.313685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.180996+0000","last_change":"2024-03-18T19:06:04.735737+0000","last_active":"2024-03-18T19:06:05.180996+0000","last_peered":"2024-03-18T19:06:05.180996+0000","last_clean":"2024-03-18T19:06:05.180996+0000","last_became_active":"2024-03-18T19:06:04.735358+0000","last_became_peered":"2024-03-18T19:06:04.735358+0000","last_unstale":"2024-03-18T19:06:05.180996+0000","last_undegraded":"2024-03-18T19:06:05.180996+0000","last_fullsized":"2024-03-18T19:06:05.180996+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:20:45.425752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157176+0000","last_change":"2024-03-18T19:06:05.171908+0000","last_active":"2024-03-18T19:06:06.157176+0000","last_peered":"2024-03-18T19:06:06.157176+0000","last_clean":"2024-03-18T19:06:06.157176+0000","last_became_active":"2024-03-18T19:06:05.171669+0000","last_became_peered":"2024-03-18T19:06:05.171669+0000","last_unstale":"2024-03-18T19:06:06.157176+0000","last_undegraded":"2024-03-18T19:06:06.157176+0000","last_fullsized":"2024-03-18T19:06:06.157176+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:45.511948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.161418+0000","last_change":"2024-03-18T19:06:04.736438+0000","last_active":"2024-03-18T19:06:05.161418+0000","last_peered":"2024-03-18T19:06:05.161418+0000","last_clean":"2024-03-18T19:06:05.161418+0000","last_became_active":"2024-03-18T19:06:04.736111+0000","last_became_peered":"2024-03-18T19:06:04.736111+0000","last_unstale":"2024-03-18T19:06:05.161418+0000","last_undegraded":"2024-03-18T19:06:05.161418+0000","last_fullsized":"2024-03-18T19:06:05.161418+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:56.438036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157371+0000","last_change":"2024-03-18T19:06:05.173451+0000","last_active":"2024-03-18T19:06:06.157371+0000","last_peered":"2024-03-18T19:06:06.157371+0000","last_clean":"2024-03-18T19:06:06.157371+0000","last_became_active":"2024-03-18T19:06:05.173171+0000","last_became_peered":"2024-03-18T19:06:05.173171+0000","last_unstale":"2024-03-18T19:06:06.157371+0000","last_undegraded":"2024-03-18T19:06:06.157371+0000","last_fullsized":"2024-03-18T19:06:06.157371+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:40:52.695671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.160706+0000","last_change":"2024-03-18T19:06:04.737213+0000","last_active":"2024-03-18T19:06:05.160706+0000","last_peered":"2024-03-18T19:06:05.160706+0000","last_clean":"2024-03-18T19:06:05.160706+0000","last_became_active":"2024-03-18T19:06:04.736846+0000","last_became_peered":"2024-03-18T19:06:04.736846+0000","last_unstale":"2024-03-18T19:06:05.160706+0000","last_undegraded":"2024-03-18T19:06:05.160706+0000","last_fullsized":"2024-03-18T19:06:05.160706+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:00.161012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.173870+0000","last_change":"2024-03-18T19:06:05.170880+0000","last_active":"2024-03-18T19:06:05.173870+0000","last_peered":"2024-03-18T19:06:05.173870+0000","last_clean":"2024-03-18T19:06:05.173870+0000","last_became_active":"2024-03-18T19:06:05.170552+0000","last_became_peered":"2024-03-18T19:06:05.170552+0000","last_unstale":"2024-03-18T19:06:05.173870+0000","last_undegraded":"2024-03-18T19:06:05.173870+0000","last_fullsized":"2024-03-18T19:06:05.173870+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:49:21.908439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"102'2","reported_seq":12,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.174733+0000","last_change":"2024-03-18T19:06:04.733203+0000","last_active":"2024-03-18T19:06:05.174733+0000","last_peered":"2024-03-18T19:06:05.174733+0000","last_clean":"2024-03-18T19:06:05.174733+0000","last_became_active":"2024-03-18T19:06:04.732922+0000","last_became_peered":"2024-03-18T19:06:04.732922+0000","last_unstale":"2024-03-18T19:06:05.174733+0000","last_undegraded":"2024-03-18T19:06:05.174733+0000","last_fullsized":"2024-03-18T19:06:05.174733+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:40:51.689786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.156881+0000","last_change":"2024-03-18T19:06:05.165633+0000","last_active":"2024-03-18T19:06:06.156881+0000","last_peered":"2024-03-18T19:06:06.156881+0000","last_clean":"2024-03-18T19:06:06.156881+0000","last_became_active":"2024-03-18T19:06:05.165174+0000","last_became_peered":"2024-03-18T19:06:05.165174+0000","last_unstale":"2024-03-18T19:06:06.156881+0000","last_undegraded":"2024-03-18T19:06:06.156881+0000","last_fullsized":"2024-03-18T19:06:06.156881+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:49:21.908439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157203+0000","last_change":"2024-03-18T19:06:04.727295+0000","last_active":"2024-03-18T19:06:06.157203+0000","last_peered":"2024-03-18T19:06:06.157203+0000","last_clean":"2024-03-18T19:06:06.157203+0000","last_became_active":"2024-03-18T19:06:04.726972+0000","last_became_peered":"2024-03-18T19:06:04.726972+0000","last_unstale":"2024-03-18T19:06:06.157203+0000","last_undegraded":"2024-03-18T19:06:06.157203+0000","last_fullsized":"2024-03-18T19:06:06.157203+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:41.776279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.184217+0000","last_change":"2024-03-18T19:06:05.183179+0000","last_active":"2024-03-18T19:06:05.184217+0000","last_peered":"2024-03-18T19:06:05.184217+0000","last_clean":"2024-03-18T19:06:05.184217+0000","last_became_active":"2024-03-18T19:06:05.182880+0000","last_became_peered":"2024-03-18T19:06:05.182880+0000","last_unstale":"2024-03-18T19:06:05.184217+0000","last_undegraded":"2024-03-18T19:06:05.184217+0000","last_fullsized":"2024-03-18T19:06:05.184217+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:30:49.975316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.154414+0000","last_change":"2024-03-18T19:06:04.732118+0000","last_active":"2024-03-18T19:06:05.154414+0000","last_peered":"2024-03-18T19:06:05.154414+0000","last_clean":"2024-03-18T19:06:05.154414+0000","last_became_active":"2024-03-18T19:06:04.731849+0000","last_became_peered":"2024-03-18T19:06:04.731849+0000","last_unstale":"2024-03-18T19:06:05.154414+0000","last_undegraded":"2024-03-18T19:06:05.154414+0000","last_fullsized":"2024-03-18T19:06:05.154414+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:09:39.307084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157240+0000","last_change":"2024-03-18T19:06:05.163264+0000","last_active":"2024-03-18T19:06:06.157240+0000","last_peered":"2024-03-18T19:06:06.157240+0000","last_clean":"2024-03-18T19:06:06.157240+0000","last_became_active":"2024-03-18T19:06:05.162967+0000","last_became_peered":"2024-03-18T19:06:05.162967+0000","last_unstale":"2024-03-18T19:06:06.157240+0000","last_undegraded":"2024-03-18T19:06:06.157240+0000","last_fullsized":"2024-03-18T19:06:06.157240+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:40:08.716187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.153804+0000","last_change":"2024-03-18T19:06:04.721220+0000","last_active":"2024-03-18T19:06:05.153804+0000","last_peered":"2024-03-18T19:06:05.153804+0000","last_clean":"2024-03-18T19:06:05.153804+0000","last_became_active":"2024-03-18T19:06:04.720956+0000","last_became_peered":"2024-03-18T19:06:04.720956+0000","last_unstale":"2024-03-18T19:06:05.153804+0000","last_undegraded":"2024-03-18T19:06:05.153804+0000","last_fullsized":"2024-03-18T19:06:05.153804+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:40:51.689786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157118+0000","last_change":"2024-03-18T19:06:05.174203+0000","last_active":"2024-03-18T19:06:06.157118+0000","last_peered":"2024-03-18T19:06:06.157118+0000","last_clean":"2024-03-18T19:06:06.157118+0000","last_became_active":"2024-03-18T19:06:05.173938+0000","last_became_peered":"2024-03-18T19:06:05.173938+0000","last_unstale":"2024-03-18T19:06:06.157118+0000","last_undegraded":"2024-03-18T19:06:06.157118+0000","last_fullsized":"2024-03-18T19:06:06.157118+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:24:32.554218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.156912+0000","last_change":"2024-03-18T19:06:04.729239+0000","last_active":"2024-03-18T19:06:06.156912+0000","last_peered":"2024-03-18T19:06:06.156912+0000","last_clean":"2024-03-18T19:06:06.156912+0000","last_became_active":"2024-03-18T19:06:04.726857+0000","last_became_peered":"2024-03-18T19:06:04.726857+0000","last_unstale":"2024-03-18T19:06:06.156912+0000","last_undegraded":"2024-03-18T19:06:06.156912+0000","last_fullsized":"2024-03-18T19:06:06.156912+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:11:44.520088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.157156+0000","last_change":"2024-03-18T19:06:05.168513+0000","last_active":"2024-03-18T19:06:06.157156+0000","last_peered":"2024-03-18T19:06:06.157156+0000","last_clean":"2024-03-18T19:06:06.157156+0000","last_became_active":"2024-03-18T19:06:05.168222+0000","last_became_peered":"2024-03-18T19:06:05.168222+0000","last_unstale":"2024-03-18T19:06:06.157156+0000","last_undegraded":"2024-03-18T19:06:06.157156+0000","last_fullsized":"2024-03-18T19:06:06.157156+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:38:46.621286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.161325+0000","last_change":"2024-03-18T19:06:04.736364+0000","last_active":"2024-03-18T19:06:05.161325+0000","last_peered":"2024-03-18T19:06:05.161325+0000","last_clean":"2024-03-18T19:06:05.161325+0000","last_became_active":"2024-03-18T19:06:04.736105+0000","last_became_peered":"2024-03-18T19:06:04.736105+0000","last_unstale":"2024-03-18T19:06:05.161325+0000","last_undegraded":"2024-03-18T19:06:05.161325+0000","last_fullsized":"2024-03-18T19:06:05.161325+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:08:47.425685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.175431+0000","last_change":"2024-03-18T19:06:05.173562+0000","last_active":"2024-03-18T19:06:05.175431+0000","last_peered":"2024-03-18T19:06:05.175431+0000","last_clean":"2024-03-18T19:06:05.175431+0000","last_became_active":"2024-03-18T19:06:05.171955+0000","last_became_peered":"2024-03-18T19:06:05.171955+0000","last_unstale":"2024-03-18T19:06:05.175431+0000","last_undegraded":"2024-03-18T19:06:05.175431+0000","last_fullsized":"2024-03-18T19:06:05.175431+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:34.217833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-18T19:06:06.156959+0000","last_change":"2024-03-18T19:06:04.732342+0000","last_active":"2024-03-18T19:06:06.156959+0000","last_peered":"2024-03-18T19:06:06.156959+0000","last_clean":"2024-03-18T19:06:06.156959+0000","last_became_active":"2024-03-18T19:06:04.732038+0000","last_became_peered":"2024-03-18T19:06:04.732038+0000","last_unstale":"2024-03-18T19:06:06.156959+0000","last_undegraded":"2024-03-18T19:06:06.156959+0000","last_fullsized":"2024-03-18T19:06:06.156959+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:38.050395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.173981+0000","last_change":"2024-03-18T19:06:05.164553+0000","last_active":"2024-03-18T19:06:05.173981+0000","last_peered":"2024-03-18T19:06:05.173981+0000","last_clean":"2024-03-18T19:06:05.173981+0000","last_became_active":"2024-03-18T19:06:05.164350+0000","last_became_peered":"2024-03-18T19:06:05.164350+0000","last_unstale":"2024-03-18T19:06:05.173981+0000","last_undegraded":"2024-03-18T19:06:05.173981+0000","last_fullsized":"2024-03-18T19:06:05.173981+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:07:30.773855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.157251+0000","last_change":"2024-03-18T19:06:04.736220+0000","last_active":"2024-03-18T19:06:05.157251+0000","last_peered":"2024-03-18T19:06:05.157251+0000","last_clean":"2024-03-18T19:06:05.157251+0000","last_became_active":"2024-03-18T19:06:04.735922+0000","last_became_peered":"2024-03-18T19:06:04.735922+0000","last_unstale":"2024-03-18T19:06:05.157251+0000","last_undegraded":"2024-03-18T19:06:05.157251+0000","last_fullsized":"2024-03-18T19:06:05.157251+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:34:07.772319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.168878+0000","last_change":"2024-03-18T19:06:05.167624+0000","last_active":"2024-03-18T19:06:05.168878+0000","last_peered":"2024-03-18T19:06:05.168878+0000","last_clean":"2024-03-18T19:06:05.168878+0000","last_became_active":"2024-03-18T19:06:05.167327+0000","last_became_peered":"2024-03-18T19:06:05.167327+0000","last_unstale":"2024-03-18T19:06:05.168878+0000","last_undegraded":"2024-03-18T19:06:05.168878+0000","last_fullsized":"2024-03-18T19:06:05.168878+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:04.710556+0000","last_clean_scrub_stamp":"2024-03-18T19:06:04.710556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:11:36.194047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-18T19:06:05.170373+0000","last_change":"2024-03-18T19:06:04.737285+0000","last_active":"2024-03-18T19:06:05.170373+0000","last_peered":"2024-03-18T19:06:05.170373+0000","last_clean":"2024-03-18T19:06:05.170373+0000","last_became_active":"2024-03-18T19:06:04.736911+0000","last_became_peered":"2024-03-18T19:06:04.736911+0000","last_unstale":"2024-03-18T19:06:05.170373+0000","last_undegraded":"2024-03-18T19:06:05.170373+0000","last_fullsized":"2024-03-18T19:06:05.170373+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:03.704671+0000","last_clean_scrub_stamp":"2024-03-18T19:06:03.704671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:27.449998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"92'172","reported_seq":436,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-18T19:06:04.715296+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T19:06:04.715296+0000","last_peered":"2024-03-18T19:06:04.715296+0000","last_clean":"2024-03-18T19:06:04.715296+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T19:06:04.715296+0000","last_undegraded":"2024-03-18T19:06:04.715296+0000","last_fullsized":"2024-03-18T19:06:04.715296+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172,"ondisk_log_size":172,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575042,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59168,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312672,"statfs":{"total":96636764160,"available":96576176128,"internally_reserved":0,"allocated":1847296,"data_stored":1213911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.55900000000000005,"15min":0.57099999999999995},"min":{"1min":0.39400000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.64100000000000001,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.56000000000000005,"15min":0.58199999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68700000000000006,"5min":1.387,"15min":1.387},"last":0.47099999999999997}]},{"osd":1,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.54000000000000004,"15min":0.57499999999999996},"min":{"1min":0.36499999999999999,"5min":0.32300000000000001,"15min":0.307},"max":{"1min":0.66200000000000003,"5min":0.81699999999999995,"15min":1.484},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.56499999999999995,"15min":0.58899999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.65600000000000003,"5min":0.80700000000000005,"15min":0.92000000000000004},"last":0.53100000000000003}]},{"osd":2,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55900000000000005,"15min":0.57799999999999996},"min":{"1min":0.42199999999999999,"5min":0.34499999999999997,"15min":0.27800000000000002},"max":{"1min":0.69799999999999995,"5min":1.3600000000000001,"15min":1.5129999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.59099999999999997},"min":{"1min":0.38300000000000001,"5min":0.36899999999999999,"15min":0.30199999999999999},"max":{"1min":0.73699999999999999,"5min":1.327,"15min":1.536},"last":0.44900000000000001}]},{"osd":3,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55200000000000005,"15min":0.57999999999999996},"min":{"1min":0.35599999999999998,"5min":0.318,"15min":0.311},"max":{"1min":0.61699999999999999,"5min":0.79100000000000004,"15min":1.5489999999999999},"last":0.5},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.57599999999999996,"15min":0.60099999999999998},"min":{"1min":0.40600000000000003,"5min":0.38800000000000001,"15min":0.38200000000000001},"max":{"1min":0.64900000000000002,"5min":0.78200000000000003,"15min":1.5660000000000001},"last":0.52000000000000002}]},{"osd":4,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.45400000000000001},"min":{"1min":0.311,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.59299999999999997,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.46999999999999997,"15min":0.47899999999999998},"min":{"1min":0.316,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.73199999999999998},"last":0.41699999999999998}]},{"osd":5,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.47799999999999998,"15min":0.47999999999999998},"min":{"1min":0.34200000000000003,"5min":0.27100000000000002,"15min":0.26000000000000001},"max":{"1min":0.71499999999999997,"5min":0.751,"15min":0.78300000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.504,"15min":0.51800000000000002},"min":{"1min":0.35699999999999998,"5min":0.30299999999999999,"15min":0.28799999999999998},"max":{"1min":0.751,"5min":0.78200000000000003,"15min":0.80300000000000005},"last":0.379}]},{"osd":6,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53500000000000003,"15min":0.54600000000000004},"min":{"1min":0.38600000000000001,"5min":0.374,"15min":0.34300000000000003},"max":{"1min":0.67700000000000005,"5min":0.81799999999999995,"15min":0.84499999999999997},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.35899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68700000000000006,"5min":0.752,"15min":0.77500000000000002},"last":0.46300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1273856,"data_stored":631705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56899999999999995,"15min":0.57199999999999995},"min":{"1min":0.33000000000000002,"5min":0.28499999999999998,"15min":0.25700000000000001},"max":{"1min":0.69799999999999995,"5min":0.75600000000000001,"15min":3.274},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56299999999999994,"15min":0.58099999999999996},"min":{"1min":0.37,"5min":0.34699999999999998,"15min":0.27400000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":3.3929999999999998},"last":0.47799999999999998}]},{"osd":1,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57699999999999996,"15min":0.56799999999999995},"min":{"1min":0.38100000000000001,"5min":0.316,"15min":0.30599999999999999},"max":{"1min":0.73899999999999999,"5min":0.77400000000000002,"15min":0.94099999999999995},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56200000000000006,"15min":0.56999999999999995},"min":{"1min":0.42699999999999999,"5min":0.32000000000000001,"15min":0.311},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.90700000000000003},"last":0.66200000000000003}]},{"osd":2,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58399999999999996,"15min":0.57599999999999996},"min":{"1min":0.42799999999999999,"5min":0.32700000000000001,"15min":0.29999999999999999},"max":{"1min":0.75700000000000001,"5min":0.80300000000000005,"15min":0.81299999999999994},"last":0.625},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57199999999999995,"15min":0.58299999999999996},"min":{"1min":0.36699999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.74199999999999999,"5min":0.83199999999999996,"15min":2.2370000000000001},"last":0.60999999999999999}]},{"osd":3,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59199999999999997,"15min":0.58499999999999996},"min":{"1min":0.41199999999999998,"5min":0.40000000000000002,"15min":0.32100000000000001},"max":{"1min":0.75600000000000001,"5min":0.89600000000000002,"15min":1.0369999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.373,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":1.595},"last":0.63900000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.34000000000000002,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.76300000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47599999999999998,"15min":0.48099999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":1.8819999999999999},"last":0.67500000000000004}]},{"osd":5,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.46899999999999997,"15min":0.47399999999999998},"min":{"1min":0.33600000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":1.022},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51200000000000001,"15min":0.51600000000000001},"min":{"1min":0.35699999999999998,"5min":0.35499999999999998,"15min":0.32200000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.81599999999999995},"last":0.72099999999999997}]},{"osd":7,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.52700000000000002,"15min":0.52600000000000002},"min":{"1min":0.34300000000000003,"5min":0.34100000000000003,"15min":0.313},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":1.532},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53000000000000003,"15min":0.53300000000000003},"min":{"1min":0.39500000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":1.978},"last":0.73399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36448,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335392,"statfs":{"total":96636764160,"available":96599441408,"internally_reserved":0,"allocated":1257472,"data_stored":623633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.375,"15min":0.373},"min":{"1min":0.23200000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.83599999999999997},"last":0.217},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42499999999999999,"15min":0.41499999999999998},"min":{"1min":0.25700000000000001,"5min":0.23799999999999999,"15min":0.23400000000000001},"max":{"1min":0.65200000000000002,"5min":0.73699999999999999,"15min":0.90400000000000003},"last":0.27800000000000002}]},{"osd":2,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41599999999999998,"15min":0.40400000000000003},"min":{"1min":0.27800000000000002,"5min":0.26500000000000001,"15min":0.23300000000000001},"max":{"1min":0.56699999999999995,"5min":0.65400000000000003,"15min":0.88400000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.45300000000000001,"15min":0.44500000000000001},"min":{"1min":0.34300000000000003,"5min":0.29699999999999999,"15min":0.23499999999999999},"max":{"1min":0.60699999999999998,"5min":0.68700000000000006,"15min":0.94999999999999996},"last":0.317}]},{"osd":3,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.27700000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.86799999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.42799999999999999,"15min":0.42099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28100000000000003,"15min":0.25600000000000001},"max":{"1min":0.65100000000000002,"5min":0.77000000000000002,"15min":0.93100000000000005},"last":0.23699999999999999}]},{"osd":4,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63800000000000001,"15min":0.63300000000000001},"min":{"1min":0.33400000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.92300000000000004,"5min":0.93999999999999995,"15min":1.052},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.63800000000000001,"15min":0.64400000000000002},"min":{"1min":0.46800000000000003,"5min":0.36899999999999999,"15min":0.34499999999999997},"max":{"1min":0.95799999999999996,"5min":1.018,"15min":1.0269999999999999},"last":0.52400000000000002}]},{"osd":5,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63300000000000001,"15min":0.64200000000000002},"min":{"1min":0.434,"5min":0.40100000000000002,"15min":0.38600000000000001},"max":{"1min":0.877,"5min":0.88500000000000001,"15min":1.0760000000000001},"last":0.504},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.65600000000000003,"15min":0.65700000000000003},"min":{"1min":0.47099999999999997,"5min":0.41099999999999998,"15min":0.40999999999999998},"max":{"1min":0.80900000000000005,"5min":1.006,"15min":1.1319999999999999},"last":0.59899999999999998}]},{"osd":6,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.93500000000000005,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67500000000000004,"15min":0.66900000000000004},"min":{"1min":0.434,"5min":0.42599999999999999,"15min":0.39100000000000001},"max":{"1min":0.93799999999999994,"5min":0.96799999999999997,"15min":1.0089999999999999},"last":0.628}]},{"osd":7,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66000000000000003},"min":{"1min":0.48599999999999999,"5min":0.44900000000000001,"15min":0.36699999999999999},"max":{"1min":0.90600000000000003,"5min":0.92900000000000005,"15min":1.0660000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66700000000000004,"15min":0.66400000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.39900000000000002},"max":{"1min":0.88100000000000001,"5min":0.94299999999999995,"15min":1.1499999999999999},"last":0.55100000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575043,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37020,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334820,"statfs":{"total":96636764160,"available":96598855680,"internally_reserved":0,"allocated":1843200,"data_stored":1213889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.377,"15min":0.377},"min":{"1min":0.23100000000000001,"5min":0.19400000000000001,"15min":0.192},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.621},"last":0.31},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.30299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.55400000000000005,"5min":0.61699999999999999,"15min":0.66400000000000003},"last":0.26900000000000002}]},{"osd":2,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41599999999999998,"15min":0.41699999999999998},"min":{"1min":0.27100000000000002,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.58999999999999997,"5min":0.60499999999999998,"15min":0.627},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.30499999999999999,"5min":0.25,"15min":0.25},"max":{"1min":0.627,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.32600000000000001}]},{"osd":3,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.40699999999999997,"15min":0.42999999999999999},"min":{"1min":0.30299999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.57499999999999996,"5min":0.61299999999999999,"15min":5.7359999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46500000000000002,"15min":0.48799999999999999},"min":{"1min":0.32000000000000001,"5min":0.28799999999999998,"15min":0.27700000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":5.9329999999999998},"last":0.33900000000000002}]},{"osd":4,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.63800000000000001,"15min":0.63900000000000001},"min":{"1min":0.49099999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.91800000000000004,"5min":1.0229999999999999,"15min":1.22},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.45900000000000002,"5min":0.315,"15min":0.315},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.1020000000000001},"last":0.68200000000000005}]},{"osd":5,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.47999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.96199999999999997,"5min":0.98099999999999998,"15min":2.9649999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.65600000000000003,"15min":0.65800000000000003},"min":{"1min":0.38,"5min":0.31,"15min":0.31},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.2609999999999999},"last":0.70699999999999996}]},{"osd":6,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.64900000000000002,"15min":0.65300000000000002},"min":{"1min":0.40500000000000003,"5min":0.39400000000000002,"15min":0.38100000000000001},"max":{"1min":0.90100000000000002,"5min":0.94599999999999995,"15min":1.133},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.66500000000000004,"15min":0.67600000000000005},"min":{"1min":0.42699999999999999,"5min":0.41199999999999998,"15min":0.40999999999999998},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":2.9300000000000002},"last":0.64000000000000001}]},{"osd":7,"last update":"Mon Mar 18 19:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.66600000000000004,"15min":0.66700000000000004},"min":{"1min":0.51900000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.2849999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.66500000000000004,"15min":0.67000000000000004},"min":{"1min":0.44,"5min":0.437,"15min":0.38500000000000001},"max":{"1min":0.94299999999999995,"5min":0.96299999999999997,"15min":1.002},"last":0.53300000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575043,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58596,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94313244,"statfs":{"total":96636764160,"available":96576761856,"internally_reserved":0,"allocated":1261568,"data_stored":624263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38200000000000001,"15min":0.39500000000000002},"min":{"1min":0.249,"5min":0.216,"15min":0.17799999999999999},"max":{"1min":0.66600000000000004,"5min":0.78300000000000003,"15min":9.1219999999999999},"last":0.252},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.46500000000000002,"15min":0.47599999999999998},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.68899999999999995,"5min":0.77200000000000002,"15min":9.1560000000000006},"last":0.41599999999999998}]},{"osd":1,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47999999999999998,"15min":0.49299999999999999},"min":{"1min":0.376,"5min":0.29399999999999998,"15min":0.27100000000000002},"max":{"1min":0.69499999999999995,"5min":0.90500000000000003,"15min":9.2460000000000004},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41299999999999998,"15min":0.42499999999999999},"min":{"1min":0.23899999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.61399999999999999,"5min":0.66700000000000004,"15min":12.932},"last":0.33100000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.42199999999999999,"15min":0.42799999999999999},"min":{"1min":0.29799999999999999,"5min":0.24199999999999999,"15min":0.22500000000000001},"max":{"1min":0.58599999999999997,"5min":1.4019999999999999,"15min":9.4120000000000008},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51300000000000001,"15min":0.52000000000000002},"min":{"1min":0.33800000000000002,"5min":0.29499999999999998,"15min":0.27700000000000002},"max":{"1min":0.67100000000000004,"5min":0.80100000000000005,"15min":9.2680000000000007},"last":0.38300000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63,"15min":0.61799999999999999},"min":{"1min":0.39400000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.98199999999999998,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63400000000000001,"15min":0.65200000000000002},"min":{"1min":0.441,"5min":0.35899999999999999,"15min":0.32100000000000001},"max":{"1min":0.874,"5min":1.4730000000000001,"15min":9.3190000000000008},"last":0.51400000000000001}]},{"osd":5,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65300000000000002,"15min":0.66800000000000004},"min":{"1min":0.45300000000000001,"5min":0.34699999999999998,"15min":0.309},"max":{"1min":0.89600000000000002,"5min":1.8109999999999999,"15min":9.1940000000000008},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65800000000000003,"15min":0.64000000000000001},"min":{"1min":0.40000000000000002,"5min":0.36799999999999999,"15min":0.33400000000000002},"max":{"1min":0.90300000000000002,"5min":1.948,"15min":1.948},"last":0.48999999999999999}]},{"osd":6,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.67700000000000005,"15min":0.67900000000000005},"min":{"1min":0.40999999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.92300000000000004,"5min":2.9470000000000001,"15min":9.3559999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69299999999999995,"15min":0.70899999999999996},"min":{"1min":0.48899999999999999,"5min":0.38100000000000001,"15min":0.37},"max":{"1min":1.0509999999999999,"5min":1.083,"15min":9.2219999999999995},"last":0.624}]},{"osd":7,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.66300000000000003,"15min":0.65200000000000002},"min":{"1min":0.42599999999999999,"5min":0.40999999999999998,"15min":0.38200000000000001},"max":{"1min":0.93600000000000005,"5min":1.083,"15min":1.083},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66800000000000004,"15min":0.68400000000000005},"min":{"1min":0.51800000000000002,"5min":0.371,"15min":0.34599999999999997},"max":{"1min":1.032,"5min":1.4350000000000001,"15min":9.3829999999999991},"last":0.61199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575043,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36444,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335396,"statfs":{"total":96636764160,"available":96599445504,"internally_reserved":0,"allocated":1253376,"data_stored":624103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.36899999999999999,"15min":0.379},"min":{"1min":0.23699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.51200000000000001,"5min":0.81100000000000005,"15min":1.204},"last":0.221},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.40400000000000003,"15min":0.39600000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.23000000000000001},"max":{"1min":0.57499999999999996,"5min":1.137,"15min":1.137},"last":0.32500000000000001}]},{"osd":1,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40000000000000002,"15min":0.39700000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.221},"max":{"1min":0.621,"5min":0.999,"15min":1.1220000000000001},"last":0.248},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.437,"15min":0.443},"min":{"1min":0.29999999999999999,"5min":0.28499999999999998,"15min":0.26500000000000001},"max":{"1min":0.51900000000000002,"5min":1.093,"15min":1.1080000000000001},"last":0.26300000000000001}]},{"osd":2,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.318,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.67500000000000004,"5min":1.0369999999999999,"15min":1.1879999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.46300000000000002,"15min":0.47499999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.28599999999999998},"max":{"1min":0.55600000000000005,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.34100000000000003}]},{"osd":4,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.052,"5min":1.4219999999999999,"15min":1.871},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.61199999999999999,"15min":0.63},"min":{"1min":0.42699999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78300000000000003,"5min":1.0700000000000001,"15min":1.764},"last":0.433}]},{"osd":5,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.65600000000000003,"15min":0.65900000000000003},"min":{"1min":0.44400000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.0249999999999999,"5min":2.5329999999999999,"15min":2.5329999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67000000000000004,"15min":0.68400000000000005},"min":{"1min":0.47099999999999997,"5min":0.45500000000000002,"15min":0.437},"max":{"1min":0.94599999999999995,"5min":1.5409999999999999,"15min":1.794},"last":0.55000000000000004}]},{"osd":6,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.66700000000000004,"15min":0.67200000000000004},"min":{"1min":0.44400000000000001,"5min":0.41499999999999998,"15min":0.40300000000000002},"max":{"1min":1.1060000000000001,"5min":2.7389999999999999,"15min":2.7389999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.66100000000000003,"15min":0.67500000000000004},"min":{"1min":0.47799999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96699999999999997,"5min":1.159,"15min":1.8069999999999999},"last":0.54100000000000004}]},{"osd":7,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.64400000000000002,"15min":0.66900000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86499999999999999,"5min":1.3959999999999999,"15min":1.732},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.66800000000000004,"15min":0.67000000000000004},"min":{"1min":0.47399999999999998,"5min":0.439,"15min":0.40500000000000003},"max":{"1min":1.085,"5min":2.7050000000000001,"15min":2.7050000000000001},"last":0.51700000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575043,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32364,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339476,"statfs":{"total":96636764160,"available":96603623424,"internally_reserved":0,"allocated":1269760,"data_stored":625189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.246},"max":{"1min":0.73299999999999998,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56399999999999995,"15min":0.58099999999999996},"min":{"1min":0.40300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.76100000000000001,"5min":0.77600000000000002,"15min":1.0860000000000001},"last":0.42599999999999999}]},{"osd":1,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57399999999999995,"15min":0.58399999999999996},"min":{"1min":0.439,"5min":0.34499999999999997,"15min":0.316},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":1.1679999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55700000000000005,"15min":0.57499999999999996},"min":{"1min":0.41599999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":1.218},"last":0.34899999999999998}]},{"osd":2,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56299999999999994,"15min":0.57299999999999995},"min":{"1min":0.45000000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72699999999999998,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57399999999999995,"15min":0.59899999999999998},"min":{"1min":0.40999999999999998,"5min":0.36799999999999999,"15min":0.32600000000000001},"max":{"1min":0.77200000000000002,"5min":0.78100000000000003,"15min":1.204},"last":0.51100000000000001}]},{"osd":3,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56899999999999995,"15min":0.58599999999999997},"min":{"1min":0.439,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.74099999999999999,"5min":0.75600000000000001,"15min":1.1799999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57799999999999996,"15min":0.59599999999999997},"min":{"1min":0.41799999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67700000000000005,"5min":1.121,"15min":1.121},"last":0.52200000000000002}]},{"osd":5,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.44600000000000001,"15min":0.46500000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.245},"max":{"1min":0.55900000000000005,"5min":0.73499999999999999,"15min":0.78900000000000003},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.435,"5min":0.46100000000000002,"15min":0.48299999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.27200000000000002},"max":{"1min":0.626,"5min":0.64400000000000002,"15min":0.77800000000000002},"last":0.499}]},{"osd":6,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.46800000000000003,"15min":0.48499999999999999},"min":{"1min":0.33900000000000002,"5min":0.32700000000000001,"15min":0.29499999999999998},"max":{"1min":0.66500000000000004,"5min":0.68600000000000005,"15min":0.81799999999999995},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.497,"15min":0.52300000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.28999999999999998},"max":{"1min":0.59799999999999998,"5min":0.72999999999999998,"15min":0.88900000000000001},"last":0.53700000000000003}]},{"osd":7,"last update":"Mon Mar 18 19:05:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.498,"15min":0.52100000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.628,"5min":0.70899999999999996,"15min":0.75800000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52600000000000002,"15min":0.55300000000000005},"min":{"1min":0.38300000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70599999999999996,"5min":0.75900000000000001,"15min":0.91900000000000004},"last":0.48599999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50392,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321448,"statfs":{"total":96636764160,"available":96585162752,"internally_reserved":0,"allocated":1253376,"data_stored":623555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56899999999999995,"15min":0.58599999999999997},"min":{"1min":0.38300000000000001,"5min":0.26800000000000002,"15min":0.24399999999999999},"max":{"1min":0.77000000000000002,"5min":0.78400000000000003,"15min":6.3079999999999998},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57899999999999996,"15min":0.57299999999999995},"min":{"1min":0.373,"5min":0.33700000000000002,"15min":0.25800000000000001},"max":{"1min":0.78800000000000003,"5min":0.79800000000000004,"15min":1.208},"last":0.57299999999999995}]},{"osd":1,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.32900000000000001,"5min":0.32600000000000001,"15min":0.28399999999999997},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.29799999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":6.2549999999999999},"last":0.70499999999999996}]},{"osd":2,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.58599999999999997,"15min":0.59699999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.28399999999999997},"max":{"1min":1.153,"5min":1.153,"15min":6.2999999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56499999999999995,"15min":0.623},"min":{"1min":0.39200000000000002,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":0.73999999999999999,"5min":0.748,"15min":17.387},"last":0.56000000000000005}]},{"osd":3,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.35799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":6.3330000000000002},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.212,"5min":1.212,"15min":1.2390000000000001},"last":0.54600000000000004}]},{"osd":4,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.47099999999999997,"15min":0.46800000000000003},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.441},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.497,"15min":0.495},"min":{"1min":0.32700000000000001,"5min":0.29499999999999998,"15min":0.23100000000000001},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.49299999999999999}]},{"osd":6,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.46700000000000003,"15min":0.47899999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.70999999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.504,"15min":0.51000000000000001},"min":{"1min":0.32100000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66300000000000003,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.50900000000000001}]},{"osd":7,"last update":"Mon Mar 18 19:05:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.53100000000000003},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55300000000000005,"15min":0.55200000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.28499999999999998},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.47599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T19:06:09.708 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T19:06:09.708 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-18T19:06:09.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.881+0000 7f214b034700 1 Processor -- start 2024-03-18T19:06:09.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.881+0000 7f214b034700 1 -- start start 2024-03-18T19:06:09.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.882+0000 7f214b034700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21440aecb0 0x7f2144070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:09.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.882+0000 7f214b034700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2144071380 0x7f214406d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:09.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.882+0000 7f214b034700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214415b750 con 0x7f214415b370 2024-03-18T19:06:09.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.882+0000 7f214b034700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21440cf510 con 0x7f2144071380 2024-03-18T19:06:09.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.882+0000 7f214b034700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2144071830 con 0x7f21440aecb0 2024-03-18T19:06:09.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.882+0000 7f21495d1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2144071380 0x7f214406d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:09.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.882+0000 7f2148dd0700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f214415b370 0x7f21440cec70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50970/0 (socket says 172.21.15.1:50970) 2024-03-18T19:06:09.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.882+0000 7f2148dd0700 1 -- 172.21.15.1:0/119825440 learned_addr learned my addr 172.21.15.1:0/119825440 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:09.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.882+0000 7f2143fff700 1 --2- 172.21.15.1:0/119825440 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21440aecb0 0x7f2144070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:09.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.882+0000 7f21495d1700 1 -- 172.21.15.1:0/119825440 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21440aecb0 msgr2=0x7f2144070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:09.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.882+0000 7f21495d1700 1 --2- 172.21.15.1:0/119825440 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21440aecb0 0x7f2144070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.882+0000 7f21495d1700 1 -- 172.21.15.1:0/119825440 >> v1:172.21.15.1:6789/0 conn(0x7f214415b370 legacy=0x7f21440cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:09.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.882+0000 7f21495d1700 1 -- 172.21.15.1:0/119825440 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21440ccc20 con 0x7f2144071380 2024-03-18T19:06:09.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.882+0000 7f21437fe700 1 -- 172.21.15.1:0/119825440 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243021750 0 0) 0x7f214415b750 con 0x7f214415b370 2024-03-18T19:06:09.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.882+0000 7f21495d1700 1 --2- 172.21.15.1:0/119825440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2144071380 0x7f214406d080 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f213c009d90 tx=0x7f213c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16db71f6e6c0af7f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:09.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.883+0000 7f21437fe700 1 -- 172.21.15.1:0/119825440 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c00e070 con 0x7f2144071380 2024-03-18T19:06:09.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.883+0000 7f21437fe700 1 -- 172.21.15.1:0/119825440 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213c00bb20 con 0x7f2144071380 2024-03-18T19:06:09.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.883+0000 7f21437fe700 1 -- 172.21.15.1:0/119825440 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c012430 con 0x7f2144071380 2024-03-18T19:06:09.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.883+0000 7f214b034700 1 -- 172.21.15.1:0/119825440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2144071380 msgr2=0x7f214406d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:09.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.883+0000 7f214b034700 1 --2- 172.21.15.1:0/119825440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2144071380 0x7f214406d080 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f213c009d90 tx=0x7f213c00b330 comp rx=0 tx=0).stop 2024-03-18T19:06:09.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.883+0000 7f214b034700 1 -- 172.21.15.1:0/119825440 shutdown_connections 2024-03-18T19:06:09.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.883+0000 7f214b034700 1 --2- 172.21.15.1:0/119825440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2144071380 0x7f214406d080 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.883+0000 7f214b034700 1 --2- 172.21.15.1:0/119825440 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21440aecb0 0x7f2144070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.883+0000 7f214b034700 1 -- 172.21.15.1:0/119825440 >> 172.21.15.1:0/119825440 conn(0x7f21440b1350 msgr2=0x7f21440b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:09.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.883+0000 7f214b034700 1 -- 172.21.15.1:0/119825440 shutdown_connections 2024-03-18T19:06:09.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.883+0000 7f214b034700 1 -- 172.21.15.1:0/119825440 wait complete. 2024-03-18T19:06:09.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.884+0000 7f214b034700 1 Processor -- start 2024-03-18T19:06:09.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.884+0000 7f214b034700 1 -- start start 2024-03-18T19:06:09.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.885+0000 7f214b034700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2144071380 0x7f2144156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.885+0000 7f214b034700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21440aecb0 0x7f21440ac010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.885+0000 7f214b034700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f214415b370 0x7f21440af9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.885+0000 7f214b034700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21441f0ec0 con 0x7f21440aecb0 2024-03-18T19:06:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.885+0000 7f214b034700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21441f1180 con 0x7f2144071380 2024-03-18T19:06:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.885+0000 7f214b034700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21441f1460 con 0x7f214415b370 2024-03-18T19:06:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.885+0000 7f2143fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21440aecb0 0x7f21440ac010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.885+0000 7f2143fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21440aecb0 0x7f21440ac010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40942/0 (socket says 172.21.15.1:40942) 2024-03-18T19:06:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.885+0000 7f2143fff700 1 -- 172.21.15.1:0/1475645495 learned_addr learned my addr 172.21.15.1:0/1475645495 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.886+0000 7f2143fff700 1 -- 172.21.15.1:0/1475645495 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f214415b370 msgr2=0x7f21440af9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.886+0000 7f2143fff700 1 --2- 172.21.15.1:0/1475645495 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f214415b370 0x7f21440af9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.886+0000 7f2143fff700 1 -- 172.21.15.1:0/1475645495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2144071380 msgr2=0x7f2144156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.886+0000 7f2143fff700 1 --2- 172.21.15.1:0/1475645495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2144071380 0x7f2144156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.886+0000 7f2143fff700 1 -- 172.21.15.1:0/1475645495 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f213c009a70 con 0x7f21440aecb0 2024-03-18T19:06:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.886+0000 7f2143fff700 1 --2- 172.21.15.1:0/1475645495 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21440aecb0 0x7f21440ac010 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f213000e9b0 tx=0x7f213000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:09.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.886+0000 7f21417fa700 1 -- 172.21.15.1:0/1475645495 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2130009710 con 0x7f21440aecb0 2024-03-18T19:06:09.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.886+0000 7f21417fa700 1 -- 172.21.15.1:0/1475645495 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213000c850 con 0x7f21440aecb0 2024-03-18T19:06:09.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.886+0000 7f214b034700 1 -- 172.21.15.1:0/1475645495 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21441f1740 con 0x7f21440aecb0 2024-03-18T19:06:09.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.886+0000 7f21417fa700 1 -- 172.21.15.1:0/1475645495 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213000cbc0 con 0x7f21440aecb0 2024-03-18T19:06:09.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.886+0000 7f214b034700 1 -- 172.21.15.1:0/1475645495 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2144156880 con 0x7f21440aecb0 2024-03-18T19:06:09.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.887+0000 7f214b034700 1 -- 172.21.15.1:0/1475645495 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21441f1d90 con 0x7f21440aecb0 2024-03-18T19:06:09.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.888+0000 7f21417fa700 1 -- 172.21.15.1:0/1475645495 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f213000cd20 con 0x7f21440aecb0 2024-03-18T19:06:09.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.888+0000 7f21417fa700 1 --2- 172.21.15.1:0/1475645495 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f212806adb0 0x7f212806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:09.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.888+0000 7f21417fa700 1 -- 172.21.15.1:0/1475645495 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2130089650 con 0x7f21440aecb0 2024-03-18T19:06:09.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.888+0000 7f2148dd0700 1 --2- 172.21.15.1:0/1475645495 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f212806adb0 0x7f212806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:09.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.889+0000 7f2148dd0700 1 --2- 172.21.15.1:0/1475645495 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f212806adb0 0x7f212806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21440a9130 tx=0x7f21380079f0 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:09.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:09.893+0000 7f21417fa700 1 -- 172.21.15.1:0/1475645495 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2130054670 con 0x7f21440aecb0 2024-03-18T19:06:10.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.177+0000 7f214b034700 1 -- 172.21.15.1:0/1475645495 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2144072dd0 con 0x7f21440aecb0 2024-03-18T19:06:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.180+0000 7f21417fa700 1 -- 172.21.15.1:0/1475645495 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+0 (secure 0 0 0) 0x7f2130087e60 con 0x7f21440aecb0 2024-03-18T19:06:10.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.182+0000 7f214b034700 1 -- 172.21.15.1:0/1475645495 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f212806adb0 msgr2=0x7f212806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:10.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.182+0000 7f214b034700 1 --2- 172.21.15.1:0/1475645495 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f212806adb0 0x7f212806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21440a9130 tx=0x7f21380079f0 comp rx=0 tx=0).stop 2024-03-18T19:06:10.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.182+0000 7f214b034700 1 -- 172.21.15.1:0/1475645495 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21440aecb0 msgr2=0x7f21440ac010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:10.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.182+0000 7f214b034700 1 --2- 172.21.15.1:0/1475645495 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21440aecb0 0x7f21440ac010 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f213000e9b0 tx=0x7f213000ed70 comp rx=0 tx=0).stop 2024-03-18T19:06:10.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.182+0000 7f214b034700 1 -- 172.21.15.1:0/1475645495 shutdown_connections 2024-03-18T19:06:10.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.182+0000 7f214b034700 1 --2- 172.21.15.1:0/1475645495 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f212806adb0 0x7f212806d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:10.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.182+0000 7f214b034700 1 --2- 172.21.15.1:0/1475645495 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f214415b370 0x7f21440af9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:10.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.182+0000 7f214b034700 1 --2- 172.21.15.1:0/1475645495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2144071380 0x7f2144156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:10.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.182+0000 7f214b034700 1 --2- 172.21.15.1:0/1475645495 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21440aecb0 0x7f21440ac010 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:10.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.182+0000 7f214b034700 1 -- 172.21.15.1:0/1475645495 >> 172.21.15.1:0/1475645495 conn(0x7f21440b1350 msgr2=0x7f214415d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:10.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.182+0000 7f214b034700 1 -- 172.21.15.1:0/1475645495 shutdown_connections 2024-03-18T19:06:10.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.182+0000 7f214b034700 1 -- 172.21.15.1:0/1475645495 wait complete. 2024-03-18T19:06:10.180 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-18T19:06:10.190 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:06:10.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.319+0000 7f81d9868700 1 Processor -- start 2024-03-18T19:06:10.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.319+0000 7f81d9868700 1 -- start start 2024-03-18T19:06:10.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.320+0000 7f81d9868700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f81d415d470 0x7f81d40cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:10.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.320+0000 7f81d9868700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81d40d10d0 0x7f81d40cf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:10.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.320+0000 7f81d9868700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d40ce7e0 con 0x7f81d415d850 2024-03-18T19:06:10.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.320+0000 7f81d9868700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d40ceac0 con 0x7f81d40d10d0 2024-03-18T19:06:10.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.320+0000 7f81d9868700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d40ceda0 con 0x7f81d415d470 2024-03-18T19:06:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.320+0000 7f81d37fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81d40d10d0 0x7f81d40cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.320+0000 7f81d37fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81d40d10d0 0x7f81d40cf880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40946/0 (socket says 172.21.15.1:40946) 2024-03-18T19:06:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.320+0000 7f81d37fe700 1 -- 172.21.15.1:0/3718088152 learned_addr learned my addr 172.21.15.1:0/3718088152 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.320+0000 7f81d27fc700 1 --1- 172.21.15.1:0/3718088152 >> v1:172.21.15.1:6789/0 conn(0x7f81d415d850 0x7f81d40d17d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50976/0 (socket says 172.21.15.1:50976) 2024-03-18T19:06:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.320+0000 7f81d2ffd700 1 --2- 172.21.15.1:0/3718088152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f81d415d470 0x7f81d40cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.320+0000 7f81d37fe700 1 -- 172.21.15.1:0/3718088152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f81d415d470 msgr2=0x7f81d40cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.320+0000 7f81d37fe700 1 --2- 172.21.15.1:0/3718088152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f81d415d470 0x7f81d40cdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.320+0000 7f81d37fe700 1 -- 172.21.15.1:0/3718088152 >> v1:172.21.15.1:6789/0 conn(0x7f81d415d850 legacy=0x7f81d40d17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.320+0000 7f81d37fe700 1 -- 172.21.15.1:0/3718088152 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d40cf080 con 0x7f81d40d10d0 2024-03-18T19:06:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.320+0000 7f81d1ffb700 1 -- 172.21.15.1:0/3718088152 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41001241 0 0) 0x7f81d40ce7e0 con 0x7f81d415d850 2024-03-18T19:06:10.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.320+0000 7f81d37fe700 1 --2- 172.21.15.1:0/3718088152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81d40d10d0 0x7f81d40cf880 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f81cc009e50 tx=0x7f81cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47ab92fb83d07f50 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:10.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.320+0000 7f81d1ffb700 1 -- 172.21.15.1:0/3718088152 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc00e070 con 0x7f81d40d10d0 2024-03-18T19:06:10.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.320+0000 7f81d1ffb700 1 -- 172.21.15.1:0/3718088152 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81cc00b9c0 con 0x7f81d40d10d0 2024-03-18T19:06:10.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.320+0000 7f81d1ffb700 1 -- 172.21.15.1:0/3718088152 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc00bd70 con 0x7f81d40d10d0 2024-03-18T19:06:10.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.321+0000 7f81d9868700 1 -- 172.21.15.1:0/3718088152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81d40d10d0 msgr2=0x7f81d40cf880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:10.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.321+0000 7f81d9868700 1 --2- 172.21.15.1:0/3718088152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81d40d10d0 0x7f81d40cf880 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f81cc009e50 tx=0x7f81cc00b330 comp rx=0 tx=0).stop 2024-03-18T19:06:10.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.321+0000 7f81d9868700 1 -- 172.21.15.1:0/3718088152 shutdown_connections 2024-03-18T19:06:10.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.321+0000 7f81d9868700 1 --2- 172.21.15.1:0/3718088152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81d40d10d0 0x7f81d40cf880 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:10.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.321+0000 7f81d9868700 1 --2- 172.21.15.1:0/3718088152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f81d415d470 0x7f81d40cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:10.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.321+0000 7f81d9868700 1 -- 172.21.15.1:0/3718088152 >> 172.21.15.1:0/3718088152 conn(0x7f81d408cc10 msgr2=0x7f81d409c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:10.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.321+0000 7f81d9868700 1 -- 172.21.15.1:0/3718088152 shutdown_connections 2024-03-18T19:06:10.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.321+0000 7f81d9868700 1 -- 172.21.15.1:0/3718088152 wait complete. 2024-03-18T19:06:10.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.322+0000 7f81d9868700 1 Processor -- start 2024-03-18T19:06:10.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.322+0000 7f81d9868700 1 -- start start 2024-03-18T19:06:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.323+0000 7f81d9868700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f81d40d10d0 0x7f81d415b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.323+0000 7f81d9868700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81d415d470 0x7f81d40cb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.323+0000 7f81d9868700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81d40cb9c0 0x7f81d40b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.323+0000 7f81d9868700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d40cb2d0 con 0x7f81d415d470 2024-03-18T19:06:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.323+0000 7f81d9868700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d415b800 con 0x7f81d40cb9c0 2024-03-18T19:06:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.323+0000 7f81d9868700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d41f6130 con 0x7f81d40d10d0 2024-03-18T19:06:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.323+0000 7f81d27fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81d415d470 0x7f81d40cb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.323+0000 7f81d27fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81d415d470 0x7f81d40cb480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40962/0 (socket says 172.21.15.1:40962) 2024-03-18T19:06:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.323+0000 7f81d27fc700 1 -- 172.21.15.1:0/1519163968 learned_addr learned my addr 172.21.15.1:0/1519163968 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.323+0000 7f81d2ffd700 1 --2- 172.21.15.1:0/1519163968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f81d40d10d0 0x7f81d415b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.323+0000 7f81d37fe700 1 --2- 172.21.15.1:0/1519163968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81d40cb9c0 0x7f81d40b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.324+0000 7f81d27fc700 1 -- 172.21.15.1:0/1519163968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f81d40d10d0 msgr2=0x7f81d415b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:10.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.324+0000 7f81d27fc700 1 --2- 172.21.15.1:0/1519163968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f81d40d10d0 0x7f81d415b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.324+0000 7f81d27fc700 1 -- 172.21.15.1:0/1519163968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81d40cb9c0 msgr2=0x7f81d40b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.324+0000 7f81d27fc700 1 --2- 172.21.15.1:0/1519163968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81d40cb9c0 0x7f81d40b0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.324+0000 7f81d27fc700 1 -- 172.21.15.1:0/1519163968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc009b30 con 0x7f81d415d470 2024-03-18T19:06:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.324+0000 7f81d2ffd700 1 --2- 172.21.15.1:0/1519163968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f81d40d10d0 0x7f81d415b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.324+0000 7f81d37fe700 1 --2- 172.21.15.1:0/1519163968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81d40cb9c0 0x7f81d40b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.324+0000 7f81d27fc700 1 --2- 172.21.15.1:0/1519163968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81d415d470 0x7f81d40cb480 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f81c400c550 tx=0x7f81c400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.324+0000 7f81bbfff700 1 -- 172.21.15.1:0/1519163968 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c4004c80 con 0x7f81d415d470 2024-03-18T19:06:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.324+0000 7f81bbfff700 1 -- 172.21.15.1:0/1519163968 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c4004e00 con 0x7f81d415d470 2024-03-18T19:06:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.324+0000 7f81bbfff700 1 -- 172.21.15.1:0/1519163968 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c4007310 con 0x7f81d415d470 2024-03-18T19:06:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.324+0000 7f81d9868700 1 -- 172.21.15.1:0/1519163968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d41f6410 con 0x7f81d415d470 2024-03-18T19:06:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.324+0000 7f81d9868700 1 -- 172.21.15.1:0/1519163968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d41f69d0 con 0x7f81d415d470 2024-03-18T19:06:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.325+0000 7f81d9868700 1 -- 172.21.15.1:0/1519163968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81d4073350 con 0x7f81d415d470 2024-03-18T19:06:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.326+0000 7f81bbfff700 1 -- 172.21.15.1:0/1519163968 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f81c4007470 con 0x7f81d415d470 2024-03-18T19:06:10.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.326+0000 7f81bbfff700 1 --2- 172.21.15.1:0/1519163968 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f81c006adb0 0x7f81c006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:10.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.326+0000 7f81d2ffd700 1 --2- 172.21.15.1:0/1519163968 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f81c006adb0 0x7f81c006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:10.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.326+0000 7f81bbfff700 1 -- 172.21.15.1:0/1519163968 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f81c40892b0 con 0x7f81d415d470 2024-03-18T19:06:10.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.327+0000 7f81d2ffd700 1 --2- 172.21.15.1:0/1519163968 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f81c006adb0 0x7f81c006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f81c80099f0 tx=0x7f81c8008040 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:10.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.331+0000 7f81bbfff700 1 -- 172.21.15.1:0/1519163968 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f81c4054240 con 0x7f81d415d470 2024-03-18T19:06:10.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.578+0000 7f81d9868700 1 -- 172.21.15.1:0/1519163968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81d406b680 con 0x7f81d415d470 2024-03-18T19:06:10.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.579+0000 7f81bbfff700 1 -- 172.21.15.1:0/1519163968 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f81c4057860 con 0x7f81d415d470 2024-03-18T19:06:10.575 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:10.575 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3926877640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3926877640},{"type":"v1","addr":"172.21.15.165:6833","nonce":3926877640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/4293662065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":4293662065},{"type":"v1","addr":"172.21.15.165:6835","nonce":4293662065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2334043797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2334043797},{"type":"v1","addr":"172.21.15.1:6835","nonce":2334043797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:06:05.147601+0000","modified":"2024-03-18T19:06:08.160700+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2526016488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":2526016488},{"type":"v1","addr":"172.21.15.1:6837","nonce":2526016488}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-18T19:06:10.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.581+0000 7f81d9868700 1 -- 172.21.15.1:0/1519163968 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f81c006adb0 msgr2=0x7f81c006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:10.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.581+0000 7f81d9868700 1 --2- 172.21.15.1:0/1519163968 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f81c006adb0 0x7f81c006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f81c80099f0 tx=0x7f81c8008040 comp rx=0 tx=0).stop 2024-03-18T19:06:10.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.581+0000 7f81d9868700 1 -- 172.21.15.1:0/1519163968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81d415d470 msgr2=0x7f81d40cb480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:10.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.581+0000 7f81d9868700 1 --2- 172.21.15.1:0/1519163968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81d415d470 0x7f81d40cb480 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f81c400c550 tx=0x7f81c400c910 comp rx=0 tx=0).stop 2024-03-18T19:06:10.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.581+0000 7f81d9868700 1 -- 172.21.15.1:0/1519163968 shutdown_connections 2024-03-18T19:06:10.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.581+0000 7f81d9868700 1 --2- 172.21.15.1:0/1519163968 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f81c006adb0 0x7f81c006d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:10.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.581+0000 7f81d9868700 1 --2- 172.21.15.1:0/1519163968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f81d40d10d0 0x7f81d415b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:10.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.581+0000 7f81d9868700 1 --2- 172.21.15.1:0/1519163968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81d40cb9c0 0x7f81d40b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:10.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.581+0000 7f81d9868700 1 --2- 172.21.15.1:0/1519163968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81d415d470 0x7f81d40cb480 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:10.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.581+0000 7f81d9868700 1 -- 172.21.15.1:0/1519163968 >> 172.21.15.1:0/1519163968 conn(0x7f81d408cc10 msgr2=0x7f81d406ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:10.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.581+0000 7f81d9868700 1 -- 172.21.15.1:0/1519163968 shutdown_connections 2024-03-18T19:06:10.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:10.581+0000 7f81d9868700 1 -- 172.21.15.1:0/1519163968 wait complete. 2024-03-18T19:06:10.578 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 42 2024-03-18T19:06:10.699 INFO:tasks.ceph.mds.a.smithi001.stderr:2024-03-18T19:06:10.703+0000 7f42ae993700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T19:06:10.992 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 42, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T19:06:05.147601+0000', 'modified': '2024-03-18T19:06:08.160700+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7296}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7296': {'gid': 7296, 'name': 'a', 'rank': 0, 'incarnation': 38, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6837/2526016488', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6836', 'nonce': 2526016488}, {'type': 'v1', 'addr': '172.21.15.1:6837', 'nonce': 2526016488}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T19:06:10.993 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T19:06:10.993 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-18T19:06:11.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.127+0000 7f1ca1f02700 1 Processor -- start 2024-03-18T19:06:11.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.127+0000 7f1ca1f02700 1 -- start start 2024-03-18T19:06:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.127+0000 7f1ca1f02700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1c9c15d500 0x7f1c9c06c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.127+0000 7f1ca1f02700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1c9c06c7b0 0x7f1c9c099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1ca1f02700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c9c06f490 con 0x7f1c9c0cb7b0 2024-03-18T19:06:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1ca1f02700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c9c06e880 con 0x7f1c9c06c7b0 2024-03-18T19:06:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1ca1f02700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c06eb60 con 0x7f1c9c15d500 2024-03-18T19:06:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9bfff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1c9c0cb7b0 0x7f1c9c0cccd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50986/0 (socket says 172.21.15.1:50986) 2024-03-18T19:06:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9bfff700 1 -- 172.21.15.1:0/759525105 learned_addr learned my addr 172.21.15.1:0/759525105 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9affd700 1 --2- 172.21.15.1:0/759525105 >> v2:172.21.15.165:3300/0 conn(0x7f1c9c06c7b0 0x7f1c9c099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9b7fe700 1 --2- 172.21.15.1:0/759525105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1c9c15d500 0x7f1c9c06c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9a7fc700 1 -- 172.21.15.1:0/759525105 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789884216 0 0) 0x7f1c9c06f490 con 0x7f1c9c0cb7b0 2024-03-18T19:06:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9a7fc700 1 -- 172.21.15.1:0/759525105 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c840035b0 con 0x7f1c9c0cb7b0 2024-03-18T19:06:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9a7fc700 1 -- 172.21.15.1:0/759525105 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2508092636 0 0) 0x7f1c840035b0 con 0x7f1c9c0cb7b0 2024-03-18T19:06:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9a7fc700 1 -- 172.21.15.1:0/759525105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1c9c15d500 msgr2=0x7f1c9c06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9a7fc700 1 --2- 172.21.15.1:0/759525105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1c9c15d500 0x7f1c9c06c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9a7fc700 1 -- 172.21.15.1:0/759525105 >> v2:172.21.15.165:3300/0 conn(0x7f1c9c06c7b0 msgr2=0x7f1c9c099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9a7fc700 1 --2- 172.21.15.1:0/759525105 >> v2:172.21.15.165:3300/0 conn(0x7f1c9c06c7b0 0x7f1c9c099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9a7fc700 1 -- 172.21.15.1:0/759525105 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c9c06ee40 con 0x7f1c9c0cb7b0 2024-03-18T19:06:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9a7fc700 1 -- 172.21.15.1:0/759525105 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f1c940019f0 con 0x7f1c9c0cb7b0 2024-03-18T19:06:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9a7fc700 1 -- 172.21.15.1:0/759525105 >> v1:172.21.15.1:6789/0 conn(0x7f1c9c0cb7b0 legacy=0x7f1c9c0cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9a7fc700 1 --2- 172.21.15.1:0/759525105 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c84003a00 0x7f1c84003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9a7fc700 1 --2- 172.21.15.1:0/759525105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c84004850 0x7f1c84004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9a7fc700 1 --2- 172.21.15.1:0/759525105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1c9c15d500 0x7f1c84005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9a7fc700 1 -- 172.21.15.1:0/759525105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c06e880 con 0x7f1c84004850 2024-03-18T19:06:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9a7fc700 1 -- 172.21.15.1:0/759525105 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c06eb60 con 0x7f1c84003a00 2024-03-18T19:06:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9a7fc700 1 -- 172.21.15.1:0/759525105 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8400ba10 con 0x7f1c9c15d500 2024-03-18T19:06:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.128+0000 7f1c9affd700 1 --2- 172.21.15.1:0/759525105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c84004850 0x7f1c84004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.129+0000 7f1c9b7fe700 1 --2- 172.21.15.1:0/759525105 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c84003a00 0x7f1c84003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.129+0000 7f1c9bfff700 1 --2- 172.21.15.1:0/759525105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1c9c15d500 0x7f1c84005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.129+0000 7f1c9affd700 1 -- 172.21.15.1:0/759525105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1c9c15d500 msgr2=0x7f1c84005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.129+0000 7f1c9affd700 1 --2- 172.21.15.1:0/759525105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1c9c15d500 0x7f1c84005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.129+0000 7f1c9affd700 1 -- 172.21.15.1:0/759525105 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c84003a00 msgr2=0x7f1c84003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.129+0000 7f1c9affd700 1 --2- 172.21.15.1:0/759525105 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c84003a00 0x7f1c84003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.129+0000 7f1c9affd700 1 -- 172.21.15.1:0/759525105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c840035b0 con 0x7f1c84004850 2024-03-18T19:06:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.129+0000 7f1c9affd700 1 --2- 172.21.15.1:0/759525105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c84004850 0x7f1c84004cd0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c009b60 tx=0x7f1c84005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=532c558eba9a4440 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.129+0000 7f1c9a7fc700 1 -- 172.21.15.1:0/759525105 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c00e6c0 con 0x7f1c84004850 2024-03-18T19:06:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.129+0000 7f1c9a7fc700 1 -- 172.21.15.1:0/759525105 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c8c00e820 con 0x7f1c84004850 2024-03-18T19:06:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.129+0000 7f1c9a7fc700 1 -- 172.21.15.1:0/759525105 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c00eb90 con 0x7f1c84004850 2024-03-18T19:06:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.129+0000 7f1ca1f02700 1 -- 172.21.15.1:0/759525105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c84004850 msgr2=0x7f1c84004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.130+0000 7f1ca1f02700 1 --2- 172.21.15.1:0/759525105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c84004850 0x7f1c84004cd0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c009b60 tx=0x7f1c84005880 comp rx=0 tx=0).stop 2024-03-18T19:06:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.130+0000 7f1c9bfff700 1 -- 172.21.15.1:0/759525105 reap_dead start 2024-03-18T19:06:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.130+0000 7f1ca1f02700 1 -- 172.21.15.1:0/759525105 shutdown_connections 2024-03-18T19:06:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.130+0000 7f1ca1f02700 1 -- 172.21.15.1:0/759525105 >> 172.21.15.1:0/759525105 conn(0x7f1c9c0b2c50 msgr2=0x7f1c9c0b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.130+0000 7f1ca1f02700 1 -- 172.21.15.1:0/759525105 shutdown_connections 2024-03-18T19:06:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.130+0000 7f1ca1f02700 1 -- 172.21.15.1:0/759525105 wait complete. 2024-03-18T19:06:11.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.131+0000 7f1ca1f02700 1 Processor -- start 2024-03-18T19:06:11.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.131+0000 7f1ca1f02700 1 -- start start 2024-03-18T19:06:11.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.132+0000 7f1ca1f02700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c84004850 0x7f1c9c1600d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.132+0000 7f1ca1f02700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1c9c15da60 0x7f1c9c14f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.132+0000 7f1ca1f02700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c9c0b1db0 0x7f1c9c0b2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.132+0000 7f1ca1f02700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c0b2780 con 0x7f1c84004850 2024-03-18T19:06:11.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.132+0000 7f1ca1f02700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c160990 con 0x7f1c9c0b1db0 2024-03-18T19:06:11.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.132+0000 7f1ca1f02700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c1606b0 con 0x7f1c9c15da60 2024-03-18T19:06:11.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.132+0000 7f1c9b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c84004850 0x7f1c9c1600d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.132+0000 7f1c9b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c84004850 0x7f1c9c1600d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40978/0 (socket says 172.21.15.1:40978) 2024-03-18T19:06:11.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.132+0000 7f1c9b7fe700 1 -- 172.21.15.1:0/1934270377 learned_addr learned my addr 172.21.15.1:0/1934270377 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:11.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.132+0000 7f1c9affd700 1 --2- 172.21.15.1:0/1934270377 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1c9c15da60 0x7f1c9c14f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.132+0000 7f1c9bfff700 1 --2- 172.21.15.1:0/1934270377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c9c0b1db0 0x7f1c9c0b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.132+0000 7f1c9b7fe700 1 -- 172.21.15.1:0/1934270377 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1c9c15da60 msgr2=0x7f1c9c14f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:11.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.132+0000 7f1c9b7fe700 1 --2- 172.21.15.1:0/1934270377 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1c9c15da60 0x7f1c9c14f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.132+0000 7f1c9b7fe700 1 -- 172.21.15.1:0/1934270377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c9c0b1db0 msgr2=0x7f1c9c0b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:11.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.132+0000 7f1c9b7fe700 1 --2- 172.21.15.1:0/1934270377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c9c0b1db0 0x7f1c9c0b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.132+0000 7f1c9b7fe700 1 -- 172.21.15.1:0/1934270377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c9c0c4e60 con 0x7f1c84004850 2024-03-18T19:06:11.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.132+0000 7f1c9affd700 1 --2- 172.21.15.1:0/1934270377 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1c9c15da60 0x7f1c9c14f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:11.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.132+0000 7f1c9bfff700 1 --2- 172.21.15.1:0/1934270377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c9c0b1db0 0x7f1c9c0b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:11.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.132+0000 7f1c9b7fe700 1 --2- 172.21.15.1:0/1934270377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c84004850 0x7f1c9c1600d0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f1c90011aa0 tx=0x7f1c90011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:11.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.133+0000 7f1c83fff700 1 -- 172.21.15.1:0/1934270377 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9000f980 con 0x7f1c84004850 2024-03-18T19:06:11.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.133+0000 7f1c83fff700 1 -- 172.21.15.1:0/1934270377 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9000fae0 con 0x7f1c84004850 2024-03-18T19:06:11.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.133+0000 7f1c83fff700 1 -- 172.21.15.1:0/1934270377 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9000fe90 con 0x7f1c84004850 2024-03-18T19:06:11.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.133+0000 7f1ca1f02700 1 -- 172.21.15.1:0/1934270377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c9c0c5140 con 0x7f1c84004850 2024-03-18T19:06:11.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.133+0000 7f1ca1f02700 1 -- 172.21.15.1:0/1934270377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c9c15bc00 con 0x7f1c84004850 2024-03-18T19:06:11.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.134+0000 7f1c83fff700 1 -- 172.21.15.1:0/1934270377 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1c90014440 con 0x7f1c84004850 2024-03-18T19:06:11.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.134+0000 7f1c83fff700 1 --2- 172.21.15.1:0/1934270377 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1c8806adb0 0x7f1c8806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.135+0000 7f1c83fff700 1 -- 172.21.15.1:0/1934270377 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1c90089bd0 con 0x7f1c84004850 2024-03-18T19:06:11.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.135+0000 7f1c9affd700 1 --2- 172.21.15.1:0/1934270377 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1c8806adb0 0x7f1c8806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.135+0000 7f1c9affd700 1 --2- 172.21.15.1:0/1934270377 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1c8806adb0 0x7f1c8806d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0090b0 tx=0x7f1c8c008030 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:11.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.136+0000 7f1cae0abb80 1 Processor -- start 2024-03-18T19:06:11.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.136+0000 7f1cae0abb80 1 -- start start 2024-03-18T19:06:11.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.137+0000 7f1cae0abb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54439cc10 0x55e54439cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.137+0000 7f1cae0abb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e54436e0a0 0x55e5443b9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.137+0000 7f1cae0abb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 0x55e544371160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.137+0000 7f1cae0abb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5442e7f20 con 0x55e5443ae630 2024-03-18T19:06:11.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.137+0000 7f1cae0abb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5442c9c30 con 0x55e54436e0a0 2024-03-18T19:06:11.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.137+0000 7f1cae0abb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e5442c7430 con 0x55e54439cc10 2024-03-18T19:06:11.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.137+0000 7f1c9bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 0x55e544371160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.137+0000 7f1c9bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 0x55e544371160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40992/0 (socket says 172.21.15.1:40992) 2024-03-18T19:06:11.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.137+0000 7f1c9bfff700 1 -- 172.21.15.1:0/2642988158 learned_addr learned my addr 172.21.15.1:0/2642988158 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:11.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.138+0000 7f1c9b7fe700 1 --2- 172.21.15.1:0/2642988158 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54439cc10 0x55e54439cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.138+0000 7f1c9affd700 1 --2- 172.21.15.1:0/2642988158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e54436e0a0 0x55e5443b9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.138+0000 7f1c9bfff700 1 -- 172.21.15.1:0/2642988158 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54439cc10 msgr2=0x55e54439cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:11.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.138+0000 7f1c9bfff700 1 --2- 172.21.15.1:0/2642988158 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54439cc10 0x55e54439cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.138+0000 7f1c9bfff700 1 -- 172.21.15.1:0/2642988158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e54436e0a0 msgr2=0x55e5443b9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:11.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.138+0000 7f1c9bfff700 1 --2- 172.21.15.1:0/2642988158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e54436e0a0 0x55e5443b9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.138+0000 7f1c9bfff700 1 -- 172.21.15.1:0/2642988158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5440b6790 con 0x55e5443ae630 2024-03-18T19:06:11.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.138+0000 7f1c9b7fe700 1 --2- 172.21.15.1:0/2642988158 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54439cc10 0x55e54439cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:11.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.138+0000 7f1c9affd700 1 --2- 172.21.15.1:0/2642988158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e54436e0a0 0x55e5443b9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:11.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.138+0000 7f1c9bfff700 1 --2- 172.21.15.1:0/2642988158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 0x55e544371160 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c072160 tx=0x7f1c94007330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c1b9b6e80853a65 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:11.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.138+0000 7f1c817fa700 1 -- 172.21.15.1:0/2642988158 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c940079d0 con 0x55e5443ae630 2024-03-18T19:06:11.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.138+0000 7f1c817fa700 1 -- 172.21.15.1:0/2642988158 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c94007b30 con 0x55e5443ae630 2024-03-18T19:06:11.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.138+0000 7f1c817fa700 1 -- 172.21.15.1:0/2642988158 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94014640 con 0x55e5443ae630 2024-03-18T19:06:11.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.139+0000 7f1cae0abb80 1 -- 172.21.15.1:0/2642988158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 msgr2=0x55e544371160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:11.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.139+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/2642988158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 0x55e544371160 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c072160 tx=0x7f1c94007330 comp rx=0 tx=0).stop 2024-03-18T19:06:11.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.139+0000 7f1cae0abb80 1 -- 172.21.15.1:0/2642988158 shutdown_connections 2024-03-18T19:06:11.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.139+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/2642988158 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54439cc10 0x55e54439cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.139+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/2642988158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 0x55e544371160 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.139+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/2642988158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e54436e0a0 0x55e5443b9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.139+0000 7f1cae0abb80 1 -- 172.21.15.1:0/2642988158 >> 172.21.15.1:0/2642988158 conn(0x55e5442d0cf0 msgr2=0x55e544370a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:11.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.139+0000 7f1cae0abb80 1 -- 172.21.15.1:0/2642988158 shutdown_connections 2024-03-18T19:06:11.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.139+0000 7f1cae0abb80 1 -- 172.21.15.1:0/2642988158 wait complete. 2024-03-18T19:06:11.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.139+0000 7f1cae0abb80 1 -- start start 2024-03-18T19:06:11.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.139+0000 7f1cae0abb80 1 Processor -- start 2024-03-18T19:06:11.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.140+0000 7f1cae0abb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54436e0a0 0x55e544394920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.140+0000 7f1cae0abb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e54439cc10 0x55e5443b55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.140+0000 7f1cae0abb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 0x55e5443b5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.140+0000 7f1cae0abb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5442c2d20 con 0x55e5443ae630 2024-03-18T19:06:11.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.140+0000 7f1cae0abb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e54425e770 con 0x55e54439cc10 2024-03-18T19:06:11.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.140+0000 7f1cae0abb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e5442816f0 con 0x55e54436e0a0 2024-03-18T19:06:11.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.140+0000 7f1cae0abb80 10 client.0 _ll_register_callbacks cb 0x55e5440ca5a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T19:06:11.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.140+0000 7f1c9bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 0x55e5443b5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.140+0000 7f1c9bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 0x55e5443b5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40996/0 (socket says 172.21.15.1:40996) 2024-03-18T19:06:11.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.140+0000 7f1c9bfff700 1 -- 172.21.15.1:0/2945464620 learned_addr learned my addr 172.21.15.1:0/2945464620 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:11.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.140+0000 7f1c9affd700 1 --2- 172.21.15.1:0/2945464620 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e54439cc10 0x55e5443b55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.140+0000 7f1c9b7fe700 1 --2- 172.21.15.1:0/2945464620 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54436e0a0 0x55e544394920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1c9bfff700 1 -- 172.21.15.1:0/2945464620 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54436e0a0 msgr2=0x55e544394920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:11.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1c9bfff700 1 --2- 172.21.15.1:0/2945464620 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54436e0a0 0x55e544394920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1c9bfff700 1 -- 172.21.15.1:0/2945464620 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e54439cc10 msgr2=0x55e5443b55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:11.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1c9bfff700 1 --2- 172.21.15.1:0/2945464620 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e54439cc10 0x55e5443b55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1c9bfff700 1 -- 172.21.15.1:0/2945464620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c9400f040 con 0x55e5443ae630 2024-03-18T19:06:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1c9bfff700 1 --2- 172.21.15.1:0/2945464620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 0x55e5443b5b10 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f1c94001cb0 tx=0x7f1c94001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb4a423f2363c7a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1c7a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T19:06:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1c7a7fc700 1 -- 172.21.15.1:0/2945464620 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c940079d0 con 0x55e5443ae630 2024-03-18T19:06:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1c7a7fc700 1 -- 172.21.15.1:0/2945464620 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1c940020a0 con 0x55e5443ae630 2024-03-18T19:06:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1cae0abb80 1 -- 172.21.15.1:0/2945464620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e544364e90 con 0x55e5443ae630 2024-03-18T19:06:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1c7a7fc700 1 -- 172.21.15.1:0/2945464620 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c94028460 con 0x55e5443ae630 2024-03-18T19:06:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1cae0abb80 10 client.7422 fetch_fsmap learned FSMap version 42 2024-03-18T19:06:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1cae0abb80 1 -- 172.21.15.1:0/2945464620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x55e5441a30a0 con 0x55e5443ae630 2024-03-18T19:06:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1c7a7fc700 1 -- 172.21.15.1:0/2945464620 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e544364e90 con 0x55e5443ae630 2024-03-18T19:06:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1c7a7fc700 1 -- 172.21.15.1:0/2945464620 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 42) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1c940285c0 con 0x55e5443ae630 2024-03-18T19:06:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1cae0abb80 10 client.7422 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T19:06:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1cae0abb80 10 client.7422 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-18T19:06:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1cae0abb80 20 client.7422 populate_metadata read hostname 'smithi001' 2024-03-18T19:06:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/2945464620 >> [v2:172.21.15.1:6836/2526016488,v1:172.21.15.1:6837/2526016488] conn(0x55e544371970 0x55e544371d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1cae0abb80 4 client.7422 mds_command: new command op to 7296 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T19:06:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1cae0abb80 1 -- 172.21.15.1:0/2945464620 --> [v2:172.21.15.1:6836/2526016488,v1:172.21.15.1:6837/2526016488] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e544342010 con 0x55e544371970 2024-03-18T19:06:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.141+0000 7f1c9b7fe700 1 --2- 172.21.15.1:0/2945464620 >> [v2:172.21.15.1:6836/2526016488,v1:172.21.15.1:6837/2526016488] conn(0x55e544371970 0x55e544371d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.142+0000 7f1c7a7fc700 10 client.7422 ms_handle_connect on v2:172.21.15.1:6836/2526016488 2024-03-18T19:06:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.142+0000 7f1c9b7fe700 1 --2- 172.21.15.1:0/2945464620 >> [v2:172.21.15.1:6836/2526016488,v1:172.21.15.1:6837/2526016488] conn(0x55e544371970 0x55e544371d50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6876f6d492b3ca53 server_cookie=83690d663e3b63d2 in_seq=0 out_seq=0 2024-03-18T19:06:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.142+0000 7f1c7a7fc700 1 -- 172.21.15.1:0/2945464620 <== mds.0 v2:172.21.15.1:6836/2526016488 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e544342010 con 0x55e544371970 2024-03-18T19:06:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.142+0000 7f1c7a7fc700 10 client.7422 handle_command_reply: tid=0 2024-03-18T19:06:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.142+0000 7f1cae0abb80 1 client.7422 shutdown 2024-03-18T19:06:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.142+0000 7f1cae0abb80 1 -- 172.21.15.1:0/2945464620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 msgr2=0x55e5443b5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.142+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/2945464620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 0x55e5443b5b10 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f1c94001cb0 tx=0x7f1c94001d80 comp rx=0 tx=0).stop 2024-03-18T19:06:11.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.142+0000 7f1cae0abb80 1 -- 172.21.15.1:0/2945464620 shutdown_connections 2024-03-18T19:06:11.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.143+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/2945464620 >> [v2:172.21.15.1:6836/2526016488,v1:172.21.15.1:6837/2526016488] conn(0x55e544371970 0x55e544371d50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.143+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/2945464620 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54436e0a0 0x55e544394920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.143+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/2945464620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 0x55e5443b5b10 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.143+0000 7f1c7a7fc700 0 client.7422 ms_handle_reset on v2:172.21.15.1:6836/2526016488 2024-03-18T19:06:11.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.143+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/2945464620 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e54439cc10 0x55e5443b55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.143+0000 7f1cae0abb80 1 -- 172.21.15.1:0/2945464620 >> 172.21.15.1:0/2945464620 conn(0x55e5442d0cf0 msgr2=0x55e54439a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:11.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.143+0000 7f1cae0abb80 1 -- 172.21.15.1:0/2945464620 shutdown_connections 2024-03-18T19:06:11.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.143+0000 7f1cae0abb80 1 -- 172.21.15.1:0/2945464620 wait complete. 2024-03-18T19:06:11.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.143+0000 7f1cae0abb80 20 client.7422 trim_cache size 0 max 16384 2024-03-18T19:06:11.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.160+0000 7f1cae0abb80 1 Processor -- start 2024-03-18T19:06:11.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.160+0000 7f1cae0abb80 1 -- start start 2024-03-18T19:06:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.161+0000 7f1cae0abb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54436e0a0 0x55e54439e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.161+0000 7f1cae0abb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e54439cc10 0x55e5443b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.161+0000 7f1cae0abb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e5443ae630 0x55e5443a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.161+0000 7f1cae0abb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5440b7390 con 0x55e54439cc10 2024-03-18T19:06:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.161+0000 7f1cae0abb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e544394020 con 0x55e5443ae630 2024-03-18T19:06:11.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.161+0000 7f1cae0abb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e544372000 con 0x55e54436e0a0 2024-03-18T19:06:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.161+0000 7f1c9affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e54439cc10 0x55e5443b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.161+0000 7f1c9affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e54439cc10 0x55e5443b6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41006/0 (socket says 172.21.15.1:41006) 2024-03-18T19:06:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.161+0000 7f1c9affd700 1 -- 172.21.15.1:0/304743496 learned_addr learned my addr 172.21.15.1:0/304743496 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.161+0000 7f1c9b7fe700 1 --2- 172.21.15.1:0/304743496 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54436e0a0 0x55e54439e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.161+0000 7f1c9bfff700 1 --2- 172.21.15.1:0/304743496 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e5443ae630 0x55e5443a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.162+0000 7f1c9affd700 1 -- 172.21.15.1:0/304743496 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54436e0a0 msgr2=0x55e54439e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.162+0000 7f1c9affd700 1 --2- 172.21.15.1:0/304743496 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54436e0a0 0x55e54439e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.162+0000 7f1c9affd700 1 -- 172.21.15.1:0/304743496 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e5443ae630 msgr2=0x55e5443a0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.162+0000 7f1c9affd700 1 --2- 172.21.15.1:0/304743496 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e5443ae630 0x55e5443a0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.162+0000 7f1c9affd700 1 -- 172.21.15.1:0/304743496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e544372d90 con 0x55e54439cc10 2024-03-18T19:06:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.162+0000 7f1c9bfff700 1 --2- 172.21.15.1:0/304743496 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e5443ae630 0x55e5443a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.162+0000 7f1c9affd700 1 --2- 172.21.15.1:0/304743496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e54439cc10 0x55e5443b6f40 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c00e4b0 tx=0x7f1c8c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b4dcbbff3c1ec39 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.162+0000 7f1c81ffb700 1 -- 172.21.15.1:0/304743496 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c02f040 con 0x55e54439cc10 2024-03-18T19:06:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.162+0000 7f1c81ffb700 1 -- 172.21.15.1:0/304743496 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c8c0040c0 con 0x55e54439cc10 2024-03-18T19:06:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.162+0000 7f1c81ffb700 1 -- 172.21.15.1:0/304743496 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c022070 con 0x55e54439cc10 2024-03-18T19:06:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.162+0000 7f1cae0abb80 1 -- 172.21.15.1:0/304743496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e54439cc10 msgr2=0x55e5443b6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.162+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/304743496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e54439cc10 0x55e5443b6f40 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c00e4b0 tx=0x7f1c8c014000 comp rx=0 tx=0).stop 2024-03-18T19:06:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.163+0000 7f1cae0abb80 1 -- 172.21.15.1:0/304743496 shutdown_connections 2024-03-18T19:06:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.163+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/304743496 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54436e0a0 0x55e54439e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.163+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/304743496 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e5443ae630 0x55e5443a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.163+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/304743496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e54439cc10 0x55e5443b6f40 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.163+0000 7f1cae0abb80 1 -- 172.21.15.1:0/304743496 >> 172.21.15.1:0/304743496 conn(0x55e5442d0cf0 msgr2=0x55e54436fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.163+0000 7f1cae0abb80 1 -- 172.21.15.1:0/304743496 shutdown_connections 2024-03-18T19:06:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.163+0000 7f1cae0abb80 1 -- 172.21.15.1:0/304743496 wait complete. 2024-03-18T19:06:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.163+0000 7f1cae0abb80 1 -- start start 2024-03-18T19:06:11.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.163+0000 7f1cae0abb80 1 Processor -- start 2024-03-18T19:06:11.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.164+0000 7f1cae0abb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54436e0a0 0x55e5443d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.164+0000 7f1cae0abb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e54439cc10 0x55e5443ad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.164+0000 7f1cae0abb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 0x55e5443bafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.164+0000 7f1cae0abb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5443949d0 con 0x55e5443ae630 2024-03-18T19:06:11.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.164+0000 7f1cae0abb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5443b4f90 con 0x55e54439cc10 2024-03-18T19:06:11.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.164+0000 7f1cae0abb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e5443d0a20 con 0x55e54436e0a0 2024-03-18T19:06:11.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.164+0000 7f1cae0abb80 10 client.0 _ll_register_callbacks cb 0x7f1c94001a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T19:06:11.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.164+0000 7f1c9bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 0x55e5443bafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.164+0000 7f1c9bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 0x55e5443bafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41022/0 (socket says 172.21.15.1:41022) 2024-03-18T19:06:11.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.164+0000 7f1c9bfff700 1 -- 172.21.15.1:0/4135769180 learned_addr learned my addr 172.21.15.1:0/4135769180 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:11.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.164+0000 7f1c9b7fe700 1 --2- 172.21.15.1:0/4135769180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54436e0a0 0x55e5443d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.164+0000 7f1c9affd700 1 --2- 172.21.15.1:0/4135769180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e54439cc10 0x55e5443ad3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.164+0000 7f1c9bfff700 1 -- 172.21.15.1:0/4135769180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54436e0a0 msgr2=0x55e5443d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:11.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.164+0000 7f1c9bfff700 1 --2- 172.21.15.1:0/4135769180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54436e0a0 0x55e5443d0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.164+0000 7f1c9bfff700 1 -- 172.21.15.1:0/4135769180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e54439cc10 msgr2=0x55e5443ad3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:11.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.165+0000 7f1c9bfff700 1 --2- 172.21.15.1:0/4135769180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e54439cc10 0x55e5443ad3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.165+0000 7f1c9bfff700 1 -- 172.21.15.1:0/4135769180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c840035b0 con 0x55e5443ae630 2024-03-18T19:06:11.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.165+0000 7f1c9bfff700 1 --2- 172.21.15.1:0/4135769180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 0x55e5443bafe0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x55e5443b1c20 tx=0x7f1c94009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95d35139f1a88dfe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:11.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.165+0000 7f1c7a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T19:06:11.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.165+0000 7f1c7a7fc700 1 -- 172.21.15.1:0/4135769180 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94014b90 con 0x55e5443ae630 2024-03-18T19:06:11.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.165+0000 7f1c7a7fc700 1 -- 172.21.15.1:0/4135769180 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1c94028c80 con 0x55e5443ae630 2024-03-18T19:06:11.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.165+0000 7f1cae0abb80 1 -- 172.21.15.1:0/4135769180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e544392c00 con 0x55e5443ae630 2024-03-18T19:06:11.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.165+0000 7f1c7a7fc700 1 -- 172.21.15.1:0/4135769180 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c94028a80 con 0x55e5443ae630 2024-03-18T19:06:11.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.165+0000 7f1cae0abb80 10 client.7428 fetch_fsmap learned FSMap version 42 2024-03-18T19:06:11.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.165+0000 7f1cae0abb80 1 -- 172.21.15.1:0/4135769180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x55e544289010 con 0x55e5443ae630 2024-03-18T19:06:11.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.165+0000 7f1c7a7fc700 1 -- 172.21.15.1:0/4135769180 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e544392c00 con 0x55e5443ae630 2024-03-18T19:06:11.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.165+0000 7f1c7a7fc700 1 -- 172.21.15.1:0/4135769180 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 42) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1c9401f410 con 0x55e5443ae630 2024-03-18T19:06:11.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.165+0000 7f1cae0abb80 10 client.7428 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T19:06:11.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.165+0000 7f1cae0abb80 10 client.7428 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-18T19:06:11.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.165+0000 7f1cae0abb80 20 client.7428 populate_metadata read hostname 'smithi001' 2024-03-18T19:06:11.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.165+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/4135769180 >> [v2:172.21.15.1:6836/2526016488,v1:172.21.15.1:6837/2526016488] conn(0x55e54441e850 0x55e5443bb520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.165+0000 7f1cae0abb80 4 client.7428 mds_command: new command op to 7296 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T19:06:11.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.165+0000 7f1cae0abb80 1 -- 172.21.15.1:0/4135769180 --> [v2:172.21.15.1:6836/2526016488,v1:172.21.15.1:6837/2526016488] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f1c94007b30 con 0x55e54441e850 2024-03-18T19:06:11.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.165+0000 7f1c9b7fe700 1 --2- 172.21.15.1:0/4135769180 >> [v2:172.21.15.1:6836/2526016488,v1:172.21.15.1:6837/2526016488] conn(0x55e54441e850 0x55e5443bb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.166+0000 7f1c7a7fc700 10 client.7428 ms_handle_connect on v2:172.21.15.1:6836/2526016488 2024-03-18T19:06:11.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.166+0000 7f1c9b7fe700 1 --2- 172.21.15.1:0/4135769180 >> [v2:172.21.15.1:6836/2526016488,v1:172.21.15.1:6837/2526016488] conn(0x55e54441e850 0x55e5443bb520 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf0d8c11c89b1814 server_cookie=82c51c1e2a5fcc5a in_seq=0 out_seq=0 2024-03-18T19:06:11.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.166+0000 7f1c7a7fc700 1 -- 172.21.15.1:0/4135769180 <== mds.0 v2:172.21.15.1:6836/2526016488 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+320 (crc 0 0 0) 0x7f1c94007b30 con 0x55e54441e850 2024-03-18T19:06:11.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.166+0000 7f1c7a7fc700 10 client.7428 handle_command_reply: tid=0 2024-03-18T19:06:11.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.166+0000 7f1cae0abb80 1 client.7428 shutdown 2024-03-18T19:06:11.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.166+0000 7f1cae0abb80 1 -- 172.21.15.1:0/4135769180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 msgr2=0x55e5443bafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:11.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.166+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/4135769180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 0x55e5443bafe0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x55e5443b1c20 tx=0x7f1c94009040 comp rx=0 tx=0).stop 2024-03-18T19:06:11.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.166+0000 7f1cae0abb80 1 -- 172.21.15.1:0/4135769180 shutdown_connections 2024-03-18T19:06:11.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.166+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/4135769180 >> [v2:172.21.15.1:6836/2526016488,v1:172.21.15.1:6837/2526016488] conn(0x55e54441e850 0x55e5443bb520 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.166+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/4135769180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55e54436e0a0 0x55e5443d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.166+0000 7f1c7a7fc700 0 client.7428 ms_handle_reset on v2:172.21.15.1:6836/2526016488 2024-03-18T19:06:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.166+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/4135769180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e5443ae630 0x55e5443bafe0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.166+0000 7f1cae0abb80 1 --2- 172.21.15.1:0/4135769180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55e54439cc10 0x55e5443ad3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.166+0000 7f1cae0abb80 1 -- 172.21.15.1:0/4135769180 >> 172.21.15.1:0/4135769180 conn(0x55e5442d0cf0 msgr2=0x55e544394db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.166+0000 7f1cae0abb80 1 -- 172.21.15.1:0/4135769180 shutdown_connections 2024-03-18T19:06:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.166+0000 7f1cae0abb80 1 -- 172.21.15.1:0/4135769180 wait complete. 2024-03-18T19:06:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.166+0000 7f1cae0abb80 20 client.7428 trim_cache size 0 max 16384 2024-03-18T19:06:11.169 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:06:11.169 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "9c1c697f-8655-4ef3-8a13-da568d2cbe86", 2024-03-18T19:06:11.169 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-18T19:06:11.169 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 7296, 2024-03-18T19:06:11.170 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-18T19:06:11.170 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-18T19:06:11.170 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-18T19:06:11.170 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 6.017529175, 2024-03-18T19:06:11.170 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 42, 2024-03-18T19:06:11.170 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 103, 2024-03-18T19:06:11.170 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 103, 2024-03-18T19:06:11.170 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 11.245684253 2024-03-18T19:06:11.170 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:06:11.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.167+0000 7f1ca1f02700 1 -- 172.21.15.1:0/1934270377 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1c8806adb0 msgr2=0x7f1c8806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:11.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.167+0000 7f1ca1f02700 1 --2- 172.21.15.1:0/1934270377 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1c8806adb0 0x7f1c8806d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0090b0 tx=0x7f1c8c008030 comp rx=0 tx=0).stop 2024-03-18T19:06:11.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.167+0000 7f1ca1f02700 1 -- 172.21.15.1:0/1934270377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c84004850 msgr2=0x7f1c9c1600d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:11.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.167+0000 7f1ca1f02700 1 --2- 172.21.15.1:0/1934270377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c84004850 0x7f1c9c1600d0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f1c90011aa0 tx=0x7f1c90011e60 comp rx=0 tx=0).stop 2024-03-18T19:06:11.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.167+0000 7f1ca1f02700 1 -- 172.21.15.1:0/1934270377 shutdown_connections 2024-03-18T19:06:11.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.167+0000 7f1ca1f02700 1 --2- 172.21.15.1:0/1934270377 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1c8806adb0 0x7f1c8806d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.167+0000 7f1ca1f02700 1 --2- 172.21.15.1:0/1934270377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c9c0b1db0 0x7f1c9c0b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.167+0000 7f1ca1f02700 1 --2- 172.21.15.1:0/1934270377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c84004850 0x7f1c9c1600d0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.167+0000 7f1ca1f02700 1 --2- 172.21.15.1:0/1934270377 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1c9c15da60 0x7f1c9c14f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.167+0000 7f1ca1f02700 1 -- 172.21.15.1:0/1934270377 >> 172.21.15.1:0/1934270377 conn(0x7f1c9c0b2c50 msgr2=0x7f1c9c0cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:11.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.167+0000 7f1ca1f02700 1 -- 172.21.15.1:0/1934270377 shutdown_connections 2024-03-18T19:06:11.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.167+0000 7f1ca1f02700 1 -- 172.21.15.1:0/1934270377 wait complete. 2024-03-18T19:06:11.176 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T19:06:11.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.305+0000 7fac23320700 1 Processor -- start 2024-03-18T19:06:11.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.305+0000 7fac23320700 1 -- start start 2024-03-18T19:06:11.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.306+0000 7fac23320700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fac1c144b70 0x7fac1c0ce980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.306+0000 7fac23320700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac1c0af610 0x7fac1c0cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.306+0000 7fac23320700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac1c072a20 con 0x7fac1c0ccf40 2024-03-18T19:06:11.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.306+0000 7fac23320700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac1c0cd3d0 con 0x7fac1c144b70 2024-03-18T19:06:11.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.306+0000 7fac23320700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c144f50 con 0x7fac1c0af610 2024-03-18T19:06:11.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.306+0000 7fac210bc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fac1c144b70 0x7fac1c0ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.306+0000 7fac210bc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fac1c144b70 0x7fac1c0ce980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41032/0 (socket says 172.21.15.1:41032) 2024-03-18T19:06:11.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.306+0000 7fac210bc700 1 -- 172.21.15.1:0/940035898 learned_addr learned my addr 172.21.15.1:0/940035898 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:11.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.306+0000 7fac208bb700 1 --2- 172.21.15.1:0/940035898 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac1c0af610 0x7fac1c0cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.307+0000 7fac210bc700 1 -- 172.21.15.1:0/940035898 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac1c0af610 msgr2=0x7fac1c0cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:11.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.307+0000 7fac210bc700 1 --2- 172.21.15.1:0/940035898 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac1c0af610 0x7fac1c0cca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.307+0000 7fac210bc700 1 -- 172.21.15.1:0/940035898 >> v1:172.21.15.165:6789/0 conn(0x7fac1c0ccf40 legacy=0x7fac1c15ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.307+0000 7fac210bc700 1 -- 172.21.15.1:0/940035898 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1c0cf2d0 con 0x7fac1c144b70 2024-03-18T19:06:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.307+0000 7fac208bb700 1 --2- 172.21.15.1:0/940035898 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac1c0af610 0x7fac1c0cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.307+0000 7fac210bc700 1 --2- 172.21.15.1:0/940035898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c144b70 0x7fac1c0ce980 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fac180098f0 tx=0x7fac18009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b463a3d5258793 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.307+0000 7fac13fff700 1 -- 172.21.15.1:0/940035898 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1800b8a0 con 0x7fac1c144b70 2024-03-18T19:06:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.307+0000 7fac13fff700 1 -- 172.21.15.1:0/940035898 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1800ba00 con 0x7fac1c144b70 2024-03-18T19:06:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.307+0000 7fac13fff700 1 -- 172.21.15.1:0/940035898 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1800bd90 con 0x7fac1c144b70 2024-03-18T19:06:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.307+0000 7fac23320700 1 -- 172.21.15.1:0/940035898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c144b70 msgr2=0x7fac1c0ce980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.307+0000 7fac23320700 1 --2- 172.21.15.1:0/940035898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c144b70 0x7fac1c0ce980 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fac180098f0 tx=0x7fac18009c00 comp rx=0 tx=0).stop 2024-03-18T19:06:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.308+0000 7fac23320700 1 -- 172.21.15.1:0/940035898 shutdown_connections 2024-03-18T19:06:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.308+0000 7fac23320700 1 --2- 172.21.15.1:0/940035898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c144b70 0x7fac1c0ce980 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.308+0000 7fac23320700 1 --2- 172.21.15.1:0/940035898 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac1c0af610 0x7fac1c0cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.308+0000 7fac23320700 1 -- 172.21.15.1:0/940035898 >> 172.21.15.1:0/940035898 conn(0x7fac1c0b1d20 msgr2=0x7fac1c0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:11.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.308+0000 7fac23320700 1 -- 172.21.15.1:0/940035898 shutdown_connections 2024-03-18T19:06:11.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.308+0000 7fac23320700 1 -- 172.21.15.1:0/940035898 wait complete. 2024-03-18T19:06:11.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.309+0000 7fac23320700 1 Processor -- start 2024-03-18T19:06:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.309+0000 7fac23320700 1 -- start start 2024-03-18T19:06:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac23320700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac1c0af610 0x7fac1c12fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac23320700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1c0ccf40 0x7fac1c154c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac23320700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c144b70 0x7fac1c155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac23320700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c1557e0 con 0x7fac1c144b70 2024-03-18T19:06:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac23320700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c130110 con 0x7fac1c0ccf40 2024-03-18T19:06:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac23320700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c155500 con 0x7fac1c0af610 2024-03-18T19:06:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac218bd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c144b70 0x7fac1c155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac218bd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c144b70 0x7fac1c155e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41038/0 (socket says 172.21.15.1:41038) 2024-03-18T19:06:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac218bd700 1 -- 172.21.15.1:0/534954420 learned_addr learned my addr 172.21.15.1:0/534954420 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac210bc700 1 --2- 172.21.15.1:0/534954420 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac1c0af610 0x7fac1c12fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac208bb700 1 --2- 172.21.15.1:0/534954420 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1c0ccf40 0x7fac1c154c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac218bd700 1 -- 172.21.15.1:0/534954420 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac1c0af610 msgr2=0x7fac1c12fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac218bd700 1 --2- 172.21.15.1:0/534954420 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac1c0af610 0x7fac1c12fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac218bd700 1 -- 172.21.15.1:0/534954420 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1c0ccf40 msgr2=0x7fac1c154c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac218bd700 1 --2- 172.21.15.1:0/534954420 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1c0ccf40 0x7fac1c154c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac218bd700 1 -- 172.21.15.1:0/534954420 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac18009550 con 0x7fac1c144b70 2024-03-18T19:06:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac218bd700 1 --2- 172.21.15.1:0/534954420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c144b70 0x7fac1c155e40 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fac1400cc50 tx=0x7fac1400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac11ffb700 1 -- 172.21.15.1:0/534954420 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac14003c00 con 0x7fac1c144b70 2024-03-18T19:06:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac11ffb700 1 -- 172.21.15.1:0/534954420 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac14003d60 con 0x7fac1c144b70 2024-03-18T19:06:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac23320700 1 -- 172.21.15.1:0/534954420 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac1c0b3250 con 0x7fac1c144b70 2024-03-18T19:06:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.310+0000 7fac11ffb700 1 -- 172.21.15.1:0/534954420 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1400ab60 con 0x7fac1c144b70 2024-03-18T19:06:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.311+0000 7fac23320700 1 -- 172.21.15.1:0/534954420 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1c0b3810 con 0x7fac1c144b70 2024-03-18T19:06:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.311+0000 7fac23320700 1 -- 172.21.15.1:0/534954420 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1c0cda90 con 0x7fac1c144b70 2024-03-18T19:06:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.312+0000 7fac11ffb700 1 -- 172.21.15.1:0/534954420 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fac1400acc0 con 0x7fac1c144b70 2024-03-18T19:06:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.312+0000 7fac11ffb700 1 --2- 172.21.15.1:0/534954420 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac0006adb0 0x7fac0006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.312+0000 7fac11ffb700 1 -- 172.21.15.1:0/534954420 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fac140895a0 con 0x7fac1c144b70 2024-03-18T19:06:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.312+0000 7fac210bc700 1 --2- 172.21.15.1:0/534954420 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac0006adb0 0x7fac0006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.313+0000 7fac210bc700 1 --2- 172.21.15.1:0/534954420 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac0006adb0 0x7fac0006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fac18014040 tx=0x7fac18008030 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:11.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.317+0000 7fac11ffb700 1 -- 172.21.15.1:0/534954420 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fac140545c0 con 0x7fac1c144b70 2024-03-18T19:06:11.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.566+0000 7fac23320700 1 -- 172.21.15.1:0/534954420 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fac1c0676a0 con 0x7fac1c144b70 2024-03-18T19:06:11.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.567+0000 7fac11ffb700 1 -- 172.21.15.1:0/534954420 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v42) v1 ==== 53+0+73 (secure 0 0 0) 0x7fac14057be0 con 0x7fac1c144b70 2024-03-18T19:06:11.563 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T19:06:11.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.569+0000 7fac23320700 1 -- 172.21.15.1:0/534954420 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac0006adb0 msgr2=0x7fac0006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:11.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.569+0000 7fac23320700 1 --2- 172.21.15.1:0/534954420 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac0006adb0 0x7fac0006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fac18014040 tx=0x7fac18008030 comp rx=0 tx=0).stop 2024-03-18T19:06:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.569+0000 7fac23320700 1 -- 172.21.15.1:0/534954420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c144b70 msgr2=0x7fac1c155e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.569+0000 7fac23320700 1 --2- 172.21.15.1:0/534954420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c144b70 0x7fac1c155e40 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fac1400cc50 tx=0x7fac1400cf60 comp rx=0 tx=0).stop 2024-03-18T19:06:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.569+0000 7fac23320700 1 -- 172.21.15.1:0/534954420 shutdown_connections 2024-03-18T19:06:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.569+0000 7fac23320700 1 --2- 172.21.15.1:0/534954420 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac0006adb0 0x7fac0006d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.569+0000 7fac23320700 1 --2- 172.21.15.1:0/534954420 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac1c0af610 0x7fac1c12fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.569+0000 7fac23320700 1 --2- 172.21.15.1:0/534954420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c144b70 0x7fac1c155e40 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.569+0000 7fac23320700 1 --2- 172.21.15.1:0/534954420 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1c0ccf40 0x7fac1c154c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.569+0000 7fac23320700 1 -- 172.21.15.1:0/534954420 >> 172.21.15.1:0/534954420 conn(0x7fac1c0b1d20 msgr2=0x7fac1c06a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.569+0000 7fac23320700 1 -- 172.21.15.1:0/534954420 shutdown_connections 2024-03-18T19:06:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:11.569+0000 7fac23320700 1 -- 172.21.15.1:0/534954420 wait complete. 2024-03-18T19:06:11.577 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T19:06:11.577 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T19:06:11.577 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-18T19:06:11.577 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T19:06:11.577 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T19:06:11.577 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T19:06:11.577 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T19:06:11.577 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T19:06:11.578 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:06:11.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:11.578 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-18T19:06:11.596 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T19:06:11.597 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T19:06:11.597 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T19:06:11.597 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:06:11.597 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T19:06:11.597 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:06:11.597 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:06:11.597 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-18T19:06:11.597 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-18T19:06:11.597 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:06:11.597 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-18T19:06:11.597 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-18T19:06:11.597 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T19:06:11.598 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:06:11.598 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-18T19:06:11.598 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T19:06:11.598 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 41598sec preferred_lft 41598sec 2024-03-18T19:06:11.598 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link 2024-03-18T19:06:11.598 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:06:11.598 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:06:11.598 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:06:11.598 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-18T19:06:11.598 INFO:teuthology.orchestra.run.smithi001.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T19:06:11.598 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:06:11.598 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T19:06:11.598 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:06:11.599 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::4ca9:f5ff:feb3:79a3/64 scope link 2024-03-18T19:06:11.599 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:06:11.599 INFO:teuthology.orchestra.run.smithi001.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T19:06:11.599 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T19:06:11.599 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::282f:a4ff:fe1a:9908/64 scope link 2024-03-18T19:06:11.599 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:06:11.599 INFO:teuthology.orchestra.run.smithi001.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T19:06:11.599 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2e:5c:60:78:b2:8c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T19:06:11.599 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::2c5c:60ff:fe78:b28c/64 scope link 2024-03-18T19:06:11.599 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:06:11.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:11.600 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T19:06:11.663 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-18T19:06:11.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:11.665 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-18T19:06:11.725 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-18T19:06:11.725 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T19:06:11.725 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T19:06:11.725 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T19:06:11.725 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T19:06:11.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:11.726 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T19:06:11.726 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T19:06:11.726 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T19:06:11.726 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T19:06:11.726 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T19:06:11.726 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T19:06:11.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:11.832 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T19:06:11.851 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T19:06:11.851 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T19:06:11.851 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:06:11.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:11.851 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-18T19:06:11.918 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T19:06:11.918 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T19:06:11.919 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T19:06:11.977 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:06:11.977 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T19:06:11.977 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T19:06:12.035 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T19:06:12.104 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T19:06:12.104 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T19:06:12.171 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T19:06:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T19:06:12.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:12.173 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T19:06:12.231 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T19:06:12.231 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T19:06:12.274 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T19:06:12.302 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T19:06:12.302 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T19:06:12.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-18T19:06:12.367+0000 7fdceec625c0 -1 init, newargv = 0x55f1512a3d60 newargc=15 2024-03-18T19:06:12.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[116850]: starting ceph client 2024-03-18T19:06:12.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[116850]: starting fuse 2024-03-18T19:06:12.373 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T19:06:12.374 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T19:06:12.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:12.375 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T19:06:12.432 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T19:06:12.433 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-03-18T19:06:12.433 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T19:06:12.433 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-18T19:06:12.433 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-18T19:06:12.433 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-18T19:06:12.433 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-18T19:06:12.433 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:06:12.433 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-18T19:06:12.433 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T19:06:12.434 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-18T19:06:12.434 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T19:06:12.434 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:06:12.434 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-18T19:06:12.434 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-18T19:06:12.434 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-18T19:06:12.434 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-18T19:06:12.434 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:06:12.434 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-18T19:06:12.434 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T19:06:12.434 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T19:06:12.434 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T19:06:12.434 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-18T19:06:12.434 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-18T19:06:12.434 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-18T19:06:12.435 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T19:06:12.435 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:06:12.435 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-18T19:06:12.435 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-18T19:06:12.557 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.116850.asok 2024-03-18T19:06:12.739 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.116850.asok 2024-03-18T19:06:12.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:12.739 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.116850.asok status 2024-03-18T19:06:12.901 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:06:12.901 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-18T19:06:12.901 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T19:06:12.901 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T19:06:12.901 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-18T19:06:12.901 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-18T19:06:12.901 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T19:06:12.901 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "116850", 2024-03-18T19:06:12.901 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-18T19:06:12.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:12.902 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-18T19:06:12.902 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-18T19:06:12.902 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 7443, 2024-03-18T19:06:12.902 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-18T19:06:12.902 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-18T19:06:12.902 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-18T19:06:12.902 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 7443 2024-03-18T19:06:12.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:12.902 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T19:06:12.902 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T19:06:12.902 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T19:06:12.902 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2328098948 2024-03-18T19:06:12.902 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:12.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:12.903 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T19:06:12.903 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T19:06:12.903 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T19:06:12.903 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2328098948 2024-03-18T19:06:12.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:12.903 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.7443 192.168.0.1:0/2328098948", 2024-03-18T19:06:12.903 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/2328098948", 2024-03-18T19:06:12.903 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-18T19:06:12.903 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 42, 2024-03-18T19:06:12.903 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 103, 2024-03-18T19:06:12.903 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-18T19:06:12.903 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-18T19:06:12.903 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-18T19:06:12.903 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:06:12.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:12.911 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-18T19:06:12.911 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T19:06:12.929 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-18T19:06:12.929 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T19:06:12.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:12.929 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T19:06:12.998 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-18T19:06:13.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.168+0000 7fee7d114700 1 Processor -- start 2024-03-18T19:06:13.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.168+0000 7fee7d114700 1 -- start start 2024-03-18T19:06:13.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.169+0000 7fee7d114700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee78071490 0x7fee7815b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:13.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.169+0000 7fee7d114700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fee780724e0 0x7fee7815baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:13.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.169+0000 7fee7d114700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee7815b6f0 con 0x7fee7815a380 2024-03-18T19:06:13.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.169+0000 7fee7d114700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee780cb510 con 0x7fee780724e0 2024-03-18T19:06:13.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.169+0000 7fee7d114700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee780ca900 con 0x7fee78071490 2024-03-18T19:06:13.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.169+0000 7fee76d9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee78071490 0x7fee7815b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:13.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.169+0000 7fee7759e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fee780724e0 0x7fee7815baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:13.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.169+0000 7fee7659c700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fee7815a380 0x7fee78071870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:43476/0 (socket says 172.21.15.1:43476) 2024-03-18T19:06:13.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.169+0000 7fee76d9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee78071490 0x7fee7815b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:56210/0 (socket says 172.21.15.1:56210) 2024-03-18T19:06:13.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.169+0000 7fee7759e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fee780724e0 0x7fee7815baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:48304/0 (socket says 172.21.15.1:48304) 2024-03-18T19:06:13.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.169+0000 7fee7659c700 1 -- 172.21.15.1:0/2742759586 learned_addr learned my addr 172.21.15.1:0/2742759586 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:13.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.169+0000 7fee76d9d700 1 -- 172.21.15.1:0/2742759586 >> v2:172.21.15.165:3300/0 conn(0x7fee780724e0 msgr2=0x7fee7815baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:13.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.169+0000 7fee76d9d700 1 --2- 172.21.15.1:0/2742759586 >> v2:172.21.15.165:3300/0 conn(0x7fee780724e0 0x7fee7815baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:13.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.169+0000 7fee76d9d700 1 -- 172.21.15.1:0/2742759586 >> v1:172.21.15.165:6789/0 conn(0x7fee7815a380 legacy=0x7fee78071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:13.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.169+0000 7fee76d9d700 1 -- 172.21.15.1:0/2742759586 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee780cabe0 con 0x7fee78071490 2024-03-18T19:06:13.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.169+0000 7fee7759e700 1 --2- 172.21.15.1:0/2742759586 >> v2:172.21.15.165:3300/0 conn(0x7fee780724e0 0x7fee7815baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:06:13.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.170+0000 7fee76d9d700 1 --2- 172.21.15.1:0/2742759586 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee78071490 0x7fee7815b1b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fee68009840 tx=0x7fee68009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a519a9bfdb4ac665 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:13.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.170+0000 7fee75d9b700 1 -- 172.21.15.1:0/2742759586 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68011830 con 0x7fee78071490 2024-03-18T19:06:13.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.170+0000 7fee75d9b700 1 -- 172.21.15.1:0/2742759586 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee68011990 con 0x7fee78071490 2024-03-18T19:06:13.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.170+0000 7fee75d9b700 1 -- 172.21.15.1:0/2742759586 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68011d20 con 0x7fee78071490 2024-03-18T19:06:13.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.170+0000 7fee7d114700 1 -- 172.21.15.1:0/2742759586 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee78071490 msgr2=0x7fee7815b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:13.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.170+0000 7fee7d114700 1 --2- 172.21.15.1:0/2742759586 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee78071490 0x7fee7815b1b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fee68009840 tx=0x7fee68009b50 comp rx=0 tx=0).stop 2024-03-18T19:06:13.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.171+0000 7fee7d114700 1 -- 172.21.15.1:0/2742759586 shutdown_connections 2024-03-18T19:06:13.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.171+0000 7fee7d114700 1 --2- 172.21.15.1:0/2742759586 >> v2:172.21.15.165:3300/0 conn(0x7fee780724e0 0x7fee7815baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:13.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.171+0000 7fee7d114700 1 --2- 172.21.15.1:0/2742759586 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee78071490 0x7fee7815b1b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:13.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.171+0000 7fee7d114700 1 -- 172.21.15.1:0/2742759586 >> 172.21.15.1:0/2742759586 conn(0x7fee780b1aa0 msgr2=0x7fee780b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:13.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.171+0000 7fee7d114700 1 -- 172.21.15.1:0/2742759586 shutdown_connections 2024-03-18T19:06:13.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.171+0000 7fee7d114700 1 -- 172.21.15.1:0/2742759586 wait complete. 2024-03-18T19:06:13.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.172+0000 7fee7d114700 1 Processor -- start 2024-03-18T19:06:13.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.172+0000 7fee7d114700 1 -- start start 2024-03-18T19:06:13.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.173+0000 7fee7d114700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee78071490 0x7fee78155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:13.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.173+0000 7fee7d114700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee780724e0 0x7fee780b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:13.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.173+0000 7fee7d114700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee7815a380 0x7fee780b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:13.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.173+0000 7fee7d114700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7809fd50 con 0x7fee78071490 2024-03-18T19:06:13.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.173+0000 7fee7d114700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7809f0d0 con 0x7fee7815a380 2024-03-18T19:06:13.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.173+0000 7fee7d114700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee7809f3b0 con 0x7fee780724e0 2024-03-18T19:06:13.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.173+0000 7fee76d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee78071490 0x7fee78155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:13.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.173+0000 7fee76d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee78071490 0x7fee78155a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41054/0 (socket says 172.21.15.1:41054) 2024-03-18T19:06:13.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.173+0000 7fee76d9d700 1 -- 172.21.15.1:0/833000351 learned_addr learned my addr 172.21.15.1:0/833000351 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:13.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.173+0000 7fee7659c700 1 --2- 172.21.15.1:0/833000351 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee780724e0 0x7fee780b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:13.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.173+0000 7fee7759e700 1 --2- 172.21.15.1:0/833000351 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee7815a380 0x7fee780b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:13.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.174+0000 7fee76d9d700 1 -- 172.21.15.1:0/833000351 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee780724e0 msgr2=0x7fee780b2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:13.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.174+0000 7fee76d9d700 1 --2- 172.21.15.1:0/833000351 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee780724e0 0x7fee780b2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:13.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.174+0000 7fee76d9d700 1 -- 172.21.15.1:0/833000351 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee7815a380 msgr2=0x7fee780b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:13.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.174+0000 7fee76d9d700 1 --2- 172.21.15.1:0/833000351 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee7815a380 0x7fee780b30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:13.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.174+0000 7fee76d9d700 1 -- 172.21.15.1:0/833000351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee70009a70 con 0x7fee78071490 2024-03-18T19:06:13.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.174+0000 7fee76d9d700 1 --2- 172.21.15.1:0/833000351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee78071490 0x7fee78155a30 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fee68009fd0 tx=0x7fee68004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:13.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.174+0000 7fee5f7fe700 1 -- 172.21.15.1:0/833000351 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee680036a0 con 0x7fee78071490 2024-03-18T19:06:13.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.174+0000 7fee5f7fe700 1 -- 172.21.15.1:0/833000351 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee68003820 con 0x7fee78071490 2024-03-18T19:06:13.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.174+0000 7fee5f7fe700 1 -- 172.21.15.1:0/833000351 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68003bd0 con 0x7fee78071490 2024-03-18T19:06:13.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.174+0000 7fee7d114700 1 -- 172.21.15.1:0/833000351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee680094a0 con 0x7fee78071490 2024-03-18T19:06:13.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.174+0000 7fee7d114700 1 -- 172.21.15.1:0/833000351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee780cbb20 con 0x7fee78071490 2024-03-18T19:06:13.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.176+0000 7fee7d114700 1 -- 172.21.15.1:0/833000351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee7815da20 con 0x7fee78071490 2024-03-18T19:06:13.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.176+0000 7fee5f7fe700 1 -- 172.21.15.1:0/833000351 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fee68003d30 con 0x7fee78071490 2024-03-18T19:06:13.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.176+0000 7fee5f7fe700 1 --2- 172.21.15.1:0/833000351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee6406ae00 0x7fee6406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:13.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.177+0000 7fee5f7fe700 1 -- 172.21.15.1:0/833000351 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fee6808c650 con 0x7fee78071490 2024-03-18T19:06:13.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.177+0000 7fee7659c700 1 --2- 172.21.15.1:0/833000351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee6406ae00 0x7fee6406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:13.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.177+0000 7fee7659c700 1 --2- 172.21.15.1:0/833000351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee6406ae00 0x7fee6406d2c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fee781598f0 tx=0x7fee6c009040 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:13.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.181+0000 7fee5f7fe700 1 -- 172.21.15.1:0/833000351 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fee68057750 con 0x7fee78071490 2024-03-18T19:06:13.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.418+0000 7fee7d114700 1 -- 172.21.15.1:0/833000351 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fee780cb9a0 con 0x7fee6406ae00 2024-03-18T19:06:13.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.419+0000 7fee5f7fe700 1 -- 172.21.15.1:0/833000351 <== mgr.6631 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fee6c001ce0 con 0x7fee6406ae00 2024-03-18T19:06:13.415 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T19:06:13.415 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:13.415 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephfs" 2024-03-18T19:06:13.415 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:13.415 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T19:06:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.421+0000 7fee7d114700 1 -- 172.21.15.1:0/833000351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee6406ae00 msgr2=0x7fee6406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:13.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.421+0000 7fee7d114700 1 --2- 172.21.15.1:0/833000351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee6406ae00 0x7fee6406d2c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fee781598f0 tx=0x7fee6c009040 comp rx=0 tx=0).stop 2024-03-18T19:06:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.421+0000 7fee7d114700 1 -- 172.21.15.1:0/833000351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee78071490 msgr2=0x7fee78155a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.421+0000 7fee7d114700 1 --2- 172.21.15.1:0/833000351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee78071490 0x7fee78155a30 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fee68009fd0 tx=0x7fee68004750 comp rx=0 tx=0).stop 2024-03-18T19:06:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.421+0000 7fee7d114700 1 -- 172.21.15.1:0/833000351 shutdown_connections 2024-03-18T19:06:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.421+0000 7fee7d114700 1 --2- 172.21.15.1:0/833000351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee6406ae00 0x7fee6406d2c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.421+0000 7fee7d114700 1 --2- 172.21.15.1:0/833000351 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee7815a380 0x7fee780b30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.421+0000 7fee7d114700 1 --2- 172.21.15.1:0/833000351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee78071490 0x7fee78155a30 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.421+0000 7fee7d114700 1 --2- 172.21.15.1:0/833000351 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee780724e0 0x7fee780b2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.421+0000 7fee7d114700 1 -- 172.21.15.1:0/833000351 >> 172.21.15.1:0/833000351 conn(0x7fee780b1aa0 msgr2=0x7fee780cdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.421+0000 7fee7d114700 1 -- 172.21.15.1:0/833000351 shutdown_connections 2024-03-18T19:06:13.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.421+0000 7fee7d114700 1 -- 172.21.15.1:0/833000351 wait complete. 2024-03-18T19:06:13.429 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-18T19:06:13.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.561+0000 7fdc12aec700 1 Processor -- start 2024-03-18T19:06:13.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.561+0000 7fdc12aec700 1 -- start start 2024-03-18T19:06:13.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.561+0000 7fdc12aec700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdc0c0cf9d0 0x7fdc0c0717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:13.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.562+0000 7fdc12aec700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdc0c144500 0x7fdc0c0ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:13.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.562+0000 7fdc12aec700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc0c071d20 con 0x7fdc0c0cfdb0 2024-03-18T19:06:13.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.562+0000 7fdc12aec700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc0c15ed40 con 0x7fdc0c144500 2024-03-18T19:06:13.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.562+0000 7fdc12aec700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc0c15ef90 con 0x7fdc0c0cf9d0 2024-03-18T19:06:13.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.562+0000 7fdc0bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdc0c144500 0x7fdc0c0ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:13.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.562+0000 7fdc11089700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdc0c0cfdb0 0x7fdc0c0cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50988/0 (socket says 172.21.15.1:50988) 2024-03-18T19:06:13.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.562+0000 7fdc11089700 1 -- 172.21.15.1:0/1589664705 learned_addr learned my addr 172.21.15.1:0/1589664705 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:13.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.562+0000 7fdc0b7fe700 1 -- 172.21.15.1:0/1589664705 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3446952376 0 0) 0x7fdc04001490 con 0x7fdc0c0cfdb0 2024-03-18T19:06:13.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.562+0000 7fdc10888700 1 --2- 172.21.15.1:0/1589664705 >> v2:172.21.15.165:3300/0 conn(0x7fdc0c0cf9d0 0x7fdc0c0717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:13.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.562+0000 7fdc0b7fe700 1 -- 172.21.15.1:0/1589664705 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbfc0035b0 con 0x7fdc0c0cfdb0 2024-03-18T19:06:13.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.562+0000 7fdc0bfff700 1 -- 172.21.15.1:0/1589664705 >> v2:172.21.15.165:3300/0 conn(0x7fdc0c0cf9d0 msgr2=0x7fdc0c0717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:13.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.562+0000 7fdc0bfff700 1 --2- 172.21.15.1:0/1589664705 >> v2:172.21.15.165:3300/0 conn(0x7fdc0c0cf9d0 0x7fdc0c0717e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:13.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.562+0000 7fdc0bfff700 1 -- 172.21.15.1:0/1589664705 >> v1:172.21.15.1:6789/0 conn(0x7fdc0c0cfdb0 legacy=0x7fdc0c0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:13.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.562+0000 7fdc0bfff700 1 -- 172.21.15.1:0/1589664705 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0c15f270 con 0x7fdc0c144500 2024-03-18T19:06:13.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.562+0000 7fdc0bfff700 1 --2- 172.21.15.1:0/1589664705 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc0c144500 0x7fdc0c0ce4d0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8009870 tx=0x7fdbf8009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d969d80a04d3a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:13.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.562+0000 7fdc0b7fe700 1 -- 172.21.15.1:0/1589664705 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8011830 con 0x7fdc0c144500 2024-03-18T19:06:13.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.562+0000 7fdc0b7fe700 1 -- 172.21.15.1:0/1589664705 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf8011990 con 0x7fdc0c144500 2024-03-18T19:06:13.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.562+0000 7fdc0b7fe700 1 -- 172.21.15.1:0/1589664705 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8011d20 con 0x7fdc0c144500 2024-03-18T19:06:13.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.563+0000 7fdc12aec700 1 -- 172.21.15.1:0/1589664705 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc0c144500 msgr2=0x7fdc0c0ce4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:13.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.563+0000 7fdc12aec700 1 --2- 172.21.15.1:0/1589664705 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc0c144500 0x7fdc0c0ce4d0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8009870 tx=0x7fdbf8009b80 comp rx=0 tx=0).stop 2024-03-18T19:06:13.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.563+0000 7fdc12aec700 1 -- 172.21.15.1:0/1589664705 shutdown_connections 2024-03-18T19:06:13.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.563+0000 7fdc12aec700 1 --2- 172.21.15.1:0/1589664705 >> v2:172.21.15.165:3300/0 conn(0x7fdc0c0cf9d0 0x7fdc0c0717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:13.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.563+0000 7fdc12aec700 1 --2- 172.21.15.1:0/1589664705 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc0c144500 0x7fdc0c0ce4d0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:13.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.563+0000 7fdc12aec700 1 -- 172.21.15.1:0/1589664705 >> 172.21.15.1:0/1589664705 conn(0x7fdc0c066b70 msgr2=0x7fdc0c06a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:13.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.563+0000 7fdc12aec700 1 -- 172.21.15.1:0/1589664705 shutdown_connections 2024-03-18T19:06:13.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.563+0000 7fdc12aec700 1 -- 172.21.15.1:0/1589664705 wait complete. 2024-03-18T19:06:13.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.564+0000 7fdc12aec700 1 Processor -- start 2024-03-18T19:06:13.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.564+0000 7fdc12aec700 1 -- start start 2024-03-18T19:06:13.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.565+0000 7fdc12aec700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc0c0cf9d0 0x7fdc0c135630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:13.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.565+0000 7fdc12aec700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc0c0cfdb0 0x7fdc0c155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:13.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.565+0000 7fdc12aec700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc0c144500 0x7fdc0c137710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:13.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.565+0000 7fdc12aec700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c13b0f0 con 0x7fdc0c0cfdb0 2024-03-18T19:06:13.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.565+0000 7fdc12aec700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c136670 con 0x7fdc0c0cf9d0 2024-03-18T19:06:13.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.565+0000 7fdc12aec700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c136950 con 0x7fdc0c144500 2024-03-18T19:06:13.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.565+0000 7fdc0bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc0c0cfdb0 0x7fdc0c155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:13.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.565+0000 7fdc0bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc0c0cfdb0 0x7fdc0c155ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41076/0 (socket says 172.21.15.1:41076) 2024-03-18T19:06:13.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.565+0000 7fdc0bfff700 1 -- 172.21.15.1:0/198399152 learned_addr learned my addr 172.21.15.1:0/198399152 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:13.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.565+0000 7fdc11089700 1 --2- 172.21.15.1:0/198399152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc0c144500 0x7fdc0c137710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:13.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.565+0000 7fdc10888700 1 --2- 172.21.15.1:0/198399152 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc0c0cf9d0 0x7fdc0c135630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:13.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.565+0000 7fdc0bfff700 1 -- 172.21.15.1:0/198399152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc0c144500 msgr2=0x7fdc0c137710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:13.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.565+0000 7fdc0bfff700 1 --2- 172.21.15.1:0/198399152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc0c144500 0x7fdc0c137710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:13.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.565+0000 7fdc0bfff700 1 -- 172.21.15.1:0/198399152 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc0c0cf9d0 msgr2=0x7fdc0c135630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:13.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.565+0000 7fdc0bfff700 1 --2- 172.21.15.1:0/198399152 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc0c0cf9d0 0x7fdc0c135630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:13.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.565+0000 7fdc0bfff700 1 -- 172.21.15.1:0/198399152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf80094a0 con 0x7fdc0c0cfdb0 2024-03-18T19:06:13.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.566+0000 7fdc0bfff700 1 --2- 172.21.15.1:0/198399152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc0c0cfdb0 0x7fdc0c155ee0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8009bc0 tx=0x7fdbf8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:13.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.566+0000 7fdc097fa700 1 -- 172.21.15.1:0/198399152 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf80045f0 con 0x7fdc0c0cfdb0 2024-03-18T19:06:13.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.566+0000 7fdc097fa700 1 -- 172.21.15.1:0/198399152 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf801e070 con 0x7fdc0c0cfdb0 2024-03-18T19:06:13.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.566+0000 7fdc097fa700 1 -- 172.21.15.1:0/198399152 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8003ae0 con 0x7fdc0c0cfdb0 2024-03-18T19:06:13.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.566+0000 7fdc12aec700 1 -- 172.21.15.1:0/198399152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc0c136c30 con 0x7fdc0c0cfdb0 2024-03-18T19:06:13.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.566+0000 7fdc12aec700 1 -- 172.21.15.1:0/198399152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc0c1371f0 con 0x7fdc0c0cfdb0 2024-03-18T19:06:13.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.567+0000 7fdc12aec700 1 -- 172.21.15.1:0/198399152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc0c136a90 con 0x7fdc0c0cfdb0 2024-03-18T19:06:13.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.567+0000 7fdc097fa700 1 -- 172.21.15.1:0/198399152 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 39) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fdbf8003c40 con 0x7fdc0c0cfdb0 2024-03-18T19:06:13.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.568+0000 7fdc097fa700 1 --2- 172.21.15.1:0/198399152 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdbf006adb0 0x7fdbf006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:13.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.568+0000 7fdc097fa700 1 -- 172.21.15.1:0/198399152 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fdbf808a590 con 0x7fdc0c0cfdb0 2024-03-18T19:06:13.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.568+0000 7fdc10888700 1 --2- 172.21.15.1:0/198399152 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdbf006adb0 0x7fdbf006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:13.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.568+0000 7fdc10888700 1 --2- 172.21.15.1:0/198399152 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdbf006adb0 0x7fdbf006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdc0000a990 tx=0x7fdc00008040 comp rx=0 tx=0).ready entity=mgr.6631 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:13.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.572+0000 7fdc097fa700 1 -- 172.21.15.1:0/198399152 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdc0c136a90 con 0x7fdc0c0cfdb0 2024-03-18T19:06:13.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:13.836+0000 7fdc12aec700 1 -- 172.21.15.1:0/198399152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fdc0c136a90 con 0x7fdc0c0cfdb0 2024-03-18T19:06:14.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.709+0000 7fdc097fa700 1 -- 172.21.15.1:0/198399152 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 40) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fdbf8055980 con 0x7fdc0c0cfdb0 2024-03-18T19:06:14.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.711+0000 7fdc097fa700 1 -- 172.21.15.1:0/198399152 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v40) v1 ==== 92+0+0 (secure 0 0 0) 0x7fdbf806f810 con 0x7fdc0c0cfdb0 2024-03-18T19:06:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.714+0000 7fdc12aec700 1 -- 172.21.15.1:0/198399152 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdbf006adb0 msgr2=0x7fdbf006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.714+0000 7fdc12aec700 1 --2- 172.21.15.1:0/198399152 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdbf006adb0 0x7fdbf006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdc0000a990 tx=0x7fdc00008040 comp rx=0 tx=0).stop 2024-03-18T19:06:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.714+0000 7fdc12aec700 1 -- 172.21.15.1:0/198399152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc0c0cfdb0 msgr2=0x7fdc0c155ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.714+0000 7fdc12aec700 1 --2- 172.21.15.1:0/198399152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc0c0cfdb0 0x7fdc0c155ee0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8009bc0 tx=0x7fdbf8004750 comp rx=0 tx=0).stop 2024-03-18T19:06:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.714+0000 7fdc12aec700 1 -- 172.21.15.1:0/198399152 shutdown_connections 2024-03-18T19:06:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.714+0000 7fdc12aec700 1 --2- 172.21.15.1:0/198399152 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdbf006adb0 0x7fdbf006d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.714+0000 7fdc12aec700 1 --2- 172.21.15.1:0/198399152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc0c144500 0x7fdc0c137710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.714+0000 7fdc12aec700 1 --2- 172.21.15.1:0/198399152 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc0c0cf9d0 0x7fdc0c135630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.714+0000 7fdc12aec700 1 --2- 172.21.15.1:0/198399152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc0c0cfdb0 0x7fdc0c155ee0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.714+0000 7fdc12aec700 1 -- 172.21.15.1:0/198399152 >> 172.21.15.1:0/198399152 conn(0x7fdc0c066b70 msgr2=0x7fdc0c15a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.714+0000 7fdc12aec700 1 -- 172.21.15.1:0/198399152 shutdown_connections 2024-03-18T19:06:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.714+0000 7fdc12aec700 1 -- 172.21.15.1:0/198399152 wait complete. 2024-03-18T19:06:14.725 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-18T19:06:14.846 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:14.849+0000 7f31e861b240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:06:14.861 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:14.866+0000 7f485edca240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:06:14.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.867+0000 7f777075a700 1 Processor -- start 2024-03-18T19:06:14.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.867+0000 7f777075a700 1 -- start start 2024-03-18T19:06:14.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.868+0000 7f777075a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680d3c60 0x7f77680ce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:14.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.868+0000 7f777075a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77680b3370 con 0x7f77680b4810 2024-03-18T19:06:14.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.868+0000 7f777075a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77680b2760 con 0x7f77680cfb50 2024-03-18T19:06:14.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.868+0000 7f777075a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77680b2a40 con 0x7f77680d3c60 2024-03-18T19:06:14.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.868+0000 7f776dcf5700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f77680b4810 0x7f77680cde70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53792/0 (socket says 172.21.15.1:53792) 2024-03-18T19:06:14.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.868+0000 7f776dcf5700 1 -- 172.21.15.1:0/1502776170 learned_addr learned my addr 172.21.15.1:0/1502776170 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:14.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.868+0000 7f776ecf7700 1 --2- 172.21.15.1:0/1502776170 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680d3c60 0x7f77680ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:14.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.868+0000 7f776d4f4700 1 -- 172.21.15.1:0/1502776170 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3108303558 0 0) 0x7f77680b3370 con 0x7f77680b4810 2024-03-18T19:06:14.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.868+0000 7f776d4f4700 1 -- 172.21.15.1:0/1502776170 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77540035b0 con 0x7f77680b4810 2024-03-18T19:06:14.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.868+0000 7f776d4f4700 1 -- 172.21.15.1:0/1502776170 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4635058 0 0) 0x7f77680b2760 con 0x7f77680cfb50 2024-03-18T19:06:14.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.868+0000 7f776d4f4700 1 -- 172.21.15.1:0/1502776170 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77680b3370 con 0x7f77680cfb50 2024-03-18T19:06:14.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.868+0000 7f776ecf7700 1 -- 172.21.15.1:0/1502776170 >> v1:172.21.15.165:6789/0 conn(0x7f77680cfb50 legacy=0x7f77680cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:14.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.868+0000 7f776ecf7700 1 -- 172.21.15.1:0/1502776170 >> v1:172.21.15.1:6789/0 conn(0x7f77680b4810 legacy=0x7f77680cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:14.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.868+0000 7f776ecf7700 1 -- 172.21.15.1:0/1502776170 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77680b2d20 con 0x7f77680d3c60 2024-03-18T19:06:14.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.868+0000 7f776ecf7700 1 --2- 172.21.15.1:0/1502776170 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680d3c60 0x7f77680ce580 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7760009dc0 tx=0x7f776000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f59dc8873d3d1bbf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:14.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.869+0000 7f776d4f4700 1 -- 172.21.15.1:0/1502776170 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776000e070 con 0x7f77680d3c60 2024-03-18T19:06:14.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.869+0000 7f776d4f4700 1 -- 172.21.15.1:0/1502776170 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776000bb20 con 0x7f77680d3c60 2024-03-18T19:06:14.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.869+0000 7f776d4f4700 1 -- 172.21.15.1:0/1502776170 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7760012430 con 0x7f77680d3c60 2024-03-18T19:06:14.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.869+0000 7f777075a700 1 -- 172.21.15.1:0/1502776170 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680d3c60 msgr2=0x7f77680ce580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:14.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.869+0000 7f777075a700 1 --2- 172.21.15.1:0/1502776170 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680d3c60 0x7f77680ce580 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7760009dc0 tx=0x7f776000b330 comp rx=0 tx=0).stop 2024-03-18T19:06:14.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.869+0000 7f777075a700 1 -- 172.21.15.1:0/1502776170 shutdown_connections 2024-03-18T19:06:14.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.869+0000 7f777075a700 1 --2- 172.21.15.1:0/1502776170 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680d3c60 0x7f77680ce580 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:14.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.869+0000 7f777075a700 1 -- 172.21.15.1:0/1502776170 >> 172.21.15.1:0/1502776170 conn(0x7f77680d3140 msgr2=0x7f77680d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:14.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.869+0000 7f777075a700 1 -- 172.21.15.1:0/1502776170 shutdown_connections 2024-03-18T19:06:14.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.869+0000 7f777075a700 1 -- 172.21.15.1:0/1502776170 wait complete. 2024-03-18T19:06:14.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.871+0000 7f777075a700 1 Processor -- start 2024-03-18T19:06:14.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.871+0000 7f777075a700 1 -- start start 2024-03-18T19:06:14.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.871+0000 7f777075a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680b4810 0x7f77680cc150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:14.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.871+0000 7f777075a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77680cfb50 0x7f7768158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:14.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.871+0000 7f777075a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77680d3c60 0x7f77680c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:14.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.871+0000 7f777075a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77680cc690 con 0x7f77680d3c60 2024-03-18T19:06:14.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.871+0000 7f777075a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77680cb0b0 con 0x7f77680cfb50 2024-03-18T19:06:14.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.871+0000 7f777075a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77680cb390 con 0x7f77680b4810 2024-03-18T19:06:14.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.871+0000 7f776ecf7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77680d3c60 0x7f77680c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:14.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.871+0000 7f776ecf7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77680d3c60 0x7f77680c7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51936/0 (socket says 172.21.15.1:51936) 2024-03-18T19:06:14.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.872+0000 7f776ecf7700 1 -- 172.21.15.1:0/3931249656 learned_addr learned my addr 172.21.15.1:0/3931249656 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:14.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.872+0000 7f776dcf5700 1 --2- 172.21.15.1:0/3931249656 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77680cfb50 0x7f7768158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:14.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.872+0000 7f776e4f6700 1 --2- 172.21.15.1:0/3931249656 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680b4810 0x7f77680cc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:14.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.872+0000 7f776ecf7700 1 -- 172.21.15.1:0/3931249656 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680b4810 msgr2=0x7f77680cc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:14.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.872+0000 7f776ecf7700 1 --2- 172.21.15.1:0/3931249656 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680b4810 0x7f77680cc150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:14.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.872+0000 7f776ecf7700 1 -- 172.21.15.1:0/3931249656 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77680cfb50 msgr2=0x7f7768158370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:14.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.872+0000 7f776ecf7700 1 --2- 172.21.15.1:0/3931249656 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77680cfb50 0x7f7768158370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:14.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.872+0000 7f776ecf7700 1 -- 172.21.15.1:0/3931249656 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7760009a70 con 0x7f77680d3c60 2024-03-18T19:06:14.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.872+0000 7f776ecf7700 1 --2- 172.21.15.1:0/3931249656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77680d3c60 0x7f77680c7bd0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f776000b370 tx=0x7f7760004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:14.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.872+0000 7f775effd700 1 -- 172.21.15.1:0/3931249656 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776001b810 con 0x7f77680d3c60 2024-03-18T19:06:14.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.872+0000 7f777075a700 1 -- 172.21.15.1:0/3931249656 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77680cb670 con 0x7f77680d3c60 2024-03-18T19:06:14.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.872+0000 7f775effd700 1 -- 172.21.15.1:0/3931249656 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7760020070 con 0x7f77680d3c60 2024-03-18T19:06:14.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.872+0000 7f775effd700 1 -- 172.21.15.1:0/3931249656 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776001bbb0 con 0x7f77680d3c60 2024-03-18T19:06:14.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.872+0000 7f777075a700 1 -- 172.21.15.1:0/3931249656 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77680cbc30 con 0x7f77680d3c60 2024-03-18T19:06:14.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.874+0000 7f777075a700 1 -- 172.21.15.1:0/3931249656 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77680cbe20 con 0x7f77680d3c60 2024-03-18T19:06:14.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.874+0000 7f775effd700 1 -- 172.21.15.1:0/3931249656 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 40) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f776001bd10 con 0x7f77680d3c60 2024-03-18T19:06:14.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.874+0000 7f775effd700 1 --2- 172.21.15.1:0/3931249656 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f774c06ae00 0x7f774c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:14.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.874+0000 7f776e4f6700 1 -- 172.21.15.1:0/3931249656 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f774c06ae00 msgr2=0x7f774c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:14.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.874+0000 7f775effd700 1 -- 172.21.15.1:0/3931249656 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7760022070 con 0x7f77680d3c60 2024-03-18T19:06:14.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.874+0000 7f776e4f6700 1 --2- 172.21.15.1:0/3931249656 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f774c06ae00 0x7f774c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:14.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:14.879+0000 7f775effd700 1 -- 172.21.15.1:0/3931249656 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f77600564c0 con 0x7f77680d3c60 2024-03-18T19:06:14.993 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:14.996+0000 7f31e861b240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:06:15.032 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:15.036+0000 7f485edca240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:06:15.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.075+0000 7f776e4f6700 1 -- 172.21.15.1:0/3931249656 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f774c06ae00 msgr2=0x7f774c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:15.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.075+0000 7f776e4f6700 1 --2- 172.21.15.1:0/3931249656 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f774c06ae00 0x7f774c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:15.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.130+0000 7f777075a700 1 -- 172.21.15.1:0/3931249656 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f7768073350 con 0x7f77680d3c60 2024-03-18T19:06:15.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.133+0000 7f775effd700 1 -- 172.21.15.1:0/3931249656 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7760059ae0 con 0x7f77680d3c60 2024-03-18T19:06:15.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.135+0000 7f777075a700 1 -- 172.21.15.1:0/3931249656 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f774c06ae00 msgr2=0x7f774c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:15.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.135+0000 7f777075a700 1 --2- 172.21.15.1:0/3931249656 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f774c06ae00 0x7f774c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:15.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.135+0000 7f777075a700 1 -- 172.21.15.1:0/3931249656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77680d3c60 msgr2=0x7f77680c7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:15.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.135+0000 7f777075a700 1 --2- 172.21.15.1:0/3931249656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77680d3c60 0x7f77680c7bd0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f776000b370 tx=0x7f7760004ad0 comp rx=0 tx=0).stop 2024-03-18T19:06:15.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.135+0000 7f777075a700 1 -- 172.21.15.1:0/3931249656 shutdown_connections 2024-03-18T19:06:15.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.135+0000 7f777075a700 1 --2- 172.21.15.1:0/3931249656 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f774c06ae00 0x7f774c06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:15.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.135+0000 7f777075a700 1 --2- 172.21.15.1:0/3931249656 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680b4810 0x7f77680cc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:15.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.135+0000 7f777075a700 1 --2- 172.21.15.1:0/3931249656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77680d3c60 0x7f77680c7bd0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:15.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.135+0000 7f777075a700 1 --2- 172.21.15.1:0/3931249656 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77680cfb50 0x7f7768158370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:15.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.135+0000 7f777075a700 1 -- 172.21.15.1:0/3931249656 >> 172.21.15.1:0/3931249656 conn(0x7f77680d3140 msgr2=0x7f776815ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:15.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.135+0000 7f777075a700 1 -- 172.21.15.1:0/3931249656 shutdown_connections 2024-03-18T19:06:15.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.135+0000 7f777075a700 1 -- 172.21.15.1:0/3931249656 wait complete. 2024-03-18T19:06:15.144 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-18T19:06:15.256 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:15.260+0000 7f31e861b240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:06:15.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.277+0000 7fecef5c7700 1 Processor -- start 2024-03-18T19:06:15.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.277+0000 7fecef5c7700 1 -- start start 2024-03-18T19:06:15.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.278+0000 7fecef5c7700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fece815b6c0 0x7fece80b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:15.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.278+0000 7fecef5c7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece80d01f0 con 0x7fece80cd430 2024-03-18T19:06:15.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.278+0000 7fecef5c7700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece80ccfa0 con 0x7fece80c87d0 2024-03-18T19:06:15.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.278+0000 7fecef5c7700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece80d1db0 con 0x7fece815b6c0 2024-03-18T19:06:15.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.278+0000 7fececb62700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fece80cd430 0x7fece80c80c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53806/0 (socket says 172.21.15.1:53806) 2024-03-18T19:06:15.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.278+0000 7fececb62700 1 -- 172.21.15.1:0/2536364025 learned_addr learned my addr 172.21.15.1:0/2536364025 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:15.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.278+0000 7feced363700 1 --2- 172.21.15.1:0/2536364025 >> v2:172.21.15.165:3300/0 conn(0x7fece815b6c0 0x7fece80b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:15.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.278+0000 7fecdffff700 1 -- 172.21.15.1:0/2536364025 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513814790 0 0) 0x7fece80d01f0 con 0x7fece80cd430 2024-03-18T19:06:15.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.278+0000 7fecdffff700 1 -- 172.21.15.1:0/2536364025 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecd40035b0 con 0x7fece80cd430 2024-03-18T19:06:15.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.278+0000 7feced363700 1 -- 172.21.15.1:0/2536364025 >> v1:172.21.15.165:6789/0 conn(0x7fece80c87d0 legacy=0x7fece80cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:15.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.278+0000 7feced363700 1 -- 172.21.15.1:0/2536364025 >> v1:172.21.15.1:6789/0 conn(0x7fece80cd430 legacy=0x7fece80c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:15.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.278+0000 7feced363700 1 -- 172.21.15.1:0/2536364025 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece80cff10 con 0x7fece815b6c0 2024-03-18T19:06:15.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.279+0000 7feced363700 1 --2- 172.21.15.1:0/2536364025 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fece815b6c0 0x7fece80b2160 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fece4009840 tx=0x7fece4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cc6a55ef80dd6d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:15.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.279+0000 7fecdffff700 1 -- 172.21.15.1:0/2536364025 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4011830 con 0x7fece815b6c0 2024-03-18T19:06:15.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.279+0000 7fecdffff700 1 -- 172.21.15.1:0/2536364025 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece4011990 con 0x7fece815b6c0 2024-03-18T19:06:15.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.279+0000 7fecdffff700 1 -- 172.21.15.1:0/2536364025 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4011d20 con 0x7fece815b6c0 2024-03-18T19:06:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.279+0000 7fecef5c7700 1 -- 172.21.15.1:0/2536364025 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fece815b6c0 msgr2=0x7fece80b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.279+0000 7fecef5c7700 1 --2- 172.21.15.1:0/2536364025 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fece815b6c0 0x7fece80b2160 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fece4009840 tx=0x7fece4009b50 comp rx=0 tx=0).stop 2024-03-18T19:06:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.279+0000 7fecef5c7700 1 -- 172.21.15.1:0/2536364025 shutdown_connections 2024-03-18T19:06:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.279+0000 7fecef5c7700 1 --2- 172.21.15.1:0/2536364025 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fece815b6c0 0x7fece80b2160 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.279+0000 7fecef5c7700 1 -- 172.21.15.1:0/2536364025 >> 172.21.15.1:0/2536364025 conn(0x7fece80d0ff0 msgr2=0x7fece80d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.280+0000 7fecef5c7700 1 -- 172.21.15.1:0/2536364025 shutdown_connections 2024-03-18T19:06:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.280+0000 7fecef5c7700 1 -- 172.21.15.1:0/2536364025 wait complete. 2024-03-18T19:06:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.281+0000 7fecef5c7700 1 Processor -- start 2024-03-18T19:06:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.281+0000 7fecef5c7700 1 -- start start 2024-03-18T19:06:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7fecef5c7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fece80c87d0 0x7fece815a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7fecef5c7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fece80cd430 0x7fece809ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7fecef5c7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fece815b6c0 0x7fece80c7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:15.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7fecef5c7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece809c120 con 0x7fece80c87d0 2024-03-18T19:06:15.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7fecef5c7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece809b880 con 0x7fece815b6c0 2024-03-18T19:06:15.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7fecef5c7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece809bb60 con 0x7fece80cd430 2024-03-18T19:06:15.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7feced363700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fece80c87d0 0x7fece815a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:15.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7feced363700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fece80c87d0 0x7fece815a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51938/0 (socket says 172.21.15.1:51938) 2024-03-18T19:06:15.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7feced363700 1 -- 172.21.15.1:0/1579563989 learned_addr learned my addr 172.21.15.1:0/1579563989 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:15.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7fececb62700 1 --2- 172.21.15.1:0/1579563989 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fece80cd430 0x7fece809ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:15.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7fecedb64700 1 --2- 172.21.15.1:0/1579563989 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fece815b6c0 0x7fece80c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:15.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7feced363700 1 -- 172.21.15.1:0/1579563989 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fece80cd430 msgr2=0x7fece809ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:15.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7feced363700 1 --2- 172.21.15.1:0/1579563989 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fece80cd430 0x7fece809ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:15.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7feced363700 1 -- 172.21.15.1:0/1579563989 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fece815b6c0 msgr2=0x7fece80c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:15.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7feced363700 1 --2- 172.21.15.1:0/1579563989 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fece815b6c0 0x7fece80c7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:15.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7feced363700 1 -- 172.21.15.1:0/1579563989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece40094a0 con 0x7fece80c87d0 2024-03-18T19:06:15.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7feced363700 1 --2- 172.21.15.1:0/1579563989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fece80c87d0 0x7fece815a590 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fece4009b90 tx=0x7fece4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:15.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7fecddffb700 1 -- 172.21.15.1:0/1579563989 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece40038a0 con 0x7fece80c87d0 2024-03-18T19:06:15.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7fecddffb700 1 -- 172.21.15.1:0/1579563989 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece401e070 con 0x7fece80c87d0 2024-03-18T19:06:15.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7fecddffb700 1 -- 172.21.15.1:0/1579563989 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4003c40 con 0x7fece80c87d0 2024-03-18T19:06:15.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.282+0000 7fecef5c7700 1 -- 172.21.15.1:0/1579563989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece809be40 con 0x7fece80c87d0 2024-03-18T19:06:15.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.283+0000 7fecef5c7700 1 -- 172.21.15.1:0/1579563989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece815aad0 con 0x7fece80c87d0 2024-03-18T19:06:15.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.284+0000 7fecef5c7700 1 -- 172.21.15.1:0/1579563989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece81f3310 con 0x7fece80c87d0 2024-03-18T19:06:15.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.284+0000 7fecddffb700 1 -- 172.21.15.1:0/1579563989 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 40) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fece402d450 con 0x7fece80c87d0 2024-03-18T19:06:15.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.285+0000 7fecddffb700 1 --2- 172.21.15.1:0/1579563989 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7feccc06ae00 0x7feccc06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:15.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.285+0000 7fececb62700 1 -- 172.21.15.1:0/1579563989 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7feccc06ae00 msgr2=0x7feccc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:15.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.285+0000 7fecddffb700 1 -- 172.21.15.1:0/1579563989 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fece408a650 con 0x7fece80c87d0 2024-03-18T19:06:15.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.285+0000 7fececb62700 1 --2- 172.21.15.1:0/1579563989 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7feccc06ae00 0x7feccc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:15.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.290+0000 7fecddffb700 1 -- 172.21.15.1:0/1579563989 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fece81f3310 con 0x7fece80c87d0 2024-03-18T19:06:15.312 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:15.317+0000 7f485edca240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:06:15.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.486+0000 7fececb62700 1 -- 172.21.15.1:0/1579563989 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7feccc06ae00 msgr2=0x7feccc06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:15.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.486+0000 7fececb62700 1 --2- 172.21.15.1:0/1579563989 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7feccc06ae00 0x7feccc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:15.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.494+0000 7fecef5c7700 1 -- 172.21.15.1:0/1579563989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fece81f36f0 con 0x7fece80c87d0 2024-03-18T19:06:15.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.498+0000 7fecddffb700 1 -- 172.21.15.1:0/1579563989 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7fece81f36f0 con 0x7fece80c87d0 2024-03-18T19:06:15.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.500+0000 7fecef5c7700 1 -- 172.21.15.1:0/1579563989 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7feccc06ae00 msgr2=0x7feccc06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:15.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.500+0000 7fecef5c7700 1 --2- 172.21.15.1:0/1579563989 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7feccc06ae00 0x7feccc06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:15.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.500+0000 7fecef5c7700 1 -- 172.21.15.1:0/1579563989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fece80c87d0 msgr2=0x7fece815a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:15.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.500+0000 7fecef5c7700 1 --2- 172.21.15.1:0/1579563989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fece80c87d0 0x7fece815a590 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fece4009b90 tx=0x7fece4004750 comp rx=0 tx=0).stop 2024-03-18T19:06:15.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.500+0000 7fecef5c7700 1 -- 172.21.15.1:0/1579563989 shutdown_connections 2024-03-18T19:06:15.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.500+0000 7fecef5c7700 1 --2- 172.21.15.1:0/1579563989 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7feccc06ae00 0x7feccc06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:15.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.500+0000 7fecef5c7700 1 --2- 172.21.15.1:0/1579563989 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fece815b6c0 0x7fece80c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:15.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.500+0000 7fecef5c7700 1 --2- 172.21.15.1:0/1579563989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fece80c87d0 0x7fece815a590 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:15.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.500+0000 7fecef5c7700 1 --2- 172.21.15.1:0/1579563989 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fece80cd430 0x7fece809ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:15.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.500+0000 7fecef5c7700 1 -- 172.21.15.1:0/1579563989 >> 172.21.15.1:0/1579563989 conn(0x7fece80d0ff0 msgr2=0x7fece80ca030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:15.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.500+0000 7fecef5c7700 1 -- 172.21.15.1:0/1579563989 shutdown_connections 2024-03-18T19:06:15.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.500+0000 7fecef5c7700 1 -- 172.21.15.1:0/1579563989 wait complete. 2024-03-18T19:06:15.509 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-03-18T19:06:15.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.643+0000 7fba47134700 1 Processor -- start 2024-03-18T19:06:15.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.643+0000 7fba47134700 1 -- start start 2024-03-18T19:06:15.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.644+0000 7fba47134700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba400cd870 0x7fba400cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:15.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.644+0000 7fba47134700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fba400cbce0 0x7fba400cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:15.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.644+0000 7fba47134700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba400d0df0 con 0x7fba400b1e20 2024-03-18T19:06:15.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.644+0000 7fba47134700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba400d0550 con 0x7fba400cbce0 2024-03-18T19:06:15.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.644+0000 7fba47134700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba400d0830 con 0x7fba400cd870 2024-03-18T19:06:15.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.644+0000 7fba3ffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fba400b1e20 0x7fba400cdc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53808/0 (socket says 172.21.15.1:53808) 2024-03-18T19:06:15.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.644+0000 7fba3ffff700 1 -- 172.21.15.1:0/947128150 learned_addr learned my addr 172.21.15.1:0/947128150 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:15.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.644+0000 7fba456d1700 1 --2- 172.21.15.1:0/947128150 >> v2:172.21.15.165:3300/0 conn(0x7fba400cbce0 0x7fba400cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:15.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.644+0000 7fba44ed0700 1 --2- 172.21.15.1:0/947128150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba400cd870 0x7fba400cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:15.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.645+0000 7fba3f7fe700 1 -- 172.21.15.1:0/947128150 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2124902394 0 0) 0x7fba400d0df0 con 0x7fba400b1e20 2024-03-18T19:06:15.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.645+0000 7fba3f7fe700 1 -- 172.21.15.1:0/947128150 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba300035b0 con 0x7fba400b1e20 2024-03-18T19:06:15.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.645+0000 7fba456d1700 1 -- 172.21.15.1:0/947128150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba400cd870 msgr2=0x7fba400cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:15.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.645+0000 7fba456d1700 1 --2- 172.21.15.1:0/947128150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba400cd870 0x7fba400cff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:15.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.645+0000 7fba456d1700 1 -- 172.21.15.1:0/947128150 >> v1:172.21.15.1:6789/0 conn(0x7fba400b1e20 legacy=0x7fba400cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:15.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.645+0000 7fba456d1700 1 -- 172.21.15.1:0/947128150 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba400d0b10 con 0x7fba400cbce0 2024-03-18T19:06:15.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.645+0000 7fba44ed0700 1 --2- 172.21.15.1:0/947128150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba400cd870 0x7fba400cff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:06:15.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.645+0000 7fba456d1700 1 --2- 172.21.15.1:0/947128150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba400cbce0 0x7fba400cc170 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fba38009d90 tx=0x7fba3800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=321e8e871fc4febb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:15.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.645+0000 7fba3f7fe700 1 -- 172.21.15.1:0/947128150 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3800e070 con 0x7fba400cbce0 2024-03-18T19:06:15.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.645+0000 7fba3f7fe700 1 -- 172.21.15.1:0/947128150 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3800bb20 con 0x7fba400cbce0 2024-03-18T19:06:15.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.645+0000 7fba3f7fe700 1 -- 172.21.15.1:0/947128150 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba38012430 con 0x7fba400cbce0 2024-03-18T19:06:15.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.646+0000 7fba47134700 1 -- 172.21.15.1:0/947128150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba400cbce0 msgr2=0x7fba400cc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:15.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.646+0000 7fba47134700 1 --2- 172.21.15.1:0/947128150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba400cbce0 0x7fba400cc170 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fba38009d90 tx=0x7fba3800b330 comp rx=0 tx=0).stop 2024-03-18T19:06:15.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.646+0000 7fba47134700 1 -- 172.21.15.1:0/947128150 shutdown_connections 2024-03-18T19:06:15.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.646+0000 7fba47134700 1 --2- 172.21.15.1:0/947128150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba400cbce0 0x7fba400cc170 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:15.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.646+0000 7fba47134700 1 --2- 172.21.15.1:0/947128150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba400cd870 0x7fba400cff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:15.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.646+0000 7fba47134700 1 -- 172.21.15.1:0/947128150 >> 172.21.15.1:0/947128150 conn(0x7fba400d0ff0 msgr2=0x7fba400d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:15.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.646+0000 7fba47134700 1 -- 172.21.15.1:0/947128150 shutdown_connections 2024-03-18T19:06:15.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.646+0000 7fba47134700 1 -- 172.21.15.1:0/947128150 wait complete. 2024-03-18T19:06:15.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.647+0000 7fba47134700 1 Processor -- start 2024-03-18T19:06:15.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.647+0000 7fba47134700 1 -- start start 2024-03-18T19:06:15.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.648+0000 7fba47134700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba400b1e20 0x7fba400c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:15.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.648+0000 7fba47134700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba400cbce0 0x7fba4015c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:15.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.648+0000 7fba47134700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba400cd870 0x7fba4015c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:15.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.648+0000 7fba47134700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba401e6a30 con 0x7fba400cbce0 2024-03-18T19:06:15.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.648+0000 7fba47134700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba401e6c80 con 0x7fba400b1e20 2024-03-18T19:06:15.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.648+0000 7fba47134700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba401e6f60 con 0x7fba400cd870 2024-03-18T19:06:15.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.648+0000 7fba3ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba400cbce0 0x7fba4015c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:15.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.648+0000 7fba44ed0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba400b1e20 0x7fba400c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:15.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.648+0000 7fba3ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba400cbce0 0x7fba4015c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51952/0 (socket says 172.21.15.1:51952) 2024-03-18T19:06:15.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.648+0000 7fba44ed0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba400b1e20 0x7fba400c80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:57936/0 (socket says 172.21.15.1:57936) 2024-03-18T19:06:15.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.648+0000 7fba3ffff700 1 -- 172.21.15.1:0/2925433770 learned_addr learned my addr 172.21.15.1:0/2925433770 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:15.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.648+0000 7fba456d1700 1 --2- 172.21.15.1:0/2925433770 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba400cd870 0x7fba4015c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:15.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.648+0000 7fba3ffff700 1 -- 172.21.15.1:0/2925433770 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba400cd870 msgr2=0x7fba4015c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:15.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.648+0000 7fba3ffff700 1 --2- 172.21.15.1:0/2925433770 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba400cd870 0x7fba4015c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:15.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.648+0000 7fba3ffff700 1 -- 172.21.15.1:0/2925433770 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba400b1e20 msgr2=0x7fba400c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:15.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.648+0000 7fba3ffff700 1 --2- 172.21.15.1:0/2925433770 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba400b1e20 0x7fba400c80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:15.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.648+0000 7fba3ffff700 1 -- 172.21.15.1:0/2925433770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba340094a0 con 0x7fba400cbce0 2024-03-18T19:06:15.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.649+0000 7fba3ffff700 1 --2- 172.21.15.1:0/2925433770 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba400cbce0 0x7fba4015c230 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00c700 tx=0x7fba2c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:15.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.649+0000 7fba3d7fa700 1 -- 172.21.15.1:0/2925433770 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c005d70 con 0x7fba400cbce0 2024-03-18T19:06:15.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.649+0000 7fba3d7fa700 1 -- 172.21.15.1:0/2925433770 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2c0070d0 con 0x7fba400cbce0 2024-03-18T19:06:15.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.649+0000 7fba47134700 1 -- 172.21.15.1:0/2925433770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba38009a70 con 0x7fba400cbce0 2024-03-18T19:06:15.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.649+0000 7fba3d7fa700 1 -- 172.21.15.1:0/2925433770 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c007460 con 0x7fba400cbce0 2024-03-18T19:06:15.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.649+0000 7fba47134700 1 -- 172.21.15.1:0/2925433770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba401e7520 con 0x7fba400cbce0 2024-03-18T19:06:15.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.650+0000 7fba47134700 1 -- 172.21.15.1:0/2925433770 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba400676a0 con 0x7fba400cbce0 2024-03-18T19:06:15.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.650+0000 7fba3d7fa700 1 -- 172.21.15.1:0/2925433770 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 40) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fba2c0075c0 con 0x7fba400cbce0 2024-03-18T19:06:15.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.651+0000 7fba3d7fa700 1 --2- 172.21.15.1:0/2925433770 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2406ae00 0x7fba2406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:15.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.651+0000 7fba44ed0700 1 -- 172.21.15.1:0/2925433770 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2406ae00 msgr2=0x7fba2406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:15.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.651+0000 7fba3d7fa700 1 -- 172.21.15.1:0/2925433770 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fba2c088ab0 con 0x7fba400cbce0 2024-03-18T19:06:15.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.651+0000 7fba44ed0700 1 --2- 172.21.15.1:0/2925433770 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2406ae00 0x7fba2406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:15.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.655+0000 7fba3d7fa700 1 -- 172.21.15.1:0/2925433770 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fba2c053a40 con 0x7fba400cbce0 2024-03-18T19:06:15.681 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:15.685+0000 7f31e861b240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:06:15.767 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:15.771+0000 7f485edca240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:06:15.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.852+0000 7fba44ed0700 1 -- 172.21.15.1:0/2925433770 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2406ae00 msgr2=0x7fba2406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:15.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.852+0000 7fba44ed0700 1 --2- 172.21.15.1:0/2925433770 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2406ae00 0x7fba2406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:15.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:15.885+0000 7fba47134700 1 -- 172.21.15.1:0/2925433770 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fba400b25a0 con 0x7fba2406ae00 2024-03-18T19:06:15.888 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:15.892+0000 7f31e861b240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:06:15.985 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:15.989+0000 7f485edca240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:06:16.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:16.253+0000 7fba44ed0700 1 -- 172.21.15.1:0/2925433770 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2406ae00 msgr2=0x7fba2406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:16.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:16.253+0000 7fba44ed0700 1 --2- 172.21.15.1:0/2925433770 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2406ae00 0x7fba2406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T19:06:16.256 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:16.260+0000 7f31e861b240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:06:16.371 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:16.375+0000 7f485edca240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:06:16.832 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:16.837+0000 7f31e861b240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:06:16.903 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:16.908+0000 7f31e861b240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:06:16.983 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:16.988+0000 7f485edca240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:06:17.029 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:17.033+0000 7f31e861b240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:06:17.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:17.054+0000 7fba44ed0700 1 -- 172.21.15.1:0/2925433770 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2406ae00 msgr2=0x7fba2406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:17.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:17.054+0000 7fba44ed0700 1 --2- 172.21.15.1:0/2925433770 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2406ae00 0x7fba2406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T19:06:17.059 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:17.063+0000 7f485edca240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:06:17.193 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:17.197+0000 7f485edca240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:06:17.208 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:17.212+0000 7f31e861b240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:06:17.279 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:17.284+0000 7f31e861b240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:06:17.356 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:17.360+0000 7f31e861b240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:06:17.382 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:17.386+0000 7f485edca240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:06:17.432 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:17.437+0000 7f31e861b240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:06:17.457 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:17.461+0000 7f485edca240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:06:17.509 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:17.514+0000 7f31e861b240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:06:17.538 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:17.543+0000 7f485edca240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:06:17.620 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:17.624+0000 7f485edca240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:06:17.676 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:17.681+0000 7f31e861b240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:06:17.701 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:17.705+0000 7f485edca240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:06:17.878 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:17.883+0000 7f485edca240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:06:17.889 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:17.894+0000 7f31e861b240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:06:18.037 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:18.042+0000 7f31e861b240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:06:18.101 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:18.105+0000 7f485edca240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:06:18.108 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:18.113+0000 7f31e861b240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:06:18.259 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:18.264+0000 7f485edca240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:06:18.335 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:18.339+0000 7f485edca240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:06:18.547 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:18.552+0000 7f31e861b240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:06:18.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:18.655+0000 7fba44ed0700 1 -- 172.21.15.1:0/2925433770 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2406ae00 msgr2=0x7fba2406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:18.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:18.655+0000 7fba44ed0700 1 --2- 172.21.15.1:0/2925433770 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2406ae00 0x7fba2406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T19:06:18.804 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:18.809+0000 7f485edca240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:06:18.881 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:18.886+0000 7f31e861b240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:06:19.041 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:19.046+0000 7f31e861b240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:06:19.155 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:19.159+0000 7f485edca240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:06:19.326 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:19.330+0000 7f485edca240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:06:19.423 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:19.428+0000 7f31e861b240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:06:19.585 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:19.590+0000 7f31e861b240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:06:19.734 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:19.739+0000 7f485edca240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:06:19.908 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:19.913+0000 7f485edca240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:06:19.943 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:19.948+0000 7f31e861b240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:06:20.028 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:20.032+0000 7f31e861b240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:06:20.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:20.102+0000 7fba3d7fa700 1 -- 172.21.15.1:0/2925433770 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 41) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fba2c086e90 con 0x7fba400cbce0 2024-03-18T19:06:20.289 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:20.294+0000 7f485edca240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:06:20.383 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:20.387+0000 7f485edca240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:06:20.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:20.406+0000 7fba3d7fa700 1 -- 172.21.15.1:0/2925433770 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 42) v1 ==== 88865+0+0 (secure 0 0 0) 0x7fba2c053e10 con 0x7fba400cbce0 2024-03-18T19:06:20.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:20.406+0000 7fba3d7fa700 1 -- 172.21.15.1:0/2925433770 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2406ae00 msgr2=0x7fba2406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:20.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:20.406+0000 7fba3d7fa700 1 --2- 172.21.15.1:0/2925433770 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2406ae00 0x7fba2406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:20.552 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:20.556+0000 7f47d49ed700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:20.552 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:20.556+0000 7f47d49ed700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:20.553 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:20.556+0000 7f47d49ed700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:20.553 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:20.556+0000 7f47d49ed700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:20.553 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:20.556+0000 7f47d49ed700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:20.553 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:20.556+0000 7f47d19e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:20.553 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:20.556+0000 7f47d19e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:20.553 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:20.556+0000 7f47d19e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:20.553 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:20.556+0000 7f47d19e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:20.553 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:20.556+0000 7f47d19e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:21.338 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:06:21.341 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:06:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.411+0000 7fba3d7fa700 1 -- 172.21.15.1:0/2925433770 <== mon.0 v2:172.21.15.1:3300/0 9 ==== mgrmap(e 43) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fba2c0736c0 con 0x7fba400cbce0 2024-03-18T19:06:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.412+0000 7fba3d7fa700 1 --2- 172.21.15.1:0/2925433770 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2406ae00 0x7fba2406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.412+0000 7fba3d7fa700 1 -- 172.21.15.1:0/2925433770 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fba2401cd40 con 0x7fba2406ae00 2024-03-18T19:06:21.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.414+0000 7fba44ed0700 1 --2- 172.21.15.1:0/2925433770 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2406ae00 0x7fba2406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:21.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.417+0000 7fba44ed0700 1 --2- 172.21.15.1:0/2925433770 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fba2406ae00 0x7fba2406d2c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fba34009810 tx=0x7fba34009210 comp rx=0 tx=0).ready entity=mgr.7467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:21.420 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:21.425+0000 7f47db9fb700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-03-18T19:06:21.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.425+0000 7fba3d7fa700 1 -- 172.21.15.1:0/2925433770 <== mgr.7467 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7fba2401cd40 con 0x7fba2406ae00 2024-03-18T19:06:21.422 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-03-18T19:06:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-18T19:06:21.427 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-18T19:06:21.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.605+0000 7f0b6fe07700 1 Processor -- start 2024-03-18T19:06:21.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.605+0000 7f0b6fe07700 1 -- start start 2024-03-18T19:06:21.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.606+0000 7f0b6fe07700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b6815d5a0 0x7f0b680ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:21.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.606+0000 7f0b6fe07700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0b680ce550 0x7f0b6806a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:21.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.606+0000 7f0b6fe07700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b680cf600 con 0x7f0b680cfa90 2024-03-18T19:06:21.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.606+0000 7f0b6fe07700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b680ce9f0 con 0x7f0b680ce550 2024-03-18T19:06:21.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.606+0000 7f0b6fe07700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b680cecd0 con 0x7f0b6815d5a0 2024-03-18T19:06:21.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.606+0000 7f0b6e3a4700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0b680cfa90 0x7f0b68071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53818/0 (socket says 172.21.15.1:53818) 2024-03-18T19:06:21.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.606+0000 7f0b6e3a4700 1 -- 172.21.15.1:0/2578819355 learned_addr learned my addr 172.21.15.1:0/2578819355 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:21.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.606+0000 7f0b6d3a2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0b680ce550 0x7f0b6806a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:21.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.606+0000 7f0b6dba3700 1 --2- 172.21.15.1:0/2578819355 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b6815d5a0 0x7f0b680ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:21.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.607+0000 7f0b6cba1700 1 -- 172.21.15.1:0/2578819355 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66962180 0 0) 0x7f0b680cf600 con 0x7f0b680cfa90 2024-03-18T19:06:21.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.607+0000 7f0b6d3a2700 1 -- 172.21.15.1:0/2578819355 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b6815d5a0 msgr2=0x7f0b680ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:21.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.607+0000 7f0b6d3a2700 1 --2- 172.21.15.1:0/2578819355 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b6815d5a0 0x7f0b680ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:21.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.607+0000 7f0b6d3a2700 1 -- 172.21.15.1:0/2578819355 >> v1:172.21.15.1:6789/0 conn(0x7f0b680cfa90 legacy=0x7f0b68071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:21.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.607+0000 7f0b6d3a2700 1 -- 172.21.15.1:0/2578819355 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b680cefb0 con 0x7f0b680ce550 2024-03-18T19:06:21.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.607+0000 7f0b6d3a2700 1 --2- 172.21.15.1:0/2578819355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b680ce550 0x7f0b6806a750 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f0b64009840 tx=0x7f0b64009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f271f988e7a7076d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:21.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.607+0000 7f0b6cba1700 1 -- 172.21.15.1:0/2578819355 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b64011830 con 0x7f0b680ce550 2024-03-18T19:06:21.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.607+0000 7f0b6cba1700 1 -- 172.21.15.1:0/2578819355 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b64011990 con 0x7f0b680ce550 2024-03-18T19:06:21.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.607+0000 7f0b6cba1700 1 -- 172.21.15.1:0/2578819355 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b64011d20 con 0x7f0b680ce550 2024-03-18T19:06:21.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.607+0000 7f0b6fe07700 1 -- 172.21.15.1:0/2578819355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b680ce550 msgr2=0x7f0b6806a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:21.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.607+0000 7f0b6fe07700 1 --2- 172.21.15.1:0/2578819355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b680ce550 0x7f0b6806a750 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f0b64009840 tx=0x7f0b64009b50 comp rx=0 tx=0).stop 2024-03-18T19:06:21.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.607+0000 7f0b6fe07700 1 -- 172.21.15.1:0/2578819355 shutdown_connections 2024-03-18T19:06:21.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.607+0000 7f0b6fe07700 1 --2- 172.21.15.1:0/2578819355 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b6815d5a0 0x7f0b680ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:21.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.608+0000 7f0b6fe07700 1 --2- 172.21.15.1:0/2578819355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b680ce550 0x7f0b6806a750 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:21.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.608+0000 7f0b6fe07700 1 -- 172.21.15.1:0/2578819355 >> 172.21.15.1:0/2578819355 conn(0x7f0b680cc9b0 msgr2=0x7f0b680ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:21.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.608+0000 7f0b6fe07700 1 -- 172.21.15.1:0/2578819355 shutdown_connections 2024-03-18T19:06:21.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.608+0000 7f0b6fe07700 1 -- 172.21.15.1:0/2578819355 wait complete. 2024-03-18T19:06:21.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.609+0000 7f0b6fe07700 1 Processor -- start 2024-03-18T19:06:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.609+0000 7f0b6fe07700 1 -- start start 2024-03-18T19:06:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.610+0000 7f0b6fe07700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b680ce550 0x7f0b680b5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.610+0000 7f0b6fe07700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b680cfa90 0x7f0b68152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.610+0000 7f0b6fe07700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b6815d5a0 0x7f0b68153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.610+0000 7f0b6fe07700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b681548c0 con 0x7f0b680cfa90 2024-03-18T19:06:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.610+0000 7f0b6fe07700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b68154ba0 con 0x7f0b680ce550 2024-03-18T19:06:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.610+0000 7f0b6fe07700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b68154e80 con 0x7f0b6815d5a0 2024-03-18T19:06:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.610+0000 7f0b6d3a2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b680cfa90 0x7f0b68152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.610+0000 7f0b6d3a2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b680cfa90 0x7f0b68152580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52058/0 (socket says 172.21.15.1:52058) 2024-03-18T19:06:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.610+0000 7f0b6d3a2700 1 -- 172.21.15.1:0/2327723751 learned_addr learned my addr 172.21.15.1:0/2327723751 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.610+0000 7f0b6dba3700 1 --2- 172.21.15.1:0/2327723751 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b680ce550 0x7f0b680b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.610+0000 7f0b6e3a4700 1 --2- 172.21.15.1:0/2327723751 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b6815d5a0 0x7f0b68153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.611+0000 7f0b6d3a2700 1 -- 172.21.15.1:0/2327723751 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b6815d5a0 msgr2=0x7f0b68153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:21.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.611+0000 7f0b6d3a2700 1 --2- 172.21.15.1:0/2327723751 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b6815d5a0 0x7f0b68153fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.611+0000 7f0b6d3a2700 1 -- 172.21.15.1:0/2327723751 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b680ce550 msgr2=0x7f0b680b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.611+0000 7f0b6d3a2700 1 --2- 172.21.15.1:0/2327723751 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b680ce550 0x7f0b680b5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.611+0000 7f0b6d3a2700 1 -- 172.21.15.1:0/2327723751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b640094a0 con 0x7f0b680cfa90 2024-03-18T19:06:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.611+0000 7f0b6d3a2700 1 --2- 172.21.15.1:0/2327723751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b680cfa90 0x7f0b68152580 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f0b64009b90 tx=0x7f0b64004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.611+0000 7f0b5e7fc700 1 -- 172.21.15.1:0/2327723751 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b640036a0 con 0x7f0b680cfa90 2024-03-18T19:06:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.611+0000 7f0b5e7fc700 1 -- 172.21.15.1:0/2327723751 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6401f070 con 0x7f0b680cfa90 2024-03-18T19:06:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.611+0000 7f0b6fe07700 1 -- 172.21.15.1:0/2327723751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b68155160 con 0x7f0b680cfa90 2024-03-18T19:06:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.611+0000 7f0b5e7fc700 1 -- 172.21.15.1:0/2327723751 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b64003a40 con 0x7f0b680cfa90 2024-03-18T19:06:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.611+0000 7f0b6fe07700 1 -- 172.21.15.1:0/2327723751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b68154520 con 0x7f0b680cfa90 2024-03-18T19:06:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.612+0000 7f0b6fe07700 1 -- 172.21.15.1:0/2327723751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b68155680 con 0x7f0b680cfa90 2024-03-18T19:06:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.612+0000 7f0b5e7fc700 1 -- 172.21.15.1:0/2327723751 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 43) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f0b64003ba0 con 0x7f0b680cfa90 2024-03-18T19:06:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.613+0000 7f0b5e7fc700 1 --2- 172.21.15.1:0/2327723751 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0b4c06ab30 0x7f0b4c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:21.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.613+0000 7f0b5e7fc700 1 -- 172.21.15.1:0/2327723751 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f0b64021070 con 0x7f0b680cfa90 2024-03-18T19:06:21.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.613+0000 7f0b6dba3700 1 --2- 172.21.15.1:0/2327723751 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0b4c06ab30 0x7f0b4c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:21.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.613+0000 7f0b6dba3700 1 --2- 172.21.15.1:0/2327723751 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0b4c06ab30 0x7f0b4c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0b58009b20 tx=0x7f0b580091c0 comp rx=0 tx=0).ready entity=mgr.7467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:21.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.617+0000 7f0b5e7fc700 1 -- 172.21.15.1:0/2327723751 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0b68155680 con 0x7f0b680cfa90 2024-03-18T19:06:21.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:21.882+0000 7f0b6fe07700 1 -- 172.21.15.1:0/2327723751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f0b68155680 con 0x7f0b680cfa90 2024-03-18T19:06:22.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.417+0000 7f0b5e7fc700 1 -- 172.21.15.1:0/2327723751 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 44) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f0b64056110 con 0x7f0b680cfa90 2024-03-18T19:06:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.418+0000 7f0b5e7fc700 1 -- 172.21.15.1:0/2327723751 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v44) v1 ==== 93+0+0 (secure 0 0 0) 0x7f0b6405c310 con 0x7f0b680cfa90 2024-03-18T19:06:22.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.421+0000 7f0b6fe07700 1 -- 172.21.15.1:0/2327723751 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0b4c06ab30 msgr2=0x7f0b4c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:22.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.421+0000 7f0b6fe07700 1 --2- 172.21.15.1:0/2327723751 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0b4c06ab30 0x7f0b4c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0b58009b20 tx=0x7f0b580091c0 comp rx=0 tx=0).stop 2024-03-18T19:06:22.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.421+0000 7f0b6fe07700 1 -- 172.21.15.1:0/2327723751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b680cfa90 msgr2=0x7f0b68152580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:22.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.421+0000 7f0b6fe07700 1 --2- 172.21.15.1:0/2327723751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b680cfa90 0x7f0b68152580 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f0b64009b90 tx=0x7f0b64004750 comp rx=0 tx=0).stop 2024-03-18T19:06:22.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.421+0000 7f0b6fe07700 1 -- 172.21.15.1:0/2327723751 shutdown_connections 2024-03-18T19:06:22.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.421+0000 7f0b6fe07700 1 --2- 172.21.15.1:0/2327723751 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0b4c06ab30 0x7f0b4c06cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:22.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.421+0000 7f0b6fe07700 1 --2- 172.21.15.1:0/2327723751 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b6815d5a0 0x7f0b68153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:22.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.421+0000 7f0b6fe07700 1 --2- 172.21.15.1:0/2327723751 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b680ce550 0x7f0b680b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:22.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.421+0000 7f0b6fe07700 1 --2- 172.21.15.1:0/2327723751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b680cfa90 0x7f0b68152580 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:22.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.421+0000 7f0b6fe07700 1 -- 172.21.15.1:0/2327723751 >> 172.21.15.1:0/2327723751 conn(0x7f0b680cc9b0 msgr2=0x7f0b680afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:22.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.421+0000 7f0b6fe07700 1 -- 172.21.15.1:0/2327723751 shutdown_connections 2024-03-18T19:06:22.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.421+0000 7f0b6fe07700 1 -- 172.21.15.1:0/2327723751 wait complete. 2024-03-18T19:06:22.433 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T19:06:22.433 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:06:22.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:06:22.518 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T19:06:22.518 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:06:22.518 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:06:22.537 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:06:22.558 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:22.563+0000 7fc75dbc3240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:06:22.565 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:22.569+0000 7f58d2245240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:06:22.705 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:22.709+0000 7fc75dbc3240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:06:22.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.735+0000 7f5dc5513700 1 Processor -- start 2024-03-18T19:06:22.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.736+0000 7f5dc5513700 1 -- start start 2024-03-18T19:06:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.738+0000 7f5dc5513700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dc0143450 0x7f5dc0167f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.738+0000 7f5dc5513700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc0142a40 con 0x7f5dc00cfb80 2024-03-18T19:06:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.738+0000 7f5dc5513700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc0140c70 con 0x7f5dc0140610 2024-03-18T19:06:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.738+0000 7f5dc5513700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc0140f50 con 0x7f5dc0143450 2024-03-18T19:06:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.738+0000 7f5dbf7fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dc0143450 0x7f5dc0167f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.738+0000 7f5dbeffd700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f5dc0140610 0x7f5dc00cf470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:41848/0 (socket says 172.21.15.1:41848) 2024-03-18T19:06:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.738+0000 7f5dbeffd700 1 -- 172.21.15.1:0/2863371091 learned_addr learned my addr 172.21.15.1:0/2863371091 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.738+0000 7f5dbf7fe700 1 -- 172.21.15.1:0/2863371091 >> v1:172.21.15.165:6789/0 conn(0x7f5dc0140610 legacy=0x7f5dc00cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.738+0000 7f5dbf7fe700 1 -- 172.21.15.1:0/2863371091 >> v1:172.21.15.1:6789/0 conn(0x7f5dc00cfb80 legacy=0x7f5dc0142d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.738+0000 7f5dbdffb700 1 -- 172.21.15.1:0/2863371091 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47521861 0 0) 0x7f5dc0140c70 con 0x7f5dc0140610 2024-03-18T19:06:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.738+0000 7f5dbf7fe700 1 -- 172.21.15.1:0/2863371091 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc0141230 con 0x7f5dc0143450 2024-03-18T19:06:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.739+0000 7f5dbf7fe700 1 --2- 172.21.15.1:0/2863371091 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dc0143450 0x7f5dc0167f10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5db8009dc0 tx=0x7f5db800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76d25106f843c326 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.739+0000 7f5dbdffb700 1 -- 172.21.15.1:0/2863371091 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db800e070 con 0x7f5dc0143450 2024-03-18T19:06:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.739+0000 7f5dbdffb700 1 -- 172.21.15.1:0/2863371091 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db800bb90 con 0x7f5dc0143450 2024-03-18T19:06:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.739+0000 7f5dbdffb700 1 -- 172.21.15.1:0/2863371091 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db8003a20 con 0x7f5dc0143450 2024-03-18T19:06:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.739+0000 7f5dc5513700 1 -- 172.21.15.1:0/2863371091 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dc0143450 msgr2=0x7f5dc0167f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:22.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.739+0000 7f5dc5513700 1 --2- 172.21.15.1:0/2863371091 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dc0143450 0x7f5dc0167f10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5db8009dc0 tx=0x7f5db800b330 comp rx=0 tx=0).stop 2024-03-18T19:06:22.737 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:22.740+0000 7f58d2245240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:06:22.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.740+0000 7f5dc5513700 1 -- 172.21.15.1:0/2863371091 shutdown_connections 2024-03-18T19:06:22.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.740+0000 7f5dc5513700 1 --2- 172.21.15.1:0/2863371091 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dc0143450 0x7f5dc0167f10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:22.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.740+0000 7f5dc5513700 1 -- 172.21.15.1:0/2863371091 >> 172.21.15.1:0/2863371091 conn(0x7f5dc0066960 msgr2=0x7f5dc006a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:22.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.740+0000 7f5dc5513700 1 -- 172.21.15.1:0/2863371091 shutdown_connections 2024-03-18T19:06:22.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.740+0000 7f5dc5513700 1 -- 172.21.15.1:0/2863371091 wait complete. 2024-03-18T19:06:22.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.742+0000 7f5dc5513700 1 Processor -- start 2024-03-18T19:06:22.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.742+0000 7f5dc5513700 1 -- start start 2024-03-18T19:06:22.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.743+0000 7f5dc5513700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dc00cfb80 0x7f5dc015c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.743+0000 7f5dc5513700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dc0140610 0x7f5dc0145c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.743+0000 7f5dc5513700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dc0143450 0x7f5dc006d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.743+0000 7f5dc5513700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc015cb50 con 0x7f5dc00cfb80 2024-03-18T19:06:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.743+0000 7f5dc5513700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc00b06b0 con 0x7f5dc0143450 2024-03-18T19:06:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.743+0000 7f5dc5513700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc00af610 con 0x7f5dc0140610 2024-03-18T19:06:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.743+0000 7f5dbeffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dc00cfb80 0x7f5dc015c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.743+0000 7f5dbeffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dc00cfb80 0x7f5dc015c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52082/0 (socket says 172.21.15.1:52082) 2024-03-18T19:06:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.743+0000 7f5dbeffd700 1 -- 172.21.15.1:0/2662100802 learned_addr learned my addr 172.21.15.1:0/2662100802 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.743+0000 7f5dbf7fe700 1 --2- 172.21.15.1:0/2662100802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dc0143450 0x7f5dc006d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.743+0000 7f5dbe7fc700 1 --2- 172.21.15.1:0/2662100802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dc0140610 0x7f5dc0145c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.744+0000 7f5dbeffd700 1 -- 172.21.15.1:0/2662100802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dc0140610 msgr2=0x7f5dc0145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.744+0000 7f5dbeffd700 1 --2- 172.21.15.1:0/2662100802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dc0140610 0x7f5dc0145c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.744+0000 7f5dbeffd700 1 -- 172.21.15.1:0/2662100802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dc0143450 msgr2=0x7f5dc006d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.744+0000 7f5dbeffd700 1 --2- 172.21.15.1:0/2662100802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dc0143450 0x7f5dc006d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.744+0000 7f5dbeffd700 1 -- 172.21.15.1:0/2662100802 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db8009a70 con 0x7f5dc00cfb80 2024-03-18T19:06:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.744+0000 7f5dbeffd700 1 --2- 172.21.15.1:0/2662100802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dc00cfb80 0x7f5dc015c610 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f5db000c520 tx=0x7f5db000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.744+0000 7f5da77fe700 1 -- 172.21.15.1:0/2662100802 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db0004c80 con 0x7f5dc00cfb80 2024-03-18T19:06:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.744+0000 7f5da77fe700 1 -- 172.21.15.1:0/2662100802 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db0004de0 con 0x7f5dc00cfb80 2024-03-18T19:06:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.744+0000 7f5da77fe700 1 -- 172.21.15.1:0/2662100802 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db0007310 con 0x7f5dc00cfb80 2024-03-18T19:06:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.744+0000 7f5dc5513700 1 -- 172.21.15.1:0/2662100802 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc006cf90 con 0x7f5dc00cfb80 2024-03-18T19:06:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.744+0000 7f5dc5513700 1 -- 172.21.15.1:0/2662100802 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc00afbd0 con 0x7f5dc00cfb80 2024-03-18T19:06:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.745+0000 7f5dc5513700 1 -- 172.21.15.1:0/2662100802 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc00aef00 con 0x7f5dc00cfb80 2024-03-18T19:06:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.746+0000 7f5da77fe700 1 -- 172.21.15.1:0/2662100802 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 44) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f5db0007470 con 0x7f5dc00cfb80 2024-03-18T19:06:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.746+0000 7f5da77fe700 1 --2- 172.21.15.1:0/2662100802 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5dac06aae0 0x7f5dac06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.746+0000 7f5dbe7fc700 1 -- 172.21.15.1:0/2662100802 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5dac06aae0 msgr2=0x7f5dac06cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.746+0000 7f5da77fe700 1 -- 172.21.15.1:0/2662100802 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f5db0088f00 con 0x7f5dc00cfb80 2024-03-18T19:06:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.746+0000 7f5dbe7fc700 1 --2- 172.21.15.1:0/2662100802 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5dac06aae0 0x7f5dac06cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:22.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.751+0000 7f5da77fe700 1 -- 172.21.15.1:0/2662100802 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5db0054020 con 0x7f5dc00cfb80 2024-03-18T19:06:22.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.947+0000 7f5dbe7fc700 1 -- 172.21.15.1:0/2662100802 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5dac06aae0 msgr2=0x7f5dac06cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.947+0000 7f5dbe7fc700 1 --2- 172.21.15.1:0/2662100802 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5dac06aae0 0x7f5dac06cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:22.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.953+0000 7f5dc5513700 1 -- 172.21.15.1:0/2662100802 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5dc0073350 con 0x7f5dc00cfb80 2024-03-18T19:06:22.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.955+0000 7f5da77fe700 1 -- 172.21.15.1:0/2662100802 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+16754 (secure 0 0 0) 0x7f5db0057640 con 0x7f5dc00cfb80 2024-03-18T19:06:22.951 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:22.951 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":104,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:06:20.400812+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:06:02.855989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-18T19:06:04.121199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/1060785910":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/3060583633":"2024-03-19T19:06:20.400777+0000","172.21.15.1:6833/76998":"2024-03-19T19:06:20.400777+0000","172.21.15.1:6832/76998":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/809308784":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/1836399926":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/3606402343":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/2494839321":"2024-03-19T19:06:20.400777+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:06:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.957+0000 7f5dc5513700 1 -- 172.21.15.1:0/2662100802 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5dac06aae0 msgr2=0x7f5dac06cfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.957+0000 7f5dc5513700 1 --2- 172.21.15.1:0/2662100802 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5dac06aae0 0x7f5dac06cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.957+0000 7f5dc5513700 1 -- 172.21.15.1:0/2662100802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dc00cfb80 msgr2=0x7f5dc015c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.957+0000 7f5dc5513700 1 --2- 172.21.15.1:0/2662100802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dc00cfb80 0x7f5dc015c610 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f5db000c520 tx=0x7f5db000c830 comp rx=0 tx=0).stop 2024-03-18T19:06:22.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.957+0000 7f5dc5513700 1 -- 172.21.15.1:0/2662100802 shutdown_connections 2024-03-18T19:06:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.957+0000 7f5dc5513700 1 --2- 172.21.15.1:0/2662100802 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5dac06aae0 0x7f5dac06cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.957+0000 7f5dc5513700 1 --2- 172.21.15.1:0/2662100802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dc0143450 0x7f5dc006d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.957+0000 7f5dc5513700 1 --2- 172.21.15.1:0/2662100802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dc00cfb80 0x7f5dc015c610 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.957+0000 7f5dc5513700 1 --2- 172.21.15.1:0/2662100802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dc0140610 0x7f5dc0145c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.957+0000 7f5dc5513700 1 -- 172.21.15.1:0/2662100802 >> 172.21.15.1:0/2662100802 conn(0x7f5dc0066960 msgr2=0x7f5dc00cf470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.957+0000 7f5dc5513700 1 -- 172.21.15.1:0/2662100802 shutdown_connections 2024-03-18T19:06:22.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:22.957+0000 7f5dc5513700 1 -- 172.21.15.1:0/2662100802 wait complete. 2024-03-18T19:06:22.965 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T19:06:02.855989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T19:06:04.121199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:06:22.966 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:06:22.973 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:22.976+0000 7fc75dbc3240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:06:23.020 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:23.024+0000 7f58d2245240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:06:23.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.107+0000 7f1548746700 1 Processor -- start 2024-03-18T19:06:23.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.107+0000 7f1548746700 1 -- start start 2024-03-18T19:06:23.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.108+0000 7f1548746700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1540071550 0x7f15400b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:23.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.108+0000 7f1548746700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15400b2b80 con 0x7f15400b2650 2024-03-18T19:06:23.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.108+0000 7f1548746700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15400b2e60 con 0x7f154015d3b0 2024-03-18T19:06:23.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.108+0000 7f1548746700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15400b3800 con 0x7f1540071550 2024-03-18T19:06:23.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.108+0000 7f1545ce1700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f15400b2650 0x7f1540070e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53844/0 (socket says 172.21.15.1:53844) 2024-03-18T19:06:23.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.108+0000 7f1545ce1700 1 -- 172.21.15.1:0/377950040 learned_addr learned my addr 172.21.15.1:0/377950040 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:23.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.108+0000 7f1546ce3700 1 --2- 172.21.15.1:0/377950040 >> v2:172.21.15.165:3300/0 conn(0x7f1540071550 0x7f15400b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.108+0000 7f15454e0700 1 -- 172.21.15.1:0/377950040 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174404341 0 0) 0x7f15400b2b80 con 0x7f15400b2650 2024-03-18T19:06:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.108+0000 7f15454e0700 1 -- 172.21.15.1:0/377950040 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f152c0035b0 con 0x7f15400b2650 2024-03-18T19:06:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.109+0000 7f15454e0700 1 -- 172.21.15.1:0/377950040 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932676775 0 0) 0x7f15400b2e60 con 0x7f154015d3b0 2024-03-18T19:06:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.109+0000 7f15454e0700 1 -- 172.21.15.1:0/377950040 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15400b2b80 con 0x7f154015d3b0 2024-03-18T19:06:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.109+0000 7f15454e0700 1 -- 172.21.15.1:0/377950040 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 741699770 0 0) 0x7f152c0035b0 con 0x7f15400b2650 2024-03-18T19:06:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.109+0000 7f1546ce3700 1 -- 172.21.15.1:0/377950040 >> v1:172.21.15.165:6789/0 conn(0x7f154015d3b0 legacy=0x7f154006cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.109+0000 7f1546ce3700 1 -- 172.21.15.1:0/377950040 >> v1:172.21.15.1:6789/0 conn(0x7f15400b2650 legacy=0x7f1540070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.109+0000 7f1546ce3700 1 -- 172.21.15.1:0/377950040 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15400b3140 con 0x7f1540071550 2024-03-18T19:06:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.109+0000 7f1546ce3700 1 --2- 172.21.15.1:0/377950040 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1540071550 0x7f15400b3c20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1538009d90 tx=0x7f153800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=398c65b6faa2ca3a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.109+0000 7f15454e0700 1 -- 172.21.15.1:0/377950040 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153800e070 con 0x7f1540071550 2024-03-18T19:06:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.109+0000 7f15454e0700 1 -- 172.21.15.1:0/377950040 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1538003a20 con 0x7f1540071550 2024-03-18T19:06:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.109+0000 7f15454e0700 1 -- 172.21.15.1:0/377950040 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1530001670 con 0x7f1540071550 2024-03-18T19:06:23.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.110+0000 7f1548746700 1 -- 172.21.15.1:0/377950040 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1540071550 msgr2=0x7f15400b3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:23.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.110+0000 7f1548746700 1 --2- 172.21.15.1:0/377950040 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1540071550 0x7f15400b3c20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1538009d90 tx=0x7f153800b330 comp rx=0 tx=0).stop 2024-03-18T19:06:23.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.110+0000 7f1548746700 1 -- 172.21.15.1:0/377950040 shutdown_connections 2024-03-18T19:06:23.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.110+0000 7f1548746700 1 --2- 172.21.15.1:0/377950040 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1540071550 0x7f15400b3c20 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:23.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.110+0000 7f1548746700 1 -- 172.21.15.1:0/377950040 >> 172.21.15.1:0/377950040 conn(0x7f15400b1350 msgr2=0x7f15400b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:23.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.110+0000 7f1548746700 1 -- 172.21.15.1:0/377950040 shutdown_connections 2024-03-18T19:06:23.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.110+0000 7f1548746700 1 -- 172.21.15.1:0/377950040 wait complete. 2024-03-18T19:06:23.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.111+0000 7f1548746700 1 Processor -- start 2024-03-18T19:06:23.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.112+0000 7f1548746700 1 -- start start 2024-03-18T19:06:23.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.112+0000 7f1548746700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1540071550 0x7f15400cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:23.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.112+0000 7f1548746700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15400b2650 0x7f15400ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:23.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.112+0000 7f1548746700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f154015d3b0 0x7f15400c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.112+0000 7f1548746700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154015be70 con 0x7f1540071550 2024-03-18T19:06:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.112+0000 7f1548746700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15400c8b70 con 0x7f154015d3b0 2024-03-18T19:06:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.112+0000 7f1548746700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15400c8890 con 0x7f15400b2650 2024-03-18T19:06:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.113+0000 7f15464e2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1540071550 0x7f15400cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.113+0000 7f15464e2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1540071550 0x7f15400cc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52088/0 (socket says 172.21.15.1:52088) 2024-03-18T19:06:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.113+0000 7f15464e2700 1 -- 172.21.15.1:0/2607461253 learned_addr learned my addr 172.21.15.1:0/2607461253 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.113+0000 7f1546ce3700 1 --2- 172.21.15.1:0/2607461253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f154015d3b0 0x7f15400c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.113+0000 7f1545ce1700 1 --2- 172.21.15.1:0/2607461253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15400b2650 0x7f15400ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.113+0000 7f15464e2700 1 -- 172.21.15.1:0/2607461253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15400b2650 msgr2=0x7f15400ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.113+0000 7f15464e2700 1 --2- 172.21.15.1:0/2607461253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15400b2650 0x7f15400ccdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.113+0000 7f15464e2700 1 -- 172.21.15.1:0/2607461253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f154015d3b0 msgr2=0x7f15400c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.113+0000 7f15464e2700 1 --2- 172.21.15.1:0/2607461253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f154015d3b0 0x7f15400c7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.113+0000 7f15464e2700 1 -- 172.21.15.1:0/2607461253 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1538009a70 con 0x7f1540071550 2024-03-18T19:06:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.113+0000 7f15464e2700 1 --2- 172.21.15.1:0/2607461253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1540071550 0x7f15400cc890 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f153c00c500 tx=0x7f153c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.113+0000 7f1536ffd700 1 -- 172.21.15.1:0/2607461253 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c0052d0 con 0x7f1540071550 2024-03-18T19:06:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.113+0000 7f1536ffd700 1 -- 172.21.15.1:0/2607461253 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153c005450 con 0x7f1540071550 2024-03-18T19:06:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.113+0000 7f1548746700 1 -- 172.21.15.1:0/2607461253 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15400ceb10 con 0x7f1540071550 2024-03-18T19:06:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.113+0000 7f1536ffd700 1 -- 172.21.15.1:0/2607461253 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c0038c0 con 0x7f1540071550 2024-03-18T19:06:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.113+0000 7f1548746700 1 -- 172.21.15.1:0/2607461253 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15400cf0d0 con 0x7f1540071550 2024-03-18T19:06:23.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.114+0000 7f1548746700 1 -- 172.21.15.1:0/2607461253 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15400676a0 con 0x7f1540071550 2024-03-18T19:06:23.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.116+0000 7f1536ffd700 1 -- 172.21.15.1:0/2607461253 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 44) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f153c003a20 con 0x7f1540071550 2024-03-18T19:06:23.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.116+0000 7f1536ffd700 1 --2- 172.21.15.1:0/2607461253 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f152406aae0 0x7f152406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:23.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.116+0000 7f1545ce1700 1 -- 172.21.15.1:0/2607461253 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f152406aae0 msgr2=0x7f152406cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:23.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.116+0000 7f1536ffd700 1 -- 172.21.15.1:0/2607461253 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f153c088ab0 con 0x7f1540071550 2024-03-18T19:06:23.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.116+0000 7f1545ce1700 1 --2- 172.21.15.1:0/2607461253 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f152406aae0 0x7f152406cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:23.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.120+0000 7f1536ffd700 1 -- 172.21.15.1:0/2607461253 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f153c053c60 con 0x7f1540071550 2024-03-18T19:06:23.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.317+0000 7f1545ce1700 1 -- 172.21.15.1:0/2607461253 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f152406aae0 msgr2=0x7f152406cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:23.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.317+0000 7f1545ce1700 1 --2- 172.21.15.1:0/2607461253 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f152406aae0 0x7f152406cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:23.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.324+0000 7f1548746700 1 -- 172.21.15.1:0/2607461253 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1540072010 con 0x7f1540071550 2024-03-18T19:06:23.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.324+0000 7f1536ffd700 1 -- 172.21.15.1:0/2607461253 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7f153c057280 con 0x7f1540071550 2024-03-18T19:06:23.320 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:06:23.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.326+0000 7f1548746700 1 -- 172.21.15.1:0/2607461253 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f152406aae0 msgr2=0x7f152406cfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:23.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.326+0000 7f1548746700 1 --2- 172.21.15.1:0/2607461253 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f152406aae0 0x7f152406cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:23.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.326+0000 7f1548746700 1 -- 172.21.15.1:0/2607461253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1540071550 msgr2=0x7f15400cc890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:23.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.326+0000 7f1548746700 1 --2- 172.21.15.1:0/2607461253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1540071550 0x7f15400cc890 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f153c00c500 tx=0x7f153c00c810 comp rx=0 tx=0).stop 2024-03-18T19:06:23.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.326+0000 7f1548746700 1 -- 172.21.15.1:0/2607461253 shutdown_connections 2024-03-18T19:06:23.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.326+0000 7f1548746700 1 --2- 172.21.15.1:0/2607461253 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f152406aae0 0x7f152406cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:23.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.326+0000 7f1548746700 1 --2- 172.21.15.1:0/2607461253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f154015d3b0 0x7f15400c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:23.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.326+0000 7f1548746700 1 --2- 172.21.15.1:0/2607461253 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1540071550 0x7f15400cc890 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:23.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.327+0000 7f1548746700 1 --2- 172.21.15.1:0/2607461253 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15400b2650 0x7f15400ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:23.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.327+0000 7f1548746700 1 -- 172.21.15.1:0/2607461253 >> 172.21.15.1:0/2607461253 conn(0x7f15400b1350 msgr2=0x7f154015c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:23.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.327+0000 7f1548746700 1 -- 172.21.15.1:0/2607461253 shutdown_connections 2024-03-18T19:06:23.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.327+0000 7f1548746700 1 -- 172.21.15.1:0/2607461253 wait complete. 2024-03-18T19:06:23.335 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T19:06:23.400 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:23.405+0000 7fc75dbc3240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:06:23.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.471+0000 7f7b045c2700 1 Processor -- start 2024-03-18T19:06:23.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.471+0000 7f7b045c2700 1 -- start start 2024-03-18T19:06:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.472+0000 7f7b045c2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7afc160e50 0x7f7afc0cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.472+0000 7f7b045c2700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7afc162170 con 0x7f7afc160a70 2024-03-18T19:06:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.472+0000 7f7b045c2700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7afc0cef30 con 0x7f7afc1612d0 2024-03-18T19:06:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.472+0000 7f7b045c2700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7afc0cf210 con 0x7f7afc160e50 2024-03-18T19:06:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.472+0000 7f7b01b5d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7afc160e50 0x7f7afc0cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.472+0000 7f7b0235e700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7afc160a70 0x7f7afc0d3b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53856/0 (socket says 172.21.15.1:53856) 2024-03-18T19:06:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.472+0000 7f7b0235e700 1 -- 172.21.15.1:0/432261993 learned_addr learned my addr 172.21.15.1:0/432261993 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.473+0000 7f7b01b5d700 1 -- 172.21.15.1:0/432261993 >> v1:172.21.15.165:6789/0 conn(0x7f7afc1612d0 legacy=0x7f7afc1616f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.473+0000 7f7b01b5d700 1 -- 172.21.15.1:0/432261993 >> v1:172.21.15.1:6789/0 conn(0x7f7afc160a70 legacy=0x7f7afc0d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.473+0000 7f7b01b5d700 1 -- 172.21.15.1:0/432261993 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc0cf4f0 con 0x7f7afc160e50 2024-03-18T19:06:23.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.473+0000 7f7b0135c700 1 -- 172.21.15.1:0/432261993 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234228043 0 0) 0x7f7af8001110 con 0x7f7afc160a70 2024-03-18T19:06:23.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.473+0000 7f7b01b5d700 1 --2- 172.21.15.1:0/432261993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7afc160e50 0x7f7afc0cffd0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f7aec009840 tx=0x7f7aec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27dcbf4e7bb52461 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:23.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.473+0000 7f7b0135c700 1 -- 172.21.15.1:0/432261993 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec011830 con 0x7f7afc160e50 2024-03-18T19:06:23.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.473+0000 7f7b0135c700 1 -- 172.21.15.1:0/432261993 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aec011990 con 0x7f7afc160e50 2024-03-18T19:06:23.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.473+0000 7f7b0135c700 1 -- 172.21.15.1:0/432261993 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec011d20 con 0x7f7afc160e50 2024-03-18T19:06:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.474+0000 7f7b045c2700 1 -- 172.21.15.1:0/432261993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7afc160e50 msgr2=0x7f7afc0cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.474+0000 7f7b045c2700 1 --2- 172.21.15.1:0/432261993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7afc160e50 0x7f7afc0cffd0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f7aec009840 tx=0x7f7aec009b50 comp rx=0 tx=0).stop 2024-03-18T19:06:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.474+0000 7f7b045c2700 1 -- 172.21.15.1:0/432261993 shutdown_connections 2024-03-18T19:06:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.474+0000 7f7b045c2700 1 --2- 172.21.15.1:0/432261993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7afc160e50 0x7f7afc0cffd0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.474+0000 7f7b045c2700 1 -- 172.21.15.1:0/432261993 >> 172.21.15.1:0/432261993 conn(0x7f7afc09c560 msgr2=0x7f7afc0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.474+0000 7f7b045c2700 1 -- 172.21.15.1:0/432261993 shutdown_connections 2024-03-18T19:06:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.474+0000 7f7b045c2700 1 -- 172.21.15.1:0/432261993 wait complete. 2024-03-18T19:06:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.475+0000 7f7b045c2700 1 Processor -- start 2024-03-18T19:06:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.475+0000 7f7b045c2700 1 -- start start 2024-03-18T19:06:23.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.476+0000 7f7b045c2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7afc160a70 0x7f7afc151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.476+0000 7f7b045c2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7afc160e50 0x7f7afc15f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.476+0000 7f7b045c2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7afc1612d0 0x7f7afc15fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.476+0000 7f7b045c2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc151e80 con 0x7f7afc160a70 2024-03-18T19:06:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.476+0000 7f7b045c2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc0b7b50 con 0x7f7afc160e50 2024-03-18T19:06:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.476+0000 7f7b045c2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc0b7e30 con 0x7f7afc1612d0 2024-03-18T19:06:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.476+0000 7f7b0235e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7afc160a70 0x7f7afc151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.476+0000 7f7b02b5f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7afc1612d0 0x7f7afc15fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.476+0000 7f7b01b5d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7afc160e50 0x7f7afc15f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.476+0000 7f7b0235e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7afc160a70 0x7f7afc151940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52106/0 (socket says 172.21.15.1:52106) 2024-03-18T19:06:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.476+0000 7f7b0235e700 1 -- 172.21.15.1:0/2061155383 learned_addr learned my addr 172.21.15.1:0/2061155383 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.476+0000 7f7b0235e700 1 -- 172.21.15.1:0/2061155383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7afc1612d0 msgr2=0x7f7afc15fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.476+0000 7f7b0235e700 1 --2- 172.21.15.1:0/2061155383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7afc1612d0 0x7f7afc15fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.476+0000 7f7b0235e700 1 -- 172.21.15.1:0/2061155383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7afc160e50 msgr2=0x7f7afc15f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.476+0000 7f7b0235e700 1 --2- 172.21.15.1:0/2061155383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7afc160e50 0x7f7afc15f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.476+0000 7f7b0235e700 1 -- 172.21.15.1:0/2061155383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aec0094a0 con 0x7f7afc160a70 2024-03-18T19:06:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.476+0000 7f7b0235e700 1 --2- 172.21.15.1:0/2061155383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7afc160a70 0x7f7afc151940 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f7af800c520 tx=0x7f7af800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.477+0000 7f7af2ffd700 1 -- 172.21.15.1:0/2061155383 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8004c80 con 0x7f7afc160a70 2024-03-18T19:06:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.477+0000 7f7af2ffd700 1 -- 172.21.15.1:0/2061155383 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af8004de0 con 0x7f7afc160a70 2024-03-18T19:06:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.477+0000 7f7b045c2700 1 -- 172.21.15.1:0/2061155383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7afc0b8110 con 0x7f7afc160a70 2024-03-18T19:06:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.477+0000 7f7af2ffd700 1 -- 172.21.15.1:0/2061155383 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8007310 con 0x7f7afc160a70 2024-03-18T19:06:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.477+0000 7f7b045c2700 1 -- 172.21.15.1:0/2061155383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7afc0b86d0 con 0x7f7afc160a70 2024-03-18T19:06:23.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.478+0000 7f7b045c2700 1 -- 172.21.15.1:0/2061155383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7afc067390 con 0x7f7afc160a70 2024-03-18T19:06:23.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.479+0000 7f7af2ffd700 1 -- 172.21.15.1:0/2061155383 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 45) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f7af8007470 con 0x7f7afc160a70 2024-03-18T19:06:23.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.480+0000 7f7af2ffd700 1 --2- 172.21.15.1:0/2061155383 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7ae006adb0 0x7f7ae006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:23.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.480+0000 7f7b01b5d700 1 -- 172.21.15.1:0/2061155383 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7ae006adb0 msgr2=0x7f7ae006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:23.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.480+0000 7f7af2ffd700 1 -- 172.21.15.1:0/2061155383 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f7af8089800 con 0x7f7afc160a70 2024-03-18T19:06:23.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.480+0000 7f7b01b5d700 1 --2- 172.21.15.1:0/2061155383 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7ae006adb0 0x7f7ae006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:23.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.483+0000 7f7af2ffd700 1 -- 172.21.15.1:0/2061155383 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7af8054630 con 0x7f7afc160a70 2024-03-18T19:06:23.487 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:23.491+0000 7f58d2245240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:06:23.612 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:23.617+0000 7fc75dbc3240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:06:23.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.681+0000 7f7b01b5d700 1 -- 172.21.15.1:0/2061155383 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7ae006adb0 msgr2=0x7f7ae006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:23.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.681+0000 7f7b01b5d700 1 --2- 172.21.15.1:0/2061155383 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7ae006adb0 0x7f7ae006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:23.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.685+0000 7f7b045c2700 1 -- 172.21.15.1:0/2061155383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7afc162170 con 0x7f7afc160a70 2024-03-18T19:06:23.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.686+0000 7f7af2ffd700 1 -- 172.21.15.1:0/2061155383 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v104) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7afc162170 con 0x7f7afc160a70 2024-03-18T19:06:23.682 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T19:06:23.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.688+0000 7f7b045c2700 1 -- 172.21.15.1:0/2061155383 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7ae006adb0 msgr2=0x7f7ae006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:23.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.688+0000 7f7b045c2700 1 --2- 172.21.15.1:0/2061155383 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7ae006adb0 0x7f7ae006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:23.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.688+0000 7f7b045c2700 1 -- 172.21.15.1:0/2061155383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7afc160a70 msgr2=0x7f7afc151940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:23.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.688+0000 7f7b045c2700 1 --2- 172.21.15.1:0/2061155383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7afc160a70 0x7f7afc151940 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f7af800c520 tx=0x7f7af800c830 comp rx=0 tx=0).stop 2024-03-18T19:06:23.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.688+0000 7f7b045c2700 1 -- 172.21.15.1:0/2061155383 shutdown_connections 2024-03-18T19:06:23.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.688+0000 7f7b045c2700 1 --2- 172.21.15.1:0/2061155383 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7ae006adb0 0x7f7ae006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:23.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.688+0000 7f7b045c2700 1 --2- 172.21.15.1:0/2061155383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7afc1612d0 0x7f7afc15fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:23.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.688+0000 7f7b045c2700 1 --2- 172.21.15.1:0/2061155383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7afc160a70 0x7f7afc151940 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:23.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.688+0000 7f7b045c2700 1 --2- 172.21.15.1:0/2061155383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7afc160e50 0x7f7afc15f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:23.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.688+0000 7f7b045c2700 1 -- 172.21.15.1:0/2061155383 >> 172.21.15.1:0/2061155383 conn(0x7f7afc09c560 msgr2=0x7f7afc0d31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:23.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.688+0000 7f7b045c2700 1 -- 172.21.15.1:0/2061155383 shutdown_connections 2024-03-18T19:06:23.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.688+0000 7f7b045c2700 1 -- 172.21.15.1:0/2061155383 wait complete. 2024-03-18T19:06:23.697 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T19:06:23.709 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:23.713+0000 7f58d2245240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:06:23.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.830+0000 7f24e7646700 1 Processor -- start 2024-03-18T19:06:23.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.830+0000 7f24e7646700 1 -- start start 2024-03-18T19:06:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.831+0000 7f24e7646700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24e006a750 0x7f24e006ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.831+0000 7f24e7646700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e006b070 con 0x7f24e006d560 2024-03-18T19:06:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.831+0000 7f24e7646700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e015ef00 con 0x7f24e015b900 2024-03-18T19:06:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.831+0000 7f24e7646700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e015f1e0 con 0x7f24e006a750 2024-03-18T19:06:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.831+0000 7f24e4be1700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f24e006d560 0x7f24e00aefd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53858/0 (socket says 172.21.15.1:53858) 2024-03-18T19:06:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.831+0000 7f24e4be1700 1 -- 172.21.15.1:0/511174216 learned_addr learned my addr 172.21.15.1:0/511174216 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.831+0000 7f24e53e2700 1 --2- 172.21.15.1:0/511174216 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24e006a750 0x7f24e006ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.831+0000 7f24d7fff700 1 -- 172.21.15.1:0/511174216 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741703275 0 0) 0x7f24e006b070 con 0x7f24e006d560 2024-03-18T19:06:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.831+0000 7f24d7fff700 1 -- 172.21.15.1:0/511174216 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24cc0035b0 con 0x7f24e006d560 2024-03-18T19:06:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.831+0000 7f24d7fff700 1 -- 172.21.15.1:0/511174216 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230485133 0 0) 0x7f24e015ef00 con 0x7f24e015b900 2024-03-18T19:06:23.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.831+0000 7f24d7fff700 1 -- 172.21.15.1:0/511174216 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24e006b070 con 0x7f24e015b900 2024-03-18T19:06:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.831+0000 7f24e53e2700 1 -- 172.21.15.1:0/511174216 >> v1:172.21.15.165:6789/0 conn(0x7f24e015b900 legacy=0x7f24e015bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.831+0000 7f24e53e2700 1 -- 172.21.15.1:0/511174216 >> v1:172.21.15.1:6789/0 conn(0x7f24e006d560 legacy=0x7f24e00aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.831+0000 7f24e53e2700 1 -- 172.21.15.1:0/511174216 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e015f4c0 con 0x7f24e006a750 2024-03-18T19:06:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.831+0000 7f24e53e2700 1 --2- 172.21.15.1:0/511174216 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24e006a750 0x7f24e006ab30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f24dc009870 tx=0x7f24dc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4ad519f6ae0fd20 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.832+0000 7f24d7fff700 1 -- 172.21.15.1:0/511174216 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc011830 con 0x7f24e006a750 2024-03-18T19:06:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.832+0000 7f24d7fff700 1 -- 172.21.15.1:0/511174216 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24dc011990 con 0x7f24e006a750 2024-03-18T19:06:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.832+0000 7f24d7fff700 1 -- 172.21.15.1:0/511174216 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc011d20 con 0x7f24e006a750 2024-03-18T19:06:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.832+0000 7f24e7646700 1 -- 172.21.15.1:0/511174216 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24e006a750 msgr2=0x7f24e006ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.832+0000 7f24e7646700 1 --2- 172.21.15.1:0/511174216 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24e006a750 0x7f24e006ab30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f24dc009870 tx=0x7f24dc009b80 comp rx=0 tx=0).stop 2024-03-18T19:06:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.832+0000 7f24e7646700 1 -- 172.21.15.1:0/511174216 shutdown_connections 2024-03-18T19:06:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.832+0000 7f24e7646700 1 --2- 172.21.15.1:0/511174216 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24e006a750 0x7f24e006ab30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.832+0000 7f24e7646700 1 -- 172.21.15.1:0/511174216 >> 172.21.15.1:0/511174216 conn(0x7f24e00cff30 msgr2=0x7f24e00d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.833+0000 7f24e7646700 1 -- 172.21.15.1:0/511174216 shutdown_connections 2024-03-18T19:06:23.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.833+0000 7f24e7646700 1 -- 172.21.15.1:0/511174216 wait complete. 2024-03-18T19:06:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.834+0000 7f24e7646700 1 Processor -- start 2024-03-18T19:06:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.834+0000 7f24e7646700 1 -- start start 2024-03-18T19:06:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.834+0000 7f24e7646700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24e006a750 0x7f24e00c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.834+0000 7f24e7646700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24e006d560 0x7f24e0153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.834+0000 7f24e7646700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24e015b900 0x7f24e015e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.834+0000 7f24e7646700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e00c73c0 con 0x7f24e006a750 2024-03-18T19:06:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.834+0000 7f24e7646700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e00c76a0 con 0x7f24e015b900 2024-03-18T19:06:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.834+0000 7f24e7646700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e00c7980 con 0x7f24e006d560 2024-03-18T19:06:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.835+0000 7f24e53e2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24e006a750 0x7f24e00c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.835+0000 7f24e53e2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24e006a750 0x7f24e00c8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52116/0 (socket says 172.21.15.1:52116) 2024-03-18T19:06:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.835+0000 7f24e53e2700 1 -- 172.21.15.1:0/2473549427 learned_addr learned my addr 172.21.15.1:0/2473549427 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.835+0000 7f24e5be3700 1 --2- 172.21.15.1:0/2473549427 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24e015b900 0x7f24e015e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.835+0000 7f24e4be1700 1 --2- 172.21.15.1:0/2473549427 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24e006d560 0x7f24e0153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.835+0000 7f24e53e2700 1 -- 172.21.15.1:0/2473549427 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24e006d560 msgr2=0x7f24e0153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.835+0000 7f24e53e2700 1 --2- 172.21.15.1:0/2473549427 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24e006d560 0x7f24e0153fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.835+0000 7f24e53e2700 1 -- 172.21.15.1:0/2473549427 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24e015b900 msgr2=0x7f24e015e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.835+0000 7f24e53e2700 1 --2- 172.21.15.1:0/2473549427 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24e015b900 0x7f24e015e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.835+0000 7f24e53e2700 1 -- 172.21.15.1:0/2473549427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24dc0094a0 con 0x7f24e006a750 2024-03-18T19:06:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.835+0000 7f24e53e2700 1 --2- 172.21.15.1:0/2473549427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24e006a750 0x7f24e00c8460 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f24dc0090b0 tx=0x7f24dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.835+0000 7f24d5ffb700 1 -- 172.21.15.1:0/2473549427 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc01a6e0 con 0x7f24e006a750 2024-03-18T19:06:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.835+0000 7f24d5ffb700 1 -- 172.21.15.1:0/2473549427 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24dc01a840 con 0x7f24e006a750 2024-03-18T19:06:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.835+0000 7f24e7646700 1 -- 172.21.15.1:0/2473549427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e00c7c60 con 0x7f24e006a750 2024-03-18T19:06:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.835+0000 7f24d5ffb700 1 -- 172.21.15.1:0/2473549427 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc01abb0 con 0x7f24e006a750 2024-03-18T19:06:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.836+0000 7f24e7646700 1 -- 172.21.15.1:0/2473549427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e00c89a0 con 0x7f24e006a750 2024-03-18T19:06:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.837+0000 7f24e7646700 1 -- 172.21.15.1:0/2473549427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24e00676a0 con 0x7f24e006a750 2024-03-18T19:06:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.837+0000 7f24d5ffb700 1 -- 172.21.15.1:0/2473549427 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 45) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f24dc022070 con 0x7f24e006a750 2024-03-18T19:06:23.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.838+0000 7f24d5ffb700 1 --2- 172.21.15.1:0/2473549427 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f24c406adb0 0x7f24c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.838+0000 7f24e4be1700 1 -- 172.21.15.1:0/2473549427 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f24c406adb0 msgr2=0x7f24c406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.838+0000 7f24d5ffb700 1 -- 172.21.15.1:0/2473549427 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f24dc08a7b0 con 0x7f24e006a750 2024-03-18T19:06:23.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.838+0000 7f24e4be1700 1 --2- 172.21.15.1:0/2473549427 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f24c406adb0 0x7f24c406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:23.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:23.842+0000 7f24d5ffb700 1 -- 172.21.15.1:0/2473549427 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f24dc055790 con 0x7f24e006a750 2024-03-18T19:06:23.984 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:23.989+0000 7fc75dbc3240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:06:24.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.039+0000 7f24e4be1700 1 -- 172.21.15.1:0/2473549427 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f24c406adb0 msgr2=0x7f24c406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:24.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.039+0000 7f24e4be1700 1 --2- 172.21.15.1:0/2473549427 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f24c406adb0 0x7f24c406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:24.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.045+0000 7f24e7646700 1 -- 172.21.15.1:0/2473549427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f24e0070010 con 0x7f24e006a750 2024-03-18T19:06:24.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.046+0000 7f24d5ffb700 1 -- 172.21.15.1:0/2473549427 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v104) v1 ==== 100+0+11 (secure 0 0 0) 0x7f24dc015020 con 0x7f24e006a750 2024-03-18T19:06:24.042 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T19:06:24.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.048+0000 7f24e7646700 1 -- 172.21.15.1:0/2473549427 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f24c406adb0 msgr2=0x7f24c406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:24.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.048+0000 7f24e7646700 1 --2- 172.21.15.1:0/2473549427 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f24c406adb0 0x7f24c406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:24.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.048+0000 7f24e7646700 1 -- 172.21.15.1:0/2473549427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24e006a750 msgr2=0x7f24e00c8460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:24.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.048+0000 7f24e7646700 1 --2- 172.21.15.1:0/2473549427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24e006a750 0x7f24e00c8460 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f24dc0090b0 tx=0x7f24dc004750 comp rx=0 tx=0).stop 2024-03-18T19:06:24.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.048+0000 7f24e7646700 1 -- 172.21.15.1:0/2473549427 shutdown_connections 2024-03-18T19:06:24.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.048+0000 7f24e7646700 1 --2- 172.21.15.1:0/2473549427 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f24c406adb0 0x7f24c406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:24.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.048+0000 7f24e7646700 1 --2- 172.21.15.1:0/2473549427 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24e015b900 0x7f24e015e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:24.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.048+0000 7f24e7646700 1 --2- 172.21.15.1:0/2473549427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24e006a750 0x7f24e00c8460 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:24.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.048+0000 7f24e7646700 1 --2- 172.21.15.1:0/2473549427 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24e006d560 0x7f24e0153fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:24.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.048+0000 7f24e7646700 1 -- 172.21.15.1:0/2473549427 >> 172.21.15.1:0/2473549427 conn(0x7f24e00cff30 msgr2=0x7f24e006e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:24.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.048+0000 7f24e7646700 1 -- 172.21.15.1:0/2473549427 shutdown_connections 2024-03-18T19:06:24.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.048+0000 7f24e7646700 1 -- 172.21.15.1:0/2473549427 wait complete. 2024-03-18T19:06:24.057 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T19:06:24.107 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:24.112+0000 7f58d2245240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:06:24.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.231+0000 7f4d2581a700 1 Processor -- start 2024-03-18T19:06:24.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.231+0000 7f4d2581a700 1 -- start start 2024-03-18T19:06:24.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.232+0000 7f4d2581a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d20071cc0 0x7f4d200ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:24.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.232+0000 7f4d2581a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4d2006d560 0x7f4d2006fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:24.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.232+0000 7f4d2581a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d20070000 con 0x7f4d2015a240 2024-03-18T19:06:24.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.232+0000 7f4d2581a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d200adff0 con 0x7f4d2006d560 2024-03-18T19:06:24.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.232+0000 7f4d2581a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d200cac80 con 0x7f4d20071cc0 2024-03-18T19:06:24.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.232+0000 7f4d1effd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4d2015a240 0x7f4d200715b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53860/0 (socket says 172.21.15.1:53860) 2024-03-18T19:06:24.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.232+0000 7f4d1effd700 1 -- 172.21.15.1:0/2277658677 learned_addr learned my addr 172.21.15.1:0/2277658677 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:24.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.232+0000 7f4d1e7fc700 1 --2- 172.21.15.1:0/2277658677 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d20071cc0 0x7f4d200ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:24.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.232+0000 7f4d1f7fe700 1 --2- 172.21.15.1:0/2277658677 >> v2:172.21.15.165:3300/0 conn(0x7f4d2006d560 0x7f4d2006fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:24.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.232+0000 7f4d1dffb700 1 -- 172.21.15.1:0/2277658677 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435250527 0 0) 0x7f4d20070000 con 0x7f4d2015a240 2024-03-18T19:06:24.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.232+0000 7f4d1dffb700 1 -- 172.21.15.1:0/2277658677 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d080035b0 con 0x7f4d2015a240 2024-03-18T19:06:24.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.233+0000 7f4d1f7fe700 1 -- 172.21.15.1:0/2277658677 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d20071cc0 msgr2=0x7f4d200ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:24.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.233+0000 7f4d1f7fe700 1 --2- 172.21.15.1:0/2277658677 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d20071cc0 0x7f4d200ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:24.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.233+0000 7f4d1f7fe700 1 -- 172.21.15.1:0/2277658677 >> v1:172.21.15.1:6789/0 conn(0x7f4d2015a240 legacy=0x7f4d200715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:24.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.233+0000 7f4d1f7fe700 1 -- 172.21.15.1:0/2277658677 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d200caf60 con 0x7f4d2006d560 2024-03-18T19:06:24.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.233+0000 7f4d1e7fc700 1 --2- 172.21.15.1:0/2277658677 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d20071cc0 0x7f4d200ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:06:24.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.233+0000 7f4d1f7fe700 1 --2- 172.21.15.1:0/2277658677 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d2006d560 0x7f4d2006fac0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009d90 tx=0x7f4d1800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afa8568d6af62ec6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:24.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.233+0000 7f4d1dffb700 1 -- 172.21.15.1:0/2277658677 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1800e070 con 0x7f4d2006d560 2024-03-18T19:06:24.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.233+0000 7f4d1dffb700 1 -- 172.21.15.1:0/2277658677 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1800bb20 con 0x7f4d2006d560 2024-03-18T19:06:24.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.233+0000 7f4d1dffb700 1 -- 172.21.15.1:0/2277658677 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d18012430 con 0x7f4d2006d560 2024-03-18T19:06:24.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.234+0000 7f4d2581a700 1 -- 172.21.15.1:0/2277658677 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d2006d560 msgr2=0x7f4d2006fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:24.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.234+0000 7f4d2581a700 1 --2- 172.21.15.1:0/2277658677 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d2006d560 0x7f4d2006fac0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009d90 tx=0x7f4d1800b330 comp rx=0 tx=0).stop 2024-03-18T19:06:24.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.234+0000 7f4d2581a700 1 -- 172.21.15.1:0/2277658677 shutdown_connections 2024-03-18T19:06:24.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.234+0000 7f4d2581a700 1 --2- 172.21.15.1:0/2277658677 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d2006d560 0x7f4d2006fac0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:24.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.234+0000 7f4d2581a700 1 --2- 172.21.15.1:0/2277658677 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d20071cc0 0x7f4d200ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:24.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.234+0000 7f4d2581a700 1 -- 172.21.15.1:0/2277658677 >> 172.21.15.1:0/2277658677 conn(0x7f4d200cff30 msgr2=0x7f4d200d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:24.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.234+0000 7f4d2581a700 1 -- 172.21.15.1:0/2277658677 shutdown_connections 2024-03-18T19:06:24.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.234+0000 7f4d2581a700 1 -- 172.21.15.1:0/2277658677 wait complete. 2024-03-18T19:06:24.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.235+0000 7f4d2581a700 1 Processor -- start 2024-03-18T19:06:24.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.235+0000 7f4d2581a700 1 -- start start 2024-03-18T19:06:24.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.236+0000 7f4d2581a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d2006d560 0x7f4d200c8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:24.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.236+0000 7f4d2581a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d20071cc0 0x7f4d200c4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:24.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.236+0000 7f4d2581a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d2015a240 0x7f4d200aea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:24.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.236+0000 7f4d2581a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d20152730 con 0x7f4d2006d560 2024-03-18T19:06:24.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.236+0000 7f4d2581a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d200c86e0 con 0x7f4d20071cc0 2024-03-18T19:06:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.236+0000 7f4d2581a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d200c89c0 con 0x7f4d2015a240 2024-03-18T19:06:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.236+0000 7f4d1effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d2006d560 0x7f4d200c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.236+0000 7f4d1effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d2006d560 0x7f4d200c8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52130/0 (socket says 172.21.15.1:52130) 2024-03-18T19:06:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.236+0000 7f4d1effd700 1 -- 172.21.15.1:0/2249729159 learned_addr learned my addr 172.21.15.1:0/2249729159 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.236+0000 7f4d1f7fe700 1 --2- 172.21.15.1:0/2249729159 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d2015a240 0x7f4d200aea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.236+0000 7f4d1effd700 1 -- 172.21.15.1:0/2249729159 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d2015a240 msgr2=0x7f4d200aea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.236+0000 7f4d1e7fc700 1 --2- 172.21.15.1:0/2249729159 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d20071cc0 0x7f4d200c4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.236+0000 7f4d1effd700 1 --2- 172.21.15.1:0/2249729159 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d2015a240 0x7f4d200aea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.236+0000 7f4d1effd700 1 -- 172.21.15.1:0/2249729159 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d20071cc0 msgr2=0x7f4d200c4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.236+0000 7f4d1effd700 1 --2- 172.21.15.1:0/2249729159 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d20071cc0 0x7f4d200c4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.236+0000 7f4d1effd700 1 -- 172.21.15.1:0/2249729159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d140094a0 con 0x7f4d2006d560 2024-03-18T19:06:24.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.236+0000 7f4d1e7fc700 1 --2- 172.21.15.1:0/2249729159 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d20071cc0 0x7f4d200c4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:24.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.237+0000 7f4d1effd700 1 --2- 172.21.15.1:0/2249729159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d2006d560 0x7f4d200c8050 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f4d10001eb0 tx=0x7f4d1000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:24.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.237+0000 7f4d07fff700 1 -- 172.21.15.1:0/2249729159 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d10005d70 con 0x7f4d2006d560 2024-03-18T19:06:24.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.237+0000 7f4d07fff700 1 -- 172.21.15.1:0/2249729159 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d100070d0 con 0x7f4d2006d560 2024-03-18T19:06:24.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.237+0000 7f4d2581a700 1 -- 172.21.15.1:0/2249729159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d18009a70 con 0x7f4d2006d560 2024-03-18T19:06:24.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.237+0000 7f4d07fff700 1 -- 172.21.15.1:0/2249729159 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d10007460 con 0x7f4d2006d560 2024-03-18T19:06:24.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.237+0000 7f4d2581a700 1 -- 172.21.15.1:0/2249729159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d200c8f80 con 0x7f4d2006d560 2024-03-18T19:06:24.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.238+0000 7f4d2581a700 1 -- 172.21.15.1:0/2249729159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d200676a0 con 0x7f4d2006d560 2024-03-18T19:06:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.239+0000 7f4d07fff700 1 -- 172.21.15.1:0/2249729159 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 45) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f4d100075c0 con 0x7f4d2006d560 2024-03-18T19:06:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.239+0000 7f4d07fff700 1 --2- 172.21.15.1:0/2249729159 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4d0c06adb0 0x7f4d0c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.239+0000 7f4d1e7fc700 1 -- 172.21.15.1:0/2249729159 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4d0c06adb0 msgr2=0x7f4d0c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.240+0000 7f4d07fff700 1 -- 172.21.15.1:0/2249729159 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f4d10088ff0 con 0x7f4d2006d560 2024-03-18T19:06:24.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.240+0000 7f4d1e7fc700 1 --2- 172.21.15.1:0/2249729159 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4d0c06adb0 0x7f4d0c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:24.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.243+0000 7f4d07fff700 1 -- 172.21.15.1:0/2249729159 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4d10053e90 con 0x7f4d2006d560 2024-03-18T19:06:24.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.441+0000 7f4d1e7fc700 1 -- 172.21.15.1:0/2249729159 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4d0c06adb0 msgr2=0x7f4d0c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:24.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.441+0000 7f4d1e7fc700 1 --2- 172.21.15.1:0/2249729159 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4d0c06adb0 0x7f4d0c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:24.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.461+0000 7f4d2581a700 1 -- 172.21.15.1:0/2249729159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4d20072860 con 0x7f4d2006d560 2024-03-18T19:06:24.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.461+0000 7f4d07fff700 1 -- 172.21.15.1:0/2249729159 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v104) v1 ==== 80+0+451 (secure 0 0 0) 0x7f4d10017080 con 0x7f4d2006d560 2024-03-18T19:06:24.457 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1060785910 2024-03-19T19:06:20.400777+0000 2024-03-18T19:06:24.458 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3060583633 2024-03-19T19:06:20.400777+0000 2024-03-18T19:06:24.458 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6833/76998 2024-03-19T19:06:20.400777+0000 2024-03-18T19:06:24.458 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6832/76998 2024-03-19T19:06:20.400777+0000 2024-03-18T19:06:24.458 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/809308784 2024-03-19T19:06:20.400777+0000 2024-03-18T19:06:24.458 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1836399926 2024-03-19T19:06:20.400777+0000 2024-03-18T19:06:24.458 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3606402343 2024-03-19T19:06:20.400777+0000 2024-03-18T19:06:24.458 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2494839321 2024-03-19T19:06:20.400777+0000 2024-03-18T19:06:24.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.464+0000 7f4d2581a700 1 -- 172.21.15.1:0/2249729159 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4d0c06adb0 msgr2=0x7f4d0c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:24.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.464+0000 7f4d2581a700 1 --2- 172.21.15.1:0/2249729159 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4d0c06adb0 0x7f4d0c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:24.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.464+0000 7f4d2581a700 1 -- 172.21.15.1:0/2249729159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d2006d560 msgr2=0x7f4d200c8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:24.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.464+0000 7f4d2581a700 1 --2- 172.21.15.1:0/2249729159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d2006d560 0x7f4d200c8050 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f4d10001eb0 tx=0x7f4d1000c7b0 comp rx=0 tx=0).stop 2024-03-18T19:06:24.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.464+0000 7f4d2581a700 1 -- 172.21.15.1:0/2249729159 shutdown_connections 2024-03-18T19:06:24.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.464+0000 7f4d2581a700 1 --2- 172.21.15.1:0/2249729159 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4d0c06adb0 0x7f4d0c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:24.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.464+0000 7f4d2581a700 1 --2- 172.21.15.1:0/2249729159 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d2015a240 0x7f4d200aea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:24.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.464+0000 7f4d2581a700 1 --2- 172.21.15.1:0/2249729159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d2006d560 0x7f4d200c8050 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:24.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.464+0000 7f4d2581a700 1 --2- 172.21.15.1:0/2249729159 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d20071cc0 0x7f4d200c4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:24.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.464+0000 7f4d2581a700 1 -- 172.21.15.1:0/2249729159 >> 172.21.15.1:0/2249729159 conn(0x7f4d200cff30 msgr2=0x7f4d2006da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:24.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.464+0000 7f4d2581a700 1 -- 172.21.15.1:0/2249729159 shutdown_connections 2024-03-18T19:06:24.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.464+0000 7f4d2581a700 1 -- 172.21.15.1:0/2249729159 wait complete. 2024-03-18T19:06:24.461 INFO:teuthology.orchestra.run.smithi001.stderr:listed 8 entries 2024-03-18T19:06:24.472 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-18T19:06:24.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:24.472 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T19:06:24.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[116850]: fuse finished with error 0 and tester_r 0 2024-03-18T19:06:24.550 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T19:06:24.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:24.550 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T19:06:24.569 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:24.574+0000 7fc75dbc3240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:06:24.622 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:06:24.622 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:06:24.640 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:06:24.640 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T19:06:24.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:24.640 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-18T19:06:24.645 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:24.650+0000 7fc75dbc3240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:06:24.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:06:24.662 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-18T19:06:24.662 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:06:24.662 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:06:24.680 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:06:24.681 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T19:06:24.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:24.681 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-18T19:06:24.735 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:24.740+0000 7f58d2245240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:06:24.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:06:24.739 INFO:teuthology.orchestra.run.smithi001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-18T19:06:24.739 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:06:24.739 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:06:24.758 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:06:24.758 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T19:06:24.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:24.758 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-18T19:06:24.781 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:24.786+0000 7fc75dbc3240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:06:24.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:06:24.799 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-18T19:06:24.799 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:06:24.811 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:24.816+0000 7f58d2245240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:06:24.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.930+0000 7f768df48700 1 Processor -- start 2024-03-18T19:06:24.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.930+0000 7f768df48700 1 -- start start 2024-03-18T19:06:24.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.931+0000 7f768df48700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f768809a150 0x7f7688166070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.931+0000 7f768df48700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768806f070 con 0x7f768809a650 2024-03-18T19:06:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.931+0000 7f768df48700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76880d0260 con 0x7f768815b4d0 2024-03-18T19:06:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.931+0000 7f768df48700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76880cff80 con 0x7f768809a150 2024-03-18T19:06:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.931+0000 7f7686ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f768809a650 0x7f7688099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52352/0 (socket says 172.21.15.1:52352) 2024-03-18T19:06:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.931+0000 7f7686ffd700 1 -- 172.21.15.1:0/4140918169 learned_addr learned my addr 172.21.15.1:0/4140918169 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.931+0000 7f7687fff700 1 --2- 172.21.15.1:0/4140918169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f768809a150 0x7f7688166070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.931+0000 7f76867fc700 1 -- 172.21.15.1:0/4140918169 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 725806491 0 0) 0x7f768806f070 con 0x7f768809a650 2024-03-18T19:06:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.931+0000 7f76867fc700 1 -- 172.21.15.1:0/4140918169 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76740035b0 con 0x7f768809a650 2024-03-18T19:06:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.932+0000 7f76867fc700 1 -- 172.21.15.1:0/4140918169 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432539381 0 0) 0x7f76880d0260 con 0x7f768815b4d0 2024-03-18T19:06:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.932+0000 7f76867fc700 1 -- 172.21.15.1:0/4140918169 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f768806f070 con 0x7f768815b4d0 2024-03-18T19:06:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.932+0000 7f76867fc700 1 -- 172.21.15.1:0/4140918169 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1099073196 0 0) 0x7f76740035b0 con 0x7f768809a650 2024-03-18T19:06:24.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.932+0000 7f7687fff700 1 -- 172.21.15.1:0/4140918169 >> v1:172.21.15.165:6789/0 conn(0x7f768815b4d0 legacy=0x7f76880cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:24.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.932+0000 7f7687fff700 1 -- 172.21.15.1:0/4140918169 >> v1:172.21.15.1:6789/0 conn(0x7f768809a650 legacy=0x7f7688099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:24.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.932+0000 7f7687fff700 1 -- 172.21.15.1:0/4140918169 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768806ea20 con 0x7f768809a150 2024-03-18T19:06:24.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.932+0000 7f7687fff700 1 --2- 172.21.15.1:0/4140918169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f768809a150 0x7f7688166070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7680009870 tx=0x7f7680009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d10e63e9cb4d0692 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:24.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.932+0000 7f76867fc700 1 -- 172.21.15.1:0/4140918169 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7670001670 con 0x7f768809a150 2024-03-18T19:06:24.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.932+0000 7f76867fc700 1 -- 172.21.15.1:0/4140918169 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7680004030 con 0x7f768809a150 2024-03-18T19:06:24.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.932+0000 7f76867fc700 1 -- 172.21.15.1:0/4140918169 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76800043c0 con 0x7f768809a150 2024-03-18T19:06:24.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.933+0000 7f768df48700 1 -- 172.21.15.1:0/4140918169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f768809a150 msgr2=0x7f7688166070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:24.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.933+0000 7f768df48700 1 --2- 172.21.15.1:0/4140918169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f768809a150 0x7f7688166070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7680009870 tx=0x7f7680009b80 comp rx=0 tx=0).stop 2024-03-18T19:06:24.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.933+0000 7f768df48700 1 -- 172.21.15.1:0/4140918169 shutdown_connections 2024-03-18T19:06:24.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.933+0000 7f768df48700 1 --2- 172.21.15.1:0/4140918169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f768809a150 0x7f7688166070 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:24.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.933+0000 7f768df48700 1 -- 172.21.15.1:0/4140918169 >> 172.21.15.1:0/4140918169 conn(0x7f76880b2cb0 msgr2=0x7f76880b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:24.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.933+0000 7f768df48700 1 -- 172.21.15.1:0/4140918169 shutdown_connections 2024-03-18T19:06:24.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.933+0000 7f768df48700 1 -- 172.21.15.1:0/4140918169 wait complete. 2024-03-18T19:06:24.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.934+0000 7f768df48700 1 Processor -- start 2024-03-18T19:06:24.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.934+0000 7f768df48700 1 -- start start 2024-03-18T19:06:24.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.935+0000 7f768df48700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f768809a150 0x7f7688156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:24.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.935+0000 7f768df48700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f768809a650 0x7f76880c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:24.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.935+0000 7f768df48700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f768815b4d0 0x7f76880af480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:24.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.935+0000 7f768df48700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76880c5780 con 0x7f768809a650 2024-03-18T19:06:24.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.935+0000 7f768df48700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76880c4e50 con 0x7f768809a150 2024-03-18T19:06:24.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.935+0000 7f768df48700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76880c5130 con 0x7f768815b4d0 2024-03-18T19:06:24.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.935+0000 7f7686ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f768809a650 0x7f76880c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:24.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.935+0000 7f7686ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f768809a650 0x7f76880c5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53984/0 (socket says 172.21.15.1:53984) 2024-03-18T19:06:24.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.935+0000 7f7686ffd700 1 -- 172.21.15.1:0/532770867 learned_addr learned my addr 172.21.15.1:0/532770867 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:24.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.935+0000 7f7687fff700 1 --2- 172.21.15.1:0/532770867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f768815b4d0 0x7f76880af480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:24.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.935+0000 7f76877fe700 1 --2- 172.21.15.1:0/532770867 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f768809a150 0x7f7688156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:24.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.935+0000 7f7686ffd700 1 -- 172.21.15.1:0/532770867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f768815b4d0 msgr2=0x7f76880af480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:24.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.935+0000 7f7686ffd700 1 --2- 172.21.15.1:0/532770867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f768815b4d0 0x7f76880af480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:24.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.935+0000 7f7686ffd700 1 -- 172.21.15.1:0/532770867 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f768809a150 msgr2=0x7f7688156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:24.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.935+0000 7f7686ffd700 1 --2- 172.21.15.1:0/532770867 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f768809a150 0x7f7688156b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:24.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.935+0000 7f7686ffd700 1 -- 172.21.15.1:0/532770867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76800094a0 con 0x7f768809a650 2024-03-18T19:06:24.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.936+0000 7f7686ffd700 1 --2- 172.21.15.1:0/532770867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f768809a650 0x7f76880c5c10 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f767000c750 tx=0x7f767000ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:24.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.936+0000 7f767ffff700 1 -- 172.21.15.1:0/532770867 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7670004d40 con 0x7f768809a650 2024-03-18T19:06:24.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.936+0000 7f767ffff700 1 -- 172.21.15.1:0/532770867 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76700070d0 con 0x7f768809a650 2024-03-18T19:06:24.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.936+0000 7f768df48700 1 -- 172.21.15.1:0/532770867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76880c5410 con 0x7f768809a650 2024-03-18T19:06:24.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.936+0000 7f767ffff700 1 -- 172.21.15.1:0/532770867 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7670004ea0 con 0x7f768809a650 2024-03-18T19:06:24.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.936+0000 7f768df48700 1 -- 172.21.15.1:0/532770867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76880c6150 con 0x7f768809a650 2024-03-18T19:06:24.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.937+0000 7f768df48700 1 -- 172.21.15.1:0/532770867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7688068240 con 0x7f768809a650 2024-03-18T19:06:24.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.937+0000 7f767ffff700 1 -- 172.21.15.1:0/532770867 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 45) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f7670007460 con 0x7f768809a650 2024-03-18T19:06:24.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.938+0000 7f767ffff700 1 --2- 172.21.15.1:0/532770867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f766806adb0 0x7f766806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:24.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.938+0000 7f76877fe700 1 -- 172.21.15.1:0/532770867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f766806adb0 msgr2=0x7f766806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:24.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.938+0000 7f767ffff700 1 -- 172.21.15.1:0/532770867 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f7670089500 con 0x7f768809a650 2024-03-18T19:06:24.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.938+0000 7f76877fe700 1 --2- 172.21.15.1:0/532770867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f766806adb0 0x7f766806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:24.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:24.942+0000 7f767ffff700 1 -- 172.21.15.1:0/532770867 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7670054330 con 0x7f768809a650 2024-03-18T19:06:24.952 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:24.956+0000 7f58d2245240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:06:24.964 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:24.969+0000 7fc75dbc3240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:06:25.036 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:25.041+0000 7fc75dbc3240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:06:25.114 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:25.119+0000 7fc75dbc3240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:06:25.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.139+0000 7f76877fe700 1 -- 172.21.15.1:0/532770867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f766806adb0 msgr2=0x7f766806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:25.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.139+0000 7f76877fe700 1 --2- 172.21.15.1:0/532770867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f766806adb0 0x7f766806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:25.144 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:25.148+0000 7f58d2245240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:06:25.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.191+0000 7f768df48700 1 -- 172.21.15.1:0/532770867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76880676a0 con 0x7f768809a650 2024-03-18T19:06:25.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.191+0000 7f767ffff700 1 -- 172.21.15.1:0/532770867 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f7670057950 con 0x7f768809a650 2024-03-18T19:06:25.187 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:25.187 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3926877640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3926877640},{"type":"v1","addr":"172.21.15.165:6833","nonce":3926877640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/4293662065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":4293662065},{"type":"v1","addr":"172.21.15.165:6835","nonce":4293662065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2334043797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2334043797},{"type":"v1","addr":"172.21.15.1:6835","nonce":2334043797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:06:05.147601+0000","modified":"2024-03-18T19:06:08.160700+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2526016488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":2526016488},{"type":"v1","addr":"172.21.15.1:6837","nonce":2526016488}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-18T19:06:25.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.193+0000 7f768df48700 1 -- 172.21.15.1:0/532770867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f766806adb0 msgr2=0x7f766806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:25.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.193+0000 7f768df48700 1 --2- 172.21.15.1:0/532770867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f766806adb0 0x7f766806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.193+0000 7f768df48700 1 -- 172.21.15.1:0/532770867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f768809a650 msgr2=0x7f76880c5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:25.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.193+0000 7f768df48700 1 --2- 172.21.15.1:0/532770867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f768809a650 0x7f76880c5c10 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f767000c750 tx=0x7f767000ca60 comp rx=0 tx=0).stop 2024-03-18T19:06:25.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.194+0000 7f768df48700 1 -- 172.21.15.1:0/532770867 shutdown_connections 2024-03-18T19:06:25.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.194+0000 7f768df48700 1 --2- 172.21.15.1:0/532770867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f766806adb0 0x7f766806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.194+0000 7f768df48700 1 --2- 172.21.15.1:0/532770867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f768815b4d0 0x7f76880af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.194+0000 7f768df48700 1 --2- 172.21.15.1:0/532770867 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f768809a150 0x7f7688156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.194+0000 7f768df48700 1 --2- 172.21.15.1:0/532770867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f768809a650 0x7f76880c5c10 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.194+0000 7f768df48700 1 -- 172.21.15.1:0/532770867 >> 172.21.15.1:0/532770867 conn(0x7f76880b2cb0 msgr2=0x7f76880cece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:25.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.194+0000 7f768df48700 1 -- 172.21.15.1:0/532770867 shutdown_connections 2024-03-18T19:06:25.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.194+0000 7f768df48700 1 -- 172.21.15.1:0/532770867 wait complete. 2024-03-18T19:06:25.191 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:25.196+0000 7fc75dbc3240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:06:25.191 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 42 2024-03-18T19:06:25.202 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:06:25.222 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:25.226+0000 7f58d2245240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:06:25.268 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:25.272+0000 7fc75dbc3240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:06:25.307 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:25.311+0000 7f58d2245240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:06:25.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.352+0000 7f602de6a700 1 Processor -- start 2024-03-18T19:06:25.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.352+0000 7f602de6a700 1 -- start start 2024-03-18T19:06:25.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.352+0000 7f602de6a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f602809a650 0x7f6028099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:25.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.352+0000 7f602de6a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6028099f80 0x7f602806f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:25.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.352+0000 7f602de6a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602806f070 con 0x7f602815b4d0 2024-03-18T19:06:25.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.353+0000 7f602de6a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f602809a3f0 con 0x7f6028099f80 2024-03-18T19:06:25.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.353+0000 7f602de6a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60280d0260 con 0x7f602809a650 2024-03-18T19:06:25.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.353+0000 7f6027fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6028099f80 0x7f602806f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:25.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.353+0000 7f6027fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6028099f80 0x7f602806f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53990/0 (socket says 172.21.15.1:53990) 2024-03-18T19:06:25.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.353+0000 7f6027fff700 1 -- 172.21.15.1:0/3268946933 learned_addr learned my addr 172.21.15.1:0/3268946933 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:25.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.353+0000 7f6026ffd700 1 --2- 172.21.15.1:0/3268946933 >> v2:172.21.15.165:3300/0 conn(0x7f602809a650 0x7f6028099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:25.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.353+0000 7f6027fff700 1 -- 172.21.15.1:0/3268946933 >> v2:172.21.15.165:3300/0 conn(0x7f602809a650 msgr2=0x7f6028099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:25.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.353+0000 7f6027fff700 1 --2- 172.21.15.1:0/3268946933 >> v2:172.21.15.165:3300/0 conn(0x7f602809a650 0x7f6028099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.353+0000 7f6027fff700 1 -- 172.21.15.1:0/3268946933 >> v1:172.21.15.165:6789/0 conn(0x7f602815b4d0 legacy=0x7f60280cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:25.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.353+0000 7f6027fff700 1 -- 172.21.15.1:0/3268946933 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60280cff80 con 0x7f6028099f80 2024-03-18T19:06:25.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.353+0000 7f6026ffd700 1 --2- 172.21.15.1:0/3268946933 >> v2:172.21.15.165:3300/0 conn(0x7f602809a650 0x7f6028099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:25.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.353+0000 7f6027fff700 1 --2- 172.21.15.1:0/3268946933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6028099f80 0x7f602806f500 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f6028071d40 tx=0x7f6020009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d482985e41bf26c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:25.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.353+0000 7f60267fc700 1 -- 172.21.15.1:0/3268946933 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602000b870 con 0x7f6028099f80 2024-03-18T19:06:25.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.353+0000 7f60267fc700 1 -- 172.21.15.1:0/3268946933 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602000b9d0 con 0x7f6028099f80 2024-03-18T19:06:25.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.353+0000 7f60267fc700 1 -- 172.21.15.1:0/3268946933 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602000bd60 con 0x7f6028099f80 2024-03-18T19:06:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.354+0000 7f602de6a700 1 -- 172.21.15.1:0/3268946933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6028099f80 msgr2=0x7f602806f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.354+0000 7f602de6a700 1 --2- 172.21.15.1:0/3268946933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6028099f80 0x7f602806f500 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f6028071d40 tx=0x7f6020009bd0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.354+0000 7f602de6a700 1 -- 172.21.15.1:0/3268946933 shutdown_connections 2024-03-18T19:06:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.354+0000 7f602de6a700 1 --2- 172.21.15.1:0/3268946933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6028099f80 0x7f602806f500 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.354+0000 7f602de6a700 1 --2- 172.21.15.1:0/3268946933 >> v2:172.21.15.165:3300/0 conn(0x7f602809a650 0x7f6028099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.354+0000 7f602de6a700 1 -- 172.21.15.1:0/3268946933 >> 172.21.15.1:0/3268946933 conn(0x7f60280b2cb0 msgr2=0x7f60280b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.354+0000 7f602de6a700 1 -- 172.21.15.1:0/3268946933 shutdown_connections 2024-03-18T19:06:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.354+0000 7f602de6a700 1 -- 172.21.15.1:0/3268946933 wait complete. 2024-03-18T19:06:25.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.356+0000 7f602de6a700 1 Processor -- start 2024-03-18T19:06:25.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.356+0000 7f602de6a700 1 -- start start 2024-03-18T19:06:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.357+0000 7f602de6a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6028099f80 0x7f60280c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.357+0000 7f602de6a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f602809a650 0x7f60281f0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.358+0000 7f602de6a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f602815b4d0 0x7f60281481a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.358+0000 7f602de6a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602814abe0 con 0x7f602815b4d0 2024-03-18T19:06:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.358+0000 7f602de6a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60280c99e0 con 0x7f6028099f80 2024-03-18T19:06:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.358+0000 7f602de6a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60280c9cc0 con 0x7f602809a650 2024-03-18T19:06:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.358+0000 7f6027fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f602815b4d0 0x7f60281481a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.358+0000 7f6027fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f602815b4d0 0x7f60281481a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54006/0 (socket says 172.21.15.1:54006) 2024-03-18T19:06:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.358+0000 7f6027fff700 1 -- 172.21.15.1:0/2703023582 learned_addr learned my addr 172.21.15.1:0/2703023582 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.358+0000 7f60277fe700 1 --2- 172.21.15.1:0/2703023582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6028099f80 0x7f60280c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.358+0000 7f6026ffd700 1 --2- 172.21.15.1:0/2703023582 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f602809a650 0x7f60281f0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.358+0000 7f6027fff700 1 -- 172.21.15.1:0/2703023582 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f602809a650 msgr2=0x7f60281f0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.358+0000 7f6027fff700 1 --2- 172.21.15.1:0/2703023582 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f602809a650 0x7f60281f0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.358+0000 7f6027fff700 1 -- 172.21.15.1:0/2703023582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6028099f80 msgr2=0x7f60280c92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:25.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.358+0000 7f6027fff700 1 --2- 172.21.15.1:0/2703023582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6028099f80 0x7f60280c92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.358+0000 7f6027fff700 1 -- 172.21.15.1:0/2703023582 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6020009550 con 0x7f602815b4d0 2024-03-18T19:06:25.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.358+0000 7f6027fff700 1 --2- 172.21.15.1:0/2703023582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f602815b4d0 0x7f60281481a0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f6020009c10 tx=0x7f6020004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:25.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.358+0000 7f601ffff700 1 -- 172.21.15.1:0/2703023582 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6020004480 con 0x7f602815b4d0 2024-03-18T19:06:25.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.358+0000 7f601ffff700 1 -- 172.21.15.1:0/2703023582 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60200045e0 con 0x7f602815b4d0 2024-03-18T19:06:25.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.358+0000 7f601ffff700 1 -- 172.21.15.1:0/2703023582 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60200038a0 con 0x7f602815b4d0 2024-03-18T19:06:25.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.358+0000 7f602de6a700 1 -- 172.21.15.1:0/2703023582 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60280c9fa0 con 0x7f602815b4d0 2024-03-18T19:06:25.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.359+0000 7f602de6a700 1 -- 172.21.15.1:0/2703023582 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60280ca560 con 0x7f602815b4d0 2024-03-18T19:06:25.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.360+0000 7f602de6a700 1 -- 172.21.15.1:0/2703023582 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6028068240 con 0x7f602815b4d0 2024-03-18T19:06:25.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.361+0000 7f601ffff700 1 -- 172.21.15.1:0/2703023582 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 45) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f6020021070 con 0x7f602815b4d0 2024-03-18T19:06:25.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.361+0000 7f601ffff700 1 --2- 172.21.15.1:0/2703023582 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f600806adb0 0x7f600806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:25.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.361+0000 7f60277fe700 1 -- 172.21.15.1:0/2703023582 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f600806adb0 msgr2=0x7f600806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:25.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.361+0000 7f601ffff700 1 -- 172.21.15.1:0/2703023582 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f602008ae60 con 0x7f602815b4d0 2024-03-18T19:06:25.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.361+0000 7f60277fe700 1 --2- 172.21.15.1:0/2703023582 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f600806adb0 0x7f600806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:25.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.365+0000 7f601ffff700 1 -- 172.21.15.1:0/2703023582 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6020055d20 con 0x7f602815b4d0 2024-03-18T19:06:25.393 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:25.397+0000 7f58d2245240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:06:25.440 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:25.445+0000 7fc75dbc3240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:06:25.476 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:25.481+0000 7f58d2245240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:06:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.562+0000 7f60277fe700 1 -- 172.21.15.1:0/2703023582 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f600806adb0 msgr2=0x7f600806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:25.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.562+0000 7f60277fe700 1 --2- 172.21.15.1:0/2703023582 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f600806adb0 0x7f600806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:25.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.570+0000 7f602de6a700 1 -- 172.21.15.1:0/2703023582 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60280676a0 con 0x7f602815b4d0 2024-03-18T19:06:25.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.571+0000 7f601ffff700 1 -- 172.21.15.1:0/2703023582 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+16754 (secure 0 0 0) 0x7f6020015020 con 0x7f602815b4d0 2024-03-18T19:06:25.567 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:25.567 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":104,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:06:20.400812+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:06:02.855989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-18T19:06:04.121199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/1060785910":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/3060583633":"2024-03-19T19:06:20.400777+0000","172.21.15.1:6833/76998":"2024-03-19T19:06:20.400777+0000","172.21.15.1:6832/76998":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/809308784":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/1836399926":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/3606402343":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/2494839321":"2024-03-19T19:06:20.400777+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:06:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.573+0000 7f602de6a700 1 -- 172.21.15.1:0/2703023582 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f600806adb0 msgr2=0x7f600806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.573+0000 7f602de6a700 1 --2- 172.21.15.1:0/2703023582 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f600806adb0 0x7f600806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.573+0000 7f602de6a700 1 -- 172.21.15.1:0/2703023582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f602815b4d0 msgr2=0x7f60281481a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.573+0000 7f602de6a700 1 --2- 172.21.15.1:0/2703023582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f602815b4d0 0x7f60281481a0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f6020009c10 tx=0x7f6020004750 comp rx=0 tx=0).stop 2024-03-18T19:06:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.573+0000 7f602de6a700 1 -- 172.21.15.1:0/2703023582 shutdown_connections 2024-03-18T19:06:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.573+0000 7f602de6a700 1 --2- 172.21.15.1:0/2703023582 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f600806adb0 0x7f600806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.573+0000 7f602de6a700 1 --2- 172.21.15.1:0/2703023582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f602815b4d0 0x7f60281481a0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.573+0000 7f602de6a700 1 --2- 172.21.15.1:0/2703023582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6028099f80 0x7f60280c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.573+0000 7f602de6a700 1 --2- 172.21.15.1:0/2703023582 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f602809a650 0x7f60281f0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.573+0000 7f602de6a700 1 -- 172.21.15.1:0/2703023582 >> 172.21.15.1:0/2703023582 conn(0x7f60280b2cb0 msgr2=0x7f60280b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.573+0000 7f602de6a700 1 -- 172.21.15.1:0/2703023582 shutdown_connections 2024-03-18T19:06:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.573+0000 7f602de6a700 1 -- 172.21.15.1:0/2703023582 wait complete. 2024-03-18T19:06:25.582 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T19:06:02.855989+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T19:06:04.121199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:06:25.582 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:06:25.654 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:25.659+0000 7fc75dbc3240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:06:25.659 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:25.663+0000 7f58d2245240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:06:25.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.715+0000 7f408796e700 1 Processor -- start 2024-03-18T19:06:25.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.715+0000 7f408796e700 1 -- start start 2024-03-18T19:06:25.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.716+0000 7f408796e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f40800cd870 0x7f40800cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:25.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.716+0000 7f408796e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f40800b1e20 0x7f40800cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.716+0000 7f408796e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40800d0df0 con 0x7f408006bc20 2024-03-18T19:06:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.716+0000 7f408796e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40800ce190 con 0x7f40800cd870 2024-03-18T19:06:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.716+0000 7f408796e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40800d0550 con 0x7f40800b1e20 2024-03-18T19:06:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.717+0000 7f408570a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f40800cd870 0x7f40800cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.717+0000 7f4085f0b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f408006bc20 0x7f408006c070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52368/0 (socket says 172.21.15.1:52368) 2024-03-18T19:06:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.717+0000 7f4085f0b700 1 -- 172.21.15.1:0/4250560703 learned_addr learned my addr 172.21.15.1:0/4250560703 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.717+0000 7f4084f09700 1 --2- 172.21.15.1:0/4250560703 >> v2:172.21.15.165:3300/0 conn(0x7f40800b1e20 0x7f40800cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.717+0000 7f408570a700 1 -- 172.21.15.1:0/4250560703 >> v2:172.21.15.165:3300/0 conn(0x7f40800b1e20 msgr2=0x7f40800cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.717+0000 7f408570a700 1 --2- 172.21.15.1:0/4250560703 >> v2:172.21.15.165:3300/0 conn(0x7f40800b1e20 0x7f40800cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.717+0000 7f408570a700 1 -- 172.21.15.1:0/4250560703 >> v1:172.21.15.1:6789/0 conn(0x7f408006bc20 legacy=0x7f408006c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.717+0000 7f4077fff700 1 -- 172.21.15.1:0/4250560703 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393271295 0 0) 0x7f40800d0df0 con 0x7f408006bc20 2024-03-18T19:06:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.717+0000 7f408570a700 1 -- 172.21.15.1:0/4250560703 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40800d0830 con 0x7f40800cd870 2024-03-18T19:06:25.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.717+0000 7f408570a700 1 --2- 172.21.15.1:0/4250560703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40800cd870 0x7f40800cff70 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f40800715c0 tx=0x7f407c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39244bd4974c66d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:25.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.717+0000 7f4077fff700 1 -- 172.21.15.1:0/4250560703 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c011830 con 0x7f40800cd870 2024-03-18T19:06:25.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.717+0000 7f4077fff700 1 -- 172.21.15.1:0/4250560703 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407c011990 con 0x7f40800cd870 2024-03-18T19:06:25.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.717+0000 7f4077fff700 1 -- 172.21.15.1:0/4250560703 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c011d20 con 0x7f40800cd870 2024-03-18T19:06:25.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.718+0000 7f408796e700 1 -- 172.21.15.1:0/4250560703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40800cd870 msgr2=0x7f40800cff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:25.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.718+0000 7f408796e700 1 --2- 172.21.15.1:0/4250560703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40800cd870 0x7f40800cff70 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f40800715c0 tx=0x7f407c009b20 comp rx=0 tx=0).stop 2024-03-18T19:06:25.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.718+0000 7f408796e700 1 -- 172.21.15.1:0/4250560703 shutdown_connections 2024-03-18T19:06:25.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.718+0000 7f408796e700 1 --2- 172.21.15.1:0/4250560703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40800cd870 0x7f40800cff70 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.718+0000 7f408796e700 1 --2- 172.21.15.1:0/4250560703 >> v2:172.21.15.165:3300/0 conn(0x7f40800b1e20 0x7f40800cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.718+0000 7f408796e700 1 -- 172.21.15.1:0/4250560703 >> 172.21.15.1:0/4250560703 conn(0x7f40800d0ff0 msgr2=0x7f40800d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:25.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.718+0000 7f408796e700 1 -- 172.21.15.1:0/4250560703 shutdown_connections 2024-03-18T19:06:25.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.719+0000 7f408796e700 1 -- 172.21.15.1:0/4250560703 wait complete. 2024-03-18T19:06:25.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.720+0000 7f408796e700 1 Processor -- start 2024-03-18T19:06:25.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.720+0000 7f408796e700 1 -- start start 2024-03-18T19:06:25.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.720+0000 7f408796e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f408006bc20 0x7f408009e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.720+0000 7f408796e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40800b1e20 0x7f408009adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.720+0000 7f408796e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40800cd870 0x7f40800caa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.720+0000 7f408796e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408009c200 con 0x7f408006bc20 2024-03-18T19:06:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.720+0000 7f408796e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408009b5f0 con 0x7f40800cd870 2024-03-18T19:06:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.720+0000 7f408796e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f408009b8d0 con 0x7f40800b1e20 2024-03-18T19:06:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.721+0000 7f408570a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f408006bc20 0x7f408009e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.721+0000 7f408570a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f408006bc20 0x7f408009e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54012/0 (socket says 172.21.15.1:54012) 2024-03-18T19:06:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.721+0000 7f408570a700 1 -- 172.21.15.1:0/452313066 learned_addr learned my addr 172.21.15.1:0/452313066 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.721+0000 7f4085f0b700 1 --2- 172.21.15.1:0/452313066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40800cd870 0x7f40800caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:25.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.721+0000 7f4084f09700 1 --2- 172.21.15.1:0/452313066 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40800b1e20 0x7f408009adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:25.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.721+0000 7f408570a700 1 -- 172.21.15.1:0/452313066 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40800b1e20 msgr2=0x7f408009adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:25.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.721+0000 7f408570a700 1 --2- 172.21.15.1:0/452313066 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40800b1e20 0x7f408009adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.721+0000 7f408570a700 1 -- 172.21.15.1:0/452313066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40800cd870 msgr2=0x7f40800caa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:25.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.721+0000 7f408570a700 1 --2- 172.21.15.1:0/452313066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40800cd870 0x7f40800caa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.721+0000 7f408570a700 1 -- 172.21.15.1:0/452313066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407c0094a0 con 0x7f408006bc20 2024-03-18T19:06:25.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.721+0000 7f4085f0b700 1 --2- 172.21.15.1:0/452313066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40800cd870 0x7f40800caa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:06:25.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.721+0000 7f408570a700 1 --2- 172.21.15.1:0/452313066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f408006bc20 0x7f408009e580 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f407c0090b0 tx=0x7f407c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:25.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.721+0000 7f4075ffb700 1 -- 172.21.15.1:0/452313066 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c0036a0 con 0x7f408006bc20 2024-03-18T19:06:25.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.721+0000 7f408796e700 1 -- 172.21.15.1:0/452313066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408009bbb0 con 0x7f408006bc20 2024-03-18T19:06:25.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.721+0000 7f4075ffb700 1 -- 172.21.15.1:0/452313066 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407c01f070 con 0x7f408006bc20 2024-03-18T19:06:25.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.721+0000 7f4075ffb700 1 -- 172.21.15.1:0/452313066 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c003a20 con 0x7f408006bc20 2024-03-18T19:06:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.722+0000 7f408796e700 1 -- 172.21.15.1:0/452313066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408009b300 con 0x7f408006bc20 2024-03-18T19:06:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.723+0000 7f408796e700 1 -- 172.21.15.1:0/452313066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40800b7190 con 0x7f408006bc20 2024-03-18T19:06:25.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.723+0000 7f4075ffb700 1 -- 172.21.15.1:0/452313066 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 45) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f407c003b80 con 0x7f408006bc20 2024-03-18T19:06:25.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.724+0000 7f4075ffb700 1 --2- 172.21.15.1:0/452313066 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f406406adb0 0x7f406406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:25.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.724+0000 7f4084f09700 1 -- 172.21.15.1:0/452313066 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f406406adb0 msgr2=0x7f406406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:25.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.724+0000 7f4075ffb700 1 -- 172.21.15.1:0/452313066 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f407c021070 con 0x7f408006bc20 2024-03-18T19:06:25.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.724+0000 7f4084f09700 1 --2- 172.21.15.1:0/452313066 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f406406adb0 0x7f406406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:25.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.728+0000 7f4075ffb700 1 -- 172.21.15.1:0/452313066 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f407c055d30 con 0x7f408006bc20 2024-03-18T19:06:25.804 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:25.808+0000 7fc75dbc3240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:06:25.876 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:25.881+0000 7fc75dbc3240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:06:25.885 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:25.890+0000 7f58d2245240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:06:25.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.925+0000 7f4084f09700 1 -- 172.21.15.1:0/452313066 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f406406adb0 msgr2=0x7f406406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:25.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.925+0000 7f4084f09700 1 --2- 172.21.15.1:0/452313066 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f406406adb0 0x7f406406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:25.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.931+0000 7f408796e700 1 -- 172.21.15.1:0/452313066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f40800676a0 con 0x7f408006bc20 2024-03-18T19:06:25.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.932+0000 7f4075ffb700 1 -- 172.21.15.1:0/452313066 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7f407c02b0a0 con 0x7f408006bc20 2024-03-18T19:06:25.928 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:06:25.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.934+0000 7f408796e700 1 -- 172.21.15.1:0/452313066 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f406406adb0 msgr2=0x7f406406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:25.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.934+0000 7f408796e700 1 --2- 172.21.15.1:0/452313066 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f406406adb0 0x7f406406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.934+0000 7f408796e700 1 -- 172.21.15.1:0/452313066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f408006bc20 msgr2=0x7f408009e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:25.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.934+0000 7f408796e700 1 --2- 172.21.15.1:0/452313066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f408006bc20 0x7f408009e580 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f407c0090b0 tx=0x7f407c004750 comp rx=0 tx=0).stop 2024-03-18T19:06:25.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.934+0000 7f408796e700 1 -- 172.21.15.1:0/452313066 shutdown_connections 2024-03-18T19:06:25.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.934+0000 7f408796e700 1 --2- 172.21.15.1:0/452313066 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f406406adb0 0x7f406406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.934+0000 7f408796e700 1 --2- 172.21.15.1:0/452313066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40800cd870 0x7f40800caa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.934+0000 7f408796e700 1 --2- 172.21.15.1:0/452313066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f408006bc20 0x7f408009e580 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.934+0000 7f408796e700 1 --2- 172.21.15.1:0/452313066 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40800b1e20 0x7f408009adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:25.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.934+0000 7f408796e700 1 -- 172.21.15.1:0/452313066 >> 172.21.15.1:0/452313066 conn(0x7f40800d0ff0 msgr2=0x7f40800c80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:25.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.934+0000 7f408796e700 1 -- 172.21.15.1:0/452313066 shutdown_connections 2024-03-18T19:06:25.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:25.934+0000 7f408796e700 1 -- 172.21.15.1:0/452313066 wait complete. 2024-03-18T19:06:25.943 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T19:06:26.054 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:26.058+0000 7f58d2245240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:06:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.076+0000 7f51ed9a9700 1 Processor -- start 2024-03-18T19:06:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.076+0000 7f51ed9a9700 1 -- start start 2024-03-18T19:06:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.077+0000 7f51ed9a9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51e815d580 0x7f51e806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.077+0000 7f51ed9a9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51e806d380 0x7f51e8071da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.077+0000 7f51ed9a9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f51e806d7e0 0x7f51e80722e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.077+0000 7f51ed9a9700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e815d960 con 0x7f51e815d580 2024-03-18T19:06:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.077+0000 7f51ed9a9700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e806f4f0 con 0x7f51e806d7e0 2024-03-18T19:06:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.077+0000 7f51ed9a9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e806ab30 con 0x7f51e806d380 2024-03-18T19:06:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.077+0000 7f51e6ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51e815d580 0x7f51e806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.077+0000 7f51e6ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51e815d580 0x7f51e806efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54018/0 (socket says 172.21.15.1:54018) 2024-03-18T19:06:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.077+0000 7f51e6ffd700 1 -- 172.21.15.1:0/60396977 learned_addr learned my addr 172.21.15.1:0/60396977 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:26.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.078+0000 7f51e77fe700 1 --2- 172.21.15.1:0/60396977 >> v2:172.21.15.165:3300/0 conn(0x7f51e806d7e0 0x7f51e80722e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.078+0000 7f51e67fc700 1 --2- 172.21.15.1:0/60396977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51e806d380 0x7f51e8071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.078+0000 7f51e6ffd700 1 -- 172.21.15.1:0/60396977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51e806d380 msgr2=0x7f51e8071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.078+0000 7f51e6ffd700 1 --2- 172.21.15.1:0/60396977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51e806d380 0x7f51e8071da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.078+0000 7f51e6ffd700 1 -- 172.21.15.1:0/60396977 >> v2:172.21.15.165:3300/0 conn(0x7f51e806d7e0 msgr2=0x7f51e80722e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.078+0000 7f51e6ffd700 1 --2- 172.21.15.1:0/60396977 >> v2:172.21.15.165:3300/0 conn(0x7f51e806d7e0 0x7f51e80722e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.078+0000 7f51e6ffd700 1 -- 172.21.15.1:0/60396977 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e806ae10 con 0x7f51e815d580 2024-03-18T19:06:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.078+0000 7f51e6ffd700 1 --2- 172.21.15.1:0/60396977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e815d580 0x7f51e806efb0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f51e8062510 tx=0x7f51d8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc10c62d9270c6c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.078+0000 7f51e5ffb700 1 -- 172.21.15.1:0/60396977 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d8011830 con 0x7f51e815d580 2024-03-18T19:06:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.078+0000 7f51e5ffb700 1 -- 172.21.15.1:0/60396977 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d8011990 con 0x7f51e815d580 2024-03-18T19:06:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.078+0000 7f51e5ffb700 1 -- 172.21.15.1:0/60396977 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d8011d20 con 0x7f51e815d580 2024-03-18T19:06:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.079+0000 7f51ed9a9700 1 -- 172.21.15.1:0/60396977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e815d580 msgr2=0x7f51e806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.079+0000 7f51ed9a9700 1 --2- 172.21.15.1:0/60396977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e815d580 0x7f51e806efb0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f51e8062510 tx=0x7f51d8009bd0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.079+0000 7f51ed9a9700 1 -- 172.21.15.1:0/60396977 shutdown_connections 2024-03-18T19:06:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.079+0000 7f51ed9a9700 1 --2- 172.21.15.1:0/60396977 >> v2:172.21.15.165:3300/0 conn(0x7f51e806d7e0 0x7f51e80722e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.079+0000 7f51ed9a9700 1 --2- 172.21.15.1:0/60396977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e815d580 0x7f51e806efb0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.079+0000 7f51ed9a9700 1 --2- 172.21.15.1:0/60396977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51e806d380 0x7f51e8071da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.079+0000 7f51ed9a9700 1 -- 172.21.15.1:0/60396977 >> 172.21.15.1:0/60396977 conn(0x7f51e80cca00 msgr2=0x7f51e80cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.079+0000 7f51ed9a9700 1 -- 172.21.15.1:0/60396977 shutdown_connections 2024-03-18T19:06:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.079+0000 7f51ed9a9700 1 -- 172.21.15.1:0/60396977 wait complete. 2024-03-18T19:06:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.080+0000 7f51ed9a9700 1 Processor -- start 2024-03-18T19:06:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.080+0000 7f51ed9a9700 1 -- start start 2024-03-18T19:06:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.081+0000 7f51ed9a9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51e806d380 0x7f51e81577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.081+0000 7f51ed9a9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e806d7e0 0x7f51e80b4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.081+0000 7f51ed9a9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51e815d580 0x7f51e80ae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.081+0000 7f51ed9a9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e8157cf0 con 0x7f51e806d7e0 2024-03-18T19:06:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.081+0000 7f51ed9a9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e80b4af0 con 0x7f51e806d380 2024-03-18T19:06:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.081+0000 7f51ed9a9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e80b4dd0 con 0x7f51e815d580 2024-03-18T19:06:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.081+0000 7f51e67fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e806d7e0 0x7f51e80b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.081+0000 7f51e67fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e806d7e0 0x7f51e80b4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54022/0 (socket says 172.21.15.1:54022) 2024-03-18T19:06:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.081+0000 7f51e6ffd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51e806d380 0x7f51e81577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.081+0000 7f51e67fc700 1 -- 172.21.15.1:0/3242884334 learned_addr learned my addr 172.21.15.1:0/3242884334 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.081+0000 7f51e77fe700 1 --2- 172.21.15.1:0/3242884334 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51e815d580 0x7f51e80ae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.081+0000 7f51e67fc700 1 -- 172.21.15.1:0/3242884334 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51e815d580 msgr2=0x7f51e80ae3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.081+0000 7f51e67fc700 1 --2- 172.21.15.1:0/3242884334 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51e815d580 0x7f51e80ae3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.081+0000 7f51e67fc700 1 -- 172.21.15.1:0/3242884334 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51e806d380 msgr2=0x7f51e81577b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.081+0000 7f51e67fc700 1 --2- 172.21.15.1:0/3242884334 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51e806d380 0x7f51e81577b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.081+0000 7f51e67fc700 1 -- 172.21.15.1:0/3242884334 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d8009550 con 0x7f51e806d7e0 2024-03-18T19:06:26.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.082+0000 7f51e67fc700 1 --2- 172.21.15.1:0/3242884334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e806d7e0 0x7f51e80b4110 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00e870 tx=0x7f51dc00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:26.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.082+0000 7f51cffff700 1 -- 172.21.15.1:0/3242884334 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc013680 con 0x7f51e806d7e0 2024-03-18T19:06:26.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.082+0000 7f51cffff700 1 -- 172.21.15.1:0/3242884334 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc0137e0 con 0x7f51e806d7e0 2024-03-18T19:06:26.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.082+0000 7f51cffff700 1 -- 172.21.15.1:0/3242884334 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc013b90 con 0x7f51e806d7e0 2024-03-18T19:06:26.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.082+0000 7f51ed9a9700 1 -- 172.21.15.1:0/3242884334 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e80b50b0 con 0x7f51e806d7e0 2024-03-18T19:06:26.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.082+0000 7f51ed9a9700 1 -- 172.21.15.1:0/3242884334 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e80b5670 con 0x7f51e806d7e0 2024-03-18T19:06:26.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.083+0000 7f51ed9a9700 1 -- 172.21.15.1:0/3242884334 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e80676a0 con 0x7f51e806d7e0 2024-03-18T19:06:26.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.083+0000 7f51cffff700 1 -- 172.21.15.1:0/3242884334 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 45) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f51dc013cf0 con 0x7f51e806d7e0 2024-03-18T19:06:26.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.084+0000 7f51cffff700 1 --2- 172.21.15.1:0/3242884334 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f51d406adb0 0x7f51d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:26.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.084+0000 7f51e6ffd700 1 -- 172.21.15.1:0/3242884334 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f51d406adb0 msgr2=0x7f51d406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:26.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.084+0000 7f51cffff700 1 -- 172.21.15.1:0/3242884334 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f51dc089570 con 0x7f51e806d7e0 2024-03-18T19:06:26.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.084+0000 7f51e6ffd700 1 --2- 172.21.15.1:0/3242884334 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f51d406adb0 0x7f51d406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:26.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.088+0000 7f51cffff700 1 -- 172.21.15.1:0/3242884334 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f51dc0543a0 con 0x7f51e806d7e0 2024-03-18T19:06:26.134 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:26.139+0000 7f58d2245240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:06:26.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.285+0000 7f51e6ffd700 1 -- 172.21.15.1:0/3242884334 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f51d406adb0 msgr2=0x7f51d406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:26.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.285+0000 7f51e6ffd700 1 --2- 172.21.15.1:0/3242884334 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f51d406adb0 0x7f51d406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:26.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.291+0000 7f51ed9a9700 1 -- 172.21.15.1:0/3242884334 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f51e806b3c0 con 0x7f51e806d7e0 2024-03-18T19:06:26.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.291+0000 7f51cffff700 1 -- 172.21.15.1:0/3242884334 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v104) v1 ==== 104+0+11 (secure 0 0 0) 0x7f51dc0579c0 con 0x7f51e806d7e0 2024-03-18T19:06:26.287 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T19:06:26.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.293+0000 7f51ed9a9700 1 -- 172.21.15.1:0/3242884334 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f51d406adb0 msgr2=0x7f51d406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:26.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.293+0000 7f51ed9a9700 1 --2- 172.21.15.1:0/3242884334 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f51d406adb0 0x7f51d406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.293+0000 7f51ed9a9700 1 -- 172.21.15.1:0/3242884334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e806d7e0 msgr2=0x7f51e80b4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:26.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.293+0000 7f51ed9a9700 1 --2- 172.21.15.1:0/3242884334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e806d7e0 0x7f51e80b4110 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00e870 tx=0x7f51dc00eb80 comp rx=0 tx=0).stop 2024-03-18T19:06:26.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.293+0000 7f51ed9a9700 1 -- 172.21.15.1:0/3242884334 shutdown_connections 2024-03-18T19:06:26.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.293+0000 7f51ed9a9700 1 --2- 172.21.15.1:0/3242884334 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f51d406adb0 0x7f51d406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.293+0000 7f51ed9a9700 1 --2- 172.21.15.1:0/3242884334 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51e815d580 0x7f51e80ae3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.293+0000 7f51ed9a9700 1 --2- 172.21.15.1:0/3242884334 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51e806d380 0x7f51e81577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.293+0000 7f51ed9a9700 1 --2- 172.21.15.1:0/3242884334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51e806d7e0 0x7f51e80b4110 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.293+0000 7f51ed9a9700 1 -- 172.21.15.1:0/3242884334 >> 172.21.15.1:0/3242884334 conn(0x7f51e80cca00 msgr2=0x7f51e80adc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:26.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.293+0000 7f51ed9a9700 1 -- 172.21.15.1:0/3242884334 shutdown_connections 2024-03-18T19:06:26.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.293+0000 7f51ed9a9700 1 -- 172.21.15.1:0/3242884334 wait complete. 2024-03-18T19:06:26.302 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T19:06:26.320 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:26.325+0000 7fc75dbc3240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:06:26.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.435+0000 7ff216be4700 1 Processor -- start 2024-03-18T19:06:26.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.435+0000 7ff216be4700 1 -- start start 2024-03-18T19:06:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.436+0000 7ff216be4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff210071cc0 0x7ff2100ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.436+0000 7ff216be4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff21006d560 0x7ff21006fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.436+0000 7ff216be4700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff210070000 con 0x7ff21015a240 2024-03-18T19:06:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.436+0000 7ff216be4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2100adff0 con 0x7ff210071cc0 2024-03-18T19:06:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.436+0000 7ff216be4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2100cac80 con 0x7ff21006d560 2024-03-18T19:06:26.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.436+0000 7ff20ffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff210071cc0 0x7ff2100ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.436+0000 7ff214980700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff21015a240 0x7ff2100715b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52382/0 (socket says 172.21.15.1:52382) 2024-03-18T19:06:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.436+0000 7ff20ffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff210071cc0 0x7ff2100ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54038/0 (socket says 172.21.15.1:54038) 2024-03-18T19:06:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.436+0000 7ff214980700 1 -- 172.21.15.1:0/2424568486 learned_addr learned my addr 172.21.15.1:0/2424568486 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.436+0000 7ff215181700 1 --2- 172.21.15.1:0/2424568486 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff21006d560 0x7ff21006fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.436+0000 7ff20ffff700 1 -- 172.21.15.1:0/2424568486 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff21006d560 msgr2=0x7ff21006fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.436+0000 7ff20ffff700 1 --2- 172.21.15.1:0/2424568486 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff21006d560 0x7ff21006fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.436+0000 7ff20ffff700 1 -- 172.21.15.1:0/2424568486 >> v1:172.21.15.1:6789/0 conn(0x7ff21015a240 legacy=0x7ff2100715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.436+0000 7ff20ffff700 1 -- 172.21.15.1:0/2424568486 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2100caf60 con 0x7ff210071cc0 2024-03-18T19:06:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.436+0000 7ff20f7fe700 1 -- 172.21.15.1:0/2424568486 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2161195966 0 0) 0x7ff210070000 con 0x7ff21015a240 2024-03-18T19:06:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.436+0000 7ff20ffff700 1 --2- 172.21.15.1:0/2424568486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff210071cc0 0x7ff2100ca580 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc009840 tx=0x7ff1fc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3914aab2138d1fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:26.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.437+0000 7ff20f7fe700 1 -- 172.21.15.1:0/2424568486 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc011830 con 0x7ff210071cc0 2024-03-18T19:06:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.437+0000 7ff20f7fe700 1 -- 172.21.15.1:0/2424568486 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1fc011990 con 0x7ff210071cc0 2024-03-18T19:06:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.437+0000 7ff20f7fe700 1 -- 172.21.15.1:0/2424568486 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc011d20 con 0x7ff210071cc0 2024-03-18T19:06:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.437+0000 7ff216be4700 1 -- 172.21.15.1:0/2424568486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff210071cc0 msgr2=0x7ff2100ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.437+0000 7ff216be4700 1 --2- 172.21.15.1:0/2424568486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff210071cc0 0x7ff2100ca580 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc009840 tx=0x7ff1fc009b50 comp rx=0 tx=0).stop 2024-03-18T19:06:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.437+0000 7ff216be4700 1 -- 172.21.15.1:0/2424568486 shutdown_connections 2024-03-18T19:06:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.437+0000 7ff216be4700 1 --2- 172.21.15.1:0/2424568486 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff21006d560 0x7ff21006fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.437+0000 7ff216be4700 1 --2- 172.21.15.1:0/2424568486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff210071cc0 0x7ff2100ca580 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.437+0000 7ff216be4700 1 -- 172.21.15.1:0/2424568486 >> 172.21.15.1:0/2424568486 conn(0x7ff2100cff30 msgr2=0x7ff2100d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.437+0000 7ff216be4700 1 -- 172.21.15.1:0/2424568486 shutdown_connections 2024-03-18T19:06:26.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.437+0000 7ff216be4700 1 -- 172.21.15.1:0/2424568486 wait complete. 2024-03-18T19:06:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.439+0000 7ff216be4700 1 Processor -- start 2024-03-18T19:06:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.439+0000 7ff216be4700 1 -- start start 2024-03-18T19:06:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.441+0000 7ff216be4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff21006d560 0x7ff2100b5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.441+0000 7ff216be4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff210071cc0 0x7ff210159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.441+0000 7ff216be4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff21015a240 0x7ff2100b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.441+0000 7ff216be4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff210159580 con 0x7ff21015a240 2024-03-18T19:06:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.441+0000 7ff216be4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2100c8240 con 0x7ff21006d560 2024-03-18T19:06:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.441+0000 7ff216be4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2100c8520 con 0x7ff210071cc0 2024-03-18T19:06:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.441+0000 7ff215181700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff21015a240 0x7ff2100b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.441+0000 7ff215181700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff21015a240 0x7ff2100b36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54052/0 (socket says 172.21.15.1:54052) 2024-03-18T19:06:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.441+0000 7ff215181700 1 -- 172.21.15.1:0/1677480961 learned_addr learned my addr 172.21.15.1:0/1677480961 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:26.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.441+0000 7ff214980700 1 --2- 172.21.15.1:0/1677480961 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff21006d560 0x7ff2100b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.441+0000 7ff20ffff700 1 --2- 172.21.15.1:0/1677480961 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff210071cc0 0x7ff210159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.441+0000 7ff215181700 1 -- 172.21.15.1:0/1677480961 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff210071cc0 msgr2=0x7ff210159040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.441+0000 7ff215181700 1 --2- 172.21.15.1:0/1677480961 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff210071cc0 0x7ff210159040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.441+0000 7ff215181700 1 -- 172.21.15.1:0/1677480961 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff21006d560 msgr2=0x7ff2100b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.441+0000 7ff215181700 1 --2- 172.21.15.1:0/1677480961 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff21006d560 0x7ff2100b5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.441+0000 7ff215181700 1 -- 172.21.15.1:0/1677480961 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc0094a0 con 0x7ff21015a240 2024-03-18T19:06:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.441+0000 7ff215181700 1 --2- 172.21.15.1:0/1677480961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff21015a240 0x7ff2100b36a0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7ff20800efc0 tx=0x7ff20800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.441+0000 7ff20d7fa700 1 -- 172.21.15.1:0/1677480961 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff208009ea0 con 0x7ff21015a240 2024-03-18T19:06:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.442+0000 7ff20d7fa700 1 -- 172.21.15.1:0/1677480961 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20800f040 con 0x7ff21015a240 2024-03-18T19:06:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.442+0000 7ff216be4700 1 -- 172.21.15.1:0/1677480961 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2100c8800 con 0x7ff21015a240 2024-03-18T19:06:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.442+0000 7ff20d7fa700 1 -- 172.21.15.1:0/1677480961 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20800ced0 con 0x7ff21015a240 2024-03-18T19:06:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.442+0000 7ff216be4700 1 -- 172.21.15.1:0/1677480961 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2100b5b30 con 0x7ff21015a240 2024-03-18T19:06:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.442+0000 7ff216be4700 1 -- 172.21.15.1:0/1677480961 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2100c8d20 con 0x7ff21015a240 2024-03-18T19:06:26.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.443+0000 7ff20d7fa700 1 -- 172.21.15.1:0/1677480961 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 45) v1 ==== 89240+0+0 (secure 0 0 0) 0x7ff208014480 con 0x7ff21015a240 2024-03-18T19:06:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.444+0000 7ff20d7fa700 1 --2- 172.21.15.1:0/1677480961 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff1f406adb0 0x7ff1f406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.444+0000 7ff214980700 1 -- 172.21.15.1:0/1677480961 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff1f406adb0 msgr2=0x7ff1f406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.444+0000 7ff20d7fa700 1 -- 172.21.15.1:0/1677480961 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6373+0+0 (secure 0 0 0) 0x7ff20808a0d0 con 0x7ff21015a240 2024-03-18T19:06:26.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.444+0000 7ff214980700 1 --2- 172.21.15.1:0/1677480961 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff1f406adb0 0x7ff1f406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:26.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.448+0000 7ff20d7fa700 1 -- 172.21.15.1:0/1677480961 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff2100c8d20 con 0x7ff21015a240 2024-03-18T19:06:26.619 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:26.623+0000 7f58d2245240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:06:26.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.645+0000 7ff214980700 1 -- 172.21.15.1:0/1677480961 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff1f406adb0 msgr2=0x7ff1f406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:26.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.645+0000 7ff214980700 1 --2- 172.21.15.1:0/1677480961 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff1f406adb0 0x7ff1f406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:26.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.652+0000 7ff216be4700 1 -- 172.21.15.1:0/1677480961 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff2100c8d20 con 0x7ff21015a240 2024-03-18T19:06:26.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.653+0000 7ff20d7fa700 1 -- 172.21.15.1:0/1677480961 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v104) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff2100c8d20 con 0x7ff21015a240 2024-03-18T19:06:26.649 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T19:06:26.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.655+0000 7ff216be4700 1 -- 172.21.15.1:0/1677480961 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff1f406adb0 msgr2=0x7ff1f406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:26.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.655+0000 7ff216be4700 1 --2- 172.21.15.1:0/1677480961 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff1f406adb0 0x7ff1f406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.655+0000 7ff216be4700 1 -- 172.21.15.1:0/1677480961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff21015a240 msgr2=0x7ff2100b36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:26.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.655+0000 7ff216be4700 1 --2- 172.21.15.1:0/1677480961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff21015a240 0x7ff2100b36a0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7ff20800efc0 tx=0x7ff20800c3a0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.655+0000 7ff216be4700 1 -- 172.21.15.1:0/1677480961 shutdown_connections 2024-03-18T19:06:26.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.655+0000 7ff216be4700 1 --2- 172.21.15.1:0/1677480961 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff1f406adb0 0x7ff1f406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.655+0000 7ff216be4700 1 --2- 172.21.15.1:0/1677480961 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff21015a240 0x7ff2100b36a0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.655+0000 7ff216be4700 1 --2- 172.21.15.1:0/1677480961 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff21006d560 0x7ff2100b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.655+0000 7ff216be4700 1 --2- 172.21.15.1:0/1677480961 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff210071cc0 0x7ff210159040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.655+0000 7ff216be4700 1 -- 172.21.15.1:0/1677480961 >> 172.21.15.1:0/1677480961 conn(0x7ff2100cff30 msgr2=0x7ff2100cf700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:26.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.655+0000 7ff216be4700 1 -- 172.21.15.1:0/1677480961 shutdown_connections 2024-03-18T19:06:26.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.655+0000 7ff216be4700 1 -- 172.21.15.1:0/1677480961 wait complete. 2024-03-18T19:06:26.657 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:26.662+0000 7fc75dbc3240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:06:26.664 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:06:26.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.800+0000 7f6c4cc47700 1 Processor -- start 2024-03-18T19:06:26.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.800+0000 7f6c4cc47700 1 -- start start 2024-03-18T19:06:26.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.801+0000 7f6c4cc47700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6c4806a750 0x7f6c4806ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:26.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.801+0000 7f6c4cc47700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c480ad790 0x7f6c48161700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:26.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.801+0000 7f6c4cc47700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c480acfb0 con 0x7f6c4815d6b0 2024-03-18T19:06:26.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.801+0000 7f6c4cc47700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c480af7a0 con 0x7f6c4806a750 2024-03-18T19:06:26.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.801+0000 7f6c4cc47700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4806b070 con 0x7f6c480ad790 2024-03-18T19:06:26.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.801+0000 7f6c45d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6c4815d6b0 0x7f6c48160ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52386/0 (socket says 172.21.15.1:52386) 2024-03-18T19:06:26.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.801+0000 7f6c45d9b700 1 -- 172.21.15.1:0/2458085903 learned_addr learned my addr 172.21.15.1:0/2458085903 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:26.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.801+0000 7f6c4659c700 1 --2- 172.21.15.1:0/2458085903 >> v2:172.21.15.165:3300/0 conn(0x7f6c4806a750 0x7f6c4806ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:26.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.801+0000 7f6c46d9d700 1 --2- 172.21.15.1:0/2458085903 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c480ad790 0x7f6c48161700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:26.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.801+0000 7f6c4559a700 1 -- 172.21.15.1:0/2458085903 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894721152 0 0) 0x7f6c480acfb0 con 0x7f6c4815d6b0 2024-03-18T19:06:26.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.801+0000 7f6c4559a700 1 -- 172.21.15.1:0/2458085903 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c300035b0 con 0x7f6c4815d6b0 2024-03-18T19:06:26.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.801+0000 7f6c4659c700 1 -- 172.21.15.1:0/2458085903 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c480ad790 msgr2=0x7f6c48161700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:26.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.801+0000 7f6c4659c700 1 --2- 172.21.15.1:0/2458085903 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c480ad790 0x7f6c48161700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.801+0000 7f6c4659c700 1 -- 172.21.15.1:0/2458085903 >> v1:172.21.15.1:6789/0 conn(0x7f6c4815d6b0 legacy=0x7f6c48160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:26.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.801+0000 7f6c4659c700 1 -- 172.21.15.1:0/2458085903 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c480b50a0 con 0x7f6c4806a750 2024-03-18T19:06:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.802+0000 7f6c4659c700 1 --2- 172.21.15.1:0/2458085903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c4806a750 0x7f6c4806ab30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6c38009840 tx=0x7f6c38009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33510ffad09cc77e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.802+0000 7f6c4559a700 1 -- 172.21.15.1:0/2458085903 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c38011830 con 0x7f6c4806a750 2024-03-18T19:06:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.802+0000 7f6c4559a700 1 -- 172.21.15.1:0/2458085903 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c38011990 con 0x7f6c4806a750 2024-03-18T19:06:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.802+0000 7f6c4559a700 1 -- 172.21.15.1:0/2458085903 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c38011d20 con 0x7f6c4806a750 2024-03-18T19:06:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.802+0000 7f6c4cc47700 1 -- 172.21.15.1:0/2458085903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c4806a750 msgr2=0x7f6c4806ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.802+0000 7f6c4cc47700 1 --2- 172.21.15.1:0/2458085903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c4806a750 0x7f6c4806ab30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6c38009840 tx=0x7f6c38009b50 comp rx=0 tx=0).stop 2024-03-18T19:06:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.802+0000 7f6c4cc47700 1 -- 172.21.15.1:0/2458085903 shutdown_connections 2024-03-18T19:06:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.802+0000 7f6c4cc47700 1 --2- 172.21.15.1:0/2458085903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c4806a750 0x7f6c4806ab30 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.802+0000 7f6c4cc47700 1 --2- 172.21.15.1:0/2458085903 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c480ad790 0x7f6c48161700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.802+0000 7f6c4cc47700 1 -- 172.21.15.1:0/2458085903 >> 172.21.15.1:0/2458085903 conn(0x7f6c4806b860 msgr2=0x7f6c4806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.803+0000 7f6c4cc47700 1 -- 172.21.15.1:0/2458085903 shutdown_connections 2024-03-18T19:06:26.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.803+0000 7f6c4cc47700 1 -- 172.21.15.1:0/2458085903 wait complete. 2024-03-18T19:06:26.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.804+0000 7f6c4cc47700 1 Processor -- start 2024-03-18T19:06:26.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.804+0000 7f6c4cc47700 1 -- start start 2024-03-18T19:06:26.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.805+0000 7f6c4cc47700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c4806a750 0x7f6c480cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:26.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.805+0000 7f6c4cc47700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c480ad790 0x7f6c480cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:26.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.805+0000 7f6c4cc47700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c4815d6b0 0x7f6c4815fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:26.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.805+0000 7f6c4cc47700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4815de40 con 0x7f6c4815d6b0 2024-03-18T19:06:26.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.805+0000 7f6c4cc47700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c480cc530 con 0x7f6c4806a750 2024-03-18T19:06:26.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.805+0000 7f6c4cc47700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c480ce9d0 con 0x7f6c480ad790 2024-03-18T19:06:26.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.805+0000 7f6c46d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c4815d6b0 0x7f6c4815fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:26.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.805+0000 7f6c4659c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c4806a750 0x7f6c480cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:26.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.805+0000 7f6c46d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c4815d6b0 0x7f6c4815fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54060/0 (socket says 172.21.15.1:54060) 2024-03-18T19:06:26.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.805+0000 7f6c46d9d700 1 -- 172.21.15.1:0/1388575548 learned_addr learned my addr 172.21.15.1:0/1388575548 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:26.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.805+0000 7f6c45d9b700 1 --2- 172.21.15.1:0/1388575548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c480ad790 0x7f6c480cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:26.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.805+0000 7f6c46d9d700 1 -- 172.21.15.1:0/1388575548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c480ad790 msgr2=0x7f6c480cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:26.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.805+0000 7f6c46d9d700 1 --2- 172.21.15.1:0/1388575548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c480ad790 0x7f6c480cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.805+0000 7f6c46d9d700 1 -- 172.21.15.1:0/1388575548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c4806a750 msgr2=0x7f6c480cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:26.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.805+0000 7f6c46d9d700 1 --2- 172.21.15.1:0/1388575548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c4806a750 0x7f6c480cbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:26.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.805+0000 7f6c46d9d700 1 -- 172.21.15.1:0/1388575548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c380094a0 con 0x7f6c4815d6b0 2024-03-18T19:06:26.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.805+0000 7f6c4659c700 1 --2- 172.21.15.1:0/1388575548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c4806a750 0x7f6c480cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:06:26.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.805+0000 7f6c46d9d700 1 --2- 172.21.15.1:0/1388575548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c4815d6b0 0x7f6c4815fad0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f6c4000efc0 tx=0x7f6c4000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:26.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.805+0000 7f6c36ffd700 1 -- 172.21.15.1:0/1388575548 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c40009ea0 con 0x7f6c4815d6b0 2024-03-18T19:06:26.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.806+0000 7f6c36ffd700 1 -- 172.21.15.1:0/1388575548 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4000f040 con 0x7f6c4815d6b0 2024-03-18T19:06:26.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.806+0000 7f6c4cc47700 1 -- 172.21.15.1:0/1388575548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c480cecb0 con 0x7f6c4815d6b0 2024-03-18T19:06:26.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.806+0000 7f6c36ffd700 1 -- 172.21.15.1:0/1388575548 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4000ced0 con 0x7f6c4815d6b0 2024-03-18T19:06:26.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.806+0000 7f6c4cc47700 1 -- 172.21.15.1:0/1388575548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c480ce5e0 con 0x7f6c4815d6b0 2024-03-18T19:06:26.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.807+0000 7f6c4cc47700 1 -- 172.21.15.1:0/1388575548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c480ceef0 con 0x7f6c4815d6b0 2024-03-18T19:06:26.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.807+0000 7f6c36ffd700 1 -- 172.21.15.1:0/1388575548 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 45) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f6c40014480 con 0x7f6c4815d6b0 2024-03-18T19:06:26.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.807+0000 7f6c36ffd700 1 --2- 172.21.15.1:0/1388575548 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c2806adb0 0x7f6c2806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:26.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.807+0000 7f6c4659c700 1 -- 172.21.15.1:0/1388575548 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c2806adb0 msgr2=0x7f6c2806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:26.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.808+0000 7f6c36ffd700 1 -- 172.21.15.1:0/1388575548 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f6c4008a320 con 0x7f6c4815d6b0 2024-03-18T19:06:26.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.808+0000 7f6c4659c700 1 --2- 172.21.15.1:0/1388575548 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c2806adb0 0x7f6c2806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:26.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:26.816+0000 7f6c36ffd700 1 -- 172.21.15.1:0/1388575548 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6c480ceef0 con 0x7f6c4815d6b0 2024-03-18T19:06:26.820 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:26.825+0000 7fc75dbc3240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:06:26.983 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:26.987+0000 7f58d2245240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:06:27.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.009+0000 7f6c4659c700 1 -- 172.21.15.1:0/1388575548 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c2806adb0 msgr2=0x7f6c2806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:27.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.009+0000 7f6c4659c700 1 --2- 172.21.15.1:0/1388575548 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c2806adb0 0x7f6c2806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:27.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.069+0000 7f6c4cc47700 1 -- 172.21.15.1:0/1388575548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c4815dc10 con 0x7f6c4815d6b0 2024-03-18T19:06:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.070+0000 7f6c36ffd700 1 -- 172.21.15.1:0/1388575548 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f6c4815dc10 con 0x7f6c4815d6b0 2024-03-18T19:06:27.066 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:27.066 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3926877640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3926877640},{"type":"v1","addr":"172.21.15.165:6833","nonce":3926877640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/4293662065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":4293662065},{"type":"v1","addr":"172.21.15.165:6835","nonce":4293662065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2334043797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2334043797},{"type":"v1","addr":"172.21.15.1:6835","nonce":2334043797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:06:05.147601+0000","modified":"2024-03-18T19:06:08.160700+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2526016488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":2526016488},{"type":"v1","addr":"172.21.15.1:6837","nonce":2526016488}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-18T19:06:27.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.072+0000 7f6c4cc47700 1 -- 172.21.15.1:0/1388575548 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c2806adb0 msgr2=0x7f6c2806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:27.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.072+0000 7f6c4cc47700 1 --2- 172.21.15.1:0/1388575548 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c2806adb0 0x7f6c2806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.072+0000 7f6c4cc47700 1 -- 172.21.15.1:0/1388575548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c4815d6b0 msgr2=0x7f6c4815fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.072+0000 7f6c4cc47700 1 --2- 172.21.15.1:0/1388575548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c4815d6b0 0x7f6c4815fad0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f6c4000efc0 tx=0x7f6c4000c3a0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.072+0000 7f6c4cc47700 1 -- 172.21.15.1:0/1388575548 shutdown_connections 2024-03-18T19:06:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.072+0000 7f6c4cc47700 1 --2- 172.21.15.1:0/1388575548 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c2806adb0 0x7f6c2806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.072+0000 7f6c4cc47700 1 --2- 172.21.15.1:0/1388575548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c4815d6b0 0x7f6c4815fad0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.072+0000 7f6c4cc47700 1 --2- 172.21.15.1:0/1388575548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c4806a750 0x7f6c480cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.072+0000 7f6c4cc47700 1 --2- 172.21.15.1:0/1388575548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c480ad790 0x7f6c480cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.072+0000 7f6c4cc47700 1 -- 172.21.15.1:0/1388575548 >> 172.21.15.1:0/1388575548 conn(0x7f6c4806b860 msgr2=0x7f6c4806d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.072+0000 7f6c4cc47700 1 -- 172.21.15.1:0/1388575548 shutdown_connections 2024-03-18T19:06:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.072+0000 7f6c4cc47700 1 -- 172.21.15.1:0/1388575548 wait complete. 2024-03-18T19:06:27.069 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 42 2024-03-18T19:06:27.080 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:06:27.172 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:27.176+0000 7f58d2245240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:06:27.207 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:27.212+0000 7fc75dbc3240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:06:27.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.220+0000 7f7c1e0d8700 1 Processor -- start 2024-03-18T19:06:27.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.220+0000 7f7c1e0d8700 1 -- start start 2024-03-18T19:06:27.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.220+0000 7f7c1e0d8700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7c1815b4d0 0x7f7c1809a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:27.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.220+0000 7f7c1e0d8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7c180cf750 0x7f7c18099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:27.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.220+0000 7f7c1e0d8700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1806f070 con 0x7f7c18099f80 2024-03-18T19:06:27.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.220+0000 7f7c1e0d8700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c1809a3f0 con 0x7f7c180cf750 2024-03-18T19:06:27.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.220+0000 7f7c1e0d8700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c180d0260 con 0x7f7c1815b4d0 2024-03-18T19:06:27.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.221+0000 7f7c16ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7c180cf750 0x7f7c18099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:27.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.221+0000 7f7c16ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7c180cf750 0x7f7c18099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54066/0 (socket says 172.21.15.1:54066) 2024-03-18T19:06:27.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.221+0000 7f7c16ffd700 1 -- 172.21.15.1:0/3832087029 learned_addr learned my addr 172.21.15.1:0/3832087029 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:27.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.221+0000 7f7c177fe700 1 --2- 172.21.15.1:0/3832087029 >> v2:172.21.15.165:3300/0 conn(0x7f7c1815b4d0 0x7f7c1809a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:27.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.221+0000 7f7c16ffd700 1 -- 172.21.15.1:0/3832087029 >> v2:172.21.15.165:3300/0 conn(0x7f7c1815b4d0 msgr2=0x7f7c1809a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:27.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.221+0000 7f7c16ffd700 1 --2- 172.21.15.1:0/3832087029 >> v2:172.21.15.165:3300/0 conn(0x7f7c1815b4d0 0x7f7c1809a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.221+0000 7f7c16ffd700 1 -- 172.21.15.1:0/3832087029 >> v1:172.21.15.165:6789/0 conn(0x7f7c18099f80 legacy=0x7f7c180cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:27.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.221+0000 7f7c16ffd700 1 -- 172.21.15.1:0/3832087029 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c180cff80 con 0x7f7c180cf750 2024-03-18T19:06:27.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.221+0000 7f7c16ffd700 1 --2- 172.21.15.1:0/3832087029 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c180cf750 0x7f7c18099a40 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f7c18071290 tx=0x7f7c0c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8819acdb7ae2a119 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:27.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.221+0000 7f7c167fc700 1 -- 172.21.15.1:0/3832087029 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c011830 con 0x7f7c180cf750 2024-03-18T19:06:27.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.221+0000 7f7c167fc700 1 -- 172.21.15.1:0/3832087029 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0c011990 con 0x7f7c180cf750 2024-03-18T19:06:27.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.221+0000 7f7c167fc700 1 -- 172.21.15.1:0/3832087029 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c011d20 con 0x7f7c180cf750 2024-03-18T19:06:27.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.222+0000 7f7c1e0d8700 1 -- 172.21.15.1:0/3832087029 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c180cf750 msgr2=0x7f7c18099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:27.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.222+0000 7f7c1e0d8700 1 --2- 172.21.15.1:0/3832087029 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c180cf750 0x7f7c18099a40 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f7c18071290 tx=0x7f7c0c009bd0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.222+0000 7f7c1e0d8700 1 -- 172.21.15.1:0/3832087029 shutdown_connections 2024-03-18T19:06:27.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.222+0000 7f7c1e0d8700 1 --2- 172.21.15.1:0/3832087029 >> v2:172.21.15.165:3300/0 conn(0x7f7c1815b4d0 0x7f7c1809a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.222+0000 7f7c1e0d8700 1 --2- 172.21.15.1:0/3832087029 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c180cf750 0x7f7c18099a40 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.222+0000 7f7c1e0d8700 1 -- 172.21.15.1:0/3832087029 >> 172.21.15.1:0/3832087029 conn(0x7f7c180b2cb0 msgr2=0x7f7c180b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:27.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.222+0000 7f7c1e0d8700 1 -- 172.21.15.1:0/3832087029 shutdown_connections 2024-03-18T19:06:27.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.222+0000 7f7c1e0d8700 1 -- 172.21.15.1:0/3832087029 wait complete. 2024-03-18T19:06:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.223+0000 7f7c1e0d8700 1 Processor -- start 2024-03-18T19:06:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.223+0000 7f7c1e0d8700 1 -- start start 2024-03-18T19:06:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.224+0000 7f7c1e0d8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c18099f80 0x7f7c180c8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.224+0000 7f7c1e0d8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c180cf750 0x7f7c180c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.224+0000 7f7c1e0d8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c1815b4d0 0x7f7c180c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.224+0000 7f7c1e0d8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c180ca5d0 con 0x7f7c180cf750 2024-03-18T19:06:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.224+0000 7f7c1e0d8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c180ca8b0 con 0x7f7c18099f80 2024-03-18T19:06:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.224+0000 7f7c1e0d8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c180cab90 con 0x7f7c1815b4d0 2024-03-18T19:06:27.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.224+0000 7f7c16ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c180cf750 0x7f7c180c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.224+0000 7f7c16ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c180cf750 0x7f7c180c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54080/0 (socket says 172.21.15.1:54080) 2024-03-18T19:06:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.224+0000 7f7c16ffd700 1 -- 172.21.15.1:0/4113062757 learned_addr learned my addr 172.21.15.1:0/4113062757 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.224+0000 7f7c177fe700 1 --2- 172.21.15.1:0/4113062757 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c18099f80 0x7f7c180c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.224+0000 7f7c17fff700 1 --2- 172.21.15.1:0/4113062757 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c1815b4d0 0x7f7c180c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.225+0000 7f7c16ffd700 1 -- 172.21.15.1:0/4113062757 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c1815b4d0 msgr2=0x7f7c180c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.225+0000 7f7c16ffd700 1 --2- 172.21.15.1:0/4113062757 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c1815b4d0 0x7f7c180c9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.225+0000 7f7c16ffd700 1 -- 172.21.15.1:0/4113062757 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c18099f80 msgr2=0x7f7c180c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.225+0000 7f7c16ffd700 1 --2- 172.21.15.1:0/4113062757 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c18099f80 0x7f7c180c8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.225+0000 7f7c16ffd700 1 -- 172.21.15.1:0/4113062757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c009550 con 0x7f7c180cf750 2024-03-18T19:06:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.225+0000 7f7c16ffd700 1 --2- 172.21.15.1:0/4113062757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c180cf750 0x7f7c180c7870 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c000c00 tx=0x7f7c0c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.225+0000 7f7bfffff700 1 -- 172.21.15.1:0/4113062757 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c004380 con 0x7f7c180cf750 2024-03-18T19:06:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.225+0000 7f7bfffff700 1 -- 172.21.15.1:0/4113062757 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0c01f070 con 0x7f7c180cf750 2024-03-18T19:06:27.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.225+0000 7f7c1e0d8700 1 -- 172.21.15.1:0/4113062757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c180cae70 con 0x7f7c180cf750 2024-03-18T19:06:27.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.225+0000 7f7bfffff700 1 -- 172.21.15.1:0/4113062757 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c0036a0 con 0x7f7c180cf750 2024-03-18T19:06:27.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.225+0000 7f7c1e0d8700 1 -- 172.21.15.1:0/4113062757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c180c7db0 con 0x7f7c180cf750 2024-03-18T19:06:27.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.226+0000 7f7c1e0d8700 1 -- 172.21.15.1:0/4113062757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c18068240 con 0x7f7c180cf750 2024-03-18T19:06:27.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.227+0000 7f7bfffff700 1 -- 172.21.15.1:0/4113062757 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 45) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f7c0c021030 con 0x7f7c180cf750 2024-03-18T19:06:27.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.227+0000 7f7bfffff700 1 --2- 172.21.15.1:0/4113062757 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7c0406adb0 0x7f7c0406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:27.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.227+0000 7f7c177fe700 1 -- 172.21.15.1:0/4113062757 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7c0406adb0 msgr2=0x7f7c0406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:27.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.227+0000 7f7bfffff700 1 -- 172.21.15.1:0/4113062757 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f7c0c08aa90 con 0x7f7c180cf750 2024-03-18T19:06:27.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.227+0000 7f7c177fe700 1 --2- 172.21.15.1:0/4113062757 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7c0406adb0 0x7f7c0406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:27.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.235+0000 7f7bfffff700 1 -- 172.21.15.1:0/4113062757 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7c0c0558d0 con 0x7f7c180cf750 2024-03-18T19:06:27.372 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:27.377+0000 7fc75dbc3240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:06:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.428+0000 7f7c177fe700 1 -- 172.21.15.1:0/4113062757 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7c0406adb0 msgr2=0x7f7c0406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:27.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.428+0000 7f7c177fe700 1 --2- 172.21.15.1:0/4113062757 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7c0406adb0 0x7f7c0406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:27.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.445+0000 7f7c1e0d8700 1 -- 172.21.15.1:0/4113062757 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c180cb390 con 0x7f7c180cf750 2024-03-18T19:06:27.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.446+0000 7f7bfffff700 1 -- 172.21.15.1:0/4113062757 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+16754 (secure 0 0 0) 0x7f7c180cb390 con 0x7f7c180cf750 2024-03-18T19:06:27.442 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:27.442 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":104,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:06:20.400812+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:06:02.855989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-18T19:06:04.121199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/1060785910":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/3060583633":"2024-03-19T19:06:20.400777+0000","172.21.15.1:6833/76998":"2024-03-19T19:06:20.400777+0000","172.21.15.1:6832/76998":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/809308784":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/1836399926":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/3606402343":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/2494839321":"2024-03-19T19:06:20.400777+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:06:27.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.448+0000 7f7c1e0d8700 1 -- 172.21.15.1:0/4113062757 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7c0406adb0 msgr2=0x7f7c0406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:27.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.448+0000 7f7c1e0d8700 1 --2- 172.21.15.1:0/4113062757 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7c0406adb0 0x7f7c0406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.448+0000 7f7c1e0d8700 1 -- 172.21.15.1:0/4113062757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c180cf750 msgr2=0x7f7c180c7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:27.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.448+0000 7f7c1e0d8700 1 --2- 172.21.15.1:0/4113062757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c180cf750 0x7f7c180c7870 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c000c00 tx=0x7f7c0c004750 comp rx=0 tx=0).stop 2024-03-18T19:06:27.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.448+0000 7f7c1e0d8700 1 -- 172.21.15.1:0/4113062757 shutdown_connections 2024-03-18T19:06:27.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.448+0000 7f7c1e0d8700 1 --2- 172.21.15.1:0/4113062757 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7c0406adb0 0x7f7c0406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.448+0000 7f7c1e0d8700 1 --2- 172.21.15.1:0/4113062757 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7c1815b4d0 0x7f7c180c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.448+0000 7f7c1e0d8700 1 --2- 172.21.15.1:0/4113062757 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c18099f80 0x7f7c180c8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.448+0000 7f7c1e0d8700 1 --2- 172.21.15.1:0/4113062757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c180cf750 0x7f7c180c7870 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.448+0000 7f7c1e0d8700 1 -- 172.21.15.1:0/4113062757 >> 172.21.15.1:0/4113062757 conn(0x7f7c180b2cb0 msgr2=0x7f7c180afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:27.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.448+0000 7f7c1e0d8700 1 -- 172.21.15.1:0/4113062757 shutdown_connections 2024-03-18T19:06:27.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.448+0000 7f7c1e0d8700 1 -- 172.21.15.1:0/4113062757 wait complete. 2024-03-18T19:06:27.457 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T19:06:27.457 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:06:27.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.588+0000 7f9880ef0700 1 Processor -- start 2024-03-18T19:06:27.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.588+0000 7f9880ef0700 1 -- start start 2024-03-18T19:06:27.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.589+0000 7f9880ef0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f987c15d500 0x7f987c0d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:27.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.589+0000 7f9880ef0700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f987c0cddc0 con 0x7f987c0d1300 2024-03-18T19:06:27.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.589+0000 7f9880ef0700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f987c0d3dc0 con 0x7f987c0d1ec0 2024-03-18T19:06:27.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.589+0000 7f9880ef0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0ce060 con 0x7f987c15d500 2024-03-18T19:06:27.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.589+0000 7f9879d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f987c0d1300 0x7f987c0d17b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52402/0 (socket says 172.21.15.1:52402) 2024-03-18T19:06:27.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.589+0000 7f9879d9b700 1 -- 172.21.15.1:0/1420785080 learned_addr learned my addr 172.21.15.1:0/1420785080 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:27.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.589+0000 7f987a59c700 1 --2- 172.21.15.1:0/1420785080 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f987c15d500 0x7f987c0d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:27.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.589+0000 7f987959a700 1 -- 172.21.15.1:0/1420785080 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90484037 0 0) 0x7f987c0cddc0 con 0x7f987c0d1300 2024-03-18T19:06:27.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.589+0000 7f987959a700 1 -- 172.21.15.1:0/1420785080 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98640035b0 con 0x7f987c0d1300 2024-03-18T19:06:27.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.589+0000 7f987959a700 1 -- 172.21.15.1:0/1420785080 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020545333 0 0) 0x7f987c0d3dc0 con 0x7f987c0d1ec0 2024-03-18T19:06:27.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.589+0000 7f987959a700 1 -- 172.21.15.1:0/1420785080 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f987c0cddc0 con 0x7f987c0d1ec0 2024-03-18T19:06:27.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.589+0000 7f987a59c700 1 -- 172.21.15.1:0/1420785080 >> v1:172.21.15.165:6789/0 conn(0x7f987c0d1ec0 legacy=0x7f987c0b83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:27.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.589+0000 7f987a59c700 1 -- 172.21.15.1:0/1420785080 >> v1:172.21.15.1:6789/0 conn(0x7f987c0d1300 legacy=0x7f987c0d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:27.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.589+0000 7f987a59c700 1 -- 172.21.15.1:0/1420785080 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c0ce340 con 0x7f987c15d500 2024-03-18T19:06:27.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.590+0000 7f987a59c700 1 --2- 172.21.15.1:0/1420785080 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f987c15d500 0x7f987c0d0dc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f986c009870 tx=0x7f986c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7cd4166a9d06d18 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:27.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.590+0000 7f987959a700 1 -- 172.21.15.1:0/1420785080 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c011830 con 0x7f987c15d500 2024-03-18T19:06:27.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.590+0000 7f987959a700 1 -- 172.21.15.1:0/1420785080 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986c011990 con 0x7f987c15d500 2024-03-18T19:06:27.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.590+0000 7f987959a700 1 -- 172.21.15.1:0/1420785080 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c011d20 con 0x7f987c15d500 2024-03-18T19:06:27.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.590+0000 7f9880ef0700 1 -- 172.21.15.1:0/1420785080 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f987c15d500 msgr2=0x7f987c0d0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:27.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.590+0000 7f9880ef0700 1 --2- 172.21.15.1:0/1420785080 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f987c15d500 0x7f987c0d0dc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f986c009870 tx=0x7f986c009b80 comp rx=0 tx=0).stop 2024-03-18T19:06:27.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.591+0000 7f9880ef0700 1 -- 172.21.15.1:0/1420785080 shutdown_connections 2024-03-18T19:06:27.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.591+0000 7f9880ef0700 1 --2- 172.21.15.1:0/1420785080 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f987c15d500 0x7f987c0d0dc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.591+0000 7f9880ef0700 1 -- 172.21.15.1:0/1420785080 >> 172.21.15.1:0/1420785080 conn(0x7f987c09c530 msgr2=0x7f987c0d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:27.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.591+0000 7f9880ef0700 1 -- 172.21.15.1:0/1420785080 shutdown_connections 2024-03-18T19:06:27.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.591+0000 7f9880ef0700 1 -- 172.21.15.1:0/1420785080 wait complete. 2024-03-18T19:06:27.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.592+0000 7f9880ef0700 1 Processor -- start 2024-03-18T19:06:27.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.592+0000 7f9880ef0700 1 -- start start 2024-03-18T19:06:27.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.593+0000 7f9880ef0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f987c0d1300 0x7f987c0b8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:27.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.593+0000 7f9880ef0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f987c0d1ec0 0x7f987c0c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:27.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.593+0000 7f9880ef0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f987c15d500 0x7f987c1f2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:27.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.593+0000 7f9880ef0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0b8cb0 con 0x7f987c0d1300 2024-03-18T19:06:27.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.593+0000 7f9880ef0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0c90d0 con 0x7f987c0d1ec0 2024-03-18T19:06:27.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.593+0000 7f9880ef0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0c9320 con 0x7f987c15d500 2024-03-18T19:06:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.593+0000 7f987a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f987c0d1300 0x7f987c0b8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.593+0000 7f987a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f987c0d1300 0x7f987c0b8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54094/0 (socket says 172.21.15.1:54094) 2024-03-18T19:06:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.593+0000 7f987a59c700 1 -- 172.21.15.1:0/641361411 learned_addr learned my addr 172.21.15.1:0/641361411 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.593+0000 7f9879d9b700 1 --2- 172.21.15.1:0/641361411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f987c0d1ec0 0x7f987c0c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.593+0000 7f987ad9d700 1 --2- 172.21.15.1:0/641361411 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f987c15d500 0x7f987c1f2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.593+0000 7f987a59c700 1 -- 172.21.15.1:0/641361411 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f987c15d500 msgr2=0x7f987c1f2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.593+0000 7f987a59c700 1 --2- 172.21.15.1:0/641361411 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f987c15d500 0x7f987c1f2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.593+0000 7f987a59c700 1 -- 172.21.15.1:0/641361411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f987c0d1ec0 msgr2=0x7f987c0c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.593+0000 7f987a59c700 1 --2- 172.21.15.1:0/641361411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f987c0d1ec0 0x7f987c0c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.593+0000 7f987a59c700 1 -- 172.21.15.1:0/641361411 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c0094a0 con 0x7f987c0d1300 2024-03-18T19:06:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.593+0000 7f987a59c700 1 --2- 172.21.15.1:0/641361411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f987c0d1300 0x7f987c0b8770 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f986c009bc0 tx=0x7f986c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.594+0000 7f986affd700 1 -- 172.21.15.1:0/641361411 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c01a6e0 con 0x7f987c0d1300 2024-03-18T19:06:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.594+0000 7f986affd700 1 -- 172.21.15.1:0/641361411 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986c01a860 con 0x7f987c0d1300 2024-03-18T19:06:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.594+0000 7f9880ef0700 1 -- 172.21.15.1:0/641361411 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f987c0c9600 con 0x7f987c0d1300 2024-03-18T19:06:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.594+0000 7f986affd700 1 -- 172.21.15.1:0/641361411 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c01ac10 con 0x7f987c0d1300 2024-03-18T19:06:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.594+0000 7f9880ef0700 1 -- 172.21.15.1:0/641361411 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f987c0c9bc0 con 0x7f987c0d1300 2024-03-18T19:06:27.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.595+0000 7f9880ef0700 1 -- 172.21.15.1:0/641361411 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f987c0676a0 con 0x7f987c0d1300 2024-03-18T19:06:27.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.596+0000 7f986affd700 1 -- 172.21.15.1:0/641361411 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 45) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f986c022070 con 0x7f987c0d1300 2024-03-18T19:06:27.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.596+0000 7f986affd700 1 --2- 172.21.15.1:0/641361411 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f985c06adb0 0x7f985c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:27.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.596+0000 7f9879d9b700 1 -- 172.21.15.1:0/641361411 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f985c06adb0 msgr2=0x7f985c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:27.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.596+0000 7f986affd700 1 -- 172.21.15.1:0/641361411 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f986c08a5a0 con 0x7f987c0d1300 2024-03-18T19:06:27.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.596+0000 7f9879d9b700 1 --2- 172.21.15.1:0/641361411 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f985c06adb0 0x7f985c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.601+0000 7f986affd700 1 -- 172.21.15.1:0/641361411 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f986c055460 con 0x7f987c0d1300 2024-03-18T19:06:27.602 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:27.607+0000 7f58d2245240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:06:27.734 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:27.739+0000 7fc75dbc3240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:06:27.779 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:27.784+0000 7f58d2245240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:06:27.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.797+0000 7f9879d9b700 1 -- 172.21.15.1:0/641361411 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f985c06adb0 msgr2=0x7f985c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:27.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.797+0000 7f9879d9b700 1 --2- 172.21.15.1:0/641361411 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f985c06adb0 0x7f985c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:27.822 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:27.827+0000 7fc75dbc3240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:06:27.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.851+0000 7f9880ef0700 1 -- 172.21.15.1:0/641361411 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f987c0d0950 con 0x7f987c0d1300 2024-03-18T19:06:27.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.852+0000 7f986affd700 1 -- 172.21.15.1:0/641361411 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f986c058a80 con 0x7f987c0d1300 2024-03-18T19:06:27.847 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:27.848 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3926877640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3926877640},{"type":"v1","addr":"172.21.15.165:6833","nonce":3926877640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/4293662065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":4293662065},{"type":"v1","addr":"172.21.15.165:6835","nonce":4293662065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2334043797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2334043797},{"type":"v1","addr":"172.21.15.1:6835","nonce":2334043797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:06:05.147601+0000","modified":"2024-03-18T19:06:08.160700+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2526016488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":2526016488},{"type":"v1","addr":"172.21.15.1:6837","nonce":2526016488}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-18T19:06:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.854+0000 7f9880ef0700 1 -- 172.21.15.1:0/641361411 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f985c06adb0 msgr2=0x7f985c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.854+0000 7f9880ef0700 1 --2- 172.21.15.1:0/641361411 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f985c06adb0 0x7f985c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.854+0000 7f9880ef0700 1 -- 172.21.15.1:0/641361411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f987c0d1300 msgr2=0x7f987c0b8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.854+0000 7f9880ef0700 1 --2- 172.21.15.1:0/641361411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f987c0d1300 0x7f987c0b8770 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f986c009bc0 tx=0x7f986c004750 comp rx=0 tx=0).stop 2024-03-18T19:06:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.854+0000 7f9880ef0700 1 -- 172.21.15.1:0/641361411 shutdown_connections 2024-03-18T19:06:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.854+0000 7f9880ef0700 1 --2- 172.21.15.1:0/641361411 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f985c06adb0 0x7f985c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.854+0000 7f9880ef0700 1 --2- 172.21.15.1:0/641361411 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f987c15d500 0x7f987c1f2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.854+0000 7f9880ef0700 1 --2- 172.21.15.1:0/641361411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f987c0d1300 0x7f987c0b8770 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.854+0000 7f9880ef0700 1 --2- 172.21.15.1:0/641361411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f987c0d1ec0 0x7f987c0c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.854+0000 7f9880ef0700 1 -- 172.21.15.1:0/641361411 >> 172.21.15.1:0/641361411 conn(0x7f987c09c530 msgr2=0x7f987c0d1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:27.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.854+0000 7f9880ef0700 1 -- 172.21.15.1:0/641361411 shutdown_connections 2024-03-18T19:06:27.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.854+0000 7f9880ef0700 1 -- 172.21.15.1:0/641361411 wait complete. 2024-03-18T19:06:27.851 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 42 2024-03-18T19:06:27.862 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:06:27.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.997+0000 7f70bd83a700 1 Processor -- start 2024-03-18T19:06:27.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.997+0000 7f70bd83a700 1 -- start start 2024-03-18T19:06:27.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.998+0000 7f70bd83a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f70b815a2e0 0x7f70b80ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:27.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.998+0000 7f70bd83a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b806e270 con 0x7f70b80ae600 2024-03-18T19:06:27.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.998+0000 7f70bd83a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b80ae050 con 0x7f70b80cc170 2024-03-18T19:06:27.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.998+0000 7f70bd83a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70b806f690 con 0x7f70b815a2e0 2024-03-18T19:06:27.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.998+0000 7f70b6ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f70b80ae600 0x7f70b80ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52408/0 (socket says 172.21.15.1:52408) 2024-03-18T19:06:27.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.998+0000 7f70b6ffd700 1 -- 172.21.15.1:0/120594011 learned_addr learned my addr 172.21.15.1:0/120594011 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:27.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.998+0000 7f70b67fc700 1 --2- 172.21.15.1:0/120594011 >> v2:172.21.15.165:3300/0 conn(0x7f70b815a2e0 0x7f70b80ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:27.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.998+0000 7f70b5ffb700 1 -- 172.21.15.1:0/120594011 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151670242 0 0) 0x7f70b806e270 con 0x7f70b80ae600 2024-03-18T19:06:27.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.999+0000 7f70b5ffb700 1 -- 172.21.15.1:0/120594011 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70a00035b0 con 0x7f70b80ae600 2024-03-18T19:06:27.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.999+0000 7f70b67fc700 1 -- 172.21.15.1:0/120594011 >> v1:172.21.15.165:6789/0 conn(0x7f70b80cc170 legacy=0x7f70b806ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:27.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.999+0000 7f70b67fc700 1 -- 172.21.15.1:0/120594011 >> v1:172.21.15.1:6789/0 conn(0x7f70b80ae600 legacy=0x7f70b80ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:27.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.999+0000 7f70b67fc700 1 -- 172.21.15.1:0/120594011 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b80add70 con 0x7f70b815a2e0 2024-03-18T19:06:27.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.999+0000 7f70b67fc700 1 --2- 172.21.15.1:0/120594011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70b815a2e0 0x7f70b80ad790 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f70ac009840 tx=0x7f70ac009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d20744066e305b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:27.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.999+0000 7f70b5ffb700 1 -- 172.21.15.1:0/120594011 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ac011830 con 0x7f70b815a2e0 2024-03-18T19:06:27.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.999+0000 7f70b5ffb700 1 -- 172.21.15.1:0/120594011 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ac011990 con 0x7f70b815a2e0 2024-03-18T19:06:27.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:27.999+0000 7f70b5ffb700 1 -- 172.21.15.1:0/120594011 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ac011d20 con 0x7f70b815a2e0 2024-03-18T19:06:27.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.000+0000 7f70bd83a700 1 -- 172.21.15.1:0/120594011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70b815a2e0 msgr2=0x7f70b80ad790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:27.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.000+0000 7f70bd83a700 1 --2- 172.21.15.1:0/120594011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70b815a2e0 0x7f70b80ad790 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f70ac009840 tx=0x7f70ac009b50 comp rx=0 tx=0).stop 2024-03-18T19:06:27.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.000+0000 7f70bd83a700 1 -- 172.21.15.1:0/120594011 shutdown_connections 2024-03-18T19:06:27.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.000+0000 7f70bd83a700 1 --2- 172.21.15.1:0/120594011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70b815a2e0 0x7f70b80ad790 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:27.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.000+0000 7f70bd83a700 1 -- 172.21.15.1:0/120594011 >> 172.21.15.1:0/120594011 conn(0x7f70b806b860 msgr2=0x7f70b806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:27.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.000+0000 7f70bd83a700 1 -- 172.21.15.1:0/120594011 shutdown_connections 2024-03-18T19:06:27.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.000+0000 7f70bd83a700 1 -- 172.21.15.1:0/120594011 wait complete. 2024-03-18T19:06:27.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.001+0000 7f70bd83a700 1 Processor -- start 2024-03-18T19:06:27.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.001+0000 7f70bd83a700 1 -- start start 2024-03-18T19:06:28.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.002+0000 7f70bd83a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b80ae600 0x7f70b80cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:28.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.002+0000 7f70bd83a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70b80cc170 0x7f70b81440a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:28.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.002+0000 7f70bd83a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70b815a2e0 0x7f70b80cee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:28.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.002+0000 7f70bd83a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b8145580 con 0x7f70b80ae600 2024-03-18T19:06:28.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.002+0000 7f70bd83a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b81445e0 con 0x7f70b80cc170 2024-03-18T19:06:28.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.002+0000 7f70bd83a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70b8144970 con 0x7f70b815a2e0 2024-03-18T19:06:28.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.002+0000 7f70b6ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b80ae600 0x7f70b80cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:28.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.002+0000 7f70b6ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b80ae600 0x7f70b80cc6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54098/0 (socket says 172.21.15.1:54098) 2024-03-18T19:06:28.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.002+0000 7f70b6ffd700 1 -- 172.21.15.1:0/4204712577 learned_addr learned my addr 172.21.15.1:0/4204712577 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:28.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.002+0000 7f70b67fc700 1 --2- 172.21.15.1:0/4204712577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70b80cc170 0x7f70b81440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:28.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.002+0000 7f70b77fe700 1 --2- 172.21.15.1:0/4204712577 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70b815a2e0 0x7f70b80cee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:28.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.003+0000 7f70b6ffd700 1 -- 172.21.15.1:0/4204712577 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70b815a2e0 msgr2=0x7f70b80cee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:28.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.003+0000 7f70b6ffd700 1 --2- 172.21.15.1:0/4204712577 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70b815a2e0 0x7f70b80cee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.003+0000 7f70b6ffd700 1 -- 172.21.15.1:0/4204712577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70b80cc170 msgr2=0x7f70b81440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:28.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.003+0000 7f70b6ffd700 1 --2- 172.21.15.1:0/4204712577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70b80cc170 0x7f70b81440a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.003+0000 7f70b6ffd700 1 -- 172.21.15.1:0/4204712577 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ac0094a0 con 0x7f70b80ae600 2024-03-18T19:06:28.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.003+0000 7f70b67fc700 1 --2- 172.21.15.1:0/4204712577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70b80cc170 0x7f70b81440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:28.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.003+0000 7f70b6ffd700 1 --2- 172.21.15.1:0/4204712577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b80ae600 0x7f70b80cc6c0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f70a800c700 tx=0x7f70a800cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:28.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.003+0000 7f709ffff700 1 -- 172.21.15.1:0/4204712577 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a8005d70 con 0x7f70b80ae600 2024-03-18T19:06:28.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.003+0000 7f709ffff700 1 -- 172.21.15.1:0/4204712577 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a80070d0 con 0x7f70b80ae600 2024-03-18T19:06:28.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.003+0000 7f70bd83a700 1 -- 172.21.15.1:0/4204712577 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70b8144c50 con 0x7f70b80ae600 2024-03-18T19:06:28.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.003+0000 7f709ffff700 1 -- 172.21.15.1:0/4204712577 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a8007460 con 0x7f70b80ae600 2024-03-18T19:06:28.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.003+0000 7f70bd83a700 1 -- 172.21.15.1:0/4204712577 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70b8145210 con 0x7f70b80ae600 2024-03-18T19:06:28.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.004+0000 7f70bd83a700 1 -- 172.21.15.1:0/4204712577 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70b80676a0 con 0x7f70b80ae600 2024-03-18T19:06:28.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.004+0000 7f709ffff700 1 -- 172.21.15.1:0/4204712577 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 45) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f70a80075c0 con 0x7f70b80ae600 2024-03-18T19:06:28.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.005+0000 7f709ffff700 1 --2- 172.21.15.1:0/4204712577 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f70a406adb0 0x7f70a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:28.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.005+0000 7f70b67fc700 1 -- 172.21.15.1:0/4204712577 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f70a406adb0 msgr2=0x7f70a406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:28.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.005+0000 7f709ffff700 1 -- 172.21.15.1:0/4204712577 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f70a8088d60 con 0x7f70b80ae600 2024-03-18T19:06:28.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.005+0000 7f70b67fc700 1 --2- 172.21.15.1:0/4204712577 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f70a406adb0 0x7f70a406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:28.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.010+0000 7f709ffff700 1 -- 172.21.15.1:0/4204712577 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f70a8053ba0 con 0x7f70b80ae600 2024-03-18T19:06:28.163 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:28.167+0000 7f58d2245240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:06:28.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.206+0000 7f70b67fc700 1 -- 172.21.15.1:0/4204712577 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f70a406adb0 msgr2=0x7f70a406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:28.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.206+0000 7f70b67fc700 1 --2- 172.21.15.1:0/4204712577 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f70a406adb0 0x7f70a406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:28.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.260+0000 7f70bd83a700 1 -- 172.21.15.1:0/4204712577 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70b806c420 con 0x7f70b80ae600 2024-03-18T19:06:28.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.261+0000 7f709ffff700 1 -- 172.21.15.1:0/4204712577 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f70a80571c0 con 0x7f70b80ae600 2024-03-18T19:06:28.257 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:28.257 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3926877640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3926877640},{"type":"v1","addr":"172.21.15.165:6833","nonce":3926877640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/4293662065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":4293662065},{"type":"v1","addr":"172.21.15.165:6835","nonce":4293662065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2334043797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2334043797},{"type":"v1","addr":"172.21.15.1:6835","nonce":2334043797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:06:05.147601+0000","modified":"2024-03-18T19:06:08.160700+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7296":{"gid":7296,"name":"a","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2526016488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":2526016488},{"type":"v1","addr":"172.21.15.1:6837","nonce":2526016488}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-18T19:06:28.258 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:28.262+0000 7f58d2245240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:06:28.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.263+0000 7f70bd83a700 1 -- 172.21.15.1:0/4204712577 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f70a406adb0 msgr2=0x7f70a406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:28.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.263+0000 7f70bd83a700 1 --2- 172.21.15.1:0/4204712577 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f70a406adb0 0x7f70a406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.263+0000 7f70bd83a700 1 -- 172.21.15.1:0/4204712577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b80ae600 msgr2=0x7f70b80cc6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:28.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.263+0000 7f70bd83a700 1 --2- 172.21.15.1:0/4204712577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b80ae600 0x7f70b80cc6c0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f70a800c700 tx=0x7f70a800cac0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.263+0000 7f70bd83a700 1 -- 172.21.15.1:0/4204712577 shutdown_connections 2024-03-18T19:06:28.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.263+0000 7f70bd83a700 1 --2- 172.21.15.1:0/4204712577 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f70a406adb0 0x7f70a406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.263+0000 7f70bd83a700 1 --2- 172.21.15.1:0/4204712577 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f70b815a2e0 0x7f70b80cee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.263+0000 7f70bd83a700 1 --2- 172.21.15.1:0/4204712577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b80ae600 0x7f70b80cc6c0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.263+0000 7f70bd83a700 1 --2- 172.21.15.1:0/4204712577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70b80cc170 0x7f70b81440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.263+0000 7f70bd83a700 1 -- 172.21.15.1:0/4204712577 >> 172.21.15.1:0/4204712577 conn(0x7f70b806b860 msgr2=0x7f70b806a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:28.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.263+0000 7f70bd83a700 1 -- 172.21.15.1:0/4204712577 shutdown_connections 2024-03-18T19:06:28.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.263+0000 7f70bd83a700 1 -- 172.21.15.1:0/4204712577 wait complete. 2024-03-18T19:06:28.260 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 42 2024-03-18T19:06:28.271 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:06:28.341 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:28.345+0000 7f584ce72700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:28.341 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:28.345+0000 7f584ce72700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:28.342 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:28.345+0000 7f584ce72700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:28.342 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:28.345+0000 7f584ce72700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:28.342 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:28.345+0000 7f584ce72700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:28.342 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:28.346+0000 7f584a66d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:28.342 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:28.346+0000 7f584a66d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:28.342 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:28.346+0000 7f584a66d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:28.342 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:28.346+0000 7f584a66d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:28.342 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:28.346+0000 7f584a66d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:28.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.412+0000 7f1468597700 1 Processor -- start 2024-03-18T19:06:28.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.412+0000 7f1468597700 1 -- start start 2024-03-18T19:06:28.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.413+0000 7f1468597700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f146015a320 0x7f14600ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:28.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.413+0000 7f1468597700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1460071d10 0x7f146006e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:28.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.413+0000 7f1468597700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14600acd80 con 0x7f14600cea30 2024-03-18T19:06:28.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.413+0000 7f1468597700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14600cf220 con 0x7f1460071d10 2024-03-18T19:06:28.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.413+0000 7f1468597700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14600cef40 con 0x7f146015a320 2024-03-18T19:06:28.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.413+0000 7f1466b34700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1460071d10 0x7f146006e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:28.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.413+0000 7f1465b32700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f14600cea30 0x7f1460071600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52424/0 (socket says 172.21.15.1:52424) 2024-03-18T19:06:28.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.413+0000 7f1465b32700 1 -- 172.21.15.1:0/1413163382 learned_addr learned my addr 172.21.15.1:0/1413163382 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:28.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.413+0000 7f1466333700 1 --2- 172.21.15.1:0/1413163382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f146015a320 0x7f14600ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:28.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.413+0000 7f1466b34700 1 -- 172.21.15.1:0/1413163382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f146015a320 msgr2=0x7f14600ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:28.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.413+0000 7f1466b34700 1 --2- 172.21.15.1:0/1413163382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f146015a320 0x7f14600ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.413+0000 7f1466b34700 1 -- 172.21.15.1:0/1413163382 >> v1:172.21.15.1:6789/0 conn(0x7f14600cea30 legacy=0x7f1460071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:28.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.413+0000 7f1466b34700 1 -- 172.21.15.1:0/1413163382 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146006d7e0 con 0x7f1460071d10 2024-03-18T19:06:28.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.413+0000 7f1465331700 1 -- 172.21.15.1:0/1413163382 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430926434 0 0) 0x7f14600acd80 con 0x7f14600cea30 2024-03-18T19:06:28.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.413+0000 7f1466b34700 1 --2- 172.21.15.1:0/1413163382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1460071d10 0x7f146006e2c0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f1458009d90 tx=0x7f145800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd8ae8ecdf8c246 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:28.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.413+0000 7f1465331700 1 -- 172.21.15.1:0/1413163382 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145800e070 con 0x7f1460071d10 2024-03-18T19:06:28.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.414+0000 7f1465331700 1 -- 172.21.15.1:0/1413163382 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145800bb20 con 0x7f1460071d10 2024-03-18T19:06:28.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.414+0000 7f1465331700 1 -- 172.21.15.1:0/1413163382 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458012430 con 0x7f1460071d10 2024-03-18T19:06:28.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.414+0000 7f1468597700 1 -- 172.21.15.1:0/1413163382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1460071d10 msgr2=0x7f146006e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:28.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.414+0000 7f1468597700 1 --2- 172.21.15.1:0/1413163382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1460071d10 0x7f146006e2c0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f1458009d90 tx=0x7f145800b330 comp rx=0 tx=0).stop 2024-03-18T19:06:28.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.414+0000 7f1468597700 1 -- 172.21.15.1:0/1413163382 shutdown_connections 2024-03-18T19:06:28.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.414+0000 7f1468597700 1 --2- 172.21.15.1:0/1413163382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1460071d10 0x7f146006e2c0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.414+0000 7f1468597700 1 --2- 172.21.15.1:0/1413163382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f146015a320 0x7f14600ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.414+0000 7f1468597700 1 -- 172.21.15.1:0/1413163382 >> 172.21.15.1:0/1413163382 conn(0x7f14600cc990 msgr2=0x7f14600ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:28.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.414+0000 7f1468597700 1 -- 172.21.15.1:0/1413163382 shutdown_connections 2024-03-18T19:06:28.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.414+0000 7f1468597700 1 -- 172.21.15.1:0/1413163382 wait complete. 2024-03-18T19:06:28.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.416+0000 7f1468597700 1 Processor -- start 2024-03-18T19:06:28.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.416+0000 7f1468597700 1 -- start start 2024-03-18T19:06:28.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1468597700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1460071d10 0x7f14600ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:28.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1468597700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14600cea30 0x7f146006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:28.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1468597700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f146006f200 0x7f1460145d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:28.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1468597700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146015c8a0 con 0x7f1460071d10 2024-03-18T19:06:28.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1468597700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146015cb80 con 0x7f14600cea30 2024-03-18T19:06:28.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1468597700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f146015ce60 con 0x7f146006f200 2024-03-18T19:06:28.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1466333700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1460071d10 0x7f14600ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:28.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1466333700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1460071d10 0x7f14600ae040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54186/0 (socket says 172.21.15.1:54186) 2024-03-18T19:06:28.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1466333700 1 -- 172.21.15.1:0/126963917 learned_addr learned my addr 172.21.15.1:0/126963917 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:28.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1466b34700 1 --2- 172.21.15.1:0/126963917 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f146006f200 0x7f1460145d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:28.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1465b32700 1 --2- 172.21.15.1:0/126963917 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14600cea30 0x7f146006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:28.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1466333700 1 -- 172.21.15.1:0/126963917 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f146006f200 msgr2=0x7f1460145d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:28.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1466333700 1 --2- 172.21.15.1:0/126963917 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f146006f200 0x7f1460145d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1466333700 1 -- 172.21.15.1:0/126963917 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14600cea30 msgr2=0x7f146006ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:28.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1466333700 1 --2- 172.21.15.1:0/126963917 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14600cea30 0x7f146006ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1466333700 1 -- 172.21.15.1:0/126963917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1458009a70 con 0x7f1460071d10 2024-03-18T19:06:28.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1466333700 1 --2- 172.21.15.1:0/126963917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1460071d10 0x7f14600ae040 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f145c00e9b0 tx=0x7f145c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:28.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1456ffd700 1 -- 172.21.15.1:0/126963917 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c009710 con 0x7f1460071d10 2024-03-18T19:06:28.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1456ffd700 1 -- 172.21.15.1:0/126963917 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145c00ee50 con 0x7f1460071d10 2024-03-18T19:06:28.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1456ffd700 1 -- 172.21.15.1:0/126963917 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c00ca50 con 0x7f1460071d10 2024-03-18T19:06:28.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.417+0000 7f1468597700 1 -- 172.21.15.1:0/126963917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f146015d140 con 0x7f1460071d10 2024-03-18T19:06:28.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.418+0000 7f1468597700 1 -- 172.21.15.1:0/126963917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14600ae580 con 0x7f1460071d10 2024-03-18T19:06:28.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.418+0000 7f1468597700 1 -- 172.21.15.1:0/126963917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f146015d4b0 con 0x7f1460071d10 2024-03-18T19:06:28.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.419+0000 7f1456ffd700 1 -- 172.21.15.1:0/126963917 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 46) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f145c00cbb0 con 0x7f1460071d10 2024-03-18T19:06:28.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.419+0000 7f1456ffd700 1 -- 172.21.15.1:0/126963917 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6631+0+0 (secure 0 0 0) 0x7f145c0891e0 con 0x7f1460071d10 2024-03-18T19:06:28.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.424+0000 7f1456ffd700 1 -- 172.21.15.1:0/126963917 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f145c054380 con 0x7f1460071d10 2024-03-18T19:06:28.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.625+0000 7f1468597700 1 -- 172.21.15.1:0/126963917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f146004ea80 con 0x7f1460071d10 2024-03-18T19:06:28.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.626+0000 7f1456ffd700 1 -- 172.21.15.1:0/126963917 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+17120 (secure 0 0 0) 0x7f145c00ce60 con 0x7f1460071d10 2024-03-18T19:06:28.623 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:28.623 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":105,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:06:28.276829+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:06:02.855989+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-18T19:06:04.121199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2460350233":"2024-03-19T19:06:28.276776+0000","172.21.15.1:0/2095460540":"2024-03-19T19:06:28.276776+0000","172.21.15.1:0/4057040655":"2024-03-19T19:06:28.276776+0000","172.21.15.1:0/1060785910":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/3060583633":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/3033255741":"2024-03-19T19:06:28.276776+0000","172.21.15.1:6833/76998":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/3832779205":"2024-03-19T19:06:28.276776+0000","172.21.15.1:0/3277934237":"2024-03-19T19:06:28.276776+0000","172.21.15.1:6832/76998":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/809308784":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/1836399926":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/3606402343":"2024-03-19T19:06:20.400777+0000","172.21.15.1:0/2494839321":"2024-03-19T19:06:20.400777+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:06:28.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.629+0000 7f1468597700 1 -- 172.21.15.1:0/126963917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1460071d10 msgr2=0x7f14600ae040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:28.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.629+0000 7f1468597700 1 --2- 172.21.15.1:0/126963917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1460071d10 0x7f14600ae040 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f145c00e9b0 tx=0x7f145c00ecc0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.629+0000 7f1468597700 1 -- 172.21.15.1:0/126963917 shutdown_connections 2024-03-18T19:06:28.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.629+0000 7f1468597700 1 --2- 172.21.15.1:0/126963917 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f146006f200 0x7f1460145d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.629+0000 7f1468597700 1 --2- 172.21.15.1:0/126963917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1460071d10 0x7f14600ae040 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.629+0000 7f1468597700 1 --2- 172.21.15.1:0/126963917 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14600cea30 0x7f146006ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.629+0000 7f1468597700 1 -- 172.21.15.1:0/126963917 >> 172.21.15.1:0/126963917 conn(0x7f14600cc990 msgr2=0x7f14600cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:28.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.629+0000 7f1468597700 1 -- 172.21.15.1:0/126963917 shutdown_connections 2024-03-18T19:06:28.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.629+0000 7f1468597700 1 -- 172.21.15.1:0/126963917 wait complete. 2024-03-18T19:06:28.637 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T19:06:28.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.767+0000 7f25912b4700 1 Processor -- start 2024-03-18T19:06:28.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.768+0000 7f25912b4700 1 -- start start 2024-03-18T19:06:28.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.768+0000 7f25912b4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f258c15d5a0 0x7f258c0ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:28.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.768+0000 7f25912b4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f258c0ce550 0x7f258c06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:28.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.768+0000 7f25912b4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f258c0cfa90 0x7f258c06ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:28.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.768+0000 7f25912b4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f258c0cf600 con 0x7f258c0cfa90 2024-03-18T19:06:28.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.768+0000 7f25912b4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f258c0ce9f0 con 0x7f258c0ce550 2024-03-18T19:06:28.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.768+0000 7f25912b4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f258c0cecd0 con 0x7f258c15d5a0 2024-03-18T19:06:28.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.769+0000 7f258b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f258c0cfa90 0x7f258c06ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:28.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.769+0000 7f258b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f258c0cfa90 0x7f258c06ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54196/0 (socket says 172.21.15.1:54196) 2024-03-18T19:06:28.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.769+0000 7f258b7fe700 1 -- 172.21.15.1:0/1479338031 learned_addr learned my addr 172.21.15.1:0/1479338031 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:28.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.769+0000 7f258a7fc700 1 --2- 172.21.15.1:0/1479338031 >> v2:172.21.15.165:3300/0 conn(0x7f258c0ce550 0x7f258c06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:28.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.769+0000 7f258affd700 1 --2- 172.21.15.1:0/1479338031 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f258c15d5a0 0x7f258c0ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:28.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.769+0000 7f258b7fe700 1 -- 172.21.15.1:0/1479338031 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f258c15d5a0 msgr2=0x7f258c0ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:28.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.769+0000 7f258b7fe700 1 --2- 172.21.15.1:0/1479338031 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f258c15d5a0 0x7f258c0ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.769+0000 7f258b7fe700 1 -- 172.21.15.1:0/1479338031 >> v2:172.21.15.165:3300/0 conn(0x7f258c0ce550 msgr2=0x7f258c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:28.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.769+0000 7f258b7fe700 1 --2- 172.21.15.1:0/1479338031 >> v2:172.21.15.165:3300/0 conn(0x7f258c0ce550 0x7f258c06a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.769+0000 7f258b7fe700 1 -- 172.21.15.1:0/1479338031 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258c0cefb0 con 0x7f258c0cfa90 2024-03-18T19:06:28.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.769+0000 7f258b7fe700 1 --2- 172.21.15.1:0/1479338031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f258c0cfa90 0x7f258c06ac90 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f258c06ce60 tx=0x7f258400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ff8fd6787e1489 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:28.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.769+0000 7f2589ffb700 1 -- 172.21.15.1:0/1479338031 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258400e070 con 0x7f258c0cfa90 2024-03-18T19:06:28.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.769+0000 7f2589ffb700 1 -- 172.21.15.1:0/1479338031 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258400b9c0 con 0x7f258c0cfa90 2024-03-18T19:06:28.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.769+0000 7f2589ffb700 1 -- 172.21.15.1:0/1479338031 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258400bd70 con 0x7f258c0cfa90 2024-03-18T19:06:28.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.770+0000 7f25912b4700 1 -- 172.21.15.1:0/1479338031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f258c0cfa90 msgr2=0x7f258c06ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:28.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.770+0000 7f25912b4700 1 --2- 172.21.15.1:0/1479338031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f258c0cfa90 0x7f258c06ac90 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f258c06ce60 tx=0x7f258400b330 comp rx=0 tx=0).stop 2024-03-18T19:06:28.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.770+0000 7f25912b4700 1 -- 172.21.15.1:0/1479338031 shutdown_connections 2024-03-18T19:06:28.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.770+0000 7f25912b4700 1 --2- 172.21.15.1:0/1479338031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f258c0cfa90 0x7f258c06ac90 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.770+0000 7f25912b4700 1 --2- 172.21.15.1:0/1479338031 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f258c15d5a0 0x7f258c0ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.770+0000 7f25912b4700 1 --2- 172.21.15.1:0/1479338031 >> v2:172.21.15.165:3300/0 conn(0x7f258c0ce550 0x7f258c06a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.770+0000 7f25912b4700 1 -- 172.21.15.1:0/1479338031 >> 172.21.15.1:0/1479338031 conn(0x7f258c0cc9b0 msgr2=0x7f258c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:28.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.770+0000 7f25912b4700 1 -- 172.21.15.1:0/1479338031 shutdown_connections 2024-03-18T19:06:28.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.770+0000 7f25912b4700 1 -- 172.21.15.1:0/1479338031 wait complete. 2024-03-18T19:06:28.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.771+0000 7f25912b4700 1 Processor -- start 2024-03-18T19:06:28.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.771+0000 7f25912b4700 1 -- start start 2024-03-18T19:06:28.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.772+0000 7f25912b4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f258c0ce550 0x7f258c15f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:28.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.772+0000 7f25912b4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f258c0cfa90 0x7f258c157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:28.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.772+0000 7f25912b4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f258c15d5a0 0x7f258c0b5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:28.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.772+0000 7f25912b4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258c15fb10 con 0x7f258c0cfa90 2024-03-18T19:06:28.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.772+0000 7f25912b4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258c15fdf0 con 0x7f258c0ce550 2024-03-18T19:06:28.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.772+0000 7f25912b4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f258c1600d0 con 0x7f258c15d5a0 2024-03-18T19:06:28.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.772+0000 7f258a7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f258c0cfa90 0x7f258c157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:28.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.772+0000 7f258a7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f258c0cfa90 0x7f258c157a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54210/0 (socket says 172.21.15.1:54210) 2024-03-18T19:06:28.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.772+0000 7f258a7fc700 1 -- 172.21.15.1:0/606245561 learned_addr learned my addr 172.21.15.1:0/606245561 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:28.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.772+0000 7f258affd700 1 --2- 172.21.15.1:0/606245561 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f258c0ce550 0x7f258c15f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:28.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.772+0000 7f258b7fe700 1 --2- 172.21.15.1:0/606245561 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f258c15d5a0 0x7f258c0b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:28.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.772+0000 7f258a7fc700 1 -- 172.21.15.1:0/606245561 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f258c15d5a0 msgr2=0x7f258c0b5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:28.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.772+0000 7f258a7fc700 1 --2- 172.21.15.1:0/606245561 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f258c15d5a0 0x7f258c0b5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.772+0000 7f258a7fc700 1 -- 172.21.15.1:0/606245561 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f258c0ce550 msgr2=0x7f258c15f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:28.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.772+0000 7f258a7fc700 1 --2- 172.21.15.1:0/606245561 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f258c0ce550 0x7f258c15f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:28.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.772+0000 7f258a7fc700 1 -- 172.21.15.1:0/606245561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2584009b30 con 0x7f258c0cfa90 2024-03-18T19:06:28.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.772+0000 7f258a7fc700 1 --2- 172.21.15.1:0/606245561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f258c0cfa90 0x7f258c157a20 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f257c00d800 tx=0x7f257c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:28.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.773+0000 7f25737fe700 1 -- 172.21.15.1:0/606245561 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c0125f0 con 0x7f258c0cfa90 2024-03-18T19:06:28.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.773+0000 7f25737fe700 1 -- 172.21.15.1:0/606245561 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257c012750 con 0x7f258c0cfa90 2024-03-18T19:06:28.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.773+0000 7f25912b4700 1 -- 172.21.15.1:0/606245561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f258c1603b0 con 0x7f258c0cfa90 2024-03-18T19:06:28.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.773+0000 7f25737fe700 1 -- 172.21.15.1:0/606245561 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c012ae0 con 0x7f258c0cfa90 2024-03-18T19:06:28.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.773+0000 7f25912b4700 1 -- 172.21.15.1:0/606245561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f258c160bb0 con 0x7f258c0cfa90 2024-03-18T19:06:28.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.774+0000 7f25912b4700 1 -- 172.21.15.1:0/606245561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f258c0676a0 con 0x7f258c0cfa90 2024-03-18T19:06:28.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.774+0000 7f25737fe700 1 -- 172.21.15.1:0/606245561 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 46) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f257c012c40 con 0x7f258c0cfa90 2024-03-18T19:06:28.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.775+0000 7f25737fe700 1 -- 172.21.15.1:0/606245561 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6631+0+0 (secure 0 0 0) 0x7f257c0890b0 con 0x7f258c0cfa90 2024-03-18T19:06:28.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:28.779+0000 7f25737fe700 1 -- 172.21.15.1:0/606245561 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f257c0543f0 con 0x7f258c0cfa90 2024-03-18T19:06:29.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.031+0000 7f25912b4700 1 -- 172.21.15.1:0/606245561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f258c159ad0 con 0x7f258c0cfa90 2024-03-18T19:06:29.027 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-18T19:06:29.031+0000 7fbdbeb46700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T19:06:29.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.042+0000 7f25737fe700 1 -- 172.21.15.1:0/606245561 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) v1 ==== 161+0+0 (secure 0 0 0) 0x7f257c057a10 con 0x7f258c0cfa90 2024-03-18T19:06:29.039 INFO:tasks.ceph.mds.a.smithi001.stderr: -1493> 2024-03-18T19:06:10.703+0000 7f42ae993700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T19:06:29.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.044+0000 7f25912b4700 1 -- 172.21.15.1:0/606245561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f258c0cfa90 msgr2=0x7f258c157a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:29.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.044+0000 7f25912b4700 1 --2- 172.21.15.1:0/606245561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f258c0cfa90 0x7f258c157a20 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f257c00d800 tx=0x7f257c00db10 comp rx=0 tx=0).stop 2024-03-18T19:06:29.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.044+0000 7f25912b4700 1 -- 172.21.15.1:0/606245561 shutdown_connections 2024-03-18T19:06:29.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.044+0000 7f25912b4700 1 --2- 172.21.15.1:0/606245561 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f258c15d5a0 0x7f258c0b5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:29.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.044+0000 7f25912b4700 1 --2- 172.21.15.1:0/606245561 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f258c0ce550 0x7f258c15f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:29.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.044+0000 7f25912b4700 1 --2- 172.21.15.1:0/606245561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f258c0cfa90 0x7f258c157a20 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:29.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.044+0000 7f25912b4700 1 -- 172.21.15.1:0/606245561 >> 172.21.15.1:0/606245561 conn(0x7f258c0cc9b0 msgr2=0x7f258c06e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:29.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.044+0000 7f25912b4700 1 -- 172.21.15.1:0/606245561 shutdown_connections 2024-03-18T19:06:29.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.044+0000 7f25912b4700 1 -- 172.21.15.1:0/606245561 wait complete. 2024-03-18T19:06:29.041 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T19:06:29.053 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T19:06:29.064 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-18T19:06:29.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.187+0000 7f460f028700 1 Processor -- start 2024-03-18T19:06:29.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.187+0000 7f460f028700 1 -- start start 2024-03-18T19:06:29.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.188+0000 7f460f028700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f460806d560 0x7f46080aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:29.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.188+0000 7f460f028700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46080af510 0x7f460815b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:29.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.188+0000 7f460f028700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460815ef00 con 0x7f460806a750 2024-03-18T19:06:29.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.188+0000 7f460f028700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f460815f1e0 con 0x7f460806d560 2024-03-18T19:06:29.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.188+0000 7f460f028700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f460815f4c0 con 0x7f46080af510 2024-03-18T19:06:29.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.188+0000 7f4607fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f460806d560 0x7f46080aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:29.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.188+0000 7f4607fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f460806d560 0x7f46080aefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54242/0 (socket says 172.21.15.1:54242) 2024-03-18T19:06:29.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.188+0000 7f4607fff700 1 -- 172.21.15.1:0/2524926517 learned_addr learned my addr 172.21.15.1:0/2524926517 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:29.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.188+0000 7f460d5c5700 1 --2- 172.21.15.1:0/2524926517 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46080af510 0x7f460815b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:29.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.188+0000 7f4607fff700 1 -- 172.21.15.1:0/2524926517 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46080af510 msgr2=0x7f460815b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:29.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.188+0000 7f4607fff700 1 --2- 172.21.15.1:0/2524926517 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46080af510 0x7f460815b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:29.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.188+0000 7f4607fff700 1 -- 172.21.15.1:0/2524926517 >> v1:172.21.15.165:6789/0 conn(0x7f460806a750 legacy=0x7f460806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:29.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.188+0000 7f4607fff700 1 -- 172.21.15.1:0/2524926517 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460815f7a0 con 0x7f460806d560 2024-03-18T19:06:29.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.188+0000 7f4607fff700 1 --2- 172.21.15.1:0/2524926517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f460806d560 0x7f46080aefd0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f45f40098f0 tx=0x7f45f4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965eacdb69d4afbc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:29.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.189+0000 7f46077fe700 1 -- 172.21.15.1:0/2524926517 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f4011830 con 0x7f460806d560 2024-03-18T19:06:29.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.189+0000 7f46077fe700 1 -- 172.21.15.1:0/2524926517 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45f4011990 con 0x7f460806d560 2024-03-18T19:06:29.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.189+0000 7f46077fe700 1 -- 172.21.15.1:0/2524926517 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f4011d20 con 0x7f460806d560 2024-03-18T19:06:29.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.189+0000 7f460f028700 1 -- 172.21.15.1:0/2524926517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f460806d560 msgr2=0x7f46080aefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:29.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.189+0000 7f460f028700 1 --2- 172.21.15.1:0/2524926517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f460806d560 0x7f46080aefd0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f45f40098f0 tx=0x7f45f4009c00 comp rx=0 tx=0).stop 2024-03-18T19:06:29.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.189+0000 7f460f028700 1 -- 172.21.15.1:0/2524926517 shutdown_connections 2024-03-18T19:06:29.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.189+0000 7f460f028700 1 --2- 172.21.15.1:0/2524926517 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46080af510 0x7f460815b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:29.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.189+0000 7f460f028700 1 --2- 172.21.15.1:0/2524926517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f460806d560 0x7f46080aefd0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:29.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.189+0000 7f460f028700 1 -- 172.21.15.1:0/2524926517 >> 172.21.15.1:0/2524926517 conn(0x7f46080cff30 msgr2=0x7f46080d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:29.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.189+0000 7f460f028700 1 -- 172.21.15.1:0/2524926517 shutdown_connections 2024-03-18T19:06:29.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.189+0000 7f460f028700 1 -- 172.21.15.1:0/2524926517 wait complete. 2024-03-18T19:06:29.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.191+0000 7f460f028700 1 Processor -- start 2024-03-18T19:06:29.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.191+0000 7f460f028700 1 -- start start 2024-03-18T19:06:29.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.192+0000 7f460f028700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f460806a750 0x7f46081493d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:29.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.192+0000 7f460f028700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f460806d560 0x7f46080b1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:29.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.192+0000 7f460f028700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46080af510 0x7f46080b4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:29.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.192+0000 7f460f028700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46080b2fc0 con 0x7f46080af510 2024-03-18T19:06:29.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.192+0000 7f460f028700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46080b32a0 con 0x7f460806a750 2024-03-18T19:06:29.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.192+0000 7f460f028700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46080b3580 con 0x7f460806d560 2024-03-18T19:06:29.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.192+0000 7f460d5c5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46080af510 0x7f46080b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:29.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.192+0000 7f4607fff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f460806d560 0x7f46080b1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:29.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.192+0000 7f460d5c5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46080af510 0x7f46080b4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54246/0 (socket says 172.21.15.1:54246) 2024-03-18T19:06:29.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.192+0000 7f460d5c5700 1 -- 172.21.15.1:0/182341762 learned_addr learned my addr 172.21.15.1:0/182341762 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:29.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.192+0000 7f460cdc4700 1 --2- 172.21.15.1:0/182341762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f460806a750 0x7f46081493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:29.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.192+0000 7f460d5c5700 1 -- 172.21.15.1:0/182341762 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f460806d560 msgr2=0x7f46080b1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:29.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.192+0000 7f460d5c5700 1 --2- 172.21.15.1:0/182341762 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f460806d560 0x7f46080b1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:29.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.192+0000 7f460d5c5700 1 -- 172.21.15.1:0/182341762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f460806a750 msgr2=0x7f46081493d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:29.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.192+0000 7f460d5c5700 1 --2- 172.21.15.1:0/182341762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f460806a750 0x7f46081493d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:29.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.192+0000 7f460d5c5700 1 -- 172.21.15.1:0/182341762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f4009550 con 0x7f46080af510 2024-03-18T19:06:29.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.193+0000 7f460d5c5700 1 --2- 172.21.15.1:0/182341762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46080af510 0x7f46080b4060 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f460000ebe0 tx=0x7f460000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:29.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.193+0000 7f46057fa700 1 -- 172.21.15.1:0/182341762 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460000ca00 con 0x7f46080af510 2024-03-18T19:06:29.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.193+0000 7f46057fa700 1 -- 172.21.15.1:0/182341762 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460000f040 con 0x7f46080af510 2024-03-18T19:06:29.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.193+0000 7f460f028700 1 -- 172.21.15.1:0/182341762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46080b3860 con 0x7f46080af510 2024-03-18T19:06:29.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.193+0000 7f46057fa700 1 -- 172.21.15.1:0/182341762 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460000cdc0 con 0x7f46080af510 2024-03-18T19:06:29.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.193+0000 7f460f028700 1 -- 172.21.15.1:0/182341762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4608156590 con 0x7f46080af510 2024-03-18T19:06:29.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.194+0000 7f460f028700 1 -- 172.21.15.1:0/182341762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4608156780 con 0x7f46080af510 2024-03-18T19:06:29.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.194+0000 7f46057fa700 1 -- 172.21.15.1:0/182341762 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 46) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f4600014480 con 0x7f46080af510 2024-03-18T19:06:29.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.195+0000 7f46057fa700 1 -- 172.21.15.1:0/182341762 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6717+0+0 (secure 0 0 0) 0x7f4600089b60 con 0x7f46080af510 2024-03-18T19:06:29.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.199+0000 7f46057fa700 1 -- 172.21.15.1:0/182341762 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4608156780 con 0x7f46080af510 2024-03-18T19:06:29.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.286+0000 7f46057fa700 1 -- 172.21.15.1:0/182341762 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 47) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f4600054850 con 0x7f46080af510 2024-03-18T19:06:29.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.287+0000 7f46057fa700 1 --2- 172.21.15.1:0/182341762 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f45ec036150 0x7f45ec0728f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:29.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.287+0000 7f460cdc4700 1 --2- 172.21.15.1:0/182341762 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f45ec036150 0x7f45ec0728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:29.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.288+0000 7f460cdc4700 1 --2- 172.21.15.1:0/182341762 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f45ec036150 0x7f45ec0728f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f45fc007fd0 tx=0x7f45fc009040 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:29.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:29.456+0000 7f460f028700 1 -- 172.21.15.1:0/182341762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f46080b3d80 con 0x7f46080af510 2024-03-18T19:06:30.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.044+0000 7f46057fa700 1 -- 172.21.15.1:0/182341762 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7f46080b3d80 con 0x7f46080af510 2024-03-18T19:06:30.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.046+0000 7f460f028700 1 -- 172.21.15.1:0/182341762 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f45ec036150 msgr2=0x7f45ec0728f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:30.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.046+0000 7f460f028700 1 --2- 172.21.15.1:0/182341762 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f45ec036150 0x7f45ec0728f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f45fc007fd0 tx=0x7f45fc009040 comp rx=0 tx=0).stop 2024-03-18T19:06:30.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.046+0000 7f460f028700 1 -- 172.21.15.1:0/182341762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46080af510 msgr2=0x7f46080b4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:30.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.046+0000 7f460f028700 1 --2- 172.21.15.1:0/182341762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46080af510 0x7f46080b4060 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f460000ebe0 tx=0x7f460000eef0 comp rx=0 tx=0).stop 2024-03-18T19:06:30.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.047+0000 7f460f028700 1 -- 172.21.15.1:0/182341762 shutdown_connections 2024-03-18T19:06:30.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.047+0000 7f460f028700 1 --2- 172.21.15.1:0/182341762 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f45ec036150 0x7f45ec0728f0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:30.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.047+0000 7f460f028700 1 --2- 172.21.15.1:0/182341762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46080af510 0x7f46080b4060 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:30.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.047+0000 7f460f028700 1 --2- 172.21.15.1:0/182341762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f460806a750 0x7f46081493d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:30.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.047+0000 7f460f028700 1 --2- 172.21.15.1:0/182341762 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f460806d560 0x7f46080b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:30.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.047+0000 7f460f028700 1 -- 172.21.15.1:0/182341762 >> 172.21.15.1:0/182341762 conn(0x7f46080cff30 msgr2=0x7f460806f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:30.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.047+0000 7f460f028700 1 -- 172.21.15.1:0/182341762 shutdown_connections 2024-03-18T19:06:30.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.047+0000 7f460f028700 1 -- 172.21.15.1:0/182341762 wait complete. 2024-03-18T19:06:30.058 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T19:06:30.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.238+0000 7fc370ce9700 1 Processor -- start 2024-03-18T19:06:30.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.238+0000 7fc370ce9700 1 -- start start 2024-03-18T19:06:30.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.239+0000 7fc370ce9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc36c070920 0x7fc36c06bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:30.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.239+0000 7fc370ce9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc36c06c0f0 0x7fc36c15e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:30.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.239+0000 7fc370ce9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc36c161390 con 0x7fc36c146d70 2024-03-18T19:06:30.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.239+0000 7fc370ce9700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc36c06c580 con 0x7fc36c06c0f0 2024-03-18T19:06:30.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.239+0000 7fc370ce9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c15dc60 con 0x7fc36c070920 2024-03-18T19:06:30.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.239+0000 7fc36ad9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc36c06c0f0 0x7fc36c15e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:30.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.239+0000 7fc36a59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc36c146d70 0x7fc36c070210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52434/0 (socket says 172.21.15.1:52434) 2024-03-18T19:06:30.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.239+0000 7fc36a59c700 1 -- 172.21.15.1:0/1447253916 learned_addr learned my addr 172.21.15.1:0/1447253916 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:30.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.239+0000 7fc369d9b700 1 --2- 172.21.15.1:0/1447253916 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc36c070920 0x7fc36c06bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:30.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.240+0000 7fc36959a700 1 -- 172.21.15.1:0/1447253916 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507992613 0 0) 0x7fc354001110 con 0x7fc36c146d70 2024-03-18T19:06:30.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.240+0000 7fc36ad9d700 1 -- 172.21.15.1:0/1447253916 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc36c070920 msgr2=0x7fc36c06bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:30.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.240+0000 7fc36ad9d700 1 --2- 172.21.15.1:0/1447253916 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc36c070920 0x7fc36c06bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:30.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.240+0000 7fc36ad9d700 1 -- 172.21.15.1:0/1447253916 >> v1:172.21.15.1:6789/0 conn(0x7fc36c146d70 legacy=0x7fc36c070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:30.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.240+0000 7fc36ad9d700 1 -- 172.21.15.1:0/1447253916 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c06d030 con 0x7fc36c06c0f0 2024-03-18T19:06:30.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.240+0000 7fc36ad9d700 1 --2- 172.21.15.1:0/1447253916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc36c06c0f0 0x7fc36c15e0f0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fc364009d90 tx=0x7fc36400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bacfa0a03f4c986 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:30.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.240+0000 7fc36959a700 1 -- 172.21.15.1:0/1447253916 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36400e070 con 0x7fc36c06c0f0 2024-03-18T19:06:30.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.240+0000 7fc36959a700 1 -- 172.21.15.1:0/1447253916 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36400bb20 con 0x7fc36c06c0f0 2024-03-18T19:06:30.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.240+0000 7fc36959a700 1 -- 172.21.15.1:0/1447253916 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc364012430 con 0x7fc36c06c0f0 2024-03-18T19:06:30.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.240+0000 7fc370ce9700 1 -- 172.21.15.1:0/1447253916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc36c06c0f0 msgr2=0x7fc36c15e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:30.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.240+0000 7fc370ce9700 1 --2- 172.21.15.1:0/1447253916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc36c06c0f0 0x7fc36c15e0f0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fc364009d90 tx=0x7fc36400b330 comp rx=0 tx=0).stop 2024-03-18T19:06:30.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.241+0000 7fc370ce9700 1 -- 172.21.15.1:0/1447253916 shutdown_connections 2024-03-18T19:06:30.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.241+0000 7fc370ce9700 1 --2- 172.21.15.1:0/1447253916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc36c06c0f0 0x7fc36c15e0f0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:30.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.241+0000 7fc370ce9700 1 --2- 172.21.15.1:0/1447253916 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc36c070920 0x7fc36c06bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:30.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.241+0000 7fc370ce9700 1 -- 172.21.15.1:0/1447253916 >> 172.21.15.1:0/1447253916 conn(0x7fc36c0b2e80 msgr2=0x7fc36c0b3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:30.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.241+0000 7fc370ce9700 1 -- 172.21.15.1:0/1447253916 shutdown_connections 2024-03-18T19:06:30.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.241+0000 7fc370ce9700 1 -- 172.21.15.1:0/1447253916 wait complete. 2024-03-18T19:06:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.242+0000 7fc370ce9700 1 Processor -- start 2024-03-18T19:06:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.242+0000 7fc370ce9700 1 -- start start 2024-03-18T19:06:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.243+0000 7fc370ce9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc36c06c0f0 0x7fc36c13bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.243+0000 7fc370ce9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc36c070920 0x7fc36c15bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.243+0000 7fc370ce9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc36c146d70 0x7fc36c1e8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.243+0000 7fc370ce9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c13f050 con 0x7fc36c070920 2024-03-18T19:06:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.243+0000 7fc370ce9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c13f580 con 0x7fc36c146d70 2024-03-18T19:06:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.243+0000 7fc370ce9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c13f2a0 con 0x7fc36c06c0f0 2024-03-18T19:06:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.243+0000 7fc369d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc36c070920 0x7fc36c15bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.243+0000 7fc369d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc36c070920 0x7fc36c15bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54268/0 (socket says 172.21.15.1:54268) 2024-03-18T19:06:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.243+0000 7fc369d9b700 1 -- 172.21.15.1:0/2545938532 learned_addr learned my addr 172.21.15.1:0/2545938532 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:30.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.244+0000 7fc369d9b700 1 -- 172.21.15.1:0/2545938532 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc36c06c0f0 msgr2=0x7fc36c13bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:30.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.244+0000 7fc369d9b700 1 --2- 172.21.15.1:0/2545938532 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc36c06c0f0 0x7fc36c13bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:30.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.244+0000 7fc369d9b700 1 -- 172.21.15.1:0/2545938532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc36c146d70 msgr2=0x7fc36c1e8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:30.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.244+0000 7fc369d9b700 1 --2- 172.21.15.1:0/2545938532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc36c146d70 0x7fc36c1e8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:30.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.244+0000 7fc369d9b700 1 -- 172.21.15.1:0/2545938532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc364009a70 con 0x7fc36c070920 2024-03-18T19:06:30.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.244+0000 7fc369d9b700 1 --2- 172.21.15.1:0/2545938532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc36c070920 0x7fc36c15bf50 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fc35c00e9e0 tx=0x7fc35c00eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:30.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.244+0000 7fc362ffd700 1 -- 172.21.15.1:0/2545938532 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c009710 con 0x7fc36c070920 2024-03-18T19:06:30.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.244+0000 7fc362ffd700 1 -- 172.21.15.1:0/2545938532 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35c00c850 con 0x7fc36c070920 2024-03-18T19:06:30.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.244+0000 7fc370ce9700 1 -- 172.21.15.1:0/2545938532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36c1e7f40 con 0x7fc36c070920 2024-03-18T19:06:30.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.244+0000 7fc362ffd700 1 -- 172.21.15.1:0/2545938532 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c00cbc0 con 0x7fc36c070920 2024-03-18T19:06:30.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.244+0000 7fc370ce9700 1 -- 172.21.15.1:0/2545938532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc36c1e8500 con 0x7fc36c070920 2024-03-18T19:06:30.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.245+0000 7fc370ce9700 1 -- 172.21.15.1:0/2545938532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36c1e7da0 con 0x7fc36c070920 2024-03-18T19:06:30.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.246+0000 7fc362ffd700 1 -- 172.21.15.1:0/2545938532 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 47) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fc35c00cd20 con 0x7fc36c070920 2024-03-18T19:06:30.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.246+0000 7fc362ffd700 1 --2- 172.21.15.1:0/2545938532 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc34c06aae0 0x7fc34c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:30.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.246+0000 7fc362ffd700 1 -- 172.21.15.1:0/2545938532 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6717+0+0 (secure 0 0 0) 0x7fc35c089420 con 0x7fc36c070920 2024-03-18T19:06:30.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.246+0000 7fc36a59c700 1 --2- 172.21.15.1:0/2545938532 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc34c06aae0 0x7fc34c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:30.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.246+0000 7fc36a59c700 1 --2- 172.21.15.1:0/2545938532 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc34c06aae0 0x7fc34c06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc3540016b0 tx=0x7fc354001310 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:30.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.251+0000 7fc362ffd700 1 -- 172.21.15.1:0/2545938532 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc36c1e7da0 con 0x7fc36c070920 2024-03-18T19:06:30.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:30.460+0000 7fc370ce9700 1 -- 172.21.15.1:0/2545938532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc36c1e7da0 con 0x7fc36c070920 2024-03-18T19:06:31.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.294+0000 7fc362ffd700 1 -- 172.21.15.1:0/2545938532 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 48) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fc35c089120 con 0x7fc36c070920 2024-03-18T19:06:31.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.434+0000 7fc362ffd700 1 -- 172.21.15.1:0/2545938532 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v107) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc35c086630 con 0x7fc36c070920 2024-03-18T19:06:31.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.437+0000 7fc370ce9700 1 -- 172.21.15.1:0/2545938532 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc34c06aae0 msgr2=0x7fc34c06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:31.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.437+0000 7fc370ce9700 1 --2- 172.21.15.1:0/2545938532 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc34c06aae0 0x7fc34c06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc3540016b0 tx=0x7fc354001310 comp rx=0 tx=0).stop 2024-03-18T19:06:31.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.437+0000 7fc370ce9700 1 -- 172.21.15.1:0/2545938532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc36c070920 msgr2=0x7fc36c15bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:31.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.437+0000 7fc370ce9700 1 --2- 172.21.15.1:0/2545938532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc36c070920 0x7fc36c15bf50 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fc35c00e9e0 tx=0x7fc35c00eda0 comp rx=0 tx=0).stop 2024-03-18T19:06:31.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.437+0000 7fc370ce9700 1 -- 172.21.15.1:0/2545938532 shutdown_connections 2024-03-18T19:06:31.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.437+0000 7fc370ce9700 1 --2- 172.21.15.1:0/2545938532 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc34c06aae0 0x7fc34c06cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:31.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.437+0000 7fc370ce9700 1 --2- 172.21.15.1:0/2545938532 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc36c06c0f0 0x7fc36c13bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:31.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.437+0000 7fc370ce9700 1 --2- 172.21.15.1:0/2545938532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc36c146d70 0x7fc36c1e8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:31.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.437+0000 7fc370ce9700 1 --2- 172.21.15.1:0/2545938532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc36c070920 0x7fc36c15bf50 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:31.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.437+0000 7fc370ce9700 1 -- 172.21.15.1:0/2545938532 >> 172.21.15.1:0/2545938532 conn(0x7fc36c0b2e80 msgr2=0x7fc36c15eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:31.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.437+0000 7fc370ce9700 1 -- 172.21.15.1:0/2545938532 shutdown_connections 2024-03-18T19:06:31.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.437+0000 7fc370ce9700 1 -- 172.21.15.1:0/2545938532 wait complete. 2024-03-18T19:06:31.434 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-18T19:06:31.448 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T19:06:31.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.589+0000 7f5341466700 1 Processor -- start 2024-03-18T19:06:31.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.589+0000 7f5341466700 1 -- start start 2024-03-18T19:06:31.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.589+0000 7f5341466700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f533c15d8b0 0x7f533c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:31.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.589+0000 7f5341466700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533c15dc90 con 0x7f533c0d1170 2024-03-18T19:06:31.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.589+0000 7f5341466700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533c15ded0 con 0x7f533c0d0040 2024-03-18T19:06:31.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.589+0000 7f5341466700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f533c15ec90 con 0x7f533c15d8b0 2024-03-18T19:06:31.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f533a7fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f533c0d1170 0x7f533c0cf930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52444/0 (socket says 172.21.15.1:52444) 2024-03-18T19:06:31.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f533a7fc700 1 -- 172.21.15.1:0/2959689404 learned_addr learned my addr 172.21.15.1:0/2959689404 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:31.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f533affd700 1 --2- 172.21.15.1:0/2959689404 >> v2:172.21.15.165:3300/0 conn(0x7f533c15d8b0 0x7f533c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:31.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f5339ffb700 1 -- 172.21.15.1:0/2959689404 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882784361 0 0) 0x7f5330001110 con 0x7f533c0d1170 2024-03-18T19:06:31.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f5339ffb700 1 -- 172.21.15.1:0/2959689404 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53240035b0 con 0x7f533c0d1170 2024-03-18T19:06:31.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f5339ffb700 1 -- 172.21.15.1:0/2959689404 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 545458020 0 0) 0x7f53240035b0 con 0x7f533c0d1170 2024-03-18T19:06:31.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f5339ffb700 1 -- 172.21.15.1:0/2959689404 >> v2:172.21.15.165:3300/0 conn(0x7f533c15d8b0 msgr2=0x7f533c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:31.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f5339ffb700 1 --2- 172.21.15.1:0/2959689404 >> v2:172.21.15.165:3300/0 conn(0x7f533c15d8b0 0x7f533c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:31.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f5339ffb700 1 -- 172.21.15.1:0/2959689404 >> v1:172.21.15.165:6789/0 conn(0x7f533c0d0040 legacy=0x7f533c0cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:31.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f5339ffb700 1 -- 172.21.15.1:0/2959689404 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533c15e1b0 con 0x7f533c0d1170 2024-03-18T19:06:31.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f5339ffb700 1 -- 172.21.15.1:0/2959689404 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f5330001800 con 0x7f533c0d1170 2024-03-18T19:06:31.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f5339ffb700 1 -- 172.21.15.1:0/2959689404 >> v1:172.21.15.1:6789/0 conn(0x7f533c0d1170 legacy=0x7f533c0cf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:31.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f5339ffb700 1 --2- 172.21.15.1:0/2959689404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5324003a00 0x7f5324003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:31.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f5339ffb700 1 --2- 172.21.15.1:0/2959689404 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5324004850 0x7f5324004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:31.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f5339ffb700 1 --2- 172.21.15.1:0/2959689404 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5324005210 0x7f532400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:31.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f5339ffb700 1 -- 172.21.15.1:0/2959689404 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533c15ec90 con 0x7f5324003a00 2024-03-18T19:06:31.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f5339ffb700 1 -- 172.21.15.1:0/2959689404 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532400c0c0 con 0x7f5324004850 2024-03-18T19:06:31.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f5339ffb700 1 -- 172.21.15.1:0/2959689404 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532400c280 con 0x7f5324005210 2024-03-18T19:06:31.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f533affd700 1 --2- 172.21.15.1:0/2959689404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5324003a00 0x7f5324003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:31.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.590+0000 7f533a7fc700 1 --2- 172.21.15.1:0/2959689404 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5324004850 0x7f5324004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:31.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.591+0000 7f533b7fe700 1 --2- 172.21.15.1:0/2959689404 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5324005210 0x7f532400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:31.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.591+0000 7f533affd700 1 -- 172.21.15.1:0/2959689404 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5324005210 msgr2=0x7f532400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:31.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.591+0000 7f533affd700 1 --2- 172.21.15.1:0/2959689404 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5324005210 0x7f532400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:31.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.591+0000 7f533affd700 1 -- 172.21.15.1:0/2959689404 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5324004850 msgr2=0x7f5324004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:31.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.591+0000 7f533affd700 1 --2- 172.21.15.1:0/2959689404 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5324004850 0x7f5324004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:31.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.591+0000 7f533b7fe700 1 -- 172.21.15.1:0/2959689404 reap_dead start 2024-03-18T19:06:31.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.591+0000 7f533affd700 1 -- 172.21.15.1:0/2959689404 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53240035b0 con 0x7f5324003a00 2024-03-18T19:06:31.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.591+0000 7f533affd700 1 --2- 172.21.15.1:0/2959689404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5324003a00 0x7f5324003e20 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f532c009b60 tx=0x7f5324005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3904a8da5dc15c72 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:31.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.591+0000 7f5339ffb700 1 -- 172.21.15.1:0/2959689404 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532c00e6c0 con 0x7f5324003a00 2024-03-18T19:06:31.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.591+0000 7f5339ffb700 1 -- 172.21.15.1:0/2959689404 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f532c00e820 con 0x7f5324003a00 2024-03-18T19:06:31.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.591+0000 7f5339ffb700 1 -- 172.21.15.1:0/2959689404 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532c00ebb0 con 0x7f5324003a00 2024-03-18T19:06:31.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.592+0000 7f5341466700 1 -- 172.21.15.1:0/2959689404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5324003a00 msgr2=0x7f5324003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:31.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.592+0000 7f5341466700 1 --2- 172.21.15.1:0/2959689404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5324003a00 0x7f5324003e20 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f532c009b60 tx=0x7f5324005880 comp rx=0 tx=0).stop 2024-03-18T19:06:31.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.592+0000 7f5341466700 1 -- 172.21.15.1:0/2959689404 shutdown_connections 2024-03-18T19:06:31.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.592+0000 7f5341466700 1 --2- 172.21.15.1:0/2959689404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5324003a00 0x7f5324003e20 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:31.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.592+0000 7f5341466700 1 -- 172.21.15.1:0/2959689404 >> 172.21.15.1:0/2959689404 conn(0x7f533c09c640 msgr2=0x7f533c0d3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:31.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.592+0000 7f5341466700 1 -- 172.21.15.1:0/2959689404 shutdown_connections 2024-03-18T19:06:31.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.592+0000 7f5341466700 1 -- 172.21.15.1:0/2959689404 wait complete. 2024-03-18T19:06:31.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.593+0000 7f5341466700 1 Processor -- start 2024-03-18T19:06:31.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.593+0000 7f5341466700 1 -- start start 2024-03-18T19:06:31.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.594+0000 7f5341466700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5324003a00 0x7f533c06c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:31.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.594+0000 7f5341466700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f533c0cb2a0 0x7f533c0cb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:31.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.594+0000 7f5341466700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f533c0c7e80 0x7f533c0c8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:31.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.594+0000 7f5341466700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533c15a040 con 0x7f533c0cb2a0 2024-03-18T19:06:31.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.594+0000 7f5341466700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533c0cbd40 con 0x7f5324003a00 2024-03-18T19:06:31.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.594+0000 7f5341466700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f533c0cc020 con 0x7f533c0c7e80 2024-03-18T19:06:31.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.594+0000 7f533a7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f533c0cb2a0 0x7f533c0cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:31.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.594+0000 7f533a7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f533c0cb2a0 0x7f533c0cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54304/0 (socket says 172.21.15.1:54304) 2024-03-18T19:06:31.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.594+0000 7f533a7fc700 1 -- 172.21.15.1:0/1369619904 learned_addr learned my addr 172.21.15.1:0/1369619904 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:31.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.594+0000 7f533affd700 1 --2- 172.21.15.1:0/1369619904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5324003a00 0x7f533c06c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:31.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.594+0000 7f533b7fe700 1 --2- 172.21.15.1:0/1369619904 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f533c0c7e80 0x7f533c0c8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:31.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.594+0000 7f533a7fc700 1 -- 172.21.15.1:0/1369619904 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f533c0c7e80 msgr2=0x7f533c0c8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:31.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.594+0000 7f533a7fc700 1 --2- 172.21.15.1:0/1369619904 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f533c0c7e80 0x7f533c0c8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:31.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.594+0000 7f533a7fc700 1 -- 172.21.15.1:0/1369619904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5324003a00 msgr2=0x7f533c06c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:31.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.594+0000 7f533a7fc700 1 --2- 172.21.15.1:0/1369619904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5324003a00 0x7f533c06c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:31.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.594+0000 7f533a7fc700 1 -- 172.21.15.1:0/1369619904 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533c0cc300 con 0x7f533c0cb2a0 2024-03-18T19:06:31.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.594+0000 7f533b7fe700 1 --2- 172.21.15.1:0/1369619904 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f533c0c7e80 0x7f533c0c8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:31.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.594+0000 7f533a7fc700 1 --2- 172.21.15.1:0/1369619904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f533c0cb2a0 0x7f533c0cb720 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f5330007060 tx=0x7f5330007370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:31.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.595+0000 7f53237fe700 1 -- 172.21.15.1:0/1369619904 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330011580 con 0x7f533c0cb2a0 2024-03-18T19:06:31.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.595+0000 7f53237fe700 1 -- 172.21.15.1:0/1369619904 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53300116e0 con 0x7f533c0cb2a0 2024-03-18T19:06:31.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.595+0000 7f53237fe700 1 -- 172.21.15.1:0/1369619904 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330011a90 con 0x7f533c0cb2a0 2024-03-18T19:06:31.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.595+0000 7f5341466700 1 -- 172.21.15.1:0/1369619904 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f533c0cc5e0 con 0x7f533c0cb2a0 2024-03-18T19:06:31.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.595+0000 7f5341466700 1 -- 172.21.15.1:0/1369619904 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f533c06cef0 con 0x7f533c0cb2a0 2024-03-18T19:06:31.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.596+0000 7f5341466700 1 -- 172.21.15.1:0/1369619904 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f533c15a180 con 0x7f533c0cb2a0 2024-03-18T19:06:31.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.596+0000 7f53237fe700 1 -- 172.21.15.1:0/1369619904 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 48) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f5330011bf0 con 0x7f533c0cb2a0 2024-03-18T19:06:31.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.596+0000 7f53237fe700 1 --2- 172.21.15.1:0/1369619904 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f532806ad20 0x7f532806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:31.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.596+0000 7f53237fe700 1 -- 172.21.15.1:0/1369619904 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6279+0+0 (secure 0 0 0) 0x7f5330089310 con 0x7f533c0cb2a0 2024-03-18T19:06:31.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.596+0000 7f533affd700 1 --2- 172.21.15.1:0/1369619904 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f532806ad20 0x7f532806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:31.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.597+0000 7f533affd700 1 --2- 172.21.15.1:0/1369619904 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f532806ad20 0x7f532806d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f532c0090b0 tx=0x7f532c01e040 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:31.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.601+0000 7f53237fe700 1 -- 172.21.15.1:0/1369619904 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f533c15a180 con 0x7f533c0cb2a0 2024-03-18T19:06:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:31.803+0000 7f5341466700 1 -- 172.21.15.1:0/1369619904 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f533c15dc90 con 0x7f533c0cb2a0 2024-03-18T19:06:32.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.436+0000 7f53237fe700 1 -- 172.21.15.1:0/1369619904 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v108) v1 ==== 169+0+0 (secure 0 0 0) 0x7f533c15dc90 con 0x7f533c0cb2a0 2024-03-18T19:06:32.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.439+0000 7f5341466700 1 -- 172.21.15.1:0/1369619904 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f532806ad20 msgr2=0x7f532806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.439+0000 7f5341466700 1 --2- 172.21.15.1:0/1369619904 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f532806ad20 0x7f532806d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f532c0090b0 tx=0x7f532c01e040 comp rx=0 tx=0).stop 2024-03-18T19:06:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.439+0000 7f5341466700 1 -- 172.21.15.1:0/1369619904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f533c0cb2a0 msgr2=0x7f533c0cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.439+0000 7f5341466700 1 --2- 172.21.15.1:0/1369619904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f533c0cb2a0 0x7f533c0cb720 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f5330007060 tx=0x7f5330007370 comp rx=0 tx=0).stop 2024-03-18T19:06:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.439+0000 7f5341466700 1 -- 172.21.15.1:0/1369619904 shutdown_connections 2024-03-18T19:06:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.439+0000 7f5341466700 1 --2- 172.21.15.1:0/1369619904 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f532806ad20 0x7f532806d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.439+0000 7f5341466700 1 --2- 172.21.15.1:0/1369619904 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f533c0c7e80 0x7f533c0c8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.439+0000 7f5341466700 1 --2- 172.21.15.1:0/1369619904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5324003a00 0x7f533c06c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.439+0000 7f5341466700 1 --2- 172.21.15.1:0/1369619904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f533c0cb2a0 0x7f533c0cb720 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.439+0000 7f5341466700 1 -- 172.21.15.1:0/1369619904 >> 172.21.15.1:0/1369619904 conn(0x7f533c09c640 msgr2=0x7f533c0b05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.440+0000 7f5341466700 1 -- 172.21.15.1:0/1369619904 shutdown_connections 2024-03-18T19:06:32.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.440+0000 7f5341466700 1 -- 172.21.15.1:0/1369619904 wait complete. 2024-03-18T19:06:32.437 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-18T19:06:32.451 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-18T19:06:32.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.590+0000 7f948179b700 1 Processor -- start 2024-03-18T19:06:32.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.590+0000 7f948179b700 1 -- start start 2024-03-18T19:06:32.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.591+0000 7f948179b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f947c15ad70 0x7f947c15d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:32.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.591+0000 7f948179b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f947c0727b0 0x7f947c161610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:32.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.591+0000 7f948179b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f947c160ea0 con 0x7f947c161b50 2024-03-18T19:06:32.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.591+0000 7f948179b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f947c0725b0 con 0x7f947c15ad70 2024-03-18T19:06:32.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.591+0000 7f948179b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f947c15ddf0 con 0x7f947c0727b0 2024-03-18T19:06:32.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.591+0000 7f947b7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f947c161b50 0x7f947c06a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52446/0 (socket says 172.21.15.1:52446) 2024-03-18T19:06:32.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.591+0000 7f947b7fe700 1 -- 172.21.15.1:0/1386215111 learned_addr learned my addr 172.21.15.1:0/1386215111 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:32.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.591+0000 7f9479ffb700 1 -- 172.21.15.1:0/1386215111 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926206025 0 0) 0x7f947c160ea0 con 0x7f947c161b50 2024-03-18T19:06:32.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f9479ffb700 1 -- 172.21.15.1:0/1386215111 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94640035b0 con 0x7f947c161b50 2024-03-18T19:06:32.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f947a7fc700 1 --2- 172.21.15.1:0/1386215111 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f947c0727b0 0x7f947c161610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:32.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f947affd700 1 --2- 172.21.15.1:0/1386215111 >> v2:172.21.15.165:3300/0 conn(0x7f947c15ad70 0x7f947c15d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:32.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f9479ffb700 1 -- 172.21.15.1:0/1386215111 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 932694019 0 0) 0x7f94640035b0 con 0x7f947c161b50 2024-03-18T19:06:32.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f9479ffb700 1 -- 172.21.15.1:0/1386215111 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f947c0727b0 msgr2=0x7f947c161610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:32.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f9479ffb700 1 --2- 172.21.15.1:0/1386215111 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f947c0727b0 0x7f947c161610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f9479ffb700 1 -- 172.21.15.1:0/1386215111 >> v2:172.21.15.165:3300/0 conn(0x7f947c15ad70 msgr2=0x7f947c15d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:32.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f9479ffb700 1 --2- 172.21.15.1:0/1386215111 >> v2:172.21.15.165:3300/0 conn(0x7f947c15ad70 0x7f947c15d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f9479ffb700 1 -- 172.21.15.1:0/1386215111 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947c15e090 con 0x7f947c161b50 2024-03-18T19:06:32.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f9479ffb700 1 -- 172.21.15.1:0/1386215111 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f94740019f0 con 0x7f947c161b50 2024-03-18T19:06:32.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f9479ffb700 1 -- 172.21.15.1:0/1386215111 >> v1:172.21.15.1:6789/0 conn(0x7f947c161b50 legacy=0x7f947c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:32.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f9479ffb700 1 --2- 172.21.15.1:0/1386215111 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9464003a00 0x7f9464003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:32.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f9479ffb700 1 --2- 172.21.15.1:0/1386215111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9464004850 0x7f9464004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:32.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f9479ffb700 1 --2- 172.21.15.1:0/1386215111 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f947c0727b0 0x7f9464005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:32.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f9479ffb700 1 -- 172.21.15.1:0/1386215111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947c15ddf0 con 0x7f9464004850 2024-03-18T19:06:32.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f9479ffb700 1 -- 172.21.15.1:0/1386215111 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946400ba40 con 0x7f9464003a00 2024-03-18T19:06:32.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f9479ffb700 1 -- 172.21.15.1:0/1386215111 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946400bc00 con 0x7f947c0727b0 2024-03-18T19:06:32.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f947a7fc700 1 --2- 172.21.15.1:0/1386215111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9464004850 0x7f9464004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:32.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f947affd700 1 --2- 172.21.15.1:0/1386215111 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9464003a00 0x7f9464003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:32.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f947b7fe700 1 --2- 172.21.15.1:0/1386215111 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f947c0727b0 0x7f9464005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:32.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f947a7fc700 1 -- 172.21.15.1:0/1386215111 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f947c0727b0 msgr2=0x7f9464005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:32.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f947a7fc700 1 --2- 172.21.15.1:0/1386215111 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f947c0727b0 0x7f9464005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f947a7fc700 1 -- 172.21.15.1:0/1386215111 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9464003a00 msgr2=0x7f9464003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:32.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f947a7fc700 1 --2- 172.21.15.1:0/1386215111 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9464003a00 0x7f9464003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.592+0000 7f947a7fc700 1 -- 172.21.15.1:0/1386215111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94640035b0 con 0x7f9464004850 2024-03-18T19:06:32.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.593+0000 7f947a7fc700 1 --2- 172.21.15.1:0/1386215111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9464004850 0x7f9464004cd0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f946c009b60 tx=0x7f9464005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecc7cd162fb74f5f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:32.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.593+0000 7f9479ffb700 1 -- 172.21.15.1:0/1386215111 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c00e6c0 con 0x7f9464004850 2024-03-18T19:06:32.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.593+0000 7f9479ffb700 1 -- 172.21.15.1:0/1386215111 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946c00e820 con 0x7f9464004850 2024-03-18T19:06:32.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.593+0000 7f9479ffb700 1 -- 172.21.15.1:0/1386215111 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946c00eb90 con 0x7f9464004850 2024-03-18T19:06:32.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.593+0000 7f948179b700 1 -- 172.21.15.1:0/1386215111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9464004850 msgr2=0x7f9464004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:32.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.593+0000 7f948179b700 1 --2- 172.21.15.1:0/1386215111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9464004850 0x7f9464004cd0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f946c009b60 tx=0x7f9464005880 comp rx=0 tx=0).stop 2024-03-18T19:06:32.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.593+0000 7f947b7fe700 1 -- 172.21.15.1:0/1386215111 reap_dead start 2024-03-18T19:06:32.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.593+0000 7f948179b700 1 -- 172.21.15.1:0/1386215111 shutdown_connections 2024-03-18T19:06:32.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.593+0000 7f948179b700 1 -- 172.21.15.1:0/1386215111 >> 172.21.15.1:0/1386215111 conn(0x7f947c0b1cb0 msgr2=0x7f947c0b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:32.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.593+0000 7f948179b700 1 -- 172.21.15.1:0/1386215111 shutdown_connections 2024-03-18T19:06:32.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.593+0000 7f948179b700 1 -- 172.21.15.1:0/1386215111 wait complete. 2024-03-18T19:06:32.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.595+0000 7f948179b700 1 Processor -- start 2024-03-18T19:06:32.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.595+0000 7f948179b700 1 -- start start 2024-03-18T19:06:32.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.595+0000 7f948179b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9464004850 0x7f947c13bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:32.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f948179b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f947c15d040 0x7f947c143cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:32.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f948179b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f947c1598c0 0x7f947c1441f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:32.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f948179b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947c13bff0 con 0x7f947c1598c0 2024-03-18T19:06:32.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f948179b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947c157af0 con 0x7f9464004850 2024-03-18T19:06:32.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f948179b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f947c0aeb40 con 0x7f947c15d040 2024-03-18T19:06:32.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f947b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f947c1598c0 0x7f947c1441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:32.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f947b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f947c1598c0 0x7f947c1441f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54332/0 (socket says 172.21.15.1:54332) 2024-03-18T19:06:32.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f947b7fe700 1 -- 172.21.15.1:0/2196286350 learned_addr learned my addr 172.21.15.1:0/2196286350 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:32.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f947affd700 1 --2- 172.21.15.1:0/2196286350 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9464004850 0x7f947c13bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:32.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f947a7fc700 1 --2- 172.21.15.1:0/2196286350 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f947c15d040 0x7f947c143cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:32.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f947b7fe700 1 -- 172.21.15.1:0/2196286350 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f947c15d040 msgr2=0x7f947c143cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:32.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f947b7fe700 1 --2- 172.21.15.1:0/2196286350 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f947c15d040 0x7f947c143cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f947b7fe700 1 -- 172.21.15.1:0/2196286350 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9464004850 msgr2=0x7f947c13bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:32.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f947b7fe700 1 --2- 172.21.15.1:0/2196286350 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9464004850 0x7f947c13bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f947b7fe700 1 -- 172.21.15.1:0/2196286350 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947c0aede0 con 0x7f947c1598c0 2024-03-18T19:06:32.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f947affd700 1 --2- 172.21.15.1:0/2196286350 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9464004850 0x7f947c13bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:32.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f947b7fe700 1 --2- 172.21.15.1:0/2196286350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f947c1598c0 0x7f947c1441f0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f947c071620 tx=0x7f947400a600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:32.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f94637fe700 1 -- 172.21.15.1:0/2196286350 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f947400ae00 con 0x7f947c1598c0 2024-03-18T19:06:32.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f94637fe700 1 -- 172.21.15.1:0/2196286350 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94740133f0 con 0x7f947c1598c0 2024-03-18T19:06:32.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f94637fe700 1 -- 172.21.15.1:0/2196286350 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94740137a0 con 0x7f947c1598c0 2024-03-18T19:06:32.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f948179b700 1 -- 172.21.15.1:0/2196286350 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f947c0af0c0 con 0x7f947c1598c0 2024-03-18T19:06:32.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.596+0000 7f948179b700 1 -- 172.21.15.1:0/2196286350 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f947c159370 con 0x7f947c1598c0 2024-03-18T19:06:32.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.597+0000 7f948179b700 1 -- 172.21.15.1:0/2196286350 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f947c0af6d0 con 0x7f947c1598c0 2024-03-18T19:06:32.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.598+0000 7f94637fe700 1 -- 172.21.15.1:0/2196286350 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 48) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f9474013900 con 0x7f947c1598c0 2024-03-18T19:06:32.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.598+0000 7f94637fe700 1 --2- 172.21.15.1:0/2196286350 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f946806ad20 0x7f946806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:32.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.599+0000 7f94637fe700 1 -- 172.21.15.1:0/2196286350 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f947408a840 con 0x7f947c1598c0 2024-03-18T19:06:32.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.599+0000 7f947affd700 1 --2- 172.21.15.1:0/2196286350 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f946806ad20 0x7f946806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:32.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.599+0000 7f947affd700 1 --2- 172.21.15.1:0/2196286350 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f946806ad20 0x7f946806d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9470009b50 tx=0x7f9470008040 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:32.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.603+0000 7f94637fe700 1 -- 172.21.15.1:0/2196286350 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9474055aa0 con 0x7f947c1598c0 2024-03-18T19:06:32.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.805+0000 7f948179b700 1 -- 172.21.15.1:0/2196286350 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f947c0676a0 con 0x7f947c1598c0 2024-03-18T19:06:32.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.809+0000 7f94637fe700 1 -- 172.21.15.1:0/2196286350 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v31) v1 ==== 115+0+0 (secure 0 0 0) 0x7f94740590c0 con 0x7f947c1598c0 2024-03-18T19:06:32.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.811+0000 7f948179b700 1 -- 172.21.15.1:0/2196286350 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f946806ad20 msgr2=0x7f946806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:32.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.811+0000 7f948179b700 1 --2- 172.21.15.1:0/2196286350 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f946806ad20 0x7f946806d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9470009b50 tx=0x7f9470008040 comp rx=0 tx=0).stop 2024-03-18T19:06:32.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.811+0000 7f948179b700 1 -- 172.21.15.1:0/2196286350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f947c1598c0 msgr2=0x7f947c1441f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:32.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.811+0000 7f948179b700 1 --2- 172.21.15.1:0/2196286350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f947c1598c0 0x7f947c1441f0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f947c071620 tx=0x7f947400a600 comp rx=0 tx=0).stop 2024-03-18T19:06:32.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.811+0000 7f948179b700 1 -- 172.21.15.1:0/2196286350 shutdown_connections 2024-03-18T19:06:32.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.811+0000 7f948179b700 1 --2- 172.21.15.1:0/2196286350 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f946806ad20 0x7f946806d1e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.811+0000 7f948179b700 1 --2- 172.21.15.1:0/2196286350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f947c1598c0 0x7f947c1441f0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.811+0000 7f948179b700 1 --2- 172.21.15.1:0/2196286350 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9464004850 0x7f947c13bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.811+0000 7f948179b700 1 --2- 172.21.15.1:0/2196286350 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f947c15d040 0x7f947c143cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.811+0000 7f948179b700 1 -- 172.21.15.1:0/2196286350 >> 172.21.15.1:0/2196286350 conn(0x7f947c0b1cb0 msgr2=0x7f947c145600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:32.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.811+0000 7f948179b700 1 -- 172.21.15.1:0/2196286350 shutdown_connections 2024-03-18T19:06:32.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.812+0000 7f948179b700 1 -- 172.21.15.1:0/2196286350 wait complete. 2024-03-18T19:06:32.820 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-18T19:06:32.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.956+0000 7f3209451700 1 Processor -- start 2024-03-18T19:06:32.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.956+0000 7f3209451700 1 -- start start 2024-03-18T19:06:32.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f3209451700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32040cfb50 0x7f32040d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:32.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f3209451700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f32040cff30 0x7f32040b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:32.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f3209451700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32040b3370 con 0x7f32040b3d40 2024-03-18T19:06:32.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f3209451700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32040b2760 con 0x7f32040cff30 2024-03-18T19:06:32.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f3209451700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32040b2a40 con 0x7f32040cfb50 2024-03-18T19:06:32.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f32037fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f32040b3d40 0x7f32040b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52450/0 (socket says 172.21.15.1:52450) 2024-03-18T19:06:32.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f32037fe700 1 -- 172.21.15.1:0/997848609 learned_addr learned my addr 172.21.15.1:0/997848609 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:32.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f32027fc700 1 --2- 172.21.15.1:0/997848609 >> v2:172.21.15.165:3300/0 conn(0x7f32040cff30 0x7f32040b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:32.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f3202ffd700 1 --2- 172.21.15.1:0/997848609 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32040cfb50 0x7f32040d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:32.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f3201ffb700 1 -- 172.21.15.1:0/997848609 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494369375 0 0) 0x7f32040b3370 con 0x7f32040b3d40 2024-03-18T19:06:32.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f3201ffb700 1 -- 172.21.15.1:0/997848609 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31ec0035b0 con 0x7f32040b3d40 2024-03-18T19:06:32.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f3201ffb700 1 -- 172.21.15.1:0/997848609 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3133068979 0 0) 0x7f31ec0035b0 con 0x7f32040b3d40 2024-03-18T19:06:32.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f3201ffb700 1 -- 172.21.15.1:0/997848609 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32040cfb50 msgr2=0x7f32040d3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:32.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f3201ffb700 1 --2- 172.21.15.1:0/997848609 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32040cfb50 0x7f32040d3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f3201ffb700 1 -- 172.21.15.1:0/997848609 >> v2:172.21.15.165:3300/0 conn(0x7f32040cff30 msgr2=0x7f32040b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:32.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f3201ffb700 1 --2- 172.21.15.1:0/997848609 >> v2:172.21.15.165:3300/0 conn(0x7f32040cff30 0x7f32040b3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f3201ffb700 1 -- 172.21.15.1:0/997848609 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32040b2d20 con 0x7f32040b3d40 2024-03-18T19:06:32.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f3201ffb700 1 -- 172.21.15.1:0/997848609 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f31fc0019f0 con 0x7f32040b3d40 2024-03-18T19:06:32.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f3201ffb700 1 -- 172.21.15.1:0/997848609 >> v1:172.21.15.1:6789/0 conn(0x7f32040b3d40 legacy=0x7f32040b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:32.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.957+0000 7f3201ffb700 1 --2- 172.21.15.1:0/997848609 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31ec003a00 0x7f31ec003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:32.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.958+0000 7f3201ffb700 1 --2- 172.21.15.1:0/997848609 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32040cfb50 0x7f31ec004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:32.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.958+0000 7f3201ffb700 1 --2- 172.21.15.1:0/997848609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31ec004d90 0x7f31ec005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:32.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.958+0000 7f3201ffb700 1 -- 172.21.15.1:0/997848609 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32040b2760 con 0x7f31ec004d90 2024-03-18T19:06:32.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.958+0000 7f3201ffb700 1 -- 172.21.15.1:0/997848609 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32040b2a40 con 0x7f31ec003a00 2024-03-18T19:06:32.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.958+0000 7f3201ffb700 1 -- 172.21.15.1:0/997848609 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31ec00ba10 con 0x7f32040cfb50 2024-03-18T19:06:32.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.958+0000 7f32037fe700 1 --2- 172.21.15.1:0/997848609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31ec004d90 0x7f31ec005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:32.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.958+0000 7f3202ffd700 1 --2- 172.21.15.1:0/997848609 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31ec003a00 0x7f31ec003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:32.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.958+0000 7f32027fc700 1 --2- 172.21.15.1:0/997848609 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32040cfb50 0x7f31ec004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:32.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.958+0000 7f32037fe700 1 -- 172.21.15.1:0/997848609 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32040cfb50 msgr2=0x7f31ec004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:32.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.958+0000 7f32037fe700 1 --2- 172.21.15.1:0/997848609 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32040cfb50 0x7f31ec004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.958+0000 7f32037fe700 1 -- 172.21.15.1:0/997848609 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31ec003a00 msgr2=0x7f31ec003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:32.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.958+0000 7f32037fe700 1 --2- 172.21.15.1:0/997848609 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31ec003a00 0x7f31ec003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.958+0000 7f32037fe700 1 -- 172.21.15.1:0/997848609 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ec0035b0 con 0x7f31ec004d90 2024-03-18T19:06:32.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.958+0000 7f32037fe700 1 --2- 172.21.15.1:0/997848609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31ec004d90 0x7f31ec005210 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f31fc00a860 tx=0x7f31ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=662750f0a5ca280c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:32.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.958+0000 7f3201ffb700 1 -- 172.21.15.1:0/997848609 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc012600 con 0x7f31ec004d90 2024-03-18T19:06:32.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.958+0000 7f3201ffb700 1 -- 172.21.15.1:0/997848609 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31fc012780 con 0x7f31ec004d90 2024-03-18T19:06:32.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.958+0000 7f3201ffb700 1 -- 172.21.15.1:0/997848609 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc012b30 con 0x7f31ec004d90 2024-03-18T19:06:32.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.959+0000 7f3209451700 1 -- 172.21.15.1:0/997848609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31ec004d90 msgr2=0x7f31ec005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:32.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.959+0000 7f3209451700 1 --2- 172.21.15.1:0/997848609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31ec004d90 0x7f31ec005210 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f31fc00a860 tx=0x7f31ec005880 comp rx=0 tx=0).stop 2024-03-18T19:06:32.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.959+0000 7f32037fe700 1 -- 172.21.15.1:0/997848609 reap_dead start 2024-03-18T19:06:32.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.959+0000 7f3209451700 1 -- 172.21.15.1:0/997848609 shutdown_connections 2024-03-18T19:06:32.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.959+0000 7f3209451700 1 -- 172.21.15.1:0/997848609 >> 172.21.15.1:0/997848609 conn(0x7f32040d3140 msgr2=0x7f32040d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:32.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.959+0000 7f3209451700 1 -- 172.21.15.1:0/997848609 shutdown_connections 2024-03-18T19:06:32.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.959+0000 7f3209451700 1 -- 172.21.15.1:0/997848609 wait complete. 2024-03-18T19:06:32.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.960+0000 7f3209451700 1 Processor -- start 2024-03-18T19:06:32.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.960+0000 7f3209451700 1 -- start start 2024-03-18T19:06:32.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.961+0000 7f3209451700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f320415a2c0 0x7f320414a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:32.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.961+0000 7f3209451700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f320415a6a0 0x7f320414a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:32.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.961+0000 7f3209451700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32040cb320 0x7f32040cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:32.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.961+0000 7f3209451700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32040c8310 con 0x7f320415a2c0 2024-03-18T19:06:32.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.961+0000 7f3209451700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32040c7700 con 0x7f320415a6a0 2024-03-18T19:06:32.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.961+0000 7f3209451700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32040c79e0 con 0x7f32040cb320 2024-03-18T19:06:32.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.961+0000 7f3202ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f320415a2c0 0x7f320414a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:32.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.961+0000 7f3202ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f320415a2c0 0x7f320414a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54344/0 (socket says 172.21.15.1:54344) 2024-03-18T19:06:32.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.961+0000 7f3202ffd700 1 -- 172.21.15.1:0/2380849902 learned_addr learned my addr 172.21.15.1:0/2380849902 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:32.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.961+0000 7f32037fe700 1 --2- 172.21.15.1:0/2380849902 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32040cb320 0x7f32040cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:32.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.961+0000 7f32027fc700 1 --2- 172.21.15.1:0/2380849902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f320415a6a0 0x7f320414a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:32.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.961+0000 7f3202ffd700 1 -- 172.21.15.1:0/2380849902 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32040cb320 msgr2=0x7f32040cb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:32.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.961+0000 7f3202ffd700 1 --2- 172.21.15.1:0/2380849902 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32040cb320 0x7f32040cb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.961+0000 7f3202ffd700 1 -- 172.21.15.1:0/2380849902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f320415a6a0 msgr2=0x7f320414a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:32.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.961+0000 7f3202ffd700 1 --2- 172.21.15.1:0/2380849902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f320415a6a0 0x7f320414a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:32.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.961+0000 7f3202ffd700 1 -- 172.21.15.1:0/2380849902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32040c7cc0 con 0x7f320415a2c0 2024-03-18T19:06:32.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.962+0000 7f3202ffd700 1 --2- 172.21.15.1:0/2380849902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f320415a2c0 0x7f320414a140 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f31f8009fd0 tx=0x7f31f8011c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:32.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.962+0000 7f31eb7fe700 1 -- 172.21.15.1:0/2380849902 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f8005a20 con 0x7f320415a2c0 2024-03-18T19:06:32.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.962+0000 7f31eb7fe700 1 -- 172.21.15.1:0/2380849902 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f8005b80 con 0x7f320415a2c0 2024-03-18T19:06:32.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.962+0000 7f31eb7fe700 1 -- 172.21.15.1:0/2380849902 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f800f730 con 0x7f320415a2c0 2024-03-18T19:06:32.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.962+0000 7f3209451700 1 -- 172.21.15.1:0/2380849902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32040c7fa0 con 0x7f320415a2c0 2024-03-18T19:06:32.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.962+0000 7f3209451700 1 -- 172.21.15.1:0/2380849902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32040c97d0 con 0x7f320415a2c0 2024-03-18T19:06:32.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.963+0000 7f3209451700 1 -- 172.21.15.1:0/2380849902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32040c8130 con 0x7f320415a2c0 2024-03-18T19:06:32.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.963+0000 7f31eb7fe700 1 -- 172.21.15.1:0/2380849902 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 48) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f31f800f890 con 0x7f320415a2c0 2024-03-18T19:06:32.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.963+0000 7f31eb7fe700 1 --2- 172.21.15.1:0/2380849902 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f31f006ad20 0x7f31f006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:32.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.964+0000 7f31eb7fe700 1 -- 172.21.15.1:0/2380849902 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f31f8089a00 con 0x7f320415a2c0 2024-03-18T19:06:32.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.964+0000 7f32027fc700 1 --2- 172.21.15.1:0/2380849902 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f31f006ad20 0x7f31f006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:32.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.964+0000 7f32027fc700 1 --2- 172.21.15.1:0/2380849902 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f31f006ad20 0x7f31f006d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f32040c6590 tx=0x7f31f4009210 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:32.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:32.968+0000 7f31eb7fe700 1 -- 172.21.15.1:0/2380849902 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f31f8054b50 con 0x7f320415a2c0 2024-03-18T19:06:33.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.074+0000 7f31eb7fe700 1 -- 172.21.15.1:0/2380849902 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f31f8054f20 con 0x7f320415a2c0 2024-03-18T19:06:33.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.176+0000 7f3209451700 1 -- 172.21.15.1:0/2380849902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f3204068220 con 0x7f320415a2c0 2024-03-18T19:06:33.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.180+0000 7f31eb7fe700 1 -- 172.21.15.1:0/2380849902 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v32) v1 ==== 120+0+0 (secure 0 0 0) 0x7f31f8086940 con 0x7f320415a2c0 2024-03-18T19:06:33.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.182+0000 7f3209451700 1 -- 172.21.15.1:0/2380849902 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f31f006ad20 msgr2=0x7f31f006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:33.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.182+0000 7f3209451700 1 --2- 172.21.15.1:0/2380849902 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f31f006ad20 0x7f31f006d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f32040c6590 tx=0x7f31f4009210 comp rx=0 tx=0).stop 2024-03-18T19:06:33.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.182+0000 7f3209451700 1 -- 172.21.15.1:0/2380849902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f320415a2c0 msgr2=0x7f320414a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:33.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.182+0000 7f3209451700 1 --2- 172.21.15.1:0/2380849902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f320415a2c0 0x7f320414a140 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f31f8009fd0 tx=0x7f31f8011c80 comp rx=0 tx=0).stop 2024-03-18T19:06:33.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.182+0000 7f3209451700 1 -- 172.21.15.1:0/2380849902 shutdown_connections 2024-03-18T19:06:33.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.182+0000 7f3209451700 1 --2- 172.21.15.1:0/2380849902 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f31f006ad20 0x7f31f006d1e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:33.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.182+0000 7f3209451700 1 --2- 172.21.15.1:0/2380849902 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32040cb320 0x7f32040cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:33.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.182+0000 7f3209451700 1 --2- 172.21.15.1:0/2380849902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f320415a2c0 0x7f320414a140 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:33.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.182+0000 7f3209451700 1 --2- 172.21.15.1:0/2380849902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f320415a6a0 0x7f320414a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:33.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.182+0000 7f3209451700 1 -- 172.21.15.1:0/2380849902 >> 172.21.15.1:0/2380849902 conn(0x7f32040d3140 msgr2=0x7f32040d1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:33.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.182+0000 7f3209451700 1 -- 172.21.15.1:0/2380849902 shutdown_connections 2024-03-18T19:06:33.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.182+0000 7f3209451700 1 -- 172.21.15.1:0/2380849902 wait complete. 2024-03-18T19:06:33.191 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-18T19:06:33.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.326+0000 7f1d856bd700 1 Processor -- start 2024-03-18T19:06:33.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.326+0000 7f1d856bd700 1 -- start start 2024-03-18T19:06:33.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.327+0000 7f1d856bd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1d800ce8d0 0x7f1d800b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:33.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.327+0000 7f1d856bd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d80070580 0x7f1d80070a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:33.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.327+0000 7f1d856bd700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d800b38d0 con 0x7f1d8006d3d0 2024-03-18T19:06:33.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.327+0000 7f1d856bd700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d800cecb0 con 0x7f1d800ce8d0 2024-03-18T19:06:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.327+0000 7f1d856bd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d80071020 con 0x7f1d80070580 2024-03-18T19:06:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.327+0000 7f1d7f7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1d8006d3d0 0x7f1d800b1f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52460/0 (socket says 172.21.15.1:52460) 2024-03-18T19:06:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.327+0000 7f1d7f7fe700 1 -- 172.21.15.1:0/4221230144 learned_addr learned my addr 172.21.15.1:0/4221230144 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.327+0000 7f1d7dffb700 1 -- 172.21.15.1:0/4221230144 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571762377 0 0) 0x7f1d800b38d0 con 0x7f1d8006d3d0 2024-03-18T19:06:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.327+0000 7f1d7dffb700 1 -- 172.21.15.1:0/4221230144 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d6c0035b0 con 0x7f1d8006d3d0 2024-03-18T19:06:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7e7fc700 1 --2- 172.21.15.1:0/4221230144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d80070580 0x7f1d80070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7effd700 1 --2- 172.21.15.1:0/4221230144 >> v2:172.21.15.165:3300/0 conn(0x7f1d800ce8d0 0x7f1d800b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7dffb700 1 -- 172.21.15.1:0/4221230144 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1996602828 0 0) 0x7f1d6c0035b0 con 0x7f1d8006d3d0 2024-03-18T19:06:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7dffb700 1 -- 172.21.15.1:0/4221230144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d80070580 msgr2=0x7f1d80070a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7dffb700 1 --2- 172.21.15.1:0/4221230144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d80070580 0x7f1d80070a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7dffb700 1 -- 172.21.15.1:0/4221230144 >> v2:172.21.15.165:3300/0 conn(0x7f1d800ce8d0 msgr2=0x7f1d800b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7dffb700 1 --2- 172.21.15.1:0/4221230144 >> v2:172.21.15.165:3300/0 conn(0x7f1d800ce8d0 0x7f1d800b3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:33.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7dffb700 1 -- 172.21.15.1:0/4221230144 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d80071300 con 0x7f1d8006d3d0 2024-03-18T19:06:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7dffb700 1 -- 172.21.15.1:0/4221230144 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f1d70001670 con 0x7f1d8006d3d0 2024-03-18T19:06:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7dffb700 1 -- 172.21.15.1:0/4221230144 >> v1:172.21.15.1:6789/0 conn(0x7f1d8006d3d0 legacy=0x7f1d800b1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7dffb700 1 --2- 172.21.15.1:0/4221230144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d6c003a00 0x7f1d6c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7dffb700 1 --2- 172.21.15.1:0/4221230144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d80070580 0x7f1d6c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7dffb700 1 --2- 172.21.15.1:0/4221230144 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d6c004d90 0x7f1d6c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7dffb700 1 -- 172.21.15.1:0/4221230144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d800cecb0 con 0x7f1d6c003a00 2024-03-18T19:06:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7dffb700 1 -- 172.21.15.1:0/4221230144 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d80071020 con 0x7f1d6c004d90 2024-03-18T19:06:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7dffb700 1 -- 172.21.15.1:0/4221230144 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c00ba40 con 0x7f1d80070580 2024-03-18T19:06:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7effd700 1 --2- 172.21.15.1:0/4221230144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d6c003a00 0x7f1d6c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7e7fc700 1 --2- 172.21.15.1:0/4221230144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d80070580 0x7f1d6c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7f7fe700 1 --2- 172.21.15.1:0/4221230144 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d6c004d90 0x7f1d6c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7effd700 1 -- 172.21.15.1:0/4221230144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d80070580 msgr2=0x7f1d6c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:33.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7effd700 1 --2- 172.21.15.1:0/4221230144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d80070580 0x7f1d6c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:33.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7effd700 1 -- 172.21.15.1:0/4221230144 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d6c004d90 msgr2=0x7f1d6c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:33.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7effd700 1 --2- 172.21.15.1:0/4221230144 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d6c004d90 0x7f1d6c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:33.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7effd700 1 -- 172.21.15.1:0/4221230144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c0035b0 con 0x7f1d6c003a00 2024-03-18T19:06:33.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7e7fc700 1 --2- 172.21.15.1:0/4221230144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d80070580 0x7f1d6c004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:33.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.328+0000 7f1d7effd700 1 --2- 172.21.15.1:0/4221230144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d6c003a00 0x7f1d6c003e20 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f1d7800f270 tx=0x7f1d6c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee1623302684622 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:33.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.329+0000 7f1d7dffb700 1 -- 172.21.15.1:0/4221230144 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7800a2d0 con 0x7f1d6c003a00 2024-03-18T19:06:33.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.329+0000 7f1d7dffb700 1 -- 172.21.15.1:0/4221230144 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d78010040 con 0x7f1d6c003a00 2024-03-18T19:06:33.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.329+0000 7f1d7dffb700 1 -- 172.21.15.1:0/4221230144 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7800fd50 con 0x7f1d6c003a00 2024-03-18T19:06:33.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.329+0000 7f1d856bd700 1 -- 172.21.15.1:0/4221230144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d6c003a00 msgr2=0x7f1d6c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:33.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.329+0000 7f1d856bd700 1 --2- 172.21.15.1:0/4221230144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d6c003a00 0x7f1d6c003e20 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f1d7800f270 tx=0x7f1d6c005880 comp rx=0 tx=0).stop 2024-03-18T19:06:33.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.329+0000 7f1d7f7fe700 1 -- 172.21.15.1:0/4221230144 reap_dead start 2024-03-18T19:06:33.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.329+0000 7f1d856bd700 1 -- 172.21.15.1:0/4221230144 shutdown_connections 2024-03-18T19:06:33.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.329+0000 7f1d856bd700 1 -- 172.21.15.1:0/4221230144 >> 172.21.15.1:0/4221230144 conn(0x7f1d8006ab70 msgr2=0x7f1d8006af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:33.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.329+0000 7f1d856bd700 1 -- 172.21.15.1:0/4221230144 shutdown_connections 2024-03-18T19:06:33.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.329+0000 7f1d856bd700 1 -- 172.21.15.1:0/4221230144 wait complete. 2024-03-18T19:06:33.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.331+0000 7f1d856bd700 1 Processor -- start 2024-03-18T19:06:33.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.331+0000 7f1d856bd700 1 -- start start 2024-03-18T19:06:33.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.331+0000 7f1d856bd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d6c003a00 0x7f1d800a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:33.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.331+0000 7f1d856bd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d800acd30 0x7f1d800ad1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:33.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.331+0000 7f1d856bd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d80148440 0x7f1d801488f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:33.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.331+0000 7f1d856bd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d800ad6f0 con 0x7f1d6c003a00 2024-03-18T19:06:33.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.331+0000 7f1d856bd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d800aa500 con 0x7f1d800acd30 2024-03-18T19:06:33.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.331+0000 7f1d856bd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d800a96e0 con 0x7f1d80148440 2024-03-18T19:06:33.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.331+0000 7f1d7effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d6c003a00 0x7f1d800a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:33.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.331+0000 7f1d7effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d6c003a00 0x7f1d800a8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54362/0 (socket says 172.21.15.1:54362) 2024-03-18T19:06:33.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.332+0000 7f1d7effd700 1 -- 172.21.15.1:0/4234116059 learned_addr learned my addr 172.21.15.1:0/4234116059 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:33.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.332+0000 7f1d7effd700 1 -- 172.21.15.1:0/4234116059 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d80148440 msgr2=0x7f1d801488f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:33.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.332+0000 7f1d7e7fc700 1 --2- 172.21.15.1:0/4234116059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d800acd30 0x7f1d800ad1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:33.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.332+0000 7f1d7f7fe700 1 --2- 172.21.15.1:0/4234116059 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d80148440 0x7f1d801488f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:33.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.332+0000 7f1d7effd700 1 --2- 172.21.15.1:0/4234116059 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d80148440 0x7f1d801488f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:33.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.332+0000 7f1d7effd700 1 -- 172.21.15.1:0/4234116059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d800acd30 msgr2=0x7f1d800ad1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:33.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.332+0000 7f1d7effd700 1 --2- 172.21.15.1:0/4234116059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d800acd30 0x7f1d800ad1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:33.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.332+0000 7f1d7effd700 1 -- 172.21.15.1:0/4234116059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d800a99c0 con 0x7f1d6c003a00 2024-03-18T19:06:33.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.332+0000 7f1d7effd700 1 --2- 172.21.15.1:0/4234116059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d6c003a00 0x7f1d800a8eb0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f1d78000f80 tx=0x7f1d7800a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:33.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.332+0000 7f1d777fe700 1 -- 172.21.15.1:0/4234116059 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d78015990 con 0x7f1d6c003a00 2024-03-18T19:06:33.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.332+0000 7f1d777fe700 1 -- 172.21.15.1:0/4234116059 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d78023070 con 0x7f1d6c003a00 2024-03-18T19:06:33.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.332+0000 7f1d856bd700 1 -- 172.21.15.1:0/4234116059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d800a9ca0 con 0x7f1d6c003a00 2024-03-18T19:06:33.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.332+0000 7f1d777fe700 1 -- 172.21.15.1:0/4234116059 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d78015d70 con 0x7f1d6c003a00 2024-03-18T19:06:33.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.332+0000 7f1d856bd700 1 -- 172.21.15.1:0/4234116059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d80148e30 con 0x7f1d6c003a00 2024-03-18T19:06:33.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.333+0000 7f1d856bd700 1 -- 172.21.15.1:0/4234116059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d80073350 con 0x7f1d6c003a00 2024-03-18T19:06:33.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.334+0000 7f1d777fe700 1 -- 172.21.15.1:0/4234116059 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1d7801e400 con 0x7f1d6c003a00 2024-03-18T19:06:33.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.334+0000 7f1d777fe700 1 --2- 172.21.15.1:0/4234116059 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1d6006adb0 0x7f1d6006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:33.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.334+0000 7f1d7e7fc700 1 --2- 172.21.15.1:0/4234116059 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1d6006adb0 0x7f1d6006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:33.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.334+0000 7f1d777fe700 1 -- 172.21.15.1:0/4234116059 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f1d78025070 con 0x7f1d6c003a00 2024-03-18T19:06:33.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.335+0000 7f1d7e7fc700 1 --2- 172.21.15.1:0/4234116059 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1d6006adb0 0x7f1d6006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1d8015d470 tx=0x7f1d68008040 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:33.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.339+0000 7f1d777fe700 1 -- 172.21.15.1:0/4234116059 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1d780565f0 con 0x7f1d6c003a00 2024-03-18T19:06:33.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.546+0000 7f1d856bd700 1 -- 172.21.15.1:0/4234116059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1d80071de0 con 0x7f1d6c003a00 2024-03-18T19:06:33.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.550+0000 7f1d777fe700 1 -- 172.21.15.1:0/4234116059 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1d78059c10 con 0x7f1d6c003a00 2024-03-18T19:06:33.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.552+0000 7f1d856bd700 1 -- 172.21.15.1:0/4234116059 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1d6006adb0 msgr2=0x7f1d6006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:33.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.552+0000 7f1d856bd700 1 --2- 172.21.15.1:0/4234116059 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1d6006adb0 0x7f1d6006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1d8015d470 tx=0x7f1d68008040 comp rx=0 tx=0).stop 2024-03-18T19:06:33.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.552+0000 7f1d856bd700 1 -- 172.21.15.1:0/4234116059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d6c003a00 msgr2=0x7f1d800a8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:33.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.552+0000 7f1d856bd700 1 --2- 172.21.15.1:0/4234116059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d6c003a00 0x7f1d800a8eb0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f1d78000f80 tx=0x7f1d7800a860 comp rx=0 tx=0).stop 2024-03-18T19:06:33.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.552+0000 7f1d856bd700 1 -- 172.21.15.1:0/4234116059 shutdown_connections 2024-03-18T19:06:33.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.552+0000 7f1d856bd700 1 --2- 172.21.15.1:0/4234116059 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1d6006adb0 0x7f1d6006d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:33.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.552+0000 7f1d856bd700 1 --2- 172.21.15.1:0/4234116059 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d80148440 0x7f1d801488f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:33.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.552+0000 7f1d856bd700 1 --2- 172.21.15.1:0/4234116059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d6c003a00 0x7f1d800a8eb0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:33.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.552+0000 7f1d856bd700 1 --2- 172.21.15.1:0/4234116059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d800acd30 0x7f1d800ad1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:33.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.552+0000 7f1d856bd700 1 -- 172.21.15.1:0/4234116059 >> 172.21.15.1:0/4234116059 conn(0x7f1d8006ab70 msgr2=0x7f1d801567b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:33.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.552+0000 7f1d856bd700 1 -- 172.21.15.1:0/4234116059 shutdown_connections 2024-03-18T19:06:33.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.552+0000 7f1d856bd700 1 -- 172.21.15.1:0/4234116059 wait complete. 2024-03-18T19:06:33.560 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2024-03-18T19:06:33.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.689+0000 7fc9153a3700 1 Processor -- start 2024-03-18T19:06:33.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.689+0000 7fc9153a3700 1 -- start start 2024-03-18T19:06:33.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.690+0000 7fc9153a3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc910159f10 0x7fc9100cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:33.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.690+0000 7fc9153a3700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc91015dd20 con 0x7fc91006e0a0 2024-03-18T19:06:33.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.690+0000 7fc9153a3700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc91015e000 con 0x7fc91015a2f0 2024-03-18T19:06:33.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.690+0000 7fc9153a3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc91015e2e0 con 0x7fc910159f10 2024-03-18T19:06:33.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.690+0000 7fc90f7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc91006e0a0 0x7fc9100b29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52466/0 (socket says 172.21.15.1:52466) 2024-03-18T19:06:33.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.690+0000 7fc90f7fe700 1 -- 172.21.15.1:0/1350010585 learned_addr learned my addr 172.21.15.1:0/1350010585 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:33.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90effd700 1 --2- 172.21.15.1:0/1350010585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc910159f10 0x7fc9100cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:33.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90dffb700 1 -- 172.21.15.1:0/1350010585 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272756669 0 0) 0x7fc91015dd20 con 0x7fc91006e0a0 2024-03-18T19:06:33.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90dffb700 1 -- 172.21.15.1:0/1350010585 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f80035b0 con 0x7fc91006e0a0 2024-03-18T19:06:33.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90dffb700 1 -- 172.21.15.1:0/1350010585 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4016526956 0 0) 0x7fc8f80035b0 con 0x7fc91006e0a0 2024-03-18T19:06:33.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90dffb700 1 -- 172.21.15.1:0/1350010585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc910159f10 msgr2=0x7fc9100cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:33.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90dffb700 1 --2- 172.21.15.1:0/1350010585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc910159f10 0x7fc9100cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:33.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90dffb700 1 -- 172.21.15.1:0/1350010585 >> v1:172.21.15.165:6789/0 conn(0x7fc91015a2f0 legacy=0x7fc910070fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:33.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90dffb700 1 -- 172.21.15.1:0/1350010585 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc91015e5c0 con 0x7fc91006e0a0 2024-03-18T19:06:33.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90dffb700 1 -- 172.21.15.1:0/1350010585 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fc9080019f0 con 0x7fc91006e0a0 2024-03-18T19:06:33.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90dffb700 1 -- 172.21.15.1:0/1350010585 >> v1:172.21.15.1:6789/0 conn(0x7fc91006e0a0 legacy=0x7fc9100b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:33.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90dffb700 1 --2- 172.21.15.1:0/1350010585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc910159f10 0x7fc8f8003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:33.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90dffb700 1 --2- 172.21.15.1:0/1350010585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8f8003fb0 0x7fc8f8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:33.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90dffb700 1 --2- 172.21.15.1:0/1350010585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8f8004d90 0x7fc8f8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:33.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90dffb700 1 -- 172.21.15.1:0/1350010585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91015e2e0 con 0x7fc8f8004d90 2024-03-18T19:06:33.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90dffb700 1 -- 172.21.15.1:0/1350010585 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f800baf0 con 0x7fc8f8003fb0 2024-03-18T19:06:33.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90dffb700 1 -- 172.21.15.1:0/1350010585 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f800dec0 con 0x7fc910159f10 2024-03-18T19:06:33.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90f7fe700 1 --2- 172.21.15.1:0/1350010585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8f8004d90 0x7fc8f8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:33.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90effd700 1 --2- 172.21.15.1:0/1350010585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc910159f10 0x7fc8f8003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:33.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90e7fc700 1 --2- 172.21.15.1:0/1350010585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8f8003fb0 0x7fc8f8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:33.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90f7fe700 1 -- 172.21.15.1:0/1350010585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc910159f10 msgr2=0x7fc8f8003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:33.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90f7fe700 1 --2- 172.21.15.1:0/1350010585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc910159f10 0x7fc8f8003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:33.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90f7fe700 1 -- 172.21.15.1:0/1350010585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8f8003fb0 msgr2=0x7fc8f8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:33.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90f7fe700 1 --2- 172.21.15.1:0/1350010585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8f8003fb0 0x7fc8f8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:33.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.691+0000 7fc90f7fe700 1 -- 172.21.15.1:0/1350010585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f80035b0 con 0x7fc8f8004d90 2024-03-18T19:06:33.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.692+0000 7fc90f7fe700 1 --2- 172.21.15.1:0/1350010585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8f8004d90 0x7fc8f8005210 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fc90800a830 tx=0x7fc8f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f3a7f2395f89e81 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:33.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.692+0000 7fc90dffb700 1 -- 172.21.15.1:0/1350010585 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90800a2f0 con 0x7fc8f8004d90 2024-03-18T19:06:33.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.692+0000 7fc90dffb700 1 -- 172.21.15.1:0/1350010585 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc908012600 con 0x7fc8f8004d90 2024-03-18T19:06:33.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.692+0000 7fc90dffb700 1 -- 172.21.15.1:0/1350010585 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9080129b0 con 0x7fc8f8004d90 2024-03-18T19:06:33.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.692+0000 7fc9153a3700 1 -- 172.21.15.1:0/1350010585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8f8004d90 msgr2=0x7fc8f8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:33.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.692+0000 7fc9153a3700 1 --2- 172.21.15.1:0/1350010585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8f8004d90 0x7fc8f8005210 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fc90800a830 tx=0x7fc8f8005880 comp rx=0 tx=0).stop 2024-03-18T19:06:33.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.692+0000 7fc90f7fe700 1 -- 172.21.15.1:0/1350010585 reap_dead start 2024-03-18T19:06:33.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.692+0000 7fc9153a3700 1 -- 172.21.15.1:0/1350010585 shutdown_connections 2024-03-18T19:06:33.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.692+0000 7fc9153a3700 1 -- 172.21.15.1:0/1350010585 >> 172.21.15.1:0/1350010585 conn(0x7fc91006a750 msgr2=0x7fc91006ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:33.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.692+0000 7fc9153a3700 1 -- 172.21.15.1:0/1350010585 shutdown_connections 2024-03-18T19:06:33.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.692+0000 7fc9153a3700 1 -- 172.21.15.1:0/1350010585 wait complete. 2024-03-18T19:06:33.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.694+0000 7fc9153a3700 1 Processor -- start 2024-03-18T19:06:33.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.694+0000 7fc9153a3700 1 -- start start 2024-03-18T19:06:33.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.695+0000 7fc9153a3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9101471c0 0x7fc910158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:33.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.695+0000 7fc9153a3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9100a9b40 0x7fc9100a9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:33.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.695+0000 7fc9153a3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9100a84a0 0x7fc9100a8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:33.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.695+0000 7fc9153a3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc910157a10 con 0x7fc9100a9b40 2024-03-18T19:06:33.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.695+0000 7fc9153a3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9100a9860 con 0x7fc9101471c0 2024-03-18T19:06:33.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.695+0000 7fc9153a3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9100aa500 con 0x7fc9100a84a0 2024-03-18T19:06:33.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.695+0000 7fc90e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9100a9b40 0x7fc9100a9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:33.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.695+0000 7fc90e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9100a9b40 0x7fc9100a9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54378/0 (socket says 172.21.15.1:54378) 2024-03-18T19:06:33.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.695+0000 7fc90e7fc700 1 -- 172.21.15.1:0/3150693052 learned_addr learned my addr 172.21.15.1:0/3150693052 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:33.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.695+0000 7fc90f7fe700 1 --2- 172.21.15.1:0/3150693052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9100a84a0 0x7fc9100a8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:33.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.695+0000 7fc90effd700 1 --2- 172.21.15.1:0/3150693052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9101471c0 0x7fc910158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:33.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.695+0000 7fc90e7fc700 1 -- 172.21.15.1:0/3150693052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9100a84a0 msgr2=0x7fc9100a8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:33.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.695+0000 7fc90e7fc700 1 --2- 172.21.15.1:0/3150693052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9100a84a0 0x7fc9100a8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:33.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.695+0000 7fc90e7fc700 1 -- 172.21.15.1:0/3150693052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9101471c0 msgr2=0x7fc910158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:33.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.695+0000 7fc90e7fc700 1 --2- 172.21.15.1:0/3150693052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9101471c0 0x7fc910158670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:33.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.696+0000 7fc90e7fc700 1 -- 172.21.15.1:0/3150693052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc910156d90 con 0x7fc9100a9b40 2024-03-18T19:06:33.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.696+0000 7fc90f7fe700 1 --2- 172.21.15.1:0/3150693052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9100a84a0 0x7fc9100a8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:33.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.696+0000 7fc90effd700 1 --2- 172.21.15.1:0/3150693052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9101471c0 0x7fc910158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:33.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.696+0000 7fc90e7fc700 1 --2- 172.21.15.1:0/3150693052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9100a9b40 0x7fc9100a9fc0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fc90000f7f0 tx=0x7fc90000fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:33.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.696+0000 7fc8f77fe700 1 -- 172.21.15.1:0/3150693052 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9000052e0 con 0x7fc9100a9b40 2024-03-18T19:06:33.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.696+0000 7fc8f77fe700 1 -- 172.21.15.1:0/3150693052 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc900005440 con 0x7fc9100a9b40 2024-03-18T19:06:33.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.696+0000 7fc8f77fe700 1 -- 172.21.15.1:0/3150693052 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9000055b0 con 0x7fc9100a9b40 2024-03-18T19:06:33.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.696+0000 7fc9153a3700 1 -- 172.21.15.1:0/3150693052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc910157070 con 0x7fc9100a9b40 2024-03-18T19:06:33.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.696+0000 7fc9153a3700 1 -- 172.21.15.1:0/3150693052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc910157630 con 0x7fc9100a9b40 2024-03-18T19:06:33.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.697+0000 7fc9153a3700 1 -- 172.21.15.1:0/3150693052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc91006b6e0 con 0x7fc9100a9b40 2024-03-18T19:06:33.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.698+0000 7fc8f77fe700 1 -- 172.21.15.1:0/3150693052 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc900003910 con 0x7fc9100a9b40 2024-03-18T19:06:33.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.698+0000 7fc8f77fe700 1 --2- 172.21.15.1:0/3150693052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc8fc06adb0 0x7fc8fc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:33.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.698+0000 7fc8f77fe700 1 -- 172.21.15.1:0/3150693052 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5865+0+0 (secure 0 0 0) 0x7fc900089340 con 0x7fc9100a9b40 2024-03-18T19:06:33.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.698+0000 7fc90effd700 1 --2- 172.21.15.1:0/3150693052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc8fc06adb0 0x7fc8fc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:33.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.698+0000 7fc90effd700 1 --2- 172.21.15.1:0/3150693052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc8fc06adb0 0x7fc8fc06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc904009c60 tx=0x7fc904008040 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:33.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.703+0000 7fc8f77fe700 1 -- 172.21.15.1:0/3150693052 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc900054400 con 0x7fc9100a9b40 2024-03-18T19:06:33.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:33.991+0000 7fc9153a3700 1 -- 172.21.15.1:0/3150693052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7fc9100676a0 con 0x7fc9100a9b40 2024-03-18T19:06:34.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.448+0000 7fc8f77fe700 1 -- 172.21.15.1:0/3150693052 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v629) v1 ==== 166+0+0 (secure 0 0 0) 0x7fc900057a20 con 0x7fc9100a9b40 2024-03-18T19:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.450+0000 7fc9153a3700 1 -- 172.21.15.1:0/3150693052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc8fc06adb0 msgr2=0x7fc8fc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.450+0000 7fc9153a3700 1 --2- 172.21.15.1:0/3150693052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc8fc06adb0 0x7fc8fc06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc904009c60 tx=0x7fc904008040 comp rx=0 tx=0).stop 2024-03-18T19:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.450+0000 7fc9153a3700 1 -- 172.21.15.1:0/3150693052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9100a9b40 msgr2=0x7fc9100a9fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.450+0000 7fc9153a3700 1 --2- 172.21.15.1:0/3150693052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9100a9b40 0x7fc9100a9fc0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fc90000f7f0 tx=0x7fc90000fbb0 comp rx=0 tx=0).stop 2024-03-18T19:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.451+0000 7fc9153a3700 1 -- 172.21.15.1:0/3150693052 shutdown_connections 2024-03-18T19:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.451+0000 7fc9153a3700 1 --2- 172.21.15.1:0/3150693052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc8fc06adb0 0x7fc8fc06d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.451+0000 7fc9153a3700 1 --2- 172.21.15.1:0/3150693052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9100a84a0 0x7fc9100a8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.451+0000 7fc9153a3700 1 --2- 172.21.15.1:0/3150693052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9101471c0 0x7fc910158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.451+0000 7fc9153a3700 1 --2- 172.21.15.1:0/3150693052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9100a9b40 0x7fc9100a9fc0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.451+0000 7fc9153a3700 1 -- 172.21.15.1:0/3150693052 >> 172.21.15.1:0/3150693052 conn(0x7fc91006a750 msgr2=0x7fc91015a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.451+0000 7fc9153a3700 1 -- 172.21.15.1:0/3150693052 shutdown_connections 2024-03-18T19:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.451+0000 7fc9153a3700 1 -- 172.21.15.1:0/3150693052 wait complete. 2024-03-18T19:06:34.460 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2024-03-18T19:06:34.460 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that a non-existent path leads to schedule deactivation after a few retries. 2024-03-18T19:06:34.460 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-18T19:06:34.460 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2024-03-18T19:06:34.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.594+0000 7f7581045700 1 Processor -- start 2024-03-18T19:06:34.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.594+0000 7f7581045700 1 -- start start 2024-03-18T19:06:34.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.594+0000 7f7581045700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f757c15b3d0 0x7f757c0cf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:34.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.594+0000 7f7581045700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f757c073b00 0x7f757c0cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:34.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.594+0000 7f7581045700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757c0ce200 con 0x7f757c15aff0 2024-03-18T19:06:34.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.594+0000 7f7581045700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f757c0ce4a0 con 0x7f757c073b00 2024-03-18T19:06:34.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.594+0000 7f7581045700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f757c0ce780 con 0x7f757c15b3d0 2024-03-18T19:06:34.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.595+0000 7f757b59e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f757c073b00 0x7f757c0cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:34.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.595+0000 7f757b59e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f757c073b00 0x7f757c0cfa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35790/0 (socket says 172.21.15.1:35790) 2024-03-18T19:06:34.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.595+0000 7f757b59e700 1 -- 172.21.15.1:0/2170323223 learned_addr learned my addr 172.21.15.1:0/2170323223 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:34.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.595+0000 7f757a59c700 1 --2- 172.21.15.1:0/2170323223 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f757c15b3d0 0x7f757c0cf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:34.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.595+0000 7f757b59e700 1 -- 172.21.15.1:0/2170323223 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f757c15b3d0 msgr2=0x7f757c0cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:34.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.595+0000 7f757b59e700 1 --2- 172.21.15.1:0/2170323223 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f757c15b3d0 0x7f757c0cf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:34.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.595+0000 7f757b59e700 1 -- 172.21.15.1:0/2170323223 >> v1:172.21.15.165:6789/0 conn(0x7f757c15aff0 legacy=0x7f757c15f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:34.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.595+0000 7f757b59e700 1 -- 172.21.15.1:0/2170323223 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c0cea60 con 0x7f757c073b00 2024-03-18T19:06:34.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.595+0000 7f757b59e700 1 --2- 172.21.15.1:0/2170323223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f757c073b00 0x7f757c0cfa80 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f7574009e50 tx=0x7f757400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9103976e4189b1bc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:34.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.595+0000 7f7579d9b700 1 -- 172.21.15.1:0/2170323223 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757400e070 con 0x7f757c073b00 2024-03-18T19:06:34.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.595+0000 7f7579d9b700 1 -- 172.21.15.1:0/2170323223 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757400b910 con 0x7f757c073b00 2024-03-18T19:06:34.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.595+0000 7f7579d9b700 1 -- 172.21.15.1:0/2170323223 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757400bcc0 con 0x7f757c073b00 2024-03-18T19:06:34.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.596+0000 7f7581045700 1 -- 172.21.15.1:0/2170323223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f757c073b00 msgr2=0x7f757c0cfa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:34.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.596+0000 7f7581045700 1 --2- 172.21.15.1:0/2170323223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f757c073b00 0x7f757c0cfa80 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f7574009e50 tx=0x7f757400b330 comp rx=0 tx=0).stop 2024-03-18T19:06:34.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.596+0000 7f7581045700 1 -- 172.21.15.1:0/2170323223 shutdown_connections 2024-03-18T19:06:34.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.596+0000 7f7581045700 1 --2- 172.21.15.1:0/2170323223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f757c073b00 0x7f757c0cfa80 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:34.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.596+0000 7f7581045700 1 --2- 172.21.15.1:0/2170323223 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f757c15b3d0 0x7f757c0cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:34.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.596+0000 7f7581045700 1 -- 172.21.15.1:0/2170323223 >> 172.21.15.1:0/2170323223 conn(0x7f757c06a750 msgr2=0x7f757c06ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:34.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.596+0000 7f7581045700 1 -- 172.21.15.1:0/2170323223 shutdown_connections 2024-03-18T19:06:34.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.596+0000 7f7581045700 1 -- 172.21.15.1:0/2170323223 wait complete. 2024-03-18T19:06:34.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.597+0000 7f7581045700 1 Processor -- start 2024-03-18T19:06:34.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.597+0000 7f7581045700 1 -- start start 2024-03-18T19:06:34.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.598+0000 7f7581045700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f757c073b00 0x7f757c138d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:34.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.598+0000 7f7581045700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f757c15aff0 0x7f757c15a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:34.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.598+0000 7f7581045700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f757c15b3d0 0x7f757c1386b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:34.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.598+0000 7f7581045700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c0b2340 con 0x7f757c15b3d0 2024-03-18T19:06:34.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.598+0000 7f7581045700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c138bf0 con 0x7f757c073b00 2024-03-18T19:06:34.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.598+0000 7f7581045700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f757c0b0a20 con 0x7f757c15aff0 2024-03-18T19:06:34.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.598+0000 7f757b59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f757c15b3d0 0x7f757c1386b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.598+0000 7f757b59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f757c15b3d0 0x7f757c1386b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35804/0 (socket says 172.21.15.1:35804) 2024-03-18T19:06:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.598+0000 7f757b59e700 1 -- 172.21.15.1:0/3181524745 learned_addr learned my addr 172.21.15.1:0/3181524745 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.598+0000 7f757a59c700 1 --2- 172.21.15.1:0/3181524745 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f757c15aff0 0x7f757c15a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.598+0000 7f757ad9d700 1 --2- 172.21.15.1:0/3181524745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f757c073b00 0x7f757c138d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.599+0000 7f757b59e700 1 -- 172.21.15.1:0/3181524745 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f757c15aff0 msgr2=0x7f757c15a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.599+0000 7f757b59e700 1 --2- 172.21.15.1:0/3181524745 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f757c15aff0 0x7f757c15a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.599+0000 7f757b59e700 1 -- 172.21.15.1:0/3181524745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f757c073b00 msgr2=0x7f757c138d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.599+0000 7f757b59e700 1 --2- 172.21.15.1:0/3181524745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f757c073b00 0x7f757c138d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.599+0000 7f757b59e700 1 -- 172.21.15.1:0/3181524745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7574009b30 con 0x7f757c15b3d0 2024-03-18T19:06:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.599+0000 7f757b59e700 1 --2- 172.21.15.1:0/3181524745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f757c15b3d0 0x7f757c1386b0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f7574012000 tx=0x7f7574004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.599+0000 7f75637fe700 1 -- 172.21.15.1:0/3181524745 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7574004800 con 0x7f757c15b3d0 2024-03-18T19:06:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.599+0000 7f75637fe700 1 -- 172.21.15.1:0/3181524745 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7574004960 con 0x7f757c15b3d0 2024-03-18T19:06:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.599+0000 7f75637fe700 1 -- 172.21.15.1:0/3181524745 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757401c680 con 0x7f757c15b3d0 2024-03-18T19:06:34.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.599+0000 7f7581045700 1 -- 172.21.15.1:0/3181524745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f757c139600 con 0x7f757c15b3d0 2024-03-18T19:06:34.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.599+0000 7f7581045700 1 -- 172.21.15.1:0/3181524745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f757c0b1d80 con 0x7f757c15b3d0 2024-03-18T19:06:34.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.600+0000 7f75637fe700 1 -- 172.21.15.1:0/3181524745 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f757401c7e0 con 0x7f757c15b3d0 2024-03-18T19:06:34.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.601+0000 7f7581045700 1 -- 172.21.15.1:0/3181524745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f757c0b0bd0 con 0x7f757c15b3d0 2024-03-18T19:06:34.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.601+0000 7f75637fe700 1 --2- 172.21.15.1:0/3181524745 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f756806adb0 0x7f756806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:34.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.601+0000 7f75637fe700 1 -- 172.21.15.1:0/3181524745 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f757408b180 con 0x7f757c15b3d0 2024-03-18T19:06:34.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.601+0000 7f757ad9d700 1 --2- 172.21.15.1:0/3181524745 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f756806adb0 0x7f756806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:34.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.601+0000 7f757ad9d700 1 --2- 172.21.15.1:0/3181524745 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f756806adb0 0x7f756806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f757c06edb0 tx=0x7f756c009040 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:34.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.605+0000 7f75637fe700 1 -- 172.21.15.1:0/3181524745 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f757c0b0bd0 con 0x7f757c15b3d0 2024-03-18T19:06:34.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:34.896+0000 7f7581045700 1 -- 172.21.15.1:0/3181524745 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f757c0b0b90 con 0x7f757c15b3d0 2024-03-18T19:06:35.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.450+0000 7f75637fe700 1 -- 172.21.15.1:0/3181524745 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v630) v1 ==== 186+0+0 (secure 0 0 0) 0x7f7574056130 con 0x7f757c15b3d0 2024-03-18T19:06:35.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.453+0000 7f7581045700 1 -- 172.21.15.1:0/3181524745 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f756806adb0 msgr2=0x7f756806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:35.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.453+0000 7f7581045700 1 --2- 172.21.15.1:0/3181524745 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f756806adb0 0x7f756806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f757c06edb0 tx=0x7f756c009040 comp rx=0 tx=0).stop 2024-03-18T19:06:35.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.453+0000 7f7581045700 1 -- 172.21.15.1:0/3181524745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f757c15b3d0 msgr2=0x7f757c1386b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:35.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.453+0000 7f7581045700 1 --2- 172.21.15.1:0/3181524745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f757c15b3d0 0x7f757c1386b0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f7574012000 tx=0x7f7574004ad0 comp rx=0 tx=0).stop 2024-03-18T19:06:35.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.453+0000 7f7581045700 1 -- 172.21.15.1:0/3181524745 shutdown_connections 2024-03-18T19:06:35.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.453+0000 7f7581045700 1 --2- 172.21.15.1:0/3181524745 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f756806adb0 0x7f756806d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:35.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.453+0000 7f7581045700 1 --2- 172.21.15.1:0/3181524745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f757c15b3d0 0x7f757c1386b0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:35.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.453+0000 7f7581045700 1 --2- 172.21.15.1:0/3181524745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f757c073b00 0x7f757c138d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:35.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.453+0000 7f7581045700 1 --2- 172.21.15.1:0/3181524745 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f757c15aff0 0x7f757c15a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:35.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.453+0000 7f7581045700 1 -- 172.21.15.1:0/3181524745 >> 172.21.15.1:0/3181524745 conn(0x7f757c06a750 msgr2=0x7f757c06dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:35.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.453+0000 7f7581045700 1 -- 172.21.15.1:0/3181524745 shutdown_connections 2024-03-18T19:06:35.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.453+0000 7f7581045700 1 -- 172.21.15.1:0/3181524745 wait complete. 2024-03-18T19:06:35.464 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-18T19:06:35.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.603+0000 7fa9bf351700 1 Processor -- start 2024-03-18T19:06:35.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.603+0000 7fa9bf351700 1 -- start start 2024-03-18T19:06:35.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.604+0000 7fa9bf351700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa9b815d5b0 0x7fa9b815d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:35.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.604+0000 7fa9bf351700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9b8099a40 0x7fa9b8099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:35.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.604+0000 7fa9bf351700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa9b809a3e0 0x7fa9b806ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:35.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.604+0000 7fa9bf351700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b80cbbb0 con 0x7fa9b815d5b0 2024-03-18T19:06:35.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.604+0000 7fa9bf351700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b809a830 con 0x7fa9b809a3e0 2024-03-18T19:06:35.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.604+0000 7fa9bf351700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b80b3b00 con 0x7fa9b8099a40 2024-03-18T19:06:35.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.604+0000 7fa9bd0ed700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa9b815d5b0 0x7fa9b815d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:35.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.604+0000 7fa9bd0ed700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa9b815d5b0 0x7fa9b815d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35808/0 (socket says 172.21.15.1:35808) 2024-03-18T19:06:35.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.604+0000 7fa9bd0ed700 1 -- 172.21.15.1:0/3415846735 learned_addr learned my addr 172.21.15.1:0/3415846735 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:35.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.604+0000 7fa9bc8ec700 1 --2- 172.21.15.1:0/3415846735 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9b8099a40 0x7fa9b8099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:35.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.604+0000 7fa9bd8ee700 1 --2- 172.21.15.1:0/3415846735 >> v2:172.21.15.165:3300/0 conn(0x7fa9b809a3e0 0x7fa9b806ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:35.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.604+0000 7fa9bd0ed700 1 -- 172.21.15.1:0/3415846735 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9b8099a40 msgr2=0x7fa9b8099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:35.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.604+0000 7fa9bd0ed700 1 --2- 172.21.15.1:0/3415846735 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9b8099a40 0x7fa9b8099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:35.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.604+0000 7fa9bd0ed700 1 -- 172.21.15.1:0/3415846735 >> v2:172.21.15.165:3300/0 conn(0x7fa9b809a3e0 msgr2=0x7fa9b806ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:35.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.604+0000 7fa9bd0ed700 1 --2- 172.21.15.1:0/3415846735 >> v2:172.21.15.165:3300/0 conn(0x7fa9b809a3e0 0x7fa9b806ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:35.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.604+0000 7fa9bd0ed700 1 -- 172.21.15.1:0/3415846735 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b80cf7f0 con 0x7fa9b815d5b0 2024-03-18T19:06:35.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.605+0000 7fa9bc8ec700 1 --2- 172.21.15.1:0/3415846735 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9b8099a40 0x7fa9b8099ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:35.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.605+0000 7fa9bd8ee700 1 --2- 172.21.15.1:0/3415846735 >> v2:172.21.15.165:3300/0 conn(0x7fa9b809a3e0 0x7fa9b806ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:35.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.605+0000 7fa9bd0ed700 1 --2- 172.21.15.1:0/3415846735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9b815d5b0 0x7fa9b815d990 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fa9b80720b0 tx=0x7fa9a8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df76bfd8dc82f56f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:35.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.605+0000 7fa9affff700 1 -- 172.21.15.1:0/3415846735 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a800b870 con 0x7fa9b815d5b0 2024-03-18T19:06:35.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.605+0000 7fa9affff700 1 -- 172.21.15.1:0/3415846735 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a800b9d0 con 0x7fa9b815d5b0 2024-03-18T19:06:35.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.605+0000 7fa9affff700 1 -- 172.21.15.1:0/3415846735 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a800bd60 con 0x7fa9b815d5b0 2024-03-18T19:06:35.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.605+0000 7fa9bf351700 1 -- 172.21.15.1:0/3415846735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9b815d5b0 msgr2=0x7fa9b815d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:35.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.605+0000 7fa9bf351700 1 --2- 172.21.15.1:0/3415846735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9b815d5b0 0x7fa9b815d990 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fa9b80720b0 tx=0x7fa9a8009bd0 comp rx=0 tx=0).stop 2024-03-18T19:06:35.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.605+0000 7fa9bf351700 1 -- 172.21.15.1:0/3415846735 shutdown_connections 2024-03-18T19:06:35.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.605+0000 7fa9bf351700 1 --2- 172.21.15.1:0/3415846735 >> v2:172.21.15.165:3300/0 conn(0x7fa9b809a3e0 0x7fa9b806ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:35.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.605+0000 7fa9bf351700 1 --2- 172.21.15.1:0/3415846735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9b815d5b0 0x7fa9b815d990 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:35.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.605+0000 7fa9bf351700 1 --2- 172.21.15.1:0/3415846735 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9b8099a40 0x7fa9b8099ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:35.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.605+0000 7fa9bf351700 1 -- 172.21.15.1:0/3415846735 >> 172.21.15.1:0/3415846735 conn(0x7fa9b80b2bd0 msgr2=0x7fa9b80b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:35.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.606+0000 7fa9bf351700 1 -- 172.21.15.1:0/3415846735 shutdown_connections 2024-03-18T19:06:35.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.606+0000 7fa9bf351700 1 -- 172.21.15.1:0/3415846735 wait complete. 2024-03-18T19:06:35.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.607+0000 7fa9bf351700 1 Processor -- start 2024-03-18T19:06:35.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.607+0000 7fa9bf351700 1 -- start start 2024-03-18T19:06:35.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9bf351700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9b8099a40 0x7fa9b80ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:35.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9bf351700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9b809a3e0 0x7fa9b80b1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:35.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9bf351700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9b815d5b0 0x7fa9b81f57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:35.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9bf351700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b80ae150 con 0x7fa9b809a3e0 2024-03-18T19:06:35.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9bf351700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b80ae430 con 0x7fa9b8099a40 2024-03-18T19:06:35.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9bf351700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b80ae710 con 0x7fa9b815d5b0 2024-03-18T19:06:35.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9bc8ec700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9b809a3e0 0x7fa9b80b1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:35.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9bc8ec700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9b809a3e0 0x7fa9b80b1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35818/0 (socket says 172.21.15.1:35818) 2024-03-18T19:06:35.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9bc8ec700 1 -- 172.21.15.1:0/3605286017 learned_addr learned my addr 172.21.15.1:0/3605286017 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:35.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9bd8ee700 1 --2- 172.21.15.1:0/3605286017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9b815d5b0 0x7fa9b81f57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:35.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9bd0ed700 1 --2- 172.21.15.1:0/3605286017 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9b8099a40 0x7fa9b80ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:35.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9bc8ec700 1 -- 172.21.15.1:0/3605286017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9b815d5b0 msgr2=0x7fa9b81f57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:35.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9bc8ec700 1 --2- 172.21.15.1:0/3605286017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9b815d5b0 0x7fa9b81f57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:35.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9bc8ec700 1 -- 172.21.15.1:0/3605286017 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9b8099a40 msgr2=0x7fa9b80ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:35.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9bc8ec700 1 --2- 172.21.15.1:0/3605286017 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9b8099a40 0x7fa9b80ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:35.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9bc8ec700 1 -- 172.21.15.1:0/3605286017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a8009550 con 0x7fa9b809a3e0 2024-03-18T19:06:35.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9bc8ec700 1 --2- 172.21.15.1:0/3605286017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9b809a3e0 0x7fa9b80b1510 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fa9b400c7f0 tx=0x7fa9b400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:35.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9adffb700 1 -- 172.21.15.1:0/3605286017 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b4011800 con 0x7fa9b809a3e0 2024-03-18T19:06:35.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9adffb700 1 -- 172.21.15.1:0/3605286017 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b4011960 con 0x7fa9b809a3e0 2024-03-18T19:06:35.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9bf351700 1 -- 172.21.15.1:0/3605286017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b80ae9f0 con 0x7fa9b809a3e0 2024-03-18T19:06:35.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.608+0000 7fa9adffb700 1 -- 172.21.15.1:0/3605286017 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b4011d10 con 0x7fa9b809a3e0 2024-03-18T19:06:35.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.609+0000 7fa9bf351700 1 -- 172.21.15.1:0/3605286017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b80add90 con 0x7fa9b809a3e0 2024-03-18T19:06:35.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.610+0000 7fa9bf351700 1 -- 172.21.15.1:0/3605286017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b80aef10 con 0x7fa9b809a3e0 2024-03-18T19:06:35.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.610+0000 7fa9adffb700 1 -- 172.21.15.1:0/3605286017 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa9b4018360 con 0x7fa9b809a3e0 2024-03-18T19:06:35.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.610+0000 7fa9adffb700 1 --2- 172.21.15.1:0/3605286017 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa99c06adb0 0x7fa99c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:35.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.611+0000 7fa9bd0ed700 1 --2- 172.21.15.1:0/3605286017 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa99c06adb0 0x7fa99c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:35.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.611+0000 7fa9adffb700 1 -- 172.21.15.1:0/3605286017 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5865+0+0 (secure 0 0 0) 0x7fa9b4089620 con 0x7fa9b809a3e0 2024-03-18T19:06:35.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.611+0000 7fa9bd0ed700 1 --2- 172.21.15.1:0/3605286017 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa99c06adb0 0x7fa99c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009130 tx=0x7fa9a8008030 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:35.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.615+0000 7fa9adffb700 1 -- 172.21.15.1:0/3605286017 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa9b80aef10 con 0x7fa9b809a3e0 2024-03-18T19:06:35.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.816+0000 7fa9bf351700 1 -- 172.21.15.1:0/3605286017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa9b80aef10 con 0x7fa9b809a3e0 2024-03-18T19:06:35.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.820+0000 7fa9adffb700 1 -- 172.21.15.1:0/3605286017 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa9b80aef10 con 0x7fa9b809a3e0 2024-03-18T19:06:35.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.822+0000 7fa9bf351700 1 -- 172.21.15.1:0/3605286017 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa99c06adb0 msgr2=0x7fa99c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:35.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.822+0000 7fa9bf351700 1 --2- 172.21.15.1:0/3605286017 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa99c06adb0 0x7fa99c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009130 tx=0x7fa9a8008030 comp rx=0 tx=0).stop 2024-03-18T19:06:35.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.822+0000 7fa9bf351700 1 -- 172.21.15.1:0/3605286017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9b809a3e0 msgr2=0x7fa9b80b1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:35.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.822+0000 7fa9bf351700 1 --2- 172.21.15.1:0/3605286017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9b809a3e0 0x7fa9b80b1510 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fa9b400c7f0 tx=0x7fa9b400cbb0 comp rx=0 tx=0).stop 2024-03-18T19:06:35.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.822+0000 7fa9bf351700 1 -- 172.21.15.1:0/3605286017 shutdown_connections 2024-03-18T19:06:35.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.822+0000 7fa9bf351700 1 --2- 172.21.15.1:0/3605286017 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa99c06adb0 0x7fa99c06d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:35.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.822+0000 7fa9bf351700 1 --2- 172.21.15.1:0/3605286017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9b815d5b0 0x7fa9b81f57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:35.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.822+0000 7fa9bf351700 1 --2- 172.21.15.1:0/3605286017 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9b8099a40 0x7fa9b80ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:35.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.822+0000 7fa9bf351700 1 --2- 172.21.15.1:0/3605286017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9b809a3e0 0x7fa9b80b1510 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:35.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.822+0000 7fa9bf351700 1 -- 172.21.15.1:0/3605286017 >> 172.21.15.1:0/3605286017 conn(0x7fa9b80b2bd0 msgr2=0x7fa9b80c6b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:35.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.822+0000 7fa9bf351700 1 -- 172.21.15.1:0/3605286017 shutdown_connections 2024-03-18T19:06:35.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:35.822+0000 7fa9bf351700 1 -- 172.21.15.1:0/3605286017 wait complete. 2024-03-18T19:06:35.830 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T19:06:35.830 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:06:35.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:06:35.922 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T19:06:35.923 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:06:35.923 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:06:35.941 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:06:35.974 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:06:35.974 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:06:35.995 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:06:35.995 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:06:35.995 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:06:36.016 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:06:36.016 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:06:36.016 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:06:36.057 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:06:36.058 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-18T19:06:36.058 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:06:36.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.187+0000 7fb09145b700 1 Processor -- start 2024-03-18T19:06:36.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.188+0000 7fb09145b700 1 -- start start 2024-03-18T19:06:36.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.188+0000 7fb09145b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb08c15d500 0x7fb08c0d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:36.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.188+0000 7fb09145b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb08c0d1300 0x7fb08c0d17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:36.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.188+0000 7fb09145b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb08c0d2120 con 0x7fb08c0d1cf0 2024-03-18T19:06:36.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.188+0000 7fb09145b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb08c0cddc0 con 0x7fb08c0d1300 2024-03-18T19:06:36.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.188+0000 7fb09145b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb08c0d3dc0 con 0x7fb08c15d500 2024-03-18T19:06:36.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.188+0000 7fb08a7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb08c0d1300 0x7fb08c0d17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:36.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.188+0000 7fb08a7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb08c0d1300 0x7fb08c0d17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35824/0 (socket says 172.21.15.1:35824) 2024-03-18T19:06:36.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.188+0000 7fb08a7fc700 1 -- 172.21.15.1:0/4205642416 learned_addr learned my addr 172.21.15.1:0/4205642416 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:36.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.189+0000 7fb08affd700 1 --2- 172.21.15.1:0/4205642416 >> v2:172.21.15.165:3300/0 conn(0x7fb08c15d500 0x7fb08c0d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:36.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.189+0000 7fb08a7fc700 1 -- 172.21.15.1:0/4205642416 >> v2:172.21.15.165:3300/0 conn(0x7fb08c15d500 msgr2=0x7fb08c0d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:36.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.189+0000 7fb08a7fc700 1 --2- 172.21.15.1:0/4205642416 >> v2:172.21.15.165:3300/0 conn(0x7fb08c15d500 0x7fb08c0d0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:36.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.189+0000 7fb08a7fc700 1 -- 172.21.15.1:0/4205642416 >> v1:172.21.15.1:6789/0 conn(0x7fb08c0d1cf0 legacy=0x7fb08c0b83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:36.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.189+0000 7fb08a7fc700 1 -- 172.21.15.1:0/4205642416 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb08c0ce060 con 0x7fb08c0d1300 2024-03-18T19:06:36.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.189+0000 7fb08a7fc700 1 --2- 172.21.15.1:0/4205642416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb08c0d1300 0x7fb08c0d17b0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fb08c08b740 tx=0x7fb074009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1505721fdd2ec9c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:36.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.189+0000 7fb089ffb700 1 -- 172.21.15.1:0/4205642416 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07400b870 con 0x7fb08c0d1300 2024-03-18T19:06:36.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.189+0000 7fb089ffb700 1 -- 172.21.15.1:0/4205642416 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb07400b9d0 con 0x7fb08c0d1300 2024-03-18T19:06:36.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.189+0000 7fb089ffb700 1 -- 172.21.15.1:0/4205642416 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07400bd60 con 0x7fb08c0d1300 2024-03-18T19:06:36.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.190+0000 7fb09145b700 1 -- 172.21.15.1:0/4205642416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb08c0d1300 msgr2=0x7fb08c0d17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:36.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.190+0000 7fb09145b700 1 --2- 172.21.15.1:0/4205642416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb08c0d1300 0x7fb08c0d17b0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fb08c08b740 tx=0x7fb074009bd0 comp rx=0 tx=0).stop 2024-03-18T19:06:36.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.190+0000 7fb09145b700 1 -- 172.21.15.1:0/4205642416 shutdown_connections 2024-03-18T19:06:36.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.190+0000 7fb09145b700 1 --2- 172.21.15.1:0/4205642416 >> v2:172.21.15.165:3300/0 conn(0x7fb08c15d500 0x7fb08c0d0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:36.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.190+0000 7fb09145b700 1 --2- 172.21.15.1:0/4205642416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb08c0d1300 0x7fb08c0d17b0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:36.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.190+0000 7fb09145b700 1 -- 172.21.15.1:0/4205642416 >> 172.21.15.1:0/4205642416 conn(0x7fb08c09c530 msgr2=0x7fb08c0d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:36.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.190+0000 7fb09145b700 1 -- 172.21.15.1:0/4205642416 shutdown_connections 2024-03-18T19:06:36.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.190+0000 7fb09145b700 1 -- 172.21.15.1:0/4205642416 wait complete. 2024-03-18T19:06:36.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.191+0000 7fb09145b700 1 Processor -- start 2024-03-18T19:06:36.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.191+0000 7fb09145b700 1 -- start start 2024-03-18T19:06:36.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.192+0000 7fb09145b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb08c0d1300 0x7fb08c0b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:36.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.192+0000 7fb09145b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb08c0d1cf0 0x7fb08c0cba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:36.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.192+0000 7fb09145b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb08c15d500 0x7fb08c14f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:36.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.192+0000 7fb09145b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08c0b2ec0 con 0x7fb08c15d500 2024-03-18T19:06:36.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.192+0000 7fb09145b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08c0b31a0 con 0x7fb08c0d1300 2024-03-18T19:06:36.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.192+0000 7fb09145b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb08c0b3480 con 0x7fb08c0d1cf0 2024-03-18T19:06:36.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.192+0000 7fb08b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb08c15d500 0x7fb08c14f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:36.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.192+0000 7fb08b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb08c15d500 0x7fb08c14f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35830/0 (socket says 172.21.15.1:35830) 2024-03-18T19:06:36.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.192+0000 7fb08b7fe700 1 -- 172.21.15.1:0/2692135005 learned_addr learned my addr 172.21.15.1:0/2692135005 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:36.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.192+0000 7fb08affd700 1 --2- 172.21.15.1:0/2692135005 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb08c0d1300 0x7fb08c0b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:36.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.192+0000 7fb08a7fc700 1 --2- 172.21.15.1:0/2692135005 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb08c0d1cf0 0x7fb08c0cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:36.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.192+0000 7fb08b7fe700 1 -- 172.21.15.1:0/2692135005 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb08c0d1cf0 msgr2=0x7fb08c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:36.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.192+0000 7fb08b7fe700 1 --2- 172.21.15.1:0/2692135005 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb08c0d1cf0 0x7fb08c0cba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:36.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.192+0000 7fb08b7fe700 1 -- 172.21.15.1:0/2692135005 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb08c0d1300 msgr2=0x7fb08c0b27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:36.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.192+0000 7fb08b7fe700 1 --2- 172.21.15.1:0/2692135005 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb08c0d1300 0x7fb08c0b27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:36.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.192+0000 7fb08b7fe700 1 -- 172.21.15.1:0/2692135005 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb074009550 con 0x7fb08c15d500 2024-03-18T19:06:36.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.192+0000 7fb08affd700 1 --2- 172.21.15.1:0/2692135005 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb08c0d1300 0x7fb08c0b27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:36.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.192+0000 7fb08b7fe700 1 --2- 172.21.15.1:0/2692135005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb08c15d500 0x7fb08c14f140 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fb08400cc50 tx=0x7fb08400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:36.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.193+0000 7fb0837fe700 1 -- 172.21.15.1:0/2692135005 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08400ad80 con 0x7fb08c15d500 2024-03-18T19:06:36.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.193+0000 7fb0837fe700 1 -- 172.21.15.1:0/2692135005 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb08400ce50 con 0x7fb08c15d500 2024-03-18T19:06:36.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.193+0000 7fb09145b700 1 -- 172.21.15.1:0/2692135005 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb08c0b3760 con 0x7fb08c15d500 2024-03-18T19:06:36.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.193+0000 7fb0837fe700 1 -- 172.21.15.1:0/2692135005 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb084003c40 con 0x7fb08c15d500 2024-03-18T19:06:36.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.193+0000 7fb09145b700 1 -- 172.21.15.1:0/2692135005 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb08c15c8c0 con 0x7fb08c15d500 2024-03-18T19:06:36.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.194+0000 7fb09145b700 1 -- 172.21.15.1:0/2692135005 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb08c0b3c80 con 0x7fb08c15d500 2024-03-18T19:06:36.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.194+0000 7fb0837fe700 1 -- 172.21.15.1:0/2692135005 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb0840050d0 con 0x7fb08c15d500 2024-03-18T19:06:36.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.195+0000 7fb0837fe700 1 --2- 172.21.15.1:0/2692135005 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb06c06adb0 0x7fb06c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:36.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.195+0000 7fb08affd700 1 --2- 172.21.15.1:0/2692135005 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb06c06adb0 0x7fb06c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:36.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.195+0000 7fb0837fe700 1 -- 172.21.15.1:0/2692135005 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5865+0+0 (secure 0 0 0) 0x7fb0840895d0 con 0x7fb08c15d500 2024-03-18T19:06:36.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.195+0000 7fb08affd700 1 --2- 172.21.15.1:0/2692135005 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb06c06adb0 0x7fb06c06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb07c009820 tx=0x7fb07c008040 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:36.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.199+0000 7fb0837fe700 1 -- 172.21.15.1:0/2692135005 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb08c0b3c80 con 0x7fb08c15d500 2024-03-18T19:06:36.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.444+0000 7fb09145b700 1 -- 172.21.15.1:0/2692135005 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb08c0b3c80 con 0x7fb08c15d500 2024-03-18T19:06:36.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.444+0000 7fb0837fe700 1 -- 172.21.15.1:0/2692135005 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+3529 (secure 0 0 0) 0x7fb08c0b3c80 con 0x7fb08c15d500 2024-03-18T19:06:36.440 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:36.441 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":44,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6802,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/3926877640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3926877640},{"type":"v1","addr":"172.21.15.165:6833","nonce":3926877640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6811,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/4293662065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":4293662065},{"type":"v1","addr":"172.21.15.165:6835","nonce":4293662065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2334043797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2334043797},{"type":"v1","addr":"172.21.15.1:6835","nonce":2334043797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":7677,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2596692225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":2596692225},{"type":"v1","addr":"172.21.15.1:6837","nonce":2596692225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[]} 2024-03-18T19:06:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.447+0000 7fb09145b700 1 -- 172.21.15.1:0/2692135005 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb06c06adb0 msgr2=0x7fb06c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.447+0000 7fb09145b700 1 --2- 172.21.15.1:0/2692135005 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb06c06adb0 0x7fb06c06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb07c009820 tx=0x7fb07c008040 comp rx=0 tx=0).stop 2024-03-18T19:06:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.447+0000 7fb09145b700 1 -- 172.21.15.1:0/2692135005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb08c15d500 msgr2=0x7fb08c14f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.447+0000 7fb09145b700 1 --2- 172.21.15.1:0/2692135005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb08c15d500 0x7fb08c14f140 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fb08400cc50 tx=0x7fb08400a340 comp rx=0 tx=0).stop 2024-03-18T19:06:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.447+0000 7fb09145b700 1 -- 172.21.15.1:0/2692135005 shutdown_connections 2024-03-18T19:06:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.447+0000 7fb09145b700 1 --2- 172.21.15.1:0/2692135005 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb06c06adb0 0x7fb06c06d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.447+0000 7fb09145b700 1 --2- 172.21.15.1:0/2692135005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb08c15d500 0x7fb08c14f140 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.447+0000 7fb09145b700 1 --2- 172.21.15.1:0/2692135005 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb08c0d1300 0x7fb08c0b27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.447+0000 7fb09145b700 1 --2- 172.21.15.1:0/2692135005 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb08c0d1cf0 0x7fb08c0cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.447+0000 7fb09145b700 1 -- 172.21.15.1:0/2692135005 >> 172.21.15.1:0/2692135005 conn(0x7fb08c09c530 msgr2=0x7fb08c06cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.447+0000 7fb09145b700 1 -- 172.21.15.1:0/2692135005 shutdown_connections 2024-03-18T19:06:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.447+0000 7fb09145b700 1 -- 172.21.15.1:0/2692135005 wait complete. 2024-03-18T19:06:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 44 2024-03-18T19:06:36.455 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T19:06:36.456 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-18T19:06:36.456 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T19:06:36.456 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:06:36.457 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T19:06:36.457 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-18T19:06:36.457 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-18T19:06:36.457 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:06:36.457 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T19:06:36.457 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-18T19:06:36.458 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-18T19:06:36.458 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:06:36.458 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T19:06:36.458 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-18T19:06:36.458 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-18T19:06:36.458 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:06:36.465 INFO:tasks.ceph.mds.a:Stopped 2024-03-18T19:06:36.465 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-18T19:06:36.466 INFO:tasks.ceph.mds.c:Stopped 2024-03-18T19:06:36.466 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-18T19:06:36.471 INFO:tasks.ceph.mds.b:Stopped 2024-03-18T19:06:36.471 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-18T19:06:36.472 INFO:tasks.ceph.mds.d:Stopped 2024-03-18T19:06:36.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-18T19:06:36.507 INFO:tasks.ceph.mds.a:Started 2024-03-18T19:06:36.507 DEBUG:teuthology.parallel:result is None 2024-03-18T19:06:36.508 INFO:tasks.ceph.mds.c:Started 2024-03-18T19:06:36.508 DEBUG:teuthology.parallel:result is None 2024-03-18T19:06:36.515 INFO:tasks.ceph.mds.b:Started 2024-03-18T19:06:36.515 DEBUG:teuthology.parallel:result is None 2024-03-18T19:06:36.515 INFO:tasks.ceph.mds.d:Started 2024-03-18T19:06:36.516 DEBUG:teuthology.parallel:result is None 2024-03-18T19:06:36.516 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-18T19:06:36.590 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-18T19:06:36.590 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T19:06:36.602 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-03-18T19:06:36.602 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-03-18T19:06:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.657+0000 7f2a8abf5700 1 Processor -- start 2024-03-18T19:06:36.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.657+0000 7f2a8abf5700 1 -- start start 2024-03-18T19:06:36.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.658+0000 7f2a8abf5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2a84071550 0x7f2a840b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:36.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.658+0000 7f2a8abf5700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a840b2b80 con 0x7f2a8415d3b0 2024-03-18T19:06:36.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.658+0000 7f2a8abf5700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a840b2e60 con 0x7f2a840b2650 2024-03-18T19:06:36.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.658+0000 7f2a8abf5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a840b3800 con 0x7f2a84071550 2024-03-18T19:06:36.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.658+0000 7f2a88991700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2a8415d3b0 0x7f2a8406cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55228/0 (socket says 172.21.15.1:55228) 2024-03-18T19:06:36.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.658+0000 7f2a88991700 1 -- 172.21.15.1:0/2717941409 learned_addr learned my addr 172.21.15.1:0/2717941409 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.658+0000 7f2a89192700 1 --2- 172.21.15.1:0/2717941409 >> v2:172.21.15.165:3300/0 conn(0x7f2a84071550 0x7f2a840b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.658+0000 7f2a837fe700 1 -- 172.21.15.1:0/2717941409 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799504993 0 0) 0x7f2a840b2b80 con 0x7f2a8415d3b0 2024-03-18T19:06:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.658+0000 7f2a837fe700 1 -- 172.21.15.1:0/2717941409 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a740035b0 con 0x7f2a8415d3b0 2024-03-18T19:06:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.658+0000 7f2a837fe700 1 -- 172.21.15.1:0/2717941409 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3788043354 0 0) 0x7f2a740035b0 con 0x7f2a8415d3b0 2024-03-18T19:06:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.658+0000 7f2a837fe700 1 -- 172.21.15.1:0/2717941409 >> v2:172.21.15.165:3300/0 conn(0x7f2a84071550 msgr2=0x7f2a840b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.658+0000 7f2a837fe700 1 --2- 172.21.15.1:0/2717941409 >> v2:172.21.15.165:3300/0 conn(0x7f2a84071550 0x7f2a840b3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.658+0000 7f2a837fe700 1 -- 172.21.15.1:0/2717941409 >> v1:172.21.15.165:6789/0 conn(0x7f2a840b2650 legacy=0x7f2a84070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a837fe700 1 -- 172.21.15.1:0/2717941409 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a840b3140 con 0x7f2a8415d3b0 2024-03-18T19:06:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a837fe700 1 -- 172.21.15.1:0/2717941409 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f2a78001670 con 0x7f2a8415d3b0 2024-03-18T19:06:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a837fe700 1 -- 172.21.15.1:0/2717941409 >> v1:172.21.15.1:6789/0 conn(0x7f2a8415d3b0 legacy=0x7f2a8406cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a837fe700 1 --2- 172.21.15.1:0/2717941409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a74003a00 0x7f2a74003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a837fe700 1 --2- 172.21.15.1:0/2717941409 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2a74004850 0x7f2a74004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:36.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a837fe700 1 --2- 172.21.15.1:0/2717941409 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2a74005210 0x7f2a7400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a837fe700 1 -- 172.21.15.1:0/2717941409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a840b3800 con 0x7f2a74003a00 2024-03-18T19:06:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a837fe700 1 -- 172.21.15.1:0/2717941409 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7400c0b0 con 0x7f2a74004850 2024-03-18T19:06:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a837fe700 1 -- 172.21.15.1:0/2717941409 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7400c270 con 0x7f2a74005210 2024-03-18T19:06:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a88991700 1 --2- 172.21.15.1:0/2717941409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a74003a00 0x7f2a74003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a83fff700 1 --2- 172.21.15.1:0/2717941409 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2a74004850 0x7f2a74004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a89192700 1 --2- 172.21.15.1:0/2717941409 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2a74005210 0x7f2a7400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a88991700 1 -- 172.21.15.1:0/2717941409 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2a74005210 msgr2=0x7f2a7400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a88991700 1 --2- 172.21.15.1:0/2717941409 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2a74005210 0x7f2a7400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a88991700 1 -- 172.21.15.1:0/2717941409 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2a74004850 msgr2=0x7f2a74004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a88991700 1 --2- 172.21.15.1:0/2717941409 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2a74004850 0x7f2a74004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a88991700 1 -- 172.21.15.1:0/2717941409 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a740035b0 con 0x7f2a74003a00 2024-03-18T19:06:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a89192700 1 -- 172.21.15.1:0/2717941409 reap_dead start 2024-03-18T19:06:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a83fff700 1 --2- 172.21.15.1:0/2717941409 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2a74004850 0x7f2a74004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:06:36.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a88991700 1 --2- 172.21.15.1:0/2717941409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a74003a00 0x7f2a74003e20 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f2a7800c240 tx=0x7f2a74005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a29157895df4801 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a837fe700 1 -- 172.21.15.1:0/2717941409 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a78007c10 con 0x7f2a74003a00 2024-03-18T19:06:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a837fe700 1 -- 172.21.15.1:0/2717941409 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7800cd60 con 0x7f2a74003a00 2024-03-18T19:06:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.659+0000 7f2a837fe700 1 -- 172.21.15.1:0/2717941409 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a78005ba0 con 0x7f2a74003a00 2024-03-18T19:06:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.660+0000 7f2a8abf5700 1 -- 172.21.15.1:0/2717941409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a74003a00 msgr2=0x7f2a74003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.660+0000 7f2a8abf5700 1 --2- 172.21.15.1:0/2717941409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a74003a00 0x7f2a74003e20 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f2a7800c240 tx=0x7f2a74005880 comp rx=0 tx=0).stop 2024-03-18T19:06:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.660+0000 7f2a8abf5700 1 -- 172.21.15.1:0/2717941409 shutdown_connections 2024-03-18T19:06:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.660+0000 7f2a8abf5700 1 --2- 172.21.15.1:0/2717941409 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a74003a00 0x7f2a74003e20 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.660+0000 7f2a8abf5700 1 -- 172.21.15.1:0/2717941409 >> 172.21.15.1:0/2717941409 conn(0x7f2a840b1350 msgr2=0x7f2a840b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.660+0000 7f2a8abf5700 1 -- 172.21.15.1:0/2717941409 shutdown_connections 2024-03-18T19:06:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.660+0000 7f2a8abf5700 1 -- 172.21.15.1:0/2717941409 wait complete. 2024-03-18T19:06:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.661+0000 7f2a8abf5700 1 Processor -- start 2024-03-18T19:06:36.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.661+0000 7f2a8abf5700 1 -- start start 2024-03-18T19:06:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.662+0000 7f2a8abf5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2a74003a00 0x7f2a841f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.662+0000 7f2a8abf5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2a840aff70 0x7f2a840b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.662+0000 7f2a8abf5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a84154140 0x7f2a841545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.662+0000 7f2a8abf5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a840b0930 con 0x7f2a84154140 2024-03-18T19:06:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.662+0000 7f2a8abf5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a84154b00 con 0x7f2a74003a00 2024-03-18T19:06:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.662+0000 7f2a8abf5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a84155f20 con 0x7f2a840aff70 2024-03-18T19:06:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.662+0000 7f2a89192700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a84154140 0x7f2a841545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.662+0000 7f2a83fff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2a840aff70 0x7f2a840b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.662+0000 7f2a88991700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2a74003a00 0x7f2a841f24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:36.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.662+0000 7f2a89192700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a84154140 0x7f2a841545c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35888/0 (socket says 172.21.15.1:35888) 2024-03-18T19:06:36.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.662+0000 7f2a83fff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2a840aff70 0x7f2a840b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:35766/0 (socket says 172.21.15.1:35766) 2024-03-18T19:06:36.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.662+0000 7f2a89192700 1 -- 172.21.15.1:0/1691898475 learned_addr learned my addr 172.21.15.1:0/1691898475 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:36.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.662+0000 7f2a89192700 1 -- 172.21.15.1:0/1691898475 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2a840aff70 msgr2=0x7f2a840b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:36.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.662+0000 7f2a89192700 1 --2- 172.21.15.1:0/1691898475 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2a840aff70 0x7f2a840b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:36.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.662+0000 7f2a89192700 1 -- 172.21.15.1:0/1691898475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2a74003a00 msgr2=0x7f2a841f24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:36.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.662+0000 7f2a89192700 1 --2- 172.21.15.1:0/1691898475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2a74003a00 0x7f2a841f24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:36.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.663+0000 7f2a89192700 1 -- 172.21.15.1:0/1691898475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a84138f00 con 0x7f2a84154140 2024-03-18T19:06:36.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.663+0000 7f2a89192700 1 --2- 172.21.15.1:0/1691898475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a84154140 0x7f2a841545c0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f2a8406cb40 tx=0x7f2a7c011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:36.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.663+0000 7f2a817fa700 1 -- 172.21.15.1:0/1691898475 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c00e040 con 0x7f2a84154140 2024-03-18T19:06:36.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.663+0000 7f2a817fa700 1 -- 172.21.15.1:0/1691898475 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7c00f040 con 0x7f2a84154140 2024-03-18T19:06:36.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.663+0000 7f2a8abf5700 1 -- 172.21.15.1:0/1691898475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a84154d20 con 0x7f2a84154140 2024-03-18T19:06:36.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.663+0000 7f2a817fa700 1 -- 172.21.15.1:0/1691898475 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c00fc10 con 0x7f2a84154140 2024-03-18T19:06:36.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.663+0000 7f2a8abf5700 1 -- 172.21.15.1:0/1691898475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a84156710 con 0x7f2a84154140 2024-03-18T19:06:36.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.664+0000 7f2a8abf5700 1 -- 172.21.15.1:0/1691898475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a840676a0 con 0x7f2a84154140 2024-03-18T19:06:36.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.664+0000 7f2a817fa700 1 -- 172.21.15.1:0/1691898475 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2a7c00fd70 con 0x7f2a84154140 2024-03-18T19:06:36.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.665+0000 7f2a817fa700 1 --2- 172.21.15.1:0/1691898475 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2a6806adb0 0x7f2a6806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:36.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.665+0000 7f2a817fa700 1 -- 172.21.15.1:0/1691898475 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5865+0+0 (secure 0 0 0) 0x7f2a7c089a00 con 0x7f2a84154140 2024-03-18T19:06:36.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.665+0000 7f2a88991700 1 --2- 172.21.15.1:0/1691898475 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2a6806adb0 0x7f2a6806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:36.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.665+0000 7f2a88991700 1 --2- 172.21.15.1:0/1691898475 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2a6806adb0 0x7f2a6806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2a78000c00 tx=0x7f2a78007000 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:36.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.669+0000 7f2a817fa700 1 -- 172.21.15.1:0/1691898475 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2a7c054880 con 0x7f2a84154140 2024-03-18T19:06:36.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:36.888+0000 7f2a8abf5700 1 -- 172.21.15.1:0/1691898475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2a840b0cf0 con 0x7f2a84154140 2024-03-18T19:06:37.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.823+0000 7f2a817fa700 1 -- 172.21.15.1:0/1691898475 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v113) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2a7c057ea0 con 0x7f2a84154140 2024-03-18T19:06:37.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.826+0000 7f2a8abf5700 1 -- 172.21.15.1:0/1691898475 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2a6806adb0 msgr2=0x7f2a6806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:37.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.826+0000 7f2a8abf5700 1 --2- 172.21.15.1:0/1691898475 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2a6806adb0 0x7f2a6806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2a78000c00 tx=0x7f2a78007000 comp rx=0 tx=0).stop 2024-03-18T19:06:37.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.826+0000 7f2a8abf5700 1 -- 172.21.15.1:0/1691898475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a84154140 msgr2=0x7f2a841545c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:37.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.826+0000 7f2a8abf5700 1 --2- 172.21.15.1:0/1691898475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a84154140 0x7f2a841545c0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f2a8406cb40 tx=0x7f2a7c011e20 comp rx=0 tx=0).stop 2024-03-18T19:06:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.826+0000 7f2a8abf5700 1 -- 172.21.15.1:0/1691898475 shutdown_connections 2024-03-18T19:06:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.826+0000 7f2a8abf5700 1 --2- 172.21.15.1:0/1691898475 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2a6806adb0 0x7f2a6806d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.826+0000 7f2a8abf5700 1 --2- 172.21.15.1:0/1691898475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a84154140 0x7f2a841545c0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.826+0000 7f2a8abf5700 1 --2- 172.21.15.1:0/1691898475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2a74003a00 0x7f2a841f24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.826+0000 7f2a8abf5700 1 --2- 172.21.15.1:0/1691898475 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2a840aff70 0x7f2a840b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.826+0000 7f2a8abf5700 1 -- 172.21.15.1:0/1691898475 >> 172.21.15.1:0/1691898475 conn(0x7f2a840b1350 msgr2=0x7f2a8406efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.826+0000 7f2a8abf5700 1 -- 172.21.15.1:0/1691898475 shutdown_connections 2024-03-18T19:06:37.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.826+0000 7f2a8abf5700 1 -- 172.21.15.1:0/1691898475 wait complete. 2024-03-18T19:06:37.823 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-18T19:06:37.839 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-18T19:06:37.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.978+0000 7f07f5050700 1 Processor -- start 2024-03-18T19:06:37.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.978+0000 7f07f5050700 1 -- start start 2024-03-18T19:06:37.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.979+0000 7f07f5050700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07f00d05a0 0x7f07f009a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:37.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.979+0000 7f07f5050700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f07f006ba70 0x7f07f015d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.979+0000 7f07f5050700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f00cf7e0 con 0x7f07f0099a40 2024-03-18T19:06:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.979+0000 7f07f5050700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07f00cfac0 con 0x7f07f006ba70 2024-03-18T19:06:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.979+0000 7f07f5050700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f00cfda0 con 0x7f07f00d05a0 2024-03-18T19:06:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.980+0000 7f07ef59e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f07f006ba70 0x7f07f015d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.980+0000 7f07ee59c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07f00d05a0 0x7f07f009a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.980+0000 7f07ef59e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f07f006ba70 0x7f07f015d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:53756/0 (socket says 172.21.15.1:53756) 2024-03-18T19:06:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.980+0000 7f07ee59c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07f00d05a0 0x7f07f009a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:35774/0 (socket says 172.21.15.1:35774) 2024-03-18T19:06:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.980+0000 7f07ef59e700 1 -- 172.21.15.1:0/1027265969 learned_addr learned my addr 172.21.15.1:0/1027265969 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.980+0000 7f07ef59e700 1 -- 172.21.15.1:0/1027265969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07f00d05a0 msgr2=0x7f07f009a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.980+0000 7f07ef59e700 1 --2- 172.21.15.1:0/1027265969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07f00d05a0 0x7f07f009a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.980+0000 7f07ef59e700 1 -- 172.21.15.1:0/1027265969 >> v1:172.21.15.165:6789/0 conn(0x7f07f0099a40 legacy=0x7f07f0099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.980+0000 7f07ef59e700 1 -- 172.21.15.1:0/1027265969 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f00d0080 con 0x7f07f006ba70 2024-03-18T19:06:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.980+0000 7f07ee59c700 1 --2- 172.21.15.1:0/1027265969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07f00d05a0 0x7f07f009a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:06:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.980+0000 7f07ef59e700 1 --2- 172.21.15.1:0/1027265969 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07f006ba70 0x7f07f015d940 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f07e8009e50 tx=0x7f07e800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0e5edf0ab3c91d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.980+0000 7f07edd9b700 1 -- 172.21.15.1:0/1027265969 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e800e070 con 0x7f07f006ba70 2024-03-18T19:06:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.980+0000 7f07edd9b700 1 -- 172.21.15.1:0/1027265969 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e800b910 con 0x7f07f006ba70 2024-03-18T19:06:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.980+0000 7f07edd9b700 1 -- 172.21.15.1:0/1027265969 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e800bcc0 con 0x7f07f006ba70 2024-03-18T19:06:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.981+0000 7f07f5050700 1 -- 172.21.15.1:0/1027265969 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07f006ba70 msgr2=0x7f07f015d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.981+0000 7f07f5050700 1 --2- 172.21.15.1:0/1027265969 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07f006ba70 0x7f07f015d940 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f07e8009e50 tx=0x7f07e800b330 comp rx=0 tx=0).stop 2024-03-18T19:06:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.981+0000 7f07f5050700 1 -- 172.21.15.1:0/1027265969 shutdown_connections 2024-03-18T19:06:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.981+0000 7f07f5050700 1 --2- 172.21.15.1:0/1027265969 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07f006ba70 0x7f07f015d940 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.981+0000 7f07f5050700 1 --2- 172.21.15.1:0/1027265969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07f00d05a0 0x7f07f009a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.981+0000 7f07f5050700 1 -- 172.21.15.1:0/1027265969 >> 172.21.15.1:0/1027265969 conn(0x7f07f00b2cb0 msgr2=0x7f07f00b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.981+0000 7f07f5050700 1 -- 172.21.15.1:0/1027265969 shutdown_connections 2024-03-18T19:06:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.981+0000 7f07f5050700 1 -- 172.21.15.1:0/1027265969 wait complete. 2024-03-18T19:06:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.983+0000 7f07f5050700 1 Processor -- start 2024-03-18T19:06:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.983+0000 7f07f5050700 1 -- start start 2024-03-18T19:06:37.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.983+0000 7f07f5050700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07f006ba70 0x7f07f00c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.983+0000 7f07f5050700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07f0099a40 0x7f07f01544c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.983+0000 7f07f5050700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07f00d05a0 0x7f07f01536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.983+0000 7f07f5050700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f00c8290 con 0x7f07f00d05a0 2024-03-18T19:06:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.983+0000 7f07f5050700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f00c8b30 con 0x7f07f006ba70 2024-03-18T19:06:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.983+0000 7f07f5050700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f00b4c60 con 0x7f07f0099a40 2024-03-18T19:06:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.984+0000 7f07ef59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07f00d05a0 0x7f07f01536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.984+0000 7f07ef59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07f00d05a0 0x7f07f01536a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35902/0 (socket says 172.21.15.1:35902) 2024-03-18T19:06:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.984+0000 7f07ef59e700 1 -- 172.21.15.1:0/1520872423 learned_addr learned my addr 172.21.15.1:0/1520872423 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.984+0000 7f07ee59c700 1 --2- 172.21.15.1:0/1520872423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07f0099a40 0x7f07f01544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.984+0000 7f07eed9d700 1 --2- 172.21.15.1:0/1520872423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07f006ba70 0x7f07f00c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:37.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.984+0000 7f07ef59e700 1 -- 172.21.15.1:0/1520872423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07f0099a40 msgr2=0x7f07f01544c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.984+0000 7f07ef59e700 1 --2- 172.21.15.1:0/1520872423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07f0099a40 0x7f07f01544c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.984+0000 7f07ef59e700 1 -- 172.21.15.1:0/1520872423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07f006ba70 msgr2=0x7f07f00c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.984+0000 7f07ef59e700 1 --2- 172.21.15.1:0/1520872423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07f006ba70 0x7f07f00c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.984+0000 7f07ef59e700 1 -- 172.21.15.1:0/1520872423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e00094a0 con 0x7f07f00d05a0 2024-03-18T19:06:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.984+0000 7f07eed9d700 1 --2- 172.21.15.1:0/1520872423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07f006ba70 0x7f07f00c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.984+0000 7f07ef59e700 1 --2- 172.21.15.1:0/1520872423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07f00d05a0 0x7f07f01536a0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f07e800b680 tx=0x7f07e8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.984+0000 7f07d77fe700 1 -- 172.21.15.1:0/1520872423 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e80048e0 con 0x7f07f00d05a0 2024-03-18T19:06:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.984+0000 7f07d77fe700 1 -- 172.21.15.1:0/1520872423 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e80099b0 con 0x7f07f00d05a0 2024-03-18T19:06:37.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.984+0000 7f07d77fe700 1 -- 172.21.15.1:0/1520872423 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e801c680 con 0x7f07f00d05a0 2024-03-18T19:06:37.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.984+0000 7f07f5050700 1 -- 172.21.15.1:0/1520872423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e8009b30 con 0x7f07f00d05a0 2024-03-18T19:06:37.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.984+0000 7f07f5050700 1 -- 172.21.15.1:0/1520872423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f00c87c0 con 0x7f07f00d05a0 2024-03-18T19:06:37.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.985+0000 7f07f5050700 1 -- 172.21.15.1:0/1520872423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f0068240 con 0x7f07f00d05a0 2024-03-18T19:06:37.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.986+0000 7f07d77fe700 1 -- 172.21.15.1:0/1520872423 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f07e801c7e0 con 0x7f07f00d05a0 2024-03-18T19:06:37.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.986+0000 7f07d77fe700 1 --2- 172.21.15.1:0/1520872423 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f07dc06adb0 0x7f07dc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:37.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.986+0000 7f07d77fe700 1 -- 172.21.15.1:0/1520872423 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f07e808b000 con 0x7f07f00d05a0 2024-03-18T19:06:37.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.986+0000 7f07eed9d700 1 --2- 172.21.15.1:0/1520872423 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f07dc06adb0 0x7f07dc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:37.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.987+0000 7f07eed9d700 1 --2- 172.21.15.1:0/1520872423 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f07dc06adb0 0x7f07dc06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f07e4000f60 tx=0x7f07e4009040 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:37.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:37.991+0000 7f07d77fe700 1 -- 172.21.15.1:0/1520872423 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f07e80561e0 con 0x7f07f00d05a0 2024-03-18T19:06:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.278+0000 7f07f5050700 1 -- 172.21.15.1:0/1520872423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f07f00cda40 con 0x7f07f00d05a0 2024-03-18T19:06:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.278+0000 7f07d77fe700 1 -- 172.21.15.1:0/1520872423 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f07e8059800 con 0x7f07f00d05a0 2024-03-18T19:06:38.275 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:38.275 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:06:38.275 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-18T19:06:38.275 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:38.275 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-18T19:06:38.275 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlR93/CxAAbpe+5n3wHRur9xi8QzdCnQ==", 2024-03-18T19:06:38.275 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:38.275 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:06:38.275 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:06:38.275 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:06:38.275 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:38.276 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.276 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:38.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:38.276 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-18T19:06:38.276 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJ3OmGBAA37u227nxrMsh/JHt71HBxQ==", 2024-03-18T19:06:38.276 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:38.276 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:06:38.276 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:06:38.276 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:06:38.276 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:38.276 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.276 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:38.277 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:38.277 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-18T19:06:38.277 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJZFoExAAmI16oxwNKbf8dKetPfyk+g==", 2024-03-18T19:06:38.277 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:38.277 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:06:38.277 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:06:38.277 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:06:38.277 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:38.277 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:38.277 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:38.277 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-18T19:06:38.277 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlQrcvIBAA1LMv+EgC7S6wspX6QF49Pg==", 2024-03-18T19:06:38.277 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:38.277 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:06:38.278 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:06:38.278 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:06:38.278 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:38.278 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.278 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:38.278 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:38.278 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-18T19:06:38.278 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCUjPhlYOVHChAARFQrksZmMIdTlntd8TgvrQ==", 2024-03-18T19:06:38.278 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:38.278 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:06:38.278 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:06:38.278 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:38.278 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:38.279 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:38.279 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-18T19:06:38.279 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCXjPhl2zSDCBAA6Al1gkWVB79ZI9djGpVChA==", 2024-03-18T19:06:38.279 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:38.279 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:06:38.279 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:06:38.279 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:38.279 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:38.279 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:38.279 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-18T19:06:38.279 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCajPhlSoG/BhAAR93USDe6DCQW+hREOsnTlw==", 2024-03-18T19:06:38.279 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:06:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:06:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-18T19:06:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCdjPhlhJg5BRAAWuQhcnxyjR5UOiM13yKdAQ==", 2024-03-18T19:06:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:06:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:06:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:38.280 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:38.281 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-18T19:06:38.281 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCjjPhlCFnpCRAAKIyOQyVdjXZUTnAS6Nw/ng==", 2024-03-18T19:06:38.281 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:38.281 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:06:38.281 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:06:38.281 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:38.281 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:38.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:38.281 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-18T19:06:38.281 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCmjPhlAeFkCBAAJcEFpRk/03lQJxj+l2hEPA==", 2024-03-18T19:06:38.281 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:38.281 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:06:38.281 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:06:38.281 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:38.282 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.282 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:38.282 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:38.282 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-18T19:06:38.282 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCpjPhl/xuuBhAA0LgKfXEvN2O11ZlRQJVOkw==", 2024-03-18T19:06:38.282 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:38.282 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:06:38.282 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:06:38.282 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:38.282 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.282 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:38.282 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:38.282 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-18T19:06:38.282 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCsjPhldiR4BRAA571oh6U8PzOv/4LzjYLF/A==", 2024-03-18T19:06:38.283 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:38.283 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:06:38.283 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:06:38.283 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:38.283 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:38.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:38.283 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-18T19:06:38.283 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlu+2xJBAANnXGGjfeqP6S2FTecsfOsg==", 2024-03-18T19:06:38.283 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:38.283 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:06:38.283 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-18T19:06:38.283 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-18T19:06:38.283 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:38.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:38.284 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-18T19:06:38.284 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlhJ9+KxAAEnrQrCrgJR6xI+pucwzAXA==", 2024-03-18T19:06:38.284 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:38.284 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:06:38.284 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:06:38.284 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:06:38.284 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:06:38.285 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.285 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:38.285 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:38.285 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.2", 2024-03-18T19:06:38.285 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlxBUpKBAAIGTwmLWyj+P2Yvl55QVcWA==", 2024-03-18T19:06:38.285 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:38.285 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:06:38.285 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:06:38.285 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:06:38.285 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:06:38.285 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.285 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:38.285 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:38.285 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.3", 2024-03-18T19:06:38.285 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlmao3MRAAHeiQ6eyBh9QjjKPCtxkC8g==", 2024-03-18T19:06:38.286 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:38.286 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:06:38.286 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:06:38.286 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:06:38.286 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:06:38.286 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:38.286 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:38.286 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-18T19:06:38.286 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCRjPhlemOuKRAAVUggmtB7+EiVgyfJZ8TCCA==", 2024-03-18T19:06:38.286 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:38.286 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:06:38.286 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:06:38.287 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:06:38.287 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:38.287 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.287 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:38.287 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:38.287 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-18T19:06:38.287 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhleIuOCBAA9CH+AHewQbn+gpVr+6lJyw==", 2024-03-18T19:06:38.287 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:38.287 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:06:38.287 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T19:06:38.287 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:38.287 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.287 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:38.287 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:38.288 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-18T19:06:38.288 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlL0pTBBAAdFiPcdwwz514cuLAvioxYA==", 2024-03-18T19:06:38.288 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:06:38.288 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:06:38.288 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T19:06:38.288 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:06:38.288 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.288 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:38.288 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-18T19:06:38.288 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:06:38.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.280+0000 7f07f5050700 1 -- 172.21.15.1:0/1520872423 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f07dc06adb0 msgr2=0x7f07dc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:38.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.280+0000 7f07f5050700 1 --2- 172.21.15.1:0/1520872423 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f07dc06adb0 0x7f07dc06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f07e4000f60 tx=0x7f07e4009040 comp rx=0 tx=0).stop 2024-03-18T19:06:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.280+0000 7f07f5050700 1 -- 172.21.15.1:0/1520872423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07f00d05a0 msgr2=0x7f07f01536a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.280+0000 7f07f5050700 1 --2- 172.21.15.1:0/1520872423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07f00d05a0 0x7f07f01536a0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f07e800b680 tx=0x7f07e8004ad0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.280+0000 7f07f5050700 1 -- 172.21.15.1:0/1520872423 shutdown_connections 2024-03-18T19:06:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.280+0000 7f07f5050700 1 --2- 172.21.15.1:0/1520872423 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f07dc06adb0 0x7f07dc06d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.280+0000 7f07f5050700 1 --2- 172.21.15.1:0/1520872423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07f00d05a0 0x7f07f01536a0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.280+0000 7f07f5050700 1 --2- 172.21.15.1:0/1520872423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07f006ba70 0x7f07f00c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.280+0000 7f07f5050700 1 --2- 172.21.15.1:0/1520872423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07f0099a40 0x7f07f01544c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.280+0000 7f07f5050700 1 -- 172.21.15.1:0/1520872423 >> 172.21.15.1:0/1520872423 conn(0x7f07f00b2cb0 msgr2=0x7f07f00b3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.280+0000 7f07f5050700 1 -- 172.21.15.1:0/1520872423 shutdown_connections 2024-03-18T19:06:38.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.280+0000 7f07f5050700 1 -- 172.21.15.1:0/1520872423 wait complete. 2024-03-18T19:06:38.291 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:06:38.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.421+0000 7fe75da9c700 1 Processor -- start 2024-03-18T19:06:38.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.421+0000 7fe75da9c700 1 -- start start 2024-03-18T19:06:38.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.421+0000 7fe75da9c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe75815a320 0x7fe7580ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:38.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.421+0000 7fe75da9c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7580acd80 con 0x7fe7580cea30 2024-03-18T19:06:38.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.422+0000 7fe75da9c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7580cf220 con 0x7fe758071d10 2024-03-18T19:06:38.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.422+0000 7fe75da9c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7580cef40 con 0x7fe75815a320 2024-03-18T19:06:38.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.422+0000 7fe7577fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe75815a320 0x7fe7580ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:38.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.422+0000 7fe756ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe7580cea30 0x7fe758071600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55236/0 (socket says 172.21.15.1:55236) 2024-03-18T19:06:38.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.422+0000 7fe756ffd700 1 -- 172.21.15.1:0/3520296108 learned_addr learned my addr 172.21.15.1:0/3520296108 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:38.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.422+0000 7fe7577fe700 1 -- 172.21.15.1:0/3520296108 >> v1:172.21.15.165:6789/0 conn(0x7fe758071d10 legacy=0x7fe75806e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:38.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.422+0000 7fe7577fe700 1 -- 172.21.15.1:0/3520296108 >> v1:172.21.15.1:6789/0 conn(0x7fe7580cea30 legacy=0x7fe758071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:38.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.422+0000 7fe7577fe700 1 -- 172.21.15.1:0/3520296108 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75806d7e0 con 0x7fe75815a320 2024-03-18T19:06:38.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.422+0000 7fe7577fe700 1 --2- 172.21.15.1:0/3520296108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe75815a320 0x7fe7580ce4f0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fe748009840 tx=0x7fe748009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810af01c660a0c00 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:38.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.422+0000 7fe7567fc700 1 -- 172.21.15.1:0/3520296108 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe748004230 con 0x7fe75815a320 2024-03-18T19:06:38.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.422+0000 7fe7567fc700 1 -- 172.21.15.1:0/3520296108 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe748004390 con 0x7fe75815a320 2024-03-18T19:06:38.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.422+0000 7fe7567fc700 1 -- 172.21.15.1:0/3520296108 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7480036a0 con 0x7fe75815a320 2024-03-18T19:06:38.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.423+0000 7fe75da9c700 1 -- 172.21.15.1:0/3520296108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe75815a320 msgr2=0x7fe7580ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:38.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.423+0000 7fe75da9c700 1 --2- 172.21.15.1:0/3520296108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe75815a320 0x7fe7580ce4f0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fe748009840 tx=0x7fe748009b50 comp rx=0 tx=0).stop 2024-03-18T19:06:38.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.423+0000 7fe75da9c700 1 -- 172.21.15.1:0/3520296108 shutdown_connections 2024-03-18T19:06:38.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.423+0000 7fe75da9c700 1 --2- 172.21.15.1:0/3520296108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe75815a320 0x7fe7580ce4f0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.423+0000 7fe75da9c700 1 -- 172.21.15.1:0/3520296108 >> 172.21.15.1:0/3520296108 conn(0x7fe7580cc990 msgr2=0x7fe7580ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:38.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.423+0000 7fe75da9c700 1 -- 172.21.15.1:0/3520296108 shutdown_connections 2024-03-18T19:06:38.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.423+0000 7fe75da9c700 1 -- 172.21.15.1:0/3520296108 wait complete. 2024-03-18T19:06:38.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.425+0000 7fe75da9c700 1 Processor -- start 2024-03-18T19:06:38.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.425+0000 7fe75da9c700 1 -- start start 2024-03-18T19:06:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe75da9c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe758071d10 0x7fe75815b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe75da9c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7580cea30 0x7fe7580c3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe75da9c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe75815a780 0x7fe758154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe75da9c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75815c0a0 con 0x7fe758071d10 2024-03-18T19:06:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe75da9c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7580b4140 con 0x7fe7580cea30 2024-03-18T19:06:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe75da9c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe758154570 con 0x7fe75815a780 2024-03-18T19:06:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe7577fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe758071d10 0x7fe75815b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe7577fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe758071d10 0x7fe75815b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35916/0 (socket says 172.21.15.1:35916) 2024-03-18T19:06:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe7577fe700 1 -- 172.21.15.1:0/9231884 learned_addr learned my addr 172.21.15.1:0/9231884 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe757fff700 1 --2- 172.21.15.1:0/9231884 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe75815a780 0x7fe758154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:38.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe756ffd700 1 --2- 172.21.15.1:0/9231884 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7580cea30 0x7fe7580c3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe7577fe700 1 -- 172.21.15.1:0/9231884 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe75815a780 msgr2=0x7fe758154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe7577fe700 1 --2- 172.21.15.1:0/9231884 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe75815a780 0x7fe758154030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe7577fe700 1 -- 172.21.15.1:0/9231884 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7580cea30 msgr2=0x7fe7580c3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe7577fe700 1 --2- 172.21.15.1:0/9231884 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7580cea30 0x7fe7580c3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe7577fe700 1 -- 172.21.15.1:0/9231884 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7480094a0 con 0x7fe758071d10 2024-03-18T19:06:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe756ffd700 1 --2- 172.21.15.1:0/9231884 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7580cea30 0x7fe7580c3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe7577fe700 1 --2- 172.21.15.1:0/9231884 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe758071d10 0x7fe75815b470 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fe748009b90 tx=0x7fe748004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe73ffff700 1 -- 172.21.15.1:0/9231884 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe748011630 con 0x7fe758071d10 2024-03-18T19:06:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe73ffff700 1 -- 172.21.15.1:0/9231884 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74801e070 con 0x7fe758071d10 2024-03-18T19:06:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe75da9c700 1 -- 172.21.15.1:0/9231884 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe75815bae0 con 0x7fe758071d10 2024-03-18T19:06:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.426+0000 7fe73ffff700 1 -- 172.21.15.1:0/9231884 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7480119b0 con 0x7fe758071d10 2024-03-18T19:06:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.427+0000 7fe75da9c700 1 -- 172.21.15.1:0/9231884 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7580b4350 con 0x7fe758071d10 2024-03-18T19:06:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.427+0000 7fe75da9c700 1 -- 172.21.15.1:0/9231884 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7580676a0 con 0x7fe758071d10 2024-03-18T19:06:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.428+0000 7fe73ffff700 1 -- 172.21.15.1:0/9231884 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe748011b10 con 0x7fe758071d10 2024-03-18T19:06:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.428+0000 7fe73ffff700 1 --2- 172.21.15.1:0/9231884 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe74406adb0 0x7fe74406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.428+0000 7fe73ffff700 1 -- 172.21.15.1:0/9231884 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fe74808a4b0 con 0x7fe758071d10 2024-03-18T19:06:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.428+0000 7fe756ffd700 1 --2- 172.21.15.1:0/9231884 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe74406adb0 0x7fe74406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.429+0000 7fe756ffd700 1 --2- 172.21.15.1:0/9231884 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe74406adb0 0x7fe74406d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe7580b40f0 tx=0x7fe74c009040 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:38.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.433+0000 7fe73ffff700 1 -- 172.21.15.1:0/9231884 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe748055820 con 0x7fe758071d10 2024-03-18T19:06:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.639+0000 7fe75da9c700 1 -- 172.21.15.1:0/9231884 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7580cf220 con 0x7fe758071d10 2024-03-18T19:06:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.640+0000 7fe73ffff700 1 -- 172.21.15.1:0/9231884 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+12556 (secure 0 0 0) 0x7fe7580cf220 con 0x7fe758071d10 2024-03-18T19:06:38.636 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:38.636 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":113,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:06:37.819970+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:06:38.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.642+0000 7fe75da9c700 1 -- 172.21.15.1:0/9231884 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe74406adb0 msgr2=0x7fe74406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.642+0000 7fe75da9c700 1 --2- 172.21.15.1:0/9231884 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe74406adb0 0x7fe74406d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe7580b40f0 tx=0x7fe74c009040 comp rx=0 tx=0).stop 2024-03-18T19:06:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.642+0000 7fe75da9c700 1 -- 172.21.15.1:0/9231884 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe758071d10 msgr2=0x7fe75815b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.642+0000 7fe75da9c700 1 --2- 172.21.15.1:0/9231884 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe758071d10 0x7fe75815b470 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fe748009b90 tx=0x7fe748004750 comp rx=0 tx=0).stop 2024-03-18T19:06:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.642+0000 7fe75da9c700 1 -- 172.21.15.1:0/9231884 shutdown_connections 2024-03-18T19:06:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.642+0000 7fe75da9c700 1 --2- 172.21.15.1:0/9231884 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe74406adb0 0x7fe74406d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.642+0000 7fe75da9c700 1 --2- 172.21.15.1:0/9231884 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe75815a780 0x7fe758154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.642+0000 7fe75da9c700 1 --2- 172.21.15.1:0/9231884 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe758071d10 0x7fe75815b470 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.642+0000 7fe75da9c700 1 --2- 172.21.15.1:0/9231884 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7580cea30 0x7fe7580c3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.642+0000 7fe75da9c700 1 -- 172.21.15.1:0/9231884 >> 172.21.15.1:0/9231884 conn(0x7fe7580cc990 msgr2=0x7fe7580b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:38.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.642+0000 7fe75da9c700 1 -- 172.21.15.1:0/9231884 shutdown_connections 2024-03-18T19:06:38.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.642+0000 7fe75da9c700 1 -- 172.21.15.1:0/9231884 wait complete. 2024-03-18T19:06:38.651 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:06:38.651 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:06:38.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.783+0000 7f0ca486a700 1 Processor -- start 2024-03-18T19:06:38.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.783+0000 7f0ca486a700 1 -- start start 2024-03-18T19:06:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.784+0000 7f0ca486a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0ca00cfa90 0x7f0ca006a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.784+0000 7f0ca486a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0ca006ac90 0x7f0ca0167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.784+0000 7f0ca486a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca00cf600 con 0x7f0ca015d5a0 2024-03-18T19:06:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.784+0000 7f0ca486a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca006b140 con 0x7f0ca00cfa90 2024-03-18T19:06:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.784+0000 7f0ca486a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca00ce720 con 0x7f0ca006ac90 2024-03-18T19:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.784+0000 7f0c9dd9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0ca00cfa90 0x7f0ca006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.784+0000 7f0c9dd9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0ca00cfa90 0x7f0ca006a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35918/0 (socket says 172.21.15.1:35918) 2024-03-18T19:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.784+0000 7f0c9dd9b700 1 -- 172.21.15.1:0/511851625 learned_addr learned my addr 172.21.15.1:0/511851625 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.784+0000 7f0c9ed9d700 1 --2- 172.21.15.1:0/511851625 >> v2:172.21.15.165:3300/0 conn(0x7f0ca006ac90 0x7f0ca0167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.784+0000 7f0c9dd9b700 1 -- 172.21.15.1:0/511851625 >> v2:172.21.15.165:3300/0 conn(0x7f0ca006ac90 msgr2=0x7f0ca0167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.784+0000 7f0c9dd9b700 1 --2- 172.21.15.1:0/511851625 >> v2:172.21.15.165:3300/0 conn(0x7f0ca006ac90 0x7f0ca0167140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.784+0000 7f0c9dd9b700 1 -- 172.21.15.1:0/511851625 >> v1:172.21.15.165:6789/0 conn(0x7f0ca015d5a0 legacy=0x7f0ca00ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.784+0000 7f0c9dd9b700 1 -- 172.21.15.1:0/511851625 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca00ce9f0 con 0x7f0ca00cfa90 2024-03-18T19:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.784+0000 7f0c9ed9d700 1 --2- 172.21.15.1:0/511851625 >> v2:172.21.15.165:3300/0 conn(0x7f0ca006ac90 0x7f0ca0167140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.785+0000 7f0c9dd9b700 1 --2- 172.21.15.1:0/511851625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca00cfa90 0x7f0ca006a750 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00cdec0 tx=0x7f0c90009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e81fd189618be3d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.785+0000 7f0c9d59a700 1 -- 172.21.15.1:0/511851625 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c90004230 con 0x7f0ca00cfa90 2024-03-18T19:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.785+0000 7f0c9d59a700 1 -- 172.21.15.1:0/511851625 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c90004390 con 0x7f0ca00cfa90 2024-03-18T19:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.785+0000 7f0c9d59a700 1 -- 172.21.15.1:0/511851625 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c900036a0 con 0x7f0ca00cfa90 2024-03-18T19:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.785+0000 7f0ca486a700 1 -- 172.21.15.1:0/511851625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca00cfa90 msgr2=0x7f0ca006a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.785+0000 7f0ca486a700 1 --2- 172.21.15.1:0/511851625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca00cfa90 0x7f0ca006a750 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00cdec0 tx=0x7f0c90009bd0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.785+0000 7f0ca486a700 1 -- 172.21.15.1:0/511851625 shutdown_connections 2024-03-18T19:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.785+0000 7f0ca486a700 1 --2- 172.21.15.1:0/511851625 >> v2:172.21.15.165:3300/0 conn(0x7f0ca006ac90 0x7f0ca0167140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.785+0000 7f0ca486a700 1 --2- 172.21.15.1:0/511851625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca00cfa90 0x7f0ca006a750 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.785+0000 7f0ca486a700 1 -- 172.21.15.1:0/511851625 >> 172.21.15.1:0/511851625 conn(0x7f0ca00cc9b0 msgr2=0x7f0ca00ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.785+0000 7f0ca486a700 1 -- 172.21.15.1:0/511851625 shutdown_connections 2024-03-18T19:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.785+0000 7f0ca486a700 1 -- 172.21.15.1:0/511851625 wait complete. 2024-03-18T19:06:38.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.787+0000 7f0ca486a700 1 Processor -- start 2024-03-18T19:06:38.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.787+0000 7f0ca486a700 1 -- start start 2024-03-18T19:06:38.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0ca486a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca006ac90 0x7f0ca00c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:38.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0ca486a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ca00cfa90 0x7f0ca01558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:38.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0ca486a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ca015d5a0 0x7f0ca00c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:38.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0ca486a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca00c82b0 con 0x7f0ca006ac90 2024-03-18T19:06:38.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0ca486a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca00c8550 con 0x7f0ca00cfa90 2024-03-18T19:06:38.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0ca486a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca00c8830 con 0x7f0ca015d5a0 2024-03-18T19:06:38.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0c9e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca006ac90 0x7f0ca00c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:38.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0c9e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca006ac90 0x7f0ca00c7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35934/0 (socket says 172.21.15.1:35934) 2024-03-18T19:06:38.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0c9e59c700 1 -- 172.21.15.1:0/1481970931 learned_addr learned my addr 172.21.15.1:0/1481970931 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:38.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0c9dd9b700 1 --2- 172.21.15.1:0/1481970931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ca00cfa90 0x7f0ca01558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:38.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0c9ed9d700 1 --2- 172.21.15.1:0/1481970931 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ca015d5a0 0x7f0ca00c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:38.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0c9e59c700 1 -- 172.21.15.1:0/1481970931 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ca015d5a0 msgr2=0x7f0ca00c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:38.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0c9e59c700 1 --2- 172.21.15.1:0/1481970931 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ca015d5a0 0x7f0ca00c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0c9e59c700 1 -- 172.21.15.1:0/1481970931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ca00cfa90 msgr2=0x7f0ca01558e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:38.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0c9e59c700 1 --2- 172.21.15.1:0/1481970931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ca00cfa90 0x7f0ca01558e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0c9e59c700 1 -- 172.21.15.1:0/1481970931 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c90009550 con 0x7f0ca006ac90 2024-03-18T19:06:38.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0c9e59c700 1 --2- 172.21.15.1:0/1481970931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca006ac90 0x7f0ca00c7000 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400b450 tx=0x7f0c9400b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:38.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0c8effd700 1 -- 172.21.15.1:0/1481970931 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c940059b0 con 0x7f0ca006ac90 2024-03-18T19:06:38.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0c8effd700 1 -- 172.21.15.1:0/1481970931 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c94005b30 con 0x7f0ca006ac90 2024-03-18T19:06:38.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0ca486a700 1 -- 172.21.15.1:0/1481970931 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca00c8b10 con 0x7f0ca006ac90 2024-03-18T19:06:38.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0c8effd700 1 -- 172.21.15.1:0/1481970931 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c94011630 con 0x7f0ca006ac90 2024-03-18T19:06:38.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.788+0000 7f0ca486a700 1 -- 172.21.15.1:0/1481970931 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca00c7540 con 0x7f0ca006ac90 2024-03-18T19:06:38.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.789+0000 7f0ca486a700 1 -- 172.21.15.1:0/1481970931 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca015e2b0 con 0x7f0ca006ac90 2024-03-18T19:06:38.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.790+0000 7f0c8effd700 1 -- 172.21.15.1:0/1481970931 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0c94011790 con 0x7f0ca006ac90 2024-03-18T19:06:38.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.790+0000 7f0c8effd700 1 --2- 172.21.15.1:0/1481970931 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c8006adb0 0x7f0c8006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:38.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.790+0000 7f0c8effd700 1 -- 172.21.15.1:0/1481970931 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f0c94088a60 con 0x7f0ca006ac90 2024-03-18T19:06:38.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.790+0000 7f0c9dd9b700 1 --2- 172.21.15.1:0/1481970931 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c8006adb0 0x7f0c8006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:38.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.791+0000 7f0c9dd9b700 1 --2- 172.21.15.1:0/1481970931 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c8006adb0 0x7f0c8006d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0c90009130 tx=0x7f0c90008030 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:38.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.795+0000 7f0c8effd700 1 -- 172.21.15.1:0/1481970931 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0c94053dd0 con 0x7f0ca006ac90 2024-03-18T19:06:38.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.996+0000 7f0ca486a700 1 -- 172.21.15.1:0/1481970931 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0ca00676a0 con 0x7f0ca006ac90 2024-03-18T19:06:38.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.996+0000 7f0c8effd700 1 -- 172.21.15.1:0/1481970931 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0c94015070 con 0x7f0ca006ac90 2024-03-18T19:06:38.992 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.998+0000 7f0ca486a700 1 -- 172.21.15.1:0/1481970931 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c8006adb0 msgr2=0x7f0c8006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.998+0000 7f0ca486a700 1 --2- 172.21.15.1:0/1481970931 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c8006adb0 0x7f0c8006d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0c90009130 tx=0x7f0c90008030 comp rx=0 tx=0).stop 2024-03-18T19:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.998+0000 7f0ca486a700 1 -- 172.21.15.1:0/1481970931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca006ac90 msgr2=0x7f0ca00c7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.998+0000 7f0ca486a700 1 --2- 172.21.15.1:0/1481970931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca006ac90 0x7f0ca00c7000 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400b450 tx=0x7f0c9400b810 comp rx=0 tx=0).stop 2024-03-18T19:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.998+0000 7f0ca486a700 1 -- 172.21.15.1:0/1481970931 shutdown_connections 2024-03-18T19:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.998+0000 7f0ca486a700 1 --2- 172.21.15.1:0/1481970931 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c8006adb0 0x7f0c8006d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.998+0000 7f0ca486a700 1 --2- 172.21.15.1:0/1481970931 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ca015d5a0 0x7f0ca00c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.998+0000 7f0ca486a700 1 --2- 172.21.15.1:0/1481970931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca006ac90 0x7f0ca00c7000 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.998+0000 7f0ca486a700 1 --2- 172.21.15.1:0/1481970931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ca00cfa90 0x7f0ca01558e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:38.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.998+0000 7f0ca486a700 1 -- 172.21.15.1:0/1481970931 >> 172.21.15.1:0/1481970931 conn(0x7f0ca00cc9b0 msgr2=0x7f0ca00aebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:38.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.998+0000 7f0ca486a700 1 -- 172.21.15.1:0/1481970931 shutdown_connections 2024-03-18T19:06:38.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:38.998+0000 7f0ca486a700 1 -- 172.21.15.1:0/1481970931 wait complete. 2024-03-18T19:06:39.006 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T19:06:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.135+0000 7f61d72d1700 1 Processor -- start 2024-03-18T19:06:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.135+0000 7f61d72d1700 1 -- start start 2024-03-18T19:06:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.136+0000 7f61d72d1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d0099a40 0x7f61d006ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.136+0000 7f61d72d1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f61d00d05a0 0x7f61d0099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.136+0000 7f61d72d1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f61d009a360 0x7f61d006e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.136+0000 7f61d72d1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61d00cf7e0 con 0x7f61d009a360 2024-03-18T19:06:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.136+0000 7f61d72d1700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61d00cfac0 con 0x7f61d00d05a0 2024-03-18T19:06:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.136+0000 7f61d72d1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d00cfda0 con 0x7f61d0099a40 2024-03-18T19:06:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.136+0000 7f61d586e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f61d009a360 0x7f61d006e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.136+0000 7f61d586e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f61d009a360 0x7f61d006e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35940/0 (socket says 172.21.15.1:35940) 2024-03-18T19:06:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.136+0000 7f61d586e700 1 -- 172.21.15.1:0/1562842777 learned_addr learned my addr 172.21.15.1:0/1562842777 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.137+0000 7f61d486c700 1 --2- 172.21.15.1:0/1562842777 >> v2:172.21.15.165:3300/0 conn(0x7f61d00d05a0 0x7f61d0099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.137+0000 7f61d506d700 1 --2- 172.21.15.1:0/1562842777 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d0099a40 0x7f61d006ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.137+0000 7f61d586e700 1 -- 172.21.15.1:0/1562842777 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d0099a40 msgr2=0x7f61d006ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.137+0000 7f61d586e700 1 --2- 172.21.15.1:0/1562842777 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d0099a40 0x7f61d006ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.137+0000 7f61d586e700 1 -- 172.21.15.1:0/1562842777 >> v2:172.21.15.165:3300/0 conn(0x7f61d00d05a0 msgr2=0x7f61d0099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.137+0000 7f61d586e700 1 --2- 172.21.15.1:0/1562842777 >> v2:172.21.15.165:3300/0 conn(0x7f61d00d05a0 0x7f61d0099e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.137+0000 7f61d586e700 1 -- 172.21.15.1:0/1562842777 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d00d0080 con 0x7f61d009a360 2024-03-18T19:06:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.137+0000 7f61d486c700 1 --2- 172.21.15.1:0/1562842777 >> v2:172.21.15.165:3300/0 conn(0x7f61d00d05a0 0x7f61d0099e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.137+0000 7f61d586e700 1 --2- 172.21.15.1:0/1562842777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d009a360 0x7f61d006e880 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f61d00d22c0 tx=0x7f61c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bf29453f607a3aa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.137+0000 7f61c7fff700 1 -- 172.21.15.1:0/1562842777 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c800e070 con 0x7f61d009a360 2024-03-18T19:06:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.137+0000 7f61c7fff700 1 -- 172.21.15.1:0/1562842777 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c800b9c0 con 0x7f61d009a360 2024-03-18T19:06:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.137+0000 7f61c7fff700 1 -- 172.21.15.1:0/1562842777 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c800bd70 con 0x7f61d009a360 2024-03-18T19:06:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.137+0000 7f61d72d1700 1 -- 172.21.15.1:0/1562842777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d009a360 msgr2=0x7f61d006e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.137+0000 7f61d72d1700 1 --2- 172.21.15.1:0/1562842777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d009a360 0x7f61d006e880 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f61d00d22c0 tx=0x7f61c800b330 comp rx=0 tx=0).stop 2024-03-18T19:06:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.138+0000 7f61d72d1700 1 -- 172.21.15.1:0/1562842777 shutdown_connections 2024-03-18T19:06:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.138+0000 7f61d72d1700 1 --2- 172.21.15.1:0/1562842777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d009a360 0x7f61d006e880 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.138+0000 7f61d72d1700 1 --2- 172.21.15.1:0/1562842777 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d0099a40 0x7f61d006ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.138+0000 7f61d72d1700 1 --2- 172.21.15.1:0/1562842777 >> v2:172.21.15.165:3300/0 conn(0x7f61d00d05a0 0x7f61d0099e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.138+0000 7f61d72d1700 1 -- 172.21.15.1:0/1562842777 >> 172.21.15.1:0/1562842777 conn(0x7f61d00b2cb0 msgr2=0x7f61d00b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.138+0000 7f61d72d1700 1 -- 172.21.15.1:0/1562842777 shutdown_connections 2024-03-18T19:06:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.138+0000 7f61d72d1700 1 -- 172.21.15.1:0/1562842777 wait complete. 2024-03-18T19:06:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.139+0000 7f61d72d1700 1 Processor -- start 2024-03-18T19:06:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.139+0000 7f61d72d1700 1 -- start start 2024-03-18T19:06:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.140+0000 7f61d72d1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d0099a40 0x7f61d00c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.140+0000 7f61d72d1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d009a360 0x7f61d00ca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.140+0000 7f61d72d1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d00d05a0 0x7f61d014b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.140+0000 7f61d72d1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d00c9280 con 0x7f61d0099a40 2024-03-18T19:06:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.140+0000 7f61d72d1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d00c9bc0 con 0x7f61d00d05a0 2024-03-18T19:06:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.140+0000 7f61d72d1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d00c8600 con 0x7f61d009a360 2024-03-18T19:06:39.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.140+0000 7f61d506d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d0099a40 0x7f61d00c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.140+0000 7f61d506d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d0099a40 0x7f61d00c9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35956/0 (socket says 172.21.15.1:35956) 2024-03-18T19:06:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.140+0000 7f61d506d700 1 -- 172.21.15.1:0/1168348830 learned_addr learned my addr 172.21.15.1:0/1168348830 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.141+0000 7f61d486c700 1 --2- 172.21.15.1:0/1168348830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d009a360 0x7f61d00ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.141+0000 7f61d586e700 1 --2- 172.21.15.1:0/1168348830 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d00d05a0 0x7f61d014b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.141+0000 7f61d506d700 1 -- 172.21.15.1:0/1168348830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d009a360 msgr2=0x7f61d00ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.141+0000 7f61d506d700 1 --2- 172.21.15.1:0/1168348830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d009a360 0x7f61d00ca990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.141+0000 7f61d506d700 1 -- 172.21.15.1:0/1168348830 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d00d05a0 msgr2=0x7f61d014b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.141+0000 7f61d506d700 1 --2- 172.21.15.1:0/1168348830 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d00d05a0 0x7f61d014b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.141+0000 7f61d506d700 1 -- 172.21.15.1:0/1168348830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c8009b30 con 0x7f61d0099a40 2024-03-18T19:06:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.141+0000 7f61d506d700 1 --2- 172.21.15.1:0/1168348830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d0099a40 0x7f61d00c9680 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f61c000c7f0 tx=0x7f61c000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.141+0000 7f61c5ffb700 1 -- 172.21.15.1:0/1168348830 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c0011800 con 0x7f61d0099a40 2024-03-18T19:06:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.141+0000 7f61c5ffb700 1 -- 172.21.15.1:0/1168348830 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c0011960 con 0x7f61d0099a40 2024-03-18T19:06:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.141+0000 7f61d72d1700 1 -- 172.21.15.1:0/1168348830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d009a950 con 0x7f61d0099a40 2024-03-18T19:06:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.141+0000 7f61c5ffb700 1 -- 172.21.15.1:0/1168348830 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c0011d10 con 0x7f61d0099a40 2024-03-18T19:06:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.141+0000 7f61d72d1700 1 -- 172.21.15.1:0/1168348830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d00c8990 con 0x7f61d0099a40 2024-03-18T19:06:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.142+0000 7f61d72d1700 1 -- 172.21.15.1:0/1168348830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d0068240 con 0x7f61d0099a40 2024-03-18T19:06:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.143+0000 7f61c5ffb700 1 -- 172.21.15.1:0/1168348830 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f61c0018360 con 0x7f61d0099a40 2024-03-18T19:06:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.143+0000 7f61c5ffb700 1 --2- 172.21.15.1:0/1168348830 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f61b406adb0 0x7f61b406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.143+0000 7f61c5ffb700 1 -- 172.21.15.1:0/1168348830 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f61c00892f0 con 0x7f61d0099a40 2024-03-18T19:06:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.143+0000 7f61d486c700 1 --2- 172.21.15.1:0/1168348830 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f61b406adb0 0x7f61b406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:39.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.144+0000 7f61d486c700 1 --2- 172.21.15.1:0/1168348830 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f61b406adb0 0x7f61b406d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f61cc006fd0 tx=0x7f61cc008040 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:39.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.148+0000 7f61c5ffb700 1 -- 172.21.15.1:0/1168348830 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f61c00544c0 con 0x7f61d0099a40 2024-03-18T19:06:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.352+0000 7f61d72d1700 1 -- 172.21.15.1:0/1168348830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f61d00cda40 con 0x7f61d0099a40 2024-03-18T19:06:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.352+0000 7f61c5ffb700 1 -- 172.21.15.1:0/1168348830 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v113) v1 ==== 84+0+65 (secure 0 0 0) 0x7f61c0057ae0 con 0x7f61d0099a40 2024-03-18T19:06:39.348 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:39.348 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T19:06:39.348 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:39.348 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-18T19:06:39.348 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-18T19:06:39.348 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:39.349 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T19:06:39.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.354+0000 7f61d72d1700 1 -- 172.21.15.1:0/1168348830 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f61b406adb0 msgr2=0x7f61b406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:39.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.354+0000 7f61d72d1700 1 --2- 172.21.15.1:0/1168348830 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f61b406adb0 0x7f61b406d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f61cc006fd0 tx=0x7f61cc008040 comp rx=0 tx=0).stop 2024-03-18T19:06:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.354+0000 7f61d72d1700 1 -- 172.21.15.1:0/1168348830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d0099a40 msgr2=0x7f61d00c9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.354+0000 7f61d72d1700 1 --2- 172.21.15.1:0/1168348830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d0099a40 0x7f61d00c9680 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f61c000c7f0 tx=0x7f61c000cbb0 comp rx=0 tx=0).stop 2024-03-18T19:06:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.354+0000 7f61d72d1700 1 -- 172.21.15.1:0/1168348830 shutdown_connections 2024-03-18T19:06:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.354+0000 7f61d72d1700 1 --2- 172.21.15.1:0/1168348830 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f61b406adb0 0x7f61b406d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.354+0000 7f61d72d1700 1 --2- 172.21.15.1:0/1168348830 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d00d05a0 0x7f61d014b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.354+0000 7f61d72d1700 1 --2- 172.21.15.1:0/1168348830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d0099a40 0x7f61d00c9680 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.354+0000 7f61d72d1700 1 --2- 172.21.15.1:0/1168348830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d009a360 0x7f61d00ca990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.354+0000 7f61d72d1700 1 -- 172.21.15.1:0/1168348830 >> 172.21.15.1:0/1168348830 conn(0x7f61d00b2cb0 msgr2=0x7f61d00b3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.354+0000 7f61d72d1700 1 -- 172.21.15.1:0/1168348830 shutdown_connections 2024-03-18T19:06:39.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.354+0000 7f61d72d1700 1 -- 172.21.15.1:0/1168348830 wait complete. 2024-03-18T19:06:39.362 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T19:06:39.362 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T19:06:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.496+0000 7fd947d78700 1 Processor -- start 2024-03-18T19:06:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.496+0000 7fd947d78700 1 -- start start 2024-03-18T19:06:39.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.497+0000 7fd947d78700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9400cfe80 0x7fd94006d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.497+0000 7fd947d78700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd94006d7f0 0x7fd9400ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.497+0000 7fd947d78700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd94015d950 con 0x7fd94015d570 2024-03-18T19:06:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.497+0000 7fd947d78700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9400ac8b0 con 0x7fd94006d7f0 2024-03-18T19:06:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.497+0000 7fd947d78700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd94006dca0 con 0x7fd9400cfe80 2024-03-18T19:06:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.497+0000 7fd945b14700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd94015d570 0x7fd9400cf770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55244/0 (socket says 172.21.15.1:55244) 2024-03-18T19:06:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.497+0000 7fd945b14700 1 -- 172.21.15.1:0/3903651012 learned_addr learned my addr 172.21.15.1:0/3903651012 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.497+0000 7fd946315700 1 --2- 172.21.15.1:0/3903651012 >> v2:172.21.15.165:3300/0 conn(0x7fd94006d7f0 0x7fd9400ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.497+0000 7fd945313700 1 --2- 172.21.15.1:0/3903651012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9400cfe80 0x7fd94006d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.497+0000 7fd944b12700 1 -- 172.21.15.1:0/3903651012 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866683430 0 0) 0x7fd94015d950 con 0x7fd94015d570 2024-03-18T19:06:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.497+0000 7fd944b12700 1 -- 172.21.15.1:0/3903651012 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd92c0035b0 con 0x7fd94015d570 2024-03-18T19:06:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd944b12700 1 -- 172.21.15.1:0/3903651012 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3539647575 0 0) 0x7fd92c0035b0 con 0x7fd94015d570 2024-03-18T19:06:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd944b12700 1 -- 172.21.15.1:0/3903651012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9400cfe80 msgr2=0x7fd94006d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:39.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd944b12700 1 --2- 172.21.15.1:0/3903651012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9400cfe80 0x7fd94006d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd944b12700 1 -- 172.21.15.1:0/3903651012 >> v2:172.21.15.165:3300/0 conn(0x7fd94006d7f0 msgr2=0x7fd9400ad5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd944b12700 1 --2- 172.21.15.1:0/3903651012 >> v2:172.21.15.165:3300/0 conn(0x7fd94006d7f0 0x7fd9400ad5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd944b12700 1 -- 172.21.15.1:0/3903651012 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd94006df50 con 0x7fd94015d570 2024-03-18T19:06:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd944b12700 1 -- 172.21.15.1:0/3903651012 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fd930001670 con 0x7fd94015d570 2024-03-18T19:06:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd944b12700 1 -- 172.21.15.1:0/3903651012 >> v1:172.21.15.1:6789/0 conn(0x7fd94015d570 legacy=0x7fd9400cf770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd944b12700 1 --2- 172.21.15.1:0/3903651012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9400cfe80 0x7fd92c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd944b12700 1 --2- 172.21.15.1:0/3903651012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd92c003f80 0x7fd92c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd944b12700 1 --2- 172.21.15.1:0/3903651012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd92c004d90 0x7fd92c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd944b12700 1 -- 172.21.15.1:0/3903651012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9400ac8b0 con 0x7fd92c004d90 2024-03-18T19:06:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd944b12700 1 -- 172.21.15.1:0/3903651012 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94006dca0 con 0x7fd92c003f80 2024-03-18T19:06:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd944b12700 1 -- 172.21.15.1:0/3903651012 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd92c00ba10 con 0x7fd9400cfe80 2024-03-18T19:06:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd946315700 1 --2- 172.21.15.1:0/3903651012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd92c004d90 0x7fd92c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd945b14700 1 --2- 172.21.15.1:0/3903651012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9400cfe80 0x7fd92c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:39.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd945313700 1 --2- 172.21.15.1:0/3903651012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd92c003f80 0x7fd92c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd946315700 1 -- 172.21.15.1:0/3903651012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9400cfe80 msgr2=0x7fd92c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd946315700 1 --2- 172.21.15.1:0/3903651012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9400cfe80 0x7fd92c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd946315700 1 -- 172.21.15.1:0/3903651012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd92c003f80 msgr2=0x7fd92c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd946315700 1 --2- 172.21.15.1:0/3903651012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd92c003f80 0x7fd92c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd946315700 1 -- 172.21.15.1:0/3903651012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92c0035b0 con 0x7fd92c004d90 2024-03-18T19:06:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd945b14700 1 --2- 172.21.15.1:0/3903651012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9400cfe80 0x7fd92c003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd946315700 1 --2- 172.21.15.1:0/3903651012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd92c004d90 0x7fd92c005210 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fd938009fc0 tx=0x7fd92c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67db44c83995ec96 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd944b12700 1 -- 172.21.15.1:0/3903651012 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93800ebd0 con 0x7fd92c004d90 2024-03-18T19:06:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd944b12700 1 -- 172.21.15.1:0/3903651012 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93800f040 con 0x7fd92c004d90 2024-03-18T19:06:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.498+0000 7fd944b12700 1 -- 172.21.15.1:0/3903651012 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd938014480 con 0x7fd92c004d90 2024-03-18T19:06:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.499+0000 7fd947d78700 1 -- 172.21.15.1:0/3903651012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd92c004d90 msgr2=0x7fd92c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.499+0000 7fd947d78700 1 --2- 172.21.15.1:0/3903651012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd92c004d90 0x7fd92c005210 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fd938009fc0 tx=0x7fd92c005880 comp rx=0 tx=0).stop 2024-03-18T19:06:39.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.499+0000 7fd946315700 1 -- 172.21.15.1:0/3903651012 reap_dead start 2024-03-18T19:06:39.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.499+0000 7fd947d78700 1 -- 172.21.15.1:0/3903651012 shutdown_connections 2024-03-18T19:06:39.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.499+0000 7fd947d78700 1 -- 172.21.15.1:0/3903651012 >> 172.21.15.1:0/3903651012 conn(0x7fd94006a750 msgr2=0x7fd94006ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:39.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.499+0000 7fd947d78700 1 -- 172.21.15.1:0/3903651012 shutdown_connections 2024-03-18T19:06:39.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.499+0000 7fd947d78700 1 -- 172.21.15.1:0/3903651012 wait complete. 2024-03-18T19:06:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.500+0000 7fd947d78700 1 Processor -- start 2024-03-18T19:06:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.500+0000 7fd947d78700 1 -- start start 2024-03-18T19:06:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.501+0000 7fd947d78700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9401383b0 0x7fd9401e2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.501+0000 7fd947d78700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9401e2d90 0x7fd940158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.501+0000 7fd947d78700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd940158cd0 0x7fd9401f6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.501+0000 7fd947d78700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9401e5380 con 0x7fd9401e2d90 2024-03-18T19:06:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.501+0000 7fd947d78700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd940159180 con 0x7fd940158cd0 2024-03-18T19:06:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.501+0000 7fd947d78700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd940138790 con 0x7fd9401383b0 2024-03-18T19:06:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.501+0000 7fd945313700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9401e2d90 0x7fd940158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.501+0000 7fd945313700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9401e2d90 0x7fd940158790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35964/0 (socket says 172.21.15.1:35964) 2024-03-18T19:06:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.501+0000 7fd945313700 1 -- 172.21.15.1:0/221565121 learned_addr learned my addr 172.21.15.1:0/221565121 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.501+0000 7fd945b14700 1 --2- 172.21.15.1:0/221565121 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9401383b0 0x7fd9401e2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.501+0000 7fd946315700 1 --2- 172.21.15.1:0/221565121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd940158cd0 0x7fd9401f6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.501+0000 7fd945313700 1 -- 172.21.15.1:0/221565121 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9401383b0 msgr2=0x7fd9401e2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.501+0000 7fd945313700 1 --2- 172.21.15.1:0/221565121 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9401383b0 0x7fd9401e2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.501+0000 7fd945313700 1 -- 172.21.15.1:0/221565121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd940158cd0 msgr2=0x7fd9401f6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.501+0000 7fd945313700 1 --2- 172.21.15.1:0/221565121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd940158cd0 0x7fd9401f6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.501+0000 7fd945313700 1 -- 172.21.15.1:0/221565121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd940148930 con 0x7fd9401e2d90 2024-03-18T19:06:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.501+0000 7fd945b14700 1 --2- 172.21.15.1:0/221565121 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9401383b0 0x7fd9401e2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.501+0000 7fd945313700 1 --2- 172.21.15.1:0/221565121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9401e2d90 0x7fd940158790 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fd93c0119a0 tx=0x7fd93c011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.502+0000 7fd9367fc700 1 -- 172.21.15.1:0/221565121 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c00f940 con 0x7fd9401e2d90 2024-03-18T19:06:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.502+0000 7fd9367fc700 1 -- 172.21.15.1:0/221565121 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93c00faa0 con 0x7fd9401e2d90 2024-03-18T19:06:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.502+0000 7fd9367fc700 1 -- 172.21.15.1:0/221565121 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c00fe30 con 0x7fd9401e2d90 2024-03-18T19:06:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.502+0000 7fd947d78700 1 -- 172.21.15.1:0/221565121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9401e33a0 con 0x7fd9401e2d90 2024-03-18T19:06:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.502+0000 7fd947d78700 1 -- 172.21.15.1:0/221565121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9401e4cc0 con 0x7fd9401e2d90 2024-03-18T19:06:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.503+0000 7fd947d78700 1 -- 172.21.15.1:0/221565121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9400ae880 con 0x7fd9401e2d90 2024-03-18T19:06:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.503+0000 7fd9367fc700 1 -- 172.21.15.1:0/221565121 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd93c014440 con 0x7fd9401e2d90 2024-03-18T19:06:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.504+0000 7fd9367fc700 1 --2- 172.21.15.1:0/221565121 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd92406adb0 0x7fd92406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.504+0000 7fd9367fc700 1 -- 172.21.15.1:0/221565121 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fd93c089940 con 0x7fd9401e2d90 2024-03-18T19:06:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.504+0000 7fd945b14700 1 --2- 172.21.15.1:0/221565121 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd92406adb0 0x7fd92406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.504+0000 7fd945b14700 1 --2- 172.21.15.1:0/221565121 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd92406adb0 0x7fd92406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd930001eb0 tx=0x7fd930004890 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.508+0000 7fd9367fc700 1 -- 172.21.15.1:0/221565121 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd93c054d60 con 0x7fd9401e2d90 2024-03-18T19:06:39.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:39.710+0000 7fd947d78700 1 -- 172.21.15.1:0/221565121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd9400676a0 con 0x7fd9401e2d90 2024-03-18T19:06:40.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.465+0000 7fd9367fc700 1 -- 172.21.15.1:0/221565121 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v114) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd93c058380 con 0x7fd9401e2d90 2024-03-18T19:06:40.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.470+0000 7fd947d78700 1 -- 172.21.15.1:0/221565121 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd92406adb0 msgr2=0x7fd92406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:40.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.470+0000 7fd947d78700 1 --2- 172.21.15.1:0/221565121 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd92406adb0 0x7fd92406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd930001eb0 tx=0x7fd930004890 comp rx=0 tx=0).stop 2024-03-18T19:06:40.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.470+0000 7fd947d78700 1 -- 172.21.15.1:0/221565121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9401e2d90 msgr2=0x7fd940158790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:40.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.470+0000 7fd947d78700 1 --2- 172.21.15.1:0/221565121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9401e2d90 0x7fd940158790 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fd93c0119a0 tx=0x7fd93c011d60 comp rx=0 tx=0).stop 2024-03-18T19:06:40.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.471+0000 7fd947d78700 1 -- 172.21.15.1:0/221565121 shutdown_connections 2024-03-18T19:06:40.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.471+0000 7fd947d78700 1 --2- 172.21.15.1:0/221565121 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd92406adb0 0x7fd92406d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:40.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.471+0000 7fd947d78700 1 --2- 172.21.15.1:0/221565121 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9401383b0 0x7fd9401e2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:40.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.471+0000 7fd947d78700 1 --2- 172.21.15.1:0/221565121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd940158cd0 0x7fd9401f6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:40.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.471+0000 7fd947d78700 1 --2- 172.21.15.1:0/221565121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9401e2d90 0x7fd940158790 secure :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fd93c0119a0 tx=0x7fd93c011d60 comp rx=0 tx=0).stop 2024-03-18T19:06:40.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.471+0000 7fd947d78700 1 -- 172.21.15.1:0/221565121 >> 172.21.15.1:0/221565121 conn(0x7fd94006a750 msgr2=0x7fd9400cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:40.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.472+0000 7fd947d78700 1 -- 172.21.15.1:0/221565121 shutdown_connections 2024-03-18T19:06:40.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.472+0000 7fd947d78700 1 -- 172.21.15.1:0/221565121 wait complete. 2024-03-18T19:06:40.469 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-18T19:06:40.483 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T19:06:40.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.626+0000 7f77cbc5e700 1 Processor -- start 2024-03-18T19:06:40.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.626+0000 7f77cbc5e700 1 -- start start 2024-03-18T19:06:40.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.627+0000 7f77cbc5e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77c4160a70 0x7f77c40d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:40.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.627+0000 7f77cbc5e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f77c41612d0 0x7f77c4161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:40.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.627+0000 7f77cbc5e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c4162170 con 0x7f77c4160e50 2024-03-18T19:06:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.627+0000 7f77cbc5e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77c4161c60 con 0x7f77c41612d0 2024-03-18T19:06:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.627+0000 7f77cbc5e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77c40d06e0 con 0x7f77c4160a70 2024-03-18T19:06:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.627+0000 7f77ca1fb700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f77c41612d0 0x7f77c4161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.627+0000 7f77c91f9700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f77c4160e50 0x7f77c40cffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55246/0 (socket says 172.21.15.1:55246) 2024-03-18T19:06:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.627+0000 7f77c91f9700 1 -- 172.21.15.1:0/110544428 learned_addr learned my addr 172.21.15.1:0/110544428 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.627+0000 7f77c99fa700 1 --2- 172.21.15.1:0/110544428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77c4160a70 0x7f77c40d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.627+0000 7f77ca1fb700 1 -- 172.21.15.1:0/110544428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77c4160a70 msgr2=0x7f77c40d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.627+0000 7f77ca1fb700 1 --2- 172.21.15.1:0/110544428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77c4160a70 0x7f77c40d3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.627+0000 7f77ca1fb700 1 -- 172.21.15.1:0/110544428 >> v1:172.21.15.1:6789/0 conn(0x7f77c4160e50 legacy=0x7f77c40cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.627+0000 7f77ca1fb700 1 -- 172.21.15.1:0/110544428 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c40cef30 con 0x7f77c41612d0 2024-03-18T19:06:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.627+0000 7f77c89f8700 1 -- 172.21.15.1:0/110544428 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1855543879 0 0) 0x7f77c0001110 con 0x7f77c4160e50 2024-03-18T19:06:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.627+0000 7f77c99fa700 1 --2- 172.21.15.1:0/110544428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77c4160a70 0x7f77c40d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.628+0000 7f77ca1fb700 1 --2- 172.21.15.1:0/110544428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77c41612d0 0x7f77c4161720 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f77bc009d90 tx=0x7f77bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3516fcc29bfd517a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.628+0000 7f77c89f8700 1 -- 172.21.15.1:0/110544428 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77bc00e070 con 0x7f77c41612d0 2024-03-18T19:06:40.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.628+0000 7f77c89f8700 1 -- 172.21.15.1:0/110544428 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77bc00bbd0 con 0x7f77c41612d0 2024-03-18T19:06:40.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.628+0000 7f77c89f8700 1 -- 172.21.15.1:0/110544428 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77bc012430 con 0x7f77c41612d0 2024-03-18T19:06:40.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.628+0000 7f77cbc5e700 1 -- 172.21.15.1:0/110544428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77c41612d0 msgr2=0x7f77c4161720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:40.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.628+0000 7f77cbc5e700 1 --2- 172.21.15.1:0/110544428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77c41612d0 0x7f77c4161720 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f77bc009d90 tx=0x7f77bc00b330 comp rx=0 tx=0).stop 2024-03-18T19:06:40.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.628+0000 7f77cbc5e700 1 -- 172.21.15.1:0/110544428 shutdown_connections 2024-03-18T19:06:40.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.628+0000 7f77cbc5e700 1 --2- 172.21.15.1:0/110544428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77c41612d0 0x7f77c4161720 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:40.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.628+0000 7f77cbc5e700 1 --2- 172.21.15.1:0/110544428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77c4160a70 0x7f77c40d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:40.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.628+0000 7f77cbc5e700 1 -- 172.21.15.1:0/110544428 >> 172.21.15.1:0/110544428 conn(0x7f77c409c560 msgr2=0x7f77c40d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:40.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.628+0000 7f77cbc5e700 1 -- 172.21.15.1:0/110544428 shutdown_connections 2024-03-18T19:06:40.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.628+0000 7f77cbc5e700 1 -- 172.21.15.1:0/110544428 wait complete. 2024-03-18T19:06:40.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.630+0000 7f77cbc5e700 1 Processor -- start 2024-03-18T19:06:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.630+0000 7f77cbc5e700 1 -- start start 2024-03-18T19:06:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.630+0000 7f77cbc5e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77c4160a70 0x7f77c4154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.630+0000 7f77cbc5e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77c4160e50 0x7f77c40c6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.630+0000 7f77cbc5e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77c41612d0 0x7f77c40ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.630+0000 7f77cbc5e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c40c69c0 con 0x7f77c4160e50 2024-03-18T19:06:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.630+0000 7f77cbc5e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c40cabb0 con 0x7f77c41612d0 2024-03-18T19:06:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.630+0000 7f77cbc5e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77c415e4f0 con 0x7f77c4160a70 2024-03-18T19:06:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.631+0000 7f77c91f9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77c4160e50 0x7f77c40c6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.631+0000 7f77c91f9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77c4160e50 0x7f77c40c6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35982/0 (socket says 172.21.15.1:35982) 2024-03-18T19:06:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.631+0000 7f77c91f9700 1 -- 172.21.15.1:0/1196859519 learned_addr learned my addr 172.21.15.1:0/1196859519 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.631+0000 7f77c99fa700 1 --2- 172.21.15.1:0/1196859519 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77c4160a70 0x7f77c4154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.631+0000 7f77ca1fb700 1 --2- 172.21.15.1:0/1196859519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77c41612d0 0x7f77c40ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.631+0000 7f77c91f9700 1 -- 172.21.15.1:0/1196859519 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77c4160a70 msgr2=0x7f77c4154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:40.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.631+0000 7f77c91f9700 1 --2- 172.21.15.1:0/1196859519 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77c4160a70 0x7f77c4154df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:40.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.631+0000 7f77c91f9700 1 -- 172.21.15.1:0/1196859519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77c41612d0 msgr2=0x7f77c40ca670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:40.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.631+0000 7f77c91f9700 1 --2- 172.21.15.1:0/1196859519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77c41612d0 0x7f77c40ca670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:40.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.631+0000 7f77c91f9700 1 -- 172.21.15.1:0/1196859519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77bc009a70 con 0x7f77c4160e50 2024-03-18T19:06:40.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.631+0000 7f77c91f9700 1 --2- 172.21.15.1:0/1196859519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77c4160e50 0x7f77c40c6480 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f77c000c520 tx=0x7f77c000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:40.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.631+0000 7f77ba7fc700 1 -- 172.21.15.1:0/1196859519 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c0004c80 con 0x7f77c4160e50 2024-03-18T19:06:40.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.631+0000 7f77ba7fc700 1 -- 172.21.15.1:0/1196859519 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c0004de0 con 0x7f77c4160e50 2024-03-18T19:06:40.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.631+0000 7f77cbc5e700 1 -- 172.21.15.1:0/1196859519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c415e7d0 con 0x7f77c4160e50 2024-03-18T19:06:40.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.631+0000 7f77ba7fc700 1 -- 172.21.15.1:0/1196859519 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c00072d0 con 0x7f77c4160e50 2024-03-18T19:06:40.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.631+0000 7f77cbc5e700 1 -- 172.21.15.1:0/1196859519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77c415ed90 con 0x7f77c4160e50 2024-03-18T19:06:40.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.632+0000 7f77cbc5e700 1 -- 172.21.15.1:0/1196859519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77c4067390 con 0x7f77c4160e50 2024-03-18T19:06:40.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.633+0000 7f77ba7fc700 1 -- 172.21.15.1:0/1196859519 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f77c0007430 con 0x7f77c4160e50 2024-03-18T19:06:40.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.633+0000 7f77ba7fc700 1 --2- 172.21.15.1:0/1196859519 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f77a806adb0 0x7f77a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:40.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.633+0000 7f77ba7fc700 1 -- 172.21.15.1:0/1196859519 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f77c0088e90 con 0x7f77c4160e50 2024-03-18T19:06:40.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.633+0000 7f77c99fa700 1 --2- 172.21.15.1:0/1196859519 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f77a806adb0 0x7f77a806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:40.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.634+0000 7f77c99fa700 1 --2- 172.21.15.1:0/1196859519 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f77a806adb0 0x7f77a806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f77b400c940 tx=0x7f77b4008040 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:40.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.638+0000 7f77ba7fc700 1 -- 172.21.15.1:0/1196859519 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f77c0053f70 con 0x7f77c4160e50 2024-03-18T19:06:40.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:40.839+0000 7f77cbc5e700 1 -- 172.21.15.1:0/1196859519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f77c4162170 con 0x7f77c4160e50 2024-03-18T19:06:41.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.470+0000 7f77ba7fc700 1 -- 172.21.15.1:0/1196859519 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v115) v1 ==== 170+0+0 (secure 0 0 0) 0x7f77c4162170 con 0x7f77c4160e50 2024-03-18T19:06:41.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.478+0000 7f77cbc5e700 1 -- 172.21.15.1:0/1196859519 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f77a806adb0 msgr2=0x7f77a806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:41.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.478+0000 7f77cbc5e700 1 --2- 172.21.15.1:0/1196859519 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f77a806adb0 0x7f77a806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f77b400c940 tx=0x7f77b4008040 comp rx=0 tx=0).stop 2024-03-18T19:06:41.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.478+0000 7f77cbc5e700 1 -- 172.21.15.1:0/1196859519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77c4160e50 msgr2=0x7f77c40c6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:41.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.478+0000 7f77cbc5e700 1 --2- 172.21.15.1:0/1196859519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77c4160e50 0x7f77c40c6480 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f77c000c520 tx=0x7f77c000c830 comp rx=0 tx=0).stop 2024-03-18T19:06:41.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.478+0000 7f77cbc5e700 1 -- 172.21.15.1:0/1196859519 shutdown_connections 2024-03-18T19:06:41.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.478+0000 7f77cbc5e700 1 --2- 172.21.15.1:0/1196859519 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f77a806adb0 0x7f77a806d270 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f77b400c940 tx=0x7f77b4008040 comp rx=0 tx=0).stop 2024-03-18T19:06:41.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.478+0000 7f77cbc5e700 1 --2- 172.21.15.1:0/1196859519 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77c4160a70 0x7f77c4154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:41.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.478+0000 7f77cbc5e700 1 --2- 172.21.15.1:0/1196859519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77c41612d0 0x7f77c40ca670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:41.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.478+0000 7f77cbc5e700 1 --2- 172.21.15.1:0/1196859519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77c4160e50 0x7f77c40c6480 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:41.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.478+0000 7f77cbc5e700 1 -- 172.21.15.1:0/1196859519 >> 172.21.15.1:0/1196859519 conn(0x7f77c409c560 msgr2=0x7f77c40cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:41.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.480+0000 7f77cbc5e700 1 -- 172.21.15.1:0/1196859519 shutdown_connections 2024-03-18T19:06:41.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.482+0000 7f77cbc5e700 1 -- 172.21.15.1:0/1196859519 wait complete. 2024-03-18T19:06:41.486 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-18T19:06:41.501 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T19:06:41.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.639+0000 7faf9f610700 1 Processor -- start 2024-03-18T19:06:41.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.639+0000 7faf9f610700 1 -- start start 2024-03-18T19:06:41.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.640+0000 7faf9f610700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf9815d5a0 0x7faf980ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:41.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.640+0000 7faf9f610700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faf9806ae60 0x7faf98167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:41.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.640+0000 7faf9f610700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf980cf600 con 0x7faf980ce550 2024-03-18T19:06:41.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.640+0000 7faf9f610700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf980ce9f0 con 0x7faf9806ae60 2024-03-18T19:06:41.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.640+0000 7faf9f610700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf980cecd0 con 0x7faf9815d5a0 2024-03-18T19:06:41.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.640+0000 7faf9dbad700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faf9806ae60 0x7faf98167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:41.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.640+0000 7faf9dbad700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faf9806ae60 0x7faf98167140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35984/0 (socket says 172.21.15.1:35984) 2024-03-18T19:06:41.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.640+0000 7faf9dbad700 1 -- 172.21.15.1:0/3498448484 learned_addr learned my addr 172.21.15.1:0/3498448484 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:41.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.640+0000 7faf9d3ac700 1 --2- 172.21.15.1:0/3498448484 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf9815d5a0 0x7faf980ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:41.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.641+0000 7faf9dbad700 1 -- 172.21.15.1:0/3498448484 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf9815d5a0 msgr2=0x7faf980ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:41.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.641+0000 7faf9dbad700 1 --2- 172.21.15.1:0/3498448484 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf9815d5a0 0x7faf980ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:41.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.641+0000 7faf9dbad700 1 -- 172.21.15.1:0/3498448484 >> v1:172.21.15.165:6789/0 conn(0x7faf980ce550 legacy=0x7faf9806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.641+0000 7faf9dbad700 1 -- 172.21.15.1:0/3498448484 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf980cefb0 con 0x7faf9806ae60 2024-03-18T19:06:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.641+0000 7faf9dbad700 1 --2- 172.21.15.1:0/3498448484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf9806ae60 0x7faf98167140 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7faf90009e50 tx=0x7faf9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a2dfb8979046b87 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.641+0000 7faf8ffff700 1 -- 172.21.15.1:0/3498448484 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9000e070 con 0x7faf9806ae60 2024-03-18T19:06:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.641+0000 7faf8ffff700 1 -- 172.21.15.1:0/3498448484 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9000b910 con 0x7faf9806ae60 2024-03-18T19:06:41.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.641+0000 7faf8ffff700 1 -- 172.21.15.1:0/3498448484 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9000bcc0 con 0x7faf9806ae60 2024-03-18T19:06:41.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.641+0000 7faf9f610700 1 -- 172.21.15.1:0/3498448484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf9806ae60 msgr2=0x7faf98167140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:41.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.641+0000 7faf9f610700 1 --2- 172.21.15.1:0/3498448484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf9806ae60 0x7faf98167140 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7faf90009e50 tx=0x7faf9000b330 comp rx=0 tx=0).stop 2024-03-18T19:06:41.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.642+0000 7faf9f610700 1 -- 172.21.15.1:0/3498448484 shutdown_connections 2024-03-18T19:06:41.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.642+0000 7faf9f610700 1 --2- 172.21.15.1:0/3498448484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf9806ae60 0x7faf98167140 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:41.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.642+0000 7faf9f610700 1 --2- 172.21.15.1:0/3498448484 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf9815d5a0 0x7faf980ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:41.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.642+0000 7faf9f610700 1 -- 172.21.15.1:0/3498448484 >> 172.21.15.1:0/3498448484 conn(0x7faf980cc9b0 msgr2=0x7faf980ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.642+0000 7faf9f610700 1 -- 172.21.15.1:0/3498448484 shutdown_connections 2024-03-18T19:06:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.642+0000 7faf9f610700 1 -- 172.21.15.1:0/3498448484 wait complete. 2024-03-18T19:06:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.643+0000 7faf9f610700 1 Processor -- start 2024-03-18T19:06:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.643+0000 7faf9f610700 1 -- start start 2024-03-18T19:06:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.643+0000 7faf9f610700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf9806ae60 0x7faf980c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.643+0000 7faf9f610700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf980ce550 0x7faf980b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.643+0000 7faf9f610700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf9815d5a0 0x7faf980b5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.644+0000 7faf9f610700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9814a140 con 0x7faf980ce550 2024-03-18T19:06:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.644+0000 7faf9f610700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf980c8c00 con 0x7faf9815d5a0 2024-03-18T19:06:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.644+0000 7faf9f610700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf980b5550 con 0x7faf9806ae60 2024-03-18T19:06:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.644+0000 7faf9cbab700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf980ce550 0x7faf980b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.644+0000 7faf9cbab700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf980ce550 0x7faf980b4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35986/0 (socket says 172.21.15.1:35986) 2024-03-18T19:06:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.644+0000 7faf9cbab700 1 -- 172.21.15.1:0/3058735925 learned_addr learned my addr 172.21.15.1:0/3058735925 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.644+0000 7faf9d3ac700 1 --2- 172.21.15.1:0/3058735925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf9806ae60 0x7faf980c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.644+0000 7faf9cbab700 1 -- 172.21.15.1:0/3058735925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf9806ae60 msgr2=0x7faf980c86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.644+0000 7faf9cbab700 1 --2- 172.21.15.1:0/3058735925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf9806ae60 0x7faf980c86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.644+0000 7faf9cbab700 1 -- 172.21.15.1:0/3058735925 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf9815d5a0 msgr2=0x7faf980b5010 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T19:06:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.644+0000 7faf9cbab700 1 --2- 172.21.15.1:0/3058735925 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf9815d5a0 0x7faf980b5010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.644+0000 7faf9cbab700 1 -- 172.21.15.1:0/3058735925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf90009b30 con 0x7faf980ce550 2024-03-18T19:06:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.644+0000 7faf9cbab700 1 --2- 172.21.15.1:0/3058735925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf980ce550 0x7faf980b4ad0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7faf9400c4e0 tx=0x7faf9400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.644+0000 7faf8dffb700 1 -- 172.21.15.1:0/3058735925 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf940052d0 con 0x7faf980ce550 2024-03-18T19:06:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.644+0000 7faf8dffb700 1 -- 172.21.15.1:0/3058735925 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf94005430 con 0x7faf980ce550 2024-03-18T19:06:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.644+0000 7faf9f610700 1 -- 172.21.15.1:0/3058735925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf9814a450 con 0x7faf980ce550 2024-03-18T19:06:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.644+0000 7faf8dffb700 1 -- 172.21.15.1:0/3058735925 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf940038c0 con 0x7faf980ce550 2024-03-18T19:06:41.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.645+0000 7faf9f610700 1 -- 172.21.15.1:0/3058735925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf9814aa10 con 0x7faf980ce550 2024-03-18T19:06:41.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.645+0000 7faf9f610700 1 -- 172.21.15.1:0/3058735925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf9806d640 con 0x7faf980ce550 2024-03-18T19:06:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.646+0000 7faf8dffb700 1 -- 172.21.15.1:0/3058735925 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7faf94003a20 con 0x7faf980ce550 2024-03-18T19:06:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.647+0000 7faf8dffb700 1 --2- 172.21.15.1:0/3058735925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf7c06adb0 0x7faf7c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.647+0000 7faf8dffb700 1 -- 172.21.15.1:0/3058735925 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5929+0+0 (secure 0 0 0) 0x7faf94088a60 con 0x7faf980ce550 2024-03-18T19:06:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.647+0000 7faf9d3ac700 1 --2- 172.21.15.1:0/3058735925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf7c06adb0 0x7faf7c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.647+0000 7faf9d3ac700 1 --2- 172.21.15.1:0/3058735925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf7c06adb0 0x7faf7c06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faf88009bc0 tx=0x7faf880091c0 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.652+0000 7faf8dffb700 1 -- 172.21.15.1:0/3058735925 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faf940539b0 con 0x7faf980ce550 2024-03-18T19:06:41.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.898+0000 7faf9f610700 1 -- 172.21.15.1:0/3058735925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7faf98073350 con 0x7faf980ce550 2024-03-18T19:06:41.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.903+0000 7faf8dffb700 1 -- 172.21.15.1:0/3058735925 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 13 and data pool 14 v46) v1 ==== 174+0+0 (secure 0 0 0) 0x7faf940170c0 con 0x7faf980ce550 2024-03-18T19:06:41.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.909+0000 7faf9f610700 1 -- 172.21.15.1:0/3058735925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf7c06adb0 msgr2=0x7faf7c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:41.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.909+0000 7faf9f610700 1 --2- 172.21.15.1:0/3058735925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf7c06adb0 0x7faf7c06d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faf88009bc0 tx=0x7faf880091c0 comp rx=0 tx=0).stop 2024-03-18T19:06:41.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.909+0000 7faf9f610700 1 -- 172.21.15.1:0/3058735925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf980ce550 msgr2=0x7faf980b4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:41.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.909+0000 7faf9f610700 1 --2- 172.21.15.1:0/3058735925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf980ce550 0x7faf980b4ad0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7faf9400c4e0 tx=0x7faf9400c7f0 comp rx=0 tx=0).stop 2024-03-18T19:06:41.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.910+0000 7faf9f610700 1 -- 172.21.15.1:0/3058735925 shutdown_connections 2024-03-18T19:06:41.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.910+0000 7faf9f610700 1 --2- 172.21.15.1:0/3058735925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf7c06adb0 0x7faf7c06d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:41.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.910+0000 7faf9f610700 1 --2- 172.21.15.1:0/3058735925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf9806ae60 0x7faf980c86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:41.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.910+0000 7faf9f610700 1 --2- 172.21.15.1:0/3058735925 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf9815d5a0 0x7faf980b5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:41.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.910+0000 7faf9f610700 1 --2- 172.21.15.1:0/3058735925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf980ce550 0x7faf980b4ad0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:41.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.910+0000 7faf9f610700 1 -- 172.21.15.1:0/3058735925 >> 172.21.15.1:0/3058735925 conn(0x7faf980cc9b0 msgr2=0x7faf98071520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:41.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.911+0000 7faf9f610700 1 -- 172.21.15.1:0/3058735925 shutdown_connections 2024-03-18T19:06:41.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:41.911+0000 7faf9f610700 1 -- 172.21.15.1:0/3058735925 wait complete. 2024-03-18T19:06:41.915 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 13 and data pool 14 2024-03-18T19:06:41.931 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:06:42.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.060+0000 7f2cbc7eb700 1 Processor -- start 2024-03-18T19:06:42.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.060+0000 7f2cbc7eb700 1 -- start start 2024-03-18T19:06:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.061+0000 7f2cbc7eb700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2cb406e290 0x7f2cb4072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.061+0000 7f2cbc7eb700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb406f490 con 0x7f2cb415d580 2024-03-18T19:06:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.061+0000 7f2cbc7eb700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb4071cf0 con 0x7f2cb406dd50 2024-03-18T19:06:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.061+0000 7f2cbc7eb700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb406f1b0 con 0x7f2cb406e290 2024-03-18T19:06:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cba587700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2cb415d580 0x7f2cb406d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55252/0 (socket says 172.21.15.1:55252) 2024-03-18T19:06:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cba587700 1 -- 172.21.15.1:0/2500182513 learned_addr learned my addr 172.21.15.1:0/2500182513 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cb9d86700 1 --2- 172.21.15.1:0/2500182513 >> v2:172.21.15.165:3300/0 conn(0x7f2cb406e290 0x7f2cb4072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cb9585700 1 -- 172.21.15.1:0/2500182513 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047146620 0 0) 0x7f2cb406f490 con 0x7f2cb415d580 2024-03-18T19:06:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cb9585700 1 -- 172.21.15.1:0/2500182513 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca00035b0 con 0x7f2cb415d580 2024-03-18T19:06:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cb9585700 1 -- 172.21.15.1:0/2500182513 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2299106180 0 0) 0x7f2ca00035b0 con 0x7f2cb415d580 2024-03-18T19:06:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cb9585700 1 -- 172.21.15.1:0/2500182513 >> v2:172.21.15.165:3300/0 conn(0x7f2cb406e290 msgr2=0x7f2cb4072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cb9585700 1 --2- 172.21.15.1:0/2500182513 >> v2:172.21.15.165:3300/0 conn(0x7f2cb406e290 0x7f2cb4072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cb9585700 1 -- 172.21.15.1:0/2500182513 >> v1:172.21.15.165:6789/0 conn(0x7f2cb406dd50 legacy=0x7f2cb40ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:42.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cb9585700 1 -- 172.21.15.1:0/2500182513 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb40716a0 con 0x7f2cb415d580 2024-03-18T19:06:42.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cb9585700 1 -- 172.21.15.1:0/2500182513 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f2cb0001670 con 0x7f2cb415d580 2024-03-18T19:06:42.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cb9585700 1 -- 172.21.15.1:0/2500182513 >> v1:172.21.15.1:6789/0 conn(0x7f2cb415d580 legacy=0x7f2cb406d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:42.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cb9585700 1 --2- 172.21.15.1:0/2500182513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ca0003a00 0x7f2ca0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:42.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cb9585700 1 --2- 172.21.15.1:0/2500182513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca0004850 0x7f2ca0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:42.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cb9585700 1 --2- 172.21.15.1:0/2500182513 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ca0005210 0x7f2ca000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:42.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cb9585700 1 -- 172.21.15.1:0/2500182513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb406f1b0 con 0x7f2ca0004850 2024-03-18T19:06:42.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cb9585700 1 -- 172.21.15.1:0/2500182513 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca000c0c0 con 0x7f2ca0003a00 2024-03-18T19:06:42.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cb9585700 1 -- 172.21.15.1:0/2500182513 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca000c280 con 0x7f2ca0005210 2024-03-18T19:06:42.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cba587700 1 --2- 172.21.15.1:0/2500182513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ca0003a00 0x7f2ca0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:42.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cb9d86700 1 --2- 172.21.15.1:0/2500182513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca0004850 0x7f2ca0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:42.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.062+0000 7f2cbad88700 1 --2- 172.21.15.1:0/2500182513 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ca0005210 0x7f2ca000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:42.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.063+0000 7f2cb9d86700 1 -- 172.21.15.1:0/2500182513 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ca0005210 msgr2=0x7f2ca000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.063+0000 7f2cb9d86700 1 --2- 172.21.15.1:0/2500182513 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ca0005210 0x7f2ca000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.063+0000 7f2cb9d86700 1 -- 172.21.15.1:0/2500182513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ca0003a00 msgr2=0x7f2ca0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.063+0000 7f2cb9d86700 1 --2- 172.21.15.1:0/2500182513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ca0003a00 0x7f2ca0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.063+0000 7f2cb9d86700 1 -- 172.21.15.1:0/2500182513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca00035b0 con 0x7f2ca0004850 2024-03-18T19:06:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.063+0000 7f2cbad88700 1 --2- 172.21.15.1:0/2500182513 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ca0005210 0x7f2ca000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.063+0000 7f2cbad88700 1 -- 172.21.15.1:0/2500182513 reap_dead start 2024-03-18T19:06:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.063+0000 7f2cb9d86700 1 --2- 172.21.15.1:0/2500182513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca0004850 0x7f2ca0004cd0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8009b60 tx=0x7f2ca0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9274ca35193df417 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.063+0000 7f2cb9585700 1 -- 172.21.15.1:0/2500182513 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca800e6c0 con 0x7f2ca0004850 2024-03-18T19:06:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.063+0000 7f2cb9585700 1 -- 172.21.15.1:0/2500182513 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca800e820 con 0x7f2ca0004850 2024-03-18T19:06:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.063+0000 7f2cb9585700 1 -- 172.21.15.1:0/2500182513 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca800eb90 con 0x7f2ca0004850 2024-03-18T19:06:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.063+0000 7f2cbc7eb700 1 -- 172.21.15.1:0/2500182513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca0004850 msgr2=0x7f2ca0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.063+0000 7f2cbc7eb700 1 --2- 172.21.15.1:0/2500182513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca0004850 0x7f2ca0004cd0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8009b60 tx=0x7f2ca0005880 comp rx=0 tx=0).stop 2024-03-18T19:06:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.064+0000 7f2cbc7eb700 1 -- 172.21.15.1:0/2500182513 shutdown_connections 2024-03-18T19:06:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.064+0000 7f2cbc7eb700 1 --2- 172.21.15.1:0/2500182513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ca0004850 0x7f2ca0004cd0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.064+0000 7f2cbc7eb700 1 -- 172.21.15.1:0/2500182513 >> 172.21.15.1:0/2500182513 conn(0x7f2cb40cc990 msgr2=0x7f2cb40ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:42.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.064+0000 7f2cbc7eb700 1 -- 172.21.15.1:0/2500182513 shutdown_connections 2024-03-18T19:06:42.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.064+0000 7f2cbc7eb700 1 -- 172.21.15.1:0/2500182513 wait complete. 2024-03-18T19:06:42.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.065+0000 7f2cbc7eb700 1 Processor -- start 2024-03-18T19:06:42.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.065+0000 7f2cbc7eb700 1 -- start start 2024-03-18T19:06:42.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.066+0000 7f2cbc7eb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ca0004850 0x7f2cb4149b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:42.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.066+0000 7f2cbc7eb700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cb415e9b0 0x7f2cb415ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:42.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.066+0000 7f2cbc7eb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cb414a5a0 0x7f2cb414aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:42.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.066+0000 7f2cbc7eb700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb415f370 con 0x7f2cb414a5a0 2024-03-18T19:06:42.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.066+0000 7f2cbc7eb700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb414a040 con 0x7f2cb415e9b0 2024-03-18T19:06:42.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.066+0000 7f2cbc7eb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb414b2f0 con 0x7f2ca0004850 2024-03-18T19:06:42.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.066+0000 7f2cbad88700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cb414a5a0 0x7f2cb414aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:42.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.066+0000 7f2cbad88700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cb414a5a0 0x7f2cb414aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36010/0 (socket says 172.21.15.1:36010) 2024-03-18T19:06:42.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.066+0000 7f2cba587700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ca0004850 0x7f2cb4149b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:42.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.066+0000 7f2cbad88700 1 -- 172.21.15.1:0/3657762387 learned_addr learned my addr 172.21.15.1:0/3657762387 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:42.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.066+0000 7f2cb9d86700 1 --2- 172.21.15.1:0/3657762387 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cb415e9b0 0x7f2cb415ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:42.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.066+0000 7f2cbad88700 1 -- 172.21.15.1:0/3657762387 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ca0004850 msgr2=0x7f2cb4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:42.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.066+0000 7f2cbad88700 1 --2- 172.21.15.1:0/3657762387 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ca0004850 0x7f2cb4149b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.066+0000 7f2cbad88700 1 -- 172.21.15.1:0/3657762387 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cb415e9b0 msgr2=0x7f2cb415ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:42.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.066+0000 7f2cbad88700 1 --2- 172.21.15.1:0/3657762387 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cb415e9b0 0x7f2cb415ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.066+0000 7f2cbad88700 1 -- 172.21.15.1:0/3657762387 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb414b5d0 con 0x7f2cb414a5a0 2024-03-18T19:06:42.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.066+0000 7f2cb9d86700 1 --2- 172.21.15.1:0/3657762387 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cb415e9b0 0x7f2cb415ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:42.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.066+0000 7f2cbad88700 1 --2- 172.21.15.1:0/3657762387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cb414a5a0 0x7f2cb414aa50 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4067260 tx=0x7f2cac00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:42.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.068+0000 7f2ca6ffd700 1 -- 172.21.15.1:0/3657762387 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac003c10 con 0x7f2cb414a5a0 2024-03-18T19:06:42.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.068+0000 7f2cbc7eb700 1 -- 172.21.15.1:0/3657762387 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb414b8b0 con 0x7f2cb414a5a0 2024-03-18T19:06:42.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.068+0000 7f2cbc7eb700 1 -- 172.21.15.1:0/3657762387 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb414be70 con 0x7f2cb414a5a0 2024-03-18T19:06:42.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.068+0000 7f2ca6ffd700 1 -- 172.21.15.1:0/3657762387 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cac003d70 con 0x7f2cb414a5a0 2024-03-18T19:06:42.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.068+0000 7f2ca6ffd700 1 -- 172.21.15.1:0/3657762387 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac0052d0 con 0x7f2cb414a5a0 2024-03-18T19:06:42.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.069+0000 7f2cbc7eb700 1 -- 172.21.15.1:0/3657762387 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c9c0052f0 con 0x7f2cb414a5a0 2024-03-18T19:06:42.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.070+0000 7f2ca6ffd700 1 -- 172.21.15.1:0/3657762387 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2cac005430 con 0x7f2cb414a5a0 2024-03-18T19:06:42.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.070+0000 7f2ca6ffd700 1 --2- 172.21.15.1:0/3657762387 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2c9806adb0 0x7f2c9806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:42.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.071+0000 7f2ca6ffd700 1 -- 172.21.15.1:0/3657762387 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2cac089510 con 0x7f2cb414a5a0 2024-03-18T19:06:42.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.071+0000 7f2cba587700 1 --2- 172.21.15.1:0/3657762387 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2c9806adb0 0x7f2c9806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:42.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.071+0000 7f2cba587700 1 --2- 172.21.15.1:0/3657762387 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2c9806adb0 0x7f2c9806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0007db0 tx=0x7f2cb0000ed0 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:42.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.076+0000 7f2ca6ffd700 1 -- 172.21.15.1:0/3657762387 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2cac054420 con 0x7f2cb414a5a0 2024-03-18T19:06:42.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.278+0000 7f2cbc7eb700 1 -- 172.21.15.1:0/3657762387 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c9c005870 con 0x7f2cb414a5a0 2024-03-18T19:06:42.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.279+0000 7f2ca6ffd700 1 -- 172.21.15.1:0/3657762387 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16225 (secure 0 0 0) 0x7f2cac057a40 con 0x7f2cb414a5a0 2024-03-18T19:06:42.275 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:42.275 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":116,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:06:41.899219+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:06:39.741939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-18T19:06:40.871304+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:06:42.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.281+0000 7f2cbc7eb700 1 -- 172.21.15.1:0/3657762387 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2c9806adb0 msgr2=0x7f2c9806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:42.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.281+0000 7f2cbc7eb700 1 --2- 172.21.15.1:0/3657762387 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2c9806adb0 0x7f2c9806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0007db0 tx=0x7f2cb0000ed0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.281+0000 7f2cbc7eb700 1 -- 172.21.15.1:0/3657762387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cb414a5a0 msgr2=0x7f2cb414aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:42.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.281+0000 7f2cbc7eb700 1 --2- 172.21.15.1:0/3657762387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cb414a5a0 0x7f2cb414aa50 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4067260 tx=0x7f2cac00ee00 comp rx=0 tx=0).stop 2024-03-18T19:06:42.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.281+0000 7f2cbc7eb700 1 -- 172.21.15.1:0/3657762387 shutdown_connections 2024-03-18T19:06:42.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.281+0000 7f2cbc7eb700 1 --2- 172.21.15.1:0/3657762387 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2c9806adb0 0x7f2c9806d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.281+0000 7f2cbc7eb700 1 --2- 172.21.15.1:0/3657762387 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ca0004850 0x7f2cb4149b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.281+0000 7f2cbc7eb700 1 --2- 172.21.15.1:0/3657762387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cb414a5a0 0x7f2cb414aa50 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.281+0000 7f2cbc7eb700 1 --2- 172.21.15.1:0/3657762387 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cb415e9b0 0x7f2cb415ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.281+0000 7f2cbc7eb700 1 -- 172.21.15.1:0/3657762387 >> 172.21.15.1:0/3657762387 conn(0x7f2cb40cc990 msgr2=0x7f2cb40ae3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:42.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.281+0000 7f2cbc7eb700 1 -- 172.21.15.1:0/3657762387 shutdown_connections 2024-03-18T19:06:42.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.281+0000 7f2cbc7eb700 1 -- 172.21.15.1:0/3657762387 wait complete. 2024-03-18T19:06:42.289 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:06:42.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.419+0000 7f03c37b2700 1 Processor -- start 2024-03-18T19:06:42.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.419+0000 7f03c37b2700 1 -- start start 2024-03-18T19:06:42.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.420+0000 7f03c37b2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03bc0d2e80 0x7f03bc0d3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:42.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.420+0000 7f03c37b2700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03bc161fa0 con 0x7f03bc161950 2024-03-18T19:06:42.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.420+0000 7f03c37b2700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03bc15eaa0 con 0x7f03bc15d810 2024-03-18T19:06:42.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.420+0000 7f03c37b2700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03bc15e610 con 0x7f03bc0d2e80 2024-03-18T19:06:42.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.420+0000 7f03c154e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03bc0d2e80 0x7f03bc0d3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.420+0000 7f03c1d4f700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f03bc161950 0x7f03bc0d0da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55254/0 (socket says 172.21.15.1:55254) 2024-03-18T19:06:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.420+0000 7f03c1d4f700 1 -- 172.21.15.1:0/1579098501 learned_addr learned my addr 172.21.15.1:0/1579098501 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.420+0000 7f03c154e700 1 -- 172.21.15.1:0/1579098501 >> v1:172.21.15.165:6789/0 conn(0x7f03bc15d810 legacy=0x7f03bc161240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.420+0000 7f03c154e700 1 -- 172.21.15.1:0/1579098501 >> v1:172.21.15.1:6789/0 conn(0x7f03bc161950 legacy=0x7f03bc0d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.420+0000 7f03b3fff700 1 -- 172.21.15.1:0/1579098501 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093126968 0 0) 0x7f03bc161fa0 con 0x7f03bc161950 2024-03-18T19:06:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.420+0000 7f03c154e700 1 -- 172.21.15.1:0/1579098501 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03bc15dce0 con 0x7f03bc0d2e80 2024-03-18T19:06:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.420+0000 7f03c154e700 1 --2- 172.21.15.1:0/1579098501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc0d2e80 0x7f03bc0d3260 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f03b8009840 tx=0x7f03b8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16847a34ccd4cb12 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.420+0000 7f03b3fff700 1 -- 172.21.15.1:0/1579098501 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8004230 con 0x7f03bc0d2e80 2024-03-18T19:06:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.421+0000 7f03b3fff700 1 -- 172.21.15.1:0/1579098501 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b8004390 con 0x7f03bc0d2e80 2024-03-18T19:06:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.421+0000 7f03b3fff700 1 -- 172.21.15.1:0/1579098501 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b80036a0 con 0x7f03bc0d2e80 2024-03-18T19:06:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.421+0000 7f03c37b2700 1 -- 172.21.15.1:0/1579098501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc0d2e80 msgr2=0x7f03bc0d3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.421+0000 7f03c37b2700 1 --2- 172.21.15.1:0/1579098501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc0d2e80 0x7f03bc0d3260 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f03b8009840 tx=0x7f03b8009b50 comp rx=0 tx=0).stop 2024-03-18T19:06:42.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.421+0000 7f03c37b2700 1 -- 172.21.15.1:0/1579098501 shutdown_connections 2024-03-18T19:06:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.421+0000 7f03c37b2700 1 --2- 172.21.15.1:0/1579098501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc0d2e80 0x7f03bc0d3260 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.421+0000 7f03c37b2700 1 -- 172.21.15.1:0/1579098501 >> 172.21.15.1:0/1579098501 conn(0x7f03bc08cbf0 msgr2=0x7f03bc09c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.421+0000 7f03c37b2700 1 -- 172.21.15.1:0/1579098501 shutdown_connections 2024-03-18T19:06:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.421+0000 7f03c37b2700 1 -- 172.21.15.1:0/1579098501 wait complete. 2024-03-18T19:06:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.422+0000 7f03c37b2700 1 Processor -- start 2024-03-18T19:06:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.422+0000 7f03c37b2700 1 -- start start 2024-03-18T19:06:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.423+0000 7f03c37b2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03bc0d2e80 0x7f03bc14f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.423+0000 7f03c37b2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc15d810 0x7f03bc0b7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.423+0000 7f03c37b2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03bc161950 0x7f03bc0c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.423+0000 7f03c37b2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc0b5d90 con 0x7f03bc15d810 2024-03-18T19:06:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.423+0000 7f03c37b2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc0b5fe0 con 0x7f03bc161950 2024-03-18T19:06:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.423+0000 7f03c37b2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc0b62c0 con 0x7f03bc0d2e80 2024-03-18T19:06:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.423+0000 7f03c0d4d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc15d810 0x7f03bc0b7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.423+0000 7f03c0d4d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc15d810 0x7f03bc0b7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36038/0 (socket says 172.21.15.1:36038) 2024-03-18T19:06:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.423+0000 7f03c0d4d700 1 -- 172.21.15.1:0/537078951 learned_addr learned my addr 172.21.15.1:0/537078951 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.423+0000 7f03c1d4f700 1 --2- 172.21.15.1:0/537078951 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03bc161950 0x7f03bc0c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.423+0000 7f03c154e700 1 --2- 172.21.15.1:0/537078951 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03bc0d2e80 0x7f03bc14f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.424+0000 7f03c0d4d700 1 -- 172.21.15.1:0/537078951 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03bc0d2e80 msgr2=0x7f03bc14f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.424+0000 7f03c0d4d700 1 --2- 172.21.15.1:0/537078951 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03bc0d2e80 0x7f03bc14f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.424+0000 7f03c0d4d700 1 -- 172.21.15.1:0/537078951 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03bc161950 msgr2=0x7f03bc0c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.424+0000 7f03c0d4d700 1 --2- 172.21.15.1:0/537078951 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03bc161950 0x7f03bc0c9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.424+0000 7f03c0d4d700 1 -- 172.21.15.1:0/537078951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b80094a0 con 0x7f03bc15d810 2024-03-18T19:06:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.424+0000 7f03c0d4d700 1 --2- 172.21.15.1:0/537078951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc15d810 0x7f03bc0b7080 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f03ac00c4a0 tx=0x7f03ac00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.424+0000 7f03b1ffb700 1 -- 172.21.15.1:0/537078951 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac011790 con 0x7f03bc15d810 2024-03-18T19:06:42.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.424+0000 7f03b1ffb700 1 -- 172.21.15.1:0/537078951 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ac0118f0 con 0x7f03bc15d810 2024-03-18T19:06:42.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.424+0000 7f03c37b2700 1 -- 172.21.15.1:0/537078951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03bc0b65a0 con 0x7f03bc15d810 2024-03-18T19:06:42.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.424+0000 7f03b1ffb700 1 -- 172.21.15.1:0/537078951 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac011ca0 con 0x7f03bc15d810 2024-03-18T19:06:42.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.424+0000 7f03c37b2700 1 -- 172.21.15.1:0/537078951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03bc0b6b60 con 0x7f03bc15d810 2024-03-18T19:06:42.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.425+0000 7f03c37b2700 1 -- 172.21.15.1:0/537078951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03bc073350 con 0x7f03bc15d810 2024-03-18T19:06:42.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.426+0000 7f03b1ffb700 1 -- 172.21.15.1:0/537078951 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f03ac01c460 con 0x7f03bc15d810 2024-03-18T19:06:42.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.426+0000 7f03b1ffb700 1 --2- 172.21.15.1:0/537078951 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f03a006adb0 0x7f03a006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:42.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.426+0000 7f03b1ffb700 1 -- 172.21.15.1:0/537078951 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f03ac089260 con 0x7f03bc15d810 2024-03-18T19:06:42.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.426+0000 7f03c154e700 1 --2- 172.21.15.1:0/537078951 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f03a006adb0 0x7f03a006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:42.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.427+0000 7f03c154e700 1 --2- 172.21.15.1:0/537078951 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f03a006adb0 0x7f03a006d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f03b8000c00 tx=0x7f03b801c040 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:42.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.431+0000 7f03b1ffb700 1 -- 172.21.15.1:0/537078951 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f03ac0541f0 con 0x7f03bc15d810 2024-03-18T19:06:42.457 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-18T19:06:42.461+0000 7f10fc1ae700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T19:06:42.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.635+0000 7f03c37b2700 1 -- 172.21.15.1:0/537078951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03bc161730 con 0x7f03bc15d810 2024-03-18T19:06:42.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.636+0000 7f03b1ffb700 1 -- 172.21.15.1:0/537078951 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16225 (secure 0 0 0) 0x7f03ac057810 con 0x7f03bc15d810 2024-03-18T19:06:42.632 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:42.632 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":116,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:06:41.899219+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:06:39.741939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-18T19:06:40.871304+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:06:42.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.638+0000 7f03c37b2700 1 -- 172.21.15.1:0/537078951 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f03a006adb0 msgr2=0x7f03a006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:42.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.638+0000 7f03c37b2700 1 --2- 172.21.15.1:0/537078951 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f03a006adb0 0x7f03a006d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f03b8000c00 tx=0x7f03b801c040 comp rx=0 tx=0).stop 2024-03-18T19:06:42.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.638+0000 7f03c37b2700 1 -- 172.21.15.1:0/537078951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc15d810 msgr2=0x7f03bc0b7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.638+0000 7f03c37b2700 1 --2- 172.21.15.1:0/537078951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc15d810 0x7f03bc0b7080 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f03ac00c4a0 tx=0x7f03ac00c7b0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.638+0000 7f03c37b2700 1 -- 172.21.15.1:0/537078951 shutdown_connections 2024-03-18T19:06:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.638+0000 7f03c37b2700 1 --2- 172.21.15.1:0/537078951 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f03a006adb0 0x7f03a006d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.638+0000 7f03c37b2700 1 --2- 172.21.15.1:0/537078951 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03bc0d2e80 0x7f03bc14f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.638+0000 7f03c37b2700 1 --2- 172.21.15.1:0/537078951 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03bc161950 0x7f03bc0c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.638+0000 7f03c37b2700 1 --2- 172.21.15.1:0/537078951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03bc15d810 0x7f03bc0b7080 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.638+0000 7f03c37b2700 1 -- 172.21.15.1:0/537078951 >> 172.21.15.1:0/537078951 conn(0x7f03bc08cbf0 msgr2=0x7f03bc0ce9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.638+0000 7f03c37b2700 1 -- 172.21.15.1:0/537078951 shutdown_connections 2024-03-18T19:06:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.638+0000 7f03c37b2700 1 -- 172.21.15.1:0/537078951 wait complete. 2024-03-18T19:06:42.646 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T19:06:42.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.779+0000 7f82c4192700 1 Processor -- start 2024-03-18T19:06:42.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.779+0000 7f82c4192700 1 -- start start 2024-03-18T19:06:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.780+0000 7f82c4192700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f82bc071cc0 0x7f82bc0ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.780+0000 7f82c4192700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82bc06d560 0x7f82bc06fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.780+0000 7f82c4192700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc070000 con 0x7f82bc15a240 2024-03-18T19:06:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.780+0000 7f82c4192700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82bc0adff0 con 0x7f82bc071cc0 2024-03-18T19:06:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.780+0000 7f82c4192700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc0cac80 con 0x7f82bc06d560 2024-03-18T19:06:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.780+0000 7f82c172d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f82bc071cc0 0x7f82bc0ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.780+0000 7f82c172d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f82bc071cc0 0x7f82bc0ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36050/0 (socket says 172.21.15.1:36050) 2024-03-18T19:06:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.780+0000 7f82c172d700 1 -- 172.21.15.1:0/265247866 learned_addr learned my addr 172.21.15.1:0/265247866 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.780+0000 7f82c272f700 1 --2- 172.21.15.1:0/265247866 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82bc06d560 0x7f82bc06fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.780+0000 7f82c172d700 1 -- 172.21.15.1:0/265247866 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82bc06d560 msgr2=0x7f82bc06fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.780+0000 7f82c172d700 1 --2- 172.21.15.1:0/265247866 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82bc06d560 0x7f82bc06fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.780+0000 7f82c172d700 1 -- 172.21.15.1:0/265247866 >> v1:172.21.15.165:6789/0 conn(0x7f82bc15a240 legacy=0x7f82bc0715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.780+0000 7f82c172d700 1 -- 172.21.15.1:0/265247866 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc0caf60 con 0x7f82bc071cc0 2024-03-18T19:06:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.780+0000 7f82c172d700 1 --2- 172.21.15.1:0/265247866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc071cc0 0x7f82bc0ca580 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f82b80098f0 tx=0x7f82b8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=268c1fe658d49552 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.780+0000 7f82c0f2c700 1 -- 172.21.15.1:0/265247866 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b8011830 con 0x7f82bc071cc0 2024-03-18T19:06:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.781+0000 7f82c0f2c700 1 -- 172.21.15.1:0/265247866 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b8011990 con 0x7f82bc071cc0 2024-03-18T19:06:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.781+0000 7f82c0f2c700 1 -- 172.21.15.1:0/265247866 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b8011d20 con 0x7f82bc071cc0 2024-03-18T19:06:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.781+0000 7f82c4192700 1 -- 172.21.15.1:0/265247866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc071cc0 msgr2=0x7f82bc0ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.781+0000 7f82c4192700 1 --2- 172.21.15.1:0/265247866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc071cc0 0x7f82bc0ca580 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f82b80098f0 tx=0x7f82b8009c00 comp rx=0 tx=0).stop 2024-03-18T19:06:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.781+0000 7f82c4192700 1 -- 172.21.15.1:0/265247866 shutdown_connections 2024-03-18T19:06:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.781+0000 7f82c4192700 1 --2- 172.21.15.1:0/265247866 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82bc06d560 0x7f82bc06fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.781+0000 7f82c4192700 1 --2- 172.21.15.1:0/265247866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc071cc0 0x7f82bc0ca580 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.781+0000 7f82c4192700 1 -- 172.21.15.1:0/265247866 >> 172.21.15.1:0/265247866 conn(0x7f82bc0cff30 msgr2=0x7f82bc0d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.781+0000 7f82c4192700 1 -- 172.21.15.1:0/265247866 shutdown_connections 2024-03-18T19:06:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.781+0000 7f82c4192700 1 -- 172.21.15.1:0/265247866 wait complete. 2024-03-18T19:06:42.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.782+0000 7f82c4192700 1 Processor -- start 2024-03-18T19:06:42.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.783+0000 7f82c4192700 1 -- start start 2024-03-18T19:06:42.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.783+0000 7f82c4192700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc06d560 0x7f82bc14c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:42.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.783+0000 7f82c4192700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82bc071cc0 0x7f82bc0c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:42.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.783+0000 7f82c4192700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82bc15a240 0x7f82bc0b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.783+0000 7f82c4192700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc0c6570 con 0x7f82bc06d560 2024-03-18T19:06:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.783+0000 7f82c4192700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc0c3ba0 con 0x7f82bc071cc0 2024-03-18T19:06:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.783+0000 7f82c4192700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc0c58f0 con 0x7f82bc15a240 2024-03-18T19:06:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.783+0000 7f82c1f2e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc06d560 0x7f82bc14c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.783+0000 7f82c1f2e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc06d560 0x7f82bc14c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36056/0 (socket says 172.21.15.1:36056) 2024-03-18T19:06:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.783+0000 7f82c1f2e700 1 -- 172.21.15.1:0/3735041763 learned_addr learned my addr 172.21.15.1:0/3735041763 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.783+0000 7f82c172d700 1 --2- 172.21.15.1:0/3735041763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82bc071cc0 0x7f82bc0c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.783+0000 7f82c272f700 1 --2- 172.21.15.1:0/3735041763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82bc15a240 0x7f82bc0b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.784+0000 7f82c1f2e700 1 -- 172.21.15.1:0/3735041763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82bc15a240 msgr2=0x7f82bc0b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.784+0000 7f82c1f2e700 1 --2- 172.21.15.1:0/3735041763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82bc15a240 0x7f82bc0b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.784+0000 7f82c1f2e700 1 -- 172.21.15.1:0/3735041763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82bc071cc0 msgr2=0x7f82bc0c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.784+0000 7f82c1f2e700 1 --2- 172.21.15.1:0/3735041763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82bc071cc0 0x7f82bc0c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.784+0000 7f82c1f2e700 1 -- 172.21.15.1:0/3735041763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b8009550 con 0x7f82bc06d560 2024-03-18T19:06:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.784+0000 7f82c272f700 1 --2- 172.21.15.1:0/3735041763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82bc15a240 0x7f82bc0b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.784+0000 7f82c1f2e700 1 --2- 172.21.15.1:0/3735041763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc06d560 0x7f82bc14c6b0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f82ac00c480 tx=0x7f82ac00c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.784+0000 7f82b27fc700 1 -- 172.21.15.1:0/3735041763 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ac0052d0 con 0x7f82bc06d560 2024-03-18T19:06:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.784+0000 7f82b27fc700 1 -- 172.21.15.1:0/3735041763 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ac005430 con 0x7f82bc06d560 2024-03-18T19:06:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.784+0000 7f82c4192700 1 -- 172.21.15.1:0/3735041763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82bc0c5bd0 con 0x7f82bc06d560 2024-03-18T19:06:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.784+0000 7f82b27fc700 1 -- 172.21.15.1:0/3735041763 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ac0038c0 con 0x7f82bc06d560 2024-03-18T19:06:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.784+0000 7f82c4192700 1 -- 172.21.15.1:0/3735041763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82bc0c6190 con 0x7f82bc06d560 2024-03-18T19:06:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.785+0000 7f82c4192700 1 -- 172.21.15.1:0/3735041763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82bc0676a0 con 0x7f82bc06d560 2024-03-18T19:06:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.785+0000 7f82b27fc700 1 -- 172.21.15.1:0/3735041763 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f82ac003a20 con 0x7f82bc06d560 2024-03-18T19:06:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.786+0000 7f82b27fc700 1 --2- 172.21.15.1:0/3735041763 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f82a006adb0 0x7f82a006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.786+0000 7f82b27fc700 1 -- 172.21.15.1:0/3735041763 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f82ac088920 con 0x7f82bc06d560 2024-03-18T19:06:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.786+0000 7f82c172d700 1 --2- 172.21.15.1:0/3735041763 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f82a006adb0 0x7f82a006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.786+0000 7f82c172d700 1 --2- 172.21.15.1:0/3735041763 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f82a006adb0 0x7f82a006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f82b8009c40 tx=0x7f82b8008030 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:42.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:42.791+0000 7f82b27fc700 1 -- 172.21.15.1:0/3735041763 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f82ac0538b0 con 0x7f82bc06d560 2024-03-18T19:06:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:43.041+0000 7f82c4192700 1 -- 172.21.15.1:0/3735041763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f82bc0b5eb0 con 0x7f82bc06d560 2024-03-18T19:06:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:43.906+0000 7f82b27fc700 1 -- 172.21.15.1:0/3735041763 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v48) v1 ==== 118+0+0 (secure 0 0 0) 0x7f82ac0170b0 con 0x7f82bc06d560 2024-03-18T19:06:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:43.909+0000 7f82c4192700 1 -- 172.21.15.1:0/3735041763 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f82a006adb0 msgr2=0x7f82a006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:43.909+0000 7f82c4192700 1 --2- 172.21.15.1:0/3735041763 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f82a006adb0 0x7f82a006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f82b8009c40 tx=0x7f82b8008030 comp rx=0 tx=0).stop 2024-03-18T19:06:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:43.909+0000 7f82c4192700 1 -- 172.21.15.1:0/3735041763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc06d560 msgr2=0x7f82bc14c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:43.909+0000 7f82c4192700 1 --2- 172.21.15.1:0/3735041763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc06d560 0x7f82bc14c6b0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f82ac00c480 tx=0x7f82ac00c790 comp rx=0 tx=0).stop 2024-03-18T19:06:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:43.909+0000 7f82c4192700 1 -- 172.21.15.1:0/3735041763 shutdown_connections 2024-03-18T19:06:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:43.909+0000 7f82c4192700 1 --2- 172.21.15.1:0/3735041763 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f82a006adb0 0x7f82a006d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:43.909+0000 7f82c4192700 1 --2- 172.21.15.1:0/3735041763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82bc15a240 0x7f82bc0b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:43.909+0000 7f82c4192700 1 --2- 172.21.15.1:0/3735041763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82bc06d560 0x7f82bc14c6b0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:43.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:43.909+0000 7f82c4192700 1 --2- 172.21.15.1:0/3735041763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82bc071cc0 0x7f82bc0c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:43.909+0000 7f82c4192700 1 -- 172.21.15.1:0/3735041763 >> 172.21.15.1:0/3735041763 conn(0x7f82bc0cff30 msgr2=0x7f82bc06f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:43.909+0000 7f82c4192700 1 -- 172.21.15.1:0/3735041763 shutdown_connections 2024-03-18T19:06:43.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:43.909+0000 7f82c4192700 1 -- 172.21.15.1:0/3735041763 wait complete. 2024-03-18T19:06:43.920 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-18T19:06:43.920 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T19:06:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.059+0000 7f9a988e6700 1 Processor -- start 2024-03-18T19:06:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.059+0000 7f9a988e6700 1 -- start start 2024-03-18T19:06:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a988e6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a940cc0f0 0x7f9a940add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a988e6700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a940ce8f0 con 0x7f9a94071d30 2024-03-18T19:06:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a988e6700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a940ae260 con 0x7f9a9415a1d0 2024-03-18T19:06:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a988e6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a940cdb00 con 0x7f9a940cc0f0 2024-03-18T19:06:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a91d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9a94071d30 0x7f9a940cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55268/0 (socket says 172.21.15.1:55268) 2024-03-18T19:06:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a91d9b700 1 -- 172.21.15.1:0/2137856780 learned_addr learned my addr 172.21.15.1:0/2137856780 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a92d9d700 1 --2- 172.21.15.1:0/2137856780 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a940cc0f0 0x7f9a940add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a9159a700 1 -- 172.21.15.1:0/2137856780 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591218913 0 0) 0x7f9a940ce8f0 con 0x7f9a94071d30 2024-03-18T19:06:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a9159a700 1 -- 172.21.15.1:0/2137856780 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a7c0035b0 con 0x7f9a94071d30 2024-03-18T19:06:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a9159a700 1 -- 172.21.15.1:0/2137856780 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3536150992 0 0) 0x7f9a7c0035b0 con 0x7f9a94071d30 2024-03-18T19:06:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a9159a700 1 -- 172.21.15.1:0/2137856780 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a940cc0f0 msgr2=0x7f9a940add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a9159a700 1 --2- 172.21.15.1:0/2137856780 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a940cc0f0 0x7f9a940add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a9159a700 1 -- 172.21.15.1:0/2137856780 >> v1:172.21.15.165:6789/0 conn(0x7f9a9415a1d0 legacy=0x7f9a94071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a9159a700 1 -- 172.21.15.1:0/2137856780 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a940cdce0 con 0x7f9a94071d30 2024-03-18T19:06:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a9159a700 1 -- 172.21.15.1:0/2137856780 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f9a88001670 con 0x7f9a94071d30 2024-03-18T19:06:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a9159a700 1 -- 172.21.15.1:0/2137856780 >> v1:172.21.15.1:6789/0 conn(0x7f9a94071d30 legacy=0x7f9a940cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a9159a700 1 --2- 172.21.15.1:0/2137856780 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a7c003a30 0x7f9a7c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a9159a700 1 --2- 172.21.15.1:0/2137856780 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a940cc0f0 0x7f9a7c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a9159a700 1 --2- 172.21.15.1:0/2137856780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a7c004d90 0x7f9a7c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a9159a700 1 -- 172.21.15.1:0/2137856780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a940cdb00 con 0x7f9a7c004d90 2024-03-18T19:06:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a9159a700 1 -- 172.21.15.1:0/2137856780 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c00ba10 con 0x7f9a7c003a30 2024-03-18T19:06:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.060+0000 7f9a9159a700 1 -- 172.21.15.1:0/2137856780 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c00bbd0 con 0x7f9a940cc0f0 2024-03-18T19:06:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.061+0000 7f9a92d9d700 1 --2- 172.21.15.1:0/2137856780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a7c004d90 0x7f9a7c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.061+0000 7f9a9259c700 1 --2- 172.21.15.1:0/2137856780 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a7c003a30 0x7f9a7c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.061+0000 7f9a91d9b700 1 --2- 172.21.15.1:0/2137856780 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a940cc0f0 0x7f9a7c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.061+0000 7f9a92d9d700 1 -- 172.21.15.1:0/2137856780 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a940cc0f0 msgr2=0x7f9a7c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.061+0000 7f9a92d9d700 1 --2- 172.21.15.1:0/2137856780 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a940cc0f0 0x7f9a7c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.061+0000 7f9a92d9d700 1 -- 172.21.15.1:0/2137856780 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a7c003a30 msgr2=0x7f9a7c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.061+0000 7f9a92d9d700 1 --2- 172.21.15.1:0/2137856780 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a7c003a30 0x7f9a7c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.061+0000 7f9a92d9d700 1 -- 172.21.15.1:0/2137856780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c0035b0 con 0x7f9a7c004d90 2024-03-18T19:06:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.061+0000 7f9a92d9d700 1 --2- 172.21.15.1:0/2137856780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a7c004d90 0x7f9a7c005210 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c010450 tx=0x7f9a7c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e35a0c3ccc67df3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.061+0000 7f9a9159a700 1 -- 172.21.15.1:0/2137856780 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c012430 con 0x7f9a7c004d90 2024-03-18T19:06:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.061+0000 7f9a9159a700 1 -- 172.21.15.1:0/2137856780 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a8c012590 con 0x7f9a7c004d90 2024-03-18T19:06:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.061+0000 7f9a9159a700 1 -- 172.21.15.1:0/2137856780 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c012920 con 0x7f9a7c004d90 2024-03-18T19:06:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.062+0000 7f9a988e6700 1 -- 172.21.15.1:0/2137856780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a7c004d90 msgr2=0x7f9a7c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.062+0000 7f9a988e6700 1 --2- 172.21.15.1:0/2137856780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a7c004d90 0x7f9a7c005210 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c010450 tx=0x7f9a7c005880 comp rx=0 tx=0).stop 2024-03-18T19:06:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.062+0000 7f9a92d9d700 1 -- 172.21.15.1:0/2137856780 reap_dead start 2024-03-18T19:06:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.062+0000 7f9a988e6700 1 -- 172.21.15.1:0/2137856780 shutdown_connections 2024-03-18T19:06:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.062+0000 7f9a988e6700 1 -- 172.21.15.1:0/2137856780 >> 172.21.15.1:0/2137856780 conn(0x7f9a940cc9b0 msgr2=0x7f9a940ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.062+0000 7f9a988e6700 1 -- 172.21.15.1:0/2137856780 shutdown_connections 2024-03-18T19:06:44.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.062+0000 7f9a988e6700 1 -- 172.21.15.1:0/2137856780 wait complete. 2024-03-18T19:06:44.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.063+0000 7f9a988e6700 1 Processor -- start 2024-03-18T19:06:44.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.063+0000 7f9a988e6700 1 -- start start 2024-03-18T19:06:44.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.064+0000 7f9a988e6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a7c004d90 0x7f9a940c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:44.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.064+0000 7f9a988e6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a940c3a60 0x7f9a941f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:44.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.064+0000 7f9a988e6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a940c3ee0 0x7f9a941f34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:44.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.064+0000 7f9a988e6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a940c76a0 con 0x7f9a940c3a60 2024-03-18T19:06:44.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.064+0000 7f9a988e6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a940c2990 con 0x7f9a940c3ee0 2024-03-18T19:06:44.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.064+0000 7f9a988e6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a940c7000 con 0x7f9a7c004d90 2024-03-18T19:06:44.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.064+0000 7f9a91d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a940c3a60 0x7f9a941f2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:44.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.064+0000 7f9a91d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a940c3a60 0x7f9a941f2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36066/0 (socket says 172.21.15.1:36066) 2024-03-18T19:06:44.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.064+0000 7f9a91d9b700 1 -- 172.21.15.1:0/3703671505 learned_addr learned my addr 172.21.15.1:0/3703671505 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:44.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.064+0000 7f9a9259c700 1 --2- 172.21.15.1:0/3703671505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a7c004d90 0x7f9a940c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:44.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.064+0000 7f9a92d9d700 1 --2- 172.21.15.1:0/3703671505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a940c3ee0 0x7f9a941f34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:44.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.064+0000 7f9a91d9b700 1 -- 172.21.15.1:0/3703671505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a7c004d90 msgr2=0x7f9a940c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:44.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.064+0000 7f9a91d9b700 1 --2- 172.21.15.1:0/3703671505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a7c004d90 0x7f9a940c89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:44.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.064+0000 7f9a91d9b700 1 -- 172.21.15.1:0/3703671505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a940c3ee0 msgr2=0x7f9a941f34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:44.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.064+0000 7f9a91d9b700 1 --2- 172.21.15.1:0/3703671505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a940c3ee0 0x7f9a941f34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:44.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.064+0000 7f9a91d9b700 1 -- 172.21.15.1:0/3703671505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a941f3b70 con 0x7f9a940c3a60 2024-03-18T19:06:44.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.064+0000 7f9a9259c700 1 --2- 172.21.15.1:0/3703671505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a7c004d90 0x7f9a940c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:44.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.064+0000 7f9a91d9b700 1 --2- 172.21.15.1:0/3703671505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a940c3a60 0x7f9a941f2fb0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f9a880072c0 tx=0x7f9a880075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:44.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.065+0000 7f9a82ffd700 1 -- 172.21.15.1:0/3703671505 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a880077b0 con 0x7f9a940c3a60 2024-03-18T19:06:44.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.065+0000 7f9a82ffd700 1 -- 172.21.15.1:0/3703671505 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a8800f870 con 0x7f9a940c3a60 2024-03-18T19:06:44.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.065+0000 7f9a988e6700 1 -- 172.21.15.1:0/3703671505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a941f3e50 con 0x7f9a940c3a60 2024-03-18T19:06:44.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.065+0000 7f9a82ffd700 1 -- 172.21.15.1:0/3703671505 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8800fc20 con 0x7f9a940c3a60 2024-03-18T19:06:44.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.065+0000 7f9a988e6700 1 -- 172.21.15.1:0/3703671505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a940c8f10 con 0x7f9a940c3a60 2024-03-18T19:06:44.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.066+0000 7f9a988e6700 1 -- 172.21.15.1:0/3703671505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a940c77e0 con 0x7f9a940c3a60 2024-03-18T19:06:44.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.066+0000 7f9a82ffd700 1 -- 172.21.15.1:0/3703671505 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9a8800fd80 con 0x7f9a940c3a60 2024-03-18T19:06:44.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.066+0000 7f9a82ffd700 1 --2- 172.21.15.1:0/3703671505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9a7406adb0 0x7f9a7406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:44.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.067+0000 7f9a82ffd700 1 -- 172.21.15.1:0/3703671505 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9a88089560 con 0x7f9a940c3a60 2024-03-18T19:06:44.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.067+0000 7f9a9259c700 1 --2- 172.21.15.1:0/3703671505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9a7406adb0 0x7f9a7406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:44.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.067+0000 7f9a9259c700 1 --2- 172.21.15.1:0/3703671505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9a7406adb0 0x7f9a7406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9a84000c00 tx=0x7f9a84009040 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:44.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.071+0000 7f9a82ffd700 1 -- 172.21.15.1:0/3703671505 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9a940c77e0 con 0x7f9a940c3a60 2024-03-18T19:06:44.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.320+0000 7f9a988e6700 1 -- 172.21.15.1:0/3703671505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f9a940c77e0 con 0x7f9a940c3a60 2024-03-18T19:06:44.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.911+0000 7f9a82ffd700 1 -- 172.21.15.1:0/3703671505 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v49) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9a88054580 con 0x7f9a940c3a60 2024-03-18T19:06:44.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.914+0000 7f9a988e6700 1 -- 172.21.15.1:0/3703671505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9a7406adb0 msgr2=0x7f9a7406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:44.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.914+0000 7f9a988e6700 1 --2- 172.21.15.1:0/3703671505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9a7406adb0 0x7f9a7406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9a84000c00 tx=0x7f9a84009040 comp rx=0 tx=0).stop 2024-03-18T19:06:44.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.914+0000 7f9a988e6700 1 -- 172.21.15.1:0/3703671505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a940c3a60 msgr2=0x7f9a941f2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.914+0000 7f9a988e6700 1 --2- 172.21.15.1:0/3703671505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a940c3a60 0x7f9a941f2fb0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f9a880072c0 tx=0x7f9a880075d0 comp rx=0 tx=0).stop 2024-03-18T19:06:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.914+0000 7f9a988e6700 1 -- 172.21.15.1:0/3703671505 shutdown_connections 2024-03-18T19:06:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.914+0000 7f9a988e6700 1 --2- 172.21.15.1:0/3703671505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9a7406adb0 0x7f9a7406d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.914+0000 7f9a988e6700 1 --2- 172.21.15.1:0/3703671505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9a7c004d90 0x7f9a940c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.914+0000 7f9a988e6700 1 --2- 172.21.15.1:0/3703671505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a940c3ee0 0x7f9a941f34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.914+0000 7f9a988e6700 1 --2- 172.21.15.1:0/3703671505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a940c3a60 0x7f9a941f2fb0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.914+0000 7f9a988e6700 1 -- 172.21.15.1:0/3703671505 >> 172.21.15.1:0/3703671505 conn(0x7f9a940cc9b0 msgr2=0x7f9a9406bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.914+0000 7f9a988e6700 1 -- 172.21.15.1:0/3703671505 shutdown_connections 2024-03-18T19:06:44.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:44.914+0000 7f9a988e6700 1 -- 172.21.15.1:0/3703671505 wait complete. 2024-03-18T19:06:44.924 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T19:06:44.924 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:06:45.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.060+0000 7f5dd4ef2700 1 Processor -- start 2024-03-18T19:06:45.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.060+0000 7f5dd4ef2700 1 -- start start 2024-03-18T19:06:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.061+0000 7f5dd4ef2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5dd015aff0 0x7f5dd015f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.061+0000 7f5dd4ef2700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd015fcc0 con 0x7f5dd015b3d0 2024-03-18T19:06:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.061+0000 7f5dd4ef2700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd00ce200 con 0x7f5dd0073b00 2024-03-18T19:06:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.061+0000 7f5dd4ef2700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd00ce4a0 con 0x7f5dd015aff0 2024-03-18T19:06:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.061+0000 7f5dce59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5dd015aff0 0x7f5dd015f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.061+0000 7f5dcdd9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5dd015b3d0 0x7f5dd00cf540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47140/0 (socket says 172.21.15.1:47140) 2024-03-18T19:06:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.061+0000 7f5dcdd9b700 1 -- 172.21.15.1:0/2156815095 learned_addr learned my addr 172.21.15.1:0/2156815095 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.062+0000 7f5dce59c700 1 -- 172.21.15.1:0/2156815095 >> v1:172.21.15.165:6789/0 conn(0x7f5dd0073b00 legacy=0x7f5dd00cb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.062+0000 7f5dce59c700 1 -- 172.21.15.1:0/2156815095 >> v1:172.21.15.1:6789/0 conn(0x7f5dd015b3d0 legacy=0x7f5dd00cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.062+0000 7f5dce59c700 1 -- 172.21.15.1:0/2156815095 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd00ce780 con 0x7f5dd015aff0 2024-03-18T19:06:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.062+0000 7f5dcd59a700 1 -- 172.21.15.1:0/2156815095 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700560483 0 0) 0x7f5dd015fcc0 con 0x7f5dd015b3d0 2024-03-18T19:06:45.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.062+0000 7f5dce59c700 1 --2- 172.21.15.1:0/2156815095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd015aff0 0x7f5dd015f780 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f5dd00cd7f0 tx=0x7f5dc4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=413621658cbde154 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.062+0000 7f5dcd59a700 1 -- 172.21.15.1:0/2156815095 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4011830 con 0x7f5dd015aff0 2024-03-18T19:06:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.062+0000 7f5dcd59a700 1 -- 172.21.15.1:0/2156815095 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc4011990 con 0x7f5dd015aff0 2024-03-18T19:06:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.062+0000 7f5dcd59a700 1 -- 172.21.15.1:0/2156815095 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4011d20 con 0x7f5dd015aff0 2024-03-18T19:06:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.063+0000 7f5dd4ef2700 1 -- 172.21.15.1:0/2156815095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd015aff0 msgr2=0x7f5dd015f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.063+0000 7f5dd4ef2700 1 --2- 172.21.15.1:0/2156815095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd015aff0 0x7f5dd015f780 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f5dd00cd7f0 tx=0x7f5dc4009b20 comp rx=0 tx=0).stop 2024-03-18T19:06:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.063+0000 7f5dd4ef2700 1 -- 172.21.15.1:0/2156815095 shutdown_connections 2024-03-18T19:06:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.063+0000 7f5dd4ef2700 1 --2- 172.21.15.1:0/2156815095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd015aff0 0x7f5dd015f780 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.063+0000 7f5dd4ef2700 1 -- 172.21.15.1:0/2156815095 >> 172.21.15.1:0/2156815095 conn(0x7f5dd006a750 msgr2=0x7f5dd006ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.063+0000 7f5dd4ef2700 1 -- 172.21.15.1:0/2156815095 shutdown_connections 2024-03-18T19:06:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.063+0000 7f5dd4ef2700 1 -- 172.21.15.1:0/2156815095 wait complete. 2024-03-18T19:06:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.064+0000 7f5dd4ef2700 1 Processor -- start 2024-03-18T19:06:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.064+0000 7f5dd4ef2700 1 -- start start 2024-03-18T19:06:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.065+0000 7f5dd4ef2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd0073b00 0x7f5dd0136f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.065+0000 7f5dd4ef2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dd015aff0 0x7f5dd00a8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.065+0000 7f5dd4ef2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dd015b3d0 0x7f5dd00b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.065+0000 7f5dd4ef2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd015a010 con 0x7f5dd0073b00 2024-03-18T19:06:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.065+0000 7f5dd4ef2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd01374d0 con 0x7f5dd015b3d0 2024-03-18T19:06:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.065+0000 7f5dd4ef2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd0158f70 con 0x7f5dd015aff0 2024-03-18T19:06:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.065+0000 7f5dce59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd0073b00 0x7f5dd0136f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.065+0000 7f5dce59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd0073b00 0x7f5dd0136f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48968/0 (socket says 172.21.15.1:48968) 2024-03-18T19:06:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.065+0000 7f5dce59c700 1 -- 172.21.15.1:0/3823609137 learned_addr learned my addr 172.21.15.1:0/3823609137 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.065+0000 7f5dce59c700 1 -- 172.21.15.1:0/3823609137 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dd015aff0 msgr2=0x7f5dd00a8910 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T19:06:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.065+0000 7f5dce59c700 1 --2- 172.21.15.1:0/3823609137 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dd015aff0 0x7f5dd00a8910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.065+0000 7f5dce59c700 1 -- 172.21.15.1:0/3823609137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dd015b3d0 msgr2=0x7f5dd00b02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.065+0000 7f5dced9d700 1 --2- 172.21.15.1:0/3823609137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dd015b3d0 0x7f5dd00b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:45.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.065+0000 7f5dce59c700 1 --2- 172.21.15.1:0/3823609137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dd015b3d0 0x7f5dd00b02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.065+0000 7f5dce59c700 1 -- 172.21.15.1:0/3823609137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc40094a0 con 0x7f5dd0073b00 2024-03-18T19:06:45.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.065+0000 7f5dce59c700 1 --2- 172.21.15.1:0/3823609137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd0073b00 0x7f5dd0136f90 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4000c00 tx=0x7f5dc4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:45.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.066+0000 7f5dbeffd700 1 -- 172.21.15.1:0/3823609137 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc40038a0 con 0x7f5dd0073b00 2024-03-18T19:06:45.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.066+0000 7f5dbeffd700 1 -- 172.21.15.1:0/3823609137 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc401e070 con 0x7f5dd0073b00 2024-03-18T19:06:45.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.066+0000 7f5dd4ef2700 1 -- 172.21.15.1:0/3823609137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd0159250 con 0x7f5dd0073b00 2024-03-18T19:06:45.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.066+0000 7f5dbeffd700 1 -- 172.21.15.1:0/3823609137 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4003c40 con 0x7f5dd0073b00 2024-03-18T19:06:45.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.066+0000 7f5dd4ef2700 1 -- 172.21.15.1:0/3823609137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd0159810 con 0x7f5dd0073b00 2024-03-18T19:06:45.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.066+0000 7f5dd4ef2700 1 -- 172.21.15.1:0/3823609137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd0144400 con 0x7f5dd0073b00 2024-03-18T19:06:45.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.067+0000 7f5dbeffd700 1 -- 172.21.15.1:0/3823609137 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5dc402d450 con 0x7f5dd0073b00 2024-03-18T19:06:45.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.067+0000 7f5dbeffd700 1 --2- 172.21.15.1:0/3823609137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5db006adb0 0x7f5db006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:45.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.067+0000 7f5dbeffd700 1 -- 172.21.15.1:0/3823609137 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5dc408a680 con 0x7f5dd0073b00 2024-03-18T19:06:45.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.068+0000 7f5dcdd9b700 1 --2- 172.21.15.1:0/3823609137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5db006adb0 0x7f5db006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:45.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.068+0000 7f5dcdd9b700 1 --2- 172.21.15.1:0/3823609137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5db006adb0 0x7f5db006d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0007fd0 tx=0x7f5dc0007920 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.072+0000 7f5dbeffd700 1 -- 172.21.15.1:0/3823609137 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5dc40556a0 con 0x7f5dd0073b00 2024-03-18T19:06:45.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.320+0000 7f5dd4ef2700 1 -- 172.21.15.1:0/3823609137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5dd00676a0 con 0x7f5dd0073b00 2024-03-18T19:06:45.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.321+0000 7f5dbeffd700 1 -- 172.21.15.1:0/3823609137 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4733 (secure 0 0 0) 0x7f5dc4020090 con 0x7f5dd0073b00 2024-03-18T19:06:45.317 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:45.317 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6431,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/983420410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":983420410},{"type":"v1","addr":"172.21.15.1:6836","nonce":983420410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6440,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1569519191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1569519191},{"type":"v1","addr":"172.21.15.165:6834","nonce":1569519191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1528229019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1528229019},{"type":"v1","addr":"172.21.15.165:6835","nonce":1528229019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:06:41.899614+0000","modified":"2024-03-18T19:06:44.904965+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"c","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2226098738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2226098738},{"type":"v1","addr":"172.21.15.1:6837","nonce":2226098738}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-18T19:06:45.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.323+0000 7f5dd4ef2700 1 -- 172.21.15.1:0/3823609137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5db006adb0 msgr2=0x7f5db006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:45.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.323+0000 7f5dd4ef2700 1 --2- 172.21.15.1:0/3823609137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5db006adb0 0x7f5db006d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0007fd0 tx=0x7f5dc0007920 comp rx=0 tx=0).stop 2024-03-18T19:06:45.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.323+0000 7f5dd4ef2700 1 -- 172.21.15.1:0/3823609137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd0073b00 msgr2=0x7f5dd0136f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:45.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.323+0000 7f5dd4ef2700 1 --2- 172.21.15.1:0/3823609137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd0073b00 0x7f5dd0136f90 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4000c00 tx=0x7f5dc4004750 comp rx=0 tx=0).stop 2024-03-18T19:06:45.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.323+0000 7f5dd4ef2700 1 -- 172.21.15.1:0/3823609137 shutdown_connections 2024-03-18T19:06:45.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.323+0000 7f5dd4ef2700 1 --2- 172.21.15.1:0/3823609137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5db006adb0 0x7f5db006d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.323+0000 7f5dd4ef2700 1 --2- 172.21.15.1:0/3823609137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dd015b3d0 0x7f5dd00b02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.323+0000 7f5dd4ef2700 1 --2- 172.21.15.1:0/3823609137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd0073b00 0x7f5dd0136f90 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.323+0000 7f5dd4ef2700 1 --2- 172.21.15.1:0/3823609137 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dd015aff0 0x7f5dd00a8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.323+0000 7f5dd4ef2700 1 -- 172.21.15.1:0/3823609137 >> 172.21.15.1:0/3823609137 conn(0x7f5dd006a750 msgr2=0x7f5dd00cffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:45.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.323+0000 7f5dd4ef2700 1 -- 172.21.15.1:0/3823609137 shutdown_connections 2024-03-18T19:06:45.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.323+0000 7f5dd4ef2700 1 -- 172.21.15.1:0/3823609137 wait complete. 2024-03-18T19:06:45.320 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 49 2024-03-18T19:06:45.331 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:06:45.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.460+0000 7fbe349c5700 1 Processor -- start 2024-03-18T19:06:45.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.460+0000 7fbe349c5700 1 -- start start 2024-03-18T19:06:45.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.461+0000 7fbe349c5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbe30099a40 0x7fbe30099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:45.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.461+0000 7fbe349c5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbe300cc310 0x7fbe300cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.461+0000 7fbe349c5700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe3009a3e0 con 0x7fbe3015d680 2024-03-18T19:06:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.461+0000 7fbe349c5700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe300d0c80 con 0x7fbe300cc310 2024-03-18T19:06:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.461+0000 7fbe349c5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe300d09a0 con 0x7fbe30099a40 2024-03-18T19:06:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.461+0000 7fbe2ed9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbe300cc310 0x7fbe300cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.461+0000 7fbe2e59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbe3015d680 0x7fbe300cff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47144/0 (socket says 172.21.15.1:47144) 2024-03-18T19:06:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.461+0000 7fbe2e59c700 1 -- 172.21.15.1:0/4049313939 learned_addr learned my addr 172.21.15.1:0/4049313939 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.461+0000 7fbe2dd9b700 1 --2- 172.21.15.1:0/4049313939 >> v2:172.21.15.165:3300/0 conn(0x7fbe30099a40 0x7fbe30099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.461+0000 7fbe2ed9d700 1 -- 172.21.15.1:0/4049313939 >> v2:172.21.15.165:3300/0 conn(0x7fbe30099a40 msgr2=0x7fbe30099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.461+0000 7fbe2ed9d700 1 --2- 172.21.15.1:0/4049313939 >> v2:172.21.15.165:3300/0 conn(0x7fbe30099a40 0x7fbe30099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.461+0000 7fbe2ed9d700 1 -- 172.21.15.1:0/4049313939 >> v1:172.21.15.1:6789/0 conn(0x7fbe3015d680 legacy=0x7fbe300cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.461+0000 7fbe2ed9d700 1 -- 172.21.15.1:0/4049313939 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe300cb830 con 0x7fbe300cc310 2024-03-18T19:06:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.461+0000 7fbe2d59a700 1 -- 172.21.15.1:0/4049313939 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422520947 0 0) 0x7fbe3009a3e0 con 0x7fbe3015d680 2024-03-18T19:06:45.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.461+0000 7fbe2dd9b700 1 --2- 172.21.15.1:0/4049313939 >> v2:172.21.15.165:3300/0 conn(0x7fbe30099a40 0x7fbe30099ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:45.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.461+0000 7fbe2ed9d700 1 --2- 172.21.15.1:0/4049313939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe300cc310 0x7fbe300cc7a0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fbe30071cb0 tx=0x7fbe2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e853ab37fa50cb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:45.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.461+0000 7fbe2d59a700 1 -- 172.21.15.1:0/4049313939 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2800e070 con 0x7fbe300cc310 2024-03-18T19:06:45.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.461+0000 7fbe2d59a700 1 -- 172.21.15.1:0/4049313939 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2800bbd0 con 0x7fbe300cc310 2024-03-18T19:06:45.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.461+0000 7fbe2d59a700 1 -- 172.21.15.1:0/4049313939 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe28012430 con 0x7fbe300cc310 2024-03-18T19:06:45.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.462+0000 7fbe349c5700 1 -- 172.21.15.1:0/4049313939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe300cc310 msgr2=0x7fbe300cc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:45.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.462+0000 7fbe349c5700 1 --2- 172.21.15.1:0/4049313939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe300cc310 0x7fbe300cc7a0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fbe30071cb0 tx=0x7fbe2800b330 comp rx=0 tx=0).stop 2024-03-18T19:06:45.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.462+0000 7fbe349c5700 1 -- 172.21.15.1:0/4049313939 shutdown_connections 2024-03-18T19:06:45.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.462+0000 7fbe349c5700 1 --2- 172.21.15.1:0/4049313939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe300cc310 0x7fbe300cc7a0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.462+0000 7fbe349c5700 1 --2- 172.21.15.1:0/4049313939 >> v2:172.21.15.165:3300/0 conn(0x7fbe30099a40 0x7fbe30099ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.462+0000 7fbe349c5700 1 -- 172.21.15.1:0/4049313939 >> 172.21.15.1:0/4049313939 conn(0x7fbe300b2ba0 msgr2=0x7fbe300b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:45.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.462+0000 7fbe349c5700 1 -- 172.21.15.1:0/4049313939 shutdown_connections 2024-03-18T19:06:45.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.462+0000 7fbe349c5700 1 -- 172.21.15.1:0/4049313939 wait complete. 2024-03-18T19:06:45.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.463+0000 7fbe349c5700 1 Processor -- start 2024-03-18T19:06:45.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.463+0000 7fbe349c5700 1 -- start start 2024-03-18T19:06:45.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.464+0000 7fbe349c5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe30099a40 0x7fbe30160270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:45.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.464+0000 7fbe349c5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe300cc310 0x7fbe30158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:45.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.464+0000 7fbe349c5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe3015d680 0x7fbe3015a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:45.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.464+0000 7fbe349c5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe300c8b50 con 0x7fbe300cc310 2024-03-18T19:06:45.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.464+0000 7fbe349c5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3015ef60 con 0x7fbe3015d680 2024-03-18T19:06:45.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.464+0000 7fbe349c5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe300c8220 con 0x7fbe30099a40 2024-03-18T19:06:45.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.464+0000 7fbe2dd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe300cc310 0x7fbe30158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:45.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.464+0000 7fbe2dd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe300cc310 0x7fbe30158a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48980/0 (socket says 172.21.15.1:48980) 2024-03-18T19:06:45.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.464+0000 7fbe2dd9b700 1 -- 172.21.15.1:0/2604584758 learned_addr learned my addr 172.21.15.1:0/2604584758 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:45.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.464+0000 7fbe2e59c700 1 --2- 172.21.15.1:0/2604584758 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe30099a40 0x7fbe30160270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.465+0000 7fbe2ed9d700 1 --2- 172.21.15.1:0/2604584758 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe3015d680 0x7fbe3015a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.465+0000 7fbe2dd9b700 1 -- 172.21.15.1:0/2604584758 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe30099a40 msgr2=0x7fbe30160270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.465+0000 7fbe2dd9b700 1 --2- 172.21.15.1:0/2604584758 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe30099a40 0x7fbe30160270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.465+0000 7fbe2dd9b700 1 -- 172.21.15.1:0/2604584758 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe3015d680 msgr2=0x7fbe3015a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.465+0000 7fbe2dd9b700 1 --2- 172.21.15.1:0/2604584758 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe3015d680 0x7fbe3015a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.465+0000 7fbe2dd9b700 1 -- 172.21.15.1:0/2604584758 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe28009a70 con 0x7fbe300cc310 2024-03-18T19:06:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.465+0000 7fbe2dd9b700 1 --2- 172.21.15.1:0/2604584758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe300cc310 0x7fbe30158a10 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400c750 tx=0x7fbe2400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.465+0000 7fbe1effd700 1 -- 172.21.15.1:0/2604584758 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe24011800 con 0x7fbe300cc310 2024-03-18T19:06:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.465+0000 7fbe1effd700 1 -- 172.21.15.1:0/2604584758 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe24011960 con 0x7fbe300cc310 2024-03-18T19:06:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.465+0000 7fbe349c5700 1 -- 172.21.15.1:0/2604584758 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe300c8500 con 0x7fbe300cc310 2024-03-18T19:06:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.465+0000 7fbe1effd700 1 -- 172.21.15.1:0/2604584758 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe24011cd0 con 0x7fbe300cc310 2024-03-18T19:06:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.465+0000 7fbe349c5700 1 -- 172.21.15.1:0/2604584758 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe3015ad10 con 0x7fbe300cc310 2024-03-18T19:06:45.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.466+0000 7fbe349c5700 1 -- 172.21.15.1:0/2604584758 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe3006d860 con 0x7fbe300cc310 2024-03-18T19:06:45.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.467+0000 7fbe1effd700 1 -- 172.21.15.1:0/2604584758 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbe24021350 con 0x7fbe300cc310 2024-03-18T19:06:45.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.467+0000 7fbe1effd700 1 --2- 172.21.15.1:0/2604584758 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbe1006adb0 0x7fbe1006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:45.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.467+0000 7fbe1effd700 1 -- 172.21.15.1:0/2604584758 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fbe24089700 con 0x7fbe300cc310 2024-03-18T19:06:45.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.467+0000 7fbe2e59c700 1 --2- 172.21.15.1:0/2604584758 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbe1006adb0 0x7fbe1006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:45.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.467+0000 7fbe2e59c700 1 --2- 172.21.15.1:0/2604584758 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbe1006adb0 0x7fbe1006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbe20000f60 tx=0x7fbe20007c90 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:45.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.472+0000 7fbe1effd700 1 -- 172.21.15.1:0/2604584758 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbe24054580 con 0x7fbe300cc310 2024-03-18T19:06:45.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.673+0000 7fbe349c5700 1 -- 172.21.15.1:0/2604584758 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe300676a0 con 0x7fbe300cc310 2024-03-18T19:06:45.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.674+0000 7fbe1effd700 1 -- 172.21.15.1:0/2604584758 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+16212 (secure 0 0 0) 0x7fbe24057ba0 con 0x7fbe300cc310 2024-03-18T19:06:45.670 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:45.670 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":117,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:06:42.901109+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:06:39.741939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-18T19:06:40.871304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:06:45.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.676+0000 7fbe349c5700 1 -- 172.21.15.1:0/2604584758 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbe1006adb0 msgr2=0x7fbe1006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:45.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.676+0000 7fbe349c5700 1 --2- 172.21.15.1:0/2604584758 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbe1006adb0 0x7fbe1006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbe20000f60 tx=0x7fbe20007c90 comp rx=0 tx=0).stop 2024-03-18T19:06:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.676+0000 7fbe349c5700 1 -- 172.21.15.1:0/2604584758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe300cc310 msgr2=0x7fbe30158a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.676+0000 7fbe349c5700 1 --2- 172.21.15.1:0/2604584758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe300cc310 0x7fbe30158a10 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400c750 tx=0x7fbe2400cb10 comp rx=0 tx=0).stop 2024-03-18T19:06:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.676+0000 7fbe349c5700 1 -- 172.21.15.1:0/2604584758 shutdown_connections 2024-03-18T19:06:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.676+0000 7fbe349c5700 1 --2- 172.21.15.1:0/2604584758 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbe1006adb0 0x7fbe1006d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.676+0000 7fbe349c5700 1 --2- 172.21.15.1:0/2604584758 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe30099a40 0x7fbe30160270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.676+0000 7fbe349c5700 1 --2- 172.21.15.1:0/2604584758 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe3015d680 0x7fbe3015a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.676+0000 7fbe349c5700 1 --2- 172.21.15.1:0/2604584758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe300cc310 0x7fbe30158a10 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.676+0000 7fbe349c5700 1 -- 172.21.15.1:0/2604584758 >> 172.21.15.1:0/2604584758 conn(0x7fbe300b2ba0 msgr2=0x7fbe300b18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.676+0000 7fbe349c5700 1 -- 172.21.15.1:0/2604584758 shutdown_connections 2024-03-18T19:06:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.676+0000 7fbe349c5700 1 -- 172.21.15.1:0/2604584758 wait complete. 2024-03-18T19:06:45.684 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T19:06:45.684 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T19:06:45.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.813+0000 7fe336962700 1 Processor -- start 2024-03-18T19:06:45.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.813+0000 7fe336962700 1 -- start start 2024-03-18T19:06:45.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.813+0000 7fe336962700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe33006ae60 0x7fe330164f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:45.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.813+0000 7fe336962700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe3301465a0 0x7fe330167480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:45.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.813+0000 7fe336962700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe330072660 con 0x7fe330146190 2024-03-18T19:06:45.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.813+0000 7fe336962700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe33006f850 con 0x7fe3301465a0 2024-03-18T19:06:45.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.813+0000 7fe336962700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe33006ebd0 con 0x7fe33006ae60 2024-03-18T19:06:45.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.814+0000 7fe334eff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe3301465a0 0x7fe330167480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:45.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.814+0000 7fe334eff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe3301465a0 0x7fe330167480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48996/0 (socket says 172.21.15.1:48996) 2024-03-18T19:06:45.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.814+0000 7fe334eff700 1 -- 172.21.15.1:0/1069270152 learned_addr learned my addr 172.21.15.1:0/1069270152 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:45.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.814+0000 7fe32f7fe700 1 --2- 172.21.15.1:0/1069270152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe33006ae60 0x7fe330164f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:45.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.814+0000 7fe334eff700 1 -- 172.21.15.1:0/1069270152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe33006ae60 msgr2=0x7fe330164f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:45.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.814+0000 7fe334eff700 1 --2- 172.21.15.1:0/1069270152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe33006ae60 0x7fe330164f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.814+0000 7fe334eff700 1 -- 172.21.15.1:0/1069270152 >> v1:172.21.15.165:6789/0 conn(0x7fe330146190 legacy=0x7fe33006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:45.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.814+0000 7fe334eff700 1 -- 172.21.15.1:0/1069270152 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33006eeb0 con 0x7fe3301465a0 2024-03-18T19:06:45.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.814+0000 7fe334eff700 1 --2- 172.21.15.1:0/1069270152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3301465a0 0x7fe330167480 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fe328009e50 tx=0x7fe32800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e911bf47741895d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:45.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.814+0000 7fe32effd700 1 -- 172.21.15.1:0/1069270152 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32800e070 con 0x7fe3301465a0 2024-03-18T19:06:45.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.814+0000 7fe32effd700 1 -- 172.21.15.1:0/1069270152 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe328003a20 con 0x7fe3301465a0 2024-03-18T19:06:45.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.814+0000 7fe32effd700 1 -- 172.21.15.1:0/1069270152 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe328003dd0 con 0x7fe3301465a0 2024-03-18T19:06:45.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.815+0000 7fe336962700 1 -- 172.21.15.1:0/1069270152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3301465a0 msgr2=0x7fe330167480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:45.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.815+0000 7fe336962700 1 --2- 172.21.15.1:0/1069270152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3301465a0 0x7fe330167480 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fe328009e50 tx=0x7fe32800b330 comp rx=0 tx=0).stop 2024-03-18T19:06:45.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.815+0000 7fe336962700 1 -- 172.21.15.1:0/1069270152 shutdown_connections 2024-03-18T19:06:45.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.815+0000 7fe336962700 1 --2- 172.21.15.1:0/1069270152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3301465a0 0x7fe330167480 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.815+0000 7fe336962700 1 --2- 172.21.15.1:0/1069270152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe33006ae60 0x7fe330164f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.815+0000 7fe336962700 1 -- 172.21.15.1:0/1069270152 >> 172.21.15.1:0/1069270152 conn(0x7fe3300b1d20 msgr2=0x7fe3300b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:45.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.815+0000 7fe336962700 1 -- 172.21.15.1:0/1069270152 shutdown_connections 2024-03-18T19:06:45.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.815+0000 7fe336962700 1 -- 172.21.15.1:0/1069270152 wait complete. 2024-03-18T19:06:45.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.816+0000 7fe336962700 1 Processor -- start 2024-03-18T19:06:45.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.816+0000 7fe336962700 1 -- start start 2024-03-18T19:06:45.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.817+0000 7fe336962700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe33006ae60 0x7fe330140970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:45.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.817+0000 7fe336962700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3301465a0 0x7fe3300cef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:45.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.817+0000 7fe336962700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe33015cd50 0x7fe3301e28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:45.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.817+0000 7fe336962700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3300cf460 con 0x7fe33015cd50 2024-03-18T19:06:45.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.817+0000 7fe336962700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe330141c80 con 0x7fe3301465a0 2024-03-18T19:06:45.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.817+0000 7fe336962700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe330142110 con 0x7fe33006ae60 2024-03-18T19:06:45.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.817+0000 7fe334eff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe33015cd50 0x7fe3301e28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:45.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.817+0000 7fe334eff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe33015cd50 0x7fe3301e28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49010/0 (socket says 172.21.15.1:49010) 2024-03-18T19:06:45.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.817+0000 7fe334eff700 1 -- 172.21.15.1:0/294243146 learned_addr learned my addr 172.21.15.1:0/294243146 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:45.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.817+0000 7fe32ffff700 1 --2- 172.21.15.1:0/294243146 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe33006ae60 0x7fe330140970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:45.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.817+0000 7fe32f7fe700 1 --2- 172.21.15.1:0/294243146 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3301465a0 0x7fe3300cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:45.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.817+0000 7fe334eff700 1 -- 172.21.15.1:0/294243146 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe33006ae60 msgr2=0x7fe330140970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:45.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.817+0000 7fe334eff700 1 --2- 172.21.15.1:0/294243146 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe33006ae60 0x7fe330140970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.817+0000 7fe334eff700 1 -- 172.21.15.1:0/294243146 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3301465a0 msgr2=0x7fe3300cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.817+0000 7fe334eff700 1 --2- 172.21.15.1:0/294243146 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3301465a0 0x7fe3300cef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.818+0000 7fe334eff700 1 -- 172.21.15.1:0/294243146 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe328009b30 con 0x7fe33015cd50 2024-03-18T19:06:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.818+0000 7fe334eff700 1 --2- 172.21.15.1:0/294243146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe33015cd50 0x7fe3301e28b0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fe328015040 tx=0x7fe328004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.818+0000 7fe32cff9700 1 -- 172.21.15.1:0/294243146 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3280094f0 con 0x7fe33015cd50 2024-03-18T19:06:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.818+0000 7fe32cff9700 1 -- 172.21.15.1:0/294243146 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe328004240 con 0x7fe33015cd50 2024-03-18T19:06:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.818+0000 7fe336962700 1 -- 172.21.15.1:0/294243146 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe330141070 con 0x7fe33015cd50 2024-03-18T19:06:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.818+0000 7fe32cff9700 1 -- 172.21.15.1:0/294243146 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3280045f0 con 0x7fe33015cd50 2024-03-18T19:06:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.818+0000 7fe336962700 1 -- 172.21.15.1:0/294243146 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe330141630 con 0x7fe33015cd50 2024-03-18T19:06:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.819+0000 7fe336962700 1 -- 172.21.15.1:0/294243146 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe33006b620 con 0x7fe33015cd50 2024-03-18T19:06:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.819+0000 7fe32cff9700 1 -- 172.21.15.1:0/294243146 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe328019070 con 0x7fe33015cd50 2024-03-18T19:06:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.820+0000 7fe32cff9700 1 --2- 172.21.15.1:0/294243146 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe31c06adb0 0x7fe31c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.820+0000 7fe32ffff700 1 --2- 172.21.15.1:0/294243146 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe31c06adb0 0x7fe31c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:45.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.820+0000 7fe32cff9700 1 -- 172.21.15.1:0/294243146 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe32808a810 con 0x7fe33015cd50 2024-03-18T19:06:45.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.820+0000 7fe32ffff700 1 --2- 172.21.15.1:0/294243146 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe31c06adb0 0x7fe31c06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe320007fd0 tx=0x7fe320007ec0 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:45.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:45.825+0000 7fe32cff9700 1 -- 172.21.15.1:0/294243146 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe328055720 con 0x7fe33015cd50 2024-03-18T19:06:46.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.026+0000 7fe336962700 1 -- 172.21.15.1:0/294243146 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe33006f470 con 0x7fe31c06adb0 2024-03-18T19:06:46.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.035+0000 7fe32cff9700 1 -- 172.21.15.1:0/294243146 <== mgr.7542 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290166 (secure 0 0 0) 0x7fe320003a90 con 0x7fe31c06adb0 2024-03-18T19:06:46.035 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:46.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.038+0000 7fe336962700 1 -- 172.21.15.1:0/294243146 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe31c06adb0 msgr2=0x7fe31c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:46.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.038+0000 7fe336962700 1 --2- 172.21.15.1:0/294243146 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe31c06adb0 0x7fe31c06d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe320007fd0 tx=0x7fe320007ec0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.038+0000 7fe336962700 1 -- 172.21.15.1:0/294243146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe33015cd50 msgr2=0x7fe3301e28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:46.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.038+0000 7fe336962700 1 --2- 172.21.15.1:0/294243146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe33015cd50 0x7fe3301e28b0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fe328015040 tx=0x7fe328004ad0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.039+0000 7fe336962700 1 -- 172.21.15.1:0/294243146 shutdown_connections 2024-03-18T19:06:46.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.039+0000 7fe336962700 1 --2- 172.21.15.1:0/294243146 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe31c06adb0 0x7fe31c06d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.039+0000 7fe336962700 1 --2- 172.21.15.1:0/294243146 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe33006ae60 0x7fe330140970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.039+0000 7fe336962700 1 --2- 172.21.15.1:0/294243146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe33015cd50 0x7fe3301e28b0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.039+0000 7fe336962700 1 --2- 172.21.15.1:0/294243146 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3301465a0 0x7fe3300cef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.039+0000 7fe336962700 1 -- 172.21.15.1:0/294243146 >> 172.21.15.1:0/294243146 conn(0x7fe3300b1d20 msgr2=0x7fe3300acab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:46.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.039+0000 7fe336962700 1 -- 172.21.15.1:0/294243146 shutdown_connections 2024-03-18T19:06:46.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.039+0000 7fe336962700 1 -- 172.21.15.1:0/294243146 wait complete. 2024-03-18T19:06:46.038 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T19:06:46.047 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-18T19:06:44.295348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":344264,"kb_used_data":12488,"kb_used_omap":12,"kb_used_meta":331635,"kb_avail":754630456,"statfs":{"total":773094113280,"available":772741586944,"internally_reserved":0,"allocated":12787712,"data_stored":6955796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":339594809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000825"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.907413+0000","last_change":"2024-03-18T19:06:41.930967+0000","last_active":"2024-03-18T19:06:42.907413+0000","last_peered":"2024-03-18T19:06:42.907413+0000","last_clean":"2024-03-18T19:06:42.907413+0000","last_became_active":"2024-03-18T19:06:41.930698+0000","last_became_peered":"2024-03-18T19:06:41.930698+0000","last_unstale":"2024-03-18T19:06:42.907413+0000","last_undegraded":"2024-03-18T19:06:42.907413+0000","last_fullsized":"2024-03-18T19:06:42.907413+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:26:06.385768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225270+0000","last_change":"2024-03-18T19:06:41.922083+0000","last_active":"2024-03-18T19:06:43.225270+0000","last_peered":"2024-03-18T19:06:43.225270+0000","last_clean":"2024-03-18T19:06:43.225270+0000","last_became_active":"2024-03-18T19:06:41.921796+0000","last_became_peered":"2024-03-18T19:06:41.921796+0000","last_unstale":"2024-03-18T19:06:43.225270+0000","last_undegraded":"2024-03-18T19:06:43.225270+0000","last_fullsized":"2024-03-18T19:06:43.225270+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:59.618124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.907005+0000","last_change":"2024-03-18T19:06:41.923251+0000","last_active":"2024-03-18T19:06:42.907005+0000","last_peered":"2024-03-18T19:06:42.907005+0000","last_clean":"2024-03-18T19:06:42.907005+0000","last_became_active":"2024-03-18T19:06:41.922974+0000","last_became_peered":"2024-03-18T19:06:41.922974+0000","last_unstale":"2024-03-18T19:06:42.907005+0000","last_undegraded":"2024-03-18T19:06:42.907005+0000","last_fullsized":"2024-03-18T19:06:42.907005+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:40.590657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905955+0000","last_change":"2024-03-18T19:06:41.919454+0000","last_active":"2024-03-18T19:06:42.905955+0000","last_peered":"2024-03-18T19:06:42.905955+0000","last_clean":"2024-03-18T19:06:42.905955+0000","last_became_active":"2024-03-18T19:06:41.919193+0000","last_became_peered":"2024-03-18T19:06:41.919193+0000","last_unstale":"2024-03-18T19:06:42.905955+0000","last_undegraded":"2024-03-18T19:06:42.905955+0000","last_fullsized":"2024-03-18T19:06:42.905955+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:39.234562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224656+0000","last_change":"2024-03-18T19:06:41.916999+0000","last_active":"2024-03-18T19:06:43.224656+0000","last_peered":"2024-03-18T19:06:43.224656+0000","last_clean":"2024-03-18T19:06:43.224656+0000","last_became_active":"2024-03-18T19:06:41.916795+0000","last_became_peered":"2024-03-18T19:06:41.916795+0000","last_unstale":"2024-03-18T19:06:43.224656+0000","last_undegraded":"2024-03-18T19:06:43.224656+0000","last_fullsized":"2024-03-18T19:06:43.224656+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:54:16.529481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223732+0000","last_change":"2024-03-18T19:06:41.934105+0000","last_active":"2024-03-18T19:06:43.223732+0000","last_peered":"2024-03-18T19:06:43.223732+0000","last_clean":"2024-03-18T19:06:43.223732+0000","last_became_active":"2024-03-18T19:06:41.933704+0000","last_became_peered":"2024-03-18T19:06:41.933704+0000","last_unstale":"2024-03-18T19:06:43.223732+0000","last_undegraded":"2024-03-18T19:06:43.223732+0000","last_fullsized":"2024-03-18T19:06:43.223732+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:58:15.616500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105595+0000","last_change":"2024-03-18T19:06:41.929703+0000","last_active":"2024-03-18T19:06:43.105595+0000","last_peered":"2024-03-18T19:06:43.105595+0000","last_clean":"2024-03-18T19:06:43.105595+0000","last_became_active":"2024-03-18T19:06:41.929476+0000","last_became_peered":"2024-03-18T19:06:41.929476+0000","last_unstale":"2024-03-18T19:06:43.105595+0000","last_undegraded":"2024-03-18T19:06:43.105595+0000","last_fullsized":"2024-03-18T19:06:43.105595+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:00:03.205630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.904851+0000","last_change":"2024-03-18T19:06:41.916504+0000","last_active":"2024-03-18T19:06:42.904851+0000","last_peered":"2024-03-18T19:06:42.904851+0000","last_clean":"2024-03-18T19:06:42.904851+0000","last_became_active":"2024-03-18T19:06:41.916197+0000","last_became_peered":"2024-03-18T19:06:41.916197+0000","last_unstale":"2024-03-18T19:06:42.904851+0000","last_undegraded":"2024-03-18T19:06:42.904851+0000","last_fullsized":"2024-03-18T19:06:42.904851+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:35.228931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225029+0000","last_change":"2024-03-18T19:06:41.918686+0000","last_active":"2024-03-18T19:06:43.225029+0000","last_peered":"2024-03-18T19:06:43.225029+0000","last_clean":"2024-03-18T19:06:43.225029+0000","last_became_active":"2024-03-18T19:06:41.917900+0000","last_became_peered":"2024-03-18T19:06:41.917900+0000","last_unstale":"2024-03-18T19:06:43.225029+0000","last_undegraded":"2024-03-18T19:06:43.225029+0000","last_fullsized":"2024-03-18T19:06:43.225029+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:03:12.024061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224018+0000","last_change":"2024-03-18T19:06:41.924735+0000","last_active":"2024-03-18T19:06:43.224018+0000","last_peered":"2024-03-18T19:06:43.224018+0000","last_clean":"2024-03-18T19:06:43.224018+0000","last_became_active":"2024-03-18T19:06:41.924491+0000","last_became_peered":"2024-03-18T19:06:41.924491+0000","last_unstale":"2024-03-18T19:06:43.224018+0000","last_undegraded":"2024-03-18T19:06:43.224018+0000","last_fullsized":"2024-03-18T19:06:43.224018+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:52.692533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225628+0000","last_change":"2024-03-18T19:06:41.931128+0000","last_active":"2024-03-18T19:06:43.225628+0000","last_peered":"2024-03-18T19:06:43.225628+0000","last_clean":"2024-03-18T19:06:43.225628+0000","last_became_active":"2024-03-18T19:06:41.930870+0000","last_became_peered":"2024-03-18T19:06:41.930870+0000","last_unstale":"2024-03-18T19:06:43.225628+0000","last_undegraded":"2024-03-18T19:06:43.225628+0000","last_fullsized":"2024-03-18T19:06:43.225628+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:43:59.126342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223680+0000","last_change":"2024-03-18T19:06:41.927393+0000","last_active":"2024-03-18T19:06:43.223680+0000","last_peered":"2024-03-18T19:06:43.223680+0000","last_clean":"2024-03-18T19:06:43.223680+0000","last_became_active":"2024-03-18T19:06:41.927127+0000","last_became_peered":"2024-03-18T19:06:41.927127+0000","last_unstale":"2024-03-18T19:06:43.223680+0000","last_undegraded":"2024-03-18T19:06:43.223680+0000","last_fullsized":"2024-03-18T19:06:43.223680+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:42.842003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905262+0000","last_change":"2024-03-18T19:06:41.916683+0000","last_active":"2024-03-18T19:06:42.905262+0000","last_peered":"2024-03-18T19:06:42.905262+0000","last_clean":"2024-03-18T19:06:42.905262+0000","last_became_active":"2024-03-18T19:06:41.916410+0000","last_became_peered":"2024-03-18T19:06:41.916410+0000","last_unstale":"2024-03-18T19:06:42.905262+0000","last_undegraded":"2024-03-18T19:06:42.905262+0000","last_fullsized":"2024-03-18T19:06:42.905262+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:08.026486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224248+0000","last_change":"2024-03-18T19:06:41.925500+0000","last_active":"2024-03-18T19:06:43.224248+0000","last_peered":"2024-03-18T19:06:43.224248+0000","last_clean":"2024-03-18T19:06:43.224248+0000","last_became_active":"2024-03-18T19:06:41.924680+0000","last_became_peered":"2024-03-18T19:06:41.924680+0000","last_unstale":"2024-03-18T19:06:43.224248+0000","last_undegraded":"2024-03-18T19:06:43.224248+0000","last_fullsized":"2024-03-18T19:06:43.224248+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:20.035954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905061+0000","last_change":"2024-03-18T19:06:41.932201+0000","last_active":"2024-03-18T19:06:42.905061+0000","last_peered":"2024-03-18T19:06:42.905061+0000","last_clean":"2024-03-18T19:06:42.905061+0000","last_became_active":"2024-03-18T19:06:41.931948+0000","last_became_peered":"2024-03-18T19:06:41.931948+0000","last_unstale":"2024-03-18T19:06:42.905061+0000","last_undegraded":"2024-03-18T19:06:42.905061+0000","last_fullsized":"2024-03-18T19:06:42.905061+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:15.089015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224052+0000","last_change":"2024-03-18T19:06:41.930105+0000","last_active":"2024-03-18T19:06:43.224052+0000","last_peered":"2024-03-18T19:06:43.224052+0000","last_clean":"2024-03-18T19:06:43.224052+0000","last_became_active":"2024-03-18T19:06:41.929840+0000","last_became_peered":"2024-03-18T19:06:41.929840+0000","last_unstale":"2024-03-18T19:06:43.224052+0000","last_undegraded":"2024-03-18T19:06:43.224052+0000","last_fullsized":"2024-03-18T19:06:43.224052+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:50.694886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224144+0000","last_change":"2024-03-18T19:06:41.930527+0000","last_active":"2024-03-18T19:06:43.224144+0000","last_peered":"2024-03-18T19:06:43.224144+0000","last_clean":"2024-03-18T19:06:43.224144+0000","last_became_active":"2024-03-18T19:06:41.929968+0000","last_became_peered":"2024-03-18T19:06:41.929968+0000","last_unstale":"2024-03-18T19:06:43.224144+0000","last_undegraded":"2024-03-18T19:06:43.224144+0000","last_fullsized":"2024-03-18T19:06:43.224144+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:13.861260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.906947+0000","last_change":"2024-03-18T19:06:41.922331+0000","last_active":"2024-03-18T19:06:42.906947+0000","last_peered":"2024-03-18T19:06:42.906947+0000","last_clean":"2024-03-18T19:06:42.906947+0000","last_became_active":"2024-03-18T19:06:41.920647+0000","last_became_peered":"2024-03-18T19:06:41.920647+0000","last_unstale":"2024-03-18T19:06:42.906947+0000","last_undegraded":"2024-03-18T19:06:42.906947+0000","last_fullsized":"2024-03-18T19:06:42.906947+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:15.655985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225010+0000","last_change":"2024-03-18T19:06:41.924022+0000","last_active":"2024-03-18T19:06:43.225010+0000","last_peered":"2024-03-18T19:06:43.225010+0000","last_clean":"2024-03-18T19:06:43.225010+0000","last_became_active":"2024-03-18T19:06:41.923770+0000","last_became_peered":"2024-03-18T19:06:41.923770+0000","last_unstale":"2024-03-18T19:06:43.225010+0000","last_undegraded":"2024-03-18T19:06:43.225010+0000","last_fullsized":"2024-03-18T19:06:43.225010+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:25.618183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905694+0000","last_change":"2024-03-18T19:06:41.930985+0000","last_active":"2024-03-18T19:06:42.905694+0000","last_peered":"2024-03-18T19:06:42.905694+0000","last_clean":"2024-03-18T19:06:42.905694+0000","last_became_active":"2024-03-18T19:06:41.927789+0000","last_became_peered":"2024-03-18T19:06:41.927789+0000","last_unstale":"2024-03-18T19:06:42.905694+0000","last_undegraded":"2024-03-18T19:06:42.905694+0000","last_fullsized":"2024-03-18T19:06:42.905694+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:00:03.205630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224745+0000","last_change":"2024-03-18T19:06:41.921090+0000","last_active":"2024-03-18T19:06:43.224745+0000","last_peered":"2024-03-18T19:06:43.224745+0000","last_clean":"2024-03-18T19:06:43.224745+0000","last_became_active":"2024-03-18T19:06:41.920828+0000","last_became_peered":"2024-03-18T19:06:41.920828+0000","last_unstale":"2024-03-18T19:06:43.224745+0000","last_undegraded":"2024-03-18T19:06:43.224745+0000","last_fullsized":"2024-03-18T19:06:43.224745+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:59.618124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224259+0000","last_change":"2024-03-18T19:06:41.924664+0000","last_active":"2024-03-18T19:06:43.224259+0000","last_peered":"2024-03-18T19:06:43.224259+0000","last_clean":"2024-03-18T19:06:43.224259+0000","last_became_active":"2024-03-18T19:06:41.924341+0000","last_became_peered":"2024-03-18T19:06:41.924341+0000","last_unstale":"2024-03-18T19:06:43.224259+0000","last_undegraded":"2024-03-18T19:06:43.224259+0000","last_fullsized":"2024-03-18T19:06:43.224259+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:00:03.205630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225066+0000","last_change":"2024-03-18T19:06:41.919722+0000","last_active":"2024-03-18T19:06:43.225066+0000","last_peered":"2024-03-18T19:06:43.225066+0000","last_clean":"2024-03-18T19:06:43.225066+0000","last_became_active":"2024-03-18T19:06:41.919362+0000","last_became_peered":"2024-03-18T19:06:41.919362+0000","last_unstale":"2024-03-18T19:06:43.225066+0000","last_undegraded":"2024-03-18T19:06:43.225066+0000","last_fullsized":"2024-03-18T19:06:43.225066+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:11:40.736064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.906806+0000","last_change":"2024-03-18T19:06:41.923714+0000","last_active":"2024-03-18T19:06:42.906806+0000","last_peered":"2024-03-18T19:06:42.906806+0000","last_clean":"2024-03-18T19:06:42.906806+0000","last_became_active":"2024-03-18T19:06:41.923245+0000","last_became_peered":"2024-03-18T19:06:41.923245+0000","last_unstale":"2024-03-18T19:06:42.906806+0000","last_undegraded":"2024-03-18T19:06:42.906806+0000","last_fullsized":"2024-03-18T19:06:42.906806+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:04:37.360539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225614+0000","last_change":"2024-03-18T19:06:41.917276+0000","last_active":"2024-03-18T19:06:43.225614+0000","last_peered":"2024-03-18T19:06:43.225614+0000","last_clean":"2024-03-18T19:06:43.225614+0000","last_became_active":"2024-03-18T19:06:41.916964+0000","last_became_peered":"2024-03-18T19:06:41.916964+0000","last_unstale":"2024-03-18T19:06:43.225614+0000","last_undegraded":"2024-03-18T19:06:43.225614+0000","last_fullsized":"2024-03-18T19:06:43.225614+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:32:37.339641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225293+0000","last_change":"2024-03-18T19:06:41.925228+0000","last_active":"2024-03-18T19:06:43.225293+0000","last_peered":"2024-03-18T19:06:43.225293+0000","last_clean":"2024-03-18T19:06:43.225293+0000","last_became_active":"2024-03-18T19:06:41.924948+0000","last_became_peered":"2024-03-18T19:06:41.924948+0000","last_unstale":"2024-03-18T19:06:43.225293+0000","last_undegraded":"2024-03-18T19:06:43.225293+0000","last_fullsized":"2024-03-18T19:06:43.225293+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:42.413430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223916+0000","last_change":"2024-03-18T19:06:41.919765+0000","last_active":"2024-03-18T19:06:43.223916+0000","last_peered":"2024-03-18T19:06:43.223916+0000","last_clean":"2024-03-18T19:06:43.223916+0000","last_became_active":"2024-03-18T19:06:41.919424+0000","last_became_peered":"2024-03-18T19:06:41.919424+0000","last_unstale":"2024-03-18T19:06:43.223916+0000","last_undegraded":"2024-03-18T19:06:43.223916+0000","last_fullsized":"2024-03-18T19:06:43.223916+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:12:06.889612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905793+0000","last_change":"2024-03-18T19:06:41.923195+0000","last_active":"2024-03-18T19:06:42.905793+0000","last_peered":"2024-03-18T19:06:42.905793+0000","last_clean":"2024-03-18T19:06:42.905793+0000","last_became_active":"2024-03-18T19:06:41.922939+0000","last_became_peered":"2024-03-18T19:06:41.922939+0000","last_unstale":"2024-03-18T19:06:42.905793+0000","last_undegraded":"2024-03-18T19:06:42.905793+0000","last_fullsized":"2024-03-18T19:06:42.905793+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:42.413430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105540+0000","last_change":"2024-03-18T19:06:41.914798+0000","last_active":"2024-03-18T19:06:43.105540+0000","last_peered":"2024-03-18T19:06:43.105540+0000","last_clean":"2024-03-18T19:06:43.105540+0000","last_became_active":"2024-03-18T19:06:41.914524+0000","last_became_peered":"2024-03-18T19:06:41.914524+0000","last_unstale":"2024-03-18T19:06:43.105540+0000","last_undegraded":"2024-03-18T19:06:43.105540+0000","last_fullsized":"2024-03-18T19:06:43.105540+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:20.035954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224360+0000","last_change":"2024-03-18T19:06:41.914890+0000","last_active":"2024-03-18T19:06:43.224360+0000","last_peered":"2024-03-18T19:06:43.224360+0000","last_clean":"2024-03-18T19:06:43.224360+0000","last_became_active":"2024-03-18T19:06:41.914687+0000","last_became_peered":"2024-03-18T19:06:41.914687+0000","last_unstale":"2024-03-18T19:06:43.224360+0000","last_undegraded":"2024-03-18T19:06:43.224360+0000","last_fullsized":"2024-03-18T19:06:43.224360+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:15.089015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224321+0000","last_change":"2024-03-18T19:06:41.923914+0000","last_active":"2024-03-18T19:06:43.224321+0000","last_peered":"2024-03-18T19:06:43.224321+0000","last_clean":"2024-03-18T19:06:43.224321+0000","last_became_active":"2024-03-18T19:06:41.923644+0000","last_became_peered":"2024-03-18T19:06:41.923644+0000","last_unstale":"2024-03-18T19:06:43.224321+0000","last_undegraded":"2024-03-18T19:06:43.224321+0000","last_fullsized":"2024-03-18T19:06:43.224321+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:54:16.529481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224151+0000","last_change":"2024-03-18T19:06:41.929950+0000","last_active":"2024-03-18T19:06:43.224151+0000","last_peered":"2024-03-18T19:06:43.224151+0000","last_clean":"2024-03-18T19:06:43.224151+0000","last_became_active":"2024-03-18T19:06:41.929716+0000","last_became_peered":"2024-03-18T19:06:41.929716+0000","last_unstale":"2024-03-18T19:06:43.224151+0000","last_undegraded":"2024-03-18T19:06:43.224151+0000","last_fullsized":"2024-03-18T19:06:43.224151+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:24.975965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224350+0000","last_change":"2024-03-18T19:06:41.925883+0000","last_active":"2024-03-18T19:06:43.224350+0000","last_peered":"2024-03-18T19:06:43.224350+0000","last_clean":"2024-03-18T19:06:43.224350+0000","last_became_active":"2024-03-18T19:06:41.925612+0000","last_became_peered":"2024-03-18T19:06:41.925612+0000","last_unstale":"2024-03-18T19:06:43.224350+0000","last_undegraded":"2024-03-18T19:06:43.224350+0000","last_fullsized":"2024-03-18T19:06:43.224350+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:02:02.068675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"116'4","reported_seq":15,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225518+0000","last_change":"2024-03-18T19:06:41.493796+0000","last_active":"2024-03-18T19:06:43.225518+0000","last_peered":"2024-03-18T19:06:43.225518+0000","last_clean":"2024-03-18T19:06:43.225518+0000","last_became_active":"2024-03-18T19:06:41.493520+0000","last_became_peered":"2024-03-18T19:06:41.493520+0000","last_unstale":"2024-03-18T19:06:43.225518+0000","last_undegraded":"2024-03-18T19:06:43.225518+0000","last_fullsized":"2024-03-18T19:06:43.225518+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:33.022886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224437+0000","last_change":"2024-03-18T19:06:41.925698+0000","last_active":"2024-03-18T19:06:43.224437+0000","last_peered":"2024-03-18T19:06:43.224437+0000","last_clean":"2024-03-18T19:06:43.224437+0000","last_became_active":"2024-03-18T19:06:41.925244+0000","last_became_peered":"2024-03-18T19:06:41.925244+0000","last_unstale":"2024-03-18T19:06:43.224437+0000","last_undegraded":"2024-03-18T19:06:43.224437+0000","last_fullsized":"2024-03-18T19:06:43.224437+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:41.487390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"116'1","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225385+0000","last_change":"2024-03-18T19:06:41.482920+0000","last_active":"2024-03-18T19:06:43.225385+0000","last_peered":"2024-03-18T19:06:43.225385+0000","last_clean":"2024-03-18T19:06:43.225385+0000","last_became_active":"2024-03-18T19:06:41.482612+0000","last_became_peered":"2024-03-18T19:06:41.482612+0000","last_unstale":"2024-03-18T19:06:43.225385+0000","last_undegraded":"2024-03-18T19:06:43.225385+0000","last_fullsized":"2024-03-18T19:06:43.225385+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:25.756251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105617+0000","last_change":"2024-03-18T19:06:41.920806+0000","last_active":"2024-03-18T19:06:43.105617+0000","last_peered":"2024-03-18T19:06:43.105617+0000","last_clean":"2024-03-18T19:06:43.105617+0000","last_became_active":"2024-03-18T19:06:41.920541+0000","last_became_peered":"2024-03-18T19:06:41.920541+0000","last_unstale":"2024-03-18T19:06:43.105617+0000","last_undegraded":"2024-03-18T19:06:43.105617+0000","last_fullsized":"2024-03-18T19:06:43.105617+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:42.413430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"116'2","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105739+0000","last_change":"2024-03-18T19:06:41.487514+0000","last_active":"2024-03-18T19:06:43.105739+0000","last_peered":"2024-03-18T19:06:43.105739+0000","last_clean":"2024-03-18T19:06:43.105739+0000","last_became_active":"2024-03-18T19:06:41.487253+0000","last_became_peered":"2024-03-18T19:06:41.487253+0000","last_unstale":"2024-03-18T19:06:43.105739+0000","last_undegraded":"2024-03-18T19:06:43.105739+0000","last_fullsized":"2024-03-18T19:06:43.105739+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:10.908853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224243+0000","last_change":"2024-03-18T19:06:41.930183+0000","last_active":"2024-03-18T19:06:43.224243+0000","last_peered":"2024-03-18T19:06:43.224243+0000","last_clean":"2024-03-18T19:06:43.224243+0000","last_became_active":"2024-03-18T19:06:41.929863+0000","last_became_peered":"2024-03-18T19:06:41.929863+0000","last_unstale":"2024-03-18T19:06:43.224243+0000","last_undegraded":"2024-03-18T19:06:43.224243+0000","last_fullsized":"2024-03-18T19:06:43.224243+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:25:37.918166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"116'2","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225278+0000","last_change":"2024-03-18T19:06:41.486199+0000","last_active":"2024-03-18T19:06:43.225278+0000","last_peered":"2024-03-18T19:06:43.225278+0000","last_clean":"2024-03-18T19:06:43.225278+0000","last_became_active":"2024-03-18T19:06:41.485977+0000","last_became_peered":"2024-03-18T19:06:41.485977+0000","last_unstale":"2024-03-18T19:06:43.225278+0000","last_undegraded":"2024-03-18T19:06:43.225278+0000","last_fullsized":"2024-03-18T19:06:43.225278+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:46.041073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.904938+0000","last_change":"2024-03-18T19:06:41.918905+0000","last_active":"2024-03-18T19:06:42.904938+0000","last_peered":"2024-03-18T19:06:42.904938+0000","last_clean":"2024-03-18T19:06:42.904938+0000","last_became_active":"2024-03-18T19:06:41.918656+0000","last_became_peered":"2024-03-18T19:06:41.918656+0000","last_unstale":"2024-03-18T19:06:42.904938+0000","last_undegraded":"2024-03-18T19:06:42.904938+0000","last_fullsized":"2024-03-18T19:06:42.904938+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:02.450766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"116'1","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224762+0000","last_change":"2024-03-18T19:06:41.483351+0000","last_active":"2024-03-18T19:06:43.224762+0000","last_peered":"2024-03-18T19:06:43.224762+0000","last_clean":"2024-03-18T19:06:43.224762+0000","last_became_active":"2024-03-18T19:06:41.483076+0000","last_became_peered":"2024-03-18T19:06:41.483076+0000","last_unstale":"2024-03-18T19:06:43.224762+0000","last_undegraded":"2024-03-18T19:06:43.224762+0000","last_fullsized":"2024-03-18T19:06:43.224762+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:49:34.183987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225177+0000","last_change":"2024-03-18T19:06:41.920874+0000","last_active":"2024-03-18T19:06:43.225177+0000","last_peered":"2024-03-18T19:06:43.225177+0000","last_clean":"2024-03-18T19:06:43.225177+0000","last_became_active":"2024-03-18T19:06:41.919913+0000","last_became_peered":"2024-03-18T19:06:41.919913+0000","last_unstale":"2024-03-18T19:06:43.225177+0000","last_undegraded":"2024-03-18T19:06:43.225177+0000","last_fullsized":"2024-03-18T19:06:43.225177+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:11.914183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"116'1","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905769+0000","last_change":"2024-03-18T19:06:41.485060+0000","last_active":"2024-03-18T19:06:42.905769+0000","last_peered":"2024-03-18T19:06:42.905769+0000","last_clean":"2024-03-18T19:06:42.905769+0000","last_became_active":"2024-03-18T19:06:41.484778+0000","last_became_peered":"2024-03-18T19:06:41.484778+0000","last_unstale":"2024-03-18T19:06:42.905769+0000","last_undegraded":"2024-03-18T19:06:42.905769+0000","last_fullsized":"2024-03-18T19:06:42.905769+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:24.017447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223815+0000","last_change":"2024-03-18T19:06:41.933950+0000","last_active":"2024-03-18T19:06:43.223815+0000","last_peered":"2024-03-18T19:06:43.223815+0000","last_clean":"2024-03-18T19:06:43.223815+0000","last_became_active":"2024-03-18T19:06:41.933411+0000","last_became_peered":"2024-03-18T19:06:41.933411+0000","last_unstale":"2024-03-18T19:06:43.223815+0000","last_undegraded":"2024-03-18T19:06:43.223815+0000","last_fullsized":"2024-03-18T19:06:43.223815+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:18:14.637126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.906967+0000","last_change":"2024-03-18T19:06:41.482438+0000","last_active":"2024-03-18T19:06:42.906967+0000","last_peered":"2024-03-18T19:06:42.906967+0000","last_clean":"2024-03-18T19:06:42.906967+0000","last_became_active":"2024-03-18T19:06:41.482139+0000","last_became_peered":"2024-03-18T19:06:41.482139+0000","last_unstale":"2024-03-18T19:06:42.906967+0000","last_undegraded":"2024-03-18T19:06:42.906967+0000","last_fullsized":"2024-03-18T19:06:42.906967+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:15:01.147997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905358+0000","last_change":"2024-03-18T19:06:41.918370+0000","last_active":"2024-03-18T19:06:42.905358+0000","last_peered":"2024-03-18T19:06:42.905358+0000","last_clean":"2024-03-18T19:06:42.905358+0000","last_became_active":"2024-03-18T19:06:41.917142+0000","last_became_peered":"2024-03-18T19:06:41.917142+0000","last_unstale":"2024-03-18T19:06:42.905358+0000","last_undegraded":"2024-03-18T19:06:42.905358+0000","last_fullsized":"2024-03-18T19:06:42.905358+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:59.618124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105424+0000","last_change":"2024-03-18T19:06:41.494458+0000","last_active":"2024-03-18T19:06:43.105424+0000","last_peered":"2024-03-18T19:06:43.105424+0000","last_clean":"2024-03-18T19:06:43.105424+0000","last_became_active":"2024-03-18T19:06:41.494280+0000","last_became_peered":"2024-03-18T19:06:41.494280+0000","last_unstale":"2024-03-18T19:06:43.105424+0000","last_undegraded":"2024-03-18T19:06:43.105424+0000","last_fullsized":"2024-03-18T19:06:43.105424+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:32:36.334311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225865+0000","last_change":"2024-03-18T19:06:41.927370+0000","last_active":"2024-03-18T19:06:43.225865+0000","last_peered":"2024-03-18T19:06:43.225865+0000","last_clean":"2024-03-18T19:06:43.225865+0000","last_became_active":"2024-03-18T19:06:41.927131+0000","last_became_peered":"2024-03-18T19:06:41.927131+0000","last_unstale":"2024-03-18T19:06:43.225865+0000","last_undegraded":"2024-03-18T19:06:43.225865+0000","last_fullsized":"2024-03-18T19:06:43.225865+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:39.234562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225752+0000","last_change":"2024-03-18T19:06:41.490868+0000","last_active":"2024-03-18T19:06:43.225752+0000","last_peered":"2024-03-18T19:06:43.225752+0000","last_clean":"2024-03-18T19:06:43.225752+0000","last_became_active":"2024-03-18T19:06:41.490615+0000","last_became_peered":"2024-03-18T19:06:41.490615+0000","last_unstale":"2024-03-18T19:06:43.225752+0000","last_undegraded":"2024-03-18T19:06:43.225752+0000","last_fullsized":"2024-03-18T19:06:43.225752+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:25.756251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105719+0000","last_change":"2024-03-18T19:06:41.921302+0000","last_active":"2024-03-18T19:06:43.105719+0000","last_peered":"2024-03-18T19:06:43.105719+0000","last_clean":"2024-03-18T19:06:43.105719+0000","last_became_active":"2024-03-18T19:06:41.921014+0000","last_became_peered":"2024-03-18T19:06:41.921014+0000","last_unstale":"2024-03-18T19:06:43.105719+0000","last_undegraded":"2024-03-18T19:06:43.105719+0000","last_fullsized":"2024-03-18T19:06:43.105719+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:15.089015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.907188+0000","last_change":"2024-03-18T19:06:41.485189+0000","last_active":"2024-03-18T19:06:42.907188+0000","last_peered":"2024-03-18T19:06:42.907188+0000","last_clean":"2024-03-18T19:06:42.907188+0000","last_became_active":"2024-03-18T19:06:41.484929+0000","last_became_peered":"2024-03-18T19:06:41.484929+0000","last_unstale":"2024-03-18T19:06:42.907188+0000","last_undegraded":"2024-03-18T19:06:42.907188+0000","last_fullsized":"2024-03-18T19:06:42.907188+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:20:43.467362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.907046+0000","last_change":"2024-03-18T19:06:41.921920+0000","last_active":"2024-03-18T19:06:42.907046+0000","last_peered":"2024-03-18T19:06:42.907046+0000","last_clean":"2024-03-18T19:06:42.907046+0000","last_became_active":"2024-03-18T19:06:41.921680+0000","last_became_peered":"2024-03-18T19:06:41.921680+0000","last_unstale":"2024-03-18T19:06:42.907046+0000","last_undegraded":"2024-03-18T19:06:42.907046+0000","last_fullsized":"2024-03-18T19:06:42.907046+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:10:31.125315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"116'3","reported_seq":14,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225102+0000","last_change":"2024-03-18T19:06:41.493620+0000","last_active":"2024-03-18T19:06:43.225102+0000","last_peered":"2024-03-18T19:06:43.225102+0000","last_clean":"2024-03-18T19:06:43.225102+0000","last_became_active":"2024-03-18T19:06:41.493271+0000","last_became_peered":"2024-03-18T19:06:41.493271+0000","last_unstale":"2024-03-18T19:06:43.225102+0000","last_undegraded":"2024-03-18T19:06:43.225102+0000","last_fullsized":"2024-03-18T19:06:43.225102+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:05.738508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905389+0000","last_change":"2024-03-18T19:06:41.918139+0000","last_active":"2024-03-18T19:06:42.905389+0000","last_peered":"2024-03-18T19:06:42.905389+0000","last_clean":"2024-03-18T19:06:42.905389+0000","last_became_active":"2024-03-18T19:06:41.917151+0000","last_became_peered":"2024-03-18T19:06:41.917151+0000","last_unstale":"2024-03-18T19:06:42.905389+0000","last_undegraded":"2024-03-18T19:06:42.905389+0000","last_fullsized":"2024-03-18T19:06:42.905389+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:20.035954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"116'6","reported_seq":17,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224394+0000","last_change":"2024-03-18T19:06:41.492273+0000","last_active":"2024-03-18T19:06:43.224394+0000","last_peered":"2024-03-18T19:06:43.224394+0000","last_clean":"2024-03-18T19:06:43.224394+0000","last_became_active":"2024-03-18T19:06:41.492024+0000","last_became_peered":"2024-03-18T19:06:41.492024+0000","last_unstale":"2024-03-18T19:06:43.224394+0000","last_undegraded":"2024-03-18T19:06:43.224394+0000","last_fullsized":"2024-03-18T19:06:43.224394+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:36:10.016972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905031+0000","last_change":"2024-03-18T19:06:41.916297+0000","last_active":"2024-03-18T19:06:42.905031+0000","last_peered":"2024-03-18T19:06:42.905031+0000","last_clean":"2024-03-18T19:06:42.905031+0000","last_became_active":"2024-03-18T19:06:41.916044+0000","last_became_peered":"2024-03-18T19:06:41.916044+0000","last_unstale":"2024-03-18T19:06:42.905031+0000","last_undegraded":"2024-03-18T19:06:42.905031+0000","last_fullsized":"2024-03-18T19:06:42.905031+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:54:16.529481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"116'2","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105804+0000","last_change":"2024-03-18T19:06:41.494026+0000","last_active":"2024-03-18T19:06:43.105804+0000","last_peered":"2024-03-18T19:06:43.105804+0000","last_clean":"2024-03-18T19:06:43.105804+0000","last_became_active":"2024-03-18T19:06:41.493749+0000","last_became_peered":"2024-03-18T19:06:41.493749+0000","last_unstale":"2024-03-18T19:06:43.105804+0000","last_undegraded":"2024-03-18T19:06:43.105804+0000","last_fullsized":"2024-03-18T19:06:43.105804+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:22.727166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224167+0000","last_change":"2024-03-18T19:06:41.917535+0000","last_active":"2024-03-18T19:06:43.224167+0000","last_peered":"2024-03-18T19:06:43.224167+0000","last_clean":"2024-03-18T19:06:43.224167+0000","last_became_active":"2024-03-18T19:06:41.917216+0000","last_became_peered":"2024-03-18T19:06:41.917216+0000","last_unstale":"2024-03-18T19:06:43.224167+0000","last_undegraded":"2024-03-18T19:06:43.224167+0000","last_fullsized":"2024-03-18T19:06:43.224167+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:33:22.354647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223830+0000","last_change":"2024-03-18T19:06:41.491449+0000","last_active":"2024-03-18T19:06:43.223830+0000","last_peered":"2024-03-18T19:06:43.223830+0000","last_clean":"2024-03-18T19:06:43.223830+0000","last_became_active":"2024-03-18T19:06:41.490953+0000","last_became_peered":"2024-03-18T19:06:41.490953+0000","last_unstale":"2024-03-18T19:06:43.223830+0000","last_undegraded":"2024-03-18T19:06:43.223830+0000","last_fullsized":"2024-03-18T19:06:43.223830+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:51:06.462173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223673+0000","last_change":"2024-03-18T19:06:41.924070+0000","last_active":"2024-03-18T19:06:43.223673+0000","last_peered":"2024-03-18T19:06:43.223673+0000","last_clean":"2024-03-18T19:06:43.223673+0000","last_became_active":"2024-03-18T19:06:41.923787+0000","last_became_peered":"2024-03-18T19:06:41.923787+0000","last_unstale":"2024-03-18T19:06:43.223673+0000","last_undegraded":"2024-03-18T19:06:43.223673+0000","last_fullsized":"2024-03-18T19:06:43.223673+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:19.898810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905160+0000","last_change":"2024-03-18T19:06:41.487812+0000","last_active":"2024-03-18T19:06:42.905160+0000","last_peered":"2024-03-18T19:06:42.905160+0000","last_clean":"2024-03-18T19:06:42.905160+0000","last_became_active":"2024-03-18T19:06:41.485619+0000","last_became_peered":"2024-03-18T19:06:41.485619+0000","last_unstale":"2024-03-18T19:06:42.905160+0000","last_undegraded":"2024-03-18T19:06:42.905160+0000","last_fullsized":"2024-03-18T19:06:42.905160+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:05.738508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224449+0000","last_change":"2024-03-18T19:06:41.919157+0000","last_active":"2024-03-18T19:06:43.224449+0000","last_peered":"2024-03-18T19:06:43.224449+0000","last_clean":"2024-03-18T19:06:43.224449+0000","last_became_active":"2024-03-18T19:06:41.918916+0000","last_became_peered":"2024-03-18T19:06:41.918916+0000","last_unstale":"2024-03-18T19:06:43.224449+0000","last_undegraded":"2024-03-18T19:06:43.224449+0000","last_fullsized":"2024-03-18T19:06:43.224449+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:42.679949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"116'8","reported_seq":19,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905036+0000","last_change":"2024-03-18T19:06:41.488447+0000","last_active":"2024-03-18T19:06:42.905036+0000","last_peered":"2024-03-18T19:06:42.905036+0000","last_clean":"2024-03-18T19:06:42.905036+0000","last_became_active":"2024-03-18T19:06:41.485582+0000","last_became_peered":"2024-03-18T19:06:41.485582+0000","last_unstale":"2024-03-18T19:06:42.905036+0000","last_undegraded":"2024-03-18T19:06:42.905036+0000","last_fullsized":"2024-03-18T19:06:42.905036+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:25.756251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224301+0000","last_change":"2024-03-18T19:06:41.929574+0000","last_active":"2024-03-18T19:06:43.224301+0000","last_peered":"2024-03-18T19:06:43.224301+0000","last_clean":"2024-03-18T19:06:43.224301+0000","last_became_active":"2024-03-18T19:06:41.929297+0000","last_became_peered":"2024-03-18T19:06:41.929297+0000","last_unstale":"2024-03-18T19:06:43.224301+0000","last_undegraded":"2024-03-18T19:06:43.224301+0000","last_fullsized":"2024-03-18T19:06:43.224301+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:46:28.072024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"116'1","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225616+0000","last_change":"2024-03-18T19:06:41.475897+0000","last_active":"2024-03-18T19:06:43.225616+0000","last_peered":"2024-03-18T19:06:43.225616+0000","last_clean":"2024-03-18T19:06:43.225616+0000","last_became_active":"2024-03-18T19:06:41.475629+0000","last_became_peered":"2024-03-18T19:06:41.475629+0000","last_unstale":"2024-03-18T19:06:43.225616+0000","last_undegraded":"2024-03-18T19:06:43.225616+0000","last_fullsized":"2024-03-18T19:06:43.225616+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:47:27.561126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225436+0000","last_change":"2024-03-18T19:06:41.915032+0000","last_active":"2024-03-18T19:06:43.225436+0000","last_peered":"2024-03-18T19:06:43.225436+0000","last_clean":"2024-03-18T19:06:43.225436+0000","last_became_active":"2024-03-18T19:06:41.914759+0000","last_became_peered":"2024-03-18T19:06:41.914759+0000","last_unstale":"2024-03-18T19:06:43.225436+0000","last_undegraded":"2024-03-18T19:06:43.225436+0000","last_fullsized":"2024-03-18T19:06:43.225436+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:17.972217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"116'2","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105743+0000","last_change":"2024-03-18T19:06:41.481574+0000","last_active":"2024-03-18T19:06:43.105743+0000","last_peered":"2024-03-18T19:06:43.105743+0000","last_clean":"2024-03-18T19:06:43.105743+0000","last_became_active":"2024-03-18T19:06:41.481320+0000","last_became_peered":"2024-03-18T19:06:41.481320+0000","last_unstale":"2024-03-18T19:06:43.105743+0000","last_undegraded":"2024-03-18T19:06:43.105743+0000","last_fullsized":"2024-03-18T19:06:43.105743+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:11:39.730734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225186+0000","last_change":"2024-03-18T19:06:41.918403+0000","last_active":"2024-03-18T19:06:43.225186+0000","last_peered":"2024-03-18T19:06:43.225186+0000","last_clean":"2024-03-18T19:06:43.225186+0000","last_became_active":"2024-03-18T19:06:41.918140+0000","last_became_peered":"2024-03-18T19:06:41.918140+0000","last_unstale":"2024-03-18T19:06:43.225186+0000","last_undegraded":"2024-03-18T19:06:43.225186+0000","last_fullsized":"2024-03-18T19:06:43.225186+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:44.947038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"116'1","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224901+0000","last_change":"2024-03-18T19:06:41.485030+0000","last_active":"2024-03-18T19:06:43.224901+0000","last_peered":"2024-03-18T19:06:43.224901+0000","last_clean":"2024-03-18T19:06:43.224901+0000","last_became_active":"2024-03-18T19:06:41.484777+0000","last_became_peered":"2024-03-18T19:06:41.484777+0000","last_unstale":"2024-03-18T19:06:43.224901+0000","last_undegraded":"2024-03-18T19:06:43.224901+0000","last_fullsized":"2024-03-18T19:06:43.224901+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:57.929295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224245+0000","last_change":"2024-03-18T19:06:41.933603+0000","last_active":"2024-03-18T19:06:43.224245+0000","last_peered":"2024-03-18T19:06:43.224245+0000","last_clean":"2024-03-18T19:06:43.224245+0000","last_became_active":"2024-03-18T19:06:41.933366+0000","last_became_peered":"2024-03-18T19:06:41.933366+0000","last_unstale":"2024-03-18T19:06:43.224245+0000","last_undegraded":"2024-03-18T19:06:43.224245+0000","last_fullsized":"2024-03-18T19:06:43.224245+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:36.486482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"116'2","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224327+0000","last_change":"2024-03-18T19:06:41.485418+0000","last_active":"2024-03-18T19:06:43.224327+0000","last_peered":"2024-03-18T19:06:43.224327+0000","last_clean":"2024-03-18T19:06:43.224327+0000","last_became_active":"2024-03-18T19:06:41.482735+0000","last_became_peered":"2024-03-18T19:06:41.482735+0000","last_unstale":"2024-03-18T19:06:43.224327+0000","last_undegraded":"2024-03-18T19:06:43.224327+0000","last_fullsized":"2024-03-18T19:06:43.224327+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:05.738508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224395+0000","last_change":"2024-03-18T19:06:41.922811+0000","last_active":"2024-03-18T19:06:43.224395+0000","last_peered":"2024-03-18T19:06:43.224395+0000","last_clean":"2024-03-18T19:06:43.224395+0000","last_became_active":"2024-03-18T19:06:41.922520+0000","last_became_peered":"2024-03-18T19:06:41.922520+0000","last_unstale":"2024-03-18T19:06:43.224395+0000","last_undegraded":"2024-03-18T19:06:43.224395+0000","last_fullsized":"2024-03-18T19:06:43.224395+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:00:42.923049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223769+0000","last_change":"2024-03-18T19:06:41.488673+0000","last_active":"2024-03-18T19:06:43.223769+0000","last_peered":"2024-03-18T19:06:43.223769+0000","last_clean":"2024-03-18T19:06:43.223769+0000","last_became_active":"2024-03-18T19:06:41.488381+0000","last_became_peered":"2024-03-18T19:06:41.488381+0000","last_unstale":"2024-03-18T19:06:43.223769+0000","last_undegraded":"2024-03-18T19:06:43.223769+0000","last_fullsized":"2024-03-18T19:06:43.223769+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:14:33.192189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224503+0000","last_change":"2024-03-18T19:06:41.923998+0000","last_active":"2024-03-18T19:06:43.224503+0000","last_peered":"2024-03-18T19:06:43.224503+0000","last_clean":"2024-03-18T19:06:43.224503+0000","last_became_active":"2024-03-18T19:06:41.923718+0000","last_became_peered":"2024-03-18T19:06:41.923718+0000","last_unstale":"2024-03-18T19:06:43.224503+0000","last_undegraded":"2024-03-18T19:06:43.224503+0000","last_fullsized":"2024-03-18T19:06:43.224503+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:08.026486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224451+0000","last_change":"2024-03-18T19:06:41.493551+0000","last_active":"2024-03-18T19:06:43.224451+0000","last_peered":"2024-03-18T19:06:43.224451+0000","last_clean":"2024-03-18T19:06:43.224451+0000","last_became_active":"2024-03-18T19:06:41.493258+0000","last_became_peered":"2024-03-18T19:06:41.493258+0000","last_unstale":"2024-03-18T19:06:43.224451+0000","last_undegraded":"2024-03-18T19:06:43.224451+0000","last_fullsized":"2024-03-18T19:06:43.224451+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:57.929295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224092+0000","last_change":"2024-03-18T19:06:41.925447+0000","last_active":"2024-03-18T19:06:43.224092+0000","last_peered":"2024-03-18T19:06:43.224092+0000","last_clean":"2024-03-18T19:06:43.224092+0000","last_became_active":"2024-03-18T19:06:41.925163+0000","last_became_peered":"2024-03-18T19:06:41.925163+0000","last_unstale":"2024-03-18T19:06:43.224092+0000","last_undegraded":"2024-03-18T19:06:43.224092+0000","last_fullsized":"2024-03-18T19:06:43.224092+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:06:46.759884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223645+0000","last_change":"2024-03-18T19:06:41.488766+0000","last_active":"2024-03-18T19:06:43.223645+0000","last_peered":"2024-03-18T19:06:43.223645+0000","last_clean":"2024-03-18T19:06:43.223645+0000","last_became_active":"2024-03-18T19:06:41.488487+0000","last_became_peered":"2024-03-18T19:06:41.488487+0000","last_unstale":"2024-03-18T19:06:43.223645+0000","last_undegraded":"2024-03-18T19:06:43.223645+0000","last_fullsized":"2024-03-18T19:06:43.223645+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:36.915166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225144+0000","last_change":"2024-03-18T19:06:41.913344+0000","last_active":"2024-03-18T19:06:43.225144+0000","last_peered":"2024-03-18T19:06:43.225144+0000","last_clean":"2024-03-18T19:06:43.225144+0000","last_became_active":"2024-03-18T19:06:41.913075+0000","last_became_peered":"2024-03-18T19:06:41.913075+0000","last_unstale":"2024-03-18T19:06:43.225144+0000","last_undegraded":"2024-03-18T19:06:43.225144+0000","last_fullsized":"2024-03-18T19:06:43.225144+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:43.067283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"116'2","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105763+0000","last_change":"2024-03-18T19:06:41.482479+0000","last_active":"2024-03-18T19:06:43.105763+0000","last_peered":"2024-03-18T19:06:43.105763+0000","last_clean":"2024-03-18T19:06:43.105763+0000","last_became_active":"2024-03-18T19:06:41.482193+0000","last_became_peered":"2024-03-18T19:06:41.482193+0000","last_unstale":"2024-03-18T19:06:43.105763+0000","last_undegraded":"2024-03-18T19:06:43.105763+0000","last_fullsized":"2024-03-18T19:06:43.105763+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:09:18.165390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224456+0000","last_change":"2024-03-18T19:06:41.925518+0000","last_active":"2024-03-18T19:06:43.224456+0000","last_peered":"2024-03-18T19:06:43.224456+0000","last_clean":"2024-03-18T19:06:43.224456+0000","last_became_active":"2024-03-18T19:06:41.924890+0000","last_became_peered":"2024-03-18T19:06:41.924890+0000","last_unstale":"2024-03-18T19:06:43.224456+0000","last_undegraded":"2024-03-18T19:06:43.224456+0000","last_fullsized":"2024-03-18T19:06:43.224456+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:33:44.303133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225477+0000","last_change":"2024-03-18T19:06:41.483079+0000","last_active":"2024-03-18T19:06:43.225477+0000","last_peered":"2024-03-18T19:06:43.225477+0000","last_clean":"2024-03-18T19:06:43.225477+0000","last_became_active":"2024-03-18T19:06:41.481825+0000","last_became_peered":"2024-03-18T19:06:41.481825+0000","last_unstale":"2024-03-18T19:06:43.225477+0000","last_undegraded":"2024-03-18T19:06:43.225477+0000","last_fullsized":"2024-03-18T19:06:43.225477+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:47:27.561126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223967+0000","last_change":"2024-03-18T19:06:41.927486+0000","last_active":"2024-03-18T19:06:43.223967+0000","last_peered":"2024-03-18T19:06:43.223967+0000","last_clean":"2024-03-18T19:06:43.223967+0000","last_became_active":"2024-03-18T19:06:41.927238+0000","last_became_peered":"2024-03-18T19:06:41.927238+0000","last_unstale":"2024-03-18T19:06:43.223967+0000","last_undegraded":"2024-03-18T19:06:43.223967+0000","last_fullsized":"2024-03-18T19:06:43.223967+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:29.079580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225182+0000","last_change":"2024-03-18T19:06:41.488337+0000","last_active":"2024-03-18T19:06:43.225182+0000","last_peered":"2024-03-18T19:06:43.225182+0000","last_clean":"2024-03-18T19:06:43.225182+0000","last_became_active":"2024-03-18T19:06:41.488126+0000","last_became_peered":"2024-03-18T19:06:41.488126+0000","last_unstale":"2024-03-18T19:06:43.225182+0000","last_undegraded":"2024-03-18T19:06:43.225182+0000","last_fullsized":"2024-03-18T19:06:43.225182+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:24.017447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225713+0000","last_change":"2024-03-18T19:06:41.918949+0000","last_active":"2024-03-18T19:06:43.225713+0000","last_peered":"2024-03-18T19:06:43.225713+0000","last_clean":"2024-03-18T19:06:43.225713+0000","last_became_active":"2024-03-18T19:06:41.918549+0000","last_became_peered":"2024-03-18T19:06:41.918549+0000","last_unstale":"2024-03-18T19:06:43.225713+0000","last_undegraded":"2024-03-18T19:06:43.225713+0000","last_fullsized":"2024-03-18T19:06:43.225713+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:23.732496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225376+0000","last_change":"2024-03-18T19:06:41.481513+0000","last_active":"2024-03-18T19:06:43.225376+0000","last_peered":"2024-03-18T19:06:43.225376+0000","last_clean":"2024-03-18T19:06:43.225376+0000","last_became_active":"2024-03-18T19:06:41.481248+0000","last_became_peered":"2024-03-18T19:06:41.481248+0000","last_unstale":"2024-03-18T19:06:43.225376+0000","last_undegraded":"2024-03-18T19:06:43.225376+0000","last_fullsized":"2024-03-18T19:06:43.225376+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:13:22.200232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223591+0000","last_change":"2024-03-18T19:06:41.927083+0000","last_active":"2024-03-18T19:06:43.223591+0000","last_peered":"2024-03-18T19:06:43.223591+0000","last_clean":"2024-03-18T19:06:43.223591+0000","last_became_active":"2024-03-18T19:06:41.926671+0000","last_became_peered":"2024-03-18T19:06:41.926671+0000","last_unstale":"2024-03-18T19:06:43.223591+0000","last_undegraded":"2024-03-18T19:06:43.223591+0000","last_fullsized":"2024-03-18T19:06:43.223591+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:05.216483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224237+0000","last_change":"2024-03-18T19:06:41.490696+0000","last_active":"2024-03-18T19:06:43.224237+0000","last_peered":"2024-03-18T19:06:43.224237+0000","last_clean":"2024-03-18T19:06:43.224237+0000","last_became_active":"2024-03-18T19:06:41.490438+0000","last_became_peered":"2024-03-18T19:06:41.490438+0000","last_unstale":"2024-03-18T19:06:43.224237+0000","last_undegraded":"2024-03-18T19:06:43.224237+0000","last_fullsized":"2024-03-18T19:06:43.224237+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:36.889621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.906913+0000","last_change":"2024-03-18T19:06:41.924694+0000","last_active":"2024-03-18T19:06:42.906913+0000","last_peered":"2024-03-18T19:06:42.906913+0000","last_clean":"2024-03-18T19:06:42.906913+0000","last_became_active":"2024-03-18T19:06:41.924446+0000","last_became_peered":"2024-03-18T19:06:41.924446+0000","last_unstale":"2024-03-18T19:06:42.906913+0000","last_undegraded":"2024-03-18T19:06:42.906913+0000","last_fullsized":"2024-03-18T19:06:42.906913+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:45.399156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225520+0000","last_change":"2024-03-18T19:06:41.482677+0000","last_active":"2024-03-18T19:06:43.225520+0000","last_peered":"2024-03-18T19:06:43.225520+0000","last_clean":"2024-03-18T19:06:43.225520+0000","last_became_active":"2024-03-18T19:06:41.482356+0000","last_became_peered":"2024-03-18T19:06:41.482356+0000","last_unstale":"2024-03-18T19:06:43.225520+0000","last_undegraded":"2024-03-18T19:06:43.225520+0000","last_fullsized":"2024-03-18T19:06:43.225520+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:42.061953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224815+0000","last_change":"2024-03-18T19:06:41.913424+0000","last_active":"2024-03-18T19:06:43.224815+0000","last_peered":"2024-03-18T19:06:43.224815+0000","last_clean":"2024-03-18T19:06:43.224815+0000","last_became_active":"2024-03-18T19:06:41.913150+0000","last_became_peered":"2024-03-18T19:06:41.913150+0000","last_unstale":"2024-03-18T19:06:43.224815+0000","last_undegraded":"2024-03-18T19:06:43.224815+0000","last_fullsized":"2024-03-18T19:06:43.224815+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:09:21.138871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225277+0000","last_change":"2024-03-18T19:06:41.484792+0000","last_active":"2024-03-18T19:06:43.225277+0000","last_peered":"2024-03-18T19:06:43.225277+0000","last_clean":"2024-03-18T19:06:43.225277+0000","last_became_active":"2024-03-18T19:06:41.484500+0000","last_became_peered":"2024-03-18T19:06:41.484500+0000","last_unstale":"2024-03-18T19:06:43.225277+0000","last_undegraded":"2024-03-18T19:06:43.225277+0000","last_fullsized":"2024-03-18T19:06:43.225277+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:10.069476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224190+0000","last_change":"2024-03-18T19:06:41.930511+0000","last_active":"2024-03-18T19:06:43.224190+0000","last_peered":"2024-03-18T19:06:43.224190+0000","last_clean":"2024-03-18T19:06:43.224190+0000","last_became_active":"2024-03-18T19:06:41.930173+0000","last_became_peered":"2024-03-18T19:06:41.930173+0000","last_unstale":"2024-03-18T19:06:43.224190+0000","last_undegraded":"2024-03-18T19:06:43.224190+0000","last_fullsized":"2024-03-18T19:06:43.224190+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:00:02.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105590+0000","last_change":"2024-03-18T19:06:41.484999+0000","last_active":"2024-03-18T19:06:43.105590+0000","last_peered":"2024-03-18T19:06:43.105590+0000","last_clean":"2024-03-18T19:06:43.105590+0000","last_became_active":"2024-03-18T19:06:41.484693+0000","last_became_peered":"2024-03-18T19:06:41.484693+0000","last_unstale":"2024-03-18T19:06:43.105590+0000","last_undegraded":"2024-03-18T19:06:43.105590+0000","last_fullsized":"2024-03-18T19:06:43.105590+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:33:21.349317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105715+0000","last_change":"2024-03-18T19:06:41.916902+0000","last_active":"2024-03-18T19:06:43.105715+0000","last_peered":"2024-03-18T19:06:43.105715+0000","last_clean":"2024-03-18T19:06:43.105715+0000","last_became_active":"2024-03-18T19:06:41.916635+0000","last_became_peered":"2024-03-18T19:06:41.916635+0000","last_unstale":"2024-03-18T19:06:43.105715+0000","last_undegraded":"2024-03-18T19:06:43.105715+0000","last_fullsized":"2024-03-18T19:06:43.105715+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:39.234562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.906036+0000","last_change":"2024-03-18T19:06:41.488714+0000","last_active":"2024-03-18T19:06:42.906036+0000","last_peered":"2024-03-18T19:06:42.906036+0000","last_clean":"2024-03-18T19:06:42.906036+0000","last_became_active":"2024-03-18T19:06:41.484846+0000","last_became_peered":"2024-03-18T19:06:41.484846+0000","last_unstale":"2024-03-18T19:06:42.906036+0000","last_undegraded":"2024-03-18T19:06:42.906036+0000","last_fullsized":"2024-03-18T19:06:42.906036+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:57.929295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"106'183","reported_seq":486,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-18T19:06:41.906777+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T19:06:41.906777+0000","last_peered":"2024-03-18T19:06:41.906777+0000","last_clean":"2024-03-18T19:06:41.906777+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T19:06:41.906777+0000","last_undegraded":"2024-03-18T19:06:41.906777+0000","last_fullsized":"2024-03-18T19:06:41.906777+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575048,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59356,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312484,"statfs":{"total":96636764160,"available":96575983616,"internally_reserved":0,"allocated":2043904,"data_stored":1311716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.55900000000000005,"15min":0.57099999999999995},"min":{"1min":0.39400000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.64100000000000001,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.432},{"interface":"front","average":{"1min":0.504,"5min":0.56000000000000005,"15min":0.58199999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68700000000000006,"5min":1.387,"15min":1.387},"last":0.58199999999999996}]},{"osd":1,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.54000000000000004,"15min":0.57499999999999996},"min":{"1min":0.36499999999999999,"5min":0.32300000000000001,"15min":0.307},"max":{"1min":0.66200000000000003,"5min":0.81699999999999995,"15min":1.484},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.503,"5min":0.56499999999999995,"15min":0.58899999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.65600000000000003,"5min":0.80700000000000005,"15min":0.92000000000000004},"last":0.377}]},{"osd":2,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55900000000000005,"15min":0.57799999999999996},"min":{"1min":0.42199999999999999,"5min":0.34499999999999997,"15min":0.27800000000000002},"max":{"1min":0.69799999999999995,"5min":1.3600000000000001,"15min":1.5129999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.59099999999999997},"min":{"1min":0.38300000000000001,"5min":0.36899999999999999,"15min":0.30199999999999999},"max":{"1min":0.73699999999999999,"5min":1.327,"15min":1.536},"last":0.47699999999999998}]},{"osd":3,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55200000000000005,"15min":0.57999999999999996},"min":{"1min":0.35599999999999998,"5min":0.318,"15min":0.311},"max":{"1min":0.61699999999999999,"5min":0.79100000000000004,"15min":1.5489999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.57599999999999996,"15min":0.60099999999999998},"min":{"1min":0.40600000000000003,"5min":0.38800000000000001,"15min":0.38200000000000001},"max":{"1min":0.64900000000000002,"5min":0.78200000000000003,"15min":1.5660000000000001},"last":0.57099999999999995}]},{"osd":4,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.45400000000000001},"min":{"1min":0.311,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.59299999999999997,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.46999999999999997,"15min":0.47899999999999998},"min":{"1min":0.316,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.73199999999999998},"last":0.501}]},{"osd":5,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.47799999999999998,"15min":0.47999999999999998},"min":{"1min":0.34200000000000003,"5min":0.27100000000000002,"15min":0.26000000000000001},"max":{"1min":0.71499999999999997,"5min":0.751,"15min":0.78300000000000003},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.504,"15min":0.51800000000000002},"min":{"1min":0.35699999999999998,"5min":0.30299999999999999,"15min":0.28799999999999998},"max":{"1min":0.751,"5min":0.78200000000000003,"15min":0.80300000000000005},"last":0.66400000000000003}]},{"osd":6,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53500000000000003,"15min":0.54600000000000004},"min":{"1min":0.38600000000000001,"5min":0.374,"15min":0.34300000000000003},"max":{"1min":0.67700000000000005,"5min":0.81799999999999995,"15min":0.84499999999999997},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.35899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68700000000000006,"5min":0.752,"15min":0.77500000000000002},"last":0.55400000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575048,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32548,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339292,"statfs":{"total":96636764160,"available":96603435008,"internally_reserved":0,"allocated":1462272,"data_stored":723460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56599999999999995,"15min":0.56699999999999995},"min":{"1min":0.38600000000000001,"5min":0.33000000000000002,"15min":0.28499999999999998},"max":{"1min":0.73699999999999999,"5min":0.75600000000000001,"15min":0.76800000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56000000000000005,"15min":0.57299999999999995},"min":{"1min":0.38400000000000001,"5min":0.34699999999999998,"15min":0.27400000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.46800000000000003}]},{"osd":1,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.35399999999999998,"5min":0.34000000000000002,"15min":0.30599999999999999},"max":{"1min":0.71499999999999997,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56299999999999994,"15min":0.57099999999999995},"min":{"1min":0.35699999999999998,"5min":0.32000000000000001,"15min":0.311},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.57699999999999996}]},{"osd":2,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.35299999999999998,"5min":0.32700000000000001,"15min":0.30499999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.57299999999999995,"15min":0.57999999999999996},"min":{"1min":0.40100000000000002,"5min":0.36699999999999999,"15min":0.30299999999999999},"max":{"1min":0.753,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.65500000000000003}]},{"osd":3,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59099999999999997,"15min":0.58499999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.32100000000000001},"max":{"1min":0.78400000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.57199999999999995,"15min":0.57799999999999996},"min":{"1min":0.41099999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79500000000000004,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.497}]},{"osd":4,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.25600000000000001},"max":{"1min":0.75,"5min":0.75900000000000001,"15min":0.76300000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.47899999999999998,"15min":0.47799999999999998},"min":{"1min":0.29599999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.73099999999999998,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.61899999999999999}]},{"osd":5,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.46999999999999997,"15min":0.47099999999999997},"min":{"1min":0.33000000000000002,"5min":0.27300000000000002,"15min":0.26900000000000002},"max":{"1min":0.71899999999999997,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.52000000000000002,"15min":0.51700000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76600000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.75600000000000001}]},{"osd":7,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53300000000000003,"15min":0.52400000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81100000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.54200000000000004,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77900000000000003,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.68700000000000006}]}]},{"osd":1,"up_from":13,"seq":55834575048,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1445888,"data_stored":721362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.36499999999999999,"15min":0.373},"min":{"1min":0.217,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52900000000000003,"5min":0.63300000000000001,"15min":0.66300000000000003},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.40600000000000003,"15min":0.41399999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.66500000000000004,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.32400000000000001}]},{"osd":2,"last update":"Mon Mar 18 19:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.40000000000000002,"15min":0.40500000000000003},"min":{"1min":0.28899999999999998,"5min":0.26500000000000001,"15min":0.254},"max":{"1min":0.64200000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.439,"15min":0.44400000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.58299999999999996,"5min":0.68700000000000006,"15min":0.71699999999999997},"last":0.35699999999999998}]},{"osd":3,"last update":"Mon Mar 18 19:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.44900000000000001,"15min":0.46200000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.38},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.41399999999999998,"15min":0.41899999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.56499999999999995,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.39300000000000002}]},{"osd":4,"last update":"Mon Mar 18 19:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.61299999999999999,"15min":0.627},"min":{"1min":0.42199999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.77200000000000002,"5min":0.93999999999999995,"15min":1.0409999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.61899999999999999,"15min":0.64100000000000001},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.34499999999999997},"max":{"1min":0.79100000000000004,"5min":1.018,"15min":1.018},"last":0.53900000000000003}]},{"osd":5,"last update":"Mon Mar 18 19:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.61299999999999999,"15min":0.63800000000000001},"min":{"1min":0.40699999999999997,"5min":0.40100000000000002,"15min":0.38600000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.99199999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.63,"15min":0.65200000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.70099999999999996,"5min":1.006,"15min":1.1319999999999999},"last":0.55200000000000005}]},{"osd":6,"last update":"Mon Mar 18 19:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.61499999999999999,"15min":0.63600000000000001},"min":{"1min":0.42299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.68400000000000005,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.39100000000000001},"max":{"1min":0.82999999999999996,"5min":0.96799999999999997,"15min":0.98099999999999998},"last":0.64800000000000002}]},{"osd":7,"last update":"Mon Mar 18 19:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.64800000000000002,"15min":0.65800000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.36699999999999999},"max":{"1min":0.84599999999999997,"5min":0.92900000000000005,"15min":1.0660000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.64400000000000002,"15min":0.66000000000000003},"min":{"1min":0.45300000000000001,"5min":0.40999999999999998,"15min":0.39900000000000002},"max":{"1min":0.81299999999999994,"5min":0.94299999999999995,"15min":1.1499999999999999},"last":0.60599999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575049,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37228,"kb_used_data":2012,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":2060288,"data_stored":1313930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 19:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36799999999999999,"15min":0.378},"min":{"1min":0.221,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.58599999999999997,"5min":0.60499999999999998,"15min":0.621},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.40000000000000002,"15min":0.41199999999999998},"min":{"1min":0.253,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.53400000000000003,"5min":0.61699999999999999,"15min":0.65100000000000002},"last":0.46800000000000003}]},{"osd":2,"last update":"Mon Mar 18 19:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.40899999999999997,"15min":0.41799999999999998},"min":{"1min":0.26200000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.627},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.42699999999999999,"15min":0.432},"min":{"1min":0.31,"5min":0.25,"15min":0.25},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.435}]},{"osd":3,"last update":"Mon Mar 18 19:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.39800000000000002,"15min":0.43099999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.217},"max":{"1min":0.53900000000000003,"5min":0.61299999999999999,"15min":5.7359999999999998},"last":0.495},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.45500000000000002,"15min":0.48799999999999999},"min":{"1min":0.33500000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64900000000000002,"5min":0.74399999999999999,"15min":5.9329999999999998},"last":0.50900000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.621,"15min":0.64000000000000001},"min":{"1min":0.47599999999999998,"5min":0.33200000000000002,"15min":0.27200000000000002},"max":{"1min":0.86699999999999999,"5min":0.91800000000000004,"15min":1.22},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.628,"15min":0.64200000000000002},"min":{"1min":0.39000000000000001,"5min":0.36799999999999999,"15min":0.315},"max":{"1min":0.83599999999999997,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.59999999999999998}]},{"osd":5,"last update":"Mon Mar 18 19:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63800000000000001,"15min":0.65500000000000003},"min":{"1min":0.45200000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.89400000000000002,"5min":0.96199999999999997,"15min":2.9649999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.63900000000000001,"15min":0.66000000000000003},"min":{"1min":0.41599999999999998,"5min":0.38,"15min":0.31},"max":{"1min":0.77100000000000002,"5min":1.0760000000000001,"15min":1.2609999999999999},"last":0.59299999999999997}]},{"osd":6,"last update":"Mon Mar 18 19:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.63600000000000001,"15min":0.65500000000000003},"min":{"1min":0.439,"5min":0.39600000000000002,"15min":0.38100000000000001},"max":{"1min":0.78800000000000003,"5min":0.94599999999999995,"15min":1.133},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.626,"5min":0.65100000000000002,"15min":0.67700000000000005},"min":{"1min":0.42699999999999999,"5min":0.41199999999999998,"15min":0.40999999999999998},"max":{"1min":0.93400000000000005,"5min":1.0409999999999999,"15min":2.9300000000000002},"last":0.70099999999999996}]},{"osd":7,"last update":"Mon Mar 18 19:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65900000000000003,"15min":0.66900000000000004},"min":{"1min":0.45800000000000002,"5min":0.38300000000000001,"15min":0.34899999999999998},"max":{"1min":0.75700000000000001,"5min":1.0229999999999999,"15min":1.2849999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.65400000000000003,"15min":0.67100000000000004},"min":{"1min":0.46400000000000002,"5min":0.437,"15min":0.38500000000000001},"max":{"1min":0.90300000000000002,"5min":0.94299999999999995,"15min":1.002},"last":0.70899999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575049,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58768,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94313072,"statfs":{"total":96636764160,"available":96576585728,"internally_reserved":0,"allocated":1441792,"data_stored":721292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.375,"15min":0.371},"min":{"1min":0.22,"5min":0.22,"15min":0.17799999999999999},"max":{"1min":0.52800000000000002,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.29599999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.57199999999999995,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.39400000000000002}]},{"osd":1,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.46899999999999997,"15min":0.46800000000000003},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.27100000000000002},"max":{"1min":0.56499999999999995,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.40400000000000003,"15min":0.39200000000000002},"min":{"1min":0.26100000000000001,"5min":0.23699999999999999,"15min":0.216},"max":{"1min":0.47999999999999998,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.28999999999999998}]},{"osd":3,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.41699999999999998,"15min":0.40600000000000003},"min":{"1min":0.27800000000000002,"5min":0.24199999999999999,"15min":0.22500000000000001},"max":{"1min":0.65800000000000003,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.504,"15min":0.498},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.27700000000000002},"max":{"1min":0.66100000000000003,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.46200000000000002}]},{"osd":4,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.61799999999999999,"15min":0.61699999999999999},"min":{"1min":0.38600000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.73599999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.61499999999999999,"15min":0.626},"min":{"1min":0.378,"5min":0.35899999999999999,"15min":0.32100000000000001},"max":{"1min":0.79200000000000004,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.48799999999999999}]},{"osd":5,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.64500000000000002,"15min":0.64400000000000002},"min":{"1min":0.39800000000000002,"5min":0.34699999999999998,"15min":0.309},"max":{"1min":0.77700000000000002,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.65300000000000002,"15min":0.64000000000000001},"min":{"1min":0.40500000000000003,"5min":0.375,"15min":0.33400000000000002},"max":{"1min":0.78200000000000003,"5min":1.948,"15min":1.948},"last":0.55600000000000005}]},{"osd":6,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.66000000000000003,"15min":0.65500000000000003},"min":{"1min":0.39600000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82299999999999995,"5min":2.9470000000000001,"15min":2.9470000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.68200000000000005,"15min":0.68400000000000005},"min":{"1min":0.44900000000000001,"5min":0.432,"15min":0.37},"max":{"1min":0.83299999999999996,"5min":1.054,"15min":1.083},"last":0.63200000000000001}]},{"osd":7,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.65500000000000003,"15min":0.65100000000000002},"min":{"1min":0.41799999999999998,"5min":0.40999999999999998,"15min":0.38200000000000001},"max":{"1min":0.78000000000000003,"5min":1.083,"15min":1.083},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.65300000000000002,"15min":0.66100000000000003},"min":{"1min":0.39500000000000002,"5min":0.371,"15min":0.34599999999999997},"max":{"1min":0.84799999999999998,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.64300000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575049,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36620,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335220,"statfs":{"total":96636764160,"available":96599265280,"internally_reserved":0,"allocated":1441792,"data_stored":721292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.36899999999999999,"15min":0.379},"min":{"1min":0.23699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.51200000000000001,"5min":0.81100000000000005,"15min":1.204},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.40400000000000003,"15min":0.39600000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.23000000000000001},"max":{"1min":0.57499999999999996,"5min":1.137,"15min":1.137},"last":0.435}]},{"osd":1,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40000000000000002,"15min":0.39700000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.221},"max":{"1min":0.621,"5min":0.999,"15min":1.1220000000000001},"last":0.307},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.437,"15min":0.443},"min":{"1min":0.29999999999999999,"5min":0.28499999999999998,"15min":0.26500000000000001},"max":{"1min":0.51900000000000002,"5min":1.093,"15min":1.1080000000000001},"last":0.41799999999999998}]},{"osd":2,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.318,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.67500000000000004,"5min":1.0369999999999999,"15min":1.1879999999999999},"last":0.378},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.46300000000000002,"15min":0.47499999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.28599999999999998},"max":{"1min":0.55600000000000005,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.45300000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.052,"5min":1.4219999999999999,"15min":1.871},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.61199999999999999,"15min":0.63},"min":{"1min":0.42699999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78300000000000003,"5min":1.0700000000000001,"15min":1.764},"last":0.83799999999999997}]},{"osd":5,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.65600000000000003,"15min":0.65900000000000003},"min":{"1min":0.44400000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.0249999999999999,"5min":2.5329999999999999,"15min":2.5329999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67000000000000004,"15min":0.68400000000000005},"min":{"1min":0.47099999999999997,"5min":0.45500000000000002,"15min":0.437},"max":{"1min":0.94599999999999995,"5min":1.5409999999999999,"15min":1.794},"last":0.76800000000000002}]},{"osd":6,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.66700000000000004,"15min":0.67200000000000004},"min":{"1min":0.44400000000000001,"5min":0.41499999999999998,"15min":0.40300000000000002},"max":{"1min":1.1060000000000001,"5min":2.7389999999999999,"15min":2.7389999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.66100000000000003,"15min":0.67500000000000004},"min":{"1min":0.47799999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96699999999999997,"5min":1.159,"15min":1.8069999999999999},"last":0.91800000000000004}]},{"osd":7,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.64400000000000002,"15min":0.66900000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86499999999999999,"5min":1.3959999999999999,"15min":1.732},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.66800000000000004,"15min":0.67000000000000004},"min":{"1min":0.47399999999999998,"5min":0.439,"15min":0.40500000000000003},"max":{"1min":1.085,"5min":2.7050000000000001,"15min":2.7050000000000001},"last":0.63100000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575049,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32536,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339304,"statfs":{"total":96636764160,"available":96603447296,"internally_reserved":0,"allocated":1449984,"data_stored":721452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55000000000000004,"15min":0.56899999999999995},"min":{"1min":0.31,"5min":0.28999999999999998,"15min":0.246},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.627,"5min":0.57299999999999995,"15min":0.58699999999999997},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.645,"5min":3.645,"15min":3.645},"last":0.53100000000000003}]},{"osd":1,"last update":"Mon Mar 18 19:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.64400000000000002,"15min":0.60899999999999999},"min":{"1min":0.36199999999999999,"5min":0.34499999999999997,"15min":0.316},"max":{"1min":9.9499999999999993,"5min":9.9499999999999993,"15min":9.9499999999999993},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53900000000000003,"15min":0.57599999999999996},"min":{"1min":0.32300000000000001,"5min":0.308,"15min":0.308},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.51700000000000002}]},{"osd":2,"last update":"Mon Mar 18 19:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.55200000000000005,"15min":0.57399999999999995},"min":{"1min":0.371,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.96299999999999997,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.58599999999999997,"15min":0.60399999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.32600000000000001},"max":{"1min":3.5990000000000002,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.56299999999999994}]},{"osd":3,"last update":"Mon Mar 18 19:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55400000000000005,"15min":0.58599999999999997},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56399999999999995,"15min":0.59399999999999997},"min":{"1min":0.39300000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70299999999999996,"5min":1.121,"15min":1.121},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Mar 18 19:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45100000000000001,"15min":0.46100000000000002},"min":{"1min":0.32800000000000001,"5min":0.29599999999999999,"15min":0.245},"max":{"1min":0.68400000000000005,"5min":0.73499999999999999,"15min":0.77500000000000002},"last":0.376},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.50700000000000001,"15min":0.49399999999999999},"min":{"1min":0.35799999999999998,"5min":0.30099999999999999,"15min":0.27200000000000002},"max":{"1min":5.4900000000000002,"5min":5.4900000000000002,"15min":5.4900000000000002},"last":0.46200000000000002}]},{"osd":6,"last update":"Mon Mar 18 19:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.46999999999999997,"15min":0.48199999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.29499999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.78900000000000003},"last":0.439},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.52600000000000002,"15min":0.52700000000000002},"min":{"1min":0.34999999999999998,"5min":0.33200000000000002,"15min":0.28999999999999998},"max":{"1min":3.7050000000000001,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":0.48199999999999998}]},{"osd":7,"last update":"Mon Mar 18 19:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.50800000000000001,"15min":0.52000000000000002},"min":{"1min":0.37,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52400000000000002,"15min":0.54800000000000004},"min":{"1min":0.39500000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.73799999999999999,"5min":0.75900000000000001,"15min":0.79700000000000004},"last":0.503}]}]},{"osd":5,"up_from":13,"seq":55834575048,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50580,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321260,"statfs":{"total":96636764160,"available":96584970240,"internally_reserved":0,"allocated":1441792,"data_stored":721292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:06:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.54500000000000004,"15min":0.56799999999999995},"min":{"1min":0.36899999999999999,"5min":0.26800000000000002,"15min":0.26600000000000001},"max":{"1min":0.67800000000000005,"5min":0.78400000000000003,"15min":1.1679999999999999},"last":0.436},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.57099999999999995,"15min":0.57199999999999995},"min":{"1min":0.38,"5min":0.33700000000000002,"15min":0.25800000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":1.1319999999999999},"last":0.48999999999999999}]},{"osd":1,"last update":"Mon Mar 18 19:06:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56100000000000005,"15min":0.57299999999999995},"min":{"1min":0.32700000000000001,"5min":0.32600000000000001,"15min":0.29599999999999999},"max":{"1min":0.748,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.57499999999999996,"15min":0.58299999999999996},"min":{"1min":0.35999999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70499999999999996,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.47099999999999997}]},{"osd":2,"last update":"Mon Mar 18 19:06:15 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.317,"5min":0.30399999999999999,"15min":0.28399999999999997},"max":{"1min":0.66500000000000004,"5min":1.153,"15min":1.1579999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56100000000000005,"15min":0.57499999999999996},"min":{"1min":0.41499999999999998,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":1.0569999999999999},"last":0.42499999999999999}]},{"osd":3,"last update":"Mon Mar 18 19:06:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.39000000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.65800000000000003,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59099999999999997,"15min":0.60099999999999998},"min":{"1min":0.42499999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.71499999999999997,"5min":1.212,"15min":1.212},"last":0.38}]},{"osd":4,"last update":"Mon Mar 18 19:06:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.47399999999999998,"15min":0.46700000000000003},"min":{"1min":0.34599999999999997,"5min":0.255,"15min":0.24199999999999999},"max":{"1min":0.69499999999999995,"5min":1.248,"15min":1.248},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.33500000000000002,"5min":0.316,"15min":0.23100000000000001},"max":{"1min":0.63800000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.33800000000000002}]},{"osd":6,"last update":"Mon Mar 18 19:06:15 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.78200000000000003,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.50700000000000001,"15min":0.51000000000000001},"min":{"1min":0.33800000000000002,"5min":0.32100000000000001,"15min":0.26800000000000002},"max":{"1min":0.83199999999999996,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.46100000000000002}]},{"osd":7,"last update":"Mon Mar 18 19:06:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.32700000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.69999999999999996,"5min":1.282,"15min":1.282},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55500000000000005,"15min":0.54800000000000004},"min":{"1min":0.35699999999999998,"5min":0.31900000000000001,"15min":0.28499999999999998},"max":{"1min":0.72199999999999998,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.41199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T19:06:46.052 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T19:06:46.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.215+0000 7f943a0b6700 1 Processor -- start 2024-03-18T19:06:46.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.215+0000 7f943a0b6700 1 -- start start 2024-03-18T19:06:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f943a0b6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9434099a40 0x7f9434099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f943a0b6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f94340cc310 0x7f94340cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f943a0b6700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f943409a3e0 con 0x7f943415d680 2024-03-18T19:06:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f943a0b6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94340d0c80 con 0x7f94340cc310 2024-03-18T19:06:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f943a0b6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94340d09a0 con 0x7f9434099a40 2024-03-18T19:06:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f94337fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f943415d680 0x7f94340cff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47154/0 (socket says 172.21.15.1:47154) 2024-03-18T19:06:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f94337fe700 1 -- 172.21.15.1:0/151541911 learned_addr learned my addr 172.21.15.1:0/151541911 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f9432ffd700 1 --2- 172.21.15.1:0/151541911 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9434099a40 0x7f9434099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f94327fc700 1 -- 172.21.15.1:0/151541911 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285543361 0 0) 0x7f943409a3e0 con 0x7f943415d680 2024-03-18T19:06:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f9433fff700 1 --2- 172.21.15.1:0/151541911 >> v2:172.21.15.165:3300/0 conn(0x7f94340cc310 0x7f94340cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f94327fc700 1 -- 172.21.15.1:0/151541911 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f941c0035b0 con 0x7f943415d680 2024-03-18T19:06:46.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f94327fc700 1 -- 172.21.15.1:0/151541911 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 201727340 0 0) 0x7f941c0035b0 con 0x7f943415d680 2024-03-18T19:06:46.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f94327fc700 1 -- 172.21.15.1:0/151541911 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9434099a40 msgr2=0x7f9434099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:46.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f94327fc700 1 --2- 172.21.15.1:0/151541911 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9434099a40 0x7f9434099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f94327fc700 1 -- 172.21.15.1:0/151541911 >> v2:172.21.15.165:3300/0 conn(0x7f94340cc310 msgr2=0x7f94340cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:46.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f94327fc700 1 --2- 172.21.15.1:0/151541911 >> v2:172.21.15.165:3300/0 conn(0x7f94340cc310 0x7f94340cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f94327fc700 1 -- 172.21.15.1:0/151541911 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94340cb830 con 0x7f943415d680 2024-03-18T19:06:46.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f94327fc700 1 -- 172.21.15.1:0/151541911 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f9428001670 con 0x7f943415d680 2024-03-18T19:06:46.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f94327fc700 1 -- 172.21.15.1:0/151541911 >> v1:172.21.15.1:6789/0 conn(0x7f943415d680 legacy=0x7f94340cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:46.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f94327fc700 1 --2- 172.21.15.1:0/151541911 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9434099a40 0x7f941c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:46.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f94327fc700 1 --2- 172.21.15.1:0/151541911 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f941c003f80 0x7f941c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:46.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f94327fc700 1 --2- 172.21.15.1:0/151541911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f941c004d90 0x7f941c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:46.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f94327fc700 1 -- 172.21.15.1:0/151541911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94340d0c80 con 0x7f941c004d90 2024-03-18T19:06:46.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f94327fc700 1 -- 172.21.15.1:0/151541911 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94340d09a0 con 0x7f941c003f80 2024-03-18T19:06:46.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.216+0000 7f94327fc700 1 -- 172.21.15.1:0/151541911 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f941c00ba10 con 0x7f9434099a40 2024-03-18T19:06:46.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.217+0000 7f9433fff700 1 --2- 172.21.15.1:0/151541911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f941c004d90 0x7f941c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:46.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.217+0000 7f94337fe700 1 --2- 172.21.15.1:0/151541911 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9434099a40 0x7f941c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:46.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.217+0000 7f9432ffd700 1 --2- 172.21.15.1:0/151541911 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f941c003f80 0x7f941c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:46.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.217+0000 7f9433fff700 1 -- 172.21.15.1:0/151541911 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9434099a40 msgr2=0x7f941c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:46.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.217+0000 7f9433fff700 1 --2- 172.21.15.1:0/151541911 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9434099a40 0x7f941c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.217+0000 7f9433fff700 1 -- 172.21.15.1:0/151541911 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f941c003f80 msgr2=0x7f941c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:46.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.217+0000 7f9433fff700 1 --2- 172.21.15.1:0/151541911 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f941c003f80 0x7f941c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.217+0000 7f9433fff700 1 -- 172.21.15.1:0/151541911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941c0035b0 con 0x7f941c004d90 2024-03-18T19:06:46.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.217+0000 7f9433fff700 1 --2- 172.21.15.1:0/151541911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f941c004d90 0x7f941c005210 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f942c009fc0 tx=0x7f941c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a8e6b784e353ef server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:46.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.217+0000 7f94327fc700 1 -- 172.21.15.1:0/151541911 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c00ebd0 con 0x7f941c004d90 2024-03-18T19:06:46.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.217+0000 7f94327fc700 1 -- 172.21.15.1:0/151541911 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f942c00f040 con 0x7f941c004d90 2024-03-18T19:06:46.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.217+0000 7f94327fc700 1 -- 172.21.15.1:0/151541911 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c014480 con 0x7f941c004d90 2024-03-18T19:06:46.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.218+0000 7f943a0b6700 1 -- 172.21.15.1:0/151541911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f941c004d90 msgr2=0x7f941c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:46.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.218+0000 7f943a0b6700 1 --2- 172.21.15.1:0/151541911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f941c004d90 0x7f941c005210 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f942c009fc0 tx=0x7f941c005880 comp rx=0 tx=0).stop 2024-03-18T19:06:46.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.218+0000 7f9433fff700 1 -- 172.21.15.1:0/151541911 reap_dead start 2024-03-18T19:06:46.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.218+0000 7f943a0b6700 1 -- 172.21.15.1:0/151541911 shutdown_connections 2024-03-18T19:06:46.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.218+0000 7f943a0b6700 1 -- 172.21.15.1:0/151541911 >> 172.21.15.1:0/151541911 conn(0x7f94340b2ba0 msgr2=0x7f94340b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:46.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.218+0000 7f943a0b6700 1 -- 172.21.15.1:0/151541911 shutdown_connections 2024-03-18T19:06:46.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.218+0000 7f943a0b6700 1 -- 172.21.15.1:0/151541911 wait complete. 2024-03-18T19:06:46.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.219+0000 7f943a0b6700 1 Processor -- start 2024-03-18T19:06:46.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.219+0000 7f943a0b6700 1 -- start start 2024-03-18T19:06:46.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.220+0000 7f943a0b6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f941c004d90 0x7f94340b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:46.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.220+0000 7f943a0b6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94340b1bc0 0x7f943415c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:46.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.220+0000 7f943a0b6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94340b2040 0x7f943415c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:46.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.220+0000 7f943a0b6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943415cae0 con 0x7f94340b1bc0 2024-03-18T19:06:46.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.220+0000 7f943a0b6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94340c8300 con 0x7f941c004d90 2024-03-18T19:06:46.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.220+0000 7f943a0b6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94340c85e0 con 0x7f94340b2040 2024-03-18T19:06:46.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.220+0000 7f9432ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94340b1bc0 0x7f943415c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:46.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.220+0000 7f9432ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94340b1bc0 0x7f943415c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49026/0 (socket says 172.21.15.1:49026) 2024-03-18T19:06:46.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.220+0000 7f9432ffd700 1 -- 172.21.15.1:0/2476319697 learned_addr learned my addr 172.21.15.1:0/2476319697 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:46.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.220+0000 7f9433fff700 1 --2- 172.21.15.1:0/2476319697 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94340b2040 0x7f943415c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:46.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.220+0000 7f94337fe700 1 --2- 172.21.15.1:0/2476319697 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f941c004d90 0x7f94340b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:46.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.220+0000 7f9432ffd700 1 -- 172.21.15.1:0/2476319697 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94340b2040 msgr2=0x7f943415c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:46.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.220+0000 7f9432ffd700 1 --2- 172.21.15.1:0/2476319697 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94340b2040 0x7f943415c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.220+0000 7f9432ffd700 1 -- 172.21.15.1:0/2476319697 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f941c004d90 msgr2=0x7f94340b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:46.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.220+0000 7f9432ffd700 1 --2- 172.21.15.1:0/2476319697 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f941c004d90 0x7f94340b2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.220+0000 7f9432ffd700 1 -- 172.21.15.1:0/2476319697 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94340c88c0 con 0x7f94340b1bc0 2024-03-18T19:06:46.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.220+0000 7f9432ffd700 1 --2- 172.21.15.1:0/2476319697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94340b1bc0 0x7f943415c060 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f94240119a0 tx=0x7f9424011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:46.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.221+0000 7f941bfff700 1 -- 172.21.15.1:0/2476319697 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942400f940 con 0x7f94340b1bc0 2024-03-18T19:06:46.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.221+0000 7f941bfff700 1 -- 172.21.15.1:0/2476319697 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f942400faa0 con 0x7f94340b1bc0 2024-03-18T19:06:46.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.221+0000 7f943a0b6700 1 -- 172.21.15.1:0/2476319697 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94340c8ba0 con 0x7f94340b1bc0 2024-03-18T19:06:46.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.221+0000 7f941bfff700 1 -- 172.21.15.1:0/2476319697 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942400fe30 con 0x7f94340b1bc0 2024-03-18T19:06:46.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.221+0000 7f943a0b6700 1 -- 172.21.15.1:0/2476319697 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f943409ea50 con 0x7f94340b1bc0 2024-03-18T19:06:46.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.222+0000 7f943a0b6700 1 -- 172.21.15.1:0/2476319697 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94340676a0 con 0x7f94340b1bc0 2024-03-18T19:06:46.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.222+0000 7f941bfff700 1 -- 172.21.15.1:0/2476319697 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9424014440 con 0x7f94340b1bc0 2024-03-18T19:06:46.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.222+0000 7f941bfff700 1 --2- 172.21.15.1:0/2476319697 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f942006adb0 0x7f942006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:46.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.223+0000 7f94337fe700 1 --2- 172.21.15.1:0/2476319697 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f942006adb0 0x7f942006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:46.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.223+0000 7f941bfff700 1 -- 172.21.15.1:0/2476319697 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9424089d70 con 0x7f94340b1bc0 2024-03-18T19:06:46.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.223+0000 7f94337fe700 1 --2- 172.21.15.1:0/2476319697 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f942006adb0 0x7f942006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9428000fb0 tx=0x7f9428009040 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:46.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.227+0000 7f941bfff700 1 -- 172.21.15.1:0/2476319697 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9424054d00 con 0x7f94340b1bc0 2024-03-18T19:06:46.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.426+0000 7f943a0b6700 1 -- 172.21.15.1:0/2476319697 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f94340d0680 con 0x7f942006adb0 2024-03-18T19:06:46.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.434+0000 7f941bfff700 1 -- 172.21.15.1:0/2476319697 <== mgr.7542 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290166 (secure 0 0 0) 0x7f94280070c0 con 0x7f942006adb0 2024-03-18T19:06:46.438 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:46.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.438+0000 7f943a0b6700 1 -- 172.21.15.1:0/2476319697 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f942006adb0 msgr2=0x7f942006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:46.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.438+0000 7f943a0b6700 1 --2- 172.21.15.1:0/2476319697 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f942006adb0 0x7f942006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9428000fb0 tx=0x7f9428009040 comp rx=0 tx=0).stop 2024-03-18T19:06:46.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.438+0000 7f943a0b6700 1 -- 172.21.15.1:0/2476319697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94340b1bc0 msgr2=0x7f943415c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:46.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.438+0000 7f943a0b6700 1 --2- 172.21.15.1:0/2476319697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94340b1bc0 0x7f943415c060 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f94240119a0 tx=0x7f9424011d60 comp rx=0 tx=0).stop 2024-03-18T19:06:46.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.438+0000 7f943a0b6700 1 -- 172.21.15.1:0/2476319697 shutdown_connections 2024-03-18T19:06:46.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.438+0000 7f943a0b6700 1 --2- 172.21.15.1:0/2476319697 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f942006adb0 0x7f942006d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.438+0000 7f943a0b6700 1 --2- 172.21.15.1:0/2476319697 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94340b2040 0x7f943415c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.438+0000 7f943a0b6700 1 --2- 172.21.15.1:0/2476319697 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f941c004d90 0x7f94340b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.438+0000 7f943a0b6700 1 --2- 172.21.15.1:0/2476319697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94340b1bc0 0x7f943415c060 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.438+0000 7f943a0b6700 1 -- 172.21.15.1:0/2476319697 >> 172.21.15.1:0/2476319697 conn(0x7f94340b2ba0 msgr2=0x7f94340cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:46.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.438+0000 7f943a0b6700 1 -- 172.21.15.1:0/2476319697 shutdown_connections 2024-03-18T19:06:46.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.438+0000 7f943a0b6700 1 -- 172.21.15.1:0/2476319697 wait complete. 2024-03-18T19:06:46.443 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T19:06:46.447 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-18T19:06:46.295725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":344264,"kb_used_data":12488,"kb_used_omap":12,"kb_used_meta":331635,"kb_avail":754630456,"statfs":{"total":773094113280,"available":772741586944,"internally_reserved":0,"allocated":12787712,"data_stored":6955796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":339594809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.831183"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.907413+0000","last_change":"2024-03-18T19:06:41.930967+0000","last_active":"2024-03-18T19:06:42.907413+0000","last_peered":"2024-03-18T19:06:42.907413+0000","last_clean":"2024-03-18T19:06:42.907413+0000","last_became_active":"2024-03-18T19:06:41.930698+0000","last_became_peered":"2024-03-18T19:06:41.930698+0000","last_unstale":"2024-03-18T19:06:42.907413+0000","last_undegraded":"2024-03-18T19:06:42.907413+0000","last_fullsized":"2024-03-18T19:06:42.907413+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:26:06.385768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225270+0000","last_change":"2024-03-18T19:06:41.922083+0000","last_active":"2024-03-18T19:06:43.225270+0000","last_peered":"2024-03-18T19:06:43.225270+0000","last_clean":"2024-03-18T19:06:43.225270+0000","last_became_active":"2024-03-18T19:06:41.921796+0000","last_became_peered":"2024-03-18T19:06:41.921796+0000","last_unstale":"2024-03-18T19:06:43.225270+0000","last_undegraded":"2024-03-18T19:06:43.225270+0000","last_fullsized":"2024-03-18T19:06:43.225270+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:59.618124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.907005+0000","last_change":"2024-03-18T19:06:41.923251+0000","last_active":"2024-03-18T19:06:42.907005+0000","last_peered":"2024-03-18T19:06:42.907005+0000","last_clean":"2024-03-18T19:06:42.907005+0000","last_became_active":"2024-03-18T19:06:41.922974+0000","last_became_peered":"2024-03-18T19:06:41.922974+0000","last_unstale":"2024-03-18T19:06:42.907005+0000","last_undegraded":"2024-03-18T19:06:42.907005+0000","last_fullsized":"2024-03-18T19:06:42.907005+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:40.590657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905955+0000","last_change":"2024-03-18T19:06:41.919454+0000","last_active":"2024-03-18T19:06:42.905955+0000","last_peered":"2024-03-18T19:06:42.905955+0000","last_clean":"2024-03-18T19:06:42.905955+0000","last_became_active":"2024-03-18T19:06:41.919193+0000","last_became_peered":"2024-03-18T19:06:41.919193+0000","last_unstale":"2024-03-18T19:06:42.905955+0000","last_undegraded":"2024-03-18T19:06:42.905955+0000","last_fullsized":"2024-03-18T19:06:42.905955+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:39.234562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224656+0000","last_change":"2024-03-18T19:06:41.916999+0000","last_active":"2024-03-18T19:06:43.224656+0000","last_peered":"2024-03-18T19:06:43.224656+0000","last_clean":"2024-03-18T19:06:43.224656+0000","last_became_active":"2024-03-18T19:06:41.916795+0000","last_became_peered":"2024-03-18T19:06:41.916795+0000","last_unstale":"2024-03-18T19:06:43.224656+0000","last_undegraded":"2024-03-18T19:06:43.224656+0000","last_fullsized":"2024-03-18T19:06:43.224656+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:54:16.529481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223732+0000","last_change":"2024-03-18T19:06:41.934105+0000","last_active":"2024-03-18T19:06:43.223732+0000","last_peered":"2024-03-18T19:06:43.223732+0000","last_clean":"2024-03-18T19:06:43.223732+0000","last_became_active":"2024-03-18T19:06:41.933704+0000","last_became_peered":"2024-03-18T19:06:41.933704+0000","last_unstale":"2024-03-18T19:06:43.223732+0000","last_undegraded":"2024-03-18T19:06:43.223732+0000","last_fullsized":"2024-03-18T19:06:43.223732+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:58:15.616500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105595+0000","last_change":"2024-03-18T19:06:41.929703+0000","last_active":"2024-03-18T19:06:43.105595+0000","last_peered":"2024-03-18T19:06:43.105595+0000","last_clean":"2024-03-18T19:06:43.105595+0000","last_became_active":"2024-03-18T19:06:41.929476+0000","last_became_peered":"2024-03-18T19:06:41.929476+0000","last_unstale":"2024-03-18T19:06:43.105595+0000","last_undegraded":"2024-03-18T19:06:43.105595+0000","last_fullsized":"2024-03-18T19:06:43.105595+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:00:03.205630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.904851+0000","last_change":"2024-03-18T19:06:41.916504+0000","last_active":"2024-03-18T19:06:42.904851+0000","last_peered":"2024-03-18T19:06:42.904851+0000","last_clean":"2024-03-18T19:06:42.904851+0000","last_became_active":"2024-03-18T19:06:41.916197+0000","last_became_peered":"2024-03-18T19:06:41.916197+0000","last_unstale":"2024-03-18T19:06:42.904851+0000","last_undegraded":"2024-03-18T19:06:42.904851+0000","last_fullsized":"2024-03-18T19:06:42.904851+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:35.228931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225029+0000","last_change":"2024-03-18T19:06:41.918686+0000","last_active":"2024-03-18T19:06:43.225029+0000","last_peered":"2024-03-18T19:06:43.225029+0000","last_clean":"2024-03-18T19:06:43.225029+0000","last_became_active":"2024-03-18T19:06:41.917900+0000","last_became_peered":"2024-03-18T19:06:41.917900+0000","last_unstale":"2024-03-18T19:06:43.225029+0000","last_undegraded":"2024-03-18T19:06:43.225029+0000","last_fullsized":"2024-03-18T19:06:43.225029+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:03:12.024061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224018+0000","last_change":"2024-03-18T19:06:41.924735+0000","last_active":"2024-03-18T19:06:43.224018+0000","last_peered":"2024-03-18T19:06:43.224018+0000","last_clean":"2024-03-18T19:06:43.224018+0000","last_became_active":"2024-03-18T19:06:41.924491+0000","last_became_peered":"2024-03-18T19:06:41.924491+0000","last_unstale":"2024-03-18T19:06:43.224018+0000","last_undegraded":"2024-03-18T19:06:43.224018+0000","last_fullsized":"2024-03-18T19:06:43.224018+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:52.692533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225628+0000","last_change":"2024-03-18T19:06:41.931128+0000","last_active":"2024-03-18T19:06:43.225628+0000","last_peered":"2024-03-18T19:06:43.225628+0000","last_clean":"2024-03-18T19:06:43.225628+0000","last_became_active":"2024-03-18T19:06:41.930870+0000","last_became_peered":"2024-03-18T19:06:41.930870+0000","last_unstale":"2024-03-18T19:06:43.225628+0000","last_undegraded":"2024-03-18T19:06:43.225628+0000","last_fullsized":"2024-03-18T19:06:43.225628+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:43:59.126342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223680+0000","last_change":"2024-03-18T19:06:41.927393+0000","last_active":"2024-03-18T19:06:43.223680+0000","last_peered":"2024-03-18T19:06:43.223680+0000","last_clean":"2024-03-18T19:06:43.223680+0000","last_became_active":"2024-03-18T19:06:41.927127+0000","last_became_peered":"2024-03-18T19:06:41.927127+0000","last_unstale":"2024-03-18T19:06:43.223680+0000","last_undegraded":"2024-03-18T19:06:43.223680+0000","last_fullsized":"2024-03-18T19:06:43.223680+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:56:42.842003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905262+0000","last_change":"2024-03-18T19:06:41.916683+0000","last_active":"2024-03-18T19:06:42.905262+0000","last_peered":"2024-03-18T19:06:42.905262+0000","last_clean":"2024-03-18T19:06:42.905262+0000","last_became_active":"2024-03-18T19:06:41.916410+0000","last_became_peered":"2024-03-18T19:06:41.916410+0000","last_unstale":"2024-03-18T19:06:42.905262+0000","last_undegraded":"2024-03-18T19:06:42.905262+0000","last_fullsized":"2024-03-18T19:06:42.905262+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:08.026486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224248+0000","last_change":"2024-03-18T19:06:41.925500+0000","last_active":"2024-03-18T19:06:43.224248+0000","last_peered":"2024-03-18T19:06:43.224248+0000","last_clean":"2024-03-18T19:06:43.224248+0000","last_became_active":"2024-03-18T19:06:41.924680+0000","last_became_peered":"2024-03-18T19:06:41.924680+0000","last_unstale":"2024-03-18T19:06:43.224248+0000","last_undegraded":"2024-03-18T19:06:43.224248+0000","last_fullsized":"2024-03-18T19:06:43.224248+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:20.035954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905061+0000","last_change":"2024-03-18T19:06:41.932201+0000","last_active":"2024-03-18T19:06:42.905061+0000","last_peered":"2024-03-18T19:06:42.905061+0000","last_clean":"2024-03-18T19:06:42.905061+0000","last_became_active":"2024-03-18T19:06:41.931948+0000","last_became_peered":"2024-03-18T19:06:41.931948+0000","last_unstale":"2024-03-18T19:06:42.905061+0000","last_undegraded":"2024-03-18T19:06:42.905061+0000","last_fullsized":"2024-03-18T19:06:42.905061+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:15.089015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224052+0000","last_change":"2024-03-18T19:06:41.930105+0000","last_active":"2024-03-18T19:06:43.224052+0000","last_peered":"2024-03-18T19:06:43.224052+0000","last_clean":"2024-03-18T19:06:43.224052+0000","last_became_active":"2024-03-18T19:06:41.929840+0000","last_became_peered":"2024-03-18T19:06:41.929840+0000","last_unstale":"2024-03-18T19:06:43.224052+0000","last_undegraded":"2024-03-18T19:06:43.224052+0000","last_fullsized":"2024-03-18T19:06:43.224052+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:50.694886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224144+0000","last_change":"2024-03-18T19:06:41.930527+0000","last_active":"2024-03-18T19:06:43.224144+0000","last_peered":"2024-03-18T19:06:43.224144+0000","last_clean":"2024-03-18T19:06:43.224144+0000","last_became_active":"2024-03-18T19:06:41.929968+0000","last_became_peered":"2024-03-18T19:06:41.929968+0000","last_unstale":"2024-03-18T19:06:43.224144+0000","last_undegraded":"2024-03-18T19:06:43.224144+0000","last_fullsized":"2024-03-18T19:06:43.224144+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:13.861260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.906947+0000","last_change":"2024-03-18T19:06:41.922331+0000","last_active":"2024-03-18T19:06:42.906947+0000","last_peered":"2024-03-18T19:06:42.906947+0000","last_clean":"2024-03-18T19:06:42.906947+0000","last_became_active":"2024-03-18T19:06:41.920647+0000","last_became_peered":"2024-03-18T19:06:41.920647+0000","last_unstale":"2024-03-18T19:06:42.906947+0000","last_undegraded":"2024-03-18T19:06:42.906947+0000","last_fullsized":"2024-03-18T19:06:42.906947+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:15.655985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225010+0000","last_change":"2024-03-18T19:06:41.924022+0000","last_active":"2024-03-18T19:06:43.225010+0000","last_peered":"2024-03-18T19:06:43.225010+0000","last_clean":"2024-03-18T19:06:43.225010+0000","last_became_active":"2024-03-18T19:06:41.923770+0000","last_became_peered":"2024-03-18T19:06:41.923770+0000","last_unstale":"2024-03-18T19:06:43.225010+0000","last_undegraded":"2024-03-18T19:06:43.225010+0000","last_fullsized":"2024-03-18T19:06:43.225010+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:25.618183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905694+0000","last_change":"2024-03-18T19:06:41.930985+0000","last_active":"2024-03-18T19:06:42.905694+0000","last_peered":"2024-03-18T19:06:42.905694+0000","last_clean":"2024-03-18T19:06:42.905694+0000","last_became_active":"2024-03-18T19:06:41.927789+0000","last_became_peered":"2024-03-18T19:06:41.927789+0000","last_unstale":"2024-03-18T19:06:42.905694+0000","last_undegraded":"2024-03-18T19:06:42.905694+0000","last_fullsized":"2024-03-18T19:06:42.905694+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:00:03.205630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224745+0000","last_change":"2024-03-18T19:06:41.921090+0000","last_active":"2024-03-18T19:06:43.224745+0000","last_peered":"2024-03-18T19:06:43.224745+0000","last_clean":"2024-03-18T19:06:43.224745+0000","last_became_active":"2024-03-18T19:06:41.920828+0000","last_became_peered":"2024-03-18T19:06:41.920828+0000","last_unstale":"2024-03-18T19:06:43.224745+0000","last_undegraded":"2024-03-18T19:06:43.224745+0000","last_fullsized":"2024-03-18T19:06:43.224745+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:59.618124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224259+0000","last_change":"2024-03-18T19:06:41.924664+0000","last_active":"2024-03-18T19:06:43.224259+0000","last_peered":"2024-03-18T19:06:43.224259+0000","last_clean":"2024-03-18T19:06:43.224259+0000","last_became_active":"2024-03-18T19:06:41.924341+0000","last_became_peered":"2024-03-18T19:06:41.924341+0000","last_unstale":"2024-03-18T19:06:43.224259+0000","last_undegraded":"2024-03-18T19:06:43.224259+0000","last_fullsized":"2024-03-18T19:06:43.224259+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:00:03.205630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225066+0000","last_change":"2024-03-18T19:06:41.919722+0000","last_active":"2024-03-18T19:06:43.225066+0000","last_peered":"2024-03-18T19:06:43.225066+0000","last_clean":"2024-03-18T19:06:43.225066+0000","last_became_active":"2024-03-18T19:06:41.919362+0000","last_became_peered":"2024-03-18T19:06:41.919362+0000","last_unstale":"2024-03-18T19:06:43.225066+0000","last_undegraded":"2024-03-18T19:06:43.225066+0000","last_fullsized":"2024-03-18T19:06:43.225066+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:11:40.736064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.906806+0000","last_change":"2024-03-18T19:06:41.923714+0000","last_active":"2024-03-18T19:06:42.906806+0000","last_peered":"2024-03-18T19:06:42.906806+0000","last_clean":"2024-03-18T19:06:42.906806+0000","last_became_active":"2024-03-18T19:06:41.923245+0000","last_became_peered":"2024-03-18T19:06:41.923245+0000","last_unstale":"2024-03-18T19:06:42.906806+0000","last_undegraded":"2024-03-18T19:06:42.906806+0000","last_fullsized":"2024-03-18T19:06:42.906806+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:04:37.360539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225614+0000","last_change":"2024-03-18T19:06:41.917276+0000","last_active":"2024-03-18T19:06:43.225614+0000","last_peered":"2024-03-18T19:06:43.225614+0000","last_clean":"2024-03-18T19:06:43.225614+0000","last_became_active":"2024-03-18T19:06:41.916964+0000","last_became_peered":"2024-03-18T19:06:41.916964+0000","last_unstale":"2024-03-18T19:06:43.225614+0000","last_undegraded":"2024-03-18T19:06:43.225614+0000","last_fullsized":"2024-03-18T19:06:43.225614+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:32:37.339641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225293+0000","last_change":"2024-03-18T19:06:41.925228+0000","last_active":"2024-03-18T19:06:43.225293+0000","last_peered":"2024-03-18T19:06:43.225293+0000","last_clean":"2024-03-18T19:06:43.225293+0000","last_became_active":"2024-03-18T19:06:41.924948+0000","last_became_peered":"2024-03-18T19:06:41.924948+0000","last_unstale":"2024-03-18T19:06:43.225293+0000","last_undegraded":"2024-03-18T19:06:43.225293+0000","last_fullsized":"2024-03-18T19:06:43.225293+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:42.413430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223916+0000","last_change":"2024-03-18T19:06:41.919765+0000","last_active":"2024-03-18T19:06:43.223916+0000","last_peered":"2024-03-18T19:06:43.223916+0000","last_clean":"2024-03-18T19:06:43.223916+0000","last_became_active":"2024-03-18T19:06:41.919424+0000","last_became_peered":"2024-03-18T19:06:41.919424+0000","last_unstale":"2024-03-18T19:06:43.223916+0000","last_undegraded":"2024-03-18T19:06:43.223916+0000","last_fullsized":"2024-03-18T19:06:43.223916+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:12:06.889612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905793+0000","last_change":"2024-03-18T19:06:41.923195+0000","last_active":"2024-03-18T19:06:42.905793+0000","last_peered":"2024-03-18T19:06:42.905793+0000","last_clean":"2024-03-18T19:06:42.905793+0000","last_became_active":"2024-03-18T19:06:41.922939+0000","last_became_peered":"2024-03-18T19:06:41.922939+0000","last_unstale":"2024-03-18T19:06:42.905793+0000","last_undegraded":"2024-03-18T19:06:42.905793+0000","last_fullsized":"2024-03-18T19:06:42.905793+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:42.413430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105540+0000","last_change":"2024-03-18T19:06:41.914798+0000","last_active":"2024-03-18T19:06:43.105540+0000","last_peered":"2024-03-18T19:06:43.105540+0000","last_clean":"2024-03-18T19:06:43.105540+0000","last_became_active":"2024-03-18T19:06:41.914524+0000","last_became_peered":"2024-03-18T19:06:41.914524+0000","last_unstale":"2024-03-18T19:06:43.105540+0000","last_undegraded":"2024-03-18T19:06:43.105540+0000","last_fullsized":"2024-03-18T19:06:43.105540+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:20.035954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224360+0000","last_change":"2024-03-18T19:06:41.914890+0000","last_active":"2024-03-18T19:06:43.224360+0000","last_peered":"2024-03-18T19:06:43.224360+0000","last_clean":"2024-03-18T19:06:43.224360+0000","last_became_active":"2024-03-18T19:06:41.914687+0000","last_became_peered":"2024-03-18T19:06:41.914687+0000","last_unstale":"2024-03-18T19:06:43.224360+0000","last_undegraded":"2024-03-18T19:06:43.224360+0000","last_fullsized":"2024-03-18T19:06:43.224360+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:15.089015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224321+0000","last_change":"2024-03-18T19:06:41.923914+0000","last_active":"2024-03-18T19:06:43.224321+0000","last_peered":"2024-03-18T19:06:43.224321+0000","last_clean":"2024-03-18T19:06:43.224321+0000","last_became_active":"2024-03-18T19:06:41.923644+0000","last_became_peered":"2024-03-18T19:06:41.923644+0000","last_unstale":"2024-03-18T19:06:43.224321+0000","last_undegraded":"2024-03-18T19:06:43.224321+0000","last_fullsized":"2024-03-18T19:06:43.224321+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:54:16.529481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224151+0000","last_change":"2024-03-18T19:06:41.929950+0000","last_active":"2024-03-18T19:06:43.224151+0000","last_peered":"2024-03-18T19:06:43.224151+0000","last_clean":"2024-03-18T19:06:43.224151+0000","last_became_active":"2024-03-18T19:06:41.929716+0000","last_became_peered":"2024-03-18T19:06:41.929716+0000","last_unstale":"2024-03-18T19:06:43.224151+0000","last_undegraded":"2024-03-18T19:06:43.224151+0000","last_fullsized":"2024-03-18T19:06:43.224151+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:24.975965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224350+0000","last_change":"2024-03-18T19:06:41.925883+0000","last_active":"2024-03-18T19:06:43.224350+0000","last_peered":"2024-03-18T19:06:43.224350+0000","last_clean":"2024-03-18T19:06:43.224350+0000","last_became_active":"2024-03-18T19:06:41.925612+0000","last_became_peered":"2024-03-18T19:06:41.925612+0000","last_unstale":"2024-03-18T19:06:43.224350+0000","last_undegraded":"2024-03-18T19:06:43.224350+0000","last_fullsized":"2024-03-18T19:06:43.224350+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:02:02.068675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"116'4","reported_seq":15,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225518+0000","last_change":"2024-03-18T19:06:41.493796+0000","last_active":"2024-03-18T19:06:43.225518+0000","last_peered":"2024-03-18T19:06:43.225518+0000","last_clean":"2024-03-18T19:06:43.225518+0000","last_became_active":"2024-03-18T19:06:41.493520+0000","last_became_peered":"2024-03-18T19:06:41.493520+0000","last_unstale":"2024-03-18T19:06:43.225518+0000","last_undegraded":"2024-03-18T19:06:43.225518+0000","last_fullsized":"2024-03-18T19:06:43.225518+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:33.022886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224437+0000","last_change":"2024-03-18T19:06:41.925698+0000","last_active":"2024-03-18T19:06:43.224437+0000","last_peered":"2024-03-18T19:06:43.224437+0000","last_clean":"2024-03-18T19:06:43.224437+0000","last_became_active":"2024-03-18T19:06:41.925244+0000","last_became_peered":"2024-03-18T19:06:41.925244+0000","last_unstale":"2024-03-18T19:06:43.224437+0000","last_undegraded":"2024-03-18T19:06:43.224437+0000","last_fullsized":"2024-03-18T19:06:43.224437+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:41.487390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"116'1","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225385+0000","last_change":"2024-03-18T19:06:41.482920+0000","last_active":"2024-03-18T19:06:43.225385+0000","last_peered":"2024-03-18T19:06:43.225385+0000","last_clean":"2024-03-18T19:06:43.225385+0000","last_became_active":"2024-03-18T19:06:41.482612+0000","last_became_peered":"2024-03-18T19:06:41.482612+0000","last_unstale":"2024-03-18T19:06:43.225385+0000","last_undegraded":"2024-03-18T19:06:43.225385+0000","last_fullsized":"2024-03-18T19:06:43.225385+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:25.756251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105617+0000","last_change":"2024-03-18T19:06:41.920806+0000","last_active":"2024-03-18T19:06:43.105617+0000","last_peered":"2024-03-18T19:06:43.105617+0000","last_clean":"2024-03-18T19:06:43.105617+0000","last_became_active":"2024-03-18T19:06:41.920541+0000","last_became_peered":"2024-03-18T19:06:41.920541+0000","last_unstale":"2024-03-18T19:06:43.105617+0000","last_undegraded":"2024-03-18T19:06:43.105617+0000","last_fullsized":"2024-03-18T19:06:43.105617+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:42.413430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"116'2","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105739+0000","last_change":"2024-03-18T19:06:41.487514+0000","last_active":"2024-03-18T19:06:43.105739+0000","last_peered":"2024-03-18T19:06:43.105739+0000","last_clean":"2024-03-18T19:06:43.105739+0000","last_became_active":"2024-03-18T19:06:41.487253+0000","last_became_peered":"2024-03-18T19:06:41.487253+0000","last_unstale":"2024-03-18T19:06:43.105739+0000","last_undegraded":"2024-03-18T19:06:43.105739+0000","last_fullsized":"2024-03-18T19:06:43.105739+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:10.908853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224243+0000","last_change":"2024-03-18T19:06:41.930183+0000","last_active":"2024-03-18T19:06:43.224243+0000","last_peered":"2024-03-18T19:06:43.224243+0000","last_clean":"2024-03-18T19:06:43.224243+0000","last_became_active":"2024-03-18T19:06:41.929863+0000","last_became_peered":"2024-03-18T19:06:41.929863+0000","last_unstale":"2024-03-18T19:06:43.224243+0000","last_undegraded":"2024-03-18T19:06:43.224243+0000","last_fullsized":"2024-03-18T19:06:43.224243+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:25:37.918166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"116'2","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225278+0000","last_change":"2024-03-18T19:06:41.486199+0000","last_active":"2024-03-18T19:06:43.225278+0000","last_peered":"2024-03-18T19:06:43.225278+0000","last_clean":"2024-03-18T19:06:43.225278+0000","last_became_active":"2024-03-18T19:06:41.485977+0000","last_became_peered":"2024-03-18T19:06:41.485977+0000","last_unstale":"2024-03-18T19:06:43.225278+0000","last_undegraded":"2024-03-18T19:06:43.225278+0000","last_fullsized":"2024-03-18T19:06:43.225278+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:46.041073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.904938+0000","last_change":"2024-03-18T19:06:41.918905+0000","last_active":"2024-03-18T19:06:42.904938+0000","last_peered":"2024-03-18T19:06:42.904938+0000","last_clean":"2024-03-18T19:06:42.904938+0000","last_became_active":"2024-03-18T19:06:41.918656+0000","last_became_peered":"2024-03-18T19:06:41.918656+0000","last_unstale":"2024-03-18T19:06:42.904938+0000","last_undegraded":"2024-03-18T19:06:42.904938+0000","last_fullsized":"2024-03-18T19:06:42.904938+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:02.450766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"116'1","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224762+0000","last_change":"2024-03-18T19:06:41.483351+0000","last_active":"2024-03-18T19:06:43.224762+0000","last_peered":"2024-03-18T19:06:43.224762+0000","last_clean":"2024-03-18T19:06:43.224762+0000","last_became_active":"2024-03-18T19:06:41.483076+0000","last_became_peered":"2024-03-18T19:06:41.483076+0000","last_unstale":"2024-03-18T19:06:43.224762+0000","last_undegraded":"2024-03-18T19:06:43.224762+0000","last_fullsized":"2024-03-18T19:06:43.224762+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:49:34.183987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225177+0000","last_change":"2024-03-18T19:06:41.920874+0000","last_active":"2024-03-18T19:06:43.225177+0000","last_peered":"2024-03-18T19:06:43.225177+0000","last_clean":"2024-03-18T19:06:43.225177+0000","last_became_active":"2024-03-18T19:06:41.919913+0000","last_became_peered":"2024-03-18T19:06:41.919913+0000","last_unstale":"2024-03-18T19:06:43.225177+0000","last_undegraded":"2024-03-18T19:06:43.225177+0000","last_fullsized":"2024-03-18T19:06:43.225177+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:11.914183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"116'1","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905769+0000","last_change":"2024-03-18T19:06:41.485060+0000","last_active":"2024-03-18T19:06:42.905769+0000","last_peered":"2024-03-18T19:06:42.905769+0000","last_clean":"2024-03-18T19:06:42.905769+0000","last_became_active":"2024-03-18T19:06:41.484778+0000","last_became_peered":"2024-03-18T19:06:41.484778+0000","last_unstale":"2024-03-18T19:06:42.905769+0000","last_undegraded":"2024-03-18T19:06:42.905769+0000","last_fullsized":"2024-03-18T19:06:42.905769+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:24.017447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223815+0000","last_change":"2024-03-18T19:06:41.933950+0000","last_active":"2024-03-18T19:06:43.223815+0000","last_peered":"2024-03-18T19:06:43.223815+0000","last_clean":"2024-03-18T19:06:43.223815+0000","last_became_active":"2024-03-18T19:06:41.933411+0000","last_became_peered":"2024-03-18T19:06:41.933411+0000","last_unstale":"2024-03-18T19:06:43.223815+0000","last_undegraded":"2024-03-18T19:06:43.223815+0000","last_fullsized":"2024-03-18T19:06:43.223815+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:18:14.637126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.906967+0000","last_change":"2024-03-18T19:06:41.482438+0000","last_active":"2024-03-18T19:06:42.906967+0000","last_peered":"2024-03-18T19:06:42.906967+0000","last_clean":"2024-03-18T19:06:42.906967+0000","last_became_active":"2024-03-18T19:06:41.482139+0000","last_became_peered":"2024-03-18T19:06:41.482139+0000","last_unstale":"2024-03-18T19:06:42.906967+0000","last_undegraded":"2024-03-18T19:06:42.906967+0000","last_fullsized":"2024-03-18T19:06:42.906967+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:15:01.147997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905358+0000","last_change":"2024-03-18T19:06:41.918370+0000","last_active":"2024-03-18T19:06:42.905358+0000","last_peered":"2024-03-18T19:06:42.905358+0000","last_clean":"2024-03-18T19:06:42.905358+0000","last_became_active":"2024-03-18T19:06:41.917142+0000","last_became_peered":"2024-03-18T19:06:41.917142+0000","last_unstale":"2024-03-18T19:06:42.905358+0000","last_undegraded":"2024-03-18T19:06:42.905358+0000","last_fullsized":"2024-03-18T19:06:42.905358+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:59.618124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105424+0000","last_change":"2024-03-18T19:06:41.494458+0000","last_active":"2024-03-18T19:06:43.105424+0000","last_peered":"2024-03-18T19:06:43.105424+0000","last_clean":"2024-03-18T19:06:43.105424+0000","last_became_active":"2024-03-18T19:06:41.494280+0000","last_became_peered":"2024-03-18T19:06:41.494280+0000","last_unstale":"2024-03-18T19:06:43.105424+0000","last_undegraded":"2024-03-18T19:06:43.105424+0000","last_fullsized":"2024-03-18T19:06:43.105424+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:32:36.334311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225865+0000","last_change":"2024-03-18T19:06:41.927370+0000","last_active":"2024-03-18T19:06:43.225865+0000","last_peered":"2024-03-18T19:06:43.225865+0000","last_clean":"2024-03-18T19:06:43.225865+0000","last_became_active":"2024-03-18T19:06:41.927131+0000","last_became_peered":"2024-03-18T19:06:41.927131+0000","last_unstale":"2024-03-18T19:06:43.225865+0000","last_undegraded":"2024-03-18T19:06:43.225865+0000","last_fullsized":"2024-03-18T19:06:43.225865+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:39.234562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225752+0000","last_change":"2024-03-18T19:06:41.490868+0000","last_active":"2024-03-18T19:06:43.225752+0000","last_peered":"2024-03-18T19:06:43.225752+0000","last_clean":"2024-03-18T19:06:43.225752+0000","last_became_active":"2024-03-18T19:06:41.490615+0000","last_became_peered":"2024-03-18T19:06:41.490615+0000","last_unstale":"2024-03-18T19:06:43.225752+0000","last_undegraded":"2024-03-18T19:06:43.225752+0000","last_fullsized":"2024-03-18T19:06:43.225752+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:25.756251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105719+0000","last_change":"2024-03-18T19:06:41.921302+0000","last_active":"2024-03-18T19:06:43.105719+0000","last_peered":"2024-03-18T19:06:43.105719+0000","last_clean":"2024-03-18T19:06:43.105719+0000","last_became_active":"2024-03-18T19:06:41.921014+0000","last_became_peered":"2024-03-18T19:06:41.921014+0000","last_unstale":"2024-03-18T19:06:43.105719+0000","last_undegraded":"2024-03-18T19:06:43.105719+0000","last_fullsized":"2024-03-18T19:06:43.105719+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:15.089015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.907188+0000","last_change":"2024-03-18T19:06:41.485189+0000","last_active":"2024-03-18T19:06:42.907188+0000","last_peered":"2024-03-18T19:06:42.907188+0000","last_clean":"2024-03-18T19:06:42.907188+0000","last_became_active":"2024-03-18T19:06:41.484929+0000","last_became_peered":"2024-03-18T19:06:41.484929+0000","last_unstale":"2024-03-18T19:06:42.907188+0000","last_undegraded":"2024-03-18T19:06:42.907188+0000","last_fullsized":"2024-03-18T19:06:42.907188+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:20:43.467362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.907046+0000","last_change":"2024-03-18T19:06:41.921920+0000","last_active":"2024-03-18T19:06:42.907046+0000","last_peered":"2024-03-18T19:06:42.907046+0000","last_clean":"2024-03-18T19:06:42.907046+0000","last_became_active":"2024-03-18T19:06:41.921680+0000","last_became_peered":"2024-03-18T19:06:41.921680+0000","last_unstale":"2024-03-18T19:06:42.907046+0000","last_undegraded":"2024-03-18T19:06:42.907046+0000","last_fullsized":"2024-03-18T19:06:42.907046+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:10:31.125315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"116'3","reported_seq":14,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225102+0000","last_change":"2024-03-18T19:06:41.493620+0000","last_active":"2024-03-18T19:06:43.225102+0000","last_peered":"2024-03-18T19:06:43.225102+0000","last_clean":"2024-03-18T19:06:43.225102+0000","last_became_active":"2024-03-18T19:06:41.493271+0000","last_became_peered":"2024-03-18T19:06:41.493271+0000","last_unstale":"2024-03-18T19:06:43.225102+0000","last_undegraded":"2024-03-18T19:06:43.225102+0000","last_fullsized":"2024-03-18T19:06:43.225102+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:05.738508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905389+0000","last_change":"2024-03-18T19:06:41.918139+0000","last_active":"2024-03-18T19:06:42.905389+0000","last_peered":"2024-03-18T19:06:42.905389+0000","last_clean":"2024-03-18T19:06:42.905389+0000","last_became_active":"2024-03-18T19:06:41.917151+0000","last_became_peered":"2024-03-18T19:06:41.917151+0000","last_unstale":"2024-03-18T19:06:42.905389+0000","last_undegraded":"2024-03-18T19:06:42.905389+0000","last_fullsized":"2024-03-18T19:06:42.905389+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:20.035954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"116'6","reported_seq":17,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224394+0000","last_change":"2024-03-18T19:06:41.492273+0000","last_active":"2024-03-18T19:06:43.224394+0000","last_peered":"2024-03-18T19:06:43.224394+0000","last_clean":"2024-03-18T19:06:43.224394+0000","last_became_active":"2024-03-18T19:06:41.492024+0000","last_became_peered":"2024-03-18T19:06:41.492024+0000","last_unstale":"2024-03-18T19:06:43.224394+0000","last_undegraded":"2024-03-18T19:06:43.224394+0000","last_fullsized":"2024-03-18T19:06:43.224394+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:36:10.016972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905031+0000","last_change":"2024-03-18T19:06:41.916297+0000","last_active":"2024-03-18T19:06:42.905031+0000","last_peered":"2024-03-18T19:06:42.905031+0000","last_clean":"2024-03-18T19:06:42.905031+0000","last_became_active":"2024-03-18T19:06:41.916044+0000","last_became_peered":"2024-03-18T19:06:41.916044+0000","last_unstale":"2024-03-18T19:06:42.905031+0000","last_undegraded":"2024-03-18T19:06:42.905031+0000","last_fullsized":"2024-03-18T19:06:42.905031+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:54:16.529481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"116'2","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105804+0000","last_change":"2024-03-18T19:06:41.494026+0000","last_active":"2024-03-18T19:06:43.105804+0000","last_peered":"2024-03-18T19:06:43.105804+0000","last_clean":"2024-03-18T19:06:43.105804+0000","last_became_active":"2024-03-18T19:06:41.493749+0000","last_became_peered":"2024-03-18T19:06:41.493749+0000","last_unstale":"2024-03-18T19:06:43.105804+0000","last_undegraded":"2024-03-18T19:06:43.105804+0000","last_fullsized":"2024-03-18T19:06:43.105804+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:22.727166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224167+0000","last_change":"2024-03-18T19:06:41.917535+0000","last_active":"2024-03-18T19:06:43.224167+0000","last_peered":"2024-03-18T19:06:43.224167+0000","last_clean":"2024-03-18T19:06:43.224167+0000","last_became_active":"2024-03-18T19:06:41.917216+0000","last_became_peered":"2024-03-18T19:06:41.917216+0000","last_unstale":"2024-03-18T19:06:43.224167+0000","last_undegraded":"2024-03-18T19:06:43.224167+0000","last_fullsized":"2024-03-18T19:06:43.224167+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:33:22.354647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223830+0000","last_change":"2024-03-18T19:06:41.491449+0000","last_active":"2024-03-18T19:06:43.223830+0000","last_peered":"2024-03-18T19:06:43.223830+0000","last_clean":"2024-03-18T19:06:43.223830+0000","last_became_active":"2024-03-18T19:06:41.490953+0000","last_became_peered":"2024-03-18T19:06:41.490953+0000","last_unstale":"2024-03-18T19:06:43.223830+0000","last_undegraded":"2024-03-18T19:06:43.223830+0000","last_fullsized":"2024-03-18T19:06:43.223830+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:51:06.462173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223673+0000","last_change":"2024-03-18T19:06:41.924070+0000","last_active":"2024-03-18T19:06:43.223673+0000","last_peered":"2024-03-18T19:06:43.223673+0000","last_clean":"2024-03-18T19:06:43.223673+0000","last_became_active":"2024-03-18T19:06:41.923787+0000","last_became_peered":"2024-03-18T19:06:41.923787+0000","last_unstale":"2024-03-18T19:06:43.223673+0000","last_undegraded":"2024-03-18T19:06:43.223673+0000","last_fullsized":"2024-03-18T19:06:43.223673+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:19.898810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905160+0000","last_change":"2024-03-18T19:06:41.487812+0000","last_active":"2024-03-18T19:06:42.905160+0000","last_peered":"2024-03-18T19:06:42.905160+0000","last_clean":"2024-03-18T19:06:42.905160+0000","last_became_active":"2024-03-18T19:06:41.485619+0000","last_became_peered":"2024-03-18T19:06:41.485619+0000","last_unstale":"2024-03-18T19:06:42.905160+0000","last_undegraded":"2024-03-18T19:06:42.905160+0000","last_fullsized":"2024-03-18T19:06:42.905160+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:05.738508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224449+0000","last_change":"2024-03-18T19:06:41.919157+0000","last_active":"2024-03-18T19:06:43.224449+0000","last_peered":"2024-03-18T19:06:43.224449+0000","last_clean":"2024-03-18T19:06:43.224449+0000","last_became_active":"2024-03-18T19:06:41.918916+0000","last_became_peered":"2024-03-18T19:06:41.918916+0000","last_unstale":"2024-03-18T19:06:43.224449+0000","last_undegraded":"2024-03-18T19:06:43.224449+0000","last_fullsized":"2024-03-18T19:06:43.224449+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:42.679949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"116'8","reported_seq":19,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.905036+0000","last_change":"2024-03-18T19:06:41.488447+0000","last_active":"2024-03-18T19:06:42.905036+0000","last_peered":"2024-03-18T19:06:42.905036+0000","last_clean":"2024-03-18T19:06:42.905036+0000","last_became_active":"2024-03-18T19:06:41.485582+0000","last_became_peered":"2024-03-18T19:06:41.485582+0000","last_unstale":"2024-03-18T19:06:42.905036+0000","last_undegraded":"2024-03-18T19:06:42.905036+0000","last_fullsized":"2024-03-18T19:06:42.905036+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:25.756251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224301+0000","last_change":"2024-03-18T19:06:41.929574+0000","last_active":"2024-03-18T19:06:43.224301+0000","last_peered":"2024-03-18T19:06:43.224301+0000","last_clean":"2024-03-18T19:06:43.224301+0000","last_became_active":"2024-03-18T19:06:41.929297+0000","last_became_peered":"2024-03-18T19:06:41.929297+0000","last_unstale":"2024-03-18T19:06:43.224301+0000","last_undegraded":"2024-03-18T19:06:43.224301+0000","last_fullsized":"2024-03-18T19:06:43.224301+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:46:28.072024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"116'1","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225616+0000","last_change":"2024-03-18T19:06:41.475897+0000","last_active":"2024-03-18T19:06:43.225616+0000","last_peered":"2024-03-18T19:06:43.225616+0000","last_clean":"2024-03-18T19:06:43.225616+0000","last_became_active":"2024-03-18T19:06:41.475629+0000","last_became_peered":"2024-03-18T19:06:41.475629+0000","last_unstale":"2024-03-18T19:06:43.225616+0000","last_undegraded":"2024-03-18T19:06:43.225616+0000","last_fullsized":"2024-03-18T19:06:43.225616+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:47:27.561126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225436+0000","last_change":"2024-03-18T19:06:41.915032+0000","last_active":"2024-03-18T19:06:43.225436+0000","last_peered":"2024-03-18T19:06:43.225436+0000","last_clean":"2024-03-18T19:06:43.225436+0000","last_became_active":"2024-03-18T19:06:41.914759+0000","last_became_peered":"2024-03-18T19:06:41.914759+0000","last_unstale":"2024-03-18T19:06:43.225436+0000","last_undegraded":"2024-03-18T19:06:43.225436+0000","last_fullsized":"2024-03-18T19:06:43.225436+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:27:17.972217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"116'2","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105743+0000","last_change":"2024-03-18T19:06:41.481574+0000","last_active":"2024-03-18T19:06:43.105743+0000","last_peered":"2024-03-18T19:06:43.105743+0000","last_clean":"2024-03-18T19:06:43.105743+0000","last_became_active":"2024-03-18T19:06:41.481320+0000","last_became_peered":"2024-03-18T19:06:41.481320+0000","last_unstale":"2024-03-18T19:06:43.105743+0000","last_undegraded":"2024-03-18T19:06:43.105743+0000","last_fullsized":"2024-03-18T19:06:43.105743+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:11:39.730734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225186+0000","last_change":"2024-03-18T19:06:41.918403+0000","last_active":"2024-03-18T19:06:43.225186+0000","last_peered":"2024-03-18T19:06:43.225186+0000","last_clean":"2024-03-18T19:06:43.225186+0000","last_became_active":"2024-03-18T19:06:41.918140+0000","last_became_peered":"2024-03-18T19:06:41.918140+0000","last_unstale":"2024-03-18T19:06:43.225186+0000","last_undegraded":"2024-03-18T19:06:43.225186+0000","last_fullsized":"2024-03-18T19:06:43.225186+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:44.947038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"116'1","reported_seq":12,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224901+0000","last_change":"2024-03-18T19:06:41.485030+0000","last_active":"2024-03-18T19:06:43.224901+0000","last_peered":"2024-03-18T19:06:43.224901+0000","last_clean":"2024-03-18T19:06:43.224901+0000","last_became_active":"2024-03-18T19:06:41.484777+0000","last_became_peered":"2024-03-18T19:06:41.484777+0000","last_unstale":"2024-03-18T19:06:43.224901+0000","last_undegraded":"2024-03-18T19:06:43.224901+0000","last_fullsized":"2024-03-18T19:06:43.224901+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:57.929295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224245+0000","last_change":"2024-03-18T19:06:41.933603+0000","last_active":"2024-03-18T19:06:43.224245+0000","last_peered":"2024-03-18T19:06:43.224245+0000","last_clean":"2024-03-18T19:06:43.224245+0000","last_became_active":"2024-03-18T19:06:41.933366+0000","last_became_peered":"2024-03-18T19:06:41.933366+0000","last_unstale":"2024-03-18T19:06:43.224245+0000","last_undegraded":"2024-03-18T19:06:43.224245+0000","last_fullsized":"2024-03-18T19:06:43.224245+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:36.486482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"116'2","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224327+0000","last_change":"2024-03-18T19:06:41.485418+0000","last_active":"2024-03-18T19:06:43.224327+0000","last_peered":"2024-03-18T19:06:43.224327+0000","last_clean":"2024-03-18T19:06:43.224327+0000","last_became_active":"2024-03-18T19:06:41.482735+0000","last_became_peered":"2024-03-18T19:06:41.482735+0000","last_unstale":"2024-03-18T19:06:43.224327+0000","last_undegraded":"2024-03-18T19:06:43.224327+0000","last_fullsized":"2024-03-18T19:06:43.224327+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:05.738508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224395+0000","last_change":"2024-03-18T19:06:41.922811+0000","last_active":"2024-03-18T19:06:43.224395+0000","last_peered":"2024-03-18T19:06:43.224395+0000","last_clean":"2024-03-18T19:06:43.224395+0000","last_became_active":"2024-03-18T19:06:41.922520+0000","last_became_peered":"2024-03-18T19:06:41.922520+0000","last_unstale":"2024-03-18T19:06:43.224395+0000","last_undegraded":"2024-03-18T19:06:43.224395+0000","last_fullsized":"2024-03-18T19:06:43.224395+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:00:42.923049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223769+0000","last_change":"2024-03-18T19:06:41.488673+0000","last_active":"2024-03-18T19:06:43.223769+0000","last_peered":"2024-03-18T19:06:43.223769+0000","last_clean":"2024-03-18T19:06:43.223769+0000","last_became_active":"2024-03-18T19:06:41.488381+0000","last_became_peered":"2024-03-18T19:06:41.488381+0000","last_unstale":"2024-03-18T19:06:43.223769+0000","last_undegraded":"2024-03-18T19:06:43.223769+0000","last_fullsized":"2024-03-18T19:06:43.223769+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:14:33.192189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224503+0000","last_change":"2024-03-18T19:06:41.923998+0000","last_active":"2024-03-18T19:06:43.224503+0000","last_peered":"2024-03-18T19:06:43.224503+0000","last_clean":"2024-03-18T19:06:43.224503+0000","last_became_active":"2024-03-18T19:06:41.923718+0000","last_became_peered":"2024-03-18T19:06:41.923718+0000","last_unstale":"2024-03-18T19:06:43.224503+0000","last_undegraded":"2024-03-18T19:06:43.224503+0000","last_fullsized":"2024-03-18T19:06:43.224503+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:08.026486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224451+0000","last_change":"2024-03-18T19:06:41.493551+0000","last_active":"2024-03-18T19:06:43.224451+0000","last_peered":"2024-03-18T19:06:43.224451+0000","last_clean":"2024-03-18T19:06:43.224451+0000","last_became_active":"2024-03-18T19:06:41.493258+0000","last_became_peered":"2024-03-18T19:06:41.493258+0000","last_unstale":"2024-03-18T19:06:43.224451+0000","last_undegraded":"2024-03-18T19:06:43.224451+0000","last_fullsized":"2024-03-18T19:06:43.224451+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:57.929295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224092+0000","last_change":"2024-03-18T19:06:41.925447+0000","last_active":"2024-03-18T19:06:43.224092+0000","last_peered":"2024-03-18T19:06:43.224092+0000","last_clean":"2024-03-18T19:06:43.224092+0000","last_became_active":"2024-03-18T19:06:41.925163+0000","last_became_peered":"2024-03-18T19:06:41.925163+0000","last_unstale":"2024-03-18T19:06:43.224092+0000","last_undegraded":"2024-03-18T19:06:43.224092+0000","last_fullsized":"2024-03-18T19:06:43.224092+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:06:46.759884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223645+0000","last_change":"2024-03-18T19:06:41.488766+0000","last_active":"2024-03-18T19:06:43.223645+0000","last_peered":"2024-03-18T19:06:43.223645+0000","last_clean":"2024-03-18T19:06:43.223645+0000","last_became_active":"2024-03-18T19:06:41.488487+0000","last_became_peered":"2024-03-18T19:06:41.488487+0000","last_unstale":"2024-03-18T19:06:43.223645+0000","last_undegraded":"2024-03-18T19:06:43.223645+0000","last_fullsized":"2024-03-18T19:06:43.223645+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:36.915166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225144+0000","last_change":"2024-03-18T19:06:41.913344+0000","last_active":"2024-03-18T19:06:43.225144+0000","last_peered":"2024-03-18T19:06:43.225144+0000","last_clean":"2024-03-18T19:06:43.225144+0000","last_became_active":"2024-03-18T19:06:41.913075+0000","last_became_peered":"2024-03-18T19:06:41.913075+0000","last_unstale":"2024-03-18T19:06:43.225144+0000","last_undegraded":"2024-03-18T19:06:43.225144+0000","last_fullsized":"2024-03-18T19:06:43.225144+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:43.067283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"116'2","reported_seq":13,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105763+0000","last_change":"2024-03-18T19:06:41.482479+0000","last_active":"2024-03-18T19:06:43.105763+0000","last_peered":"2024-03-18T19:06:43.105763+0000","last_clean":"2024-03-18T19:06:43.105763+0000","last_became_active":"2024-03-18T19:06:41.482193+0000","last_became_peered":"2024-03-18T19:06:41.482193+0000","last_unstale":"2024-03-18T19:06:43.105763+0000","last_undegraded":"2024-03-18T19:06:43.105763+0000","last_fullsized":"2024-03-18T19:06:43.105763+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:09:18.165390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224456+0000","last_change":"2024-03-18T19:06:41.925518+0000","last_active":"2024-03-18T19:06:43.224456+0000","last_peered":"2024-03-18T19:06:43.224456+0000","last_clean":"2024-03-18T19:06:43.224456+0000","last_became_active":"2024-03-18T19:06:41.924890+0000","last_became_peered":"2024-03-18T19:06:41.924890+0000","last_unstale":"2024-03-18T19:06:43.224456+0000","last_undegraded":"2024-03-18T19:06:43.224456+0000","last_fullsized":"2024-03-18T19:06:43.224456+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:33:44.303133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225477+0000","last_change":"2024-03-18T19:06:41.483079+0000","last_active":"2024-03-18T19:06:43.225477+0000","last_peered":"2024-03-18T19:06:43.225477+0000","last_clean":"2024-03-18T19:06:43.225477+0000","last_became_active":"2024-03-18T19:06:41.481825+0000","last_became_peered":"2024-03-18T19:06:41.481825+0000","last_unstale":"2024-03-18T19:06:43.225477+0000","last_undegraded":"2024-03-18T19:06:43.225477+0000","last_fullsized":"2024-03-18T19:06:43.225477+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:47:27.561126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223967+0000","last_change":"2024-03-18T19:06:41.927486+0000","last_active":"2024-03-18T19:06:43.223967+0000","last_peered":"2024-03-18T19:06:43.223967+0000","last_clean":"2024-03-18T19:06:43.223967+0000","last_became_active":"2024-03-18T19:06:41.927238+0000","last_became_peered":"2024-03-18T19:06:41.927238+0000","last_unstale":"2024-03-18T19:06:43.223967+0000","last_undegraded":"2024-03-18T19:06:43.223967+0000","last_fullsized":"2024-03-18T19:06:43.223967+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:16:29.079580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225182+0000","last_change":"2024-03-18T19:06:41.488337+0000","last_active":"2024-03-18T19:06:43.225182+0000","last_peered":"2024-03-18T19:06:43.225182+0000","last_clean":"2024-03-18T19:06:43.225182+0000","last_became_active":"2024-03-18T19:06:41.488126+0000","last_became_peered":"2024-03-18T19:06:41.488126+0000","last_unstale":"2024-03-18T19:06:43.225182+0000","last_undegraded":"2024-03-18T19:06:43.225182+0000","last_fullsized":"2024-03-18T19:06:43.225182+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:24.017447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225713+0000","last_change":"2024-03-18T19:06:41.918949+0000","last_active":"2024-03-18T19:06:43.225713+0000","last_peered":"2024-03-18T19:06:43.225713+0000","last_clean":"2024-03-18T19:06:43.225713+0000","last_became_active":"2024-03-18T19:06:41.918549+0000","last_became_peered":"2024-03-18T19:06:41.918549+0000","last_unstale":"2024-03-18T19:06:43.225713+0000","last_undegraded":"2024-03-18T19:06:43.225713+0000","last_fullsized":"2024-03-18T19:06:43.225713+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:23.732496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225376+0000","last_change":"2024-03-18T19:06:41.481513+0000","last_active":"2024-03-18T19:06:43.225376+0000","last_peered":"2024-03-18T19:06:43.225376+0000","last_clean":"2024-03-18T19:06:43.225376+0000","last_became_active":"2024-03-18T19:06:41.481248+0000","last_became_peered":"2024-03-18T19:06:41.481248+0000","last_unstale":"2024-03-18T19:06:43.225376+0000","last_undegraded":"2024-03-18T19:06:43.225376+0000","last_fullsized":"2024-03-18T19:06:43.225376+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:13:22.200232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.223591+0000","last_change":"2024-03-18T19:06:41.927083+0000","last_active":"2024-03-18T19:06:43.223591+0000","last_peered":"2024-03-18T19:06:43.223591+0000","last_clean":"2024-03-18T19:06:43.223591+0000","last_became_active":"2024-03-18T19:06:41.926671+0000","last_became_peered":"2024-03-18T19:06:41.926671+0000","last_unstale":"2024-03-18T19:06:43.223591+0000","last_undegraded":"2024-03-18T19:06:43.223591+0000","last_fullsized":"2024-03-18T19:06:43.223591+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:05.216483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224237+0000","last_change":"2024-03-18T19:06:41.490696+0000","last_active":"2024-03-18T19:06:43.224237+0000","last_peered":"2024-03-18T19:06:43.224237+0000","last_clean":"2024-03-18T19:06:43.224237+0000","last_became_active":"2024-03-18T19:06:41.490438+0000","last_became_peered":"2024-03-18T19:06:41.490438+0000","last_unstale":"2024-03-18T19:06:43.224237+0000","last_undegraded":"2024-03-18T19:06:43.224237+0000","last_fullsized":"2024-03-18T19:06:43.224237+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:36.889621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.906913+0000","last_change":"2024-03-18T19:06:41.924694+0000","last_active":"2024-03-18T19:06:42.906913+0000","last_peered":"2024-03-18T19:06:42.906913+0000","last_clean":"2024-03-18T19:06:42.906913+0000","last_became_active":"2024-03-18T19:06:41.924446+0000","last_became_peered":"2024-03-18T19:06:41.924446+0000","last_unstale":"2024-03-18T19:06:42.906913+0000","last_undegraded":"2024-03-18T19:06:42.906913+0000","last_fullsized":"2024-03-18T19:06:42.906913+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:45.399156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225520+0000","last_change":"2024-03-18T19:06:41.482677+0000","last_active":"2024-03-18T19:06:43.225520+0000","last_peered":"2024-03-18T19:06:43.225520+0000","last_clean":"2024-03-18T19:06:43.225520+0000","last_became_active":"2024-03-18T19:06:41.482356+0000","last_became_peered":"2024-03-18T19:06:41.482356+0000","last_unstale":"2024-03-18T19:06:43.225520+0000","last_undegraded":"2024-03-18T19:06:43.225520+0000","last_fullsized":"2024-03-18T19:06:43.225520+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:42.061953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224815+0000","last_change":"2024-03-18T19:06:41.913424+0000","last_active":"2024-03-18T19:06:43.224815+0000","last_peered":"2024-03-18T19:06:43.224815+0000","last_clean":"2024-03-18T19:06:43.224815+0000","last_became_active":"2024-03-18T19:06:41.913150+0000","last_became_peered":"2024-03-18T19:06:41.913150+0000","last_unstale":"2024-03-18T19:06:43.224815+0000","last_undegraded":"2024-03-18T19:06:43.224815+0000","last_fullsized":"2024-03-18T19:06:43.224815+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:09:21.138871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.225277+0000","last_change":"2024-03-18T19:06:41.484792+0000","last_active":"2024-03-18T19:06:43.225277+0000","last_peered":"2024-03-18T19:06:43.225277+0000","last_clean":"2024-03-18T19:06:43.225277+0000","last_became_active":"2024-03-18T19:06:41.484500+0000","last_became_peered":"2024-03-18T19:06:41.484500+0000","last_unstale":"2024-03-18T19:06:43.225277+0000","last_undegraded":"2024-03-18T19:06:43.225277+0000","last_fullsized":"2024-03-18T19:06:43.225277+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:10.069476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.224190+0000","last_change":"2024-03-18T19:06:41.930511+0000","last_active":"2024-03-18T19:06:43.224190+0000","last_peered":"2024-03-18T19:06:43.224190+0000","last_clean":"2024-03-18T19:06:43.224190+0000","last_became_active":"2024-03-18T19:06:41.930173+0000","last_became_peered":"2024-03-18T19:06:41.930173+0000","last_unstale":"2024-03-18T19:06:43.224190+0000","last_undegraded":"2024-03-18T19:06:43.224190+0000","last_fullsized":"2024-03-18T19:06:43.224190+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:00:02.294652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105590+0000","last_change":"2024-03-18T19:06:41.484999+0000","last_active":"2024-03-18T19:06:43.105590+0000","last_peered":"2024-03-18T19:06:43.105590+0000","last_clean":"2024-03-18T19:06:43.105590+0000","last_became_active":"2024-03-18T19:06:41.484693+0000","last_became_peered":"2024-03-18T19:06:41.484693+0000","last_unstale":"2024-03-18T19:06:43.105590+0000","last_undegraded":"2024-03-18T19:06:43.105590+0000","last_fullsized":"2024-03-18T19:06:43.105590+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:33:21.349317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":10,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:43.105715+0000","last_change":"2024-03-18T19:06:41.916902+0000","last_active":"2024-03-18T19:06:43.105715+0000","last_peered":"2024-03-18T19:06:43.105715+0000","last_clean":"2024-03-18T19:06:43.105715+0000","last_became_active":"2024-03-18T19:06:41.916635+0000","last_became_peered":"2024-03-18T19:06:41.916635+0000","last_unstale":"2024-03-18T19:06:43.105715+0000","last_undegraded":"2024-03-18T19:06:43.105715+0000","last_fullsized":"2024-03-18T19:06:43.105715+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:41.464154+0000","last_clean_scrub_stamp":"2024-03-18T19:06:41.464154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:39.234562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-03-18T19:06:42.906036+0000","last_change":"2024-03-18T19:06:41.488714+0000","last_active":"2024-03-18T19:06:42.906036+0000","last_peered":"2024-03-18T19:06:42.906036+0000","last_clean":"2024-03-18T19:06:42.906036+0000","last_became_active":"2024-03-18T19:06:41.484846+0000","last_became_peered":"2024-03-18T19:06:41.484846+0000","last_unstale":"2024-03-18T19:06:42.906036+0000","last_undegraded":"2024-03-18T19:06:42.906036+0000","last_fullsized":"2024-03-18T19:06:42.906036+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:06:40.458824+0000","last_clean_scrub_stamp":"2024-03-18T19:06:40.458824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:57.929295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"106'183","reported_seq":486,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-03-18T19:06:41.906777+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T19:06:41.906777+0000","last_peered":"2024-03-18T19:06:41.906777+0000","last_clean":"2024-03-18T19:06:41.906777+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T19:06:41.906777+0000","last_undegraded":"2024-03-18T19:06:41.906777+0000","last_fullsized":"2024-03-18T19:06:41.906777+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575048,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59356,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312484,"statfs":{"total":96636764160,"available":96575983616,"internally_reserved":0,"allocated":2043904,"data_stored":1311716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.55900000000000005,"15min":0.57099999999999995},"min":{"1min":0.39400000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.64100000000000001,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.432},{"interface":"front","average":{"1min":0.504,"5min":0.56000000000000005,"15min":0.58199999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68700000000000006,"5min":1.387,"15min":1.387},"last":0.58199999999999996}]},{"osd":1,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.54000000000000004,"15min":0.57499999999999996},"min":{"1min":0.36499999999999999,"5min":0.32300000000000001,"15min":0.307},"max":{"1min":0.66200000000000003,"5min":0.81699999999999995,"15min":1.484},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.503,"5min":0.56499999999999995,"15min":0.58899999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.65600000000000003,"5min":0.80700000000000005,"15min":0.92000000000000004},"last":0.377}]},{"osd":2,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55900000000000005,"15min":0.57799999999999996},"min":{"1min":0.42199999999999999,"5min":0.34499999999999997,"15min":0.27800000000000002},"max":{"1min":0.69799999999999995,"5min":1.3600000000000001,"15min":1.5129999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.59099999999999997},"min":{"1min":0.38300000000000001,"5min":0.36899999999999999,"15min":0.30199999999999999},"max":{"1min":0.73699999999999999,"5min":1.327,"15min":1.536},"last":0.47699999999999998}]},{"osd":3,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55200000000000005,"15min":0.57999999999999996},"min":{"1min":0.35599999999999998,"5min":0.318,"15min":0.311},"max":{"1min":0.61699999999999999,"5min":0.79100000000000004,"15min":1.5489999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.57599999999999996,"15min":0.60099999999999998},"min":{"1min":0.40600000000000003,"5min":0.38800000000000001,"15min":0.38200000000000001},"max":{"1min":0.64900000000000002,"5min":0.78200000000000003,"15min":1.5660000000000001},"last":0.57099999999999995}]},{"osd":4,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.45400000000000001},"min":{"1min":0.311,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.59299999999999997,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.46999999999999997,"15min":0.47899999999999998},"min":{"1min":0.316,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.73199999999999998},"last":0.501}]},{"osd":5,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.47799999999999998,"15min":0.47999999999999998},"min":{"1min":0.34200000000000003,"5min":0.27100000000000002,"15min":0.26000000000000001},"max":{"1min":0.71499999999999997,"5min":0.751,"15min":0.78300000000000003},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.504,"15min":0.51800000000000002},"min":{"1min":0.35699999999999998,"5min":0.30299999999999999,"15min":0.28799999999999998},"max":{"1min":0.751,"5min":0.78200000000000003,"15min":0.80300000000000005},"last":0.66400000000000003}]},{"osd":6,"last update":"Mon Mar 18 19:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53500000000000003,"15min":0.54600000000000004},"min":{"1min":0.38600000000000001,"5min":0.374,"15min":0.34300000000000003},"max":{"1min":0.67700000000000005,"5min":0.81799999999999995,"15min":0.84499999999999997},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.35899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68700000000000006,"5min":0.752,"15min":0.77500000000000002},"last":0.55400000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575048,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32548,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339292,"statfs":{"total":96636764160,"available":96603435008,"internally_reserved":0,"allocated":1462272,"data_stored":723460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56599999999999995,"15min":0.56699999999999995},"min":{"1min":0.38600000000000001,"5min":0.33000000000000002,"15min":0.28499999999999998},"max":{"1min":0.73699999999999999,"5min":0.75600000000000001,"15min":0.76800000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56000000000000005,"15min":0.57299999999999995},"min":{"1min":0.38400000000000001,"5min":0.34699999999999998,"15min":0.27400000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.46800000000000003}]},{"osd":1,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.35399999999999998,"5min":0.34000000000000002,"15min":0.30599999999999999},"max":{"1min":0.71499999999999997,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56299999999999994,"15min":0.57099999999999995},"min":{"1min":0.35699999999999998,"5min":0.32000000000000001,"15min":0.311},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.57699999999999996}]},{"osd":2,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.35299999999999998,"5min":0.32700000000000001,"15min":0.30499999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.57299999999999995,"15min":0.57999999999999996},"min":{"1min":0.40100000000000002,"5min":0.36699999999999999,"15min":0.30299999999999999},"max":{"1min":0.753,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.65500000000000003}]},{"osd":3,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59099999999999997,"15min":0.58499999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.32100000000000001},"max":{"1min":0.78400000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.57199999999999995,"15min":0.57799999999999996},"min":{"1min":0.41099999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79500000000000004,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.497}]},{"osd":4,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.25600000000000001},"max":{"1min":0.75,"5min":0.75900000000000001,"15min":0.76300000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.47899999999999998,"15min":0.47799999999999998},"min":{"1min":0.29599999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.73099999999999998,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.61899999999999999}]},{"osd":5,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.46999999999999997,"15min":0.47099999999999997},"min":{"1min":0.33000000000000002,"5min":0.27300000000000002,"15min":0.26900000000000002},"max":{"1min":0.71899999999999997,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.52000000000000002,"15min":0.51700000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76600000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.75600000000000001}]},{"osd":7,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53300000000000003,"15min":0.52400000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81100000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.54200000000000004,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77900000000000003,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.68700000000000006}]}]},{"osd":1,"up_from":13,"seq":55834575048,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1445888,"data_stored":721362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.36499999999999999,"15min":0.373},"min":{"1min":0.217,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52900000000000003,"5min":0.63300000000000001,"15min":0.66300000000000003},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.40600000000000003,"15min":0.41399999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.66500000000000004,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.32400000000000001}]},{"osd":2,"last update":"Mon Mar 18 19:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.40000000000000002,"15min":0.40500000000000003},"min":{"1min":0.28899999999999998,"5min":0.26500000000000001,"15min":0.254},"max":{"1min":0.64200000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.439,"15min":0.44400000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.58299999999999996,"5min":0.68700000000000006,"15min":0.71699999999999997},"last":0.35699999999999998}]},{"osd":3,"last update":"Mon Mar 18 19:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.44900000000000001,"15min":0.46200000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.38},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.41399999999999998,"15min":0.41899999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.56499999999999995,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.39300000000000002}]},{"osd":4,"last update":"Mon Mar 18 19:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.61299999999999999,"15min":0.627},"min":{"1min":0.42199999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.77200000000000002,"5min":0.93999999999999995,"15min":1.0409999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.61899999999999999,"15min":0.64100000000000001},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.34499999999999997},"max":{"1min":0.79100000000000004,"5min":1.018,"15min":1.018},"last":0.53900000000000003}]},{"osd":5,"last update":"Mon Mar 18 19:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.61299999999999999,"15min":0.63800000000000001},"min":{"1min":0.40699999999999997,"5min":0.40100000000000002,"15min":0.38600000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.99199999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.63,"15min":0.65200000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.70099999999999996,"5min":1.006,"15min":1.1319999999999999},"last":0.55200000000000005}]},{"osd":6,"last update":"Mon Mar 18 19:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.61499999999999999,"15min":0.63600000000000001},"min":{"1min":0.42299999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.68400000000000005,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.39100000000000001},"max":{"1min":0.82999999999999996,"5min":0.96799999999999997,"15min":0.98099999999999998},"last":0.64800000000000002}]},{"osd":7,"last update":"Mon Mar 18 19:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.64800000000000002,"15min":0.65800000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.36699999999999999},"max":{"1min":0.84599999999999997,"5min":0.92900000000000005,"15min":1.0660000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.64400000000000002,"15min":0.66000000000000003},"min":{"1min":0.45300000000000001,"5min":0.40999999999999998,"15min":0.39900000000000002},"max":{"1min":0.81299999999999994,"5min":0.94299999999999995,"15min":1.1499999999999999},"last":0.60599999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575049,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37228,"kb_used_data":2012,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":2060288,"data_stored":1313930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 19:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36799999999999999,"15min":0.378},"min":{"1min":0.221,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.58599999999999997,"5min":0.60499999999999998,"15min":0.621},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.40000000000000002,"15min":0.41199999999999998},"min":{"1min":0.253,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.53400000000000003,"5min":0.61699999999999999,"15min":0.65100000000000002},"last":0.46800000000000003}]},{"osd":2,"last update":"Mon Mar 18 19:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.40899999999999997,"15min":0.41799999999999998},"min":{"1min":0.26200000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.627},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.42699999999999999,"15min":0.432},"min":{"1min":0.31,"5min":0.25,"15min":0.25},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.435}]},{"osd":3,"last update":"Mon Mar 18 19:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.39800000000000002,"15min":0.43099999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.217},"max":{"1min":0.53900000000000003,"5min":0.61299999999999999,"15min":5.7359999999999998},"last":0.495},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.45500000000000002,"15min":0.48799999999999999},"min":{"1min":0.33500000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.64900000000000002,"5min":0.74399999999999999,"15min":5.9329999999999998},"last":0.50900000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.621,"15min":0.64000000000000001},"min":{"1min":0.47599999999999998,"5min":0.33200000000000002,"15min":0.27200000000000002},"max":{"1min":0.86699999999999999,"5min":0.91800000000000004,"15min":1.22},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.628,"15min":0.64200000000000002},"min":{"1min":0.39000000000000001,"5min":0.36799999999999999,"15min":0.315},"max":{"1min":0.83599999999999997,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.59999999999999998}]},{"osd":5,"last update":"Mon Mar 18 19:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63800000000000001,"15min":0.65500000000000003},"min":{"1min":0.45200000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.89400000000000002,"5min":0.96199999999999997,"15min":2.9649999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.63900000000000001,"15min":0.66000000000000003},"min":{"1min":0.41599999999999998,"5min":0.38,"15min":0.31},"max":{"1min":0.77100000000000002,"5min":1.0760000000000001,"15min":1.2609999999999999},"last":0.59299999999999997}]},{"osd":6,"last update":"Mon Mar 18 19:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.63600000000000001,"15min":0.65500000000000003},"min":{"1min":0.439,"5min":0.39600000000000002,"15min":0.38100000000000001},"max":{"1min":0.78800000000000003,"5min":0.94599999999999995,"15min":1.133},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.626,"5min":0.65100000000000002,"15min":0.67700000000000005},"min":{"1min":0.42699999999999999,"5min":0.41199999999999998,"15min":0.40999999999999998},"max":{"1min":0.93400000000000005,"5min":1.0409999999999999,"15min":2.9300000000000002},"last":0.70099999999999996}]},{"osd":7,"last update":"Mon Mar 18 19:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65900000000000003,"15min":0.66900000000000004},"min":{"1min":0.45800000000000002,"5min":0.38300000000000001,"15min":0.34899999999999998},"max":{"1min":0.75700000000000001,"5min":1.0229999999999999,"15min":1.2849999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.65400000000000003,"15min":0.67100000000000004},"min":{"1min":0.46400000000000002,"5min":0.437,"15min":0.38500000000000001},"max":{"1min":0.90300000000000002,"5min":0.94299999999999995,"15min":1.002},"last":0.70899999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575049,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58768,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94313072,"statfs":{"total":96636764160,"available":96576585728,"internally_reserved":0,"allocated":1441792,"data_stored":721292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.375,"15min":0.371},"min":{"1min":0.22,"5min":0.22,"15min":0.17799999999999999},"max":{"1min":0.52800000000000002,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.29599999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.57199999999999995,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.39400000000000002}]},{"osd":1,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.46899999999999997,"15min":0.46800000000000003},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.27100000000000002},"max":{"1min":0.56499999999999995,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.40400000000000003,"15min":0.39200000000000002},"min":{"1min":0.26100000000000001,"5min":0.23699999999999999,"15min":0.216},"max":{"1min":0.47999999999999998,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.28999999999999998}]},{"osd":3,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.41699999999999998,"15min":0.40600000000000003},"min":{"1min":0.27800000000000002,"5min":0.24199999999999999,"15min":0.22500000000000001},"max":{"1min":0.65800000000000003,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.504,"15min":0.498},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.27700000000000002},"max":{"1min":0.66100000000000003,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.46200000000000002}]},{"osd":4,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.61799999999999999,"15min":0.61699999999999999},"min":{"1min":0.38600000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.73599999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.61499999999999999,"15min":0.626},"min":{"1min":0.378,"5min":0.35899999999999999,"15min":0.32100000000000001},"max":{"1min":0.79200000000000004,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.48799999999999999}]},{"osd":5,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.64500000000000002,"15min":0.64400000000000002},"min":{"1min":0.39800000000000002,"5min":0.34699999999999998,"15min":0.309},"max":{"1min":0.77700000000000002,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.65300000000000002,"15min":0.64000000000000001},"min":{"1min":0.40500000000000003,"5min":0.375,"15min":0.33400000000000002},"max":{"1min":0.78200000000000003,"5min":1.948,"15min":1.948},"last":0.55600000000000005}]},{"osd":6,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.66000000000000003,"15min":0.65500000000000003},"min":{"1min":0.39600000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82299999999999995,"5min":2.9470000000000001,"15min":2.9470000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.68200000000000005,"15min":0.68400000000000005},"min":{"1min":0.44900000000000001,"5min":0.432,"15min":0.37},"max":{"1min":0.83299999999999996,"5min":1.054,"15min":1.083},"last":0.63200000000000001}]},{"osd":7,"last update":"Mon Mar 18 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.65500000000000003,"15min":0.65100000000000002},"min":{"1min":0.41799999999999998,"5min":0.40999999999999998,"15min":0.38200000000000001},"max":{"1min":0.78000000000000003,"5min":1.083,"15min":1.083},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.65300000000000002,"15min":0.66100000000000003},"min":{"1min":0.39500000000000002,"5min":0.371,"15min":0.34599999999999997},"max":{"1min":0.84799999999999998,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.64300000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575049,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36620,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335220,"statfs":{"total":96636764160,"available":96599265280,"internally_reserved":0,"allocated":1441792,"data_stored":721292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.36899999999999999,"15min":0.379},"min":{"1min":0.23699999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.51200000000000001,"5min":0.81100000000000005,"15min":1.204},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.40400000000000003,"15min":0.39600000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.23000000000000001},"max":{"1min":0.57499999999999996,"5min":1.137,"15min":1.137},"last":0.435}]},{"osd":1,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40000000000000002,"15min":0.39700000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.221},"max":{"1min":0.621,"5min":0.999,"15min":1.1220000000000001},"last":0.307},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.437,"15min":0.443},"min":{"1min":0.29999999999999999,"5min":0.28499999999999998,"15min":0.26500000000000001},"max":{"1min":0.51900000000000002,"5min":1.093,"15min":1.1080000000000001},"last":0.41799999999999998}]},{"osd":2,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.318,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.67500000000000004,"5min":1.0369999999999999,"15min":1.1879999999999999},"last":0.378},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.46300000000000002,"15min":0.47499999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.28599999999999998},"max":{"1min":0.55600000000000005,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.45300000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.052,"5min":1.4219999999999999,"15min":1.871},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.61199999999999999,"15min":0.63},"min":{"1min":0.42699999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78300000000000003,"5min":1.0700000000000001,"15min":1.764},"last":0.83799999999999997}]},{"osd":5,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.65600000000000003,"15min":0.65900000000000003},"min":{"1min":0.44400000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.0249999999999999,"5min":2.5329999999999999,"15min":2.5329999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67000000000000004,"15min":0.68400000000000005},"min":{"1min":0.47099999999999997,"5min":0.45500000000000002,"15min":0.437},"max":{"1min":0.94599999999999995,"5min":1.5409999999999999,"15min":1.794},"last":0.76800000000000002}]},{"osd":6,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.66700000000000004,"15min":0.67200000000000004},"min":{"1min":0.44400000000000001,"5min":0.41499999999999998,"15min":0.40300000000000002},"max":{"1min":1.1060000000000001,"5min":2.7389999999999999,"15min":2.7389999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.66100000000000003,"15min":0.67500000000000004},"min":{"1min":0.47799999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96699999999999997,"5min":1.159,"15min":1.8069999999999999},"last":0.91800000000000004}]},{"osd":7,"last update":"Mon Mar 18 19:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.64400000000000002,"15min":0.66900000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86499999999999999,"5min":1.3959999999999999,"15min":1.732},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.66800000000000004,"15min":0.67000000000000004},"min":{"1min":0.47399999999999998,"5min":0.439,"15min":0.40500000000000003},"max":{"1min":1.085,"5min":2.7050000000000001,"15min":2.7050000000000001},"last":0.63100000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575049,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32536,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339304,"statfs":{"total":96636764160,"available":96603447296,"internally_reserved":0,"allocated":1449984,"data_stored":721452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55000000000000004,"15min":0.56899999999999995},"min":{"1min":0.31,"5min":0.28999999999999998,"15min":0.246},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.627,"5min":0.57299999999999995,"15min":0.58699999999999997},"min":{"1min":0.35299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.645,"5min":3.645,"15min":3.645},"last":0.53100000000000003}]},{"osd":1,"last update":"Mon Mar 18 19:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.64400000000000002,"15min":0.60899999999999999},"min":{"1min":0.36199999999999999,"5min":0.34499999999999997,"15min":0.316},"max":{"1min":9.9499999999999993,"5min":9.9499999999999993,"15min":9.9499999999999993},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53900000000000003,"15min":0.57599999999999996},"min":{"1min":0.32300000000000001,"5min":0.308,"15min":0.308},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.51700000000000002}]},{"osd":2,"last update":"Mon Mar 18 19:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.55200000000000005,"15min":0.57399999999999995},"min":{"1min":0.371,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.96299999999999997,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.58599999999999997,"15min":0.60399999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.32600000000000001},"max":{"1min":3.5990000000000002,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.56299999999999994}]},{"osd":3,"last update":"Mon Mar 18 19:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55400000000000005,"15min":0.58599999999999997},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56399999999999995,"15min":0.59399999999999997},"min":{"1min":0.39300000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70299999999999996,"5min":1.121,"15min":1.121},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Mar 18 19:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45100000000000001,"15min":0.46100000000000002},"min":{"1min":0.32800000000000001,"5min":0.29599999999999999,"15min":0.245},"max":{"1min":0.68400000000000005,"5min":0.73499999999999999,"15min":0.77500000000000002},"last":0.376},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.50700000000000001,"15min":0.49399999999999999},"min":{"1min":0.35799999999999998,"5min":0.30099999999999999,"15min":0.27200000000000002},"max":{"1min":5.4900000000000002,"5min":5.4900000000000002,"15min":5.4900000000000002},"last":0.46200000000000002}]},{"osd":6,"last update":"Mon Mar 18 19:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.46999999999999997,"15min":0.48199999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.29499999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.78900000000000003},"last":0.439},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.52600000000000002,"15min":0.52700000000000002},"min":{"1min":0.34999999999999998,"5min":0.33200000000000002,"15min":0.28999999999999998},"max":{"1min":3.7050000000000001,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":0.48199999999999998}]},{"osd":7,"last update":"Mon Mar 18 19:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.50800000000000001,"15min":0.52000000000000002},"min":{"1min":0.37,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52400000000000002,"15min":0.54800000000000004},"min":{"1min":0.39500000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.73799999999999999,"5min":0.75900000000000001,"15min":0.79700000000000004},"last":0.503}]}]},{"osd":5,"up_from":13,"seq":55834575048,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50580,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321260,"statfs":{"total":96636764160,"available":96584970240,"internally_reserved":0,"allocated":1441792,"data_stored":721292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:06:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.54500000000000004,"15min":0.56799999999999995},"min":{"1min":0.36899999999999999,"5min":0.26800000000000002,"15min":0.26600000000000001},"max":{"1min":0.67800000000000005,"5min":0.78400000000000003,"15min":1.1679999999999999},"last":0.436},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.57099999999999995,"15min":0.57199999999999995},"min":{"1min":0.38,"5min":0.33700000000000002,"15min":0.25800000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":1.1319999999999999},"last":0.48999999999999999}]},{"osd":1,"last update":"Mon Mar 18 19:06:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56100000000000005,"15min":0.57299999999999995},"min":{"1min":0.32700000000000001,"5min":0.32600000000000001,"15min":0.29599999999999999},"max":{"1min":0.748,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.57499999999999996,"15min":0.58299999999999996},"min":{"1min":0.35999999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70499999999999996,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.47099999999999997}]},{"osd":2,"last update":"Mon Mar 18 19:06:15 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.317,"5min":0.30399999999999999,"15min":0.28399999999999997},"max":{"1min":0.66500000000000004,"5min":1.153,"15min":1.1579999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56100000000000005,"15min":0.57499999999999996},"min":{"1min":0.41499999999999998,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":1.0569999999999999},"last":0.42499999999999999}]},{"osd":3,"last update":"Mon Mar 18 19:06:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.39000000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.65800000000000003,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59099999999999997,"15min":0.60099999999999998},"min":{"1min":0.42499999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.71499999999999997,"5min":1.212,"15min":1.212},"last":0.38}]},{"osd":4,"last update":"Mon Mar 18 19:06:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.47399999999999998,"15min":0.46700000000000003},"min":{"1min":0.34599999999999997,"5min":0.255,"15min":0.24199999999999999},"max":{"1min":0.69499999999999995,"5min":1.248,"15min":1.248},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.33500000000000002,"5min":0.316,"15min":0.23100000000000001},"max":{"1min":0.63800000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.33800000000000002}]},{"osd":6,"last update":"Mon Mar 18 19:06:15 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.78200000000000003,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.50700000000000001,"15min":0.51000000000000001},"min":{"1min":0.33800000000000002,"5min":0.32100000000000001,"15min":0.26800000000000002},"max":{"1min":0.83199999999999996,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.46100000000000002}]},{"osd":7,"last update":"Mon Mar 18 19:06:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.52200000000000002,"15min":0.52700000000000002},"min":{"1min":0.32700000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.69999999999999996,"5min":1.282,"15min":1.282},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55500000000000005,"15min":0.54800000000000004},"min":{"1min":0.35699999999999998,"5min":0.31900000000000001,"15min":0.28499999999999998},"max":{"1min":0.72199999999999998,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.41199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T19:06:46.455 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T19:06:46.456 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-18T19:06:46.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.623+0000 7fe3ff6b4700 1 Processor -- start 2024-03-18T19:06:46.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.624+0000 7fe3ff6b4700 1 -- start start 2024-03-18T19:06:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.624+0000 7fe3ff6b4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3f815b6c0 0x7fe3f80b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.624+0000 7fe3ff6b4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe3f80cd430 0x7fe3f80c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.624+0000 7fe3ff6b4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe3f80c8600 0x7fe3f80c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.624+0000 7fe3ff6b4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3f80c8fd0 con 0x7fe3f80c8600 2024-03-18T19:06:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.624+0000 7fe3ff6b4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3f80d01f0 con 0x7fe3f80cd430 2024-03-18T19:06:46.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.624+0000 7fe3ff6b4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f80ccfa0 con 0x7fe3f815b6c0 2024-03-18T19:06:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.625+0000 7fe3fdc51700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe3f80c8600 0x7fe3f80c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.625+0000 7fe3fdc51700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe3f80c8600 0x7fe3f80c8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49028/0 (socket says 172.21.15.1:49028) 2024-03-18T19:06:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.625+0000 7fe3fdc51700 1 -- 172.21.15.1:0/116522384 learned_addr learned my addr 172.21.15.1:0/116522384 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.625+0000 7fe3fd450700 1 --2- 172.21.15.1:0/116522384 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3f815b6c0 0x7fe3f80b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.625+0000 7fe3fcc4f700 1 --2- 172.21.15.1:0/116522384 >> v2:172.21.15.165:3300/0 conn(0x7fe3f80cd430 0x7fe3f80c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.625+0000 7fe3fdc51700 1 -- 172.21.15.1:0/116522384 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3f815b6c0 msgr2=0x7fe3f80b2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.625+0000 7fe3fdc51700 1 --2- 172.21.15.1:0/116522384 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3f815b6c0 0x7fe3f80b2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.625+0000 7fe3fdc51700 1 -- 172.21.15.1:0/116522384 >> v2:172.21.15.165:3300/0 conn(0x7fe3f80cd430 msgr2=0x7fe3f80c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.625+0000 7fe3fdc51700 1 --2- 172.21.15.1:0/116522384 >> v2:172.21.15.165:3300/0 conn(0x7fe3f80cd430 0x7fe3f80c80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.625+0000 7fe3fdc51700 1 -- 172.21.15.1:0/116522384 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f80d1db0 con 0x7fe3f80c8600 2024-03-18T19:06:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.625+0000 7fe3fcc4f700 1 --2- 172.21.15.1:0/116522384 >> v2:172.21.15.165:3300/0 conn(0x7fe3f80cd430 0x7fe3f80c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.625+0000 7fe3fdc51700 1 --2- 172.21.15.1:0/116522384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3f80c8600 0x7fe3f80c8a90 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8072030 tx=0x7fe3f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dc5a5d97f010008 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.625+0000 7fe3effff700 1 -- 172.21.15.1:0/116522384 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f000e070 con 0x7fe3f80c8600 2024-03-18T19:06:46.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.625+0000 7fe3effff700 1 -- 172.21.15.1:0/116522384 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f000b9c0 con 0x7fe3f80c8600 2024-03-18T19:06:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.625+0000 7fe3effff700 1 -- 172.21.15.1:0/116522384 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f000bd70 con 0x7fe3f80c8600 2024-03-18T19:06:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.626+0000 7fe3ff6b4700 1 -- 172.21.15.1:0/116522384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3f80c8600 msgr2=0x7fe3f80c8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.626+0000 7fe3ff6b4700 1 --2- 172.21.15.1:0/116522384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3f80c8600 0x7fe3f80c8a90 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8072030 tx=0x7fe3f000b330 comp rx=0 tx=0).stop 2024-03-18T19:06:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.626+0000 7fe3ff6b4700 1 -- 172.21.15.1:0/116522384 shutdown_connections 2024-03-18T19:06:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.626+0000 7fe3ff6b4700 1 --2- 172.21.15.1:0/116522384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3f80c8600 0x7fe3f80c8a90 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.626+0000 7fe3ff6b4700 1 --2- 172.21.15.1:0/116522384 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3f815b6c0 0x7fe3f80b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.626+0000 7fe3ff6b4700 1 --2- 172.21.15.1:0/116522384 >> v2:172.21.15.165:3300/0 conn(0x7fe3f80cd430 0x7fe3f80c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.626+0000 7fe3ff6b4700 1 -- 172.21.15.1:0/116522384 >> 172.21.15.1:0/116522384 conn(0x7fe3f80d0ff0 msgr2=0x7fe3f80d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.626+0000 7fe3ff6b4700 1 -- 172.21.15.1:0/116522384 shutdown_connections 2024-03-18T19:06:46.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.626+0000 7fe3ff6b4700 1 -- 172.21.15.1:0/116522384 wait complete. 2024-03-18T19:06:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.627+0000 7fe3ff6b4700 1 Processor -- start 2024-03-18T19:06:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.627+0000 7fe3ff6b4700 1 -- start start 2024-03-18T19:06:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.628+0000 7fe3ff6b4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3f80c8600 0x7fe3f80c5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.628+0000 7fe3ff6b4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3f80cd430 0x7fe3f809adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.628+0000 7fe3ff6b4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3f815b6c0 0x7fe3f809b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.628+0000 7fe3ff6b4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f80c6ed0 con 0x7fe3f80c8600 2024-03-18T19:06:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.628+0000 7fe3ff6b4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f80c4ca0 con 0x7fe3f80cd430 2024-03-18T19:06:46.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.628+0000 7fe3ff6b4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f80c4f80 con 0x7fe3f815b6c0 2024-03-18T19:06:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.628+0000 7fe3fd450700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3f80c8600 0x7fe3f80c5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.628+0000 7fe3fd450700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3f80c8600 0x7fe3f80c5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49030/0 (socket says 172.21.15.1:49030) 2024-03-18T19:06:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.628+0000 7fe3fd450700 1 -- 172.21.15.1:0/3661041914 learned_addr learned my addr 172.21.15.1:0/3661041914 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.628+0000 7fe3fdc51700 1 --2- 172.21.15.1:0/3661041914 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3f815b6c0 0x7fe3f809b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.628+0000 7fe3fcc4f700 1 --2- 172.21.15.1:0/3661041914 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3f80cd430 0x7fe3f809adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.628+0000 7fe3fd450700 1 -- 172.21.15.1:0/3661041914 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3f815b6c0 msgr2=0x7fe3f809b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.628+0000 7fe3fd450700 1 --2- 172.21.15.1:0/3661041914 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3f815b6c0 0x7fe3f809b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.628+0000 7fe3fd450700 1 -- 172.21.15.1:0/3661041914 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3f80cd430 msgr2=0x7fe3f809adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.628+0000 7fe3fd450700 1 --2- 172.21.15.1:0/3661041914 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3f80cd430 0x7fe3f809adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.628+0000 7fe3fd450700 1 -- 172.21.15.1:0/3661041914 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f0009b30 con 0x7fe3f80c8600 2024-03-18T19:06:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.628+0000 7fe3fd450700 1 --2- 172.21.15.1:0/3661041914 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3f80c8600 0x7fe3f80c5d40 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fe3e800d800 tx=0x7fe3e800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.629+0000 7fe3edffb700 1 -- 172.21.15.1:0/3661041914 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e80125f0 con 0x7fe3f80c8600 2024-03-18T19:06:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.629+0000 7fe3edffb700 1 -- 172.21.15.1:0/3661041914 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e8012750 con 0x7fe3f80c8600 2024-03-18T19:06:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.629+0000 7fe3ff6b4700 1 -- 172.21.15.1:0/3661041914 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f80c5260 con 0x7fe3f80c8600 2024-03-18T19:06:46.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.629+0000 7fe3edffb700 1 -- 172.21.15.1:0/3661041914 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8012b00 con 0x7fe3f80c8600 2024-03-18T19:06:46.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.629+0000 7fe3ff6b4700 1 -- 172.21.15.1:0/3661041914 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f80c5820 con 0x7fe3f80c8600 2024-03-18T19:06:46.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.630+0000 7fe3ff6b4700 1 -- 172.21.15.1:0/3661041914 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f80676a0 con 0x7fe3f80c8600 2024-03-18T19:06:46.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.630+0000 7fe3edffb700 1 -- 172.21.15.1:0/3661041914 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe3e8012c60 con 0x7fe3f80c8600 2024-03-18T19:06:46.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.631+0000 7fe3edffb700 1 --2- 172.21.15.1:0/3661041914 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe3dc06adb0 0x7fe3dc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:46.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.631+0000 7fe3edffb700 1 -- 172.21.15.1:0/3661041914 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe3e8089570 con 0x7fe3f80c8600 2024-03-18T19:06:46.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.631+0000 7fe3fcc4f700 1 --2- 172.21.15.1:0/3661041914 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe3dc06adb0 0x7fe3dc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:46.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.631+0000 7fe3fcc4f700 1 --2- 172.21.15.1:0/3661041914 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe3dc06adb0 0x7fe3dc06d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe3f81593c0 tx=0x7fe3f4008040 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:46.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.635+0000 7fe3edffb700 1 -- 172.21.15.1:0/3661041914 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe3e8054640 con 0x7fe3f80c8600 2024-03-18T19:06:46.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.925+0000 7fe3ff6b4700 1 -- 172.21.15.1:0/3661041914 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fe3f80c9aa0 con 0x7fe3f80c8600 2024-03-18T19:06:46.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.928+0000 7fe3edffb700 1 -- 172.21.15.1:0/3661041914 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+0 (secure 0 0 0) 0x7fe3e8057c60 con 0x7fe3f80c8600 2024-03-18T19:06:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.930+0000 7fe3ff6b4700 1 -- 172.21.15.1:0/3661041914 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe3dc06adb0 msgr2=0x7fe3dc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.930+0000 7fe3ff6b4700 1 --2- 172.21.15.1:0/3661041914 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe3dc06adb0 0x7fe3dc06d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe3f81593c0 tx=0x7fe3f4008040 comp rx=0 tx=0).stop 2024-03-18T19:06:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.930+0000 7fe3ff6b4700 1 -- 172.21.15.1:0/3661041914 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3f80c8600 msgr2=0x7fe3f80c5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.930+0000 7fe3ff6b4700 1 --2- 172.21.15.1:0/3661041914 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3f80c8600 0x7fe3f80c5d40 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fe3e800d800 tx=0x7fe3e800db10 comp rx=0 tx=0).stop 2024-03-18T19:06:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.930+0000 7fe3ff6b4700 1 -- 172.21.15.1:0/3661041914 shutdown_connections 2024-03-18T19:06:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.930+0000 7fe3ff6b4700 1 --2- 172.21.15.1:0/3661041914 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe3dc06adb0 0x7fe3dc06d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.930+0000 7fe3ff6b4700 1 --2- 172.21.15.1:0/3661041914 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3f815b6c0 0x7fe3f809b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.930+0000 7fe3ff6b4700 1 --2- 172.21.15.1:0/3661041914 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3f80c8600 0x7fe3f80c5d40 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.930+0000 7fe3ff6b4700 1 --2- 172.21.15.1:0/3661041914 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3f80cd430 0x7fe3f809adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.930+0000 7fe3ff6b4700 1 -- 172.21.15.1:0/3661041914 >> 172.21.15.1:0/3661041914 conn(0x7fe3f80d0ff0 msgr2=0x7fe3f806fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.930+0000 7fe3ff6b4700 1 -- 172.21.15.1:0/3661041914 shutdown_connections 2024-03-18T19:06:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:46.930+0000 7fe3ff6b4700 1 -- 172.21.15.1:0/3661041914 wait complete. 2024-03-18T19:06:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-18T19:06:46.939 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:06:47.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.067+0000 7f7fb0f4a700 1 Processor -- start 2024-03-18T19:06:47.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.067+0000 7f7fb0f4a700 1 -- start start 2024-03-18T19:06:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.068+0000 7f7fb0f4a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fac099ea0 0x7f7fac0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.068+0000 7f7fb0f4a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac0d0c80 con 0x7f7fac15d680 2024-03-18T19:06:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.068+0000 7f7fb0f4a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac0d09a0 con 0x7f7fac099a40 2024-03-18T19:06:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.068+0000 7f7fb0f4a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac0cb830 con 0x7f7fac099ea0 2024-03-18T19:06:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.068+0000 7f7faa59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7fac15d680 0x7f7fac0cff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47156/0 (socket says 172.21.15.1:47156) 2024-03-18T19:06:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.068+0000 7f7faa59c700 1 -- 172.21.15.1:0/1399058445 learned_addr learned my addr 172.21.15.1:0/1399058445 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.068+0000 7f7faad9d700 1 --2- 172.21.15.1:0/1399058445 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fac099ea0 0x7f7fac0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.068+0000 7f7fa959a700 1 -- 172.21.15.1:0/1399058445 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208249945 0 0) 0x7f7fac0d0c80 con 0x7f7fac15d680 2024-03-18T19:06:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.068+0000 7f7fa959a700 1 -- 172.21.15.1:0/1399058445 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f940035b0 con 0x7f7fac15d680 2024-03-18T19:06:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.068+0000 7f7fa959a700 1 -- 172.21.15.1:0/1399058445 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1379897720 0 0) 0x7f7f940035b0 con 0x7f7fac15d680 2024-03-18T19:06:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7fa959a700 1 -- 172.21.15.1:0/1399058445 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fac099ea0 msgr2=0x7f7fac0cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7fa959a700 1 --2- 172.21.15.1:0/1399058445 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fac099ea0 0x7f7fac0cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7fa959a700 1 -- 172.21.15.1:0/1399058445 >> v1:172.21.15.165:6789/0 conn(0x7f7fac099a40 legacy=0x7f7fac0cc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7fa959a700 1 -- 172.21.15.1:0/1399058445 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac0cbb10 con 0x7f7fac15d680 2024-03-18T19:06:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7fa959a700 1 -- 172.21.15.1:0/1399058445 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f7f9c001670 con 0x7f7fac15d680 2024-03-18T19:06:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7fa959a700 1 -- 172.21.15.1:0/1399058445 >> v1:172.21.15.1:6789/0 conn(0x7f7fac15d680 legacy=0x7f7fac0cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7fa959a700 1 --2- 172.21.15.1:0/1399058445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f94003a30 0x7f7f94003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7fa959a700 1 --2- 172.21.15.1:0/1399058445 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fac099ea0 0x7f7f94004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7fa959a700 1 --2- 172.21.15.1:0/1399058445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f94004d90 0x7f7f94005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7fa959a700 1 -- 172.21.15.1:0/1399058445 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac0cb830 con 0x7f7f94004d90 2024-03-18T19:06:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7fa959a700 1 -- 172.21.15.1:0/1399058445 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9400baf0 con 0x7f7f94003a30 2024-03-18T19:06:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7fa959a700 1 -- 172.21.15.1:0/1399058445 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9400bcb0 con 0x7f7fac099ea0 2024-03-18T19:06:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7faad9d700 1 --2- 172.21.15.1:0/1399058445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f94004d90 0x7f7f94005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7faa59c700 1 --2- 172.21.15.1:0/1399058445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f94003a30 0x7f7f94003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7fa9d9b700 1 --2- 172.21.15.1:0/1399058445 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fac099ea0 0x7f7f94004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7faad9d700 1 -- 172.21.15.1:0/1399058445 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fac099ea0 msgr2=0x7f7f94004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7faad9d700 1 --2- 172.21.15.1:0/1399058445 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fac099ea0 0x7f7f94004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7faad9d700 1 -- 172.21.15.1:0/1399058445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f94003a30 msgr2=0x7f7f94003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:47.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7faad9d700 1 --2- 172.21.15.1:0/1399058445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f94003a30 0x7f7f94003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7faad9d700 1 -- 172.21.15.1:0/1399058445 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f940035b0 con 0x7f7f94004d90 2024-03-18T19:06:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7faa59c700 1 --2- 172.21.15.1:0/1399058445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f94003a30 0x7f7f94003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7faad9d700 1 --2- 172.21.15.1:0/1399058445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f94004d90 0x7f7f94005210 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4009f90 tx=0x7f7f94005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a5db14a10d08746 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7fa959a700 1 -- 172.21.15.1:0/1399058445 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa400ec30 con 0x7f7f94004d90 2024-03-18T19:06:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7fa959a700 1 -- 172.21.15.1:0/1399058445 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa400f040 con 0x7f7f94004d90 2024-03-18T19:06:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.069+0000 7f7fa959a700 1 -- 172.21.15.1:0/1399058445 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4014480 con 0x7f7f94004d90 2024-03-18T19:06:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.070+0000 7f7fb0f4a700 1 -- 172.21.15.1:0/1399058445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f94004d90 msgr2=0x7f7f94005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.070+0000 7f7fb0f4a700 1 --2- 172.21.15.1:0/1399058445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f94004d90 0x7f7f94005210 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4009f90 tx=0x7f7f94005880 comp rx=0 tx=0).stop 2024-03-18T19:06:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.070+0000 7f7faad9d700 1 -- 172.21.15.1:0/1399058445 reap_dead start 2024-03-18T19:06:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.070+0000 7f7fb0f4a700 1 -- 172.21.15.1:0/1399058445 shutdown_connections 2024-03-18T19:06:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.070+0000 7f7fb0f4a700 1 -- 172.21.15.1:0/1399058445 >> 172.21.15.1:0/1399058445 conn(0x7f7fac0b2ba0 msgr2=0x7f7fac0b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.070+0000 7f7fb0f4a700 1 -- 172.21.15.1:0/1399058445 shutdown_connections 2024-03-18T19:06:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.070+0000 7f7fb0f4a700 1 -- 172.21.15.1:0/1399058445 wait complete. 2024-03-18T19:06:47.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.072+0000 7f7fb0f4a700 1 Processor -- start 2024-03-18T19:06:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.072+0000 7f7fb0f4a700 1 -- start start 2024-03-18T19:06:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.072+0000 7f7fb0f4a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f94004d90 0x7f7fac09d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.072+0000 7f7fb0f4a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fac09b820 0x7f7fac1f69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.072+0000 7f7fb0f4a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fac09eb50 0x7f7fac1f6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.072+0000 7f7fb0f4a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac1f4c10 con 0x7f7fac09eb50 2024-03-18T19:06:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.072+0000 7f7fb0f4a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac09bcb0 con 0x7f7f94004d90 2024-03-18T19:06:47.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.072+0000 7f7fb0f4a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac1f45c0 con 0x7f7fac09b820 2024-03-18T19:06:47.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.072+0000 7f7faad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fac09eb50 0x7f7fac1f6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.072+0000 7f7faad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fac09eb50 0x7f7fac1f6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49042/0 (socket says 172.21.15.1:49042) 2024-03-18T19:06:47.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.072+0000 7f7faad9d700 1 -- 172.21.15.1:0/1749490342 learned_addr learned my addr 172.21.15.1:0/1749490342 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:47.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.073+0000 7f7fa9d9b700 1 --2- 172.21.15.1:0/1749490342 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fac09b820 0x7f7fac1f69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.073+0000 7f7faa59c700 1 --2- 172.21.15.1:0/1749490342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f94004d90 0x7f7fac09d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.073+0000 7f7faad9d700 1 -- 172.21.15.1:0/1749490342 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fac09b820 msgr2=0x7f7fac1f69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.073+0000 7f7faad9d700 1 --2- 172.21.15.1:0/1749490342 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fac09b820 0x7f7fac1f69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.073+0000 7f7faad9d700 1 -- 172.21.15.1:0/1749490342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f94004d90 msgr2=0x7f7fac09d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.073+0000 7f7faad9d700 1 --2- 172.21.15.1:0/1749490342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f94004d90 0x7f7fac09d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.073+0000 7f7faad9d700 1 -- 172.21.15.1:0/1749490342 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac1f48a0 con 0x7f7fac09eb50 2024-03-18T19:06:47.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.073+0000 7f7faad9d700 1 --2- 172.21.15.1:0/1749490342 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fac09eb50 0x7f7fac1f6f30 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4016000 tx=0x7f7fa4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:47.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.073+0000 7f7f9affd700 1 -- 172.21.15.1:0/1749490342 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4009410 con 0x7f7fac09eb50 2024-03-18T19:06:47.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.073+0000 7f7f9affd700 1 -- 172.21.15.1:0/1749490342 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa400ee40 con 0x7f7fac09eb50 2024-03-18T19:06:47.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.073+0000 7f7f9affd700 1 -- 172.21.15.1:0/1749490342 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4014e50 con 0x7f7fac09eb50 2024-03-18T19:06:47.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.073+0000 7f7fb0f4a700 1 -- 172.21.15.1:0/1749490342 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fac09bf90 con 0x7f7fac09eb50 2024-03-18T19:06:47.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.073+0000 7f7fb0f4a700 1 -- 172.21.15.1:0/1749490342 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fac09def0 con 0x7f7fac09eb50 2024-03-18T19:06:47.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.074+0000 7f7fb0f4a700 1 -- 172.21.15.1:0/1749490342 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fac0676a0 con 0x7f7fac09eb50 2024-03-18T19:06:47.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.075+0000 7f7f9affd700 1 -- 172.21.15.1:0/1749490342 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7fa4023020 con 0x7f7fac09eb50 2024-03-18T19:06:47.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.075+0000 7f7f9affd700 1 --2- 172.21.15.1:0/1749490342 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f8c06adb0 0x7f7f8c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.075+0000 7f7f9affd700 1 -- 172.21.15.1:0/1749490342 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7fa408a9f0 con 0x7f7fac09eb50 2024-03-18T19:06:47.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.075+0000 7f7faa59c700 1 --2- 172.21.15.1:0/1749490342 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f8c06adb0 0x7f7f8c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.075+0000 7f7faa59c700 1 --2- 172.21.15.1:0/1749490342 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f8c06adb0 0x7f7f8c06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c001200 tx=0x7f7f9c007040 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:47.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.080+0000 7f7f9affd700 1 -- 172.21.15.1:0/1749490342 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7fa4055a10 con 0x7f7fac09eb50 2024-03-18T19:06:47.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.328+0000 7f7fb0f4a700 1 -- 172.21.15.1:0/1749490342 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fac09def0 con 0x7f7fac09eb50 2024-03-18T19:06:47.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.329+0000 7f7f9affd700 1 -- 172.21.15.1:0/1749490342 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4733 (secure 0 0 0) 0x7f7fac09def0 con 0x7f7fac09eb50 2024-03-18T19:06:47.326 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:47.326 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6431,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/983420410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":983420410},{"type":"v1","addr":"172.21.15.1:6836","nonce":983420410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6440,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1569519191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1569519191},{"type":"v1","addr":"172.21.15.165:6834","nonce":1569519191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1528229019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1528229019},{"type":"v1","addr":"172.21.15.165:6835","nonce":1528229019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:06:41.899614+0000","modified":"2024-03-18T19:06:44.904965+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"c","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2226098738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2226098738},{"type":"v1","addr":"172.21.15.1:6837","nonce":2226098738}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-18T19:06:47.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.331+0000 7f7fb0f4a700 1 -- 172.21.15.1:0/1749490342 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f8c06adb0 msgr2=0x7f7f8c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:47.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.331+0000 7f7fb0f4a700 1 --2- 172.21.15.1:0/1749490342 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f8c06adb0 0x7f7f8c06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c001200 tx=0x7f7f9c007040 comp rx=0 tx=0).stop 2024-03-18T19:06:47.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.331+0000 7f7fb0f4a700 1 -- 172.21.15.1:0/1749490342 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fac09eb50 msgr2=0x7f7fac1f6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:47.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.331+0000 7f7fb0f4a700 1 --2- 172.21.15.1:0/1749490342 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fac09eb50 0x7f7fac1f6f30 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4016000 tx=0x7f7fa4009750 comp rx=0 tx=0).stop 2024-03-18T19:06:47.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.331+0000 7f7fb0f4a700 1 -- 172.21.15.1:0/1749490342 shutdown_connections 2024-03-18T19:06:47.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.331+0000 7f7fb0f4a700 1 --2- 172.21.15.1:0/1749490342 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f8c06adb0 0x7f7f8c06d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.331+0000 7f7fb0f4a700 1 --2- 172.21.15.1:0/1749490342 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fac09eb50 0x7f7fac1f6f30 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.331+0000 7f7fb0f4a700 1 --2- 172.21.15.1:0/1749490342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f94004d90 0x7f7fac09d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.331+0000 7f7fb0f4a700 1 --2- 172.21.15.1:0/1749490342 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fac09b820 0x7f7fac1f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.331+0000 7f7fb0f4a700 1 -- 172.21.15.1:0/1749490342 >> 172.21.15.1:0/1749490342 conn(0x7f7fac0b2ba0 msgr2=0x7f7fac068560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:47.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.331+0000 7f7fb0f4a700 1 -- 172.21.15.1:0/1749490342 shutdown_connections 2024-03-18T19:06:47.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.331+0000 7f7fb0f4a700 1 -- 172.21.15.1:0/1749490342 wait complete. 2024-03-18T19:06:47.330 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 49 2024-03-18T19:06:47.743 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T19:06:41.899614+0000', 'modified': '2024-03-18T19:06:44.904965+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7755}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7755': {'gid': 7755, 'name': 'c', 'rank': 0, 'incarnation': 46, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6837/2226098738', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6835', 'nonce': 2226098738}, {'type': 'v1', 'addr': '172.21.15.1:6837', 'nonce': 2226098738}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T19:06:47.743 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T19:06:47.744 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T19:06:47.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.887+0000 7f12669da700 1 Processor -- start 2024-03-18T19:06:47.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.887+0000 7f12669da700 1 -- start start 2024-03-18T19:06:47.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.888+0000 7f12669da700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1260071d30 0x7f12600add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.888+0000 7f12669da700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f12600cb9e0 0x7f12600cbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.888+0000 7f12669da700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12600ce8f0 con 0x7f126015a1d0 2024-03-18T19:06:47.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.888+0000 7f12669da700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12600cdb00 con 0x7f12600cb9e0 2024-03-18T19:06:47.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.888+0000 7f12669da700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f126015a5b0 con 0x7f1260071d30 2024-03-18T19:06:47.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.888+0000 7f1264f77700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f12600cb9e0 0x7f12600cbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.888+0000 7f125ffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f126015a1d0 0x7f1260071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47162/0 (socket says 172.21.15.1:47162) 2024-03-18T19:06:47.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.888+0000 7f125ffff700 1 -- 172.21.15.1:0/3986616839 learned_addr learned my addr 172.21.15.1:0/3986616839 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:47.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.889+0000 7f1264f77700 1 -- 172.21.15.1:0/3986616839 >> v2:172.21.15.165:3300/0 conn(0x7f1260071d30 msgr2=0x7f12600add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:47.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.889+0000 7f1264f77700 1 --2- 172.21.15.1:0/3986616839 >> v2:172.21.15.165:3300/0 conn(0x7f1260071d30 0x7f12600add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.889+0000 7f1264f77700 1 -- 172.21.15.1:0/3986616839 >> v1:172.21.15.1:6789/0 conn(0x7f126015a1d0 legacy=0x7f1260071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:47.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.889+0000 7f1264f77700 1 -- 172.21.15.1:0/3986616839 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12600ae260 con 0x7f12600cb9e0 2024-03-18T19:06:47.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.889+0000 7f1264f77700 1 --2- 172.21.15.1:0/3986616839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12600cb9e0 0x7f12600cbe30 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f12600cebb0 tx=0x7f125800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15fb5aff7a39648e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:47.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.889+0000 7f125effd700 1 -- 172.21.15.1:0/3986616839 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125800e070 con 0x7f12600cb9e0 2024-03-18T19:06:47.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.889+0000 7f125effd700 1 -- 172.21.15.1:0/3986616839 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125800bbd0 con 0x7f12600cb9e0 2024-03-18T19:06:47.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.889+0000 7f125effd700 1 -- 172.21.15.1:0/3986616839 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1258012430 con 0x7f12600cb9e0 2024-03-18T19:06:47.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.889+0000 7f12669da700 1 -- 172.21.15.1:0/3986616839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12600cb9e0 msgr2=0x7f12600cbe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:47.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.889+0000 7f12669da700 1 --2- 172.21.15.1:0/3986616839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12600cb9e0 0x7f12600cbe30 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f12600cebb0 tx=0x7f125800b330 comp rx=0 tx=0).stop 2024-03-18T19:06:47.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.889+0000 7f12669da700 1 -- 172.21.15.1:0/3986616839 shutdown_connections 2024-03-18T19:06:47.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.889+0000 7f12669da700 1 --2- 172.21.15.1:0/3986616839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12600cb9e0 0x7f12600cbe30 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.889+0000 7f12669da700 1 --2- 172.21.15.1:0/3986616839 >> v2:172.21.15.165:3300/0 conn(0x7f1260071d30 0x7f12600add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.890+0000 7f12669da700 1 -- 172.21.15.1:0/3986616839 >> 172.21.15.1:0/3986616839 conn(0x7f12600cc9b0 msgr2=0x7f12600ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:47.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.890+0000 7f12669da700 1 -- 172.21.15.1:0/3986616839 shutdown_connections 2024-03-18T19:06:47.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.890+0000 7f12669da700 1 -- 172.21.15.1:0/3986616839 wait complete. 2024-03-18T19:06:47.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.891+0000 7f12669da700 1 Processor -- start 2024-03-18T19:06:47.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.891+0000 7f12669da700 1 -- start start 2024-03-18T19:06:47.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.892+0000 7f12669da700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1260071d30 0x7f12601e2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.892+0000 7f12669da700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12600cb9e0 0x7f12600c6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.892+0000 7f12669da700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f126015a1d0 0x7f126015ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.892+0000 7f12669da700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12601e3410 con 0x7f126015a1d0 2024-03-18T19:06:47.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.892+0000 7f12669da700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12600c6950 con 0x7f1260071d30 2024-03-18T19:06:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.892+0000 7f12669da700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12601e48c0 con 0x7f12600cb9e0 2024-03-18T19:06:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.892+0000 7f1264f77700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f126015a1d0 0x7f126015ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.892+0000 7f1264f77700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f126015a1d0 0x7f126015ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49058/0 (socket says 172.21.15.1:49058) 2024-03-18T19:06:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.892+0000 7f1264f77700 1 -- 172.21.15.1:0/1646389983 learned_addr learned my addr 172.21.15.1:0/1646389983 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.892+0000 7f125ffff700 1 --2- 172.21.15.1:0/1646389983 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1260071d30 0x7f12601e2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.892+0000 7f125f7fe700 1 --2- 172.21.15.1:0/1646389983 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12600cb9e0 0x7f12600c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.892+0000 7f1264f77700 1 -- 172.21.15.1:0/1646389983 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12600cb9e0 msgr2=0x7f12600c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.892+0000 7f1264f77700 1 --2- 172.21.15.1:0/1646389983 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12600cb9e0 0x7f12600c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.892+0000 7f1264f77700 1 -- 172.21.15.1:0/1646389983 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1260071d30 msgr2=0x7f12601e2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.892+0000 7f1264f77700 1 --2- 172.21.15.1:0/1646389983 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1260071d30 0x7f12601e2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.892+0000 7f1264f77700 1 -- 172.21.15.1:0/1646389983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1258009a70 con 0x7f126015a1d0 2024-03-18T19:06:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.892+0000 7f125f7fe700 1 --2- 172.21.15.1:0/1646389983 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12600cb9e0 0x7f12600c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.892+0000 7f125ffff700 1 --2- 172.21.15.1:0/1646389983 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1260071d30 0x7f12601e2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.892+0000 7f1264f77700 1 --2- 172.21.15.1:0/1646389983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f126015a1d0 0x7f126015ced0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f1258014040 tx=0x7f1258004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:47.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.893+0000 7f125cff9700 1 -- 172.21.15.1:0/1646389983 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1258012430 con 0x7f126015a1d0 2024-03-18T19:06:47.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.893+0000 7f125cff9700 1 -- 172.21.15.1:0/1646389983 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1258020070 con 0x7f126015a1d0 2024-03-18T19:06:47.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.893+0000 7f12669da700 1 -- 172.21.15.1:0/1646389983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12601e3130 con 0x7f126015a1d0 2024-03-18T19:06:47.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.893+0000 7f125cff9700 1 -- 172.21.15.1:0/1646389983 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1258004280 con 0x7f126015a1d0 2024-03-18T19:06:47.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.893+0000 7f12669da700 1 -- 172.21.15.1:0/1646389983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12601e4e40 con 0x7f126015a1d0 2024-03-18T19:06:47.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.894+0000 7f125cff9700 1 -- 172.21.15.1:0/1646389983 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f12580043e0 con 0x7f126015a1d0 2024-03-18T19:06:47.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.895+0000 7f125cff9700 1 --2- 172.21.15.1:0/1646389983 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f124006adb0 0x7f124006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.895+0000 7f125ffff700 1 --2- 172.21.15.1:0/1646389983 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f124006adb0 0x7f124006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.895+0000 7f125cff9700 1 -- 172.21.15.1:0/1646389983 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1258022070 con 0x7f126015a1d0 2024-03-18T19:06:47.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.895+0000 7f125ffff700 1 --2- 172.21.15.1:0/1646389983 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f124006adb0 0x7f124006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f12480010e0 tx=0x7f1248009040 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:47.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.897+0000 7f1272b83b80 1 Processor -- start 2024-03-18T19:06:47.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.897+0000 7f1272b83b80 1 -- start start 2024-03-18T19:06:47.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.898+0000 7f1272b83b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bbffc10 0x55a08bbffff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.898+0000 7f1272b83b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bbd10a0 0x55a08bc1c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.898+0000 7f1272b83b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bc11630 0x55a08bbd4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.898+0000 7f1272b83b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a08bb4af20 con 0x55a08bc11630 2024-03-18T19:06:47.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.898+0000 7f1272b83b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a08bb2cc30 con 0x55a08bbffc10 2024-03-18T19:06:47.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.898+0000 7f1272b83b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a08bb2a430 con 0x55a08bbd10a0 2024-03-18T19:06:47.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.898+0000 7f1264f77700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bc11630 0x55a08bbd4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.898+0000 7f1264f77700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bc11630 0x55a08bbd4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49068/0 (socket says 172.21.15.1:49068) 2024-03-18T19:06:47.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.898+0000 7f1264f77700 1 -- 172.21.15.1:0/2110033403 learned_addr learned my addr 172.21.15.1:0/2110033403 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:47.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.898+0000 7f125f7fe700 1 --2- 172.21.15.1:0/2110033403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bbd10a0 0x55a08bc1c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.898+0000 7f125ffff700 1 --2- 172.21.15.1:0/2110033403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bbffc10 0x55a08bbffff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.898+0000 7f1264f77700 1 -- 172.21.15.1:0/2110033403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bbd10a0 msgr2=0x55a08bc1c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:47.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.898+0000 7f1264f77700 1 --2- 172.21.15.1:0/2110033403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bbd10a0 0x55a08bc1c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.898+0000 7f1264f77700 1 -- 172.21.15.1:0/2110033403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bbffc10 msgr2=0x55a08bbffff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:47.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.898+0000 7f1264f77700 1 --2- 172.21.15.1:0/2110033403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bbffc10 0x55a08bbffff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.898+0000 7f1264f77700 1 -- 172.21.15.1:0/2110033403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a08b919790 con 0x55a08bc11630 2024-03-18T19:06:47.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.898+0000 7f125ffff700 1 --2- 172.21.15.1:0/2110033403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bbffc10 0x55a08bbffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:47.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.898+0000 7f125f7fe700 1 --2- 172.21.15.1:0/2110033403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bbd10a0 0x55a08bc1c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:47.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.898+0000 7f1264f77700 1 --2- 172.21.15.1:0/2110033403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bc11630 0x55a08bbd4160 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f1258061fe0 tx=0x7f1258059b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b422371397c6a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:47.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.899+0000 7f1255ffb700 1 -- 172.21.15.1:0/2110033403 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125805c070 con 0x55a08bc11630 2024-03-18T19:06:47.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.899+0000 7f1255ffb700 1 -- 172.21.15.1:0/2110033403 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125805df50 con 0x55a08bc11630 2024-03-18T19:06:47.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.899+0000 7f1255ffb700 1 -- 172.21.15.1:0/2110033403 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1258062510 con 0x55a08bc11630 2024-03-18T19:06:47.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.899+0000 7f1272b83b80 1 -- 172.21.15.1:0/2110033403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bc11630 msgr2=0x55a08bbd4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:47.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.899+0000 7f1272b83b80 1 --2- 172.21.15.1:0/2110033403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bc11630 0x55a08bbd4160 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f1258061fe0 tx=0x7f1258059b50 comp rx=0 tx=0).stop 2024-03-18T19:06:47.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.899+0000 7f1272b83b80 1 -- 172.21.15.1:0/2110033403 shutdown_connections 2024-03-18T19:06:47.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.899+0000 7f1272b83b80 1 --2- 172.21.15.1:0/2110033403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bc11630 0x55a08bbd4160 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.899+0000 7f1272b83b80 1 --2- 172.21.15.1:0/2110033403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bbffc10 0x55a08bbffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.899+0000 7f1272b83b80 1 --2- 172.21.15.1:0/2110033403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bbd10a0 0x55a08bc1c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.899+0000 7f1272b83b80 1 -- 172.21.15.1:0/2110033403 >> 172.21.15.1:0/2110033403 conn(0x55a08bb33cf0 msgr2=0x55a08bbd3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:47.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.899+0000 7f1272b83b80 1 -- 172.21.15.1:0/2110033403 shutdown_connections 2024-03-18T19:06:47.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.899+0000 7f1272b83b80 1 -- 172.21.15.1:0/2110033403 wait complete. 2024-03-18T19:06:47.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.900+0000 7f1272b83b80 1 -- start start 2024-03-18T19:06:47.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.900+0000 7f1272b83b80 1 Processor -- start 2024-03-18T19:06:47.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f1272b83b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bbd10a0 0x55a08bbf4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f1272b83b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bbffc10 0x55a08bbf4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f1272b83b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bc11630 0x55a08bc01b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f1272b83b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a08bb25d20 con 0x55a08bc11630 2024-03-18T19:06:47.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f1272b83b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a08bac1770 con 0x55a08bbd10a0 2024-03-18T19:06:47.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f1272b83b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a08bae46f0 con 0x55a08bbffc10 2024-03-18T19:06:47.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f1272b83b80 10 client.0 _ll_register_callbacks cb 0x55a08b92d5a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T19:06:47.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f1264f77700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bc11630 0x55a08bc01b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f1264f77700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bc11630 0x55a08bc01b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49076/0 (socket says 172.21.15.1:49076) 2024-03-18T19:06:47.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f1264f77700 1 -- 172.21.15.1:0/2253314713 learned_addr learned my addr 172.21.15.1:0/2253314713 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:47.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f125f7fe700 1 --2- 172.21.15.1:0/2253314713 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bbffc10 0x55a08bbf4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f125ffff700 1 --2- 172.21.15.1:0/2253314713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bbd10a0 0x55a08bbf4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f1264f77700 1 -- 172.21.15.1:0/2253314713 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bbffc10 msgr2=0x55a08bbf4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:47.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f1264f77700 1 --2- 172.21.15.1:0/2253314713 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bbffc10 0x55a08bbf4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f1264f77700 1 -- 172.21.15.1:0/2253314713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bbd10a0 msgr2=0x55a08bbf4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:47.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f1264f77700 1 --2- 172.21.15.1:0/2253314713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bbd10a0 0x55a08bbf4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f1264f77700 1 -- 172.21.15.1:0/2253314713 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f125805ee70 con 0x55a08bc11630 2024-03-18T19:06:47.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f1264f77700 1 --2- 172.21.15.1:0/2253314713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bc11630 0x55a08bc01b40 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f1258056620 tx=0x7f1258009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3798f8cf598ad6cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:47.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f123effd700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T19:06:47.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f123effd700 1 -- 172.21.15.1:0/2253314713 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125805c040 con 0x55a08bc11630 2024-03-18T19:06:47.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f123effd700 1 -- 172.21.15.1:0/2253314713 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(117..117 src has 1..117) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f125808b2c0 con 0x55a08bc11630 2024-03-18T19:06:47.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.902+0000 7f1272b83b80 1 -- 172.21.15.1:0/2253314713 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a08bbc7e90 con 0x55a08bc11630 2024-03-18T19:06:47.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.903+0000 7f123effd700 1 -- 172.21.15.1:0/2253314713 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f125805a580 con 0x55a08bc11630 2024-03-18T19:06:47.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.903+0000 7f1272b83b80 10 client.7905 fetch_fsmap learned FSMap version 49 2024-03-18T19:06:47.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.903+0000 7f1272b83b80 1 -- 172.21.15.1:0/2253314713 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x55a08ba060a0 con 0x55a08bc11630 2024-03-18T19:06:47.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.903+0000 7f123effd700 1 -- 172.21.15.1:0/2253314713 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a08bbc7e90 con 0x55a08bc11630 2024-03-18T19:06:47.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.903+0000 7f123effd700 1 -- 172.21.15.1:0/2253314713 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1258059610 con 0x55a08bc11630 2024-03-18T19:06:47.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.903+0000 7f1272b83b80 10 client.7905 fetch_fsmap finished waiting for FSMap version 49 2024-03-18T19:06:47.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.903+0000 7f1272b83b80 10 client.7905 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T19:06:47.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.903+0000 7f1272b83b80 20 client.7905 populate_metadata read hostname 'smithi001' 2024-03-18T19:06:47.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.903+0000 7f1272b83b80 1 --2- 172.21.15.1:0/2253314713 >> [v2:172.21.15.1:6835/2226098738,v1:172.21.15.1:6837/2226098738] conn(0x55a08bbd5050 0x55a08bbf7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.903+0000 7f1272b83b80 4 client.7905 mds_command: new command op to 7755 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T19:06:47.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.903+0000 7f1272b83b80 1 -- 172.21.15.1:0/2253314713 --> [v2:172.21.15.1:6835/2226098738,v1:172.21.15.1:6837/2226098738] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a08bba5010 con 0x55a08bbd5050 2024-03-18T19:06:47.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.903+0000 7f125ffff700 1 --2- 172.21.15.1:0/2253314713 >> [v2:172.21.15.1:6835/2226098738,v1:172.21.15.1:6837/2226098738] conn(0x55a08bbd5050 0x55a08bbf7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.903+0000 7f125ffff700 1 --2- 172.21.15.1:0/2253314713 >> [v2:172.21.15.1:6835/2226098738,v1:172.21.15.1:6837/2226098738] conn(0x55a08bbd5050 0x55a08bbf7200 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5e47f2e4124fad5 server_cookie=7c9fc056c59be63a in_seq=0 out_seq=0 2024-03-18T19:06:47.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.903+0000 7f123effd700 10 client.7905 ms_handle_connect on v2:172.21.15.1:6835/2226098738 2024-03-18T19:06:47.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.904+0000 7f123effd700 1 -- 172.21.15.1:0/2253314713 <== mds.0 v2:172.21.15.1:6835/2226098738 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a08bba5010 con 0x55a08bbd5050 2024-03-18T19:06:47.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.904+0000 7f123effd700 10 client.7905 handle_command_reply: tid=0 2024-03-18T19:06:47.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.904+0000 7f1272b83b80 1 client.7905 shutdown 2024-03-18T19:06:47.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.904+0000 7f1272b83b80 1 -- 172.21.15.1:0/2253314713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bc11630 msgr2=0x55a08bc01b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:47.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.904+0000 7f1272b83b80 1 --2- 172.21.15.1:0/2253314713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bc11630 0x55a08bc01b40 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f1258056620 tx=0x7f1258009750 comp rx=0 tx=0).stop 2024-03-18T19:06:47.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.904+0000 7f1272b83b80 1 -- 172.21.15.1:0/2253314713 shutdown_connections 2024-03-18T19:06:47.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.904+0000 7f1272b83b80 1 --2- 172.21.15.1:0/2253314713 >> [v2:172.21.15.1:6835/2226098738,v1:172.21.15.1:6837/2226098738] conn(0x55a08bbd5050 0x55a08bbf7200 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.904+0000 7f1272b83b80 1 --2- 172.21.15.1:0/2253314713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bc11630 0x55a08bc01b40 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.904+0000 7f1272b83b80 1 --2- 172.21.15.1:0/2253314713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bbd10a0 0x55a08bbf4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.904+0000 7f123effd700 0 client.7905 ms_handle_reset on v2:172.21.15.1:6835/2226098738 2024-03-18T19:06:47.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.904+0000 7f1272b83b80 1 --2- 172.21.15.1:0/2253314713 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bbffc10 0x55a08bbf4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.904+0000 7f1272b83b80 1 -- 172.21.15.1:0/2253314713 >> 172.21.15.1:0/2253314713 conn(0x55a08bb33cf0 msgr2=0x55a08bc00290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:47.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.904+0000 7f1272b83b80 1 -- 172.21.15.1:0/2253314713 shutdown_connections 2024-03-18T19:06:47.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.904+0000 7f1272b83b80 1 -- 172.21.15.1:0/2253314713 wait complete. 2024-03-18T19:06:47.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.904+0000 7f1272b83b80 20 client.7905 trim_cache size 0 max 16384 2024-03-18T19:06:47.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.924+0000 7f1272b83b80 1 Processor -- start 2024-03-18T19:06:47.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.924+0000 7f1272b83b80 1 -- start start 2024-03-18T19:06:47.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.925+0000 7f1272b83b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bbd10a0 0x55a08bbd2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.925+0000 7f1272b83b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bbffc10 0x55a08bc11fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.925+0000 7f1272b83b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bc11630 0x55a08bc12500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.925+0000 7f1272b83b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a08b91a390 con 0x55a08bbd10a0 2024-03-18T19:06:47.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.925+0000 7f1272b83b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a08bbd3470 con 0x55a08bc11630 2024-03-18T19:06:47.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.925+0000 7f1272b83b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a08bbd0a10 con 0x55a08bbffc10 2024-03-18T19:06:47.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.925+0000 7f125ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bbd10a0 0x55a08bbd2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.925+0000 7f125ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bbd10a0 0x55a08bbd2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49082/0 (socket says 172.21.15.1:49082) 2024-03-18T19:06:47.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.925+0000 7f125ffff700 1 -- 172.21.15.1:0/835573002 learned_addr learned my addr 172.21.15.1:0/835573002 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:47.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.925+0000 7f125f7fe700 1 --2- 172.21.15.1:0/835573002 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bbffc10 0x55a08bc11fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.925+0000 7f125ffff700 1 -- 172.21.15.1:0/835573002 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bbffc10 msgr2=0x55a08bc11fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.925+0000 7f1264f77700 1 --2- 172.21.15.1:0/835573002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bc11630 0x55a08bc12500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.925+0000 7f125ffff700 1 --2- 172.21.15.1:0/835573002 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bbffc10 0x55a08bc11fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.925+0000 7f125ffff700 1 -- 172.21.15.1:0/835573002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bc11630 msgr2=0x55a08bc12500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.925+0000 7f125ffff700 1 --2- 172.21.15.1:0/835573002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bc11630 0x55a08bc12500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.925+0000 7f125ffff700 1 -- 172.21.15.1:0/835573002 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a08baec010 con 0x55a08bbd10a0 2024-03-18T19:06:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.925+0000 7f125f7fe700 1 --2- 172.21.15.1:0/835573002 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bbffc10 0x55a08bc11fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.926+0000 7f125ffff700 1 --2- 172.21.15.1:0/835573002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bbd10a0 0x55a08bbd2e50 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x55a08bbf5210 tx=0x7f1248011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fadfab5579581aaf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.926+0000 7f12567fc700 1 -- 172.21.15.1:0/835573002 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12480073a0 con 0x55a08bbd10a0 2024-03-18T19:06:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.926+0000 7f12567fc700 1 -- 172.21.15.1:0/835573002 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1248029070 con 0x55a08bbd10a0 2024-03-18T19:06:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.926+0000 7f12567fc700 1 -- 172.21.15.1:0/835573002 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248005e10 con 0x55a08bbd10a0 2024-03-18T19:06:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.926+0000 7f1272b83b80 1 -- 172.21.15.1:0/835573002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bbd10a0 msgr2=0x55a08bbd2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.926+0000 7f1272b83b80 1 --2- 172.21.15.1:0/835573002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bbd10a0 0x55a08bbd2e50 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x55a08bbf5210 tx=0x7f1248011000 comp rx=0 tx=0).stop 2024-03-18T19:06:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.927+0000 7f1272b83b80 1 -- 172.21.15.1:0/835573002 shutdown_connections 2024-03-18T19:06:47.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.927+0000 7f1272b83b80 1 --2- 172.21.15.1:0/835573002 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bc11630 0x55a08bc12500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.927+0000 7f1272b83b80 1 --2- 172.21.15.1:0/835573002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bbd10a0 0x55a08bbd2e50 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.927+0000 7f1272b83b80 1 --2- 172.21.15.1:0/835573002 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bbffc10 0x55a08bc11fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.927+0000 7f1272b83b80 1 -- 172.21.15.1:0/835573002 >> 172.21.15.1:0/835573002 conn(0x55a08bb33cf0 msgr2=0x55a08bbfd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:47.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.927+0000 7f1272b83b80 1 -- 172.21.15.1:0/835573002 shutdown_connections 2024-03-18T19:06:47.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.927+0000 7f1272b83b80 1 -- 172.21.15.1:0/835573002 wait complete. 2024-03-18T19:06:47.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.927+0000 7f1272b83b80 1 -- start start 2024-03-18T19:06:47.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.927+0000 7f1272b83b80 1 Processor -- start 2024-03-18T19:06:47.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.928+0000 7f1272b83b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bbd10a0 0x55a08bbcf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.928+0000 7f1272b83b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bbffc10 0x55a08bbf8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.928+0000 7f1272b83b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bc11630 0x55a08bc33a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.928+0000 7f1272b83b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a08bbf9130 con 0x55a08bbffc10 2024-03-18T19:06:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.928+0000 7f1272b83b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a08bbd6b40 con 0x55a08bbd10a0 2024-03-18T19:06:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.928+0000 7f1272b83b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a08bbd6d80 con 0x55a08bc11630 2024-03-18T19:06:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.928+0000 7f1272b83b80 10 client.0 _ll_register_callbacks cb 0x7f1258059d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T19:06:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.928+0000 7f125f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bbffc10 0x55a08bbf8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.928+0000 7f125f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bbffc10 0x55a08bbf8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49090/0 (socket says 172.21.15.1:49090) 2024-03-18T19:06:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.928+0000 7f125f7fe700 1 -- 172.21.15.1:0/1015741790 learned_addr learned my addr 172.21.15.1:0/1015741790 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.928+0000 7f125ffff700 1 --2- 172.21.15.1:0/1015741790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bbd10a0 0x55a08bbcf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.928+0000 7f1264f77700 1 --2- 172.21.15.1:0/1015741790 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bc11630 0x55a08bc33a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f125f7fe700 1 -- 172.21.15.1:0/1015741790 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bc11630 msgr2=0x55a08bc33a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f125f7fe700 1 --2- 172.21.15.1:0/1015741790 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bc11630 0x55a08bc33a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f125f7fe700 1 -- 172.21.15.1:0/1015741790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bbd10a0 msgr2=0x55a08bbcf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f125f7fe700 1 --2- 172.21.15.1:0/1015741790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bbd10a0 0x55a08bbcf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f125f7fe700 1 -- 172.21.15.1:0/1015741790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f124801c040 con 0x55a08bbffc10 2024-03-18T19:06:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f125f7fe700 1 --2- 172.21.15.1:0/1015741790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bbffc10 0x55a08bbf8970 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f1250012550 tx=0x7f1250012860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d427364f6437074 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f123effd700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T19:06:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f123effd700 1 -- 172.21.15.1:0/1015741790 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1250015450 con 0x55a08bbffc10 2024-03-18T19:06:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f123effd700 1 -- 172.21.15.1:0/1015741790 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(117..117 src has 1..117) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f12500155b0 con 0x55a08bbffc10 2024-03-18T19:06:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f1272b83b80 1 -- 172.21.15.1:0/1015741790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f125805df50 con 0x55a08bbffc10 2024-03-18T19:06:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f123effd700 1 -- 172.21.15.1:0/1015741790 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1250015790 con 0x55a08bbffc10 2024-03-18T19:06:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f1272b83b80 10 client.7911 fetch_fsmap learned FSMap version 49 2024-03-18T19:06:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f1272b83b80 1 -- 172.21.15.1:0/1015741790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x55a08b97a1d0 con 0x55a08bbffc10 2024-03-18T19:06:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f123effd700 1 -- 172.21.15.1:0/1015741790 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125805df50 con 0x55a08bbffc10 2024-03-18T19:06:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f123effd700 1 -- 172.21.15.1:0/1015741790 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f12500158f0 con 0x55a08bbffc10 2024-03-18T19:06:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f1272b83b80 10 client.7911 fetch_fsmap finished waiting for FSMap version 49 2024-03-18T19:06:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f1272b83b80 10 client.7911 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T19:06:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f1272b83b80 20 client.7911 populate_metadata read hostname 'smithi001' 2024-03-18T19:06:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f1272b83b80 1 --2- 172.21.15.1:0/1015741790 >> [v2:172.21.15.1:6835/2226098738,v1:172.21.15.1:6837/2226098738] conn(0x55a08bbd37d0 0x55a08bbd3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f1272b83b80 4 client.7911 mds_command: new command op to 7755 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T19:06:47.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.929+0000 7f1272b83b80 1 -- 172.21.15.1:0/1015741790 --> [v2:172.21.15.1:6835/2226098738,v1:172.21.15.1:6837/2226098738] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a08b95f040 con 0x55a08bbd37d0 2024-03-18T19:06:47.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.930+0000 7f125ffff700 1 --2- 172.21.15.1:0/1015741790 >> [v2:172.21.15.1:6835/2226098738,v1:172.21.15.1:6837/2226098738] conn(0x55a08bbd37d0 0x55a08bbd3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:47.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.930+0000 7f123effd700 10 client.7911 ms_handle_connect on v2:172.21.15.1:6835/2226098738 2024-03-18T19:06:47.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.930+0000 7f125ffff700 1 --2- 172.21.15.1:0/1015741790 >> [v2:172.21.15.1:6835/2226098738,v1:172.21.15.1:6837/2226098738] conn(0x55a08bbd37d0 0x55a08bbd3bb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25b9b9dbf1cbf797 server_cookie=d5266deec7607f37 in_seq=0 out_seq=0 2024-03-18T19:06:47.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.930+0000 7f123effd700 1 -- 172.21.15.1:0/1015741790 <== mds.0 v2:172.21.15.1:6835/2226098738 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55a08b95f040 con 0x55a08bbd37d0 2024-03-18T19:06:47.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.930+0000 7f123effd700 10 client.7911 handle_command_reply: tid=0 2024-03-18T19:06:47.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.930+0000 7f1272b83b80 1 client.7911 shutdown 2024-03-18T19:06:47.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.930+0000 7f1272b83b80 1 -- 172.21.15.1:0/1015741790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bbffc10 msgr2=0x55a08bbf8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:47.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.930+0000 7f1272b83b80 1 --2- 172.21.15.1:0/1015741790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bbffc10 0x55a08bbf8970 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f1250012550 tx=0x7f1250012860 comp rx=0 tx=0).stop 2024-03-18T19:06:47.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.930+0000 7f1272b83b80 1 -- 172.21.15.1:0/1015741790 shutdown_connections 2024-03-18T19:06:47.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.930+0000 7f1272b83b80 1 --2- 172.21.15.1:0/1015741790 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55a08bc11630 0x55a08bc33a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.930+0000 7f1272b83b80 1 --2- 172.21.15.1:0/1015741790 >> [v2:172.21.15.1:6835/2226098738,v1:172.21.15.1:6837/2226098738] conn(0x55a08bbd37d0 0x55a08bbd3bb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.930+0000 7f1272b83b80 1 --2- 172.21.15.1:0/1015741790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a08bbd10a0 0x55a08bbcf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.930+0000 7f1272b83b80 1 --2- 172.21.15.1:0/1015741790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a08bbffc10 0x55a08bbf8970 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.930+0000 7f123effd700 0 client.7911 ms_handle_reset on v2:172.21.15.1:6835/2226098738 2024-03-18T19:06:47.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.930+0000 7f1272b83b80 1 -- 172.21.15.1:0/1015741790 >> 172.21.15.1:0/1015741790 conn(0x55a08bb33cf0 msgr2=0x55a08bc11fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:47.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.930+0000 7f1272b83b80 1 -- 172.21.15.1:0/1015741790 shutdown_connections 2024-03-18T19:06:47.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.930+0000 7f1272b83b80 1 -- 172.21.15.1:0/1015741790 wait complete. 2024-03-18T19:06:47.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.931+0000 7f1272b83b80 20 client.7911 trim_cache size 0 max 16384 2024-03-18T19:06:47.934 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:06:47.934 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "9c1c697f-8655-4ef3-8a13-da568d2cbe86", 2024-03-18T19:06:47.934 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-18T19:06:47.934 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 7755, 2024-03-18T19:06:47.934 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-18T19:06:47.934 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-18T19:06:47.934 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-18T19:06:47.935 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 6.0307115080000004, 2024-03-18T19:06:47.935 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 49, 2024-03-18T19:06:47.935 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 117, 2024-03-18T19:06:47.935 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 117, 2024-03-18T19:06:47.935 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 11.335968371 2024-03-18T19:06:47.935 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:06:47.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.931+0000 7f12669da700 1 -- 172.21.15.1:0/1646389983 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f124006adb0 msgr2=0x7f124006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:47.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.931+0000 7f12669da700 1 --2- 172.21.15.1:0/1646389983 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f124006adb0 0x7f124006d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f12480010e0 tx=0x7f1248009040 comp rx=0 tx=0).stop 2024-03-18T19:06:47.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.931+0000 7f12669da700 1 -- 172.21.15.1:0/1646389983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f126015a1d0 msgr2=0x7f126015ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:47.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.931+0000 7f12669da700 1 --2- 172.21.15.1:0/1646389983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f126015a1d0 0x7f126015ced0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f1258014040 tx=0x7f1258004ad0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.931+0000 7f12669da700 1 -- 172.21.15.1:0/1646389983 shutdown_connections 2024-03-18T19:06:47.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.931+0000 7f12669da700 1 --2- 172.21.15.1:0/1646389983 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f124006adb0 0x7f124006d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.931+0000 7f12669da700 1 --2- 172.21.15.1:0/1646389983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f126015a1d0 0x7f126015ced0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.931+0000 7f12669da700 1 --2- 172.21.15.1:0/1646389983 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1260071d30 0x7f12601e2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.931+0000 7f12669da700 1 --2- 172.21.15.1:0/1646389983 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12600cb9e0 0x7f12600c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:47.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.931+0000 7f12669da700 1 -- 172.21.15.1:0/1646389983 >> 172.21.15.1:0/1646389983 conn(0x7f12600cc9b0 msgr2=0x7f12600ca560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:47.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.931+0000 7f12669da700 1 -- 172.21.15.1:0/1646389983 shutdown_connections 2024-03-18T19:06:47.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:47.931+0000 7f12669da700 1 -- 172.21.15.1:0/1646389983 wait complete. 2024-03-18T19:06:47.940 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T19:06:48.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.068+0000 7fb224f96700 1 Processor -- start 2024-03-18T19:06:48.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.068+0000 7fb224f96700 1 -- start start 2024-03-18T19:06:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.069+0000 7fb224f96700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2200b3980 0x7fb22006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.069+0000 7fb224f96700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb22015e0f0 0x7fb22006dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.069+0000 7fb224f96700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22015dc60 con 0x7fb220159620 2024-03-18T19:06:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.069+0000 7fb224f96700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb22015d400 con 0x7fb2200b3980 2024-03-18T19:06:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.069+0000 7fb224f96700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb22015ac20 con 0x7fb22015e0f0 2024-03-18T19:06:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.069+0000 7fb21dd9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2200b3980 0x7fb22006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.069+0000 7fb21e59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb220159620 0x7fb2200b3270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47178/0 (socket says 172.21.15.1:47178) 2024-03-18T19:06:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.069+0000 7fb21e59c700 1 -- 172.21.15.1:0/3076807195 learned_addr learned my addr 172.21.15.1:0/3076807195 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:48.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.069+0000 7fb21ed9d700 1 --2- 172.21.15.1:0/3076807195 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb22015e0f0 0x7fb22006dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.070+0000 7fb21dd9b700 1 -- 172.21.15.1:0/3076807195 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb22015e0f0 msgr2=0x7fb22006dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.070+0000 7fb21dd9b700 1 --2- 172.21.15.1:0/3076807195 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb22015e0f0 0x7fb22006dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.070+0000 7fb21d59a700 1 -- 172.21.15.1:0/3076807195 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765014856 0 0) 0x7fb22015dc60 con 0x7fb220159620 2024-03-18T19:06:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.070+0000 7fb21dd9b700 1 -- 172.21.15.1:0/3076807195 >> v1:172.21.15.1:6789/0 conn(0x7fb220159620 legacy=0x7fb2200b3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.070+0000 7fb21dd9b700 1 -- 172.21.15.1:0/3076807195 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22015d610 con 0x7fb2200b3980 2024-03-18T19:06:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.070+0000 7fb21ed9d700 1 --2- 172.21.15.1:0/3076807195 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb22015e0f0 0x7fb22006dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.070+0000 7fb21dd9b700 1 --2- 172.21.15.1:0/3076807195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2200b3980 0x7fb22006d6b0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fb210009820 tx=0x7fb210009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e6c1a111cccb8a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:48.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.070+0000 7fb21d59a700 1 -- 172.21.15.1:0/3076807195 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb210005e30 con 0x7fb2200b3980 2024-03-18T19:06:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.070+0000 7fb21d59a700 1 -- 172.21.15.1:0/3076807195 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb210006030 con 0x7fb2200b3980 2024-03-18T19:06:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.070+0000 7fb21d59a700 1 -- 172.21.15.1:0/3076807195 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2100063e0 con 0x7fb2200b3980 2024-03-18T19:06:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.070+0000 7fb224f96700 1 -- 172.21.15.1:0/3076807195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2200b3980 msgr2=0x7fb22006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.070+0000 7fb224f96700 1 --2- 172.21.15.1:0/3076807195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2200b3980 0x7fb22006d6b0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fb210009820 tx=0x7fb210009b30 comp rx=0 tx=0).stop 2024-03-18T19:06:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.071+0000 7fb224f96700 1 -- 172.21.15.1:0/3076807195 shutdown_connections 2024-03-18T19:06:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.071+0000 7fb224f96700 1 --2- 172.21.15.1:0/3076807195 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb22015e0f0 0x7fb22006dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.071+0000 7fb224f96700 1 --2- 172.21.15.1:0/3076807195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2200b3980 0x7fb22006d6b0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.071+0000 7fb224f96700 1 -- 172.21.15.1:0/3076807195 >> 172.21.15.1:0/3076807195 conn(0x7fb22006a750 msgr2=0x7fb22006ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:48.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.071+0000 7fb224f96700 1 -- 172.21.15.1:0/3076807195 shutdown_connections 2024-03-18T19:06:48.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.071+0000 7fb224f96700 1 -- 172.21.15.1:0/3076807195 wait complete. 2024-03-18T19:06:48.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.072+0000 7fb224f96700 1 Processor -- start 2024-03-18T19:06:48.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.072+0000 7fb224f96700 1 -- start start 2024-03-18T19:06:48.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb224f96700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2200b3980 0x7fb2200c46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:48.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb224f96700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb220159620 0x7fb2201f2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:48.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb224f96700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb22015e0f0 0x7fb2201f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:48.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb224f96700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2201f30c0 con 0x7fb22015e0f0 2024-03-18T19:06:48.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb224f96700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2200c4d20 con 0x7fb220159620 2024-03-18T19:06:48.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb224f96700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2200cccc0 con 0x7fb2200b3980 2024-03-18T19:06:48.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb21ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb22015e0f0 0x7fb2201f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:48.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb21ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb22015e0f0 0x7fb2201f2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49112/0 (socket says 172.21.15.1:49112) 2024-03-18T19:06:48.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb21ed9d700 1 -- 172.21.15.1:0/1284867134 learned_addr learned my addr 172.21.15.1:0/1284867134 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:48.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb21e59c700 1 --2- 172.21.15.1:0/1284867134 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2200b3980 0x7fb2200c46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:48.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb21dd9b700 1 --2- 172.21.15.1:0/1284867134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb220159620 0x7fb2201f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:48.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb21ed9d700 1 -- 172.21.15.1:0/1284867134 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2200b3980 msgr2=0x7fb2200c46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:48.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb21ed9d700 1 --2- 172.21.15.1:0/1284867134 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2200b3980 0x7fb2200c46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:48.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb21ed9d700 1 -- 172.21.15.1:0/1284867134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb220159620 msgr2=0x7fb2201f2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:48.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb21ed9d700 1 --2- 172.21.15.1:0/1284867134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb220159620 0x7fb2201f2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:48.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb21ed9d700 1 -- 172.21.15.1:0/1284867134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb210009430 con 0x7fb22015e0f0 2024-03-18T19:06:48.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb21dd9b700 1 --2- 172.21.15.1:0/1284867134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb220159620 0x7fb2201f2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:48.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb21ed9d700 1 --2- 172.21.15.1:0/1284867134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb22015e0f0 0x7fb2201f2b80 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fb21800cd50 tx=0x7fb21800a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:48.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb20effd700 1 -- 172.21.15.1:0/1284867134 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21800f480 con 0x7fb22015e0f0 2024-03-18T19:06:48.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.073+0000 7fb20effd700 1 -- 172.21.15.1:0/1284867134 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21800f600 con 0x7fb22015e0f0 2024-03-18T19:06:48.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.074+0000 7fb224f96700 1 -- 172.21.15.1:0/1284867134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2200ccfa0 con 0x7fb22015e0f0 2024-03-18T19:06:48.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.074+0000 7fb20effd700 1 -- 172.21.15.1:0/1284867134 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21800f9b0 con 0x7fb22015e0f0 2024-03-18T19:06:48.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.074+0000 7fb224f96700 1 -- 172.21.15.1:0/1284867134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2200cd560 con 0x7fb22015e0f0 2024-03-18T19:06:48.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.074+0000 7fb224f96700 1 -- 172.21.15.1:0/1284867134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2200676a0 con 0x7fb22015e0f0 2024-03-18T19:06:48.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.075+0000 7fb20effd700 1 -- 172.21.15.1:0/1284867134 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb21800b040 con 0x7fb22015e0f0 2024-03-18T19:06:48.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.076+0000 7fb20effd700 1 --2- 172.21.15.1:0/1284867134 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb20006adb0 0x7fb20006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:48.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.076+0000 7fb21e59c700 1 --2- 172.21.15.1:0/1284867134 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb20006adb0 0x7fb20006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:48.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.076+0000 7fb20effd700 1 -- 172.21.15.1:0/1284867134 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb218089f20 con 0x7fb22015e0f0 2024-03-18T19:06:48.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.076+0000 7fb21e59c700 1 --2- 172.21.15.1:0/1284867134 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb20006adb0 0x7fb20006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb214000f60 tx=0x7fb214007be0 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:48.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.080+0000 7fb20effd700 1 -- 172.21.15.1:0/1284867134 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb218054da0 con 0x7fb22015e0f0 2024-03-18T19:06:48.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.327+0000 7fb224f96700 1 -- 172.21.15.1:0/1284867134 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb22006ea00 con 0x7fb22015e0f0 2024-03-18T19:06:48.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.328+0000 7fb20effd700 1 -- 172.21.15.1:0/1284867134 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v49) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb21801d020 con 0x7fb22015e0f0 2024-03-18T19:06:48.325 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T19:06:48.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.329+0000 7fb224f96700 1 -- 172.21.15.1:0/1284867134 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb20006adb0 msgr2=0x7fb20006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:48.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.329+0000 7fb224f96700 1 --2- 172.21.15.1:0/1284867134 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb20006adb0 0x7fb20006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb214000f60 tx=0x7fb214007be0 comp rx=0 tx=0).stop 2024-03-18T19:06:48.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.329+0000 7fb224f96700 1 -- 172.21.15.1:0/1284867134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb22015e0f0 msgr2=0x7fb2201f2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:48.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.329+0000 7fb224f96700 1 --2- 172.21.15.1:0/1284867134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb22015e0f0 0x7fb2201f2b80 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fb21800cd50 tx=0x7fb21800a560 comp rx=0 tx=0).stop 2024-03-18T19:06:48.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.330+0000 7fb224f96700 1 -- 172.21.15.1:0/1284867134 shutdown_connections 2024-03-18T19:06:48.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.330+0000 7fb224f96700 1 --2- 172.21.15.1:0/1284867134 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb20006adb0 0x7fb20006d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:48.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.330+0000 7fb224f96700 1 --2- 172.21.15.1:0/1284867134 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2200b3980 0x7fb2200c46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:48.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.330+0000 7fb224f96700 1 --2- 172.21.15.1:0/1284867134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb22015e0f0 0x7fb2201f2b80 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:48.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.330+0000 7fb224f96700 1 --2- 172.21.15.1:0/1284867134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb220159620 0x7fb2201f2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:48.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.330+0000 7fb224f96700 1 -- 172.21.15.1:0/1284867134 >> 172.21.15.1:0/1284867134 conn(0x7fb22006a750 msgr2=0x7fb2200cbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:48.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.330+0000 7fb224f96700 1 -- 172.21.15.1:0/1284867134 shutdown_connections 2024-03-18T19:06:48.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:48.330+0000 7fb224f96700 1 -- 172.21.15.1:0/1284867134 wait complete. 2024-03-18T19:06:48.338 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T19:06:48.339 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T19:06:48.339 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-18T19:06:48.339 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T19:06:48.339 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T19:06:48.339 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T19:06:48.339 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T19:06:48.339 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T19:06:48.339 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:06:48.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:48.339 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-18T19:06:48.359 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T19:06:48.359 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T19:06:48.359 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T19:06:48.359 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:06:48.359 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T19:06:48.359 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:06:48.359 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:06:48.359 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-18T19:06:48.359 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-18T19:06:48.359 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:06:48.359 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-18T19:06:48.360 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-18T19:06:48.360 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T19:06:48.360 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:06:48.360 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-18T19:06:48.360 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T19:06:48.360 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 41561sec preferred_lft 41561sec 2024-03-18T19:06:48.360 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link 2024-03-18T19:06:48.360 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:06:48.360 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:06:48.360 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:06:48.360 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-18T19:06:48.360 INFO:teuthology.orchestra.run.smithi001.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T19:06:48.360 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:06:48.361 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T19:06:48.361 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:06:48.361 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::4ca9:f5ff:feb3:79a3/64 scope link 2024-03-18T19:06:48.361 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:06:48.361 INFO:teuthology.orchestra.run.smithi001.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T19:06:48.361 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T19:06:48.361 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::282f:a4ff:fe1a:9908/64 scope link 2024-03-18T19:06:48.361 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:06:48.361 INFO:teuthology.orchestra.run.smithi001.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T19:06:48.361 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2e:5c:60:78:b2:8c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T19:06:48.361 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::2c5c:60ff:fe78:b28c/64 scope link 2024-03-18T19:06:48.361 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:06:48.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:48.362 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T19:06:48.425 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-18T19:06:48.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:48.426 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-18T19:06:48.487 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-18T19:06:48.487 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T19:06:48.487 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T19:06:48.487 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T19:06:48.487 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T19:06:48.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:48.487 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T19:06:48.487 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T19:06:48.488 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T19:06:48.488 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T19:06:48.488 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T19:06:48.488 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T19:06:48.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:48.594 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T19:06:48.613 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T19:06:48.613 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T19:06:48.613 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:06:48.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:48.613 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-18T19:06:48.680 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T19:06:48.680 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T19:06:48.680 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T19:06:48.739 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:06:48.739 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T19:06:48.739 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T19:06:48.798 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T19:06:48.867 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T19:06:48.867 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T19:06:48.933 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T19:06:48.934 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T19:06:48.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:48.935 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T19:06:48.993 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T19:06:48.994 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T19:06:49.036 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T19:06:49.064 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T19:06:49.064 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T19:06:49.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-18T19:06:49.127+0000 7f2011cb55c0 -1 init, newargv = 0x55f534ab5050 newargc=15 2024-03-18T19:06:49.124 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[120695]: starting ceph client 2024-03-18T19:06:49.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[120695]: starting fuse 2024-03-18T19:06:49.132 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T19:06:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T19:06:49.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:49.134 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T19:06:49.192 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T19:06:49.192 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-03-18T19:06:49.192 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T19:06:49.192 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-18T19:06:49.192 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-18T19:06:49.192 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-18T19:06:49.193 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-18T19:06:49.193 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:06:49.193 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-18T19:06:49.193 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T19:06:49.193 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-18T19:06:49.193 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T19:06:49.193 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:06:49.193 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-18T19:06:49.193 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-18T19:06:49.193 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-18T19:06:49.193 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-18T19:06:49.193 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:06:49.193 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-18T19:06:49.193 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T19:06:49.194 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T19:06:49.194 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T19:06:49.194 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-18T19:06:49.194 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-18T19:06:49.194 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-18T19:06:49.194 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T19:06:49.194 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:06:49.194 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-18T19:06:49.194 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-18T19:06:49.317 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.120695.asok 2024-03-18T19:06:49.499 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.120695.asok 2024-03-18T19:06:49.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:49.500 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.120695.asok status 2024-03-18T19:06:49.661 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:06:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-18T19:06:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T19:06:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T19:06:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-18T19:06:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-18T19:06:49.661 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T19:06:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "120695", 2024-03-18T19:06:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-18T19:06:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-18T19:06:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-18T19:06:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 6536, 2024-03-18T19:06:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-18T19:06:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-18T19:06:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-18T19:06:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6536 2024-03-18T19:06:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T19:06:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T19:06:49.662 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T19:06:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3726143884 2024-03-18T19:06:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T19:06:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T19:06:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T19:06:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3726143884 2024-03-18T19:06:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:06:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.6536 192.168.0.1:0/3726143884", 2024-03-18T19:06:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/3726143884", 2024-03-18T19:06:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-18T19:06:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 49, 2024-03-18T19:06:49.663 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 117, 2024-03-18T19:06:49.664 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-18T19:06:49.664 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-18T19:06:49.664 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-18T19:06:49.664 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:06:49.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:49.671 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-18T19:06:49.671 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T19:06:49.690 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-18T19:06:49.690 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T19:06:49.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:06:49.691 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T19:06:49.759 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-18T19:06:49.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.929+0000 7ffa25e14700 1 Processor -- start 2024-03-18T19:06:49.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.929+0000 7ffa25e14700 1 -- start start 2024-03-18T19:06:49.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.930+0000 7ffa25e14700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa200cfa10 0x7ffa20168490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:49.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.930+0000 7ffa25e14700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa20142fc0 con 0x7ffa200cb450 2024-03-18T19:06:49.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.930+0000 7ffa25e14700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa200cfff0 con 0x7ffa200b1e80 2024-03-18T19:06:49.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.930+0000 7ffa25e14700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa20140cb0 con 0x7ffa200cfa10 2024-03-18T19:06:49.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.930+0000 7ffa1effd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ffa200cb450 0x7ffa200cf300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47194/0 (socket says 172.21.15.1:47194) 2024-03-18T19:06:49.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.930+0000 7ffa1effd700 1 -- 172.21.15.1:0/3586394577 learned_addr learned my addr 172.21.15.1:0/3586394577 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:49.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.931+0000 7ffa1ffff700 1 --2- 172.21.15.1:0/3586394577 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa200cfa10 0x7ffa20168490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:49.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.931+0000 7ffa1e7fc700 1 -- 172.21.15.1:0/3586394577 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680482923 0 0) 0x7ffa20142fc0 con 0x7ffa200cb450 2024-03-18T19:06:49.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.931+0000 7ffa1e7fc700 1 -- 172.21.15.1:0/3586394577 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa080035b0 con 0x7ffa200cb450 2024-03-18T19:06:49.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.931+0000 7ffa1e7fc700 1 -- 172.21.15.1:0/3586394577 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2735997704 0 0) 0x7ffa080035b0 con 0x7ffa200cb450 2024-03-18T19:06:49.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.931+0000 7ffa1e7fc700 1 -- 172.21.15.1:0/3586394577 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa200cfa10 msgr2=0x7ffa20168490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:49.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.931+0000 7ffa1e7fc700 1 --2- 172.21.15.1:0/3586394577 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa200cfa10 0x7ffa20168490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:49.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.931+0000 7ffa1e7fc700 1 -- 172.21.15.1:0/3586394577 >> v1:172.21.15.165:6789/0 conn(0x7ffa200b1e80 legacy=0x7ffa200cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:49.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.931+0000 7ffa1e7fc700 1 -- 172.21.15.1:0/3586394577 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa20143210 con 0x7ffa200cb450 2024-03-18T19:06:49.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.931+0000 7ffa1e7fc700 1 -- 172.21.15.1:0/3586394577 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7ffa14001670 con 0x7ffa200cb450 2024-03-18T19:06:49.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.931+0000 7ffa1e7fc700 1 -- 172.21.15.1:0/3586394577 >> v1:172.21.15.1:6789/0 conn(0x7ffa200cb450 legacy=0x7ffa200cf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:49.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.931+0000 7ffa1e7fc700 1 --2- 172.21.15.1:0/3586394577 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa200cfa10 0x7ffa08003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:49.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.931+0000 7ffa1e7fc700 1 --2- 172.21.15.1:0/3586394577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa08003f70 0x7ffa08004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:49.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.931+0000 7ffa1e7fc700 1 --2- 172.21.15.1:0/3586394577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa08004d90 0x7ffa08005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:49.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.931+0000 7ffa1e7fc700 1 -- 172.21.15.1:0/3586394577 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa20140cb0 con 0x7ffa08003f70 2024-03-18T19:06:49.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.931+0000 7ffa1e7fc700 1 -- 172.21.15.1:0/3586394577 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0800bb20 con 0x7ffa08004d90 2024-03-18T19:06:49.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.931+0000 7ffa1e7fc700 1 -- 172.21.15.1:0/3586394577 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0800bce0 con 0x7ffa200cfa10 2024-03-18T19:06:49.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.931+0000 7ffa1effd700 1 --2- 172.21.15.1:0/3586394577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa08003f70 0x7ffa08004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:49.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.931+0000 7ffa1f7fe700 1 --2- 172.21.15.1:0/3586394577 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa200cfa10 0x7ffa08003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:49.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.931+0000 7ffa1ffff700 1 --2- 172.21.15.1:0/3586394577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa08004d90 0x7ffa08005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:49.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.932+0000 7ffa1effd700 1 -- 172.21.15.1:0/3586394577 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa200cfa10 msgr2=0x7ffa08003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:49.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.932+0000 7ffa1effd700 1 --2- 172.21.15.1:0/3586394577 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa200cfa10 0x7ffa08003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:49.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.932+0000 7ffa1effd700 1 -- 172.21.15.1:0/3586394577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa08004d90 msgr2=0x7ffa08005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:49.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.932+0000 7ffa1effd700 1 --2- 172.21.15.1:0/3586394577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa08004d90 0x7ffa08005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:49.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.932+0000 7ffa1effd700 1 -- 172.21.15.1:0/3586394577 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa080035b0 con 0x7ffa08003f70 2024-03-18T19:06:49.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.932+0000 7ffa1effd700 1 --2- 172.21.15.1:0/3586394577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa08003f70 0x7ffa08004850 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7ffa1400c2f0 tx=0x7ffa08005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca4a2ac46d81a0e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:49.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.932+0000 7ffa1e7fc700 1 -- 172.21.15.1:0/3586394577 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1400ce30 con 0x7ffa08003f70 2024-03-18T19:06:49.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.932+0000 7ffa1e7fc700 1 -- 172.21.15.1:0/3586394577 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa14005960 con 0x7ffa08003f70 2024-03-18T19:06:49.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.932+0000 7ffa1e7fc700 1 -- 172.21.15.1:0/3586394577 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa14005d10 con 0x7ffa08003f70 2024-03-18T19:06:49.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.932+0000 7ffa25e14700 1 -- 172.21.15.1:0/3586394577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa08003f70 msgr2=0x7ffa08004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:49.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.932+0000 7ffa25e14700 1 --2- 172.21.15.1:0/3586394577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa08003f70 0x7ffa08004850 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7ffa1400c2f0 tx=0x7ffa08005880 comp rx=0 tx=0).stop 2024-03-18T19:06:49.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.932+0000 7ffa1ffff700 1 -- 172.21.15.1:0/3586394577 reap_dead start 2024-03-18T19:06:49.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.932+0000 7ffa25e14700 1 -- 172.21.15.1:0/3586394577 shutdown_connections 2024-03-18T19:06:49.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.932+0000 7ffa25e14700 1 -- 172.21.15.1:0/3586394577 >> 172.21.15.1:0/3586394577 conn(0x7ffa20066c80 msgr2=0x7ffa200b2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:49.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.933+0000 7ffa25e14700 1 -- 172.21.15.1:0/3586394577 shutdown_connections 2024-03-18T19:06:49.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.933+0000 7ffa25e14700 1 -- 172.21.15.1:0/3586394577 wait complete. 2024-03-18T19:06:49.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.934+0000 7ffa25e14700 1 Processor -- start 2024-03-18T19:06:49.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.934+0000 7ffa25e14700 1 -- start start 2024-03-18T19:06:49.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa25e14700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa08003f70 0x7ffa2015cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:49.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa25e14700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa2015d1c0 0x7ffa2015c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:49.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa25e14700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa201461c0 0x7ffa201440b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:49.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa25e14700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2015c9e0 con 0x7ffa201461c0 2024-03-18T19:06:49.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa25e14700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa20146750 con 0x7ffa08003f70 2024-03-18T19:06:49.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa25e14700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa20146a30 con 0x7ffa2015d1c0 2024-03-18T19:06:49.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa1ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa201461c0 0x7ffa201440b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:49.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa1ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa201461c0 0x7ffa201440b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49134/0 (socket says 172.21.15.1:49134) 2024-03-18T19:06:49.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa1ffff700 1 -- 172.21.15.1:0/2853800792 learned_addr learned my addr 172.21.15.1:0/2853800792 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:49.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa1effd700 1 --2- 172.21.15.1:0/2853800792 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa2015d1c0 0x7ffa2015c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:49.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa1f7fe700 1 --2- 172.21.15.1:0/2853800792 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa08003f70 0x7ffa2015cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:49.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa1ffff700 1 -- 172.21.15.1:0/2853800792 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa2015d1c0 msgr2=0x7ffa2015c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:49.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa1ffff700 1 --2- 172.21.15.1:0/2853800792 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa2015d1c0 0x7ffa2015c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:49.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa1ffff700 1 -- 172.21.15.1:0/2853800792 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa08003f70 msgr2=0x7ffa2015cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:49.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa1ffff700 1 --2- 172.21.15.1:0/2853800792 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa08003f70 0x7ffa2015cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:49.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa1ffff700 1 -- 172.21.15.1:0/2853800792 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa20146d10 con 0x7ffa201461c0 2024-03-18T19:06:49.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa1f7fe700 1 --2- 172.21.15.1:0/2853800792 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa08003f70 0x7ffa2015cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:06:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa1ffff700 1 --2- 172.21.15.1:0/2853800792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa201461c0 0x7ffa201440b0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7ffa201473c0 tx=0x7ffa18011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa07fff700 1 -- 172.21.15.1:0/2853800792 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1800e040 con 0x7ffa201461c0 2024-03-18T19:06:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa07fff700 1 -- 172.21.15.1:0/2853800792 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1800f040 con 0x7ffa201461c0 2024-03-18T19:06:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa07fff700 1 -- 172.21.15.1:0/2853800792 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1800fdb0 con 0x7ffa201461c0 2024-03-18T19:06:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa25e14700 1 -- 172.21.15.1:0/2853800792 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa20146ff0 con 0x7ffa201461c0 2024-03-18T19:06:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.935+0000 7ffa25e14700 1 -- 172.21.15.1:0/2853800792 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa201445f0 con 0x7ffa201461c0 2024-03-18T19:06:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.936+0000 7ffa25e14700 1 -- 172.21.15.1:0/2853800792 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2015e950 con 0x7ffa201461c0 2024-03-18T19:06:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.937+0000 7ffa07fff700 1 -- 172.21.15.1:0/2853800792 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ffa18022460 con 0x7ffa201461c0 2024-03-18T19:06:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.937+0000 7ffa07fff700 1 --2- 172.21.15.1:0/2853800792 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa0c06adb0 0x7ffa0c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.937+0000 7ffa07fff700 1 -- 172.21.15.1:0/2853800792 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ffa18089cf0 con 0x7ffa201461c0 2024-03-18T19:06:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.937+0000 7ffa1f7fe700 1 --2- 172.21.15.1:0/2853800792 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa0c06adb0 0x7ffa0c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:49.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.938+0000 7ffa1f7fe700 1 --2- 172.21.15.1:0/2853800792 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa0c06adb0 0x7ffa0c06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffa200cb180 tx=0x7ffa10009040 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:49.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:49.942+0000 7ffa07fff700 1 -- 172.21.15.1:0/2853800792 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffa2015e950 con 0x7ffa201461c0 2024-03-18T19:06:50.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.173+0000 7ffa25e14700 1 -- 172.21.15.1:0/2853800792 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa20146ba0 con 0x7ffa0c06adb0 2024-03-18T19:06:50.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.174+0000 7ffa07fff700 1 -- 172.21.15.1:0/2853800792 <== mgr.7542 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7ffa20146ba0 con 0x7ffa0c06adb0 2024-03-18T19:06:50.171 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T19:06:50.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:06:50.171 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephfs" 2024-03-18T19:06:50.171 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:06:50.171 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T19:06:50.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.176+0000 7ffa25e14700 1 -- 172.21.15.1:0/2853800792 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa0c06adb0 msgr2=0x7ffa0c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:50.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.176+0000 7ffa25e14700 1 --2- 172.21.15.1:0/2853800792 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa0c06adb0 0x7ffa0c06d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffa200cb180 tx=0x7ffa10009040 comp rx=0 tx=0).stop 2024-03-18T19:06:50.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.176+0000 7ffa25e14700 1 -- 172.21.15.1:0/2853800792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa201461c0 msgr2=0x7ffa201440b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:50.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.176+0000 7ffa25e14700 1 --2- 172.21.15.1:0/2853800792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa201461c0 0x7ffa201440b0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7ffa201473c0 tx=0x7ffa18011f80 comp rx=0 tx=0).stop 2024-03-18T19:06:50.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.176+0000 7ffa25e14700 1 -- 172.21.15.1:0/2853800792 shutdown_connections 2024-03-18T19:06:50.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.176+0000 7ffa25e14700 1 --2- 172.21.15.1:0/2853800792 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffa0c06adb0 0x7ffa0c06d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:50.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.176+0000 7ffa25e14700 1 --2- 172.21.15.1:0/2853800792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa201461c0 0x7ffa201440b0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:50.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.176+0000 7ffa25e14700 1 --2- 172.21.15.1:0/2853800792 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa08003f70 0x7ffa2015cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:50.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.176+0000 7ffa25e14700 1 --2- 172.21.15.1:0/2853800792 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa2015d1c0 0x7ffa2015c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:50.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.176+0000 7ffa25e14700 1 -- 172.21.15.1:0/2853800792 >> 172.21.15.1:0/2853800792 conn(0x7ffa20066c80 msgr2=0x7ffa20142970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:50.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.176+0000 7ffa25e14700 1 -- 172.21.15.1:0/2853800792 shutdown_connections 2024-03-18T19:06:50.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.176+0000 7ffa25e14700 1 -- 172.21.15.1:0/2853800792 wait complete. 2024-03-18T19:06:50.185 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-18T19:06:50.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.315+0000 7f2091f40700 1 Processor -- start 2024-03-18T19:06:50.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.315+0000 7f2091f40700 1 -- start start 2024-03-18T19:06:50.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.316+0000 7f2091f40700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f208c06d560 0x7f208c0ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:50.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.316+0000 7f2091f40700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c0cac80 con 0x7f208c071cc0 2024-03-18T19:06:50.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.316+0000 7f2091f40700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c0caf60 con 0x7f208c15a240 2024-03-18T19:06:50.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.316+0000 7f2091f40700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208c0adff0 con 0x7f208c06d560 2024-03-18T19:06:50.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.316+0000 7f208bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f208c06d560 0x7f208c0ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:50.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.316+0000 7f208affd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f208c071cc0 0x7f208c0ae410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47196/0 (socket says 172.21.15.1:47196) 2024-03-18T19:06:50.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.316+0000 7f208affd700 1 -- 172.21.15.1:0/1097034909 learned_addr learned my addr 172.21.15.1:0/1097034909 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:50.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.317+0000 7f208bfff700 1 -- 172.21.15.1:0/1097034909 >> v1:172.21.15.165:6789/0 conn(0x7f208c15a240 legacy=0x7f208c0715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:50.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.317+0000 7f208bfff700 1 -- 172.21.15.1:0/1097034909 >> v1:172.21.15.1:6789/0 conn(0x7f208c071cc0 legacy=0x7f208c0ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:50.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.317+0000 7f208bfff700 1 -- 172.21.15.1:0/1097034909 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c0cb240 con 0x7f208c06d560 2024-03-18T19:06:50.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.317+0000 7f208bfff700 1 --2- 172.21.15.1:0/1097034909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c06d560 0x7f208c0ca580 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f2084009d90 tx=0x7f208400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e44a7464197b23 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:50.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.317+0000 7f208a7fc700 1 -- 172.21.15.1:0/1097034909 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208400e070 con 0x7f208c06d560 2024-03-18T19:06:50.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.317+0000 7f208a7fc700 1 -- 172.21.15.1:0/1097034909 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208400bb90 con 0x7f208c06d560 2024-03-18T19:06:50.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.317+0000 7f208a7fc700 1 -- 172.21.15.1:0/1097034909 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2084003a20 con 0x7f208c06d560 2024-03-18T19:06:50.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.317+0000 7f2091f40700 1 -- 172.21.15.1:0/1097034909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c06d560 msgr2=0x7f208c0ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:50.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.317+0000 7f2091f40700 1 --2- 172.21.15.1:0/1097034909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c06d560 0x7f208c0ca580 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f2084009d90 tx=0x7f208400b330 comp rx=0 tx=0).stop 2024-03-18T19:06:50.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.317+0000 7f2091f40700 1 -- 172.21.15.1:0/1097034909 shutdown_connections 2024-03-18T19:06:50.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.317+0000 7f2091f40700 1 --2- 172.21.15.1:0/1097034909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c06d560 0x7f208c0ca580 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:50.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.317+0000 7f2091f40700 1 -- 172.21.15.1:0/1097034909 >> 172.21.15.1:0/1097034909 conn(0x7f208c0cff30 msgr2=0x7f208c0d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:50.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.317+0000 7f2091f40700 1 -- 172.21.15.1:0/1097034909 shutdown_connections 2024-03-18T19:06:50.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.318+0000 7f2091f40700 1 -- 172.21.15.1:0/1097034909 wait complete. 2024-03-18T19:06:50.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.319+0000 7f2091f40700 1 Processor -- start 2024-03-18T19:06:50.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.319+0000 7f2091f40700 1 -- start start 2024-03-18T19:06:50.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f2091f40700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f208c06d560 0x7f208c0cf630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:50.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f2091f40700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c071cc0 0x7f208c06e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:50.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f2091f40700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f208c15a240 0x7f208c14df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f2091f40700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c0c3c50 con 0x7f208c071cc0 2024-03-18T19:06:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f2091f40700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c0c3f30 con 0x7f208c06d560 2024-03-18T19:06:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f2091f40700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208c0c4210 con 0x7f208c15a240 2024-03-18T19:06:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f208affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c071cc0 0x7f208c06e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f208affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c071cc0 0x7f208c06e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49160/0 (socket says 172.21.15.1:49160) 2024-03-18T19:06:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f208affd700 1 -- 172.21.15.1:0/3751881532 learned_addr learned my addr 172.21.15.1:0/3751881532 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f208b7fe700 1 --2- 172.21.15.1:0/3751881532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f208c06d560 0x7f208c0cf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f208bfff700 1 --2- 172.21.15.1:0/3751881532 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f208c15a240 0x7f208c14df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f208affd700 1 -- 172.21.15.1:0/3751881532 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f208c15a240 msgr2=0x7f208c14df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f208affd700 1 --2- 172.21.15.1:0/3751881532 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f208c15a240 0x7f208c14df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f208affd700 1 -- 172.21.15.1:0/3751881532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f208c06d560 msgr2=0x7f208c0cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f208affd700 1 --2- 172.21.15.1:0/3751881532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f208c06d560 0x7f208c0cf630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f208affd700 1 -- 172.21.15.1:0/3751881532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2084009a70 con 0x7f208c071cc0 2024-03-18T19:06:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f208b7fe700 1 --2- 172.21.15.1:0/3751881532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f208c06d560 0x7f208c0cf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:06:50.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f208affd700 1 --2- 172.21.15.1:0/3751881532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c071cc0 0x7f208c06e200 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f207400c490 tx=0x7f207400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:50.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f2083fff700 1 -- 172.21.15.1:0/3751881532 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2074005d70 con 0x7f208c071cc0 2024-03-18T19:06:50.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f2083fff700 1 -- 172.21.15.1:0/3751881532 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20740070d0 con 0x7f208c071cc0 2024-03-18T19:06:50.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f2091f40700 1 -- 172.21.15.1:0/3751881532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f208c0c44f0 con 0x7f208c071cc0 2024-03-18T19:06:50.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f2083fff700 1 -- 172.21.15.1:0/3751881532 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2074007460 con 0x7f208c071cc0 2024-03-18T19:06:50.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.320+0000 7f2091f40700 1 -- 172.21.15.1:0/3751881532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f208c0ceee0 con 0x7f208c071cc0 2024-03-18T19:06:50.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.322+0000 7f2091f40700 1 -- 172.21.15.1:0/3751881532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208c0cf070 con 0x7f208c071cc0 2024-03-18T19:06:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.322+0000 7f2083fff700 1 -- 172.21.15.1:0/3751881532 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 49) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f20740075c0 con 0x7f208c071cc0 2024-03-18T19:06:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.322+0000 7f2083fff700 1 --2- 172.21.15.1:0/3751881532 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f206c06adb0 0x7f206c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.322+0000 7f2083fff700 1 -- 172.21.15.1:0/3751881532 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2074088a50 con 0x7f208c071cc0 2024-03-18T19:06:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.322+0000 7f208b7fe700 1 --2- 172.21.15.1:0/3751881532 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f206c06adb0 0x7f206c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:50.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.323+0000 7f208b7fe700 1 --2- 172.21.15.1:0/3751881532 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f206c06adb0 0x7f206c06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f207c009930 tx=0x7f207c008040 comp rx=0 tx=0).ready entity=mgr.7542 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:50.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.327+0000 7f2083fff700 1 -- 172.21.15.1:0/3751881532 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2074053a70 con 0x7f208c071cc0 2024-03-18T19:06:50.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:50.588+0000 7f2091f40700 1 -- 172.21.15.1:0/3751881532 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f208c0676a0 con 0x7f208c071cc0 2024-03-18T19:06:51.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.356+0000 7f2083fff700 1 -- 172.21.15.1:0/3751881532 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 50) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f2074053e40 con 0x7f208c071cc0 2024-03-18T19:06:51.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.358+0000 7f2083fff700 1 -- 172.21.15.1:0/3751881532 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v50) v1 ==== 92+0+0 (secure 0 0 0) 0x7f2074073840 con 0x7f208c071cc0 2024-03-18T19:06:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.361+0000 7f2091f40700 1 -- 172.21.15.1:0/3751881532 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f206c06adb0 msgr2=0x7f206c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.361+0000 7f2091f40700 1 --2- 172.21.15.1:0/3751881532 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f206c06adb0 0x7f206c06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f207c009930 tx=0x7f207c008040 comp rx=0 tx=0).stop 2024-03-18T19:06:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.361+0000 7f2091f40700 1 -- 172.21.15.1:0/3751881532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c071cc0 msgr2=0x7f208c06e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.361+0000 7f2091f40700 1 --2- 172.21.15.1:0/3751881532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c071cc0 0x7f208c06e200 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f207400c490 tx=0x7f207400c850 comp rx=0 tx=0).stop 2024-03-18T19:06:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.362+0000 7f2091f40700 1 -- 172.21.15.1:0/3751881532 shutdown_connections 2024-03-18T19:06:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.362+0000 7f2091f40700 1 --2- 172.21.15.1:0/3751881532 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f206c06adb0 0x7f206c06d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.362+0000 7f2091f40700 1 --2- 172.21.15.1:0/3751881532 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f208c15a240 0x7f208c14df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.362+0000 7f2091f40700 1 --2- 172.21.15.1:0/3751881532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f208c06d560 0x7f208c0cf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.362+0000 7f2091f40700 1 --2- 172.21.15.1:0/3751881532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f208c071cc0 0x7f208c06e200 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.362+0000 7f2091f40700 1 -- 172.21.15.1:0/3751881532 >> 172.21.15.1:0/3751881532 conn(0x7f208c0cff30 msgr2=0x7f208c0b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:51.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.362+0000 7f2091f40700 1 -- 172.21.15.1:0/3751881532 shutdown_connections 2024-03-18T19:06:51.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.362+0000 7f2091f40700 1 -- 172.21.15.1:0/3751881532 wait complete. 2024-03-18T19:06:51.374 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-18T19:06:51.404 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:06:51.421 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:06:51.497 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:51.501+0000 7fd7a16e2240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:06:51.511 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:51.514+0000 7f5de70e4240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:06:51.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.517+0000 7fa3c60f4700 1 Processor -- start 2024-03-18T19:06:51.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.517+0000 7fa3c60f4700 1 -- start start 2024-03-18T19:06:51.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.518+0000 7fa3c60f4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa3c006eda0 0x7fa3c0161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:51.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.518+0000 7fa3c60f4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3c006f180 0x7fa3c00ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.518+0000 7fa3c60f4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa3c006a750 0x7fa3c006ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.518+0000 7fa3c60f4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c00cb320 con 0x7fa3c006eda0 2024-03-18T19:06:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.518+0000 7fa3c60f4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c00ce010 con 0x7fa3c006a750 2024-03-18T19:06:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.518+0000 7fa3c60f4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c006b0c0 con 0x7fa3c006f180 2024-03-18T19:06:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.518+0000 7fa3bf7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa3c006eda0 0x7fa3c0161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.518+0000 7fa3bf7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa3c006eda0 0x7fa3c0161c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49186/0 (socket says 172.21.15.1:49186) 2024-03-18T19:06:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.518+0000 7fa3bf7fe700 1 -- 172.21.15.1:0/4241446653 learned_addr learned my addr 172.21.15.1:0/4241446653 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.518+0000 7fa3bffff700 1 --2- 172.21.15.1:0/4241446653 >> v2:172.21.15.165:3300/0 conn(0x7fa3c006a750 0x7fa3c006ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.518+0000 7fa3beffd700 1 --2- 172.21.15.1:0/4241446653 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3c006f180 0x7fa3c00ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.518+0000 7fa3bf7fe700 1 -- 172.21.15.1:0/4241446653 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3c006f180 msgr2=0x7fa3c00ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.518+0000 7fa3bf7fe700 1 --2- 172.21.15.1:0/4241446653 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3c006f180 0x7fa3c00ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.518+0000 7fa3bf7fe700 1 -- 172.21.15.1:0/4241446653 >> v2:172.21.15.165:3300/0 conn(0x7fa3c006a750 msgr2=0x7fa3c006ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.518+0000 7fa3bf7fe700 1 --2- 172.21.15.1:0/4241446653 >> v2:172.21.15.165:3300/0 conn(0x7fa3c006a750 0x7fa3c006ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.518+0000 7fa3bf7fe700 1 -- 172.21.15.1:0/4241446653 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c00caac0 con 0x7fa3c006eda0 2024-03-18T19:06:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.518+0000 7fa3bf7fe700 1 --2- 172.21.15.1:0/4241446653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c006eda0 0x7fa3c0161c60 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0161040 tx=0x7fa3b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=342bbaebf54b8508 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.519+0000 7fa3be7fc700 1 -- 172.21.15.1:0/4241446653 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0011830 con 0x7fa3c006eda0 2024-03-18T19:06:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.519+0000 7fa3be7fc700 1 -- 172.21.15.1:0/4241446653 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b0011990 con 0x7fa3c006eda0 2024-03-18T19:06:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.519+0000 7fa3be7fc700 1 -- 172.21.15.1:0/4241446653 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0011d20 con 0x7fa3c006eda0 2024-03-18T19:06:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.519+0000 7fa3c60f4700 1 -- 172.21.15.1:0/4241446653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c006eda0 msgr2=0x7fa3c0161c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.519+0000 7fa3c60f4700 1 --2- 172.21.15.1:0/4241446653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c006eda0 0x7fa3c0161c60 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0161040 tx=0x7fa3b0009bd0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.519+0000 7fa3c60f4700 1 -- 172.21.15.1:0/4241446653 shutdown_connections 2024-03-18T19:06:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.519+0000 7fa3c60f4700 1 --2- 172.21.15.1:0/4241446653 >> v2:172.21.15.165:3300/0 conn(0x7fa3c006a750 0x7fa3c006ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.519+0000 7fa3c60f4700 1 --2- 172.21.15.1:0/4241446653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c006eda0 0x7fa3c0161c60 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.519+0000 7fa3c60f4700 1 --2- 172.21.15.1:0/4241446653 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3c006f180 0x7fa3c00ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.519+0000 7fa3c60f4700 1 -- 172.21.15.1:0/4241446653 >> 172.21.15.1:0/4241446653 conn(0x7fa3c00cff30 msgr2=0x7fa3c00d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.519+0000 7fa3c60f4700 1 -- 172.21.15.1:0/4241446653 shutdown_connections 2024-03-18T19:06:51.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.519+0000 7fa3c60f4700 1 -- 172.21.15.1:0/4241446653 wait complete. 2024-03-18T19:06:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.521+0000 7fa3c60f4700 1 Processor -- start 2024-03-18T19:06:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.521+0000 7fa3c60f4700 1 -- start start 2024-03-18T19:06:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.521+0000 7fa3c60f4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa3c006a750 0x7fa3c00b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.521+0000 7fa3c60f4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c006eda0 0x7fa3c00ca420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.521+0000 7fa3c60f4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3c006f180 0x7fa3c00c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.522+0000 7fa3c60f4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c00ca000 con 0x7fa3c006eda0 2024-03-18T19:06:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.522+0000 7fa3c60f4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c00c9150 con 0x7fa3c006a750 2024-03-18T19:06:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.522+0000 7fa3c60f4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c00c9380 con 0x7fa3c006f180 2024-03-18T19:06:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.522+0000 7fa3beffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c006eda0 0x7fa3c00ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.522+0000 7fa3bf7fe700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa3c006a750 0x7fa3c00b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.522+0000 7fa3beffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c006eda0 0x7fa3c00ca420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49200/0 (socket says 172.21.15.1:49200) 2024-03-18T19:06:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.522+0000 7fa3beffd700 1 -- 172.21.15.1:0/1076859440 learned_addr learned my addr 172.21.15.1:0/1076859440 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.522+0000 7fa3bffff700 1 --2- 172.21.15.1:0/1076859440 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3c006f180 0x7fa3c00c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.522+0000 7fa3bf7fe700 1 -- 172.21.15.1:0/1076859440 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3c006f180 msgr2=0x7fa3c00c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.522+0000 7fa3bf7fe700 1 --2- 172.21.15.1:0/1076859440 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3c006f180 0x7fa3c00c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.522+0000 7fa3bf7fe700 1 -- 172.21.15.1:0/1076859440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c006eda0 msgr2=0x7fa3c00ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.522+0000 7fa3bf7fe700 1 --2- 172.21.15.1:0/1076859440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c006eda0 0x7fa3c00ca420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.522+0000 7fa3bf7fe700 1 -- 172.21.15.1:0/1076859440 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b0009550 con 0x7fa3c006a750 2024-03-18T19:06:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.522+0000 7fa3beffd700 1 --2- 172.21.15.1:0/1076859440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c006eda0 0x7fa3c00ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:06:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.522+0000 7fa3bf7fe700 1 --2- 172.21.15.1:0/1076859440 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa3c006a750 0x7fa3c00b2f20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa3c006d3d0 tx=0x7fa3b0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.522+0000 7fa3a7fff700 1 -- 172.21.15.1:0/1076859440 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0004240 con 0x7fa3c006a750 2024-03-18T19:06:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.522+0000 7fa3a7fff700 1 -- 172.21.15.1:0/1076859440 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b001f070 con 0x7fa3c006a750 2024-03-18T19:06:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.522+0000 7fa3c60f4700 1 -- 172.21.15.1:0/1076859440 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c00c9660 con 0x7fa3c006a750 2024-03-18T19:06:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.522+0000 7fa3a7fff700 1 -- 172.21.15.1:0/1076859440 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b00045e0 con 0x7fa3c006a750 2024-03-18T19:06:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.523+0000 7fa3c60f4700 1 -- 172.21.15.1:0/1076859440 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c00c9c20 con 0x7fa3c006a750 2024-03-18T19:06:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.523+0000 7fa3c60f4700 1 -- 172.21.15.1:0/1076859440 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c00ca140 con 0x7fa3c006a750 2024-03-18T19:06:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.524+0000 7fa3a7fff700 1 -- 172.21.15.1:0/1076859440 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 50) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fa3b0021030 con 0x7fa3c006a750 2024-03-18T19:06:51.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.524+0000 7fa3a7fff700 1 --2- 172.21.15.1:0/1076859440 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa3ac06ae00 0x7fa3ac06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:51.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.524+0000 7fa3beffd700 1 -- 172.21.15.1:0/1076859440 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa3ac06ae00 msgr2=0x7fa3ac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:51.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.524+0000 7fa3a7fff700 1 -- 172.21.15.1:0/1076859440 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa3b008a730 con 0x7fa3c006a750 2024-03-18T19:06:51.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.524+0000 7fa3beffd700 1 --2- 172.21.15.1:0/1076859440 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa3ac06ae00 0x7fa3ac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:51.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.529+0000 7fa3a7fff700 1 -- 172.21.15.1:0/1076859440 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa3c00ca140 con 0x7fa3c006a750 2024-03-18T19:06:51.644 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:51.647+0000 7fd7a16e2240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:06:51.680 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:51.684+0000 7f5de70e4240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:06:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.725+0000 7fa3beffd700 1 -- 172.21.15.1:0/1076859440 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa3ac06ae00 msgr2=0x7fa3ac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.725+0000 7fa3beffd700 1 --2- 172.21.15.1:0/1076859440 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa3ac06ae00 0x7fa3ac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:51.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.780+0000 7fa3c60f4700 1 -- 172.21.15.1:0/1076859440 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fa3c00ca140 con 0x7fa3c006a750 2024-03-18T19:06:51.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.788+0000 7fa3a7fff700 1 -- 172.21.15.1:0/1076859440 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v35)=0 v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa3c00ca140 con 0x7fa3c006a750 2024-03-18T19:06:51.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.790+0000 7fa3c60f4700 1 -- 172.21.15.1:0/1076859440 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa3ac06ae00 msgr2=0x7fa3ac06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:51.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.790+0000 7fa3c60f4700 1 --2- 172.21.15.1:0/1076859440 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa3ac06ae00 0x7fa3ac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.790+0000 7fa3c60f4700 1 -- 172.21.15.1:0/1076859440 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa3c006a750 msgr2=0x7fa3c00b2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:51.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.790+0000 7fa3c60f4700 1 --2- 172.21.15.1:0/1076859440 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa3c006a750 0x7fa3c00b2f20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa3c006d3d0 tx=0x7fa3b0004750 comp rx=0 tx=0).stop 2024-03-18T19:06:51.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.790+0000 7fa3c60f4700 1 -- 172.21.15.1:0/1076859440 shutdown_connections 2024-03-18T19:06:51.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.790+0000 7fa3c60f4700 1 --2- 172.21.15.1:0/1076859440 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa3ac06ae00 0x7fa3ac06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.790+0000 7fa3c60f4700 1 --2- 172.21.15.1:0/1076859440 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3c006f180 0x7fa3c00c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.790+0000 7fa3c60f4700 1 --2- 172.21.15.1:0/1076859440 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa3c006a750 0x7fa3c00b2f20 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.790+0000 7fa3c60f4700 1 --2- 172.21.15.1:0/1076859440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3c006eda0 0x7fa3c00ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.790+0000 7fa3c60f4700 1 -- 172.21.15.1:0/1076859440 >> 172.21.15.1:0/1076859440 conn(0x7fa3c00cff30 msgr2=0x7fa3c00cea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:51.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.790+0000 7fa3c60f4700 1 -- 172.21.15.1:0/1076859440 shutdown_connections 2024-03-18T19:06:51.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.790+0000 7fa3c60f4700 1 -- 172.21.15.1:0/1076859440 wait complete. 2024-03-18T19:06:51.800 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-18T19:06:51.910 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:51.913+0000 7fd7a16e2240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:06:51.962 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:51.965+0000 7f5de70e4240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:06:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.974+0000 7f10e7259700 1 Processor -- start 2024-03-18T19:06:51.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.974+0000 7f10e7259700 1 -- start start 2024-03-18T19:06:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.974+0000 7f10e7259700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f10e01612d0 0x7f10e01616f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.974+0000 7f10e7259700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e0162170 con 0x7f10e0160e50 2024-03-18T19:06:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.975+0000 7f10e7259700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e0161c30 con 0x7f10e0160a70 2024-03-18T19:06:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.975+0000 7f10e7259700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e00d06e0 con 0x7f10e01612d0 2024-03-18T19:06:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.975+0000 7f10e57f6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f10e01612d0 0x7f10e01616f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.975+0000 7f10dffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f10e0160e50 0x7f10e00cffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47204/0 (socket says 172.21.15.1:47204) 2024-03-18T19:06:51.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.975+0000 7f10e57f6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f10e01612d0 0x7f10e01616f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49214/0 (socket says 172.21.15.1:49214) 2024-03-18T19:06:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.975+0000 7f10dffff700 1 -- 172.21.15.1:0/3333210051 learned_addr learned my addr 172.21.15.1:0/3333210051 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.975+0000 7f10e57f6700 1 -- 172.21.15.1:0/3333210051 >> v1:172.21.15.165:6789/0 conn(0x7f10e0160a70 legacy=0x7f10e00d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.975+0000 7f10e57f6700 1 -- 172.21.15.1:0/3333210051 >> v1:172.21.15.1:6789/0 conn(0x7f10e0160e50 legacy=0x7f10e00cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.975+0000 7f10e57f6700 1 -- 172.21.15.1:0/3333210051 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e00cef30 con 0x7f10e01612d0 2024-03-18T19:06:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.975+0000 7f10df7fe700 1 -- 172.21.15.1:0/3333210051 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245008997 0 0) 0x7f10cc001110 con 0x7f10e0160e50 2024-03-18T19:06:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.975+0000 7f10e57f6700 1 --2- 172.21.15.1:0/3333210051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10e01612d0 0x7f10e01616f0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f10d8009d90 tx=0x7f10d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e461d78415aa60b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.975+0000 7f10df7fe700 1 -- 172.21.15.1:0/3333210051 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d800e070 con 0x7f10e01612d0 2024-03-18T19:06:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.975+0000 7f10df7fe700 1 -- 172.21.15.1:0/3333210051 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d800bbd0 con 0x7f10e01612d0 2024-03-18T19:06:51.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.975+0000 7f10df7fe700 1 -- 172.21.15.1:0/3333210051 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d8012430 con 0x7f10e01612d0 2024-03-18T19:06:51.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.976+0000 7f10e7259700 1 -- 172.21.15.1:0/3333210051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10e01612d0 msgr2=0x7f10e01616f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.976+0000 7f10e7259700 1 --2- 172.21.15.1:0/3333210051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10e01612d0 0x7f10e01616f0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f10d8009d90 tx=0x7f10d800b330 comp rx=0 tx=0).stop 2024-03-18T19:06:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.976+0000 7f10e7259700 1 -- 172.21.15.1:0/3333210051 shutdown_connections 2024-03-18T19:06:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.976+0000 7f10e7259700 1 --2- 172.21.15.1:0/3333210051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10e01612d0 0x7f10e01616f0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.976+0000 7f10e7259700 1 -- 172.21.15.1:0/3333210051 >> 172.21.15.1:0/3333210051 conn(0x7f10e009c560 msgr2=0x7f10e00d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.976+0000 7f10e7259700 1 -- 172.21.15.1:0/3333210051 shutdown_connections 2024-03-18T19:06:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.976+0000 7f10e7259700 1 -- 172.21.15.1:0/3333210051 wait complete. 2024-03-18T19:06:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.977+0000 7f10e7259700 1 Processor -- start 2024-03-18T19:06:51.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.977+0000 7f10e7259700 1 -- start start 2024-03-18T19:06:51.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.978+0000 7f10e7259700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10e0160a70 0x7f10e015a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:51.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.978+0000 7f10e7259700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10e0160e50 0x7f10e00c7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:51.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.978+0000 7f10e7259700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10e01612d0 0x7f10e01f6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:51.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.978+0000 7f10e7259700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e01f74a0 con 0x7f10e0160e50 2024-03-18T19:06:51.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.978+0000 7f10e7259700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e00c7070 con 0x7f10e01612d0 2024-03-18T19:06:51.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.978+0000 7f10e7259700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e01f6ee0 con 0x7f10e0160a70 2024-03-18T19:06:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.978+0000 7f10dffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10e0160e50 0x7f10e00c7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.979+0000 7f10dffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10e0160e50 0x7f10e00c7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49222/0 (socket says 172.21.15.1:49222) 2024-03-18T19:06:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.979+0000 7f10dffff700 1 -- 172.21.15.1:0/564277474 learned_addr learned my addr 172.21.15.1:0/564277474 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.979+0000 7f10e57f6700 1 --2- 172.21.15.1:0/564277474 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10e01612d0 0x7f10e01f6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.979+0000 7f10e4ff5700 1 --2- 172.21.15.1:0/564277474 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10e0160a70 0x7f10e015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.979+0000 7f10dffff700 1 -- 172.21.15.1:0/564277474 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10e0160a70 msgr2=0x7f10e015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.979+0000 7f10dffff700 1 --2- 172.21.15.1:0/564277474 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10e0160a70 0x7f10e015a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.979+0000 7f10dffff700 1 -- 172.21.15.1:0/564277474 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10e01612d0 msgr2=0x7f10e01f6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.979+0000 7f10dffff700 1 --2- 172.21.15.1:0/564277474 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10e01612d0 0x7f10e01f6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.979+0000 7f10dffff700 1 -- 172.21.15.1:0/564277474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d8009a70 con 0x7f10e0160e50 2024-03-18T19:06:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.979+0000 7f10dffff700 1 --2- 172.21.15.1:0/564277474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10e0160e50 0x7f10e00c7500 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f10cc00c780 tx=0x7f10cc00ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.979+0000 7f10dd7fa700 1 -- 172.21.15.1:0/564277474 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc004c80 con 0x7f10e0160e50 2024-03-18T19:06:51.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.979+0000 7f10dd7fa700 1 -- 172.21.15.1:0/564277474 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10cc004e00 con 0x7f10e0160e50 2024-03-18T19:06:51.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.979+0000 7f10e7259700 1 -- 172.21.15.1:0/564277474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e01f71c0 con 0x7f10e0160e50 2024-03-18T19:06:51.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.979+0000 7f10dd7fa700 1 -- 172.21.15.1:0/564277474 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc007310 con 0x7f10e0160e50 2024-03-18T19:06:51.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.979+0000 7f10e7259700 1 -- 172.21.15.1:0/564277474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e015ada0 con 0x7f10e0160e50 2024-03-18T19:06:51.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.981+0000 7f10e7259700 1 -- 172.21.15.1:0/564277474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e0067390 con 0x7f10e0160e50 2024-03-18T19:06:51.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.981+0000 7f10dd7fa700 1 -- 172.21.15.1:0/564277474 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 50) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f10cc007470 con 0x7f10e0160e50 2024-03-18T19:06:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.981+0000 7f10dd7fa700 1 --2- 172.21.15.1:0/564277474 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f10c406ae00 0x7f10c406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.981+0000 7f10e4ff5700 1 -- 172.21.15.1:0/564277474 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f10c406ae00 msgr2=0x7f10c406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.981+0000 7f10dd7fa700 1 -- 172.21.15.1:0/564277474 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f10cc089000 con 0x7f10e0160e50 2024-03-18T19:06:51.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.981+0000 7f10e4ff5700 1 --2- 172.21.15.1:0/564277474 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f10c406ae00 0x7f10c406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:51.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:51.987+0000 7f10dd7fa700 1 -- 172.21.15.1:0/564277474 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f10cc053f80 con 0x7f10e0160e50 2024-03-18T19:06:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.182+0000 7f10e4ff5700 1 -- 172.21.15.1:0/564277474 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f10c406ae00 msgr2=0x7f10c406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.182+0000 7f10e4ff5700 1 --2- 172.21.15.1:0/564277474 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f10c406ae00 0x7f10c406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:52.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.189+0000 7f10e7259700 1 -- 172.21.15.1:0/564277474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f10e0162170 con 0x7f10e0160e50 2024-03-18T19:06:52.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.194+0000 7f10dd7fa700 1 -- 172.21.15.1:0/564277474 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v36)=0 v36) v1 ==== 133+0+0 (secure 0 0 0) 0x7f10e0162170 con 0x7f10e0160e50 2024-03-18T19:06:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.196+0000 7f10e7259700 1 -- 172.21.15.1:0/564277474 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f10c406ae00 msgr2=0x7f10c406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.196+0000 7f10e7259700 1 --2- 172.21.15.1:0/564277474 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f10c406ae00 0x7f10c406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.196+0000 7f10e7259700 1 -- 172.21.15.1:0/564277474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10e0160e50 msgr2=0x7f10e00c7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.196+0000 7f10e7259700 1 --2- 172.21.15.1:0/564277474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10e0160e50 0x7f10e00c7500 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f10cc00c780 tx=0x7f10cc00ca90 comp rx=0 tx=0).stop 2024-03-18T19:06:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.197+0000 7f10e7259700 1 -- 172.21.15.1:0/564277474 shutdown_connections 2024-03-18T19:06:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.197+0000 7f10e7259700 1 --2- 172.21.15.1:0/564277474 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f10c406ae00 0x7f10c406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.197+0000 7f10e7259700 1 --2- 172.21.15.1:0/564277474 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10e0160a70 0x7f10e015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.197+0000 7f10e7259700 1 --2- 172.21.15.1:0/564277474 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10e01612d0 0x7f10e01f6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.197+0000 7f10e7259700 1 --2- 172.21.15.1:0/564277474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10e0160e50 0x7f10e00c7500 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.197+0000 7f10e7259700 1 -- 172.21.15.1:0/564277474 >> 172.21.15.1:0/564277474 conn(0x7f10e009c560 msgr2=0x7f10e006b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.197+0000 7f10e7259700 1 -- 172.21.15.1:0/564277474 shutdown_connections 2024-03-18T19:06:52.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.197+0000 7f10e7259700 1 -- 172.21.15.1:0/564277474 wait complete. 2024-03-18T19:06:52.206 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /bad-path 1m 2024-03-18T19:06:52.332 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:52.336+0000 7fd7a16e2240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:06:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.341+0000 7f715a7f6700 1 Processor -- start 2024-03-18T19:06:52.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.341+0000 7f715a7f6700 1 -- start start 2024-03-18T19:06:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.341+0000 7f715a7f6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7154099a40 0x7f715415e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.341+0000 7f715a7f6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f715415d6f0 0x7f7154099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.341+0000 7f715a7f6700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715415e9e0 con 0x7f715406e530 2024-03-18T19:06:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.341+0000 7f715a7f6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f715406ea40 con 0x7f7154099a40 2024-03-18T19:06:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.341+0000 7f715a7f6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715409a360 con 0x7f715415d6f0 2024-03-18T19:06:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.342+0000 7f7153fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7154099a40 0x7f715415e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.342+0000 7f7153fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7154099a40 0x7f715415e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49232/0 (socket says 172.21.15.1:49232) 2024-03-18T19:06:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.342+0000 7f7153fff700 1 -- 172.21.15.1:0/1234567907 learned_addr learned my addr 172.21.15.1:0/1234567907 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.342+0000 7f71537fe700 1 --2- 172.21.15.1:0/1234567907 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f715415d6f0 0x7f7154099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.342+0000 7f7153fff700 1 -- 172.21.15.1:0/1234567907 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f715415d6f0 msgr2=0x7f7154099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.342+0000 7f7153fff700 1 --2- 172.21.15.1:0/1234567907 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f715415d6f0 0x7f7154099e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.342+0000 7f7153fff700 1 -- 172.21.15.1:0/1234567907 >> v1:172.21.15.165:6789/0 conn(0x7f715406e530 legacy=0x7f7154160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.342+0000 7f7153fff700 1 -- 172.21.15.1:0/1234567907 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715406ece0 con 0x7f7154099a40 2024-03-18T19:06:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.342+0000 7f7153fff700 1 --2- 172.21.15.1:0/1234567907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7154099a40 0x7f715415e4a0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f71440098f0 tx=0x7f7144009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d6b98f6f065cb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.342+0000 7f7152ffd700 1 -- 172.21.15.1:0/1234567907 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7144011830 con 0x7f7154099a40 2024-03-18T19:06:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.342+0000 7f7152ffd700 1 -- 172.21.15.1:0/1234567907 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7144011990 con 0x7f7154099a40 2024-03-18T19:06:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.342+0000 7f7152ffd700 1 -- 172.21.15.1:0/1234567907 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7144011d20 con 0x7f7154099a40 2024-03-18T19:06:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.343+0000 7f715a7f6700 1 -- 172.21.15.1:0/1234567907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7154099a40 msgr2=0x7f715415e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.343+0000 7f715a7f6700 1 --2- 172.21.15.1:0/1234567907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7154099a40 0x7f715415e4a0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f71440098f0 tx=0x7f7144009c00 comp rx=0 tx=0).stop 2024-03-18T19:06:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.343+0000 7f715a7f6700 1 -- 172.21.15.1:0/1234567907 shutdown_connections 2024-03-18T19:06:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.343+0000 7f715a7f6700 1 --2- 172.21.15.1:0/1234567907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7154099a40 0x7f715415e4a0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.343+0000 7f715a7f6700 1 --2- 172.21.15.1:0/1234567907 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f715415d6f0 0x7f7154099e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.343+0000 7f715a7f6700 1 -- 172.21.15.1:0/1234567907 >> 172.21.15.1:0/1234567907 conn(0x7f71540b2af0 msgr2=0x7f71540b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.343+0000 7f715a7f6700 1 -- 172.21.15.1:0/1234567907 shutdown_connections 2024-03-18T19:06:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.343+0000 7f715a7f6700 1 -- 172.21.15.1:0/1234567907 wait complete. 2024-03-18T19:06:52.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.344+0000 7f715a7f6700 1 Processor -- start 2024-03-18T19:06:52.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.344+0000 7f715a7f6700 1 -- start start 2024-03-18T19:06:52.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.345+0000 7f715a7f6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f715406e530 0x7f715415a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:52.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.345+0000 7f715a7f6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7154099a40 0x7f71541585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:52.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.345+0000 7f715a7f6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71540c5c00 0x7f7154153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:52.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.345+0000 7f715a7f6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7154156b70 con 0x7f71540c5c00 2024-03-18T19:06:52.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.345+0000 7f715a7f6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7154156dc0 con 0x7f7154099a40 2024-03-18T19:06:52.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.345+0000 7f715a7f6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71541570a0 con 0x7f715406e530 2024-03-18T19:06:52.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.345+0000 7f7158d93700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71540c5c00 0x7f7154153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:52.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.345+0000 7f71537fe700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7154099a40 0x7f71541585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:52.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.345+0000 7f7158d93700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71540c5c00 0x7f7154153aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49240/0 (socket says 172.21.15.1:49240) 2024-03-18T19:06:52.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.345+0000 7f7158d93700 1 -- 172.21.15.1:0/3927155479 learned_addr learned my addr 172.21.15.1:0/3927155479 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:52.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.346+0000 7f7153fff700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f715406e530 0x7f715415a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:52.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.346+0000 7f7158d93700 1 -- 172.21.15.1:0/3927155479 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f715406e530 msgr2=0x7f715415a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.346+0000 7f7158d93700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f715406e530 0x7f715415a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.346+0000 7f7158d93700 1 -- 172.21.15.1:0/3927155479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7154099a40 msgr2=0x7f71541585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.346+0000 7f7158d93700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7154099a40 0x7f71541585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.346+0000 7f7158d93700 1 -- 172.21.15.1:0/3927155479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7144009550 con 0x7f71540c5c00 2024-03-18T19:06:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.346+0000 7f71537fe700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7154099a40 0x7f71541585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:06:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.346+0000 7f7158d93700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71540c5c00 0x7f7154153aa0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f714c00cbc0 tx=0x7f714c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.346+0000 7f7150ff9700 1 -- 172.21.15.1:0/3927155479 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c003c00 con 0x7f71540c5c00 2024-03-18T19:06:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.346+0000 7f7150ff9700 1 -- 172.21.15.1:0/3927155479 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714c003d60 con 0x7f71540c5c00 2024-03-18T19:06:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.346+0000 7f715a7f6700 1 -- 172.21.15.1:0/3927155479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7154157380 con 0x7f71540c5c00 2024-03-18T19:06:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.346+0000 7f7150ff9700 1 -- 172.21.15.1:0/3927155479 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c00ab40 con 0x7f71540c5c00 2024-03-18T19:06:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.346+0000 7f715a7f6700 1 -- 172.21.15.1:0/3927155479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f715415a8a0 con 0x7f71540c5c00 2024-03-18T19:06:52.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.347+0000 7f715a7f6700 1 -- 172.21.15.1:0/3927155479 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71541578a0 con 0x7f71540c5c00 2024-03-18T19:06:52.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.347+0000 7f7150ff9700 1 -- 172.21.15.1:0/3927155479 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 50) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f714c00aca0 con 0x7f71540c5c00 2024-03-18T19:06:52.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.348+0000 7f7150ff9700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f714006ae00 0x7f714006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:52.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.348+0000 7f7153fff700 1 -- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f714006ae00 msgr2=0x7f714006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:52.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.348+0000 7f7150ff9700 1 -- 172.21.15.1:0/3927155479 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f714c0896b0 con 0x7f71540c5c00 2024-03-18T19:06:52.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.348+0000 7f7153fff700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f714006ae00 0x7f714006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:06:52.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.353+0000 7f7150ff9700 1 -- 172.21.15.1:0/3927155479 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f714c054640 con 0x7f71540c5c00 2024-03-18T19:06:52.426 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:52.430+0000 7f5de70e4240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:06:52.542 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:52.546+0000 7fd7a16e2240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:06:52.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.549+0000 7f7153fff700 1 -- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f714006ae00 msgr2=0x7f714006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:52.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.549+0000 7f7153fff700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f714006ae00 0x7f714006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:06:52.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.582+0000 7f715a7f6700 1 -- 172.21.15.1:0/3927155479 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f715406f580 con 0x7f714006ae00 2024-03-18T19:06:52.645 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:52.648+0000 7f5de70e4240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:06:52.910 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:52.914+0000 7fd7a16e2240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:06:52.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.950+0000 7f7153fff700 1 -- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f714006ae00 msgr2=0x7f714006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:52.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:52.950+0000 7f7153fff700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f714006ae00 0x7f714006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T19:06:53.033 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:53.037+0000 7f5de70e4240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:06:53.488 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:53.492+0000 7fd7a16e2240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:06:53.558 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:53.562+0000 7fd7a16e2240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:06:53.651 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:53.654+0000 7f5de70e4240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:06:53.685 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:53.689+0000 7fd7a16e2240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:06:53.727 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:53.730+0000 7f5de70e4240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:06:53.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:53.751+0000 7f7153fff700 1 -- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f714006ae00 msgr2=0x7f714006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:53.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:53.751+0000 7f7153fff700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f714006ae00 0x7f714006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T19:06:53.861 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:53.865+0000 7fd7a16e2240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:06:53.862 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:53.865+0000 7f5de70e4240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:06:53.932 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:53.936+0000 7fd7a16e2240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:06:54.009 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:54.013+0000 7fd7a16e2240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:06:54.053 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:54.056+0000 7f5de70e4240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:06:54.087 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:54.090+0000 7fd7a16e2240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:06:54.129 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:54.132+0000 7f5de70e4240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:06:54.164 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:54.168+0000 7fd7a16e2240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:06:54.211 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:54.214+0000 7f5de70e4240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:06:54.293 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:54.296+0000 7f5de70e4240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:06:54.331 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:54.335+0000 7fd7a16e2240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:06:54.375 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:54.378+0000 7f5de70e4240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:06:54.546 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:54.549+0000 7fd7a16e2240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:06:54.554 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:54.558+0000 7f5de70e4240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:06:54.695 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:54.699+0000 7fd7a16e2240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:06:54.766 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:54.770+0000 7fd7a16e2240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:06:54.776 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:54.779+0000 7f5de70e4240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:06:54.936 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:54.939+0000 7f5de70e4240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:06:55.012 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:55.015+0000 7f5de70e4240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:06:55.207 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:55.210+0000 7fd7a16e2240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:06:55.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:55.352+0000 7f7153fff700 1 -- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f714006ae00 msgr2=0x7f714006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:06:55.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:55.352+0000 7f7153fff700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f714006ae00 0x7f714006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T19:06:55.488 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:55.492+0000 7f5de70e4240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:06:55.540 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:55.544+0000 7fd7a16e2240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:06:55.700 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:55.704+0000 7fd7a16e2240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:06:55.841 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:55.845+0000 7f5de70e4240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:06:56.013 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:56.017+0000 7f5de70e4240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:06:56.082 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:56.086+0000 7fd7a16e2240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:06:56.245 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:56.249+0000 7fd7a16e2240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:06:56.428 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:56.431+0000 7f5de70e4240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:06:56.604 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:56.607+0000 7f5de70e4240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:06:56.611 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:56.615+0000 7fd7a16e2240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:06:56.698 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:06:56.702+0000 7fd7a16e2240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:06:56.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:56.773+0000 7f7150ff9700 1 -- 172.21.15.1:0/3927155479 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 51) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f714c010dc0 con 0x7f71540c5c00 2024-03-18T19:06:56.985 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:56.988+0000 7f5de70e4240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:06:57.080 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:57.083+0000 7f5de70e4240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:06:57.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:57.105+0000 7f7150ff9700 1 -- 172.21.15.1:0/3927155479 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 52) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f714c087a90 con 0x7f71540c5c00 2024-03-18T19:06:57.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:57.105+0000 7f7150ff9700 1 -- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f714006ae00 msgr2=0x7f714006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:06:57.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:57.105+0000 7f7150ff9700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f714006ae00 0x7f714006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:57.253 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:57.256+0000 7f5d5cd07700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:57.253 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:57.256+0000 7f5d5cd07700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:57.253 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:57.256+0000 7f5d5cd07700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:57.254 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:57.256+0000 7f5d5cd07700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:57.254 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:57.256+0000 7f5d5cd07700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:57.254 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:57.256+0000 7f5d59d01700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:57.254 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:57.256+0000 7f5d59d01700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:57.254 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:57.256+0000 7f5d59d01700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:57.254 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:57.256+0000 7f5d59d01700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:57.254 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:06:57.256+0000 7f5d59d01700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:06:58.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.112+0000 7f7150ff9700 1 -- 172.21.15.1:0/3927155479 <== mon.0 v2:172.21.15.1:3300/0 9 ==== mgrmap(e 53) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f714c0552e0 con 0x7f71540c5c00 2024-03-18T19:06:58.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.112+0000 7f7150ff9700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f714006ae00 0x7f714006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:58.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.112+0000 7f7150ff9700 1 -- 172.21.15.1:0/3927155479 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f714001cd40 con 0x7f714006ae00 2024-03-18T19:06:58.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.115+0000 7f7153fff700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f714006ae00 0x7f714006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:58.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.116+0000 7f7153fff700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f714006ae00 0x7f714006d2c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7144009c40 tx=0x7f7144008030 comp rx=0 tx=0).ready entity=mgr.7947 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:58.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.180+0000 7f7150ff9700 1 -- 172.21.15.1:0/3927155479 <== mgr.7947 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f714001cd40 con 0x7f714006ae00 2024-03-18T19:06:58.177 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path /bad-path 2024-03-18T19:06:58.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.182+0000 7f715a7f6700 1 -- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f714006ae00 msgr2=0x7f714006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:58.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.182+0000 7f715a7f6700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f714006ae00 0x7f714006d2c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7144009c40 tx=0x7f7144008030 comp rx=0 tx=0).stop 2024-03-18T19:06:58.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.182+0000 7f715a7f6700 1 -- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71540c5c00 msgr2=0x7f7154153aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:58.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.182+0000 7f715a7f6700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71540c5c00 0x7f7154153aa0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f714c00cbc0 tx=0x7f714c00ced0 comp rx=0 tx=0).stop 2024-03-18T19:06:58.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.183+0000 7f715a7f6700 1 -- 172.21.15.1:0/3927155479 shutdown_connections 2024-03-18T19:06:58.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.183+0000 7f715a7f6700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f714006ae00 0x7f714006d2c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:58.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.183+0000 7f715a7f6700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f715406e530 0x7f715415a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:58.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.183+0000 7f715a7f6700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71540c5c00 0x7f7154153aa0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:58.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.183+0000 7f715a7f6700 1 --2- 172.21.15.1:0/3927155479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7154099a40 0x7f71541585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:58.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.183+0000 7f715a7f6700 1 -- 172.21.15.1:0/3927155479 >> 172.21.15.1:0/3927155479 conn(0x7f71540b2af0 msgr2=0x7f71540cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:58.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.183+0000 7f715a7f6700 1 -- 172.21.15.1:0/3927155479 shutdown_connections 2024-03-18T19:06:58.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.183+0000 7f715a7f6700 1 -- 172.21.15.1:0/3927155479 wait complete. 2024-03-18T19:06:58.194 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2024-03-18T19:06:58.194 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2024-03-18T19:06:58.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.329+0000 7fc8b5d7f700 1 Processor -- start 2024-03-18T19:06:58.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.329+0000 7fc8b5d7f700 1 -- start start 2024-03-18T19:06:58.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.330+0000 7fc8b5d7f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc8b006b130 0x7fc8b00cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:58.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.330+0000 7fc8b5d7f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b00cf730 con 0x7fc8b00c3670 2024-03-18T19:06:58.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.330+0000 7fc8b5d7f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b00cf980 con 0x7fc8b00cbd40 2024-03-18T19:06:58.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.330+0000 7fc8b5d7f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b00cfc60 con 0x7fc8b006b130 2024-03-18T19:06:58.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.330+0000 7fc8aeffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc8b00c3670 0x7fc8b015e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46126/0 (socket says 172.21.15.1:46126) 2024-03-18T19:06:58.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.330+0000 7fc8aeffd700 1 -- 172.21.15.1:0/181429758 learned_addr learned my addr 172.21.15.1:0/181429758 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:06:58.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.330+0000 7fc8affff700 1 --2- 172.21.15.1:0/181429758 >> v2:172.21.15.165:3300/0 conn(0x7fc8b006b130 0x7fc8b00cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:58.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.330+0000 7fc8ae7fc700 1 -- 172.21.15.1:0/181429758 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68232606 0 0) 0x7fc8b00cf730 con 0x7fc8b00c3670 2024-03-18T19:06:58.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8ae7fc700 1 -- 172.21.15.1:0/181429758 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc89c0035b0 con 0x7fc8b00c3670 2024-03-18T19:06:58.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8ae7fc700 1 -- 172.21.15.1:0/181429758 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983714877 0 0) 0x7fc8b00cf980 con 0x7fc8b00cbd40 2024-03-18T19:06:58.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8ae7fc700 1 -- 172.21.15.1:0/181429758 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b00cf730 con 0x7fc8b00cbd40 2024-03-18T19:06:58.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8ae7fc700 1 -- 172.21.15.1:0/181429758 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 329221462 0 0) 0x7fc89c0035b0 con 0x7fc8b00c3670 2024-03-18T19:06:58.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8ae7fc700 1 -- 172.21.15.1:0/181429758 >> v2:172.21.15.165:3300/0 conn(0x7fc8b006b130 msgr2=0x7fc8b00cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:58.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8ae7fc700 1 --2- 172.21.15.1:0/181429758 >> v2:172.21.15.165:3300/0 conn(0x7fc8b006b130 0x7fc8b00cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:58.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8ae7fc700 1 -- 172.21.15.1:0/181429758 >> v1:172.21.15.165:6789/0 conn(0x7fc8b00cbd40 legacy=0x7fc8b00cc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:58.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8ae7fc700 1 -- 172.21.15.1:0/181429758 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b015dc00 con 0x7fc8b00c3670 2024-03-18T19:06:58.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8ae7fc700 1 -- 172.21.15.1:0/181429758 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fc898001670 con 0x7fc8b00c3670 2024-03-18T19:06:58.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8ae7fc700 1 -- 172.21.15.1:0/181429758 >> v1:172.21.15.1:6789/0 conn(0x7fc8b00c3670 legacy=0x7fc8b015e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:58.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8ae7fc700 1 --2- 172.21.15.1:0/181429758 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc89c003b90 0x7fc89c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:58.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8ae7fc700 1 --2- 172.21.15.1:0/181429758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc89c004850 0x7fc89c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:58.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8ae7fc700 1 --2- 172.21.15.1:0/181429758 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc89c005210 0x7fc89c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:58.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8ae7fc700 1 -- 172.21.15.1:0/181429758 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b00cfc60 con 0x7fc89c004850 2024-03-18T19:06:58.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8ae7fc700 1 -- 172.21.15.1:0/181429758 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c00c140 con 0x7fc89c005210 2024-03-18T19:06:58.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8ae7fc700 1 -- 172.21.15.1:0/181429758 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c00c300 con 0x7fc89c003b90 2024-03-18T19:06:58.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8aeffd700 1 --2- 172.21.15.1:0/181429758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc89c004850 0x7fc89c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:58.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8affff700 1 --2- 172.21.15.1:0/181429758 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc89c005210 0x7fc89c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:58.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8af7fe700 1 --2- 172.21.15.1:0/181429758 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc89c003b90 0x7fc89c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:58.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8aeffd700 1 -- 172.21.15.1:0/181429758 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc89c003b90 msgr2=0x7fc89c003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:58.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8aeffd700 1 --2- 172.21.15.1:0/181429758 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc89c003b90 0x7fc89c003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:58.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8aeffd700 1 -- 172.21.15.1:0/181429758 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc89c005210 msgr2=0x7fc89c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:06:58.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8aeffd700 1 --2- 172.21.15.1:0/181429758 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc89c005210 0x7fc89c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:58.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8aeffd700 1 -- 172.21.15.1:0/181429758 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89c0035b0 con 0x7fc89c004850 2024-03-18T19:06:58.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8affff700 1 -- 172.21.15.1:0/181429758 reap_dead start 2024-03-18T19:06:58.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.331+0000 7fc8aeffd700 1 --2- 172.21.15.1:0/181429758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc89c004850 0x7fc89c004cd0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fc89800c210 tx=0x7fc89c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=489f66af8fa0ce52 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:58.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.332+0000 7fc8ae7fc700 1 -- 172.21.15.1:0/181429758 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89800cd10 con 0x7fc89c004850 2024-03-18T19:06:58.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.332+0000 7fc8ae7fc700 1 -- 172.21.15.1:0/181429758 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc89800ce90 con 0x7fc89c004850 2024-03-18T19:06:58.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.332+0000 7fc8ae7fc700 1 -- 172.21.15.1:0/181429758 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc898005ba0 con 0x7fc89c004850 2024-03-18T19:06:58.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.332+0000 7fc8b5d7f700 1 -- 172.21.15.1:0/181429758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc89c004850 msgr2=0x7fc89c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:58.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.332+0000 7fc8b5d7f700 1 --2- 172.21.15.1:0/181429758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc89c004850 0x7fc89c004cd0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fc89800c210 tx=0x7fc89c005880 comp rx=0 tx=0).stop 2024-03-18T19:06:58.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.332+0000 7fc8b5d7f700 1 -- 172.21.15.1:0/181429758 shutdown_connections 2024-03-18T19:06:58.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.332+0000 7fc8b5d7f700 1 --2- 172.21.15.1:0/181429758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc89c004850 0x7fc89c004cd0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:58.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.332+0000 7fc8b5d7f700 1 -- 172.21.15.1:0/181429758 >> 172.21.15.1:0/181429758 conn(0x7fc8b00b1aa0 msgr2=0x7fc8b00b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:58.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.332+0000 7fc8b5d7f700 1 -- 172.21.15.1:0/181429758 shutdown_connections 2024-03-18T19:06:58.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.332+0000 7fc8b5d7f700 1 -- 172.21.15.1:0/181429758 wait complete. 2024-03-18T19:06:58.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.333+0000 7fc8b5d7f700 1 Processor -- start 2024-03-18T19:06:58.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.334+0000 7fc8b5d7f700 1 -- start start 2024-03-18T19:06:58.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.334+0000 7fc8b5d7f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc89c004850 0x7fc8b015e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:58.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.334+0000 7fc8b5d7f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8b01e96c0 0x7fc8b015bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:58.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.334+0000 7fc8b5d7f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8b01e9b40 0x7fc8b015c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:58.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.334+0000 7fc8b5d7f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b015c780 con 0x7fc8b01e9b40 2024-03-18T19:06:58.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.334+0000 7fc8b5d7f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b00c4d00 con 0x7fc8b01e96c0 2024-03-18T19:06:58.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.334+0000 7fc8b5d7f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b00c4fe0 con 0x7fc89c004850 2024-03-18T19:06:58.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.334+0000 7fc8affff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8b01e9b40 0x7fc8b015c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:58.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.334+0000 7fc8affff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8b01e9b40 0x7fc8b015c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36936/0 (socket says 172.21.15.1:36936) 2024-03-18T19:06:58.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.334+0000 7fc8affff700 1 -- 172.21.15.1:0/1063599445 learned_addr learned my addr 172.21.15.1:0/1063599445 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:06:58.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.335+0000 7fc8af7fe700 1 --2- 172.21.15.1:0/1063599445 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc89c004850 0x7fc8b015e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:58.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.335+0000 7fc8affff700 1 -- 172.21.15.1:0/1063599445 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc89c004850 msgr2=0x7fc8b015e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:58.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.335+0000 7fc8affff700 1 --2- 172.21.15.1:0/1063599445 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc89c004850 0x7fc8b015e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:58.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.335+0000 7fc8affff700 1 -- 172.21.15.1:0/1063599445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8b01e96c0 msgr2=0x7fc8b015bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:58.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.335+0000 7fc8affff700 1 --2- 172.21.15.1:0/1063599445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8b01e96c0 0x7fc8b015bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:58.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.335+0000 7fc8affff700 1 -- 172.21.15.1:0/1063599445 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b015e540 con 0x7fc8b01e9b40 2024-03-18T19:06:58.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.335+0000 7fc8affff700 1 --2- 172.21.15.1:0/1063599445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8b01e9b40 0x7fc8b015c240 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fc8a8011bf0 tx=0x7fc8a800f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:58.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.335+0000 7fc8a7fff700 1 -- 172.21.15.1:0/1063599445 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a800e040 con 0x7fc8b01e9b40 2024-03-18T19:06:58.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.335+0000 7fc8a7fff700 1 -- 172.21.15.1:0/1063599445 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a800f040 con 0x7fc8b01e9b40 2024-03-18T19:06:58.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.335+0000 7fc8b5d7f700 1 -- 172.21.15.1:0/1063599445 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b015ebc0 con 0x7fc8b01e9b40 2024-03-18T19:06:58.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.335+0000 7fc8a7fff700 1 -- 172.21.15.1:0/1063599445 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a800fe50 con 0x7fc8b01e9b40 2024-03-18T19:06:58.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.335+0000 7fc8b5d7f700 1 -- 172.21.15.1:0/1063599445 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b015f180 con 0x7fc8b01e9b40 2024-03-18T19:06:58.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.336+0000 7fc8b5d7f700 1 -- 172.21.15.1:0/1063599445 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b00c5170 con 0x7fc8b01e9b40 2024-03-18T19:06:58.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.337+0000 7fc8a7fff700 1 -- 172.21.15.1:0/1063599445 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 53) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fc8a8022460 con 0x7fc8b01e9b40 2024-03-18T19:06:58.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.337+0000 7fc8a7fff700 1 --2- 172.21.15.1:0/1063599445 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc89006ab30 0x7fc89006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:06:58.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.337+0000 7fc8a7fff700 1 -- 172.21.15.1:0/1063599445 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fc8a8089840 con 0x7fc8b01e9b40 2024-03-18T19:06:58.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.337+0000 7fc8af7fe700 1 --2- 172.21.15.1:0/1063599445 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc89006ab30 0x7fc89006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:06:58.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.337+0000 7fc8af7fe700 1 --2- 172.21.15.1:0/1063599445 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc89006ab30 0x7fc89006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0000c00 tx=0x7fc8a0009040 comp rx=0 tx=0).ready entity=mgr.7947 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:06:58.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.342+0000 7fc8a7fff700 1 -- 172.21.15.1:0/1063599445 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc8a8054ac0 con 0x7fc8b01e9b40 2024-03-18T19:06:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.573+0000 7fc8b5d7f700 1 -- 172.21.15.1:0/1063599445 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc8b015d920 con 0x7fc89006ab30 2024-03-18T19:06:58.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.583+0000 7fc8a7fff700 1 -- 172.21.15.1:0/1063599445 <== mgr.7947 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+302 (secure 0 0 0) 0x7fc8a0001430 con 0x7fc89006ab30 2024-03-18T19:06:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:06:58.580 INFO:teuthology.orchestra.run.smithi001.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T19:06:58", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-03-18T19:06:58.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.585+0000 7fc8b5d7f700 1 -- 172.21.15.1:0/1063599445 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc89006ab30 msgr2=0x7fc89006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.585+0000 7fc8b5d7f700 1 --2- 172.21.15.1:0/1063599445 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc89006ab30 0x7fc89006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0000c00 tx=0x7fc8a0009040 comp rx=0 tx=0).stop 2024-03-18T19:06:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.585+0000 7fc8b5d7f700 1 -- 172.21.15.1:0/1063599445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8b01e9b40 msgr2=0x7fc8b015c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:06:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.585+0000 7fc8b5d7f700 1 --2- 172.21.15.1:0/1063599445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8b01e9b40 0x7fc8b015c240 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fc8a8011bf0 tx=0x7fc8a800f1b0 comp rx=0 tx=0).stop 2024-03-18T19:06:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.585+0000 7fc8b5d7f700 1 -- 172.21.15.1:0/1063599445 shutdown_connections 2024-03-18T19:06:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.585+0000 7fc8b5d7f700 1 --2- 172.21.15.1:0/1063599445 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc89006ab30 0x7fc89006cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.585+0000 7fc8b5d7f700 1 --2- 172.21.15.1:0/1063599445 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc89c004850 0x7fc8b015e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.585+0000 7fc8b5d7f700 1 --2- 172.21.15.1:0/1063599445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8b01e9b40 0x7fc8b015c240 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.585+0000 7fc8b5d7f700 1 --2- 172.21.15.1:0/1063599445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8b01e96c0 0x7fc8b015bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:06:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.585+0000 7fc8b5d7f700 1 -- 172.21.15.1:0/1063599445 >> 172.21.15.1:0/1063599445 conn(0x7fc8b00b1aa0 msgr2=0x7fc8b00af910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:06:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.586+0000 7fc8b5d7f700 1 -- 172.21.15.1:0/1063599445 shutdown_connections 2024-03-18T19:06:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:06:58.586+0000 7fc8b5d7f700 1 -- 172.21.15.1:0/1063599445 wait complete. 2024-03-18T19:06:58.595 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T19:06:58", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-03-18T19:07:00.188 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:07:00.191+0000 7f5d4c4e6700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:07:00.188 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:07:00.191+0000 7f5d4c4e6700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:07:00.188 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:07:00.191+0000 7f5d4c4e6700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:07:00.188 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:07:00.191+0000 7f5d4c4e6700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:07:00.188 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:07:00.191+0000 7f5d4c4e6700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:07:21.474 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:07:21.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:07:51.618 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:07:51.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:07:58.596 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2024-03-18T19:07:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.741+0000 7f853eea8700 1 Processor -- start 2024-03-18T19:07:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.741+0000 7f853eea8700 1 -- start start 2024-03-18T19:07:58.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.741+0000 7f853eea8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8538070f10 0x7f853806f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:58.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.741+0000 7f853eea8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f85380d0bb0 0x7f853815b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:58.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.742+0000 7f853eea8700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8538155e80 con 0x7f85380cb740 2024-03-18T19:07:58.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.742+0000 7f853eea8700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f853815efd0 con 0x7f85380d0bb0 2024-03-18T19:07:58.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.742+0000 7f853eea8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85380cbcc0 con 0x7f8538070f10 2024-03-18T19:07:58.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.742+0000 7f853d445700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f85380d0bb0 0x7f853815b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:58.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.742+0000 7f8537fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f85380cb740 0x7f85380d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34924/0 (socket says 172.21.15.1:34924) 2024-03-18T19:07:58.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.742+0000 7f8537fff700 1 -- 172.21.15.1:0/3113846107 learned_addr learned my addr 172.21.15.1:0/3113846107 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:07:58.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.742+0000 7f853cc44700 1 --2- 172.21.15.1:0/3113846107 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8538070f10 0x7f853806f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:58.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.742+0000 7f853d445700 1 -- 172.21.15.1:0/3113846107 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8538070f10 msgr2=0x7f853806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:07:58.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.742+0000 7f853d445700 1 --2- 172.21.15.1:0/3113846107 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8538070f10 0x7f853806f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:58.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.742+0000 7f853d445700 1 -- 172.21.15.1:0/3113846107 >> v1:172.21.15.1:6789/0 conn(0x7f85380cb740 legacy=0x7f85380d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:07:58.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.742+0000 7f853d445700 1 -- 172.21.15.1:0/3113846107 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853815f220 con 0x7f85380d0bb0 2024-03-18T19:07:58.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.742+0000 7f85377fe700 1 -- 172.21.15.1:0/3113846107 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3493925206 0 0) 0x7f8538155e80 con 0x7f85380cb740 2024-03-18T19:07:58.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.742+0000 7f853d445700 1 --2- 172.21.15.1:0/3113846107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85380d0bb0 0x7f853815b9e0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f8530009d90 tx=0x7f853000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad1af11153286250 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:07:58.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.742+0000 7f85377fe700 1 -- 172.21.15.1:0/3113846107 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853000e070 con 0x7f85380d0bb0 2024-03-18T19:07:58.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.742+0000 7f85377fe700 1 -- 172.21.15.1:0/3113846107 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853000bbd0 con 0x7f85380d0bb0 2024-03-18T19:07:58.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.742+0000 7f85377fe700 1 -- 172.21.15.1:0/3113846107 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8530012430 con 0x7f85380d0bb0 2024-03-18T19:07:58.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.743+0000 7f853eea8700 1 -- 172.21.15.1:0/3113846107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85380d0bb0 msgr2=0x7f853815b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:07:58.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.743+0000 7f853eea8700 1 --2- 172.21.15.1:0/3113846107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85380d0bb0 0x7f853815b9e0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f8530009d90 tx=0x7f853000b330 comp rx=0 tx=0).stop 2024-03-18T19:07:58.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.743+0000 7f853eea8700 1 -- 172.21.15.1:0/3113846107 shutdown_connections 2024-03-18T19:07:58.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.743+0000 7f853eea8700 1 --2- 172.21.15.1:0/3113846107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85380d0bb0 0x7f853815b9e0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:58.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.743+0000 7f853eea8700 1 --2- 172.21.15.1:0/3113846107 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8538070f10 0x7f853806f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:58.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.743+0000 7f853eea8700 1 -- 172.21.15.1:0/3113846107 >> 172.21.15.1:0/3113846107 conn(0x7f85380b2cc0 msgr2=0x7f85380b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:07:58.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.743+0000 7f853eea8700 1 -- 172.21.15.1:0/3113846107 shutdown_connections 2024-03-18T19:07:58.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.743+0000 7f853eea8700 1 -- 172.21.15.1:0/3113846107 wait complete. 2024-03-18T19:07:58.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.744+0000 7f853eea8700 1 Processor -- start 2024-03-18T19:07:58.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.744+0000 7f853eea8700 1 -- start start 2024-03-18T19:07:58.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.745+0000 7f853eea8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8538070f10 0x7f8538152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:58.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.745+0000 7f853eea8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85380cb740 0x7f8538152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:58.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.745+0000 7f853eea8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85380d0bb0 0x7f85380ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:58.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.745+0000 7f853eea8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85380c60b0 con 0x7f85380d0bb0 2024-03-18T19:07:58.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.745+0000 7f853eea8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85380c5810 con 0x7f8538070f10 2024-03-18T19:07:58.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.745+0000 7f853eea8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85380c5af0 con 0x7f85380cb740 2024-03-18T19:07:58.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.745+0000 7f853d445700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85380d0bb0 0x7f85380ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:58.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.745+0000 7f853d445700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85380d0bb0 0x7f85380ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46460/0 (socket says 172.21.15.1:46460) 2024-03-18T19:07:58.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.745+0000 7f853d445700 1 -- 172.21.15.1:0/3508615416 learned_addr learned my addr 172.21.15.1:0/3508615416 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:07:58.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.745+0000 7f8537fff700 1 --2- 172.21.15.1:0/3508615416 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85380cb740 0x7f8538152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:58.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.745+0000 7f853cc44700 1 --2- 172.21.15.1:0/3508615416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8538070f10 0x7f8538152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:58.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.746+0000 7f853d445700 1 -- 172.21.15.1:0/3508615416 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85380cb740 msgr2=0x7f8538152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:07:58.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.746+0000 7f853d445700 1 --2- 172.21.15.1:0/3508615416 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85380cb740 0x7f8538152ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:58.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.746+0000 7f853d445700 1 -- 172.21.15.1:0/3508615416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8538070f10 msgr2=0x7f8538152710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:07:58.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.746+0000 7f853d445700 1 --2- 172.21.15.1:0/3508615416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8538070f10 0x7f8538152710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:58.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.746+0000 7f853d445700 1 -- 172.21.15.1:0/3508615416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8530009a70 con 0x7f85380d0bb0 2024-03-18T19:07:58.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.746+0000 7f853cc44700 1 --2- 172.21.15.1:0/3508615416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8538070f10 0x7f8538152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:07:58.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.746+0000 7f853d445700 1 --2- 172.21.15.1:0/3508615416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85380d0bb0 0x7f85380ca1d0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f85380b3b90 tx=0x7f8530004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:07:58.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.746+0000 7f85357fa700 1 -- 172.21.15.1:0/3508615416 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8530003ea0 con 0x7f85380d0bb0 2024-03-18T19:07:58.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.746+0000 7f85357fa700 1 -- 172.21.15.1:0/3508615416 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8530020070 con 0x7f85380d0bb0 2024-03-18T19:07:58.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.746+0000 7f85357fa700 1 -- 172.21.15.1:0/3508615416 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8530004280 con 0x7f85380d0bb0 2024-03-18T19:07:58.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.746+0000 7f853eea8700 1 -- 172.21.15.1:0/3508615416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85380c5dd0 con 0x7f85380d0bb0 2024-03-18T19:07:58.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.746+0000 7f853eea8700 1 -- 172.21.15.1:0/3508615416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8538152c50 con 0x7f85380d0bb0 2024-03-18T19:07:58.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.747+0000 7f853eea8700 1 -- 172.21.15.1:0/3508615416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f853815d1e0 con 0x7f85380d0bb0 2024-03-18T19:07:58.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.747+0000 7f85357fa700 1 -- 172.21.15.1:0/3508615416 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 55) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f85300043e0 con 0x7f85380d0bb0 2024-03-18T19:07:58.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.748+0000 7f85357fa700 1 --2- 172.21.15.1:0/3508615416 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f851c06af20 0x7f851c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:58.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.748+0000 7f85357fa700 1 -- 172.21.15.1:0/3508615416 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f8530022070 con 0x7f85380d0bb0 2024-03-18T19:07:58.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.748+0000 7f853cc44700 1 --2- 172.21.15.1:0/3508615416 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f851c06af20 0x7f851c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:58.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.748+0000 7f853cc44700 1 --2- 172.21.15.1:0/3508615416 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f851c06af20 0x7f851c06d3e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f852c00a9b0 tx=0x7f852c008040 comp rx=0 tx=0).ready entity=mgr.7947 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:07:58.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.753+0000 7f85357fa700 1 -- 172.21.15.1:0/3508615416 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f853815d1e0 con 0x7f85380d0bb0 2024-03-18T19:07:58.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.982+0000 7f853eea8700 1 -- 172.21.15.1:0/3508615416 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f853806f980 con 0x7f851c06af20 2024-03-18T19:07:58.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.989+0000 7f85357fa700 1 -- 172.21.15.1:0/3508615416 <== mgr.7947 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+303 (secure 0 0 0) 0x7f852c00ae20 con 0x7f851c06af20 2024-03-18T19:07:58.986 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:07:58.986 INFO:teuthology.orchestra.run.smithi001.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T19:06:58", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2024-03-18T19:07:58.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.992+0000 7f853eea8700 1 -- 172.21.15.1:0/3508615416 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f851c06af20 msgr2=0x7f851c06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:07:58.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.992+0000 7f853eea8700 1 --2- 172.21.15.1:0/3508615416 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f851c06af20 0x7f851c06d3e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f852c00a9b0 tx=0x7f852c008040 comp rx=0 tx=0).stop 2024-03-18T19:07:58.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.992+0000 7f853eea8700 1 -- 172.21.15.1:0/3508615416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85380d0bb0 msgr2=0x7f85380ca1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:07:58.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.992+0000 7f853eea8700 1 --2- 172.21.15.1:0/3508615416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85380d0bb0 0x7f85380ca1d0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f85380b3b90 tx=0x7f8530004ad0 comp rx=0 tx=0).stop 2024-03-18T19:07:58.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.992+0000 7f853eea8700 1 -- 172.21.15.1:0/3508615416 shutdown_connections 2024-03-18T19:07:58.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.992+0000 7f853eea8700 1 --2- 172.21.15.1:0/3508615416 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f851c06af20 0x7f851c06d3e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:58.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.992+0000 7f853eea8700 1 --2- 172.21.15.1:0/3508615416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85380d0bb0 0x7f85380ca1d0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:58.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.992+0000 7f853eea8700 1 --2- 172.21.15.1:0/3508615416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8538070f10 0x7f8538152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:58.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.992+0000 7f853eea8700 1 --2- 172.21.15.1:0/3508615416 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85380cb740 0x7f8538152ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:58.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.992+0000 7f853eea8700 1 -- 172.21.15.1:0/3508615416 >> 172.21.15.1:0/3508615416 conn(0x7f85380b2cc0 msgr2=0x7f85380b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:07:58.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.992+0000 7f853eea8700 1 -- 172.21.15.1:0/3508615416 shutdown_connections 2024-03-18T19:07:58.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:58.992+0000 7f853eea8700 1 -- 172.21.15.1:0/3508615416 wait complete. 2024-03-18T19:07:59.000 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T19:06:58", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2024-03-18T19:07:59.001 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /bad-path 2024-03-18T19:07:59.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.132+0000 7f2b411ce700 1 Processor -- start 2024-03-18T19:07:59.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.132+0000 7f2b411ce700 1 -- start start 2024-03-18T19:07:59.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.133+0000 7f2b411ce700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b3c0cc0f0 0x7f2b3c0add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:59.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.133+0000 7f2b411ce700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b3c0ce8f0 con 0x7f2b3c071d30 2024-03-18T19:07:59.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.133+0000 7f2b411ce700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b3c0ae260 con 0x7f2b3c15a1d0 2024-03-18T19:07:59.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.133+0000 7f2b411ce700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c0cdb00 con 0x7f2b3c0cc0f0 2024-03-18T19:07:59.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.133+0000 7f2b3a59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2b3c071d30 0x7f2b3c0cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34936/0 (socket says 172.21.15.1:34936) 2024-03-18T19:07:59.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.133+0000 7f2b3a59c700 1 -- 172.21.15.1:0/1613261638 learned_addr learned my addr 172.21.15.1:0/1613261638 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:07:59.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b3b59e700 1 --2- 172.21.15.1:0/1613261638 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b3c0cc0f0 0x7f2b3c0add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:59.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b39d9b700 1 -- 172.21.15.1:0/1613261638 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995897043 0 0) 0x7f2b3c0ce8f0 con 0x7f2b3c071d30 2024-03-18T19:07:59.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b39d9b700 1 -- 172.21.15.1:0/1613261638 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b240035b0 con 0x7f2b3c071d30 2024-03-18T19:07:59.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b39d9b700 1 -- 172.21.15.1:0/1613261638 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 573479575 0 0) 0x7f2b240035b0 con 0x7f2b3c071d30 2024-03-18T19:07:59.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b39d9b700 1 -- 172.21.15.1:0/1613261638 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b3c0cc0f0 msgr2=0x7f2b3c0add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:07:59.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b39d9b700 1 --2- 172.21.15.1:0/1613261638 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b3c0cc0f0 0x7f2b3c0add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:59.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b39d9b700 1 -- 172.21.15.1:0/1613261638 >> v1:172.21.15.165:6789/0 conn(0x7f2b3c15a1d0 legacy=0x7f2b3c071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:07:59.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b39d9b700 1 -- 172.21.15.1:0/1613261638 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c0cdce0 con 0x7f2b3c071d30 2024-03-18T19:07:59.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b39d9b700 1 -- 172.21.15.1:0/1613261638 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f2b2c001670 con 0x7f2b3c071d30 2024-03-18T19:07:59.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b39d9b700 1 -- 172.21.15.1:0/1613261638 >> v1:172.21.15.1:6789/0 conn(0x7f2b3c071d30 legacy=0x7f2b3c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:07:59.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b39d9b700 1 --2- 172.21.15.1:0/1613261638 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b3c0cc0f0 0x7f2b24003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:59.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b39d9b700 1 --2- 172.21.15.1:0/1613261638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b24003f70 0x7f2b24004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:59.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b39d9b700 1 --2- 172.21.15.1:0/1613261638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b24004d90 0x7f2b24005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:59.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b39d9b700 1 -- 172.21.15.1:0/1613261638 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c0cdb00 con 0x7f2b24004d90 2024-03-18T19:07:59.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b39d9b700 1 -- 172.21.15.1:0/1613261638 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2400ba70 con 0x7f2b24003f70 2024-03-18T19:07:59.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b39d9b700 1 -- 172.21.15.1:0/1613261638 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2400bc30 con 0x7f2b3c0cc0f0 2024-03-18T19:07:59.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b3b59e700 1 --2- 172.21.15.1:0/1613261638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b24004d90 0x7f2b24005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:59.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b3ad9d700 1 --2- 172.21.15.1:0/1613261638 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b3c0cc0f0 0x7f2b24003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:59.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b3a59c700 1 --2- 172.21.15.1:0/1613261638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b24003f70 0x7f2b24004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:59.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b3b59e700 1 -- 172.21.15.1:0/1613261638 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b3c0cc0f0 msgr2=0x7f2b24003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:07:59.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b3b59e700 1 --2- 172.21.15.1:0/1613261638 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b3c0cc0f0 0x7f2b24003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:59.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b3b59e700 1 -- 172.21.15.1:0/1613261638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b24003f70 msgr2=0x7f2b24004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:07:59.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b3b59e700 1 --2- 172.21.15.1:0/1613261638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b24003f70 0x7f2b24004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:59.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.134+0000 7f2b3b59e700 1 -- 172.21.15.1:0/1613261638 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b240035b0 con 0x7f2b24004d90 2024-03-18T19:07:59.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.135+0000 7f2b3b59e700 1 --2- 172.21.15.1:0/1613261638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b24004d90 0x7f2b24005210 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009f90 tx=0x7f2b24005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f1731ebbb41484c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:07:59.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.135+0000 7f2b39d9b700 1 -- 172.21.15.1:0/1613261638 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3400ec30 con 0x7f2b24004d90 2024-03-18T19:07:59.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.135+0000 7f2b39d9b700 1 -- 172.21.15.1:0/1613261638 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3400f040 con 0x7f2b24004d90 2024-03-18T19:07:59.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.135+0000 7f2b39d9b700 1 -- 172.21.15.1:0/1613261638 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34014480 con 0x7f2b24004d90 2024-03-18T19:07:59.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.135+0000 7f2b411ce700 1 -- 172.21.15.1:0/1613261638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b24004d90 msgr2=0x7f2b24005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:07:59.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.135+0000 7f2b411ce700 1 --2- 172.21.15.1:0/1613261638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b24004d90 0x7f2b24005210 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009f90 tx=0x7f2b24005880 comp rx=0 tx=0).stop 2024-03-18T19:07:59.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.135+0000 7f2b3b59e700 1 -- 172.21.15.1:0/1613261638 reap_dead start 2024-03-18T19:07:59.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.135+0000 7f2b411ce700 1 -- 172.21.15.1:0/1613261638 shutdown_connections 2024-03-18T19:07:59.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.135+0000 7f2b411ce700 1 -- 172.21.15.1:0/1613261638 >> 172.21.15.1:0/1613261638 conn(0x7f2b3c0cc9b0 msgr2=0x7f2b3c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:07:59.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.135+0000 7f2b411ce700 1 -- 172.21.15.1:0/1613261638 shutdown_connections 2024-03-18T19:07:59.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.135+0000 7f2b411ce700 1 -- 172.21.15.1:0/1613261638 wait complete. 2024-03-18T19:07:59.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.137+0000 7f2b411ce700 1 Processor -- start 2024-03-18T19:07:59.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.137+0000 7f2b411ce700 1 -- start start 2024-03-18T19:07:59.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.138+0000 7f2b411ce700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b3c157440 0x7f2b3c15b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:59.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.138+0000 7f2b411ce700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b3c15d310 0x7f2b3c0c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:59.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.138+0000 7f2b411ce700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b3c15bd60 0x7f2b3c0c6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:59.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.138+0000 7f2b411ce700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c157820 con 0x7f2b3c157440 2024-03-18T19:07:59.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.138+0000 7f2b411ce700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c15c270 con 0x7f2b3c15bd60 2024-03-18T19:07:59.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.138+0000 7f2b411ce700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c15c550 con 0x7f2b3c15d310 2024-03-18T19:07:59.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.138+0000 7f2b3ad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b3c157440 0x7f2b3c15b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:59.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.138+0000 7f2b3ad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b3c157440 0x7f2b3c15b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46482/0 (socket says 172.21.15.1:46482) 2024-03-18T19:07:59.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.138+0000 7f2b3ad9d700 1 -- 172.21.15.1:0/1023576188 learned_addr learned my addr 172.21.15.1:0/1023576188 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:07:59.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.138+0000 7f2b3a59c700 1 --2- 172.21.15.1:0/1023576188 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b3c15d310 0x7f2b3c0c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:59.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.138+0000 7f2b3b59e700 1 --2- 172.21.15.1:0/1023576188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b3c15bd60 0x7f2b3c0c6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:59.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.139+0000 7f2b3ad9d700 1 -- 172.21.15.1:0/1023576188 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b3c15d310 msgr2=0x7f2b3c0c5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:07:59.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.139+0000 7f2b3ad9d700 1 --2- 172.21.15.1:0/1023576188 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b3c15d310 0x7f2b3c0c5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:59.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.139+0000 7f2b3ad9d700 1 -- 172.21.15.1:0/1023576188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b3c15bd60 msgr2=0x7f2b3c0c6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:07:59.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.139+0000 7f2b3ad9d700 1 --2- 172.21.15.1:0/1023576188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b3c15bd60 0x7f2b3c0c6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:59.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.139+0000 7f2b3ad9d700 1 -- 172.21.15.1:0/1023576188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c15c830 con 0x7f2b3c157440 2024-03-18T19:07:59.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.139+0000 7f2b3b59e700 1 --2- 172.21.15.1:0/1023576188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b3c15bd60 0x7f2b3c0c6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:07:59.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.139+0000 7f2b3ad9d700 1 --2- 172.21.15.1:0/1023576188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b3c157440 0x7f2b3c15b820 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f2b3000f820 tx=0x7f2b3000fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:07:59.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.139+0000 7f2b237fe700 1 -- 172.21.15.1:0/1023576188 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b300052e0 con 0x7f2b3c157440 2024-03-18T19:07:59.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.139+0000 7f2b237fe700 1 -- 172.21.15.1:0/1023576188 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b30005480 con 0x7f2b3c157440 2024-03-18T19:07:59.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.139+0000 7f2b237fe700 1 -- 172.21.15.1:0/1023576188 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b30003910 con 0x7f2b3c157440 2024-03-18T19:07:59.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.139+0000 7f2b411ce700 1 -- 172.21.15.1:0/1023576188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b3c15cb10 con 0x7f2b3c157440 2024-03-18T19:07:59.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.139+0000 7f2b411ce700 1 -- 172.21.15.1:0/1023576188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b3c0b5970 con 0x7f2b3c157440 2024-03-18T19:07:59.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.140+0000 7f2b411ce700 1 -- 172.21.15.1:0/1023576188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b3c0b5b00 con 0x7f2b3c157440 2024-03-18T19:07:59.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.140+0000 7f2b237fe700 1 -- 172.21.15.1:0/1023576188 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 55) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f2b30003a70 con 0x7f2b3c157440 2024-03-18T19:07:59.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.141+0000 7f2b237fe700 1 --2- 172.21.15.1:0/1023576188 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2b2806af20 0x7f2b2806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:59.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.141+0000 7f2b237fe700 1 -- 172.21.15.1:0/1023576188 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f2b30089930 con 0x7f2b3c157440 2024-03-18T19:07:59.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.141+0000 7f2b3a59c700 1 --2- 172.21.15.1:0/1023576188 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2b2806af20 0x7f2b2806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:59.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.141+0000 7f2b3a59c700 1 --2- 172.21.15.1:0/1023576188 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2b2806af20 0x7f2b2806d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c0c3790 tx=0x7f2b2c000ed0 comp rx=0 tx=0).ready entity=mgr.7947 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:07:59.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.146+0000 7f2b237fe700 1 -- 172.21.15.1:0/1023576188 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2b30054530 con 0x7f2b3c157440 2024-03-18T19:07:59.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.374+0000 7f2b411ce700 1 -- 172.21.15.1:0/1023576188 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/bad-path", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b3c15c6c0 con 0x7f2b2806af20 2024-03-18T19:07:59.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.449+0000 7f2b237fe700 1 -- 172.21.15.1:0/1023576188 <== mgr.7947 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+36 (secure 0 0 0) 0x7f2b3c15c6c0 con 0x7f2b2806af20 2024-03-18T19:07:59.446 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule removed for path /bad-path 2024-03-18T19:07:59.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.451+0000 7f2b411ce700 1 -- 172.21.15.1:0/1023576188 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2b2806af20 msgr2=0x7f2b2806d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:07:59.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.451+0000 7f2b411ce700 1 --2- 172.21.15.1:0/1023576188 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2b2806af20 0x7f2b2806d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c0c3790 tx=0x7f2b2c000ed0 comp rx=0 tx=0).stop 2024-03-18T19:07:59.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.451+0000 7f2b411ce700 1 -- 172.21.15.1:0/1023576188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b3c157440 msgr2=0x7f2b3c15b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:07:59.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.451+0000 7f2b411ce700 1 --2- 172.21.15.1:0/1023576188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b3c157440 0x7f2b3c15b820 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f2b3000f820 tx=0x7f2b3000fbe0 comp rx=0 tx=0).stop 2024-03-18T19:07:59.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.451+0000 7f2b411ce700 1 -- 172.21.15.1:0/1023576188 shutdown_connections 2024-03-18T19:07:59.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.451+0000 7f2b411ce700 1 --2- 172.21.15.1:0/1023576188 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2b2806af20 0x7f2b2806d3e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:59.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.451+0000 7f2b411ce700 1 --2- 172.21.15.1:0/1023576188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b3c15bd60 0x7f2b3c0c6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:59.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.451+0000 7f2b411ce700 1 --2- 172.21.15.1:0/1023576188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2b3c157440 0x7f2b3c15b820 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:59.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.451+0000 7f2b411ce700 1 --2- 172.21.15.1:0/1023576188 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2b3c15d310 0x7f2b3c0c5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:59.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.451+0000 7f2b411ce700 1 -- 172.21.15.1:0/1023576188 >> 172.21.15.1:0/1023576188 conn(0x7f2b3c0cc9b0 msgr2=0x7f2b3c1594a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:07:59.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.452+0000 7f2b411ce700 1 -- 172.21.15.1:0/1023576188 shutdown_connections 2024-03-18T19:07:59.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.452+0000 7f2b411ce700 1 -- 172.21.15.1:0/1023576188 wait complete. 2024-03-18T19:07:59.460 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2024-03-18T19:07:59.460 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-18T19:07:59.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.591+0000 7f3575444700 1 Processor -- start 2024-03-18T19:07:59.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.591+0000 7f3575444700 1 -- start start 2024-03-18T19:07:59.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.591+0000 7f3575444700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35700cbd40 0x7f35700cc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:59.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.591+0000 7f3575444700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f357006b130 0x7f35700cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:59.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.591+0000 7f3575444700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35700cf730 con 0x7f35700c3670 2024-03-18T19:07:59.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.591+0000 7f3575444700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35700cc660 con 0x7f35700cbd40 2024-03-18T19:07:59.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.591+0000 7f3575444700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35700cf980 con 0x7f357006b130 2024-03-18T19:07:59.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.591+0000 7f356effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f35700cbd40 0x7f35700cc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:59.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.592+0000 7f356e7fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f35700c3670 0x7f357015e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34946/0 (socket says 172.21.15.1:34946) 2024-03-18T19:07:59.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.592+0000 7f356e7fc700 1 -- 172.21.15.1:0/2776138880 learned_addr learned my addr 172.21.15.1:0/2776138880 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:07:59.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.592+0000 7f356f7fe700 1 --2- 172.21.15.1:0/2776138880 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f357006b130 0x7f35700cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:59.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.592+0000 7f356effd700 1 -- 172.21.15.1:0/2776138880 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f357006b130 msgr2=0x7f35700cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:07:59.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.592+0000 7f356effd700 1 --2- 172.21.15.1:0/2776138880 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f357006b130 0x7f35700cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:59.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.592+0000 7f356effd700 1 -- 172.21.15.1:0/2776138880 >> v1:172.21.15.1:6789/0 conn(0x7f35700c3670 legacy=0x7f357015e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:07:59.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.592+0000 7f356effd700 1 -- 172.21.15.1:0/2776138880 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35700cfc60 con 0x7f35700cbd40 2024-03-18T19:07:59.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.592+0000 7f356dffb700 1 -- 172.21.15.1:0/2776138880 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811827774 0 0) 0x7f35700cf730 con 0x7f35700c3670 2024-03-18T19:07:59.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.592+0000 7f356f7fe700 1 --2- 172.21.15.1:0/2776138880 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f357006b130 0x7f35700cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:07:59.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.592+0000 7f356effd700 1 --2- 172.21.15.1:0/2776138880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35700cbd40 0x7f35700cc120 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f3558009840 tx=0x7f3558009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15c61478a7f7ee1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:07:59.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.592+0000 7f356dffb700 1 -- 172.21.15.1:0/2776138880 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3558011830 con 0x7f35700cbd40 2024-03-18T19:07:59.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.592+0000 7f356dffb700 1 -- 172.21.15.1:0/2776138880 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3558011990 con 0x7f35700cbd40 2024-03-18T19:07:59.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.592+0000 7f356dffb700 1 -- 172.21.15.1:0/2776138880 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3558011d20 con 0x7f35700cbd40 2024-03-18T19:07:59.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.593+0000 7f3575444700 1 -- 172.21.15.1:0/2776138880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35700cbd40 msgr2=0x7f35700cc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:07:59.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.593+0000 7f3575444700 1 --2- 172.21.15.1:0/2776138880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35700cbd40 0x7f35700cc120 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f3558009840 tx=0x7f3558009b50 comp rx=0 tx=0).stop 2024-03-18T19:07:59.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.593+0000 7f3575444700 1 -- 172.21.15.1:0/2776138880 shutdown_connections 2024-03-18T19:07:59.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.593+0000 7f3575444700 1 --2- 172.21.15.1:0/2776138880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f35700cbd40 0x7f35700cc120 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:59.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.593+0000 7f3575444700 1 --2- 172.21.15.1:0/2776138880 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f357006b130 0x7f35700cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:59.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.593+0000 7f3575444700 1 -- 172.21.15.1:0/2776138880 >> 172.21.15.1:0/2776138880 conn(0x7f35700b1aa0 msgr2=0x7f35700b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:07:59.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.593+0000 7f3575444700 1 -- 172.21.15.1:0/2776138880 shutdown_connections 2024-03-18T19:07:59.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.593+0000 7f3575444700 1 -- 172.21.15.1:0/2776138880 wait complete. 2024-03-18T19:07:59.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.594+0000 7f3575444700 1 Processor -- start 2024-03-18T19:07:59.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.595+0000 7f3575444700 1 -- start start 2024-03-18T19:07:59.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.595+0000 7f3575444700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f357006b130 0x7f357015ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:59.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.595+0000 7f3575444700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35700c3670 0x7f35700b0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:59.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.595+0000 7f3575444700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f35700cbd40 0x7f35700b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:59.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.595+0000 7f3575444700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35700b5cc0 con 0x7f357006b130 2024-03-18T19:07:59.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.595+0000 7f3575444700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35700b59e0 con 0x7f35700c3670 2024-03-18T19:07:59.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.595+0000 7f3575444700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f357009f4c0 con 0x7f35700cbd40 2024-03-18T19:07:59.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.596+0000 7f356effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f357006b130 0x7f357015ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:59.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.596+0000 7f356effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f357006b130 0x7f357015ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46500/0 (socket says 172.21.15.1:46500) 2024-03-18T19:07:59.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.596+0000 7f356effd700 1 -- 172.21.15.1:0/3417336759 learned_addr learned my addr 172.21.15.1:0/3417336759 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:07:59.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.596+0000 7f356f7fe700 1 --2- 172.21.15.1:0/3417336759 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f35700cbd40 0x7f35700b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:59.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.596+0000 7f356e7fc700 1 --2- 172.21.15.1:0/3417336759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35700c3670 0x7f35700b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:59.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.596+0000 7f356effd700 1 -- 172.21.15.1:0/3417336759 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f35700cbd40 msgr2=0x7f35700b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:07:59.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.596+0000 7f356effd700 1 --2- 172.21.15.1:0/3417336759 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f35700cbd40 0x7f35700b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:59.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.596+0000 7f356effd700 1 -- 172.21.15.1:0/3417336759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35700c3670 msgr2=0x7f35700b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:07:59.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.596+0000 7f356effd700 1 --2- 172.21.15.1:0/3417336759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35700c3670 0x7f35700b0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:07:59.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.596+0000 7f356effd700 1 -- 172.21.15.1:0/3417336759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35580094a0 con 0x7f357006b130 2024-03-18T19:07:59.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.596+0000 7f356effd700 1 --2- 172.21.15.1:0/3417336759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f357006b130 0x7f357015ea40 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f3558014040 tx=0x7f3558004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:07:59.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.596+0000 7f35677fe700 1 -- 172.21.15.1:0/3417336759 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35580045f0 con 0x7f357006b130 2024-03-18T19:07:59.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.596+0000 7f35677fe700 1 -- 172.21.15.1:0/3417336759 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355801e070 con 0x7f357006b130 2024-03-18T19:07:59.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.596+0000 7f3575444700 1 -- 172.21.15.1:0/3417336759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f357009f7a0 con 0x7f357006b130 2024-03-18T19:07:59.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.596+0000 7f35677fe700 1 -- 172.21.15.1:0/3417336759 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3558003ae0 con 0x7f357006b130 2024-03-18T19:07:59.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.596+0000 7f3575444700 1 -- 172.21.15.1:0/3417336759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f357015ef80 con 0x7f357006b130 2024-03-18T19:07:59.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.597+0000 7f3575444700 1 -- 172.21.15.1:0/3417336759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35700cf580 con 0x7f357006b130 2024-03-18T19:07:59.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.598+0000 7f35677fe700 1 -- 172.21.15.1:0/3417336759 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 55) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f3558003c40 con 0x7f357006b130 2024-03-18T19:07:59.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.598+0000 7f35677fe700 1 --2- 172.21.15.1:0/3417336759 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f355006af20 0x7f355006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:07:59.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.598+0000 7f35677fe700 1 -- 172.21.15.1:0/3417336759 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f355808ab60 con 0x7f357006b130 2024-03-18T19:07:59.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.598+0000 7f356e7fc700 1 --2- 172.21.15.1:0/3417336759 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f355006af20 0x7f355006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:07:59.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.599+0000 7f356e7fc700 1 --2- 172.21.15.1:0/3417336759 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f355006af20 0x7f355006d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3560000f60 tx=0x7f3560007be0 comp rx=0 tx=0).ready entity=mgr.7947 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:07:59.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.603+0000 7f35677fe700 1 -- 172.21.15.1:0/3417336759 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3558055760 con 0x7f357006b130 2024-03-18T19:07:59.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:07:59.861+0000 7f3575444700 1 -- 172.21.15.1:0/3417336759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f357009fcc0 con 0x7f357006b130 2024-03-18T19:08:00.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.104+0000 7f35677fe700 1 -- 172.21.15.1:0/3417336759 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 56) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f3558055b30 con 0x7f357006b130 2024-03-18T19:08:00.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.105+0000 7f35677fe700 1 -- 172.21.15.1:0/3417336759 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v56) v1 ==== 93+0+0 (secure 0 0 0) 0x7f35580591f0 con 0x7f357006b130 2024-03-18T19:08:00.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.107+0000 7f3575444700 1 -- 172.21.15.1:0/3417336759 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f355006af20 msgr2=0x7f355006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:00.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.107+0000 7f3575444700 1 --2- 172.21.15.1:0/3417336759 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f355006af20 0x7f355006d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3560000f60 tx=0x7f3560007be0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.107+0000 7f3575444700 1 -- 172.21.15.1:0/3417336759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f357006b130 msgr2=0x7f357015ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:00.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.107+0000 7f3575444700 1 --2- 172.21.15.1:0/3417336759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f357006b130 0x7f357015ea40 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f3558014040 tx=0x7f3558004750 comp rx=0 tx=0).stop 2024-03-18T19:08:00.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.107+0000 7f3575444700 1 -- 172.21.15.1:0/3417336759 shutdown_connections 2024-03-18T19:08:00.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.107+0000 7f3575444700 1 --2- 172.21.15.1:0/3417336759 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f355006af20 0x7f355006d3e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.107+0000 7f3575444700 1 --2- 172.21.15.1:0/3417336759 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f35700cbd40 0x7f35700b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.107+0000 7f3575444700 1 --2- 172.21.15.1:0/3417336759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f357006b130 0x7f357015ea40 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.107+0000 7f3575444700 1 --2- 172.21.15.1:0/3417336759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35700c3670 0x7f35700b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.107+0000 7f3575444700 1 -- 172.21.15.1:0/3417336759 >> 172.21.15.1:0/3417336759 conn(0x7f35700b1aa0 msgr2=0x7f357015e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:00.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.107+0000 7f3575444700 1 -- 172.21.15.1:0/3417336759 shutdown_connections 2024-03-18T19:08:00.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.107+0000 7f3575444700 1 -- 172.21.15.1:0/3417336759 wait complete. 2024-03-18T19:08:00.118 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T19:08:00.118 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:08:00.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:08:00.204 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T19:08:00.205 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:08:00.205 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:08:00.224 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:08:00.245 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:00.249+0000 7f30b35e4240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:08:00.251 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:00.255+0000 7f1b3c187240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:08:00.400 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:00.394+0000 7f30b35e4240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:08:00.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.425+0000 7f98a72ed700 1 Processor -- start 2024-03-18T19:08:00.422 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:00.425+0000 7f1b3c187240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:08:00.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.425+0000 7f98a72ed700 1 -- start start 2024-03-18T19:08:00.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.426+0000 7f98a72ed700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f98a015aff0 0x7f98a015f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:00.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.426+0000 7f98a72ed700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98a015b3d0 0x7f98a00cf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:00.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.426+0000 7f98a72ed700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a015fcc0 con 0x7f98a0073b00 2024-03-18T19:08:00.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.426+0000 7f98a72ed700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98a00ce200 con 0x7f98a015aff0 2024-03-18T19:08:00.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.426+0000 7f98a72ed700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00ce4a0 con 0x7f98a015b3d0 2024-03-18T19:08:00.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.426+0000 7f98a5089700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f98a015aff0 0x7f98a015f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.426+0000 7f98a5089700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f98a015aff0 0x7f98a015f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46526/0 (socket says 172.21.15.1:46526) 2024-03-18T19:08:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.426+0000 7f98a5089700 1 -- 172.21.15.1:0/3801935836 learned_addr learned my addr 172.21.15.1:0/3801935836 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.427+0000 7f98a4888700 1 --2- 172.21.15.1:0/3801935836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98a015b3d0 0x7f98a00cf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.427+0000 7f98a5089700 1 -- 172.21.15.1:0/3801935836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98a015b3d0 msgr2=0x7f98a00cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.427+0000 7f98a5089700 1 --2- 172.21.15.1:0/3801935836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98a015b3d0 0x7f98a00cf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.427+0000 7f98a5089700 1 -- 172.21.15.1:0/3801935836 >> v1:172.21.15.165:6789/0 conn(0x7f98a0073b00 legacy=0x7f98a00cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.427+0000 7f98a5089700 1 -- 172.21.15.1:0/3801935836 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a00ce780 con 0x7f98a015aff0 2024-03-18T19:08:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.427+0000 7f98a5089700 1 --2- 172.21.15.1:0/3801935836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a015aff0 0x7f98a015f780 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f9898009e50 tx=0x7f989800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e71ba01df0ef61de server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.427+0000 7f9897fff700 1 -- 172.21.15.1:0/3801935836 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989800e070 con 0x7f98a015aff0 2024-03-18T19:08:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.427+0000 7f9897fff700 1 -- 172.21.15.1:0/3801935836 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989800b910 con 0x7f98a015aff0 2024-03-18T19:08:00.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.427+0000 7f9897fff700 1 -- 172.21.15.1:0/3801935836 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989800bcc0 con 0x7f98a015aff0 2024-03-18T19:08:00.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.428+0000 7f98a72ed700 1 -- 172.21.15.1:0/3801935836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a015aff0 msgr2=0x7f98a015f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:00.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.428+0000 7f98a72ed700 1 --2- 172.21.15.1:0/3801935836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a015aff0 0x7f98a015f780 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f9898009e50 tx=0x7f989800b330 comp rx=0 tx=0).stop 2024-03-18T19:08:00.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.428+0000 7f98a72ed700 1 -- 172.21.15.1:0/3801935836 shutdown_connections 2024-03-18T19:08:00.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.428+0000 7f98a72ed700 1 --2- 172.21.15.1:0/3801935836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a015aff0 0x7f98a015f780 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.428+0000 7f98a72ed700 1 --2- 172.21.15.1:0/3801935836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98a015b3d0 0x7f98a00cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.428+0000 7f98a72ed700 1 -- 172.21.15.1:0/3801935836 >> 172.21.15.1:0/3801935836 conn(0x7f98a006a750 msgr2=0x7f98a006ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.428+0000 7f98a72ed700 1 -- 172.21.15.1:0/3801935836 shutdown_connections 2024-03-18T19:08:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.428+0000 7f98a72ed700 1 -- 172.21.15.1:0/3801935836 wait complete. 2024-03-18T19:08:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.429+0000 7f98a72ed700 1 Processor -- start 2024-03-18T19:08:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.429+0000 7f98a72ed700 1 -- start start 2024-03-18T19:08:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.430+0000 7f98a72ed700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98a0073b00 0x7f98a0155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.430+0000 7f98a72ed700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a015aff0 0x7f98a013cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.430+0000 7f98a72ed700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98a015b3d0 0x7f98a013d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.430+0000 7f98a72ed700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a0156110 con 0x7f98a015aff0 2024-03-18T19:08:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.430+0000 7f98a72ed700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00b0420 con 0x7f98a015b3d0 2024-03-18T19:08:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.430+0000 7f98a72ed700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00b06c0 con 0x7f98a0073b00 2024-03-18T19:08:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.430+0000 7f98a5089700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98a0073b00 0x7f98a0155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.430+0000 7f98a5089700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98a0073b00 0x7f98a0155bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:45050/0 (socket says 172.21.15.1:45050) 2024-03-18T19:08:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.430+0000 7f98a5089700 1 -- 172.21.15.1:0/29988463 learned_addr learned my addr 172.21.15.1:0/29988463 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:00.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.430+0000 7f98a588a700 1 --2- 172.21.15.1:0/29988463 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98a015b3d0 0x7f98a013d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.430+0000 7f98a4888700 1 --2- 172.21.15.1:0/29988463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a015aff0 0x7f98a013cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.430+0000 7f98a5089700 1 -- 172.21.15.1:0/29988463 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98a015b3d0 msgr2=0x7f98a013d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.430+0000 7f98a5089700 1 --2- 172.21.15.1:0/29988463 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98a015b3d0 0x7f98a013d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.430+0000 7f98a5089700 1 -- 172.21.15.1:0/29988463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a015aff0 msgr2=0x7f98a013cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.430+0000 7f98a5089700 1 --2- 172.21.15.1:0/29988463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a015aff0 0x7f98a013cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.430+0000 7f98a5089700 1 -- 172.21.15.1:0/29988463 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9898009b30 con 0x7f98a0073b00 2024-03-18T19:08:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.430+0000 7f98a4888700 1 --2- 172.21.15.1:0/29988463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a015aff0 0x7f98a013cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.431+0000 7f98a5089700 1 --2- 172.21.15.1:0/29988463 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98a0073b00 0x7f98a0155bd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f989800b370 tx=0x7f9898004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.431+0000 7f9895ffb700 1 -- 172.21.15.1:0/29988463 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9898004680 con 0x7f98a0073b00 2024-03-18T19:08:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.431+0000 7f9895ffb700 1 -- 172.21.15.1:0/29988463 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9898004800 con 0x7f98a0073b00 2024-03-18T19:08:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.431+0000 7f98a72ed700 1 -- 172.21.15.1:0/29988463 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a013d8e0 con 0x7f98a0073b00 2024-03-18T19:08:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.431+0000 7f9895ffb700 1 -- 172.21.15.1:0/29988463 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9898004970 con 0x7f98a0073b00 2024-03-18T19:08:00.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.431+0000 7f98a72ed700 1 -- 172.21.15.1:0/29988463 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a013dea0 con 0x7f98a0073b00 2024-03-18T19:08:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.432+0000 7f98a72ed700 1 -- 172.21.15.1:0/29988463 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a013e060 con 0x7f98a0073b00 2024-03-18T19:08:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.433+0000 7f9895ffb700 1 -- 172.21.15.1:0/29988463 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 56) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f9898024070 con 0x7f98a0073b00 2024-03-18T19:08:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.433+0000 7f9895ffb700 1 --2- 172.21.15.1:0/29988463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f988406aed0 0x7f988406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:00.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.433+0000 7f98a4888700 1 -- 172.21.15.1:0/29988463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f988406aed0 msgr2=0x7f988406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:00.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.433+0000 7f9895ffb700 1 -- 172.21.15.1:0/29988463 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f989808eff0 con 0x7f98a0073b00 2024-03-18T19:08:00.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.433+0000 7f98a4888700 1 --2- 172.21.15.1:0/29988463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f988406aed0 0x7f988406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:08:00.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.438+0000 7f9895ffb700 1 -- 172.21.15.1:0/29988463 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f989805d5e0 con 0x7f98a0073b00 2024-03-18T19:08:00.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.634+0000 7f98a4888700 1 -- 172.21.15.1:0/29988463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f988406aed0 msgr2=0x7f988406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:00.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.634+0000 7f98a4888700 1 --2- 172.21.15.1:0/29988463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f988406aed0 0x7f988406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:08:00.656 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:00.660+0000 7f30b35e4240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:08:00.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.666+0000 7f98a72ed700 1 -- 172.21.15.1:0/29988463 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98a00676a0 con 0x7f98a0073b00 2024-03-18T19:08:00.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.668+0000 7f9895ffb700 1 -- 172.21.15.1:0/29988463 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16887 (secure 0 0 0) 0x7f989807ae60 con 0x7f98a0073b00 2024-03-18T19:08:00.665 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:00.665 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":120,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:07:59.090022+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:06:39.741939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-18T19:06:40.871304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.10","mappings":[{"from":1,"to":2}]},{"pgid":"14.18","mappings":[{"from":1,"to":5}]},{"pgid":"14.21","mappings":[{"from":1,"to":7}]},{"pgid":"14.29","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/1072638685":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/67600845":"2024-03-19T19:06:57.097725+0000","172.21.15.1:6833/76998":"2024-03-19T19:06:57.097725+0000","172.21.15.1:6832/76998":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/1805165487":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/1343251370":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/3890713729":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/2823326236":"2024-03-19T19:06:57.097725+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:08:00.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.670+0000 7f98a72ed700 1 -- 172.21.15.1:0/29988463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f988406aed0 msgr2=0x7f988406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:08:00.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.670+0000 7f98a72ed700 1 --2- 172.21.15.1:0/29988463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f988406aed0 0x7f988406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.670+0000 7f98a72ed700 1 -- 172.21.15.1:0/29988463 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98a0073b00 msgr2=0x7f98a0155bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:00.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.670+0000 7f98a72ed700 1 --2- 172.21.15.1:0/29988463 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98a0073b00 0x7f98a0155bd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f989800b370 tx=0x7f9898004ad0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.670+0000 7f98a72ed700 1 -- 172.21.15.1:0/29988463 shutdown_connections 2024-03-18T19:08:00.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.670+0000 7f98a72ed700 1 --2- 172.21.15.1:0/29988463 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f988406aed0 0x7f988406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.670+0000 7f98a72ed700 1 --2- 172.21.15.1:0/29988463 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f98a0073b00 0x7f98a0155bd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.670+0000 7f98a72ed700 1 --2- 172.21.15.1:0/29988463 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98a015b3d0 0x7f98a013d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.670+0000 7f98a72ed700 1 --2- 172.21.15.1:0/29988463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98a015aff0 0x7f98a013cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.670+0000 7f98a72ed700 1 -- 172.21.15.1:0/29988463 >> 172.21.15.1:0/29988463 conn(0x7f98a006a750 msgr2=0x7f98a00cf540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:00.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.670+0000 7f98a72ed700 1 -- 172.21.15.1:0/29988463 shutdown_connections 2024-03-18T19:08:00.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.670+0000 7f98a72ed700 1 -- 172.21.15.1:0/29988463 wait complete. 2024-03-18T19:08:00.680 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T19:06:39.741939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T19:06:40.871304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:08:00.680 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:08:00.722 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:00.726+0000 7f1b3c187240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:08:00.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.814+0000 7f266c5af700 1 Processor -- start 2024-03-18T19:08:00.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.814+0000 7f266c5af700 1 -- start start 2024-03-18T19:08:00.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.815+0000 7f266c5af700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2664099a40 0x7f26640ade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:00.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.815+0000 7f266c5af700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2664099e20 0x7f26640cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:00.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.815+0000 7f266c5af700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26640b3aa0 con 0x7f26640ccc20 2024-03-18T19:08:00.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.815+0000 7f266c5af700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f266406e9b0 con 0x7f2664099e20 2024-03-18T19:08:00.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.815+0000 7f266c5af700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f266406ec90 con 0x7f2664099a40 2024-03-18T19:08:00.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.815+0000 7f2669b4a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2664099e20 0x7f26640cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:00.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.815+0000 7f266ab4c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f26640ccc20 0x7f266406fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34954/0 (socket says 172.21.15.1:34954) 2024-03-18T19:08:00.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.815+0000 7f266ab4c700 1 -- 172.21.15.1:0/1338582175 learned_addr learned my addr 172.21.15.1:0/1338582175 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:00.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.816+0000 7f266a34b700 1 --2- 172.21.15.1:0/1338582175 >> v2:172.21.15.165:3300/0 conn(0x7f2664099a40 0x7f26640ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:00.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.816+0000 7f2669b4a700 1 -- 172.21.15.1:0/1338582175 >> v2:172.21.15.165:3300/0 conn(0x7f2664099a40 msgr2=0x7f26640ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:00.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.816+0000 7f2669b4a700 1 --2- 172.21.15.1:0/1338582175 >> v2:172.21.15.165:3300/0 conn(0x7f2664099a40 0x7f26640ade00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.816+0000 7f2669b4a700 1 -- 172.21.15.1:0/1338582175 >> v1:172.21.15.1:6789/0 conn(0x7f26640ccc20 legacy=0x7f266406fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:00.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.816+0000 7f2669349700 1 -- 172.21.15.1:0/1338582175 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237283037 0 0) 0x7f26640b3aa0 con 0x7f26640ccc20 2024-03-18T19:08:00.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.816+0000 7f2669b4a700 1 -- 172.21.15.1:0/1338582175 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266406ef70 con 0x7f2664099e20 2024-03-18T19:08:00.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.816+0000 7f2669b4a700 1 --2- 172.21.15.1:0/1338582175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2664099e20 0x7f26640cc6e0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f2664071d40 tx=0x7f2654009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c947218d94d7dd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:00.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.816+0000 7f2669349700 1 -- 172.21.15.1:0/1338582175 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2654011830 con 0x7f2664099e20 2024-03-18T19:08:00.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.816+0000 7f2669349700 1 -- 172.21.15.1:0/1338582175 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2654011990 con 0x7f2664099e20 2024-03-18T19:08:00.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.816+0000 7f2669349700 1 -- 172.21.15.1:0/1338582175 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2654011d20 con 0x7f2664099e20 2024-03-18T19:08:00.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.817+0000 7f266c5af700 1 -- 172.21.15.1:0/1338582175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2664099e20 msgr2=0x7f26640cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:00.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.817+0000 7f266c5af700 1 --2- 172.21.15.1:0/1338582175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2664099e20 0x7f26640cc6e0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f2664071d40 tx=0x7f2654009b20 comp rx=0 tx=0).stop 2024-03-18T19:08:00.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.817+0000 7f266c5af700 1 -- 172.21.15.1:0/1338582175 shutdown_connections 2024-03-18T19:08:00.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.817+0000 7f266c5af700 1 --2- 172.21.15.1:0/1338582175 >> v2:172.21.15.165:3300/0 conn(0x7f2664099a40 0x7f26640ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.817+0000 7f266c5af700 1 --2- 172.21.15.1:0/1338582175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2664099e20 0x7f26640cc6e0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.817+0000 7f266c5af700 1 -- 172.21.15.1:0/1338582175 >> 172.21.15.1:0/1338582175 conn(0x7f26640b2c50 msgr2=0x7f26640b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:00.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.817+0000 7f266c5af700 1 -- 172.21.15.1:0/1338582175 shutdown_connections 2024-03-18T19:08:00.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.817+0000 7f266c5af700 1 -- 172.21.15.1:0/1338582175 wait complete. 2024-03-18T19:08:00.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.818+0000 7f266c5af700 1 Processor -- start 2024-03-18T19:08:00.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.818+0000 7f266c5af700 1 -- start start 2024-03-18T19:08:00.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.819+0000 7f266c5af700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2664099a40 0x7f26640b6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:00.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.819+0000 7f266c5af700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2664099e20 0x7f26641f6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:00.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.819+0000 7f266c5af700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26640ccc20 0x7f266415daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:00.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.819+0000 7f266c5af700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26641f63a0 con 0x7f2664099a40 2024-03-18T19:08:00.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.819+0000 7f266c5af700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26640c5770 con 0x7f26640ccc20 2024-03-18T19:08:00.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.819+0000 7f266c5af700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26640c7120 con 0x7f2664099e20 2024-03-18T19:08:00.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.819+0000 7f266a34b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2664099a40 0x7f26640b6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:00.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.819+0000 7f266a34b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2664099a40 0x7f26640b6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46552/0 (socket says 172.21.15.1:46552) 2024-03-18T19:08:00.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.819+0000 7f266a34b700 1 -- 172.21.15.1:0/1867877059 learned_addr learned my addr 172.21.15.1:0/1867877059 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:00.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.819+0000 7f266ab4c700 1 --2- 172.21.15.1:0/1867877059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26640ccc20 0x7f266415daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:00.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.819+0000 7f2669b4a700 1 --2- 172.21.15.1:0/1867877059 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2664099e20 0x7f26641f6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:00.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.819+0000 7f266a34b700 1 -- 172.21.15.1:0/1867877059 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2664099e20 msgr2=0x7f26641f6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:00.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.819+0000 7f266a34b700 1 --2- 172.21.15.1:0/1867877059 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2664099e20 0x7f26641f6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.819+0000 7f266a34b700 1 -- 172.21.15.1:0/1867877059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26640ccc20 msgr2=0x7f266415daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:00.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.819+0000 7f266a34b700 1 --2- 172.21.15.1:0/1867877059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26640ccc20 0x7f266415daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:00.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.819+0000 7f266a34b700 1 -- 172.21.15.1:0/1867877059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26540094a0 con 0x7f2664099a40 2024-03-18T19:08:00.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.819+0000 7f266a34b700 1 --2- 172.21.15.1:0/1867877059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2664099a40 0x7f26640b6770 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f266000e870 tx=0x7f266000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:00.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.820+0000 7f265affd700 1 -- 172.21.15.1:0/1867877059 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2660009590 con 0x7f2664099a40 2024-03-18T19:08:00.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.820+0000 7f265affd700 1 -- 172.21.15.1:0/1867877059 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2660013680 con 0x7f2664099a40 2024-03-18T19:08:00.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.820+0000 7f266c5af700 1 -- 172.21.15.1:0/1867877059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26640c7400 con 0x7f2664099a40 2024-03-18T19:08:00.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.820+0000 7f265affd700 1 -- 172.21.15.1:0/1867877059 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26600139f0 con 0x7f2664099a40 2024-03-18T19:08:00.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.820+0000 7f266c5af700 1 -- 172.21.15.1:0/1867877059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26641f71d0 con 0x7f2664099a40 2024-03-18T19:08:00.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.820+0000 7f266c5af700 1 -- 172.21.15.1:0/1867877059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26640c7920 con 0x7f2664099a40 2024-03-18T19:08:00.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.821+0000 7f265affd700 1 -- 172.21.15.1:0/1867877059 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 56) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f2660013b50 con 0x7f2664099a40 2024-03-18T19:08:00.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.822+0000 7f265affd700 1 --2- 172.21.15.1:0/1867877059 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f264806aed0 0x7f264806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:00.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.822+0000 7f2669b4a700 1 -- 172.21.15.1:0/1867877059 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f264806aed0 msgr2=0x7f264806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:00.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.822+0000 7f265affd700 1 -- 172.21.15.1:0/1867877059 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f26600898c0 con 0x7f2664099a40 2024-03-18T19:08:00.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.822+0000 7f2669b4a700 1 --2- 172.21.15.1:0/1867877059 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f264806aed0 0x7f264806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:08:00.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:00.826+0000 7f265affd700 1 -- 172.21.15.1:0/1867877059 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f26640c7920 con 0x7f2664099a40 2024-03-18T19:08:01.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.023+0000 7f2669b4a700 1 -- 172.21.15.1:0/1867877059 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f264806aed0 msgr2=0x7f264806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:01.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.023+0000 7f2669b4a700 1 --2- 172.21.15.1:0/1867877059 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f264806aed0 0x7f264806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:08:01.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.028+0000 7f266c5af700 1 -- 172.21.15.1:0/1867877059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f26640c7920 con 0x7f2664099a40 2024-03-18T19:08:01.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.029+0000 7f265affd700 1 -- 172.21.15.1:0/1867877059 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) v1 ==== 93+0+10 (secure 0 0 0) 0x7f26640c7920 con 0x7f2664099a40 2024-03-18T19:08:01.026 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:08:01.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.031+0000 7f266c5af700 1 -- 172.21.15.1:0/1867877059 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f264806aed0 msgr2=0x7f264806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:08:01.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.031+0000 7f266c5af700 1 --2- 172.21.15.1:0/1867877059 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f264806aed0 0x7f264806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.031+0000 7f266c5af700 1 -- 172.21.15.1:0/1867877059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2664099a40 msgr2=0x7f26640b6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:01.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.031+0000 7f266c5af700 1 --2- 172.21.15.1:0/1867877059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2664099a40 0x7f26640b6770 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f266000e870 tx=0x7f266000eb80 comp rx=0 tx=0).stop 2024-03-18T19:08:01.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.031+0000 7f266c5af700 1 -- 172.21.15.1:0/1867877059 shutdown_connections 2024-03-18T19:08:01.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.031+0000 7f266c5af700 1 --2- 172.21.15.1:0/1867877059 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f264806aed0 0x7f264806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.031+0000 7f266c5af700 1 --2- 172.21.15.1:0/1867877059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26640ccc20 0x7f266415daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.031+0000 7f266c5af700 1 --2- 172.21.15.1:0/1867877059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2664099a40 0x7f26640b6770 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.031+0000 7f266c5af700 1 --2- 172.21.15.1:0/1867877059 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2664099e20 0x7f26641f6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.031+0000 7f266c5af700 1 -- 172.21.15.1:0/1867877059 >> 172.21.15.1:0/1867877059 conn(0x7f26640b2c50 msgr2=0x7f26640ca320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:01.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.031+0000 7f266c5af700 1 -- 172.21.15.1:0/1867877059 shutdown_connections 2024-03-18T19:08:01.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.031+0000 7f266c5af700 1 -- 172.21.15.1:0/1867877059 wait complete. 2024-03-18T19:08:01.040 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T19:08:01.080 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:01.083+0000 7f30b35e4240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:08:01.187 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:01.190+0000 7f1b3c187240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:08:01.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.214+0000 7f000aad0700 1 Processor -- start 2024-03-18T19:08:01.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.214+0000 7f000aad0700 1 -- start start 2024-03-18T19:08:01.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.215+0000 7f000aad0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f00040cc770 0x7f00040cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:01.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.215+0000 7f000aad0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000406f370 0x7f000406f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:01.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.215+0000 7f000aad0700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00040cd030 con 0x7f000415b600 2024-03-18T19:08:01.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.215+0000 7f000aad0700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00040b3a90 con 0x7f00040cc770 2024-03-18T19:08:01.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.215+0000 7f000aad0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000409aa50 con 0x7f000406f370 2024-03-18T19:08:01.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.215+0000 7f0003fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f00040cc770 0x7f00040cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:01.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.216+0000 7f0003fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f00040cc770 0x7f00040cf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46558/0 (socket says 172.21.15.1:46558) 2024-03-18T19:08:01.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.216+0000 7f000886c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f000415b600 0x7f00040ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34962/0 (socket says 172.21.15.1:34962) 2024-03-18T19:08:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.216+0000 7f0003fff700 1 -- 172.21.15.1:0/2620310188 learned_addr learned my addr 172.21.15.1:0/2620310188 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.216+0000 7f000906d700 1 --2- 172.21.15.1:0/2620310188 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000406f370 0x7f000406f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.216+0000 7f0003fff700 1 -- 172.21.15.1:0/2620310188 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000406f370 msgr2=0x7f000406f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.216+0000 7f0003fff700 1 --2- 172.21.15.1:0/2620310188 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000406f370 0x7f000406f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.216+0000 7f0003fff700 1 -- 172.21.15.1:0/2620310188 >> v1:172.21.15.1:6789/0 conn(0x7f000415b600 legacy=0x7f00040ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.216+0000 7f0003fff700 1 -- 172.21.15.1:0/2620310188 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000415b9e0 con 0x7f00040cc770 2024-03-18T19:08:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.216+0000 7f00037fe700 1 -- 172.21.15.1:0/2620310188 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527336638 0 0) 0x7f00040cd030 con 0x7f000415b600 2024-03-18T19:08:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.216+0000 7f0003fff700 1 --2- 172.21.15.1:0/2620310188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00040cc770 0x7f00040cf5b0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7efff0009920 tx=0x7efff0009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d31ec539c8ba36 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.216+0000 7f00037fe700 1 -- 172.21.15.1:0/2620310188 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff0011830 con 0x7f00040cc770 2024-03-18T19:08:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.216+0000 7f00037fe700 1 -- 172.21.15.1:0/2620310188 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff0011990 con 0x7f00040cc770 2024-03-18T19:08:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.216+0000 7f00037fe700 1 -- 172.21.15.1:0/2620310188 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff0011d20 con 0x7f00040cc770 2024-03-18T19:08:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.217+0000 7f000aad0700 1 -- 172.21.15.1:0/2620310188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00040cc770 msgr2=0x7f00040cf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.217+0000 7f000aad0700 1 --2- 172.21.15.1:0/2620310188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00040cc770 0x7f00040cf5b0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7efff0009920 tx=0x7efff0009c30 comp rx=0 tx=0).stop 2024-03-18T19:08:01.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.217+0000 7f000aad0700 1 -- 172.21.15.1:0/2620310188 shutdown_connections 2024-03-18T19:08:01.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.217+0000 7f000aad0700 1 --2- 172.21.15.1:0/2620310188 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000406f370 0x7f000406f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.217+0000 7f000aad0700 1 --2- 172.21.15.1:0/2620310188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00040cc770 0x7f00040cf5b0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.217+0000 7f000aad0700 1 -- 172.21.15.1:0/2620310188 >> 172.21.15.1:0/2620310188 conn(0x7f00040b2bb0 msgr2=0x7f00040b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:01.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.217+0000 7f000aad0700 1 -- 172.21.15.1:0/2620310188 shutdown_connections 2024-03-18T19:08:01.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.217+0000 7f000aad0700 1 -- 172.21.15.1:0/2620310188 wait complete. 2024-03-18T19:08:01.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.218+0000 7f000aad0700 1 Processor -- start 2024-03-18T19:08:01.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.218+0000 7f000aad0700 1 -- start start 2024-03-18T19:08:01.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.219+0000 7f000aad0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000406f370 0x7f00040b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:01.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.219+0000 7f000aad0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00040cc770 0x7f00040cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:01.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.219+0000 7f000aad0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f000415b600 0x7f0004158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:01.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.219+0000 7f000aad0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00040ca0d0 con 0x7f00040cc770 2024-03-18T19:08:01.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.219+0000 7f000aad0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00040c91e0 con 0x7f000415b600 2024-03-18T19:08:01.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.219+0000 7f000aad0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00040c94c0 con 0x7f000406f370 2024-03-18T19:08:01.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.219+0000 7f0003fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00040cc770 0x7f00040cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:01.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.219+0000 7f0003fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00040cc770 0x7f00040cd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46566/0 (socket says 172.21.15.1:46566) 2024-03-18T19:08:01.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.219+0000 7f0003fff700 1 -- 172.21.15.1:0/913582610 learned_addr learned my addr 172.21.15.1:0/913582610 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:01.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.219+0000 7f000886c700 1 --2- 172.21.15.1:0/913582610 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000406f370 0x7f00040b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.219+0000 7f000906d700 1 --2- 172.21.15.1:0/913582610 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f000415b600 0x7f0004158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.219+0000 7f0003fff700 1 -- 172.21.15.1:0/913582610 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000406f370 msgr2=0x7f00040b3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.219+0000 7f0003fff700 1 --2- 172.21.15.1:0/913582610 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000406f370 0x7f00040b3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.219+0000 7f0003fff700 1 -- 172.21.15.1:0/913582610 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f000415b600 msgr2=0x7f0004158d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.219+0000 7f0003fff700 1 --2- 172.21.15.1:0/913582610 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f000415b600 0x7f0004158d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.219+0000 7f0003fff700 1 -- 172.21.15.1:0/913582610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff0009550 con 0x7f00040cc770 2024-03-18T19:08:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.220+0000 7f0003fff700 1 --2- 172.21.15.1:0/913582610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00040cc770 0x7f00040cd6f0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7efff0009c70 tx=0x7efff0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.220+0000 7f00017fa700 1 -- 172.21.15.1:0/913582610 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff0004450 con 0x7f00040cc770 2024-03-18T19:08:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.220+0000 7f00017fa700 1 -- 172.21.15.1:0/913582610 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff001e070 con 0x7f00040cc770 2024-03-18T19:08:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.220+0000 7f00017fa700 1 -- 172.21.15.1:0/913582610 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff00038e0 con 0x7f00040cc770 2024-03-18T19:08:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.220+0000 7f000aad0700 1 -- 172.21.15.1:0/913582610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00040c97a0 con 0x7f00040cc770 2024-03-18T19:08:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.220+0000 7f000aad0700 1 -- 172.21.15.1:0/913582610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00040c9d60 con 0x7f00040cc770 2024-03-18T19:08:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.221+0000 7f000aad0700 1 -- 172.21.15.1:0/913582610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00040ca240 con 0x7f00040cc770 2024-03-18T19:08:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.221+0000 7f00017fa700 1 -- 172.21.15.1:0/913582610 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 56) v1 ==== 89353+0+0 (secure 0 0 0) 0x7efff0003a40 con 0x7f00040cc770 2024-03-18T19:08:01.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.222+0000 7f00017fa700 1 --2- 172.21.15.1:0/913582610 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effe806aed0 0x7effe806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.222+0000 7f000886c700 1 -- 172.21.15.1:0/913582610 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effe806aed0 msgr2=0x7effe806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.222+0000 7f00017fa700 1 -- 172.21.15.1:0/913582610 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6489+0+0 (secure 0 0 0) 0x7efff008a6a0 con 0x7f00040cc770 2024-03-18T19:08:01.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.222+0000 7f000886c700 1 --2- 172.21.15.1:0/913582610 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effe806aed0 0x7effe806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:08:01.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.226+0000 7f00017fa700 1 -- 172.21.15.1:0/913582610 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efff0055370 con 0x7f00040cc770 2024-03-18T19:08:01.289 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:01.293+0000 7f30b35e4240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:08:01.409 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:01.412+0000 7f1b3c187240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:08:01.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.423+0000 7f000886c700 1 -- 172.21.15.1:0/913582610 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effe806aed0 msgr2=0x7effe806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:01.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.423+0000 7f000886c700 1 --2- 172.21.15.1:0/913582610 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effe806aed0 0x7effe806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:08:01.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.428+0000 7f000aad0700 1 -- 172.21.15.1:0/913582610 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f00040676a0 con 0x7f00040cc770 2024-03-18T19:08:01.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.429+0000 7f00017fa700 1 -- 172.21.15.1:0/913582610 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v120) v1 ==== 104+0+11 (secure 0 0 0) 0x7efff00045b0 con 0x7f00040cc770 2024-03-18T19:08:01.426 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T19:08:01.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.431+0000 7f000aad0700 1 -- 172.21.15.1:0/913582610 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effe806aed0 msgr2=0x7effe806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:08:01.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.431+0000 7f000aad0700 1 --2- 172.21.15.1:0/913582610 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effe806aed0 0x7effe806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.431+0000 7f000aad0700 1 -- 172.21.15.1:0/913582610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00040cc770 msgr2=0x7f00040cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:01.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.431+0000 7f000aad0700 1 --2- 172.21.15.1:0/913582610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00040cc770 0x7f00040cd6f0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7efff0009c70 tx=0x7efff0004750 comp rx=0 tx=0).stop 2024-03-18T19:08:01.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.431+0000 7f000aad0700 1 -- 172.21.15.1:0/913582610 shutdown_connections 2024-03-18T19:08:01.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.431+0000 7f000aad0700 1 --2- 172.21.15.1:0/913582610 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7effe806aed0 0x7effe806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.431+0000 7f000aad0700 1 --2- 172.21.15.1:0/913582610 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000406f370 0x7f00040b3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.431+0000 7f000aad0700 1 --2- 172.21.15.1:0/913582610 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f000415b600 0x7f0004158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.431+0000 7f000aad0700 1 --2- 172.21.15.1:0/913582610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00040cc770 0x7f00040cd6f0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.431+0000 7f000aad0700 1 -- 172.21.15.1:0/913582610 >> 172.21.15.1:0/913582610 conn(0x7f00040b2bb0 msgr2=0x7f00040b2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:01.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.431+0000 7f000aad0700 1 -- 172.21.15.1:0/913582610 shutdown_connections 2024-03-18T19:08:01.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.431+0000 7f000aad0700 1 -- 172.21.15.1:0/913582610 wait complete. 2024-03-18T19:08:01.440 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T19:08:01.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.574+0000 7f55ef5ae700 1 Processor -- start 2024-03-18T19:08:01.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.574+0000 7f55ef5ae700 1 -- start start 2024-03-18T19:08:01.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.575+0000 7f55ef5ae700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f55e80cd430 0x7f55e80b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:01.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.575+0000 7f55ef5ae700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f55e80c87d0 0x7f55e80cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:01.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.575+0000 7f55ef5ae700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55e80d01f0 con 0x7f55e815b6c0 2024-03-18T19:08:01.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.575+0000 7f55ef5ae700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55e80ccfa0 con 0x7f55e80c87d0 2024-03-18T19:08:01.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.575+0000 7f55ef5ae700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55e80d1db0 con 0x7f55e80cd430 2024-03-18T19:08:01.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.575+0000 7f55edb4b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f55e80c87d0 0x7f55e80cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:01.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.575+0000 7f55ed34a700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f55e815b6c0 0x7f55e80c80c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34974/0 (socket says 172.21.15.1:34974) 2024-03-18T19:08:01.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.575+0000 7f55ed34a700 1 -- 172.21.15.1:0/2469460028 learned_addr learned my addr 172.21.15.1:0/2469460028 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:01.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.575+0000 7f55edb4b700 1 --2- 172.21.15.1:0/2469460028 >> v2:172.21.15.1:3300/0 conn(0x7f55e80c87d0 0x7f55e80cd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46574/0 (socket says 172.21.15.1:46574) 2024-03-18T19:08:01.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.575+0000 7f55ecb49700 1 --2- 172.21.15.1:0/2469460028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f55e80cd430 0x7f55e80b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:01.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.575+0000 7f55edb4b700 1 -- 172.21.15.1:0/2469460028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f55e80cd430 msgr2=0x7f55e80b2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:01.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.575+0000 7f55edb4b700 1 --2- 172.21.15.1:0/2469460028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f55e80cd430 0x7f55e80b2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.575+0000 7f55edb4b700 1 -- 172.21.15.1:0/2469460028 >> v1:172.21.15.1:6789/0 conn(0x7f55e815b6c0 legacy=0x7f55e80c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:01.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.576+0000 7f55dffff700 1 -- 172.21.15.1:0/2469460028 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137710316 0 0) 0x7f55e80d01f0 con 0x7f55e815b6c0 2024-03-18T19:08:01.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.576+0000 7f55edb4b700 1 -- 172.21.15.1:0/2469460028 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e80cff10 con 0x7f55e80c87d0 2024-03-18T19:08:01.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.576+0000 7f55ecb49700 1 --2- 172.21.15.1:0/2469460028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f55e80cd430 0x7f55e80b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:01.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.576+0000 7f55edb4b700 1 --2- 172.21.15.1:0/2469460028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55e80c87d0 0x7f55e80cd9e0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f55e0009dc0 tx=0x7f55e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c835ad2d57495ab server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:01.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.576+0000 7f55dffff700 1 -- 172.21.15.1:0/2469460028 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e000e070 con 0x7f55e80c87d0 2024-03-18T19:08:01.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.576+0000 7f55dffff700 1 -- 172.21.15.1:0/2469460028 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e000bb90 con 0x7f55e80c87d0 2024-03-18T19:08:01.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.576+0000 7f55dffff700 1 -- 172.21.15.1:0/2469460028 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e0012430 con 0x7f55e80c87d0 2024-03-18T19:08:01.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.576+0000 7f55ef5ae700 1 -- 172.21.15.1:0/2469460028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55e80c87d0 msgr2=0x7f55e80cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:01.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.576+0000 7f55ef5ae700 1 --2- 172.21.15.1:0/2469460028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55e80c87d0 0x7f55e80cd9e0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f55e0009dc0 tx=0x7f55e000b330 comp rx=0 tx=0).stop 2024-03-18T19:08:01.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.577+0000 7f55ef5ae700 1 -- 172.21.15.1:0/2469460028 shutdown_connections 2024-03-18T19:08:01.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.577+0000 7f55ef5ae700 1 --2- 172.21.15.1:0/2469460028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55e80c87d0 0x7f55e80cd9e0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.577+0000 7f55ef5ae700 1 --2- 172.21.15.1:0/2469460028 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f55e80cd430 0x7f55e80b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.577+0000 7f55ef5ae700 1 -- 172.21.15.1:0/2469460028 >> 172.21.15.1:0/2469460028 conn(0x7f55e80d0ff0 msgr2=0x7f55e80d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:01.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.577+0000 7f55ef5ae700 1 -- 172.21.15.1:0/2469460028 shutdown_connections 2024-03-18T19:08:01.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.577+0000 7f55ef5ae700 1 -- 172.21.15.1:0/2469460028 wait complete. 2024-03-18T19:08:01.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.578+0000 7f55ef5ae700 1 Processor -- start 2024-03-18T19:08:01.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.578+0000 7f55ef5ae700 1 -- start start 2024-03-18T19:08:01.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ef5ae700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f55e80c87d0 0x7f55e806cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:01.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ef5ae700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55e80cd430 0x7f55e80c5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:01.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ef5ae700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f55e815b6c0 0x7f55e81555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:01.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ef5ae700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e806bcd0 con 0x7f55e80cd430 2024-03-18T19:08:01.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ef5ae700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e806bfb0 con 0x7f55e80c87d0 2024-03-18T19:08:01.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ef5ae700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55e806c290 con 0x7f55e815b6c0 2024-03-18T19:08:01.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ecb49700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55e80cd430 0x7f55e80c5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:01.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ecb49700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55e80cd430 0x7f55e80c5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46580/0 (socket says 172.21.15.1:46580) 2024-03-18T19:08:01.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ecb49700 1 -- 172.21.15.1:0/3277535221 learned_addr learned my addr 172.21.15.1:0/3277535221 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:01.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55edb4b700 1 --2- 172.21.15.1:0/3277535221 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f55e815b6c0 0x7f55e81555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:01.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ed34a700 1 --2- 172.21.15.1:0/3277535221 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f55e80c87d0 0x7f55e806cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:01.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ecb49700 1 -- 172.21.15.1:0/3277535221 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f55e815b6c0 msgr2=0x7f55e81555d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:01.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ecb49700 1 --2- 172.21.15.1:0/3277535221 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f55e815b6c0 0x7f55e81555d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ecb49700 1 -- 172.21.15.1:0/3277535221 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f55e80c87d0 msgr2=0x7f55e806cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:01.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ecb49700 1 --2- 172.21.15.1:0/3277535221 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f55e80c87d0 0x7f55e806cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ecb49700 1 -- 172.21.15.1:0/3277535221 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e0009a70 con 0x7f55e80cd430 2024-03-18T19:08:01.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ed34a700 1 --2- 172.21.15.1:0/3277535221 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f55e80c87d0 0x7f55e806cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:01.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55edb4b700 1 --2- 172.21.15.1:0/3277535221 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f55e815b6c0 0x7f55e81555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:01.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ecb49700 1 --2- 172.21.15.1:0/3277535221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55e80cd430 0x7f55e80c5f60 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f55d800e810 tx=0x7f55d800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:01.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ddffb700 1 -- 172.21.15.1:0/3277535221 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d800c020 con 0x7f55e80cd430 2024-03-18T19:08:01.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ddffb700 1 -- 172.21.15.1:0/3277535221 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d8004250 con 0x7f55e80cd430 2024-03-18T19:08:01.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ddffb700 1 -- 172.21.15.1:0/3277535221 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d80036a0 con 0x7f55e80cd430 2024-03-18T19:08:01.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.579+0000 7f55ef5ae700 1 -- 172.21.15.1:0/3277535221 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55e806c570 con 0x7f55e80cd430 2024-03-18T19:08:01.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.580+0000 7f55ef5ae700 1 -- 172.21.15.1:0/3277535221 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55e80c64a0 con 0x7f55e80cd430 2024-03-18T19:08:01.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.581+0000 7f55ef5ae700 1 -- 172.21.15.1:0/3277535221 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55e806ca90 con 0x7f55e80cd430 2024-03-18T19:08:01.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.581+0000 7f55ddffb700 1 -- 172.21.15.1:0/3277535221 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 56) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f55d8003800 con 0x7f55e80cd430 2024-03-18T19:08:01.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.581+0000 7f55ddffb700 1 --2- 172.21.15.1:0/3277535221 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f55cc06aed0 0x7f55cc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:01.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.582+0000 7f55ed34a700 1 -- 172.21.15.1:0/3277535221 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f55cc06aed0 msgr2=0x7f55cc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:01.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.582+0000 7f55ddffb700 1 -- 172.21.15.1:0/3277535221 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f55d8089660 con 0x7f55e80cd430 2024-03-18T19:08:01.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.582+0000 7f55ed34a700 1 --2- 172.21.15.1:0/3277535221 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f55cc06aed0 0x7f55cc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:08:01.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.586+0000 7f55ddffb700 1 -- 172.21.15.1:0/3277535221 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f55e806ca90 con 0x7f55e80cd430 2024-03-18T19:08:01.661 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:01.665+0000 7f30b35e4240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:08:01.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.783+0000 7f55ed34a700 1 -- 172.21.15.1:0/3277535221 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f55cc06aed0 msgr2=0x7f55cc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:01.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.783+0000 7f55ed34a700 1 --2- 172.21.15.1:0/3277535221 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f55cc06aed0 0x7f55cc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:08:01.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.789+0000 7f55ef5ae700 1 -- 172.21.15.1:0/3277535221 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f55e806ca90 con 0x7f55e80cd430 2024-03-18T19:08:01.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.790+0000 7f55ddffb700 1 -- 172.21.15.1:0/3277535221 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v120) v1 ==== 100+0+11 (secure 0 0 0) 0x7f55e806ca90 con 0x7f55e80cd430 2024-03-18T19:08:01.787 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T19:08:01.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.792+0000 7f55ef5ae700 1 -- 172.21.15.1:0/3277535221 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f55cc06aed0 msgr2=0x7f55cc06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:08:01.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.792+0000 7f55ef5ae700 1 --2- 172.21.15.1:0/3277535221 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f55cc06aed0 0x7f55cc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.792+0000 7f55ef5ae700 1 -- 172.21.15.1:0/3277535221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55e80cd430 msgr2=0x7f55e80c5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:01.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.792+0000 7f55ef5ae700 1 --2- 172.21.15.1:0/3277535221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55e80cd430 0x7f55e80c5f60 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f55d800e810 tx=0x7f55d800ebd0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.792+0000 7f55ef5ae700 1 -- 172.21.15.1:0/3277535221 shutdown_connections 2024-03-18T19:08:01.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.792+0000 7f55ef5ae700 1 --2- 172.21.15.1:0/3277535221 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f55cc06aed0 0x7f55cc06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.792+0000 7f55ef5ae700 1 --2- 172.21.15.1:0/3277535221 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f55e815b6c0 0x7f55e81555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.792+0000 7f55ef5ae700 1 --2- 172.21.15.1:0/3277535221 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f55e80c87d0 0x7f55e806cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.792+0000 7f55ef5ae700 1 --2- 172.21.15.1:0/3277535221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55e80cd430 0x7f55e80c5f60 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.792+0000 7f55ef5ae700 1 -- 172.21.15.1:0/3277535221 >> 172.21.15.1:0/3277535221 conn(0x7f55e80d0ff0 msgr2=0x7f55e80c8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:01.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.792+0000 7f55ef5ae700 1 -- 172.21.15.1:0/3277535221 shutdown_connections 2024-03-18T19:08:01.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.792+0000 7f55ef5ae700 1 -- 172.21.15.1:0/3277535221 wait complete. 2024-03-18T19:08:01.801 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T19:08:01.811 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:01.814+0000 7f1b3c187240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:08:01.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.934+0000 7f531e99d700 1 Processor -- start 2024-03-18T19:08:01.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.934+0000 7f531e99d700 1 -- start start 2024-03-18T19:08:01.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.935+0000 7f531e99d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f531806ae60 0x7f531806ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:01.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.935+0000 7f531e99d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f531806ef70 0x7f5318071520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:01.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.935+0000 7f531e99d700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531806e690 con 0x7f531815b360 2024-03-18T19:08:01.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.935+0000 7f531e99d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f531806e270 con 0x7f531806ae60 2024-03-18T19:08:01.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.935+0000 7f531e99d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531806f400 con 0x7f531806ef70 2024-03-18T19:08:01.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.935+0000 7f531d19a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f531806ae60 0x7f531806ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:01.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.935+0000 7f531d19a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f531806ae60 0x7f531806ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46594/0 (socket says 172.21.15.1:46594) 2024-03-18T19:08:01.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.935+0000 7f531d19a700 1 -- 172.21.15.1:0/2422208407 learned_addr learned my addr 172.21.15.1:0/2422208407 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:01.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.936+0000 7f531e19c700 1 --2- 172.21.15.1:0/2422208407 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f531806ef70 0x7f5318071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:01.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.936+0000 7f531d19a700 1 -- 172.21.15.1:0/2422208407 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f531806ef70 msgr2=0x7f5318071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:01.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.936+0000 7f531d19a700 1 --2- 172.21.15.1:0/2422208407 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f531806ef70 0x7f5318071520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.936+0000 7f531d19a700 1 -- 172.21.15.1:0/2422208407 >> v1:172.21.15.165:6789/0 conn(0x7f531815b360 legacy=0x7f531806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:01.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.936+0000 7f531d19a700 1 -- 172.21.15.1:0/2422208407 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531806d680 con 0x7f531806ae60 2024-03-18T19:08:01.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.936+0000 7f531d19a700 1 --2- 172.21.15.1:0/2422208407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f531806ae60 0x7f531806ea30 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f53080098f0 tx=0x7f5308009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e6ccc92d835e7a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:01.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.936+0000 7f531c999700 1 -- 172.21.15.1:0/2422208407 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5308011830 con 0x7f531806ae60 2024-03-18T19:08:01.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.936+0000 7f531c999700 1 -- 172.21.15.1:0/2422208407 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5308011990 con 0x7f531806ae60 2024-03-18T19:08:01.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.936+0000 7f531c999700 1 -- 172.21.15.1:0/2422208407 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5308011d20 con 0x7f531806ae60 2024-03-18T19:08:01.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.937+0000 7f531e99d700 1 -- 172.21.15.1:0/2422208407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f531806ae60 msgr2=0x7f531806ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:01.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.937+0000 7f531e99d700 1 --2- 172.21.15.1:0/2422208407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f531806ae60 0x7f531806ea30 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f53080098f0 tx=0x7f5308009c00 comp rx=0 tx=0).stop 2024-03-18T19:08:01.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.937+0000 7f531e99d700 1 -- 172.21.15.1:0/2422208407 shutdown_connections 2024-03-18T19:08:01.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.937+0000 7f531e99d700 1 --2- 172.21.15.1:0/2422208407 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f531806ef70 0x7f5318071520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.937+0000 7f531e99d700 1 --2- 172.21.15.1:0/2422208407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f531806ae60 0x7f531806ea30 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.937+0000 7f531e99d700 1 -- 172.21.15.1:0/2422208407 >> 172.21.15.1:0/2422208407 conn(0x7f53180cca30 msgr2=0x7f53180cce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:01.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.937+0000 7f531e99d700 1 -- 172.21.15.1:0/2422208407 shutdown_connections 2024-03-18T19:08:01.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.937+0000 7f531e99d700 1 -- 172.21.15.1:0/2422208407 wait complete. 2024-03-18T19:08:01.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.938+0000 7f531e99d700 1 Processor -- start 2024-03-18T19:08:01.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.938+0000 7f531e99d700 1 -- start start 2024-03-18T19:08:01.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.939+0000 7f531e99d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f531806ae60 0x7f5318153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:01.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.939+0000 7f531e99d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f531806ef70 0x7f5318154730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:01.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.939+0000 7f531e99d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f531815b360 0x7f5318154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:01.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.939+0000 7f531e99d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53181f1390 con 0x7f531815b360 2024-03-18T19:08:01.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.939+0000 7f531e99d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5318155e50 con 0x7f531806ae60 2024-03-18T19:08:01.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.939+0000 7f531e99d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53181552d0 con 0x7f531806ef70 2024-03-18T19:08:01.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.940+0000 7f531e19c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f531815b360 0x7f5318154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:01.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.940+0000 7f531e19c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f531815b360 0x7f5318154c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46602/0 (socket says 172.21.15.1:46602) 2024-03-18T19:08:01.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.940+0000 7f531e19c700 1 -- 172.21.15.1:0/3184921481 learned_addr learned my addr 172.21.15.1:0/3184921481 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:01.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.940+0000 7f531d99b700 1 --2- 172.21.15.1:0/3184921481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f531806ae60 0x7f5318153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:01.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.940+0000 7f531d19a700 1 --2- 172.21.15.1:0/3184921481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f531806ef70 0x7f5318154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:01.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.940+0000 7f531e19c700 1 -- 172.21.15.1:0/3184921481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f531806ef70 msgr2=0x7f5318154730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:01.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.940+0000 7f531e19c700 1 --2- 172.21.15.1:0/3184921481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f531806ef70 0x7f5318154730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.940+0000 7f531e19c700 1 -- 172.21.15.1:0/3184921481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f531806ae60 msgr2=0x7f5318153350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:01.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.940+0000 7f531e19c700 1 --2- 172.21.15.1:0/3184921481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f531806ae60 0x7f5318153350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:01.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.940+0000 7f531e19c700 1 -- 172.21.15.1:0/3184921481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5308009550 con 0x7f531815b360 2024-03-18T19:08:01.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.940+0000 7f531e19c700 1 --2- 172.21.15.1:0/3184921481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f531815b360 0x7f5318154c70 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f531000efc0 tx=0x7f531000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:01.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.940+0000 7f530e7fc700 1 -- 172.21.15.1:0/3184921481 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5310009ea0 con 0x7f531815b360 2024-03-18T19:08:01.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.940+0000 7f530e7fc700 1 -- 172.21.15.1:0/3184921481 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f531000f040 con 0x7f531815b360 2024-03-18T19:08:01.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.940+0000 7f531e99d700 1 -- 172.21.15.1:0/3184921481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53181555b0 con 0x7f531815b360 2024-03-18T19:08:01.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.940+0000 7f530e7fc700 1 -- 172.21.15.1:0/3184921481 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531000ced0 con 0x7f531815b360 2024-03-18T19:08:01.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.940+0000 7f531e99d700 1 -- 172.21.15.1:0/3184921481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5318155b70 con 0x7f531815b360 2024-03-18T19:08:01.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.942+0000 7f531e99d700 1 -- 172.21.15.1:0/3184921481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5318071100 con 0x7f531815b360 2024-03-18T19:08:01.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.942+0000 7f530e7fc700 1 -- 172.21.15.1:0/3184921481 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 56) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f5310014480 con 0x7f531815b360 2024-03-18T19:08:01.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.942+0000 7f530e7fc700 1 --2- 172.21.15.1:0/3184921481 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f52fc06aed0 0x7f52fc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:01.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.942+0000 7f531d99b700 1 -- 172.21.15.1:0/3184921481 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f52fc06aed0 msgr2=0x7f52fc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:01.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.942+0000 7f530e7fc700 1 -- 172.21.15.1:0/3184921481 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f531008a370 con 0x7f531815b360 2024-03-18T19:08:01.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.942+0000 7f531d99b700 1 --2- 172.21.15.1:0/3184921481 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f52fc06aed0 0x7f52fc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:08:01.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:01.947+0000 7f530e7fc700 1 -- 172.21.15.1:0/3184921481 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5310054dd0 con 0x7f531815b360 2024-03-18T19:08:02.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.143+0000 7f531d99b700 1 -- 172.21.15.1:0/3184921481 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f52fc06aed0 msgr2=0x7f52fc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:02.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.143+0000 7f531d99b700 1 --2- 172.21.15.1:0/3184921481 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f52fc06aed0 0x7f52fc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:08:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.185+0000 7f531e99d700 1 -- 172.21.15.1:0/3184921481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f53180747a0 con 0x7f531815b360 2024-03-18T19:08:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.186+0000 7f530e7fc700 1 -- 172.21.15.1:0/3184921481 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v120) v1 ==== 80+0+450 (secure 0 0 0) 0x7f53100583f0 con 0x7f531815b360 2024-03-18T19:08:02.183 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1072638685 2024-03-19T19:06:57.097725+0000 2024-03-18T19:08:02.183 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/67600845 2024-03-19T19:06:57.097725+0000 2024-03-18T19:08:02.183 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6833/76998 2024-03-19T19:06:57.097725+0000 2024-03-18T19:08:02.183 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6832/76998 2024-03-19T19:06:57.097725+0000 2024-03-18T19:08:02.183 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1805165487 2024-03-19T19:06:57.097725+0000 2024-03-18T19:08:02.183 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1343251370 2024-03-19T19:06:57.097725+0000 2024-03-18T19:08:02.183 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3890713729 2024-03-19T19:06:57.097725+0000 2024-03-18T19:08:02.183 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2823326236 2024-03-19T19:06:57.097725+0000 2024-03-18T19:08:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.188+0000 7f531e99d700 1 -- 172.21.15.1:0/3184921481 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f52fc06aed0 msgr2=0x7f52fc06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:08:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.188+0000 7f531e99d700 1 --2- 172.21.15.1:0/3184921481 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f52fc06aed0 0x7f52fc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.188+0000 7f531e99d700 1 -- 172.21.15.1:0/3184921481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f531815b360 msgr2=0x7f5318154c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.188+0000 7f531e99d700 1 --2- 172.21.15.1:0/3184921481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f531815b360 0x7f5318154c70 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f531000efc0 tx=0x7f531000c3a0 comp rx=0 tx=0).stop 2024-03-18T19:08:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.188+0000 7f531e99d700 1 -- 172.21.15.1:0/3184921481 shutdown_connections 2024-03-18T19:08:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.188+0000 7f531e99d700 1 --2- 172.21.15.1:0/3184921481 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f52fc06aed0 0x7f52fc06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.188+0000 7f531e99d700 1 --2- 172.21.15.1:0/3184921481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f531815b360 0x7f5318154c70 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.188+0000 7f531e99d700 1 --2- 172.21.15.1:0/3184921481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f531806ae60 0x7f5318153350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.188+0000 7f531e99d700 1 --2- 172.21.15.1:0/3184921481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f531806ef70 0x7f5318154730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.188+0000 7f531e99d700 1 -- 172.21.15.1:0/3184921481 >> 172.21.15.1:0/3184921481 conn(0x7f53180cca30 msgr2=0x7f53180cbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.188+0000 7f531e99d700 1 -- 172.21.15.1:0/3184921481 shutdown_connections 2024-03-18T19:08:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.188+0000 7f531e99d700 1 -- 172.21.15.1:0/3184921481 wait complete. 2024-03-18T19:08:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:listed 8 entries 2024-03-18T19:08:02.197 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-18T19:08:02.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:02.197 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T19:08:02.249 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:02.252+0000 7f30b35e4240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:08:02.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[120695]: fuse finished with error 0 and tester_r 0 2024-03-18T19:08:02.274 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T19:08:02.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:02.275 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T19:08:02.320 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:02.324+0000 7f30b35e4240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:08:02.414 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:08:02.414 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:08:02.432 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:08:02.432 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T19:08:02.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:02.432 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-18T19:08:02.449 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:02.453+0000 7f30b35e4240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:08:02.460 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:02.464+0000 7f1b3c187240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:08:02.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:08:02.466 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-18T19:08:02.466 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:08:02.466 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:08:02.485 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:08:02.485 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T19:08:02.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:02.485 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-18T19:08:02.538 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:02.541+0000 7f1b3c187240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:08:02.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:08:02.542 INFO:teuthology.orchestra.run.smithi001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-18T19:08:02.542 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:08:02.543 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:08:02.561 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:08:02.561 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T19:08:02.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:02.561 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-18T19:08:02.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:08:02.618 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-18T19:08:02.619 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:08:02.633 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:02.637+0000 7f30b35e4240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:08:02.674 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:02.678+0000 7f1b3c187240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:08:02.705 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:02.709+0000 7f30b35e4240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:08:02.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.748+0000 7f302b24a700 1 Processor -- start 2024-03-18T19:08:02.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.748+0000 7f302b24a700 1 -- start start 2024-03-18T19:08:02.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.749+0000 7f302b24a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f302415d6b0 0x7f30240b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:02.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.749+0000 7f302b24a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f30240ad790 0x7f30240b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:02.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.749+0000 7f302b24a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30240acfb0 con 0x7f302406a750 2024-03-18T19:08:02.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.749+0000 7f302b24a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30240af7a0 con 0x7f30240ad790 2024-03-18T19:08:02.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.749+0000 7f302b24a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30240b50a0 con 0x7f302415d6b0 2024-03-18T19:08:02.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.749+0000 7f3028fe6700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f302406a750 0x7f302406ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34982/0 (socket says 172.21.15.1:34982) 2024-03-18T19:08:02.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.749+0000 7f3028fe6700 1 -- 172.21.15.1:0/3578760800 learned_addr learned my addr 172.21.15.1:0/3578760800 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:02.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.749+0000 7f30297e7700 1 --2- 172.21.15.1:0/3578760800 >> v2:172.21.15.165:3300/0 conn(0x7f30240ad790 0x7f30240b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:02.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.749+0000 7f3023fff700 1 --2- 172.21.15.1:0/3578760800 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f302415d6b0 0x7f30240b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:02.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.750+0000 7f30237fe700 1 -- 172.21.15.1:0/3578760800 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81736295 0 0) 0x7f30240acfb0 con 0x7f302406a750 2024-03-18T19:08:02.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.750+0000 7f30237fe700 1 -- 172.21.15.1:0/3578760800 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30140035b0 con 0x7f302406a750 2024-03-18T19:08:02.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.750+0000 7f30297e7700 1 -- 172.21.15.1:0/3578760800 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f302415d6b0 msgr2=0x7f30240b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:02.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.750+0000 7f30297e7700 1 --2- 172.21.15.1:0/3578760800 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f302415d6b0 0x7f30240b2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:02.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.750+0000 7f30297e7700 1 -- 172.21.15.1:0/3578760800 >> v1:172.21.15.1:6789/0 conn(0x7f302406a750 legacy=0x7f302406ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:02.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.750+0000 7f30297e7700 1 -- 172.21.15.1:0/3578760800 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30240af1e0 con 0x7f30240ad790 2024-03-18T19:08:02.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.750+0000 7f3023fff700 1 --2- 172.21.15.1:0/3578760800 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f302415d6b0 0x7f30240b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:08:02.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.750+0000 7f30297e7700 1 --2- 172.21.15.1:0/3578760800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30240ad790 0x7f30240b32e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f301c009d90 tx=0x7f301c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a8641609e454ef7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:02.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.750+0000 7f30237fe700 1 -- 172.21.15.1:0/3578760800 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c00e070 con 0x7f30240ad790 2024-03-18T19:08:02.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.750+0000 7f30237fe700 1 -- 172.21.15.1:0/3578760800 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c00bb20 con 0x7f30240ad790 2024-03-18T19:08:02.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.750+0000 7f30237fe700 1 -- 172.21.15.1:0/3578760800 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c012430 con 0x7f30240ad790 2024-03-18T19:08:02.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.751+0000 7f302b24a700 1 -- 172.21.15.1:0/3578760800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30240ad790 msgr2=0x7f30240b32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:02.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.751+0000 7f302b24a700 1 --2- 172.21.15.1:0/3578760800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30240ad790 0x7f30240b32e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f301c009d90 tx=0x7f301c00b330 comp rx=0 tx=0).stop 2024-03-18T19:08:02.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.751+0000 7f302b24a700 1 -- 172.21.15.1:0/3578760800 shutdown_connections 2024-03-18T19:08:02.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.751+0000 7f302b24a700 1 --2- 172.21.15.1:0/3578760800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30240ad790 0x7f30240b32e0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:02.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.751+0000 7f302b24a700 1 --2- 172.21.15.1:0/3578760800 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f302415d6b0 0x7f30240b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:02.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.751+0000 7f302b24a700 1 -- 172.21.15.1:0/3578760800 >> 172.21.15.1:0/3578760800 conn(0x7f302406b860 msgr2=0x7f302406bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:02.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.751+0000 7f302b24a700 1 -- 172.21.15.1:0/3578760800 shutdown_connections 2024-03-18T19:08:02.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.751+0000 7f302b24a700 1 -- 172.21.15.1:0/3578760800 wait complete. 2024-03-18T19:08:02.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.752+0000 7f302b24a700 1 Processor -- start 2024-03-18T19:08:02.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.752+0000 7f302b24a700 1 -- start start 2024-03-18T19:08:02.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.753+0000 7f302b24a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f302406a750 0x7f302415b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:02.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.753+0000 7f302b24a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30240ad790 0x7f30240cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:02.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.753+0000 7f302b24a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f302415d6b0 0x7f30241468d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:02.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.754+0000 7f302b24a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30241451d0 con 0x7f302415d6b0 2024-03-18T19:08:02.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.754+0000 7f302b24a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3024146440 con 0x7f30240ad790 2024-03-18T19:08:02.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.754+0000 7f302b24a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3024145830 con 0x7f302406a750 2024-03-18T19:08:02.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.754+0000 7f30297e7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f302415d6b0 0x7f30241468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:02.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.754+0000 7f30297e7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f302415d6b0 0x7f30241468d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46616/0 (socket says 172.21.15.1:46616) 2024-03-18T19:08:02.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.754+0000 7f30297e7700 1 -- 172.21.15.1:0/379380782 learned_addr learned my addr 172.21.15.1:0/379380782 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:02.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.754+0000 7f3028fe6700 1 --2- 172.21.15.1:0/379380782 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f302406a750 0x7f302415b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:02.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.754+0000 7f3023fff700 1 --2- 172.21.15.1:0/379380782 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30240ad790 0x7f30240cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:02.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.754+0000 7f30297e7700 1 -- 172.21.15.1:0/379380782 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f302406a750 msgr2=0x7f302415b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:02.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.754+0000 7f30297e7700 1 --2- 172.21.15.1:0/379380782 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f302406a750 0x7f302415b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:02.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.754+0000 7f30297e7700 1 -- 172.21.15.1:0/379380782 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30240ad790 msgr2=0x7f30240cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:02.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.754+0000 7f30297e7700 1 --2- 172.21.15.1:0/379380782 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30240ad790 0x7f30240cd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:02.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.754+0000 7f30297e7700 1 -- 172.21.15.1:0/379380782 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30100094a0 con 0x7f302415d6b0 2024-03-18T19:08:02.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.754+0000 7f30297e7700 1 --2- 172.21.15.1:0/379380782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f302415d6b0 0x7f30241468d0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f301c014040 tx=0x7f301c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:02.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.754+0000 7f30217fa700 1 -- 172.21.15.1:0/379380782 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c0129f0 con 0x7f302415d6b0 2024-03-18T19:08:02.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.754+0000 7f30217fa700 1 -- 172.21.15.1:0/379380782 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c020070 con 0x7f302415d6b0 2024-03-18T19:08:02.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.754+0000 7f302b24a700 1 -- 172.21.15.1:0/379380782 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f301c009a70 con 0x7f302415d6b0 2024-03-18T19:08:02.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.754+0000 7f30217fa700 1 -- 172.21.15.1:0/379380782 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c012db0 con 0x7f302415d6b0 2024-03-18T19:08:02.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.754+0000 7f302b24a700 1 -- 172.21.15.1:0/379380782 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3024145df0 con 0x7f302415d6b0 2024-03-18T19:08:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.755+0000 7f302b24a700 1 -- 172.21.15.1:0/379380782 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30240676a0 con 0x7f302415d6b0 2024-03-18T19:08:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.756+0000 7f30217fa700 1 -- 172.21.15.1:0/379380782 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 56) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f301c004040 con 0x7f302415d6b0 2024-03-18T19:08:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.756+0000 7f30217fa700 1 --2- 172.21.15.1:0/379380782 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f300806aed0 0x7f300806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.756+0000 7f3028fe6700 1 -- 172.21.15.1:0/379380782 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f300806aed0 msgr2=0x7f300806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.756+0000 7f30217fa700 1 -- 172.21.15.1:0/379380782 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f301c022070 con 0x7f302415d6b0 2024-03-18T19:08:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.756+0000 7f3028fe6700 1 --2- 172.21.15.1:0/379380782 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f300806aed0 0x7f300806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:08:02.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.766+0000 7f30217fa700 1 -- 172.21.15.1:0/379380782 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f301c0569f0 con 0x7f302415d6b0 2024-03-18T19:08:02.783 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:02.787+0000 7f30b35e4240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:08:02.861 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:02.865+0000 7f30b35e4240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:08:02.867 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:02.870+0000 7f1b3c187240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:08:02.938 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:02.941+0000 7f30b35e4240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:08:02.944 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:02.947+0000 7f1b3c187240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:08:02.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.957+0000 7f3028fe6700 1 -- 172.21.15.1:0/379380782 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f300806aed0 msgr2=0x7f300806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:02.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:02.957+0000 7f3028fe6700 1 --2- 172.21.15.1:0/379380782 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f300806aed0 0x7f300806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:08:03.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.015+0000 7f302b24a700 1 -- 172.21.15.1:0/379380782 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3024146580 con 0x7f302415d6b0 2024-03-18T19:08:03.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.015+0000 7f30217fa700 1 -- 172.21.15.1:0/379380782 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4733 (secure 0 0 0) 0x7f301c05a010 con 0x7f302415d6b0 2024-03-18T19:08:03.015 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:03.015 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6431,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/983420410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":983420410},{"type":"v1","addr":"172.21.15.1:6836","nonce":983420410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6440,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1569519191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1569519191},{"type":"v1","addr":"172.21.15.165:6834","nonce":1569519191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1528229019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1528229019},{"type":"v1","addr":"172.21.15.165:6835","nonce":1528229019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:06:41.899614+0000","modified":"2024-03-18T19:06:44.904965+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"c","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2226098738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2226098738},{"type":"v1","addr":"172.21.15.1:6837","nonce":2226098738}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-18T19:08:03.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.017+0000 7f302b24a700 1 -- 172.21.15.1:0/379380782 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f300806aed0 msgr2=0x7f300806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:08:03.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.017+0000 7f302b24a700 1 --2- 172.21.15.1:0/379380782 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f300806aed0 0x7f300806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.017+0000 7f302b24a700 1 -- 172.21.15.1:0/379380782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f302415d6b0 msgr2=0x7f30241468d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:03.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.017+0000 7f302b24a700 1 --2- 172.21.15.1:0/379380782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f302415d6b0 0x7f30241468d0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f301c014040 tx=0x7f301c004ad0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.018+0000 7f302b24a700 1 -- 172.21.15.1:0/379380782 shutdown_connections 2024-03-18T19:08:03.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.018+0000 7f302b24a700 1 --2- 172.21.15.1:0/379380782 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f300806aed0 0x7f300806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.018+0000 7f302b24a700 1 --2- 172.21.15.1:0/379380782 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f302406a750 0x7f302415b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.018+0000 7f302b24a700 1 --2- 172.21.15.1:0/379380782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f302415d6b0 0x7f30241468d0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.018+0000 7f302b24a700 1 --2- 172.21.15.1:0/379380782 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30240ad790 0x7f30240cd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.018+0000 7f302b24a700 1 -- 172.21.15.1:0/379380782 >> 172.21.15.1:0/379380782 conn(0x7f302406b860 msgr2=0x7f30240b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:03.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.018+0000 7f302b24a700 1 -- 172.21.15.1:0/379380782 shutdown_connections 2024-03-18T19:08:03.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.018+0000 7f302b24a700 1 -- 172.21.15.1:0/379380782 wait complete. 2024-03-18T19:08:03.017 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 49 2024-03-18T19:08:03.027 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:08:03.029 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:03.032+0000 7f1b3c187240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:08:03.108 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:03.111+0000 7f30b35e4240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:08:03.115 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:03.118+0000 7f1b3c187240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:08:03.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.157+0000 7f8cb664f700 1 Processor -- start 2024-03-18T19:08:03.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.158+0000 7f8cb664f700 1 -- start start 2024-03-18T19:08:03.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.158+0000 7f8cb664f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cb006ae60 0x7f8cb0166270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:03.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.158+0000 7f8cb664f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb0072720 con 0x7f8cb015b120 2024-03-18T19:08:03.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.158+0000 7f8cb664f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb006ffc0 con 0x7f8cb015b500 2024-03-18T19:08:03.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.158+0000 7f8cb664f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb006b520 con 0x7f8cb006ae60 2024-03-18T19:08:03.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.158+0000 7f8caffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8cb015b120 0x7f8cb0072010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34998/0 (socket says 172.21.15.1:34998) 2024-03-18T19:08:03.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.158+0000 7f8caffff700 1 -- 172.21.15.1:0/868945445 learned_addr learned my addr 172.21.15.1:0/868945445 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:03.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.159+0000 7f8cb4bec700 1 --2- 172.21.15.1:0/868945445 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cb006ae60 0x7f8cb0166270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:03.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.159+0000 7f8caeffd700 1 -- 172.21.15.1:0/868945445 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400648566 0 0) 0x7f8ca0001110 con 0x7f8cb015b120 2024-03-18T19:08:03.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.159+0000 7f8caeffd700 1 -- 172.21.15.1:0/868945445 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c980035b0 con 0x7f8cb015b120 2024-03-18T19:08:03.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.159+0000 7f8caeffd700 1 -- 172.21.15.1:0/868945445 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933615873 0 0) 0x7f8cb006ffc0 con 0x7f8cb015b500 2024-03-18T19:08:03.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.159+0000 7f8caeffd700 1 -- 172.21.15.1:0/868945445 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca0001110 con 0x7f8cb015b500 2024-03-18T19:08:03.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.159+0000 7f8caeffd700 1 -- 172.21.15.1:0/868945445 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3386411972 0 0) 0x7f8c980035b0 con 0x7f8cb015b120 2024-03-18T19:08:03.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.159+0000 7f8cb4bec700 1 -- 172.21.15.1:0/868945445 >> v1:172.21.15.165:6789/0 conn(0x7f8cb015b500 legacy=0x7f8cb006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:03.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.159+0000 7f8cb4bec700 1 -- 172.21.15.1:0/868945445 >> v1:172.21.15.1:6789/0 conn(0x7f8cb015b120 legacy=0x7f8cb0072010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:03.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.159+0000 7f8cb4bec700 1 -- 172.21.15.1:0/868945445 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb006ef20 con 0x7f8cb006ae60 2024-03-18T19:08:03.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.159+0000 7f8cb4bec700 1 --2- 172.21.15.1:0/868945445 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cb006ae60 0x7f8cb0166270 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8009dc0 tx=0x7f8ca800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=687aa11ff3e7a6aa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:03.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.159+0000 7f8caeffd700 1 -- 172.21.15.1:0/868945445 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca800e070 con 0x7f8cb006ae60 2024-03-18T19:08:03.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.160+0000 7f8caeffd700 1 -- 172.21.15.1:0/868945445 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca8003a20 con 0x7f8cb006ae60 2024-03-18T19:08:03.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.160+0000 7f8caeffd700 1 -- 172.21.15.1:0/868945445 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0001800 con 0x7f8cb006ae60 2024-03-18T19:08:03.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.160+0000 7f8cb664f700 1 -- 172.21.15.1:0/868945445 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cb006ae60 msgr2=0x7f8cb0166270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:03.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.160+0000 7f8cb664f700 1 --2- 172.21.15.1:0/868945445 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cb006ae60 0x7f8cb0166270 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8009dc0 tx=0x7f8ca800b330 comp rx=0 tx=0).stop 2024-03-18T19:08:03.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.160+0000 7f8cb664f700 1 -- 172.21.15.1:0/868945445 shutdown_connections 2024-03-18T19:08:03.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.160+0000 7f8cb664f700 1 --2- 172.21.15.1:0/868945445 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cb006ae60 0x7f8cb0166270 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.160+0000 7f8cb664f700 1 -- 172.21.15.1:0/868945445 >> 172.21.15.1:0/868945445 conn(0x7f8cb00ccab0 msgr2=0x7f8cb00ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:03.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.160+0000 7f8cb664f700 1 -- 172.21.15.1:0/868945445 shutdown_connections 2024-03-18T19:08:03.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.160+0000 7f8cb664f700 1 -- 172.21.15.1:0/868945445 wait complete. 2024-03-18T19:08:03.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.161+0000 7f8cb664f700 1 Processor -- start 2024-03-18T19:08:03.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.162+0000 7f8cb664f700 1 -- start start 2024-03-18T19:08:03.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.162+0000 7f8cb664f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cb006ae60 0x7f8cb00cf460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:03.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.162+0000 7f8cb664f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb015b120 0x7f8cb0157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:03.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.162+0000 7f8cb664f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cb015b500 0x7f8cb01e5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:03.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.162+0000 7f8cb664f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb01e6e60 con 0x7f8cb015b120 2024-03-18T19:08:03.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.162+0000 7f8cb664f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb01584a0 con 0x7f8cb006ae60 2024-03-18T19:08:03.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.162+0000 7f8cb664f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb01e62e0 con 0x7f8cb015b500 2024-03-18T19:08:03.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.162+0000 7f8caf7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb015b120 0x7f8cb0157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:03.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.162+0000 7f8caf7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb015b120 0x7f8cb0157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46620/0 (socket says 172.21.15.1:46620) 2024-03-18T19:08:03.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.162+0000 7f8caf7fe700 1 -- 172.21.15.1:0/918953097 learned_addr learned my addr 172.21.15.1:0/918953097 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:03.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.162+0000 7f8caffff700 1 --2- 172.21.15.1:0/918953097 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cb006ae60 0x7f8cb00cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:03.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.162+0000 7f8cb4bec700 1 --2- 172.21.15.1:0/918953097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cb015b500 0x7f8cb01e5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:03.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.163+0000 7f8caf7fe700 1 -- 172.21.15.1:0/918953097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cb015b500 msgr2=0x7f8cb01e5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:03.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.163+0000 7f8caf7fe700 1 --2- 172.21.15.1:0/918953097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cb015b500 0x7f8cb01e5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.163+0000 7f8caf7fe700 1 -- 172.21.15.1:0/918953097 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cb006ae60 msgr2=0x7f8cb00cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:03.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.163+0000 7f8caf7fe700 1 --2- 172.21.15.1:0/918953097 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cb006ae60 0x7f8cb00cf460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.163+0000 7f8caf7fe700 1 -- 172.21.15.1:0/918953097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca8009a70 con 0x7f8cb015b120 2024-03-18T19:08:03.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.163+0000 7f8caf7fe700 1 --2- 172.21.15.1:0/918953097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb015b120 0x7f8cb0157f60 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f8ca400c500 tx=0x7f8ca400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:03.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.163+0000 7f8cacff9700 1 -- 172.21.15.1:0/918953097 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca40052d0 con 0x7f8cb015b120 2024-03-18T19:08:03.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.163+0000 7f8cacff9700 1 -- 172.21.15.1:0/918953097 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca4005430 con 0x7f8cb015b120 2024-03-18T19:08:03.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.163+0000 7f8cacff9700 1 -- 172.21.15.1:0/918953097 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca40038c0 con 0x7f8cb015b120 2024-03-18T19:08:03.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.163+0000 7f8cb664f700 1 -- 172.21.15.1:0/918953097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb01e65c0 con 0x7f8cb015b120 2024-03-18T19:08:03.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.163+0000 7f8cb664f700 1 -- 172.21.15.1:0/918953097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb01e6b80 con 0x7f8cb015b120 2024-03-18T19:08:03.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.164+0000 7f8cb664f700 1 -- 172.21.15.1:0/918953097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb00b3d10 con 0x7f8cb015b120 2024-03-18T19:08:03.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.165+0000 7f8cacff9700 1 -- 172.21.15.1:0/918953097 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 56) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f8ca4003a20 con 0x7f8cb015b120 2024-03-18T19:08:03.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.165+0000 7f8cacff9700 1 --2- 172.21.15.1:0/918953097 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8c9c06aed0 0x7f8c9c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:03.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.165+0000 7f8caffff700 1 -- 172.21.15.1:0/918953097 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8c9c06aed0 msgr2=0x7f8c9c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:03.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.165+0000 7f8cacff9700 1 -- 172.21.15.1:0/918953097 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f8ca4088ee0 con 0x7f8cb015b120 2024-03-18T19:08:03.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.165+0000 7f8caffff700 1 --2- 172.21.15.1:0/918953097 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8c9c06aed0 0x7f8c9c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:08:03.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.170+0000 7f8cacff9700 1 -- 172.21.15.1:0/918953097 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8ca4053d50 con 0x7f8cb015b120 2024-03-18T19:08:03.203 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:03.207+0000 7f1b3c187240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:08:03.323 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:03.327+0000 7f30b35e4240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:08:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.366+0000 7f8caffff700 1 -- 172.21.15.1:0/918953097 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8c9c06aed0 msgr2=0x7f8c9c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.366+0000 7f8caffff700 1 --2- 172.21.15.1:0/918953097 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8c9c06aed0 0x7f8c9c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:08:03.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.375+0000 7f8cb664f700 1 -- 172.21.15.1:0/918953097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cb00b3d10 con 0x7f8cb015b120 2024-03-18T19:08:03.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.376+0000 7f8cacff9700 1 -- 172.21.15.1:0/918953097 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16887 (secure 0 0 0) 0x7f8cb00b3d10 con 0x7f8cb015b120 2024-03-18T19:08:03.373 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:03.374 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":120,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:07:59.090022+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:06:39.741939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-18T19:06:40.871304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.10","mappings":[{"from":1,"to":2}]},{"pgid":"14.18","mappings":[{"from":1,"to":5}]},{"pgid":"14.21","mappings":[{"from":1,"to":7}]},{"pgid":"14.29","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/1072638685":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/67600845":"2024-03-19T19:06:57.097725+0000","172.21.15.1:6833/76998":"2024-03-19T19:06:57.097725+0000","172.21.15.1:6832/76998":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/1805165487":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/1343251370":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/3890713729":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/2823326236":"2024-03-19T19:06:57.097725+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:08:03.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.378+0000 7f8cb664f700 1 -- 172.21.15.1:0/918953097 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8c9c06aed0 msgr2=0x7f8c9c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:08:03.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.378+0000 7f8cb664f700 1 --2- 172.21.15.1:0/918953097 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8c9c06aed0 0x7f8c9c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.378+0000 7f8cb664f700 1 -- 172.21.15.1:0/918953097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb015b120 msgr2=0x7f8cb0157f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:03.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.378+0000 7f8cb664f700 1 --2- 172.21.15.1:0/918953097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb015b120 0x7f8cb0157f60 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f8ca400c500 tx=0x7f8ca400c810 comp rx=0 tx=0).stop 2024-03-18T19:08:03.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.379+0000 7f8cb664f700 1 -- 172.21.15.1:0/918953097 shutdown_connections 2024-03-18T19:08:03.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.379+0000 7f8cb664f700 1 --2- 172.21.15.1:0/918953097 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8c9c06aed0 0x7f8c9c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.379+0000 7f8cb664f700 1 --2- 172.21.15.1:0/918953097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cb015b500 0x7f8cb01e5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.379+0000 7f8cb664f700 1 --2- 172.21.15.1:0/918953097 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cb006ae60 0x7f8cb00cf460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.379+0000 7f8cb664f700 1 --2- 172.21.15.1:0/918953097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cb015b120 0x7f8cb0157f60 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.379+0000 7f8cb664f700 1 -- 172.21.15.1:0/918953097 >> 172.21.15.1:0/918953097 conn(0x7f8cb00ccab0 msgr2=0x7f8cb00cce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:03.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.379+0000 7f8cb664f700 1 -- 172.21.15.1:0/918953097 shutdown_connections 2024-03-18T19:08:03.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.379+0000 7f8cb664f700 1 -- 172.21.15.1:0/918953097 wait complete. 2024-03-18T19:08:03.386 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:03.389+0000 7f1b3c187240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:08:03.388 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T19:06:39.741939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T19:06:40.871304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:08:03.388 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:08:03.472 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:03.476+0000 7f30b35e4240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:08:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.525+0000 7fb76a27a700 1 Processor -- start 2024-03-18T19:08:03.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.525+0000 7fb76a27a700 1 -- start start 2024-03-18T19:08:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.526+0000 7fb76a27a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb764071cc0 0x7fb7640ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.526+0000 7fb76a27a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb76406d560 0x7fb76406fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.526+0000 7fb76a27a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb764070000 con 0x7fb76415a240 2024-03-18T19:08:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.526+0000 7fb76a27a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7640adff0 con 0x7fb76406d560 2024-03-18T19:08:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.526+0000 7fb76a27a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7640cac80 con 0x7fb764071cc0 2024-03-18T19:08:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.526+0000 7fb768817700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb76406d560 0x7fb76406fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.526+0000 7fb768817700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb76406d560 0x7fb76406fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46622/0 (socket says 172.21.15.1:46622) 2024-03-18T19:08:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.526+0000 7fb768817700 1 -- 172.21.15.1:0/1387310201 learned_addr learned my addr 172.21.15.1:0/1387310201 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.526+0000 7fb7637fe700 1 --2- 172.21.15.1:0/1387310201 >> v2:172.21.15.165:3300/0 conn(0x7fb764071cc0 0x7fb7640ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.527+0000 7fb768817700 1 -- 172.21.15.1:0/1387310201 >> v2:172.21.15.165:3300/0 conn(0x7fb764071cc0 msgr2=0x7fb7640ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:03.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.527+0000 7fb768817700 1 --2- 172.21.15.1:0/1387310201 >> v2:172.21.15.165:3300/0 conn(0x7fb764071cc0 0x7fb7640ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.527+0000 7fb768817700 1 -- 172.21.15.1:0/1387310201 >> v1:172.21.15.165:6789/0 conn(0x7fb76415a240 legacy=0x7fb7640715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.527+0000 7fb768817700 1 -- 172.21.15.1:0/1387310201 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7640caf60 con 0x7fb76406d560 2024-03-18T19:08:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.527+0000 7fb762ffd700 1 -- 172.21.15.1:0/1387310201 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954240225 0 0) 0x7fb764070000 con 0x7fb76415a240 2024-03-18T19:08:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.527+0000 7fb768817700 1 --2- 172.21.15.1:0/1387310201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb76406d560 0x7fb76406fac0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fb76415eeb0 tx=0x7fb75c00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d48560642ee4b509 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.527+0000 7fb762ffd700 1 -- 172.21.15.1:0/1387310201 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c010070 con 0x7fb76406d560 2024-03-18T19:08:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.527+0000 7fb762ffd700 1 -- 172.21.15.1:0/1387310201 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75c00dce0 con 0x7fb76406d560 2024-03-18T19:08:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.527+0000 7fb762ffd700 1 -- 172.21.15.1:0/1387310201 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c00b310 con 0x7fb76406d560 2024-03-18T19:08:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.528+0000 7fb76a27a700 1 -- 172.21.15.1:0/1387310201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb76406d560 msgr2=0x7fb76406fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.528+0000 7fb76a27a700 1 --2- 172.21.15.1:0/1387310201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb76406d560 0x7fb76406fac0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fb76415eeb0 tx=0x7fb75c00d430 comp rx=0 tx=0).stop 2024-03-18T19:08:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.528+0000 7fb76a27a700 1 -- 172.21.15.1:0/1387310201 shutdown_connections 2024-03-18T19:08:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.528+0000 7fb76a27a700 1 --2- 172.21.15.1:0/1387310201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb76406d560 0x7fb76406fac0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.528+0000 7fb76a27a700 1 --2- 172.21.15.1:0/1387310201 >> v2:172.21.15.165:3300/0 conn(0x7fb764071cc0 0x7fb7640ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.528+0000 7fb76a27a700 1 -- 172.21.15.1:0/1387310201 >> 172.21.15.1:0/1387310201 conn(0x7fb7640cff30 msgr2=0x7fb7640d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:03.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.528+0000 7fb76a27a700 1 -- 172.21.15.1:0/1387310201 shutdown_connections 2024-03-18T19:08:03.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.528+0000 7fb76a27a700 1 -- 172.21.15.1:0/1387310201 wait complete. 2024-03-18T19:08:03.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.529+0000 7fb76a27a700 1 Processor -- start 2024-03-18T19:08:03.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.529+0000 7fb76a27a700 1 -- start start 2024-03-18T19:08:03.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.530+0000 7fb76a27a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb76406d560 0x7fb7641587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:03.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.530+0000 7fb76a27a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb764071cc0 0x7fb764156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:03.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.530+0000 7fb76a27a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb76415a240 0x7fb764157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:03.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.530+0000 7fb76a27a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7640c6e60 con 0x7fb764071cc0 2024-03-18T19:08:03.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.530+0000 7fb76a27a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7641579b0 con 0x7fb76415a240 2024-03-18T19:08:03.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.530+0000 7fb76a27a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb764157c00 con 0x7fb76406d560 2024-03-18T19:08:03.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.530+0000 7fb7637fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb764071cc0 0x7fb764156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:03.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.530+0000 7fb7637fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb764071cc0 0x7fb764156df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46638/0 (socket says 172.21.15.1:46638) 2024-03-18T19:08:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.530+0000 7fb7637fe700 1 -- 172.21.15.1:0/773204914 learned_addr learned my addr 172.21.15.1:0/773204914 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.530+0000 7fb768817700 1 --2- 172.21.15.1:0/773204914 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb76415a240 0x7fb764157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.530+0000 7fb763fff700 1 --2- 172.21.15.1:0/773204914 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb76406d560 0x7fb7641587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.530+0000 7fb7637fe700 1 -- 172.21.15.1:0/773204914 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb76406d560 msgr2=0x7fb7641587e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.530+0000 7fb7637fe700 1 --2- 172.21.15.1:0/773204914 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb76406d560 0x7fb7641587e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.530+0000 7fb7637fe700 1 -- 172.21.15.1:0/773204914 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb76415a240 msgr2=0x7fb764157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.530+0000 7fb7637fe700 1 --2- 172.21.15.1:0/773204914 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb76415a240 0x7fb764157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.530+0000 7fb7637fe700 1 -- 172.21.15.1:0/773204914 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75c00bd40 con 0x7fb764071cc0 2024-03-18T19:08:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.531+0000 7fb7637fe700 1 --2- 172.21.15.1:0/773204914 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb764071cc0 0x7fb764156df0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fb75800e9b0 tx=0x7fb75800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.531+0000 7fb760ff9700 1 -- 172.21.15.1:0/773204914 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb758009710 con 0x7fb764071cc0 2024-03-18T19:08:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.531+0000 7fb760ff9700 1 -- 172.21.15.1:0/773204914 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75800ee50 con 0x7fb764071cc0 2024-03-18T19:08:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.531+0000 7fb76a27a700 1 -- 172.21.15.1:0/773204914 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76409d1b0 con 0x7fb764071cc0 2024-03-18T19:08:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.531+0000 7fb760ff9700 1 -- 172.21.15.1:0/773204914 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75800ca50 con 0x7fb764071cc0 2024-03-18T19:08:03.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.531+0000 7fb76a27a700 1 -- 172.21.15.1:0/773204914 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb76409d770 con 0x7fb764071cc0 2024-03-18T19:08:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.532+0000 7fb76a27a700 1 -- 172.21.15.1:0/773204914 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7640addb0 con 0x7fb764071cc0 2024-03-18T19:08:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.532+0000 7fb760ff9700 1 -- 172.21.15.1:0/773204914 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 56) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fb75800cbb0 con 0x7fb764071cc0 2024-03-18T19:08:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.533+0000 7fb760ff9700 1 --2- 172.21.15.1:0/773204914 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb75006aed0 0x7fb75006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:03.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.533+0000 7fb763fff700 1 -- 172.21.15.1:0/773204914 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb75006aed0 msgr2=0x7fb75006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:03.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.533+0000 7fb760ff9700 1 -- 172.21.15.1:0/773204914 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fb758089a30 con 0x7fb764071cc0 2024-03-18T19:08:03.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.533+0000 7fb763fff700 1 --2- 172.21.15.1:0/773204914 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb75006aed0 0x7fb75006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:08:03.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.537+0000 7fb760ff9700 1 -- 172.21.15.1:0/773204914 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb758054590 con 0x7fb764071cc0 2024-03-18T19:08:03.544 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:03.548+0000 7f30b35e4240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:08:03.616 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:03.620+0000 7f1b3c187240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:08:03.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.734+0000 7fb763fff700 1 -- 172.21.15.1:0/773204914 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb75006aed0 msgr2=0x7fb75006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:03.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.734+0000 7fb763fff700 1 --2- 172.21.15.1:0/773204914 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb75006aed0 0x7fb75006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:08:03.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.739+0000 7fb76a27a700 1 -- 172.21.15.1:0/773204914 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb7640676a0 con 0x7fb764071cc0 2024-03-18T19:08:03.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.739+0000 7fb760ff9700 1 -- 172.21.15.1:0/773204914 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb75800ce60 con 0x7fb764071cc0 2024-03-18T19:08:03.736 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:08:03.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.741+0000 7fb76a27a700 1 -- 172.21.15.1:0/773204914 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb75006aed0 msgr2=0x7fb75006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:08:03.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.741+0000 7fb76a27a700 1 --2- 172.21.15.1:0/773204914 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb75006aed0 0x7fb75006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.741+0000 7fb76a27a700 1 -- 172.21.15.1:0/773204914 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb764071cc0 msgr2=0x7fb764156df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:03.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.741+0000 7fb76a27a700 1 --2- 172.21.15.1:0/773204914 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb764071cc0 0x7fb764156df0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fb75800e9b0 tx=0x7fb75800ecc0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.742+0000 7fb76a27a700 1 -- 172.21.15.1:0/773204914 shutdown_connections 2024-03-18T19:08:03.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.742+0000 7fb76a27a700 1 --2- 172.21.15.1:0/773204914 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb75006aed0 0x7fb75006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.742+0000 7fb76a27a700 1 --2- 172.21.15.1:0/773204914 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb76406d560 0x7fb7641587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.742+0000 7fb76a27a700 1 --2- 172.21.15.1:0/773204914 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb76415a240 0x7fb764157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.742+0000 7fb76a27a700 1 --2- 172.21.15.1:0/773204914 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb764071cc0 0x7fb764156df0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.742+0000 7fb76a27a700 1 -- 172.21.15.1:0/773204914 >> 172.21.15.1:0/773204914 conn(0x7fb7640cff30 msgr2=0x7fb7640ca460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:03.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.742+0000 7fb76a27a700 1 -- 172.21.15.1:0/773204914 shutdown_connections 2024-03-18T19:08:03.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.742+0000 7fb76a27a700 1 -- 172.21.15.1:0/773204914 wait complete. 2024-03-18T19:08:03.751 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T19:08:03.782 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:03.785+0000 7f1b3c187240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:08:03.863 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:03.866+0000 7f1b3c187240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:08:03.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.885+0000 7fe8e8843700 1 Processor -- start 2024-03-18T19:08:03.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.885+0000 7fe8e8843700 1 -- start start 2024-03-18T19:08:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.887+0000 7fe8e8843700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe8e415b6c0 0x7fe8e40b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.887+0000 7fe8e8843700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe8e40cd430 0x7fe8e40c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.887+0000 7fe8e8843700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e40d01f0 con 0x7fe8e40c8600 2024-03-18T19:08:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.887+0000 7fe8e8843700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8e40ccfa0 con 0x7fe8e415b6c0 2024-03-18T19:08:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.887+0000 7fe8e8843700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e40d1db0 con 0x7fe8e40cd430 2024-03-18T19:08:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.887+0000 7fe8e259c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe8e415b6c0 0x7fe8e40b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.887+0000 7fe8e259c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe8e415b6c0 0x7fe8e40b2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46648/0 (socket says 172.21.15.1:46648) 2024-03-18T19:08:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.887+0000 7fe8e259c700 1 -- 172.21.15.1:0/1805272325 learned_addr learned my addr 172.21.15.1:0/1805272325 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:03.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.887+0000 7fe8e1d9b700 1 --2- 172.21.15.1:0/1805272325 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe8e40cd430 0x7fe8e40c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:03.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.888+0000 7fe8e259c700 1 -- 172.21.15.1:0/1805272325 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe8e40cd430 msgr2=0x7fe8e40c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:03.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.888+0000 7fe8e259c700 1 --2- 172.21.15.1:0/1805272325 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe8e40cd430 0x7fe8e40c80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.888+0000 7fe8e259c700 1 -- 172.21.15.1:0/1805272325 >> v1:172.21.15.165:6789/0 conn(0x7fe8e40c8600 legacy=0x7fe8e40cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:03.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.888+0000 7fe8e259c700 1 -- 172.21.15.1:0/1805272325 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e40cff10 con 0x7fe8e415b6c0 2024-03-18T19:08:03.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.888+0000 7fe8e259c700 1 --2- 172.21.15.1:0/1805272325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8e415b6c0 0x7fe8e40b2160 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4009920 tx=0x7fe8d4009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a7c72ee36ca55cc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:03.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.888+0000 7fe8e159a700 1 -- 172.21.15.1:0/1805272325 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d4011830 con 0x7fe8e415b6c0 2024-03-18T19:08:03.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.888+0000 7fe8e159a700 1 -- 172.21.15.1:0/1805272325 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d4011990 con 0x7fe8e415b6c0 2024-03-18T19:08:03.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.888+0000 7fe8e159a700 1 -- 172.21.15.1:0/1805272325 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d4011d20 con 0x7fe8e415b6c0 2024-03-18T19:08:03.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.889+0000 7fe8e8843700 1 -- 172.21.15.1:0/1805272325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8e415b6c0 msgr2=0x7fe8e40b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:03.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.889+0000 7fe8e8843700 1 --2- 172.21.15.1:0/1805272325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8e415b6c0 0x7fe8e40b2160 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4009920 tx=0x7fe8d4009c30 comp rx=0 tx=0).stop 2024-03-18T19:08:03.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.889+0000 7fe8e8843700 1 -- 172.21.15.1:0/1805272325 shutdown_connections 2024-03-18T19:08:03.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.889+0000 7fe8e8843700 1 --2- 172.21.15.1:0/1805272325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8e415b6c0 0x7fe8e40b2160 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.889+0000 7fe8e8843700 1 --2- 172.21.15.1:0/1805272325 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe8e40cd430 0x7fe8e40c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.889+0000 7fe8e8843700 1 -- 172.21.15.1:0/1805272325 >> 172.21.15.1:0/1805272325 conn(0x7fe8e40d0ff0 msgr2=0x7fe8e40d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:03.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.889+0000 7fe8e8843700 1 -- 172.21.15.1:0/1805272325 shutdown_connections 2024-03-18T19:08:03.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.889+0000 7fe8e8843700 1 -- 172.21.15.1:0/1805272325 wait complete. 2024-03-18T19:08:03.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.891+0000 7fe8e8843700 1 Processor -- start 2024-03-18T19:08:03.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.891+0000 7fe8e8843700 1 -- start start 2024-03-18T19:08:03.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8e8843700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe8e40c8600 0x7fe8e409e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:03.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8e8843700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8e40cd430 0x7fe8e4156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:03.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8e8843700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8e415b6c0 0x7fe8e409d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:03.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8e8843700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e4157000 con 0x7fe8e40cd430 2024-03-18T19:08:03.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8e8843700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e40b5dd0 con 0x7fe8e415b6c0 2024-03-18T19:08:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8e8843700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e409c230 con 0x7fe8e40c8600 2024-03-18T19:08:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8e1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8e40cd430 0x7fe8e4156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8e1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8e40cd430 0x7fe8e4156ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46660/0 (socket says 172.21.15.1:46660) 2024-03-18T19:08:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8e1d9b700 1 -- 172.21.15.1:0/960616480 learned_addr learned my addr 172.21.15.1:0/960616480 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8e259c700 1 --2- 172.21.15.1:0/960616480 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe8e40c8600 0x7fe8e409e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8e2d9d700 1 --2- 172.21.15.1:0/960616480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8e415b6c0 0x7fe8e409d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8e1d9b700 1 -- 172.21.15.1:0/960616480 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe8e40c8600 msgr2=0x7fe8e409e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8e1d9b700 1 --2- 172.21.15.1:0/960616480 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe8e40c8600 0x7fe8e409e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8e1d9b700 1 -- 172.21.15.1:0/960616480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8e415b6c0 msgr2=0x7fe8e409d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8e1d9b700 1 --2- 172.21.15.1:0/960616480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8e415b6c0 0x7fe8e409d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8e1d9b700 1 -- 172.21.15.1:0/960616480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d4009550 con 0x7fe8e40cd430 2024-03-18T19:08:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8e259c700 1 --2- 172.21.15.1:0/960616480 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe8e40c8600 0x7fe8e409e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:08:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8e1d9b700 1 --2- 172.21.15.1:0/960616480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8e40cd430 0x7fe8e4156ac0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fe8d800e9e0 tx=0x7fe8d800ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:03.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8d2ffd700 1 -- 172.21.15.1:0/960616480 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d8009710 con 0x7fe8e40cd430 2024-03-18T19:08:03.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8d2ffd700 1 -- 172.21.15.1:0/960616480 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d800ee80 con 0x7fe8e40cd430 2024-03-18T19:08:03.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8e8843700 1 -- 172.21.15.1:0/960616480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e409b900 con 0x7fe8e40cd430 2024-03-18T19:08:03.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.892+0000 7fe8d2ffd700 1 -- 172.21.15.1:0/960616480 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d800ca50 con 0x7fe8e40cd430 2024-03-18T19:08:03.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.893+0000 7fe8e8843700 1 -- 172.21.15.1:0/960616480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e409bec0 con 0x7fe8e40cd430 2024-03-18T19:08:03.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.894+0000 7fe8e8843700 1 -- 172.21.15.1:0/960616480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e40676a0 con 0x7fe8e40cd430 2024-03-18T19:08:03.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.894+0000 7fe8d2ffd700 1 -- 172.21.15.1:0/960616480 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 56) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fe8d800cbb0 con 0x7fe8e40cd430 2024-03-18T19:08:03.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.894+0000 7fe8d2ffd700 1 --2- 172.21.15.1:0/960616480 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe8c406aed0 0x7fe8c406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:03.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.894+0000 7fe8e259c700 1 -- 172.21.15.1:0/960616480 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe8c406aed0 msgr2=0x7fe8c406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:03.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.894+0000 7fe8d2ffd700 1 -- 172.21.15.1:0/960616480 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fe8d8089a10 con 0x7fe8e40cd430 2024-03-18T19:08:03.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.894+0000 7fe8e259c700 1 --2- 172.21.15.1:0/960616480 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe8c406aed0 0x7fe8c406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:08:03.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:03.899+0000 7fe8d2ffd700 1 -- 172.21.15.1:0/960616480 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe8d8054570 con 0x7fe8e40cd430 2024-03-18T19:08:03.990 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:03.994+0000 7f30b35e4240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:08:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.095+0000 7fe8e259c700 1 -- 172.21.15.1:0/960616480 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe8c406aed0 msgr2=0x7fe8c406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:04.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.095+0000 7fe8e259c700 1 --2- 172.21.15.1:0/960616480 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe8c406aed0 0x7fe8c406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:08:04.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.104+0000 7fe8e8843700 1 -- 172.21.15.1:0/960616480 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe8e409c370 con 0x7fe8e40cd430 2024-03-18T19:08:04.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.104+0000 7fe8d2ffd700 1 -- 172.21.15.1:0/960616480 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v120) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe8d8057b90 con 0x7fe8e40cd430 2024-03-18T19:08:04.101 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T19:08:04.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.106+0000 7fe8e8843700 1 -- 172.21.15.1:0/960616480 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe8c406aed0 msgr2=0x7fe8c406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:08:04.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.106+0000 7fe8e8843700 1 --2- 172.21.15.1:0/960616480 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe8c406aed0 0x7fe8c406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.106+0000 7fe8e8843700 1 -- 172.21.15.1:0/960616480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8e40cd430 msgr2=0x7fe8e4156ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:04.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.106+0000 7fe8e8843700 1 --2- 172.21.15.1:0/960616480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8e40cd430 0x7fe8e4156ac0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fe8d800e9e0 tx=0x7fe8d800ecf0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.107+0000 7fe8e8843700 1 -- 172.21.15.1:0/960616480 shutdown_connections 2024-03-18T19:08:04.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.107+0000 7fe8e8843700 1 --2- 172.21.15.1:0/960616480 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe8c406aed0 0x7fe8c406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.107+0000 7fe8e8843700 1 --2- 172.21.15.1:0/960616480 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe8e40c8600 0x7fe8e409e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.107+0000 7fe8e8843700 1 --2- 172.21.15.1:0/960616480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8e415b6c0 0x7fe8e409d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.107+0000 7fe8e8843700 1 --2- 172.21.15.1:0/960616480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8e40cd430 0x7fe8e4156ac0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.107+0000 7fe8e8843700 1 -- 172.21.15.1:0/960616480 >> 172.21.15.1:0/960616480 conn(0x7fe8e40d0ff0 msgr2=0x7fe8e40c9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:04.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.107+0000 7fe8e8843700 1 -- 172.21.15.1:0/960616480 shutdown_connections 2024-03-18T19:08:04.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.107+0000 7fe8e8843700 1 -- 172.21.15.1:0/960616480 wait complete. 2024-03-18T19:08:04.116 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T19:08:04.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.288+0000 7fb37e8ef700 1 Processor -- start 2024-03-18T19:08:04.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.288+0000 7fb37e8ef700 1 -- start start 2024-03-18T19:08:04.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.289+0000 7fb37e8ef700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb37815b6c0 0x7fb3780b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:04.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.289+0000 7fb37e8ef700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3780c87d0 0x7fb3780cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:04.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.289+0000 7fb37e8ef700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3780d01f0 con 0x7fb3780cd430 2024-03-18T19:08:04.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.289+0000 7fb37e8ef700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3780ccfa0 con 0x7fb37815b6c0 2024-03-18T19:08:04.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.289+0000 7fb37e8ef700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3780d1db0 con 0x7fb3780c87d0 2024-03-18T19:08:04.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.289+0000 7fb3777fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb3780cd430 0x7fb3780c80c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35002/0 (socket says 172.21.15.1:35002) 2024-03-18T19:08:04.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.289+0000 7fb3777fe700 1 -- 172.21.15.1:0/1727344186 learned_addr learned my addr 172.21.15.1:0/1727344186 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:04.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.289+0000 7fb377fff700 1 --2- 172.21.15.1:0/1727344186 >> v2:172.21.15.1:3300/0 conn(0x7fb37815b6c0 0x7fb3780b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:04.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.289+0000 7fb37ce8c700 1 --2- 172.21.15.1:0/1727344186 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3780c87d0 0x7fb3780cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:04.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.290+0000 7fb377fff700 1 -- 172.21.15.1:0/1727344186 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3780c87d0 msgr2=0x7fb3780cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:04.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.290+0000 7fb377fff700 1 --2- 172.21.15.1:0/1727344186 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3780c87d0 0x7fb3780cd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.290+0000 7fb377fff700 1 -- 172.21.15.1:0/1727344186 >> v1:172.21.15.1:6789/0 conn(0x7fb3780cd430 legacy=0x7fb3780c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:04.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.290+0000 7fb377fff700 1 -- 172.21.15.1:0/1727344186 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3780cff10 con 0x7fb37815b6c0 2024-03-18T19:08:04.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.290+0000 7fb376ffd700 1 -- 172.21.15.1:0/1727344186 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169910831 0 0) 0x7fb3780d01f0 con 0x7fb3780cd430 2024-03-18T19:08:04.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.290+0000 7fb377fff700 1 --2- 172.21.15.1:0/1727344186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb37815b6c0 0x7fb3780b2160 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fb368009840 tx=0x7fb368009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61cee6b825e085c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:04.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.290+0000 7fb376ffd700 1 -- 172.21.15.1:0/1727344186 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb368011830 con 0x7fb37815b6c0 2024-03-18T19:08:04.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.290+0000 7fb376ffd700 1 -- 172.21.15.1:0/1727344186 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb368011990 con 0x7fb37815b6c0 2024-03-18T19:08:04.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.290+0000 7fb376ffd700 1 -- 172.21.15.1:0/1727344186 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb368011d20 con 0x7fb37815b6c0 2024-03-18T19:08:04.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.291+0000 7fb37e8ef700 1 -- 172.21.15.1:0/1727344186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb37815b6c0 msgr2=0x7fb3780b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:04.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.291+0000 7fb37e8ef700 1 --2- 172.21.15.1:0/1727344186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb37815b6c0 0x7fb3780b2160 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fb368009840 tx=0x7fb368009b50 comp rx=0 tx=0).stop 2024-03-18T19:08:04.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.291+0000 7fb37e8ef700 1 -- 172.21.15.1:0/1727344186 shutdown_connections 2024-03-18T19:08:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.291+0000 7fb37e8ef700 1 --2- 172.21.15.1:0/1727344186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb37815b6c0 0x7fb3780b2160 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.291+0000 7fb37e8ef700 1 --2- 172.21.15.1:0/1727344186 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3780c87d0 0x7fb3780cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.291+0000 7fb37e8ef700 1 -- 172.21.15.1:0/1727344186 >> 172.21.15.1:0/1727344186 conn(0x7fb3780d0ff0 msgr2=0x7fb3780d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.291+0000 7fb37e8ef700 1 -- 172.21.15.1:0/1727344186 shutdown_connections 2024-03-18T19:08:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.291+0000 7fb37e8ef700 1 -- 172.21.15.1:0/1727344186 wait complete. 2024-03-18T19:08:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.292+0000 7fb37e8ef700 1 Processor -- start 2024-03-18T19:08:04.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.292+0000 7fb37e8ef700 1 -- start start 2024-03-18T19:08:04.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.293+0000 7fb37e8ef700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3780c87d0 0x7fb37809b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:04.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.293+0000 7fb37e8ef700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3780cd430 0x7fb37814e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:04.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.293+0000 7fb37e8ef700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb37815b6c0 0x7fb37814e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:04.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.293+0000 7fb37e8ef700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3780b1780 con 0x7fb37815b6c0 2024-03-18T19:08:04.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.293+0000 7fb37e8ef700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3780b0c00 con 0x7fb3780cd430 2024-03-18T19:08:04.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.293+0000 7fb37e8ef700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3780b0ee0 con 0x7fb3780c87d0 2024-03-18T19:08:04.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.293+0000 7fb37ce8c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb37815b6c0 0x7fb37814e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:04.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.293+0000 7fb377fff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3780c87d0 0x7fb37809b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:04.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.293+0000 7fb377fff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3780c87d0 0x7fb37809b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:45166/0 (socket says 172.21.15.1:45166) 2024-03-18T19:08:04.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.293+0000 7fb37ce8c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb37815b6c0 0x7fb37814e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46668/0 (socket says 172.21.15.1:46668) 2024-03-18T19:08:04.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.293+0000 7fb377fff700 1 -- 172.21.15.1:0/3539470066 learned_addr learned my addr 172.21.15.1:0/3539470066 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:04.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.293+0000 7fb3777fe700 1 --2- 172.21.15.1:0/3539470066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3780cd430 0x7fb37814e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:04.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.294+0000 7fb377fff700 1 -- 172.21.15.1:0/3539470066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3780cd430 msgr2=0x7fb37814e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:04.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.294+0000 7fb377fff700 1 --2- 172.21.15.1:0/3539470066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3780cd430 0x7fb37814e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.294+0000 7fb377fff700 1 -- 172.21.15.1:0/3539470066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb37815b6c0 msgr2=0x7fb37814e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:04.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.294+0000 7fb377fff700 1 --2- 172.21.15.1:0/3539470066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb37815b6c0 0x7fb37814e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.294+0000 7fb377fff700 1 -- 172.21.15.1:0/3539470066 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3680094a0 con 0x7fb3780c87d0 2024-03-18T19:08:04.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.294+0000 7fb37ce8c700 1 --2- 172.21.15.1:0/3539470066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb37815b6c0 0x7fb37814e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:08:04.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.294+0000 7fb377fff700 1 --2- 172.21.15.1:0/3539470066 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3780c87d0 0x7fb37809b820 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb368009fd0 tx=0x7fb368004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:04.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.294+0000 7fb374ff9700 1 -- 172.21.15.1:0/3539470066 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3680036a0 con 0x7fb3780c87d0 2024-03-18T19:08:04.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.294+0000 7fb374ff9700 1 -- 172.21.15.1:0/3539470066 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36801f070 con 0x7fb3780c87d0 2024-03-18T19:08:04.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.294+0000 7fb374ff9700 1 -- 172.21.15.1:0/3539470066 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb368003a60 con 0x7fb3780c87d0 2024-03-18T19:08:04.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.294+0000 7fb37e8ef700 1 -- 172.21.15.1:0/3539470066 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3780b11c0 con 0x7fb3780c87d0 2024-03-18T19:08:04.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.294+0000 7fb37e8ef700 1 -- 172.21.15.1:0/3539470066 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3780b0820 con 0x7fb3780c87d0 2024-03-18T19:08:04.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.295+0000 7fb37e8ef700 1 -- 172.21.15.1:0/3539470066 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3780b09b0 con 0x7fb3780c87d0 2024-03-18T19:08:04.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.296+0000 7fb374ff9700 1 -- 172.21.15.1:0/3539470066 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 56) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fb368003bc0 con 0x7fb3780c87d0 2024-03-18T19:08:04.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.296+0000 7fb374ff9700 1 --2- 172.21.15.1:0/3539470066 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb36406aed0 0x7fb36406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:04.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.296+0000 7fb3777fe700 1 -- 172.21.15.1:0/3539470066 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb36406aed0 msgr2=0x7fb36406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:04.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.296+0000 7fb374ff9700 1 -- 172.21.15.1:0/3539470066 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fb368021070 con 0x7fb3780c87d0 2024-03-18T19:08:04.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.296+0000 7fb3777fe700 1 --2- 172.21.15.1:0/3539470066 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb36406aed0 0x7fb36406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:08:04.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.301+0000 7fb374ff9700 1 -- 172.21.15.1:0/3539470066 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb368055df0 con 0x7fb3780c87d0 2024-03-18T19:08:04.328 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:04.332+0000 7f30b35e4240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:08:04.353 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:04.356+0000 7f1b3c187240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:08:04.488 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:04.492+0000 7f30b35e4240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:08:04.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.497+0000 7fb3777fe700 1 -- 172.21.15.1:0/3539470066 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb36406aed0 msgr2=0x7fb36406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:04.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.497+0000 7fb3777fe700 1 --2- 172.21.15.1:0/3539470066 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb36406aed0 0x7fb36406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:08:04.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.505+0000 7fb37e8ef700 1 -- 172.21.15.1:0/3539470066 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb3780b1020 con 0x7fb3780c87d0 2024-03-18T19:08:04.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.506+0000 7fb374ff9700 1 -- 172.21.15.1:0/3539470066 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v120) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb3780b1020 con 0x7fb3780c87d0 2024-03-18T19:08:04.503 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T19:08:04.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.508+0000 7fb37e8ef700 1 -- 172.21.15.1:0/3539470066 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb36406aed0 msgr2=0x7fb36406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:08:04.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.508+0000 7fb37e8ef700 1 --2- 172.21.15.1:0/3539470066 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb36406aed0 0x7fb36406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.508+0000 7fb37e8ef700 1 -- 172.21.15.1:0/3539470066 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3780c87d0 msgr2=0x7fb37809b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:04.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.508+0000 7fb37e8ef700 1 --2- 172.21.15.1:0/3539470066 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3780c87d0 0x7fb37809b820 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb368009fd0 tx=0x7fb368004750 comp rx=0 tx=0).stop 2024-03-18T19:08:04.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.508+0000 7fb37e8ef700 1 -- 172.21.15.1:0/3539470066 shutdown_connections 2024-03-18T19:08:04.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.508+0000 7fb37e8ef700 1 --2- 172.21.15.1:0/3539470066 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb36406aed0 0x7fb36406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.508+0000 7fb37e8ef700 1 --2- 172.21.15.1:0/3539470066 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3780c87d0 0x7fb37809b820 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.508+0000 7fb37e8ef700 1 --2- 172.21.15.1:0/3539470066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb37815b6c0 0x7fb37814e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.508+0000 7fb37e8ef700 1 --2- 172.21.15.1:0/3539470066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3780cd430 0x7fb37814e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.508+0000 7fb37e8ef700 1 -- 172.21.15.1:0/3539470066 >> 172.21.15.1:0/3539470066 conn(0x7fb3780d0ff0 msgr2=0x7fb3780c9130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:04.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.508+0000 7fb37e8ef700 1 -- 172.21.15.1:0/3539470066 shutdown_connections 2024-03-18T19:08:04.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.508+0000 7fb37e8ef700 1 -- 172.21.15.1:0/3539470066 wait complete. 2024-03-18T19:08:04.518 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:08:04.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.648+0000 7f3840364700 1 Processor -- start 2024-03-18T19:08:04.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.648+0000 7f3840364700 1 -- start start 2024-03-18T19:08:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.650+0000 7f3840364700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f38380cc170 0x7f38380ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:04.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.650+0000 7f3840364700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383806e270 con 0x7f38380ae600 2024-03-18T19:08:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.650+0000 7f3840364700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38380ae050 con 0x7f383815a2e0 2024-03-18T19:08:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.650+0000 7f3840364700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f383806f690 con 0x7f38380cc170 2024-03-18T19:08:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.650+0000 7f383e100700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f38380ae600 0x7f38380ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54798/0 (socket says 172.21.15.1:54798) 2024-03-18T19:08:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.650+0000 7f383e100700 1 -- 172.21.15.1:0/904776548 learned_addr learned my addr 172.21.15.1:0/904776548 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.650+0000 7f383e901700 1 --2- 172.21.15.1:0/904776548 >> v2:172.21.15.165:3300/0 conn(0x7f38380cc170 0x7f38380ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.650+0000 7f383d0fe700 1 -- 172.21.15.1:0/904776548 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727541738 0 0) 0x7f383806e270 con 0x7f38380ae600 2024-03-18T19:08:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.650+0000 7f383d0fe700 1 -- 172.21.15.1:0/904776548 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38240035b0 con 0x7f38380ae600 2024-03-18T19:08:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.650+0000 7f383e901700 1 -- 172.21.15.1:0/904776548 >> v1:172.21.15.165:6789/0 conn(0x7f383815a2e0 legacy=0x7f383806ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.650+0000 7f383e901700 1 -- 172.21.15.1:0/904776548 >> v1:172.21.15.1:6789/0 conn(0x7f38380ae600 legacy=0x7f38380ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.650+0000 7f383d0fe700 1 -- 172.21.15.1:0/904776548 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402358346 0 0) 0x7f38380ae050 con 0x7f383815a2e0 2024-03-18T19:08:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.650+0000 7f383e901700 1 -- 172.21.15.1:0/904776548 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38380add70 con 0x7f38380cc170 2024-03-18T19:08:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.651+0000 7f383e901700 1 --2- 172.21.15.1:0/904776548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38380cc170 0x7f38380ad790 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3830009d90 tx=0x7f383000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6d680927dcfe0d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.651+0000 7f383d0fe700 1 -- 172.21.15.1:0/904776548 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383000e070 con 0x7f38380cc170 2024-03-18T19:08:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.651+0000 7f383d0fe700 1 -- 172.21.15.1:0/904776548 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383000bb20 con 0x7f38380cc170 2024-03-18T19:08:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.651+0000 7f383d0fe700 1 -- 172.21.15.1:0/904776548 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383000bed0 con 0x7f38380cc170 2024-03-18T19:08:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.651+0000 7f3840364700 1 -- 172.21.15.1:0/904776548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38380cc170 msgr2=0x7f38380ad790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.651+0000 7f3840364700 1 --2- 172.21.15.1:0/904776548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38380cc170 0x7f38380ad790 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3830009d90 tx=0x7f383000b330 comp rx=0 tx=0).stop 2024-03-18T19:08:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.651+0000 7f3840364700 1 -- 172.21.15.1:0/904776548 shutdown_connections 2024-03-18T19:08:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.651+0000 7f3840364700 1 --2- 172.21.15.1:0/904776548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38380cc170 0x7f38380ad790 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.651+0000 7f3840364700 1 -- 172.21.15.1:0/904776548 >> 172.21.15.1:0/904776548 conn(0x7f383806b860 msgr2=0x7f383806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.652+0000 7f3840364700 1 -- 172.21.15.1:0/904776548 shutdown_connections 2024-03-18T19:08:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.652+0000 7f3840364700 1 -- 172.21.15.1:0/904776548 wait complete. 2024-03-18T19:08:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.653+0000 7f3840364700 1 Processor -- start 2024-03-18T19:08:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.653+0000 7f3840364700 1 -- start start 2024-03-18T19:08:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.654+0000 7f3840364700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38380ae600 0x7f38381f16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.654+0000 7f3840364700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38380cc170 0x7f383809d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.654+0000 7f3840364700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f383815a2e0 0x7f38380afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.654+0000 7f3840364700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38381f1be0 con 0x7f38380ae600 2024-03-18T19:08:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.654+0000 7f3840364700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383814c2d0 con 0x7f38380cc170 2024-03-18T19:08:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.654+0000 7f3840364700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38381f5150 con 0x7f383815a2e0 2024-03-18T19:08:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.654+0000 7f383e100700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38380ae600 0x7f38381f16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.654+0000 7f383e100700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38380ae600 0x7f38381f16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40112/0 (socket says 172.21.15.1:40112) 2024-03-18T19:08:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.654+0000 7f383e100700 1 -- 172.21.15.1:0/3242693276 learned_addr learned my addr 172.21.15.1:0/3242693276 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.654+0000 7f383d8ff700 1 --2- 172.21.15.1:0/3242693276 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38380cc170 0x7f383809d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.654+0000 7f383e901700 1 --2- 172.21.15.1:0/3242693276 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f383815a2e0 0x7f38380afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.655+0000 7f383e100700 1 -- 172.21.15.1:0/3242693276 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f383815a2e0 msgr2=0x7f38380afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.655+0000 7f383e100700 1 --2- 172.21.15.1:0/3242693276 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f383815a2e0 0x7f38380afaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.655+0000 7f383e100700 1 -- 172.21.15.1:0/3242693276 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38380cc170 msgr2=0x7f383809d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.655+0000 7f383e100700 1 --2- 172.21.15.1:0/3242693276 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38380cc170 0x7f383809d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.655+0000 7f383e100700 1 -- 172.21.15.1:0/3242693276 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3830009a70 con 0x7f38380ae600 2024-03-18T19:08:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.655+0000 7f383e100700 1 --2- 172.21.15.1:0/3242693276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38380ae600 0x7f38381f16a0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f383400c530 tx=0x7f383400c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.655+0000 7f382effd700 1 -- 172.21.15.1:0/3242693276 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834005d70 con 0x7f38380ae600 2024-03-18T19:08:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.655+0000 7f382effd700 1 -- 172.21.15.1:0/3242693276 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38340070d0 con 0x7f38380ae600 2024-03-18T19:08:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.655+0000 7f382effd700 1 -- 172.21.15.1:0/3242693276 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834007440 con 0x7f38380ae600 2024-03-18T19:08:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.655+0000 7f3840364700 1 -- 172.21.15.1:0/3242693276 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f383809db90 con 0x7f38380ae600 2024-03-18T19:08:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.655+0000 7f3840364700 1 -- 172.21.15.1:0/3242693276 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38381f4820 con 0x7f38380ae600 2024-03-18T19:08:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.656+0000 7f3840364700 1 -- 172.21.15.1:0/3242693276 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38380676a0 con 0x7f38380ae600 2024-03-18T19:08:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.656+0000 7f382effd700 1 -- 172.21.15.1:0/3242693276 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 56) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f38340075a0 con 0x7f38380ae600 2024-03-18T19:08:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.657+0000 7f382effd700 1 --2- 172.21.15.1:0/3242693276 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f381c06aed0 0x7f381c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.657+0000 7f383d8ff700 1 -- 172.21.15.1:0/3242693276 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f381c06aed0 msgr2=0x7f381c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.657+0000 7f382effd700 1 -- 172.21.15.1:0/3242693276 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f3834088d30 con 0x7f38380ae600 2024-03-18T19:08:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.657+0000 7f383d8ff700 1 --2- 172.21.15.1:0/3242693276 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f381c06aed0 0x7f381c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:08:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.661+0000 7f382effd700 1 -- 172.21.15.1:0/3242693276 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3834053950 con 0x7f38380ae600 2024-03-18T19:08:04.714 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:04.717+0000 7f1b3c187240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:08:04.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.858+0000 7f383d8ff700 1 -- 172.21.15.1:0/3242693276 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f381c06aed0 msgr2=0x7f381c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:04.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.858+0000 7f383d8ff700 1 --2- 172.21.15.1:0/3242693276 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f381c06aed0 0x7f381c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:08:04.876 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:04.879+0000 7f30b35e4240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:08:04.889 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:04.892+0000 7f1b3c187240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:08:04.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.909+0000 7f3840364700 1 -- 172.21.15.1:0/3242693276 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f383806da70 con 0x7f38380ae600 2024-03-18T19:08:04.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.910+0000 7f382effd700 1 -- 172.21.15.1:0/3242693276 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4733 (secure 0 0 0) 0x7f3834017020 con 0x7f38380ae600 2024-03-18T19:08:04.907 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:04.907 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6431,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/983420410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":983420410},{"type":"v1","addr":"172.21.15.1:6836","nonce":983420410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6440,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1569519191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1569519191},{"type":"v1","addr":"172.21.15.165:6834","nonce":1569519191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1528229019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1528229019},{"type":"v1","addr":"172.21.15.165:6835","nonce":1528229019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:06:41.899614+0000","modified":"2024-03-18T19:06:44.904965+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"c","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2226098738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2226098738},{"type":"v1","addr":"172.21.15.1:6837","nonce":2226098738}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-18T19:08:04.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.912+0000 7f3840364700 1 -- 172.21.15.1:0/3242693276 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f381c06aed0 msgr2=0x7f381c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:08:04.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.912+0000 7f3840364700 1 --2- 172.21.15.1:0/3242693276 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f381c06aed0 0x7f381c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.912+0000 7f3840364700 1 -- 172.21.15.1:0/3242693276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38380ae600 msgr2=0x7f38381f16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:04.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.912+0000 7f3840364700 1 --2- 172.21.15.1:0/3242693276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38380ae600 0x7f38381f16a0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f383400c530 tx=0x7f383400c840 comp rx=0 tx=0).stop 2024-03-18T19:08:04.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.912+0000 7f3840364700 1 -- 172.21.15.1:0/3242693276 shutdown_connections 2024-03-18T19:08:04.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.912+0000 7f3840364700 1 --2- 172.21.15.1:0/3242693276 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f381c06aed0 0x7f381c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.912+0000 7f3840364700 1 --2- 172.21.15.1:0/3242693276 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f383815a2e0 0x7f38380afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.912+0000 7f3840364700 1 --2- 172.21.15.1:0/3242693276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38380ae600 0x7f38381f16a0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.912+0000 7f3840364700 1 --2- 172.21.15.1:0/3242693276 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38380cc170 0x7f383809d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:04.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.912+0000 7f3840364700 1 -- 172.21.15.1:0/3242693276 >> 172.21.15.1:0/3242693276 conn(0x7f383806b860 msgr2=0x7f38380684d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:04.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.912+0000 7f3840364700 1 -- 172.21.15.1:0/3242693276 shutdown_connections 2024-03-18T19:08:04.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:04.912+0000 7f3840364700 1 -- 172.21.15.1:0/3242693276 wait complete. 2024-03-18T19:08:04.911 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 49 2024-03-18T19:08:04.926 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:08:05.039 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:05.042+0000 7f30b35e4240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:08:05.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.056+0000 7f1b9d5bd700 1 Processor -- start 2024-03-18T19:08:05.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.057+0000 7f1b9d5bd700 1 -- start start 2024-03-18T19:08:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.057+0000 7f1b9d5bd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b980d1ea0 0x7f1b98168560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.057+0000 7f1b9d5bd700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b980cddc0 con 0x7f1b98160940 2024-03-18T19:08:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.057+0000 7f1b9d5bd700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b980cdff0 con 0x7f1b980cf090 2024-03-18T19:08:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.057+0000 7f1b9d5bd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b980ce2d0 con 0x7f1b980d1ea0 2024-03-18T19:08:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.057+0000 7f1b96ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1b98160940 0x7f1b980d0190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54804/0 (socket says 172.21.15.1:54804) 2024-03-18T19:08:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.058+0000 7f1b96ffd700 1 -- 172.21.15.1:0/176385536 learned_addr learned my addr 172.21.15.1:0/176385536 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.058+0000 7f1b977fe700 1 --2- 172.21.15.1:0/176385536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b980d1ea0 0x7f1b98168560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.058+0000 7f1b95ffb700 1 -- 172.21.15.1:0/176385536 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706590966 0 0) 0x7f1b980cddc0 con 0x7f1b98160940 2024-03-18T19:08:05.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.058+0000 7f1b95ffb700 1 -- 172.21.15.1:0/176385536 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b840035b0 con 0x7f1b98160940 2024-03-18T19:08:05.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.058+0000 7f1b95ffb700 1 -- 172.21.15.1:0/176385536 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3100711737 0 0) 0x7f1b980cdff0 con 0x7f1b980cf090 2024-03-18T19:08:05.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.058+0000 7f1b95ffb700 1 -- 172.21.15.1:0/176385536 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b980cddc0 con 0x7f1b980cf090 2024-03-18T19:08:05.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.058+0000 7f1b977fe700 1 -- 172.21.15.1:0/176385536 >> v1:172.21.15.165:6789/0 conn(0x7f1b980cf090 legacy=0x7f1b980d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:05.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.058+0000 7f1b977fe700 1 -- 172.21.15.1:0/176385536 >> v1:172.21.15.1:6789/0 conn(0x7f1b98160940 legacy=0x7f1b980d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:05.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.058+0000 7f1b977fe700 1 -- 172.21.15.1:0/176385536 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b980ce5b0 con 0x7f1b980d1ea0 2024-03-18T19:08:05.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.058+0000 7f1b977fe700 1 --2- 172.21.15.1:0/176385536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b980d1ea0 0x7f1b98168560 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1b90009dc0 tx=0x7f1b9000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcbfc56cb179ade7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:05.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.058+0000 7f1b95ffb700 1 -- 172.21.15.1:0/176385536 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9000e070 con 0x7f1b980d1ea0 2024-03-18T19:08:05.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.058+0000 7f1b95ffb700 1 -- 172.21.15.1:0/176385536 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b9000bb20 con 0x7f1b980d1ea0 2024-03-18T19:08:05.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.058+0000 7f1b95ffb700 1 -- 172.21.15.1:0/176385536 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b90012430 con 0x7f1b980d1ea0 2024-03-18T19:08:05.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.059+0000 7f1b9d5bd700 1 -- 172.21.15.1:0/176385536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b980d1ea0 msgr2=0x7f1b98168560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:05.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.059+0000 7f1b9d5bd700 1 --2- 172.21.15.1:0/176385536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b980d1ea0 0x7f1b98168560 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1b90009dc0 tx=0x7f1b9000b330 comp rx=0 tx=0).stop 2024-03-18T19:08:05.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.059+0000 7f1b9d5bd700 1 -- 172.21.15.1:0/176385536 shutdown_connections 2024-03-18T19:08:05.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.059+0000 7f1b9d5bd700 1 --2- 172.21.15.1:0/176385536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b980d1ea0 0x7f1b98168560 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.059+0000 7f1b9d5bd700 1 -- 172.21.15.1:0/176385536 >> 172.21.15.1:0/176385536 conn(0x7f1b9809c520 msgr2=0x7f1b980d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:05.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.059+0000 7f1b9d5bd700 1 -- 172.21.15.1:0/176385536 shutdown_connections 2024-03-18T19:08:05.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.059+0000 7f1b9d5bd700 1 -- 172.21.15.1:0/176385536 wait complete. 2024-03-18T19:08:05.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.060+0000 7f1b9d5bd700 1 Processor -- start 2024-03-18T19:08:05.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.060+0000 7f1b9d5bd700 1 -- start start 2024-03-18T19:08:05.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.061+0000 7f1b9d5bd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b980cf090 0x7f1b98155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:05.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.061+0000 7f1b9d5bd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b980d1ea0 0x7f1b9815e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:05.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.061+0000 7f1b9d5bd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b98160940 0x7f1b9815ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:05.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.061+0000 7f1b9d5bd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b980c8470 con 0x7f1b980cf090 2024-03-18T19:08:05.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.061+0000 7f1b9d5bd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9815f250 con 0x7f1b980d1ea0 2024-03-18T19:08:05.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.061+0000 7f1b9d5bd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9815f530 con 0x7f1b98160940 2024-03-18T19:08:05.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.061+0000 7f1b96ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b980cf090 0x7f1b98155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:05.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.061+0000 7f1b96ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b980cf090 0x7f1b98155220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40128/0 (socket says 172.21.15.1:40128) 2024-03-18T19:08:05.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.061+0000 7f1b96ffd700 1 -- 172.21.15.1:0/2580024410 learned_addr learned my addr 172.21.15.1:0/2580024410 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:05.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.061+0000 7f1b977fe700 1 --2- 172.21.15.1:0/2580024410 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b98160940 0x7f1b9815ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:05.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.061+0000 7f1b967fc700 1 --2- 172.21.15.1:0/2580024410 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b980d1ea0 0x7f1b9815e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:05.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.062+0000 7f1b96ffd700 1 -- 172.21.15.1:0/2580024410 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b98160940 msgr2=0x7f1b9815ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:05.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.062+0000 7f1b96ffd700 1 --2- 172.21.15.1:0/2580024410 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b98160940 0x7f1b9815ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.062+0000 7f1b96ffd700 1 -- 172.21.15.1:0/2580024410 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b980d1ea0 msgr2=0x7f1b9815e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:05.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.062+0000 7f1b96ffd700 1 --2- 172.21.15.1:0/2580024410 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b980d1ea0 0x7f1b9815e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.062+0000 7f1b96ffd700 1 -- 172.21.15.1:0/2580024410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b90009a70 con 0x7f1b980cf090 2024-03-18T19:08:05.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.062+0000 7f1b96ffd700 1 --2- 172.21.15.1:0/2580024410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b980cf090 0x7f1b98155220 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f1b88001eb0 tx=0x7f1b8800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:05.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.062+0000 7f1b8f7fe700 1 -- 172.21.15.1:0/2580024410 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b88005d70 con 0x7f1b980cf090 2024-03-18T19:08:05.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.062+0000 7f1b8f7fe700 1 -- 172.21.15.1:0/2580024410 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b880070d0 con 0x7f1b980cf090 2024-03-18T19:08:05.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.062+0000 7f1b9d5bd700 1 -- 172.21.15.1:0/2580024410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b9815f810 con 0x7f1b980cf090 2024-03-18T19:08:05.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.062+0000 7f1b8f7fe700 1 -- 172.21.15.1:0/2580024410 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b88007460 con 0x7f1b980cf090 2024-03-18T19:08:05.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.062+0000 7f1b9d5bd700 1 -- 172.21.15.1:0/2580024410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b9815fdd0 con 0x7f1b980cf090 2024-03-18T19:08:05.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.063+0000 7f1b9d5bd700 1 -- 172.21.15.1:0/2580024410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b98073350 con 0x7f1b980cf090 2024-03-18T19:08:05.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.064+0000 7f1b8f7fe700 1 -- 172.21.15.1:0/2580024410 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 56) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f1b880075c0 con 0x7f1b980cf090 2024-03-18T19:08:05.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.064+0000 7f1b8f7fe700 1 --2- 172.21.15.1:0/2580024410 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1b7806aed0 0x7f1b7806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:05.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.064+0000 7f1b967fc700 1 -- 172.21.15.1:0/2580024410 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1b7806aed0 msgr2=0x7f1b7806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:05.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.064+0000 7f1b8f7fe700 1 -- 172.21.15.1:0/2580024410 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f1b88088e40 con 0x7f1b980cf090 2024-03-18T19:08:05.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.064+0000 7f1b967fc700 1 --2- 172.21.15.1:0/2580024410 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1b7806aed0 0x7f1b7806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:08:05.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.073+0000 7f1b8f7fe700 1 -- 172.21.15.1:0/2580024410 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1b880539b0 con 0x7f1b980cf090 2024-03-18T19:08:05.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.265+0000 7f1b967fc700 1 -- 172.21.15.1:0/2580024410 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1b7806aed0 msgr2=0x7f1b7806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:05.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.265+0000 7f1b967fc700 1 --2- 172.21.15.1:0/2580024410 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1b7806aed0 0x7f1b7806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:08:05.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.278+0000 7f1b9d5bd700 1 -- 172.21.15.1:0/2580024410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b980d0980 con 0x7f1b980cf090 2024-03-18T19:08:05.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.279+0000 7f1b8f7fe700 1 -- 172.21.15.1:0/2580024410 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16887 (secure 0 0 0) 0x7f1b88017090 con 0x7f1b980cf090 2024-03-18T19:08:05.276 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:05.276 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":120,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:07:59.090022+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:06:39.741939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-18T19:06:40.871304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.10","mappings":[{"from":1,"to":2}]},{"pgid":"14.18","mappings":[{"from":1,"to":5}]},{"pgid":"14.21","mappings":[{"from":1,"to":7}]},{"pgid":"14.29","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/1072638685":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/67600845":"2024-03-19T19:06:57.097725+0000","172.21.15.1:6833/76998":"2024-03-19T19:06:57.097725+0000","172.21.15.1:6832/76998":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/1805165487":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/1343251370":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/3890713729":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/2823326236":"2024-03-19T19:06:57.097725+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:08:05.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.281+0000 7f1b9d5bd700 1 -- 172.21.15.1:0/2580024410 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1b7806aed0 msgr2=0x7f1b7806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:08:05.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.281+0000 7f1b9d5bd700 1 --2- 172.21.15.1:0/2580024410 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1b7806aed0 0x7f1b7806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.281+0000 7f1b9d5bd700 1 -- 172.21.15.1:0/2580024410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b980cf090 msgr2=0x7f1b98155220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:05.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.281+0000 7f1b9d5bd700 1 --2- 172.21.15.1:0/2580024410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b980cf090 0x7f1b98155220 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f1b88001eb0 tx=0x7f1b8800c7b0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.281+0000 7f1b9d5bd700 1 -- 172.21.15.1:0/2580024410 shutdown_connections 2024-03-18T19:08:05.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.281+0000 7f1b9d5bd700 1 --2- 172.21.15.1:0/2580024410 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1b7806aed0 0x7f1b7806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.281+0000 7f1b9d5bd700 1 --2- 172.21.15.1:0/2580024410 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b98160940 0x7f1b9815ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.281+0000 7f1b9d5bd700 1 --2- 172.21.15.1:0/2580024410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b980cf090 0x7f1b98155220 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.281+0000 7f1b9d5bd700 1 --2- 172.21.15.1:0/2580024410 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b980d1ea0 0x7f1b9815e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.281+0000 7f1b9d5bd700 1 -- 172.21.15.1:0/2580024410 >> 172.21.15.1:0/2580024410 conn(0x7f1b9809c520 msgr2=0x7f1b9815d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:05.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.281+0000 7f1b9d5bd700 1 -- 172.21.15.1:0/2580024410 shutdown_connections 2024-03-18T19:08:05.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.281+0000 7f1b9d5bd700 1 -- 172.21.15.1:0/2580024410 wait complete. 2024-03-18T19:08:05.291 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T19:08:05.291 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:08:05.328 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:05.331+0000 7f1b3c187240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:08:05.401 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:05.405+0000 7f30b35e4240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:08:05.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.421+0000 7f205ddba700 1 Processor -- start 2024-03-18T19:08:05.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.421+0000 7f205ddba700 1 -- start start 2024-03-18T19:08:05.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.422+0000 7f205ddba700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f20580d17b0 0x7f20580d1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:05.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.422+0000 7f205ddba700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20580b83f0 0x7f20580b8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:05.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.422+0000 7f205ddba700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20580cddc0 con 0x7f205815d500 2024-03-18T19:08:05.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.422+0000 7f205ddba700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20580d21a0 con 0x7f20580d17b0 2024-03-18T19:08:05.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.422+0000 7f205ddba700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20580d3dc0 con 0x7f20580b83f0 2024-03-18T19:08:05.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.422+0000 7f2056ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f20580d17b0 0x7f20580d1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:05.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.422+0000 7f2056ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f20580d17b0 0x7f20580d1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40132/0 (socket says 172.21.15.1:40132) 2024-03-18T19:08:05.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.422+0000 7f2056ffd700 1 -- 172.21.15.1:0/1236071240 learned_addr learned my addr 172.21.15.1:0/1236071240 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:05.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.422+0000 7f2057fff700 1 --2- 172.21.15.1:0/1236071240 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20580b83f0 0x7f20580b8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:05.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.422+0000 7f2056ffd700 1 -- 172.21.15.1:0/1236071240 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20580b83f0 msgr2=0x7f20580b8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:05.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.422+0000 7f2056ffd700 1 --2- 172.21.15.1:0/1236071240 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20580b83f0 0x7f20580b8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.422+0000 7f2056ffd700 1 -- 172.21.15.1:0/1236071240 >> v1:172.21.15.165:6789/0 conn(0x7f205815d500 legacy=0x7f20580d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:05.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.422+0000 7f2056ffd700 1 -- 172.21.15.1:0/1236071240 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20580ce060 con 0x7f20580d17b0 2024-03-18T19:08:05.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.422+0000 7f2056ffd700 1 --2- 172.21.15.1:0/1236071240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20580d17b0 0x7f20580d1c60 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f204c0098f0 tx=0x7f204c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1b036aaa8ec63f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:05.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.422+0000 7f20567fc700 1 -- 172.21.15.1:0/1236071240 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204c011830 con 0x7f20580d17b0 2024-03-18T19:08:05.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.422+0000 7f20567fc700 1 -- 172.21.15.1:0/1236071240 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204c011990 con 0x7f20580d17b0 2024-03-18T19:08:05.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.422+0000 7f20567fc700 1 -- 172.21.15.1:0/1236071240 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204c011d20 con 0x7f20580d17b0 2024-03-18T19:08:05.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.423+0000 7f205ddba700 1 -- 172.21.15.1:0/1236071240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20580d17b0 msgr2=0x7f20580d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:05.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.423+0000 7f205ddba700 1 --2- 172.21.15.1:0/1236071240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20580d17b0 0x7f20580d1c60 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f204c0098f0 tx=0x7f204c009c00 comp rx=0 tx=0).stop 2024-03-18T19:08:05.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.423+0000 7f205ddba700 1 -- 172.21.15.1:0/1236071240 shutdown_connections 2024-03-18T19:08:05.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.423+0000 7f205ddba700 1 --2- 172.21.15.1:0/1236071240 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20580b83f0 0x7f20580b8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.423+0000 7f205ddba700 1 --2- 172.21.15.1:0/1236071240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20580d17b0 0x7f20580d1c60 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.423+0000 7f205ddba700 1 -- 172.21.15.1:0/1236071240 >> 172.21.15.1:0/1236071240 conn(0x7f205809c530 msgr2=0x7f20580d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:05.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.423+0000 7f205ddba700 1 -- 172.21.15.1:0/1236071240 shutdown_connections 2024-03-18T19:08:05.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.423+0000 7f205ddba700 1 -- 172.21.15.1:0/1236071240 wait complete. 2024-03-18T19:08:05.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.424+0000 7f205ddba700 1 Processor -- start 2024-03-18T19:08:05.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.425+0000 7f205ddba700 1 -- start start 2024-03-18T19:08:05.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.425+0000 7f205ddba700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20580b83f0 0x7f2058152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:05.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.425+0000 7f205ddba700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20580d17b0 0x7f20580b6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:05.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.425+0000 7f205ddba700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f205815d500 0x7f20580b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:05.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.425+0000 7f205ddba700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20580b7390 con 0x7f20580d17b0 2024-03-18T19:08:05.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.425+0000 7f205ddba700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20580b7670 con 0x7f20580b83f0 2024-03-18T19:08:05.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.425+0000 7f205ddba700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20580b7950 con 0x7f205815d500 2024-03-18T19:08:05.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.425+0000 7f2056ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20580d17b0 0x7f20580b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:05.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.425+0000 7f2056ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20580d17b0 0x7f20580b6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40136/0 (socket says 172.21.15.1:40136) 2024-03-18T19:08:05.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.425+0000 7f2056ffd700 1 -- 172.21.15.1:0/621478079 learned_addr learned my addr 172.21.15.1:0/621478079 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:05.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.425+0000 7f20577fe700 1 --2- 172.21.15.1:0/621478079 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20580b83f0 0x7f2058152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:05.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.426+0000 7f2057fff700 1 --2- 172.21.15.1:0/621478079 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f205815d500 0x7f20580b6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:05.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.426+0000 7f2056ffd700 1 -- 172.21.15.1:0/621478079 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f205815d500 msgr2=0x7f20580b6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:05.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.426+0000 7f2056ffd700 1 --2- 172.21.15.1:0/621478079 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f205815d500 0x7f20580b6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.426+0000 7f2056ffd700 1 -- 172.21.15.1:0/621478079 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20580b83f0 msgr2=0x7f2058152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:05.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.426+0000 7f2056ffd700 1 --2- 172.21.15.1:0/621478079 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20580b83f0 0x7f2058152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.426+0000 7f2056ffd700 1 -- 172.21.15.1:0/621478079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204c009550 con 0x7f20580d17b0 2024-03-18T19:08:05.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.426+0000 7f2056ffd700 1 --2- 172.21.15.1:0/621478079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20580d17b0 0x7f20580b6820 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f204c000c00 tx=0x7f204c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:05.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.426+0000 7f203ffff700 1 -- 172.21.15.1:0/621478079 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204c004400 con 0x7f20580d17b0 2024-03-18T19:08:05.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.426+0000 7f203ffff700 1 -- 172.21.15.1:0/621478079 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204c01f070 con 0x7f20580d17b0 2024-03-18T19:08:05.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.426+0000 7f203ffff700 1 -- 172.21.15.1:0/621478079 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204c0038e0 con 0x7f20580d17b0 2024-03-18T19:08:05.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.426+0000 7f205ddba700 1 -- 172.21.15.1:0/621478079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20580b7c30 con 0x7f20580d17b0 2024-03-18T19:08:05.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.426+0000 7f205ddba700 1 -- 172.21.15.1:0/621478079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2058152ac0 con 0x7f20580d17b0 2024-03-18T19:08:05.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.427+0000 7f205ddba700 1 -- 172.21.15.1:0/621478079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20580b8150 con 0x7f20580d17b0 2024-03-18T19:08:05.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.428+0000 7f203ffff700 1 -- 172.21.15.1:0/621478079 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 56) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f204c021030 con 0x7f20580d17b0 2024-03-18T19:08:05.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.428+0000 7f203ffff700 1 --2- 172.21.15.1:0/621478079 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f204406aed0 0x7f204406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:05.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.428+0000 7f20577fe700 1 -- 172.21.15.1:0/621478079 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f204406aed0 msgr2=0x7f204406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:05.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.428+0000 7f203ffff700 1 -- 172.21.15.1:0/621478079 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f204c08afe0 con 0x7f20580d17b0 2024-03-18T19:08:05.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.428+0000 7f20577fe700 1 --2- 172.21.15.1:0/621478079 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f204406aed0 0x7f204406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:08:05.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.432+0000 7f203ffff700 1 -- 172.21.15.1:0/621478079 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f204c055be0 con 0x7f20580d17b0 2024-03-18T19:08:05.490 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:05.494+0000 7f30b35e4240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:08:05.511 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:05.514+0000 7f1b3c187240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:08:05.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.629+0000 7f20577fe700 1 -- 172.21.15.1:0/621478079 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f204406aed0 msgr2=0x7f204406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:05.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.629+0000 7f20577fe700 1 --2- 172.21.15.1:0/621478079 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f204406aed0 0x7f204406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:08:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.680+0000 7f205ddba700 1 -- 172.21.15.1:0/621478079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20580b8150 con 0x7f20580d17b0 2024-03-18T19:08:05.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.680+0000 7f203ffff700 1 -- 172.21.15.1:0/621478079 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4733 (secure 0 0 0) 0x7f20580b8150 con 0x7f20580d17b0 2024-03-18T19:08:05.677 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:05.677 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6431,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/983420410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":983420410},{"type":"v1","addr":"172.21.15.1:6836","nonce":983420410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6440,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1569519191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1569519191},{"type":"v1","addr":"172.21.15.165:6834","nonce":1569519191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1528229019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1528229019},{"type":"v1","addr":"172.21.15.165:6835","nonce":1528229019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:06:41.899614+0000","modified":"2024-03-18T19:06:44.904965+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"c","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2226098738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2226098738},{"type":"v1","addr":"172.21.15.1:6837","nonce":2226098738}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-18T19:08:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.683+0000 7f205ddba700 1 -- 172.21.15.1:0/621478079 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f204406aed0 msgr2=0x7f204406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:08:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.683+0000 7f205ddba700 1 --2- 172.21.15.1:0/621478079 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f204406aed0 0x7f204406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.683+0000 7f205ddba700 1 -- 172.21.15.1:0/621478079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20580d17b0 msgr2=0x7f20580b6820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.683+0000 7f205ddba700 1 --2- 172.21.15.1:0/621478079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20580d17b0 0x7f20580b6820 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f204c000c00 tx=0x7f204c004750 comp rx=0 tx=0).stop 2024-03-18T19:08:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.683+0000 7f205ddba700 1 -- 172.21.15.1:0/621478079 shutdown_connections 2024-03-18T19:08:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.683+0000 7f205ddba700 1 --2- 172.21.15.1:0/621478079 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f204406aed0 0x7f204406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.683+0000 7f205ddba700 1 --2- 172.21.15.1:0/621478079 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f205815d500 0x7f20580b6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.683+0000 7f205ddba700 1 --2- 172.21.15.1:0/621478079 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20580b83f0 0x7f2058152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.683+0000 7f205ddba700 1 --2- 172.21.15.1:0/621478079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20580d17b0 0x7f20580b6820 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.683+0000 7f205ddba700 1 -- 172.21.15.1:0/621478079 >> 172.21.15.1:0/621478079 conn(0x7f205809c530 msgr2=0x7f20581614a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:05.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.683+0000 7f205ddba700 1 -- 172.21.15.1:0/621478079 shutdown_connections 2024-03-18T19:08:05.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.683+0000 7f205ddba700 1 -- 172.21.15.1:0/621478079 wait complete. 2024-03-18T19:08:05.681 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 49 2024-03-18T19:08:05.693 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:08:05.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.865+0000 7f8f7cc3c700 1 Processor -- start 2024-03-18T19:08:05.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.865+0000 7f8f7cc3c700 1 -- start start 2024-03-18T19:08:05.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.866+0000 7f8f7cc3c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8f780d05a0 0x7f8f7809a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:05.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.866+0000 7f8f7cc3c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f780cf7e0 con 0x7f8f7806ba70 2024-03-18T19:08:05.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.866+0000 7f8f7cc3c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f780cfac0 con 0x7f8f78099a40 2024-03-18T19:08:05.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.866+0000 7f8f7cc3c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f7806bec0 con 0x7f8f780d05a0 2024-03-18T19:08:05.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.866+0000 7f8f76d9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8f7806ba70 0x7f8f7815d940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54816/0 (socket says 172.21.15.1:54816) 2024-03-18T19:08:05.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.866+0000 7f8f76d9d700 1 -- 172.21.15.1:0/4259984453 learned_addr learned my addr 172.21.15.1:0/4259984453 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:05.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.866+0000 7f8f75d9b700 1 --2- 172.21.15.1:0/4259984453 >> v2:172.21.15.165:3300/0 conn(0x7f8f780d05a0 0x7f8f7809a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:05.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.867+0000 7f8f7559a700 1 -- 172.21.15.1:0/4259984453 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572062943 0 0) 0x7f8f780cf7e0 con 0x7f8f7806ba70 2024-03-18T19:08:05.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.867+0000 7f8f7559a700 1 -- 172.21.15.1:0/4259984453 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f600035b0 con 0x7f8f7806ba70 2024-03-18T19:08:05.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.867+0000 7f8f7559a700 1 -- 172.21.15.1:0/4259984453 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590441233 0 0) 0x7f8f780cfac0 con 0x7f8f78099a40 2024-03-18T19:08:05.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.867+0000 7f8f7559a700 1 -- 172.21.15.1:0/4259984453 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f780cf7e0 con 0x7f8f78099a40 2024-03-18T19:08:05.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.867+0000 7f8f75d9b700 1 -- 172.21.15.1:0/4259984453 >> v1:172.21.15.165:6789/0 conn(0x7f8f78099a40 legacy=0x7f8f78099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:05.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.867+0000 7f8f75d9b700 1 -- 172.21.15.1:0/4259984453 >> v1:172.21.15.1:6789/0 conn(0x7f8f7806ba70 legacy=0x7f8f7815d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:05.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.867+0000 7f8f75d9b700 1 -- 172.21.15.1:0/4259984453 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f780cfda0 con 0x7f8f780d05a0 2024-03-18T19:08:05.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.867+0000 7f8f75d9b700 1 --2- 172.21.15.1:0/4259984453 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f780d05a0 0x7f8f7809a530 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c009840 tx=0x7f8f6c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ecd149218cfad91 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:05.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.867+0000 7f8f7559a700 1 -- 172.21.15.1:0/4259984453 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c011830 con 0x7f8f780d05a0 2024-03-18T19:08:05.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.867+0000 7f8f7559a700 1 -- 172.21.15.1:0/4259984453 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6c011990 con 0x7f8f780d05a0 2024-03-18T19:08:05.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.867+0000 7f8f7559a700 1 -- 172.21.15.1:0/4259984453 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c011d20 con 0x7f8f780d05a0 2024-03-18T19:08:05.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.868+0000 7f8f7cc3c700 1 -- 172.21.15.1:0/4259984453 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f780d05a0 msgr2=0x7f8f7809a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:05.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.868+0000 7f8f7cc3c700 1 --2- 172.21.15.1:0/4259984453 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f780d05a0 0x7f8f7809a530 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c009840 tx=0x7f8f6c009b50 comp rx=0 tx=0).stop 2024-03-18T19:08:05.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.868+0000 7f8f7cc3c700 1 -- 172.21.15.1:0/4259984453 shutdown_connections 2024-03-18T19:08:05.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.868+0000 7f8f7cc3c700 1 --2- 172.21.15.1:0/4259984453 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f780d05a0 0x7f8f7809a530 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.868+0000 7f8f7cc3c700 1 -- 172.21.15.1:0/4259984453 >> 172.21.15.1:0/4259984453 conn(0x7f8f780b2cb0 msgr2=0x7f8f780b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:05.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.868+0000 7f8f7cc3c700 1 -- 172.21.15.1:0/4259984453 shutdown_connections 2024-03-18T19:08:05.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.868+0000 7f8f7cc3c700 1 -- 172.21.15.1:0/4259984453 wait complete. 2024-03-18T19:08:05.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.870+0000 7f8f7cc3c700 1 Processor -- start 2024-03-18T19:08:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.870+0000 7f8f7cc3c700 1 -- start start 2024-03-18T19:08:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.870+0000 7f8f7cc3c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f7806ba70 0x7f8f7815be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.870+0000 7f8f7cc3c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f78099a40 0x7f8f781f63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.870+0000 7f8f7cc3c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f780d05a0 0x7f8f781f6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.870+0000 7f8f7cc3c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f780cb200 con 0x7f8f78099a40 2024-03-18T19:08:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.870+0000 7f8f7cc3c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7809bd70 con 0x7f8f780d05a0 2024-03-18T19:08:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.870+0000 7f8f7cc3c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7815c360 con 0x7f8f7806ba70 2024-03-18T19:08:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.870+0000 7f8f75d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f78099a40 0x7f8f781f63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.871+0000 7f8f75d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f78099a40 0x7f8f781f63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40146/0 (socket says 172.21.15.1:40146) 2024-03-18T19:08:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.871+0000 7f8f75d9b700 1 -- 172.21.15.1:0/2477276654 learned_addr learned my addr 172.21.15.1:0/2477276654 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.871+0000 7f8f7659c700 1 --2- 172.21.15.1:0/2477276654 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f7806ba70 0x7f8f7815be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.871+0000 7f8f76d9d700 1 --2- 172.21.15.1:0/2477276654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f780d05a0 0x7f8f781f6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:05.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.871+0000 7f8f75d9b700 1 -- 172.21.15.1:0/2477276654 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f7806ba70 msgr2=0x7f8f7815be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.871+0000 7f8f75d9b700 1 --2- 172.21.15.1:0/2477276654 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f7806ba70 0x7f8f7815be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.871+0000 7f8f75d9b700 1 -- 172.21.15.1:0/2477276654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f780d05a0 msgr2=0x7f8f781f6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.871+0000 7f8f75d9b700 1 --2- 172.21.15.1:0/2477276654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f780d05a0 0x7f8f781f6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.871+0000 7f8f75d9b700 1 -- 172.21.15.1:0/2477276654 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c0094a0 con 0x7f8f78099a40 2024-03-18T19:08:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.871+0000 7f8f75d9b700 1 --2- 172.21.15.1:0/2477276654 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f78099a40 0x7f8f781f63c0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0090b0 tx=0x7f8f6c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.871+0000 7f8f66ffd700 1 -- 172.21.15.1:0/2477276654 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c01a860 con 0x7f8f78099a40 2024-03-18T19:08:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.871+0000 7f8f66ffd700 1 -- 172.21.15.1:0/2477276654 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6c01a9c0 con 0x7f8f78099a40 2024-03-18T19:08:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.871+0000 7f8f7cc3c700 1 -- 172.21.15.1:0/2477276654 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f780b3ff0 con 0x7f8f78099a40 2024-03-18T19:08:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.871+0000 7f8f66ffd700 1 -- 172.21.15.1:0/2477276654 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c01ad30 con 0x7f8f78099a40 2024-03-18T19:08:05.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.871+0000 7f8f7cc3c700 1 -- 172.21.15.1:0/2477276654 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f780cac40 con 0x7f8f78099a40 2024-03-18T19:08:05.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.872+0000 7f8f7cc3c700 1 -- 172.21.15.1:0/2477276654 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f78068240 con 0x7f8f78099a40 2024-03-18T19:08:05.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.873+0000 7f8f66ffd700 1 -- 172.21.15.1:0/2477276654 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 56) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f8f6c022070 con 0x7f8f78099a40 2024-03-18T19:08:05.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.874+0000 7f8f66ffd700 1 --2- 172.21.15.1:0/2477276654 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8f5806aed0 0x7f8f5806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:05.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.874+0000 7f8f7659c700 1 -- 172.21.15.1:0/2477276654 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8f5806aed0 msgr2=0x7f8f5806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:05.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.874+0000 7f8f66ffd700 1 -- 172.21.15.1:0/2477276654 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f8f6c08a660 con 0x7f8f78099a40 2024-03-18T19:08:05.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.874+0000 7f8f7659c700 1 --2- 172.21.15.1:0/2477276654 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8f5806aed0 0x7f8f5806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:08:05.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:05.879+0000 7f8f66ffd700 1 -- 172.21.15.1:0/2477276654 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8f6c055310 con 0x7f8f78099a40 2024-03-18T19:08:05.909 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:05.913+0000 7f1b3c187240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:08:06.006 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:06.010+0000 7f1b3c187240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:08:06.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.028+0000 7f8f66ffd700 1 -- 172.21.15.1:0/2477276654 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 57) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f8f6c0556e0 con 0x7f8f78099a40 2024-03-18T19:08:06.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.028+0000 7f8f66ffd700 1 -- 172.21.15.1:0/2477276654 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8f5806aed0 msgr2=0x7f8f5806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:08:06.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.028+0000 7f8f66ffd700 1 --2- 172.21.15.1:0/2477276654 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8f5806aed0 0x7f8f5806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.088 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:06.091+0000 7f1ab35ad700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:06.088 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:06.091+0000 7f1ab35ad700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:06.088 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:06.091+0000 7f1ab35ad700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:06.088 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:06.091+0000 7f1ab35ad700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:06.088 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:06.091+0000 7f1ab35ad700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:06.088 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:06.091+0000 7f1ab75b5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:06.088 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:06.091+0000 7f1ab75b5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:06.088 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:06.091+0000 7f1ab75b5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:06.088 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:06.091+0000 7f1ab75b5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:06.088 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:06.091+0000 7f1ab75b5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:06.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.158+0000 7f8f7cc3c700 1 -- 172.21.15.1:0/2477276654 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f7809aae0 con 0x7f8f78099a40 2024-03-18T19:08:06.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.158+0000 7f8f66ffd700 1 -- 172.21.15.1:0/2477276654 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4733 (secure 0 0 0) 0x7f8f6c05a480 con 0x7f8f78099a40 2024-03-18T19:08:06.155 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:06.155 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6431,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/983420410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":983420410},{"type":"v1","addr":"172.21.15.1:6836","nonce":983420410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6440,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1569519191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1569519191},{"type":"v1","addr":"172.21.15.165:6834","nonce":1569519191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1528229019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1528229019},{"type":"v1","addr":"172.21.15.165:6835","nonce":1528229019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:06:41.899614+0000","modified":"2024-03-18T19:06:44.904965+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"c","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2226098738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2226098738},{"type":"v1","addr":"172.21.15.1:6837","nonce":2226098738}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-18T19:08:06.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.160+0000 7f8f7cc3c700 1 -- 172.21.15.1:0/2477276654 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f78099a40 msgr2=0x7f8f781f63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:06.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.160+0000 7f8f7cc3c700 1 --2- 172.21.15.1:0/2477276654 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f78099a40 0x7f8f781f63c0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0090b0 tx=0x7f8f6c004750 comp rx=0 tx=0).stop 2024-03-18T19:08:06.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.160+0000 7f8f7cc3c700 1 -- 172.21.15.1:0/2477276654 shutdown_connections 2024-03-18T19:08:06.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.160+0000 7f8f7cc3c700 1 --2- 172.21.15.1:0/2477276654 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8f5806aed0 0x7f8f5806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.160+0000 7f8f7cc3c700 1 --2- 172.21.15.1:0/2477276654 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f7806ba70 0x7f8f7815be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.160+0000 7f8f7cc3c700 1 --2- 172.21.15.1:0/2477276654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f780d05a0 0x7f8f781f6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.160+0000 7f8f7cc3c700 1 --2- 172.21.15.1:0/2477276654 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f78099a40 0x7f8f781f63c0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.160+0000 7f8f7cc3c700 1 -- 172.21.15.1:0/2477276654 >> 172.21.15.1:0/2477276654 conn(0x7f8f780b2cb0 msgr2=0x7f8f780b3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:06.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.160+0000 7f8f7cc3c700 1 -- 172.21.15.1:0/2477276654 shutdown_connections 2024-03-18T19:08:06.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.160+0000 7f8f7cc3c700 1 -- 172.21.15.1:0/2477276654 wait complete. 2024-03-18T19:08:06.159 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 49 2024-03-18T19:08:06.169 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:08:06.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.297+0000 7fa9e0fc5700 1 Processor -- start 2024-03-18T19:08:06.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.297+0000 7fa9e0fc5700 1 -- start start 2024-03-18T19:08:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.298+0000 7fa9e0fc5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa9dc06a750 0x7fa9dc0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.298+0000 7fa9e0fc5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa9dc06ea10 0x7fa9dc06ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.298+0000 7fa9e0fc5700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9dc06ee90 con 0x7fa9dc071600 2024-03-18T19:08:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.298+0000 7fa9e0fc5700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9dc06b070 con 0x7fa9dc06a750 2024-03-18T19:08:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.298+0000 7fa9e0fc5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9dc06e1f0 con 0x7fa9dc06ea10 2024-03-18T19:08:06.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.299+0000 7fa9da59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa9dc06a750 0x7fa9dc0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.299+0000 7fa9dad9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa9dc071600 0x7fa9dc071a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54820/0 (socket says 172.21.15.1:54820) 2024-03-18T19:08:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.299+0000 7fa9dad9d700 1 -- 172.21.15.1:0/3848712281 learned_addr learned my addr 172.21.15.1:0/3848712281 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.299+0000 7fa9d9d9b700 1 --2- 172.21.15.1:0/3848712281 >> v2:172.21.15.165:3300/0 conn(0x7fa9dc06ea10 0x7fa9dc06ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.299+0000 7fa9da59c700 1 -- 172.21.15.1:0/3848712281 >> v2:172.21.15.165:3300/0 conn(0x7fa9dc06ea10 msgr2=0x7fa9dc06ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.299+0000 7fa9da59c700 1 --2- 172.21.15.1:0/3848712281 >> v2:172.21.15.165:3300/0 conn(0x7fa9dc06ea10 0x7fa9dc06ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.299+0000 7fa9da59c700 1 -- 172.21.15.1:0/3848712281 >> v1:172.21.15.1:6789/0 conn(0x7fa9dc071600 legacy=0x7fa9dc071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.299+0000 7fa9da59c700 1 -- 172.21.15.1:0/3848712281 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc06e4d0 con 0x7fa9dc06a750 2024-03-18T19:08:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.299+0000 7fa9d959a700 1 -- 172.21.15.1:0/3848712281 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226951040 0 0) 0x7fa9dc06ee90 con 0x7fa9dc071600 2024-03-18T19:08:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.299+0000 7fa9d9d9b700 1 --2- 172.21.15.1:0/3848712281 >> v2:172.21.15.165:3300/0 conn(0x7fa9dc06ea10 0x7fa9dc06ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:08:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.299+0000 7fa9da59c700 1 --2- 172.21.15.1:0/3848712281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9dc06a750 0x7fa9dc0ca5f0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc0701b0 tx=0x7fa9d0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2993dd9daf2655f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.299+0000 7fa9d959a700 1 -- 172.21.15.1:0/3848712281 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0011830 con 0x7fa9dc06a750 2024-03-18T19:08:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.299+0000 7fa9d959a700 1 -- 172.21.15.1:0/3848712281 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d0011990 con 0x7fa9dc06a750 2024-03-18T19:08:06.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.299+0000 7fa9d959a700 1 -- 172.21.15.1:0/3848712281 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0011d20 con 0x7fa9dc06a750 2024-03-18T19:08:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.300+0000 7fa9e0fc5700 1 -- 172.21.15.1:0/3848712281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9dc06a750 msgr2=0x7fa9dc0ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.300+0000 7fa9e0fc5700 1 --2- 172.21.15.1:0/3848712281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9dc06a750 0x7fa9dc0ca5f0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc0701b0 tx=0x7fa9d0009b20 comp rx=0 tx=0).stop 2024-03-18T19:08:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.300+0000 7fa9e0fc5700 1 -- 172.21.15.1:0/3848712281 shutdown_connections 2024-03-18T19:08:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.300+0000 7fa9e0fc5700 1 --2- 172.21.15.1:0/3848712281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9dc06a750 0x7fa9dc0ca5f0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.300+0000 7fa9e0fc5700 1 --2- 172.21.15.1:0/3848712281 >> v2:172.21.15.165:3300/0 conn(0x7fa9dc06ea10 0x7fa9dc06ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.300+0000 7fa9e0fc5700 1 -- 172.21.15.1:0/3848712281 >> 172.21.15.1:0/3848712281 conn(0x7fa9dc0cc990 msgr2=0x7fa9dc0ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.300+0000 7fa9e0fc5700 1 -- 172.21.15.1:0/3848712281 shutdown_connections 2024-03-18T19:08:06.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.300+0000 7fa9e0fc5700 1 -- 172.21.15.1:0/3848712281 wait complete. 2024-03-18T19:08:06.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.301+0000 7fa9e0fc5700 1 Processor -- start 2024-03-18T19:08:06.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.301+0000 7fa9e0fc5700 1 -- start start 2024-03-18T19:08:06.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.302+0000 7fa9e0fc5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9dc06a750 0x7fa9dc1597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:06.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.302+0000 7fa9e0fc5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9dc06ea10 0x7fa9dc0c85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:06.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.302+0000 7fa9e0fc5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9dc071600 0x7fa9dc0c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:06.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.302+0000 7fa9e0fc5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc1f6e70 con 0x7fa9dc06ea10 2024-03-18T19:08:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.302+0000 7fa9e0fc5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc0cf9d0 con 0x7fa9dc071600 2024-03-18T19:08:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.302+0000 7fa9e0fc5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc0ace80 con 0x7fa9dc06a750 2024-03-18T19:08:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.302+0000 7fa9d9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9dc06ea10 0x7fa9dc0c85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.302+0000 7fa9d9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9dc06ea10 0x7fa9dc0c85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40266/0 (socket says 172.21.15.1:40266) 2024-03-18T19:08:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.302+0000 7fa9d9d9b700 1 -- 172.21.15.1:0/3894005640 learned_addr learned my addr 172.21.15.1:0/3894005640 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.302+0000 7fa9dad9d700 1 --2- 172.21.15.1:0/3894005640 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9dc071600 0x7fa9dc0c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.302+0000 7fa9da59c700 1 --2- 172.21.15.1:0/3894005640 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9dc06a750 0x7fa9dc1597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.303+0000 7fa9d9d9b700 1 -- 172.21.15.1:0/3894005640 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9dc06a750 msgr2=0x7fa9dc1597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.303+0000 7fa9d9d9b700 1 --2- 172.21.15.1:0/3894005640 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9dc06a750 0x7fa9dc1597d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.303+0000 7fa9d9d9b700 1 -- 172.21.15.1:0/3894005640 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9dc071600 msgr2=0x7fa9dc0c8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.303+0000 7fa9d9d9b700 1 --2- 172.21.15.1:0/3894005640 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9dc071600 0x7fa9dc0c8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.303+0000 7fa9d9d9b700 1 -- 172.21.15.1:0/3894005640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d00094a0 con 0x7fa9dc06ea10 2024-03-18T19:08:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.303+0000 7fa9d9d9b700 1 --2- 172.21.15.1:0/3894005640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9dc06ea10 0x7fa9dc0c85b0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00c750 tx=0x7fa9cc00ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.303+0000 7fa9caffd700 1 -- 172.21.15.1:0/3894005640 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc011800 con 0x7fa9dc06ea10 2024-03-18T19:08:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.303+0000 7fa9caffd700 1 -- 172.21.15.1:0/3894005640 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9cc011960 con 0x7fa9dc06ea10 2024-03-18T19:08:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.303+0000 7fa9e0fc5700 1 -- 172.21.15.1:0/3894005640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9dc1f60b0 con 0x7fa9dc06ea10 2024-03-18T19:08:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.303+0000 7fa9caffd700 1 -- 172.21.15.1:0/3894005640 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc011cd0 con 0x7fa9dc06ea10 2024-03-18T19:08:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.303+0000 7fa9e0fc5700 1 -- 172.21.15.1:0/3894005640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9dc1f6670 con 0x7fa9dc06ea10 2024-03-18T19:08:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.304+0000 7fa9e0fc5700 1 -- 172.21.15.1:0/3894005640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9dc0676a0 con 0x7fa9dc06ea10 2024-03-18T19:08:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.304+0000 7fa9caffd700 1 -- 172.21.15.1:0/3894005640 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 57) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fa9cc021350 con 0x7fa9dc06ea10 2024-03-18T19:08:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.305+0000 7fa9caffd700 1 -- 172.21.15.1:0/3894005640 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6833+0+0 (secure 0 0 0) 0x7fa9cc088f70 con 0x7fa9dc06ea10 2024-03-18T19:08:06.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.309+0000 7fa9caffd700 1 -- 172.21.15.1:0/3894005640 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa9cc0bc580 con 0x7fa9dc06ea10 2024-03-18T19:08:06.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.512+0000 7fa9e0fc5700 1 -- 172.21.15.1:0/3894005640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9dc073fe0 con 0x7fa9dc06ea10 2024-03-18T19:08:06.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.513+0000 7fa9caffd700 1 -- 172.21.15.1:0/3894005640 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+17373 (secure 0 0 0) 0x7fa9cc01e090 con 0x7fa9dc06ea10 2024-03-18T19:08:06.510 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:06.510 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":121,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:08:06.024686+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:06:39.741939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-18T19:06:40.871304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.10","mappings":[{"from":1,"to":2}]},{"pgid":"14.18","mappings":[{"from":1,"to":5}]},{"pgid":"14.21","mappings":[{"from":1,"to":7}]},{"pgid":"14.29","mappings":[{"from":1,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3720501989":"2024-03-19T19:08:06.024643+0000","172.21.15.1:0/3609120550":"2024-03-19T19:08:06.024643+0000","172.21.15.1:0/1510974066":"2024-03-19T19:08:06.024643+0000","172.21.15.1:0/1123406931":"2024-03-19T19:08:06.024643+0000","172.21.15.1:0/423620372":"2024-03-19T19:08:06.024643+0000","172.21.15.1:0/342320400":"2024-03-19T19:08:06.024643+0000","172.21.15.1:0/1072638685":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/2264868470":"2024-03-19T19:08:06.024643+0000","172.21.15.1:0/67600845":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/2487441726":"2024-03-19T19:08:06.024643+0000","172.21.15.1:6833/76998":"2024-03-19T19:06:57.097725+0000","172.21.15.1:6832/76998":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/1805165487":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/1343251370":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/3890713729":"2024-03-19T19:06:57.097725+0000","172.21.15.1:0/2823326236":"2024-03-19T19:06:57.097725+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:08:06.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.515+0000 7fa9e0fc5700 1 -- 172.21.15.1:0/3894005640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9dc06ea10 msgr2=0x7fa9dc0c85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:06.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.515+0000 7fa9e0fc5700 1 --2- 172.21.15.1:0/3894005640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9dc06ea10 0x7fa9dc0c85b0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00c750 tx=0x7fa9cc00ca60 comp rx=0 tx=0).stop 2024-03-18T19:08:06.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.515+0000 7fa9e0fc5700 1 -- 172.21.15.1:0/3894005640 shutdown_connections 2024-03-18T19:08:06.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.515+0000 7fa9e0fc5700 1 --2- 172.21.15.1:0/3894005640 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9dc06a750 0x7fa9dc1597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.515+0000 7fa9e0fc5700 1 --2- 172.21.15.1:0/3894005640 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9dc071600 0x7fa9dc0c8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.515+0000 7fa9e0fc5700 1 --2- 172.21.15.1:0/3894005640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9dc06ea10 0x7fa9dc0c85b0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.515+0000 7fa9e0fc5700 1 -- 172.21.15.1:0/3894005640 >> 172.21.15.1:0/3894005640 conn(0x7fa9dc0cc990 msgr2=0x7fa9dc0b12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:06.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.515+0000 7fa9e0fc5700 1 -- 172.21.15.1:0/3894005640 shutdown_connections 2024-03-18T19:08:06.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.515+0000 7fa9e0fc5700 1 -- 172.21.15.1:0/3894005640 wait complete. 2024-03-18T19:08:06.525 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T19:08:06.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.654+0000 7f3947355700 1 Processor -- start 2024-03-18T19:08:06.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.655+0000 7f3947355700 1 -- start start 2024-03-18T19:08:06.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.655+0000 7f3947355700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3940071490 0x7f394015b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:06.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.655+0000 7f3947355700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f39400724e0 0x7f394015baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:06.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.655+0000 7f3947355700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394015b6f0 con 0x7f394015a380 2024-03-18T19:08:06.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.655+0000 7f3947355700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39400cb510 con 0x7f3940071490 2024-03-18T19:08:06.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.655+0000 7f3947355700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39400ca900 con 0x7f39400724e0 2024-03-18T19:08:06.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.656+0000 7f39450f1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3940071490 0x7f394015b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:06.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.656+0000 7f39448f0700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f394015a380 0x7f3940071870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54828/0 (socket says 172.21.15.1:54828) 2024-03-18T19:08:06.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.656+0000 7f39448f0700 1 -- 172.21.15.1:0/1584787127 learned_addr learned my addr 172.21.15.1:0/1584787127 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:06.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.656+0000 7f39450f1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3940071490 0x7f394015b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40276/0 (socket says 172.21.15.1:40276) 2024-03-18T19:08:06.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.656+0000 7f39458f2700 1 --2- 172.21.15.1:0/1584787127 >> v2:172.21.15.165:3300/0 conn(0x7f39400724e0 0x7f394015baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:06.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.656+0000 7f39450f1700 1 -- 172.21.15.1:0/1584787127 >> v2:172.21.15.165:3300/0 conn(0x7f39400724e0 msgr2=0x7f394015baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:06.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.656+0000 7f39450f1700 1 --2- 172.21.15.1:0/1584787127 >> v2:172.21.15.165:3300/0 conn(0x7f39400724e0 0x7f394015baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.656+0000 7f39450f1700 1 -- 172.21.15.1:0/1584787127 >> v1:172.21.15.1:6789/0 conn(0x7f394015a380 legacy=0x7f3940071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:06.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.656+0000 7f39450f1700 1 -- 172.21.15.1:0/1584787127 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39400cabe0 con 0x7f3940071490 2024-03-18T19:08:06.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.656+0000 7f3937fff700 1 -- 172.21.15.1:0/1584787127 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1613569253 0 0) 0x7f393c001110 con 0x7f394015a380 2024-03-18T19:08:06.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.656+0000 7f39450f1700 1 --2- 172.21.15.1:0/1584787127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3940071490 0x7f394015b1b0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f394015ed40 tx=0x7f3930009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d4bb05c3322bd21 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:06.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.656+0000 7f3937fff700 1 -- 172.21.15.1:0/1584787127 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930011830 con 0x7f3940071490 2024-03-18T19:08:06.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.656+0000 7f3937fff700 1 -- 172.21.15.1:0/1584787127 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3930011990 con 0x7f3940071490 2024-03-18T19:08:06.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.656+0000 7f3937fff700 1 -- 172.21.15.1:0/1584787127 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930011d20 con 0x7f3940071490 2024-03-18T19:08:06.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.657+0000 7f3947355700 1 -- 172.21.15.1:0/1584787127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3940071490 msgr2=0x7f394015b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:06.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.657+0000 7f3947355700 1 --2- 172.21.15.1:0/1584787127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3940071490 0x7f394015b1b0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f394015ed40 tx=0x7f3930009b20 comp rx=0 tx=0).stop 2024-03-18T19:08:06.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.657+0000 7f3947355700 1 -- 172.21.15.1:0/1584787127 shutdown_connections 2024-03-18T19:08:06.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.657+0000 7f3947355700 1 --2- 172.21.15.1:0/1584787127 >> v2:172.21.15.165:3300/0 conn(0x7f39400724e0 0x7f394015baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.657+0000 7f3947355700 1 --2- 172.21.15.1:0/1584787127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3940071490 0x7f394015b1b0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.657+0000 7f3947355700 1 -- 172.21.15.1:0/1584787127 >> 172.21.15.1:0/1584787127 conn(0x7f39400b1aa0 msgr2=0x7f39400b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:06.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.657+0000 7f3947355700 1 -- 172.21.15.1:0/1584787127 shutdown_connections 2024-03-18T19:08:06.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.657+0000 7f3947355700 1 -- 172.21.15.1:0/1584787127 wait complete. 2024-03-18T19:08:06.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.658+0000 7f3947355700 1 Processor -- start 2024-03-18T19:08:06.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.658+0000 7f3947355700 1 -- start start 2024-03-18T19:08:06.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.659+0000 7f3947355700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3940071490 0x7f3940159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:06.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.659+0000 7f3947355700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f39400724e0 0x7f39400af910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:06.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.659+0000 7f3947355700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f394015a380 0x7f39400afe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:06.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.659+0000 7f3947355700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394009fc00 con 0x7f3940071490 2024-03-18T19:08:06.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.659+0000 7f3947355700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3940159780 con 0x7f39400724e0 2024-03-18T19:08:06.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.659+0000 7f3947355700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39400b48d0 con 0x7f394015a380 2024-03-18T19:08:06.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.659+0000 7f39450f1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3940071490 0x7f3940159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:06.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.659+0000 7f39450f1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3940071490 0x7f3940159240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40278/0 (socket says 172.21.15.1:40278) 2024-03-18T19:08:06.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.659+0000 7f39450f1700 1 -- 172.21.15.1:0/276813935 learned_addr learned my addr 172.21.15.1:0/276813935 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:06.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.659+0000 7f39448f0700 1 --2- 172.21.15.1:0/276813935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f39400724e0 0x7f39400af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:06.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.659+0000 7f39458f2700 1 --2- 172.21.15.1:0/276813935 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f394015a380 0x7f39400afe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:06.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.659+0000 7f39450f1700 1 -- 172.21.15.1:0/276813935 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f394015a380 msgr2=0x7f39400afe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:06.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.659+0000 7f39450f1700 1 --2- 172.21.15.1:0/276813935 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f394015a380 0x7f39400afe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.659+0000 7f39450f1700 1 -- 172.21.15.1:0/276813935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f39400724e0 msgr2=0x7f39400af910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:06.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.659+0000 7f39450f1700 1 --2- 172.21.15.1:0/276813935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f39400724e0 0x7f39400af910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.659+0000 7f39450f1700 1 -- 172.21.15.1:0/276813935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39300094a0 con 0x7f3940071490 2024-03-18T19:08:06.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.659+0000 7f39448f0700 1 --2- 172.21.15.1:0/276813935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f39400724e0 0x7f39400af910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:06.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.660+0000 7f39450f1700 1 --2- 172.21.15.1:0/276813935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3940071490 0x7f3940159240 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f39300090b0 tx=0x7f3930004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:06.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.660+0000 7f3935ffb700 1 -- 172.21.15.1:0/276813935 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39300036a0 con 0x7f3940071490 2024-03-18T19:08:06.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.660+0000 7f3935ffb700 1 -- 172.21.15.1:0/276813935 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393001f070 con 0x7f3940071490 2024-03-18T19:08:06.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.660+0000 7f3947355700 1 -- 172.21.15.1:0/276813935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39400cbb20 con 0x7f3940071490 2024-03-18T19:08:06.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.660+0000 7f3935ffb700 1 -- 172.21.15.1:0/276813935 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930003a00 con 0x7f3940071490 2024-03-18T19:08:06.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.660+0000 7f3947355700 1 -- 172.21.15.1:0/276813935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39400b4e90 con 0x7f3940071490 2024-03-18T19:08:06.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.661+0000 7f3947355700 1 -- 172.21.15.1:0/276813935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39400cb980 con 0x7f3940071490 2024-03-18T19:08:06.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.661+0000 7f3935ffb700 1 -- 172.21.15.1:0/276813935 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 57) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f3930003b60 con 0x7f3940071490 2024-03-18T19:08:06.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.662+0000 7f3935ffb700 1 -- 172.21.15.1:0/276813935 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6833+0+0 (secure 0 0 0) 0x7f3930021070 con 0x7f3940071490 2024-03-18T19:08:06.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.667+0000 7f3935ffb700 1 -- 172.21.15.1:0/276813935 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f39400cb980 con 0x7f3940071490 2024-03-18T19:08:06.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.917+0000 7f3947355700 1 -- 172.21.15.1:0/276813935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f39400cb980 con 0x7f3940071490 2024-03-18T19:08:06.915 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-18T19:08:06.918+0000 7fbdbeb46700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T19:08:06.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.926+0000 7f3935ffb700 1 -- 172.21.15.1:0/276813935 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v50) v1 ==== 161+0+0 (secure 0 0 0) 0x7f39400cb980 con 0x7f3940071490 2024-03-18T19:08:06.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.928+0000 7f3947355700 1 -- 172.21.15.1:0/276813935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3940071490 msgr2=0x7f3940159240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:06.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.928+0000 7f3947355700 1 --2- 172.21.15.1:0/276813935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3940071490 0x7f3940159240 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f39300090b0 tx=0x7f3930004750 comp rx=0 tx=0).stop 2024-03-18T19:08:06.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.928+0000 7f3947355700 1 -- 172.21.15.1:0/276813935 shutdown_connections 2024-03-18T19:08:06.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.928+0000 7f3947355700 1 --2- 172.21.15.1:0/276813935 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f394015a380 0x7f39400afe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.928+0000 7f3947355700 1 --2- 172.21.15.1:0/276813935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3940071490 0x7f3940159240 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.928+0000 7f3947355700 1 --2- 172.21.15.1:0/276813935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f39400724e0 0x7f39400af910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:06.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.928+0000 7f3947355700 1 -- 172.21.15.1:0/276813935 >> 172.21.15.1:0/276813935 conn(0x7f39400b1aa0 msgr2=0x7f394006f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:06.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.928+0000 7f3947355700 1 -- 172.21.15.1:0/276813935 shutdown_connections 2024-03-18T19:08:06.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:06.928+0000 7f3947355700 1 -- 172.21.15.1:0/276813935 wait complete. 2024-03-18T19:08:06.926 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T19:08:06.927 INFO:tasks.ceph.mds.c.smithi001.stderr: -6697> 2024-03-18T19:06:42.461+0000 7f10fc1ae700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T19:08:06.938 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T19:08:06.958 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T19:08:07.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.076+0000 7ffbecfaf700 1 Processor -- start 2024-03-18T19:08:07.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.076+0000 7ffbecfaf700 1 -- start start 2024-03-18T19:08:07.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.077+0000 7ffbecfaf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbe80cfd30 0x7ffbe8141390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:07.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.077+0000 7ffbecfaf700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe815ec90 con 0x7ffbe815aee0 2024-03-18T19:08:07.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.077+0000 7ffbecfaf700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe815ef70 con 0x7ffbe815b2c0 2024-03-18T19:08:07.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.077+0000 7ffbecfaf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe81418d0 con 0x7ffbe80cfd30 2024-03-18T19:08:07.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.077+0000 7ffbe659c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ffbe815aee0 0x7ffbe80cf620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54836/0 (socket says 172.21.15.1:54836) 2024-03-18T19:08:07.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.077+0000 7ffbe659c700 1 -- 172.21.15.1:0/3718467434 learned_addr learned my addr 172.21.15.1:0/3718467434 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:07.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.077+0000 7ffbe5d9b700 1 --2- 172.21.15.1:0/3718467434 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbe80cfd30 0x7ffbe8141390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:07.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.077+0000 7ffbe559a700 1 -- 172.21.15.1:0/3718467434 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354209415 0 0) 0x7ffbe815ec90 con 0x7ffbe815aee0 2024-03-18T19:08:07.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.077+0000 7ffbe559a700 1 -- 172.21.15.1:0/3718467434 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbd00035b0 con 0x7ffbe815aee0 2024-03-18T19:08:07.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.077+0000 7ffbe559a700 1 -- 172.21.15.1:0/3718467434 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720258624 0 0) 0x7ffbe815ef70 con 0x7ffbe815b2c0 2024-03-18T19:08:07.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.077+0000 7ffbe559a700 1 -- 172.21.15.1:0/3718467434 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbe815ec90 con 0x7ffbe815b2c0 2024-03-18T19:08:07.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.077+0000 7ffbe5d9b700 1 -- 172.21.15.1:0/3718467434 >> v1:172.21.15.165:6789/0 conn(0x7ffbe815b2c0 legacy=0x7ffbe80ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:07.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.077+0000 7ffbe5d9b700 1 -- 172.21.15.1:0/3718467434 >> v1:172.21.15.1:6789/0 conn(0x7ffbe815aee0 legacy=0x7ffbe80cf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:07.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.077+0000 7ffbe5d9b700 1 -- 172.21.15.1:0/3718467434 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe815f250 con 0x7ffbe80cfd30 2024-03-18T19:08:07.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.077+0000 7ffbe5d9b700 1 --2- 172.21.15.1:0/3718467434 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbe80cfd30 0x7ffbe8141390 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc009870 tx=0x7ffbdc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=257e894696d36ba3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:07.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.078+0000 7ffbe559a700 1 -- 172.21.15.1:0/3718467434 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc011830 con 0x7ffbe80cfd30 2024-03-18T19:08:07.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.078+0000 7ffbe559a700 1 -- 172.21.15.1:0/3718467434 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbdc011990 con 0x7ffbe80cfd30 2024-03-18T19:08:07.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.078+0000 7ffbe559a700 1 -- 172.21.15.1:0/3718467434 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc011d20 con 0x7ffbe80cfd30 2024-03-18T19:08:07.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.078+0000 7ffbecfaf700 1 -- 172.21.15.1:0/3718467434 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbe80cfd30 msgr2=0x7ffbe8141390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:07.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.078+0000 7ffbecfaf700 1 --2- 172.21.15.1:0/3718467434 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbe80cfd30 0x7ffbe8141390 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc009870 tx=0x7ffbdc009b80 comp rx=0 tx=0).stop 2024-03-18T19:08:07.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.078+0000 7ffbecfaf700 1 -- 172.21.15.1:0/3718467434 shutdown_connections 2024-03-18T19:08:07.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.078+0000 7ffbecfaf700 1 --2- 172.21.15.1:0/3718467434 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbe80cfd30 0x7ffbe8141390 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:07.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.078+0000 7ffbecfaf700 1 -- 172.21.15.1:0/3718467434 >> 172.21.15.1:0/3718467434 conn(0x7ffbe8066930 msgr2=0x7ffbe806a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:07.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.078+0000 7ffbecfaf700 1 -- 172.21.15.1:0/3718467434 shutdown_connections 2024-03-18T19:08:07.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.078+0000 7ffbecfaf700 1 -- 172.21.15.1:0/3718467434 wait complete. 2024-03-18T19:08:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.080+0000 7ffbecfaf700 1 Processor -- start 2024-03-18T19:08:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.080+0000 7ffbecfaf700 1 -- start start 2024-03-18T19:08:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.080+0000 7ffbecfaf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbe80cfd30 0x7ffbe8156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.080+0000 7ffbecfaf700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffbe815aee0 0x7ffbe815d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.080+0000 7ffbecfaf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbe815b2c0 0x7ffbe812de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.080+0000 7ffbecfaf700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe815d170 con 0x7ffbe80cfd30 2024-03-18T19:08:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.080+0000 7ffbecfaf700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe8134290 con 0x7ffbe815aee0 2024-03-18T19:08:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.080+0000 7ffbecfaf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe8157990 con 0x7ffbe815b2c0 2024-03-18T19:08:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.081+0000 7ffbe659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbe80cfd30 0x7ffbe8156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.081+0000 7ffbe659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbe80cfd30 0x7ffbe8156ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40304/0 (socket says 172.21.15.1:40304) 2024-03-18T19:08:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.081+0000 7ffbe659c700 1 -- 172.21.15.1:0/3209341037 learned_addr learned my addr 172.21.15.1:0/3209341037 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:07.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.081+0000 7ffbe6d9d700 1 --2- 172.21.15.1:0/3209341037 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbe815b2c0 0x7ffbe812de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:07.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.081+0000 7ffbe5d9b700 1 --2- 172.21.15.1:0/3209341037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffbe815aee0 0x7ffbe815d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:07.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.081+0000 7ffbe659c700 1 -- 172.21.15.1:0/3209341037 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbe815b2c0 msgr2=0x7ffbe812de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:07.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.081+0000 7ffbe659c700 1 --2- 172.21.15.1:0/3209341037 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbe815b2c0 0x7ffbe812de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:07.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.081+0000 7ffbe659c700 1 -- 172.21.15.1:0/3209341037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffbe815aee0 msgr2=0x7ffbe815d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:07.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.081+0000 7ffbe659c700 1 --2- 172.21.15.1:0/3209341037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffbe815aee0 0x7ffbe815d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:07.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.081+0000 7ffbe659c700 1 -- 172.21.15.1:0/3209341037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbdc0094a0 con 0x7ffbe80cfd30 2024-03-18T19:08:07.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.081+0000 7ffbe6d9d700 1 --2- 172.21.15.1:0/3209341037 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbe815b2c0 0x7ffbe812de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:07.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.081+0000 7ffbe5d9b700 1 --2- 172.21.15.1:0/3209341037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffbe815aee0 0x7ffbe815d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:07.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.081+0000 7ffbe659c700 1 --2- 172.21.15.1:0/3209341037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbe80cfd30 0x7ffbe8156ad0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7ffbd800c700 tx=0x7ffbd800cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:07.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.081+0000 7ffbd6ffd700 1 -- 172.21.15.1:0/3209341037 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8005d70 con 0x7ffbe80cfd30 2024-03-18T19:08:07.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.081+0000 7ffbd6ffd700 1 -- 172.21.15.1:0/3209341037 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd80070d0 con 0x7ffbe80cfd30 2024-03-18T19:08:07.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.081+0000 7ffbecfaf700 1 -- 172.21.15.1:0/3209341037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe8157c70 con 0x7ffbe80cfd30 2024-03-18T19:08:07.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.081+0000 7ffbd6ffd700 1 -- 172.21.15.1:0/3209341037 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8007460 con 0x7ffbe80cfd30 2024-03-18T19:08:07.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.081+0000 7ffbecfaf700 1 -- 172.21.15.1:0/3209341037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe8133ed0 con 0x7ffbe80cfd30 2024-03-18T19:08:07.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.082+0000 7ffbecfaf700 1 -- 172.21.15.1:0/3209341037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe8134060 con 0x7ffbe80cfd30 2024-03-18T19:08:07.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.083+0000 7ffbd6ffd700 1 -- 172.21.15.1:0/3209341037 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 58) v1 ==== 88975+0+0 (secure 0 0 0) 0x7ffbd80075c0 con 0x7ffbe80cfd30 2024-03-18T19:08:07.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.083+0000 7ffbd6ffd700 1 --2- 172.21.15.1:0/3209341037 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffbc806aae0 0x7ffbc806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:07.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.083+0000 7ffbd6ffd700 1 -- 172.21.15.1:0/3209341037 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6919+0+0 (secure 0 0 0) 0x7ffbd8088a20 con 0x7ffbe80cfd30 2024-03-18T19:08:07.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.083+0000 7ffbe5d9b700 1 --2- 172.21.15.1:0/3209341037 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffbc806aae0 0x7ffbc806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:07.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.084+0000 7ffbe5d9b700 1 --2- 172.21.15.1:0/3209341037 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffbc806aae0 0x7ffbc806cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffbe80d0150 tx=0x7ffbdc01c040 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:07.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.088+0000 7ffbd6ffd700 1 -- 172.21.15.1:0/3209341037 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffbd8053a00 con 0x7ffbe80cfd30 2024-03-18T19:08:07.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.346+0000 7ffbecfaf700 1 -- 172.21.15.1:0/3209341037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ffbe80676a0 con 0x7ffbe80cfd30 2024-03-18T19:08:07.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.928+0000 7ffbd6ffd700 1 -- 172.21.15.1:0/3209341037 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v51) v1 ==== 104+0+0 (secure 0 0 0) 0x7ffbd8057020 con 0x7ffbe80cfd30 2024-03-18T19:08:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.932+0000 7ffbecfaf700 1 -- 172.21.15.1:0/3209341037 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffbc806aae0 msgr2=0x7ffbc806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.932+0000 7ffbecfaf700 1 --2- 172.21.15.1:0/3209341037 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffbc806aae0 0x7ffbc806cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffbe80d0150 tx=0x7ffbdc01c040 comp rx=0 tx=0).stop 2024-03-18T19:08:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.932+0000 7ffbecfaf700 1 -- 172.21.15.1:0/3209341037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbe80cfd30 msgr2=0x7ffbe8156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.932+0000 7ffbecfaf700 1 --2- 172.21.15.1:0/3209341037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbe80cfd30 0x7ffbe8156ad0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7ffbd800c700 tx=0x7ffbd800cac0 comp rx=0 tx=0).stop 2024-03-18T19:08:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.932+0000 7ffbecfaf700 1 -- 172.21.15.1:0/3209341037 shutdown_connections 2024-03-18T19:08:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.932+0000 7ffbecfaf700 1 --2- 172.21.15.1:0/3209341037 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ffbc806aae0 0x7ffbc806cfa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.932+0000 7ffbecfaf700 1 --2- 172.21.15.1:0/3209341037 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbe815b2c0 0x7ffbe812de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.932+0000 7ffbecfaf700 1 --2- 172.21.15.1:0/3209341037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbe80cfd30 0x7ffbe8156ad0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.932+0000 7ffbecfaf700 1 --2- 172.21.15.1:0/3209341037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffbe815aee0 0x7ffbe815d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:07.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.932+0000 7ffbecfaf700 1 -- 172.21.15.1:0/3209341037 >> 172.21.15.1:0/3209341037 conn(0x7ffbe8066930 msgr2=0x7ffbe806df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:07.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.932+0000 7ffbecfaf700 1 -- 172.21.15.1:0/3209341037 shutdown_connections 2024-03-18T19:08:07.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:07.932+0000 7ffbecfaf700 1 -- 172.21.15.1:0/3209341037 wait complete. 2024-03-18T19:08:07.944 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T19:08:08.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.128+0000 7f1961c0c700 1 Processor -- start 2024-03-18T19:08:08.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.128+0000 7f1961c0c700 1 -- start start 2024-03-18T19:08:08.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.129+0000 7f1961c0c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f195c06ae60 0x7f195c0af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:08.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.129+0000 7f1961c0c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f195c161b60 0x7f195c0b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:08.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.129+0000 7f1961c0c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195c15dd70 con 0x7f195c15d540 2024-03-18T19:08:08.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.129+0000 7f1961c0c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f195c162100 con 0x7f195c06ae60 2024-03-18T19:08:08.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.129+0000 7f1961c0c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f195c0af630 con 0x7f195c161b60 2024-03-18T19:08:08.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.129+0000 7f195bfff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f195c161b60 0x7f195c0b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:08.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.129+0000 7f195affd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f195c06ae60 0x7f195c0af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:08.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.129+0000 7f195bfff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f195c161b60 0x7f195c0b2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:60370/0 (socket says 172.21.15.1:60370) 2024-03-18T19:08:08.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.129+0000 7f195bfff700 1 -- 172.21.15.1:0/4104489155 learned_addr learned my addr 172.21.15.1:0/4104489155 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:08.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.130+0000 7f195affd700 1 -- 172.21.15.1:0/4104489155 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f195c161b60 msgr2=0x7f195c0b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:08.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.130+0000 7f195affd700 1 --2- 172.21.15.1:0/4104489155 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f195c161b60 0x7f195c0b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:08.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.130+0000 7f195affd700 1 -- 172.21.15.1:0/4104489155 >> v1:172.21.15.165:6789/0 conn(0x7f195c15d540 legacy=0x7f195c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:08.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.130+0000 7f195affd700 1 -- 172.21.15.1:0/4104489155 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195c15eaf0 con 0x7f195c06ae60 2024-03-18T19:08:08.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.130+0000 7f195affd700 1 --2- 172.21.15.1:0/4104489155 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f195c06ae60 0x7f195c0af0f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1950009870 tx=0x7f1950009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=529df38eb54b6c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:08.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.130+0000 7f195a7fc700 1 -- 172.21.15.1:0/4104489155 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1950011830 con 0x7f195c06ae60 2024-03-18T19:08:08.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.130+0000 7f195a7fc700 1 -- 172.21.15.1:0/4104489155 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1950011990 con 0x7f195c06ae60 2024-03-18T19:08:08.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.130+0000 7f195a7fc700 1 -- 172.21.15.1:0/4104489155 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1950011d20 con 0x7f195c06ae60 2024-03-18T19:08:08.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.131+0000 7f1961c0c700 1 -- 172.21.15.1:0/4104489155 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f195c06ae60 msgr2=0x7f195c0af0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:08.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.131+0000 7f1961c0c700 1 --2- 172.21.15.1:0/4104489155 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f195c06ae60 0x7f195c0af0f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1950009870 tx=0x7f1950009b80 comp rx=0 tx=0).stop 2024-03-18T19:08:08.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.131+0000 7f1961c0c700 1 -- 172.21.15.1:0/4104489155 shutdown_connections 2024-03-18T19:08:08.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.131+0000 7f1961c0c700 1 --2- 172.21.15.1:0/4104489155 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f195c161b60 0x7f195c0b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:08.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.131+0000 7f1961c0c700 1 --2- 172.21.15.1:0/4104489155 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f195c06ae60 0x7f195c0af0f0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:08.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.131+0000 7f1961c0c700 1 -- 172.21.15.1:0/4104489155 >> 172.21.15.1:0/4104489155 conn(0x7f195c0b1cb0 msgr2=0x7f195c0b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:08.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.131+0000 7f1961c0c700 1 -- 172.21.15.1:0/4104489155 shutdown_connections 2024-03-18T19:08:08.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.131+0000 7f1961c0c700 1 -- 172.21.15.1:0/4104489155 wait complete. 2024-03-18T19:08:08.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.132+0000 7f1961c0c700 1 Processor -- start 2024-03-18T19:08:08.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.132+0000 7f1961c0c700 1 -- start start 2024-03-18T19:08:08.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.133+0000 7f1961c0c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f195c06ae60 0x7f195c0ae010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:08.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.133+0000 7f1961c0c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f195c15d540 0x7f195c0cd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:08.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.133+0000 7f1961c0c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f195c161b60 0x7f195c14ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:08.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.133+0000 7f1961c0c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c0cd550 con 0x7f195c06ae60 2024-03-18T19:08:08.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.133+0000 7f1961c0c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c0cd7d0 con 0x7f195c161b60 2024-03-18T19:08:08.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.133+0000 7f1961c0c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f195c0cdab0 con 0x7f195c15d540 2024-03-18T19:08:08.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.133+0000 7f195b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f195c06ae60 0x7f195c0ae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:08.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.133+0000 7f195b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f195c06ae60 0x7f195c0ae010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40326/0 (socket says 172.21.15.1:40326) 2024-03-18T19:08:08.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.133+0000 7f195b7fe700 1 -- 172.21.15.1:0/3197517394 learned_addr learned my addr 172.21.15.1:0/3197517394 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:08.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.133+0000 7f195affd700 1 --2- 172.21.15.1:0/3197517394 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f195c15d540 0x7f195c0cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:08.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.133+0000 7f195bfff700 1 --2- 172.21.15.1:0/3197517394 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f195c161b60 0x7f195c14ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:08.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.133+0000 7f195b7fe700 1 -- 172.21.15.1:0/3197517394 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f195c15d540 msgr2=0x7f195c0cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:08.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.133+0000 7f195b7fe700 1 --2- 172.21.15.1:0/3197517394 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f195c15d540 0x7f195c0cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:08.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.133+0000 7f195b7fe700 1 -- 172.21.15.1:0/3197517394 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f195c161b60 msgr2=0x7f195c14ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:08.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.133+0000 7f195b7fe700 1 --2- 172.21.15.1:0/3197517394 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f195c161b60 0x7f195c14ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:08.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.133+0000 7f195b7fe700 1 -- 172.21.15.1:0/3197517394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19500094a0 con 0x7f195c06ae60 2024-03-18T19:08:08.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.134+0000 7f195b7fe700 1 --2- 172.21.15.1:0/3197517394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f195c06ae60 0x7f195c0ae010 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f1954001ed0 tx=0x7f19540021e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:08.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.134+0000 7f1943fff700 1 -- 172.21.15.1:0/3197517394 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954003c00 con 0x7f195c06ae60 2024-03-18T19:08:08.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.134+0000 7f1943fff700 1 -- 172.21.15.1:0/3197517394 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1954003d60 con 0x7f195c06ae60 2024-03-18T19:08:08.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.134+0000 7f1961c0c700 1 -- 172.21.15.1:0/3197517394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f195c0cdd90 con 0x7f195c06ae60 2024-03-18T19:08:08.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.134+0000 7f1943fff700 1 -- 172.21.15.1:0/3197517394 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195400ab40 con 0x7f195c06ae60 2024-03-18T19:08:08.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.134+0000 7f1961c0c700 1 -- 172.21.15.1:0/3197517394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f195c0ce350 con 0x7f195c06ae60 2024-03-18T19:08:08.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.135+0000 7f1961c0c700 1 -- 172.21.15.1:0/3197517394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f195c0676a0 con 0x7f195c06ae60 2024-03-18T19:08:08.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.135+0000 7f1943fff700 1 -- 172.21.15.1:0/3197517394 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 58) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f195400aca0 con 0x7f195c06ae60 2024-03-18T19:08:08.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.135+0000 7f1943fff700 1 --2- 172.21.15.1:0/3197517394 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f194806aae0 0x7f194806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:08.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.136+0000 7f1943fff700 1 -- 172.21.15.1:0/3197517394 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6919+0+0 (secure 0 0 0) 0x7f1954089470 con 0x7f195c06ae60 2024-03-18T19:08:08.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.136+0000 7f195affd700 1 --2- 172.21.15.1:0/3197517394 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f194806aae0 0x7f194806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:08.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.136+0000 7f195affd700 1 --2- 172.21.15.1:0/3197517394 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f194806aae0 0x7f194806cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f19500090b0 tx=0x7f1950008030 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:08.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.140+0000 7f1943fff700 1 -- 172.21.15.1:0/3197517394 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f19540544d0 con 0x7f195c06ae60 2024-03-18T19:08:08.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:08.342+0000 7f1961c0c700 1 -- 172.21.15.1:0/3197517394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f195c06b430 con 0x7f195c06ae60 2024-03-18T19:08:09.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.042+0000 7f1943fff700 1 -- 172.21.15.1:0/3197517394 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 59) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f1954010dc0 con 0x7f195c06ae60 2024-03-18T19:08:09.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.110+0000 7f1943fff700 1 -- 172.21.15.1:0/3197517394 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v123) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1954057f60 con 0x7f195c06ae60 2024-03-18T19:08:09.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.113+0000 7f1961c0c700 1 -- 172.21.15.1:0/3197517394 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f194806aae0 msgr2=0x7f194806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:09.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.113+0000 7f1961c0c700 1 --2- 172.21.15.1:0/3197517394 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f194806aae0 0x7f194806cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f19500090b0 tx=0x7f1950008030 comp rx=0 tx=0).stop 2024-03-18T19:08:09.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.113+0000 7f1961c0c700 1 -- 172.21.15.1:0/3197517394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f195c06ae60 msgr2=0x7f195c0ae010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:09.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.113+0000 7f1961c0c700 1 --2- 172.21.15.1:0/3197517394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f195c06ae60 0x7f195c0ae010 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f1954001ed0 tx=0x7f19540021e0 comp rx=0 tx=0).stop 2024-03-18T19:08:09.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.113+0000 7f1961c0c700 1 -- 172.21.15.1:0/3197517394 shutdown_connections 2024-03-18T19:08:09.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.113+0000 7f1961c0c700 1 --2- 172.21.15.1:0/3197517394 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f194806aae0 0x7f194806cfa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:09.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.113+0000 7f1961c0c700 1 --2- 172.21.15.1:0/3197517394 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f195c161b60 0x7f195c14ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:09.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.113+0000 7f1961c0c700 1 --2- 172.21.15.1:0/3197517394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f195c06ae60 0x7f195c0ae010 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:09.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.113+0000 7f1961c0c700 1 --2- 172.21.15.1:0/3197517394 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f195c15d540 0x7f195c0cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:09.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.113+0000 7f1961c0c700 1 -- 172.21.15.1:0/3197517394 >> 172.21.15.1:0/3197517394 conn(0x7f195c0b1cb0 msgr2=0x7f195c1601c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:09.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.114+0000 7f1961c0c700 1 -- 172.21.15.1:0/3197517394 shutdown_connections 2024-03-18T19:08:09.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.114+0000 7f1961c0c700 1 -- 172.21.15.1:0/3197517394 wait complete. 2024-03-18T19:08:09.121 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-18T19:08:09.128 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T19:08:09.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.267+0000 7efcb0b94700 1 Processor -- start 2024-03-18T19:08:09.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.267+0000 7efcb0b94700 1 -- start start 2024-03-18T19:08:09.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.268+0000 7efcb0b94700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7efcac15af20 0x7efcac072480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:09.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.268+0000 7efcb0b94700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcac0af6b0 0x7efcac15dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:09.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.268+0000 7efcb0b94700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcac15e860 con 0x7efcac15b300 2024-03-18T19:08:09.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.268+0000 7efcb0b94700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcac0719a0 con 0x7efcac15af20 2024-03-18T19:08:09.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.268+0000 7efcb0b94700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcac071c80 con 0x7efcac0af6b0 2024-03-18T19:08:09.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.268+0000 7efcaad9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcac0af6b0 0x7efcac15dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:09.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.268+0000 7efca9d9b700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7efcac15b300 0x7efcac0aefa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:40204/0 (socket says 172.21.15.1:40204) 2024-03-18T19:08:09.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.268+0000 7efcaa59c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7efcac15af20 0x7efcac072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:09.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.268+0000 7efcaad9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcac0af6b0 0x7efcac15dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:60384/0 (socket says 172.21.15.1:60384) 2024-03-18T19:08:09.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.268+0000 7efca9d9b700 1 -- 172.21.15.1:0/1477274089 learned_addr learned my addr 172.21.15.1:0/1477274089 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:09.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.269+0000 7efcaad9d700 1 -- 172.21.15.1:0/1477274089 >> v2:172.21.15.165:3300/0 conn(0x7efcac15af20 msgr2=0x7efcac072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:09.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.269+0000 7efcaad9d700 1 --2- 172.21.15.1:0/1477274089 >> v2:172.21.15.165:3300/0 conn(0x7efcac15af20 0x7efcac072480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:09.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.269+0000 7efcaad9d700 1 -- 172.21.15.1:0/1477274089 >> v1:172.21.15.165:6789/0 conn(0x7efcac15b300 legacy=0x7efcac0aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:09.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.269+0000 7efcaad9d700 1 -- 172.21.15.1:0/1477274089 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcac071f60 con 0x7efcac0af6b0 2024-03-18T19:08:09.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.269+0000 7efcaa59c700 1 --2- 172.21.15.1:0/1477274089 >> v2:172.21.15.165:3300/0 conn(0x7efcac15af20 0x7efcac072480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:08:09.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.269+0000 7efcaad9d700 1 --2- 172.21.15.1:0/1477274089 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcac0af6b0 0x7efcac15dac0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efca4009dc0 tx=0x7efca400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcba1230c51c2a89 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:09.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.269+0000 7efca959a700 1 -- 172.21.15.1:0/1477274089 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca400e070 con 0x7efcac0af6b0 2024-03-18T19:08:09.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.269+0000 7efca959a700 1 -- 172.21.15.1:0/1477274089 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca400bae0 con 0x7efcac0af6b0 2024-03-18T19:08:09.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.269+0000 7efca959a700 1 -- 172.21.15.1:0/1477274089 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca400be90 con 0x7efcac0af6b0 2024-03-18T19:08:09.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.269+0000 7efcb0b94700 1 -- 172.21.15.1:0/1477274089 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcac0af6b0 msgr2=0x7efcac15dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:09.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.269+0000 7efcb0b94700 1 --2- 172.21.15.1:0/1477274089 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcac0af6b0 0x7efcac15dac0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efca4009dc0 tx=0x7efca400b330 comp rx=0 tx=0).stop 2024-03-18T19:08:09.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.270+0000 7efcb0b94700 1 -- 172.21.15.1:0/1477274089 shutdown_connections 2024-03-18T19:08:09.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.270+0000 7efcb0b94700 1 --2- 172.21.15.1:0/1477274089 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcac0af6b0 0x7efcac15dac0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:09.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.270+0000 7efcb0b94700 1 --2- 172.21.15.1:0/1477274089 >> v2:172.21.15.165:3300/0 conn(0x7efcac15af20 0x7efcac072480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:09.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.270+0000 7efcb0b94700 1 -- 172.21.15.1:0/1477274089 >> 172.21.15.1:0/1477274089 conn(0x7efcac0d00a0 msgr2=0x7efcac0d04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:09.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.270+0000 7efcb0b94700 1 -- 172.21.15.1:0/1477274089 shutdown_connections 2024-03-18T19:08:09.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.270+0000 7efcb0b94700 1 -- 172.21.15.1:0/1477274089 wait complete. 2024-03-18T19:08:09.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.271+0000 7efcb0b94700 1 Processor -- start 2024-03-18T19:08:09.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.271+0000 7efcb0b94700 1 -- start start 2024-03-18T19:08:09.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efcb0b94700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcac0af6b0 0x7efcac0ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:09.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efcb0b94700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcac15af20 0x7efcac1429c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:09.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efcb0b94700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcac15b300 0x7efcac0aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:09.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efcb0b94700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac142f00 con 0x7efcac15af20 2024-03-18T19:08:09.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efcb0b94700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac155fd0 con 0x7efcac0af6b0 2024-03-18T19:08:09.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efcb0b94700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcac1553c0 con 0x7efcac15b300 2024-03-18T19:08:09.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efca9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcac15af20 0x7efcac1429c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:09.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efca9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcac15af20 0x7efcac1429c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40330/0 (socket says 172.21.15.1:40330) 2024-03-18T19:08:09.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efca9d9b700 1 -- 172.21.15.1:0/3554384720 learned_addr learned my addr 172.21.15.1:0/3554384720 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:09.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efcaad9d700 1 --2- 172.21.15.1:0/3554384720 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcac15b300 0x7efcac0aebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:09.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efcaa59c700 1 --2- 172.21.15.1:0/3554384720 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcac0af6b0 0x7efcac0ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:09.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efca9d9b700 1 -- 172.21.15.1:0/3554384720 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcac15b300 msgr2=0x7efcac0aebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:09.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efca9d9b700 1 --2- 172.21.15.1:0/3554384720 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcac15b300 0x7efcac0aebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:09.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efca9d9b700 1 -- 172.21.15.1:0/3554384720 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcac0af6b0 msgr2=0x7efcac0ad2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:09.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efca9d9b700 1 --2- 172.21.15.1:0/3554384720 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcac0af6b0 0x7efcac0ad2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:09.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efca9d9b700 1 -- 172.21.15.1:0/3554384720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca00094a0 con 0x7efcac15af20 2024-03-18T19:08:09.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efcaa59c700 1 --2- 172.21.15.1:0/3554384720 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcac0af6b0 0x7efcac0ad2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:08:09.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efca9d9b700 1 --2- 172.21.15.1:0/3554384720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcac15af20 0x7efcac1429c0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00c480 tx=0x7efc9c00c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:09.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efc9affd700 1 -- 172.21.15.1:0/3554384720 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c005d70 con 0x7efcac15af20 2024-03-18T19:08:09.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efc9affd700 1 -- 172.21.15.1:0/3554384720 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9c0070d0 con 0x7efcac15af20 2024-03-18T19:08:09.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efc9affd700 1 -- 172.21.15.1:0/3554384720 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c007440 con 0x7efcac15af20 2024-03-18T19:08:09.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.272+0000 7efcb0b94700 1 -- 172.21.15.1:0/3554384720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca4009a70 con 0x7efcac15af20 2024-03-18T19:08:09.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.273+0000 7efcb0b94700 1 -- 172.21.15.1:0/3554384720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcac155980 con 0x7efcac15af20 2024-03-18T19:08:09.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.273+0000 7efcb0b94700 1 -- 172.21.15.1:0/3554384720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcac070270 con 0x7efcac15af20 2024-03-18T19:08:09.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.274+0000 7efc9affd700 1 -- 172.21.15.1:0/3554384720 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 59) v1 ==== 89183+0+0 (secure 0 0 0) 0x7efc9c0075a0 con 0x7efcac15af20 2024-03-18T19:08:09.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.274+0000 7efc9affd700 1 --2- 172.21.15.1:0/3554384720 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efc8c06ad20 0x7efc8c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:09.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.275+0000 7efcaa59c700 1 --2- 172.21.15.1:0/3554384720 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efc8c06ad20 0x7efc8c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:09.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.275+0000 7efc9affd700 1 -- 172.21.15.1:0/3554384720 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6481+0+0 (secure 0 0 0) 0x7efc9c0888b0 con 0x7efcac15af20 2024-03-18T19:08:09.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.275+0000 7efcaa59c700 1 --2- 172.21.15.1:0/3554384720 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efc8c06ad20 0x7efc8c06d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efca0009cf0 tx=0x7efca0008040 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:09.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.280+0000 7efc9affd700 1 -- 172.21.15.1:0/3554384720 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efc9c0538d0 con 0x7efcac15af20 2024-03-18T19:08:09.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:09.484+0000 7efcb0b94700 1 -- 172.21.15.1:0/3554384720 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efcac0676a0 con 0x7efcac15af20 2024-03-18T19:08:10.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.133+0000 7efc9affd700 1 -- 172.21.15.1:0/3554384720 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7efc9c053ca0 con 0x7efcac15af20 2024-03-18T19:08:10.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.134+0000 7efc9affd700 1 -- 172.21.15.1:0/3554384720 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v124) v1 ==== 169+0+0 (secure 0 0 0) 0x7efc9c0509e0 con 0x7efcac15af20 2024-03-18T19:08:10.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.137+0000 7efcb0b94700 1 -- 172.21.15.1:0/3554384720 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efc8c06ad20 msgr2=0x7efc8c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:10.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.137+0000 7efcb0b94700 1 --2- 172.21.15.1:0/3554384720 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efc8c06ad20 0x7efc8c06d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efca0009cf0 tx=0x7efca0008040 comp rx=0 tx=0).stop 2024-03-18T19:08:10.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.137+0000 7efcb0b94700 1 -- 172.21.15.1:0/3554384720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcac15af20 msgr2=0x7efcac1429c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.137+0000 7efcb0b94700 1 --2- 172.21.15.1:0/3554384720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcac15af20 0x7efcac1429c0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00c480 tx=0x7efc9c00c840 comp rx=0 tx=0).stop 2024-03-18T19:08:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.137+0000 7efcb0b94700 1 -- 172.21.15.1:0/3554384720 shutdown_connections 2024-03-18T19:08:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.137+0000 7efcb0b94700 1 --2- 172.21.15.1:0/3554384720 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efc8c06ad20 0x7efc8c06d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.137+0000 7efcb0b94700 1 --2- 172.21.15.1:0/3554384720 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcac15b300 0x7efcac0aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.137+0000 7efcb0b94700 1 --2- 172.21.15.1:0/3554384720 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcac0af6b0 0x7efcac0ad2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.137+0000 7efcb0b94700 1 --2- 172.21.15.1:0/3554384720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcac15af20 0x7efcac1429c0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.137+0000 7efcb0b94700 1 -- 172.21.15.1:0/3554384720 >> 172.21.15.1:0/3554384720 conn(0x7efcac0d00a0 msgr2=0x7efcac06f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.137+0000 7efcb0b94700 1 -- 172.21.15.1:0/3554384720 shutdown_connections 2024-03-18T19:08:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.137+0000 7efcb0b94700 1 -- 172.21.15.1:0/3554384720 wait complete. 2024-03-18T19:08:10.135 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-18T19:08:10.151 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-18T19:08:10.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.285+0000 7f49f6c1a700 1 Processor -- start 2024-03-18T19:08:10.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.285+0000 7f49f6c1a700 1 -- start start 2024-03-18T19:08:10.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.286+0000 7f49f6c1a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f00cfb50 0x7f49f00d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:10.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.286+0000 7f49f6c1a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f49f00cff30 0x7f49f00b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:10.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.286+0000 7f49f6c1a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f49f00b3d40 0x7f49f00b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:10.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.286+0000 7f49f6c1a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49f00b3370 con 0x7f49f00b3d40 2024-03-18T19:08:10.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.286+0000 7f49f6c1a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49f00b2760 con 0x7f49f00cff30 2024-03-18T19:08:10.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.286+0000 7f49f6c1a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f00b2a40 con 0x7f49f00cfb50 2024-03-18T19:08:10.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.286+0000 7f49f51b7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f49f00b3d40 0x7f49f00b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.286+0000 7f49f51b7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f49f00b3d40 0x7f49f00b41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40344/0 (socket says 172.21.15.1:40344) 2024-03-18T19:08:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.286+0000 7f49f51b7700 1 -- 172.21.15.1:0/3484992693 learned_addr learned my addr 172.21.15.1:0/3484992693 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.286+0000 7f49effff700 1 --2- 172.21.15.1:0/3484992693 >> v2:172.21.15.165:3300/0 conn(0x7f49f00cff30 0x7f49f00b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.286+0000 7f49f49b6700 1 --2- 172.21.15.1:0/3484992693 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f00cfb50 0x7f49f00d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.286+0000 7f49f51b7700 1 -- 172.21.15.1:0/3484992693 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f00cfb50 msgr2=0x7f49f00d3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.286+0000 7f49f51b7700 1 --2- 172.21.15.1:0/3484992693 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f00cfb50 0x7f49f00d3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.286+0000 7f49f51b7700 1 -- 172.21.15.1:0/3484992693 >> v2:172.21.15.165:3300/0 conn(0x7f49f00cff30 msgr2=0x7f49f00b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.286+0000 7f49f51b7700 1 --2- 172.21.15.1:0/3484992693 >> v2:172.21.15.165:3300/0 conn(0x7f49f00cff30 0x7f49f00b3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.286+0000 7f49f51b7700 1 -- 172.21.15.1:0/3484992693 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f00b2d20 con 0x7f49f00b3d40 2024-03-18T19:08:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.286+0000 7f49effff700 1 --2- 172.21.15.1:0/3484992693 >> v2:172.21.15.165:3300/0 conn(0x7f49f00cff30 0x7f49f00b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.286+0000 7f49f49b6700 1 --2- 172.21.15.1:0/3484992693 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f00cfb50 0x7f49f00d3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.287+0000 7f49f51b7700 1 --2- 172.21.15.1:0/3484992693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f00b3d40 0x7f49f00b41d0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f49f00b5cc0 tx=0x7f49e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc38842813fa5f97 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.287+0000 7f49ef7fe700 1 -- 172.21.15.1:0/3484992693 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e800e070 con 0x7f49f00b3d40 2024-03-18T19:08:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.287+0000 7f49ef7fe700 1 -- 172.21.15.1:0/3484992693 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e800b9c0 con 0x7f49f00b3d40 2024-03-18T19:08:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.287+0000 7f49ef7fe700 1 -- 172.21.15.1:0/3484992693 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e800bd70 con 0x7f49f00b3d40 2024-03-18T19:08:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.287+0000 7f49f6c1a700 1 -- 172.21.15.1:0/3484992693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f00b3d40 msgr2=0x7f49f00b41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.287+0000 7f49f6c1a700 1 --2- 172.21.15.1:0/3484992693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f00b3d40 0x7f49f00b41d0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f49f00b5cc0 tx=0x7f49e800b330 comp rx=0 tx=0).stop 2024-03-18T19:08:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.287+0000 7f49f6c1a700 1 -- 172.21.15.1:0/3484992693 shutdown_connections 2024-03-18T19:08:10.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.287+0000 7f49f6c1a700 1 --2- 172.21.15.1:0/3484992693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f00b3d40 0x7f49f00b41d0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.287+0000 7f49f6c1a700 1 --2- 172.21.15.1:0/3484992693 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f00cfb50 0x7f49f00d3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.287+0000 7f49f6c1a700 1 --2- 172.21.15.1:0/3484992693 >> v2:172.21.15.165:3300/0 conn(0x7f49f00cff30 0x7f49f00b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.287+0000 7f49f6c1a700 1 -- 172.21.15.1:0/3484992693 >> 172.21.15.1:0/3484992693 conn(0x7f49f00d3140 msgr2=0x7f49f00d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:10.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.287+0000 7f49f6c1a700 1 -- 172.21.15.1:0/3484992693 shutdown_connections 2024-03-18T19:08:10.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.287+0000 7f49f6c1a700 1 -- 172.21.15.1:0/3484992693 wait complete. 2024-03-18T19:08:10.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.289+0000 7f49f6c1a700 1 Processor -- start 2024-03-18T19:08:10.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.289+0000 7f49f6c1a700 1 -- start start 2024-03-18T19:08:10.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.290+0000 7f49f6c1a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f00b3d40 0x7f49f00ca210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:10.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.290+0000 7f49f6c1a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f00cfb50 0x7f49f00ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:10.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.290+0000 7f49f6c1a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49f00cff30 0x7f49f00b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:10.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.290+0000 7f49f6c1a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f00cbb80 con 0x7f49f00b3d40 2024-03-18T19:08:10.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.290+0000 7f49f6c1a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f00cac90 con 0x7f49f00cff30 2024-03-18T19:08:10.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.290+0000 7f49f6c1a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f00caf00 con 0x7f49f00cfb50 2024-03-18T19:08:10.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.290+0000 7f49f49b6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f00b3d40 0x7f49f00ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:10.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.290+0000 7f49f49b6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f00b3d40 0x7f49f00ca210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40360/0 (socket says 172.21.15.1:40360) 2024-03-18T19:08:10.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.290+0000 7f49f49b6700 1 -- 172.21.15.1:0/1831597938 learned_addr learned my addr 172.21.15.1:0/1831597938 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:10.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.290+0000 7f49effff700 1 --2- 172.21.15.1:0/1831597938 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f00cfb50 0x7f49f00ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:10.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.290+0000 7f49f51b7700 1 --2- 172.21.15.1:0/1831597938 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49f00cff30 0x7f49f00b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:10.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.290+0000 7f49f49b6700 1 -- 172.21.15.1:0/1831597938 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f00cfb50 msgr2=0x7f49f00ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:10.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.290+0000 7f49f49b6700 1 --2- 172.21.15.1:0/1831597938 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f00cfb50 0x7f49f00ca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.290+0000 7f49f49b6700 1 -- 172.21.15.1:0/1831597938 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49f00cff30 msgr2=0x7f49f00b6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:10.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.290+0000 7f49f49b6700 1 --2- 172.21.15.1:0/1831597938 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49f00cff30 0x7f49f00b6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.290+0000 7f49f49b6700 1 -- 172.21.15.1:0/1831597938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e8009b30 con 0x7f49f00b3d40 2024-03-18T19:08:10.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.291+0000 7f49f49b6700 1 --2- 172.21.15.1:0/1831597938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f00b3d40 0x7f49f00ca210 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f49e400e840 tx=0x7f49e400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:10.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.291+0000 7f49ed7fa700 1 -- 172.21.15.1:0/1831597938 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e400c070 con 0x7f49f00b3d40 2024-03-18T19:08:10.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.291+0000 7f49ed7fa700 1 -- 172.21.15.1:0/1831597938 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e4005c20 con 0x7f49f00b3d40 2024-03-18T19:08:10.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.291+0000 7f49f6c1a700 1 -- 172.21.15.1:0/1831597938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f00cb1e0 con 0x7f49f00b3d40 2024-03-18T19:08:10.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.291+0000 7f49ed7fa700 1 -- 172.21.15.1:0/1831597938 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e4004030 con 0x7f49f00b3d40 2024-03-18T19:08:10.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.291+0000 7f49f6c1a700 1 -- 172.21.15.1:0/1831597938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f00cb7a0 con 0x7f49f00b3d40 2024-03-18T19:08:10.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.292+0000 7f49f6c1a700 1 -- 172.21.15.1:0/1831597938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f0068220 con 0x7f49f00b3d40 2024-03-18T19:08:10.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.292+0000 7f49ed7fa700 1 -- 172.21.15.1:0/1831597938 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f49e40041b0 con 0x7f49f00b3d40 2024-03-18T19:08:10.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.293+0000 7f49ed7fa700 1 --2- 172.21.15.1:0/1831597938 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49d406adb0 0x7f49d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:10.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.293+0000 7f49ed7fa700 1 -- 172.21.15.1:0/1831597938 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f49e4088d10 con 0x7f49f00b3d40 2024-03-18T19:08:10.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.293+0000 7f49effff700 1 --2- 172.21.15.1:0/1831597938 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49d406adb0 0x7f49d406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:10.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.293+0000 7f49effff700 1 --2- 172.21.15.1:0/1831597938 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49d406adb0 0x7f49d406d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f49f00cd720 tx=0x7f49dc008040 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:10.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.297+0000 7f49ed7fa700 1 -- 172.21.15.1:0/1831597938 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f49e4053c70 con 0x7f49f00b3d40 2024-03-18T19:08:10.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.510+0000 7f49f6c1a700 1 -- 172.21.15.1:0/1831597938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f49f00d3aa0 con 0x7f49f00b3d40 2024-03-18T19:08:10.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.513+0000 7f49ed7fa700 1 -- 172.21.15.1:0/1831597938 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v37) v1 ==== 115+0+0 (secure 0 0 0) 0x7f49e4017020 con 0x7f49f00b3d40 2024-03-18T19:08:10.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.515+0000 7f49f6c1a700 1 -- 172.21.15.1:0/1831597938 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49d406adb0 msgr2=0x7f49d406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:10.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.515+0000 7f49f6c1a700 1 --2- 172.21.15.1:0/1831597938 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49d406adb0 0x7f49d406d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f49f00cd720 tx=0x7f49dc008040 comp rx=0 tx=0).stop 2024-03-18T19:08:10.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.515+0000 7f49f6c1a700 1 -- 172.21.15.1:0/1831597938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f00b3d40 msgr2=0x7f49f00ca210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:10.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.515+0000 7f49f6c1a700 1 --2- 172.21.15.1:0/1831597938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f00b3d40 0x7f49f00ca210 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f49e400e840 tx=0x7f49e400eb50 comp rx=0 tx=0).stop 2024-03-18T19:08:10.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.515+0000 7f49f6c1a700 1 -- 172.21.15.1:0/1831597938 shutdown_connections 2024-03-18T19:08:10.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.515+0000 7f49f6c1a700 1 --2- 172.21.15.1:0/1831597938 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49d406adb0 0x7f49d406d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.515+0000 7f49f6c1a700 1 --2- 172.21.15.1:0/1831597938 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49f00cff30 0x7f49f00b6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.515+0000 7f49f6c1a700 1 --2- 172.21.15.1:0/1831597938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f00b3d40 0x7f49f00ca210 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.515+0000 7f49f6c1a700 1 --2- 172.21.15.1:0/1831597938 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f00cfb50 0x7f49f00ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.515+0000 7f49f6c1a700 1 -- 172.21.15.1:0/1831597938 >> 172.21.15.1:0/1831597938 conn(0x7f49f00d3140 msgr2=0x7f49f00b1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:10.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.516+0000 7f49f6c1a700 1 -- 172.21.15.1:0/1831597938 shutdown_connections 2024-03-18T19:08:10.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.516+0000 7f49f6c1a700 1 -- 172.21.15.1:0/1831597938 wait complete. 2024-03-18T19:08:10.524 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-18T19:08:10.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.656+0000 7f92a187b700 1 Processor -- start 2024-03-18T19:08:10.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.656+0000 7f92a187b700 1 -- start start 2024-03-18T19:08:10.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.656+0000 7f92a187b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f929c0cfa90 0x7f929c06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:10.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.656+0000 7f92a187b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f929c06ac90 0x7f929c167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:10.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.657+0000 7f92a187b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f929c0cf600 con 0x7f929c15d5a0 2024-03-18T19:08:10.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.657+0000 7f92a187b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f929c06b140 con 0x7f929c06ac90 2024-03-18T19:08:10.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.657+0000 7f92a187b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f929c0ce720 con 0x7f929c0cfa90 2024-03-18T19:08:10.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.657+0000 7f929a7fc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f929c0cfa90 0x7f929c06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:10.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.657+0000 7f929b7fe700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f929c06ac90 0x7f929c167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:10.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.657+0000 7f929affd700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f929c15d5a0 0x7f929c0ce010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:40208/0 (socket says 172.21.15.1:40208) 2024-03-18T19:08:10.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.657+0000 7f929affd700 1 -- 172.21.15.1:0/2506782957 learned_addr learned my addr 172.21.15.1:0/2506782957 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:10.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.657+0000 7f9299ffb700 1 -- 172.21.15.1:0/2506782957 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930596265 0 0) 0x7f929c0cf600 con 0x7f929c15d5a0 2024-03-18T19:08:10.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.657+0000 7f9299ffb700 1 -- 172.21.15.1:0/2506782957 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92840035b0 con 0x7f929c15d5a0 2024-03-18T19:08:10.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.657+0000 7f929b7fe700 1 -- 172.21.15.1:0/2506782957 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f929c0cfa90 msgr2=0x7f929c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:10.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.657+0000 7f929b7fe700 1 --2- 172.21.15.1:0/2506782957 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f929c0cfa90 0x7f929c06a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.657+0000 7f929b7fe700 1 -- 172.21.15.1:0/2506782957 >> v1:172.21.15.165:6789/0 conn(0x7f929c15d5a0 legacy=0x7f929c0ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:10.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.657+0000 7f929b7fe700 1 -- 172.21.15.1:0/2506782957 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929c0ce9f0 con 0x7f929c06ac90 2024-03-18T19:08:10.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.657+0000 7f929a7fc700 1 --2- 172.21.15.1:0/2506782957 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f929c0cfa90 0x7f929c06a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:08:10.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.658+0000 7f929b7fe700 1 --2- 172.21.15.1:0/2506782957 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f929c06ac90 0x7f929c167140 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9294009dc0 tx=0x7f929400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f901585daab335b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:10.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.658+0000 7f9299ffb700 1 -- 172.21.15.1:0/2506782957 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929400e070 con 0x7f929c06ac90 2024-03-18T19:08:10.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.658+0000 7f9299ffb700 1 -- 172.21.15.1:0/2506782957 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f929400bae0 con 0x7f929c06ac90 2024-03-18T19:08:10.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.658+0000 7f9299ffb700 1 -- 172.21.15.1:0/2506782957 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929400be90 con 0x7f929c06ac90 2024-03-18T19:08:10.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.658+0000 7f92a187b700 1 -- 172.21.15.1:0/2506782957 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f929c06ac90 msgr2=0x7f929c167140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:10.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.658+0000 7f92a187b700 1 --2- 172.21.15.1:0/2506782957 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f929c06ac90 0x7f929c167140 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9294009dc0 tx=0x7f929400b330 comp rx=0 tx=0).stop 2024-03-18T19:08:10.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.658+0000 7f92a187b700 1 -- 172.21.15.1:0/2506782957 shutdown_connections 2024-03-18T19:08:10.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.658+0000 7f92a187b700 1 --2- 172.21.15.1:0/2506782957 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f929c06ac90 0x7f929c167140 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.658+0000 7f92a187b700 1 --2- 172.21.15.1:0/2506782957 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f929c0cfa90 0x7f929c06a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.658+0000 7f92a187b700 1 -- 172.21.15.1:0/2506782957 >> 172.21.15.1:0/2506782957 conn(0x7f929c0cc9b0 msgr2=0x7f929c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:10.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.658+0000 7f92a187b700 1 -- 172.21.15.1:0/2506782957 shutdown_connections 2024-03-18T19:08:10.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.658+0000 7f92a187b700 1 -- 172.21.15.1:0/2506782957 wait complete. 2024-03-18T19:08:10.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.659+0000 7f92a187b700 1 Processor -- start 2024-03-18T19:08:10.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.660+0000 7f92a187b700 1 -- start start 2024-03-18T19:08:10.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.660+0000 7f92a187b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f929c06ac90 0x7f929c1f27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:10.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.660+0000 7f92a187b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f929c0cfa90 0x7f929c15e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:10.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.660+0000 7f92a187b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f929c15d5a0 0x7f929c1603e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:10.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.660+0000 7f92a187b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c152d30 con 0x7f929c06ac90 2024-03-18T19:08:10.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.660+0000 7f92a187b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c15ff50 con 0x7f929c15d5a0 2024-03-18T19:08:10.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.660+0000 7f92a187b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f929c1f2d00 con 0x7f929c0cfa90 2024-03-18T19:08:10.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.661+0000 7f929affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f929c06ac90 0x7f929c1f27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:10.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.661+0000 7f929affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f929c06ac90 0x7f929c1f27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40366/0 (socket says 172.21.15.1:40366) 2024-03-18T19:08:10.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.661+0000 7f929affd700 1 -- 172.21.15.1:0/732184353 learned_addr learned my addr 172.21.15.1:0/732184353 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:10.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.661+0000 7f929a7fc700 1 --2- 172.21.15.1:0/732184353 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f929c0cfa90 0x7f929c15e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:10.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.661+0000 7f929b7fe700 1 --2- 172.21.15.1:0/732184353 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f929c15d5a0 0x7f929c1603e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:10.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.661+0000 7f929affd700 1 -- 172.21.15.1:0/732184353 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f929c0cfa90 msgr2=0x7f929c15e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:10.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.661+0000 7f929affd700 1 --2- 172.21.15.1:0/732184353 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f929c0cfa90 0x7f929c15e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.661+0000 7f929affd700 1 -- 172.21.15.1:0/732184353 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f929c15d5a0 msgr2=0x7f929c1603e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:10.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.661+0000 7f929affd700 1 --2- 172.21.15.1:0/732184353 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f929c15d5a0 0x7f929c1603e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.661+0000 7f929affd700 1 -- 172.21.15.1:0/732184353 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92900094a0 con 0x7f929c06ac90 2024-03-18T19:08:10.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.661+0000 7f929b7fe700 1 --2- 172.21.15.1:0/732184353 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f929c15d5a0 0x7f929c1603e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:10.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.661+0000 7f929a7fc700 1 --2- 172.21.15.1:0/732184353 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f929c0cfa90 0x7f929c15e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:10.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.661+0000 7f929affd700 1 --2- 172.21.15.1:0/732184353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f929c06ac90 0x7f929c1f27c0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f928c001130 tx=0x7f928c00ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:10.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.661+0000 7f9283fff700 1 -- 172.21.15.1:0/732184353 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c005dc0 con 0x7f929c06ac90 2024-03-18T19:08:10.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.661+0000 7f9283fff700 1 -- 172.21.15.1:0/732184353 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928c0070d0 con 0x7f929c06ac90 2024-03-18T19:08:10.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.661+0000 7f92a187b700 1 -- 172.21.15.1:0/732184353 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9294009a70 con 0x7f929c06ac90 2024-03-18T19:08:10.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.661+0000 7f9283fff700 1 -- 172.21.15.1:0/732184353 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c007460 con 0x7f929c06ac90 2024-03-18T19:08:10.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.661+0000 7f92a187b700 1 -- 172.21.15.1:0/732184353 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929c15f620 con 0x7f929c06ac90 2024-03-18T19:08:10.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.662+0000 7f92a187b700 1 -- 172.21.15.1:0/732184353 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929c160230 con 0x7f929c06ac90 2024-03-18T19:08:10.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.663+0000 7f9283fff700 1 -- 172.21.15.1:0/732184353 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f928c0075c0 con 0x7f929c06ac90 2024-03-18T19:08:10.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.663+0000 7f9283fff700 1 --2- 172.21.15.1:0/732184353 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f928806adb0 0x7f928806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:10.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.663+0000 7f929a7fc700 1 --2- 172.21.15.1:0/732184353 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f928806adb0 0x7f928806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:10.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.663+0000 7f9283fff700 1 -- 172.21.15.1:0/732184353 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f928c088be0 con 0x7f929c06ac90 2024-03-18T19:08:10.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.664+0000 7f929a7fc700 1 --2- 172.21.15.1:0/732184353 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f928806adb0 0x7f928806d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f929c155f60 tx=0x7f9290008040 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:10.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.668+0000 7f9283fff700 1 -- 172.21.15.1:0/732184353 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f928c053b30 con 0x7f929c06ac90 2024-03-18T19:08:10.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.873+0000 7f92a187b700 1 -- 172.21.15.1:0/732184353 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f929c0676a0 con 0x7f929c06ac90 2024-03-18T19:08:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.877+0000 7f9283fff700 1 -- 172.21.15.1:0/732184353 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v38) v1 ==== 120+0+0 (secure 0 0 0) 0x7f928c017020 con 0x7f929c06ac90 2024-03-18T19:08:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.880+0000 7f92a187b700 1 -- 172.21.15.1:0/732184353 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f928806adb0 msgr2=0x7f928806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.880+0000 7f92a187b700 1 --2- 172.21.15.1:0/732184353 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f928806adb0 0x7f928806d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f929c155f60 tx=0x7f9290008040 comp rx=0 tx=0).stop 2024-03-18T19:08:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.880+0000 7f92a187b700 1 -- 172.21.15.1:0/732184353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f929c06ac90 msgr2=0x7f929c1f27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.880+0000 7f92a187b700 1 --2- 172.21.15.1:0/732184353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f929c06ac90 0x7f929c1f27c0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f928c001130 tx=0x7f928c00ca90 comp rx=0 tx=0).stop 2024-03-18T19:08:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.880+0000 7f92a187b700 1 -- 172.21.15.1:0/732184353 shutdown_connections 2024-03-18T19:08:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.880+0000 7f92a187b700 1 --2- 172.21.15.1:0/732184353 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f928806adb0 0x7f928806d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.880+0000 7f92a187b700 1 --2- 172.21.15.1:0/732184353 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f929c15d5a0 0x7f929c1603e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.880+0000 7f92a187b700 1 --2- 172.21.15.1:0/732184353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f929c06ac90 0x7f929c1f27c0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.880+0000 7f92a187b700 1 --2- 172.21.15.1:0/732184353 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f929c0cfa90 0x7f929c15e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:10.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.880+0000 7f92a187b700 1 -- 172.21.15.1:0/732184353 >> 172.21.15.1:0/732184353 conn(0x7f929c0cc9b0 msgr2=0x7f929c0b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:10.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.880+0000 7f92a187b700 1 -- 172.21.15.1:0/732184353 shutdown_connections 2024-03-18T19:08:10.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:10.880+0000 7f92a187b700 1 -- 172.21.15.1:0/732184353 wait complete. 2024-03-18T19:08:10.889 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-18T19:08:11.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.020+0000 7fe16de6b700 1 Processor -- start 2024-03-18T19:08:11.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.020+0000 7fe16de6b700 1 -- start start 2024-03-18T19:08:11.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.021+0000 7fe16de6b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe1680cfa90 0x7fe16806a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:11.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.021+0000 7fe16de6b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1680cf600 con 0x7fe16815d5a0 2024-03-18T19:08:11.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.021+0000 7fe16de6b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1680ce720 con 0x7fe16806ac90 2024-03-18T19:08:11.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.021+0000 7fe16de6b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe16806b140 con 0x7fe1680cfa90 2024-03-18T19:08:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.021+0000 7fe1677fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe16815d5a0 0x7fe1680ce010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54850/0 (socket says 172.21.15.1:54850) 2024-03-18T19:08:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.021+0000 7fe1677fe700 1 -- 172.21.15.1:0/3422135020 learned_addr learned my addr 172.21.15.1:0/3422135020 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.021+0000 7fe166ffd700 1 --2- 172.21.15.1:0/3422135020 >> v2:172.21.15.165:3300/0 conn(0x7fe1680cfa90 0x7fe16806a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.021+0000 7fe1667fc700 1 -- 172.21.15.1:0/3422135020 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198791622 0 0) 0x7fe1680cf600 con 0x7fe16815d5a0 2024-03-18T19:08:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.021+0000 7fe1667fc700 1 -- 172.21.15.1:0/3422135020 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1500035b0 con 0x7fe16815d5a0 2024-03-18T19:08:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.021+0000 7fe1667fc700 1 -- 172.21.15.1:0/3422135020 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523704225 0 0) 0x7fe1680ce720 con 0x7fe16806ac90 2024-03-18T19:08:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1667fc700 1 -- 172.21.15.1:0/3422135020 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1680cf600 con 0x7fe16806ac90 2024-03-18T19:08:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1667fc700 1 -- 172.21.15.1:0/3422135020 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2095628951 0 0) 0x7fe1500035b0 con 0x7fe16815d5a0 2024-03-18T19:08:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1667fc700 1 -- 172.21.15.1:0/3422135020 >> v2:172.21.15.165:3300/0 conn(0x7fe1680cfa90 msgr2=0x7fe16806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1667fc700 1 --2- 172.21.15.1:0/3422135020 >> v2:172.21.15.165:3300/0 conn(0x7fe1680cfa90 0x7fe16806a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1667fc700 1 -- 172.21.15.1:0/3422135020 >> v1:172.21.15.165:6789/0 conn(0x7fe16806ac90 legacy=0x7fe168167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1667fc700 1 -- 172.21.15.1:0/3422135020 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1680ce9f0 con 0x7fe16815d5a0 2024-03-18T19:08:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1667fc700 1 -- 172.21.15.1:0/3422135020 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fe15c0017f0 con 0x7fe16815d5a0 2024-03-18T19:08:11.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1667fc700 1 -- 172.21.15.1:0/3422135020 >> v1:172.21.15.1:6789/0 conn(0x7fe16815d5a0 legacy=0x7fe1680ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:11.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1667fc700 1 --2- 172.21.15.1:0/3422135020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe150003be0 0x7fe150003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:11.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1667fc700 1 --2- 172.21.15.1:0/3422135020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe150004850 0x7fe150004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:11.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1667fc700 1 --2- 172.21.15.1:0/3422135020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe150005210 0x7fe15000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:11.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1667fc700 1 -- 172.21.15.1:0/3422135020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16806b140 con 0x7fe150003be0 2024-03-18T19:08:11.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1667fc700 1 -- 172.21.15.1:0/3422135020 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15000c0e0 con 0x7fe150005210 2024-03-18T19:08:11.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1667fc700 1 -- 172.21.15.1:0/3422135020 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15000c2a0 con 0x7fe150004850 2024-03-18T19:08:11.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1677fe700 1 --2- 172.21.15.1:0/3422135020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe150003be0 0x7fe150003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:11.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe166ffd700 1 --2- 172.21.15.1:0/3422135020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe150004850 0x7fe150004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:11.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe167fff700 1 --2- 172.21.15.1:0/3422135020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe150005210 0x7fe15000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:11.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1677fe700 1 -- 172.21.15.1:0/3422135020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe150004850 msgr2=0x7fe150004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:11.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1677fe700 1 --2- 172.21.15.1:0/3422135020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe150004850 0x7fe150004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:11.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1677fe700 1 -- 172.21.15.1:0/3422135020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe150005210 msgr2=0x7fe15000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:11.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1677fe700 1 --2- 172.21.15.1:0/3422135020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe150005210 0x7fe15000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:11.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe167fff700 1 -- 172.21.15.1:0/3422135020 reap_dead start 2024-03-18T19:08:11.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1677fe700 1 -- 172.21.15.1:0/3422135020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1500035b0 con 0x7fe150003be0 2024-03-18T19:08:11.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1677fe700 1 --2- 172.21.15.1:0/3422135020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe150003be0 0x7fe150003fe0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fe15c00c110 tx=0x7fe150005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fd0c28105e1e6d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:11.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1667fc700 1 -- 172.21.15.1:0/3422135020 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15c00ccb0 con 0x7fe150003be0 2024-03-18T19:08:11.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1667fc700 1 -- 172.21.15.1:0/3422135020 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15c00ce10 con 0x7fe150003be0 2024-03-18T19:08:11.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.022+0000 7fe1667fc700 1 -- 172.21.15.1:0/3422135020 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15c007310 con 0x7fe150003be0 2024-03-18T19:08:11.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.023+0000 7fe16de6b700 1 -- 172.21.15.1:0/3422135020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe150003be0 msgr2=0x7fe150003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:11.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.023+0000 7fe16de6b700 1 --2- 172.21.15.1:0/3422135020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe150003be0 0x7fe150003fe0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fe15c00c110 tx=0x7fe150005880 comp rx=0 tx=0).stop 2024-03-18T19:08:11.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.023+0000 7fe16de6b700 1 -- 172.21.15.1:0/3422135020 shutdown_connections 2024-03-18T19:08:11.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.023+0000 7fe16de6b700 1 --2- 172.21.15.1:0/3422135020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe150003be0 0x7fe150003fe0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:11.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.023+0000 7fe16de6b700 1 -- 172.21.15.1:0/3422135020 >> 172.21.15.1:0/3422135020 conn(0x7fe1680cc9b0 msgr2=0x7fe1680ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:11.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.023+0000 7fe16de6b700 1 -- 172.21.15.1:0/3422135020 shutdown_connections 2024-03-18T19:08:11.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.023+0000 7fe16de6b700 1 -- 172.21.15.1:0/3422135020 wait complete. 2024-03-18T19:08:11.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.025+0000 7fe16de6b700 1 Processor -- start 2024-03-18T19:08:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.025+0000 7fe16de6b700 1 -- start start 2024-03-18T19:08:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.026+0000 7fe16de6b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe150003be0 0x7fe1680c3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.026+0000 7fe16de6b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1680b51b0 0x7fe1680c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.026+0000 7fe16de6b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1680b3b50 0x7fe1680c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.026+0000 7fe16de6b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1681f3070 con 0x7fe150003be0 2024-03-18T19:08:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.026+0000 7fe16de6b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1680b4110 con 0x7fe1680b51b0 2024-03-18T19:08:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.026+0000 7fe16de6b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1680b43f0 con 0x7fe1680b3b50 2024-03-18T19:08:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.026+0000 7fe1677fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe150003be0 0x7fe1680c3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.026+0000 7fe1677fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe150003be0 0x7fe1680c3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40392/0 (socket says 172.21.15.1:40392) 2024-03-18T19:08:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.026+0000 7fe1677fe700 1 -- 172.21.15.1:0/2346704043 learned_addr learned my addr 172.21.15.1:0/2346704043 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:11.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.026+0000 7fe1677fe700 1 -- 172.21.15.1:0/2346704043 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1680b3b50 msgr2=0x7fe1680c5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.026+0000 7fe1677fe700 1 --2- 172.21.15.1:0/2346704043 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1680b3b50 0x7fe1680c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.026+0000 7fe1677fe700 1 -- 172.21.15.1:0/2346704043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1680b51b0 msgr2=0x7fe1680c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.026+0000 7fe1677fe700 1 --2- 172.21.15.1:0/2346704043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1680b51b0 0x7fe1680c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.026+0000 7fe1677fe700 1 -- 172.21.15.1:0/2346704043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1680b46d0 con 0x7fe150003be0 2024-03-18T19:08:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.026+0000 7fe1677fe700 1 --2- 172.21.15.1:0/2346704043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe150003be0 0x7fe1680c3a60 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fe15c011000 tx=0x7fe1680cf570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.026+0000 7fe14ffff700 1 -- 172.21.15.1:0/2346704043 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15c000f70 con 0x7fe150003be0 2024-03-18T19:08:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.026+0000 7fe14ffff700 1 -- 172.21.15.1:0/2346704043 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15c005830 con 0x7fe150003be0 2024-03-18T19:08:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.027+0000 7fe16de6b700 1 -- 172.21.15.1:0/2346704043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1680b49b0 con 0x7fe150003be0 2024-03-18T19:08:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.027+0000 7fe14ffff700 1 -- 172.21.15.1:0/2346704043 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15c005bc0 con 0x7fe150003be0 2024-03-18T19:08:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.027+0000 7fe16de6b700 1 -- 172.21.15.1:0/2346704043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1680c6410 con 0x7fe150003be0 2024-03-18T19:08:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.028+0000 7fe16de6b700 1 -- 172.21.15.1:0/2346704043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1680b4ed0 con 0x7fe150003be0 2024-03-18T19:08:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.028+0000 7fe14ffff700 1 -- 172.21.15.1:0/2346704043 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe15c005d20 con 0x7fe150003be0 2024-03-18T19:08:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.028+0000 7fe14ffff700 1 --2- 172.21.15.1:0/2346704043 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe15406ade0 0x7fe15406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:11.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.028+0000 7fe166ffd700 1 --2- 172.21.15.1:0/2346704043 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe15406ade0 0x7fe15406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:11.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.028+0000 7fe14ffff700 1 -- 172.21.15.1:0/2346704043 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5951+0+0 (secure 0 0 0) 0x7fe15c08a220 con 0x7fe150003be0 2024-03-18T19:08:11.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.029+0000 7fe166ffd700 1 --2- 172.21.15.1:0/2346704043 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe15406ade0 0x7fe15406d2a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe158006f10 tx=0x7fe158006e80 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:11.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.033+0000 7fe14ffff700 1 -- 172.21.15.1:0/2346704043 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe1680b4ed0 con 0x7fe150003be0 2024-03-18T19:08:11.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.235+0000 7fe16de6b700 1 -- 172.21.15.1:0/2346704043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe1680b4ed0 con 0x7fe150003be0 2024-03-18T19:08:11.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.238+0000 7fe14ffff700 1 -- 172.21.15.1:0/2346704043 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v39) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe1680b4ed0 con 0x7fe150003be0 2024-03-18T19:08:11.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.241+0000 7fe16de6b700 1 -- 172.21.15.1:0/2346704043 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe15406ade0 msgr2=0x7fe15406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:11.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.241+0000 7fe16de6b700 1 --2- 172.21.15.1:0/2346704043 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe15406ade0 0x7fe15406d2a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe158006f10 tx=0x7fe158006e80 comp rx=0 tx=0).stop 2024-03-18T19:08:11.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.241+0000 7fe16de6b700 1 -- 172.21.15.1:0/2346704043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe150003be0 msgr2=0x7fe1680c3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:11.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.241+0000 7fe16de6b700 1 --2- 172.21.15.1:0/2346704043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe150003be0 0x7fe1680c3a60 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fe15c011000 tx=0x7fe1680cf570 comp rx=0 tx=0).stop 2024-03-18T19:08:11.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.241+0000 7fe16de6b700 1 -- 172.21.15.1:0/2346704043 shutdown_connections 2024-03-18T19:08:11.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.241+0000 7fe16de6b700 1 --2- 172.21.15.1:0/2346704043 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe15406ade0 0x7fe15406d2a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:11.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.241+0000 7fe16de6b700 1 --2- 172.21.15.1:0/2346704043 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1680b3b50 0x7fe1680c5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:11.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.241+0000 7fe16de6b700 1 --2- 172.21.15.1:0/2346704043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe150003be0 0x7fe1680c3a60 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:11.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.241+0000 7fe16de6b700 1 --2- 172.21.15.1:0/2346704043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1680b51b0 0x7fe1680c3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:11.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.241+0000 7fe16de6b700 1 -- 172.21.15.1:0/2346704043 >> 172.21.15.1:0/2346704043 conn(0x7fe1680cc9b0 msgr2=0x7fe1680b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:11.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.241+0000 7fe16de6b700 1 -- 172.21.15.1:0/2346704043 shutdown_connections 2024-03-18T19:08:11.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.241+0000 7fe16de6b700 1 -- 172.21.15.1:0/2346704043 wait complete. 2024-03-18T19:08:11.249 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2024-03-18T19:08:11.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.378+0000 7f612f9c6700 1 Processor -- start 2024-03-18T19:08:11.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.379+0000 7f612f9c6700 1 -- start start 2024-03-18T19:08:11.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.379+0000 7f612f9c6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6128143c20 0x7f61280b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:11.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.379+0000 7f612f9c6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f612806f240 0x7f61280cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:11.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.379+0000 7f612f9c6700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6128144000 con 0x7f61280b3f40 2024-03-18T19:08:11.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.379+0000 7f612f9c6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61280b43c0 con 0x7f6128143c20 2024-03-18T19:08:11.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.379+0000 7f612f9c6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61280b51b0 con 0x7f612806f240 2024-03-18T19:08:11.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.380+0000 7f612d762700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6128143c20 0x7f61280b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:11.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.380+0000 7f612d762700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6128143c20 0x7f61280b3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40400/0 (socket says 172.21.15.1:40400) 2024-03-18T19:08:11.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.380+0000 7f612d762700 1 -- 172.21.15.1:0/2538121076 learned_addr learned my addr 172.21.15.1:0/2538121076 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:11.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.380+0000 7f612df63700 1 --2- 172.21.15.1:0/2538121076 >> v2:172.21.15.165:3300/0 conn(0x7f612806f240 0x7f61280cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:11.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.380+0000 7f612d762700 1 -- 172.21.15.1:0/2538121076 >> v2:172.21.15.165:3300/0 conn(0x7f612806f240 msgr2=0x7f61280cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:11.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.380+0000 7f612d762700 1 --2- 172.21.15.1:0/2538121076 >> v2:172.21.15.165:3300/0 conn(0x7f612806f240 0x7f61280cddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:11.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.380+0000 7f612d762700 1 -- 172.21.15.1:0/2538121076 >> v1:172.21.15.165:6789/0 conn(0x7f61280b3f40 legacy=0x7f612806eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:11.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.380+0000 7f612d762700 1 -- 172.21.15.1:0/2538121076 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61280cd8d0 con 0x7f6128143c20 2024-03-18T19:08:11.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.380+0000 7f612d762700 1 --2- 172.21.15.1:0/2538121076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6128143c20 0x7f61280b3a00 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f6128067530 tx=0x7f6124009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7aee7d3d933ed02 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:11.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.380+0000 7f611ffff700 1 -- 172.21.15.1:0/2538121076 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6124011830 con 0x7f6128143c20 2024-03-18T19:08:11.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.380+0000 7f611ffff700 1 -- 172.21.15.1:0/2538121076 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6124011990 con 0x7f6128143c20 2024-03-18T19:08:11.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.380+0000 7f611ffff700 1 -- 172.21.15.1:0/2538121076 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6124011d20 con 0x7f6128143c20 2024-03-18T19:08:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.381+0000 7f612f9c6700 1 -- 172.21.15.1:0/2538121076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6128143c20 msgr2=0x7f61280b3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.381+0000 7f612f9c6700 1 --2- 172.21.15.1:0/2538121076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6128143c20 0x7f61280b3a00 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f6128067530 tx=0x7f6124009bd0 comp rx=0 tx=0).stop 2024-03-18T19:08:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.381+0000 7f612f9c6700 1 -- 172.21.15.1:0/2538121076 shutdown_connections 2024-03-18T19:08:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.381+0000 7f612f9c6700 1 --2- 172.21.15.1:0/2538121076 >> v2:172.21.15.165:3300/0 conn(0x7f612806f240 0x7f61280cddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.381+0000 7f612f9c6700 1 --2- 172.21.15.1:0/2538121076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6128143c20 0x7f61280b3a00 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.381+0000 7f612f9c6700 1 -- 172.21.15.1:0/2538121076 >> 172.21.15.1:0/2538121076 conn(0x7f61280b1d20 msgr2=0x7f61280b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.381+0000 7f612f9c6700 1 -- 172.21.15.1:0/2538121076 shutdown_connections 2024-03-18T19:08:11.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.381+0000 7f612f9c6700 1 -- 172.21.15.1:0/2538121076 wait complete. 2024-03-18T19:08:11.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.382+0000 7f612f9c6700 1 Processor -- start 2024-03-18T19:08:11.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.382+0000 7f612f9c6700 1 -- start start 2024-03-18T19:08:11.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.383+0000 7f612f9c6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f612806f240 0x7f61280ae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:11.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.383+0000 7f612f9c6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61280b3f40 0x7f61281d4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:11.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.383+0000 7f612f9c6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6128143c20 0x7f61281d48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:11.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.383+0000 7f612f9c6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61281d5140 con 0x7f6128143c20 2024-03-18T19:08:11.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.383+0000 7f612f9c6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61281d5420 con 0x7f61280b3f40 2024-03-18T19:08:11.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.383+0000 7f612f9c6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61281d5700 con 0x7f612806f240 2024-03-18T19:08:11.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.383+0000 7f612df63700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6128143c20 0x7f61281d48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:11.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.383+0000 7f612df63700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6128143c20 0x7f61281d48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40406/0 (socket says 172.21.15.1:40406) 2024-03-18T19:08:11.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.383+0000 7f612df63700 1 -- 172.21.15.1:0/1766326577 learned_addr learned my addr 172.21.15.1:0/1766326577 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:11.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.383+0000 7f612d762700 1 --2- 172.21.15.1:0/1766326577 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f612806f240 0x7f61280ae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:11.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.383+0000 7f612cf61700 1 --2- 172.21.15.1:0/1766326577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61280b3f40 0x7f61281d4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:11.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.383+0000 7f612df63700 1 -- 172.21.15.1:0/1766326577 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f612806f240 msgr2=0x7f61280ae0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:11.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.383+0000 7f612df63700 1 --2- 172.21.15.1:0/1766326577 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f612806f240 0x7f61280ae0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:11.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.383+0000 7f612df63700 1 -- 172.21.15.1:0/1766326577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61280b3f40 msgr2=0x7f61281d4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:11.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.383+0000 7f612df63700 1 --2- 172.21.15.1:0/1766326577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61280b3f40 0x7f61281d4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:11.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.383+0000 7f612df63700 1 -- 172.21.15.1:0/1766326577 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6124009550 con 0x7f6128143c20 2024-03-18T19:08:11.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.383+0000 7f612df63700 1 --2- 172.21.15.1:0/1766326577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6128143c20 0x7f61281d48c0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f612000efc0 tx=0x7f612000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:11.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.384+0000 7f611dffb700 1 -- 172.21.15.1:0/1766326577 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6120009ea0 con 0x7f6128143c20 2024-03-18T19:08:11.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.384+0000 7f611dffb700 1 -- 172.21.15.1:0/1766326577 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f612000f040 con 0x7f6128143c20 2024-03-18T19:08:11.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.384+0000 7f612f9c6700 1 -- 172.21.15.1:0/1766326577 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61281d59e0 con 0x7f6128143c20 2024-03-18T19:08:11.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.384+0000 7f611dffb700 1 -- 172.21.15.1:0/1766326577 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612000ced0 con 0x7f6128143c20 2024-03-18T19:08:11.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.384+0000 7f612f9c6700 1 -- 172.21.15.1:0/1766326577 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61281d4e00 con 0x7f6128143c20 2024-03-18T19:08:11.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.385+0000 7f612f9c6700 1 -- 172.21.15.1:0/1766326577 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61281d5f00 con 0x7f6128143c20 2024-03-18T19:08:11.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.385+0000 7f611dffb700 1 -- 172.21.15.1:0/1766326577 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6120014480 con 0x7f6128143c20 2024-03-18T19:08:11.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.386+0000 7f611dffb700 1 --2- 172.21.15.1:0/1766326577 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f610c06adb0 0x7f610c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:11.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.386+0000 7f611dffb700 1 -- 172.21.15.1:0/1766326577 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f6120089f30 con 0x7f6128143c20 2024-03-18T19:08:11.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.386+0000 7f612d762700 1 --2- 172.21.15.1:0/1766326577 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f610c06adb0 0x7f610c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:11.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.386+0000 7f612d762700 1 --2- 172.21.15.1:0/1766326577 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f610c06adb0 0x7f610c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6124014040 tx=0x7f6124008030 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:11.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.390+0000 7f611dffb700 1 -- 172.21.15.1:0/1766326577 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6120054d60 con 0x7f6128143c20 2024-03-18T19:08:11.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:11.679+0000 7f612f9c6700 1 -- 172.21.15.1:0/1766326577 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f61280cd1d0 con 0x7f6128143c20 2024-03-18T19:08:12.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.129+0000 7f611dffb700 1 -- 172.21.15.1:0/1766326577 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v693) v1 ==== 183+0+0 (secure 0 0 0) 0x7f6120018020 con 0x7f6128143c20 2024-03-18T19:08:12.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.131+0000 7f612f9c6700 1 -- 172.21.15.1:0/1766326577 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f610c06adb0 msgr2=0x7f610c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:12.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.131+0000 7f612f9c6700 1 --2- 172.21.15.1:0/1766326577 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f610c06adb0 0x7f610c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6124014040 tx=0x7f6124008030 comp rx=0 tx=0).stop 2024-03-18T19:08:12.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.131+0000 7f612f9c6700 1 -- 172.21.15.1:0/1766326577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6128143c20 msgr2=0x7f61281d48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:12.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.131+0000 7f612f9c6700 1 --2- 172.21.15.1:0/1766326577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6128143c20 0x7f61281d48c0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f612000efc0 tx=0x7f612000c3a0 comp rx=0 tx=0).stop 2024-03-18T19:08:12.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.132+0000 7f612f9c6700 1 -- 172.21.15.1:0/1766326577 shutdown_connections 2024-03-18T19:08:12.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.132+0000 7f612f9c6700 1 --2- 172.21.15.1:0/1766326577 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f610c06adb0 0x7f610c06d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:12.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.132+0000 7f612f9c6700 1 --2- 172.21.15.1:0/1766326577 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f612806f240 0x7f61280ae0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:12.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.132+0000 7f612f9c6700 1 --2- 172.21.15.1:0/1766326577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6128143c20 0x7f61281d48c0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:12.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.132+0000 7f612f9c6700 1 --2- 172.21.15.1:0/1766326577 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61280b3f40 0x7f61281d4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:12.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.132+0000 7f612f9c6700 1 -- 172.21.15.1:0/1766326577 >> 172.21.15.1:0/1766326577 conn(0x7f61280b1d20 msgr2=0x7f61280b2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:12.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.132+0000 7f612f9c6700 1 -- 172.21.15.1:0/1766326577 shutdown_connections 2024-03-18T19:08:12.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.132+0000 7f612f9c6700 1 -- 172.21.15.1:0/1766326577 wait complete. 2024-03-18T19:08:12.142 INFO:tasks.cephfs_test_runner:Test that a non-existent path leads to schedule deactivation after a few retries. ... ok 2024-03-18T19:08:12.142 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2024-03-18T19:08:12.142 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-18T19:08:12.142 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2024-03-18T19:08:12.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.277+0000 7f2f05292700 1 Processor -- start 2024-03-18T19:08:12.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.277+0000 7f2f05292700 1 -- start start 2024-03-18T19:08:12.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.278+0000 7f2f05292700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2f0015d350 0x7f2f0006de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:12.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.278+0000 7f2f05292700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f000b4f20 con 0x7f2f0006ee10 2024-03-18T19:08:12.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.278+0000 7f2f05292700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f000b5200 con 0x7f2f0006ae60 2024-03-18T19:08:12.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.278+0000 7f2f05292700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f0006d380 con 0x7f2f0015d350 2024-03-18T19:08:12.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.278+0000 7f2efe7fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2f0006ee10 0x7f2f0006a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54852/0 (socket says 172.21.15.1:54852) 2024-03-18T19:08:12.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.278+0000 7f2efe7fc700 1 -- 172.21.15.1:0/3964801582 learned_addr learned my addr 172.21.15.1:0/3964801582 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:12.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.278+0000 7f2efdffb700 1 -- 172.21.15.1:0/3964801582 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182327749 0 0) 0x7f2f000b4f20 con 0x7f2f0006ee10 2024-03-18T19:08:12.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.278+0000 7f2efeffd700 1 --2- 172.21.15.1:0/3964801582 >> v2:172.21.15.165:3300/0 conn(0x7f2f0015d350 0x7f2f0006de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:12.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.278+0000 7f2efdffb700 1 -- 172.21.15.1:0/3964801582 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee80035b0 con 0x7f2f0006ee10 2024-03-18T19:08:12.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.278+0000 7f2efdffb700 1 -- 172.21.15.1:0/3964801582 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3909226758 0 0) 0x7f2ee80035b0 con 0x7f2f0006ee10 2024-03-18T19:08:12.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efdffb700 1 -- 172.21.15.1:0/3964801582 >> v2:172.21.15.165:3300/0 conn(0x7f2f0015d350 msgr2=0x7f2f0006de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:12.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efdffb700 1 --2- 172.21.15.1:0/3964801582 >> v2:172.21.15.165:3300/0 conn(0x7f2f0015d350 0x7f2f0006de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:12.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efdffb700 1 -- 172.21.15.1:0/3964801582 >> v1:172.21.15.165:6789/0 conn(0x7f2f0006ae60 legacy=0x7f2f00071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:12.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efdffb700 1 -- 172.21.15.1:0/3964801582 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0006d660 con 0x7f2f0006ee10 2024-03-18T19:08:12.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efdffb700 1 -- 172.21.15.1:0/3964801582 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f2ef0001670 con 0x7f2f0006ee10 2024-03-18T19:08:12.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efdffb700 1 -- 172.21.15.1:0/3964801582 >> v1:172.21.15.1:6789/0 conn(0x7f2f0006ee10 legacy=0x7f2f0006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:12.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efdffb700 1 --2- 172.21.15.1:0/3964801582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ee8003a00 0x7f2ee8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:12.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efdffb700 1 --2- 172.21.15.1:0/3964801582 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ee8004850 0x7f2ee8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:12.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efdffb700 1 --2- 172.21.15.1:0/3964801582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ee8005210 0x7f2ee800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:12.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efdffb700 1 -- 172.21.15.1:0/3964801582 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0006d380 con 0x7f2ee8003a00 2024-03-18T19:08:12.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efdffb700 1 -- 172.21.15.1:0/3964801582 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee800c0b0 con 0x7f2ee8005210 2024-03-18T19:08:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efdffb700 1 -- 172.21.15.1:0/3964801582 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee800c270 con 0x7f2ee8004850 2024-03-18T19:08:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efeffd700 1 --2- 172.21.15.1:0/3964801582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ee8003a00 0x7f2ee8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2eff7fe700 1 --2- 172.21.15.1:0/3964801582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ee8005210 0x7f2ee800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efe7fc700 1 --2- 172.21.15.1:0/3964801582 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ee8004850 0x7f2ee8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efeffd700 1 -- 172.21.15.1:0/3964801582 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ee8004850 msgr2=0x7f2ee8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efeffd700 1 --2- 172.21.15.1:0/3964801582 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ee8004850 0x7f2ee8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efeffd700 1 -- 172.21.15.1:0/3964801582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ee8005210 msgr2=0x7f2ee800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efeffd700 1 --2- 172.21.15.1:0/3964801582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ee8005210 0x7f2ee800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efeffd700 1 -- 172.21.15.1:0/3964801582 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee80035b0 con 0x7f2ee8003a00 2024-03-18T19:08:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efe7fc700 1 --2- 172.21.15.1:0/3964801582 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ee8004850 0x7f2ee8004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2eff7fe700 1 --2- 172.21.15.1:0/3964801582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ee8005210 0x7f2ee800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2eff7fe700 1 -- 172.21.15.1:0/3964801582 reap_dead start 2024-03-18T19:08:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efeffd700 1 --2- 172.21.15.1:0/3964801582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ee8003a00 0x7f2ee8003e20 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4009b60 tx=0x7f2ee8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca80fa4a89601af server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:12.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efdffb700 1 -- 172.21.15.1:0/3964801582 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef400e6c0 con 0x7f2ee8003a00 2024-03-18T19:08:12.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efdffb700 1 -- 172.21.15.1:0/3964801582 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef400e820 con 0x7f2ee8003a00 2024-03-18T19:08:12.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.279+0000 7f2efdffb700 1 -- 172.21.15.1:0/3964801582 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef400eb90 con 0x7f2ee8003a00 2024-03-18T19:08:12.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.280+0000 7f2f05292700 1 -- 172.21.15.1:0/3964801582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ee8003a00 msgr2=0x7f2ee8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:12.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.280+0000 7f2f05292700 1 --2- 172.21.15.1:0/3964801582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ee8003a00 0x7f2ee8003e20 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4009b60 tx=0x7f2ee8005880 comp rx=0 tx=0).stop 2024-03-18T19:08:12.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.280+0000 7f2f05292700 1 -- 172.21.15.1:0/3964801582 shutdown_connections 2024-03-18T19:08:12.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.280+0000 7f2f05292700 1 --2- 172.21.15.1:0/3964801582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ee8003a00 0x7f2ee8003e20 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:12.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.280+0000 7f2f05292700 1 -- 172.21.15.1:0/3964801582 >> 172.21.15.1:0/3964801582 conn(0x7f2f000cff00 msgr2=0x7f2f000d0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:12.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.280+0000 7f2f05292700 1 -- 172.21.15.1:0/3964801582 shutdown_connections 2024-03-18T19:08:12.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.280+0000 7f2f05292700 1 -- 172.21.15.1:0/3964801582 wait complete. 2024-03-18T19:08:12.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.281+0000 7f2f05292700 1 Processor -- start 2024-03-18T19:08:12.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.281+0000 7f2f05292700 1 -- start start 2024-03-18T19:08:12.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.282+0000 7f2f05292700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ee8003a00 0x7f2f00143a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.282+0000 7f2f05292700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f0015b7f0 0x7f2f0015bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.282+0000 7f2f05292700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f000cc8f0 0x7f2f000ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.282+0000 7f2f05292700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0015c1b0 con 0x7f2ee8003a00 2024-03-18T19:08:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.282+0000 7f2f05292700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f000ce1f0 con 0x7f2f0015b7f0 2024-03-18T19:08:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.282+0000 7f2f05292700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f000cd5e0 con 0x7f2f000cc8f0 2024-03-18T19:08:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.282+0000 7f2efeffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ee8003a00 0x7f2f00143a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.282+0000 7f2efeffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ee8003a00 0x7f2f00143a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40422/0 (socket says 172.21.15.1:40422) 2024-03-18T19:08:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.282+0000 7f2efeffd700 1 -- 172.21.15.1:0/581531683 learned_addr learned my addr 172.21.15.1:0/581531683 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.282+0000 7f2eff7fe700 1 --2- 172.21.15.1:0/581531683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f000cc8f0 0x7f2f000ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.282+0000 7f2efe7fc700 1 --2- 172.21.15.1:0/581531683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f0015b7f0 0x7f2f0015bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.282+0000 7f2efeffd700 1 -- 172.21.15.1:0/581531683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f000cc8f0 msgr2=0x7f2f000ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.282+0000 7f2efeffd700 1 --2- 172.21.15.1:0/581531683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f000cc8f0 0x7f2f000ccd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:12.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.282+0000 7f2efeffd700 1 -- 172.21.15.1:0/581531683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f0015b7f0 msgr2=0x7f2f0015bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:12.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.282+0000 7f2efeffd700 1 --2- 172.21.15.1:0/581531683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f0015b7f0 0x7f2f0015bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:12.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.282+0000 7f2efeffd700 1 -- 172.21.15.1:0/581531683 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f000cd8c0 con 0x7f2ee8003a00 2024-03-18T19:08:12.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.282+0000 7f2eff7fe700 1 --2- 172.21.15.1:0/581531683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f000cc8f0 0x7f2f000ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:12.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.282+0000 7f2efe7fc700 1 --2- 172.21.15.1:0/581531683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f0015b7f0 0x7f2f0015bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:12.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.283+0000 7f2efeffd700 1 --2- 172.21.15.1:0/581531683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ee8003a00 0x7f2f00143a50 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f2ee800c080 tx=0x7f2ef40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:12.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.283+0000 7f2ee77fe700 1 -- 172.21.15.1:0/581531683 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef400e6c0 con 0x7f2ee8003a00 2024-03-18T19:08:12.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.283+0000 7f2ee77fe700 1 -- 172.21.15.1:0/581531683 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef4003ad0 con 0x7f2ee8003a00 2024-03-18T19:08:12.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.283+0000 7f2f05292700 1 -- 172.21.15.1:0/581531683 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f000cdba0 con 0x7f2ee8003a00 2024-03-18T19:08:12.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.283+0000 7f2ee77fe700 1 -- 172.21.15.1:0/581531683 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4003e80 con 0x7f2ee8003a00 2024-03-18T19:08:12.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.283+0000 7f2f05292700 1 -- 172.21.15.1:0/581531683 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f000cd2b0 con 0x7f2ee8003a00 2024-03-18T19:08:12.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.284+0000 7f2f05292700 1 -- 172.21.15.1:0/581531683 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f000676a0 con 0x7f2ee8003a00 2024-03-18T19:08:12.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.285+0000 7f2ee77fe700 1 -- 172.21.15.1:0/581531683 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2ef4013470 con 0x7f2ee8003a00 2024-03-18T19:08:12.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.285+0000 7f2ee77fe700 1 --2- 172.21.15.1:0/581531683 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2eec06adb0 0x7f2eec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:12.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.285+0000 7f2ee77fe700 1 -- 172.21.15.1:0/581531683 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f2ef408b4f0 con 0x7f2ee8003a00 2024-03-18T19:08:12.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.285+0000 7f2efe7fc700 1 --2- 172.21.15.1:0/581531683 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2eec06adb0 0x7f2eec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:12.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.286+0000 7f2efe7fc700 1 --2- 172.21.15.1:0/581531683 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2eec06adb0 0x7f2eec06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2f00147300 tx=0x7f2ef0009040 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:12.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.289+0000 7f2ee77fe700 1 -- 172.21.15.1:0/581531683 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2ef4056760 con 0x7f2ee8003a00 2024-03-18T19:08:12.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:12.574+0000 7f2f05292700 1 -- 172.21.15.1:0/581531683 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f2f000afb20 con 0x7f2ee8003a00 2024-03-18T19:08:13.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.131+0000 7f2ee77fe700 1 -- 172.21.15.1:0/581531683 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v694) v1 ==== 151+0+0 (secure 0 0 0) 0x7f2ef40170d0 con 0x7f2ee8003a00 2024-03-18T19:08:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.134+0000 7f2f05292700 1 -- 172.21.15.1:0/581531683 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2eec06adb0 msgr2=0x7f2eec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.134+0000 7f2f05292700 1 --2- 172.21.15.1:0/581531683 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2eec06adb0 0x7f2eec06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2f00147300 tx=0x7f2ef0009040 comp rx=0 tx=0).stop 2024-03-18T19:08:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.134+0000 7f2f05292700 1 -- 172.21.15.1:0/581531683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ee8003a00 msgr2=0x7f2f00143a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.134+0000 7f2f05292700 1 --2- 172.21.15.1:0/581531683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ee8003a00 0x7f2f00143a50 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f2ee800c080 tx=0x7f2ef40047c0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.134+0000 7f2f05292700 1 -- 172.21.15.1:0/581531683 shutdown_connections 2024-03-18T19:08:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.134+0000 7f2f05292700 1 --2- 172.21.15.1:0/581531683 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2eec06adb0 0x7f2eec06d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.134+0000 7f2f05292700 1 --2- 172.21.15.1:0/581531683 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f000cc8f0 0x7f2f000ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.134+0000 7f2f05292700 1 --2- 172.21.15.1:0/581531683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ee8003a00 0x7f2f00143a50 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.134+0000 7f2f05292700 1 --2- 172.21.15.1:0/581531683 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f0015b7f0 0x7f2f0015bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.134+0000 7f2f05292700 1 -- 172.21.15.1:0/581531683 >> 172.21.15.1:0/581531683 conn(0x7f2f000cff00 msgr2=0x7f2f0006a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.134+0000 7f2f05292700 1 -- 172.21.15.1:0/581531683 shutdown_connections 2024-03-18T19:08:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.134+0000 7f2f05292700 1 -- 172.21.15.1:0/581531683 wait complete. 2024-03-18T19:08:13.145 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-18T19:08:13.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.284+0000 7f6cf3155700 1 Processor -- start 2024-03-18T19:08:13.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.284+0000 7f6cf3155700 1 -- start start 2024-03-18T19:08:13.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.285+0000 7f6cf3155700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6cec0b0500 0x7f6cec0cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.285+0000 7f6cf3155700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cec15ef80 con 0x7f6cec15eba0 2024-03-18T19:08:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.285+0000 7f6cf3155700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cec0cc9a0 con 0x7f6cec0b0e20 2024-03-18T19:08:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.285+0000 7f6cf3155700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cec0ccc80 con 0x7f6cec0b0500 2024-03-18T19:08:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.285+0000 7f6cf0ef1700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6cec15eba0 0x7f6cec0cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54858/0 (socket says 172.21.15.1:54858) 2024-03-18T19:08:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.285+0000 7f6cf0ef1700 1 -- 172.21.15.1:0/2750431399 learned_addr learned my addr 172.21.15.1:0/2750431399 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6cebfff700 1 --2- 172.21.15.1:0/2750431399 >> v2:172.21.15.165:3300/0 conn(0x7f6cec0b0500 0x7f6cec0cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6ceb7fe700 1 -- 172.21.15.1:0/2750431399 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608987948 0 0) 0x7f6cec15ef80 con 0x7f6cec15eba0 2024-03-18T19:08:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6ceb7fe700 1 -- 172.21.15.1:0/2750431399 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cdc0035b0 con 0x7f6cec15eba0 2024-03-18T19:08:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6ceb7fe700 1 -- 172.21.15.1:0/2750431399 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 276320256 0 0) 0x7f6cdc0035b0 con 0x7f6cec15eba0 2024-03-18T19:08:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6ceb7fe700 1 -- 172.21.15.1:0/2750431399 >> v2:172.21.15.165:3300/0 conn(0x7f6cec0b0500 msgr2=0x7f6cec0cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6ceb7fe700 1 --2- 172.21.15.1:0/2750431399 >> v2:172.21.15.165:3300/0 conn(0x7f6cec0b0500 0x7f6cec0cfa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6ceb7fe700 1 -- 172.21.15.1:0/2750431399 >> v1:172.21.15.165:6789/0 conn(0x7f6cec0b0e20 legacy=0x7f6cec0b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6ceb7fe700 1 -- 172.21.15.1:0/2750431399 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cec0ccf60 con 0x7f6cec15eba0 2024-03-18T19:08:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6ceb7fe700 1 -- 172.21.15.1:0/2750431399 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f6ce0001670 con 0x7f6cec15eba0 2024-03-18T19:08:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6ceb7fe700 1 -- 172.21.15.1:0/2750431399 >> v1:172.21.15.1:6789/0 conn(0x7f6cec15eba0 legacy=0x7f6cec0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6ceb7fe700 1 --2- 172.21.15.1:0/2750431399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cdc003a00 0x7f6cdc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6ceb7fe700 1 --2- 172.21.15.1:0/2750431399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cdc004850 0x7f6cdc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6ceb7fe700 1 --2- 172.21.15.1:0/2750431399 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cdc005210 0x7f6cdc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6ceb7fe700 1 -- 172.21.15.1:0/2750431399 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec0ccc80 con 0x7f6cdc004850 2024-03-18T19:08:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6ceb7fe700 1 -- 172.21.15.1:0/2750431399 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc00c1b0 con 0x7f6cdc003a00 2024-03-18T19:08:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6ceb7fe700 1 -- 172.21.15.1:0/2750431399 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc00c370 con 0x7f6cdc005210 2024-03-18T19:08:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6cebfff700 1 --2- 172.21.15.1:0/2750431399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cdc004850 0x7f6cdc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6cf0ef1700 1 --2- 172.21.15.1:0/2750431399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cdc003a00 0x7f6cdc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6cf16f2700 1 --2- 172.21.15.1:0/2750431399 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cdc005210 0x7f6cdc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6cebfff700 1 -- 172.21.15.1:0/2750431399 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cdc005210 msgr2=0x7f6cdc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6cebfff700 1 --2- 172.21.15.1:0/2750431399 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cdc005210 0x7f6cdc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6cebfff700 1 -- 172.21.15.1:0/2750431399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cdc003a00 msgr2=0x7f6cdc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6cebfff700 1 --2- 172.21.15.1:0/2750431399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cdc003a00 0x7f6cdc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6cebfff700 1 -- 172.21.15.1:0/2750431399 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc0035b0 con 0x7f6cdc004850 2024-03-18T19:08:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.286+0000 7f6cf16f2700 1 -- 172.21.15.1:0/2750431399 reap_dead start 2024-03-18T19:08:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.287+0000 7f6cebfff700 1 --2- 172.21.15.1:0/2750431399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cdc004850 0x7f6cdc004cd0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8009b60 tx=0x7f6cdc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6be63d3802661c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.287+0000 7f6ceb7fe700 1 -- 172.21.15.1:0/2750431399 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd800e6c0 con 0x7f6cdc004850 2024-03-18T19:08:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.287+0000 7f6ceb7fe700 1 -- 172.21.15.1:0/2750431399 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd800e820 con 0x7f6cdc004850 2024-03-18T19:08:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.287+0000 7f6ceb7fe700 1 -- 172.21.15.1:0/2750431399 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd800eb90 con 0x7f6cdc004850 2024-03-18T19:08:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.287+0000 7f6cf3155700 1 -- 172.21.15.1:0/2750431399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cdc004850 msgr2=0x7f6cdc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.287+0000 7f6cf3155700 1 --2- 172.21.15.1:0/2750431399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cdc004850 0x7f6cdc004cd0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8009b60 tx=0x7f6cdc005880 comp rx=0 tx=0).stop 2024-03-18T19:08:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.287+0000 7f6cf3155700 1 -- 172.21.15.1:0/2750431399 shutdown_connections 2024-03-18T19:08:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.287+0000 7f6cf3155700 1 --2- 172.21.15.1:0/2750431399 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cdc004850 0x7f6cdc004cd0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.287+0000 7f6cf3155700 1 -- 172.21.15.1:0/2750431399 >> 172.21.15.1:0/2750431399 conn(0x7f6cec0b2bd0 msgr2=0x7f6cec0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.287+0000 7f6cf3155700 1 -- 172.21.15.1:0/2750431399 shutdown_connections 2024-03-18T19:08:13.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.287+0000 7f6cf3155700 1 -- 172.21.15.1:0/2750431399 wait complete. 2024-03-18T19:08:13.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.289+0000 7f6cf3155700 1 Processor -- start 2024-03-18T19:08:13.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.289+0000 7f6cf3155700 1 -- start start 2024-03-18T19:08:13.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.290+0000 7f6cf3155700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cdc004850 0x7f6cec0c8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:13.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.290+0000 7f6cf3155700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cec153f40 0x7f6cec14b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:13.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.290+0000 7f6cf3155700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cec1f12a0 0x7f6cec0aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:13.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.290+0000 7f6cf3155700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec1547c0 con 0x7f6cec1f12a0 2024-03-18T19:08:13.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.290+0000 7f6cf3155700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec0c9540 con 0x7f6cdc004850 2024-03-18T19:08:13.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.290+0000 7f6cf3155700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec0c9820 con 0x7f6cec153f40 2024-03-18T19:08:13.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.290+0000 7f6cf16f2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cec1f12a0 0x7f6cec0aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:13.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.290+0000 7f6cf16f2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cec1f12a0 0x7f6cec0aede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40444/0 (socket says 172.21.15.1:40444) 2024-03-18T19:08:13.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.290+0000 7f6cf16f2700 1 -- 172.21.15.1:0/2459747464 learned_addr learned my addr 172.21.15.1:0/2459747464 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:13.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.290+0000 7f6cf0ef1700 1 --2- 172.21.15.1:0/2459747464 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cdc004850 0x7f6cec0c8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:13.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.290+0000 7f6cebfff700 1 --2- 172.21.15.1:0/2459747464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cec153f40 0x7f6cec14b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:13.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.290+0000 7f6cf16f2700 1 -- 172.21.15.1:0/2459747464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cec153f40 msgr2=0x7f6cec14b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:13.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.290+0000 7f6cf16f2700 1 --2- 172.21.15.1:0/2459747464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cec153f40 0x7f6cec14b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.290+0000 7f6cf16f2700 1 -- 172.21.15.1:0/2459747464 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cdc004850 msgr2=0x7f6cec0c8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:13.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.290+0000 7f6cf16f2700 1 --2- 172.21.15.1:0/2459747464 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cdc004850 0x7f6cec0c8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.290+0000 7f6cf16f2700 1 -- 172.21.15.1:0/2459747464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cec0c9b00 con 0x7f6cec1f12a0 2024-03-18T19:08:13.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.291+0000 7f6cf16f2700 1 --2- 172.21.15.1:0/2459747464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cec1f12a0 0x7f6cec0aede0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f6cec0cd870 tx=0x7f6ce400ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:13.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.291+0000 7f6ce97fa700 1 -- 172.21.15.1:0/2459747464 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce400ee80 con 0x7f6cec1f12a0 2024-03-18T19:08:13.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.291+0000 7f6ce97fa700 1 -- 172.21.15.1:0/2459747464 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce4003c10 con 0x7f6cec1f12a0 2024-03-18T19:08:13.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.291+0000 7f6cf3155700 1 -- 172.21.15.1:0/2459747464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cec0c9de0 con 0x7f6cec1f12a0 2024-03-18T19:08:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.291+0000 7f6ce97fa700 1 -- 172.21.15.1:0/2459747464 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce40050d0 con 0x7f6cec1f12a0 2024-03-18T19:08:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.291+0000 7f6cf3155700 1 -- 172.21.15.1:0/2459747464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cec14b9f0 con 0x7f6cec1f12a0 2024-03-18T19:08:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.292+0000 7f6cf3155700 1 -- 172.21.15.1:0/2459747464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cec0676a0 con 0x7f6cec1f12a0 2024-03-18T19:08:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.292+0000 7f6ce97fa700 1 -- 172.21.15.1:0/2459747464 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6ce4005230 con 0x7f6cec1f12a0 2024-03-18T19:08:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.293+0000 7f6ce97fa700 1 --2- 172.21.15.1:0/2459747464 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6cd006adb0 0x7f6cd006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.293+0000 7f6ce97fa700 1 -- 172.21.15.1:0/2459747464 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f6ce4089350 con 0x7f6cec1f12a0 2024-03-18T19:08:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.293+0000 7f6cf0ef1700 1 --2- 172.21.15.1:0/2459747464 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6cd006adb0 0x7f6cd006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.293+0000 7f6cf0ef1700 1 --2- 172.21.15.1:0/2459747464 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6cd006adb0 0x7f6cd006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00017a0 tx=0x7f6ce00015b0 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:13.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.297+0000 7f6ce97fa700 1 -- 172.21.15.1:0/2459747464 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6ce40543c0 con 0x7f6cec1f12a0 2024-03-18T19:08:13.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.499+0000 7f6cf3155700 1 -- 172.21.15.1:0/2459747464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6cec0b2240 con 0x7f6cec1f12a0 2024-03-18T19:08:13.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.502+0000 7f6ce97fa700 1 -- 172.21.15.1:0/2459747464 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6ce40579e0 con 0x7f6cec1f12a0 2024-03-18T19:08:13.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.505+0000 7f6cf3155700 1 -- 172.21.15.1:0/2459747464 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6cd006adb0 msgr2=0x7f6cd006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:13.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.505+0000 7f6cf3155700 1 --2- 172.21.15.1:0/2459747464 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6cd006adb0 0x7f6cd006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00017a0 tx=0x7f6ce00015b0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.505+0000 7f6cf3155700 1 -- 172.21.15.1:0/2459747464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cec1f12a0 msgr2=0x7f6cec0aede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:13.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.505+0000 7f6cf3155700 1 --2- 172.21.15.1:0/2459747464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cec1f12a0 0x7f6cec0aede0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f6cec0cd870 tx=0x7f6ce400ed50 comp rx=0 tx=0).stop 2024-03-18T19:08:13.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.505+0000 7f6cf3155700 1 -- 172.21.15.1:0/2459747464 shutdown_connections 2024-03-18T19:08:13.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.505+0000 7f6cf3155700 1 --2- 172.21.15.1:0/2459747464 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6cd006adb0 0x7f6cd006d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.505+0000 7f6cf3155700 1 --2- 172.21.15.1:0/2459747464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6cec1f12a0 0x7f6cec0aede0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.505+0000 7f6cf3155700 1 --2- 172.21.15.1:0/2459747464 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6cdc004850 0x7f6cec0c8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.505+0000 7f6cf3155700 1 --2- 172.21.15.1:0/2459747464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6cec153f40 0x7f6cec14b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.505+0000 7f6cf3155700 1 -- 172.21.15.1:0/2459747464 >> 172.21.15.1:0/2459747464 conn(0x7f6cec0b2bd0 msgr2=0x7f6cec15d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:13.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.505+0000 7f6cf3155700 1 -- 172.21.15.1:0/2459747464 shutdown_connections 2024-03-18T19:08:13.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.505+0000 7f6cf3155700 1 -- 172.21.15.1:0/2459747464 wait complete. 2024-03-18T19:08:13.513 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T19:08:13.513 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:08:13.550 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:08:13.598 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T19:08:13.598 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:08:13.598 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:08:13.617 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:08:13.617 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:08:13.617 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:08:13.638 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:08:13.638 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:08:13.638 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:08:13.676 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:08:13.676 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:08:13.676 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:08:13.698 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:08:13.698 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-18T19:08:13.699 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:08:13.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.844+0000 7f567f188700 1 Processor -- start 2024-03-18T19:08:13.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.844+0000 7f567f188700 1 -- start start 2024-03-18T19:08:13.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.845+0000 7f567f188700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f567815d580 0x7f5678072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:13.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.845+0000 7f567f188700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f567806e290 0x7f56780cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:13.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.845+0000 7f567f188700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f567806d640 0x7f567806dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:13.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.845+0000 7f567f188700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f567806e010 con 0x7f567815d580 2024-03-18T19:08:13.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.845+0000 7f567f188700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f567806f490 con 0x7f567806d640 2024-03-18T19:08:13.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.845+0000 7f567f188700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5678071cf0 con 0x7f567806e290 2024-03-18T19:08:13.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.845+0000 7f567cf24700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f567815d580 0x7f5678072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:13.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.845+0000 7f567cf24700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f567815d580 0x7f5678072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40460/0 (socket says 172.21.15.1:40460) 2024-03-18T19:08:13.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.845+0000 7f567cf24700 1 -- 172.21.15.1:0/2069650920 learned_addr learned my addr 172.21.15.1:0/2069650920 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:13.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.845+0000 7f5677fff700 1 --2- 172.21.15.1:0/2069650920 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f567806e290 0x7f56780cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:13.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.845+0000 7f567d725700 1 --2- 172.21.15.1:0/2069650920 >> v2:172.21.15.165:3300/0 conn(0x7f567806d640 0x7f567806dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:13.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.846+0000 7f567cf24700 1 -- 172.21.15.1:0/2069650920 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f567806e290 msgr2=0x7f56780cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:13.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.846+0000 7f567cf24700 1 --2- 172.21.15.1:0/2069650920 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f567806e290 0x7f56780cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.846+0000 7f567cf24700 1 -- 172.21.15.1:0/2069650920 >> v2:172.21.15.165:3300/0 conn(0x7f567806d640 msgr2=0x7f567806dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:13.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.846+0000 7f567cf24700 1 --2- 172.21.15.1:0/2069650920 >> v2:172.21.15.165:3300/0 conn(0x7f567806d640 0x7f567806dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.846+0000 7f567cf24700 1 -- 172.21.15.1:0/2069650920 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567806f1b0 con 0x7f567815d580 2024-03-18T19:08:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.846+0000 7f567d725700 1 --2- 172.21.15.1:0/2069650920 >> v2:172.21.15.165:3300/0 conn(0x7f567806d640 0x7f567806dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.846+0000 7f567cf24700 1 --2- 172.21.15.1:0/2069650920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567815d580 0x7f5678072180 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f5678072eb0 tx=0x7f566c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d2cda0b629e437 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.846+0000 7f56777fe700 1 -- 172.21.15.1:0/2069650920 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566c00b870 con 0x7f567815d580 2024-03-18T19:08:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.846+0000 7f56777fe700 1 -- 172.21.15.1:0/2069650920 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566c00b9d0 con 0x7f567815d580 2024-03-18T19:08:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.846+0000 7f56777fe700 1 -- 172.21.15.1:0/2069650920 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566c00bd60 con 0x7f567815d580 2024-03-18T19:08:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.847+0000 7f567f188700 1 -- 172.21.15.1:0/2069650920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567815d580 msgr2=0x7f5678072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.847+0000 7f567f188700 1 --2- 172.21.15.1:0/2069650920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567815d580 0x7f5678072180 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f5678072eb0 tx=0x7f566c009bd0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.847+0000 7f567f188700 1 -- 172.21.15.1:0/2069650920 shutdown_connections 2024-03-18T19:08:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.847+0000 7f567f188700 1 --2- 172.21.15.1:0/2069650920 >> v2:172.21.15.165:3300/0 conn(0x7f567806d640 0x7f567806dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.847+0000 7f567f188700 1 --2- 172.21.15.1:0/2069650920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567815d580 0x7f5678072180 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.847+0000 7f567f188700 1 --2- 172.21.15.1:0/2069650920 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f567806e290 0x7f56780cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.847+0000 7f567f188700 1 -- 172.21.15.1:0/2069650920 >> 172.21.15.1:0/2069650920 conn(0x7f56780cc990 msgr2=0x7f56780ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.847+0000 7f567f188700 1 -- 172.21.15.1:0/2069650920 shutdown_connections 2024-03-18T19:08:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.847+0000 7f567f188700 1 -- 172.21.15.1:0/2069650920 wait complete. 2024-03-18T19:08:13.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.848+0000 7f567f188700 1 Processor -- start 2024-03-18T19:08:13.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.848+0000 7f567f188700 1 -- start start 2024-03-18T19:08:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.849+0000 7f567f188700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f567806d640 0x7f567815ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.849+0000 7f567f188700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567815d580 0x7f56781523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.849+0000 7f567f188700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56780c91b0 0x7f56780c9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.849+0000 7f567f188700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56780c9ba0 con 0x7f567815d580 2024-03-18T19:08:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.849+0000 7f567f188700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56780b53d0 con 0x7f56780c91b0 2024-03-18T19:08:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.849+0000 7f567f188700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567815f130 con 0x7f567806d640 2024-03-18T19:08:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.849+0000 7f5677fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567815d580 0x7f56781523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.849+0000 7f5677fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567815d580 0x7f56781523d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40468/0 (socket says 172.21.15.1:40468) 2024-03-18T19:08:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.849+0000 7f5677fff700 1 -- 172.21.15.1:0/3563475160 learned_addr learned my addr 172.21.15.1:0/3563475160 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.849+0000 7f567cf24700 1 --2- 172.21.15.1:0/3563475160 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f567806d640 0x7f567815ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.849+0000 7f567d725700 1 --2- 172.21.15.1:0/3563475160 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56780c91b0 0x7f56780c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.849+0000 7f5677fff700 1 -- 172.21.15.1:0/3563475160 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f567806d640 msgr2=0x7f567815ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.849+0000 7f5677fff700 1 --2- 172.21.15.1:0/3563475160 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f567806d640 0x7f567815ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.849+0000 7f5677fff700 1 -- 172.21.15.1:0/3563475160 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56780c91b0 msgr2=0x7f56780c9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.849+0000 7f5677fff700 1 --2- 172.21.15.1:0/3563475160 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56780c91b0 0x7f56780c9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.849+0000 7f5677fff700 1 -- 172.21.15.1:0/3563475160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566c009550 con 0x7f567815d580 2024-03-18T19:08:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.849+0000 7f567cf24700 1 --2- 172.21.15.1:0/3563475160 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f567806d640 0x7f567815ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.849+0000 7f5677fff700 1 --2- 172.21.15.1:0/3563475160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567815d580 0x7f56781523d0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f566400e840 tx=0x7f566400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.849+0000 7f56757fa700 1 -- 172.21.15.1:0/3563475160 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5664013610 con 0x7f567815d580 2024-03-18T19:08:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.850+0000 7f56757fa700 1 -- 172.21.15.1:0/3563475160 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5664013770 con 0x7f567815d580 2024-03-18T19:08:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.850+0000 7f567f188700 1 -- 172.21.15.1:0/3563475160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5678152910 con 0x7f567815d580 2024-03-18T19:08:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.850+0000 7f56757fa700 1 -- 172.21.15.1:0/3563475160 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5664013b20 con 0x7f567815d580 2024-03-18T19:08:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.850+0000 7f567f188700 1 -- 172.21.15.1:0/3563475160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56780b4aa0 con 0x7f567815d580 2024-03-18T19:08:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.850+0000 7f567f188700 1 -- 172.21.15.1:0/3563475160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56780b11e0 con 0x7f567815d580 2024-03-18T19:08:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.851+0000 7f56757fa700 1 -- 172.21.15.1:0/3563475160 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5664013c80 con 0x7f567815d580 2024-03-18T19:08:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.851+0000 7f56757fa700 1 --2- 172.21.15.1:0/3563475160 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f565c06adb0 0x7f565c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.851+0000 7f56757fa700 1 -- 172.21.15.1:0/3563475160 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f56640892b0 con 0x7f567815d580 2024-03-18T19:08:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.852+0000 7f567cf24700 1 --2- 172.21.15.1:0/3563475160 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f565c06adb0 0x7f565c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.852+0000 7f567cf24700 1 --2- 172.21.15.1:0/3563475160 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f565c06adb0 0x7f565c06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f566c014040 tx=0x7f566c008030 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:13.856+0000 7f56757fa700 1 -- 172.21.15.1:0/3563475160 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5664054180 con 0x7f567815d580 2024-03-18T19:08:14.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.103+0000 7f567f188700 1 -- 172.21.15.1:0/3563475160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56780676a0 con 0x7f567815d580 2024-03-18T19:08:14.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.104+0000 7f56757fa700 1 -- 172.21.15.1:0/3563475160 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+3526 (secure 0 0 0) 0x7f5664021de0 con 0x7f567815d580 2024-03-18T19:08:14.100 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:14.100 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":51,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6431,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/983420410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":983420410},{"type":"v1","addr":"172.21.15.1:6836","nonce":983420410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6440,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1569519191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1569519191},{"type":"v1","addr":"172.21.15.165:6834","nonce":1569519191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7237,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1528229019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1528229019},{"type":"v1","addr":"172.21.15.165:6835","nonce":1528229019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":8214,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1392315794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1392315794},{"type":"v1","addr":"172.21.15.1:6837","nonce":1392315794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[]} 2024-03-18T19:08:14.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.106+0000 7f567f188700 1 -- 172.21.15.1:0/3563475160 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f565c06adb0 msgr2=0x7f565c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:14.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.106+0000 7f567f188700 1 --2- 172.21.15.1:0/3563475160 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f565c06adb0 0x7f565c06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f566c014040 tx=0x7f566c008030 comp rx=0 tx=0).stop 2024-03-18T19:08:14.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.106+0000 7f567f188700 1 -- 172.21.15.1:0/3563475160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567815d580 msgr2=0x7f56781523d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:14.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.106+0000 7f567f188700 1 --2- 172.21.15.1:0/3563475160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567815d580 0x7f56781523d0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f566400e840 tx=0x7f566400ec00 comp rx=0 tx=0).stop 2024-03-18T19:08:14.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.106+0000 7f567f188700 1 -- 172.21.15.1:0/3563475160 shutdown_connections 2024-03-18T19:08:14.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.106+0000 7f567f188700 1 --2- 172.21.15.1:0/3563475160 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f565c06adb0 0x7f565c06d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:14.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.106+0000 7f567f188700 1 --2- 172.21.15.1:0/3563475160 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f567806d640 0x7f567815ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:14.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.106+0000 7f567f188700 1 --2- 172.21.15.1:0/3563475160 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56780c91b0 0x7f56780c9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:14.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.106+0000 7f567f188700 1 --2- 172.21.15.1:0/3563475160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567815d580 0x7f56781523d0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:14.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.106+0000 7f567f188700 1 -- 172.21.15.1:0/3563475160 >> 172.21.15.1:0/3563475160 conn(0x7f56780cc990 msgr2=0x7f56780b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:14.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.106+0000 7f567f188700 1 -- 172.21.15.1:0/3563475160 shutdown_connections 2024-03-18T19:08:14.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.106+0000 7f567f188700 1 -- 172.21.15.1:0/3563475160 wait complete. 2024-03-18T19:08:14.104 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 51 2024-03-18T19:08:14.114 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T19:08:14.114 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-18T19:08:14.115 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T19:08:14.115 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:08:14.115 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T19:08:14.115 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-18T19:08:14.115 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-18T19:08:14.115 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:08:14.115 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T19:08:14.116 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-18T19:08:14.116 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-18T19:08:14.116 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:08:14.116 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T19:08:14.116 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-18T19:08:14.116 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-18T19:08:14.116 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:08:14.123 INFO:tasks.ceph.mds.a:Stopped 2024-03-18T19:08:14.123 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-18T19:08:14.124 INFO:tasks.ceph.mds.c:Stopped 2024-03-18T19:08:14.124 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-18T19:08:14.127 INFO:tasks.ceph.mds.b:Stopped 2024-03-18T19:08:14.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-18T19:08:14.128 INFO:tasks.ceph.mds.d:Stopped 2024-03-18T19:08:14.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-18T19:08:14.166 INFO:tasks.ceph.mds.a:Started 2024-03-18T19:08:14.166 DEBUG:teuthology.parallel:result is None 2024-03-18T19:08:14.167 INFO:tasks.ceph.mds.c:Started 2024-03-18T19:08:14.167 DEBUG:teuthology.parallel:result is None 2024-03-18T19:08:14.169 INFO:tasks.ceph.mds.b:Started 2024-03-18T19:08:14.170 DEBUG:teuthology.parallel:result is None 2024-03-18T19:08:14.170 INFO:tasks.ceph.mds.d:Started 2024-03-18T19:08:14.170 DEBUG:teuthology.parallel:result is None 2024-03-18T19:08:14.171 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-18T19:08:14.250 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T19:08:14.250 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-18T19:08:14.256 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-03-18T19:08:14.256 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-03-18T19:08:14.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.308+0000 7f5c71d44700 1 Processor -- start 2024-03-18T19:08:14.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.308+0000 7f5c71d44700 1 -- start start 2024-03-18T19:08:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.309+0000 7f5c71d44700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6c06d560 0x7f5c6c0aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.309+0000 7f5c71d44700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5c6c0af510 0x7f5c6c15b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.309+0000 7f5c71d44700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c6c15ef00 con 0x7f5c6c06a750 2024-03-18T19:08:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.309+0000 7f5c71d44700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c6c15f1e0 con 0x7f5c6c0af510 2024-03-18T19:08:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.309+0000 7f5c71d44700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c15f4c0 con 0x7f5c6c06d560 2024-03-18T19:08:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.309+0000 7f5c6b7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5c6c06a750 0x7f5c6c06ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54884/0 (socket says 172.21.15.1:54884) 2024-03-18T19:08:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.309+0000 7f5c6b7fe700 1 -- 172.21.15.1:0/132009409 learned_addr learned my addr 172.21.15.1:0/132009409 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.309+0000 7f5c6bfff700 1 --2- 172.21.15.1:0/132009409 >> v2:172.21.15.165:3300/0 conn(0x7f5c6c0af510 0x7f5c6c15b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.309+0000 7f5c6affd700 1 --2- 172.21.15.1:0/132009409 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6c06d560 0x7f5c6c0aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.309+0000 7f5c6a7fc700 1 -- 172.21.15.1:0/132009409 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708502570 0 0) 0x7f5c6c15ef00 con 0x7f5c6c06a750 2024-03-18T19:08:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.309+0000 7f5c6a7fc700 1 -- 172.21.15.1:0/132009409 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c540035b0 con 0x7f5c6c06a750 2024-03-18T19:08:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.309+0000 7f5c6a7fc700 1 -- 172.21.15.1:0/132009409 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3982907812 0 0) 0x7f5c540035b0 con 0x7f5c6c06a750 2024-03-18T19:08:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.309+0000 7f5c6bfff700 1 -- 172.21.15.1:0/132009409 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6c06d560 msgr2=0x7f5c6c0aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.309+0000 7f5c6bfff700 1 --2- 172.21.15.1:0/132009409 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6c06d560 0x7f5c6c0aefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.309+0000 7f5c6bfff700 1 -- 172.21.15.1:0/132009409 >> v1:172.21.15.1:6789/0 conn(0x7f5c6c06a750 legacy=0x7f5c6c06ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.309+0000 7f5c6bfff700 1 -- 172.21.15.1:0/132009409 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6c15f7a0 con 0x7f5c6c0af510 2024-03-18T19:08:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.309+0000 7f5c6affd700 1 --2- 172.21.15.1:0/132009409 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6c06d560 0x7f5c6c0aefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:08:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.310+0000 7f5c6bfff700 1 --2- 172.21.15.1:0/132009409 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c6c0af510 0x7f5c6c15b900 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c009840 tx=0x7f5c5c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d647c15ea8b4eec0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.310+0000 7f5c6a7fc700 1 -- 172.21.15.1:0/132009409 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c640019f0 con 0x7f5c6c0af510 2024-03-18T19:08:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.310+0000 7f5c6a7fc700 1 -- 172.21.15.1:0/132009409 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c5c004030 con 0x7f5c6c0af510 2024-03-18T19:08:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.310+0000 7f5c6a7fc700 1 -- 172.21.15.1:0/132009409 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c0043c0 con 0x7f5c6c0af510 2024-03-18T19:08:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.310+0000 7f5c71d44700 1 -- 172.21.15.1:0/132009409 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c6c0af510 msgr2=0x7f5c6c15b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.310+0000 7f5c71d44700 1 --2- 172.21.15.1:0/132009409 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c6c0af510 0x7f5c6c15b900 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c009840 tx=0x7f5c5c009b50 comp rx=0 tx=0).stop 2024-03-18T19:08:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.310+0000 7f5c71d44700 1 -- 172.21.15.1:0/132009409 shutdown_connections 2024-03-18T19:08:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.310+0000 7f5c71d44700 1 --2- 172.21.15.1:0/132009409 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c6c0af510 0x7f5c6c15b900 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.310+0000 7f5c71d44700 1 --2- 172.21.15.1:0/132009409 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6c06d560 0x7f5c6c0aefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:14.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.310+0000 7f5c71d44700 1 -- 172.21.15.1:0/132009409 >> 172.21.15.1:0/132009409 conn(0x7f5c6c0cff30 msgr2=0x7f5c6c0d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:14.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.310+0000 7f5c71d44700 1 -- 172.21.15.1:0/132009409 shutdown_connections 2024-03-18T19:08:14.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.310+0000 7f5c71d44700 1 -- 172.21.15.1:0/132009409 wait complete. 2024-03-18T19:08:14.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.312+0000 7f5c71d44700 1 Processor -- start 2024-03-18T19:08:14.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.312+0000 7f5c71d44700 1 -- start start 2024-03-18T19:08:14.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.313+0000 7f5c71d44700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6c06a750 0x7f5c6c1e5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:14.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.313+0000 7f5c71d44700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c6c06d560 0x7f5c6c1e65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:14.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.313+0000 7f5c71d44700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6c0af510 0x7f5c6c146fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:14.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.313+0000 7f5c71d44700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c1e5b30 con 0x7f5c6c06a750 2024-03-18T19:08:14.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.313+0000 7f5c71d44700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c0afce0 con 0x7f5c6c06d560 2024-03-18T19:08:14.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.313+0000 7f5c71d44700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c1e5dd0 con 0x7f5c6c0af510 2024-03-18T19:08:14.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.313+0000 7f5c6b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6c06a750 0x7f5c6c1e5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:14.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.313+0000 7f5c6b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6c06a750 0x7f5c6c1e5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40526/0 (socket says 172.21.15.1:40526) 2024-03-18T19:08:14.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.313+0000 7f5c6b7fe700 1 -- 172.21.15.1:0/1315496766 learned_addr learned my addr 172.21.15.1:0/1315496766 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:14.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.313+0000 7f5c6affd700 1 --2- 172.21.15.1:0/1315496766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c6c06d560 0x7f5c6c1e65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:14.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.313+0000 7f5c6bfff700 1 --2- 172.21.15.1:0/1315496766 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6c0af510 0x7f5c6c146fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:14.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.313+0000 7f5c6b7fe700 1 -- 172.21.15.1:0/1315496766 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6c0af510 msgr2=0x7f5c6c146fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.313+0000 7f5c6b7fe700 1 --2- 172.21.15.1:0/1315496766 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6c0af510 0x7f5c6c146fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.313+0000 7f5c6b7fe700 1 -- 172.21.15.1:0/1315496766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c6c06d560 msgr2=0x7f5c6c1e65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.314+0000 7f5c6b7fe700 1 --2- 172.21.15.1:0/1315496766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c6c06d560 0x7f5c6c1e65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.314+0000 7f5c6b7fe700 1 -- 172.21.15.1:0/1315496766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c600094a0 con 0x7f5c6c06a750 2024-03-18T19:08:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.314+0000 7f5c6b7fe700 1 --2- 172.21.15.1:0/1315496766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6c06a750 0x7f5c6c1e5490 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f5c640019b0 tx=0x7f5c6400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.314+0000 7f5c53fff700 1 -- 172.21.15.1:0/1315496766 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6400acd0 con 0x7f5c6c06a750 2024-03-18T19:08:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.314+0000 7f5c53fff700 1 -- 172.21.15.1:0/1315496766 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6400ae30 con 0x7f5c6c06a750 2024-03-18T19:08:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.314+0000 7f5c71d44700 1 -- 172.21.15.1:0/1315496766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5c0094a0 con 0x7f5c6c06a750 2024-03-18T19:08:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.314+0000 7f5c53fff700 1 -- 172.21.15.1:0/1315496766 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c64007310 con 0x7f5c6c06a750 2024-03-18T19:08:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.314+0000 7f5c71d44700 1 -- 172.21.15.1:0/1315496766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c6c0affc0 con 0x7f5c6c06a750 2024-03-18T19:08:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.315+0000 7f5c71d44700 1 -- 172.21.15.1:0/1315496766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c6c0676a0 con 0x7f5c6c06a750 2024-03-18T19:08:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.315+0000 7f5c53fff700 1 -- 172.21.15.1:0/1315496766 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5c64007470 con 0x7f5c6c06a750 2024-03-18T19:08:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.316+0000 7f5c53fff700 1 --2- 172.21.15.1:0/1315496766 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5c5806adb0 0x7f5c5806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.316+0000 7f5c53fff700 1 -- 172.21.15.1:0/1315496766 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5951+0+0 (secure 0 0 0) 0x7f5c640894b0 con 0x7f5c6c06a750 2024-03-18T19:08:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.316+0000 7f5c6affd700 1 --2- 172.21.15.1:0/1315496766 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5c5806adb0 0x7f5c5806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:14.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.316+0000 7f5c6affd700 1 --2- 172.21.15.1:0/1315496766 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5c5806adb0 0x7f5c5806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c0ca360 tx=0x7f5c60009210 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:14.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.320+0000 7f5c53fff700 1 -- 172.21.15.1:0/1315496766 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c64054490 con 0x7f5c6c06a750 2024-03-18T19:08:14.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:14.541+0000 7f5c71d44700 1 -- 172.21.15.1:0/1315496766 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5c6c0b5eb0 con 0x7f5c6c06a750 2024-03-18T19:08:15.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.505+0000 7f5c53fff700 1 -- 172.21.15.1:0/1315496766 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v129) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5c64057ab0 con 0x7f5c6c06a750 2024-03-18T19:08:15.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.508+0000 7f5c71d44700 1 -- 172.21.15.1:0/1315496766 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5c5806adb0 msgr2=0x7f5c5806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:15.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.508+0000 7f5c71d44700 1 --2- 172.21.15.1:0/1315496766 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5c5806adb0 0x7f5c5806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c0ca360 tx=0x7f5c60009210 comp rx=0 tx=0).stop 2024-03-18T19:08:15.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.508+0000 7f5c71d44700 1 -- 172.21.15.1:0/1315496766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6c06a750 msgr2=0x7f5c6c1e5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:15.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.508+0000 7f5c71d44700 1 --2- 172.21.15.1:0/1315496766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6c06a750 0x7f5c6c1e5490 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f5c640019b0 tx=0x7f5c6400a340 comp rx=0 tx=0).stop 2024-03-18T19:08:15.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.508+0000 7f5c71d44700 1 -- 172.21.15.1:0/1315496766 shutdown_connections 2024-03-18T19:08:15.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.508+0000 7f5c71d44700 1 --2- 172.21.15.1:0/1315496766 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5c5806adb0 0x7f5c5806d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:15.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.508+0000 7f5c71d44700 1 --2- 172.21.15.1:0/1315496766 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c6c0af510 0x7f5c6c146fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:15.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.508+0000 7f5c71d44700 1 --2- 172.21.15.1:0/1315496766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6c06a750 0x7f5c6c1e5490 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.508+0000 7f5c71d44700 1 --2- 172.21.15.1:0/1315496766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c6c06d560 0x7f5c6c1e65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.509+0000 7f5c71d44700 1 -- 172.21.15.1:0/1315496766 >> 172.21.15.1:0/1315496766 conn(0x7f5c6c0cff30 msgr2=0x7f5c6c0ca560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.509+0000 7f5c71d44700 1 -- 172.21.15.1:0/1315496766 shutdown_connections 2024-03-18T19:08:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.509+0000 7f5c71d44700 1 -- 172.21.15.1:0/1315496766 wait complete. 2024-03-18T19:08:15.506 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-18T19:08:15.520 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-18T19:08:15.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.656+0000 7fe59412b700 1 Processor -- start 2024-03-18T19:08:15.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.656+0000 7fe59412b700 1 -- start start 2024-03-18T19:08:15.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.657+0000 7fe59412b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe58c0cfab0 0x7fe58c15c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:15.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.657+0000 7fe59412b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe58c140b20 0x7fe58c0cfe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:15.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.657+0000 7fe59412b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe58c15c660 con 0x7fe58c0cdf80 2024-03-18T19:08:15.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.657+0000 7fe59412b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe58c0d03d0 con 0x7fe58c140b20 2024-03-18T19:08:15.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.657+0000 7fe59412b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c15fbe0 con 0x7fe58c0cfab0 2024-03-18T19:08:15.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.657+0000 7fe5916c6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe58c140b20 0x7fe58c0cfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:15.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.657+0000 7fe5916c6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe58c140b20 0x7fe58c0cfe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59196/0 (socket says 172.21.15.1:59196) 2024-03-18T19:08:15.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.657+0000 7fe5916c6700 1 -- 172.21.15.1:0/1288238207 learned_addr learned my addr 172.21.15.1:0/1288238207 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:15.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.657+0000 7fe591ec7700 1 --2- 172.21.15.1:0/1288238207 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe58c0cfab0 0x7fe58c15c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:15.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.657+0000 7fe5916c6700 1 -- 172.21.15.1:0/1288238207 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe58c0cfab0 msgr2=0x7fe58c15c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:15.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.657+0000 7fe5916c6700 1 --2- 172.21.15.1:0/1288238207 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe58c0cfab0 0x7fe58c15c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:15.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.657+0000 7fe5916c6700 1 -- 172.21.15.1:0/1288238207 >> v1:172.21.15.165:6789/0 conn(0x7fe58c0cdf80 legacy=0x7fe58c070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:15.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.657+0000 7fe5916c6700 1 -- 172.21.15.1:0/1288238207 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58c15f060 con 0x7fe58c140b20 2024-03-18T19:08:15.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.657+0000 7fe5916c6700 1 --2- 172.21.15.1:0/1288238207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe58c140b20 0x7fe58c0cfe90 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fe5880098f0 tx=0x7fe588009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=357778a9974857a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:15.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.657+0000 7fe590ec5700 1 -- 172.21.15.1:0/1288238207 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58800b8a0 con 0x7fe58c140b20 2024-03-18T19:08:15.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.657+0000 7fe590ec5700 1 -- 172.21.15.1:0/1288238207 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe58800ba00 con 0x7fe58c140b20 2024-03-18T19:08:15.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.657+0000 7fe590ec5700 1 -- 172.21.15.1:0/1288238207 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58800bd90 con 0x7fe58c140b20 2024-03-18T19:08:15.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.658+0000 7fe59412b700 1 -- 172.21.15.1:0/1288238207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe58c140b20 msgr2=0x7fe58c0cfe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:15.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.658+0000 7fe59412b700 1 --2- 172.21.15.1:0/1288238207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe58c140b20 0x7fe58c0cfe90 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fe5880098f0 tx=0x7fe588009c00 comp rx=0 tx=0).stop 2024-03-18T19:08:15.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.658+0000 7fe59412b700 1 -- 172.21.15.1:0/1288238207 shutdown_connections 2024-03-18T19:08:15.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.658+0000 7fe59412b700 1 --2- 172.21.15.1:0/1288238207 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe58c0cfab0 0x7fe58c15c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:15.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.658+0000 7fe59412b700 1 --2- 172.21.15.1:0/1288238207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe58c140b20 0x7fe58c0cfe90 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:15.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.658+0000 7fe59412b700 1 -- 172.21.15.1:0/1288238207 >> 172.21.15.1:0/1288238207 conn(0x7fe58c06ab70 msgr2=0x7fe58c06af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:15.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.658+0000 7fe59412b700 1 -- 172.21.15.1:0/1288238207 shutdown_connections 2024-03-18T19:08:15.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.658+0000 7fe59412b700 1 -- 172.21.15.1:0/1288238207 wait complete. 2024-03-18T19:08:15.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.659+0000 7fe59412b700 1 Processor -- start 2024-03-18T19:08:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.659+0000 7fe59412b700 1 -- start start 2024-03-18T19:08:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.660+0000 7fe59412b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe58c0cdf80 0x7fe58c13e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.660+0000 7fe59412b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe58c0cfab0 0x7fe58c157e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.660+0000 7fe59412b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe58c15db90 0x7fe58c13bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.660+0000 7fe59412b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c15c850 con 0x7fe58c0cfab0 2024-03-18T19:08:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.660+0000 7fe59412b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c13e970 con 0x7fe58c0cdf80 2024-03-18T19:08:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.660+0000 7fe59412b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c15caf0 con 0x7fe58c15db90 2024-03-18T19:08:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.660+0000 7fe5916c6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe58c0cfab0 0x7fe58c157e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.660+0000 7fe5916c6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe58c0cfab0 0x7fe58c157e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59198/0 (socket says 172.21.15.1:59198) 2024-03-18T19:08:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.660+0000 7fe5916c6700 1 -- 172.21.15.1:0/1950202244 learned_addr learned my addr 172.21.15.1:0/1950202244 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.660+0000 7fe591ec7700 1 --2- 172.21.15.1:0/1950202244 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe58c0cdf80 0x7fe58c13e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.660+0000 7fe5926c8700 1 --2- 172.21.15.1:0/1950202244 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe58c15db90 0x7fe58c13bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.661+0000 7fe5916c6700 1 -- 172.21.15.1:0/1950202244 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe58c15db90 msgr2=0x7fe58c13bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:15.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.661+0000 7fe5916c6700 1 --2- 172.21.15.1:0/1950202244 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe58c15db90 0x7fe58c13bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:15.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.661+0000 7fe5916c6700 1 -- 172.21.15.1:0/1950202244 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe58c0cdf80 msgr2=0x7fe58c13e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:15.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.661+0000 7fe5916c6700 1 --2- 172.21.15.1:0/1950202244 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe58c0cdf80 0x7fe58c13e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:15.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.661+0000 7fe5916c6700 1 -- 172.21.15.1:0/1950202244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe588009550 con 0x7fe58c0cfab0 2024-03-18T19:08:15.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.661+0000 7fe5916c6700 1 --2- 172.21.15.1:0/1950202244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe58c0cfab0 0x7fe58c157e10 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fe588009c40 tx=0x7fe588004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:15.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.661+0000 7fe5827fc700 1 -- 172.21.15.1:0/1950202244 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5880044d0 con 0x7fe58c0cfab0 2024-03-18T19:08:15.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.661+0000 7fe5827fc700 1 -- 172.21.15.1:0/1950202244 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe58801e070 con 0x7fe58c0cfab0 2024-03-18T19:08:15.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.661+0000 7fe59412b700 1 -- 172.21.15.1:0/1950202244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58c15cdd0 con 0x7fe58c0cfab0 2024-03-18T19:08:15.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.661+0000 7fe5827fc700 1 -- 172.21.15.1:0/1950202244 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5880038e0 con 0x7fe58c0cfab0 2024-03-18T19:08:15.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.661+0000 7fe59412b700 1 -- 172.21.15.1:0/1950202244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe58c15d390 con 0x7fe58c0cfab0 2024-03-18T19:08:15.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.662+0000 7fe59412b700 1 -- 172.21.15.1:0/1950202244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe58c0676a0 con 0x7fe58c0cfab0 2024-03-18T19:08:15.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.662+0000 7fe5827fc700 1 -- 172.21.15.1:0/1950202244 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe588003a40 con 0x7fe58c0cfab0 2024-03-18T19:08:15.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.663+0000 7fe5827fc700 1 --2- 172.21.15.1:0/1950202244 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe57006adb0 0x7fe57006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:15.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.663+0000 7fe5827fc700 1 -- 172.21.15.1:0/1950202244 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fe588020070 con 0x7fe58c0cfab0 2024-03-18T19:08:15.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.663+0000 7fe591ec7700 1 --2- 172.21.15.1:0/1950202244 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe57006adb0 0x7fe57006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:15.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.663+0000 7fe591ec7700 1 --2- 172.21.15.1:0/1950202244 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe57006adb0 0x7fe57006d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe58c158fa0 tx=0x7fe57c008040 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:15.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.668+0000 7fe5827fc700 1 -- 172.21.15.1:0/1950202244 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe588055a60 con 0x7fe58c0cfab0 2024-03-18T19:08:15.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.951+0000 7fe59412b700 1 -- 172.21.15.1:0/1950202244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe58c06cf10 con 0x7fe58c0cfab0 2024-03-18T19:08:15.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.951+0000 7fe5827fc700 1 -- 172.21.15.1:0/1950202244 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fe588059080 con 0x7fe58c0cfab0 2024-03-18T19:08:15.948 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:15.948 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:08:15.948 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-18T19:08:15.948 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:15.948 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-18T19:08:15.948 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlR93/CxAAbpe+5n3wHRur9xi8QzdCnQ==", 2024-03-18T19:08:15.948 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:08:15.948 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:08:15.948 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:08:15.948 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:08:15.948 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:08:15.948 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:15.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:15.949 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-18T19:08:15.949 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJ3OmGBAA37u227nxrMsh/JHt71HBxQ==", 2024-03-18T19:08:15.949 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:08:15.949 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:08:15.949 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:08:15.949 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:08:15.949 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:08:15.949 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:15.949 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:15.949 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-18T19:08:15.949 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJZFoExAAmI16oxwNKbf8dKetPfyk+g==", 2024-03-18T19:08:15.949 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:08:15.949 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:08:15.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:08:15.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:08:15.950 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:08:15.950 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:15.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:15.950 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-18T19:08:15.950 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlQrcvIBAA1LMv+EgC7S6wspX6QF49Pg==", 2024-03-18T19:08:15.950 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:08:15.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:08:15.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:08:15.951 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:08:15.951 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:08:15.951 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:15.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:15.951 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-18T19:08:15.951 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCUjPhlYOVHChAARFQrksZmMIdTlntd8TgvrQ==", 2024-03-18T19:08:15.951 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:08:15.951 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:08:15.951 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:08:15.951 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:08:15.951 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:15.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:15.952 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-18T19:08:15.952 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCXjPhl2zSDCBAA6Al1gkWVB79ZI9djGpVChA==", 2024-03-18T19:08:15.952 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:08:15.952 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:08:15.952 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:08:15.952 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:08:15.952 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:15.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:15.952 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-18T19:08:15.952 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCajPhlSoG/BhAAR93USDe6DCQW+hREOsnTlw==", 2024-03-18T19:08:15.952 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:08:15.952 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:08:15.952 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:08:15.953 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:08:15.953 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:15.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:15.953 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-18T19:08:15.953 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCdjPhlhJg5BRAAWuQhcnxyjR5UOiM13yKdAQ==", 2024-03-18T19:08:15.953 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:08:15.953 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:08:15.953 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:08:15.953 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:08:15.953 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:15.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:15.953 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-18T19:08:15.954 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCjjPhlCFnpCRAAKIyOQyVdjXZUTnAS6Nw/ng==", 2024-03-18T19:08:15.954 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:08:15.954 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:08:15.954 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:08:15.954 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:08:15.954 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:15.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:15.954 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-18T19:08:15.954 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCmjPhlAeFkCBAAJcEFpRk/03lQJxj+l2hEPA==", 2024-03-18T19:08:15.954 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:08:15.954 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:08:15.954 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:08:15.954 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:08:15.955 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:15.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:15.955 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-18T19:08:15.955 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCpjPhl/xuuBhAA0LgKfXEvN2O11ZlRQJVOkw==", 2024-03-18T19:08:15.955 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:08:15.955 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:08:15.955 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:08:15.955 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:08:15.955 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:15.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:15.955 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-18T19:08:15.955 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCsjPhldiR4BRAA571oh6U8PzOv/4LzjYLF/A==", 2024-03-18T19:08:15.956 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:08:15.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:08:15.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:08:15.956 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:08:15.956 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:15.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:15.957 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-18T19:08:15.957 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlu+2xJBAANnXGGjfeqP6S2FTecsfOsg==", 2024-03-18T19:08:15.957 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:08:15.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:08:15.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-18T19:08:15.957 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-18T19:08:15.957 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:15.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:15.958 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-18T19:08:15.958 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlhJ9+KxAAEnrQrCrgJR6xI+pucwzAXA==", 2024-03-18T19:08:15.958 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:08:15.958 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:08:15.958 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:08:15.958 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:08:15.958 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:08:15.958 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:15.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:15.959 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.2", 2024-03-18T19:08:15.959 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlxBUpKBAAIGTwmLWyj+P2Yvl55QVcWA==", 2024-03-18T19:08:15.959 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:08:15.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:08:15.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:08:15.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:08:15.959 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:08:15.959 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:15.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:15.959 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.3", 2024-03-18T19:08:15.959 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlmao3MRAAHeiQ6eyBh9QjjKPCtxkC8g==", 2024-03-18T19:08:15.959 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:08:15.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:08:15.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:08:15.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:08:15.960 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:08:15.960 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:15.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:15.960 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-18T19:08:15.960 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCRjPhlemOuKRAAVUggmtB7+EiVgyfJZ8TCCA==", 2024-03-18T19:08:15.960 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:08:15.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:08:15.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:08:15.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:08:15.960 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:08:15.961 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:15.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:15.961 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-18T19:08:15.961 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhleIuOCBAA9CH+AHewQbn+gpVr+6lJyw==", 2024-03-18T19:08:15.961 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:08:15.961 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:08:15.961 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T19:08:15.961 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:08:15.961 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:15.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:15.961 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-18T19:08:15.961 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlL0pTBBAAdFiPcdwwz514cuLAvioxYA==", 2024-03-18T19:08:15.961 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:08:15.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:08:15.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T19:08:15.962 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:08:15.962 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.962 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:15.962 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-18T19:08:15.962 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:08:15.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.953+0000 7fe59412b700 1 -- 172.21.15.1:0/1950202244 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe57006adb0 msgr2=0x7fe57006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:15.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.953+0000 7fe59412b700 1 --2- 172.21.15.1:0/1950202244 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe57006adb0 0x7fe57006d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe58c158fa0 tx=0x7fe57c008040 comp rx=0 tx=0).stop 2024-03-18T19:08:15.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.953+0000 7fe59412b700 1 -- 172.21.15.1:0/1950202244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe58c0cfab0 msgr2=0x7fe58c157e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:15.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.953+0000 7fe59412b700 1 --2- 172.21.15.1:0/1950202244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe58c0cfab0 0x7fe58c157e10 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fe588009c40 tx=0x7fe588004750 comp rx=0 tx=0).stop 2024-03-18T19:08:15.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.953+0000 7fe59412b700 1 -- 172.21.15.1:0/1950202244 shutdown_connections 2024-03-18T19:08:15.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.953+0000 7fe59412b700 1 --2- 172.21.15.1:0/1950202244 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe57006adb0 0x7fe57006d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:15.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.953+0000 7fe59412b700 1 --2- 172.21.15.1:0/1950202244 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe58c15db90 0x7fe58c13bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:15.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.953+0000 7fe59412b700 1 --2- 172.21.15.1:0/1950202244 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe58c0cdf80 0x7fe58c13e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:15.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.953+0000 7fe59412b700 1 --2- 172.21.15.1:0/1950202244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe58c0cfab0 0x7fe58c157e10 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:15.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.953+0000 7fe59412b700 1 -- 172.21.15.1:0/1950202244 >> 172.21.15.1:0/1950202244 conn(0x7fe58c06ab70 msgr2=0x7fe58c13b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:15.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.954+0000 7fe59412b700 1 -- 172.21.15.1:0/1950202244 shutdown_connections 2024-03-18T19:08:15.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:15.954+0000 7fe59412b700 1 -- 172.21.15.1:0/1950202244 wait complete. 2024-03-18T19:08:15.965 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:08:16.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.095+0000 7fb80472a700 1 Processor -- start 2024-03-18T19:08:16.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.095+0000 7fb80472a700 1 -- start start 2024-03-18T19:08:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.096+0000 7fb80472a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7fc15d680 0x7fb7fc0cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.096+0000 7fb80472a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb7fc09a150 0x7fb7fc0cc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.096+0000 7fb80472a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc0d0c80 con 0x7fb7fc0d04b0 2024-03-18T19:08:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.096+0000 7fb80472a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7fc0d09a0 con 0x7fb7fc09a150 2024-03-18T19:08:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.096+0000 7fb80472a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc0cb830 con 0x7fb7fc15d680 2024-03-18T19:08:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.096+0000 7fb802cc7700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb7fc09a150 0x7fb7fc0cc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.096+0000 7fb801cc5700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fb7fc0d04b0 0x7fb7fc099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:37810/0 (socket says 172.21.15.1:37810) 2024-03-18T19:08:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.096+0000 7fb8024c6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7fc15d680 0x7fb7fc0cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.096+0000 7fb801cc5700 1 -- 172.21.15.1:0/120826322 learned_addr learned my addr 172.21.15.1:0/120826322 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.096+0000 7fb8014c4700 1 -- 172.21.15.1:0/120826322 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852515721 0 0) 0x7fb7fc0d0c80 con 0x7fb7fc0d04b0 2024-03-18T19:08:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.096+0000 7fb8014c4700 1 -- 172.21.15.1:0/120826322 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7e80035b0 con 0x7fb7fc0d04b0 2024-03-18T19:08:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.096+0000 7fb802cc7700 1 -- 172.21.15.1:0/120826322 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7fc15d680 msgr2=0x7fb7fc0cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.096+0000 7fb802cc7700 1 --2- 172.21.15.1:0/120826322 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7fc15d680 0x7fb7fc0cff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.096+0000 7fb802cc7700 1 -- 172.21.15.1:0/120826322 >> v1:172.21.15.165:6789/0 conn(0x7fb7fc0d04b0 legacy=0x7fb7fc099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:16.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.096+0000 7fb802cc7700 1 -- 172.21.15.1:0/120826322 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc0cbb10 con 0x7fb7fc09a150 2024-03-18T19:08:16.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.097+0000 7fb802cc7700 1 --2- 172.21.15.1:0/120826322 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7fc09a150 0x7fb7fc0cc310 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb7f4009d90 tx=0x7fb7f400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5015650a18e4aba2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:16.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.097+0000 7fb8014c4700 1 -- 172.21.15.1:0/120826322 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f400e070 con 0x7fb7fc09a150 2024-03-18T19:08:16.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.097+0000 7fb8014c4700 1 -- 172.21.15.1:0/120826322 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f400bb20 con 0x7fb7fc09a150 2024-03-18T19:08:16.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.097+0000 7fb8014c4700 1 -- 172.21.15.1:0/120826322 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f4012430 con 0x7fb7fc09a150 2024-03-18T19:08:16.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.097+0000 7fb80472a700 1 -- 172.21.15.1:0/120826322 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7fc09a150 msgr2=0x7fb7fc0cc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:16.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.097+0000 7fb80472a700 1 --2- 172.21.15.1:0/120826322 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7fc09a150 0x7fb7fc0cc310 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb7f4009d90 tx=0x7fb7f400b330 comp rx=0 tx=0).stop 2024-03-18T19:08:16.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.097+0000 7fb80472a700 1 -- 172.21.15.1:0/120826322 shutdown_connections 2024-03-18T19:08:16.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.097+0000 7fb80472a700 1 --2- 172.21.15.1:0/120826322 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7fc09a150 0x7fb7fc0cc310 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.097+0000 7fb80472a700 1 --2- 172.21.15.1:0/120826322 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7fc15d680 0x7fb7fc0cff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.097+0000 7fb80472a700 1 -- 172.21.15.1:0/120826322 >> 172.21.15.1:0/120826322 conn(0x7fb7fc0b2ba0 msgr2=0x7fb7fc0b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:16.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.097+0000 7fb80472a700 1 -- 172.21.15.1:0/120826322 shutdown_connections 2024-03-18T19:08:16.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.097+0000 7fb80472a700 1 -- 172.21.15.1:0/120826322 wait complete. 2024-03-18T19:08:16.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.099+0000 7fb80472a700 1 Processor -- start 2024-03-18T19:08:16.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.099+0000 7fb80472a700 1 -- start start 2024-03-18T19:08:16.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.099+0000 7fb80472a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7fc09a150 0x7fb7fc1f3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.099+0000 7fb80472a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7fc0d04b0 0x7fb7fc15a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.099+0000 7fb80472a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7fc15d680 0x7fb7fc15f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.099+0000 7fb80472a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc0b3b80 con 0x7fb7fc09a150 2024-03-18T19:08:16.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.099+0000 7fb80472a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc0b3e40 con 0x7fb7fc0d04b0 2024-03-18T19:08:16.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.099+0000 7fb80472a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc0b4120 con 0x7fb7fc15d680 2024-03-18T19:08:16.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.100+0000 7fb8024c6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7fc09a150 0x7fb7fc1f3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:16.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.100+0000 7fb8024c6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7fc09a150 0x7fb7fc1f3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59210/0 (socket says 172.21.15.1:59210) 2024-03-18T19:08:16.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.100+0000 7fb8024c6700 1 -- 172.21.15.1:0/2451527921 learned_addr learned my addr 172.21.15.1:0/2451527921 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:16.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.100+0000 7fb802cc7700 1 --2- 172.21.15.1:0/2451527921 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7fc15d680 0x7fb7fc15f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:16.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.100+0000 7fb801cc5700 1 --2- 172.21.15.1:0/2451527921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7fc0d04b0 0x7fb7fc15a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:16.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.100+0000 7fb8024c6700 1 -- 172.21.15.1:0/2451527921 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7fc15d680 msgr2=0x7fb7fc15f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:16.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.100+0000 7fb8024c6700 1 --2- 172.21.15.1:0/2451527921 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7fc15d680 0x7fb7fc15f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.100+0000 7fb8024c6700 1 -- 172.21.15.1:0/2451527921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7fc0d04b0 msgr2=0x7fb7fc15a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:16.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.100+0000 7fb8024c6700 1 --2- 172.21.15.1:0/2451527921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7fc0d04b0 0x7fb7fc15a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.100+0000 7fb8024c6700 1 -- 172.21.15.1:0/2451527921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f4009a70 con 0x7fb7fc09a150 2024-03-18T19:08:16.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.100+0000 7fb801cc5700 1 --2- 172.21.15.1:0/2451527921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7fc0d04b0 0x7fb7fc15a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:16.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.100+0000 7fb8024c6700 1 --2- 172.21.15.1:0/2451527921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7fc09a150 0x7fb7fc1f3da0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000e9b0 tx=0x7fb7f000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:16.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.100+0000 7fb7eeffd700 1 -- 172.21.15.1:0/2451527921 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f0009710 con 0x7fb7fc09a150 2024-03-18T19:08:16.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.100+0000 7fb7eeffd700 1 -- 172.21.15.1:0/2451527921 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f000c850 con 0x7fb7fc09a150 2024-03-18T19:08:16.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.100+0000 7fb80472a700 1 -- 172.21.15.1:0/2451527921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7fc15dbe0 con 0x7fb7fc09a150 2024-03-18T19:08:16.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.100+0000 7fb7eeffd700 1 -- 172.21.15.1:0/2451527921 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f000cbc0 con 0x7fb7fc09a150 2024-03-18T19:08:16.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.100+0000 7fb80472a700 1 -- 172.21.15.1:0/2451527921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7fc0b46e0 con 0x7fb7fc09a150 2024-03-18T19:08:16.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.101+0000 7fb80472a700 1 -- 172.21.15.1:0/2451527921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7fc0b4a50 con 0x7fb7fc09a150 2024-03-18T19:08:16.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.102+0000 7fb7eeffd700 1 -- 172.21.15.1:0/2451527921 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb7f000cd20 con 0x7fb7fc09a150 2024-03-18T19:08:16.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.102+0000 7fb7eeffd700 1 --2- 172.21.15.1:0/2451527921 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb7e006adb0 0x7fb7e006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.102+0000 7fb7eeffd700 1 -- 172.21.15.1:0/2451527921 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fb7f00896a0 con 0x7fb7fc09a150 2024-03-18T19:08:16.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.102+0000 7fb801cc5700 1 --2- 172.21.15.1:0/2451527921 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb7e006adb0 0x7fb7e006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:16.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.102+0000 7fb801cc5700 1 --2- 172.21.15.1:0/2451527921 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb7e006adb0 0x7fb7e006d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc0cf3e0 tx=0x7fb7f8009040 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:16.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.106+0000 7fb7eeffd700 1 -- 172.21.15.1:0/2451527921 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb7f0054980 con 0x7fb7fc09a150 2024-03-18T19:08:16.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.310+0000 7fb80472a700 1 -- 172.21.15.1:0/2451527921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7fc0676a0 con 0x7fb7fc09a150 2024-03-18T19:08:16.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.311+0000 7fb7eeffd700 1 -- 172.21.15.1:0/2451527921 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+12556 (secure 0 0 0) 0x7fb7f0057fa0 con 0x7fb7fc09a150 2024-03-18T19:08:16.307 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:16.307 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":129,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:08:15.502376+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:08:16.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.313+0000 7fb80472a700 1 -- 172.21.15.1:0/2451527921 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb7e006adb0 msgr2=0x7fb7e006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:16.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.313+0000 7fb80472a700 1 --2- 172.21.15.1:0/2451527921 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb7e006adb0 0x7fb7e006d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc0cf3e0 tx=0x7fb7f8009040 comp rx=0 tx=0).stop 2024-03-18T19:08:16.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.313+0000 7fb80472a700 1 -- 172.21.15.1:0/2451527921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7fc09a150 msgr2=0x7fb7fc1f3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:16.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.313+0000 7fb80472a700 1 --2- 172.21.15.1:0/2451527921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7fc09a150 0x7fb7fc1f3da0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000e9b0 tx=0x7fb7f000ecc0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.313+0000 7fb80472a700 1 -- 172.21.15.1:0/2451527921 shutdown_connections 2024-03-18T19:08:16.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.313+0000 7fb80472a700 1 --2- 172.21.15.1:0/2451527921 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb7e006adb0 0x7fb7e006d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.313+0000 7fb80472a700 1 --2- 172.21.15.1:0/2451527921 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb7fc15d680 0x7fb7fc15f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.313+0000 7fb80472a700 1 --2- 172.21.15.1:0/2451527921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb7fc09a150 0x7fb7fc1f3da0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.313+0000 7fb80472a700 1 --2- 172.21.15.1:0/2451527921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7fc0d04b0 0x7fb7fc15a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.313+0000 7fb80472a700 1 -- 172.21.15.1:0/2451527921 >> 172.21.15.1:0/2451527921 conn(0x7fb7fc0b2ba0 msgr2=0x7fb7fc0afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:16.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.313+0000 7fb80472a700 1 -- 172.21.15.1:0/2451527921 shutdown_connections 2024-03-18T19:08:16.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.313+0000 7fb80472a700 1 -- 172.21.15.1:0/2451527921 wait complete. 2024-03-18T19:08:16.321 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:08:16.321 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:08:16.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.454+0000 7fe188db1700 1 Processor -- start 2024-03-18T19:08:16.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.454+0000 7fe188db1700 1 -- start start 2024-03-18T19:08:16.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.455+0000 7fe188db1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe18415d5a0 0x7fe1840ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.455+0000 7fe188db1700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1840cf600 con 0x7fe1840ce550 2024-03-18T19:08:16.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.455+0000 7fe188db1700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1840ce9f0 con 0x7fe18406ae60 2024-03-18T19:08:16.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.455+0000 7fe188db1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1840cecd0 con 0x7fe18415d5a0 2024-03-18T19:08:16.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.455+0000 7fe18259c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe18415d5a0 0x7fe1840ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:16.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.455+0000 7fe18259c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe18415d5a0 0x7fe1840ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59212/0 (socket says 172.21.15.1:59212) 2024-03-18T19:08:16.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.455+0000 7fe18259c700 1 -- 172.21.15.1:0/2528360659 learned_addr learned my addr 172.21.15.1:0/2528360659 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:16.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.455+0000 7fe181d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe1840ce550 0x7fe18406a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54856/0 (socket says 172.21.15.1:54856) 2024-03-18T19:08:16.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.456+0000 7fe18259c700 1 -- 172.21.15.1:0/2528360659 >> v1:172.21.15.165:6789/0 conn(0x7fe18406ae60 legacy=0x7fe184167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:16.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.456+0000 7fe18259c700 1 -- 172.21.15.1:0/2528360659 >> v1:172.21.15.1:6789/0 conn(0x7fe1840ce550 legacy=0x7fe18406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:16.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.456+0000 7fe18259c700 1 -- 172.21.15.1:0/2528360659 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1840cefb0 con 0x7fe18415d5a0 2024-03-18T19:08:16.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.456+0000 7fe18259c700 1 --2- 172.21.15.1:0/2528360659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18415d5a0 0x7fe1840ce010 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fe178006fd0 tx=0x7fe178009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99071380db32db8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:16.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.456+0000 7fe18159a700 1 -- 172.21.15.1:0/2528360659 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178005e30 con 0x7fe18415d5a0 2024-03-18T19:08:16.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.456+0000 7fe18159a700 1 -- 172.21.15.1:0/2528360659 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe178006030 con 0x7fe18415d5a0 2024-03-18T19:08:16.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.456+0000 7fe18159a700 1 -- 172.21.15.1:0/2528360659 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1780063e0 con 0x7fe18415d5a0 2024-03-18T19:08:16.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.457+0000 7fe188db1700 1 -- 172.21.15.1:0/2528360659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18415d5a0 msgr2=0x7fe1840ce010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:16.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.457+0000 7fe188db1700 1 --2- 172.21.15.1:0/2528360659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18415d5a0 0x7fe1840ce010 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fe178006fd0 tx=0x7fe178009bd0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.457+0000 7fe188db1700 1 -- 172.21.15.1:0/2528360659 shutdown_connections 2024-03-18T19:08:16.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.457+0000 7fe188db1700 1 --2- 172.21.15.1:0/2528360659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18415d5a0 0x7fe1840ce010 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.457+0000 7fe188db1700 1 -- 172.21.15.1:0/2528360659 >> 172.21.15.1:0/2528360659 conn(0x7fe1840cc9b0 msgr2=0x7fe1840ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:16.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.457+0000 7fe188db1700 1 -- 172.21.15.1:0/2528360659 shutdown_connections 2024-03-18T19:08:16.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.457+0000 7fe188db1700 1 -- 172.21.15.1:0/2528360659 wait complete. 2024-03-18T19:08:16.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.458+0000 7fe188db1700 1 Processor -- start 2024-03-18T19:08:16.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.458+0000 7fe188db1700 1 -- start start 2024-03-18T19:08:16.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.459+0000 7fe188db1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe18406ae60 0x7fe18414b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.459+0000 7fe188db1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1840ce550 0x7fe1840b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.459+0000 7fe188db1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18415d5a0 0x7fe1840b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.459+0000 7fe188db1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1840b3940 con 0x7fe18415d5a0 2024-03-18T19:08:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.459+0000 7fe188db1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1840b3c20 con 0x7fe1840ce550 2024-03-18T19:08:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.459+0000 7fe188db1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1840b3f00 con 0x7fe18406ae60 2024-03-18T19:08:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.459+0000 7fe182d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18415d5a0 0x7fe1840b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.459+0000 7fe182d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18415d5a0 0x7fe1840b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59226/0 (socket says 172.21.15.1:59226) 2024-03-18T19:08:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.459+0000 7fe182d9d700 1 -- 172.21.15.1:0/647847866 learned_addr learned my addr 172.21.15.1:0/647847866 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.459+0000 7fe182d9d700 1 -- 172.21.15.1:0/647847866 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe18406ae60 msgr2=0x7fe18414b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.459+0000 7fe182d9d700 1 --2- 172.21.15.1:0/647847866 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe18406ae60 0x7fe18414b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.459+0000 7fe182d9d700 1 -- 172.21.15.1:0/647847866 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1840ce550 msgr2=0x7fe1840b07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.459+0000 7fe181d9b700 1 --2- 172.21.15.1:0/647847866 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1840ce550 0x7fe1840b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.459+0000 7fe182d9d700 1 --2- 172.21.15.1:0/647847866 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1840ce550 0x7fe1840b07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.459+0000 7fe182d9d700 1 -- 172.21.15.1:0/647847866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe178009500 con 0x7fe18415d5a0 2024-03-18T19:08:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.459+0000 7fe181d9b700 1 --2- 172.21.15.1:0/647847866 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1840ce550 0x7fe1840b07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:08:16.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.459+0000 7fe182d9d700 1 --2- 172.21.15.1:0/647847866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18415d5a0 0x7fe1840b0cf0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00cc10 tx=0x7fe17c004990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:16.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.459+0000 7fe172ffd700 1 -- 172.21.15.1:0/647847866 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c00ab80 con 0x7fe18415d5a0 2024-03-18T19:08:16.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.460+0000 7fe172ffd700 1 -- 172.21.15.1:0/647847866 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe17c00ace0 con 0x7fe18415d5a0 2024-03-18T19:08:16.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.460+0000 7fe172ffd700 1 -- 172.21.15.1:0/647847866 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c012660 con 0x7fe18415d5a0 2024-03-18T19:08:16.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.460+0000 7fe188db1700 1 -- 172.21.15.1:0/647847866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1840b41e0 con 0x7fe18415d5a0 2024-03-18T19:08:16.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.460+0000 7fe188db1700 1 -- 172.21.15.1:0/647847866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe18414b850 con 0x7fe18415d5a0 2024-03-18T19:08:16.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.460+0000 7fe188db1700 1 -- 172.21.15.1:0/647847866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1840676a0 con 0x7fe18415d5a0 2024-03-18T19:08:16.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.461+0000 7fe172ffd700 1 -- 172.21.15.1:0/647847866 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe17c0127c0 con 0x7fe18415d5a0 2024-03-18T19:08:16.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.461+0000 7fe172ffd700 1 --2- 172.21.15.1:0/647847866 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe16406adb0 0x7fe16406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.461+0000 7fe172ffd700 1 -- 172.21.15.1:0/647847866 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fe17c089640 con 0x7fe18415d5a0 2024-03-18T19:08:16.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.461+0000 7fe18259c700 1 --2- 172.21.15.1:0/647847866 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe16406adb0 0x7fe16406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:16.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.462+0000 7fe18259c700 1 --2- 172.21.15.1:0/647847866 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe16406adb0 0x7fe16406d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe178013040 tx=0x7fe178010000 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:16.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.466+0000 7fe172ffd700 1 -- 172.21.15.1:0/647847866 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe17c0549b0 con 0x7fe18415d5a0 2024-03-18T19:08:16.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.669+0000 7fe188db1700 1 -- 172.21.15.1:0/647847866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe18415ccc0 con 0x7fe18415d5a0 2024-03-18T19:08:16.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.670+0000 7fe172ffd700 1 -- 172.21.15.1:0/647847866 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe17c0160a0 con 0x7fe18415d5a0 2024-03-18T19:08:16.666 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:08:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.672+0000 7fe188db1700 1 -- 172.21.15.1:0/647847866 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe16406adb0 msgr2=0x7fe16406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.672+0000 7fe188db1700 1 --2- 172.21.15.1:0/647847866 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe16406adb0 0x7fe16406d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe178013040 tx=0x7fe178010000 comp rx=0 tx=0).stop 2024-03-18T19:08:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.672+0000 7fe188db1700 1 -- 172.21.15.1:0/647847866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18415d5a0 msgr2=0x7fe1840b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.672+0000 7fe188db1700 1 --2- 172.21.15.1:0/647847866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18415d5a0 0x7fe1840b0cf0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00cc10 tx=0x7fe17c004990 comp rx=0 tx=0).stop 2024-03-18T19:08:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.672+0000 7fe188db1700 1 -- 172.21.15.1:0/647847866 shutdown_connections 2024-03-18T19:08:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.672+0000 7fe188db1700 1 --2- 172.21.15.1:0/647847866 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe16406adb0 0x7fe16406d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.672+0000 7fe188db1700 1 --2- 172.21.15.1:0/647847866 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe18406ae60 0x7fe18414b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.672+0000 7fe188db1700 1 --2- 172.21.15.1:0/647847866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe18415d5a0 0x7fe1840b0cf0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.672+0000 7fe188db1700 1 --2- 172.21.15.1:0/647847866 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1840ce550 0x7fe1840b07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.672+0000 7fe188db1700 1 -- 172.21.15.1:0/647847866 >> 172.21.15.1:0/647847866 conn(0x7fe1840cc9b0 msgr2=0x7fe184071520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:16.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.672+0000 7fe188db1700 1 -- 172.21.15.1:0/647847866 shutdown_connections 2024-03-18T19:08:16.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.672+0000 7fe188db1700 1 -- 172.21.15.1:0/647847866 wait complete. 2024-03-18T19:08:16.682 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T19:08:16.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.811+0000 7f67c0f2d700 1 Processor -- start 2024-03-18T19:08:16.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.811+0000 7f67c0f2d700 1 -- start start 2024-03-18T19:08:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.812+0000 7f67c0f2d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67bc0cf090 0x7f67bc0d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.812+0000 7f67c0f2d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f67bc160d20 0x7f67bc0d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.812+0000 7f67c0f2d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67bc0cddc0 con 0x7f67bc160940 2024-03-18T19:08:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.812+0000 7f67c0f2d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67bc0cdff0 con 0x7f67bc160d20 2024-03-18T19:08:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.812+0000 7f67c0f2d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67bc0ce2d0 con 0x7f67bc0cf090 2024-03-18T19:08:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67ba59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f67bc160940 0x7f67bc0d0190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54864/0 (socket says 172.21.15.1:54864) 2024-03-18T19:08:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67ba59c700 1 -- 172.21.15.1:0/1792910879 learned_addr learned my addr 172.21.15.1:0/1792910879 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67b959a700 1 -- 172.21.15.1:0/1792910879 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26529238 0 0) 0x7f67bc0cddc0 con 0x7f67bc160940 2024-03-18T19:08:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67b959a700 1 -- 172.21.15.1:0/1792910879 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67a80035b0 con 0x7f67bc160940 2024-03-18T19:08:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67bad9d700 1 --2- 172.21.15.1:0/1792910879 >> v2:172.21.15.165:3300/0 conn(0x7f67bc160d20 0x7f67bc0d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67b9d9b700 1 --2- 172.21.15.1:0/1792910879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67bc0cf090 0x7f67bc0d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67b959a700 1 -- 172.21.15.1:0/1792910879 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3713653338 0 0) 0x7f67a80035b0 con 0x7f67bc160940 2024-03-18T19:08:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67b959a700 1 -- 172.21.15.1:0/1792910879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67bc0cf090 msgr2=0x7f67bc0d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67b959a700 1 --2- 172.21.15.1:0/1792910879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67bc0cf090 0x7f67bc0d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67b959a700 1 -- 172.21.15.1:0/1792910879 >> v2:172.21.15.165:3300/0 conn(0x7f67bc160d20 msgr2=0x7f67bc0d1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67b959a700 1 --2- 172.21.15.1:0/1792910879 >> v2:172.21.15.165:3300/0 conn(0x7f67bc160d20 0x7f67bc0d1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67b959a700 1 -- 172.21.15.1:0/1792910879 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67bc0ce5b0 con 0x7f67bc160940 2024-03-18T19:08:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67b959a700 1 -- 172.21.15.1:0/1792910879 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f67a4001670 con 0x7f67bc160940 2024-03-18T19:08:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67b959a700 1 -- 172.21.15.1:0/1792910879 >> v1:172.21.15.1:6789/0 conn(0x7f67bc160940 legacy=0x7f67bc0d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67b959a700 1 --2- 172.21.15.1:0/1792910879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67a8003a00 0x7f67a8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67b959a700 1 --2- 172.21.15.1:0/1792910879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a8004850 0x7f67a8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67b959a700 1 --2- 172.21.15.1:0/1792910879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67bc0cf090 0x7f67a8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67b959a700 1 -- 172.21.15.1:0/1792910879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67bc0cdff0 con 0x7f67a8004850 2024-03-18T19:08:16.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67b959a700 1 -- 172.21.15.1:0/1792910879 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67bc0ce2d0 con 0x7f67a8003a00 2024-03-18T19:08:16.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67b959a700 1 -- 172.21.15.1:0/1792910879 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a800ba60 con 0x7f67bc0cf090 2024-03-18T19:08:16.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67b9d9b700 1 --2- 172.21.15.1:0/1792910879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a8004850 0x7f67a8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:16.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67ba59c700 1 --2- 172.21.15.1:0/1792910879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67a8003a00 0x7f67a8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:16.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.813+0000 7f67bad9d700 1 --2- 172.21.15.1:0/1792910879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67bc0cf090 0x7f67a8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:16.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.814+0000 7f67b9d9b700 1 -- 172.21.15.1:0/1792910879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67bc0cf090 msgr2=0x7f67a8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:16.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.814+0000 7f67b9d9b700 1 --2- 172.21.15.1:0/1792910879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67bc0cf090 0x7f67a8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.814+0000 7f67b9d9b700 1 -- 172.21.15.1:0/1792910879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67a8003a00 msgr2=0x7f67a8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:16.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.814+0000 7f67b9d9b700 1 --2- 172.21.15.1:0/1792910879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67a8003a00 0x7f67a8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.814+0000 7f67b9d9b700 1 -- 172.21.15.1:0/1792910879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a80035b0 con 0x7f67a8004850 2024-03-18T19:08:16.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.814+0000 7f67b9d9b700 1 --2- 172.21.15.1:0/1792910879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a8004850 0x7f67a8004cd0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009b90 tx=0x7f67a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3899d6fb157b52e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:16.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.814+0000 7f67b959a700 1 -- 172.21.15.1:0/1792910879 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ac00e6c0 con 0x7f67a8004850 2024-03-18T19:08:16.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.814+0000 7f67b959a700 1 -- 172.21.15.1:0/1792910879 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67ac00e820 con 0x7f67a8004850 2024-03-18T19:08:16.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.814+0000 7f67b959a700 1 -- 172.21.15.1:0/1792910879 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ac00eb90 con 0x7f67a8004850 2024-03-18T19:08:16.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.815+0000 7f67c0f2d700 1 -- 172.21.15.1:0/1792910879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a8004850 msgr2=0x7f67a8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:16.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.815+0000 7f67c0f2d700 1 --2- 172.21.15.1:0/1792910879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a8004850 0x7f67a8004cd0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009b90 tx=0x7f67a8005880 comp rx=0 tx=0).stop 2024-03-18T19:08:16.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.815+0000 7f67bad9d700 1 -- 172.21.15.1:0/1792910879 reap_dead start 2024-03-18T19:08:16.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.815+0000 7f67c0f2d700 1 -- 172.21.15.1:0/1792910879 shutdown_connections 2024-03-18T19:08:16.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.815+0000 7f67c0f2d700 1 -- 172.21.15.1:0/1792910879 >> 172.21.15.1:0/1792910879 conn(0x7f67bc09c520 msgr2=0x7f67bc0d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:16.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.815+0000 7f67c0f2d700 1 -- 172.21.15.1:0/1792910879 shutdown_connections 2024-03-18T19:08:16.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.815+0000 7f67c0f2d700 1 -- 172.21.15.1:0/1792910879 wait complete. 2024-03-18T19:08:16.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.816+0000 7f67c0f2d700 1 Processor -- start 2024-03-18T19:08:16.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.816+0000 7f67c0f2d700 1 -- start start 2024-03-18T19:08:16.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.817+0000 7f67c0f2d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a8004850 0x7f67bc14c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.817+0000 7f67c0f2d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67bc0b84d0 0x7f67bc0b8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.817+0000 7f67c0f2d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67bc0b5dd0 0x7f67bc0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.817+0000 7f67c0f2d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67bc0cbd10 con 0x7f67a8004850 2024-03-18T19:08:16.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.817+0000 7f67c0f2d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67bc0b8e90 con 0x7f67bc0b84d0 2024-03-18T19:08:16.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.817+0000 7f67c0f2d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67bc14c8e0 con 0x7f67bc0b5dd0 2024-03-18T19:08:16.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.817+0000 7f67ba59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a8004850 0x7f67bc14c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:16.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.817+0000 7f67ba59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a8004850 0x7f67bc14c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59254/0 (socket says 172.21.15.1:59254) 2024-03-18T19:08:16.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.817+0000 7f67ba59c700 1 -- 172.21.15.1:0/1750844970 learned_addr learned my addr 172.21.15.1:0/1750844970 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:16.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.817+0000 7f67ba59c700 1 -- 172.21.15.1:0/1750844970 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67bc0b5dd0 msgr2=0x7f67bc0b6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:16.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.817+0000 7f67ba59c700 1 --2- 172.21.15.1:0/1750844970 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67bc0b5dd0 0x7f67bc0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.817+0000 7f67ba59c700 1 -- 172.21.15.1:0/1750844970 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67bc0b84d0 msgr2=0x7f67bc0b8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:16.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.817+0000 7f67ba59c700 1 --2- 172.21.15.1:0/1750844970 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67bc0b84d0 0x7f67bc0b8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:16.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.817+0000 7f67ba59c700 1 -- 172.21.15.1:0/1750844970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67bc0cb090 con 0x7f67a8004850 2024-03-18T19:08:16.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.817+0000 7f67ba59c700 1 --2- 172.21.15.1:0/1750844970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a8004850 0x7f67bc14c3a0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f67a4007310 tx=0x7f67a40076d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:16.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.817+0000 7f67b2ffd700 1 -- 172.21.15.1:0/1750844970 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a40077f0 con 0x7f67a8004850 2024-03-18T19:08:16.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.817+0000 7f67b2ffd700 1 -- 172.21.15.1:0/1750844970 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a400f7f0 con 0x7f67a8004850 2024-03-18T19:08:16.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.817+0000 7f67c0f2d700 1 -- 172.21.15.1:0/1750844970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67bc0cb370 con 0x7f67a8004850 2024-03-18T19:08:16.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.817+0000 7f67b2ffd700 1 -- 172.21.15.1:0/1750844970 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a400fbc0 con 0x7f67a8004850 2024-03-18T19:08:16.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.818+0000 7f67c0f2d700 1 -- 172.21.15.1:0/1750844970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67bc0cb930 con 0x7f67a8004850 2024-03-18T19:08:16.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.818+0000 7f67c0f2d700 1 -- 172.21.15.1:0/1750844970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67bc0cbaf0 con 0x7f67a8004850 2024-03-18T19:08:16.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.819+0000 7f67b2ffd700 1 -- 172.21.15.1:0/1750844970 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f67a400fd20 con 0x7f67a8004850 2024-03-18T19:08:16.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.819+0000 7f67b2ffd700 1 --2- 172.21.15.1:0/1750844970 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f679c06adb0 0x7f679c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:16.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.819+0000 7f67b2ffd700 1 -- 172.21.15.1:0/1750844970 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f67a4089250 con 0x7f67a8004850 2024-03-18T19:08:16.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.819+0000 7f67b9d9b700 1 --2- 172.21.15.1:0/1750844970 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f679c06adb0 0x7f679c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:16.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.820+0000 7f67b9d9b700 1 --2- 172.21.15.1:0/1750844970 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f679c06adb0 0x7f679c06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f67bc0d06b0 tx=0x7f67ac013000 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:16.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:16.824+0000 7f67b2ffd700 1 -- 172.21.15.1:0/1750844970 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f67a4054530 con 0x7f67a8004850 2024-03-18T19:08:17.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.028+0000 7f67c0f2d700 1 -- 172.21.15.1:0/1750844970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f67bc073350 con 0x7f67a8004850 2024-03-18T19:08:17.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.029+0000 7f67b2ffd700 1 -- 172.21.15.1:0/1750844970 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v129) v1 ==== 84+0+65 (secure 0 0 0) 0x7f67a4057b50 con 0x7f67a8004850 2024-03-18T19:08:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:17.025 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T19:08:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-18T19:08:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-18T19:08:17.025 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:17.025 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T19:08:17.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.031+0000 7f67c0f2d700 1 -- 172.21.15.1:0/1750844970 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f679c06adb0 msgr2=0x7f679c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:17.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.031+0000 7f67c0f2d700 1 --2- 172.21.15.1:0/1750844970 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f679c06adb0 0x7f679c06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f67bc0d06b0 tx=0x7f67ac013000 comp rx=0 tx=0).stop 2024-03-18T19:08:17.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.031+0000 7f67c0f2d700 1 -- 172.21.15.1:0/1750844970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a8004850 msgr2=0x7f67bc14c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:17.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.031+0000 7f67c0f2d700 1 --2- 172.21.15.1:0/1750844970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a8004850 0x7f67bc14c3a0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f67a4007310 tx=0x7f67a40076d0 comp rx=0 tx=0).stop 2024-03-18T19:08:17.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.031+0000 7f67c0f2d700 1 -- 172.21.15.1:0/1750844970 shutdown_connections 2024-03-18T19:08:17.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.031+0000 7f67c0f2d700 1 --2- 172.21.15.1:0/1750844970 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f679c06adb0 0x7f679c06d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:17.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.031+0000 7f67c0f2d700 1 --2- 172.21.15.1:0/1750844970 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67bc0b5dd0 0x7f67bc0b6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:17.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.031+0000 7f67c0f2d700 1 --2- 172.21.15.1:0/1750844970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a8004850 0x7f67bc14c3a0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:17.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.031+0000 7f67c0f2d700 1 --2- 172.21.15.1:0/1750844970 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67bc0b84d0 0x7f67bc0b8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:17.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.031+0000 7f67c0f2d700 1 -- 172.21.15.1:0/1750844970 >> 172.21.15.1:0/1750844970 conn(0x7f67bc09c520 msgr2=0x7f67bc06b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:17.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.031+0000 7f67c0f2d700 1 -- 172.21.15.1:0/1750844970 shutdown_connections 2024-03-18T19:08:17.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.031+0000 7f67c0f2d700 1 -- 172.21.15.1:0/1750844970 wait complete. 2024-03-18T19:08:17.040 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T19:08:17.040 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T19:08:17.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.174+0000 7f43f28a7700 1 Processor -- start 2024-03-18T19:08:17.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.174+0000 7f43f28a7700 1 -- start start 2024-03-18T19:08:17.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.174+0000 7f43f28a7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f43ec144b30 0x7f43ec0723f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:17.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.174+0000 7f43f28a7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ec0d0ee0 con 0x7f43ec0ccd50 2024-03-18T19:08:17.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.174+0000 7f43f28a7700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ec15c8b0 con 0x7f43ec15c050 2024-03-18T19:08:17.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.174+0000 7f43f28a7700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43ec15c5d0 con 0x7f43ec144b30 2024-03-18T19:08:17.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.175+0000 7f43ebfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f43ec144b30 0x7f43ec0723f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:17.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.175+0000 7f43ebfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f43ec144b30 0x7f43ec0723f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59262/0 (socket says 172.21.15.1:59262) 2024-03-18T19:08:17.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.175+0000 7f43ebfff700 1 -- 172.21.15.1:0/2631789689 learned_addr learned my addr 172.21.15.1:0/2631789689 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:17.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.175+0000 7f43ebfff700 1 -- 172.21.15.1:0/2631789689 >> v1:172.21.15.165:6789/0 conn(0x7f43ec15c050 legacy=0x7f43ec0cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:17.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.175+0000 7f43ebfff700 1 -- 172.21.15.1:0/2631789689 >> v1:172.21.15.1:6789/0 conn(0x7f43ec0ccd50 legacy=0x7f43ec15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:17.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.175+0000 7f43eaffd700 1 -- 172.21.15.1:0/2631789689 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768478209 0 0) 0x7f43ec15c8b0 con 0x7f43ec15c050 2024-03-18T19:08:17.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.175+0000 7f43ebfff700 1 -- 172.21.15.1:0/2631789689 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ec071910 con 0x7f43ec144b30 2024-03-18T19:08:17.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.175+0000 7f43ebfff700 1 --2- 172.21.15.1:0/2631789689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43ec144b30 0x7f43ec0723f0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0098f0 tx=0x7f43dc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=974f47304e5acc45 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:17.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.175+0000 7f43eaffd700 1 -- 172.21.15.1:0/2631789689 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc00b8a0 con 0x7f43ec144b30 2024-03-18T19:08:17.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.175+0000 7f43eaffd700 1 -- 172.21.15.1:0/2631789689 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43dc00ba00 con 0x7f43ec144b30 2024-03-18T19:08:17.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.175+0000 7f43eaffd700 1 -- 172.21.15.1:0/2631789689 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc00bd90 con 0x7f43ec144b30 2024-03-18T19:08:17.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.176+0000 7f43f28a7700 1 -- 172.21.15.1:0/2631789689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43ec144b30 msgr2=0x7f43ec0723f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:17.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.176+0000 7f43f28a7700 1 --2- 172.21.15.1:0/2631789689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43ec144b30 0x7f43ec0723f0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0098f0 tx=0x7f43dc009c00 comp rx=0 tx=0).stop 2024-03-18T19:08:17.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.176+0000 7f43f28a7700 1 -- 172.21.15.1:0/2631789689 shutdown_connections 2024-03-18T19:08:17.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.176+0000 7f43f28a7700 1 --2- 172.21.15.1:0/2631789689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43ec144b30 0x7f43ec0723f0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:17.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.176+0000 7f43f28a7700 1 -- 172.21.15.1:0/2631789689 >> 172.21.15.1:0/2631789689 conn(0x7f43ec0d00c0 msgr2=0x7f43ec0d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:17.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.176+0000 7f43f28a7700 1 -- 172.21.15.1:0/2631789689 shutdown_connections 2024-03-18T19:08:17.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.176+0000 7f43f28a7700 1 -- 172.21.15.1:0/2631789689 wait complete. 2024-03-18T19:08:17.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.177+0000 7f43f28a7700 1 Processor -- start 2024-03-18T19:08:17.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.177+0000 7f43f28a7700 1 -- start start 2024-03-18T19:08:17.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.178+0000 7f43f28a7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43ec0ccd50 0x7f43ec15ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:17.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.178+0000 7f43f28a7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43ec144b30 0x7f43ec0cebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:17.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.178+0000 7f43f28a7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43ec15c050 0x7f43ec0cf120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:17.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.178+0000 7f43f28a7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec1e5aa0 con 0x7f43ec15c050 2024-03-18T19:08:17.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.178+0000 7f43f28a7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec1e5d80 con 0x7f43ec144b30 2024-03-18T19:08:17.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.178+0000 7f43f28a7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec1f12b0 con 0x7f43ec0ccd50 2024-03-18T19:08:17.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.178+0000 7f43f0e44700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43ec15c050 0x7f43ec0cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:17.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.178+0000 7f43f0e44700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43ec15c050 0x7f43ec0cf120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59266/0 (socket says 172.21.15.1:59266) 2024-03-18T19:08:17.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.178+0000 7f43f0e44700 1 -- 172.21.15.1:0/2851722533 learned_addr learned my addr 172.21.15.1:0/2851722533 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:17.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.178+0000 7f43eb7fe700 1 --2- 172.21.15.1:0/2851722533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43ec144b30 0x7f43ec0cebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:17.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.178+0000 7f43ebfff700 1 --2- 172.21.15.1:0/2851722533 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43ec0ccd50 0x7f43ec15ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:17.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.179+0000 7f43f0e44700 1 -- 172.21.15.1:0/2851722533 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43ec0ccd50 msgr2=0x7f43ec15ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:17.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.179+0000 7f43f0e44700 1 --2- 172.21.15.1:0/2851722533 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43ec0ccd50 0x7f43ec15ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:17.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.179+0000 7f43f0e44700 1 -- 172.21.15.1:0/2851722533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43ec144b30 msgr2=0x7f43ec0cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:17.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.179+0000 7f43f0e44700 1 --2- 172.21.15.1:0/2851722533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43ec144b30 0x7f43ec0cebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:17.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.179+0000 7f43f0e44700 1 -- 172.21.15.1:0/2851722533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43dc009550 con 0x7f43ec15c050 2024-03-18T19:08:17.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.179+0000 7f43eb7fe700 1 --2- 172.21.15.1:0/2851722533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43ec144b30 0x7f43ec0cebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:17.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.179+0000 7f43f0e44700 1 --2- 172.21.15.1:0/2851722533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43ec15c050 0x7f43ec0cf120 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f43e400c760 tx=0x7f43e400cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:17.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.179+0000 7f43e8ff9700 1 -- 172.21.15.1:0/2851722533 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e4003e30 con 0x7f43ec15c050 2024-03-18T19:08:17.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.179+0000 7f43e8ff9700 1 -- 172.21.15.1:0/2851722533 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e4005070 con 0x7f43ec15c050 2024-03-18T19:08:17.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.179+0000 7f43f28a7700 1 -- 172.21.15.1:0/2851722533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43ec1f1590 con 0x7f43ec15c050 2024-03-18T19:08:17.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.179+0000 7f43e8ff9700 1 -- 172.21.15.1:0/2851722533 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e4005400 con 0x7f43ec15c050 2024-03-18T19:08:17.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.179+0000 7f43f28a7700 1 -- 172.21.15.1:0/2851722533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43ec15b190 con 0x7f43ec15c050 2024-03-18T19:08:17.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.180+0000 7f43f28a7700 1 -- 172.21.15.1:0/2851722533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43ec1f1ab0 con 0x7f43ec15c050 2024-03-18T19:08:17.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.180+0000 7f43e8ff9700 1 -- 172.21.15.1:0/2851722533 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f43e4005560 con 0x7f43ec15c050 2024-03-18T19:08:17.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.181+0000 7f43e8ff9700 1 --2- 172.21.15.1:0/2851722533 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43d806adb0 0x7f43d806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:17.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.181+0000 7f43e8ff9700 1 -- 172.21.15.1:0/2851722533 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f43e4088ea0 con 0x7f43ec15c050 2024-03-18T19:08:17.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.181+0000 7f43ebfff700 1 --2- 172.21.15.1:0/2851722533 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43d806adb0 0x7f43d806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:17.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.181+0000 7f43ebfff700 1 --2- 172.21.15.1:0/2851722533 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43d806adb0 0x7f43d806d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f43dc000c00 tx=0x7f43dc008030 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:17.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.186+0000 7f43e8ff9700 1 -- 172.21.15.1:0/2851722533 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f43ec1f1ab0 con 0x7f43ec15c050 2024-03-18T19:08:17.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:17.387+0000 7f43f28a7700 1 -- 172.21.15.1:0/2851722533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f43ec1f1ab0 con 0x7f43ec15c050 2024-03-18T19:08:18.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.145+0000 7f43e8ff9700 1 -- 172.21.15.1:0/2851722533 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v130) v1 ==== 138+0+0 (secure 0 0 0) 0x7f43ec1f1ab0 con 0x7f43ec15c050 2024-03-18T19:08:18.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.151+0000 7f43f28a7700 1 -- 172.21.15.1:0/2851722533 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43d806adb0 msgr2=0x7f43d806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.151+0000 7f43f28a7700 1 --2- 172.21.15.1:0/2851722533 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43d806adb0 0x7f43d806d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f43dc000c00 tx=0x7f43dc008030 comp rx=0 tx=0).stop 2024-03-18T19:08:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.151+0000 7f43f28a7700 1 -- 172.21.15.1:0/2851722533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43ec15c050 msgr2=0x7f43ec0cf120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.151+0000 7f43f28a7700 1 --2- 172.21.15.1:0/2851722533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43ec15c050 0x7f43ec0cf120 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f43e400c760 tx=0x7f43e400cb20 comp rx=0 tx=0).stop 2024-03-18T19:08:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.152+0000 7f43f28a7700 1 -- 172.21.15.1:0/2851722533 shutdown_connections 2024-03-18T19:08:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.152+0000 7f43f28a7700 1 --2- 172.21.15.1:0/2851722533 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43d806adb0 0x7f43d806d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.152+0000 7f43f28a7700 1 --2- 172.21.15.1:0/2851722533 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43ec0ccd50 0x7f43ec15ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.152+0000 7f43f28a7700 1 --2- 172.21.15.1:0/2851722533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43ec15c050 0x7f43ec0cf120 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.152+0000 7f43f28a7700 1 --2- 172.21.15.1:0/2851722533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43ec144b30 0x7f43ec0cebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.152+0000 7f43f28a7700 1 -- 172.21.15.1:0/2851722533 >> 172.21.15.1:0/2851722533 conn(0x7f43ec0d00c0 msgr2=0x7f43ec0b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:18.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.152+0000 7f43f28a7700 1 -- 172.21.15.1:0/2851722533 shutdown_connections 2024-03-18T19:08:18.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.153+0000 7f43f28a7700 1 -- 172.21.15.1:0/2851722533 wait complete. 2024-03-18T19:08:18.150 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-18T19:08:18.165 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T19:08:18.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.308+0000 7f0081279700 1 Processor -- start 2024-03-18T19:08:18.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.308+0000 7f0081279700 1 -- start start 2024-03-18T19:08:18.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.309+0000 7f0081279700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f007c0cfb50 0x7f007c0d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:18.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.309+0000 7f0081279700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f007c0cee80 0x7f007c0b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:18.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.309+0000 7f0081279700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f007c0b3370 con 0x7f007c0cff30 2024-03-18T19:08:18.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.309+0000 7f0081279700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f007c0cf310 con 0x7f007c0cfb50 2024-03-18T19:08:18.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.309+0000 7f0081279700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f007c0b2760 con 0x7f007c0cee80 2024-03-18T19:08:18.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.309+0000 7f007affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f007c0cfb50 0x7f007c0d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:18.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.309+0000 7f007a7fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f007c0cff30 0x7f007c0b3800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54880/0 (socket says 172.21.15.1:54880) 2024-03-18T19:08:18.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.309+0000 7f007affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f007c0cfb50 0x7f007c0d3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59272/0 (socket says 172.21.15.1:59272) 2024-03-18T19:08:18.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.309+0000 7f007a7fc700 1 -- 172.21.15.1:0/3630834428 learned_addr learned my addr 172.21.15.1:0/3630834428 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:18.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.309+0000 7f007b7fe700 1 --2- 172.21.15.1:0/3630834428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f007c0cee80 0x7f007c0b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:18.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.309+0000 7f007affd700 1 -- 172.21.15.1:0/3630834428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f007c0cee80 msgr2=0x7f007c0b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:18.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.309+0000 7f007affd700 1 --2- 172.21.15.1:0/3630834428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f007c0cee80 0x7f007c0b3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.309+0000 7f007affd700 1 -- 172.21.15.1:0/3630834428 >> v1:172.21.15.1:6789/0 conn(0x7f007c0cff30 legacy=0x7f007c0b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.309+0000 7f0079ffb700 1 -- 172.21.15.1:0/3630834428 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755787214 0 0) 0x7f007c0b3370 con 0x7f007c0cff30 2024-03-18T19:08:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.309+0000 7f007affd700 1 -- 172.21.15.1:0/3630834428 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007c0b2a40 con 0x7f007c0cfb50 2024-03-18T19:08:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.310+0000 7f007affd700 1 --2- 172.21.15.1:0/3630834428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f007c0cfb50 0x7f007c0d3c60 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f0070009820 tx=0x7f0070009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab2c04df01546a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.310+0000 7f0079ffb700 1 -- 172.21.15.1:0/3630834428 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0070005e30 con 0x7f007c0cfb50 2024-03-18T19:08:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.310+0000 7f0079ffb700 1 -- 172.21.15.1:0/3630834428 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0070006030 con 0x7f007c0cfb50 2024-03-18T19:08:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.310+0000 7f0079ffb700 1 -- 172.21.15.1:0/3630834428 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00700063e0 con 0x7f007c0cfb50 2024-03-18T19:08:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.310+0000 7f0081279700 1 -- 172.21.15.1:0/3630834428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f007c0cfb50 msgr2=0x7f007c0d3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.310+0000 7f0081279700 1 --2- 172.21.15.1:0/3630834428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f007c0cfb50 0x7f007c0d3c60 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f0070009820 tx=0x7f0070009b30 comp rx=0 tx=0).stop 2024-03-18T19:08:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.310+0000 7f0081279700 1 -- 172.21.15.1:0/3630834428 shutdown_connections 2024-03-18T19:08:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.310+0000 7f0081279700 1 --2- 172.21.15.1:0/3630834428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f007c0cfb50 0x7f007c0d3c60 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.310+0000 7f0081279700 1 --2- 172.21.15.1:0/3630834428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f007c0cee80 0x7f007c0b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:18.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.310+0000 7f0081279700 1 -- 172.21.15.1:0/3630834428 >> 172.21.15.1:0/3630834428 conn(0x7f007c0d3140 msgr2=0x7f007c0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:18.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.310+0000 7f0081279700 1 -- 172.21.15.1:0/3630834428 shutdown_connections 2024-03-18T19:08:18.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.310+0000 7f0081279700 1 -- 172.21.15.1:0/3630834428 wait complete. 2024-03-18T19:08:18.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.312+0000 7f0081279700 1 Processor -- start 2024-03-18T19:08:18.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.312+0000 7f0081279700 1 -- start start 2024-03-18T19:08:18.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f0081279700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f007c0cee80 0x7f007c154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:18.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f0081279700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f007c0cfb50 0x7f007c0cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:18.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f0081279700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f007c0cff30 0x7f007c0b1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:18.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f0081279700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007c154cc0 con 0x7f007c0cff30 2024-03-18T19:08:18.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f0081279700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007c0cc700 con 0x7f007c0cfb50 2024-03-18T19:08:18.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f0081279700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f007c0b6750 con 0x7f007c0cee80 2024-03-18T19:08:18.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f007b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f007c0cff30 0x7f007c0b1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:18.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f007b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f007c0cff30 0x7f007c0b1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59274/0 (socket says 172.21.15.1:59274) 2024-03-18T19:08:18.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f007b7fe700 1 -- 172.21.15.1:0/3216788813 learned_addr learned my addr 172.21.15.1:0/3216788813 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:18.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f007a7fc700 1 --2- 172.21.15.1:0/3216788813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f007c0cfb50 0x7f007c0cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:18.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f007affd700 1 --2- 172.21.15.1:0/3216788813 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f007c0cee80 0x7f007c154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:18.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f007b7fe700 1 -- 172.21.15.1:0/3216788813 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f007c0cee80 msgr2=0x7f007c154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:18.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f007b7fe700 1 --2- 172.21.15.1:0/3216788813 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f007c0cee80 0x7f007c154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:18.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f007b7fe700 1 -- 172.21.15.1:0/3216788813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f007c0cfb50 msgr2=0x7f007c0cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:18.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f007b7fe700 1 --2- 172.21.15.1:0/3216788813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f007c0cfb50 0x7f007c0cbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:18.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f007b7fe700 1 -- 172.21.15.1:0/3216788813 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0070009430 con 0x7f007c0cff30 2024-03-18T19:08:18.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f007b7fe700 1 --2- 172.21.15.1:0/3216788813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f007c0cff30 0x7f007c0b1460 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f0074007e90 tx=0x7f007400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:18.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f00637fe700 1 -- 172.21.15.1:0/3216788813 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007400ede0 con 0x7f007c0cff30 2024-03-18T19:08:18.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f00637fe700 1 -- 172.21.15.1:0/3216788813 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0074014420 con 0x7f007c0cff30 2024-03-18T19:08:18.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f00637fe700 1 -- 172.21.15.1:0/3216788813 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00740147d0 con 0x7f007c0cff30 2024-03-18T19:08:18.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f0081279700 1 -- 172.21.15.1:0/3216788813 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f007c0b5e60 con 0x7f007c0cff30 2024-03-18T19:08:18.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.313+0000 7f0081279700 1 -- 172.21.15.1:0/3216788813 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f007c0b63e0 con 0x7f007c0cff30 2024-03-18T19:08:18.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.314+0000 7f0081279700 1 -- 172.21.15.1:0/3216788813 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f007c0b68e0 con 0x7f007c0cff30 2024-03-18T19:08:18.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.315+0000 7f00637fe700 1 -- 172.21.15.1:0/3216788813 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0074014930 con 0x7f007c0cff30 2024-03-18T19:08:18.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.315+0000 7f00637fe700 1 --2- 172.21.15.1:0/3216788813 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f006806adb0 0x7f006806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:18.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.316+0000 7f00637fe700 1 -- 172.21.15.1:0/3216788813 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f0074089ad0 con 0x7f007c0cff30 2024-03-18T19:08:18.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.316+0000 7f007affd700 1 --2- 172.21.15.1:0/3216788813 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f006806adb0 0x7f006806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:18.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.316+0000 7f007affd700 1 --2- 172.21.15.1:0/3216788813 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f006806adb0 0x7f006806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0070013040 tx=0x7f0070008070 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:18.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.320+0000 7f00637fe700 1 -- 172.21.15.1:0/3216788813 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0074054cd0 con 0x7f007c0cff30 2024-03-18T19:08:18.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:18.521+0000 7f0081279700 1 -- 172.21.15.1:0/3216788813 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f007c073350 con 0x7f007c0cff30 2024-03-18T19:08:19.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.149+0000 7f00637fe700 1 -- 172.21.15.1:0/3216788813 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v131) v1 ==== 170+0+0 (secure 0 0 0) 0x7f00740582f0 con 0x7f007c0cff30 2024-03-18T19:08:19.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.154+0000 7f0081279700 1 -- 172.21.15.1:0/3216788813 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f006806adb0 msgr2=0x7f006806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:19.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.154+0000 7f0081279700 1 --2- 172.21.15.1:0/3216788813 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f006806adb0 0x7f006806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0070013040 tx=0x7f0070008070 comp rx=0 tx=0).stop 2024-03-18T19:08:19.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.154+0000 7f0081279700 1 -- 172.21.15.1:0/3216788813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f007c0cff30 msgr2=0x7f007c0b1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:19.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.154+0000 7f0081279700 1 --2- 172.21.15.1:0/3216788813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f007c0cff30 0x7f007c0b1460 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f0074007e90 tx=0x7f007400e3c0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.156+0000 7f0081279700 1 -- 172.21.15.1:0/3216788813 shutdown_connections 2024-03-18T19:08:19.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.156+0000 7f0081279700 1 --2- 172.21.15.1:0/3216788813 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f006806adb0 0x7f006806d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.156+0000 7f0081279700 1 --2- 172.21.15.1:0/3216788813 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f007c0cee80 0x7f007c154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.156+0000 7f0081279700 1 --2- 172.21.15.1:0/3216788813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f007c0cff30 0x7f007c0b1460 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.156+0000 7f0081279700 1 --2- 172.21.15.1:0/3216788813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f007c0cfb50 0x7f007c0cbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.156+0000 7f0081279700 1 -- 172.21.15.1:0/3216788813 >> 172.21.15.1:0/3216788813 conn(0x7f007c0d3140 msgr2=0x7f007c0b53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:19.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.169+0000 7f0081279700 1 -- 172.21.15.1:0/3216788813 shutdown_connections 2024-03-18T19:08:19.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.171+0000 7f0081279700 1 -- 172.21.15.1:0/3216788813 wait complete. 2024-03-18T19:08:19.168 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-18T19:08:19.182 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T19:08:19.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.319+0000 7fe0b357e700 1 Processor -- start 2024-03-18T19:08:19.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.319+0000 7fe0b357e700 1 -- start start 2024-03-18T19:08:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.320+0000 7fe0b357e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe0ac15a1d0 0x7fe0ac0add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.320+0000 7fe0b357e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe0ac0cc0f0 0x7fe0ac071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.320+0000 7fe0b357e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac0ce8f0 con 0x7fe0ac071620 2024-03-18T19:08:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.320+0000 7fe0b357e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0ac0cdb00 con 0x7fe0ac0cc0f0 2024-03-18T19:08:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.320+0000 7fe0b357e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0ac15a5b0 con 0x7fe0ac15a1d0 2024-03-18T19:08:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.320+0000 7fe0b1b1b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe0ac0cc0f0 0x7fe0ac071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.320+0000 7fe0b1b1b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe0ac0cc0f0 0x7fe0ac071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59284/0 (socket says 172.21.15.1:59284) 2024-03-18T19:08:19.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.320+0000 7fe0b1b1b700 1 -- 172.21.15.1:0/1741109889 learned_addr learned my addr 172.21.15.1:0/1741109889 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.320+0000 7fe0b1b1b700 1 -- 172.21.15.1:0/1741109889 >> v2:172.21.15.165:3300/0 conn(0x7fe0ac15a1d0 msgr2=0x7fe0ac0add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.320+0000 7fe0b131a700 1 --2- 172.21.15.1:0/1741109889 >> v2:172.21.15.165:3300/0 conn(0x7fe0ac15a1d0 0x7fe0ac0add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.320+0000 7fe0b1b1b700 1 --2- 172.21.15.1:0/1741109889 >> v2:172.21.15.165:3300/0 conn(0x7fe0ac15a1d0 0x7fe0ac0add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.320+0000 7fe0b1b1b700 1 -- 172.21.15.1:0/1741109889 >> v1:172.21.15.165:6789/0 conn(0x7fe0ac071620 legacy=0x7fe0ac0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.320+0000 7fe0b1b1b700 1 -- 172.21.15.1:0/1741109889 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac071fc0 con 0x7fe0ac0cc0f0 2024-03-18T19:08:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.320+0000 7fe0b131a700 1 --2- 172.21.15.1:0/1741109889 >> v2:172.21.15.165:3300/0 conn(0x7fe0ac15a1d0 0x7fe0ac0add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:08:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.320+0000 7fe0b1b1b700 1 --2- 172.21.15.1:0/1741109889 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0ac0cc0f0 0x7fe0ac071a80 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac0726b0 tx=0x7fe0a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abcfa8b6ed5f2753 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.321+0000 7fe0a3fff700 1 -- 172.21.15.1:0/1741109889 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a400e070 con 0x7fe0ac0cc0f0 2024-03-18T19:08:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.321+0000 7fe0a3fff700 1 -- 172.21.15.1:0/1741109889 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a4003a20 con 0x7fe0ac0cc0f0 2024-03-18T19:08:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.321+0000 7fe0a3fff700 1 -- 172.21.15.1:0/1741109889 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4003dd0 con 0x7fe0ac0cc0f0 2024-03-18T19:08:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.321+0000 7fe0b357e700 1 -- 172.21.15.1:0/1741109889 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0ac0cc0f0 msgr2=0x7fe0ac071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.321+0000 7fe0b357e700 1 --2- 172.21.15.1:0/1741109889 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0ac0cc0f0 0x7fe0ac071a80 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac0726b0 tx=0x7fe0a400b330 comp rx=0 tx=0).stop 2024-03-18T19:08:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.321+0000 7fe0b357e700 1 -- 172.21.15.1:0/1741109889 shutdown_connections 2024-03-18T19:08:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.321+0000 7fe0b357e700 1 --2- 172.21.15.1:0/1741109889 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0ac0cc0f0 0x7fe0ac071a80 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.321+0000 7fe0b357e700 1 --2- 172.21.15.1:0/1741109889 >> v2:172.21.15.165:3300/0 conn(0x7fe0ac15a1d0 0x7fe0ac0add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.321+0000 7fe0b357e700 1 -- 172.21.15.1:0/1741109889 >> 172.21.15.1:0/1741109889 conn(0x7fe0ac0cc9b0 msgr2=0x7fe0ac0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:19.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.321+0000 7fe0b357e700 1 -- 172.21.15.1:0/1741109889 shutdown_connections 2024-03-18T19:08:19.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.321+0000 7fe0b357e700 1 -- 172.21.15.1:0/1741109889 wait complete. 2024-03-18T19:08:19.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.323+0000 7fe0b357e700 1 Processor -- start 2024-03-18T19:08:19.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.323+0000 7fe0b357e700 1 -- start start 2024-03-18T19:08:19.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0b357e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0ac071620 0x7fe0ac0c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:19.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0b357e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0ac0cc0f0 0x7fe0ac06e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:19.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0b357e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0ac15a1d0 0x7fe0ac0c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:19.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0b357e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac0c2df0 con 0x7fe0ac0cc0f0 2024-03-18T19:08:19.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0b357e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac06f0c0 con 0x7fe0ac15a1d0 2024-03-18T19:08:19.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0b357e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac06f3a0 con 0x7fe0ac071620 2024-03-18T19:08:19.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0b0b19700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0ac0cc0f0 0x7fe0ac06e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:19.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0b0b19700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0ac0cc0f0 0x7fe0ac06e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59294/0 (socket says 172.21.15.1:59294) 2024-03-18T19:08:19.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0b0b19700 1 -- 172.21.15.1:0/2407697700 learned_addr learned my addr 172.21.15.1:0/2407697700 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:19.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0b1b1b700 1 --2- 172.21.15.1:0/2407697700 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0ac15a1d0 0x7fe0ac0c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:19.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0b0b19700 1 -- 172.21.15.1:0/2407697700 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0ac071620 msgr2=0x7fe0ac0c7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:19.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0b131a700 1 --2- 172.21.15.1:0/2407697700 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0ac071620 0x7fe0ac0c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:19.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0b0b19700 1 --2- 172.21.15.1:0/2407697700 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0ac071620 0x7fe0ac0c7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0b0b19700 1 -- 172.21.15.1:0/2407697700 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0ac15a1d0 msgr2=0x7fe0ac0c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:19.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0b0b19700 1 --2- 172.21.15.1:0/2407697700 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0ac15a1d0 0x7fe0ac0c9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0b0b19700 1 -- 172.21.15.1:0/2407697700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a4009b30 con 0x7fe0ac0cc0f0 2024-03-18T19:08:19.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0b1b1b700 1 --2- 172.21.15.1:0/2407697700 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0ac15a1d0 0x7fe0ac0c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:08:19.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0b0b19700 1 --2- 172.21.15.1:0/2407697700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0ac0cc0f0 0x7fe0ac06e690 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fe09c00b4a0 tx=0x7fe09c00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:19.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0a1ffb700 1 -- 172.21.15.1:0/2407697700 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c011830 con 0x7fe0ac0cc0f0 2024-03-18T19:08:19.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.324+0000 7fe0a1ffb700 1 -- 172.21.15.1:0/2407697700 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09c011990 con 0x7fe0ac0cc0f0 2024-03-18T19:08:19.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.325+0000 7fe0b357e700 1 -- 172.21.15.1:0/2407697700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ac06f680 con 0x7fe0ac0cc0f0 2024-03-18T19:08:19.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.325+0000 7fe0a1ffb700 1 -- 172.21.15.1:0/2407697700 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c011d40 con 0x7fe0ac0cc0f0 2024-03-18T19:08:19.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.325+0000 7fe0b357e700 1 -- 172.21.15.1:0/2407697700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ac06fc40 con 0x7fe0ac0cc0f0 2024-03-18T19:08:19.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.326+0000 7fe0b357e700 1 -- 172.21.15.1:0/2407697700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ac0cf8e0 con 0x7fe0ac0cc0f0 2024-03-18T19:08:19.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.327+0000 7fe0a1ffb700 1 -- 172.21.15.1:0/2407697700 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe09c004020 con 0x7fe0ac0cc0f0 2024-03-18T19:08:19.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.327+0000 7fe0a1ffb700 1 --2- 172.21.15.1:0/2407697700 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe09006adb0 0x7fe09006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:19.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.327+0000 7fe0a1ffb700 1 -- 172.21.15.1:0/2407697700 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5929+0+0 (secure 0 0 0) 0x7fe09c089610 con 0x7fe0ac0cc0f0 2024-03-18T19:08:19.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.327+0000 7fe0b131a700 1 --2- 172.21.15.1:0/2407697700 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe09006adb0 0x7fe09006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:19.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.328+0000 7fe0b131a700 1 --2- 172.21.15.1:0/2407697700 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe09006adb0 0x7fe09006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe0a80088d0 tx=0x7fe0a8008040 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:19.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.331+0000 7fe0a1ffb700 1 -- 172.21.15.1:0/2407697700 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe09c054770 con 0x7fe0ac0cc0f0 2024-03-18T19:08:19.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.580+0000 7fe0b357e700 1 -- 172.21.15.1:0/2407697700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe0ac0ad060 con 0x7fe0ac0cc0f0 2024-03-18T19:08:19.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.585+0000 7fe0a1ffb700 1 -- 172.21.15.1:0/2407697700 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v53) v1 ==== 174+0+0 (secure 0 0 0) 0x7fe09c057d90 con 0x7fe0ac0cc0f0 2024-03-18T19:08:19.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.587+0000 7fe0b357e700 1 -- 172.21.15.1:0/2407697700 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe09006adb0 msgr2=0x7fe09006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:19.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.587+0000 7fe0b357e700 1 --2- 172.21.15.1:0/2407697700 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe09006adb0 0x7fe09006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe0a80088d0 tx=0x7fe0a8008040 comp rx=0 tx=0).stop 2024-03-18T19:08:19.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.587+0000 7fe0b357e700 1 -- 172.21.15.1:0/2407697700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0ac0cc0f0 msgr2=0x7fe0ac06e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:19.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.587+0000 7fe0b357e700 1 --2- 172.21.15.1:0/2407697700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0ac0cc0f0 0x7fe0ac06e690 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fe09c00b4a0 tx=0x7fe09c00b860 comp rx=0 tx=0).stop 2024-03-18T19:08:19.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.588+0000 7fe0b357e700 1 -- 172.21.15.1:0/2407697700 shutdown_connections 2024-03-18T19:08:19.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.588+0000 7fe0b357e700 1 --2- 172.21.15.1:0/2407697700 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe09006adb0 0x7fe09006d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.588+0000 7fe0b357e700 1 --2- 172.21.15.1:0/2407697700 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0ac071620 0x7fe0ac0c7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.588+0000 7fe0b357e700 1 --2- 172.21.15.1:0/2407697700 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0ac15a1d0 0x7fe0ac0c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.588+0000 7fe0b357e700 1 --2- 172.21.15.1:0/2407697700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0ac0cc0f0 0x7fe0ac06e690 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.588+0000 7fe0b357e700 1 -- 172.21.15.1:0/2407697700 >> 172.21.15.1:0/2407697700 conn(0x7fe0ac0cc9b0 msgr2=0x7fe0ac06bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:19.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.588+0000 7fe0b357e700 1 -- 172.21.15.1:0/2407697700 shutdown_connections 2024-03-18T19:08:19.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.588+0000 7fe0b357e700 1 -- 172.21.15.1:0/2407697700 wait complete. 2024-03-18T19:08:19.585 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 15 and data pool 16 2024-03-18T19:08:19.609 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:08:19.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.739+0000 7fee23a62700 1 Processor -- start 2024-03-18T19:08:19.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.739+0000 7fee23a62700 1 -- start start 2024-03-18T19:08:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.740+0000 7fee23a62700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fee1c161090 0x7fee1c1614f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.740+0000 7fee23a62700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee1c161a30 con 0x7fee1c15cc40 2024-03-18T19:08:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.740+0000 7fee23a62700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee1c0d0330 con 0x7fee1c06d440 2024-03-18T19:08:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.740+0000 7fee23a62700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee1c0d07c0 con 0x7fee1c161090 2024-03-18T19:08:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.740+0000 7fee20ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fee1c161090 0x7fee1c1614f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.740+0000 7fee20ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fee1c161090 0x7fee1c1614f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59302/0 (socket says 172.21.15.1:59302) 2024-03-18T19:08:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.740+0000 7fee20ffd700 1 -- 172.21.15.1:0/2297704567 learned_addr learned my addr 172.21.15.1:0/2297704567 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.740+0000 7fee20ffd700 1 -- 172.21.15.1:0/2297704567 >> v1:172.21.15.165:6789/0 conn(0x7fee1c06d440 legacy=0x7fee1c06f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.741+0000 7fee20ffd700 1 -- 172.21.15.1:0/2297704567 >> v1:172.21.15.1:6789/0 conn(0x7fee1c15cc40 legacy=0x7fee1c15d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.741+0000 7fee20ffd700 1 -- 172.21.15.1:0/2297704567 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1c0cf720 con 0x7fee1c161090 2024-03-18T19:08:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.741+0000 7fee20ffd700 1 --2- 172.21.15.1:0/2297704567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee1c161090 0x7fee1c1614f0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fee100098f0 tx=0x7fee10009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e9ce05160234f95 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.741+0000 7fee0ffff700 1 -- 172.21.15.1:0/2297704567 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1000b8a0 con 0x7fee1c161090 2024-03-18T19:08:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.741+0000 7fee0ffff700 1 -- 172.21.15.1:0/2297704567 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1000ba00 con 0x7fee1c161090 2024-03-18T19:08:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.741+0000 7fee0ffff700 1 -- 172.21.15.1:0/2297704567 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1000bd90 con 0x7fee1c161090 2024-03-18T19:08:19.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.741+0000 7fee23a62700 1 -- 172.21.15.1:0/2297704567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee1c161090 msgr2=0x7fee1c1614f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.741+0000 7fee23a62700 1 --2- 172.21.15.1:0/2297704567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee1c161090 0x7fee1c1614f0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fee100098f0 tx=0x7fee10009c00 comp rx=0 tx=0).stop 2024-03-18T19:08:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.742+0000 7fee23a62700 1 -- 172.21.15.1:0/2297704567 shutdown_connections 2024-03-18T19:08:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.742+0000 7fee23a62700 1 --2- 172.21.15.1:0/2297704567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee1c161090 0x7fee1c1614f0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.742+0000 7fee23a62700 1 -- 172.21.15.1:0/2297704567 >> 172.21.15.1:0/2297704567 conn(0x7fee1c0b2ac0 msgr2=0x7fee1c0b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.742+0000 7fee23a62700 1 -- 172.21.15.1:0/2297704567 shutdown_connections 2024-03-18T19:08:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.742+0000 7fee23a62700 1 -- 172.21.15.1:0/2297704567 wait complete. 2024-03-18T19:08:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.743+0000 7fee23a62700 1 Processor -- start 2024-03-18T19:08:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.743+0000 7fee23a62700 1 -- start start 2024-03-18T19:08:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.743+0000 7fee23a62700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee1c06d440 0x7fee1c0ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee23a62700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee1c15cc40 0x7fee1c0add00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee23a62700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee1c161090 0x7fee1c0ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee23a62700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c15d150 con 0x7fee1c06d440 2024-03-18T19:08:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee23a62700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c0caeb0 con 0x7fee1c161090 2024-03-18T19:08:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee23a62700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c0b3ec0 con 0x7fee1c15cc40 2024-03-18T19:08:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee217fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee1c06d440 0x7fee1c0ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee217fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee1c06d440 0x7fee1c0ca970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59310/0 (socket says 172.21.15.1:59310) 2024-03-18T19:08:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee217fe700 1 -- 172.21.15.1:0/693539462 learned_addr learned my addr 172.21.15.1:0/693539462 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee20ffd700 1 --2- 172.21.15.1:0/693539462 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee1c15cc40 0x7fee1c0add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee21fff700 1 --2- 172.21.15.1:0/693539462 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee1c161090 0x7fee1c0ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee217fe700 1 -- 172.21.15.1:0/693539462 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee1c15cc40 msgr2=0x7fee1c0add00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee217fe700 1 --2- 172.21.15.1:0/693539462 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee1c15cc40 0x7fee1c0add00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee217fe700 1 -- 172.21.15.1:0/693539462 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee1c161090 msgr2=0x7fee1c0ae240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee217fe700 1 --2- 172.21.15.1:0/693539462 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee1c161090 0x7fee1c0ae240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee217fe700 1 -- 172.21.15.1:0/693539462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee10009550 con 0x7fee1c06d440 2024-03-18T19:08:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee20ffd700 1 --2- 172.21.15.1:0/693539462 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee1c15cc40 0x7fee1c0add00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:08:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee21fff700 1 --2- 172.21.15.1:0/693539462 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee1c161090 0x7fee1c0ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:08:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee217fe700 1 --2- 172.21.15.1:0/693539462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee1c06d440 0x7fee1c0ca970 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fee1800c450 tx=0x7fee1800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee0dffb700 1 -- 172.21.15.1:0/693539462 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee180052d0 con 0x7fee1c06d440 2024-03-18T19:08:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee0dffb700 1 -- 172.21.15.1:0/693539462 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee18005430 con 0x7fee1c06d440 2024-03-18T19:08:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee23a62700 1 -- 172.21.15.1:0/693539462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1c0b41a0 con 0x7fee1c06d440 2024-03-18T19:08:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.744+0000 7fee0dffb700 1 -- 172.21.15.1:0/693539462 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee180038c0 con 0x7fee1c06d440 2024-03-18T19:08:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.745+0000 7fee23a62700 1 -- 172.21.15.1:0/693539462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1c0b4480 con 0x7fee1c06d440 2024-03-18T19:08:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.745+0000 7fee23a62700 1 -- 172.21.15.1:0/693539462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee1c072cd0 con 0x7fee1c06d440 2024-03-18T19:08:19.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.746+0000 7fee0dffb700 1 -- 172.21.15.1:0/693539462 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fee18003a20 con 0x7fee1c06d440 2024-03-18T19:08:19.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.746+0000 7fee0dffb700 1 --2- 172.21.15.1:0/693539462 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee0006adb0 0x7fee0006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:19.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.747+0000 7fee20ffd700 1 --2- 172.21.15.1:0/693539462 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee0006adb0 0x7fee0006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:19.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.747+0000 7fee0dffb700 1 -- 172.21.15.1:0/693539462 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fee18088a20 con 0x7fee1c06d440 2024-03-18T19:08:19.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.747+0000 7fee20ffd700 1 --2- 172.21.15.1:0/693539462 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee0006adb0 0x7fee0006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fee1c15f8f0 tx=0x7fee10008030 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:19.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.751+0000 7fee0dffb700 1 -- 172.21.15.1:0/693539462 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fee180539b0 con 0x7fee1c06d440 2024-03-18T19:08:19.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.956+0000 7fee23a62700 1 -- 172.21.15.1:0/693539462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee1c073350 con 0x7fee1c06d440 2024-03-18T19:08:19.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.958+0000 7fee0dffb700 1 -- 172.21.15.1:0/693539462 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+16213 (secure 0 0 0) 0x7fee18017090 con 0x7fee1c06d440 2024-03-18T19:08:19.954 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:19.954 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":132,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:08:19.581312+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:08:17.419247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-18T19:08:18.552871+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:08:19.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.960+0000 7fee23a62700 1 -- 172.21.15.1:0/693539462 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee0006adb0 msgr2=0x7fee0006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:19.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.960+0000 7fee23a62700 1 --2- 172.21.15.1:0/693539462 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee0006adb0 0x7fee0006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fee1c15f8f0 tx=0x7fee10008030 comp rx=0 tx=0).stop 2024-03-18T19:08:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.960+0000 7fee23a62700 1 -- 172.21.15.1:0/693539462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee1c06d440 msgr2=0x7fee1c0ca970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.960+0000 7fee23a62700 1 --2- 172.21.15.1:0/693539462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee1c06d440 0x7fee1c0ca970 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fee1800c450 tx=0x7fee1800c810 comp rx=0 tx=0).stop 2024-03-18T19:08:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.960+0000 7fee23a62700 1 -- 172.21.15.1:0/693539462 shutdown_connections 2024-03-18T19:08:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.960+0000 7fee23a62700 1 --2- 172.21.15.1:0/693539462 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fee0006adb0 0x7fee0006d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.960+0000 7fee23a62700 1 --2- 172.21.15.1:0/693539462 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee1c161090 0x7fee1c0ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.960+0000 7fee23a62700 1 --2- 172.21.15.1:0/693539462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee1c06d440 0x7fee1c0ca970 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.960+0000 7fee23a62700 1 --2- 172.21.15.1:0/693539462 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee1c15cc40 0x7fee1c0add00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.960+0000 7fee23a62700 1 -- 172.21.15.1:0/693539462 >> 172.21.15.1:0/693539462 conn(0x7fee1c0b2ac0 msgr2=0x7fee1c0b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.960+0000 7fee23a62700 1 -- 172.21.15.1:0/693539462 shutdown_connections 2024-03-18T19:08:19.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:19.960+0000 7fee23a62700 1 -- 172.21.15.1:0/693539462 wait complete. 2024-03-18T19:08:19.969 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:08:20.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.099+0000 7fd38965d700 1 Processor -- start 2024-03-18T19:08:20.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.099+0000 7fd38965d700 1 -- start start 2024-03-18T19:08:20.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.099+0000 7fd38965d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd38415d470 0x7fd3840cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:20.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.099+0000 7fd38965d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd38415d850 0x7fd3840d17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:20.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.099+0000 7fd38965d700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3840ce7e0 con 0x7fd3840d10d0 2024-03-18T19:08:20.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.099+0000 7fd38965d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3840ceac0 con 0x7fd38415d850 2024-03-18T19:08:20.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.099+0000 7fd38965d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3840ceda0 con 0x7fd38415d470 2024-03-18T19:08:20.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.100+0000 7fd3827fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd38415d850 0x7fd3840d17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:20.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.100+0000 7fd3827fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd38415d850 0x7fd3840d17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59320/0 (socket says 172.21.15.1:59320) 2024-03-18T19:08:20.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.100+0000 7fd3827fc700 1 -- 172.21.15.1:0/1755052309 learned_addr learned my addr 172.21.15.1:0/1755052309 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:20.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.100+0000 7fd382ffd700 1 --2- 172.21.15.1:0/1755052309 >> v2:172.21.15.165:3300/0 conn(0x7fd38415d470 0x7fd3840cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:20.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.100+0000 7fd3827fc700 1 -- 172.21.15.1:0/1755052309 >> v2:172.21.15.165:3300/0 conn(0x7fd38415d470 msgr2=0x7fd3840cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:20.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.100+0000 7fd3827fc700 1 --2- 172.21.15.1:0/1755052309 >> v2:172.21.15.165:3300/0 conn(0x7fd38415d470 0x7fd3840cdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:20.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.100+0000 7fd3827fc700 1 -- 172.21.15.1:0/1755052309 >> v1:172.21.15.165:6789/0 conn(0x7fd3840d10d0 legacy=0x7fd3840b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:20.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.100+0000 7fd3827fc700 1 -- 172.21.15.1:0/1755052309 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3840cf080 con 0x7fd38415d850 2024-03-18T19:08:20.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.100+0000 7fd3827fc700 1 --2- 172.21.15.1:0/1755052309 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd38415d850 0x7fd3840d17d0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fd3840b5db0 tx=0x7fd378009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ddf50a268d0d6c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:20.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.100+0000 7fd381ffb700 1 -- 172.21.15.1:0/1755052309 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd378004230 con 0x7fd38415d850 2024-03-18T19:08:20.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.100+0000 7fd381ffb700 1 -- 172.21.15.1:0/1755052309 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd378004390 con 0x7fd38415d850 2024-03-18T19:08:20.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.100+0000 7fd381ffb700 1 -- 172.21.15.1:0/1755052309 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3780036a0 con 0x7fd38415d850 2024-03-18T19:08:20.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.101+0000 7fd38965d700 1 -- 172.21.15.1:0/1755052309 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd38415d850 msgr2=0x7fd3840d17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:20.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.101+0000 7fd38965d700 1 --2- 172.21.15.1:0/1755052309 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd38415d850 0x7fd3840d17d0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fd3840b5db0 tx=0x7fd378009bd0 comp rx=0 tx=0).stop 2024-03-18T19:08:20.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.101+0000 7fd38965d700 1 -- 172.21.15.1:0/1755052309 shutdown_connections 2024-03-18T19:08:20.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.101+0000 7fd38965d700 1 --2- 172.21.15.1:0/1755052309 >> v2:172.21.15.165:3300/0 conn(0x7fd38415d470 0x7fd3840cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:20.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.101+0000 7fd38965d700 1 --2- 172.21.15.1:0/1755052309 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd38415d850 0x7fd3840d17d0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:20.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.101+0000 7fd38965d700 1 -- 172.21.15.1:0/1755052309 >> 172.21.15.1:0/1755052309 conn(0x7fd38408cc10 msgr2=0x7fd38409c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:20.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.101+0000 7fd38965d700 1 -- 172.21.15.1:0/1755052309 shutdown_connections 2024-03-18T19:08:20.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.101+0000 7fd38965d700 1 -- 172.21.15.1:0/1755052309 wait complete. 2024-03-18T19:08:20.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.102+0000 7fd38965d700 1 Processor -- start 2024-03-18T19:08:20.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.102+0000 7fd38965d700 1 -- start start 2024-03-18T19:08:20.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.103+0000 7fd38965d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3840d10d0 0x7fd3840b5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:20.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.103+0000 7fd38965d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd38415d470 0x7fd38415afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:20.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.103+0000 7fd38965d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3841e70f0 0x7fd3841e7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:20.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.103+0000 7fd38965d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3841e6c60 con 0x7fd3840d10d0 2024-03-18T19:08:20.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.103+0000 7fd38965d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38415f5f0 con 0x7fd38415d470 2024-03-18T19:08:20.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.103+0000 7fd38965d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3841e6330 con 0x7fd3841e70f0 2024-03-18T19:08:20.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.103+0000 7fd382ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3840d10d0 0x7fd3840b5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:20.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.103+0000 7fd382ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3840d10d0 0x7fd3840b5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59332/0 (socket says 172.21.15.1:59332) 2024-03-18T19:08:20.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.103+0000 7fd382ffd700 1 -- 172.21.15.1:0/1932459056 learned_addr learned my addr 172.21.15.1:0/1932459056 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:20.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.103+0000 7fd3827fc700 1 --2- 172.21.15.1:0/1932459056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd38415d470 0x7fd38415afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:20.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.103+0000 7fd3837fe700 1 --2- 172.21.15.1:0/1932459056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3841e70f0 0x7fd3841e7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:20.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.103+0000 7fd382ffd700 1 -- 172.21.15.1:0/1932459056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3841e70f0 msgr2=0x7fd3841e7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:20.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.103+0000 7fd382ffd700 1 --2- 172.21.15.1:0/1932459056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3841e70f0 0x7fd3841e7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:20.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.103+0000 7fd382ffd700 1 -- 172.21.15.1:0/1932459056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd38415d470 msgr2=0x7fd38415afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:20.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.103+0000 7fd382ffd700 1 --2- 172.21.15.1:0/1932459056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd38415d470 0x7fd38415afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:20.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.103+0000 7fd382ffd700 1 -- 172.21.15.1:0/1932459056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd378009550 con 0x7fd3840d10d0 2024-03-18T19:08:20.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.103+0000 7fd3837fe700 1 --2- 172.21.15.1:0/1932459056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3841e70f0 0x7fd3841e7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:20.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.103+0000 7fd382ffd700 1 --2- 172.21.15.1:0/1932459056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3840d10d0 0x7fd3840b5de0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fd37400e810 tx=0x7fd37400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:20.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.104+0000 7fd36b7fe700 1 -- 172.21.15.1:0/1932459056 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd374013610 con 0x7fd3840d10d0 2024-03-18T19:08:20.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.104+0000 7fd36b7fe700 1 -- 172.21.15.1:0/1932459056 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd374013770 con 0x7fd3840d10d0 2024-03-18T19:08:20.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.104+0000 7fd38965d700 1 -- 172.21.15.1:0/1932459056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3841e6610 con 0x7fd3840d10d0 2024-03-18T19:08:20.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.104+0000 7fd36b7fe700 1 -- 172.21.15.1:0/1932459056 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd374013b20 con 0x7fd3840d10d0 2024-03-18T19:08:20.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.104+0000 7fd38965d700 1 -- 172.21.15.1:0/1932459056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3841e7ac0 con 0x7fd3840d10d0 2024-03-18T19:08:20.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.104+0000 7fd38965d700 1 -- 172.21.15.1:0/1932459056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd384073350 con 0x7fd3840d10d0 2024-03-18T19:08:20.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.105+0000 7fd36b7fe700 1 -- 172.21.15.1:0/1932459056 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd374013c80 con 0x7fd3840d10d0 2024-03-18T19:08:20.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.105+0000 7fd36b7fe700 1 --2- 172.21.15.1:0/1932459056 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd37006adb0 0x7fd37006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:20.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.105+0000 7fd36b7fe700 1 -- 172.21.15.1:0/1932459056 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd374089330 con 0x7fd3840d10d0 2024-03-18T19:08:20.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.106+0000 7fd3827fc700 1 --2- 172.21.15.1:0/1932459056 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd37006adb0 0x7fd37006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:20.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.106+0000 7fd3827fc700 1 --2- 172.21.15.1:0/1932459056 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd37006adb0 0x7fd37006d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd378009130 tx=0x7fd378008030 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:20.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.110+0000 7fd36b7fe700 1 -- 172.21.15.1:0/1932459056 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd3740541c0 con 0x7fd3840d10d0 2024-03-18T19:08:20.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.312+0000 7fd38965d700 1 -- 172.21.15.1:0/1932459056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd38415cc70 con 0x7fd3840d10d0 2024-03-18T19:08:20.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.313+0000 7fd36b7fe700 1 -- 172.21.15.1:0/1932459056 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+16213 (secure 0 0 0) 0x7fd3740577e0 con 0x7fd3840d10d0 2024-03-18T19:08:20.309 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:20.309 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":132,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:08:19.581312+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:08:17.419247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-18T19:08:18.552871+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:08:20.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.315+0000 7fd38965d700 1 -- 172.21.15.1:0/1932459056 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd37006adb0 msgr2=0x7fd37006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:20.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.315+0000 7fd38965d700 1 --2- 172.21.15.1:0/1932459056 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd37006adb0 0x7fd37006d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd378009130 tx=0x7fd378008030 comp rx=0 tx=0).stop 2024-03-18T19:08:20.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.315+0000 7fd38965d700 1 -- 172.21.15.1:0/1932459056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3840d10d0 msgr2=0x7fd3840b5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:20.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.315+0000 7fd38965d700 1 --2- 172.21.15.1:0/1932459056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3840d10d0 0x7fd3840b5de0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fd37400e810 tx=0x7fd37400eb20 comp rx=0 tx=0).stop 2024-03-18T19:08:20.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.315+0000 7fd38965d700 1 -- 172.21.15.1:0/1932459056 shutdown_connections 2024-03-18T19:08:20.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.315+0000 7fd38965d700 1 --2- 172.21.15.1:0/1932459056 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd37006adb0 0x7fd37006d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:20.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.315+0000 7fd38965d700 1 --2- 172.21.15.1:0/1932459056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3841e70f0 0x7fd3841e7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:20.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.315+0000 7fd38965d700 1 --2- 172.21.15.1:0/1932459056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3840d10d0 0x7fd3840b5de0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:20.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.315+0000 7fd38965d700 1 --2- 172.21.15.1:0/1932459056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd38415d470 0x7fd38415afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:20.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.315+0000 7fd38965d700 1 -- 172.21.15.1:0/1932459056 >> 172.21.15.1:0/1932459056 conn(0x7fd38408cc10 msgr2=0x7fd3840b3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:20.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.316+0000 7fd38965d700 1 -- 172.21.15.1:0/1932459056 shutdown_connections 2024-03-18T19:08:20.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.316+0000 7fd38965d700 1 -- 172.21.15.1:0/1932459056 wait complete. 2024-03-18T19:08:20.324 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T19:08:20.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.457+0000 7f5a91ebc700 1 Processor -- start 2024-03-18T19:08:20.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.457+0000 7f5a91ebc700 1 -- start start 2024-03-18T19:08:20.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.458+0000 7f5a91ebc700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5a8c15b370 0x7f5a8c0cec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:20.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.458+0000 7f5a91ebc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5a8c0aecb0 0x7f5a8c070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:20.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.458+0000 7f5a91ebc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a8c071380 0x7f5a8c15ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:20.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.458+0000 7f5a91ebc700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a8c0cf510 con 0x7f5a8c0aecb0 2024-03-18T19:08:20.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.458+0000 7f5a91ebc700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a8c071830 con 0x7f5a8c15b370 2024-03-18T19:08:20.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.458+0000 7f5a91ebc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c15b750 con 0x7f5a8c071380 2024-03-18T19:08:20.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.458+0000 7f5a8affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5a8c0aecb0 0x7f5a8c070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:20.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.458+0000 7f5a8affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5a8c0aecb0 0x7f5a8c070e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59346/0 (socket says 172.21.15.1:59346) 2024-03-18T19:08:20.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.458+0000 7f5a8affd700 1 -- 172.21.15.1:0/4287984548 learned_addr learned my addr 172.21.15.1:0/4287984548 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:20.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.458+0000 7f5a8b7fe700 1 --2- 172.21.15.1:0/4287984548 >> v2:172.21.15.165:3300/0 conn(0x7f5a8c15b370 0x7f5a8c0cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.458+0000 7f5a8bfff700 1 --2- 172.21.15.1:0/4287984548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a8c071380 0x7f5a8c15ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.458+0000 7f5a8affd700 1 -- 172.21.15.1:0/4287984548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a8c071380 msgr2=0x7f5a8c15ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.458+0000 7f5a8affd700 1 --2- 172.21.15.1:0/4287984548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a8c071380 0x7f5a8c15ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.458+0000 7f5a8affd700 1 -- 172.21.15.1:0/4287984548 >> v2:172.21.15.165:3300/0 conn(0x7f5a8c15b370 msgr2=0x7f5a8c0cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.458+0000 7f5a8affd700 1 --2- 172.21.15.1:0/4287984548 >> v2:172.21.15.165:3300/0 conn(0x7f5a8c15b370 0x7f5a8c0cec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.458+0000 7f5a8affd700 1 -- 172.21.15.1:0/4287984548 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8c0ccc20 con 0x7f5a8c0aecb0 2024-03-18T19:08:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.458+0000 7f5a8affd700 1 --2- 172.21.15.1:0/4287984548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c0aecb0 0x7f5a8c070e40 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c0b2110 tx=0x7f5a7c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cedb38f3f9917c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.459+0000 7f5a8a7fc700 1 -- 172.21.15.1:0/4287984548 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c00b870 con 0x7f5a8c0aecb0 2024-03-18T19:08:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.459+0000 7f5a8a7fc700 1 -- 172.21.15.1:0/4287984548 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7c00b9d0 con 0x7f5a8c0aecb0 2024-03-18T19:08:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.459+0000 7f5a8a7fc700 1 -- 172.21.15.1:0/4287984548 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c00bd60 con 0x7f5a8c0aecb0 2024-03-18T19:08:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.459+0000 7f5a91ebc700 1 -- 172.21.15.1:0/4287984548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c0aecb0 msgr2=0x7f5a8c070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.459+0000 7f5a91ebc700 1 --2- 172.21.15.1:0/4287984548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c0aecb0 0x7f5a8c070e40 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c0b2110 tx=0x7f5a7c009bd0 comp rx=0 tx=0).stop 2024-03-18T19:08:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.459+0000 7f5a91ebc700 1 -- 172.21.15.1:0/4287984548 shutdown_connections 2024-03-18T19:08:20.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.459+0000 7f5a91ebc700 1 --2- 172.21.15.1:0/4287984548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a8c071380 0x7f5a8c15ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:20.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.459+0000 7f5a91ebc700 1 --2- 172.21.15.1:0/4287984548 >> v2:172.21.15.165:3300/0 conn(0x7f5a8c15b370 0x7f5a8c0cec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:20.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.459+0000 7f5a91ebc700 1 --2- 172.21.15.1:0/4287984548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c0aecb0 0x7f5a8c070e40 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:20.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.459+0000 7f5a91ebc700 1 -- 172.21.15.1:0/4287984548 >> 172.21.15.1:0/4287984548 conn(0x7f5a8c0b1350 msgr2=0x7f5a8c0b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:20.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.459+0000 7f5a91ebc700 1 -- 172.21.15.1:0/4287984548 shutdown_connections 2024-03-18T19:08:20.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.459+0000 7f5a91ebc700 1 -- 172.21.15.1:0/4287984548 wait complete. 2024-03-18T19:08:20.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.460+0000 7f5a91ebc700 1 Processor -- start 2024-03-18T19:08:20.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.461+0000 7f5a91ebc700 1 -- start start 2024-03-18T19:08:20.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.461+0000 7f5a91ebc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a8c071380 0x7f5a8c14c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:20.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.461+0000 7f5a91ebc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c0aecb0 0x7f5a8c15e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:20.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.461+0000 7f5a91ebc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a8c15b370 0x7f5a8c15ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:20.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.461+0000 7f5a91ebc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c14c6e0 con 0x7f5a8c0aecb0 2024-03-18T19:08:20.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.461+0000 7f5a91ebc700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c1592b0 con 0x7f5a8c071380 2024-03-18T19:08:20.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.461+0000 7f5a91ebc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c15e6a0 con 0x7f5a8c15b370 2024-03-18T19:08:20.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.461+0000 7f5a8affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c0aecb0 0x7f5a8c15e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:20.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.461+0000 7f5a8affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c0aecb0 0x7f5a8c15e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59350/0 (socket says 172.21.15.1:59350) 2024-03-18T19:08:20.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.462+0000 7f5a8affd700 1 -- 172.21.15.1:0/1230422618 learned_addr learned my addr 172.21.15.1:0/1230422618 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:20.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.462+0000 7f5a8b7fe700 1 --2- 172.21.15.1:0/1230422618 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a8c071380 0x7f5a8c14c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:20.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.462+0000 7f5a8bfff700 1 --2- 172.21.15.1:0/1230422618 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a8c15b370 0x7f5a8c15ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:20.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.462+0000 7f5a8affd700 1 -- 172.21.15.1:0/1230422618 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a8c15b370 msgr2=0x7f5a8c15ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:20.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.462+0000 7f5a8affd700 1 --2- 172.21.15.1:0/1230422618 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a8c15b370 0x7f5a8c15ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:20.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.462+0000 7f5a8affd700 1 -- 172.21.15.1:0/1230422618 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a8c071380 msgr2=0x7f5a8c14c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:20.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.462+0000 7f5a8affd700 1 --2- 172.21.15.1:0/1230422618 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a8c071380 0x7f5a8c14c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:20.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.462+0000 7f5a8affd700 1 -- 172.21.15.1:0/1230422618 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7c009550 con 0x7f5a8c0aecb0 2024-03-18T19:08:20.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.462+0000 7f5a8b7fe700 1 --2- 172.21.15.1:0/1230422618 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a8c071380 0x7f5a8c14c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:20.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.462+0000 7f5a8affd700 1 --2- 172.21.15.1:0/1230422618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c0aecb0 0x7f5a8c15e160 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c000c00 tx=0x7f5a7c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:20.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.462+0000 7f5a83fff700 1 -- 172.21.15.1:0/1230422618 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c004310 con 0x7f5a8c0aecb0 2024-03-18T19:08:20.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.462+0000 7f5a83fff700 1 -- 172.21.15.1:0/1230422618 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7c01f070 con 0x7f5a8c0aecb0 2024-03-18T19:08:20.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.462+0000 7f5a83fff700 1 -- 172.21.15.1:0/1230422618 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c0036a0 con 0x7f5a8c0aecb0 2024-03-18T19:08:20.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.462+0000 7f5a91ebc700 1 -- 172.21.15.1:0/1230422618 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a8c159530 con 0x7f5a8c0aecb0 2024-03-18T19:08:20.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.462+0000 7f5a91ebc700 1 -- 172.21.15.1:0/1230422618 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a8c159af0 con 0x7f5a8c0aecb0 2024-03-18T19:08:20.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.463+0000 7f5a91ebc700 1 -- 172.21.15.1:0/1230422618 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a8c072dd0 con 0x7f5a8c0aecb0 2024-03-18T19:08:20.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.464+0000 7f5a83fff700 1 -- 172.21.15.1:0/1230422618 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5a7c021030 con 0x7f5a8c0aecb0 2024-03-18T19:08:20.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.464+0000 7f5a83fff700 1 --2- 172.21.15.1:0/1230422618 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5a6c06adb0 0x7f5a6c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:20.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.464+0000 7f5a83fff700 1 -- 172.21.15.1:0/1230422618 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5a7c08a7a0 con 0x7f5a8c0aecb0 2024-03-18T19:08:20.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.464+0000 7f5a8b7fe700 1 --2- 172.21.15.1:0/1230422618 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5a6c06adb0 0x7f5a6c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:20.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.465+0000 7f5a8b7fe700 1 --2- 172.21.15.1:0/1230422618 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5a6c06adb0 0x7f5a6c06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5a74007a50 tx=0x7f5a74008040 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:20.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.468+0000 7f5a83fff700 1 -- 172.21.15.1:0/1230422618 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5a7c0557c0 con 0x7f5a8c0aecb0 2024-03-18T19:08:20.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:20.721+0000 7f5a91ebc700 1 -- 172.21.15.1:0/1230422618 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5a8c0ae730 con 0x7f5a8c0aecb0 2024-03-18T19:08:21.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.590+0000 7f5a83fff700 1 -- 172.21.15.1:0/1230422618 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v56) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5a7c058de0 con 0x7f5a8c0aecb0 2024-03-18T19:08:21.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.593+0000 7f5a91ebc700 1 -- 172.21.15.1:0/1230422618 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5a6c06adb0 msgr2=0x7f5a6c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:21.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.593+0000 7f5a91ebc700 1 --2- 172.21.15.1:0/1230422618 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5a6c06adb0 0x7f5a6c06d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5a74007a50 tx=0x7f5a74008040 comp rx=0 tx=0).stop 2024-03-18T19:08:21.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.593+0000 7f5a91ebc700 1 -- 172.21.15.1:0/1230422618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c0aecb0 msgr2=0x7f5a8c15e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:21.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.593+0000 7f5a91ebc700 1 --2- 172.21.15.1:0/1230422618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c0aecb0 0x7f5a8c15e160 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c000c00 tx=0x7f5a7c004750 comp rx=0 tx=0).stop 2024-03-18T19:08:21.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.593+0000 7f5a91ebc700 1 -- 172.21.15.1:0/1230422618 shutdown_connections 2024-03-18T19:08:21.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.593+0000 7f5a91ebc700 1 --2- 172.21.15.1:0/1230422618 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5a6c06adb0 0x7f5a6c06d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:21.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.593+0000 7f5a91ebc700 1 --2- 172.21.15.1:0/1230422618 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a8c15b370 0x7f5a8c15ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:21.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.593+0000 7f5a91ebc700 1 --2- 172.21.15.1:0/1230422618 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a8c071380 0x7f5a8c14c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:21.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.593+0000 7f5a91ebc700 1 --2- 172.21.15.1:0/1230422618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a8c0aecb0 0x7f5a8c15e160 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:21.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.593+0000 7f5a91ebc700 1 -- 172.21.15.1:0/1230422618 >> 172.21.15.1:0/1230422618 conn(0x7f5a8c0b1350 msgr2=0x7f5a8c0af2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:21.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.593+0000 7f5a91ebc700 1 -- 172.21.15.1:0/1230422618 shutdown_connections 2024-03-18T19:08:21.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.593+0000 7f5a91ebc700 1 -- 172.21.15.1:0/1230422618 wait complete. 2024-03-18T19:08:21.604 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-18T19:08:21.605 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T19:08:21.738 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:08:21.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.743+0000 7f1627fc1700 1 Processor -- start 2024-03-18T19:08:21.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.743+0000 7f1627fc1700 1 -- start start 2024-03-18T19:08:21.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.744+0000 7f1627fc1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f162006eda0 0x7f1620161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:21.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.744+0000 7f1627fc1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f162006f180 0x7f16200ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:21.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.744+0000 7f1627fc1700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16200caac0 con 0x7f162006a750 2024-03-18T19:08:21.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.744+0000 7f1627fc1700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16200cb320 con 0x7f162006eda0 2024-03-18T19:08:21.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.744+0000 7f1627fc1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16200ce010 con 0x7f162006f180 2024-03-18T19:08:21.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.744+0000 7f1625d5d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f162006eda0 0x7f1620161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:21.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.744+0000 7f162555c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f162006f180 0x7f16200ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:21.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.744+0000 7f162655e700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f162006a750 0x7f16200715b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:37890/0 (socket says 172.21.15.1:37890) 2024-03-18T19:08:21.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.744+0000 7f162655e700 1 -- 172.21.15.1:0/3850376163 learned_addr learned my addr 172.21.15.1:0/3850376163 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:21.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:08:21.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.745+0000 7f1624d5b700 1 -- 172.21.15.1:0/3850376163 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388319079 0 0) 0x7f16200caac0 con 0x7f162006a750 2024-03-18T19:08:21.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.745+0000 7f1624d5b700 1 -- 172.21.15.1:0/3850376163 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f160c0035b0 con 0x7f162006a750 2024-03-18T19:08:21.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.745+0000 7f1625d5d700 1 -- 172.21.15.1:0/3850376163 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f162006f180 msgr2=0x7f16200ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:21.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.745+0000 7f1625d5d700 1 --2- 172.21.15.1:0/3850376163 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f162006f180 0x7f16200ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:21.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.745+0000 7f1625d5d700 1 -- 172.21.15.1:0/3850376163 >> v1:172.21.15.165:6789/0 conn(0x7f162006a750 legacy=0x7f16200715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:21.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.745+0000 7f1625d5d700 1 -- 172.21.15.1:0/3850376163 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16200cad60 con 0x7f162006eda0 2024-03-18T19:08:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.745+0000 7f1625d5d700 1 --2- 172.21.15.1:0/3850376163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f162006eda0 0x7f1620161c60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f1610009840 tx=0x7f1610009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79f18d8edb44ddaa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.745+0000 7f1624d5b700 1 -- 172.21.15.1:0/3850376163 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1610011830 con 0x7f162006eda0 2024-03-18T19:08:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.745+0000 7f1624d5b700 1 -- 172.21.15.1:0/3850376163 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1610011990 con 0x7f162006eda0 2024-03-18T19:08:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.745+0000 7f1624d5b700 1 -- 172.21.15.1:0/3850376163 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1610011d20 con 0x7f162006eda0 2024-03-18T19:08:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.746+0000 7f1627fc1700 1 -- 172.21.15.1:0/3850376163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f162006eda0 msgr2=0x7f1620161c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.746+0000 7f1627fc1700 1 --2- 172.21.15.1:0/3850376163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f162006eda0 0x7f1620161c60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f1610009840 tx=0x7f1610009b50 comp rx=0 tx=0).stop 2024-03-18T19:08:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.746+0000 7f1627fc1700 1 -- 172.21.15.1:0/3850376163 shutdown_connections 2024-03-18T19:08:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.746+0000 7f1627fc1700 1 --2- 172.21.15.1:0/3850376163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f162006eda0 0x7f1620161c60 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.746+0000 7f1627fc1700 1 --2- 172.21.15.1:0/3850376163 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f162006f180 0x7f16200ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.746+0000 7f1627fc1700 1 -- 172.21.15.1:0/3850376163 >> 172.21.15.1:0/3850376163 conn(0x7f16200cff30 msgr2=0x7f16200d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.746+0000 7f1627fc1700 1 -- 172.21.15.1:0/3850376163 shutdown_connections 2024-03-18T19:08:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.746+0000 7f1627fc1700 1 -- 172.21.15.1:0/3850376163 wait complete. 2024-03-18T19:08:21.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.747+0000 7f1627fc1700 1 Processor -- start 2024-03-18T19:08:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.747+0000 7f1627fc1700 1 -- start start 2024-03-18T19:08:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.748+0000 7f1627fc1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f162006a750 0x7f16200c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.748+0000 7f1627fc1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f162006eda0 0x7f16200b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.748+0000 7f1627fc1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f162006f180 0x7f16200ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.748+0000 7f1627fc1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16201f2f00 con 0x7f162006eda0 2024-03-18T19:08:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.748+0000 7f1627fc1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16200c9d20 con 0x7f162006f180 2024-03-18T19:08:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.748+0000 7f1627fc1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16200ca000 con 0x7f162006a750 2024-03-18T19:08:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.748+0000 7f162555c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f162006eda0 0x7f16200b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.748+0000 7f162555c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f162006eda0 0x7f16200b3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59366/0 (socket says 172.21.15.1:59366) 2024-03-18T19:08:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.748+0000 7f162555c700 1 -- 172.21.15.1:0/366893716 learned_addr learned my addr 172.21.15.1:0/366893716 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.748+0000 7f162655e700 1 --2- 172.21.15.1:0/366893716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f162006f180 0x7f16200ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.748+0000 7f1625d5d700 1 --2- 172.21.15.1:0/366893716 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f162006a750 0x7f16200c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:21.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.748+0000 7f162555c700 1 -- 172.21.15.1:0/366893716 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f162006a750 msgr2=0x7f16200c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.748+0000 7f162555c700 1 --2- 172.21.15.1:0/366893716 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f162006a750 0x7f16200c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.748+0000 7f162555c700 1 -- 172.21.15.1:0/366893716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f162006f180 msgr2=0x7f16200ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.748+0000 7f162555c700 1 --2- 172.21.15.1:0/366893716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f162006f180 0x7f16200ae040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.748+0000 7f162555c700 1 -- 172.21.15.1:0/366893716 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16100094a0 con 0x7f162006eda0 2024-03-18T19:08:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.748+0000 7f1625d5d700 1 --2- 172.21.15.1:0/366893716 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f162006a750 0x7f16200c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.748+0000 7f162655e700 1 --2- 172.21.15.1:0/366893716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f162006f180 0x7f16200ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.749+0000 7f162555c700 1 --2- 172.21.15.1:0/366893716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f162006eda0 0x7f16200b3dd0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f161c00e870 tx=0x7f161c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.749+0000 7f16167fc700 1 -- 172.21.15.1:0/366893716 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c009590 con 0x7f162006eda0 2024-03-18T19:08:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.749+0000 7f16167fc700 1 -- 172.21.15.1:0/366893716 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f161c00ee70 con 0x7f162006eda0 2024-03-18T19:08:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.749+0000 7f16167fc700 1 -- 172.21.15.1:0/366893716 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c00ca50 con 0x7f162006eda0 2024-03-18T19:08:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.749+0000 7f1627fc1700 1 -- 172.21.15.1:0/366893716 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16200ca2e0 con 0x7f162006eda0 2024-03-18T19:08:21.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.749+0000 7f1627fc1700 1 -- 172.21.15.1:0/366893716 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16200b4310 con 0x7f162006eda0 2024-03-18T19:08:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.750+0000 7f1627fc1700 1 -- 172.21.15.1:0/366893716 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16200cba30 con 0x7f162006eda0 2024-03-18T19:08:21.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.751+0000 7f16167fc700 1 -- 172.21.15.1:0/366893716 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f161c00cbb0 con 0x7f162006eda0 2024-03-18T19:08:21.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.751+0000 7f16167fc700 1 --2- 172.21.15.1:0/366893716 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f160406adb0 0x7f160406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:21.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.751+0000 7f16167fc700 1 -- 172.21.15.1:0/366893716 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f161c089cb0 con 0x7f162006eda0 2024-03-18T19:08:21.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.751+0000 7f1625d5d700 1 --2- 172.21.15.1:0/366893716 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f160406adb0 0x7f160406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:21.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.752+0000 7f1625d5d700 1 --2- 172.21.15.1:0/366893716 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f160406adb0 0x7f160406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1610000c00 tx=0x7f1610008030 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:21.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:21.755+0000 7f16167fc700 1 -- 172.21.15.1:0/366893716 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f161c054d20 con 0x7f162006eda0 2024-03-18T19:08:21.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.003+0000 7f1627fc1700 1 -- 172.21.15.1:0/366893716 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f16200676a0 con 0x7f162006eda0 2024-03-18T19:08:22.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.596+0000 7f16167fc700 1 -- 172.21.15.1:0/366893716 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v57) v1 ==== 122+0+0 (secure 0 0 0) 0x7f161c0170c0 con 0x7f162006eda0 2024-03-18T19:08:22.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.598+0000 7f1627fc1700 1 -- 172.21.15.1:0/366893716 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f160406adb0 msgr2=0x7f160406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:22.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.598+0000 7f1627fc1700 1 --2- 172.21.15.1:0/366893716 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f160406adb0 0x7f160406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1610000c00 tx=0x7f1610008030 comp rx=0 tx=0).stop 2024-03-18T19:08:22.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.598+0000 7f1627fc1700 1 -- 172.21.15.1:0/366893716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f162006eda0 msgr2=0x7f16200b3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:22.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.598+0000 7f1627fc1700 1 --2- 172.21.15.1:0/366893716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f162006eda0 0x7f16200b3dd0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f161c00e870 tx=0x7f161c00ec30 comp rx=0 tx=0).stop 2024-03-18T19:08:22.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.599+0000 7f1627fc1700 1 -- 172.21.15.1:0/366893716 shutdown_connections 2024-03-18T19:08:22.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.599+0000 7f1627fc1700 1 --2- 172.21.15.1:0/366893716 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f160406adb0 0x7f160406d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:22.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.599+0000 7f1627fc1700 1 --2- 172.21.15.1:0/366893716 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f162006a750 0x7f16200c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:22.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.599+0000 7f1627fc1700 1 --2- 172.21.15.1:0/366893716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f162006f180 0x7f16200ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:22.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.599+0000 7f1627fc1700 1 --2- 172.21.15.1:0/366893716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f162006eda0 0x7f16200b3dd0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:22.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.599+0000 7f1627fc1700 1 -- 172.21.15.1:0/366893716 >> 172.21.15.1:0/366893716 conn(0x7f16200cff30 msgr2=0x7f16200b01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:22.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.599+0000 7f1627fc1700 1 -- 172.21.15.1:0/366893716 shutdown_connections 2024-03-18T19:08:22.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.599+0000 7f1627fc1700 1 -- 172.21.15.1:0/366893716 wait complete. 2024-03-18T19:08:22.610 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T19:08:22.610 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:08:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.745+0000 7fb0c7ec2700 1 Processor -- start 2024-03-18T19:08:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.746+0000 7fb0c7ec2700 1 -- start start 2024-03-18T19:08:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.746+0000 7fb0c7ec2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb0c015b4d0 0x7fb0c009a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.746+0000 7fb0c7ec2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb0c00cf750 0x7fb0c0099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.746+0000 7fb0c7ec2700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c006f070 con 0x7fb0c0099f80 2024-03-18T19:08:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.746+0000 7fb0c7ec2700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c009a410 con 0x7fb0c00cf750 2024-03-18T19:08:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.746+0000 7fb0c7ec2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c00d0260 con 0x7fb0c015b4d0 2024-03-18T19:08:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.746+0000 7fb0c545d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb0c00cf750 0x7fb0c0099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.747+0000 7fb0c545d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb0c00cf750 0x7fb0c0099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59378/0 (socket says 172.21.15.1:59378) 2024-03-18T19:08:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.747+0000 7fb0c545d700 1 -- 172.21.15.1:0/607058896 learned_addr learned my addr 172.21.15.1:0/607058896 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.747+0000 7fb0c5c5e700 1 --2- 172.21.15.1:0/607058896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb0c015b4d0 0x7fb0c009a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.747+0000 7fb0c545d700 1 -- 172.21.15.1:0/607058896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb0c015b4d0 msgr2=0x7fb0c009a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.747+0000 7fb0c545d700 1 --2- 172.21.15.1:0/607058896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb0c015b4d0 0x7fb0c009a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.747+0000 7fb0c545d700 1 -- 172.21.15.1:0/607058896 >> v1:172.21.15.1:6789/0 conn(0x7fb0c0099f80 legacy=0x7fb0c00cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.747+0000 7fb0c545d700 1 -- 172.21.15.1:0/607058896 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c00cff80 con 0x7fb0c00cf750 2024-03-18T19:08:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.747+0000 7fb0c4c5c700 1 -- 172.21.15.1:0/607058896 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500063212 0 0) 0x7fb0c006f070 con 0x7fb0c0099f80 2024-03-18T19:08:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.747+0000 7fb0c545d700 1 --2- 172.21.15.1:0/607058896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0c00cf750 0x7fb0c0099a40 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fb0b40098f0 tx=0x7fb0b4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da15be29aa9be655 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.747+0000 7fb0c4c5c700 1 -- 172.21.15.1:0/607058896 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b400b8a0 con 0x7fb0c00cf750 2024-03-18T19:08:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.747+0000 7fb0c4c5c700 1 -- 172.21.15.1:0/607058896 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b400ba00 con 0x7fb0c00cf750 2024-03-18T19:08:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.747+0000 7fb0c4c5c700 1 -- 172.21.15.1:0/607058896 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b400bd90 con 0x7fb0c00cf750 2024-03-18T19:08:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.748+0000 7fb0c7ec2700 1 -- 172.21.15.1:0/607058896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0c00cf750 msgr2=0x7fb0c0099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.748+0000 7fb0c7ec2700 1 --2- 172.21.15.1:0/607058896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0c00cf750 0x7fb0c0099a40 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fb0b40098f0 tx=0x7fb0b4009c00 comp rx=0 tx=0).stop 2024-03-18T19:08:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.748+0000 7fb0c7ec2700 1 -- 172.21.15.1:0/607058896 shutdown_connections 2024-03-18T19:08:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.748+0000 7fb0c7ec2700 1 --2- 172.21.15.1:0/607058896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb0c015b4d0 0x7fb0c009a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:22.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.748+0000 7fb0c7ec2700 1 --2- 172.21.15.1:0/607058896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0c00cf750 0x7fb0c0099a40 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:22.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.748+0000 7fb0c7ec2700 1 -- 172.21.15.1:0/607058896 >> 172.21.15.1:0/607058896 conn(0x7fb0c00b2cb0 msgr2=0x7fb0c00b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:22.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.748+0000 7fb0c7ec2700 1 -- 172.21.15.1:0/607058896 shutdown_connections 2024-03-18T19:08:22.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.748+0000 7fb0c7ec2700 1 -- 172.21.15.1:0/607058896 wait complete. 2024-03-18T19:08:22.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.749+0000 7fb0c7ec2700 1 Processor -- start 2024-03-18T19:08:22.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.749+0000 7fb0c7ec2700 1 -- start start 2024-03-18T19:08:22.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.750+0000 7fb0c7ec2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0c0099f80 0x7fb0c0157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.750+0000 7fb0c7ec2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0c00cf750 0x7fb0c00b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.750+0000 7fb0c7ec2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb0c015b4d0 0x7fb0c0158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.750+0000 7fb0c7ec2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c00b6380 con 0x7fb0c00cf750 2024-03-18T19:08:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.750+0000 7fb0c7ec2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c00b60a0 con 0x7fb0c0099f80 2024-03-18T19:08:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.750+0000 7fb0c7ec2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c00b46a0 con 0x7fb0c015b4d0 2024-03-18T19:08:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.750+0000 7fb0c545d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0c00cf750 0x7fb0c00b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.750+0000 7fb0c545d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0c00cf750 0x7fb0c00b5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59382/0 (socket says 172.21.15.1:59382) 2024-03-18T19:08:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.750+0000 7fb0c545d700 1 -- 172.21.15.1:0/2560295560 learned_addr learned my addr 172.21.15.1:0/2560295560 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.750+0000 7fb0c645f700 1 --2- 172.21.15.1:0/2560295560 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb0c015b4d0 0x7fb0c0158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.750+0000 7fb0c5c5e700 1 --2- 172.21.15.1:0/2560295560 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0c0099f80 0x7fb0c0157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.750+0000 7fb0c545d700 1 -- 172.21.15.1:0/2560295560 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb0c015b4d0 msgr2=0x7fb0c0158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.750+0000 7fb0c545d700 1 --2- 172.21.15.1:0/2560295560 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb0c015b4d0 0x7fb0c0158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:22.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.750+0000 7fb0c545d700 1 -- 172.21.15.1:0/2560295560 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0c0099f80 msgr2=0x7fb0c0157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.750+0000 7fb0c545d700 1 --2- 172.21.15.1:0/2560295560 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0c0099f80 0x7fb0c0157ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.750+0000 7fb0c545d700 1 -- 172.21.15.1:0/2560295560 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b4009550 con 0x7fb0c00cf750 2024-03-18T19:08:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.751+0000 7fb0c545d700 1 --2- 172.21.15.1:0/2560295560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0c00cf750 0x7fb0c00b5180 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0072230 tx=0x7fb0b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.751+0000 7fb0b27fc700 1 -- 172.21.15.1:0/2560295560 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4004240 con 0x7fb0c00cf750 2024-03-18T19:08:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.751+0000 7fb0b27fc700 1 -- 172.21.15.1:0/2560295560 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b401f070 con 0x7fb0c00cf750 2024-03-18T19:08:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.751+0000 7fb0c7ec2700 1 -- 172.21.15.1:0/2560295560 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c00b4980 con 0x7fb0c00cf750 2024-03-18T19:08:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.751+0000 7fb0b27fc700 1 -- 172.21.15.1:0/2560295560 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b40036a0 con 0x7fb0c00cf750 2024-03-18T19:08:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.751+0000 7fb0c7ec2700 1 -- 172.21.15.1:0/2560295560 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c00b6920 con 0x7fb0c00cf750 2024-03-18T19:08:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.752+0000 7fb0c7ec2700 1 -- 172.21.15.1:0/2560295560 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c0068240 con 0x7fb0c00cf750 2024-03-18T19:08:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.752+0000 7fb0b27fc700 1 -- 172.21.15.1:0/2560295560 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb0b4021030 con 0x7fb0c00cf750 2024-03-18T19:08:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.753+0000 7fb0b27fc700 1 --2- 172.21.15.1:0/2560295560 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb0a406adb0 0x7fb0a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.753+0000 7fb0b27fc700 1 -- 172.21.15.1:0/2560295560 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb0b408a920 con 0x7fb0c00cf750 2024-03-18T19:08:22.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.753+0000 7fb0c5c5e700 1 --2- 172.21.15.1:0/2560295560 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb0a406adb0 0x7fb0a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:22.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.753+0000 7fb0c5c5e700 1 --2- 172.21.15.1:0/2560295560 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb0a406adb0 0x7fb0a406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc006b20 tx=0x7fb0bc008040 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:22.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:22.757+0000 7fb0b27fc700 1 -- 172.21.15.1:0/2560295560 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb0b40558c0 con 0x7fb0c00cf750 2024-03-18T19:08:22.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.001+0000 7fb0c7ec2700 1 -- 172.21.15.1:0/2560295560 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0c00b4ea0 con 0x7fb0c00cf750 2024-03-18T19:08:22.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.002+0000 7fb0b27fc700 1 -- 172.21.15.1:0/2560295560 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4730 (secure 0 0 0) 0x7fb0c00b4ea0 con 0x7fb0c00cf750 2024-03-18T19:08:22.998 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:22.998 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6800,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3354770356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3354770356},{"type":"v1","addr":"172.21.15.1:6837","nonce":3354770356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":6809,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/695588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":695588699},{"type":"v1","addr":"172.21.15.165:6834","nonce":695588699}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7660,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3186346302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3186346302},{"type":"v1","addr":"172.21.15.1:6836","nonce":3186346302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:08:19.581625+0000","modified":"2024-03-18T19:08:22.589436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7675},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7675":{"gid":7675,"name":"d","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/450810178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":450810178},{"type":"v1","addr":"172.21.15.165:6835","nonce":450810178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-18T19:08:23.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.004+0000 7fb0c7ec2700 1 -- 172.21.15.1:0/2560295560 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb0a406adb0 msgr2=0x7fb0a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:23.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.004+0000 7fb0c7ec2700 1 --2- 172.21.15.1:0/2560295560 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb0a406adb0 0x7fb0a406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc006b20 tx=0x7fb0bc008040 comp rx=0 tx=0).stop 2024-03-18T19:08:23.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.004+0000 7fb0c7ec2700 1 -- 172.21.15.1:0/2560295560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0c00cf750 msgr2=0x7fb0c00b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:23.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.004+0000 7fb0c7ec2700 1 --2- 172.21.15.1:0/2560295560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0c00cf750 0x7fb0c00b5180 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0072230 tx=0x7fb0b4004750 comp rx=0 tx=0).stop 2024-03-18T19:08:23.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.004+0000 7fb0c7ec2700 1 -- 172.21.15.1:0/2560295560 shutdown_connections 2024-03-18T19:08:23.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.004+0000 7fb0c7ec2700 1 --2- 172.21.15.1:0/2560295560 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb0a406adb0 0x7fb0a406d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.004+0000 7fb0c7ec2700 1 --2- 172.21.15.1:0/2560295560 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb0c015b4d0 0x7fb0c0158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.004+0000 7fb0c7ec2700 1 --2- 172.21.15.1:0/2560295560 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0c0099f80 0x7fb0c0157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.004+0000 7fb0c7ec2700 1 --2- 172.21.15.1:0/2560295560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0c00cf750 0x7fb0c00b5180 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.004+0000 7fb0c7ec2700 1 -- 172.21.15.1:0/2560295560 >> 172.21.15.1:0/2560295560 conn(0x7fb0c00b2cb0 msgr2=0x7fb0c00afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:23.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.004+0000 7fb0c7ec2700 1 -- 172.21.15.1:0/2560295560 shutdown_connections 2024-03-18T19:08:23.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.004+0000 7fb0c7ec2700 1 -- 172.21.15.1:0/2560295560 wait complete. 2024-03-18T19:08:23.001 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 57 2024-03-18T19:08:23.012 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:08:23.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.140+0000 7f6c65b83700 1 Processor -- start 2024-03-18T19:08:23.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.140+0000 7f6c65b83700 1 -- start start 2024-03-18T19:08:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.141+0000 7f6c65b83700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6c60143c20 0x7f6c600b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.141+0000 7f6c65b83700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6c6006f240 0x7f6c600cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.141+0000 7f6c65b83700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c60144000 con 0x7f6c600b3f40 2024-03-18T19:08:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.141+0000 7f6c65b83700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c600b43c0 con 0x7f6c6006f240 2024-03-18T19:08:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.141+0000 7f6c65b83700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c600b51b0 con 0x7f6c60143c20 2024-03-18T19:08:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.141+0000 7f6c5ffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6c6006f240 0x7f6c600cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.141+0000 7f6c5ffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6c6006f240 0x7f6c600cddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59390/0 (socket says 172.21.15.1:59390) 2024-03-18T19:08:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.141+0000 7f6c5ffff700 1 -- 172.21.15.1:0/862919865 learned_addr learned my addr 172.21.15.1:0/862919865 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.141+0000 7f6c5f7fe700 1 --2- 172.21.15.1:0/862919865 >> v2:172.21.15.165:3300/0 conn(0x7f6c60143c20 0x7f6c600b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.142+0000 7f6c5ffff700 1 -- 172.21.15.1:0/862919865 >> v2:172.21.15.165:3300/0 conn(0x7f6c60143c20 msgr2=0x7f6c600b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.142+0000 7f6c5ffff700 1 --2- 172.21.15.1:0/862919865 >> v2:172.21.15.165:3300/0 conn(0x7f6c60143c20 0x7f6c600b3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.142+0000 7f6c5ffff700 1 -- 172.21.15.1:0/862919865 >> v1:172.21.15.165:6789/0 conn(0x7f6c600b3f40 legacy=0x7f6c6006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.142+0000 7f6c5ffff700 1 -- 172.21.15.1:0/862919865 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c600cd8d0 con 0x7f6c6006f240 2024-03-18T19:08:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.142+0000 7f6c5ffff700 1 --2- 172.21.15.1:0/862919865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6006f240 0x7f6c600cddb0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f6c6006d080 tx=0x7f6c5800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=128ba1c5d02569e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.142+0000 7f6c5e7fc700 1 -- 172.21.15.1:0/862919865 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c58010070 con 0x7f6c6006f240 2024-03-18T19:08:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.142+0000 7f6c5e7fc700 1 -- 172.21.15.1:0/862919865 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c5800dce0 con 0x7f6c6006f240 2024-03-18T19:08:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.142+0000 7f6c5e7fc700 1 -- 172.21.15.1:0/862919865 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c5800b310 con 0x7f6c6006f240 2024-03-18T19:08:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.143+0000 7f6c65b83700 1 -- 172.21.15.1:0/862919865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6006f240 msgr2=0x7f6c600cddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.143+0000 7f6c65b83700 1 --2- 172.21.15.1:0/862919865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6006f240 0x7f6c600cddb0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f6c6006d080 tx=0x7f6c5800d430 comp rx=0 tx=0).stop 2024-03-18T19:08:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.143+0000 7f6c65b83700 1 -- 172.21.15.1:0/862919865 shutdown_connections 2024-03-18T19:08:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.143+0000 7f6c65b83700 1 --2- 172.21.15.1:0/862919865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6006f240 0x7f6c600cddb0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.143+0000 7f6c65b83700 1 --2- 172.21.15.1:0/862919865 >> v2:172.21.15.165:3300/0 conn(0x7f6c60143c20 0x7f6c600b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.143+0000 7f6c65b83700 1 -- 172.21.15.1:0/862919865 >> 172.21.15.1:0/862919865 conn(0x7f6c600b1d20 msgr2=0x7f6c600b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.143+0000 7f6c65b83700 1 -- 172.21.15.1:0/862919865 shutdown_connections 2024-03-18T19:08:23.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.143+0000 7f6c65b83700 1 -- 172.21.15.1:0/862919865 wait complete. 2024-03-18T19:08:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.144+0000 7f6c65b83700 1 Processor -- start 2024-03-18T19:08:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.144+0000 7f6c65b83700 1 -- start start 2024-03-18T19:08:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.145+0000 7f6c65b83700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6006f240 0x7f6c601ef5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.145+0000 7f6c65b83700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c600b3f40 0x7f6c601efd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.145+0000 7f6c65b83700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c60143c20 0x7f6c601ee1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.145+0000 7f6c65b83700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c601efb20 con 0x7f6c6006f240 2024-03-18T19:08:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.145+0000 7f6c65b83700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6013fd80 con 0x7f6c600b3f40 2024-03-18T19:08:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.145+0000 7f6c65b83700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c60140060 con 0x7f6c60143c20 2024-03-18T19:08:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.145+0000 7f6c5f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6006f240 0x7f6c601ef5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.145+0000 7f6c5f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6006f240 0x7f6c601ef5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59396/0 (socket says 172.21.15.1:59396) 2024-03-18T19:08:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.145+0000 7f6c5f7fe700 1 -- 172.21.15.1:0/1763882257 learned_addr learned my addr 172.21.15.1:0/1763882257 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.145+0000 7f6c5ffff700 1 --2- 172.21.15.1:0/1763882257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c60143c20 0x7f6c601ee1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.145+0000 7f6c5effd700 1 --2- 172.21.15.1:0/1763882257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c600b3f40 0x7f6c601efd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.145+0000 7f6c5f7fe700 1 -- 172.21.15.1:0/1763882257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c60143c20 msgr2=0x7f6c601ee1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.145+0000 7f6c5f7fe700 1 --2- 172.21.15.1:0/1763882257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c60143c20 0x7f6c601ee1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.145+0000 7f6c5f7fe700 1 -- 172.21.15.1:0/1763882257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c600b3f40 msgr2=0x7f6c601efd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.145+0000 7f6c5f7fe700 1 --2- 172.21.15.1:0/1763882257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c600b3f40 0x7f6c601efd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.145+0000 7f6c5f7fe700 1 -- 172.21.15.1:0/1763882257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5800bd40 con 0x7f6c6006f240 2024-03-18T19:08:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.145+0000 7f6c5f7fe700 1 --2- 172.21.15.1:0/1763882257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6006f240 0x7f6c601ef5e0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f6c5400e870 tx=0x7f6c5400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.146+0000 7f6c47fff700 1 -- 172.21.15.1:0/1763882257 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c54009590 con 0x7f6c6006f240 2024-03-18T19:08:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.146+0000 7f6c47fff700 1 -- 172.21.15.1:0/1763882257 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c5400ee70 con 0x7f6c6006f240 2024-03-18T19:08:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.146+0000 7f6c65b83700 1 -- 172.21.15.1:0/1763882257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c60140340 con 0x7f6c6006f240 2024-03-18T19:08:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.146+0000 7f6c47fff700 1 -- 172.21.15.1:0/1763882257 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c5400ca50 con 0x7f6c6006f240 2024-03-18T19:08:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.146+0000 7f6c65b83700 1 -- 172.21.15.1:0/1763882257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c60140900 con 0x7f6c6006f240 2024-03-18T19:08:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.147+0000 7f6c65b83700 1 -- 172.21.15.1:0/1763882257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c600cc3a0 con 0x7f6c6006f240 2024-03-18T19:08:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.147+0000 7f6c47fff700 1 -- 172.21.15.1:0/1763882257 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6c5400cbb0 con 0x7f6c6006f240 2024-03-18T19:08:23.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.147+0000 7f6c47fff700 1 --2- 172.21.15.1:0/1763882257 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c4c06adb0 0x7f6c4c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.147+0000 7f6c47fff700 1 -- 172.21.15.1:0/1763882257 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6c54089540 con 0x7f6c6006f240 2024-03-18T19:08:23.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.148+0000 7f6c5effd700 1 --2- 172.21.15.1:0/1763882257 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c4c06adb0 0x7f6c4c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.148+0000 7f6c5effd700 1 --2- 172.21.15.1:0/1763882257 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c4c06adb0 0x7f6c4c06d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6c600b0e70 tx=0x7f6c50009040 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:23.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.152+0000 7f6c47fff700 1 -- 172.21.15.1:0/1763882257 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6c540544d0 con 0x7f6c6006f240 2024-03-18T19:08:23.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.353+0000 7f6c65b83700 1 -- 172.21.15.1:0/1763882257 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c600676a0 con 0x7f6c6006f240 2024-03-18T19:08:23.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.355+0000 7f6c47fff700 1 -- 172.21.15.1:0/1763882257 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v133) v1 ==== 74+0+16200 (secure 0 0 0) 0x7f6c5400ce60 con 0x7f6c6006f240 2024-03-18T19:08:23.351 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:23.351 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":133,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:08:20.583382+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:08:17.419247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-18T19:08:18.552871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:08:23.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.357+0000 7f6c65b83700 1 -- 172.21.15.1:0/1763882257 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c4c06adb0 msgr2=0x7f6c4c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:23.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.357+0000 7f6c65b83700 1 --2- 172.21.15.1:0/1763882257 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c4c06adb0 0x7f6c4c06d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6c600b0e70 tx=0x7f6c50009040 comp rx=0 tx=0).stop 2024-03-18T19:08:23.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.357+0000 7f6c65b83700 1 -- 172.21.15.1:0/1763882257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6006f240 msgr2=0x7f6c601ef5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:23.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.357+0000 7f6c65b83700 1 --2- 172.21.15.1:0/1763882257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6006f240 0x7f6c601ef5e0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f6c5400e870 tx=0x7f6c5400ec30 comp rx=0 tx=0).stop 2024-03-18T19:08:23.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.357+0000 7f6c65b83700 1 -- 172.21.15.1:0/1763882257 shutdown_connections 2024-03-18T19:08:23.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.357+0000 7f6c65b83700 1 --2- 172.21.15.1:0/1763882257 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c4c06adb0 0x7f6c4c06d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.357+0000 7f6c65b83700 1 --2- 172.21.15.1:0/1763882257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c60143c20 0x7f6c601ee1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.357+0000 7f6c65b83700 1 --2- 172.21.15.1:0/1763882257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c6006f240 0x7f6c601ef5e0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.357+0000 7f6c65b83700 1 --2- 172.21.15.1:0/1763882257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c600b3f40 0x7f6c601efd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.357+0000 7f6c65b83700 1 -- 172.21.15.1:0/1763882257 >> 172.21.15.1:0/1763882257 conn(0x7f6c600b1d20 msgr2=0x7f6c6006a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:23.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.357+0000 7f6c65b83700 1 -- 172.21.15.1:0/1763882257 shutdown_connections 2024-03-18T19:08:23.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.357+0000 7f6c65b83700 1 -- 172.21.15.1:0/1763882257 wait complete. 2024-03-18T19:08:23.367 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T19:08:23.367 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T19:08:23.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.497+0000 7fd8be03a700 1 Processor -- start 2024-03-18T19:08:23.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.497+0000 7fd8be03a700 1 -- start start 2024-03-18T19:08:23.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.498+0000 7fd8be03a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd8b815a2e0 0x7fd8b80ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.498+0000 7fd8be03a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8b80cc170 0x7fd8b806ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.498+0000 7fd8be03a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b806e270 con 0x7fd8b80ae600 2024-03-18T19:08:23.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.498+0000 7fd8be03a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b80ae050 con 0x7fd8b815a2e0 2024-03-18T19:08:23.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.498+0000 7fd8be03a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b80add70 con 0x7fd8b80cc170 2024-03-18T19:08:23.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.498+0000 7fd8b6ffd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd8b815a2e0 0x7fd8b80ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.498+0000 7fd8b7fff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8b80cc170 0x7fd8b806ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.498+0000 7fd8b6ffd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd8b815a2e0 0x7fd8b80ad790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:43944/0 (socket says 172.21.15.1:43944) 2024-03-18T19:08:23.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.498+0000 7fd8b77fe700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fd8b80ae600 0x7fd8b80ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:37902/0 (socket says 172.21.15.1:37902) 2024-03-18T19:08:23.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.498+0000 7fd8b6ffd700 1 -- 172.21.15.1:0/82932270 learned_addr learned my addr 172.21.15.1:0/82932270 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:23.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.499+0000 7fd8b67fc700 1 -- 172.21.15.1:0/82932270 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753987183 0 0) 0x7fd8b806e270 con 0x7fd8b80ae600 2024-03-18T19:08:23.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.499+0000 7fd8b7fff700 1 -- 172.21.15.1:0/82932270 >> v2:172.21.15.165:3300/0 conn(0x7fd8b815a2e0 msgr2=0x7fd8b80ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:23.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.499+0000 7fd8b7fff700 1 --2- 172.21.15.1:0/82932270 >> v2:172.21.15.165:3300/0 conn(0x7fd8b815a2e0 0x7fd8b80ad790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.499+0000 7fd8b7fff700 1 -- 172.21.15.1:0/82932270 >> v1:172.21.15.165:6789/0 conn(0x7fd8b80ae600 legacy=0x7fd8b80ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:23.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.499+0000 7fd8b7fff700 1 -- 172.21.15.1:0/82932270 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b806e4a0 con 0x7fd8b80cc170 2024-03-18T19:08:23.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.499+0000 7fd8b6ffd700 1 --2- 172.21.15.1:0/82932270 >> v2:172.21.15.165:3300/0 conn(0x7fd8b815a2e0 0x7fd8b80ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:08:23.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.499+0000 7fd8b7fff700 1 --2- 172.21.15.1:0/82932270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8b80cc170 0x7fd8b806ef80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009d90 tx=0x7fd8b000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38593214ee69f62b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:23.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.499+0000 7fd8b67fc700 1 -- 172.21.15.1:0/82932270 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b000e070 con 0x7fd8b80cc170 2024-03-18T19:08:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.499+0000 7fd8b67fc700 1 -- 172.21.15.1:0/82932270 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b000bb20 con 0x7fd8b80cc170 2024-03-18T19:08:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.499+0000 7fd8b67fc700 1 -- 172.21.15.1:0/82932270 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0012430 con 0x7fd8b80cc170 2024-03-18T19:08:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.500+0000 7fd8be03a700 1 -- 172.21.15.1:0/82932270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8b80cc170 msgr2=0x7fd8b806ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.500+0000 7fd8be03a700 1 --2- 172.21.15.1:0/82932270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8b80cc170 0x7fd8b806ef80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009d90 tx=0x7fd8b000b330 comp rx=0 tx=0).stop 2024-03-18T19:08:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.500+0000 7fd8be03a700 1 -- 172.21.15.1:0/82932270 shutdown_connections 2024-03-18T19:08:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.500+0000 7fd8be03a700 1 --2- 172.21.15.1:0/82932270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8b80cc170 0x7fd8b806ef80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.500+0000 7fd8be03a700 1 --2- 172.21.15.1:0/82932270 >> v2:172.21.15.165:3300/0 conn(0x7fd8b815a2e0 0x7fd8b80ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.500+0000 7fd8be03a700 1 -- 172.21.15.1:0/82932270 >> 172.21.15.1:0/82932270 conn(0x7fd8b806b860 msgr2=0x7fd8b806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.500+0000 7fd8be03a700 1 -- 172.21.15.1:0/82932270 shutdown_connections 2024-03-18T19:08:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.500+0000 7fd8be03a700 1 -- 172.21.15.1:0/82932270 wait complete. 2024-03-18T19:08:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.501+0000 7fd8be03a700 1 Processor -- start 2024-03-18T19:08:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.501+0000 7fd8be03a700 1 -- start start 2024-03-18T19:08:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.502+0000 7fd8be03a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8b80ae600 0x7fd8b8144010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.502+0000 7fd8be03a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8b80cc170 0x7fd8b80ceac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.502+0000 7fd8be03a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8b815a2e0 0x7fd8b8145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.502+0000 7fd8be03a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b80cf000 con 0x7fd8b815a2e0 2024-03-18T19:08:23.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.502+0000 7fd8be03a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b81446f0 con 0x7fd8b80ae600 2024-03-18T19:08:23.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.502+0000 7fd8be03a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b81449d0 con 0x7fd8b80cc170 2024-03-18T19:08:23.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.502+0000 7fd8b7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8b815a2e0 0x7fd8b8145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.502+0000 7fd8b77fe700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8b80ae600 0x7fd8b8144010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.502+0000 7fd8b7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8b815a2e0 0x7fd8b8145790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59412/0 (socket says 172.21.15.1:59412) 2024-03-18T19:08:23.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.502+0000 7fd8b7fff700 1 -- 172.21.15.1:0/3393622641 learned_addr learned my addr 172.21.15.1:0/3393622641 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:23.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.502+0000 7fd8b6ffd700 1 --2- 172.21.15.1:0/3393622641 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8b80cc170 0x7fd8b80ceac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.503+0000 7fd8b7fff700 1 -- 172.21.15.1:0/3393622641 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8b80cc170 msgr2=0x7fd8b80ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:23.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.503+0000 7fd8b7fff700 1 --2- 172.21.15.1:0/3393622641 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8b80cc170 0x7fd8b80ceac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.503+0000 7fd8b7fff700 1 -- 172.21.15.1:0/3393622641 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8b80ae600 msgr2=0x7fd8b8144010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:23.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.503+0000 7fd8b7fff700 1 --2- 172.21.15.1:0/3393622641 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8b80ae600 0x7fd8b8144010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.503+0000 7fd8b7fff700 1 -- 172.21.15.1:0/3393622641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a800b7f0 con 0x7fd8b815a2e0 2024-03-18T19:08:23.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.503+0000 7fd8b77fe700 1 --2- 172.21.15.1:0/3393622641 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8b80ae600 0x7fd8b8144010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:08:23.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.503+0000 7fd8b7fff700 1 --2- 172.21.15.1:0/3393622641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8b815a2e0 0x7fd8b8145790 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0014040 tx=0x7fd8b0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:23.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.503+0000 7fd8affff700 1 -- 172.21.15.1:0/3393622641 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0012430 con 0x7fd8b815a2e0 2024-03-18T19:08:23.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.503+0000 7fd8be03a700 1 -- 172.21.15.1:0/3393622641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b0009a70 con 0x7fd8b815a2e0 2024-03-18T19:08:23.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.503+0000 7fd8affff700 1 -- 172.21.15.1:0/3393622641 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b0020070 con 0x7fd8b815a2e0 2024-03-18T19:08:23.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.503+0000 7fd8affff700 1 -- 172.21.15.1:0/3393622641 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0004280 con 0x7fd8b815a2e0 2024-03-18T19:08:23.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.503+0000 7fd8be03a700 1 -- 172.21.15.1:0/3393622641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b8144f90 con 0x7fd8b815a2e0 2024-03-18T19:08:23.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.504+0000 7fd8be03a700 1 -- 172.21.15.1:0/3393622641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b80b3e00 con 0x7fd8b815a2e0 2024-03-18T19:08:23.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.504+0000 7fd8affff700 1 -- 172.21.15.1:0/3393622641 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd8b00043e0 con 0x7fd8b815a2e0 2024-03-18T19:08:23.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.505+0000 7fd8affff700 1 --2- 172.21.15.1:0/3393622641 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd89806adb0 0x7fd89806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.505+0000 7fd8affff700 1 -- 172.21.15.1:0/3393622641 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd8b0022070 con 0x7fd8b815a2e0 2024-03-18T19:08:23.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.505+0000 7fd8b77fe700 1 --2- 172.21.15.1:0/3393622641 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd89806adb0 0x7fd89806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.505+0000 7fd8b77fe700 1 --2- 172.21.15.1:0/3393622641 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd89806adb0 0x7fd89806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0007fd0 tx=0x7fd8a0007de0 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:23.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.509+0000 7fd8affff700 1 -- 172.21.15.1:0/3393622641 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd8b00561a0 con 0x7fd8b815a2e0 2024-03-18T19:08:23.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.711+0000 7fd8be03a700 1 -- 172.21.15.1:0/3393622641 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd8b8144b40 con 0x7fd89806adb0 2024-03-18T19:08:23.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.720+0000 7fd8affff700 1 -- 172.21.15.1:0/3393622641 <== mgr.8061 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290244 (secure 0 0 0) 0x7fd8b8144b40 con 0x7fd89806adb0 2024-03-18T19:08:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:23.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.723+0000 7fd8be03a700 1 -- 172.21.15.1:0/3393622641 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd89806adb0 msgr2=0x7fd89806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:23.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.723+0000 7fd8be03a700 1 --2- 172.21.15.1:0/3393622641 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd89806adb0 0x7fd89806d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0007fd0 tx=0x7fd8a0007de0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.723+0000 7fd8be03a700 1 -- 172.21.15.1:0/3393622641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8b815a2e0 msgr2=0x7fd8b8145790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.723+0000 7fd8be03a700 1 --2- 172.21.15.1:0/3393622641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8b815a2e0 0x7fd8b8145790 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0014040 tx=0x7fd8b0004ad0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.723+0000 7fd8be03a700 1 -- 172.21.15.1:0/3393622641 shutdown_connections 2024-03-18T19:08:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.723+0000 7fd8be03a700 1 --2- 172.21.15.1:0/3393622641 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd89806adb0 0x7fd89806d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.723+0000 7fd8be03a700 1 --2- 172.21.15.1:0/3393622641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8b815a2e0 0x7fd8b8145790 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.723+0000 7fd8be03a700 1 --2- 172.21.15.1:0/3393622641 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8b80ae600 0x7fd8b8144010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.723+0000 7fd8be03a700 1 --2- 172.21.15.1:0/3393622641 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8b80cc170 0x7fd8b80ceac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.723+0000 7fd8be03a700 1 -- 172.21.15.1:0/3393622641 >> 172.21.15.1:0/3393622641 conn(0x7fd8b806b860 msgr2=0x7fd8b80ad4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.723+0000 7fd8be03a700 1 -- 172.21.15.1:0/3393622641 shutdown_connections 2024-03-18T19:08:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.723+0000 7fd8be03a700 1 -- 172.21.15.1:0/3393622641 wait complete. 2024-03-18T19:08:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T19:08:23.732 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-18T19:08:22.042383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":363,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":345844,"kb_used_data":14028,"kb_used_omap":12,"kb_used_meta":331635,"kb_avail":754628876,"statfs":{"total":773094113280,"available":772739969024,"internally_reserved":0,"allocated":14364672,"data_stored":7770128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":339594809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000745"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536456+0000","last_change":"2024-03-18T19:08:19.601950+0000","last_active":"2024-03-18T19:08:21.536456+0000","last_peered":"2024-03-18T19:08:21.536456+0000","last_clean":"2024-03-18T19:08:21.536456+0000","last_became_active":"2024-03-18T19:08:19.601675+0000","last_became_peered":"2024-03-18T19:08:19.601675+0000","last_unstale":"2024-03-18T19:08:21.536456+0000","last_undegraded":"2024-03-18T19:08:21.536456+0000","last_fullsized":"2024-03-18T19:08:21.536456+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:57:06.127424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735398+0000","last_change":"2024-03-18T19:08:19.610511+0000","last_active":"2024-03-18T19:08:20.735398+0000","last_peered":"2024-03-18T19:08:20.735398+0000","last_clean":"2024-03-18T19:08:20.735398+0000","last_became_active":"2024-03-18T19:08:19.609922+0000","last_became_peered":"2024-03-18T19:08:19.609922+0000","last_unstale":"2024-03-18T19:08:20.735398+0000","last_undegraded":"2024-03-18T19:08:20.735398+0000","last_fullsized":"2024-03-18T19:08:20.735398+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:32.794300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536211+0000","last_change":"2024-03-18T19:08:19.607208+0000","last_active":"2024-03-18T19:08:21.536211+0000","last_peered":"2024-03-18T19:08:21.536211+0000","last_clean":"2024-03-18T19:08:21.536211+0000","last_became_active":"2024-03-18T19:08:19.606969+0000","last_became_peered":"2024-03-18T19:08:19.606969+0000","last_unstale":"2024-03-18T19:08:21.536211+0000","last_undegraded":"2024-03-18T19:08:21.536211+0000","last_fullsized":"2024-03-18T19:08:21.536211+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:54:19.583130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590473+0000","last_change":"2024-03-18T19:08:19.602888+0000","last_active":"2024-03-18T19:08:20.590473+0000","last_peered":"2024-03-18T19:08:20.590473+0000","last_clean":"2024-03-18T19:08:20.590473+0000","last_became_active":"2024-03-18T19:08:19.602487+0000","last_became_peered":"2024-03-18T19:08:19.602487+0000","last_unstale":"2024-03-18T19:08:20.590473+0000","last_undegraded":"2024-03-18T19:08:20.590473+0000","last_fullsized":"2024-03-18T19:08:20.590473+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:51:29.329926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589345+0000","last_change":"2024-03-18T19:08:19.600315+0000","last_active":"2024-03-18T19:08:20.589345+0000","last_peered":"2024-03-18T19:08:20.589345+0000","last_clean":"2024-03-18T19:08:20.589345+0000","last_became_active":"2024-03-18T19:08:19.600044+0000","last_became_peered":"2024-03-18T19:08:19.600044+0000","last_unstale":"2024-03-18T19:08:20.589345+0000","last_undegraded":"2024-03-18T19:08:20.589345+0000","last_fullsized":"2024-03-18T19:08:20.589345+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:51.825688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735115+0000","last_change":"2024-03-18T19:08:19.601463+0000","last_active":"2024-03-18T19:08:20.735115+0000","last_peered":"2024-03-18T19:08:20.735115+0000","last_clean":"2024-03-18T19:08:20.735115+0000","last_became_active":"2024-03-18T19:08:19.601177+0000","last_became_peered":"2024-03-18T19:08:19.601177+0000","last_unstale":"2024-03-18T19:08:20.735115+0000","last_undegraded":"2024-03-18T19:08:20.735115+0000","last_fullsized":"2024-03-18T19:08:20.735115+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:54.114622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.734965+0000","last_change":"2024-03-18T19:08:19.600528+0000","last_active":"2024-03-18T19:08:20.734965+0000","last_peered":"2024-03-18T19:08:20.734965+0000","last_clean":"2024-03-18T19:08:20.734965+0000","last_became_active":"2024-03-18T19:08:19.600263+0000","last_became_peered":"2024-03-18T19:08:19.600263+0000","last_unstale":"2024-03-18T19:08:20.734965+0000","last_undegraded":"2024-03-18T19:08:20.734965+0000","last_fullsized":"2024-03-18T19:08:20.734965+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:12:11.205891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.616276+0000","last_change":"2024-03-18T19:08:19.614620+0000","last_active":"2024-03-18T19:08:19.616276+0000","last_peered":"2024-03-18T19:08:19.616276+0000","last_clean":"2024-03-18T19:08:19.616276+0000","last_became_active":"2024-03-18T19:08:19.614436+0000","last_became_peered":"2024-03-18T19:08:19.614436+0000","last_unstale":"2024-03-18T19:08:19.616276+0000","last_undegraded":"2024-03-18T19:08:19.616276+0000","last_fullsized":"2024-03-18T19:08:19.616276+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:55:39.823009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735332+0000","last_change":"2024-03-18T19:08:19.603641+0000","last_active":"2024-03-18T19:08:20.735332+0000","last_peered":"2024-03-18T19:08:20.735332+0000","last_clean":"2024-03-18T19:08:20.735332+0000","last_became_active":"2024-03-18T19:08:19.603353+0000","last_became_peered":"2024-03-18T19:08:19.603353+0000","last_unstale":"2024-03-18T19:08:20.735332+0000","last_undegraded":"2024-03-18T19:08:20.735332+0000","last_fullsized":"2024-03-18T19:08:20.735332+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:46.115796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589965+0000","last_change":"2024-03-18T19:08:19.609437+0000","last_active":"2024-03-18T19:08:20.589965+0000","last_peered":"2024-03-18T19:08:20.589965+0000","last_clean":"2024-03-18T19:08:20.589965+0000","last_became_active":"2024-03-18T19:08:19.609162+0000","last_became_peered":"2024-03-18T19:08:19.609162+0000","last_unstale":"2024-03-18T19:08:20.589965+0000","last_undegraded":"2024-03-18T19:08:20.589965+0000","last_fullsized":"2024-03-18T19:08:20.589965+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:36:41.211897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.616737+0000","last_change":"2024-03-18T19:08:19.614990+0000","last_active":"2024-03-18T19:08:19.616737+0000","last_peered":"2024-03-18T19:08:19.616737+0000","last_clean":"2024-03-18T19:08:19.616737+0000","last_became_active":"2024-03-18T19:08:19.614660+0000","last_became_peered":"2024-03-18T19:08:19.614660+0000","last_unstale":"2024-03-18T19:08:19.616737+0000","last_undegraded":"2024-03-18T19:08:19.616737+0000","last_fullsized":"2024-03-18T19:08:19.616737+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:31.979650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.606684+0000","last_change":"2024-03-18T19:08:19.602172+0000","last_active":"2024-03-18T19:08:19.606684+0000","last_peered":"2024-03-18T19:08:19.606684+0000","last_clean":"2024-03-18T19:08:19.606684+0000","last_became_active":"2024-03-18T19:08:19.601913+0000","last_became_peered":"2024-03-18T19:08:19.601913+0000","last_unstale":"2024-03-18T19:08:19.606684+0000","last_undegraded":"2024-03-18T19:08:19.606684+0000","last_fullsized":"2024-03-18T19:08:19.606684+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:19:47.819745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590535+0000","last_change":"2024-03-18T19:08:19.603927+0000","last_active":"2024-03-18T19:08:20.590535+0000","last_peered":"2024-03-18T19:08:20.590535+0000","last_clean":"2024-03-18T19:08:20.590535+0000","last_became_active":"2024-03-18T19:08:19.603525+0000","last_became_peered":"2024-03-18T19:08:19.603525+0000","last_unstale":"2024-03-18T19:08:20.590535+0000","last_undegraded":"2024-03-18T19:08:20.590535+0000","last_fullsized":"2024-03-18T19:08:20.590535+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:13:04.646632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.597520+0000","last_change":"2024-03-18T19:08:19.596954+0000","last_active":"2024-03-18T19:08:19.597520+0000","last_peered":"2024-03-18T19:08:19.597520+0000","last_clean":"2024-03-18T19:08:19.597520+0000","last_became_active":"2024-03-18T19:08:19.595777+0000","last_became_peered":"2024-03-18T19:08:19.595777+0000","last_unstale":"2024-03-18T19:08:19.597520+0000","last_undegraded":"2024-03-18T19:08:19.597520+0000","last_fullsized":"2024-03-18T19:08:19.597520+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:23:14.545049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.605093+0000","last_change":"2024-03-18T19:08:19.603344+0000","last_active":"2024-03-18T19:08:19.605093+0000","last_peered":"2024-03-18T19:08:19.605093+0000","last_clean":"2024-03-18T19:08:19.605093+0000","last_became_active":"2024-03-18T19:08:19.603048+0000","last_became_peered":"2024-03-18T19:08:19.603048+0000","last_unstale":"2024-03-18T19:08:19.605093+0000","last_undegraded":"2024-03-18T19:08:19.605093+0000","last_fullsized":"2024-03-18T19:08:19.605093+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:36:41.211897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.616367+0000","last_change":"2024-03-18T19:08:19.615532+0000","last_active":"2024-03-18T19:08:19.616367+0000","last_peered":"2024-03-18T19:08:19.616367+0000","last_clean":"2024-03-18T19:08:19.616367+0000","last_became_active":"2024-03-18T19:08:19.615134+0000","last_became_peered":"2024-03-18T19:08:19.615134+0000","last_unstale":"2024-03-18T19:08:19.616367+0000","last_undegraded":"2024-03-18T19:08:19.616367+0000","last_fullsized":"2024-03-18T19:08:19.616367+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:46.115796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536455+0000","last_change":"2024-03-18T19:08:19.608311+0000","last_active":"2024-03-18T19:08:21.536455+0000","last_peered":"2024-03-18T19:08:21.536455+0000","last_clean":"2024-03-18T19:08:21.536455+0000","last_became_active":"2024-03-18T19:08:19.607879+0000","last_became_peered":"2024-03-18T19:08:19.607879+0000","last_unstale":"2024-03-18T19:08:21.536455+0000","last_undegraded":"2024-03-18T19:08:21.536455+0000","last_fullsized":"2024-03-18T19:08:21.536455+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:12:11.441836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735497+0000","last_change":"2024-03-18T19:08:19.611314+0000","last_active":"2024-03-18T19:08:20.735497+0000","last_peered":"2024-03-18T19:08:20.735497+0000","last_clean":"2024-03-18T19:08:20.735497+0000","last_became_active":"2024-03-18T19:08:19.611000+0000","last_became_peered":"2024-03-18T19:08:19.611000+0000","last_unstale":"2024-03-18T19:08:20.735497+0000","last_undegraded":"2024-03-18T19:08:20.735497+0000","last_fullsized":"2024-03-18T19:08:20.735497+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:23:14.545049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536358+0000","last_change":"2024-03-18T19:08:19.615396+0000","last_active":"2024-03-18T19:08:21.536358+0000","last_peered":"2024-03-18T19:08:21.536358+0000","last_clean":"2024-03-18T19:08:21.536358+0000","last_became_active":"2024-03-18T19:08:19.614729+0000","last_became_peered":"2024-03-18T19:08:19.614729+0000","last_unstale":"2024-03-18T19:08:21.536358+0000","last_undegraded":"2024-03-18T19:08:21.536358+0000","last_fullsized":"2024-03-18T19:08:21.536358+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:42.662793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.609840+0000","last_change":"2024-03-18T19:08:19.602688+0000","last_active":"2024-03-18T19:08:19.609840+0000","last_peered":"2024-03-18T19:08:19.609840+0000","last_clean":"2024-03-18T19:08:19.609840+0000","last_became_active":"2024-03-18T19:08:19.602401+0000","last_became_peered":"2024-03-18T19:08:19.602401+0000","last_unstale":"2024-03-18T19:08:19.609840+0000","last_undegraded":"2024-03-18T19:08:19.609840+0000","last_fullsized":"2024-03-18T19:08:19.609840+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:05:03.506261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.608563+0000","last_change":"2024-03-18T19:08:19.607660+0000","last_active":"2024-03-18T19:08:19.608563+0000","last_peered":"2024-03-18T19:08:19.608563+0000","last_clean":"2024-03-18T19:08:19.608563+0000","last_became_active":"2024-03-18T19:08:19.607371+0000","last_became_peered":"2024-03-18T19:08:19.607371+0000","last_unstale":"2024-03-18T19:08:19.608563+0000","last_undegraded":"2024-03-18T19:08:19.608563+0000","last_fullsized":"2024-03-18T19:08:19.608563+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:50:21.786428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589502+0000","last_change":"2024-03-18T19:08:19.609931+0000","last_active":"2024-03-18T19:08:20.589502+0000","last_peered":"2024-03-18T19:08:20.589502+0000","last_clean":"2024-03-18T19:08:20.589502+0000","last_became_active":"2024-03-18T19:08:19.608632+0000","last_became_peered":"2024-03-18T19:08:19.608632+0000","last_unstale":"2024-03-18T19:08:20.589502+0000","last_undegraded":"2024-03-18T19:08:20.589502+0000","last_fullsized":"2024-03-18T19:08:20.589502+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:46.115796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536337+0000","last_change":"2024-03-18T19:08:19.601572+0000","last_active":"2024-03-18T19:08:21.536337+0000","last_peered":"2024-03-18T19:08:21.536337+0000","last_clean":"2024-03-18T19:08:21.536337+0000","last_became_active":"2024-03-18T19:08:19.601326+0000","last_became_peered":"2024-03-18T19:08:19.601326+0000","last_unstale":"2024-03-18T19:08:21.536337+0000","last_undegraded":"2024-03-18T19:08:21.536337+0000","last_fullsized":"2024-03-18T19:08:21.536337+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:50.261113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589354+0000","last_change":"2024-03-18T19:08:19.599930+0000","last_active":"2024-03-18T19:08:20.589354+0000","last_peered":"2024-03-18T19:08:20.589354+0000","last_clean":"2024-03-18T19:08:20.589354+0000","last_became_active":"2024-03-18T19:08:19.599654+0000","last_became_peered":"2024-03-18T19:08:19.599654+0000","last_unstale":"2024-03-18T19:08:20.589354+0000","last_undegraded":"2024-03-18T19:08:20.589354+0000","last_fullsized":"2024-03-18T19:08:20.589354+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:31.979650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.610321+0000","last_change":"2024-03-18T19:08:19.607615+0000","last_active":"2024-03-18T19:08:19.610321+0000","last_peered":"2024-03-18T19:08:19.610321+0000","last_clean":"2024-03-18T19:08:19.610321+0000","last_became_active":"2024-03-18T19:08:19.607332+0000","last_became_peered":"2024-03-18T19:08:19.607332+0000","last_unstale":"2024-03-18T19:08:19.610321+0000","last_undegraded":"2024-03-18T19:08:19.610321+0000","last_fullsized":"2024-03-18T19:08:19.610321+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:36.075071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.613344+0000","last_change":"2024-03-18T19:08:19.610335+0000","last_active":"2024-03-18T19:08:19.613344+0000","last_peered":"2024-03-18T19:08:19.613344+0000","last_clean":"2024-03-18T19:08:19.613344+0000","last_became_active":"2024-03-18T19:08:19.610073+0000","last_became_peered":"2024-03-18T19:08:19.610073+0000","last_unstale":"2024-03-18T19:08:19.613344+0000","last_undegraded":"2024-03-18T19:08:19.613344+0000","last_fullsized":"2024-03-18T19:08:19.613344+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:11:13.131189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.609048+0000","last_change":"2024-03-18T19:08:19.605712+0000","last_active":"2024-03-18T19:08:19.609048+0000","last_peered":"2024-03-18T19:08:19.609048+0000","last_clean":"2024-03-18T19:08:19.609048+0000","last_became_active":"2024-03-18T19:08:19.605463+0000","last_became_peered":"2024-03-18T19:08:19.605463+0000","last_unstale":"2024-03-18T19:08:19.609048+0000","last_undegraded":"2024-03-18T19:08:19.609048+0000","last_fullsized":"2024-03-18T19:08:19.609048+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:23:14.545049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735318+0000","last_change":"2024-03-18T19:08:19.611955+0000","last_active":"2024-03-18T19:08:20.735318+0000","last_peered":"2024-03-18T19:08:20.735318+0000","last_clean":"2024-03-18T19:08:20.735318+0000","last_became_active":"2024-03-18T19:08:19.611686+0000","last_became_peered":"2024-03-18T19:08:19.611686+0000","last_unstale":"2024-03-18T19:08:20.735318+0000","last_undegraded":"2024-03-18T19:08:20.735318+0000","last_fullsized":"2024-03-18T19:08:20.735318+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:39:17.365258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.608526+0000","last_change":"2024-03-18T19:08:19.606635+0000","last_active":"2024-03-18T19:08:19.608526+0000","last_peered":"2024-03-18T19:08:19.608526+0000","last_clean":"2024-03-18T19:08:19.608526+0000","last_became_active":"2024-03-18T19:08:19.606387+0000","last_became_peered":"2024-03-18T19:08:19.606387+0000","last_unstale":"2024-03-18T19:08:19.608526+0000","last_undegraded":"2024-03-18T19:08:19.608526+0000","last_fullsized":"2024-03-18T19:08:19.608526+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:47:16.302386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.610496+0000","last_change":"2024-03-18T19:08:19.607687+0000","last_active":"2024-03-18T19:08:19.610496+0000","last_peered":"2024-03-18T19:08:19.610496+0000","last_clean":"2024-03-18T19:08:19.610496+0000","last_became_active":"2024-03-18T19:08:19.607375+0000","last_became_peered":"2024-03-18T19:08:19.607375+0000","last_unstale":"2024-03-18T19:08:19.610496+0000","last_undegraded":"2024-03-18T19:08:19.610496+0000","last_fullsized":"2024-03-18T19:08:19.610496+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:17.177831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735125+0000","last_change":"2024-03-18T19:08:19.606174+0000","last_active":"2024-03-18T19:08:20.735125+0000","last_peered":"2024-03-18T19:08:20.735125+0000","last_clean":"2024-03-18T19:08:20.735125+0000","last_became_active":"2024-03-18T19:08:19.605924+0000","last_became_peered":"2024-03-18T19:08:19.605924+0000","last_unstale":"2024-03-18T19:08:20.735125+0000","last_undegraded":"2024-03-18T19:08:20.735125+0000","last_fullsized":"2024-03-18T19:08:20.735125+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:49:04.996335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.617428+0000","last_change":"2024-03-18T19:08:19.616439+0000","last_active":"2024-03-18T19:08:19.617428+0000","last_peered":"2024-03-18T19:08:19.617428+0000","last_clean":"2024-03-18T19:08:19.617428+0000","last_became_active":"2024-03-18T19:08:19.615997+0000","last_became_peered":"2024-03-18T19:08:19.615997+0000","last_unstale":"2024-03-18T19:08:19.617428+0000","last_undegraded":"2024-03-18T19:08:19.617428+0000","last_fullsized":"2024-03-18T19:08:19.617428+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:58:13.838042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536235+0000","last_change":"2024-03-18T19:08:19.607013+0000","last_active":"2024-03-18T19:08:21.536235+0000","last_peered":"2024-03-18T19:08:21.536235+0000","last_clean":"2024-03-18T19:08:21.536235+0000","last_became_active":"2024-03-18T19:08:19.606591+0000","last_became_peered":"2024-03-18T19:08:19.606591+0000","last_unstale":"2024-03-18T19:08:21.536235+0000","last_undegraded":"2024-03-18T19:08:21.536235+0000","last_fullsized":"2024-03-18T19:08:21.536235+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:41.496312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"132'4","reported_seq":15,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590608+0000","last_change":"2024-03-18T19:08:19.165661+0000","last_active":"2024-03-18T19:08:20.590608+0000","last_peered":"2024-03-18T19:08:20.590608+0000","last_clean":"2024-03-18T19:08:20.590608+0000","last_became_active":"2024-03-18T19:08:19.165328+0000","last_became_peered":"2024-03-18T19:08:19.165328+0000","last_unstale":"2024-03-18T19:08:20.590608+0000","last_undegraded":"2024-03-18T19:08:20.590608+0000","last_fullsized":"2024-03-18T19:08:20.590608+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:35.069998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.612335+0000","last_change":"2024-03-18T19:08:19.610079+0000","last_active":"2024-03-18T19:08:19.612335+0000","last_peered":"2024-03-18T19:08:19.612335+0000","last_clean":"2024-03-18T19:08:19.612335+0000","last_became_active":"2024-03-18T19:08:19.609716+0000","last_became_peered":"2024-03-18T19:08:19.609716+0000","last_unstale":"2024-03-18T19:08:19.612335+0000","last_undegraded":"2024-03-18T19:08:19.612335+0000","last_fullsized":"2024-03-18T19:08:19.612335+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:13:50.766336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"132'1","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.616040+0000","last_change":"2024-03-18T19:08:19.168730+0000","last_active":"2024-03-18T19:08:19.616040+0000","last_peered":"2024-03-18T19:08:19.616040+0000","last_clean":"2024-03-18T19:08:19.616040+0000","last_became_active":"2024-03-18T19:08:19.168221+0000","last_became_peered":"2024-03-18T19:08:19.168221+0000","last_unstale":"2024-03-18T19:08:19.616040+0000","last_undegraded":"2024-03-18T19:08:19.616040+0000","last_fullsized":"2024-03-18T19:08:19.616040+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:30.492658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.609794+0000","last_change":"2024-03-18T19:08:19.603336+0000","last_active":"2024-03-18T19:08:19.609794+0000","last_peered":"2024-03-18T19:08:19.609794+0000","last_clean":"2024-03-18T19:08:19.609794+0000","last_became_active":"2024-03-18T19:08:19.598930+0000","last_became_peered":"2024-03-18T19:08:19.598930+0000","last_unstale":"2024-03-18T19:08:19.609794+0000","last_undegraded":"2024-03-18T19:08:19.609794+0000","last_fullsized":"2024-03-18T19:08:19.609794+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:52:38.917172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"132'2","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.608103+0000","last_change":"2024-03-18T19:08:19.168965+0000","last_active":"2024-03-18T19:08:19.608103+0000","last_peered":"2024-03-18T19:08:19.608103+0000","last_clean":"2024-03-18T19:08:19.608103+0000","last_became_active":"2024-03-18T19:08:19.168573+0000","last_became_peered":"2024-03-18T19:08:19.168573+0000","last_unstale":"2024-03-18T19:08:19.608103+0000","last_undegraded":"2024-03-18T19:08:19.608103+0000","last_fullsized":"2024-03-18T19:08:19.608103+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:23:13.539977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.610829+0000","last_change":"2024-03-18T19:08:19.610336+0000","last_active":"2024-03-18T19:08:19.610829+0000","last_peered":"2024-03-18T19:08:19.610829+0000","last_clean":"2024-03-18T19:08:19.610829+0000","last_became_active":"2024-03-18T19:08:19.610077+0000","last_became_peered":"2024-03-18T19:08:19.610077+0000","last_unstale":"2024-03-18T19:08:19.610829+0000","last_undegraded":"2024-03-18T19:08:19.610829+0000","last_fullsized":"2024-03-18T19:08:19.610829+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:54.114622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"132'2","reported_seq":13,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589777+0000","last_change":"2024-03-18T19:08:19.166836+0000","last_active":"2024-03-18T19:08:20.589777+0000","last_peered":"2024-03-18T19:08:20.589777+0000","last_clean":"2024-03-18T19:08:20.589777+0000","last_became_active":"2024-03-18T19:08:19.166571+0000","last_became_peered":"2024-03-18T19:08:19.166571+0000","last_unstale":"2024-03-18T19:08:20.589777+0000","last_undegraded":"2024-03-18T19:08:20.589777+0000","last_fullsized":"2024-03-18T19:08:20.589777+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:04:25.186213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590568+0000","last_change":"2024-03-18T19:08:19.596579+0000","last_active":"2024-03-18T19:08:20.590568+0000","last_peered":"2024-03-18T19:08:20.590568+0000","last_clean":"2024-03-18T19:08:20.590568+0000","last_became_active":"2024-03-18T19:08:19.596283+0000","last_became_peered":"2024-03-18T19:08:19.596283+0000","last_unstale":"2024-03-18T19:08:20.590568+0000","last_undegraded":"2024-03-18T19:08:20.590568+0000","last_fullsized":"2024-03-18T19:08:20.590568+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:11:00.314452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"132'1","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.605733+0000","last_change":"2024-03-18T19:08:19.160250+0000","last_active":"2024-03-18T19:08:19.605733+0000","last_peered":"2024-03-18T19:08:19.605733+0000","last_clean":"2024-03-18T19:08:19.605733+0000","last_became_active":"2024-03-18T19:08:19.159902+0000","last_became_peered":"2024-03-18T19:08:19.159902+0000","last_unstale":"2024-03-18T19:08:19.605733+0000","last_undegraded":"2024-03-18T19:08:19.605733+0000","last_fullsized":"2024-03-18T19:08:19.605733+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:59:31.879611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536293+0000","last_change":"2024-03-18T19:08:19.615764+0000","last_active":"2024-03-18T19:08:21.536293+0000","last_peered":"2024-03-18T19:08:21.536293+0000","last_clean":"2024-03-18T19:08:21.536293+0000","last_became_active":"2024-03-18T19:08:19.615081+0000","last_became_peered":"2024-03-18T19:08:19.615081+0000","last_unstale":"2024-03-18T19:08:21.536293+0000","last_undegraded":"2024-03-18T19:08:21.536293+0000","last_fullsized":"2024-03-18T19:08:21.536293+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:39:02.694519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"132'1","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.609994+0000","last_change":"2024-03-18T19:08:19.165945+0000","last_active":"2024-03-18T19:08:19.609994+0000","last_peered":"2024-03-18T19:08:19.609994+0000","last_clean":"2024-03-18T19:08:19.609994+0000","last_became_active":"2024-03-18T19:08:19.165670+0000","last_became_peered":"2024-03-18T19:08:19.165670+0000","last_unstale":"2024-03-18T19:08:19.609994+0000","last_undegraded":"2024-03-18T19:08:19.609994+0000","last_fullsized":"2024-03-18T19:08:19.609994+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:20.352711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735661+0000","last_change":"2024-03-18T19:08:19.610692+0000","last_active":"2024-03-18T19:08:20.735661+0000","last_peered":"2024-03-18T19:08:20.735661+0000","last_clean":"2024-03-18T19:08:20.735661+0000","last_became_active":"2024-03-18T19:08:19.610265+0000","last_became_peered":"2024-03-18T19:08:19.610265+0000","last_unstale":"2024-03-18T19:08:20.735661+0000","last_undegraded":"2024-03-18T19:08:20.735661+0000","last_fullsized":"2024-03-18T19:08:20.735661+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:06.122862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.591816+0000","last_change":"2024-03-18T19:08:19.164485+0000","last_active":"2024-03-18T19:08:19.591816+0000","last_peered":"2024-03-18T19:08:19.591816+0000","last_clean":"2024-03-18T19:08:19.591816+0000","last_became_active":"2024-03-18T19:08:19.164217+0000","last_became_peered":"2024-03-18T19:08:19.164217+0000","last_unstale":"2024-03-18T19:08:19.591816+0000","last_undegraded":"2024-03-18T19:08:19.591816+0000","last_fullsized":"2024-03-18T19:08:19.591816+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:31.789228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590632+0000","last_change":"2024-03-18T19:08:19.602044+0000","last_active":"2024-03-18T19:08:20.590632+0000","last_peered":"2024-03-18T19:08:20.590632+0000","last_clean":"2024-03-18T19:08:20.590632+0000","last_became_active":"2024-03-18T19:08:19.601277+0000","last_became_peered":"2024-03-18T19:08:19.601277+0000","last_unstale":"2024-03-18T19:08:20.590632+0000","last_undegraded":"2024-03-18T19:08:20.590632+0000","last_fullsized":"2024-03-18T19:08:20.590632+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:23:07.262496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590305+0000","last_change":"2024-03-18T19:08:19.170497+0000","last_active":"2024-03-18T19:08:20.590305+0000","last_peered":"2024-03-18T19:08:20.590305+0000","last_clean":"2024-03-18T19:08:20.590305+0000","last_became_active":"2024-03-18T19:08:19.170284+0000","last_became_peered":"2024-03-18T19:08:19.170284+0000","last_unstale":"2024-03-18T19:08:20.590305+0000","last_undegraded":"2024-03-18T19:08:20.590305+0000","last_fullsized":"2024-03-18T19:08:20.590305+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:57.194723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536144+0000","last_change":"2024-03-18T19:08:19.606838+0000","last_active":"2024-03-18T19:08:21.536144+0000","last_peered":"2024-03-18T19:08:21.536144+0000","last_clean":"2024-03-18T19:08:21.536144+0000","last_became_active":"2024-03-18T19:08:19.606257+0000","last_became_peered":"2024-03-18T19:08:19.606257+0000","last_unstale":"2024-03-18T19:08:21.536144+0000","last_undegraded":"2024-03-18T19:08:21.536144+0000","last_fullsized":"2024-03-18T19:08:21.536144+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:40:11.381260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.591378+0000","last_change":"2024-03-18T19:08:19.168791+0000","last_active":"2024-03-18T19:08:19.591378+0000","last_peered":"2024-03-18T19:08:19.591378+0000","last_clean":"2024-03-18T19:08:19.591378+0000","last_became_active":"2024-03-18T19:08:19.168522+0000","last_became_peered":"2024-03-18T19:08:19.168522+0000","last_unstale":"2024-03-18T19:08:19.591378+0000","last_undegraded":"2024-03-18T19:08:19.591378+0000","last_fullsized":"2024-03-18T19:08:19.591378+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:36:40.206825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589461+0000","last_change":"2024-03-18T19:08:19.605621+0000","last_active":"2024-03-18T19:08:20.589461+0000","last_peered":"2024-03-18T19:08:20.589461+0000","last_clean":"2024-03-18T19:08:20.589461+0000","last_became_active":"2024-03-18T19:08:19.605185+0000","last_became_peered":"2024-03-18T19:08:19.605185+0000","last_unstale":"2024-03-18T19:08:20.589461+0000","last_undegraded":"2024-03-18T19:08:20.589461+0000","last_fullsized":"2024-03-18T19:08:20.589461+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:23:14.545049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536146+0000","last_change":"2024-03-18T19:08:19.169385+0000","last_active":"2024-03-18T19:08:21.536146+0000","last_peered":"2024-03-18T19:08:21.536146+0000","last_clean":"2024-03-18T19:08:21.536146+0000","last_became_active":"2024-03-18T19:08:19.169106+0000","last_became_peered":"2024-03-18T19:08:19.169106+0000","last_unstale":"2024-03-18T19:08:21.536146+0000","last_undegraded":"2024-03-18T19:08:21.536146+0000","last_fullsized":"2024-03-18T19:08:21.536146+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:10:26.147698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.608175+0000","last_change":"2024-03-18T19:08:19.605584+0000","last_active":"2024-03-18T19:08:19.608175+0000","last_peered":"2024-03-18T19:08:19.608175+0000","last_clean":"2024-03-18T19:08:19.608175+0000","last_became_active":"2024-03-18T19:08:19.605356+0000","last_became_peered":"2024-03-18T19:08:19.605356+0000","last_unstale":"2024-03-18T19:08:19.608175+0000","last_undegraded":"2024-03-18T19:08:19.608175+0000","last_fullsized":"2024-03-18T19:08:19.608175+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:40:40.883829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"132'3","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.604670+0000","last_change":"2024-03-18T19:08:19.166809+0000","last_active":"2024-03-18T19:08:19.604670+0000","last_peered":"2024-03-18T19:08:19.604670+0000","last_clean":"2024-03-18T19:08:19.604670+0000","last_became_active":"2024-03-18T19:08:19.166512+0000","last_became_peered":"2024-03-18T19:08:19.166512+0000","last_unstale":"2024-03-18T19:08:19.604670+0000","last_undegraded":"2024-03-18T19:08:19.604670+0000","last_fullsized":"2024-03-18T19:08:19.604670+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:33.982221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589515+0000","last_change":"2024-03-18T19:08:19.612111+0000","last_active":"2024-03-18T19:08:20.589515+0000","last_peered":"2024-03-18T19:08:20.589515+0000","last_clean":"2024-03-18T19:08:20.589515+0000","last_became_active":"2024-03-18T19:08:19.611853+0000","last_became_peered":"2024-03-18T19:08:19.611853+0000","last_unstale":"2024-03-18T19:08:20.589515+0000","last_undegraded":"2024-03-18T19:08:20.589515+0000","last_fullsized":"2024-03-18T19:08:20.589515+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:49:04.996335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"132'6","reported_seq":17,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735406+0000","last_change":"2024-03-18T19:08:19.164339+0000","last_active":"2024-03-18T19:08:20.735406+0000","last_peered":"2024-03-18T19:08:20.735406+0000","last_clean":"2024-03-18T19:08:20.735406+0000","last_became_active":"2024-03-18T19:08:19.164062+0000","last_became_peered":"2024-03-18T19:08:19.164062+0000","last_unstale":"2024-03-18T19:08:20.735406+0000","last_undegraded":"2024-03-18T19:08:20.735406+0000","last_fullsized":"2024-03-18T19:08:20.735406+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:47:15.297314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.617049+0000","last_change":"2024-03-18T19:08:19.616275+0000","last_active":"2024-03-18T19:08:19.617049+0000","last_peered":"2024-03-18T19:08:19.617049+0000","last_clean":"2024-03-18T19:08:19.617049+0000","last_became_active":"2024-03-18T19:08:19.616022+0000","last_became_peered":"2024-03-18T19:08:19.616022+0000","last_unstale":"2024-03-18T19:08:19.617049+0000","last_undegraded":"2024-03-18T19:08:19.617049+0000","last_fullsized":"2024-03-18T19:08:19.617049+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:11.705994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"132'2","reported_seq":13,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589684+0000","last_change":"2024-03-18T19:08:19.165654+0000","last_active":"2024-03-18T19:08:20.589684+0000","last_peered":"2024-03-18T19:08:20.589684+0000","last_clean":"2024-03-18T19:08:20.589684+0000","last_became_active":"2024-03-18T19:08:19.165394+0000","last_became_peered":"2024-03-18T19:08:19.165394+0000","last_unstale":"2024-03-18T19:08:20.589684+0000","last_undegraded":"2024-03-18T19:08:20.589684+0000","last_fullsized":"2024-03-18T19:08:20.589684+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:33.982221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735247+0000","last_change":"2024-03-18T19:08:19.603045+0000","last_active":"2024-03-18T19:08:20.735247+0000","last_peered":"2024-03-18T19:08:20.735247+0000","last_clean":"2024-03-18T19:08:20.735247+0000","last_became_active":"2024-03-18T19:08:19.602799+0000","last_became_peered":"2024-03-18T19:08:19.602799+0000","last_unstale":"2024-03-18T19:08:20.735247+0000","last_undegraded":"2024-03-18T19:08:20.735247+0000","last_fullsized":"2024-03-18T19:08:20.735247+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:51.825688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590559+0000","last_change":"2024-03-18T19:08:19.169402+0000","last_active":"2024-03-18T19:08:20.590559+0000","last_peered":"2024-03-18T19:08:20.590559+0000","last_clean":"2024-03-18T19:08:20.590559+0000","last_became_active":"2024-03-18T19:08:19.169108+0000","last_became_peered":"2024-03-18T19:08:19.169108+0000","last_unstale":"2024-03-18T19:08:20.590559+0000","last_undegraded":"2024-03-18T19:08:20.590559+0000","last_fullsized":"2024-03-18T19:08:20.590559+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:19:46.814673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.612135+0000","last_change":"2024-03-18T19:08:19.611243+0000","last_active":"2024-03-18T19:08:19.612135+0000","last_peered":"2024-03-18T19:08:19.612135+0000","last_clean":"2024-03-18T19:08:19.612135+0000","last_became_active":"2024-03-18T19:08:19.611001+0000","last_became_peered":"2024-03-18T19:08:19.611001+0000","last_unstale":"2024-03-18T19:08:19.612135+0000","last_undegraded":"2024-03-18T19:08:19.612135+0000","last_fullsized":"2024-03-18T19:08:19.612135+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:16:53.833044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.601446+0000","last_change":"2024-03-18T19:08:19.170909+0000","last_active":"2024-03-18T19:08:19.601446+0000","last_peered":"2024-03-18T19:08:19.601446+0000","last_clean":"2024-03-18T19:08:19.601446+0000","last_became_active":"2024-03-18T19:08:19.168675+0000","last_became_peered":"2024-03-18T19:08:19.168675+0000","last_unstale":"2024-03-18T19:08:19.601446+0000","last_undegraded":"2024-03-18T19:08:19.601446+0000","last_fullsized":"2024-03-18T19:08:19.601446+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:18.748842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536546+0000","last_change":"2024-03-18T19:08:19.608133+0000","last_active":"2024-03-18T19:08:21.536546+0000","last_peered":"2024-03-18T19:08:21.536546+0000","last_clean":"2024-03-18T19:08:21.536546+0000","last_became_active":"2024-03-18T19:08:19.607531+0000","last_became_peered":"2024-03-18T19:08:19.607531+0000","last_unstale":"2024-03-18T19:08:21.536546+0000","last_undegraded":"2024-03-18T19:08:21.536546+0000","last_fullsized":"2024-03-18T19:08:21.536546+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:44:49.610735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"132'8","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.608768+0000","last_change":"2024-03-18T19:08:19.165160+0000","last_active":"2024-03-18T19:08:19.608768+0000","last_peered":"2024-03-18T19:08:19.608768+0000","last_clean":"2024-03-18T19:08:19.608768+0000","last_became_active":"2024-03-18T19:08:19.164883+0000","last_became_peered":"2024-03-18T19:08:19.164883+0000","last_unstale":"2024-03-18T19:08:19.608768+0000","last_undegraded":"2024-03-18T19:08:19.608768+0000","last_fullsized":"2024-03-18T19:08:19.608768+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:45.110724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.616173+0000","last_change":"2024-03-18T19:08:19.608700+0000","last_active":"2024-03-18T19:08:19.616173+0000","last_peered":"2024-03-18T19:08:19.616173+0000","last_clean":"2024-03-18T19:08:19.616173+0000","last_became_active":"2024-03-18T19:08:19.608373+0000","last_became_peered":"2024-03-18T19:08:19.608373+0000","last_unstale":"2024-03-18T19:08:19.616173+0000","last_undegraded":"2024-03-18T19:08:19.616173+0000","last_fullsized":"2024-03-18T19:08:19.616173+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:29:17.927761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"132'1","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735398+0000","last_change":"2024-03-18T19:08:19.168227+0000","last_active":"2024-03-18T19:08:20.735398+0000","last_peered":"2024-03-18T19:08:20.735398+0000","last_clean":"2024-03-18T19:08:20.735398+0000","last_became_active":"2024-03-18T19:08:19.167990+0000","last_became_peered":"2024-03-18T19:08:19.167990+0000","last_unstale":"2024-03-18T19:08:20.735398+0000","last_undegraded":"2024-03-18T19:08:20.735398+0000","last_fullsized":"2024-03-18T19:08:20.735398+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:16:52.827972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735232+0000","last_change":"2024-03-18T19:08:19.605813+0000","last_active":"2024-03-18T19:08:20.735232+0000","last_peered":"2024-03-18T19:08:20.735232+0000","last_clean":"2024-03-18T19:08:20.735232+0000","last_became_active":"2024-03-18T19:08:19.605536+0000","last_became_peered":"2024-03-18T19:08:19.605536+0000","last_unstale":"2024-03-18T19:08:20.735232+0000","last_undegraded":"2024-03-18T19:08:20.735232+0000","last_fullsized":"2024-03-18T19:08:20.735232+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:31.979650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"132'1","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.607027+0000","last_change":"2024-03-18T19:08:19.168439+0000","last_active":"2024-03-18T19:08:19.607027+0000","last_peered":"2024-03-18T19:08:19.607027+0000","last_clean":"2024-03-18T19:08:19.607027+0000","last_became_active":"2024-03-18T19:08:19.168119+0000","last_became_peered":"2024-03-18T19:08:19.168119+0000","last_unstale":"2024-03-18T19:08:19.607027+0000","last_undegraded":"2024-03-18T19:08:19.607027+0000","last_fullsized":"2024-03-18T19:08:19.607027+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:57.747186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589492+0000","last_change":"2024-03-18T19:08:19.594726+0000","last_active":"2024-03-18T19:08:20.589492+0000","last_peered":"2024-03-18T19:08:20.589492+0000","last_clean":"2024-03-18T19:08:20.589492+0000","last_became_active":"2024-03-18T19:08:19.594469+0000","last_became_peered":"2024-03-18T19:08:19.594469+0000","last_unstale":"2024-03-18T19:08:20.589492+0000","last_undegraded":"2024-03-18T19:08:20.589492+0000","last_fullsized":"2024-03-18T19:08:20.589492+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:54.114622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.596127+0000","last_change":"2024-03-18T19:08:19.164565+0000","last_active":"2024-03-18T19:08:19.596127+0000","last_peered":"2024-03-18T19:08:19.596127+0000","last_clean":"2024-03-18T19:08:19.596127+0000","last_became_active":"2024-03-18T19:08:19.164307+0000","last_became_peered":"2024-03-18T19:08:19.164307+0000","last_unstale":"2024-03-18T19:08:19.596127+0000","last_undegraded":"2024-03-18T19:08:19.596127+0000","last_fullsized":"2024-03-18T19:08:19.596127+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:57.747186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536236+0000","last_change":"2024-03-18T19:08:19.603564+0000","last_active":"2024-03-18T19:08:21.536236+0000","last_peered":"2024-03-18T19:08:21.536236+0000","last_clean":"2024-03-18T19:08:21.536236+0000","last_became_active":"2024-03-18T19:08:19.603281+0000","last_became_peered":"2024-03-18T19:08:19.603281+0000","last_unstale":"2024-03-18T19:08:21.536236+0000","last_undegraded":"2024-03-18T19:08:21.536236+0000","last_fullsized":"2024-03-18T19:08:21.536236+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:17.394276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"132'2","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.618024+0000","last_change":"2024-03-18T19:08:19.171553+0000","last_active":"2024-03-18T19:08:19.618024+0000","last_peered":"2024-03-18T19:08:19.618024+0000","last_clean":"2024-03-18T19:08:19.618024+0000","last_became_active":"2024-03-18T19:08:19.168493+0000","last_became_peered":"2024-03-18T19:08:19.168493+0000","last_unstale":"2024-03-18T19:08:19.618024+0000","last_undegraded":"2024-03-18T19:08:19.618024+0000","last_fullsized":"2024-03-18T19:08:19.618024+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:53.557992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536435+0000","last_change":"2024-03-18T19:08:19.601793+0000","last_active":"2024-03-18T19:08:21.536435+0000","last_peered":"2024-03-18T19:08:21.536435+0000","last_clean":"2024-03-18T19:08:21.536435+0000","last_became_active":"2024-03-18T19:08:19.601489+0000","last_became_peered":"2024-03-18T19:08:19.601489+0000","last_unstale":"2024-03-18T19:08:21.536435+0000","last_undegraded":"2024-03-18T19:08:21.536435+0000","last_fullsized":"2024-03-18T19:08:21.536435+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:09:30.271851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.601566+0000","last_change":"2024-03-18T19:08:19.171812+0000","last_active":"2024-03-18T19:08:19.601566+0000","last_peered":"2024-03-18T19:08:19.601566+0000","last_clean":"2024-03-18T19:08:19.601566+0000","last_became_active":"2024-03-18T19:08:19.171520+0000","last_became_peered":"2024-03-18T19:08:19.171520+0000","last_unstale":"2024-03-18T19:08:19.601566+0000","last_undegraded":"2024-03-18T19:08:19.601566+0000","last_fullsized":"2024-03-18T19:08:19.601566+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:03:24.832833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590711+0000","last_change":"2024-03-18T19:08:19.615504+0000","last_active":"2024-03-18T19:08:20.590711+0000","last_peered":"2024-03-18T19:08:20.590711+0000","last_clean":"2024-03-18T19:08:20.590711+0000","last_became_active":"2024-03-18T19:08:19.615260+0000","last_became_peered":"2024-03-18T19:08:19.615260+0000","last_unstale":"2024-03-18T19:08:20.590711+0000","last_undegraded":"2024-03-18T19:08:20.590711+0000","last_fullsized":"2024-03-18T19:08:20.590711+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:21.361005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.734959+0000","last_change":"2024-03-18T19:08:19.163683+0000","last_active":"2024-03-18T19:08:20.734959+0000","last_peered":"2024-03-18T19:08:20.734959+0000","last_clean":"2024-03-18T19:08:20.734959+0000","last_became_active":"2024-03-18T19:08:19.163414+0000","last_became_peered":"2024-03-18T19:08:19.163414+0000","last_unstale":"2024-03-18T19:08:20.734959+0000","last_undegraded":"2024-03-18T19:08:20.734959+0000","last_fullsized":"2024-03-18T19:08:20.734959+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:58:12.832970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.608204+0000","last_change":"2024-03-18T19:08:19.605932+0000","last_active":"2024-03-18T19:08:19.608204+0000","last_peered":"2024-03-18T19:08:19.608204+0000","last_clean":"2024-03-18T19:08:19.608204+0000","last_became_active":"2024-03-18T19:08:19.602651+0000","last_became_peered":"2024-03-18T19:08:19.602651+0000","last_unstale":"2024-03-18T19:08:19.608204+0000","last_undegraded":"2024-03-18T19:08:19.608204+0000","last_fullsized":"2024-03-18T19:08:19.608204+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:07:58.870811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.601302+0000","last_change":"2024-03-18T19:08:19.171795+0000","last_active":"2024-03-18T19:08:19.601302+0000","last_peered":"2024-03-18T19:08:19.601302+0000","last_clean":"2024-03-18T19:08:19.601302+0000","last_became_active":"2024-03-18T19:08:19.171464+0000","last_became_peered":"2024-03-18T19:08:19.171464+0000","last_unstale":"2024-03-18T19:08:19.601302+0000","last_undegraded":"2024-03-18T19:08:19.601302+0000","last_fullsized":"2024-03-18T19:08:19.601302+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:54:22.449223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536086+0000","last_change":"2024-03-18T19:08:19.600863+0000","last_active":"2024-03-18T19:08:21.536086+0000","last_peered":"2024-03-18T19:08:21.536086+0000","last_clean":"2024-03-18T19:08:21.536086+0000","last_became_active":"2024-03-18T19:08:19.600608+0000","last_became_peered":"2024-03-18T19:08:19.600608+0000","last_unstale":"2024-03-18T19:08:21.536086+0000","last_undegraded":"2024-03-18T19:08:21.536086+0000","last_fullsized":"2024-03-18T19:08:21.536086+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:59.796154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"132'2","reported_seq":13,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590032+0000","last_change":"2024-03-18T19:08:19.165314+0000","last_active":"2024-03-18T19:08:20.590032+0000","last_peered":"2024-03-18T19:08:20.590032+0000","last_clean":"2024-03-18T19:08:20.590032+0000","last_became_active":"2024-03-18T19:08:19.165036+0000","last_became_peered":"2024-03-18T19:08:19.165036+0000","last_unstale":"2024-03-18T19:08:20.590032+0000","last_undegraded":"2024-03-18T19:08:20.590032+0000","last_fullsized":"2024-03-18T19:08:20.590032+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:57.137780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.603953+0000","last_change":"2024-03-18T19:08:19.600746+0000","last_active":"2024-03-18T19:08:19.603953+0000","last_peered":"2024-03-18T19:08:19.603953+0000","last_clean":"2024-03-18T19:08:19.603953+0000","last_became_active":"2024-03-18T19:08:19.600483+0000","last_became_peered":"2024-03-18T19:08:19.600483+0000","last_unstale":"2024-03-18T19:08:19.603953+0000","last_undegraded":"2024-03-18T19:08:19.603953+0000","last_fullsized":"2024-03-18T19:08:19.603953+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:58.199795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536300+0000","last_change":"2024-03-18T19:08:19.164044+0000","last_active":"2024-03-18T19:08:21.536300+0000","last_peered":"2024-03-18T19:08:21.536300+0000","last_clean":"2024-03-18T19:08:21.536300+0000","last_became_active":"2024-03-18T19:08:19.163786+0000","last_became_peered":"2024-03-18T19:08:19.163786+0000","last_unstale":"2024-03-18T19:08:21.536300+0000","last_undegraded":"2024-03-18T19:08:21.536300+0000","last_fullsized":"2024-03-18T19:08:21.536300+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:04.663910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590475+0000","last_change":"2024-03-18T19:08:19.601877+0000","last_active":"2024-03-18T19:08:20.590475+0000","last_peered":"2024-03-18T19:08:20.590475+0000","last_clean":"2024-03-18T19:08:20.590475+0000","last_became_active":"2024-03-18T19:08:19.601628+0000","last_became_peered":"2024-03-18T19:08:19.601628+0000","last_unstale":"2024-03-18T19:08:20.590475+0000","last_undegraded":"2024-03-18T19:08:20.590475+0000","last_fullsized":"2024-03-18T19:08:20.590475+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:35.010798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735302+0000","last_change":"2024-03-18T19:08:19.165924+0000","last_active":"2024-03-18T19:08:20.735302+0000","last_peered":"2024-03-18T19:08:20.735302+0000","last_clean":"2024-03-18T19:08:20.735302+0000","last_became_active":"2024-03-18T19:08:19.165670+0000","last_became_peered":"2024-03-18T19:08:19.165670+0000","last_unstale":"2024-03-18T19:08:20.735302+0000","last_undegraded":"2024-03-18T19:08:20.735302+0000","last_fullsized":"2024-03-18T19:08:20.735302+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:50:20.781356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.613877+0000","last_change":"2024-03-18T19:08:19.613090+0000","last_active":"2024-03-18T19:08:19.613877+0000","last_peered":"2024-03-18T19:08:19.613877+0000","last_clean":"2024-03-18T19:08:19.613877+0000","last_became_active":"2024-03-18T19:08:19.612803+0000","last_became_peered":"2024-03-18T19:08:19.612803+0000","last_unstale":"2024-03-18T19:08:19.613877+0000","last_undegraded":"2024-03-18T19:08:19.613877+0000","last_fullsized":"2024-03-18T19:08:19.613877+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:12:11.205891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589821+0000","last_change":"2024-03-18T19:08:19.161643+0000","last_active":"2024-03-18T19:08:20.589821+0000","last_peered":"2024-03-18T19:08:20.589821+0000","last_clean":"2024-03-18T19:08:20.589821+0000","last_became_active":"2024-03-18T19:08:19.161320+0000","last_became_peered":"2024-03-18T19:08:19.161320+0000","last_unstale":"2024-03-18T19:08:20.589821+0000","last_undegraded":"2024-03-18T19:08:20.589821+0000","last_fullsized":"2024-03-18T19:08:20.589821+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:19:58.390893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.608119+0000","last_change":"2024-03-18T19:08:19.606097+0000","last_active":"2024-03-18T19:08:19.608119+0000","last_peered":"2024-03-18T19:08:19.608119+0000","last_clean":"2024-03-18T19:08:19.608119+0000","last_became_active":"2024-03-18T19:08:19.602719+0000","last_became_peered":"2024-03-18T19:08:19.602719+0000","last_unstale":"2024-03-18T19:08:19.608119+0000","last_undegraded":"2024-03-18T19:08:19.608119+0000","last_fullsized":"2024-03-18T19:08:19.608119+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:09:36.235776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.591082+0000","last_change":"2024-03-18T19:08:19.170578+0000","last_active":"2024-03-18T19:08:19.591082+0000","last_peered":"2024-03-18T19:08:19.591082+0000","last_clean":"2024-03-18T19:08:19.591082+0000","last_became_active":"2024-03-18T19:08:19.170332+0000","last_became_peered":"2024-03-18T19:08:19.170332+0000","last_unstale":"2024-03-18T19:08:19.591082+0000","last_undegraded":"2024-03-18T19:08:19.591082+0000","last_fullsized":"2024-03-18T19:08:19.591082+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:06:13.807256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.608146+0000","last_change":"2024-03-18T19:08:19.605113+0000","last_active":"2024-03-18T19:08:19.608146+0000","last_peered":"2024-03-18T19:08:19.608146+0000","last_clean":"2024-03-18T19:08:19.608146+0000","last_became_active":"2024-03-18T19:08:19.604849+0000","last_became_peered":"2024-03-18T19:08:19.604849+0000","last_unstale":"2024-03-18T19:08:19.608146+0000","last_undegraded":"2024-03-18T19:08:19.608146+0000","last_fullsized":"2024-03-18T19:08:19.608146+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:31.326003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.601649+0000","last_change":"2024-03-18T19:08:19.171031+0000","last_active":"2024-03-18T19:08:19.601649+0000","last_peered":"2024-03-18T19:08:19.601649+0000","last_clean":"2024-03-18T19:08:19.601649+0000","last_became_active":"2024-03-18T19:08:19.170750+0000","last_became_peered":"2024-03-18T19:08:19.170750+0000","last_unstale":"2024-03-18T19:08:19.601649+0000","last_undegraded":"2024-03-18T19:08:19.601649+0000","last_fullsized":"2024-03-18T19:08:19.601649+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:53.730763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.605665+0000","last_change":"2024-03-18T19:08:19.604372+0000","last_active":"2024-03-18T19:08:19.605665+0000","last_peered":"2024-03-18T19:08:19.605665+0000","last_clean":"2024-03-18T19:08:19.605665+0000","last_became_active":"2024-03-18T19:08:19.604098+0000","last_became_peered":"2024-03-18T19:08:19.604098+0000","last_unstale":"2024-03-18T19:08:19.605665+0000","last_undegraded":"2024-03-18T19:08:19.605665+0000","last_fullsized":"2024-03-18T19:08:19.605665+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:32.794300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735279+0000","last_change":"2024-03-18T19:08:19.166840+0000","last_active":"2024-03-18T19:08:20.735279+0000","last_peered":"2024-03-18T19:08:20.735279+0000","last_clean":"2024-03-18T19:08:20.735279+0000","last_became_active":"2024-03-18T19:08:19.166520+0000","last_became_peered":"2024-03-18T19:08:19.166520+0000","last_unstale":"2024-03-18T19:08:20.735279+0000","last_undegraded":"2024-03-18T19:08:20.735279+0000","last_fullsized":"2024-03-18T19:08:20.735279+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:55:38.817937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.607881+0000","last_change":"2024-03-18T19:08:19.606522+0000","last_active":"2024-03-18T19:08:19.607881+0000","last_peered":"2024-03-18T19:08:19.607881+0000","last_clean":"2024-03-18T19:08:19.607881+0000","last_became_active":"2024-03-18T19:08:19.606269+0000","last_became_peered":"2024-03-18T19:08:19.606269+0000","last_unstale":"2024-03-18T19:08:19.607881+0000","last_undegraded":"2024-03-18T19:08:19.607881+0000","last_fullsized":"2024-03-18T19:08:19.607881+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:51.825688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.601181+0000","last_change":"2024-03-18T19:08:19.171148+0000","last_active":"2024-03-18T19:08:19.601181+0000","last_peered":"2024-03-18T19:08:19.601181+0000","last_clean":"2024-03-18T19:08:19.601181+0000","last_became_active":"2024-03-18T19:08:19.170726+0000","last_became_peered":"2024-03-18T19:08:19.170726+0000","last_unstale":"2024-03-18T19:08:19.601181+0000","last_undegraded":"2024-03-18T19:08:19.601181+0000","last_fullsized":"2024-03-18T19:08:19.601181+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:52:04.509926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.616474+0000","last_change":"2024-03-18T19:08:19.614738+0000","last_active":"2024-03-18T19:08:19.616474+0000","last_peered":"2024-03-18T19:08:19.616474+0000","last_clean":"2024-03-18T19:08:19.616474+0000","last_became_active":"2024-03-18T19:08:19.614329+0000","last_became_peered":"2024-03-18T19:08:19.614329+0000","last_unstale":"2024-03-18T19:08:19.616474+0000","last_undegraded":"2024-03-18T19:08:19.616474+0000","last_fullsized":"2024-03-18T19:08:19.616474+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:49:04.996335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"132'2","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.618117+0000","last_change":"2024-03-18T19:08:19.171491+0000","last_active":"2024-03-18T19:08:19.618117+0000","last_peered":"2024-03-18T19:08:19.618117+0000","last_clean":"2024-03-18T19:08:19.618117+0000","last_became_active":"2024-03-18T19:08:19.171212+0000","last_became_peered":"2024-03-18T19:08:19.171212+0000","last_unstale":"2024-03-18T19:08:19.618117+0000","last_undegraded":"2024-03-18T19:08:19.618117+0000","last_fullsized":"2024-03-18T19:08:19.618117+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:41:01.232974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"122'205","reported_seq":575,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.153271+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T19:08:19.153271+0000","last_peered":"2024-03-18T19:08:19.153271+0000","last_clean":"2024-03-18T19:08:19.153271+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T19:08:19.153271+0000","last_undegraded":"2024-03-18T19:08:19.153271+0000","last_fullsized":"2024-03-18T19:08:19.153271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575066,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59548,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312292,"statfs":{"total":96636764160,"available":96575787008,"internally_reserved":0,"allocated":2236416,"data_stored":1411129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56200000000000006,"15min":0.57099999999999995},"min":{"1min":0.39600000000000002,"5min":0.35799999999999998,"15min":0.27800000000000002},"max":{"1min":0.78400000000000003,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55700000000000005,"15min":0.58199999999999996},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.318},"max":{"1min":0.70699999999999996,"5min":1.387,"15min":1.387},"last":0.39600000000000002}]},{"osd":1,"last update":"Mon Mar 18 19:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54200000000000004,"15min":0.57399999999999995},"min":{"1min":0.373,"5min":0.33200000000000002,"15min":0.307},"max":{"1min":0.74099999999999999,"5min":0.81699999999999995,"15min":1.484},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55100000000000005,"15min":0.58799999999999997},"min":{"1min":0.376,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67600000000000005,"5min":0.80700000000000005,"15min":0.92000000000000004},"last":0.497}]},{"osd":2,"last update":"Mon Mar 18 19:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.55000000000000004,"15min":0.57299999999999995},"min":{"1min":0.35099999999999998,"5min":0.34799999999999998,"15min":0.27800000000000002},"max":{"1min":0.70999999999999996,"5min":1.3600000000000001,"15min":1.5129999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56999999999999995,"15min":0.59299999999999997},"min":{"1min":0.372,"5min":0.35999999999999999,"15min":0.35299999999999998},"max":{"1min":0.78100000000000003,"5min":1.327,"15min":1.536},"last":0.58499999999999996}]},{"osd":3,"last update":"Mon Mar 18 19:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55200000000000005,"15min":0.57899999999999996},"min":{"1min":0.38900000000000001,"5min":0.35599999999999998,"15min":0.311},"max":{"1min":0.749,"5min":0.79100000000000004,"15min":1.5489999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56699999999999995,"15min":0.59899999999999998},"min":{"1min":0.44400000000000001,"5min":0.38600000000000001,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":1.5660000000000001},"last":0.629}]},{"osd":4,"last update":"Mon Mar 18 19:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.44600000000000001,"15min":0.45100000000000001},"min":{"1min":0.33300000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.67300000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.443,"5min":0.45500000000000002,"15min":0.47199999999999998},"min":{"1min":0.34599999999999997,"5min":0.27500000000000002,"15min":0.25800000000000001},"max":{"1min":0.56899999999999995,"5min":0.71599999999999997,"15min":0.72599999999999998},"last":0.55200000000000005}]},{"osd":5,"last update":"Mon Mar 18 19:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.47499999999999998,"15min":0.47599999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.26000000000000001},"max":{"1min":0.66000000000000003,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.50600000000000001,"15min":0.51400000000000001},"min":{"1min":0.35899999999999999,"5min":0.30299999999999999,"15min":0.28799999999999998},"max":{"1min":0.71499999999999997,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.56899999999999995}]},{"osd":6,"last update":"Mon Mar 18 19:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52400000000000002,"15min":0.54100000000000004},"min":{"1min":0.38,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.74099999999999999,"5min":0.81799999999999995,"15min":0.84499999999999997},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.52100000000000002,"15min":0.52800000000000002},"min":{"1min":0.311,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.64800000000000002,"5min":0.74099999999999999,"15min":0.77500000000000002},"last":0.68200000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575066,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32756,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339084,"statfs":{"total":96636764160,"available":96603222016,"internally_reserved":0,"allocated":1667072,"data_stored":829389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54500000000000004,"15min":0.56100000000000005},"min":{"1min":0.40799999999999997,"5min":0.31900000000000001,"15min":0.28499999999999998},"max":{"1min":0.73699999999999999,"5min":0.749,"15min":0.76800000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54700000000000004,"15min":0.56799999999999995},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.27400000000000002},"max":{"1min":0.70999999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.57699999999999996}]},{"osd":1,"last update":"Mon Mar 18 19:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54900000000000004,"15min":0.56399999999999995},"min":{"1min":0.39200000000000002,"5min":0.33700000000000002,"15min":0.316},"max":{"1min":0.71099999999999997,"5min":0.76700000000000002,"15min":0.77400000000000002},"last":0.628},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55600000000000005,"15min":0.56799999999999995},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32000000000000001},"max":{"1min":0.72299999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.72199999999999998}]},{"osd":2,"last update":"Mon Mar 18 19:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56299999999999994,"15min":0.57499999999999996},"min":{"1min":0.41199999999999998,"5min":0.32700000000000001,"15min":0.30499999999999999},"max":{"1min":0.78200000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.55900000000000005,"15min":0.57899999999999996},"min":{"1min":0.36899999999999999,"5min":0.35299999999999998,"15min":0.30299999999999999},"max":{"1min":0.751,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.70299999999999996}]},{"osd":3,"last update":"Mon Mar 18 19:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.49399999999999999,"5min":0.372,"15min":0.32100000000000001},"max":{"1min":0.72599999999999998,"5min":0.80700000000000005,"15min":0.89600000000000002},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56100000000000005,"15min":0.57499999999999996},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73899999999999999,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.68999999999999995}]},{"osd":4,"last update":"Mon Mar 18 19:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.46000000000000002,"15min":0.44700000000000001},"min":{"1min":0.317,"5min":0.26900000000000002,"15min":0.25600000000000001},"max":{"1min":0.65000000000000002,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48599999999999999,"15min":0.47499999999999998},"min":{"1min":0.35299999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.81699999999999995}]},{"osd":5,"last update":"Mon Mar 18 19:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.46999999999999997,"15min":0.46800000000000003},"min":{"1min":0.34499999999999997,"5min":0.27300000000000002,"15min":0.26900000000000002},"max":{"1min":0.67600000000000005,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.75},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.52800000000000002,"15min":0.51500000000000001},"min":{"1min":0.38500000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.753,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.879}]},{"osd":7,"last update":"Mon Mar 18 19:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.54100000000000004,"15min":0.52400000000000002},"min":{"1min":0.40500000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.76800000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.872},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55400000000000005,"15min":0.53100000000000003},"min":{"1min":0.39400000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.74399999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.78800000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575066,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36836,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335004,"statfs":{"total":96636764160,"available":96599044096,"internally_reserved":0,"allocated":1650688,"data_stored":827255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.36199999999999999,"15min":0.374},"min":{"1min":0.246,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.59099999999999997,"5min":0.63300000000000001,"15min":0.66300000000000003},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.41399999999999998},"min":{"1min":0.28399999999999997,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.60199999999999998,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Mon Mar 18 19:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.39700000000000002,"15min":0.40600000000000003},"min":{"1min":0.29299999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.55700000000000005,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.44500000000000001},"min":{"1min":0.29999999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.57499999999999996,"5min":0.66500000000000004,"15min":0.71699999999999997},"last":0.29599999999999999}]},{"osd":3,"last update":"Mon Mar 18 19:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44500000000000001,"15min":0.46500000000000002},"min":{"1min":0.29599999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.60799999999999998,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40400000000000003,"15min":0.41899999999999998},"min":{"1min":0.26900000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.53600000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.25900000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.60399999999999998,"15min":0.627},"min":{"1min":0.44600000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.84899999999999998,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.61499999999999999,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84599999999999997,"5min":1.018,"15min":1.018},"last":0.52700000000000002}]},{"osd":5,"last update":"Mon Mar 18 19:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.60399999999999998,"15min":0.63700000000000001},"min":{"1min":0.41199999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.872,"5min":0.89500000000000002,"15min":0.99199999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.629,"15min":0.65100000000000002},"min":{"1min":0.48299999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.90800000000000003,"5min":1.006,"15min":1.006},"last":0.55200000000000005}]},{"osd":6,"last update":"Mon Mar 18 19:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.60199999999999998,"15min":0.63300000000000001},"min":{"1min":0.42499999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83999999999999997,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64500000000000002,"15min":0.66600000000000004},"min":{"1min":0.42699999999999999,"5min":0.42199999999999999,"15min":0.39100000000000001},"max":{"1min":0.93600000000000005,"5min":0.96799999999999997,"15min":0.98099999999999998},"last":0.58199999999999996}]},{"osd":7,"last update":"Mon Mar 18 19:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.63500000000000001,"15min":0.65800000000000003},"min":{"1min":0.41299999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81000000000000005,"5min":0.92900000000000005,"15min":0.94799999999999995},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.63,"15min":0.65900000000000003},"min":{"1min":0.50900000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.53900000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575067,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37412,"kb_used_data":2192,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334428,"statfs":{"total":96636764160,"available":96598454272,"internally_reserved":0,"allocated":2244608,"data_stored":1412623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 19:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35799999999999998,"15min":0.378},"min":{"1min":0.251,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.52900000000000003,"5min":0.60499999999999998,"15min":0.621},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.39500000000000002,"15min":0.41399999999999998},"min":{"1min":0.33500000000000002,"5min":0.22700000000000001,"15min":0.20000000000000001},"max":{"1min":0.52400000000000002,"5min":0.57499999999999996,"15min":0.65100000000000002},"last":0.47499999999999998}]},{"osd":2,"last update":"Mon Mar 18 19:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.40699999999999997,"15min":0.41999999999999998},"min":{"1min":0.318,"5min":0.25600000000000001,"15min":0.21099999999999999},"max":{"1min":0.57699999999999996,"5min":0.60899999999999999,"15min":0.627},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.42499999999999999,"15min":0.437},"min":{"1min":0.318,"5min":0.25700000000000001,"15min":0.25},"max":{"1min":0.55500000000000005,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.39200000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39700000000000002,"15min":0.41499999999999998},"min":{"1min":0.28699999999999998,"5min":0.26500000000000001,"15min":0.217},"max":{"1min":0.58099999999999996,"5min":0.61299999999999999,"15min":0.64100000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.45600000000000002,"15min":0.47299999999999998},"min":{"1min":0.39000000000000001,"5min":0.32000000000000001,"15min":0.28799999999999998},"max":{"1min":0.62,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.51000000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.61599999999999999,"15min":0.64300000000000002},"min":{"1min":0.41299999999999998,"5min":0.33200000000000002,"15min":0.27200000000000002},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.61399999999999999,"15min":0.64300000000000002},"min":{"1min":0.42399999999999999,"5min":0.38200000000000001,"15min":0.315},"max":{"1min":0.91600000000000004,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.64100000000000001}]},{"osd":5,"last update":"Mon Mar 18 19:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63400000000000001,"15min":0.64900000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.34300000000000003},"max":{"1min":1.2030000000000001,"5min":1.248,"15min":1.248},"last":0.753},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.64700000000000002,"15min":0.66600000000000004},"min":{"1min":0.46999999999999997,"5min":0.38,"15min":0.31},"max":{"1min":1.2889999999999999,"5min":1.399,"15min":1.399},"last":0.624}]},{"osd":6,"last update":"Mon Mar 18 19:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.63500000000000001,"15min":0.65900000000000003},"min":{"1min":0.47899999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.1719999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.65500000000000003,"15min":0.67400000000000004},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.40999999999999998},"max":{"1min":1.2689999999999999,"5min":1.284,"15min":1.284},"last":0.71399999999999997}]},{"osd":7,"last update":"Mon Mar 18 19:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.65200000000000002,"15min":0.67100000000000004},"min":{"1min":0.443,"5min":0.38300000000000001,"15min":0.34899999999999998},"max":{"1min":1.2390000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.64400000000000002,"15min":0.67000000000000004},"min":{"1min":0.45000000000000001,"5min":0.42899999999999999,"15min":0.38500000000000001},"max":{"1min":0.86599999999999999,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.72599999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575067,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":1638400,"data_stored":820635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:08:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.36699999999999999,"15min":0.372},"min":{"1min":0.27800000000000002,"5min":0.19500000000000001,"15min":0.17799999999999999},"max":{"1min":0.58299999999999996,"5min":0.67400000000000004,"15min":0.78300000000000003},"last":0.25},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.45400000000000001,"15min":0.45700000000000002},"min":{"1min":0.32800000000000001,"5min":0.29399999999999998,"15min":0.217},"max":{"1min":0.68799999999999994,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.35499999999999998}]},{"osd":1,"last update":"Mon Mar 18 19:08:16 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45200000000000001,"15min":0.46800000000000003},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27100000000000002},"max":{"1min":0.60899999999999999,"5min":0.71899999999999997,"15min":0.90500000000000003},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.39200000000000002,"15min":0.39600000000000002},"min":{"1min":0.26300000000000001,"5min":0.22800000000000001,"15min":0.216},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.66700000000000004},"last":0.30399999999999999}]},{"osd":3,"last update":"Mon Mar 18 19:08:16 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.40600000000000003,"15min":0.41099999999999998},"min":{"1min":0.28000000000000003,"5min":0.247,"15min":0.22500000000000001},"max":{"1min":0.59399999999999997,"5min":0.65800000000000003,"15min":1.4019999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.48499999999999999,"15min":0.5},"min":{"1min":0.36699999999999999,"5min":0.27900000000000003,"15min":0.27700000000000002},"max":{"1min":0.70099999999999996,"5min":0.76400000000000001,"15min":0.80100000000000005},"last":0.438}]},{"osd":4,"last update":"Mon Mar 18 19:08:16 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61699999999999999},"min":{"1min":0.47899999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.71299999999999997,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60699999999999998,"15min":0.628},"min":{"1min":0.45000000000000001,"5min":0.36499999999999999,"15min":0.32100000000000001},"max":{"1min":0.78600000000000003,"5min":1.448,"15min":1.4730000000000001},"last":0.44800000000000001}]},{"osd":5,"last update":"Mon Mar 18 19:08:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.63800000000000001,"15min":0.64600000000000002},"min":{"1min":0.53900000000000003,"5min":0.39800000000000002,"15min":0.309},"max":{"1min":1.327,"5min":1.327,"15min":1.8109999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.626,"15min":0.63800000000000001},"min":{"1min":0.499,"5min":0.36899999999999999,"15min":0.33400000000000002},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.948},"last":0.56799999999999995}]},{"osd":6,"last update":"Mon Mar 18 19:08:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.63700000000000001,"15min":0.65800000000000003},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.35899999999999999},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":2.9470000000000001},"last":0.504},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68100000000000005,"15min":0.68500000000000005},"min":{"1min":0.53300000000000003,"5min":0.432,"15min":0.37},"max":{"1min":0.878,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.51200000000000001}]},{"osd":7,"last update":"Mon Mar 18 19:08:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.65100000000000002,"15min":0.65200000000000002},"min":{"1min":0.435,"5min":0.40999999999999998,"15min":0.38200000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.64500000000000002,"15min":0.66300000000000003},"min":{"1min":0.46400000000000002,"5min":0.38300000000000001,"15min":0.34599999999999997},"max":{"1min":0.80500000000000005,"5min":1.482,"15min":1.482},"last":0.47599999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575067,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36832,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335008,"statfs":{"total":96636764160,"available":96599048192,"internally_reserved":0,"allocated":1650688,"data_stored":827803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35299999999999998,"15min":0.379},"min":{"1min":0.22500000000000001,"5min":0.221,"15min":0.21099999999999999},"max":{"1min":0.51400000000000001,"5min":0.63900000000000001,"15min":1.204},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.377,"15min":0.40100000000000002},"min":{"1min":0.29799999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57399999999999995,"5min":0.66900000000000004,"15min":1.137},"last":0.28999999999999998}]},{"osd":1,"last update":"Mon Mar 18 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.374,"15min":0.39800000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.629,"5min":0.629,"15min":1.1220000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.40999999999999998,"15min":0.44500000000000001},"min":{"1min":0.33100000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":1.1080000000000001},"last":0.30299999999999999}]},{"osd":2,"last update":"Mon Mar 18 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.42399999999999999,"15min":0.45000000000000001},"min":{"1min":0.314,"5min":0.29499999999999998,"15min":0.28399999999999997},"max":{"1min":0.59799999999999998,"5min":0.68999999999999995,"15min":1.1879999999999999},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.44800000000000001,"15min":0.47799999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66600000000000004,"5min":0.71499999999999997,"15min":1.2290000000000001},"last":0.33900000000000002}]},{"osd":4,"last update":"Mon Mar 18 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57499999999999996,"15min":0.625},"min":{"1min":0.38100000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83399999999999996,"5min":1.052,"15min":1.871},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.58999999999999997,"15min":0.63200000000000001},"min":{"1min":0.47199999999999998,"5min":0.40300000000000002,"15min":0.39600000000000002},"max":{"1min":0.88100000000000001,"5min":0.90600000000000003,"15min":1.764},"last":0.40699999999999997}]},{"osd":5,"last update":"Mon Mar 18 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.624,"15min":0.66400000000000003},"min":{"1min":0.47699999999999998,"5min":0.39200000000000002,"15min":0.379},"max":{"1min":0.94199999999999995,"5min":1.0249999999999999,"15min":2.5329999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.64600000000000002,"15min":0.68500000000000005},"min":{"1min":0.48499999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.83299999999999996,"5min":0.94599999999999995,"15min":1.794},"last":0.47799999999999998}]},{"osd":6,"last update":"Mon Mar 18 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.628,"15min":0.67300000000000004},"min":{"1min":0.42499999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.88500000000000001,"5min":1.1060000000000001,"15min":2.7389999999999999},"last":0.496},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.64200000000000002,"15min":0.67900000000000005},"min":{"1min":0.50900000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.98199999999999998,"5min":0.98299999999999998,"15min":1.8069999999999999},"last":0.45000000000000001}]},{"osd":7,"last update":"Mon Mar 18 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63100000000000001,"15min":0.66900000000000004},"min":{"1min":0.45100000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.90000000000000002,"5min":0.97599999999999998,"15min":1.732},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.61899999999999999,"15min":0.67200000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.40799999999999997},"max":{"1min":0.874,"5min":1.085,"15min":2.7050000000000001},"last":0.51800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575067,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32728,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339112,"statfs":{"total":96636764160,"available":96603250688,"internally_reserved":0,"allocated":1642496,"data_stored":820693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54500000000000004,"15min":0.56899999999999995},"min":{"1min":0.35599999999999998,"5min":0.31,"15min":0.246},"max":{"1min":0.78800000000000003,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56799999999999995,"15min":0.58399999999999996},"min":{"1min":0.38900000000000001,"5min":0.35299999999999998,"15min":0.26000000000000001},"max":{"1min":0.748,"5min":3.645,"15min":3.645},"last":0.49299999999999999}]},{"osd":1,"last update":"Mon Mar 18 19:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.63200000000000001,"15min":0.60799999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76000000000000001,"5min":9.9499999999999993,"15min":9.9499999999999993},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53800000000000003,"15min":0.57399999999999995},"min":{"1min":0.34599999999999997,"5min":0.32300000000000001,"15min":0.308},"max":{"1min":0.73799999999999999,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon Mar 18 19:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54900000000000004,"15min":0.57199999999999995},"min":{"1min":0.39500000000000002,"5min":0.35899999999999999,"15min":0.28999999999999998},"max":{"1min":0.752,"5min":0.96299999999999997,"15min":1.0940000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.41999999999999998,"5min":0.35099999999999998,"15min":0.32600000000000001},"max":{"1min":0.79800000000000004,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.68100000000000005}]},{"osd":3,"last update":"Mon Mar 18 19:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55000000000000004,"15min":0.58199999999999996},"min":{"1min":0.40100000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.77000000000000002,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56100000000000005,"15min":0.59499999999999997},"min":{"1min":0.39900000000000002,"5min":0.39300000000000002,"15min":0.33900000000000002},"max":{"1min":0.77300000000000002,"5min":0.78200000000000003,"15min":1.121},"last":0.69899999999999995}]},{"osd":5,"last update":"Mon Mar 18 19:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.45900000000000002,"15min":0.46000000000000002},"min":{"1min":0.32600000000000001,"5min":0.254,"15min":0.245},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.77500000000000002},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.51100000000000001,"15min":0.49099999999999999},"min":{"1min":0.33500000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.78500000000000003,"5min":5.4900000000000002,"15min":5.4900000000000002},"last":0.629}]},{"osd":6,"last update":"Mon Mar 18 19:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.46800000000000003,"15min":0.47599999999999998},"min":{"1min":0.29899999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53500000000000003,"15min":0.52400000000000002},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.28999999999999998},"max":{"1min":0.82099999999999995,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":0.76200000000000001}]},{"osd":7,"last update":"Mon Mar 18 19:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51600000000000001,"15min":0.51500000000000001},"min":{"1min":0.374,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.74199999999999999,"5min":1.55,"15min":1.55},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.34699999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.73799999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575066,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50768,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321072,"statfs":{"total":96636764160,"available":96584777728,"internally_reserved":0,"allocated":1634304,"data_stored":820601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.54400000000000004,"15min":0.57199999999999995},"min":{"1min":0.36399999999999999,"5min":0.26800000000000002,"15min":0.26600000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":1.1679999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56299999999999994,"15min":0.57399999999999995},"min":{"1min":0.35299999999999998,"5min":0.33700000000000002,"15min":0.25800000000000001},"max":{"1min":0.69299999999999995,"5min":0.80600000000000005,"15min":1.1319999999999999},"last":0.58199999999999996}]},{"osd":1,"last update":"Mon Mar 18 19:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.54600000000000004,"15min":0.57399999999999995},"min":{"1min":0.34699999999999998,"5min":0.32600000000000001,"15min":0.29599999999999999},"max":{"1min":0.73899999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56699999999999995,"15min":0.58299999999999996},"min":{"1min":0.38200000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78400000000000003,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.56899999999999995}]},{"osd":2,"last update":"Mon Mar 18 19:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.55100000000000005,"15min":0.57699999999999996},"min":{"1min":0.377,"5min":0.30399999999999999,"15min":0.28399999999999997},"max":{"1min":0.66900000000000004,"5min":1.153,"15min":1.1579999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.55800000000000005,"15min":0.57599999999999996},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":0.77100000000000002,"5min":0.79500000000000004,"15min":1.0569999999999999},"last":0.55200000000000005}]},{"osd":3,"last update":"Mon Mar 18 19:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.58699999999999997},"min":{"1min":0.38,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82799999999999996,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58599999999999997,"15min":0.60199999999999998},"min":{"1min":0.37,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.73199999999999998,"5min":1.212,"15min":1.212},"last":0.49299999999999999}]},{"osd":4,"last update":"Mon Mar 18 19:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.47499999999999998,"15min":0.46400000000000002},"min":{"1min":0.30399999999999999,"5min":0.255,"15min":0.24199999999999999},"max":{"1min":0.64100000000000001,"5min":1.248,"15min":1.248},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.503,"15min":0.48899999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.23100000000000001},"max":{"1min":0.71899999999999997,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.377}]},{"osd":6,"last update":"Mon Mar 18 19:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47399999999999998,"15min":0.47199999999999998},"min":{"1min":0.26100000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.67500000000000004,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51500000000000001,"15min":0.50800000000000001},"min":{"1min":0.34100000000000003,"5min":0.32100000000000001,"15min":0.26800000000000002},"max":{"1min":0.68899999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.47999999999999998}]},{"osd":7,"last update":"Mon Mar 18 19:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52700000000000002,"15min":0.52300000000000002},"min":{"1min":0.32500000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.748,"5min":1.282,"15min":1.282},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55300000000000005,"15min":0.54400000000000004},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28499999999999998},"max":{"1min":0.73999999999999999,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.44800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T19:08:23.740 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T19:08:23.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.900+0000 7fa46f945700 1 Processor -- start 2024-03-18T19:08:23.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.900+0000 7fa46f945700 1 -- start start 2024-03-18T19:08:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.901+0000 7fa46f945700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa4681465a0 0x7fa468168660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.901+0000 7fa46f945700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa468072660 con 0x7fa468146190 2024-03-18T19:08:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.901+0000 7fa46f945700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa46806ebd0 con 0x7fa46806ae60 2024-03-18T19:08:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.901+0000 7fa46f945700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa46806f850 con 0x7fa4681465a0 2024-03-18T19:08:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.901+0000 7fa46d6e1700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa468146190 0x7fa46806a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54902/0 (socket says 172.21.15.1:54902) 2024-03-18T19:08:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.901+0000 7fa46d6e1700 1 -- 172.21.15.1:0/1194275126 learned_addr learned my addr 172.21.15.1:0/1194275126 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa45ffff700 1 -- 172.21.15.1:0/1194275126 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668524136 0 0) 0x7fa468072660 con 0x7fa468146190 2024-03-18T19:08:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa45ffff700 1 -- 172.21.15.1:0/1194275126 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4540035b0 con 0x7fa468146190 2024-03-18T19:08:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa46dee2700 1 --2- 172.21.15.1:0/1194275126 >> v2:172.21.15.165:3300/0 conn(0x7fa4681465a0 0x7fa468168660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa45ffff700 1 -- 172.21.15.1:0/1194275126 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1852229082 0 0) 0x7fa4540035b0 con 0x7fa468146190 2024-03-18T19:08:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa45ffff700 1 -- 172.21.15.1:0/1194275126 >> v2:172.21.15.165:3300/0 conn(0x7fa4681465a0 msgr2=0x7fa468168660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa45ffff700 1 --2- 172.21.15.1:0/1194275126 >> v2:172.21.15.165:3300/0 conn(0x7fa4681465a0 0x7fa468168660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa45ffff700 1 -- 172.21.15.1:0/1194275126 >> v1:172.21.15.165:6789/0 conn(0x7fa46806ae60 legacy=0x7fa468164f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa45ffff700 1 -- 172.21.15.1:0/1194275126 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa46806eeb0 con 0x7fa468146190 2024-03-18T19:08:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa45ffff700 1 -- 172.21.15.1:0/1194275126 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fa458001670 con 0x7fa468146190 2024-03-18T19:08:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa45ffff700 1 -- 172.21.15.1:0/1194275126 >> v1:172.21.15.1:6789/0 conn(0x7fa468146190 legacy=0x7fa46806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa45ffff700 1 --2- 172.21.15.1:0/1194275126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa454003a00 0x7fa454003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa45ffff700 1 --2- 172.21.15.1:0/1194275126 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa454004850 0x7fa454004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa45ffff700 1 --2- 172.21.15.1:0/1194275126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa454005210 0x7fa45400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa45ffff700 1 -- 172.21.15.1:0/1194275126 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46806f850 con 0x7fa454005210 2024-03-18T19:08:23.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa45ffff700 1 -- 172.21.15.1:0/1194275126 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45400c0b0 con 0x7fa454003a00 2024-03-18T19:08:23.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa45ffff700 1 -- 172.21.15.1:0/1194275126 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45400c270 con 0x7fa454004850 2024-03-18T19:08:23.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa46dee2700 1 --2- 172.21.15.1:0/1194275126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa454005210 0x7fa45400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa46d6e1700 1 --2- 172.21.15.1:0/1194275126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa454003a00 0x7fa454003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa46cee0700 1 --2- 172.21.15.1:0/1194275126 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa454004850 0x7fa454004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa46dee2700 1 -- 172.21.15.1:0/1194275126 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa454004850 msgr2=0x7fa454004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:23.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa46dee2700 1 --2- 172.21.15.1:0/1194275126 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa454004850 0x7fa454004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa46dee2700 1 -- 172.21.15.1:0/1194275126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa454003a00 msgr2=0x7fa454003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:23.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa46dee2700 1 --2- 172.21.15.1:0/1194275126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa454003a00 0x7fa454003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.902+0000 7fa46dee2700 1 -- 172.21.15.1:0/1194275126 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4540035b0 con 0x7fa454005210 2024-03-18T19:08:23.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.903+0000 7fa46dee2700 1 -- 172.21.15.1:0/1194275126 reap_dead start 2024-03-18T19:08:23.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.903+0000 7fa46dee2700 1 --2- 172.21.15.1:0/1194275126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa454005210 0x7fa45400ba10 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fa460009fc0 tx=0x7fa454005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5afa08723e9a6c84 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:23.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.903+0000 7fa45ffff700 1 -- 172.21.15.1:0/1194275126 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa460003a20 con 0x7fa454005210 2024-03-18T19:08:23.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.903+0000 7fa45ffff700 1 -- 172.21.15.1:0/1194275126 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa46000f040 con 0x7fa454005210 2024-03-18T19:08:23.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.903+0000 7fa45ffff700 1 -- 172.21.15.1:0/1194275126 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa460003da0 con 0x7fa454005210 2024-03-18T19:08:23.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.903+0000 7fa46f945700 1 -- 172.21.15.1:0/1194275126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa454005210 msgr2=0x7fa45400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:23.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.903+0000 7fa46f945700 1 --2- 172.21.15.1:0/1194275126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa454005210 0x7fa45400ba10 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fa460009fc0 tx=0x7fa454005880 comp rx=0 tx=0).stop 2024-03-18T19:08:23.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.903+0000 7fa46f945700 1 -- 172.21.15.1:0/1194275126 shutdown_connections 2024-03-18T19:08:23.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.903+0000 7fa46f945700 1 --2- 172.21.15.1:0/1194275126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa454005210 0x7fa45400ba10 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.903+0000 7fa46f945700 1 -- 172.21.15.1:0/1194275126 >> 172.21.15.1:0/1194275126 conn(0x7fa4680b1d20 msgr2=0x7fa4680b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:23.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.903+0000 7fa46f945700 1 -- 172.21.15.1:0/1194275126 shutdown_connections 2024-03-18T19:08:23.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.903+0000 7fa46f945700 1 -- 172.21.15.1:0/1194275126 wait complete. 2024-03-18T19:08:23.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.905+0000 7fa46f945700 1 Processor -- start 2024-03-18T19:08:23.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.905+0000 7fa46f945700 1 -- start start 2024-03-18T19:08:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.905+0000 7fa46f945700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa454005210 0x7fa46815b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.905+0000 7fa46f945700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4681e8e00 0x7fa4681e9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.905+0000 7fa46f945700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa468140740 0x7fa468140bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.905+0000 7fa46f945700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4681e97c0 con 0x7fa4681e8e00 2024-03-18T19:08:23.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.905+0000 7fa46f945700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa468141470 con 0x7fa468140740 2024-03-18T19:08:23.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.905+0000 7fa46f945700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa468141750 con 0x7fa454005210 2024-03-18T19:08:23.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.905+0000 7fa46cee0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4681e8e00 0x7fa4681e9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.905+0000 7fa46cee0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4681e8e00 0x7fa4681e9280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59430/0 (socket says 172.21.15.1:59430) 2024-03-18T19:08:23.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.906+0000 7fa46cee0700 1 -- 172.21.15.1:0/2427405966 learned_addr learned my addr 172.21.15.1:0/2427405966 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:23.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.906+0000 7fa46d6e1700 1 --2- 172.21.15.1:0/2427405966 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa454005210 0x7fa46815b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.906+0000 7fa46dee2700 1 --2- 172.21.15.1:0/2427405966 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa468140740 0x7fa468140bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.906+0000 7fa46cee0700 1 -- 172.21.15.1:0/2427405966 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa454005210 msgr2=0x7fa46815b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.906+0000 7fa46cee0700 1 --2- 172.21.15.1:0/2427405966 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa454005210 0x7fa46815b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.906+0000 7fa46cee0700 1 -- 172.21.15.1:0/2427405966 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa468140740 msgr2=0x7fa468140bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.906+0000 7fa46cee0700 1 --2- 172.21.15.1:0/2427405966 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa468140740 0x7fa468140bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.906+0000 7fa46cee0700 1 -- 172.21.15.1:0/2427405966 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa468141a30 con 0x7fa4681e8e00 2024-03-18T19:08:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.906+0000 7fa46d6e1700 1 --2- 172.21.15.1:0/2427405966 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa454005210 0x7fa46815b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:08:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.906+0000 7fa46dee2700 1 --2- 172.21.15.1:0/2427405966 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa468140740 0x7fa468140bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:08:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.906+0000 7fa46cee0700 1 --2- 172.21.15.1:0/2427405966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4681e8e00 0x7fa4681e9280 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fa46400f7f0 tx=0x7fa46400fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.906+0000 7fa45dffb700 1 -- 172.21.15.1:0/2427405966 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4640052e0 con 0x7fa4681e8e00 2024-03-18T19:08:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.906+0000 7fa45dffb700 1 -- 172.21.15.1:0/2427405966 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa464005440 con 0x7fa4681e8e00 2024-03-18T19:08:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.906+0000 7fa46f945700 1 -- 172.21.15.1:0/2427405966 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa468141d10 con 0x7fa4681e8e00 2024-03-18T19:08:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.906+0000 7fa45dffb700 1 -- 172.21.15.1:0/2427405966 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4640038a0 con 0x7fa4681e8e00 2024-03-18T19:08:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.906+0000 7fa46f945700 1 -- 172.21.15.1:0/2427405966 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa468141110 con 0x7fa4681e8e00 2024-03-18T19:08:23.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.907+0000 7fa46f945700 1 -- 172.21.15.1:0/2427405966 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4680676a0 con 0x7fa4681e8e00 2024-03-18T19:08:23.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.908+0000 7fa45dffb700 1 -- 172.21.15.1:0/2427405966 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa464003a00 con 0x7fa4681e8e00 2024-03-18T19:08:23.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.908+0000 7fa45dffb700 1 --2- 172.21.15.1:0/2427405966 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa44c06adb0 0x7fa44c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:23.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.908+0000 7fa45dffb700 1 -- 172.21.15.1:0/2427405966 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa464089460 con 0x7fa4681e8e00 2024-03-18T19:08:23.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.908+0000 7fa46d6e1700 1 --2- 172.21.15.1:0/2427405966 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa44c06adb0 0x7fa44c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:23.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.909+0000 7fa46d6e1700 1 --2- 172.21.15.1:0/2427405966 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa44c06adb0 0x7fa44c06d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa458007cf0 tx=0x7fa458009040 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:23.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:23.913+0000 7fa45dffb700 1 -- 172.21.15.1:0/2427405966 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa464054350 con 0x7fa4681e8e00 2024-03-18T19:08:24.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.112+0000 7fa46f945700 1 -- 172.21.15.1:0/2427405966 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa46806f470 con 0x7fa44c06adb0 2024-03-18T19:08:24.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.120+0000 7fa45dffb700 1 -- 172.21.15.1:0/2427405966 <== mgr.8061 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290244 (secure 0 0 0) 0x7fa458007e30 con 0x7fa44c06adb0 2024-03-18T19:08:24.120 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:24.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.123+0000 7fa46f945700 1 -- 172.21.15.1:0/2427405966 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa44c06adb0 msgr2=0x7fa44c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:24.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.123+0000 7fa46f945700 1 --2- 172.21.15.1:0/2427405966 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa44c06adb0 0x7fa44c06d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa458007cf0 tx=0x7fa458009040 comp rx=0 tx=0).stop 2024-03-18T19:08:24.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.123+0000 7fa46f945700 1 -- 172.21.15.1:0/2427405966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4681e8e00 msgr2=0x7fa4681e9280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:24.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.123+0000 7fa46f945700 1 --2- 172.21.15.1:0/2427405966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4681e8e00 0x7fa4681e9280 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fa46400f7f0 tx=0x7fa46400fbb0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.124+0000 7fa46f945700 1 -- 172.21.15.1:0/2427405966 shutdown_connections 2024-03-18T19:08:24.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.124+0000 7fa46f945700 1 --2- 172.21.15.1:0/2427405966 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa44c06adb0 0x7fa44c06d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.124+0000 7fa46f945700 1 --2- 172.21.15.1:0/2427405966 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa454005210 0x7fa46815b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.124+0000 7fa46f945700 1 --2- 172.21.15.1:0/2427405966 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa468140740 0x7fa468140bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.124+0000 7fa46f945700 1 --2- 172.21.15.1:0/2427405966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4681e8e00 0x7fa4681e9280 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.124+0000 7fa46f945700 1 -- 172.21.15.1:0/2427405966 >> 172.21.15.1:0/2427405966 conn(0x7fa4680b1d20 msgr2=0x7fa468071650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.124+0000 7fa46f945700 1 -- 172.21.15.1:0/2427405966 shutdown_connections 2024-03-18T19:08:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.124+0000 7fa46f945700 1 -- 172.21.15.1:0/2427405966 wait complete. 2024-03-18T19:08:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T19:08:24.131 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-18T19:08:24.042763+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":363,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":345844,"kb_used_data":14028,"kb_used_omap":12,"kb_used_meta":331635,"kb_avail":754628876,"statfs":{"total":773094113280,"available":772739969024,"internally_reserved":0,"allocated":14364672,"data_stored":7770128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":339594809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.897967"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536456+0000","last_change":"2024-03-18T19:08:19.601950+0000","last_active":"2024-03-18T19:08:21.536456+0000","last_peered":"2024-03-18T19:08:21.536456+0000","last_clean":"2024-03-18T19:08:21.536456+0000","last_became_active":"2024-03-18T19:08:19.601675+0000","last_became_peered":"2024-03-18T19:08:19.601675+0000","last_unstale":"2024-03-18T19:08:21.536456+0000","last_undegraded":"2024-03-18T19:08:21.536456+0000","last_fullsized":"2024-03-18T19:08:21.536456+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:57:06.127424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735398+0000","last_change":"2024-03-18T19:08:19.610511+0000","last_active":"2024-03-18T19:08:20.735398+0000","last_peered":"2024-03-18T19:08:20.735398+0000","last_clean":"2024-03-18T19:08:20.735398+0000","last_became_active":"2024-03-18T19:08:19.609922+0000","last_became_peered":"2024-03-18T19:08:19.609922+0000","last_unstale":"2024-03-18T19:08:20.735398+0000","last_undegraded":"2024-03-18T19:08:20.735398+0000","last_fullsized":"2024-03-18T19:08:20.735398+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:32.794300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536211+0000","last_change":"2024-03-18T19:08:19.607208+0000","last_active":"2024-03-18T19:08:21.536211+0000","last_peered":"2024-03-18T19:08:21.536211+0000","last_clean":"2024-03-18T19:08:21.536211+0000","last_became_active":"2024-03-18T19:08:19.606969+0000","last_became_peered":"2024-03-18T19:08:19.606969+0000","last_unstale":"2024-03-18T19:08:21.536211+0000","last_undegraded":"2024-03-18T19:08:21.536211+0000","last_fullsized":"2024-03-18T19:08:21.536211+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:54:19.583130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590473+0000","last_change":"2024-03-18T19:08:19.602888+0000","last_active":"2024-03-18T19:08:20.590473+0000","last_peered":"2024-03-18T19:08:20.590473+0000","last_clean":"2024-03-18T19:08:20.590473+0000","last_became_active":"2024-03-18T19:08:19.602487+0000","last_became_peered":"2024-03-18T19:08:19.602487+0000","last_unstale":"2024-03-18T19:08:20.590473+0000","last_undegraded":"2024-03-18T19:08:20.590473+0000","last_fullsized":"2024-03-18T19:08:20.590473+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:51:29.329926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589345+0000","last_change":"2024-03-18T19:08:19.600315+0000","last_active":"2024-03-18T19:08:20.589345+0000","last_peered":"2024-03-18T19:08:20.589345+0000","last_clean":"2024-03-18T19:08:20.589345+0000","last_became_active":"2024-03-18T19:08:19.600044+0000","last_became_peered":"2024-03-18T19:08:19.600044+0000","last_unstale":"2024-03-18T19:08:20.589345+0000","last_undegraded":"2024-03-18T19:08:20.589345+0000","last_fullsized":"2024-03-18T19:08:20.589345+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:51.825688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735115+0000","last_change":"2024-03-18T19:08:19.601463+0000","last_active":"2024-03-18T19:08:20.735115+0000","last_peered":"2024-03-18T19:08:20.735115+0000","last_clean":"2024-03-18T19:08:20.735115+0000","last_became_active":"2024-03-18T19:08:19.601177+0000","last_became_peered":"2024-03-18T19:08:19.601177+0000","last_unstale":"2024-03-18T19:08:20.735115+0000","last_undegraded":"2024-03-18T19:08:20.735115+0000","last_fullsized":"2024-03-18T19:08:20.735115+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:54.114622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.734965+0000","last_change":"2024-03-18T19:08:19.600528+0000","last_active":"2024-03-18T19:08:20.734965+0000","last_peered":"2024-03-18T19:08:20.734965+0000","last_clean":"2024-03-18T19:08:20.734965+0000","last_became_active":"2024-03-18T19:08:19.600263+0000","last_became_peered":"2024-03-18T19:08:19.600263+0000","last_unstale":"2024-03-18T19:08:20.734965+0000","last_undegraded":"2024-03-18T19:08:20.734965+0000","last_fullsized":"2024-03-18T19:08:20.734965+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:12:11.205891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.616276+0000","last_change":"2024-03-18T19:08:19.614620+0000","last_active":"2024-03-18T19:08:19.616276+0000","last_peered":"2024-03-18T19:08:19.616276+0000","last_clean":"2024-03-18T19:08:19.616276+0000","last_became_active":"2024-03-18T19:08:19.614436+0000","last_became_peered":"2024-03-18T19:08:19.614436+0000","last_unstale":"2024-03-18T19:08:19.616276+0000","last_undegraded":"2024-03-18T19:08:19.616276+0000","last_fullsized":"2024-03-18T19:08:19.616276+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:55:39.823009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735332+0000","last_change":"2024-03-18T19:08:19.603641+0000","last_active":"2024-03-18T19:08:20.735332+0000","last_peered":"2024-03-18T19:08:20.735332+0000","last_clean":"2024-03-18T19:08:20.735332+0000","last_became_active":"2024-03-18T19:08:19.603353+0000","last_became_peered":"2024-03-18T19:08:19.603353+0000","last_unstale":"2024-03-18T19:08:20.735332+0000","last_undegraded":"2024-03-18T19:08:20.735332+0000","last_fullsized":"2024-03-18T19:08:20.735332+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:46.115796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589965+0000","last_change":"2024-03-18T19:08:19.609437+0000","last_active":"2024-03-18T19:08:20.589965+0000","last_peered":"2024-03-18T19:08:20.589965+0000","last_clean":"2024-03-18T19:08:20.589965+0000","last_became_active":"2024-03-18T19:08:19.609162+0000","last_became_peered":"2024-03-18T19:08:19.609162+0000","last_unstale":"2024-03-18T19:08:20.589965+0000","last_undegraded":"2024-03-18T19:08:20.589965+0000","last_fullsized":"2024-03-18T19:08:20.589965+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:36:41.211897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.616737+0000","last_change":"2024-03-18T19:08:19.614990+0000","last_active":"2024-03-18T19:08:19.616737+0000","last_peered":"2024-03-18T19:08:19.616737+0000","last_clean":"2024-03-18T19:08:19.616737+0000","last_became_active":"2024-03-18T19:08:19.614660+0000","last_became_peered":"2024-03-18T19:08:19.614660+0000","last_unstale":"2024-03-18T19:08:19.616737+0000","last_undegraded":"2024-03-18T19:08:19.616737+0000","last_fullsized":"2024-03-18T19:08:19.616737+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:31.979650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.606684+0000","last_change":"2024-03-18T19:08:19.602172+0000","last_active":"2024-03-18T19:08:19.606684+0000","last_peered":"2024-03-18T19:08:19.606684+0000","last_clean":"2024-03-18T19:08:19.606684+0000","last_became_active":"2024-03-18T19:08:19.601913+0000","last_became_peered":"2024-03-18T19:08:19.601913+0000","last_unstale":"2024-03-18T19:08:19.606684+0000","last_undegraded":"2024-03-18T19:08:19.606684+0000","last_fullsized":"2024-03-18T19:08:19.606684+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:19:47.819745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590535+0000","last_change":"2024-03-18T19:08:19.603927+0000","last_active":"2024-03-18T19:08:20.590535+0000","last_peered":"2024-03-18T19:08:20.590535+0000","last_clean":"2024-03-18T19:08:20.590535+0000","last_became_active":"2024-03-18T19:08:19.603525+0000","last_became_peered":"2024-03-18T19:08:19.603525+0000","last_unstale":"2024-03-18T19:08:20.590535+0000","last_undegraded":"2024-03-18T19:08:20.590535+0000","last_fullsized":"2024-03-18T19:08:20.590535+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:13:04.646632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.597520+0000","last_change":"2024-03-18T19:08:19.596954+0000","last_active":"2024-03-18T19:08:19.597520+0000","last_peered":"2024-03-18T19:08:19.597520+0000","last_clean":"2024-03-18T19:08:19.597520+0000","last_became_active":"2024-03-18T19:08:19.595777+0000","last_became_peered":"2024-03-18T19:08:19.595777+0000","last_unstale":"2024-03-18T19:08:19.597520+0000","last_undegraded":"2024-03-18T19:08:19.597520+0000","last_fullsized":"2024-03-18T19:08:19.597520+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:23:14.545049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.605093+0000","last_change":"2024-03-18T19:08:19.603344+0000","last_active":"2024-03-18T19:08:19.605093+0000","last_peered":"2024-03-18T19:08:19.605093+0000","last_clean":"2024-03-18T19:08:19.605093+0000","last_became_active":"2024-03-18T19:08:19.603048+0000","last_became_peered":"2024-03-18T19:08:19.603048+0000","last_unstale":"2024-03-18T19:08:19.605093+0000","last_undegraded":"2024-03-18T19:08:19.605093+0000","last_fullsized":"2024-03-18T19:08:19.605093+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:36:41.211897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.616367+0000","last_change":"2024-03-18T19:08:19.615532+0000","last_active":"2024-03-18T19:08:19.616367+0000","last_peered":"2024-03-18T19:08:19.616367+0000","last_clean":"2024-03-18T19:08:19.616367+0000","last_became_active":"2024-03-18T19:08:19.615134+0000","last_became_peered":"2024-03-18T19:08:19.615134+0000","last_unstale":"2024-03-18T19:08:19.616367+0000","last_undegraded":"2024-03-18T19:08:19.616367+0000","last_fullsized":"2024-03-18T19:08:19.616367+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:46.115796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536455+0000","last_change":"2024-03-18T19:08:19.608311+0000","last_active":"2024-03-18T19:08:21.536455+0000","last_peered":"2024-03-18T19:08:21.536455+0000","last_clean":"2024-03-18T19:08:21.536455+0000","last_became_active":"2024-03-18T19:08:19.607879+0000","last_became_peered":"2024-03-18T19:08:19.607879+0000","last_unstale":"2024-03-18T19:08:21.536455+0000","last_undegraded":"2024-03-18T19:08:21.536455+0000","last_fullsized":"2024-03-18T19:08:21.536455+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:12:11.441836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735497+0000","last_change":"2024-03-18T19:08:19.611314+0000","last_active":"2024-03-18T19:08:20.735497+0000","last_peered":"2024-03-18T19:08:20.735497+0000","last_clean":"2024-03-18T19:08:20.735497+0000","last_became_active":"2024-03-18T19:08:19.611000+0000","last_became_peered":"2024-03-18T19:08:19.611000+0000","last_unstale":"2024-03-18T19:08:20.735497+0000","last_undegraded":"2024-03-18T19:08:20.735497+0000","last_fullsized":"2024-03-18T19:08:20.735497+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:23:14.545049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536358+0000","last_change":"2024-03-18T19:08:19.615396+0000","last_active":"2024-03-18T19:08:21.536358+0000","last_peered":"2024-03-18T19:08:21.536358+0000","last_clean":"2024-03-18T19:08:21.536358+0000","last_became_active":"2024-03-18T19:08:19.614729+0000","last_became_peered":"2024-03-18T19:08:19.614729+0000","last_unstale":"2024-03-18T19:08:21.536358+0000","last_undegraded":"2024-03-18T19:08:21.536358+0000","last_fullsized":"2024-03-18T19:08:21.536358+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:42.662793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.609840+0000","last_change":"2024-03-18T19:08:19.602688+0000","last_active":"2024-03-18T19:08:19.609840+0000","last_peered":"2024-03-18T19:08:19.609840+0000","last_clean":"2024-03-18T19:08:19.609840+0000","last_became_active":"2024-03-18T19:08:19.602401+0000","last_became_peered":"2024-03-18T19:08:19.602401+0000","last_unstale":"2024-03-18T19:08:19.609840+0000","last_undegraded":"2024-03-18T19:08:19.609840+0000","last_fullsized":"2024-03-18T19:08:19.609840+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:05:03.506261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.608563+0000","last_change":"2024-03-18T19:08:19.607660+0000","last_active":"2024-03-18T19:08:19.608563+0000","last_peered":"2024-03-18T19:08:19.608563+0000","last_clean":"2024-03-18T19:08:19.608563+0000","last_became_active":"2024-03-18T19:08:19.607371+0000","last_became_peered":"2024-03-18T19:08:19.607371+0000","last_unstale":"2024-03-18T19:08:19.608563+0000","last_undegraded":"2024-03-18T19:08:19.608563+0000","last_fullsized":"2024-03-18T19:08:19.608563+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:50:21.786428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589502+0000","last_change":"2024-03-18T19:08:19.609931+0000","last_active":"2024-03-18T19:08:20.589502+0000","last_peered":"2024-03-18T19:08:20.589502+0000","last_clean":"2024-03-18T19:08:20.589502+0000","last_became_active":"2024-03-18T19:08:19.608632+0000","last_became_peered":"2024-03-18T19:08:19.608632+0000","last_unstale":"2024-03-18T19:08:20.589502+0000","last_undegraded":"2024-03-18T19:08:20.589502+0000","last_fullsized":"2024-03-18T19:08:20.589502+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:46.115796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536337+0000","last_change":"2024-03-18T19:08:19.601572+0000","last_active":"2024-03-18T19:08:21.536337+0000","last_peered":"2024-03-18T19:08:21.536337+0000","last_clean":"2024-03-18T19:08:21.536337+0000","last_became_active":"2024-03-18T19:08:19.601326+0000","last_became_peered":"2024-03-18T19:08:19.601326+0000","last_unstale":"2024-03-18T19:08:21.536337+0000","last_undegraded":"2024-03-18T19:08:21.536337+0000","last_fullsized":"2024-03-18T19:08:21.536337+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:50.261113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589354+0000","last_change":"2024-03-18T19:08:19.599930+0000","last_active":"2024-03-18T19:08:20.589354+0000","last_peered":"2024-03-18T19:08:20.589354+0000","last_clean":"2024-03-18T19:08:20.589354+0000","last_became_active":"2024-03-18T19:08:19.599654+0000","last_became_peered":"2024-03-18T19:08:19.599654+0000","last_unstale":"2024-03-18T19:08:20.589354+0000","last_undegraded":"2024-03-18T19:08:20.589354+0000","last_fullsized":"2024-03-18T19:08:20.589354+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:31.979650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.610321+0000","last_change":"2024-03-18T19:08:19.607615+0000","last_active":"2024-03-18T19:08:19.610321+0000","last_peered":"2024-03-18T19:08:19.610321+0000","last_clean":"2024-03-18T19:08:19.610321+0000","last_became_active":"2024-03-18T19:08:19.607332+0000","last_became_peered":"2024-03-18T19:08:19.607332+0000","last_unstale":"2024-03-18T19:08:19.610321+0000","last_undegraded":"2024-03-18T19:08:19.610321+0000","last_fullsized":"2024-03-18T19:08:19.610321+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:36.075071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.613344+0000","last_change":"2024-03-18T19:08:19.610335+0000","last_active":"2024-03-18T19:08:19.613344+0000","last_peered":"2024-03-18T19:08:19.613344+0000","last_clean":"2024-03-18T19:08:19.613344+0000","last_became_active":"2024-03-18T19:08:19.610073+0000","last_became_peered":"2024-03-18T19:08:19.610073+0000","last_unstale":"2024-03-18T19:08:19.613344+0000","last_undegraded":"2024-03-18T19:08:19.613344+0000","last_fullsized":"2024-03-18T19:08:19.613344+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:11:13.131189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.609048+0000","last_change":"2024-03-18T19:08:19.605712+0000","last_active":"2024-03-18T19:08:19.609048+0000","last_peered":"2024-03-18T19:08:19.609048+0000","last_clean":"2024-03-18T19:08:19.609048+0000","last_became_active":"2024-03-18T19:08:19.605463+0000","last_became_peered":"2024-03-18T19:08:19.605463+0000","last_unstale":"2024-03-18T19:08:19.609048+0000","last_undegraded":"2024-03-18T19:08:19.609048+0000","last_fullsized":"2024-03-18T19:08:19.609048+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:23:14.545049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735318+0000","last_change":"2024-03-18T19:08:19.611955+0000","last_active":"2024-03-18T19:08:20.735318+0000","last_peered":"2024-03-18T19:08:20.735318+0000","last_clean":"2024-03-18T19:08:20.735318+0000","last_became_active":"2024-03-18T19:08:19.611686+0000","last_became_peered":"2024-03-18T19:08:19.611686+0000","last_unstale":"2024-03-18T19:08:20.735318+0000","last_undegraded":"2024-03-18T19:08:20.735318+0000","last_fullsized":"2024-03-18T19:08:20.735318+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:39:17.365258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.608526+0000","last_change":"2024-03-18T19:08:19.606635+0000","last_active":"2024-03-18T19:08:19.608526+0000","last_peered":"2024-03-18T19:08:19.608526+0000","last_clean":"2024-03-18T19:08:19.608526+0000","last_became_active":"2024-03-18T19:08:19.606387+0000","last_became_peered":"2024-03-18T19:08:19.606387+0000","last_unstale":"2024-03-18T19:08:19.608526+0000","last_undegraded":"2024-03-18T19:08:19.608526+0000","last_fullsized":"2024-03-18T19:08:19.608526+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:47:16.302386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.610496+0000","last_change":"2024-03-18T19:08:19.607687+0000","last_active":"2024-03-18T19:08:19.610496+0000","last_peered":"2024-03-18T19:08:19.610496+0000","last_clean":"2024-03-18T19:08:19.610496+0000","last_became_active":"2024-03-18T19:08:19.607375+0000","last_became_peered":"2024-03-18T19:08:19.607375+0000","last_unstale":"2024-03-18T19:08:19.610496+0000","last_undegraded":"2024-03-18T19:08:19.610496+0000","last_fullsized":"2024-03-18T19:08:19.610496+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:17.177831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735125+0000","last_change":"2024-03-18T19:08:19.606174+0000","last_active":"2024-03-18T19:08:20.735125+0000","last_peered":"2024-03-18T19:08:20.735125+0000","last_clean":"2024-03-18T19:08:20.735125+0000","last_became_active":"2024-03-18T19:08:19.605924+0000","last_became_peered":"2024-03-18T19:08:19.605924+0000","last_unstale":"2024-03-18T19:08:20.735125+0000","last_undegraded":"2024-03-18T19:08:20.735125+0000","last_fullsized":"2024-03-18T19:08:20.735125+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:49:04.996335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.617428+0000","last_change":"2024-03-18T19:08:19.616439+0000","last_active":"2024-03-18T19:08:19.617428+0000","last_peered":"2024-03-18T19:08:19.617428+0000","last_clean":"2024-03-18T19:08:19.617428+0000","last_became_active":"2024-03-18T19:08:19.615997+0000","last_became_peered":"2024-03-18T19:08:19.615997+0000","last_unstale":"2024-03-18T19:08:19.617428+0000","last_undegraded":"2024-03-18T19:08:19.617428+0000","last_fullsized":"2024-03-18T19:08:19.617428+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:58:13.838042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536235+0000","last_change":"2024-03-18T19:08:19.607013+0000","last_active":"2024-03-18T19:08:21.536235+0000","last_peered":"2024-03-18T19:08:21.536235+0000","last_clean":"2024-03-18T19:08:21.536235+0000","last_became_active":"2024-03-18T19:08:19.606591+0000","last_became_peered":"2024-03-18T19:08:19.606591+0000","last_unstale":"2024-03-18T19:08:21.536235+0000","last_undegraded":"2024-03-18T19:08:21.536235+0000","last_fullsized":"2024-03-18T19:08:21.536235+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:19:41.496312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"132'4","reported_seq":15,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590608+0000","last_change":"2024-03-18T19:08:19.165661+0000","last_active":"2024-03-18T19:08:20.590608+0000","last_peered":"2024-03-18T19:08:20.590608+0000","last_clean":"2024-03-18T19:08:20.590608+0000","last_became_active":"2024-03-18T19:08:19.165328+0000","last_became_peered":"2024-03-18T19:08:19.165328+0000","last_unstale":"2024-03-18T19:08:20.590608+0000","last_undegraded":"2024-03-18T19:08:20.590608+0000","last_fullsized":"2024-03-18T19:08:20.590608+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:35.069998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.612335+0000","last_change":"2024-03-18T19:08:19.610079+0000","last_active":"2024-03-18T19:08:19.612335+0000","last_peered":"2024-03-18T19:08:19.612335+0000","last_clean":"2024-03-18T19:08:19.612335+0000","last_became_active":"2024-03-18T19:08:19.609716+0000","last_became_peered":"2024-03-18T19:08:19.609716+0000","last_unstale":"2024-03-18T19:08:19.612335+0000","last_undegraded":"2024-03-18T19:08:19.612335+0000","last_fullsized":"2024-03-18T19:08:19.612335+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:13:50.766336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"132'1","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.616040+0000","last_change":"2024-03-18T19:08:19.168730+0000","last_active":"2024-03-18T19:08:19.616040+0000","last_peered":"2024-03-18T19:08:19.616040+0000","last_clean":"2024-03-18T19:08:19.616040+0000","last_became_active":"2024-03-18T19:08:19.168221+0000","last_became_peered":"2024-03-18T19:08:19.168221+0000","last_unstale":"2024-03-18T19:08:19.616040+0000","last_undegraded":"2024-03-18T19:08:19.616040+0000","last_fullsized":"2024-03-18T19:08:19.616040+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:30.492658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.609794+0000","last_change":"2024-03-18T19:08:19.603336+0000","last_active":"2024-03-18T19:08:19.609794+0000","last_peered":"2024-03-18T19:08:19.609794+0000","last_clean":"2024-03-18T19:08:19.609794+0000","last_became_active":"2024-03-18T19:08:19.598930+0000","last_became_peered":"2024-03-18T19:08:19.598930+0000","last_unstale":"2024-03-18T19:08:19.609794+0000","last_undegraded":"2024-03-18T19:08:19.609794+0000","last_fullsized":"2024-03-18T19:08:19.609794+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:52:38.917172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"132'2","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.608103+0000","last_change":"2024-03-18T19:08:19.168965+0000","last_active":"2024-03-18T19:08:19.608103+0000","last_peered":"2024-03-18T19:08:19.608103+0000","last_clean":"2024-03-18T19:08:19.608103+0000","last_became_active":"2024-03-18T19:08:19.168573+0000","last_became_peered":"2024-03-18T19:08:19.168573+0000","last_unstale":"2024-03-18T19:08:19.608103+0000","last_undegraded":"2024-03-18T19:08:19.608103+0000","last_fullsized":"2024-03-18T19:08:19.608103+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:23:13.539977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.610829+0000","last_change":"2024-03-18T19:08:19.610336+0000","last_active":"2024-03-18T19:08:19.610829+0000","last_peered":"2024-03-18T19:08:19.610829+0000","last_clean":"2024-03-18T19:08:19.610829+0000","last_became_active":"2024-03-18T19:08:19.610077+0000","last_became_peered":"2024-03-18T19:08:19.610077+0000","last_unstale":"2024-03-18T19:08:19.610829+0000","last_undegraded":"2024-03-18T19:08:19.610829+0000","last_fullsized":"2024-03-18T19:08:19.610829+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:54.114622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"132'2","reported_seq":13,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589777+0000","last_change":"2024-03-18T19:08:19.166836+0000","last_active":"2024-03-18T19:08:20.589777+0000","last_peered":"2024-03-18T19:08:20.589777+0000","last_clean":"2024-03-18T19:08:20.589777+0000","last_became_active":"2024-03-18T19:08:19.166571+0000","last_became_peered":"2024-03-18T19:08:19.166571+0000","last_unstale":"2024-03-18T19:08:20.589777+0000","last_undegraded":"2024-03-18T19:08:20.589777+0000","last_fullsized":"2024-03-18T19:08:20.589777+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:04:25.186213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590568+0000","last_change":"2024-03-18T19:08:19.596579+0000","last_active":"2024-03-18T19:08:20.590568+0000","last_peered":"2024-03-18T19:08:20.590568+0000","last_clean":"2024-03-18T19:08:20.590568+0000","last_became_active":"2024-03-18T19:08:19.596283+0000","last_became_peered":"2024-03-18T19:08:19.596283+0000","last_unstale":"2024-03-18T19:08:20.590568+0000","last_undegraded":"2024-03-18T19:08:20.590568+0000","last_fullsized":"2024-03-18T19:08:20.590568+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:11:00.314452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"132'1","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.605733+0000","last_change":"2024-03-18T19:08:19.160250+0000","last_active":"2024-03-18T19:08:19.605733+0000","last_peered":"2024-03-18T19:08:19.605733+0000","last_clean":"2024-03-18T19:08:19.605733+0000","last_became_active":"2024-03-18T19:08:19.159902+0000","last_became_peered":"2024-03-18T19:08:19.159902+0000","last_unstale":"2024-03-18T19:08:19.605733+0000","last_undegraded":"2024-03-18T19:08:19.605733+0000","last_fullsized":"2024-03-18T19:08:19.605733+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:59:31.879611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536293+0000","last_change":"2024-03-18T19:08:19.615764+0000","last_active":"2024-03-18T19:08:21.536293+0000","last_peered":"2024-03-18T19:08:21.536293+0000","last_clean":"2024-03-18T19:08:21.536293+0000","last_became_active":"2024-03-18T19:08:19.615081+0000","last_became_peered":"2024-03-18T19:08:19.615081+0000","last_unstale":"2024-03-18T19:08:21.536293+0000","last_undegraded":"2024-03-18T19:08:21.536293+0000","last_fullsized":"2024-03-18T19:08:21.536293+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:39:02.694519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"132'1","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.609994+0000","last_change":"2024-03-18T19:08:19.165945+0000","last_active":"2024-03-18T19:08:19.609994+0000","last_peered":"2024-03-18T19:08:19.609994+0000","last_clean":"2024-03-18T19:08:19.609994+0000","last_became_active":"2024-03-18T19:08:19.165670+0000","last_became_peered":"2024-03-18T19:08:19.165670+0000","last_unstale":"2024-03-18T19:08:19.609994+0000","last_undegraded":"2024-03-18T19:08:19.609994+0000","last_fullsized":"2024-03-18T19:08:19.609994+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:20.352711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735661+0000","last_change":"2024-03-18T19:08:19.610692+0000","last_active":"2024-03-18T19:08:20.735661+0000","last_peered":"2024-03-18T19:08:20.735661+0000","last_clean":"2024-03-18T19:08:20.735661+0000","last_became_active":"2024-03-18T19:08:19.610265+0000","last_became_peered":"2024-03-18T19:08:19.610265+0000","last_unstale":"2024-03-18T19:08:20.735661+0000","last_undegraded":"2024-03-18T19:08:20.735661+0000","last_fullsized":"2024-03-18T19:08:20.735661+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:06.122862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.591816+0000","last_change":"2024-03-18T19:08:19.164485+0000","last_active":"2024-03-18T19:08:19.591816+0000","last_peered":"2024-03-18T19:08:19.591816+0000","last_clean":"2024-03-18T19:08:19.591816+0000","last_became_active":"2024-03-18T19:08:19.164217+0000","last_became_peered":"2024-03-18T19:08:19.164217+0000","last_unstale":"2024-03-18T19:08:19.591816+0000","last_undegraded":"2024-03-18T19:08:19.591816+0000","last_fullsized":"2024-03-18T19:08:19.591816+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:31.789228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590632+0000","last_change":"2024-03-18T19:08:19.602044+0000","last_active":"2024-03-18T19:08:20.590632+0000","last_peered":"2024-03-18T19:08:20.590632+0000","last_clean":"2024-03-18T19:08:20.590632+0000","last_became_active":"2024-03-18T19:08:19.601277+0000","last_became_peered":"2024-03-18T19:08:19.601277+0000","last_unstale":"2024-03-18T19:08:20.590632+0000","last_undegraded":"2024-03-18T19:08:20.590632+0000","last_fullsized":"2024-03-18T19:08:20.590632+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:23:07.262496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590305+0000","last_change":"2024-03-18T19:08:19.170497+0000","last_active":"2024-03-18T19:08:20.590305+0000","last_peered":"2024-03-18T19:08:20.590305+0000","last_clean":"2024-03-18T19:08:20.590305+0000","last_became_active":"2024-03-18T19:08:19.170284+0000","last_became_peered":"2024-03-18T19:08:19.170284+0000","last_unstale":"2024-03-18T19:08:20.590305+0000","last_undegraded":"2024-03-18T19:08:20.590305+0000","last_fullsized":"2024-03-18T19:08:20.590305+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:57.194723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536144+0000","last_change":"2024-03-18T19:08:19.606838+0000","last_active":"2024-03-18T19:08:21.536144+0000","last_peered":"2024-03-18T19:08:21.536144+0000","last_clean":"2024-03-18T19:08:21.536144+0000","last_became_active":"2024-03-18T19:08:19.606257+0000","last_became_peered":"2024-03-18T19:08:19.606257+0000","last_unstale":"2024-03-18T19:08:21.536144+0000","last_undegraded":"2024-03-18T19:08:21.536144+0000","last_fullsized":"2024-03-18T19:08:21.536144+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:40:11.381260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.591378+0000","last_change":"2024-03-18T19:08:19.168791+0000","last_active":"2024-03-18T19:08:19.591378+0000","last_peered":"2024-03-18T19:08:19.591378+0000","last_clean":"2024-03-18T19:08:19.591378+0000","last_became_active":"2024-03-18T19:08:19.168522+0000","last_became_peered":"2024-03-18T19:08:19.168522+0000","last_unstale":"2024-03-18T19:08:19.591378+0000","last_undegraded":"2024-03-18T19:08:19.591378+0000","last_fullsized":"2024-03-18T19:08:19.591378+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:36:40.206825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589461+0000","last_change":"2024-03-18T19:08:19.605621+0000","last_active":"2024-03-18T19:08:20.589461+0000","last_peered":"2024-03-18T19:08:20.589461+0000","last_clean":"2024-03-18T19:08:20.589461+0000","last_became_active":"2024-03-18T19:08:19.605185+0000","last_became_peered":"2024-03-18T19:08:19.605185+0000","last_unstale":"2024-03-18T19:08:20.589461+0000","last_undegraded":"2024-03-18T19:08:20.589461+0000","last_fullsized":"2024-03-18T19:08:20.589461+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:23:14.545049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536146+0000","last_change":"2024-03-18T19:08:19.169385+0000","last_active":"2024-03-18T19:08:21.536146+0000","last_peered":"2024-03-18T19:08:21.536146+0000","last_clean":"2024-03-18T19:08:21.536146+0000","last_became_active":"2024-03-18T19:08:19.169106+0000","last_became_peered":"2024-03-18T19:08:19.169106+0000","last_unstale":"2024-03-18T19:08:21.536146+0000","last_undegraded":"2024-03-18T19:08:21.536146+0000","last_fullsized":"2024-03-18T19:08:21.536146+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:10:26.147698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.608175+0000","last_change":"2024-03-18T19:08:19.605584+0000","last_active":"2024-03-18T19:08:19.608175+0000","last_peered":"2024-03-18T19:08:19.608175+0000","last_clean":"2024-03-18T19:08:19.608175+0000","last_became_active":"2024-03-18T19:08:19.605356+0000","last_became_peered":"2024-03-18T19:08:19.605356+0000","last_unstale":"2024-03-18T19:08:19.608175+0000","last_undegraded":"2024-03-18T19:08:19.608175+0000","last_fullsized":"2024-03-18T19:08:19.608175+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:40:40.883829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"132'3","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.604670+0000","last_change":"2024-03-18T19:08:19.166809+0000","last_active":"2024-03-18T19:08:19.604670+0000","last_peered":"2024-03-18T19:08:19.604670+0000","last_clean":"2024-03-18T19:08:19.604670+0000","last_became_active":"2024-03-18T19:08:19.166512+0000","last_became_peered":"2024-03-18T19:08:19.166512+0000","last_unstale":"2024-03-18T19:08:19.604670+0000","last_undegraded":"2024-03-18T19:08:19.604670+0000","last_fullsized":"2024-03-18T19:08:19.604670+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:33.982221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589515+0000","last_change":"2024-03-18T19:08:19.612111+0000","last_active":"2024-03-18T19:08:20.589515+0000","last_peered":"2024-03-18T19:08:20.589515+0000","last_clean":"2024-03-18T19:08:20.589515+0000","last_became_active":"2024-03-18T19:08:19.611853+0000","last_became_peered":"2024-03-18T19:08:19.611853+0000","last_unstale":"2024-03-18T19:08:20.589515+0000","last_undegraded":"2024-03-18T19:08:20.589515+0000","last_fullsized":"2024-03-18T19:08:20.589515+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:49:04.996335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"132'6","reported_seq":17,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735406+0000","last_change":"2024-03-18T19:08:19.164339+0000","last_active":"2024-03-18T19:08:20.735406+0000","last_peered":"2024-03-18T19:08:20.735406+0000","last_clean":"2024-03-18T19:08:20.735406+0000","last_became_active":"2024-03-18T19:08:19.164062+0000","last_became_peered":"2024-03-18T19:08:19.164062+0000","last_unstale":"2024-03-18T19:08:20.735406+0000","last_undegraded":"2024-03-18T19:08:20.735406+0000","last_fullsized":"2024-03-18T19:08:20.735406+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:47:15.297314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.617049+0000","last_change":"2024-03-18T19:08:19.616275+0000","last_active":"2024-03-18T19:08:19.617049+0000","last_peered":"2024-03-18T19:08:19.617049+0000","last_clean":"2024-03-18T19:08:19.617049+0000","last_became_active":"2024-03-18T19:08:19.616022+0000","last_became_peered":"2024-03-18T19:08:19.616022+0000","last_unstale":"2024-03-18T19:08:19.617049+0000","last_undegraded":"2024-03-18T19:08:19.617049+0000","last_fullsized":"2024-03-18T19:08:19.617049+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:11.705994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"132'2","reported_seq":13,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589684+0000","last_change":"2024-03-18T19:08:19.165654+0000","last_active":"2024-03-18T19:08:20.589684+0000","last_peered":"2024-03-18T19:08:20.589684+0000","last_clean":"2024-03-18T19:08:20.589684+0000","last_became_active":"2024-03-18T19:08:19.165394+0000","last_became_peered":"2024-03-18T19:08:19.165394+0000","last_unstale":"2024-03-18T19:08:20.589684+0000","last_undegraded":"2024-03-18T19:08:20.589684+0000","last_fullsized":"2024-03-18T19:08:20.589684+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:33.982221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735247+0000","last_change":"2024-03-18T19:08:19.603045+0000","last_active":"2024-03-18T19:08:20.735247+0000","last_peered":"2024-03-18T19:08:20.735247+0000","last_clean":"2024-03-18T19:08:20.735247+0000","last_became_active":"2024-03-18T19:08:19.602799+0000","last_became_peered":"2024-03-18T19:08:19.602799+0000","last_unstale":"2024-03-18T19:08:20.735247+0000","last_undegraded":"2024-03-18T19:08:20.735247+0000","last_fullsized":"2024-03-18T19:08:20.735247+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:51.825688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590559+0000","last_change":"2024-03-18T19:08:19.169402+0000","last_active":"2024-03-18T19:08:20.590559+0000","last_peered":"2024-03-18T19:08:20.590559+0000","last_clean":"2024-03-18T19:08:20.590559+0000","last_became_active":"2024-03-18T19:08:19.169108+0000","last_became_peered":"2024-03-18T19:08:19.169108+0000","last_unstale":"2024-03-18T19:08:20.590559+0000","last_undegraded":"2024-03-18T19:08:20.590559+0000","last_fullsized":"2024-03-18T19:08:20.590559+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:19:46.814673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.612135+0000","last_change":"2024-03-18T19:08:19.611243+0000","last_active":"2024-03-18T19:08:19.612135+0000","last_peered":"2024-03-18T19:08:19.612135+0000","last_clean":"2024-03-18T19:08:19.612135+0000","last_became_active":"2024-03-18T19:08:19.611001+0000","last_became_peered":"2024-03-18T19:08:19.611001+0000","last_unstale":"2024-03-18T19:08:19.612135+0000","last_undegraded":"2024-03-18T19:08:19.612135+0000","last_fullsized":"2024-03-18T19:08:19.612135+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:16:53.833044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.601446+0000","last_change":"2024-03-18T19:08:19.170909+0000","last_active":"2024-03-18T19:08:19.601446+0000","last_peered":"2024-03-18T19:08:19.601446+0000","last_clean":"2024-03-18T19:08:19.601446+0000","last_became_active":"2024-03-18T19:08:19.168675+0000","last_became_peered":"2024-03-18T19:08:19.168675+0000","last_unstale":"2024-03-18T19:08:19.601446+0000","last_undegraded":"2024-03-18T19:08:19.601446+0000","last_fullsized":"2024-03-18T19:08:19.601446+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:18.748842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536546+0000","last_change":"2024-03-18T19:08:19.608133+0000","last_active":"2024-03-18T19:08:21.536546+0000","last_peered":"2024-03-18T19:08:21.536546+0000","last_clean":"2024-03-18T19:08:21.536546+0000","last_became_active":"2024-03-18T19:08:19.607531+0000","last_became_peered":"2024-03-18T19:08:19.607531+0000","last_unstale":"2024-03-18T19:08:21.536546+0000","last_undegraded":"2024-03-18T19:08:21.536546+0000","last_fullsized":"2024-03-18T19:08:21.536546+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:44:49.610735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"132'8","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.608768+0000","last_change":"2024-03-18T19:08:19.165160+0000","last_active":"2024-03-18T19:08:19.608768+0000","last_peered":"2024-03-18T19:08:19.608768+0000","last_clean":"2024-03-18T19:08:19.608768+0000","last_became_active":"2024-03-18T19:08:19.164883+0000","last_became_peered":"2024-03-18T19:08:19.164883+0000","last_unstale":"2024-03-18T19:08:19.608768+0000","last_undegraded":"2024-03-18T19:08:19.608768+0000","last_fullsized":"2024-03-18T19:08:19.608768+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:45.110724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.616173+0000","last_change":"2024-03-18T19:08:19.608700+0000","last_active":"2024-03-18T19:08:19.616173+0000","last_peered":"2024-03-18T19:08:19.616173+0000","last_clean":"2024-03-18T19:08:19.616173+0000","last_became_active":"2024-03-18T19:08:19.608373+0000","last_became_peered":"2024-03-18T19:08:19.608373+0000","last_unstale":"2024-03-18T19:08:19.616173+0000","last_undegraded":"2024-03-18T19:08:19.616173+0000","last_fullsized":"2024-03-18T19:08:19.616173+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:29:17.927761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"132'1","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735398+0000","last_change":"2024-03-18T19:08:19.168227+0000","last_active":"2024-03-18T19:08:20.735398+0000","last_peered":"2024-03-18T19:08:20.735398+0000","last_clean":"2024-03-18T19:08:20.735398+0000","last_became_active":"2024-03-18T19:08:19.167990+0000","last_became_peered":"2024-03-18T19:08:19.167990+0000","last_unstale":"2024-03-18T19:08:20.735398+0000","last_undegraded":"2024-03-18T19:08:20.735398+0000","last_fullsized":"2024-03-18T19:08:20.735398+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:16:52.827972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735232+0000","last_change":"2024-03-18T19:08:19.605813+0000","last_active":"2024-03-18T19:08:20.735232+0000","last_peered":"2024-03-18T19:08:20.735232+0000","last_clean":"2024-03-18T19:08:20.735232+0000","last_became_active":"2024-03-18T19:08:19.605536+0000","last_became_peered":"2024-03-18T19:08:19.605536+0000","last_unstale":"2024-03-18T19:08:20.735232+0000","last_undegraded":"2024-03-18T19:08:20.735232+0000","last_fullsized":"2024-03-18T19:08:20.735232+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:31.979650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"132'1","reported_seq":11,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.607027+0000","last_change":"2024-03-18T19:08:19.168439+0000","last_active":"2024-03-18T19:08:19.607027+0000","last_peered":"2024-03-18T19:08:19.607027+0000","last_clean":"2024-03-18T19:08:19.607027+0000","last_became_active":"2024-03-18T19:08:19.168119+0000","last_became_peered":"2024-03-18T19:08:19.168119+0000","last_unstale":"2024-03-18T19:08:19.607027+0000","last_undegraded":"2024-03-18T19:08:19.607027+0000","last_fullsized":"2024-03-18T19:08:19.607027+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:57.747186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589492+0000","last_change":"2024-03-18T19:08:19.594726+0000","last_active":"2024-03-18T19:08:20.589492+0000","last_peered":"2024-03-18T19:08:20.589492+0000","last_clean":"2024-03-18T19:08:20.589492+0000","last_became_active":"2024-03-18T19:08:19.594469+0000","last_became_peered":"2024-03-18T19:08:19.594469+0000","last_unstale":"2024-03-18T19:08:20.589492+0000","last_undegraded":"2024-03-18T19:08:20.589492+0000","last_fullsized":"2024-03-18T19:08:20.589492+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:54.114622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.596127+0000","last_change":"2024-03-18T19:08:19.164565+0000","last_active":"2024-03-18T19:08:19.596127+0000","last_peered":"2024-03-18T19:08:19.596127+0000","last_clean":"2024-03-18T19:08:19.596127+0000","last_became_active":"2024-03-18T19:08:19.164307+0000","last_became_peered":"2024-03-18T19:08:19.164307+0000","last_unstale":"2024-03-18T19:08:19.596127+0000","last_undegraded":"2024-03-18T19:08:19.596127+0000","last_fullsized":"2024-03-18T19:08:19.596127+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:57.747186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536236+0000","last_change":"2024-03-18T19:08:19.603564+0000","last_active":"2024-03-18T19:08:21.536236+0000","last_peered":"2024-03-18T19:08:21.536236+0000","last_clean":"2024-03-18T19:08:21.536236+0000","last_became_active":"2024-03-18T19:08:19.603281+0000","last_became_peered":"2024-03-18T19:08:19.603281+0000","last_unstale":"2024-03-18T19:08:21.536236+0000","last_undegraded":"2024-03-18T19:08:21.536236+0000","last_fullsized":"2024-03-18T19:08:21.536236+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:17.394276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"132'2","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.618024+0000","last_change":"2024-03-18T19:08:19.171553+0000","last_active":"2024-03-18T19:08:19.618024+0000","last_peered":"2024-03-18T19:08:19.618024+0000","last_clean":"2024-03-18T19:08:19.618024+0000","last_became_active":"2024-03-18T19:08:19.168493+0000","last_became_peered":"2024-03-18T19:08:19.168493+0000","last_unstale":"2024-03-18T19:08:19.618024+0000","last_undegraded":"2024-03-18T19:08:19.618024+0000","last_fullsized":"2024-03-18T19:08:19.618024+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:53.557992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536435+0000","last_change":"2024-03-18T19:08:19.601793+0000","last_active":"2024-03-18T19:08:21.536435+0000","last_peered":"2024-03-18T19:08:21.536435+0000","last_clean":"2024-03-18T19:08:21.536435+0000","last_became_active":"2024-03-18T19:08:19.601489+0000","last_became_peered":"2024-03-18T19:08:19.601489+0000","last_unstale":"2024-03-18T19:08:21.536435+0000","last_undegraded":"2024-03-18T19:08:21.536435+0000","last_fullsized":"2024-03-18T19:08:21.536435+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:09:30.271851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.601566+0000","last_change":"2024-03-18T19:08:19.171812+0000","last_active":"2024-03-18T19:08:19.601566+0000","last_peered":"2024-03-18T19:08:19.601566+0000","last_clean":"2024-03-18T19:08:19.601566+0000","last_became_active":"2024-03-18T19:08:19.171520+0000","last_became_peered":"2024-03-18T19:08:19.171520+0000","last_unstale":"2024-03-18T19:08:19.601566+0000","last_undegraded":"2024-03-18T19:08:19.601566+0000","last_fullsized":"2024-03-18T19:08:19.601566+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:03:24.832833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590711+0000","last_change":"2024-03-18T19:08:19.615504+0000","last_active":"2024-03-18T19:08:20.590711+0000","last_peered":"2024-03-18T19:08:20.590711+0000","last_clean":"2024-03-18T19:08:20.590711+0000","last_became_active":"2024-03-18T19:08:19.615260+0000","last_became_peered":"2024-03-18T19:08:19.615260+0000","last_unstale":"2024-03-18T19:08:20.590711+0000","last_undegraded":"2024-03-18T19:08:20.590711+0000","last_fullsized":"2024-03-18T19:08:20.590711+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:21.361005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.734959+0000","last_change":"2024-03-18T19:08:19.163683+0000","last_active":"2024-03-18T19:08:20.734959+0000","last_peered":"2024-03-18T19:08:20.734959+0000","last_clean":"2024-03-18T19:08:20.734959+0000","last_became_active":"2024-03-18T19:08:19.163414+0000","last_became_peered":"2024-03-18T19:08:19.163414+0000","last_unstale":"2024-03-18T19:08:20.734959+0000","last_undegraded":"2024-03-18T19:08:20.734959+0000","last_fullsized":"2024-03-18T19:08:20.734959+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:58:12.832970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.608204+0000","last_change":"2024-03-18T19:08:19.605932+0000","last_active":"2024-03-18T19:08:19.608204+0000","last_peered":"2024-03-18T19:08:19.608204+0000","last_clean":"2024-03-18T19:08:19.608204+0000","last_became_active":"2024-03-18T19:08:19.602651+0000","last_became_peered":"2024-03-18T19:08:19.602651+0000","last_unstale":"2024-03-18T19:08:19.608204+0000","last_undegraded":"2024-03-18T19:08:19.608204+0000","last_fullsized":"2024-03-18T19:08:19.608204+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:07:58.870811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.601302+0000","last_change":"2024-03-18T19:08:19.171795+0000","last_active":"2024-03-18T19:08:19.601302+0000","last_peered":"2024-03-18T19:08:19.601302+0000","last_clean":"2024-03-18T19:08:19.601302+0000","last_became_active":"2024-03-18T19:08:19.171464+0000","last_became_peered":"2024-03-18T19:08:19.171464+0000","last_unstale":"2024-03-18T19:08:19.601302+0000","last_undegraded":"2024-03-18T19:08:19.601302+0000","last_fullsized":"2024-03-18T19:08:19.601302+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:54:22.449223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536086+0000","last_change":"2024-03-18T19:08:19.600863+0000","last_active":"2024-03-18T19:08:21.536086+0000","last_peered":"2024-03-18T19:08:21.536086+0000","last_clean":"2024-03-18T19:08:21.536086+0000","last_became_active":"2024-03-18T19:08:19.600608+0000","last_became_peered":"2024-03-18T19:08:19.600608+0000","last_unstale":"2024-03-18T19:08:21.536086+0000","last_undegraded":"2024-03-18T19:08:21.536086+0000","last_fullsized":"2024-03-18T19:08:21.536086+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:59.796154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"132'2","reported_seq":13,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590032+0000","last_change":"2024-03-18T19:08:19.165314+0000","last_active":"2024-03-18T19:08:20.590032+0000","last_peered":"2024-03-18T19:08:20.590032+0000","last_clean":"2024-03-18T19:08:20.590032+0000","last_became_active":"2024-03-18T19:08:19.165036+0000","last_became_peered":"2024-03-18T19:08:19.165036+0000","last_unstale":"2024-03-18T19:08:20.590032+0000","last_undegraded":"2024-03-18T19:08:20.590032+0000","last_fullsized":"2024-03-18T19:08:20.590032+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:57.137780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.603953+0000","last_change":"2024-03-18T19:08:19.600746+0000","last_active":"2024-03-18T19:08:19.603953+0000","last_peered":"2024-03-18T19:08:19.603953+0000","last_clean":"2024-03-18T19:08:19.603953+0000","last_became_active":"2024-03-18T19:08:19.600483+0000","last_became_peered":"2024-03-18T19:08:19.600483+0000","last_unstale":"2024-03-18T19:08:19.603953+0000","last_undegraded":"2024-03-18T19:08:19.603953+0000","last_fullsized":"2024-03-18T19:08:19.603953+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:58.199795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:21.536300+0000","last_change":"2024-03-18T19:08:19.164044+0000","last_active":"2024-03-18T19:08:21.536300+0000","last_peered":"2024-03-18T19:08:21.536300+0000","last_clean":"2024-03-18T19:08:21.536300+0000","last_became_active":"2024-03-18T19:08:19.163786+0000","last_became_peered":"2024-03-18T19:08:19.163786+0000","last_unstale":"2024-03-18T19:08:21.536300+0000","last_undegraded":"2024-03-18T19:08:21.536300+0000","last_fullsized":"2024-03-18T19:08:21.536300+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:04.663910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":10,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.590475+0000","last_change":"2024-03-18T19:08:19.601877+0000","last_active":"2024-03-18T19:08:20.590475+0000","last_peered":"2024-03-18T19:08:20.590475+0000","last_clean":"2024-03-18T19:08:20.590475+0000","last_became_active":"2024-03-18T19:08:19.601628+0000","last_became_peered":"2024-03-18T19:08:19.601628+0000","last_unstale":"2024-03-18T19:08:20.590475+0000","last_undegraded":"2024-03-18T19:08:20.590475+0000","last_fullsized":"2024-03-18T19:08:20.590475+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:35.010798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735302+0000","last_change":"2024-03-18T19:08:19.165924+0000","last_active":"2024-03-18T19:08:20.735302+0000","last_peered":"2024-03-18T19:08:20.735302+0000","last_clean":"2024-03-18T19:08:20.735302+0000","last_became_active":"2024-03-18T19:08:19.165670+0000","last_became_peered":"2024-03-18T19:08:19.165670+0000","last_unstale":"2024-03-18T19:08:20.735302+0000","last_undegraded":"2024-03-18T19:08:20.735302+0000","last_fullsized":"2024-03-18T19:08:20.735302+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:50:20.781356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.613877+0000","last_change":"2024-03-18T19:08:19.613090+0000","last_active":"2024-03-18T19:08:19.613877+0000","last_peered":"2024-03-18T19:08:19.613877+0000","last_clean":"2024-03-18T19:08:19.613877+0000","last_became_active":"2024-03-18T19:08:19.612803+0000","last_became_peered":"2024-03-18T19:08:19.612803+0000","last_unstale":"2024-03-18T19:08:19.613877+0000","last_undegraded":"2024-03-18T19:08:19.613877+0000","last_fullsized":"2024-03-18T19:08:19.613877+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:12:11.205891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.589821+0000","last_change":"2024-03-18T19:08:19.161643+0000","last_active":"2024-03-18T19:08:20.589821+0000","last_peered":"2024-03-18T19:08:20.589821+0000","last_clean":"2024-03-18T19:08:20.589821+0000","last_became_active":"2024-03-18T19:08:19.161320+0000","last_became_peered":"2024-03-18T19:08:19.161320+0000","last_unstale":"2024-03-18T19:08:20.589821+0000","last_undegraded":"2024-03-18T19:08:20.589821+0000","last_fullsized":"2024-03-18T19:08:20.589821+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:19:58.390893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.608119+0000","last_change":"2024-03-18T19:08:19.606097+0000","last_active":"2024-03-18T19:08:19.608119+0000","last_peered":"2024-03-18T19:08:19.608119+0000","last_clean":"2024-03-18T19:08:19.608119+0000","last_became_active":"2024-03-18T19:08:19.602719+0000","last_became_peered":"2024-03-18T19:08:19.602719+0000","last_unstale":"2024-03-18T19:08:19.608119+0000","last_undegraded":"2024-03-18T19:08:19.608119+0000","last_fullsized":"2024-03-18T19:08:19.608119+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:09:36.235776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.591082+0000","last_change":"2024-03-18T19:08:19.170578+0000","last_active":"2024-03-18T19:08:19.591082+0000","last_peered":"2024-03-18T19:08:19.591082+0000","last_clean":"2024-03-18T19:08:19.591082+0000","last_became_active":"2024-03-18T19:08:19.170332+0000","last_became_peered":"2024-03-18T19:08:19.170332+0000","last_unstale":"2024-03-18T19:08:19.591082+0000","last_undegraded":"2024-03-18T19:08:19.591082+0000","last_fullsized":"2024-03-18T19:08:19.591082+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:06:13.807256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.608146+0000","last_change":"2024-03-18T19:08:19.605113+0000","last_active":"2024-03-18T19:08:19.608146+0000","last_peered":"2024-03-18T19:08:19.608146+0000","last_clean":"2024-03-18T19:08:19.608146+0000","last_became_active":"2024-03-18T19:08:19.604849+0000","last_became_peered":"2024-03-18T19:08:19.604849+0000","last_unstale":"2024-03-18T19:08:19.608146+0000","last_undegraded":"2024-03-18T19:08:19.608146+0000","last_fullsized":"2024-03-18T19:08:19.608146+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:31.326003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.601649+0000","last_change":"2024-03-18T19:08:19.171031+0000","last_active":"2024-03-18T19:08:19.601649+0000","last_peered":"2024-03-18T19:08:19.601649+0000","last_clean":"2024-03-18T19:08:19.601649+0000","last_became_active":"2024-03-18T19:08:19.170750+0000","last_became_peered":"2024-03-18T19:08:19.170750+0000","last_unstale":"2024-03-18T19:08:19.601649+0000","last_undegraded":"2024-03-18T19:08:19.601649+0000","last_fullsized":"2024-03-18T19:08:19.601649+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:53.730763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.605665+0000","last_change":"2024-03-18T19:08:19.604372+0000","last_active":"2024-03-18T19:08:19.605665+0000","last_peered":"2024-03-18T19:08:19.605665+0000","last_clean":"2024-03-18T19:08:19.605665+0000","last_became_active":"2024-03-18T19:08:19.604098+0000","last_became_peered":"2024-03-18T19:08:19.604098+0000","last_unstale":"2024-03-18T19:08:19.605665+0000","last_undegraded":"2024-03-18T19:08:19.605665+0000","last_fullsized":"2024-03-18T19:08:19.605665+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:32.794300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-18T19:08:20.735279+0000","last_change":"2024-03-18T19:08:19.166840+0000","last_active":"2024-03-18T19:08:20.735279+0000","last_peered":"2024-03-18T19:08:20.735279+0000","last_clean":"2024-03-18T19:08:20.735279+0000","last_became_active":"2024-03-18T19:08:19.166520+0000","last_became_peered":"2024-03-18T19:08:19.166520+0000","last_unstale":"2024-03-18T19:08:20.735279+0000","last_undegraded":"2024-03-18T19:08:20.735279+0000","last_fullsized":"2024-03-18T19:08:20.735279+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:55:38.817937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.607881+0000","last_change":"2024-03-18T19:08:19.606522+0000","last_active":"2024-03-18T19:08:19.607881+0000","last_peered":"2024-03-18T19:08:19.607881+0000","last_clean":"2024-03-18T19:08:19.607881+0000","last_became_active":"2024-03-18T19:08:19.606269+0000","last_became_peered":"2024-03-18T19:08:19.606269+0000","last_unstale":"2024-03-18T19:08:19.607881+0000","last_undegraded":"2024-03-18T19:08:19.607881+0000","last_fullsized":"2024-03-18T19:08:19.607881+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:51.825688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":10,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.601181+0000","last_change":"2024-03-18T19:08:19.171148+0000","last_active":"2024-03-18T19:08:19.601181+0000","last_peered":"2024-03-18T19:08:19.601181+0000","last_clean":"2024-03-18T19:08:19.601181+0000","last_became_active":"2024-03-18T19:08:19.170726+0000","last_became_peered":"2024-03-18T19:08:19.170726+0000","last_unstale":"2024-03-18T19:08:19.601181+0000","last_undegraded":"2024-03-18T19:08:19.601181+0000","last_fullsized":"2024-03-18T19:08:19.601181+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:52:04.509926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.616474+0000","last_change":"2024-03-18T19:08:19.614738+0000","last_active":"2024-03-18T19:08:19.616474+0000","last_peered":"2024-03-18T19:08:19.616474+0000","last_clean":"2024-03-18T19:08:19.616474+0000","last_became_active":"2024-03-18T19:08:19.614329+0000","last_became_peered":"2024-03-18T19:08:19.614329+0000","last_unstale":"2024-03-18T19:08:19.616474+0000","last_undegraded":"2024-03-18T19:08:19.616474+0000","last_fullsized":"2024-03-18T19:08:19.616474+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:19.144487+0000","last_clean_scrub_stamp":"2024-03-18T19:08:19.144487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:49:04.996335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"132'2","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.618117+0000","last_change":"2024-03-18T19:08:19.171491+0000","last_active":"2024-03-18T19:08:19.618117+0000","last_peered":"2024-03-18T19:08:19.618117+0000","last_clean":"2024-03-18T19:08:19.618117+0000","last_became_active":"2024-03-18T19:08:19.171212+0000","last_became_peered":"2024-03-18T19:08:19.171212+0000","last_unstale":"2024-03-18T19:08:19.618117+0000","last_undegraded":"2024-03-18T19:08:19.618117+0000","last_fullsized":"2024-03-18T19:08:19.618117+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:08:18.139415+0000","last_clean_scrub_stamp":"2024-03-18T19:08:18.139415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:41:01.232974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"122'205","reported_seq":575,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-18T19:08:19.153271+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T19:08:19.153271+0000","last_peered":"2024-03-18T19:08:19.153271+0000","last_clean":"2024-03-18T19:08:19.153271+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T19:08:19.153271+0000","last_undegraded":"2024-03-18T19:08:19.153271+0000","last_fullsized":"2024-03-18T19:08:19.153271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575066,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59548,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312292,"statfs":{"total":96636764160,"available":96575787008,"internally_reserved":0,"allocated":2236416,"data_stored":1411129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56200000000000006,"15min":0.57099999999999995},"min":{"1min":0.39600000000000002,"5min":0.35799999999999998,"15min":0.27800000000000002},"max":{"1min":0.78400000000000003,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55700000000000005,"15min":0.58199999999999996},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.318},"max":{"1min":0.70699999999999996,"5min":1.387,"15min":1.387},"last":0.39600000000000002}]},{"osd":1,"last update":"Mon Mar 18 19:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54200000000000004,"15min":0.57399999999999995},"min":{"1min":0.373,"5min":0.33200000000000002,"15min":0.307},"max":{"1min":0.74099999999999999,"5min":0.81699999999999995,"15min":1.484},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55100000000000005,"15min":0.58799999999999997},"min":{"1min":0.376,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67600000000000005,"5min":0.80700000000000005,"15min":0.92000000000000004},"last":0.497}]},{"osd":2,"last update":"Mon Mar 18 19:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.55000000000000004,"15min":0.57299999999999995},"min":{"1min":0.35099999999999998,"5min":0.34799999999999998,"15min":0.27800000000000002},"max":{"1min":0.70999999999999996,"5min":1.3600000000000001,"15min":1.5129999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56999999999999995,"15min":0.59299999999999997},"min":{"1min":0.372,"5min":0.35999999999999999,"15min":0.35299999999999998},"max":{"1min":0.78100000000000003,"5min":1.327,"15min":1.536},"last":0.58499999999999996}]},{"osd":3,"last update":"Mon Mar 18 19:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55200000000000005,"15min":0.57899999999999996},"min":{"1min":0.38900000000000001,"5min":0.35599999999999998,"15min":0.311},"max":{"1min":0.749,"5min":0.79100000000000004,"15min":1.5489999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56699999999999995,"15min":0.59899999999999998},"min":{"1min":0.44400000000000001,"5min":0.38600000000000001,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":1.5660000000000001},"last":0.629}]},{"osd":4,"last update":"Mon Mar 18 19:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.44600000000000001,"15min":0.45100000000000001},"min":{"1min":0.33300000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.67300000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.443,"5min":0.45500000000000002,"15min":0.47199999999999998},"min":{"1min":0.34599999999999997,"5min":0.27500000000000002,"15min":0.25800000000000001},"max":{"1min":0.56899999999999995,"5min":0.71599999999999997,"15min":0.72599999999999998},"last":0.55200000000000005}]},{"osd":5,"last update":"Mon Mar 18 19:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.47499999999999998,"15min":0.47599999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.26000000000000001},"max":{"1min":0.66000000000000003,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.50600000000000001,"15min":0.51400000000000001},"min":{"1min":0.35899999999999999,"5min":0.30299999999999999,"15min":0.28799999999999998},"max":{"1min":0.71499999999999997,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.56899999999999995}]},{"osd":6,"last update":"Mon Mar 18 19:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52400000000000002,"15min":0.54100000000000004},"min":{"1min":0.38,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.74099999999999999,"5min":0.81799999999999995,"15min":0.84499999999999997},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.52100000000000002,"15min":0.52800000000000002},"min":{"1min":0.311,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.64800000000000002,"5min":0.74099999999999999,"15min":0.77500000000000002},"last":0.68200000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575066,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32756,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339084,"statfs":{"total":96636764160,"available":96603222016,"internally_reserved":0,"allocated":1667072,"data_stored":829389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54500000000000004,"15min":0.56100000000000005},"min":{"1min":0.40799999999999997,"5min":0.31900000000000001,"15min":0.28499999999999998},"max":{"1min":0.73699999999999999,"5min":0.749,"15min":0.76800000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54700000000000004,"15min":0.56799999999999995},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.27400000000000002},"max":{"1min":0.70999999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.57699999999999996}]},{"osd":1,"last update":"Mon Mar 18 19:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54900000000000004,"15min":0.56399999999999995},"min":{"1min":0.39200000000000002,"5min":0.33700000000000002,"15min":0.316},"max":{"1min":0.71099999999999997,"5min":0.76700000000000002,"15min":0.77400000000000002},"last":0.628},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55600000000000005,"15min":0.56799999999999995},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32000000000000001},"max":{"1min":0.72299999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.72199999999999998}]},{"osd":2,"last update":"Mon Mar 18 19:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56299999999999994,"15min":0.57499999999999996},"min":{"1min":0.41199999999999998,"5min":0.32700000000000001,"15min":0.30499999999999999},"max":{"1min":0.78200000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.55900000000000005,"15min":0.57899999999999996},"min":{"1min":0.36899999999999999,"5min":0.35299999999999998,"15min":0.30299999999999999},"max":{"1min":0.751,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.70299999999999996}]},{"osd":3,"last update":"Mon Mar 18 19:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.49399999999999999,"5min":0.372,"15min":0.32100000000000001},"max":{"1min":0.72599999999999998,"5min":0.80700000000000005,"15min":0.89600000000000002},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56100000000000005,"15min":0.57499999999999996},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73899999999999999,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.68999999999999995}]},{"osd":4,"last update":"Mon Mar 18 19:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.46000000000000002,"15min":0.44700000000000001},"min":{"1min":0.317,"5min":0.26900000000000002,"15min":0.25600000000000001},"max":{"1min":0.65000000000000002,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48599999999999999,"15min":0.47499999999999998},"min":{"1min":0.35299999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.81699999999999995}]},{"osd":5,"last update":"Mon Mar 18 19:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.46999999999999997,"15min":0.46800000000000003},"min":{"1min":0.34499999999999997,"5min":0.27300000000000002,"15min":0.26900000000000002},"max":{"1min":0.67600000000000005,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.75},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.52800000000000002,"15min":0.51500000000000001},"min":{"1min":0.38500000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.753,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.879}]},{"osd":7,"last update":"Mon Mar 18 19:08:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.54100000000000004,"15min":0.52400000000000002},"min":{"1min":0.40500000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.76800000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.872},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55400000000000005,"15min":0.53100000000000003},"min":{"1min":0.39400000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.74399999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.78800000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575066,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36836,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335004,"statfs":{"total":96636764160,"available":96599044096,"internally_reserved":0,"allocated":1650688,"data_stored":827255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.36199999999999999,"15min":0.374},"min":{"1min":0.246,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.59099999999999997,"5min":0.63300000000000001,"15min":0.66300000000000003},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.41399999999999998},"min":{"1min":0.28399999999999997,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.60199999999999998,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Mon Mar 18 19:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.39700000000000002,"15min":0.40600000000000003},"min":{"1min":0.29299999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.55700000000000005,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.44500000000000001},"min":{"1min":0.29999999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.57499999999999996,"5min":0.66500000000000004,"15min":0.71699999999999997},"last":0.29599999999999999}]},{"osd":3,"last update":"Mon Mar 18 19:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44500000000000001,"15min":0.46500000000000002},"min":{"1min":0.29599999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.60799999999999998,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40400000000000003,"15min":0.41899999999999998},"min":{"1min":0.26900000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.53600000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.25900000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.60399999999999998,"15min":0.627},"min":{"1min":0.44600000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.84899999999999998,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.61499999999999999,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84599999999999997,"5min":1.018,"15min":1.018},"last":0.52700000000000002}]},{"osd":5,"last update":"Mon Mar 18 19:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.60399999999999998,"15min":0.63700000000000001},"min":{"1min":0.41199999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.872,"5min":0.89500000000000002,"15min":0.99199999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.629,"15min":0.65100000000000002},"min":{"1min":0.48299999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.90800000000000003,"5min":1.006,"15min":1.006},"last":0.55200000000000005}]},{"osd":6,"last update":"Mon Mar 18 19:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.60199999999999998,"15min":0.63300000000000001},"min":{"1min":0.42499999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83999999999999997,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64500000000000002,"15min":0.66600000000000004},"min":{"1min":0.42699999999999999,"5min":0.42199999999999999,"15min":0.39100000000000001},"max":{"1min":0.93600000000000005,"5min":0.96799999999999997,"15min":0.98099999999999998},"last":0.58199999999999996}]},{"osd":7,"last update":"Mon Mar 18 19:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.63500000000000001,"15min":0.65800000000000003},"min":{"1min":0.41299999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81000000000000005,"5min":0.92900000000000005,"15min":0.94799999999999995},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.63,"15min":0.65900000000000003},"min":{"1min":0.50900000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.53900000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575067,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37412,"kb_used_data":2192,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334428,"statfs":{"total":96636764160,"available":96598454272,"internally_reserved":0,"allocated":2244608,"data_stored":1412623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 19:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35799999999999998,"15min":0.378},"min":{"1min":0.251,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.52900000000000003,"5min":0.60499999999999998,"15min":0.621},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.39500000000000002,"15min":0.41399999999999998},"min":{"1min":0.33500000000000002,"5min":0.22700000000000001,"15min":0.20000000000000001},"max":{"1min":0.52400000000000002,"5min":0.57499999999999996,"15min":0.65100000000000002},"last":0.47499999999999998}]},{"osd":2,"last update":"Mon Mar 18 19:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.40699999999999997,"15min":0.41999999999999998},"min":{"1min":0.318,"5min":0.25600000000000001,"15min":0.21099999999999999},"max":{"1min":0.57699999999999996,"5min":0.60899999999999999,"15min":0.627},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.42499999999999999,"15min":0.437},"min":{"1min":0.318,"5min":0.25700000000000001,"15min":0.25},"max":{"1min":0.55500000000000005,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.39200000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39700000000000002,"15min":0.41499999999999998},"min":{"1min":0.28699999999999998,"5min":0.26500000000000001,"15min":0.217},"max":{"1min":0.58099999999999996,"5min":0.61299999999999999,"15min":0.64100000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.45600000000000002,"15min":0.47299999999999998},"min":{"1min":0.39000000000000001,"5min":0.32000000000000001,"15min":0.28799999999999998},"max":{"1min":0.62,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.51000000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.61599999999999999,"15min":0.64300000000000002},"min":{"1min":0.41299999999999998,"5min":0.33200000000000002,"15min":0.27200000000000002},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.61399999999999999,"15min":0.64300000000000002},"min":{"1min":0.42399999999999999,"5min":0.38200000000000001,"15min":0.315},"max":{"1min":0.91600000000000004,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.64100000000000001}]},{"osd":5,"last update":"Mon Mar 18 19:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63400000000000001,"15min":0.64900000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.34300000000000003},"max":{"1min":1.2030000000000001,"5min":1.248,"15min":1.248},"last":0.753},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.64700000000000002,"15min":0.66600000000000004},"min":{"1min":0.46999999999999997,"5min":0.38,"15min":0.31},"max":{"1min":1.2889999999999999,"5min":1.399,"15min":1.399},"last":0.624}]},{"osd":6,"last update":"Mon Mar 18 19:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.63500000000000001,"15min":0.65900000000000003},"min":{"1min":0.47899999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.1719999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.65500000000000003,"15min":0.67400000000000004},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.40999999999999998},"max":{"1min":1.2689999999999999,"5min":1.284,"15min":1.284},"last":0.71399999999999997}]},{"osd":7,"last update":"Mon Mar 18 19:08:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.65200000000000002,"15min":0.67100000000000004},"min":{"1min":0.443,"5min":0.38300000000000001,"15min":0.34899999999999998},"max":{"1min":1.2390000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.64400000000000002,"15min":0.67000000000000004},"min":{"1min":0.45000000000000001,"5min":0.42899999999999999,"15min":0.38500000000000001},"max":{"1min":0.86599999999999999,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.72599999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575067,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":1638400,"data_stored":820635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:08:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.36699999999999999,"15min":0.372},"min":{"1min":0.27800000000000002,"5min":0.19500000000000001,"15min":0.17799999999999999},"max":{"1min":0.58299999999999996,"5min":0.67400000000000004,"15min":0.78300000000000003},"last":0.25},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.45400000000000001,"15min":0.45700000000000002},"min":{"1min":0.32800000000000001,"5min":0.29399999999999998,"15min":0.217},"max":{"1min":0.68799999999999994,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.35499999999999998}]},{"osd":1,"last update":"Mon Mar 18 19:08:16 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45200000000000001,"15min":0.46800000000000003},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27100000000000002},"max":{"1min":0.60899999999999999,"5min":0.71899999999999997,"15min":0.90500000000000003},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.39200000000000002,"15min":0.39600000000000002},"min":{"1min":0.26300000000000001,"5min":0.22800000000000001,"15min":0.216},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.66700000000000004},"last":0.30399999999999999}]},{"osd":3,"last update":"Mon Mar 18 19:08:16 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.40600000000000003,"15min":0.41099999999999998},"min":{"1min":0.28000000000000003,"5min":0.247,"15min":0.22500000000000001},"max":{"1min":0.59399999999999997,"5min":0.65800000000000003,"15min":1.4019999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.48499999999999999,"15min":0.5},"min":{"1min":0.36699999999999999,"5min":0.27900000000000003,"15min":0.27700000000000002},"max":{"1min":0.70099999999999996,"5min":0.76400000000000001,"15min":0.80100000000000005},"last":0.438}]},{"osd":4,"last update":"Mon Mar 18 19:08:16 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61699999999999999},"min":{"1min":0.47899999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.71299999999999997,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60699999999999998,"15min":0.628},"min":{"1min":0.45000000000000001,"5min":0.36499999999999999,"15min":0.32100000000000001},"max":{"1min":0.78600000000000003,"5min":1.448,"15min":1.4730000000000001},"last":0.44800000000000001}]},{"osd":5,"last update":"Mon Mar 18 19:08:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.63800000000000001,"15min":0.64600000000000002},"min":{"1min":0.53900000000000003,"5min":0.39800000000000002,"15min":0.309},"max":{"1min":1.327,"5min":1.327,"15min":1.8109999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.626,"15min":0.63800000000000001},"min":{"1min":0.499,"5min":0.36899999999999999,"15min":0.33400000000000002},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.948},"last":0.56799999999999995}]},{"osd":6,"last update":"Mon Mar 18 19:08:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.63700000000000001,"15min":0.65800000000000003},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.35899999999999999},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":2.9470000000000001},"last":0.504},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68100000000000005,"15min":0.68500000000000005},"min":{"1min":0.53300000000000003,"5min":0.432,"15min":0.37},"max":{"1min":0.878,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.51200000000000001}]},{"osd":7,"last update":"Mon Mar 18 19:08:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.65100000000000002,"15min":0.65200000000000002},"min":{"1min":0.435,"5min":0.40999999999999998,"15min":0.38200000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.64500000000000002,"15min":0.66300000000000003},"min":{"1min":0.46400000000000002,"5min":0.38300000000000001,"15min":0.34599999999999997},"max":{"1min":0.80500000000000005,"5min":1.482,"15min":1.482},"last":0.47599999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575067,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36832,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335008,"statfs":{"total":96636764160,"available":96599048192,"internally_reserved":0,"allocated":1650688,"data_stored":827803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35299999999999998,"15min":0.379},"min":{"1min":0.22500000000000001,"5min":0.221,"15min":0.21099999999999999},"max":{"1min":0.51400000000000001,"5min":0.63900000000000001,"15min":1.204},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.377,"15min":0.40100000000000002},"min":{"1min":0.29799999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57399999999999995,"5min":0.66900000000000004,"15min":1.137},"last":0.28999999999999998}]},{"osd":1,"last update":"Mon Mar 18 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.374,"15min":0.39800000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.629,"5min":0.629,"15min":1.1220000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.40999999999999998,"15min":0.44500000000000001},"min":{"1min":0.33100000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":1.1080000000000001},"last":0.30299999999999999}]},{"osd":2,"last update":"Mon Mar 18 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.42399999999999999,"15min":0.45000000000000001},"min":{"1min":0.314,"5min":0.29499999999999998,"15min":0.28399999999999997},"max":{"1min":0.59799999999999998,"5min":0.68999999999999995,"15min":1.1879999999999999},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.44800000000000001,"15min":0.47799999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66600000000000004,"5min":0.71499999999999997,"15min":1.2290000000000001},"last":0.33900000000000002}]},{"osd":4,"last update":"Mon Mar 18 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57499999999999996,"15min":0.625},"min":{"1min":0.38100000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83399999999999996,"5min":1.052,"15min":1.871},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.58999999999999997,"15min":0.63200000000000001},"min":{"1min":0.47199999999999998,"5min":0.40300000000000002,"15min":0.39600000000000002},"max":{"1min":0.88100000000000001,"5min":0.90600000000000003,"15min":1.764},"last":0.40699999999999997}]},{"osd":5,"last update":"Mon Mar 18 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.624,"15min":0.66400000000000003},"min":{"1min":0.47699999999999998,"5min":0.39200000000000002,"15min":0.379},"max":{"1min":0.94199999999999995,"5min":1.0249999999999999,"15min":2.5329999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.64600000000000002,"15min":0.68500000000000005},"min":{"1min":0.48499999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.83299999999999996,"5min":0.94599999999999995,"15min":1.794},"last":0.47799999999999998}]},{"osd":6,"last update":"Mon Mar 18 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.628,"15min":0.67300000000000004},"min":{"1min":0.42499999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.88500000000000001,"5min":1.1060000000000001,"15min":2.7389999999999999},"last":0.496},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.64200000000000002,"15min":0.67900000000000005},"min":{"1min":0.50900000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.98199999999999998,"5min":0.98299999999999998,"15min":1.8069999999999999},"last":0.45000000000000001}]},{"osd":7,"last update":"Mon Mar 18 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63100000000000001,"15min":0.66900000000000004},"min":{"1min":0.45100000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.90000000000000002,"5min":0.97599999999999998,"15min":1.732},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.61899999999999999,"15min":0.67200000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.40799999999999997},"max":{"1min":0.874,"5min":1.085,"15min":2.7050000000000001},"last":0.51800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575067,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32728,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339112,"statfs":{"total":96636764160,"available":96603250688,"internally_reserved":0,"allocated":1642496,"data_stored":820693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54500000000000004,"15min":0.56899999999999995},"min":{"1min":0.35599999999999998,"5min":0.31,"15min":0.246},"max":{"1min":0.78800000000000003,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56799999999999995,"15min":0.58399999999999996},"min":{"1min":0.38900000000000001,"5min":0.35299999999999998,"15min":0.26000000000000001},"max":{"1min":0.748,"5min":3.645,"15min":3.645},"last":0.49299999999999999}]},{"osd":1,"last update":"Mon Mar 18 19:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.63200000000000001,"15min":0.60799999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76000000000000001,"5min":9.9499999999999993,"15min":9.9499999999999993},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53800000000000003,"15min":0.57399999999999995},"min":{"1min":0.34599999999999997,"5min":0.32300000000000001,"15min":0.308},"max":{"1min":0.73799999999999999,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon Mar 18 19:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54900000000000004,"15min":0.57199999999999995},"min":{"1min":0.39500000000000002,"5min":0.35899999999999999,"15min":0.28999999999999998},"max":{"1min":0.752,"5min":0.96299999999999997,"15min":1.0940000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.41999999999999998,"5min":0.35099999999999998,"15min":0.32600000000000001},"max":{"1min":0.79800000000000004,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.68100000000000005}]},{"osd":3,"last update":"Mon Mar 18 19:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55000000000000004,"15min":0.58199999999999996},"min":{"1min":0.40100000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.77000000000000002,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56100000000000005,"15min":0.59499999999999997},"min":{"1min":0.39900000000000002,"5min":0.39300000000000002,"15min":0.33900000000000002},"max":{"1min":0.77300000000000002,"5min":0.78200000000000003,"15min":1.121},"last":0.69899999999999995}]},{"osd":5,"last update":"Mon Mar 18 19:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.45900000000000002,"15min":0.46000000000000002},"min":{"1min":0.32600000000000001,"5min":0.254,"15min":0.245},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.77500000000000002},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.51100000000000001,"15min":0.49099999999999999},"min":{"1min":0.33500000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.78500000000000003,"5min":5.4900000000000002,"15min":5.4900000000000002},"last":0.629}]},{"osd":6,"last update":"Mon Mar 18 19:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.46800000000000003,"15min":0.47599999999999998},"min":{"1min":0.29899999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53500000000000003,"15min":0.52400000000000002},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.28999999999999998},"max":{"1min":0.82099999999999995,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":0.76200000000000001}]},{"osd":7,"last update":"Mon Mar 18 19:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51600000000000001,"15min":0.51500000000000001},"min":{"1min":0.374,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.74199999999999999,"5min":1.55,"15min":1.55},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.34699999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.73799999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575066,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50768,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321072,"statfs":{"total":96636764160,"available":96584777728,"internally_reserved":0,"allocated":1634304,"data_stored":820601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.54400000000000004,"15min":0.57199999999999995},"min":{"1min":0.36399999999999999,"5min":0.26800000000000002,"15min":0.26600000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":1.1679999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56299999999999994,"15min":0.57399999999999995},"min":{"1min":0.35299999999999998,"5min":0.33700000000000002,"15min":0.25800000000000001},"max":{"1min":0.69299999999999995,"5min":0.80600000000000005,"15min":1.1319999999999999},"last":0.58199999999999996}]},{"osd":1,"last update":"Mon Mar 18 19:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.54600000000000004,"15min":0.57399999999999995},"min":{"1min":0.34699999999999998,"5min":0.32600000000000001,"15min":0.29599999999999999},"max":{"1min":0.73899999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56699999999999995,"15min":0.58299999999999996},"min":{"1min":0.38200000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78400000000000003,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.56899999999999995}]},{"osd":2,"last update":"Mon Mar 18 19:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.55100000000000005,"15min":0.57699999999999996},"min":{"1min":0.377,"5min":0.30399999999999999,"15min":0.28399999999999997},"max":{"1min":0.66900000000000004,"5min":1.153,"15min":1.1579999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.55800000000000005,"15min":0.57599999999999996},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.33200000000000002},"max":{"1min":0.77100000000000002,"5min":0.79500000000000004,"15min":1.0569999999999999},"last":0.55200000000000005}]},{"osd":3,"last update":"Mon Mar 18 19:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.58699999999999997},"min":{"1min":0.38,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82799999999999996,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58599999999999997,"15min":0.60199999999999998},"min":{"1min":0.37,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.73199999999999998,"5min":1.212,"15min":1.212},"last":0.49299999999999999}]},{"osd":4,"last update":"Mon Mar 18 19:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.47499999999999998,"15min":0.46400000000000002},"min":{"1min":0.30399999999999999,"5min":0.255,"15min":0.24199999999999999},"max":{"1min":0.64100000000000001,"5min":1.248,"15min":1.248},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.503,"15min":0.48899999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.23100000000000001},"max":{"1min":0.71899999999999997,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.377}]},{"osd":6,"last update":"Mon Mar 18 19:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47399999999999998,"15min":0.47199999999999998},"min":{"1min":0.26100000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.67500000000000004,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51500000000000001,"15min":0.50800000000000001},"min":{"1min":0.34100000000000003,"5min":0.32100000000000001,"15min":0.26800000000000002},"max":{"1min":0.68899999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.47999999999999998}]},{"osd":7,"last update":"Mon Mar 18 19:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52700000000000002,"15min":0.52300000000000002},"min":{"1min":0.32500000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.748,"5min":1.282,"15min":1.282},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55300000000000005,"15min":0.54400000000000004},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28499999999999998},"max":{"1min":0.73999999999999999,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.44800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T19:08:24.136 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T19:08:24.137 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-18T19:08:24.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.308+0000 7f6a96670700 1 Processor -- start 2024-03-18T19:08:24.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.308+0000 7f6a96670700 1 -- start start 2024-03-18T19:08:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.309+0000 7f6a96670700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6a900cfa90 0x7f6a9006a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.309+0000 7f6a96670700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6a9006ac90 0x7f6a90167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.309+0000 7f6a96670700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a900cf600 con 0x7f6a9015d5a0 2024-03-18T19:08:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.309+0000 7f6a96670700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a9006b140 con 0x7f6a900cfa90 2024-03-18T19:08:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.309+0000 7f6a96670700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a900ce720 con 0x7f6a9006ac90 2024-03-18T19:08:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.309+0000 7f6a8f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6a900cfa90 0x7f6a9006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.309+0000 7f6a8f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6a900cfa90 0x7f6a9006a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59436/0 (socket says 172.21.15.1:59436) 2024-03-18T19:08:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.309+0000 7f6a8f7fe700 1 -- 172.21.15.1:0/3794963196 learned_addr learned my addr 172.21.15.1:0/3794963196 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.309+0000 7f6a94c0d700 1 --2- 172.21.15.1:0/3794963196 >> v2:172.21.15.165:3300/0 conn(0x7f6a9006ac90 0x7f6a90167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.309+0000 7f6a8f7fe700 1 -- 172.21.15.1:0/3794963196 >> v2:172.21.15.165:3300/0 conn(0x7f6a9006ac90 msgr2=0x7f6a90167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.309+0000 7f6a8f7fe700 1 --2- 172.21.15.1:0/3794963196 >> v2:172.21.15.165:3300/0 conn(0x7f6a9006ac90 0x7f6a90167140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.309+0000 7f6a8f7fe700 1 -- 172.21.15.1:0/3794963196 >> v1:172.21.15.1:6789/0 conn(0x7f6a9015d5a0 legacy=0x7f6a900ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.309+0000 7f6a8f7fe700 1 -- 172.21.15.1:0/3794963196 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a900ce9f0 con 0x7f6a900cfa90 2024-03-18T19:08:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.309+0000 7f6a94c0d700 1 --2- 172.21.15.1:0/3794963196 >> v2:172.21.15.165:3300/0 conn(0x7f6a9006ac90 0x7f6a90167140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.309+0000 7f6a8f7fe700 1 --2- 172.21.15.1:0/3794963196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a900cfa90 0x7f6a9006a750 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f6a900cdec0 tx=0x7f6a78009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82f5b18614df349c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.310+0000 7f6a8effd700 1 -- 172.21.15.1:0/3794963196 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7800b870 con 0x7f6a900cfa90 2024-03-18T19:08:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.310+0000 7f6a8effd700 1 -- 172.21.15.1:0/3794963196 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7800b9d0 con 0x7f6a900cfa90 2024-03-18T19:08:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.310+0000 7f6a8effd700 1 -- 172.21.15.1:0/3794963196 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7800bd60 con 0x7f6a900cfa90 2024-03-18T19:08:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.310+0000 7f6a96670700 1 -- 172.21.15.1:0/3794963196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a900cfa90 msgr2=0x7f6a9006a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.310+0000 7f6a96670700 1 --2- 172.21.15.1:0/3794963196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a900cfa90 0x7f6a9006a750 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f6a900cdec0 tx=0x7f6a78009bd0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.310+0000 7f6a96670700 1 -- 172.21.15.1:0/3794963196 shutdown_connections 2024-03-18T19:08:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.310+0000 7f6a96670700 1 --2- 172.21.15.1:0/3794963196 >> v2:172.21.15.165:3300/0 conn(0x7f6a9006ac90 0x7f6a90167140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.310+0000 7f6a96670700 1 --2- 172.21.15.1:0/3794963196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a900cfa90 0x7f6a9006a750 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.310+0000 7f6a96670700 1 -- 172.21.15.1:0/3794963196 >> 172.21.15.1:0/3794963196 conn(0x7f6a900cc9b0 msgr2=0x7f6a900ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.310+0000 7f6a96670700 1 -- 172.21.15.1:0/3794963196 shutdown_connections 2024-03-18T19:08:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.310+0000 7f6a96670700 1 -- 172.21.15.1:0/3794963196 wait complete. 2024-03-18T19:08:24.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.311+0000 7f6a96670700 1 Processor -- start 2024-03-18T19:08:24.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.312+0000 7f6a96670700 1 -- start start 2024-03-18T19:08:24.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.312+0000 7f6a96670700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a9006ac90 0x7f6a90159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:24.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.312+0000 7f6a96670700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a900cfa90 0x7f6a9014a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:24.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.312+0000 7f6a96670700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a9015d5a0 0x7f6a9014b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:24.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.312+0000 7f6a96670700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9014a5a0 con 0x7f6a9006ac90 2024-03-18T19:08:24.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.312+0000 7f6a96670700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9014a840 con 0x7f6a9015d5a0 2024-03-18T19:08:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.312+0000 7f6a96670700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9014ab20 con 0x7f6a900cfa90 2024-03-18T19:08:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.312+0000 7f6a8ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a9006ac90 0x7f6a90159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.312+0000 7f6a8ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a9006ac90 0x7f6a90159120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59448/0 (socket says 172.21.15.1:59448) 2024-03-18T19:08:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.313+0000 7f6a8ffff700 1 -- 172.21.15.1:0/2405642307 learned_addr learned my addr 172.21.15.1:0/2405642307 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.313+0000 7f6a8f7fe700 1 --2- 172.21.15.1:0/2405642307 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a900cfa90 0x7f6a9014a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.313+0000 7f6a94c0d700 1 --2- 172.21.15.1:0/2405642307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a9015d5a0 0x7f6a9014b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.313+0000 7f6a8ffff700 1 -- 172.21.15.1:0/2405642307 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a900cfa90 msgr2=0x7f6a9014a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.313+0000 7f6a8ffff700 1 --2- 172.21.15.1:0/2405642307 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a900cfa90 0x7f6a9014a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.313+0000 7f6a8ffff700 1 -- 172.21.15.1:0/2405642307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a9015d5a0 msgr2=0x7f6a9014b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.313+0000 7f6a8ffff700 1 --2- 172.21.15.1:0/2405642307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a9015d5a0 0x7f6a9014b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.313+0000 7f6a8ffff700 1 -- 172.21.15.1:0/2405642307 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a78009550 con 0x7f6a9006ac90 2024-03-18T19:08:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.313+0000 7f6a94c0d700 1 --2- 172.21.15.1:0/2405642307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a9015d5a0 0x7f6a9014b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:08:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.313+0000 7f6a8f7fe700 1 --2- 172.21.15.1:0/2405642307 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a900cfa90 0x7f6a9014a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:08:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.313+0000 7f6a8ffff700 1 --2- 172.21.15.1:0/2405642307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a9006ac90 0x7f6a90159120 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f6a8000c430 tx=0x7f6a8000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.313+0000 7f6a8cff9700 1 -- 172.21.15.1:0/2405642307 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a800054e0 con 0x7f6a9006ac90 2024-03-18T19:08:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.313+0000 7f6a8cff9700 1 -- 172.21.15.1:0/2405642307 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a80003680 con 0x7f6a9006ac90 2024-03-18T19:08:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.313+0000 7f6a96670700 1 -- 172.21.15.1:0/2405642307 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a9014ae00 con 0x7f6a9006ac90 2024-03-18T19:08:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.313+0000 7f6a8cff9700 1 -- 172.21.15.1:0/2405642307 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a80003a30 con 0x7f6a9006ac90 2024-03-18T19:08:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.313+0000 7f6a96670700 1 -- 172.21.15.1:0/2405642307 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a9014b3c0 con 0x7f6a9006ac90 2024-03-18T19:08:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.314+0000 7f6a96670700 1 -- 172.21.15.1:0/2405642307 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a900676a0 con 0x7f6a9006ac90 2024-03-18T19:08:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.315+0000 7f6a8cff9700 1 -- 172.21.15.1:0/2405642307 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6a80003b90 con 0x7f6a9006ac90 2024-03-18T19:08:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.315+0000 7f6a8cff9700 1 --2- 172.21.15.1:0/2405642307 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6a7006adb0 0x7f6a7006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.315+0000 7f6a8cff9700 1 -- 172.21.15.1:0/2405642307 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6a80088a70 con 0x7f6a9006ac90 2024-03-18T19:08:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.315+0000 7f6a8f7fe700 1 --2- 172.21.15.1:0/2405642307 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6a7006adb0 0x7f6a7006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.316+0000 7f6a8f7fe700 1 --2- 172.21.15.1:0/2405642307 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6a7006adb0 0x7f6a7006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6a78009130 tx=0x7f6a78008030 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:24.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.320+0000 7f6a8cff9700 1 -- 172.21.15.1:0/2405642307 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6a80053a00 con 0x7f6a9006ac90 2024-03-18T19:08:24.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.604+0000 7f6a96670700 1 -- 172.21.15.1:0/2405642307 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f6a90070080 con 0x7f6a9006ac90 2024-03-18T19:08:24.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.607+0000 7f6a8cff9700 1 -- 172.21.15.1:0/2405642307 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+0 (secure 0 0 0) 0x7f6a80017020 con 0x7f6a9006ac90 2024-03-18T19:08:24.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.609+0000 7f6a96670700 1 -- 172.21.15.1:0/2405642307 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6a7006adb0 msgr2=0x7f6a7006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:24.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.609+0000 7f6a96670700 1 --2- 172.21.15.1:0/2405642307 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6a7006adb0 0x7f6a7006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6a78009130 tx=0x7f6a78008030 comp rx=0 tx=0).stop 2024-03-18T19:08:24.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.609+0000 7f6a96670700 1 -- 172.21.15.1:0/2405642307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a9006ac90 msgr2=0x7f6a90159120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:24.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.609+0000 7f6a96670700 1 --2- 172.21.15.1:0/2405642307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a9006ac90 0x7f6a90159120 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f6a8000c430 tx=0x7f6a8000c7f0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.609+0000 7f6a96670700 1 -- 172.21.15.1:0/2405642307 shutdown_connections 2024-03-18T19:08:24.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.609+0000 7f6a96670700 1 --2- 172.21.15.1:0/2405642307 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6a7006adb0 0x7f6a7006d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.609+0000 7f6a96670700 1 --2- 172.21.15.1:0/2405642307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a9015d5a0 0x7f6a9014b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.609+0000 7f6a96670700 1 --2- 172.21.15.1:0/2405642307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a9006ac90 0x7f6a90159120 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.609+0000 7f6a96670700 1 --2- 172.21.15.1:0/2405642307 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a900cfa90 0x7f6a9014a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.609+0000 7f6a96670700 1 -- 172.21.15.1:0/2405642307 >> 172.21.15.1:0/2405642307 conn(0x7f6a900cc9b0 msgr2=0x7f6a900c68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:24.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.609+0000 7f6a96670700 1 -- 172.21.15.1:0/2405642307 shutdown_connections 2024-03-18T19:08:24.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.609+0000 7f6a96670700 1 -- 172.21.15.1:0/2405642307 wait complete. 2024-03-18T19:08:24.607 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-18T19:08:24.617 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:08:24.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.746+0000 7f76743ea700 1 Processor -- start 2024-03-18T19:08:24.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.746+0000 7f76743ea700 1 -- start start 2024-03-18T19:08:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.747+0000 7f76743ea700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f766c15a320 0x7f766c0ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.747+0000 7f76743ea700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f766c0acd80 con 0x7f766c071d10 2024-03-18T19:08:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.747+0000 7f76743ea700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f766c0cf220 con 0x7f766c0cea30 2024-03-18T19:08:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.747+0000 7f76743ea700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f766c0cef40 con 0x7f766c15a320 2024-03-18T19:08:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.747+0000 7f7672987700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f766c071d10 0x7f766c06e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60882/0 (socket says 172.21.15.1:60882) 2024-03-18T19:08:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.747+0000 7f7672987700 1 -- 172.21.15.1:0/1472273767 learned_addr learned my addr 172.21.15.1:0/1472273767 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.747+0000 7f7672186700 1 --2- 172.21.15.1:0/1472273767 >> v2:172.21.15.165:3300/0 conn(0x7f766c15a320 0x7f766c0ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7671184700 1 -- 172.21.15.1:0/1472273767 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459234335 0 0) 0x7f766c0acd80 con 0x7f766c071d10 2024-03-18T19:08:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7671184700 1 -- 172.21.15.1:0/1472273767 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76580035b0 con 0x7f766c071d10 2024-03-18T19:08:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7671184700 1 -- 172.21.15.1:0/1472273767 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591655870 0 0) 0x7f7668001110 con 0x7f766c0cea30 2024-03-18T19:08:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7671184700 1 -- 172.21.15.1:0/1472273767 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f766c0acd80 con 0x7f766c0cea30 2024-03-18T19:08:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7671184700 1 -- 172.21.15.1:0/1472273767 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2375725231 0 0) 0x7f76580035b0 con 0x7f766c071d10 2024-03-18T19:08:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7671184700 1 -- 172.21.15.1:0/1472273767 >> v2:172.21.15.165:3300/0 conn(0x7f766c15a320 msgr2=0x7f766c0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7671184700 1 --2- 172.21.15.1:0/1472273767 >> v2:172.21.15.165:3300/0 conn(0x7f766c15a320 0x7f766c0ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7671184700 1 -- 172.21.15.1:0/1472273767 >> v1:172.21.15.165:6789/0 conn(0x7f766c0cea30 legacy=0x7f766c071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7671184700 1 -- 172.21.15.1:0/1472273767 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766c06d7e0 con 0x7f766c071d10 2024-03-18T19:08:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7671184700 1 -- 172.21.15.1:0/1472273767 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f76640019f0 con 0x7f766c071d10 2024-03-18T19:08:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7672186700 1 --2- 172.21.15.1:0/1472273767 >> v2:172.21.15.165:3300/0 conn(0x7f766c15a320 0x7f766c0ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:08:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7671184700 1 -- 172.21.15.1:0/1472273767 >> v1:172.21.15.1:6789/0 conn(0x7f766c071d10 legacy=0x7f766c06e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7671184700 1 --2- 172.21.15.1:0/1472273767 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658003be0 0x7f7658003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7671184700 1 --2- 172.21.15.1:0/1472273767 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7658004850 0x7f7658004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7671184700 1 --2- 172.21.15.1:0/1472273767 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7658005210 0x7f765800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7671184700 1 -- 172.21.15.1:0/1472273767 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c0cef40 con 0x7f7658003be0 2024-03-18T19:08:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7671184700 1 -- 172.21.15.1:0/1472273767 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765800c0e0 con 0x7f7658005210 2024-03-18T19:08:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7671184700 1 -- 172.21.15.1:0/1472273767 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f765800c2a0 con 0x7f7658004850 2024-03-18T19:08:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7672186700 1 --2- 172.21.15.1:0/1472273767 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658003be0 0x7f7658003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7671985700 1 --2- 172.21.15.1:0/1472273767 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7658004850 0x7f7658004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7672987700 1 --2- 172.21.15.1:0/1472273767 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7658005210 0x7f765800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7672186700 1 -- 172.21.15.1:0/1472273767 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7658004850 msgr2=0x7f7658004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7672186700 1 --2- 172.21.15.1:0/1472273767 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7658004850 0x7f7658004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7672186700 1 -- 172.21.15.1:0/1472273767 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7658005210 msgr2=0x7f765800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7672186700 1 --2- 172.21.15.1:0/1472273767 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7658005210 0x7f765800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7672186700 1 -- 172.21.15.1:0/1472273767 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76580035b0 con 0x7f7658003be0 2024-03-18T19:08:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7672987700 1 -- 172.21.15.1:0/1472273767 reap_dead start 2024-03-18T19:08:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.748+0000 7f7672186700 1 --2- 172.21.15.1:0/1472273767 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658003be0 0x7f7658003fe0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f765c009d30 tx=0x7f7658005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e864411e7be13b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.749+0000 7f7671184700 1 -- 172.21.15.1:0/1472273767 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c009e80 con 0x7f7658003be0 2024-03-18T19:08:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.749+0000 7f7671184700 1 -- 172.21.15.1:0/1472273767 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765c00e9a0 con 0x7f7658003be0 2024-03-18T19:08:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.749+0000 7f7671184700 1 -- 172.21.15.1:0/1472273767 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765c00ed10 con 0x7f7658003be0 2024-03-18T19:08:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.749+0000 7f76743ea700 1 -- 172.21.15.1:0/1472273767 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658003be0 msgr2=0x7f7658003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.749+0000 7f76743ea700 1 --2- 172.21.15.1:0/1472273767 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658003be0 0x7f7658003fe0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f765c009d30 tx=0x7f7658005880 comp rx=0 tx=0).stop 2024-03-18T19:08:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.749+0000 7f76743ea700 1 -- 172.21.15.1:0/1472273767 shutdown_connections 2024-03-18T19:08:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.749+0000 7f76743ea700 1 --2- 172.21.15.1:0/1472273767 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658003be0 0x7f7658003fe0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.749+0000 7f76743ea700 1 -- 172.21.15.1:0/1472273767 >> 172.21.15.1:0/1472273767 conn(0x7f766c0cc990 msgr2=0x7f766c0ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.749+0000 7f76743ea700 1 -- 172.21.15.1:0/1472273767 shutdown_connections 2024-03-18T19:08:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.749+0000 7f76743ea700 1 -- 172.21.15.1:0/1472273767 wait complete. 2024-03-18T19:08:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.750+0000 7f76743ea700 1 Processor -- start 2024-03-18T19:08:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.750+0000 7f76743ea700 1 -- start start 2024-03-18T19:08:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.751+0000 7f76743ea700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7658003be0 0x7f766c15b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.751+0000 7f76743ea700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f766c153d70 0x7f766c0af290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.751+0000 7f76743ea700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f766c0c88a0 0x7f766c0c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.751+0000 7f76743ea700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c0c9260 con 0x7f766c0c88a0 2024-03-18T19:08:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.751+0000 7f76743ea700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c15cf90 con 0x7f7658003be0 2024-03-18T19:08:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.751+0000 7f76743ea700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766c0c9ce0 con 0x7f766c153d70 2024-03-18T19:08:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.751+0000 7f7672987700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f766c0c88a0 0x7f766c0c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.751+0000 7f7672987700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f766c0c88a0 0x7f766c0c8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57096/0 (socket says 172.21.15.1:57096) 2024-03-18T19:08:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.751+0000 7f7672987700 1 -- 172.21.15.1:0/4210562600 learned_addr learned my addr 172.21.15.1:0/4210562600 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.751+0000 7f7672186700 1 --2- 172.21.15.1:0/4210562600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7658003be0 0x7f766c15b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.751+0000 7f7671985700 1 --2- 172.21.15.1:0/4210562600 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f766c153d70 0x7f766c0af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.751+0000 7f7672987700 1 -- 172.21.15.1:0/4210562600 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f766c153d70 msgr2=0x7f766c0af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.751+0000 7f7672987700 1 --2- 172.21.15.1:0/4210562600 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f766c153d70 0x7f766c0af290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.751+0000 7f7672987700 1 -- 172.21.15.1:0/4210562600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7658003be0 msgr2=0x7f766c15b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.751+0000 7f7672987700 1 --2- 172.21.15.1:0/4210562600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7658003be0 0x7f766c15b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.752+0000 7f7672987700 1 -- 172.21.15.1:0/4210562600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766c15cb00 con 0x7f766c0c88a0 2024-03-18T19:08:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.752+0000 7f7672987700 1 --2- 172.21.15.1:0/4210562600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f766c0c88a0 0x7f766c0c8d20 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f766c067230 tx=0x7f7664007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.752+0000 7f7662ffd700 1 -- 172.21.15.1:0/4210562600 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766400ea60 con 0x7f766c0c88a0 2024-03-18T19:08:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.752+0000 7f7662ffd700 1 -- 172.21.15.1:0/4210562600 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766400ebc0 con 0x7f766c0c88a0 2024-03-18T19:08:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.752+0000 7f76743ea700 1 -- 172.21.15.1:0/4210562600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f766c0af7d0 con 0x7f766c0c88a0 2024-03-18T19:08:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.752+0000 7f7662ffd700 1 -- 172.21.15.1:0/4210562600 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7664013420 con 0x7f766c0c88a0 2024-03-18T19:08:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.752+0000 7f76743ea700 1 -- 172.21.15.1:0/4210562600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f766c15bef0 con 0x7f766c0c88a0 2024-03-18T19:08:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.753+0000 7f76743ea700 1 -- 172.21.15.1:0/4210562600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766c0ae5b0 con 0x7f766c0c88a0 2024-03-18T19:08:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.753+0000 7f7662ffd700 1 -- 172.21.15.1:0/4210562600 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7664013580 con 0x7f766c0c88a0 2024-03-18T19:08:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.754+0000 7f7662ffd700 1 --2- 172.21.15.1:0/4210562600 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f765006adb0 0x7f765006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:24.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.754+0000 7f7662ffd700 1 -- 172.21.15.1:0/4210562600 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7664089ad0 con 0x7f766c0c88a0 2024-03-18T19:08:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.754+0000 7f7672186700 1 --2- 172.21.15.1:0/4210562600 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f765006adb0 0x7f765006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.754+0000 7f7672186700 1 --2- 172.21.15.1:0/4210562600 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f765006adb0 0x7f765006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f765c000c00 tx=0x7f765c008030 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:24.758+0000 7f7662ffd700 1 -- 172.21.15.1:0/4210562600 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7664054a60 con 0x7f766c0c88a0 2024-03-18T19:08:25.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.004+0000 7f76743ea700 1 -- 172.21.15.1:0/4210562600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f766c0ae5b0 con 0x7f766c0c88a0 2024-03-18T19:08:25.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.004+0000 7f7662ffd700 1 -- 172.21.15.1:0/4210562600 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4730 (secure 0 0 0) 0x7f7664058080 con 0x7f766c0c88a0 2024-03-18T19:08:25.000 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:25.001 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6800,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3354770356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3354770356},{"type":"v1","addr":"172.21.15.1:6837","nonce":3354770356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":6809,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/695588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":695588699},{"type":"v1","addr":"172.21.15.165:6834","nonce":695588699}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7660,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3186346302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3186346302},{"type":"v1","addr":"172.21.15.1:6836","nonce":3186346302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:08:19.581625+0000","modified":"2024-03-18T19:08:22.589436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7675},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7675":{"gid":7675,"name":"d","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/450810178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":450810178},{"type":"v1","addr":"172.21.15.165:6835","nonce":450810178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-18T19:08:25.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.007+0000 7f76743ea700 1 -- 172.21.15.1:0/4210562600 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f765006adb0 msgr2=0x7f765006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:25.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.007+0000 7f76743ea700 1 --2- 172.21.15.1:0/4210562600 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f765006adb0 0x7f765006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f765c000c00 tx=0x7f765c008030 comp rx=0 tx=0).stop 2024-03-18T19:08:25.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.007+0000 7f76743ea700 1 -- 172.21.15.1:0/4210562600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f766c0c88a0 msgr2=0x7f766c0c8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:25.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.007+0000 7f76743ea700 1 --2- 172.21.15.1:0/4210562600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f766c0c88a0 0x7f766c0c8d20 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f766c067230 tx=0x7f7664007c90 comp rx=0 tx=0).stop 2024-03-18T19:08:25.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.007+0000 7f76743ea700 1 -- 172.21.15.1:0/4210562600 shutdown_connections 2024-03-18T19:08:25.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.007+0000 7f76743ea700 1 --2- 172.21.15.1:0/4210562600 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f765006adb0 0x7f765006d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.007+0000 7f76743ea700 1 --2- 172.21.15.1:0/4210562600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f766c0c88a0 0x7f766c0c8d20 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.007+0000 7f76743ea700 1 --2- 172.21.15.1:0/4210562600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7658003be0 0x7f766c15b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.007+0000 7f76743ea700 1 --2- 172.21.15.1:0/4210562600 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f766c153d70 0x7f766c0af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.007+0000 7f76743ea700 1 -- 172.21.15.1:0/4210562600 >> 172.21.15.1:0/4210562600 conn(0x7f766c0cc990 msgr2=0x7f766c06e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:25.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.007+0000 7f76743ea700 1 -- 172.21.15.1:0/4210562600 shutdown_connections 2024-03-18T19:08:25.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.007+0000 7f76743ea700 1 -- 172.21.15.1:0/4210562600 wait complete. 2024-03-18T19:08:25.004 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 57 2024-03-18T19:08:25.136 INFO:tasks.ceph.mds.d.smithi165.stderr:2024-03-18T19:08:25.140+0000 7f896cab9700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T19:08:25.417 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T19:08:19.581625+0000', 'modified': '2024-03-18T19:08:22.589436+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7675}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7675': {'gid': 7675, 'name': 'd', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.165:6835/450810178', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6833', 'nonce': 450810178}, {'type': 'v1', 'addr': '172.21.15.165:6835', 'nonce': 450810178}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T19:08:25.418 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T19:08:25.418 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-18T19:08:25.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.557+0000 7f628bec0700 1 Processor -- start 2024-03-18T19:08:25.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.557+0000 7f628bec0700 1 -- start start 2024-03-18T19:08:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.558+0000 7f628bec0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f628415d5b0 0x7f628415d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.558+0000 7f628bec0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6284099a40 0x7f6284099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.558+0000 7f628bec0700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f628409a3e0 0x7f628406ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.558+0000 7f628bec0700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62840cbbb0 con 0x7f628415d5b0 2024-03-18T19:08:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.558+0000 7f628bec0700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628409a830 con 0x7f628409a3e0 2024-03-18T19:08:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.558+0000 7f628bec0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62840b3b00 con 0x7f6284099a40 2024-03-18T19:08:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.558+0000 7f6289c5c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f628415d5b0 0x7f628415d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.558+0000 7f6289c5c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f628415d5b0 0x7f628415d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57106/0 (socket says 172.21.15.1:57106) 2024-03-18T19:08:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.558+0000 7f6289c5c700 1 -- 172.21.15.1:0/3395910071 learned_addr learned my addr 172.21.15.1:0/3395910071 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.558+0000 7f628a45d700 1 --2- 172.21.15.1:0/3395910071 >> v2:172.21.15.165:3300/0 conn(0x7f628409a3e0 0x7f628406ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.558+0000 7f628945b700 1 --2- 172.21.15.1:0/3395910071 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6284099a40 0x7f6284099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.558+0000 7f6289c5c700 1 -- 172.21.15.1:0/3395910071 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6284099a40 msgr2=0x7f6284099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.558+0000 7f6289c5c700 1 --2- 172.21.15.1:0/3395910071 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6284099a40 0x7f6284099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.558+0000 7f6289c5c700 1 -- 172.21.15.1:0/3395910071 >> v2:172.21.15.165:3300/0 conn(0x7f628409a3e0 msgr2=0x7f628406ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.558+0000 7f6289c5c700 1 --2- 172.21.15.1:0/3395910071 >> v2:172.21.15.165:3300/0 conn(0x7f628409a3e0 0x7f628406ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.558+0000 7f6289c5c700 1 -- 172.21.15.1:0/3395910071 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62840cf7f0 con 0x7f628415d5b0 2024-03-18T19:08:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.558+0000 7f6289c5c700 1 --2- 172.21.15.1:0/3395910071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f628415d5b0 0x7f628415d990 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f62840720b0 tx=0x7f6280009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3fa3c96935a5366 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:25.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.558+0000 7f6288c5a700 1 -- 172.21.15.1:0/3395910071 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628000b870 con 0x7f628415d5b0 2024-03-18T19:08:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.559+0000 7f6288c5a700 1 -- 172.21.15.1:0/3395910071 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f628000b9d0 con 0x7f628415d5b0 2024-03-18T19:08:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.559+0000 7f6288c5a700 1 -- 172.21.15.1:0/3395910071 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628000bd60 con 0x7f628415d5b0 2024-03-18T19:08:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.559+0000 7f628bec0700 1 -- 172.21.15.1:0/3395910071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f628415d5b0 msgr2=0x7f628415d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.559+0000 7f628bec0700 1 --2- 172.21.15.1:0/3395910071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f628415d5b0 0x7f628415d990 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f62840720b0 tx=0x7f6280009bd0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.559+0000 7f628bec0700 1 -- 172.21.15.1:0/3395910071 shutdown_connections 2024-03-18T19:08:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.559+0000 7f628bec0700 1 --2- 172.21.15.1:0/3395910071 >> v2:172.21.15.165:3300/0 conn(0x7f628409a3e0 0x7f628406ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.559+0000 7f628bec0700 1 --2- 172.21.15.1:0/3395910071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f628415d5b0 0x7f628415d990 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.559+0000 7f628bec0700 1 --2- 172.21.15.1:0/3395910071 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6284099a40 0x7f6284099ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.559+0000 7f628bec0700 1 -- 172.21.15.1:0/3395910071 >> 172.21.15.1:0/3395910071 conn(0x7f62840b2bd0 msgr2=0x7f62840b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.559+0000 7f628bec0700 1 -- 172.21.15.1:0/3395910071 shutdown_connections 2024-03-18T19:08:25.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.559+0000 7f628bec0700 1 -- 172.21.15.1:0/3395910071 wait complete. 2024-03-18T19:08:25.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.560+0000 7f628bec0700 1 Processor -- start 2024-03-18T19:08:25.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.560+0000 7f628bec0700 1 -- start start 2024-03-18T19:08:25.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.561+0000 7f628bec0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6284099a40 0x7f628414f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.561+0000 7f628bec0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f628409a3e0 0x7f62840b1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.561+0000 7f628bec0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f628415d5b0 0x7f6284159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.561+0000 7f628bec0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62840aef40 con 0x7f628415d5b0 2024-03-18T19:08:25.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.561+0000 7f628bec0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62840ae330 con 0x7f628409a3e0 2024-03-18T19:08:25.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.561+0000 7f628bec0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62840ae610 con 0x7f6284099a40 2024-03-18T19:08:25.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.561+0000 7f628a45d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f628415d5b0 0x7f6284159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.561+0000 7f628a45d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f628415d5b0 0x7f6284159fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57118/0 (socket says 172.21.15.1:57118) 2024-03-18T19:08:25.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.561+0000 7f628a45d700 1 -- 172.21.15.1:0/1719973060 learned_addr learned my addr 172.21.15.1:0/1719973060 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:25.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.561+0000 7f628945b700 1 --2- 172.21.15.1:0/1719973060 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f628409a3e0 0x7f62840b1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.561+0000 7f6289c5c700 1 --2- 172.21.15.1:0/1719973060 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6284099a40 0x7f628414f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.561+0000 7f628a45d700 1 -- 172.21.15.1:0/1719973060 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6284099a40 msgr2=0x7f628414f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:25.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.561+0000 7f628a45d700 1 --2- 172.21.15.1:0/1719973060 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6284099a40 0x7f628414f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.561+0000 7f628a45d700 1 -- 172.21.15.1:0/1719973060 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f628409a3e0 msgr2=0x7f62840b1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:25.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.561+0000 7f628a45d700 1 --2- 172.21.15.1:0/1719973060 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f628409a3e0 0x7f62840b1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.561+0000 7f628a45d700 1 -- 172.21.15.1:0/1719973060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6280009550 con 0x7f628415d5b0 2024-03-18T19:08:25.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.562+0000 7f628a45d700 1 --2- 172.21.15.1:0/1719973060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f628415d5b0 0x7f6284159fa0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f627c007e60 tx=0x7f627c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:25.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.562+0000 7f627a7fc700 1 -- 172.21.15.1:0/1719973060 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c00ed30 con 0x7f628415d5b0 2024-03-18T19:08:25.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.562+0000 7f627a7fc700 1 -- 172.21.15.1:0/1719973060 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627c014420 con 0x7f628415d5b0 2024-03-18T19:08:25.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.562+0000 7f628bec0700 1 -- 172.21.15.1:0/1719973060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62840ae8f0 con 0x7f628415d5b0 2024-03-18T19:08:25.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.562+0000 7f627a7fc700 1 -- 172.21.15.1:0/1719973060 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c0147d0 con 0x7f628415d5b0 2024-03-18T19:08:25.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.562+0000 7f628bec0700 1 -- 172.21.15.1:0/1719973060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62840b1b50 con 0x7f628415d5b0 2024-03-18T19:08:25.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.564+0000 7f627a7fc700 1 -- 172.21.15.1:0/1719973060 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f627c014930 con 0x7f628415d5b0 2024-03-18T19:08:25.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.564+0000 7f627a7fc700 1 --2- 172.21.15.1:0/1719973060 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f626806adb0 0x7f626806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.564+0000 7f627a7fc700 1 -- 172.21.15.1:0/1719973060 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f627c089c00 con 0x7f628415d5b0 2024-03-18T19:08:25.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.564+0000 7f6289c5c700 1 --2- 172.21.15.1:0/1719973060 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f626806adb0 0x7f626806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.565+0000 7f6289c5c700 1 --2- 172.21.15.1:0/1719973060 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f626806adb0 0x7f626806d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6280000c00 tx=0x7f6280008030 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.566+0000 7f6298069b80 1 Processor -- start 2024-03-18T19:08:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.566+0000 7f6298069b80 1 -- start start 2024-03-18T19:08:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f6298069b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc4dc10 0x55ededc4dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f6298069b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc1f0a0 0x55ededc6a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f6298069b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc5f630 0x55ededc22160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f6298069b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ededb98f20 con 0x55ededc5f630 2024-03-18T19:08:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f6298069b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ededb7ac30 con 0x55ededc1f0a0 2024-03-18T19:08:25.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f6298069b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ededb78430 con 0x55ededc4dc10 2024-03-18T19:08:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f628a45d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc5f630 0x55ededc22160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f628a45d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc5f630 0x55ededc22160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57132/0 (socket says 172.21.15.1:57132) 2024-03-18T19:08:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f628a45d700 1 -- 172.21.15.1:0/1890957601 learned_addr learned my addr 172.21.15.1:0/1890957601 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f628945b700 1 --2- 172.21.15.1:0/1890957601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc1f0a0 0x55ededc6a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f6289c5c700 1 --2- 172.21.15.1:0/1890957601 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc4dc10 0x55ededc4dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f628a45d700 1 -- 172.21.15.1:0/1890957601 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc4dc10 msgr2=0x55ededc4dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f628a45d700 1 --2- 172.21.15.1:0/1890957601 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc4dc10 0x55ededc4dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f628a45d700 1 -- 172.21.15.1:0/1890957601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc1f0a0 msgr2=0x55ededc6a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f628a45d700 1 --2- 172.21.15.1:0/1890957601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc1f0a0 0x55ededc6a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f628a45d700 1 -- 172.21.15.1:0/1890957601 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eded967790 con 0x55ededc5f630 2024-03-18T19:08:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f628945b700 1 --2- 172.21.15.1:0/1890957601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc1f0a0 0x55ededc6a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:08:25.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f6289c5c700 1 --2- 172.21.15.1:0/1890957601 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc4dc10 0x55ededc4dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:08:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f628a45d700 1 --2- 172.21.15.1:0/1890957601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc5f630 0x55ededc22160 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f627c007e30 tx=0x7f627c058280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e36fc475efde3d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.567+0000 7f62677fe700 1 -- 172.21.15.1:0/1890957601 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c05c680 con 0x55ededc5f630 2024-03-18T19:08:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.568+0000 7f62677fe700 1 -- 172.21.15.1:0/1890957601 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627c060c40 con 0x55ededc5f630 2024-03-18T19:08:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.568+0000 7f62677fe700 1 -- 172.21.15.1:0/1890957601 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c014be0 con 0x55ededc5f630 2024-03-18T19:08:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.568+0000 7f6298069b80 1 -- 172.21.15.1:0/1890957601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc5f630 msgr2=0x55ededc22160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.568+0000 7f6298069b80 1 --2- 172.21.15.1:0/1890957601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc5f630 0x55ededc22160 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f627c007e30 tx=0x7f627c058280 comp rx=0 tx=0).stop 2024-03-18T19:08:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.568+0000 7f6298069b80 1 -- 172.21.15.1:0/1890957601 shutdown_connections 2024-03-18T19:08:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.568+0000 7f6298069b80 1 --2- 172.21.15.1:0/1890957601 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc4dc10 0x55ededc4dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.568+0000 7f6298069b80 1 --2- 172.21.15.1:0/1890957601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc5f630 0x55ededc22160 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.568+0000 7f6298069b80 1 --2- 172.21.15.1:0/1890957601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc1f0a0 0x55ededc6a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.568+0000 7f6298069b80 1 -- 172.21.15.1:0/1890957601 >> 172.21.15.1:0/1890957601 conn(0x55ededb81cf0 msgr2=0x55ededc21a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.568+0000 7f6298069b80 1 -- 172.21.15.1:0/1890957601 shutdown_connections 2024-03-18T19:08:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.568+0000 7f6298069b80 1 -- 172.21.15.1:0/1890957601 wait complete. 2024-03-18T19:08:25.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.569+0000 7f6298069b80 1 -- start start 2024-03-18T19:08:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.569+0000 7f6298069b80 1 Processor -- start 2024-03-18T19:08:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f6298069b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc1f0a0 0x55ededc4fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f6298069b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc4dc10 0x55ededc6c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f6298069b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc5f630 0x55ededc22af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f6298069b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ededb73d20 con 0x55ededc5f630 2024-03-18T19:08:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f6298069b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ededb0f770 con 0x55ededc4dc10 2024-03-18T19:08:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f6298069b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ededb326f0 con 0x55ededc1f0a0 2024-03-18T19:08:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f6298069b80 10 client.0 _ll_register_callbacks cb 0x55eded97b5a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T19:08:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f628a45d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc5f630 0x55ededc22af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f628a45d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc5f630 0x55ededc22af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57146/0 (socket says 172.21.15.1:57146) 2024-03-18T19:08:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f628a45d700 1 -- 172.21.15.1:0/2349209928 learned_addr learned my addr 172.21.15.1:0/2349209928 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f6289c5c700 1 --2- 172.21.15.1:0/2349209928 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc1f0a0 0x55ededc4fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f628945b700 1 --2- 172.21.15.1:0/2349209928 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc4dc10 0x55ededc6c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f628a45d700 1 -- 172.21.15.1:0/2349209928 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc1f0a0 msgr2=0x55ededc4fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:25.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f628a45d700 1 --2- 172.21.15.1:0/2349209928 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc1f0a0 0x55ededc4fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f628a45d700 1 -- 172.21.15.1:0/2349209928 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc4dc10 msgr2=0x55ededc6c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f628a45d700 1 --2- 172.21.15.1:0/2349209928 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc4dc10 0x55ededc6c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f628a45d700 1 -- 172.21.15.1:0/2349209928 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62840b1b50 con 0x55ededc5f630 2024-03-18T19:08:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f628945b700 1 --2- 172.21.15.1:0/2349209928 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc4dc10 0x55ededc6c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f628a45d700 1 --2- 172.21.15.1:0/2349209928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc5f630 0x55ededc22af0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f627c056ba0 tx=0x7f627c020070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be180da2c1f12a4c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.570+0000 7f6264ff9700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T19:08:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.571+0000 7f6264ff9700 1 -- 172.21.15.1:0/2349209928 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c014de0 con 0x55ededc5f630 2024-03-18T19:08:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.571+0000 7f6264ff9700 1 -- 172.21.15.1:0/2349209928 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f627c058ab0 con 0x55ededc5f630 2024-03-18T19:08:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.571+0000 7f6298069b80 1 -- 172.21.15.1:0/2349209928 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ededc15e90 con 0x55ededc5f630 2024-03-18T19:08:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.571+0000 7f6264ff9700 1 -- 172.21.15.1:0/2349209928 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7f627c058c90 con 0x55ededc5f630 2024-03-18T19:08:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.571+0000 7f6298069b80 10 client.8427 fetch_fsmap learned FSMap version 57 2024-03-18T19:08:25.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.571+0000 7f6298069b80 1 -- 172.21.15.1:0/2349209928 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x55ededa540a0 con 0x55ededc5f630 2024-03-18T19:08:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.571+0000 7f6264ff9700 1 -- 172.21.15.1:0/2349209928 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ededc15e90 con 0x55ededc5f630 2024-03-18T19:08:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.571+0000 7f6264ff9700 1 -- 172.21.15.1:0/2349209928 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 57) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f627c057d40 con 0x55ededc5f630 2024-03-18T19:08:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.571+0000 7f6298069b80 10 client.8427 fetch_fsmap finished waiting for FSMap version 57 2024-03-18T19:08:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.571+0000 7f6298069b80 10 client.8427 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-18T19:08:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.571+0000 7f6298069b80 20 client.8427 populate_metadata read hostname 'smithi001' 2024-03-18T19:08:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.571+0000 7f6298069b80 1 --2- 172.21.15.1:0/2349209928 >> [v2:172.21.15.165:6833/450810178,v1:172.21.15.165:6835/450810178] conn(0x55ededc46a30 0x55ededc46e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.571+0000 7f6298069b80 4 client.8427 mds_command: new command op to 7675 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T19:08:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.571+0000 7f6298069b80 1 -- 172.21.15.1:0/2349209928 --> [v2:172.21.15.165:6833/450810178,v1:172.21.15.165:6835/450810178] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ededbf3010 con 0x55ededc46a30 2024-03-18T19:08:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.571+0000 7f6289c5c700 1 --2- 172.21.15.1:0/2349209928 >> [v2:172.21.15.165:6833/450810178,v1:172.21.15.165:6835/450810178] conn(0x55ededc46a30 0x55ededc46e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.572+0000 7f6264ff9700 10 client.8427 ms_handle_connect on v2:172.21.15.165:6833/450810178 2024-03-18T19:08:25.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.572+0000 7f6289c5c700 1 --2- 172.21.15.1:0/2349209928 >> [v2:172.21.15.165:6833/450810178,v1:172.21.15.165:6835/450810178] conn(0x55ededc46a30 0x55ededc46e10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a16e28ad88ab3d6e server_cookie=da92e6c65a91b992 in_seq=0 out_seq=0 2024-03-18T19:08:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.572+0000 7f6264ff9700 1 -- 172.21.15.1:0/2349209928 <== mds.0 v2:172.21.15.165:6833/450810178 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ededbf3010 con 0x55ededc46a30 2024-03-18T19:08:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.572+0000 7f6264ff9700 10 client.8427 handle_command_reply: tid=0 2024-03-18T19:08:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.572+0000 7f6298069b80 1 client.8427 shutdown 2024-03-18T19:08:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.572+0000 7f6298069b80 1 -- 172.21.15.1:0/2349209928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc5f630 msgr2=0x55ededc22af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.572+0000 7f6298069b80 1 --2- 172.21.15.1:0/2349209928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc5f630 0x55ededc22af0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f627c056ba0 tx=0x7f627c020070 comp rx=0 tx=0).stop 2024-03-18T19:08:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.573+0000 7f6298069b80 1 -- 172.21.15.1:0/2349209928 shutdown_connections 2024-03-18T19:08:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.573+0000 7f6298069b80 1 --2- 172.21.15.1:0/2349209928 >> [v2:172.21.15.165:6833/450810178,v1:172.21.15.165:6835/450810178] conn(0x55ededc46a30 0x55ededc46e10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.573+0000 7f6298069b80 1 --2- 172.21.15.1:0/2349209928 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc1f0a0 0x55ededc4fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.573+0000 7f6298069b80 1 --2- 172.21.15.1:0/2349209928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc5f630 0x55ededc22af0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.573+0000 7f6264ff9700 0 client.8427 ms_handle_reset on v2:172.21.15.165:6833/450810178 2024-03-18T19:08:25.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.573+0000 7f6298069b80 1 --2- 172.21.15.1:0/2349209928 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc4dc10 0x55ededc6c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.573+0000 7f6298069b80 1 -- 172.21.15.1:0/2349209928 >> 172.21.15.1:0/2349209928 conn(0x55ededb81cf0 msgr2=0x55ededc1d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.573+0000 7f6298069b80 1 -- 172.21.15.1:0/2349209928 shutdown_connections 2024-03-18T19:08:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.573+0000 7f6298069b80 1 -- 172.21.15.1:0/2349209928 wait complete. 2024-03-18T19:08:25.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.573+0000 7f6298069b80 20 client.8427 trim_cache size 0 max 16384 2024-03-18T19:08:25.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.590+0000 7f6298069b80 1 Processor -- start 2024-03-18T19:08:25.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.590+0000 7f6298069b80 1 -- start start 2024-03-18T19:08:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.591+0000 7f6298069b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc1f0a0 0x55ededc1dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.591+0000 7f6298069b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc4dc10 0x55ededc4f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.591+0000 7f6298069b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc5f630 0x55ededc4fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.591+0000 7f6298069b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eded968390 con 0x55ededc4dc10 2024-03-18T19:08:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.591+0000 7f6298069b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ededc21900 con 0x55ededc1f0a0 2024-03-18T19:08:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.591+0000 7f6298069b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ededc22b50 con 0x55ededc5f630 2024-03-18T19:08:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.591+0000 7f628945b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc4dc10 0x55ededc4f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.591+0000 7f628945b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc4dc10 0x55ededc4f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57160/0 (socket says 172.21.15.1:57160) 2024-03-18T19:08:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.591+0000 7f628945b700 1 -- 172.21.15.1:0/83752227 learned_addr learned my addr 172.21.15.1:0/83752227 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.591+0000 7f6289c5c700 1 --2- 172.21.15.1:0/83752227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc1f0a0 0x55ededc1dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.591+0000 7f628a45d700 1 --2- 172.21.15.1:0/83752227 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc5f630 0x55ededc4fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.592+0000 7f628945b700 1 -- 172.21.15.1:0/83752227 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc5f630 msgr2=0x55ededc4fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.592+0000 7f628945b700 1 --2- 172.21.15.1:0/83752227 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc5f630 0x55ededc4fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.592+0000 7f628945b700 1 -- 172.21.15.1:0/83752227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc1f0a0 msgr2=0x55ededc1dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.592+0000 7f628945b700 1 --2- 172.21.15.1:0/83752227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc1f0a0 0x55ededc1dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.592+0000 7f628945b700 1 -- 172.21.15.1:0/83752227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ededb3a010 con 0x55ededc4dc10 2024-03-18T19:08:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.592+0000 7f6289c5c700 1 --2- 172.21.15.1:0/83752227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc1f0a0 0x55ededc1dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.592+0000 7f628945b700 1 --2- 172.21.15.1:0/83752227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc4dc10 0x55ededc4f4d0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f62840cb220 tx=0x7f62740036a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe05f259f8bff2b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.592+0000 7f6267fff700 1 -- 172.21.15.1:0/83752227 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6274003d70 con 0x55ededc4dc10 2024-03-18T19:08:25.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.592+0000 7f6267fff700 1 -- 172.21.15.1:0/83752227 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627400b500 con 0x55ededc4dc10 2024-03-18T19:08:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.592+0000 7f6267fff700 1 -- 172.21.15.1:0/83752227 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627400b8b0 con 0x55ededc4dc10 2024-03-18T19:08:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.592+0000 7f6298069b80 1 -- 172.21.15.1:0/83752227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc4dc10 msgr2=0x55ededc4f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.592+0000 7f6298069b80 1 --2- 172.21.15.1:0/83752227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc4dc10 0x55ededc4f4d0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f62840cb220 tx=0x7f62740036a0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.592+0000 7f6298069b80 1 -- 172.21.15.1:0/83752227 shutdown_connections 2024-03-18T19:08:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.592+0000 7f6298069b80 1 --2- 172.21.15.1:0/83752227 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc5f630 0x55ededc4fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.592+0000 7f6298069b80 1 --2- 172.21.15.1:0/83752227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc1f0a0 0x55ededc1dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.592+0000 7f6298069b80 1 --2- 172.21.15.1:0/83752227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc4dc10 0x55ededc4f4d0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.592+0000 7f6298069b80 1 -- 172.21.15.1:0/83752227 >> 172.21.15.1:0/83752227 conn(0x55ededb81cf0 msgr2=0x55ededc81ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.592+0000 7f6298069b80 1 -- 172.21.15.1:0/83752227 shutdown_connections 2024-03-18T19:08:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.593+0000 7f6298069b80 1 -- 172.21.15.1:0/83752227 wait complete. 2024-03-18T19:08:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.593+0000 7f6298069b80 1 -- start start 2024-03-18T19:08:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.593+0000 7f6298069b80 1 Processor -- start 2024-03-18T19:08:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.594+0000 7f6298069b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc1f0a0 0x55ededc510b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.594+0000 7f6298069b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc4dc10 0x55ededc515f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.594+0000 7f6298069b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc5f630 0x55ededccfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.594+0000 7f6298069b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ededc50c90 con 0x55ededc4dc10 2024-03-18T19:08:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.594+0000 7f6298069b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ededcd0e10 con 0x55ededc5f630 2024-03-18T19:08:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.594+0000 7f6298069b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ededc5e5d0 con 0x55ededc1f0a0 2024-03-18T19:08:25.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.594+0000 7f6298069b80 10 client.0 _ll_register_callbacks cb 0x7f627c058460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T19:08:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.594+0000 7f628945b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc4dc10 0x55ededc515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.594+0000 7f628945b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc4dc10 0x55ededc515f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57168/0 (socket says 172.21.15.1:57168) 2024-03-18T19:08:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.594+0000 7f628945b700 1 -- 172.21.15.1:0/1955999716 learned_addr learned my addr 172.21.15.1:0/1955999716 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.594+0000 7f628a45d700 1 --2- 172.21.15.1:0/1955999716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc5f630 0x55ededccfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.594+0000 7f6289c5c700 1 --2- 172.21.15.1:0/1955999716 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc1f0a0 0x55ededc510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.594+0000 7f628945b700 1 -- 172.21.15.1:0/1955999716 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc1f0a0 msgr2=0x55ededc510b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.594+0000 7f628945b700 1 --2- 172.21.15.1:0/1955999716 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc1f0a0 0x55ededc510b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.594+0000 7f628945b700 1 -- 172.21.15.1:0/1955999716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc5f630 msgr2=0x55ededccfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.594+0000 7f628945b700 1 --2- 172.21.15.1:0/1955999716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc5f630 0x55ededccfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.594+0000 7f628945b700 1 -- 172.21.15.1:0/1955999716 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f627400e040 con 0x55ededc4dc10 2024-03-18T19:08:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.595+0000 7f628945b700 1 --2- 172.21.15.1:0/1955999716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc4dc10 0x55ededc515f0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f6274000c00 tx=0x7f62740047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ce69516d8f23cf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.595+0000 7f6264ff9700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T19:08:25.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.595+0000 7f6264ff9700 1 -- 172.21.15.1:0/1955999716 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627400b9b0 con 0x55ededc4dc10 2024-03-18T19:08:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.595+0000 7f6264ff9700 1 -- 172.21.15.1:0/1955999716 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f627400bb10 con 0x55ededc4dc10 2024-03-18T19:08:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.595+0000 7f6298069b80 1 -- 172.21.15.1:0/1955999716 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f627c060c40 con 0x55ededc4dc10 2024-03-18T19:08:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.595+0000 7f6264ff9700 1 -- 172.21.15.1:0/1955999716 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7f627400bd10 con 0x55ededc4dc10 2024-03-18T19:08:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.595+0000 7f6298069b80 10 client.8433 fetch_fsmap learned FSMap version 57 2024-03-18T19:08:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.595+0000 7f6298069b80 1 -- 172.21.15.1:0/1955999716 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x55eded9c81d0 con 0x55ededc4dc10 2024-03-18T19:08:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.595+0000 7f6264ff9700 1 -- 172.21.15.1:0/1955999716 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627c060c40 con 0x55ededc4dc10 2024-03-18T19:08:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.595+0000 7f6264ff9700 1 -- 172.21.15.1:0/1955999716 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 57) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f6274016460 con 0x55ededc4dc10 2024-03-18T19:08:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.595+0000 7f6298069b80 10 client.8433 fetch_fsmap finished waiting for FSMap version 57 2024-03-18T19:08:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.595+0000 7f6298069b80 10 client.8433 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-18T19:08:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.595+0000 7f6298069b80 20 client.8433 populate_metadata read hostname 'smithi001' 2024-03-18T19:08:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.595+0000 7f6298069b80 1 --2- 172.21.15.1:0/1955999716 >> [v2:172.21.15.165:6833/450810178,v1:172.21.15.165:6835/450810178] conn(0x55ededcd0150 0x55ededcd0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.595+0000 7f6298069b80 4 client.8433 mds_command: new command op to 7675 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T19:08:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.595+0000 7f6298069b80 1 -- 172.21.15.1:0/1955999716 --> [v2:172.21.15.165:6833/450810178,v1:172.21.15.165:6835/450810178] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55eded9ad040 con 0x55ededcd0150 2024-03-18T19:08:25.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.595+0000 7f6289c5c700 1 --2- 172.21.15.1:0/1955999716 >> [v2:172.21.15.165:6833/450810178,v1:172.21.15.165:6835/450810178] conn(0x55ededcd0150 0x55ededcd0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.596+0000 7f6264ff9700 10 client.8433 ms_handle_connect on v2:172.21.15.165:6833/450810178 2024-03-18T19:08:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.596+0000 7f6289c5c700 1 --2- 172.21.15.1:0/1955999716 >> [v2:172.21.15.165:6833/450810178,v1:172.21.15.165:6835/450810178] conn(0x55ededcd0150 0x55ededcd0530 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18fca3d080f383 server_cookie=6a36aec42f3e85d9 in_seq=0 out_seq=0 2024-03-18T19:08:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.596+0000 7f6264ff9700 1 -- 172.21.15.1:0/1955999716 <== mds.0 v2:172.21.15.165:6833/450810178 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55eded9ad040 con 0x55ededcd0150 2024-03-18T19:08:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.596+0000 7f6264ff9700 10 client.8433 handle_command_reply: tid=0 2024-03-18T19:08:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.596+0000 7f6298069b80 1 client.8433 shutdown 2024-03-18T19:08:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.596+0000 7f6298069b80 1 -- 172.21.15.1:0/1955999716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc4dc10 msgr2=0x55ededc515f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.596+0000 7f6298069b80 1 --2- 172.21.15.1:0/1955999716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc4dc10 0x55ededc515f0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f6274000c00 tx=0x7f62740047c0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.596+0000 7f6298069b80 1 -- 172.21.15.1:0/1955999716 shutdown_connections 2024-03-18T19:08:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.596+0000 7f6298069b80 1 --2- 172.21.15.1:0/1955999716 >> [v2:172.21.15.165:6833/450810178,v1:172.21.15.165:6835/450810178] conn(0x55ededcd0150 0x55ededcd0530 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.596+0000 7f6298069b80 1 --2- 172.21.15.1:0/1955999716 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ededc1f0a0 0x55ededc510b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.596+0000 7f6298069b80 1 --2- 172.21.15.1:0/1955999716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ededc5f630 0x55ededccfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.596+0000 7f6264ff9700 0 client.8433 ms_handle_reset on v2:172.21.15.165:6833/450810178 2024-03-18T19:08:25.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.596+0000 7f6298069b80 1 --2- 172.21.15.1:0/1955999716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ededc4dc10 0x55ededc515f0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.596+0000 7f6298069b80 1 -- 172.21.15.1:0/1955999716 >> 172.21.15.1:0/1955999716 conn(0x55ededb81cf0 msgr2=0x55ededc1df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:25.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.596+0000 7f6298069b80 1 -- 172.21.15.1:0/1955999716 shutdown_connections 2024-03-18T19:08:25.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.596+0000 7f6298069b80 1 -- 172.21.15.1:0/1955999716 wait complete. 2024-03-18T19:08:25.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.597+0000 7f6298069b80 20 client.8433 trim_cache size 0 max 16384 2024-03-18T19:08:25.599 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:08:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "9c1c697f-8655-4ef3-8a13-da568d2cbe86", 2024-03-18T19:08:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-18T19:08:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 7675, 2024-03-18T19:08:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-18T19:08:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-18T19:08:25.600 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-18T19:08:25.600 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 6.0102947369999997, 2024-03-18T19:08:25.600 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 57, 2024-03-18T19:08:25.600 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 133, 2024-03-18T19:08:25.600 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 133, 2024-03-18T19:08:25.600 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 11.334127559000001 2024-03-18T19:08:25.600 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:08:25.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.597+0000 7f628bec0700 1 -- 172.21.15.1:0/1719973060 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f626806adb0 msgr2=0x7f626806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:25.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.597+0000 7f628bec0700 1 --2- 172.21.15.1:0/1719973060 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f626806adb0 0x7f626806d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6280000c00 tx=0x7f6280008030 comp rx=0 tx=0).stop 2024-03-18T19:08:25.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.597+0000 7f628bec0700 1 -- 172.21.15.1:0/1719973060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f628415d5b0 msgr2=0x7f6284159fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:25.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.597+0000 7f628bec0700 1 --2- 172.21.15.1:0/1719973060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f628415d5b0 0x7f6284159fa0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f627c007e60 tx=0x7f627c00e3c0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.597+0000 7f628bec0700 1 -- 172.21.15.1:0/1719973060 shutdown_connections 2024-03-18T19:08:25.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.597+0000 7f628bec0700 1 --2- 172.21.15.1:0/1719973060 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f626806adb0 0x7f626806d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.597+0000 7f628bec0700 1 --2- 172.21.15.1:0/1719973060 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6284099a40 0x7f628414f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.597+0000 7f628bec0700 1 --2- 172.21.15.1:0/1719973060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f628415d5b0 0x7f6284159fa0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.597+0000 7f628bec0700 1 --2- 172.21.15.1:0/1719973060 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f628409a3e0 0x7f62840b1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.597+0000 7f628bec0700 1 -- 172.21.15.1:0/1719973060 >> 172.21.15.1:0/1719973060 conn(0x7f62840b2bd0 msgr2=0x7f62840b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.597+0000 7f628bec0700 1 -- 172.21.15.1:0/1719973060 shutdown_connections 2024-03-18T19:08:25.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.597+0000 7f628bec0700 1 -- 172.21.15.1:0/1719973060 wait complete. 2024-03-18T19:08:25.605 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T19:08:25.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.735+0000 7f546017a700 1 Processor -- start 2024-03-18T19:08:25.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.735+0000 7f546017a700 1 -- start start 2024-03-18T19:08:25.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.736+0000 7f546017a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54580b6b60 0x7f54580cc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.736+0000 7f546017a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54580cee90 0x7f54580cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.736+0000 7f546017a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54580cd0d0 con 0x7f54580cfbd0 2024-03-18T19:08:25.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.736+0000 7f546017a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54580d0060 con 0x7f54580cee90 2024-03-18T19:08:25.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.736+0000 7f546017a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54580b3f00 con 0x7f54580b6b60 2024-03-18T19:08:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.736+0000 7f545d715700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54580cee90 0x7f54580cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.736+0000 7f545d715700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54580cee90 0x7f54580cf690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57172/0 (socket says 172.21.15.1:57172) 2024-03-18T19:08:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.736+0000 7f545d715700 1 -- 172.21.15.1:0/1727804008 learned_addr learned my addr 172.21.15.1:0/1727804008 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.736+0000 7f545df16700 1 --2- 172.21.15.1:0/1727804008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54580b6b60 0x7f54580cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.736+0000 7f545d715700 1 -- 172.21.15.1:0/1727804008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54580b6b60 msgr2=0x7f54580cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.736+0000 7f545d715700 1 --2- 172.21.15.1:0/1727804008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54580b6b60 0x7f54580cc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.736+0000 7f545d715700 1 -- 172.21.15.1:0/1727804008 >> v1:172.21.15.165:6789/0 conn(0x7f54580cfbd0 legacy=0x7f5458099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.736+0000 7f545d715700 1 -- 172.21.15.1:0/1727804008 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54580b41e0 con 0x7f54580cee90 2024-03-18T19:08:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.736+0000 7f545d715700 1 --2- 172.21.15.1:0/1727804008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54580cee90 0x7f54580cf690 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f54480098f0 tx=0x7f5448009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bd7f3302ddf92bd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.736+0000 7f545cf14700 1 -- 172.21.15.1:0/1727804008 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5448011830 con 0x7f54580cee90 2024-03-18T19:08:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.736+0000 7f545cf14700 1 -- 172.21.15.1:0/1727804008 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5448011990 con 0x7f54580cee90 2024-03-18T19:08:25.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.736+0000 7f545cf14700 1 -- 172.21.15.1:0/1727804008 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5448011d20 con 0x7f54580cee90 2024-03-18T19:08:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.737+0000 7f546017a700 1 -- 172.21.15.1:0/1727804008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54580cee90 msgr2=0x7f54580cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.737+0000 7f546017a700 1 --2- 172.21.15.1:0/1727804008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54580cee90 0x7f54580cf690 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f54480098f0 tx=0x7f5448009c00 comp rx=0 tx=0).stop 2024-03-18T19:08:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.737+0000 7f546017a700 1 -- 172.21.15.1:0/1727804008 shutdown_connections 2024-03-18T19:08:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.737+0000 7f546017a700 1 --2- 172.21.15.1:0/1727804008 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54580b6b60 0x7f54580cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.737+0000 7f546017a700 1 --2- 172.21.15.1:0/1727804008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54580cee90 0x7f54580cf690 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.737+0000 7f546017a700 1 -- 172.21.15.1:0/1727804008 >> 172.21.15.1:0/1727804008 conn(0x7f54580b2ba0 msgr2=0x7f54580b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.737+0000 7f546017a700 1 -- 172.21.15.1:0/1727804008 shutdown_connections 2024-03-18T19:08:25.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.737+0000 7f546017a700 1 -- 172.21.15.1:0/1727804008 wait complete. 2024-03-18T19:08:25.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.738+0000 7f546017a700 1 Processor -- start 2024-03-18T19:08:25.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.738+0000 7f546017a700 1 -- start start 2024-03-18T19:08:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.739+0000 7f546017a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54580b6b60 0x7f545809d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.739+0000 7f546017a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54580cee90 0x7f54580c5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.739+0000 7f546017a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54580cfbd0 0x7f54581f01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.739+0000 7f546017a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545809e860 con 0x7f54580b6b60 2024-03-18T19:08:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.739+0000 7f546017a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54581f0710 con 0x7f54580cfbd0 2024-03-18T19:08:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.739+0000 7f546017a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f545809dc50 con 0x7f54580cee90 2024-03-18T19:08:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.739+0000 7f545df16700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54580b6b60 0x7f545809d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.739+0000 7f545df16700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54580b6b60 0x7f545809d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57176/0 (socket says 172.21.15.1:57176) 2024-03-18T19:08:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.739+0000 7f545df16700 1 -- 172.21.15.1:0/1333754504 learned_addr learned my addr 172.21.15.1:0/1333754504 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.739+0000 7f545d715700 1 --2- 172.21.15.1:0/1333754504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54580cee90 0x7f54580c5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.739+0000 7f545e717700 1 --2- 172.21.15.1:0/1333754504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54580cfbd0 0x7f54581f01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.739+0000 7f545df16700 1 -- 172.21.15.1:0/1333754504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54580cee90 msgr2=0x7f54580c5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.739+0000 7f545df16700 1 --2- 172.21.15.1:0/1333754504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54580cee90 0x7f54580c5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.739+0000 7f545df16700 1 -- 172.21.15.1:0/1333754504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54580cfbd0 msgr2=0x7f54581f01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.739+0000 7f545df16700 1 --2- 172.21.15.1:0/1333754504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54580cfbd0 0x7f54581f01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.739+0000 7f545df16700 1 -- 172.21.15.1:0/1333754504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5448009550 con 0x7f54580b6b60 2024-03-18T19:08:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.740+0000 7f545df16700 1 --2- 172.21.15.1:0/1333754504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54580b6b60 0x7f545809d5e0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f545400e9b0 tx=0x7f545400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.740+0000 7f544e7fc700 1 -- 172.21.15.1:0/1333754504 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5454009710 con 0x7f54580b6b60 2024-03-18T19:08:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.740+0000 7f544e7fc700 1 -- 172.21.15.1:0/1333754504 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545400ee50 con 0x7f54580b6b60 2024-03-18T19:08:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.740+0000 7f546017a700 1 -- 172.21.15.1:0/1333754504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f545809df30 con 0x7f54580b6b60 2024-03-18T19:08:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.740+0000 7f544e7fc700 1 -- 172.21.15.1:0/1333754504 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545400ca50 con 0x7f54580b6b60 2024-03-18T19:08:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.740+0000 7f546017a700 1 -- 172.21.15.1:0/1333754504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f545809e4f0 con 0x7f54580b6b60 2024-03-18T19:08:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.741+0000 7f546017a700 1 -- 172.21.15.1:0/1333754504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f545809eb40 con 0x7f54580b6b60 2024-03-18T19:08:25.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.741+0000 7f544e7fc700 1 -- 172.21.15.1:0/1333754504 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f545400cbb0 con 0x7f54580b6b60 2024-03-18T19:08:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.742+0000 7f544e7fc700 1 --2- 172.21.15.1:0/1333754504 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f543c06adb0 0x7f543c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.742+0000 7f544e7fc700 1 -- 172.21.15.1:0/1333754504 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5454089590 con 0x7f54580b6b60 2024-03-18T19:08:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.742+0000 7f545d715700 1 --2- 172.21.15.1:0/1333754504 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f543c06adb0 0x7f543c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:25.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.742+0000 7f545d715700 1 --2- 172.21.15.1:0/1333754504 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f543c06adb0 0x7f543c06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5448009c40 tx=0x7f5448008030 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:25.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.746+0000 7f544e7fc700 1 -- 172.21.15.1:0/1333754504 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f54540545b0 con 0x7f54580b6b60 2024-03-18T19:08:25.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.994+0000 7f546017a700 1 -- 172.21.15.1:0/1333754504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f54580676a0 con 0x7f54580b6b60 2024-03-18T19:08:25.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.994+0000 7f544e7fc700 1 -- 172.21.15.1:0/1333754504 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v57) v1 ==== 53+0+73 (secure 0 0 0) 0x7f545400ce60 con 0x7f54580b6b60 2024-03-18T19:08:25.990 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T19:08:25.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.996+0000 7f546017a700 1 -- 172.21.15.1:0/1333754504 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f543c06adb0 msgr2=0x7f543c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:25.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.996+0000 7f546017a700 1 --2- 172.21.15.1:0/1333754504 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f543c06adb0 0x7f543c06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5448009c40 tx=0x7f5448008030 comp rx=0 tx=0).stop 2024-03-18T19:08:25.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.996+0000 7f546017a700 1 -- 172.21.15.1:0/1333754504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54580b6b60 msgr2=0x7f545809d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:25.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.996+0000 7f546017a700 1 --2- 172.21.15.1:0/1333754504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54580b6b60 0x7f545809d5e0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f545400e9b0 tx=0x7f545400ecc0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.996+0000 7f546017a700 1 -- 172.21.15.1:0/1333754504 shutdown_connections 2024-03-18T19:08:25.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.996+0000 7f546017a700 1 --2- 172.21.15.1:0/1333754504 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f543c06adb0 0x7f543c06d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.996+0000 7f546017a700 1 --2- 172.21.15.1:0/1333754504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54580cfbd0 0x7f54581f01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.996+0000 7f546017a700 1 --2- 172.21.15.1:0/1333754504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54580b6b60 0x7f545809d5e0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.996+0000 7f546017a700 1 --2- 172.21.15.1:0/1333754504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54580cee90 0x7f54580c5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:25.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.996+0000 7f546017a700 1 -- 172.21.15.1:0/1333754504 >> 172.21.15.1:0/1333754504 conn(0x7f54580b2ba0 msgr2=0x7f545806fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:25.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.996+0000 7f546017a700 1 -- 172.21.15.1:0/1333754504 shutdown_connections 2024-03-18T19:08:25.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:25.996+0000 7f546017a700 1 -- 172.21.15.1:0/1333754504 wait complete. 2024-03-18T19:08:26.004 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T19:08:26.004 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T19:08:26.004 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-18T19:08:26.004 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T19:08:26.005 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T19:08:26.005 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T19:08:26.005 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T19:08:26.005 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T19:08:26.005 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:08:26.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:26.005 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-18T19:08:26.024 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T19:08:26.024 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T19:08:26.024 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T19:08:26.024 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:08:26.024 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T19:08:26.024 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:08:26.025 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:08:26.025 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-18T19:08:26.025 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-18T19:08:26.025 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:08:26.025 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-18T19:08:26.025 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-18T19:08:26.025 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T19:08:26.025 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:08:26.025 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-18T19:08:26.025 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T19:08:26.025 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 41463sec preferred_lft 41463sec 2024-03-18T19:08:26.025 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link 2024-03-18T19:08:26.025 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:08:26.025 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:08:26.026 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:08:26.026 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-18T19:08:26.026 INFO:teuthology.orchestra.run.smithi001.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T19:08:26.026 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:08:26.026 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T19:08:26.026 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:08:26.026 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::4ca9:f5ff:feb3:79a3/64 scope link 2024-03-18T19:08:26.026 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:08:26.026 INFO:teuthology.orchestra.run.smithi001.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T19:08:26.026 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T19:08:26.026 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::282f:a4ff:fe1a:9908/64 scope link 2024-03-18T19:08:26.026 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:08:26.026 INFO:teuthology.orchestra.run.smithi001.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T19:08:26.027 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2e:5c:60:78:b2:8c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T19:08:26.027 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::2c5c:60ff:fe78:b28c/64 scope link 2024-03-18T19:08:26.027 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:08:26.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:26.027 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T19:08:26.091 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-18T19:08:26.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:26.092 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-18T19:08:26.153 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-18T19:08:26.153 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T19:08:26.154 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T19:08:26.154 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T19:08:26.154 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T19:08:26.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:26.154 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T19:08:26.154 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T19:08:26.154 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T19:08:26.154 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T19:08:26.154 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T19:08:26.155 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T19:08:26.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:26.260 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T19:08:26.278 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T19:08:26.278 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T19:08:26.279 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:08:26.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:26.279 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-18T19:08:26.346 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T19:08:26.346 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T19:08:26.346 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T19:08:26.404 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:08:26.404 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T19:08:26.404 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T19:08:26.461 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T19:08:26.528 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T19:08:26.528 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T19:08:26.595 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T19:08:26.596 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T19:08:26.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:26.597 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T19:08:26.656 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T19:08:26.656 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T19:08:26.698 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T19:08:26.725 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T19:08:26.726 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T19:08:26.788 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-18T19:08:26.792+0000 7fb41fb3a5c0 -1 init, newargv = 0x55b749ac2fa0 newargc=15 2024-03-18T19:08:26.788 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[124790]: starting ceph client 2024-03-18T19:08:26.793 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[124790]: starting fuse 2024-03-18T19:08:26.797 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T19:08:26.798 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T19:08:26.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:26.799 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T19:08:26.857 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T19:08:26.857 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-03-18T19:08:26.857 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T19:08:26.857 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-18T19:08:26.857 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-18T19:08:26.857 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-18T19:08:26.857 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-18T19:08:26.857 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:08:26.858 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-18T19:08:26.858 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T19:08:26.858 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-18T19:08:26.858 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T19:08:26.858 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:08:26.858 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-18T19:08:26.858 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-18T19:08:26.858 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-18T19:08:26.858 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-18T19:08:26.858 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:08:26.858 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-18T19:08:26.858 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T19:08:26.858 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T19:08:26.858 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T19:08:26.858 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-18T19:08:26.858 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-18T19:08:26.859 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-18T19:08:26.859 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T19:08:26.859 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:08:26.859 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-18T19:08:26.859 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-18T19:08:26.982 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.124790.asok 2024-03-18T19:08:27.163 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.124790.asok 2024-03-18T19:08:27.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:27.164 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.124790.asok status 2024-03-18T19:08:27.324 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:08:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-18T19:08:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T19:08:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T19:08:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-18T19:08:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-18T19:08:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T19:08:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "124790", 2024-03-18T19:08:27.324 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-18T19:08:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-18T19:08:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-18T19:08:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 8448, 2024-03-18T19:08:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-18T19:08:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-18T19:08:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-18T19:08:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8448 2024-03-18T19:08:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T19:08:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T19:08:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T19:08:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2128494678 2024-03-18T19:08:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:27.325 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T19:08:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T19:08:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T19:08:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2128494678 2024-03-18T19:08:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:08:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.8448 192.168.0.1:0/2128494678", 2024-03-18T19:08:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/2128494678", 2024-03-18T19:08:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-18T19:08:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 57, 2024-03-18T19:08:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 133, 2024-03-18T19:08:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-18T19:08:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-18T19:08:27.326 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-18T19:08:27.326 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:08:27.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:27.335 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-18T19:08:27.335 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T19:08:27.353 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-18T19:08:27.353 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T19:08:27.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:27.354 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T19:08:27.422 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-18T19:08:27.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.593+0000 7fcb0a7a2700 1 Processor -- start 2024-03-18T19:08:27.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.593+0000 7fcb0a7a2700 1 -- start start 2024-03-18T19:08:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.594+0000 7fcb0a7a2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb040cdd90 0x7fcb0406a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.594+0000 7fcb0a7a2700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb04071520 con 0x7fcb04072250 2024-03-18T19:08:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.594+0000 7fcb0a7a2700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb0406f360 con 0x7fcb040af5d0 2024-03-18T19:08:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.594+0000 7fcb0a7a2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb040af9b0 con 0x7fcb040cdd90 2024-03-18T19:08:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.594+0000 7fcb09fa1700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcb04072250 0x7fcb0415cc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60888/0 (socket says 172.21.15.1:60888) 2024-03-18T19:08:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.594+0000 7fcb09fa1700 1 -- 172.21.15.1:0/3672790561 learned_addr learned my addr 172.21.15.1:0/3672790561 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.594+0000 7fcb08f9f700 1 --2- 172.21.15.1:0/3672790561 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb040cdd90 0x7fcb0406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.594+0000 7fcafbfff700 1 -- 172.21.15.1:0/3672790561 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875262601 0 0) 0x7fcb04071520 con 0x7fcb04072250 2024-03-18T19:08:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.594+0000 7fcafbfff700 1 -- 172.21.15.1:0/3672790561 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf00035b0 con 0x7fcb04072250 2024-03-18T19:08:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.594+0000 7fcafbfff700 1 -- 172.21.15.1:0/3672790561 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918598106 0 0) 0x7fcb0406f360 con 0x7fcb040af5d0 2024-03-18T19:08:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.594+0000 7fcafbfff700 1 -- 172.21.15.1:0/3672790561 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb04071520 con 0x7fcb040af5d0 2024-03-18T19:08:27.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.594+0000 7fcb08f9f700 1 -- 172.21.15.1:0/3672790561 >> v1:172.21.15.165:6789/0 conn(0x7fcb040af5d0 legacy=0x7fcb0406e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.594+0000 7fcb08f9f700 1 -- 172.21.15.1:0/3672790561 >> v1:172.21.15.1:6789/0 conn(0x7fcb04072250 legacy=0x7fcb0415cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.594+0000 7fcb08f9f700 1 -- 172.21.15.1:0/3672790561 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb04071770 con 0x7fcb040cdd90 2024-03-18T19:08:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.595+0000 7fcb08f9f700 1 --2- 172.21.15.1:0/3672790561 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb040cdd90 0x7fcb0406a750 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009870 tx=0x7fcb00009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10ed897bf20484dc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.595+0000 7fcafbfff700 1 -- 172.21.15.1:0/3672790561 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb00011830 con 0x7fcb040cdd90 2024-03-18T19:08:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.595+0000 7fcafbfff700 1 -- 172.21.15.1:0/3672790561 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb00011990 con 0x7fcb040cdd90 2024-03-18T19:08:27.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.595+0000 7fcafbfff700 1 -- 172.21.15.1:0/3672790561 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb00011d20 con 0x7fcb040cdd90 2024-03-18T19:08:27.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.595+0000 7fcb0a7a2700 1 -- 172.21.15.1:0/3672790561 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb040cdd90 msgr2=0x7fcb0406a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:27.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.595+0000 7fcb0a7a2700 1 --2- 172.21.15.1:0/3672790561 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb040cdd90 0x7fcb0406a750 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009870 tx=0x7fcb00009b80 comp rx=0 tx=0).stop 2024-03-18T19:08:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.595+0000 7fcb0a7a2700 1 -- 172.21.15.1:0/3672790561 shutdown_connections 2024-03-18T19:08:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.595+0000 7fcb0a7a2700 1 --2- 172.21.15.1:0/3672790561 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb040cdd90 0x7fcb0406a750 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.595+0000 7fcb0a7a2700 1 -- 172.21.15.1:0/3672790561 >> 172.21.15.1:0/3672790561 conn(0x7fcb040cca30 msgr2=0x7fcb040cce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.596+0000 7fcb0a7a2700 1 -- 172.21.15.1:0/3672790561 shutdown_connections 2024-03-18T19:08:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.596+0000 7fcb0a7a2700 1 -- 172.21.15.1:0/3672790561 wait complete. 2024-03-18T19:08:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.597+0000 7fcb0a7a2700 1 Processor -- start 2024-03-18T19:08:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.597+0000 7fcb0a7a2700 1 -- start start 2024-03-18T19:08:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.597+0000 7fcb0a7a2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb04072250 0x7fcb04145d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.597+0000 7fcb0a7a2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb040af5d0 0x7fcb040c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.597+0000 7fcb0a7a2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb040c92e0 0x7fcb040acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.597+0000 7fcb0a7a2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb040ca340 con 0x7fcb040c92e0 2024-03-18T19:08:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.597+0000 7fcb0a7a2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb040c97c0 con 0x7fcb04072250 2024-03-18T19:08:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.597+0000 7fcb0a7a2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb040c9aa0 con 0x7fcb040af5d0 2024-03-18T19:08:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.597+0000 7fcb09fa1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb040c92e0 0x7fcb040acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:27.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.598+0000 7fcb09fa1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb040c92e0 0x7fcb040acdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57190/0 (socket says 172.21.15.1:57190) 2024-03-18T19:08:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.598+0000 7fcb09fa1700 1 -- 172.21.15.1:0/1333173585 learned_addr learned my addr 172.21.15.1:0/1333173585 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.598+0000 7fcb097a0700 1 --2- 172.21.15.1:0/1333173585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb04072250 0x7fcb04145d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.598+0000 7fcb08f9f700 1 --2- 172.21.15.1:0/1333173585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb040af5d0 0x7fcb040c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.598+0000 7fcb09fa1700 1 -- 172.21.15.1:0/1333173585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb040af5d0 msgr2=0x7fcb040c7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.598+0000 7fcb09fa1700 1 --2- 172.21.15.1:0/1333173585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb040af5d0 0x7fcb040c7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.598+0000 7fcb09fa1700 1 -- 172.21.15.1:0/1333173585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb04072250 msgr2=0x7fcb04145d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.598+0000 7fcb09fa1700 1 --2- 172.21.15.1:0/1333173585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb04072250 0x7fcb04145d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.598+0000 7fcb09fa1700 1 -- 172.21.15.1:0/1333173585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb000094a0 con 0x7fcb040c92e0 2024-03-18T19:08:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.598+0000 7fcb08f9f700 1 --2- 172.21.15.1:0/1333173585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb040af5d0 0x7fcb040c7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:08:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.598+0000 7fcb09fa1700 1 --2- 172.21.15.1:0/1333173585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb040c92e0 0x7fcb040acdf0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fcafc00cba0 tx=0x7fcafc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.598+0000 7fcaf9ffb700 1 -- 172.21.15.1:0/1333173585 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcafc00ab20 con 0x7fcb040c92e0 2024-03-18T19:08:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.598+0000 7fcaf9ffb700 1 -- 172.21.15.1:0/1333173585 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcafc00ac80 con 0x7fcb040c92e0 2024-03-18T19:08:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.598+0000 7fcb0a7a2700 1 -- 172.21.15.1:0/1333173585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb0406f130 con 0x7fcb040c92e0 2024-03-18T19:08:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.598+0000 7fcaf9ffb700 1 -- 172.21.15.1:0/1333173585 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcafc012400 con 0x7fcb040c92e0 2024-03-18T19:08:27.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.598+0000 7fcb0a7a2700 1 -- 172.21.15.1:0/1333173585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb040ca060 con 0x7fcb040c92e0 2024-03-18T19:08:27.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.600+0000 7fcb0a7a2700 1 -- 172.21.15.1:0/1333173585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb041464f0 con 0x7fcb040c92e0 2024-03-18T19:08:27.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.600+0000 7fcaf9ffb700 1 -- 172.21.15.1:0/1333173585 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcafc01b3b0 con 0x7fcb040c92e0 2024-03-18T19:08:27.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.600+0000 7fcaf9ffb700 1 --2- 172.21.15.1:0/1333173585 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcae806adb0 0x7fcae806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:27.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.600+0000 7fcaf9ffb700 1 -- 172.21.15.1:0/1333173585 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fcafc089390 con 0x7fcb040c92e0 2024-03-18T19:08:27.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.600+0000 7fcb097a0700 1 --2- 172.21.15.1:0/1333173585 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcae806adb0 0x7fcae806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:27.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.601+0000 7fcb097a0700 1 --2- 172.21.15.1:0/1333173585 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcae806adb0 0x7fcae806d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcaf40011e0 tx=0x7fcaf4009040 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:27.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.605+0000 7fcaf9ffb700 1 -- 172.21.15.1:0/1333173585 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcb041464f0 con 0x7fcb040c92e0 2024-03-18T19:08:27.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.839+0000 7fcb0a7a2700 1 -- 172.21.15.1:0/1333173585 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb0406efb0 con 0x7fcae806adb0 2024-03-18T19:08:27.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.840+0000 7fcaf9ffb700 1 -- 172.21.15.1:0/1333173585 <== mgr.8061 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fcaf40050d0 con 0x7fcae806adb0 2024-03-18T19:08:27.836 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T19:08:27.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:08:27.836 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephfs" 2024-03-18T19:08:27.836 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:08:27.836 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T19:08:27.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.842+0000 7fcb0a7a2700 1 -- 172.21.15.1:0/1333173585 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcae806adb0 msgr2=0x7fcae806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:27.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.842+0000 7fcb0a7a2700 1 --2- 172.21.15.1:0/1333173585 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcae806adb0 0x7fcae806d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcaf40011e0 tx=0x7fcaf4009040 comp rx=0 tx=0).stop 2024-03-18T19:08:27.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.842+0000 7fcb0a7a2700 1 -- 172.21.15.1:0/1333173585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb040c92e0 msgr2=0x7fcb040acdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:27.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.842+0000 7fcb0a7a2700 1 --2- 172.21.15.1:0/1333173585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb040c92e0 0x7fcb040acdf0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fcafc00cba0 tx=0x7fcafc00cf60 comp rx=0 tx=0).stop 2024-03-18T19:08:27.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.843+0000 7fcb0a7a2700 1 -- 172.21.15.1:0/1333173585 shutdown_connections 2024-03-18T19:08:27.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.843+0000 7fcb0a7a2700 1 --2- 172.21.15.1:0/1333173585 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcae806adb0 0x7fcae806d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:27.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.843+0000 7fcb0a7a2700 1 --2- 172.21.15.1:0/1333173585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb040c92e0 0x7fcb040acdf0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:27.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.843+0000 7fcb0a7a2700 1 --2- 172.21.15.1:0/1333173585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb04072250 0x7fcb04145d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:27.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.843+0000 7fcb0a7a2700 1 --2- 172.21.15.1:0/1333173585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb040af5d0 0x7fcb040c7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:27.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.843+0000 7fcb0a7a2700 1 -- 172.21.15.1:0/1333173585 >> 172.21.15.1:0/1333173585 conn(0x7fcb040cca30 msgr2=0x7fcb0406c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:27.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.843+0000 7fcb0a7a2700 1 -- 172.21.15.1:0/1333173585 shutdown_connections 2024-03-18T19:08:27.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.843+0000 7fcb0a7a2700 1 -- 172.21.15.1:0/1333173585 wait complete. 2024-03-18T19:08:27.850 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-18T19:08:27.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.981+0000 7f46a271e700 1 Processor -- start 2024-03-18T19:08:27.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.981+0000 7f46a271e700 1 -- start start 2024-03-18T19:08:27.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.982+0000 7f46a271e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f469c0cfa90 0x7f469c06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:27.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.982+0000 7f46a271e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c0cf600 con 0x7f469c06ac90 2024-03-18T19:08:27.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.982+0000 7f46a271e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c0ce720 con 0x7f469c15d5a0 2024-03-18T19:08:27.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.982+0000 7f46a271e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f469c06b140 con 0x7f469c0cfa90 2024-03-18T19:08:27.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.982+0000 7f46a0cbb700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f469c06ac90 0x7f469c167140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60900/0 (socket says 172.21.15.1:60900) 2024-03-18T19:08:27.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.982+0000 7f46a0cbb700 1 -- 172.21.15.1:0/620137714 learned_addr learned my addr 172.21.15.1:0/620137714 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:27.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469affd700 1 -- 172.21.15.1:0/620137714 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863233025 0 0) 0x7f469c0cf600 con 0x7f469c06ac90 2024-03-18T19:08:27.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469b7fe700 1 --2- 172.21.15.1:0/620137714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f469c0cfa90 0x7f469c06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:27.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469affd700 1 -- 172.21.15.1:0/620137714 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46880035b0 con 0x7f469c06ac90 2024-03-18T19:08:27.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469affd700 1 -- 172.21.15.1:0/620137714 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1944570434 0 0) 0x7f46880035b0 con 0x7f469c06ac90 2024-03-18T19:08:27.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469affd700 1 -- 172.21.15.1:0/620137714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f469c0cfa90 msgr2=0x7f469c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:27.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469affd700 1 --2- 172.21.15.1:0/620137714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f469c0cfa90 0x7f469c06a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:27.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469affd700 1 -- 172.21.15.1:0/620137714 >> v1:172.21.15.165:6789/0 conn(0x7f469c15d5a0 legacy=0x7f469c0ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469affd700 1 -- 172.21.15.1:0/620137714 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469c0ce9f0 con 0x7f469c06ac90 2024-03-18T19:08:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469affd700 1 -- 172.21.15.1:0/620137714 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f46940019f0 con 0x7f469c06ac90 2024-03-18T19:08:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469affd700 1 -- 172.21.15.1:0/620137714 >> v1:172.21.15.1:6789/0 conn(0x7f469c06ac90 legacy=0x7f469c167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469affd700 1 --2- 172.21.15.1:0/620137714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f469c0cfa90 0x7f4688003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469affd700 1 --2- 172.21.15.1:0/620137714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4688003f70 0x7f4688004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469affd700 1 --2- 172.21.15.1:0/620137714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4688004d90 0x7f4688005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469affd700 1 -- 172.21.15.1:0/620137714 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c06b140 con 0x7f4688003f70 2024-03-18T19:08:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469affd700 1 -- 172.21.15.1:0/620137714 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468800ba70 con 0x7f4688004d90 2024-03-18T19:08:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469affd700 1 -- 172.21.15.1:0/620137714 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468800bc30 con 0x7f469c0cfa90 2024-03-18T19:08:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469b7fe700 1 --2- 172.21.15.1:0/620137714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4688003f70 0x7f4688004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469bfff700 1 --2- 172.21.15.1:0/620137714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f469c0cfa90 0x7f4688003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f46a0cbb700 1 --2- 172.21.15.1:0/620137714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4688004d90 0x7f4688005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:27.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469b7fe700 1 -- 172.21.15.1:0/620137714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f469c0cfa90 msgr2=0x7f4688003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469b7fe700 1 --2- 172.21.15.1:0/620137714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f469c0cfa90 0x7f4688003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469b7fe700 1 -- 172.21.15.1:0/620137714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4688004d90 msgr2=0x7f4688005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469b7fe700 1 --2- 172.21.15.1:0/620137714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4688004d90 0x7f4688005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469b7fe700 1 -- 172.21.15.1:0/620137714 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46880035b0 con 0x7f4688003f70 2024-03-18T19:08:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.983+0000 7f469bfff700 1 --2- 172.21.15.1:0/620137714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f469c0cfa90 0x7f4688003a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:08:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.984+0000 7f469b7fe700 1 --2- 172.21.15.1:0/620137714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4688003f70 0x7f4688004850 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f4684009b60 tx=0x7f4688005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caddc4fa1feac23 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.984+0000 7f469affd700 1 -- 172.21.15.1:0/620137714 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468400e6c0 con 0x7f4688003f70 2024-03-18T19:08:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.984+0000 7f469affd700 1 -- 172.21.15.1:0/620137714 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468400e820 con 0x7f4688003f70 2024-03-18T19:08:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.984+0000 7f469affd700 1 -- 172.21.15.1:0/620137714 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468400eb90 con 0x7f4688003f70 2024-03-18T19:08:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.984+0000 7f46a271e700 1 -- 172.21.15.1:0/620137714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4688003f70 msgr2=0x7f4688004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.984+0000 7f46a271e700 1 --2- 172.21.15.1:0/620137714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4688003f70 0x7f4688004850 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f4684009b60 tx=0x7f4688005880 comp rx=0 tx=0).stop 2024-03-18T19:08:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.984+0000 7f46a0cbb700 1 -- 172.21.15.1:0/620137714 reap_dead start 2024-03-18T19:08:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.984+0000 7f46a271e700 1 -- 172.21.15.1:0/620137714 shutdown_connections 2024-03-18T19:08:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.984+0000 7f46a271e700 1 -- 172.21.15.1:0/620137714 >> 172.21.15.1:0/620137714 conn(0x7f469c0cc9b0 msgr2=0x7f469c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:27.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.984+0000 7f46a271e700 1 -- 172.21.15.1:0/620137714 shutdown_connections 2024-03-18T19:08:27.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.984+0000 7f46a271e700 1 -- 172.21.15.1:0/620137714 wait complete. 2024-03-18T19:08:27.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.986+0000 7f46a271e700 1 Processor -- start 2024-03-18T19:08:27.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.986+0000 7f46a271e700 1 -- start start 2024-03-18T19:08:27.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f46a271e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4688003f70 0x7f469c0b3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:27.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f46a271e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c154d50 0x7f469c0c6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:27.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f46a271e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f469c15ac50 0x7f469c15b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:27.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f46a271e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c15b640 con 0x7f469c154d50 2024-03-18T19:08:27.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f46a271e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c0b3b80 con 0x7f469c15ac50 2024-03-18T19:08:27.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f46a271e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f469c0b3e60 con 0x7f4688003f70 2024-03-18T19:08:27.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f469b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c154d50 0x7f469c0c6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:27.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f469b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c154d50 0x7f469c0c6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57202/0 (socket says 172.21.15.1:57202) 2024-03-18T19:08:27.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f469b7fe700 1 -- 172.21.15.1:0/1368684585 learned_addr learned my addr 172.21.15.1:0/1368684585 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:27.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f46a0cbb700 1 --2- 172.21.15.1:0/1368684585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f469c15ac50 0x7f469c15b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:27.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f469bfff700 1 --2- 172.21.15.1:0/1368684585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4688003f70 0x7f469c0b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:27.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f469b7fe700 1 -- 172.21.15.1:0/1368684585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4688003f70 msgr2=0x7f469c0b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:27.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f469b7fe700 1 --2- 172.21.15.1:0/1368684585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4688003f70 0x7f469c0b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:27.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f469b7fe700 1 -- 172.21.15.1:0/1368684585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f469c15ac50 msgr2=0x7f469c15b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:27.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f469b7fe700 1 --2- 172.21.15.1:0/1368684585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f469c15ac50 0x7f469c15b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:27.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f469b7fe700 1 -- 172.21.15.1:0/1368684585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469c0b4140 con 0x7f469c154d50 2024-03-18T19:08:27.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f469b7fe700 1 --2- 172.21.15.1:0/1368684585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c154d50 0x7f469c0c6750 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f46840090b0 tx=0x7f46840047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:27.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f4698ff9700 1 -- 172.21.15.1:0/1368684585 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4684021070 con 0x7f469c154d50 2024-03-18T19:08:27.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f4698ff9700 1 -- 172.21.15.1:0/1368684585 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4684009ed0 con 0x7f469c154d50 2024-03-18T19:08:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f4698ff9700 1 -- 172.21.15.1:0/1368684585 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468400e6c0 con 0x7f469c154d50 2024-03-18T19:08:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f46a271e700 1 -- 172.21.15.1:0/1368684585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f469c0b4420 con 0x7f469c154d50 2024-03-18T19:08:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.987+0000 7f46a271e700 1 -- 172.21.15.1:0/1368684585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f469c1551d0 con 0x7f469c154d50 2024-03-18T19:08:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.988+0000 7f46a271e700 1 -- 172.21.15.1:0/1368684585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f469c0676a0 con 0x7f469c154d50 2024-03-18T19:08:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.989+0000 7f4698ff9700 1 -- 172.21.15.1:0/1368684585 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 60) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4684025070 con 0x7f469c154d50 2024-03-18T19:08:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.989+0000 7f4698ff9700 1 --2- 172.21.15.1:0/1368684585 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f467c06adb0 0x7f467c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.989+0000 7f4698ff9700 1 -- 172.21.15.1:0/1368684585 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f468408a590 con 0x7f469c154d50 2024-03-18T19:08:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.989+0000 7f469bfff700 1 --2- 172.21.15.1:0/1368684585 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f467c06adb0 0x7f467c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:27.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.990+0000 7f469bfff700 1 --2- 172.21.15.1:0/1368684585 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f467c06adb0 0x7f467c06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f468c007fd0 tx=0x7f468c009040 comp rx=0 tx=0).ready entity=mgr.8061 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:27.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:27.994+0000 7f4698ff9700 1 -- 172.21.15.1:0/1368684585 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f46840556d0 con 0x7f469c154d50 2024-03-18T19:08:28.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:28.257+0000 7f46a271e700 1 -- 172.21.15.1:0/1368684585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f469c06e960 con 0x7f469c154d50 2024-03-18T19:08:29.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.103+0000 7f4698ff9700 1 -- 172.21.15.1:0/1368684585 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 61) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f4684009700 con 0x7f469c154d50 2024-03-18T19:08:29.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.106+0000 7f4698ff9700 1 -- 172.21.15.1:0/1368684585 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v61) v1 ==== 92+0+0 (secure 0 0 0) 0x7f4684059160 con 0x7f469c154d50 2024-03-18T19:08:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.108+0000 7f46a271e700 1 -- 172.21.15.1:0/1368684585 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f467c06adb0 msgr2=0x7f467c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.108+0000 7f46a271e700 1 --2- 172.21.15.1:0/1368684585 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f467c06adb0 0x7f467c06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f468c007fd0 tx=0x7f468c009040 comp rx=0 tx=0).stop 2024-03-18T19:08:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.108+0000 7f46a271e700 1 -- 172.21.15.1:0/1368684585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c154d50 msgr2=0x7f469c0c6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.108+0000 7f46a271e700 1 --2- 172.21.15.1:0/1368684585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c154d50 0x7f469c0c6750 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f46840090b0 tx=0x7f46840047c0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.109+0000 7f46a271e700 1 -- 172.21.15.1:0/1368684585 shutdown_connections 2024-03-18T19:08:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.109+0000 7f46a271e700 1 --2- 172.21.15.1:0/1368684585 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f467c06adb0 0x7f467c06d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.109+0000 7f46a271e700 1 --2- 172.21.15.1:0/1368684585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4688003f70 0x7f469c0b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.109+0000 7f46a271e700 1 --2- 172.21.15.1:0/1368684585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f469c15ac50 0x7f469c15b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.109+0000 7f46a271e700 1 --2- 172.21.15.1:0/1368684585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f469c154d50 0x7f469c0c6750 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.109+0000 7f46a271e700 1 -- 172.21.15.1:0/1368684585 >> 172.21.15.1:0/1368684585 conn(0x7f469c0cc9b0 msgr2=0x7f469c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.109+0000 7f46a271e700 1 -- 172.21.15.1:0/1368684585 shutdown_connections 2024-03-18T19:08:29.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.109+0000 7f46a271e700 1 -- 172.21.15.1:0/1368684585 wait complete. 2024-03-18T19:08:29.119 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-18T19:08:29.242 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:29.246+0000 7fc0125d2240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:08:29.250 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:29.255+0000 7f433248b240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:08:29.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.257+0000 7f3acf0ad700 1 Processor -- start 2024-03-18T19:08:29.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.257+0000 7f3acf0ad700 1 -- start start 2024-03-18T19:08:29.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.258+0000 7f3acf0ad700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ac806eda0 0x7f3ac8161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:29.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.258+0000 7f3acf0ad700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3ac806f180 0x7f3ac80ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:29.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.258+0000 7f3acf0ad700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac80caac0 con 0x7f3ac806a750 2024-03-18T19:08:29.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.258+0000 7f3acf0ad700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ac80cb320 con 0x7f3ac806f180 2024-03-18T19:08:29.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.258+0000 7f3acf0ad700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac80ce010 con 0x7f3ac806eda0 2024-03-18T19:08:29.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.258+0000 7f3acce49700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ac806eda0 0x7f3ac8161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:29.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.258+0000 7f3ac7fff700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3ac806f180 0x7f3ac80ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:29.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.258+0000 7f3acce49700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ac806eda0 0x7f3ac8161c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:34310/0 (socket says 172.21.15.1:34310) 2024-03-18T19:08:29.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.258+0000 7f3acce49700 1 -- 172.21.15.1:0/1018339322 learned_addr learned my addr 172.21.15.1:0/1018339322 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:29.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.258+0000 7f3acd64a700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f3ac806a750 0x7f3ac806ab60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:53858/0 (socket says 172.21.15.1:53858) 2024-03-18T19:08:29.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.258+0000 7f3acce49700 1 -- 172.21.15.1:0/1018339322 >> v2:172.21.15.165:3300/0 conn(0x7f3ac806f180 msgr2=0x7f3ac80ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:08:29.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.258+0000 7f3acce49700 1 --2- 172.21.15.1:0/1018339322 >> v2:172.21.15.165:3300/0 conn(0x7f3ac806f180 0x7f3ac80ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.258+0000 7f3acce49700 1 -- 172.21.15.1:0/1018339322 >> v1:172.21.15.165:6789/0 conn(0x7f3ac806a750 legacy=0x7f3ac806ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:29.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.258+0000 7f3acce49700 1 -- 172.21.15.1:0/1018339322 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac80cad60 con 0x7f3ac806eda0 2024-03-18T19:08:29.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.258+0000 7f3ac77fe700 1 -- 172.21.15.1:0/1018339322 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115716240 0 0) 0x7f3ac80caac0 con 0x7f3ac806a750 2024-03-18T19:08:29.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.258+0000 7f3ac7fff700 1 --2- 172.21.15.1:0/1018339322 >> v2:172.21.15.165:3300/0 conn(0x7f3ac806f180 0x7f3ac80ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:08:29.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.259+0000 7f3acce49700 1 --2- 172.21.15.1:0/1018339322 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ac806eda0 0x7f3ac8161c60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3abc0098f0 tx=0x7f3abc009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19cf0ba8e8d135fa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:29.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.259+0000 7f3ac77fe700 1 -- 172.21.15.1:0/1018339322 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc011830 con 0x7f3ac806eda0 2024-03-18T19:08:29.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.259+0000 7f3ac77fe700 1 -- 172.21.15.1:0/1018339322 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3abc011990 con 0x7f3ac806eda0 2024-03-18T19:08:29.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.259+0000 7f3ac77fe700 1 -- 172.21.15.1:0/1018339322 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc011d20 con 0x7f3ac806eda0 2024-03-18T19:08:29.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.259+0000 7f3acf0ad700 1 -- 172.21.15.1:0/1018339322 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ac806eda0 msgr2=0x7f3ac8161c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:29.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.259+0000 7f3acf0ad700 1 --2- 172.21.15.1:0/1018339322 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ac806eda0 0x7f3ac8161c60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3abc0098f0 tx=0x7f3abc009c00 comp rx=0 tx=0).stop 2024-03-18T19:08:29.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.259+0000 7f3acf0ad700 1 -- 172.21.15.1:0/1018339322 shutdown_connections 2024-03-18T19:08:29.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.259+0000 7f3acf0ad700 1 --2- 172.21.15.1:0/1018339322 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ac806eda0 0x7f3ac8161c60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.259+0000 7f3acf0ad700 1 --2- 172.21.15.1:0/1018339322 >> v2:172.21.15.165:3300/0 conn(0x7f3ac806f180 0x7f3ac80ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.259+0000 7f3acf0ad700 1 -- 172.21.15.1:0/1018339322 >> 172.21.15.1:0/1018339322 conn(0x7f3ac80cff30 msgr2=0x7f3ac80d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:29.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.259+0000 7f3acf0ad700 1 -- 172.21.15.1:0/1018339322 shutdown_connections 2024-03-18T19:08:29.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.259+0000 7f3acf0ad700 1 -- 172.21.15.1:0/1018339322 wait complete. 2024-03-18T19:08:29.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.261+0000 7f3acf0ad700 1 Processor -- start 2024-03-18T19:08:29.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.261+0000 7f3acf0ad700 1 -- start start 2024-03-18T19:08:29.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.262+0000 7f3acf0ad700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ac806a750 0x7f3ac80b3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:29.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.262+0000 7f3acf0ad700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ac806eda0 0x7f3ac80ae210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:29.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.262+0000 7f3acf0ad700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ac806f180 0x7f3ac81f2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:29.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.262+0000 7f3acf0ad700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac815d280 con 0x7f3ac806a750 2024-03-18T19:08:29.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.262+0000 7f3acf0ad700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac81603b0 con 0x7f3ac806f180 2024-03-18T19:08:29.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.262+0000 7f3acf0ad700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac80b2b20 con 0x7f3ac806eda0 2024-03-18T19:08:29.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.262+0000 7f3acce49700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ac806a750 0x7f3ac80b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:29.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.262+0000 7f3ac7fff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ac806eda0 0x7f3ac80ae210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:29.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.262+0000 7f3acd64a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ac806f180 0x7f3ac81f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:29.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.262+0000 7f3ac7fff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ac806eda0 0x7f3ac80ae210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:34324/0 (socket says 172.21.15.1:34324) 2024-03-18T19:08:29.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.262+0000 7f3ac7fff700 1 -- 172.21.15.1:0/4182500289 learned_addr learned my addr 172.21.15.1:0/4182500289 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:29.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.263+0000 7f3acce49700 1 -- 172.21.15.1:0/4182500289 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ac806eda0 msgr2=0x7f3ac80ae210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:29.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.263+0000 7f3acce49700 1 --2- 172.21.15.1:0/4182500289 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ac806eda0 0x7f3ac80ae210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.263+0000 7f3acce49700 1 -- 172.21.15.1:0/4182500289 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ac806f180 msgr2=0x7f3ac81f2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.263+0000 7f3acce49700 1 --2- 172.21.15.1:0/4182500289 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ac806f180 0x7f3ac81f2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.263+0000 7f3acce49700 1 -- 172.21.15.1:0/4182500289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab40094a0 con 0x7f3ac806a750 2024-03-18T19:08:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.263+0000 7f3ac7fff700 1 --2- 172.21.15.1:0/4182500289 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ac806eda0 0x7f3ac80ae210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:08:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.263+0000 7f3acce49700 1 --2- 172.21.15.1:0/4182500289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ac806a750 0x7f3ac80b3530 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f3abc016000 tx=0x7f3abc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.263+0000 7f3ac57fa700 1 -- 172.21.15.1:0/4182500289 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc004300 con 0x7f3ac806a750 2024-03-18T19:08:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.263+0000 7f3ac57fa700 1 -- 172.21.15.1:0/4182500289 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3abc01f070 con 0x7f3ac806a750 2024-03-18T19:08:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.263+0000 7f3acf0ad700 1 -- 172.21.15.1:0/4182500289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3abc009550 con 0x7f3ac806a750 2024-03-18T19:08:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.263+0000 7f3ac57fa700 1 -- 172.21.15.1:0/4182500289 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc0036a0 con 0x7f3ac806a750 2024-03-18T19:08:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.263+0000 7f3acf0ad700 1 -- 172.21.15.1:0/4182500289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac80b2de0 con 0x7f3ac806a750 2024-03-18T19:08:29.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.264+0000 7f3acf0ad700 1 -- 172.21.15.1:0/4182500289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac80d0310 con 0x7f3ac806a750 2024-03-18T19:08:29.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.265+0000 7f3ac57fa700 1 -- 172.21.15.1:0/4182500289 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 61) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f3abc021030 con 0x7f3ac806a750 2024-03-18T19:08:29.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.265+0000 7f3ac57fa700 1 --2- 172.21.15.1:0/4182500289 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3aac06ae00 0x7f3aac06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:29.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.265+0000 7f3ac7fff700 1 -- 172.21.15.1:0/4182500289 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3aac06ae00 msgr2=0x7f3aac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:29.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.265+0000 7f3ac57fa700 1 -- 172.21.15.1:0/4182500289 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3abc08aaf0 con 0x7f3ac806a750 2024-03-18T19:08:29.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.265+0000 7f3ac7fff700 1 --2- 172.21.15.1:0/4182500289 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3aac06ae00 0x7f3aac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:08:29.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.274+0000 7f3ac57fa700 1 -- 172.21.15.1:0/4182500289 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3abc055bb0 con 0x7f3ac806a750 2024-03-18T19:08:29.391 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:29.394+0000 7fc0125d2240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:08:29.423 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:29.427+0000 7f433248b240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:08:29.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.466+0000 7f3ac7fff700 1 -- 172.21.15.1:0/4182500289 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3aac06ae00 msgr2=0x7f3aac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:29.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.466+0000 7f3ac7fff700 1 --2- 172.21.15.1:0/4182500289 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3aac06ae00 0x7f3aac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:08:29.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.525+0000 7f3acf0ad700 1 -- 172.21.15.1:0/4182500289 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f3ac80676a0 con 0x7f3ac806a750 2024-03-18T19:08:29.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.528+0000 7f3ac57fa700 1 -- 172.21.15.1:0/4182500289 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v41)=0 v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3abc015070 con 0x7f3ac806a750 2024-03-18T19:08:29.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.531+0000 7f3acf0ad700 1 -- 172.21.15.1:0/4182500289 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3aac06ae00 msgr2=0x7f3aac06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:08:29.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.531+0000 7f3acf0ad700 1 --2- 172.21.15.1:0/4182500289 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3aac06ae00 0x7f3aac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.531+0000 7f3acf0ad700 1 -- 172.21.15.1:0/4182500289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ac806a750 msgr2=0x7f3ac80b3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:29.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.531+0000 7f3acf0ad700 1 --2- 172.21.15.1:0/4182500289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ac806a750 0x7f3ac80b3530 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f3abc016000 tx=0x7f3abc004750 comp rx=0 tx=0).stop 2024-03-18T19:08:29.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.531+0000 7f3acf0ad700 1 -- 172.21.15.1:0/4182500289 shutdown_connections 2024-03-18T19:08:29.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.531+0000 7f3acf0ad700 1 --2- 172.21.15.1:0/4182500289 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3aac06ae00 0x7f3aac06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.531+0000 7f3acf0ad700 1 --2- 172.21.15.1:0/4182500289 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ac806f180 0x7f3ac81f2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.531+0000 7f3acf0ad700 1 --2- 172.21.15.1:0/4182500289 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ac806a750 0x7f3ac80b3530 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.531+0000 7f3acf0ad700 1 --2- 172.21.15.1:0/4182500289 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ac806eda0 0x7f3ac80ae210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.531+0000 7f3acf0ad700 1 -- 172.21.15.1:0/4182500289 >> 172.21.15.1:0/4182500289 conn(0x7f3ac80cff30 msgr2=0x7f3ac80b0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:29.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.531+0000 7f3acf0ad700 1 -- 172.21.15.1:0/4182500289 shutdown_connections 2024-03-18T19:08:29.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.531+0000 7f3acf0ad700 1 -- 172.21.15.1:0/4182500289 wait complete. 2024-03-18T19:08:29.539 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-18T19:08:29.655 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:29.659+0000 7fc0125d2240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:08:29.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.672+0000 7fe1cd78f700 1 Processor -- start 2024-03-18T19:08:29.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.673+0000 7fe1cd78f700 1 -- start start 2024-03-18T19:08:29.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.673+0000 7fe1cd78f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe1c806eda0 0x7fe1c8161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:29.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.673+0000 7fe1cd78f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c80cad60 con 0x7fe1c806a750 2024-03-18T19:08:29.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.673+0000 7fe1cd78f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c80cb040 con 0x7fe1c806f180 2024-03-18T19:08:29.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.673+0000 7fe1cd78f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c80cb320 con 0x7fe1c806eda0 2024-03-18T19:08:29.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.674+0000 7fe1c6ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe1c806eda0 0x7fe1c8161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:29.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.674+0000 7fe1c6ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe1c806eda0 0x7fe1c8161c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57236/0 (socket says 172.21.15.1:57236) 2024-03-18T19:08:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.674+0000 7fe1c77fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe1c806a750 0x7fe1c806ab50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60920/0 (socket says 172.21.15.1:60920) 2024-03-18T19:08:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.674+0000 7fe1c6ffd700 1 -- 172.21.15.1:0/4047030676 learned_addr learned my addr 172.21.15.1:0/4047030676 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.674+0000 7fe1c6ffd700 1 -- 172.21.15.1:0/4047030676 >> v1:172.21.15.165:6789/0 conn(0x7fe1c806f180 legacy=0x7fe1c80ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.674+0000 7fe1c6ffd700 1 -- 172.21.15.1:0/4047030676 >> v1:172.21.15.1:6789/0 conn(0x7fe1c806a750 legacy=0x7fe1c806ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.674+0000 7fe1c5ffb700 1 -- 172.21.15.1:0/4047030676 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009114633 0 0) 0x7fe1c80cad60 con 0x7fe1c806a750 2024-03-18T19:08:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.674+0000 7fe1c6ffd700 1 -- 172.21.15.1:0/4047030676 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c80ce220 con 0x7fe1c806eda0 2024-03-18T19:08:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.674+0000 7fe1c6ffd700 1 --2- 172.21.15.1:0/4047030676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1c806eda0 0x7fe1c8161c60 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fe1b80098f0 tx=0x7fe1b8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b17d053c47efaca server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.674+0000 7fe1c5ffb700 1 -- 172.21.15.1:0/4047030676 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b800b8a0 con 0x7fe1c806eda0 2024-03-18T19:08:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.674+0000 7fe1c5ffb700 1 -- 172.21.15.1:0/4047030676 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b800ba00 con 0x7fe1c806eda0 2024-03-18T19:08:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.674+0000 7fe1c5ffb700 1 -- 172.21.15.1:0/4047030676 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b800bd90 con 0x7fe1c806eda0 2024-03-18T19:08:29.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.675+0000 7fe1cd78f700 1 -- 172.21.15.1:0/4047030676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1c806eda0 msgr2=0x7fe1c8161c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:29.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.675+0000 7fe1cd78f700 1 --2- 172.21.15.1:0/4047030676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1c806eda0 0x7fe1c8161c60 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fe1b80098f0 tx=0x7fe1b8009c00 comp rx=0 tx=0).stop 2024-03-18T19:08:29.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.675+0000 7fe1cd78f700 1 -- 172.21.15.1:0/4047030676 shutdown_connections 2024-03-18T19:08:29.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.675+0000 7fe1cd78f700 1 --2- 172.21.15.1:0/4047030676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1c806eda0 0x7fe1c8161c60 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.675+0000 7fe1cd78f700 1 -- 172.21.15.1:0/4047030676 >> 172.21.15.1:0/4047030676 conn(0x7fe1c80cff30 msgr2=0x7fe1c80d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:29.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.675+0000 7fe1cd78f700 1 -- 172.21.15.1:0/4047030676 shutdown_connections 2024-03-18T19:08:29.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.675+0000 7fe1cd78f700 1 -- 172.21.15.1:0/4047030676 wait complete. 2024-03-18T19:08:29.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.676+0000 7fe1cd78f700 1 Processor -- start 2024-03-18T19:08:29.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.676+0000 7fe1cd78f700 1 -- start start 2024-03-18T19:08:29.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.677+0000 7fe1cd78f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1c806a750 0x7fe1c8159e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:29.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.677+0000 7fe1cd78f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1c806eda0 0x7fe1c814eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:29.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.677+0000 7fe1cd78f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1c806f180 0x7fe1c80ae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:29.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.677+0000 7fe1cd78f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c814f0e0 con 0x7fe1c806a750 2024-03-18T19:08:29.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.677+0000 7fe1cd78f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c814bff0 con 0x7fe1c806f180 2024-03-18T19:08:29.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.677+0000 7fe1cd78f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c814c890 con 0x7fe1c806eda0 2024-03-18T19:08:29.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.677+0000 7fe1c6ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1c806a750 0x7fe1c8159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:29.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.677+0000 7fe1c6ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1c806a750 0x7fe1c8159e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57244/0 (socket says 172.21.15.1:57244) 2024-03-18T19:08:29.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.677+0000 7fe1c6ffd700 1 -- 172.21.15.1:0/2393121603 learned_addr learned my addr 172.21.15.1:0/2393121603 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:29.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.677+0000 7fe1c67fc700 1 --2- 172.21.15.1:0/2393121603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1c806eda0 0x7fe1c814eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:29.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.677+0000 7fe1c77fe700 1 --2- 172.21.15.1:0/2393121603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1c806f180 0x7fe1c80ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:29.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.678+0000 7fe1c6ffd700 1 -- 172.21.15.1:0/2393121603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1c806eda0 msgr2=0x7fe1c814eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:29.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.678+0000 7fe1c6ffd700 1 --2- 172.21.15.1:0/2393121603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1c806eda0 0x7fe1c814eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.678+0000 7fe1c6ffd700 1 -- 172.21.15.1:0/2393121603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1c806f180 msgr2=0x7fe1c80ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:29.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.678+0000 7fe1c6ffd700 1 --2- 172.21.15.1:0/2393121603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1c806f180 0x7fe1c80ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.678+0000 7fe1c6ffd700 1 -- 172.21.15.1:0/2393121603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b8009550 con 0x7fe1c806a750 2024-03-18T19:08:29.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.678+0000 7fe1c6ffd700 1 --2- 172.21.15.1:0/2393121603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1c806a750 0x7fe1c8159e10 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fe1c809e080 tx=0x7fe1b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:29.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.678+0000 7fe1af7fe700 1 -- 172.21.15.1:0/2393121603 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b801a900 con 0x7fe1c806a750 2024-03-18T19:08:29.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.678+0000 7fe1af7fe700 1 -- 172.21.15.1:0/2393121603 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b801f070 con 0x7fe1c806a750 2024-03-18T19:08:29.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.678+0000 7fe1af7fe700 1 -- 172.21.15.1:0/2393121603 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b801aca0 con 0x7fe1c806a750 2024-03-18T19:08:29.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.678+0000 7fe1cd78f700 1 -- 172.21.15.1:0/2393121603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c815a410 con 0x7fe1c806a750 2024-03-18T19:08:29.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.678+0000 7fe1cd78f700 1 -- 172.21.15.1:0/2393121603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c814c520 con 0x7fe1c806a750 2024-03-18T19:08:29.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.679+0000 7fe1cd78f700 1 -- 172.21.15.1:0/2393121603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c80676a0 con 0x7fe1c806a750 2024-03-18T19:08:29.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.680+0000 7fe1af7fe700 1 -- 172.21.15.1:0/2393121603 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 61) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fe1b8026480 con 0x7fe1c806a750 2024-03-18T19:08:29.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.680+0000 7fe1af7fe700 1 --2- 172.21.15.1:0/2393121603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1b406ae00 0x7fe1b406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:29.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.680+0000 7fe1c67fc700 1 -- 172.21.15.1:0/2393121603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1b406ae00 msgr2=0x7fe1b406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:29.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.680+0000 7fe1af7fe700 1 -- 172.21.15.1:0/2393121603 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe1b8021070 con 0x7fe1c806a750 2024-03-18T19:08:29.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.680+0000 7fe1c67fc700 1 --2- 172.21.15.1:0/2393121603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1b406ae00 0x7fe1b406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:08:29.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.689+0000 7fe1af7fe700 1 -- 172.21.15.1:0/2393121603 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe1b8055f40 con 0x7fe1c806a750 2024-03-18T19:08:29.706 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:29.710+0000 7f433248b240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:08:29.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.881+0000 7fe1c67fc700 1 -- 172.21.15.1:0/2393121603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1b406ae00 msgr2=0x7fe1b406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:29.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.881+0000 7fe1c67fc700 1 --2- 172.21.15.1:0/2393121603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1b406ae00 0x7fe1b406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:08:29.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.893+0000 7fe1cd78f700 1 -- 172.21.15.1:0/2393121603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fe1c80d0bc0 con 0x7fe1c806a750 2024-03-18T19:08:29.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.897+0000 7fe1af7fe700 1 -- 172.21.15.1:0/2393121603 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v42)=0 v42) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe1b802b0c0 con 0x7fe1c806a750 2024-03-18T19:08:29.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.899+0000 7fe1cd78f700 1 -- 172.21.15.1:0/2393121603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1b406ae00 msgr2=0x7fe1b406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:08:29.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.899+0000 7fe1cd78f700 1 --2- 172.21.15.1:0/2393121603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1b406ae00 0x7fe1b406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.899+0000 7fe1cd78f700 1 -- 172.21.15.1:0/2393121603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1c806a750 msgr2=0x7fe1c8159e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:29.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.899+0000 7fe1cd78f700 1 --2- 172.21.15.1:0/2393121603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1c806a750 0x7fe1c8159e10 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fe1c809e080 tx=0x7fe1b8004750 comp rx=0 tx=0).stop 2024-03-18T19:08:29.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.899+0000 7fe1cd78f700 1 -- 172.21.15.1:0/2393121603 shutdown_connections 2024-03-18T19:08:29.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.899+0000 7fe1cd78f700 1 --2- 172.21.15.1:0/2393121603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe1b406ae00 0x7fe1b406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.899+0000 7fe1cd78f700 1 --2- 172.21.15.1:0/2393121603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1c806f180 0x7fe1c80ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.899+0000 7fe1cd78f700 1 --2- 172.21.15.1:0/2393121603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1c806a750 0x7fe1c8159e10 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.899+0000 7fe1cd78f700 1 --2- 172.21.15.1:0/2393121603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1c806eda0 0x7fe1c814eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:29.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.899+0000 7fe1cd78f700 1 -- 172.21.15.1:0/2393121603 >> 172.21.15.1:0/2393121603 conn(0x7fe1c80cff30 msgr2=0x7fe1c80b0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:29.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.899+0000 7fe1cd78f700 1 -- 172.21.15.1:0/2393121603 shutdown_connections 2024-03-18T19:08:29.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:29.899+0000 7fe1cd78f700 1 -- 172.21.15.1:0/2393121603 wait complete. 2024-03-18T19:08:29.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:29.908 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-03-18T19:08:29.930 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-03-18T19:08:30.081 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:30.085+0000 7fc0125d2240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:08:30.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.104+0000 7f6244e38700 1 Processor -- start 2024-03-18T19:08:30.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.104+0000 7f6244e38700 1 -- start start 2024-03-18T19:08:30.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.105+0000 7f6244e38700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6240158e30 0x7f6240071500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:30.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.105+0000 7f6244e38700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f624015c500 con 0x7f62400cbf70 2024-03-18T19:08:30.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.105+0000 7f6244e38700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62400af8b0 con 0x7f624006ae60 2024-03-18T19:08:30.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.105+0000 7f6244e38700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f624015b8f0 con 0x7f6240158e30 2024-03-18T19:08:30.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.105+0000 7f623dd9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f62400cbf70 0x7f624006a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60926/0 (socket says 172.21.15.1:60926) 2024-03-18T19:08:30.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.105+0000 7f623dd9b700 1 -- 172.21.15.1:0/1181961366 learned_addr learned my addr 172.21.15.1:0/1181961366 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:30.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.105+0000 7f623e59c700 1 --2- 172.21.15.1:0/1181961366 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6240158e30 0x7f6240071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:30.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.105+0000 7f623d59a700 1 -- 172.21.15.1:0/1181961366 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845647031 0 0) 0x7f624015c500 con 0x7f62400cbf70 2024-03-18T19:08:30.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.105+0000 7f623d59a700 1 -- 172.21.15.1:0/1181961366 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62280035b0 con 0x7f62400cbf70 2024-03-18T19:08:30.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.105+0000 7f623d59a700 1 -- 172.21.15.1:0/1181961366 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778880878 0 0) 0x7f62400af8b0 con 0x7f624006ae60 2024-03-18T19:08:30.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.105+0000 7f623d59a700 1 -- 172.21.15.1:0/1181961366 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f624015c500 con 0x7f624006ae60 2024-03-18T19:08:30.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.105+0000 7f623e59c700 1 -- 172.21.15.1:0/1181961366 >> v1:172.21.15.165:6789/0 conn(0x7f624006ae60 legacy=0x7f62400cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:30.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.105+0000 7f623e59c700 1 -- 172.21.15.1:0/1181961366 >> v1:172.21.15.1:6789/0 conn(0x7f62400cbf70 legacy=0x7f624006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:30.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.105+0000 7f623e59c700 1 -- 172.21.15.1:0/1181961366 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f624015bbd0 con 0x7f6240158e30 2024-03-18T19:08:30.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.106+0000 7f623e59c700 1 --2- 172.21.15.1:0/1181961366 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6240158e30 0x7f6240071500 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6230009870 tx=0x7f6230009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a3c4ef871da6715 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:30.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.106+0000 7f623d59a700 1 -- 172.21.15.1:0/1181961366 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6230011830 con 0x7f6240158e30 2024-03-18T19:08:30.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.106+0000 7f623d59a700 1 -- 172.21.15.1:0/1181961366 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6230011990 con 0x7f6240158e30 2024-03-18T19:08:30.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.106+0000 7f623d59a700 1 -- 172.21.15.1:0/1181961366 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6230011d20 con 0x7f6240158e30 2024-03-18T19:08:30.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.106+0000 7f6244e38700 1 -- 172.21.15.1:0/1181961366 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6240158e30 msgr2=0x7f6240071500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:30.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.106+0000 7f6244e38700 1 --2- 172.21.15.1:0/1181961366 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6240158e30 0x7f6240071500 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6230009870 tx=0x7f6230009b80 comp rx=0 tx=0).stop 2024-03-18T19:08:30.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.107+0000 7f6244e38700 1 -- 172.21.15.1:0/1181961366 shutdown_connections 2024-03-18T19:08:30.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.107+0000 7f6244e38700 1 --2- 172.21.15.1:0/1181961366 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6240158e30 0x7f6240071500 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:30.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.107+0000 7f6244e38700 1 -- 172.21.15.1:0/1181961366 >> 172.21.15.1:0/1181961366 conn(0x7f62400cca00 msgr2=0x7f62400cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:30.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.107+0000 7f6244e38700 1 -- 172.21.15.1:0/1181961366 shutdown_connections 2024-03-18T19:08:30.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.107+0000 7f6244e38700 1 -- 172.21.15.1:0/1181961366 wait complete. 2024-03-18T19:08:30.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.108+0000 7f6244e38700 1 Processor -- start 2024-03-18T19:08:30.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.108+0000 7f6244e38700 1 -- start start 2024-03-18T19:08:30.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.109+0000 7f6244e38700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f624006ae60 0x7f62400b2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:30.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.109+0000 7f6244e38700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f62400cbf70 0x7f624006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:30.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.109+0000 7f6244e38700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6240158e30 0x7f6240156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:30.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.109+0000 7f6244e38700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624006f450 con 0x7f6240158e30 2024-03-18T19:08:30.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.109+0000 7f6244e38700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624006f1b0 con 0x7f624006ae60 2024-03-18T19:08:30.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.109+0000 7f6244e38700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62400acdf0 con 0x7f62400cbf70 2024-03-18T19:08:30.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.109+0000 7f623ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6240158e30 0x7f6240156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:30.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.109+0000 7f623dd9b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f62400cbf70 0x7f624006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:30.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.109+0000 7f623ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6240158e30 0x7f6240156330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57246/0 (socket says 172.21.15.1:57246) 2024-03-18T19:08:30.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.109+0000 7f623ed9d700 1 -- 172.21.15.1:0/429576366 learned_addr learned my addr 172.21.15.1:0/429576366 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:30.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.109+0000 7f623e59c700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f624006ae60 0x7f62400b2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:30.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.109+0000 7f623ed9d700 1 -- 172.21.15.1:0/429576366 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f62400cbf70 msgr2=0x7f624006ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:30.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.109+0000 7f623ed9d700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f62400cbf70 0x7f624006ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:30.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.109+0000 7f623ed9d700 1 -- 172.21.15.1:0/429576366 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f624006ae60 msgr2=0x7f62400b2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:30.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.109+0000 7f623ed9d700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f624006ae60 0x7f62400b2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:30.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.109+0000 7f623ed9d700 1 -- 172.21.15.1:0/429576366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62300094a0 con 0x7f6240158e30 2024-03-18T19:08:30.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.110+0000 7f623ed9d700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6240158e30 0x7f6240156330 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f623800cc20 tx=0x7f623800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:30.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.110+0000 7f622effd700 1 -- 172.21.15.1:0/429576366 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6238003c00 con 0x7f6240158e30 2024-03-18T19:08:30.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.110+0000 7f6244e38700 1 -- 172.21.15.1:0/429576366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f624006df30 con 0x7f6240158e30 2024-03-18T19:08:30.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.110+0000 7f6244e38700 1 -- 172.21.15.1:0/429576366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f624006e4f0 con 0x7f6240158e30 2024-03-18T19:08:30.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.110+0000 7f622effd700 1 -- 172.21.15.1:0/429576366 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6238001d00 con 0x7f6240158e30 2024-03-18T19:08:30.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.110+0000 7f622effd700 1 -- 172.21.15.1:0/429576366 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623800a8a0 con 0x7f6240158e30 2024-03-18T19:08:30.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.111+0000 7f6244e38700 1 -- 172.21.15.1:0/429576366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62240052f0 con 0x7f6240158e30 2024-03-18T19:08:30.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.111+0000 7f622effd700 1 -- 172.21.15.1:0/429576366 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 61) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f623800aa00 con 0x7f6240158e30 2024-03-18T19:08:30.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.112+0000 7f622effd700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622006ae30 0x7f622006d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:30.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.112+0000 7f623e59c700 1 -- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622006ae30 msgr2=0x7f622006d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:30.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.112+0000 7f622effd700 1 -- 172.21.15.1:0/429576366 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f62380896d0 con 0x7f6240158e30 2024-03-18T19:08:30.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.112+0000 7f623e59c700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622006ae30 0x7f622006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:08:30.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.121+0000 7f622effd700 1 -- 172.21.15.1:0/429576366 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f62380545e0 con 0x7f6240158e30 2024-03-18T19:08:30.168 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:30.172+0000 7f433248b240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:08:30.290 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:30.294+0000 7fc0125d2240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:08:30.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.313+0000 7f623e59c700 1 -- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622006ae30 msgr2=0x7f622006d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:30.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.313+0000 7f623e59c700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622006ae30 0x7f622006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:08:30.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.365+0000 7f6244e38700 1 -- 172.21.15.1:0/429576366 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6224000bc0 con 0x7f622006ae30 2024-03-18T19:08:30.398 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:30.402+0000 7f433248b240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:08:30.661 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:30.665+0000 7fc0125d2240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:08:30.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.714+0000 7f623e59c700 1 -- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622006ae30 msgr2=0x7f622006d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:30.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:30.714+0000 7f623e59c700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622006ae30 0x7f622006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T19:08:30.784 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:30.789+0000 7f433248b240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:08:31.250 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:31.254+0000 7fc0125d2240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:08:31.322 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:31.326+0000 7fc0125d2240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:08:31.398 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:31.402+0000 7f433248b240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:08:31.449 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:31.452+0000 7fc0125d2240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:08:31.473 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:31.478+0000 7f433248b240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:08:31.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:31.515+0000 7f623e59c700 1 -- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622006ae30 msgr2=0x7f622006d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:31.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:31.515+0000 7f623e59c700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622006ae30 0x7f622006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T19:08:31.608 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:31.612+0000 7f433248b240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:08:31.626 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:31.629+0000 7fc0125d2240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:08:31.697 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:31.701+0000 7fc0125d2240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:08:31.775 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:31.779+0000 7fc0125d2240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:08:31.794 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:31.798+0000 7f433248b240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:08:31.852 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:31.856+0000 7fc0125d2240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:08:31.870 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:31.874+0000 7f433248b240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:08:31.931 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:31.935+0000 7fc0125d2240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:08:31.952 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:31.956+0000 7f433248b240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:08:32.034 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:32.038+0000 7f433248b240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:08:32.102 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:32.105+0000 7fc0125d2240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:08:32.116 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:32.120+0000 7f433248b240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:08:32.294 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:32.298+0000 7f433248b240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:08:32.317 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:32.321+0000 7fc0125d2240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:08:32.467 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:32.471+0000 7fc0125d2240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:08:32.516 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:32.520+0000 7f433248b240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:08:32.541 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:32.544+0000 7fc0125d2240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:08:32.675 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:32.679+0000 7f433248b240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:08:32.751 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:32.756+0000 7f433248b240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:08:32.984 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:32.988+0000 7fc0125d2240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:08:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:33.116+0000 7f623e59c700 1 -- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622006ae30 msgr2=0x7f622006d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:08:33.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:33.116+0000 7f623e59c700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622006ae30 0x7f622006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T19:08:33.222 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:33.226+0000 7f433248b240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:08:33.323 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:33.327+0000 7fc0125d2240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:08:33.487 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:33.490+0000 7fc0125d2240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:08:33.572 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:33.576+0000 7f433248b240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:08:33.742 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:33.746+0000 7f433248b240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:08:33.872 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:33.876+0000 7fc0125d2240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:08:34.036 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:34.040+0000 7fc0125d2240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:08:34.151 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:34.155+0000 7f433248b240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:08:34.324 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:34.328+0000 7f433248b240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:08:34.398 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:34.402+0000 7fc0125d2240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:08:34.486 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:08:34.489+0000 7fc0125d2240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:08:34.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:34.559+0000 7f622effd700 1 -- 172.21.15.1:0/429576366 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 62) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f6238087ab0 con 0x7f6240158e30 2024-03-18T19:08:34.703 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:34.707+0000 7f433248b240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:08:34.797 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:34.801+0000 7f433248b240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:08:34.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:34.820+0000 7f622effd700 1 -- 172.21.15.1:0/429576366 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 63) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f6238058500 con 0x7f6240158e30 2024-03-18T19:08:34.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:34.821+0000 7f622effd700 1 -- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622006ae30 msgr2=0x7f622006d2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:08:34.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:34.821+0000 7f622effd700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622006ae30 0x7f622006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:34.974 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:34.978+0000 7f42a80ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:34.975 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:34.978+0000 7f42a80ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:34.975 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:34.978+0000 7f42a80ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:34.975 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:34.978+0000 7f42a80ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:34.975 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:34.978+0000 7f42a80ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:34.975 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:34.978+0000 7f42a50a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:34.975 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:34.978+0000 7f42a50a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:34.975 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:34.978+0000 7f42a50a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:34.975 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:34.978+0000 7f42a50a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:34.975 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:08:34.978+0000 7f42a50a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:08:35.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:35.827+0000 7f622effd700 1 -- 172.21.15.1:0/429576366 <== mon.0 v2:172.21.15.1:3300/0 9 ==== mgrmap(e 64) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f6238055280 con 0x7f6240158e30 2024-03-18T19:08:35.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:35.828+0000 7f622effd700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622006ae30 0x7f622006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:35.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:35.828+0000 7f622effd700 1 -- 172.21.15.1:0/429576366 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6224000bc0 con 0x7f622006ae30 2024-03-18T19:08:35.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:35.829+0000 7f623e59c700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622006ae30 0x7f622006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:35.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:35.832+0000 7f623e59c700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622006ae30 0x7f622006d270 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f62300097c0 tx=0x7f623001c040 comp rx=0 tx=0).ready entity=mgr.8469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:35.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:35.889+0000 7f622effd700 1 -- 172.21.15.1:0/429576366 <== mgr.8469 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6224000bc0 con 0x7f622006ae30 2024-03-18T19:08:35.885 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path snap_test_dir1 2024-03-18T19:08:35.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:35.892+0000 7f6244e38700 1 -- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622006ae30 msgr2=0x7f622006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:35.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:35.892+0000 7f6244e38700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622006ae30 0x7f622006d270 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f62300097c0 tx=0x7f623001c040 comp rx=0 tx=0).stop 2024-03-18T19:08:35.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:35.892+0000 7f6244e38700 1 -- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6240158e30 msgr2=0x7f6240156330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:35.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:35.892+0000 7f6244e38700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6240158e30 0x7f6240156330 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f623800cc20 tx=0x7f623800cf30 comp rx=0 tx=0).stop 2024-03-18T19:08:35.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:35.892+0000 7f6244e38700 1 -- 172.21.15.1:0/429576366 shutdown_connections 2024-03-18T19:08:35.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:35.892+0000 7f6244e38700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f622006ae30 0x7f622006d270 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:35.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:35.892+0000 7f6244e38700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6240158e30 0x7f6240156330 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:35.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:35.892+0000 7f6244e38700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f624006ae60 0x7f62400b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:35.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:35.892+0000 7f6244e38700 1 --2- 172.21.15.1:0/429576366 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f62400cbf70 0x7f624006ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:35.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:35.892+0000 7f6244e38700 1 -- 172.21.15.1:0/429576366 >> 172.21.15.1:0/429576366 conn(0x7f62400cca00 msgr2=0x7f624006c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:35.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:35.892+0000 7f6244e38700 1 -- 172.21.15.1:0/429576366 shutdown_connections 2024-03-18T19:08:35.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:35.892+0000 7f6244e38700 1 -- 172.21.15.1:0/429576366 wait complete. 2024-03-18T19:08:35.901 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-03-18T19:08:35.902 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-03-18-19_09_00 in ~25s... 2024-03-18T19:08:35.902 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2024-03-18T19:08:35.902 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-03-18T19:08:36.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.036+0000 7fb32114a700 1 Processor -- start 2024-03-18T19:08:36.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.036+0000 7fb32114a700 1 -- start start 2024-03-18T19:08:36.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.036+0000 7fb32114a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb31c0cfa90 0x7fb31c06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:36.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.036+0000 7fb32114a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31c0cf600 con 0x7fb31c15d5a0 2024-03-18T19:08:36.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.036+0000 7fb32114a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31c0ce720 con 0x7fb31c06ac90 2024-03-18T19:08:36.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.036+0000 7fb32114a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb31c06b140 con 0x7fb31c0cfa90 2024-03-18T19:08:36.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.037+0000 7fb31ad9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb31c15d5a0 0x7fb31c0ce010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48158/0 (socket says 172.21.15.1:48158) 2024-03-18T19:08:36.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.037+0000 7fb31ad9d700 1 -- 172.21.15.1:0/1043349037 learned_addr learned my addr 172.21.15.1:0/1043349037 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:08:36.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.037+0000 7fb31a59c700 1 --2- 172.21.15.1:0/1043349037 >> v2:172.21.15.165:3300/0 conn(0x7fb31c0cfa90 0x7fb31c06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:36.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.037+0000 7fb319d9b700 1 -- 172.21.15.1:0/1043349037 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058041013 0 0) 0x7fb31c0cf600 con 0x7fb31c15d5a0 2024-03-18T19:08:36.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.037+0000 7fb319d9b700 1 -- 172.21.15.1:0/1043349037 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3040035b0 con 0x7fb31c15d5a0 2024-03-18T19:08:36.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.037+0000 7fb31a59c700 1 -- 172.21.15.1:0/1043349037 >> v1:172.21.15.165:6789/0 conn(0x7fb31c06ac90 legacy=0x7fb31c167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:36.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.037+0000 7fb31a59c700 1 -- 172.21.15.1:0/1043349037 >> v1:172.21.15.1:6789/0 conn(0x7fb31c15d5a0 legacy=0x7fb31c0ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:36.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.037+0000 7fb31a59c700 1 -- 172.21.15.1:0/1043349037 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c0ce9f0 con 0x7fb31c0cfa90 2024-03-18T19:08:36.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.037+0000 7fb31a59c700 1 --2- 172.21.15.1:0/1043349037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb31c0cfa90 0x7fb31c06a750 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb310009840 tx=0x7fb310009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92efe8eebd8e045a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:36.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.037+0000 7fb319d9b700 1 -- 172.21.15.1:0/1043349037 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb310011830 con 0x7fb31c0cfa90 2024-03-18T19:08:36.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.037+0000 7fb319d9b700 1 -- 172.21.15.1:0/1043349037 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb310011990 con 0x7fb31c0cfa90 2024-03-18T19:08:36.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.037+0000 7fb319d9b700 1 -- 172.21.15.1:0/1043349037 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb310011d20 con 0x7fb31c0cfa90 2024-03-18T19:08:36.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.038+0000 7fb32114a700 1 -- 172.21.15.1:0/1043349037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb31c0cfa90 msgr2=0x7fb31c06a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:36.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.038+0000 7fb32114a700 1 --2- 172.21.15.1:0/1043349037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb31c0cfa90 0x7fb31c06a750 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb310009840 tx=0x7fb310009b50 comp rx=0 tx=0).stop 2024-03-18T19:08:36.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.038+0000 7fb32114a700 1 -- 172.21.15.1:0/1043349037 shutdown_connections 2024-03-18T19:08:36.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.038+0000 7fb32114a700 1 --2- 172.21.15.1:0/1043349037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb31c0cfa90 0x7fb31c06a750 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:36.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.038+0000 7fb32114a700 1 -- 172.21.15.1:0/1043349037 >> 172.21.15.1:0/1043349037 conn(0x7fb31c0cc9b0 msgr2=0x7fb31c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:36.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.038+0000 7fb32114a700 1 -- 172.21.15.1:0/1043349037 shutdown_connections 2024-03-18T19:08:36.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.038+0000 7fb32114a700 1 -- 172.21.15.1:0/1043349037 wait complete. 2024-03-18T19:08:36.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.039+0000 7fb32114a700 1 Processor -- start 2024-03-18T19:08:36.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.040+0000 7fb32114a700 1 -- start start 2024-03-18T19:08:36.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.040+0000 7fb32114a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb31c06ac90 0x7fb31c153f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:36.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.040+0000 7fb32114a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb31c0cfa90 0x7fb31c147060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:36.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.040+0000 7fb32114a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb31c15d5a0 0x7fb31c1475a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:36.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.040+0000 7fb32114a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c1544c0 con 0x7fb31c15d5a0 2024-03-18T19:08:36.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.040+0000 7fb32114a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c0b0d80 con 0x7fb31c06ac90 2024-03-18T19:08:36.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.040+0000 7fb32114a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c1548e0 con 0x7fb31c0cfa90 2024-03-18T19:08:36.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.040+0000 7fb31b59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb31c15d5a0 0x7fb31c1475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:36.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.040+0000 7fb31b59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb31c15d5a0 0x7fb31c1475a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46986/0 (socket says 172.21.15.1:46986) 2024-03-18T19:08:36.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.040+0000 7fb31b59e700 1 -- 172.21.15.1:0/1044183931 learned_addr learned my addr 172.21.15.1:0/1044183931 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:08:36.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.041+0000 7fb31ad9d700 1 --2- 172.21.15.1:0/1044183931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb31c06ac90 0x7fb31c153f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:36.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.041+0000 7fb31a59c700 1 --2- 172.21.15.1:0/1044183931 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb31c0cfa90 0x7fb31c147060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:36.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.041+0000 7fb31b59e700 1 -- 172.21.15.1:0/1044183931 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb31c0cfa90 msgr2=0x7fb31c147060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:36.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.041+0000 7fb31b59e700 1 --2- 172.21.15.1:0/1044183931 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb31c0cfa90 0x7fb31c147060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:36.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.041+0000 7fb31b59e700 1 -- 172.21.15.1:0/1044183931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb31c06ac90 msgr2=0x7fb31c153f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:36.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.041+0000 7fb31b59e700 1 --2- 172.21.15.1:0/1044183931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb31c06ac90 0x7fb31c153f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:36.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.041+0000 7fb31b59e700 1 -- 172.21.15.1:0/1044183931 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3100094a0 con 0x7fb31c15d5a0 2024-03-18T19:08:36.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.041+0000 7fb31b59e700 1 --2- 172.21.15.1:0/1044183931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb31c15d5a0 0x7fb31c1475a0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fb31400cc80 tx=0x7fb31400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:36.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.041+0000 7fb3037fe700 1 -- 172.21.15.1:0/1044183931 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31400ad40 con 0x7fb31c15d5a0 2024-03-18T19:08:36.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.041+0000 7fb3037fe700 1 -- 172.21.15.1:0/1044183931 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31400ce80 con 0x7fb31c15d5a0 2024-03-18T19:08:36.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.041+0000 7fb32114a700 1 -- 172.21.15.1:0/1044183931 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31c0b0170 con 0x7fb31c15d5a0 2024-03-18T19:08:36.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.041+0000 7fb3037fe700 1 -- 172.21.15.1:0/1044183931 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31400aea0 con 0x7fb31c15d5a0 2024-03-18T19:08:36.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.041+0000 7fb32114a700 1 -- 172.21.15.1:0/1044183931 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb31c0b0730 con 0x7fb31c15d5a0 2024-03-18T19:08:36.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.042+0000 7fb32114a700 1 -- 172.21.15.1:0/1044183931 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31c0676a0 con 0x7fb31c15d5a0 2024-03-18T19:08:36.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.043+0000 7fb3037fe700 1 -- 172.21.15.1:0/1044183931 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 64) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fb314003c40 con 0x7fb31c15d5a0 2024-03-18T19:08:36.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.043+0000 7fb3037fe700 1 --2- 172.21.15.1:0/1044183931 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb30806ab30 0x7fb30806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:08:36.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.043+0000 7fb31ad9d700 1 --2- 172.21.15.1:0/1044183931 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb30806ab30 0x7fb30806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:08:36.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.043+0000 7fb3037fe700 1 -- 172.21.15.1:0/1044183931 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fb314089430 con 0x7fb31c15d5a0 2024-03-18T19:08:36.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.044+0000 7fb31ad9d700 1 --2- 172.21.15.1:0/1044183931 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb30806ab30 0x7fb30806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0010e0 tx=0x7fb30c009040 comp rx=0 tx=0).ready entity=mgr.8469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:08:36.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.048+0000 7fb3037fe700 1 -- 172.21.15.1:0/1044183931 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb3140545e0 con 0x7fb31c15d5a0 2024-03-18T19:08:36.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.277+0000 7fb32114a700 1 -- 172.21.15.1:0/1044183931 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb31c0cf290 con 0x7fb30806ab30 2024-03-18T19:08:36.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.286+0000 7fb3037fe700 1 -- 172.21.15.1:0/1044183931 <== mgr.8469 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fb30c001450 con 0x7fb30806ab30 2024-03-18T19:08:36.282 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:08:36.282 INFO:teuthology.orchestra.run.smithi001.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2024-03-18T19:08:36.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.288+0000 7fb32114a700 1 -- 172.21.15.1:0/1044183931 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb30806ab30 msgr2=0x7fb30806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:36.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.288+0000 7fb32114a700 1 --2- 172.21.15.1:0/1044183931 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb30806ab30 0x7fb30806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0010e0 tx=0x7fb30c009040 comp rx=0 tx=0).stop 2024-03-18T19:08:36.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.288+0000 7fb32114a700 1 -- 172.21.15.1:0/1044183931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb31c15d5a0 msgr2=0x7fb31c1475a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:08:36.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.288+0000 7fb32114a700 1 --2- 172.21.15.1:0/1044183931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb31c15d5a0 0x7fb31c1475a0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fb31400cc80 tx=0x7fb31400a340 comp rx=0 tx=0).stop 2024-03-18T19:08:36.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.289+0000 7fb32114a700 1 -- 172.21.15.1:0/1044183931 shutdown_connections 2024-03-18T19:08:36.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.289+0000 7fb32114a700 1 --2- 172.21.15.1:0/1044183931 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb30806ab30 0x7fb30806cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:36.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.289+0000 7fb32114a700 1 --2- 172.21.15.1:0/1044183931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb31c15d5a0 0x7fb31c1475a0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:36.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.289+0000 7fb32114a700 1 --2- 172.21.15.1:0/1044183931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb31c06ac90 0x7fb31c153f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:36.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.289+0000 7fb32114a700 1 --2- 172.21.15.1:0/1044183931 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb31c0cfa90 0x7fb31c147060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:08:36.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.289+0000 7fb32114a700 1 -- 172.21.15.1:0/1044183931 >> 172.21.15.1:0/1044183931 conn(0x7fb31c0cc9b0 msgr2=0x7fb31c15c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:08:36.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.289+0000 7fb32114a700 1 -- 172.21.15.1:0/1044183931 shutdown_connections 2024-03-18T19:08:36.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:08:36.289+0000 7fb32114a700 1 -- 172.21.15.1:0/1044183931 wait complete. 2024-03-18T19:08:36.297 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2024-03-18T19:08:36.297 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{}]} 2024-03-18T19:08:36.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:36.297 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:36.321 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:36.321 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:36.321 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:37.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:37.322 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:37.351 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:37.351 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:37.351 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:38.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:38.352 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:38.380 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:38.380 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:38.380 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:39.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:39.382 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:39.410 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:39.410 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:39.410 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:40.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:40.411 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:40.439 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:40.440 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:40.440 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:41.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:41.441 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:41.469 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:41.469 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:41.469 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:42.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:42.471 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:42.500 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:42.500 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:42.500 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:43.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:43.501 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:43.530 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:43.530 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:43.530 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:44.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:44.531 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:44.559 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:44.559 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:44.559 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:45.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:45.559 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:45.588 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:45.588 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:45.588 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:46.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:46.589 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:46.617 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:46.618 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:46.618 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:47.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:47.619 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:47.647 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:47.647 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:47.647 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:48.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:48.648 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:48.676 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:48.677 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:48.677 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:49.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:49.678 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:49.706 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:49.706 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:49.706 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:50.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:50.707 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:50.736 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:50.736 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:50.736 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:51.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:51.737 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:51.766 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:51.766 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:51.766 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:51.876 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:08:51.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:08:52.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:52.767 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:52.795 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:52.795 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:52.795 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:53.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:53.796 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:53.824 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:53.824 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:53.824 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:54.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:54.825 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:54.853 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:54.853 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:54.854 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:55.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:55.855 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:55.884 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:55.884 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:55.884 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:56.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:56.885 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:56.914 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:56.914 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:56.914 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:57.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:57.915 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:57.944 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:57.944 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:57.944 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:58.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:58.945 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:08:58.973 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:08:58.973 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:08:58.974 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:08:59.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:08:59.974 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:09:00.004 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T19:09:00.005 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T19:09:00.005 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:09:00.896 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:00.899+0000 7f429788d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:00.896 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:00.899+0000 7f429788d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:00.896 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:00.899+0000 7f429788d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:00.896 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:00.899+0000 7f429788d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:00.896 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:00.899+0000 7f429788d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:01.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:01.005 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:09:01.031 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_09_00_UTC 2024-03-18T19:09:01.031 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-19_09_00_UTC'} 2024-03-18T19:09:01.031 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-03-18-19_09_00_UTC'} 2024-03-18T19:09:01.031 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T19:09:01.031 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-03-18-19_09_00_UTC'] 2024-03-18T19:09:01.031 DEBUG:tasks.cephfs.test_snap_schedules:exec=1710788915.9020681, now = 1710788941.0317426, timo = 25 2024-03-18T19:09:02.033 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-03-18T19:09:02.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.180+0000 7f318076c700 1 Processor -- start 2024-03-18T19:09:02.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.180+0000 7f318076c700 1 -- start start 2024-03-18T19:09:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.181+0000 7f318076c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f317815d780 0x7f31780cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.181+0000 7f318076c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f317809dd40 0x7f31780cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.181+0000 7f318076c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31780ce090 con 0x7f31780ce5a0 2024-03-18T19:09:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.181+0000 7f318076c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f317806db30 con 0x7f317815d780 2024-03-18T19:09:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.181+0000 7f318076c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f317806d520 con 0x7f317809dd40 2024-03-18T19:09:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.181+0000 7f317e508700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f317815d780 0x7f31780cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.182+0000 7f317e508700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f317815d780 0x7f31780cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57194/0 (socket says 172.21.15.1:57194) 2024-03-18T19:09:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.182+0000 7f317e508700 1 -- 172.21.15.1:0/298872588 learned_addr learned my addr 172.21.15.1:0/298872588 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.182+0000 7f317dd07700 1 --2- 172.21.15.1:0/298872588 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f317809dd40 0x7f31780cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.182+0000 7f317e508700 1 -- 172.21.15.1:0/298872588 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f317809dd40 msgr2=0x7f31780cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.182+0000 7f317e508700 1 --2- 172.21.15.1:0/298872588 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f317809dd40 0x7f31780cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.182+0000 7f317e508700 1 -- 172.21.15.1:0/298872588 >> v1:172.21.15.165:6789/0 conn(0x7f31780ce5a0 legacy=0x7f31780cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.182+0000 7f317e508700 1 -- 172.21.15.1:0/298872588 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317806f460 con 0x7f317815d780 2024-03-18T19:09:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.182+0000 7f317dd07700 1 --2- 172.21.15.1:0/298872588 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f317809dd40 0x7f31780cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.182+0000 7f317e508700 1 --2- 172.21.15.1:0/298872588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317815d780 0x7f31780cdb50 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f316800bb90 tx=0x7f316800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52027b33f9c493c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.182+0000 7f317d506700 1 -- 172.21.15.1:0/298872588 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316800dd10 con 0x7f317815d780 2024-03-18T19:09:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.182+0000 7f317d506700 1 -- 172.21.15.1:0/298872588 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f316800de70 con 0x7f317815d780 2024-03-18T19:09:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.182+0000 7f317d506700 1 -- 172.21.15.1:0/298872588 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3168004230 con 0x7f317815d780 2024-03-18T19:09:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.183+0000 7f318076c700 1 -- 172.21.15.1:0/298872588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317815d780 msgr2=0x7f31780cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.183+0000 7f318076c700 1 --2- 172.21.15.1:0/298872588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317815d780 0x7f31780cdb50 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f316800bb90 tx=0x7f316800bea0 comp rx=0 tx=0).stop 2024-03-18T19:09:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.183+0000 7f318076c700 1 -- 172.21.15.1:0/298872588 shutdown_connections 2024-03-18T19:09:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.183+0000 7f318076c700 1 --2- 172.21.15.1:0/298872588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317815d780 0x7f31780cdb50 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.183+0000 7f318076c700 1 --2- 172.21.15.1:0/298872588 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f317809dd40 0x7f31780cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.183+0000 7f318076c700 1 -- 172.21.15.1:0/298872588 >> 172.21.15.1:0/298872588 conn(0x7f31780cca00 msgr2=0x7f31780cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.183+0000 7f318076c700 1 -- 172.21.15.1:0/298872588 shutdown_connections 2024-03-18T19:09:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.183+0000 7f318076c700 1 -- 172.21.15.1:0/298872588 wait complete. 2024-03-18T19:09:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.184+0000 7f318076c700 1 Processor -- start 2024-03-18T19:09:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.184+0000 7f318076c700 1 -- start start 2024-03-18T19:09:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.185+0000 7f318076c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f317809dd40 0x7f31781525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.185+0000 7f318076c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31780ce5a0 0x7f31780c9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.185+0000 7f318076c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317815d780 0x7f3178157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.185+0000 7f318076c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31780c7050 con 0x7f317815d780 2024-03-18T19:09:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.185+0000 7f318076c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3178154e20 con 0x7f31780ce5a0 2024-03-18T19:09:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.185+0000 7f318076c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3178155100 con 0x7f317809dd40 2024-03-18T19:09:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.185+0000 7f317ed09700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317815d780 0x7f3178157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.185+0000 7f317ed09700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317815d780 0x7f3178157900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57206/0 (socket says 172.21.15.1:57206) 2024-03-18T19:09:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.185+0000 7f317ed09700 1 -- 172.21.15.1:0/800961611 learned_addr learned my addr 172.21.15.1:0/800961611 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.185+0000 7f317e508700 1 --2- 172.21.15.1:0/800961611 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f317809dd40 0x7f31781525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.185+0000 7f317dd07700 1 --2- 172.21.15.1:0/800961611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31780ce5a0 0x7f31780c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.185+0000 7f317ed09700 1 -- 172.21.15.1:0/800961611 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f317809dd40 msgr2=0x7f31781525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.185+0000 7f317ed09700 1 --2- 172.21.15.1:0/800961611 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f317809dd40 0x7f31781525d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.185+0000 7f317ed09700 1 -- 172.21.15.1:0/800961611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31780ce5a0 msgr2=0x7f31780c9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.185+0000 7f317ed09700 1 --2- 172.21.15.1:0/800961611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31780ce5a0 0x7f31780c9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.185+0000 7f317ed09700 1 -- 172.21.15.1:0/800961611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316800b7f0 con 0x7f317815d780 2024-03-18T19:09:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.185+0000 7f317ed09700 1 --2- 172.21.15.1:0/800961611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317815d780 0x7f3178157900 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f317000cc10 tx=0x7f317000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.186+0000 7f316effd700 1 -- 172.21.15.1:0/800961611 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317000aa60 con 0x7f317815d780 2024-03-18T19:09:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.186+0000 7f316effd700 1 -- 172.21.15.1:0/800961611 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317000abc0 con 0x7f317815d780 2024-03-18T19:09:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.186+0000 7f318076c700 1 -- 172.21.15.1:0/800961611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31781553e0 con 0x7f317815d780 2024-03-18T19:09:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.186+0000 7f316effd700 1 -- 172.21.15.1:0/800961611 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3170012420 con 0x7f317815d780 2024-03-18T19:09:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.186+0000 7f318076c700 1 -- 172.21.15.1:0/800961611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3178152b10 con 0x7f317815d780 2024-03-18T19:09:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.187+0000 7f318076c700 1 -- 172.21.15.1:0/800961611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31780c74e0 con 0x7f317815d780 2024-03-18T19:09:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.187+0000 7f316effd700 1 -- 172.21.15.1:0/800961611 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 67) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f3170012580 con 0x7f317815d780 2024-03-18T19:09:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.188+0000 7f316effd700 1 --2- 172.21.15.1:0/800961611 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f315c06af50 0x7f315c06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.188+0000 7f316effd700 1 -- 172.21.15.1:0/800961611 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f3170089c00 con 0x7f317815d780 2024-03-18T19:09:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.188+0000 7f317e508700 1 --2- 172.21.15.1:0/800961611 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f315c06af50 0x7f315c06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.188+0000 7f317e508700 1 --2- 172.21.15.1:0/800961611 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f315c06af50 0x7f315c06d410 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f316800b340 tx=0x7f316801d040 comp rx=0 tx=0).ready entity=mgr.8469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:02.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.192+0000 7f316effd700 1 -- 172.21.15.1:0/800961611 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f31780c74e0 con 0x7f317815d780 2024-03-18T19:09:02.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.422+0000 7f318076c700 1 -- 172.21.15.1:0/800961611 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f317806d7c0 con 0x7f315c06af50 2024-03-18T19:09:02.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.484+0000 7f316effd700 1 -- 172.21.15.1:0/800961611 <== mgr.8469 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f3168021040 con 0x7f315c06af50 2024-03-18T19:09:02.481 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule removed for path snap_test_dir1 2024-03-18T19:09:02.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.487+0000 7f318076c700 1 -- 172.21.15.1:0/800961611 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f315c06af50 msgr2=0x7f315c06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:02.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.487+0000 7f318076c700 1 --2- 172.21.15.1:0/800961611 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f315c06af50 0x7f315c06d410 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f316800b340 tx=0x7f316801d040 comp rx=0 tx=0).stop 2024-03-18T19:09:02.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.487+0000 7f318076c700 1 -- 172.21.15.1:0/800961611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317815d780 msgr2=0x7f3178157900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:02.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.487+0000 7f318076c700 1 --2- 172.21.15.1:0/800961611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317815d780 0x7f3178157900 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f317000cc10 tx=0x7f317000cf20 comp rx=0 tx=0).stop 2024-03-18T19:09:02.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.487+0000 7f318076c700 1 -- 172.21.15.1:0/800961611 shutdown_connections 2024-03-18T19:09:02.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.487+0000 7f318076c700 1 --2- 172.21.15.1:0/800961611 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f315c06af50 0x7f315c06d410 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:02.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.487+0000 7f318076c700 1 --2- 172.21.15.1:0/800961611 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f317809dd40 0x7f31781525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:02.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.487+0000 7f318076c700 1 --2- 172.21.15.1:0/800961611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317815d780 0x7f3178157900 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:02.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.487+0000 7f318076c700 1 --2- 172.21.15.1:0/800961611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31780ce5a0 0x7f31780c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:02.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.487+0000 7f318076c700 1 -- 172.21.15.1:0/800961611 >> 172.21.15.1:0/800961611 conn(0x7f31780cca00 msgr2=0x7f31780cea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:02.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.487+0000 7f318076c700 1 -- 172.21.15.1:0/800961611 shutdown_connections 2024-03-18T19:09:02.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.487+0000 7f318076c700 1 -- 172.21.15.1:0/800961611 wait complete. 2024-03-18T19:09:02.495 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-03-18T19:09:02.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:02.496 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T19:09:02.518 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_09_00_UTC 2024-03-18T19:09:02.518 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_09_00_UTC 2024-03-18T19:09:02.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:02.518 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_09_00_UTC) 2024-03-18T19:09:02.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:02.583 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-03-18T19:09:02.645 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-18T19:09:02.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.815+0000 7f27fab4f700 1 Processor -- start 2024-03-18T19:09:02.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.815+0000 7f27fab4f700 1 -- start start 2024-03-18T19:09:02.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.816+0000 7f27fab4f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f4071550 0x7f27f40b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:02.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.816+0000 7f27fab4f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f40b2b80 con 0x7f27f40b2650 2024-03-18T19:09:02.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.816+0000 7f27fab4f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f40b2e60 con 0x7f27f415d3b0 2024-03-18T19:09:02.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.816+0000 7f27fab4f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f40b3800 con 0x7f27f4071550 2024-03-18T19:09:02.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.816+0000 7f27f3fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f27f40b2650 0x7f27f4070e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54818/0 (socket says 172.21.15.1:54818) 2024-03-18T19:09:02.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.816+0000 7f27f3fff700 1 -- 172.21.15.1:0/445016341 learned_addr learned my addr 172.21.15.1:0/445016341 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:02.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.816+0000 7f27f90ec700 1 --2- 172.21.15.1:0/445016341 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f4071550 0x7f27f40b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:02.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.816+0000 7f27f37fe700 1 -- 172.21.15.1:0/445016341 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723635138 0 0) 0x7f27f40b2b80 con 0x7f27f40b2650 2024-03-18T19:09:02.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.816+0000 7f27f37fe700 1 -- 172.21.15.1:0/445016341 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27e40035b0 con 0x7f27f40b2650 2024-03-18T19:09:02.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f37fe700 1 -- 172.21.15.1:0/445016341 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284407857 0 0) 0x7f27f40b2e60 con 0x7f27f415d3b0 2024-03-18T19:09:02.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f37fe700 1 -- 172.21.15.1:0/445016341 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27f40b2b80 con 0x7f27f415d3b0 2024-03-18T19:09:02.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f37fe700 1 -- 172.21.15.1:0/445016341 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2911853473 0 0) 0x7f27e40035b0 con 0x7f27f40b2650 2024-03-18T19:09:02.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f37fe700 1 -- 172.21.15.1:0/445016341 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f4071550 msgr2=0x7f27f40b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:02.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f37fe700 1 --2- 172.21.15.1:0/445016341 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f4071550 0x7f27f40b3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:02.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f37fe700 1 -- 172.21.15.1:0/445016341 >> v1:172.21.15.165:6789/0 conn(0x7f27f415d3b0 legacy=0x7f27f406cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:02.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f37fe700 1 -- 172.21.15.1:0/445016341 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f40b3140 con 0x7f27f40b2650 2024-03-18T19:09:02.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f37fe700 1 -- 172.21.15.1:0/445016341 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f27e0001670 con 0x7f27f40b2650 2024-03-18T19:09:02.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f90ec700 1 --2- 172.21.15.1:0/445016341 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f4071550 0x7f27f40b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:02.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f37fe700 1 -- 172.21.15.1:0/445016341 >> v1:172.21.15.1:6789/0 conn(0x7f27f40b2650 legacy=0x7f27f4070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:02.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f37fe700 1 --2- 172.21.15.1:0/445016341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27e4003bc0 0x7f27e4003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:02.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f37fe700 1 --2- 172.21.15.1:0/445016341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27e4004850 0x7f27e4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:02.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f37fe700 1 --2- 172.21.15.1:0/445016341 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f4071550 0x7f27e4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:02.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f37fe700 1 -- 172.21.15.1:0/445016341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f40b3800 con 0x7f27e4004850 2024-03-18T19:09:02.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f37fe700 1 -- 172.21.15.1:0/445016341 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e400ba70 con 0x7f27e4003bc0 2024-03-18T19:09:02.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f37fe700 1 -- 172.21.15.1:0/445016341 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e400bc30 con 0x7f27f4071550 2024-03-18T19:09:02.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f3fff700 1 --2- 172.21.15.1:0/445016341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27e4004850 0x7f27e4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:02.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f88eb700 1 --2- 172.21.15.1:0/445016341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27e4003bc0 0x7f27e4003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:02.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f90ec700 1 --2- 172.21.15.1:0/445016341 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f4071550 0x7f27e4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:02.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f3fff700 1 -- 172.21.15.1:0/445016341 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f4071550 msgr2=0x7f27e4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:02.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f3fff700 1 --2- 172.21.15.1:0/445016341 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f4071550 0x7f27e4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:02.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f3fff700 1 -- 172.21.15.1:0/445016341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27e4003bc0 msgr2=0x7f27e4003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:02.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f3fff700 1 --2- 172.21.15.1:0/445016341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27e4003bc0 0x7f27e4003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:02.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f3fff700 1 -- 172.21.15.1:0/445016341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ec009a70 con 0x7f27e4004850 2024-03-18T19:09:02.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f90ec700 1 --2- 172.21.15.1:0/445016341 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f4071550 0x7f27e4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:02.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f3fff700 1 --2- 172.21.15.1:0/445016341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27e4004850 0x7f27e4004cd0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f27e000c210 tx=0x7f27e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b5754e517b265ba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:02.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f37fe700 1 -- 172.21.15.1:0/445016341 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e000cd50 con 0x7f27e4004850 2024-03-18T19:09:02.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f37fe700 1 -- 172.21.15.1:0/445016341 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e0005960 con 0x7f27e4004850 2024-03-18T19:09:02.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.817+0000 7f27f37fe700 1 -- 172.21.15.1:0/445016341 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e0005cf0 con 0x7f27e4004850 2024-03-18T19:09:02.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.818+0000 7f27fab4f700 1 -- 172.21.15.1:0/445016341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27e4004850 msgr2=0x7f27e4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:02.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.818+0000 7f27fab4f700 1 --2- 172.21.15.1:0/445016341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27e4004850 0x7f27e4004cd0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f27e000c210 tx=0x7f27e4005880 comp rx=0 tx=0).stop 2024-03-18T19:09:02.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.818+0000 7f27f90ec700 1 -- 172.21.15.1:0/445016341 reap_dead start 2024-03-18T19:09:02.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.818+0000 7f27fab4f700 1 -- 172.21.15.1:0/445016341 shutdown_connections 2024-03-18T19:09:02.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.818+0000 7f27fab4f700 1 -- 172.21.15.1:0/445016341 >> 172.21.15.1:0/445016341 conn(0x7f27f40b1350 msgr2=0x7f27f40b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:02.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.818+0000 7f27fab4f700 1 -- 172.21.15.1:0/445016341 shutdown_connections 2024-03-18T19:09:02.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.818+0000 7f27fab4f700 1 -- 172.21.15.1:0/445016341 wait complete. 2024-03-18T19:09:02.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.819+0000 7f27fab4f700 1 Processor -- start 2024-03-18T19:09:02.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.819+0000 7f27fab4f700 1 -- start start 2024-03-18T19:09:02.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.820+0000 7f27fab4f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27e4004850 0x7f27f4156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:02.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.820+0000 7f27fab4f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f41f2b40 0x7f27f4133e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:02.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.820+0000 7f27fab4f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27f41343a0 0x7f27f4134820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:02.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.820+0000 7f27fab4f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f4153010 con 0x7f27f41343a0 2024-03-18T19:09:02.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.820+0000 7f27fab4f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f4154ee0 con 0x7f27e4004850 2024-03-18T19:09:02.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.820+0000 7f27fab4f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f41f2fc0 con 0x7f27f41f2b40 2024-03-18T19:09:02.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.820+0000 7f27f90ec700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27f41343a0 0x7f27f4134820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:02.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.820+0000 7f27f90ec700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27f41343a0 0x7f27f4134820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57220/0 (socket says 172.21.15.1:57220) 2024-03-18T19:09:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.820+0000 7f27f90ec700 1 -- 172.21.15.1:0/2030087525 learned_addr learned my addr 172.21.15.1:0/2030087525 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.820+0000 7f27f3fff700 1 --2- 172.21.15.1:0/2030087525 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f41f2b40 0x7f27f4133e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.820+0000 7f27f88eb700 1 --2- 172.21.15.1:0/2030087525 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27e4004850 0x7f27f4156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.820+0000 7f27f90ec700 1 -- 172.21.15.1:0/2030087525 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f41f2b40 msgr2=0x7f27f4133e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.820+0000 7f27f90ec700 1 --2- 172.21.15.1:0/2030087525 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f41f2b40 0x7f27f4133e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.820+0000 7f27f90ec700 1 -- 172.21.15.1:0/2030087525 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27e4004850 msgr2=0x7f27f4156700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.820+0000 7f27f90ec700 1 --2- 172.21.15.1:0/2030087525 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27e4004850 0x7f27f4156700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.820+0000 7f27f90ec700 1 -- 172.21.15.1:0/2030087525 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f4154640 con 0x7f27f41343a0 2024-03-18T19:09:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.821+0000 7f27f90ec700 1 --2- 172.21.15.1:0/2030087525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27f41343a0 0x7f27f4134820 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f27f40725a0 tx=0x7f27ec0116e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.821+0000 7f27f17fa700 1 -- 172.21.15.1:0/2030087525 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec00e070 con 0x7f27f41343a0 2024-03-18T19:09:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.821+0000 7f27f17fa700 1 -- 172.21.15.1:0/2030087525 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ec011c40 con 0x7f27f41343a0 2024-03-18T19:09:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.821+0000 7f27f17fa700 1 -- 172.21.15.1:0/2030087525 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec011dd0 con 0x7f27f41343a0 2024-03-18T19:09:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.821+0000 7f27fab4f700 1 -- 172.21.15.1:0/2030087525 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f4154920 con 0x7f27f41343a0 2024-03-18T19:09:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.821+0000 7f27fab4f700 1 -- 172.21.15.1:0/2030087525 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f4145180 con 0x7f27f41343a0 2024-03-18T19:09:02.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.822+0000 7f27fab4f700 1 -- 172.21.15.1:0/2030087525 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f40676a0 con 0x7f27f41343a0 2024-03-18T19:09:02.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.822+0000 7f27f17fa700 1 -- 172.21.15.1:0/2030087525 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 67) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f27ec003a20 con 0x7f27f41343a0 2024-03-18T19:09:02.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.823+0000 7f27f17fa700 1 --2- 172.21.15.1:0/2030087525 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f27d806af20 0x7f27d806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:02.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.823+0000 7f27f17fa700 1 -- 172.21.15.1:0/2030087525 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f27ec089f30 con 0x7f27f41343a0 2024-03-18T19:09:02.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.823+0000 7f27f88eb700 1 --2- 172.21.15.1:0/2030087525 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f27d806af20 0x7f27d806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:02.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.823+0000 7f27f88eb700 1 --2- 172.21.15.1:0/2030087525 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f27d806af20 0x7f27d806d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f27e8007fd0 tx=0x7f27e8009040 comp rx=0 tx=0).ready entity=mgr.8469 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:02.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:02.827+0000 7f27f17fa700 1 -- 172.21.15.1:0/2030087525 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f27ec054a00 con 0x7f27f41343a0 2024-03-18T19:09:03.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.087+0000 7f27fab4f700 1 -- 172.21.15.1:0/2030087525 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f27f40b0e20 con 0x7f27f41343a0 2024-03-18T19:09:03.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.155+0000 7f27f17fa700 1 -- 172.21.15.1:0/2030087525 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 68) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f27ec003cd0 con 0x7f27f41343a0 2024-03-18T19:09:03.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.155+0000 7f27f17fa700 1 -- 172.21.15.1:0/2030087525 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v68) v1 ==== 93+0+0 (secure 0 0 0) 0x7f27ec0860e0 con 0x7f27f41343a0 2024-03-18T19:09:03.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.157+0000 7f27fab4f700 1 -- 172.21.15.1:0/2030087525 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f27d806af20 msgr2=0x7f27d806d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:03.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.157+0000 7f27fab4f700 1 --2- 172.21.15.1:0/2030087525 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f27d806af20 0x7f27d806d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f27e8007fd0 tx=0x7f27e8009040 comp rx=0 tx=0).stop 2024-03-18T19:09:03.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.157+0000 7f27fab4f700 1 -- 172.21.15.1:0/2030087525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27f41343a0 msgr2=0x7f27f4134820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:03.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.157+0000 7f27fab4f700 1 --2- 172.21.15.1:0/2030087525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27f41343a0 0x7f27f4134820 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f27f40725a0 tx=0x7f27ec0116e0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.158+0000 7f27fab4f700 1 -- 172.21.15.1:0/2030087525 shutdown_connections 2024-03-18T19:09:03.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.158+0000 7f27fab4f700 1 --2- 172.21.15.1:0/2030087525 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f27d806af20 0x7f27d806d3e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.158+0000 7f27fab4f700 1 --2- 172.21.15.1:0/2030087525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27f41343a0 0x7f27f4134820 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.158+0000 7f27fab4f700 1 --2- 172.21.15.1:0/2030087525 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27e4004850 0x7f27f4156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.158+0000 7f27fab4f700 1 --2- 172.21.15.1:0/2030087525 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27f41f2b40 0x7f27f4133e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.158+0000 7f27fab4f700 1 -- 172.21.15.1:0/2030087525 >> 172.21.15.1:0/2030087525 conn(0x7f27f40b1350 msgr2=0x7f27f406df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:03.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.158+0000 7f27fab4f700 1 -- 172.21.15.1:0/2030087525 shutdown_connections 2024-03-18T19:09:03.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.158+0000 7f27fab4f700 1 -- 172.21.15.1:0/2030087525 wait complete. 2024-03-18T19:09:03.167 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T19:09:03.167 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:09:03.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:09:03.249 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T19:09:03.250 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:09:03.250 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:09:03.269 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:09:03.298 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:03.301+0000 7f868eda6240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:09:03.298 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:03.302+0000 7fcaa747b240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:09:03.443 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:03.446+0000 7fcaa747b240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:09:03.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.471+0000 7f4485bfb700 1 Processor -- start 2024-03-18T19:09:03.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.471+0000 7f4485bfb700 1 -- start start 2024-03-18T19:09:03.468 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:03.471+0000 7f868eda6240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:09:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.472+0000 7f4485bfb700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4480154300 0x7f4480070500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.472+0000 7f4485bfb700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4480070a40 0x7f448006ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.472+0000 7f4485bfb700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4480160e90 con 0x7f448006b080 2024-03-18T19:09:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.472+0000 7f4485bfb700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4480161d60 con 0x7f4480154300 2024-03-18T19:09:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.472+0000 7f4485bfb700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4480070ea0 con 0x7f4480070a40 2024-03-18T19:09:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.472+0000 7f447f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4480154300 0x7f4480070500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.472+0000 7f447f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4480154300 0x7f4480070500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57238/0 (socket says 172.21.15.1:57238) 2024-03-18T19:09:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.472+0000 7f447f7fe700 1 -- 172.21.15.1:0/1678150587 learned_addr learned my addr 172.21.15.1:0/1678150587 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.472+0000 7f447effd700 1 --2- 172.21.15.1:0/1678150587 >> v2:172.21.15.165:3300/0 conn(0x7f4480070a40 0x7f448006ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.472+0000 7f447f7fe700 1 -- 172.21.15.1:0/1678150587 >> v2:172.21.15.165:3300/0 conn(0x7f4480070a40 msgr2=0x7f448006ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.472+0000 7f447f7fe700 1 --2- 172.21.15.1:0/1678150587 >> v2:172.21.15.165:3300/0 conn(0x7f4480070a40 0x7f448006ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.472+0000 7f447f7fe700 1 -- 172.21.15.1:0/1678150587 >> v1:172.21.15.165:6789/0 conn(0x7f448006b080 legacy=0x7f448015dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.472+0000 7f447f7fe700 1 -- 172.21.15.1:0/1678150587 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44800cec40 con 0x7f4480154300 2024-03-18T19:09:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.472+0000 7f447f7fe700 1 --2- 172.21.15.1:0/1678150587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4480154300 0x7f4480070500 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f448006dc50 tx=0x7f4470009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e41e8b8f12424077 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.472+0000 7f447e7fc700 1 -- 172.21.15.1:0/1678150587 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447000b870 con 0x7f4480154300 2024-03-18T19:09:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.472+0000 7f447e7fc700 1 -- 172.21.15.1:0/1678150587 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f447000b9d0 con 0x7f4480154300 2024-03-18T19:09:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.473+0000 7f447e7fc700 1 -- 172.21.15.1:0/1678150587 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447000bd60 con 0x7f4480154300 2024-03-18T19:09:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.473+0000 7f4485bfb700 1 -- 172.21.15.1:0/1678150587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4480154300 msgr2=0x7f4480070500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.473+0000 7f4485bfb700 1 --2- 172.21.15.1:0/1678150587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4480154300 0x7f4480070500 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f448006dc50 tx=0x7f4470009bd0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.473+0000 7f4485bfb700 1 -- 172.21.15.1:0/1678150587 shutdown_connections 2024-03-18T19:09:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.473+0000 7f4485bfb700 1 --2- 172.21.15.1:0/1678150587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4480154300 0x7f4480070500 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.473+0000 7f4485bfb700 1 --2- 172.21.15.1:0/1678150587 >> v2:172.21.15.165:3300/0 conn(0x7f4480070a40 0x7f448006ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.473+0000 7f4485bfb700 1 -- 172.21.15.1:0/1678150587 >> 172.21.15.1:0/1678150587 conn(0x7f44800b0a00 msgr2=0x7f44800b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.473+0000 7f4485bfb700 1 -- 172.21.15.1:0/1678150587 shutdown_connections 2024-03-18T19:09:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.473+0000 7f4485bfb700 1 -- 172.21.15.1:0/1678150587 wait complete. 2024-03-18T19:09:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.475+0000 7f4485bfb700 1 Processor -- start 2024-03-18T19:09:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.475+0000 7f4485bfb700 1 -- start start 2024-03-18T19:09:03.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.476+0000 7f4485bfb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f448006b080 0x7f448009c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.476+0000 7f4485bfb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4480070a40 0x7f4480155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.476+0000 7f4485bfb700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4480154300 0x7f448009ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.476+0000 7f4485bfb700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4480146470 con 0x7f448006b080 2024-03-18T19:09:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.476+0000 7f4485bfb700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4480099d70 con 0x7f4480154300 2024-03-18T19:09:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.476+0000 7f4485bfb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f448009a050 con 0x7f4480070a40 2024-03-18T19:09:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.476+0000 7f447f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f448006b080 0x7f448009c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.476+0000 7f447f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f448006b080 0x7f448009c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57242/0 (socket says 172.21.15.1:57242) 2024-03-18T19:09:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.476+0000 7f447f7fe700 1 -- 172.21.15.1:0/1487523024 learned_addr learned my addr 172.21.15.1:0/1487523024 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.476+0000 7f447ffff700 1 --2- 172.21.15.1:0/1487523024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4480154300 0x7f448009ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.476+0000 7f447effd700 1 --2- 172.21.15.1:0/1487523024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4480070a40 0x7f4480155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.476+0000 7f447f7fe700 1 -- 172.21.15.1:0/1487523024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4480070a40 msgr2=0x7f4480155bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.476+0000 7f447f7fe700 1 --2- 172.21.15.1:0/1487523024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4480070a40 0x7f4480155bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.476+0000 7f447f7fe700 1 -- 172.21.15.1:0/1487523024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4480154300 msgr2=0x7f448009ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.476+0000 7f447f7fe700 1 --2- 172.21.15.1:0/1487523024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4480154300 0x7f448009ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.476+0000 7f447f7fe700 1 -- 172.21.15.1:0/1487523024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4470009550 con 0x7f448006b080 2024-03-18T19:09:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.477+0000 7f447f7fe700 1 --2- 172.21.15.1:0/1487523024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f448006b080 0x7f448009c370 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f4470000c00 tx=0x7f4470004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.477+0000 7f4477fff700 1 -- 172.21.15.1:0/1487523024 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4470004310 con 0x7f448006b080 2024-03-18T19:09:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.477+0000 7f4477fff700 1 -- 172.21.15.1:0/1487523024 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f447001e070 con 0x7f448006b080 2024-03-18T19:09:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.477+0000 7f4485bfb700 1 -- 172.21.15.1:0/1487523024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f448009a330 con 0x7f448006b080 2024-03-18T19:09:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.477+0000 7f4477fff700 1 -- 172.21.15.1:0/1487523024 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44700036a0 con 0x7f448006b080 2024-03-18T19:09:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.477+0000 7f4485bfb700 1 -- 172.21.15.1:0/1487523024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f448009a8f0 con 0x7f448006b080 2024-03-18T19:09:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.478+0000 7f4485bfb700 1 -- 172.21.15.1:0/1487523024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4480073350 con 0x7f448006b080 2024-03-18T19:09:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.478+0000 7f4477fff700 1 -- 172.21.15.1:0/1487523024 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 68) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f4470020030 con 0x7f448006b080 2024-03-18T19:09:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.479+0000 7f4477fff700 1 --2- 172.21.15.1:0/1487523024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f446006aed0 0x7f446006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.479+0000 7f447effd700 1 -- 172.21.15.1:0/1487523024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f446006aed0 msgr2=0x7f446006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.479+0000 7f4477fff700 1 -- 172.21.15.1:0/1487523024 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f447008a860 con 0x7f448006b080 2024-03-18T19:09:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.479+0000 7f447effd700 1 --2- 172.21.15.1:0/1487523024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f446006aed0 0x7f446006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:03.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.484+0000 7f4477fff700 1 -- 172.21.15.1:0/1487523024 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4470055520 con 0x7f448006b080 2024-03-18T19:09:03.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.680+0000 7f447effd700 1 -- 172.21.15.1:0/1487523024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f446006aed0 msgr2=0x7f446006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:03.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.680+0000 7f447effd700 1 --2- 172.21.15.1:0/1487523024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f446006aed0 0x7f446006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:03.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.687+0000 7f4485bfb700 1 -- 172.21.15.1:0/1487523024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44800b3ff0 con 0x7f448006b080 2024-03-18T19:09:03.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.688+0000 7f4477fff700 1 -- 172.21.15.1:0/1487523024 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+16682 (secure 0 0 0) 0x7f4470058b40 con 0x7f448006b080 2024-03-18T19:09:03.685 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:03.686 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":134,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:08:34.815984+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:08:17.419247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-18T19:08:18.552871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3989116924":"2024-03-19T19:08:34.815944+0000","172.21.15.1:6832/76998":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/3859645093":"2024-03-19T19:08:34.815944+0000","172.21.15.1:6833/76998":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/748689501":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/1964926516":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/2849309461":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/3265489674":"2024-03-19T19:08:34.815944+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:09:03.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.691+0000 7f4485bfb700 1 -- 172.21.15.1:0/1487523024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f446006aed0 msgr2=0x7f446006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:03.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.691+0000 7f4485bfb700 1 --2- 172.21.15.1:0/1487523024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f446006aed0 0x7f446006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.691+0000 7f4485bfb700 1 -- 172.21.15.1:0/1487523024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f448006b080 msgr2=0x7f448009c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:03.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.691+0000 7f4485bfb700 1 --2- 172.21.15.1:0/1487523024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f448006b080 0x7f448009c370 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f4470000c00 tx=0x7f4470004750 comp rx=0 tx=0).stop 2024-03-18T19:09:03.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.691+0000 7f4485bfb700 1 -- 172.21.15.1:0/1487523024 shutdown_connections 2024-03-18T19:09:03.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.691+0000 7f4485bfb700 1 --2- 172.21.15.1:0/1487523024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f446006aed0 0x7f446006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.691+0000 7f4485bfb700 1 --2- 172.21.15.1:0/1487523024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4480154300 0x7f448009ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.691+0000 7f4485bfb700 1 --2- 172.21.15.1:0/1487523024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f448006b080 0x7f448009c370 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.691+0000 7f4485bfb700 1 --2- 172.21.15.1:0/1487523024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4480070a40 0x7f4480155bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.691+0000 7f4485bfb700 1 -- 172.21.15.1:0/1487523024 >> 172.21.15.1:0/1487523024 conn(0x7f44800b0a00 msgr2=0x7f4480153860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:03.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.691+0000 7f4485bfb700 1 -- 172.21.15.1:0/1487523024 shutdown_connections 2024-03-18T19:09:03.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.691+0000 7f4485bfb700 1 -- 172.21.15.1:0/1487523024 wait complete. 2024-03-18T19:09:03.700 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T19:08:17.419247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '132', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T19:08:18.552871+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '133', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:09:03.701 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:09:03.707 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:03.711+0000 7fcaa747b240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:09:03.752 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:03.756+0000 7f868eda6240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:09:03.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.835+0000 7f1f02206700 1 Processor -- start 2024-03-18T19:09:03.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.835+0000 7f1f02206700 1 -- start start 2024-03-18T19:09:03.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.836+0000 7f1f02206700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1efc099a40 0x7f1efc0ade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:03.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.836+0000 7f1f02206700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1efc099e20 0x7f1efc0cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:03.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.836+0000 7f1f02206700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1efc0b3aa0 con 0x7f1efc0ccc20 2024-03-18T19:09:03.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.836+0000 7f1f02206700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1efc06e9b0 con 0x7f1efc099a40 2024-03-18T19:09:03.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.836+0000 7f1f02206700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1efc06ec90 con 0x7f1efc099e20 2024-03-18T19:09:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.836+0000 7f1efbfff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1efc0ccc20 0x7f1efc06fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54826/0 (socket says 172.21.15.1:54826) 2024-03-18T19:09:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.836+0000 7f1efbfff700 1 -- 172.21.15.1:0/713091747 learned_addr learned my addr 172.21.15.1:0/713091747 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.836+0000 7f1efb7fe700 1 --2- 172.21.15.1:0/713091747 >> v2:172.21.15.165:3300/0 conn(0x7f1efc099a40 0x7f1efc0ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.836+0000 7f1ef2ffd700 1 --2- 172.21.15.1:0/713091747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1efc099e20 0x7f1efc0cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.837+0000 7f1efaffd700 1 -- 172.21.15.1:0/713091747 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445267051 0 0) 0x7f1efc0b3aa0 con 0x7f1efc0ccc20 2024-03-18T19:09:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.837+0000 7f1efaffd700 1 -- 172.21.15.1:0/713091747 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ee80035b0 con 0x7f1efc0ccc20 2024-03-18T19:09:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.837+0000 7f1efb7fe700 1 -- 172.21.15.1:0/713091747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1efc099e20 msgr2=0x7f1efc0cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.837+0000 7f1efb7fe700 1 --2- 172.21.15.1:0/713091747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1efc099e20 0x7f1efc0cc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.837+0000 7f1efb7fe700 1 -- 172.21.15.1:0/713091747 >> v1:172.21.15.1:6789/0 conn(0x7f1efc0ccc20 legacy=0x7f1efc06fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.837+0000 7f1efb7fe700 1 -- 172.21.15.1:0/713091747 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1efc06ef70 con 0x7f1efc099a40 2024-03-18T19:09:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.837+0000 7f1ef2ffd700 1 --2- 172.21.15.1:0/713091747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1efc099e20 0x7f1efc0cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.837+0000 7f1efb7fe700 1 --2- 172.21.15.1:0/713091747 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1efc099a40 0x7f1efc0ade00 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1eec009840 tx=0x7f1eec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d7d64d090d72bda server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.837+0000 7f1efaffd700 1 -- 172.21.15.1:0/713091747 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eec011830 con 0x7f1efc099a40 2024-03-18T19:09:03.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.837+0000 7f1efaffd700 1 -- 172.21.15.1:0/713091747 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eec011990 con 0x7f1efc099a40 2024-03-18T19:09:03.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.837+0000 7f1efaffd700 1 -- 172.21.15.1:0/713091747 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eec011d20 con 0x7f1efc099a40 2024-03-18T19:09:03.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.838+0000 7f1f02206700 1 -- 172.21.15.1:0/713091747 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1efc099a40 msgr2=0x7f1efc0ade00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.838+0000 7f1f02206700 1 --2- 172.21.15.1:0/713091747 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1efc099a40 0x7f1efc0ade00 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1eec009840 tx=0x7f1eec009b50 comp rx=0 tx=0).stop 2024-03-18T19:09:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.838+0000 7f1f02206700 1 -- 172.21.15.1:0/713091747 shutdown_connections 2024-03-18T19:09:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.838+0000 7f1f02206700 1 --2- 172.21.15.1:0/713091747 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1efc099a40 0x7f1efc0ade00 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.838+0000 7f1f02206700 1 --2- 172.21.15.1:0/713091747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1efc099e20 0x7f1efc0cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.838+0000 7f1f02206700 1 -- 172.21.15.1:0/713091747 >> 172.21.15.1:0/713091747 conn(0x7f1efc0b2c50 msgr2=0x7f1efc0b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.838+0000 7f1f02206700 1 -- 172.21.15.1:0/713091747 shutdown_connections 2024-03-18T19:09:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.838+0000 7f1f02206700 1 -- 172.21.15.1:0/713091747 wait complete. 2024-03-18T19:09:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.839+0000 7f1f02206700 1 Processor -- start 2024-03-18T19:09:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.839+0000 7f1f02206700 1 -- start start 2024-03-18T19:09:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.840+0000 7f1f02206700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1efc099a40 0x7f1efc157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.840+0000 7f1f02206700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1efc099e20 0x7f1efc0b46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.840+0000 7f1f02206700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1efc0ccc20 0x7f1efc0b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.840+0000 7f1f02206700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1efc158790 con 0x7f1efc099a40 2024-03-18T19:09:03.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.840+0000 7f1f02206700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1efc1582c0 con 0x7f1efc0ccc20 2024-03-18T19:09:03.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.840+0000 7f1f02206700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1efc0b5410 con 0x7f1efc099e20 2024-03-18T19:09:03.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.840+0000 7f1efb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1efc099a40 0x7f1efc157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:03.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.840+0000 7f1efb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1efc099a40 0x7f1efc157d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57248/0 (socket says 172.21.15.1:57248) 2024-03-18T19:09:03.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.840+0000 7f1efb7fe700 1 -- 172.21.15.1:0/4203305967 learned_addr learned my addr 172.21.15.1:0/4203305967 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:03.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.840+0000 7f1ef2ffd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1efc099e20 0x7f1efc0b46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:03.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.840+0000 7f1efbfff700 1 --2- 172.21.15.1:0/4203305967 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1efc0ccc20 0x7f1efc0b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:03.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.841+0000 7f1efb7fe700 1 -- 172.21.15.1:0/4203305967 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1efc099e20 msgr2=0x7f1efc0b46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:03.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.841+0000 7f1efb7fe700 1 --2- 172.21.15.1:0/4203305967 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1efc099e20 0x7f1efc0b46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.841+0000 7f1efb7fe700 1 -- 172.21.15.1:0/4203305967 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1efc0ccc20 msgr2=0x7f1efc0b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:03.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.841+0000 7f1efb7fe700 1 --2- 172.21.15.1:0/4203305967 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1efc0ccc20 0x7f1efc0b4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:03.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.841+0000 7f1efb7fe700 1 -- 172.21.15.1:0/4203305967 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee40094a0 con 0x7f1efc099a40 2024-03-18T19:09:03.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.841+0000 7f1ef2ffd700 1 --2- 172.21.15.1:0/4203305967 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1efc099e20 0x7f1efc0b46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:03.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.841+0000 7f1efbfff700 1 --2- 172.21.15.1:0/4203305967 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1efc0ccc20 0x7f1efc0b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:03.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.841+0000 7f1efb7fe700 1 --2- 172.21.15.1:0/4203305967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1efc099a40 0x7f1efc157d80 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f1eec009fd0 tx=0x7f1eec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:03.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.841+0000 7f1ef8ff9700 1 -- 172.21.15.1:0/4203305967 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eec004590 con 0x7f1efc099a40 2024-03-18T19:09:03.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.841+0000 7f1ef8ff9700 1 -- 172.21.15.1:0/4203305967 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eec01e070 con 0x7f1efc099a40 2024-03-18T19:09:03.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.841+0000 7f1f02206700 1 -- 172.21.15.1:0/4203305967 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eec0094a0 con 0x7f1efc099a40 2024-03-18T19:09:03.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.841+0000 7f1ef8ff9700 1 -- 172.21.15.1:0/4203305967 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eec0038e0 con 0x7f1efc099a40 2024-03-18T19:09:03.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.841+0000 7f1f02206700 1 -- 172.21.15.1:0/4203305967 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1efc0b59d0 con 0x7f1efc099a40 2024-03-18T19:09:03.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.842+0000 7f1f02206700 1 -- 172.21.15.1:0/4203305967 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1efc0b5bc0 con 0x7f1efc099a40 2024-03-18T19:09:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.843+0000 7f1ef8ff9700 1 -- 172.21.15.1:0/4203305967 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 68) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f1eec003a40 con 0x7f1efc099a40 2024-03-18T19:09:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.843+0000 7f1ef8ff9700 1 --2- 172.21.15.1:0/4203305967 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1edc06aed0 0x7f1edc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:03.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.843+0000 7f1ef2ffd700 1 -- 172.21.15.1:0/4203305967 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1edc06aed0 msgr2=0x7f1edc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:03.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.843+0000 7f1ef8ff9700 1 -- 172.21.15.1:0/4203305967 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f1eec08a790 con 0x7f1efc099a40 2024-03-18T19:09:03.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.843+0000 7f1ef2ffd700 1 --2- 172.21.15.1:0/4203305967 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1edc06aed0 0x7f1edc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:03.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:03.852+0000 7f1ef8ff9700 1 -- 172.21.15.1:0/4203305967 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1eec055380 con 0x7f1efc099a40 2024-03-18T19:09:04.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.044+0000 7f1ef2ffd700 1 -- 172.21.15.1:0/4203305967 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1edc06aed0 msgr2=0x7f1edc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:04.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.044+0000 7f1ef2ffd700 1 --2- 172.21.15.1:0/4203305967 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1edc06aed0 0x7f1edc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:04.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.056+0000 7f1f02206700 1 -- 172.21.15.1:0/4203305967 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1efc0676a0 con 0x7f1efc099a40 2024-03-18T19:09:04.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.056+0000 7f1ef8ff9700 1 -- 172.21.15.1:0/4203305967 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v134) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1eec0589a0 con 0x7f1efc099a40 2024-03-18T19:09:04.053 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:09:04.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.058+0000 7f1f02206700 1 -- 172.21.15.1:0/4203305967 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1edc06aed0 msgr2=0x7f1edc06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:04.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.058+0000 7f1f02206700 1 --2- 172.21.15.1:0/4203305967 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1edc06aed0 0x7f1edc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.058+0000 7f1f02206700 1 -- 172.21.15.1:0/4203305967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1efc099a40 msgr2=0x7f1efc157d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:04.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.058+0000 7f1f02206700 1 --2- 172.21.15.1:0/4203305967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1efc099a40 0x7f1efc157d80 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f1eec009fd0 tx=0x7f1eec004750 comp rx=0 tx=0).stop 2024-03-18T19:09:04.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.058+0000 7f1f02206700 1 -- 172.21.15.1:0/4203305967 shutdown_connections 2024-03-18T19:09:04.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.058+0000 7f1f02206700 1 --2- 172.21.15.1:0/4203305967 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1edc06aed0 0x7f1edc06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.058+0000 7f1f02206700 1 --2- 172.21.15.1:0/4203305967 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1efc0ccc20 0x7f1efc0b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.058+0000 7f1f02206700 1 --2- 172.21.15.1:0/4203305967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1efc099a40 0x7f1efc157d80 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.058+0000 7f1f02206700 1 --2- 172.21.15.1:0/4203305967 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1efc099e20 0x7f1efc0b46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.058+0000 7f1f02206700 1 -- 172.21.15.1:0/4203305967 >> 172.21.15.1:0/4203305967 conn(0x7f1efc0b2c50 msgr2=0x7f1efc06d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:04.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.058+0000 7f1f02206700 1 -- 172.21.15.1:0/4203305967 shutdown_connections 2024-03-18T19:09:04.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.058+0000 7f1f02206700 1 -- 172.21.15.1:0/4203305967 wait complete. 2024-03-18T19:09:04.069 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T19:09:04.130 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:04.134+0000 7fcaa747b240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:09:04.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.202+0000 7f40a756b700 1 Processor -- start 2024-03-18T19:09:04.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.203+0000 7f40a756b700 1 -- start start 2024-03-18T19:09:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.203+0000 7f40a756b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f40a00b1e30 0x7f40a0166210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.203+0000 7f40a756b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40a015ef80 con 0x7f40a00b0500 2024-03-18T19:09:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.203+0000 7f40a756b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40a00cc9a0 con 0x7f40a015eba0 2024-03-18T19:09:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.203+0000 7f40a756b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40a00d0190 con 0x7f40a00b1e30 2024-03-18T19:09:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.204+0000 7f40a4b06700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f40a00b0500 0x7f40a00cfa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54834/0 (socket says 172.21.15.1:54834) 2024-03-18T19:09:04.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.204+0000 7f40a4b06700 1 -- 172.21.15.1:0/205219191 learned_addr learned my addr 172.21.15.1:0/205219191 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:04.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.204+0000 7f40a5b08700 1 --2- 172.21.15.1:0/205219191 >> v2:172.21.15.165:3300/0 conn(0x7f40a00b1e30 0x7f40a0166210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:04.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.204+0000 7f4097fff700 1 -- 172.21.15.1:0/205219191 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3796933519 0 0) 0x7f40a015ef80 con 0x7f40a00b0500 2024-03-18T19:09:04.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.204+0000 7f4097fff700 1 -- 172.21.15.1:0/205219191 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f408c0035b0 con 0x7f40a00b0500 2024-03-18T19:09:04.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.204+0000 7f40a5b08700 1 -- 172.21.15.1:0/205219191 >> v1:172.21.15.165:6789/0 conn(0x7f40a015eba0 legacy=0x7f40a00cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:04.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.204+0000 7f4097fff700 1 -- 172.21.15.1:0/205219191 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367644534 0 0) 0x7f40a00cc9a0 con 0x7f40a015eba0 2024-03-18T19:09:04.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.204+0000 7f40a5b08700 1 -- 172.21.15.1:0/205219191 >> v1:172.21.15.1:6789/0 conn(0x7f40a00b0500 legacy=0x7f40a00cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:04.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.204+0000 7f40a5b08700 1 -- 172.21.15.1:0/205219191 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a00ccc80 con 0x7f40a00b1e30 2024-03-18T19:09:04.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.204+0000 7f40a5b08700 1 --2- 172.21.15.1:0/205219191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40a00b1e30 0x7f40a0166210 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4098009d90 tx=0x7f409800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bbec4f103ee9b8d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:04.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.204+0000 7f4097fff700 1 -- 172.21.15.1:0/205219191 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409800e070 con 0x7f40a00b1e30 2024-03-18T19:09:04.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.204+0000 7f4097fff700 1 -- 172.21.15.1:0/205219191 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409800bb20 con 0x7f40a00b1e30 2024-03-18T19:09:04.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.204+0000 7f4097fff700 1 -- 172.21.15.1:0/205219191 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409800bed0 con 0x7f40a00b1e30 2024-03-18T19:09:04.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.205+0000 7f40a756b700 1 -- 172.21.15.1:0/205219191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40a00b1e30 msgr2=0x7f40a0166210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:04.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.205+0000 7f40a756b700 1 --2- 172.21.15.1:0/205219191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40a00b1e30 0x7f40a0166210 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4098009d90 tx=0x7f409800b330 comp rx=0 tx=0).stop 2024-03-18T19:09:04.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.205+0000 7f40a756b700 1 -- 172.21.15.1:0/205219191 shutdown_connections 2024-03-18T19:09:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.205+0000 7f40a756b700 1 --2- 172.21.15.1:0/205219191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40a00b1e30 0x7f40a0166210 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.205+0000 7f40a756b700 1 -- 172.21.15.1:0/205219191 >> 172.21.15.1:0/205219191 conn(0x7f40a00b2bd0 msgr2=0x7f40a00b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.205+0000 7f40a756b700 1 -- 172.21.15.1:0/205219191 shutdown_connections 2024-03-18T19:09:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.205+0000 7f40a756b700 1 -- 172.21.15.1:0/205219191 wait complete. 2024-03-18T19:09:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.206+0000 7f40a756b700 1 Processor -- start 2024-03-18T19:09:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.206+0000 7f40a756b700 1 -- start start 2024-03-18T19:09:04.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.207+0000 7f40a756b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40a00b0500 0x7f40a014ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:04.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.207+0000 7f40a756b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40a00b1e30 0x7f40a0151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:04.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.207+0000 7f40a756b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40a015eba0 0x7f40a006b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:04.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.207+0000 7f40a756b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a014f200 con 0x7f40a00b0500 2024-03-18T19:09:04.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.207+0000 7f40a756b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a0164ff0 con 0x7f40a00b1e30 2024-03-18T19:09:04.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.207+0000 7f40a756b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a006c3d0 con 0x7f40a015eba0 2024-03-18T19:09:04.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.207+0000 7f40a5307700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40a00b0500 0x7f40a014ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:04.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.207+0000 7f40a5307700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40a00b0500 0x7f40a014ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57258/0 (socket says 172.21.15.1:57258) 2024-03-18T19:09:04.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.207+0000 7f40a5307700 1 -- 172.21.15.1:0/550672533 learned_addr learned my addr 172.21.15.1:0/550672533 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:04.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.207+0000 7f40a4b06700 1 --2- 172.21.15.1:0/550672533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40a00b1e30 0x7f40a0151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:04.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.207+0000 7f40a5b08700 1 --2- 172.21.15.1:0/550672533 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40a015eba0 0x7f40a006b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:04.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.208+0000 7f40a5307700 1 -- 172.21.15.1:0/550672533 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40a015eba0 msgr2=0x7f40a006b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:04.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.208+0000 7f40a5307700 1 --2- 172.21.15.1:0/550672533 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40a015eba0 0x7f40a006b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.208+0000 7f40a5307700 1 -- 172.21.15.1:0/550672533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40a00b1e30 msgr2=0x7f40a0151d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:04.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.208+0000 7f40a5307700 1 --2- 172.21.15.1:0/550672533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40a00b1e30 0x7f40a0151d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.208+0000 7f40a5307700 1 -- 172.21.15.1:0/550672533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4098009a70 con 0x7f40a00b0500 2024-03-18T19:09:04.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.208+0000 7f40a5307700 1 --2- 172.21.15.1:0/550672533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40a00b0500 0x7f40a014ecc0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f409c00c550 tx=0x7f409c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:04.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.208+0000 7f4095ffb700 1 -- 172.21.15.1:0/550672533 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c0054e0 con 0x7f40a00b0500 2024-03-18T19:09:04.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.208+0000 7f4095ffb700 1 -- 172.21.15.1:0/550672533 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409c003680 con 0x7f40a00b0500 2024-03-18T19:09:04.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.208+0000 7f40a756b700 1 -- 172.21.15.1:0/550672533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a006c6b0 con 0x7f40a00b0500 2024-03-18T19:09:04.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.208+0000 7f4095ffb700 1 -- 172.21.15.1:0/550672533 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c003a30 con 0x7f40a00b0500 2024-03-18T19:09:04.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.208+0000 7f40a756b700 1 -- 172.21.15.1:0/550672533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a0164c80 con 0x7f40a00b0500 2024-03-18T19:09:04.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.209+0000 7f40a756b700 1 -- 172.21.15.1:0/550672533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40a00676a0 con 0x7f40a00b0500 2024-03-18T19:09:04.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.210+0000 7f4095ffb700 1 -- 172.21.15.1:0/550672533 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 68) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f409c003b90 con 0x7f40a00b0500 2024-03-18T19:09:04.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.210+0000 7f4095ffb700 1 --2- 172.21.15.1:0/550672533 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f408406aed0 0x7f408406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:04.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.211+0000 7f40a4b06700 1 -- 172.21.15.1:0/550672533 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f408406aed0 msgr2=0x7f408406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:04.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.211+0000 7f4095ffb700 1 -- 172.21.15.1:0/550672533 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f409c089240 con 0x7f40a00b0500 2024-03-18T19:09:04.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.211+0000 7f40a4b06700 1 --2- 172.21.15.1:0/550672533 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f408406aed0 0x7f408406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:04.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.219+0000 7f4095ffb700 1 -- 172.21.15.1:0/550672533 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f409c053d80 con 0x7f40a00b0500 2024-03-18T19:09:04.223 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:04.226+0000 7f868eda6240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:09:04.340 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:04.343+0000 7fcaa747b240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:09:04.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.412+0000 7f40a4b06700 1 -- 172.21.15.1:0/550672533 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f408406aed0 msgr2=0x7f408406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:04.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.412+0000 7f40a4b06700 1 --2- 172.21.15.1:0/550672533 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f408406aed0 0x7f408406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:04.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.421+0000 7f40a756b700 1 -- 172.21.15.1:0/550672533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f40a00b2800 con 0x7f40a00b0500 2024-03-18T19:09:04.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.422+0000 7f4095ffb700 1 -- 172.21.15.1:0/550672533 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v134) v1 ==== 104+0+11 (secure 0 0 0) 0x7f409c0573a0 con 0x7f40a00b0500 2024-03-18T19:09:04.419 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T19:09:04.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.424+0000 7f40a756b700 1 -- 172.21.15.1:0/550672533 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f408406aed0 msgr2=0x7f408406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.424+0000 7f40a756b700 1 --2- 172.21.15.1:0/550672533 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f408406aed0 0x7f408406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.424+0000 7f40a756b700 1 -- 172.21.15.1:0/550672533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40a00b0500 msgr2=0x7f40a014ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.424+0000 7f40a756b700 1 --2- 172.21.15.1:0/550672533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40a00b0500 0x7f40a014ecc0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f409c00c550 tx=0x7f409c00c860 comp rx=0 tx=0).stop 2024-03-18T19:09:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.424+0000 7f40a756b700 1 -- 172.21.15.1:0/550672533 shutdown_connections 2024-03-18T19:09:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.424+0000 7f40a756b700 1 --2- 172.21.15.1:0/550672533 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f408406aed0 0x7f408406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.424+0000 7f40a756b700 1 --2- 172.21.15.1:0/550672533 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40a015eba0 0x7f40a006b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.424+0000 7f40a756b700 1 --2- 172.21.15.1:0/550672533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40a00b0500 0x7f40a014ecc0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.424+0000 7f40a756b700 1 --2- 172.21.15.1:0/550672533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40a00b1e30 0x7f40a0151d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.424+0000 7f40a756b700 1 -- 172.21.15.1:0/550672533 >> 172.21.15.1:0/550672533 conn(0x7f40a00b2bd0 msgr2=0x7f40a0156b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.424+0000 7f40a756b700 1 -- 172.21.15.1:0/550672533 shutdown_connections 2024-03-18T19:09:04.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.424+0000 7f40a756b700 1 -- 172.21.15.1:0/550672533 wait complete. 2024-03-18T19:09:04.433 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T19:09:04.445 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:04.448+0000 7f868eda6240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:09:04.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.567+0000 7f14e6cc5700 1 Processor -- start 2024-03-18T19:09:04.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.567+0000 7f14e6cc5700 1 -- start start 2024-03-18T19:09:04.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.568+0000 7f14e6cc5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14e0164000 0x7f14e01663f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:04.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.568+0000 7f14e6cc5700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e015c340 con 0x7f14e00ace80 2024-03-18T19:09:04.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.568+0000 7f14e6cc5700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e015b7a0 con 0x7f14e006a750 2024-03-18T19:09:04.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.568+0000 7f14e6cc5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e015fc90 con 0x7f14e0164000 2024-03-18T19:09:04.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.568+0000 7f14dffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f14e00ace80 0x7f14e00ad310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43512/0 (socket says 172.21.15.1:43512) 2024-03-18T19:09:04.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.568+0000 7f14dffff700 1 -- 172.21.15.1:0/4018760845 learned_addr learned my addr 172.21.15.1:0/4018760845 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:04.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.568+0000 7f14e5262700 1 --2- 172.21.15.1:0/4018760845 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14e0164000 0x7f14e01663f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:04.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.568+0000 7f14df7fe700 1 -- 172.21.15.1:0/4018760845 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368315746 0 0) 0x7f14e015c340 con 0x7f14e00ace80 2024-03-18T19:09:04.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.568+0000 7f14df7fe700 1 -- 172.21.15.1:0/4018760845 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d00035b0 con 0x7f14e00ace80 2024-03-18T19:09:04.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.568+0000 7f14df7fe700 1 -- 172.21.15.1:0/4018760845 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031546581 0 0) 0x7f14e015b7a0 con 0x7f14e006a750 2024-03-18T19:09:04.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.568+0000 7f14df7fe700 1 -- 172.21.15.1:0/4018760845 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14e015c340 con 0x7f14e006a750 2024-03-18T19:09:04.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.568+0000 7f14e5262700 1 -- 172.21.15.1:0/4018760845 >> v1:172.21.15.165:6789/0 conn(0x7f14e006a750 legacy=0x7f14e006ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:04.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.568+0000 7f14e5262700 1 -- 172.21.15.1:0/4018760845 >> v1:172.21.15.1:6789/0 conn(0x7f14e00ace80 legacy=0x7f14e00ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:04.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.568+0000 7f14e5262700 1 -- 172.21.15.1:0/4018760845 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e015ba10 con 0x7f14e0164000 2024-03-18T19:09:04.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.569+0000 7f14e5262700 1 --2- 172.21.15.1:0/4018760845 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14e0164000 0x7f14e01663f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009dc0 tx=0x7f14d800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c138fab802ea54d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:04.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.569+0000 7f14df7fe700 1 -- 172.21.15.1:0/4018760845 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d800e070 con 0x7f14e0164000 2024-03-18T19:09:04.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.569+0000 7f14df7fe700 1 -- 172.21.15.1:0/4018760845 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d800bb20 con 0x7f14e0164000 2024-03-18T19:09:04.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.569+0000 7f14df7fe700 1 -- 172.21.15.1:0/4018760845 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d8012430 con 0x7f14e0164000 2024-03-18T19:09:04.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.569+0000 7f14e6cc5700 1 -- 172.21.15.1:0/4018760845 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14e0164000 msgr2=0x7f14e01663f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:04.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.569+0000 7f14e6cc5700 1 --2- 172.21.15.1:0/4018760845 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14e0164000 0x7f14e01663f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009dc0 tx=0x7f14d800b330 comp rx=0 tx=0).stop 2024-03-18T19:09:04.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.569+0000 7f14e6cc5700 1 -- 172.21.15.1:0/4018760845 shutdown_connections 2024-03-18T19:09:04.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.569+0000 7f14e6cc5700 1 --2- 172.21.15.1:0/4018760845 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14e0164000 0x7f14e01663f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.569+0000 7f14e6cc5700 1 -- 172.21.15.1:0/4018760845 >> 172.21.15.1:0/4018760845 conn(0x7f14e00cc9b0 msgr2=0x7f14e00ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:04.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.569+0000 7f14e6cc5700 1 -- 172.21.15.1:0/4018760845 shutdown_connections 2024-03-18T19:09:04.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.569+0000 7f14e6cc5700 1 -- 172.21.15.1:0/4018760845 wait complete. 2024-03-18T19:09:04.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.571+0000 7f14e6cc5700 1 Processor -- start 2024-03-18T19:09:04.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.571+0000 7f14e6cc5700 1 -- start start 2024-03-18T19:09:04.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14e6cc5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14e006a750 0x7f14e00c5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:04.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14e6cc5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14e00ace80 0x7f14e0157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:04.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14e6cc5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14e0164000 0x7f14e00b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:04.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14e6cc5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e00c5290 con 0x7f14e0164000 2024-03-18T19:09:04.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14e6cc5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e00c4680 con 0x7f14e00ace80 2024-03-18T19:09:04.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14e6cc5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e00c4960 con 0x7f14e006a750 2024-03-18T19:09:04.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14e5262700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14e0164000 0x7f14e00b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:04.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14e5262700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14e0164000 0x7f14e00b3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37312/0 (socket says 172.21.15.1:37312) 2024-03-18T19:09:04.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14e5262700 1 -- 172.21.15.1:0/3291092728 learned_addr learned my addr 172.21.15.1:0/3291092728 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:04.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14dffff700 1 --2- 172.21.15.1:0/3291092728 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14e00ace80 0x7f14e0157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:04.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14e4a61700 1 --2- 172.21.15.1:0/3291092728 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14e006a750 0x7f14e00c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:04.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14dffff700 1 -- 172.21.15.1:0/3291092728 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14e006a750 msgr2=0x7f14e00c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:04.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14dffff700 1 --2- 172.21.15.1:0/3291092728 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14e006a750 0x7f14e00c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14dffff700 1 -- 172.21.15.1:0/3291092728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14e0164000 msgr2=0x7f14e00b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:04.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14dffff700 1 --2- 172.21.15.1:0/3291092728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14e0164000 0x7f14e00b3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14dffff700 1 -- 172.21.15.1:0/3291092728 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d8009a70 con 0x7f14e00ace80 2024-03-18T19:09:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14e4a61700 1 --2- 172.21.15.1:0/3291092728 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14e006a750 0x7f14e00c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14dffff700 1 --2- 172.21.15.1:0/3291092728 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14e00ace80 0x7f14e0157990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f14cc00c4d0 tx=0x7f14cc00c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14dd7fa700 1 -- 172.21.15.1:0/3291092728 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc005d70 con 0x7f14e00ace80 2024-03-18T19:09:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14dd7fa700 1 -- 172.21.15.1:0/3291092728 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14cc0070d0 con 0x7f14e00ace80 2024-03-18T19:09:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.572+0000 7f14e6cc5700 1 -- 172.21.15.1:0/3291092728 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e00c4c40 con 0x7f14e00ace80 2024-03-18T19:09:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.573+0000 7f14dd7fa700 1 -- 172.21.15.1:0/3291092728 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc007480 con 0x7f14e00ace80 2024-03-18T19:09:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.573+0000 7f14e6cc5700 1 -- 172.21.15.1:0/3291092728 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e00c4310 con 0x7f14e00ace80 2024-03-18T19:09:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.573+0000 7f14e6cc5700 1 -- 172.21.15.1:0/3291092728 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e00c5440 con 0x7f14e00ace80 2024-03-18T19:09:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.574+0000 7f14dd7fa700 1 -- 172.21.15.1:0/3291092728 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 68) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f14cc0075e0 con 0x7f14e00ace80 2024-03-18T19:09:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.574+0000 7f14dd7fa700 1 --2- 172.21.15.1:0/3291092728 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14c406af00 0x7f14c406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.574+0000 7f14e4a61700 1 -- 172.21.15.1:0/3291092728 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14c406af00 msgr2=0x7f14c406d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.574+0000 7f14dd7fa700 1 -- 172.21.15.1:0/3291092728 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f14cc088e00 con 0x7f14e00ace80 2024-03-18T19:09:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.574+0000 7f14e4a61700 1 --2- 172.21.15.1:0/3291092728 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14c406af00 0x7f14c406d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:04.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.579+0000 7f14dd7fa700 1 -- 172.21.15.1:0/3291092728 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f14e00c5440 con 0x7f14e00ace80 2024-03-18T19:09:04.709 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:04.713+0000 7fcaa747b240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:09:04.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.775+0000 7f14e4a61700 1 -- 172.21.15.1:0/3291092728 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14c406af00 msgr2=0x7f14c406d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:04.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.775+0000 7f14e4a61700 1 --2- 172.21.15.1:0/3291092728 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14c406af00 0x7f14c406d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.783+0000 7f14e6cc5700 1 -- 172.21.15.1:0/3291092728 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f14e00ae4b0 con 0x7f14e00ace80 2024-03-18T19:09:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.784+0000 7f14dd7fa700 1 -- 172.21.15.1:0/3291092728 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v134) v1 ==== 100+0+11 (secure 0 0 0) 0x7f14cc0170a0 con 0x7f14e00ace80 2024-03-18T19:09:04.781 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T19:09:04.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.786+0000 7f14e6cc5700 1 -- 172.21.15.1:0/3291092728 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14c406af00 msgr2=0x7f14c406d3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:04.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.786+0000 7f14e6cc5700 1 --2- 172.21.15.1:0/3291092728 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14c406af00 0x7f14c406d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.786+0000 7f14e6cc5700 1 -- 172.21.15.1:0/3291092728 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14e00ace80 msgr2=0x7f14e0157990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:04.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.786+0000 7f14e6cc5700 1 --2- 172.21.15.1:0/3291092728 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14e00ace80 0x7f14e0157990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f14cc00c4d0 tx=0x7f14cc00c7e0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.786+0000 7f14e6cc5700 1 -- 172.21.15.1:0/3291092728 shutdown_connections 2024-03-18T19:09:04.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.786+0000 7f14e6cc5700 1 --2- 172.21.15.1:0/3291092728 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14c406af00 0x7f14c406d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.786+0000 7f14e6cc5700 1 --2- 172.21.15.1:0/3291092728 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14e006a750 0x7f14e00c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.786+0000 7f14e6cc5700 1 --2- 172.21.15.1:0/3291092728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14e0164000 0x7f14e00b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.786+0000 7f14e6cc5700 1 --2- 172.21.15.1:0/3291092728 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14e00ace80 0x7f14e0157990 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.786+0000 7f14e6cc5700 1 -- 172.21.15.1:0/3291092728 >> 172.21.15.1:0/3291092728 conn(0x7f14e00cc9b0 msgr2=0x7f14e0071e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:04.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.786+0000 7f14e6cc5700 1 -- 172.21.15.1:0/3291092728 shutdown_connections 2024-03-18T19:09:04.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.787+0000 7f14e6cc5700 1 -- 172.21.15.1:0/3291092728 wait complete. 2024-03-18T19:09:04.797 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T19:09:04.845 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:04.848+0000 7f868eda6240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:09:04.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.968+0000 7f2e0980b700 1 Processor -- start 2024-03-18T19:09:04.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.968+0000 7f2e0980b700 1 -- start start 2024-03-18T19:09:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.968+0000 7f2e0980b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e0406cc60 0x7f2e0406ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.969+0000 7f2e0980b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e0406d040 con 0x7f2e04099a40 2024-03-18T19:09:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.969+0000 7f2e0980b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e04162120 con 0x7f2e04099ed0 2024-03-18T19:09:04.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.969+0000 7f2e0980b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e0415ed60 con 0x7f2e0406cc60 2024-03-18T19:09:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.969+0000 7f2e02ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e0406cc60 0x7f2e0406ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.969+0000 7f2e02ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e0406cc60 0x7f2e0406ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37318/0 (socket says 172.21.15.1:37318) 2024-03-18T19:09:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.969+0000 7f2e027fc700 1 --1- 172.21.15.1:0/552177174 >> v1:172.21.15.1:6789/0 conn(0x7f2e04099a40 0x7f2e04161300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43516/0 (socket says 172.21.15.1:43516) 2024-03-18T19:09:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.969+0000 7f2e02ffd700 1 -- 172.21.15.1:0/552177174 learned_addr learned my addr 172.21.15.1:0/552177174 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.969+0000 7f2e02ffd700 1 -- 172.21.15.1:0/552177174 >> v1:172.21.15.165:6789/0 conn(0x7f2e04099ed0 legacy=0x7f2e04161a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.969+0000 7f2e02ffd700 1 -- 172.21.15.1:0/552177174 >> v1:172.21.15.1:6789/0 conn(0x7f2e04099a40 legacy=0x7f2e04161300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.969+0000 7f2e01ffb700 1 -- 172.21.15.1:0/552177174 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1931869095 0 0) 0x7f2e0406d040 con 0x7f2e04099a40 2024-03-18T19:09:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.969+0000 7f2e02ffd700 1 -- 172.21.15.1:0/552177174 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e040d09e0 con 0x7f2e0406cc60 2024-03-18T19:09:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.969+0000 7f2e02ffd700 1 --2- 172.21.15.1:0/552177174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e0406cc60 0x7f2e0406ba70 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f2df80098f0 tx=0x7f2df8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=165d9939dc77515c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.969+0000 7f2e01ffb700 1 -- 172.21.15.1:0/552177174 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df8004230 con 0x7f2e0406cc60 2024-03-18T19:09:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.969+0000 7f2e01ffb700 1 -- 172.21.15.1:0/552177174 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df8004390 con 0x7f2e0406cc60 2024-03-18T19:09:04.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.969+0000 7f2e01ffb700 1 -- 172.21.15.1:0/552177174 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df80036a0 con 0x7f2e0406cc60 2024-03-18T19:09:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.970+0000 7f2e0980b700 1 -- 172.21.15.1:0/552177174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e0406cc60 msgr2=0x7f2e0406ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.970+0000 7f2e0980b700 1 --2- 172.21.15.1:0/552177174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e0406cc60 0x7f2e0406ba70 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f2df80098f0 tx=0x7f2df8009c00 comp rx=0 tx=0).stop 2024-03-18T19:09:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.970+0000 7f2e0980b700 1 -- 172.21.15.1:0/552177174 shutdown_connections 2024-03-18T19:09:04.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.970+0000 7f2e0980b700 1 --2- 172.21.15.1:0/552177174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e0406cc60 0x7f2e0406ba70 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.970+0000 7f2e0980b700 1 -- 172.21.15.1:0/552177174 >> 172.21.15.1:0/552177174 conn(0x7f2e040b2b90 msgr2=0x7f2e040b2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.970+0000 7f2e0980b700 1 -- 172.21.15.1:0/552177174 shutdown_connections 2024-03-18T19:09:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.970+0000 7f2e0980b700 1 -- 172.21.15.1:0/552177174 wait complete. 2024-03-18T19:09:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.971+0000 7f2e0980b700 1 Processor -- start 2024-03-18T19:09:04.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.971+0000 7f2e0980b700 1 -- start start 2024-03-18T19:09:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.972+0000 7f2e0980b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e0406cc60 0x7f2e041e7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.972+0000 7f2e0980b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e04099a40 0x7f2e040b66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.972+0000 7f2e0980b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e04099ed0 0x7f2e0414fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.972+0000 7f2e0980b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e040b5620 con 0x7f2e04099a40 2024-03-18T19:09:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.972+0000 7f2e0980b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e040b5900 con 0x7f2e0406cc60 2024-03-18T19:09:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.972+0000 7f2e0980b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e040b5be0 con 0x7f2e04099ed0 2024-03-18T19:09:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.972+0000 7f2e027fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e04099a40 0x7f2e040b66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.972+0000 7f2e027fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e04099a40 0x7f2e040b66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37320/0 (socket says 172.21.15.1:37320) 2024-03-18T19:09:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.972+0000 7f2e02ffd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e0406cc60 0x7f2e041e7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.972+0000 7f2e027fc700 1 -- 172.21.15.1:0/3679514809 learned_addr learned my addr 172.21.15.1:0/3679514809 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.972+0000 7f2e037fe700 1 --2- 172.21.15.1:0/3679514809 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e04099ed0 0x7f2e0414fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:04.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.973+0000 7f2e027fc700 1 -- 172.21.15.1:0/3679514809 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e04099ed0 msgr2=0x7f2e0414fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.973+0000 7f2e027fc700 1 --2- 172.21.15.1:0/3679514809 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e04099ed0 0x7f2e0414fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.973+0000 7f2e027fc700 1 -- 172.21.15.1:0/3679514809 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e0406cc60 msgr2=0x7f2e041e7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.973+0000 7f2e027fc700 1 --2- 172.21.15.1:0/3679514809 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e0406cc60 0x7f2e041e7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.973+0000 7f2e027fc700 1 -- 172.21.15.1:0/3679514809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df8009550 con 0x7f2e04099a40 2024-03-18T19:09:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.973+0000 7f2e027fc700 1 --2- 172.21.15.1:0/3679514809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e04099a40 0x7f2e040b66c0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f2df40011e0 tx=0x7f2df400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.973+0000 7f2debfff700 1 -- 172.21.15.1:0/3679514809 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df4004c80 con 0x7f2e04099a40 2024-03-18T19:09:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.973+0000 7f2debfff700 1 -- 172.21.15.1:0/3679514809 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df4004e00 con 0x7f2e04099a40 2024-03-18T19:09:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.973+0000 7f2e0980b700 1 -- 172.21.15.1:0/3679514809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e040b5ec0 con 0x7f2e04099a40 2024-03-18T19:09:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.973+0000 7f2debfff700 1 -- 172.21.15.1:0/3679514809 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df4007310 con 0x7f2e04099a40 2024-03-18T19:09:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.973+0000 7f2e0980b700 1 -- 172.21.15.1:0/3679514809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e04150990 con 0x7f2e04099a40 2024-03-18T19:09:04.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.974+0000 7f2e0980b700 1 -- 172.21.15.1:0/3679514809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e040d0ea0 con 0x7f2e04099a40 2024-03-18T19:09:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.974+0000 7f2debfff700 1 -- 172.21.15.1:0/3679514809 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 68) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f2df4007470 con 0x7f2e04099a40 2024-03-18T19:09:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.975+0000 7f2debfff700 1 --2- 172.21.15.1:0/3679514809 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2df006aed0 0x7f2df006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.975+0000 7f2e02ffd700 1 -- 172.21.15.1:0/3679514809 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2df006aed0 msgr2=0x7f2df006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.975+0000 7f2debfff700 1 -- 172.21.15.1:0/3679514809 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f2df40895f0 con 0x7f2e04099a40 2024-03-18T19:09:04.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.975+0000 7f2e02ffd700 1 --2- 172.21.15.1:0/3679514809 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2df006aed0 0x7f2df006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:04.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:04.979+0000 7f2debfff700 1 -- 172.21.15.1:0/3679514809 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2df4054260 con 0x7f2e04099a40 2024-03-18T19:09:05.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.176+0000 7f2e02ffd700 1 -- 172.21.15.1:0/3679514809 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2df006aed0 msgr2=0x7f2df006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:05.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.176+0000 7f2e02ffd700 1 --2- 172.21.15.1:0/3679514809 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2df006aed0 0x7f2df006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:05.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.218+0000 7f2e0980b700 1 -- 172.21.15.1:0/3679514809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2e040b63e0 con 0x7f2e04099a40 2024-03-18T19:09:05.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.219+0000 7f2debfff700 1 -- 172.21.15.1:0/3679514809 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v134) v1 ==== 80+0+451 (secure 0 0 0) 0x7f2e040b63e0 con 0x7f2e04099a40 2024-03-18T19:09:05.216 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3989116924 2024-03-19T19:08:34.815944+0000 2024-03-18T19:09:05.216 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6832/76998 2024-03-19T19:08:34.815944+0000 2024-03-18T19:09:05.216 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3859645093 2024-03-19T19:08:34.815944+0000 2024-03-18T19:09:05.216 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6833/76998 2024-03-19T19:08:34.815944+0000 2024-03-18T19:09:05.216 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/748689501 2024-03-19T19:08:34.815944+0000 2024-03-18T19:09:05.216 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1964926516 2024-03-19T19:08:34.815944+0000 2024-03-18T19:09:05.216 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2849309461 2024-03-19T19:08:34.815944+0000 2024-03-18T19:09:05.216 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3265489674 2024-03-19T19:08:34.815944+0000 2024-03-18T19:09:05.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.221+0000 7f2e0980b700 1 -- 172.21.15.1:0/3679514809 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2df006aed0 msgr2=0x7f2df006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:05.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.221+0000 7f2e0980b700 1 --2- 172.21.15.1:0/3679514809 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2df006aed0 0x7f2df006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:05.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.221+0000 7f2e0980b700 1 -- 172.21.15.1:0/3679514809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e04099a40 msgr2=0x7f2e040b66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:05.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.221+0000 7f2e0980b700 1 --2- 172.21.15.1:0/3679514809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e04099a40 0x7f2e040b66c0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f2df40011e0 tx=0x7f2df400ca30 comp rx=0 tx=0).stop 2024-03-18T19:09:05.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.221+0000 7f2e0980b700 1 -- 172.21.15.1:0/3679514809 shutdown_connections 2024-03-18T19:09:05.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.221+0000 7f2e0980b700 1 --2- 172.21.15.1:0/3679514809 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2df006aed0 0x7f2df006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:05.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.221+0000 7f2e0980b700 1 --2- 172.21.15.1:0/3679514809 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e04099ed0 0x7f2e0414fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:05.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.221+0000 7f2e0980b700 1 --2- 172.21.15.1:0/3679514809 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e0406cc60 0x7f2e041e7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:05.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.221+0000 7f2e0980b700 1 --2- 172.21.15.1:0/3679514809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e04099a40 0x7f2e040b66c0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:05.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.221+0000 7f2e0980b700 1 -- 172.21.15.1:0/3679514809 >> 172.21.15.1:0/3679514809 conn(0x7f2e040b2b90 msgr2=0x7f2e0406f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:05.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.222+0000 7f2e0980b700 1 -- 172.21.15.1:0/3679514809 shutdown_connections 2024-03-18T19:09:05.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.222+0000 7f2e0980b700 1 -- 172.21.15.1:0/3679514809 wait complete. 2024-03-18T19:09:05.220 INFO:teuthology.orchestra.run.smithi001.stderr:listed 8 entries 2024-03-18T19:09:05.231 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-18T19:09:05.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:05.231 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T19:09:05.290 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:05.294+0000 7fcaa747b240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:09:05.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[124790]: fuse finished with error 0 and tester_r 0 2024-03-18T19:09:05.307 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T19:09:05.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:05.307 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T19:09:05.361 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:05.365+0000 7fcaa747b240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:09:05.436 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:09:05.436 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:09:05.454 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:09:05.455 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T19:09:05.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:05.455 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-18T19:09:05.490 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:05.494+0000 7fcaa747b240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:09:05.496 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:05.499+0000 7f868eda6240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:09:05.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:09:05.506 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-18T19:09:05.507 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:09:05.507 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:09:05.526 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:09:05.526 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T19:09:05.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:05.526 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-18T19:09:05.576 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:05.579+0000 7f868eda6240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:09:05.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:09:05.584 INFO:teuthology.orchestra.run.smithi001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-18T19:09:05.584 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:09:05.584 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:09:05.603 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:09:05.603 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T19:09:05.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:05.603 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-18T19:09:05.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:09:05.660 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-18T19:09:05.661 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:09:05.673 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:05.677+0000 7fcaa747b240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:09:05.715 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:05.719+0000 7f868eda6240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:09:05.745 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:05.749+0000 7fcaa747b240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:09:05.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.790+0000 7f1908104700 1 Processor -- start 2024-03-18T19:09:05.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.790+0000 7f1908104700 1 -- start start 2024-03-18T19:09:05.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.791+0000 7f1908104700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f190015b4d0 0x7f190009a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:05.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.792+0000 7f1908104700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19000cf750 0x7f1900099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:05.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.792+0000 7f1908104700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1900099f80 0x7f19000cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:05.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.792+0000 7f1908104700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f190009a410 con 0x7f190015b4d0 2024-03-18T19:09:05.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.792+0000 7f1908104700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f190006f070 con 0x7f1900099f80 2024-03-18T19:09:05.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.792+0000 7f1908104700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19000d0260 con 0x7f19000cf750 2024-03-18T19:09:05.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.792+0000 7f1905ea0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f190015b4d0 0x7f190009a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:05.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.792+0000 7f1905ea0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f190015b4d0 0x7f190009a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37328/0 (socket says 172.21.15.1:37328) 2024-03-18T19:09:05.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.792+0000 7f1905ea0700 1 -- 172.21.15.1:0/1340816102 learned_addr learned my addr 172.21.15.1:0/1340816102 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:05.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.792+0000 7f190569f700 1 --2- 172.21.15.1:0/1340816102 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19000cf750 0x7f1900099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:05.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.792+0000 7f19066a1700 1 --2- 172.21.15.1:0/1340816102 >> v2:172.21.15.165:3300/0 conn(0x7f1900099f80 0x7f19000cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:05.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.792+0000 7f1905ea0700 1 -- 172.21.15.1:0/1340816102 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19000cf750 msgr2=0x7f1900099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:05.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.792+0000 7f1905ea0700 1 --2- 172.21.15.1:0/1340816102 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19000cf750 0x7f1900099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:05.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.792+0000 7f1905ea0700 1 -- 172.21.15.1:0/1340816102 >> v2:172.21.15.165:3300/0 conn(0x7f1900099f80 msgr2=0x7f19000cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:05.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.792+0000 7f1905ea0700 1 --2- 172.21.15.1:0/1340816102 >> v2:172.21.15.165:3300/0 conn(0x7f1900099f80 0x7f19000cbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:05.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.792+0000 7f1905ea0700 1 -- 172.21.15.1:0/1340816102 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19000cff80 con 0x7f190015b4d0 2024-03-18T19:09:05.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.792+0000 7f1905ea0700 1 --2- 172.21.15.1:0/1340816102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f190015b4d0 0x7f190009a650 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f19000cb7a0 tx=0x7f18fc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63d935bb0387b50 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:05.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.792+0000 7f1904e9e700 1 -- 172.21.15.1:0/1340816102 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc011830 con 0x7f190015b4d0 2024-03-18T19:09:05.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.792+0000 7f1904e9e700 1 -- 172.21.15.1:0/1340816102 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18fc011990 con 0x7f190015b4d0 2024-03-18T19:09:05.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.792+0000 7f1904e9e700 1 -- 172.21.15.1:0/1340816102 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc011d20 con 0x7f190015b4d0 2024-03-18T19:09:05.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.793+0000 7f1908104700 1 -- 172.21.15.1:0/1340816102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f190015b4d0 msgr2=0x7f190009a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.793+0000 7f1908104700 1 --2- 172.21.15.1:0/1340816102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f190015b4d0 0x7f190009a650 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f19000cb7a0 tx=0x7f18fc009bd0 comp rx=0 tx=0).stop 2024-03-18T19:09:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.793+0000 7f1908104700 1 -- 172.21.15.1:0/1340816102 shutdown_connections 2024-03-18T19:09:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.793+0000 7f1908104700 1 --2- 172.21.15.1:0/1340816102 >> v2:172.21.15.165:3300/0 conn(0x7f1900099f80 0x7f19000cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.793+0000 7f1908104700 1 --2- 172.21.15.1:0/1340816102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f190015b4d0 0x7f190009a650 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.793+0000 7f1908104700 1 --2- 172.21.15.1:0/1340816102 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19000cf750 0x7f1900099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.793+0000 7f1908104700 1 -- 172.21.15.1:0/1340816102 >> 172.21.15.1:0/1340816102 conn(0x7f19000b2cb0 msgr2=0x7f19000b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.793+0000 7f1908104700 1 -- 172.21.15.1:0/1340816102 shutdown_connections 2024-03-18T19:09:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.793+0000 7f1908104700 1 -- 172.21.15.1:0/1340816102 wait complete. 2024-03-18T19:09:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.795+0000 7f1908104700 1 Processor -- start 2024-03-18T19:09:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.795+0000 7f1908104700 1 -- start start 2024-03-18T19:09:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.796+0000 7f1908104700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1900099f80 0x7f19001567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.796+0000 7f1908104700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19000cf750 0x7f19000aee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.796+0000 7f1908104700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f190015b4d0 0x7f19000af390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.796+0000 7f1908104700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19000c74d0 con 0x7f1900099f80 2024-03-18T19:09:05.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.796+0000 7f1908104700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19000afd50 con 0x7f19000cf750 2024-03-18T19:09:05.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.796+0000 7f1908104700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19000c6850 con 0x7f190015b4d0 2024-03-18T19:09:05.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.796+0000 7f1905ea0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1900099f80 0x7f19001567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:05.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.796+0000 7f1905ea0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1900099f80 0x7f19001567b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37336/0 (socket says 172.21.15.1:37336) 2024-03-18T19:09:05.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.796+0000 7f1905ea0700 1 -- 172.21.15.1:0/906063521 learned_addr learned my addr 172.21.15.1:0/906063521 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:05.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.796+0000 7f190569f700 1 --2- 172.21.15.1:0/906063521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19000cf750 0x7f19000aee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:05.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.796+0000 7f19066a1700 1 --2- 172.21.15.1:0/906063521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f190015b4d0 0x7f19000af390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:05.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.796+0000 7f1905ea0700 1 -- 172.21.15.1:0/906063521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f190015b4d0 msgr2=0x7f19000af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:05.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.796+0000 7f1905ea0700 1 --2- 172.21.15.1:0/906063521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f190015b4d0 0x7f19000af390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:05.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.796+0000 7f1905ea0700 1 -- 172.21.15.1:0/906063521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19000cf750 msgr2=0x7f19000aee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:05.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.796+0000 7f1905ea0700 1 --2- 172.21.15.1:0/906063521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19000cf750 0x7f19000aee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:05.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.796+0000 7f1905ea0700 1 -- 172.21.15.1:0/906063521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18fc009550 con 0x7f1900099f80 2024-03-18T19:09:05.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.797+0000 7f1905ea0700 1 --2- 172.21.15.1:0/906063521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1900099f80 0x7f19001567b0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f18fc014040 tx=0x7f18fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:05.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.797+0000 7f18f67fc700 1 -- 172.21.15.1:0/906063521 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc004510 con 0x7f1900099f80 2024-03-18T19:09:05.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.797+0000 7f18f67fc700 1 -- 172.21.15.1:0/906063521 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18fc01e070 con 0x7f1900099f80 2024-03-18T19:09:05.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.797+0000 7f1908104700 1 -- 172.21.15.1:0/906063521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19000c6b30 con 0x7f1900099f80 2024-03-18T19:09:05.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.797+0000 7f18f67fc700 1 -- 172.21.15.1:0/906063521 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc0038e0 con 0x7f1900099f80 2024-03-18T19:09:05.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.797+0000 7f1908104700 1 -- 172.21.15.1:0/906063521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19000c70f0 con 0x7f1900099f80 2024-03-18T19:09:05.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.798+0000 7f1908104700 1 -- 172.21.15.1:0/906063521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19000c7280 con 0x7f1900099f80 2024-03-18T19:09:05.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.798+0000 7f18f67fc700 1 -- 172.21.15.1:0/906063521 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 68) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f18fc003a40 con 0x7f1900099f80 2024-03-18T19:09:05.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.799+0000 7f18f67fc700 1 --2- 172.21.15.1:0/906063521 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f18e406aed0 0x7f18e406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:05.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.799+0000 7f190569f700 1 -- 172.21.15.1:0/906063521 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f18e406aed0 msgr2=0x7f18e406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:05.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.799+0000 7f18f67fc700 1 -- 172.21.15.1:0/906063521 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f18fc020070 con 0x7f1900099f80 2024-03-18T19:09:05.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.799+0000 7f190569f700 1 --2- 172.21.15.1:0/906063521 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f18e406aed0 0x7f18e406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:05.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:05.803+0000 7f18f67fc700 1 -- 172.21.15.1:0/906063521 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f18fc055910 con 0x7f1900099f80 2024-03-18T19:09:05.822 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:05.826+0000 7fcaa747b240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:09:05.898 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:05.902+0000 7fcaa747b240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:09:05.906 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:05.909+0000 7f868eda6240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:09:05.974 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:05.978+0000 7fcaa747b240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:09:05.983 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:05.986+0000 7f868eda6240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:09:05.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.000+0000 7f190569f700 1 -- 172.21.15.1:0/906063521 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f18e406aed0 msgr2=0x7f18e406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:05.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.000+0000 7f190569f700 1 --2- 172.21.15.1:0/906063521 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f18e406aed0 0x7f18e406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:06.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.056+0000 7f1908104700 1 -- 172.21.15.1:0/906063521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1900068240 con 0x7f1900099f80 2024-03-18T19:09:06.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.057+0000 7f18f67fc700 1 -- 172.21.15.1:0/906063521 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4730 (secure 0 0 0) 0x7f18fc015020 con 0x7f1900099f80 2024-03-18T19:09:06.054 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:06.054 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6800,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3354770356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3354770356},{"type":"v1","addr":"172.21.15.1:6837","nonce":3354770356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":6809,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/695588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":695588699},{"type":"v1","addr":"172.21.15.165:6834","nonce":695588699}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7660,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3186346302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3186346302},{"type":"v1","addr":"172.21.15.1:6836","nonce":3186346302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:08:19.581625+0000","modified":"2024-03-18T19:08:22.589436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7675},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7675":{"gid":7675,"name":"d","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/450810178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":450810178},{"type":"v1","addr":"172.21.15.165:6835","nonce":450810178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-18T19:09:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.059+0000 7f1908104700 1 -- 172.21.15.1:0/906063521 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f18e406aed0 msgr2=0x7f18e406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.059+0000 7f1908104700 1 --2- 172.21.15.1:0/906063521 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f18e406aed0 0x7f18e406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.059+0000 7f1908104700 1 -- 172.21.15.1:0/906063521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1900099f80 msgr2=0x7f19001567b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:06.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.059+0000 7f1908104700 1 --2- 172.21.15.1:0/906063521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1900099f80 0x7f19001567b0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f18fc014040 tx=0x7f18fc004750 comp rx=0 tx=0).stop 2024-03-18T19:09:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.059+0000 7f1908104700 1 -- 172.21.15.1:0/906063521 shutdown_connections 2024-03-18T19:09:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.059+0000 7f1908104700 1 --2- 172.21.15.1:0/906063521 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f18e406aed0 0x7f18e406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.059+0000 7f1908104700 1 --2- 172.21.15.1:0/906063521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f190015b4d0 0x7f19000af390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.059+0000 7f1908104700 1 --2- 172.21.15.1:0/906063521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1900099f80 0x7f19001567b0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.059+0000 7f1908104700 1 --2- 172.21.15.1:0/906063521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19000cf750 0x7f19000aee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.059+0000 7f1908104700 1 -- 172.21.15.1:0/906063521 >> 172.21.15.1:0/906063521 conn(0x7f19000b2cb0 msgr2=0x7f190006f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.059+0000 7f1908104700 1 -- 172.21.15.1:0/906063521 shutdown_connections 2024-03-18T19:09:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.059+0000 7f1908104700 1 -- 172.21.15.1:0/906063521 wait complete. 2024-03-18T19:09:06.058 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 57 2024-03-18T19:09:06.067 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:06.071+0000 7f868eda6240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:09:06.069 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:09:06.142 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:06.146+0000 7fcaa747b240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:09:06.152 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:06.156+0000 7f868eda6240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:09:06.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.199+0000 7f60978b1700 1 Processor -- start 2024-03-18T19:09:06.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.200+0000 7f60978b1700 1 -- start start 2024-03-18T19:09:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.201+0000 7f60978b1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f609015d500 0x7f60900d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.201+0000 7f60978b1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f60900d1ec0 0x7f60900b83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.201+0000 7f60978b1700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60900cddc0 con 0x7f60900d1300 2024-03-18T19:09:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.201+0000 7f60978b1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60900d3dc0 con 0x7f60900d1ec0 2024-03-18T19:09:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.201+0000 7f60978b1700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60900ce060 con 0x7f609015d500 2024-03-18T19:09:06.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.201+0000 7f6094e4c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f60900d1300 0x7f60900d17b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43524/0 (socket says 172.21.15.1:43524) 2024-03-18T19:09:06.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.201+0000 7f6094e4c700 1 -- 172.21.15.1:0/2026261521 learned_addr learned my addr 172.21.15.1:0/2026261521 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:06.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.201+0000 7f6095e4e700 1 --2- 172.21.15.1:0/2026261521 >> v2:172.21.15.1:3300/0 conn(0x7f60900d1ec0 0x7f60900b83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:06.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.201+0000 7f609564d700 1 --2- 172.21.15.1:0/2026261521 >> v2:172.21.15.165:3300/0 conn(0x7f609015d500 0x7f60900d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:06.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.202+0000 7f6087fff700 1 -- 172.21.15.1:0/2026261521 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195328831 0 0) 0x7f60900cddc0 con 0x7f60900d1300 2024-03-18T19:09:06.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.202+0000 7f6087fff700 1 -- 172.21.15.1:0/2026261521 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f607c0035b0 con 0x7f60900d1300 2024-03-18T19:09:06.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.202+0000 7f6095e4e700 1 -- 172.21.15.1:0/2026261521 >> v2:172.21.15.165:3300/0 conn(0x7f609015d500 msgr2=0x7f60900d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:06.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.202+0000 7f6095e4e700 1 --2- 172.21.15.1:0/2026261521 >> v2:172.21.15.165:3300/0 conn(0x7f609015d500 0x7f60900d0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.202+0000 7f6095e4e700 1 -- 172.21.15.1:0/2026261521 >> v1:172.21.15.1:6789/0 conn(0x7f60900d1300 legacy=0x7f60900d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:06.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.202+0000 7f6095e4e700 1 -- 172.21.15.1:0/2026261521 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60900ce340 con 0x7f60900d1ec0 2024-03-18T19:09:06.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.202+0000 7f6095e4e700 1 --2- 172.21.15.1:0/2026261521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60900d1ec0 0x7f60900b83f0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f609008b740 tx=0x7f608800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5ee9c1030fb365 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:06.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.202+0000 7f6087fff700 1 -- 172.21.15.1:0/2026261521 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608800e070 con 0x7f60900d1ec0 2024-03-18T19:09:06.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.202+0000 7f6087fff700 1 -- 172.21.15.1:0/2026261521 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f608800bb20 con 0x7f60900d1ec0 2024-03-18T19:09:06.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.202+0000 7f6087fff700 1 -- 172.21.15.1:0/2026261521 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608800be90 con 0x7f60900d1ec0 2024-03-18T19:09:06.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.203+0000 7f60978b1700 1 -- 172.21.15.1:0/2026261521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60900d1ec0 msgr2=0x7f60900b83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:06.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.203+0000 7f60978b1700 1 --2- 172.21.15.1:0/2026261521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60900d1ec0 0x7f60900b83f0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f609008b740 tx=0x7f608800b330 comp rx=0 tx=0).stop 2024-03-18T19:09:06.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.203+0000 7f60978b1700 1 -- 172.21.15.1:0/2026261521 shutdown_connections 2024-03-18T19:09:06.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.203+0000 7f60978b1700 1 --2- 172.21.15.1:0/2026261521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60900d1ec0 0x7f60900b83f0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.203+0000 7f60978b1700 1 --2- 172.21.15.1:0/2026261521 >> v2:172.21.15.165:3300/0 conn(0x7f609015d500 0x7f60900d0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.203+0000 7f60978b1700 1 -- 172.21.15.1:0/2026261521 >> 172.21.15.1:0/2026261521 conn(0x7f609009c530 msgr2=0x7f60900d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:06.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.203+0000 7f60978b1700 1 -- 172.21.15.1:0/2026261521 shutdown_connections 2024-03-18T19:09:06.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.203+0000 7f60978b1700 1 -- 172.21.15.1:0/2026261521 wait complete. 2024-03-18T19:09:06.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.205+0000 7f60978b1700 1 Processor -- start 2024-03-18T19:09:06.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.205+0000 7f60978b1700 1 -- start start 2024-03-18T19:09:06.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f60978b1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60900d1300 0x7f60900cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:06.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f60978b1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60900d1ec0 0x7f60900c9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:06.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f60978b1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f609015d500 0x7f60900c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:06.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f60978b1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60900c9c40 con 0x7f609015d500 2024-03-18T19:09:06.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f60978b1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60900cb900 con 0x7f60900d1ec0 2024-03-18T19:09:06.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f60978b1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60900b6c70 con 0x7f60900d1300 2024-03-18T19:09:06.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f6095e4e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f609015d500 0x7f60900c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:06.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f6095e4e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f609015d500 0x7f60900c5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37348/0 (socket says 172.21.15.1:37348) 2024-03-18T19:09:06.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f6095e4e700 1 -- 172.21.15.1:0/1917640958 learned_addr learned my addr 172.21.15.1:0/1917640958 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:06.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f6094e4c700 1 --2- 172.21.15.1:0/1917640958 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60900d1ec0 0x7f60900c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:06.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f609564d700 1 --2- 172.21.15.1:0/1917640958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60900d1300 0x7f60900cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:06.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f6095e4e700 1 -- 172.21.15.1:0/1917640958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60900d1300 msgr2=0x7f60900cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:06.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f6095e4e700 1 --2- 172.21.15.1:0/1917640958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60900d1300 0x7f60900cacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f6095e4e700 1 -- 172.21.15.1:0/1917640958 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60900d1ec0 msgr2=0x7f60900c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:06.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f6095e4e700 1 --2- 172.21.15.1:0/1917640958 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60900d1ec0 0x7f60900c9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f6095e4e700 1 -- 172.21.15.1:0/1917640958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6088009a70 con 0x7f609015d500 2024-03-18T19:09:06.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f609564d700 1 --2- 172.21.15.1:0/1917640958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60900d1300 0x7f60900cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:06.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f6095e4e700 1 --2- 172.21.15.1:0/1917640958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f609015d500 0x7f60900c5cb0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f6088014040 tx=0x7f6088004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:06.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f6085ffb700 1 -- 172.21.15.1:0/1917640958 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60880048b0 con 0x7f609015d500 2024-03-18T19:09:06.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f6085ffb700 1 -- 172.21.15.1:0/1917640958 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6088020070 con 0x7f609015d500 2024-03-18T19:09:06.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f60978b1700 1 -- 172.21.15.1:0/1917640958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60900cb340 con 0x7f609015d500 2024-03-18T19:09:06.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.206+0000 7f6085ffb700 1 -- 172.21.15.1:0/1917640958 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60880276c0 con 0x7f609015d500 2024-03-18T19:09:06.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.207+0000 7f60978b1700 1 -- 172.21.15.1:0/1917640958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60900c6e50 con 0x7f609015d500 2024-03-18T19:09:06.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.208+0000 7f60978b1700 1 -- 172.21.15.1:0/1917640958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60900676a0 con 0x7f609015d500 2024-03-18T19:09:06.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.209+0000 7f6085ffb700 1 -- 172.21.15.1:0/1917640958 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 68) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f6088012430 con 0x7f609015d500 2024-03-18T19:09:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.209+0000 7f6085ffb700 1 --2- 172.21.15.1:0/1917640958 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f607406aed0 0x7f607406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.209+0000 7f609564d700 1 -- 172.21.15.1:0/1917640958 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f607406aed0 msgr2=0x7f607406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.209+0000 7f6085ffb700 1 -- 172.21.15.1:0/1917640958 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f6088022070 con 0x7f609015d500 2024-03-18T19:09:06.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.209+0000 7f609564d700 1 --2- 172.21.15.1:0/1917640958 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f607406aed0 0x7f607406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:06.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.214+0000 7f6085ffb700 1 -- 172.21.15.1:0/1917640958 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f60880561e0 con 0x7f609015d500 2024-03-18T19:09:06.243 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:06.246+0000 7f868eda6240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:09:06.357 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:06.361+0000 7fcaa747b240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:09:06.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.410+0000 7f609564d700 1 -- 172.21.15.1:0/1917640958 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f607406aed0 msgr2=0x7f607406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:06.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.410+0000 7f609564d700 1 --2- 172.21.15.1:0/1917640958 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f607406aed0 0x7f607406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:06.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.422+0000 7f60978b1700 1 -- 172.21.15.1:0/1917640958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60900d0950 con 0x7f609015d500 2024-03-18T19:09:06.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.423+0000 7f6085ffb700 1 -- 172.21.15.1:0/1917640958 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+16682 (secure 0 0 0) 0x7f6088059800 con 0x7f609015d500 2024-03-18T19:09:06.420 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:06.421 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":134,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:08:34.815984+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:08:17.419247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-18T19:08:18.552871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3989116924":"2024-03-19T19:08:34.815944+0000","172.21.15.1:6832/76998":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/3859645093":"2024-03-19T19:08:34.815944+0000","172.21.15.1:6833/76998":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/748689501":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/1964926516":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/2849309461":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/3265489674":"2024-03-19T19:08:34.815944+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:09:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.426+0000 7f60978b1700 1 -- 172.21.15.1:0/1917640958 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f607406aed0 msgr2=0x7f607406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.426+0000 7f60978b1700 1 --2- 172.21.15.1:0/1917640958 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f607406aed0 0x7f607406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.426+0000 7f60978b1700 1 -- 172.21.15.1:0/1917640958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f609015d500 msgr2=0x7f60900c5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.426+0000 7f60978b1700 1 --2- 172.21.15.1:0/1917640958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f609015d500 0x7f60900c5cb0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f6088014040 tx=0x7f6088004ad0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.426+0000 7f60978b1700 1 -- 172.21.15.1:0/1917640958 shutdown_connections 2024-03-18T19:09:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.426+0000 7f60978b1700 1 --2- 172.21.15.1:0/1917640958 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f607406aed0 0x7f607406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.426+0000 7f60978b1700 1 --2- 172.21.15.1:0/1917640958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60900d1300 0x7f60900cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.426+0000 7f60978b1700 1 --2- 172.21.15.1:0/1917640958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f609015d500 0x7f60900c5cb0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.426+0000 7f60978b1700 1 --2- 172.21.15.1:0/1917640958 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60900d1ec0 0x7f60900c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.426+0000 7f60978b1700 1 -- 172.21.15.1:0/1917640958 >> 172.21.15.1:0/1917640958 conn(0x7f609009c530 msgr2=0x7f609015ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:06.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.426+0000 7f60978b1700 1 -- 172.21.15.1:0/1917640958 shutdown_connections 2024-03-18T19:09:06.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.426+0000 7f60978b1700 1 -- 172.21.15.1:0/1917640958 wait complete. 2024-03-18T19:09:06.425 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:06.428+0000 7f868eda6240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:09:06.436 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T19:08:17.419247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '132', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T19:08:18.552871+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '133', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:09:06.436 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:09:06.505 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:06.509+0000 7fcaa747b240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:09:06.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.572+0000 7fade5220700 1 Processor -- start 2024-03-18T19:09:06.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.572+0000 7fade5220700 1 -- start start 2024-03-18T19:09:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.573+0000 7fade5220700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fade015b6c0 0x7fade00b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.573+0000 7fade5220700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fade00cd430 0x7fade00c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.573+0000 7fade5220700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade00d01f0 con 0x7fade00c8600 2024-03-18T19:09:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.573+0000 7fade5220700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade00ccfa0 con 0x7fade00cd430 2024-03-18T19:09:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.573+0000 7fade5220700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade00d1db0 con 0x7fade015b6c0 2024-03-18T19:09:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.573+0000 7faddf59e700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fade00c8600 0x7fade00cedd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43532/0 (socket says 172.21.15.1:43532) 2024-03-18T19:09:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.573+0000 7faddf59e700 1 -- 172.21.15.1:0/3669556937 learned_addr learned my addr 172.21.15.1:0/3669556937 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:06.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.573+0000 7fadded9d700 1 --2- 172.21.15.1:0/3669556937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fade015b6c0 0x7fade00b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:06.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.573+0000 7fadde59c700 1 --2- 172.21.15.1:0/3669556937 >> v2:172.21.15.165:3300/0 conn(0x7fade00cd430 0x7fade00c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:06.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.573+0000 7fadddd9b700 1 -- 172.21.15.1:0/3669556937 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233592681 0 0) 0x7fade00d01f0 con 0x7fade00c8600 2024-03-18T19:09:06.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.573+0000 7fadddd9b700 1 -- 172.21.15.1:0/3669556937 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc80035b0 con 0x7fade00c8600 2024-03-18T19:09:06.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.573+0000 7fadde59c700 1 -- 172.21.15.1:0/3669556937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fade015b6c0 msgr2=0x7fade00b2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:06.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.573+0000 7fadde59c700 1 --2- 172.21.15.1:0/3669556937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fade015b6c0 0x7fade00b2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.573+0000 7fadde59c700 1 -- 172.21.15.1:0/3669556937 >> v1:172.21.15.1:6789/0 conn(0x7fade00c8600 legacy=0x7fade00cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:06.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.573+0000 7fadde59c700 1 -- 172.21.15.1:0/3669556937 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade00cff10 con 0x7fade00cd430 2024-03-18T19:09:06.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.573+0000 7fadded9d700 1 --2- 172.21.15.1:0/3669556937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fade015b6c0 0x7fade00b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:06.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.574+0000 7fadde59c700 1 --2- 172.21.15.1:0/3669556937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fade00cd430 0x7fade00c80c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009840 tx=0x7fadd4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b9f3020f1a52a86 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:06.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.574+0000 7fadddd9b700 1 -- 172.21.15.1:0/3669556937 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd4011830 con 0x7fade00cd430 2024-03-18T19:09:06.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.574+0000 7fadddd9b700 1 -- 172.21.15.1:0/3669556937 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd4011990 con 0x7fade00cd430 2024-03-18T19:09:06.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.574+0000 7fadddd9b700 1 -- 172.21.15.1:0/3669556937 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd4011d20 con 0x7fade00cd430 2024-03-18T19:09:06.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.575+0000 7fade5220700 1 -- 172.21.15.1:0/3669556937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fade00cd430 msgr2=0x7fade00c80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:06.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.575+0000 7fade5220700 1 --2- 172.21.15.1:0/3669556937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fade00cd430 0x7fade00c80c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009840 tx=0x7fadd4009b50 comp rx=0 tx=0).stop 2024-03-18T19:09:06.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.575+0000 7fade5220700 1 -- 172.21.15.1:0/3669556937 shutdown_connections 2024-03-18T19:09:06.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.575+0000 7fade5220700 1 --2- 172.21.15.1:0/3669556937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fade015b6c0 0x7fade00b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.575+0000 7fade5220700 1 --2- 172.21.15.1:0/3669556937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fade00cd430 0x7fade00c80c0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.575+0000 7fade5220700 1 -- 172.21.15.1:0/3669556937 >> 172.21.15.1:0/3669556937 conn(0x7fade00d0ff0 msgr2=0x7fade00d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:06.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.575+0000 7fade5220700 1 -- 172.21.15.1:0/3669556937 shutdown_connections 2024-03-18T19:09:06.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.575+0000 7fade5220700 1 -- 172.21.15.1:0/3669556937 wait complete. 2024-03-18T19:09:06.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.576+0000 7fade5220700 1 Processor -- start 2024-03-18T19:09:06.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.576+0000 7fade5220700 1 -- start start 2024-03-18T19:09:06.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.577+0000 7fade5220700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fade00c8600 0x7fade00c6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:06.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.577+0000 7fade5220700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fade00cd430 0x7fade015a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:06.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.577+0000 7fade5220700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fade015b6c0 0x7fade00c5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:06.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.577+0000 7fade5220700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade00c52f0 con 0x7fade00cd430 2024-03-18T19:09:06.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.577+0000 7fade5220700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade00c6e10 con 0x7fade00c8600 2024-03-18T19:09:06.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.577+0000 7fade5220700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade00c4ca0 con 0x7fade015b6c0 2024-03-18T19:09:06.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.577+0000 7fadde59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fade00cd430 0x7fade015a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:06.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.577+0000 7fadde59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fade00cd430 0x7fade015a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37356/0 (socket says 172.21.15.1:37356) 2024-03-18T19:09:06.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.577+0000 7fadde59c700 1 -- 172.21.15.1:0/2541962654 learned_addr learned my addr 172.21.15.1:0/2541962654 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:06.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.577+0000 7fadded9d700 1 --2- 172.21.15.1:0/2541962654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fade00c8600 0x7fade00c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:06.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.577+0000 7faddf59e700 1 --2- 172.21.15.1:0/2541962654 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fade015b6c0 0x7fade00c5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:06.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.578+0000 7fadde59c700 1 -- 172.21.15.1:0/2541962654 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fade015b6c0 msgr2=0x7fade00c5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:06.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.578+0000 7fadde59c700 1 --2- 172.21.15.1:0/2541962654 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fade015b6c0 0x7fade00c5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.578+0000 7fadde59c700 1 -- 172.21.15.1:0/2541962654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fade00c8600 msgr2=0x7fade00c6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:06.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.578+0000 7fadde59c700 1 --2- 172.21.15.1:0/2541962654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fade00c8600 0x7fade00c6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.578+0000 7fadde59c700 1 -- 172.21.15.1:0/2541962654 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd00094a0 con 0x7fade00cd430 2024-03-18T19:09:06.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.578+0000 7fadded9d700 1 --2- 172.21.15.1:0/2541962654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fade00c8600 0x7fade00c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:06.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.578+0000 7fadde59c700 1 --2- 172.21.15.1:0/2541962654 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fade00cd430 0x7fade015a640 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fadd40090b0 tx=0x7fadd4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:06.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.578+0000 7fadc77fe700 1 -- 172.21.15.1:0/2541962654 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd4004590 con 0x7fade00cd430 2024-03-18T19:09:06.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.578+0000 7fadc77fe700 1 -- 172.21.15.1:0/2541962654 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd401f070 con 0x7fade00cd430 2024-03-18T19:09:06.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.578+0000 7fade5220700 1 -- 172.21.15.1:0/2541962654 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd40094a0 con 0x7fade00cd430 2024-03-18T19:09:06.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.578+0000 7fadc77fe700 1 -- 172.21.15.1:0/2541962654 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd40038e0 con 0x7fade00cd430 2024-03-18T19:09:06.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.578+0000 7fade5220700 1 -- 172.21.15.1:0/2541962654 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade00b6400 con 0x7fade00cd430 2024-03-18T19:09:06.578 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:06.582+0000 7fcaa747b240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:09:06.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.579+0000 7fade5220700 1 -- 172.21.15.1:0/2541962654 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade00c4ae0 con 0x7fade00cd430 2024-03-18T19:09:06.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.580+0000 7fadc77fe700 1 -- 172.21.15.1:0/2541962654 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 68) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fadd4003a40 con 0x7fade00cd430 2024-03-18T19:09:06.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.580+0000 7fadc77fe700 1 --2- 172.21.15.1:0/2541962654 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fadcc06aed0 0x7fadcc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:06.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.580+0000 7fadded9d700 1 -- 172.21.15.1:0/2541962654 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fadcc06aed0 msgr2=0x7fadcc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:06.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.580+0000 7fadc77fe700 1 -- 172.21.15.1:0/2541962654 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fadd4021070 con 0x7fade00cd430 2024-03-18T19:09:06.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.580+0000 7fadded9d700 1 --2- 172.21.15.1:0/2541962654 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fadcc06aed0 0x7fadcc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:06.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.589+0000 7fadc77fe700 1 -- 172.21.15.1:0/2541962654 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fadd4055f20 con 0x7fade00cd430 2024-03-18T19:09:06.655 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:06.658+0000 7f868eda6240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:09:06.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.781+0000 7fadded9d700 1 -- 172.21.15.1:0/2541962654 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fadcc06aed0 msgr2=0x7fadcc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:06.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.781+0000 7fadded9d700 1 --2- 172.21.15.1:0/2541962654 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fadcc06aed0 0x7fadcc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:06.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.792+0000 7fade5220700 1 -- 172.21.15.1:0/2541962654 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fade00676a0 con 0x7fade00cd430 2024-03-18T19:09:06.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.793+0000 7fadc77fe700 1 -- 172.21.15.1:0/2541962654 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v134) v1 ==== 93+0+10 (secure 0 0 0) 0x7fadd4059540 con 0x7fade00cd430 2024-03-18T19:09:06.790 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:09:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.795+0000 7fade5220700 1 -- 172.21.15.1:0/2541962654 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fadcc06aed0 msgr2=0x7fadcc06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.795+0000 7fade5220700 1 --2- 172.21.15.1:0/2541962654 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fadcc06aed0 0x7fadcc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.795+0000 7fade5220700 1 -- 172.21.15.1:0/2541962654 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fade00cd430 msgr2=0x7fade015a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.795+0000 7fade5220700 1 --2- 172.21.15.1:0/2541962654 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fade00cd430 0x7fade015a640 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fadd40090b0 tx=0x7fadd4004750 comp rx=0 tx=0).stop 2024-03-18T19:09:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.795+0000 7fade5220700 1 -- 172.21.15.1:0/2541962654 shutdown_connections 2024-03-18T19:09:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.795+0000 7fade5220700 1 --2- 172.21.15.1:0/2541962654 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fadcc06aed0 0x7fadcc06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.795+0000 7fade5220700 1 --2- 172.21.15.1:0/2541962654 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fade015b6c0 0x7fade00c5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.795+0000 7fade5220700 1 --2- 172.21.15.1:0/2541962654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fade00c8600 0x7fade00c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.795+0000 7fade5220700 1 --2- 172.21.15.1:0/2541962654 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fade00cd430 0x7fade015a640 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.795+0000 7fade5220700 1 -- 172.21.15.1:0/2541962654 >> 172.21.15.1:0/2541962654 conn(0x7fade00d0ff0 msgr2=0x7fade015e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.795+0000 7fade5220700 1 -- 172.21.15.1:0/2541962654 shutdown_connections 2024-03-18T19:09:06.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.795+0000 7fade5220700 1 -- 172.21.15.1:0/2541962654 wait complete. 2024-03-18T19:09:06.805 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T19:09:06.819 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:06.822+0000 7f868eda6240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:09:06.899 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:06.902+0000 7f868eda6240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:09:06.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.937+0000 7eff45204700 1 Processor -- start 2024-03-18T19:09:06.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.937+0000 7eff45204700 1 -- start start 2024-03-18T19:09:06.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.938+0000 7eff45204700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff4015b6c0 0x7eff400b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:06.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.938+0000 7eff45204700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7eff400c87d0 0x7eff400cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:06.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.938+0000 7eff45204700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff400d01f0 con 0x7eff400cd430 2024-03-18T19:09:06.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.938+0000 7eff45204700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff400ccfa0 con 0x7eff400c87d0 2024-03-18T19:09:06.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.938+0000 7eff45204700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff400d1db0 con 0x7eff4015b6c0 2024-03-18T19:09:06.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.938+0000 7eff3f59e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7eff400c87d0 0x7eff400cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:06.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.938+0000 7eff3f59e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7eff400c87d0 0x7eff400cd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37358/0 (socket says 172.21.15.1:37358) 2024-03-18T19:09:06.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.938+0000 7eff3f59e700 1 -- 172.21.15.1:0/2687356987 learned_addr learned my addr 172.21.15.1:0/2687356987 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:06.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.938+0000 7eff3ed9d700 1 --2- 172.21.15.1:0/2687356987 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff4015b6c0 0x7eff400b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:06.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.938+0000 7eff3f59e700 1 -- 172.21.15.1:0/2687356987 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff4015b6c0 msgr2=0x7eff400b2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:06.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.938+0000 7eff3f59e700 1 --2- 172.21.15.1:0/2687356987 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff4015b6c0 0x7eff400b2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.938+0000 7eff3f59e700 1 -- 172.21.15.1:0/2687356987 >> v1:172.21.15.165:6789/0 conn(0x7eff400cd430 legacy=0x7eff400c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:06.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.938+0000 7eff3f59e700 1 -- 172.21.15.1:0/2687356987 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff400cff10 con 0x7eff400c87d0 2024-03-18T19:09:06.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.939+0000 7eff3f59e700 1 --2- 172.21.15.1:0/2687356987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400c87d0 0x7eff400cd9e0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7eff38009e50 tx=0x7eff3800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c78219c4bece5d6f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:06.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.939+0000 7eff3dd9b700 1 -- 172.21.15.1:0/2687356987 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3800e070 con 0x7eff400c87d0 2024-03-18T19:09:06.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.939+0000 7eff3dd9b700 1 -- 172.21.15.1:0/2687356987 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3800b910 con 0x7eff400c87d0 2024-03-18T19:09:06.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.939+0000 7eff3dd9b700 1 -- 172.21.15.1:0/2687356987 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3800bcc0 con 0x7eff400c87d0 2024-03-18T19:09:06.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.940+0000 7eff45204700 1 -- 172.21.15.1:0/2687356987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400c87d0 msgr2=0x7eff400cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:06.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.940+0000 7eff45204700 1 --2- 172.21.15.1:0/2687356987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400c87d0 0x7eff400cd9e0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7eff38009e50 tx=0x7eff3800b330 comp rx=0 tx=0).stop 2024-03-18T19:09:06.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.940+0000 7eff45204700 1 -- 172.21.15.1:0/2687356987 shutdown_connections 2024-03-18T19:09:06.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.940+0000 7eff45204700 1 --2- 172.21.15.1:0/2687356987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400c87d0 0x7eff400cd9e0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.940+0000 7eff45204700 1 --2- 172.21.15.1:0/2687356987 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff4015b6c0 0x7eff400b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.940+0000 7eff45204700 1 -- 172.21.15.1:0/2687356987 >> 172.21.15.1:0/2687356987 conn(0x7eff400d0ff0 msgr2=0x7eff400d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:06.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.940+0000 7eff45204700 1 -- 172.21.15.1:0/2687356987 shutdown_connections 2024-03-18T19:09:06.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.940+0000 7eff45204700 1 -- 172.21.15.1:0/2687356987 wait complete. 2024-03-18T19:09:06.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.942+0000 7eff45204700 1 Processor -- start 2024-03-18T19:09:06.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.942+0000 7eff45204700 1 -- start start 2024-03-18T19:09:06.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff45204700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400c87d0 0x7eff400c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:06.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff45204700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff400cd430 0x7eff400c7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:06.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff45204700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff4015b6c0 0x7eff4014c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:06.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff45204700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4014a210 con 0x7eff400c87d0 2024-03-18T19:09:06.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff45204700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff40149f80 con 0x7eff4015b6c0 2024-03-18T19:09:06.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff45204700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff4014ad20 con 0x7eff400cd430 2024-03-18T19:09:06.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff3ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400c87d0 0x7eff400c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:06.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff3ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400c87d0 0x7eff400c7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37366/0 (socket says 172.21.15.1:37366) 2024-03-18T19:09:06.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff3ed9d700 1 -- 172.21.15.1:0/1519725505 learned_addr learned my addr 172.21.15.1:0/1519725505 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:06.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff3f59e700 1 --2- 172.21.15.1:0/1519725505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff4015b6c0 0x7eff4014c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff3e59c700 1 --2- 172.21.15.1:0/1519725505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff400cd430 0x7eff400c7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff3ed9d700 1 -- 172.21.15.1:0/1519725505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff400cd430 msgr2=0x7eff400c7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff3ed9d700 1 --2- 172.21.15.1:0/1519725505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff400cd430 0x7eff400c7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff3ed9d700 1 -- 172.21.15.1:0/1519725505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff4015b6c0 msgr2=0x7eff4014c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff3ed9d700 1 --2- 172.21.15.1:0/1519725505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff4015b6c0 0x7eff4014c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff3ed9d700 1 -- 172.21.15.1:0/1519725505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff38009b30 con 0x7eff400c87d0 2024-03-18T19:09:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff3ed9d700 1 --2- 172.21.15.1:0/1519725505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400c87d0 0x7eff400c7410 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7eff3000e9e0 tx=0x7eff3000ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff277fe700 1 -- 172.21.15.1:0/1519725505 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff30009710 con 0x7eff400c87d0 2024-03-18T19:09:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff277fe700 1 -- 172.21.15.1:0/1519725505 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3000ee80 con 0x7eff400c87d0 2024-03-18T19:09:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff45204700 1 -- 172.21.15.1:0/1519725505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff4014a3f0 con 0x7eff400c87d0 2024-03-18T19:09:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.944+0000 7eff277fe700 1 -- 172.21.15.1:0/1519725505 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3000ca50 con 0x7eff400c87d0 2024-03-18T19:09:06.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.945+0000 7eff45204700 1 -- 172.21.15.1:0/1519725505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff4014a9b0 con 0x7eff400c87d0 2024-03-18T19:09:06.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.946+0000 7eff45204700 1 -- 172.21.15.1:0/1519725505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff400676a0 con 0x7eff400c87d0 2024-03-18T19:09:06.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.946+0000 7eff277fe700 1 -- 172.21.15.1:0/1519725505 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 68) v1 ==== 89353+0+0 (secure 0 0 0) 0x7eff3000cbb0 con 0x7eff400c87d0 2024-03-18T19:09:06.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.946+0000 7eff277fe700 1 --2- 172.21.15.1:0/1519725505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7eff2c06aed0 0x7eff2c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:06.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.946+0000 7eff3e59c700 1 -- 172.21.15.1:0/1519725505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7eff2c06aed0 msgr2=0x7eff2c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:06.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.946+0000 7eff277fe700 1 -- 172.21.15.1:0/1519725505 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7eff300899b0 con 0x7eff400c87d0 2024-03-18T19:09:06.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.946+0000 7eff3e59c700 1 --2- 172.21.15.1:0/1519725505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7eff2c06aed0 0x7eff2c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:06.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:06.951+0000 7eff277fe700 1 -- 172.21.15.1:0/1519725505 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7eff30054590 con 0x7eff400c87d0 2024-03-18T19:09:07.021 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:07.024+0000 7fcaa747b240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:09:07.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.147+0000 7eff3e59c700 1 -- 172.21.15.1:0/1519725505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7eff2c06aed0 msgr2=0x7eff2c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:07.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.147+0000 7eff3e59c700 1 --2- 172.21.15.1:0/1519725505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7eff2c06aed0 0x7eff2c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:07.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.205+0000 7eff45204700 1 -- 172.21.15.1:0/1519725505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7eff4006f7c0 con 0x7eff400c87d0 2024-03-18T19:09:07.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.205+0000 7eff277fe700 1 -- 172.21.15.1:0/1519725505 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v134) v1 ==== 104+0+11 (secure 0 0 0) 0x7eff30057bb0 con 0x7eff400c87d0 2024-03-18T19:09:07.202 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T19:09:07.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.207+0000 7eff45204700 1 -- 172.21.15.1:0/1519725505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7eff2c06aed0 msgr2=0x7eff2c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:07.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.207+0000 7eff45204700 1 --2- 172.21.15.1:0/1519725505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7eff2c06aed0 0x7eff2c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.207+0000 7eff45204700 1 -- 172.21.15.1:0/1519725505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400c87d0 msgr2=0x7eff400c7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:07.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.207+0000 7eff45204700 1 --2- 172.21.15.1:0/1519725505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400c87d0 0x7eff400c7410 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7eff3000e9e0 tx=0x7eff3000ecf0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.207+0000 7eff45204700 1 -- 172.21.15.1:0/1519725505 shutdown_connections 2024-03-18T19:09:07.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.207+0000 7eff45204700 1 --2- 172.21.15.1:0/1519725505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7eff2c06aed0 0x7eff2c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.207+0000 7eff45204700 1 --2- 172.21.15.1:0/1519725505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff4015b6c0 0x7eff4014c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.207+0000 7eff45204700 1 --2- 172.21.15.1:0/1519725505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff400c87d0 0x7eff400c7410 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.207+0000 7eff45204700 1 --2- 172.21.15.1:0/1519725505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff400cd430 0x7eff400c7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.207+0000 7eff45204700 1 -- 172.21.15.1:0/1519725505 >> 172.21.15.1:0/1519725505 conn(0x7eff400d0ff0 msgr2=0x7eff4015e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:07.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.208+0000 7eff45204700 1 -- 172.21.15.1:0/1519725505 shutdown_connections 2024-03-18T19:09:07.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.208+0000 7eff45204700 1 -- 172.21.15.1:0/1519725505 wait complete. 2024-03-18T19:09:07.217 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T19:09:07.357 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:07.361+0000 7fcaa747b240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:09:07.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.388+0000 7f2278237700 1 Processor -- start 2024-03-18T19:09:07.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.389+0000 7f2278237700 1 -- start start 2024-03-18T19:09:07.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.389+0000 7f2278237700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f227006cdf0 0x7f227006d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:07.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.389+0000 7f2278237700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f22700b3d90 0x7f22700b4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:07.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.389+0000 7f2278237700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f227006d790 con 0x7f227015b4b0 2024-03-18T19:09:07.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.390+0000 7f2278237700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f227015e960 con 0x7f227006cdf0 2024-03-18T19:09:07.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.390+0000 7f2278237700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f227015edf0 con 0x7f22700b3d90 2024-03-18T19:09:07.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.390+0000 7f2275fd3700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f227015b4b0 0x7f2270070d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43548/0 (socket says 172.21.15.1:43548) 2024-03-18T19:09:07.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.390+0000 7f2275fd3700 1 -- 172.21.15.1:0/2850201049 learned_addr learned my addr 172.21.15.1:0/2850201049 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:07.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.390+0000 7f22757d2700 1 --2- 172.21.15.1:0/2850201049 >> v2:172.21.15.165:3300/0 conn(0x7f227006cdf0 0x7f227006d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:07.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.390+0000 7f22767d4700 1 --2- 172.21.15.1:0/2850201049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f22700b3d90 0x7f22700b4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:07.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.390+0000 7f2274fd1700 1 -- 172.21.15.1:0/2850201049 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591870278 0 0) 0x7f227006d790 con 0x7f227015b4b0 2024-03-18T19:09:07.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.390+0000 7f2274fd1700 1 -- 172.21.15.1:0/2850201049 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f225c0035b0 con 0x7f227015b4b0 2024-03-18T19:09:07.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.390+0000 7f22757d2700 1 -- 172.21.15.1:0/2850201049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f22700b3d90 msgr2=0x7f22700b4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:07.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.390+0000 7f22757d2700 1 --2- 172.21.15.1:0/2850201049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f22700b3d90 0x7f22700b4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.390+0000 7f22757d2700 1 -- 172.21.15.1:0/2850201049 >> v1:172.21.15.1:6789/0 conn(0x7f227015b4b0 legacy=0x7f2270070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:07.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.390+0000 7f22757d2700 1 -- 172.21.15.1:0/2850201049 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22700717b0 con 0x7f227006cdf0 2024-03-18T19:09:07.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.390+0000 7f22767d4700 1 --2- 172.21.15.1:0/2850201049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f22700b3d90 0x7f22700b4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:07.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.390+0000 7f22757d2700 1 --2- 172.21.15.1:0/2850201049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f227006cdf0 0x7f227006d250 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f2260009840 tx=0x7f2260009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41523917f500266 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:07.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.391+0000 7f2274fd1700 1 -- 172.21.15.1:0/2850201049 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2260011830 con 0x7f227006cdf0 2024-03-18T19:09:07.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.391+0000 7f2274fd1700 1 -- 172.21.15.1:0/2850201049 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2260011990 con 0x7f227006cdf0 2024-03-18T19:09:07.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.391+0000 7f2274fd1700 1 -- 172.21.15.1:0/2850201049 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2260011d20 con 0x7f227006cdf0 2024-03-18T19:09:07.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.391+0000 7f2278237700 1 -- 172.21.15.1:0/2850201049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f227006cdf0 msgr2=0x7f227006d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:07.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.391+0000 7f2278237700 1 --2- 172.21.15.1:0/2850201049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f227006cdf0 0x7f227006d250 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f2260009840 tx=0x7f2260009b50 comp rx=0 tx=0).stop 2024-03-18T19:09:07.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.391+0000 7f2278237700 1 -- 172.21.15.1:0/2850201049 shutdown_connections 2024-03-18T19:09:07.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.391+0000 7f2278237700 1 --2- 172.21.15.1:0/2850201049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f22700b3d90 0x7f22700b4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.391+0000 7f2278237700 1 --2- 172.21.15.1:0/2850201049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f227006cdf0 0x7f227006d250 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.391+0000 7f2278237700 1 -- 172.21.15.1:0/2850201049 >> 172.21.15.1:0/2850201049 conn(0x7f2270066c80 msgr2=0x7f22700b2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:07.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.391+0000 7f2278237700 1 -- 172.21.15.1:0/2850201049 shutdown_connections 2024-03-18T19:09:07.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.391+0000 7f2278237700 1 -- 172.21.15.1:0/2850201049 wait complete. 2024-03-18T19:09:07.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.393+0000 7f2278237700 1 Processor -- start 2024-03-18T19:09:07.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.393+0000 7f2278237700 1 -- start start 2024-03-18T19:09:07.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.394+0000 7f2278237700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f227006cdf0 0x7f2270140a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:07.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.394+0000 7f2278237700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f22700b3d90 0x7f2270159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:07.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.394+0000 7f2278237700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f227015b4b0 0x7f22701401b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:07.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.394+0000 7f2278237700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2270141d90 con 0x7f227006cdf0 2024-03-18T19:09:07.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.394+0000 7f2278237700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2270159ad0 con 0x7f22700b3d90 2024-03-18T19:09:07.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.394+0000 7f2278237700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2270141110 con 0x7f227015b4b0 2024-03-18T19:09:07.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.394+0000 7f2275fd3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f227006cdf0 0x7f2270140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:07.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.394+0000 7f2275fd3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f227006cdf0 0x7f2270140a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37370/0 (socket says 172.21.15.1:37370) 2024-03-18T19:09:07.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.394+0000 7f2275fd3700 1 -- 172.21.15.1:0/3470734841 learned_addr learned my addr 172.21.15.1:0/3470734841 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:07.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.394+0000 7f22767d4700 1 --2- 172.21.15.1:0/3470734841 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f227015b4b0 0x7f22701401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:07.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.394+0000 7f22757d2700 1 --2- 172.21.15.1:0/3470734841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f22700b3d90 0x7f2270159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:07.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.394+0000 7f2275fd3700 1 -- 172.21.15.1:0/3470734841 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f227015b4b0 msgr2=0x7f22701401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:07.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.394+0000 7f2275fd3700 1 --2- 172.21.15.1:0/3470734841 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f227015b4b0 0x7f22701401b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.394+0000 7f2275fd3700 1 -- 172.21.15.1:0/3470734841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f22700b3d90 msgr2=0x7f2270159590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:07.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.394+0000 7f2275fd3700 1 --2- 172.21.15.1:0/3470734841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f22700b3d90 0x7f2270159590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.394+0000 7f2275fd3700 1 -- 172.21.15.1:0/3470734841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2268009a70 con 0x7f227006cdf0 2024-03-18T19:09:07.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.395+0000 7f2275fd3700 1 --2- 172.21.15.1:0/3470734841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f227006cdf0 0x7f2270140a80 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f226c00c700 tx=0x7f226c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:07.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.395+0000 7f22667fc700 1 -- 172.21.15.1:0/3470734841 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226c005d70 con 0x7f227006cdf0 2024-03-18T19:09:07.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.395+0000 7f22667fc700 1 -- 172.21.15.1:0/3470734841 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f226c0070d0 con 0x7f227006cdf0 2024-03-18T19:09:07.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.395+0000 7f2278237700 1 -- 172.21.15.1:0/3470734841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22600094a0 con 0x7f227006cdf0 2024-03-18T19:09:07.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.395+0000 7f22667fc700 1 -- 172.21.15.1:0/3470734841 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226c007460 con 0x7f227006cdf0 2024-03-18T19:09:07.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.395+0000 7f2278237700 1 -- 172.21.15.1:0/3470734841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22701416d0 con 0x7f227006cdf0 2024-03-18T19:09:07.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.396+0000 7f2278237700 1 -- 172.21.15.1:0/3470734841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2270141ed0 con 0x7f227006cdf0 2024-03-18T19:09:07.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.396+0000 7f22667fc700 1 -- 172.21.15.1:0/3470734841 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 68) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f226c0075c0 con 0x7f227006cdf0 2024-03-18T19:09:07.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.397+0000 7f22667fc700 1 --2- 172.21.15.1:0/3470734841 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f225406aed0 0x7f225406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:07.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.397+0000 7f22757d2700 1 -- 172.21.15.1:0/3470734841 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f225406aed0 msgr2=0x7f225406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:07.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.397+0000 7f22667fc700 1 -- 172.21.15.1:0/3470734841 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f226c088e10 con 0x7f227006cdf0 2024-03-18T19:09:07.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.397+0000 7f22757d2700 1 --2- 172.21.15.1:0/3470734841 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f225406aed0 0x7f225406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:07.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.406+0000 7f22667fc700 1 -- 172.21.15.1:0/3470734841 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2270141ed0 con 0x7f227006cdf0 2024-03-18T19:09:07.434 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:07.437+0000 7f868eda6240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:09:07.520 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:07.524+0000 7fcaa747b240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:09:07.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.598+0000 7f22757d2700 1 -- 172.21.15.1:0/3470734841 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f225406aed0 msgr2=0x7f225406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:07.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.598+0000 7f22757d2700 1 --2- 172.21.15.1:0/3470734841 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f225406aed0 0x7f225406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:07.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.613+0000 7f2278237700 1 -- 172.21.15.1:0/3470734841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2270141ed0 con 0x7f227006cdf0 2024-03-18T19:09:07.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.614+0000 7f22667fc700 1 -- 172.21.15.1:0/3470734841 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v134) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2270141ed0 con 0x7f227006cdf0 2024-03-18T19:09:07.611 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T19:09:07.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.616+0000 7f2278237700 1 -- 172.21.15.1:0/3470734841 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f225406aed0 msgr2=0x7f225406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:07.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.616+0000 7f2278237700 1 --2- 172.21.15.1:0/3470734841 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f225406aed0 0x7f225406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.616+0000 7f2278237700 1 -- 172.21.15.1:0/3470734841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f227006cdf0 msgr2=0x7f2270140a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:07.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.616+0000 7f2278237700 1 --2- 172.21.15.1:0/3470734841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f227006cdf0 0x7f2270140a80 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f226c00c700 tx=0x7f226c00ca10 comp rx=0 tx=0).stop 2024-03-18T19:09:07.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.616+0000 7f2278237700 1 -- 172.21.15.1:0/3470734841 shutdown_connections 2024-03-18T19:09:07.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.616+0000 7f2278237700 1 --2- 172.21.15.1:0/3470734841 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f225406aed0 0x7f225406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.616+0000 7f2278237700 1 --2- 172.21.15.1:0/3470734841 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f227015b4b0 0x7f22701401b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.616+0000 7f2278237700 1 --2- 172.21.15.1:0/3470734841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f227006cdf0 0x7f2270140a80 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.616+0000 7f2278237700 1 --2- 172.21.15.1:0/3470734841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f22700b3d90 0x7f2270159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.616+0000 7f2278237700 1 -- 172.21.15.1:0/3470734841 >> 172.21.15.1:0/3470734841 conn(0x7f2270066c80 msgr2=0x7f2270151f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:07.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.616+0000 7f2278237700 1 -- 172.21.15.1:0/3470734841 shutdown_connections 2024-03-18T19:09:07.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.616+0000 7f2278237700 1 -- 172.21.15.1:0/3470734841 wait complete. 2024-03-18T19:09:07.626 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:09:07.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.758+0000 7fceb70e4700 1 Processor -- start 2024-03-18T19:09:07.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.758+0000 7fceb70e4700 1 -- start start 2024-03-18T19:09:07.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.759+0000 7fceb70e4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fceb015d580 0x7fceb0072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:07.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.759+0000 7fceb70e4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fceb006dd50 0x7fceb00cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:07.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.759+0000 7fceb70e4700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb006f490 con 0x7fceb006e290 2024-03-18T19:09:07.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.759+0000 7fceb70e4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb0071cf0 con 0x7fceb015d580 2024-03-18T19:09:07.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.759+0000 7fceb70e4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb006f1b0 con 0x7fceb006dd50 2024-03-18T19:09:07.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.759+0000 7fceaffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fceb006e290 0x7fceb006d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43550/0 (socket says 172.21.15.1:43550) 2024-03-18T19:09:07.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.759+0000 7fceaffff700 1 -- 172.21.15.1:0/3637855912 learned_addr learned my addr 172.21.15.1:0/3637855912 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:07.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.759+0000 7fceb4e80700 1 --2- 172.21.15.1:0/3637855912 >> v2:172.21.15.165:3300/0 conn(0x7fceb015d580 0x7fceb0072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:07.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.759+0000 7fceb5681700 1 --2- 172.21.15.1:0/3637855912 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fceb006dd50 0x7fceb00cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:07.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.759+0000 7fceaf7fe700 1 -- 172.21.15.1:0/3637855912 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366501905 0 0) 0x7fceb006f490 con 0x7fceb006e290 2024-03-18T19:09:07.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.759+0000 7fceaf7fe700 1 -- 172.21.15.1:0/3637855912 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea00035b0 con 0x7fceb006e290 2024-03-18T19:09:07.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.759+0000 7fceb5681700 1 -- 172.21.15.1:0/3637855912 >> v2:172.21.15.165:3300/0 conn(0x7fceb015d580 msgr2=0x7fceb0072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:07.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.759+0000 7fceaf7fe700 1 -- 172.21.15.1:0/3637855912 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3531051921 0 0) 0x7fcea00035b0 con 0x7fceb006e290 2024-03-18T19:09:07.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.759+0000 7fceb5681700 1 --2- 172.21.15.1:0/3637855912 >> v2:172.21.15.165:3300/0 conn(0x7fceb015d580 0x7fceb0072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.759+0000 7fceb5681700 1 -- 172.21.15.1:0/3637855912 >> v1:172.21.15.1:6789/0 conn(0x7fceb006e290 legacy=0x7fceb006d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:07.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.759+0000 7fceb5681700 1 -- 172.21.15.1:0/3637855912 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb00716a0 con 0x7fceb006dd50 2024-03-18T19:09:07.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.760+0000 7fceb5681700 1 --2- 172.21.15.1:0/3637855912 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fceb006dd50 0x7fceb00cd4b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcea8009d90 tx=0x7fcea800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44729bad6ecc1576 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:07.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.760+0000 7fceaf7fe700 1 -- 172.21.15.1:0/3637855912 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea800e070 con 0x7fceb006dd50 2024-03-18T19:09:07.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.760+0000 7fceaf7fe700 1 -- 172.21.15.1:0/3637855912 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea8003a20 con 0x7fceb006dd50 2024-03-18T19:09:07.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.760+0000 7fceaf7fe700 1 -- 172.21.15.1:0/3637855912 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c001670 con 0x7fceb006dd50 2024-03-18T19:09:07.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.760+0000 7fceb70e4700 1 -- 172.21.15.1:0/3637855912 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fceb006dd50 msgr2=0x7fceb00cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:07.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.760+0000 7fceb70e4700 1 --2- 172.21.15.1:0/3637855912 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fceb006dd50 0x7fceb00cd4b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcea8009d90 tx=0x7fcea800b330 comp rx=0 tx=0).stop 2024-03-18T19:09:07.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.760+0000 7fceb70e4700 1 -- 172.21.15.1:0/3637855912 shutdown_connections 2024-03-18T19:09:07.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.760+0000 7fceb70e4700 1 --2- 172.21.15.1:0/3637855912 >> v2:172.21.15.165:3300/0 conn(0x7fceb015d580 0x7fceb0072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.760+0000 7fceb70e4700 1 --2- 172.21.15.1:0/3637855912 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fceb006dd50 0x7fceb00cd4b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.760+0000 7fceb70e4700 1 -- 172.21.15.1:0/3637855912 >> 172.21.15.1:0/3637855912 conn(0x7fceb00cc990 msgr2=0x7fceb00ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:07.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.761+0000 7fceb70e4700 1 -- 172.21.15.1:0/3637855912 shutdown_connections 2024-03-18T19:09:07.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.761+0000 7fceb70e4700 1 -- 172.21.15.1:0/3637855912 wait complete. 2024-03-18T19:09:07.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.762+0000 7fceb70e4700 1 Processor -- start 2024-03-18T19:09:07.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.762+0000 7fceb70e4700 1 -- start start 2024-03-18T19:09:07.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.763+0000 7fceb70e4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fceb006dd50 0x7fceb015a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:07.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.763+0000 7fceb70e4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fceb015d580 0x7fceb00c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:07.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.763+0000 7fceb70e4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fceb015ace0 0x7fceb00c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:07.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.763+0000 7fceb70e4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb0160900 con 0x7fceb015d580 2024-03-18T19:09:07.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.763+0000 7fceb70e4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb015ab00 con 0x7fceb006dd50 2024-03-18T19:09:07.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.763+0000 7fceb70e4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb0160620 con 0x7fceb015ace0 2024-03-18T19:09:07.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.763+0000 7fceaffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fceb015d580 0x7fceb00c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:07.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.763+0000 7fceaffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fceb015d580 0x7fceb00c82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37386/0 (socket says 172.21.15.1:37386) 2024-03-18T19:09:07.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.763+0000 7fceaffff700 1 -- 172.21.15.1:0/3458300624 learned_addr learned my addr 172.21.15.1:0/3458300624 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:07.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.763+0000 7fceb4e80700 1 --2- 172.21.15.1:0/3458300624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fceb006dd50 0x7fceb015a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:07.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.763+0000 7fceb5681700 1 --2- 172.21.15.1:0/3458300624 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fceb015ace0 0x7fceb00c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:07.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.763+0000 7fceaffff700 1 -- 172.21.15.1:0/3458300624 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fceb015ace0 msgr2=0x7fceb00c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:07.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.763+0000 7fceaffff700 1 --2- 172.21.15.1:0/3458300624 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fceb015ace0 0x7fceb00c8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.763+0000 7fceaffff700 1 -- 172.21.15.1:0/3458300624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fceb006dd50 msgr2=0x7fceb015a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:07.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.763+0000 7fceaffff700 1 --2- 172.21.15.1:0/3458300624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fceb006dd50 0x7fceb015a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:07.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.763+0000 7fceaffff700 1 -- 172.21.15.1:0/3458300624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea8009a70 con 0x7fceb015d580 2024-03-18T19:09:07.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.763+0000 7fceaffff700 1 --2- 172.21.15.1:0/3458300624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fceb015d580 0x7fceb00c82d0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fce9c00c780 tx=0x7fce9c00ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:07.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.763+0000 7fcead7fa700 1 -- 172.21.15.1:0/3458300624 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c004c80 con 0x7fceb015d580 2024-03-18T19:09:07.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.764+0000 7fcead7fa700 1 -- 172.21.15.1:0/3458300624 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9c004de0 con 0x7fceb015d580 2024-03-18T19:09:07.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.764+0000 7fceb70e4700 1 -- 172.21.15.1:0/3458300624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb00c4b70 con 0x7fceb015d580 2024-03-18T19:09:07.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.764+0000 7fcead7fa700 1 -- 172.21.15.1:0/3458300624 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c0072f0 con 0x7fceb015d580 2024-03-18T19:09:07.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.764+0000 7fceb70e4700 1 -- 172.21.15.1:0/3458300624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb00c5130 con 0x7fceb015d580 2024-03-18T19:09:07.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.764+0000 7fceb70e4700 1 -- 172.21.15.1:0/3458300624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb00c49d0 con 0x7fceb015d580 2024-03-18T19:09:07.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.765+0000 7fcead7fa700 1 -- 172.21.15.1:0/3458300624 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 68) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fce9c007450 con 0x7fceb015d580 2024-03-18T19:09:07.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.766+0000 7fcead7fa700 1 --2- 172.21.15.1:0/3458300624 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce9406af00 0x7fce9406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:07.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.766+0000 7fceb4e80700 1 -- 172.21.15.1:0/3458300624 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce9406af00 msgr2=0x7fce9406d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:07.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.766+0000 7fcead7fa700 1 -- 172.21.15.1:0/3458300624 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fce9c089d00 con 0x7fceb015d580 2024-03-18T19:09:07.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.766+0000 7fceb4e80700 1 --2- 172.21.15.1:0/3458300624 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce9406af00 0x7fce9406d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:07.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.770+0000 7fcead7fa700 1 -- 172.21.15.1:0/3458300624 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fce9c054830 con 0x7fceb015d580 2024-03-18T19:09:07.800 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:07.803+0000 7f868eda6240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:09:07.905 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:07.909+0000 7fcaa747b240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:09:07.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.967+0000 7fceb4e80700 1 -- 172.21.15.1:0/3458300624 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce9406af00 msgr2=0x7fce9406d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:07.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:07.967+0000 7fceb4e80700 1 --2- 172.21.15.1:0/3458300624 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce9406af00 0x7fce9406d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:07.974 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:07.977+0000 7f868eda6240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:09:08.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.019+0000 7fceb70e4700 1 -- 172.21.15.1:0/3458300624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fceb00c49d0 con 0x7fceb015d580 2024-03-18T19:09:08.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.019+0000 7fcead7fa700 1 -- 172.21.15.1:0/3458300624 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4730 (secure 0 0 0) 0x7fceb00c49d0 con 0x7fceb015d580 2024-03-18T19:09:08.016 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:08.016 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6800,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3354770356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3354770356},{"type":"v1","addr":"172.21.15.1:6837","nonce":3354770356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":6809,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/695588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":695588699},{"type":"v1","addr":"172.21.15.165:6834","nonce":695588699}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7660,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3186346302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3186346302},{"type":"v1","addr":"172.21.15.1:6836","nonce":3186346302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:08:19.581625+0000","modified":"2024-03-18T19:08:22.589436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7675},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7675":{"gid":7675,"name":"d","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/450810178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":450810178},{"type":"v1","addr":"172.21.15.165:6835","nonce":450810178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-18T19:09:08.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.022+0000 7fceb70e4700 1 -- 172.21.15.1:0/3458300624 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce9406af00 msgr2=0x7fce9406d3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:08.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.022+0000 7fceb70e4700 1 --2- 172.21.15.1:0/3458300624 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce9406af00 0x7fce9406d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.022+0000 7fceb70e4700 1 -- 172.21.15.1:0/3458300624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fceb015d580 msgr2=0x7fceb00c82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:08.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.022+0000 7fceb70e4700 1 --2- 172.21.15.1:0/3458300624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fceb015d580 0x7fceb00c82d0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fce9c00c780 tx=0x7fce9c00ca90 comp rx=0 tx=0).stop 2024-03-18T19:09:08.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.022+0000 7fceb70e4700 1 -- 172.21.15.1:0/3458300624 shutdown_connections 2024-03-18T19:09:08.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.022+0000 7fceb70e4700 1 --2- 172.21.15.1:0/3458300624 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fce9406af00 0x7fce9406d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.022+0000 7fceb70e4700 1 --2- 172.21.15.1:0/3458300624 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fceb015ace0 0x7fceb00c8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.022+0000 7fceb70e4700 1 --2- 172.21.15.1:0/3458300624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fceb006dd50 0x7fceb015a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.022+0000 7fceb70e4700 1 --2- 172.21.15.1:0/3458300624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fceb015d580 0x7fceb00c82d0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.022+0000 7fceb70e4700 1 -- 172.21.15.1:0/3458300624 >> 172.21.15.1:0/3458300624 conn(0x7fceb00cc990 msgr2=0x7fceb015c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:08.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.022+0000 7fceb70e4700 1 -- 172.21.15.1:0/3458300624 shutdown_connections 2024-03-18T19:09:08.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.022+0000 7fceb70e4700 1 -- 172.21.15.1:0/3458300624 wait complete. 2024-03-18T19:09:08.020 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 57 2024-03-18T19:09:08.032 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:09:08.068 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:08.072+0000 7fcaa747b240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:09:08.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.163+0000 7f117f9c1700 1 Processor -- start 2024-03-18T19:09:08.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.163+0000 7f117f9c1700 1 -- start start 2024-03-18T19:09:08.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.164+0000 7f117f9c1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f117806e290 0x7f1178072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:08.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.164+0000 7f117f9c1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f117806dd50 0x7f11780cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:08.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.164+0000 7f117f9c1700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f117806f490 con 0x7f117815d580 2024-03-18T19:09:08.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.164+0000 7f117f9c1700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1178071cf0 con 0x7f117806dd50 2024-03-18T19:09:08.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.164+0000 7f117f9c1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117806f1b0 con 0x7f117806e290 2024-03-18T19:09:08.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.164+0000 7f117cf5c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f117806e290 0x7f1178072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:08.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.165+0000 7f117df5e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f117806dd50 0x7f11780cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:08.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.165+0000 7f117d75d700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f117815d580 0x7f117806d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:52446/0 (socket says 172.21.15.1:52446) 2024-03-18T19:09:08.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.165+0000 7f117d75d700 1 -- 172.21.15.1:0/2120061870 learned_addr learned my addr 172.21.15.1:0/2120061870 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:08.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.165+0000 7f117cf5c700 1 --2- 172.21.15.1:0/2120061870 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f117806e290 0x7f1178072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:43306/0 (socket says 172.21.15.1:43306) 2024-03-18T19:09:08.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.165+0000 7f116ffff700 1 -- 172.21.15.1:0/2120061870 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848261659 0 0) 0x7f117806f490 con 0x7f117815d580 2024-03-18T19:09:08.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.165+0000 7f116ffff700 1 -- 172.21.15.1:0/2120061870 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11640035b0 con 0x7f117815d580 2024-03-18T19:09:08.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.165+0000 7f117df5e700 1 -- 172.21.15.1:0/2120061870 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f117806e290 msgr2=0x7f1178072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:08.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.165+0000 7f117df5e700 1 --2- 172.21.15.1:0/2120061870 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f117806e290 0x7f1178072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.165+0000 7f117df5e700 1 -- 172.21.15.1:0/2120061870 >> v1:172.21.15.165:6789/0 conn(0x7f117815d580 legacy=0x7f117806d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:08.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.165+0000 7f117df5e700 1 -- 172.21.15.1:0/2120061870 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11780716a0 con 0x7f117806dd50 2024-03-18T19:09:08.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.165+0000 7f117cf5c700 1 --2- 172.21.15.1:0/2120061870 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f117806e290 0x7f1178072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:08.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.165+0000 7f117df5e700 1 --2- 172.21.15.1:0/2120061870 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117806dd50 0x7f11780cd4b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1170009d90 tx=0x7f117000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd53496df8a9e6e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:08.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.165+0000 7f116ffff700 1 -- 172.21.15.1:0/2120061870 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117000e070 con 0x7f117806dd50 2024-03-18T19:09:08.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.165+0000 7f116ffff700 1 -- 172.21.15.1:0/2120061870 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117000bb20 con 0x7f117806dd50 2024-03-18T19:09:08.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.165+0000 7f116ffff700 1 -- 172.21.15.1:0/2120061870 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1170012430 con 0x7f117806dd50 2024-03-18T19:09:08.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.166+0000 7f117f9c1700 1 -- 172.21.15.1:0/2120061870 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117806dd50 msgr2=0x7f11780cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:08.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.166+0000 7f117f9c1700 1 --2- 172.21.15.1:0/2120061870 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117806dd50 0x7f11780cd4b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1170009d90 tx=0x7f117000b330 comp rx=0 tx=0).stop 2024-03-18T19:09:08.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.166+0000 7f117f9c1700 1 -- 172.21.15.1:0/2120061870 shutdown_connections 2024-03-18T19:09:08.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.166+0000 7f117f9c1700 1 --2- 172.21.15.1:0/2120061870 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117806dd50 0x7f11780cd4b0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.166+0000 7f117f9c1700 1 --2- 172.21.15.1:0/2120061870 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f117806e290 0x7f1178072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.166+0000 7f117f9c1700 1 -- 172.21.15.1:0/2120061870 >> 172.21.15.1:0/2120061870 conn(0x7f11780cc990 msgr2=0x7f11780ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:08.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.166+0000 7f117f9c1700 1 -- 172.21.15.1:0/2120061870 shutdown_connections 2024-03-18T19:09:08.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.166+0000 7f117f9c1700 1 -- 172.21.15.1:0/2120061870 wait complete. 2024-03-18T19:09:08.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.167+0000 7f117f9c1700 1 Processor -- start 2024-03-18T19:09:08.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.167+0000 7f117f9c1700 1 -- start start 2024-03-18T19:09:08.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.168+0000 7f117f9c1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f117806dd50 0x7f11780b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:08.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.168+0000 7f117f9c1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117815d580 0x7f117815b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:08.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.168+0000 7f117f9c1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1178160570 0x7f117815ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:08.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.168+0000 7f117f9c1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11780b4400 con 0x7f117806dd50 2024-03-18T19:09:08.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.168+0000 7f117f9c1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1178160150 con 0x7f117815d580 2024-03-18T19:09:08.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.168+0000 7f117f9c1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117815b910 con 0x7f1178160570 2024-03-18T19:09:08.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.168+0000 7f117d75d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f117806dd50 0x7f11780b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:08.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.168+0000 7f117d75d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f117806dd50 0x7f11780b57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37394/0 (socket says 172.21.15.1:37394) 2024-03-18T19:09:08.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.168+0000 7f117d75d700 1 -- 172.21.15.1:0/1355639598 learned_addr learned my addr 172.21.15.1:0/1355639598 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:08.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.168+0000 7f117df5e700 1 --2- 172.21.15.1:0/1355639598 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1178160570 0x7f117815ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:08.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.168+0000 7f117cf5c700 1 --2- 172.21.15.1:0/1355639598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117815d580 0x7f117815b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:08.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.168+0000 7f117d75d700 1 -- 172.21.15.1:0/1355639598 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1178160570 msgr2=0x7f117815ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:08.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.168+0000 7f117d75d700 1 --2- 172.21.15.1:0/1355639598 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1178160570 0x7f117815ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.168+0000 7f117d75d700 1 -- 172.21.15.1:0/1355639598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117815d580 msgr2=0x7f117815b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:08.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.168+0000 7f117d75d700 1 --2- 172.21.15.1:0/1355639598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117815d580 0x7f117815b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.168+0000 7f117d75d700 1 -- 172.21.15.1:0/1355639598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11740094a0 con 0x7f117806dd50 2024-03-18T19:09:08.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.168+0000 7f117df5e700 1 --2- 172.21.15.1:0/1355639598 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1178160570 0x7f117815ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:08.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.169+0000 7f117cf5c700 1 --2- 172.21.15.1:0/1355639598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117815d580 0x7f117815b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:08.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.169+0000 7f117d75d700 1 --2- 172.21.15.1:0/1355639598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f117806dd50 0x7f11780b57c0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f116800c4d0 tx=0x7f116800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:08.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.169+0000 7f116dffb700 1 -- 172.21.15.1:0/1355639598 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1168005d70 con 0x7f117806dd50 2024-03-18T19:09:08.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.169+0000 7f116dffb700 1 -- 172.21.15.1:0/1355639598 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116800ce40 con 0x7f117806dd50 2024-03-18T19:09:08.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.169+0000 7f116dffb700 1 -- 172.21.15.1:0/1355639598 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11680072f0 con 0x7f117806dd50 2024-03-18T19:09:08.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.169+0000 7f117f9c1700 1 -- 172.21.15.1:0/1355639598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1170009a70 con 0x7f117806dd50 2024-03-18T19:09:08.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.169+0000 7f117f9c1700 1 -- 172.21.15.1:0/1355639598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117815f7b0 con 0x7f117806dd50 2024-03-18T19:09:08.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.170+0000 7f117f9c1700 1 -- 172.21.15.1:0/1355639598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11780676a0 con 0x7f117806dd50 2024-03-18T19:09:08.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.171+0000 7f116dffb700 1 -- 172.21.15.1:0/1355639598 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 68) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f1168007450 con 0x7f117806dd50 2024-03-18T19:09:08.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.171+0000 7f116dffb700 1 --2- 172.21.15.1:0/1355639598 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f115c06aed0 0x7f115c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:08.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.171+0000 7f117cf5c700 1 -- 172.21.15.1:0/1355639598 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f115c06aed0 msgr2=0x7f115c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:08.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.171+0000 7f116dffb700 1 -- 172.21.15.1:0/1355639598 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f1168088db0 con 0x7f117806dd50 2024-03-18T19:09:08.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.171+0000 7f117cf5c700 1 --2- 172.21.15.1:0/1355639598 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f115c06aed0 0x7f115c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:08.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.176+0000 7f116dffb700 1 -- 172.21.15.1:0/1355639598 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1168053990 con 0x7f117806dd50 2024-03-18T19:09:08.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.372+0000 7f117cf5c700 1 -- 172.21.15.1:0/1355639598 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f115c06aed0 msgr2=0x7f115c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:08.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.372+0000 7f117cf5c700 1 --2- 172.21.15.1:0/1355639598 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f115c06aed0 0x7f115c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:08.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.379+0000 7f117f9c1700 1 -- 172.21.15.1:0/1355639598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f117806d6e0 con 0x7f117806dd50 2024-03-18T19:09:08.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.380+0000 7f116dffb700 1 -- 172.21.15.1:0/1355639598 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+16682 (secure 0 0 0) 0x7f1168017090 con 0x7f117806dd50 2024-03-18T19:09:08.378 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:08.378 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":134,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:08:34.815984+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:08:17.419247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-18T19:08:18.552871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3989116924":"2024-03-19T19:08:34.815944+0000","172.21.15.1:6832/76998":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/3859645093":"2024-03-19T19:08:34.815944+0000","172.21.15.1:6833/76998":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/748689501":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/1964926516":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/2849309461":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/3265489674":"2024-03-19T19:08:34.815944+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:09:08.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.383+0000 7f117f9c1700 1 -- 172.21.15.1:0/1355639598 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f115c06aed0 msgr2=0x7f115c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:08.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.383+0000 7f117f9c1700 1 --2- 172.21.15.1:0/1355639598 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f115c06aed0 0x7f115c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.383+0000 7f117f9c1700 1 -- 172.21.15.1:0/1355639598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f117806dd50 msgr2=0x7f11780b57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:08.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.383+0000 7f117f9c1700 1 --2- 172.21.15.1:0/1355639598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f117806dd50 0x7f11780b57c0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f116800c4d0 tx=0x7f116800c7e0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.383+0000 7f117f9c1700 1 -- 172.21.15.1:0/1355639598 shutdown_connections 2024-03-18T19:09:08.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.383+0000 7f117f9c1700 1 --2- 172.21.15.1:0/1355639598 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f115c06aed0 0x7f115c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.383+0000 7f117f9c1700 1 --2- 172.21.15.1:0/1355639598 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1178160570 0x7f117815ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.383+0000 7f117f9c1700 1 --2- 172.21.15.1:0/1355639598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f117806dd50 0x7f11780b57c0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.383+0000 7f117f9c1700 1 --2- 172.21.15.1:0/1355639598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117815d580 0x7f117815b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.383+0000 7f117f9c1700 1 -- 172.21.15.1:0/1355639598 >> 172.21.15.1:0/1355639598 conn(0x7f11780cc990 msgr2=0x7f11780cb680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:08.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.383+0000 7f117f9c1700 1 -- 172.21.15.1:0/1355639598 shutdown_connections 2024-03-18T19:09:08.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.383+0000 7f117f9c1700 1 -- 172.21.15.1:0/1355639598 wait complete. 2024-03-18T19:09:08.392 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T19:09:08.392 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:09:08.404 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:08.407+0000 7f868eda6240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:09:08.429 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:08.433+0000 7fcaa747b240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:09:08.516 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:08.520+0000 7fcaa747b240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:09:08.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.521+0000 7fccfe4bc700 1 Processor -- start 2024-03-18T19:09:08.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.521+0000 7fccfe4bc700 1 -- start start 2024-03-18T19:09:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.522+0000 7fccfe4bc700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fccf806db70 0x7fccf8165100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.522+0000 7fccfe4bc700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf815a6f0 con 0x7fccf8071d10 2024-03-18T19:09:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.522+0000 7fccfe4bc700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf80cfe50 con 0x7fccf815a310 2024-03-18T19:09:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.522+0000 7fccfe4bc700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf80cdda0 con 0x7fccf806db70 2024-03-18T19:09:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.522+0000 7fccf77fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fccf8071d10 0x7fccf80cb090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43560/0 (socket says 172.21.15.1:43560) 2024-03-18T19:09:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.522+0000 7fccf77fe700 1 -- 172.21.15.1:0/3002615364 learned_addr learned my addr 172.21.15.1:0/3002615364 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccfca59700 1 --2- 172.21.15.1:0/3002615364 >> v2:172.21.15.165:3300/0 conn(0x7fccf806db70 0x7fccf8165100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf6ffd700 1 -- 172.21.15.1:0/3002615364 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136498834 0 0) 0x7fccf815a6f0 con 0x7fccf8071d10 2024-03-18T19:09:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf6ffd700 1 -- 172.21.15.1:0/3002615364 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcce00035b0 con 0x7fccf8071d10 2024-03-18T19:09:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf6ffd700 1 -- 172.21.15.1:0/3002615364 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2119080208 0 0) 0x7fcce00035b0 con 0x7fccf8071d10 2024-03-18T19:09:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf6ffd700 1 -- 172.21.15.1:0/3002615364 >> v2:172.21.15.165:3300/0 conn(0x7fccf806db70 msgr2=0x7fccf8165100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf6ffd700 1 --2- 172.21.15.1:0/3002615364 >> v2:172.21.15.165:3300/0 conn(0x7fccf806db70 0x7fccf8165100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf6ffd700 1 -- 172.21.15.1:0/3002615364 >> v1:172.21.15.165:6789/0 conn(0x7fccf815a310 legacy=0x7fccf8071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf6ffd700 1 -- 172.21.15.1:0/3002615364 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf80ce080 con 0x7fccf8071d10 2024-03-18T19:09:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf6ffd700 1 -- 172.21.15.1:0/3002615364 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fccec001670 con 0x7fccf8071d10 2024-03-18T19:09:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf6ffd700 1 -- 172.21.15.1:0/3002615364 >> v1:172.21.15.1:6789/0 conn(0x7fccf8071d10 legacy=0x7fccf80cb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf6ffd700 1 --2- 172.21.15.1:0/3002615364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcce0003a00 0x7fcce0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf6ffd700 1 --2- 172.21.15.1:0/3002615364 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcce0004850 0x7fcce0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf6ffd700 1 --2- 172.21.15.1:0/3002615364 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcce0005210 0x7fcce000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf6ffd700 1 -- 172.21.15.1:0/3002615364 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf80cdda0 con 0x7fcce0003a00 2024-03-18T19:09:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf6ffd700 1 -- 172.21.15.1:0/3002615364 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce000c1b0 con 0x7fcce0004850 2024-03-18T19:09:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf6ffd700 1 -- 172.21.15.1:0/3002615364 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce000c370 con 0x7fcce0005210 2024-03-18T19:09:08.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf7fff700 1 --2- 172.21.15.1:0/3002615364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcce0003a00 0x7fcce0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:08.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccfca59700 1 --2- 172.21.15.1:0/3002615364 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcce0005210 0x7fcce000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:08.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf77fe700 1 --2- 172.21.15.1:0/3002615364 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcce0004850 0x7fcce0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:08.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf7fff700 1 -- 172.21.15.1:0/3002615364 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcce0005210 msgr2=0x7fcce000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:08.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf7fff700 1 --2- 172.21.15.1:0/3002615364 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcce0005210 0x7fcce000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf7fff700 1 -- 172.21.15.1:0/3002615364 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcce0004850 msgr2=0x7fcce0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:08.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf7fff700 1 --2- 172.21.15.1:0/3002615364 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcce0004850 0x7fcce0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccf7fff700 1 -- 172.21.15.1:0/3002615364 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce00035b0 con 0x7fcce0003a00 2024-03-18T19:09:08.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.523+0000 7fccfca59700 1 -- 172.21.15.1:0/3002615364 reap_dead start 2024-03-18T19:09:08.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.524+0000 7fccf7fff700 1 --2- 172.21.15.1:0/3002615364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcce0003a00 0x7fcce0003e20 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fcce8001800 tx=0x7fcce0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=238eb2264c6998c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:08.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.524+0000 7fccf6ffd700 1 -- 172.21.15.1:0/3002615364 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce80012f0 con 0x7fcce0003a00 2024-03-18T19:09:08.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.524+0000 7fccf6ffd700 1 -- 172.21.15.1:0/3002615364 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce80052f0 con 0x7fcce0003a00 2024-03-18T19:09:08.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.524+0000 7fccf6ffd700 1 -- 172.21.15.1:0/3002615364 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce8006860 con 0x7fcce0003a00 2024-03-18T19:09:08.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.524+0000 7fccfe4bc700 1 -- 172.21.15.1:0/3002615364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcce0003a00 msgr2=0x7fcce0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:08.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.524+0000 7fccfe4bc700 1 --2- 172.21.15.1:0/3002615364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcce0003a00 0x7fcce0003e20 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fcce8001800 tx=0x7fcce0005880 comp rx=0 tx=0).stop 2024-03-18T19:09:08.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.524+0000 7fccfe4bc700 1 -- 172.21.15.1:0/3002615364 shutdown_connections 2024-03-18T19:09:08.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.524+0000 7fccfe4bc700 1 --2- 172.21.15.1:0/3002615364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcce0003a00 0x7fcce0003e20 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.524+0000 7fccfe4bc700 1 -- 172.21.15.1:0/3002615364 >> 172.21.15.1:0/3002615364 conn(0x7fccf80ccab0 msgr2=0x7fccf80ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:08.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.524+0000 7fccfe4bc700 1 -- 172.21.15.1:0/3002615364 shutdown_connections 2024-03-18T19:09:08.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.524+0000 7fccfe4bc700 1 -- 172.21.15.1:0/3002615364 wait complete. 2024-03-18T19:09:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.526+0000 7fccfe4bc700 1 Processor -- start 2024-03-18T19:09:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.526+0000 7fccfe4bc700 1 -- start start 2024-03-18T19:09:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccfe4bc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcce0003a00 0x7fccf81f43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccfe4bc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccf8152ae0 0x7fccf814db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccfe4bc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccf80aea00 0x7fccf814e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccfe4bc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf809e380 con 0x7fccf8152ae0 2024-03-18T19:09:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccfe4bc700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf8152fe0 con 0x7fccf80aea00 2024-03-18T19:09:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccfe4bc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf809e5d0 con 0x7fcce0003a00 2024-03-18T19:09:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccf77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccf8152ae0 0x7fccf814db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccf77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccf8152ae0 0x7fccf814db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37420/0 (socket says 172.21.15.1:37420) 2024-03-18T19:09:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccf77fe700 1 -- 172.21.15.1:0/2644236203 learned_addr learned my addr 172.21.15.1:0/2644236203 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:08.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccfca59700 1 --2- 172.21.15.1:0/2644236203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccf80aea00 0x7fccf814e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:08.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccf7fff700 1 --2- 172.21.15.1:0/2644236203 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcce0003a00 0x7fccf81f43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:08.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccf77fe700 1 -- 172.21.15.1:0/2644236203 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcce0003a00 msgr2=0x7fccf81f43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:08.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccf77fe700 1 --2- 172.21.15.1:0/2644236203 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcce0003a00 0x7fccf81f43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccf77fe700 1 -- 172.21.15.1:0/2644236203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccf80aea00 msgr2=0x7fccf814e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:08.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccf77fe700 1 --2- 172.21.15.1:0/2644236203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccf80aea00 0x7fccf814e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccf77fe700 1 -- 172.21.15.1:0/2644236203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf809e8b0 con 0x7fccf8152ae0 2024-03-18T19:09:08.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccf77fe700 1 --2- 172.21.15.1:0/2644236203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccf8152ae0 0x7fccf814db00 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fccec0072c0 tx=0x7fccec0075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:08.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccf4ff9700 1 -- 172.21.15.1:0/2644236203 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec011580 con 0x7fccf8152ae0 2024-03-18T19:09:08.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccf4ff9700 1 -- 172.21.15.1:0/2644236203 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccec011700 con 0x7fccf8152ae0 2024-03-18T19:09:08.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccfe4bc700 1 -- 172.21.15.1:0/2644236203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf809eb90 con 0x7fccf8152ae0 2024-03-18T19:09:08.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccf4ff9700 1 -- 172.21.15.1:0/2644236203 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec011ab0 con 0x7fccf8152ae0 2024-03-18T19:09:08.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.527+0000 7fccfe4bc700 1 -- 172.21.15.1:0/2644236203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf80af1a0 con 0x7fccf8152ae0 2024-03-18T19:09:08.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.528+0000 7fccfe4bc700 1 -- 172.21.15.1:0/2644236203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf8071990 con 0x7fccf8152ae0 2024-03-18T19:09:08.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.529+0000 7fccf4ff9700 1 -- 172.21.15.1:0/2644236203 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 68) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fccec011c10 con 0x7fccf8152ae0 2024-03-18T19:09:08.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.529+0000 7fccf4ff9700 1 --2- 172.21.15.1:0/2644236203 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcce406aed0 0x7fcce406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:08.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.529+0000 7fccf7fff700 1 -- 172.21.15.1:0/2644236203 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcce406aed0 msgr2=0x7fcce406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:08.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.529+0000 7fccf4ff9700 1 -- 172.21.15.1:0/2644236203 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7fccec089800 con 0x7fccf8152ae0 2024-03-18T19:09:08.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.529+0000 7fccf7fff700 1 --2- 172.21.15.1:0/2644236203 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcce406aed0 0x7fcce406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:08.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.534+0000 7fccf4ff9700 1 -- 172.21.15.1:0/2644236203 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fccec054540 con 0x7fccf8152ae0 2024-03-18T19:09:08.586 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:08.589+0000 7f868eda6240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:09:08.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.730+0000 7fccf7fff700 1 -- 172.21.15.1:0/2644236203 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcce406aed0 msgr2=0x7fcce406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:08.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.730+0000 7fccf7fff700 1 --2- 172.21.15.1:0/2644236203 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcce406aed0 0x7fcce406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:08.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.785+0000 7fccfe4bc700 1 -- 172.21.15.1:0/2644236203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccf80676a0 con 0x7fccf8152ae0 2024-03-18T19:09:08.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.786+0000 7fccf4ff9700 1 -- 172.21.15.1:0/2644236203 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4730 (secure 0 0 0) 0x7fccec057b60 con 0x7fccf8152ae0 2024-03-18T19:09:08.783 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:08.783 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6800,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3354770356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3354770356},{"type":"v1","addr":"172.21.15.1:6837","nonce":3354770356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":6809,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/695588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":695588699},{"type":"v1","addr":"172.21.15.165:6834","nonce":695588699}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7660,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3186346302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3186346302},{"type":"v1","addr":"172.21.15.1:6836","nonce":3186346302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:08:19.581625+0000","modified":"2024-03-18T19:08:22.589436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7675},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7675":{"gid":7675,"name":"d","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/450810178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":450810178},{"type":"v1","addr":"172.21.15.165:6835","nonce":450810178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-18T19:09:08.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.788+0000 7fccfe4bc700 1 -- 172.21.15.1:0/2644236203 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcce406aed0 msgr2=0x7fcce406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:08.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.788+0000 7fccfe4bc700 1 --2- 172.21.15.1:0/2644236203 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcce406aed0 0x7fcce406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.788+0000 7fccfe4bc700 1 -- 172.21.15.1:0/2644236203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccf8152ae0 msgr2=0x7fccf814db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:08.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.788+0000 7fccfe4bc700 1 --2- 172.21.15.1:0/2644236203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccf8152ae0 0x7fccf814db00 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fccec0072c0 tx=0x7fccec0075d0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.788+0000 7fccfe4bc700 1 -- 172.21.15.1:0/2644236203 shutdown_connections 2024-03-18T19:09:08.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.788+0000 7fccfe4bc700 1 --2- 172.21.15.1:0/2644236203 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcce406aed0 0x7fcce406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.788+0000 7fccfe4bc700 1 --2- 172.21.15.1:0/2644236203 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcce0003a00 0x7fccf81f43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.788+0000 7fccfe4bc700 1 --2- 172.21.15.1:0/2644236203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccf80aea00 0x7fccf814e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.788+0000 7fccfe4bc700 1 --2- 172.21.15.1:0/2644236203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccf8152ae0 0x7fccf814db00 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.788+0000 7fccfe4bc700 1 -- 172.21.15.1:0/2644236203 >> 172.21.15.1:0/2644236203 conn(0x7fccf80ccab0 msgr2=0x7fccf80ca660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:08.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.788+0000 7fccfe4bc700 1 -- 172.21.15.1:0/2644236203 shutdown_connections 2024-03-18T19:09:08.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.788+0000 7fccfe4bc700 1 -- 172.21.15.1:0/2644236203 wait complete. 2024-03-18T19:09:08.786 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 57 2024-03-18T19:09:08.799 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:09:08.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.971+0000 7f2f64f79700 1 Processor -- start 2024-03-18T19:09:08.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.971+0000 7f2f64f79700 1 -- start start 2024-03-18T19:09:08.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.972+0000 7f2f64f79700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2f600ae600 0x7f2f600ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:08.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.972+0000 7f2f64f79700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6006e270 con 0x7f2f600cc170 2024-03-18T19:09:08.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.973+0000 7f2f64f79700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f600ae050 con 0x7f2f6015a2e0 2024-03-18T19:09:08.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.973+0000 7f2f64f79700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f6006f690 con 0x7f2f600ae600 2024-03-18T19:09:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.973+0000 7f2f5ed9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2f600cc170 0x7f2f6006ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43564/0 (socket says 172.21.15.1:43564) 2024-03-18T19:09:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.973+0000 7f2f5ed9d700 1 -- 172.21.15.1:0/1768751413 learned_addr learned my addr 172.21.15.1:0/1768751413 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.973+0000 7f2f5e59c700 1 --2- 172.21.15.1:0/1768751413 >> v2:172.21.15.165:3300/0 conn(0x7f2f600ae600 0x7f2f600ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.973+0000 7f2f5d59a700 1 -- 172.21.15.1:0/1768751413 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279493126 0 0) 0x7f2f6006e270 con 0x7f2f600cc170 2024-03-18T19:09:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.973+0000 7f2f5d59a700 1 -- 172.21.15.1:0/1768751413 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f480035b0 con 0x7f2f600cc170 2024-03-18T19:09:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.973+0000 7f2f5d59a700 1 -- 172.21.15.1:0/1768751413 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924918357 0 0) 0x7f2f600ae050 con 0x7f2f6015a2e0 2024-03-18T19:09:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.973+0000 7f2f5d59a700 1 -- 172.21.15.1:0/1768751413 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f6006e270 con 0x7f2f6015a2e0 2024-03-18T19:09:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.973+0000 7f2f5d59a700 1 -- 172.21.15.1:0/1768751413 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2625389801 0 0) 0x7f2f480035b0 con 0x7f2f600cc170 2024-03-18T19:09:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.973+0000 7f2f5e59c700 1 -- 172.21.15.1:0/1768751413 >> v1:172.21.15.165:6789/0 conn(0x7f2f6015a2e0 legacy=0x7f2f600ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.973+0000 7f2f5e59c700 1 -- 172.21.15.1:0/1768751413 >> v1:172.21.15.1:6789/0 conn(0x7f2f600cc170 legacy=0x7f2f6006ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.973+0000 7f2f5e59c700 1 -- 172.21.15.1:0/1768751413 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f600add70 con 0x7f2f600ae600 2024-03-18T19:09:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.973+0000 7f2f5e59c700 1 --2- 172.21.15.1:0/1768751413 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f600ae600 0x7f2f600ad790 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009840 tx=0x7f2f50009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81552189f2c75e73 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.974+0000 7f2f5d59a700 1 -- 172.21.15.1:0/1768751413 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f580019f0 con 0x7f2f600ae600 2024-03-18T19:09:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.974+0000 7f2f5d59a700 1 -- 172.21.15.1:0/1768751413 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f50004030 con 0x7f2f600ae600 2024-03-18T19:09:08.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.974+0000 7f2f5d59a700 1 -- 172.21.15.1:0/1768751413 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f500043c0 con 0x7f2f600ae600 2024-03-18T19:09:08.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.974+0000 7f2f64f79700 1 -- 172.21.15.1:0/1768751413 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f600ae600 msgr2=0x7f2f600ad790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:08.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.974+0000 7f2f64f79700 1 --2- 172.21.15.1:0/1768751413 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f600ae600 0x7f2f600ad790 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009840 tx=0x7f2f50009b50 comp rx=0 tx=0).stop 2024-03-18T19:09:08.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.974+0000 7f2f64f79700 1 -- 172.21.15.1:0/1768751413 shutdown_connections 2024-03-18T19:09:08.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.974+0000 7f2f64f79700 1 --2- 172.21.15.1:0/1768751413 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f600ae600 0x7f2f600ad790 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.974+0000 7f2f64f79700 1 -- 172.21.15.1:0/1768751413 >> 172.21.15.1:0/1768751413 conn(0x7f2f6006b860 msgr2=0x7f2f6006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:08.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.974+0000 7f2f64f79700 1 -- 172.21.15.1:0/1768751413 shutdown_connections 2024-03-18T19:09:08.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.974+0000 7f2f64f79700 1 -- 172.21.15.1:0/1768751413 wait complete. 2024-03-18T19:09:08.973 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:08.975+0000 7f868eda6240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:09:08.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.976+0000 7f2f64f79700 1 Processor -- start 2024-03-18T19:09:08.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.976+0000 7f2f64f79700 1 -- start start 2024-03-18T19:09:08.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.976+0000 7f2f64f79700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f600ae600 0x7f2f601418f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:08.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.976+0000 7f2f64f79700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f600cc170 0x7f2f60150ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:08.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.976+0000 7f2f64f79700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f6015a2e0 0x7f2f6013f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:08.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.976+0000 7f2f64f79700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f601520e0 con 0x7f2f6015a2e0 2024-03-18T19:09:08.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.976+0000 7f2f64f79700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f601502f0 con 0x7f2f600ae600 2024-03-18T19:09:08.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.976+0000 7f2f64f79700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f60142140 con 0x7f2f600cc170 2024-03-18T19:09:08.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.977+0000 7f2f5ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f6015a2e0 0x7f2f6013f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:08.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.977+0000 7f2f5ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f6015a2e0 0x7f2f6013f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37432/0 (socket says 172.21.15.1:37432) 2024-03-18T19:09:08.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.977+0000 7f2f5ed9d700 1 -- 172.21.15.1:0/522933990 learned_addr learned my addr 172.21.15.1:0/522933990 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:08.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.977+0000 7f2f5dd9b700 1 --2- 172.21.15.1:0/522933990 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f600cc170 0x7f2f60150ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.977+0000 7f2f5e59c700 1 --2- 172.21.15.1:0/522933990 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f600ae600 0x7f2f601418f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.977+0000 7f2f5ed9d700 1 -- 172.21.15.1:0/522933990 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f600cc170 msgr2=0x7f2f60150ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.977+0000 7f2f5ed9d700 1 --2- 172.21.15.1:0/522933990 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f600cc170 0x7f2f60150ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.977+0000 7f2f5ed9d700 1 -- 172.21.15.1:0/522933990 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f600ae600 msgr2=0x7f2f601418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.977+0000 7f2f5ed9d700 1 --2- 172.21.15.1:0/522933990 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f600ae600 0x7f2f601418f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.977+0000 7f2f5ed9d700 1 -- 172.21.15.1:0/522933990 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f500094a0 con 0x7f2f6015a2e0 2024-03-18T19:09:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.977+0000 7f2f5ed9d700 1 --2- 172.21.15.1:0/522933990 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f6015a2e0 0x7f2f6013f860 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800cc50 tx=0x7f2f5800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.977+0000 7f2f4effd700 1 -- 172.21.15.1:0/522933990 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5800aad0 con 0x7f2f6015a2e0 2024-03-18T19:09:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.977+0000 7f2f4effd700 1 -- 172.21.15.1:0/522933990 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5800ac30 con 0x7f2f6015a2e0 2024-03-18T19:09:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.977+0000 7f2f4effd700 1 -- 172.21.15.1:0/522933990 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f580070d0 con 0x7f2f6015a2e0 2024-03-18T19:09:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.977+0000 7f2f64f79700 1 -- 172.21.15.1:0/522933990 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f60142420 con 0x7f2f6015a2e0 2024-03-18T19:09:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.977+0000 7f2f64f79700 1 -- 172.21.15.1:0/522933990 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f60150590 con 0x7f2f6015a2e0 2024-03-18T19:09:08.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.979+0000 7f2f64f79700 1 -- 172.21.15.1:0/522933990 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f600676a0 con 0x7f2f6015a2e0 2024-03-18T19:09:08.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.979+0000 7f2f4effd700 1 -- 172.21.15.1:0/522933990 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 68) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f2f58007230 con 0x7f2f6015a2e0 2024-03-18T19:09:08.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.979+0000 7f2f4effd700 1 --2- 172.21.15.1:0/522933990 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2f4006aed0 0x7f2f4006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:08.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.979+0000 7f2f5e59c700 1 -- 172.21.15.1:0/522933990 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2f4006aed0 msgr2=0x7f2f4006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:08.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.980+0000 7f2f4effd700 1 -- 172.21.15.1:0/522933990 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6373+0+0 (secure 0 0 0) 0x7f2f58089430 con 0x7f2f6015a2e0 2024-03-18T19:09:08.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.980+0000 7f2f5e59c700 1 --2- 172.21.15.1:0/522933990 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2f4006aed0 0x7f2f4006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:08.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:08.984+0000 7f2f4effd700 1 -- 172.21.15.1:0/522933990 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2f58054020 con 0x7f2f6015a2e0 2024-03-18T19:09:09.070 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:09.073+0000 7f868eda6240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:09:09.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.093+0000 7f2f4effd700 1 -- 172.21.15.1:0/522933990 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 69) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f2f580543f0 con 0x7f2f6015a2e0 2024-03-18T19:09:09.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.094+0000 7f2f4effd700 1 -- 172.21.15.1:0/522933990 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2f4006aed0 msgr2=0x7f2f4006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.094+0000 7f2f4effd700 1 --2- 172.21.15.1:0/522933990 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2f4006aed0 0x7f2f4006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:09.150 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:09.153+0000 7f86099d3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:09.150 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:09.153+0000 7f86099d3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:09.150 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:09.153+0000 7f86099d3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:09.151 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:09.153+0000 7f86099d3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:09.151 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:09.153+0000 7f86099d3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:09.151 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:09.154+0000 7f86061cc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:09.151 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:09.154+0000 7f86061cc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:09.151 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:09.154+0000 7f86061cc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:09.151 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:09.154+0000 7f86061cc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:09.151 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:09.154+0000 7f86061cc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:09.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.250+0000 7f2f64f79700 1 -- 172.21.15.1:0/522933990 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f600cbee0 con 0x7f2f6015a2e0 2024-03-18T19:09:09.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.250+0000 7f2f4effd700 1 -- 172.21.15.1:0/522933990 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4730 (secure 0 0 0) 0x7f2f58018090 con 0x7f2f6015a2e0 2024-03-18T19:09:09.248 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:09.248 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6800,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3354770356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3354770356},{"type":"v1","addr":"172.21.15.1:6837","nonce":3354770356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":6809,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/695588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":695588699},{"type":"v1","addr":"172.21.15.165:6834","nonce":695588699}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7660,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3186346302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3186346302},{"type":"v1","addr":"172.21.15.1:6836","nonce":3186346302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:08:19.581625+0000","modified":"2024-03-18T19:08:22.589436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7675},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7675":{"gid":7675,"name":"d","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/450810178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":450810178},{"type":"v1","addr":"172.21.15.165:6835","nonce":450810178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-18T19:09:09.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.252+0000 7f2f64f79700 1 -- 172.21.15.1:0/522933990 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f6015a2e0 msgr2=0x7f2f6013f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:09.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.252+0000 7f2f64f79700 1 --2- 172.21.15.1:0/522933990 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f6015a2e0 0x7f2f6013f860 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800cc50 tx=0x7f2f5800cf60 comp rx=0 tx=0).stop 2024-03-18T19:09:09.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.252+0000 7f2f64f79700 1 -- 172.21.15.1:0/522933990 shutdown_connections 2024-03-18T19:09:09.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.252+0000 7f2f64f79700 1 --2- 172.21.15.1:0/522933990 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2f4006aed0 0x7f2f4006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:09.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.252+0000 7f2f64f79700 1 --2- 172.21.15.1:0/522933990 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f6015a2e0 0x7f2f6013f860 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:09.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.252+0000 7f2f64f79700 1 --2- 172.21.15.1:0/522933990 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f600ae600 0x7f2f601418f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:09.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.252+0000 7f2f64f79700 1 --2- 172.21.15.1:0/522933990 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2f600cc170 0x7f2f60150ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:09.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.252+0000 7f2f64f79700 1 -- 172.21.15.1:0/522933990 >> 172.21.15.1:0/522933990 conn(0x7f2f6006b860 msgr2=0x7f2f600b0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:09.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.252+0000 7f2f64f79700 1 -- 172.21.15.1:0/522933990 shutdown_connections 2024-03-18T19:09:09.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.252+0000 7f2f64f79700 1 -- 172.21.15.1:0/522933990 wait complete. 2024-03-18T19:09:09.250 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 57 2024-03-18T19:09:09.261 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:09:09.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.389+0000 7f99867ee700 1 Processor -- start 2024-03-18T19:09:09.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.389+0000 7f99867ee700 1 -- start start 2024-03-18T19:09:09.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.389+0000 7f99867ee700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f99800b0570 0x7f9980099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:09.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.389+0000 7f99867ee700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f998006e880 0x7f9980099f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:09.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.389+0000 7f99867ee700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f998006c7a0 con 0x7f998006ba70 2024-03-18T19:09:09.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.389+0000 7f99867ee700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f998006ca80 con 0x7f99800b0570 2024-03-18T19:09:09.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.390+0000 7f99867ee700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99800cf760 con 0x7f998006e880 2024-03-18T19:09:09.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.390+0000 7f9984d8b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f998006ba70 0x7f998006bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43568/0 (socket says 172.21.15.1:43568) 2024-03-18T19:09:09.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.390+0000 7f9984d8b700 1 -- 172.21.15.1:0/2354726921 learned_addr learned my addr 172.21.15.1:0/2354726921 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:09.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.390+0000 7f997f7fe700 1 --2- 172.21.15.1:0/2354726921 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f998006e880 0x7f9980099f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:09.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.390+0000 7f997ffff700 1 --2- 172.21.15.1:0/2354726921 >> v2:172.21.15.165:3300/0 conn(0x7f99800b0570 0x7f9980099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:09.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.390+0000 7f997effd700 1 -- 172.21.15.1:0/2354726921 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 339472276 0 0) 0x7f998006c7a0 con 0x7f998006ba70 2024-03-18T19:09:09.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.390+0000 7f997effd700 1 -- 172.21.15.1:0/2354726921 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99680035b0 con 0x7f998006ba70 2024-03-18T19:09:09.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.390+0000 7f997f7fe700 1 -- 172.21.15.1:0/2354726921 >> v2:172.21.15.165:3300/0 conn(0x7f99800b0570 msgr2=0x7f9980099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:09.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.390+0000 7f997f7fe700 1 --2- 172.21.15.1:0/2354726921 >> v2:172.21.15.165:3300/0 conn(0x7f99800b0570 0x7f9980099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:09.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.390+0000 7f997f7fe700 1 -- 172.21.15.1:0/2354726921 >> v1:172.21.15.1:6789/0 conn(0x7f998006ba70 legacy=0x7f998006bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:09.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.390+0000 7f997effd700 1 -- 172.21.15.1:0/2354726921 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1175126864 0 0) 0x7f99680035b0 con 0x7f998006ba70 2024-03-18T19:09:09.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.390+0000 7f997f7fe700 1 -- 172.21.15.1:0/2354726921 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99800cfa40 con 0x7f998006e880 2024-03-18T19:09:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.390+0000 7f997f7fe700 1 --2- 172.21.15.1:0/2354726921 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f998006e880 0x7f9980099f80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9974009840 tx=0x7f9974009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7669856a39c7d1fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.391+0000 7f997effd700 1 -- 172.21.15.1:0/2354726921 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99780019f0 con 0x7f998006e880 2024-03-18T19:09:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.391+0000 7f997effd700 1 -- 172.21.15.1:0/2354726921 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9974004030 con 0x7f998006e880 2024-03-18T19:09:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.391+0000 7f997effd700 1 -- 172.21.15.1:0/2354726921 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99740043c0 con 0x7f998006e880 2024-03-18T19:09:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.391+0000 7f99867ee700 1 -- 172.21.15.1:0/2354726921 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f998006e880 msgr2=0x7f9980099f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.391+0000 7f99867ee700 1 --2- 172.21.15.1:0/2354726921 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f998006e880 0x7f9980099f80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9974009840 tx=0x7f9974009b50 comp rx=0 tx=0).stop 2024-03-18T19:09:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.391+0000 7f99867ee700 1 -- 172.21.15.1:0/2354726921 shutdown_connections 2024-03-18T19:09:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.391+0000 7f99867ee700 1 --2- 172.21.15.1:0/2354726921 >> v2:172.21.15.165:3300/0 conn(0x7f99800b0570 0x7f9980099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.391+0000 7f99867ee700 1 --2- 172.21.15.1:0/2354726921 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f998006e880 0x7f9980099f80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.391+0000 7f99867ee700 1 -- 172.21.15.1:0/2354726921 >> 172.21.15.1:0/2354726921 conn(0x7f99800b2c30 msgr2=0x7f99800b3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.391+0000 7f99867ee700 1 -- 172.21.15.1:0/2354726921 shutdown_connections 2024-03-18T19:09:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.391+0000 7f99867ee700 1 -- 172.21.15.1:0/2354726921 wait complete. 2024-03-18T19:09:09.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.392+0000 7f99867ee700 1 Processor -- start 2024-03-18T19:09:09.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.393+0000 7f99867ee700 1 -- start start 2024-03-18T19:09:09.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.393+0000 7f99867ee700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f998006ba70 0x7f99801f0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:09.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.393+0000 7f99867ee700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f998006e880 0x7f9980153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:09.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.393+0000 7f99867ee700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f99800b0570 0x7f9980153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:09.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.393+0000 7f99867ee700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9980156070 con 0x7f998006ba70 2024-03-18T19:09:09.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.393+0000 7f99867ee700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9980156350 con 0x7f99800b0570 2024-03-18T19:09:09.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.393+0000 7f99867ee700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9980159bd0 con 0x7f998006e880 2024-03-18T19:09:09.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.393+0000 7f997ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f998006ba70 0x7f99801f0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:09.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.393+0000 7f997ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f998006ba70 0x7f99801f0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37536/0 (socket says 172.21.15.1:37536) 2024-03-18T19:09:09.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.393+0000 7f997ffff700 1 -- 172.21.15.1:0/3246705388 learned_addr learned my addr 172.21.15.1:0/3246705388 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:09.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.394+0000 7f997f7fe700 1 --2- 172.21.15.1:0/3246705388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f998006e880 0x7f9980153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:09.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.394+0000 7f9984d8b700 1 --2- 172.21.15.1:0/3246705388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f99800b0570 0x7f9980153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:09.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.394+0000 7f997ffff700 1 -- 172.21.15.1:0/3246705388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f998006e880 msgr2=0x7f9980153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:09.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.394+0000 7f997ffff700 1 --2- 172.21.15.1:0/3246705388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f998006e880 0x7f9980153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:09.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.394+0000 7f997ffff700 1 -- 172.21.15.1:0/3246705388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f99800b0570 msgr2=0x7f9980153570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:09.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.394+0000 7f997ffff700 1 --2- 172.21.15.1:0/3246705388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f99800b0570 0x7f9980153570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:09.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.394+0000 7f997ffff700 1 -- 172.21.15.1:0/3246705388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99740094a0 con 0x7f998006ba70 2024-03-18T19:09:09.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.394+0000 7f9984d8b700 1 --2- 172.21.15.1:0/3246705388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f99800b0570 0x7f9980153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:09.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.394+0000 7f997ffff700 1 --2- 172.21.15.1:0/3246705388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f998006ba70 0x7f99801f0450 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f997000e870 tx=0x7f997000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:09.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.394+0000 7f997cff9700 1 -- 172.21.15.1:0/3246705388 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9970009590 con 0x7f998006ba70 2024-03-18T19:09:09.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.394+0000 7f997cff9700 1 -- 172.21.15.1:0/3246705388 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997000ee70 con 0x7f998006ba70 2024-03-18T19:09:09.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.394+0000 7f99867ee700 1 -- 172.21.15.1:0/3246705388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9980159eb0 con 0x7f998006ba70 2024-03-18T19:09:09.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.394+0000 7f997cff9700 1 -- 172.21.15.1:0/3246705388 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997000ca50 con 0x7f998006ba70 2024-03-18T19:09:09.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.394+0000 7f99867ee700 1 -- 172.21.15.1:0/3246705388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99801f0990 con 0x7f998006ba70 2024-03-18T19:09:09.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.395+0000 7f99867ee700 1 -- 172.21.15.1:0/3246705388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f998015a500 con 0x7f998006ba70 2024-03-18T19:09:09.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.395+0000 7f997cff9700 1 -- 172.21.15.1:0/3246705388 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 69) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f997000cbb0 con 0x7f998006ba70 2024-03-18T19:09:09.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.396+0000 7f997cff9700 1 -- 172.21.15.1:0/3246705388 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6717+0+0 (secure 0 0 0) 0x7f99700891a0 con 0x7f998006ba70 2024-03-18T19:09:09.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.400+0000 7f997cff9700 1 -- 172.21.15.1:0/3246705388 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9970054360 con 0x7f998006ba70 2024-03-18T19:09:09.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.604+0000 7f99867ee700 1 -- 172.21.15.1:0/3246705388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99800676a0 con 0x7f998006ba70 2024-03-18T19:09:09.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.606+0000 7f997cff9700 1 -- 172.21.15.1:0/3246705388 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+17168 (secure 0 0 0) 0x7f997000ce60 con 0x7f998006ba70 2024-03-18T19:09:09.603 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:09.603 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":135,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:09:09.088722+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:08:17.419247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-18T19:08:18.552871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/1276764796":"2024-03-19T19:09:09.088679+0000","172.21.15.1:0/3452920932":"2024-03-19T19:09:09.088679+0000","172.21.15.1:0/2595728197":"2024-03-19T19:09:09.088679+0000","172.21.15.1:0/2429556029":"2024-03-19T19:09:09.088679+0000","172.21.15.1:0/608960572":"2024-03-19T19:09:09.088679+0000","172.21.15.1:0/3063582767":"2024-03-19T19:09:09.088679+0000","172.21.15.1:0/1028352806":"2024-03-19T19:09:09.088679+0000","172.21.15.1:0/905856574":"2024-03-19T19:09:09.088679+0000","172.21.15.1:0/3989116924":"2024-03-19T19:08:34.815944+0000","172.21.15.1:6832/76998":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/3859645093":"2024-03-19T19:08:34.815944+0000","172.21.15.1:6833/76998":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/748689501":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/1964926516":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/2849309461":"2024-03-19T19:08:34.815944+0000","172.21.15.1:0/3265489674":"2024-03-19T19:08:34.815944+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:09:09.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.608+0000 7f99867ee700 1 -- 172.21.15.1:0/3246705388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f998006ba70 msgr2=0x7f99801f0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:09.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.608+0000 7f99867ee700 1 --2- 172.21.15.1:0/3246705388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f998006ba70 0x7f99801f0450 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f997000e870 tx=0x7f997000ec30 comp rx=0 tx=0).stop 2024-03-18T19:09:09.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.608+0000 7f99867ee700 1 -- 172.21.15.1:0/3246705388 shutdown_connections 2024-03-18T19:09:09.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.608+0000 7f99867ee700 1 --2- 172.21.15.1:0/3246705388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f99800b0570 0x7f9980153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:09.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.608+0000 7f99867ee700 1 --2- 172.21.15.1:0/3246705388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f998006ba70 0x7f99801f0450 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:09.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.608+0000 7f99867ee700 1 --2- 172.21.15.1:0/3246705388 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f998006e880 0x7f9980153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:09.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.608+0000 7f99867ee700 1 -- 172.21.15.1:0/3246705388 >> 172.21.15.1:0/3246705388 conn(0x7f99800b2c30 msgr2=0x7f998006cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:09.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.608+0000 7f99867ee700 1 -- 172.21.15.1:0/3246705388 shutdown_connections 2024-03-18T19:09:09.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.608+0000 7f99867ee700 1 -- 172.21.15.1:0/3246705388 wait complete. 2024-03-18T19:09:09.617 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T19:09:09.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.747+0000 7f53454e4700 1 Processor -- start 2024-03-18T19:09:09.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.747+0000 7f53454e4700 1 -- start start 2024-03-18T19:09:09.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.748+0000 7f53454e4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f534009dd40 0x7f53400cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:09.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.748+0000 7f53454e4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f53400ce5a0 0x7f53400cea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:09.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.748+0000 7f53454e4700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53400ce090 con 0x7f534015d780 2024-03-18T19:09:09.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.748+0000 7f53454e4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53400cb480 con 0x7f53400ce5a0 2024-03-18T19:09:09.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.748+0000 7f53454e4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534006db30 con 0x7f534009dd40 2024-03-18T19:09:09.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.748+0000 7f533effd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f534015d780 0x7f53400cad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43584/0 (socket says 172.21.15.1:43584) 2024-03-18T19:09:09.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.748+0000 7f533effd700 1 -- 172.21.15.1:0/4150686648 learned_addr learned my addr 172.21.15.1:0/4150686648 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:09.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.748+0000 7f533e7fc700 1 --2- 172.21.15.1:0/4150686648 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f534009dd40 0x7f53400cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:09.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.748+0000 7f533f7fe700 1 --2- 172.21.15.1:0/4150686648 >> v2:172.21.15.165:3300/0 conn(0x7f53400ce5a0 0x7f53400cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:09.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.748+0000 7f533dffb700 1 -- 172.21.15.1:0/4150686648 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199070363 0 0) 0x7f53400ce090 con 0x7f534015d780 2024-03-18T19:09:09.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.748+0000 7f533dffb700 1 -- 172.21.15.1:0/4150686648 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53280035b0 con 0x7f534015d780 2024-03-18T19:09:09.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533dffb700 1 -- 172.21.15.1:0/4150686648 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 390896134 0 0) 0x7f53280035b0 con 0x7f534015d780 2024-03-18T19:09:09.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533dffb700 1 -- 172.21.15.1:0/4150686648 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f534009dd40 msgr2=0x7f53400cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:09.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533dffb700 1 --2- 172.21.15.1:0/4150686648 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f534009dd40 0x7f53400cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:09.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533dffb700 1 -- 172.21.15.1:0/4150686648 >> v2:172.21.15.165:3300/0 conn(0x7f53400ce5a0 msgr2=0x7f53400cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:09.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533dffb700 1 --2- 172.21.15.1:0/4150686648 >> v2:172.21.15.165:3300/0 conn(0x7f53400ce5a0 0x7f53400cea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:09.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533dffb700 1 -- 172.21.15.1:0/4150686648 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534006d520 con 0x7f534015d780 2024-03-18T19:09:09.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533dffb700 1 -- 172.21.15.1:0/4150686648 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f5334001670 con 0x7f534015d780 2024-03-18T19:09:09.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533dffb700 1 -- 172.21.15.1:0/4150686648 >> v1:172.21.15.1:6789/0 conn(0x7f534015d780 legacy=0x7f53400cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:09.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533dffb700 1 --2- 172.21.15.1:0/4150686648 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f534009dd40 0x7f5328003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:09.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533dffb700 1 --2- 172.21.15.1:0/4150686648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5328003f80 0x7f5328004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:09.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533dffb700 1 --2- 172.21.15.1:0/4150686648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5328004d90 0x7f5328005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:09.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533dffb700 1 -- 172.21.15.1:0/4150686648 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53400cb480 con 0x7f5328004d90 2024-03-18T19:09:09.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533dffb700 1 -- 172.21.15.1:0/4150686648 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534006db30 con 0x7f5328003f80 2024-03-18T19:09:09.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533dffb700 1 -- 172.21.15.1:0/4150686648 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532800dc20 con 0x7f534009dd40 2024-03-18T19:09:09.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533f7fe700 1 --2- 172.21.15.1:0/4150686648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5328004d90 0x7f5328005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:09.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533effd700 1 --2- 172.21.15.1:0/4150686648 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f534009dd40 0x7f5328003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:09.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533e7fc700 1 --2- 172.21.15.1:0/4150686648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5328003f80 0x7f5328004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:09.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533f7fe700 1 -- 172.21.15.1:0/4150686648 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f534009dd40 msgr2=0x7f5328003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:09.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533f7fe700 1 --2- 172.21.15.1:0/4150686648 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f534009dd40 0x7f5328003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:09.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533f7fe700 1 -- 172.21.15.1:0/4150686648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5328003f80 msgr2=0x7f5328004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:09.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533f7fe700 1 --2- 172.21.15.1:0/4150686648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5328003f80 0x7f5328004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533f7fe700 1 -- 172.21.15.1:0/4150686648 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53280035b0 con 0x7f5328004d90 2024-03-18T19:09:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533effd700 1 --2- 172.21.15.1:0/4150686648 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f534009dd40 0x7f5328003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.749+0000 7f533f7fe700 1 --2- 172.21.15.1:0/4150686648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5328004d90 0x7f5328005210 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f5338009fc0 tx=0x7f5328005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82658ccd2869eb8e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.750+0000 7f533dffb700 1 -- 172.21.15.1:0/4150686648 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533800ebd0 con 0x7f5328004d90 2024-03-18T19:09:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.750+0000 7f533dffb700 1 -- 172.21.15.1:0/4150686648 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533800f040 con 0x7f5328004d90 2024-03-18T19:09:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.750+0000 7f533dffb700 1 -- 172.21.15.1:0/4150686648 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5338014480 con 0x7f5328004d90 2024-03-18T19:09:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.750+0000 7f53454e4700 1 -- 172.21.15.1:0/4150686648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5328004d90 msgr2=0x7f5328005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.750+0000 7f53454e4700 1 --2- 172.21.15.1:0/4150686648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5328004d90 0x7f5328005210 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f5338009fc0 tx=0x7f5328005880 comp rx=0 tx=0).stop 2024-03-18T19:09:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.750+0000 7f533f7fe700 1 -- 172.21.15.1:0/4150686648 reap_dead start 2024-03-18T19:09:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.750+0000 7f53454e4700 1 -- 172.21.15.1:0/4150686648 shutdown_connections 2024-03-18T19:09:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.750+0000 7f53454e4700 1 -- 172.21.15.1:0/4150686648 >> 172.21.15.1:0/4150686648 conn(0x7f53400cca00 msgr2=0x7f53400cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.750+0000 7f53454e4700 1 -- 172.21.15.1:0/4150686648 shutdown_connections 2024-03-18T19:09:09.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.750+0000 7f53454e4700 1 -- 172.21.15.1:0/4150686648 wait complete. 2024-03-18T19:09:09.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.752+0000 7f53454e4700 1 Processor -- start 2024-03-18T19:09:09.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.752+0000 7f53454e4700 1 -- start start 2024-03-18T19:09:09.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.752+0000 7f53454e4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5328004d90 0x7f534014e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:09.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.752+0000 7f53454e4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f53400c82c0 0x7f53400c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:09.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.752+0000 7f53454e4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5340155510 0x7f534015bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:09.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.752+0000 7f53454e4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53400c8830 con 0x7f5328004d90 2024-03-18T19:09:09.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.752+0000 7f53454e4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53400c8b10 con 0x7f5340155510 2024-03-18T19:09:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.752+0000 7f53454e4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53400c8df0 con 0x7f53400c82c0 2024-03-18T19:09:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.752+0000 7f533effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5328004d90 0x7f534014e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.752+0000 7f533effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5328004d90 0x7f534014e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37558/0 (socket says 172.21.15.1:37558) 2024-03-18T19:09:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.752+0000 7f533effd700 1 -- 172.21.15.1:0/889461879 learned_addr learned my addr 172.21.15.1:0/889461879 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.753+0000 7f533f7fe700 1 --2- 172.21.15.1:0/889461879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5340155510 0x7f534015bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.753+0000 7f533e7fc700 1 --2- 172.21.15.1:0/889461879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f53400c82c0 0x7f53400c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.753+0000 7f533effd700 1 -- 172.21.15.1:0/889461879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f53400c82c0 msgr2=0x7f53400c98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.753+0000 7f533effd700 1 --2- 172.21.15.1:0/889461879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f53400c82c0 0x7f53400c98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.753+0000 7f533effd700 1 -- 172.21.15.1:0/889461879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5340155510 msgr2=0x7f534015bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.753+0000 7f533effd700 1 --2- 172.21.15.1:0/889461879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5340155510 0x7f534015bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.753+0000 7f533effd700 1 -- 172.21.15.1:0/889461879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53400c90d0 con 0x7f5328004d90 2024-03-18T19:09:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.753+0000 7f533effd700 1 --2- 172.21.15.1:0/889461879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5328004d90 0x7f534014e490 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f5334000fb0 tx=0x7f533400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:09.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.753+0000 7f53277fe700 1 -- 172.21.15.1:0/889461879 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533400c780 con 0x7f5328004d90 2024-03-18T19:09:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.753+0000 7f53277fe700 1 -- 172.21.15.1:0/889461879 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533400c8e0 con 0x7f5328004d90 2024-03-18T19:09:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.753+0000 7f53454e4700 1 -- 172.21.15.1:0/889461879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53400c93b0 con 0x7f5328004d90 2024-03-18T19:09:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.753+0000 7f53277fe700 1 -- 172.21.15.1:0/889461879 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533400cc50 con 0x7f5328004d90 2024-03-18T19:09:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.753+0000 7f53454e4700 1 -- 172.21.15.1:0/889461879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53400c3be0 con 0x7f5328004d90 2024-03-18T19:09:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.754+0000 7f53454e4700 1 -- 172.21.15.1:0/889461879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5340073940 con 0x7f5328004d90 2024-03-18T19:09:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.754+0000 7f53277fe700 1 -- 172.21.15.1:0/889461879 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 69) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f533401d460 con 0x7f5328004d90 2024-03-18T19:09:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.755+0000 7f53277fe700 1 -- 172.21.15.1:0/889461879 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6717+0+0 (secure 0 0 0) 0x7f5334089160 con 0x7f5328004d90 2024-03-18T19:09:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:09.759+0000 7f53277fe700 1 -- 172.21.15.1:0/889461879 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f53340543a0 con 0x7f5328004d90 2024-03-18T19:09:10.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.007+0000 7f53454e4700 1 -- 172.21.15.1:0/889461879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f53400676a0 con 0x7f5328004d90 2024-03-18T19:09:10.005 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-18T19:09:10.008+0000 7fbdbeb46700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T19:09:10.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.016+0000 7f53277fe700 1 -- 172.21.15.1:0/889461879 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v58) v1 ==== 161+0+0 (secure 0 0 0) 0x7f533400cdb0 con 0x7f5328004d90 2024-03-18T19:09:10.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.019+0000 7f53454e4700 1 -- 172.21.15.1:0/889461879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5328004d90 msgr2=0x7f534014e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:10.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.019+0000 7f53454e4700 1 --2- 172.21.15.1:0/889461879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5328004d90 0x7f534014e490 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f5334000fb0 tx=0x7f533400ee80 comp rx=0 tx=0).stop 2024-03-18T19:09:10.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.019+0000 7f53454e4700 1 -- 172.21.15.1:0/889461879 shutdown_connections 2024-03-18T19:09:10.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.019+0000 7f53454e4700 1 --2- 172.21.15.1:0/889461879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5340155510 0x7f534015bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:10.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.019+0000 7f53454e4700 1 --2- 172.21.15.1:0/889461879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5328004d90 0x7f534014e490 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:10.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.019+0000 7f53454e4700 1 --2- 172.21.15.1:0/889461879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f53400c82c0 0x7f53400c98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:10.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.019+0000 7f53454e4700 1 -- 172.21.15.1:0/889461879 >> 172.21.15.1:0/889461879 conn(0x7f53400cca00 msgr2=0x7f534006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:10.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.019+0000 7f53454e4700 1 -- 172.21.15.1:0/889461879 shutdown_connections 2024-03-18T19:09:10.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.019+0000 7f53454e4700 1 -- 172.21.15.1:0/889461879 wait complete. 2024-03-18T19:09:10.017 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T19:09:10.028 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T19:09:10.053 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-03-18T19:09:10.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.170+0000 7f814df34700 1 Processor -- start 2024-03-18T19:09:10.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.170+0000 7f814df34700 1 -- start start 2024-03-18T19:09:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.171+0000 7f814df34700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81480cfa90 0x7f814806a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.171+0000 7f814df34700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f814806ac90 0x7f8148167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.171+0000 7f814df34700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81480cf600 con 0x7f814815d5a0 2024-03-18T19:09:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.171+0000 7f814df34700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f814806b140 con 0x7f81480cfa90 2024-03-18T19:09:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.171+0000 7f814df34700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81480ce720 con 0x7f814806ac90 2024-03-18T19:09:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.171+0000 7f813ffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81480cfa90 0x7f814806a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.171+0000 7f81477fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f814815d5a0 0x7f81480ce010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43594/0 (socket says 172.21.15.1:43594) 2024-03-18T19:09:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.171+0000 7f813ffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81480cfa90 0x7f814806a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37574/0 (socket says 172.21.15.1:37574) 2024-03-18T19:09:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.171+0000 7f81477fe700 1 -- 172.21.15.1:0/1400426804 learned_addr learned my addr 172.21.15.1:0/1400426804 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.171+0000 7f8147fff700 1 --2- 172.21.15.1:0/1400426804 >> v2:172.21.15.165:3300/0 conn(0x7f814806ac90 0x7f8148167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.172+0000 7f813ffff700 1 -- 172.21.15.1:0/1400426804 >> v2:172.21.15.165:3300/0 conn(0x7f814806ac90 msgr2=0x7f8148167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.172+0000 7f813ffff700 1 --2- 172.21.15.1:0/1400426804 >> v2:172.21.15.165:3300/0 conn(0x7f814806ac90 0x7f8148167140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.172+0000 7f813ffff700 1 -- 172.21.15.1:0/1400426804 >> v1:172.21.15.1:6789/0 conn(0x7f814815d5a0 legacy=0x7f81480ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.172+0000 7f8146ffd700 1 -- 172.21.15.1:0/1400426804 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159982272 0 0) 0x7f81480cf600 con 0x7f814815d5a0 2024-03-18T19:09:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.172+0000 7f813ffff700 1 -- 172.21.15.1:0/1400426804 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81480ce9f0 con 0x7f81480cfa90 2024-03-18T19:09:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.172+0000 7f813ffff700 1 --2- 172.21.15.1:0/1400426804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81480cfa90 0x7f814806a750 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f81480cdec0 tx=0x7f8130009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=528e4b062d871d03 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.172+0000 7f8146ffd700 1 -- 172.21.15.1:0/1400426804 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8130011830 con 0x7f81480cfa90 2024-03-18T19:09:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.172+0000 7f8146ffd700 1 -- 172.21.15.1:0/1400426804 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8130011990 con 0x7f81480cfa90 2024-03-18T19:09:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.172+0000 7f8146ffd700 1 -- 172.21.15.1:0/1400426804 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8130011d20 con 0x7f81480cfa90 2024-03-18T19:09:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.172+0000 7f814df34700 1 -- 172.21.15.1:0/1400426804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81480cfa90 msgr2=0x7f814806a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.172+0000 7f814df34700 1 --2- 172.21.15.1:0/1400426804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81480cfa90 0x7f814806a750 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f81480cdec0 tx=0x7f8130009b20 comp rx=0 tx=0).stop 2024-03-18T19:09:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.172+0000 7f814df34700 1 -- 172.21.15.1:0/1400426804 shutdown_connections 2024-03-18T19:09:10.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.172+0000 7f814df34700 1 --2- 172.21.15.1:0/1400426804 >> v2:172.21.15.165:3300/0 conn(0x7f814806ac90 0x7f8148167140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:10.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.172+0000 7f814df34700 1 --2- 172.21.15.1:0/1400426804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81480cfa90 0x7f814806a750 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:10.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.172+0000 7f814df34700 1 -- 172.21.15.1:0/1400426804 >> 172.21.15.1:0/1400426804 conn(0x7f81480cc9b0 msgr2=0x7f81480ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:10.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.173+0000 7f814df34700 1 -- 172.21.15.1:0/1400426804 shutdown_connections 2024-03-18T19:09:10.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.173+0000 7f814df34700 1 -- 172.21.15.1:0/1400426804 wait complete. 2024-03-18T19:09:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.174+0000 7f814df34700 1 Processor -- start 2024-03-18T19:09:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.174+0000 7f814df34700 1 -- start start 2024-03-18T19:09:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.174+0000 7f814df34700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f814806ac90 0x7f81480c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.174+0000 7f814df34700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81480cfa90 0x7f81481f26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.174+0000 7f814df34700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f814815d5a0 0x7f814815e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.174+0000 7f814df34700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81481f72a0 con 0x7f81480cfa90 2024-03-18T19:09:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.174+0000 7f814df34700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81481586d0 con 0x7f814815d5a0 2024-03-18T19:09:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.174+0000 7f814df34700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81481f2c00 con 0x7f814806ac90 2024-03-18T19:09:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.175+0000 7f813ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81480cfa90 0x7f81481f26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.175+0000 7f813ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81480cfa90 0x7f81481f26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37582/0 (socket says 172.21.15.1:37582) 2024-03-18T19:09:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.175+0000 7f813ffff700 1 -- 172.21.15.1:0/1513103589 learned_addr learned my addr 172.21.15.1:0/1513103589 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.175+0000 7f8147fff700 1 --2- 172.21.15.1:0/1513103589 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f814815d5a0 0x7f814815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.175+0000 7f81477fe700 1 --2- 172.21.15.1:0/1513103589 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f814806ac90 0x7f81480c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:10.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.175+0000 7f813ffff700 1 -- 172.21.15.1:0/1513103589 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f814806ac90 msgr2=0x7f81480c7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:10.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.175+0000 7f813ffff700 1 --2- 172.21.15.1:0/1513103589 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f814806ac90 0x7f81480c7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:10.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.175+0000 7f813ffff700 1 -- 172.21.15.1:0/1513103589 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f814815d5a0 msgr2=0x7f814815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:10.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.175+0000 7f813ffff700 1 --2- 172.21.15.1:0/1513103589 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f814815d5a0 0x7f814815e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:10.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.175+0000 7f813ffff700 1 -- 172.21.15.1:0/1513103589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81300094a0 con 0x7f81480cfa90 2024-03-18T19:09:10.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.175+0000 7f8147fff700 1 --2- 172.21.15.1:0/1513103589 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f814815d5a0 0x7f814815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:10.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.175+0000 7f81477fe700 1 --2- 172.21.15.1:0/1513103589 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f814806ac90 0x7f81480c7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:10.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.175+0000 7f813ffff700 1 --2- 172.21.15.1:0/1513103589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81480cfa90 0x7f81481f26c0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f8130009470 tx=0x7f8130004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:10.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.175+0000 7f8144ff9700 1 -- 172.21.15.1:0/1513103589 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81300038a0 con 0x7f81480cfa90 2024-03-18T19:09:10.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.175+0000 7f8144ff9700 1 -- 172.21.15.1:0/1513103589 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813001f070 con 0x7f81480cfa90 2024-03-18T19:09:10.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.175+0000 7f814df34700 1 -- 172.21.15.1:0/1513103589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8148157b20 con 0x7f81480cfa90 2024-03-18T19:09:10.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.175+0000 7f8144ff9700 1 -- 172.21.15.1:0/1513103589 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8130003c40 con 0x7f81480cfa90 2024-03-18T19:09:10.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.175+0000 7f814df34700 1 -- 172.21.15.1:0/1513103589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8148158080 con 0x7f81480cfa90 2024-03-18T19:09:10.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.176+0000 7f814df34700 1 -- 172.21.15.1:0/1513103589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81480676a0 con 0x7f81480cfa90 2024-03-18T19:09:10.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.177+0000 7f8144ff9700 1 -- 172.21.15.1:0/1513103589 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 70) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f8130026480 con 0x7f81480cfa90 2024-03-18T19:09:10.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.177+0000 7f8144ff9700 1 --2- 172.21.15.1:0/1513103589 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f812806aae0 0x7f812806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:10.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.177+0000 7f8144ff9700 1 -- 172.21.15.1:0/1513103589 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6859+0+0 (secure 0 0 0) 0x7f8130021070 con 0x7f81480cfa90 2024-03-18T19:09:10.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.177+0000 7f81477fe700 1 --2- 172.21.15.1:0/1513103589 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f812806aae0 0x7f812806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:10.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.178+0000 7f81477fe700 1 --2- 172.21.15.1:0/1513103589 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f812806aae0 0x7f812806cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8138001730 tx=0x7f8138009040 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:10.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.182+0000 7f8144ff9700 1 -- 172.21.15.1:0/1513103589 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8130055db0 con 0x7f81480cfa90 2024-03-18T19:09:10.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:10.431+0000 7f814df34700 1 -- 172.21.15.1:0/1513103589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f814815a8b0 con 0x7f81480cfa90 2024-03-18T19:09:11.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.018+0000 7f8144ff9700 1 -- 172.21.15.1:0/1513103589 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v59) v1 ==== 104+0+0 (secure 0 0 0) 0x7f813002b0a0 con 0x7f81480cfa90 2024-03-18T19:09:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.021+0000 7f814df34700 1 -- 172.21.15.1:0/1513103589 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f812806aae0 msgr2=0x7f812806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.021+0000 7f814df34700 1 --2- 172.21.15.1:0/1513103589 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f812806aae0 0x7f812806cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8138001730 tx=0x7f8138009040 comp rx=0 tx=0).stop 2024-03-18T19:09:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.021+0000 7f814df34700 1 -- 172.21.15.1:0/1513103589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81480cfa90 msgr2=0x7f81481f26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.021+0000 7f814df34700 1 --2- 172.21.15.1:0/1513103589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81480cfa90 0x7f81481f26c0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f8130009470 tx=0x7f8130004750 comp rx=0 tx=0).stop 2024-03-18T19:09:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.021+0000 7f814df34700 1 -- 172.21.15.1:0/1513103589 shutdown_connections 2024-03-18T19:09:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.021+0000 7f814df34700 1 --2- 172.21.15.1:0/1513103589 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f812806aae0 0x7f812806cfa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.021+0000 7f814df34700 1 --2- 172.21.15.1:0/1513103589 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f814806ac90 0x7f81480c7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.021+0000 7f814df34700 1 --2- 172.21.15.1:0/1513103589 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f814815d5a0 0x7f814815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.021+0000 7f814df34700 1 --2- 172.21.15.1:0/1513103589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81480cfa90 0x7f81481f26c0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.021+0000 7f814df34700 1 -- 172.21.15.1:0/1513103589 >> 172.21.15.1:0/1513103589 conn(0x7f81480cc9b0 msgr2=0x7f814815db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.021+0000 7f814df34700 1 -- 172.21.15.1:0/1513103589 shutdown_connections 2024-03-18T19:09:11.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.021+0000 7f814df34700 1 -- 172.21.15.1:0/1513103589 wait complete. 2024-03-18T19:09:11.033 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T19:09:11.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.179+0000 7f766c85f700 1 Processor -- start 2024-03-18T19:09:11.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.179+0000 7f766c85f700 1 -- start start 2024-03-18T19:09:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.180+0000 7f766c85f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f766806c270 0x7f76680aff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.180+0000 7f766c85f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f76680b05c0 0x7f76680cb790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.180+0000 7f766c85f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76680cbcd0 con 0x7f766806c650 2024-03-18T19:09:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.180+0000 7f766c85f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f766806bb10 con 0x7f76680b05c0 2024-03-18T19:09:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.180+0000 7f766c85f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766806bdb0 con 0x7f766806c270 2024-03-18T19:09:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.180+0000 7f7665d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f76680b05c0 0x7f76680cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.181+0000 7f7665d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f76680b05c0 0x7f76680cb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37606/0 (socket says 172.21.15.1:37606) 2024-03-18T19:09:11.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.181+0000 7f7665d9b700 1 -- 172.21.15.1:0/2427879665 learned_addr learned my addr 172.21.15.1:0/2427879665 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.181+0000 7f766659c700 1 --2- 172.21.15.1:0/2427879665 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f766806c270 0x7f76680aff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.181+0000 7f7665d9b700 1 -- 172.21.15.1:0/2427879665 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f766806c270 msgr2=0x7f76680aff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.181+0000 7f7665d9b700 1 --2- 172.21.15.1:0/2427879665 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f766806c270 0x7f76680aff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.181+0000 7f7665d9b700 1 -- 172.21.15.1:0/2427879665 >> v1:172.21.15.165:6789/0 conn(0x7f766806c650 legacy=0x7f76680cfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.181+0000 7f7665d9b700 1 -- 172.21.15.1:0/2427879665 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76680cf690 con 0x7f76680b05c0 2024-03-18T19:09:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.181+0000 7f7665d9b700 1 --2- 172.21.15.1:0/2427879665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76680b05c0 0x7f76680cb790 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f76580098f0 tx=0x7f7658009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d306d45c7ea82130 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.181+0000 7f766559a700 1 -- 172.21.15.1:0/2427879665 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7658011830 con 0x7f76680b05c0 2024-03-18T19:09:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.181+0000 7f766559a700 1 -- 172.21.15.1:0/2427879665 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7658011990 con 0x7f76680b05c0 2024-03-18T19:09:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.181+0000 7f766559a700 1 -- 172.21.15.1:0/2427879665 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7658011d20 con 0x7f76680b05c0 2024-03-18T19:09:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.182+0000 7f766c85f700 1 -- 172.21.15.1:0/2427879665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76680b05c0 msgr2=0x7f76680cb790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.182+0000 7f766c85f700 1 --2- 172.21.15.1:0/2427879665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76680b05c0 0x7f76680cb790 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f76580098f0 tx=0x7f7658009c00 comp rx=0 tx=0).stop 2024-03-18T19:09:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.182+0000 7f766c85f700 1 -- 172.21.15.1:0/2427879665 shutdown_connections 2024-03-18T19:09:11.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.182+0000 7f766c85f700 1 --2- 172.21.15.1:0/2427879665 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f766806c270 0x7f76680aff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.182+0000 7f766c85f700 1 --2- 172.21.15.1:0/2427879665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76680b05c0 0x7f76680cb790 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.182+0000 7f766c85f700 1 -- 172.21.15.1:0/2427879665 >> 172.21.15.1:0/2427879665 conn(0x7f76680b2c30 msgr2=0x7f76680b3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.182+0000 7f766c85f700 1 -- 172.21.15.1:0/2427879665 shutdown_connections 2024-03-18T19:09:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.182+0000 7f766c85f700 1 -- 172.21.15.1:0/2427879665 wait complete. 2024-03-18T19:09:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.183+0000 7f766c85f700 1 Processor -- start 2024-03-18T19:09:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.183+0000 7f766c85f700 1 -- start start 2024-03-18T19:09:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.184+0000 7f766c85f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f766806c270 0x7f76680b6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.184+0000 7f766c85f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f766806c650 0x7f76680c7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.184+0000 7f766c85f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76680b05c0 0x7f76680ca850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.184+0000 7f766c85f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76680c61d0 con 0x7f76680b05c0 2024-03-18T19:09:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.184+0000 7f766c85f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76680c64b0 con 0x7f766806c650 2024-03-18T19:09:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.184+0000 7f766c85f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76680c6790 con 0x7f766806c270 2024-03-18T19:09:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.184+0000 7f766659c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f766806c270 0x7f76680b6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:11.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.184+0000 7f7666d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76680b05c0 0x7f76680ca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:11.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.184+0000 7f766659c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f766806c270 0x7f76680b6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:43544/0 (socket says 172.21.15.1:43544) 2024-03-18T19:09:11.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.184+0000 7f766659c700 1 -- 172.21.15.1:0/2649687906 learned_addr learned my addr 172.21.15.1:0/2649687906 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:11.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.184+0000 7f7665d9b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f766806c650 0x7f76680c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:11.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.184+0000 7f7666d9d700 1 -- 172.21.15.1:0/2649687906 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f766806c270 msgr2=0x7f76680b6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:11.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.184+0000 7f7666d9d700 1 --2- 172.21.15.1:0/2649687906 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f766806c270 0x7f76680b6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:11.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.184+0000 7f7666d9d700 1 -- 172.21.15.1:0/2649687906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f766806c650 msgr2=0x7f76680c7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:11.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.184+0000 7f7666d9d700 1 --2- 172.21.15.1:0/2649687906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f766806c650 0x7f76680c7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:11.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.184+0000 7f7666d9d700 1 -- 172.21.15.1:0/2649687906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7658009550 con 0x7f76680b05c0 2024-03-18T19:09:11.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.185+0000 7f7666d9d700 1 --2- 172.21.15.1:0/2649687906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76680b05c0 0x7f76680ca850 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f766000cae0 tx=0x7f766000cdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:11.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.185+0000 7f7656ffd700 1 -- 172.21.15.1:0/2649687906 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7660003c00 con 0x7f76680b05c0 2024-03-18T19:09:11.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.185+0000 7f7656ffd700 1 -- 172.21.15.1:0/2649687906 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7660003d60 con 0x7f76680b05c0 2024-03-18T19:09:11.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.185+0000 7f7656ffd700 1 -- 172.21.15.1:0/2649687906 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766000aa20 con 0x7f76680b05c0 2024-03-18T19:09:11.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.185+0000 7f766c85f700 1 -- 172.21.15.1:0/2649687906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76680c6a70 con 0x7f76680b05c0 2024-03-18T19:09:11.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.185+0000 7f766c85f700 1 -- 172.21.15.1:0/2649687906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f766815c3a0 con 0x7f76680b05c0 2024-03-18T19:09:11.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.186+0000 7f766c85f700 1 -- 172.21.15.1:0/2649687906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76680c6f90 con 0x7f76680b05c0 2024-03-18T19:09:11.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.186+0000 7f7656ffd700 1 -- 172.21.15.1:0/2649687906 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 70) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f766000ab80 con 0x7f76680b05c0 2024-03-18T19:09:11.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.187+0000 7f7656ffd700 1 --2- 172.21.15.1:0/2649687906 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f764806aae0 0x7f764806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:11.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.187+0000 7f766659c700 1 --2- 172.21.15.1:0/2649687906 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f764806aae0 0x7f764806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:11.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.187+0000 7f7656ffd700 1 -- 172.21.15.1:0/2649687906 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6859+0+0 (secure 0 0 0) 0x7f76600892f0 con 0x7f76680b05c0 2024-03-18T19:09:11.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.187+0000 7f766659c700 1 --2- 172.21.15.1:0/2649687906 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f764806aae0 0x7f764806cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f765c009c00 tx=0x7f765c009210 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:11.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.191+0000 7f7656ffd700 1 -- 172.21.15.1:0/2649687906 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f76680c6f90 con 0x7f76680b05c0 2024-03-18T19:09:11.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:11.394+0000 7f766c85f700 1 -- 172.21.15.1:0/2649687906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f76680c6f90 con 0x7f76680b05c0 2024-03-18T19:09:12.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.104+0000 7f7656ffd700 1 -- 172.21.15.1:0/2649687906 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 71) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f7660054690 con 0x7f76680b05c0 2024-03-18T19:09:12.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.165+0000 7f7656ffd700 1 -- 172.21.15.1:0/2649687906 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v137) v1 ==== 181+0+0 (secure 0 0 0) 0x7f76600542c0 con 0x7f76680b05c0 2024-03-18T19:09:12.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.168+0000 7f766c85f700 1 -- 172.21.15.1:0/2649687906 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f764806aae0 msgr2=0x7f764806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:12.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.168+0000 7f766c85f700 1 --2- 172.21.15.1:0/2649687906 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f764806aae0 0x7f764806cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f765c009c00 tx=0x7f765c009210 comp rx=0 tx=0).stop 2024-03-18T19:09:12.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.168+0000 7f766c85f700 1 -- 172.21.15.1:0/2649687906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76680b05c0 msgr2=0x7f76680ca850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:12.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.168+0000 7f766c85f700 1 --2- 172.21.15.1:0/2649687906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76680b05c0 0x7f76680ca850 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f766000cae0 tx=0x7f766000cdf0 comp rx=0 tx=0).stop 2024-03-18T19:09:12.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.168+0000 7f766c85f700 1 -- 172.21.15.1:0/2649687906 shutdown_connections 2024-03-18T19:09:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.168+0000 7f766c85f700 1 --2- 172.21.15.1:0/2649687906 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f764806aae0 0x7f764806cfa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.168+0000 7f766c85f700 1 --2- 172.21.15.1:0/2649687906 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f766806c270 0x7f76680b6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.168+0000 7f766c85f700 1 --2- 172.21.15.1:0/2649687906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76680b05c0 0x7f76680ca850 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.168+0000 7f766c85f700 1 --2- 172.21.15.1:0/2649687906 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f766806c650 0x7f76680c7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.168+0000 7f766c85f700 1 -- 172.21.15.1:0/2649687906 >> 172.21.15.1:0/2649687906 conn(0x7f76680b2c30 msgr2=0x7f7668099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.168+0000 7f766c85f700 1 -- 172.21.15.1:0/2649687906 shutdown_connections 2024-03-18T19:09:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.168+0000 7f766c85f700 1 -- 172.21.15.1:0/2649687906 wait complete. 2024-03-18T19:09:12.166 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-18T19:09:12.180 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T19:09:12.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.319+0000 7ff3ee3bb700 1 Processor -- start 2024-03-18T19:09:12.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.319+0000 7ff3ee3bb700 1 -- start start 2024-03-18T19:09:12.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.320+0000 7ff3ee3bb700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff3e80b2fa0 0x7ff3e815c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:12.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.320+0000 7ff3ee3bb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3e80d1dc0 0x7ff3e80b60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:12.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.320+0000 7ff3ee3bb700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e815c8a0 con 0x7ff3e815fc10 2024-03-18T19:09:12.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.320+0000 7ff3ee3bb700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e80cc0a0 con 0x7ff3e80b2fa0 2024-03-18T19:09:12.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.320+0000 7ff3ee3bb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e815ef20 con 0x7ff3e80d1dc0 2024-03-18T19:09:12.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.320+0000 7ff3e7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff3e80b2fa0 0x7ff3e815c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:12.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.320+0000 7ff3e7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff3e80b2fa0 0x7ff3e815c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37626/0 (socket says 172.21.15.1:37626) 2024-03-18T19:09:12.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.320+0000 7ff3e7fff700 1 -- 172.21.15.1:0/1448432102 learned_addr learned my addr 172.21.15.1:0/1448432102 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:12.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.320+0000 7ff3ec958700 1 --2- 172.21.15.1:0/1448432102 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3e80d1dc0 0x7ff3e80b60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:12.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.321+0000 7ff3e7fff700 1 -- 172.21.15.1:0/1448432102 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3e80d1dc0 msgr2=0x7ff3e80b60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:12.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.321+0000 7ff3e7fff700 1 --2- 172.21.15.1:0/1448432102 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3e80d1dc0 0x7ff3e80b60c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:12.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.321+0000 7ff3e7fff700 1 -- 172.21.15.1:0/1448432102 >> v1:172.21.15.165:6789/0 conn(0x7ff3e815fc10 legacy=0x7ff3e80d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:12.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.321+0000 7ff3e7fff700 1 -- 172.21.15.1:0/1448432102 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e80cbdc0 con 0x7ff3e80b2fa0 2024-03-18T19:09:12.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.321+0000 7ff3e7fff700 1 --2- 172.21.15.1:0/1448432102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3e80b2fa0 0x7ff3e815c360 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7ff3d80098f0 tx=0x7ff3d8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c919514a19e3c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:12.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.321+0000 7ff3e6ffd700 1 -- 172.21.15.1:0/1448432102 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8004230 con 0x7ff3e80b2fa0 2024-03-18T19:09:12.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.321+0000 7ff3e6ffd700 1 -- 172.21.15.1:0/1448432102 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d8004390 con 0x7ff3e80b2fa0 2024-03-18T19:09:12.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.321+0000 7ff3e6ffd700 1 -- 172.21.15.1:0/1448432102 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d80036a0 con 0x7ff3e80b2fa0 2024-03-18T19:09:12.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.321+0000 7ff3ee3bb700 1 -- 172.21.15.1:0/1448432102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3e80b2fa0 msgr2=0x7ff3e815c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:12.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.321+0000 7ff3ee3bb700 1 --2- 172.21.15.1:0/1448432102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3e80b2fa0 0x7ff3e815c360 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7ff3d80098f0 tx=0x7ff3d8009c00 comp rx=0 tx=0).stop 2024-03-18T19:09:12.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.321+0000 7ff3ee3bb700 1 -- 172.21.15.1:0/1448432102 shutdown_connections 2024-03-18T19:09:12.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.321+0000 7ff3ee3bb700 1 --2- 172.21.15.1:0/1448432102 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3e80d1dc0 0x7ff3e80b60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:12.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.321+0000 7ff3ee3bb700 1 --2- 172.21.15.1:0/1448432102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3e80b2fa0 0x7ff3e815c360 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:12.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.321+0000 7ff3ee3bb700 1 -- 172.21.15.1:0/1448432102 >> 172.21.15.1:0/1448432102 conn(0x7ff3e806a750 msgr2=0x7ff3e806ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:12.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.322+0000 7ff3ee3bb700 1 -- 172.21.15.1:0/1448432102 shutdown_connections 2024-03-18T19:09:12.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.322+0000 7ff3ee3bb700 1 -- 172.21.15.1:0/1448432102 wait complete. 2024-03-18T19:09:12.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.323+0000 7ff3ee3bb700 1 Processor -- start 2024-03-18T19:09:12.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.323+0000 7ff3ee3bb700 1 -- start start 2024-03-18T19:09:12.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3ee3bb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3e80b2fa0 0x7ff3e806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:12.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3ee3bb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3e80d1dc0 0x7ff3e80c6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:12.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3ee3bb700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3e815fc10 0x7ff3e814f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:12.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3ee3bb700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e81f31c0 con 0x7ff3e80d1dc0 2024-03-18T19:09:12.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3ee3bb700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e806d8f0 con 0x7ff3e815fc10 2024-03-18T19:09:12.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3ee3bb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e80c7410 con 0x7ff3e80b2fa0 2024-03-18T19:09:12.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3e77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3e80d1dc0 0x7ff3e80c6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:12.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3e77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3e80d1dc0 0x7ff3e80c6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37628/0 (socket says 172.21.15.1:37628) 2024-03-18T19:09:12.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3e77fe700 1 -- 172.21.15.1:0/4075505466 learned_addr learned my addr 172.21.15.1:0/4075505466 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:12.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3e7fff700 1 --2- 172.21.15.1:0/4075505466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3e80b2fa0 0x7ff3e806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:12.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3ec958700 1 --2- 172.21.15.1:0/4075505466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3e815fc10 0x7ff3e814f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:12.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3e77fe700 1 -- 172.21.15.1:0/4075505466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3e80b2fa0 msgr2=0x7ff3e806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:12.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3e77fe700 1 --2- 172.21.15.1:0/4075505466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3e80b2fa0 0x7ff3e806d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:12.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3e77fe700 1 -- 172.21.15.1:0/4075505466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3e815fc10 msgr2=0x7ff3e814f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:12.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3e77fe700 1 --2- 172.21.15.1:0/4075505466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3e815fc10 0x7ff3e814f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:12.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3e77fe700 1 -- 172.21.15.1:0/4075505466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d8009550 con 0x7ff3e80d1dc0 2024-03-18T19:09:12.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3ec958700 1 --2- 172.21.15.1:0/4075505466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3e815fc10 0x7ff3e814f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:12.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3e77fe700 1 --2- 172.21.15.1:0/4075505466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3e80d1dc0 0x7ff3e80c6ed0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7ff3d000b480 tx=0x7ff3d000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:12.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3e4ff9700 1 -- 172.21.15.1:0/4075505466 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0011830 con 0x7ff3e80d1dc0 2024-03-18T19:09:12.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3e4ff9700 1 -- 172.21.15.1:0/4075505466 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d0011990 con 0x7ff3e80d1dc0 2024-03-18T19:09:12.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3e4ff9700 1 -- 172.21.15.1:0/4075505466 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0011d40 con 0x7ff3e80d1dc0 2024-03-18T19:09:12.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.324+0000 7ff3ee3bb700 1 -- 172.21.15.1:0/4075505466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e806c310 con 0x7ff3e80d1dc0 2024-03-18T19:09:12.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.325+0000 7ff3ee3bb700 1 -- 172.21.15.1:0/4075505466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e806c8d0 con 0x7ff3e80d1dc0 2024-03-18T19:09:12.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.325+0000 7ff3ee3bb700 1 -- 172.21.15.1:0/4075505466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e80676a0 con 0x7ff3e80d1dc0 2024-03-18T19:09:12.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.326+0000 7ff3e4ff9700 1 -- 172.21.15.1:0/4075505466 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 71) v1 ==== 89183+0+0 (secure 0 0 0) 0x7ff3d001a400 con 0x7ff3e80d1dc0 2024-03-18T19:09:12.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.326+0000 7ff3e4ff9700 1 --2- 172.21.15.1:0/4075505466 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff3c806ad20 0x7ff3c806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:12.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.326+0000 7ff3e7fff700 1 --2- 172.21.15.1:0/4075505466 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff3c806ad20 0x7ff3c806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:12.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.326+0000 7ff3e4ff9700 1 -- 172.21.15.1:0/4075505466 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6393+0+0 (secure 0 0 0) 0x7ff3d0089310 con 0x7ff3e80d1dc0 2024-03-18T19:09:12.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.327+0000 7ff3e7fff700 1 --2- 172.21.15.1:0/4075505466 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff3c806ad20 0x7ff3c806d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8009130 tx=0x7ff3d8008030 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:12.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.331+0000 7ff3e4ff9700 1 -- 172.21.15.1:0/4075505466 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff3d00541c0 con 0x7ff3e80d1dc0 2024-03-18T19:09:12.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:12.533+0000 7ff3ee3bb700 1 -- 172.21.15.1:0/4075505466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff3e80ce510 con 0x7ff3e80d1dc0 2024-03-18T19:09:13.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.179+0000 7ff3e4ff9700 1 -- 172.21.15.1:0/4075505466 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff3d0054590 con 0x7ff3e80d1dc0 2024-03-18T19:09:13.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.180+0000 7ff3e4ff9700 1 -- 172.21.15.1:0/4075505466 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v138) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff3d0050860 con 0x7ff3e80d1dc0 2024-03-18T19:09:13.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.186+0000 7ff3ee3bb700 1 -- 172.21.15.1:0/4075505466 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff3c806ad20 msgr2=0x7ff3c806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:13.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.186+0000 7ff3ee3bb700 1 --2- 172.21.15.1:0/4075505466 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff3c806ad20 0x7ff3c806d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8009130 tx=0x7ff3d8008030 comp rx=0 tx=0).stop 2024-03-18T19:09:13.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.186+0000 7ff3ee3bb700 1 -- 172.21.15.1:0/4075505466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3e80d1dc0 msgr2=0x7ff3e80c6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:13.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.186+0000 7ff3ee3bb700 1 --2- 172.21.15.1:0/4075505466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3e80d1dc0 0x7ff3e80c6ed0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7ff3d000b480 tx=0x7ff3d000b840 comp rx=0 tx=0).stop 2024-03-18T19:09:13.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.186+0000 7ff3ee3bb700 1 -- 172.21.15.1:0/4075505466 shutdown_connections 2024-03-18T19:09:13.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.186+0000 7ff3ee3bb700 1 --2- 172.21.15.1:0/4075505466 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff3c806ad20 0x7ff3c806d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.186+0000 7ff3ee3bb700 1 --2- 172.21.15.1:0/4075505466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff3e80b2fa0 0x7ff3e806d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.186+0000 7ff3ee3bb700 1 --2- 172.21.15.1:0/4075505466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3e815fc10 0x7ff3e814f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.186+0000 7ff3ee3bb700 1 --2- 172.21.15.1:0/4075505466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3e80d1dc0 0x7ff3e80c6ed0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.186+0000 7ff3ee3bb700 1 -- 172.21.15.1:0/4075505466 >> 172.21.15.1:0/4075505466 conn(0x7ff3e806a750 msgr2=0x7ff3e815c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:13.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.188+0000 7ff3ee3bb700 1 -- 172.21.15.1:0/4075505466 shutdown_connections 2024-03-18T19:09:13.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.190+0000 7ff3ee3bb700 1 -- 172.21.15.1:0/4075505466 wait complete. 2024-03-18T19:09:13.189 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-18T19:09:13.203 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-18T19:09:13.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.339+0000 7fafb85b1700 1 Processor -- start 2024-03-18T19:09:13.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.339+0000 7fafb85b1700 1 -- start start 2024-03-18T19:09:13.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.340+0000 7fafb85b1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fafb015a240 0x7fafb00ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:13.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.340+0000 7fafb85b1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fafb006d560 0x7fafb006fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:13.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.340+0000 7fafb85b1700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb0070000 con 0x7fafb00715b0 2024-03-18T19:09:13.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.340+0000 7fafb85b1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb00adff0 con 0x7fafb006d560 2024-03-18T19:09:13.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.340+0000 7fafb85b1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb00cac80 con 0x7fafb015a240 2024-03-18T19:09:13.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.340+0000 7fafb5b4c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fafb006d560 0x7fafb006fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:13.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.340+0000 7fafb5b4c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fafb006d560 0x7fafb006fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37638/0 (socket says 172.21.15.1:37638) 2024-03-18T19:09:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.340+0000 7fafb5b4c700 1 -- 172.21.15.1:0/3159680335 learned_addr learned my addr 172.21.15.1:0/3159680335 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.340+0000 7fafb634d700 1 --2- 172.21.15.1:0/3159680335 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fafb015a240 0x7fafb00ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.340+0000 7fafb5b4c700 1 -- 172.21.15.1:0/3159680335 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fafb015a240 msgr2=0x7fafb00ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.340+0000 7fafb5b4c700 1 --2- 172.21.15.1:0/3159680335 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fafb015a240 0x7fafb00ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.340+0000 7fafb5b4c700 1 -- 172.21.15.1:0/3159680335 >> v1:172.21.15.165:6789/0 conn(0x7fafb00715b0 legacy=0x7fafb015de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.340+0000 7fafb5b4c700 1 -- 172.21.15.1:0/3159680335 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb00caf60 con 0x7fafb006d560 2024-03-18T19:09:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.340+0000 7fafb5b4c700 1 --2- 172.21.15.1:0/3159680335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafb006d560 0x7fafb006fac0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fafa00098f0 tx=0x7fafa0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d441438c4071d653 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.340+0000 7fafb534b700 1 -- 172.21.15.1:0/3159680335 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa0004230 con 0x7fafb006d560 2024-03-18T19:09:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.341+0000 7fafb534b700 1 -- 172.21.15.1:0/3159680335 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa0004390 con 0x7fafb006d560 2024-03-18T19:09:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.341+0000 7fafb534b700 1 -- 172.21.15.1:0/3159680335 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa00036a0 con 0x7fafb006d560 2024-03-18T19:09:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.341+0000 7fafb85b1700 1 -- 172.21.15.1:0/3159680335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafb006d560 msgr2=0x7fafb006fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.341+0000 7fafb85b1700 1 --2- 172.21.15.1:0/3159680335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafb006d560 0x7fafb006fac0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fafa00098f0 tx=0x7fafa0009c00 comp rx=0 tx=0).stop 2024-03-18T19:09:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.341+0000 7fafb85b1700 1 -- 172.21.15.1:0/3159680335 shutdown_connections 2024-03-18T19:09:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.341+0000 7fafb85b1700 1 --2- 172.21.15.1:0/3159680335 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fafb015a240 0x7fafb00ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.341+0000 7fafb85b1700 1 --2- 172.21.15.1:0/3159680335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafb006d560 0x7fafb006fac0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.341+0000 7fafb85b1700 1 -- 172.21.15.1:0/3159680335 >> 172.21.15.1:0/3159680335 conn(0x7fafb00cff30 msgr2=0x7fafb00d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:13.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.341+0000 7fafb85b1700 1 -- 172.21.15.1:0/3159680335 shutdown_connections 2024-03-18T19:09:13.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.341+0000 7fafb85b1700 1 -- 172.21.15.1:0/3159680335 wait complete. 2024-03-18T19:09:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.342+0000 7fafb85b1700 1 Processor -- start 2024-03-18T19:09:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.342+0000 7fafb85b1700 1 -- start start 2024-03-18T19:09:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.343+0000 7fafb85b1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafb006d560 0x7fafb0150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.343+0000 7fafb85b1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fafb00715b0 0x7fafb006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.343+0000 7fafb85b1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafb015a240 0x7fafb00b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.343+0000 7fafb85b1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb006de80 con 0x7fafb006d560 2024-03-18T19:09:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.343+0000 7fafb85b1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb006e160 con 0x7fafb015a240 2024-03-18T19:09:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.343+0000 7fafb85b1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb006e440 con 0x7fafb00715b0 2024-03-18T19:09:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.343+0000 7fafb634d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafb006d560 0x7fafb0150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.343+0000 7fafb634d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafb006d560 0x7fafb0150f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37652/0 (socket says 172.21.15.1:37652) 2024-03-18T19:09:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.343+0000 7fafb634d700 1 -- 172.21.15.1:0/693782755 learned_addr learned my addr 172.21.15.1:0/693782755 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.343+0000 7fafb5b4c700 1 --2- 172.21.15.1:0/693782755 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fafb00715b0 0x7fafb006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.343+0000 7fafb6b4e700 1 --2- 172.21.15.1:0/693782755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafb015a240 0x7fafb00b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.343+0000 7fafb634d700 1 -- 172.21.15.1:0/693782755 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fafb00715b0 msgr2=0x7fafb006ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.343+0000 7fafb634d700 1 --2- 172.21.15.1:0/693782755 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fafb00715b0 0x7fafb006ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.343+0000 7fafb634d700 1 -- 172.21.15.1:0/693782755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafb015a240 msgr2=0x7fafb00b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.343+0000 7fafb634d700 1 --2- 172.21.15.1:0/693782755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafb015a240 0x7fafb00b2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.343+0000 7fafb634d700 1 -- 172.21.15.1:0/693782755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa0009550 con 0x7fafb006d560 2024-03-18T19:09:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.344+0000 7fafb634d700 1 --2- 172.21.15.1:0/693782755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafb006d560 0x7fafb0150f00 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fafac00c7f0 tx=0x7fafac00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.344+0000 7fafa6ffd700 1 -- 172.21.15.1:0/693782755 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac011800 con 0x7fafb006d560 2024-03-18T19:09:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.344+0000 7fafa6ffd700 1 -- 172.21.15.1:0/693782755 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafac011960 con 0x7fafb006d560 2024-03-18T19:09:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.344+0000 7fafa6ffd700 1 -- 172.21.15.1:0/693782755 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac011cd0 con 0x7fafb006d560 2024-03-18T19:09:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.344+0000 7fafb85b1700 1 -- 172.21.15.1:0/693782755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb006e720 con 0x7fafb006d560 2024-03-18T19:09:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.344+0000 7fafb85b1700 1 -- 172.21.15.1:0/693782755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb0151440 con 0x7fafb006d560 2024-03-18T19:09:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.345+0000 7fafb85b1700 1 -- 172.21.15.1:0/693782755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb00cfd30 con 0x7fafb006d560 2024-03-18T19:09:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.345+0000 7fafa6ffd700 1 -- 172.21.15.1:0/693782755 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fafac022350 con 0x7fafb006d560 2024-03-18T19:09:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.346+0000 7fafa6ffd700 1 --2- 172.21.15.1:0/693782755 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf9406ade0 0x7faf9406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.346+0000 7fafa6ffd700 1 -- 172.21.15.1:0/693782755 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fafac0894a0 con 0x7fafb006d560 2024-03-18T19:09:13.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.346+0000 7fafb5b4c700 1 --2- 172.21.15.1:0/693782755 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf9406ade0 0x7faf9406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:13.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.346+0000 7fafb5b4c700 1 --2- 172.21.15.1:0/693782755 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf9406ade0 0x7faf9406d2a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fafa0009130 tx=0x7fafa0008030 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:13.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.350+0000 7fafa6ffd700 1 -- 172.21.15.1:0/693782755 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fafac0542c0 con 0x7fafb006d560 2024-03-18T19:09:13.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.551+0000 7fafb85b1700 1 -- 172.21.15.1:0/693782755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fafb00676a0 con 0x7fafb006d560 2024-03-18T19:09:13.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.555+0000 7fafa6ffd700 1 -- 172.21.15.1:0/693782755 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v43) v1 ==== 115+0+0 (secure 0 0 0) 0x7fafac0578e0 con 0x7fafb006d560 2024-03-18T19:09:13.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.557+0000 7fafb85b1700 1 -- 172.21.15.1:0/693782755 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf9406ade0 msgr2=0x7faf9406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:13.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.558+0000 7fafb85b1700 1 --2- 172.21.15.1:0/693782755 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf9406ade0 0x7faf9406d2a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fafa0009130 tx=0x7fafa0008030 comp rx=0 tx=0).stop 2024-03-18T19:09:13.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.558+0000 7fafb85b1700 1 -- 172.21.15.1:0/693782755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafb006d560 msgr2=0x7fafb0150f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:13.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.558+0000 7fafb85b1700 1 --2- 172.21.15.1:0/693782755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafb006d560 0x7fafb0150f00 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fafac00c7f0 tx=0x7fafac00cb00 comp rx=0 tx=0).stop 2024-03-18T19:09:13.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.558+0000 7fafb85b1700 1 -- 172.21.15.1:0/693782755 shutdown_connections 2024-03-18T19:09:13.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.558+0000 7fafb85b1700 1 --2- 172.21.15.1:0/693782755 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf9406ade0 0x7faf9406d2a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.558+0000 7fafb85b1700 1 --2- 172.21.15.1:0/693782755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafb015a240 0x7fafb00b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.558+0000 7fafb85b1700 1 --2- 172.21.15.1:0/693782755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafb006d560 0x7fafb0150f00 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.558+0000 7fafb85b1700 1 --2- 172.21.15.1:0/693782755 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fafb00715b0 0x7fafb006ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.558+0000 7fafb85b1700 1 -- 172.21.15.1:0/693782755 >> 172.21.15.1:0/693782755 conn(0x7fafb00cff30 msgr2=0x7fafb006a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:13.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.558+0000 7fafb85b1700 1 -- 172.21.15.1:0/693782755 shutdown_connections 2024-03-18T19:09:13.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.558+0000 7fafb85b1700 1 -- 172.21.15.1:0/693782755 wait complete. 2024-03-18T19:09:13.567 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-18T19:09:13.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.697+0000 7fda9f5b8700 1 Processor -- start 2024-03-18T19:09:13.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.697+0000 7fda9f5b8700 1 -- start start 2024-03-18T19:09:13.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.698+0000 7fda9f5b8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fda9806fdd0 0x7fda980b3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:13.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.698+0000 7fda9f5b8700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fda980b3870 0x7fda98071570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:13.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.698+0000 7fda9f5b8700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda9806ed30 con 0x7fda98146020 2024-03-18T19:09:13.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.698+0000 7fda9f5b8700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda9806eb50 con 0x7fda9806fdd0 2024-03-18T19:09:13.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.698+0000 7fda9f5b8700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda980b3d20 con 0x7fda980b3870 2024-03-18T19:09:13.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.698+0000 7fda9cb53700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fda9806fdd0 0x7fda980b3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:13.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.698+0000 7fda9d354700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fda98146020 0x7fda981465a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43604/0 (socket says 172.21.15.1:43604) 2024-03-18T19:09:13.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.698+0000 7fda9d354700 1 -- 172.21.15.1:0/2157600017 learned_addr learned my addr 172.21.15.1:0/2157600017 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:13.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.698+0000 7fda9db55700 1 --2- 172.21.15.1:0/2157600017 >> v2:172.21.15.165:3300/0 conn(0x7fda980b3870 0x7fda98071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:13.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.698+0000 7fda9cb53700 1 -- 172.21.15.1:0/2157600017 >> v2:172.21.15.165:3300/0 conn(0x7fda980b3870 msgr2=0x7fda98071570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:13.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.698+0000 7fda9cb53700 1 --2- 172.21.15.1:0/2157600017 >> v2:172.21.15.165:3300/0 conn(0x7fda980b3870 0x7fda98071570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.698+0000 7fda9cb53700 1 -- 172.21.15.1:0/2157600017 >> v1:172.21.15.1:6789/0 conn(0x7fda98146020 legacy=0x7fda981465a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:13.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.698+0000 7fda9cb53700 1 -- 172.21.15.1:0/2157600017 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda9806f010 con 0x7fda9806fdd0 2024-03-18T19:09:13.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.698+0000 7fda8ffff700 1 -- 172.21.15.1:0/2157600017 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042310775 0 0) 0x7fda9806ed30 con 0x7fda98146020 2024-03-18T19:09:13.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.698+0000 7fda9db55700 1 --2- 172.21.15.1:0/2157600017 >> v2:172.21.15.165:3300/0 conn(0x7fda980b3870 0x7fda98071570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:13.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.699+0000 7fda9cb53700 1 --2- 172.21.15.1:0/2157600017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda9806fdd0 0x7fda980b3330 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fda98072d30 tx=0x7fda94009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea53006c355933d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.699+0000 7fda8ffff700 1 -- 172.21.15.1:0/2157600017 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda94011830 con 0x7fda9806fdd0 2024-03-18T19:09:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.699+0000 7fda8ffff700 1 -- 172.21.15.1:0/2157600017 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda94011990 con 0x7fda9806fdd0 2024-03-18T19:09:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.699+0000 7fda8ffff700 1 -- 172.21.15.1:0/2157600017 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda94011d20 con 0x7fda9806fdd0 2024-03-18T19:09:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.699+0000 7fda9f5b8700 1 -- 172.21.15.1:0/2157600017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda9806fdd0 msgr2=0x7fda980b3330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.699+0000 7fda9f5b8700 1 --2- 172.21.15.1:0/2157600017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda9806fdd0 0x7fda980b3330 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fda98072d30 tx=0x7fda94009b20 comp rx=0 tx=0).stop 2024-03-18T19:09:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.699+0000 7fda9f5b8700 1 -- 172.21.15.1:0/2157600017 shutdown_connections 2024-03-18T19:09:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.699+0000 7fda9f5b8700 1 --2- 172.21.15.1:0/2157600017 >> v2:172.21.15.165:3300/0 conn(0x7fda980b3870 0x7fda98071570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.699+0000 7fda9f5b8700 1 --2- 172.21.15.1:0/2157600017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda9806fdd0 0x7fda980b3330 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.699+0000 7fda9f5b8700 1 -- 172.21.15.1:0/2157600017 >> 172.21.15.1:0/2157600017 conn(0x7fda980b1d20 msgr2=0x7fda980b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.699+0000 7fda9f5b8700 1 -- 172.21.15.1:0/2157600017 shutdown_connections 2024-03-18T19:09:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.699+0000 7fda9f5b8700 1 -- 172.21.15.1:0/2157600017 wait complete. 2024-03-18T19:09:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.701+0000 7fda9f5b8700 1 Processor -- start 2024-03-18T19:09:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.701+0000 7fda9f5b8700 1 -- start start 2024-03-18T19:09:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.701+0000 7fda9f5b8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda9806fdd0 0x7fda981e4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.701+0000 7fda9f5b8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda980b3870 0x7fda98139340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.701+0000 7fda9f5b8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fda98146020 0x7fda9813d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.701+0000 7fda9f5b8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9806f9b0 con 0x7fda980b3870 2024-03-18T19:09:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.701+0000 7fda9f5b8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda981e4c70 con 0x7fda9806fdd0 2024-03-18T19:09:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.701+0000 7fda9f5b8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda980cec80 con 0x7fda98146020 2024-03-18T19:09:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.701+0000 7fda9cb53700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda980b3870 0x7fda98139340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:13.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.701+0000 7fda9cb53700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda980b3870 0x7fda98139340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37662/0 (socket says 172.21.15.1:37662) 2024-03-18T19:09:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.701+0000 7fda9cb53700 1 -- 172.21.15.1:0/1030066808 learned_addr learned my addr 172.21.15.1:0/1030066808 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.702+0000 7fda9d354700 1 --2- 172.21.15.1:0/1030066808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda9806fdd0 0x7fda981e4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.702+0000 7fda9db55700 1 --2- 172.21.15.1:0/1030066808 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fda98146020 0x7fda9813d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.702+0000 7fda9cb53700 1 -- 172.21.15.1:0/1030066808 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fda98146020 msgr2=0x7fda9813d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.702+0000 7fda9cb53700 1 --2- 172.21.15.1:0/1030066808 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fda98146020 0x7fda9813d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.702+0000 7fda9cb53700 1 -- 172.21.15.1:0/1030066808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda9806fdd0 msgr2=0x7fda981e4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.702+0000 7fda9cb53700 1 --2- 172.21.15.1:0/1030066808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda9806fdd0 0x7fda981e4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.702+0000 7fda9cb53700 1 -- 172.21.15.1:0/1030066808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda940094a0 con 0x7fda980b3870 2024-03-18T19:09:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.702+0000 7fda9cb53700 1 --2- 172.21.15.1:0/1030066808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda980b3870 0x7fda98139340 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fda940090b0 tx=0x7fda94004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.702+0000 7fda8dffb700 1 -- 172.21.15.1:0/1030066808 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda940038a0 con 0x7fda980b3870 2024-03-18T19:09:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.702+0000 7fda8dffb700 1 -- 172.21.15.1:0/1030066808 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9401e070 con 0x7fda980b3870 2024-03-18T19:09:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.702+0000 7fda8dffb700 1 -- 172.21.15.1:0/1030066808 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda94003c40 con 0x7fda980b3870 2024-03-18T19:09:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.702+0000 7fda9f5b8700 1 -- 172.21.15.1:0/1030066808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda980cdd90 con 0x7fda980b3870 2024-03-18T19:09:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.702+0000 7fda9f5b8700 1 -- 172.21.15.1:0/1030066808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda980ce350 con 0x7fda980b3870 2024-03-18T19:09:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.703+0000 7fda9f5b8700 1 -- 172.21.15.1:0/1030066808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda98070f20 con 0x7fda980b3870 2024-03-18T19:09:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.704+0000 7fda8dffb700 1 -- 172.21.15.1:0/1030066808 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fda9402d450 con 0x7fda980b3870 2024-03-18T19:09:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.704+0000 7fda8dffb700 1 --2- 172.21.15.1:0/1030066808 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fda7c06adb0 0x7fda7c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.704+0000 7fda8dffb700 1 -- 172.21.15.1:0/1030066808 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fda9408a620 con 0x7fda980b3870 2024-03-18T19:09:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.704+0000 7fda9d354700 1 --2- 172.21.15.1:0/1030066808 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fda7c06adb0 0x7fda7c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.705+0000 7fda9d354700 1 --2- 172.21.15.1:0/1030066808 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fda7c06adb0 0x7fda7c06d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fda88007fd0 tx=0x7fda88007de0 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:13.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.709+0000 7fda8dffb700 1 -- 172.21.15.1:0/1030066808 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fda94055670 con 0x7fda980b3870 2024-03-18T19:09:13.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.914+0000 7fda9f5b8700 1 -- 172.21.15.1:0/1030066808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fda980676a0 con 0x7fda980b3870 2024-03-18T19:09:13.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.918+0000 7fda8dffb700 1 -- 172.21.15.1:0/1030066808 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v44) v1 ==== 120+0+0 (secure 0 0 0) 0x7fda94088e60 con 0x7fda980b3870 2024-03-18T19:09:13.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.920+0000 7fda9f5b8700 1 -- 172.21.15.1:0/1030066808 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fda7c06adb0 msgr2=0x7fda7c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:13.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.920+0000 7fda9f5b8700 1 --2- 172.21.15.1:0/1030066808 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fda7c06adb0 0x7fda7c06d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fda88007fd0 tx=0x7fda88007de0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.920+0000 7fda9f5b8700 1 -- 172.21.15.1:0/1030066808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda980b3870 msgr2=0x7fda98139340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:13.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.920+0000 7fda9f5b8700 1 --2- 172.21.15.1:0/1030066808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda980b3870 0x7fda98139340 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fda940090b0 tx=0x7fda94004750 comp rx=0 tx=0).stop 2024-03-18T19:09:13.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.920+0000 7fda9f5b8700 1 -- 172.21.15.1:0/1030066808 shutdown_connections 2024-03-18T19:09:13.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.920+0000 7fda9f5b8700 1 --2- 172.21.15.1:0/1030066808 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fda7c06adb0 0x7fda7c06d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.920+0000 7fda9f5b8700 1 --2- 172.21.15.1:0/1030066808 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fda98146020 0x7fda9813d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.920+0000 7fda9f5b8700 1 --2- 172.21.15.1:0/1030066808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda9806fdd0 0x7fda981e4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.920+0000 7fda9f5b8700 1 --2- 172.21.15.1:0/1030066808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda980b3870 0x7fda98139340 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:13.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.920+0000 7fda9f5b8700 1 -- 172.21.15.1:0/1030066808 >> 172.21.15.1:0/1030066808 conn(0x7fda980b1d20 msgr2=0x7fda980ac8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:13.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.921+0000 7fda9f5b8700 1 -- 172.21.15.1:0/1030066808 shutdown_connections 2024-03-18T19:09:13.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:13.921+0000 7fda9f5b8700 1 -- 172.21.15.1:0/1030066808 wait complete. 2024-03-18T19:09:13.930 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-18T19:09:14.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.068+0000 7f8681da5700 1 Processor -- start 2024-03-18T19:09:14.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.068+0000 7f8681da5700 1 -- start start 2024-03-18T19:09:14.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.069+0000 7f8681da5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f867c06e6b0 0x7f867c0b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:14.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.069+0000 7f8681da5700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867c15df40 con 0x7f867c070e40 2024-03-18T19:09:14.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.069+0000 7f8681da5700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867c15e220 con 0x7f867c06e200 2024-03-18T19:09:14.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.069+0000 7f8681da5700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f867c15e500 con 0x7f867c06e6b0 2024-03-18T19:09:14.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.069+0000 7f867bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f867c06e6b0 0x7f867c0b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:14.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.069+0000 7f867b7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f867c070e40 0x7f867c071220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43612/0 (socket says 172.21.15.1:43612) 2024-03-18T19:09:14.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.069+0000 7f867b7fe700 1 -- 172.21.15.1:0/3930393440 learned_addr learned my addr 172.21.15.1:0/3930393440 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:14.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.070+0000 7f867bfff700 1 -- 172.21.15.1:0/3930393440 >> v1:172.21.15.165:6789/0 conn(0x7f867c06e200 legacy=0x7f867c0b2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:14.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.070+0000 7f867bfff700 1 -- 172.21.15.1:0/3930393440 >> v1:172.21.15.1:6789/0 conn(0x7f867c070e40 legacy=0x7f867c071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:14.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.070+0000 7f867bfff700 1 -- 172.21.15.1:0/3930393440 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c15e7e0 con 0x7f867c06e6b0 2024-03-18T19:09:14.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.070+0000 7f867a7fc700 1 -- 172.21.15.1:0/3930393440 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533506571 0 0) 0x7f867c15df40 con 0x7f867c070e40 2024-03-18T19:09:14.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.070+0000 7f867bfff700 1 --2- 172.21.15.1:0/3930393440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f867c06e6b0 0x7f867c0b2f70 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f8674009d90 tx=0x7f867400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ad9ee02e7cb051f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:14.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.070+0000 7f867a7fc700 1 -- 172.21.15.1:0/3930393440 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867400e070 con 0x7f867c06e6b0 2024-03-18T19:09:14.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.070+0000 7f867a7fc700 1 -- 172.21.15.1:0/3930393440 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867400bb90 con 0x7f867c06e6b0 2024-03-18T19:09:14.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.070+0000 7f867a7fc700 1 -- 172.21.15.1:0/3930393440 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8674003a20 con 0x7f867c06e6b0 2024-03-18T19:09:14.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.070+0000 7f8681da5700 1 -- 172.21.15.1:0/3930393440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f867c06e6b0 msgr2=0x7f867c0b2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:14.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.070+0000 7f8681da5700 1 --2- 172.21.15.1:0/3930393440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f867c06e6b0 0x7f867c0b2f70 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f8674009d90 tx=0x7f867400b330 comp rx=0 tx=0).stop 2024-03-18T19:09:14.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.070+0000 7f8681da5700 1 -- 172.21.15.1:0/3930393440 shutdown_connections 2024-03-18T19:09:14.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.070+0000 7f8681da5700 1 --2- 172.21.15.1:0/3930393440 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f867c06e6b0 0x7f867c0b2f70 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:14.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.070+0000 7f8681da5700 1 -- 172.21.15.1:0/3930393440 >> 172.21.15.1:0/3930393440 conn(0x7f867c06a750 msgr2=0x7f867c06ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:14.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.070+0000 7f8681da5700 1 -- 172.21.15.1:0/3930393440 shutdown_connections 2024-03-18T19:09:14.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.070+0000 7f8681da5700 1 -- 172.21.15.1:0/3930393440 wait complete. 2024-03-18T19:09:14.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.072+0000 7f8681da5700 1 Processor -- start 2024-03-18T19:09:14.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.072+0000 7f8681da5700 1 -- start start 2024-03-18T19:09:14.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.072+0000 7f8681da5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f867c06e200 0x7f867c14de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:14.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.072+0000 7f8681da5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f867c06e6b0 0x7f867c14f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:14.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.072+0000 7f8681da5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f867c070e40 0x7f867c0b05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:14.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.072+0000 7f8681da5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c14e340 con 0x7f867c06e200 2024-03-18T19:09:14.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.072+0000 7f8681da5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c0ad960 con 0x7f867c06e6b0 2024-03-18T19:09:14.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.072+0000 7f8681da5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f867c0adc40 con 0x7f867c070e40 2024-03-18T19:09:14.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.073+0000 7f867b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f867c06e200 0x7f867c14de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:14.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.073+0000 7f867b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f867c06e200 0x7f867c14de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37668/0 (socket says 172.21.15.1:37668) 2024-03-18T19:09:14.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.073+0000 7f867b7fe700 1 -- 172.21.15.1:0/2829586810 learned_addr learned my addr 172.21.15.1:0/2829586810 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:14.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.073+0000 7f867bfff700 1 --2- 172.21.15.1:0/2829586810 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f867c070e40 0x7f867c0b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:14.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.073+0000 7f867affd700 1 --2- 172.21.15.1:0/2829586810 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f867c06e6b0 0x7f867c14f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:14.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.073+0000 7f867b7fe700 1 -- 172.21.15.1:0/2829586810 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f867c070e40 msgr2=0x7f867c0b05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:14.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.073+0000 7f867b7fe700 1 --2- 172.21.15.1:0/2829586810 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f867c070e40 0x7f867c0b05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:14.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.073+0000 7f867b7fe700 1 -- 172.21.15.1:0/2829586810 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f867c06e6b0 msgr2=0x7f867c14f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:14.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.073+0000 7f867b7fe700 1 --2- 172.21.15.1:0/2829586810 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f867c06e6b0 0x7f867c14f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:14.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.073+0000 7f867b7fe700 1 -- 172.21.15.1:0/2829586810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8674009a70 con 0x7f867c06e200 2024-03-18T19:09:14.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.073+0000 7f867affd700 1 --2- 172.21.15.1:0/2829586810 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f867c06e6b0 0x7f867c14f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:14.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.073+0000 7f867b7fe700 1 --2- 172.21.15.1:0/2829586810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f867c06e200 0x7f867c14de00 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f866c00c520 tx=0x7f866c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:14.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.073+0000 7f8673fff700 1 -- 172.21.15.1:0/2829586810 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866c004c80 con 0x7f867c06e200 2024-03-18T19:09:14.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.073+0000 7f8673fff700 1 -- 172.21.15.1:0/2829586810 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866c004de0 con 0x7f867c06e200 2024-03-18T19:09:14.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.073+0000 7f8681da5700 1 -- 172.21.15.1:0/2829586810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f867c0adf20 con 0x7f867c06e200 2024-03-18T19:09:14.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.073+0000 7f8673fff700 1 -- 172.21.15.1:0/2829586810 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866c007310 con 0x7f867c06e200 2024-03-18T19:09:14.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.073+0000 7f8681da5700 1 -- 172.21.15.1:0/2829586810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f867c150270 con 0x7f867c06e200 2024-03-18T19:09:14.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.074+0000 7f8681da5700 1 -- 172.21.15.1:0/2829586810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f867c0676a0 con 0x7f867c06e200 2024-03-18T19:09:14.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.075+0000 7f8673fff700 1 -- 172.21.15.1:0/2829586810 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f866c007470 con 0x7f867c06e200 2024-03-18T19:09:14.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.075+0000 7f8673fff700 1 --2- 172.21.15.1:0/2829586810 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f865c06adb0 0x7f865c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:14.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.075+0000 7f8673fff700 1 -- 172.21.15.1:0/2829586810 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f866c089230 con 0x7f867c06e200 2024-03-18T19:09:14.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.075+0000 7f867affd700 1 --2- 172.21.15.1:0/2829586810 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f865c06adb0 0x7f865c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:14.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.076+0000 7f867affd700 1 --2- 172.21.15.1:0/2829586810 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f865c06adb0 0x7f865c06d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f867c15c460 tx=0x7f8664009040 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:14.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.080+0000 7f8673fff700 1 -- 172.21.15.1:0/2829586810 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f866c0541f0 con 0x7f867c06e200 2024-03-18T19:09:14.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.285+0000 7f8681da5700 1 -- 172.21.15.1:0/2829586810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f867c0d0700 con 0x7f867c06e200 2024-03-18T19:09:14.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.288+0000 7f8673fff700 1 -- 172.21.15.1:0/2829586810 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v45) v1 ==== 104+0+0 (secure 0 0 0) 0x7f866c057810 con 0x7f867c06e200 2024-03-18T19:09:14.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.290+0000 7f8681da5700 1 -- 172.21.15.1:0/2829586810 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f865c06adb0 msgr2=0x7f865c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:14.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.290+0000 7f8681da5700 1 --2- 172.21.15.1:0/2829586810 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f865c06adb0 0x7f865c06d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f867c15c460 tx=0x7f8664009040 comp rx=0 tx=0).stop 2024-03-18T19:09:14.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.290+0000 7f8681da5700 1 -- 172.21.15.1:0/2829586810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f867c06e200 msgr2=0x7f867c14de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:14.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.290+0000 7f8681da5700 1 --2- 172.21.15.1:0/2829586810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f867c06e200 0x7f867c14de00 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f866c00c520 tx=0x7f866c00c830 comp rx=0 tx=0).stop 2024-03-18T19:09:14.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.290+0000 7f8681da5700 1 -- 172.21.15.1:0/2829586810 shutdown_connections 2024-03-18T19:09:14.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.290+0000 7f8681da5700 1 --2- 172.21.15.1:0/2829586810 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f865c06adb0 0x7f865c06d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:14.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.290+0000 7f8681da5700 1 --2- 172.21.15.1:0/2829586810 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f867c070e40 0x7f867c0b05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:14.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.290+0000 7f8681da5700 1 --2- 172.21.15.1:0/2829586810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f867c06e200 0x7f867c14de00 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:14.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.290+0000 7f8681da5700 1 --2- 172.21.15.1:0/2829586810 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f867c06e6b0 0x7f867c14f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:14.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.290+0000 7f8681da5700 1 -- 172.21.15.1:0/2829586810 >> 172.21.15.1:0/2829586810 conn(0x7f867c06a750 msgr2=0x7f867c15a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:14.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.290+0000 7f8681da5700 1 -- 172.21.15.1:0/2829586810 shutdown_connections 2024-03-18T19:09:14.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.291+0000 7f8681da5700 1 -- 172.21.15.1:0/2829586810 wait complete. 2024-03-18T19:09:14.299 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2024-03-18T19:09:14.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.427+0000 7fa740bd5700 1 Processor -- start 2024-03-18T19:09:14.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.427+0000 7fa740bd5700 1 -- start start 2024-03-18T19:09:14.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.428+0000 7fa740bd5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa73c0af610 0x7fa73c0cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:14.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.428+0000 7fa740bd5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa73c0ccf40 0x7fa73c071a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:14.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.428+0000 7fa740bd5700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73c0cf090 con 0x7fa73c144b70 2024-03-18T19:09:14.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.428+0000 7fa740bd5700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa73c144f50 con 0x7fa73c0af610 2024-03-18T19:09:14.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.428+0000 7fa740bd5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa73c0cd3b0 con 0x7fa73c0ccf40 2024-03-18T19:09:14.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.429+0000 7fa739d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa73c0af610 0x7fa73c0cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:14.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.429+0000 7fa739d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa73c0af610 0x7fa73c0cca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43772/0 (socket says 172.21.15.1:43772) 2024-03-18T19:09:14.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.429+0000 7fa739d9b700 1 -- 172.21.15.1:0/588968755 learned_addr learned my addr 172.21.15.1:0/588968755 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:14.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.429+0000 7fa739d9b700 1 -- 172.21.15.1:0/588968755 >> v2:172.21.15.165:3300/0 conn(0x7fa73c0ccf40 msgr2=0x7fa73c071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:14.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.429+0000 7fa73ad9d700 1 --2- 172.21.15.1:0/588968755 >> v2:172.21.15.165:3300/0 conn(0x7fa73c0ccf40 0x7fa73c071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:14.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.429+0000 7fa739d9b700 1 --2- 172.21.15.1:0/588968755 >> v2:172.21.15.165:3300/0 conn(0x7fa73c0ccf40 0x7fa73c071a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:14.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.429+0000 7fa739d9b700 1 -- 172.21.15.1:0/588968755 >> v1:172.21.15.165:6789/0 conn(0x7fa73c144b70 legacy=0x7fa73c0ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:14.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.429+0000 7fa739d9b700 1 -- 172.21.15.1:0/588968755 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73c0cf2d0 con 0x7fa73c0af610 2024-03-18T19:09:14.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.429+0000 7fa73ad9d700 1 --2- 172.21.15.1:0/588968755 >> v2:172.21.15.165:3300/0 conn(0x7fa73c0ccf40 0x7fa73c071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:14.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.429+0000 7fa739d9b700 1 --2- 172.21.15.1:0/588968755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa73c0af610 0x7fa73c0cca00 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fa73c0715e0 tx=0x7fa724009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91b62ac0568b3e27 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:14.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.429+0000 7fa73959a700 1 -- 172.21.15.1:0/588968755 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa724004230 con 0x7fa73c0af610 2024-03-18T19:09:14.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.429+0000 7fa73959a700 1 -- 172.21.15.1:0/588968755 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa724004390 con 0x7fa73c0af610 2024-03-18T19:09:14.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.429+0000 7fa73959a700 1 -- 172.21.15.1:0/588968755 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7240036a0 con 0x7fa73c0af610 2024-03-18T19:09:14.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.430+0000 7fa740bd5700 1 -- 172.21.15.1:0/588968755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa73c0af610 msgr2=0x7fa73c0cca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:14.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.430+0000 7fa740bd5700 1 --2- 172.21.15.1:0/588968755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa73c0af610 0x7fa73c0cca00 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fa73c0715e0 tx=0x7fa724009bd0 comp rx=0 tx=0).stop 2024-03-18T19:09:14.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.430+0000 7fa740bd5700 1 -- 172.21.15.1:0/588968755 shutdown_connections 2024-03-18T19:09:14.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.430+0000 7fa740bd5700 1 --2- 172.21.15.1:0/588968755 >> v2:172.21.15.165:3300/0 conn(0x7fa73c0ccf40 0x7fa73c071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:14.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.430+0000 7fa740bd5700 1 --2- 172.21.15.1:0/588968755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa73c0af610 0x7fa73c0cca00 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:14.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.430+0000 7fa740bd5700 1 -- 172.21.15.1:0/588968755 >> 172.21.15.1:0/588968755 conn(0x7fa73c0b1d20 msgr2=0x7fa73c0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:14.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.430+0000 7fa740bd5700 1 -- 172.21.15.1:0/588968755 shutdown_connections 2024-03-18T19:09:14.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.430+0000 7fa740bd5700 1 -- 172.21.15.1:0/588968755 wait complete. 2024-03-18T19:09:14.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.432+0000 7fa740bd5700 1 Processor -- start 2024-03-18T19:09:14.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.432+0000 7fa740bd5700 1 -- start start 2024-03-18T19:09:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.432+0000 7fa740bd5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa73c0af610 0x7fa73c1f1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.432+0000 7fa740bd5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa73c0ccf40 0x7fa73c157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa740bd5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa73c144b70 0x7fa73c13d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa740bd5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73c1f1ac0 con 0x7fa73c144b70 2024-03-18T19:09:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa740bd5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73c158900 con 0x7fa73c0ccf40 2024-03-18T19:09:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa740bd5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa73c158be0 con 0x7fa73c0af610 2024-03-18T19:09:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa73ad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa73c144b70 0x7fa73c13d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa73ad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa73c144b70 0x7fa73c13d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43784/0 (socket says 172.21.15.1:43784) 2024-03-18T19:09:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa73ad9d700 1 -- 172.21.15.1:0/2949293800 learned_addr learned my addr 172.21.15.1:0/2949293800 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa739d9b700 1 --2- 172.21.15.1:0/2949293800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa73c0ccf40 0x7fa73c157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:14.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa73a59c700 1 --2- 172.21.15.1:0/2949293800 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa73c0af610 0x7fa73c1f1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa73ad9d700 1 -- 172.21.15.1:0/2949293800 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa73c0af610 msgr2=0x7fa73c1f1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa73ad9d700 1 --2- 172.21.15.1:0/2949293800 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa73c0af610 0x7fa73c1f1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa73ad9d700 1 -- 172.21.15.1:0/2949293800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa73c0ccf40 msgr2=0x7fa73c157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa73ad9d700 1 --2- 172.21.15.1:0/2949293800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa73c0ccf40 0x7fa73c157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa73ad9d700 1 -- 172.21.15.1:0/2949293800 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa724009550 con 0x7fa73c144b70 2024-03-18T19:09:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa739d9b700 1 --2- 172.21.15.1:0/2949293800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa73c0ccf40 0x7fa73c157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa73a59c700 1 --2- 172.21.15.1:0/2949293800 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa73c0af610 0x7fa73c1f1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa73ad9d700 1 --2- 172.21.15.1:0/2949293800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa73c144b70 0x7fa73c13d080 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fa73400cef0 tx=0x7fa73400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa732ffd700 1 -- 172.21.15.1:0/2949293800 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73400ae50 con 0x7fa73c144b70 2024-03-18T19:09:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa740bd5700 1 -- 172.21.15.1:0/2949293800 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa73c158ec0 con 0x7fa73c144b70 2024-03-18T19:09:14.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa732ffd700 1 -- 172.21.15.1:0/2949293800 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa734015440 con 0x7fa73c144b70 2024-03-18T19:09:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa732ffd700 1 -- 172.21.15.1:0/2949293800 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7340157f0 con 0x7fa73c144b70 2024-03-18T19:09:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.433+0000 7fa740bd5700 1 -- 172.21.15.1:0/2949293800 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa73c158510 con 0x7fa73c144b70 2024-03-18T19:09:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.434+0000 7fa740bd5700 1 -- 172.21.15.1:0/2949293800 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa73c0cda90 con 0x7fa73c144b70 2024-03-18T19:09:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.435+0000 7fa732ffd700 1 -- 172.21.15.1:0/2949293800 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa73400d430 con 0x7fa73c144b70 2024-03-18T19:09:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.435+0000 7fa732ffd700 1 --2- 172.21.15.1:0/2949293800 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa71c06adb0 0x7fa71c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.435+0000 7fa73a59c700 1 --2- 172.21.15.1:0/2949293800 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa71c06adb0 0x7fa71c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.436+0000 7fa732ffd700 1 -- 172.21.15.1:0/2949293800 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fa734089bd0 con 0x7fa73c144b70 2024-03-18T19:09:14.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.436+0000 7fa73a59c700 1 --2- 172.21.15.1:0/2949293800 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa71c06adb0 0x7fa71c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009930 tx=0x7fa72c008040 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:14.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.440+0000 7fa732ffd700 1 -- 172.21.15.1:0/2949293800 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa734054ba0 con 0x7fa73c144b70 2024-03-18T19:09:14.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:14.729+0000 7fa740bd5700 1 -- 172.21.15.1:0/2949293800 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7fa73c0b6310 con 0x7fa73c144b70 2024-03-18T19:09:15.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.175+0000 7fa732ffd700 1 -- 172.21.15.1:0/2949293800 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v738) v1 ==== 148+0+0 (secure 0 0 0) 0x7fa7340581c0 con 0x7fa73c144b70 2024-03-18T19:09:15.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.177+0000 7fa740bd5700 1 -- 172.21.15.1:0/2949293800 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa71c06adb0 msgr2=0x7fa71c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:15.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.177+0000 7fa740bd5700 1 --2- 172.21.15.1:0/2949293800 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa71c06adb0 0x7fa71c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009930 tx=0x7fa72c008040 comp rx=0 tx=0).stop 2024-03-18T19:09:15.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.177+0000 7fa740bd5700 1 -- 172.21.15.1:0/2949293800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa73c144b70 msgr2=0x7fa73c13d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:15.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.177+0000 7fa740bd5700 1 --2- 172.21.15.1:0/2949293800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa73c144b70 0x7fa73c13d080 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fa73400cef0 tx=0x7fa73400a340 comp rx=0 tx=0).stop 2024-03-18T19:09:15.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.178+0000 7fa740bd5700 1 -- 172.21.15.1:0/2949293800 shutdown_connections 2024-03-18T19:09:15.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.178+0000 7fa740bd5700 1 --2- 172.21.15.1:0/2949293800 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa71c06adb0 0x7fa71c06d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:15.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.178+0000 7fa740bd5700 1 --2- 172.21.15.1:0/2949293800 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa73c0af610 0x7fa73c1f1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:15.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.178+0000 7fa740bd5700 1 --2- 172.21.15.1:0/2949293800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa73c144b70 0x7fa73c13d080 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:15.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.178+0000 7fa740bd5700 1 --2- 172.21.15.1:0/2949293800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa73c0ccf40 0x7fa73c157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:15.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.178+0000 7fa740bd5700 1 -- 172.21.15.1:0/2949293800 >> 172.21.15.1:0/2949293800 conn(0x7fa73c0b1d20 msgr2=0x7fa73c15ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:15.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.178+0000 7fa740bd5700 1 -- 172.21.15.1:0/2949293800 shutdown_connections 2024-03-18T19:09:15.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.178+0000 7fa740bd5700 1 -- 172.21.15.1:0/2949293800 wait complete. 2024-03-18T19:09:15.189 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2024-03-18T19:09:15.189 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-18T19:09:15.189 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2024-03-18T19:09:15.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.324+0000 7fa36db7c700 1 Processor -- start 2024-03-18T19:09:15.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.324+0000 7fa36db7c700 1 -- start start 2024-03-18T19:09:15.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.325+0000 7fa36db7c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa368144c30 0x7fa3680ce980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:15.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.325+0000 7fa36db7c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa368071650 con 0x7fa3680ceec0 2024-03-18T19:09:15.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.325+0000 7fa36db7c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3680cf380 con 0x7fa3680cd110 2024-03-18T19:09:15.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.325+0000 7fa36db7c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3680cf660 con 0x7fa368144c30 2024-03-18T19:09:15.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.325+0000 7fa366ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa3680ceec0 0x7fa3680cca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48304/0 (socket says 172.21.15.1:48304) 2024-03-18T19:09:15.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.325+0000 7fa366ffd700 1 -- 172.21.15.1:0/187564286 learned_addr learned my addr 172.21.15.1:0/187564286 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:15.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.325+0000 7fa3677fe700 1 --2- 172.21.15.1:0/187564286 >> v2:172.21.15.165:3300/0 conn(0x7fa368144c30 0x7fa3680ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:15.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.325+0000 7fa3667fc700 1 -- 172.21.15.1:0/187564286 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170194293 0 0) 0x7fa368071650 con 0x7fa3680ceec0 2024-03-18T19:09:15.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.325+0000 7fa3667fc700 1 -- 172.21.15.1:0/187564286 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3540035b0 con 0x7fa3680ceec0 2024-03-18T19:09:15.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.325+0000 7fa3667fc700 1 -- 172.21.15.1:0/187564286 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979609363 0 0) 0x7fa3680cf380 con 0x7fa3680cd110 2024-03-18T19:09:15.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.325+0000 7fa3667fc700 1 -- 172.21.15.1:0/187564286 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa368071650 con 0x7fa3680cd110 2024-03-18T19:09:15.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.325+0000 7fa3667fc700 1 -- 172.21.15.1:0/187564286 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3709155178 0 0) 0x7fa3540035b0 con 0x7fa3680ceec0 2024-03-18T19:09:15.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa3667fc700 1 -- 172.21.15.1:0/187564286 >> v2:172.21.15.165:3300/0 conn(0x7fa368144c30 msgr2=0x7fa3680ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:15.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa3667fc700 1 --2- 172.21.15.1:0/187564286 >> v2:172.21.15.165:3300/0 conn(0x7fa368144c30 0x7fa3680ce980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:15.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa3667fc700 1 -- 172.21.15.1:0/187564286 >> v1:172.21.15.165:6789/0 conn(0x7fa3680cd110 legacy=0x7fa368071ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:15.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa3667fc700 1 -- 172.21.15.1:0/187564286 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3680cf940 con 0x7fa3680ceec0 2024-03-18T19:09:15.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa3667fc700 1 -- 172.21.15.1:0/187564286 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fa350001670 con 0x7fa3680ceec0 2024-03-18T19:09:15.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa3667fc700 1 -- 172.21.15.1:0/187564286 >> v1:172.21.15.1:6789/0 conn(0x7fa3680ceec0 legacy=0x7fa3680cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:15.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa3667fc700 1 --2- 172.21.15.1:0/187564286 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa354003b90 0x7fa354003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:15.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa3667fc700 1 --2- 172.21.15.1:0/187564286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa354004850 0x7fa354004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:15.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa3667fc700 1 --2- 172.21.15.1:0/187564286 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa354005210 0x7fa35400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:15.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa3667fc700 1 -- 172.21.15.1:0/187564286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3680cf660 con 0x7fa354004850 2024-03-18T19:09:15.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa3667fc700 1 -- 172.21.15.1:0/187564286 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35400c090 con 0x7fa354005210 2024-03-18T19:09:15.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa3667fc700 1 -- 172.21.15.1:0/187564286 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa35400c250 con 0x7fa354003b90 2024-03-18T19:09:15.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa366ffd700 1 --2- 172.21.15.1:0/187564286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa354004850 0x7fa354004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:15.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa367fff700 1 --2- 172.21.15.1:0/187564286 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa354005210 0x7fa35400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:15.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa3677fe700 1 --2- 172.21.15.1:0/187564286 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa354003b90 0x7fa354003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:15.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa366ffd700 1 -- 172.21.15.1:0/187564286 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa354003b90 msgr2=0x7fa354003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:15.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa366ffd700 1 --2- 172.21.15.1:0/187564286 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa354003b90 0x7fa354003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:15.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa366ffd700 1 -- 172.21.15.1:0/187564286 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa354005210 msgr2=0x7fa35400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:15.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa366ffd700 1 --2- 172.21.15.1:0/187564286 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa354005210 0x7fa35400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:15.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa366ffd700 1 -- 172.21.15.1:0/187564286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3540035b0 con 0x7fa354004850 2024-03-18T19:09:15.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa367fff700 1 -- 172.21.15.1:0/187564286 reap_dead start 2024-03-18T19:09:15.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa366ffd700 1 --2- 172.21.15.1:0/187564286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa354004850 0x7fa354004cd0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fa35000c210 tx=0x7fa354005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e322a98e651999 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:15.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa3667fc700 1 -- 172.21.15.1:0/187564286 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35000cd50 con 0x7fa354004850 2024-03-18T19:09:15.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa3667fc700 1 -- 172.21.15.1:0/187564286 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa350005960 con 0x7fa354004850 2024-03-18T19:09:15.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.326+0000 7fa3667fc700 1 -- 172.21.15.1:0/187564286 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa350005d10 con 0x7fa354004850 2024-03-18T19:09:15.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.327+0000 7fa36db7c700 1 -- 172.21.15.1:0/187564286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa354004850 msgr2=0x7fa354004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:15.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.327+0000 7fa36db7c700 1 --2- 172.21.15.1:0/187564286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa354004850 0x7fa354004cd0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fa35000c210 tx=0x7fa354005880 comp rx=0 tx=0).stop 2024-03-18T19:09:15.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.327+0000 7fa36db7c700 1 -- 172.21.15.1:0/187564286 shutdown_connections 2024-03-18T19:09:15.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.327+0000 7fa36db7c700 1 --2- 172.21.15.1:0/187564286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa354004850 0x7fa354004cd0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:15.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.327+0000 7fa36db7c700 1 -- 172.21.15.1:0/187564286 >> 172.21.15.1:0/187564286 conn(0x7fa3680b1d20 msgr2=0x7fa3680b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:15.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.327+0000 7fa36db7c700 1 -- 172.21.15.1:0/187564286 shutdown_connections 2024-03-18T19:09:15.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.327+0000 7fa36db7c700 1 -- 172.21.15.1:0/187564286 wait complete. 2024-03-18T19:09:15.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.328+0000 7fa36db7c700 1 Processor -- start 2024-03-18T19:09:15.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.328+0000 7fa36db7c700 1 -- start start 2024-03-18T19:09:15.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.329+0000 7fa36db7c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa354004850 0x7fa368135f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:15.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.329+0000 7fa36db7c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa368142bc0 0x7fa368143040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:15.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.329+0000 7fa36db7c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa368136740 0x7fa368136bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:15.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.329+0000 7fa36db7c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa368137130 con 0x7fa368136740 2024-03-18T19:09:15.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.329+0000 7fa36db7c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa368143c40 con 0x7fa354004850 2024-03-18T19:09:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.329+0000 7fa36db7c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa36815cab0 con 0x7fa368142bc0 2024-03-18T19:09:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.329+0000 7fa367fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa368136740 0x7fa368136bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.329+0000 7fa367fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa368136740 0x7fa368136bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43802/0 (socket says 172.21.15.1:43802) 2024-03-18T19:09:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.329+0000 7fa367fff700 1 -- 172.21.15.1:0/2671685949 learned_addr learned my addr 172.21.15.1:0/2671685949 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.329+0000 7fa3677fe700 1 --2- 172.21.15.1:0/2671685949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa354004850 0x7fa368135f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.329+0000 7fa366ffd700 1 --2- 172.21.15.1:0/2671685949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa368142bc0 0x7fa368143040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.330+0000 7fa367fff700 1 -- 172.21.15.1:0/2671685949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa368142bc0 msgr2=0x7fa368143040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.330+0000 7fa367fff700 1 --2- 172.21.15.1:0/2671685949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa368142bc0 0x7fa368143040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.330+0000 7fa367fff700 1 -- 172.21.15.1:0/2671685949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa354004850 msgr2=0x7fa368135f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.330+0000 7fa367fff700 1 --2- 172.21.15.1:0/2671685949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa354004850 0x7fa368135f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.330+0000 7fa367fff700 1 -- 172.21.15.1:0/2671685949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa368143680 con 0x7fa368136740 2024-03-18T19:09:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.330+0000 7fa366ffd700 1 --2- 172.21.15.1:0/2671685949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa368142bc0 0x7fa368143040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.330+0000 7fa367fff700 1 --2- 172.21.15.1:0/2671685949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa368136740 0x7fa368136bf0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fa3680b5620 tx=0x7fa36000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:15.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.330+0000 7fa35ffff700 1 -- 172.21.15.1:0/2671685949 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa360003c10 con 0x7fa368136740 2024-03-18T19:09:15.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.330+0000 7fa35ffff700 1 -- 172.21.15.1:0/2671685949 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa360003d70 con 0x7fa368136740 2024-03-18T19:09:15.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.330+0000 7fa36db7c700 1 -- 172.21.15.1:0/2671685949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa368143960 con 0x7fa368136740 2024-03-18T19:09:15.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.330+0000 7fa35ffff700 1 -- 172.21.15.1:0/2671685949 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36000daf0 con 0x7fa368136740 2024-03-18T19:09:15.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.330+0000 7fa36db7c700 1 -- 172.21.15.1:0/2671685949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa368137440 con 0x7fa368136740 2024-03-18T19:09:15.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.331+0000 7fa36db7c700 1 -- 172.21.15.1:0/2671685949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa36815c5a0 con 0x7fa368136740 2024-03-18T19:09:15.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.331+0000 7fa35ffff700 1 -- 172.21.15.1:0/2671685949 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa36000dc50 con 0x7fa368136740 2024-03-18T19:09:15.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.332+0000 7fa35ffff700 1 --2- 172.21.15.1:0/2671685949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa34806adb0 0x7fa34806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:15.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.332+0000 7fa35ffff700 1 -- 172.21.15.1:0/2671685949 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fa360089750 con 0x7fa368136740 2024-03-18T19:09:15.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.332+0000 7fa3677fe700 1 --2- 172.21.15.1:0/2671685949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa34806adb0 0x7fa34806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:15.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.332+0000 7fa3677fe700 1 --2- 172.21.15.1:0/2671685949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa34806adb0 0x7fa34806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa358009b80 tx=0x7fa358009210 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:15.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.336+0000 7fa35ffff700 1 -- 172.21.15.1:0/2671685949 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa36815c5a0 con 0x7fa368136740 2024-03-18T19:09:15.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:15.625+0000 7fa36db7c700 1 -- 172.21.15.1:0/2671685949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7fa36815c5a0 con 0x7fa368136740 2024-03-18T19:09:16.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.176+0000 7fa35ffff700 1 -- 172.21.15.1:0/2671685949 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v739) v1 ==== 163+0+0 (secure 0 0 0) 0x7fa36815c5a0 con 0x7fa368136740 2024-03-18T19:09:16.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.179+0000 7fa36db7c700 1 -- 172.21.15.1:0/2671685949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa34806adb0 msgr2=0x7fa34806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:16.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.179+0000 7fa36db7c700 1 --2- 172.21.15.1:0/2671685949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa34806adb0 0x7fa34806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa358009b80 tx=0x7fa358009210 comp rx=0 tx=0).stop 2024-03-18T19:09:16.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.179+0000 7fa36db7c700 1 -- 172.21.15.1:0/2671685949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa368136740 msgr2=0x7fa368136bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:16.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.179+0000 7fa36db7c700 1 --2- 172.21.15.1:0/2671685949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa368136740 0x7fa368136bf0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fa3680b5620 tx=0x7fa36000d430 comp rx=0 tx=0).stop 2024-03-18T19:09:16.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.179+0000 7fa36db7c700 1 -- 172.21.15.1:0/2671685949 shutdown_connections 2024-03-18T19:09:16.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.179+0000 7fa36db7c700 1 --2- 172.21.15.1:0/2671685949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa34806adb0 0x7fa34806d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.179+0000 7fa36db7c700 1 --2- 172.21.15.1:0/2671685949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa368136740 0x7fa368136bf0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.179+0000 7fa36db7c700 1 --2- 172.21.15.1:0/2671685949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa354004850 0x7fa368135f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.179+0000 7fa36db7c700 1 --2- 172.21.15.1:0/2671685949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa368142bc0 0x7fa368143040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.179+0000 7fa36db7c700 1 -- 172.21.15.1:0/2671685949 >> 172.21.15.1:0/2671685949 conn(0x7fa3680b1d20 msgr2=0x7fa3680b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:16.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.179+0000 7fa36db7c700 1 -- 172.21.15.1:0/2671685949 shutdown_connections 2024-03-18T19:09:16.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.179+0000 7fa36db7c700 1 -- 172.21.15.1:0/2671685949 wait complete. 2024-03-18T19:09:16.191 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-18T19:09:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.330+0000 7f01ae683700 1 Processor -- start 2024-03-18T19:09:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.330+0000 7f01ae683700 1 -- start start 2024-03-18T19:09:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.330+0000 7f01ae683700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01a815a1d0 0x7f01a80add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.330+0000 7f01ae683700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f01a80cc0f0 0x7f01a8071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.330+0000 7f01ae683700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a80ce8f0 con 0x7f01a8071620 2024-03-18T19:09:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.330+0000 7f01ae683700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a80cdb00 con 0x7f01a80cc0f0 2024-03-18T19:09:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.330+0000 7f01ae683700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a815a5b0 con 0x7f01a815a1d0 2024-03-18T19:09:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.330+0000 7f01a77fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f01a8071620 0x7f01a80cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48312/0 (socket says 172.21.15.1:48312) 2024-03-18T19:09:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.330+0000 7f01a77fe700 1 -- 172.21.15.1:0/3501966318 learned_addr learned my addr 172.21.15.1:0/3501966318 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.330+0000 7f01acc20700 1 --2- 172.21.15.1:0/3501966318 >> v2:172.21.15.165:3300/0 conn(0x7f01a80cc0f0 0x7f01a8071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.330+0000 7f01a7fff700 1 --2- 172.21.15.1:0/3501966318 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01a815a1d0 0x7f01a80add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.330+0000 7f01a6ffd700 1 -- 172.21.15.1:0/3501966318 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214768141 0 0) 0x7f01a80ce8f0 con 0x7f01a8071620 2024-03-18T19:09:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.330+0000 7f01a6ffd700 1 -- 172.21.15.1:0/3501966318 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01940035b0 con 0x7f01a8071620 2024-03-18T19:09:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01a6ffd700 1 -- 172.21.15.1:0/3501966318 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2345198749 0 0) 0x7f01940035b0 con 0x7f01a8071620 2024-03-18T19:09:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01a6ffd700 1 -- 172.21.15.1:0/3501966318 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01a815a1d0 msgr2=0x7f01a80add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01a6ffd700 1 --2- 172.21.15.1:0/3501966318 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01a815a1d0 0x7f01a80add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01a6ffd700 1 -- 172.21.15.1:0/3501966318 >> v2:172.21.15.165:3300/0 conn(0x7f01a80cc0f0 msgr2=0x7f01a8071aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01a6ffd700 1 --2- 172.21.15.1:0/3501966318 >> v2:172.21.15.165:3300/0 conn(0x7f01a80cc0f0 0x7f01a8071aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01a6ffd700 1 -- 172.21.15.1:0/3501966318 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a80ae260 con 0x7f01a8071620 2024-03-18T19:09:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01a6ffd700 1 -- 172.21.15.1:0/3501966318 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f0190001670 con 0x7f01a8071620 2024-03-18T19:09:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01acc20700 1 --2- 172.21.15.1:0/3501966318 >> v2:172.21.15.165:3300/0 conn(0x7f01a80cc0f0 0x7f01a8071aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01a6ffd700 1 -- 172.21.15.1:0/3501966318 >> v1:172.21.15.1:6789/0 conn(0x7f01a8071620 legacy=0x7f01a80cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01a6ffd700 1 --2- 172.21.15.1:0/3501966318 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0194003a00 0x7f0194003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01a6ffd700 1 --2- 172.21.15.1:0/3501966318 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01a815a1d0 0x7f0194004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01a6ffd700 1 --2- 172.21.15.1:0/3501966318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0194004d90 0x7f0194005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01a6ffd700 1 -- 172.21.15.1:0/3501966318 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80cdb00 con 0x7f0194004d90 2024-03-18T19:09:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01a6ffd700 1 -- 172.21.15.1:0/3501966318 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019400ba10 con 0x7f0194003a00 2024-03-18T19:09:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01a6ffd700 1 -- 172.21.15.1:0/3501966318 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019400bb50 con 0x7f01a815a1d0 2024-03-18T19:09:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01acc20700 1 --2- 172.21.15.1:0/3501966318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0194004d90 0x7f0194005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01a7fff700 1 --2- 172.21.15.1:0/3501966318 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0194003a00 0x7f0194003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01a77fe700 1 --2- 172.21.15.1:0/3501966318 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01a815a1d0 0x7f0194004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01acc20700 1 -- 172.21.15.1:0/3501966318 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01a815a1d0 msgr2=0x7f0194004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01acc20700 1 --2- 172.21.15.1:0/3501966318 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01a815a1d0 0x7f0194004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01acc20700 1 -- 172.21.15.1:0/3501966318 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0194003a00 msgr2=0x7f0194003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01acc20700 1 --2- 172.21.15.1:0/3501966318 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0194003a00 0x7f0194003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01acc20700 1 -- 172.21.15.1:0/3501966318 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01940035b0 con 0x7f0194004d90 2024-03-18T19:09:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01acc20700 1 --2- 172.21.15.1:0/3501966318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0194004d90 0x7f0194005210 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f01a000e320 tx=0x7f0194005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc13ea688ee0cff server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.331+0000 7f01a6ffd700 1 -- 172.21.15.1:0/3501966318 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a000edc0 con 0x7f0194004d90 2024-03-18T19:09:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.332+0000 7f01a6ffd700 1 -- 172.21.15.1:0/3501966318 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a000f040 con 0x7f0194004d90 2024-03-18T19:09:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.332+0000 7f01a6ffd700 1 -- 172.21.15.1:0/3501966318 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a00146a0 con 0x7f0194004d90 2024-03-18T19:09:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.332+0000 7f01ae683700 1 -- 172.21.15.1:0/3501966318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0194004d90 msgr2=0x7f0194005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.332+0000 7f01ae683700 1 --2- 172.21.15.1:0/3501966318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0194004d90 0x7f0194005210 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f01a000e320 tx=0x7f0194005880 comp rx=0 tx=0).stop 2024-03-18T19:09:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.332+0000 7f01acc20700 1 -- 172.21.15.1:0/3501966318 reap_dead start 2024-03-18T19:09:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.332+0000 7f01ae683700 1 -- 172.21.15.1:0/3501966318 shutdown_connections 2024-03-18T19:09:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.332+0000 7f01ae683700 1 -- 172.21.15.1:0/3501966318 >> 172.21.15.1:0/3501966318 conn(0x7f01a80cc9b0 msgr2=0x7f01a80ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.332+0000 7f01ae683700 1 -- 172.21.15.1:0/3501966318 shutdown_connections 2024-03-18T19:09:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.332+0000 7f01ae683700 1 -- 172.21.15.1:0/3501966318 wait complete. 2024-03-18T19:09:16.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.334+0000 7f01ae683700 1 Processor -- start 2024-03-18T19:09:16.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.334+0000 7f01ae683700 1 -- start start 2024-03-18T19:09:16.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.335+0000 7f01ae683700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01a8157c10 0x7f01a8155d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:16.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.335+0000 7f01ae683700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a8157ff0 0x7f01a80c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:16.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.335+0000 7f01ae683700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01a80c7540 0x7f01a80ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:16.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.335+0000 7f01ae683700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80c79a0 con 0x7f01a8157ff0 2024-03-18T19:09:16.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.335+0000 7f01ae683700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80c32c0 con 0x7f01a8157c10 2024-03-18T19:09:16.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.335+0000 7f01ae683700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80c9570 con 0x7f01a80c7540 2024-03-18T19:09:16.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.335+0000 7f01a77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a8157ff0 0x7f01a80c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:16.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.335+0000 7f01a77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a8157ff0 0x7f01a80c7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43820/0 (socket says 172.21.15.1:43820) 2024-03-18T19:09:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.335+0000 7f01a77fe700 1 -- 172.21.15.1:0/3686983848 learned_addr learned my addr 172.21.15.1:0/3686983848 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.335+0000 7f01a7fff700 1 --2- 172.21.15.1:0/3686983848 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01a8157c10 0x7f01a8155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.335+0000 7f01acc20700 1 --2- 172.21.15.1:0/3686983848 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01a80c7540 0x7f01a80ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.335+0000 7f01a77fe700 1 -- 172.21.15.1:0/3686983848 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01a80c7540 msgr2=0x7f01a80ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.335+0000 7f01a77fe700 1 --2- 172.21.15.1:0/3686983848 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01a80c7540 0x7f01a80ca330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.335+0000 7f01a77fe700 1 -- 172.21.15.1:0/3686983848 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01a8157c10 msgr2=0x7f01a8155d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.335+0000 7f01a77fe700 1 --2- 172.21.15.1:0/3686983848 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01a8157c10 0x7f01a8155d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.335+0000 7f01a77fe700 1 -- 172.21.15.1:0/3686983848 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a80c9850 con 0x7f01a8157ff0 2024-03-18T19:09:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.335+0000 7f01a77fe700 1 --2- 172.21.15.1:0/3686983848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a8157ff0 0x7f01a80c7000 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f01900017a0 tx=0x7f01900076a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.335+0000 7f01a4ff9700 1 -- 172.21.15.1:0/3686983848 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019000f8d0 con 0x7f01a8157ff0 2024-03-18T19:09:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.336+0000 7f01a4ff9700 1 -- 172.21.15.1:0/3686983848 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019000fa50 con 0x7f01a8157ff0 2024-03-18T19:09:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.336+0000 7f01a4ff9700 1 -- 172.21.15.1:0/3686983848 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019000fe00 con 0x7f01a8157ff0 2024-03-18T19:09:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.336+0000 7f01ae683700 1 -- 172.21.15.1:0/3686983848 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a80c9b30 con 0x7f01a8157ff0 2024-03-18T19:09:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.336+0000 7f01ae683700 1 -- 172.21.15.1:0/3686983848 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a80c9260 con 0x7f01a8157ff0 2024-03-18T19:09:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.337+0000 7f01ae683700 1 -- 172.21.15.1:0/3686983848 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a80ca050 con 0x7f01a8157ff0 2024-03-18T19:09:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.337+0000 7f01a4ff9700 1 -- 172.21.15.1:0/3686983848 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f019001d460 con 0x7f01a8157ff0 2024-03-18T19:09:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.338+0000 7f01a4ff9700 1 --2- 172.21.15.1:0/3686983848 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f018806adb0 0x7f018806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.338+0000 7f01a4ff9700 1 -- 172.21.15.1:0/3686983848 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f0190089690 con 0x7f01a8157ff0 2024-03-18T19:09:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.338+0000 7f01a7fff700 1 --2- 172.21.15.1:0/3686983848 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f018806adb0 0x7f018806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.338+0000 7f01a7fff700 1 --2- 172.21.15.1:0/3686983848 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f018806adb0 0x7f018806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0198009b80 tx=0x7f0198008040 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:16.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.342+0000 7f01a4ff9700 1 -- 172.21.15.1:0/3686983848 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f01a80ca050 con 0x7f01a8157ff0 2024-03-18T19:09:16.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.545+0000 7f01ae683700 1 -- 172.21.15.1:0/3686983848 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f01a80ca050 con 0x7f01a8157ff0 2024-03-18T19:09:16.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.549+0000 7f01a4ff9700 1 -- 172.21.15.1:0/3686983848 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7f01a80ca050 con 0x7f01a8157ff0 2024-03-18T19:09:16.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.551+0000 7f01ae683700 1 -- 172.21.15.1:0/3686983848 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f018806adb0 msgr2=0x7f018806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:16.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.551+0000 7f01ae683700 1 --2- 172.21.15.1:0/3686983848 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f018806adb0 0x7f018806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0198009b80 tx=0x7f0198008040 comp rx=0 tx=0).stop 2024-03-18T19:09:16.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.551+0000 7f01ae683700 1 -- 172.21.15.1:0/3686983848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a8157ff0 msgr2=0x7f01a80c7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:16.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.551+0000 7f01ae683700 1 --2- 172.21.15.1:0/3686983848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a8157ff0 0x7f01a80c7000 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f01900017a0 tx=0x7f01900076a0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.551+0000 7f01ae683700 1 -- 172.21.15.1:0/3686983848 shutdown_connections 2024-03-18T19:09:16.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.551+0000 7f01ae683700 1 --2- 172.21.15.1:0/3686983848 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f018806adb0 0x7f018806d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.551+0000 7f01ae683700 1 --2- 172.21.15.1:0/3686983848 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01a80c7540 0x7f01a80ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.551+0000 7f01ae683700 1 --2- 172.21.15.1:0/3686983848 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01a8157c10 0x7f01a8155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.551+0000 7f01ae683700 1 --2- 172.21.15.1:0/3686983848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01a8157ff0 0x7f01a80c7000 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.551+0000 7f01ae683700 1 -- 172.21.15.1:0/3686983848 >> 172.21.15.1:0/3686983848 conn(0x7f01a80cc9b0 msgr2=0x7f01a80aec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:16.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.551+0000 7f01ae683700 1 -- 172.21.15.1:0/3686983848 shutdown_connections 2024-03-18T19:09:16.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.551+0000 7f01ae683700 1 -- 172.21.15.1:0/3686983848 wait complete. 2024-03-18T19:09:16.560 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T19:09:16.560 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:09:16.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:09:16.652 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T19:09:16.653 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:09:16.653 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:09:16.671 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:09:16.671 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:09:16.671 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:09:16.692 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:09:16.692 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:09:16.692 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:09:16.731 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:09:16.731 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:09:16.731 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:09:16.752 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:09:16.752 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-18T19:09:16.752 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:09:16.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.898+0000 7fbf298c4700 1 Processor -- start 2024-03-18T19:09:16.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.898+0000 7fbf298c4700 1 -- start start 2024-03-18T19:09:16.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.899+0000 7fbf298c4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbf240b6b60 0x7fbf240cc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:16.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.899+0000 7fbf298c4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf240cee90 0x7fbf240cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:16.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.899+0000 7fbf298c4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf240cfbd0 0x7fbf24099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:16.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.899+0000 7fbf298c4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf240d0060 con 0x7fbf240cee90 2024-03-18T19:09:16.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.899+0000 7fbf298c4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf240cd0d0 con 0x7fbf240b6b60 2024-03-18T19:09:16.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.899+0000 7fbf298c4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf240b3f00 con 0x7fbf240cfbd0 2024-03-18T19:09:16.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.899+0000 7fbf227fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf240cee90 0x7fbf240cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:16.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.899+0000 7fbf227fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf240cee90 0x7fbf240cf690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43826/0 (socket says 172.21.15.1:43826) 2024-03-18T19:09:16.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.899+0000 7fbf227fc700 1 -- 172.21.15.1:0/114543386 learned_addr learned my addr 172.21.15.1:0/114543386 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:16.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf22ffd700 1 --2- 172.21.15.1:0/114543386 >> v2:172.21.15.165:3300/0 conn(0x7fbf240b6b60 0x7fbf240cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:16.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf237fe700 1 --2- 172.21.15.1:0/114543386 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf240cfbd0 0x7fbf24099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:16.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf227fc700 1 -- 172.21.15.1:0/114543386 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf240cfbd0 msgr2=0x7fbf24099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:16.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf227fc700 1 --2- 172.21.15.1:0/114543386 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf240cfbd0 0x7fbf24099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf227fc700 1 -- 172.21.15.1:0/114543386 >> v2:172.21.15.165:3300/0 conn(0x7fbf240b6b60 msgr2=0x7fbf240cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:16.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf227fc700 1 --2- 172.21.15.1:0/114543386 >> v2:172.21.15.165:3300/0 conn(0x7fbf240b6b60 0x7fbf240cc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf227fc700 1 -- 172.21.15.1:0/114543386 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf240b41e0 con 0x7fbf240cee90 2024-03-18T19:09:16.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf22ffd700 1 --2- 172.21.15.1:0/114543386 >> v2:172.21.15.165:3300/0 conn(0x7fbf240b6b60 0x7fbf240cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:16.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf237fe700 1 --2- 172.21.15.1:0/114543386 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf240cfbd0 0x7fbf24099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:16.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf227fc700 1 --2- 172.21.15.1:0/114543386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf240cee90 0x7fbf240cf690 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fbf2406e730 tx=0x7fbf14009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd034a77921261e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:16.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf21ffb700 1 -- 172.21.15.1:0/114543386 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1400b870 con 0x7fbf240cee90 2024-03-18T19:09:16.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf21ffb700 1 -- 172.21.15.1:0/114543386 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf1400b9d0 con 0x7fbf240cee90 2024-03-18T19:09:16.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf21ffb700 1 -- 172.21.15.1:0/114543386 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1400bd60 con 0x7fbf240cee90 2024-03-18T19:09:16.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf298c4700 1 -- 172.21.15.1:0/114543386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf240cee90 msgr2=0x7fbf240cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:16.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf298c4700 1 --2- 172.21.15.1:0/114543386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf240cee90 0x7fbf240cf690 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fbf2406e730 tx=0x7fbf14009bd0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf298c4700 1 -- 172.21.15.1:0/114543386 shutdown_connections 2024-03-18T19:09:16.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf298c4700 1 --2- 172.21.15.1:0/114543386 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf240cfbd0 0x7fbf24099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf298c4700 1 --2- 172.21.15.1:0/114543386 >> v2:172.21.15.165:3300/0 conn(0x7fbf240b6b60 0x7fbf240cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf298c4700 1 --2- 172.21.15.1:0/114543386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf240cee90 0x7fbf240cf690 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.900+0000 7fbf298c4700 1 -- 172.21.15.1:0/114543386 >> 172.21.15.1:0/114543386 conn(0x7fbf240b2ba0 msgr2=0x7fbf240b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:16.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.901+0000 7fbf298c4700 1 -- 172.21.15.1:0/114543386 shutdown_connections 2024-03-18T19:09:16.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.901+0000 7fbf298c4700 1 -- 172.21.15.1:0/114543386 wait complete. 2024-03-18T19:09:16.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.902+0000 7fbf298c4700 1 Processor -- start 2024-03-18T19:09:16.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.902+0000 7fbf298c4700 1 -- start start 2024-03-18T19:09:16.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf298c4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf240b6b60 0x7fbf240c91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:16.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf298c4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf240cee90 0x7fbf241f6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:16.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf298c4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf240cfbd0 0x7fbf241f6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:16.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf298c4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf240aebb0 con 0x7fbf240b6b60 2024-03-18T19:09:16.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf298c4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf240ae810 con 0x7fbf240cfbd0 2024-03-18T19:09:16.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf298c4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf240ae1a0 con 0x7fbf240cee90 2024-03-18T19:09:16.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf22ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf240b6b60 0x7fbf240c91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:16.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf22ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf240b6b60 0x7fbf240c91d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43828/0 (socket says 172.21.15.1:43828) 2024-03-18T19:09:16.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf22ffd700 1 -- 172.21.15.1:0/1229712312 learned_addr learned my addr 172.21.15.1:0/1229712312 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:16.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf227fc700 1 --2- 172.21.15.1:0/1229712312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf240cee90 0x7fbf241f6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:16.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf237fe700 1 --2- 172.21.15.1:0/1229712312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf240cfbd0 0x7fbf241f6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:16.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf22ffd700 1 -- 172.21.15.1:0/1229712312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf240cee90 msgr2=0x7fbf241f6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:16.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf22ffd700 1 --2- 172.21.15.1:0/1229712312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf240cee90 0x7fbf241f6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf22ffd700 1 -- 172.21.15.1:0/1229712312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf240cfbd0 msgr2=0x7fbf241f6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:16.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf22ffd700 1 --2- 172.21.15.1:0/1229712312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf240cfbd0 0x7fbf241f6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:16.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf22ffd700 1 -- 172.21.15.1:0/1229712312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf14009550 con 0x7fbf240b6b60 2024-03-18T19:09:16.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf22ffd700 1 --2- 172.21.15.1:0/1229712312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf240b6b60 0x7fbf240c91d0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fbf1800c740 tx=0x7fbf1800cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:16.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf0bfff700 1 -- 172.21.15.1:0/1229712312 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf18011800 con 0x7fbf240b6b60 2024-03-18T19:09:16.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf0bfff700 1 -- 172.21.15.1:0/1229712312 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf18011960 con 0x7fbf240b6b60 2024-03-18T19:09:16.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf298c4700 1 -- 172.21.15.1:0/1229712312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf241f70f0 con 0x7fbf240b6b60 2024-03-18T19:09:16.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf0bfff700 1 -- 172.21.15.1:0/1229712312 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf18011cd0 con 0x7fbf240b6b60 2024-03-18T19:09:16.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.903+0000 7fbf298c4700 1 -- 172.21.15.1:0/1229712312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf240ade00 con 0x7fbf240b6b60 2024-03-18T19:09:16.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.904+0000 7fbf298c4700 1 -- 172.21.15.1:0/1229712312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf240676a0 con 0x7fbf240b6b60 2024-03-18T19:09:16.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.905+0000 7fbf0bfff700 1 -- 172.21.15.1:0/1229712312 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbf18021350 con 0x7fbf240b6b60 2024-03-18T19:09:16.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.905+0000 7fbf0bfff700 1 --2- 172.21.15.1:0/1229712312 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf1006adb0 0x7fbf1006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:16.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.905+0000 7fbf0bfff700 1 -- 172.21.15.1:0/1229712312 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fbf18089680 con 0x7fbf240b6b60 2024-03-18T19:09:16.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.905+0000 7fbf227fc700 1 --2- 172.21.15.1:0/1229712312 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf1006adb0 0x7fbf1006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:16.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.906+0000 7fbf227fc700 1 --2- 172.21.15.1:0/1229712312 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf1006adb0 0x7fbf1006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbf14000c00 tx=0x7fbf14008030 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:16.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:16.911+0000 7fbf0bfff700 1 -- 172.21.15.1:0/1229712312 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbf18054540 con 0x7fbf240b6b60 2024-03-18T19:09:17.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.158+0000 7fbf298c4700 1 -- 172.21.15.1:0/1229712312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf240ae980 con 0x7fbf240b6b60 2024-03-18T19:09:17.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.158+0000 7fbf0bfff700 1 -- 172.21.15.1:0/1229712312 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+3526 (secure 0 0 0) 0x7fbf1801ede0 con 0x7fbf240b6b60 2024-03-18T19:09:17.155 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:17.156 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":59,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6800,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3354770356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3354770356},{"type":"v1","addr":"172.21.15.1:6837","nonce":3354770356}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":6809,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/695588699","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":695588699},{"type":"v1","addr":"172.21.15.165:6834","nonce":695588699}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7109,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1653043150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1653043150},{"type":"v1","addr":"172.21.15.165:6835","nonce":1653043150}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":7660,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3186346302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3186346302},{"type":"v1","addr":"172.21.15.1:6836","nonce":3186346302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[]} 2024-03-18T19:09:17.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.160+0000 7fbf298c4700 1 -- 172.21.15.1:0/1229712312 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf1006adb0 msgr2=0x7fbf1006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:17.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.160+0000 7fbf298c4700 1 --2- 172.21.15.1:0/1229712312 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf1006adb0 0x7fbf1006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbf14000c00 tx=0x7fbf14008030 comp rx=0 tx=0).stop 2024-03-18T19:09:17.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.160+0000 7fbf298c4700 1 -- 172.21.15.1:0/1229712312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf240b6b60 msgr2=0x7fbf240c91d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:17.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.160+0000 7fbf298c4700 1 --2- 172.21.15.1:0/1229712312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf240b6b60 0x7fbf240c91d0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fbf1800c740 tx=0x7fbf1800cb00 comp rx=0 tx=0).stop 2024-03-18T19:09:17.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.160+0000 7fbf298c4700 1 -- 172.21.15.1:0/1229712312 shutdown_connections 2024-03-18T19:09:17.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.160+0000 7fbf298c4700 1 --2- 172.21.15.1:0/1229712312 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf1006adb0 0x7fbf1006d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:17.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.160+0000 7fbf298c4700 1 --2- 172.21.15.1:0/1229712312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf240cfbd0 0x7fbf241f6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:17.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.160+0000 7fbf298c4700 1 --2- 172.21.15.1:0/1229712312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf240b6b60 0x7fbf240c91d0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:17.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.160+0000 7fbf298c4700 1 --2- 172.21.15.1:0/1229712312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf240cee90 0x7fbf241f6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:17.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.160+0000 7fbf298c4700 1 -- 172.21.15.1:0/1229712312 >> 172.21.15.1:0/1229712312 conn(0x7fbf240b2ba0 msgr2=0x7fbf240c69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:17.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.160+0000 7fbf298c4700 1 -- 172.21.15.1:0/1229712312 shutdown_connections 2024-03-18T19:09:17.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.160+0000 7fbf298c4700 1 -- 172.21.15.1:0/1229712312 wait complete. 2024-03-18T19:09:17.158 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 59 2024-03-18T19:09:17.170 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T19:09:17.170 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-18T19:09:17.170 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T19:09:17.170 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:09:17.170 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T19:09:17.171 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-18T19:09:17.171 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-18T19:09:17.171 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:09:17.171 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T19:09:17.171 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-18T19:09:17.171 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-18T19:09:17.171 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:09:17.172 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T19:09:17.172 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-18T19:09:17.172 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-18T19:09:17.172 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:09:17.181 INFO:tasks.ceph.mds.a:Stopped 2024-03-18T19:09:17.181 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-18T19:09:17.182 INFO:tasks.ceph.mds.c:Stopped 2024-03-18T19:09:17.182 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-18T19:09:17.183 INFO:tasks.ceph.mds.b:Stopped 2024-03-18T19:09:17.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-18T19:09:17.184 INFO:tasks.ceph.mds.d:Stopped 2024-03-18T19:09:17.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-18T19:09:17.223 INFO:tasks.ceph.mds.a:Started 2024-03-18T19:09:17.223 DEBUG:teuthology.parallel:result is None 2024-03-18T19:09:17.224 INFO:tasks.ceph.mds.c:Started 2024-03-18T19:09:17.224 DEBUG:teuthology.parallel:result is None 2024-03-18T19:09:17.226 INFO:tasks.ceph.mds.b:Started 2024-03-18T19:09:17.226 DEBUG:teuthology.parallel:result is None 2024-03-18T19:09:17.227 INFO:tasks.ceph.mds.d:Started 2024-03-18T19:09:17.227 DEBUG:teuthology.parallel:result is None 2024-03-18T19:09:17.227 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-18T19:09:17.304 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T19:09:17.304 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-18T19:09:17.310 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-03-18T19:09:17.311 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-03-18T19:09:17.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.366+0000 7fe937b92700 1 Processor -- start 2024-03-18T19:09:17.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.366+0000 7fe937b92700 1 -- start start 2024-03-18T19:09:17.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.367+0000 7fe937b92700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe93015d6f0 0x7fe930099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:17.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.367+0000 7fe937b92700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe93006e530 0x7fe9300cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:17.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.367+0000 7fe937b92700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe93006ea40 con 0x7fe930099a40 2024-03-18T19:09:17.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.367+0000 7fe937b92700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe93009a360 con 0x7fe93015d6f0 2024-03-18T19:09:17.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.367+0000 7fe937b92700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe93006ece0 con 0x7fe93006e530 2024-03-18T19:09:17.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.367+0000 7fe93512d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe93015d6f0 0x7fe930099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:17.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.367+0000 7fe93512d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe93015d6f0 0x7fe930099e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43852/0 (socket says 172.21.15.1:43852) 2024-03-18T19:09:17.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.367+0000 7fe93512d700 1 -- 172.21.15.1:0/3970087720 learned_addr learned my addr 172.21.15.1:0/3970087720 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:17.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.367+0000 7fe93612f700 1 --2- 172.21.15.1:0/3970087720 >> v2:172.21.15.165:3300/0 conn(0x7fe93006e530 0x7fe9300cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:17.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.367+0000 7fe93512d700 1 -- 172.21.15.1:0/3970087720 >> v2:172.21.15.165:3300/0 conn(0x7fe93006e530 msgr2=0x7fe9300cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:17.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.367+0000 7fe93512d700 1 --2- 172.21.15.1:0/3970087720 >> v2:172.21.15.165:3300/0 conn(0x7fe93006e530 0x7fe9300cf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:17.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.367+0000 7fe93512d700 1 -- 172.21.15.1:0/3970087720 >> v1:172.21.15.165:6789/0 conn(0x7fe930099a40 legacy=0x7fe93015e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:17.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.367+0000 7fe93512d700 1 -- 172.21.15.1:0/3970087720 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93006efc0 con 0x7fe93015d6f0 2024-03-18T19:09:17.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.367+0000 7fe93612f700 1 --2- 172.21.15.1:0/3970087720 >> v2:172.21.15.165:3300/0 conn(0x7fe93006e530 0x7fe9300cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:17.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.367+0000 7fe93512d700 1 --2- 172.21.15.1:0/3970087720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe93015d6f0 0x7fe930099e20 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fe930070240 tx=0x7fe92c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0db621dc2a32dcf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:17.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.368+0000 7fe93492c700 1 -- 172.21.15.1:0/3970087720 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c004230 con 0x7fe93015d6f0 2024-03-18T19:09:17.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.368+0000 7fe93492c700 1 -- 172.21.15.1:0/3970087720 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92c004390 con 0x7fe93015d6f0 2024-03-18T19:09:17.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.368+0000 7fe93492c700 1 -- 172.21.15.1:0/3970087720 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c0036a0 con 0x7fe93015d6f0 2024-03-18T19:09:17.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.368+0000 7fe937b92700 1 -- 172.21.15.1:0/3970087720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe93015d6f0 msgr2=0x7fe930099e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:17.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.368+0000 7fe937b92700 1 --2- 172.21.15.1:0/3970087720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe93015d6f0 0x7fe930099e20 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fe930070240 tx=0x7fe92c009bd0 comp rx=0 tx=0).stop 2024-03-18T19:09:17.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.368+0000 7fe937b92700 1 -- 172.21.15.1:0/3970087720 shutdown_connections 2024-03-18T19:09:17.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.368+0000 7fe937b92700 1 --2- 172.21.15.1:0/3970087720 >> v2:172.21.15.165:3300/0 conn(0x7fe93006e530 0x7fe9300cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:17.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.368+0000 7fe937b92700 1 --2- 172.21.15.1:0/3970087720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe93015d6f0 0x7fe930099e20 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:17.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.368+0000 7fe937b92700 1 -- 172.21.15.1:0/3970087720 >> 172.21.15.1:0/3970087720 conn(0x7fe9300b2af0 msgr2=0x7fe9300b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:17.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.368+0000 7fe937b92700 1 -- 172.21.15.1:0/3970087720 shutdown_connections 2024-03-18T19:09:17.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.368+0000 7fe937b92700 1 -- 172.21.15.1:0/3970087720 wait complete. 2024-03-18T19:09:17.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.369+0000 7fe937b92700 1 Processor -- start 2024-03-18T19:09:17.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.369+0000 7fe937b92700 1 -- start start 2024-03-18T19:09:17.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.370+0000 7fe937b92700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe93006e530 0x7fe9300b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:17.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.370+0000 7fe937b92700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe930099a40 0x7fe9300b5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:17.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.370+0000 7fe937b92700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9300b5d50 0x7fe93015faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:17.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.370+0000 7fe937b92700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93014f6b0 con 0x7fe9300b5d50 2024-03-18T19:09:17.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.370+0000 7fe937b92700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93015cfb0 con 0x7fe930099a40 2024-03-18T19:09:17.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.370+0000 7fe937b92700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9300b4320 con 0x7fe93006e530 2024-03-18T19:09:17.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.370+0000 7fe93612f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9300b5d50 0x7fe93015faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:17.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.370+0000 7fe93612f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9300b5d50 0x7fe93015faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43866/0 (socket says 172.21.15.1:43866) 2024-03-18T19:09:17.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.370+0000 7fe93592e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe93006e530 0x7fe9300b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:17.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.370+0000 7fe93612f700 1 -- 172.21.15.1:0/1125083024 learned_addr learned my addr 172.21.15.1:0/1125083024 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:17.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.370+0000 7fe93512d700 1 --2- 172.21.15.1:0/1125083024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe930099a40 0x7fe9300b5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:17.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.371+0000 7fe93612f700 1 -- 172.21.15.1:0/1125083024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe93006e530 msgr2=0x7fe9300b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:17.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.371+0000 7fe93612f700 1 --2- 172.21.15.1:0/1125083024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe93006e530 0x7fe9300b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:17.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.371+0000 7fe93612f700 1 -- 172.21.15.1:0/1125083024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe930099a40 msgr2=0x7fe9300b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:17.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.371+0000 7fe93612f700 1 --2- 172.21.15.1:0/1125083024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe930099a40 0x7fe9300b5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:17.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.371+0000 7fe93612f700 1 -- 172.21.15.1:0/1125083024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c009550 con 0x7fe9300b5d50 2024-03-18T19:09:17.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.371+0000 7fe93612f700 1 --2- 172.21.15.1:0/1125083024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9300b5d50 0x7fe93015faf0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fe92800cf50 tx=0x7fe92800cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:17.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.371+0000 7fe9227fc700 1 -- 172.21.15.1:0/1125083024 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92800ad60 con 0x7fe9300b5d50 2024-03-18T19:09:17.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.371+0000 7fe9227fc700 1 -- 172.21.15.1:0/1125083024 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe928015440 con 0x7fe9300b5d50 2024-03-18T19:09:17.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.371+0000 7fe9227fc700 1 -- 172.21.15.1:0/1125083024 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9280157f0 con 0x7fe9300b5d50 2024-03-18T19:09:17.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.371+0000 7fe937b92700 1 -- 172.21.15.1:0/1125083024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9300b6200 con 0x7fe9300b5d50 2024-03-18T19:09:17.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.371+0000 7fe937b92700 1 -- 172.21.15.1:0/1125083024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93015f010 con 0x7fe9300b5d50 2024-03-18T19:09:17.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.372+0000 7fe937b92700 1 -- 172.21.15.1:0/1125083024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe930073350 con 0x7fe9300b5d50 2024-03-18T19:09:17.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.373+0000 7fe9227fc700 1 -- 172.21.15.1:0/1125083024 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe92800d430 con 0x7fe9300b5d50 2024-03-18T19:09:17.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.373+0000 7fe9227fc700 1 --2- 172.21.15.1:0/1125083024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe91406adb0 0x7fe91406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:17.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.373+0000 7fe9227fc700 1 -- 172.21.15.1:0/1125083024 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5979+0+0 (secure 0 0 0) 0x7fe928089af0 con 0x7fe9300b5d50 2024-03-18T19:09:17.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.373+0000 7fe93592e700 1 --2- 172.21.15.1:0/1125083024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe91406adb0 0x7fe91406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:17.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.373+0000 7fe93592e700 1 --2- 172.21.15.1:0/1125083024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe91406adb0 0x7fe91406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe924007fd0 tx=0x7fe924007ec0 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:17.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.378+0000 7fe9227fc700 1 -- 172.21.15.1:0/1125083024 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe928054a20 con 0x7fe9300b5d50 2024-03-18T19:09:17.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:17.595+0000 7fe937b92700 1 -- 172.21.15.1:0/1125083024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe93006b520 con 0x7fe9300b5d50 2024-03-18T19:09:18.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.552+0000 7fe9227fc700 1 -- 172.21.15.1:0/1125083024 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v143) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe92801a070 con 0x7fe9300b5d50 2024-03-18T19:09:18.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.556+0000 7fe937b92700 1 -- 172.21.15.1:0/1125083024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe91406adb0 msgr2=0x7fe91406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:18.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.556+0000 7fe937b92700 1 --2- 172.21.15.1:0/1125083024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe91406adb0 0x7fe91406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe924007fd0 tx=0x7fe924007ec0 comp rx=0 tx=0).stop 2024-03-18T19:09:18.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.556+0000 7fe937b92700 1 -- 172.21.15.1:0/1125083024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9300b5d50 msgr2=0x7fe93015faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:18.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.556+0000 7fe937b92700 1 --2- 172.21.15.1:0/1125083024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9300b5d50 0x7fe93015faf0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fe92800cf50 tx=0x7fe92800cf80 comp rx=0 tx=0).stop 2024-03-18T19:09:18.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.556+0000 7fe937b92700 1 -- 172.21.15.1:0/1125083024 shutdown_connections 2024-03-18T19:09:18.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.556+0000 7fe937b92700 1 --2- 172.21.15.1:0/1125083024 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe91406adb0 0x7fe91406d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:18.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.556+0000 7fe937b92700 1 --2- 172.21.15.1:0/1125083024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe93006e530 0x7fe9300b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:18.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.556+0000 7fe937b92700 1 --2- 172.21.15.1:0/1125083024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9300b5d50 0x7fe93015faf0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:18.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.556+0000 7fe937b92700 1 --2- 172.21.15.1:0/1125083024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe930099a40 0x7fe9300b5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:18.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.556+0000 7fe937b92700 1 -- 172.21.15.1:0/1125083024 >> 172.21.15.1:0/1125083024 conn(0x7fe9300b2af0 msgr2=0x7fe93006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:18.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.556+0000 7fe937b92700 1 -- 172.21.15.1:0/1125083024 shutdown_connections 2024-03-18T19:09:18.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.556+0000 7fe937b92700 1 -- 172.21.15.1:0/1125083024 wait complete. 2024-03-18T19:09:18.553 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-18T19:09:18.569 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-18T19:09:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.707+0000 7f7f5db8a700 1 Processor -- start 2024-03-18T19:09:18.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.707+0000 7f7f5db8a700 1 -- start start 2024-03-18T19:09:18.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.708+0000 7f7f5db8a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7f5815d580 0x7f7f58072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.708+0000 7f7f5db8a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f5806f490 con 0x7f7f5806e290 2024-03-18T19:09:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.708+0000 7f7f5db8a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f58071cf0 con 0x7f7f5806dd50 2024-03-18T19:09:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.708+0000 7f7f5db8a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f5806f1b0 con 0x7f7f5815d580 2024-03-18T19:09:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.708+0000 7f7f56ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7f5806e290 0x7f7f5806d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48322/0 (socket says 172.21.15.1:48322) 2024-03-18T19:09:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.708+0000 7f7f56ffd700 1 -- 172.21.15.1:0/3227911922 learned_addr learned my addr 172.21.15.1:0/3227911922 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f577fe700 1 --2- 172.21.15.1:0/3227911922 >> v2:172.21.15.165:3300/0 conn(0x7f7f5815d580 0x7f7f58072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f567fc700 1 -- 172.21.15.1:0/3227911922 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162120012 0 0) 0x7f7f5806f490 con 0x7f7f5806e290 2024-03-18T19:09:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f567fc700 1 -- 172.21.15.1:0/3227911922 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f440035b0 con 0x7f7f5806e290 2024-03-18T19:09:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f567fc700 1 -- 172.21.15.1:0/3227911922 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762435562 0 0) 0x7f7f58071cf0 con 0x7f7f5806dd50 2024-03-18T19:09:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f567fc700 1 -- 172.21.15.1:0/3227911922 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f5806f490 con 0x7f7f5806dd50 2024-03-18T19:09:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f567fc700 1 -- 172.21.15.1:0/3227911922 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3442296690 0 0) 0x7f7f440035b0 con 0x7f7f5806e290 2024-03-18T19:09:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f567fc700 1 -- 172.21.15.1:0/3227911922 >> v2:172.21.15.165:3300/0 conn(0x7f7f5815d580 msgr2=0x7f7f58072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f567fc700 1 --2- 172.21.15.1:0/3227911922 >> v2:172.21.15.165:3300/0 conn(0x7f7f5815d580 0x7f7f58072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f567fc700 1 -- 172.21.15.1:0/3227911922 >> v1:172.21.15.165:6789/0 conn(0x7f7f5806dd50 legacy=0x7f7f580ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f567fc700 1 -- 172.21.15.1:0/3227911922 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f580716a0 con 0x7f7f5806e290 2024-03-18T19:09:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f567fc700 1 -- 172.21.15.1:0/3227911922 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f7f40001670 con 0x7f7f5806e290 2024-03-18T19:09:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f567fc700 1 -- 172.21.15.1:0/3227911922 >> v1:172.21.15.1:6789/0 conn(0x7f7f5806e290 legacy=0x7f7f5806d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f567fc700 1 --2- 172.21.15.1:0/3227911922 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f44003be0 0x7f7f44003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f567fc700 1 --2- 172.21.15.1:0/3227911922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f44004850 0x7f7f44004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f567fc700 1 --2- 172.21.15.1:0/3227911922 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f44005210 0x7f7f4400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f567fc700 1 -- 172.21.15.1:0/3227911922 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5806f1b0 con 0x7f7f44004850 2024-03-18T19:09:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f567fc700 1 -- 172.21.15.1:0/3227911922 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4400c030 con 0x7f7f44005210 2024-03-18T19:09:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f567fc700 1 -- 172.21.15.1:0/3227911922 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4400c1f0 con 0x7f7f44003be0 2024-03-18T19:09:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f56ffd700 1 --2- 172.21.15.1:0/3227911922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f44004850 0x7f7f44004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f57fff700 1 --2- 172.21.15.1:0/3227911922 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f44005210 0x7f7f4400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f577fe700 1 --2- 172.21.15.1:0/3227911922 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f44003be0 0x7f7f44003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:18.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f56ffd700 1 -- 172.21.15.1:0/3227911922 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f44003be0 msgr2=0x7f7f44003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:18.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f56ffd700 1 --2- 172.21.15.1:0/3227911922 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f44003be0 0x7f7f44003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:18.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f56ffd700 1 -- 172.21.15.1:0/3227911922 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f44005210 msgr2=0x7f7f4400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:18.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f56ffd700 1 --2- 172.21.15.1:0/3227911922 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f44005210 0x7f7f4400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:18.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f56ffd700 1 -- 172.21.15.1:0/3227911922 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f440035b0 con 0x7f7f44004850 2024-03-18T19:09:18.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f57fff700 1 -- 172.21.15.1:0/3227911922 reap_dead start 2024-03-18T19:09:18.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.709+0000 7f7f577fe700 1 --2- 172.21.15.1:0/3227911922 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f44003be0 0x7f7f44003fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:18.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.710+0000 7f7f56ffd700 1 --2- 172.21.15.1:0/3227911922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f44004850 0x7f7f44004cd0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000c160 tx=0x7f7f44005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad37091a3b93ae61 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:18.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.710+0000 7f7f567fc700 1 -- 172.21.15.1:0/3227911922 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4000cca0 con 0x7f7f44004850 2024-03-18T19:09:18.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.710+0000 7f7f567fc700 1 -- 172.21.15.1:0/3227911922 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4000ce20 con 0x7f7f44004850 2024-03-18T19:09:18.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.710+0000 7f7f567fc700 1 -- 172.21.15.1:0/3227911922 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f40005ba0 con 0x7f7f44004850 2024-03-18T19:09:18.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.710+0000 7f7f5db8a700 1 -- 172.21.15.1:0/3227911922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f44004850 msgr2=0x7f7f44004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:18.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.710+0000 7f7f5db8a700 1 --2- 172.21.15.1:0/3227911922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f44004850 0x7f7f44004cd0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000c160 tx=0x7f7f44005880 comp rx=0 tx=0).stop 2024-03-18T19:09:18.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.710+0000 7f7f5db8a700 1 -- 172.21.15.1:0/3227911922 shutdown_connections 2024-03-18T19:09:18.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.710+0000 7f7f5db8a700 1 --2- 172.21.15.1:0/3227911922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f44004850 0x7f7f44004cd0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:18.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.710+0000 7f7f5db8a700 1 -- 172.21.15.1:0/3227911922 >> 172.21.15.1:0/3227911922 conn(0x7f7f580cc990 msgr2=0x7f7f580ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:18.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.710+0000 7f7f5db8a700 1 -- 172.21.15.1:0/3227911922 shutdown_connections 2024-03-18T19:09:18.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.710+0000 7f7f5db8a700 1 -- 172.21.15.1:0/3227911922 wait complete. 2024-03-18T19:09:18.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.712+0000 7f7f5db8a700 1 Processor -- start 2024-03-18T19:09:18.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.712+0000 7f7f5db8a700 1 -- start start 2024-03-18T19:09:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f5db8a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f44004850 0x7f7f58150320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f5db8a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f580c7800 0x7f7f580c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f5db8a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f5815eca0 0x7f7f5815f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f5db8a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f580c81c0 con 0x7f7f44004850 2024-03-18T19:09:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f5db8a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f580b3780 con 0x7f7f5815eca0 2024-03-18T19:09:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f5db8a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f580b4020 con 0x7f7f580c7800 2024-03-18T19:09:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f577fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f44004850 0x7f7f58150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f577fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f44004850 0x7f7f58150320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43884/0 (socket says 172.21.15.1:43884) 2024-03-18T19:09:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f577fe700 1 -- 172.21.15.1:0/3806919634 learned_addr learned my addr 172.21.15.1:0/3806919634 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f56ffd700 1 --2- 172.21.15.1:0/3806919634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f580c7800 0x7f7f580c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f57fff700 1 --2- 172.21.15.1:0/3806919634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f5815eca0 0x7f7f5815f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f577fe700 1 -- 172.21.15.1:0/3806919634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f580c7800 msgr2=0x7f7f580c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f577fe700 1 --2- 172.21.15.1:0/3806919634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f580c7800 0x7f7f580c7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f577fe700 1 -- 172.21.15.1:0/3806919634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f5815eca0 msgr2=0x7f7f5815f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f577fe700 1 --2- 172.21.15.1:0/3806919634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f5815eca0 0x7f7f5815f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f577fe700 1 -- 172.21.15.1:0/3806919634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5815fa00 con 0x7f7f44004850 2024-03-18T19:09:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f56ffd700 1 --2- 172.21.15.1:0/3806919634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f580c7800 0x7f7f580c7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f57fff700 1 --2- 172.21.15.1:0/3806919634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f5815eca0 0x7f7f5815f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f577fe700 1 --2- 172.21.15.1:0/3806919634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f44004850 0x7f7f58150320 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f7f480115c0 tx=0x7f7f48011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f4ffff700 1 -- 172.21.15.1:0/3806919634 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f48004220 con 0x7f7f44004850 2024-03-18T19:09:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f4ffff700 1 -- 172.21.15.1:0/3806919634 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f48004380 con 0x7f7f44004850 2024-03-18T19:09:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f4ffff700 1 -- 172.21.15.1:0/3806919634 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f480036a0 con 0x7f7f44004850 2024-03-18T19:09:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f5db8a700 1 -- 172.21.15.1:0/3806919634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f580b39d0 con 0x7f7f44004850 2024-03-18T19:09:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.713+0000 7f7f5db8a700 1 -- 172.21.15.1:0/3806919634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f58150860 con 0x7f7f44004850 2024-03-18T19:09:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.714+0000 7f7f5db8a700 1 -- 172.21.15.1:0/3806919634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f580676a0 con 0x7f7f44004850 2024-03-18T19:09:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.715+0000 7f7f4ffff700 1 -- 172.21.15.1:0/3806919634 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7f48003800 con 0x7f7f44004850 2024-03-18T19:09:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.715+0000 7f7f4ffff700 1 --2- 172.21.15.1:0/3806919634 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f3806adb0 0x7f7f3806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.715+0000 7f7f4ffff700 1 -- 172.21.15.1:0/3806919634 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f7f48089470 con 0x7f7f44004850 2024-03-18T19:09:18.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.715+0000 7f7f56ffd700 1 --2- 172.21.15.1:0/3806919634 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f3806adb0 0x7f7f3806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:18.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.716+0000 7f7f56ffd700 1 --2- 172.21.15.1:0/3806919634 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f3806adb0 0x7f7f3806d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7f40000c00 tx=0x7f7f40007000 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:18.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:18.720+0000 7f7f4ffff700 1 -- 172.21.15.1:0/3806919634 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7f480547c0 con 0x7f7f44004850 2024-03-18T19:09:19.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.005+0000 7f7f5db8a700 1 -- 172.21.15.1:0/3806919634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f7f580cc450 con 0x7f7f44004850 2024-03-18T19:09:19.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.006+0000 7f7f4ffff700 1 -- 172.21.15.1:0/3806919634 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f7f48057de0 con 0x7f7f44004850 2024-03-18T19:09:19.002 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:19.002 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:09:19.002 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-18T19:09:19.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:19.002 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-18T19:09:19.003 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlR93/CxAAbpe+5n3wHRur9xi8QzdCnQ==", 2024-03-18T19:09:19.003 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:09:19.003 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:09:19.003 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:09:19.003 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:09:19.003 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:09:19.003 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:19.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:19.003 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-18T19:09:19.003 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJ3OmGBAA37u227nxrMsh/JHt71HBxQ==", 2024-03-18T19:09:19.003 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:09:19.003 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:09:19.004 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:09:19.004 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:09:19.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:09:19.004 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:19.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:19.004 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-18T19:09:19.004 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJZFoExAAmI16oxwNKbf8dKetPfyk+g==", 2024-03-18T19:09:19.004 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:09:19.004 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:09:19.004 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:09:19.004 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:09:19.004 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:09:19.004 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:19.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:19.005 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-18T19:09:19.005 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlQrcvIBAA1LMv+EgC7S6wspX6QF49Pg==", 2024-03-18T19:09:19.005 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:09:19.005 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:09:19.005 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:09:19.005 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:09:19.005 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:09:19.005 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:19.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:19.005 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-18T19:09:19.005 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCUjPhlYOVHChAARFQrksZmMIdTlntd8TgvrQ==", 2024-03-18T19:09:19.006 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:09:19.006 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:09:19.006 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:09:19.006 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:09:19.006 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:19.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:19.006 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-18T19:09:19.006 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCXjPhl2zSDCBAA6Al1gkWVB79ZI9djGpVChA==", 2024-03-18T19:09:19.006 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:09:19.006 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:09:19.006 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:09:19.006 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:09:19.006 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:19.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:19.007 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-18T19:09:19.007 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCajPhlSoG/BhAAR93USDe6DCQW+hREOsnTlw==", 2024-03-18T19:09:19.007 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:09:19.007 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:09:19.007 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:09:19.007 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:09:19.007 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:19.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:19.007 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-18T19:09:19.007 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCdjPhlhJg5BRAAWuQhcnxyjR5UOiM13yKdAQ==", 2024-03-18T19:09:19.007 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:09:19.007 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:09:19.008 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:09:19.008 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:09:19.008 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:19.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:19.008 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-18T19:09:19.008 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCjjPhlCFnpCRAAKIyOQyVdjXZUTnAS6Nw/ng==", 2024-03-18T19:09:19.008 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:09:19.008 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:09:19.008 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:09:19.008 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:09:19.008 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:19.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:19.008 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-18T19:09:19.009 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCmjPhlAeFkCBAAJcEFpRk/03lQJxj+l2hEPA==", 2024-03-18T19:09:19.009 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:09:19.009 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:09:19.009 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:09:19.009 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:09:19.009 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:19.009 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:19.009 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-18T19:09:19.009 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCpjPhl/xuuBhAA0LgKfXEvN2O11ZlRQJVOkw==", 2024-03-18T19:09:19.009 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:09:19.009 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:09:19.009 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:09:19.009 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:09:19.009 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:19.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:19.010 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-18T19:09:19.010 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCsjPhldiR4BRAA571oh6U8PzOv/4LzjYLF/A==", 2024-03-18T19:09:19.010 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:09:19.010 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:09:19.010 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:09:19.010 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:09:19.010 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:19.010 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:19.010 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-18T19:09:19.010 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlu+2xJBAANnXGGjfeqP6S2FTecsfOsg==", 2024-03-18T19:09:19.011 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:09:19.011 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:09:19.011 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-18T19:09:19.011 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-18T19:09:19.011 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:19.011 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:19.011 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-18T19:09:19.011 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlhJ9+KxAAEnrQrCrgJR6xI+pucwzAXA==", 2024-03-18T19:09:19.011 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:09:19.011 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:09:19.011 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:09:19.011 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:09:19.011 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:09:19.011 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:19.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:19.012 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.2", 2024-03-18T19:09:19.012 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlxBUpKBAAIGTwmLWyj+P2Yvl55QVcWA==", 2024-03-18T19:09:19.012 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:09:19.012 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:09:19.012 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:09:19.012 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:09:19.012 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:09:19.012 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:19.012 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:19.012 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.3", 2024-03-18T19:09:19.012 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlmao3MRAAHeiQ6eyBh9QjjKPCtxkC8g==", 2024-03-18T19:09:19.012 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:09:19.013 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:09:19.013 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:09:19.013 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:09:19.013 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:09:19.013 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:19.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:19.013 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-18T19:09:19.013 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCRjPhlemOuKRAAVUggmtB7+EiVgyfJZ8TCCA==", 2024-03-18T19:09:19.013 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:09:19.013 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:09:19.013 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:09:19.013 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:09:19.013 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:09:19.013 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:19.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:19.014 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-18T19:09:19.014 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhleIuOCBAA9CH+AHewQbn+gpVr+6lJyw==", 2024-03-18T19:09:19.014 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:09:19.014 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:09:19.014 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T19:09:19.014 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:09:19.014 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:19.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:19.014 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-18T19:09:19.014 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlL0pTBBAAdFiPcdwwz514cuLAvioxYA==", 2024-03-18T19:09:19.014 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:09:19.015 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:09:19.015 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T19:09:19.015 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:09:19.015 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.015 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:19.015 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-18T19:09:19.015 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:09:19.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.008+0000 7f7f5db8a700 1 -- 172.21.15.1:0/3806919634 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f3806adb0 msgr2=0x7f7f3806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:19.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.008+0000 7f7f5db8a700 1 --2- 172.21.15.1:0/3806919634 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f3806adb0 0x7f7f3806d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7f40000c00 tx=0x7f7f40007000 comp rx=0 tx=0).stop 2024-03-18T19:09:19.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.008+0000 7f7f5db8a700 1 -- 172.21.15.1:0/3806919634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f44004850 msgr2=0x7f7f58150320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:19.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.008+0000 7f7f5db8a700 1 --2- 172.21.15.1:0/3806919634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f44004850 0x7f7f58150320 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f7f480115c0 tx=0x7f7f48011980 comp rx=0 tx=0).stop 2024-03-18T19:09:19.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.008+0000 7f7f5db8a700 1 -- 172.21.15.1:0/3806919634 shutdown_connections 2024-03-18T19:09:19.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.008+0000 7f7f5db8a700 1 --2- 172.21.15.1:0/3806919634 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f3806adb0 0x7f7f3806d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.008+0000 7f7f5db8a700 1 --2- 172.21.15.1:0/3806919634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f5815eca0 0x7f7f5815f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.008+0000 7f7f5db8a700 1 --2- 172.21.15.1:0/3806919634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f44004850 0x7f7f58150320 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.008+0000 7f7f5db8a700 1 --2- 172.21.15.1:0/3806919634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f580c7800 0x7f7f580c7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.008+0000 7f7f5db8a700 1 -- 172.21.15.1:0/3806919634 >> 172.21.15.1:0/3806919634 conn(0x7f7f580cc990 msgr2=0x7f7f5815c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:19.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.008+0000 7f7f5db8a700 1 -- 172.21.15.1:0/3806919634 shutdown_connections 2024-03-18T19:09:19.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.008+0000 7f7f5db8a700 1 -- 172.21.15.1:0/3806919634 wait complete. 2024-03-18T19:09:19.018 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:09:19.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.147+0000 7f5281526700 1 Processor -- start 2024-03-18T19:09:19.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.147+0000 7f5281526700 1 -- start start 2024-03-18T19:09:19.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.148+0000 7f5281526700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f527c1545e0 0x7f527c06e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.148+0000 7f5281526700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f527c0aebd0 0x7f527c15b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.148+0000 7f5281526700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c15b630 con 0x7f527c06e760 2024-03-18T19:09:19.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.148+0000 7f5281526700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f527c06cfc0 con 0x7f527c1545e0 2024-03-18T19:09:19.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.148+0000 7f5281526700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f527c06d280 con 0x7f527c0aebd0 2024-03-18T19:09:19.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.148+0000 7f527affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f527c1545e0 0x7f527c06e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:19.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.148+0000 7f527affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f527c1545e0 0x7f527c06e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43900/0 (socket says 172.21.15.1:43900) 2024-03-18T19:09:19.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.148+0000 7f527affd700 1 -- 172.21.15.1:0/147276539 learned_addr learned my addr 172.21.15.1:0/147276539 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:19.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.148+0000 7f527affd700 1 -- 172.21.15.1:0/147276539 >> v2:172.21.15.165:3300/0 conn(0x7f527c0aebd0 msgr2=0x7f527c15b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:19.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.148+0000 7f527affd700 1 --2- 172.21.15.1:0/147276539 >> v2:172.21.15.165:3300/0 conn(0x7f527c0aebd0 0x7f527c15b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.148+0000 7f527affd700 1 -- 172.21.15.1:0/147276539 >> v1:172.21.15.165:6789/0 conn(0x7f527c06e760 legacy=0x7f527c0b2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:19.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.148+0000 7f527affd700 1 -- 172.21.15.1:0/147276539 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527c06d560 con 0x7f527c1545e0 2024-03-18T19:09:19.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.148+0000 7f527affd700 1 --2- 172.21.15.1:0/147276539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527c1545e0 0x7f527c06e040 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f527c0af290 tx=0x7f526c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4506b806644ecdb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:19.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.148+0000 7f5279ffb700 1 -- 172.21.15.1:0/147276539 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c004230 con 0x7f527c1545e0 2024-03-18T19:09:19.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.148+0000 7f5279ffb700 1 -- 172.21.15.1:0/147276539 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526c004390 con 0x7f527c1545e0 2024-03-18T19:09:19.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.148+0000 7f5279ffb700 1 -- 172.21.15.1:0/147276539 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c0036a0 con 0x7f527c1545e0 2024-03-18T19:09:19.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.149+0000 7f5281526700 1 -- 172.21.15.1:0/147276539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527c1545e0 msgr2=0x7f527c06e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:19.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.149+0000 7f5281526700 1 --2- 172.21.15.1:0/147276539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527c1545e0 0x7f527c06e040 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f527c0af290 tx=0x7f526c009bd0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.149+0000 7f5281526700 1 -- 172.21.15.1:0/147276539 shutdown_connections 2024-03-18T19:09:19.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.149+0000 7f5281526700 1 --2- 172.21.15.1:0/147276539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527c1545e0 0x7f527c06e040 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.149+0000 7f5281526700 1 --2- 172.21.15.1:0/147276539 >> v2:172.21.15.165:3300/0 conn(0x7f527c0aebd0 0x7f527c15b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.149+0000 7f5281526700 1 -- 172.21.15.1:0/147276539 >> 172.21.15.1:0/147276539 conn(0x7f527c0b1350 msgr2=0x7f527c0b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:19.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.149+0000 7f5281526700 1 -- 172.21.15.1:0/147276539 shutdown_connections 2024-03-18T19:09:19.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.149+0000 7f5281526700 1 -- 172.21.15.1:0/147276539 wait complete. 2024-03-18T19:09:19.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.150+0000 7f5281526700 1 Processor -- start 2024-03-18T19:09:19.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.150+0000 7f5281526700 1 -- start start 2024-03-18T19:09:19.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.151+0000 7f5281526700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527c06e760 0x7f527c153300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.151+0000 7f5281526700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f527c0aebd0 0x7f527c157d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.151+0000 7f5281526700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f527c1545e0 0x7f527c0ac450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.151+0000 7f5281526700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527c158c10 con 0x7f527c06e760 2024-03-18T19:09:19.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.151+0000 7f5281526700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527c158650 con 0x7f527c1545e0 2024-03-18T19:09:19.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.151+0000 7f5281526700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f527c158930 con 0x7f527c0aebd0 2024-03-18T19:09:19.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.151+0000 7f527affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527c06e760 0x7f527c153300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:19.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.151+0000 7f527affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527c06e760 0x7f527c153300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43904/0 (socket says 172.21.15.1:43904) 2024-03-18T19:09:19.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.151+0000 7f527affd700 1 -- 172.21.15.1:0/1937924637 learned_addr learned my addr 172.21.15.1:0/1937924637 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:19.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.151+0000 7f527a7fc700 1 --2- 172.21.15.1:0/1937924637 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f527c0aebd0 0x7f527c157d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:19.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.151+0000 7f527b7fe700 1 --2- 172.21.15.1:0/1937924637 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f527c1545e0 0x7f527c0ac450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:19.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.151+0000 7f527affd700 1 -- 172.21.15.1:0/1937924637 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f527c0aebd0 msgr2=0x7f527c157d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:19.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.151+0000 7f527affd700 1 --2- 172.21.15.1:0/1937924637 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f527c0aebd0 0x7f527c157d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.151+0000 7f527affd700 1 -- 172.21.15.1:0/1937924637 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f527c1545e0 msgr2=0x7f527c0ac450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:19.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.151+0000 7f527affd700 1 --2- 172.21.15.1:0/1937924637 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f527c1545e0 0x7f527c0ac450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.151+0000 7f527affd700 1 -- 172.21.15.1:0/1937924637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526c009550 con 0x7f527c06e760 2024-03-18T19:09:19.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.151+0000 7f527a7fc700 1 --2- 172.21.15.1:0/1937924637 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f527c0aebd0 0x7f527c157d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:19.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.151+0000 7f527b7fe700 1 --2- 172.21.15.1:0/1937924637 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f527c1545e0 0x7f527c0ac450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:19.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.152+0000 7f527affd700 1 --2- 172.21.15.1:0/1937924637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527c06e760 0x7f527c153300 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f526c009130 tx=0x7f526c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:19.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.152+0000 7f52637fe700 1 -- 172.21.15.1:0/1937924637 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c011630 con 0x7f527c06e760 2024-03-18T19:09:19.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.152+0000 7f52637fe700 1 -- 172.21.15.1:0/1937924637 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526c011790 con 0x7f527c06e760 2024-03-18T19:09:19.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.152+0000 7f5281526700 1 -- 172.21.15.1:0/1937924637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f527c0a85d0 con 0x7f527c06e760 2024-03-18T19:09:19.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.152+0000 7f52637fe700 1 -- 172.21.15.1:0/1937924637 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526c011b00 con 0x7f527c06e760 2024-03-18T19:09:19.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.152+0000 7f5281526700 1 -- 172.21.15.1:0/1937924637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f527c153840 con 0x7f527c06e760 2024-03-18T19:09:19.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.153+0000 7f5281526700 1 -- 172.21.15.1:0/1937924637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f527c0a8af0 con 0x7f527c06e760 2024-03-18T19:09:19.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.153+0000 7f52637fe700 1 -- 172.21.15.1:0/1937924637 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f526c011c60 con 0x7f527c06e760 2024-03-18T19:09:19.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.154+0000 7f52637fe700 1 --2- 172.21.15.1:0/1937924637 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526806ade0 0x7f526806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.154+0000 7f52637fe700 1 -- 172.21.15.1:0/1937924637 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f526c08a900 con 0x7f527c06e760 2024-03-18T19:09:19.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.154+0000 7f527a7fc700 1 --2- 172.21.15.1:0/1937924637 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526806ade0 0x7f526806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:19.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.154+0000 7f527a7fc700 1 --2- 172.21.15.1:0/1937924637 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526806ade0 0x7f526806d2a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5270006fd0 tx=0x7f5270008040 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:19.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.158+0000 7f52637fe700 1 -- 172.21.15.1:0/1937924637 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f527c0a8af0 con 0x7f527c06e760 2024-03-18T19:09:19.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.366+0000 7f5281526700 1 -- 172.21.15.1:0/1937924637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f527c0a8af0 con 0x7f527c06e760 2024-03-18T19:09:19.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.367+0000 7f52637fe700 1 -- 172.21.15.1:0/1937924637 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+12600 (secure 0 0 0) 0x7f527c0a8af0 con 0x7f527c06e760 2024-03-18T19:09:19.363 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:19.363 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":143,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:09:18.549724+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:09:19.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.369+0000 7f5281526700 1 -- 172.21.15.1:0/1937924637 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526806ade0 msgr2=0x7f526806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.369+0000 7f5281526700 1 --2- 172.21.15.1:0/1937924637 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526806ade0 0x7f526806d2a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5270006fd0 tx=0x7f5270008040 comp rx=0 tx=0).stop 2024-03-18T19:09:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.369+0000 7f5281526700 1 -- 172.21.15.1:0/1937924637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527c06e760 msgr2=0x7f527c153300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.369+0000 7f5281526700 1 --2- 172.21.15.1:0/1937924637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527c06e760 0x7f527c153300 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f526c009130 tx=0x7f526c004750 comp rx=0 tx=0).stop 2024-03-18T19:09:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.369+0000 7f5281526700 1 -- 172.21.15.1:0/1937924637 shutdown_connections 2024-03-18T19:09:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.369+0000 7f5281526700 1 --2- 172.21.15.1:0/1937924637 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526806ade0 0x7f526806d2a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.369+0000 7f5281526700 1 --2- 172.21.15.1:0/1937924637 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f527c1545e0 0x7f527c0ac450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.369+0000 7f5281526700 1 --2- 172.21.15.1:0/1937924637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f527c06e760 0x7f527c153300 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.369+0000 7f5281526700 1 --2- 172.21.15.1:0/1937924637 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f527c0aebd0 0x7f527c157d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.369+0000 7f5281526700 1 -- 172.21.15.1:0/1937924637 >> 172.21.15.1:0/1937924637 conn(0x7f527c0b1350 msgr2=0x7f527c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.369+0000 7f5281526700 1 -- 172.21.15.1:0/1937924637 shutdown_connections 2024-03-18T19:09:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.369+0000 7f5281526700 1 -- 172.21.15.1:0/1937924637 wait complete. 2024-03-18T19:09:19.378 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:09:19.378 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:09:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.509+0000 7f97d3ab4700 1 Processor -- start 2024-03-18T19:09:19.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.509+0000 7f97d3ab4700 1 -- start start 2024-03-18T19:09:19.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.510+0000 7f97d3ab4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f97cc09dd40 0x7f97cc0cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.510+0000 7f97d3ab4700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97cc0ce090 con 0x7f97cc15d780 2024-03-18T19:09:19.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.510+0000 7f97d3ab4700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97cc0cf0e0 con 0x7f97cc0ce5a0 2024-03-18T19:09:19.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.510+0000 7f97d3ab4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97cc0cb480 con 0x7f97cc09dd40 2024-03-18T19:09:19.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.510+0000 7f97d1850700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f97cc15d780 0x7f97cc0cad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48324/0 (socket says 172.21.15.1:48324) 2024-03-18T19:09:19.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.510+0000 7f97d1850700 1 -- 172.21.15.1:0/1134188211 learned_addr learned my addr 172.21.15.1:0/1134188211 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:19.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d104f700 1 --2- 172.21.15.1:0/1134188211 >> v2:172.21.15.165:3300/0 conn(0x7f97cc09dd40 0x7f97cc0cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:19.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d084e700 1 -- 172.21.15.1:0/1134188211 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755018955 0 0) 0x7f97cc0ce090 con 0x7f97cc15d780 2024-03-18T19:09:19.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d084e700 1 -- 172.21.15.1:0/1134188211 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97b80035b0 con 0x7f97cc15d780 2024-03-18T19:09:19.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d084e700 1 -- 172.21.15.1:0/1134188211 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3231497105 0 0) 0x7f97b80035b0 con 0x7f97cc15d780 2024-03-18T19:09:19.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d084e700 1 -- 172.21.15.1:0/1134188211 >> v2:172.21.15.165:3300/0 conn(0x7f97cc09dd40 msgr2=0x7f97cc0cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:19.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d084e700 1 --2- 172.21.15.1:0/1134188211 >> v2:172.21.15.165:3300/0 conn(0x7f97cc09dd40 0x7f97cc0cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d084e700 1 -- 172.21.15.1:0/1134188211 >> v1:172.21.15.165:6789/0 conn(0x7f97cc0ce5a0 legacy=0x7f97cc0ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:19.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d084e700 1 -- 172.21.15.1:0/1134188211 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97cc06db30 con 0x7f97cc15d780 2024-03-18T19:09:19.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d084e700 1 -- 172.21.15.1:0/1134188211 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f97bc001670 con 0x7f97cc15d780 2024-03-18T19:09:19.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d084e700 1 -- 172.21.15.1:0/1134188211 >> v1:172.21.15.1:6789/0 conn(0x7f97cc15d780 legacy=0x7f97cc0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:19.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d084e700 1 --2- 172.21.15.1:0/1134188211 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97b8003a00 0x7f97b8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d084e700 1 --2- 172.21.15.1:0/1134188211 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97b8004400 0x7f97b80048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d084e700 1 --2- 172.21.15.1:0/1134188211 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97b80050d0 0x7f97b800dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d084e700 1 -- 172.21.15.1:0/1134188211 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc0cb480 con 0x7f97b8003a00 2024-03-18T19:09:19.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d084e700 1 -- 172.21.15.1:0/1134188211 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b800e240 con 0x7f97b80050d0 2024-03-18T19:09:19.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d084e700 1 -- 172.21.15.1:0/1134188211 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b800e400 con 0x7f97b8004400 2024-03-18T19:09:19.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d1850700 1 --2- 172.21.15.1:0/1134188211 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97b8003a00 0x7f97b8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:19.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d104f700 1 --2- 172.21.15.1:0/1134188211 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97b8004400 0x7f97b80048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:19.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d2051700 1 --2- 172.21.15.1:0/1134188211 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97b80050d0 0x7f97b800dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:19.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d1850700 1 -- 172.21.15.1:0/1134188211 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97b8004400 msgr2=0x7f97b80048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:19.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d1850700 1 --2- 172.21.15.1:0/1134188211 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97b8004400 0x7f97b80048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d1850700 1 -- 172.21.15.1:0/1134188211 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97b80050d0 msgr2=0x7f97b800dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:19.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d1850700 1 --2- 172.21.15.1:0/1134188211 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97b80050d0 0x7f97b800dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d1850700 1 -- 172.21.15.1:0/1134188211 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b80035b0 con 0x7f97b8003a00 2024-03-18T19:09:19.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d2051700 1 -- 172.21.15.1:0/1134188211 reap_dead start 2024-03-18T19:09:19.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d104f700 1 --2- 172.21.15.1:0/1134188211 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97b8004400 0x7f97b80048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:19.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.511+0000 7f97d1850700 1 --2- 172.21.15.1:0/1134188211 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97b8003a00 0x7f97b8003de0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00c2f0 tx=0x7f97b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a351170fb4404db1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:19.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.512+0000 7f97d084e700 1 -- 172.21.15.1:0/1134188211 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc00ce30 con 0x7f97b8003a00 2024-03-18T19:09:19.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.512+0000 7f97d084e700 1 -- 172.21.15.1:0/1134188211 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97bc005960 con 0x7f97b8003a00 2024-03-18T19:09:19.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.512+0000 7f97d084e700 1 -- 172.21.15.1:0/1134188211 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc005cf0 con 0x7f97b8003a00 2024-03-18T19:09:19.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.512+0000 7f97d3ab4700 1 -- 172.21.15.1:0/1134188211 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97b8003a00 msgr2=0x7f97b8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:19.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.512+0000 7f97d3ab4700 1 --2- 172.21.15.1:0/1134188211 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97b8003a00 0x7f97b8003de0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00c2f0 tx=0x7f97b8005880 comp rx=0 tx=0).stop 2024-03-18T19:09:19.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.512+0000 7f97d3ab4700 1 -- 172.21.15.1:0/1134188211 shutdown_connections 2024-03-18T19:09:19.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.512+0000 7f97d3ab4700 1 --2- 172.21.15.1:0/1134188211 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97b8003a00 0x7f97b8003de0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.512+0000 7f97d3ab4700 1 -- 172.21.15.1:0/1134188211 >> 172.21.15.1:0/1134188211 conn(0x7f97cc0cca00 msgr2=0x7f97cc0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:19.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.512+0000 7f97d3ab4700 1 -- 172.21.15.1:0/1134188211 shutdown_connections 2024-03-18T19:09:19.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.512+0000 7f97d3ab4700 1 -- 172.21.15.1:0/1134188211 wait complete. 2024-03-18T19:09:19.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.513+0000 7f97d3ab4700 1 Processor -- start 2024-03-18T19:09:19.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.514+0000 7f97d3ab4700 1 -- start start 2024-03-18T19:09:19.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.514+0000 7f97d3ab4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97b8003a00 0x7f97cc1549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.514+0000 7f97d3ab4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97cc0c9530 0x7f97cc0c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.514+0000 7f97d3ab4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97cc159480 0x7f97cc159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.514+0000 7f97d3ab4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc0c9ef0 con 0x7f97cc159480 2024-03-18T19:09:19.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.514+0000 7f97d3ab4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc0c3c10 con 0x7f97cc0c9530 2024-03-18T19:09:19.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.514+0000 7f97d3ab4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc0c3ef0 con 0x7f97b8003a00 2024-03-18T19:09:19.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.514+0000 7f97d2051700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97cc159480 0x7f97cc159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.514+0000 7f97d2051700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97cc159480 0x7f97cc159900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43936/0 (socket says 172.21.15.1:43936) 2024-03-18T19:09:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.514+0000 7f97d2051700 1 -- 172.21.15.1:0/99664948 learned_addr learned my addr 172.21.15.1:0/99664948 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.515+0000 7f97d104f700 1 --2- 172.21.15.1:0/99664948 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97cc0c9530 0x7f97cc0c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.515+0000 7f97d1850700 1 --2- 172.21.15.1:0/99664948 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97b8003a00 0x7f97cc1549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.515+0000 7f97d2051700 1 -- 172.21.15.1:0/99664948 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97b8003a00 msgr2=0x7f97cc1549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.515+0000 7f97d2051700 1 --2- 172.21.15.1:0/99664948 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97b8003a00 0x7f97cc1549b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.515+0000 7f97d2051700 1 -- 172.21.15.1:0/99664948 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97cc0c9530 msgr2=0x7f97cc0c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.515+0000 7f97d2051700 1 --2- 172.21.15.1:0/99664948 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97cc0c9530 0x7f97cc0c99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.515+0000 7f97d2051700 1 -- 172.21.15.1:0/99664948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97cc0c41d0 con 0x7f97cc159480 2024-03-18T19:09:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.515+0000 7f97d2051700 1 --2- 172.21.15.1:0/99664948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97cc159480 0x7f97cc159900 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f97cc071270 tx=0x7f97c400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.515+0000 7f97c27fc700 1 -- 172.21.15.1:0/99664948 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4003c10 con 0x7f97cc159480 2024-03-18T19:09:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.515+0000 7f97c27fc700 1 -- 172.21.15.1:0/99664948 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c4003d70 con 0x7f97cc159480 2024-03-18T19:09:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.515+0000 7f97c27fc700 1 -- 172.21.15.1:0/99664948 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4005310 con 0x7f97cc159480 2024-03-18T19:09:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.515+0000 7f97d3ab4700 1 -- 172.21.15.1:0/99664948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97cc0c44b0 con 0x7f97cc159480 2024-03-18T19:09:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.515+0000 7f97d3ab4700 1 -- 172.21.15.1:0/99664948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97cc154ef0 con 0x7f97cc159480 2024-03-18T19:09:19.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.516+0000 7f97d3ab4700 1 -- 172.21.15.1:0/99664948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97cc1591a0 con 0x7f97cc159480 2024-03-18T19:09:19.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.516+0000 7f97c27fc700 1 -- 172.21.15.1:0/99664948 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f97c4005470 con 0x7f97cc159480 2024-03-18T19:09:19.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.517+0000 7f97c27fc700 1 --2- 172.21.15.1:0/99664948 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97b006adb0 0x7f97b006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.517+0000 7f97d1850700 1 --2- 172.21.15.1:0/99664948 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97b006adb0 0x7f97b006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:19.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.517+0000 7f97c27fc700 1 -- 172.21.15.1:0/99664948 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f97c4089290 con 0x7f97cc159480 2024-03-18T19:09:19.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.517+0000 7f97d1850700 1 --2- 172.21.15.1:0/99664948 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97b006adb0 0x7f97b006d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f97bc000c00 tx=0x7f97bc007000 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:19.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.522+0000 7f97c27fc700 1 -- 172.21.15.1:0/99664948 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f97cc1591a0 con 0x7f97cc159480 2024-03-18T19:09:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.724+0000 7f97d3ab4700 1 -- 172.21.15.1:0/99664948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f97cc1591a0 con 0x7f97cc159480 2024-03-18T19:09:19.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.724+0000 7f97c27fc700 1 -- 172.21.15.1:0/99664948 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7f97cc1591a0 con 0x7f97cc159480 2024-03-18T19:09:19.720 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:09:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.727+0000 7f97d3ab4700 1 -- 172.21.15.1:0/99664948 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97b006adb0 msgr2=0x7f97b006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.727+0000 7f97d3ab4700 1 --2- 172.21.15.1:0/99664948 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97b006adb0 0x7f97b006d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f97bc000c00 tx=0x7f97bc007000 comp rx=0 tx=0).stop 2024-03-18T19:09:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.727+0000 7f97d3ab4700 1 -- 172.21.15.1:0/99664948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97cc159480 msgr2=0x7f97cc159900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.727+0000 7f97d3ab4700 1 --2- 172.21.15.1:0/99664948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97cc159480 0x7f97cc159900 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f97cc071270 tx=0x7f97c400ee70 comp rx=0 tx=0).stop 2024-03-18T19:09:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.727+0000 7f97d3ab4700 1 -- 172.21.15.1:0/99664948 shutdown_connections 2024-03-18T19:09:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.727+0000 7f97d3ab4700 1 --2- 172.21.15.1:0/99664948 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f97b006adb0 0x7f97b006d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.727+0000 7f97d3ab4700 1 --2- 172.21.15.1:0/99664948 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f97b8003a00 0x7f97cc1549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.727+0000 7f97d3ab4700 1 --2- 172.21.15.1:0/99664948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97cc159480 0x7f97cc159900 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.727+0000 7f97d3ab4700 1 --2- 172.21.15.1:0/99664948 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97cc0c9530 0x7f97cc0c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.727+0000 7f97d3ab4700 1 -- 172.21.15.1:0/99664948 >> 172.21.15.1:0/99664948 conn(0x7f97cc0cca00 msgr2=0x7f97cc0b0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.727+0000 7f97d3ab4700 1 -- 172.21.15.1:0/99664948 shutdown_connections 2024-03-18T19:09:19.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.727+0000 7f97d3ab4700 1 -- 172.21.15.1:0/99664948 wait complete. 2024-03-18T19:09:19.735 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T19:09:19.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.864+0000 7fadbeaf4700 1 Processor -- start 2024-03-18T19:09:19.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.864+0000 7fadbeaf4700 1 -- start start 2024-03-18T19:09:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.865+0000 7fadbeaf4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fadb80b3980 0x7fadb806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.865+0000 7fadbeaf4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fadb815e0f0 0x7fadb806dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.865+0000 7fadbeaf4700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadb815dc60 con 0x7fadb8159620 2024-03-18T19:09:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.865+0000 7fadbeaf4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadb815d400 con 0x7fadb80b3980 2024-03-18T19:09:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.865+0000 7fadbeaf4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadb815ac20 con 0x7fadb815e0f0 2024-03-18T19:09:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.865+0000 7fadb7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fadb80b3980 0x7fadb806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.865+0000 7fadbc890700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fadb8159620 0x7fadb80b3270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48338/0 (socket says 172.21.15.1:48338) 2024-03-18T19:09:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.865+0000 7fadbc890700 1 -- 172.21.15.1:0/2898915915 learned_addr learned my addr 172.21.15.1:0/2898915915 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.865+0000 7fadbd091700 1 --2- 172.21.15.1:0/2898915915 >> v2:172.21.15.165:3300/0 conn(0x7fadb815e0f0 0x7fadb806dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.866+0000 7fadb7fff700 1 -- 172.21.15.1:0/2898915915 >> v2:172.21.15.165:3300/0 conn(0x7fadb815e0f0 msgr2=0x7fadb806dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.866+0000 7fadb7fff700 1 --2- 172.21.15.1:0/2898915915 >> v2:172.21.15.165:3300/0 conn(0x7fadb815e0f0 0x7fadb806dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.866+0000 7fadb7fff700 1 -- 172.21.15.1:0/2898915915 >> v1:172.21.15.1:6789/0 conn(0x7fadb8159620 legacy=0x7fadb80b3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.866+0000 7fadb7fff700 1 -- 172.21.15.1:0/2898915915 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb815d610 con 0x7fadb80b3980 2024-03-18T19:09:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.866+0000 7fadb77fe700 1 -- 172.21.15.1:0/2898915915 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260795341 0 0) 0x7fadb815dc60 con 0x7fadb8159620 2024-03-18T19:09:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.866+0000 7fadb7fff700 1 --2- 172.21.15.1:0/2898915915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadb80b3980 0x7fadb806d6b0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fadb815d330 tx=0x7fada4009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fb520e13a2ffc65 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.866+0000 7fadb77fe700 1 -- 172.21.15.1:0/2898915915 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada4005e30 con 0x7fadb80b3980 2024-03-18T19:09:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.866+0000 7fadb77fe700 1 -- 172.21.15.1:0/2898915915 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada4006030 con 0x7fadb80b3980 2024-03-18T19:09:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.866+0000 7fadb77fe700 1 -- 172.21.15.1:0/2898915915 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada40063c0 con 0x7fadb80b3980 2024-03-18T19:09:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.866+0000 7fadbeaf4700 1 -- 172.21.15.1:0/2898915915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadb80b3980 msgr2=0x7fadb806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.866+0000 7fadbeaf4700 1 --2- 172.21.15.1:0/2898915915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadb80b3980 0x7fadb806d6b0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fadb815d330 tx=0x7fada4009b00 comp rx=0 tx=0).stop 2024-03-18T19:09:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.867+0000 7fadbeaf4700 1 -- 172.21.15.1:0/2898915915 shutdown_connections 2024-03-18T19:09:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.867+0000 7fadbeaf4700 1 --2- 172.21.15.1:0/2898915915 >> v2:172.21.15.165:3300/0 conn(0x7fadb815e0f0 0x7fadb806dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.867+0000 7fadbeaf4700 1 --2- 172.21.15.1:0/2898915915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadb80b3980 0x7fadb806d6b0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.867+0000 7fadbeaf4700 1 -- 172.21.15.1:0/2898915915 >> 172.21.15.1:0/2898915915 conn(0x7fadb806a750 msgr2=0x7fadb806ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:19.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.867+0000 7fadbeaf4700 1 -- 172.21.15.1:0/2898915915 shutdown_connections 2024-03-18T19:09:19.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.867+0000 7fadbeaf4700 1 -- 172.21.15.1:0/2898915915 wait complete. 2024-03-18T19:09:19.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.868+0000 7fadbeaf4700 1 Processor -- start 2024-03-18T19:09:19.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.868+0000 7fadbeaf4700 1 -- start start 2024-03-18T19:09:19.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadbeaf4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fadb80b3980 0x7fadb814f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadbeaf4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadb8159620 0x7fadb814e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadbeaf4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fadb815e0f0 0x7fadb8151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadbeaf4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb81534a0 con 0x7fadb8159620 2024-03-18T19:09:19.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadbeaf4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb814f570 con 0x7fadb815e0f0 2024-03-18T19:09:19.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadbeaf4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadb8152540 con 0x7fadb80b3980 2024-03-18T19:09:19.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadb7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadb8159620 0x7fadb814e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:19.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadb7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadb8159620 0x7fadb814e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43966/0 (socket says 172.21.15.1:43966) 2024-03-18T19:09:19.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadb7fff700 1 -- 172.21.15.1:0/405433591 learned_addr learned my addr 172.21.15.1:0/405433591 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:19.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadbd091700 1 --2- 172.21.15.1:0/405433591 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fadb815e0f0 0x7fadb8151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:19.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadbc890700 1 --2- 172.21.15.1:0/405433591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fadb80b3980 0x7fadb814f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:19.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadb7fff700 1 -- 172.21.15.1:0/405433591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fadb80b3980 msgr2=0x7fadb814f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:19.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadb7fff700 1 --2- 172.21.15.1:0/405433591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fadb80b3980 0x7fadb814f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadb7fff700 1 -- 172.21.15.1:0/405433591 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fadb815e0f0 msgr2=0x7fadb8151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:19.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadb7fff700 1 --2- 172.21.15.1:0/405433591 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fadb815e0f0 0x7fadb8151ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:19.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadb7fff700 1 -- 172.21.15.1:0/405433591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada4009430 con 0x7fadb8159620 2024-03-18T19:09:19.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadb7fff700 1 --2- 172.21.15.1:0/405433591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadb8159620 0x7fadb814e120 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fada4009b40 tx=0x7fada4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:19.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadb57fa700 1 -- 172.21.15.1:0/405433591 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada401f070 con 0x7fadb8159620 2024-03-18T19:09:19.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadb57fa700 1 -- 172.21.15.1:0/405433591 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada400c050 con 0x7fadb8159620 2024-03-18T19:09:19.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadb57fa700 1 -- 172.21.15.1:0/405433591 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada4019980 con 0x7fadb8159620 2024-03-18T19:09:19.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.869+0000 7fadbeaf4700 1 -- 172.21.15.1:0/405433591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb8152820 con 0x7fadb8159620 2024-03-18T19:09:19.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.870+0000 7fadbeaf4700 1 -- 172.21.15.1:0/405433591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb8152de0 con 0x7fadb8159620 2024-03-18T19:09:19.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.871+0000 7fadbeaf4700 1 -- 172.21.15.1:0/405433591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb815a830 con 0x7fadb8159620 2024-03-18T19:09:19.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.871+0000 7fadb57fa700 1 -- 172.21.15.1:0/405433591 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fada4019ae0 con 0x7fadb8159620 2024-03-18T19:09:19.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.871+0000 7fadb57fa700 1 --2- 172.21.15.1:0/405433591 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fad9c06adb0 0x7fad9c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:19.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.871+0000 7fadb57fa700 1 -- 172.21.15.1:0/405433591 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fada408a6b0 con 0x7fadb8159620 2024-03-18T19:09:19.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.871+0000 7fadbc890700 1 --2- 172.21.15.1:0/405433591 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fad9c06adb0 0x7fad9c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:19.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.872+0000 7fadbc890700 1 --2- 172.21.15.1:0/405433591 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fad9c06adb0 0x7fad9c06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fadac000f60 tx=0x7fadac007be0 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:19.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:19.876+0000 7fadb57fa700 1 -- 172.21.15.1:0/405433591 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fada40558f0 con 0x7fadb8159620 2024-03-18T19:09:20.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.079+0000 7fadbeaf4700 1 -- 172.21.15.1:0/405433591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fadb80676a0 con 0x7fadb8159620 2024-03-18T19:09:20.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.079+0000 7fadb57fa700 1 -- 172.21.15.1:0/405433591 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v143) v1 ==== 84+0+65 (secure 0 0 0) 0x7fada4058f10 con 0x7fadb8159620 2024-03-18T19:09:20.075 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:20.075 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T19:09:20.075 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:20.076 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-18T19:09:20.076 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-18T19:09:20.076 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:20.076 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T19:09:20.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.081+0000 7fadbeaf4700 1 -- 172.21.15.1:0/405433591 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fad9c06adb0 msgr2=0x7fad9c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.081+0000 7fadbeaf4700 1 --2- 172.21.15.1:0/405433591 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fad9c06adb0 0x7fad9c06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fadac000f60 tx=0x7fadac007be0 comp rx=0 tx=0).stop 2024-03-18T19:09:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.081+0000 7fadbeaf4700 1 -- 172.21.15.1:0/405433591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadb8159620 msgr2=0x7fadb814e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.081+0000 7fadbeaf4700 1 --2- 172.21.15.1:0/405433591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadb8159620 0x7fadb814e120 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fada4009b40 tx=0x7fada4004750 comp rx=0 tx=0).stop 2024-03-18T19:09:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.081+0000 7fadbeaf4700 1 -- 172.21.15.1:0/405433591 shutdown_connections 2024-03-18T19:09:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.081+0000 7fadbeaf4700 1 --2- 172.21.15.1:0/405433591 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fad9c06adb0 0x7fad9c06d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.081+0000 7fadbeaf4700 1 --2- 172.21.15.1:0/405433591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fadb80b3980 0x7fadb814f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.081+0000 7fadbeaf4700 1 --2- 172.21.15.1:0/405433591 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fadb815e0f0 0x7fadb8151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.081+0000 7fadbeaf4700 1 --2- 172.21.15.1:0/405433591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadb8159620 0x7fadb814e120 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.081+0000 7fadbeaf4700 1 -- 172.21.15.1:0/405433591 >> 172.21.15.1:0/405433591 conn(0x7fadb806a750 msgr2=0x7fadb80afdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.081+0000 7fadbeaf4700 1 -- 172.21.15.1:0/405433591 shutdown_connections 2024-03-18T19:09:20.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.082+0000 7fadbeaf4700 1 -- 172.21.15.1:0/405433591 wait complete. 2024-03-18T19:09:20.089 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T19:09:20.090 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T19:09:20.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.223+0000 7f076bf52700 1 Processor -- start 2024-03-18T19:09:20.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.223+0000 7f076bf52700 1 -- start start 2024-03-18T19:09:20.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.224+0000 7f076bf52700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f076415dca0 0x7f076415a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:20.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.224+0000 7f076bf52700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f07640717b0 0x7f0764071c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:20.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.224+0000 7f076bf52700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076415ac60 con 0x7f076415e8d0 2024-03-18T19:09:20.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.224+0000 7f076bf52700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0764072180 con 0x7f076415dca0 2024-03-18T19:09:20.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.224+0000 7f076bf52700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07640701d0 con 0x7f07640717b0 2024-03-18T19:09:20.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.224+0000 7f07694ed700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f076415dca0 0x7f076415a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:20.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.224+0000 7f0769cee700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f076415e8d0 0x7f07640b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48350/0 (socket says 172.21.15.1:48350) 2024-03-18T19:09:20.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.224+0000 7f0769cee700 1 -- 172.21.15.1:0/4179358651 learned_addr learned my addr 172.21.15.1:0/4179358651 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:20.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.225+0000 7f076a4ef700 1 --2- 172.21.15.1:0/4179358651 >> v2:172.21.15.165:3300/0 conn(0x7f07640717b0 0x7f0764071c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:20.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.225+0000 7f07694ed700 1 -- 172.21.15.1:0/4179358651 >> v2:172.21.15.165:3300/0 conn(0x7f07640717b0 msgr2=0x7f0764071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:20.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.225+0000 7f07694ed700 1 --2- 172.21.15.1:0/4179358651 >> v2:172.21.15.165:3300/0 conn(0x7f07640717b0 0x7f0764071c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:20.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.225+0000 7f07694ed700 1 -- 172.21.15.1:0/4179358651 >> v1:172.21.15.1:6789/0 conn(0x7f076415e8d0 legacy=0x7f07640b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:20.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.225+0000 7f07694ed700 1 -- 172.21.15.1:0/4179358651 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07640b30b0 con 0x7f076415dca0 2024-03-18T19:09:20.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.225+0000 7f0768cec700 1 -- 172.21.15.1:0/4179358651 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4080294022 0 0) 0x7f076415ac60 con 0x7f076415e8d0 2024-03-18T19:09:20.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.225+0000 7f07694ed700 1 --2- 172.21.15.1:0/4179358651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f076415dca0 0x7f076415a720 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f0760009840 tx=0x7f0760009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e96cedfe374986fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:20.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.225+0000 7f0768cec700 1 -- 172.21.15.1:0/4179358651 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0760011830 con 0x7f076415dca0 2024-03-18T19:09:20.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.225+0000 7f0768cec700 1 -- 172.21.15.1:0/4179358651 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0760011990 con 0x7f076415dca0 2024-03-18T19:09:20.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.225+0000 7f0768cec700 1 -- 172.21.15.1:0/4179358651 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0760011d20 con 0x7f076415dca0 2024-03-18T19:09:20.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.225+0000 7f076bf52700 1 -- 172.21.15.1:0/4179358651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f076415dca0 msgr2=0x7f076415a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:20.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.225+0000 7f076bf52700 1 --2- 172.21.15.1:0/4179358651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f076415dca0 0x7f076415a720 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f0760009840 tx=0x7f0760009b50 comp rx=0 tx=0).stop 2024-03-18T19:09:20.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.226+0000 7f076bf52700 1 -- 172.21.15.1:0/4179358651 shutdown_connections 2024-03-18T19:09:20.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.226+0000 7f076bf52700 1 --2- 172.21.15.1:0/4179358651 >> v2:172.21.15.165:3300/0 conn(0x7f07640717b0 0x7f0764071c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:20.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.226+0000 7f076bf52700 1 --2- 172.21.15.1:0/4179358651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f076415dca0 0x7f076415a720 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:20.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.226+0000 7f076bf52700 1 -- 172.21.15.1:0/4179358651 >> 172.21.15.1:0/4179358651 conn(0x7f07640d0080 msgr2=0x7f07640d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:20.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.226+0000 7f076bf52700 1 -- 172.21.15.1:0/4179358651 shutdown_connections 2024-03-18T19:09:20.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.226+0000 7f076bf52700 1 -- 172.21.15.1:0/4179358651 wait complete. 2024-03-18T19:09:20.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.227+0000 7f076bf52700 1 Processor -- start 2024-03-18T19:09:20.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.227+0000 7f076bf52700 1 -- start start 2024-03-18T19:09:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f076bf52700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640717b0 0x7f0764145a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f076bf52700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f076415dca0 0x7f076415b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f076bf52700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f076415e8d0 0x7f076415bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f076bf52700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07641f3400 con 0x7f07640717b0 2024-03-18T19:09:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f076bf52700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07641e5150 con 0x7f076415dca0 2024-03-18T19:09:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f076bf52700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07641e5430 con 0x7f076415e8d0 2024-03-18T19:09:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f0769cee700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640717b0 0x7f0764145a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f0769cee700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640717b0 0x7f0764145a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43986/0 (socket says 172.21.15.1:43986) 2024-03-18T19:09:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f0769cee700 1 -- 172.21.15.1:0/3749068126 learned_addr learned my addr 172.21.15.1:0/3749068126 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f07694ed700 1 --2- 172.21.15.1:0/3749068126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f076415dca0 0x7f076415b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f076a4ef700 1 --2- 172.21.15.1:0/3749068126 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f076415e8d0 0x7f076415bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f0769cee700 1 -- 172.21.15.1:0/3749068126 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f076415e8d0 msgr2=0x7f076415bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f0769cee700 1 --2- 172.21.15.1:0/3749068126 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f076415e8d0 0x7f076415bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f0769cee700 1 -- 172.21.15.1:0/3749068126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f076415dca0 msgr2=0x7f076415b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f0769cee700 1 --2- 172.21.15.1:0/3749068126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f076415dca0 0x7f076415b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f0769cee700 1 -- 172.21.15.1:0/3749068126 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07600094a0 con 0x7f07640717b0 2024-03-18T19:09:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f076a4ef700 1 --2- 172.21.15.1:0/3749068126 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f076415e8d0 0x7f076415bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f0769cee700 1 --2- 172.21.15.1:0/3749068126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640717b0 0x7f0764145a70 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f075400c550 tx=0x7f075400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f075a7fc700 1 -- 172.21.15.1:0/3749068126 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0754004c80 con 0x7f07640717b0 2024-03-18T19:09:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f075a7fc700 1 -- 172.21.15.1:0/3749068126 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0754004e00 con 0x7f07640717b0 2024-03-18T19:09:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f076bf52700 1 -- 172.21.15.1:0/3749068126 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07641e5710 con 0x7f07640717b0 2024-03-18T19:09:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f075a7fc700 1 -- 172.21.15.1:0/3749068126 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0754007310 con 0x7f07640717b0 2024-03-18T19:09:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.228+0000 7f076bf52700 1 -- 172.21.15.1:0/3749068126 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0764145fb0 con 0x7f07640717b0 2024-03-18T19:09:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.229+0000 7f076bf52700 1 -- 172.21.15.1:0/3749068126 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07640cd070 con 0x7f07640717b0 2024-03-18T19:09:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.230+0000 7f075a7fc700 1 -- 172.21.15.1:0/3749068126 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0754007470 con 0x7f07640717b0 2024-03-18T19:09:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.230+0000 7f075a7fc700 1 --2- 172.21.15.1:0/3749068126 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f074806adb0 0x7f074806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.230+0000 7f075a7fc700 1 -- 172.21.15.1:0/3749068126 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f0754088f30 con 0x7f07640717b0 2024-03-18T19:09:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.230+0000 7f07694ed700 1 --2- 172.21.15.1:0/3749068126 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f074806adb0 0x7f074806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:20.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.231+0000 7f07694ed700 1 --2- 172.21.15.1:0/3749068126 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f074806adb0 0x7f074806d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0760009840 tx=0x7f0760008030 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:20.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.235+0000 7f075a7fc700 1 -- 172.21.15.1:0/3749068126 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0754054280 con 0x7f07640717b0 2024-03-18T19:09:20.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:20.437+0000 7f076bf52700 1 -- 172.21.15.1:0/3749068126 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f07641e5c30 con 0x7f07640717b0 2024-03-18T19:09:21.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.192+0000 7f075a7fc700 1 -- 172.21.15.1:0/3749068126 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v144) v1 ==== 138+0+0 (secure 0 0 0) 0x7f07641e5c30 con 0x7f07640717b0 2024-03-18T19:09:21.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.197+0000 7f076bf52700 1 -- 172.21.15.1:0/3749068126 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f074806adb0 msgr2=0x7f074806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:21.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.197+0000 7f076bf52700 1 --2- 172.21.15.1:0/3749068126 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f074806adb0 0x7f074806d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0760009840 tx=0x7f0760008030 comp rx=0 tx=0).stop 2024-03-18T19:09:21.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.197+0000 7f076bf52700 1 -- 172.21.15.1:0/3749068126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640717b0 msgr2=0x7f0764145a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:21.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.197+0000 7f076bf52700 1 --2- 172.21.15.1:0/3749068126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640717b0 0x7f0764145a70 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f075400c550 tx=0x7f075400c910 comp rx=0 tx=0).stop 2024-03-18T19:09:21.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.197+0000 7f076bf52700 1 -- 172.21.15.1:0/3749068126 shutdown_connections 2024-03-18T19:09:21.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.197+0000 7f076bf52700 1 --2- 172.21.15.1:0/3749068126 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f074806adb0 0x7f074806d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:21.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.197+0000 7f076bf52700 1 --2- 172.21.15.1:0/3749068126 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f076415e8d0 0x7f076415bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:21.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.197+0000 7f076bf52700 1 --2- 172.21.15.1:0/3749068126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640717b0 0x7f0764145a70 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:21.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.197+0000 7f076bf52700 1 --2- 172.21.15.1:0/3749068126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f076415dca0 0x7f076415b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:21.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.197+0000 7f076bf52700 1 -- 172.21.15.1:0/3749068126 >> 172.21.15.1:0/3749068126 conn(0x7f07640d0080 msgr2=0x7f076415c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:21.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.198+0000 7f076bf52700 1 -- 172.21.15.1:0/3749068126 shutdown_connections 2024-03-18T19:09:21.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.198+0000 7f076bf52700 1 -- 172.21.15.1:0/3749068126 wait complete. 2024-03-18T19:09:21.196 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-18T19:09:21.210 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T19:09:21.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.353+0000 7f3022279700 1 Processor -- start 2024-03-18T19:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.353+0000 7f3022279700 1 -- start start 2024-03-18T19:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.354+0000 7f3022279700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f301c146db0 0x7f301c0cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.354+0000 7f3022279700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f301c06b1f0 0x7f301c160fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.354+0000 7f3022279700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f301c147190 con 0x7f301c0b3430 2024-03-18T19:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.354+0000 7f3022279700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f301c161820 con 0x7f301c146db0 2024-03-18T19:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.354+0000 7f3022279700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f301c161b00 con 0x7f301c06b1f0 2024-03-18T19:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.354+0000 7f3020816700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f301c0b3430 0x7f301c06f510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48352/0 (socket says 172.21.15.1:48352) 2024-03-18T19:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.354+0000 7f3020816700 1 -- 172.21.15.1:0/4120770370 learned_addr learned my addr 172.21.15.1:0/4120770370 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.354+0000 7f301affd700 1 -- 172.21.15.1:0/4120770370 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537856346 0 0) 0x7f301c147190 con 0x7f301c0b3430 2024-03-18T19:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.354+0000 7f301affd700 1 -- 172.21.15.1:0/4120770370 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30080035b0 con 0x7f301c0b3430 2024-03-18T19:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.354+0000 7f301b7fe700 1 --2- 172.21.15.1:0/4120770370 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f301c06b1f0 0x7f301c160fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:21.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.354+0000 7f301bfff700 1 --2- 172.21.15.1:0/4120770370 >> v2:172.21.15.165:3300/0 conn(0x7f301c146db0 0x7f301c0cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.354+0000 7f301affd700 1 -- 172.21.15.1:0/4120770370 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3217405520 0 0) 0x7f30080035b0 con 0x7f301c0b3430 2024-03-18T19:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.354+0000 7f301affd700 1 -- 172.21.15.1:0/4120770370 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f301c06b1f0 msgr2=0x7f301c160fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.354+0000 7f301affd700 1 --2- 172.21.15.1:0/4120770370 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f301c06b1f0 0x7f301c160fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.354+0000 7f301affd700 1 -- 172.21.15.1:0/4120770370 >> v2:172.21.15.165:3300/0 conn(0x7f301c146db0 msgr2=0x7f301c0cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.354+0000 7f301affd700 1 --2- 172.21.15.1:0/4120770370 >> v2:172.21.15.165:3300/0 conn(0x7f301c146db0 0x7f301c0cca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.354+0000 7f301affd700 1 -- 172.21.15.1:0/4120770370 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c161de0 con 0x7f301c0b3430 2024-03-18T19:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.354+0000 7f301affd700 1 -- 172.21.15.1:0/4120770370 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f30140019f0 con 0x7f301c0b3430 2024-03-18T19:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.354+0000 7f301affd700 1 -- 172.21.15.1:0/4120770370 >> v1:172.21.15.1:6789/0 conn(0x7f301c0b3430 legacy=0x7f301c06f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.355+0000 7f301affd700 1 --2- 172.21.15.1:0/4120770370 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3008003a00 0x7f3008003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.355+0000 7f301affd700 1 --2- 172.21.15.1:0/4120770370 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f301c06b1f0 0x7f3008004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.355+0000 7f301affd700 1 --2- 172.21.15.1:0/4120770370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3008004d90 0x7f3008005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.355+0000 7f301affd700 1 -- 172.21.15.1:0/4120770370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301c161820 con 0x7f3008004d90 2024-03-18T19:09:21.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.355+0000 7f301affd700 1 -- 172.21.15.1:0/4120770370 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301c161b00 con 0x7f3008003a00 2024-03-18T19:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.355+0000 7f301affd700 1 -- 172.21.15.1:0/4120770370 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f300800ba10 con 0x7f301c06b1f0 2024-03-18T19:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.355+0000 7f3020816700 1 --2- 172.21.15.1:0/4120770370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3008004d90 0x7f3008005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.355+0000 7f301bfff700 1 --2- 172.21.15.1:0/4120770370 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3008003a00 0x7f3008003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.355+0000 7f301b7fe700 1 --2- 172.21.15.1:0/4120770370 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f301c06b1f0 0x7f3008004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.355+0000 7f3020816700 1 -- 172.21.15.1:0/4120770370 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f301c06b1f0 msgr2=0x7f3008004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.355+0000 7f3020816700 1 --2- 172.21.15.1:0/4120770370 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f301c06b1f0 0x7f3008004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.355+0000 7f3020816700 1 -- 172.21.15.1:0/4120770370 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3008003a00 msgr2=0x7f3008003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.355+0000 7f3020816700 1 --2- 172.21.15.1:0/4120770370 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3008003a00 0x7f3008003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.355+0000 7f3020816700 1 -- 172.21.15.1:0/4120770370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30080035b0 con 0x7f3008004d90 2024-03-18T19:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.355+0000 7f3020816700 1 --2- 172.21.15.1:0/4120770370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3008004d90 0x7f3008005210 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f301400a860 tx=0x7f3008005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a880739a70f94d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.355+0000 7f301affd700 1 -- 172.21.15.1:0/4120770370 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3014012600 con 0x7f3008004d90 2024-03-18T19:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.355+0000 7f301affd700 1 -- 172.21.15.1:0/4120770370 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3014012780 con 0x7f3008004d90 2024-03-18T19:09:21.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.355+0000 7f301affd700 1 -- 172.21.15.1:0/4120770370 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3014012b30 con 0x7f3008004d90 2024-03-18T19:09:21.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.356+0000 7f3022279700 1 -- 172.21.15.1:0/4120770370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3008004d90 msgr2=0x7f3008005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:21.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.356+0000 7f3022279700 1 --2- 172.21.15.1:0/4120770370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3008004d90 0x7f3008005210 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f301400a860 tx=0x7f3008005880 comp rx=0 tx=0).stop 2024-03-18T19:09:21.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.356+0000 7f3020816700 1 -- 172.21.15.1:0/4120770370 reap_dead start 2024-03-18T19:09:21.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.356+0000 7f3022279700 1 -- 172.21.15.1:0/4120770370 shutdown_connections 2024-03-18T19:09:21.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.356+0000 7f3022279700 1 -- 172.21.15.1:0/4120770370 >> 172.21.15.1:0/4120770370 conn(0x7f301c0d0070 msgr2=0x7f301c0d0480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:21.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.356+0000 7f3022279700 1 -- 172.21.15.1:0/4120770370 shutdown_connections 2024-03-18T19:09:21.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.356+0000 7f3022279700 1 -- 172.21.15.1:0/4120770370 wait complete. 2024-03-18T19:09:21.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.357+0000 7f3022279700 1 Processor -- start 2024-03-18T19:09:21.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.357+0000 7f3022279700 1 -- start start 2024-03-18T19:09:21.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.358+0000 7f3022279700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f301c15cdb0 0x7f301c0af3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:21.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.358+0000 7f3022279700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f301c0ad2c0 0x7f301c15f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:21.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.358+0000 7f3022279700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f301c15fd10 0x7f301c15d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:21.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.358+0000 7f3022279700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301c0af900 con 0x7f301c15fd10 2024-03-18T19:09:21.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.358+0000 7f3022279700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301c160190 con 0x7f301c15cdb0 2024-03-18T19:09:21.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.358+0000 7f3022279700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f301c139100 con 0x7f301c0ad2c0 2024-03-18T19:09:21.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.358+0000 7f3020816700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f301c15fd10 0x7f301c15d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:21.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.358+0000 7f3020816700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f301c15fd10 0x7f301c15d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44002/0 (socket says 172.21.15.1:44002) 2024-03-18T19:09:21.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.358+0000 7f3020816700 1 -- 172.21.15.1:0/3970225509 learned_addr learned my addr 172.21.15.1:0/3970225509 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:21.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.358+0000 7f301bfff700 1 --2- 172.21.15.1:0/3970225509 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f301c15cdb0 0x7f301c0af3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:21.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.358+0000 7f301b7fe700 1 --2- 172.21.15.1:0/3970225509 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f301c0ad2c0 0x7f301c15f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:21.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.359+0000 7f3020816700 1 -- 172.21.15.1:0/3970225509 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f301c0ad2c0 msgr2=0x7f301c15f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:21.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.359+0000 7f3020816700 1 --2- 172.21.15.1:0/3970225509 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f301c0ad2c0 0x7f301c15f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:21.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.359+0000 7f3020816700 1 -- 172.21.15.1:0/3970225509 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f301c15cdb0 msgr2=0x7f301c0af3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:21.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.359+0000 7f3020816700 1 --2- 172.21.15.1:0/3970225509 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f301c15cdb0 0x7f301c0af3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:21.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.359+0000 7f3020816700 1 -- 172.21.15.1:0/3970225509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c0ada30 con 0x7f301c15fd10 2024-03-18T19:09:21.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.359+0000 7f3020816700 1 --2- 172.21.15.1:0/3970225509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f301c15fd10 0x7f301c15d8f0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f3014001cb0 tx=0x7f3014001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:21.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.359+0000 7f3018ff9700 1 -- 172.21.15.1:0/3970225509 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30140019c0 con 0x7f301c15fd10 2024-03-18T19:09:21.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.359+0000 7f3018ff9700 1 -- 172.21.15.1:0/3970225509 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3014027410 con 0x7f301c15fd10 2024-03-18T19:09:21.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.359+0000 7f3022279700 1 -- 172.21.15.1:0/3970225509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f301c0add10 con 0x7f301c15fd10 2024-03-18T19:09:21.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.359+0000 7f3018ff9700 1 -- 172.21.15.1:0/3970225509 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30140277e0 con 0x7f301c15fd10 2024-03-18T19:09:21.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.359+0000 7f3022279700 1 -- 172.21.15.1:0/3970225509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f301c0ae2d0 con 0x7f301c15fd10 2024-03-18T19:09:21.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.360+0000 7f3022279700 1 -- 172.21.15.1:0/3970225509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f301c138ed0 con 0x7f301c15fd10 2024-03-18T19:09:21.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.361+0000 7f3018ff9700 1 -- 172.21.15.1:0/3970225509 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3014019070 con 0x7f301c15fd10 2024-03-18T19:09:21.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.361+0000 7f3018ff9700 1 --2- 172.21.15.1:0/3970225509 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2ffc06ade0 0x7f2ffc06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:21.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.361+0000 7f3018ff9700 1 -- 172.21.15.1:0/3970225509 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f301408a8c0 con 0x7f301c15fd10 2024-03-18T19:09:21.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.361+0000 7f301bfff700 1 --2- 172.21.15.1:0/3970225509 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2ffc06ade0 0x7f2ffc06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:21.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.361+0000 7f301bfff700 1 --2- 172.21.15.1:0/3970225509 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2ffc06ade0 0x7f2ffc06d2a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f300c009c60 tx=0x7f300c009210 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:21.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.365+0000 7f3018ff9700 1 -- 172.21.15.1:0/3970225509 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3014055c40 con 0x7f301c15fd10 2024-03-18T19:09:21.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:21.565+0000 7f3022279700 1 -- 172.21.15.1:0/3970225509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f301c138ed0 con 0x7f301c15fd10 2024-03-18T19:09:21.998 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:09:22.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:09:22.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.195+0000 7f3018ff9700 1 -- 172.21.15.1:0/3970225509 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v145) v1 ==== 170+0+0 (secure 0 0 0) 0x7f301c138ed0 con 0x7f301c15fd10 2024-03-18T19:09:22.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.202+0000 7f3022279700 1 -- 172.21.15.1:0/3970225509 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2ffc06ade0 msgr2=0x7f2ffc06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:22.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.202+0000 7f3022279700 1 --2- 172.21.15.1:0/3970225509 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2ffc06ade0 0x7f2ffc06d2a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f300c009c60 tx=0x7f300c009210 comp rx=0 tx=0).stop 2024-03-18T19:09:22.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.202+0000 7f3022279700 1 -- 172.21.15.1:0/3970225509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f301c15fd10 msgr2=0x7f301c15d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:22.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.202+0000 7f3022279700 1 --2- 172.21.15.1:0/3970225509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f301c15fd10 0x7f301c15d8f0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f3014001cb0 tx=0x7f3014001d80 comp rx=0 tx=0).stop 2024-03-18T19:09:22.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.202+0000 7f3022279700 1 -- 172.21.15.1:0/3970225509 shutdown_connections 2024-03-18T19:09:22.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.202+0000 7f3022279700 1 --2- 172.21.15.1:0/3970225509 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2ffc06ade0 0x7f2ffc06d2a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.202+0000 7f3022279700 1 --2- 172.21.15.1:0/3970225509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f301c15fd10 0x7f301c15d8f0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.202+0000 7f3022279700 1 --2- 172.21.15.1:0/3970225509 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f301c15cdb0 0x7f301c0af3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.202+0000 7f3022279700 1 --2- 172.21.15.1:0/3970225509 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f301c0ad2c0 0x7f301c15f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.202+0000 7f3022279700 1 -- 172.21.15.1:0/3970225509 >> 172.21.15.1:0/3970225509 conn(0x7f301c0d0070 msgr2=0x7f301c0b07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:22.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.204+0000 7f3022279700 1 -- 172.21.15.1:0/3970225509 shutdown_connections 2024-03-18T19:09:22.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.204+0000 7f3022279700 1 -- 172.21.15.1:0/3970225509 wait complete. 2024-03-18T19:09:22.209 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-18T19:09:22.223 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T19:09:22.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.361+0000 7f60ca585700 1 Processor -- start 2024-03-18T19:09:22.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.361+0000 7f60ca585700 1 -- start start 2024-03-18T19:09:22.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.362+0000 7f60ca585700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60c40b2cf0 0x7f60c415d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:22.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.362+0000 7f60ca585700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c406d300 con 0x7f60c415eb40 2024-03-18T19:09:22.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.362+0000 7f60ca585700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c4071600 con 0x7f60c4157be0 2024-03-18T19:09:22.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.362+0000 7f60ca585700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c40b3450 con 0x7f60c40b2cf0 2024-03-18T19:09:22.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.362+0000 7f60c37fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f60c415eb40 0x7f60c40b25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48358/0 (socket says 172.21.15.1:48358) 2024-03-18T19:09:22.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.362+0000 7f60c37fe700 1 -- 172.21.15.1:0/2650047173 learned_addr learned my addr 172.21.15.1:0/2650047173 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:22.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c8b22700 1 --2- 172.21.15.1:0/2650047173 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60c40b2cf0 0x7f60c415d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:22.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c2ffd700 1 -- 172.21.15.1:0/2650047173 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286574295 0 0) 0x7f60c406d300 con 0x7f60c415eb40 2024-03-18T19:09:22.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c2ffd700 1 -- 172.21.15.1:0/2650047173 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60ac0035b0 con 0x7f60c415eb40 2024-03-18T19:09:22.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c2ffd700 1 -- 172.21.15.1:0/2650047173 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 121471403 0 0) 0x7f60ac0035b0 con 0x7f60c415eb40 2024-03-18T19:09:22.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c2ffd700 1 -- 172.21.15.1:0/2650047173 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60c40b2cf0 msgr2=0x7f60c415d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:22.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c2ffd700 1 --2- 172.21.15.1:0/2650047173 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60c40b2cf0 0x7f60c415d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c2ffd700 1 -- 172.21.15.1:0/2650047173 >> v1:172.21.15.165:6789/0 conn(0x7f60c4157be0 legacy=0x7f60c4070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:22.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c2ffd700 1 -- 172.21.15.1:0/2650047173 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c40718e0 con 0x7f60c415eb40 2024-03-18T19:09:22.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c2ffd700 1 -- 172.21.15.1:0/2650047173 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f60b4001670 con 0x7f60c415eb40 2024-03-18T19:09:22.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c2ffd700 1 -- 172.21.15.1:0/2650047173 >> v1:172.21.15.1:6789/0 conn(0x7f60c415eb40 legacy=0x7f60c40b25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:22.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c2ffd700 1 --2- 172.21.15.1:0/2650047173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60ac003a30 0x7f60ac003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:22.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c2ffd700 1 --2- 172.21.15.1:0/2650047173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60ac004850 0x7f60ac004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:22.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c2ffd700 1 --2- 172.21.15.1:0/2650047173 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60c40b2cf0 0x7f60ac005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:22.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c2ffd700 1 -- 172.21.15.1:0/2650047173 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c40b3450 con 0x7f60ac004850 2024-03-18T19:09:22.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c2ffd700 1 -- 172.21.15.1:0/2650047173 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac00baf0 con 0x7f60ac003a30 2024-03-18T19:09:22.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c2ffd700 1 -- 172.21.15.1:0/2650047173 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac00bcb0 con 0x7f60c40b2cf0 2024-03-18T19:09:22.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c37fe700 1 --2- 172.21.15.1:0/2650047173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60ac004850 0x7f60ac004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:22.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c3fff700 1 --2- 172.21.15.1:0/2650047173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60ac003a30 0x7f60ac003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:22.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c8b22700 1 --2- 172.21.15.1:0/2650047173 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60c40b2cf0 0x7f60ac005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:22.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c37fe700 1 -- 172.21.15.1:0/2650047173 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60c40b2cf0 msgr2=0x7f60ac005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:22.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c37fe700 1 --2- 172.21.15.1:0/2650047173 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60c40b2cf0 0x7f60ac005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c37fe700 1 -- 172.21.15.1:0/2650047173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60ac003a30 msgr2=0x7f60ac003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:22.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.363+0000 7f60c37fe700 1 --2- 172.21.15.1:0/2650047173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60ac003a30 0x7f60ac003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.364+0000 7f60c37fe700 1 -- 172.21.15.1:0/2650047173 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac0035b0 con 0x7f60ac004850 2024-03-18T19:09:22.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.364+0000 7f60c37fe700 1 --2- 172.21.15.1:0/2650047173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60ac004850 0x7f60ac004cd0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f60b400c270 tx=0x7f60ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86920df9a3610c33 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:22.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.364+0000 7f60c2ffd700 1 -- 172.21.15.1:0/2650047173 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b4007c10 con 0x7f60ac004850 2024-03-18T19:09:22.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.364+0000 7f60c2ffd700 1 -- 172.21.15.1:0/2650047173 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b400cd90 con 0x7f60ac004850 2024-03-18T19:09:22.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.364+0000 7f60c2ffd700 1 -- 172.21.15.1:0/2650047173 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b4005ba0 con 0x7f60ac004850 2024-03-18T19:09:22.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.364+0000 7f60ca585700 1 -- 172.21.15.1:0/2650047173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60ac004850 msgr2=0x7f60ac004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:22.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.364+0000 7f60ca585700 1 --2- 172.21.15.1:0/2650047173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60ac004850 0x7f60ac004cd0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f60b400c270 tx=0x7f60ac005880 comp rx=0 tx=0).stop 2024-03-18T19:09:22.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.364+0000 7f60c8b22700 1 -- 172.21.15.1:0/2650047173 reap_dead start 2024-03-18T19:09:22.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.364+0000 7f60ca585700 1 -- 172.21.15.1:0/2650047173 shutdown_connections 2024-03-18T19:09:22.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.364+0000 7f60ca585700 1 -- 172.21.15.1:0/2650047173 >> 172.21.15.1:0/2650047173 conn(0x7f60c406a750 msgr2=0x7f60c406ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:22.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.364+0000 7f60ca585700 1 -- 172.21.15.1:0/2650047173 shutdown_connections 2024-03-18T19:09:22.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.364+0000 7f60ca585700 1 -- 172.21.15.1:0/2650047173 wait complete. 2024-03-18T19:09:22.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.366+0000 7f60ca585700 1 Processor -- start 2024-03-18T19:09:22.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.366+0000 7f60ca585700 1 -- start start 2024-03-18T19:09:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60ca585700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60ac004850 0x7f60c41439a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60ca585700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60c414fdd0 0x7f60c41f30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60ca585700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60c415f710 0x7f60c415fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60ca585700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c40cbb10 con 0x7f60ac004850 2024-03-18T19:09:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60ca585700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c41600e0 con 0x7f60c414fdd0 2024-03-18T19:09:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60ca585700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c41f3630 con 0x7f60c415f710 2024-03-18T19:09:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60c3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60ac004850 0x7f60c41439a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60c3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60ac004850 0x7f60c41439a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44012/0 (socket says 172.21.15.1:44012) 2024-03-18T19:09:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60c3fff700 1 -- 172.21.15.1:0/3693427058 learned_addr learned my addr 172.21.15.1:0/3693427058 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60c8b22700 1 --2- 172.21.15.1:0/3693427058 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60c415f710 0x7f60c415fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60c37fe700 1 --2- 172.21.15.1:0/3693427058 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60c414fdd0 0x7f60c41f30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60c3fff700 1 -- 172.21.15.1:0/3693427058 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60c415f710 msgr2=0x7f60c415fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60c3fff700 1 --2- 172.21.15.1:0/3693427058 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60c415f710 0x7f60c415fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60c3fff700 1 -- 172.21.15.1:0/3693427058 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60c414fdd0 msgr2=0x7f60c41f30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60c3fff700 1 --2- 172.21.15.1:0/3693427058 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60c414fdd0 0x7f60c41f30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60c3fff700 1 -- 172.21.15.1:0/3693427058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c41505b0 con 0x7f60ac004850 2024-03-18T19:09:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60c8b22700 1 --2- 172.21.15.1:0/3693427058 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60c415f710 0x7f60c415fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60c3fff700 1 --2- 172.21.15.1:0/3693427058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60ac004850 0x7f60c41439a0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f60b800f7f0 tx=0x7f60b800fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60c0ff9700 1 -- 172.21.15.1:0/3693427058 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b80052e0 con 0x7f60ac004850 2024-03-18T19:09:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60c0ff9700 1 -- 172.21.15.1:0/3693427058 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b8005440 con 0x7f60ac004850 2024-03-18T19:09:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60ca585700 1 -- 172.21.15.1:0/3693427058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c4150890 con 0x7f60ac004850 2024-03-18T19:09:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60c0ff9700 1 -- 172.21.15.1:0/3693427058 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b80038a0 con 0x7f60ac004850 2024-03-18T19:09:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.367+0000 7f60ca585700 1 -- 172.21.15.1:0/3693427058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c4151aa0 con 0x7f60ac004850 2024-03-18T19:09:22.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.368+0000 7f60ca585700 1 -- 172.21.15.1:0/3693427058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c4150db0 con 0x7f60ac004850 2024-03-18T19:09:22.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.369+0000 7f60c0ff9700 1 -- 172.21.15.1:0/3693427058 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f60b8003a00 con 0x7f60ac004850 2024-03-18T19:09:22.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.370+0000 7f60c0ff9700 1 --2- 172.21.15.1:0/3693427058 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f60b006adb0 0x7f60b006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:22.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.370+0000 7f60c0ff9700 1 -- 172.21.15.1:0/3693427058 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f60b8089bb0 con 0x7f60ac004850 2024-03-18T19:09:22.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.370+0000 7f60c37fe700 1 --2- 172.21.15.1:0/3693427058 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f60b006adb0 0x7f60b006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:22.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.370+0000 7f60c37fe700 1 --2- 172.21.15.1:0/3693427058 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f60b006adb0 0x7f60b006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f60b4015040 tx=0x7f60b4007000 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:22.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.374+0000 7f60c0ff9700 1 -- 172.21.15.1:0/3693427058 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f60b8054c60 con 0x7f60ac004850 2024-03-18T19:09:22.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.622+0000 7f60ca585700 1 -- 172.21.15.1:0/3693427058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f60c4151900 con 0x7f60ac004850 2024-03-18T19:09:22.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.626+0000 7f60c0ff9700 1 -- 172.21.15.1:0/3693427058 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v61) v1 ==== 174+0+0 (secure 0 0 0) 0x7f60c4151900 con 0x7f60ac004850 2024-03-18T19:09:22.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.634+0000 7f60ca585700 1 -- 172.21.15.1:0/3693427058 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f60b006adb0 msgr2=0x7f60b006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:22.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.634+0000 7f60ca585700 1 --2- 172.21.15.1:0/3693427058 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f60b006adb0 0x7f60b006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f60b4015040 tx=0x7f60b4007000 comp rx=0 tx=0).stop 2024-03-18T19:09:22.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.634+0000 7f60ca585700 1 -- 172.21.15.1:0/3693427058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60ac004850 msgr2=0x7f60c41439a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:22.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.634+0000 7f60ca585700 1 --2- 172.21.15.1:0/3693427058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60ac004850 0x7f60c41439a0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f60b800f7f0 tx=0x7f60b800fbb0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.636+0000 7f60ca585700 1 -- 172.21.15.1:0/3693427058 shutdown_connections 2024-03-18T19:09:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.636+0000 7f60ca585700 1 --2- 172.21.15.1:0/3693427058 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f60b006adb0 0x7f60b006d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.636+0000 7f60ca585700 1 --2- 172.21.15.1:0/3693427058 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f60c415f710 0x7f60c415fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.636+0000 7f60ca585700 1 --2- 172.21.15.1:0/3693427058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f60ac004850 0x7f60c41439a0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.636+0000 7f60ca585700 1 --2- 172.21.15.1:0/3693427058 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60c414fdd0 0x7f60c41f30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.636+0000 7f60ca585700 1 -- 172.21.15.1:0/3693427058 >> 172.21.15.1:0/3693427058 conn(0x7f60c406a750 msgr2=0x7f60c40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:22.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.640+0000 7f60ca585700 1 -- 172.21.15.1:0/3693427058 shutdown_connections 2024-03-18T19:09:22.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.642+0000 7f60ca585700 1 -- 172.21.15.1:0/3693427058 wait complete. 2024-03-18T19:09:22.645 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 17 and data pool 18 2024-03-18T19:09:22.659 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:09:22.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.788+0000 7fc646431700 1 Processor -- start 2024-03-18T19:09:22.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.788+0000 7fc646431700 1 -- start start 2024-03-18T19:09:22.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.789+0000 7fc646431700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc6400ae600 0x7fc6400ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:22.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.789+0000 7fc646431700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc64015a2e0 0x7fc6400ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:22.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.789+0000 7fc646431700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc64006e270 con 0x7fc6400cc170 2024-03-18T19:09:22.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.789+0000 7fc646431700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6400ae050 con 0x7fc64015a2e0 2024-03-18T19:09:22.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.789+0000 7fc646431700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc64006f690 con 0x7fc6400ae600 2024-03-18T19:09:22.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.789+0000 7fc63f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc64015a2e0 0x7fc6400ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:22.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.789+0000 7fc63f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc64015a2e0 0x7fc6400ae9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44026/0 (socket says 172.21.15.1:44026) 2024-03-18T19:09:22.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.789+0000 7fc6449ce700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc6400cc170 0x7fc64006ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48364/0 (socket says 172.21.15.1:48364) 2024-03-18T19:09:22.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.789+0000 7fc63f7fe700 1 -- 172.21.15.1:0/2686050194 learned_addr learned my addr 172.21.15.1:0/2686050194 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:22.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.789+0000 7fc63ffff700 1 --2- 172.21.15.1:0/2686050194 >> v2:172.21.15.165:3300/0 conn(0x7fc6400ae600 0x7fc6400ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:22.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.789+0000 7fc63f7fe700 1 -- 172.21.15.1:0/2686050194 >> v2:172.21.15.165:3300/0 conn(0x7fc6400ae600 msgr2=0x7fc6400ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:22.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.789+0000 7fc63f7fe700 1 --2- 172.21.15.1:0/2686050194 >> v2:172.21.15.165:3300/0 conn(0x7fc6400ae600 0x7fc6400ad790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.789+0000 7fc63f7fe700 1 -- 172.21.15.1:0/2686050194 >> v1:172.21.15.1:6789/0 conn(0x7fc6400cc170 legacy=0x7fc64006ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:22.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.789+0000 7fc63effd700 1 -- 172.21.15.1:0/2686050194 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949626340 0 0) 0x7fc64006e270 con 0x7fc6400cc170 2024-03-18T19:09:22.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.790+0000 7fc63f7fe700 1 -- 172.21.15.1:0/2686050194 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6400aef20 con 0x7fc64015a2e0 2024-03-18T19:09:22.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.790+0000 7fc63ffff700 1 --2- 172.21.15.1:0/2686050194 >> v2:172.21.15.165:3300/0 conn(0x7fc6400ae600 0x7fc6400ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:22.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.790+0000 7fc63f7fe700 1 --2- 172.21.15.1:0/2686050194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc64015a2e0 0x7fc6400ae9e0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fc64004ebb0 tx=0x7fc634009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcae5b2d431278fa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:22.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.790+0000 7fc63effd700 1 -- 172.21.15.1:0/2686050194 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63400b870 con 0x7fc64015a2e0 2024-03-18T19:09:22.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.790+0000 7fc63effd700 1 -- 172.21.15.1:0/2686050194 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63400b9d0 con 0x7fc64015a2e0 2024-03-18T19:09:22.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.790+0000 7fc63effd700 1 -- 172.21.15.1:0/2686050194 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63400bd60 con 0x7fc64015a2e0 2024-03-18T19:09:22.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.790+0000 7fc646431700 1 -- 172.21.15.1:0/2686050194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc64015a2e0 msgr2=0x7fc6400ae9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:22.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.790+0000 7fc646431700 1 --2- 172.21.15.1:0/2686050194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc64015a2e0 0x7fc6400ae9e0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fc64004ebb0 tx=0x7fc634009bd0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.790+0000 7fc646431700 1 -- 172.21.15.1:0/2686050194 shutdown_connections 2024-03-18T19:09:22.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.790+0000 7fc646431700 1 --2- 172.21.15.1:0/2686050194 >> v2:172.21.15.165:3300/0 conn(0x7fc6400ae600 0x7fc6400ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.790+0000 7fc646431700 1 --2- 172.21.15.1:0/2686050194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc64015a2e0 0x7fc6400ae9e0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.790+0000 7fc646431700 1 -- 172.21.15.1:0/2686050194 >> 172.21.15.1:0/2686050194 conn(0x7fc64006b860 msgr2=0x7fc64006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:22.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.790+0000 7fc646431700 1 -- 172.21.15.1:0/2686050194 shutdown_connections 2024-03-18T19:09:22.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.790+0000 7fc646431700 1 -- 172.21.15.1:0/2686050194 wait complete. 2024-03-18T19:09:22.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.792+0000 7fc646431700 1 Processor -- start 2024-03-18T19:09:22.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.792+0000 7fc646431700 1 -- start start 2024-03-18T19:09:22.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.793+0000 7fc646431700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6400ae600 0x7fc64014aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:22.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.793+0000 7fc646431700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6400cc170 0x7fc6400cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:22.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.793+0000 7fc646431700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc64015a2e0 0x7fc640158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:22.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.793+0000 7fc646431700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64014afc0 con 0x7fc6400cc170 2024-03-18T19:09:22.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.793+0000 7fc646431700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6401f13d0 con 0x7fc64015a2e0 2024-03-18T19:09:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.793+0000 7fc646431700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6401f16b0 con 0x7fc6400ae600 2024-03-18T19:09:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.793+0000 7fc63f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6400cc170 0x7fc6400cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.793+0000 7fc63f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6400cc170 0x7fc6400cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44036/0 (socket says 172.21.15.1:44036) 2024-03-18T19:09:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.793+0000 7fc63f7fe700 1 -- 172.21.15.1:0/4204220998 learned_addr learned my addr 172.21.15.1:0/4204220998 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.793+0000 7fc6449ce700 1 --2- 172.21.15.1:0/4204220998 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc64015a2e0 0x7fc640158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.793+0000 7fc63ffff700 1 --2- 172.21.15.1:0/4204220998 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6400ae600 0x7fc64014aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.793+0000 7fc63f7fe700 1 -- 172.21.15.1:0/4204220998 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6400ae600 msgr2=0x7fc64014aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.794+0000 7fc63f7fe700 1 --2- 172.21.15.1:0/4204220998 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6400ae600 0x7fc64014aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.794+0000 7fc63f7fe700 1 -- 172.21.15.1:0/4204220998 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc64015a2e0 msgr2=0x7fc640158710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.794+0000 7fc63f7fe700 1 --2- 172.21.15.1:0/4204220998 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc64015a2e0 0x7fc640158710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.794+0000 7fc63f7fe700 1 -- 172.21.15.1:0/4204220998 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc634009550 con 0x7fc6400cc170 2024-03-18T19:09:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.794+0000 7fc63ffff700 1 --2- 172.21.15.1:0/4204220998 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6400ae600 0x7fc64014aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:22.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.794+0000 7fc63f7fe700 1 --2- 172.21.15.1:0/4204220998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6400cc170 0x7fc6400cf4d0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fc634014040 tx=0x7fc634004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.794+0000 7fc63cff9700 1 -- 172.21.15.1:0/4204220998 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc634004220 con 0x7fc6400cc170 2024-03-18T19:09:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.794+0000 7fc63cff9700 1 -- 172.21.15.1:0/4204220998 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63401e070 con 0x7fc6400cc170 2024-03-18T19:09:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.794+0000 7fc63cff9700 1 -- 172.21.15.1:0/4204220998 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc634004580 con 0x7fc6400cc170 2024-03-18T19:09:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.794+0000 7fc646431700 1 -- 172.21.15.1:0/4204220998 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6401f1990 con 0x7fc6400cc170 2024-03-18T19:09:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.794+0000 7fc646431700 1 -- 172.21.15.1:0/4204220998 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6401f1f50 con 0x7fc6400cc170 2024-03-18T19:09:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.795+0000 7fc646431700 1 -- 172.21.15.1:0/4204220998 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc640072490 con 0x7fc6400cc170 2024-03-18T19:09:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.795+0000 7fc63cff9700 1 -- 172.21.15.1:0/4204220998 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc634012800 con 0x7fc6400cc170 2024-03-18T19:09:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.796+0000 7fc63cff9700 1 --2- 172.21.15.1:0/4204220998 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc62c06adb0 0x7fc62c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.796+0000 7fc63cff9700 1 -- 172.21.15.1:0/4204220998 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc63408a840 con 0x7fc6400cc170 2024-03-18T19:09:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.796+0000 7fc63ffff700 1 --2- 172.21.15.1:0/4204220998 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc62c06adb0 0x7fc62c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:22.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.796+0000 7fc63ffff700 1 --2- 172.21.15.1:0/4204220998 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc62c06adb0 0x7fc62c06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc6300089b0 tx=0x7fc630008040 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:22.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:22.801+0000 7fc63cff9700 1 -- 172.21.15.1:0/4204220998 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc634055840 con 0x7fc6400cc170 2024-03-18T19:09:22.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.003+0000 7fc646431700 1 -- 172.21.15.1:0/4204220998 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc640073350 con 0x7fc6400cc170 2024-03-18T19:09:23.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.004+0000 7fc63cff9700 1 -- 172.21.15.1:0/4204220998 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+16325 (secure 0 0 0) 0x7fc634020090 con 0x7fc6400cc170 2024-03-18T19:09:23.000 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:23.000 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":146,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:09:22.623112+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:09:20.468744+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-18T19:09:21.597476+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:09:23.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.006+0000 7fc646431700 1 -- 172.21.15.1:0/4204220998 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc62c06adb0 msgr2=0x7fc62c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:23.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.006+0000 7fc646431700 1 --2- 172.21.15.1:0/4204220998 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc62c06adb0 0x7fc62c06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc6300089b0 tx=0x7fc630008040 comp rx=0 tx=0).stop 2024-03-18T19:09:23.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.006+0000 7fc646431700 1 -- 172.21.15.1:0/4204220998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6400cc170 msgr2=0x7fc6400cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:23.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.006+0000 7fc646431700 1 --2- 172.21.15.1:0/4204220998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6400cc170 0x7fc6400cf4d0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fc634014040 tx=0x7fc634004750 comp rx=0 tx=0).stop 2024-03-18T19:09:23.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.006+0000 7fc646431700 1 -- 172.21.15.1:0/4204220998 shutdown_connections 2024-03-18T19:09:23.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.006+0000 7fc646431700 1 --2- 172.21.15.1:0/4204220998 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc62c06adb0 0x7fc62c06d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:23.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.006+0000 7fc646431700 1 --2- 172.21.15.1:0/4204220998 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6400ae600 0x7fc64014aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:23.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.007+0000 7fc646431700 1 --2- 172.21.15.1:0/4204220998 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc64015a2e0 0x7fc640158710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:23.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.007+0000 7fc646431700 1 --2- 172.21.15.1:0/4204220998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6400cc170 0x7fc6400cf4d0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:23.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.007+0000 7fc646431700 1 -- 172.21.15.1:0/4204220998 >> 172.21.15.1:0/4204220998 conn(0x7fc64006b860 msgr2=0x7fc6400b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:23.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.007+0000 7fc646431700 1 -- 172.21.15.1:0/4204220998 shutdown_connections 2024-03-18T19:09:23.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.007+0000 7fc646431700 1 -- 172.21.15.1:0/4204220998 wait complete. 2024-03-18T19:09:23.015 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:09:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.144+0000 7f7f4f3af700 1 Processor -- start 2024-03-18T19:09:23.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.144+0000 7f7f4f3af700 1 -- start start 2024-03-18T19:09:23.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.145+0000 7f7f4f3af700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7f48146190 0x7f7f4806a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:23.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.145+0000 7f7f4f3af700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f48146af0 0x7f7f4806ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:23.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.145+0000 7f7f4f3af700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4806ebd0 con 0x7f7f481465a0 2024-03-18T19:09:23.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.145+0000 7f7f4f3af700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f4806f850 con 0x7f7f48146190 2024-03-18T19:09:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.145+0000 7f7f4f3af700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4806eeb0 con 0x7f7f48146af0 2024-03-18T19:09:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.145+0000 7f7f4d14b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7f48146190 0x7f7f4806a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.145+0000 7f7f4d14b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7f48146190 0x7f7f4806a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44052/0 (socket says 172.21.15.1:44052) 2024-03-18T19:09:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.145+0000 7f7f4d14b700 1 -- 172.21.15.1:0/295016711 learned_addr learned my addr 172.21.15.1:0/295016711 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.145+0000 7f7f4d94c700 1 --2- 172.21.15.1:0/295016711 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f48146af0 0x7f7f4806ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.145+0000 7f7f4d14b700 1 -- 172.21.15.1:0/295016711 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f48146af0 msgr2=0x7f7f4806ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.145+0000 7f7f4d14b700 1 --2- 172.21.15.1:0/295016711 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f48146af0 0x7f7f4806ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.145+0000 7f7f4d14b700 1 -- 172.21.15.1:0/295016711 >> v1:172.21.15.1:6789/0 conn(0x7f7f481465a0 legacy=0x7f7f48071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.145+0000 7f7f4d14b700 1 -- 172.21.15.1:0/295016711 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4806f190 con 0x7f7f48146190 2024-03-18T19:09:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.145+0000 7f7f3ffff700 1 -- 172.21.15.1:0/295016711 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921934154 0 0) 0x7f7f4806ebd0 con 0x7f7f481465a0 2024-03-18T19:09:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.145+0000 7f7f4d14b700 1 --2- 172.21.15.1:0/295016711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f48146190 0x7f7f4806a750 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f7f440098f0 tx=0x7f7f44009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b73a8de7ed9ffcf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.145+0000 7f7f3ffff700 1 -- 172.21.15.1:0/295016711 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4400b8a0 con 0x7f7f48146190 2024-03-18T19:09:23.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.146+0000 7f7f3ffff700 1 -- 172.21.15.1:0/295016711 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4400ba00 con 0x7f7f48146190 2024-03-18T19:09:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.146+0000 7f7f3ffff700 1 -- 172.21.15.1:0/295016711 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4400bd90 con 0x7f7f48146190 2024-03-18T19:09:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.146+0000 7f7f4f3af700 1 -- 172.21.15.1:0/295016711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f48146190 msgr2=0x7f7f4806a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.146+0000 7f7f4f3af700 1 --2- 172.21.15.1:0/295016711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f48146190 0x7f7f4806a750 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f7f440098f0 tx=0x7f7f44009c00 comp rx=0 tx=0).stop 2024-03-18T19:09:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.146+0000 7f7f4f3af700 1 -- 172.21.15.1:0/295016711 shutdown_connections 2024-03-18T19:09:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.146+0000 7f7f4f3af700 1 --2- 172.21.15.1:0/295016711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f48146190 0x7f7f4806a750 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.146+0000 7f7f4f3af700 1 --2- 172.21.15.1:0/295016711 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f48146af0 0x7f7f4806ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.146+0000 7f7f4f3af700 1 -- 172.21.15.1:0/295016711 >> 172.21.15.1:0/295016711 conn(0x7f7f480b1d20 msgr2=0x7f7f480b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.146+0000 7f7f4f3af700 1 -- 172.21.15.1:0/295016711 shutdown_connections 2024-03-18T19:09:23.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.146+0000 7f7f4f3af700 1 -- 172.21.15.1:0/295016711 wait complete. 2024-03-18T19:09:23.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.147+0000 7f7f4f3af700 1 Processor -- start 2024-03-18T19:09:23.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.147+0000 7f7f4f3af700 1 -- start start 2024-03-18T19:09:23.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.148+0000 7f7f4f3af700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f481465a0 0x7f7f481f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:23.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.148+0000 7f7f4f3af700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f48146af0 0x7f7f480ac940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:23.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.148+0000 7f7f4f3af700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f480c4f90 0x7f7f480ace80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:23.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.148+0000 7f7f4f3af700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f480c60a0 con 0x7f7f48146af0 2024-03-18T19:09:23.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.148+0000 7f7f4f3af700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f480c5420 con 0x7f7f481465a0 2024-03-18T19:09:23.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.148+0000 7f7f4f3af700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f480c5700 con 0x7f7f480c4f90 2024-03-18T19:09:23.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.148+0000 7f7f4c94a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f48146af0 0x7f7f480ac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:23.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.148+0000 7f7f4c94a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f48146af0 0x7f7f480ac940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44054/0 (socket says 172.21.15.1:44054) 2024-03-18T19:09:23.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.148+0000 7f7f4c94a700 1 -- 172.21.15.1:0/981106597 learned_addr learned my addr 172.21.15.1:0/981106597 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:23.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.148+0000 7f7f4d94c700 1 --2- 172.21.15.1:0/981106597 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f480c4f90 0x7f7f480ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:23.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.148+0000 7f7f4d14b700 1 --2- 172.21.15.1:0/981106597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f481465a0 0x7f7f481f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:23.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.149+0000 7f7f4c94a700 1 -- 172.21.15.1:0/981106597 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f480c4f90 msgr2=0x7f7f480ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.149+0000 7f7f4c94a700 1 --2- 172.21.15.1:0/981106597 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f480c4f90 0x7f7f480ace80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.149+0000 7f7f4c94a700 1 -- 172.21.15.1:0/981106597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f481465a0 msgr2=0x7f7f481f33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.149+0000 7f7f4c94a700 1 --2- 172.21.15.1:0/981106597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f481465a0 0x7f7f481f33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.149+0000 7f7f4c94a700 1 -- 172.21.15.1:0/981106597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f44009550 con 0x7f7f48146af0 2024-03-18T19:09:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.149+0000 7f7f4c94a700 1 --2- 172.21.15.1:0/981106597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f48146af0 0x7f7f480ac940 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f7f380011e0 tx=0x7f7f3800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.149+0000 7f7f3dffb700 1 -- 172.21.15.1:0/981106597 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f380054e0 con 0x7f7f48146af0 2024-03-18T19:09:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.149+0000 7f7f3dffb700 1 -- 172.21.15.1:0/981106597 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f38003680 con 0x7f7f48146af0 2024-03-18T19:09:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.149+0000 7f7f4f3af700 1 -- 172.21.15.1:0/981106597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f480c59e0 con 0x7f7f48146af0 2024-03-18T19:09:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.149+0000 7f7f3dffb700 1 -- 172.21.15.1:0/981106597 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f38003a30 con 0x7f7f48146af0 2024-03-18T19:09:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.149+0000 7f7f4f3af700 1 -- 172.21.15.1:0/981106597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f480cf270 con 0x7f7f48146af0 2024-03-18T19:09:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.150+0000 7f7f4f3af700 1 -- 172.21.15.1:0/981106597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f480676a0 con 0x7f7f48146af0 2024-03-18T19:09:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.150+0000 7f7f3dffb700 1 -- 172.21.15.1:0/981106597 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7f38003b90 con 0x7f7f48146af0 2024-03-18T19:09:23.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.151+0000 7f7f3dffb700 1 --2- 172.21.15.1:0/981106597 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f2c06adb0 0x7f7f2c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:23.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.151+0000 7f7f3dffb700 1 -- 172.21.15.1:0/981106597 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f7f38088ad0 con 0x7f7f48146af0 2024-03-18T19:09:23.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.151+0000 7f7f4d14b700 1 --2- 172.21.15.1:0/981106597 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f2c06adb0 0x7f7f2c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:23.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.151+0000 7f7f4d14b700 1 --2- 172.21.15.1:0/981106597 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f2c06adb0 0x7f7f2c06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7f44014040 tx=0x7f7f44008030 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:23.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.156+0000 7f7f3dffb700 1 -- 172.21.15.1:0/981106597 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7f38053ad0 con 0x7f7f48146af0 2024-03-18T19:09:23.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.361+0000 7f7f4f3af700 1 -- 172.21.15.1:0/981106597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f480cc430 con 0x7f7f48146af0 2024-03-18T19:09:23.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.363+0000 7f7f3dffb700 1 -- 172.21.15.1:0/981106597 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+16325 (secure 0 0 0) 0x7f7f38017070 con 0x7f7f48146af0 2024-03-18T19:09:23.359 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:23.359 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":146,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:09:22.623112+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:09:20.468744+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-18T19:09:21.597476+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:09:23.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.365+0000 7f7f4f3af700 1 -- 172.21.15.1:0/981106597 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f2c06adb0 msgr2=0x7f7f2c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:23.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.365+0000 7f7f4f3af700 1 --2- 172.21.15.1:0/981106597 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f2c06adb0 0x7f7f2c06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7f44014040 tx=0x7f7f44008030 comp rx=0 tx=0).stop 2024-03-18T19:09:23.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.365+0000 7f7f4f3af700 1 -- 172.21.15.1:0/981106597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f48146af0 msgr2=0x7f7f480ac940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:23.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.365+0000 7f7f4f3af700 1 --2- 172.21.15.1:0/981106597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f48146af0 0x7f7f480ac940 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f7f380011e0 tx=0x7f7f3800cae0 comp rx=0 tx=0).stop 2024-03-18T19:09:23.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.365+0000 7f7f4f3af700 1 -- 172.21.15.1:0/981106597 shutdown_connections 2024-03-18T19:09:23.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.365+0000 7f7f4f3af700 1 --2- 172.21.15.1:0/981106597 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f2c06adb0 0x7f7f2c06d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:23.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.365+0000 7f7f4f3af700 1 --2- 172.21.15.1:0/981106597 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f480c4f90 0x7f7f480ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:23.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.365+0000 7f7f4f3af700 1 --2- 172.21.15.1:0/981106597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f481465a0 0x7f7f481f33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:23.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.365+0000 7f7f4f3af700 1 --2- 172.21.15.1:0/981106597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f48146af0 0x7f7f480ac940 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:23.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.365+0000 7f7f4f3af700 1 -- 172.21.15.1:0/981106597 >> 172.21.15.1:0/981106597 conn(0x7f7f480b1d20 msgr2=0x7f7f481454a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:23.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.365+0000 7f7f4f3af700 1 -- 172.21.15.1:0/981106597 shutdown_connections 2024-03-18T19:09:23.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.365+0000 7f7f4f3af700 1 -- 172.21.15.1:0/981106597 wait complete. 2024-03-18T19:09:23.373 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T19:09:23.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.506+0000 7f82e6d99700 1 Processor -- start 2024-03-18T19:09:23.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.506+0000 7f82e6d99700 1 -- start start 2024-03-18T19:09:23.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.507+0000 7f82e6d99700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f82e00b0500 0x7f82e00cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:23.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.507+0000 7f82e6d99700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f82e00b0e20 0x7f82e00b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:23.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.507+0000 7f82e6d99700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e015ef80 con 0x7f82e015eba0 2024-03-18T19:09:23.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.507+0000 7f82e6d99700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e00cc9a0 con 0x7f82e00b0e20 2024-03-18T19:09:23.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.507+0000 7f82e6d99700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e00ccc80 con 0x7f82e00b0500 2024-03-18T19:09:23.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.507+0000 7f82e5336700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f82e00b0e20 0x7f82e00b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:23.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.507+0000 7f82e5336700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f82e00b0e20 0x7f82e00b1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44062/0 (socket says 172.21.15.1:44062) 2024-03-18T19:09:23.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.507+0000 7f82e4b35700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f82e015eba0 0x7f82e00cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48384/0 (socket says 172.21.15.1:48384) 2024-03-18T19:09:23.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.507+0000 7f82e5336700 1 -- 172.21.15.1:0/792418474 learned_addr learned my addr 172.21.15.1:0/792418474 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:23.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.507+0000 7f82dffff700 1 --2- 172.21.15.1:0/792418474 >> v2:172.21.15.165:3300/0 conn(0x7f82e00b0500 0x7f82e00cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:23.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.507+0000 7f82e5336700 1 -- 172.21.15.1:0/792418474 >> v2:172.21.15.165:3300/0 conn(0x7f82e00b0500 msgr2=0x7f82e00cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:23.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.507+0000 7f82e5336700 1 --2- 172.21.15.1:0/792418474 >> v2:172.21.15.165:3300/0 conn(0x7f82e00b0500 0x7f82e00cfa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:23.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.507+0000 7f82e5336700 1 -- 172.21.15.1:0/792418474 >> v1:172.21.15.1:6789/0 conn(0x7f82e015eba0 legacy=0x7f82e00cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:23.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.507+0000 7f82df7fe700 1 -- 172.21.15.1:0/792418474 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112926625 0 0) 0x7f82e015ef80 con 0x7f82e015eba0 2024-03-18T19:09:23.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.507+0000 7f82e5336700 1 -- 172.21.15.1:0/792418474 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e00ccf60 con 0x7f82e00b0e20 2024-03-18T19:09:23.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.507+0000 7f82e5336700 1 --2- 172.21.15.1:0/792418474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00b0e20 0x7f82e00b1290 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f82e009f0e0 tx=0x7f82d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c28b665491ac29f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:23.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.507+0000 7f82df7fe700 1 -- 172.21.15.1:0/792418474 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d800e070 con 0x7f82e00b0e20 2024-03-18T19:09:23.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.507+0000 7f82df7fe700 1 -- 172.21.15.1:0/792418474 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d800b9c0 con 0x7f82e00b0e20 2024-03-18T19:09:23.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.507+0000 7f82df7fe700 1 -- 172.21.15.1:0/792418474 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d800bd70 con 0x7f82e00b0e20 2024-03-18T19:09:23.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.508+0000 7f82e6d99700 1 -- 172.21.15.1:0/792418474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00b0e20 msgr2=0x7f82e00b1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:23.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.508+0000 7f82e6d99700 1 --2- 172.21.15.1:0/792418474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00b0e20 0x7f82e00b1290 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f82e009f0e0 tx=0x7f82d800b330 comp rx=0 tx=0).stop 2024-03-18T19:09:23.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.508+0000 7f82e6d99700 1 -- 172.21.15.1:0/792418474 shutdown_connections 2024-03-18T19:09:23.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.508+0000 7f82e6d99700 1 --2- 172.21.15.1:0/792418474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00b0e20 0x7f82e00b1290 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:23.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.508+0000 7f82e6d99700 1 --2- 172.21.15.1:0/792418474 >> v2:172.21.15.165:3300/0 conn(0x7f82e00b0500 0x7f82e00cfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:23.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.508+0000 7f82e6d99700 1 -- 172.21.15.1:0/792418474 >> 172.21.15.1:0/792418474 conn(0x7f82e00b2bd0 msgr2=0x7f82e00b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:23.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.508+0000 7f82e6d99700 1 -- 172.21.15.1:0/792418474 shutdown_connections 2024-03-18T19:09:23.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.508+0000 7f82e6d99700 1 -- 172.21.15.1:0/792418474 wait complete. 2024-03-18T19:09:23.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.509+0000 7f82e6d99700 1 Processor -- start 2024-03-18T19:09:23.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.509+0000 7f82e6d99700 1 -- start start 2024-03-18T19:09:23.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.510+0000 7f82e6d99700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82e00b0500 0x7f82e00c72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:23.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.510+0000 7f82e6d99700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00b0e20 0x7f82e00c3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:23.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.510+0000 7f82e6d99700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82e015eba0 0x7f82e015ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.510+0000 7f82e6d99700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e00c5140 con 0x7f82e00b0e20 2024-03-18T19:09:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.510+0000 7f82e6d99700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e00c7820 con 0x7f82e015eba0 2024-03-18T19:09:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.510+0000 7f82e6d99700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e00c6560 con 0x7f82e00b0500 2024-03-18T19:09:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.510+0000 7f82dffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00b0e20 0x7f82e00c3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.510+0000 7f82dffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00b0e20 0x7f82e00c3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44072/0 (socket says 172.21.15.1:44072) 2024-03-18T19:09:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.510+0000 7f82dffff700 1 -- 172.21.15.1:0/1774754862 learned_addr learned my addr 172.21.15.1:0/1774754862 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.510+0000 7f82e5336700 1 --2- 172.21.15.1:0/1774754862 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82e015eba0 0x7f82e015ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.510+0000 7f82e4b35700 1 --2- 172.21.15.1:0/1774754862 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82e00b0500 0x7f82e00c72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.510+0000 7f82dffff700 1 -- 172.21.15.1:0/1774754862 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82e00b0500 msgr2=0x7f82e00c72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.510+0000 7f82dffff700 1 --2- 172.21.15.1:0/1774754862 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82e00b0500 0x7f82e00c72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.510+0000 7f82dffff700 1 -- 172.21.15.1:0/1774754862 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82e015eba0 msgr2=0x7f82e015ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.510+0000 7f82dffff700 1 --2- 172.21.15.1:0/1774754862 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82e015eba0 0x7f82e015ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.511+0000 7f82dffff700 1 -- 172.21.15.1:0/1774754862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d8009b30 con 0x7f82e00b0e20 2024-03-18T19:09:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.511+0000 7f82dffff700 1 --2- 172.21.15.1:0/1774754862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00b0e20 0x7f82e00c3cd0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f82cc00e810 tx=0x7f82cc00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.511+0000 7f82dd7fa700 1 -- 172.21.15.1:0/1774754862 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc00c070 con 0x7f82e00b0e20 2024-03-18T19:09:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.511+0000 7f82dd7fa700 1 -- 172.21.15.1:0/1774754862 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82cc013610 con 0x7f82e00b0e20 2024-03-18T19:09:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.511+0000 7f82e6d99700 1 -- 172.21.15.1:0/1774754862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e00c4e60 con 0x7f82e00b0e20 2024-03-18T19:09:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.511+0000 7f82dd7fa700 1 -- 172.21.15.1:0/1774754862 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc0139c0 con 0x7f82e00b0e20 2024-03-18T19:09:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.511+0000 7f82e6d99700 1 -- 172.21.15.1:0/1774754862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e00c6ae0 con 0x7f82e00b0e20 2024-03-18T19:09:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.512+0000 7f82e6d99700 1 -- 172.21.15.1:0/1774754862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e00676a0 con 0x7f82e00b0e20 2024-03-18T19:09:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.512+0000 7f82dd7fa700 1 -- 172.21.15.1:0/1774754862 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f82cc013b20 con 0x7f82e00b0e20 2024-03-18T19:09:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.513+0000 7f82dd7fa700 1 --2- 172.21.15.1:0/1774754862 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f82c406adb0 0x7f82c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.513+0000 7f82dd7fa700 1 -- 172.21.15.1:0/1774754862 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f82cc089440 con 0x7f82e00b0e20 2024-03-18T19:09:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.513+0000 7f82e4b35700 1 --2- 172.21.15.1:0/1774754862 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f82c406adb0 0x7f82c406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:23.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.513+0000 7f82e4b35700 1 --2- 172.21.15.1:0/1774754862 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f82c406adb0 0x7f82c406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f82d4000f60 tx=0x7f82d4007e70 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:23.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.517+0000 7f82dd7fa700 1 -- 172.21.15.1:0/1774754862 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f82cc0542b0 con 0x7f82e00b0e20 2024-03-18T19:09:23.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:23.769+0000 7f82e6d99700 1 -- 172.21.15.1:0/1774754862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f82e006eb60 con 0x7f82e00b0e20 2024-03-18T19:09:24.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.641+0000 7f82dd7fa700 1 -- 172.21.15.1:0/1774754862 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v64) v1 ==== 118+0+0 (secure 0 0 0) 0x7f82cc0578d0 con 0x7f82e00b0e20 2024-03-18T19:09:24.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.644+0000 7f82e6d99700 1 -- 172.21.15.1:0/1774754862 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f82c406adb0 msgr2=0x7f82c406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:24.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.644+0000 7f82e6d99700 1 --2- 172.21.15.1:0/1774754862 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f82c406adb0 0x7f82c406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f82d4000f60 tx=0x7f82d4007e70 comp rx=0 tx=0).stop 2024-03-18T19:09:24.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.644+0000 7f82e6d99700 1 -- 172.21.15.1:0/1774754862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00b0e20 msgr2=0x7f82e00c3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:24.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.644+0000 7f82e6d99700 1 --2- 172.21.15.1:0/1774754862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00b0e20 0x7f82e00c3cd0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f82cc00e810 tx=0x7f82cc00eb20 comp rx=0 tx=0).stop 2024-03-18T19:09:24.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.644+0000 7f82e6d99700 1 -- 172.21.15.1:0/1774754862 shutdown_connections 2024-03-18T19:09:24.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.644+0000 7f82e6d99700 1 --2- 172.21.15.1:0/1774754862 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f82c406adb0 0x7f82c406d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:24.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.644+0000 7f82e6d99700 1 --2- 172.21.15.1:0/1774754862 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82e00b0500 0x7f82e00c72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:24.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.644+0000 7f82e6d99700 1 --2- 172.21.15.1:0/1774754862 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82e015eba0 0x7f82e015ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:24.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.644+0000 7f82e6d99700 1 --2- 172.21.15.1:0/1774754862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82e00b0e20 0x7f82e00c3cd0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:24.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.644+0000 7f82e6d99700 1 -- 172.21.15.1:0/1774754862 >> 172.21.15.1:0/1774754862 conn(0x7f82e00b2bd0 msgr2=0x7f82e00cbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:24.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.644+0000 7f82e6d99700 1 -- 172.21.15.1:0/1774754862 shutdown_connections 2024-03-18T19:09:24.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.644+0000 7f82e6d99700 1 -- 172.21.15.1:0/1774754862 wait complete. 2024-03-18T19:09:24.655 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-18T19:09:24.655 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T19:09:24.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.794+0000 7f56f2cb9700 1 Processor -- start 2024-03-18T19:09:24.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.794+0000 7f56f2cb9700 1 -- start start 2024-03-18T19:09:24.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.795+0000 7f56f2cb9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f56ec15d5a0 0x7f56ec0ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:24.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.795+0000 7f56f2cb9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ec0cf600 con 0x7f56ec06ae60 2024-03-18T19:09:24.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.795+0000 7f56f2cb9700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ec0ce9f0 con 0x7f56ec0ce550 2024-03-18T19:09:24.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.795+0000 7f56f2cb9700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56ec0cecd0 con 0x7f56ec15d5a0 2024-03-18T19:09:24.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.795+0000 7f56f1256700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f56ec06ae60 0x7f56ec167140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51098/0 (socket says 172.21.15.1:51098) 2024-03-18T19:09:24.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.795+0000 7f56f1256700 1 -- 172.21.15.1:0/1262346862 learned_addr learned my addr 172.21.15.1:0/1262346862 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:24.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.795+0000 7f56f0a55700 1 --2- 172.21.15.1:0/1262346862 >> v2:172.21.15.165:3300/0 conn(0x7f56ec15d5a0 0x7f56ec0ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:24.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.795+0000 7f56eb7fe700 1 -- 172.21.15.1:0/1262346862 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958217119 0 0) 0x7f56ec0cf600 con 0x7f56ec06ae60 2024-03-18T19:09:24.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.795+0000 7f56eb7fe700 1 -- 172.21.15.1:0/1262346862 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56dc0035b0 con 0x7f56ec06ae60 2024-03-18T19:09:24.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.795+0000 7f56eb7fe700 1 -- 172.21.15.1:0/1262346862 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2849160496 0 0) 0x7f56dc0035b0 con 0x7f56ec06ae60 2024-03-18T19:09:24.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.795+0000 7f56eb7fe700 1 -- 172.21.15.1:0/1262346862 >> v2:172.21.15.165:3300/0 conn(0x7f56ec15d5a0 msgr2=0x7f56ec0ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:24.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.795+0000 7f56eb7fe700 1 --2- 172.21.15.1:0/1262346862 >> v2:172.21.15.165:3300/0 conn(0x7f56ec15d5a0 0x7f56ec0ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:24.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.795+0000 7f56eb7fe700 1 -- 172.21.15.1:0/1262346862 >> v1:172.21.15.165:6789/0 conn(0x7f56ec0ce550 legacy=0x7f56ec06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:24.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.795+0000 7f56eb7fe700 1 -- 172.21.15.1:0/1262346862 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ec0cefb0 con 0x7f56ec06ae60 2024-03-18T19:09:24.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.795+0000 7f56eb7fe700 1 -- 172.21.15.1:0/1262346862 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f56e40019f0 con 0x7f56ec06ae60 2024-03-18T19:09:24.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56eb7fe700 1 -- 172.21.15.1:0/1262346862 >> v1:172.21.15.1:6789/0 conn(0x7f56ec06ae60 legacy=0x7f56ec167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:24.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56eb7fe700 1 --2- 172.21.15.1:0/1262346862 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56dc003a00 0x7f56dc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:24.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56eb7fe700 1 --2- 172.21.15.1:0/1262346862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc004850 0x7f56dc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:24.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56eb7fe700 1 --2- 172.21.15.1:0/1262346862 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56dc005210 0x7f56dc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:24.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56eb7fe700 1 -- 172.21.15.1:0/1262346862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec0cecd0 con 0x7f56dc004850 2024-03-18T19:09:24.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56eb7fe700 1 -- 172.21.15.1:0/1262346862 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc00c0b0 con 0x7f56dc003a00 2024-03-18T19:09:24.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56eb7fe700 1 -- 172.21.15.1:0/1262346862 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc00c270 con 0x7f56dc005210 2024-03-18T19:09:24.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56ebfff700 1 --2- 172.21.15.1:0/1262346862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc004850 0x7f56dc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:24.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56f0a55700 1 --2- 172.21.15.1:0/1262346862 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56dc003a00 0x7f56dc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:24.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56f1256700 1 --2- 172.21.15.1:0/1262346862 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56dc005210 0x7f56dc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:24.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56ebfff700 1 -- 172.21.15.1:0/1262346862 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56dc005210 msgr2=0x7f56dc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:24.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56ebfff700 1 --2- 172.21.15.1:0/1262346862 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56dc005210 0x7f56dc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:24.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56ebfff700 1 -- 172.21.15.1:0/1262346862 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56dc003a00 msgr2=0x7f56dc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:24.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56ebfff700 1 --2- 172.21.15.1:0/1262346862 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56dc003a00 0x7f56dc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:24.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56ebfff700 1 -- 172.21.15.1:0/1262346862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56dc0035b0 con 0x7f56dc004850 2024-03-18T19:09:24.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56f1256700 1 --2- 172.21.15.1:0/1262346862 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56dc005210 0x7f56dc00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:24.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56f1256700 1 -- 172.21.15.1:0/1262346862 reap_dead start 2024-03-18T19:09:24.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56ebfff700 1 --2- 172.21.15.1:0/1262346862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc004850 0x7f56dc004cd0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f56d80017c0 tx=0x7f56dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7bcdde3a2274e1a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:24.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56eb7fe700 1 -- 172.21.15.1:0/1262346862 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d80052d0 con 0x7f56dc004850 2024-03-18T19:09:24.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56eb7fe700 1 -- 172.21.15.1:0/1262346862 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d8005430 con 0x7f56dc004850 2024-03-18T19:09:24.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.796+0000 7f56eb7fe700 1 -- 172.21.15.1:0/1262346862 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d80038c0 con 0x7f56dc004850 2024-03-18T19:09:24.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.797+0000 7f56f2cb9700 1 -- 172.21.15.1:0/1262346862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc004850 msgr2=0x7f56dc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:24.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.797+0000 7f56f2cb9700 1 --2- 172.21.15.1:0/1262346862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc004850 0x7f56dc004cd0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f56d80017c0 tx=0x7f56dc005880 comp rx=0 tx=0).stop 2024-03-18T19:09:24.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.797+0000 7f56f2cb9700 1 -- 172.21.15.1:0/1262346862 shutdown_connections 2024-03-18T19:09:24.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.797+0000 7f56f2cb9700 1 --2- 172.21.15.1:0/1262346862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56dc004850 0x7f56dc004cd0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:24.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.797+0000 7f56f2cb9700 1 -- 172.21.15.1:0/1262346862 >> 172.21.15.1:0/1262346862 conn(0x7f56ec0cc9b0 msgr2=0x7f56ec0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:24.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.797+0000 7f56f2cb9700 1 -- 172.21.15.1:0/1262346862 shutdown_connections 2024-03-18T19:09:24.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.797+0000 7f56f2cb9700 1 -- 172.21.15.1:0/1262346862 wait complete. 2024-03-18T19:09:24.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.798+0000 7f56f2cb9700 1 Processor -- start 2024-03-18T19:09:24.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.799+0000 7f56f2cb9700 1 -- start start 2024-03-18T19:09:24.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56f2cb9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56dc004850 0x7f56ec1f2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:24.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56f2cb9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56ec158d60 0x7f56ec153390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:24.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56f2cb9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56ec1591b0 0x7f56ec1538d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:24.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56f2cb9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec1f3540 con 0x7f56ec1591b0 2024-03-18T19:09:24.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56f2cb9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec1f3820 con 0x7f56ec158d60 2024-03-18T19:09:24.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56f2cb9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec1f3b00 con 0x7f56dc004850 2024-03-18T19:09:24.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56f1256700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56ec1591b0 0x7f56ec1538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:24.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56f1256700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56ec1591b0 0x7f56ec1538d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38898/0 (socket says 172.21.15.1:38898) 2024-03-18T19:09:24.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56f1256700 1 -- 172.21.15.1:0/2397844130 learned_addr learned my addr 172.21.15.1:0/2397844130 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:24.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56ebfff700 1 --2- 172.21.15.1:0/2397844130 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56ec158d60 0x7f56ec153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:24.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56f0a55700 1 --2- 172.21.15.1:0/2397844130 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56dc004850 0x7f56ec1f2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:24.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56f1256700 1 -- 172.21.15.1:0/2397844130 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56dc004850 msgr2=0x7f56ec1f2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:24.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56f1256700 1 --2- 172.21.15.1:0/2397844130 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56dc004850 0x7f56ec1f2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:24.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56f1256700 1 -- 172.21.15.1:0/2397844130 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56ec158d60 msgr2=0x7f56ec153390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:24.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56f1256700 1 --2- 172.21.15.1:0/2397844130 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56ec158d60 0x7f56ec153390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:24.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56f1256700 1 -- 172.21.15.1:0/2397844130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ec1f3de0 con 0x7f56ec1591b0 2024-03-18T19:09:24.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56f1256700 1 --2- 172.21.15.1:0/2397844130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56ec1591b0 0x7f56ec1538d0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f56ec0b2a50 tx=0x7f56e4007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:24.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56e97fa700 1 -- 172.21.15.1:0/2397844130 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e400eba0 con 0x7f56ec1591b0 2024-03-18T19:09:24.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56e97fa700 1 -- 172.21.15.1:0/2397844130 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e400ed00 con 0x7f56ec1591b0 2024-03-18T19:09:24.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56f2cb9700 1 -- 172.21.15.1:0/2397844130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56ec1561a0 con 0x7f56ec1591b0 2024-03-18T19:09:24.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.800+0000 7f56e97fa700 1 -- 172.21.15.1:0/2397844130 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e4013620 con 0x7f56ec1591b0 2024-03-18T19:09:24.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.801+0000 7f56f2cb9700 1 -- 172.21.15.1:0/2397844130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56ec1f6680 con 0x7f56ec1591b0 2024-03-18T19:09:24.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.801+0000 7f56f2cb9700 1 -- 172.21.15.1:0/2397844130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56ec1f3f70 con 0x7f56ec1591b0 2024-03-18T19:09:24.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.802+0000 7f56e97fa700 1 -- 172.21.15.1:0/2397844130 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f56e4013780 con 0x7f56ec1591b0 2024-03-18T19:09:24.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.802+0000 7f56e97fa700 1 --2- 172.21.15.1:0/2397844130 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56d006adb0 0x7f56d006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:24.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.802+0000 7f56e97fa700 1 -- 172.21.15.1:0/2397844130 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f56e4089a20 con 0x7f56ec1591b0 2024-03-18T19:09:24.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.802+0000 7f56f0a55700 1 --2- 172.21.15.1:0/2397844130 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56d006adb0 0x7f56d006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:24.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.803+0000 7f56f0a55700 1 --2- 172.21.15.1:0/2397844130 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56d006adb0 0x7f56d006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f56e0009cc0 tx=0x7f56e00091c0 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:24.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:24.807+0000 7f56e97fa700 1 -- 172.21.15.1:0/2397844130 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f56e4054a20 con 0x7f56ec1591b0 2024-03-18T19:09:25.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.055+0000 7f56f2cb9700 1 -- 172.21.15.1:0/2397844130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f56ec0676a0 con 0x7f56ec1591b0 2024-03-18T19:09:25.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.649+0000 7f56e97fa700 1 -- 172.21.15.1:0/2397844130 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v65) v1 ==== 122+0+0 (secure 0 0 0) 0x7f56e4058040 con 0x7f56ec1591b0 2024-03-18T19:09:25.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.652+0000 7f56f2cb9700 1 -- 172.21.15.1:0/2397844130 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56d006adb0 msgr2=0x7f56d006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:25.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.652+0000 7f56f2cb9700 1 --2- 172.21.15.1:0/2397844130 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56d006adb0 0x7f56d006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f56e0009cc0 tx=0x7f56e00091c0 comp rx=0 tx=0).stop 2024-03-18T19:09:25.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.652+0000 7f56f2cb9700 1 -- 172.21.15.1:0/2397844130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56ec1591b0 msgr2=0x7f56ec1538d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:25.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.652+0000 7f56f2cb9700 1 --2- 172.21.15.1:0/2397844130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56ec1591b0 0x7f56ec1538d0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f56ec0b2a50 tx=0x7f56e4007c90 comp rx=0 tx=0).stop 2024-03-18T19:09:25.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.652+0000 7f56f2cb9700 1 -- 172.21.15.1:0/2397844130 shutdown_connections 2024-03-18T19:09:25.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.652+0000 7f56f2cb9700 1 --2- 172.21.15.1:0/2397844130 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f56d006adb0 0x7f56d006d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:25.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.652+0000 7f56f2cb9700 1 --2- 172.21.15.1:0/2397844130 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56dc004850 0x7f56ec1f2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:25.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.652+0000 7f56f2cb9700 1 --2- 172.21.15.1:0/2397844130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56ec1591b0 0x7f56ec1538d0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:25.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.652+0000 7f56f2cb9700 1 --2- 172.21.15.1:0/2397844130 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56ec158d60 0x7f56ec153390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:25.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.652+0000 7f56f2cb9700 1 -- 172.21.15.1:0/2397844130 >> 172.21.15.1:0/2397844130 conn(0x7f56ec0cc9b0 msgr2=0x7f56ec06f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:25.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.652+0000 7f56f2cb9700 1 -- 172.21.15.1:0/2397844130 shutdown_connections 2024-03-18T19:09:25.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.652+0000 7f56f2cb9700 1 -- 172.21.15.1:0/2397844130 wait complete. 2024-03-18T19:09:25.663 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T19:09:25.664 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:09:25.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.799+0000 7ff90bf10700 1 Processor -- start 2024-03-18T19:09:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.799+0000 7ff90bf10700 1 -- start start 2024-03-18T19:09:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.800+0000 7ff90bf10700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff904161360 0x7ff90406b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.800+0000 7ff90bf10700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9040cfd20 con 0x7ff9040ce080 2024-03-18T19:09:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.800+0000 7ff90bf10700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9040717c0 con 0x7ff9040cdc00 2024-03-18T19:09:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.800+0000 7ff90bf10700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff90415da00 con 0x7ff904161360 2024-03-18T19:09:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.800+0000 7ff909cac700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff904161360 0x7ff90406b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.800+0000 7ff90a4ad700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff9040ce080 0x7ff904071c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51100/0 (socket says 172.21.15.1:51100) 2024-03-18T19:09:25.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.800+0000 7ff90a4ad700 1 -- 172.21.15.1:0/298944201 learned_addr learned my addr 172.21.15.1:0/298944201 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.801+0000 7ff909cac700 1 -- 172.21.15.1:0/298944201 >> v1:172.21.15.165:6789/0 conn(0x7ff9040cdc00 legacy=0x7ff904161740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.801+0000 7ff909cac700 1 -- 172.21.15.1:0/298944201 >> v1:172.21.15.1:6789/0 conn(0x7ff9040ce080 legacy=0x7ff904071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.801+0000 7ff909cac700 1 -- 172.21.15.1:0/298944201 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90406b670 con 0x7ff904161360 2024-03-18T19:09:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.801+0000 7ff908caa700 1 -- 172.21.15.1:0/298944201 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891462929 0 0) 0x7ff9040cfd20 con 0x7ff9040ce080 2024-03-18T19:09:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.801+0000 7ff909cac700 1 --2- 172.21.15.1:0/298944201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff904161360 0x7ff90406b130 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7ff900009840 tx=0x7ff900009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5abf940244213f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.801+0000 7ff908caa700 1 -- 172.21.15.1:0/298944201 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff900011830 con 0x7ff904161360 2024-03-18T19:09:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.801+0000 7ff908caa700 1 -- 172.21.15.1:0/298944201 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff900011990 con 0x7ff904161360 2024-03-18T19:09:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.801+0000 7ff908caa700 1 -- 172.21.15.1:0/298944201 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff900011d20 con 0x7ff904161360 2024-03-18T19:09:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.802+0000 7ff90bf10700 1 -- 172.21.15.1:0/298944201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff904161360 msgr2=0x7ff90406b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.802+0000 7ff90bf10700 1 --2- 172.21.15.1:0/298944201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff904161360 0x7ff90406b130 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7ff900009840 tx=0x7ff900009b50 comp rx=0 tx=0).stop 2024-03-18T19:09:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.802+0000 7ff90bf10700 1 -- 172.21.15.1:0/298944201 shutdown_connections 2024-03-18T19:09:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.802+0000 7ff90bf10700 1 --2- 172.21.15.1:0/298944201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff904161360 0x7ff90406b130 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.802+0000 7ff90bf10700 1 -- 172.21.15.1:0/298944201 >> 172.21.15.1:0/298944201 conn(0x7ff9040ccab0 msgr2=0x7ff9040ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.802+0000 7ff90bf10700 1 -- 172.21.15.1:0/298944201 shutdown_connections 2024-03-18T19:09:25.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.802+0000 7ff90bf10700 1 -- 172.21.15.1:0/298944201 wait complete. 2024-03-18T19:09:25.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.803+0000 7ff90bf10700 1 Processor -- start 2024-03-18T19:09:25.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.803+0000 7ff90bf10700 1 -- start start 2024-03-18T19:09:25.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.804+0000 7ff90bf10700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9040cdc00 0x7ff9041f3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:25.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.804+0000 7ff90bf10700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9040ce080 0x7ff9040b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:25.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.804+0000 7ff90bf10700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff904161360 0x7ff9040c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:25.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.804+0000 7ff90bf10700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9040b35e0 con 0x7ff904161360 2024-03-18T19:09:25.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.804+0000 7ff90bf10700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9040b3880 con 0x7ff9040ce080 2024-03-18T19:09:25.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.804+0000 7ff90bf10700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9040b3b60 con 0x7ff9040cdc00 2024-03-18T19:09:25.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.804+0000 7ff90a4ad700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff904161360 0x7ff9040c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:25.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.804+0000 7ff90a4ad700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff904161360 0x7ff9040c9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38916/0 (socket says 172.21.15.1:38916) 2024-03-18T19:09:25.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.804+0000 7ff90a4ad700 1 -- 172.21.15.1:0/1104867188 learned_addr learned my addr 172.21.15.1:0/1104867188 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:25.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.804+0000 7ff909cac700 1 --2- 172.21.15.1:0/1104867188 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9040cdc00 0x7ff9041f3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:25.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.804+0000 7ff9094ab700 1 --2- 172.21.15.1:0/1104867188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9040ce080 0x7ff9040b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:25.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.804+0000 7ff90a4ad700 1 -- 172.21.15.1:0/1104867188 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9040cdc00 msgr2=0x7ff9041f3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:25.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.804+0000 7ff90a4ad700 1 --2- 172.21.15.1:0/1104867188 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9040cdc00 0x7ff9041f3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:25.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.804+0000 7ff90a4ad700 1 -- 172.21.15.1:0/1104867188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9040ce080 msgr2=0x7ff9040b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:25.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.804+0000 7ff90a4ad700 1 --2- 172.21.15.1:0/1104867188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9040ce080 0x7ff9040b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:25.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.804+0000 7ff90a4ad700 1 -- 172.21.15.1:0/1104867188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9000094a0 con 0x7ff904161360 2024-03-18T19:09:25.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.804+0000 7ff90a4ad700 1 --2- 172.21.15.1:0/1104867188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff904161360 0x7ff9040c9650 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc002230 tx=0x7ff8fc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:25.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.804+0000 7ff8f67fc700 1 -- 172.21.15.1:0/1104867188 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc00ac20 con 0x7ff904161360 2024-03-18T19:09:25.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.804+0000 7ff8f67fc700 1 -- 172.21.15.1:0/1104867188 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8fc00ada0 con 0x7ff904161360 2024-03-18T19:09:25.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.805+0000 7ff90bf10700 1 -- 172.21.15.1:0/1104867188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9040b3e40 con 0x7ff904161360 2024-03-18T19:09:25.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.805+0000 7ff8f67fc700 1 -- 172.21.15.1:0/1104867188 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc007310 con 0x7ff904161360 2024-03-18T19:09:25.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.805+0000 7ff90bf10700 1 -- 172.21.15.1:0/1104867188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9040c9b90 con 0x7ff904161360 2024-03-18T19:09:25.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.805+0000 7ff90bf10700 1 -- 172.21.15.1:0/1104867188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9040676a0 con 0x7ff904161360 2024-03-18T19:09:25.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.806+0000 7ff8f67fc700 1 -- 172.21.15.1:0/1104867188 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff8fc007470 con 0x7ff904161360 2024-03-18T19:09:25.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.806+0000 7ff8f67fc700 1 --2- 172.21.15.1:0/1104867188 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff8e806adb0 0x7ff8e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:25.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.806+0000 7ff8f67fc700 1 -- 172.21.15.1:0/1104867188 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff8fc089180 con 0x7ff904161360 2024-03-18T19:09:25.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.806+0000 7ff909cac700 1 --2- 172.21.15.1:0/1104867188 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff8e806adb0 0x7ff8e806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:25.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.807+0000 7ff909cac700 1 --2- 172.21.15.1:0/1104867188 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff8e806adb0 0x7ff8e806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff900000c00 tx=0x7ff90001c040 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:25.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:25.811+0000 7ff8f67fc700 1 -- 172.21.15.1:0/1104867188 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff8fc0540e0 con 0x7ff904161360 2024-03-18T19:09:26.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.055+0000 7ff90bf10700 1 -- 172.21.15.1:0/1104867188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff904072870 con 0x7ff904161360 2024-03-18T19:09:26.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.056+0000 7ff8f67fc700 1 -- 172.21.15.1:0/1104867188 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+4727 (secure 0 0 0) 0x7ff8fc057700 con 0x7ff904161360 2024-03-18T19:09:26.052 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:26.052 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8116,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3723189348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3723189348},{"type":"v1","addr":"172.21.15.1:6837","nonce":3723189348}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":8125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/530206363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":530206363},{"type":"v1","addr":"172.21.15.165:6835","nonce":530206363}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":8784,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/332827595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":332827595},{"type":"v1","addr":"172.21.15.1:6836","nonce":332827595}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:09:22.623415+0000","modified":"2024-03-18T19:09:25.640243+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6834/311436411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":311436411},{"type":"v1","addr":"172.21.15.165:6834","nonce":311436411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-18T19:09:26.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.058+0000 7ff90bf10700 1 -- 172.21.15.1:0/1104867188 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff8e806adb0 msgr2=0x7ff8e806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:26.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.058+0000 7ff90bf10700 1 --2- 172.21.15.1:0/1104867188 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff8e806adb0 0x7ff8e806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff900000c00 tx=0x7ff90001c040 comp rx=0 tx=0).stop 2024-03-18T19:09:26.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.058+0000 7ff90bf10700 1 -- 172.21.15.1:0/1104867188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff904161360 msgr2=0x7ff9040c9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:26.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.058+0000 7ff90bf10700 1 --2- 172.21.15.1:0/1104867188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff904161360 0x7ff9040c9650 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc002230 tx=0x7ff8fc00a060 comp rx=0 tx=0).stop 2024-03-18T19:09:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.058+0000 7ff90bf10700 1 -- 172.21.15.1:0/1104867188 shutdown_connections 2024-03-18T19:09:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.058+0000 7ff90bf10700 1 --2- 172.21.15.1:0/1104867188 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff8e806adb0 0x7ff8e806d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.058+0000 7ff90bf10700 1 --2- 172.21.15.1:0/1104867188 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9040cdc00 0x7ff9041f3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.058+0000 7ff90bf10700 1 --2- 172.21.15.1:0/1104867188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff904161360 0x7ff9040c9650 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.058+0000 7ff90bf10700 1 --2- 172.21.15.1:0/1104867188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9040ce080 0x7ff9040b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.058+0000 7ff90bf10700 1 -- 172.21.15.1:0/1104867188 >> 172.21.15.1:0/1104867188 conn(0x7ff9040ccab0 msgr2=0x7ff90406f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.058+0000 7ff90bf10700 1 -- 172.21.15.1:0/1104867188 shutdown_connections 2024-03-18T19:09:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.058+0000 7ff90bf10700 1 -- 172.21.15.1:0/1104867188 wait complete. 2024-03-18T19:09:26.055 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 65 2024-03-18T19:09:26.066 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:09:26.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.196+0000 7f8fc14e3700 1 Processor -- start 2024-03-18T19:09:26.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.196+0000 7f8fc14e3700 1 -- start start 2024-03-18T19:09:26.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.196+0000 7f8fc14e3700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8fbc0d2e80 0x7f8fbc0d3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:26.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.196+0000 7f8fc14e3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8fbc15d810 0x7f8fbc161240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:26.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.196+0000 7f8fc14e3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8fbc161780 0x7f8fbc0b3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:26.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.196+0000 7f8fc14e3700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fbc161fa0 con 0x7f8fbc15d810 2024-03-18T19:09:26.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.196+0000 7f8fc14e3700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fbc15eaa0 con 0x7f8fbc0d2e80 2024-03-18T19:09:26.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.196+0000 7f8fc14e3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc15e610 con 0x7f8fbc161780 2024-03-18T19:09:26.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.197+0000 7f8fba7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8fbc15d810 0x7f8fbc161240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:26.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.197+0000 7f8fba7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8fbc15d810 0x7f8fbc161240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38924/0 (socket says 172.21.15.1:38924) 2024-03-18T19:09:26.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.197+0000 7f8fba7fc700 1 -- 172.21.15.1:0/706322795 learned_addr learned my addr 172.21.15.1:0/706322795 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:26.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.197+0000 7f8fbaffd700 1 --2- 172.21.15.1:0/706322795 >> v2:172.21.15.165:3300/0 conn(0x7f8fbc0d2e80 0x7f8fbc0d3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:26.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.197+0000 7f8fbb7fe700 1 --2- 172.21.15.1:0/706322795 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8fbc161780 0x7f8fbc0b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:26.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.197+0000 7f8fba7fc700 1 -- 172.21.15.1:0/706322795 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8fbc161780 msgr2=0x7f8fbc0b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:26.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.197+0000 7f8fba7fc700 1 --2- 172.21.15.1:0/706322795 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8fbc161780 0x7f8fbc0b3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.197+0000 7f8fba7fc700 1 -- 172.21.15.1:0/706322795 >> v2:172.21.15.165:3300/0 conn(0x7f8fbc0d2e80 msgr2=0x7f8fbc0d3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:26.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.197+0000 7f8fba7fc700 1 --2- 172.21.15.1:0/706322795 >> v2:172.21.15.165:3300/0 conn(0x7f8fbc0d2e80 0x7f8fbc0d3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.197+0000 7f8fba7fc700 1 -- 172.21.15.1:0/706322795 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fbc15dce0 con 0x7f8fbc15d810 2024-03-18T19:09:26.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.197+0000 7f8fbb7fe700 1 --2- 172.21.15.1:0/706322795 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8fbc161780 0x7f8fbc0b3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:26.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.197+0000 7f8fbaffd700 1 --2- 172.21.15.1:0/706322795 >> v2:172.21.15.165:3300/0 conn(0x7f8fbc0d2e80 0x7f8fbc0d3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:26.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.197+0000 7f8fba7fc700 1 --2- 172.21.15.1:0/706322795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fbc15d810 0x7f8fbc161240 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc08d0c0 tx=0x7f8fb0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a6b68ec7a16a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:26.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.197+0000 7f8fb9ffb700 1 -- 172.21.15.1:0/706322795 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb000b870 con 0x7f8fbc15d810 2024-03-18T19:09:26.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.197+0000 7f8fb9ffb700 1 -- 172.21.15.1:0/706322795 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb000b9d0 con 0x7f8fbc15d810 2024-03-18T19:09:26.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.197+0000 7f8fb9ffb700 1 -- 172.21.15.1:0/706322795 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb000bd60 con 0x7f8fbc15d810 2024-03-18T19:09:26.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.198+0000 7f8fc14e3700 1 -- 172.21.15.1:0/706322795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fbc15d810 msgr2=0x7f8fbc161240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:26.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.198+0000 7f8fc14e3700 1 --2- 172.21.15.1:0/706322795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fbc15d810 0x7f8fbc161240 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc08d0c0 tx=0x7f8fb0009bd0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.198+0000 7f8fc14e3700 1 -- 172.21.15.1:0/706322795 shutdown_connections 2024-03-18T19:09:26.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.198+0000 7f8fc14e3700 1 --2- 172.21.15.1:0/706322795 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8fbc161780 0x7f8fbc0b3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.198+0000 7f8fc14e3700 1 --2- 172.21.15.1:0/706322795 >> v2:172.21.15.165:3300/0 conn(0x7f8fbc0d2e80 0x7f8fbc0d3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.198+0000 7f8fc14e3700 1 --2- 172.21.15.1:0/706322795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fbc15d810 0x7f8fbc161240 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.198+0000 7f8fc14e3700 1 -- 172.21.15.1:0/706322795 >> 172.21.15.1:0/706322795 conn(0x7f8fbc08cbf0 msgr2=0x7f8fbc09c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:26.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.198+0000 7f8fc14e3700 1 -- 172.21.15.1:0/706322795 shutdown_connections 2024-03-18T19:09:26.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.198+0000 7f8fc14e3700 1 -- 172.21.15.1:0/706322795 wait complete. 2024-03-18T19:09:26.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.199+0000 7f8fc14e3700 1 Processor -- start 2024-03-18T19:09:26.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.199+0000 7f8fc14e3700 1 -- start start 2024-03-18T19:09:26.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.200+0000 7f8fc14e3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8fbc0d2e80 0x7f8fbc06b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:26.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.200+0000 7f8fc14e3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8fbc15d810 0x7f8fbc0c6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:26.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.200+0000 7f8fc14e3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fbc161780 0x7f8fbc1550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:26.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.200+0000 7f8fc14e3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc0c8440 con 0x7f8fbc161780 2024-03-18T19:09:26.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.200+0000 7f8fc14e3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc0c77c0 con 0x7f8fbc0d2e80 2024-03-18T19:09:26.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.200+0000 7f8fc14e3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc0c7aa0 con 0x7f8fbc15d810 2024-03-18T19:09:26.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.200+0000 7f8fbb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fbc161780 0x7f8fbc1550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:26.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.200+0000 7f8fbb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fbc161780 0x7f8fbc1550f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38940/0 (socket says 172.21.15.1:38940) 2024-03-18T19:09:26.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.200+0000 7f8fbb7fe700 1 -- 172.21.15.1:0/3389718676 learned_addr learned my addr 172.21.15.1:0/3389718676 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:26.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.200+0000 7f8fba7fc700 1 --2- 172.21.15.1:0/3389718676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8fbc15d810 0x7f8fbc0c6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:26.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.200+0000 7f8fbaffd700 1 --2- 172.21.15.1:0/3389718676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8fbc0d2e80 0x7f8fbc06b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:26.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.200+0000 7f8fbb7fe700 1 -- 172.21.15.1:0/3389718676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8fbc15d810 msgr2=0x7f8fbc0c6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:26.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.200+0000 7f8fbb7fe700 1 --2- 172.21.15.1:0/3389718676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8fbc15d810 0x7f8fbc0c6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.200+0000 7f8fbb7fe700 1 -- 172.21.15.1:0/3389718676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8fbc0d2e80 msgr2=0x7f8fbc06b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:26.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.200+0000 7f8fbb7fe700 1 --2- 172.21.15.1:0/3389718676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8fbc0d2e80 0x7f8fbc06b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.200+0000 7f8fbb7fe700 1 -- 172.21.15.1:0/3389718676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb0009550 con 0x7f8fbc161780 2024-03-18T19:09:26.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.200+0000 7f8fba7fc700 1 --2- 172.21.15.1:0/3389718676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8fbc15d810 0x7f8fbc0c6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:26.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.200+0000 7f8fbaffd700 1 --2- 172.21.15.1:0/3389718676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8fbc0d2e80 0x7f8fbc06b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:26.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.200+0000 7f8fbb7fe700 1 --2- 172.21.15.1:0/3389718676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fbc161780 0x7f8fbc1550f0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400cd50 tx=0x7f8fb400a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:26.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.201+0000 7f8fa37fe700 1 -- 172.21.15.1:0/3389718676 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb400f480 con 0x7f8fbc161780 2024-03-18T19:09:26.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.201+0000 7f8fa37fe700 1 -- 172.21.15.1:0/3389718676 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb400f600 con 0x7f8fbc161780 2024-03-18T19:09:26.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.201+0000 7f8fc14e3700 1 -- 172.21.15.1:0/3389718676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fbc0c7d80 con 0x7f8fbc161780 2024-03-18T19:09:26.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.201+0000 7f8fa37fe700 1 -- 172.21.15.1:0/3389718676 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb400f9b0 con 0x7f8fbc161780 2024-03-18T19:09:26.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.201+0000 7f8fc14e3700 1 -- 172.21.15.1:0/3389718676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fbc15fe40 con 0x7f8fbc161780 2024-03-18T19:09:26.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.202+0000 7f8fc14e3700 1 -- 172.21.15.1:0/3389718676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fbc073350 con 0x7f8fbc161780 2024-03-18T19:09:26.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.202+0000 7f8fa37fe700 1 -- 172.21.15.1:0/3389718676 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8fb400b040 con 0x7f8fbc161780 2024-03-18T19:09:26.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.202+0000 7f8fa37fe700 1 --2- 172.21.15.1:0/3389718676 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8fa806adb0 0x7f8fa806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:26.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.203+0000 7f8fa37fe700 1 -- 172.21.15.1:0/3389718676 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f8fb4089ea0 con 0x7f8fbc161780 2024-03-18T19:09:26.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.203+0000 7f8fbaffd700 1 --2- 172.21.15.1:0/3389718676 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8fa806adb0 0x7f8fa806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:26.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.203+0000 7f8fbaffd700 1 --2- 172.21.15.1:0/3389718676 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8fa806adb0 0x7f8fa806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8fac006ba0 tx=0x7f8fac008040 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:26.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.207+0000 7f8fa37fe700 1 -- 172.21.15.1:0/3389718676 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8fb4054d90 con 0x7f8fbc161780 2024-03-18T19:09:26.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.415+0000 7f8fc14e3700 1 -- 172.21.15.1:0/3389718676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fbc0d3be0 con 0x7f8fbc161780 2024-03-18T19:09:26.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.416+0000 7f8fa37fe700 1 -- 172.21.15.1:0/3389718676 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+16312 (secure 0 0 0) 0x7f8fb40583b0 con 0x7f8fbc161780 2024-03-18T19:09:26.412 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:26.412 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":147,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:09:23.625297+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:09:20.468744+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-18T19:09:21.597476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:09:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.418+0000 7f8fc14e3700 1 -- 172.21.15.1:0/3389718676 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8fa806adb0 msgr2=0x7f8fa806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.418+0000 7f8fc14e3700 1 --2- 172.21.15.1:0/3389718676 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8fa806adb0 0x7f8fa806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8fac006ba0 tx=0x7f8fac008040 comp rx=0 tx=0).stop 2024-03-18T19:09:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.418+0000 7f8fc14e3700 1 -- 172.21.15.1:0/3389718676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fbc161780 msgr2=0x7f8fbc1550f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.418+0000 7f8fc14e3700 1 --2- 172.21.15.1:0/3389718676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fbc161780 0x7f8fbc1550f0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400cd50 tx=0x7f8fb400a560 comp rx=0 tx=0).stop 2024-03-18T19:09:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.418+0000 7f8fc14e3700 1 -- 172.21.15.1:0/3389718676 shutdown_connections 2024-03-18T19:09:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.418+0000 7f8fc14e3700 1 --2- 172.21.15.1:0/3389718676 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8fa806adb0 0x7f8fa806d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.418+0000 7f8fc14e3700 1 --2- 172.21.15.1:0/3389718676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fbc161780 0x7f8fbc1550f0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.418+0000 7f8fc14e3700 1 --2- 172.21.15.1:0/3389718676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8fbc0d2e80 0x7f8fbc06b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.418+0000 7f8fc14e3700 1 --2- 172.21.15.1:0/3389718676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8fbc15d810 0x7f8fbc0c6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.418+0000 7f8fc14e3700 1 -- 172.21.15.1:0/3389718676 >> 172.21.15.1:0/3389718676 conn(0x7f8fbc08cbf0 msgr2=0x7f8fbc0cddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.418+0000 7f8fc14e3700 1 -- 172.21.15.1:0/3389718676 shutdown_connections 2024-03-18T19:09:26.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.418+0000 7f8fc14e3700 1 -- 172.21.15.1:0/3389718676 wait complete. 2024-03-18T19:09:26.427 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T19:09:26.427 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T19:09:26.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.556+0000 7f0fa54e6700 1 Processor -- start 2024-03-18T19:09:26.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.557+0000 7f0fa54e6700 1 -- start start 2024-03-18T19:09:26.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.557+0000 7f0fa54e6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0fa00cf090 0x7f0fa00d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:26.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.557+0000 7f0fa54e6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0fa0160d20 0x7f0fa00d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:26.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.557+0000 7f0fa54e6700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa00cddc0 con 0x7f0fa0160940 2024-03-18T19:09:26.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.557+0000 7f0fa54e6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fa00cdff0 con 0x7f0fa0160d20 2024-03-18T19:09:26.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.557+0000 7f0fa54e6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fa00ce2d0 con 0x7f0fa00cf090 2024-03-18T19:09:26.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.557+0000 7f0f9f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0fa0160d20 0x7f0fa00d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:26.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.558+0000 7f0f9effd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0fa0160940 0x7f0fa00d0190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51116/0 (socket says 172.21.15.1:51116) 2024-03-18T19:09:26.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.558+0000 7f0f9f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0fa0160d20 0x7f0fa00d1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38948/0 (socket says 172.21.15.1:38948) 2024-03-18T19:09:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.558+0000 7f0f9effd700 1 -- 172.21.15.1:0/1725211828 learned_addr learned my addr 172.21.15.1:0/1725211828 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.558+0000 7f0f9e7fc700 1 --2- 172.21.15.1:0/1725211828 >> v2:172.21.15.165:3300/0 conn(0x7f0fa00cf090 0x7f0fa00d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.558+0000 7f0f9f7fe700 1 -- 172.21.15.1:0/1725211828 >> v2:172.21.15.165:3300/0 conn(0x7f0fa00cf090 msgr2=0x7f0fa00d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.558+0000 7f0f9f7fe700 1 --2- 172.21.15.1:0/1725211828 >> v2:172.21.15.165:3300/0 conn(0x7f0fa00cf090 0x7f0fa00d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.558+0000 7f0f9f7fe700 1 -- 172.21.15.1:0/1725211828 >> v1:172.21.15.1:6789/0 conn(0x7f0fa0160940 legacy=0x7f0fa00d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.558+0000 7f0f9f7fe700 1 -- 172.21.15.1:0/1725211828 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa00ce5b0 con 0x7f0fa0160d20 2024-03-18T19:09:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.558+0000 7f0f9e7fc700 1 --2- 172.21.15.1:0/1725211828 >> v2:172.21.15.165:3300/0 conn(0x7f0fa00cf090 0x7f0fa00d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.558+0000 7f0f9dffb700 1 -- 172.21.15.1:0/1725211828 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355656794 0 0) 0x7f0fa00cddc0 con 0x7f0fa0160940 2024-03-18T19:09:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.558+0000 7f0f9f7fe700 1 --2- 172.21.15.1:0/1725211828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fa0160d20 0x7f0fa00d1cd0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f0fa00b9230 tx=0x7f0f9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f97cadc0eb03dc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.558+0000 7f0f9dffb700 1 -- 172.21.15.1:0/1725211828 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9800e070 con 0x7f0fa0160d20 2024-03-18T19:09:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.558+0000 7f0f9dffb700 1 -- 172.21.15.1:0/1725211828 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9800bbd0 con 0x7f0fa0160d20 2024-03-18T19:09:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.558+0000 7f0f9dffb700 1 -- 172.21.15.1:0/1725211828 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f98012430 con 0x7f0fa0160d20 2024-03-18T19:09:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.559+0000 7f0fa54e6700 1 -- 172.21.15.1:0/1725211828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fa0160d20 msgr2=0x7f0fa00d1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:26.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.559+0000 7f0fa54e6700 1 --2- 172.21.15.1:0/1725211828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fa0160d20 0x7f0fa00d1cd0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f0fa00b9230 tx=0x7f0f9800b330 comp rx=0 tx=0).stop 2024-03-18T19:09:26.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.559+0000 7f0fa54e6700 1 -- 172.21.15.1:0/1725211828 shutdown_connections 2024-03-18T19:09:26.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.559+0000 7f0fa54e6700 1 --2- 172.21.15.1:0/1725211828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fa0160d20 0x7f0fa00d1cd0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.559+0000 7f0fa54e6700 1 --2- 172.21.15.1:0/1725211828 >> v2:172.21.15.165:3300/0 conn(0x7f0fa00cf090 0x7f0fa00d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.559+0000 7f0fa54e6700 1 -- 172.21.15.1:0/1725211828 >> 172.21.15.1:0/1725211828 conn(0x7f0fa009c520 msgr2=0x7f0fa00d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:26.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.559+0000 7f0fa54e6700 1 -- 172.21.15.1:0/1725211828 shutdown_connections 2024-03-18T19:09:26.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.559+0000 7f0fa54e6700 1 -- 172.21.15.1:0/1725211828 wait complete. 2024-03-18T19:09:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.560+0000 7f0fa54e6700 1 Processor -- start 2024-03-18T19:09:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.560+0000 7f0fa54e6700 1 -- start start 2024-03-18T19:09:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.561+0000 7f0fa54e6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fa00cf090 0x7f0fa00c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.561+0000 7f0fa54e6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fa0160940 0x7f0fa015b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.561+0000 7f0fa54e6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fa0160d20 0x7f0fa015bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:26.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.561+0000 7f0fa54e6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa00c7c30 con 0x7f0fa0160d20 2024-03-18T19:09:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.561+0000 7f0fa54e6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa00c7f00 con 0x7f0fa00cf090 2024-03-18T19:09:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.561+0000 7f0fa54e6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa00c81e0 con 0x7f0fa0160940 2024-03-18T19:09:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.561+0000 7f0f9f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fa0160d20 0x7f0fa015bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.561+0000 7f0f9f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fa0160d20 0x7f0fa015bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38956/0 (socket says 172.21.15.1:38956) 2024-03-18T19:09:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.561+0000 7f0f9f7fe700 1 -- 172.21.15.1:0/1801660837 learned_addr learned my addr 172.21.15.1:0/1801660837 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.561+0000 7f0f9effd700 1 --2- 172.21.15.1:0/1801660837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fa00cf090 0x7f0fa00c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.561+0000 7f0f9e7fc700 1 --2- 172.21.15.1:0/1801660837 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fa0160940 0x7f0fa015b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.561+0000 7f0f9f7fe700 1 -- 172.21.15.1:0/1801660837 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fa0160940 msgr2=0x7f0fa015b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.561+0000 7f0f9f7fe700 1 --2- 172.21.15.1:0/1801660837 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fa0160940 0x7f0fa015b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.561+0000 7f0f9f7fe700 1 -- 172.21.15.1:0/1801660837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fa00cf090 msgr2=0x7f0fa00c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.561+0000 7f0f9f7fe700 1 --2- 172.21.15.1:0/1801660837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fa00cf090 0x7f0fa00c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.561+0000 7f0f9f7fe700 1 -- 172.21.15.1:0/1801660837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f98009a70 con 0x7f0fa0160d20 2024-03-18T19:09:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.562+0000 7f0f9f7fe700 1 --2- 172.21.15.1:0/1801660837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fa0160d20 0x7f0fa015bb20 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f0f98013000 tx=0x7f0f98004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.562+0000 7f0f977fe700 1 -- 172.21.15.1:0/1801660837 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f98012430 con 0x7f0fa0160d20 2024-03-18T19:09:26.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.562+0000 7f0f977fe700 1 -- 172.21.15.1:0/1801660837 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f98020070 con 0x7f0fa0160d20 2024-03-18T19:09:26.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.562+0000 7f0fa54e6700 1 -- 172.21.15.1:0/1801660837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa00c84c0 con 0x7f0fa0160d20 2024-03-18T19:09:26.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.562+0000 7f0f977fe700 1 -- 172.21.15.1:0/1801660837 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f98004280 con 0x7f0fa0160d20 2024-03-18T19:09:26.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.562+0000 7f0fa54e6700 1 -- 172.21.15.1:0/1801660837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa00ca4a0 con 0x7f0fa0160d20 2024-03-18T19:09:26.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.563+0000 7f0fa54e6700 1 -- 172.21.15.1:0/1801660837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa00c8980 con 0x7f0fa0160d20 2024-03-18T19:09:26.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.563+0000 7f0f977fe700 1 -- 172.21.15.1:0/1801660837 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0f980043e0 con 0x7f0fa0160d20 2024-03-18T19:09:26.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.564+0000 7f0f977fe700 1 --2- 172.21.15.1:0/1801660837 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0f8006adb0 0x7f0f8006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.564+0000 7f0f977fe700 1 -- 172.21.15.1:0/1801660837 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0f98022070 con 0x7f0fa0160d20 2024-03-18T19:09:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.564+0000 7f0f9effd700 1 --2- 172.21.15.1:0/1801660837 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0f8006adb0 0x7f0f8006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:26.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.564+0000 7f0f9effd700 1 --2- 172.21.15.1:0/1801660837 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0f8006adb0 0x7f0f8006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0f90007fd0 tx=0x7f0f90007de0 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:26.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.568+0000 7f0f977fe700 1 -- 172.21.15.1:0/1801660837 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0f980568c0 con 0x7f0fa0160d20 2024-03-18T19:09:26.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.768+0000 7f0fa54e6700 1 -- 172.21.15.1:0/1801660837 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0fa00c8350 con 0x7f0f8006adb0 2024-03-18T19:09:26.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.777+0000 7f0f977fe700 1 -- 172.21.15.1:0/1801660837 <== mgr.8565 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+283908 (secure 0 0 0) 0x7f0fa00c8350 con 0x7f0f8006adb0 2024-03-18T19:09:26.777 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:26.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.780+0000 7f0fa54e6700 1 -- 172.21.15.1:0/1801660837 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0f8006adb0 msgr2=0x7f0f8006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:26.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.780+0000 7f0fa54e6700 1 --2- 172.21.15.1:0/1801660837 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0f8006adb0 0x7f0f8006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0f90007fd0 tx=0x7f0f90007de0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.780+0000 7f0fa54e6700 1 -- 172.21.15.1:0/1801660837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fa0160d20 msgr2=0x7f0fa015bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:26.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.780+0000 7f0fa54e6700 1 --2- 172.21.15.1:0/1801660837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fa0160d20 0x7f0fa015bb20 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f0f98013000 tx=0x7f0f98004ad0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.780+0000 7f0fa54e6700 1 -- 172.21.15.1:0/1801660837 shutdown_connections 2024-03-18T19:09:26.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.780+0000 7f0fa54e6700 1 --2- 172.21.15.1:0/1801660837 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0f8006adb0 0x7f0f8006d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.780+0000 7f0fa54e6700 1 --2- 172.21.15.1:0/1801660837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fa0160d20 0x7f0fa015bb20 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.780+0000 7f0fa54e6700 1 --2- 172.21.15.1:0/1801660837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fa00cf090 0x7f0fa00c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.780+0000 7f0fa54e6700 1 --2- 172.21.15.1:0/1801660837 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fa0160940 0x7f0fa015b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.780+0000 7f0fa54e6700 1 -- 172.21.15.1:0/1801660837 >> 172.21.15.1:0/1801660837 conn(0x7f0fa009c520 msgr2=0x7f0fa00cf4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:26.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.780+0000 7f0fa54e6700 1 -- 172.21.15.1:0/1801660837 shutdown_connections 2024-03-18T19:09:26.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.780+0000 7f0fa54e6700 1 -- 172.21.15.1:0/1801660837 wait complete. 2024-03-18T19:09:26.780 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T19:09:26.788 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-18T19:09:25.105901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592640,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":395,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":256,"ondisk_log_size":256,"up":116,"acting":116,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":91,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":347072,"kb_used_data":15280,"kb_used_omap":12,"kb_used_meta":331635,"kb_avail":754627648,"statfs":{"total":773094113280,"available":772738711552,"internally_reserved":0,"allocated":15646720,"data_stored":8437437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":339594809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2272,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000666"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.650652+0000","last_change":"2024-03-18T19:09:22.649460+0000","last_active":"2024-03-18T19:09:22.650652+0000","last_peered":"2024-03-18T19:09:22.650652+0000","last_clean":"2024-03-18T19:09:22.650652+0000","last_became_active":"2024-03-18T19:09:22.648770+0000","last_became_peered":"2024-03-18T19:09:22.648770+0000","last_unstale":"2024-03-18T19:09:22.650652+0000","last_undegraded":"2024-03-18T19:09:22.650652+0000","last_fullsized":"2024-03-18T19:09:22.650652+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:28.480818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.651238+0000","last_change":"2024-03-18T19:09:22.651017+0000","last_active":"2024-03-18T19:09:22.651238+0000","last_peered":"2024-03-18T19:09:22.651238+0000","last_clean":"2024-03-18T19:09:22.651238+0000","last_became_active":"2024-03-18T19:09:22.650823+0000","last_became_peered":"2024-03-18T19:09:22.650823+0000","last_unstale":"2024-03-18T19:09:22.651238+0000","last_undegraded":"2024-03-18T19:09:22.651238+0000","last_fullsized":"2024-03-18T19:09:22.651238+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:20:20.852061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327783+0000","last_change":"2024-03-18T19:09:22.646262+0000","last_active":"2024-03-18T19:09:24.327783+0000","last_peered":"2024-03-18T19:09:24.327783+0000","last_clean":"2024-03-18T19:09:24.327783+0000","last_became_active":"2024-03-18T19:09:22.645926+0000","last_became_peered":"2024-03-18T19:09:22.645926+0000","last_unstale":"2024-03-18T19:09:24.327783+0000","last_undegraded":"2024-03-18T19:09:24.327783+0000","last_fullsized":"2024-03-18T19:09:24.327783+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:44.544055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.651585+0000","last_change":"2024-03-18T19:09:22.651211+0000","last_active":"2024-03-18T19:09:22.651585+0000","last_peered":"2024-03-18T19:09:22.651585+0000","last_clean":"2024-03-18T19:09:22.651585+0000","last_became_active":"2024-03-18T19:09:22.650920+0000","last_became_peered":"2024-03-18T19:09:22.650920+0000","last_unstale":"2024-03-18T19:09:22.651585+0000","last_undegraded":"2024-03-18T19:09:22.651585+0000","last_fullsized":"2024-03-18T19:09:22.651585+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:07:46.533619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.328030+0000","last_change":"2024-03-18T19:09:22.649568+0000","last_active":"2024-03-18T19:09:24.328030+0000","last_peered":"2024-03-18T19:09:24.328030+0000","last_clean":"2024-03-18T19:09:24.328030+0000","last_became_active":"2024-03-18T19:09:22.648829+0000","last_became_peered":"2024-03-18T19:09:22.648829+0000","last_unstale":"2024-03-18T19:09:24.328030+0000","last_undegraded":"2024-03-18T19:09:24.328030+0000","last_fullsized":"2024-03-18T19:09:24.328030+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:55:10.360362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.640141+0000","last_change":"2024-03-18T19:09:22.637079+0000","last_active":"2024-03-18T19:09:22.640141+0000","last_peered":"2024-03-18T19:09:22.640141+0000","last_clean":"2024-03-18T19:09:22.640141+0000","last_became_active":"2024-03-18T19:09:22.636411+0000","last_became_peered":"2024-03-18T19:09:22.636411+0000","last_unstale":"2024-03-18T19:09:22.640141+0000","last_undegraded":"2024-03-18T19:09:22.640141+0000","last_fullsized":"2024-03-18T19:09:22.640141+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:57:27.805513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.658061+0000","last_change":"2024-03-18T19:09:22.654312+0000","last_active":"2024-03-18T19:09:22.658061+0000","last_peered":"2024-03-18T19:09:22.658061+0000","last_clean":"2024-03-18T19:09:22.658061+0000","last_became_active":"2024-03-18T19:09:22.654071+0000","last_became_peered":"2024-03-18T19:09:22.654071+0000","last_unstale":"2024-03-18T19:09:22.658061+0000","last_undegraded":"2024-03-18T19:09:22.658061+0000","last_fullsized":"2024-03-18T19:09:22.658061+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:44.544055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.649403+0000","last_change":"2024-03-18T19:09:22.646993+0000","last_active":"2024-03-18T19:09:22.649403+0000","last_peered":"2024-03-18T19:09:22.649403+0000","last_clean":"2024-03-18T19:09:22.649403+0000","last_became_active":"2024-03-18T19:09:22.646684+0000","last_became_peered":"2024-03-18T19:09:22.646684+0000","last_unstale":"2024-03-18T19:09:22.649403+0000","last_undegraded":"2024-03-18T19:09:22.649403+0000","last_fullsized":"2024-03-18T19:09:22.649403+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:47:24.499751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.658137+0000","last_change":"2024-03-18T19:09:22.655395+0000","last_active":"2024-03-18T19:09:22.658137+0000","last_peered":"2024-03-18T19:09:22.658137+0000","last_clean":"2024-03-18T19:09:22.658137+0000","last_became_active":"2024-03-18T19:09:22.655157+0000","last_became_peered":"2024-03-18T19:09:22.655157+0000","last_unstale":"2024-03-18T19:09:22.658137+0000","last_undegraded":"2024-03-18T19:09:22.658137+0000","last_fullsized":"2024-03-18T19:09:22.658137+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:55:10.360362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.649517+0000","last_change":"2024-03-18T19:09:22.646963+0000","last_active":"2024-03-18T19:09:22.649517+0000","last_peered":"2024-03-18T19:09:22.649517+0000","last_clean":"2024-03-18T19:09:22.649517+0000","last_became_active":"2024-03-18T19:09:22.646274+0000","last_became_peered":"2024-03-18T19:09:22.646274+0000","last_unstale":"2024-03-18T19:09:22.649517+0000","last_undegraded":"2024-03-18T19:09:22.649517+0000","last_fullsized":"2024-03-18T19:09:22.649517+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:04:29.218281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.633637+0000","last_change":"2024-03-18T19:09:22.643401+0000","last_active":"2024-03-18T19:09:23.633637+0000","last_peered":"2024-03-18T19:09:23.633637+0000","last_clean":"2024-03-18T19:09:23.633637+0000","last_became_active":"2024-03-18T19:09:22.642990+0000","last_became_peered":"2024-03-18T19:09:22.642990+0000","last_unstale":"2024-03-18T19:09:23.633637+0000","last_undegraded":"2024-03-18T19:09:23.633637+0000","last_fullsized":"2024-03-18T19:09:23.633637+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:30.062865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.644965+0000","last_change":"2024-03-18T19:09:22.642770+0000","last_active":"2024-03-18T19:09:22.644965+0000","last_peered":"2024-03-18T19:09:22.644965+0000","last_clean":"2024-03-18T19:09:22.644965+0000","last_became_active":"2024-03-18T19:09:22.642497+0000","last_became_peered":"2024-03-18T19:09:22.642497+0000","last_unstale":"2024-03-18T19:09:22.644965+0000","last_undegraded":"2024-03-18T19:09:22.644965+0000","last_fullsized":"2024-03-18T19:09:22.644965+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:25:41.879440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327766+0000","last_change":"2024-03-18T19:09:22.640110+0000","last_active":"2024-03-18T19:09:24.327766+0000","last_peered":"2024-03-18T19:09:24.327766+0000","last_clean":"2024-03-18T19:09:24.327766+0000","last_became_active":"2024-03-18T19:09:22.639807+0000","last_became_peered":"2024-03-18T19:09:22.639807+0000","last_unstale":"2024-03-18T19:09:24.327766+0000","last_undegraded":"2024-03-18T19:09:24.327766+0000","last_fullsized":"2024-03-18T19:09:24.327766+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:19.098002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327631+0000","last_change":"2024-03-18T19:09:22.650369+0000","last_active":"2024-03-18T19:09:24.327631+0000","last_peered":"2024-03-18T19:09:24.327631+0000","last_clean":"2024-03-18T19:09:24.327631+0000","last_became_active":"2024-03-18T19:09:22.650096+0000","last_became_peered":"2024-03-18T19:09:22.650096+0000","last_unstale":"2024-03-18T19:09:24.327631+0000","last_undegraded":"2024-03-18T19:09:24.327631+0000","last_fullsized":"2024-03-18T19:09:24.327631+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:35.575921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.644993+0000","last_change":"2024-03-18T19:09:22.642852+0000","last_active":"2024-03-18T19:09:22.644993+0000","last_peered":"2024-03-18T19:09:22.644993+0000","last_clean":"2024-03-18T19:09:22.644993+0000","last_became_active":"2024-03-18T19:09:22.642577+0000","last_became_peered":"2024-03-18T19:09:22.642577+0000","last_unstale":"2024-03-18T19:09:22.644993+0000","last_undegraded":"2024-03-18T19:09:22.644993+0000","last_fullsized":"2024-03-18T19:09:22.644993+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:11.209178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.633727+0000","last_change":"2024-03-18T19:09:22.643822+0000","last_active":"2024-03-18T19:09:23.633727+0000","last_peered":"2024-03-18T19:09:23.633727+0000","last_clean":"2024-03-18T19:09:23.633727+0000","last_became_active":"2024-03-18T19:09:22.642602+0000","last_became_peered":"2024-03-18T19:09:22.642602+0000","last_unstale":"2024-03-18T19:09:23.633727+0000","last_undegraded":"2024-03-18T19:09:23.633727+0000","last_fullsized":"2024-03-18T19:09:23.633727+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:32.074373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.649253+0000","last_change":"2024-03-18T19:09:22.648927+0000","last_active":"2024-03-18T19:09:22.649253+0000","last_peered":"2024-03-18T19:09:22.649253+0000","last_clean":"2024-03-18T19:09:22.649253+0000","last_became_active":"2024-03-18T19:09:22.648652+0000","last_became_peered":"2024-03-18T19:09:22.648652+0000","last_unstale":"2024-03-18T19:09:22.649253+0000","last_undegraded":"2024-03-18T19:09:22.649253+0000","last_fullsized":"2024-03-18T19:09:22.649253+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:02:45.260371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.649280+0000","last_change":"2024-03-18T19:09:22.645632+0000","last_active":"2024-03-18T19:09:22.649280+0000","last_peered":"2024-03-18T19:09:22.649280+0000","last_clean":"2024-03-18T19:09:22.649280+0000","last_became_active":"2024-03-18T19:09:22.644855+0000","last_became_peered":"2024-03-18T19:09:22.644855+0000","last_unstale":"2024-03-18T19:09:22.649280+0000","last_undegraded":"2024-03-18T19:09:22.649280+0000","last_fullsized":"2024-03-18T19:09:22.649280+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:28:35.293994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.640249+0000","last_change":"2024-03-18T19:09:22.637640+0000","last_active":"2024-03-18T19:09:22.640249+0000","last_peered":"2024-03-18T19:09:22.640249+0000","last_clean":"2024-03-18T19:09:22.640249+0000","last_became_active":"2024-03-18T19:09:22.637385+0000","last_became_peered":"2024-03-18T19:09:22.637385+0000","last_unstale":"2024-03-18T19:09:22.640249+0000","last_undegraded":"2024-03-18T19:09:22.640249+0000","last_fullsized":"2024-03-18T19:09:22.640249+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:48:33.430169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"146'4","reported_seq":15,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.633920+0000","last_change":"2024-03-18T19:09:22.213624+0000","last_active":"2024-03-18T19:09:23.633920+0000","last_peered":"2024-03-18T19:09:23.633920+0000","last_clean":"2024-03-18T19:09:23.633920+0000","last_became_active":"2024-03-18T19:09:22.213358+0000","last_became_peered":"2024-03-18T19:09:22.213358+0000","last_unstale":"2024-03-18T19:09:23.633920+0000","last_undegraded":"2024-03-18T19:09:23.633920+0000","last_fullsized":"2024-03-18T19:09:23.633920+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:23:13.662431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1e","version":"146'1","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.641409+0000","last_change":"2024-03-18T19:09:22.210909+0000","last_active":"2024-03-18T19:09:22.641409+0000","last_peered":"2024-03-18T19:09:22.641409+0000","last_clean":"2024-03-18T19:09:22.641409+0000","last_became_active":"2024-03-18T19:09:22.210359+0000","last_became_peered":"2024-03-18T19:09:22.210359+0000","last_unstale":"2024-03-18T19:09:22.641409+0000","last_undegraded":"2024-03-18T19:09:22.641409+0000","last_fullsized":"2024-03-18T19:09:22.641409+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:23:44.469348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1d","version":"146'2","reported_seq":13,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327963+0000","last_change":"2024-03-18T19:09:22.210238+0000","last_active":"2024-03-18T19:09:24.327963+0000","last_peered":"2024-03-18T19:09:24.327963+0000","last_clean":"2024-03-18T19:09:24.327963+0000","last_became_active":"2024-03-18T19:09:22.207542+0000","last_became_peered":"2024-03-18T19:09:22.207542+0000","last_unstale":"2024-03-18T19:09:24.327963+0000","last_undegraded":"2024-03-18T19:09:24.327963+0000","last_fullsized":"2024-03-18T19:09:24.327963+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:23:44.469348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.328091+0000","last_change":"2024-03-18T19:09:22.644465+0000","last_active":"2024-03-18T19:09:24.328091+0000","last_peered":"2024-03-18T19:09:24.328091+0000","last_clean":"2024-03-18T19:09:24.328091+0000","last_became_active":"2024-03-18T19:09:22.644008+0000","last_became_peered":"2024-03-18T19:09:22.644008+0000","last_unstale":"2024-03-18T19:09:24.328091+0000","last_undegraded":"2024-03-18T19:09:24.328091+0000","last_fullsized":"2024-03-18T19:09:24.328091+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:30.062865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"146'2","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.648630+0000","last_change":"2024-03-18T19:09:22.212515+0000","last_active":"2024-03-18T19:09:22.648630+0000","last_peered":"2024-03-18T19:09:22.648630+0000","last_clean":"2024-03-18T19:09:22.648630+0000","last_became_active":"2024-03-18T19:09:22.212230+0000","last_became_peered":"2024-03-18T19:09:22.212230+0000","last_unstale":"2024-03-18T19:09:22.648630+0000","last_undegraded":"2024-03-18T19:09:22.648630+0000","last_fullsized":"2024-03-18T19:09:22.648630+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:23:13.662431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.636090+0000","last_change":"2024-03-18T19:09:22.634750+0000","last_active":"2024-03-18T19:09:22.636090+0000","last_peered":"2024-03-18T19:09:22.636090+0000","last_clean":"2024-03-18T19:09:22.636090+0000","last_became_active":"2024-03-18T19:09:22.634468+0000","last_became_peered":"2024-03-18T19:09:22.634468+0000","last_unstale":"2024-03-18T19:09:22.636090+0000","last_undegraded":"2024-03-18T19:09:22.636090+0000","last_fullsized":"2024-03-18T19:09:22.636090+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:32.074373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:21.185417+0000","last_change":"2024-03-18T19:09:21.185417+0000","last_active":"2024-03-18T19:09:21.185417+0000","last_peered":"2024-03-18T19:09:21.185417+0000","last_clean":"2024-03-18T19:09:21.185417+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:21.185417+0000","last_undegraded":"2024-03-18T19:09:21.185417+0000","last_fullsized":"2024-03-18T19:09:21.185417+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:21.185417+0000","last_change":"2024-03-18T19:09:21.185417+0000","last_active":"2024-03-18T19:09:21.185417+0000","last_peered":"2024-03-18T19:09:21.185417+0000","last_clean":"2024-03-18T19:09:21.185417+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:21.185417+0000","last_undegraded":"2024-03-18T19:09:21.185417+0000","last_fullsized":"2024-03-18T19:09:21.185417+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.648192+0000","last_change":"2024-03-18T19:09:22.640975+0000","last_active":"2024-03-18T19:09:22.648192+0000","last_peered":"2024-03-18T19:09:22.648192+0000","last_clean":"2024-03-18T19:09:22.648192+0000","last_became_active":"2024-03-18T19:09:22.638298+0000","last_became_peered":"2024-03-18T19:09:22.638298+0000","last_unstale":"2024-03-18T19:09:22.648192+0000","last_undegraded":"2024-03-18T19:09:22.648192+0000","last_fullsized":"2024-03-18T19:09:22.648192+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:42:47.124196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:21.185417+0000","last_change":"2024-03-18T19:09:21.185417+0000","last_active":"2024-03-18T19:09:21.185417+0000","last_peered":"2024-03-18T19:09:21.185417+0000","last_clean":"2024-03-18T19:09:21.185417+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:21.185417+0000","last_undegraded":"2024-03-18T19:09:21.185417+0000","last_fullsized":"2024-03-18T19:09:21.185417+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.641350+0000","last_change":"2024-03-18T19:09:22.639238+0000","last_active":"2024-03-18T19:09:22.641350+0000","last_peered":"2024-03-18T19:09:22.641350+0000","last_clean":"2024-03-18T19:09:22.641350+0000","last_became_active":"2024-03-18T19:09:22.639052+0000","last_became_peered":"2024-03-18T19:09:22.639052+0000","last_unstale":"2024-03-18T19:09:22.641350+0000","last_undegraded":"2024-03-18T19:09:22.641350+0000","last_fullsized":"2024-03-18T19:09:22.641350+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:08.116636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:21.185417+0000","last_change":"2024-03-18T19:09:21.185417+0000","last_active":"2024-03-18T19:09:21.185417+0000","last_peered":"2024-03-18T19:09:21.185417+0000","last_clean":"2024-03-18T19:09:21.185417+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:21.185417+0000","last_undegraded":"2024-03-18T19:09:21.185417+0000","last_fullsized":"2024-03-18T19:09:21.185417+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.633971+0000","last_change":"2024-03-18T19:09:22.644752+0000","last_active":"2024-03-18T19:09:23.633971+0000","last_peered":"2024-03-18T19:09:23.633971+0000","last_clean":"2024-03-18T19:09:23.633971+0000","last_became_active":"2024-03-18T19:09:22.644452+0000","last_became_peered":"2024-03-18T19:09:22.644452+0000","last_unstale":"2024-03-18T19:09:23.633971+0000","last_undegraded":"2024-03-18T19:09:23.633971+0000","last_fullsized":"2024-03-18T19:09:23.633971+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:50:11.074283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.634003+0000","last_change":"2024-03-18T19:09:22.210552+0000","last_active":"2024-03-18T19:09:23.634003+0000","last_peered":"2024-03-18T19:09:23.634003+0000","last_clean":"2024-03-18T19:09:23.634003+0000","last_became_active":"2024-03-18T19:09:22.210152+0000","last_became_peered":"2024-03-18T19:09:22.210152+0000","last_unstale":"2024-03-18T19:09:23.634003+0000","last_undegraded":"2024-03-18T19:09:23.634003+0000","last_fullsized":"2024-03-18T19:09:23.634003+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:42:54.314098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:21.185417+0000","last_change":"2024-03-18T19:09:21.185417+0000","last_active":"2024-03-18T19:09:21.185417+0000","last_peered":"2024-03-18T19:09:21.185417+0000","last_clean":"2024-03-18T19:09:21.185417+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:21.185417+0000","last_undegraded":"2024-03-18T19:09:21.185417+0000","last_fullsized":"2024-03-18T19:09:21.185417+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.649528+0000","last_change":"2024-03-18T19:09:22.646996+0000","last_active":"2024-03-18T19:09:22.649528+0000","last_peered":"2024-03-18T19:09:22.649528+0000","last_clean":"2024-03-18T19:09:22.649528+0000","last_became_active":"2024-03-18T19:09:22.646627+0000","last_became_peered":"2024-03-18T19:09:22.646627+0000","last_unstale":"2024-03-18T19:09:22.649528+0000","last_undegraded":"2024-03-18T19:09:22.649528+0000","last_fullsized":"2024-03-18T19:09:22.649528+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:58:33.254876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:21.185417+0000","last_change":"2024-03-18T19:09:21.185417+0000","last_active":"2024-03-18T19:09:21.185417+0000","last_peered":"2024-03-18T19:09:21.185417+0000","last_clean":"2024-03-18T19:09:21.185417+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:21.185417+0000","last_undegraded":"2024-03-18T19:09:21.185417+0000","last_fullsized":"2024-03-18T19:09:21.185417+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.14","version":"146'6","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.657735+0000","last_change":"2024-03-18T19:09:22.209065+0000","last_active":"2024-03-18T19:09:22.657735+0000","last_peered":"2024-03-18T19:09:22.657735+0000","last_clean":"2024-03-18T19:09:22.657735+0000","last_became_active":"2024-03-18T19:09:22.208780+0000","last_became_peered":"2024-03-18T19:09:22.208780+0000","last_unstale":"2024-03-18T19:09:22.657735+0000","last_undegraded":"2024-03-18T19:09:22.657735+0000","last_fullsized":"2024-03-18T19:09:22.657735+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:48:23.276451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.633437+0000","last_change":"2024-03-18T19:09:22.644783+0000","last_active":"2024-03-18T19:09:23.633437+0000","last_peered":"2024-03-18T19:09:23.633437+0000","last_clean":"2024-03-18T19:09:23.633437+0000","last_became_active":"2024-03-18T19:09:22.644575+0000","last_became_peered":"2024-03-18T19:09:22.644575+0000","last_unstale":"2024-03-18T19:09:23.633437+0000","last_undegraded":"2024-03-18T19:09:23.633437+0000","last_fullsized":"2024-03-18T19:09:23.633437+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:51.984733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"146'2","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.652517+0000","last_change":"2024-03-18T19:09:22.206823+0000","last_active":"2024-03-18T19:09:22.652517+0000","last_peered":"2024-03-18T19:09:22.652517+0000","last_clean":"2024-03-18T19:09:22.652517+0000","last_became_active":"2024-03-18T19:09:22.206396+0000","last_became_peered":"2024-03-18T19:09:22.206396+0000","last_unstale":"2024-03-18T19:09:22.652517+0000","last_undegraded":"2024-03-18T19:09:22.652517+0000","last_fullsized":"2024-03-18T19:09:22.652517+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:59:05.543413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.641134+0000","last_change":"2024-03-18T19:09:22.637886+0000","last_active":"2024-03-18T19:09:22.641134+0000","last_peered":"2024-03-18T19:09:22.641134+0000","last_clean":"2024-03-18T19:09:22.641134+0000","last_became_active":"2024-03-18T19:09:22.637568+0000","last_became_peered":"2024-03-18T19:09:22.637568+0000","last_unstale":"2024-03-18T19:09:22.641134+0000","last_undegraded":"2024-03-18T19:09:22.641134+0000","last_fullsized":"2024-03-18T19:09:22.641134+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:35.575921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.637953+0000","last_change":"2024-03-18T19:09:22.209622+0000","last_active":"2024-03-18T19:09:22.637953+0000","last_peered":"2024-03-18T19:09:22.637953+0000","last_clean":"2024-03-18T19:09:22.637953+0000","last_became_active":"2024-03-18T19:09:22.206732+0000","last_became_peered":"2024-03-18T19:09:22.206732+0000","last_unstale":"2024-03-18T19:09:22.637953+0000","last_undegraded":"2024-03-18T19:09:22.637953+0000","last_fullsized":"2024-03-18T19:09:22.637953+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:21:43.532626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.645250+0000","last_change":"2024-03-18T19:09:22.643878+0000","last_active":"2024-03-18T19:09:22.645250+0000","last_peered":"2024-03-18T19:09:22.645250+0000","last_clean":"2024-03-18T19:09:22.645250+0000","last_became_active":"2024-03-18T19:09:22.643632+0000","last_became_peered":"2024-03-18T19:09:22.643632+0000","last_unstale":"2024-03-18T19:09:22.645250+0000","last_undegraded":"2024-03-18T19:09:22.645250+0000","last_fullsized":"2024-03-18T19:09:22.645250+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:49:31.795796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.637797+0000","last_change":"2024-03-18T19:09:22.210688+0000","last_active":"2024-03-18T19:09:22.637797+0000","last_peered":"2024-03-18T19:09:22.637797+0000","last_clean":"2024-03-18T19:09:22.637797+0000","last_became_active":"2024-03-18T19:09:22.210334+0000","last_became_peered":"2024-03-18T19:09:22.210334+0000","last_unstale":"2024-03-18T19:09:22.637797+0000","last_undegraded":"2024-03-18T19:09:22.637797+0000","last_fullsized":"2024-03-18T19:09:22.637797+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:30:31.862656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.f","version":"146'1","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.646771+0000","last_change":"2024-03-18T19:09:22.211398+0000","last_active":"2024-03-18T19:09:22.646771+0000","last_peered":"2024-03-18T19:09:22.646771+0000","last_clean":"2024-03-18T19:09:22.646771+0000","last_became_active":"2024-03-18T19:09:22.211147+0000","last_became_peered":"2024-03-18T19:09:22.211147+0000","last_unstale":"2024-03-18T19:09:22.646771+0000","last_undegraded":"2024-03-18T19:09:22.646771+0000","last_fullsized":"2024-03-18T19:09:22.646771+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:49:48.268109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.e","version":"146'2","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.647069+0000","last_change":"2024-03-18T19:09:22.217258+0000","last_active":"2024-03-18T19:09:22.647069+0000","last_peered":"2024-03-18T19:09:22.647069+0000","last_clean":"2024-03-18T19:09:22.647069+0000","last_became_active":"2024-03-18T19:09:22.216952+0000","last_became_peered":"2024-03-18T19:09:22.216952+0000","last_unstale":"2024-03-18T19:09:22.647069+0000","last_undegraded":"2024-03-18T19:09:22.647069+0000","last_fullsized":"2024-03-18T19:09:22.647069+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:28:39.908259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.d","version":"146'1","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.328031+0000","last_change":"2024-03-18T19:09:22.210772+0000","last_active":"2024-03-18T19:09:24.328031+0000","last_peered":"2024-03-18T19:09:24.328031+0000","last_clean":"2024-03-18T19:09:24.328031+0000","last_became_active":"2024-03-18T19:09:22.210511+0000","last_became_peered":"2024-03-18T19:09:22.210511+0000","last_unstale":"2024-03-18T19:09:24.328031+0000","last_undegraded":"2024-03-18T19:09:24.328031+0000","last_fullsized":"2024-03-18T19:09:24.328031+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:46:43.196183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327829+0000","last_change":"2024-03-18T19:09:22.216994+0000","last_active":"2024-03-18T19:09:24.327829+0000","last_peered":"2024-03-18T19:09:24.327829+0000","last_clean":"2024-03-18T19:09:24.327829+0000","last_became_active":"2024-03-18T19:09:22.216719+0000","last_became_peered":"2024-03-18T19:09:22.216719+0000","last_unstale":"2024-03-18T19:09:24.327829+0000","last_undegraded":"2024-03-18T19:09:24.327829+0000","last_fullsized":"2024-03-18T19:09:24.327829+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:12:31.452285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:21.185417+0000","last_change":"2024-03-18T19:09:21.185417+0000","last_active":"2024-03-18T19:09:21.185417+0000","last_peered":"2024-03-18T19:09:21.185417+0000","last_clean":"2024-03-18T19:09:21.185417+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:21.185417+0000","last_undegraded":"2024-03-18T19:09:21.185417+0000","last_fullsized":"2024-03-18T19:09:21.185417+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.654972+0000","last_change":"2024-03-18T19:09:22.653564+0000","last_active":"2024-03-18T19:09:22.654972+0000","last_peered":"2024-03-18T19:09:22.654972+0000","last_clean":"2024-03-18T19:09:22.654972+0000","last_became_active":"2024-03-18T19:09:22.653288+0000","last_became_peered":"2024-03-18T19:09:22.653288+0000","last_unstale":"2024-03-18T19:09:22.654972+0000","last_undegraded":"2024-03-18T19:09:22.654972+0000","last_fullsized":"2024-03-18T19:09:22.654972+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:51:38.942600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327668+0000","last_change":"2024-03-18T19:09:22.214404+0000","last_active":"2024-03-18T19:09:24.327668+0000","last_peered":"2024-03-18T19:09:24.327668+0000","last_clean":"2024-03-18T19:09:24.327668+0000","last_became_active":"2024-03-18T19:09:22.214057+0000","last_became_peered":"2024-03-18T19:09:22.214057+0000","last_unstale":"2024-03-18T19:09:24.327668+0000","last_undegraded":"2024-03-18T19:09:24.327668+0000","last_fullsized":"2024-03-18T19:09:24.327668+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:49:48.268109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.649333+0000","last_change":"2024-03-18T19:09:22.646657+0000","last_active":"2024-03-18T19:09:22.649333+0000","last_peered":"2024-03-18T19:09:22.649333+0000","last_clean":"2024-03-18T19:09:22.649333+0000","last_became_active":"2024-03-18T19:09:22.646337+0000","last_became_peered":"2024-03-18T19:09:22.646337+0000","last_unstale":"2024-03-18T19:09:22.649333+0000","last_undegraded":"2024-03-18T19:09:22.649333+0000","last_fullsized":"2024-03-18T19:09:22.649333+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:01.296112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.637902+0000","last_change":"2024-03-18T19:09:22.205466+0000","last_active":"2024-03-18T19:09:22.637902+0000","last_peered":"2024-03-18T19:09:22.637902+0000","last_clean":"2024-03-18T19:09:22.637902+0000","last_became_active":"2024-03-18T19:09:22.205156+0000","last_became_peered":"2024-03-18T19:09:22.205156+0000","last_unstale":"2024-03-18T19:09:22.637902+0000","last_undegraded":"2024-03-18T19:09:22.637902+0000","last_fullsized":"2024-03-18T19:09:22.637902+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:12:58.020407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327461+0000","last_change":"2024-03-18T19:09:22.206066+0000","last_active":"2024-03-18T19:09:24.327461+0000","last_peered":"2024-03-18T19:09:24.327461+0000","last_clean":"2024-03-18T19:09:24.327461+0000","last_became_active":"2024-03-18T19:09:22.205784+0000","last_became_peered":"2024-03-18T19:09:22.205784+0000","last_unstale":"2024-03-18T19:09:24.327461+0000","last_undegraded":"2024-03-18T19:09:24.327461+0000","last_fullsized":"2024-03-18T19:09:24.327461+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:23:13.662431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:21.185417+0000","last_change":"2024-03-18T19:09:21.185417+0000","last_active":"2024-03-18T19:09:21.185417+0000","last_peered":"2024-03-18T19:09:21.185417+0000","last_clean":"2024-03-18T19:09:21.185417+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:21.185417+0000","last_undegraded":"2024-03-18T19:09:21.185417+0000","last_fullsized":"2024-03-18T19:09:21.185417+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.640184+0000","last_change":"2024-03-18T19:09:22.637480+0000","last_active":"2024-03-18T19:09:22.640184+0000","last_peered":"2024-03-18T19:09:22.640184+0000","last_clean":"2024-03-18T19:09:22.640184+0000","last_became_active":"2024-03-18T19:09:22.637210+0000","last_became_peered":"2024-03-18T19:09:22.637210+0000","last_unstale":"2024-03-18T19:09:22.640184+0000","last_undegraded":"2024-03-18T19:09:22.640184+0000","last_fullsized":"2024-03-18T19:09:22.640184+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:30.062865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.635335+0000","last_change":"2024-03-18T19:09:22.215407+0000","last_active":"2024-03-18T19:09:23.635335+0000","last_peered":"2024-03-18T19:09:23.635335+0000","last_clean":"2024-03-18T19:09:23.635335+0000","last_became_active":"2024-03-18T19:09:22.215180+0000","last_became_peered":"2024-03-18T19:09:22.215180+0000","last_unstale":"2024-03-18T19:09:23.635335+0000","last_undegraded":"2024-03-18T19:09:23.635335+0000","last_fullsized":"2024-03-18T19:09:23.635335+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:49:48.268109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327985+0000","last_change":"2024-03-18T19:09:22.644776+0000","last_active":"2024-03-18T19:09:24.327985+0000","last_peered":"2024-03-18T19:09:24.327985+0000","last_clean":"2024-03-18T19:09:24.327985+0000","last_became_active":"2024-03-18T19:09:22.644275+0000","last_became_peered":"2024-03-18T19:09:22.644275+0000","last_unstale":"2024-03-18T19:09:24.327985+0000","last_undegraded":"2024-03-18T19:09:24.327985+0000","last_fullsized":"2024-03-18T19:09:24.327985+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:08.116636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.633808+0000","last_change":"2024-03-18T19:09:22.214530+0000","last_active":"2024-03-18T19:09:23.633808+0000","last_peered":"2024-03-18T19:09:23.633808+0000","last_clean":"2024-03-18T19:09:23.633808+0000","last_became_active":"2024-03-18T19:09:22.214220+0000","last_became_peered":"2024-03-18T19:09:22.214220+0000","last_unstale":"2024-03-18T19:09:23.633808+0000","last_undegraded":"2024-03-18T19:09:23.633808+0000","last_fullsized":"2024-03-18T19:09:23.633808+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:12:31.452285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.656146+0000","last_change":"2024-03-18T19:09:22.655155+0000","last_active":"2024-03-18T19:09:22.656146+0000","last_peered":"2024-03-18T19:09:22.656146+0000","last_clean":"2024-03-18T19:09:22.656146+0000","last_became_active":"2024-03-18T19:09:22.654886+0000","last_became_peered":"2024-03-18T19:09:22.654886+0000","last_unstale":"2024-03-18T19:09:22.656146+0000","last_undegraded":"2024-03-18T19:09:22.656146+0000","last_fullsized":"2024-03-18T19:09:22.656146+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:58:31.158678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:21.185417+0000","last_change":"2024-03-18T19:09:21.185417+0000","last_active":"2024-03-18T19:09:21.185417+0000","last_peered":"2024-03-18T19:09:21.185417+0000","last_clean":"2024-03-18T19:09:21.185417+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:21.185417+0000","last_undegraded":"2024-03-18T19:09:21.185417+0000","last_fullsized":"2024-03-18T19:09:21.185417+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:22.190544+0000","last_change":"2024-03-18T19:09:22.190544+0000","last_active":"2024-03-18T19:09:22.190544+0000","last_peered":"2024-03-18T19:09:22.190544+0000","last_clean":"2024-03-18T19:09:22.190544+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:22.190544+0000","last_undegraded":"2024-03-18T19:09:22.190544+0000","last_fullsized":"2024-03-18T19:09:22.190544+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:21.185417+0000","last_change":"2024-03-18T19:09:21.185417+0000","last_active":"2024-03-18T19:09:21.185417+0000","last_peered":"2024-03-18T19:09:21.185417+0000","last_clean":"2024-03-18T19:09:21.185417+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:21.185417+0000","last_undegraded":"2024-03-18T19:09:21.185417+0000","last_fullsized":"2024-03-18T19:09:21.185417+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.649758+0000","last_change":"2024-03-18T19:09:22.647165+0000","last_active":"2024-03-18T19:09:22.649758+0000","last_peered":"2024-03-18T19:09:22.649758+0000","last_clean":"2024-03-18T19:09:22.649758+0000","last_became_active":"2024-03-18T19:09:22.646619+0000","last_became_peered":"2024-03-18T19:09:22.646619+0000","last_unstale":"2024-03-18T19:09:22.649758+0000","last_undegraded":"2024-03-18T19:09:22.649758+0000","last_fullsized":"2024-03-18T19:09:22.649758+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:15.237092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:21.185417+0000","last_change":"2024-03-18T19:09:21.185417+0000","last_active":"2024-03-18T19:09:21.185417+0000","last_peered":"2024-03-18T19:09:21.185417+0000","last_clean":"2024-03-18T19:09:21.185417+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:21.185417+0000","last_undegraded":"2024-03-18T19:09:21.185417+0000","last_fullsized":"2024-03-18T19:09:21.185417+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327782+0000","last_change":"2024-03-18T19:09:22.637700+0000","last_active":"2024-03-18T19:09:24.327782+0000","last_peered":"2024-03-18T19:09:24.327782+0000","last_clean":"2024-03-18T19:09:24.327782+0000","last_became_active":"2024-03-18T19:09:22.637417+0000","last_became_peered":"2024-03-18T19:09:22.637417+0000","last_unstale":"2024-03-18T19:09:24.327782+0000","last_undegraded":"2024-03-18T19:09:24.327782+0000","last_fullsized":"2024-03-18T19:09:24.327782+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:48:33.430169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:09:21.185417+0000","last_change":"2024-03-18T19:09:21.185417+0000","last_active":"2024-03-18T19:09:21.185417+0000","last_peered":"2024-03-18T19:09:21.185417+0000","last_clean":"2024-03-18T19:09:21.185417+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:09:21.185417+0000","last_undegraded":"2024-03-18T19:09:21.185417+0000","last_fullsized":"2024-03-18T19:09:21.185417+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.640313+0000","last_change":"2024-03-18T19:09:22.637602+0000","last_active":"2024-03-18T19:09:22.640313+0000","last_peered":"2024-03-18T19:09:22.640313+0000","last_clean":"2024-03-18T19:09:22.640313+0000","last_became_active":"2024-03-18T19:09:22.637351+0000","last_became_peered":"2024-03-18T19:09:22.637351+0000","last_unstale":"2024-03-18T19:09:22.640313+0000","last_undegraded":"2024-03-18T19:09:22.640313+0000","last_fullsized":"2024-03-18T19:09:22.640313+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:19.098002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327456+0000","last_change":"2024-03-18T19:09:22.207744+0000","last_active":"2024-03-18T19:09:24.327456+0000","last_peered":"2024-03-18T19:09:24.327456+0000","last_clean":"2024-03-18T19:09:24.327456+0000","last_became_active":"2024-03-18T19:09:22.206341+0000","last_became_peered":"2024-03-18T19:09:22.206341+0000","last_unstale":"2024-03-18T19:09:24.327456+0000","last_undegraded":"2024-03-18T19:09:24.327456+0000","last_fullsized":"2024-03-18T19:09:24.327456+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:45:55.970780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327830+0000","last_change":"2024-03-18T19:09:22.639977+0000","last_active":"2024-03-18T19:09:24.327830+0000","last_peered":"2024-03-18T19:09:24.327830+0000","last_clean":"2024-03-18T19:09:24.327830+0000","last_became_active":"2024-03-18T19:09:22.639700+0000","last_became_peered":"2024-03-18T19:09:22.639700+0000","last_unstale":"2024-03-18T19:09:24.327830+0000","last_undegraded":"2024-03-18T19:09:24.327830+0000","last_fullsized":"2024-03-18T19:09:24.327830+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:43:09.563639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"146'8","reported_seq":18,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.652528+0000","last_change":"2024-03-18T19:09:22.215782+0000","last_active":"2024-03-18T19:09:22.652528+0000","last_peered":"2024-03-18T19:09:22.652528+0000","last_clean":"2024-03-18T19:09:22.652528+0000","last_became_active":"2024-03-18T19:09:22.215478+0000","last_became_peered":"2024-03-18T19:09:22.215478+0000","last_unstale":"2024-03-18T19:09:22.652528+0000","last_undegraded":"2024-03-18T19:09:22.652528+0000","last_fullsized":"2024-03-18T19:09:22.652528+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:08.578704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"136'227","reported_seq":664,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.630751+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T19:09:22.630751+0000","last_peered":"2024-03-18T19:09:22.630751+0000","last_clean":"2024-03-18T19:09:22.630751+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T19:09:22.630751+0000","last_undegraded":"2024-03-18T19:09:22.630751+0000","last_fullsized":"2024-03-18T19:09:22.630751+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":74,"acting":74,"num_store_stats":5},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2272,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":40,"acting":40,"num_store_stats":5},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575077,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59732,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312108,"statfs":{"total":96636764160,"available":96575598592,"internally_reserved":0,"allocated":2424832,"data_stored":1504495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.55300000000000005,"15min":0.56599999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.27800000000000002},"max":{"1min":0.76200000000000001,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55800000000000005,"15min":0.57899999999999996},"min":{"1min":0.373,"5min":0.33700000000000002,"15min":0.318},"max":{"1min":0.78300000000000003,"5min":1.387,"15min":1.387},"last":0.67800000000000005}]},{"osd":1,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.53500000000000003,"15min":0.56799999999999995},"min":{"1min":0.34399999999999997,"5min":0.33200000000000002,"15min":0.307},"max":{"1min":0.73299999999999998,"5min":0.81699999999999995,"15min":1.484},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55100000000000005,"15min":0.58599999999999997},"min":{"1min":0.39500000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.92000000000000004},"last":0.72299999999999998}]},{"osd":2,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55200000000000005,"15min":0.57099999999999995},"min":{"1min":0.38200000000000001,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":0.68000000000000005,"5min":1.3600000000000001,"15min":1.5129999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57499999999999996,"15min":0.59099999999999997},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73499999999999999,"5min":1.327,"15min":1.536},"last":0.65100000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55400000000000005,"15min":0.57599999999999996},"min":{"1min":0.38800000000000001,"5min":0.35599999999999998,"15min":0.318},"max":{"1min":0.745,"5min":0.79100000000000004,"15min":1.5489999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57399999999999995,"15min":0.59799999999999998},"min":{"1min":0.441,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.78300000000000003,"5min":0.81699999999999995,"15min":1.5660000000000001},"last":0.68700000000000006}]},{"osd":4,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.45800000000000002,"15min":0.45300000000000001},"min":{"1min":0.30299999999999999,"5min":0.27800000000000002,"15min":0.27100000000000002},"max":{"1min":0.748,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.495},{"interface":"front","average":{"1min":0.505,"5min":0.46999999999999997,"15min":0.47299999999999998},"min":{"1min":0.35299999999999998,"5min":0.27500000000000002,"15min":0.25800000000000001},"max":{"1min":0.63900000000000001,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.46700000000000003}]},{"osd":5,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.48499999999999999,"15min":0.47699999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.26000000000000001},"max":{"1min":0.68600000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.51500000000000001,"15min":0.51400000000000001},"min":{"1min":0.33600000000000002,"5min":0.317,"15min":0.30299999999999999},"max":{"1min":0.70699999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.56499999999999995}]},{"osd":6,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.54300000000000004},"min":{"1min":0.34000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.74099999999999999,"5min":0.81799999999999995,"15min":0.82799999999999996},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53600000000000003,"15min":0.53000000000000003},"min":{"1min":0.36399999999999999,"5min":0.311,"15min":0.29099999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.66500000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575077,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32912,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338928,"statfs":{"total":96636764160,"available":96603062272,"internally_reserved":0,"allocated":1830912,"data_stored":920581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52500000000000002,"15min":0.55500000000000005},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28499999999999998},"max":{"1min":0.67800000000000005,"5min":0.73699999999999999,"15min":0.76800000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53900000000000003,"15min":0.56499999999999995},"min":{"1min":0.36199999999999999,"5min":0.35199999999999998,"15min":0.27400000000000002},"max":{"1min":0.71099999999999997,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.46300000000000002}]},{"osd":1,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.56000000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.77400000000000002},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.56499999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.32000000000000001},"max":{"1min":0.78900000000000003,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.374}]},{"osd":2,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55400000000000005,"15min":0.57299999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.30499999999999999},"max":{"1min":0.73699999999999999,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54500000000000004,"15min":0.57399999999999995},"min":{"1min":0.40999999999999998,"5min":0.35299999999999998,"15min":0.30299999999999999},"max":{"1min":0.73399999999999999,"5min":0.753,"15min":0.83199999999999996},"last":0.55400000000000005}]},{"osd":3,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.58399999999999996},"min":{"1min":0.40799999999999997,"5min":0.372,"15min":0.32100000000000001},"max":{"1min":0.76400000000000001,"5min":0.78400000000000003,"15min":0.89600000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55400000000000005,"15min":0.57199999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72099999999999997,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.442}]},{"osd":4,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46700000000000003,"15min":0.45000000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.432},{"interface":"front","average":{"1min":0.495,"5min":0.48899999999999999,"15min":0.47699999999999998},"min":{"1min":0.32200000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.623}]},{"osd":5,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.75,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53400000000000003,"15min":0.51600000000000001},"min":{"1min":0.34899999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.57999999999999996}]},{"osd":7,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54400000000000004,"15min":0.52900000000000003},"min":{"1min":0.371,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55600000000000005,"15min":0.53600000000000003},"min":{"1min":0.371,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.80500000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.56799999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575077,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36996,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334844,"statfs":{"total":96636764160,"available":96598880256,"internally_reserved":0,"allocated":1818624,"data_stored":913373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.35499999999999998,"15min":0.373},"min":{"1min":0.216,"5min":0.216,"15min":0.19600000000000001},"max":{"1min":0.58699999999999997,"5min":0.63300000000000001,"15min":0.66300000000000003},"last":0.371},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.38600000000000001,"15min":0.41299999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.57099999999999995,"5min":0.66500000000000004,"15min":0.73699999999999999},"last":0.35899999999999999}]},{"osd":2,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38900000000000001,"15min":0.40400000000000003},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.59299999999999997,"5min":0.64200000000000002,"15min":0.65400000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.434,"15min":0.44500000000000001},"min":{"1min":0.29599999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.63300000000000001,"5min":0.66500000000000004,"15min":0.71699999999999997},"last":0.39300000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.44500000000000001,"15min":0.46400000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.60399999999999998,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40200000000000002,"15min":0.41899999999999998},"min":{"1min":0.25900000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.61199999999999999,"5min":0.67800000000000005,"15min":0.77000000000000002},"last":0.34100000000000003}]},{"osd":4,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60499999999999998,"15min":0.628},"min":{"1min":0.46200000000000002,"5min":0.33400000000000002,"15min":0.32800000000000001},"max":{"1min":0.89600000000000002,"5min":0.92300000000000004,"15min":0.93999999999999995},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.62,"15min":0.64100000000000001},"min":{"1min":0.443,"5min":0.439,"15min":0.36899999999999999},"max":{"1min":0.86499999999999999,"5min":0.97499999999999998,"15min":1.018},"last":0.499}]},{"osd":5,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60399999999999998,"15min":0.63700000000000001},"min":{"1min":0.40500000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88700000000000001,"5min":0.89500000000000002,"15min":0.99199999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.63200000000000001,"15min":0.65600000000000003},"min":{"1min":0.47199999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":1.006},"last":0.54800000000000004}]},{"osd":6,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.60499999999999998,"15min":0.63400000000000001},"min":{"1min":0.42899999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.878,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.64600000000000002,"15min":0.67000000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.39100000000000001},"max":{"1min":0.91300000000000003,"5min":0.94899999999999995,"15min":0.98099999999999998},"last":0.46100000000000002}]},{"osd":7,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63200000000000001,"15min":0.65900000000000003},"min":{"1min":0.45100000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81899999999999995,"5min":0.90600000000000003,"15min":0.94799999999999995},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63400000000000001,"15min":0.66200000000000003},"min":{"1min":0.48299999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90800000000000003,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.59199999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575077,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37532,"kb_used_data":2316,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334308,"statfs":{"total":96636764160,"available":96598331392,"internally_reserved":0,"allocated":2371584,"data_stored":1483808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36299999999999999,"15min":0.376},"min":{"1min":0.218,"5min":0.19700000000000001,"15min":0.19400000000000001},"max":{"1min":0.59899999999999998,"5min":0.60499999999999998,"15min":0.621},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39400000000000002,"15min":0.40999999999999998},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20000000000000001},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.63200000000000001},"last":0.35699999999999998}]},{"osd":2,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40699999999999997,"15min":0.41699999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.21099999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.42299999999999999,"15min":0.432},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.59399999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.39900000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.40300000000000002,"15min":0.41299999999999998},"min":{"1min":0.27200000000000002,"5min":0.26500000000000001,"15min":0.217},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.64100000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.443,"5min":0.45500000000000002,"15min":0.46800000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.28799999999999998},"max":{"1min":0.64300000000000002,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.312}]},{"osd":4,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.60099999999999998,"15min":0.63300000000000001},"min":{"1min":0.376,"5min":0.35699999999999998,"15min":0.27200000000000002},"max":{"1min":0.76900000000000002,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.60899999999999999,"15min":0.63500000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.315},"max":{"1min":0.75600000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.68600000000000005}]},{"osd":5,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.627,"15min":0.64300000000000002},"min":{"1min":0.432,"5min":0.376,"15min":0.34300000000000003},"max":{"1min":0.78100000000000003,"5min":1.248,"15min":1.248},"last":0.629},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.63800000000000001,"15min":0.65800000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.31},"max":{"1min":0.84899999999999998,"5min":1.399,"15min":1.399},"last":0.69899999999999995}]},{"osd":6,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.63200000000000001,"15min":0.65500000000000003},"min":{"1min":0.44700000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.86099999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.64900000000000002,"15min":0.66700000000000004},"min":{"1min":0.47599999999999998,"5min":0.42399999999999999,"15min":0.40999999999999998},"max":{"1min":0.83499999999999996,"5min":1.284,"15min":1.284},"last":0.53600000000000003}]},{"osd":7,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.64100000000000001,"15min":0.66300000000000003},"min":{"1min":0.42399999999999999,"5min":0.39300000000000002,"15min":0.34899999999999998},"max":{"1min":0.878,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64000000000000001,"15min":0.66400000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.38500000000000001},"max":{"1min":0.86599999999999999,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.58199999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575078,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59136,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312704,"statfs":{"total":96636764160,"available":96576208896,"internally_reserved":0,"allocated":1814528,"data_stored":913351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34999999999999998,"15min":0.36799999999999999},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.498,"5min":0.66600000000000004,"15min":0.78300000000000003},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.44400000000000001,"15min":0.45500000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.217},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.77200000000000002},"last":0.31900000000000001}]},{"osd":1,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.435,"15min":0.46400000000000002},"min":{"1min":0.31,"5min":0.27300000000000002,"15min":0.27100000000000002},"max":{"1min":0.62,"5min":0.69499999999999995,"15min":0.90500000000000003},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.38400000000000001,"15min":0.39600000000000002},"min":{"1min":0.26700000000000002,"5min":0.22800000000000001,"15min":0.216},"max":{"1min":0.54300000000000004,"5min":0.64500000000000002,"15min":0.66700000000000004},"last":0.34699999999999998}]},{"osd":3,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.24199999999999999},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47599999999999998,"15min":0.499},"min":{"1min":0.33400000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.80100000000000005},"last":0.36299999999999999}]},{"osd":4,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58399999999999996,"15min":0.60999999999999999},"min":{"1min":0.33500000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76100000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.59799999999999998,"15min":0.624},"min":{"1min":0.38600000000000001,"5min":0.378,"15min":0.32100000000000001},"max":{"1min":1.355,"5min":1.448,"15min":1.4730000000000001},"last":0.45300000000000001}]},{"osd":5,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.61899999999999999,"15min":0.64100000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.34699999999999998},"max":{"1min":0.90600000000000003,"5min":1.327,"15min":1.8109999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.59999999999999998,"15min":0.63100000000000001},"min":{"1min":0.376,"5min":0.36899999999999999,"15min":0.33400000000000002},"max":{"1min":0.75,"5min":1.4630000000000001,"15min":1.948},"last":0.52800000000000002}]},{"osd":6,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.624,"15min":0.65500000000000003},"min":{"1min":0.42599999999999999,"5min":0.39600000000000002,"15min":0.35899999999999999},"max":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":2.9470000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65600000000000003,"15min":0.67900000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.38100000000000001},"max":{"1min":0.94899999999999995,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.48099999999999998}]},{"osd":7,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.625,"15min":0.64700000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.89100000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64000000000000001,"15min":0.66100000000000003},"min":{"1min":0.437,"5min":0.38300000000000001,"15min":0.371},"max":{"1min":1.409,"5min":1.482,"15min":1.482},"last":0.51400000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575078,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37004,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334836,"statfs":{"total":96636764160,"available":96598872064,"internally_reserved":0,"allocated":1830912,"data_stored":914949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34699999999999998,"15min":0.374},"min":{"1min":0.23499999999999999,"5min":0.221,"15min":0.21099999999999999},"max":{"1min":0.442,"5min":0.63900000000000001,"15min":1.204},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38,"15min":0.39900000000000002},"min":{"1min":0.25700000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.52900000000000003,"5min":0.66900000000000004,"15min":1.137},"last":0.33600000000000002}]},{"osd":1,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.36899999999999999,"15min":0.39500000000000002},"min":{"1min":0.27700000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.502,"5min":0.629,"15min":1.1220000000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40699999999999997,"15min":0.441},"min":{"1min":0.29499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.57399999999999995,"5min":0.76600000000000001,"15min":1.1080000000000001},"last":0.312}]},{"osd":2,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.41799999999999998,"15min":0.44600000000000001},"min":{"1min":0.308,"5min":0.29499999999999998,"15min":0.28399999999999997},"max":{"1min":0.61799999999999999,"5min":0.67500000000000004,"15min":1.1879999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.44500000000000001,"15min":0.47299999999999998},"min":{"1min":0.33400000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.55600000000000005,"5min":0.71499999999999997,"15min":1.2290000000000001},"last":0.44500000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.58099999999999996,"15min":0.62},"min":{"1min":0.441,"5min":0.38100000000000001,"15min":0.34300000000000003},"max":{"1min":0.82899999999999996,"5min":1.052,"15min":1.871},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60199999999999998,"15min":0.629},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.39600000000000002},"max":{"1min":0.81000000000000005,"5min":0.90600000000000003,"15min":1.764},"last":0.59099999999999997}]},{"osd":5,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.63400000000000001,"15min":0.66200000000000003},"min":{"1min":0.47099999999999997,"5min":0.39200000000000002,"15min":0.379},"max":{"1min":0.84999999999999998,"5min":1.0249999999999999,"15min":2.5329999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.65500000000000003,"15min":0.68200000000000005},"min":{"1min":0.43099999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.92900000000000005,"5min":0.94599999999999995,"15min":1.794},"last":0.51000000000000001}]},{"osd":6,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.63900000000000001,"15min":0.67300000000000004},"min":{"1min":0.496,"5min":0.42499999999999999,"15min":0.41499999999999998},"max":{"1min":0.89400000000000002,"5min":1.1060000000000001,"15min":2.7389999999999999},"last":0.625},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.65000000000000002,"15min":0.67600000000000005},"min":{"1min":0.45000000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.91200000000000003,"5min":0.98299999999999998,"15min":1.8069999999999999},"last":0.54200000000000004}]},{"osd":7,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63100000000000001,"15min":0.66400000000000003},"min":{"1min":0.43099999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":1.732},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.63400000000000001,"15min":0.66900000000000004},"min":{"1min":0.51800000000000002,"5min":0.45900000000000002,"15min":0.40799999999999997},"max":{"1min":0.86299999999999999,"5min":1.085,"15min":2.7050000000000001},"last":0.52600000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575077,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32852,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1777664,"data_stored":893440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.56699999999999995,"15min":0.57499999999999996},"min":{"1min":0.43099999999999999,"5min":0.31,"15min":0.246},"max":{"1min":2.3210000000000002,"5min":2.3210000000000002,"15min":2.3210000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.41699999999999998,"5min":0.35299999999999998,"15min":0.26000000000000001},"max":{"1min":1.7430000000000001,"5min":3.645,"15min":3.645},"last":0.66000000000000003}]},{"osd":1,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65000000000000002,"15min":0.61199999999999999},"min":{"1min":0.40699999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.6739999999999999,"5min":9.9499999999999993,"15min":9.9499999999999993},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54400000000000004,"15min":0.57399999999999995},"min":{"1min":0.40200000000000002,"5min":0.32300000000000001,"15min":0.308},"max":{"1min":0.80000000000000004,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.502}]},{"osd":2,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56699999999999995,"15min":0.57799999999999996},"min":{"1min":0.439,"5min":0.35899999999999999,"15min":0.28999999999999998},"max":{"1min":2.431,"5min":2.431,"15min":2.431},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.61499999999999999,"15min":0.61299999999999999},"min":{"1min":0.45700000000000002,"5min":0.35099999999999998,"15min":0.32600000000000001},"max":{"1min":2.165,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.58899999999999997}]},{"osd":3,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.55800000000000005,"15min":0.58299999999999996},"min":{"1min":0.39700000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76800000000000002,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.58799999999999997,"15min":0.59999999999999998},"min":{"1min":0.42599999999999999,"5min":0.39300000000000002,"15min":0.33900000000000002},"max":{"1min":2.5510000000000002,"5min":2.5510000000000002,"15min":2.5510000000000002},"last":0.72099999999999997}]},{"osd":5,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46300000000000002,"15min":0.46000000000000002},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.245},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.77500000000000002},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.51600000000000001,"15min":0.49199999999999999},"min":{"1min":0.307,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.73599999999999999,"5min":5.4900000000000002,"15min":5.4900000000000002},"last":0.61699999999999999}]},{"osd":6,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.49199999999999999,"15min":0.48499999999999999},"min":{"1min":0.33200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.5190000000000001,"5min":2.5190000000000001,"15min":2.5190000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.55100000000000005,"15min":0.52900000000000003},"min":{"1min":0.32600000000000001,"5min":0.317,"15min":0.28999999999999998},"max":{"1min":1.2689999999999999,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":0.70099999999999996}]},{"osd":7,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52200000000000002,"15min":0.51900000000000002},"min":{"1min":0.34999999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.94599999999999995,"5min":1.55,"15min":1.55},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.54800000000000004,"15min":0.54900000000000004},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.751}]}]},{"osd":5,"up_from":13,"seq":55834575076,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50908,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320932,"statfs":{"total":96636764160,"available":96584634368,"internally_reserved":0,"allocated":1777664,"data_stored":893440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56299999999999994,"15min":0.56999999999999995},"min":{"1min":0.38600000000000001,"5min":0.36399999999999999,"15min":0.26600000000000001},"max":{"1min":0.75600000000000001,"5min":0.86099999999999999,"15min":1.1679999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.40300000000000002,"5min":0.35299999999999998,"15min":0.25800000000000001},"max":{"1min":0.71099999999999997,"5min":0.80600000000000005,"15min":1.1319999999999999},"last":0.64000000000000001}]},{"osd":1,"last update":"Mon Mar 18 19:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55500000000000005,"15min":0.56999999999999995},"min":{"1min":0.39300000000000002,"5min":0.32600000000000001,"15min":0.29599999999999999},"max":{"1min":0.73899999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57699999999999996,"15min":0.58199999999999996},"min":{"1min":0.36499999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.80600000000000005,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.60499999999999998}]},{"osd":2,"last update":"Mon Mar 18 19:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55800000000000005,"15min":0.57699999999999996},"min":{"1min":0.41699999999999998,"5min":0.30399999999999999,"15min":0.28399999999999997},"max":{"1min":0.72699999999999998,"5min":1.153,"15min":1.1579999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56200000000000006,"15min":0.57199999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33200000000000002},"max":{"1min":0.77500000000000002,"5min":0.79500000000000004,"15min":1.0569999999999999},"last":0.62}]},{"osd":3,"last update":"Mon Mar 18 19:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58299999999999996,"15min":0.58699999999999997},"min":{"1min":0.38600000000000001,"5min":0.35799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78400000000000003,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.58199999999999996,"15min":0.59799999999999998},"min":{"1min":0.39000000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76100000000000001,"5min":1.212,"15min":1.212},"last":0.77000000000000002}]},{"osd":4,"last update":"Mon Mar 18 19:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46800000000000003,"15min":0.46300000000000002},"min":{"1min":0.33300000000000002,"5min":0.255,"15min":0.24199999999999999},"max":{"1min":0.69899999999999995,"5min":1.248,"15min":1.248},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.49299999999999999,"15min":0.48799999999999999},"min":{"1min":0.32500000000000001,"5min":0.28299999999999997,"15min":0.23100000000000001},"max":{"1min":0.72299999999999998,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.82599999999999996}]},{"osd":6,"last update":"Mon Mar 18 19:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46200000000000002,"15min":0.47099999999999997},"min":{"1min":0.32400000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.68899999999999995,"5min":0.78200000000000003,"15min":1.0069999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.51200000000000001,"15min":0.50900000000000001},"min":{"1min":0.41799999999999998,"5min":0.32100000000000001,"15min":0.26800000000000002},"max":{"1min":0.67200000000000004,"5min":0.83199999999999996,"15min":0.94599999999999995},"last":0.71799999999999997}]},{"osd":7,"last update":"Mon Mar 18 19:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52200000000000002,"15min":0.52300000000000002},"min":{"1min":0.373,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.77600000000000002,"5min":1.282,"15min":1.282},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28499999999999998},"max":{"1min":0.76900000000000002,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.81000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T19:09:26.794 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T19:09:26.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.958+0000 7f52e65f2700 1 Processor -- start 2024-03-18T19:09:26.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.958+0000 7f52e65f2700 1 -- start start 2024-03-18T19:09:26.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.959+0000 7f52e65f2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f52e00ae600 0x7f52e00ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:26.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.959+0000 7f52e65f2700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f52e015a2e0 0x7f52e00ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:26.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.959+0000 7f52e65f2700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e006e270 con 0x7f52e00cc170 2024-03-18T19:09:26.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.959+0000 7f52e65f2700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52e00ae050 con 0x7f52e00ae600 2024-03-18T19:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.959+0000 7f52e65f2700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52e006f690 con 0x7f52e015a2e0 2024-03-18T19:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.959+0000 7f52dffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f52e00ae600 0x7f52e00ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.959+0000 7f52dffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f52e00ae600 0x7f52e00ad790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38962/0 (socket says 172.21.15.1:38962) 2024-03-18T19:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.959+0000 7f52dffff700 1 -- 172.21.15.1:0/2800092100 learned_addr learned my addr 172.21.15.1:0/2800092100 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.959+0000 7f52dffff700 1 -- 172.21.15.1:0/2800092100 >> v2:172.21.15.165:3300/0 conn(0x7f52e015a2e0 msgr2=0x7f52e00ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.959+0000 7f52dffff700 1 --2- 172.21.15.1:0/2800092100 >> v2:172.21.15.165:3300/0 conn(0x7f52e015a2e0 0x7f52e00ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.959+0000 7f52dffff700 1 -- 172.21.15.1:0/2800092100 >> v1:172.21.15.165:6789/0 conn(0x7f52e00cc170 legacy=0x7f52e006ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.959+0000 7f52dffff700 1 -- 172.21.15.1:0/2800092100 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e00aef20 con 0x7f52e00ae600 2024-03-18T19:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.959+0000 7f52dffff700 1 --2- 172.21.15.1:0/2800092100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52e00ae600 0x7f52e00ad790 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f52e0073a90 tx=0x7f52c800be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4da1e8474aab1166 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.960+0000 7f52deffd700 1 -- 172.21.15.1:0/2800092100 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c800dcf0 con 0x7f52e00ae600 2024-03-18T19:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.960+0000 7f52deffd700 1 -- 172.21.15.1:0/2800092100 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c800de50 con 0x7f52e00ae600 2024-03-18T19:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.960+0000 7f52deffd700 1 -- 172.21.15.1:0/2800092100 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8004250 con 0x7f52e00ae600 2024-03-18T19:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.960+0000 7f52e65f2700 1 -- 172.21.15.1:0/2800092100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52e00ae600 msgr2=0x7f52e00ad790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:26.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.960+0000 7f52e65f2700 1 --2- 172.21.15.1:0/2800092100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52e00ae600 0x7f52e00ad790 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f52e0073a90 tx=0x7f52c800be70 comp rx=0 tx=0).stop 2024-03-18T19:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.960+0000 7f52e65f2700 1 -- 172.21.15.1:0/2800092100 shutdown_connections 2024-03-18T19:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.960+0000 7f52e65f2700 1 --2- 172.21.15.1:0/2800092100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52e00ae600 0x7f52e00ad790 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.960+0000 7f52e65f2700 1 --2- 172.21.15.1:0/2800092100 >> v2:172.21.15.165:3300/0 conn(0x7f52e015a2e0 0x7f52e00ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.960+0000 7f52e65f2700 1 -- 172.21.15.1:0/2800092100 >> 172.21.15.1:0/2800092100 conn(0x7f52e006b860 msgr2=0x7f52e006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.960+0000 7f52e65f2700 1 -- 172.21.15.1:0/2800092100 shutdown_connections 2024-03-18T19:09:26.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.960+0000 7f52e65f2700 1 -- 172.21.15.1:0/2800092100 wait complete. 2024-03-18T19:09:26.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.961+0000 7f52e65f2700 1 Processor -- start 2024-03-18T19:09:26.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.961+0000 7f52e65f2700 1 -- start start 2024-03-18T19:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.962+0000 7f52e65f2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52e00ae600 0x7f52e01ef070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.962+0000 7f52e65f2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52e00cc170 0x7f52e0148a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.962+0000 7f52e65f2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f52e015a2e0 0x7f52e015d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.962+0000 7f52e65f2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e0148fa0 con 0x7f52e00ae600 2024-03-18T19:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.962+0000 7f52e65f2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e00cf4a0 con 0x7f52e00cc170 2024-03-18T19:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.962+0000 7f52e65f2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e015a840 con 0x7f52e015a2e0 2024-03-18T19:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.962+0000 7f52dffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52e00ae600 0x7f52e01ef070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.962+0000 7f52dffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52e00ae600 0x7f52e01ef070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38970/0 (socket says 172.21.15.1:38970) 2024-03-18T19:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.962+0000 7f52dffff700 1 -- 172.21.15.1:0/754678629 learned_addr learned my addr 172.21.15.1:0/754678629 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.963+0000 7f52e4b8f700 1 --2- 172.21.15.1:0/754678629 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f52e015a2e0 0x7f52e015d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.963+0000 7f52df7fe700 1 --2- 172.21.15.1:0/754678629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52e00cc170 0x7f52e0148a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.963+0000 7f52dffff700 1 -- 172.21.15.1:0/754678629 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f52e015a2e0 msgr2=0x7f52e015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:26.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.963+0000 7f52dffff700 1 --2- 172.21.15.1:0/754678629 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f52e015a2e0 0x7f52e015d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.963+0000 7f52dffff700 1 -- 172.21.15.1:0/754678629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52e00cc170 msgr2=0x7f52e0148a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.963+0000 7f52dffff700 1 --2- 172.21.15.1:0/754678629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52e00cc170 0x7f52e0148a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.963+0000 7f52dffff700 1 -- 172.21.15.1:0/754678629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c800b7f0 con 0x7f52e00ae600 2024-03-18T19:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.963+0000 7f52dffff700 1 --2- 172.21.15.1:0/754678629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52e00ae600 0x7f52e01ef070 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f52c8000c00 tx=0x7f52c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.963+0000 7f52dcff9700 1 -- 172.21.15.1:0/754678629 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8020070 con 0x7f52e00ae600 2024-03-18T19:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.963+0000 7f52dcff9700 1 -- 172.21.15.1:0/754678629 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c8004240 con 0x7f52e00ae600 2024-03-18T19:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.963+0000 7f52e65f2700 1 -- 172.21.15.1:0/754678629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e015ab20 con 0x7f52e00ae600 2024-03-18T19:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.963+0000 7f52dcff9700 1 -- 172.21.15.1:0/754678629 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8010690 con 0x7f52e00ae600 2024-03-18T19:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.963+0000 7f52e65f2700 1 -- 172.21.15.1:0/754678629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e015b0e0 con 0x7f52e00ae600 2024-03-18T19:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.964+0000 7f52e65f2700 1 -- 172.21.15.1:0/754678629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e00676a0 con 0x7f52e00ae600 2024-03-18T19:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.964+0000 7f52dcff9700 1 -- 172.21.15.1:0/754678629 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f52c80193f0 con 0x7f52e00ae600 2024-03-18T19:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.965+0000 7f52dcff9700 1 --2- 172.21.15.1:0/754678629 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f52c006adb0 0x7f52c006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.965+0000 7f52dcff9700 1 -- 172.21.15.1:0/754678629 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f52c808ccf0 con 0x7f52e00ae600 2024-03-18T19:09:26.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.965+0000 7f52df7fe700 1 --2- 172.21.15.1:0/754678629 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f52c006adb0 0x7f52c006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:26.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.965+0000 7f52df7fe700 1 --2- 172.21.15.1:0/754678629 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f52c006adb0 0x7f52c006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f52d0006fd0 tx=0x7f52d0008040 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:26.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:26.969+0000 7f52dcff9700 1 -- 172.21.15.1:0/754678629 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f52c8091070 con 0x7f52e00ae600 2024-03-18T19:09:27.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.168+0000 7f52e65f2700 1 -- 172.21.15.1:0/754678629 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f52e006e780 con 0x7f52c006adb0 2024-03-18T19:09:27.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.176+0000 7f52dcff9700 1 -- 172.21.15.1:0/754678629 <== mgr.8565 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289862 (secure 0 0 0) 0x7f52d00123e0 con 0x7f52c006adb0 2024-03-18T19:09:27.175 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:27.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.179+0000 7f52e65f2700 1 -- 172.21.15.1:0/754678629 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f52c006adb0 msgr2=0x7f52c006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.179+0000 7f52e65f2700 1 --2- 172.21.15.1:0/754678629 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f52c006adb0 0x7f52c006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f52d0006fd0 tx=0x7f52d0008040 comp rx=0 tx=0).stop 2024-03-18T19:09:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.179+0000 7f52e65f2700 1 -- 172.21.15.1:0/754678629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52e00ae600 msgr2=0x7f52e01ef070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.179+0000 7f52e65f2700 1 --2- 172.21.15.1:0/754678629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52e00ae600 0x7f52e01ef070 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f52c8000c00 tx=0x7f52c8004750 comp rx=0 tx=0).stop 2024-03-18T19:09:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.179+0000 7f52e65f2700 1 -- 172.21.15.1:0/754678629 shutdown_connections 2024-03-18T19:09:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.179+0000 7f52e65f2700 1 --2- 172.21.15.1:0/754678629 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f52c006adb0 0x7f52c006d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.179+0000 7f52e65f2700 1 --2- 172.21.15.1:0/754678629 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f52e015a2e0 0x7f52e015d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.179+0000 7f52e65f2700 1 --2- 172.21.15.1:0/754678629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52e00ae600 0x7f52e01ef070 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.179+0000 7f52e65f2700 1 --2- 172.21.15.1:0/754678629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52e00cc170 0x7f52e0148a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.179+0000 7f52e65f2700 1 -- 172.21.15.1:0/754678629 >> 172.21.15.1:0/754678629 conn(0x7f52e006b860 msgr2=0x7f52e006ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.180+0000 7f52e65f2700 1 -- 172.21.15.1:0/754678629 shutdown_connections 2024-03-18T19:09:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.180+0000 7f52e65f2700 1 -- 172.21.15.1:0/754678629 wait complete. 2024-03-18T19:09:27.178 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T19:09:27.188 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-18T19:09:27.106271+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":403,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":265,"ondisk_log_size":265,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":347220,"kb_used_data":15424,"kb_used_omap":12,"kb_used_meta":331635,"kb_avail":754627500,"statfs":{"total":773094113280,"available":772738560000,"internally_reserved":0,"allocated":15794176,"data_stored":8505564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":339594809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.915360"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.643541+0000","last_change":"2024-03-18T19:09:22.641031+0000","last_active":"2024-03-18T19:09:22.643541+0000","last_peered":"2024-03-18T19:09:22.643541+0000","last_clean":"2024-03-18T19:09:22.643541+0000","last_became_active":"2024-03-18T19:09:22.640849+0000","last_became_peered":"2024-03-18T19:09:22.640849+0000","last_unstale":"2024-03-18T19:09:22.643541+0000","last_undegraded":"2024-03-18T19:09:22.643541+0000","last_fullsized":"2024-03-18T19:09:22.643541+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:09.583831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.650652+0000","last_change":"2024-03-18T19:09:22.649460+0000","last_active":"2024-03-18T19:09:22.650652+0000","last_peered":"2024-03-18T19:09:22.650652+0000","last_clean":"2024-03-18T19:09:22.650652+0000","last_became_active":"2024-03-18T19:09:22.648770+0000","last_became_peered":"2024-03-18T19:09:22.648770+0000","last_unstale":"2024-03-18T19:09:22.650652+0000","last_undegraded":"2024-03-18T19:09:22.650652+0000","last_fullsized":"2024-03-18T19:09:22.650652+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:13:28.480818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.651238+0000","last_change":"2024-03-18T19:09:22.651017+0000","last_active":"2024-03-18T19:09:22.651238+0000","last_peered":"2024-03-18T19:09:22.651238+0000","last_clean":"2024-03-18T19:09:22.651238+0000","last_became_active":"2024-03-18T19:09:22.650823+0000","last_became_peered":"2024-03-18T19:09:22.650823+0000","last_unstale":"2024-03-18T19:09:22.651238+0000","last_undegraded":"2024-03-18T19:09:22.651238+0000","last_fullsized":"2024-03-18T19:09:22.651238+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:20:20.852061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327783+0000","last_change":"2024-03-18T19:09:22.646262+0000","last_active":"2024-03-18T19:09:24.327783+0000","last_peered":"2024-03-18T19:09:24.327783+0000","last_clean":"2024-03-18T19:09:24.327783+0000","last_became_active":"2024-03-18T19:09:22.645926+0000","last_became_peered":"2024-03-18T19:09:22.645926+0000","last_unstale":"2024-03-18T19:09:24.327783+0000","last_undegraded":"2024-03-18T19:09:24.327783+0000","last_fullsized":"2024-03-18T19:09:24.327783+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:44.544055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.651043+0000","last_change":"2024-03-18T19:09:22.644837+0000","last_active":"2024-03-18T19:09:22.651043+0000","last_peered":"2024-03-18T19:09:22.651043+0000","last_clean":"2024-03-18T19:09:22.651043+0000","last_became_active":"2024-03-18T19:09:22.644093+0000","last_became_peered":"2024-03-18T19:09:22.644093+0000","last_unstale":"2024-03-18T19:09:22.651043+0000","last_undegraded":"2024-03-18T19:09:22.651043+0000","last_fullsized":"2024-03-18T19:09:22.651043+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:06:04.875225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.543942+0000","last_change":"2024-03-18T19:09:22.646388+0000","last_active":"2024-03-18T19:09:24.543942+0000","last_peered":"2024-03-18T19:09:24.543942+0000","last_clean":"2024-03-18T19:09:24.543942+0000","last_became_active":"2024-03-18T19:09:22.646099+0000","last_became_peered":"2024-03-18T19:09:22.646099+0000","last_unstale":"2024-03-18T19:09:24.543942+0000","last_undegraded":"2024-03-18T19:09:24.543942+0000","last_fullsized":"2024-03-18T19:09:24.543942+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:14.500045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.543861+0000","last_change":"2024-03-18T19:09:22.635542+0000","last_active":"2024-03-18T19:09:24.543861+0000","last_peered":"2024-03-18T19:09:24.543861+0000","last_clean":"2024-03-18T19:09:24.543861+0000","last_became_active":"2024-03-18T19:09:22.635216+0000","last_became_peered":"2024-03-18T19:09:22.635216+0000","last_unstale":"2024-03-18T19:09:24.543861+0000","last_undegraded":"2024-03-18T19:09:24.543861+0000","last_fullsized":"2024-03-18T19:09:24.543861+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:07:57.607914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.651585+0000","last_change":"2024-03-18T19:09:22.651211+0000","last_active":"2024-03-18T19:09:22.651585+0000","last_peered":"2024-03-18T19:09:22.651585+0000","last_clean":"2024-03-18T19:09:22.651585+0000","last_became_active":"2024-03-18T19:09:22.650920+0000","last_became_peered":"2024-03-18T19:09:22.650920+0000","last_unstale":"2024-03-18T19:09:22.651585+0000","last_undegraded":"2024-03-18T19:09:22.651585+0000","last_fullsized":"2024-03-18T19:09:22.651585+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:07:46.533619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.328030+0000","last_change":"2024-03-18T19:09:22.649568+0000","last_active":"2024-03-18T19:09:24.328030+0000","last_peered":"2024-03-18T19:09:24.328030+0000","last_clean":"2024-03-18T19:09:24.328030+0000","last_became_active":"2024-03-18T19:09:22.648829+0000","last_became_peered":"2024-03-18T19:09:22.648829+0000","last_unstale":"2024-03-18T19:09:24.328030+0000","last_undegraded":"2024-03-18T19:09:24.328030+0000","last_fullsized":"2024-03-18T19:09:24.328030+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:55:10.360362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.640141+0000","last_change":"2024-03-18T19:09:22.637079+0000","last_active":"2024-03-18T19:09:22.640141+0000","last_peered":"2024-03-18T19:09:22.640141+0000","last_clean":"2024-03-18T19:09:22.640141+0000","last_became_active":"2024-03-18T19:09:22.636411+0000","last_became_peered":"2024-03-18T19:09:22.636411+0000","last_unstale":"2024-03-18T19:09:22.640141+0000","last_undegraded":"2024-03-18T19:09:22.640141+0000","last_fullsized":"2024-03-18T19:09:22.640141+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:57:27.805513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.656977+0000","last_change":"2024-03-18T19:09:22.656194+0000","last_active":"2024-03-18T19:09:22.656977+0000","last_peered":"2024-03-18T19:09:22.656977+0000","last_clean":"2024-03-18T19:09:22.656977+0000","last_became_active":"2024-03-18T19:09:22.655886+0000","last_became_peered":"2024-03-18T19:09:22.655886+0000","last_unstale":"2024-03-18T19:09:22.656977+0000","last_undegraded":"2024-03-18T19:09:22.656977+0000","last_fullsized":"2024-03-18T19:09:22.656977+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:17.416505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.544400+0000","last_change":"2024-03-18T19:09:22.643996+0000","last_active":"2024-03-18T19:09:24.544400+0000","last_peered":"2024-03-18T19:09:24.544400+0000","last_clean":"2024-03-18T19:09:24.544400+0000","last_became_active":"2024-03-18T19:09:22.643620+0000","last_became_peered":"2024-03-18T19:09:22.643620+0000","last_unstale":"2024-03-18T19:09:24.544400+0000","last_undegraded":"2024-03-18T19:09:24.544400+0000","last_fullsized":"2024-03-18T19:09:24.544400+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:29:26.326903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.658061+0000","last_change":"2024-03-18T19:09:22.654312+0000","last_active":"2024-03-18T19:09:22.658061+0000","last_peered":"2024-03-18T19:09:22.658061+0000","last_clean":"2024-03-18T19:09:22.658061+0000","last_became_active":"2024-03-18T19:09:22.654071+0000","last_became_peered":"2024-03-18T19:09:22.654071+0000","last_unstale":"2024-03-18T19:09:22.658061+0000","last_undegraded":"2024-03-18T19:09:22.658061+0000","last_fullsized":"2024-03-18T19:09:22.658061+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:44.544055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.649403+0000","last_change":"2024-03-18T19:09:22.646993+0000","last_active":"2024-03-18T19:09:22.649403+0000","last_peered":"2024-03-18T19:09:22.649403+0000","last_clean":"2024-03-18T19:09:22.649403+0000","last_became_active":"2024-03-18T19:09:22.646684+0000","last_became_peered":"2024-03-18T19:09:22.646684+0000","last_unstale":"2024-03-18T19:09:22.649403+0000","last_undegraded":"2024-03-18T19:09:22.649403+0000","last_fullsized":"2024-03-18T19:09:22.649403+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:47:24.499751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.656990+0000","last_change":"2024-03-18T19:09:22.656195+0000","last_active":"2024-03-18T19:09:22.656990+0000","last_peered":"2024-03-18T19:09:22.656990+0000","last_clean":"2024-03-18T19:09:22.656990+0000","last_became_active":"2024-03-18T19:09:22.655942+0000","last_became_peered":"2024-03-18T19:09:22.655942+0000","last_unstale":"2024-03-18T19:09:22.656990+0000","last_undegraded":"2024-03-18T19:09:22.656990+0000","last_fullsized":"2024-03-18T19:09:22.656990+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:36:25.838234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.543808+0000","last_change":"2024-03-18T19:09:22.642908+0000","last_active":"2024-03-18T19:09:24.543808+0000","last_peered":"2024-03-18T19:09:24.543808+0000","last_clean":"2024-03-18T19:09:24.543808+0000","last_became_active":"2024-03-18T19:09:22.642438+0000","last_became_peered":"2024-03-18T19:09:22.642438+0000","last_unstale":"2024-03-18T19:09:24.543808+0000","last_undegraded":"2024-03-18T19:09:24.543808+0000","last_fullsized":"2024-03-18T19:09:24.543808+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:07:29.956879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.658137+0000","last_change":"2024-03-18T19:09:22.655395+0000","last_active":"2024-03-18T19:09:22.658137+0000","last_peered":"2024-03-18T19:09:22.658137+0000","last_clean":"2024-03-18T19:09:22.658137+0000","last_became_active":"2024-03-18T19:09:22.655157+0000","last_became_peered":"2024-03-18T19:09:22.655157+0000","last_unstale":"2024-03-18T19:09:22.658137+0000","last_undegraded":"2024-03-18T19:09:22.658137+0000","last_fullsized":"2024-03-18T19:09:22.658137+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:55:10.360362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.649517+0000","last_change":"2024-03-18T19:09:22.646963+0000","last_active":"2024-03-18T19:09:22.649517+0000","last_peered":"2024-03-18T19:09:22.649517+0000","last_clean":"2024-03-18T19:09:22.649517+0000","last_became_active":"2024-03-18T19:09:22.646274+0000","last_became_peered":"2024-03-18T19:09:22.646274+0000","last_unstale":"2024-03-18T19:09:22.649517+0000","last_undegraded":"2024-03-18T19:09:22.649517+0000","last_fullsized":"2024-03-18T19:09:22.649517+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:04:29.218281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.633637+0000","last_change":"2024-03-18T19:09:22.643401+0000","last_active":"2024-03-18T19:09:23.633637+0000","last_peered":"2024-03-18T19:09:23.633637+0000","last_clean":"2024-03-18T19:09:23.633637+0000","last_became_active":"2024-03-18T19:09:22.642990+0000","last_became_peered":"2024-03-18T19:09:22.642990+0000","last_unstale":"2024-03-18T19:09:23.633637+0000","last_undegraded":"2024-03-18T19:09:23.633637+0000","last_fullsized":"2024-03-18T19:09:23.633637+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:30.062865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.644965+0000","last_change":"2024-03-18T19:09:22.642770+0000","last_active":"2024-03-18T19:09:22.644965+0000","last_peered":"2024-03-18T19:09:22.644965+0000","last_clean":"2024-03-18T19:09:22.644965+0000","last_became_active":"2024-03-18T19:09:22.642497+0000","last_became_peered":"2024-03-18T19:09:22.642497+0000","last_unstale":"2024-03-18T19:09:22.644965+0000","last_undegraded":"2024-03-18T19:09:22.644965+0000","last_fullsized":"2024-03-18T19:09:22.644965+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:25:41.879440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327766+0000","last_change":"2024-03-18T19:09:22.640110+0000","last_active":"2024-03-18T19:09:24.327766+0000","last_peered":"2024-03-18T19:09:24.327766+0000","last_clean":"2024-03-18T19:09:24.327766+0000","last_became_active":"2024-03-18T19:09:22.639807+0000","last_became_peered":"2024-03-18T19:09:22.639807+0000","last_unstale":"2024-03-18T19:09:24.327766+0000","last_undegraded":"2024-03-18T19:09:24.327766+0000","last_fullsized":"2024-03-18T19:09:24.327766+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:19.098002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.651010+0000","last_change":"2024-03-18T19:09:22.648054+0000","last_active":"2024-03-18T19:09:22.651010+0000","last_peered":"2024-03-18T19:09:22.651010+0000","last_clean":"2024-03-18T19:09:22.651010+0000","last_became_active":"2024-03-18T19:09:22.647796+0000","last_became_peered":"2024-03-18T19:09:22.647796+0000","last_unstale":"2024-03-18T19:09:22.651010+0000","last_undegraded":"2024-03-18T19:09:22.651010+0000","last_fullsized":"2024-03-18T19:09:22.651010+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:11:49.496234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327631+0000","last_change":"2024-03-18T19:09:22.650369+0000","last_active":"2024-03-18T19:09:24.327631+0000","last_peered":"2024-03-18T19:09:24.327631+0000","last_clean":"2024-03-18T19:09:24.327631+0000","last_became_active":"2024-03-18T19:09:22.650096+0000","last_became_peered":"2024-03-18T19:09:22.650096+0000","last_unstale":"2024-03-18T19:09:24.327631+0000","last_undegraded":"2024-03-18T19:09:24.327631+0000","last_fullsized":"2024-03-18T19:09:24.327631+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:35.575921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.634045+0000","last_change":"2024-03-18T19:09:22.657507+0000","last_active":"2024-03-18T19:09:23.634045+0000","last_peered":"2024-03-18T19:09:23.634045+0000","last_clean":"2024-03-18T19:09:23.634045+0000","last_became_active":"2024-03-18T19:09:22.657223+0000","last_became_peered":"2024-03-18T19:09:22.657223+0000","last_unstale":"2024-03-18T19:09:23.634045+0000","last_undegraded":"2024-03-18T19:09:23.634045+0000","last_fullsized":"2024-03-18T19:09:23.634045+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:19.098002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.635192+0000","last_change":"2024-03-18T19:09:22.654820+0000","last_active":"2024-03-18T19:09:23.635192+0000","last_peered":"2024-03-18T19:09:23.635192+0000","last_clean":"2024-03-18T19:09:23.635192+0000","last_became_active":"2024-03-18T19:09:22.654587+0000","last_became_peered":"2024-03-18T19:09:22.654587+0000","last_unstale":"2024-03-18T19:09:23.635192+0000","last_undegraded":"2024-03-18T19:09:23.635192+0000","last_fullsized":"2024-03-18T19:09:23.635192+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:35.575921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.644993+0000","last_change":"2024-03-18T19:09:22.642852+0000","last_active":"2024-03-18T19:09:22.644993+0000","last_peered":"2024-03-18T19:09:22.644993+0000","last_clean":"2024-03-18T19:09:22.644993+0000","last_became_active":"2024-03-18T19:09:22.642577+0000","last_became_peered":"2024-03-18T19:09:22.642577+0000","last_unstale":"2024-03-18T19:09:22.644993+0000","last_undegraded":"2024-03-18T19:09:22.644993+0000","last_fullsized":"2024-03-18T19:09:22.644993+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:11.209178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.633727+0000","last_change":"2024-03-18T19:09:22.643822+0000","last_active":"2024-03-18T19:09:23.633727+0000","last_peered":"2024-03-18T19:09:23.633727+0000","last_clean":"2024-03-18T19:09:23.633727+0000","last_became_active":"2024-03-18T19:09:22.642602+0000","last_became_peered":"2024-03-18T19:09:22.642602+0000","last_unstale":"2024-03-18T19:09:23.633727+0000","last_undegraded":"2024-03-18T19:09:23.633727+0000","last_fullsized":"2024-03-18T19:09:23.633727+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:32.074373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.544279+0000","last_change":"2024-03-18T19:09:22.653778+0000","last_active":"2024-03-18T19:09:24.544279+0000","last_peered":"2024-03-18T19:09:24.544279+0000","last_clean":"2024-03-18T19:09:24.544279+0000","last_became_active":"2024-03-18T19:09:22.653463+0000","last_became_peered":"2024-03-18T19:09:22.653463+0000","last_unstale":"2024-03-18T19:09:24.544279+0000","last_undegraded":"2024-03-18T19:09:24.544279+0000","last_fullsized":"2024-03-18T19:09:24.544279+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:41.406056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.649253+0000","last_change":"2024-03-18T19:09:22.648927+0000","last_active":"2024-03-18T19:09:22.649253+0000","last_peered":"2024-03-18T19:09:22.649253+0000","last_clean":"2024-03-18T19:09:22.649253+0000","last_became_active":"2024-03-18T19:09:22.648652+0000","last_became_peered":"2024-03-18T19:09:22.648652+0000","last_unstale":"2024-03-18T19:09:22.649253+0000","last_undegraded":"2024-03-18T19:09:22.649253+0000","last_fullsized":"2024-03-18T19:09:22.649253+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:02:45.260371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.649280+0000","last_change":"2024-03-18T19:09:22.645632+0000","last_active":"2024-03-18T19:09:22.649280+0000","last_peered":"2024-03-18T19:09:22.649280+0000","last_clean":"2024-03-18T19:09:22.649280+0000","last_became_active":"2024-03-18T19:09:22.644855+0000","last_became_peered":"2024-03-18T19:09:22.644855+0000","last_unstale":"2024-03-18T19:09:22.649280+0000","last_undegraded":"2024-03-18T19:09:22.649280+0000","last_fullsized":"2024-03-18T19:09:22.649280+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:28:35.293994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.634347+0000","last_change":"2024-03-18T19:09:22.655231+0000","last_active":"2024-03-18T19:09:23.634347+0000","last_peered":"2024-03-18T19:09:23.634347+0000","last_clean":"2024-03-18T19:09:23.634347+0000","last_became_active":"2024-03-18T19:09:22.654032+0000","last_became_peered":"2024-03-18T19:09:22.654032+0000","last_unstale":"2024-03-18T19:09:23.634347+0000","last_undegraded":"2024-03-18T19:09:23.634347+0000","last_fullsized":"2024-03-18T19:09:23.634347+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:44.544055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.634142+0000","last_change":"2024-03-18T19:09:22.654727+0000","last_active":"2024-03-18T19:09:23.634142+0000","last_peered":"2024-03-18T19:09:23.634142+0000","last_clean":"2024-03-18T19:09:23.634142+0000","last_became_active":"2024-03-18T19:09:22.654324+0000","last_became_peered":"2024-03-18T19:09:22.654324+0000","last_unstale":"2024-03-18T19:09:23.634142+0000","last_undegraded":"2024-03-18T19:09:23.634142+0000","last_fullsized":"2024-03-18T19:09:23.634142+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:43:09.563639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.640249+0000","last_change":"2024-03-18T19:09:22.637640+0000","last_active":"2024-03-18T19:09:22.640249+0000","last_peered":"2024-03-18T19:09:22.640249+0000","last_clean":"2024-03-18T19:09:22.640249+0000","last_became_active":"2024-03-18T19:09:22.637385+0000","last_became_peered":"2024-03-18T19:09:22.637385+0000","last_unstale":"2024-03-18T19:09:22.640249+0000","last_undegraded":"2024-03-18T19:09:22.640249+0000","last_fullsized":"2024-03-18T19:09:22.640249+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:48:33.430169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"146'4","reported_seq":15,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.633920+0000","last_change":"2024-03-18T19:09:22.213624+0000","last_active":"2024-03-18T19:09:23.633920+0000","last_peered":"2024-03-18T19:09:23.633920+0000","last_clean":"2024-03-18T19:09:23.633920+0000","last_became_active":"2024-03-18T19:09:22.213358+0000","last_became_peered":"2024-03-18T19:09:22.213358+0000","last_unstale":"2024-03-18T19:09:23.633920+0000","last_undegraded":"2024-03-18T19:09:23.633920+0000","last_fullsized":"2024-03-18T19:09:23.633920+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:23:13.662431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.544222+0000","last_change":"2024-03-18T19:09:22.642416+0000","last_active":"2024-03-18T19:09:24.544222+0000","last_peered":"2024-03-18T19:09:24.544222+0000","last_clean":"2024-03-18T19:09:24.544222+0000","last_became_active":"2024-03-18T19:09:22.641728+0000","last_became_peered":"2024-03-18T19:09:22.641728+0000","last_unstale":"2024-03-18T19:09:24.544222+0000","last_undegraded":"2024-03-18T19:09:24.544222+0000","last_fullsized":"2024-03-18T19:09:24.544222+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:00:35.089597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"146'1","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.641409+0000","last_change":"2024-03-18T19:09:22.210909+0000","last_active":"2024-03-18T19:09:22.641409+0000","last_peered":"2024-03-18T19:09:22.641409+0000","last_clean":"2024-03-18T19:09:22.641409+0000","last_became_active":"2024-03-18T19:09:22.210359+0000","last_became_peered":"2024-03-18T19:09:22.210359+0000","last_unstale":"2024-03-18T19:09:22.641409+0000","last_undegraded":"2024-03-18T19:09:22.641409+0000","last_fullsized":"2024-03-18T19:09:22.641409+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:23:44.469348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.634135+0000","last_change":"2024-03-18T19:09:22.653159+0000","last_active":"2024-03-18T19:09:23.634135+0000","last_peered":"2024-03-18T19:09:23.634135+0000","last_clean":"2024-03-18T19:09:23.634135+0000","last_became_active":"2024-03-18T19:09:22.652895+0000","last_became_peered":"2024-03-18T19:09:22.652895+0000","last_unstale":"2024-03-18T19:09:23.634135+0000","last_undegraded":"2024-03-18T19:09:23.634135+0000","last_fullsized":"2024-03-18T19:09:23.634135+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:48:33.430169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"146'2","reported_seq":13,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327963+0000","last_change":"2024-03-18T19:09:22.210238+0000","last_active":"2024-03-18T19:09:24.327963+0000","last_peered":"2024-03-18T19:09:24.327963+0000","last_clean":"2024-03-18T19:09:24.327963+0000","last_became_active":"2024-03-18T19:09:22.207542+0000","last_became_peered":"2024-03-18T19:09:22.207542+0000","last_unstale":"2024-03-18T19:09:24.327963+0000","last_undegraded":"2024-03-18T19:09:24.327963+0000","last_fullsized":"2024-03-18T19:09:24.327963+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:23:44.469348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.328091+0000","last_change":"2024-03-18T19:09:22.644465+0000","last_active":"2024-03-18T19:09:24.328091+0000","last_peered":"2024-03-18T19:09:24.328091+0000","last_clean":"2024-03-18T19:09:24.328091+0000","last_became_active":"2024-03-18T19:09:22.644008+0000","last_became_peered":"2024-03-18T19:09:22.644008+0000","last_unstale":"2024-03-18T19:09:24.328091+0000","last_undegraded":"2024-03-18T19:09:24.328091+0000","last_fullsized":"2024-03-18T19:09:24.328091+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:30.062865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"146'2","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.648630+0000","last_change":"2024-03-18T19:09:22.212515+0000","last_active":"2024-03-18T19:09:22.648630+0000","last_peered":"2024-03-18T19:09:22.648630+0000","last_clean":"2024-03-18T19:09:22.648630+0000","last_became_active":"2024-03-18T19:09:22.212230+0000","last_became_peered":"2024-03-18T19:09:22.212230+0000","last_unstale":"2024-03-18T19:09:22.648630+0000","last_undegraded":"2024-03-18T19:09:22.648630+0000","last_fullsized":"2024-03-18T19:09:22.648630+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:23:13.662431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.636090+0000","last_change":"2024-03-18T19:09:22.634750+0000","last_active":"2024-03-18T19:09:22.636090+0000","last_peered":"2024-03-18T19:09:22.636090+0000","last_clean":"2024-03-18T19:09:22.636090+0000","last_became_active":"2024-03-18T19:09:22.634468+0000","last_became_peered":"2024-03-18T19:09:22.634468+0000","last_unstale":"2024-03-18T19:09:22.636090+0000","last_undegraded":"2024-03-18T19:09:22.636090+0000","last_fullsized":"2024-03-18T19:09:22.636090+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:32.074373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"146'1","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.643954+0000","last_change":"2024-03-18T19:09:22.216606+0000","last_active":"2024-03-18T19:09:22.643954+0000","last_peered":"2024-03-18T19:09:22.643954+0000","last_clean":"2024-03-18T19:09:22.643954+0000","last_became_active":"2024-03-18T19:09:22.216273+0000","last_became_peered":"2024-03-18T19:09:22.216273+0000","last_unstale":"2024-03-18T19:09:22.643954+0000","last_undegraded":"2024-03-18T19:09:22.643954+0000","last_fullsized":"2024-03-18T19:09:22.643954+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:31.069246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.643533+0000","last_change":"2024-03-18T19:09:22.640753+0000","last_active":"2024-03-18T19:09:22.643533+0000","last_peered":"2024-03-18T19:09:22.643533+0000","last_clean":"2024-03-18T19:09:22.643533+0000","last_became_active":"2024-03-18T19:09:22.640464+0000","last_became_peered":"2024-03-18T19:09:22.640464+0000","last_unstale":"2024-03-18T19:09:22.643533+0000","last_undegraded":"2024-03-18T19:09:22.643533+0000","last_fullsized":"2024-03-18T19:09:22.643533+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:59:06.548540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"146'1","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.544043+0000","last_change":"2024-03-18T19:09:22.212425+0000","last_active":"2024-03-18T19:09:24.544043+0000","last_peered":"2024-03-18T19:09:24.544043+0000","last_clean":"2024-03-18T19:09:24.544043+0000","last_became_active":"2024-03-18T19:09:22.212115+0000","last_became_peered":"2024-03-18T19:09:22.212115+0000","last_unstale":"2024-03-18T19:09:24.544043+0000","last_undegraded":"2024-03-18T19:09:24.544043+0000","last_fullsized":"2024-03-18T19:09:24.544043+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:45:16.280219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.648192+0000","last_change":"2024-03-18T19:09:22.640975+0000","last_active":"2024-03-18T19:09:22.648192+0000","last_peered":"2024-03-18T19:09:22.648192+0000","last_clean":"2024-03-18T19:09:22.648192+0000","last_became_active":"2024-03-18T19:09:22.638298+0000","last_became_peered":"2024-03-18T19:09:22.638298+0000","last_unstale":"2024-03-18T19:09:22.648192+0000","last_undegraded":"2024-03-18T19:09:22.648192+0000","last_fullsized":"2024-03-18T19:09:22.648192+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:42:47.124196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.634438+0000","last_change":"2024-03-18T19:09:22.207973+0000","last_active":"2024-03-18T19:09:23.634438+0000","last_peered":"2024-03-18T19:09:23.634438+0000","last_clean":"2024-03-18T19:09:23.634438+0000","last_became_active":"2024-03-18T19:09:22.207627+0000","last_became_peered":"2024-03-18T19:09:22.207627+0000","last_unstale":"2024-03-18T19:09:23.634438+0000","last_undegraded":"2024-03-18T19:09:23.634438+0000","last_fullsized":"2024-03-18T19:09:23.634438+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:23:44.469348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.641350+0000","last_change":"2024-03-18T19:09:22.639238+0000","last_active":"2024-03-18T19:09:22.641350+0000","last_peered":"2024-03-18T19:09:22.641350+0000","last_clean":"2024-03-18T19:09:22.641350+0000","last_became_active":"2024-03-18T19:09:22.639052+0000","last_became_peered":"2024-03-18T19:09:22.639052+0000","last_unstale":"2024-03-18T19:09:22.641350+0000","last_undegraded":"2024-03-18T19:09:22.641350+0000","last_fullsized":"2024-03-18T19:09:22.641350+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:08.116636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.544039+0000","last_change":"2024-03-18T19:09:22.216954+0000","last_active":"2024-03-18T19:09:24.544039+0000","last_peered":"2024-03-18T19:09:24.544039+0000","last_clean":"2024-03-18T19:09:24.544039+0000","last_became_active":"2024-03-18T19:09:22.216518+0000","last_became_peered":"2024-03-18T19:09:22.216518+0000","last_unstale":"2024-03-18T19:09:24.544039+0000","last_undegraded":"2024-03-18T19:09:24.544039+0000","last_fullsized":"2024-03-18T19:09:24.544039+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:17:26.526424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.633971+0000","last_change":"2024-03-18T19:09:22.644752+0000","last_active":"2024-03-18T19:09:23.633971+0000","last_peered":"2024-03-18T19:09:23.633971+0000","last_clean":"2024-03-18T19:09:23.633971+0000","last_became_active":"2024-03-18T19:09:22.644452+0000","last_became_peered":"2024-03-18T19:09:22.644452+0000","last_unstale":"2024-03-18T19:09:23.633971+0000","last_undegraded":"2024-03-18T19:09:23.633971+0000","last_fullsized":"2024-03-18T19:09:23.633971+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:50:11.074283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.634003+0000","last_change":"2024-03-18T19:09:22.210552+0000","last_active":"2024-03-18T19:09:23.634003+0000","last_peered":"2024-03-18T19:09:23.634003+0000","last_clean":"2024-03-18T19:09:23.634003+0000","last_became_active":"2024-03-18T19:09:22.210152+0000","last_became_peered":"2024-03-18T19:09:22.210152+0000","last_unstale":"2024-03-18T19:09:23.634003+0000","last_undegraded":"2024-03-18T19:09:23.634003+0000","last_fullsized":"2024-03-18T19:09:23.634003+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:42:54.314098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.635768+0000","last_change":"2024-03-18T19:09:22.653068+0000","last_active":"2024-03-18T19:09:23.635768+0000","last_peered":"2024-03-18T19:09:23.635768+0000","last_clean":"2024-03-18T19:09:23.635768+0000","last_became_active":"2024-03-18T19:09:22.652743+0000","last_became_peered":"2024-03-18T19:09:22.652743+0000","last_unstale":"2024-03-18T19:09:23.635768+0000","last_undegraded":"2024-03-18T19:09:23.635768+0000","last_fullsized":"2024-03-18T19:09:23.635768+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:55:10.360362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.634241+0000","last_change":"2024-03-18T19:09:22.211243+0000","last_active":"2024-03-18T19:09:23.634241+0000","last_peered":"2024-03-18T19:09:23.634241+0000","last_clean":"2024-03-18T19:09:23.634241+0000","last_became_active":"2024-03-18T19:09:22.210793+0000","last_became_peered":"2024-03-18T19:09:22.210793+0000","last_unstale":"2024-03-18T19:09:23.634241+0000","last_undegraded":"2024-03-18T19:09:23.634241+0000","last_fullsized":"2024-03-18T19:09:23.634241+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:49:48.268109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.649528+0000","last_change":"2024-03-18T19:09:22.646996+0000","last_active":"2024-03-18T19:09:22.649528+0000","last_peered":"2024-03-18T19:09:22.649528+0000","last_clean":"2024-03-18T19:09:22.649528+0000","last_became_active":"2024-03-18T19:09:22.646627+0000","last_became_peered":"2024-03-18T19:09:22.646627+0000","last_unstale":"2024-03-18T19:09:22.649528+0000","last_undegraded":"2024-03-18T19:09:22.649528+0000","last_fullsized":"2024-03-18T19:09:22.649528+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:58:33.254876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"146'3","reported_seq":14,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.635942+0000","last_change":"2024-03-18T19:09:22.213592+0000","last_active":"2024-03-18T19:09:23.635942+0000","last_peered":"2024-03-18T19:09:23.635942+0000","last_clean":"2024-03-18T19:09:23.635942+0000","last_became_active":"2024-03-18T19:09:22.213290+0000","last_became_peered":"2024-03-18T19:09:22.213290+0000","last_unstale":"2024-03-18T19:09:23.635942+0000","last_undegraded":"2024-03-18T19:09:23.635942+0000","last_fullsized":"2024-03-18T19:09:23.635942+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:23:13.662431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.635278+0000","last_change":"2024-03-18T19:09:22.654387+0000","last_active":"2024-03-18T19:09:23.635278+0000","last_peered":"2024-03-18T19:09:23.635278+0000","last_clean":"2024-03-18T19:09:23.635278+0000","last_became_active":"2024-03-18T19:09:22.653641+0000","last_became_peered":"2024-03-18T19:09:22.653641+0000","last_unstale":"2024-03-18T19:09:23.635278+0000","last_undegraded":"2024-03-18T19:09:23.635278+0000","last_fullsized":"2024-03-18T19:09:23.635278+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:32:54.149288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"146'6","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.657735+0000","last_change":"2024-03-18T19:09:22.209065+0000","last_active":"2024-03-18T19:09:22.657735+0000","last_peered":"2024-03-18T19:09:22.657735+0000","last_clean":"2024-03-18T19:09:22.657735+0000","last_became_active":"2024-03-18T19:09:22.208780+0000","last_became_peered":"2024-03-18T19:09:22.208780+0000","last_unstale":"2024-03-18T19:09:22.657735+0000","last_undegraded":"2024-03-18T19:09:22.657735+0000","last_fullsized":"2024-03-18T19:09:22.657735+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:48:23.276451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.633437+0000","last_change":"2024-03-18T19:09:22.644783+0000","last_active":"2024-03-18T19:09:23.633437+0000","last_peered":"2024-03-18T19:09:23.633437+0000","last_clean":"2024-03-18T19:09:23.633437+0000","last_became_active":"2024-03-18T19:09:22.644575+0000","last_became_peered":"2024-03-18T19:09:22.644575+0000","last_unstale":"2024-03-18T19:09:23.633437+0000","last_undegraded":"2024-03-18T19:09:23.633437+0000","last_fullsized":"2024-03-18T19:09:23.633437+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:51.984733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"146'2","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.652517+0000","last_change":"2024-03-18T19:09:22.206823+0000","last_active":"2024-03-18T19:09:22.652517+0000","last_peered":"2024-03-18T19:09:22.652517+0000","last_clean":"2024-03-18T19:09:22.652517+0000","last_became_active":"2024-03-18T19:09:22.206396+0000","last_became_peered":"2024-03-18T19:09:22.206396+0000","last_unstale":"2024-03-18T19:09:22.652517+0000","last_undegraded":"2024-03-18T19:09:22.652517+0000","last_fullsized":"2024-03-18T19:09:22.652517+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:59:05.543413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.641134+0000","last_change":"2024-03-18T19:09:22.637886+0000","last_active":"2024-03-18T19:09:22.641134+0000","last_peered":"2024-03-18T19:09:22.641134+0000","last_clean":"2024-03-18T19:09:22.641134+0000","last_became_active":"2024-03-18T19:09:22.637568+0000","last_became_peered":"2024-03-18T19:09:22.637568+0000","last_unstale":"2024-03-18T19:09:22.641134+0000","last_undegraded":"2024-03-18T19:09:22.641134+0000","last_fullsized":"2024-03-18T19:09:22.641134+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:03:35.575921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.637953+0000","last_change":"2024-03-18T19:09:22.209622+0000","last_active":"2024-03-18T19:09:22.637953+0000","last_peered":"2024-03-18T19:09:22.637953+0000","last_clean":"2024-03-18T19:09:22.637953+0000","last_became_active":"2024-03-18T19:09:22.206732+0000","last_became_peered":"2024-03-18T19:09:22.206732+0000","last_unstale":"2024-03-18T19:09:22.637953+0000","last_undegraded":"2024-03-18T19:09:22.637953+0000","last_fullsized":"2024-03-18T19:09:22.637953+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:21:43.532626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.645250+0000","last_change":"2024-03-18T19:09:22.643878+0000","last_active":"2024-03-18T19:09:22.645250+0000","last_peered":"2024-03-18T19:09:22.645250+0000","last_clean":"2024-03-18T19:09:22.645250+0000","last_became_active":"2024-03-18T19:09:22.643632+0000","last_became_peered":"2024-03-18T19:09:22.643632+0000","last_unstale":"2024-03-18T19:09:22.645250+0000","last_undegraded":"2024-03-18T19:09:22.645250+0000","last_fullsized":"2024-03-18T19:09:22.645250+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:49:31.795796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.637797+0000","last_change":"2024-03-18T19:09:22.210688+0000","last_active":"2024-03-18T19:09:22.637797+0000","last_peered":"2024-03-18T19:09:22.637797+0000","last_clean":"2024-03-18T19:09:22.637797+0000","last_became_active":"2024-03-18T19:09:22.210334+0000","last_became_peered":"2024-03-18T19:09:22.210334+0000","last_unstale":"2024-03-18T19:09:22.637797+0000","last_undegraded":"2024-03-18T19:09:22.637797+0000","last_fullsized":"2024-03-18T19:09:22.637797+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:30:31.862656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.635565+0000","last_change":"2024-03-18T19:09:22.651971+0000","last_active":"2024-03-18T19:09:23.635565+0000","last_peered":"2024-03-18T19:09:23.635565+0000","last_clean":"2024-03-18T19:09:23.635565+0000","last_became_active":"2024-03-18T19:09:22.651728+0000","last_became_peered":"2024-03-18T19:09:22.651728+0000","last_unstale":"2024-03-18T19:09:23.635565+0000","last_undegraded":"2024-03-18T19:09:23.635565+0000","last_fullsized":"2024-03-18T19:09:23.635565+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:57:27.805513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"146'1","reported_seq":11,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.646771+0000","last_change":"2024-03-18T19:09:22.211398+0000","last_active":"2024-03-18T19:09:22.646771+0000","last_peered":"2024-03-18T19:09:22.646771+0000","last_clean":"2024-03-18T19:09:22.646771+0000","last_became_active":"2024-03-18T19:09:22.211147+0000","last_became_peered":"2024-03-18T19:09:22.211147+0000","last_unstale":"2024-03-18T19:09:22.646771+0000","last_undegraded":"2024-03-18T19:09:22.646771+0000","last_fullsized":"2024-03-18T19:09:22.646771+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:49:48.268109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.651632+0000","last_change":"2024-03-18T19:09:22.649071+0000","last_active":"2024-03-18T19:09:22.651632+0000","last_peered":"2024-03-18T19:09:22.651632+0000","last_clean":"2024-03-18T19:09:22.651632+0000","last_became_active":"2024-03-18T19:09:22.648709+0000","last_became_peered":"2024-03-18T19:09:22.648709+0000","last_unstale":"2024-03-18T19:09:22.651632+0000","last_undegraded":"2024-03-18T19:09:22.651632+0000","last_fullsized":"2024-03-18T19:09:22.651632+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:21:44.537753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"146'2","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.647069+0000","last_change":"2024-03-18T19:09:22.217258+0000","last_active":"2024-03-18T19:09:22.647069+0000","last_peered":"2024-03-18T19:09:22.647069+0000","last_clean":"2024-03-18T19:09:22.647069+0000","last_became_active":"2024-03-18T19:09:22.216952+0000","last_became_peered":"2024-03-18T19:09:22.216952+0000","last_unstale":"2024-03-18T19:09:22.647069+0000","last_undegraded":"2024-03-18T19:09:22.647069+0000","last_fullsized":"2024-03-18T19:09:22.647069+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:28:39.908259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.544396+0000","last_change":"2024-03-18T19:09:22.634767+0000","last_active":"2024-03-18T19:09:24.544396+0000","last_peered":"2024-03-18T19:09:24.544396+0000","last_clean":"2024-03-18T19:09:24.544396+0000","last_became_active":"2024-03-18T19:09:22.634515+0000","last_became_peered":"2024-03-18T19:09:22.634515+0000","last_unstale":"2024-03-18T19:09:24.544396+0000","last_undegraded":"2024-03-18T19:09:24.544396+0000","last_fullsized":"2024-03-18T19:09:24.544396+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:47:58.250987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"146'1","reported_seq":12,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.328031+0000","last_change":"2024-03-18T19:09:22.210772+0000","last_active":"2024-03-18T19:09:24.328031+0000","last_peered":"2024-03-18T19:09:24.328031+0000","last_clean":"2024-03-18T19:09:24.328031+0000","last_became_active":"2024-03-18T19:09:22.210511+0000","last_became_peered":"2024-03-18T19:09:22.210511+0000","last_unstale":"2024-03-18T19:09:24.328031+0000","last_undegraded":"2024-03-18T19:09:24.328031+0000","last_fullsized":"2024-03-18T19:09:24.328031+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:46:43.196183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.635327+0000","last_change":"2024-03-18T19:09:22.651154+0000","last_active":"2024-03-18T19:09:23.635327+0000","last_peered":"2024-03-18T19:09:23.635327+0000","last_clean":"2024-03-18T19:09:23.635327+0000","last_became_active":"2024-03-18T19:09:22.650883+0000","last_became_peered":"2024-03-18T19:09:22.650883+0000","last_unstale":"2024-03-18T19:09:23.635327+0000","last_undegraded":"2024-03-18T19:09:23.635327+0000","last_fullsized":"2024-03-18T19:09:23.635327+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:08.116636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327829+0000","last_change":"2024-03-18T19:09:22.216994+0000","last_active":"2024-03-18T19:09:24.327829+0000","last_peered":"2024-03-18T19:09:24.327829+0000","last_clean":"2024-03-18T19:09:24.327829+0000","last_became_active":"2024-03-18T19:09:22.216719+0000","last_became_peered":"2024-03-18T19:09:22.216719+0000","last_unstale":"2024-03-18T19:09:24.327829+0000","last_undegraded":"2024-03-18T19:09:24.327829+0000","last_fullsized":"2024-03-18T19:09:24.327829+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:12:31.452285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.543793+0000","last_change":"2024-03-18T19:09:22.633713+0000","last_active":"2024-03-18T19:09:24.543793+0000","last_peered":"2024-03-18T19:09:24.543793+0000","last_clean":"2024-03-18T19:09:24.543793+0000","last_became_active":"2024-03-18T19:09:22.633419+0000","last_became_peered":"2024-03-18T19:09:22.633419+0000","last_unstale":"2024-03-18T19:09:24.543793+0000","last_undegraded":"2024-03-18T19:09:24.543793+0000","last_fullsized":"2024-03-18T19:09:24.543793+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:43.874991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"146'2","reported_seq":13,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.543971+0000","last_change":"2024-03-18T19:09:22.215805+0000","last_active":"2024-03-18T19:09:24.543971+0000","last_peered":"2024-03-18T19:09:24.543971+0000","last_clean":"2024-03-18T19:09:24.543971+0000","last_became_active":"2024-03-18T19:09:22.215539+0000","last_became_peered":"2024-03-18T19:09:22.215539+0000","last_unstale":"2024-03-18T19:09:24.543971+0000","last_undegraded":"2024-03-18T19:09:24.543971+0000","last_fullsized":"2024-03-18T19:09:24.543971+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:33:28.327640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.654972+0000","last_change":"2024-03-18T19:09:22.653564+0000","last_active":"2024-03-18T19:09:22.654972+0000","last_peered":"2024-03-18T19:09:22.654972+0000","last_clean":"2024-03-18T19:09:22.654972+0000","last_became_active":"2024-03-18T19:09:22.653288+0000","last_became_peered":"2024-03-18T19:09:22.653288+0000","last_unstale":"2024-03-18T19:09:22.654972+0000","last_undegraded":"2024-03-18T19:09:22.654972+0000","last_fullsized":"2024-03-18T19:09:22.654972+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:51:38.942600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327668+0000","last_change":"2024-03-18T19:09:22.214404+0000","last_active":"2024-03-18T19:09:24.327668+0000","last_peered":"2024-03-18T19:09:24.327668+0000","last_clean":"2024-03-18T19:09:24.327668+0000","last_became_active":"2024-03-18T19:09:22.214057+0000","last_became_peered":"2024-03-18T19:09:22.214057+0000","last_unstale":"2024-03-18T19:09:24.327668+0000","last_undegraded":"2024-03-18T19:09:24.327668+0000","last_fullsized":"2024-03-18T19:09:24.327668+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:49:48.268109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.649333+0000","last_change":"2024-03-18T19:09:22.646657+0000","last_active":"2024-03-18T19:09:22.649333+0000","last_peered":"2024-03-18T19:09:22.649333+0000","last_clean":"2024-03-18T19:09:22.649333+0000","last_became_active":"2024-03-18T19:09:22.646337+0000","last_became_peered":"2024-03-18T19:09:22.646337+0000","last_unstale":"2024-03-18T19:09:22.649333+0000","last_undegraded":"2024-03-18T19:09:22.649333+0000","last_fullsized":"2024-03-18T19:09:22.649333+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:01.296112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.637902+0000","last_change":"2024-03-18T19:09:22.205466+0000","last_active":"2024-03-18T19:09:22.637902+0000","last_peered":"2024-03-18T19:09:22.637902+0000","last_clean":"2024-03-18T19:09:22.637902+0000","last_became_active":"2024-03-18T19:09:22.205156+0000","last_became_peered":"2024-03-18T19:09:22.205156+0000","last_unstale":"2024-03-18T19:09:22.637902+0000","last_undegraded":"2024-03-18T19:09:22.637902+0000","last_fullsized":"2024-03-18T19:09:22.637902+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:12:58.020407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.634760+0000","last_change":"2024-03-18T19:09:22.656882+0000","last_active":"2024-03-18T19:09:23.634760+0000","last_peered":"2024-03-18T19:09:23.634760+0000","last_clean":"2024-03-18T19:09:23.634760+0000","last_became_active":"2024-03-18T19:09:22.656625+0000","last_became_peered":"2024-03-18T19:09:22.656625+0000","last_unstale":"2024-03-18T19:09:23.634760+0000","last_undegraded":"2024-03-18T19:09:23.634760+0000","last_fullsized":"2024-03-18T19:09:23.634760+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:06:04.875225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327461+0000","last_change":"2024-03-18T19:09:22.206066+0000","last_active":"2024-03-18T19:09:24.327461+0000","last_peered":"2024-03-18T19:09:24.327461+0000","last_clean":"2024-03-18T19:09:24.327461+0000","last_became_active":"2024-03-18T19:09:22.205784+0000","last_became_peered":"2024-03-18T19:09:22.205784+0000","last_unstale":"2024-03-18T19:09:24.327461+0000","last_undegraded":"2024-03-18T19:09:24.327461+0000","last_fullsized":"2024-03-18T19:09:24.327461+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:23:13.662431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.544248+0000","last_change":"2024-03-18T19:09:22.635761+0000","last_active":"2024-03-18T19:09:24.544248+0000","last_peered":"2024-03-18T19:09:24.544248+0000","last_clean":"2024-03-18T19:09:24.544248+0000","last_became_active":"2024-03-18T19:09:22.635470+0000","last_became_peered":"2024-03-18T19:09:22.635470+0000","last_unstale":"2024-03-18T19:09:24.544248+0000","last_undegraded":"2024-03-18T19:09:24.544248+0000","last_fullsized":"2024-03-18T19:09:24.544248+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:39.450378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"146'2","reported_seq":13,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.544394+0000","last_change":"2024-03-18T19:09:22.208389+0000","last_active":"2024-03-18T19:09:24.544394+0000","last_peered":"2024-03-18T19:09:24.544394+0000","last_clean":"2024-03-18T19:09:24.544394+0000","last_became_active":"2024-03-18T19:09:22.208079+0000","last_became_peered":"2024-03-18T19:09:22.208079+0000","last_unstale":"2024-03-18T19:09:24.544394+0000","last_undegraded":"2024-03-18T19:09:24.544394+0000","last_fullsized":"2024-03-18T19:09:24.544394+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:34:18.760056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.640184+0000","last_change":"2024-03-18T19:09:22.637480+0000","last_active":"2024-03-18T19:09:22.640184+0000","last_peered":"2024-03-18T19:09:22.640184+0000","last_clean":"2024-03-18T19:09:22.640184+0000","last_became_active":"2024-03-18T19:09:22.637210+0000","last_became_peered":"2024-03-18T19:09:22.637210+0000","last_unstale":"2024-03-18T19:09:22.640184+0000","last_undegraded":"2024-03-18T19:09:22.640184+0000","last_fullsized":"2024-03-18T19:09:22.640184+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:30.062865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.635335+0000","last_change":"2024-03-18T19:09:22.215407+0000","last_active":"2024-03-18T19:09:23.635335+0000","last_peered":"2024-03-18T19:09:23.635335+0000","last_clean":"2024-03-18T19:09:23.635335+0000","last_became_active":"2024-03-18T19:09:22.215180+0000","last_became_peered":"2024-03-18T19:09:22.215180+0000","last_unstale":"2024-03-18T19:09:23.635335+0000","last_undegraded":"2024-03-18T19:09:23.635335+0000","last_fullsized":"2024-03-18T19:09:23.635335+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:49:48.268109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327985+0000","last_change":"2024-03-18T19:09:22.644776+0000","last_active":"2024-03-18T19:09:24.327985+0000","last_peered":"2024-03-18T19:09:24.327985+0000","last_clean":"2024-03-18T19:09:24.327985+0000","last_became_active":"2024-03-18T19:09:22.644275+0000","last_became_peered":"2024-03-18T19:09:22.644275+0000","last_unstale":"2024-03-18T19:09:24.327985+0000","last_undegraded":"2024-03-18T19:09:24.327985+0000","last_fullsized":"2024-03-18T19:09:24.327985+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:41:08.116636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:23.633808+0000","last_change":"2024-03-18T19:09:22.214530+0000","last_active":"2024-03-18T19:09:23.633808+0000","last_peered":"2024-03-18T19:09:23.633808+0000","last_clean":"2024-03-18T19:09:23.633808+0000","last_became_active":"2024-03-18T19:09:22.214220+0000","last_became_peered":"2024-03-18T19:09:22.214220+0000","last_unstale":"2024-03-18T19:09:23.633808+0000","last_undegraded":"2024-03-18T19:09:23.633808+0000","last_fullsized":"2024-03-18T19:09:23.633808+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:12:31.452285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.656146+0000","last_change":"2024-03-18T19:09:22.655155+0000","last_active":"2024-03-18T19:09:22.656146+0000","last_peered":"2024-03-18T19:09:22.656146+0000","last_clean":"2024-03-18T19:09:22.656146+0000","last_became_active":"2024-03-18T19:09:22.654886+0000","last_became_peered":"2024-03-18T19:09:22.654886+0000","last_unstale":"2024-03-18T19:09:22.656146+0000","last_undegraded":"2024-03-18T19:09:22.656146+0000","last_fullsized":"2024-03-18T19:09:22.656146+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:58:31.158678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.544336+0000","last_change":"2024-03-18T19:09:22.217529+0000","last_active":"2024-03-18T19:09:24.544336+0000","last_peered":"2024-03-18T19:09:24.544336+0000","last_clean":"2024-03-18T19:09:24.544336+0000","last_became_active":"2024-03-18T19:09:22.217033+0000","last_became_peered":"2024-03-18T19:09:22.217033+0000","last_unstale":"2024-03-18T19:09:24.544336+0000","last_undegraded":"2024-03-18T19:09:24.544336+0000","last_fullsized":"2024-03-18T19:09:24.544336+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:03:15.951011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.543989+0000","last_change":"2024-03-18T19:09:22.642681+0000","last_active":"2024-03-18T19:09:24.543989+0000","last_peered":"2024-03-18T19:09:24.543989+0000","last_clean":"2024-03-18T19:09:24.543989+0000","last_became_active":"2024-03-18T19:09:22.641787+0000","last_became_peered":"2024-03-18T19:09:22.641787+0000","last_unstale":"2024-03-18T19:09:24.543989+0000","last_undegraded":"2024-03-18T19:09:24.543989+0000","last_fullsized":"2024-03-18T19:09:24.543989+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:07:28.636110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.637987+0000","last_change":"2024-03-18T19:09:22.213628+0000","last_active":"2024-03-18T19:09:22.637987+0000","last_peered":"2024-03-18T19:09:22.637987+0000","last_clean":"2024-03-18T19:09:22.637987+0000","last_became_active":"2024-03-18T19:09:22.213369+0000","last_became_peered":"2024-03-18T19:09:22.213369+0000","last_unstale":"2024-03-18T19:09:22.637987+0000","last_undegraded":"2024-03-18T19:09:22.637987+0000","last_fullsized":"2024-03-18T19:09:22.637987+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:29.057738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.649758+0000","last_change":"2024-03-18T19:09:22.647165+0000","last_active":"2024-03-18T19:09:22.649758+0000","last_peered":"2024-03-18T19:09:22.649758+0000","last_clean":"2024-03-18T19:09:22.649758+0000","last_became_active":"2024-03-18T19:09:22.646619+0000","last_became_peered":"2024-03-18T19:09:22.646619+0000","last_unstale":"2024-03-18T19:09:22.649758+0000","last_undegraded":"2024-03-18T19:09:22.649758+0000","last_fullsized":"2024-03-18T19:09:22.649758+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:15.237092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":10,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.638735+0000","last_change":"2024-03-18T19:09:22.211020+0000","last_active":"2024-03-18T19:09:22.638735+0000","last_peered":"2024-03-18T19:09:22.638735+0000","last_clean":"2024-03-18T19:09:22.638735+0000","last_became_active":"2024-03-18T19:09:22.210761+0000","last_became_peered":"2024-03-18T19:09:22.210761+0000","last_unstale":"2024-03-18T19:09:22.638735+0000","last_undegraded":"2024-03-18T19:09:22.638735+0000","last_fullsized":"2024-03-18T19:09:22.638735+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:25:50.979606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327782+0000","last_change":"2024-03-18T19:09:22.637700+0000","last_active":"2024-03-18T19:09:24.327782+0000","last_peered":"2024-03-18T19:09:24.327782+0000","last_clean":"2024-03-18T19:09:24.327782+0000","last_became_active":"2024-03-18T19:09:22.637417+0000","last_became_peered":"2024-03-18T19:09:22.637417+0000","last_unstale":"2024-03-18T19:09:24.327782+0000","last_undegraded":"2024-03-18T19:09:24.327782+0000","last_fullsized":"2024-03-18T19:09:24.327782+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:48:33.430169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.543956+0000","last_change":"2024-03-18T19:09:22.216857+0000","last_active":"2024-03-18T19:09:24.543956+0000","last_peered":"2024-03-18T19:09:24.543956+0000","last_clean":"2024-03-18T19:09:24.543956+0000","last_became_active":"2024-03-18T19:09:22.216540+0000","last_became_peered":"2024-03-18T19:09:22.216540+0000","last_unstale":"2024-03-18T19:09:24.543956+0000","last_undegraded":"2024-03-18T19:09:24.543956+0000","last_fullsized":"2024-03-18T19:09:24.543956+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:23.822038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.640313+0000","last_change":"2024-03-18T19:09:22.637602+0000","last_active":"2024-03-18T19:09:22.640313+0000","last_peered":"2024-03-18T19:09:22.640313+0000","last_clean":"2024-03-18T19:09:22.640313+0000","last_became_active":"2024-03-18T19:09:22.637351+0000","last_became_peered":"2024-03-18T19:09:22.637351+0000","last_unstale":"2024-03-18T19:09:22.640313+0000","last_undegraded":"2024-03-18T19:09:22.640313+0000","last_fullsized":"2024-03-18T19:09:22.640313+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:19.098002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327456+0000","last_change":"2024-03-18T19:09:22.207744+0000","last_active":"2024-03-18T19:09:24.327456+0000","last_peered":"2024-03-18T19:09:24.327456+0000","last_clean":"2024-03-18T19:09:24.327456+0000","last_became_active":"2024-03-18T19:09:22.206341+0000","last_became_peered":"2024-03-18T19:09:22.206341+0000","last_unstale":"2024-03-18T19:09:24.327456+0000","last_undegraded":"2024-03-18T19:09:24.327456+0000","last_fullsized":"2024-03-18T19:09:24.327456+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:45:55.970780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":10,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-18T19:09:24.327830+0000","last_change":"2024-03-18T19:09:22.639977+0000","last_active":"2024-03-18T19:09:24.327830+0000","last_peered":"2024-03-18T19:09:24.327830+0000","last_clean":"2024-03-18T19:09:24.327830+0000","last_became_active":"2024-03-18T19:09:22.639700+0000","last_became_peered":"2024-03-18T19:09:22.639700+0000","last_unstale":"2024-03-18T19:09:24.327830+0000","last_undegraded":"2024-03-18T19:09:24.327830+0000","last_fullsized":"2024-03-18T19:09:24.327830+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":145,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:22.190544+0000","last_clean_scrub_stamp":"2024-03-18T19:09:22.190544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:43:09.563639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"146'8","reported_seq":18,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.652528+0000","last_change":"2024-03-18T19:09:22.215782+0000","last_active":"2024-03-18T19:09:22.652528+0000","last_peered":"2024-03-18T19:09:22.652528+0000","last_clean":"2024-03-18T19:09:22.652528+0000","last_became_active":"2024-03-18T19:09:22.215478+0000","last_became_peered":"2024-03-18T19:09:22.215478+0000","last_unstale":"2024-03-18T19:09:22.652528+0000","last_undegraded":"2024-03-18T19:09:22.652528+0000","last_fullsized":"2024-03-18T19:09:22.652528+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:09:21.185417+0000","last_clean_scrub_stamp":"2024-03-18T19:09:21.185417+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:08.578704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"136'227","reported_seq":664,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-18T19:09:22.630751+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T19:09:22.630751+0000","last_peered":"2024-03-18T19:09:22.630751+0000","last_clean":"2024-03-18T19:09:22.630751+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T19:09:22.630751+0000","last_undegraded":"2024-03-18T19:09:22.630751+0000","last_fullsized":"2024-03-18T19:09:22.630751+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575077,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59732,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312108,"statfs":{"total":96636764160,"available":96575598592,"internally_reserved":0,"allocated":2424832,"data_stored":1504495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.55300000000000005,"15min":0.56599999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.27800000000000002},"max":{"1min":0.76200000000000001,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55800000000000005,"15min":0.57899999999999996},"min":{"1min":0.373,"5min":0.33700000000000002,"15min":0.318},"max":{"1min":0.78300000000000003,"5min":1.387,"15min":1.387},"last":0.67800000000000005}]},{"osd":1,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.53500000000000003,"15min":0.56799999999999995},"min":{"1min":0.34399999999999997,"5min":0.33200000000000002,"15min":0.307},"max":{"1min":0.73299999999999998,"5min":0.81699999999999995,"15min":1.484},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55100000000000005,"15min":0.58599999999999997},"min":{"1min":0.39500000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.92000000000000004},"last":0.72299999999999998}]},{"osd":2,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55200000000000005,"15min":0.57099999999999995},"min":{"1min":0.38200000000000001,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":0.68000000000000005,"5min":1.3600000000000001,"15min":1.5129999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57499999999999996,"15min":0.59099999999999997},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73499999999999999,"5min":1.327,"15min":1.536},"last":0.65100000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55400000000000005,"15min":0.57599999999999996},"min":{"1min":0.38800000000000001,"5min":0.35599999999999998,"15min":0.318},"max":{"1min":0.745,"5min":0.79100000000000004,"15min":1.5489999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57399999999999995,"15min":0.59799999999999998},"min":{"1min":0.441,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.78300000000000003,"5min":0.81699999999999995,"15min":1.5660000000000001},"last":0.68700000000000006}]},{"osd":4,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.45800000000000002,"15min":0.45300000000000001},"min":{"1min":0.30299999999999999,"5min":0.27800000000000002,"15min":0.27100000000000002},"max":{"1min":0.748,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.495},{"interface":"front","average":{"1min":0.505,"5min":0.46999999999999997,"15min":0.47299999999999998},"min":{"1min":0.35299999999999998,"5min":0.27500000000000002,"15min":0.25800000000000001},"max":{"1min":0.63900000000000001,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.46700000000000003}]},{"osd":5,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.48499999999999999,"15min":0.47699999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.26000000000000001},"max":{"1min":0.68600000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.51500000000000001,"15min":0.51400000000000001},"min":{"1min":0.33600000000000002,"5min":0.317,"15min":0.30299999999999999},"max":{"1min":0.70699999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.56499999999999995}]},{"osd":6,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.54300000000000004},"min":{"1min":0.34000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.74099999999999999,"5min":0.81799999999999995,"15min":0.82799999999999996},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53600000000000003,"15min":0.53000000000000003},"min":{"1min":0.36399999999999999,"5min":0.311,"15min":0.29099999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.66500000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575077,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32912,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338928,"statfs":{"total":96636764160,"available":96603062272,"internally_reserved":0,"allocated":1830912,"data_stored":920581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52500000000000002,"15min":0.55500000000000005},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28499999999999998},"max":{"1min":0.67800000000000005,"5min":0.73699999999999999,"15min":0.76800000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53900000000000003,"15min":0.56499999999999995},"min":{"1min":0.36199999999999999,"5min":0.35199999999999998,"15min":0.27400000000000002},"max":{"1min":0.71099999999999997,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.46300000000000002}]},{"osd":1,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.56000000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.77400000000000002},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.56499999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.32000000000000001},"max":{"1min":0.78900000000000003,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.374}]},{"osd":2,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55400000000000005,"15min":0.57299999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.30499999999999999},"max":{"1min":0.73699999999999999,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54500000000000004,"15min":0.57399999999999995},"min":{"1min":0.40999999999999998,"5min":0.35299999999999998,"15min":0.30299999999999999},"max":{"1min":0.73399999999999999,"5min":0.753,"15min":0.83199999999999996},"last":0.55400000000000005}]},{"osd":3,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.58399999999999996},"min":{"1min":0.40799999999999997,"5min":0.372,"15min":0.32100000000000001},"max":{"1min":0.76400000000000001,"5min":0.78400000000000003,"15min":0.89600000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55400000000000005,"15min":0.57199999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72099999999999997,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.442}]},{"osd":4,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46700000000000003,"15min":0.45000000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.432},{"interface":"front","average":{"1min":0.495,"5min":0.48899999999999999,"15min":0.47699999999999998},"min":{"1min":0.32200000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.623}]},{"osd":5,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.75,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53400000000000003,"15min":0.51600000000000001},"min":{"1min":0.34899999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.57999999999999996}]},{"osd":7,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54400000000000004,"15min":0.52900000000000003},"min":{"1min":0.371,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55600000000000005,"15min":0.53600000000000003},"min":{"1min":0.371,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.80500000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.56799999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575077,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36996,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334844,"statfs":{"total":96636764160,"available":96598880256,"internally_reserved":0,"allocated":1818624,"data_stored":913373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.35499999999999998,"15min":0.373},"min":{"1min":0.216,"5min":0.216,"15min":0.19600000000000001},"max":{"1min":0.58699999999999997,"5min":0.63300000000000001,"15min":0.66300000000000003},"last":0.371},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.38600000000000001,"15min":0.41299999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.57099999999999995,"5min":0.66500000000000004,"15min":0.73699999999999999},"last":0.35899999999999999}]},{"osd":2,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38900000000000001,"15min":0.40400000000000003},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.59299999999999997,"5min":0.64200000000000002,"15min":0.65400000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.434,"15min":0.44500000000000001},"min":{"1min":0.29599999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.63300000000000001,"5min":0.66500000000000004,"15min":0.71699999999999997},"last":0.39300000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.44500000000000001,"15min":0.46400000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.60399999999999998,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40200000000000002,"15min":0.41899999999999998},"min":{"1min":0.25900000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.61199999999999999,"5min":0.67800000000000005,"15min":0.77000000000000002},"last":0.34100000000000003}]},{"osd":4,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60499999999999998,"15min":0.628},"min":{"1min":0.46200000000000002,"5min":0.33400000000000002,"15min":0.32800000000000001},"max":{"1min":0.89600000000000002,"5min":0.92300000000000004,"15min":0.93999999999999995},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.62,"15min":0.64100000000000001},"min":{"1min":0.443,"5min":0.439,"15min":0.36899999999999999},"max":{"1min":0.86499999999999999,"5min":0.97499999999999998,"15min":1.018},"last":0.499}]},{"osd":5,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60399999999999998,"15min":0.63700000000000001},"min":{"1min":0.40500000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88700000000000001,"5min":0.89500000000000002,"15min":0.99199999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.63200000000000001,"15min":0.65600000000000003},"min":{"1min":0.47199999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":1.006},"last":0.54800000000000004}]},{"osd":6,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.60499999999999998,"15min":0.63400000000000001},"min":{"1min":0.42899999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.878,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.64600000000000002,"15min":0.67000000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.39100000000000001},"max":{"1min":0.91300000000000003,"5min":0.94899999999999995,"15min":0.98099999999999998},"last":0.46100000000000002}]},{"osd":7,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63200000000000001,"15min":0.65900000000000003},"min":{"1min":0.45100000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81899999999999995,"5min":0.90600000000000003,"15min":0.94799999999999995},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63400000000000001,"15min":0.66200000000000003},"min":{"1min":0.48299999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90800000000000003,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.59199999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575078,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37576,"kb_used_data":2356,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334264,"statfs":{"total":96636764160,"available":96598286336,"internally_reserved":0,"allocated":2412544,"data_stored":1503741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36299999999999999,"15min":0.376},"min":{"1min":0.218,"5min":0.19700000000000001,"15min":0.19400000000000001},"max":{"1min":0.59899999999999998,"5min":0.60499999999999998,"15min":0.621},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39400000000000002,"15min":0.40999999999999998},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20000000000000001},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.63200000000000001},"last":0.29199999999999998}]},{"osd":2,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40699999999999997,"15min":0.41699999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.21099999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.42299999999999999,"15min":0.432},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.59399999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.42299999999999999}]},{"osd":3,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.40300000000000002,"15min":0.41299999999999998},"min":{"1min":0.27200000000000002,"5min":0.26500000000000001,"15min":0.217},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.64100000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.45500000000000002,"15min":0.46800000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.28799999999999998},"max":{"1min":0.64300000000000002,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.442}]},{"osd":4,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.60099999999999998,"15min":0.63300000000000001},"min":{"1min":0.376,"5min":0.35699999999999998,"15min":0.27200000000000002},"max":{"1min":0.76900000000000002,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.60899999999999999,"15min":0.63500000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.315},"max":{"1min":0.75600000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.58299999999999996}]},{"osd":5,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.627,"15min":0.64300000000000002},"min":{"1min":0.432,"5min":0.376,"15min":0.34300000000000003},"max":{"1min":0.78100000000000003,"5min":1.248,"15min":1.248},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.63800000000000001,"15min":0.65800000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.31},"max":{"1min":0.84899999999999998,"5min":1.399,"15min":1.399},"last":0.49399999999999999}]},{"osd":6,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.63200000000000001,"15min":0.65500000000000003},"min":{"1min":0.44700000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.86099999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.503},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.64900000000000002,"15min":0.66700000000000004},"min":{"1min":0.47599999999999998,"5min":0.42399999999999999,"15min":0.40999999999999998},"max":{"1min":0.83499999999999996,"5min":1.284,"15min":1.284},"last":0.71199999999999997}]},{"osd":7,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.64100000000000001,"15min":0.66300000000000003},"min":{"1min":0.42399999999999999,"5min":0.39300000000000002,"15min":0.34899999999999998},"max":{"1min":0.878,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64000000000000001,"15min":0.66400000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.38500000000000001},"max":{"1min":0.86599999999999999,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.65900000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575078,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59136,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312704,"statfs":{"total":96636764160,"available":96576208896,"internally_reserved":0,"allocated":1814528,"data_stored":913351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34999999999999998,"15min":0.36799999999999999},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.498,"5min":0.66600000000000004,"15min":0.78300000000000003},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.44400000000000001,"15min":0.45500000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.217},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.77200000000000002},"last":0.31900000000000001}]},{"osd":1,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.435,"15min":0.46400000000000002},"min":{"1min":0.31,"5min":0.27300000000000002,"15min":0.27100000000000002},"max":{"1min":0.62,"5min":0.69499999999999995,"15min":0.90500000000000003},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.38400000000000001,"15min":0.39600000000000002},"min":{"1min":0.26700000000000002,"5min":0.22800000000000001,"15min":0.216},"max":{"1min":0.54300000000000004,"5min":0.64500000000000002,"15min":0.66700000000000004},"last":0.34699999999999998}]},{"osd":3,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.24199999999999999},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47599999999999998,"15min":0.499},"min":{"1min":0.33400000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.80100000000000005},"last":0.36299999999999999}]},{"osd":4,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58399999999999996,"15min":0.60999999999999999},"min":{"1min":0.33500000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76100000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.59799999999999998,"15min":0.624},"min":{"1min":0.38600000000000001,"5min":0.378,"15min":0.32100000000000001},"max":{"1min":1.355,"5min":1.448,"15min":1.4730000000000001},"last":0.45300000000000001}]},{"osd":5,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.61899999999999999,"15min":0.64100000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.34699999999999998},"max":{"1min":0.90600000000000003,"5min":1.327,"15min":1.8109999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.59999999999999998,"15min":0.63100000000000001},"min":{"1min":0.376,"5min":0.36899999999999999,"15min":0.33400000000000002},"max":{"1min":0.75,"5min":1.4630000000000001,"15min":1.948},"last":0.52800000000000002}]},{"osd":6,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.624,"15min":0.65500000000000003},"min":{"1min":0.42599999999999999,"5min":0.39600000000000002,"15min":0.35899999999999999},"max":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":2.9470000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65600000000000003,"15min":0.67900000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.38100000000000001},"max":{"1min":0.94899999999999995,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.48099999999999998}]},{"osd":7,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.625,"15min":0.64700000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.89100000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64000000000000001,"15min":0.66100000000000003},"min":{"1min":0.437,"5min":0.38300000000000001,"15min":0.371},"max":{"1min":1.409,"5min":1.482,"15min":1.482},"last":0.51400000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575078,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37004,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334836,"statfs":{"total":96636764160,"available":96598872064,"internally_reserved":0,"allocated":1830912,"data_stored":914949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34699999999999998,"15min":0.374},"min":{"1min":0.23499999999999999,"5min":0.221,"15min":0.21099999999999999},"max":{"1min":0.442,"5min":0.63900000000000001,"15min":1.204},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38,"15min":0.39900000000000002},"min":{"1min":0.25700000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.52900000000000003,"5min":0.66900000000000004,"15min":1.137},"last":0.33600000000000002}]},{"osd":1,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.36899999999999999,"15min":0.39500000000000002},"min":{"1min":0.27700000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.502,"5min":0.629,"15min":1.1220000000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40699999999999997,"15min":0.441},"min":{"1min":0.29499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.57399999999999995,"5min":0.76600000000000001,"15min":1.1080000000000001},"last":0.312}]},{"osd":2,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.41799999999999998,"15min":0.44600000000000001},"min":{"1min":0.308,"5min":0.29499999999999998,"15min":0.28399999999999997},"max":{"1min":0.61799999999999999,"5min":0.67500000000000004,"15min":1.1879999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.44500000000000001,"15min":0.47299999999999998},"min":{"1min":0.33400000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.55600000000000005,"5min":0.71499999999999997,"15min":1.2290000000000001},"last":0.44500000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.58099999999999996,"15min":0.62},"min":{"1min":0.441,"5min":0.38100000000000001,"15min":0.34300000000000003},"max":{"1min":0.82899999999999996,"5min":1.052,"15min":1.871},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60199999999999998,"15min":0.629},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.39600000000000002},"max":{"1min":0.81000000000000005,"5min":0.90600000000000003,"15min":1.764},"last":0.59099999999999997}]},{"osd":5,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.63400000000000001,"15min":0.66200000000000003},"min":{"1min":0.47099999999999997,"5min":0.39200000000000002,"15min":0.379},"max":{"1min":0.84999999999999998,"5min":1.0249999999999999,"15min":2.5329999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.65500000000000003,"15min":0.68200000000000005},"min":{"1min":0.43099999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.92900000000000005,"5min":0.94599999999999995,"15min":1.794},"last":0.51000000000000001}]},{"osd":6,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.63900000000000001,"15min":0.67300000000000004},"min":{"1min":0.496,"5min":0.42499999999999999,"15min":0.41499999999999998},"max":{"1min":0.89400000000000002,"5min":1.1060000000000001,"15min":2.7389999999999999},"last":0.625},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.65000000000000002,"15min":0.67600000000000005},"min":{"1min":0.45000000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.91200000000000003,"5min":0.98299999999999998,"15min":1.8069999999999999},"last":0.54200000000000004}]},{"osd":7,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63100000000000001,"15min":0.66400000000000003},"min":{"1min":0.43099999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":1.732},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.63400000000000001,"15min":0.66900000000000004},"min":{"1min":0.51800000000000002,"5min":0.45900000000000002,"15min":0.40799999999999997},"max":{"1min":0.86299999999999999,"5min":1.085,"15min":2.7050000000000001},"last":0.52600000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575078,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32908,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338932,"statfs":{"total":96636764160,"available":96603066368,"internally_reserved":0,"allocated":1826816,"data_stored":914915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.56699999999999995,"15min":0.57499999999999996},"min":{"1min":0.43099999999999999,"5min":0.31,"15min":0.246},"max":{"1min":2.3210000000000002,"5min":2.3210000000000002,"15min":2.3210000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.41699999999999998,"5min":0.35299999999999998,"15min":0.26000000000000001},"max":{"1min":1.7430000000000001,"5min":3.645,"15min":3.645},"last":0.36899999999999999}]},{"osd":1,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65000000000000002,"15min":0.61199999999999999},"min":{"1min":0.40699999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.6739999999999999,"5min":9.9499999999999993,"15min":9.9499999999999993},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54400000000000004,"15min":0.57399999999999995},"min":{"1min":0.40200000000000002,"5min":0.32300000000000001,"15min":0.308},"max":{"1min":0.80000000000000004,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.92700000000000005}]},{"osd":2,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56699999999999995,"15min":0.57799999999999996},"min":{"1min":0.439,"5min":0.35899999999999999,"15min":0.28999999999999998},"max":{"1min":2.431,"5min":2.431,"15min":2.431},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.61499999999999999,"15min":0.61299999999999999},"min":{"1min":0.45700000000000002,"5min":0.35099999999999998,"15min":0.32600000000000001},"max":{"1min":2.165,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.46500000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.55800000000000005,"15min":0.58299999999999996},"min":{"1min":0.39700000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76800000000000002,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.58799999999999997,"15min":0.59999999999999998},"min":{"1min":0.42599999999999999,"5min":0.39300000000000002,"15min":0.33900000000000002},"max":{"1min":2.5510000000000002,"5min":2.5510000000000002,"15min":2.5510000000000002},"last":0.35099999999999998}]},{"osd":5,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46300000000000002,"15min":0.46000000000000002},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.245},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.77500000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.51600000000000001,"15min":0.49199999999999999},"min":{"1min":0.307,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.73599999999999999,"5min":5.4900000000000002,"15min":5.4900000000000002},"last":0.28599999999999998}]},{"osd":6,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.49199999999999999,"15min":0.48499999999999999},"min":{"1min":0.33200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.5190000000000001,"5min":2.5190000000000001,"15min":2.5190000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.55100000000000005,"15min":0.52900000000000003},"min":{"1min":0.32600000000000001,"5min":0.317,"15min":0.28999999999999998},"max":{"1min":1.2689999999999999,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":0.44}]},{"osd":7,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52200000000000002,"15min":0.51900000000000002},"min":{"1min":0.34999999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.94599999999999995,"5min":1.55,"15min":1.55},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.54800000000000004,"15min":0.54900000000000004},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.45400000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575077,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50956,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320884,"statfs":{"total":96636764160,"available":96584585216,"internally_reserved":0,"allocated":1835008,"data_stored":920159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55400000000000005,"15min":0.56599999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.26600000000000001},"max":{"1min":0.76400000000000001,"5min":0.86099999999999999,"15min":1.1679999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55200000000000005,"15min":0.56599999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.25800000000000001},"max":{"1min":0.69699999999999995,"5min":0.80600000000000005,"15min":1.1319999999999999},"last":0.32700000000000001}]},{"osd":1,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54800000000000004,"15min":0.56599999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.29599999999999999},"max":{"1min":0.78300000000000003,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.307},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.56100000000000005,"15min":0.57899999999999996},"min":{"1min":0.34599999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71999999999999997,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.53200000000000003}]},{"osd":2,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55300000000000005,"15min":0.57399999999999995},"min":{"1min":0.371,"5min":0.30399999999999999,"15min":0.28399999999999997},"max":{"1min":0.79700000000000004,"5min":1.153,"15min":1.1579999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55100000000000005,"15min":0.56899999999999995},"min":{"1min":0.377,"5min":0.35599999999999998,"15min":0.33200000000000002},"max":{"1min":0.71199999999999997,"5min":0.79500000000000004,"15min":1.0569999999999999},"last":0.41799999999999998}]},{"osd":3,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.39700000000000002,"5min":0.35799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72499999999999998,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57399999999999995,"15min":0.59599999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77000000000000002,"5min":1.212,"15min":1.212},"last":0.51000000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.46300000000000002},"min":{"1min":0.34799999999999998,"5min":0.255,"15min":0.24199999999999999},"max":{"1min":0.64700000000000002,"5min":1.248,"15min":1.248},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.496,"15min":0.48899999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.23100000000000001},"max":{"1min":0.82599999999999996,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.625}]},{"osd":6,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.29399999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.68000000000000005,"5min":0.78200000000000003,"15min":1.0069999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.496,"5min":0.51100000000000001,"15min":0.50900000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.26800000000000002},"max":{"1min":0.71799999999999997,"5min":0.83199999999999996,"15min":0.94599999999999995},"last":0.55400000000000005}]},{"osd":7,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.32700000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.85699999999999998,"5min":1.282,"15min":1.282},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.34200000000000003,"5min":0.28799999999999998,"15min":0.28499999999999998},"max":{"1min":0.81000000000000005,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.64200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T19:09:27.192 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T19:09:27.193 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-18T19:09:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.364+0000 7fbb9dbbf700 1 Processor -- start 2024-03-18T19:09:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.364+0000 7fbb9dbbf700 1 -- start start 2024-03-18T19:09:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.365+0000 7fbb9dbbf700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbb980cfa90 0x7fbb9806a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.365+0000 7fbb9dbbf700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb980cf600 con 0x7fbb9806ac90 2024-03-18T19:09:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.365+0000 7fbb9dbbf700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb980ce720 con 0x7fbb9815d5a0 2024-03-18T19:09:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.365+0000 7fbb9dbbf700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb9806b140 con 0x7fbb980cfa90 2024-03-18T19:09:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.365+0000 7fbb97fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbb9806ac90 0x7fbb98167140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51132/0 (socket says 172.21.15.1:51132) 2024-03-18T19:09:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.365+0000 7fbb97fff700 1 -- 172.21.15.1:0/2294316573 learned_addr learned my addr 172.21.15.1:0/2294316573 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.365+0000 7fbb96ffd700 1 --2- 172.21.15.1:0/2294316573 >> v2:172.21.15.165:3300/0 conn(0x7fbb980cfa90 0x7fbb9806a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.365+0000 7fbb967fc700 1 -- 172.21.15.1:0/2294316573 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648752690 0 0) 0x7fbb980cf600 con 0x7fbb9806ac90 2024-03-18T19:09:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.365+0000 7fbb967fc700 1 -- 172.21.15.1:0/2294316573 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb840035b0 con 0x7fbb9806ac90 2024-03-18T19:09:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.365+0000 7fbb967fc700 1 -- 172.21.15.1:0/2294316573 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 525819766 0 0) 0x7fbb840035b0 con 0x7fbb9806ac90 2024-03-18T19:09:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.365+0000 7fbb967fc700 1 -- 172.21.15.1:0/2294316573 >> v2:172.21.15.165:3300/0 conn(0x7fbb980cfa90 msgr2=0x7fbb9806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.365+0000 7fbb967fc700 1 --2- 172.21.15.1:0/2294316573 >> v2:172.21.15.165:3300/0 conn(0x7fbb980cfa90 0x7fbb9806a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.365+0000 7fbb967fc700 1 -- 172.21.15.1:0/2294316573 >> v1:172.21.15.165:6789/0 conn(0x7fbb9815d5a0 legacy=0x7fbb980ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.365+0000 7fbb967fc700 1 -- 172.21.15.1:0/2294316573 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb980ce9f0 con 0x7fbb9806ac90 2024-03-18T19:09:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb967fc700 1 -- 172.21.15.1:0/2294316573 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fbb900019f0 con 0x7fbb9806ac90 2024-03-18T19:09:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb967fc700 1 -- 172.21.15.1:0/2294316573 >> v1:172.21.15.1:6789/0 conn(0x7fbb9806ac90 legacy=0x7fbb98167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb967fc700 1 --2- 172.21.15.1:0/2294316573 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb84003a00 0x7fbb84003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb967fc700 1 --2- 172.21.15.1:0/2294316573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbb84004850 0x7fbb84004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb967fc700 1 --2- 172.21.15.1:0/2294316573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb84005210 0x7fbb8400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb967fc700 1 -- 172.21.15.1:0/2294316573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9806b140 con 0x7fbb84005210 2024-03-18T19:09:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb967fc700 1 -- 172.21.15.1:0/2294316573 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8400c0b0 con 0x7fbb84003a00 2024-03-18T19:09:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb967fc700 1 -- 172.21.15.1:0/2294316573 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8400c270 con 0x7fbb84004850 2024-03-18T19:09:27.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb97fff700 1 --2- 172.21.15.1:0/2294316573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb84005210 0x7fbb8400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:27.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb977fe700 1 --2- 172.21.15.1:0/2294316573 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb84003a00 0x7fbb84003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:27.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb96ffd700 1 --2- 172.21.15.1:0/2294316573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbb84004850 0x7fbb84004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:27.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb97fff700 1 -- 172.21.15.1:0/2294316573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbb84004850 msgr2=0x7fbb84004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:27.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb97fff700 1 --2- 172.21.15.1:0/2294316573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbb84004850 0x7fbb84004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:27.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb97fff700 1 -- 172.21.15.1:0/2294316573 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb84003a00 msgr2=0x7fbb84003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:27.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb97fff700 1 --2- 172.21.15.1:0/2294316573 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb84003a00 0x7fbb84003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:27.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb97fff700 1 -- 172.21.15.1:0/2294316573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb840035b0 con 0x7fbb84005210 2024-03-18T19:09:27.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb96ffd700 1 --2- 172.21.15.1:0/2294316573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbb84004850 0x7fbb84004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:27.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb97fff700 1 -- 172.21.15.1:0/2294316573 reap_dead start 2024-03-18T19:09:27.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb97fff700 1 --2- 172.21.15.1:0/2294316573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb84005210 0x7fbb8400ba10 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fbb9000a820 tx=0x7fbb84005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6acf55dff8e2288e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:27.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb967fc700 1 -- 172.21.15.1:0/2294316573 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9000a2f0 con 0x7fbb84005210 2024-03-18T19:09:27.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb967fc700 1 -- 172.21.15.1:0/2294316573 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb90012620 con 0x7fbb84005210 2024-03-18T19:09:27.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.366+0000 7fbb967fc700 1 -- 172.21.15.1:0/2294316573 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb900129d0 con 0x7fbb84005210 2024-03-18T19:09:27.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.367+0000 7fbb9dbbf700 1 -- 172.21.15.1:0/2294316573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb84005210 msgr2=0x7fbb8400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:27.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.367+0000 7fbb9dbbf700 1 --2- 172.21.15.1:0/2294316573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb84005210 0x7fbb8400ba10 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fbb9000a820 tx=0x7fbb84005880 comp rx=0 tx=0).stop 2024-03-18T19:09:27.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.367+0000 7fbb9dbbf700 1 -- 172.21.15.1:0/2294316573 shutdown_connections 2024-03-18T19:09:27.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.367+0000 7fbb9dbbf700 1 --2- 172.21.15.1:0/2294316573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb84005210 0x7fbb8400ba10 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:27.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.367+0000 7fbb9dbbf700 1 -- 172.21.15.1:0/2294316573 >> 172.21.15.1:0/2294316573 conn(0x7fbb980cc9b0 msgr2=0x7fbb980ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:27.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.367+0000 7fbb9dbbf700 1 -- 172.21.15.1:0/2294316573 shutdown_connections 2024-03-18T19:09:27.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.367+0000 7fbb9dbbf700 1 -- 172.21.15.1:0/2294316573 wait complete. 2024-03-18T19:09:27.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.368+0000 7fbb9dbbf700 1 Processor -- start 2024-03-18T19:09:27.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.369+0000 7fbb9dbbf700 1 -- start start 2024-03-18T19:09:27.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.370+0000 7fbb9dbbf700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb84005210 0x7fbb981551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:27.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.370+0000 7fbb9dbbf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbb9815ece0 0x7fbb981492d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:27.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.370+0000 7fbb9dbbf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb98153180 0x7fbb98153610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:27.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.370+0000 7fbb9dbbf700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb98153b50 con 0x7fbb98153180 2024-03-18T19:09:27.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.370+0000 7fbb9dbbf700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9815f510 con 0x7fbb84005210 2024-03-18T19:09:27.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.370+0000 7fbb9dbbf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9815f7f0 con 0x7fbb9815ece0 2024-03-18T19:09:27.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.370+0000 7fbb97fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb98153180 0x7fbb98153610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:27.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.370+0000 7fbb97fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb98153180 0x7fbb98153610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38986/0 (socket says 172.21.15.1:38986) 2024-03-18T19:09:27.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.370+0000 7fbb97fff700 1 -- 172.21.15.1:0/1606025210 learned_addr learned my addr 172.21.15.1:0/1606025210 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.370+0000 7fbb977fe700 1 --2- 172.21.15.1:0/1606025210 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb84005210 0x7fbb981551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.370+0000 7fbb96ffd700 1 --2- 172.21.15.1:0/1606025210 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbb9815ece0 0x7fbb981492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.370+0000 7fbb97fff700 1 -- 172.21.15.1:0/1606025210 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbb9815ece0 msgr2=0x7fbb981492d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.370+0000 7fbb97fff700 1 --2- 172.21.15.1:0/1606025210 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbb9815ece0 0x7fbb981492d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.370+0000 7fbb97fff700 1 -- 172.21.15.1:0/1606025210 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb84005210 msgr2=0x7fbb981551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.370+0000 7fbb97fff700 1 --2- 172.21.15.1:0/1606025210 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb84005210 0x7fbb981551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.370+0000 7fbb97fff700 1 -- 172.21.15.1:0/1606025210 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9815fad0 con 0x7fbb98153180 2024-03-18T19:09:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.370+0000 7fbb97fff700 1 --2- 172.21.15.1:0/1606025210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb98153180 0x7fbb98153610 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fbb980b2a50 tx=0x7fbb980cf570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.370+0000 7fbb8ffff700 1 -- 172.21.15.1:0/1606025210 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb90012ba0 con 0x7fbb98153180 2024-03-18T19:09:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.371+0000 7fbb8ffff700 1 -- 172.21.15.1:0/1606025210 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb90012d00 con 0x7fbb98153180 2024-03-18T19:09:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.371+0000 7fbb9dbbf700 1 -- 172.21.15.1:0/1606025210 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb9815fdb0 con 0x7fbb98153180 2024-03-18T19:09:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.371+0000 7fbb8ffff700 1 -- 172.21.15.1:0/1606025210 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9001e670 con 0x7fbb98153180 2024-03-18T19:09:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.371+0000 7fbb9dbbf700 1 -- 172.21.15.1:0/1606025210 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb981605b0 con 0x7fbb98153180 2024-03-18T19:09:27.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.372+0000 7fbb9dbbf700 1 -- 172.21.15.1:0/1606025210 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb980676a0 con 0x7fbb98153180 2024-03-18T19:09:27.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.372+0000 7fbb8ffff700 1 -- 172.21.15.1:0/1606025210 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbb9001e7d0 con 0x7fbb98153180 2024-03-18T19:09:27.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.372+0000 7fbb8ffff700 1 --2- 172.21.15.1:0/1606025210 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbb7806adb0 0x7fbb7806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:27.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.373+0000 7fbb8ffff700 1 -- 172.21.15.1:0/1606025210 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fbb9008b490 con 0x7fbb98153180 2024-03-18T19:09:27.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.373+0000 7fbb977fe700 1 --2- 172.21.15.1:0/1606025210 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbb7806adb0 0x7fbb7806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:27.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.373+0000 7fbb977fe700 1 --2- 172.21.15.1:0/1606025210 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbb7806adb0 0x7fbb7806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbb88000c00 tx=0x7fbb88009040 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:27.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.377+0000 7fbb8ffff700 1 -- 172.21.15.1:0/1606025210 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbb90056490 con 0x7fbb98153180 2024-03-18T19:09:27.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.666+0000 7fbb9dbbf700 1 -- 172.21.15.1:0/1606025210 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fbb9815a6f0 con 0x7fbb98153180 2024-03-18T19:09:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.669+0000 7fbb8ffff700 1 -- 172.21.15.1:0/1606025210 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+0 (secure 0 0 0) 0x7fbb90059ab0 con 0x7fbb98153180 2024-03-18T19:09:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.672+0000 7fbb9dbbf700 1 -- 172.21.15.1:0/1606025210 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbb7806adb0 msgr2=0x7fbb7806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.672+0000 7fbb9dbbf700 1 --2- 172.21.15.1:0/1606025210 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbb7806adb0 0x7fbb7806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbb88000c00 tx=0x7fbb88009040 comp rx=0 tx=0).stop 2024-03-18T19:09:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.672+0000 7fbb9dbbf700 1 -- 172.21.15.1:0/1606025210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb98153180 msgr2=0x7fbb98153610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.672+0000 7fbb9dbbf700 1 --2- 172.21.15.1:0/1606025210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb98153180 0x7fbb98153610 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fbb980b2a50 tx=0x7fbb980cf570 comp rx=0 tx=0).stop 2024-03-18T19:09:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.672+0000 7fbb9dbbf700 1 -- 172.21.15.1:0/1606025210 shutdown_connections 2024-03-18T19:09:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.672+0000 7fbb9dbbf700 1 --2- 172.21.15.1:0/1606025210 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbb7806adb0 0x7fbb7806d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.672+0000 7fbb9dbbf700 1 --2- 172.21.15.1:0/1606025210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb98153180 0x7fbb98153610 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.672+0000 7fbb9dbbf700 1 --2- 172.21.15.1:0/1606025210 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb84005210 0x7fbb981551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.672+0000 7fbb9dbbf700 1 --2- 172.21.15.1:0/1606025210 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbb9815ece0 0x7fbb981492d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.672+0000 7fbb9dbbf700 1 -- 172.21.15.1:0/1606025210 >> 172.21.15.1:0/1606025210 conn(0x7fbb980cc9b0 msgr2=0x7fbb98159b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.672+0000 7fbb9dbbf700 1 -- 172.21.15.1:0/1606025210 shutdown_connections 2024-03-18T19:09:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.672+0000 7fbb9dbbf700 1 -- 172.21.15.1:0/1606025210 wait complete. 2024-03-18T19:09:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-18T19:09:27.683 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:09:27.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.811+0000 7fcbebdee700 1 Processor -- start 2024-03-18T19:09:27.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.811+0000 7fcbebdee700 1 -- start start 2024-03-18T19:09:27.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.812+0000 7fcbebdee700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcbe40ae600 0x7fcbe40ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:27.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.812+0000 7fcbebdee700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcbe40cc170 0x7fcbe406ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:27.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.812+0000 7fcbebdee700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe406e270 con 0x7fcbe415a2e0 2024-03-18T19:09:27.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.812+0000 7fcbebdee700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe40ae050 con 0x7fcbe40cc170 2024-03-18T19:09:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.812+0000 7fcbebdee700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe40add70 con 0x7fcbe40ae600 2024-03-18T19:09:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.812+0000 7fcbea38b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcbe40cc170 0x7fcbe406ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.812+0000 7fcbe9389700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcbe415a2e0 0x7fcbe40ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51134/0 (socket says 172.21.15.1:51134) 2024-03-18T19:09:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.812+0000 7fcbea38b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcbe40cc170 0x7fcbe406ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38998/0 (socket says 172.21.15.1:38998) 2024-03-18T19:09:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.812+0000 7fcbe9389700 1 -- 172.21.15.1:0/3013144738 learned_addr learned my addr 172.21.15.1:0/3013144738 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.813+0000 7fcbe9b8a700 1 --2- 172.21.15.1:0/3013144738 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcbe40ae600 0x7fcbe40ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.813+0000 7fcbea38b700 1 -- 172.21.15.1:0/3013144738 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcbe40ae600 msgr2=0x7fcbe40ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.813+0000 7fcbea38b700 1 --2- 172.21.15.1:0/3013144738 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcbe40ae600 0x7fcbe40ad790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.813+0000 7fcbea38b700 1 -- 172.21.15.1:0/3013144738 >> v1:172.21.15.1:6789/0 conn(0x7fcbe415a2e0 legacy=0x7fcbe40ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.813+0000 7fcbea38b700 1 -- 172.21.15.1:0/3013144738 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe406e4a0 con 0x7fcbe40cc170 2024-03-18T19:09:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.813+0000 7fcbe8b88700 1 -- 172.21.15.1:0/3013144738 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405704720 0 0) 0x7fcbe406e270 con 0x7fcbe415a2e0 2024-03-18T19:09:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.813+0000 7fcbea38b700 1 --2- 172.21.15.1:0/3013144738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbe40cc170 0x7fcbe406ef80 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00bfa0 tx=0x7fcbdc00d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81effb04dcd42da0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.813+0000 7fcbe8b88700 1 -- 172.21.15.1:0/3013144738 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc010070 con 0x7fcbe40cc170 2024-03-18T19:09:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.813+0000 7fcbe8b88700 1 -- 172.21.15.1:0/3013144738 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbdc00b2e0 con 0x7fcbe40cc170 2024-03-18T19:09:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.813+0000 7fcbe8b88700 1 -- 172.21.15.1:0/3013144738 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc00dcf0 con 0x7fcbe40cc170 2024-03-18T19:09:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.814+0000 7fcbebdee700 1 -- 172.21.15.1:0/3013144738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbe40cc170 msgr2=0x7fcbe406ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.814+0000 7fcbebdee700 1 --2- 172.21.15.1:0/3013144738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbe40cc170 0x7fcbe406ef80 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00bfa0 tx=0x7fcbdc00d330 comp rx=0 tx=0).stop 2024-03-18T19:09:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.814+0000 7fcbebdee700 1 -- 172.21.15.1:0/3013144738 shutdown_connections 2024-03-18T19:09:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.814+0000 7fcbebdee700 1 --2- 172.21.15.1:0/3013144738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbe40cc170 0x7fcbe406ef80 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.814+0000 7fcbebdee700 1 --2- 172.21.15.1:0/3013144738 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcbe40ae600 0x7fcbe40ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.814+0000 7fcbebdee700 1 -- 172.21.15.1:0/3013144738 >> 172.21.15.1:0/3013144738 conn(0x7fcbe406b860 msgr2=0x7fcbe406bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.814+0000 7fcbebdee700 1 -- 172.21.15.1:0/3013144738 shutdown_connections 2024-03-18T19:09:27.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.814+0000 7fcbebdee700 1 -- 172.21.15.1:0/3013144738 wait complete. 2024-03-18T19:09:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.815+0000 7fcbebdee700 1 Processor -- start 2024-03-18T19:09:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.815+0000 7fcbebdee700 1 -- start start 2024-03-18T19:09:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.816+0000 7fcbebdee700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcbe40ae600 0x7fcbe4150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.816+0000 7fcbebdee700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbe40cc170 0x7fcbe40ce790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.816+0000 7fcbebdee700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcbe415a2e0 0x7fcbe4153fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.816+0000 7fcbebdee700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe40cf070 con 0x7fcbe40cc170 2024-03-18T19:09:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.816+0000 7fcbebdee700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe40cf350 con 0x7fcbe40ae600 2024-03-18T19:09:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.816+0000 7fcbebdee700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe40cf630 con 0x7fcbe415a2e0 2024-03-18T19:09:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.816+0000 7fcbe9389700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbe40cc170 0x7fcbe40ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.816+0000 7fcbe9389700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbe40cc170 0x7fcbe40ce790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39000/0 (socket says 172.21.15.1:39000) 2024-03-18T19:09:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.816+0000 7fcbe9389700 1 -- 172.21.15.1:0/1190431136 learned_addr learned my addr 172.21.15.1:0/1190431136 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.816+0000 7fcbe9b8a700 1 --2- 172.21.15.1:0/1190431136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcbe40ae600 0x7fcbe4150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.816+0000 7fcbea38b700 1 --2- 172.21.15.1:0/1190431136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcbe415a2e0 0x7fcbe4153fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.816+0000 7fcbe9389700 1 -- 172.21.15.1:0/1190431136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcbe415a2e0 msgr2=0x7fcbe4153fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.816+0000 7fcbe9389700 1 --2- 172.21.15.1:0/1190431136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcbe415a2e0 0x7fcbe4153fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.816+0000 7fcbe9389700 1 -- 172.21.15.1:0/1190431136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcbe40ae600 msgr2=0x7fcbe4150c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.816+0000 7fcbe9389700 1 --2- 172.21.15.1:0/1190431136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcbe40ae600 0x7fcbe4150c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.816+0000 7fcbe9389700 1 -- 172.21.15.1:0/1190431136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbdc00bc80 con 0x7fcbe40cc170 2024-03-18T19:09:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.816+0000 7fcbe9389700 1 --2- 172.21.15.1:0/1190431136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbe40cc170 0x7fcbe40ce790 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fcbe000c520 tx=0x7fcbe000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.816+0000 7fcbda7fc700 1 -- 172.21.15.1:0/1190431136 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0004c80 con 0x7fcbe40cc170 2024-03-18T19:09:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.817+0000 7fcbda7fc700 1 -- 172.21.15.1:0/1190431136 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe0004de0 con 0x7fcbe40cc170 2024-03-18T19:09:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.817+0000 7fcbebdee700 1 -- 172.21.15.1:0/1190431136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe40cf910 con 0x7fcbe40cc170 2024-03-18T19:09:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.817+0000 7fcbda7fc700 1 -- 172.21.15.1:0/1190431136 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe00072d0 con 0x7fcbe40cc170 2024-03-18T19:09:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.817+0000 7fcbebdee700 1 -- 172.21.15.1:0/1190431136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe40cecd0 con 0x7fcbe40cc170 2024-03-18T19:09:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.818+0000 7fcbebdee700 1 -- 172.21.15.1:0/1190431136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe40676a0 con 0x7fcbe40cc170 2024-03-18T19:09:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.818+0000 7fcbda7fc700 1 -- 172.21.15.1:0/1190431136 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcbe0007430 con 0x7fcbe40cc170 2024-03-18T19:09:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.819+0000 7fcbda7fc700 1 --2- 172.21.15.1:0/1190431136 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcbc806adb0 0x7fcbc806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.819+0000 7fcbe9b8a700 1 --2- 172.21.15.1:0/1190431136 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcbc806adb0 0x7fcbc806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.819+0000 7fcbda7fc700 1 -- 172.21.15.1:0/1190431136 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fcbe00891f0 con 0x7fcbe40cc170 2024-03-18T19:09:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.819+0000 7fcbe9b8a700 1 --2- 172.21.15.1:0/1190431136 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcbc806adb0 0x7fcbc806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcbe41f2370 tx=0x7fcbd4008040 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:27.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:27.823+0000 7fcbda7fc700 1 -- 172.21.15.1:0/1190431136 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcbe00540c0 con 0x7fcbe40cc170 2024-03-18T19:09:28.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.071+0000 7fcbebdee700 1 -- 172.21.15.1:0/1190431136 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbe40b46b0 con 0x7fcbe40cc170 2024-03-18T19:09:28.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.071+0000 7fcbda7fc700 1 -- 172.21.15.1:0/1190431136 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+4727 (secure 0 0 0) 0x7fcbe00576e0 con 0x7fcbe40cc170 2024-03-18T19:09:28.067 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:28.068 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8116,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3723189348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3723189348},{"type":"v1","addr":"172.21.15.1:6837","nonce":3723189348}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":8125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/530206363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":530206363},{"type":"v1","addr":"172.21.15.165:6835","nonce":530206363}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":8784,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/332827595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":332827595},{"type":"v1","addr":"172.21.15.1:6836","nonce":332827595}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:09:22.623415+0000","modified":"2024-03-18T19:09:25.640243+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6834/311436411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":311436411},{"type":"v1","addr":"172.21.15.165:6834","nonce":311436411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-18T19:09:28.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.074+0000 7fcbebdee700 1 -- 172.21.15.1:0/1190431136 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcbc806adb0 msgr2=0x7fcbc806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:28.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.074+0000 7fcbebdee700 1 --2- 172.21.15.1:0/1190431136 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcbc806adb0 0x7fcbc806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcbe41f2370 tx=0x7fcbd4008040 comp rx=0 tx=0).stop 2024-03-18T19:09:28.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.074+0000 7fcbebdee700 1 -- 172.21.15.1:0/1190431136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbe40cc170 msgr2=0x7fcbe40ce790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:28.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.074+0000 7fcbebdee700 1 --2- 172.21.15.1:0/1190431136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbe40cc170 0x7fcbe40ce790 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fcbe000c520 tx=0x7fcbe000c830 comp rx=0 tx=0).stop 2024-03-18T19:09:28.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.074+0000 7fcbebdee700 1 -- 172.21.15.1:0/1190431136 shutdown_connections 2024-03-18T19:09:28.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.074+0000 7fcbebdee700 1 --2- 172.21.15.1:0/1190431136 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcbc806adb0 0x7fcbc806d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.074+0000 7fcbebdee700 1 --2- 172.21.15.1:0/1190431136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcbe415a2e0 0x7fcbe4153fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.074+0000 7fcbebdee700 1 --2- 172.21.15.1:0/1190431136 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcbe40ae600 0x7fcbe4150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.074+0000 7fcbebdee700 1 --2- 172.21.15.1:0/1190431136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbe40cc170 0x7fcbe40ce790 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.074+0000 7fcbebdee700 1 -- 172.21.15.1:0/1190431136 >> 172.21.15.1:0/1190431136 conn(0x7fcbe406b860 msgr2=0x7fcbe4154ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:28.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.074+0000 7fcbebdee700 1 -- 172.21.15.1:0/1190431136 shutdown_connections 2024-03-18T19:09:28.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.074+0000 7fcbebdee700 1 -- 172.21.15.1:0/1190431136 wait complete. 2024-03-18T19:09:28.071 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 65 2024-03-18T19:09:28.181 INFO:tasks.ceph.mds.b.smithi165.stderr:2024-03-18T19:09:28.185+0000 7fdf146d4700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T19:09:28.484 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 65, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T19:09:22.623415+0000', 'modified': '2024-03-18T19:09:25.640243+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8802}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8802': {'gid': 8802, 'name': 'b', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.165:6834/311436411', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6832', 'nonce': 311436411}, {'type': 'v1', 'addr': '172.21.15.165:6834', 'nonce': 311436411}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T19:09:28.484 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T19:09:28.485 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T19:09:28.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.622+0000 7f8ff8865700 1 Processor -- start 2024-03-18T19:09:28.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.622+0000 7f8ff8865700 1 -- start start 2024-03-18T19:09:28.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff8865700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff4071d30 0x7f8ff40add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff8865700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8ff40cb9e0 0x7f8ff40cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff8865700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff40ce8f0 con 0x7f8ff415a1d0 2024-03-18T19:09:28.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff8865700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff40cdb00 con 0x7f8ff40cb9e0 2024-03-18T19:09:28.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff8865700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff415a5b0 con 0x7f8ff4071d30 2024-03-18T19:09:28.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff259c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8ff415a1d0 0x7f8ff4071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51138/0 (socket says 172.21.15.1:51138) 2024-03-18T19:09:28.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff259c700 1 -- 172.21.15.1:0/3841638879 learned_addr learned my addr 172.21.15.1:0/3841638879 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:28.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff1d9b700 1 --2- 172.21.15.1:0/3841638879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff4071d30 0x7f8ff40add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff2d9d700 1 --2- 172.21.15.1:0/3841638879 >> v2:172.21.15.165:3300/0 conn(0x7f8ff40cb9e0 0x7f8ff40cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff159a700 1 -- 172.21.15.1:0/3841638879 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494930580 0 0) 0x7f8ff40ce8f0 con 0x7f8ff415a1d0 2024-03-18T19:09:28.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff159a700 1 -- 172.21.15.1:0/3841638879 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe00035b0 con 0x7f8ff415a1d0 2024-03-18T19:09:28.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff159a700 1 -- 172.21.15.1:0/3841638879 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3336030120 0 0) 0x7f8fe00035b0 con 0x7f8ff415a1d0 2024-03-18T19:09:28.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff159a700 1 -- 172.21.15.1:0/3841638879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff4071d30 msgr2=0x7f8ff40add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:28.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff159a700 1 --2- 172.21.15.1:0/3841638879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff4071d30 0x7f8ff40add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff159a700 1 -- 172.21.15.1:0/3841638879 >> v2:172.21.15.165:3300/0 conn(0x7f8ff40cb9e0 msgr2=0x7f8ff40cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:28.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff159a700 1 --2- 172.21.15.1:0/3841638879 >> v2:172.21.15.165:3300/0 conn(0x7f8ff40cb9e0 0x7f8ff40cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff159a700 1 -- 172.21.15.1:0/3841638879 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff40ae260 con 0x7f8ff415a1d0 2024-03-18T19:09:28.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff159a700 1 -- 172.21.15.1:0/3841638879 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f8fe40017f0 con 0x7f8ff415a1d0 2024-03-18T19:09:28.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff159a700 1 -- 172.21.15.1:0/3841638879 >> v1:172.21.15.1:6789/0 conn(0x7f8ff415a1d0 legacy=0x7f8ff4071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:28.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff159a700 1 --2- 172.21.15.1:0/3841638879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fe0003a00 0x7f8fe0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff159a700 1 --2- 172.21.15.1:0/3841638879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff4071d30 0x7f8fe0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff159a700 1 --2- 172.21.15.1:0/3841638879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8fe0004d90 0x7f8fe0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff159a700 1 -- 172.21.15.1:0/3841638879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40cdb00 con 0x7f8fe0003a00 2024-03-18T19:09:28.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff159a700 1 -- 172.21.15.1:0/3841638879 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe000ba10 con 0x7f8fe0004d90 2024-03-18T19:09:28.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.623+0000 7f8ff159a700 1 -- 172.21.15.1:0/3841638879 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe000bb50 con 0x7f8ff4071d30 2024-03-18T19:09:28.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.624+0000 7f8ff259c700 1 --2- 172.21.15.1:0/3841638879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fe0003a00 0x7f8fe0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.624+0000 7f8ff1d9b700 1 --2- 172.21.15.1:0/3841638879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff4071d30 0x7f8fe0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.624+0000 7f8ff2d9d700 1 --2- 172.21.15.1:0/3841638879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8fe0004d90 0x7f8fe0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.624+0000 7f8ff259c700 1 -- 172.21.15.1:0/3841638879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff4071d30 msgr2=0x7f8fe0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:28.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.624+0000 7f8ff259c700 1 --2- 172.21.15.1:0/3841638879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff4071d30 0x7f8fe0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.624+0000 7f8ff259c700 1 -- 172.21.15.1:0/3841638879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8fe0004d90 msgr2=0x7f8fe0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:28.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.624+0000 7f8ff259c700 1 --2- 172.21.15.1:0/3841638879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8fe0004d90 0x7f8fe0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.624+0000 7f8ff259c700 1 -- 172.21.15.1:0/3841638879 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe00035b0 con 0x7f8fe0003a00 2024-03-18T19:09:28.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.624+0000 7f8ff259c700 1 --2- 172.21.15.1:0/3841638879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fe0003a00 0x7f8fe0003e20 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400c240 tx=0x7f8fe0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8640d531fbd9d40 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:28.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.624+0000 7f8ff159a700 1 -- 172.21.15.1:0/3841638879 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe400cde0 con 0x7f8fe0003a00 2024-03-18T19:09:28.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.624+0000 7f8ff159a700 1 -- 172.21.15.1:0/3841638879 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe4007bd0 con 0x7f8fe0003a00 2024-03-18T19:09:28.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.624+0000 7f8ff159a700 1 -- 172.21.15.1:0/3841638879 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4007310 con 0x7f8fe0003a00 2024-03-18T19:09:28.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.625+0000 7f8ff8865700 1 -- 172.21.15.1:0/3841638879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fe0003a00 msgr2=0x7f8fe0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:28.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.625+0000 7f8ff8865700 1 --2- 172.21.15.1:0/3841638879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fe0003a00 0x7f8fe0003e20 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400c240 tx=0x7f8fe0005880 comp rx=0 tx=0).stop 2024-03-18T19:09:28.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.625+0000 7f8ff2d9d700 1 -- 172.21.15.1:0/3841638879 reap_dead start 2024-03-18T19:09:28.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.625+0000 7f8ff8865700 1 -- 172.21.15.1:0/3841638879 shutdown_connections 2024-03-18T19:09:28.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.625+0000 7f8ff8865700 1 -- 172.21.15.1:0/3841638879 >> 172.21.15.1:0/3841638879 conn(0x7f8ff40cc9b0 msgr2=0x7f8ff40ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:28.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.625+0000 7f8ff8865700 1 -- 172.21.15.1:0/3841638879 shutdown_connections 2024-03-18T19:09:28.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.625+0000 7f8ff8865700 1 -- 172.21.15.1:0/3841638879 wait complete. 2024-03-18T19:09:28.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.626+0000 7f8ff8865700 1 Processor -- start 2024-03-18T19:09:28.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.626+0000 7f8ff8865700 1 -- start start 2024-03-18T19:09:28.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.627+0000 7f8ff8865700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8fe0003a00 0x7f8ff4148090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.627+0000 7f8ff8865700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff4148760 0x7f8ff415a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.627+0000 7f8ff8865700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ff40b5180 0x7f8ff40b5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.627+0000 7f8ff8865700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff415c260 con 0x7f8ff4148760 2024-03-18T19:09:28.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.627+0000 7f8ff8865700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40b5b70 con 0x7f8ff40b5180 2024-03-18T19:09:28.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.627+0000 7f8ff8865700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff415b1c0 con 0x7f8fe0003a00 2024-03-18T19:09:28.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.627+0000 7f8ff1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff4148760 0x7f8ff415a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.627+0000 7f8ff1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff4148760 0x7f8ff415a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39028/0 (socket says 172.21.15.1:39028) 2024-03-18T19:09:28.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.627+0000 7f8ff1d9b700 1 -- 172.21.15.1:0/658860395 learned_addr learned my addr 172.21.15.1:0/658860395 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:28.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.627+0000 7f8ff2d9d700 1 --2- 172.21.15.1:0/658860395 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ff40b5180 0x7f8ff40b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.627+0000 7f8ff259c700 1 --2- 172.21.15.1:0/658860395 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8fe0003a00 0x7f8ff4148090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.627+0000 7f8ff1d9b700 1 -- 172.21.15.1:0/658860395 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8fe0003a00 msgr2=0x7f8ff4148090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:28.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.627+0000 7f8ff1d9b700 1 --2- 172.21.15.1:0/658860395 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8fe0003a00 0x7f8ff4148090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.627+0000 7f8ff1d9b700 1 -- 172.21.15.1:0/658860395 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ff40b5180 msgr2=0x7f8ff40b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:28.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.627+0000 7f8ff1d9b700 1 --2- 172.21.15.1:0/658860395 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ff40b5180 0x7f8ff40b5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.627+0000 7f8ff1d9b700 1 -- 172.21.15.1:0/658860395 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff415b4a0 con 0x7f8ff4148760 2024-03-18T19:09:28.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.628+0000 7f8ff1d9b700 1 --2- 172.21.15.1:0/658860395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff4148760 0x7f8ff415a850 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc0119a0 tx=0x7f8fdc011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:28.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.628+0000 7f8feaffd700 1 -- 172.21.15.1:0/658860395 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc00f780 con 0x7f8ff4148760 2024-03-18T19:09:28.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.628+0000 7f8feaffd700 1 -- 172.21.15.1:0/658860395 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc00f8e0 con 0x7f8ff4148760 2024-03-18T19:09:28.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.628+0000 7f8ff8865700 1 -- 172.21.15.1:0/658860395 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff415b780 con 0x7f8ff4148760 2024-03-18T19:09:28.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.628+0000 7f8feaffd700 1 -- 172.21.15.1:0/658860395 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc00fc70 con 0x7f8ff4148760 2024-03-18T19:09:28.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.628+0000 7f8ff8865700 1 -- 172.21.15.1:0/658860395 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff415bd40 con 0x7f8ff4148760 2024-03-18T19:09:28.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.629+0000 7f8feaffd700 1 -- 172.21.15.1:0/658860395 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8fdc014440 con 0x7f8ff4148760 2024-03-18T19:09:28.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.630+0000 7f8feaffd700 1 --2- 172.21.15.1:0/658860395 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8fd406adb0 0x7f8fd406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.630+0000 7f8feaffd700 1 -- 172.21.15.1:0/658860395 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f8fdc089c00 con 0x7f8ff4148760 2024-03-18T19:09:28.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.630+0000 7f8ff259c700 1 --2- 172.21.15.1:0/658860395 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8fd406adb0 0x7f8fd406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.630+0000 7f8ff259c700 1 --2- 172.21.15.1:0/658860395 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8fd406adb0 0x7f8fd406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4000ef0 tx=0x7f8fe400a000 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:28.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.632+0000 7f9004a0eb80 1 Processor -- start 2024-03-18T19:09:28.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.632+0000 7f9004a0eb80 1 -- start start 2024-03-18T19:09:28.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.632+0000 7f9004a0eb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1ebc10 0x56194f1ebff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.632+0000 7f9004a0eb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1bd0a0 0x56194f208580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.632+0000 7f9004a0eb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1fd630 0x56194f1c0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.632+0000 7f9004a0eb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194f136f20 con 0x56194f1ebc10 2024-03-18T19:09:28.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.632+0000 7f9004a0eb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194f118c30 con 0x56194f1bd0a0 2024-03-18T19:09:28.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.632+0000 7f9004a0eb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x56194f116430 con 0x56194f1fd630 2024-03-18T19:09:28.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.633+0000 7f8ff259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1ebc10 0x56194f1ebff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.633+0000 7f8ff259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1ebc10 0x56194f1ebff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39036/0 (socket says 172.21.15.1:39036) 2024-03-18T19:09:28.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.633+0000 7f8ff259c700 1 -- 172.21.15.1:0/41988780 learned_addr learned my addr 172.21.15.1:0/41988780 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:28.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.633+0000 7f8ff1d9b700 1 --2- 172.21.15.1:0/41988780 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1bd0a0 0x56194f208580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.633+0000 7f8ff2d9d700 1 --2- 172.21.15.1:0/41988780 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1fd630 0x56194f1c0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.633+0000 7f8ff259c700 1 -- 172.21.15.1:0/41988780 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1fd630 msgr2=0x56194f1c0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:28.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.633+0000 7f8ff259c700 1 --2- 172.21.15.1:0/41988780 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1fd630 0x56194f1c0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.633+0000 7f8ff259c700 1 -- 172.21.15.1:0/41988780 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1bd0a0 msgr2=0x56194f208580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:28.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.633+0000 7f8ff259c700 1 --2- 172.21.15.1:0/41988780 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1bd0a0 0x56194f208580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.633+0000 7f8ff259c700 1 -- 172.21.15.1:0/41988780 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56194ef05790 con 0x56194f1ebc10 2024-03-18T19:09:28.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.633+0000 7f8ff259c700 1 --2- 172.21.15.1:0/41988780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1ebc10 0x56194f1ebff0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400c190 tx=0x7f8fe400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49977b56c9e83c96 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:28.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.633+0000 7f8fd3fff700 1 -- 172.21.15.1:0/41988780 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4005e90 con 0x56194f1ebc10 2024-03-18T19:09:28.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.633+0000 7f8fd3fff700 1 -- 172.21.15.1:0/41988780 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe4023070 con 0x56194f1ebc10 2024-03-18T19:09:28.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.633+0000 7f8fd3fff700 1 -- 172.21.15.1:0/41988780 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe40072d0 con 0x56194f1ebc10 2024-03-18T19:09:28.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.634+0000 7f9004a0eb80 1 -- 172.21.15.1:0/41988780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1ebc10 msgr2=0x56194f1ebff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:28.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.634+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/41988780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1ebc10 0x56194f1ebff0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400c190 tx=0x7f8fe400f000 comp rx=0 tx=0).stop 2024-03-18T19:09:28.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.634+0000 7f9004a0eb80 1 -- 172.21.15.1:0/41988780 shutdown_connections 2024-03-18T19:09:28.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.634+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/41988780 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1fd630 0x56194f1c0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.634+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/41988780 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1ebc10 0x56194f1ebff0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.634+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/41988780 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1bd0a0 0x56194f208580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.634+0000 7f9004a0eb80 1 -- 172.21.15.1:0/41988780 >> 172.21.15.1:0/41988780 conn(0x56194f11fcf0 msgr2=0x56194f1bfa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:28.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.634+0000 7f9004a0eb80 1 -- 172.21.15.1:0/41988780 shutdown_connections 2024-03-18T19:09:28.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.634+0000 7f9004a0eb80 1 -- 172.21.15.1:0/41988780 wait complete. 2024-03-18T19:09:28.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.635+0000 7f9004a0eb80 1 -- start start 2024-03-18T19:09:28.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.635+0000 7f9004a0eb80 1 Processor -- start 2024-03-18T19:09:28.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f9004a0eb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1bd0a0 0x56194f205ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f9004a0eb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1ebc10 0x56194f206430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f9004a0eb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1fd630 0x56194f1eefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f9004a0eb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194f111d20 con 0x56194f1fd630 2024-03-18T19:09:28.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f9004a0eb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194f0ad770 con 0x56194f1ebc10 2024-03-18T19:09:28.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f9004a0eb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x56194f0d06f0 con 0x56194f1bd0a0 2024-03-18T19:09:28.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f9004a0eb80 10 client.0 _ll_register_callbacks cb 0x56194ef195a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T19:09:28.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f8ff2d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1fd630 0x56194f1eefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f8ff2d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1fd630 0x56194f1eefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39038/0 (socket says 172.21.15.1:39038) 2024-03-18T19:09:28.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f8ff259c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1bd0a0 0x56194f205ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f8ff2d9d700 1 -- 172.21.15.1:0/2418719260 learned_addr learned my addr 172.21.15.1:0/2418719260 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:28.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f8ff1d9b700 1 --2- 172.21.15.1:0/2418719260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1ebc10 0x56194f206430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f8ff2d9d700 1 -- 172.21.15.1:0/2418719260 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1bd0a0 msgr2=0x56194f205ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:28.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f8ff2d9d700 1 --2- 172.21.15.1:0/2418719260 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1bd0a0 0x56194f205ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f8ff2d9d700 1 -- 172.21.15.1:0/2418719260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1ebc10 msgr2=0x56194f206430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:28.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f8ff2d9d700 1 --2- 172.21.15.1:0/2418719260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1ebc10 0x56194f206430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f8ff2d9d700 1 -- 172.21.15.1:0/2418719260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe00035b0 con 0x56194f1fd630 2024-03-18T19:09:28.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f8ff2d9d700 1 --2- 172.21.15.1:0/2418719260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1fd630 0x56194f1eefe0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f8ff409df60 tx=0x7f8fec014600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65fb02146ca0c847 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:28.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f8fd17fa700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T19:09:28.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f8fd17fa700 1 -- 172.21.15.1:0/2418719260 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec00e040 con 0x56194f1fd630 2024-03-18T19:09:28.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f8fd17fa700 1 -- 172.21.15.1:0/2418719260 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(147..147 src has 1..147) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f8fec014ce0 con 0x56194f1fd630 2024-03-18T19:09:28.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.636+0000 7f9004a0eb80 1 -- 172.21.15.1:0/2418719260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56194f1b3e90 con 0x56194f1fd630 2024-03-18T19:09:28.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.637+0000 7f8fd17fa700 1 -- 172.21.15.1:0/2418719260 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fec016eb0 con 0x56194f1fd630 2024-03-18T19:09:28.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.637+0000 7f9004a0eb80 10 client.8949 fetch_fsmap learned FSMap version 65 2024-03-18T19:09:28.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.637+0000 7f9004a0eb80 1 -- 172.21.15.1:0/2418719260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=2+}) v3 -- 0x56194eff20a0 con 0x56194f1fd630 2024-03-18T19:09:28.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.637+0000 7f8fd17fa700 1 -- 172.21.15.1:0/2418719260 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56194f1b3e90 con 0x56194f1fd630 2024-03-18T19:09:28.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.637+0000 7f8fd17fa700 1 -- 172.21.15.1:0/2418719260 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 65) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f8fec01f030 con 0x56194f1fd630 2024-03-18T19:09:28.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.637+0000 7f9004a0eb80 10 client.8949 fetch_fsmap finished waiting for FSMap version 65 2024-03-18T19:09:28.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.637+0000 7f9004a0eb80 10 client.8949 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T19:09:28.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.637+0000 7f9004a0eb80 20 client.8949 populate_metadata read hostname 'smithi001' 2024-03-18T19:09:28.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.637+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/2418719260 >> [v2:172.21.15.165:6832/311436411,v1:172.21.15.165:6834/311436411] conn(0x56194f1c02c0 0x56194f1c06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.637+0000 7f9004a0eb80 4 client.8949 mds_command: new command op to 8802 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T19:09:28.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.637+0000 7f9004a0eb80 1 -- 172.21.15.1:0/2418719260 --> [v2:172.21.15.165:6832/311436411,v1:172.21.15.165:6834/311436411] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56194f191010 con 0x56194f1c02c0 2024-03-18T19:09:28.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.637+0000 7f8ff259c700 1 --2- 172.21.15.1:0/2418719260 >> [v2:172.21.15.165:6832/311436411,v1:172.21.15.165:6834/311436411] conn(0x56194f1c02c0 0x56194f1c06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.637+0000 7f8fd17fa700 10 client.8949 ms_handle_connect on v2:172.21.15.165:6832/311436411 2024-03-18T19:09:28.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.637+0000 7f8ff259c700 1 --2- 172.21.15.1:0/2418719260 >> [v2:172.21.15.165:6832/311436411,v1:172.21.15.165:6834/311436411] conn(0x56194f1c02c0 0x56194f1c06a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df91cdfb3c90b304 server_cookie=74f22d5ebb3a7aaa in_seq=0 out_seq=0 2024-03-18T19:09:28.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.638+0000 7f8fd17fa700 1 -- 172.21.15.1:0/2418719260 <== mds.0 v2:172.21.15.165:6832/311436411 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56194f191010 con 0x56194f1c02c0 2024-03-18T19:09:28.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.638+0000 7f8fd17fa700 10 client.8949 handle_command_reply: tid=0 2024-03-18T19:09:28.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.638+0000 7f9004a0eb80 1 client.8949 shutdown 2024-03-18T19:09:28.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.638+0000 7f9004a0eb80 1 -- 172.21.15.1:0/2418719260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1fd630 msgr2=0x56194f1eefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:28.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.638+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/2418719260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1fd630 0x56194f1eefe0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f8ff409df60 tx=0x7f8fec014600 comp rx=0 tx=0).stop 2024-03-18T19:09:28.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.638+0000 7f9004a0eb80 1 -- 172.21.15.1:0/2418719260 shutdown_connections 2024-03-18T19:09:28.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.638+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/2418719260 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1bd0a0 0x56194f205ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.638+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/2418719260 >> [v2:172.21.15.165:6832/311436411,v1:172.21.15.165:6834/311436411] conn(0x56194f1c02c0 0x56194f1c06a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.638+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/2418719260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1fd630 0x56194f1eefe0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.638+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/2418719260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1ebc10 0x56194f206430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.638+0000 7f8fd17fa700 0 client.8949 ms_handle_reset on v2:172.21.15.165:6832/311436411 2024-03-18T19:09:28.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.638+0000 7f9004a0eb80 1 -- 172.21.15.1:0/2418719260 >> 172.21.15.1:0/2418719260 conn(0x56194f11fcf0 msgr2=0x56194f1e9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:28.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.638+0000 7f9004a0eb80 1 -- 172.21.15.1:0/2418719260 shutdown_connections 2024-03-18T19:09:28.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.638+0000 7f9004a0eb80 1 -- 172.21.15.1:0/2418719260 wait complete. 2024-03-18T19:09:28.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.639+0000 7f9004a0eb80 20 client.8949 trim_cache size 0 max 16384 2024-03-18T19:09:28.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.656+0000 7f9004a0eb80 1 Processor -- start 2024-03-18T19:09:28.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.656+0000 7f9004a0eb80 1 -- start start 2024-03-18T19:09:28.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f9004a0eb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1bd0a0 0x56194f1e69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f9004a0eb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1ebc10 0x56194f217150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f9004a0eb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1fd630 0x56194f217690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f9004a0eb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194ef06390 con 0x56194f1ebc10 2024-03-18T19:09:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f9004a0eb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194f1be0e0 con 0x56194f1bd0a0 2024-03-18T19:09:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f9004a0eb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x56194f1eb420 con 0x56194f1fd630 2024-03-18T19:09:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f8ff1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1ebc10 0x56194f217150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f8ff1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1ebc10 0x56194f217150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39054/0 (socket says 172.21.15.1:39054) 2024-03-18T19:09:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f8ff1d9b700 1 -- 172.21.15.1:0/1647259404 learned_addr learned my addr 172.21.15.1:0/1647259404 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f8ff2d9d700 1 --2- 172.21.15.1:0/1647259404 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1fd630 0x56194f217690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f8ff259c700 1 --2- 172.21.15.1:0/1647259404 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1bd0a0 0x56194f1e69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f8ff1d9b700 1 -- 172.21.15.1:0/1647259404 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1fd630 msgr2=0x56194f217690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f8ff1d9b700 1 --2- 172.21.15.1:0/1647259404 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1fd630 0x56194f217690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f8ff1d9b700 1 -- 172.21.15.1:0/1647259404 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1bd0a0 msgr2=0x56194f1e69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f8ff1d9b700 1 --2- 172.21.15.1:0/1647259404 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1bd0a0 0x56194f1e69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f8ff1d9b700 1 -- 172.21.15.1:0/1647259404 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56194f1eff00 con 0x56194f1ebc10 2024-03-18T19:09:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f8ff259c700 1 --2- 172.21.15.1:0/1647259404 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1bd0a0 0x56194f1e69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:28.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f8ff2d9d700 1 --2- 172.21.15.1:0/1647259404 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1fd630 0x56194f217690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:28.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f8ff1d9b700 1 --2- 172.21.15.1:0/1647259404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1ebc10 0x56194f217150 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc054d30 tx=0x7f8fdc058950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f696110a1233acb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:28.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.657+0000 7f8fe8ff9700 1 -- 172.21.15.1:0/1647259404 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc060c20 con 0x56194f1ebc10 2024-03-18T19:09:28.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.658+0000 7f8fe8ff9700 1 -- 172.21.15.1:0/1647259404 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc056bd0 con 0x56194f1ebc10 2024-03-18T19:09:28.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.658+0000 7f8fe8ff9700 1 -- 172.21.15.1:0/1647259404 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc01b300 con 0x56194f1ebc10 2024-03-18T19:09:28.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.658+0000 7f9004a0eb80 1 -- 172.21.15.1:0/1647259404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1ebc10 msgr2=0x56194f217150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:28.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.658+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/1647259404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1ebc10 0x56194f217150 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc054d30 tx=0x7f8fdc058950 comp rx=0 tx=0).stop 2024-03-18T19:09:28.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.658+0000 7f9004a0eb80 1 -- 172.21.15.1:0/1647259404 shutdown_connections 2024-03-18T19:09:28.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.658+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/1647259404 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1fd630 0x56194f217690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.658+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/1647259404 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1bd0a0 0x56194f1e69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.658+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/1647259404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1ebc10 0x56194f217150 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.658+0000 7f9004a0eb80 1 -- 172.21.15.1:0/1647259404 >> 172.21.15.1:0/1647259404 conn(0x56194f11fcf0 msgr2=0x56194f1e8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:28.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.658+0000 7f9004a0eb80 1 -- 172.21.15.1:0/1647259404 shutdown_connections 2024-03-18T19:09:28.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.658+0000 7f9004a0eb80 1 -- 172.21.15.1:0/1647259404 wait complete. 2024-03-18T19:09:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.659+0000 7f9004a0eb80 1 -- start start 2024-03-18T19:09:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.659+0000 7f9004a0eb80 1 Processor -- start 2024-03-18T19:09:28.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.660+0000 7f9004a0eb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1bd0a0 0x56194f217550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.660+0000 7f9004a0eb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1ebc10 0x56194f1c06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.660+0000 7f9004a0eb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1fd630 0x56194f1c0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.660+0000 7f9004a0eb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194f1fd410 con 0x56194f1bd0a0 2024-03-18T19:09:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.660+0000 7f9004a0eb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194f1c1160 con 0x56194f1fd630 2024-03-18T19:09:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.660+0000 7f9004a0eb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x56194f1c13b0 con 0x56194f1ebc10 2024-03-18T19:09:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.660+0000 7f9004a0eb80 10 client.0 _ll_register_callbacks cb 0x7f8fec016740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T19:09:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.660+0000 7f8ff259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1bd0a0 0x56194f217550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.660+0000 7f8ff259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1bd0a0 0x56194f217550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39066/0 (socket says 172.21.15.1:39066) 2024-03-18T19:09:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.660+0000 7f8ff259c700 1 -- 172.21.15.1:0/2404631714 learned_addr learned my addr 172.21.15.1:0/2404631714 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.660+0000 7f8ff1d9b700 1 --2- 172.21.15.1:0/2404631714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1ebc10 0x56194f1c06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.660+0000 7f8ff2d9d700 1 --2- 172.21.15.1:0/2404631714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1fd630 0x56194f1c0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f8ff259c700 1 -- 172.21.15.1:0/2404631714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1ebc10 msgr2=0x56194f1c06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f8ff259c700 1 --2- 172.21.15.1:0/2404631714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1ebc10 0x56194f1c06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f8ff259c700 1 -- 172.21.15.1:0/2404631714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1fd630 msgr2=0x56194f1c0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f8ff259c700 1 --2- 172.21.15.1:0/2404631714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1fd630 0x56194f1c0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f8ff259c700 1 -- 172.21.15.1:0/2404631714 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff415bd40 con 0x56194f1bd0a0 2024-03-18T19:09:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f8ff259c700 1 --2- 172.21.15.1:0/2404631714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1bd0a0 0x56194f217550 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4015040 tx=0x7f8fe4009050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=264c3b6a947156de server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f8fd17fa700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T19:09:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f8fd17fa700 1 -- 172.21.15.1:0/2404631714 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4038040 con 0x56194f1bd0a0 2024-03-18T19:09:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f8fd17fa700 1 -- 172.21.15.1:0/2404631714 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(147..147 src has 1..147) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f8fe403f420 con 0x56194f1bd0a0 2024-03-18T19:09:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f9004a0eb80 1 -- 172.21.15.1:0/2404631714 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8fe4023070 con 0x56194f1bd0a0 2024-03-18T19:09:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f8fd17fa700 1 -- 172.21.15.1:0/2404631714 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fe4001870 con 0x56194f1bd0a0 2024-03-18T19:09:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f9004a0eb80 10 client.8955 fetch_fsmap learned FSMap version 65 2024-03-18T19:09:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f9004a0eb80 1 -- 172.21.15.1:0/2404631714 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=2+}) v3 -- 0x56194f0d8010 con 0x56194f1bd0a0 2024-03-18T19:09:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f8fd17fa700 1 -- 172.21.15.1:0/2404631714 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe4023070 con 0x56194f1bd0a0 2024-03-18T19:09:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f8fd17fa700 1 -- 172.21.15.1:0/2404631714 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 65) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f8fe403f980 con 0x56194f1bd0a0 2024-03-18T19:09:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f9004a0eb80 10 client.8955 fetch_fsmap finished waiting for FSMap version 65 2024-03-18T19:09:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f9004a0eb80 10 client.8955 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T19:09:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f9004a0eb80 20 client.8955 populate_metadata read hostname 'smithi001' 2024-03-18T19:09:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/2404631714 >> [v2:172.21.15.165:6832/311436411,v1:172.21.15.165:6834/311436411] conn(0x56194f1fc3b0 0x56194f1fc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f9004a0eb80 4 client.8955 mds_command: new command op to 8802 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T19:09:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.661+0000 7f9004a0eb80 1 -- 172.21.15.1:0/2404631714 --> [v2:172.21.15.165:6832/311436411,v1:172.21.15.165:6834/311436411] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56194ef4b040 con 0x56194f1fc3b0 2024-03-18T19:09:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.662+0000 7f8ff1d9b700 1 --2- 172.21.15.1:0/2404631714 >> [v2:172.21.15.165:6832/311436411,v1:172.21.15.165:6834/311436411] conn(0x56194f1fc3b0 0x56194f1fc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.662+0000 7f8ff1d9b700 1 --2- 172.21.15.1:0/2404631714 >> [v2:172.21.15.165:6832/311436411,v1:172.21.15.165:6834/311436411] conn(0x56194f1fc3b0 0x56194f1fc790 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39d50970bb6a4c49 server_cookie=70d5ebd49585a4e9 in_seq=0 out_seq=0 2024-03-18T19:09:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.662+0000 7f8fd17fa700 10 client.8955 ms_handle_connect on v2:172.21.15.165:6832/311436411 2024-03-18T19:09:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.662+0000 7f8fd17fa700 1 -- 172.21.15.1:0/2404631714 <== mds.0 v2:172.21.15.165:6832/311436411 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x56194ef4b040 con 0x56194f1fc3b0 2024-03-18T19:09:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.662+0000 7f8fd17fa700 10 client.8955 handle_command_reply: tid=0 2024-03-18T19:09:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.662+0000 7f9004a0eb80 1 client.8955 shutdown 2024-03-18T19:09:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.662+0000 7f9004a0eb80 1 -- 172.21.15.1:0/2404631714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1bd0a0 msgr2=0x56194f217550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.662+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/2404631714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1bd0a0 0x56194f217550 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4015040 tx=0x7f8fe4009050 comp rx=0 tx=0).stop 2024-03-18T19:09:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f9004a0eb80 1 -- 172.21.15.1:0/2404631714 shutdown_connections 2024-03-18T19:09:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/2404631714 >> [v2:172.21.15.165:6832/311436411,v1:172.21.15.165:6834/311436411] conn(0x56194f1fc3b0 0x56194f1fc790 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/2404631714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56194f1fd630 0x56194f1c0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f8fd17fa700 0 client.8955 ms_handle_reset on v2:172.21.15.165:6832/311436411 2024-03-18T19:09:28.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/2404631714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56194f1bd0a0 0x56194f217550 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f9004a0eb80 1 --2- 172.21.15.1:0/2404631714 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x56194f1ebc10 0x56194f1c06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f9004a0eb80 1 -- 172.21.15.1:0/2404631714 >> 172.21.15.1:0/2404631714 conn(0x56194f11fcf0 msgr2=0x56194f206c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:28.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f9004a0eb80 1 -- 172.21.15.1:0/2404631714 shutdown_connections 2024-03-18T19:09:28.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f9004a0eb80 1 -- 172.21.15.1:0/2404631714 wait complete. 2024-03-18T19:09:28.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f9004a0eb80 20 client.8955 trim_cache size 0 max 16384 2024-03-18T19:09:28.664 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:09:28.664 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "9c1c697f-8655-4ef3-8a13-da568d2cbe86", 2024-03-18T19:09:28.664 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-18T19:09:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 8802, 2024-03-18T19:09:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-18T19:09:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-18T19:09:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-18T19:09:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 6.0378010629999999, 2024-03-18T19:09:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 65, 2024-03-18T19:09:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 147, 2024-03-18T19:09:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 147, 2024-03-18T19:09:28.665 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 11.346375741999999 2024-03-18T19:09:28.665 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:09:28.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f8ff8865700 1 -- 172.21.15.1:0/658860395 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8fd406adb0 msgr2=0x7f8fd406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:28.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f8ff8865700 1 --2- 172.21.15.1:0/658860395 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8fd406adb0 0x7f8fd406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4000ef0 tx=0x7f8fe400a000 comp rx=0 tx=0).stop 2024-03-18T19:09:28.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f8ff8865700 1 -- 172.21.15.1:0/658860395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff4148760 msgr2=0x7f8ff415a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:28.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f8ff8865700 1 --2- 172.21.15.1:0/658860395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff4148760 0x7f8ff415a850 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc0119a0 tx=0x7f8fdc011cb0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f8ff8865700 1 -- 172.21.15.1:0/658860395 shutdown_connections 2024-03-18T19:09:28.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f8ff8865700 1 --2- 172.21.15.1:0/658860395 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8fd406adb0 0x7f8fd406d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f8ff8865700 1 --2- 172.21.15.1:0/658860395 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8fe0003a00 0x7f8ff4148090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f8ff8865700 1 --2- 172.21.15.1:0/658860395 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ff40b5180 0x7f8ff40b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f8ff8865700 1 --2- 172.21.15.1:0/658860395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff4148760 0x7f8ff415a850 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f8ff8865700 1 -- 172.21.15.1:0/658860395 >> 172.21.15.1:0/658860395 conn(0x7f8ff40cc9b0 msgr2=0x7f8ff406ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:28.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f8ff8865700 1 -- 172.21.15.1:0/658860395 shutdown_connections 2024-03-18T19:09:28.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.663+0000 7f8ff8865700 1 -- 172.21.15.1:0/658860395 wait complete. 2024-03-18T19:09:28.671 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T19:09:28.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.800+0000 7f150cd90700 1 Processor -- start 2024-03-18T19:09:28.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.800+0000 7f150cd90700 1 -- start start 2024-03-18T19:09:28.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.801+0000 7f150cd90700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f150809a150 0x7f150806ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.801+0000 7f150cd90700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f150806c7a0 con 0x7f15080b0570 2024-03-18T19:09:28.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.801+0000 7f150cd90700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f150806ca80 con 0x7f150806e880 2024-03-18T19:09:28.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.801+0000 7f150cd90700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15080cf760 con 0x7f150809a150 2024-03-18T19:09:28.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.801+0000 7f150659c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f15080b0570 0x7f1508099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51150/0 (socket says 172.21.15.1:51150) 2024-03-18T19:09:28.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.801+0000 7f150659c700 1 -- 172.21.15.1:0/2145194020 learned_addr learned my addr 172.21.15.1:0/2145194020 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:28.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.801+0000 7f1505d9b700 1 --2- 172.21.15.1:0/2145194020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f150809a150 0x7f150806ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.802+0000 7f150559a700 1 -- 172.21.15.1:0/2145194020 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647936172 0 0) 0x7f150806c7a0 con 0x7f15080b0570 2024-03-18T19:09:28.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.802+0000 7f150559a700 1 -- 172.21.15.1:0/2145194020 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14f00035b0 con 0x7f15080b0570 2024-03-18T19:09:28.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.802+0000 7f150559a700 1 -- 172.21.15.1:0/2145194020 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623893493 0 0) 0x7f150806ca80 con 0x7f150806e880 2024-03-18T19:09:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.802+0000 7f150559a700 1 -- 172.21.15.1:0/2145194020 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f150806c7a0 con 0x7f150806e880 2024-03-18T19:09:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.802+0000 7f1505d9b700 1 -- 172.21.15.1:0/2145194020 >> v1:172.21.15.165:6789/0 conn(0x7f150806e880 legacy=0x7f150806bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.802+0000 7f1505d9b700 1 -- 172.21.15.1:0/2145194020 >> v1:172.21.15.1:6789/0 conn(0x7f15080b0570 legacy=0x7f1508099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.802+0000 7f150559a700 1 -- 172.21.15.1:0/2145194020 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3433760419 0 0) 0x7f14f00035b0 con 0x7f15080b0570 2024-03-18T19:09:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.802+0000 7f1505d9b700 1 -- 172.21.15.1:0/2145194020 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15080cfa40 con 0x7f150809a150 2024-03-18T19:09:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.802+0000 7f1505d9b700 1 --2- 172.21.15.1:0/2145194020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f150809a150 0x7f150806ba70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f14f8009870 tx=0x7f14f8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0c47c8984890e58 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.802+0000 7f150559a700 1 -- 172.21.15.1:0/2145194020 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc001670 con 0x7f150809a150 2024-03-18T19:09:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.802+0000 7f150559a700 1 -- 172.21.15.1:0/2145194020 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f8004030 con 0x7f150809a150 2024-03-18T19:09:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.802+0000 7f150559a700 1 -- 172.21.15.1:0/2145194020 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f80043a0 con 0x7f150809a150 2024-03-18T19:09:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.803+0000 7f150cd90700 1 -- 172.21.15.1:0/2145194020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f150809a150 msgr2=0x7f150806ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.803+0000 7f150cd90700 1 --2- 172.21.15.1:0/2145194020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f150809a150 0x7f150806ba70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f14f8009870 tx=0x7f14f8009b80 comp rx=0 tx=0).stop 2024-03-18T19:09:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.803+0000 7f150cd90700 1 -- 172.21.15.1:0/2145194020 shutdown_connections 2024-03-18T19:09:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.803+0000 7f150cd90700 1 --2- 172.21.15.1:0/2145194020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f150809a150 0x7f150806ba70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.803+0000 7f150cd90700 1 -- 172.21.15.1:0/2145194020 >> 172.21.15.1:0/2145194020 conn(0x7f15080b2c30 msgr2=0x7f15080b3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:28.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.803+0000 7f150cd90700 1 -- 172.21.15.1:0/2145194020 shutdown_connections 2024-03-18T19:09:28.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.803+0000 7f150cd90700 1 -- 172.21.15.1:0/2145194020 wait complete. 2024-03-18T19:09:28.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.805+0000 7f150cd90700 1 Processor -- start 2024-03-18T19:09:28.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.805+0000 7f150cd90700 1 -- start start 2024-03-18T19:09:28.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f150cd90700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f150806e880 0x7f15080c6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f150cd90700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f150809a150 0x7f15081505d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f150cd90700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15080b0570 0x7f15080aee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f150cd90700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15081592f0 con 0x7f150806e880 2024-03-18T19:09:28.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f150cd90700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15080ce910 con 0x7f150809a150 2024-03-18T19:09:28.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f150cd90700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15081589c0 con 0x7f15080b0570 2024-03-18T19:09:28.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f150659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f150806e880 0x7f15080c6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f150659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f150806e880 0x7f15080c6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39080/0 (socket says 172.21.15.1:39080) 2024-03-18T19:09:28.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f150659c700 1 -- 172.21.15.1:0/781085540 learned_addr learned my addr 172.21.15.1:0/781085540 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:28.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f1505d9b700 1 --2- 172.21.15.1:0/781085540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f150809a150 0x7f15081505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f1506d9d700 1 --2- 172.21.15.1:0/781085540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15080b0570 0x7f15080aee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f150659c700 1 -- 172.21.15.1:0/781085540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15080b0570 msgr2=0x7f15080aee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:28.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f150659c700 1 --2- 172.21.15.1:0/781085540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15080b0570 0x7f15080aee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f150659c700 1 -- 172.21.15.1:0/781085540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f150809a150 msgr2=0x7f15081505d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:28.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f150659c700 1 --2- 172.21.15.1:0/781085540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f150809a150 0x7f15081505d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:28.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f150659c700 1 -- 172.21.15.1:0/781085540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f80094a0 con 0x7f150806e880 2024-03-18T19:09:28.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f150659c700 1 --2- 172.21.15.1:0/781085540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f150806e880 0x7f15080c6670 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00c500 tx=0x7f14fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:28.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f14f6ffd700 1 -- 172.21.15.1:0/781085540 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc004c80 con 0x7f150806e880 2024-03-18T19:09:28.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f14f6ffd700 1 -- 172.21.15.1:0/781085540 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14fc004e00 con 0x7f150806e880 2024-03-18T19:09:28.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f150cd90700 1 -- 172.21.15.1:0/781085540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1508158ca0 con 0x7f150806e880 2024-03-18T19:09:28.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f14f6ffd700 1 -- 172.21.15.1:0/781085540 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc007310 con 0x7f150806e880 2024-03-18T19:09:28.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.806+0000 7f150cd90700 1 -- 172.21.15.1:0/781085540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1508151680 con 0x7f150806e880 2024-03-18T19:09:28.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.807+0000 7f150cd90700 1 -- 172.21.15.1:0/781085540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15080676a0 con 0x7f150806e880 2024-03-18T19:09:28.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.808+0000 7f14f6ffd700 1 -- 172.21.15.1:0/781085540 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f14fc007470 con 0x7f150806e880 2024-03-18T19:09:28.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.808+0000 7f14f6ffd700 1 --2- 172.21.15.1:0/781085540 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14e806adb0 0x7f14e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:28.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.808+0000 7f14f6ffd700 1 -- 172.21.15.1:0/781085540 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f14fc089450 con 0x7f150806e880 2024-03-18T19:09:28.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.808+0000 7f1505d9b700 1 --2- 172.21.15.1:0/781085540 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14e806adb0 0x7f14e806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:28.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.809+0000 7f1505d9b700 1 --2- 172.21.15.1:0/781085540 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14e806adb0 0x7f14e806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f150806f5b0 tx=0x7f14f8008030 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:28.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:28.813+0000 7f14f6ffd700 1 -- 172.21.15.1:0/781085540 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f14fc054320 con 0x7f150806e880 2024-03-18T19:09:29.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:29.063+0000 7f150cd90700 1 -- 172.21.15.1:0/781085540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f150806f610 con 0x7f150806e880 2024-03-18T19:09:29.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:29.063+0000 7f14f6ffd700 1 -- 172.21.15.1:0/781085540 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v65) v1 ==== 53+0+73 (secure 0 0 0) 0x7f14fc057940 con 0x7f150806e880 2024-03-18T19:09:29.059 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T19:09:29.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:29.065+0000 7f150cd90700 1 -- 172.21.15.1:0/781085540 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14e806adb0 msgr2=0x7f14e806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:29.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:29.065+0000 7f150cd90700 1 --2- 172.21.15.1:0/781085540 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14e806adb0 0x7f14e806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f150806f5b0 tx=0x7f14f8008030 comp rx=0 tx=0).stop 2024-03-18T19:09:29.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:29.065+0000 7f150cd90700 1 -- 172.21.15.1:0/781085540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f150806e880 msgr2=0x7f15080c6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:29.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:29.065+0000 7f150cd90700 1 --2- 172.21.15.1:0/781085540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f150806e880 0x7f15080c6670 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00c500 tx=0x7f14fc00c8c0 comp rx=0 tx=0).stop 2024-03-18T19:09:29.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:29.065+0000 7f150cd90700 1 -- 172.21.15.1:0/781085540 shutdown_connections 2024-03-18T19:09:29.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:29.065+0000 7f150cd90700 1 --2- 172.21.15.1:0/781085540 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f14e806adb0 0x7f14e806d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:29.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:29.065+0000 7f150cd90700 1 --2- 172.21.15.1:0/781085540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15080b0570 0x7f15080aee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:29.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:29.065+0000 7f150cd90700 1 --2- 172.21.15.1:0/781085540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f150806e880 0x7f15080c6670 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:29.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:29.065+0000 7f150cd90700 1 --2- 172.21.15.1:0/781085540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f150809a150 0x7f15081505d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:29.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:29.065+0000 7f150cd90700 1 -- 172.21.15.1:0/781085540 >> 172.21.15.1:0/781085540 conn(0x7f15080b2c30 msgr2=0x7f150806f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:29.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:29.065+0000 7f150cd90700 1 -- 172.21.15.1:0/781085540 shutdown_connections 2024-03-18T19:09:29.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:29.066+0000 7f150cd90700 1 -- 172.21.15.1:0/781085540 wait complete. 2024-03-18T19:09:29.073 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T19:09:29.073 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T19:09:29.074 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-18T19:09:29.074 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T19:09:29.074 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T19:09:29.074 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T19:09:29.074 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T19:09:29.074 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T19:09:29.074 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:09:29.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:29.074 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-18T19:09:29.093 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T19:09:29.094 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T19:09:29.094 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T19:09:29.094 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:09:29.094 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T19:09:29.094 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:09:29.094 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:09:29.094 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-18T19:09:29.094 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-18T19:09:29.094 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:09:29.094 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-18T19:09:29.094 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-18T19:09:29.095 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T19:09:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:09:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-18T19:09:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T19:09:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 41400sec preferred_lft 41400sec 2024-03-18T19:09:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link 2024-03-18T19:09:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:09:29.095 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:09:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:09:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-18T19:09:29.095 INFO:teuthology.orchestra.run.smithi001.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T19:09:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:09:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T19:09:29.095 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:09:29.096 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::4ca9:f5ff:feb3:79a3/64 scope link 2024-03-18T19:09:29.096 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:09:29.096 INFO:teuthology.orchestra.run.smithi001.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T19:09:29.096 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T19:09:29.096 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::282f:a4ff:fe1a:9908/64 scope link 2024-03-18T19:09:29.096 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:09:29.096 INFO:teuthology.orchestra.run.smithi001.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T19:09:29.096 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2e:5c:60:78:b2:8c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T19:09:29.096 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::2c5c:60ff:fe78:b28c/64 scope link 2024-03-18T19:09:29.096 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:09:29.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:29.097 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T19:09:29.160 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-18T19:09:29.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:29.162 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-18T19:09:29.222 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-18T19:09:29.222 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T19:09:29.222 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T19:09:29.222 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T19:09:29.222 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T19:09:29.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:29.223 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T19:09:29.223 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T19:09:29.223 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T19:09:29.223 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T19:09:29.223 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T19:09:29.223 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T19:09:29.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:29.329 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T19:09:29.347 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T19:09:29.347 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T19:09:29.348 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:09:29.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:29.348 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-18T19:09:29.415 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T19:09:29.416 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T19:09:29.416 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T19:09:29.473 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:09:29.473 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T19:09:29.474 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T19:09:29.534 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T19:09:29.602 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T19:09:29.603 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T19:09:29.669 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T19:09:29.670 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T19:09:29.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:29.671 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T19:09:29.728 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T19:09:29.729 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T19:09:29.771 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T19:09:29.799 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T19:09:29.799 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T19:09:29.858 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-18T19:09:29.863+0000 7f2254ada5c0 -1 init, newargv = 0x559f7c13ca40 newargc=15 2024-03-18T19:09:29.859 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[129429]: starting ceph client 2024-03-18T19:09:29.864 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[129429]: starting fuse 2024-03-18T19:09:29.867 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T19:09:29.869 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T19:09:29.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:29.869 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T19:09:29.928 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T19:09:29.928 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-03-18T19:09:29.928 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T19:09:29.928 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-18T19:09:29.928 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-18T19:09:29.928 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-18T19:09:29.928 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-18T19:09:29.929 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:09:29.929 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-18T19:09:29.929 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T19:09:29.929 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-18T19:09:29.929 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T19:09:29.929 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:09:29.929 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-18T19:09:29.929 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-18T19:09:29.929 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-18T19:09:29.929 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-18T19:09:29.929 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:09:29.929 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-18T19:09:29.929 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T19:09:29.929 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T19:09:29.929 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T19:09:29.929 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-18T19:09:29.929 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-18T19:09:29.930 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-18T19:09:29.930 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T19:09:29.930 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:09:29.930 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-18T19:09:29.930 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-18T19:09:30.053 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.129429.asok 2024-03-18T19:09:30.235 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.129429.asok 2024-03-18T19:09:30.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:30.236 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.129429.asok status 2024-03-18T19:09:30.397 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:09:30.397 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-18T19:09:30.397 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T19:09:30.397 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T19:09:30.397 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-18T19:09:30.397 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-18T19:09:30.397 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T19:09:30.398 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "129429", 2024-03-18T19:09:30.398 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-18T19:09:30.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:30.398 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-18T19:09:30.398 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-18T19:09:30.398 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 8967, 2024-03-18T19:09:30.398 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-18T19:09:30.398 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-18T19:09:30.398 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-18T19:09:30.398 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8967 2024-03-18T19:09:30.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:30.398 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T19:09:30.398 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T19:09:30.399 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T19:09:30.399 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4259278421 2024-03-18T19:09:30.399 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:30.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:30.399 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T19:09:30.399 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T19:09:30.399 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T19:09:30.399 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4259278421 2024-03-18T19:09:30.399 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:09:30.399 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.8967 192.168.0.1:0/4259278421", 2024-03-18T19:09:30.399 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/4259278421", 2024-03-18T19:09:30.399 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-18T19:09:30.399 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 65, 2024-03-18T19:09:30.399 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 147, 2024-03-18T19:09:30.399 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-18T19:09:30.399 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-18T19:09:30.400 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-18T19:09:30.400 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:09:30.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:30.407 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-18T19:09:30.407 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T19:09:30.425 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-18T19:09:30.425 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T19:09:30.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:30.426 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T19:09:30.494 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-18T19:09:30.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.666+0000 7fabb2e6c700 1 Processor -- start 2024-03-18T19:09:30.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.666+0000 7fabb2e6c700 1 -- start start 2024-03-18T19:09:30.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.666+0000 7fabb2e6c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fabac15d780 0x7fabac0cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:30.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.666+0000 7fabb2e6c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabac09dd40 0x7fabac0cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:30.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.666+0000 7fabb2e6c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabac0ce090 con 0x7fabac0ce5a0 2024-03-18T19:09:30.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.666+0000 7fabb2e6c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabac06db30 con 0x7fabac15d780 2024-03-18T19:09:30.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.666+0000 7fabb2e6c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabac06d520 con 0x7fabac09dd40 2024-03-18T19:09:30.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.667+0000 7fabb0c08700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fabac15d780 0x7fabac0cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:30.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.667+0000 7fabb0c08700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fabac15d780 0x7fabac0cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39082/0 (socket says 172.21.15.1:39082) 2024-03-18T19:09:30.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.667+0000 7fabb0c08700 1 -- 172.21.15.1:0/3009927064 learned_addr learned my addr 172.21.15.1:0/3009927064 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:30.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.667+0000 7fabb1409700 1 --1- 172.21.15.1:0/3009927064 >> v1:172.21.15.1:6789/0 conn(0x7fabac0ce5a0 0x7fabac0cea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51166/0 (socket says 172.21.15.1:51166) 2024-03-18T19:09:30.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.667+0000 7fababfff700 1 --2- 172.21.15.1:0/3009927064 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabac09dd40 0x7fabac0cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:30.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.667+0000 7fabb0c08700 1 -- 172.21.15.1:0/3009927064 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabac09dd40 msgr2=0x7fabac0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:30.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.667+0000 7fabb0c08700 1 --2- 172.21.15.1:0/3009927064 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabac09dd40 0x7fabac0cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:30.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.667+0000 7fabb0c08700 1 -- 172.21.15.1:0/3009927064 >> v1:172.21.15.1:6789/0 conn(0x7fabac0ce5a0 legacy=0x7fabac0cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:30.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.667+0000 7fabab7fe700 1 -- 172.21.15.1:0/3009927064 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759456536 0 0) 0x7fabac0ce090 con 0x7fabac0ce5a0 2024-03-18T19:09:30.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.667+0000 7fabb0c08700 1 -- 172.21.15.1:0/3009927064 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabac06f460 con 0x7fabac15d780 2024-03-18T19:09:30.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.667+0000 7fababfff700 1 --2- 172.21.15.1:0/3009927064 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabac09dd40 0x7fabac0cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:30.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.667+0000 7fabb0c08700 1 --2- 172.21.15.1:0/3009927064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabac15d780 0x7fabac0cdb50 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7faba00098f0 tx=0x7faba0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e96ec5b8293610c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:30.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.667+0000 7fabab7fe700 1 -- 172.21.15.1:0/3009927064 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba000b8a0 con 0x7fabac15d780 2024-03-18T19:09:30.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.667+0000 7fabab7fe700 1 -- 172.21.15.1:0/3009927064 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba000ba00 con 0x7fabac15d780 2024-03-18T19:09:30.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.667+0000 7fabab7fe700 1 -- 172.21.15.1:0/3009927064 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba000bd90 con 0x7fabac15d780 2024-03-18T19:09:30.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.668+0000 7fabb2e6c700 1 -- 172.21.15.1:0/3009927064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabac15d780 msgr2=0x7fabac0cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:30.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.668+0000 7fabb2e6c700 1 --2- 172.21.15.1:0/3009927064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabac15d780 0x7fabac0cdb50 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7faba00098f0 tx=0x7faba0009c00 comp rx=0 tx=0).stop 2024-03-18T19:09:30.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.668+0000 7fabb2e6c700 1 -- 172.21.15.1:0/3009927064 shutdown_connections 2024-03-18T19:09:30.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.668+0000 7fabb2e6c700 1 --2- 172.21.15.1:0/3009927064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabac15d780 0x7fabac0cdb50 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:30.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.668+0000 7fabb2e6c700 1 --2- 172.21.15.1:0/3009927064 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabac09dd40 0x7fabac0cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:30.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.668+0000 7fabb2e6c700 1 -- 172.21.15.1:0/3009927064 >> 172.21.15.1:0/3009927064 conn(0x7fabac0cca00 msgr2=0x7fabac0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:30.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.668+0000 7fabb2e6c700 1 -- 172.21.15.1:0/3009927064 shutdown_connections 2024-03-18T19:09:30.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.668+0000 7fabb2e6c700 1 -- 172.21.15.1:0/3009927064 wait complete. 2024-03-18T19:09:30.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.669+0000 7fabb2e6c700 1 Processor -- start 2024-03-18T19:09:30.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.669+0000 7fabb2e6c700 1 -- start start 2024-03-18T19:09:30.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.670+0000 7fabb2e6c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabac09dd40 0x7fabac099ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:30.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.670+0000 7fabb2e6c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabac0ce5a0 0x7fabac09beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:30.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.670+0000 7fabb2e6c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabac15d780 0x7fabac1e62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:30.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.670+0000 7fabb2e6c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabac09c3f0 con 0x7fabac09dd40 2024-03-18T19:09:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.670+0000 7fabb2e6c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabac1e6800 con 0x7fabac0ce5a0 2024-03-18T19:09:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.670+0000 7fabb2e6c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabac09c650 con 0x7fabac15d780 2024-03-18T19:09:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.670+0000 7fabb0c08700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabac09dd40 0x7fabac099ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.670+0000 7fabb0c08700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabac09dd40 0x7fabac099ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39086/0 (socket says 172.21.15.1:39086) 2024-03-18T19:09:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.670+0000 7fabb0c08700 1 -- 172.21.15.1:0/2664297365 learned_addr learned my addr 172.21.15.1:0/2664297365 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.670+0000 7fabb1409700 1 --2- 172.21.15.1:0/2664297365 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabac15d780 0x7fabac1e62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.670+0000 7fababfff700 1 --2- 172.21.15.1:0/2664297365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabac0ce5a0 0x7fabac09beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.670+0000 7fabb0c08700 1 -- 172.21.15.1:0/2664297365 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabac15d780 msgr2=0x7fabac1e62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.670+0000 7fabb0c08700 1 --2- 172.21.15.1:0/2664297365 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabac15d780 0x7fabac1e62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.670+0000 7fabb0c08700 1 -- 172.21.15.1:0/2664297365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabac0ce5a0 msgr2=0x7fabac09beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.670+0000 7fabb0c08700 1 --2- 172.21.15.1:0/2664297365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabac0ce5a0 0x7fabac09beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.670+0000 7fabb0c08700 1 -- 172.21.15.1:0/2664297365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba0009550 con 0x7fabac09dd40 2024-03-18T19:09:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.670+0000 7fababfff700 1 --2- 172.21.15.1:0/2664297365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabac0ce5a0 0x7fabac09beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.671+0000 7fabb0c08700 1 --2- 172.21.15.1:0/2664297365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabac09dd40 0x7fabac099ab0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7faba0000c00 tx=0x7faba0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.671+0000 7faba97fa700 1 -- 172.21.15.1:0/2664297365 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba00038a0 con 0x7fabac09dd40 2024-03-18T19:09:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.671+0000 7faba97fa700 1 -- 172.21.15.1:0/2664297365 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba001e070 con 0x7fabac09dd40 2024-03-18T19:09:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.671+0000 7fabb2e6c700 1 -- 172.21.15.1:0/2664297365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabac09c930 con 0x7fabac09dd40 2024-03-18T19:09:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.671+0000 7faba97fa700 1 -- 172.21.15.1:0/2664297365 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba0003c20 con 0x7fabac09dd40 2024-03-18T19:09:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.671+0000 7fabb2e6c700 1 -- 172.21.15.1:0/2664297365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabac09cef0 con 0x7fabac09dd40 2024-03-18T19:09:30.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.672+0000 7fabb2e6c700 1 -- 172.21.15.1:0/2664297365 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabac073350 con 0x7fabac09dd40 2024-03-18T19:09:30.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.672+0000 7faba97fa700 1 -- 172.21.15.1:0/2664297365 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7faba0011420 con 0x7fabac09dd40 2024-03-18T19:09:30.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.673+0000 7faba97fa700 1 --2- 172.21.15.1:0/2664297365 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fab9006adb0 0x7fab9006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:30.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.673+0000 7faba97fa700 1 -- 172.21.15.1:0/2664297365 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6057+0+0 (secure 0 0 0) 0x7faba0020070 con 0x7fabac09dd40 2024-03-18T19:09:30.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.673+0000 7fababfff700 1 --2- 172.21.15.1:0/2664297365 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fab9006adb0 0x7fab9006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:30.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.673+0000 7fababfff700 1 --2- 172.21.15.1:0/2664297365 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fab9006adb0 0x7fab9006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fabac159c10 tx=0x7fab98008040 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:30.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.678+0000 7faba97fa700 1 -- 172.21.15.1:0/2664297365 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faba0055bd0 con 0x7fabac09dd40 2024-03-18T19:09:30.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.906+0000 7fabb2e6c700 1 -- 172.21.15.1:0/2664297365 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fabac06d7c0 con 0x7fab9006adb0 2024-03-18T19:09:30.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.908+0000 7faba97fa700 1 -- 172.21.15.1:0/2664297365 <== mgr.8565 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fab9800c050 con 0x7fab9006adb0 2024-03-18T19:09:30.904 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T19:09:30.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:09:30.904 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephfs" 2024-03-18T19:09:30.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:09:30.904 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T19:09:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.909+0000 7fabb2e6c700 1 -- 172.21.15.1:0/2664297365 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fab9006adb0 msgr2=0x7fab9006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.909+0000 7fabb2e6c700 1 --2- 172.21.15.1:0/2664297365 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fab9006adb0 0x7fab9006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fabac159c10 tx=0x7fab98008040 comp rx=0 tx=0).stop 2024-03-18T19:09:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.909+0000 7fabb2e6c700 1 -- 172.21.15.1:0/2664297365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabac09dd40 msgr2=0x7fabac099ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.909+0000 7fabb2e6c700 1 --2- 172.21.15.1:0/2664297365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabac09dd40 0x7fabac099ab0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7faba0000c00 tx=0x7faba0004750 comp rx=0 tx=0).stop 2024-03-18T19:09:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.910+0000 7fabb2e6c700 1 -- 172.21.15.1:0/2664297365 shutdown_connections 2024-03-18T19:09:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.910+0000 7fabb2e6c700 1 --2- 172.21.15.1:0/2664297365 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fab9006adb0 0x7fab9006d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:30.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.910+0000 7fabb2e6c700 1 --2- 172.21.15.1:0/2664297365 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabac15d780 0x7fabac1e62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.910+0000 7fabb2e6c700 1 --2- 172.21.15.1:0/2664297365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabac09dd40 0x7fabac099ab0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.910+0000 7fabb2e6c700 1 --2- 172.21.15.1:0/2664297365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabac0ce5a0 0x7fabac09beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.910+0000 7fabb2e6c700 1 -- 172.21.15.1:0/2664297365 >> 172.21.15.1:0/2664297365 conn(0x7fabac0cca00 msgr2=0x7fabac06e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.910+0000 7fabb2e6c700 1 -- 172.21.15.1:0/2664297365 shutdown_connections 2024-03-18T19:09:30.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:30.910+0000 7fabb2e6c700 1 -- 172.21.15.1:0/2664297365 wait complete. 2024-03-18T19:09:30.917 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-18T19:09:31.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.048+0000 7f5bc6b3a700 1 Processor -- start 2024-03-18T19:09:31.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.048+0000 7f5bc6b3a700 1 -- start start 2024-03-18T19:09:31.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.049+0000 7f5bc6b3a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5bc015d5b0 0x7f5bc015d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:31.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.049+0000 7f5bc6b3a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5bc0099a40 0x7f5bc0099ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:31.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.049+0000 7f5bc6b3a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc009a820 con 0x7f5bc009a410 2024-03-18T19:09:31.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.049+0000 7f5bc6b3a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bc00cbbb0 con 0x7f5bc015d5b0 2024-03-18T19:09:31.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.049+0000 7f5bc6b3a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bc00b3b00 con 0x7f5bc0099a40 2024-03-18T19:09:31.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.049+0000 7f5bc48d6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5bc015d5b0 0x7f5bc015d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:31.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.049+0000 7f5bc50d7700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5bc009a410 0x7f5bc006ba70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51172/0 (socket says 172.21.15.1:51172) 2024-03-18T19:09:31.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.049+0000 7f5bc50d7700 1 -- 172.21.15.1:0/2280866259 learned_addr learned my addr 172.21.15.1:0/2280866259 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:31.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.049+0000 7f5bc48d6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5bc015d5b0 0x7f5bc015d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39088/0 (socket says 172.21.15.1:39088) 2024-03-18T19:09:31.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.049+0000 7f5bbffff700 1 --2- 172.21.15.1:0/2280866259 >> v2:172.21.15.165:3300/0 conn(0x7f5bc0099a40 0x7f5bc0099ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:31.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.049+0000 7f5bc48d6700 1 -- 172.21.15.1:0/2280866259 >> v2:172.21.15.165:3300/0 conn(0x7f5bc0099a40 msgr2=0x7f5bc0099ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:31.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.049+0000 7f5bc48d6700 1 --2- 172.21.15.1:0/2280866259 >> v2:172.21.15.165:3300/0 conn(0x7f5bc0099a40 0x7f5bc0099ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:31.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.049+0000 7f5bc48d6700 1 -- 172.21.15.1:0/2280866259 >> v1:172.21.15.1:6789/0 conn(0x7f5bc009a410 legacy=0x7f5bc006ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:31.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.049+0000 7f5bc48d6700 1 -- 172.21.15.1:0/2280866259 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc00cf7f0 con 0x7f5bc015d5b0 2024-03-18T19:09:31.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.049+0000 7f5bbf7fe700 1 -- 172.21.15.1:0/2280866259 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160038638 0 0) 0x7f5bc009a820 con 0x7f5bc009a410 2024-03-18T19:09:31.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.050+0000 7f5bc48d6700 1 --2- 172.21.15.1:0/2280866259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc015d5b0 0x7f5bc015d990 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0072c10 tx=0x7f5bb4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a8cf7ff6fa81d4d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:31.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.050+0000 7f5bbf7fe700 1 -- 172.21.15.1:0/2280866259 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4011830 con 0x7f5bc015d5b0 2024-03-18T19:09:31.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.050+0000 7f5bbf7fe700 1 -- 172.21.15.1:0/2280866259 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb4011990 con 0x7f5bc015d5b0 2024-03-18T19:09:31.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.050+0000 7f5bbf7fe700 1 -- 172.21.15.1:0/2280866259 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4011d20 con 0x7f5bc015d5b0 2024-03-18T19:09:31.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.050+0000 7f5bc6b3a700 1 -- 172.21.15.1:0/2280866259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc015d5b0 msgr2=0x7f5bc015d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:31.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.050+0000 7f5bc6b3a700 1 --2- 172.21.15.1:0/2280866259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc015d5b0 0x7f5bc015d990 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0072c10 tx=0x7f5bb4009b20 comp rx=0 tx=0).stop 2024-03-18T19:09:31.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.050+0000 7f5bc6b3a700 1 -- 172.21.15.1:0/2280866259 shutdown_connections 2024-03-18T19:09:31.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.050+0000 7f5bc6b3a700 1 --2- 172.21.15.1:0/2280866259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc015d5b0 0x7f5bc015d990 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:31.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.050+0000 7f5bc6b3a700 1 --2- 172.21.15.1:0/2280866259 >> v2:172.21.15.165:3300/0 conn(0x7f5bc0099a40 0x7f5bc0099ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:31.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.050+0000 7f5bc6b3a700 1 -- 172.21.15.1:0/2280866259 >> 172.21.15.1:0/2280866259 conn(0x7f5bc00b2bd0 msgr2=0x7f5bc00b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:31.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.050+0000 7f5bc6b3a700 1 -- 172.21.15.1:0/2280866259 shutdown_connections 2024-03-18T19:09:31.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.050+0000 7f5bc6b3a700 1 -- 172.21.15.1:0/2280866259 wait complete. 2024-03-18T19:09:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.052+0000 7f5bc6b3a700 1 Processor -- start 2024-03-18T19:09:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.052+0000 7f5bc6b3a700 1 -- start start 2024-03-18T19:09:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.052+0000 7f5bc6b3a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc0099a40 0x7f5bc00c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.052+0000 7f5bc6b3a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5bc009a410 0x7f5bc00ae860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.052+0000 7f5bc6b3a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bc015d5b0 0x7f5bc015b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bc6b3a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc00b06e0 con 0x7f5bc0099a40 2024-03-18T19:09:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bc6b3a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc00aeda0 con 0x7f5bc009a410 2024-03-18T19:09:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bc6b3a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc015be60 con 0x7f5bc015d5b0 2024-03-18T19:09:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bc48d6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc0099a40 0x7f5bc00c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bc48d6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc0099a40 0x7f5bc00c9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39104/0 (socket says 172.21.15.1:39104) 2024-03-18T19:09:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bc48d6700 1 -- 172.21.15.1:0/2156641510 learned_addr learned my addr 172.21.15.1:0/2156641510 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bc50d7700 1 --2- 172.21.15.1:0/2156641510 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bc015d5b0 0x7f5bc015b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bbffff700 1 --2- 172.21.15.1:0/2156641510 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5bc009a410 0x7f5bc00ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bc48d6700 1 -- 172.21.15.1:0/2156641510 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bc015d5b0 msgr2=0x7f5bc015b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bc48d6700 1 --2- 172.21.15.1:0/2156641510 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bc015d5b0 0x7f5bc015b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bc48d6700 1 -- 172.21.15.1:0/2156641510 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5bc009a410 msgr2=0x7f5bc00ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bc48d6700 1 --2- 172.21.15.1:0/2156641510 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5bc009a410 0x7f5bc00ae860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bc48d6700 1 -- 172.21.15.1:0/2156641510 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb40094a0 con 0x7f5bc0099a40 2024-03-18T19:09:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bbffff700 1 --2- 172.21.15.1:0/2156641510 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5bc009a410 0x7f5bc00ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bc50d7700 1 --2- 172.21.15.1:0/2156641510 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bc015d5b0 0x7f5bc015b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bc48d6700 1 --2- 172.21.15.1:0/2156641510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc0099a40 0x7f5bc00c9cd0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40090b0 tx=0x7f5bb4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bbd7fa700 1 -- 172.21.15.1:0/2156641510 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb40038a0 con 0x7f5bc0099a40 2024-03-18T19:09:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bbd7fa700 1 -- 172.21.15.1:0/2156641510 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb401f070 con 0x7f5bc0099a40 2024-03-18T19:09:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bc6b3a700 1 -- 172.21.15.1:0/2156641510 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc01f2e10 con 0x7f5bc0099a40 2024-03-18T19:09:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bbd7fa700 1 -- 172.21.15.1:0/2156641510 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb4003c20 con 0x7f5bc0099a40 2024-03-18T19:09:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.053+0000 7f5bc6b3a700 1 -- 172.21.15.1:0/2156641510 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc00ca640 con 0x7f5bc0099a40 2024-03-18T19:09:31.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.054+0000 7f5bc6b3a700 1 -- 172.21.15.1:0/2156641510 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc00676a0 con 0x7f5bc0099a40 2024-03-18T19:09:31.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.055+0000 7f5bbd7fa700 1 -- 172.21.15.1:0/2156641510 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 72) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5bb4026480 con 0x7f5bc0099a40 2024-03-18T19:09:31.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.055+0000 7f5bbd7fa700 1 --2- 172.21.15.1:0/2156641510 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ba406adb0 0x7f5ba406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:31.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.055+0000 7f5bbd7fa700 1 -- 172.21.15.1:0/2156641510 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5bb4021070 con 0x7f5bc0099a40 2024-03-18T19:09:31.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.055+0000 7f5bbffff700 1 --2- 172.21.15.1:0/2156641510 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ba406adb0 0x7f5ba406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:31.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.056+0000 7f5bbffff700 1 --2- 172.21.15.1:0/2156641510 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ba406adb0 0x7f5ba406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5bc01f62e0 tx=0x7f5bac008040 comp rx=0 tx=0).ready entity=mgr.8565 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:31.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.060+0000 7f5bbd7fa700 1 -- 172.21.15.1:0/2156641510 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5bb4055f80 con 0x7f5bc0099a40 2024-03-18T19:09:31.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:31.321+0000 7f5bc6b3a700 1 -- 172.21.15.1:0/2156641510 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f5bc006ba00 con 0x7f5bc0099a40 2024-03-18T19:09:32.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.137+0000 7f5bbd7fa700 1 -- 172.21.15.1:0/2156641510 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 73) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f5bb4056350 con 0x7f5bc0099a40 2024-03-18T19:09:32.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.138+0000 7f5bbd7fa700 1 -- 172.21.15.1:0/2156641510 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v73) v1 ==== 92+0+0 (secure 0 0 0) 0x7f5bb4052ba0 con 0x7f5bc0099a40 2024-03-18T19:09:32.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.142+0000 7f5bc6b3a700 1 -- 172.21.15.1:0/2156641510 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ba406adb0 msgr2=0x7f5ba406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.142+0000 7f5bc6b3a700 1 --2- 172.21.15.1:0/2156641510 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ba406adb0 0x7f5ba406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5bc01f62e0 tx=0x7f5bac008040 comp rx=0 tx=0).stop 2024-03-18T19:09:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.142+0000 7f5bc6b3a700 1 -- 172.21.15.1:0/2156641510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc0099a40 msgr2=0x7f5bc00c9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.142+0000 7f5bc6b3a700 1 --2- 172.21.15.1:0/2156641510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc0099a40 0x7f5bc00c9cd0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40090b0 tx=0x7f5bb4004750 comp rx=0 tx=0).stop 2024-03-18T19:09:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.142+0000 7f5bc6b3a700 1 -- 172.21.15.1:0/2156641510 shutdown_connections 2024-03-18T19:09:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.142+0000 7f5bc6b3a700 1 --2- 172.21.15.1:0/2156641510 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ba406adb0 0x7f5ba406d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.142+0000 7f5bc6b3a700 1 --2- 172.21.15.1:0/2156641510 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bc015d5b0 0x7f5bc015b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.142+0000 7f5bc6b3a700 1 --2- 172.21.15.1:0/2156641510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bc0099a40 0x7f5bc00c9cd0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.142+0000 7f5bc6b3a700 1 --2- 172.21.15.1:0/2156641510 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5bc009a410 0x7f5bc00ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.142+0000 7f5bc6b3a700 1 -- 172.21.15.1:0/2156641510 >> 172.21.15.1:0/2156641510 conn(0x7f5bc00b2bd0 msgr2=0x7f5bc00c6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.142+0000 7f5bc6b3a700 1 -- 172.21.15.1:0/2156641510 shutdown_connections 2024-03-18T19:09:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.142+0000 7f5bc6b3a700 1 -- 172.21.15.1:0/2156641510 wait complete. 2024-03-18T19:09:32.153 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-18T19:09:32.273 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:32.277+0000 7f5eaa49c240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:09:32.282 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:32.286+0000 7f91a31d7240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:09:32.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.290+0000 7f542c0a5700 1 Processor -- start 2024-03-18T19:09:32.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.291+0000 7f542c0a5700 1 -- start start 2024-03-18T19:09:32.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.292+0000 7f542c0a5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f54240cd430 0x7f54240b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.292+0000 7f542c0a5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54240c87d0 0x7f54240cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.292+0000 7f542c0a5700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54240d01f0 con 0x7f542415b6c0 2024-03-18T19:09:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.292+0000 7f542c0a5700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54240ccfa0 con 0x7f54240c87d0 2024-03-18T19:09:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.292+0000 7f542c0a5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54240d1db0 con 0x7f54240cd430 2024-03-18T19:09:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.292+0000 7f5429e41700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f542415b6c0 0x7f54240c80c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51190/0 (socket says 172.21.15.1:51190) 2024-03-18T19:09:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.292+0000 7f5429e41700 1 -- 172.21.15.1:0/2520003054 learned_addr learned my addr 172.21.15.1:0/2520003054 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.292+0000 7f542a642700 1 --2- 172.21.15.1:0/2520003054 >> v2:172.21.15.1:3300/0 conn(0x7f54240c87d0 0x7f54240cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.292+0000 7f5429640700 1 --2- 172.21.15.1:0/2520003054 >> v2:172.21.15.165:3300/0 conn(0x7f54240cd430 0x7f54240b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.292+0000 7f5428e3f700 1 -- 172.21.15.1:0/2520003054 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318822314 0 0) 0x7f54240d01f0 con 0x7f542415b6c0 2024-03-18T19:09:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.292+0000 7f5428e3f700 1 -- 172.21.15.1:0/2520003054 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54100035b0 con 0x7f542415b6c0 2024-03-18T19:09:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.292+0000 7f542a642700 1 -- 172.21.15.1:0/2520003054 >> v2:172.21.15.165:3300/0 conn(0x7f54240cd430 msgr2=0x7f54240b2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.292+0000 7f542a642700 1 --2- 172.21.15.1:0/2520003054 >> v2:172.21.15.165:3300/0 conn(0x7f54240cd430 0x7f54240b2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.292+0000 7f542a642700 1 -- 172.21.15.1:0/2520003054 >> v1:172.21.15.1:6789/0 conn(0x7f542415b6c0 legacy=0x7f54240c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.292+0000 7f542a642700 1 -- 172.21.15.1:0/2520003054 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54240cff10 con 0x7f54240c87d0 2024-03-18T19:09:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.292+0000 7f5429640700 1 --2- 172.21.15.1:0/2520003054 >> v2:172.21.15.165:3300/0 conn(0x7f54240cd430 0x7f54240b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.292+0000 7f542a642700 1 --2- 172.21.15.1:0/2520003054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54240c87d0 0x7f54240cd9e0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f542406e620 tx=0x7f541c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa67e49e6c07ed4d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.292+0000 7f5428e3f700 1 -- 172.21.15.1:0/2520003054 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c00e070 con 0x7f54240c87d0 2024-03-18T19:09:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.292+0000 7f5428e3f700 1 -- 172.21.15.1:0/2520003054 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541c00bb20 con 0x7f54240c87d0 2024-03-18T19:09:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.293+0000 7f5428e3f700 1 -- 172.21.15.1:0/2520003054 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c012430 con 0x7f54240c87d0 2024-03-18T19:09:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.293+0000 7f542c0a5700 1 -- 172.21.15.1:0/2520003054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54240c87d0 msgr2=0x7f54240cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.293+0000 7f542c0a5700 1 --2- 172.21.15.1:0/2520003054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54240c87d0 0x7f54240cd9e0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f542406e620 tx=0x7f541c00b330 comp rx=0 tx=0).stop 2024-03-18T19:09:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.293+0000 7f542c0a5700 1 -- 172.21.15.1:0/2520003054 shutdown_connections 2024-03-18T19:09:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.293+0000 7f542c0a5700 1 --2- 172.21.15.1:0/2520003054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54240c87d0 0x7f54240cd9e0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.293+0000 7f542c0a5700 1 --2- 172.21.15.1:0/2520003054 >> v2:172.21.15.165:3300/0 conn(0x7f54240cd430 0x7f54240b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.293+0000 7f542c0a5700 1 -- 172.21.15.1:0/2520003054 >> 172.21.15.1:0/2520003054 conn(0x7f54240d0ff0 msgr2=0x7f54240d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:32.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.293+0000 7f542c0a5700 1 -- 172.21.15.1:0/2520003054 shutdown_connections 2024-03-18T19:09:32.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.293+0000 7f542c0a5700 1 -- 172.21.15.1:0/2520003054 wait complete. 2024-03-18T19:09:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.294+0000 7f542c0a5700 1 Processor -- start 2024-03-18T19:09:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.294+0000 7f542c0a5700 1 -- start start 2024-03-18T19:09:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.295+0000 7f542c0a5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54240c87d0 0x7f542406c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.295+0000 7f542c0a5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54240cd430 0x7f54241f61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.295+0000 7f542c0a5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f542415b6c0 0x7f54240ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.295+0000 7f542c0a5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542406d780 con 0x7f542415b6c0 2024-03-18T19:09:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.295+0000 7f542c0a5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542406cb70 con 0x7f54240c87d0 2024-03-18T19:09:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.295+0000 7f542c0a5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f542406ce50 con 0x7f54240cd430 2024-03-18T19:09:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.295+0000 7f542a642700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f542415b6c0 0x7f54240ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.295+0000 7f542a642700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f542415b6c0 0x7f54240ca0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39112/0 (socket says 172.21.15.1:39112) 2024-03-18T19:09:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.295+0000 7f542a642700 1 -- 172.21.15.1:0/2916929049 learned_addr learned my addr 172.21.15.1:0/2916929049 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.295+0000 7f5429e41700 1 --2- 172.21.15.1:0/2916929049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54240c87d0 0x7f542406c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.295+0000 7f5429640700 1 --2- 172.21.15.1:0/2916929049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54240cd430 0x7f54241f61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.296+0000 7f542a642700 1 -- 172.21.15.1:0/2916929049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54240cd430 msgr2=0x7f54241f61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.296+0000 7f542a642700 1 --2- 172.21.15.1:0/2916929049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54240cd430 0x7f54241f61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.296+0000 7f542a642700 1 -- 172.21.15.1:0/2916929049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54240c87d0 msgr2=0x7f542406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.296+0000 7f542a642700 1 --2- 172.21.15.1:0/2916929049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54240c87d0 0x7f542406c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.296+0000 7f542a642700 1 -- 172.21.15.1:0/2916929049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54140094a0 con 0x7f542415b6c0 2024-03-18T19:09:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.296+0000 7f542a642700 1 --2- 172.21.15.1:0/2916929049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f542415b6c0 0x7f54240ca0c0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f541c014040 tx=0x7f541c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.296+0000 7f541a7fc700 1 -- 172.21.15.1:0/2916929049 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c012430 con 0x7f542415b6c0 2024-03-18T19:09:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.296+0000 7f541a7fc700 1 -- 172.21.15.1:0/2916929049 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541c020070 con 0x7f542415b6c0 2024-03-18T19:09:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.296+0000 7f542c0a5700 1 -- 172.21.15.1:0/2916929049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f541c009a70 con 0x7f542415b6c0 2024-03-18T19:09:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.296+0000 7f541a7fc700 1 -- 172.21.15.1:0/2916929049 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c0098f0 con 0x7f542415b6c0 2024-03-18T19:09:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.296+0000 7f542c0a5700 1 -- 172.21.15.1:0/2916929049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f542406d410 con 0x7f542415b6c0 2024-03-18T19:09:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.297+0000 7f542c0a5700 1 -- 172.21.15.1:0/2916929049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f542406d5a0 con 0x7f542415b6c0 2024-03-18T19:09:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.298+0000 7f541a7fc700 1 -- 172.21.15.1:0/2916929049 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 73) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f541c004040 con 0x7f542415b6c0 2024-03-18T19:09:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.298+0000 7f541a7fc700 1 --2- 172.21.15.1:0/2916929049 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f540806ae00 0x7f540806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:32.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.298+0000 7f5429e41700 1 -- 172.21.15.1:0/2916929049 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f540806ae00 msgr2=0x7f540806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:32.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.299+0000 7f541a7fc700 1 -- 172.21.15.1:0/2916929049 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f541c022070 con 0x7f542415b6c0 2024-03-18T19:09:32.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.299+0000 7f5429e41700 1 --2- 172.21.15.1:0/2916929049 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f540806ae00 0x7f540806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:32.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.303+0000 7f541a7fc700 1 -- 172.21.15.1:0/2916929049 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f541c0be580 con 0x7f542415b6c0 2024-03-18T19:09:32.419 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:32.423+0000 7f5eaa49c240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:09:32.451 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:32.455+0000 7f91a31d7240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:09:32.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.500+0000 7f5429e41700 1 -- 172.21.15.1:0/2916929049 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f540806ae00 msgr2=0x7f540806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:32.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.500+0000 7f5429e41700 1 --2- 172.21.15.1:0/2916929049 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f540806ae00 0x7f540806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:32.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.554+0000 7f542c0a5700 1 -- 172.21.15.1:0/2916929049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f54240676a0 con 0x7f542415b6c0 2024-03-18T19:09:32.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.558+0000 7f541a7fc700 1 -- 172.21.15.1:0/2916929049 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v47)=0 v47) v1 ==== 138+0+0 (secure 0 0 0) 0x7f541c059bf0 con 0x7f542415b6c0 2024-03-18T19:09:32.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.560+0000 7f542c0a5700 1 -- 172.21.15.1:0/2916929049 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f540806ae00 msgr2=0x7f540806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:32.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.560+0000 7f542c0a5700 1 --2- 172.21.15.1:0/2916929049 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f540806ae00 0x7f540806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.560+0000 7f542c0a5700 1 -- 172.21.15.1:0/2916929049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f542415b6c0 msgr2=0x7f54240ca0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:32.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.560+0000 7f542c0a5700 1 --2- 172.21.15.1:0/2916929049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f542415b6c0 0x7f54240ca0c0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f541c014040 tx=0x7f541c004ad0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.561+0000 7f542c0a5700 1 -- 172.21.15.1:0/2916929049 shutdown_connections 2024-03-18T19:09:32.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.561+0000 7f542c0a5700 1 --2- 172.21.15.1:0/2916929049 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f540806ae00 0x7f540806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.561+0000 7f542c0a5700 1 --2- 172.21.15.1:0/2916929049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f542415b6c0 0x7f54240ca0c0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.561+0000 7f542c0a5700 1 --2- 172.21.15.1:0/2916929049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54240c87d0 0x7f542406c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.561+0000 7f542c0a5700 1 --2- 172.21.15.1:0/2916929049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54240cd430 0x7f54241f61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.561+0000 7f542c0a5700 1 -- 172.21.15.1:0/2916929049 >> 172.21.15.1:0/2916929049 conn(0x7f54240d0ff0 msgr2=0x7f54240c9340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:32.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.561+0000 7f542c0a5700 1 -- 172.21.15.1:0/2916929049 shutdown_connections 2024-03-18T19:09:32.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.561+0000 7f542c0a5700 1 -- 172.21.15.1:0/2916929049 wait complete. 2024-03-18T19:09:32.569 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-18T19:09:32.684 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:32.687+0000 7f5eaa49c240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:09:32.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.703+0000 7fe490fbc700 1 Processor -- start 2024-03-18T19:09:32.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.703+0000 7fe490fbc700 1 -- start start 2024-03-18T19:09:32.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.704+0000 7fe490fbc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe48c15dbe0 0x7fe48c0b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:32.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.704+0000 7fe490fbc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c15a5f0 0x7fe48c0cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:32.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.704+0000 7fe490fbc700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48c15aad0 con 0x7fe48c0cc4d0 2024-03-18T19:09:32.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.704+0000 7fe490fbc700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48c13f750 con 0x7fe48c15dbe0 2024-03-18T19:09:32.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.704+0000 7fe490fbc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0cc930 con 0x7fe48c15a5f0 2024-03-18T19:09:32.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.704+0000 7fe48ad9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe48c0cc4d0 0x7fe48c06d500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51204/0 (socket says 172.21.15.1:51204) 2024-03-18T19:09:32.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.704+0000 7fe48a59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe48c15dbe0 0x7fe48c0b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:32.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.704+0000 7fe48ad9d700 1 -- 172.21.15.1:0/1626814747 learned_addr learned my addr 172.21.15.1:0/1626814747 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:32.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.704+0000 7fe489d9b700 1 --2- 172.21.15.1:0/1626814747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c15a5f0 0x7fe48c0cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:32.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.704+0000 7fe48959a700 1 -- 172.21.15.1:0/1626814747 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303530079 0 0) 0x7fe48c15aad0 con 0x7fe48c0cc4d0 2024-03-18T19:09:32.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.704+0000 7fe48959a700 1 -- 172.21.15.1:0/1626814747 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4740035b0 con 0x7fe48c0cc4d0 2024-03-18T19:09:32.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.704+0000 7fe48a59c700 1 -- 172.21.15.1:0/1626814747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c15a5f0 msgr2=0x7fe48c0cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:32.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.704+0000 7fe48a59c700 1 --2- 172.21.15.1:0/1626814747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c15a5f0 0x7fe48c0cbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.704+0000 7fe48a59c700 1 -- 172.21.15.1:0/1626814747 >> v1:172.21.15.1:6789/0 conn(0x7fe48c0cc4d0 legacy=0x7fe48c06d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:32.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.704+0000 7fe48a59c700 1 -- 172.21.15.1:0/1626814747 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c15adb0 con 0x7fe48c15dbe0 2024-03-18T19:09:32.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.704+0000 7fe489d9b700 1 --2- 172.21.15.1:0/1626814747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c15a5f0 0x7fe48c0cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:32.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.704+0000 7fe48a59c700 1 --2- 172.21.15.1:0/1626814747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe48c15dbe0 0x7fe48c0b2fb0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009840 tx=0x7fe47c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4571bbe246064328 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:32.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.705+0000 7fe48959a700 1 -- 172.21.15.1:0/1626814747 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c011830 con 0x7fe48c15dbe0 2024-03-18T19:09:32.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.705+0000 7fe48959a700 1 -- 172.21.15.1:0/1626814747 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47c011990 con 0x7fe48c15dbe0 2024-03-18T19:09:32.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.705+0000 7fe48959a700 1 -- 172.21.15.1:0/1626814747 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c011d20 con 0x7fe48c15dbe0 2024-03-18T19:09:32.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.705+0000 7fe490fbc700 1 -- 172.21.15.1:0/1626814747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe48c15dbe0 msgr2=0x7fe48c0b2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:32.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.705+0000 7fe490fbc700 1 --2- 172.21.15.1:0/1626814747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe48c15dbe0 0x7fe48c0b2fb0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009840 tx=0x7fe47c009b50 comp rx=0 tx=0).stop 2024-03-18T19:09:32.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.705+0000 7fe490fbc700 1 -- 172.21.15.1:0/1626814747 shutdown_connections 2024-03-18T19:09:32.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.705+0000 7fe490fbc700 1 --2- 172.21.15.1:0/1626814747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe48c15dbe0 0x7fe48c0b2fb0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.705+0000 7fe490fbc700 1 --2- 172.21.15.1:0/1626814747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c15a5f0 0x7fe48c0cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.705+0000 7fe490fbc700 1 -- 172.21.15.1:0/1626814747 >> 172.21.15.1:0/1626814747 conn(0x7fe48c0b1cb0 msgr2=0x7fe48c0b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:32.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.705+0000 7fe490fbc700 1 -- 172.21.15.1:0/1626814747 shutdown_connections 2024-03-18T19:09:32.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.705+0000 7fe490fbc700 1 -- 172.21.15.1:0/1626814747 wait complete. 2024-03-18T19:09:32.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.707+0000 7fe490fbc700 1 Processor -- start 2024-03-18T19:09:32.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.707+0000 7fe490fbc700 1 -- start start 2024-03-18T19:09:32.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe490fbc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe48c0cc4d0 0x7fe48c0cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:32.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe490fbc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe48c15a5f0 0x7fe48c1f2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:32.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe490fbc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c15dbe0 0x7fe48c0ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:32.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe490fbc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0ae180 con 0x7fe48c15a5f0 2024-03-18T19:09:32.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe490fbc700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c151000 con 0x7fe48c0cc4d0 2024-03-18T19:09:32.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe490fbc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0ad0e0 con 0x7fe48c15dbe0 2024-03-18T19:09:32.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe489d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe48c15a5f0 0x7fe48c1f2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:32.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe489d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe48c15a5f0 0x7fe48c1f2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39128/0 (socket says 172.21.15.1:39128) 2024-03-18T19:09:32.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe489d9b700 1 -- 172.21.15.1:0/1079138977 learned_addr learned my addr 172.21.15.1:0/1079138977 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:32.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe48ad9d700 1 --2- 172.21.15.1:0/1079138977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c15dbe0 0x7fe48c0ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe48a59c700 1 --2- 172.21.15.1:0/1079138977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe48c0cc4d0 0x7fe48c0cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe489d9b700 1 -- 172.21.15.1:0/1079138977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c15dbe0 msgr2=0x7fe48c0ac8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe489d9b700 1 --2- 172.21.15.1:0/1079138977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c15dbe0 0x7fe48c0ac8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe489d9b700 1 -- 172.21.15.1:0/1079138977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe48c0cc4d0 msgr2=0x7fe48c0cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe489d9b700 1 --2- 172.21.15.1:0/1079138977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe48c0cc4d0 0x7fe48c0cd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe489d9b700 1 -- 172.21.15.1:0/1079138977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47c0094a0 con 0x7fe48c15a5f0 2024-03-18T19:09:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe489d9b700 1 --2- 172.21.15.1:0/1079138977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe48c15a5f0 0x7fe48c1f2570 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fe48000e810 tx=0x7fe48000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe47affd700 1 -- 172.21.15.1:0/1079138977 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48000c070 con 0x7fe48c15a5f0 2024-03-18T19:09:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe47affd700 1 -- 172.21.15.1:0/1079138977 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe480004230 con 0x7fe48c15a5f0 2024-03-18T19:09:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe490fbc700 1 -- 172.21.15.1:0/1079138977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48c0ad420 con 0x7fe48c15a5f0 2024-03-18T19:09:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe47affd700 1 -- 172.21.15.1:0/1079138977 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4800036a0 con 0x7fe48c15a5f0 2024-03-18T19:09:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.708+0000 7fe490fbc700 1 -- 172.21.15.1:0/1079138977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48c0adc60 con 0x7fe48c15a5f0 2024-03-18T19:09:32.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.709+0000 7fe490fbc700 1 -- 172.21.15.1:0/1079138977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe48c0676a0 con 0x7fe48c15a5f0 2024-03-18T19:09:32.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.710+0000 7fe47affd700 1 -- 172.21.15.1:0/1079138977 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 73) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fe480003800 con 0x7fe48c15a5f0 2024-03-18T19:09:32.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.711+0000 7fe47affd700 1 --2- 172.21.15.1:0/1079138977 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe46c06ae00 0x7fe46c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:32.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.711+0000 7fe48a59c700 1 -- 172.21.15.1:0/1079138977 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe46c06ae00 msgr2=0x7fe46c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:32.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.711+0000 7fe47affd700 1 -- 172.21.15.1:0/1079138977 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe480089730 con 0x7fe48c15a5f0 2024-03-18T19:09:32.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.711+0000 7fe48a59c700 1 --2- 172.21.15.1:0/1079138977 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe46c06ae00 0x7fe46c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:32.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.715+0000 7fe47affd700 1 -- 172.21.15.1:0/1079138977 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe480054670 con 0x7fe48c15a5f0 2024-03-18T19:09:32.736 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:32.740+0000 7f91a31d7240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:09:32.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.912+0000 7fe48a59c700 1 -- 172.21.15.1:0/1079138977 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe46c06ae00 msgr2=0x7fe46c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:32.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.912+0000 7fe48a59c700 1 --2- 172.21.15.1:0/1079138977 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe46c06ae00 0x7fe46c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:32.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.919+0000 7fe490fbc700 1 -- 172.21.15.1:0/1079138977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fe48c1f2ab0 con 0x7fe48c15a5f0 2024-03-18T19:09:32.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.923+0000 7fe47affd700 1 -- 172.21.15.1:0/1079138977 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v48)=0 v48) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe480057c90 con 0x7fe48c15a5f0 2024-03-18T19:09:32.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.925+0000 7fe490fbc700 1 -- 172.21.15.1:0/1079138977 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe46c06ae00 msgr2=0x7fe46c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:32.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.925+0000 7fe490fbc700 1 --2- 172.21.15.1:0/1079138977 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe46c06ae00 0x7fe46c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.925+0000 7fe490fbc700 1 -- 172.21.15.1:0/1079138977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe48c15a5f0 msgr2=0x7fe48c1f2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:32.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.925+0000 7fe490fbc700 1 --2- 172.21.15.1:0/1079138977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe48c15a5f0 0x7fe48c1f2570 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fe48000e810 tx=0x7fe48000eb20 comp rx=0 tx=0).stop 2024-03-18T19:09:32.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.925+0000 7fe490fbc700 1 -- 172.21.15.1:0/1079138977 shutdown_connections 2024-03-18T19:09:32.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.925+0000 7fe490fbc700 1 --2- 172.21.15.1:0/1079138977 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe46c06ae00 0x7fe46c06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.925+0000 7fe490fbc700 1 --2- 172.21.15.1:0/1079138977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c15dbe0 0x7fe48c0ac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.925+0000 7fe490fbc700 1 --2- 172.21.15.1:0/1079138977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe48c0cc4d0 0x7fe48c0cd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.925+0000 7fe490fbc700 1 --2- 172.21.15.1:0/1079138977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe48c15a5f0 0x7fe48c1f2570 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:32.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.925+0000 7fe490fbc700 1 -- 172.21.15.1:0/1079138977 >> 172.21.15.1:0/1079138977 conn(0x7fe48c0b1cb0 msgr2=0x7fe48c06eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:32.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.925+0000 7fe490fbc700 1 -- 172.21.15.1:0/1079138977 shutdown_connections 2024-03-18T19:09:32.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:32.925+0000 7fe490fbc700 1 -- 172.21.15.1:0/1079138977 wait complete. 2024-03-18T19:09:32.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:32.934 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/minutes) 2024-03-18T19:09:32.958 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/minutes 1m 2024-03-18T19:09:33.109 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:33.112+0000 7f5eaa49c240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:09:33.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.133+0000 7f528fe77700 1 Processor -- start 2024-03-18T19:09:33.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.133+0000 7f528fe77700 1 -- start start 2024-03-18T19:09:33.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.134+0000 7f528fe77700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5288099a40 0x7f52880ade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:33.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.134+0000 7f528fe77700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5288099e20 0x7f52880cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:33.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.134+0000 7f528fe77700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f52880ccc20 0x7f528806fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:33.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.134+0000 7f528fe77700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52880b3aa0 con 0x7f52880ccc20 2024-03-18T19:09:33.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.134+0000 7f528fe77700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f528806e9b0 con 0x7f5288099a40 2024-03-18T19:09:33.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.134+0000 7f528fe77700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f528806ec90 con 0x7f5288099e20 2024-03-18T19:09:33.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.134+0000 7f528e414700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f52880ccc20 0x7f528806fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:33.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.134+0000 7f528e414700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f52880ccc20 0x7f528806fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39138/0 (socket says 172.21.15.1:39138) 2024-03-18T19:09:33.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.134+0000 7f528dc13700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5288099a40 0x7f52880ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:33.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.134+0000 7f528e414700 1 -- 172.21.15.1:0/3130066713 learned_addr learned my addr 172.21.15.1:0/3130066713 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:33.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.134+0000 7f528d412700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5288099e20 0x7f52880cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:33.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.134+0000 7f528e414700 1 -- 172.21.15.1:0/3130066713 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5288099e20 msgr2=0x7f52880cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:33.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.134+0000 7f528e414700 1 --2- 172.21.15.1:0/3130066713 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5288099e20 0x7f52880cc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:33.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.134+0000 7f528e414700 1 -- 172.21.15.1:0/3130066713 >> v2:172.21.15.165:3300/0 conn(0x7f5288099a40 msgr2=0x7f52880ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:33.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.134+0000 7f528e414700 1 --2- 172.21.15.1:0/3130066713 >> v2:172.21.15.165:3300/0 conn(0x7f5288099a40 0x7f52880ade00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:33.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.134+0000 7f528e414700 1 -- 172.21.15.1:0/3130066713 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528806ef70 con 0x7f52880ccc20 2024-03-18T19:09:33.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.135+0000 7f528e414700 1 --2- 172.21.15.1:0/3130066713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52880ccc20 0x7f528806fa50 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f5288071d40 tx=0x7f528000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=490b07d8c2355f9a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:33.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.135+0000 7f528cc11700 1 -- 172.21.15.1:0/3130066713 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528000e070 con 0x7f52880ccc20 2024-03-18T19:09:33.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.135+0000 7f528cc11700 1 -- 172.21.15.1:0/3130066713 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528000b910 con 0x7f52880ccc20 2024-03-18T19:09:33.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.135+0000 7f528cc11700 1 -- 172.21.15.1:0/3130066713 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528000bcc0 con 0x7f52880ccc20 2024-03-18T19:09:33.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.135+0000 7f528fe77700 1 -- 172.21.15.1:0/3130066713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52880ccc20 msgr2=0x7f528806fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:33.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.135+0000 7f528fe77700 1 --2- 172.21.15.1:0/3130066713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52880ccc20 0x7f528806fa50 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f5288071d40 tx=0x7f528000b330 comp rx=0 tx=0).stop 2024-03-18T19:09:33.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.135+0000 7f528fe77700 1 -- 172.21.15.1:0/3130066713 shutdown_connections 2024-03-18T19:09:33.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.135+0000 7f528fe77700 1 --2- 172.21.15.1:0/3130066713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52880ccc20 0x7f528806fa50 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:33.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.135+0000 7f528fe77700 1 --2- 172.21.15.1:0/3130066713 >> v2:172.21.15.165:3300/0 conn(0x7f5288099a40 0x7f52880ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:33.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.135+0000 7f528fe77700 1 --2- 172.21.15.1:0/3130066713 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5288099e20 0x7f52880cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:33.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.135+0000 7f528fe77700 1 -- 172.21.15.1:0/3130066713 >> 172.21.15.1:0/3130066713 conn(0x7f52880b2c50 msgr2=0x7f52880b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:33.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.136+0000 7f528fe77700 1 -- 172.21.15.1:0/3130066713 shutdown_connections 2024-03-18T19:09:33.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.136+0000 7f528fe77700 1 -- 172.21.15.1:0/3130066713 wait complete. 2024-03-18T19:09:33.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.137+0000 7f528fe77700 1 Processor -- start 2024-03-18T19:09:33.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.137+0000 7f528fe77700 1 -- start start 2024-03-18T19:09:33.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.137+0000 7f528fe77700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5288099a40 0x7f528814a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:33.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.137+0000 7f528fe77700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5288099e20 0x7f52880c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:33.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f528fe77700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52880ccc20 0x7f52880c8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:33.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f528fe77700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52880c4c50 con 0x7f5288099a40 2024-03-18T19:09:33.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f528fe77700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52880cb460 con 0x7f52880ccc20 2024-03-18T19:09:33.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f528fe77700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52880c4ea0 con 0x7f5288099e20 2024-03-18T19:09:33.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f528dc13700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5288099a40 0x7f528814a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:33.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f528dc13700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5288099a40 0x7f528814a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39146/0 (socket says 172.21.15.1:39146) 2024-03-18T19:09:33.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f528dc13700 1 -- 172.21.15.1:0/1839652734 learned_addr learned my addr 172.21.15.1:0/1839652734 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:33.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f528d412700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5288099e20 0x7f52880c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:33.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f528e414700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52880ccc20 0x7f52880c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:33.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f528dc13700 1 -- 172.21.15.1:0/1839652734 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5288099e20 msgr2=0x7f52880c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:33.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f528dc13700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5288099e20 0x7f52880c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:33.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f528dc13700 1 -- 172.21.15.1:0/1839652734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52880ccc20 msgr2=0x7f52880c8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:33.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f528dc13700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52880ccc20 0x7f52880c8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:33.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f528dc13700 1 -- 172.21.15.1:0/1839652734 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5280009b30 con 0x7f5288099a40 2024-03-18T19:09:33.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f528e414700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52880ccc20 0x7f52880c8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:33.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f528dc13700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5288099a40 0x7f528814a600 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f528400e870 tx=0x7f528400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:33.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f527e7fc700 1 -- 172.21.15.1:0/1839652734 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5284013680 con 0x7f5288099a40 2024-03-18T19:09:33.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f527e7fc700 1 -- 172.21.15.1:0/1839652734 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52840137e0 con 0x7f5288099a40 2024-03-18T19:09:33.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f527e7fc700 1 -- 172.21.15.1:0/1839652734 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5284013b90 con 0x7f5288099a40 2024-03-18T19:09:33.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f528fe77700 1 -- 172.21.15.1:0/1839652734 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52880c5180 con 0x7f5288099a40 2024-03-18T19:09:33.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.138+0000 7f528fe77700 1 -- 172.21.15.1:0/1839652734 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5288157950 con 0x7f5288099a40 2024-03-18T19:09:33.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.139+0000 7f528fe77700 1 -- 172.21.15.1:0/1839652734 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52880676a0 con 0x7f5288099a40 2024-03-18T19:09:33.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.140+0000 7f527e7fc700 1 -- 172.21.15.1:0/1839652734 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 73) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f5284013cf0 con 0x7f5288099a40 2024-03-18T19:09:33.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.140+0000 7f527e7fc700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526c06ae00 0x7f526c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:33.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.140+0000 7f528d412700 1 -- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526c06ae00 msgr2=0x7f526c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:33.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.140+0000 7f527e7fc700 1 -- 172.21.15.1:0/1839652734 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5284089ca0 con 0x7f5288099a40 2024-03-18T19:09:33.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.141+0000 7f528d412700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526c06ae00 0x7f526c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:33.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.145+0000 7f527e7fc700 1 -- 172.21.15.1:0/1839652734 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5284054c60 con 0x7f5288099a40 2024-03-18T19:09:33.196 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:33.200+0000 7f91a31d7240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:09:33.319 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:33.323+0000 7f5eaa49c240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:09:33.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.342+0000 7f528d412700 1 -- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526c06ae00 msgr2=0x7f526c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:33.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.342+0000 7f528d412700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526c06ae00 0x7f526c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:33.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.374+0000 7f528fe77700 1 -- 172.21.15.1:0/1839652734 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f528809a2a0 con 0x7f526c06ae00 2024-03-18T19:09:33.415 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:33.419+0000 7f91a31d7240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:09:33.690 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:33.694+0000 7f5eaa49c240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:09:33.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.743+0000 7f528d412700 1 -- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526c06ae00 msgr2=0x7f526c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:33.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:33.743+0000 7f528d412700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526c06ae00 0x7f526c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T19:09:33.803 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:33.807+0000 7f91a31d7240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:09:34.272 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:34.276+0000 7f5eaa49c240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:09:34.344 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:34.348+0000 7f5eaa49c240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:09:34.417 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:34.422+0000 7f91a31d7240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:09:34.471 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:34.475+0000 7f5eaa49c240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:09:34.493 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:34.497+0000 7f91a31d7240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:09:34.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:34.544+0000 7f528d412700 1 -- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526c06ae00 msgr2=0x7f526c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:34.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:34.544+0000 7f528d412700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526c06ae00 0x7f526c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T19:09:34.627 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:34.631+0000 7f91a31d7240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:09:34.649 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:34.653+0000 7f5eaa49c240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:09:34.721 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:34.724+0000 7f5eaa49c240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:09:34.798 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:34.802+0000 7f5eaa49c240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:09:34.813 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:34.817+0000 7f91a31d7240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:09:34.877 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:34.880+0000 7f5eaa49c240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:09:34.889 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:34.893+0000 7f91a31d7240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:09:34.954 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:34.958+0000 7f5eaa49c240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:09:34.971 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:34.975+0000 7f91a31d7240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:09:35.053 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:35.057+0000 7f91a31d7240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:09:35.122 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:35.126+0000 7f5eaa49c240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:09:35.135 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:35.139+0000 7f91a31d7240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:09:35.313 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:35.317+0000 7f91a31d7240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:09:35.335 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:35.339+0000 7f5eaa49c240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:09:35.485 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:35.489+0000 7f5eaa49c240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:09:35.537 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:35.541+0000 7f91a31d7240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:09:35.556 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:35.560+0000 7f5eaa49c240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:09:35.695 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:35.700+0000 7f91a31d7240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:09:35.772 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:35.777+0000 7f91a31d7240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:09:35.997 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:36.000+0000 7f5eaa49c240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:09:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:36.145+0000 7f528d412700 1 -- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526c06ae00 msgr2=0x7f526c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:36.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:36.145+0000 7f528d412700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526c06ae00 0x7f526c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T19:09:36.244 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:36.248+0000 7f91a31d7240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:09:36.331 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:36.335+0000 7f5eaa49c240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:09:36.494 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:36.498+0000 7f5eaa49c240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:09:36.597 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:36.602+0000 7f91a31d7240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:09:36.770 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:36.774+0000 7f91a31d7240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:09:36.881 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:36.885+0000 7f5eaa49c240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:09:37.046 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:37.049+0000 7f5eaa49c240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:09:37.182 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:37.186+0000 7f91a31d7240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:09:37.357 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:37.361+0000 7f91a31d7240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:09:37.410 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:37.413+0000 7f5eaa49c240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:09:37.498 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:37.502+0000 7f5eaa49c240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:09:37.737 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:37.741+0000 7f91a31d7240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:09:37.831 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:37.836+0000 7f91a31d7240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:09:37.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:37.856+0000 7f527e7fc700 1 -- 172.21.15.1:0/1839652734 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 74) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f5284055030 con 0x7f5288099a40 2024-03-18T19:09:37.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:37.856+0000 7f527e7fc700 1 -- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526c06ae00 msgr2=0x7f526c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:37.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:37.856+0000 7f527e7fc700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526c06ae00 0x7f526c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:37.993 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:37.997+0000 7f9115df4700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:37.993 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:37.997+0000 7f9115df4700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:37.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:37.997+0000 7f9115df4700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:37.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:37.997+0000 7f9115df4700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:37.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:37.997+0000 7f9115df4700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:37.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:37.997+0000 7f9117df8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:37.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:37.997+0000 7f9117df8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:37.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:37.997+0000 7f9117df8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:37.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:37.997+0000 7f9117df8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:37.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:37.997+0000 7f9117df8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:38.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:38.861+0000 7f527e7fc700 1 -- 172.21.15.1:0/1839652734 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 75) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f5284057e10 con 0x7f5288099a40 2024-03-18T19:09:38.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:38.862+0000 7f527e7fc700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526c06ae00 0x7f526c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:38.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:38.862+0000 7f527e7fc700 1 -- 172.21.15.1:0/1839652734 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f526c0283e0 con 0x7f526c06ae00 2024-03-18T19:09:38.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:38.865+0000 7f528d412700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526c06ae00 0x7f526c06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:38.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:38.867+0000 7f528d412700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526c06ae00 0x7f526c06d2c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f528809ef70 tx=0x7f5278009210 comp rx=0 tx=0).ready entity=mgr.7262 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:38.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:38.923+0000 7f527e7fc700 1 -- 172.21.15.1:0/1839652734 <== mgr.7262 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7f526c0283e0 con 0x7f526c06ae00 2024-03-18T19:09:38.919 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path snap_test_dir1/minutes 2024-03-18T19:09:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:38.926+0000 7f528fe77700 1 -- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526c06ae00 msgr2=0x7f526c06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:38.926+0000 7f528fe77700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526c06ae00 0x7f526c06d2c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f528809ef70 tx=0x7f5278009210 comp rx=0 tx=0).stop 2024-03-18T19:09:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:38.926+0000 7f528fe77700 1 -- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5288099a40 msgr2=0x7f528814a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:38.926+0000 7f528fe77700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5288099a40 0x7f528814a600 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f528400e870 tx=0x7f528400eb80 comp rx=0 tx=0).stop 2024-03-18T19:09:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:38.926+0000 7f528fe77700 1 -- 172.21.15.1:0/1839652734 shutdown_connections 2024-03-18T19:09:38.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:38.926+0000 7f528fe77700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f526c06ae00 0x7f526c06d2c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:38.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:38.926+0000 7f528fe77700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52880ccc20 0x7f52880c8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:38.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:38.926+0000 7f528fe77700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5288099a40 0x7f528814a600 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:38.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:38.926+0000 7f528fe77700 1 --2- 172.21.15.1:0/1839652734 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5288099e20 0x7f52880c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:38.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:38.926+0000 7f528fe77700 1 -- 172.21.15.1:0/1839652734 >> 172.21.15.1:0/1839652734 conn(0x7f52880b2c50 msgr2=0x7f52880cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:38.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:38.926+0000 7f528fe77700 1 -- 172.21.15.1:0/1839652734 shutdown_connections 2024-03-18T19:09:38.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:38.926+0000 7f528fe77700 1 -- 172.21.15.1:0/1839652734 wait complete. 2024-03-18T19:09:38.935 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/minutes 2024-03-18T19:09:38.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:38.935 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/hourly) 2024-03-18T19:09:38.958 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/hourly 1h 2024-03-18T19:09:39.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.133+0000 7f7328e14700 1 Processor -- start 2024-03-18T19:09:39.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.133+0000 7f7328e14700 1 -- start start 2024-03-18T19:09:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.134+0000 7f7328e14700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7324071490 0x7f732415b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:39.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.134+0000 7f7328e14700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f73240724e0 0x7f732415baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.134+0000 7f7328e14700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f732415b6f0 con 0x7f732415a380 2024-03-18T19:09:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.134+0000 7f7328e14700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73240cb510 con 0x7f73240724e0 2024-03-18T19:09:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.134+0000 7f7328e14700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73240ca900 con 0x7f7324071490 2024-03-18T19:09:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.134+0000 7f7322d9d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f73240724e0 0x7f732415baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.134+0000 7f732259c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7324071490 0x7f732415b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.134+0000 7f7321d9b700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f732415a380 0x7f7324071870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:55766/0 (socket says 172.21.15.1:55766) 2024-03-18T19:09:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.134+0000 7f7321d9b700 1 -- 172.21.15.1:0/2803823879 learned_addr learned my addr 172.21.15.1:0/2803823879 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.134+0000 7f7322d9d700 1 -- 172.21.15.1:0/2803823879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7324071490 msgr2=0x7f732415b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.134+0000 7f7322d9d700 1 --2- 172.21.15.1:0/2803823879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7324071490 0x7f732415b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.134+0000 7f7322d9d700 1 -- 172.21.15.1:0/2803823879 >> v1:172.21.15.165:6789/0 conn(0x7f732415a380 legacy=0x7f7324071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.134+0000 7f7322d9d700 1 -- 172.21.15.1:0/2803823879 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73240cabe0 con 0x7f73240724e0 2024-03-18T19:09:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.134+0000 7f7322d9d700 1 --2- 172.21.15.1:0/2803823879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73240724e0 0x7f732415baa0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f731c009d90 tx=0x7f731c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35155d5e46f51291 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:39.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.135+0000 7f732159a700 1 -- 172.21.15.1:0/2803823879 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c00e070 con 0x7f73240724e0 2024-03-18T19:09:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.135+0000 7f732159a700 1 -- 172.21.15.1:0/2803823879 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f731c00bb20 con 0x7f73240724e0 2024-03-18T19:09:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.135+0000 7f732159a700 1 -- 172.21.15.1:0/2803823879 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c012430 con 0x7f73240724e0 2024-03-18T19:09:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.135+0000 7f7328e14700 1 -- 172.21.15.1:0/2803823879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73240724e0 msgr2=0x7f732415baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.135+0000 7f7328e14700 1 --2- 172.21.15.1:0/2803823879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73240724e0 0x7f732415baa0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f731c009d90 tx=0x7f731c00b330 comp rx=0 tx=0).stop 2024-03-18T19:09:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.135+0000 7f7328e14700 1 -- 172.21.15.1:0/2803823879 shutdown_connections 2024-03-18T19:09:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.135+0000 7f7328e14700 1 --2- 172.21.15.1:0/2803823879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73240724e0 0x7f732415baa0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.135+0000 7f7328e14700 1 --2- 172.21.15.1:0/2803823879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7324071490 0x7f732415b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.135+0000 7f7328e14700 1 -- 172.21.15.1:0/2803823879 >> 172.21.15.1:0/2803823879 conn(0x7f73240b1aa0 msgr2=0x7f73240b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.135+0000 7f7328e14700 1 -- 172.21.15.1:0/2803823879 shutdown_connections 2024-03-18T19:09:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.135+0000 7f7328e14700 1 -- 172.21.15.1:0/2803823879 wait complete. 2024-03-18T19:09:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.137+0000 7f7328e14700 1 Processor -- start 2024-03-18T19:09:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.137+0000 7f7328e14700 1 -- start start 2024-03-18T19:09:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.138+0000 7f7328e14700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7324071490 0x7f732406fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.138+0000 7f7328e14700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f73240724e0 0x7f73241f2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.138+0000 7f7328e14700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f732415a380 0x7f73240c51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.138+0000 7f7328e14700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732406ff90 con 0x7f732415a380 2024-03-18T19:09:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.138+0000 7f7328e14700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732406ab30 con 0x7f7324071490 2024-03-18T19:09:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.138+0000 7f7328e14700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732406ae10 con 0x7f73240724e0 2024-03-18T19:09:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.138+0000 7f7322d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f732415a380 0x7f73240c51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.138+0000 7f7322d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f732415a380 0x7f73240c51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46150/0 (socket says 172.21.15.1:46150) 2024-03-18T19:09:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.138+0000 7f7322d9d700 1 -- 172.21.15.1:0/4206302455 learned_addr learned my addr 172.21.15.1:0/4206302455 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.138+0000 7f7321d9b700 1 --2- 172.21.15.1:0/4206302455 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f73240724e0 0x7f73241f2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.138+0000 7f732259c700 1 --2- 172.21.15.1:0/4206302455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7324071490 0x7f732406fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.138+0000 7f7322d9d700 1 -- 172.21.15.1:0/4206302455 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f73240724e0 msgr2=0x7f73241f2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.138+0000 7f7322d9d700 1 --2- 172.21.15.1:0/4206302455 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f73240724e0 0x7f73241f2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.138+0000 7f7322d9d700 1 -- 172.21.15.1:0/4206302455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7324071490 msgr2=0x7f732406fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.138+0000 7f7322d9d700 1 --2- 172.21.15.1:0/4206302455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7324071490 0x7f732406fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.138+0000 7f7322d9d700 1 -- 172.21.15.1:0/4206302455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c009a70 con 0x7f732415a380 2024-03-18T19:09:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.139+0000 7f7322d9d700 1 --2- 172.21.15.1:0/4206302455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f732415a380 0x7f73240c51c0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f731c014040 tx=0x7f731c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.139+0000 7f7312ffd700 1 -- 172.21.15.1:0/4206302455 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c012430 con 0x7f732415a380 2024-03-18T19:09:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.139+0000 7f7312ffd700 1 -- 172.21.15.1:0/4206302455 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f731c020070 con 0x7f732415a380 2024-03-18T19:09:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.139+0000 7f7312ffd700 1 -- 172.21.15.1:0/4206302455 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c003ea0 con 0x7f732415a380 2024-03-18T19:09:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.139+0000 7f7328e14700 1 -- 172.21.15.1:0/4206302455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73240cbb20 con 0x7f732415a380 2024-03-18T19:09:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.139+0000 7f7328e14700 1 -- 172.21.15.1:0/4206302455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f732406b3d0 con 0x7f732415a380 2024-03-18T19:09:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.140+0000 7f7310ff9700 1 -- 172.21.15.1:0/4206302455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72fc0052f0 con 0x7f732415a380 2024-03-18T19:09:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.140+0000 7f7312ffd700 1 -- 172.21.15.1:0/4206302455 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 75) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f731c004040 con 0x7f732415a380 2024-03-18T19:09:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.141+0000 7f7312ffd700 1 --2- 172.21.15.1:0/4206302455 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f730406ab30 0x7f730406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.141+0000 7f7312ffd700 1 -- 172.21.15.1:0/4206302455 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f731c022070 con 0x7f732415a380 2024-03-18T19:09:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.141+0000 7f732259c700 1 --2- 172.21.15.1:0/4206302455 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f730406ab30 0x7f730406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.141+0000 7f732259c700 1 --2- 172.21.15.1:0/4206302455 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f730406ab30 0x7f730406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7314009be0 tx=0x7f7314009210 comp rx=0 tx=0).ready entity=mgr.7262 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:39.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.145+0000 7f7312ffd700 1 -- 172.21.15.1:0/4206302455 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f731c0560f0 con 0x7f732415a380 2024-03-18T19:09:39.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.375+0000 7f7310ff9700 1 -- 172.21.15.1:0/4206302455 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/hourly", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f72fc000bc0 con 0x7f730406ab30 2024-03-18T19:09:39.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.425+0000 7f7312ffd700 1 -- 172.21.15.1:0/4206302455 <== mgr.7262 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f72fc000bc0 con 0x7f730406ab30 2024-03-18T19:09:39.421 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path snap_test_dir1/hourly 2024-03-18T19:09:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.427+0000 7f7310ff9700 1 -- 172.21.15.1:0/4206302455 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f730406ab30 msgr2=0x7f730406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.427+0000 7f7310ff9700 1 --2- 172.21.15.1:0/4206302455 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f730406ab30 0x7f730406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7314009be0 tx=0x7f7314009210 comp rx=0 tx=0).stop 2024-03-18T19:09:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.427+0000 7f7310ff9700 1 -- 172.21.15.1:0/4206302455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f732415a380 msgr2=0x7f73240c51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.427+0000 7f7310ff9700 1 --2- 172.21.15.1:0/4206302455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f732415a380 0x7f73240c51c0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f731c014040 tx=0x7f731c004ad0 comp rx=0 tx=0).stop 2024-03-18T19:09:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.427+0000 7f7310ff9700 1 -- 172.21.15.1:0/4206302455 shutdown_connections 2024-03-18T19:09:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.427+0000 7f7310ff9700 1 --2- 172.21.15.1:0/4206302455 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f730406ab30 0x7f730406cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.427+0000 7f7310ff9700 1 --2- 172.21.15.1:0/4206302455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f732415a380 0x7f73240c51c0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.427+0000 7f7310ff9700 1 --2- 172.21.15.1:0/4206302455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7324071490 0x7f732406fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.427+0000 7f7310ff9700 1 --2- 172.21.15.1:0/4206302455 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f73240724e0 0x7f73241f2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.427+0000 7f7310ff9700 1 -- 172.21.15.1:0/4206302455 >> 172.21.15.1:0/4206302455 conn(0x7f73240b1aa0 msgr2=0x7f7324154e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.428+0000 7f7310ff9700 1 -- 172.21.15.1:0/4206302455 shutdown_connections 2024-03-18T19:09:39.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.428+0000 7f7310ff9700 1 -- 172.21.15.1:0/4206302455 wait complete. 2024-03-18T19:09:39.436 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/hourly 2024-03-18T19:09:39.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:39.436 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/daily) 2024-03-18T19:09:39.457 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/daily 1d 2024-03-18T19:09:39.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.628+0000 7fcfbc5f2700 1 Processor -- start 2024-03-18T19:09:39.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.628+0000 7fcfbc5f2700 1 -- start start 2024-03-18T19:09:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.629+0000 7fcfbc5f2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcfb40cff70 0x7fcfb40cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.629+0000 7fcfbc5f2700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfb40d0df0 con 0x7fcfb40cd870 2024-03-18T19:09:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.629+0000 7fcfbc5f2700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfb40d0550 con 0x7fcfb40b1e20 2024-03-18T19:09:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.629+0000 7fcfbc5f2700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfb40d0830 con 0x7fcfb40cff70 2024-03-18T19:09:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.629+0000 7fcfbab8f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcfb40cff70 0x7fcfb40cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.629+0000 7fcfbab8f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcfb40cff70 0x7fcfb40cc3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46164/0 (socket says 172.21.15.1:46164) 2024-03-18T19:09:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.629+0000 7fcfbab8f700 1 -- 172.21.15.1:0/3448561990 learned_addr learned my addr 172.21.15.1:0/3448561990 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.630+0000 7fcfbab8f700 1 -- 172.21.15.1:0/3448561990 >> v1:172.21.15.165:6789/0 conn(0x7fcfb40b1e20 legacy=0x7fcfb40cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:39.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.630+0000 7fcfbab8f700 1 -- 172.21.15.1:0/3448561990 >> v1:172.21.15.1:6789/0 conn(0x7fcfb40cd870 legacy=0x7fcfb40cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:39.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.630+0000 7fcfbab8f700 1 -- 172.21.15.1:0/3448561990 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb40d0b10 con 0x7fcfb40cff70 2024-03-18T19:09:39.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.630+0000 7fcfb938c700 1 -- 172.21.15.1:0/3448561990 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622220966 0 0) 0x7fcfb40d0df0 con 0x7fcfb40cd870 2024-03-18T19:09:39.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.630+0000 7fcfbab8f700 1 --2- 172.21.15.1:0/3448561990 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb40cff70 0x7fcfb40cc3f0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fcfac009e50 tx=0x7fcfac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=546ba352fbb20c2f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:39.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.630+0000 7fcfb938c700 1 -- 172.21.15.1:0/3448561990 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfac00e070 con 0x7fcfb40cff70 2024-03-18T19:09:39.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.630+0000 7fcfb938c700 1 -- 172.21.15.1:0/3448561990 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfac00b9c0 con 0x7fcfb40cff70 2024-03-18T19:09:39.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.630+0000 7fcfb938c700 1 -- 172.21.15.1:0/3448561990 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfac00bd70 con 0x7fcfb40cff70 2024-03-18T19:09:39.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.630+0000 7fcfbc5f2700 1 -- 172.21.15.1:0/3448561990 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb40cff70 msgr2=0x7fcfb40cc3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:39.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.630+0000 7fcfbc5f2700 1 --2- 172.21.15.1:0/3448561990 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb40cff70 0x7fcfb40cc3f0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fcfac009e50 tx=0x7fcfac00b330 comp rx=0 tx=0).stop 2024-03-18T19:09:39.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.631+0000 7fcfbc5f2700 1 -- 172.21.15.1:0/3448561990 shutdown_connections 2024-03-18T19:09:39.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.631+0000 7fcfbc5f2700 1 --2- 172.21.15.1:0/3448561990 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb40cff70 0x7fcfb40cc3f0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:39.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.631+0000 7fcfbc5f2700 1 -- 172.21.15.1:0/3448561990 >> 172.21.15.1:0/3448561990 conn(0x7fcfb40d0ff0 msgr2=0x7fcfb40d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:39.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.631+0000 7fcfbc5f2700 1 -- 172.21.15.1:0/3448561990 shutdown_connections 2024-03-18T19:09:39.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.631+0000 7fcfbc5f2700 1 -- 172.21.15.1:0/3448561990 wait complete. 2024-03-18T19:09:39.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.632+0000 7fcfbc5f2700 1 Processor -- start 2024-03-18T19:09:39.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.632+0000 7fcfbc5f2700 1 -- start start 2024-03-18T19:09:39.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.632+0000 7fcfbc5f2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcfb40b1e20 0x7fcfb41f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:39.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.632+0000 7fcfbc5f2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfb40cd870 0x7fcfb40b4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:39.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.632+0000 7fcfbc5f2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb40cff70 0x7fcfb40b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:39.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.633+0000 7fcfbc5f2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb41f3370 con 0x7fcfb40cff70 2024-03-18T19:09:39.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.633+0000 7fcfbc5f2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb40b5460 con 0x7fcfb40cd870 2024-03-18T19:09:39.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.633+0000 7fcfbc5f2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb40b5740 con 0x7fcfb40b1e20 2024-03-18T19:09:39.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.633+0000 7fcfbab8f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb40cff70 0x7fcfb40b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:39.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.633+0000 7fcfbab8f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb40cff70 0x7fcfb40b2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46172/0 (socket says 172.21.15.1:46172) 2024-03-18T19:09:39.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.633+0000 7fcfbab8f700 1 -- 172.21.15.1:0/492665231 learned_addr learned my addr 172.21.15.1:0/492665231 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:39.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.633+0000 7fcfba38e700 1 --2- 172.21.15.1:0/492665231 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcfb40b1e20 0x7fcfb41f2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:39.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.633+0000 7fcfb9b8d700 1 --2- 172.21.15.1:0/492665231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfb40cd870 0x7fcfb40b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:39.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.633+0000 7fcfbab8f700 1 -- 172.21.15.1:0/492665231 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcfb40b1e20 msgr2=0x7fcfb41f2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:39.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.633+0000 7fcfbab8f700 1 --2- 172.21.15.1:0/492665231 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcfb40b1e20 0x7fcfb41f2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:39.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.633+0000 7fcfbab8f700 1 -- 172.21.15.1:0/492665231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfb40cd870 msgr2=0x7fcfb40b4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:39.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.633+0000 7fcfbab8f700 1 --2- 172.21.15.1:0/492665231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfb40cd870 0x7fcfb40b4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:39.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.633+0000 7fcfbab8f700 1 -- 172.21.15.1:0/492665231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfac009b30 con 0x7fcfb40cff70 2024-03-18T19:09:39.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.633+0000 7fcfbab8f700 1 --2- 172.21.15.1:0/492665231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb40cff70 0x7fcfb40b2a70 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fcfac00b730 tx=0x7fcfac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:39.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.633+0000 7fcfaaffd700 1 -- 172.21.15.1:0/492665231 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfac01b630 con 0x7fcfb40cff70 2024-03-18T19:09:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.633+0000 7fcfaaffd700 1 -- 172.21.15.1:0/492665231 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfac01b7b0 con 0x7fcfb40cff70 2024-03-18T19:09:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.633+0000 7fcfbc5f2700 1 -- 172.21.15.1:0/492665231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb40b5a20 con 0x7fcfb40cff70 2024-03-18T19:09:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.633+0000 7fcfaaffd700 1 -- 172.21.15.1:0/492665231 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfac01bb60 con 0x7fcfb40cff70 2024-03-18T19:09:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.633+0000 7fcfbc5f2700 1 -- 172.21.15.1:0/492665231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb40b5fe0 con 0x7fcfb40cff70 2024-03-18T19:09:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.634+0000 7fcfbc5f2700 1 -- 172.21.15.1:0/492665231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb40cc8e0 con 0x7fcfb40cff70 2024-03-18T19:09:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.635+0000 7fcfaaffd700 1 -- 172.21.15.1:0/492665231 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 75) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fcfac01bcc0 con 0x7fcfb40cff70 2024-03-18T19:09:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.635+0000 7fcfaaffd700 1 --2- 172.21.15.1:0/492665231 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcf9806ab30 0x7fcf9806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.635+0000 7fcfba38e700 1 --2- 172.21.15.1:0/492665231 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcf9806ab30 0x7fcf9806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:39.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.635+0000 7fcfaaffd700 1 -- 172.21.15.1:0/492665231 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fcfac08a950 con 0x7fcfb40cff70 2024-03-18T19:09:39.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.636+0000 7fcfba38e700 1 --2- 172.21.15.1:0/492665231 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcf9806ab30 0x7fcf9806cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4000f60 tx=0x7fcfa4009040 comp rx=0 tx=0).ready entity=mgr.7262 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:39.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.640+0000 7fcfaaffd700 1 -- 172.21.15.1:0/492665231 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcfac055bb0 con 0x7fcfb40cff70 2024-03-18T19:09:39.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.869+0000 7fcfbc5f2700 1 -- 172.21.15.1:0/492665231 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/daily", "snap_schedule": "1d", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfb40b25a0 con 0x7fcf9806ab30 2024-03-18T19:09:39.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.930+0000 7fcfaaffd700 1 -- 172.21.15.1:0/492665231 <== mgr.7262 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fcfa40013d0 con 0x7fcf9806ab30 2024-03-18T19:09:39.926 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path snap_test_dir1/daily 2024-03-18T19:09:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.932+0000 7fcfbc5f2700 1 -- 172.21.15.1:0/492665231 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcf9806ab30 msgr2=0x7fcf9806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.932+0000 7fcfbc5f2700 1 --2- 172.21.15.1:0/492665231 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcf9806ab30 0x7fcf9806cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4000f60 tx=0x7fcfa4009040 comp rx=0 tx=0).stop 2024-03-18T19:09:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.932+0000 7fcfbc5f2700 1 -- 172.21.15.1:0/492665231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb40cff70 msgr2=0x7fcfb40b2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.932+0000 7fcfbc5f2700 1 --2- 172.21.15.1:0/492665231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb40cff70 0x7fcfb40b2a70 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fcfac00b730 tx=0x7fcfac004ad0 comp rx=0 tx=0).stop 2024-03-18T19:09:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.932+0000 7fcfbc5f2700 1 -- 172.21.15.1:0/492665231 shutdown_connections 2024-03-18T19:09:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.932+0000 7fcfbc5f2700 1 --2- 172.21.15.1:0/492665231 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcf9806ab30 0x7fcf9806cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.932+0000 7fcfbc5f2700 1 --2- 172.21.15.1:0/492665231 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcfb40b1e20 0x7fcfb41f2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.932+0000 7fcfbc5f2700 1 --2- 172.21.15.1:0/492665231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb40cff70 0x7fcfb40b2a70 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.932+0000 7fcfbc5f2700 1 --2- 172.21.15.1:0/492665231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcfb40cd870 0x7fcfb40b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.932+0000 7fcfbc5f2700 1 -- 172.21.15.1:0/492665231 >> 172.21.15.1:0/492665231 conn(0x7fcfb40d0ff0 msgr2=0x7fcfb415b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.932+0000 7fcfbc5f2700 1 -- 172.21.15.1:0/492665231 shutdown_connections 2024-03-18T19:09:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:39.932+0000 7fcfbc5f2700 1 -- 172.21.15.1:0/492665231 wait complete. 2024-03-18T19:09:39.940 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/daily 2024-03-18T19:09:39.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:39.940 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/weekly) 2024-03-18T19:09:39.961 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/weekly 1w 2024-03-18T19:09:40.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.133+0000 7fc026697700 1 Processor -- start 2024-03-18T19:09:40.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.133+0000 7fc026697700 1 -- start start 2024-03-18T19:09:40.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.134+0000 7fc026697700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc020143c60 0x7fc02006eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:40.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.134+0000 7fc026697700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc0200b4a50 0x7fc0200ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:40.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.134+0000 7fc026697700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02006f430 con 0x7fc020072330 2024-03-18T19:09:40.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.134+0000 7fc026697700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc02006f710 con 0x7fc0200b4a50 2024-03-18T19:09:40.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.134+0000 7fc026697700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0200cda00 con 0x7fc020143c60 2024-03-18T19:09:40.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.134+0000 7fc024c34700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc0200b4a50 0x7fc0200ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:40.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.134+0000 7fc01f7fe700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fc020072330 0x7fc0200b4340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:55780/0 (socket says 172.21.15.1:55780) 2024-03-18T19:09:40.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.134+0000 7fc01f7fe700 1 -- 172.21.15.1:0/1835122218 learned_addr learned my addr 172.21.15.1:0/1835122218 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:40.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.134+0000 7fc01ffff700 1 --2- 172.21.15.1:0/1835122218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc020143c60 0x7fc02006eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:40.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.135+0000 7fc024c34700 1 -- 172.21.15.1:0/1835122218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc020143c60 msgr2=0x7fc02006eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:40.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.135+0000 7fc024c34700 1 --2- 172.21.15.1:0/1835122218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc020143c60 0x7fc02006eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:40.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.135+0000 7fc024c34700 1 -- 172.21.15.1:0/1835122218 >> v1:172.21.15.165:6789/0 conn(0x7fc020072330 legacy=0x7fc0200b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:40.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.135+0000 7fc024c34700 1 -- 172.21.15.1:0/1835122218 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0200cdce0 con 0x7fc0200b4a50 2024-03-18T19:09:40.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.135+0000 7fc01effd700 1 -- 172.21.15.1:0/1835122218 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311923778 0 0) 0x7fc02006f430 con 0x7fc020072330 2024-03-18T19:09:40.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.135+0000 7fc01ffff700 1 --2- 172.21.15.1:0/1835122218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc020143c60 0x7fc02006eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:40.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.135+0000 7fc024c34700 1 --2- 172.21.15.1:0/1835122218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0200b4a50 0x7fc0200ce4e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fc018009d90 tx=0x7fc01800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6ad28e0c9fb6a8d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:40.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.135+0000 7fc01effd700 1 -- 172.21.15.1:0/1835122218 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01800e070 con 0x7fc0200b4a50 2024-03-18T19:09:40.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.135+0000 7fc01effd700 1 -- 172.21.15.1:0/1835122218 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01800bb20 con 0x7fc0200b4a50 2024-03-18T19:09:40.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.135+0000 7fc01effd700 1 -- 172.21.15.1:0/1835122218 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018012430 con 0x7fc0200b4a50 2024-03-18T19:09:40.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.136+0000 7fc026697700 1 -- 172.21.15.1:0/1835122218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0200b4a50 msgr2=0x7fc0200ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:40.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.136+0000 7fc026697700 1 --2- 172.21.15.1:0/1835122218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0200b4a50 0x7fc0200ce4e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fc018009d90 tx=0x7fc01800b330 comp rx=0 tx=0).stop 2024-03-18T19:09:40.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.136+0000 7fc026697700 1 -- 172.21.15.1:0/1835122218 shutdown_connections 2024-03-18T19:09:40.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.136+0000 7fc026697700 1 --2- 172.21.15.1:0/1835122218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0200b4a50 0x7fc0200ce4e0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:40.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.136+0000 7fc026697700 1 --2- 172.21.15.1:0/1835122218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc020143c60 0x7fc02006eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:40.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.136+0000 7fc026697700 1 -- 172.21.15.1:0/1835122218 >> 172.21.15.1:0/1835122218 conn(0x7fc0200b1d20 msgr2=0x7fc0200b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:40.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.136+0000 7fc026697700 1 -- 172.21.15.1:0/1835122218 shutdown_connections 2024-03-18T19:09:40.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.136+0000 7fc026697700 1 -- 172.21.15.1:0/1835122218 wait complete. 2024-03-18T19:09:40.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.137+0000 7fc026697700 1 Processor -- start 2024-03-18T19:09:40.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.137+0000 7fc026697700 1 -- start start 2024-03-18T19:09:40.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.138+0000 7fc026697700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc020072330 0x7fc02015c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:40.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.138+0000 7fc026697700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0200b4a50 0x7fc0201e37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:40.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.138+0000 7fc026697700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc020143c60 0x7fc02013fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:40.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.138+0000 7fc026697700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02015a960 con 0x7fc020143c60 2024-03-18T19:09:40.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.138+0000 7fc026697700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02015c950 con 0x7fc0200b4a50 2024-03-18T19:09:40.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.138+0000 7fc026697700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02015cc30 con 0x7fc020072330 2024-03-18T19:09:40.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.138+0000 7fc024c34700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc020143c60 0x7fc02013fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:40.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.138+0000 7fc024c34700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc020143c60 0x7fc02013fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46194/0 (socket says 172.21.15.1:46194) 2024-03-18T19:09:40.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.138+0000 7fc024c34700 1 -- 172.21.15.1:0/2717687964 learned_addr learned my addr 172.21.15.1:0/2717687964 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:40.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.138+0000 7fc01ffff700 1 --2- 172.21.15.1:0/2717687964 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc020072330 0x7fc02015c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:40.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.138+0000 7fc01f7fe700 1 --2- 172.21.15.1:0/2717687964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0200b4a50 0x7fc0201e37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:40.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.138+0000 7fc024c34700 1 -- 172.21.15.1:0/2717687964 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc020072330 msgr2=0x7fc02015c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:40.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.138+0000 7fc024c34700 1 --2- 172.21.15.1:0/2717687964 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc020072330 0x7fc02015c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:40.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.138+0000 7fc024c34700 1 -- 172.21.15.1:0/2717687964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0200b4a50 msgr2=0x7fc0201e37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.138+0000 7fc024c34700 1 --2- 172.21.15.1:0/2717687964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0200b4a50 0x7fc0201e37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.138+0000 7fc024c34700 1 -- 172.21.15.1:0/2717687964 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0080094a0 con 0x7fc020143c60 2024-03-18T19:09:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.138+0000 7fc01f7fe700 1 --2- 172.21.15.1:0/2717687964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0200b4a50 0x7fc0201e37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.139+0000 7fc024c34700 1 --2- 172.21.15.1:0/2717687964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc020143c60 0x7fc02013fa50 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fc018013000 tx=0x7fc018004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.139+0000 7fc01cff9700 1 -- 172.21.15.1:0/2717687964 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018012430 con 0x7fc020143c60 2024-03-18T19:09:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.139+0000 7fc01cff9700 1 -- 172.21.15.1:0/2717687964 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc018020070 con 0x7fc020143c60 2024-03-18T19:09:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.139+0000 7fc026697700 1 -- 172.21.15.1:0/2717687964 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc018009a70 con 0x7fc020143c60 2024-03-18T19:09:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.139+0000 7fc01cff9700 1 -- 172.21.15.1:0/2717687964 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018004280 con 0x7fc020143c60 2024-03-18T19:09:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.139+0000 7fc026697700 1 -- 172.21.15.1:0/2717687964 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc02015d1f0 con 0x7fc020143c60 2024-03-18T19:09:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.140+0000 7fc026697700 1 -- 172.21.15.1:0/2717687964 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc02015aad0 con 0x7fc020143c60 2024-03-18T19:09:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.140+0000 7fc01cff9700 1 -- 172.21.15.1:0/2717687964 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 75) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fc0180043e0 con 0x7fc020143c60 2024-03-18T19:09:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.140+0000 7fc01cff9700 1 --2- 172.21.15.1:0/2717687964 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc00006ab30 0x7fc00006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.141+0000 7fc01cff9700 1 -- 172.21.15.1:0/2717687964 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fc018022070 con 0x7fc020143c60 2024-03-18T19:09:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.141+0000 7fc01ffff700 1 --2- 172.21.15.1:0/2717687964 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc00006ab30 0x7fc00006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:40.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.141+0000 7fc01ffff700 1 --2- 172.21.15.1:0/2717687964 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc00006ab30 0x7fc00006cff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc020140690 tx=0x7fc008009210 comp rx=0 tx=0).ready entity=mgr.7262 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:40.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.145+0000 7fc01cff9700 1 -- 172.21.15.1:0/2717687964 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc02015aad0 con 0x7fc020143c60 2024-03-18T19:09:40.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.371+0000 7fc026697700 1 -- 172.21.15.1:0/2717687964 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/weekly", "snap_schedule": "1w", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0200cea20 con 0x7fc00006ab30 2024-03-18T19:09:40.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.425+0000 7fc01cff9700 1 -- 172.21.15.1:0/2717687964 <== mgr.7262 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7fc008009de0 con 0x7fc00006ab30 2024-03-18T19:09:40.421 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path snap_test_dir1/weekly 2024-03-18T19:09:40.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.427+0000 7fc026697700 1 -- 172.21.15.1:0/2717687964 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc00006ab30 msgr2=0x7fc00006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:40.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.427+0000 7fc026697700 1 --2- 172.21.15.1:0/2717687964 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc00006ab30 0x7fc00006cff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc020140690 tx=0x7fc008009210 comp rx=0 tx=0).stop 2024-03-18T19:09:40.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.427+0000 7fc026697700 1 -- 172.21.15.1:0/2717687964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc020143c60 msgr2=0x7fc02013fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:40.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.427+0000 7fc026697700 1 --2- 172.21.15.1:0/2717687964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc020143c60 0x7fc02013fa50 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fc018013000 tx=0x7fc018004ad0 comp rx=0 tx=0).stop 2024-03-18T19:09:40.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.427+0000 7fc026697700 1 -- 172.21.15.1:0/2717687964 shutdown_connections 2024-03-18T19:09:40.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.427+0000 7fc026697700 1 --2- 172.21.15.1:0/2717687964 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc00006ab30 0x7fc00006cff0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:40.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.427+0000 7fc026697700 1 --2- 172.21.15.1:0/2717687964 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc020072330 0x7fc02015c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:40.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.427+0000 7fc026697700 1 --2- 172.21.15.1:0/2717687964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc020143c60 0x7fc02013fa50 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:40.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.427+0000 7fc026697700 1 --2- 172.21.15.1:0/2717687964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0200b4a50 0x7fc0201e37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:40.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.427+0000 7fc026697700 1 -- 172.21.15.1:0/2717687964 >> 172.21.15.1:0/2717687964 conn(0x7fc0200b1d20 msgr2=0x7fc020142340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:40.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.427+0000 7fc026697700 1 -- 172.21.15.1:0/2717687964 shutdown_connections 2024-03-18T19:09:40.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.428+0000 7fc026697700 1 -- 172.21.15.1:0/2717687964 wait complete. 2024-03-18T19:09:40.436 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/weekly 2024-03-18T19:09:40.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:40.436 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/monthly) 2024-03-18T19:09:40.456 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/monthly 1M 2024-03-18T19:09:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.629+0000 7fa302a58700 1 Processor -- start 2024-03-18T19:09:40.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.629+0000 7fa302a58700 1 -- start start 2024-03-18T19:09:40.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.630+0000 7fa302a58700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa2fc160a70 0x7fa2fc0d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:40.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.630+0000 7fa302a58700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2fc1612d0 0x7fa2fc1616f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:40.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.630+0000 7fa302a58700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2fc162170 con 0x7fa2fc160e50 2024-03-18T19:09:40.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.630+0000 7fa302a58700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2fc161c30 con 0x7fa2fc1612d0 2024-03-18T19:09:40.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.630+0000 7fa302a58700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2fc0d06e0 con 0x7fa2fc160a70 2024-03-18T19:09:40.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.630+0000 7fa300ff5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2fc1612d0 0x7fa2fc1616f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:40.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.630+0000 7fa300ff5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2fc1612d0 0x7fa2fc1616f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46200/0 (socket says 172.21.15.1:46200) 2024-03-18T19:09:40.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.630+0000 7fa300ff5700 1 -- 172.21.15.1:0/2307324578 learned_addr learned my addr 172.21.15.1:0/2307324578 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:40.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.630+0000 7fa2fbfff700 1 --2- 172.21.15.1:0/2307324578 >> v2:172.21.15.165:3300/0 conn(0x7fa2fc160a70 0x7fa2fc0d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:40.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.630+0000 7fa300ff5700 1 -- 172.21.15.1:0/2307324578 >> v2:172.21.15.165:3300/0 conn(0x7fa2fc160a70 msgr2=0x7fa2fc0d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:40.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.630+0000 7fa300ff5700 1 --2- 172.21.15.1:0/2307324578 >> v2:172.21.15.165:3300/0 conn(0x7fa2fc160a70 0x7fa2fc0d3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:40.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.630+0000 7fa300ff5700 1 -- 172.21.15.1:0/2307324578 >> v1:172.21.15.165:6789/0 conn(0x7fa2fc160e50 legacy=0x7fa2fc0cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:40.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.630+0000 7fa300ff5700 1 -- 172.21.15.1:0/2307324578 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2fc0cef30 con 0x7fa2fc1612d0 2024-03-18T19:09:40.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.630+0000 7fa300ff5700 1 --2- 172.21.15.1:0/2307324578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2fc1612d0 0x7fa2fc1616f0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc08d8f0 tx=0x7fa2f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4b891fff0139eb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:40.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.630+0000 7fa2faffd700 1 -- 172.21.15.1:0/2307324578 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f400e070 con 0x7fa2fc1612d0 2024-03-18T19:09:40.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.631+0000 7fa2faffd700 1 -- 172.21.15.1:0/2307324578 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f400b9c0 con 0x7fa2fc1612d0 2024-03-18T19:09:40.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.631+0000 7fa2faffd700 1 -- 172.21.15.1:0/2307324578 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f400bd70 con 0x7fa2fc1612d0 2024-03-18T19:09:40.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.631+0000 7fa302a58700 1 -- 172.21.15.1:0/2307324578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2fc1612d0 msgr2=0x7fa2fc1616f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.631+0000 7fa302a58700 1 --2- 172.21.15.1:0/2307324578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2fc1612d0 0x7fa2fc1616f0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc08d8f0 tx=0x7fa2f400b330 comp rx=0 tx=0).stop 2024-03-18T19:09:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.631+0000 7fa302a58700 1 -- 172.21.15.1:0/2307324578 shutdown_connections 2024-03-18T19:09:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.631+0000 7fa302a58700 1 --2- 172.21.15.1:0/2307324578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2fc1612d0 0x7fa2fc1616f0 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.631+0000 7fa302a58700 1 --2- 172.21.15.1:0/2307324578 >> v2:172.21.15.165:3300/0 conn(0x7fa2fc160a70 0x7fa2fc0d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.631+0000 7fa302a58700 1 -- 172.21.15.1:0/2307324578 >> 172.21.15.1:0/2307324578 conn(0x7fa2fc09c560 msgr2=0x7fa2fc0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.631+0000 7fa302a58700 1 -- 172.21.15.1:0/2307324578 shutdown_connections 2024-03-18T19:09:40.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.631+0000 7fa302a58700 1 -- 172.21.15.1:0/2307324578 wait complete. 2024-03-18T19:09:40.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.632+0000 7fa302a58700 1 Processor -- start 2024-03-18T19:09:40.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.633+0000 7fa302a58700 1 -- start start 2024-03-18T19:09:40.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.633+0000 7fa302a58700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2fc160a70 0x7fa2fc0b8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:40.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.633+0000 7fa302a58700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2fc160e50 0x7fa2fc0c81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:40.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.633+0000 7fa302a58700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2fc1612d0 0x7fa2fc0cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:40.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.633+0000 7fa302a58700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc0c8700 con 0x7fa2fc160e50 2024-03-18T19:09:40.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.633+0000 7fa302a58700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc0c6bd0 con 0x7fa2fc160a70 2024-03-18T19:09:40.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.633+0000 7fa302a58700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc0c8930 con 0x7fa2fc1612d0 2024-03-18T19:09:40.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.633+0000 7fa2fb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2fc160e50 0x7fa2fc0c81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:40.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.633+0000 7fa2fb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2fc160e50 0x7fa2fc0c81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46202/0 (socket says 172.21.15.1:46202) 2024-03-18T19:09:40.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.633+0000 7fa2fb7fe700 1 -- 172.21.15.1:0/2087340072 learned_addr learned my addr 172.21.15.1:0/2087340072 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:40.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.633+0000 7fa2fbfff700 1 --2- 172.21.15.1:0/2087340072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2fc160a70 0x7fa2fc0b8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:40.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.633+0000 7fa300ff5700 1 --2- 172.21.15.1:0/2087340072 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2fc1612d0 0x7fa2fc0cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:40.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.634+0000 7fa2fb7fe700 1 -- 172.21.15.1:0/2087340072 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2fc1612d0 msgr2=0x7fa2fc0cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:40.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.634+0000 7fa2fb7fe700 1 --2- 172.21.15.1:0/2087340072 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2fc1612d0 0x7fa2fc0cc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:40.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.634+0000 7fa2fb7fe700 1 -- 172.21.15.1:0/2087340072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2fc160a70 msgr2=0x7fa2fc0b8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:40.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.634+0000 7fa2fb7fe700 1 --2- 172.21.15.1:0/2087340072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2fc160a70 0x7fa2fc0b8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:40.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.634+0000 7fa2fb7fe700 1 -- 172.21.15.1:0/2087340072 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f4009b30 con 0x7fa2fc160e50 2024-03-18T19:09:40.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.634+0000 7fa2fb7fe700 1 --2- 172.21.15.1:0/2087340072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2fc160e50 0x7fa2fc0c81c0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00c480 tx=0x7fa2ec00c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:40.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.634+0000 7fa2f8ff9700 1 -- 172.21.15.1:0/2087340072 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec0052d0 con 0x7fa2fc160e50 2024-03-18T19:09:40.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.634+0000 7fa2f8ff9700 1 -- 172.21.15.1:0/2087340072 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ec00ce90 con 0x7fa2fc160e50 2024-03-18T19:09:40.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.634+0000 7fa302a58700 1 -- 172.21.15.1:0/2087340072 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2fc0c8c10 con 0x7fa2fc160e50 2024-03-18T19:09:40.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.634+0000 7fa2f8ff9700 1 -- 172.21.15.1:0/2087340072 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec003680 con 0x7fa2fc160e50 2024-03-18T19:09:40.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.634+0000 7fa302a58700 1 -- 172.21.15.1:0/2087340072 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2fc0c7170 con 0x7fa2fc160e50 2024-03-18T19:09:40.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.635+0000 7fa302a58700 1 -- 172.21.15.1:0/2087340072 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2fc0d24f0 con 0x7fa2fc160e50 2024-03-18T19:09:40.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.635+0000 7fa2f8ff9700 1 -- 172.21.15.1:0/2087340072 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 75) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fa2ec0037e0 con 0x7fa2fc160e50 2024-03-18T19:09:40.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.636+0000 7fa2f8ff9700 1 --2- 172.21.15.1:0/2087340072 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa2e806ab30 0x7fa2e806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:40.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.636+0000 7fa2f8ff9700 1 -- 172.21.15.1:0/2087340072 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fa2ec088470 con 0x7fa2fc160e50 2024-03-18T19:09:40.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.636+0000 7fa2fbfff700 1 --2- 172.21.15.1:0/2087340072 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa2e806ab30 0x7fa2e806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:40.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.636+0000 7fa2fbfff700 1 --2- 172.21.15.1:0/2087340072 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa2e806ab30 0x7fa2e806cff0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0006b20 tx=0x7fa2f0008040 comp rx=0 tx=0).ready entity=mgr.7262 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:40.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.640+0000 7fa2f8ff9700 1 -- 172.21.15.1:0/2087340072 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa2ec053600 con 0x7fa2fc160e50 2024-03-18T19:09:40.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.870+0000 7fa2f8ff9700 1 -- 172.21.15.1:0/2087340072 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 76) v1 ==== 89257+0+0 (secure 0 0 0) 0x7fa2ec0539d0 con 0x7fa2fc160e50 2024-03-18T19:09:40.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.872+0000 7fa302a58700 1 -- 172.21.15.1:0/2087340072 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/monthly", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2fc0cf4f0 con 0x7fa2e806ab30 2024-03-18T19:09:40.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.930+0000 7fa2f8ff9700 1 -- 172.21.15.1:0/2087340072 <== mgr.7262 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7fa2f00076b0 con 0x7fa2e806ab30 2024-03-18T19:09:40.926 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path snap_test_dir1/monthly 2024-03-18T19:09:40.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.932+0000 7fa302a58700 1 -- 172.21.15.1:0/2087340072 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa2e806ab30 msgr2=0x7fa2e806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:40.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.932+0000 7fa302a58700 1 --2- 172.21.15.1:0/2087340072 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa2e806ab30 0x7fa2e806cff0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0006b20 tx=0x7fa2f0008040 comp rx=0 tx=0).stop 2024-03-18T19:09:40.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.932+0000 7fa302a58700 1 -- 172.21.15.1:0/2087340072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2fc160e50 msgr2=0x7fa2fc0c81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:40.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.932+0000 7fa302a58700 1 --2- 172.21.15.1:0/2087340072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2fc160e50 0x7fa2fc0c81c0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00c480 tx=0x7fa2ec00c790 comp rx=0 tx=0).stop 2024-03-18T19:09:40.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.932+0000 7fa302a58700 1 -- 172.21.15.1:0/2087340072 shutdown_connections 2024-03-18T19:09:40.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.932+0000 7fa302a58700 1 --2- 172.21.15.1:0/2087340072 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa2e806ab30 0x7fa2e806cff0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:40.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.932+0000 7fa302a58700 1 --2- 172.21.15.1:0/2087340072 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2fc1612d0 0x7fa2fc0cc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:40.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.932+0000 7fa302a58700 1 --2- 172.21.15.1:0/2087340072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2fc160a70 0x7fa2fc0b8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:40.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.932+0000 7fa302a58700 1 --2- 172.21.15.1:0/2087340072 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2fc160e50 0x7fa2fc0c81c0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:40.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.933+0000 7fa302a58700 1 -- 172.21.15.1:0/2087340072 >> 172.21.15.1:0/2087340072 conn(0x7fa2fc09c560 msgr2=0x7fa2fc15b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:40.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.933+0000 7fa302a58700 1 -- 172.21.15.1:0/2087340072 shutdown_connections 2024-03-18T19:09:40.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:40.933+0000 7fa302a58700 1 -- 172.21.15.1:0/2087340072 wait complete. 2024-03-18T19:09:40.941 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/monthly 2024-03-18T19:09:40.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:40.941 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/yearly) 2024-03-18T19:09:40.962 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/yearly 1Y 2024-03-18T19:09:41.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.135+0000 7f0058bd1700 1 Processor -- start 2024-03-18T19:09:41.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.135+0000 7f0058bd1700 1 -- start start 2024-03-18T19:09:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.136+0000 7f0058bd1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0054071cc0 0x7f00540ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.136+0000 7f0058bd1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f005406d560 0x7f005406fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.136+0000 7f0058bd1700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0054070000 con 0x7f005415a240 2024-03-18T19:09:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.136+0000 7f0058bd1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00540adff0 con 0x7f005406d560 2024-03-18T19:09:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.136+0000 7f0058bd1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00540cac80 con 0x7f0054071cc0 2024-03-18T19:09:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.136+0000 7f0052d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f005406d560 0x7f005406fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.136+0000 7f005259c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f005415a240 0x7f00540715b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43798/0 (socket says 172.21.15.1:43798) 2024-03-18T19:09:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.136+0000 7f005259c700 1 -- 172.21.15.1:0/1143842305 learned_addr learned my addr 172.21.15.1:0/1143842305 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.136+0000 7f0051d9b700 1 --2- 172.21.15.1:0/1143842305 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0054071cc0 0x7f00540ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.136+0000 7f0052d9d700 1 -- 172.21.15.1:0/1143842305 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0054071cc0 msgr2=0x7f00540ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.136+0000 7f0052d9d700 1 --2- 172.21.15.1:0/1143842305 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0054071cc0 0x7f00540ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.136+0000 7f0052d9d700 1 -- 172.21.15.1:0/1143842305 >> v1:172.21.15.1:6789/0 conn(0x7f005415a240 legacy=0x7f00540715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.136+0000 7f005159a700 1 -- 172.21.15.1:0/1143842305 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971430008 0 0) 0x7f0054070000 con 0x7f005415a240 2024-03-18T19:09:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.136+0000 7f0052d9d700 1 -- 172.21.15.1:0/1143842305 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00540caf60 con 0x7f005406d560 2024-03-18T19:09:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.136+0000 7f0052d9d700 1 --2- 172.21.15.1:0/1143842305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f005406d560 0x7f005406fac0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f004c009d90 tx=0x7f004c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=268d515e757d8ba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.137+0000 7f005159a700 1 -- 172.21.15.1:0/1143842305 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c00e070 con 0x7f005406d560 2024-03-18T19:09:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.137+0000 7f005159a700 1 -- 172.21.15.1:0/1143842305 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f004c00bbd0 con 0x7f005406d560 2024-03-18T19:09:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.137+0000 7f005159a700 1 -- 172.21.15.1:0/1143842305 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c012430 con 0x7f005406d560 2024-03-18T19:09:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.137+0000 7f0058bd1700 1 -- 172.21.15.1:0/1143842305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f005406d560 msgr2=0x7f005406fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.137+0000 7f0058bd1700 1 --2- 172.21.15.1:0/1143842305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f005406d560 0x7f005406fac0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f004c009d90 tx=0x7f004c00b330 comp rx=0 tx=0).stop 2024-03-18T19:09:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.137+0000 7f0058bd1700 1 -- 172.21.15.1:0/1143842305 shutdown_connections 2024-03-18T19:09:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.137+0000 7f0058bd1700 1 --2- 172.21.15.1:0/1143842305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f005406d560 0x7f005406fac0 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.137+0000 7f0058bd1700 1 --2- 172.21.15.1:0/1143842305 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0054071cc0 0x7f00540ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.137+0000 7f0058bd1700 1 -- 172.21.15.1:0/1143842305 >> 172.21.15.1:0/1143842305 conn(0x7f00540cff30 msgr2=0x7f00540d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.137+0000 7f0058bd1700 1 -- 172.21.15.1:0/1143842305 shutdown_connections 2024-03-18T19:09:41.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.137+0000 7f0058bd1700 1 -- 172.21.15.1:0/1143842305 wait complete. 2024-03-18T19:09:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.139+0000 7f0058bd1700 1 Processor -- start 2024-03-18T19:09:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.139+0000 7f0058bd1700 1 -- start start 2024-03-18T19:09:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.139+0000 7f0058bd1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f005406d560 0x7f00540c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.139+0000 7f0058bd1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0054071cc0 0x7f0054158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.139+0000 7f0058bd1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f005415a240 0x7f00540ae820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.139+0000 7f0058bd1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00541f74b0 con 0x7f005415a240 2024-03-18T19:09:41.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.139+0000 7f0058bd1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00540c3a40 con 0x7f0054071cc0 2024-03-18T19:09:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.139+0000 7f0058bd1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00540c8160 con 0x7f005406d560 2024-03-18T19:09:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.139+0000 7f0052d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f005415a240 0x7f00540ae820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.139+0000 7f0052d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f005415a240 0x7f00540ae820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46214/0 (socket says 172.21.15.1:46214) 2024-03-18T19:09:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.140+0000 7f0052d9d700 1 -- 172.21.15.1:0/1929779968 learned_addr learned my addr 172.21.15.1:0/1929779968 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.140+0000 7f005259c700 1 --2- 172.21.15.1:0/1929779968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f005406d560 0x7f00540c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.140+0000 7f0051d9b700 1 --2- 172.21.15.1:0/1929779968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0054071cc0 0x7f0054158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.140+0000 7f0052d9d700 1 -- 172.21.15.1:0/1929779968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f005406d560 msgr2=0x7f00540c7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.140+0000 7f0052d9d700 1 --2- 172.21.15.1:0/1929779968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f005406d560 0x7f00540c7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.140+0000 7f0052d9d700 1 -- 172.21.15.1:0/1929779968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0054071cc0 msgr2=0x7f0054158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.140+0000 7f0052d9d700 1 --2- 172.21.15.1:0/1929779968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0054071cc0 0x7f0054158450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.140+0000 7f0052d9d700 1 -- 172.21.15.1:0/1929779968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f004c009a70 con 0x7f005415a240 2024-03-18T19:09:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.140+0000 7f0052d9d700 1 --2- 172.21.15.1:0/1929779968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f005415a240 0x7f00540ae820 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f004c013000 tx=0x7f004c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:41.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.140+0000 7f0042ffd700 1 -- 172.21.15.1:0/1929779968 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c012430 con 0x7f005415a240 2024-03-18T19:09:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.140+0000 7f0042ffd700 1 -- 172.21.15.1:0/1929779968 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f004c020070 con 0x7f005415a240 2024-03-18T19:09:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.140+0000 7f0058bd1700 1 -- 172.21.15.1:0/1929779968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00541f66f0 con 0x7f005415a240 2024-03-18T19:09:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.140+0000 7f0042ffd700 1 -- 172.21.15.1:0/1929779968 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c004280 con 0x7f005415a240 2024-03-18T19:09:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.140+0000 7f0058bd1700 1 -- 172.21.15.1:0/1929779968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00541f6cb0 con 0x7f005415a240 2024-03-18T19:09:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.141+0000 7f0058bd1700 1 -- 172.21.15.1:0/1929779968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00540676a0 con 0x7f005415a240 2024-03-18T19:09:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.142+0000 7f0042ffd700 1 -- 172.21.15.1:0/1929779968 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 76) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f004c0043e0 con 0x7f005415a240 2024-03-18T19:09:41.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.142+0000 7f0042ffd700 1 --2- 172.21.15.1:0/1929779968 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f003406ae00 0x7f003406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:41.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.142+0000 7f0042ffd700 1 -- 172.21.15.1:0/1929779968 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f004c022070 con 0x7f005415a240 2024-03-18T19:09:41.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.142+0000 7f005259c700 1 --2- 172.21.15.1:0/1929779968 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f003406ae00 0x7f003406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:41.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.143+0000 7f005259c700 1 --2- 172.21.15.1:0/1929779968 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f003406ae00 0x7f003406d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0044000f60 tx=0x7f0044007e70 comp rx=0 tx=0).ready entity=mgr.7262 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:41.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.147+0000 7f0042ffd700 1 -- 172.21.15.1:0/1929779968 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f004c056800 con 0x7f005415a240 2024-03-18T19:09:41.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.379+0000 7f0058bd1700 1 -- 172.21.15.1:0/1929779968 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/yearly", "snap_schedule": "1Y", "target": ["mon-mgr", ""]}) v1 -- 0x7f005415a620 con 0x7f003406ae00 2024-03-18T19:09:41.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.439+0000 7f0042ffd700 1 -- 172.21.15.1:0/1929779968 <== mgr.7262 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f0044004ad0 con 0x7f003406ae00 2024-03-18T19:09:41.435 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path snap_test_dir1/yearly 2024-03-18T19:09:41.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.441+0000 7f0058bd1700 1 -- 172.21.15.1:0/1929779968 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f003406ae00 msgr2=0x7f003406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.441+0000 7f0058bd1700 1 --2- 172.21.15.1:0/1929779968 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f003406ae00 0x7f003406d2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0044000f60 tx=0x7f0044007e70 comp rx=0 tx=0).stop 2024-03-18T19:09:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.441+0000 7f0058bd1700 1 -- 172.21.15.1:0/1929779968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f005415a240 msgr2=0x7f00540ae820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.441+0000 7f0058bd1700 1 --2- 172.21.15.1:0/1929779968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f005415a240 0x7f00540ae820 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f004c013000 tx=0x7f004c004ad0 comp rx=0 tx=0).stop 2024-03-18T19:09:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.441+0000 7f0058bd1700 1 -- 172.21.15.1:0/1929779968 shutdown_connections 2024-03-18T19:09:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.441+0000 7f0058bd1700 1 --2- 172.21.15.1:0/1929779968 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f003406ae00 0x7f003406d2c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.441+0000 7f0058bd1700 1 --2- 172.21.15.1:0/1929779968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f005406d560 0x7f00540c7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.441+0000 7f0058bd1700 1 --2- 172.21.15.1:0/1929779968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f005415a240 0x7f00540ae820 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.441+0000 7f0058bd1700 1 --2- 172.21.15.1:0/1929779968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0054071cc0 0x7f0054158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.441+0000 7f0058bd1700 1 -- 172.21.15.1:0/1929779968 >> 172.21.15.1:0/1929779968 conn(0x7f00540cff30 msgr2=0x7f00541594a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.441+0000 7f0058bd1700 1 -- 172.21.15.1:0/1929779968 shutdown_connections 2024-03-18T19:09:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.441+0000 7f0058bd1700 1 -- 172.21.15.1:0/1929779968 wait complete. 2024-03-18T19:09:41.449 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/yearly 2024-03-18T19:09:41.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:41.450 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/bad_period_spec) 2024-03-18T19:09:41.470 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1X 2024-03-18T19:09:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.643+0000 7fe9e3bd8700 1 Processor -- start 2024-03-18T19:09:41.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.643+0000 7fe9e3bd8700 1 -- start start 2024-03-18T19:09:41.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.644+0000 7fe9e3bd8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe9dc15a1d0 0x7fe9dc0add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:41.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.644+0000 7fe9e3bd8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe9dc071620 0x7fe9dc071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:41.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.644+0000 7fe9e3bd8700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9dc0ce8f0 con 0x7fe9dc0cb9e0 2024-03-18T19:09:41.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.644+0000 7fe9e3bd8700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9dc0cdb00 con 0x7fe9dc071620 2024-03-18T19:09:41.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.644+0000 7fe9e3bd8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9dc15a5b0 con 0x7fe9dc15a1d0 2024-03-18T19:09:41.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.644+0000 7fe9e1173700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe9dc071620 0x7fe9dc071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.644+0000 7fe9e2175700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe9dc0cb9e0 0x7fe9dc0cbe70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43802/0 (socket says 172.21.15.1:43802) 2024-03-18T19:09:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.644+0000 7fe9e2175700 1 -- 172.21.15.1:0/394467307 learned_addr learned my addr 172.21.15.1:0/394467307 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.644+0000 7fe9e1173700 1 -- 172.21.15.1:0/394467307 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe9dc15a1d0 msgr2=0x7fe9dc0add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.644+0000 7fe9e1173700 1 --2- 172.21.15.1:0/394467307 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe9dc15a1d0 0x7fe9dc0add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.644+0000 7fe9e1173700 1 -- 172.21.15.1:0/394467307 >> v1:172.21.15.1:6789/0 conn(0x7fe9dc0cb9e0 legacy=0x7fe9dc0cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.644+0000 7fe9e1173700 1 -- 172.21.15.1:0/394467307 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9dc0ae260 con 0x7fe9dc071620 2024-03-18T19:09:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.644+0000 7fe9e0972700 1 -- 172.21.15.1:0/394467307 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1575499511 0 0) 0x7fe9dc0ce8f0 con 0x7fe9dc0cb9e0 2024-03-18T19:09:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.644+0000 7fe9e1173700 1 --2- 172.21.15.1:0/394467307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9dc071620 0x7fe9dc071aa0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8009840 tx=0x7fe9d8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d86b705e8992454c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.645+0000 7fe9e0972700 1 -- 172.21.15.1:0/394467307 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8011830 con 0x7fe9dc071620 2024-03-18T19:09:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.645+0000 7fe9e0972700 1 -- 172.21.15.1:0/394467307 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d8011990 con 0x7fe9dc071620 2024-03-18T19:09:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.645+0000 7fe9e0972700 1 -- 172.21.15.1:0/394467307 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8011d20 con 0x7fe9dc071620 2024-03-18T19:09:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.645+0000 7fe9e3bd8700 1 -- 172.21.15.1:0/394467307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9dc071620 msgr2=0x7fe9dc071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:41.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.645+0000 7fe9e3bd8700 1 --2- 172.21.15.1:0/394467307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9dc071620 0x7fe9dc071aa0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8009840 tx=0x7fe9d8009b50 comp rx=0 tx=0).stop 2024-03-18T19:09:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.645+0000 7fe9e3bd8700 1 -- 172.21.15.1:0/394467307 shutdown_connections 2024-03-18T19:09:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.645+0000 7fe9e3bd8700 1 --2- 172.21.15.1:0/394467307 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe9dc15a1d0 0x7fe9dc0add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.645+0000 7fe9e3bd8700 1 --2- 172.21.15.1:0/394467307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9dc071620 0x7fe9dc071aa0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.645+0000 7fe9e3bd8700 1 -- 172.21.15.1:0/394467307 >> 172.21.15.1:0/394467307 conn(0x7fe9dc0cc9b0 msgr2=0x7fe9dc0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.645+0000 7fe9e3bd8700 1 -- 172.21.15.1:0/394467307 shutdown_connections 2024-03-18T19:09:41.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.645+0000 7fe9e3bd8700 1 -- 172.21.15.1:0/394467307 wait complete. 2024-03-18T19:09:41.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.646+0000 7fe9e3bd8700 1 Processor -- start 2024-03-18T19:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.646+0000 7fe9e3bd8700 1 -- start start 2024-03-18T19:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.647+0000 7fe9e3bd8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9dc071620 0x7fe9dc0b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.647+0000 7fe9e3bd8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe9dc0cb9e0 0x7fe9dc15cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.647+0000 7fe9e3bd8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe9dc15a1d0 0x7fe9dc154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.647+0000 7fe9e3bd8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9dc0b4040 con 0x7fe9dc071620 2024-03-18T19:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.647+0000 7fe9e3bd8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9dc0b4320 con 0x7fe9dc0cb9e0 2024-03-18T19:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.647+0000 7fe9e3bd8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9dc0b4600 con 0x7fe9dc15a1d0 2024-03-18T19:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.647+0000 7fe9e1974700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9dc071620 0x7fe9dc0b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.647+0000 7fe9e1974700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9dc071620 0x7fe9dc0b50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46238/0 (socket says 172.21.15.1:46238) 2024-03-18T19:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.647+0000 7fe9e1974700 1 -- 172.21.15.1:0/3963260023 learned_addr learned my addr 172.21.15.1:0/3963260023 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.647+0000 7fe9e2175700 1 --2- 172.21.15.1:0/3963260023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe9dc15a1d0 0x7fe9dc154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:41.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.647+0000 7fe9e1173700 1 --2- 172.21.15.1:0/3963260023 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe9dc0cb9e0 0x7fe9dc15cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.648+0000 7fe9e1974700 1 -- 172.21.15.1:0/3963260023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe9dc15a1d0 msgr2=0x7fe9dc154250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.648+0000 7fe9e1974700 1 --2- 172.21.15.1:0/3963260023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe9dc15a1d0 0x7fe9dc154250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.648+0000 7fe9e1974700 1 -- 172.21.15.1:0/3963260023 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe9dc0cb9e0 msgr2=0x7fe9dc15cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.648+0000 7fe9e1974700 1 --2- 172.21.15.1:0/3963260023 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe9dc0cb9e0 0x7fe9dc15cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.648+0000 7fe9e1974700 1 -- 172.21.15.1:0/3963260023 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d80094a0 con 0x7fe9dc071620 2024-03-18T19:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.648+0000 7fe9e2175700 1 --2- 172.21.15.1:0/3963260023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe9dc15a1d0 0x7fe9dc154250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.648+0000 7fe9e1974700 1 --2- 172.21.15.1:0/3963260023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9dc071620 0x7fe9dc0b50e0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc00b520 tx=0x7fe9cc00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.648+0000 7fe9d27fc700 1 -- 172.21.15.1:0/3963260023 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc005aa0 con 0x7fe9dc071620 2024-03-18T19:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.648+0000 7fe9d27fc700 1 -- 172.21.15.1:0/3963260023 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9cc005c00 con 0x7fe9dc071620 2024-03-18T19:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.648+0000 7fe9e3bd8700 1 -- 172.21.15.1:0/3963260023 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9dc0b48e0 con 0x7fe9dc071620 2024-03-18T19:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.648+0000 7fe9d27fc700 1 -- 172.21.15.1:0/3963260023 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc011630 con 0x7fe9dc071620 2024-03-18T19:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.648+0000 7fe9e3bd8700 1 -- 172.21.15.1:0/3963260023 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9dc0b5620 con 0x7fe9dc071620 2024-03-18T19:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.649+0000 7fe9e3bd8700 1 -- 172.21.15.1:0/3963260023 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9dc0b4e00 con 0x7fe9dc071620 2024-03-18T19:09:41.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.649+0000 7fe9d27fc700 1 -- 172.21.15.1:0/3963260023 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 76) v1 ==== 89257+0+0 (secure 0 0 0) 0x7fe9cc011790 con 0x7fe9dc071620 2024-03-18T19:09:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.650+0000 7fe9d27fc700 1 --2- 172.21.15.1:0/3963260023 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe9c006ae00 0x7fe9c006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.650+0000 7fe9d27fc700 1 -- 172.21.15.1:0/3963260023 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fe9cc0893f0 con 0x7fe9dc071620 2024-03-18T19:09:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.650+0000 7fe9e1173700 1 --2- 172.21.15.1:0/3963260023 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe9c006ae00 0x7fe9c006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:41.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.650+0000 7fe9e1173700 1 --2- 172.21.15.1:0/3963260023 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe9c006ae00 0x7fe9c006d2c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe9d80090b0 tx=0x7fe9d801c040 comp rx=0 tx=0).ready entity=mgr.7262 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:41.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.654+0000 7fe9d27fc700 1 -- 172.21.15.1:0/3963260023 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe9dc0b4e00 con 0x7fe9dc071620 2024-03-18T19:09:41.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.885+0000 7fe9e3bd8700 1 -- 172.21.15.1:0/3963260023 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1X", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9dc0b4490 con 0x7fe9c006ae00 2024-03-18T19:09:41.882 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:41.886+0000 7f911fe08700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - multiplier should be one of h,d,w,M,Y 2024-03-18T19:09:41.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:41.886+0000 7fe9d27fc700 1 -- 172.21.15.1:0/3963260023 <== mgr.7262 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - multiplier should be one of h,d,w,M,Y) v1 ==== 74+0+0 (secure 0 0 0) 0x7fe9dc0b4490 con 0x7fe9c006ae00 2024-03-18T19:09:41.884 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENOENT: invalid schedule specified - multiplier should be one of h,d,w,M,Y 2024-03-18T19:09:41.887 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-18T19:09:41.888 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1MM 2024-03-18T19:09:42.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.021+0000 7f50a1a7d700 1 Processor -- start 2024-03-18T19:09:42.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.021+0000 7f50a1a7d700 1 -- start start 2024-03-18T19:09:42.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.022+0000 7f50a1a7d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f509c0cd520 0x7f509c0cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.022+0000 7f50a1a7d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f509c0cbf00 0x7f509c0cc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.022+0000 7f50a1a7d700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f509c0cf2c0 con 0x7f509c15b4d0 2024-03-18T19:09:42.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.022+0000 7f50a1a7d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f509c0725f0 con 0x7f509c0cbf00 2024-03-18T19:09:42.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.022+0000 7f50a1a7d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f509c0cf750 con 0x7f509c0cd520 2024-03-18T19:09:42.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.022+0000 7f509bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f509c0cbf00 0x7f509c0cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.022+0000 7f509bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f509c0cbf00 0x7f509c0cc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46252/0 (socket says 172.21.15.1:46252) 2024-03-18T19:09:42.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.022+0000 7f509bfff700 1 -- 172.21.15.1:0/3748214656 learned_addr learned my addr 172.21.15.1:0/3748214656 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:42.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.022+0000 7f509affd700 1 --2- 172.21.15.1:0/3748214656 >> v2:172.21.15.165:3300/0 conn(0x7f509c0cd520 0x7f509c0cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.022+0000 7f509bfff700 1 -- 172.21.15.1:0/3748214656 >> v2:172.21.15.165:3300/0 conn(0x7f509c0cd520 msgr2=0x7f509c0cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:42.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.022+0000 7f509bfff700 1 --2- 172.21.15.1:0/3748214656 >> v2:172.21.15.165:3300/0 conn(0x7f509c0cd520 0x7f509c0cf9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:42.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.022+0000 7f509bfff700 1 -- 172.21.15.1:0/3748214656 >> v1:172.21.15.165:6789/0 conn(0x7f509c15b4d0 legacy=0x7f509c071ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:42.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.022+0000 7f509bfff700 1 -- 172.21.15.1:0/3748214656 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f509c0ca5d0 con 0x7f509c0cbf00 2024-03-18T19:09:42.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.023+0000 7f509bfff700 1 --2- 172.21.15.1:0/3748214656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f509c0cbf00 0x7f509c0cc370 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f509c073f40 tx=0x7f509400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3218f2923f57cc89 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:42.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.023+0000 7f509a7fc700 1 -- 172.21.15.1:0/3748214656 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509400e070 con 0x7f509c0cbf00 2024-03-18T19:09:42.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.023+0000 7f509a7fc700 1 -- 172.21.15.1:0/3748214656 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509400b9c0 con 0x7f509c0cbf00 2024-03-18T19:09:42.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.023+0000 7f509a7fc700 1 -- 172.21.15.1:0/3748214656 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509400bd70 con 0x7f509c0cbf00 2024-03-18T19:09:42.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.023+0000 7f50a1a7d700 1 -- 172.21.15.1:0/3748214656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f509c0cbf00 msgr2=0x7f509c0cc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:42.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.023+0000 7f50a1a7d700 1 --2- 172.21.15.1:0/3748214656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f509c0cbf00 0x7f509c0cc370 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f509c073f40 tx=0x7f509400b330 comp rx=0 tx=0).stop 2024-03-18T19:09:42.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.023+0000 7f50a1a7d700 1 -- 172.21.15.1:0/3748214656 shutdown_connections 2024-03-18T19:09:42.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.023+0000 7f50a1a7d700 1 --2- 172.21.15.1:0/3748214656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f509c0cbf00 0x7f509c0cc370 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:42.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.023+0000 7f50a1a7d700 1 --2- 172.21.15.1:0/3748214656 >> v2:172.21.15.165:3300/0 conn(0x7f509c0cd520 0x7f509c0cf9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:42.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.023+0000 7f50a1a7d700 1 -- 172.21.15.1:0/3748214656 >> 172.21.15.1:0/3748214656 conn(0x7f509c0cca00 msgr2=0x7f509c0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:42.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.023+0000 7f50a1a7d700 1 -- 172.21.15.1:0/3748214656 shutdown_connections 2024-03-18T19:09:42.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.023+0000 7f50a1a7d700 1 -- 172.21.15.1:0/3748214656 wait complete. 2024-03-18T19:09:42.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.025+0000 7f50a1a7d700 1 Processor -- start 2024-03-18T19:09:42.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.025+0000 7f50a1a7d700 1 -- start start 2024-03-18T19:09:42.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f50a1a7d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f509c0cbf00 0x7f509c150840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f50a1a7d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f509c0cd520 0x7f509c0cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f50a1a7d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f509c15b4d0 0x7f509c0ce090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f50a1a7d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509c06e490 con 0x7f509c15b4d0 2024-03-18T19:09:42.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f50a1a7d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509c06e920 con 0x7f509c0cd520 2024-03-18T19:09:42.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f50a1a7d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509c06d880 con 0x7f509c0cbf00 2024-03-18T19:09:42.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f509bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f509c15b4d0 0x7f509c0ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f509bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f509c15b4d0 0x7f509c0ce090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46258/0 (socket says 172.21.15.1:46258) 2024-03-18T19:09:42.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f509bfff700 1 -- 172.21.15.1:0/1007053438 learned_addr learned my addr 172.21.15.1:0/1007053438 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:42.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f509affd700 1 --2- 172.21.15.1:0/1007053438 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f509c0cd520 0x7f509c0cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f509b7fe700 1 --2- 172.21.15.1:0/1007053438 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f509c0cbf00 0x7f509c150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f509bfff700 1 -- 172.21.15.1:0/1007053438 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f509c0cbf00 msgr2=0x7f509c150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:42.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f509bfff700 1 --2- 172.21.15.1:0/1007053438 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f509c0cbf00 0x7f509c150840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:42.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f509bfff700 1 -- 172.21.15.1:0/1007053438 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f509c0cd520 msgr2=0x7f509c0cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:42.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f509bfff700 1 --2- 172.21.15.1:0/1007053438 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f509c0cd520 0x7f509c0cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:42.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f509bfff700 1 -- 172.21.15.1:0/1007053438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5094009b30 con 0x7f509c15b4d0 2024-03-18T19:09:42.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f509bfff700 1 --2- 172.21.15.1:0/1007053438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f509c15b4d0 0x7f509c0ce090 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f509400b370 tx=0x7f5094004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:42.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f5083fff700 1 -- 172.21.15.1:0/1007053438 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5094004880 con 0x7f509c15b4d0 2024-03-18T19:09:42.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f5083fff700 1 -- 172.21.15.1:0/1007053438 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50940099b0 con 0x7f509c15b4d0 2024-03-18T19:09:42.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f50a1a7d700 1 -- 172.21.15.1:0/1007053438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509c06db60 con 0x7f509c15b4d0 2024-03-18T19:09:42.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f5083fff700 1 -- 172.21.15.1:0/1007053438 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509401c680 con 0x7f509c15b4d0 2024-03-18T19:09:42.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.026+0000 7f50a1a7d700 1 -- 172.21.15.1:0/1007053438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f509c06e120 con 0x7f509c15b4d0 2024-03-18T19:09:42.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.028+0000 7f50a1a7d700 1 -- 172.21.15.1:0/1007053438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f509c06e2b0 con 0x7f509c15b4d0 2024-03-18T19:09:42.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.028+0000 7f5083fff700 1 -- 172.21.15.1:0/1007053438 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 76) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f509401c7e0 con 0x7f509c15b4d0 2024-03-18T19:09:42.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.028+0000 7f5083fff700 1 --2- 172.21.15.1:0/1007053438 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f508806ae00 0x7f508806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.028+0000 7f5083fff700 1 -- 172.21.15.1:0/1007053438 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f509408b390 con 0x7f509c15b4d0 2024-03-18T19:09:42.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.028+0000 7f509b7fe700 1 --2- 172.21.15.1:0/1007053438 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f508806ae00 0x7f508806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.029+0000 7f509b7fe700 1 --2- 172.21.15.1:0/1007053438 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f508806ae00 0x7f508806d2c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f509c15b9d0 tx=0x7f508c009040 comp rx=0 tx=0).ready entity=mgr.7262 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:42.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.033+0000 7f5083fff700 1 -- 172.21.15.1:0/1007053438 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5094056120 con 0x7f509c15b4d0 2024-03-18T19:09:42.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.261+0000 7f50a1a7d700 1 -- 172.21.15.1:0/1007053438 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1MM", "target": ["mon-mgr", ""]}) v1 -- 0x7f509c0af700 con 0x7f508806ae00 2024-03-18T19:09:42.258 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:42.262+0000 7f911fe08700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-03-18T19:09:42.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.262+0000 7f5083fff700 1 -- 172.21.15.1:0/1007053438 <== mgr.7262 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f508c001580 con 0x7f508806ae00 2024-03-18T19:09:42.259 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-03-18T19:09:42.262 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-18T19:09:42.263 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1 2024-03-18T19:09:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.396+0000 7f6c6193b700 1 Processor -- start 2024-03-18T19:09:42.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.396+0000 7f6c6193b700 1 -- start start 2024-03-18T19:09:42.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.397+0000 7f6c6193b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6c5c0b0500 0x7f6c5c0cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.397+0000 7f6c6193b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6c5c0b0e20 0x7f6c5c0b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.397+0000 7f6c6193b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5c15ef80 con 0x7f6c5c15eba0 2024-03-18T19:09:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.397+0000 7f6c6193b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c5c0cc9a0 con 0x7f6c5c0b0500 2024-03-18T19:09:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.397+0000 7f6c6193b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c5c0ccc80 con 0x7f6c5c0b0e20 2024-03-18T19:09:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.397+0000 7f6c5a7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6c5c0b0500 0x7f6c5c0cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.397+0000 7f6c5affd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6c5c15eba0 0x7f6c5c0cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43818/0 (socket says 172.21.15.1:43818) 2024-03-18T19:09:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.397+0000 7f6c5affd700 1 -- 172.21.15.1:0/2763952882 learned_addr learned my addr 172.21.15.1:0/2763952882 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.397+0000 7f6c5a7fc700 1 -- 172.21.15.1:0/2763952882 >> v2:172.21.15.165:3300/0 conn(0x7f6c5c0b0e20 msgr2=0x7f6c5c0b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.397+0000 7f6c5b7fe700 1 --2- 172.21.15.1:0/2763952882 >> v2:172.21.15.165:3300/0 conn(0x7f6c5c0b0e20 0x7f6c5c0b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.397+0000 7f6c5a7fc700 1 --2- 172.21.15.1:0/2763952882 >> v2:172.21.15.165:3300/0 conn(0x7f6c5c0b0e20 0x7f6c5c0b1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.397+0000 7f6c5a7fc700 1 -- 172.21.15.1:0/2763952882 >> v1:172.21.15.1:6789/0 conn(0x7f6c5c15eba0 legacy=0x7f6c5c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.397+0000 7f6c5a7fc700 1 -- 172.21.15.1:0/2763952882 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5c0ccf60 con 0x7f6c5c0b0500 2024-03-18T19:09:42.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.397+0000 7f6c59ffb700 1 -- 172.21.15.1:0/2763952882 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711884009 0 0) 0x7f6c5c15ef80 con 0x7f6c5c15eba0 2024-03-18T19:09:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.398+0000 7f6c5a7fc700 1 --2- 172.21.15.1:0/2763952882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c5c0b0500 0x7f6c5c0cfa80 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c0720b0 tx=0x7f6c4c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a725dd683140f0b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.398+0000 7f6c59ffb700 1 -- 172.21.15.1:0/2763952882 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c011830 con 0x7f6c5c0b0500 2024-03-18T19:09:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.398+0000 7f6c59ffb700 1 -- 172.21.15.1:0/2763952882 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4c011990 con 0x7f6c5c0b0500 2024-03-18T19:09:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.398+0000 7f6c59ffb700 1 -- 172.21.15.1:0/2763952882 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c011d20 con 0x7f6c5c0b0500 2024-03-18T19:09:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.398+0000 7f6c6193b700 1 -- 172.21.15.1:0/2763952882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c5c0b0500 msgr2=0x7f6c5c0cfa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.398+0000 7f6c6193b700 1 --2- 172.21.15.1:0/2763952882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c5c0b0500 0x7f6c5c0cfa80 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c0720b0 tx=0x7f6c4c009b20 comp rx=0 tx=0).stop 2024-03-18T19:09:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.398+0000 7f6c6193b700 1 -- 172.21.15.1:0/2763952882 shutdown_connections 2024-03-18T19:09:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.398+0000 7f6c6193b700 1 --2- 172.21.15.1:0/2763952882 >> v2:172.21.15.165:3300/0 conn(0x7f6c5c0b0e20 0x7f6c5c0b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.398+0000 7f6c6193b700 1 --2- 172.21.15.1:0/2763952882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c5c0b0500 0x7f6c5c0cfa80 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.398+0000 7f6c6193b700 1 -- 172.21.15.1:0/2763952882 >> 172.21.15.1:0/2763952882 conn(0x7f6c5c0b2bd0 msgr2=0x7f6c5c0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.398+0000 7f6c6193b700 1 -- 172.21.15.1:0/2763952882 shutdown_connections 2024-03-18T19:09:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.398+0000 7f6c6193b700 1 -- 172.21.15.1:0/2763952882 wait complete. 2024-03-18T19:09:42.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.400+0000 7f6c6193b700 1 Processor -- start 2024-03-18T19:09:42.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.400+0000 7f6c6193b700 1 -- start start 2024-03-18T19:09:42.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.400+0000 7f6c6193b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c5c0b0500 0x7f6c5c1f60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.400+0000 7f6c6193b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c5c0b0e20 0x7f6c5c1f6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.400+0000 7f6c6193b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c5c15eba0 0x7f6c5c159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.400+0000 7f6c6193b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c0c8380 con 0x7f6c5c0b0e20 2024-03-18T19:09:42.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.400+0000 7f6c6193b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c0c9e10 con 0x7f6c5c15eba0 2024-03-18T19:09:42.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.400+0000 7f6c6193b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c0c9830 con 0x7f6c5c0b0500 2024-03-18T19:09:42.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.401+0000 7f6c5a7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c5c0b0e20 0x7f6c5c1f6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.401+0000 7f6c5a7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c5c0b0e20 0x7f6c5c1f6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46286/0 (socket says 172.21.15.1:46286) 2024-03-18T19:09:42.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.401+0000 7f6c5a7fc700 1 -- 172.21.15.1:0/3058639520 learned_addr learned my addr 172.21.15.1:0/3058639520 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:42.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.401+0000 7f6c5affd700 1 --2- 172.21.15.1:0/3058639520 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c5c0b0500 0x7f6c5c1f60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.401+0000 7f6c5b7fe700 1 --2- 172.21.15.1:0/3058639520 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c5c15eba0 0x7f6c5c159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.401+0000 7f6c5a7fc700 1 -- 172.21.15.1:0/3058639520 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c5c0b0500 msgr2=0x7f6c5c1f60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:42.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.401+0000 7f6c5a7fc700 1 --2- 172.21.15.1:0/3058639520 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c5c0b0500 0x7f6c5c1f60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:42.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.401+0000 7f6c5a7fc700 1 -- 172.21.15.1:0/3058639520 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c5c15eba0 msgr2=0x7f6c5c159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:42.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.401+0000 7f6c5a7fc700 1 --2- 172.21.15.1:0/3058639520 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c5c15eba0 0x7f6c5c159770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:42.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.401+0000 7f6c5a7fc700 1 -- 172.21.15.1:0/3058639520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c4c0094a0 con 0x7f6c5c0b0e20 2024-03-18T19:09:42.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.401+0000 7f6c5affd700 1 --2- 172.21.15.1:0/3058639520 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c5c0b0500 0x7f6c5c1f60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:42.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.401+0000 7f6c5b7fe700 1 --2- 172.21.15.1:0/3058639520 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c5c15eba0 0x7f6c5c159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:42.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.401+0000 7f6c5a7fc700 1 --2- 172.21.15.1:0/3058639520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c5c0b0e20 0x7f6c5c1f6750 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c0090b0 tx=0x7f6c4c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:42.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.401+0000 7f6c43fff700 1 -- 172.21.15.1:0/3058639520 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c0038a0 con 0x7f6c5c0b0e20 2024-03-18T19:09:42.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.401+0000 7f6c43fff700 1 -- 172.21.15.1:0/3058639520 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4c01f070 con 0x7f6c5c0b0e20 2024-03-18T19:09:42.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.401+0000 7f6c6193b700 1 -- 172.21.15.1:0/3058639520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c5c1f7210 con 0x7f6c5c0b0e20 2024-03-18T19:09:42.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.401+0000 7f6c43fff700 1 -- 172.21.15.1:0/3058639520 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c003c40 con 0x7f6c5c0b0e20 2024-03-18T19:09:42.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.401+0000 7f6c6193b700 1 -- 172.21.15.1:0/3058639520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c5c0c9aa0 con 0x7f6c5c0b0e20 2024-03-18T19:09:42.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.402+0000 7f6c6193b700 1 -- 172.21.15.1:0/3058639520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c5c0c9c60 con 0x7f6c5c0b0e20 2024-03-18T19:09:42.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.403+0000 7f6c43fff700 1 -- 172.21.15.1:0/3058639520 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 76) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f6c4c026480 con 0x7f6c5c0b0e20 2024-03-18T19:09:42.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.403+0000 7f6c43fff700 1 --2- 172.21.15.1:0/3058639520 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c4806ae00 0x7f6c4806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.403+0000 7f6c43fff700 1 -- 172.21.15.1:0/3058639520 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f6c4c021070 con 0x7f6c5c0b0e20 2024-03-18T19:09:42.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.403+0000 7f6c5affd700 1 --2- 172.21.15.1:0/3058639520 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c4806ae00 0x7f6c4806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.404+0000 7f6c5affd700 1 --2- 172.21.15.1:0/3058639520 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c4806ae00 0x7f6c4806d2c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6c50007fd0 tx=0x7f6c50009040 comp rx=0 tx=0).ready entity=mgr.7262 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:42.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.408+0000 7f6c43fff700 1 -- 172.21.15.1:0/3058639520 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6c4c0be580 con 0x7f6c5c0b0e20 2024-03-18T19:09:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.637+0000 7f6c6193b700 1 -- 172.21.15.1:0/3058639520 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c5c0b17d0 con 0x7f6c4806ae00 2024-03-18T19:09:42.634 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:42.638+0000 7f911fe08700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-03-18T19:09:42.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.638+0000 7f6c43fff700 1 -- 172.21.15.1:0/3058639520 <== mgr.7262 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6c50001370 con 0x7f6c4806ae00 2024-03-18T19:09:42.636 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-03-18T19:09:42.639 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-18T19:09:42.640 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec M 2024-03-18T19:09:42.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.772+0000 7f1f6663f700 1 Processor -- start 2024-03-18T19:09:42.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.772+0000 7f1f6663f700 1 -- start start 2024-03-18T19:09:42.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.773+0000 7f1f6663f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1f6009a530 0x7f1f600cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.773+0000 7f1f6663f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f600d0730 con 0x7f1f600d0cd0 2024-03-18T19:09:42.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.773+0000 7f1f6663f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f6006bb10 con 0x7f1f60099a40 2024-03-18T19:09:42.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.773+0000 7f1f6663f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f6006bdf0 con 0x7f1f6009a530 2024-03-18T19:09:42.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.773+0000 7f1f5f7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1f600d0cd0 0x7f1f600cc1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43834/0 (socket says 172.21.15.1:43834) 2024-03-18T19:09:42.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.773+0000 7f1f5f7fe700 1 -- 172.21.15.1:0/20399595 learned_addr learned my addr 172.21.15.1:0/20399595 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:42.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.773+0000 7f1f64bdc700 1 --2- 172.21.15.1:0/20399595 >> v2:172.21.15.165:3300/0 conn(0x7f1f6009a530 0x7f1f600cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.773+0000 7f1f5effd700 1 -- 172.21.15.1:0/20399595 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234547519 0 0) 0x7f1f600d0730 con 0x7f1f600d0cd0 2024-03-18T19:09:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.773+0000 7f1f5effd700 1 -- 172.21.15.1:0/20399595 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f480035b0 con 0x7f1f600d0cd0 2024-03-18T19:09:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.773+0000 7f1f5effd700 1 -- 172.21.15.1:0/20399595 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4217046563 0 0) 0x7f1f480035b0 con 0x7f1f600d0cd0 2024-03-18T19:09:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.773+0000 7f1f5effd700 1 -- 172.21.15.1:0/20399595 >> v2:172.21.15.165:3300/0 conn(0x7f1f6009a530 msgr2=0x7f1f600cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.773+0000 7f1f5effd700 1 --2- 172.21.15.1:0/20399595 >> v2:172.21.15.165:3300/0 conn(0x7f1f6009a530 0x7f1f600cc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.773+0000 7f1f5effd700 1 -- 172.21.15.1:0/20399595 >> v1:172.21.15.165:6789/0 conn(0x7f1f60099a40 legacy=0x7f1f60099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f5effd700 1 -- 172.21.15.1:0/20399595 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6006c410 con 0x7f1f600d0cd0 2024-03-18T19:09:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f5effd700 1 -- 172.21.15.1:0/20399595 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f1f50001670 con 0x7f1f600d0cd0 2024-03-18T19:09:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f5effd700 1 -- 172.21.15.1:0/20399595 >> v1:172.21.15.1:6789/0 conn(0x7f1f600d0cd0 legacy=0x7f1f600cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f5effd700 1 --2- 172.21.15.1:0/20399595 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f48003a00 0x7f1f48003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f5effd700 1 --2- 172.21.15.1:0/20399595 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f48004850 0x7f1f48004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f5effd700 1 --2- 172.21.15.1:0/20399595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f48005210 0x7f1f4800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f5effd700 1 -- 172.21.15.1:0/20399595 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6006bdf0 con 0x7f1f48005210 2024-03-18T19:09:42.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f5effd700 1 -- 172.21.15.1:0/20399595 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4800c1b0 con 0x7f1f48003a00 2024-03-18T19:09:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f5effd700 1 -- 172.21.15.1:0/20399595 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4800c370 con 0x7f1f48004850 2024-03-18T19:09:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f64bdc700 1 --2- 172.21.15.1:0/20399595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f48005210 0x7f1f4800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f5ffff700 1 --2- 172.21.15.1:0/20399595 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f48003a00 0x7f1f48003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f5f7fe700 1 --2- 172.21.15.1:0/20399595 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f48004850 0x7f1f48004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f64bdc700 1 -- 172.21.15.1:0/20399595 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f48004850 msgr2=0x7f1f48004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f64bdc700 1 --2- 172.21.15.1:0/20399595 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f48004850 0x7f1f48004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f64bdc700 1 -- 172.21.15.1:0/20399595 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f48003a00 msgr2=0x7f1f48003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f64bdc700 1 --2- 172.21.15.1:0/20399595 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f48003a00 0x7f1f48003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f64bdc700 1 -- 172.21.15.1:0/20399595 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f480035b0 con 0x7f1f48005210 2024-03-18T19:09:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f64bdc700 1 -- 172.21.15.1:0/20399595 reap_dead start 2024-03-18T19:09:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f64bdc700 1 --2- 172.21.15.1:0/20399595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f48005210 0x7f1f4800ba10 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f1f580149d0 tx=0x7f1f48005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4422f77f16c2d433 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f5effd700 1 -- 172.21.15.1:0/20399595 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58019070 con 0x7f1f48005210 2024-03-18T19:09:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f5effd700 1 -- 172.21.15.1:0/20399595 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f58003a20 con 0x7f1f48005210 2024-03-18T19:09:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.774+0000 7f1f5effd700 1 -- 172.21.15.1:0/20399595 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58003dd0 con 0x7f1f48005210 2024-03-18T19:09:42.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.775+0000 7f1f6663f700 1 -- 172.21.15.1:0/20399595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f48005210 msgr2=0x7f1f4800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:42.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.775+0000 7f1f6663f700 1 --2- 172.21.15.1:0/20399595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f48005210 0x7f1f4800ba10 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f1f580149d0 tx=0x7f1f48005880 comp rx=0 tx=0).stop 2024-03-18T19:09:42.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.775+0000 7f1f6663f700 1 -- 172.21.15.1:0/20399595 shutdown_connections 2024-03-18T19:09:42.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.775+0000 7f1f6663f700 1 --2- 172.21.15.1:0/20399595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f48005210 0x7f1f4800ba10 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:42.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.775+0000 7f1f6663f700 1 -- 172.21.15.1:0/20399595 >> 172.21.15.1:0/20399595 conn(0x7f1f600b2bf0 msgr2=0x7f1f600b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:42.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.775+0000 7f1f6663f700 1 -- 172.21.15.1:0/20399595 shutdown_connections 2024-03-18T19:09:42.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.775+0000 7f1f6663f700 1 -- 172.21.15.1:0/20399595 wait complete. 2024-03-18T19:09:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.776+0000 7f1f6663f700 1 Processor -- start 2024-03-18T19:09:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.776+0000 7f1f6663f700 1 -- start start 2024-03-18T19:09:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.777+0000 7f1f6663f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f48005210 0x7f1f601537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.777+0000 7f1f6663f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f600b2470 0x7f1f601561e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.777+0000 7f1f6663f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f60153f00 0x7f1f6015be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.777+0000 7f1f6663f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f60155790 con 0x7f1f60153f00 2024-03-18T19:09:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.777+0000 7f1f6663f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f60153d20 con 0x7f1f48005210 2024-03-18T19:09:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.777+0000 7f1f6663f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f600cabc0 con 0x7f1f600b2470 2024-03-18T19:09:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.777+0000 7f1f64bdc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f60153f00 0x7f1f6015be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.777+0000 7f1f64bdc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f60153f00 0x7f1f6015be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46308/0 (socket says 172.21.15.1:46308) 2024-03-18T19:09:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.777+0000 7f1f64bdc700 1 -- 172.21.15.1:0/4287601370 learned_addr learned my addr 172.21.15.1:0/4287601370 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.777+0000 7f1f5f7fe700 1 --2- 172.21.15.1:0/4287601370 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f600b2470 0x7f1f601561e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.777+0000 7f1f5ffff700 1 --2- 172.21.15.1:0/4287601370 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f48005210 0x7f1f601537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.778+0000 7f1f64bdc700 1 -- 172.21.15.1:0/4287601370 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f600b2470 msgr2=0x7f1f601561e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.778+0000 7f1f64bdc700 1 --2- 172.21.15.1:0/4287601370 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f600b2470 0x7f1f601561e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.778+0000 7f1f64bdc700 1 -- 172.21.15.1:0/4287601370 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f48005210 msgr2=0x7f1f601537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.778+0000 7f1f64bdc700 1 --2- 172.21.15.1:0/4287601370 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f48005210 0x7f1f601537e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.778+0000 7f1f64bdc700 1 -- 172.21.15.1:0/4287601370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f600caea0 con 0x7f1f60153f00 2024-03-18T19:09:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.778+0000 7f1f5ffff700 1 --2- 172.21.15.1:0/4287601370 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f48005210 0x7f1f601537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.778+0000 7f1f64bdc700 1 --2- 172.21.15.1:0/4287601370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f60153f00 0x7f1f6015be30 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f1f6006d630 tx=0x7f1f58009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.778+0000 7f1f5cff9700 1 -- 172.21.15.1:0/4287601370 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58027660 con 0x7f1f60153f00 2024-03-18T19:09:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.778+0000 7f1f5cff9700 1 -- 172.21.15.1:0/4287601370 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f580277c0 con 0x7f1f60153f00 2024-03-18T19:09:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.778+0000 7f1f5cff9700 1 -- 172.21.15.1:0/4287601370 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58027b50 con 0x7f1f60153f00 2024-03-18T19:09:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.778+0000 7f1f6663f700 1 -- 172.21.15.1:0/4287601370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f601559e0 con 0x7f1f60153f00 2024-03-18T19:09:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.778+0000 7f1f6663f700 1 -- 172.21.15.1:0/4287601370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f60156720 con 0x7f1f60153f00 2024-03-18T19:09:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.779+0000 7f1f6663f700 1 -- 172.21.15.1:0/4287601370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f600cb030 con 0x7f1f60153f00 2024-03-18T19:09:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.779+0000 7f1f5cff9700 1 -- 172.21.15.1:0/4287601370 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 76) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f1f58027cb0 con 0x7f1f60153f00 2024-03-18T19:09:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.780+0000 7f1f5cff9700 1 --2- 172.21.15.1:0/4287601370 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1f4c06ae00 0x7f1f4c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.780+0000 7f1f5cff9700 1 -- 172.21.15.1:0/4287601370 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f1f58093700 con 0x7f1f60153f00 2024-03-18T19:09:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.780+0000 7f1f5ffff700 1 --2- 172.21.15.1:0/4287601370 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1f4c06ae00 0x7f1f4c06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.780+0000 7f1f5ffff700 1 --2- 172.21.15.1:0/4287601370 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1f4c06ae00 0x7f1f4c06d2c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1f54000c00 tx=0x7f1f54009040 comp rx=0 tx=0).ready entity=mgr.7262 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:42.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.785+0000 7f1f5cff9700 1 -- 172.21.15.1:0/4287601370 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1f5805e490 con 0x7f1f60153f00 2024-03-18T19:09:42.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:42.876+0000 7f1f5cff9700 1 -- 172.21.15.1:0/4287601370 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 77) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f1f58091960 con 0x7f1f60153f00 2024-03-18T19:09:43.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.016+0000 7f1f6663f700 1 -- 172.21.15.1:0/4287601370 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "M", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f6015d9f0 con 0x7f1f4c06ae00 2024-03-18T19:09:43.013 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:43.017+0000 7f911fe08700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-03-18T19:09:43.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.017+0000 7f1f5cff9700 1 -- 172.21.15.1:0/4287601370 <== mgr.7262 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1f540052c0 con 0x7f1f4c06ae00 2024-03-18T19:09:43.015 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-03-18T19:09:43.018 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-18T19:09:43.018 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec -1m 2024-03-18T19:09:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.151+0000 7f28895c5700 1 Processor -- start 2024-03-18T19:09:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.151+0000 7f28895c5700 1 -- start start 2024-03-18T19:09:43.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.152+0000 7f28895c5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f28840b4810 0x7f28840d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:43.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.152+0000 7f28895c5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f28840cde70 0x7f28840ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:43.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.152+0000 7f28895c5700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28840b3370 con 0x7f28840cfb50 2024-03-18T19:09:43.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.152+0000 7f28895c5700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28840b2760 con 0x7f28840b4810 2024-03-18T19:09:43.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.152+0000 7f28895c5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28840b2a40 con 0x7f28840cde70 2024-03-18T19:09:43.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.152+0000 7f28827fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f28840b4810 0x7f28840d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:43.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.152+0000 7f28827fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f28840b4810 0x7f28840d3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46310/0 (socket says 172.21.15.1:46310) 2024-03-18T19:09:43.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.152+0000 7f28827fc700 1 -- 172.21.15.1:0/3332256582 learned_addr learned my addr 172.21.15.1:0/3332256582 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:43.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.152+0000 7f28837fe700 1 --2- 172.21.15.1:0/3332256582 >> v2:172.21.15.165:3300/0 conn(0x7f28840cde70 0x7f28840ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.152+0000 7f28827fc700 1 -- 172.21.15.1:0/3332256582 >> v2:172.21.15.165:3300/0 conn(0x7f28840cde70 msgr2=0x7f28840ce2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.152+0000 7f28827fc700 1 --2- 172.21.15.1:0/3332256582 >> v2:172.21.15.165:3300/0 conn(0x7f28840cde70 0x7f28840ce2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.152+0000 7f28827fc700 1 -- 172.21.15.1:0/3332256582 >> v1:172.21.15.165:6789/0 conn(0x7f28840cfb50 legacy=0x7f28840cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.152+0000 7f28827fc700 1 -- 172.21.15.1:0/3332256582 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28840b2d20 con 0x7f28840b4810 2024-03-18T19:09:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.153+0000 7f28827fc700 1 --2- 172.21.15.1:0/3332256582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28840b4810 0x7f28840d3c60 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f288406ab10 tx=0x7f2874009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18351ee92e2cc67b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.153+0000 7f2881ffb700 1 -- 172.21.15.1:0/3332256582 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2874011830 con 0x7f28840b4810 2024-03-18T19:09:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.153+0000 7f2881ffb700 1 -- 172.21.15.1:0/3332256582 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2874011990 con 0x7f28840b4810 2024-03-18T19:09:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.153+0000 7f2881ffb700 1 -- 172.21.15.1:0/3332256582 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2874011d20 con 0x7f28840b4810 2024-03-18T19:09:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.153+0000 7f28895c5700 1 -- 172.21.15.1:0/3332256582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28840b4810 msgr2=0x7f28840d3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.153+0000 7f28895c5700 1 --2- 172.21.15.1:0/3332256582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28840b4810 0x7f28840d3c60 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f288406ab10 tx=0x7f2874009bd0 comp rx=0 tx=0).stop 2024-03-18T19:09:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.153+0000 7f28895c5700 1 -- 172.21.15.1:0/3332256582 shutdown_connections 2024-03-18T19:09:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.153+0000 7f28895c5700 1 --2- 172.21.15.1:0/3332256582 >> v2:172.21.15.165:3300/0 conn(0x7f28840cde70 0x7f28840ce2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.153+0000 7f28895c5700 1 --2- 172.21.15.1:0/3332256582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28840b4810 0x7f28840d3c60 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.153+0000 7f28895c5700 1 -- 172.21.15.1:0/3332256582 >> 172.21.15.1:0/3332256582 conn(0x7f28840d3140 msgr2=0x7f28840d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.153+0000 7f28895c5700 1 -- 172.21.15.1:0/3332256582 shutdown_connections 2024-03-18T19:09:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.154+0000 7f28895c5700 1 -- 172.21.15.1:0/3332256582 wait complete. 2024-03-18T19:09:43.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.155+0000 7f28895c5700 1 Processor -- start 2024-03-18T19:09:43.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.155+0000 7f28895c5700 1 -- start start 2024-03-18T19:09:43.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f28895c5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28840b4810 0x7f288414a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:43.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f28895c5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28840cde70 0x7f2884157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:43.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f28895c5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28840cfb50 0x7f28840b0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:43.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f28895c5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2884157870 con 0x7f28840cde70 2024-03-18T19:09:43.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f28895c5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28840cadf0 con 0x7f28840b4810 2024-03-18T19:09:43.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f28895c5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28840c8350 con 0x7f28840cfb50 2024-03-18T19:09:43.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f28827fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28840cde70 0x7f2884157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:43.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f28827fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28840cde70 0x7f2884157fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46316/0 (socket says 172.21.15.1:46316) 2024-03-18T19:09:43.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f28827fc700 1 -- 172.21.15.1:0/2745600007 learned_addr learned my addr 172.21.15.1:0/2745600007 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:43.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f28837fe700 1 --2- 172.21.15.1:0/2745600007 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28840cfb50 0x7f28840b0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:43.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f2882ffd700 1 --2- 172.21.15.1:0/2745600007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28840b4810 0x7f288414a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:43.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f28827fc700 1 -- 172.21.15.1:0/2745600007 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28840cfb50 msgr2=0x7f28840b0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:43.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f28827fc700 1 --2- 172.21.15.1:0/2745600007 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28840cfb50 0x7f28840b0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:43.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f28827fc700 1 -- 172.21.15.1:0/2745600007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28840b4810 msgr2=0x7f288414a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:43.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f28827fc700 1 --2- 172.21.15.1:0/2745600007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28840b4810 0x7f288414a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:43.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f28827fc700 1 -- 172.21.15.1:0/2745600007 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2874009550 con 0x7f28840cde70 2024-03-18T19:09:43.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f28827fc700 1 --2- 172.21.15.1:0/2745600007 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28840cde70 0x7f2884157fa0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f2874000c00 tx=0x7f2874004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:43.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f286b7fe700 1 -- 172.21.15.1:0/2745600007 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2874004570 con 0x7f28840cde70 2024-03-18T19:09:43.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f286b7fe700 1 -- 172.21.15.1:0/2745600007 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287401f070 con 0x7f28840cde70 2024-03-18T19:09:43.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f286b7fe700 1 -- 172.21.15.1:0/2745600007 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28740038e0 con 0x7f28840cde70 2024-03-18T19:09:43.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f28895c5700 1 -- 172.21.15.1:0/2745600007 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28840c8630 con 0x7f28840cde70 2024-03-18T19:09:43.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.156+0000 7f28895c5700 1 -- 172.21.15.1:0/2745600007 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28840cb0d0 con 0x7f28840cde70 2024-03-18T19:09:43.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.157+0000 7f28895c5700 1 -- 172.21.15.1:0/2745600007 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28840c8b50 con 0x7f28840cde70 2024-03-18T19:09:43.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.158+0000 7f286b7fe700 1 -- 172.21.15.1:0/2745600007 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 77) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f2874021030 con 0x7f28840cde70 2024-03-18T19:09:43.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.158+0000 7f286b7fe700 1 --2- 172.21.15.1:0/2745600007 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f287006aec0 0x7f287006d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:43.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.158+0000 7f286b7fe700 1 -- 172.21.15.1:0/2745600007 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f287408ad30 con 0x7f28840cde70 2024-03-18T19:09:43.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.159+0000 7f2882ffd700 1 --2- 172.21.15.1:0/2745600007 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f287006aec0 0x7f287006d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:43.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.159+0000 7f2882ffd700 1 --2- 172.21.15.1:0/2745600007 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f287006aec0 0x7f287006d380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2878000fb0 tx=0x7f2878009040 comp rx=0 tx=0).ready entity=mgr.7262 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:43.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.163+0000 7f286b7fe700 1 -- 172.21.15.1:0/2745600007 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f28840c8b50 con 0x7f28840cde70 2024-03-18T19:09:43.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.391+0000 7f28895c5700 1 -- 172.21.15.1:0/2745600007 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "-1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f2884160d90 con 0x7f287006aec0 2024-03-18T19:09:43.387 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:43.392+0000 7f911fe08700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2024-03-18T19:09:43.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.392+0000 7f286b7fe700 1 -- 172.21.15.1:0/2745600007 <== mgr.7262 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc.) v1 ==== 96+0+0 (secure 0 0 0) 0x7f2878001310 con 0x7f287006aec0 2024-03-18T19:09:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENOENT: invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2024-03-18T19:09:43.393 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-18T19:09:43.393 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec '' 2024-03-18T19:09:43.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.526+0000 7f4401eb6700 1 Processor -- start 2024-03-18T19:09:43.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.526+0000 7f4401eb6700 1 -- start start 2024-03-18T19:09:43.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.526+0000 7f4401eb6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f43fc0cfa90 0x7f43fc06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:43.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.526+0000 7f4401eb6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f43fc06ac90 0x7f43fc167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:43.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.526+0000 7f4401eb6700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43fc0cf600 con 0x7f43fc15d5a0 2024-03-18T19:09:43.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.526+0000 7f4401eb6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43fc06b140 con 0x7f43fc06ac90 2024-03-18T19:09:43.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.526+0000 7f4401eb6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43fc0ce720 con 0x7f43fc0cfa90 2024-03-18T19:09:43.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.527+0000 7f43fbfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f43fc06ac90 0x7f43fc167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.527+0000 7f43fbfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f43fc06ac90 0x7f43fc167140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46324/0 (socket says 172.21.15.1:46324) 2024-03-18T19:09:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.527+0000 7f43fbfff700 1 -- 172.21.15.1:0/989990473 learned_addr learned my addr 172.21.15.1:0/989990473 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.527+0000 7f43fb7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f43fc15d5a0 0x7f43fc0ce010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43850/0 (socket says 172.21.15.1:43850) 2024-03-18T19:09:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.527+0000 7f43faffd700 1 --2- 172.21.15.1:0/989990473 >> v2:172.21.15.165:3300/0 conn(0x7f43fc0cfa90 0x7f43fc06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.527+0000 7f43fbfff700 1 -- 172.21.15.1:0/989990473 >> v2:172.21.15.165:3300/0 conn(0x7f43fc0cfa90 msgr2=0x7f43fc06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.527+0000 7f43fbfff700 1 --2- 172.21.15.1:0/989990473 >> v2:172.21.15.165:3300/0 conn(0x7f43fc0cfa90 0x7f43fc06a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.527+0000 7f43fbfff700 1 -- 172.21.15.1:0/989990473 >> v1:172.21.15.1:6789/0 conn(0x7f43fc15d5a0 legacy=0x7f43fc0ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.527+0000 7f43fbfff700 1 -- 172.21.15.1:0/989990473 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43fc0ce9f0 con 0x7f43fc06ac90 2024-03-18T19:09:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.527+0000 7f43fa7fc700 1 -- 172.21.15.1:0/989990473 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186789175 0 0) 0x7f43fc0cf600 con 0x7f43fc15d5a0 2024-03-18T19:09:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.527+0000 7f43faffd700 1 --2- 172.21.15.1:0/989990473 >> v2:172.21.15.165:3300/0 conn(0x7f43fc0cfa90 0x7f43fc06a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.527+0000 7f43fbfff700 1 --2- 172.21.15.1:0/989990473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43fc06ac90 0x7f43fc167140 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f43fc0af730 tx=0x7f43f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f88a46be9cb1e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.527+0000 7f43fa7fc700 1 -- 172.21.15.1:0/989990473 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f400e070 con 0x7f43fc06ac90 2024-03-18T19:09:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.527+0000 7f43fa7fc700 1 -- 172.21.15.1:0/989990473 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f400b9c0 con 0x7f43fc06ac90 2024-03-18T19:09:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.527+0000 7f43fa7fc700 1 -- 172.21.15.1:0/989990473 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f400bd70 con 0x7f43fc06ac90 2024-03-18T19:09:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.528+0000 7f4401eb6700 1 -- 172.21.15.1:0/989990473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43fc06ac90 msgr2=0x7f43fc167140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.528+0000 7f4401eb6700 1 --2- 172.21.15.1:0/989990473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43fc06ac90 0x7f43fc167140 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f43fc0af730 tx=0x7f43f400b330 comp rx=0 tx=0).stop 2024-03-18T19:09:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.528+0000 7f4401eb6700 1 -- 172.21.15.1:0/989990473 shutdown_connections 2024-03-18T19:09:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.528+0000 7f4401eb6700 1 --2- 172.21.15.1:0/989990473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43fc06ac90 0x7f43fc167140 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.528+0000 7f4401eb6700 1 --2- 172.21.15.1:0/989990473 >> v2:172.21.15.165:3300/0 conn(0x7f43fc0cfa90 0x7f43fc06a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.528+0000 7f4401eb6700 1 -- 172.21.15.1:0/989990473 >> 172.21.15.1:0/989990473 conn(0x7f43fc0cc9b0 msgr2=0x7f43fc0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.528+0000 7f4401eb6700 1 -- 172.21.15.1:0/989990473 shutdown_connections 2024-03-18T19:09:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.528+0000 7f4401eb6700 1 -- 172.21.15.1:0/989990473 wait complete. 2024-03-18T19:09:43.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.529+0000 7f4401eb6700 1 Processor -- start 2024-03-18T19:09:43.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.529+0000 7f4401eb6700 1 -- start start 2024-03-18T19:09:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.530+0000 7f4401eb6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43fc06ac90 0x7f43fc152e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.530+0000 7f4401eb6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43fc0cfa90 0x7f43fc0c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.530+0000 7f4401eb6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43fc15d5a0 0x7f43fc0c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.530+0000 7f4401eb6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43fc152920 con 0x7f43fc15d5a0 2024-03-18T19:09:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.530+0000 7f4401eb6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43fc1534c0 con 0x7f43fc06ac90 2024-03-18T19:09:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.530+0000 7f4401eb6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43fc1537a0 con 0x7f43fc0cfa90 2024-03-18T19:09:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.530+0000 7f43fbfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43fc15d5a0 0x7f43fc0c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.530+0000 7f43fbfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43fc15d5a0 0x7f43fc0c7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46332/0 (socket says 172.21.15.1:46332) 2024-03-18T19:09:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.530+0000 7f43fbfff700 1 -- 172.21.15.1:0/3742427996 learned_addr learned my addr 172.21.15.1:0/3742427996 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.530+0000 7f43faffd700 1 --2- 172.21.15.1:0/3742427996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43fc0cfa90 0x7f43fc0c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.530+0000 7f43fb7fe700 1 --2- 172.21.15.1:0/3742427996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43fc06ac90 0x7f43fc152e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.530+0000 7f43fbfff700 1 -- 172.21.15.1:0/3742427996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43fc0cfa90 msgr2=0x7f43fc0c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.530+0000 7f43fbfff700 1 --2- 172.21.15.1:0/3742427996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43fc0cfa90 0x7f43fc0c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.530+0000 7f43fbfff700 1 -- 172.21.15.1:0/3742427996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43fc06ac90 msgr2=0x7f43fc152e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.530+0000 7f43fbfff700 1 --2- 172.21.15.1:0/3742427996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43fc06ac90 0x7f43fc152e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.530+0000 7f43fbfff700 1 -- 172.21.15.1:0/3742427996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f4009b30 con 0x7f43fc15d5a0 2024-03-18T19:09:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.530+0000 7f43fb7fe700 1 --2- 172.21.15.1:0/3742427996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43fc06ac90 0x7f43fc152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.531+0000 7f43fbfff700 1 --2- 172.21.15.1:0/3742427996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43fc15d5a0 0x7f43fc0c7000 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f43fc06cfd0 tx=0x7f43f4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.531+0000 7f43e3fff700 1 -- 172.21.15.1:0/3742427996 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f40048f0 con 0x7f43fc15d5a0 2024-03-18T19:09:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.531+0000 7f43e3fff700 1 -- 172.21.15.1:0/3742427996 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f40092e0 con 0x7f43fc15d5a0 2024-03-18T19:09:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.531+0000 7f4401eb6700 1 -- 172.21.15.1:0/3742427996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43fc153a80 con 0x7f43fc15d5a0 2024-03-18T19:09:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.531+0000 7f43e3fff700 1 -- 172.21.15.1:0/3742427996 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f401c660 con 0x7f43fc15d5a0 2024-03-18T19:09:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.531+0000 7f4401eb6700 1 -- 172.21.15.1:0/3742427996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43fc154040 con 0x7f43fc15d5a0 2024-03-18T19:09:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.532+0000 7f4401eb6700 1 -- 172.21.15.1:0/3742427996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43fc0676a0 con 0x7f43fc15d5a0 2024-03-18T19:09:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.532+0000 7f43e3fff700 1 -- 172.21.15.1:0/3742427996 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 77) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f43f401c7c0 con 0x7f43fc15d5a0 2024-03-18T19:09:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.533+0000 7f43e3fff700 1 --2- 172.21.15.1:0/3742427996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43e806ae90 0x7f43e806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.533+0000 7f43e3fff700 1 -- 172.21.15.1:0/3742427996 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f43f408b480 con 0x7f43fc15d5a0 2024-03-18T19:09:43.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.533+0000 7f43fb7fe700 1 --2- 172.21.15.1:0/3742427996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43e806ae90 0x7f43e806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.533+0000 7f43fb7fe700 1 --2- 172.21.15.1:0/3742427996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43e806ae90 0x7f43e806d350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f43ec0017b0 tx=0x7f43ec009040 comp rx=0 tx=0).ready entity=mgr.7262 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.537+0000 7f43e3fff700 1 -- 172.21.15.1:0/3742427996 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f43f4056150 con 0x7f43fc15d5a0 2024-03-18T19:09:43.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.766+0000 7f4401eb6700 1 -- 172.21.15.1:0/3742427996 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f43fc0cf290 con 0x7f43e806ae90 2024-03-18T19:09:43.763 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:43.767+0000 7f911fe08700 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-03-18T19:09:43.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:43.767+0000 7f43e3fff700 1 -- 172.21.15.1:0/3742427996 <== mgr.7262 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f43ec001c80 con 0x7f43e806ae90 2024-03-18T19:09:43.764 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2024-03-18T19:09:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-18T19:09:43.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:43.768 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/minutes) 2024-03-18T19:09:43.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:43.790 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/hourly) 2024-03-18T19:09:43.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:43.852 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/daily) 2024-03-18T19:09:43.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:43.913 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/weekly) 2024-03-18T19:09:43.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:43.975 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/monthly) 2024-03-18T19:09:44.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:44.037 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/yearly) 2024-03-18T19:09:44.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:44.098 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/bad_period_spec) 2024-03-18T19:09:44.161 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-18T19:09:44.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.332+0000 7faca1ae4700 1 Processor -- start 2024-03-18T19:09:44.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.332+0000 7faca1ae4700 1 -- start start 2024-03-18T19:09:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.333+0000 7faca1ae4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fac9c144bb0 0x7fac9c15b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.333+0000 7faca1ae4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac9c0cc990 0x7fac9c0ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.333+0000 7faca1ae4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fac9c0ce8d0 0x7fac9c0ced60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:44.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.333+0000 7faca1ae4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac9c06eb50 con 0x7fac9c144bb0 2024-03-18T19:09:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.333+0000 7faca1ae4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac9c0cd330 con 0x7fac9c0ce8d0 2024-03-18T19:09:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.333+0000 7faca1ae4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c144f90 con 0x7fac9c0cc990 2024-03-18T19:09:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.333+0000 7fac9b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fac9c144bb0 0x7fac9c15b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.333+0000 7fac9b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fac9c144bb0 0x7fac9c15b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46340/0 (socket says 172.21.15.1:46340) 2024-03-18T19:09:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.333+0000 7fac9b7fe700 1 -- 172.21.15.1:0/2388549619 learned_addr learned my addr 172.21.15.1:0/2388549619 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.334+0000 7fac9affd700 1 --2- 172.21.15.1:0/2388549619 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac9c0cc990 0x7fac9c0ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.334+0000 7fac9bfff700 1 --2- 172.21.15.1:0/2388549619 >> v2:172.21.15.165:3300/0 conn(0x7fac9c0ce8d0 0x7fac9c0ced60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.334+0000 7fac9b7fe700 1 -- 172.21.15.1:0/2388549619 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac9c0cc990 msgr2=0x7fac9c0ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.334+0000 7fac9b7fe700 1 --2- 172.21.15.1:0/2388549619 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac9c0cc990 0x7fac9c0ccdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.334+0000 7fac9b7fe700 1 -- 172.21.15.1:0/2388549619 >> v2:172.21.15.165:3300/0 conn(0x7fac9c0ce8d0 msgr2=0x7fac9c0ced60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.334+0000 7fac9b7fe700 1 --2- 172.21.15.1:0/2388549619 >> v2:172.21.15.165:3300/0 conn(0x7fac9c0ce8d0 0x7fac9c0ced60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:44.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.334+0000 7fac9b7fe700 1 -- 172.21.15.1:0/2388549619 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9c06edc0 con 0x7fac9c144bb0 2024-03-18T19:09:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.334+0000 7fac9bfff700 1 --2- 172.21.15.1:0/2388549619 >> v2:172.21.15.165:3300/0 conn(0x7fac9c0ce8d0 0x7fac9c0ced60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.334+0000 7fac9b7fe700 1 --2- 172.21.15.1:0/2388549619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac9c144bb0 0x7fac9c15b7e0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0cc8b0 tx=0x7fac84009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64cb7c16c35311a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.334+0000 7fac9a7fc700 1 -- 172.21.15.1:0/2388549619 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8400b870 con 0x7fac9c144bb0 2024-03-18T19:09:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.334+0000 7fac9a7fc700 1 -- 172.21.15.1:0/2388549619 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8400b9d0 con 0x7fac9c144bb0 2024-03-18T19:09:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.334+0000 7fac9a7fc700 1 -- 172.21.15.1:0/2388549619 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8400bd60 con 0x7fac9c144bb0 2024-03-18T19:09:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.334+0000 7faca1ae4700 1 -- 172.21.15.1:0/2388549619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac9c144bb0 msgr2=0x7fac9c15b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.334+0000 7faca1ae4700 1 --2- 172.21.15.1:0/2388549619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac9c144bb0 0x7fac9c15b7e0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0cc8b0 tx=0x7fac84009bd0 comp rx=0 tx=0).stop 2024-03-18T19:09:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.335+0000 7faca1ae4700 1 -- 172.21.15.1:0/2388549619 shutdown_connections 2024-03-18T19:09:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.335+0000 7faca1ae4700 1 --2- 172.21.15.1:0/2388549619 >> v2:172.21.15.165:3300/0 conn(0x7fac9c0ce8d0 0x7fac9c0ced60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.335+0000 7faca1ae4700 1 --2- 172.21.15.1:0/2388549619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac9c144bb0 0x7fac9c15b7e0 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.335+0000 7faca1ae4700 1 --2- 172.21.15.1:0/2388549619 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac9c0cc990 0x7fac9c0ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.335+0000 7faca1ae4700 1 -- 172.21.15.1:0/2388549619 >> 172.21.15.1:0/2388549619 conn(0x7fac9c0b1cb0 msgr2=0x7fac9c0b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.335+0000 7faca1ae4700 1 -- 172.21.15.1:0/2388549619 shutdown_connections 2024-03-18T19:09:44.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.335+0000 7faca1ae4700 1 -- 172.21.15.1:0/2388549619 wait complete. 2024-03-18T19:09:44.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.336+0000 7faca1ae4700 1 Processor -- start 2024-03-18T19:09:44.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.336+0000 7faca1ae4700 1 -- start start 2024-03-18T19:09:44.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7faca1ae4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac9c0cc990 0x7fac9c13ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:44.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7faca1ae4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac9c0ce8d0 0x7fac9c1f39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7faca1ae4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac9c144bb0 0x7fac9c15ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7faca1ae4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c1f7440 con 0x7fac9c0ce8d0 2024-03-18T19:09:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7faca1ae4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c13b780 con 0x7fac9c0cc990 2024-03-18T19:09:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7faca1ae4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c0b30b0 con 0x7fac9c144bb0 2024-03-18T19:09:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7fac9affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac9c0ce8d0 0x7fac9c1f39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7fac9affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac9c0ce8d0 0x7fac9c1f39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46356/0 (socket says 172.21.15.1:46356) 2024-03-18T19:09:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7fac9affd700 1 -- 172.21.15.1:0/1588957338 learned_addr learned my addr 172.21.15.1:0/1588957338 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7fac9b7fe700 1 --2- 172.21.15.1:0/1588957338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac9c0cc990 0x7fac9c13ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7fac9bfff700 1 --2- 172.21.15.1:0/1588957338 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac9c144bb0 0x7fac9c15ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7fac9affd700 1 -- 172.21.15.1:0/1588957338 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac9c144bb0 msgr2=0x7fac9c15ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7fac9affd700 1 --2- 172.21.15.1:0/1588957338 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac9c144bb0 0x7fac9c15ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7fac9affd700 1 -- 172.21.15.1:0/1588957338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac9c0cc990 msgr2=0x7fac9c13ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7fac9affd700 1 --2- 172.21.15.1:0/1588957338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac9c0cc990 0x7fac9c13ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:44.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7fac9affd700 1 -- 172.21.15.1:0/1588957338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac84009550 con 0x7fac9c0ce8d0 2024-03-18T19:09:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7fac9b7fe700 1 --2- 172.21.15.1:0/1588957338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac9c0cc990 0x7fac9c13ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7fac9affd700 1 --2- 172.21.15.1:0/1588957338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac9c0ce8d0 0x7fac9c1f39b0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fac8c00e810 tx=0x7fac8c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7fac93fff700 1 -- 172.21.15.1:0/1588957338 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c013610 con 0x7fac9c0ce8d0 2024-03-18T19:09:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7fac93fff700 1 -- 172.21.15.1:0/1588957338 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8c013770 con 0x7fac9c0ce8d0 2024-03-18T19:09:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.337+0000 7faca1ae4700 1 -- 172.21.15.1:0/1588957338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9c1f3ef0 con 0x7fac9c0ce8d0 2024-03-18T19:09:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.338+0000 7fac93fff700 1 -- 172.21.15.1:0/1588957338 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c013b00 con 0x7fac9c0ce8d0 2024-03-18T19:09:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.338+0000 7faca1ae4700 1 -- 172.21.15.1:0/1588957338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9c0b32c0 con 0x7fac9c0ce8d0 2024-03-18T19:09:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.338+0000 7faca1ae4700 1 -- 172.21.15.1:0/1588957338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9c0676a0 con 0x7fac9c0ce8d0 2024-03-18T19:09:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.339+0000 7fac93fff700 1 -- 172.21.15.1:0/1588957338 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 77) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fac8c013c60 con 0x7fac9c0ce8d0 2024-03-18T19:09:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.339+0000 7fac93fff700 1 --2- 172.21.15.1:0/1588957338 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac7c06ae90 0x7fac7c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.339+0000 7fac93fff700 1 -- 172.21.15.1:0/1588957338 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fac8c089500 con 0x7fac9c0ce8d0 2024-03-18T19:09:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.340+0000 7fac9b7fe700 1 --2- 172.21.15.1:0/1588957338 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac7c06ae90 0x7fac7c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:44.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.340+0000 7fac9b7fe700 1 --2- 172.21.15.1:0/1588957338 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac7c06ae90 0x7fac7c06d350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fac84014040 tx=0x7fac84008030 comp rx=0 tx=0).ready entity=mgr.7262 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:44.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.344+0000 7fac93fff700 1 -- 172.21.15.1:0/1588957338 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fac8c054080 con 0x7fac9c0ce8d0 2024-03-18T19:09:44.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:44.602+0000 7faca1ae4700 1 -- 172.21.15.1:0/1588957338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fac9c1f3ef0 con 0x7fac9c0ce8d0 2024-03-18T19:09:45.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.139+0000 7fac93fff700 1 -- 172.21.15.1:0/1588957338 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 78) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fac8c054450 con 0x7fac9c0ce8d0 2024-03-18T19:09:45.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.140+0000 7fac93fff700 1 -- 172.21.15.1:0/1588957338 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v78) v1 ==== 93+0+0 (secure 0 0 0) 0x7fac8c085040 con 0x7fac9c0ce8d0 2024-03-18T19:09:45.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.143+0000 7faca1ae4700 1 -- 172.21.15.1:0/1588957338 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac7c06ae90 msgr2=0x7fac7c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:45.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.143+0000 7faca1ae4700 1 --2- 172.21.15.1:0/1588957338 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac7c06ae90 0x7fac7c06d350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fac84014040 tx=0x7fac84008030 comp rx=0 tx=0).stop 2024-03-18T19:09:45.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.143+0000 7faca1ae4700 1 -- 172.21.15.1:0/1588957338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac9c0ce8d0 msgr2=0x7fac9c1f39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:45.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.143+0000 7faca1ae4700 1 --2- 172.21.15.1:0/1588957338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac9c0ce8d0 0x7fac9c1f39b0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fac8c00e810 tx=0x7fac8c00ebd0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.143+0000 7faca1ae4700 1 -- 172.21.15.1:0/1588957338 shutdown_connections 2024-03-18T19:09:45.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.143+0000 7faca1ae4700 1 --2- 172.21.15.1:0/1588957338 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fac7c06ae90 0x7fac7c06d350 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.143+0000 7faca1ae4700 1 --2- 172.21.15.1:0/1588957338 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac9c144bb0 0x7fac9c15ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.143+0000 7faca1ae4700 1 --2- 172.21.15.1:0/1588957338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac9c0cc990 0x7fac9c13ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.143+0000 7faca1ae4700 1 --2- 172.21.15.1:0/1588957338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac9c0ce8d0 0x7fac9c1f39b0 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.143+0000 7faca1ae4700 1 -- 172.21.15.1:0/1588957338 >> 172.21.15.1:0/1588957338 conn(0x7fac9c0b1cb0 msgr2=0x7fac9c0b00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:45.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.143+0000 7faca1ae4700 1 -- 172.21.15.1:0/1588957338 shutdown_connections 2024-03-18T19:09:45.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.143+0000 7faca1ae4700 1 -- 172.21.15.1:0/1588957338 wait complete. 2024-03-18T19:09:45.154 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T19:09:45.155 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:09:45.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:09:45.247 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T19:09:45.247 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:09:45.247 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:09:45.266 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:09:45.273 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:45.277+0000 7f441fadd240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:09:45.291 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:45.295+0000 7ffbca1f9240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:09:45.420 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:45.424+0000 7f441fadd240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:09:45.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.457+0000 7f704552a700 1 Processor -- start 2024-03-18T19:09:45.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.458+0000 7f704552a700 1 -- start start 2024-03-18T19:09:45.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.460+0000 7f704552a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f704006ae60 0x7f7040165270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:45.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.460+0000 7f704552a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7040161d90 con 0x7f704015f140 2024-03-18T19:09:45.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.461+0000 7f704552a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70400b2780 con 0x7f704006dcf0 2024-03-18T19:09:45.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.461+0000 7f704552a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7040165850 con 0x7f704006ae60 2024-03-18T19:09:45.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.461+0000 7f703effd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f704015f140 0x7f70400b2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59500/0 (socket says 172.21.15.1:59500) 2024-03-18T19:09:45.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.461+0000 7f703effd700 1 -- 172.21.15.1:0/34784931 learned_addr learned my addr 172.21.15.1:0/34784931 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:45.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.461+0000 7f703f7fe700 1 --2- 172.21.15.1:0/34784931 >> v2:172.21.15.165:3300/0 conn(0x7f704006ae60 0x7f7040165270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.461+0000 7f703f7fe700 1 -- 172.21.15.1:0/34784931 >> v1:172.21.15.165:6789/0 conn(0x7f704006dcf0 legacy=0x7f704006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.461+0000 7f703f7fe700 1 -- 172.21.15.1:0/34784931 >> v1:172.21.15.1:6789/0 conn(0x7f704015f140 legacy=0x7f70400b2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.461+0000 7f703f7fe700 1 -- 172.21.15.1:0/34784931 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7040165b30 con 0x7f704006ae60 2024-03-18T19:09:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.461+0000 7f703dffb700 1 -- 172.21.15.1:0/34784931 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745926088 0 0) 0x7f7040161d90 con 0x7f704015f140 2024-03-18T19:09:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.462+0000 7f703f7fe700 1 --2- 172.21.15.1:0/34784931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f704006ae60 0x7f7040165270 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7038009d90 tx=0x7f703800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a69ccbade58fa264 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.462+0000 7f703dffb700 1 -- 172.21.15.1:0/34784931 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703800e070 con 0x7f704006ae60 2024-03-18T19:09:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.462+0000 7f703dffb700 1 -- 172.21.15.1:0/34784931 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703800bbd0 con 0x7f704006ae60 2024-03-18T19:09:45.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.462+0000 7f703dffb700 1 -- 172.21.15.1:0/34784931 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038012430 con 0x7f704006ae60 2024-03-18T19:09:45.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.463+0000 7f704552a700 1 -- 172.21.15.1:0/34784931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f704006ae60 msgr2=0x7f7040165270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:45.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.463+0000 7f704552a700 1 --2- 172.21.15.1:0/34784931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f704006ae60 0x7f7040165270 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7038009d90 tx=0x7f703800b330 comp rx=0 tx=0).stop 2024-03-18T19:09:45.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.463+0000 7f704552a700 1 -- 172.21.15.1:0/34784931 shutdown_connections 2024-03-18T19:09:45.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.463+0000 7f704552a700 1 --2- 172.21.15.1:0/34784931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f704006ae60 0x7f7040165270 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.463+0000 7f704552a700 1 -- 172.21.15.1:0/34784931 >> 172.21.15.1:0/34784931 conn(0x7f70400b0a00 msgr2=0x7f70400b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:45.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.463+0000 7f704552a700 1 -- 172.21.15.1:0/34784931 shutdown_connections 2024-03-18T19:09:45.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.464+0000 7f704552a700 1 -- 172.21.15.1:0/34784931 wait complete. 2024-03-18T19:09:45.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.467+0000 7f704552a700 1 Processor -- start 2024-03-18T19:09:45.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.469+0000 7f704552a700 1 -- start start 2024-03-18T19:09:45.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.469+0000 7f704552a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f704006ae60 0x7f70400cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:45.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.469+0000 7f704552a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f704006dcf0 0x7f704014ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:45.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.469+0000 7f704552a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f704015f140 0x7f70401500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:45.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.469+0000 7f704552a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704014d3a0 con 0x7f704015f140 2024-03-18T19:09:45.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.469+0000 7f704552a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7040150620 con 0x7f704006ae60 2024-03-18T19:09:45.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.469+0000 7f704552a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70400cd880 con 0x7f704006dcf0 2024-03-18T19:09:45.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.469+0000 7f703f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f704015f140 0x7f70401500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:45.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.469+0000 7f703f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f704015f140 0x7f70401500e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43832/0 (socket says 172.21.15.1:43832) 2024-03-18T19:09:45.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.469+0000 7f703f7fe700 1 -- 172.21.15.1:0/1040619811 learned_addr learned my addr 172.21.15.1:0/1040619811 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:45.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.469+0000 7f703effd700 1 --2- 172.21.15.1:0/1040619811 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f704006ae60 0x7f70400cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:45.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.469+0000 7f703e7fc700 1 --2- 172.21.15.1:0/1040619811 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f704006dcf0 0x7f704014ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:45.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.470+0000 7f703f7fe700 1 -- 172.21.15.1:0/1040619811 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f704006dcf0 msgr2=0x7f704014ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:45.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.470+0000 7f703f7fe700 1 --2- 172.21.15.1:0/1040619811 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f704006dcf0 0x7f704014ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.470+0000 7f703f7fe700 1 -- 172.21.15.1:0/1040619811 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f704006ae60 msgr2=0x7f70400cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:45.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.470+0000 7f703f7fe700 1 --2- 172.21.15.1:0/1040619811 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f704006ae60 0x7f70400cd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.470+0000 7f703f7fe700 1 -- 172.21.15.1:0/1040619811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7038009a70 con 0x7f704015f140 2024-03-18T19:09:45.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.470+0000 7f703f7fe700 1 --2- 172.21.15.1:0/1040619811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f704015f140 0x7f70401500e0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f7038014040 tx=0x7f7038004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:45.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.470+0000 7f70277fe700 1 -- 172.21.15.1:0/1040619811 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703801b810 con 0x7f704015f140 2024-03-18T19:09:45.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.470+0000 7f70277fe700 1 -- 172.21.15.1:0/1040619811 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7038020070 con 0x7f704015f140 2024-03-18T19:09:45.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.470+0000 7f704552a700 1 -- 172.21.15.1:0/1040619811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70400cdb60 con 0x7f704015f140 2024-03-18T19:09:45.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.470+0000 7f70277fe700 1 -- 172.21.15.1:0/1040619811 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703801bbb0 con 0x7f704015f140 2024-03-18T19:09:45.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.470+0000 7f704552a700 1 -- 172.21.15.1:0/1040619811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70400ce120 con 0x7f704015f140 2024-03-18T19:09:45.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.471+0000 7f704552a700 1 -- 172.21.15.1:0/1040619811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7040072630 con 0x7f704015f140 2024-03-18T19:09:45.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.472+0000 7f70277fe700 1 -- 172.21.15.1:0/1040619811 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 78) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f703801bd10 con 0x7f704015f140 2024-03-18T19:09:45.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.472+0000 7f70277fe700 1 --2- 172.21.15.1:0/1040619811 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f702c06ae40 0x7f702c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:45.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.472+0000 7f703effd700 1 -- 172.21.15.1:0/1040619811 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f702c06ae40 msgr2=0x7f702c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:45.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.472+0000 7f70277fe700 1 -- 172.21.15.1:0/1040619811 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f7038022070 con 0x7f704015f140 2024-03-18T19:09:45.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.472+0000 7f703effd700 1 --2- 172.21.15.1:0/1040619811 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f702c06ae40 0x7f702c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:45.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.477+0000 7f70277fe700 1 -- 172.21.15.1:0/1040619811 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7038056a40 con 0x7f704015f140 2024-03-18T19:09:45.474 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:45.479+0000 7ffbca1f9240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:09:45.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.673+0000 7f703effd700 1 -- 172.21.15.1:0/1040619811 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f702c06ae40 msgr2=0x7f702c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:45.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.673+0000 7f703effd700 1 --2- 172.21.15.1:0/1040619811 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f702c06ae40 0x7f702c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:45.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.683+0000 7f704552a700 1 -- 172.21.15.1:0/1040619811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7040073350 con 0x7f704015f140 2024-03-18T19:09:45.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.684+0000 7f70277fe700 1 -- 172.21.15.1:0/1040619811 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+16790 (secure 0 0 0) 0x7f703805a060 con 0x7f704015f140 2024-03-18T19:09:45.681 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:45.681 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":148,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:09:37.850698+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:09:20.468744+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-18T19:09:21.597476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/69006586":"2024-03-19T19:09:37.850642+0000","172.21.15.1:6833/76998":"2024-03-19T19:09:37.850642+0000","172.21.15.1:6832/76998":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/1403689662":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/806972823":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/619565916":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/2042700873":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/744112928":"2024-03-19T19:09:37.850642+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:09:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.687+0000 7f704552a700 1 -- 172.21.15.1:0/1040619811 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f702c06ae40 msgr2=0x7f702c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.687+0000 7f704552a700 1 --2- 172.21.15.1:0/1040619811 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f702c06ae40 0x7f702c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.687+0000 7f704552a700 1 -- 172.21.15.1:0/1040619811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f704015f140 msgr2=0x7f70401500e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.687+0000 7f704552a700 1 --2- 172.21.15.1:0/1040619811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f704015f140 0x7f70401500e0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f7038014040 tx=0x7f7038004ad0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.687+0000 7f704552a700 1 -- 172.21.15.1:0/1040619811 shutdown_connections 2024-03-18T19:09:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.687+0000 7f704552a700 1 --2- 172.21.15.1:0/1040619811 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f702c06ae40 0x7f702c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.687+0000 7f704552a700 1 --2- 172.21.15.1:0/1040619811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f704015f140 0x7f70401500e0 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.687+0000 7f704552a700 1 --2- 172.21.15.1:0/1040619811 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f704006ae60 0x7f70400cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.687+0000 7f704552a700 1 --2- 172.21.15.1:0/1040619811 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f704006dcf0 0x7f704014ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.687+0000 7f704552a700 1 -- 172.21.15.1:0/1040619811 >> 172.21.15.1:0/1040619811 conn(0x7f70400b0a00 msgr2=0x7f704015e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.687+0000 7f704552a700 1 -- 172.21.15.1:0/1040619811 shutdown_connections 2024-03-18T19:09:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.687+0000 7f704552a700 1 -- 172.21.15.1:0/1040619811 wait complete. 2024-03-18T19:09:45.684 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:45.688+0000 7f441fadd240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:09:45.696 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T19:09:20.468744+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T19:09:21.597476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '147', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:09:45.697 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:09:45.755 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:45.759+0000 7ffbca1f9240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:09:45.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.834+0000 7ff50c585700 1 Processor -- start 2024-03-18T19:09:45.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.834+0000 7ff50c585700 1 -- start start 2024-03-18T19:09:45.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.835+0000 7ff50c585700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff50415b6c0 0x7ff5040b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:45.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.835+0000 7ff50c585700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5040cd430 0x7ff5040c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:45.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.835+0000 7ff50c585700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff5040c8600 0x7ff5040c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:45.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.835+0000 7ff50c585700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5040c8ff0 con 0x7ff5040c8600 2024-03-18T19:09:45.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.835+0000 7ff50c585700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5040d01f0 con 0x7ff50415b6c0 2024-03-18T19:09:45.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.835+0000 7ff50c585700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5040ccfa0 con 0x7ff5040cd430 2024-03-18T19:09:45.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.835+0000 7ff50ab22700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff5040c8600 0x7ff5040c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:45.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.835+0000 7ff50ab22700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff5040c8600 0x7ff5040c8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43838/0 (socket says 172.21.15.1:43838) 2024-03-18T19:09:45.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.835+0000 7ff50ab22700 1 -- 172.21.15.1:0/3956893752 learned_addr learned my addr 172.21.15.1:0/3956893752 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:45.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.835+0000 7ff509b20700 1 --2- 172.21.15.1:0/3956893752 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5040cd430 0x7ff5040c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:45.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.835+0000 7ff50a321700 1 --2- 172.21.15.1:0/3956893752 >> v2:172.21.15.165:3300/0 conn(0x7ff50415b6c0 0x7ff5040b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:45.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.835+0000 7ff50ab22700 1 -- 172.21.15.1:0/3956893752 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5040cd430 msgr2=0x7ff5040c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:45.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.835+0000 7ff50ab22700 1 --2- 172.21.15.1:0/3956893752 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5040cd430 0x7ff5040c80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.835+0000 7ff50ab22700 1 -- 172.21.15.1:0/3956893752 >> v2:172.21.15.165:3300/0 conn(0x7ff50415b6c0 msgr2=0x7ff5040b2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:45.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.835+0000 7ff50ab22700 1 --2- 172.21.15.1:0/3956893752 >> v2:172.21.15.165:3300/0 conn(0x7ff50415b6c0 0x7ff5040b2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.835+0000 7ff50ab22700 1 -- 172.21.15.1:0/3956893752 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5040d1db0 con 0x7ff5040c8600 2024-03-18T19:09:45.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.835+0000 7ff50ab22700 1 --2- 172.21.15.1:0/3956893752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5040c8600 0x7ff5040c8ab0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009e50 tx=0x7ff4fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523dbe41de362009 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:45.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.835+0000 7ff50931f700 1 -- 172.21.15.1:0/3956893752 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc00e070 con 0x7ff5040c8600 2024-03-18T19:09:45.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.836+0000 7ff50931f700 1 -- 172.21.15.1:0/3956893752 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4fc00b8d0 con 0x7ff5040c8600 2024-03-18T19:09:45.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.836+0000 7ff50931f700 1 -- 172.21.15.1:0/3956893752 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc00bc80 con 0x7ff5040c8600 2024-03-18T19:09:45.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.836+0000 7ff50c585700 1 -- 172.21.15.1:0/3956893752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5040c8600 msgr2=0x7ff5040c8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:45.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.836+0000 7ff50c585700 1 --2- 172.21.15.1:0/3956893752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5040c8600 0x7ff5040c8ab0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009e50 tx=0x7ff4fc00b330 comp rx=0 tx=0).stop 2024-03-18T19:09:45.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.836+0000 7ff50c585700 1 -- 172.21.15.1:0/3956893752 shutdown_connections 2024-03-18T19:09:45.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.836+0000 7ff50c585700 1 --2- 172.21.15.1:0/3956893752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5040c8600 0x7ff5040c8ab0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.836+0000 7ff50c585700 1 --2- 172.21.15.1:0/3956893752 >> v2:172.21.15.165:3300/0 conn(0x7ff50415b6c0 0x7ff5040b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.836+0000 7ff50c585700 1 --2- 172.21.15.1:0/3956893752 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5040cd430 0x7ff5040c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.836+0000 7ff50c585700 1 -- 172.21.15.1:0/3956893752 >> 172.21.15.1:0/3956893752 conn(0x7ff5040d0ff0 msgr2=0x7ff5040d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:45.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.836+0000 7ff50c585700 1 -- 172.21.15.1:0/3956893752 shutdown_connections 2024-03-18T19:09:45.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.836+0000 7ff50c585700 1 -- 172.21.15.1:0/3956893752 wait complete. 2024-03-18T19:09:45.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.838+0000 7ff50c585700 1 Processor -- start 2024-03-18T19:09:45.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.838+0000 7ff50c585700 1 -- start start 2024-03-18T19:09:45.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.839+0000 7ff50c585700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5040c8600 0x7ff50414cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:45.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.839+0000 7ff50c585700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5040cd430 0x7ff50409d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:45.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.839+0000 7ff50c585700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff50415b6c0 0x7ff50409dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:45.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.839+0000 7ff50c585700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50406d240 con 0x7ff5040cd430 2024-03-18T19:09:45.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.839+0000 7ff50c585700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50414d7a0 con 0x7ff5040c8600 2024-03-18T19:09:45.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.839+0000 7ff50c585700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50414da80 con 0x7ff50415b6c0 2024-03-18T19:09:45.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.839+0000 7ff509b20700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5040cd430 0x7ff50409d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:45.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.839+0000 7ff509b20700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5040cd430 0x7ff50409d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43854/0 (socket says 172.21.15.1:43854) 2024-03-18T19:09:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.839+0000 7ff509b20700 1 -- 172.21.15.1:0/80303153 learned_addr learned my addr 172.21.15.1:0/80303153 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.839+0000 7ff50a321700 1 --2- 172.21.15.1:0/80303153 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5040c8600 0x7ff50414cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.839+0000 7ff50ab22700 1 --2- 172.21.15.1:0/80303153 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff50415b6c0 0x7ff50409dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.839+0000 7ff509b20700 1 -- 172.21.15.1:0/80303153 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff50415b6c0 msgr2=0x7ff50409dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.839+0000 7ff509b20700 1 --2- 172.21.15.1:0/80303153 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff50415b6c0 0x7ff50409dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.839+0000 7ff509b20700 1 -- 172.21.15.1:0/80303153 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5040c8600 msgr2=0x7ff50414cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.839+0000 7ff509b20700 1 --2- 172.21.15.1:0/80303153 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5040c8600 0x7ff50414cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.839+0000 7ff509b20700 1 -- 172.21.15.1:0/80303153 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4fc009b30 con 0x7ff5040cd430 2024-03-18T19:09:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.840+0000 7ff509b20700 1 --2- 172.21.15.1:0/80303153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5040cd430 0x7ff50409d770 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400e930 tx=0x7ff4f400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.840+0000 7ff4faffd700 1 -- 172.21.15.1:0/80303153 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f4013680 con 0x7ff5040cd430 2024-03-18T19:09:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.840+0000 7ff4faffd700 1 -- 172.21.15.1:0/80303153 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f40137e0 con 0x7ff5040cd430 2024-03-18T19:09:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.840+0000 7ff50c585700 1 -- 172.21.15.1:0/80303153 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50414dd60 con 0x7ff5040cd430 2024-03-18T19:09:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.840+0000 7ff4faffd700 1 -- 172.21.15.1:0/80303153 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f4013b70 con 0x7ff5040cd430 2024-03-18T19:09:45.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.840+0000 7ff50c585700 1 -- 172.21.15.1:0/80303153 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff50409adc0 con 0x7ff5040cd430 2024-03-18T19:09:45.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.841+0000 7ff50c585700 1 -- 172.21.15.1:0/80303153 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff50414e280 con 0x7ff5040cd430 2024-03-18T19:09:45.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.841+0000 7ff4faffd700 1 -- 172.21.15.1:0/80303153 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 78) v1 ==== 89296+0+0 (secure 0 0 0) 0x7ff4f4013cd0 con 0x7ff5040cd430 2024-03-18T19:09:45.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.842+0000 7ff4faffd700 1 --2- 172.21.15.1:0/80303153 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff4e806ae40 0x7ff4e806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:45.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.842+0000 7ff50a321700 1 -- 172.21.15.1:0/80303153 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff4e806ae40 msgr2=0x7ff4e806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:45.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.842+0000 7ff4faffd700 1 -- 172.21.15.1:0/80303153 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7ff4f4089830 con 0x7ff5040cd430 2024-03-18T19:09:45.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.842+0000 7ff50a321700 1 --2- 172.21.15.1:0/80303153 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff4e806ae40 0x7ff4e806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:45.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:45.850+0000 7ff4faffd700 1 -- 172.21.15.1:0/80303153 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff50414e280 con 0x7ff5040cd430 2024-03-18T19:09:46.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.043+0000 7ff50a321700 1 -- 172.21.15.1:0/80303153 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff4e806ae40 msgr2=0x7ff4e806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:46.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.043+0000 7ff50a321700 1 --2- 172.21.15.1:0/80303153 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff4e806ae40 0x7ff4e806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:46.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.052+0000 7ff50c585700 1 -- 172.21.15.1:0/80303153 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff50414e280 con 0x7ff5040cd430 2024-03-18T19:09:46.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.052+0000 7ff4faffd700 1 -- 172.21.15.1:0/80303153 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v148) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff50414e280 con 0x7ff5040cd430 2024-03-18T19:09:46.048 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:09:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.055+0000 7ff50c585700 1 -- 172.21.15.1:0/80303153 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff4e806ae40 msgr2=0x7ff4e806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.055+0000 7ff50c585700 1 --2- 172.21.15.1:0/80303153 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff4e806ae40 0x7ff4e806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.055+0000 7ff50c585700 1 -- 172.21.15.1:0/80303153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5040cd430 msgr2=0x7ff50409d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.055+0000 7ff50c585700 1 --2- 172.21.15.1:0/80303153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5040cd430 0x7ff50409d770 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400e930 tx=0x7ff4f400ec40 comp rx=0 tx=0).stop 2024-03-18T19:09:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.055+0000 7ff50c585700 1 -- 172.21.15.1:0/80303153 shutdown_connections 2024-03-18T19:09:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.055+0000 7ff50c585700 1 --2- 172.21.15.1:0/80303153 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff4e806ae40 0x7ff4e806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.055+0000 7ff50c585700 1 --2- 172.21.15.1:0/80303153 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff50415b6c0 0x7ff50409dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.055+0000 7ff50c585700 1 --2- 172.21.15.1:0/80303153 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5040c8600 0x7ff50414cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.055+0000 7ff50c585700 1 --2- 172.21.15.1:0/80303153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5040cd430 0x7ff50409d770 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.055+0000 7ff50c585700 1 -- 172.21.15.1:0/80303153 >> 172.21.15.1:0/80303153 conn(0x7ff5040d0ff0 msgr2=0x7ff5040cd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.055+0000 7ff50c585700 1 -- 172.21.15.1:0/80303153 shutdown_connections 2024-03-18T19:09:46.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.055+0000 7ff50c585700 1 -- 172.21.15.1:0/80303153 wait complete. 2024-03-18T19:09:46.064 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T19:09:46.108 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:46.111+0000 7f441fadd240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:09:46.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.198+0000 7f339ff84700 1 Processor -- start 2024-03-18T19:09:46.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.198+0000 7f339ff84700 1 -- start start 2024-03-18T19:09:46.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.199+0000 7f339ff84700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f339815a1d0 0x7f33980add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:46.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.199+0000 7f339ff84700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3398071620 0x7f3398071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:46.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.199+0000 7f339ff84700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33980ce8f0 con 0x7f33980cb9e0 2024-03-18T19:09:46.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.199+0000 7f339ff84700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33980cdb00 con 0x7f3398071620 2024-03-18T19:09:46.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.199+0000 7f339ff84700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f339815a5b0 con 0x7f339815a1d0 2024-03-18T19:09:46.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.199+0000 7f339d51f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3398071620 0x7f3398071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:46.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.199+0000 7f339e521700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f33980cb9e0 0x7f33980cbe50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59506/0 (socket says 172.21.15.1:59506) 2024-03-18T19:09:46.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.199+0000 7f339e521700 1 -- 172.21.15.1:0/120831481 learned_addr learned my addr 172.21.15.1:0/120831481 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:46.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.199+0000 7f339dd20700 1 --2- 172.21.15.1:0/120831481 >> v2:172.21.15.165:3300/0 conn(0x7f339815a1d0 0x7f33980add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:46.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.200+0000 7f339d51f700 1 -- 172.21.15.1:0/120831481 >> v2:172.21.15.165:3300/0 conn(0x7f339815a1d0 msgr2=0x7f33980add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:46.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.200+0000 7f339d51f700 1 --2- 172.21.15.1:0/120831481 >> v2:172.21.15.165:3300/0 conn(0x7f339815a1d0 0x7f33980add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.200+0000 7f339d51f700 1 -- 172.21.15.1:0/120831481 >> v1:172.21.15.1:6789/0 conn(0x7f33980cb9e0 legacy=0x7f33980cbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:46.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.200+0000 7f339d51f700 1 -- 172.21.15.1:0/120831481 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3398071fc0 con 0x7f3398071620 2024-03-18T19:09:46.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.200+0000 7f339cd1e700 1 -- 172.21.15.1:0/120831481 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533355382 0 0) 0x7f33980ce8f0 con 0x7f33980cb9e0 2024-03-18T19:09:46.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.200+0000 7f339dd20700 1 --2- 172.21.15.1:0/120831481 >> v2:172.21.15.165:3300/0 conn(0x7f339815a1d0 0x7f33980add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:46.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.200+0000 7f339d51f700 1 --2- 172.21.15.1:0/120831481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3398071620 0x7f3398071a80 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f33980726b0 tx=0x7f3394009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1176542a99466759 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:46.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.200+0000 7f339cd1e700 1 -- 172.21.15.1:0/120831481 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3394011830 con 0x7f3398071620 2024-03-18T19:09:46.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.200+0000 7f339cd1e700 1 -- 172.21.15.1:0/120831481 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3394011990 con 0x7f3398071620 2024-03-18T19:09:46.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.200+0000 7f339cd1e700 1 -- 172.21.15.1:0/120831481 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3394011d20 con 0x7f3398071620 2024-03-18T19:09:46.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.200+0000 7f339ff84700 1 -- 172.21.15.1:0/120831481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3398071620 msgr2=0x7f3398071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:46.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.200+0000 7f339ff84700 1 --2- 172.21.15.1:0/120831481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3398071620 0x7f3398071a80 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f33980726b0 tx=0x7f3394009b20 comp rx=0 tx=0).stop 2024-03-18T19:09:46.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.200+0000 7f339ff84700 1 -- 172.21.15.1:0/120831481 shutdown_connections 2024-03-18T19:09:46.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.200+0000 7f339ff84700 1 --2- 172.21.15.1:0/120831481 >> v2:172.21.15.165:3300/0 conn(0x7f339815a1d0 0x7f33980add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.200+0000 7f339ff84700 1 --2- 172.21.15.1:0/120831481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3398071620 0x7f3398071a80 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.201+0000 7f339ff84700 1 -- 172.21.15.1:0/120831481 >> 172.21.15.1:0/120831481 conn(0x7f33980cc9b0 msgr2=0x7f33980ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:46.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.201+0000 7f339ff84700 1 -- 172.21.15.1:0/120831481 shutdown_connections 2024-03-18T19:09:46.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.201+0000 7f339ff84700 1 -- 172.21.15.1:0/120831481 wait complete. 2024-03-18T19:09:46.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.202+0000 7f339ff84700 1 Processor -- start 2024-03-18T19:09:46.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.202+0000 7f339ff84700 1 -- start start 2024-03-18T19:09:46.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f339ff84700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3398071620 0x7f3398153020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:46.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f339ff84700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33980cb9e0 0x7f339815c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f339ff84700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339815a1d0 0x7f339815c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f339ff84700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33980c87f0 con 0x7f339815a1d0 2024-03-18T19:09:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f339ff84700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3398153af0 con 0x7f33980cb9e0 2024-03-18T19:09:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f339ff84700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33980c7be0 con 0x7f3398071620 2024-03-18T19:09:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f339e521700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339815a1d0 0x7f339815c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f339e521700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339815a1d0 0x7f339815c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43868/0 (socket says 172.21.15.1:43868) 2024-03-18T19:09:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f339e521700 1 -- 172.21.15.1:0/2811998526 learned_addr learned my addr 172.21.15.1:0/2811998526 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f339d51f700 1 --2- 172.21.15.1:0/2811998526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33980cb9e0 0x7f339815c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f339dd20700 1 --2- 172.21.15.1:0/2811998526 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3398071620 0x7f3398153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f339e521700 1 -- 172.21.15.1:0/2811998526 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3398071620 msgr2=0x7f3398153020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f339e521700 1 --2- 172.21.15.1:0/2811998526 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3398071620 0x7f3398153020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f339e521700 1 -- 172.21.15.1:0/2811998526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33980cb9e0 msgr2=0x7f339815c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f339e521700 1 --2- 172.21.15.1:0/2811998526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33980cb9e0 0x7f339815c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f339e521700 1 -- 172.21.15.1:0/2811998526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33940094a0 con 0x7f339815a1d0 2024-03-18T19:09:46.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f339e521700 1 --2- 172.21.15.1:0/2811998526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339815a1d0 0x7f339815c590 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f339000cc10 tx=0x7f339000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:46.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f338e7fc700 1 -- 172.21.15.1:0/2811998526 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339000aa60 con 0x7f339815a1d0 2024-03-18T19:09:46.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f338e7fc700 1 -- 172.21.15.1:0/2811998526 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f339000abc0 con 0x7f339815a1d0 2024-03-18T19:09:46.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f339ff84700 1 -- 172.21.15.1:0/2811998526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33980c7ec0 con 0x7f339815a1d0 2024-03-18T19:09:46.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f338e7fc700 1 -- 172.21.15.1:0/2811998526 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33900070d0 con 0x7f339815a1d0 2024-03-18T19:09:46.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.203+0000 7f339ff84700 1 -- 172.21.15.1:0/2811998526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33980c8480 con 0x7f339815a1d0 2024-03-18T19:09:46.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.204+0000 7f339ff84700 1 -- 172.21.15.1:0/2811998526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33980676a0 con 0x7f339815a1d0 2024-03-18T19:09:46.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.206+0000 7f338e7fc700 1 -- 172.21.15.1:0/2811998526 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 78) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f3390007230 con 0x7f339815a1d0 2024-03-18T19:09:46.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.206+0000 7f338e7fc700 1 --2- 172.21.15.1:0/2811998526 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f337c06ae40 0x7f337c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:46.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.206+0000 7f339dd20700 1 -- 172.21.15.1:0/2811998526 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f337c06ae40 msgr2=0x7f337c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:46.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.206+0000 7f338e7fc700 1 -- 172.21.15.1:0/2811998526 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f33900897d0 con 0x7f339815a1d0 2024-03-18T19:09:46.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.206+0000 7f339dd20700 1 --2- 172.21.15.1:0/2811998526 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f337c06ae40 0x7f337c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:46.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.210+0000 7f338e7fc700 1 -- 172.21.15.1:0/2811998526 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f33900545e0 con 0x7f339815a1d0 2024-03-18T19:09:46.221 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:46.226+0000 7ffbca1f9240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:09:46.319 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:46.322+0000 7f441fadd240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:09:46.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.407+0000 7f339dd20700 1 -- 172.21.15.1:0/2811998526 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f337c06ae40 msgr2=0x7f337c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:46.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.407+0000 7f339dd20700 1 --2- 172.21.15.1:0/2811998526 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f337c06ae40 0x7f337c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:46.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.412+0000 7f339ff84700 1 -- 172.21.15.1:0/2811998526 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f33980cb590 con 0x7f339815a1d0 2024-03-18T19:09:46.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.412+0000 7f338e7fc700 1 -- 172.21.15.1:0/2811998526 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v148) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3390018020 con 0x7f339815a1d0 2024-03-18T19:09:46.409 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T19:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.414+0000 7f339ff84700 1 -- 172.21.15.1:0/2811998526 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f337c06ae40 msgr2=0x7f337c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.414+0000 7f339ff84700 1 --2- 172.21.15.1:0/2811998526 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f337c06ae40 0x7f337c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.414+0000 7f339ff84700 1 -- 172.21.15.1:0/2811998526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339815a1d0 msgr2=0x7f339815c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.414+0000 7f339ff84700 1 --2- 172.21.15.1:0/2811998526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339815a1d0 0x7f339815c590 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f339000cc10 tx=0x7f339000cf20 comp rx=0 tx=0).stop 2024-03-18T19:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.415+0000 7f339ff84700 1 -- 172.21.15.1:0/2811998526 shutdown_connections 2024-03-18T19:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.415+0000 7f339ff84700 1 --2- 172.21.15.1:0/2811998526 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f337c06ae40 0x7f337c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.415+0000 7f339ff84700 1 --2- 172.21.15.1:0/2811998526 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3398071620 0x7f3398153020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.415+0000 7f339ff84700 1 --2- 172.21.15.1:0/2811998526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339815a1d0 0x7f339815c590 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.415+0000 7f339ff84700 1 --2- 172.21.15.1:0/2811998526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33980cb9e0 0x7f339815c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.415+0000 7f339ff84700 1 -- 172.21.15.1:0/2811998526 >> 172.21.15.1:0/2811998526 conn(0x7f33980cc9b0 msgr2=0x7f339806fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:46.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.415+0000 7f339ff84700 1 -- 172.21.15.1:0/2811998526 shutdown_connections 2024-03-18T19:09:46.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.415+0000 7f339ff84700 1 -- 172.21.15.1:0/2811998526 wait complete. 2024-03-18T19:09:46.423 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T19:09:46.445 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:46.449+0000 7ffbca1f9240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:09:46.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.559+0000 7fbfbf7ea700 1 Processor -- start 2024-03-18T19:09:46.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.559+0000 7fbfbf7ea700 1 -- start start 2024-03-18T19:09:46.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.560+0000 7fbfbf7ea700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbfb806d560 0x7fbfb80ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:46.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.560+0000 7fbfbf7ea700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb80cac80 con 0x7fbfb8071cc0 2024-03-18T19:09:46.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.560+0000 7fbfbf7ea700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb80caf60 con 0x7fbfb815a240 2024-03-18T19:09:46.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.560+0000 7fbfbf7ea700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfb80adff0 con 0x7fbfb806d560 2024-03-18T19:09:46.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.560+0000 7fbfbdd87700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbfb806d560 0x7fbfb80ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:46.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.560+0000 7fbfbcd85700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbfb8071cc0 0x7fbfb80ae410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59508/0 (socket says 172.21.15.1:59508) 2024-03-18T19:09:46.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.560+0000 7fbfbcd85700 1 -- 172.21.15.1:0/1384981029 learned_addr learned my addr 172.21.15.1:0/1384981029 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:46.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.561+0000 7fbfbdd87700 1 -- 172.21.15.1:0/1384981029 >> v1:172.21.15.165:6789/0 conn(0x7fbfb815a240 legacy=0x7fbfb80715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.561+0000 7fbfbdd87700 1 -- 172.21.15.1:0/1384981029 >> v1:172.21.15.1:6789/0 conn(0x7fbfb8071cc0 legacy=0x7fbfb80ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.561+0000 7fbfbdd87700 1 -- 172.21.15.1:0/1384981029 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb80cb240 con 0x7fbfb806d560 2024-03-18T19:09:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.561+0000 7fbfaffff700 1 -- 172.21.15.1:0/1384981029 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2959771243 0 0) 0x7fbfb80cac80 con 0x7fbfb8071cc0 2024-03-18T19:09:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.561+0000 7fbfbdd87700 1 --2- 172.21.15.1:0/1384981029 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb806d560 0x7fbfb80ca580 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009d90 tx=0x7fbfb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=602856026fad8d74 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.561+0000 7fbfaffff700 1 -- 172.21.15.1:0/1384981029 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb000e070 con 0x7fbfb806d560 2024-03-18T19:09:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.561+0000 7fbfaffff700 1 -- 172.21.15.1:0/1384981029 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb000bbd0 con 0x7fbfb806d560 2024-03-18T19:09:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.561+0000 7fbfaffff700 1 -- 172.21.15.1:0/1384981029 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0012430 con 0x7fbfb806d560 2024-03-18T19:09:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.561+0000 7fbfbf7ea700 1 -- 172.21.15.1:0/1384981029 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb806d560 msgr2=0x7fbfb80ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.562+0000 7fbfbf7ea700 1 --2- 172.21.15.1:0/1384981029 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb806d560 0x7fbfb80ca580 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009d90 tx=0x7fbfb000b330 comp rx=0 tx=0).stop 2024-03-18T19:09:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.562+0000 7fbfbf7ea700 1 -- 172.21.15.1:0/1384981029 shutdown_connections 2024-03-18T19:09:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.562+0000 7fbfbf7ea700 1 --2- 172.21.15.1:0/1384981029 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb806d560 0x7fbfb80ca580 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.562+0000 7fbfbf7ea700 1 -- 172.21.15.1:0/1384981029 >> 172.21.15.1:0/1384981029 conn(0x7fbfb80cff30 msgr2=0x7fbfb80d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.562+0000 7fbfbf7ea700 1 -- 172.21.15.1:0/1384981029 shutdown_connections 2024-03-18T19:09:46.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.562+0000 7fbfbf7ea700 1 -- 172.21.15.1:0/1384981029 wait complete. 2024-03-18T19:09:46.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.563+0000 7fbfbf7ea700 1 Processor -- start 2024-03-18T19:09:46.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.563+0000 7fbfbf7ea700 1 -- start start 2024-03-18T19:09:46.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfbf7ea700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfb806d560 0x7fbfb8157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:46.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfbf7ea700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb8071cc0 0x7fbfb80c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:46.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfbf7ea700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfb815a240 0x7fbfb80c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:46.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfbf7ea700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb80c3bb0 con 0x7fbfb8071cc0 2024-03-18T19:09:46.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfbf7ea700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb80c3e90 con 0x7fbfb806d560 2024-03-18T19:09:46.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfbf7ea700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb80c4170 con 0x7fbfb815a240 2024-03-18T19:09:46.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfbcd85700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb8071cc0 0x7fbfb80c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:46.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfbcd85700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb8071cc0 0x7fbfb80c94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43888/0 (socket says 172.21.15.1:43888) 2024-03-18T19:09:46.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfbcd85700 1 -- 172.21.15.1:0/2829253501 learned_addr learned my addr 172.21.15.1:0/2829253501 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:46.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfbd586700 1 --2- 172.21.15.1:0/2829253501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfb806d560 0x7fbfb8157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:46.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfbdd87700 1 --2- 172.21.15.1:0/2829253501 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfb815a240 0x7fbfb80c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:46.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfbcd85700 1 -- 172.21.15.1:0/2829253501 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfb815a240 msgr2=0x7fbfb80c99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:46.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfbcd85700 1 --2- 172.21.15.1:0/2829253501 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfb815a240 0x7fbfb80c99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfbcd85700 1 -- 172.21.15.1:0/2829253501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfb806d560 msgr2=0x7fbfb8157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:46.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfbcd85700 1 --2- 172.21.15.1:0/2829253501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfb806d560 0x7fbfb8157dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfbcd85700 1 -- 172.21.15.1:0/2829253501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb0009a70 con 0x7fbfb8071cc0 2024-03-18T19:09:46.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfbcd85700 1 --2- 172.21.15.1:0/2829253501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb8071cc0 0x7fbfb80c94b0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800c520 tx=0x7fbfa800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:46.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfadffb700 1 -- 172.21.15.1:0/2829253501 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8004c80 con 0x7fbfb8071cc0 2024-03-18T19:09:46.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfadffb700 1 -- 172.21.15.1:0/2829253501 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa8004e00 con 0x7fbfb8071cc0 2024-03-18T19:09:46.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfbf7ea700 1 -- 172.21.15.1:0/2829253501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb80c4450 con 0x7fbfb8071cc0 2024-03-18T19:09:46.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.564+0000 7fbfadffb700 1 -- 172.21.15.1:0/2829253501 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8007310 con 0x7fbfb8071cc0 2024-03-18T19:09:46.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.565+0000 7fbfbf7ea700 1 -- 172.21.15.1:0/2829253501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb80c4c50 con 0x7fbfb8071cc0 2024-03-18T19:09:46.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.566+0000 7fbfbf7ea700 1 -- 172.21.15.1:0/2829253501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb80c4970 con 0x7fbfb8071cc0 2024-03-18T19:09:46.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.567+0000 7fbfadffb700 1 -- 172.21.15.1:0/2829253501 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 78) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fbfa8007470 con 0x7fbfb8071cc0 2024-03-18T19:09:46.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.567+0000 7fbfadffb700 1 --2- 172.21.15.1:0/2829253501 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf9c06ae40 0x7fbf9c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:46.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.567+0000 7fbfbd586700 1 -- 172.21.15.1:0/2829253501 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf9c06ae40 msgr2=0x7fbf9c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:46.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.567+0000 7fbfadffb700 1 -- 172.21.15.1:0/2829253501 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fbfa80894e0 con 0x7fbfb8071cc0 2024-03-18T19:09:46.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.567+0000 7fbfbd586700 1 --2- 172.21.15.1:0/2829253501 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf9c06ae40 0x7fbf9c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:46.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.572+0000 7fbfadffb700 1 -- 172.21.15.1:0/2829253501 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbfb80c4970 con 0x7fbfb8071cc0 2024-03-18T19:09:46.690 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:46.694+0000 7f441fadd240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:09:46.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.768+0000 7fbfbd586700 1 -- 172.21.15.1:0/2829253501 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf9c06ae40 msgr2=0x7fbf9c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:46.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.768+0000 7fbfbd586700 1 --2- 172.21.15.1:0/2829253501 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf9c06ae40 0x7fbf9c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:46.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.777+0000 7fbfbf7ea700 1 -- 172.21.15.1:0/2829253501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbfb80c4970 con 0x7fbfb8071cc0 2024-03-18T19:09:46.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.777+0000 7fbfadffb700 1 -- 172.21.15.1:0/2829253501 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v148) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbfb80c4970 con 0x7fbfb8071cc0 2024-03-18T19:09:46.773 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T19:09:46.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.780+0000 7fbfbf7ea700 1 -- 172.21.15.1:0/2829253501 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf9c06ae40 msgr2=0x7fbf9c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:46.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.780+0000 7fbfbf7ea700 1 --2- 172.21.15.1:0/2829253501 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf9c06ae40 0x7fbf9c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.780+0000 7fbfbf7ea700 1 -- 172.21.15.1:0/2829253501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb8071cc0 msgr2=0x7fbfb80c94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:46.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.780+0000 7fbfbf7ea700 1 --2- 172.21.15.1:0/2829253501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb8071cc0 0x7fbfb80c94b0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800c520 tx=0x7fbfa800c830 comp rx=0 tx=0).stop 2024-03-18T19:09:46.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.780+0000 7fbfbf7ea700 1 -- 172.21.15.1:0/2829253501 shutdown_connections 2024-03-18T19:09:46.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.780+0000 7fbfbf7ea700 1 --2- 172.21.15.1:0/2829253501 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf9c06ae40 0x7fbf9c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.780+0000 7fbfbf7ea700 1 --2- 172.21.15.1:0/2829253501 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfb815a240 0x7fbfb80c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.780+0000 7fbfbf7ea700 1 --2- 172.21.15.1:0/2829253501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfb806d560 0x7fbfb8157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.780+0000 7fbfbf7ea700 1 --2- 172.21.15.1:0/2829253501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb8071cc0 0x7fbfb80c94b0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.780+0000 7fbfbf7ea700 1 -- 172.21.15.1:0/2829253501 >> 172.21.15.1:0/2829253501 conn(0x7fbfb80cff30 msgr2=0x7fbfb81593e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:46.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.780+0000 7fbfbf7ea700 1 -- 172.21.15.1:0/2829253501 shutdown_connections 2024-03-18T19:09:46.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.780+0000 7fbfbf7ea700 1 -- 172.21.15.1:0/2829253501 wait complete. 2024-03-18T19:09:46.790 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T19:09:46.844 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:46.848+0000 7ffbca1f9240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:09:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.961+0000 7fa43e4d2700 1 Processor -- start 2024-03-18T19:09:46.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.961+0000 7fa43e4d2700 1 -- start start 2024-03-18T19:09:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.962+0000 7fa43e4d2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa43815a240 0x7fa4380ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.962+0000 7fa43e4d2700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4380cac80 con 0x7fa43806d560 2024-03-18T19:09:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.962+0000 7fa43e4d2700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4380caf60 con 0x7fa438071cc0 2024-03-18T19:09:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.962+0000 7fa43e4d2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4380adff0 con 0x7fa43815a240 2024-03-18T19:09:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.962+0000 7fa4377fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa43806d560 0x7fa4380715b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59516/0 (socket says 172.21.15.1:59516) 2024-03-18T19:09:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.962+0000 7fa4377fe700 1 -- 172.21.15.1:0/2059057615 learned_addr learned my addr 172.21.15.1:0/2059057615 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.962+0000 7fa437fff700 1 --2- 172.21.15.1:0/2059057615 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa43815a240 0x7fa4380ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.963+0000 7fa436ffd700 1 -- 172.21.15.1:0/2059057615 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429466093 0 0) 0x7fa4380cac80 con 0x7fa43806d560 2024-03-18T19:09:46.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.963+0000 7fa436ffd700 1 -- 172.21.15.1:0/2059057615 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4240035b0 con 0x7fa43806d560 2024-03-18T19:09:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.963+0000 7fa436ffd700 1 -- 172.21.15.1:0/2059057615 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646976548 0 0) 0x7fa4380caf60 con 0x7fa438071cc0 2024-03-18T19:09:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.963+0000 7fa436ffd700 1 -- 172.21.15.1:0/2059057615 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4380cac80 con 0x7fa438071cc0 2024-03-18T19:09:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.963+0000 7fa437fff700 1 -- 172.21.15.1:0/2059057615 >> v1:172.21.15.165:6789/0 conn(0x7fa438071cc0 legacy=0x7fa4380ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.963+0000 7fa437fff700 1 -- 172.21.15.1:0/2059057615 >> v1:172.21.15.1:6789/0 conn(0x7fa43806d560 legacy=0x7fa4380715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.963+0000 7fa437fff700 1 -- 172.21.15.1:0/2059057615 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4380cb240 con 0x7fa43815a240 2024-03-18T19:09:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.963+0000 7fa437fff700 1 --2- 172.21.15.1:0/2059057615 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa43815a240 0x7fa4380ca580 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa420009870 tx=0x7fa420009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6de2bd324a5d410f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.963+0000 7fa436ffd700 1 -- 172.21.15.1:0/2059057615 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa420011830 con 0x7fa43815a240 2024-03-18T19:09:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.963+0000 7fa436ffd700 1 -- 172.21.15.1:0/2059057615 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa420011990 con 0x7fa43815a240 2024-03-18T19:09:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.963+0000 7fa436ffd700 1 -- 172.21.15.1:0/2059057615 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa420011d20 con 0x7fa43815a240 2024-03-18T19:09:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.964+0000 7fa43e4d2700 1 -- 172.21.15.1:0/2059057615 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa43815a240 msgr2=0x7fa4380ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.964+0000 7fa43e4d2700 1 --2- 172.21.15.1:0/2059057615 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa43815a240 0x7fa4380ca580 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa420009870 tx=0x7fa420009b80 comp rx=0 tx=0).stop 2024-03-18T19:09:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.964+0000 7fa43e4d2700 1 -- 172.21.15.1:0/2059057615 shutdown_connections 2024-03-18T19:09:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.964+0000 7fa43e4d2700 1 --2- 172.21.15.1:0/2059057615 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa43815a240 0x7fa4380ca580 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.964+0000 7fa43e4d2700 1 -- 172.21.15.1:0/2059057615 >> 172.21.15.1:0/2059057615 conn(0x7fa4380cff30 msgr2=0x7fa4380d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:46.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.964+0000 7fa43e4d2700 1 -- 172.21.15.1:0/2059057615 shutdown_connections 2024-03-18T19:09:46.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.964+0000 7fa43e4d2700 1 -- 172.21.15.1:0/2059057615 wait complete. 2024-03-18T19:09:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.965+0000 7fa43e4d2700 1 Processor -- start 2024-03-18T19:09:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.965+0000 7fa43e4d2700 1 -- start start 2024-03-18T19:09:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.966+0000 7fa43e4d2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa43806d560 0x7fa4380cbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.966+0000 7fa43e4d2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa438071cc0 0x7fa438155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.966+0000 7fa43e4d2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa43815a240 0x7fa4381556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.966+0000 7fa43e4d2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4381f7450 con 0x7fa43806d560 2024-03-18T19:09:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.966+0000 7fa43e4d2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4381e1fd0 con 0x7fa438071cc0 2024-03-18T19:09:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.966+0000 7fa43e4d2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4381e23f0 con 0x7fa43815a240 2024-03-18T19:09:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.966+0000 7fa437fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa43806d560 0x7fa4380cbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:46.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.966+0000 7fa437fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa43806d560 0x7fa4380cbbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43906/0 (socket says 172.21.15.1:43906) 2024-03-18T19:09:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.966+0000 7fa437fff700 1 -- 172.21.15.1:0/2338611926 learned_addr learned my addr 172.21.15.1:0/2338611926 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.966+0000 7fa4377fe700 1 --2- 172.21.15.1:0/2338611926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa438071cc0 0x7fa438155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.966+0000 7fa43ca6f700 1 --2- 172.21.15.1:0/2338611926 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa43815a240 0x7fa4381556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.966+0000 7fa437fff700 1 -- 172.21.15.1:0/2338611926 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa43815a240 msgr2=0x7fa4381556d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.966+0000 7fa437fff700 1 --2- 172.21.15.1:0/2338611926 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa43815a240 0x7fa4381556d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.966+0000 7fa437fff700 1 -- 172.21.15.1:0/2338611926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa438071cc0 msgr2=0x7fa438155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.966+0000 7fa437fff700 1 --2- 172.21.15.1:0/2338611926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa438071cc0 0x7fa438155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.966+0000 7fa437fff700 1 -- 172.21.15.1:0/2338611926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4200094a0 con 0x7fa43806d560 2024-03-18T19:09:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.967+0000 7fa437fff700 1 --2- 172.21.15.1:0/2338611926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa43806d560 0x7fa4380cbbb0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fa420009bc0 tx=0x7fa420004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.967+0000 7fa434ff9700 1 -- 172.21.15.1:0/2338611926 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42001a6e0 con 0x7fa43806d560 2024-03-18T19:09:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.967+0000 7fa434ff9700 1 -- 172.21.15.1:0/2338611926 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa42001a860 con 0x7fa43806d560 2024-03-18T19:09:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.967+0000 7fa434ff9700 1 -- 172.21.15.1:0/2338611926 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42001ac10 con 0x7fa43806d560 2024-03-18T19:09:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.967+0000 7fa43e4d2700 1 -- 172.21.15.1:0/2338611926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4381e1630 con 0x7fa43806d560 2024-03-18T19:09:46.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.967+0000 7fa43e4d2700 1 -- 172.21.15.1:0/2338611926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4381e1bf0 con 0x7fa43806d560 2024-03-18T19:09:46.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.968+0000 7fa43e4d2700 1 -- 172.21.15.1:0/2338611926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4380676a0 con 0x7fa43806d560 2024-03-18T19:09:46.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.968+0000 7fa434ff9700 1 -- 172.21.15.1:0/2338611926 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 78) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fa420022070 con 0x7fa43806d560 2024-03-18T19:09:46.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.969+0000 7fa434ff9700 1 --2- 172.21.15.1:0/2338611926 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa41806ae40 0x7fa41806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:46.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.969+0000 7fa4377fe700 1 -- 172.21.15.1:0/2338611926 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa41806ae40 msgr2=0x7fa41806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.969+0000 7fa434ff9700 1 -- 172.21.15.1:0/2338611926 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fa42008a440 con 0x7fa43806d560 2024-03-18T19:09:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.969+0000 7fa4377fe700 1 --2- 172.21.15.1:0/2338611926 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa41806ae40 0x7fa41806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:46.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:46.973+0000 7fa434ff9700 1 -- 172.21.15.1:0/2338611926 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa420055170 con 0x7fa43806d560 2024-03-18T19:09:47.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.170+0000 7fa4377fe700 1 -- 172.21.15.1:0/2338611926 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa41806ae40 msgr2=0x7fa41806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:47.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.170+0000 7fa4377fe700 1 --2- 172.21.15.1:0/2338611926 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa41806ae40 0x7fa41806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:47.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.193+0000 7fa43e4d2700 1 -- 172.21.15.1:0/2338611926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa438072860 con 0x7fa43806d560 2024-03-18T19:09:47.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.194+0000 7fa434ff9700 1 -- 172.21.15.1:0/2338611926 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v148) v1 ==== 80+0+447 (secure 0 0 0) 0x7fa420058790 con 0x7fa43806d560 2024-03-18T19:09:47.189 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/69006586 2024-03-19T19:09:37.850642+0000 2024-03-18T19:09:47.190 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6833/76998 2024-03-19T19:09:37.850642+0000 2024-03-18T19:09:47.190 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6832/76998 2024-03-19T19:09:37.850642+0000 2024-03-18T19:09:47.190 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1403689662 2024-03-19T19:09:37.850642+0000 2024-03-18T19:09:47.190 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/806972823 2024-03-19T19:09:37.850642+0000 2024-03-18T19:09:47.190 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/619565916 2024-03-19T19:09:37.850642+0000 2024-03-18T19:09:47.190 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2042700873 2024-03-19T19:09:37.850642+0000 2024-03-18T19:09:47.190 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/744112928 2024-03-19T19:09:37.850642+0000 2024-03-18T19:09:47.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.196+0000 7fa43e4d2700 1 -- 172.21.15.1:0/2338611926 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa41806ae40 msgr2=0x7fa41806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:47.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.196+0000 7fa43e4d2700 1 --2- 172.21.15.1:0/2338611926 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa41806ae40 0x7fa41806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:47.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.196+0000 7fa43e4d2700 1 -- 172.21.15.1:0/2338611926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa43806d560 msgr2=0x7fa4380cbbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:47.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.196+0000 7fa43e4d2700 1 --2- 172.21.15.1:0/2338611926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa43806d560 0x7fa4380cbbb0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fa420009bc0 tx=0x7fa420004750 comp rx=0 tx=0).stop 2024-03-18T19:09:47.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.196+0000 7fa43e4d2700 1 -- 172.21.15.1:0/2338611926 shutdown_connections 2024-03-18T19:09:47.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.196+0000 7fa43e4d2700 1 --2- 172.21.15.1:0/2338611926 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa41806ae40 0x7fa41806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:47.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.196+0000 7fa43e4d2700 1 --2- 172.21.15.1:0/2338611926 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa43815a240 0x7fa4381556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:47.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.196+0000 7fa43e4d2700 1 --2- 172.21.15.1:0/2338611926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa43806d560 0x7fa4380cbbb0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:47.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.196+0000 7fa43e4d2700 1 --2- 172.21.15.1:0/2338611926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa438071cc0 0x7fa438155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:47.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.196+0000 7fa43e4d2700 1 -- 172.21.15.1:0/2338611926 >> 172.21.15.1:0/2338611926 conn(0x7fa4380cff30 msgr2=0x7fa4380b00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:47.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.196+0000 7fa43e4d2700 1 -- 172.21.15.1:0/2338611926 shutdown_connections 2024-03-18T19:09:47.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.196+0000 7fa43e4d2700 1 -- 172.21.15.1:0/2338611926 wait complete. 2024-03-18T19:09:47.193 INFO:teuthology.orchestra.run.smithi001.stderr:listed 8 entries 2024-03-18T19:09:47.204 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-18T19:09:47.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:47.205 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T19:09:47.274 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:47.278+0000 7f441fadd240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:09:47.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[129429]: fuse finished with error 0 and tester_r 0 2024-03-18T19:09:47.282 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T19:09:47.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:47.282 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T19:09:47.348 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:47.352+0000 7f441fadd240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:09:47.451 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:09:47.452 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:09:47.471 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:09:47.471 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T19:09:47.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:47.472 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-18T19:09:47.475 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:47.480+0000 7ffbca1f9240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:09:47.478 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:47.482+0000 7f441fadd240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:09:47.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:09:47.497 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-18T19:09:47.498 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:09:47.498 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:09:47.517 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:09:47.517 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T19:09:47.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:47.518 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-18T19:09:47.553 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:47.558+0000 7ffbca1f9240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:09:47.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:09:47.571 INFO:teuthology.orchestra.run.smithi001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-18T19:09:47.572 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:09:47.572 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:09:47.592 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:09:47.592 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T19:09:47.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:09:47.592 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-18T19:09:47.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T19:09:47.653 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-18T19:09:47.653 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:09:47.667 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:47.671+0000 7f441fadd240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:09:47.696 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:47.700+0000 7ffbca1f9240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:09:47.740 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:47.744+0000 7f441fadd240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:09:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.784+0000 7fea3e504700 1 Processor -- start 2024-03-18T19:09:47.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.784+0000 7fea3e504700 1 -- start start 2024-03-18T19:09:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.785+0000 7fea3e504700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fea380d05a0 0x7fea3809a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.785+0000 7fea3e504700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fea3806ba70 0x7fea3815d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.785+0000 7fea3e504700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea380cf7e0 con 0x7fea38099a40 2024-03-18T19:09:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.785+0000 7fea3e504700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea3806bec0 con 0x7fea3806ba70 2024-03-18T19:09:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.785+0000 7fea3e504700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea380cfac0 con 0x7fea380d05a0 2024-03-18T19:09:47.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.785+0000 7fea3caa1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fea3806ba70 0x7fea3815d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.785+0000 7fea3caa1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fea3806ba70 0x7fea3815d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43914/0 (socket says 172.21.15.1:43914) 2024-03-18T19:09:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.785+0000 7fea3caa1700 1 -- 172.21.15.1:0/846339232 learned_addr learned my addr 172.21.15.1:0/846339232 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.786+0000 7fea377fe700 1 --2- 172.21.15.1:0/846339232 >> v2:172.21.15.165:3300/0 conn(0x7fea380d05a0 0x7fea3809a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.786+0000 7fea3caa1700 1 -- 172.21.15.1:0/846339232 >> v2:172.21.15.165:3300/0 conn(0x7fea380d05a0 msgr2=0x7fea3809a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.786+0000 7fea3caa1700 1 --2- 172.21.15.1:0/846339232 >> v2:172.21.15.165:3300/0 conn(0x7fea380d05a0 0x7fea3809a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.786+0000 7fea3caa1700 1 -- 172.21.15.1:0/846339232 >> v1:172.21.15.165:6789/0 conn(0x7fea38099a40 legacy=0x7fea38099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.786+0000 7fea3caa1700 1 -- 172.21.15.1:0/846339232 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea380cfda0 con 0x7fea3806ba70 2024-03-18T19:09:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.786+0000 7fea3caa1700 1 --2- 172.21.15.1:0/846339232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3806ba70 0x7fea3815d940 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fea38071d40 tx=0x7fea3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a0f1b1aa33ebe51 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.786+0000 7fea36ffd700 1 -- 172.21.15.1:0/846339232 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3000e070 con 0x7fea3806ba70 2024-03-18T19:09:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.786+0000 7fea36ffd700 1 -- 172.21.15.1:0/846339232 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3000b910 con 0x7fea3806ba70 2024-03-18T19:09:47.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.786+0000 7fea36ffd700 1 -- 172.21.15.1:0/846339232 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3000bcc0 con 0x7fea3806ba70 2024-03-18T19:09:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.787+0000 7fea3e504700 1 -- 172.21.15.1:0/846339232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3806ba70 msgr2=0x7fea3815d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.787+0000 7fea3e504700 1 --2- 172.21.15.1:0/846339232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3806ba70 0x7fea3815d940 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fea38071d40 tx=0x7fea3000b330 comp rx=0 tx=0).stop 2024-03-18T19:09:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.787+0000 7fea3e504700 1 -- 172.21.15.1:0/846339232 shutdown_connections 2024-03-18T19:09:47.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.787+0000 7fea3e504700 1 --2- 172.21.15.1:0/846339232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3806ba70 0x7fea3815d940 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.787+0000 7fea3e504700 1 --2- 172.21.15.1:0/846339232 >> v2:172.21.15.165:3300/0 conn(0x7fea380d05a0 0x7fea3809a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.787+0000 7fea3e504700 1 -- 172.21.15.1:0/846339232 >> 172.21.15.1:0/846339232 conn(0x7fea380b2cb0 msgr2=0x7fea380b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.787+0000 7fea3e504700 1 -- 172.21.15.1:0/846339232 shutdown_connections 2024-03-18T19:09:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.787+0000 7fea3e504700 1 -- 172.21.15.1:0/846339232 wait complete. 2024-03-18T19:09:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.788+0000 7fea3e504700 1 Processor -- start 2024-03-18T19:09:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.788+0000 7fea3e504700 1 -- start start 2024-03-18T19:09:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.789+0000 7fea3e504700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3806ba70 0x7fea3814d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.789+0000 7fea3e504700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea38099a40 0x7fea380b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.789+0000 7fea3e504700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea380d05a0 0x7fea380cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.789+0000 7fea3e504700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea380c9d80 con 0x7fea3806ba70 2024-03-18T19:09:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.789+0000 7fea3e504700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea380ca060 con 0x7fea380d05a0 2024-03-18T19:09:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.789+0000 7fea3e504700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea380ca340 con 0x7fea38099a40 2024-03-18T19:09:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.789+0000 7fea37fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3806ba70 0x7fea3814d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.789+0000 7fea37fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3806ba70 0x7fea3814d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43916/0 (socket says 172.21.15.1:43916) 2024-03-18T19:09:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.789+0000 7fea37fff700 1 -- 172.21.15.1:0/652041244 learned_addr learned my addr 172.21.15.1:0/652041244 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.789+0000 7fea3caa1700 1 --2- 172.21.15.1:0/652041244 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea380d05a0 0x7fea380cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.789+0000 7fea377fe700 1 --2- 172.21.15.1:0/652041244 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea38099a40 0x7fea380b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.790+0000 7fea37fff700 1 -- 172.21.15.1:0/652041244 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea38099a40 msgr2=0x7fea380b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.790+0000 7fea37fff700 1 --2- 172.21.15.1:0/652041244 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea38099a40 0x7fea380b3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.790+0000 7fea37fff700 1 -- 172.21.15.1:0/652041244 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea380d05a0 msgr2=0x7fea380cae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.790+0000 7fea37fff700 1 --2- 172.21.15.1:0/652041244 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea380d05a0 0x7fea380cae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.790+0000 7fea37fff700 1 -- 172.21.15.1:0/652041244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea30009b30 con 0x7fea3806ba70 2024-03-18T19:09:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.790+0000 7fea37fff700 1 --2- 172.21.15.1:0/652041244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3806ba70 0x7fea3814d7b0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fea2800c4e0 tx=0x7fea2800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.790+0000 7fea34ff9700 1 -- 172.21.15.1:0/652041244 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea280052d0 con 0x7fea3806ba70 2024-03-18T19:09:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.790+0000 7fea3e504700 1 -- 172.21.15.1:0/652041244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea380ca620 con 0x7fea3806ba70 2024-03-18T19:09:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.790+0000 7fea34ff9700 1 -- 172.21.15.1:0/652041244 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea28005430 con 0x7fea3806ba70 2024-03-18T19:09:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.790+0000 7fea34ff9700 1 -- 172.21.15.1:0/652041244 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea280038c0 con 0x7fea3806ba70 2024-03-18T19:09:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.790+0000 7fea3e504700 1 -- 172.21.15.1:0/652041244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea380b4300 con 0x7fea3806ba70 2024-03-18T19:09:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.791+0000 7fea3e504700 1 -- 172.21.15.1:0/652041244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea38068240 con 0x7fea3806ba70 2024-03-18T19:09:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.791+0000 7fea34ff9700 1 -- 172.21.15.1:0/652041244 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 78) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fea28003a20 con 0x7fea3806ba70 2024-03-18T19:09:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.792+0000 7fea34ff9700 1 --2- 172.21.15.1:0/652041244 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fea2406ae40 0x7fea2406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.792+0000 7fea377fe700 1 -- 172.21.15.1:0/652041244 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fea2406ae40 msgr2=0x7fea2406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.792+0000 7fea34ff9700 1 -- 172.21.15.1:0/652041244 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fea28088cf0 con 0x7fea3806ba70 2024-03-18T19:09:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.792+0000 7fea377fe700 1 --2- 172.21.15.1:0/652041244 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fea2406ae40 0x7fea2406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:47.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.796+0000 7fea34ff9700 1 -- 172.21.15.1:0/652041244 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fea2808d070 con 0x7fea3806ba70 2024-03-18T19:09:47.818 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:47.821+0000 7f441fadd240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:09:47.885 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:47.889+0000 7ffbca1f9240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:09:47.896 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:47.900+0000 7f441fadd240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:09:47.961 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:47.966+0000 7ffbca1f9240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:09:47.973 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:47.977+0000 7f441fadd240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:09:47.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.993+0000 7fea377fe700 1 -- 172.21.15.1:0/652041244 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fea2406ae40 msgr2=0x7fea2406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:47.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:47.993+0000 7fea377fe700 1 --2- 172.21.15.1:0/652041244 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fea2406ae40 0x7fea2406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:48.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.045+0000 7fea3e504700 1 -- 172.21.15.1:0/652041244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea3809aae0 con 0x7fea3806ba70 2024-03-18T19:09:48.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.045+0000 7fea34ff9700 1 -- 172.21.15.1:0/652041244 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+4727 (secure 0 0 0) 0x7fea28053910 con 0x7fea3806ba70 2024-03-18T19:09:48.041 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:48.041 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8116,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3723189348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3723189348},{"type":"v1","addr":"172.21.15.1:6837","nonce":3723189348}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":8125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/530206363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":530206363},{"type":"v1","addr":"172.21.15.165:6835","nonce":530206363}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":8784,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/332827595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":332827595},{"type":"v1","addr":"172.21.15.1:6836","nonce":332827595}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:09:22.623415+0000","modified":"2024-03-18T19:09:25.640243+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6834/311436411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":311436411},{"type":"v1","addr":"172.21.15.165:6834","nonce":311436411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-18T19:09:48.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.047+0000 7fea3e504700 1 -- 172.21.15.1:0/652041244 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fea2406ae40 msgr2=0x7fea2406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:48.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.047+0000 7fea3e504700 1 --2- 172.21.15.1:0/652041244 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fea2406ae40 0x7fea2406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.047+0000 7fea3e504700 1 -- 172.21.15.1:0/652041244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3806ba70 msgr2=0x7fea3814d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:48.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.047+0000 7fea3e504700 1 --2- 172.21.15.1:0/652041244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3806ba70 0x7fea3814d7b0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fea2800c4e0 tx=0x7fea2800c7f0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.047+0000 7fea3e504700 1 -- 172.21.15.1:0/652041244 shutdown_connections 2024-03-18T19:09:48.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.047+0000 7fea3e504700 1 --2- 172.21.15.1:0/652041244 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fea2406ae40 0x7fea2406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.047+0000 7fea3e504700 1 --2- 172.21.15.1:0/652041244 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea380d05a0 0x7fea380cae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.047+0000 7fea3e504700 1 --2- 172.21.15.1:0/652041244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea3806ba70 0x7fea3814d7b0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.048+0000 7fea3e504700 1 --2- 172.21.15.1:0/652041244 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea38099a40 0x7fea380b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.048+0000 7fea3e504700 1 -- 172.21.15.1:0/652041244 >> 172.21.15.1:0/652041244 conn(0x7fea380b2cb0 msgr2=0x7fea380b3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:48.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.048+0000 7fea3e504700 1 -- 172.21.15.1:0/652041244 shutdown_connections 2024-03-18T19:09:48.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.048+0000 7fea3e504700 1 -- 172.21.15.1:0/652041244 wait complete. 2024-03-18T19:09:48.045 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 65 2024-03-18T19:09:48.045 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:48.049+0000 7ffbca1f9240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:09:48.056 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:09:48.130 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:48.134+0000 7ffbca1f9240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:09:48.142 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:48.146+0000 7f441fadd240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:09:48.215 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:48.219+0000 7ffbca1f9240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:09:48.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.228+0000 7fb44db8f700 1 Processor -- start 2024-03-18T19:09:48.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.229+0000 7fb44db8f700 1 -- start start 2024-03-18T19:09:48.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.229+0000 7fb44db8f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb44809a650 0x7fb448099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:48.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.229+0000 7fb44db8f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44806f070 con 0x7fb448099f80 2024-03-18T19:09:48.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.229+0000 7fb44db8f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4480d0260 con 0x7fb44815b4d0 2024-03-18T19:09:48.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.229+0000 7fb44db8f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb44809a410 con 0x7fb44809a650 2024-03-18T19:09:48.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.229+0000 7fb447fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb448099f80 0x7fb44806f500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59518/0 (socket says 172.21.15.1:59518) 2024-03-18T19:09:48.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.229+0000 7fb447fff700 1 -- 172.21.15.1:0/1650279669 learned_addr learned my addr 172.21.15.1:0/1650279669 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:48.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.230+0000 7fb446ffd700 1 --2- 172.21.15.1:0/1650279669 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb44809a650 0x7fb448099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:48.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.230+0000 7fb4467fc700 1 -- 172.21.15.1:0/1650279669 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758013056 0 0) 0x7fb44806f070 con 0x7fb448099f80 2024-03-18T19:09:48.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.230+0000 7fb4467fc700 1 -- 172.21.15.1:0/1650279669 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4340035b0 con 0x7fb448099f80 2024-03-18T19:09:48.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.230+0000 7fb4467fc700 1 -- 172.21.15.1:0/1650279669 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222602250 0 0) 0x7fb4480d0260 con 0x7fb44815b4d0 2024-03-18T19:09:48.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.230+0000 7fb4467fc700 1 -- 172.21.15.1:0/1650279669 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb44806f070 con 0x7fb44815b4d0 2024-03-18T19:09:48.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.230+0000 7fb446ffd700 1 -- 172.21.15.1:0/1650279669 >> v1:172.21.15.165:6789/0 conn(0x7fb44815b4d0 legacy=0x7fb4480cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:48.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.230+0000 7fb446ffd700 1 -- 172.21.15.1:0/1650279669 >> v1:172.21.15.1:6789/0 conn(0x7fb448099f80 legacy=0x7fb44806f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:48.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.230+0000 7fb446ffd700 1 -- 172.21.15.1:0/1650279669 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4480cff80 con 0x7fb44809a650 2024-03-18T19:09:48.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.230+0000 7fb446ffd700 1 --2- 172.21.15.1:0/1650279669 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb44809a650 0x7fb448099a40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb430009870 tx=0x7fb430009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7624cee57c850045 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:48.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.230+0000 7fb4467fc700 1 -- 172.21.15.1:0/1650279669 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb430011830 con 0x7fb44809a650 2024-03-18T19:09:48.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.230+0000 7fb4467fc700 1 -- 172.21.15.1:0/1650279669 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb430011990 con 0x7fb44809a650 2024-03-18T19:09:48.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.230+0000 7fb4467fc700 1 -- 172.21.15.1:0/1650279669 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb430011d20 con 0x7fb44809a650 2024-03-18T19:09:48.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.231+0000 7fb44db8f700 1 -- 172.21.15.1:0/1650279669 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb44809a650 msgr2=0x7fb448099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:48.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.231+0000 7fb44db8f700 1 --2- 172.21.15.1:0/1650279669 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb44809a650 0x7fb448099a40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb430009870 tx=0x7fb430009b80 comp rx=0 tx=0).stop 2024-03-18T19:09:48.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.231+0000 7fb44db8f700 1 -- 172.21.15.1:0/1650279669 shutdown_connections 2024-03-18T19:09:48.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.231+0000 7fb44db8f700 1 --2- 172.21.15.1:0/1650279669 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb44809a650 0x7fb448099a40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.231+0000 7fb44db8f700 1 -- 172.21.15.1:0/1650279669 >> 172.21.15.1:0/1650279669 conn(0x7fb4480b2cb0 msgr2=0x7fb4480b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:48.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.231+0000 7fb44db8f700 1 -- 172.21.15.1:0/1650279669 shutdown_connections 2024-03-18T19:09:48.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.231+0000 7fb44db8f700 1 -- 172.21.15.1:0/1650279669 wait complete. 2024-03-18T19:09:48.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.232+0000 7fb44db8f700 1 Processor -- start 2024-03-18T19:09:48.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.233+0000 7fb44db8f700 1 -- start start 2024-03-18T19:09:48.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.233+0000 7fb44db8f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb448099f80 0x7fb4480b6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:48.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.233+0000 7fb44db8f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb44809a650 0x7fb4480ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:48.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.233+0000 7fb44db8f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb44815b4d0 0x7fb44806cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:48.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.233+0000 7fb44db8f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480ca2c0 con 0x7fb448099f80 2024-03-18T19:09:48.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.233+0000 7fb44db8f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480b6aa0 con 0x7fb44815b4d0 2024-03-18T19:09:48.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.233+0000 7fb44db8f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480c9410 con 0x7fb44809a650 2024-03-18T19:09:48.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.233+0000 7fb4477fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb448099f80 0x7fb4480b6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:48.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.233+0000 7fb4477fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb448099f80 0x7fb4480b6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43930/0 (socket says 172.21.15.1:43930) 2024-03-18T19:09:48.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.233+0000 7fb4477fe700 1 -- 172.21.15.1:0/3591833822 learned_addr learned my addr 172.21.15.1:0/3591833822 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.234+0000 7fb447fff700 1 --2- 172.21.15.1:0/3591833822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb44815b4d0 0x7fb44806cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.234+0000 7fb446ffd700 1 --2- 172.21.15.1:0/3591833822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb44809a650 0x7fb4480ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.234+0000 7fb4477fe700 1 -- 172.21.15.1:0/3591833822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb44809a650 msgr2=0x7fb4480ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.234+0000 7fb4477fe700 1 --2- 172.21.15.1:0/3591833822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb44809a650 0x7fb4480ca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.234+0000 7fb4477fe700 1 -- 172.21.15.1:0/3591833822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb44815b4d0 msgr2=0x7fb44806cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.234+0000 7fb4477fe700 1 --2- 172.21.15.1:0/3591833822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb44815b4d0 0x7fb44806cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.234+0000 7fb4477fe700 1 -- 172.21.15.1:0/3591833822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4300094a0 con 0x7fb448099f80 2024-03-18T19:09:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.234+0000 7fb447fff700 1 --2- 172.21.15.1:0/3591833822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb44815b4d0 0x7fb44806cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.234+0000 7fb4477fe700 1 --2- 172.21.15.1:0/3591833822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb448099f80 0x7fb4480b6560 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fb43800c500 tx=0x7fb43800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.234+0000 7fb43ffff700 1 -- 172.21.15.1:0/3591833822 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4380052d0 con 0x7fb448099f80 2024-03-18T19:09:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.234+0000 7fb43ffff700 1 -- 172.21.15.1:0/3591833822 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb438005450 con 0x7fb448099f80 2024-03-18T19:09:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.234+0000 7fb44db8f700 1 -- 172.21.15.1:0/3591833822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4480c96b0 con 0x7fb448099f80 2024-03-18T19:09:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.234+0000 7fb43ffff700 1 -- 172.21.15.1:0/3591833822 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4380038c0 con 0x7fb448099f80 2024-03-18T19:09:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.234+0000 7fb44db8f700 1 -- 172.21.15.1:0/3591833822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4480c9c70 con 0x7fb448099f80 2024-03-18T19:09:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.235+0000 7fb44db8f700 1 -- 172.21.15.1:0/3591833822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb448068240 con 0x7fb448099f80 2024-03-18T19:09:48.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.236+0000 7fb43ffff700 1 -- 172.21.15.1:0/3591833822 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 78) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fb438003a20 con 0x7fb448099f80 2024-03-18T19:09:48.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.236+0000 7fb43ffff700 1 --2- 172.21.15.1:0/3591833822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb42806ae40 0x7fb42806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:48.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.236+0000 7fb446ffd700 1 -- 172.21.15.1:0/3591833822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb42806ae40 msgr2=0x7fb42806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:48.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.237+0000 7fb43ffff700 1 -- 172.21.15.1:0/3591833822 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fb438088f60 con 0x7fb448099f80 2024-03-18T19:09:48.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.237+0000 7fb446ffd700 1 --2- 172.21.15.1:0/3591833822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb42806ae40 0x7fb42806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:48.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.240+0000 7fb43ffff700 1 -- 172.21.15.1:0/3591833822 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb438053d70 con 0x7fb448099f80 2024-03-18T19:09:48.357 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:48.361+0000 7f441fadd240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:09:48.398 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:48.402+0000 7ffbca1f9240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:09:48.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.438+0000 7fb446ffd700 1 -- 172.21.15.1:0/3591833822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb42806ae40 msgr2=0x7fb42806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:48.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.438+0000 7fb446ffd700 1 --2- 172.21.15.1:0/3591833822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb42806ae40 0x7fb42806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:48.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.444+0000 7fb44db8f700 1 -- 172.21.15.1:0/3591833822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4480ca5a0 con 0x7fb448099f80 2024-03-18T19:09:48.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.445+0000 7fb43ffff700 1 -- 172.21.15.1:0/3591833822 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+16790 (secure 0 0 0) 0x7fb438057390 con 0x7fb448099f80 2024-03-18T19:09:48.441 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:48.441 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":148,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:09:37.850698+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:09:20.468744+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-18T19:09:21.597476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/69006586":"2024-03-19T19:09:37.850642+0000","172.21.15.1:6833/76998":"2024-03-19T19:09:37.850642+0000","172.21.15.1:6832/76998":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/1403689662":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/806972823":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/619565916":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/2042700873":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/744112928":"2024-03-19T19:09:37.850642+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:09:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.447+0000 7fb44db8f700 1 -- 172.21.15.1:0/3591833822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb42806ae40 msgr2=0x7fb42806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.447+0000 7fb44db8f700 1 --2- 172.21.15.1:0/3591833822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb42806ae40 0x7fb42806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.447+0000 7fb44db8f700 1 -- 172.21.15.1:0/3591833822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb448099f80 msgr2=0x7fb4480b6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.447+0000 7fb44db8f700 1 --2- 172.21.15.1:0/3591833822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb448099f80 0x7fb4480b6560 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fb43800c500 tx=0x7fb43800c810 comp rx=0 tx=0).stop 2024-03-18T19:09:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.447+0000 7fb44db8f700 1 -- 172.21.15.1:0/3591833822 shutdown_connections 2024-03-18T19:09:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.447+0000 7fb44db8f700 1 --2- 172.21.15.1:0/3591833822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb42806ae40 0x7fb42806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.447+0000 7fb44db8f700 1 --2- 172.21.15.1:0/3591833822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb44815b4d0 0x7fb44806cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.447+0000 7fb44db8f700 1 --2- 172.21.15.1:0/3591833822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb448099f80 0x7fb4480b6560 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.447+0000 7fb44db8f700 1 --2- 172.21.15.1:0/3591833822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb44809a650 0x7fb4480ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.447+0000 7fb44db8f700 1 -- 172.21.15.1:0/3591833822 >> 172.21.15.1:0/3591833822 conn(0x7fb4480b2cb0 msgr2=0x7fb4480afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:48.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.447+0000 7fb44db8f700 1 -- 172.21.15.1:0/3591833822 shutdown_connections 2024-03-18T19:09:48.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.447+0000 7fb44db8f700 1 -- 172.21.15.1:0/3591833822 wait complete. 2024-03-18T19:09:48.455 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T19:09:20.468744+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T19:09:21.597476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '147', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:09:48.456 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:09:48.507 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:48.511+0000 7f441fadd240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:09:48.579 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:48.583+0000 7f441fadd240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:09:48.628 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:48.632+0000 7ffbca1f9240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:09:48.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.632+0000 7f565ccda700 1 Processor -- start 2024-03-18T19:09:48.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.632+0000 7f565ccda700 1 -- start start 2024-03-18T19:09:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.633+0000 7f565ccda700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f565806ae60 0x7f5658168150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.634+0000 7f565ccda700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56580cf600 con 0x7f56580cfa90 2024-03-18T19:09:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.634+0000 7f565ccda700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56580ce720 con 0x7f565815d5a0 2024-03-18T19:09:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.634+0000 7f565ccda700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56580ce9f0 con 0x7f565806ae60 2024-03-18T19:09:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.634+0000 7f5655d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f56580cfa90 0x7f565806a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59522/0 (socket says 172.21.15.1:59522) 2024-03-18T19:09:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.634+0000 7f5655d9b700 1 -- 172.21.15.1:0/2949355764 learned_addr learned my addr 172.21.15.1:0/2949355764 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:48.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.634+0000 7f5656d9d700 1 --2- 172.21.15.1:0/2949355764 >> v2:172.21.15.165:3300/0 conn(0x7f565806ae60 0x7f5658168150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:48.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.634+0000 7f565559a700 1 -- 172.21.15.1:0/2949355764 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800838670 0 0) 0x7f56580cf600 con 0x7f56580cfa90 2024-03-18T19:09:48.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.634+0000 7f565559a700 1 -- 172.21.15.1:0/2949355764 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56400035b0 con 0x7f56580cfa90 2024-03-18T19:09:48.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.634+0000 7f565559a700 1 -- 172.21.15.1:0/2949355764 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569926906 0 0) 0x7f56580ce720 con 0x7f565815d5a0 2024-03-18T19:09:48.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.634+0000 7f565559a700 1 -- 172.21.15.1:0/2949355764 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56580cf600 con 0x7f565815d5a0 2024-03-18T19:09:48.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.634+0000 7f5656d9d700 1 -- 172.21.15.1:0/2949355764 >> v1:172.21.15.165:6789/0 conn(0x7f565815d5a0 legacy=0x7f56580ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:48.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.634+0000 7f5656d9d700 1 -- 172.21.15.1:0/2949355764 >> v1:172.21.15.1:6789/0 conn(0x7f56580cfa90 legacy=0x7f565806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:48.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.634+0000 7f5656d9d700 1 -- 172.21.15.1:0/2949355764 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56580cecd0 con 0x7f565806ae60 2024-03-18T19:09:48.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.634+0000 7f5656d9d700 1 --2- 172.21.15.1:0/2949355764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f565806ae60 0x7f5658168150 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f5650009d90 tx=0x7f565000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84e9691e5c45108c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:48.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.635+0000 7f565559a700 1 -- 172.21.15.1:0/2949355764 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565000e070 con 0x7f565806ae60 2024-03-18T19:09:48.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.635+0000 7f565559a700 1 -- 172.21.15.1:0/2949355764 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565000bb20 con 0x7f565806ae60 2024-03-18T19:09:48.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.635+0000 7f565559a700 1 -- 172.21.15.1:0/2949355764 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5650012430 con 0x7f565806ae60 2024-03-18T19:09:48.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.635+0000 7f565ccda700 1 -- 172.21.15.1:0/2949355764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f565806ae60 msgr2=0x7f5658168150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:48.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.635+0000 7f565ccda700 1 --2- 172.21.15.1:0/2949355764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f565806ae60 0x7f5658168150 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f5650009d90 tx=0x7f565000b330 comp rx=0 tx=0).stop 2024-03-18T19:09:48.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.635+0000 7f565ccda700 1 -- 172.21.15.1:0/2949355764 shutdown_connections 2024-03-18T19:09:48.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.635+0000 7f565ccda700 1 --2- 172.21.15.1:0/2949355764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f565806ae60 0x7f5658168150 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.635+0000 7f565ccda700 1 -- 172.21.15.1:0/2949355764 >> 172.21.15.1:0/2949355764 conn(0x7f56580cc9b0 msgr2=0x7f56580ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:48.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.635+0000 7f565ccda700 1 -- 172.21.15.1:0/2949355764 shutdown_connections 2024-03-18T19:09:48.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.635+0000 7f565ccda700 1 -- 172.21.15.1:0/2949355764 wait complete. 2024-03-18T19:09:48.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.637+0000 7f565ccda700 1 Processor -- start 2024-03-18T19:09:48.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.638+0000 7f565ccda700 1 -- start start 2024-03-18T19:09:48.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.639+0000 7f565ccda700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f565806ae60 0x7f56581f6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:48.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.639+0000 7f565ccda700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56580cfa90 0x7f5658147f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:48.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.639+0000 7f565ccda700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f565815d5a0 0x7f56581f2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.639+0000 7f565ccda700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56580c7330 con 0x7f565806ae60 2024-03-18T19:09:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.639+0000 7f565ccda700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56581f7280 con 0x7f56580cfa90 2024-03-18T19:09:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.639+0000 7f565ccda700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565814a570 con 0x7f565815d5a0 2024-03-18T19:09:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.639+0000 7f565659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f565806ae60 0x7f56581f6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.639+0000 7f565659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f565806ae60 0x7f56581f6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43934/0 (socket says 172.21.15.1:43934) 2024-03-18T19:09:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.639+0000 7f565659c700 1 -- 172.21.15.1:0/4152451740 learned_addr learned my addr 172.21.15.1:0/4152451740 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.639+0000 7f5655d9b700 1 --2- 172.21.15.1:0/4152451740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56580cfa90 0x7f5658147f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.639+0000 7f5656d9d700 1 --2- 172.21.15.1:0/4152451740 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f565815d5a0 0x7f56581f2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.639+0000 7f565659c700 1 -- 172.21.15.1:0/4152451740 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f565815d5a0 msgr2=0x7f56581f2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.639+0000 7f565659c700 1 --2- 172.21.15.1:0/4152451740 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f565815d5a0 0x7f56581f2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.639+0000 7f565659c700 1 -- 172.21.15.1:0/4152451740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56580cfa90 msgr2=0x7f5658147f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.639+0000 7f565659c700 1 --2- 172.21.15.1:0/4152451740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56580cfa90 0x7f5658147f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.639+0000 7f565659c700 1 -- 172.21.15.1:0/4152451740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5650009a70 con 0x7f565806ae60 2024-03-18T19:09:48.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.639+0000 7f5655d9b700 1 --2- 172.21.15.1:0/4152451740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56580cfa90 0x7f5658147f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:48.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.639+0000 7f565659c700 1 --2- 172.21.15.1:0/4152451740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f565806ae60 0x7f56581f6d40 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f5648001f00 tx=0x7f564800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:48.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.640+0000 7f5646ffd700 1 -- 172.21.15.1:0/4152451740 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56480052d0 con 0x7f565806ae60 2024-03-18T19:09:48.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.640+0000 7f5646ffd700 1 -- 172.21.15.1:0/4152451740 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5648005430 con 0x7f565806ae60 2024-03-18T19:09:48.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.640+0000 7f565ccda700 1 -- 172.21.15.1:0/4152451740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565814a850 con 0x7f565806ae60 2024-03-18T19:09:48.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.640+0000 7f5646ffd700 1 -- 172.21.15.1:0/4152451740 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56480038f0 con 0x7f565806ae60 2024-03-18T19:09:48.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.640+0000 7f565ccda700 1 -- 172.21.15.1:0/4152451740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565814ae10 con 0x7f565806ae60 2024-03-18T19:09:48.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.641+0000 7f565ccda700 1 -- 172.21.15.1:0/4152451740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56580676a0 con 0x7f565806ae60 2024-03-18T19:09:48.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.641+0000 7f5646ffd700 1 -- 172.21.15.1:0/4152451740 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 78) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f5648003a50 con 0x7f565806ae60 2024-03-18T19:09:48.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.641+0000 7f5646ffd700 1 --2- 172.21.15.1:0/4152451740 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f563806ae40 0x7f563806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:48.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.641+0000 7f5655d9b700 1 -- 172.21.15.1:0/4152451740 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f563806ae40 msgr2=0x7f563806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:48.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.642+0000 7f5646ffd700 1 -- 172.21.15.1:0/4152451740 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f5648088ca0 con 0x7f565806ae60 2024-03-18T19:09:48.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.642+0000 7f5655d9b700 1 --2- 172.21.15.1:0/4152451740 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f563806ae40 0x7f563806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:48.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.646+0000 7f5646ffd700 1 -- 172.21.15.1:0/4152451740 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f56480539d0 con 0x7f565806ae60 2024-03-18T19:09:48.792 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:48.797+0000 7ffbca1f9240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:09:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.843+0000 7f5655d9b700 1 -- 172.21.15.1:0/4152451740 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f563806ae40 msgr2=0x7f563806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.843+0000 7f5655d9b700 1 --2- 172.21.15.1:0/4152451740 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f563806ae40 0x7f563806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:48.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.850+0000 7f565ccda700 1 -- 172.21.15.1:0/4152451740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f56581484a0 con 0x7f565806ae60 2024-03-18T19:09:48.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.850+0000 7f5646ffd700 1 -- 172.21.15.1:0/4152451740 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v148) v1 ==== 93+0+10 (secure 0 0 0) 0x7f56480170a0 con 0x7f565806ae60 2024-03-18T19:09:48.846 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:09:48.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.853+0000 7f565ccda700 1 -- 172.21.15.1:0/4152451740 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f563806ae40 msgr2=0x7f563806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.853+0000 7f565ccda700 1 --2- 172.21.15.1:0/4152451740 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f563806ae40 0x7f563806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.853+0000 7f565ccda700 1 -- 172.21.15.1:0/4152451740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f565806ae60 msgr2=0x7f56581f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.853+0000 7f565ccda700 1 --2- 172.21.15.1:0/4152451740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f565806ae60 0x7f56581f6d40 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f5648001f00 tx=0x7f564800c7b0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.853+0000 7f565ccda700 1 -- 172.21.15.1:0/4152451740 shutdown_connections 2024-03-18T19:09:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.853+0000 7f565ccda700 1 --2- 172.21.15.1:0/4152451740 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f563806ae40 0x7f563806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.853+0000 7f565ccda700 1 --2- 172.21.15.1:0/4152451740 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f565815d5a0 0x7f56581f2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.853+0000 7f565ccda700 1 --2- 172.21.15.1:0/4152451740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f565806ae60 0x7f56581f6d40 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.853+0000 7f565ccda700 1 --2- 172.21.15.1:0/4152451740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56580cfa90 0x7f5658147f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.853+0000 7f565ccda700 1 -- 172.21.15.1:0/4152451740 >> 172.21.15.1:0/4152451740 conn(0x7f56580cc9b0 msgr2=0x7f56580b1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.853+0000 7f565ccda700 1 -- 172.21.15.1:0/4152451740 shutdown_connections 2024-03-18T19:09:48.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.853+0000 7f565ccda700 1 -- 172.21.15.1:0/4152451740 wait complete. 2024-03-18T19:09:48.861 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T19:09:48.872 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:48.876+0000 7ffbca1f9240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:09:48.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.995+0000 7f4c6f204700 1 Processor -- start 2024-03-18T19:09:48.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.995+0000 7f4c6f204700 1 -- start start 2024-03-18T19:09:48.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.996+0000 7f4c6f204700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c680cd870 0x7f4c680cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:48.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.996+0000 7f4c6f204700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4c680cbce0 0x7f4c680cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:48.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.996+0000 7f4c6f204700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c680d0df0 con 0x7f4c680b1e20 2024-03-18T19:09:48.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.996+0000 7f4c6f204700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c680d0550 con 0x7f4c680cbce0 2024-03-18T19:09:48.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.996+0000 7f4c6f204700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c680d0830 con 0x7f4c680cd870 2024-03-18T19:09:48.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.996+0000 7f4c6cfa0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c680cd870 0x7f4c680cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:48.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.996+0000 7f4c6d7a1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4c680cbce0 0x7f4c680cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:48.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.996+0000 7f4c6d7a1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4c680cbce0 0x7f4c680cc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:57214/0 (socket says 172.21.15.1:57214) 2024-03-18T19:09:48.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.996+0000 7f4c67fff700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f4c680b1e20 0x7f4c680cdc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:59660/0 (socket says 172.21.15.1:59660) 2024-03-18T19:09:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.996+0000 7f4c6d7a1700 1 -- 172.21.15.1:0/1401762222 learned_addr learned my addr 172.21.15.1:0/1401762222 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.996+0000 7f4c6cfa0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c680cd870 0x7f4c680cff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:48970/0 (socket says 172.21.15.1:48970) 2024-03-18T19:09:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.997+0000 7f4c6cfa0700 1 -- 172.21.15.1:0/1401762222 >> v2:172.21.15.165:3300/0 conn(0x7f4c680cbce0 msgr2=0x7f4c680cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.997+0000 7f4c6cfa0700 1 --2- 172.21.15.1:0/1401762222 >> v2:172.21.15.165:3300/0 conn(0x7f4c680cbce0 0x7f4c680cc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.997+0000 7f4c6cfa0700 1 -- 172.21.15.1:0/1401762222 >> v1:172.21.15.165:6789/0 conn(0x7f4c680b1e20 legacy=0x7f4c680cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.997+0000 7f4c6cfa0700 1 -- 172.21.15.1:0/1401762222 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c680d0b10 con 0x7f4c680cd870 2024-03-18T19:09:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.997+0000 7f4c677fe700 1 -- 172.21.15.1:0/1401762222 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665834856 0 0) 0x7f4c680d0df0 con 0x7f4c680b1e20 2024-03-18T19:09:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.997+0000 7f4c6d7a1700 1 --2- 172.21.15.1:0/1401762222 >> v2:172.21.15.165:3300/0 conn(0x7f4c680cbce0 0x7f4c680cc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.997+0000 7f4c6cfa0700 1 --2- 172.21.15.1:0/1401762222 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c680cd870 0x7f4c680cff70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c009840 tx=0x7f4c5c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a33cf2d35ff9f30 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.997+0000 7f4c677fe700 1 -- 172.21.15.1:0/1401762222 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c011830 con 0x7f4c680cd870 2024-03-18T19:09:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.997+0000 7f4c677fe700 1 -- 172.21.15.1:0/1401762222 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5c011990 con 0x7f4c680cd870 2024-03-18T19:09:48.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.997+0000 7f4c677fe700 1 -- 172.21.15.1:0/1401762222 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c011d20 con 0x7f4c680cd870 2024-03-18T19:09:48.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.998+0000 7f4c6f204700 1 -- 172.21.15.1:0/1401762222 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c680cd870 msgr2=0x7f4c680cff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:48.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.998+0000 7f4c6f204700 1 --2- 172.21.15.1:0/1401762222 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c680cd870 0x7f4c680cff70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c009840 tx=0x7f4c5c009b50 comp rx=0 tx=0).stop 2024-03-18T19:09:48.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.998+0000 7f4c6f204700 1 -- 172.21.15.1:0/1401762222 shutdown_connections 2024-03-18T19:09:48.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.998+0000 7f4c6f204700 1 --2- 172.21.15.1:0/1401762222 >> v2:172.21.15.165:3300/0 conn(0x7f4c680cbce0 0x7f4c680cc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.998+0000 7f4c6f204700 1 --2- 172.21.15.1:0/1401762222 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c680cd870 0x7f4c680cff70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.998+0000 7f4c6f204700 1 -- 172.21.15.1:0/1401762222 >> 172.21.15.1:0/1401762222 conn(0x7f4c680d0ff0 msgr2=0x7f4c680d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:48.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.998+0000 7f4c6f204700 1 -- 172.21.15.1:0/1401762222 shutdown_connections 2024-03-18T19:09:48.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.998+0000 7f4c6f204700 1 -- 172.21.15.1:0/1401762222 wait complete. 2024-03-18T19:09:48.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.999+0000 7f4c6f204700 1 Processor -- start 2024-03-18T19:09:48.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:48.999+0000 7f4c6f204700 1 -- start start 2024-03-18T19:09:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.000+0000 7f4c6f204700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c680b1e20 0x7f4c68158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.000+0000 7f4c6f204700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c680cbce0 0x7f4c681f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.000+0000 7f4c6f204700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c680cd870 0x7f4c681f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.000+0000 7f4c6f204700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c68154d70 con 0x7f4c680cbce0 2024-03-18T19:09:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.000+0000 7f4c6f204700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c68158e60 con 0x7f4c680b1e20 2024-03-18T19:09:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.000+0000 7f4c6f204700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c68159140 con 0x7f4c680cd870 2024-03-18T19:09:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.001+0000 7f4c67fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c680cbce0 0x7f4c681f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.001+0000 7f4c67fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c680cbce0 0x7f4c681f3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43938/0 (socket says 172.21.15.1:43938) 2024-03-18T19:09:48.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.001+0000 7f4c67fff700 1 -- 172.21.15.1:0/275564518 learned_addr learned my addr 172.21.15.1:0/275564518 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.001+0000 7f4c6cfa0700 1 --2- 172.21.15.1:0/275564518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c680b1e20 0x7f4c68158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.001+0000 7f4c6d7a1700 1 --2- 172.21.15.1:0/275564518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c680cd870 0x7f4c681f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.001+0000 7f4c67fff700 1 -- 172.21.15.1:0/275564518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c680cd870 msgr2=0x7f4c681f35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.001+0000 7f4c67fff700 1 --2- 172.21.15.1:0/275564518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c680cd870 0x7f4c681f35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.001+0000 7f4c67fff700 1 -- 172.21.15.1:0/275564518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c680b1e20 msgr2=0x7f4c68158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.001+0000 7f4c67fff700 1 --2- 172.21.15.1:0/275564518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c680b1e20 0x7f4c68158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.001+0000 7f4c67fff700 1 -- 172.21.15.1:0/275564518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c60009b30 con 0x7f4c680cbce0 2024-03-18T19:09:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.001+0000 7f4c67fff700 1 --2- 172.21.15.1:0/275564518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c680cbce0 0x7f4c681f3080 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f4c540011e0 tx=0x7f4c5400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.001+0000 7f4c657fa700 1 -- 172.21.15.1:0/275564518 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54004c80 con 0x7f4c680cbce0 2024-03-18T19:09:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.001+0000 7f4c657fa700 1 -- 172.21.15.1:0/275564518 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c54004e00 con 0x7f4c680cbce0 2024-03-18T19:09:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.001+0000 7f4c6f204700 1 -- 172.21.15.1:0/275564518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5c0094a0 con 0x7f4c680cbce0 2024-03-18T19:09:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.001+0000 7f4c657fa700 1 -- 172.21.15.1:0/275564518 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54007310 con 0x7f4c680cbce0 2024-03-18T19:09:48.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.001+0000 7f4c6f204700 1 -- 172.21.15.1:0/275564518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c68159700 con 0x7f4c680cbce0 2024-03-18T19:09:49.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.003+0000 7f4c6f204700 1 -- 172.21.15.1:0/275564518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c680b2790 con 0x7f4c680cbce0 2024-03-18T19:09:49.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.003+0000 7f4c657fa700 1 -- 172.21.15.1:0/275564518 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 78) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f4c54007470 con 0x7f4c680cbce0 2024-03-18T19:09:49.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.003+0000 7f4c657fa700 1 --2- 172.21.15.1:0/275564518 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4c4c06ae40 0x7f4c4c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:49.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.003+0000 7f4c6cfa0700 1 -- 172.21.15.1:0/275564518 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4c4c06ae40 msgr2=0x7f4c4c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:49.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.003+0000 7f4c657fa700 1 -- 172.21.15.1:0/275564518 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f4c54089540 con 0x7f4c680cbce0 2024-03-18T19:09:49.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.003+0000 7f4c6cfa0700 1 --2- 172.21.15.1:0/275564518 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4c4c06ae40 0x7f4c4c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:49.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.008+0000 7f4c657fa700 1 -- 172.21.15.1:0/275564518 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4c540541e0 con 0x7f4c680cbce0 2024-03-18T19:09:49.022 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:49.026+0000 7f441fadd240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:09:49.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.204+0000 7f4c6cfa0700 1 -- 172.21.15.1:0/275564518 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4c4c06ae40 msgr2=0x7f4c4c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:49.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.204+0000 7f4c6cfa0700 1 --2- 172.21.15.1:0/275564518 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4c4c06ae40 0x7f4c4c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:49.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.244+0000 7f4c6f204700 1 -- 172.21.15.1:0/275564518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4c680676a0 con 0x7f4c680cbce0 2024-03-18T19:09:49.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.244+0000 7f4c657fa700 1 -- 172.21.15.1:0/275564518 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v148) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4c54057800 con 0x7f4c680cbce0 2024-03-18T19:09:49.240 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T19:09:49.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.246+0000 7f4c6f204700 1 -- 172.21.15.1:0/275564518 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4c4c06ae40 msgr2=0x7f4c4c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:49.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.247+0000 7f4c6f204700 1 --2- 172.21.15.1:0/275564518 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4c4c06ae40 0x7f4c4c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.247+0000 7f4c6f204700 1 -- 172.21.15.1:0/275564518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c680cbce0 msgr2=0x7f4c681f3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:49.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.247+0000 7f4c6f204700 1 --2- 172.21.15.1:0/275564518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c680cbce0 0x7f4c681f3080 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f4c540011e0 tx=0x7f4c5400ca30 comp rx=0 tx=0).stop 2024-03-18T19:09:49.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.247+0000 7f4c6f204700 1 -- 172.21.15.1:0/275564518 shutdown_connections 2024-03-18T19:09:49.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.247+0000 7f4c6f204700 1 --2- 172.21.15.1:0/275564518 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4c4c06ae40 0x7f4c4c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.247+0000 7f4c6f204700 1 --2- 172.21.15.1:0/275564518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c680cd870 0x7f4c681f35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.247+0000 7f4c6f204700 1 --2- 172.21.15.1:0/275564518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c680b1e20 0x7f4c68158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.247+0000 7f4c6f204700 1 --2- 172.21.15.1:0/275564518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c680cbce0 0x7f4c681f3080 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.247+0000 7f4c6f204700 1 -- 172.21.15.1:0/275564518 >> 172.21.15.1:0/275564518 conn(0x7f4c680d0ff0 msgr2=0x7f4c6815cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:49.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.247+0000 7f4c6f204700 1 -- 172.21.15.1:0/275564518 shutdown_connections 2024-03-18T19:09:49.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.247+0000 7f4c6f204700 1 -- 172.21.15.1:0/275564518 wait complete. 2024-03-18T19:09:49.255 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T19:09:49.358 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:49.362+0000 7f441fadd240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:09:49.393 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:49.397+0000 7ffbca1f9240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:09:49.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.429+0000 7ff22a74c700 1 Processor -- start 2024-03-18T19:09:49.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.429+0000 7ff22a74c700 1 -- start start 2024-03-18T19:09:49.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.431+0000 7ff22a74c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff22415b6c0 0x7ff2240b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:49.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.431+0000 7ff22a74c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2240cd430 0x7ff2240c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:49.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.431+0000 7ff22a74c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2240d01f0 con 0x7ff2240c8600 2024-03-18T19:09:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.431+0000 7ff22a74c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2240ccfa0 con 0x7ff22415b6c0 2024-03-18T19:09:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.431+0000 7ff22a74c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2240d1db0 con 0x7ff2240cd430 2024-03-18T19:09:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.431+0000 7ff223fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff22415b6c0 0x7ff2240b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.431+0000 7ff228ce9700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff2240c8600 0x7ff2240cedd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59530/0 (socket says 172.21.15.1:59530) 2024-03-18T19:09:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.431+0000 7ff228ce9700 1 -- 172.21.15.1:0/1259926382 learned_addr learned my addr 172.21.15.1:0/1259926382 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.431+0000 7ff2237fe700 1 --2- 172.21.15.1:0/1259926382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2240cd430 0x7ff2240c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.431+0000 7ff223fff700 1 -- 172.21.15.1:0/1259926382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2240cd430 msgr2=0x7ff2240c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.431+0000 7ff223fff700 1 --2- 172.21.15.1:0/1259926382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2240cd430 0x7ff2240c80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.431+0000 7ff223fff700 1 -- 172.21.15.1:0/1259926382 >> v1:172.21.15.1:6789/0 conn(0x7ff2240c8600 legacy=0x7ff2240cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.431+0000 7ff223fff700 1 -- 172.21.15.1:0/1259926382 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2240cff10 con 0x7ff22415b6c0 2024-03-18T19:09:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.431+0000 7ff222ffd700 1 -- 172.21.15.1:0/1259926382 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845298991 0 0) 0x7ff2240d01f0 con 0x7ff2240c8600 2024-03-18T19:09:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.431+0000 7ff223fff700 1 --2- 172.21.15.1:0/1259926382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff22415b6c0 0x7ff2240b2160 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7ff214009870 tx=0x7ff214009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81b1b3d9ec6899f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:49.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.431+0000 7ff222ffd700 1 -- 172.21.15.1:0/1259926382 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff214011830 con 0x7ff22415b6c0 2024-03-18T19:09:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.431+0000 7ff222ffd700 1 -- 172.21.15.1:0/1259926382 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff214011990 con 0x7ff22415b6c0 2024-03-18T19:09:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.432+0000 7ff222ffd700 1 -- 172.21.15.1:0/1259926382 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff214011d20 con 0x7ff22415b6c0 2024-03-18T19:09:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.432+0000 7ff22a74c700 1 -- 172.21.15.1:0/1259926382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff22415b6c0 msgr2=0x7ff2240b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.432+0000 7ff22a74c700 1 --2- 172.21.15.1:0/1259926382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff22415b6c0 0x7ff2240b2160 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7ff214009870 tx=0x7ff214009b80 comp rx=0 tx=0).stop 2024-03-18T19:09:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.432+0000 7ff22a74c700 1 -- 172.21.15.1:0/1259926382 shutdown_connections 2024-03-18T19:09:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.432+0000 7ff22a74c700 1 --2- 172.21.15.1:0/1259926382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff22415b6c0 0x7ff2240b2160 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.432+0000 7ff22a74c700 1 --2- 172.21.15.1:0/1259926382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2240cd430 0x7ff2240c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.432+0000 7ff22a74c700 1 -- 172.21.15.1:0/1259926382 >> 172.21.15.1:0/1259926382 conn(0x7ff2240d0ff0 msgr2=0x7ff2240d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.432+0000 7ff22a74c700 1 -- 172.21.15.1:0/1259926382 shutdown_connections 2024-03-18T19:09:49.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.432+0000 7ff22a74c700 1 -- 172.21.15.1:0/1259926382 wait complete. 2024-03-18T19:09:49.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.434+0000 7ff22a74c700 1 Processor -- start 2024-03-18T19:09:49.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.434+0000 7ff22a74c700 1 -- start start 2024-03-18T19:09:49.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.435+0000 7ff22a74c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2240c8600 0x7ff2240b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:49.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.435+0000 7ff22a74c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2240cd430 0x7ff22409e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:49.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.435+0000 7ff22a74c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff22415b6c0 0x7ff22409e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:49.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.435+0000 7ff22a74c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2240b6820 con 0x7ff22415b6c0 2024-03-18T19:09:49.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.435+0000 7ff22a74c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22409ae60 con 0x7ff2240c8600 2024-03-18T19:09:49.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.435+0000 7ff22a74c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22409b140 con 0x7ff2240cd430 2024-03-18T19:09:49.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.435+0000 7ff228ce9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff22415b6c0 0x7ff22409e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:49.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.435+0000 7ff2237fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2240cd430 0x7ff22409e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:49.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.435+0000 7ff228ce9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff22415b6c0 0x7ff22409e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43960/0 (socket says 172.21.15.1:43960) 2024-03-18T19:09:49.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.435+0000 7ff228ce9700 1 -- 172.21.15.1:0/606766564 learned_addr learned my addr 172.21.15.1:0/606766564 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:49.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.435+0000 7ff223fff700 1 --2- 172.21.15.1:0/606766564 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2240c8600 0x7ff2240b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:49.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.435+0000 7ff228ce9700 1 -- 172.21.15.1:0/606766564 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2240cd430 msgr2=0x7ff22409e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:49.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.435+0000 7ff228ce9700 1 --2- 172.21.15.1:0/606766564 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2240cd430 0x7ff22409e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.435+0000 7ff228ce9700 1 -- 172.21.15.1:0/606766564 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2240c8600 msgr2=0x7ff2240b62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:49.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.435+0000 7ff228ce9700 1 --2- 172.21.15.1:0/606766564 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2240c8600 0x7ff2240b62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.435+0000 7ff228ce9700 1 -- 172.21.15.1:0/606766564 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2140094a0 con 0x7ff22415b6c0 2024-03-18T19:09:49.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.436+0000 7ff228ce9700 1 --2- 172.21.15.1:0/606766564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff22415b6c0 0x7ff22409e680 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7ff21c00cba0 tx=0x7ff21c00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:49.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.436+0000 7ff220ff9700 1 -- 172.21.15.1:0/606766564 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c00aaa0 con 0x7ff22415b6c0 2024-03-18T19:09:49.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.436+0000 7ff220ff9700 1 -- 172.21.15.1:0/606766564 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21c00ac20 con 0x7ff22415b6c0 2024-03-18T19:09:49.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.436+0000 7ff22a74c700 1 -- 172.21.15.1:0/606766564 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22409b420 con 0x7ff22415b6c0 2024-03-18T19:09:49.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.436+0000 7ff220ff9700 1 -- 172.21.15.1:0/606766564 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c0070d0 con 0x7ff22415b6c0 2024-03-18T19:09:49.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.436+0000 7ff22a74c700 1 -- 172.21.15.1:0/606766564 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2240c7bb0 con 0x7ff22415b6c0 2024-03-18T19:09:49.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.437+0000 7ff22a74c700 1 -- 172.21.15.1:0/606766564 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff22409b940 con 0x7ff22415b6c0 2024-03-18T19:09:49.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.437+0000 7ff220ff9700 1 -- 172.21.15.1:0/606766564 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 78) v1 ==== 89296+0+0 (secure 0 0 0) 0x7ff21c007230 con 0x7ff22415b6c0 2024-03-18T19:09:49.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.438+0000 7ff220ff9700 1 --2- 172.21.15.1:0/606766564 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff21006ae40 0x7ff21006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:49.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.438+0000 7ff223fff700 1 -- 172.21.15.1:0/606766564 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff21006ae40 msgr2=0x7ff21006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:49.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.438+0000 7ff220ff9700 1 -- 172.21.15.1:0/606766564 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7ff21c089380 con 0x7ff22415b6c0 2024-03-18T19:09:49.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.438+0000 7ff223fff700 1 --2- 172.21.15.1:0/606766564 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff21006ae40 0x7ff21006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:49.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.443+0000 7ff220ff9700 1 -- 172.21.15.1:0/606766564 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff21c054020 con 0x7ff22415b6c0 2024-03-18T19:09:49.520 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:49.524+0000 7f441fadd240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:09:49.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.639+0000 7ff223fff700 1 -- 172.21.15.1:0/606766564 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff21006ae40 msgr2=0x7ff21006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:49.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.639+0000 7ff223fff700 1 --2- 172.21.15.1:0/606766564 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff21006ae40 0x7ff21006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:49.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.648+0000 7ff22a74c700 1 -- 172.21.15.1:0/606766564 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff2240676a0 con 0x7ff22415b6c0 2024-03-18T19:09:49.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.648+0000 7ff220ff9700 1 -- 172.21.15.1:0/606766564 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v148) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff21c0180a0 con 0x7ff22415b6c0 2024-03-18T19:09:49.644 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T19:09:49.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.650+0000 7ff22a74c700 1 -- 172.21.15.1:0/606766564 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff21006ae40 msgr2=0x7ff21006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:49.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.650+0000 7ff22a74c700 1 --2- 172.21.15.1:0/606766564 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff21006ae40 0x7ff21006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.650+0000 7ff22a74c700 1 -- 172.21.15.1:0/606766564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff22415b6c0 msgr2=0x7ff22409e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:49.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.650+0000 7ff22a74c700 1 --2- 172.21.15.1:0/606766564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff22415b6c0 0x7ff22409e680 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7ff21c00cba0 tx=0x7ff21c00ceb0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.651+0000 7ff22a74c700 1 -- 172.21.15.1:0/606766564 shutdown_connections 2024-03-18T19:09:49.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.651+0000 7ff22a74c700 1 --2- 172.21.15.1:0/606766564 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff21006ae40 0x7ff21006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.651+0000 7ff22a74c700 1 --2- 172.21.15.1:0/606766564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff22415b6c0 0x7ff22409e680 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.651+0000 7ff22a74c700 1 --2- 172.21.15.1:0/606766564 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2240c8600 0x7ff2240b62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.651+0000 7ff22a74c700 1 --2- 172.21.15.1:0/606766564 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2240cd430 0x7ff22409e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.651+0000 7ff22a74c700 1 -- 172.21.15.1:0/606766564 >> 172.21.15.1:0/606766564 conn(0x7ff2240d0ff0 msgr2=0x7ff22415e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:49.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.651+0000 7ff22a74c700 1 -- 172.21.15.1:0/606766564 shutdown_connections 2024-03-18T19:09:49.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.651+0000 7ff22a74c700 1 -- 172.21.15.1:0/606766564 wait complete. 2024-03-18T19:09:49.659 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:09:49.760 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:49.764+0000 7ffbca1f9240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:09:49.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.796+0000 7fb8415de700 1 Processor -- start 2024-03-18T19:09:49.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.796+0000 7fb8415de700 1 -- start start 2024-03-18T19:09:49.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.797+0000 7fb8415de700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb83c099a40 0x7fb83c06ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:49.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.797+0000 7fb8415de700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb83c09a530 0x7fb83c15d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:49.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.797+0000 7fb8415de700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb83c0cf7e0 con 0x7fb83c0d05a0 2024-03-18T19:09:49.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.797+0000 7fb8415de700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb83c09a9a0 con 0x7fb83c099a40 2024-03-18T19:09:49.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.797+0000 7fb8415de700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c0cfac0 con 0x7fb83c09a530 2024-03-18T19:09:49.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.797+0000 7fb83affd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb83c099a40 0x7fb83c06ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.798+0000 7fb83b7fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb83c09a530 0x7fb83c15d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.798+0000 7fb83a7fc700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fb83c0d05a0 0x7fb83c099e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:59670/0 (socket says 172.21.15.1:59670) 2024-03-18T19:09:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.798+0000 7fb83a7fc700 1 -- 172.21.15.1:0/3652610206 learned_addr learned my addr 172.21.15.1:0/3652610206 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.798+0000 7fb83affd700 1 -- 172.21.15.1:0/3652610206 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb83c09a530 msgr2=0x7fb83c15d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.798+0000 7fb83affd700 1 --2- 172.21.15.1:0/3652610206 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb83c09a530 0x7fb83c15d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.798+0000 7fb83affd700 1 -- 172.21.15.1:0/3652610206 >> v1:172.21.15.165:6789/0 conn(0x7fb83c0d05a0 legacy=0x7fb83c099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.798+0000 7fb83affd700 1 -- 172.21.15.1:0/3652610206 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83c0cfda0 con 0x7fb83c099a40 2024-03-18T19:09:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.798+0000 7fb83b7fe700 1 --2- 172.21.15.1:0/3652610206 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb83c09a530 0x7fb83c15d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.798+0000 7fb83affd700 1 --2- 172.21.15.1:0/3652610206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb83c099a40 0x7fb83c06ba70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fb82c009840 tx=0x7fb82c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d25a2a83f1d53e6a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.798+0000 7fb839ffb700 1 -- 172.21.15.1:0/3652610206 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c011830 con 0x7fb83c099a40 2024-03-18T19:09:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.798+0000 7fb839ffb700 1 -- 172.21.15.1:0/3652610206 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82c011990 con 0x7fb83c099a40 2024-03-18T19:09:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.798+0000 7fb839ffb700 1 -- 172.21.15.1:0/3652610206 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c011d20 con 0x7fb83c099a40 2024-03-18T19:09:49.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.799+0000 7fb8415de700 1 -- 172.21.15.1:0/3652610206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb83c099a40 msgr2=0x7fb83c06ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:49.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.799+0000 7fb8415de700 1 --2- 172.21.15.1:0/3652610206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb83c099a40 0x7fb83c06ba70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fb82c009840 tx=0x7fb82c009b50 comp rx=0 tx=0).stop 2024-03-18T19:09:49.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.799+0000 7fb8415de700 1 -- 172.21.15.1:0/3652610206 shutdown_connections 2024-03-18T19:09:49.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.799+0000 7fb8415de700 1 --2- 172.21.15.1:0/3652610206 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb83c09a530 0x7fb83c15d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.799+0000 7fb8415de700 1 --2- 172.21.15.1:0/3652610206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb83c099a40 0x7fb83c06ba70 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.799+0000 7fb8415de700 1 -- 172.21.15.1:0/3652610206 >> 172.21.15.1:0/3652610206 conn(0x7fb83c0b2cb0 msgr2=0x7fb83c0b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:49.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.799+0000 7fb8415de700 1 -- 172.21.15.1:0/3652610206 shutdown_connections 2024-03-18T19:09:49.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.799+0000 7fb8415de700 1 -- 172.21.15.1:0/3652610206 wait complete. 2024-03-18T19:09:49.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.800+0000 7fb8415de700 1 Processor -- start 2024-03-18T19:09:49.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.800+0000 7fb8415de700 1 -- start start 2024-03-18T19:09:49.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.801+0000 7fb8415de700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb83c099a40 0x7fb83c14bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:49.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.801+0000 7fb8415de700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb83c09a530 0x7fb83c15e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:49.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.801+0000 7fb8415de700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb83c0d05a0 0x7fb83c15e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:49.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.801+0000 7fb8415de700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c0c77a0 con 0x7fb83c0d05a0 2024-03-18T19:09:49.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.801+0000 7fb8415de700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c0c6e00 con 0x7fb83c099a40 2024-03-18T19:09:49.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.801+0000 7fb8415de700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c0c70e0 con 0x7fb83c09a530 2024-03-18T19:09:49.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.801+0000 7fb83b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb83c0d05a0 0x7fb83c15e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.801+0000 7fb83b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb83c0d05a0 0x7fb83c15e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43974/0 (socket says 172.21.15.1:43974) 2024-03-18T19:09:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.801+0000 7fb83affd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb83c099a40 0x7fb83c14bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.801+0000 7fb83b7fe700 1 -- 172.21.15.1:0/2723566398 learned_addr learned my addr 172.21.15.1:0/2723566398 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.801+0000 7fb83a7fc700 1 --2- 172.21.15.1:0/2723566398 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb83c09a530 0x7fb83c15e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.802+0000 7fb83b7fe700 1 -- 172.21.15.1:0/2723566398 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb83c09a530 msgr2=0x7fb83c15e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.802+0000 7fb83b7fe700 1 --2- 172.21.15.1:0/2723566398 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb83c09a530 0x7fb83c15e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.802+0000 7fb83b7fe700 1 -- 172.21.15.1:0/2723566398 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb83c099a40 msgr2=0x7fb83c14bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.802+0000 7fb83b7fe700 1 --2- 172.21.15.1:0/2723566398 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb83c099a40 0x7fb83c14bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.802+0000 7fb83b7fe700 1 -- 172.21.15.1:0/2723566398 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb834009a70 con 0x7fb83c0d05a0 2024-03-18T19:09:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.802+0000 7fb83affd700 1 --2- 172.21.15.1:0/2723566398 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb83c099a40 0x7fb83c14bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.802+0000 7fb83b7fe700 1 --2- 172.21.15.1:0/2723566398 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb83c0d05a0 0x7fb83c15e750 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fb83400efc0 tx=0x7fb83400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.802+0000 7fb8237fe700 1 -- 172.21.15.1:0/2723566398 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83400ca90 con 0x7fb83c0d05a0 2024-03-18T19:09:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.802+0000 7fb8237fe700 1 -- 172.21.15.1:0/2723566398 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83400f040 con 0x7fb83c0d05a0 2024-03-18T19:09:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.802+0000 7fb8415de700 1 -- 172.21.15.1:0/2723566398 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb82c0094a0 con 0x7fb83c0d05a0 2024-03-18T19:09:49.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.802+0000 7fb8237fe700 1 -- 172.21.15.1:0/2723566398 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83400ce10 con 0x7fb83c0d05a0 2024-03-18T19:09:49.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.802+0000 7fb8415de700 1 -- 172.21.15.1:0/2723566398 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb83c0c6af0 con 0x7fb83c0d05a0 2024-03-18T19:09:49.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.803+0000 7fb8415de700 1 -- 172.21.15.1:0/2723566398 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb83c068240 con 0x7fb83c0d05a0 2024-03-18T19:09:49.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.803+0000 7fb8237fe700 1 -- 172.21.15.1:0/2723566398 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 78) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fb834014480 con 0x7fb83c0d05a0 2024-03-18T19:09:49.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.804+0000 7fb8237fe700 1 --2- 172.21.15.1:0/2723566398 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb82806ae40 0x7fb82806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:49.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.804+0000 7fb83affd700 1 -- 172.21.15.1:0/2723566398 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb82806ae40 msgr2=0x7fb82806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.804+0000 7fb8237fe700 1 -- 172.21.15.1:0/2723566398 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fb83408a150 con 0x7fb83c0d05a0 2024-03-18T19:09:49.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.804+0000 7fb83affd700 1 --2- 172.21.15.1:0/2723566398 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb82806ae40 0x7fb82806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:49.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:49.809+0000 7fb8237fe700 1 -- 172.21.15.1:0/2723566398 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb834054bd0 con 0x7fb83c0d05a0 2024-03-18T19:09:49.908 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:49.911+0000 7f441fadd240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:09:49.939 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:49.943+0000 7ffbca1f9240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:09:50.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.005+0000 7fb83affd700 1 -- 172.21.15.1:0/2723566398 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb82806ae40 msgr2=0x7fb82806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:50.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.005+0000 7fb83affd700 1 --2- 172.21.15.1:0/2723566398 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb82806ae40 0x7fb82806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:50.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.061+0000 7fb8415de700 1 -- 172.21.15.1:0/2723566398 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb83c0cda40 con 0x7fb83c0d05a0 2024-03-18T19:09:50.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.061+0000 7fb8237fe700 1 -- 172.21.15.1:0/2723566398 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+4727 (secure 0 0 0) 0x7fb834009d70 con 0x7fb83c0d05a0 2024-03-18T19:09:50.057 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:50.058 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8116,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3723189348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3723189348},{"type":"v1","addr":"172.21.15.1:6837","nonce":3723189348}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":8125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/530206363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":530206363},{"type":"v1","addr":"172.21.15.165:6835","nonce":530206363}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":8784,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/332827595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":332827595},{"type":"v1","addr":"172.21.15.1:6836","nonce":332827595}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:09:22.623415+0000","modified":"2024-03-18T19:09:25.640243+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6834/311436411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":311436411},{"type":"v1","addr":"172.21.15.165:6834","nonce":311436411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-18T19:09:50.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.064+0000 7fb8415de700 1 -- 172.21.15.1:0/2723566398 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb82806ae40 msgr2=0x7fb82806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:50.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.064+0000 7fb8415de700 1 --2- 172.21.15.1:0/2723566398 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb82806ae40 0x7fb82806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.064+0000 7fb8415de700 1 -- 172.21.15.1:0/2723566398 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb83c0d05a0 msgr2=0x7fb83c15e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:50.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.064+0000 7fb8415de700 1 --2- 172.21.15.1:0/2723566398 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb83c0d05a0 0x7fb83c15e750 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fb83400efc0 tx=0x7fb83400c3a0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.064+0000 7fb8415de700 1 -- 172.21.15.1:0/2723566398 shutdown_connections 2024-03-18T19:09:50.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.064+0000 7fb8415de700 1 --2- 172.21.15.1:0/2723566398 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb82806ae40 0x7fb82806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.064+0000 7fb8415de700 1 --2- 172.21.15.1:0/2723566398 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb83c0d05a0 0x7fb83c15e750 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.064+0000 7fb8415de700 1 --2- 172.21.15.1:0/2723566398 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb83c099a40 0x7fb83c14bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.064+0000 7fb8415de700 1 --2- 172.21.15.1:0/2723566398 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb83c09a530 0x7fb83c15e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.064+0000 7fb8415de700 1 -- 172.21.15.1:0/2723566398 >> 172.21.15.1:0/2723566398 conn(0x7fb83c0b2cb0 msgr2=0x7fb83c06c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.064+0000 7fb8415de700 1 -- 172.21.15.1:0/2723566398 shutdown_connections 2024-03-18T19:09:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.064+0000 7fb8415de700 1 -- 172.21.15.1:0/2723566398 wait complete. 2024-03-18T19:09:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 65 2024-03-18T19:09:50.070 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:50.074+0000 7f441fadd240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:09:50.072 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:09:50.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.202+0000 7effbcd92700 1 Processor -- start 2024-03-18T19:09:50.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.203+0000 7effbcd92700 1 -- start start 2024-03-18T19:09:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.203+0000 7effbcd92700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7effb80d17b0 0x7effb80d1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.203+0000 7effbcd92700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7effb80b83f0 0x7effb80b8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.203+0000 7effbcd92700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb80cddc0 con 0x7effb815d500 2024-03-18T19:09:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.203+0000 7effbcd92700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb80d21a0 con 0x7effb80d17b0 2024-03-18T19:09:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.203+0000 7effbcd92700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb80d3dc0 con 0x7effb80b83f0 2024-03-18T19:09:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.204+0000 7effb5d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7effb80d17b0 0x7effb80d1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.204+0000 7effb5d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7effb80d17b0 0x7effb80d1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43978/0 (socket says 172.21.15.1:43978) 2024-03-18T19:09:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.204+0000 7effb5d9b700 1 -- 172.21.15.1:0/361082028 learned_addr learned my addr 172.21.15.1:0/361082028 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.204+0000 7effb6d9d700 1 --2- 172.21.15.1:0/361082028 >> v2:172.21.15.165:3300/0 conn(0x7effb80b83f0 0x7effb80b8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.204+0000 7effb5d9b700 1 -- 172.21.15.1:0/361082028 >> v2:172.21.15.165:3300/0 conn(0x7effb80b83f0 msgr2=0x7effb80b8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.204+0000 7effb5d9b700 1 --2- 172.21.15.1:0/361082028 >> v2:172.21.15.165:3300/0 conn(0x7effb80b83f0 0x7effb80b8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.204+0000 7effb5d9b700 1 -- 172.21.15.1:0/361082028 >> v1:172.21.15.165:6789/0 conn(0x7effb815d500 legacy=0x7effb80d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.204+0000 7effb5d9b700 1 -- 172.21.15.1:0/361082028 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb80ce060 con 0x7effb80d17b0 2024-03-18T19:09:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.204+0000 7effb5d9b700 1 --2- 172.21.15.1:0/361082028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effb80d17b0 0x7effb80d1c60 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7effb808d2b0 tx=0x7effac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae93410550d95c5a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.204+0000 7effb559a700 1 -- 172.21.15.1:0/361082028 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac011830 con 0x7effb80d17b0 2024-03-18T19:09:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.204+0000 7effb559a700 1 -- 172.21.15.1:0/361082028 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effac011990 con 0x7effb80d17b0 2024-03-18T19:09:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.204+0000 7effb559a700 1 -- 172.21.15.1:0/361082028 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac011d20 con 0x7effb80d17b0 2024-03-18T19:09:50.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.205+0000 7effbcd92700 1 -- 172.21.15.1:0/361082028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effb80d17b0 msgr2=0x7effb80d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:50.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.205+0000 7effbcd92700 1 --2- 172.21.15.1:0/361082028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effb80d17b0 0x7effb80d1c60 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7effb808d2b0 tx=0x7effac009bd0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.205+0000 7effbcd92700 1 -- 172.21.15.1:0/361082028 shutdown_connections 2024-03-18T19:09:50.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.205+0000 7effbcd92700 1 --2- 172.21.15.1:0/361082028 >> v2:172.21.15.165:3300/0 conn(0x7effb80b83f0 0x7effb80b8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.205+0000 7effbcd92700 1 --2- 172.21.15.1:0/361082028 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effb80d17b0 0x7effb80d1c60 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.205+0000 7effbcd92700 1 -- 172.21.15.1:0/361082028 >> 172.21.15.1:0/361082028 conn(0x7effb809c530 msgr2=0x7effb80d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:50.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.205+0000 7effbcd92700 1 -- 172.21.15.1:0/361082028 shutdown_connections 2024-03-18T19:09:50.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.205+0000 7effbcd92700 1 -- 172.21.15.1:0/361082028 wait complete. 2024-03-18T19:09:50.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.206+0000 7effbcd92700 1 Processor -- start 2024-03-18T19:09:50.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.206+0000 7effbcd92700 1 -- start start 2024-03-18T19:09:50.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.207+0000 7effbcd92700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effb80b83f0 0x7effb8151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:50.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.207+0000 7effbcd92700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effb80d17b0 0x7effb80c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:50.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.207+0000 7effbcd92700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effb815d500 0x7effb80b7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:50.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.207+0000 7effbcd92700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb80c9900 con 0x7effb815d500 2024-03-18T19:09:50.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.207+0000 7effbcd92700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb81e71d0 con 0x7effb80d17b0 2024-03-18T19:09:50.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.207+0000 7effbcd92700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb81e74b0 con 0x7effb80b83f0 2024-03-18T19:09:50.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.207+0000 7effb6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effb815d500 0x7effb80b7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:50.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.207+0000 7effb5d9b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effb80d17b0 0x7effb80c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:50.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.207+0000 7effb6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effb815d500 0x7effb80b7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43992/0 (socket says 172.21.15.1:43992) 2024-03-18T19:09:50.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.207+0000 7effb5d9b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effb80d17b0 0x7effb80c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:57250/0 (socket says 172.21.15.1:57250) 2024-03-18T19:09:50.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.207+0000 7effb6d9d700 1 -- 172.21.15.1:0/1132371348 learned_addr learned my addr 172.21.15.1:0/1132371348 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:50.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.207+0000 7effb659c700 1 --2- 172.21.15.1:0/1132371348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effb80b83f0 0x7effb8151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:50.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.207+0000 7effb6d9d700 1 -- 172.21.15.1:0/1132371348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effb80b83f0 msgr2=0x7effb8151890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:50.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.207+0000 7effb6d9d700 1 --2- 172.21.15.1:0/1132371348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effb80b83f0 0x7effb8151890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.207+0000 7effb6d9d700 1 -- 172.21.15.1:0/1132371348 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effb80d17b0 msgr2=0x7effb80c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:50.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.208+0000 7effb6d9d700 1 --2- 172.21.15.1:0/1132371348 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effb80d17b0 0x7effb80c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.208+0000 7effb6d9d700 1 -- 172.21.15.1:0/1132371348 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac009550 con 0x7effb815d500 2024-03-18T19:09:50.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.208+0000 7effb5d9b700 1 --2- 172.21.15.1:0/1132371348 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effb80d17b0 0x7effb80c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:50.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.208+0000 7effb6d9d700 1 --2- 172.21.15.1:0/1132371348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effb815d500 0x7effb80b7570 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7effb000efc0 tx=0x7effb000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:50.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.208+0000 7effa6ffd700 1 -- 172.21.15.1:0/1132371348 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0009ea0 con 0x7effb815d500 2024-03-18T19:09:50.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.208+0000 7effa6ffd700 1 -- 172.21.15.1:0/1132371348 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb000f040 con 0x7effb815d500 2024-03-18T19:09:50.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.208+0000 7effbcd92700 1 -- 172.21.15.1:0/1132371348 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb81e7790 con 0x7effb815d500 2024-03-18T19:09:50.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.208+0000 7effa6ffd700 1 -- 172.21.15.1:0/1132371348 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb000ced0 con 0x7effb815d500 2024-03-18T19:09:50.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.208+0000 7effbcd92700 1 -- 172.21.15.1:0/1132371348 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb81e7d50 con 0x7effb815d500 2024-03-18T19:09:50.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.209+0000 7effbcd92700 1 -- 172.21.15.1:0/1132371348 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb81e7f40 con 0x7effb815d500 2024-03-18T19:09:50.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.209+0000 7effa6ffd700 1 -- 172.21.15.1:0/1132371348 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 78) v1 ==== 89296+0+0 (secure 0 0 0) 0x7effb0014480 con 0x7effb815d500 2024-03-18T19:09:50.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.210+0000 7effa6ffd700 1 --2- 172.21.15.1:0/1132371348 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7eff9806ae40 0x7eff9806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:50.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.210+0000 7effb659c700 1 -- 172.21.15.1:0/1132371348 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7eff9806ae40 msgr2=0x7eff9806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:50.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.210+0000 7effa6ffd700 1 -- 172.21.15.1:0/1132371348 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7effb008a1f0 con 0x7effb815d500 2024-03-18T19:09:50.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.210+0000 7effb659c700 1 --2- 172.21.15.1:0/1132371348 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7eff9806ae40 0x7eff9806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:50.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.214+0000 7effa6ffd700 1 -- 172.21.15.1:0/1132371348 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7effb0054d80 con 0x7effb815d500 2024-03-18T19:09:50.378 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:50.383+0000 7ffbca1f9240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:09:50.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.411+0000 7effb659c700 1 -- 172.21.15.1:0/1132371348 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7eff9806ae40 msgr2=0x7eff9806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:50.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.411+0000 7effb659c700 1 --2- 172.21.15.1:0/1132371348 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7eff9806ae40 0x7eff9806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:50.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.434+0000 7effbcd92700 1 -- 172.21.15.1:0/1132371348 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effb80676a0 con 0x7effb815d500 2024-03-18T19:09:50.431 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:50.434+0000 7f441fadd240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:09:50.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.435+0000 7effa6ffd700 1 -- 172.21.15.1:0/1132371348 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+16790 (secure 0 0 0) 0x7effb00583a0 con 0x7effb815d500 2024-03-18T19:09:50.432 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:50.432 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":148,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:09:37.850698+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:09:20.468744+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-18T19:09:21.597476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/69006586":"2024-03-19T19:09:37.850642+0000","172.21.15.1:6833/76998":"2024-03-19T19:09:37.850642+0000","172.21.15.1:6832/76998":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/1403689662":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/806972823":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/619565916":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/2042700873":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/744112928":"2024-03-19T19:09:37.850642+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:09:50.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.437+0000 7effbcd92700 1 -- 172.21.15.1:0/1132371348 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7eff9806ae40 msgr2=0x7eff9806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:50.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.437+0000 7effbcd92700 1 --2- 172.21.15.1:0/1132371348 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7eff9806ae40 0x7eff9806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.438+0000 7effbcd92700 1 -- 172.21.15.1:0/1132371348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effb815d500 msgr2=0x7effb80b7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:50.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.438+0000 7effbcd92700 1 --2- 172.21.15.1:0/1132371348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effb815d500 0x7effb80b7570 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7effb000efc0 tx=0x7effb000c3a0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.438+0000 7effbcd92700 1 -- 172.21.15.1:0/1132371348 shutdown_connections 2024-03-18T19:09:50.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.438+0000 7effbcd92700 1 --2- 172.21.15.1:0/1132371348 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7eff9806ae40 0x7eff9806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.438+0000 7effbcd92700 1 --2- 172.21.15.1:0/1132371348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effb80b83f0 0x7effb8151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.438+0000 7effbcd92700 1 --2- 172.21.15.1:0/1132371348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effb815d500 0x7effb80b7570 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.438+0000 7effbcd92700 1 --2- 172.21.15.1:0/1132371348 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effb80d17b0 0x7effb80c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.438+0000 7effbcd92700 1 -- 172.21.15.1:0/1132371348 >> 172.21.15.1:0/1132371348 conn(0x7effb809c530 msgr2=0x7effb806b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:50.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.438+0000 7effbcd92700 1 -- 172.21.15.1:0/1132371348 shutdown_connections 2024-03-18T19:09:50.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.438+0000 7effbcd92700 1 -- 172.21.15.1:0/1132371348 wait complete. 2024-03-18T19:09:50.447 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T19:09:50.447 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:09:50.518 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:09:50.522+0000 7f441fadd240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:09:50.560 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:50.564+0000 7ffbca1f9240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:09:50.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.576+0000 7f7fa6b46700 1 Processor -- start 2024-03-18T19:09:50.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.577+0000 7f7fa6b46700 1 -- start start 2024-03-18T19:09:50.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.578+0000 7f7fa6b46700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7fa015b4d0 0x7f7fa009a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:50.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.578+0000 7f7fa6b46700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fa00cf750 0x7f7fa0099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:50.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.578+0000 7f7fa6b46700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa006f070 con 0x7f7fa0099f80 2024-03-18T19:09:50.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.578+0000 7f7fa6b46700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fa009a410 con 0x7f7fa015b4d0 2024-03-18T19:09:50.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.578+0000 7f7fa6b46700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa00d0260 con 0x7f7fa00cf750 2024-03-18T19:09:50.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.578+0000 7f7fa48e2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7fa015b4d0 0x7f7fa009a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:50.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.578+0000 7f7fa48e2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7fa015b4d0 0x7f7fa009a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44006/0 (socket says 172.21.15.1:44006) 2024-03-18T19:09:50.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.578+0000 7f7fa48e2700 1 -- 172.21.15.1:0/1438119850 learned_addr learned my addr 172.21.15.1:0/1438119850 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:50.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.578+0000 7f7fa50e3700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7fa0099f80 0x7f7fa00cbd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59542/0 (socket says 172.21.15.1:59542) 2024-03-18T19:09:50.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.578+0000 7f7f9ffff700 1 --2- 172.21.15.1:0/1438119850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fa00cf750 0x7f7fa0099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:50.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.578+0000 7f7fa48e2700 1 -- 172.21.15.1:0/1438119850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fa00cf750 msgr2=0x7f7fa0099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:50.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.578+0000 7f7fa48e2700 1 --2- 172.21.15.1:0/1438119850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fa00cf750 0x7f7fa0099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.578+0000 7f7fa48e2700 1 -- 172.21.15.1:0/1438119850 >> v1:172.21.15.1:6789/0 conn(0x7f7fa0099f80 legacy=0x7f7fa00cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:50.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.578+0000 7f7fa48e2700 1 -- 172.21.15.1:0/1438119850 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa00cff80 con 0x7f7fa015b4d0 2024-03-18T19:09:50.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.578+0000 7f7f9f7fe700 1 -- 172.21.15.1:0/1438119850 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888698571 0 0) 0x7f7fa006f070 con 0x7f7fa0099f80 2024-03-18T19:09:50.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.578+0000 7f7f9ffff700 1 --2- 172.21.15.1:0/1438119850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fa00cf750 0x7f7fa0099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:50.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.578+0000 7f7fa48e2700 1 --2- 172.21.15.1:0/1438119850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa015b4d0 0x7f7fa009a650 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f7f940098f0 tx=0x7f7f94009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac35135ef04db2f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:50.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.579+0000 7f7f9f7fe700 1 -- 172.21.15.1:0/1438119850 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9400b8a0 con 0x7f7fa015b4d0 2024-03-18T19:09:50.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.579+0000 7f7f9f7fe700 1 -- 172.21.15.1:0/1438119850 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9400ba00 con 0x7f7fa015b4d0 2024-03-18T19:09:50.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.579+0000 7f7f9f7fe700 1 -- 172.21.15.1:0/1438119850 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9400bd90 con 0x7f7fa015b4d0 2024-03-18T19:09:50.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.579+0000 7f7fa6b46700 1 -- 172.21.15.1:0/1438119850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa015b4d0 msgr2=0x7f7fa009a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:50.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.579+0000 7f7fa6b46700 1 --2- 172.21.15.1:0/1438119850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa015b4d0 0x7f7fa009a650 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f7f940098f0 tx=0x7f7f94009c00 comp rx=0 tx=0).stop 2024-03-18T19:09:50.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.579+0000 7f7fa6b46700 1 -- 172.21.15.1:0/1438119850 shutdown_connections 2024-03-18T19:09:50.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.579+0000 7f7fa6b46700 1 --2- 172.21.15.1:0/1438119850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa015b4d0 0x7f7fa009a650 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.579+0000 7f7fa6b46700 1 --2- 172.21.15.1:0/1438119850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fa00cf750 0x7f7fa0099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.579+0000 7f7fa6b46700 1 -- 172.21.15.1:0/1438119850 >> 172.21.15.1:0/1438119850 conn(0x7f7fa00b2cb0 msgr2=0x7f7fa00b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:50.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.579+0000 7f7fa6b46700 1 -- 172.21.15.1:0/1438119850 shutdown_connections 2024-03-18T19:09:50.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.579+0000 7f7fa6b46700 1 -- 172.21.15.1:0/1438119850 wait complete. 2024-03-18T19:09:50.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.581+0000 7f7fa6b46700 1 Processor -- start 2024-03-18T19:09:50.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.581+0000 7f7fa6b46700 1 -- start start 2024-03-18T19:09:50.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.583+0000 7f7fa6b46700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa0099f80 0x7f7fa01f04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:50.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.583+0000 7f7fa6b46700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fa00cf750 0x7f7fa00ca460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:50.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.583+0000 7f7fa6b46700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fa015b4d0 0x7f7fa015c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:50.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.583+0000 7f7fa6b46700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa00c5760 con 0x7f7fa0099f80 2024-03-18T19:09:50.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.583+0000 7f7fa6b46700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa00cae20 con 0x7f7fa015b4d0 2024-03-18T19:09:50.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.583+0000 7f7fa6b46700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa00c4dc0 con 0x7f7fa00cf750 2024-03-18T19:09:50.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.583+0000 7f7fa48e2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa0099f80 0x7f7fa01f04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:50.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.583+0000 7f7fa48e2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa0099f80 0x7f7fa01f04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44014/0 (socket says 172.21.15.1:44014) 2024-03-18T19:09:50.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.583+0000 7f7fa48e2700 1 -- 172.21.15.1:0/2272616743 learned_addr learned my addr 172.21.15.1:0/2272616743 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:50.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.583+0000 7f7f9ffff700 1 --2- 172.21.15.1:0/2272616743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fa00cf750 0x7f7fa00ca460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:50.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.583+0000 7f7fa50e3700 1 --2- 172.21.15.1:0/2272616743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fa015b4d0 0x7f7fa015c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:50.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.583+0000 7f7fa48e2700 1 -- 172.21.15.1:0/2272616743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fa00cf750 msgr2=0x7f7fa00ca460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:50.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.583+0000 7f7fa48e2700 1 --2- 172.21.15.1:0/2272616743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fa00cf750 0x7f7fa00ca460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.583+0000 7f7fa48e2700 1 -- 172.21.15.1:0/2272616743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fa015b4d0 msgr2=0x7f7fa015c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:50.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.583+0000 7f7fa48e2700 1 --2- 172.21.15.1:0/2272616743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fa015b4d0 0x7f7fa015c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.583+0000 7f7fa48e2700 1 -- 172.21.15.1:0/2272616743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f94009550 con 0x7f7fa0099f80 2024-03-18T19:09:50.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.583+0000 7f7f9ffff700 1 --2- 172.21.15.1:0/2272616743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fa00cf750 0x7f7fa00ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:50.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.583+0000 7f7fa48e2700 1 --2- 172.21.15.1:0/2272616743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa0099f80 0x7f7fa01f04e0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f7f94014040 tx=0x7f7f94004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:50.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.584+0000 7f7f9d7fa700 1 -- 172.21.15.1:0/2272616743 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f940036a0 con 0x7f7fa0099f80 2024-03-18T19:09:50.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.584+0000 7f7f9d7fa700 1 -- 172.21.15.1:0/2272616743 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9401e070 con 0x7f7fa0099f80 2024-03-18T19:09:50.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.584+0000 7f7f9d7fa700 1 -- 172.21.15.1:0/2272616743 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f94003a40 con 0x7f7fa0099f80 2024-03-18T19:09:50.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.584+0000 7f7fa6b46700 1 -- 172.21.15.1:0/2272616743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa00c50a0 con 0x7f7fa0099f80 2024-03-18T19:09:50.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.584+0000 7f7fa6b46700 1 -- 172.21.15.1:0/2272616743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa01f0a20 con 0x7f7fa0099f80 2024-03-18T19:09:50.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.585+0000 7f7fa6b46700 1 -- 172.21.15.1:0/2272616743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa0068240 con 0x7f7fa0099f80 2024-03-18T19:09:50.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.585+0000 7f7f9d7fa700 1 -- 172.21.15.1:0/2272616743 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 78) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f7f94003ba0 con 0x7f7fa0099f80 2024-03-18T19:09:50.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.586+0000 7f7f9d7fa700 1 --2- 172.21.15.1:0/2272616743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f8406ae40 0x7f7f8406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:50.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.586+0000 7f7f9ffff700 1 -- 172.21.15.1:0/2272616743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f8406ae40 msgr2=0x7f7f8406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:50.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.586+0000 7f7f9d7fa700 1 -- 172.21.15.1:0/2272616743 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f7f94020070 con 0x7f7fa0099f80 2024-03-18T19:09:50.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.586+0000 7f7f9ffff700 1 --2- 172.21.15.1:0/2272616743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f8406ae40 0x7f7f8406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:50.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.590+0000 7f7f9d7fa700 1 -- 172.21.15.1:0/2272616743 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7f94055a40 con 0x7f7fa0099f80 2024-03-18T19:09:50.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.787+0000 7f7f9ffff700 1 -- 172.21.15.1:0/2272616743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f8406ae40 msgr2=0x7f7f8406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:50.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.787+0000 7f7f9ffff700 1 --2- 172.21.15.1:0/2272616743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f8406ae40 0x7f7f8406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:09:50.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.837+0000 7f7fa6b46700 1 -- 172.21.15.1:0/2272616743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fa006e030 con 0x7f7fa0099f80 2024-03-18T19:09:50.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.838+0000 7f7f9d7fa700 1 -- 172.21.15.1:0/2272616743 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+4727 (secure 0 0 0) 0x7f7f9402a090 con 0x7f7fa0099f80 2024-03-18T19:09:50.834 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:50.834 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8116,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3723189348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3723189348},{"type":"v1","addr":"172.21.15.1:6837","nonce":3723189348}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":8125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/530206363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":530206363},{"type":"v1","addr":"172.21.15.165:6835","nonce":530206363}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":8784,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/332827595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":332827595},{"type":"v1","addr":"172.21.15.1:6836","nonce":332827595}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:09:22.623415+0000","modified":"2024-03-18T19:09:25.640243+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6834/311436411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":311436411},{"type":"v1","addr":"172.21.15.165:6834","nonce":311436411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-18T19:09:50.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.840+0000 7f7fa6b46700 1 -- 172.21.15.1:0/2272616743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f8406ae40 msgr2=0x7f7f8406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:50.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.840+0000 7f7fa6b46700 1 --2- 172.21.15.1:0/2272616743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f8406ae40 0x7f7f8406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.840+0000 7f7fa6b46700 1 -- 172.21.15.1:0/2272616743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa0099f80 msgr2=0x7f7fa01f04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:50.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.840+0000 7f7fa6b46700 1 --2- 172.21.15.1:0/2272616743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa0099f80 0x7f7fa01f04e0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f7f94014040 tx=0x7f7f94004750 comp rx=0 tx=0).stop 2024-03-18T19:09:50.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.840+0000 7f7fa6b46700 1 -- 172.21.15.1:0/2272616743 shutdown_connections 2024-03-18T19:09:50.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.840+0000 7f7fa6b46700 1 --2- 172.21.15.1:0/2272616743 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f7f8406ae40 0x7f7f8406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.840+0000 7f7fa6b46700 1 --2- 172.21.15.1:0/2272616743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fa015b4d0 0x7f7fa015c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.840+0000 7f7fa6b46700 1 --2- 172.21.15.1:0/2272616743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fa0099f80 0x7f7fa01f04e0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.840+0000 7f7fa6b46700 1 --2- 172.21.15.1:0/2272616743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fa00cf750 0x7f7fa00ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.840+0000 7f7fa6b46700 1 -- 172.21.15.1:0/2272616743 >> 172.21.15.1:0/2272616743 conn(0x7f7fa00b2cb0 msgr2=0x7f7fa015a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:50.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.840+0000 7f7fa6b46700 1 -- 172.21.15.1:0/2272616743 shutdown_connections 2024-03-18T19:09:50.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.840+0000 7f7fa6b46700 1 -- 172.21.15.1:0/2272616743 wait complete. 2024-03-18T19:09:50.837 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 65 2024-03-18T19:09:50.849 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:09:50.948 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:50.952+0000 7ffbca1f9240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:09:50.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.980+0000 7f43dcc4e700 1 Processor -- start 2024-03-18T19:09:50.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.980+0000 7f43dcc4e700 1 -- start start 2024-03-18T19:09:50.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.981+0000 7f43dcc4e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f43d80cf960 0x7f43d80cf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:50.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.981+0000 7f43dcc4e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d806d640 con 0x7f43d80cdda0 2024-03-18T19:09:50.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.981+0000 7f43dcc4e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d806de50 con 0x7f43d806a750 2024-03-18T19:09:50.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.981+0000 7f43dcc4e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d80acd80 con 0x7f43d80cf960 2024-03-18T19:09:50.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.981+0000 7f43d5d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f43d80cf960 0x7f43d80cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:50.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.982+0000 7f43d6d9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f43d80cdda0 0x7f43d80ce250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59558/0 (socket says 172.21.15.1:59558) 2024-03-18T19:09:50.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.982+0000 7f43d5d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f43d80cf960 0x7f43d80cf280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44020/0 (socket says 172.21.15.1:44020) 2024-03-18T19:09:50.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.982+0000 7f43d6d9d700 1 -- 172.21.15.1:0/3384968728 learned_addr learned my addr 172.21.15.1:0/3384968728 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:50.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.982+0000 7f43d5d9b700 1 -- 172.21.15.1:0/3384968728 >> v1:172.21.15.165:6789/0 conn(0x7f43d806a750 legacy=0x7f43d806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:50.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.982+0000 7f43d5d9b700 1 -- 172.21.15.1:0/3384968728 >> v1:172.21.15.1:6789/0 conn(0x7f43d80cdda0 legacy=0x7f43d80ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:50.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.982+0000 7f43d5d9b700 1 -- 172.21.15.1:0/3384968728 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d806d890 con 0x7f43d80cf960 2024-03-18T19:09:50.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.982+0000 7f43d559a700 1 -- 172.21.15.1:0/3384968728 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623850934 0 0) 0x7f43d806d640 con 0x7f43d80cdda0 2024-03-18T19:09:50.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.982+0000 7f43d5d9b700 1 --2- 172.21.15.1:0/3384968728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d80cf960 0x7f43d80cf280 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f43cc009840 tx=0x7f43cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c784c9d89fb7f5a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:50.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.982+0000 7f43d559a700 1 -- 172.21.15.1:0/3384968728 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc011830 con 0x7f43d80cf960 2024-03-18T19:09:50.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.982+0000 7f43d559a700 1 -- 172.21.15.1:0/3384968728 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43cc011990 con 0x7f43d80cf960 2024-03-18T19:09:50.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.982+0000 7f43d559a700 1 -- 172.21.15.1:0/3384968728 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc011d20 con 0x7f43d80cf960 2024-03-18T19:09:50.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.983+0000 7f43dcc4e700 1 -- 172.21.15.1:0/3384968728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d80cf960 msgr2=0x7f43d80cf280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:50.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.983+0000 7f43dcc4e700 1 --2- 172.21.15.1:0/3384968728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d80cf960 0x7f43d80cf280 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f43cc009840 tx=0x7f43cc009b50 comp rx=0 tx=0).stop 2024-03-18T19:09:50.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.983+0000 7f43dcc4e700 1 -- 172.21.15.1:0/3384968728 shutdown_connections 2024-03-18T19:09:50.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.983+0000 7f43dcc4e700 1 --2- 172.21.15.1:0/3384968728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d80cf960 0x7f43d80cf280 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.983+0000 7f43dcc4e700 1 -- 172.21.15.1:0/3384968728 >> 172.21.15.1:0/3384968728 conn(0x7f43d806b860 msgr2=0x7f43d806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:50.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.983+0000 7f43dcc4e700 1 -- 172.21.15.1:0/3384968728 shutdown_connections 2024-03-18T19:09:50.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.983+0000 7f43dcc4e700 1 -- 172.21.15.1:0/3384968728 wait complete. 2024-03-18T19:09:50.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.984+0000 7f43dcc4e700 1 Processor -- start 2024-03-18T19:09:50.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.984+0000 7f43dcc4e700 1 -- start start 2024-03-18T19:09:50.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.985+0000 7f43dcc4e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43d806a750 0x7f43d81458a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:50.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.985+0000 7f43dcc4e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43d80cdda0 0x7f43d8154c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:50.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.985+0000 7f43dcc4e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d80cf960 0x7f43d8155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:50.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.985+0000 7f43dcc4e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d815cfd0 con 0x7f43d80cf960 2024-03-18T19:09:50.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.985+0000 7f43dcc4e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d815c450 con 0x7f43d80cdda0 2024-03-18T19:09:50.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.985+0000 7f43dcc4e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d815c730 con 0x7f43d806a750 2024-03-18T19:09:50.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.985+0000 7f43d6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d80cf960 0x7f43d8155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:50.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.985+0000 7f43d6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d80cf960 0x7f43d8155170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44022/0 (socket says 172.21.15.1:44022) 2024-03-18T19:09:50.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.985+0000 7f43d6d9d700 1 -- 172.21.15.1:0/3880330860 learned_addr learned my addr 172.21.15.1:0/3880330860 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:50.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.985+0000 7f43d5d9b700 1 --2- 172.21.15.1:0/3880330860 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43d80cdda0 0x7f43d8154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:50.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.985+0000 7f43d659c700 1 --2- 172.21.15.1:0/3880330860 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43d806a750 0x7f43d81458a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:50.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.986+0000 7f43d6d9d700 1 -- 172.21.15.1:0/3880330860 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43d806a750 msgr2=0x7f43d81458a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:50.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.986+0000 7f43d6d9d700 1 --2- 172.21.15.1:0/3880330860 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43d806a750 0x7f43d81458a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.986+0000 7f43d6d9d700 1 -- 172.21.15.1:0/3880330860 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43d80cdda0 msgr2=0x7f43d8154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:50.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.986+0000 7f43d6d9d700 1 --2- 172.21.15.1:0/3880330860 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43d80cdda0 0x7f43d8154c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:50.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.986+0000 7f43d6d9d700 1 -- 172.21.15.1:0/3880330860 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43cc0094a0 con 0x7f43d80cf960 2024-03-18T19:09:50.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.986+0000 7f43d6d9d700 1 --2- 172.21.15.1:0/3880330860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d80cf960 0x7f43d8155170 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f43d0002230 tx=0x7f43d000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:50.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.986+0000 7f43c6ffd700 1 -- 172.21.15.1:0/3880330860 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d000aac0 con 0x7f43d80cf960 2024-03-18T19:09:50.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.986+0000 7f43c6ffd700 1 -- 172.21.15.1:0/3880330860 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d000ac40 con 0x7f43d80cf960 2024-03-18T19:09:50.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.986+0000 7f43c6ffd700 1 -- 172.21.15.1:0/3880330860 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d00070d0 con 0x7f43d80cf960 2024-03-18T19:09:50.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.986+0000 7f43dcc4e700 1 -- 172.21.15.1:0/3880330860 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d815ca10 con 0x7f43d80cf960 2024-03-18T19:09:50.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.986+0000 7f43dcc4e700 1 -- 172.21.15.1:0/3880330860 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d8145de0 con 0x7f43d80cf960 2024-03-18T19:09:50.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.987+0000 7f43dcc4e700 1 -- 172.21.15.1:0/3880330860 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d81f0d40 con 0x7f43d80cf960 2024-03-18T19:09:50.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.987+0000 7f43c6ffd700 1 -- 172.21.15.1:0/3880330860 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 78) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f43d0007250 con 0x7f43d80cf960 2024-03-18T19:09:50.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.988+0000 7f43c6ffd700 1 --2- 172.21.15.1:0/3880330860 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43b806ae40 0x7f43b806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:50.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.988+0000 7f43d659c700 1 -- 172.21.15.1:0/3880330860 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43b806ae40 msgr2=0x7f43b806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:09:50.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.988+0000 7f43c6ffd700 1 -- 172.21.15.1:0/3880330860 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f43d00893b0 con 0x7f43d80cf960 2024-03-18T19:09:50.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.988+0000 7f43d659c700 1 --2- 172.21.15.1:0/3880330860 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43b806ae40 0x7f43b806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:09:50.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:50.992+0000 7f43c6ffd700 1 -- 172.21.15.1:0/3880330860 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f43d81f0d40 con 0x7f43d80cf960 2024-03-18T19:09:51.048 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:51.052+0000 7ffbca1f9240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:09:51.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.070+0000 7f43c6ffd700 1 -- 172.21.15.1:0/3880330860 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 79) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f43d0054380 con 0x7f43d80cf960 2024-03-18T19:09:51.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.071+0000 7f43c6ffd700 1 -- 172.21.15.1:0/3880330860 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43b806ae40 msgr2=0x7f43b806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:09:51.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.071+0000 7f43c6ffd700 1 --2- 172.21.15.1:0/3880330860 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43b806ae40 0x7f43b806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.131 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:51.135+0000 7ffb42621700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:51.131 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:51.135+0000 7ffb42621700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:51.132 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:51.135+0000 7ffb42621700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:51.132 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:51.135+0000 7ffb42621700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:51.132 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:51.135+0000 7ffb42621700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:51.132 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:51.136+0000 7ffb45627700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:51.132 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:51.136+0000 7ffb45627700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:51.132 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:51.136+0000 7ffb45627700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:51.132 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:51.136+0000 7ffb45627700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:51.132 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:09:51.136+0000 7ffb45627700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:09:51.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.258+0000 7f43dcc4e700 1 -- 172.21.15.1:0/3880330860 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43d81f0d40 con 0x7f43d80cf960 2024-03-18T19:09:51.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.259+0000 7f43c6ffd700 1 -- 172.21.15.1:0/3880330860 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+4727 (secure 0 0 0) 0x7f43d81f0d40 con 0x7f43d80cf960 2024-03-18T19:09:51.255 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:51.255 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8116,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3723189348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3723189348},{"type":"v1","addr":"172.21.15.1:6837","nonce":3723189348}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":8125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/530206363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":530206363},{"type":"v1","addr":"172.21.15.165:6835","nonce":530206363}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":8784,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/332827595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":332827595},{"type":"v1","addr":"172.21.15.1:6836","nonce":332827595}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:09:22.623415+0000","modified":"2024-03-18T19:09:25.640243+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6834/311436411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":311436411},{"type":"v1","addr":"172.21.15.165:6834","nonce":311436411}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-03-18T19:09:51.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.261+0000 7f43dcc4e700 1 -- 172.21.15.1:0/3880330860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d80cf960 msgr2=0x7f43d8155170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:51.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.261+0000 7f43dcc4e700 1 --2- 172.21.15.1:0/3880330860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d80cf960 0x7f43d8155170 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f43d0002230 tx=0x7f43d000cf50 comp rx=0 tx=0).stop 2024-03-18T19:09:51.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.261+0000 7f43dcc4e700 1 -- 172.21.15.1:0/3880330860 shutdown_connections 2024-03-18T19:09:51.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.261+0000 7f43dcc4e700 1 --2- 172.21.15.1:0/3880330860 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f43b806ae40 0x7f43b806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.261+0000 7f43dcc4e700 1 --2- 172.21.15.1:0/3880330860 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43d806a750 0x7f43d81458a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.261+0000 7f43dcc4e700 1 --2- 172.21.15.1:0/3880330860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d80cf960 0x7f43d8155170 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.261+0000 7f43dcc4e700 1 --2- 172.21.15.1:0/3880330860 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43d80cdda0 0x7f43d8154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.261+0000 7f43dcc4e700 1 -- 172.21.15.1:0/3880330860 >> 172.21.15.1:0/3880330860 conn(0x7f43d806b860 msgr2=0x7f43d80ce1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:51.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.261+0000 7f43dcc4e700 1 -- 172.21.15.1:0/3880330860 shutdown_connections 2024-03-18T19:09:51.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.261+0000 7f43dcc4e700 1 -- 172.21.15.1:0/3880330860 wait complete. 2024-03-18T19:09:51.259 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 65 2024-03-18T19:09:51.270 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:09:51.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.398+0000 7f59473f8700 1 Processor -- start 2024-03-18T19:09:51.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.399+0000 7f59473f8700 1 -- start start 2024-03-18T19:09:51.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.399+0000 7f59473f8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5940146190 0x7f594006a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:51.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.399+0000 7f59473f8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59401465a0 0x7f594006ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:51.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.399+0000 7f59473f8700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5940146af0 0x7f5940071650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:51.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.399+0000 7f59473f8700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594006f850 con 0x7f5940146190 2024-03-18T19:09:51.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.399+0000 7f59473f8700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594006ebd0 con 0x7f5940146af0 2024-03-18T19:09:51.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.399+0000 7f59473f8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594006eeb0 con 0x7f59401465a0 2024-03-18T19:09:51.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.400+0000 7f5945194700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5940146190 0x7f594006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:51.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.400+0000 7f5945194700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5940146190 0x7f594006a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44122/0 (socket says 172.21.15.1:44122) 2024-03-18T19:09:51.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.400+0000 7f5945194700 1 -- 172.21.15.1:0/2727167393 learned_addr learned my addr 172.21.15.1:0/2727167393 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:51.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.400+0000 7f5944993700 1 --2- 172.21.15.1:0/2727167393 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59401465a0 0x7f594006ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:51.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.400+0000 7f5945995700 1 --2- 172.21.15.1:0/2727167393 >> v2:172.21.15.165:3300/0 conn(0x7f5940146af0 0x7f5940071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:51.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.400+0000 7f5945194700 1 -- 172.21.15.1:0/2727167393 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59401465a0 msgr2=0x7f594006ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:51.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.400+0000 7f5945194700 1 --2- 172.21.15.1:0/2727167393 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59401465a0 0x7f594006ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.400+0000 7f5945194700 1 -- 172.21.15.1:0/2727167393 >> v2:172.21.15.165:3300/0 conn(0x7f5940146af0 msgr2=0x7f5940071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:51.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.400+0000 7f5945194700 1 --2- 172.21.15.1:0/2727167393 >> v2:172.21.15.165:3300/0 conn(0x7f5940146af0 0x7f5940071650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.400+0000 7f5945194700 1 -- 172.21.15.1:0/2727167393 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594006f190 con 0x7f5940146190 2024-03-18T19:09:51.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.400+0000 7f5945194700 1 --2- 172.21.15.1:0/2727167393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5940146190 0x7f594006a750 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f59400715e0 tx=0x7f5930009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb6a0c5750fbd9c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:51.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.400+0000 7f5937fff700 1 -- 172.21.15.1:0/2727167393 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593000b870 con 0x7f5940146190 2024-03-18T19:09:51.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.400+0000 7f5937fff700 1 -- 172.21.15.1:0/2727167393 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593000b9d0 con 0x7f5940146190 2024-03-18T19:09:51.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.400+0000 7f5937fff700 1 -- 172.21.15.1:0/2727167393 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593000bd60 con 0x7f5940146190 2024-03-18T19:09:51.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.401+0000 7f59473f8700 1 -- 172.21.15.1:0/2727167393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5940146190 msgr2=0x7f594006a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:51.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.401+0000 7f59473f8700 1 --2- 172.21.15.1:0/2727167393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5940146190 0x7f594006a750 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f59400715e0 tx=0x7f5930009bd0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.401+0000 7f59473f8700 1 -- 172.21.15.1:0/2727167393 shutdown_connections 2024-03-18T19:09:51.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.401+0000 7f59473f8700 1 --2- 172.21.15.1:0/2727167393 >> v2:172.21.15.165:3300/0 conn(0x7f5940146af0 0x7f5940071650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.401+0000 7f59473f8700 1 --2- 172.21.15.1:0/2727167393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5940146190 0x7f594006a750 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.401+0000 7f59473f8700 1 --2- 172.21.15.1:0/2727167393 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59401465a0 0x7f594006ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.401+0000 7f59473f8700 1 -- 172.21.15.1:0/2727167393 >> 172.21.15.1:0/2727167393 conn(0x7f59400b1d20 msgr2=0x7f59400b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:51.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.401+0000 7f59473f8700 1 -- 172.21.15.1:0/2727167393 shutdown_connections 2024-03-18T19:09:51.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.401+0000 7f59473f8700 1 -- 172.21.15.1:0/2727167393 wait complete. 2024-03-18T19:09:51.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.402+0000 7f59473f8700 1 Processor -- start 2024-03-18T19:09:51.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.402+0000 7f59473f8700 1 -- start start 2024-03-18T19:09:51.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.403+0000 7f59473f8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59401465a0 0x7f59401e88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:51.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.403+0000 7f59473f8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5940146af0 0x7f59401e2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:51.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.403+0000 7f59473f8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59401e3380 0x7f59401318b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:51.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.403+0000 7f59473f8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59401e3810 con 0x7f59401465a0 2024-03-18T19:09:51.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.403+0000 7f59473f8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5940144c50 con 0x7f59401e3380 2024-03-18T19:09:51.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.403+0000 7f59473f8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594015d610 con 0x7f5940146af0 2024-03-18T19:09:51.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.403+0000 7f5945194700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59401465a0 0x7f59401e88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:51.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.403+0000 7f5945194700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59401465a0 0x7f59401e88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44124/0 (socket says 172.21.15.1:44124) 2024-03-18T19:09:51.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.403+0000 7f5945194700 1 -- 172.21.15.1:0/191296338 learned_addr learned my addr 172.21.15.1:0/191296338 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:51.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.403+0000 7f5944993700 1 --2- 172.21.15.1:0/191296338 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5940146af0 0x7f59401e2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:51.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.403+0000 7f5945995700 1 --2- 172.21.15.1:0/191296338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59401e3380 0x7f59401318b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:51.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.403+0000 7f5945194700 1 -- 172.21.15.1:0/191296338 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5940146af0 msgr2=0x7f59401e2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:51.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.403+0000 7f5945194700 1 --2- 172.21.15.1:0/191296338 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5940146af0 0x7f59401e2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.403+0000 7f5945194700 1 -- 172.21.15.1:0/191296338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59401e3380 msgr2=0x7f59401318b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:51.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.403+0000 7f5945194700 1 --2- 172.21.15.1:0/191296338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59401e3380 0x7f59401318b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.403+0000 7f5945194700 1 -- 172.21.15.1:0/191296338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5930009550 con 0x7f59401465a0 2024-03-18T19:09:51.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.403+0000 7f5945194700 1 --2- 172.21.15.1:0/191296338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59401465a0 0x7f59401e88a0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f5930000c00 tx=0x7f5930004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:51.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.404+0000 7f5935ffb700 1 -- 172.21.15.1:0/191296338 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5930004480 con 0x7f59401465a0 2024-03-18T19:09:51.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.404+0000 7f5935ffb700 1 -- 172.21.15.1:0/191296338 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59300045e0 con 0x7f59401465a0 2024-03-18T19:09:51.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.404+0000 7f59473f8700 1 -- 172.21.15.1:0/191296338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59401e3a20 con 0x7f59401465a0 2024-03-18T19:09:51.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.404+0000 7f5935ffb700 1 -- 172.21.15.1:0/191296338 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59300038e0 con 0x7f59401465a0 2024-03-18T19:09:51.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.404+0000 7f59473f8700 1 -- 172.21.15.1:0/191296338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59401e3fe0 con 0x7f59401465a0 2024-03-18T19:09:51.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.405+0000 7f59473f8700 1 -- 172.21.15.1:0/191296338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59400cc760 con 0x7f59401465a0 2024-03-18T19:09:51.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.405+0000 7f5935ffb700 1 -- 172.21.15.1:0/191296338 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 79) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f5930021070 con 0x7f59401465a0 2024-03-18T19:09:51.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.406+0000 7f5935ffb700 1 -- 172.21.15.1:0/191296338 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6702+0+0 (secure 0 0 0) 0x7f593008a680 con 0x7f59401465a0 2024-03-18T19:09:51.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.410+0000 7f5935ffb700 1 -- 172.21.15.1:0/191296338 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f59300558d0 con 0x7f59401465a0 2024-03-18T19:09:51.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.613+0000 7f59473f8700 1 -- 172.21.15.1:0/191296338 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59400676a0 con 0x7f59401465a0 2024-03-18T19:09:51.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.614+0000 7f5935ffb700 1 -- 172.21.15.1:0/191296338 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+17216 (secure 0 0 0) 0x7f5930058ef0 con 0x7f59401465a0 2024-03-18T19:09:51.610 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:51.610 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":149,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:09:51.065705+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:09:20.468744+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-18T19:09:21.597476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/1268717803":"2024-03-19T19:09:51.065667+0000","172.21.15.1:0/4096703182":"2024-03-19T19:09:51.065667+0000","172.21.15.1:0/3827666525":"2024-03-19T19:09:51.065667+0000","172.21.15.1:0/382823491":"2024-03-19T19:09:51.065667+0000","172.21.15.1:0/2890345743":"2024-03-19T19:09:51.065667+0000","172.21.15.1:0/3308101132":"2024-03-19T19:09:51.065667+0000","172.21.15.1:0/69006586":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/3167978554":"2024-03-19T19:09:51.065667+0000","172.21.15.1:6833/76998":"2024-03-19T19:09:37.850642+0000","172.21.15.1:6832/76998":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/1403689662":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/806972823":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/619565916":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/2042700873":"2024-03-19T19:09:37.850642+0000","172.21.15.1:0/744112928":"2024-03-19T19:09:37.850642+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:09:51.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.616+0000 7f59473f8700 1 -- 172.21.15.1:0/191296338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59401465a0 msgr2=0x7f59401e88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:51.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.616+0000 7f59473f8700 1 --2- 172.21.15.1:0/191296338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59401465a0 0x7f59401e88a0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f5930000c00 tx=0x7f5930004750 comp rx=0 tx=0).stop 2024-03-18T19:09:51.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.617+0000 7f59473f8700 1 -- 172.21.15.1:0/191296338 shutdown_connections 2024-03-18T19:09:51.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.617+0000 7f59473f8700 1 --2- 172.21.15.1:0/191296338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59401e3380 0x7f59401318b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.617+0000 7f59473f8700 1 --2- 172.21.15.1:0/191296338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59401465a0 0x7f59401e88a0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.617+0000 7f59473f8700 1 --2- 172.21.15.1:0/191296338 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5940146af0 0x7f59401e2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.617+0000 7f59473f8700 1 -- 172.21.15.1:0/191296338 >> 172.21.15.1:0/191296338 conn(0x7f59400b1d20 msgr2=0x7f59400b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:51.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.617+0000 7f59473f8700 1 -- 172.21.15.1:0/191296338 shutdown_connections 2024-03-18T19:09:51.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.617+0000 7f59473f8700 1 -- 172.21.15.1:0/191296338 wait complete. 2024-03-18T19:09:51.625 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T19:09:51.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.755+0000 7f859de81700 1 Processor -- start 2024-03-18T19:09:51.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.755+0000 7f859de81700 1 -- start start 2024-03-18T19:09:51.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.755+0000 7f859de81700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85980cd870 0x7f85980cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.755+0000 7f859de81700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85980d0df0 con 0x7f85980cbce0 2024-03-18T19:09:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.755+0000 7f859de81700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85980d0550 con 0x7f85980b1e20 2024-03-18T19:09:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.755+0000 7f859de81700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85980d0830 con 0x7f85980cd870 2024-03-18T19:09:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.755+0000 7f8597fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f85980cbce0 0x7f85980cc170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59572/0 (socket says 172.21.15.1:59572) 2024-03-18T19:09:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f8597fff700 1 -- 172.21.15.1:0/512910598 learned_addr learned my addr 172.21.15.1:0/512910598 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f85977fe700 1 --2- 172.21.15.1:0/512910598 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85980cd870 0x7f85980cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f85967fc700 1 -- 172.21.15.1:0/512910598 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860010363 0 0) 0x7f85980d0df0 con 0x7f85980cbce0 2024-03-18T19:09:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f85967fc700 1 -- 172.21.15.1:0/512910598 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85800035b0 con 0x7f85980cbce0 2024-03-18T19:09:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f85967fc700 1 -- 172.21.15.1:0/512910598 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 195042964 0 0) 0x7f85800035b0 con 0x7f85980cbce0 2024-03-18T19:09:51.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f85967fc700 1 -- 172.21.15.1:0/512910598 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85980cd870 msgr2=0x7f85980cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f85967fc700 1 --2- 172.21.15.1:0/512910598 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85980cd870 0x7f85980cff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f85967fc700 1 -- 172.21.15.1:0/512910598 >> v1:172.21.15.165:6789/0 conn(0x7f85980b1e20 legacy=0x7f85980cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f85967fc700 1 -- 172.21.15.1:0/512910598 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85980d0b10 con 0x7f85980cbce0 2024-03-18T19:09:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f85967fc700 1 -- 172.21.15.1:0/512910598 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f85900019f0 con 0x7f85980cbce0 2024-03-18T19:09:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f85967fc700 1 -- 172.21.15.1:0/512910598 >> v1:172.21.15.1:6789/0 conn(0x7f85980cbce0 legacy=0x7f85980cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f85967fc700 1 --2- 172.21.15.1:0/512910598 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85980cd870 0x7f8580003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f85967fc700 1 --2- 172.21.15.1:0/512910598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8580003fb0 0x7f8580004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f85967fc700 1 --2- 172.21.15.1:0/512910598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8580004d90 0x7f8580005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f85967fc700 1 -- 172.21.15.1:0/512910598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85980d0830 con 0x7f8580004d90 2024-03-18T19:09:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f85967fc700 1 -- 172.21.15.1:0/512910598 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858000baf0 con 0x7f8580003fb0 2024-03-18T19:09:51.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f85967fc700 1 -- 172.21.15.1:0/512910598 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f858000bcb0 con 0x7f85980cd870 2024-03-18T19:09:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f8597fff700 1 --2- 172.21.15.1:0/512910598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8580004d90 0x7f8580005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f85977fe700 1 --2- 172.21.15.1:0/512910598 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85980cd870 0x7f8580003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.756+0000 7f8596ffd700 1 --2- 172.21.15.1:0/512910598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8580003fb0 0x7f8580004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.757+0000 7f8597fff700 1 -- 172.21.15.1:0/512910598 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85980cd870 msgr2=0x7f8580003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.757+0000 7f8597fff700 1 --2- 172.21.15.1:0/512910598 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85980cd870 0x7f8580003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.757+0000 7f8597fff700 1 -- 172.21.15.1:0/512910598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8580003fb0 msgr2=0x7f8580004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.757+0000 7f8597fff700 1 --2- 172.21.15.1:0/512910598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8580003fb0 0x7f8580004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.757+0000 7f8597fff700 1 -- 172.21.15.1:0/512910598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85800035b0 con 0x7f8580004d90 2024-03-18T19:09:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.757+0000 7f8597fff700 1 --2- 172.21.15.1:0/512910598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8580004d90 0x7f8580005210 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f859000a830 tx=0x7f8580005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab69f446c4c1f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.757+0000 7f85967fc700 1 -- 172.21.15.1:0/512910598 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859000a2f0 con 0x7f8580004d90 2024-03-18T19:09:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.757+0000 7f85967fc700 1 -- 172.21.15.1:0/512910598 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8590012620 con 0x7f8580004d90 2024-03-18T19:09:51.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.757+0000 7f85967fc700 1 -- 172.21.15.1:0/512910598 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85900129d0 con 0x7f8580004d90 2024-03-18T19:09:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.757+0000 7f859de81700 1 -- 172.21.15.1:0/512910598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8580004d90 msgr2=0x7f8580005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.757+0000 7f859de81700 1 --2- 172.21.15.1:0/512910598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8580004d90 0x7f8580005210 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f859000a830 tx=0x7f8580005880 comp rx=0 tx=0).stop 2024-03-18T19:09:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.757+0000 7f8597fff700 1 -- 172.21.15.1:0/512910598 reap_dead start 2024-03-18T19:09:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.757+0000 7f859de81700 1 -- 172.21.15.1:0/512910598 shutdown_connections 2024-03-18T19:09:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.757+0000 7f859de81700 1 -- 172.21.15.1:0/512910598 >> 172.21.15.1:0/512910598 conn(0x7f85980d0ff0 msgr2=0x7f85980d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.757+0000 7f859de81700 1 -- 172.21.15.1:0/512910598 shutdown_connections 2024-03-18T19:09:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.758+0000 7f859de81700 1 -- 172.21.15.1:0/512910598 wait complete. 2024-03-18T19:09:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.759+0000 7f859de81700 1 Processor -- start 2024-03-18T19:09:51.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.759+0000 7f859de81700 1 -- start start 2024-03-18T19:09:51.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f859de81700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8580004d90 0x7f8598156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:51.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f859de81700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f859806c8b0 0x7f85981f3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:51.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f859de81700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8598157660 0x7f85981589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:51.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f859de81700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859815b270 con 0x7f859806c8b0 2024-03-18T19:09:51.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f859de81700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85981596b0 con 0x7f8598157660 2024-03-18T19:09:51.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f859de81700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8598157110 con 0x7f8580004d90 2024-03-18T19:09:51.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f8596ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f859806c8b0 0x7f85981f3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:51.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f8596ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f859806c8b0 0x7f85981f3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44132/0 (socket says 172.21.15.1:44132) 2024-03-18T19:09:51.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f8596ffd700 1 -- 172.21.15.1:0/3590694667 learned_addr learned my addr 172.21.15.1:0/3590694667 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:51.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f85977fe700 1 --2- 172.21.15.1:0/3590694667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8580004d90 0x7f8598156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:51.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f8597fff700 1 --2- 172.21.15.1:0/3590694667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8598157660 0x7f85981589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:51.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f8596ffd700 1 -- 172.21.15.1:0/3590694667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8580004d90 msgr2=0x7f8598156b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:51.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f8596ffd700 1 --2- 172.21.15.1:0/3590694667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8580004d90 0x7f8598156b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f8596ffd700 1 -- 172.21.15.1:0/3590694667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8598157660 msgr2=0x7f85981589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:51.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f8596ffd700 1 --2- 172.21.15.1:0/3590694667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8598157660 0x7f85981589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:51.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f8596ffd700 1 -- 172.21.15.1:0/3590694667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8598157360 con 0x7f859806c8b0 2024-03-18T19:09:51.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f85977fe700 1 --2- 172.21.15.1:0/3590694667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8580004d90 0x7f8598156b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:51.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f8597fff700 1 --2- 172.21.15.1:0/3590694667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8598157660 0x7f85981589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:51.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f8596ffd700 1 --2- 172.21.15.1:0/3590694667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f859806c8b0 0x7f85981f3510 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f858c00f7f0 tx=0x7f858c00fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:51.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f857ffff700 1 -- 172.21.15.1:0/3590694667 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858c0052e0 con 0x7f859806c8b0 2024-03-18T19:09:51.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f857ffff700 1 -- 172.21.15.1:0/3590694667 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f858c005440 con 0x7f859806c8b0 2024-03-18T19:09:51.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f859de81700 1 -- 172.21.15.1:0/3590694667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f859815af90 con 0x7f859806c8b0 2024-03-18T19:09:51.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f857ffff700 1 -- 172.21.15.1:0/3590694667 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858c0038c0 con 0x7f859806c8b0 2024-03-18T19:09:51.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.760+0000 7f859de81700 1 -- 172.21.15.1:0/3590694667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85981f3a50 con 0x7f859806c8b0 2024-03-18T19:09:51.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.761+0000 7f859de81700 1 -- 172.21.15.1:0/3590694667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85980c8ad0 con 0x7f859806c8b0 2024-03-18T19:09:51.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.762+0000 7f857ffff700 1 -- 172.21.15.1:0/3590694667 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 79) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f858c003a20 con 0x7f859806c8b0 2024-03-18T19:09:51.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.762+0000 7f857ffff700 1 -- 172.21.15.1:0/3590694667 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6702+0+0 (secure 0 0 0) 0x7f858c089000 con 0x7f859806c8b0 2024-03-18T19:09:51.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:51.767+0000 7f857ffff700 1 -- 172.21.15.1:0/3590694667 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f85980c8ad0 con 0x7f859806c8b0 2024-03-18T19:09:52.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.016+0000 7f859de81700 1 -- 172.21.15.1:0/3590694667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f85980c8eb0 con 0x7f859806c8b0 2024-03-18T19:09:52.012 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-18T19:09:52.017+0000 7fbdbeb46700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T19:09:52.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.023+0000 7f857ffff700 1 -- 172.21.15.1:0/3590694667 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v66) v1 ==== 161+0+0 (secure 0 0 0) 0x7f85980c8eb0 con 0x7f859806c8b0 2024-03-18T19:09:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.025+0000 7f859de81700 1 -- 172.21.15.1:0/3590694667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f859806c8b0 msgr2=0x7f85981f3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.025+0000 7f859de81700 1 --2- 172.21.15.1:0/3590694667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f859806c8b0 0x7f85981f3510 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f858c00f7f0 tx=0x7f858c00fbb0 comp rx=0 tx=0).stop 2024-03-18T19:09:52.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.025+0000 7f859de81700 1 -- 172.21.15.1:0/3590694667 shutdown_connections 2024-03-18T19:09:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.025+0000 7f859de81700 1 --2- 172.21.15.1:0/3590694667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8580004d90 0x7f8598156b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.025+0000 7f859de81700 1 --2- 172.21.15.1:0/3590694667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8598157660 0x7f85981589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.025+0000 7f859de81700 1 --2- 172.21.15.1:0/3590694667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f859806c8b0 0x7f85981f3510 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.025+0000 7f859de81700 1 -- 172.21.15.1:0/3590694667 >> 172.21.15.1:0/3590694667 conn(0x7f85980d0ff0 msgr2=0x7f85980c4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.025+0000 7f859de81700 1 -- 172.21.15.1:0/3590694667 shutdown_connections 2024-03-18T19:09:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.025+0000 7f859de81700 1 -- 172.21.15.1:0/3590694667 wait complete. 2024-03-18T19:09:52.022 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T19:09:52.024 INFO:tasks.ceph.mds.b.smithi165.stderr: -8614> 2024-03-18T19:09:28.185+0000 7fdf146d4700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T19:09:52.033 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T19:09:52.059 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-03-18T19:09:52.060 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:09:52.079 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:09:52.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.171+0000 7fb4e2198700 1 Processor -- start 2024-03-18T19:09:52.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.171+0000 7fb4e2198700 1 -- start start 2024-03-18T19:09:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.173+0000 7fb4e2198700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb4dc0ade00 0x7fb4dc0cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.173+0000 7fb4e2198700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4dc0b3aa0 con 0x7fb4dc0ccc20 2024-03-18T19:09:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.173+0000 7fb4e2198700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4dc0ae280 con 0x7fb4dc099a40 2024-03-18T19:09:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.173+0000 7fb4e2198700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4dc06e9b0 con 0x7fb4dc0ade00 2024-03-18T19:09:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.173+0000 7fb4daffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb4dc0ade00 0x7fb4dc0cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:52.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.173+0000 7fb4dbfff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb4dc0ccc20 0x7fb4dc06fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59588/0 (socket says 172.21.15.1:59588) 2024-03-18T19:09:52.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.173+0000 7fb4dbfff700 1 -- 172.21.15.1:0/468996918 learned_addr learned my addr 172.21.15.1:0/468996918 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:52.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.173+0000 7fb4daffd700 1 -- 172.21.15.1:0/468996918 >> v1:172.21.15.165:6789/0 conn(0x7fb4dc099a40 legacy=0x7fb4dc099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:52.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.173+0000 7fb4daffd700 1 -- 172.21.15.1:0/468996918 >> v1:172.21.15.1:6789/0 conn(0x7fb4dc0ccc20 legacy=0x7fb4dc06fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:52.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.173+0000 7fb4daffd700 1 -- 172.21.15.1:0/468996918 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc06ec90 con 0x7fb4dc0ade00 2024-03-18T19:09:52.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.173+0000 7fb4da7fc700 1 -- 172.21.15.1:0/468996918 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990522494 0 0) 0x7fb4dc0b3aa0 con 0x7fb4dc0ccc20 2024-03-18T19:09:52.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.173+0000 7fb4daffd700 1 --2- 172.21.15.1:0/468996918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4dc0ade00 0x7fb4dc0cc6e0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc009840 tx=0x7fb4cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2a98339c32a15c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:52.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.173+0000 7fb4da7fc700 1 -- 172.21.15.1:0/468996918 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc011830 con 0x7fb4dc0ade00 2024-03-18T19:09:52.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.173+0000 7fb4da7fc700 1 -- 172.21.15.1:0/468996918 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4cc011990 con 0x7fb4dc0ade00 2024-03-18T19:09:52.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.173+0000 7fb4da7fc700 1 -- 172.21.15.1:0/468996918 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc011d20 con 0x7fb4dc0ade00 2024-03-18T19:09:52.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.174+0000 7fb4e2198700 1 -- 172.21.15.1:0/468996918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4dc0ade00 msgr2=0x7fb4dc0cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:52.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.174+0000 7fb4e2198700 1 --2- 172.21.15.1:0/468996918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4dc0ade00 0x7fb4dc0cc6e0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc009840 tx=0x7fb4cc009b50 comp rx=0 tx=0).stop 2024-03-18T19:09:52.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.174+0000 7fb4e2198700 1 -- 172.21.15.1:0/468996918 shutdown_connections 2024-03-18T19:09:52.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.174+0000 7fb4e2198700 1 --2- 172.21.15.1:0/468996918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4dc0ade00 0x7fb4dc0cc6e0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:52.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.174+0000 7fb4e2198700 1 -- 172.21.15.1:0/468996918 >> 172.21.15.1:0/468996918 conn(0x7fb4dc0b2c50 msgr2=0x7fb4dc0b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:52.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.174+0000 7fb4e2198700 1 -- 172.21.15.1:0/468996918 shutdown_connections 2024-03-18T19:09:52.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.174+0000 7fb4e2198700 1 -- 172.21.15.1:0/468996918 wait complete. 2024-03-18T19:09:52.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.175+0000 7fb4e2198700 1 Processor -- start 2024-03-18T19:09:52.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.175+0000 7fb4e2198700 1 -- start start 2024-03-18T19:09:52.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.176+0000 7fb4e2198700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4dc099a40 0x7fb4dc0c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:52.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.176+0000 7fb4e2198700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4dc0ade00 0x7fb4dc1598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:52.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.176+0000 7fb4e2198700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4dc0ccc20 0x7fb4dc0af670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:52.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.176+0000 7fb4e2198700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc15a950 con 0x7fb4dc0ccc20 2024-03-18T19:09:52.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.176+0000 7fb4e2198700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc0c7740 con 0x7fb4dc099a40 2024-03-18T19:09:52.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.176+0000 7fb4e2198700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc1e48c0 con 0x7fb4dc0ade00 2024-03-18T19:09:52.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.176+0000 7fb4dbfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4dc0ccc20 0x7fb4dc0af670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:52.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.176+0000 7fb4dbfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4dc0ccc20 0x7fb4dc0af670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44152/0 (socket says 172.21.15.1:44152) 2024-03-18T19:09:52.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.176+0000 7fb4dbfff700 1 -- 172.21.15.1:0/3769677923 learned_addr learned my addr 172.21.15.1:0/3769677923 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:52.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.176+0000 7fb4db7fe700 1 --2- 172.21.15.1:0/3769677923 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4dc099a40 0x7fb4dc0c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:52.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.176+0000 7fb4daffd700 1 --2- 172.21.15.1:0/3769677923 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4dc0ade00 0x7fb4dc1598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:52.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.177+0000 7fb4dbfff700 1 -- 172.21.15.1:0/3769677923 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4dc0ade00 msgr2=0x7fb4dc1598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.177+0000 7fb4dbfff700 1 --2- 172.21.15.1:0/3769677923 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4dc0ade00 0x7fb4dc1598a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.177+0000 7fb4dbfff700 1 -- 172.21.15.1:0/3769677923 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4dc099a40 msgr2=0x7fb4dc0c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.177+0000 7fb4dbfff700 1 --2- 172.21.15.1:0/3769677923 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4dc099a40 0x7fb4dc0c3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.177+0000 7fb4dbfff700 1 -- 172.21.15.1:0/3769677923 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc0094a0 con 0x7fb4dc0ccc20 2024-03-18T19:09:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.177+0000 7fb4dbfff700 1 --2- 172.21.15.1:0/3769677923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4dc0ccc20 0x7fb4dc0af670 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4002230 tx=0x7fb4d400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.177+0000 7fb4d3fff700 1 -- 172.21.15.1:0/3769677923 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d400aac0 con 0x7fb4dc0ccc20 2024-03-18T19:09:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.177+0000 7fb4d3fff700 1 -- 172.21.15.1:0/3769677923 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d400ac40 con 0x7fb4dc0ccc20 2024-03-18T19:09:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.177+0000 7fb4e2198700 1 -- 172.21.15.1:0/3769677923 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4dc0c72b0 con 0x7fb4dc0ccc20 2024-03-18T19:09:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.177+0000 7fb4d3fff700 1 -- 172.21.15.1:0/3769677923 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d40070d0 con 0x7fb4dc0ccc20 2024-03-18T19:09:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.177+0000 7fb4e2198700 1 -- 172.21.15.1:0/3769677923 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4dc1e4df0 con 0x7fb4dc0ccc20 2024-03-18T19:09:52.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.178+0000 7fb4e2198700 1 -- 172.21.15.1:0/3769677923 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4dc0c7440 con 0x7fb4dc0ccc20 2024-03-18T19:09:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.179+0000 7fb4d3fff700 1 -- 172.21.15.1:0/3769677923 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 80) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fb4d4007250 con 0x7fb4dc0ccc20 2024-03-18T19:09:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.179+0000 7fb4d3fff700 1 --2- 172.21.15.1:0/3769677923 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb4bc06aae0 0x7fb4bc06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.179+0000 7fb4d3fff700 1 -- 172.21.15.1:0/3769677923 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6788+0+0 (secure 0 0 0) 0x7fb4d4089110 con 0x7fb4dc0ccc20 2024-03-18T19:09:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.179+0000 7fb4db7fe700 1 --2- 172.21.15.1:0/3769677923 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb4bc06aae0 0x7fb4bc06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:52.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.179+0000 7fb4db7fe700 1 --2- 172.21.15.1:0/3769677923 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb4bc06aae0 0x7fb4bc06cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4007fd0 tx=0x7fb4c4009040 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.184+0000 7fb4d3fff700 1 -- 172.21.15.1:0/3769677923 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb4d40541f0 con 0x7fb4dc0ccc20 2024-03-18T19:09:52.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:52.432+0000 7fb4e2198700 1 -- 172.21.15.1:0/3769677923 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb4dc0676a0 con 0x7fb4dc0ccc20 2024-03-18T19:09:53.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.082+0000 7fb4d3fff700 1 -- 172.21.15.1:0/3769677923 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v67) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb4d4057810 con 0x7fb4dc0ccc20 2024-03-18T19:09:53.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.085+0000 7fb4e2198700 1 -- 172.21.15.1:0/3769677923 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb4bc06aae0 msgr2=0x7fb4bc06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:53.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.085+0000 7fb4e2198700 1 --2- 172.21.15.1:0/3769677923 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb4bc06aae0 0x7fb4bc06cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4007fd0 tx=0x7fb4c4009040 comp rx=0 tx=0).stop 2024-03-18T19:09:53.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.085+0000 7fb4e2198700 1 -- 172.21.15.1:0/3769677923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4dc0ccc20 msgr2=0x7fb4dc0af670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:53.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.085+0000 7fb4e2198700 1 --2- 172.21.15.1:0/3769677923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4dc0ccc20 0x7fb4dc0af670 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4002230 tx=0x7fb4d400cf50 comp rx=0 tx=0).stop 2024-03-18T19:09:53.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.085+0000 7fb4e2198700 1 -- 172.21.15.1:0/3769677923 shutdown_connections 2024-03-18T19:09:53.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.085+0000 7fb4e2198700 1 --2- 172.21.15.1:0/3769677923 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb4bc06aae0 0x7fb4bc06cfa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:53.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.085+0000 7fb4e2198700 1 --2- 172.21.15.1:0/3769677923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4dc0ccc20 0x7fb4dc0af670 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:53.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.085+0000 7fb4e2198700 1 --2- 172.21.15.1:0/3769677923 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4dc099a40 0x7fb4dc0c3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:53.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.085+0000 7fb4e2198700 1 --2- 172.21.15.1:0/3769677923 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4dc0ade00 0x7fb4dc1598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:53.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.085+0000 7fb4e2198700 1 -- 172.21.15.1:0/3769677923 >> 172.21.15.1:0/3769677923 conn(0x7fb4dc0b2c50 msgr2=0x7fb4dc15d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:53.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.085+0000 7fb4e2198700 1 -- 172.21.15.1:0/3769677923 shutdown_connections 2024-03-18T19:09:53.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.085+0000 7fb4e2198700 1 -- 172.21.15.1:0/3769677923 wait complete. 2024-03-18T19:09:53.096 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T19:09:53.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.273+0000 7f030de0e700 1 Processor -- start 2024-03-18T19:09:53.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.273+0000 7f030de0e700 1 -- start start 2024-03-18T19:09:53.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.274+0000 7f030de0e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f030815dca0 0x7f030815a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:53.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.274+0000 7f030de0e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030815ac60 con 0x7f03080717b0 2024-03-18T19:09:53.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.274+0000 7f030de0e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03080b30b0 con 0x7f030815e8d0 2024-03-18T19:09:53.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.274+0000 7f030de0e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03080701d0 con 0x7f030815dca0 2024-03-18T19:09:53.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.275+0000 7f0307fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f03080717b0 0x7f03080b45b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59594/0 (socket says 172.21.15.1:59594) 2024-03-18T19:09:53.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.275+0000 7f0307fff700 1 -- 172.21.15.1:0/3810029795 learned_addr learned my addr 172.21.15.1:0/3810029795 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:53.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.275+0000 7f0306ffd700 1 --2- 172.21.15.1:0/3810029795 >> v2:172.21.15.165:3300/0 conn(0x7f030815dca0 0x7f030815a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:53.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.275+0000 7f03067fc700 1 -- 172.21.15.1:0/3810029795 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013187997 0 0) 0x7f030815ac60 con 0x7f03080717b0 2024-03-18T19:09:53.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.275+0000 7f03067fc700 1 -- 172.21.15.1:0/3810029795 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f00035b0 con 0x7f03080717b0 2024-03-18T19:09:53.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.275+0000 7f03067fc700 1 -- 172.21.15.1:0/3810029795 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293209363 0 0) 0x7f03080b30b0 con 0x7f030815e8d0 2024-03-18T19:09:53.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.275+0000 7f03067fc700 1 -- 172.21.15.1:0/3810029795 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f030815ac60 con 0x7f030815e8d0 2024-03-18T19:09:53.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.275+0000 7f03067fc700 1 -- 172.21.15.1:0/3810029795 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3524819175 0 0) 0x7f02f00035b0 con 0x7f03080717b0 2024-03-18T19:09:53.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.275+0000 7f0306ffd700 1 -- 172.21.15.1:0/3810029795 >> v1:172.21.15.165:6789/0 conn(0x7f030815e8d0 legacy=0x7f03080b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:53.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.275+0000 7f0306ffd700 1 -- 172.21.15.1:0/3810029795 >> v1:172.21.15.1:6789/0 conn(0x7f03080717b0 legacy=0x7f03080b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:53.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.275+0000 7f0306ffd700 1 -- 172.21.15.1:0/3810029795 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03080b3390 con 0x7f030815dca0 2024-03-18T19:09:53.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.275+0000 7f0306ffd700 1 --2- 172.21.15.1:0/3810029795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f030815dca0 0x7f030815a720 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f02fc009870 tx=0x7f02fc009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49d1ee61c3ac1f00 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:53.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.276+0000 7f03067fc700 1 -- 172.21.15.1:0/3810029795 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03000019f0 con 0x7f030815dca0 2024-03-18T19:09:53.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.276+0000 7f03067fc700 1 -- 172.21.15.1:0/3810029795 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc004030 con 0x7f030815dca0 2024-03-18T19:09:53.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.276+0000 7f03067fc700 1 -- 172.21.15.1:0/3810029795 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc0043c0 con 0x7f030815dca0 2024-03-18T19:09:53.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.276+0000 7f030de0e700 1 -- 172.21.15.1:0/3810029795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f030815dca0 msgr2=0x7f030815a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:53.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.276+0000 7f030de0e700 1 --2- 172.21.15.1:0/3810029795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f030815dca0 0x7f030815a720 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f02fc009870 tx=0x7f02fc009b80 comp rx=0 tx=0).stop 2024-03-18T19:09:53.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.276+0000 7f030de0e700 1 -- 172.21.15.1:0/3810029795 shutdown_connections 2024-03-18T19:09:53.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.276+0000 7f030de0e700 1 --2- 172.21.15.1:0/3810029795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f030815dca0 0x7f030815a720 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:53.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.276+0000 7f030de0e700 1 -- 172.21.15.1:0/3810029795 >> 172.21.15.1:0/3810029795 conn(0x7f03080d0080 msgr2=0x7f03080d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:53.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.276+0000 7f030de0e700 1 -- 172.21.15.1:0/3810029795 shutdown_connections 2024-03-18T19:09:53.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.276+0000 7f030de0e700 1 -- 172.21.15.1:0/3810029795 wait complete. 2024-03-18T19:09:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.278+0000 7f030de0e700 1 Processor -- start 2024-03-18T19:09:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.278+0000 7f030de0e700 1 -- start start 2024-03-18T19:09:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.279+0000 7f030de0e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03080717b0 0x7f0308138fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.279+0000 7f030de0e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f030815dca0 0x7f030813c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.279+0000 7f030de0e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f030815e8d0 0x7f0308153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.279+0000 7f030de0e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0308139510 con 0x7f030815e8d0 2024-03-18T19:09:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.279+0000 7f030de0e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030813c7b0 con 0x7f030815dca0 2024-03-18T19:09:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.279+0000 7f030de0e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03081535d0 con 0x7f03080717b0 2024-03-18T19:09:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.279+0000 7f0307fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f030815e8d0 0x7f0308153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.279+0000 7f0307fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f030815e8d0 0x7f0308153090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44180/0 (socket says 172.21.15.1:44180) 2024-03-18T19:09:53.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.279+0000 7f0307fff700 1 -- 172.21.15.1:0/3246979929 learned_addr learned my addr 172.21.15.1:0/3246979929 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:53.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.279+0000 7f0306ffd700 1 --2- 172.21.15.1:0/3246979929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f030815dca0 0x7f030813c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:53.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.279+0000 7f03077fe700 1 --2- 172.21.15.1:0/3246979929 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03080717b0 0x7f0308138fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:53.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.279+0000 7f0307fff700 1 -- 172.21.15.1:0/3246979929 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03080717b0 msgr2=0x7f0308138fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:53.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.279+0000 7f0307fff700 1 --2- 172.21.15.1:0/3246979929 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03080717b0 0x7f0308138fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:53.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.279+0000 7f0307fff700 1 -- 172.21.15.1:0/3246979929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f030815dca0 msgr2=0x7f030813c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:53.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.279+0000 7f0307fff700 1 --2- 172.21.15.1:0/3246979929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f030815dca0 0x7f030813c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:53.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.279+0000 7f0307fff700 1 -- 172.21.15.1:0/3246979929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02fc0094a0 con 0x7f030815e8d0 2024-03-18T19:09:53.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.279+0000 7f03077fe700 1 --2- 172.21.15.1:0/3246979929 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03080717b0 0x7f0308138fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:53.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.279+0000 7f0307fff700 1 --2- 172.21.15.1:0/3246979929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f030815e8d0 0x7f0308153090 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f03000021d0 tx=0x7f030000c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:53.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.280+0000 7f02effff700 1 -- 172.21.15.1:0/3246979929 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030000cdc0 con 0x7f030815e8d0 2024-03-18T19:09:53.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.280+0000 7f02effff700 1 -- 172.21.15.1:0/3246979929 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03000070d0 con 0x7f030815e8d0 2024-03-18T19:09:53.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.280+0000 7f030de0e700 1 -- 172.21.15.1:0/3246979929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0308153820 con 0x7f030815e8d0 2024-03-18T19:09:53.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.280+0000 7f02effff700 1 -- 172.21.15.1:0/3246979929 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0300007480 con 0x7f030815e8d0 2024-03-18T19:09:53.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.280+0000 7f030de0e700 1 -- 172.21.15.1:0/3246979929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0308153de0 con 0x7f030815e8d0 2024-03-18T19:09:53.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.281+0000 7f030de0e700 1 -- 172.21.15.1:0/3246979929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03080676a0 con 0x7f030815e8d0 2024-03-18T19:09:53.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.281+0000 7f02effff700 1 -- 172.21.15.1:0/3246979929 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 80) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f03000075e0 con 0x7f030815e8d0 2024-03-18T19:09:53.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.281+0000 7f02effff700 1 --2- 172.21.15.1:0/3246979929 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f02f406aae0 0x7f02f406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:53.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.282+0000 7f02effff700 1 -- 172.21.15.1:0/3246979929 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6788+0+0 (secure 0 0 0) 0x7f030008c3d0 con 0x7f030815e8d0 2024-03-18T19:09:53.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.282+0000 7f03077fe700 1 --2- 172.21.15.1:0/3246979929 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f02f406aae0 0x7f02f406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:53.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.282+0000 7f03077fe700 1 --2- 172.21.15.1:0/3246979929 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f02f406aae0 0x7f02f406cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f02f80011e0 tx=0x7f02f8009040 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:53.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.286+0000 7f02effff700 1 -- 172.21.15.1:0/3246979929 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0300057430 con 0x7f030815e8d0 2024-03-18T19:09:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:53.489+0000 7f030de0e700 1 -- 172.21.15.1:0/3246979929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f03080d0f70 con 0x7f030815e8d0 2024-03-18T19:09:54.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.086+0000 7f02effff700 1 -- 172.21.15.1:0/3246979929 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 81) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f030008c120 con 0x7f030815e8d0 2024-03-18T19:09:54.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.145+0000 7f02effff700 1 -- 172.21.15.1:0/3246979929 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v151) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0300086a90 con 0x7f030815e8d0 2024-03-18T19:09:54.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.151+0000 7f030de0e700 1 -- 172.21.15.1:0/3246979929 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f02f406aae0 msgr2=0x7f02f406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:54.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.151+0000 7f030de0e700 1 --2- 172.21.15.1:0/3246979929 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f02f406aae0 0x7f02f406cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f02f80011e0 tx=0x7f02f8009040 comp rx=0 tx=0).stop 2024-03-18T19:09:54.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.151+0000 7f030de0e700 1 -- 172.21.15.1:0/3246979929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f030815e8d0 msgr2=0x7f0308153090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:54.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.151+0000 7f030de0e700 1 --2- 172.21.15.1:0/3246979929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f030815e8d0 0x7f0308153090 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f03000021d0 tx=0x7f030000c550 comp rx=0 tx=0).stop 2024-03-18T19:09:54.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.151+0000 7f030de0e700 1 -- 172.21.15.1:0/3246979929 shutdown_connections 2024-03-18T19:09:54.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.151+0000 7f030de0e700 1 --2- 172.21.15.1:0/3246979929 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f02f406aae0 0x7f02f406cfa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:54.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.151+0000 7f030de0e700 1 --2- 172.21.15.1:0/3246979929 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f03080717b0 0x7f0308138fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.151+0000 7f030de0e700 1 --2- 172.21.15.1:0/3246979929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f030815e8d0 0x7f0308153090 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.151+0000 7f030de0e700 1 --2- 172.21.15.1:0/3246979929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f030815dca0 0x7f030813c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.151+0000 7f030de0e700 1 -- 172.21.15.1:0/3246979929 >> 172.21.15.1:0/3246979929 conn(0x7f03080d0080 msgr2=0x7f03080cc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.151+0000 7f030de0e700 1 -- 172.21.15.1:0/3246979929 shutdown_connections 2024-03-18T19:09:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.151+0000 7f030de0e700 1 -- 172.21.15.1:0/3246979929 wait complete. 2024-03-18T19:09:54.148 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-18T19:09:54.162 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T19:09:54.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.302+0000 7fe688a13700 1 Processor -- start 2024-03-18T19:09:54.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.302+0000 7fe688a13700 1 -- start start 2024-03-18T19:09:54.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.303+0000 7fe688a13700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe6840ae5b0 0x7fe684168120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:54.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.303+0000 7fe688a13700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe68415d950 con 0x7fe6840cfe80 2024-03-18T19:09:54.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.303+0000 7fe688a13700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe68406df50 con 0x7fe68415d570 2024-03-18T19:09:54.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.303+0000 7fe688a13700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6840ac8b0 con 0x7fe6840ae5b0 2024-03-18T19:09:54.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.303+0000 7fe682d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe6840ae5b0 0x7fe684168120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:54.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.303+0000 7fe681d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe6840cfe80 0x7fe68406d2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59602/0 (socket says 172.21.15.1:59602) 2024-03-18T19:09:54.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.303+0000 7fe681d9b700 1 -- 172.21.15.1:0/2089951292 learned_addr learned my addr 172.21.15.1:0/2089951292 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:54.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.304+0000 7fe682d9d700 1 -- 172.21.15.1:0/2089951292 >> v1:172.21.15.165:6789/0 conn(0x7fe68415d570 legacy=0x7fe6840cf770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:54.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.304+0000 7fe682d9d700 1 -- 172.21.15.1:0/2089951292 >> v1:172.21.15.1:6789/0 conn(0x7fe6840cfe80 legacy=0x7fe68406d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:54.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.304+0000 7fe682d9d700 1 -- 172.21.15.1:0/2089951292 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6840acac0 con 0x7fe6840ae5b0 2024-03-18T19:09:54.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.304+0000 7fe682d9d700 1 --2- 172.21.15.1:0/2089951292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6840ae5b0 0x7fe684168120 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009d90 tx=0x7fe67c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=422057942a2b5735 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:54.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.304+0000 7fe68159a700 1 -- 172.21.15.1:0/2089951292 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c00e070 con 0x7fe6840ae5b0 2024-03-18T19:09:54.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.304+0000 7fe68159a700 1 -- 172.21.15.1:0/2089951292 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67c00bb90 con 0x7fe6840ae5b0 2024-03-18T19:09:54.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.304+0000 7fe68159a700 1 -- 172.21.15.1:0/2089951292 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c003a20 con 0x7fe6840ae5b0 2024-03-18T19:09:54.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.304+0000 7fe688a13700 1 -- 172.21.15.1:0/2089951292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6840ae5b0 msgr2=0x7fe684168120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:54.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.304+0000 7fe688a13700 1 --2- 172.21.15.1:0/2089951292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6840ae5b0 0x7fe684168120 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009d90 tx=0x7fe67c00b330 comp rx=0 tx=0).stop 2024-03-18T19:09:54.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.305+0000 7fe688a13700 1 -- 172.21.15.1:0/2089951292 shutdown_connections 2024-03-18T19:09:54.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.305+0000 7fe688a13700 1 --2- 172.21.15.1:0/2089951292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6840ae5b0 0x7fe684168120 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:54.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.305+0000 7fe688a13700 1 -- 172.21.15.1:0/2089951292 >> 172.21.15.1:0/2089951292 conn(0x7fe68406a750 msgr2=0x7fe68406ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:54.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.305+0000 7fe688a13700 1 -- 172.21.15.1:0/2089951292 shutdown_connections 2024-03-18T19:09:54.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.305+0000 7fe688a13700 1 -- 172.21.15.1:0/2089951292 wait complete. 2024-03-18T19:09:54.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.306+0000 7fe688a13700 1 Processor -- start 2024-03-18T19:09:54.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.306+0000 7fe688a13700 1 -- start start 2024-03-18T19:09:54.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe688a13700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6840ae5b0 0x7fe6841f3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:54.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe688a13700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6840cfe80 0x7fe6841f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:54.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe688a13700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe68415d570 0x7fe68414ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:54.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe688a13700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6840c8500 con 0x7fe68415d570 2024-03-18T19:09:54.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe688a13700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6841f3c80 con 0x7fe6840cfe80 2024-03-18T19:09:54.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe688a13700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe684149ae0 con 0x7fe6840ae5b0 2024-03-18T19:09:54.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe682d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe68415d570 0x7fe68414ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:54.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe682d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe68415d570 0x7fe68414ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44194/0 (socket says 172.21.15.1:44194) 2024-03-18T19:09:54.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe682d9d700 1 -- 172.21.15.1:0/2258996949 learned_addr learned my addr 172.21.15.1:0/2258996949 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:54.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe682d9d700 1 -- 172.21.15.1:0/2258996949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6840ae5b0 msgr2=0x7fe6841f3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:54.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe682d9d700 1 --2- 172.21.15.1:0/2258996949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6840ae5b0 0x7fe6841f3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:54.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe682d9d700 1 -- 172.21.15.1:0/2258996949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6840cfe80 msgr2=0x7fe6841f3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:54.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe681d9b700 1 --2- 172.21.15.1:0/2258996949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6840cfe80 0x7fe6841f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:54.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe682d9d700 1 --2- 172.21.15.1:0/2258996949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6840cfe80 0x7fe6841f3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:54.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe682d9d700 1 -- 172.21.15.1:0/2258996949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c009a70 con 0x7fe68415d570 2024-03-18T19:09:54.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe682d9d700 1 --2- 172.21.15.1:0/2258996949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe68415d570 0x7fe68414ab80 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fe67c004000 tx=0x7fe67c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:54.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe672ffd700 1 -- 172.21.15.1:0/2258996949 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c012630 con 0x7fe68415d570 2024-03-18T19:09:54.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe672ffd700 1 -- 172.21.15.1:0/2258996949 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67c0127b0 con 0x7fe68415d570 2024-03-18T19:09:54.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe688a13700 1 -- 172.21.15.1:0/2258996949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe684149dc0 con 0x7fe68415d570 2024-03-18T19:09:54.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.307+0000 7fe672ffd700 1 -- 172.21.15.1:0/2258996949 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c012b60 con 0x7fe68415d570 2024-03-18T19:09:54.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.308+0000 7fe688a13700 1 -- 172.21.15.1:0/2258996949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe68414a380 con 0x7fe68415d570 2024-03-18T19:09:54.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.308+0000 7fe688a13700 1 -- 172.21.15.1:0/2258996949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6840676a0 con 0x7fe68415d570 2024-03-18T19:09:54.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.310+0000 7fe672ffd700 1 -- 172.21.15.1:0/2258996949 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 81) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fe67c01d420 con 0x7fe68415d570 2024-03-18T19:09:54.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.310+0000 7fe672ffd700 1 --2- 172.21.15.1:0/2258996949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe66406ad20 0x7fe66406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:54.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.310+0000 7fe672ffd700 1 -- 172.21.15.1:0/2258996949 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6350+0+0 (secure 0 0 0) 0x7fe67c08bf00 con 0x7fe68415d570 2024-03-18T19:09:54.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.310+0000 7fe68259c700 1 --2- 172.21.15.1:0/2258996949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe66406ad20 0x7fe66406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:54.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.310+0000 7fe68259c700 1 --2- 172.21.15.1:0/2258996949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe66406ad20 0x7fe66406d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe6780096f0 tx=0x7fe678006c50 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:54.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.314+0000 7fe672ffd700 1 -- 172.21.15.1:0/2258996949 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe67c056e70 con 0x7fe68415d570 2024-03-18T19:09:54.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:54.517+0000 7fe688a13700 1 -- 172.21.15.1:0/2258996949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe6840cedc0 con 0x7fe68415d570 2024-03-18T19:09:55.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.162+0000 7fe672ffd700 1 -- 172.21.15.1:0/2258996949 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe67c057240 con 0x7fe68415d570 2024-03-18T19:09:55.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.163+0000 7fe672ffd700 1 -- 172.21.15.1:0/2258996949 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v152) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe67c05a900 con 0x7fe68415d570 2024-03-18T19:09:55.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.166+0000 7fe688a13700 1 -- 172.21.15.1:0/2258996949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe66406ad20 msgr2=0x7fe66406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:55.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.166+0000 7fe688a13700 1 --2- 172.21.15.1:0/2258996949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe66406ad20 0x7fe66406d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe6780096f0 tx=0x7fe678006c50 comp rx=0 tx=0).stop 2024-03-18T19:09:55.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.166+0000 7fe688a13700 1 -- 172.21.15.1:0/2258996949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe68415d570 msgr2=0x7fe68414ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:55.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.166+0000 7fe688a13700 1 --2- 172.21.15.1:0/2258996949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe68415d570 0x7fe68414ab80 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fe67c004000 tx=0x7fe67c004ad0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.166+0000 7fe688a13700 1 -- 172.21.15.1:0/2258996949 shutdown_connections 2024-03-18T19:09:55.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.166+0000 7fe688a13700 1 --2- 172.21.15.1:0/2258996949 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe66406ad20 0x7fe66406d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.166+0000 7fe688a13700 1 --2- 172.21.15.1:0/2258996949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6840ae5b0 0x7fe6841f3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.166+0000 7fe688a13700 1 --2- 172.21.15.1:0/2258996949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe68415d570 0x7fe68414ab80 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.166+0000 7fe688a13700 1 --2- 172.21.15.1:0/2258996949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6840cfe80 0x7fe6841f3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.166+0000 7fe688a13700 1 -- 172.21.15.1:0/2258996949 >> 172.21.15.1:0/2258996949 conn(0x7fe68406a750 msgr2=0x7fe68406ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:55.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.167+0000 7fe688a13700 1 -- 172.21.15.1:0/2258996949 shutdown_connections 2024-03-18T19:09:55.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.167+0000 7fe688a13700 1 -- 172.21.15.1:0/2258996949 wait complete. 2024-03-18T19:09:55.164 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-18T19:09:55.177 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-18T19:09:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.318+0000 7fcf23b27700 1 Processor -- start 2024-03-18T19:09:55.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.318+0000 7fcf23b27700 1 -- start start 2024-03-18T19:09:55.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.319+0000 7fcf23b27700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcf1c15d580 0x7fcf1c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.319+0000 7fcf23b27700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf1c06d380 0x7fcf1c071da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.319+0000 7fcf23b27700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c06f4f0 con 0x7fcf1c06d7e0 2024-03-18T19:09:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.319+0000 7fcf23b27700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf1c15d960 con 0x7fcf1c15d580 2024-03-18T19:09:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.319+0000 7fcf23b27700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c06ab30 con 0x7fcf1c06d380 2024-03-18T19:09:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.319+0000 7fcf218c3700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcf1c15d580 0x7fcf1c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.319+0000 7fcf210c2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf1c06d380 0x7fcf1c071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.319+0000 7fcf220c4700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fcf1c06d7e0 0x7fcf1c0cf2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:46794/0 (socket says 172.21.15.1:46794) 2024-03-18T19:09:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.319+0000 7fcf220c4700 1 -- 172.21.15.1:0/2828230320 learned_addr learned my addr 172.21.15.1:0/2828230320 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.320+0000 7fcf208c1700 1 -- 172.21.15.1:0/2828230320 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3751487637 0 0) 0x7fcf14001490 con 0x7fcf1c06d7e0 2024-03-18T19:09:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.320+0000 7fcf208c1700 1 -- 172.21.15.1:0/2828230320 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf080035b0 con 0x7fcf1c06d7e0 2024-03-18T19:09:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.320+0000 7fcf210c2700 1 -- 172.21.15.1:0/2828230320 >> v2:172.21.15.165:3300/0 conn(0x7fcf1c15d580 msgr2=0x7fcf1c06efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:55.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.320+0000 7fcf210c2700 1 --2- 172.21.15.1:0/2828230320 >> v2:172.21.15.165:3300/0 conn(0x7fcf1c15d580 0x7fcf1c06efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.320+0000 7fcf210c2700 1 -- 172.21.15.1:0/2828230320 >> v1:172.21.15.165:6789/0 conn(0x7fcf1c06d7e0 legacy=0x7fcf1c0cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:55.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.320+0000 7fcf210c2700 1 -- 172.21.15.1:0/2828230320 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c06ae10 con 0x7fcf1c06d380 2024-03-18T19:09:55.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.320+0000 7fcf218c3700 1 --2- 172.21.15.1:0/2828230320 >> v2:172.21.15.165:3300/0 conn(0x7fcf1c15d580 0x7fcf1c06efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:55.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.320+0000 7fcf210c2700 1 --2- 172.21.15.1:0/2828230320 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf1c06d380 0x7fcf1c071da0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcf18009840 tx=0x7fcf18009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92318ba11cc8210a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:55.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.320+0000 7fcf208c1700 1 -- 172.21.15.1:0/2828230320 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf18011830 con 0x7fcf1c06d380 2024-03-18T19:09:55.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.320+0000 7fcf208c1700 1 -- 172.21.15.1:0/2828230320 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf18011990 con 0x7fcf1c06d380 2024-03-18T19:09:55.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.320+0000 7fcf208c1700 1 -- 172.21.15.1:0/2828230320 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf18011d20 con 0x7fcf1c06d380 2024-03-18T19:09:55.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.321+0000 7fcf23b27700 1 -- 172.21.15.1:0/2828230320 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf1c06d380 msgr2=0x7fcf1c071da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:55.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.321+0000 7fcf23b27700 1 --2- 172.21.15.1:0/2828230320 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf1c06d380 0x7fcf1c071da0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcf18009840 tx=0x7fcf18009b50 comp rx=0 tx=0).stop 2024-03-18T19:09:55.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.321+0000 7fcf23b27700 1 -- 172.21.15.1:0/2828230320 shutdown_connections 2024-03-18T19:09:55.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.321+0000 7fcf23b27700 1 --2- 172.21.15.1:0/2828230320 >> v2:172.21.15.165:3300/0 conn(0x7fcf1c15d580 0x7fcf1c06efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.321+0000 7fcf23b27700 1 --2- 172.21.15.1:0/2828230320 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf1c06d380 0x7fcf1c071da0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.321+0000 7fcf23b27700 1 -- 172.21.15.1:0/2828230320 >> 172.21.15.1:0/2828230320 conn(0x7fcf1c0cca00 msgr2=0x7fcf1c0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:55.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.321+0000 7fcf23b27700 1 -- 172.21.15.1:0/2828230320 shutdown_connections 2024-03-18T19:09:55.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.321+0000 7fcf23b27700 1 -- 172.21.15.1:0/2828230320 wait complete. 2024-03-18T19:09:55.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.322+0000 7fcf23b27700 1 Processor -- start 2024-03-18T19:09:55.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.322+0000 7fcf23b27700 1 -- start start 2024-03-18T19:09:55.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.323+0000 7fcf23b27700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf1c06d380 0x7fcf1c0ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:55.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.323+0000 7fcf23b27700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf1c06d7e0 0x7fcf1c1f6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:55.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.323+0000 7fcf23b27700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf1c15d580 0x7fcf1c0c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:55.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.323+0000 7fcf23b27700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c1f4030 con 0x7fcf1c06d7e0 2024-03-18T19:09:55.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.323+0000 7fcf23b27700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c15c250 con 0x7fcf1c06d380 2024-03-18T19:09:55.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.323+0000 7fcf23b27700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c1f7480 con 0x7fcf1c15d580 2024-03-18T19:09:55.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.323+0000 7fcf210c2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf1c06d7e0 0x7fcf1c1f6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:55.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.323+0000 7fcf210c2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf1c06d7e0 0x7fcf1c1f6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35090/0 (socket says 172.21.15.1:35090) 2024-03-18T19:09:55.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.323+0000 7fcf210c2700 1 -- 172.21.15.1:0/1303280812 learned_addr learned my addr 172.21.15.1:0/1303280812 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:55.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.323+0000 7fcf218c3700 1 --2- 172.21.15.1:0/1303280812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf1c06d380 0x7fcf1c0ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:55.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.323+0000 7fcf220c4700 1 --2- 172.21.15.1:0/1303280812 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf1c15d580 0x7fcf1c0c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:55.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.323+0000 7fcf210c2700 1 -- 172.21.15.1:0/1303280812 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf1c15d580 msgr2=0x7fcf1c0c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:55.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.323+0000 7fcf210c2700 1 --2- 172.21.15.1:0/1303280812 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf1c15d580 0x7fcf1c0c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.323+0000 7fcf210c2700 1 -- 172.21.15.1:0/1303280812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf1c06d380 msgr2=0x7fcf1c0ca8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:55.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.323+0000 7fcf210c2700 1 --2- 172.21.15.1:0/1303280812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf1c06d380 0x7fcf1c0ca8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.323+0000 7fcf210c2700 1 -- 172.21.15.1:0/1303280812 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0c0094a0 con 0x7fcf1c06d7e0 2024-03-18T19:09:55.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.323+0000 7fcf218c3700 1 --2- 172.21.15.1:0/1303280812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf1c06d380 0x7fcf1c0ca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:55.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.323+0000 7fcf210c2700 1 --2- 172.21.15.1:0/1303280812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf1c06d7e0 0x7fcf1c1f6550 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fcf18009fd0 tx=0x7fcf18004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:55.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.324+0000 7fcf127fc700 1 -- 172.21.15.1:0/1303280812 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf180036a0 con 0x7fcf1c06d7e0 2024-03-18T19:09:55.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.324+0000 7fcf127fc700 1 -- 172.21.15.1:0/1303280812 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1801e070 con 0x7fcf1c06d7e0 2024-03-18T19:09:55.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.324+0000 7fcf127fc700 1 -- 172.21.15.1:0/1303280812 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf18003a60 con 0x7fcf1c06d7e0 2024-03-18T19:09:55.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.324+0000 7fcf23b27700 1 -- 172.21.15.1:0/1303280812 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf180094a0 con 0x7fcf1c06d7e0 2024-03-18T19:09:55.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.324+0000 7fcf23b27700 1 -- 172.21.15.1:0/1303280812 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1c0c6960 con 0x7fcf1c06d7e0 2024-03-18T19:09:55.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.325+0000 7fcf23b27700 1 -- 172.21.15.1:0/1303280812 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1c0ceea0 con 0x7fcf1c06d7e0 2024-03-18T19:09:55.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.325+0000 7fcf127fc700 1 -- 172.21.15.1:0/1303280812 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcf18003bc0 con 0x7fcf1c06d7e0 2024-03-18T19:09:55.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.326+0000 7fcf127fc700 1 --2- 172.21.15.1:0/1303280812 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcf0006adb0 0x7fcf0006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:55.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.326+0000 7fcf127fc700 1 -- 172.21.15.1:0/1303280812 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5936+0+0 (secure 0 0 0) 0x7fcf1808a2c0 con 0x7fcf1c06d7e0 2024-03-18T19:09:55.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.326+0000 7fcf218c3700 1 --2- 172.21.15.1:0/1303280812 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcf0006adb0 0x7fcf0006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:55.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.326+0000 7fcf218c3700 1 --2- 172.21.15.1:0/1303280812 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcf0006adb0 0x7fcf0006d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c009dc0 tx=0x7fcf0c008040 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:55.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.330+0000 7fcf127fc700 1 -- 172.21.15.1:0/1303280812 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcf18055340 con 0x7fcf1c06d7e0 2024-03-18T19:09:55.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.539+0000 7fcf23b27700 1 -- 172.21.15.1:0/1303280812 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fcf1c0c6960 con 0x7fcf1c06d7e0 2024-03-18T19:09:55.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.543+0000 7fcf127fc700 1 -- 172.21.15.1:0/1303280812 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v49) v1 ==== 115+0+0 (secure 0 0 0) 0x7fcf1c0c6960 con 0x7fcf1c06d7e0 2024-03-18T19:09:55.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.546+0000 7fcf23b27700 1 -- 172.21.15.1:0/1303280812 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcf0006adb0 msgr2=0x7fcf0006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:55.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.546+0000 7fcf23b27700 1 --2- 172.21.15.1:0/1303280812 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcf0006adb0 0x7fcf0006d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c009dc0 tx=0x7fcf0c008040 comp rx=0 tx=0).stop 2024-03-18T19:09:55.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.546+0000 7fcf23b27700 1 -- 172.21.15.1:0/1303280812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf1c06d7e0 msgr2=0x7fcf1c1f6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:55.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.546+0000 7fcf23b27700 1 --2- 172.21.15.1:0/1303280812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf1c06d7e0 0x7fcf1c1f6550 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fcf18009fd0 tx=0x7fcf18004750 comp rx=0 tx=0).stop 2024-03-18T19:09:55.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.546+0000 7fcf23b27700 1 -- 172.21.15.1:0/1303280812 shutdown_connections 2024-03-18T19:09:55.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.546+0000 7fcf23b27700 1 --2- 172.21.15.1:0/1303280812 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcf0006adb0 0x7fcf0006d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.546+0000 7fcf23b27700 1 --2- 172.21.15.1:0/1303280812 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf1c15d580 0x7fcf1c0c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.546+0000 7fcf23b27700 1 --2- 172.21.15.1:0/1303280812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf1c06d380 0x7fcf1c0ca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.546+0000 7fcf23b27700 1 --2- 172.21.15.1:0/1303280812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf1c06d7e0 0x7fcf1c1f6550 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.546+0000 7fcf23b27700 1 -- 172.21.15.1:0/1303280812 >> 172.21.15.1:0/1303280812 conn(0x7fcf1c0cca00 msgr2=0x7fcf1c06dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:55.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.546+0000 7fcf23b27700 1 -- 172.21.15.1:0/1303280812 shutdown_connections 2024-03-18T19:09:55.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.546+0000 7fcf23b27700 1 -- 172.21.15.1:0/1303280812 wait complete. 2024-03-18T19:09:55.554 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-18T19:09:55.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.685+0000 7f4138816700 1 Processor -- start 2024-03-18T19:09:55.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.685+0000 7f4138816700 1 -- start start 2024-03-18T19:09:55.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.686+0000 7f4138816700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f413415eba0 0x7f41340cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:55.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.686+0000 7f4138816700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41340b0500 0x7f41340cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:55.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.686+0000 7f4138816700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413415ef80 con 0x7f41340cfdf0 2024-03-18T19:09:55.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.686+0000 7f4138816700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41340cc9a0 con 0x7f413415eba0 2024-03-18T19:09:55.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.686+0000 7f4138816700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41340ccc80 con 0x7f41340b0500 2024-03-18T19:09:55.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.687+0000 7f413259c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f413415eba0 0x7f41340cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:55.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.687+0000 7f4131d9b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41340b0500 0x7f41340cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:55.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.687+0000 7f4132d9d700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f41340cfdf0 0x7f4134099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:46806/0 (socket says 172.21.15.1:46806) 2024-03-18T19:09:55.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.687+0000 7f4132d9d700 1 -- 172.21.15.1:0/443003428 learned_addr learned my addr 172.21.15.1:0/443003428 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:55.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.687+0000 7f413159a700 1 -- 172.21.15.1:0/443003428 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689724101 0 0) 0x7f413415ef80 con 0x7f41340cfdf0 2024-03-18T19:09:55.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.687+0000 7f413159a700 1 -- 172.21.15.1:0/443003428 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41200035b0 con 0x7f41340cfdf0 2024-03-18T19:09:55.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.687+0000 7f413259c700 1 -- 172.21.15.1:0/443003428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41340b0500 msgr2=0x7f41340cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:55.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.687+0000 7f413259c700 1 --2- 172.21.15.1:0/443003428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41340b0500 0x7f41340cf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.687+0000 7f413259c700 1 -- 172.21.15.1:0/443003428 >> v1:172.21.15.165:6789/0 conn(0x7f41340cfdf0 legacy=0x7f4134099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:55.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.687+0000 7f413259c700 1 -- 172.21.15.1:0/443003428 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41340ccf60 con 0x7f413415eba0 2024-03-18T19:09:55.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.687+0000 7f4131d9b700 1 --2- 172.21.15.1:0/443003428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41340b0500 0x7f41340cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:55.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.687+0000 7f413259c700 1 --2- 172.21.15.1:0/443003428 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f413415eba0 0x7f41340cf370 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f411c009840 tx=0x7f411c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ccfee467b09b58e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:55.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.687+0000 7f413159a700 1 -- 172.21.15.1:0/443003428 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c011830 con 0x7f413415eba0 2024-03-18T19:09:55.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.687+0000 7f413159a700 1 -- 172.21.15.1:0/443003428 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411c011990 con 0x7f413415eba0 2024-03-18T19:09:55.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.687+0000 7f413159a700 1 -- 172.21.15.1:0/443003428 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c011d20 con 0x7f413415eba0 2024-03-18T19:09:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.688+0000 7f4138816700 1 -- 172.21.15.1:0/443003428 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f413415eba0 msgr2=0x7f41340cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.688+0000 7f4138816700 1 --2- 172.21.15.1:0/443003428 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f413415eba0 0x7f41340cf370 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f411c009840 tx=0x7f411c009b50 comp rx=0 tx=0).stop 2024-03-18T19:09:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.688+0000 7f4138816700 1 -- 172.21.15.1:0/443003428 shutdown_connections 2024-03-18T19:09:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.688+0000 7f4138816700 1 --2- 172.21.15.1:0/443003428 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f413415eba0 0x7f41340cf370 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.688+0000 7f4138816700 1 --2- 172.21.15.1:0/443003428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41340b0500 0x7f41340cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.688+0000 7f4138816700 1 -- 172.21.15.1:0/443003428 >> 172.21.15.1:0/443003428 conn(0x7f41340b2bd0 msgr2=0x7f41340b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.688+0000 7f4138816700 1 -- 172.21.15.1:0/443003428 shutdown_connections 2024-03-18T19:09:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.688+0000 7f4138816700 1 -- 172.21.15.1:0/443003428 wait complete. 2024-03-18T19:09:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.690+0000 7f4138816700 1 Processor -- start 2024-03-18T19:09:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.690+0000 7f4138816700 1 -- start start 2024-03-18T19:09:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.691+0000 7f4138816700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41340b0500 0x7f41340c4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.691+0000 7f4138816700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41340cfdf0 0x7f41341595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.691+0000 7f4138816700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f413415eba0 0x7f4134159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.691+0000 7f4138816700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41340c60d0 con 0x7f41340b0500 2024-03-18T19:09:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.691+0000 7f4138816700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41340c5550 con 0x7f41340cfdf0 2024-03-18T19:09:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.691+0000 7f4138816700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41340c5830 con 0x7f413415eba0 2024-03-18T19:09:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.691+0000 7f413259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41340b0500 0x7f41340c4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.691+0000 7f413259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41340b0500 0x7f41340c4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35106/0 (socket says 172.21.15.1:35106) 2024-03-18T19:09:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.691+0000 7f413259c700 1 -- 172.21.15.1:0/267522811 learned_addr learned my addr 172.21.15.1:0/267522811 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.691+0000 7f4131d9b700 1 --2- 172.21.15.1:0/267522811 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41340cfdf0 0x7f41341595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:55.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.691+0000 7f4132d9d700 1 --2- 172.21.15.1:0/267522811 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f413415eba0 0x7f4134159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:55.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.691+0000 7f413259c700 1 -- 172.21.15.1:0/267522811 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f413415eba0 msgr2=0x7f4134159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:55.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.691+0000 7f413259c700 1 --2- 172.21.15.1:0/267522811 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f413415eba0 0x7f4134159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.691+0000 7f413259c700 1 -- 172.21.15.1:0/267522811 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41340cfdf0 msgr2=0x7f41341595c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:55.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.691+0000 7f413259c700 1 --2- 172.21.15.1:0/267522811 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41340cfdf0 0x7f41341595c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.691+0000 7f413259c700 1 -- 172.21.15.1:0/267522811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41240094a0 con 0x7f41340b0500 2024-03-18T19:09:55.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.691+0000 7f413259c700 1 --2- 172.21.15.1:0/267522811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41340b0500 0x7f41340c4d00 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f411c009fd0 tx=0x7f411c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:55.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.692+0000 7f412affd700 1 -- 172.21.15.1:0/267522811 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c0036a0 con 0x7f41340b0500 2024-03-18T19:09:55.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.692+0000 7f412affd700 1 -- 172.21.15.1:0/267522811 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411c01e070 con 0x7f41340b0500 2024-03-18T19:09:55.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.692+0000 7f4138816700 1 -- 172.21.15.1:0/267522811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f411c0094a0 con 0x7f41340b0500 2024-03-18T19:09:55.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.692+0000 7f412affd700 1 -- 172.21.15.1:0/267522811 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c003a60 con 0x7f41340b0500 2024-03-18T19:09:55.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.692+0000 7f4138816700 1 -- 172.21.15.1:0/267522811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41340c5df0 con 0x7f41340b0500 2024-03-18T19:09:55.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.693+0000 7f4138816700 1 -- 172.21.15.1:0/267522811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41340676a0 con 0x7f41340b0500 2024-03-18T19:09:55.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.693+0000 7f412affd700 1 -- 172.21.15.1:0/267522811 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f411c003bc0 con 0x7f41340b0500 2024-03-18T19:09:55.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.693+0000 7f412affd700 1 --2- 172.21.15.1:0/267522811 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f411406adb0 0x7f411406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.694+0000 7f412affd700 1 -- 172.21.15.1:0/267522811 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5936+0+0 (secure 0 0 0) 0x7f411c08a2c0 con 0x7f41340b0500 2024-03-18T19:09:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.694+0000 7f4131d9b700 1 --2- 172.21.15.1:0/267522811 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f411406adb0 0x7f411406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.694+0000 7f4131d9b700 1 --2- 172.21.15.1:0/267522811 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f411406adb0 0x7f411406d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f41341f44f0 tx=0x7f4124008040 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:55.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.698+0000 7f412affd700 1 -- 172.21.15.1:0/267522811 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f411c055340 con 0x7f41340b0500 2024-03-18T19:09:55.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.902+0000 7f4138816700 1 -- 172.21.15.1:0/267522811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f41340b0270 con 0x7f41340b0500 2024-03-18T19:09:55.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.906+0000 7f412affd700 1 -- 172.21.15.1:0/267522811 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v50) v1 ==== 120+0+0 (secure 0 0 0) 0x7f411c058960 con 0x7f41340b0500 2024-03-18T19:09:55.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.908+0000 7f4138816700 1 -- 172.21.15.1:0/267522811 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f411406adb0 msgr2=0x7f411406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:55.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.908+0000 7f4138816700 1 --2- 172.21.15.1:0/267522811 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f411406adb0 0x7f411406d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f41341f44f0 tx=0x7f4124008040 comp rx=0 tx=0).stop 2024-03-18T19:09:55.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.908+0000 7f4138816700 1 -- 172.21.15.1:0/267522811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41340b0500 msgr2=0x7f41340c4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:55.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.908+0000 7f4138816700 1 --2- 172.21.15.1:0/267522811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41340b0500 0x7f41340c4d00 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f411c009fd0 tx=0x7f411c004750 comp rx=0 tx=0).stop 2024-03-18T19:09:55.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.908+0000 7f4138816700 1 -- 172.21.15.1:0/267522811 shutdown_connections 2024-03-18T19:09:55.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.908+0000 7f4138816700 1 --2- 172.21.15.1:0/267522811 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f411406adb0 0x7f411406d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.908+0000 7f4138816700 1 --2- 172.21.15.1:0/267522811 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f413415eba0 0x7f4134159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.908+0000 7f4138816700 1 --2- 172.21.15.1:0/267522811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41340b0500 0x7f41340c4d00 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.908+0000 7f4138816700 1 --2- 172.21.15.1:0/267522811 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41340cfdf0 0x7f41341595c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:55.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.908+0000 7f4138816700 1 -- 172.21.15.1:0/267522811 >> 172.21.15.1:0/267522811 conn(0x7f41340b2bd0 msgr2=0x7f413406fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:55.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.909+0000 7f4138816700 1 -- 172.21.15.1:0/267522811 shutdown_connections 2024-03-18T19:09:55.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:55.909+0000 7f4138816700 1 -- 172.21.15.1:0/267522811 wait complete. 2024-03-18T19:09:55.917 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-18T19:09:56.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.048+0000 7fae7a868700 1 Processor -- start 2024-03-18T19:09:56.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.048+0000 7fae7a868700 1 -- start start 2024-03-18T19:09:56.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.049+0000 7fae7a868700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae740cbd40 0x7fae740cc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:56.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.049+0000 7fae7a868700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fae740c3670 0x7fae7415e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:56.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.049+0000 7fae7a868700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae740cf730 con 0x7fae7415e660 2024-03-18T19:09:56.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.049+0000 7fae7a868700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae740cc660 con 0x7fae740c3670 2024-03-18T19:09:56.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.049+0000 7fae7a868700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae740cf980 con 0x7fae740cbd40 2024-03-18T19:09:56.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.049+0000 7fae737fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fae740c3670 0x7fae7415e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:56.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.049+0000 7fae737fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fae740c3670 0x7fae7415e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35118/0 (socket says 172.21.15.1:35118) 2024-03-18T19:09:56.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.049+0000 7fae737fe700 1 -- 172.21.15.1:0/81123217 learned_addr learned my addr 172.21.15.1:0/81123217 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:56.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.049+0000 7fae73fff700 1 --2- 172.21.15.1:0/81123217 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae740cbd40 0x7fae740cc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:56.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.049+0000 7fae737fe700 1 -- 172.21.15.1:0/81123217 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae740cbd40 msgr2=0x7fae740cc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:56.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.049+0000 7fae737fe700 1 --2- 172.21.15.1:0/81123217 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae740cbd40 0x7fae740cc120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:56.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.049+0000 7fae737fe700 1 -- 172.21.15.1:0/81123217 >> v1:172.21.15.165:6789/0 conn(0x7fae7415e660 legacy=0x7fae740cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:56.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.049+0000 7fae737fe700 1 -- 172.21.15.1:0/81123217 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae740cfc60 con 0x7fae740c3670 2024-03-18T19:09:56.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.050+0000 7fae737fe700 1 --2- 172.21.15.1:0/81123217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740c3670 0x7fae7415e120 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fae640098f0 tx=0x7fae64009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d35d9bf3ffeb1f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:56.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.050+0000 7fae72ffd700 1 -- 172.21.15.1:0/81123217 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6400b8a0 con 0x7fae740c3670 2024-03-18T19:09:56.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.050+0000 7fae72ffd700 1 -- 172.21.15.1:0/81123217 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae6400ba00 con 0x7fae740c3670 2024-03-18T19:09:56.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.050+0000 7fae72ffd700 1 -- 172.21.15.1:0/81123217 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6400bd90 con 0x7fae740c3670 2024-03-18T19:09:56.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.050+0000 7fae7a868700 1 -- 172.21.15.1:0/81123217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740c3670 msgr2=0x7fae7415e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:56.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.050+0000 7fae7a868700 1 --2- 172.21.15.1:0/81123217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740c3670 0x7fae7415e120 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fae640098f0 tx=0x7fae64009c00 comp rx=0 tx=0).stop 2024-03-18T19:09:56.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.050+0000 7fae7a868700 1 -- 172.21.15.1:0/81123217 shutdown_connections 2024-03-18T19:09:56.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.050+0000 7fae7a868700 1 --2- 172.21.15.1:0/81123217 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae740cbd40 0x7fae740cc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:56.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.050+0000 7fae7a868700 1 --2- 172.21.15.1:0/81123217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740c3670 0x7fae7415e120 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:56.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.050+0000 7fae7a868700 1 -- 172.21.15.1:0/81123217 >> 172.21.15.1:0/81123217 conn(0x7fae740b1aa0 msgr2=0x7fae740b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:56.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.050+0000 7fae7a868700 1 -- 172.21.15.1:0/81123217 shutdown_connections 2024-03-18T19:09:56.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.050+0000 7fae7a868700 1 -- 172.21.15.1:0/81123217 wait complete. 2024-03-18T19:09:56.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.052+0000 7fae7a868700 1 Processor -- start 2024-03-18T19:09:56.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.052+0000 7fae7a868700 1 -- start start 2024-03-18T19:09:56.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.053+0000 7fae7a868700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740c3670 0x7fae74158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:56.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.053+0000 7fae7a868700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae740cbd40 0x7fae7415f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:56.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.053+0000 7fae7a868700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae7415e660 0x7fae740b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:56.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.053+0000 7fae7a868700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7415fdd0 con 0x7fae740c3670 2024-03-18T19:09:56.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.053+0000 7fae7a868700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae741593f0 con 0x7fae740cbd40 2024-03-18T19:09:56.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.053+0000 7fae7a868700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7415faf0 con 0x7fae7415e660 2024-03-18T19:09:56.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.053+0000 7fae73fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740c3670 0x7fae74158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:56.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.053+0000 7fae73fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740c3670 0x7fae74158eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35130/0 (socket says 172.21.15.1:35130) 2024-03-18T19:09:56.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.053+0000 7fae73fff700 1 -- 172.21.15.1:0/1542612025 learned_addr learned my addr 172.21.15.1:0/1542612025 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:56.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.053+0000 7fae737fe700 1 --2- 172.21.15.1:0/1542612025 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae740cbd40 0x7fae7415f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:56.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.053+0000 7fae78e05700 1 --2- 172.21.15.1:0/1542612025 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae7415e660 0x7fae740b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:56.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.053+0000 7fae73fff700 1 -- 172.21.15.1:0/1542612025 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae7415e660 msgr2=0x7fae740b2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:56.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.053+0000 7fae73fff700 1 --2- 172.21.15.1:0/1542612025 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae7415e660 0x7fae740b2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:56.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.053+0000 7fae73fff700 1 -- 172.21.15.1:0/1542612025 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae740cbd40 msgr2=0x7fae7415f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:56.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.053+0000 7fae73fff700 1 --2- 172.21.15.1:0/1542612025 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae740cbd40 0x7fae7415f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:56.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.053+0000 7fae73fff700 1 -- 172.21.15.1:0/1542612025 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae64009550 con 0x7fae740c3670 2024-03-18T19:09:56.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.053+0000 7fae78e05700 1 --2- 172.21.15.1:0/1542612025 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae7415e660 0x7fae740b2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:09:56.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.054+0000 7fae73fff700 1 --2- 172.21.15.1:0/1542612025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740c3670 0x7fae74158eb0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00e810 tx=0x7fae5c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:56.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.054+0000 7fae70ff9700 1 -- 172.21.15.1:0/1542612025 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c013610 con 0x7fae740c3670 2024-03-18T19:09:56.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.054+0000 7fae70ff9700 1 -- 172.21.15.1:0/1542612025 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5c013770 con 0x7fae740c3670 2024-03-18T19:09:56.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.054+0000 7fae7a868700 1 -- 172.21.15.1:0/1542612025 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae740c4c20 con 0x7fae740c3670 2024-03-18T19:09:56.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.054+0000 7fae70ff9700 1 -- 172.21.15.1:0/1542612025 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c013b20 con 0x7fae740c3670 2024-03-18T19:09:56.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.054+0000 7fae7a868700 1 -- 172.21.15.1:0/1542612025 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae740c51e0 con 0x7fae740c3670 2024-03-18T19:09:56.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.055+0000 7fae7a868700 1 -- 172.21.15.1:0/1542612025 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae740cf580 con 0x7fae740c3670 2024-03-18T19:09:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.055+0000 7fae70ff9700 1 -- 172.21.15.1:0/1542612025 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fae5c013c80 con 0x7fae740c3670 2024-03-18T19:09:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.056+0000 7fae70ff9700 1 --2- 172.21.15.1:0/1542612025 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae5406adb0 0x7fae5406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.056+0000 7fae70ff9700 1 -- 172.21.15.1:0/1542612025 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5936+0+0 (secure 0 0 0) 0x7fae5c089350 con 0x7fae740c3670 2024-03-18T19:09:56.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.056+0000 7fae737fe700 1 --2- 172.21.15.1:0/1542612025 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae5406adb0 0x7fae5406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:56.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.056+0000 7fae737fe700 1 --2- 172.21.15.1:0/1542612025 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae5406adb0 0x7fae5406d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fae64009c40 tx=0x7fae64008030 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:56.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.060+0000 7fae70ff9700 1 -- 172.21.15.1:0/1542612025 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fae5c0541c0 con 0x7fae740c3670 2024-03-18T19:09:56.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.268+0000 7fae7a868700 1 -- 172.21.15.1:0/1542612025 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fae7415d6f0 con 0x7fae740c3670 2024-03-18T19:09:56.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.272+0000 7fae70ff9700 1 -- 172.21.15.1:0/1542612025 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v51) v1 ==== 104+0+0 (secure 0 0 0) 0x7fae7415d6f0 con 0x7fae740c3670 2024-03-18T19:09:56.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.274+0000 7fae7a868700 1 -- 172.21.15.1:0/1542612025 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae5406adb0 msgr2=0x7fae5406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:56.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.274+0000 7fae7a868700 1 --2- 172.21.15.1:0/1542612025 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae5406adb0 0x7fae5406d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fae64009c40 tx=0x7fae64008030 comp rx=0 tx=0).stop 2024-03-18T19:09:56.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.274+0000 7fae7a868700 1 -- 172.21.15.1:0/1542612025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740c3670 msgr2=0x7fae74158eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:56.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.274+0000 7fae7a868700 1 --2- 172.21.15.1:0/1542612025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740c3670 0x7fae74158eb0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00e810 tx=0x7fae5c00eb20 comp rx=0 tx=0).stop 2024-03-18T19:09:56.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.274+0000 7fae7a868700 1 -- 172.21.15.1:0/1542612025 shutdown_connections 2024-03-18T19:09:56.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.274+0000 7fae7a868700 1 --2- 172.21.15.1:0/1542612025 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fae5406adb0 0x7fae5406d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:56.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.274+0000 7fae7a868700 1 --2- 172.21.15.1:0/1542612025 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fae7415e660 0x7fae740b2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:56.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.274+0000 7fae7a868700 1 --2- 172.21.15.1:0/1542612025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae740c3670 0x7fae74158eb0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:56.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.274+0000 7fae7a868700 1 --2- 172.21.15.1:0/1542612025 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae740cbd40 0x7fae7415f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:56.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.274+0000 7fae7a868700 1 -- 172.21.15.1:0/1542612025 >> 172.21.15.1:0/1542612025 conn(0x7fae740b1aa0 msgr2=0x7fae7406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:56.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.274+0000 7fae7a868700 1 -- 172.21.15.1:0/1542612025 shutdown_connections 2024-03-18T19:09:56.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.274+0000 7fae7a868700 1 -- 172.21.15.1:0/1542612025 wait complete. 2024-03-18T19:09:56.282 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2024-03-18T19:09:56.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.411+0000 7fd114e54700 1 Processor -- start 2024-03-18T19:09:56.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.411+0000 7fd114e54700 1 -- start start 2024-03-18T19:09:56.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.412+0000 7fd114e54700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1100b6b60 0x7fd1100cc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:56.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.412+0000 7fd114e54700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd1100cfda0 0x7fd1100b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:56.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.412+0000 7fd114e54700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1100cd0d0 con 0x7fd1100cee90 2024-03-18T19:09:56.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.412+0000 7fd114e54700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1100b3f00 con 0x7fd1100cfda0 2024-03-18T19:09:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.412+0000 7fd114e54700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1100b41e0 con 0x7fd1100b6b60 2024-03-18T19:09:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.412+0000 7fd10dd9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd1100cee90 0x7fd1100cf690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58552/0 (socket says 172.21.15.1:58552) 2024-03-18T19:09:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.412+0000 7fd10dd9b700 1 -- 172.21.15.1:0/1392806796 learned_addr learned my addr 172.21.15.1:0/1392806796 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.412+0000 7fd10ed9d700 1 --2- 172.21.15.1:0/1392806796 >> v2:172.21.15.165:3300/0 conn(0x7fd1100cfda0 0x7fd1100b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.412+0000 7fd10e59c700 1 --2- 172.21.15.1:0/1392806796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1100b6b60 0x7fd1100cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.412+0000 7fd10d59a700 1 -- 172.21.15.1:0/1392806796 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319240412 0 0) 0x7fd1100cd0d0 con 0x7fd1100cee90 2024-03-18T19:09:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.412+0000 7fd10d59a700 1 -- 172.21.15.1:0/1392806796 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0f80035b0 con 0x7fd1100cee90 2024-03-18T19:09:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10d59a700 1 -- 172.21.15.1:0/1392806796 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3946446436 0 0) 0x7fd0f80035b0 con 0x7fd1100cee90 2024-03-18T19:09:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10d59a700 1 -- 172.21.15.1:0/1392806796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1100b6b60 msgr2=0x7fd1100cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10d59a700 1 --2- 172.21.15.1:0/1392806796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1100b6b60 0x7fd1100cc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10d59a700 1 -- 172.21.15.1:0/1392806796 >> v2:172.21.15.165:3300/0 conn(0x7fd1100cfda0 msgr2=0x7fd1100b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10d59a700 1 --2- 172.21.15.1:0/1392806796 >> v2:172.21.15.165:3300/0 conn(0x7fd1100cfda0 0x7fd1100b4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10d59a700 1 -- 172.21.15.1:0/1392806796 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1100b44c0 con 0x7fd1100cee90 2024-03-18T19:09:56.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10d59a700 1 -- 172.21.15.1:0/1392806796 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fd100001670 con 0x7fd1100cee90 2024-03-18T19:09:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10ed9d700 1 --2- 172.21.15.1:0/1392806796 >> v2:172.21.15.165:3300/0 conn(0x7fd1100cfda0 0x7fd1100b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:09:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10d59a700 1 -- 172.21.15.1:0/1392806796 >> v1:172.21.15.1:6789/0 conn(0x7fd1100cee90 legacy=0x7fd1100cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10d59a700 1 --2- 172.21.15.1:0/1392806796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1100b6b60 0x7fd0f8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10d59a700 1 --2- 172.21.15.1:0/1392806796 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0f8003f80 0x7fd0f8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10d59a700 1 --2- 172.21.15.1:0/1392806796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0f8004d90 0x7fd0f8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10d59a700 1 -- 172.21.15.1:0/1392806796 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1100b3f00 con 0x7fd0f8004d90 2024-03-18T19:09:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10d59a700 1 -- 172.21.15.1:0/1392806796 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1100b41e0 con 0x7fd0f8003f80 2024-03-18T19:09:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10d59a700 1 -- 172.21.15.1:0/1392806796 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f800ba10 con 0x7fd1100b6b60 2024-03-18T19:09:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10ed9d700 1 --2- 172.21.15.1:0/1392806796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0f8004d90 0x7fd0f8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10e59c700 1 --2- 172.21.15.1:0/1392806796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1100b6b60 0x7fd0f8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10dd9b700 1 --2- 172.21.15.1:0/1392806796 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0f8003f80 0x7fd0f8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10ed9d700 1 -- 172.21.15.1:0/1392806796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1100b6b60 msgr2=0x7fd0f8003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10ed9d700 1 --2- 172.21.15.1:0/1392806796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1100b6b60 0x7fd0f8003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10ed9d700 1 -- 172.21.15.1:0/1392806796 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0f8003f80 msgr2=0x7fd0f8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:56.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10ed9d700 1 --2- 172.21.15.1:0/1392806796 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0f8003f80 0x7fd0f8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:56.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10ed9d700 1 -- 172.21.15.1:0/1392806796 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f80035b0 con 0x7fd0f8004d90 2024-03-18T19:09:56.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10ed9d700 1 --2- 172.21.15.1:0/1392806796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0f8004d90 0x7fd0f8005210 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fd10800e320 tx=0x7fd0f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6471b22bb70b7259 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:56.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10d59a700 1 -- 172.21.15.1:0/1392806796 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10800ed60 con 0x7fd0f8004d90 2024-03-18T19:09:56.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10d59a700 1 -- 172.21.15.1:0/1392806796 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd10800f040 con 0x7fd0f8004d90 2024-03-18T19:09:56.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.413+0000 7fd10d59a700 1 -- 172.21.15.1:0/1392806796 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1080146a0 con 0x7fd0f8004d90 2024-03-18T19:09:56.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.414+0000 7fd114e54700 1 -- 172.21.15.1:0/1392806796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0f8004d90 msgr2=0x7fd0f8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:56.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.414+0000 7fd114e54700 1 --2- 172.21.15.1:0/1392806796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd0f8004d90 0x7fd0f8005210 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fd10800e320 tx=0x7fd0f8005880 comp rx=0 tx=0).stop 2024-03-18T19:09:56.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.414+0000 7fd10ed9d700 1 -- 172.21.15.1:0/1392806796 reap_dead start 2024-03-18T19:09:56.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.414+0000 7fd114e54700 1 -- 172.21.15.1:0/1392806796 shutdown_connections 2024-03-18T19:09:56.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.414+0000 7fd114e54700 1 -- 172.21.15.1:0/1392806796 >> 172.21.15.1:0/1392806796 conn(0x7fd1100b2ba0 msgr2=0x7fd1100b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:56.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.414+0000 7fd114e54700 1 -- 172.21.15.1:0/1392806796 shutdown_connections 2024-03-18T19:09:56.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.414+0000 7fd114e54700 1 -- 172.21.15.1:0/1392806796 wait complete. 2024-03-18T19:09:56.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.416+0000 7fd114e54700 1 Processor -- start 2024-03-18T19:09:56.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.416+0000 7fd114e54700 1 -- start start 2024-03-18T19:09:56.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.416+0000 7fd114e54700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd0f8004d90 0x7fd11014f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:56.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.416+0000 7fd114e54700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd11014ff80 0x7fd11009ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:56.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.416+0000 7fd114e54700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd110150400 0x7fd11009cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:56.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.416+0000 7fd114e54700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11009ee30 con 0x7fd110150400 2024-03-18T19:09:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.416+0000 7fd114e54700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1100c7120 con 0x7fd11014ff80 2024-03-18T19:09:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.416+0000 7fd114e54700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1100c6510 con 0x7fd0f8004d90 2024-03-18T19:09:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.416+0000 7fd10ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd110150400 0x7fd11009cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.416+0000 7fd10ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd110150400 0x7fd11009cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35144/0 (socket says 172.21.15.1:35144) 2024-03-18T19:09:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.416+0000 7fd10ed9d700 1 -- 172.21.15.1:0/3297146273 learned_addr learned my addr 172.21.15.1:0/3297146273 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.417+0000 7fd10dd9b700 1 --2- 172.21.15.1:0/3297146273 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd11014ff80 0x7fd11009ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.417+0000 7fd10e59c700 1 --2- 172.21.15.1:0/3297146273 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd0f8004d90 0x7fd11014f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.417+0000 7fd10ed9d700 1 -- 172.21.15.1:0/3297146273 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd0f8004d90 msgr2=0x7fd11014f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:56.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.417+0000 7fd10ed9d700 1 --2- 172.21.15.1:0/3297146273 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd0f8004d90 0x7fd11014f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.417+0000 7fd10ed9d700 1 -- 172.21.15.1:0/3297146273 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd11014ff80 msgr2=0x7fd11009ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.417+0000 7fd10ed9d700 1 --2- 172.21.15.1:0/3297146273 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd11014ff80 0x7fd11009ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.417+0000 7fd10ed9d700 1 -- 172.21.15.1:0/3297146273 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1100c67f0 con 0x7fd110150400 2024-03-18T19:09:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.417+0000 7fd10e59c700 1 --2- 172.21.15.1:0/3297146273 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd0f8004d90 0x7fd11014f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.417+0000 7fd10ed9d700 1 --2- 172.21.15.1:0/3297146273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd110150400 0x7fd11009cf40 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fd108016000 tx=0x7fd108009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.417+0000 7fd0feffd700 1 -- 172.21.15.1:0/3297146273 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd108007000 con 0x7fd110150400 2024-03-18T19:09:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.417+0000 7fd0feffd700 1 -- 172.21.15.1:0/3297146273 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd108009410 con 0x7fd110150400 2024-03-18T19:09:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.417+0000 7fd114e54700 1 -- 172.21.15.1:0/3297146273 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1100c6ad0 con 0x7fd110150400 2024-03-18T19:09:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.417+0000 7fd0feffd700 1 -- 172.21.15.1:0/3297146273 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd108003c60 con 0x7fd110150400 2024-03-18T19:09:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.417+0000 7fd114e54700 1 -- 172.21.15.1:0/3297146273 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd11014fa00 con 0x7fd110150400 2024-03-18T19:09:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.418+0000 7fd114e54700 1 -- 172.21.15.1:0/3297146273 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1100676a0 con 0x7fd110150400 2024-03-18T19:09:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.419+0000 7fd0feffd700 1 -- 172.21.15.1:0/3297146273 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd108024020 con 0x7fd110150400 2024-03-18T19:09:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.419+0000 7fd0feffd700 1 --2- 172.21.15.1:0/3297146273 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd0f006adb0 0x7fd0f006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.419+0000 7fd0feffd700 1 -- 172.21.15.1:0/3297146273 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5936+0+0 (secure 0 0 0) 0x7fd10808b170 con 0x7fd110150400 2024-03-18T19:09:56.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.419+0000 7fd10e59c700 1 --2- 172.21.15.1:0/3297146273 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd0f006adb0 0x7fd0f006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:56.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.420+0000 7fd10e59c700 1 --2- 172.21.15.1:0/3297146273 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd0f006adb0 0x7fd0f006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd104009b50 tx=0x7fd104008040 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:56.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.424+0000 7fd0feffd700 1 -- 172.21.15.1:0/3297146273 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd108056310 con 0x7fd110150400 2024-03-18T19:09:56.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:56.718+0000 7fd114e54700 1 -- 172.21.15.1:0/3297146273 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7fd1101e4080 con 0x7fd110150400 2024-03-18T19:09:57.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.160+0000 7fd0feffd700 1 -- 172.21.15.1:0/3297146273 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v773) v1 ==== 160+0+0 (secure 0 0 0) 0x7fd108059930 con 0x7fd110150400 2024-03-18T19:09:57.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.162+0000 7fd114e54700 1 -- 172.21.15.1:0/3297146273 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd0f006adb0 msgr2=0x7fd0f006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:57.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.162+0000 7fd114e54700 1 --2- 172.21.15.1:0/3297146273 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd0f006adb0 0x7fd0f006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd104009b50 tx=0x7fd104008040 comp rx=0 tx=0).stop 2024-03-18T19:09:57.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.162+0000 7fd114e54700 1 -- 172.21.15.1:0/3297146273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd110150400 msgr2=0x7fd11009cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:57.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.162+0000 7fd114e54700 1 --2- 172.21.15.1:0/3297146273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd110150400 0x7fd11009cf40 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fd108016000 tx=0x7fd108009750 comp rx=0 tx=0).stop 2024-03-18T19:09:57.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.162+0000 7fd114e54700 1 -- 172.21.15.1:0/3297146273 shutdown_connections 2024-03-18T19:09:57.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.162+0000 7fd114e54700 1 --2- 172.21.15.1:0/3297146273 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd0f006adb0 0x7fd0f006d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:57.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.162+0000 7fd114e54700 1 --2- 172.21.15.1:0/3297146273 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd0f8004d90 0x7fd11014f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:57.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.162+0000 7fd114e54700 1 --2- 172.21.15.1:0/3297146273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd110150400 0x7fd11009cf40 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:57.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.162+0000 7fd114e54700 1 --2- 172.21.15.1:0/3297146273 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd11014ff80 0x7fd11009ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:57.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.162+0000 7fd114e54700 1 -- 172.21.15.1:0/3297146273 >> 172.21.15.1:0/3297146273 conn(0x7fd1100b2ba0 msgr2=0x7fd1100cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:57.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.163+0000 7fd114e54700 1 -- 172.21.15.1:0/3297146273 shutdown_connections 2024-03-18T19:09:57.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.163+0000 7fd114e54700 1 -- 172.21.15.1:0/3297146273 wait complete. 2024-03-18T19:09:57.173 INFO:tasks.cephfs_test_runner:test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) ... ok 2024-03-18T19:09:57.173 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that number of snaps retained are as per user spec. 2024-03-18T19:09:57.173 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-18T19:09:57.173 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2024-03-18T19:09:57.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.308+0000 7fe0abdb1700 1 Processor -- start 2024-03-18T19:09:57.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.309+0000 7fe0abdb1700 1 -- start start 2024-03-18T19:09:57.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.310+0000 7fe0abdb1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe0a415d6b0 0x7fe0a40b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:57.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.310+0000 7fe0abdb1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a40ad790 0x7fe0a40b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:57.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.310+0000 7fe0abdb1700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a40acfb0 con 0x7fe0a406a750 2024-03-18T19:09:57.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.310+0000 7fe0abdb1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a40af7a0 con 0x7fe0a415d6b0 2024-03-18T19:09:57.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.310+0000 7fe0abdb1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a40b50a0 con 0x7fe0a40ad790 2024-03-18T19:09:57.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.310+0000 7fe0a934c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe0a415d6b0 0x7fe0a40b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.310+0000 7fe0a9b4d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe0a406a750 0x7fe0a406ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58566/0 (socket says 172.21.15.1:58566) 2024-03-18T19:09:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.310+0000 7fe0a9b4d700 1 -- 172.21.15.1:0/815349697 learned_addr learned my addr 172.21.15.1:0/815349697 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.310+0000 7fe0aa34e700 1 --2- 172.21.15.1:0/815349697 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a40ad790 0x7fe0a40b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.310+0000 7fe0a934c700 1 -- 172.21.15.1:0/815349697 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a40ad790 msgr2=0x7fe0a40b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.310+0000 7fe0a934c700 1 --2- 172.21.15.1:0/815349697 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a40ad790 0x7fe0a40b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.310+0000 7fe0a934c700 1 -- 172.21.15.1:0/815349697 >> v1:172.21.15.1:6789/0 conn(0x7fe0a406a750 legacy=0x7fe0a406ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.310+0000 7fe0a8b4b700 1 -- 172.21.15.1:0/815349697 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1372919235 0 0) 0x7fe0a40acfb0 con 0x7fe0a406a750 2024-03-18T19:09:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.310+0000 7fe0a934c700 1 -- 172.21.15.1:0/815349697 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a40af1e0 con 0x7fe0a415d6b0 2024-03-18T19:09:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.310+0000 7fe0aa34e700 1 --2- 172.21.15.1:0/815349697 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a40ad790 0x7fe0a40b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:09:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.310+0000 7fe0a934c700 1 --2- 172.21.15.1:0/815349697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a415d6b0 0x7fe0a40b2da0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fe098009840 tx=0x7fe098009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5c5f0f3f587dde3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.310+0000 7fe0a8b4b700 1 -- 172.21.15.1:0/815349697 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe098011830 con 0x7fe0a415d6b0 2024-03-18T19:09:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.310+0000 7fe0a8b4b700 1 -- 172.21.15.1:0/815349697 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe098011990 con 0x7fe0a415d6b0 2024-03-18T19:09:57.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.310+0000 7fe0a8b4b700 1 -- 172.21.15.1:0/815349697 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe098011d20 con 0x7fe0a415d6b0 2024-03-18T19:09:57.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.311+0000 7fe0abdb1700 1 -- 172.21.15.1:0/815349697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a415d6b0 msgr2=0x7fe0a40b2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:57.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.311+0000 7fe0abdb1700 1 --2- 172.21.15.1:0/815349697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a415d6b0 0x7fe0a40b2da0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fe098009840 tx=0x7fe098009b50 comp rx=0 tx=0).stop 2024-03-18T19:09:57.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.311+0000 7fe0abdb1700 1 -- 172.21.15.1:0/815349697 shutdown_connections 2024-03-18T19:09:57.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.311+0000 7fe0abdb1700 1 --2- 172.21.15.1:0/815349697 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a40ad790 0x7fe0a40b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:57.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.311+0000 7fe0abdb1700 1 --2- 172.21.15.1:0/815349697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a415d6b0 0x7fe0a40b2da0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:57.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.311+0000 7fe0abdb1700 1 -- 172.21.15.1:0/815349697 >> 172.21.15.1:0/815349697 conn(0x7fe0a406b860 msgr2=0x7fe0a406bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:57.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.311+0000 7fe0abdb1700 1 -- 172.21.15.1:0/815349697 shutdown_connections 2024-03-18T19:09:57.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.311+0000 7fe0abdb1700 1 -- 172.21.15.1:0/815349697 wait complete. 2024-03-18T19:09:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.312+0000 7fe0abdb1700 1 Processor -- start 2024-03-18T19:09:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.312+0000 7fe0abdb1700 1 -- start start 2024-03-18T19:09:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.313+0000 7fe0abdb1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0a406a750 0x7fe0a415ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.313+0000 7fe0abdb1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a40ad790 0x7fe0a40ce830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.313+0000 7fe0abdb1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a415d6b0 0x7fe0a4157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.313+0000 7fe0abdb1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a40cbff0 con 0x7fe0a40ad790 2024-03-18T19:09:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.313+0000 7fe0abdb1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a415b7f0 con 0x7fe0a406a750 2024-03-18T19:09:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.313+0000 7fe0abdb1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a415ba90 con 0x7fe0a415d6b0 2024-03-18T19:09:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.313+0000 7fe0a934c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a40ad790 0x7fe0a40ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.313+0000 7fe0a934c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a40ad790 0x7fe0a40ce830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35170/0 (socket says 172.21.15.1:35170) 2024-03-18T19:09:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.313+0000 7fe0a934c700 1 -- 172.21.15.1:0/1986064197 learned_addr learned my addr 172.21.15.1:0/1986064197 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.313+0000 7fe0a9b4d700 1 --2- 172.21.15.1:0/1986064197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0a406a750 0x7fe0a415ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.313+0000 7fe0aa34e700 1 --2- 172.21.15.1:0/1986064197 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a415d6b0 0x7fe0a4157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:57.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.313+0000 7fe0a934c700 1 -- 172.21.15.1:0/1986064197 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a415d6b0 msgr2=0x7fe0a4157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.313+0000 7fe0a934c700 1 --2- 172.21.15.1:0/1986064197 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a415d6b0 0x7fe0a4157ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.313+0000 7fe0a934c700 1 -- 172.21.15.1:0/1986064197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0a406a750 msgr2=0x7fe0a415ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.313+0000 7fe0a934c700 1 --2- 172.21.15.1:0/1986064197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0a406a750 0x7fe0a415ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.313+0000 7fe0a934c700 1 -- 172.21.15.1:0/1986064197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0980094a0 con 0x7fe0a40ad790 2024-03-18T19:09:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.314+0000 7fe0a934c700 1 --2- 172.21.15.1:0/1986064197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a40ad790 0x7fe0a40ce830 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fe098014040 tx=0x7fe098004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.314+0000 7fe0967fc700 1 -- 172.21.15.1:0/1986064197 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0980036a0 con 0x7fe0a40ad790 2024-03-18T19:09:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.314+0000 7fe0967fc700 1 -- 172.21.15.1:0/1986064197 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09801e070 con 0x7fe0a40ad790 2024-03-18T19:09:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.314+0000 7fe0967fc700 1 -- 172.21.15.1:0/1986064197 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe098003a40 con 0x7fe0a40ad790 2024-03-18T19:09:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.314+0000 7fe0abdb1700 1 -- 172.21.15.1:0/1986064197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a415bd70 con 0x7fe0a40ad790 2024-03-18T19:09:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.314+0000 7fe0abdb1700 1 -- 172.21.15.1:0/1986064197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a40cef30 con 0x7fe0a40ad790 2024-03-18T19:09:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.315+0000 7fe0abdb1700 1 -- 172.21.15.1:0/1986064197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a415bf30 con 0x7fe0a40ad790 2024-03-18T19:09:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.315+0000 7fe0967fc700 1 -- 172.21.15.1:0/1986064197 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe098003ba0 con 0x7fe0a40ad790 2024-03-18T19:09:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.316+0000 7fe0967fc700 1 --2- 172.21.15.1:0/1986064197 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe08806adb0 0x7fe08806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.316+0000 7fe0967fc700 1 -- 172.21.15.1:0/1986064197 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5936+0+0 (secure 0 0 0) 0x7fe09808a530 con 0x7fe0a40ad790 2024-03-18T19:09:57.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.316+0000 7fe0a9b4d700 1 --2- 172.21.15.1:0/1986064197 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe08806adb0 0x7fe08806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:57.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.316+0000 7fe0a9b4d700 1 --2- 172.21.15.1:0/1986064197 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe08806adb0 0x7fe08806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0007fd0 tx=0x7fe0a0007de0 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:57.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.320+0000 7fe0967fc700 1 -- 172.21.15.1:0/1986064197 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe0980555b0 con 0x7fe0a40ad790 2024-03-18T19:09:57.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:57.603+0000 7fe0abdb1700 1 -- 172.21.15.1:0/1986064197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7fe0a40ad420 con 0x7fe0a40ad790 2024-03-18T19:09:58.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.162+0000 7fe0967fc700 1 -- 172.21.15.1:0/1986064197 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v774) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe0a40ad420 con 0x7fe0a40ad790 2024-03-18T19:09:58.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.165+0000 7fe0abdb1700 1 -- 172.21.15.1:0/1986064197 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe08806adb0 msgr2=0x7fe08806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:58.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.165+0000 7fe0abdb1700 1 --2- 172.21.15.1:0/1986064197 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe08806adb0 0x7fe08806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0007fd0 tx=0x7fe0a0007de0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.165+0000 7fe0abdb1700 1 -- 172.21.15.1:0/1986064197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a40ad790 msgr2=0x7fe0a40ce830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:58.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.165+0000 7fe0abdb1700 1 --2- 172.21.15.1:0/1986064197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a40ad790 0x7fe0a40ce830 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fe098014040 tx=0x7fe098004750 comp rx=0 tx=0).stop 2024-03-18T19:09:58.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.165+0000 7fe0abdb1700 1 -- 172.21.15.1:0/1986064197 shutdown_connections 2024-03-18T19:09:58.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.165+0000 7fe0abdb1700 1 --2- 172.21.15.1:0/1986064197 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe08806adb0 0x7fe08806d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.165+0000 7fe0abdb1700 1 --2- 172.21.15.1:0/1986064197 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0a415d6b0 0x7fe0a4157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.165+0000 7fe0abdb1700 1 --2- 172.21.15.1:0/1986064197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0a406a750 0x7fe0a415ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.165+0000 7fe0abdb1700 1 --2- 172.21.15.1:0/1986064197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0a40ad790 0x7fe0a40ce830 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.165+0000 7fe0abdb1700 1 -- 172.21.15.1:0/1986064197 >> 172.21.15.1:0/1986064197 conn(0x7fe0a406b860 msgr2=0x7fe0a41e8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:58.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.165+0000 7fe0abdb1700 1 -- 172.21.15.1:0/1986064197 shutdown_connections 2024-03-18T19:09:58.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.165+0000 7fe0abdb1700 1 -- 172.21.15.1:0/1986064197 wait complete. 2024-03-18T19:09:58.176 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-18T19:09:58.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.315+0000 7fb350037700 1 Processor -- start 2024-03-18T19:09:58.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.315+0000 7fb350037700 1 -- start start 2024-03-18T19:09:58.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.316+0000 7fb350037700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb348071d30 0x7fb3480add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:58.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.316+0000 7fb350037700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb3480cb9e0 0x7fb3480cbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:58.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.316+0000 7fb350037700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3480ce8f0 con 0x7fb34815a1d0 2024-03-18T19:09:58.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.316+0000 7fb350037700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3480cdb00 con 0x7fb3480cb9e0 2024-03-18T19:09:58.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.316+0000 7fb350037700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb34815a5b0 con 0x7fb348071d30 2024-03-18T19:09:58.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.316+0000 7fb34e5d4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb3480cb9e0 0x7fb3480cbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:58.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.316+0000 7fb34e5d4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb3480cb9e0 0x7fb3480cbe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35176/0 (socket says 172.21.15.1:35176) 2024-03-18T19:09:58.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.316+0000 7fb34e5d4700 1 -- 172.21.15.1:0/1772933454 learned_addr learned my addr 172.21.15.1:0/1772933454 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:58.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.316+0000 7fb34d5d2700 1 --2- 172.21.15.1:0/1772933454 >> v2:172.21.15.165:3300/0 conn(0x7fb348071d30 0x7fb3480add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:58.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.316+0000 7fb34e5d4700 1 -- 172.21.15.1:0/1772933454 >> v2:172.21.15.165:3300/0 conn(0x7fb348071d30 msgr2=0x7fb3480add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:58.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.316+0000 7fb34e5d4700 1 --2- 172.21.15.1:0/1772933454 >> v2:172.21.15.165:3300/0 conn(0x7fb348071d30 0x7fb3480add20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.316+0000 7fb34e5d4700 1 -- 172.21.15.1:0/1772933454 >> v1:172.21.15.165:6789/0 conn(0x7fb34815a1d0 legacy=0x7fb348071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:58.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.316+0000 7fb34e5d4700 1 -- 172.21.15.1:0/1772933454 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3480ae260 con 0x7fb3480cb9e0 2024-03-18T19:09:58.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.317+0000 7fb34e5d4700 1 --2- 172.21.15.1:0/1772933454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3480cb9e0 0x7fb3480cbe30 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fb3480cebb0 tx=0x7fb34000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=630e3f396c7a1f53 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:58.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.317+0000 7fb34cdd1700 1 -- 172.21.15.1:0/1772933454 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34000e070 con 0x7fb3480cb9e0 2024-03-18T19:09:58.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.317+0000 7fb34cdd1700 1 -- 172.21.15.1:0/1772933454 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb340003a20 con 0x7fb3480cb9e0 2024-03-18T19:09:58.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.317+0000 7fb34cdd1700 1 -- 172.21.15.1:0/1772933454 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb340003dd0 con 0x7fb3480cb9e0 2024-03-18T19:09:58.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.317+0000 7fb350037700 1 -- 172.21.15.1:0/1772933454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3480cb9e0 msgr2=0x7fb3480cbe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:58.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.317+0000 7fb350037700 1 --2- 172.21.15.1:0/1772933454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3480cb9e0 0x7fb3480cbe30 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fb3480cebb0 tx=0x7fb34000b330 comp rx=0 tx=0).stop 2024-03-18T19:09:58.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.317+0000 7fb350037700 1 -- 172.21.15.1:0/1772933454 shutdown_connections 2024-03-18T19:09:58.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.317+0000 7fb350037700 1 --2- 172.21.15.1:0/1772933454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3480cb9e0 0x7fb3480cbe30 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.317+0000 7fb350037700 1 --2- 172.21.15.1:0/1772933454 >> v2:172.21.15.165:3300/0 conn(0x7fb348071d30 0x7fb3480add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.317+0000 7fb350037700 1 -- 172.21.15.1:0/1772933454 >> 172.21.15.1:0/1772933454 conn(0x7fb3480cc9b0 msgr2=0x7fb3480ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:58.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.317+0000 7fb350037700 1 -- 172.21.15.1:0/1772933454 shutdown_connections 2024-03-18T19:09:58.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.317+0000 7fb350037700 1 -- 172.21.15.1:0/1772933454 wait complete. 2024-03-18T19:09:58.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.319+0000 7fb350037700 1 Processor -- start 2024-03-18T19:09:58.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.319+0000 7fb350037700 1 -- start start 2024-03-18T19:09:58.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.319+0000 7fb350037700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb348071d30 0x7fb3480c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:58.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.319+0000 7fb350037700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3480cb9e0 0x7fb3480c7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:58.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.319+0000 7fb350037700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb34815a1d0 0x7fb3480c5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:58.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.319+0000 7fb350037700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480c4c30 con 0x7fb348071d30 2024-03-18T19:09:58.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.319+0000 7fb350037700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480c3fb0 con 0x7fb3480cb9e0 2024-03-18T19:09:58.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.319+0000 7fb350037700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480c4290 con 0x7fb34815a1d0 2024-03-18T19:09:58.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.319+0000 7fb34ddd3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb348071d30 0x7fb3480c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:58.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.319+0000 7fb34ddd3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb348071d30 0x7fb3480c9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35184/0 (socket says 172.21.15.1:35184) 2024-03-18T19:09:58.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.319+0000 7fb34ddd3700 1 -- 172.21.15.1:0/3067389582 learned_addr learned my addr 172.21.15.1:0/3067389582 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:58.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.320+0000 7fb34d5d2700 1 --2- 172.21.15.1:0/3067389582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3480cb9e0 0x7fb3480c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:58.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.320+0000 7fb34e5d4700 1 --2- 172.21.15.1:0/3067389582 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb34815a1d0 0x7fb3480c5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:58.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.320+0000 7fb34ddd3700 1 -- 172.21.15.1:0/3067389582 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb34815a1d0 msgr2=0x7fb3480c5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:58.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.320+0000 7fb34ddd3700 1 --2- 172.21.15.1:0/3067389582 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb34815a1d0 0x7fb3480c5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.320+0000 7fb34ddd3700 1 -- 172.21.15.1:0/3067389582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3480cb9e0 msgr2=0x7fb3480c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:58.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.320+0000 7fb34ddd3700 1 --2- 172.21.15.1:0/3067389582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3480cb9e0 0x7fb3480c7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.320+0000 7fb34ddd3700 1 -- 172.21.15.1:0/3067389582 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb340009b30 con 0x7fb348071d30 2024-03-18T19:09:58.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.320+0000 7fb34ddd3700 1 --2- 172.21.15.1:0/3067389582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb348071d30 0x7fb3480c9fa0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fb33800c4b0 tx=0x7fb33800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:58.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.320+0000 7fb33e7fc700 1 -- 172.21.15.1:0/3067389582 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338011790 con 0x7fb348071d30 2024-03-18T19:09:58.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.320+0000 7fb33e7fc700 1 -- 172.21.15.1:0/3067389582 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3380118f0 con 0x7fb348071d30 2024-03-18T19:09:58.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.320+0000 7fb350037700 1 -- 172.21.15.1:0/3067389582 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3480c4570 con 0x7fb348071d30 2024-03-18T19:09:58.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.320+0000 7fb33e7fc700 1 -- 172.21.15.1:0/3067389582 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338011ca0 con 0x7fb348071d30 2024-03-18T19:09:58.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.320+0000 7fb350037700 1 -- 172.21.15.1:0/3067389582 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3480adfb0 con 0x7fb348071d30 2024-03-18T19:09:58.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.321+0000 7fb350037700 1 -- 172.21.15.1:0/3067389582 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3480c4ea0 con 0x7fb348071d30 2024-03-18T19:09:58.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.322+0000 7fb33e7fc700 1 -- 172.21.15.1:0/3067389582 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb338012420 con 0x7fb348071d30 2024-03-18T19:09:58.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.322+0000 7fb33e7fc700 1 --2- 172.21.15.1:0/3067389582 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb32c06adb0 0x7fb32c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:58.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.322+0000 7fb33e7fc700 1 -- 172.21.15.1:0/3067389582 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5936+0+0 (secure 0 0 0) 0x7fb338088f00 con 0x7fb348071d30 2024-03-18T19:09:58.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.322+0000 7fb34d5d2700 1 --2- 172.21.15.1:0/3067389582 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb32c06adb0 0x7fb32c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:58.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.323+0000 7fb34d5d2700 1 --2- 172.21.15.1:0/3067389582 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb32c06adb0 0x7fb32c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb3480ca4e0 tx=0x7fb344008040 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:58.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.327+0000 7fb33e7fc700 1 -- 172.21.15.1:0/3067389582 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb338053f80 con 0x7fb348071d30 2024-03-18T19:09:58.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.529+0000 7fb350037700 1 -- 172.21.15.1:0/3067389582 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb3480676a0 con 0x7fb348071d30 2024-03-18T19:09:58.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.532+0000 7fb33e7fc700 1 -- 172.21.15.1:0/3067389582 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb3380575a0 con 0x7fb348071d30 2024-03-18T19:09:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.535+0000 7fb350037700 1 -- 172.21.15.1:0/3067389582 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb32c06adb0 msgr2=0x7fb32c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.535+0000 7fb350037700 1 --2- 172.21.15.1:0/3067389582 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb32c06adb0 0x7fb32c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb3480ca4e0 tx=0x7fb344008040 comp rx=0 tx=0).stop 2024-03-18T19:09:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.535+0000 7fb350037700 1 -- 172.21.15.1:0/3067389582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb348071d30 msgr2=0x7fb3480c9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.535+0000 7fb350037700 1 --2- 172.21.15.1:0/3067389582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb348071d30 0x7fb3480c9fa0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fb33800c4b0 tx=0x7fb33800c7c0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.535+0000 7fb350037700 1 -- 172.21.15.1:0/3067389582 shutdown_connections 2024-03-18T19:09:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.535+0000 7fb350037700 1 --2- 172.21.15.1:0/3067389582 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb32c06adb0 0x7fb32c06d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.535+0000 7fb350037700 1 --2- 172.21.15.1:0/3067389582 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb34815a1d0 0x7fb3480c5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.535+0000 7fb350037700 1 --2- 172.21.15.1:0/3067389582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb348071d30 0x7fb3480c9fa0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.535+0000 7fb350037700 1 --2- 172.21.15.1:0/3067389582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3480cb9e0 0x7fb3480c7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.535+0000 7fb350037700 1 -- 172.21.15.1:0/3067389582 >> 172.21.15.1:0/3067389582 conn(0x7fb3480cc9b0 msgr2=0x7fb3480ca540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.535+0000 7fb350037700 1 -- 172.21.15.1:0/3067389582 shutdown_connections 2024-03-18T19:09:58.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.535+0000 7fb350037700 1 -- 172.21.15.1:0/3067389582 wait complete. 2024-03-18T19:09:58.543 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T19:09:58.543 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:09:58.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T19:09:58.635 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T19:09:58.636 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:09:58.636 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:09:58.656 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:09:58.656 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:09:58.657 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:09:58.678 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:09:58.679 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T19:09:58.679 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:09:58.715 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T19:09:58.715 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T19:09:58.715 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T19:09:58.739 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T19:09:58.739 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-18T19:09:58.740 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:09:58.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.884+0000 7fc0f79d0700 1 Processor -- start 2024-03-18T19:09:58.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.885+0000 7fc0f79d0700 1 -- start start 2024-03-18T19:09:58.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.885+0000 7fc0f79d0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0f00d05a0 0x7fc0f009a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:58.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.885+0000 7fc0f79d0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc0f006ba70 0x7fc0f015d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:58.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.885+0000 7fc0f79d0700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f00cf7e0 con 0x7fc0f0099a40 2024-03-18T19:09:58.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.885+0000 7fc0f79d0700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f00cfac0 con 0x7fc0f006ba70 2024-03-18T19:09:58.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.885+0000 7fc0f79d0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f00cfda0 con 0x7fc0f00d05a0 2024-03-18T19:09:58.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.886+0000 7fc0f5f6d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc0f006ba70 0x7fc0f015d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:58.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.886+0000 7fc0f5f6d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc0f006ba70 0x7fc0f015d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35200/0 (socket says 172.21.15.1:35200) 2024-03-18T19:09:58.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.886+0000 7fc0f5f6d700 1 -- 172.21.15.1:0/1028044822 learned_addr learned my addr 172.21.15.1:0/1028044822 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:58.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.886+0000 7fc0f4f6b700 1 --2- 172.21.15.1:0/1028044822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0f00d05a0 0x7fc0f009a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:58.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.886+0000 7fc0f5f6d700 1 -- 172.21.15.1:0/1028044822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0f00d05a0 msgr2=0x7fc0f009a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:58.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.886+0000 7fc0f5f6d700 1 --2- 172.21.15.1:0/1028044822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0f00d05a0 0x7fc0f009a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.886+0000 7fc0f5f6d700 1 -- 172.21.15.1:0/1028044822 >> v1:172.21.15.165:6789/0 conn(0x7fc0f0099a40 legacy=0x7fc0f0099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:58.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.886+0000 7fc0f5f6d700 1 -- 172.21.15.1:0/1028044822 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f00d0080 con 0x7fc0f006ba70 2024-03-18T19:09:58.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.886+0000 7fc0f5f6d700 1 --2- 172.21.15.1:0/1028044822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0f006ba70 0x7fc0f015d940 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8009e50 tx=0x7fc0e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29d95882511ec9cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:58.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.886+0000 7fc0e3fff700 1 -- 172.21.15.1:0/1028044822 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e800e070 con 0x7fc0f006ba70 2024-03-18T19:09:58.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.886+0000 7fc0e3fff700 1 -- 172.21.15.1:0/1028044822 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e800b910 con 0x7fc0f006ba70 2024-03-18T19:09:58.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.886+0000 7fc0e3fff700 1 -- 172.21.15.1:0/1028044822 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e800bcc0 con 0x7fc0f006ba70 2024-03-18T19:09:58.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.887+0000 7fc0f79d0700 1 -- 172.21.15.1:0/1028044822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0f006ba70 msgr2=0x7fc0f015d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:58.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.887+0000 7fc0f79d0700 1 --2- 172.21.15.1:0/1028044822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0f006ba70 0x7fc0f015d940 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8009e50 tx=0x7fc0e800b330 comp rx=0 tx=0).stop 2024-03-18T19:09:58.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.887+0000 7fc0f79d0700 1 -- 172.21.15.1:0/1028044822 shutdown_connections 2024-03-18T19:09:58.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.887+0000 7fc0f79d0700 1 --2- 172.21.15.1:0/1028044822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0f006ba70 0x7fc0f015d940 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.887+0000 7fc0f79d0700 1 --2- 172.21.15.1:0/1028044822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0f00d05a0 0x7fc0f009a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.887+0000 7fc0f79d0700 1 -- 172.21.15.1:0/1028044822 >> 172.21.15.1:0/1028044822 conn(0x7fc0f00b2cb0 msgr2=0x7fc0f00b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:58.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.887+0000 7fc0f79d0700 1 -- 172.21.15.1:0/1028044822 shutdown_connections 2024-03-18T19:09:58.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.887+0000 7fc0f79d0700 1 -- 172.21.15.1:0/1028044822 wait complete. 2024-03-18T19:09:58.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.888+0000 7fc0f79d0700 1 Processor -- start 2024-03-18T19:09:58.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.888+0000 7fc0f79d0700 1 -- start start 2024-03-18T19:09:58.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.889+0000 7fc0f79d0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0f006ba70 0x7fc0f01542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:58.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.889+0000 7fc0f79d0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0f0099a40 0x7fc0f01f3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:58.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.889+0000 7fc0f79d0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0f00d05a0 0x7fc0f00aef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:58.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.889+0000 7fc0f79d0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f01f3b50 con 0x7fc0f00d05a0 2024-03-18T19:09:58.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.889+0000 7fc0f79d0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f01f6140 con 0x7fc0f006ba70 2024-03-18T19:09:58.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.889+0000 7fc0f79d0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f01f6420 con 0x7fc0f0099a40 2024-03-18T19:09:58.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.889+0000 7fc0f5f6d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0f00d05a0 0x7fc0f00aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:58.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.889+0000 7fc0f5f6d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0f00d05a0 0x7fc0f00aef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35214/0 (socket says 172.21.15.1:35214) 2024-03-18T19:09:58.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.889+0000 7fc0f5f6d700 1 -- 172.21.15.1:0/4204601422 learned_addr learned my addr 172.21.15.1:0/4204601422 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:58.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.889+0000 7fc0f576c700 1 --2- 172.21.15.1:0/4204601422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0f006ba70 0x7fc0f01542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:58.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.889+0000 7fc0f4f6b700 1 --2- 172.21.15.1:0/4204601422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0f0099a40 0x7fc0f01f3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:58.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.889+0000 7fc0f5f6d700 1 -- 172.21.15.1:0/4204601422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0f0099a40 msgr2=0x7fc0f01f3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:58.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.889+0000 7fc0f5f6d700 1 --2- 172.21.15.1:0/4204601422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0f0099a40 0x7fc0f01f3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.889+0000 7fc0f5f6d700 1 -- 172.21.15.1:0/4204601422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0f006ba70 msgr2=0x7fc0f01542d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:58.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.889+0000 7fc0f5f6d700 1 --2- 172.21.15.1:0/4204601422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0f006ba70 0x7fc0f01542d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:58.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.889+0000 7fc0f5f6d700 1 -- 172.21.15.1:0/4204601422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e8009b30 con 0x7fc0f00d05a0 2024-03-18T19:09:58.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.889+0000 7fc0f5f6d700 1 --2- 172.21.15.1:0/4204601422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0f00d05a0 0x7fc0f00aef80 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0072230 tx=0x7fc0e8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:58.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.889+0000 7fc0e1ffb700 1 -- 172.21.15.1:0/4204601422 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8004800 con 0x7fc0f00d05a0 2024-03-18T19:09:58.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.890+0000 7fc0e1ffb700 1 -- 172.21.15.1:0/4204601422 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e8004960 con 0x7fc0f00d05a0 2024-03-18T19:09:58.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.890+0000 7fc0e1ffb700 1 -- 172.21.15.1:0/4204601422 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e801c680 con 0x7fc0f00d05a0 2024-03-18T19:09:58.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.890+0000 7fc0f79d0700 1 -- 172.21.15.1:0/4204601422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f01f6700 con 0x7fc0f00d05a0 2024-03-18T19:09:58.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.890+0000 7fc0f79d0700 1 -- 172.21.15.1:0/4204601422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f01f6cc0 con 0x7fc0f00d05a0 2024-03-18T19:09:58.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.891+0000 7fc0f79d0700 1 -- 172.21.15.1:0/4204601422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f0068240 con 0x7fc0f00d05a0 2024-03-18T19:09:58.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.891+0000 7fc0e1ffb700 1 -- 172.21.15.1:0/4204601422 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc0e801c7e0 con 0x7fc0f00d05a0 2024-03-18T19:09:58.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.891+0000 7fc0e1ffb700 1 --2- 172.21.15.1:0/4204601422 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc0d406adb0 0x7fc0d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:58.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.892+0000 7fc0e1ffb700 1 -- 172.21.15.1:0/4204601422 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5936+0+0 (secure 0 0 0) 0x7fc0e808b1c0 con 0x7fc0f00d05a0 2024-03-18T19:09:58.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.892+0000 7fc0f576c700 1 --2- 172.21.15.1:0/4204601422 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc0d406adb0 0x7fc0d406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:58.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.892+0000 7fc0f576c700 1 --2- 172.21.15.1:0/4204601422 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc0d406adb0 0x7fc0d406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc0f015dd00 tx=0x7fc0e4009040 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:58.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:58.896+0000 7fc0e1ffb700 1 -- 172.21.15.1:0/4204601422 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc0e8056130 con 0x7fc0f00d05a0 2024-03-18T19:09:59.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.145+0000 7fc0f79d0700 1 -- 172.21.15.1:0/4204601422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0f00cda40 con 0x7fc0f00d05a0 2024-03-18T19:09:59.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.145+0000 7fc0e1ffb700 1 -- 172.21.15.1:0/4204601422 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+3523 (secure 0 0 0) 0x7fc0e8059750 con 0x7fc0f00d05a0 2024-03-18T19:09:59.142 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:09:59.142 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":67,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8116,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3723189348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3723189348},{"type":"v1","addr":"172.21.15.1:6837","nonce":3723189348}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":8125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/530206363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":530206363},{"type":"v1","addr":"172.21.15.165:6835","nonce":530206363}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60},{"gid":8524,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/2052416572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":2052416572},{"type":"v1","addr":"172.21.15.165:6834","nonce":2052416572}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":67},{"gid":8784,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/332827595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":332827595},{"type":"v1","addr":"172.21.15.1:6836","nonce":332827595}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[]} 2024-03-18T19:09:59.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.148+0000 7fc0f79d0700 1 -- 172.21.15.1:0/4204601422 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc0d406adb0 msgr2=0x7fc0d406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:59.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.148+0000 7fc0f79d0700 1 --2- 172.21.15.1:0/4204601422 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc0d406adb0 0x7fc0d406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc0f015dd00 tx=0x7fc0e4009040 comp rx=0 tx=0).stop 2024-03-18T19:09:59.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.148+0000 7fc0f79d0700 1 -- 172.21.15.1:0/4204601422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0f00d05a0 msgr2=0x7fc0f00aef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:59.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.148+0000 7fc0f79d0700 1 --2- 172.21.15.1:0/4204601422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0f00d05a0 0x7fc0f00aef80 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0072230 tx=0x7fc0e8004ad0 comp rx=0 tx=0).stop 2024-03-18T19:09:59.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.148+0000 7fc0f79d0700 1 -- 172.21.15.1:0/4204601422 shutdown_connections 2024-03-18T19:09:59.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.148+0000 7fc0f79d0700 1 --2- 172.21.15.1:0/4204601422 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc0d406adb0 0x7fc0d406d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:59.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.148+0000 7fc0f79d0700 1 --2- 172.21.15.1:0/4204601422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0f00d05a0 0x7fc0f00aef80 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:59.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.148+0000 7fc0f79d0700 1 --2- 172.21.15.1:0/4204601422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0f006ba70 0x7fc0f01542d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:59.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.148+0000 7fc0f79d0700 1 --2- 172.21.15.1:0/4204601422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0f0099a40 0x7fc0f01f3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:59.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.148+0000 7fc0f79d0700 1 -- 172.21.15.1:0/4204601422 >> 172.21.15.1:0/4204601422 conn(0x7fc0f00b2cb0 msgr2=0x7fc0f015e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:59.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.148+0000 7fc0f79d0700 1 -- 172.21.15.1:0/4204601422 shutdown_connections 2024-03-18T19:09:59.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.148+0000 7fc0f79d0700 1 -- 172.21.15.1:0/4204601422 wait complete. 2024-03-18T19:09:59.145 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 67 2024-03-18T19:09:59.158 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T19:09:59.159 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-18T19:09:59.159 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T19:09:59.159 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:09:59.159 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T19:09:59.159 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-18T19:09:59.159 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-18T19:09:59.159 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:09:59.160 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T19:09:59.160 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-18T19:09:59.160 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-18T19:09:59.160 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:09:59.160 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T19:09:59.160 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-18T19:09:59.161 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-18T19:09:59.161 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T19:09:59.167 INFO:tasks.ceph.mds.a:Stopped 2024-03-18T19:09:59.168 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-18T19:09:59.168 INFO:tasks.ceph.mds.c:Stopped 2024-03-18T19:09:59.169 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-18T19:09:59.171 INFO:tasks.ceph.mds.b:Stopped 2024-03-18T19:09:59.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-18T19:09:59.172 INFO:tasks.ceph.mds.d:Stopped 2024-03-18T19:09:59.172 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-18T19:09:59.210 INFO:tasks.ceph.mds.a:Started 2024-03-18T19:09:59.211 DEBUG:teuthology.parallel:result is None 2024-03-18T19:09:59.211 INFO:tasks.ceph.mds.c:Started 2024-03-18T19:09:59.211 DEBUG:teuthology.parallel:result is None 2024-03-18T19:09:59.214 INFO:tasks.ceph.mds.b:Started 2024-03-18T19:09:59.214 DEBUG:teuthology.parallel:result is None 2024-03-18T19:09:59.214 INFO:tasks.ceph.mds.d:Started 2024-03-18T19:09:59.215 DEBUG:teuthology.parallel:result is None 2024-03-18T19:09:59.215 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-18T19:09:59.293 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T19:09:59.294 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-18T19:09:59.308 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-03-18T19:09:59.308 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-03-18T19:09:59.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.354+0000 7f5ff24c3700 1 Processor -- start 2024-03-18T19:09:59.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.354+0000 7f5ff24c3700 1 -- start start 2024-03-18T19:09:59.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5ff24c3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5fec15b480 0x7f5fec0cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:59.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5ff24c3700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fec0c4c50 con 0x7f5fec0cfdd0 2024-03-18T19:09:59.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5ff24c3700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fec0c5210 con 0x7f5fec0cc7b0 2024-03-18T19:09:59.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5ff24c3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec0cccb0 con 0x7f5fec15b480 2024-03-18T19:09:59.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5ff0a60700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5fec0cfdd0 0x7f5fec0c56a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58602/0 (socket says 172.21.15.1:58602) 2024-03-18T19:09:59.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5ff0a60700 1 -- 172.21.15.1:0/3740178310 learned_addr learned my addr 172.21.15.1:0/3740178310 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:09:59.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5feaffd700 1 -- 172.21.15.1:0/3740178310 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576889079 0 0) 0x7f5fec0c4c50 con 0x7f5fec0cfdd0 2024-03-18T19:09:59.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5febfff700 1 --2- 172.21.15.1:0/3740178310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5fec15b480 0x7f5fec0cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:59.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5feaffd700 1 -- 172.21.15.1:0/3740178310 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd40035b0 con 0x7f5fec0cfdd0 2024-03-18T19:09:59.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5feaffd700 1 -- 172.21.15.1:0/3740178310 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3022405491 0 0) 0x7f5fd40035b0 con 0x7f5fec0cfdd0 2024-03-18T19:09:59.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5feaffd700 1 -- 172.21.15.1:0/3740178310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5fec15b480 msgr2=0x7f5fec0cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:59.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5feaffd700 1 --2- 172.21.15.1:0/3740178310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5fec15b480 0x7f5fec0cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:59.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5feaffd700 1 -- 172.21.15.1:0/3740178310 >> v1:172.21.15.165:6789/0 conn(0x7f5fec0cc7b0 legacy=0x7f5fec0cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:59.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5feaffd700 1 -- 172.21.15.1:0/3740178310 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec0ccf90 con 0x7f5fec0cfdd0 2024-03-18T19:09:59.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5feaffd700 1 -- 172.21.15.1:0/3740178310 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f5fe40019f0 con 0x7f5fec0cfdd0 2024-03-18T19:09:59.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5feaffd700 1 -- 172.21.15.1:0/3740178310 >> v1:172.21.15.1:6789/0 conn(0x7f5fec0cfdd0 legacy=0x7f5fec0c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:59.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5feaffd700 1 --2- 172.21.15.1:0/3740178310 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd4003a30 0x7f5fd4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:59.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5feaffd700 1 --2- 172.21.15.1:0/3740178310 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5fd4004850 0x7f5fd4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:59.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5feaffd700 1 --2- 172.21.15.1:0/3740178310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5fec15b480 0x7f5fd4005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:59.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5feaffd700 1 -- 172.21.15.1:0/3740178310 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec0cccb0 con 0x7f5fd4003a30 2024-03-18T19:09:59.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5feaffd700 1 -- 172.21.15.1:0/3740178310 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd400bb40 con 0x7f5fd4004850 2024-03-18T19:09:59.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5feaffd700 1 -- 172.21.15.1:0/3740178310 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd400bd00 con 0x7f5fec15b480 2024-03-18T19:09:59.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.355+0000 7f5febfff700 1 --2- 172.21.15.1:0/3740178310 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd4003a30 0x7f5fd4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:59.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.356+0000 7f5feb7fe700 1 --2- 172.21.15.1:0/3740178310 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5fd4004850 0x7f5fd4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:59.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.356+0000 7f5ff0a60700 1 --2- 172.21.15.1:0/3740178310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5fec15b480 0x7f5fd4005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:59.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.356+0000 7f5febfff700 1 -- 172.21.15.1:0/3740178310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5fec15b480 msgr2=0x7f5fd4005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:59.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.356+0000 7f5febfff700 1 --2- 172.21.15.1:0/3740178310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5fec15b480 0x7f5fd4005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:59.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.356+0000 7f5febfff700 1 -- 172.21.15.1:0/3740178310 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5fd4004850 msgr2=0x7f5fd4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:09:59.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.356+0000 7f5febfff700 1 --2- 172.21.15.1:0/3740178310 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5fd4004850 0x7f5fd4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:59.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.356+0000 7f5febfff700 1 -- 172.21.15.1:0/3740178310 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd40035b0 con 0x7f5fd4003a30 2024-03-18T19:09:59.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.356+0000 7f5febfff700 1 --2- 172.21.15.1:0/3740178310 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd4003a30 0x7f5fd4003e50 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0009b60 tx=0x7f5fd4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75021c4ae7184ef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.356+0000 7f5feaffd700 1 -- 172.21.15.1:0/3740178310 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe000e6c0 con 0x7f5fd4003a30 2024-03-18T19:09:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.356+0000 7f5feaffd700 1 -- 172.21.15.1:0/3740178310 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe000e820 con 0x7f5fd4003a30 2024-03-18T19:09:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.356+0000 7f5feaffd700 1 -- 172.21.15.1:0/3740178310 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe000eb90 con 0x7f5fd4003a30 2024-03-18T19:09:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.357+0000 7f5ff24c3700 1 -- 172.21.15.1:0/3740178310 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd4003a30 msgr2=0x7f5fd4003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.357+0000 7f5ff24c3700 1 --2- 172.21.15.1:0/3740178310 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd4003a30 0x7f5fd4003e50 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0009b60 tx=0x7f5fd4005880 comp rx=0 tx=0).stop 2024-03-18T19:09:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.357+0000 7f5ff0a60700 1 -- 172.21.15.1:0/3740178310 reap_dead start 2024-03-18T19:09:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.357+0000 7f5ff24c3700 1 -- 172.21.15.1:0/3740178310 shutdown_connections 2024-03-18T19:09:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.357+0000 7f5ff24c3700 1 -- 172.21.15.1:0/3740178310 >> 172.21.15.1:0/3740178310 conn(0x7f5fec0b2bf0 msgr2=0x7f5fec0b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:09:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.357+0000 7f5ff24c3700 1 -- 172.21.15.1:0/3740178310 shutdown_connections 2024-03-18T19:09:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.357+0000 7f5ff24c3700 1 -- 172.21.15.1:0/3740178310 wait complete. 2024-03-18T19:09:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.358+0000 7f5ff24c3700 1 Processor -- start 2024-03-18T19:09:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.358+0000 7f5ff24c3700 1 -- start start 2024-03-18T19:09:59.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.359+0000 7f5ff24c3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5fd4003a30 0x7f5fec06bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:59.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.359+0000 7f5ff24c3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fec154a40 0x7f5fec154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:59.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.359+0000 7f5ff24c3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5fec0b5400 0x7f5fec0b58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:59.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.359+0000 7f5ff24c3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec1e5ad0 con 0x7f5fec154a40 2024-03-18T19:09:59.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.359+0000 7f5ff24c3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec0b6130 con 0x7f5fec0b5400 2024-03-18T19:09:59.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.359+0000 7f5ff24c3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec1e4d10 con 0x7f5fd4003a30 2024-03-18T19:09:59.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.359+0000 7f5feb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fec154a40 0x7f5fec154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:59.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.359+0000 7f5feb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fec154a40 0x7f5fec154ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35276/0 (socket says 172.21.15.1:35276) 2024-03-18T19:09:59.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.359+0000 7f5feb7fe700 1 -- 172.21.15.1:0/4234845450 learned_addr learned my addr 172.21.15.1:0/4234845450 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:09:59.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.359+0000 7f5febfff700 1 --2- 172.21.15.1:0/4234845450 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5fd4003a30 0x7f5fec06bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:59.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.359+0000 7f5ff0a60700 1 --2- 172.21.15.1:0/4234845450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5fec0b5400 0x7f5fec0b58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:59.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.359+0000 7f5feb7fe700 1 -- 172.21.15.1:0/4234845450 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5fd4003a30 msgr2=0x7f5fec06bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:59.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.359+0000 7f5feb7fe700 1 --2- 172.21.15.1:0/4234845450 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5fd4003a30 0x7f5fec06bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:59.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.359+0000 7f5feb7fe700 1 -- 172.21.15.1:0/4234845450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5fec0b5400 msgr2=0x7f5fec0b58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:09:59.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.359+0000 7f5feb7fe700 1 --2- 172.21.15.1:0/4234845450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5fec0b5400 0x7f5fec0b58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:09:59.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.359+0000 7f5feb7fe700 1 -- 172.21.15.1:0/4234845450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec1e4ff0 con 0x7f5fec154a40 2024-03-18T19:09:59.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.359+0000 7f5feb7fe700 1 --2- 172.21.15.1:0/4234845450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fec154a40 0x7f5fec154ec0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc00f7f0 tx=0x7f5fdc00fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:59.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.360+0000 7f5fe8ff9700 1 -- 172.21.15.1:0/4234845450 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc005320 con 0x7f5fec154a40 2024-03-18T19:09:59.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.360+0000 7f5fe8ff9700 1 -- 172.21.15.1:0/4234845450 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fdc005480 con 0x7f5fec154a40 2024-03-18T19:09:59.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.360+0000 7f5ff24c3700 1 -- 172.21.15.1:0/4234845450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fec1e52d0 con 0x7f5fec154a40 2024-03-18T19:09:59.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.360+0000 7f5fe8ff9700 1 -- 172.21.15.1:0/4234845450 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc006aa0 con 0x7f5fec154a40 2024-03-18T19:09:59.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.360+0000 7f5ff24c3700 1 -- 172.21.15.1:0/4234845450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fec155400 con 0x7f5fec154a40 2024-03-18T19:09:59.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.361+0000 7f5ff24c3700 1 -- 172.21.15.1:0/4234845450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fec1e5920 con 0x7f5fec154a40 2024-03-18T19:09:59.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.361+0000 7f5fe8ff9700 1 -- 172.21.15.1:0/4234845450 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5fdc006c00 con 0x7f5fec154a40 2024-03-18T19:09:59.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.361+0000 7f5fe8ff9700 1 --2- 172.21.15.1:0/4234845450 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5fd806adb0 0x7f5fd806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:09:59.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.361+0000 7f5fe8ff9700 1 -- 172.21.15.1:0/4234845450 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5936+0+0 (secure 0 0 0) 0x7f5fdc088b10 con 0x7f5fec154a40 2024-03-18T19:09:59.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.361+0000 7f5febfff700 1 --2- 172.21.15.1:0/4234845450 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5fd806adb0 0x7f5fd806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:09:59.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.362+0000 7f5febfff700 1 --2- 172.21.15.1:0/4234845450 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5fd806adb0 0x7f5fd806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5fe00090b0 tx=0x7f5fe0008030 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:09:59.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.366+0000 7f5fe8ff9700 1 -- 172.21.15.1:0/4234845450 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5fdc053b00 con 0x7f5fec154a40 2024-03-18T19:09:59.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:09:59.587+0000 7f5ff24c3700 1 -- 172.21.15.1:0/4234845450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5fec0676a0 con 0x7f5fec154a40 2024-03-18T19:10:00.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.535+0000 7f5fe8ff9700 1 -- 172.21.15.1:0/4234845450 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v157) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5fdc01a080 con 0x7f5fec154a40 2024-03-18T19:10:00.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.538+0000 7f5ff24c3700 1 -- 172.21.15.1:0/4234845450 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5fd806adb0 msgr2=0x7f5fd806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:00.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.538+0000 7f5ff24c3700 1 --2- 172.21.15.1:0/4234845450 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5fd806adb0 0x7f5fd806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5fe00090b0 tx=0x7f5fe0008030 comp rx=0 tx=0).stop 2024-03-18T19:10:00.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.538+0000 7f5ff24c3700 1 -- 172.21.15.1:0/4234845450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fec154a40 msgr2=0x7f5fec154ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:00.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.538+0000 7f5ff24c3700 1 --2- 172.21.15.1:0/4234845450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fec154a40 0x7f5fec154ec0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc00f7f0 tx=0x7f5fdc00fbb0 comp rx=0 tx=0).stop 2024-03-18T19:10:00.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.538+0000 7f5ff24c3700 1 -- 172.21.15.1:0/4234845450 shutdown_connections 2024-03-18T19:10:00.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.538+0000 7f5ff24c3700 1 --2- 172.21.15.1:0/4234845450 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5fd806adb0 0x7f5fd806d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:00.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.538+0000 7f5ff24c3700 1 --2- 172.21.15.1:0/4234845450 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5fd4003a30 0x7f5fec06bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:00.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.538+0000 7f5ff24c3700 1 --2- 172.21.15.1:0/4234845450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5fec0b5400 0x7f5fec0b58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:00.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.538+0000 7f5ff24c3700 1 --2- 172.21.15.1:0/4234845450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fec154a40 0x7f5fec154ec0 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:00.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.538+0000 7f5ff24c3700 1 -- 172.21.15.1:0/4234845450 >> 172.21.15.1:0/4234845450 conn(0x7f5fec0b2bf0 msgr2=0x7f5fec0c75b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:00.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.539+0000 7f5ff24c3700 1 -- 172.21.15.1:0/4234845450 shutdown_connections 2024-03-18T19:10:00.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.539+0000 7f5ff24c3700 1 -- 172.21.15.1:0/4234845450 wait complete. 2024-03-18T19:10:00.536 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-18T19:10:00.549 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-18T19:10:00.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.688+0000 7f4c7824e700 1 Processor -- start 2024-03-18T19:10:00.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.688+0000 7f4c7824e700 1 -- start start 2024-03-18T19:10:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.689+0000 7f4c7824e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c700ad140 0x7f4c7015a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.689+0000 7f4c7824e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c70072100 con 0x7f4c700ae070 2024-03-18T19:10:00.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.689+0000 7f4c7824e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c700b3560 con 0x7f4c7015a1b0 2024-03-18T19:10:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.689+0000 7f4c7824e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c70070610 con 0x7f4c700ad140 2024-03-18T19:10:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.689+0000 7f4c757e9700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4c700ae070 0x7f4c700b1d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58608/0 (socket says 172.21.15.1:58608) 2024-03-18T19:10:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.689+0000 7f4c757e9700 1 -- 172.21.15.1:0/4033075420 learned_addr learned my addr 172.21.15.1:0/4033075420 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:10:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.689+0000 7f4c767eb700 1 --2- 172.21.15.1:0/4033075420 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c700ad140 0x7f4c7015a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.689+0000 7f4c74fe8700 1 -- 172.21.15.1:0/4033075420 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 387856945 0 0) 0x7f4c70072100 con 0x7f4c700ae070 2024-03-18T19:10:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.689+0000 7f4c74fe8700 1 -- 172.21.15.1:0/4033075420 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c5c0035b0 con 0x7f4c700ae070 2024-03-18T19:10:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.689+0000 7f4c74fe8700 1 -- 172.21.15.1:0/4033075420 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1885907873 0 0) 0x7f4c5c0035b0 con 0x7f4c700ae070 2024-03-18T19:10:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.689+0000 7f4c74fe8700 1 -- 172.21.15.1:0/4033075420 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c700ad140 msgr2=0x7f4c7015a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.689+0000 7f4c74fe8700 1 --2- 172.21.15.1:0/4033075420 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c700ad140 0x7f4c7015a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.689+0000 7f4c74fe8700 1 -- 172.21.15.1:0/4033075420 >> v1:172.21.15.165:6789/0 conn(0x7f4c7015a1b0 legacy=0x7f4c700aca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c74fe8700 1 -- 172.21.15.1:0/4033075420 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c700708f0 con 0x7f4c700ae070 2024-03-18T19:10:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c74fe8700 1 -- 172.21.15.1:0/4033075420 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f4c60001670 con 0x7f4c700ae070 2024-03-18T19:10:00.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c74fe8700 1 -- 172.21.15.1:0/4033075420 >> v1:172.21.15.1:6789/0 conn(0x7f4c700ae070 legacy=0x7f4c700b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c74fe8700 1 --2- 172.21.15.1:0/4033075420 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c5c003a30 0x7f4c5c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c74fe8700 1 --2- 172.21.15.1:0/4033075420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c5c0043e0 0x7f4c5c0048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c74fe8700 1 --2- 172.21.15.1:0/4033075420 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c700ad140 0x7f4c5c005110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c74fe8700 1 -- 172.21.15.1:0/4033075420 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c70070610 con 0x7f4c5c0043e0 2024-03-18T19:10:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c74fe8700 1 -- 172.21.15.1:0/4033075420 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c00ba10 con 0x7f4c5c003a30 2024-03-18T19:10:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c74fe8700 1 -- 172.21.15.1:0/4033075420 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c00dde0 con 0x7f4c700ad140 2024-03-18T19:10:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c757e9700 1 --2- 172.21.15.1:0/4033075420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c5c0043e0 0x7f4c5c0048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c75fea700 1 --2- 172.21.15.1:0/4033075420 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c5c003a30 0x7f4c5c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c767eb700 1 --2- 172.21.15.1:0/4033075420 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c700ad140 0x7f4c5c005110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c757e9700 1 -- 172.21.15.1:0/4033075420 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c700ad140 msgr2=0x7f4c5c005110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c757e9700 1 --2- 172.21.15.1:0/4033075420 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c700ad140 0x7f4c5c005110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c757e9700 1 -- 172.21.15.1:0/4033075420 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c5c003a30 msgr2=0x7f4c5c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c757e9700 1 --2- 172.21.15.1:0/4033075420 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c5c003a30 0x7f4c5c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:00.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c757e9700 1 -- 172.21.15.1:0/4033075420 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c0035b0 con 0x7f4c5c0043e0 2024-03-18T19:10:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c767eb700 1 --2- 172.21.15.1:0/4033075420 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c700ad140 0x7f4c5c005110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:10:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c757e9700 1 --2- 172.21.15.1:0/4033075420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c5c0043e0 0x7f4c5c0048b0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f4c6000c240 tx=0x7f4c5c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7116110e7b1d15ef server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c74fe8700 1 -- 172.21.15.1:0/4033075420 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6000cde0 con 0x7f4c5c0043e0 2024-03-18T19:10:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c74fe8700 1 -- 172.21.15.1:0/4033075420 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c60007ba0 con 0x7f4c5c0043e0 2024-03-18T19:10:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.690+0000 7f4c74fe8700 1 -- 172.21.15.1:0/4033075420 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60005ba0 con 0x7f4c5c0043e0 2024-03-18T19:10:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.691+0000 7f4c7824e700 1 -- 172.21.15.1:0/4033075420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c5c0043e0 msgr2=0x7f4c5c0048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.691+0000 7f4c7824e700 1 --2- 172.21.15.1:0/4033075420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c5c0043e0 0x7f4c5c0048b0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f4c6000c240 tx=0x7f4c5c005880 comp rx=0 tx=0).stop 2024-03-18T19:10:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.691+0000 7f4c767eb700 1 -- 172.21.15.1:0/4033075420 reap_dead start 2024-03-18T19:10:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.691+0000 7f4c7824e700 1 -- 172.21.15.1:0/4033075420 shutdown_connections 2024-03-18T19:10:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.691+0000 7f4c7824e700 1 -- 172.21.15.1:0/4033075420 >> 172.21.15.1:0/4033075420 conn(0x7f4c700cee10 msgr2=0x7f4c700cf220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.691+0000 7f4c7824e700 1 -- 172.21.15.1:0/4033075420 shutdown_connections 2024-03-18T19:10:00.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.691+0000 7f4c7824e700 1 -- 172.21.15.1:0/4033075420 wait complete. 2024-03-18T19:10:00.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.693+0000 7f4c7824e700 1 Processor -- start 2024-03-18T19:10:00.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.693+0000 7f4c7824e700 1 -- start start 2024-03-18T19:10:00.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c7824e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c5c0043e0 0x7f4c70156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c7824e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c70145be0 0x7f4c70144290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c7824e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c70071d90 0x7f4c701f6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c7824e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c70099e00 con 0x7f4c70071d90 2024-03-18T19:10:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c7824e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7009a0e0 con 0x7f4c5c0043e0 2024-03-18T19:10:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c7824e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7009a3c0 con 0x7f4c70145be0 2024-03-18T19:10:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c767eb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c70071d90 0x7f4c701f6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c767eb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c70071d90 0x7f4c701f6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35286/0 (socket says 172.21.15.1:35286) 2024-03-18T19:10:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c767eb700 1 -- 172.21.15.1:0/2407600571 learned_addr learned my addr 172.21.15.1:0/2407600571 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c757e9700 1 --2- 172.21.15.1:0/2407600571 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c70145be0 0x7f4c70144290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c75fea700 1 --2- 172.21.15.1:0/2407600571 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c5c0043e0 0x7f4c70156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c767eb700 1 -- 172.21.15.1:0/2407600571 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c70145be0 msgr2=0x7f4c70144290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c767eb700 1 --2- 172.21.15.1:0/2407600571 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c70145be0 0x7f4c70144290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:00.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c767eb700 1 -- 172.21.15.1:0/2407600571 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c5c0043e0 msgr2=0x7f4c70156330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c767eb700 1 --2- 172.21.15.1:0/2407600571 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c5c0043e0 0x7f4c70156330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c767eb700 1 -- 172.21.15.1:0/2407600571 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c701f6810 con 0x7f4c70071d90 2024-03-18T19:10:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c75fea700 1 --2- 172.21.15.1:0/2407600571 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c5c0043e0 0x7f4c70156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:10:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c757e9700 1 --2- 172.21.15.1:0/2407600571 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c70145be0 0x7f4c70144290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:10:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c767eb700 1 --2- 172.21.15.1:0/2407600571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c70071d90 0x7f4c701f6d30 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f4c7006c910 tx=0x7f4c680104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.694+0000 7f4c667fc700 1 -- 172.21.15.1:0/2407600571 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6800e050 con 0x7f4c70071d90 2024-03-18T19:10:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.695+0000 7f4c667fc700 1 -- 172.21.15.1:0/2407600571 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6800f040 con 0x7f4c70071d90 2024-03-18T19:10:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.695+0000 7f4c667fc700 1 -- 172.21.15.1:0/2407600571 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c68010e90 con 0x7f4c70071d90 2024-03-18T19:10:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.695+0000 7f4c7824e700 1 -- 172.21.15.1:0/2407600571 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c7009a6a0 con 0x7f4c70071d90 2024-03-18T19:10:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.695+0000 7f4c7824e700 1 -- 172.21.15.1:0/2407600571 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c70156870 con 0x7f4c70071d90 2024-03-18T19:10:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.696+0000 7f4c7824e700 1 -- 172.21.15.1:0/2407600571 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c70073350 con 0x7f4c70071d90 2024-03-18T19:10:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.696+0000 7f4c667fc700 1 -- 172.21.15.1:0/2407600571 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4c68015440 con 0x7f4c70071d90 2024-03-18T19:10:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.696+0000 7f4c667fc700 1 --2- 172.21.15.1:0/2407600571 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4c5406adb0 0x7f4c5406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:00.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.697+0000 7f4c667fc700 1 -- 172.21.15.1:0/2407600571 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f4c68089620 con 0x7f4c70071d90 2024-03-18T19:10:00.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.697+0000 7f4c75fea700 1 --2- 172.21.15.1:0/2407600571 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4c5406adb0 0x7f4c5406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:00.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.697+0000 7f4c75fea700 1 --2- 172.21.15.1:0/2407600571 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4c5406adb0 0x7f4c5406d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c0011a0 tx=0x7f4c6c009040 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:00.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.701+0000 7f4c667fc700 1 -- 172.21.15.1:0/2407600571 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4c68054860 con 0x7f4c70071d90 2024-03-18T19:10:00.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.986+0000 7f4c7824e700 1 -- 172.21.15.1:0/2407600571 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4c700cfd20 con 0x7f4c70071d90 2024-03-18T19:10:00.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.987+0000 7f4c667fc700 1 -- 172.21.15.1:0/2407600571 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f4c68057e80 con 0x7f4c70071d90 2024-03-18T19:10:00.983 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:10:00.983 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:10:00.983 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-18T19:10:00.983 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:00.983 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-18T19:10:00.983 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlR93/CxAAbpe+5n3wHRur9xi8QzdCnQ==", 2024-03-18T19:10:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:10:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:10:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:10:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:10:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:10:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-18T19:10:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJ3OmGBAA37u227nxrMsh/JHt71HBxQ==", 2024-03-18T19:10:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:10:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:10:00.984 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:10:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:10:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:10:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-18T19:10:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJZFoExAAmI16oxwNKbf8dKetPfyk+g==", 2024-03-18T19:10:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:10:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:10:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:10:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:10:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:10:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-18T19:10:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlQrcvIBAA1LMv+EgC7S6wspX6QF49Pg==", 2024-03-18T19:10:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:10:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:10:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:10:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:10:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:10:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-18T19:10:00.986 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCUjPhlYOVHChAARFQrksZmMIdTlntd8TgvrQ==", 2024-03-18T19:10:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:10:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:10:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:10:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:10:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-18T19:10:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCXjPhl2zSDCBAA6Al1gkWVB79ZI9djGpVChA==", 2024-03-18T19:10:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:10:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:10:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:10:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:10:00.987 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-18T19:10:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCajPhlSoG/BhAAR93USDe6DCQW+hREOsnTlw==", 2024-03-18T19:10:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:10:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:10:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:10:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:10:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-18T19:10:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCdjPhlhJg5BRAAWuQhcnxyjR5UOiM13yKdAQ==", 2024-03-18T19:10:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:10:00.988 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:10:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:10:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:10:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-18T19:10:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCjjPhlCFnpCRAAKIyOQyVdjXZUTnAS6Nw/ng==", 2024-03-18T19:10:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:10:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:10:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:10:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:10:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:00.989 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-18T19:10:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCmjPhlAeFkCBAAJcEFpRk/03lQJxj+l2hEPA==", 2024-03-18T19:10:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:10:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:10:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:10:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:10:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-18T19:10:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCpjPhl/xuuBhAA0LgKfXEvN2O11ZlRQJVOkw==", 2024-03-18T19:10:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:10:00.990 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:10:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:10:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:10:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-18T19:10:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCsjPhldiR4BRAA571oh6U8PzOv/4LzjYLF/A==", 2024-03-18T19:10:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:10:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T19:10:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T19:10:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:10:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:00.991 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-18T19:10:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlu+2xJBAANnXGGjfeqP6S2FTecsfOsg==", 2024-03-18T19:10:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:10:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:10:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-18T19:10:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-18T19:10:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-18T19:10:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlhJ9+KxAAEnrQrCrgJR6xI+pucwzAXA==", 2024-03-18T19:10:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:10:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:10:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:10:00.992 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:10:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:10:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.2", 2024-03-18T19:10:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlxBUpKBAAIGTwmLWyj+P2Yvl55QVcWA==", 2024-03-18T19:10:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:10:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:10:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:10:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:10:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:10:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.993 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.3", 2024-03-18T19:10:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlmao3MRAAHeiQ6eyBh9QjjKPCtxkC8g==", 2024-03-18T19:10:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:10:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T19:10:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T19:10:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T19:10:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T19:10:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-18T19:10:00.994 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCRjPhlemOuKRAAVUggmtB7+EiVgyfJZ8TCCA==", 2024-03-18T19:10:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:10:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:10:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T19:10:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T19:10:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:10:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-18T19:10:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhleIuOCBAA9CH+AHewQbn+gpVr+6lJyw==", 2024-03-18T19:10:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:10:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:10:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T19:10:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:10:00.995 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-18T19:10:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlL0pTBBAAdFiPcdwwz514cuLAvioxYA==", 2024-03-18T19:10:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T19:10:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T19:10:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T19:10:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T19:10:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:00.996 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-18T19:10:00.996 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:10:00.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.989+0000 7f4c7824e700 1 -- 172.21.15.1:0/2407600571 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4c5406adb0 msgr2=0x7f4c5406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:00.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.989+0000 7f4c7824e700 1 --2- 172.21.15.1:0/2407600571 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4c5406adb0 0x7f4c5406d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c0011a0 tx=0x7f4c6c009040 comp rx=0 tx=0).stop 2024-03-18T19:10:00.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.989+0000 7f4c7824e700 1 -- 172.21.15.1:0/2407600571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c70071d90 msgr2=0x7f4c701f6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:00.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.989+0000 7f4c7824e700 1 --2- 172.21.15.1:0/2407600571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c70071d90 0x7f4c701f6d30 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f4c7006c910 tx=0x7f4c680104a0 comp rx=0 tx=0).stop 2024-03-18T19:10:00.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.989+0000 7f4c7824e700 1 -- 172.21.15.1:0/2407600571 shutdown_connections 2024-03-18T19:10:00.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.989+0000 7f4c7824e700 1 --2- 172.21.15.1:0/2407600571 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4c5406adb0 0x7f4c5406d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:00.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.989+0000 7f4c7824e700 1 --2- 172.21.15.1:0/2407600571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c70071d90 0x7f4c701f6d30 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:00.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.989+0000 7f4c7824e700 1 --2- 172.21.15.1:0/2407600571 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c5c0043e0 0x7f4c70156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:00.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.989+0000 7f4c7824e700 1 --2- 172.21.15.1:0/2407600571 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c70145be0 0x7f4c70144290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:00.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.989+0000 7f4c7824e700 1 -- 172.21.15.1:0/2407600571 >> 172.21.15.1:0/2407600571 conn(0x7f4c700cee10 msgr2=0x7f4c70158ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:00.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.989+0000 7f4c7824e700 1 -- 172.21.15.1:0/2407600571 shutdown_connections 2024-03-18T19:10:00.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:00.989+0000 7f4c7824e700 1 -- 172.21.15.1:0/2407600571 wait complete. 2024-03-18T19:10:00.999 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:10:01.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.129+0000 7fe89e121700 1 Processor -- start 2024-03-18T19:10:01.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.129+0000 7fe89e121700 1 -- start start 2024-03-18T19:10:01.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.130+0000 7fe89e121700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe89815b4d0 0x7fe89809a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:01.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.130+0000 7fe89e121700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe8980cf750 0x7fe898099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:01.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.130+0000 7fe89e121700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89806f070 con 0x7fe898099f80 2024-03-18T19:10:01.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.130+0000 7fe89e121700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89809a3f0 con 0x7fe89815b4d0 2024-03-18T19:10:01.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.130+0000 7fe89e121700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8980d0260 con 0x7fe8980cf750 2024-03-18T19:10:01.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.130+0000 7fe8977fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe89815b4d0 0x7fe89809a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:01.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.130+0000 7fe8977fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe89815b4d0 0x7fe89809a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35296/0 (socket says 172.21.15.1:35296) 2024-03-18T19:10:01.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.130+0000 7fe8977fe700 1 -- 172.21.15.1:0/2894594693 learned_addr learned my addr 172.21.15.1:0/2894594693 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:01.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.130+0000 7fe896ffd700 1 --2- 172.21.15.1:0/2894594693 >> v2:172.21.15.165:3300/0 conn(0x7fe8980cf750 0x7fe898099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:01.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.130+0000 7fe8977fe700 1 -- 172.21.15.1:0/2894594693 >> v2:172.21.15.165:3300/0 conn(0x7fe8980cf750 msgr2=0x7fe898099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.130+0000 7fe8977fe700 1 --2- 172.21.15.1:0/2894594693 >> v2:172.21.15.165:3300/0 conn(0x7fe8980cf750 0x7fe898099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.130+0000 7fe8977fe700 1 -- 172.21.15.1:0/2894594693 >> v1:172.21.15.165:6789/0 conn(0x7fe898099f80 legacy=0x7fe8980cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.130+0000 7fe8977fe700 1 -- 172.21.15.1:0/2894594693 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8980cff80 con 0x7fe89815b4d0 2024-03-18T19:10:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.131+0000 7fe8977fe700 1 --2- 172.21.15.1:0/2894594693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe89815b4d0 0x7fe89809a650 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fe898071d40 tx=0x7fe888009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c4c3f9cd29971b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.131+0000 7fe8967fc700 1 -- 172.21.15.1:0/2894594693 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe888011830 con 0x7fe89815b4d0 2024-03-18T19:10:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.131+0000 7fe8967fc700 1 -- 172.21.15.1:0/2894594693 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe888011990 con 0x7fe89815b4d0 2024-03-18T19:10:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.131+0000 7fe8967fc700 1 -- 172.21.15.1:0/2894594693 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe888011d20 con 0x7fe89815b4d0 2024-03-18T19:10:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.131+0000 7fe89e121700 1 -- 172.21.15.1:0/2894594693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe89815b4d0 msgr2=0x7fe89809a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.131+0000 7fe89e121700 1 --2- 172.21.15.1:0/2894594693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe89815b4d0 0x7fe89809a650 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fe898071d40 tx=0x7fe888009bd0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.131+0000 7fe89e121700 1 -- 172.21.15.1:0/2894594693 shutdown_connections 2024-03-18T19:10:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.131+0000 7fe89e121700 1 --2- 172.21.15.1:0/2894594693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe89815b4d0 0x7fe89809a650 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.131+0000 7fe89e121700 1 --2- 172.21.15.1:0/2894594693 >> v2:172.21.15.165:3300/0 conn(0x7fe8980cf750 0x7fe898099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.131+0000 7fe89e121700 1 -- 172.21.15.1:0/2894594693 >> 172.21.15.1:0/2894594693 conn(0x7fe8980b2cb0 msgr2=0x7fe8980b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:01.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.131+0000 7fe89e121700 1 -- 172.21.15.1:0/2894594693 shutdown_connections 2024-03-18T19:10:01.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.131+0000 7fe89e121700 1 -- 172.21.15.1:0/2894594693 wait complete. 2024-03-18T19:10:01.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.133+0000 7fe89e121700 1 Processor -- start 2024-03-18T19:10:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.133+0000 7fe89e121700 1 -- start start 2024-03-18T19:10:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.134+0000 7fe89e121700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe898099f80 0x7fe898157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.134+0000 7fe89e121700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8980cf750 0x7fe8980c56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.134+0000 7fe89e121700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe89815b4d0 0x7fe89806f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.134+0000 7fe89e121700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8980b5d00 con 0x7fe8980cf750 2024-03-18T19:10:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.134+0000 7fe89e121700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe898158100 con 0x7fe898099f80 2024-03-18T19:10:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.134+0000 7fe89e121700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8981583e0 con 0x7fe89815b4d0 2024-03-18T19:10:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.134+0000 7fe896ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8980cf750 0x7fe8980c56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.134+0000 7fe896ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8980cf750 0x7fe8980c56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35300/0 (socket says 172.21.15.1:35300) 2024-03-18T19:10:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.134+0000 7fe896ffd700 1 -- 172.21.15.1:0/3700123020 learned_addr learned my addr 172.21.15.1:0/3700123020 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:01.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.134+0000 7fe8977fe700 1 --2- 172.21.15.1:0/3700123020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe898099f80 0x7fe898157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.134+0000 7fe897fff700 1 --2- 172.21.15.1:0/3700123020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe89815b4d0 0x7fe89806f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.135+0000 7fe896ffd700 1 -- 172.21.15.1:0/3700123020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe89815b4d0 msgr2=0x7fe89806f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.135+0000 7fe896ffd700 1 --2- 172.21.15.1:0/3700123020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe89815b4d0 0x7fe89806f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.135+0000 7fe896ffd700 1 -- 172.21.15.1:0/3700123020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe898099f80 msgr2=0x7fe898157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.135+0000 7fe896ffd700 1 --2- 172.21.15.1:0/3700123020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe898099f80 0x7fe898157b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.135+0000 7fe896ffd700 1 -- 172.21.15.1:0/3700123020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe888009550 con 0x7fe8980cf750 2024-03-18T19:10:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.135+0000 7fe896ffd700 1 --2- 172.21.15.1:0/3700123020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8980cf750 0x7fe8980c56b0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fe88c00e9b0 tx=0x7fe88c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.135+0000 7fe87ffff700 1 -- 172.21.15.1:0/3700123020 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c009710 con 0x7fe8980cf750 2024-03-18T19:10:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.135+0000 7fe87ffff700 1 -- 172.21.15.1:0/3700123020 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88c00ee50 con 0x7fe8980cf750 2024-03-18T19:10:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.135+0000 7fe89e121700 1 -- 172.21.15.1:0/3700123020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8981586c0 con 0x7fe8980cf750 2024-03-18T19:10:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.135+0000 7fe87ffff700 1 -- 172.21.15.1:0/3700123020 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c00ca50 con 0x7fe8980cf750 2024-03-18T19:10:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.135+0000 7fe89e121700 1 -- 172.21.15.1:0/3700123020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe898158c80 con 0x7fe8980cf750 2024-03-18T19:10:01.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.136+0000 7fe89e121700 1 -- 172.21.15.1:0/3700123020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe898158e40 con 0x7fe8980cf750 2024-03-18T19:10:01.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.136+0000 7fe87ffff700 1 -- 172.21.15.1:0/3700123020 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe88c00cbb0 con 0x7fe8980cf750 2024-03-18T19:10:01.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.137+0000 7fe87ffff700 1 --2- 172.21.15.1:0/3700123020 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe88406adb0 0x7fe88406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:01.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.137+0000 7fe87ffff700 1 -- 172.21.15.1:0/3700123020 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fe88c089250 con 0x7fe8980cf750 2024-03-18T19:10:01.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.137+0000 7fe8977fe700 1 --2- 172.21.15.1:0/3700123020 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe88406adb0 0x7fe88406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:01.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.137+0000 7fe8977fe700 1 --2- 172.21.15.1:0/3700123020 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe88406adb0 0x7fe88406d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe888014040 tx=0x7fe888008030 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:01.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.141+0000 7fe87ffff700 1 -- 172.21.15.1:0/3700123020 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe88c054510 con 0x7fe8980cf750 2024-03-18T19:10:01.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.346+0000 7fe89e121700 1 -- 172.21.15.1:0/3700123020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe898068240 con 0x7fe8980cf750 2024-03-18T19:10:01.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.347+0000 7fe87ffff700 1 -- 172.21.15.1:0/3700123020 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) v1 ==== 74+0+12600 (secure 0 0 0) 0x7fe88c00ce60 con 0x7fe8980cf750 2024-03-18T19:10:01.343 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:10:01.343 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":157,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:10:00.532449+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:10:01.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.349+0000 7fe89e121700 1 -- 172.21.15.1:0/3700123020 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe88406adb0 msgr2=0x7fe88406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:01.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.349+0000 7fe89e121700 1 --2- 172.21.15.1:0/3700123020 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe88406adb0 0x7fe88406d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe888014040 tx=0x7fe888008030 comp rx=0 tx=0).stop 2024-03-18T19:10:01.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.349+0000 7fe89e121700 1 -- 172.21.15.1:0/3700123020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8980cf750 msgr2=0x7fe8980c56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:01.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.349+0000 7fe89e121700 1 --2- 172.21.15.1:0/3700123020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8980cf750 0x7fe8980c56b0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fe88c00e9b0 tx=0x7fe88c00ecc0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.349+0000 7fe89e121700 1 -- 172.21.15.1:0/3700123020 shutdown_connections 2024-03-18T19:10:01.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.349+0000 7fe89e121700 1 --2- 172.21.15.1:0/3700123020 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe88406adb0 0x7fe88406d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.349+0000 7fe89e121700 1 --2- 172.21.15.1:0/3700123020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe89815b4d0 0x7fe89806f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.349+0000 7fe89e121700 1 --2- 172.21.15.1:0/3700123020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe898099f80 0x7fe898157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.349+0000 7fe89e121700 1 --2- 172.21.15.1:0/3700123020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8980cf750 0x7fe8980c56b0 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.349+0000 7fe89e121700 1 -- 172.21.15.1:0/3700123020 >> 172.21.15.1:0/3700123020 conn(0x7fe8980b2cb0 msgr2=0x7fe89815a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:01.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.349+0000 7fe89e121700 1 -- 172.21.15.1:0/3700123020 shutdown_connections 2024-03-18T19:10:01.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.349+0000 7fe89e121700 1 -- 172.21.15.1:0/3700123020 wait complete. 2024-03-18T19:10:01.357 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T19:10:01.357 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T19:10:01.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.489+0000 7f5b5fa4c700 1 Processor -- start 2024-03-18T19:10:01.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.489+0000 7f5b5fa4c700 1 -- start start 2024-03-18T19:10:01.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.490+0000 7f5b5fa4c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5b5815d3b0 0x7f5b5806cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:01.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.490+0000 7f5b5fa4c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b5806d430 0x7f5b58070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:01.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.490+0000 7f5b5fa4c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5b580b2650 0x7f5b58071380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:01.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.490+0000 7f5b5fa4c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b580b3800 con 0x7f5b580b2650 2024-03-18T19:10:01.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.490+0000 7f5b5fa4c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b580b2b80 con 0x7f5b5815d3b0 2024-03-18T19:10:01.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.490+0000 7f5b5fa4c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b580b2e60 con 0x7f5b5806d430 2024-03-18T19:10:01.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.490+0000 7f5b5dfe9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5b580b2650 0x7f5b58071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:01.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.490+0000 7f5b5dfe9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5b580b2650 0x7f5b58071380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35312/0 (socket says 172.21.15.1:35312) 2024-03-18T19:10:01.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.490+0000 7f5b5dfe9700 1 -- 172.21.15.1:0/684529058 learned_addr learned my addr 172.21.15.1:0/684529058 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:01.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.490+0000 7f5b5d7e8700 1 --2- 172.21.15.1:0/684529058 >> v2:172.21.15.165:3300/0 conn(0x7f5b5815d3b0 0x7f5b5806cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:01.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.490+0000 7f5b5cfe7700 1 --2- 172.21.15.1:0/684529058 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b5806d430 0x7f5b58070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:01.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.490+0000 7f5b5dfe9700 1 -- 172.21.15.1:0/684529058 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b5806d430 msgr2=0x7f5b58070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:01.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.490+0000 7f5b5dfe9700 1 --2- 172.21.15.1:0/684529058 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b5806d430 0x7f5b58070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.491+0000 7f5b5dfe9700 1 -- 172.21.15.1:0/684529058 >> v2:172.21.15.165:3300/0 conn(0x7f5b5815d3b0 msgr2=0x7f5b5806cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:01.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.491+0000 7f5b5dfe9700 1 --2- 172.21.15.1:0/684529058 >> v2:172.21.15.165:3300/0 conn(0x7f5b5815d3b0 0x7f5b5806cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.491+0000 7f5b5dfe9700 1 -- 172.21.15.1:0/684529058 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b580b3140 con 0x7f5b580b2650 2024-03-18T19:10:01.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.491+0000 7f5b5d7e8700 1 --2- 172.21.15.1:0/684529058 >> v2:172.21.15.165:3300/0 conn(0x7f5b5815d3b0 0x7f5b5806cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:10:01.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.491+0000 7f5b5dfe9700 1 --2- 172.21.15.1:0/684529058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b580b2650 0x7f5b58071380 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f5b580d0810 tx=0x7f5b5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a31da38d5160510 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:01.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.491+0000 7f5b4ffff700 1 -- 172.21.15.1:0/684529058 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5000e070 con 0x7f5b580b2650 2024-03-18T19:10:01.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.491+0000 7f5b4ffff700 1 -- 172.21.15.1:0/684529058 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5000b9c0 con 0x7f5b580b2650 2024-03-18T19:10:01.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.491+0000 7f5b4ffff700 1 -- 172.21.15.1:0/684529058 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5000bd70 con 0x7f5b580b2650 2024-03-18T19:10:01.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.491+0000 7f5b5fa4c700 1 -- 172.21.15.1:0/684529058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b580b2650 msgr2=0x7f5b58071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:01.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.491+0000 7f5b5fa4c700 1 --2- 172.21.15.1:0/684529058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b580b2650 0x7f5b58071380 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f5b580d0810 tx=0x7f5b5000b330 comp rx=0 tx=0).stop 2024-03-18T19:10:01.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.491+0000 7f5b5fa4c700 1 -- 172.21.15.1:0/684529058 shutdown_connections 2024-03-18T19:10:01.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.491+0000 7f5b5fa4c700 1 --2- 172.21.15.1:0/684529058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b580b2650 0x7f5b58071380 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.491+0000 7f5b5fa4c700 1 --2- 172.21.15.1:0/684529058 >> v2:172.21.15.165:3300/0 conn(0x7f5b5815d3b0 0x7f5b5806cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.491+0000 7f5b5fa4c700 1 --2- 172.21.15.1:0/684529058 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b5806d430 0x7f5b58070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.491+0000 7f5b5fa4c700 1 -- 172.21.15.1:0/684529058 >> 172.21.15.1:0/684529058 conn(0x7f5b580b1350 msgr2=0x7f5b580b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:01.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.492+0000 7f5b5fa4c700 1 -- 172.21.15.1:0/684529058 shutdown_connections 2024-03-18T19:10:01.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.492+0000 7f5b5fa4c700 1 -- 172.21.15.1:0/684529058 wait complete. 2024-03-18T19:10:01.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.493+0000 7f5b5fa4c700 1 Processor -- start 2024-03-18T19:10:01.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.493+0000 7f5b5fa4c700 1 -- start start 2024-03-18T19:10:01.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.494+0000 7f5b5fa4c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b5806d430 0x7f5b581f6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:01.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.494+0000 7f5b5fa4c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b580b2650 0x7f5b58159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:01.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.494+0000 7f5b5fa4c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b5815d3b0 0x7f5b580afa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:01.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.494+0000 7f5b5fa4c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b581f6930 con 0x7f5b5815d3b0 2024-03-18T19:10:01.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.494+0000 7f5b5fa4c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b581f6b80 con 0x7f5b580b2650 2024-03-18T19:10:01.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.494+0000 7f5b5fa4c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b581f6e60 con 0x7f5b5806d430 2024-03-18T19:10:01.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.494+0000 7f5b5dfe9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b5815d3b0 0x7f5b580afa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:01.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.494+0000 7f5b5dfe9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b5815d3b0 0x7f5b580afa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35322/0 (socket says 172.21.15.1:35322) 2024-03-18T19:10:01.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.494+0000 7f5b5dfe9700 1 -- 172.21.15.1:0/3351605322 learned_addr learned my addr 172.21.15.1:0/3351605322 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:01.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.494+0000 7f5b5d7e8700 1 --2- 172.21.15.1:0/3351605322 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b5806d430 0x7f5b581f6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:01.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.494+0000 7f5b5cfe7700 1 --2- 172.21.15.1:0/3351605322 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b580b2650 0x7f5b58159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:01.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.495+0000 7f5b5dfe9700 1 -- 172.21.15.1:0/3351605322 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b5806d430 msgr2=0x7f5b581f6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:01.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.495+0000 7f5b5dfe9700 1 --2- 172.21.15.1:0/3351605322 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b5806d430 0x7f5b581f6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.495+0000 7f5b5dfe9700 1 -- 172.21.15.1:0/3351605322 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b580b2650 msgr2=0x7f5b58159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:01.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.495+0000 7f5b5dfe9700 1 --2- 172.21.15.1:0/3351605322 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b580b2650 0x7f5b58159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.495+0000 7f5b5dfe9700 1 -- 172.21.15.1:0/3351605322 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b50009b30 con 0x7f5b5815d3b0 2024-03-18T19:10:01.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.495+0000 7f5b5cfe7700 1 --2- 172.21.15.1:0/3351605322 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b580b2650 0x7f5b58159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:10:01.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.495+0000 7f5b5dfe9700 1 --2- 172.21.15.1:0/3351605322 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b5815d3b0 0x7f5b580afa50 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f5b5806c740 tx=0x7f5b50004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:01.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.495+0000 7f5b4dffb700 1 -- 172.21.15.1:0/3351605322 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b500092e0 con 0x7f5b5815d3b0 2024-03-18T19:10:01.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.495+0000 7f5b4dffb700 1 -- 172.21.15.1:0/3351605322 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b50004250 con 0x7f5b5815d3b0 2024-03-18T19:10:01.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.495+0000 7f5b5fa4c700 1 -- 172.21.15.1:0/3351605322 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b581f7140 con 0x7f5b5815d3b0 2024-03-18T19:10:01.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.495+0000 7f5b4dffb700 1 -- 172.21.15.1:0/3351605322 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b50004600 con 0x7f5b5815d3b0 2024-03-18T19:10:01.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.495+0000 7f5b5fa4c700 1 -- 172.21.15.1:0/3351605322 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b581f65c0 con 0x7f5b5815d3b0 2024-03-18T19:10:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.496+0000 7f5b5fa4c700 1 -- 172.21.15.1:0/3351605322 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b580676a0 con 0x7f5b5815d3b0 2024-03-18T19:10:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.496+0000 7f5b4dffb700 1 -- 172.21.15.1:0/3351605322 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5b50004760 con 0x7f5b5815d3b0 2024-03-18T19:10:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.497+0000 7f5b4dffb700 1 --2- 172.21.15.1:0/3351605322 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b3c06adb0 0x7f5b3c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.497+0000 7f5b4dffb700 1 -- 172.21.15.1:0/3351605322 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f5b5008a8a0 con 0x7f5b5815d3b0 2024-03-18T19:10:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.497+0000 7f5b5d7e8700 1 --2- 172.21.15.1:0/3351605322 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b3c06adb0 0x7f5b3c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.497+0000 7f5b5d7e8700 1 --2- 172.21.15.1:0/3351605322 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b3c06adb0 0x7f5b3c06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5b581550f0 tx=0x7f5b48008040 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:01.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.501+0000 7f5b4dffb700 1 -- 172.21.15.1:0/3351605322 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5b50055ae0 con 0x7f5b5815d3b0 2024-03-18T19:10:01.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.701+0000 7f5b5fa4c700 1 -- 172.21.15.1:0/3351605322 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5b5806f5b0 con 0x7f5b5815d3b0 2024-03-18T19:10:01.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.701+0000 7f5b4dffb700 1 -- 172.21.15.1:0/3351605322 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v157) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5b5002a0a0 con 0x7f5b5815d3b0 2024-03-18T19:10:01.697 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T19:10:01.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.704+0000 7f5b5fa4c700 1 -- 172.21.15.1:0/3351605322 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b3c06adb0 msgr2=0x7f5b3c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:01.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.704+0000 7f5b5fa4c700 1 --2- 172.21.15.1:0/3351605322 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b3c06adb0 0x7f5b3c06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5b581550f0 tx=0x7f5b48008040 comp rx=0 tx=0).stop 2024-03-18T19:10:01.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.704+0000 7f5b5fa4c700 1 -- 172.21.15.1:0/3351605322 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b5815d3b0 msgr2=0x7f5b580afa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:01.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.704+0000 7f5b5fa4c700 1 --2- 172.21.15.1:0/3351605322 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b5815d3b0 0x7f5b580afa50 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f5b5806c740 tx=0x7f5b50004ad0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.704+0000 7f5b5fa4c700 1 -- 172.21.15.1:0/3351605322 shutdown_connections 2024-03-18T19:10:01.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.704+0000 7f5b5fa4c700 1 --2- 172.21.15.1:0/3351605322 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b3c06adb0 0x7f5b3c06d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.704+0000 7f5b5fa4c700 1 --2- 172.21.15.1:0/3351605322 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b5806d430 0x7f5b581f6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.704+0000 7f5b5fa4c700 1 --2- 172.21.15.1:0/3351605322 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b5815d3b0 0x7f5b580afa50 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.704+0000 7f5b5fa4c700 1 --2- 172.21.15.1:0/3351605322 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b580b2650 0x7f5b58159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.704+0000 7f5b5fa4c700 1 -- 172.21.15.1:0/3351605322 >> 172.21.15.1:0/3351605322 conn(0x7f5b580b1350 msgr2=0x7f5b58070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:01.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.704+0000 7f5b5fa4c700 1 -- 172.21.15.1:0/3351605322 shutdown_connections 2024-03-18T19:10:01.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.704+0000 7f5b5fa4c700 1 -- 172.21.15.1:0/3351605322 wait complete. 2024-03-18T19:10:01.712 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T19:10:01.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.841+0000 7f219cf02700 1 Processor -- start 2024-03-18T19:10:01.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.841+0000 7f219cf02700 1 -- start start 2024-03-18T19:10:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.842+0000 7f219cf02700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f219806eb30 0x7f21980720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.842+0000 7f219cf02700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f219806efb0 0x7f2198072630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.842+0000 7f219cf02700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2198144dc0 con 0x7f219806a750 2024-03-18T19:10:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.842+0000 7f219cf02700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2198071650 con 0x7f219806efb0 2024-03-18T19:10:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.842+0000 7f219cf02700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21980cf970 con 0x7f219806eb30 2024-03-18T19:10:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.842+0000 7f2196d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f219806efb0 0x7f2198072630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.842+0000 7f219659c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f219806a750 0x7f219806ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58612/0 (socket says 172.21.15.1:58612) 2024-03-18T19:10:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.842+0000 7f219659c700 1 -- 172.21.15.1:0/2613166950 learned_addr learned my addr 172.21.15.1:0/2613166950 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:10:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.843+0000 7f2195d9b700 1 --2- 172.21.15.1:0/2613166950 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f219806eb30 0x7f21980720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.843+0000 7f2196d9d700 1 -- 172.21.15.1:0/2613166950 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f219806eb30 msgr2=0x7f21980720f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.843+0000 7f2196d9d700 1 --2- 172.21.15.1:0/2613166950 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f219806eb30 0x7f21980720f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.843+0000 7f2196d9d700 1 -- 172.21.15.1:0/2613166950 >> v1:172.21.15.1:6789/0 conn(0x7f219806a750 legacy=0x7f219806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.843+0000 7f2196d9d700 1 -- 172.21.15.1:0/2613166950 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21980cfc50 con 0x7f219806efb0 2024-03-18T19:10:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.843+0000 7f219559a700 1 -- 172.21.15.1:0/2613166950 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036165985 0 0) 0x7f2198144dc0 con 0x7f219806a750 2024-03-18T19:10:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.843+0000 7f2196d9d700 1 --2- 172.21.15.1:0/2613166950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f219806efb0 0x7f2198072630 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f2190009dc0 tx=0x7f219000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609ebebba97ec00d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.843+0000 7f219559a700 1 -- 172.21.15.1:0/2613166950 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219000e070 con 0x7f219806efb0 2024-03-18T19:10:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.843+0000 7f219559a700 1 -- 172.21.15.1:0/2613166950 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219000bb90 con 0x7f219806efb0 2024-03-18T19:10:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.843+0000 7f219559a700 1 -- 172.21.15.1:0/2613166950 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2190012430 con 0x7f219806efb0 2024-03-18T19:10:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.844+0000 7f219cf02700 1 -- 172.21.15.1:0/2613166950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f219806efb0 msgr2=0x7f2198072630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.844+0000 7f219cf02700 1 --2- 172.21.15.1:0/2613166950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f219806efb0 0x7f2198072630 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f2190009dc0 tx=0x7f219000b330 comp rx=0 tx=0).stop 2024-03-18T19:10:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.844+0000 7f219cf02700 1 -- 172.21.15.1:0/2613166950 shutdown_connections 2024-03-18T19:10:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.844+0000 7f219cf02700 1 --2- 172.21.15.1:0/2613166950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f219806efb0 0x7f2198072630 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.844+0000 7f219cf02700 1 --2- 172.21.15.1:0/2613166950 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f219806eb30 0x7f21980720f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.844+0000 7f219cf02700 1 -- 172.21.15.1:0/2613166950 >> 172.21.15.1:0/2613166950 conn(0x7f21980b1d20 msgr2=0x7f21980b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.844+0000 7f219cf02700 1 -- 172.21.15.1:0/2613166950 shutdown_connections 2024-03-18T19:10:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.844+0000 7f219cf02700 1 -- 172.21.15.1:0/2613166950 wait complete. 2024-03-18T19:10:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.845+0000 7f219cf02700 1 Processor -- start 2024-03-18T19:10:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.845+0000 7f219cf02700 1 -- start start 2024-03-18T19:10:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f219cf02700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f219806a750 0x7f21981f0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f219cf02700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f219806eb30 0x7f219813cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f219cf02700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f219806efb0 0x7f21981427f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f219cf02700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21981389c0 con 0x7f219806eb30 2024-03-18T19:10:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f219cf02700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2198137db0 con 0x7f219806a750 2024-03-18T19:10:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f219cf02700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2198138090 con 0x7f219806efb0 2024-03-18T19:10:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f2195d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f219806eb30 0x7f219813cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f2195d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f219806eb30 0x7f219813cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35350/0 (socket says 172.21.15.1:35350) 2024-03-18T19:10:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f2195d9b700 1 -- 172.21.15.1:0/4015971496 learned_addr learned my addr 172.21.15.1:0/4015971496 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:01.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f2196d9d700 1 --2- 172.21.15.1:0/4015971496 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f219806efb0 0x7f21981427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f219659c700 1 --2- 172.21.15.1:0/4015971496 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f219806a750 0x7f21981f0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f2195d9b700 1 -- 172.21.15.1:0/4015971496 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f219806efb0 msgr2=0x7f21981427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f2195d9b700 1 --2- 172.21.15.1:0/4015971496 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f219806efb0 0x7f21981427f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f2195d9b700 1 -- 172.21.15.1:0/4015971496 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f219806a750 msgr2=0x7f21981f0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f2195d9b700 1 --2- 172.21.15.1:0/4015971496 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f219806a750 0x7f21981f0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f2195d9b700 1 -- 172.21.15.1:0/4015971496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2190009a70 con 0x7f219806eb30 2024-03-18T19:10:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f219659c700 1 --2- 172.21.15.1:0/4015971496 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f219806a750 0x7f21981f0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:10:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f2195d9b700 1 --2- 172.21.15.1:0/4015971496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f219806eb30 0x7f219813cec0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f218800e810 tx=0x7f218800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f2186ffd700 1 -- 172.21.15.1:0/4015971496 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2188013610 con 0x7f219806eb30 2024-03-18T19:10:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f2186ffd700 1 -- 172.21.15.1:0/4015971496 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2188013770 con 0x7f219806eb30 2024-03-18T19:10:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.846+0000 7f219cf02700 1 -- 172.21.15.1:0/4015971496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2198138370 con 0x7f219806eb30 2024-03-18T19:10:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.847+0000 7f2186ffd700 1 -- 172.21.15.1:0/4015971496 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2188013b20 con 0x7f219806eb30 2024-03-18T19:10:01.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.847+0000 7f219cf02700 1 -- 172.21.15.1:0/4015971496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f219815aae0 con 0x7f219806eb30 2024-03-18T19:10:01.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.848+0000 7f219cf02700 1 -- 172.21.15.1:0/4015971496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2198138b70 con 0x7f219806eb30 2024-03-18T19:10:01.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.848+0000 7f2186ffd700 1 -- 172.21.15.1:0/4015971496 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2188013c80 con 0x7f219806eb30 2024-03-18T19:10:01.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.849+0000 7f2186ffd700 1 --2- 172.21.15.1:0/4015971496 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f217806adb0 0x7f217806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:01.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.849+0000 7f219659c700 1 --2- 172.21.15.1:0/4015971496 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f217806adb0 0x7f217806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:01.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.849+0000 7f2186ffd700 1 -- 172.21.15.1:0/4015971496 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5205+0+0 (secure 0 0 0) 0x7f2188089020 con 0x7f219806eb30 2024-03-18T19:10:01.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.849+0000 7f219659c700 1 --2- 172.21.15.1:0/4015971496 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f217806adb0 0x7f217806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f218c000f60 tx=0x7f218c009040 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:01.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:01.853+0000 7f2186ffd700 1 -- 172.21.15.1:0/4015971496 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2198138b70 con 0x7f219806eb30 2024-03-18T19:10:02.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.057+0000 7f219cf02700 1 -- 172.21.15.1:0/4015971496 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2198071130 con 0x7f219806eb30 2024-03-18T19:10:02.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.058+0000 7f2186ffd700 1 -- 172.21.15.1:0/4015971496 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v157) v1 ==== 84+0+65 (secure 0 0 0) 0x7f21880541e0 con 0x7f219806eb30 2024-03-18T19:10:02.054 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:10:02.054 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T19:10:02.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:02.054 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-18T19:10:02.054 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-18T19:10:02.054 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:02.054 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T19:10:02.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.060+0000 7f219cf02700 1 -- 172.21.15.1:0/4015971496 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f217806adb0 msgr2=0x7f217806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:02.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.060+0000 7f219cf02700 1 --2- 172.21.15.1:0/4015971496 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f217806adb0 0x7f217806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f218c000f60 tx=0x7f218c009040 comp rx=0 tx=0).stop 2024-03-18T19:10:02.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.060+0000 7f219cf02700 1 -- 172.21.15.1:0/4015971496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f219806eb30 msgr2=0x7f219813cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:02.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.060+0000 7f219cf02700 1 --2- 172.21.15.1:0/4015971496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f219806eb30 0x7f219813cec0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f218800e810 tx=0x7f218800ebd0 comp rx=0 tx=0).stop 2024-03-18T19:10:02.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.060+0000 7f219cf02700 1 -- 172.21.15.1:0/4015971496 shutdown_connections 2024-03-18T19:10:02.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.060+0000 7f219cf02700 1 --2- 172.21.15.1:0/4015971496 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f217806adb0 0x7f217806d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:02.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.060+0000 7f219cf02700 1 --2- 172.21.15.1:0/4015971496 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f219806efb0 0x7f21981427f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:02.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.060+0000 7f219cf02700 1 --2- 172.21.15.1:0/4015971496 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f219806a750 0x7f21981f0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:02.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.060+0000 7f219cf02700 1 --2- 172.21.15.1:0/4015971496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f219806eb30 0x7f219813cec0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:02.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.060+0000 7f219cf02700 1 -- 172.21.15.1:0/4015971496 >> 172.21.15.1:0/4015971496 conn(0x7f21980b1d20 msgr2=0x7f219806ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:02.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.060+0000 7f219cf02700 1 -- 172.21.15.1:0/4015971496 shutdown_connections 2024-03-18T19:10:02.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.060+0000 7f219cf02700 1 -- 172.21.15.1:0/4015971496 wait complete. 2024-03-18T19:10:02.068 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T19:10:02.068 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T19:10:02.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.203+0000 7fc6c37b6700 1 Processor -- start 2024-03-18T19:10:02.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.203+0000 7fc6c37b6700 1 -- start start 2024-03-18T19:10:02.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.204+0000 7fc6c37b6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc6bc15d610 0x7fc6bc0cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:02.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.204+0000 7fc6c37b6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc6bc0cfbb0 0x7fc6bc06e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:02.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.204+0000 7fc6c37b6700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6bc0b2920 con 0x7fc6bc06e540 2024-03-18T19:10:02.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.204+0000 7fc6c37b6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6bc06eda0 con 0x7fc6bc15d610 2024-03-18T19:10:02.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.204+0000 7fc6c37b6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6bc0b2bc0 con 0x7fc6bc0cfbb0 2024-03-18T19:10:02.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.204+0000 7fc6c1552700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc6bc15d610 0x7fc6bc0cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:02.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.204+0000 7fc6c1552700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc6bc15d610 0x7fc6bc0cf670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35356/0 (socket says 172.21.15.1:35356) 2024-03-18T19:10:02.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.204+0000 7fc6c1552700 1 -- 172.21.15.1:0/1820364498 learned_addr learned my addr 172.21.15.1:0/1820364498 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:02.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.205+0000 7fc6c0d51700 1 --2- 172.21.15.1:0/1820364498 >> v2:172.21.15.165:3300/0 conn(0x7fc6bc0cfbb0 0x7fc6bc06e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:02.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.205+0000 7fc6c1552700 1 -- 172.21.15.1:0/1820364498 >> v2:172.21.15.165:3300/0 conn(0x7fc6bc0cfbb0 msgr2=0x7fc6bc06e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.205+0000 7fc6c1552700 1 --2- 172.21.15.1:0/1820364498 >> v2:172.21.15.165:3300/0 conn(0x7fc6bc0cfbb0 0x7fc6bc06e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.205+0000 7fc6c1552700 1 -- 172.21.15.1:0/1820364498 >> v1:172.21.15.165:6789/0 conn(0x7fc6bc06e540 legacy=0x7fc6bc0cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.205+0000 7fc6c1552700 1 -- 172.21.15.1:0/1820364498 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6bc0b2ea0 con 0x7fc6bc15d610 2024-03-18T19:10:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.205+0000 7fc6c0d51700 1 --2- 172.21.15.1:0/1820364498 >> v2:172.21.15.165:3300/0 conn(0x7fc6bc0cfbb0 0x7fc6bc06e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:10:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.205+0000 7fc6c1552700 1 --2- 172.21.15.1:0/1820364498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6bc15d610 0x7fc6bc0cf670 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc06da90 tx=0x7fc6ac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc1293dadf0a9c69 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.205+0000 7fc6b3fff700 1 -- 172.21.15.1:0/1820364498 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac00b870 con 0x7fc6bc15d610 2024-03-18T19:10:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.205+0000 7fc6b3fff700 1 -- 172.21.15.1:0/1820364498 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ac00b9d0 con 0x7fc6bc15d610 2024-03-18T19:10:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.205+0000 7fc6b3fff700 1 -- 172.21.15.1:0/1820364498 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac00bd60 con 0x7fc6bc15d610 2024-03-18T19:10:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.206+0000 7fc6c37b6700 1 -- 172.21.15.1:0/1820364498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6bc15d610 msgr2=0x7fc6bc0cf670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.206+0000 7fc6c37b6700 1 --2- 172.21.15.1:0/1820364498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6bc15d610 0x7fc6bc0cf670 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc06da90 tx=0x7fc6ac009bd0 comp rx=0 tx=0).stop 2024-03-18T19:10:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.206+0000 7fc6c37b6700 1 -- 172.21.15.1:0/1820364498 shutdown_connections 2024-03-18T19:10:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.206+0000 7fc6c37b6700 1 --2- 172.21.15.1:0/1820364498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6bc15d610 0x7fc6bc0cf670 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.206+0000 7fc6c37b6700 1 --2- 172.21.15.1:0/1820364498 >> v2:172.21.15.165:3300/0 conn(0x7fc6bc0cfbb0 0x7fc6bc06e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:02.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.206+0000 7fc6c37b6700 1 -- 172.21.15.1:0/1820364498 >> 172.21.15.1:0/1820364498 conn(0x7fc6bc06a750 msgr2=0x7fc6bc06ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:02.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.206+0000 7fc6c37b6700 1 -- 172.21.15.1:0/1820364498 shutdown_connections 2024-03-18T19:10:02.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.206+0000 7fc6c37b6700 1 -- 172.21.15.1:0/1820364498 wait complete. 2024-03-18T19:10:02.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.207+0000 7fc6c37b6700 1 Processor -- start 2024-03-18T19:10:02.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.207+0000 7fc6c37b6700 1 -- start start 2024-03-18T19:10:02.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.208+0000 7fc6c37b6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6bc06e540 0x7fc6bc0c64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:02.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.208+0000 7fc6c37b6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6bc0cfbb0 0x7fc6bc157d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:02.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.208+0000 7fc6c37b6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6bc15d610 0x7fc6bc158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:02.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.208+0000 7fc6c37b6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc1604f0 con 0x7fc6bc15d610 2024-03-18T19:10:02.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.208+0000 7fc6c37b6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc15ff30 con 0x7fc6bc06e540 2024-03-18T19:10:02.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.208+0000 7fc6c37b6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc160210 con 0x7fc6bc0cfbb0 2024-03-18T19:10:02.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.208+0000 7fc6c1d53700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6bc15d610 0x7fc6bc158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:02.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.208+0000 7fc6c1d53700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6bc15d610 0x7fc6bc158290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35362/0 (socket says 172.21.15.1:35362) 2024-03-18T19:10:02.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.208+0000 7fc6c1d53700 1 -- 172.21.15.1:0/3103576467 learned_addr learned my addr 172.21.15.1:0/3103576467 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:02.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.208+0000 7fc6c1552700 1 --2- 172.21.15.1:0/3103576467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6bc06e540 0x7fc6bc0c64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:02.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.208+0000 7fc6c0d51700 1 --2- 172.21.15.1:0/3103576467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6bc0cfbb0 0x7fc6bc157d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:02.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.208+0000 7fc6c1d53700 1 -- 172.21.15.1:0/3103576467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6bc0cfbb0 msgr2=0x7fc6bc157d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:02.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.208+0000 7fc6c1d53700 1 --2- 172.21.15.1:0/3103576467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6bc0cfbb0 0x7fc6bc157d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:02.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.208+0000 7fc6c1d53700 1 -- 172.21.15.1:0/3103576467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6bc06e540 msgr2=0x7fc6bc0c64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:02.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.208+0000 7fc6c1d53700 1 --2- 172.21.15.1:0/3103576467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6bc06e540 0x7fc6bc0c64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:02.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.208+0000 7fc6c1d53700 1 -- 172.21.15.1:0/3103576467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ac009550 con 0x7fc6bc15d610 2024-03-18T19:10:02.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.209+0000 7fc6c1d53700 1 --2- 172.21.15.1:0/3103576467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6bc15d610 0x7fc6bc158290 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fc6b400cc50 tx=0x7fc6b400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:02.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.209+0000 7fc6b1ffb700 1 -- 172.21.15.1:0/3103576467 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b4003c00 con 0x7fc6bc15d610 2024-03-18T19:10:02.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.209+0000 7fc6b1ffb700 1 -- 172.21.15.1:0/3103576467 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b4003d60 con 0x7fc6bc15d610 2024-03-18T19:10:02.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.209+0000 7fc6c37b6700 1 -- 172.21.15.1:0/3103576467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6bc15dd90 con 0x7fc6bc15d610 2024-03-18T19:10:02.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.209+0000 7fc6b1ffb700 1 -- 172.21.15.1:0/3103576467 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b400aab0 con 0x7fc6bc15d610 2024-03-18T19:10:02.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.209+0000 7fc6c37b6700 1 -- 172.21.15.1:0/3103576467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6bc160ab0 con 0x7fc6bc15d610 2024-03-18T19:10:02.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.210+0000 7fc6c37b6700 1 -- 172.21.15.1:0/3103576467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6bc15e2b0 con 0x7fc6bc15d610 2024-03-18T19:10:02.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.210+0000 7fc6b1ffb700 1 -- 172.21.15.1:0/3103576467 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc6b400ac10 con 0x7fc6bc15d610 2024-03-18T19:10:02.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.211+0000 7fc6b1ffb700 1 --2- 172.21.15.1:0/3103576467 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc6a006adb0 0x7fc6a006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:02.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.211+0000 7fc6b1ffb700 1 -- 172.21.15.1:0/3103576467 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5205+0+0 (secure 0 0 0) 0x7fc6b40893b0 con 0x7fc6bc15d610 2024-03-18T19:10:02.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.211+0000 7fc6c1552700 1 --2- 172.21.15.1:0/3103576467 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc6a006adb0 0x7fc6a006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:02.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.211+0000 7fc6c1552700 1 --2- 172.21.15.1:0/3103576467 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc6a006adb0 0x7fc6a006d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac014040 tx=0x7fc6ac008030 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:02.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.215+0000 7fc6b1ffb700 1 -- 172.21.15.1:0/3103576467 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc6bc15e2b0 con 0x7fc6bc15d610 2024-03-18T19:10:02.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:02.421+0000 7fc6c37b6700 1 -- 172.21.15.1:0/3103576467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc6bc15e2b0 con 0x7fc6bc15d610 2024-03-18T19:10:03.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.177+0000 7fc6b1ffb700 1 -- 172.21.15.1:0/3103576467 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v158) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc6bc15e2b0 con 0x7fc6bc15d610 2024-03-18T19:10:03.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.182+0000 7fc6c37b6700 1 -- 172.21.15.1:0/3103576467 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc6a006adb0 msgr2=0x7fc6a006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:03.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.182+0000 7fc6c37b6700 1 --2- 172.21.15.1:0/3103576467 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc6a006adb0 0x7fc6a006d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac014040 tx=0x7fc6ac008030 comp rx=0 tx=0).stop 2024-03-18T19:10:03.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.182+0000 7fc6c37b6700 1 -- 172.21.15.1:0/3103576467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6bc15d610 msgr2=0x7fc6bc158290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:03.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.182+0000 7fc6c37b6700 1 --2- 172.21.15.1:0/3103576467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6bc15d610 0x7fc6bc158290 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fc6b400cc50 tx=0x7fc6b400cf60 comp rx=0 tx=0).stop 2024-03-18T19:10:03.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.183+0000 7fc6c37b6700 1 -- 172.21.15.1:0/3103576467 shutdown_connections 2024-03-18T19:10:03.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.183+0000 7fc6c37b6700 1 --2- 172.21.15.1:0/3103576467 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc6a006adb0 0x7fc6a006d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:03.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.183+0000 7fc6c37b6700 1 --2- 172.21.15.1:0/3103576467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6bc15d610 0x7fc6bc158290 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:03.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.183+0000 7fc6c37b6700 1 --2- 172.21.15.1:0/3103576467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6bc06e540 0x7fc6bc0c64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:03.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.183+0000 7fc6c37b6700 1 --2- 172.21.15.1:0/3103576467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6bc0cfbb0 0x7fc6bc157d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:03.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.183+0000 7fc6c37b6700 1 -- 172.21.15.1:0/3103576467 >> 172.21.15.1:0/3103576467 conn(0x7fc6bc06a750 msgr2=0x7fc6bc0af3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:03.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.183+0000 7fc6c37b6700 1 -- 172.21.15.1:0/3103576467 shutdown_connections 2024-03-18T19:10:03.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.183+0000 7fc6c37b6700 1 -- 172.21.15.1:0/3103576467 wait complete. 2024-03-18T19:10:03.181 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-18T19:10:03.194 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T19:10:03.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.338+0000 7f4fa0eab700 1 Processor -- start 2024-03-18T19:10:03.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.338+0000 7f4fa0eab700 1 -- start start 2024-03-18T19:10:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.339+0000 7f4fa0eab700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4f9c0727b0 0x7f4f9c161610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.339+0000 7f4fa0eab700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4f9c161b50 0x7f4f9c06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.339+0000 7f4fa0eab700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f9c160ea0 con 0x7f4f9c15ad70 2024-03-18T19:10:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.339+0000 7f4fa0eab700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f9c0725b0 con 0x7f4f9c0727b0 2024-03-18T19:10:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.339+0000 7f4fa0eab700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f9c15e090 con 0x7f4f9c161b50 2024-03-18T19:10:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.340+0000 7f4f99d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4f9c0727b0 0x7f4f9c161610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.340+0000 7f4f99d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4f9c0727b0 0x7f4f9c161610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35366/0 (socket says 172.21.15.1:35366) 2024-03-18T19:10:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.340+0000 7f4f99d9b700 1 -- 172.21.15.1:0/62078565 learned_addr learned my addr 172.21.15.1:0/62078565 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:03.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.340+0000 7f4f9ad9d700 1 --2- 172.21.15.1:0/62078565 >> v2:172.21.15.165:3300/0 conn(0x7f4f9c161b50 0x7f4f9c06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.340+0000 7f4f99d9b700 1 -- 172.21.15.1:0/62078565 >> v2:172.21.15.165:3300/0 conn(0x7f4f9c161b50 msgr2=0x7f4f9c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.340+0000 7f4f99d9b700 1 --2- 172.21.15.1:0/62078565 >> v2:172.21.15.165:3300/0 conn(0x7f4f9c161b50 0x7f4f9c06a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.340+0000 7f4f99d9b700 1 -- 172.21.15.1:0/62078565 >> v1:172.21.15.165:6789/0 conn(0x7f4f9c15ad70 legacy=0x7f4f9c15d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.340+0000 7f4f99d9b700 1 -- 172.21.15.1:0/62078565 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f9c15e370 con 0x7f4f9c0727b0 2024-03-18T19:10:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.340+0000 7f4f99d9b700 1 --2- 172.21.15.1:0/62078565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f9c0727b0 0x7f4f9c161610 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c071600 tx=0x7f4f8c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6439c9868ad95a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.340+0000 7f4f9959a700 1 -- 172.21.15.1:0/62078565 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c004230 con 0x7f4f9c0727b0 2024-03-18T19:10:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.340+0000 7f4f9959a700 1 -- 172.21.15.1:0/62078565 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8c004390 con 0x7f4f9c0727b0 2024-03-18T19:10:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.340+0000 7f4f9959a700 1 -- 172.21.15.1:0/62078565 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c0036a0 con 0x7f4f9c0727b0 2024-03-18T19:10:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.341+0000 7f4fa0eab700 1 -- 172.21.15.1:0/62078565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f9c0727b0 msgr2=0x7f4f9c161610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.341+0000 7f4fa0eab700 1 --2- 172.21.15.1:0/62078565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f9c0727b0 0x7f4f9c161610 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c071600 tx=0x7f4f8c009bd0 comp rx=0 tx=0).stop 2024-03-18T19:10:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.341+0000 7f4fa0eab700 1 -- 172.21.15.1:0/62078565 shutdown_connections 2024-03-18T19:10:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.341+0000 7f4fa0eab700 1 --2- 172.21.15.1:0/62078565 >> v2:172.21.15.165:3300/0 conn(0x7f4f9c161b50 0x7f4f9c06a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.341+0000 7f4fa0eab700 1 --2- 172.21.15.1:0/62078565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f9c0727b0 0x7f4f9c161610 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:03.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.341+0000 7f4fa0eab700 1 -- 172.21.15.1:0/62078565 >> 172.21.15.1:0/62078565 conn(0x7f4f9c0b1cb0 msgr2=0x7f4f9c0b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:03.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.341+0000 7f4fa0eab700 1 -- 172.21.15.1:0/62078565 shutdown_connections 2024-03-18T19:10:03.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.341+0000 7f4fa0eab700 1 -- 172.21.15.1:0/62078565 wait complete. 2024-03-18T19:10:03.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.343+0000 7f4fa0eab700 1 Processor -- start 2024-03-18T19:10:03.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.343+0000 7f4fa0eab700 1 -- start start 2024-03-18T19:10:03.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.344+0000 7f4fa0eab700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f9c0727b0 0x7f4f9c14ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:03.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.344+0000 7f4fa0eab700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f9c15ad70 0x7f4f9c0c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:03.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.344+0000 7f4fa0eab700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f9c161b50 0x7f4f9c0c5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:03.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.344+0000 7f4fa0eab700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c14bc20 con 0x7f4f9c0727b0 2024-03-18T19:10:03.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.344+0000 7f4fa0eab700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c14b080 con 0x7f4f9c15ad70 2024-03-18T19:10:03.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.344+0000 7f4fa0eab700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c14b2f0 con 0x7f4f9c161b50 2024-03-18T19:10:03.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.344+0000 7f4f9a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f9c0727b0 0x7f4f9c14ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:03.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.344+0000 7f4f9a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f9c0727b0 0x7f4f9c14ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35382/0 (socket says 172.21.15.1:35382) 2024-03-18T19:10:03.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.344+0000 7f4f9a59c700 1 -- 172.21.15.1:0/3381424287 learned_addr learned my addr 172.21.15.1:0/3381424287 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:03.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.344+0000 7f4f99d9b700 1 --2- 172.21.15.1:0/3381424287 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f9c15ad70 0x7f4f9c0c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:03.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.344+0000 7f4f9a59c700 1 -- 172.21.15.1:0/3381424287 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f9c161b50 msgr2=0x7f4f9c0c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:03.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.344+0000 7f4f9a59c700 1 --2- 172.21.15.1:0/3381424287 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f9c161b50 0x7f4f9c0c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:03.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.344+0000 7f4f9a59c700 1 -- 172.21.15.1:0/3381424287 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f9c15ad70 msgr2=0x7f4f9c0c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:03.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.344+0000 7f4f9a59c700 1 --2- 172.21.15.1:0/3381424287 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f9c15ad70 0x7f4f9c0c4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:03.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.344+0000 7f4f9a59c700 1 -- 172.21.15.1:0/3381424287 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c009550 con 0x7f4f9c0727b0 2024-03-18T19:10:03.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.344+0000 7f4f9a59c700 1 --2- 172.21.15.1:0/3381424287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f9c0727b0 0x7f4f9c14ab40 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000c4d0 tx=0x7f4f9000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:03.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.344+0000 7f4f8affd700 1 -- 172.21.15.1:0/3381424287 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f90011790 con 0x7f4f9c0727b0 2024-03-18T19:10:03.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.345+0000 7f4f8affd700 1 -- 172.21.15.1:0/3381424287 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f900118f0 con 0x7f4f9c0727b0 2024-03-18T19:10:03.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.345+0000 7f4fa0eab700 1 -- 172.21.15.1:0/3381424287 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f9c14b5d0 con 0x7f4f9c0727b0 2024-03-18T19:10:03.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.345+0000 7f4f8affd700 1 -- 172.21.15.1:0/3381424287 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f90011ca0 con 0x7f4f9c0727b0 2024-03-18T19:10:03.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.345+0000 7f4fa0eab700 1 -- 172.21.15.1:0/3381424287 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f9c1f3df0 con 0x7f4f9c0727b0 2024-03-18T19:10:03.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.345+0000 7f4fa0eab700 1 -- 172.21.15.1:0/3381424287 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f9c0b0bc0 con 0x7f4f9c0727b0 2024-03-18T19:10:03.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.346+0000 7f4f8affd700 1 -- 172.21.15.1:0/3381424287 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4f9001c460 con 0x7f4f9c0727b0 2024-03-18T19:10:03.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.346+0000 7f4f8affd700 1 --2- 172.21.15.1:0/3381424287 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4f7c06adb0 0x7f4f7c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:03.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.347+0000 7f4f8affd700 1 -- 172.21.15.1:0/3381424287 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f4f90089220 con 0x7f4f9c0727b0 2024-03-18T19:10:03.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.347+0000 7f4f99d9b700 1 --2- 172.21.15.1:0/3381424287 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4f7c06adb0 0x7f4f7c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:03.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.347+0000 7f4f99d9b700 1 --2- 172.21.15.1:0/3381424287 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4f7c06adb0 0x7f4f7c06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0cf990 tx=0x7f4f8c008030 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:03.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.351+0000 7f4f8affd700 1 -- 172.21.15.1:0/3381424287 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4f90054370 con 0x7f4f9c0727b0 2024-03-18T19:10:03.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:03.553+0000 7f4fa0eab700 1 -- 172.21.15.1:0/3381424287 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4f9c0676a0 con 0x7f4f9c0727b0 2024-03-18T19:10:04.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.183+0000 7f4f8affd700 1 -- 172.21.15.1:0/3381424287 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v159) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4f90057990 con 0x7f4f9c0727b0 2024-03-18T19:10:04.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.194+0000 7f4fa0eab700 1 -- 172.21.15.1:0/3381424287 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4f7c06adb0 msgr2=0x7f4f7c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:04.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.194+0000 7f4fa0eab700 1 --2- 172.21.15.1:0/3381424287 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4f7c06adb0 0x7f4f7c06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0cf990 tx=0x7f4f8c008030 comp rx=0 tx=0).stop 2024-03-18T19:10:04.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.194+0000 7f4fa0eab700 1 -- 172.21.15.1:0/3381424287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f9c0727b0 msgr2=0x7f4f9c14ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:04.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.194+0000 7f4fa0eab700 1 --2- 172.21.15.1:0/3381424287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f9c0727b0 0x7f4f9c14ab40 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000c4d0 tx=0x7f4f9000c890 comp rx=0 tx=0).stop 2024-03-18T19:10:04.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.195+0000 7f4fa0eab700 1 -- 172.21.15.1:0/3381424287 shutdown_connections 2024-03-18T19:10:04.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.195+0000 7f4fa0eab700 1 --2- 172.21.15.1:0/3381424287 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4f7c06adb0 0x7f4f7c06d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.195+0000 7f4fa0eab700 1 --2- 172.21.15.1:0/3381424287 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f9c161b50 0x7f4f9c0c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.195+0000 7f4fa0eab700 1 --2- 172.21.15.1:0/3381424287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f9c0727b0 0x7f4f9c14ab40 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.195+0000 7f4fa0eab700 1 --2- 172.21.15.1:0/3381424287 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f9c15ad70 0x7f4f9c0c4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.195+0000 7f4fa0eab700 1 -- 172.21.15.1:0/3381424287 >> 172.21.15.1:0/3381424287 conn(0x7f4f9c0b1cb0 msgr2=0x7f4f9c0b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:04.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.196+0000 7f4fa0eab700 1 -- 172.21.15.1:0/3381424287 shutdown_connections 2024-03-18T19:10:04.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.196+0000 7f4fa0eab700 1 -- 172.21.15.1:0/3381424287 wait complete. 2024-03-18T19:10:04.198 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-18T19:10:04.212 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T19:10:04.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.351+0000 7fa88e197700 1 Processor -- start 2024-03-18T19:10:04.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.351+0000 7fa88e197700 1 -- start start 2024-03-18T19:10:04.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa88e197700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa88806a750 0x7fa88806ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:04.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa88e197700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa8880ace80 0x7fa8880ad300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:04.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa88e197700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa8880ad840 0x7fa8880caa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:04.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa88e197700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa888070090 con 0x7fa8880ace80 2024-03-18T19:10:04.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa88e197700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa88815c340 con 0x7fa8880ad840 2024-03-18T19:10:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa88e197700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa88815b7a0 con 0x7fa88806a750 2024-03-18T19:10:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa886ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa8880ace80 0x7fa8880ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa886ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa8880ace80 0x7fa8880ad300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35396/0 (socket says 172.21.15.1:35396) 2024-03-18T19:10:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa886ffd700 1 -- 172.21.15.1:0/499157802 learned_addr learned my addr 172.21.15.1:0/499157802 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa8877fe700 1 --2- 172.21.15.1:0/499157802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa88806a750 0x7fa88806ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa887fff700 1 --2- 172.21.15.1:0/499157802 >> v2:172.21.15.165:3300/0 conn(0x7fa8880ad840 0x7fa8880caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa886ffd700 1 -- 172.21.15.1:0/499157802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa88806a750 msgr2=0x7fa88806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa886ffd700 1 --2- 172.21.15.1:0/499157802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa88806a750 0x7fa88806ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa886ffd700 1 -- 172.21.15.1:0/499157802 >> v2:172.21.15.165:3300/0 conn(0x7fa8880ad840 msgr2=0x7fa8880caa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa886ffd700 1 --2- 172.21.15.1:0/499157802 >> v2:172.21.15.165:3300/0 conn(0x7fa8880ad840 0x7fa8880caa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa886ffd700 1 -- 172.21.15.1:0/499157802 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa88806b070 con 0x7fa8880ace80 2024-03-18T19:10:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa8877fe700 1 --2- 172.21.15.1:0/499157802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa88806a750 0x7fa88806ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:10:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa887fff700 1 --2- 172.21.15.1:0/499157802 >> v2:172.21.15.165:3300/0 conn(0x7fa8880ad840 0x7fa8880caa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:10:04.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa886ffd700 1 --2- 172.21.15.1:0/499157802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8880ace80 0x7fa8880ad300 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fa8880cd5f0 tx=0x7fa878009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588754e0d78f0d72 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:04.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa8867fc700 1 -- 172.21.15.1:0/499157802 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87800b870 con 0x7fa8880ace80 2024-03-18T19:10:04.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.352+0000 7fa8867fc700 1 -- 172.21.15.1:0/499157802 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa87800b9d0 con 0x7fa8880ace80 2024-03-18T19:10:04.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.353+0000 7fa8867fc700 1 -- 172.21.15.1:0/499157802 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87800bd60 con 0x7fa8880ace80 2024-03-18T19:10:04.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.353+0000 7fa88e197700 1 -- 172.21.15.1:0/499157802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8880ace80 msgr2=0x7fa8880ad300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:04.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.353+0000 7fa88e197700 1 --2- 172.21.15.1:0/499157802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8880ace80 0x7fa8880ad300 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fa8880cd5f0 tx=0x7fa878009bd0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.353+0000 7fa88e197700 1 -- 172.21.15.1:0/499157802 shutdown_connections 2024-03-18T19:10:04.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.353+0000 7fa88e197700 1 --2- 172.21.15.1:0/499157802 >> v2:172.21.15.165:3300/0 conn(0x7fa8880ad840 0x7fa8880caa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.353+0000 7fa88e197700 1 --2- 172.21.15.1:0/499157802 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa88806a750 0x7fa88806ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.353+0000 7fa88e197700 1 --2- 172.21.15.1:0/499157802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8880ace80 0x7fa8880ad300 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.353+0000 7fa88e197700 1 -- 172.21.15.1:0/499157802 >> 172.21.15.1:0/499157802 conn(0x7fa8880cc9b0 msgr2=0x7fa8880ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:04.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.353+0000 7fa88e197700 1 -- 172.21.15.1:0/499157802 shutdown_connections 2024-03-18T19:10:04.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.353+0000 7fa88e197700 1 -- 172.21.15.1:0/499157802 wait complete. 2024-03-18T19:10:04.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.354+0000 7fa88e197700 1 Processor -- start 2024-03-18T19:10:04.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.355+0000 7fa88e197700 1 -- start start 2024-03-18T19:10:04.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.355+0000 7fa88e197700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa88806a750 0x7fa8880c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:04.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.355+0000 7fa88e197700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8880ace80 0x7fa888146ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:04.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.355+0000 7fa88e197700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8880ad840 0x7fa8880c3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:04.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.355+0000 7fa88e197700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa888147430 con 0x7fa88806a750 2024-03-18T19:10:04.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.355+0000 7fa88e197700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8880c3f80 con 0x7fa8880ace80 2024-03-18T19:10:04.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.355+0000 7fa88e197700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8880c6b50 con 0x7fa8880ad840 2024-03-18T19:10:04.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.355+0000 7fa8877fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa88806a750 0x7fa8880c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:04.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.355+0000 7fa8877fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa88806a750 0x7fa8880c5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35412/0 (socket says 172.21.15.1:35412) 2024-03-18T19:10:04.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.355+0000 7fa8877fe700 1 -- 172.21.15.1:0/2636895371 learned_addr learned my addr 172.21.15.1:0/2636895371 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:04.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.356+0000 7fa887fff700 1 --2- 172.21.15.1:0/2636895371 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8880ad840 0x7fa8880c3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:04.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.356+0000 7fa886ffd700 1 --2- 172.21.15.1:0/2636895371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8880ace80 0x7fa888146ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:04.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.356+0000 7fa8877fe700 1 -- 172.21.15.1:0/2636895371 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8880ad840 msgr2=0x7fa8880c3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:04.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.356+0000 7fa8877fe700 1 --2- 172.21.15.1:0/2636895371 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8880ad840 0x7fa8880c3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.356+0000 7fa8877fe700 1 -- 172.21.15.1:0/2636895371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8880ace80 msgr2=0x7fa888146ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:04.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.356+0000 7fa8877fe700 1 --2- 172.21.15.1:0/2636895371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8880ace80 0x7fa888146ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.356+0000 7fa8877fe700 1 -- 172.21.15.1:0/2636895371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa878009550 con 0x7fa88806a750 2024-03-18T19:10:04.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.356+0000 7fa8877fe700 1 --2- 172.21.15.1:0/2636895371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa88806a750 0x7fa8880c5e80 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fa870006fd0 tx=0x7fa87000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:04.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.356+0000 7fa87ffff700 1 -- 172.21.15.1:0/2636895371 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa870005c20 con 0x7fa88806a750 2024-03-18T19:10:04.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.356+0000 7fa87ffff700 1 -- 172.21.15.1:0/2636895371 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa870005d80 con 0x7fa88806a750 2024-03-18T19:10:04.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.356+0000 7fa88e197700 1 -- 172.21.15.1:0/2636895371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa88814d550 con 0x7fa88806a750 2024-03-18T19:10:04.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.356+0000 7fa87ffff700 1 -- 172.21.15.1:0/2636895371 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa870011840 con 0x7fa88806a750 2024-03-18T19:10:04.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.356+0000 7fa88e197700 1 -- 172.21.15.1:0/2636895371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8880c6870 con 0x7fa88806a750 2024-03-18T19:10:04.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.357+0000 7fa88e197700 1 -- 172.21.15.1:0/2636895371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8880676a0 con 0x7fa88806a750 2024-03-18T19:10:04.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.358+0000 7fa87ffff700 1 -- 172.21.15.1:0/2636895371 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa8700119a0 con 0x7fa88806a750 2024-03-18T19:10:04.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.358+0000 7fa87ffff700 1 --2- 172.21.15.1:0/2636895371 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa86806adb0 0x7fa86806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:04.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.358+0000 7fa87ffff700 1 -- 172.21.15.1:0/2636895371 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fa8700898f0 con 0x7fa88806a750 2024-03-18T19:10:04.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.358+0000 7fa886ffd700 1 --2- 172.21.15.1:0/2636895371 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa86806adb0 0x7fa86806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:04.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.358+0000 7fa886ffd700 1 --2- 172.21.15.1:0/2636895371 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa86806adb0 0x7fa86806d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa878000c00 tx=0x7fa878008030 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:04.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.363+0000 7fa87ffff700 1 -- 172.21.15.1:0/2636895371 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa870054840 con 0x7fa88806a750 2024-03-18T19:10:04.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.611+0000 7fa88e197700 1 -- 172.21.15.1:0/2636895371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa88806f2f0 con 0x7fa88806a750 2024-03-18T19:10:04.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.616+0000 7fa87ffff700 1 -- 172.21.15.1:0/2636895371 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 19 and data pool 20 v69) v1 ==== 174+0+0 (secure 0 0 0) 0x7fa870057e60 con 0x7fa88806a750 2024-03-18T19:10:04.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.619+0000 7fa88e197700 1 -- 172.21.15.1:0/2636895371 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa86806adb0 msgr2=0x7fa86806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:04.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.620+0000 7fa88e197700 1 --2- 172.21.15.1:0/2636895371 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa86806adb0 0x7fa86806d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa878000c00 tx=0x7fa878008030 comp rx=0 tx=0).stop 2024-03-18T19:10:04.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.620+0000 7fa88e197700 1 -- 172.21.15.1:0/2636895371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa88806a750 msgr2=0x7fa8880c5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:04.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.620+0000 7fa88e197700 1 --2- 172.21.15.1:0/2636895371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa88806a750 0x7fa8880c5e80 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fa870006fd0 tx=0x7fa87000cae0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.620+0000 7fa88e197700 1 -- 172.21.15.1:0/2636895371 shutdown_connections 2024-03-18T19:10:04.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.620+0000 7fa88e197700 1 --2- 172.21.15.1:0/2636895371 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa86806adb0 0x7fa86806d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.620+0000 7fa88e197700 1 --2- 172.21.15.1:0/2636895371 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8880ad840 0x7fa8880c3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.620+0000 7fa88e197700 1 --2- 172.21.15.1:0/2636895371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa88806a750 0x7fa8880c5e80 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.620+0000 7fa88e197700 1 --2- 172.21.15.1:0/2636895371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8880ace80 0x7fa888146ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.620+0000 7fa88e197700 1 -- 172.21.15.1:0/2636895371 >> 172.21.15.1:0/2636895371 conn(0x7fa8880cc9b0 msgr2=0x7fa8880cb710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:04.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.621+0000 7fa88e197700 1 -- 172.21.15.1:0/2636895371 shutdown_connections 2024-03-18T19:10:04.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.621+0000 7fa88e197700 1 -- 172.21.15.1:0/2636895371 wait complete. 2024-03-18T19:10:04.628 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 19 and data pool 20 2024-03-18T19:10:04.639 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:10:04.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.768+0000 7fdb9b94b700 1 Processor -- start 2024-03-18T19:10:04.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.768+0000 7fdb9b94b700 1 -- start start 2024-03-18T19:10:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.769+0000 7fdb9b94b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdb94140b20 0x7fdb940cfe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.769+0000 7fdb9b94b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdb940cdf80 0x7fdb940cba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.769+0000 7fdb9b94b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb940d03d0 con 0x7fdb940cfab0 2024-03-18T19:10:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.769+0000 7fdb9b94b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb9415fbe0 con 0x7fdb940cdf80 2024-03-18T19:10:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.769+0000 7fdb9b94b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9415f060 con 0x7fdb94140b20 2024-03-18T19:10:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.769+0000 7fdb99ee8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdb940cdf80 0x7fdb940cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.769+0000 7fdb996e7700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdb940cfab0 0x7fdb9415c120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52596/0 (socket says 172.21.15.1:52596) 2024-03-18T19:10:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.769+0000 7fdb99ee8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdb940cdf80 0x7fdb940cba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50792/0 (socket says 172.21.15.1:50792) 2024-03-18T19:10:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.769+0000 7fdb996e7700 1 -- 172.21.15.1:0/2649288191 learned_addr learned my addr 172.21.15.1:0/2649288191 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:10:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.770+0000 7fdb98ee6700 1 --2- 172.21.15.1:0/2649288191 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdb94140b20 0x7fdb940cfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.770+0000 7fdb99ee8700 1 -- 172.21.15.1:0/2649288191 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdb94140b20 msgr2=0x7fdb940cfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.770+0000 7fdb99ee8700 1 --2- 172.21.15.1:0/2649288191 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdb94140b20 0x7fdb940cfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.770+0000 7fdb99ee8700 1 -- 172.21.15.1:0/2649288191 >> v1:172.21.15.1:6789/0 conn(0x7fdb940cfab0 legacy=0x7fdb9415c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.770+0000 7fdb99ee8700 1 -- 172.21.15.1:0/2649288191 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9415f340 con 0x7fdb940cdf80 2024-03-18T19:10:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.770+0000 7fdb8bfff700 1 -- 172.21.15.1:0/2649288191 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843541571 0 0) 0x7fdb940d03d0 con 0x7fdb940cfab0 2024-03-18T19:10:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.770+0000 7fdb99ee8700 1 --2- 172.21.15.1:0/2649288191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb940cdf80 0x7fdb940cba70 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c009d90 tx=0x7fdb8c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fe42e5cfbb16d6f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.770+0000 7fdb8bfff700 1 -- 172.21.15.1:0/2649288191 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c00e070 con 0x7fdb940cdf80 2024-03-18T19:10:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.770+0000 7fdb8bfff700 1 -- 172.21.15.1:0/2649288191 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8c00bbd0 con 0x7fdb940cdf80 2024-03-18T19:10:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.770+0000 7fdb8bfff700 1 -- 172.21.15.1:0/2649288191 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c012430 con 0x7fdb940cdf80 2024-03-18T19:10:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.770+0000 7fdb9b94b700 1 -- 172.21.15.1:0/2649288191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb940cdf80 msgr2=0x7fdb940cba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.770+0000 7fdb9b94b700 1 --2- 172.21.15.1:0/2649288191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb940cdf80 0x7fdb940cba70 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c009d90 tx=0x7fdb8c00b330 comp rx=0 tx=0).stop 2024-03-18T19:10:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.771+0000 7fdb9b94b700 1 -- 172.21.15.1:0/2649288191 shutdown_connections 2024-03-18T19:10:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.771+0000 7fdb9b94b700 1 --2- 172.21.15.1:0/2649288191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb940cdf80 0x7fdb940cba70 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.771+0000 7fdb9b94b700 1 --2- 172.21.15.1:0/2649288191 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdb94140b20 0x7fdb940cfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.771+0000 7fdb9b94b700 1 -- 172.21.15.1:0/2649288191 >> 172.21.15.1:0/2649288191 conn(0x7fdb9406ab70 msgr2=0x7fdb9406af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:04.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.771+0000 7fdb9b94b700 1 -- 172.21.15.1:0/2649288191 shutdown_connections 2024-03-18T19:10:04.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.771+0000 7fdb9b94b700 1 -- 172.21.15.1:0/2649288191 wait complete. 2024-03-18T19:10:04.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.772+0000 7fdb9b94b700 1 Processor -- start 2024-03-18T19:10:04.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.772+0000 7fdb9b94b700 1 -- start start 2024-03-18T19:10:04.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb9b94b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdb940cdf80 0x7fdb9415be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:04.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb9b94b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb940cfab0 0x7fdb94158b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:04.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb9b94b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb9415e510 0x7fdb9413fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:04.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb9b94b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb940afaa0 con 0x7fdb9415e510 2024-03-18T19:10:04.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb9b94b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9415cfc0 con 0x7fdb940cfab0 2024-03-18T19:10:04.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb9b94b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9415e9c0 con 0x7fdb940cdf80 2024-03-18T19:10:04.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb99ee8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb9415e510 0x7fdb9413fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:04.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb99ee8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb9415e510 0x7fdb9413fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50808/0 (socket says 172.21.15.1:50808) 2024-03-18T19:10:04.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb99ee8700 1 -- 172.21.15.1:0/173490617 learned_addr learned my addr 172.21.15.1:0/173490617 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:04.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb996e7700 1 --2- 172.21.15.1:0/173490617 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdb940cdf80 0x7fdb9415be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:04.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb98ee6700 1 --2- 172.21.15.1:0/173490617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb940cfab0 0x7fdb94158b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:04.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb99ee8700 1 -- 172.21.15.1:0/173490617 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdb940cdf80 msgr2=0x7fdb9415be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:04.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb99ee8700 1 --2- 172.21.15.1:0/173490617 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdb940cdf80 0x7fdb9415be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb99ee8700 1 -- 172.21.15.1:0/173490617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb940cfab0 msgr2=0x7fdb94158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:04.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb99ee8700 1 --2- 172.21.15.1:0/173490617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb940cfab0 0x7fdb94158b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb99ee8700 1 -- 172.21.15.1:0/173490617 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8c009a70 con 0x7fdb9415e510 2024-03-18T19:10:04.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb98ee6700 1 --2- 172.21.15.1:0/173490617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb940cfab0 0x7fdb94158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:10:04.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb99ee8700 1 --2- 172.21.15.1:0/173490617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb9415e510 0x7fdb9413fa60 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c013000 tx=0x7fdb8c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:04.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb89ffb700 1 -- 172.21.15.1:0/173490617 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c012430 con 0x7fdb9415e510 2024-03-18T19:10:04.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb89ffb700 1 -- 172.21.15.1:0/173490617 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8c020070 con 0x7fdb9415e510 2024-03-18T19:10:04.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb9b94b700 1 -- 172.21.15.1:0/173490617 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb9415c440 con 0x7fdb9415e510 2024-03-18T19:10:04.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.773+0000 7fdb89ffb700 1 -- 172.21.15.1:0/173490617 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c004280 con 0x7fdb9415e510 2024-03-18T19:10:04.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.774+0000 7fdb9b94b700 1 -- 172.21.15.1:0/173490617 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb9415ca00 con 0x7fdb9415e510 2024-03-18T19:10:04.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.774+0000 7fdb9b94b700 1 -- 172.21.15.1:0/173490617 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb940afbe0 con 0x7fdb9415e510 2024-03-18T19:10:04.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.775+0000 7fdb89ffb700 1 -- 172.21.15.1:0/173490617 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fdb8c0043e0 con 0x7fdb9415e510 2024-03-18T19:10:04.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.775+0000 7fdb89ffb700 1 --2- 172.21.15.1:0/173490617 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdb7806adb0 0x7fdb7806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:04.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.775+0000 7fdb89ffb700 1 -- 172.21.15.1:0/173490617 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fdb8c022070 con 0x7fdb9415e510 2024-03-18T19:10:04.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.776+0000 7fdb996e7700 1 --2- 172.21.15.1:0/173490617 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdb7806adb0 0x7fdb7806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:04.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.776+0000 7fdb996e7700 1 --2- 172.21.15.1:0/173490617 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdb7806adb0 0x7fdb7806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdb84007fd0 tx=0x7fdb84007de0 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:04.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.780+0000 7fdb89ffb700 1 -- 172.21.15.1:0/173490617 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdb8c056650 con 0x7fdb9415e510 2024-03-18T19:10:04.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.983+0000 7fdb9b94b700 1 -- 172.21.15.1:0/173490617 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb940676a0 con 0x7fdb9415e510 2024-03-18T19:10:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.985+0000 7fdb89ffb700 1 -- 172.21.15.1:0/173490617 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+16273 (secure 0 0 0) 0x7fdb8c059c70 con 0x7fdb9415e510 2024-03-18T19:10:04.981 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:10:04.981 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":160,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:10:04.612904+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:10:02.452367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-18T19:10:03.585045+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:10:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.987+0000 7fdb9b94b700 1 -- 172.21.15.1:0/173490617 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdb7806adb0 msgr2=0x7fdb7806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.987+0000 7fdb9b94b700 1 --2- 172.21.15.1:0/173490617 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdb7806adb0 0x7fdb7806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdb84007fd0 tx=0x7fdb84007de0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.987+0000 7fdb9b94b700 1 -- 172.21.15.1:0/173490617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb9415e510 msgr2=0x7fdb9413fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.987+0000 7fdb9b94b700 1 --2- 172.21.15.1:0/173490617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb9415e510 0x7fdb9413fa60 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c013000 tx=0x7fdb8c004ad0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.987+0000 7fdb9b94b700 1 -- 172.21.15.1:0/173490617 shutdown_connections 2024-03-18T19:10:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.987+0000 7fdb9b94b700 1 --2- 172.21.15.1:0/173490617 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdb7806adb0 0x7fdb7806d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.987+0000 7fdb9b94b700 1 --2- 172.21.15.1:0/173490617 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdb940cdf80 0x7fdb9415be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.987+0000 7fdb9b94b700 1 --2- 172.21.15.1:0/173490617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb9415e510 0x7fdb9413fa60 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.987+0000 7fdb9b94b700 1 --2- 172.21.15.1:0/173490617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb940cfab0 0x7fdb94158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.987+0000 7fdb9b94b700 1 -- 172.21.15.1:0/173490617 >> 172.21.15.1:0/173490617 conn(0x7fdb9406ab70 msgr2=0x7fdb9406e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.987+0000 7fdb9b94b700 1 -- 172.21.15.1:0/173490617 shutdown_connections 2024-03-18T19:10:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:04.987+0000 7fdb9b94b700 1 -- 172.21.15.1:0/173490617 wait complete. 2024-03-18T19:10:04.995 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:10:05.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.124+0000 7fcbabc45700 1 Processor -- start 2024-03-18T19:10:05.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.124+0000 7fcbabc45700 1 -- start start 2024-03-18T19:10:05.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.125+0000 7fcbabc45700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcba415a320 0x7fcba40ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:05.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.125+0000 7fcbabc45700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcba4071d10 0x7fcba406e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:05.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.125+0000 7fcbabc45700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba40acd80 con 0x7fcba40cea30 2024-03-18T19:10:05.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.125+0000 7fcbabc45700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcba40cf220 con 0x7fcba4071d10 2024-03-18T19:10:05.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.125+0000 7fcbabc45700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba40cef40 con 0x7fcba415a320 2024-03-18T19:10:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.126+0000 7fcba99e1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcba415a320 0x7fcba40ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.126+0000 7fcbaa1e2700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcba4071d10 0x7fcba406e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.126+0000 7fcba91e0700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fcba40cea30 0x7fcba4071600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:44866/0 (socket says 172.21.15.1:44866) 2024-03-18T19:10:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.126+0000 7fcba91e0700 1 -- 172.21.15.1:0/3796982288 learned_addr learned my addr 172.21.15.1:0/3796982288 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:10:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.126+0000 7fcbaa1e2700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcba4071d10 0x7fcba406e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:50720/0 (socket says 172.21.15.1:50720) 2024-03-18T19:10:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.126+0000 7fcba89df700 1 -- 172.21.15.1:0/3796982288 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45405454 0 0) 0x7fcba40acd80 con 0x7fcba40cea30 2024-03-18T19:10:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.126+0000 7fcba89df700 1 -- 172.21.15.1:0/3796982288 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb900035b0 con 0x7fcba40cea30 2024-03-18T19:10:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.126+0000 7fcbaa1e2700 1 -- 172.21.15.1:0/3796982288 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcba415a320 msgr2=0x7fcba40ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.126+0000 7fcbaa1e2700 1 --2- 172.21.15.1:0/3796982288 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcba415a320 0x7fcba40ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.126+0000 7fcbaa1e2700 1 -- 172.21.15.1:0/3796982288 >> v1:172.21.15.165:6789/0 conn(0x7fcba40cea30 legacy=0x7fcba4071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.126+0000 7fcbaa1e2700 1 -- 172.21.15.1:0/3796982288 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba406d7e0 con 0x7fcba4071d10 2024-03-18T19:10:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.126+0000 7fcbaa1e2700 1 --2- 172.21.15.1:0/3796982288 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcba4071d10 0x7fcba406e2c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c009d90 tx=0x7fcb9c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7245305fefc2d935 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.126+0000 7fcba89df700 1 -- 172.21.15.1:0/3796982288 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c00e070 con 0x7fcba4071d10 2024-03-18T19:10:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.126+0000 7fcba89df700 1 -- 172.21.15.1:0/3796982288 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9c00bb20 con 0x7fcba4071d10 2024-03-18T19:10:05.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.127+0000 7fcba89df700 1 -- 172.21.15.1:0/3796982288 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c012430 con 0x7fcba4071d10 2024-03-18T19:10:05.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.127+0000 7fcbabc45700 1 -- 172.21.15.1:0/3796982288 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcba4071d10 msgr2=0x7fcba406e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:05.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.127+0000 7fcbabc45700 1 --2- 172.21.15.1:0/3796982288 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcba4071d10 0x7fcba406e2c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c009d90 tx=0x7fcb9c00b330 comp rx=0 tx=0).stop 2024-03-18T19:10:05.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.127+0000 7fcbabc45700 1 -- 172.21.15.1:0/3796982288 shutdown_connections 2024-03-18T19:10:05.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.127+0000 7fcbabc45700 1 --2- 172.21.15.1:0/3796982288 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcba4071d10 0x7fcba406e2c0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:05.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.127+0000 7fcbabc45700 1 --2- 172.21.15.1:0/3796982288 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcba415a320 0x7fcba40ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:05.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.127+0000 7fcbabc45700 1 -- 172.21.15.1:0/3796982288 >> 172.21.15.1:0/3796982288 conn(0x7fcba40cc990 msgr2=0x7fcba40ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:05.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.127+0000 7fcbabc45700 1 -- 172.21.15.1:0/3796982288 shutdown_connections 2024-03-18T19:10:05.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.127+0000 7fcbabc45700 1 -- 172.21.15.1:0/3796982288 wait complete. 2024-03-18T19:10:05.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.128+0000 7fcbabc45700 1 Processor -- start 2024-03-18T19:10:05.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.128+0000 7fcbabc45700 1 -- start start 2024-03-18T19:10:05.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.129+0000 7fcbabc45700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcba4071d10 0x7fcba4153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:05.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.129+0000 7fcbabc45700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcba40cea30 0x7fcba41f3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:05.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.129+0000 7fcbabc45700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcba40c9340 0x7fcba40c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:05.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.129+0000 7fcbabc45700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba40c9d30 con 0x7fcba4071d10 2024-03-18T19:10:05.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.129+0000 7fcbabc45700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba4153ab0 con 0x7fcba40cea30 2024-03-18T19:10:05.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.129+0000 7fcbabc45700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba40acd80 con 0x7fcba40c9340 2024-03-18T19:10:05.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.129+0000 7fcba99e1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcba4071d10 0x7fcba4153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:05.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.129+0000 7fcba99e1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcba4071d10 0x7fcba4153570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50818/0 (socket says 172.21.15.1:50818) 2024-03-18T19:10:05.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.129+0000 7fcba99e1700 1 -- 172.21.15.1:0/1678077872 learned_addr learned my addr 172.21.15.1:0/1678077872 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:05.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.130+0000 7fcbaa1e2700 1 --2- 172.21.15.1:0/1678077872 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcba40c9340 0x7fcba40c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:05.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.130+0000 7fcba91e0700 1 --2- 172.21.15.1:0/1678077872 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcba40cea30 0x7fcba41f3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:05.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.130+0000 7fcba99e1700 1 -- 172.21.15.1:0/1678077872 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcba40c9340 msgr2=0x7fcba40c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:05.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.130+0000 7fcba99e1700 1 --2- 172.21.15.1:0/1678077872 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcba40c9340 0x7fcba40c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:05.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.130+0000 7fcba99e1700 1 -- 172.21.15.1:0/1678077872 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcba40cea30 msgr2=0x7fcba41f3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:05.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.130+0000 7fcba99e1700 1 --2- 172.21.15.1:0/1678077872 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcba40cea30 0x7fcba41f3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:05.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.130+0000 7fcba99e1700 1 -- 172.21.15.1:0/1678077872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c009a70 con 0x7fcba4071d10 2024-03-18T19:10:05.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.130+0000 7fcbaa1e2700 1 --2- 172.21.15.1:0/1678077872 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcba40c9340 0x7fcba40c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:10:05.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.130+0000 7fcba99e1700 1 --2- 172.21.15.1:0/1678077872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcba4071d10 0x7fcba4153570 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7fcba000e9b0 tx=0x7fcba000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:05.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.130+0000 7fcb967fc700 1 -- 172.21.15.1:0/1678077872 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba0009710 con 0x7fcba4071d10 2024-03-18T19:10:05.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.130+0000 7fcb967fc700 1 -- 172.21.15.1:0/1678077872 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba000c850 con 0x7fcba4071d10 2024-03-18T19:10:05.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.130+0000 7fcb967fc700 1 -- 172.21.15.1:0/1678077872 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba000cbc0 con 0x7fcba4071d10 2024-03-18T19:10:05.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.130+0000 7fcbabc45700 1 -- 172.21.15.1:0/1678077872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba41f3ac0 con 0x7fcba4071d10 2024-03-18T19:10:05.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.130+0000 7fcbabc45700 1 -- 172.21.15.1:0/1678077872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba40c4090 con 0x7fcba4071d10 2024-03-18T19:10:05.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.131+0000 7fcbabc45700 1 -- 172.21.15.1:0/1678077872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba40c7b60 con 0x7fcba4071d10 2024-03-18T19:10:05.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.131+0000 7fcb967fc700 1 -- 172.21.15.1:0/1678077872 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcba000cd20 con 0x7fcba4071d10 2024-03-18T19:10:05.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.132+0000 7fcb967fc700 1 --2- 172.21.15.1:0/1678077872 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcb8806adb0 0x7fcb8806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:05.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.132+0000 7fcb967fc700 1 -- 172.21.15.1:0/1678077872 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fcba00895e0 con 0x7fcba4071d10 2024-03-18T19:10:05.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.132+0000 7fcba91e0700 1 --2- 172.21.15.1:0/1678077872 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcb8806adb0 0x7fcb8806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:05.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.132+0000 7fcba91e0700 1 --2- 172.21.15.1:0/1678077872 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcb8806adb0 0x7fcb8806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcba40ca070 tx=0x7fcb98009040 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:05.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.137+0000 7fcb967fc700 1 -- 172.21.15.1:0/1678077872 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcba00544d0 con 0x7fcba4071d10 2024-03-18T19:10:05.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.340+0000 7fcbabc45700 1 -- 172.21.15.1:0/1678077872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcba40c49c0 con 0x7fcba4071d10 2024-03-18T19:10:05.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.341+0000 7fcb967fc700 1 -- 172.21.15.1:0/1678077872 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+16273 (secure 0 0 0) 0x7fcba0057af0 con 0x7fcba4071d10 2024-03-18T19:10:05.337 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:10:05.337 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":160,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:10:04.612904+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:10:02.452367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-18T19:10:03.585045+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:10:05.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.343+0000 7fcbabc45700 1 -- 172.21.15.1:0/1678077872 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcb8806adb0 msgr2=0x7fcb8806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:05.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.343+0000 7fcbabc45700 1 --2- 172.21.15.1:0/1678077872 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcb8806adb0 0x7fcb8806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcba40ca070 tx=0x7fcb98009040 comp rx=0 tx=0).stop 2024-03-18T19:10:05.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.343+0000 7fcbabc45700 1 -- 172.21.15.1:0/1678077872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcba4071d10 msgr2=0x7fcba4153570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:05.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.343+0000 7fcbabc45700 1 --2- 172.21.15.1:0/1678077872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcba4071d10 0x7fcba4153570 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7fcba000e9b0 tx=0x7fcba000ed70 comp rx=0 tx=0).stop 2024-03-18T19:10:05.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.343+0000 7fcbabc45700 1 -- 172.21.15.1:0/1678077872 shutdown_connections 2024-03-18T19:10:05.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.343+0000 7fcbabc45700 1 --2- 172.21.15.1:0/1678077872 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcb8806adb0 0x7fcb8806d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:05.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.343+0000 7fcbabc45700 1 --2- 172.21.15.1:0/1678077872 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcba40c9340 0x7fcba40c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:05.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.343+0000 7fcbabc45700 1 --2- 172.21.15.1:0/1678077872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcba4071d10 0x7fcba4153570 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:05.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.343+0000 7fcbabc45700 1 --2- 172.21.15.1:0/1678077872 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcba40cea30 0x7fcba41f3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:05.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.343+0000 7fcbabc45700 1 -- 172.21.15.1:0/1678077872 >> 172.21.15.1:0/1678077872 conn(0x7fcba40cc990 msgr2=0x7fcba40ccd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:05.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.343+0000 7fcbabc45700 1 -- 172.21.15.1:0/1678077872 shutdown_connections 2024-03-18T19:10:05.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.343+0000 7fcbabc45700 1 -- 172.21.15.1:0/1678077872 wait complete. 2024-03-18T19:10:05.352 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T19:10:05.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.484+0000 7fa946a2b700 1 Processor -- start 2024-03-18T19:10:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.484+0000 7fa946a2b700 1 -- start start 2024-03-18T19:10:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.485+0000 7fa946a2b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9400724b0 0x7fa9400cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.485+0000 7fa946a2b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa94006da90 con 0x7fa94006efb0 2024-03-18T19:10:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.485+0000 7fa946a2b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa94006ab30 con 0x7fa94015d580 2024-03-18T19:10:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.485+0000 7fa946a2b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa94015d960 con 0x7fa9400724b0 2024-03-18T19:10:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.485+0000 7fa93f7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa94006efb0 0x7fa940071da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52602/0 (socket says 172.21.15.1:52602) 2024-03-18T19:10:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.485+0000 7fa93f7fe700 1 -- 172.21.15.1:0/885518521 learned_addr learned my addr 172.21.15.1:0/885518521 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:10:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.485+0000 7fa944fc8700 1 --2- 172.21.15.1:0/885518521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9400724b0 0x7fa9400cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93effd700 1 -- 172.21.15.1:0/885518521 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753397570 0 0) 0x7fa94006da90 con 0x7fa94006efb0 2024-03-18T19:10:05.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93effd700 1 -- 172.21.15.1:0/885518521 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa92c0035b0 con 0x7fa94006efb0 2024-03-18T19:10:05.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93effd700 1 -- 172.21.15.1:0/885518521 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584522666 0 0) 0x7fa94006ab30 con 0x7fa94015d580 2024-03-18T19:10:05.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93effd700 1 -- 172.21.15.1:0/885518521 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa94006da90 con 0x7fa94015d580 2024-03-18T19:10:05.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93effd700 1 -- 172.21.15.1:0/885518521 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1313219883 0 0) 0x7fa92c0035b0 con 0x7fa94006efb0 2024-03-18T19:10:05.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93effd700 1 -- 172.21.15.1:0/885518521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9400724b0 msgr2=0x7fa9400cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:05.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93effd700 1 --2- 172.21.15.1:0/885518521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9400724b0 0x7fa9400cf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:05.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93effd700 1 -- 172.21.15.1:0/885518521 >> v1:172.21.15.165:6789/0 conn(0x7fa94015d580 legacy=0x7fa94006d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:05.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93effd700 1 -- 172.21.15.1:0/885518521 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94006ae10 con 0x7fa94006efb0 2024-03-18T19:10:05.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93effd700 1 -- 172.21.15.1:0/885518521 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fa930001670 con 0x7fa94006efb0 2024-03-18T19:10:05.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa944fc8700 1 --2- 172.21.15.1:0/885518521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9400724b0 0x7fa9400cf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:10:05.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93effd700 1 -- 172.21.15.1:0/885518521 >> v1:172.21.15.1:6789/0 conn(0x7fa94006efb0 legacy=0x7fa940071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:05.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93effd700 1 --2- 172.21.15.1:0/885518521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9400724b0 0x7fa92c003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:05.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93effd700 1 --2- 172.21.15.1:0/885518521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c004100 0x7fa92c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:05.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93effd700 1 --2- 172.21.15.1:0/885518521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa92c004d90 0x7fa92c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93effd700 1 -- 172.21.15.1:0/885518521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94015d960 con 0x7fa92c004100 2024-03-18T19:10:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93effd700 1 -- 172.21.15.1:0/885518521 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92c00bac0 con 0x7fa92c004d90 2024-03-18T19:10:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93effd700 1 -- 172.21.15.1:0/885518521 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa92c00bc80 con 0x7fa9400724b0 2024-03-18T19:10:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93f7fe700 1 --2- 172.21.15.1:0/885518521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c004100 0x7fa92c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93ffff700 1 --2- 172.21.15.1:0/885518521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9400724b0 0x7fa92c003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa944fc8700 1 --2- 172.21.15.1:0/885518521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa92c004d90 0x7fa92c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93f7fe700 1 -- 172.21.15.1:0/885518521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9400724b0 msgr2=0x7fa92c003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93f7fe700 1 --2- 172.21.15.1:0/885518521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa9400724b0 0x7fa92c003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93f7fe700 1 -- 172.21.15.1:0/885518521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa92c004d90 msgr2=0x7fa92c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93f7fe700 1 --2- 172.21.15.1:0/885518521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa92c004d90 0x7fa92c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.486+0000 7fa93f7fe700 1 -- 172.21.15.1:0/885518521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa938009a70 con 0x7fa92c004100 2024-03-18T19:10:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.487+0000 7fa93f7fe700 1 --2- 172.21.15.1:0/885518521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c004100 0x7fa92c004850 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fa93000c210 tx=0x7fa92c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad90751717c87e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.487+0000 7fa93effd700 1 -- 172.21.15.1:0/885518521 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93000cd50 con 0x7fa92c004100 2024-03-18T19:10:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.487+0000 7fa93effd700 1 -- 172.21.15.1:0/885518521 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa930005960 con 0x7fa92c004100 2024-03-18T19:10:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.487+0000 7fa93effd700 1 -- 172.21.15.1:0/885518521 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa930005d10 con 0x7fa92c004100 2024-03-18T19:10:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.487+0000 7fa946a2b700 1 -- 172.21.15.1:0/885518521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c004100 msgr2=0x7fa92c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.487+0000 7fa946a2b700 1 --2- 172.21.15.1:0/885518521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa92c004100 0x7fa92c004850 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fa93000c210 tx=0x7fa92c005880 comp rx=0 tx=0).stop 2024-03-18T19:10:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.487+0000 7fa944fc8700 1 -- 172.21.15.1:0/885518521 reap_dead start 2024-03-18T19:10:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.487+0000 7fa946a2b700 1 -- 172.21.15.1:0/885518521 shutdown_connections 2024-03-18T19:10:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.487+0000 7fa946a2b700 1 -- 172.21.15.1:0/885518521 >> 172.21.15.1:0/885518521 conn(0x7fa9400cca00 msgr2=0x7fa9400cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.487+0000 7fa946a2b700 1 -- 172.21.15.1:0/885518521 shutdown_connections 2024-03-18T19:10:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.487+0000 7fa946a2b700 1 -- 172.21.15.1:0/885518521 wait complete. 2024-03-18T19:10:05.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.489+0000 7fa946a2b700 1 Processor -- start 2024-03-18T19:10:05.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.489+0000 7fa946a2b700 1 -- start start 2024-03-18T19:10:05.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.490+0000 7fa946a2b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa92c004100 0x7fa940157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:05.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.490+0000 7fa946a2b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa94015daa0 0x7fa94015df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:05.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.490+0000 7fa946a2b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa940159b10 0x7fa940159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:05.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.490+0000 7fa946a2b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94015a4d0 con 0x7fa940159b10 2024-03-18T19:10:05.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.490+0000 7fa946a2b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa940157ad0 con 0x7fa92c004100 2024-03-18T19:10:05.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.490+0000 7fa946a2b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9401601c0 con 0x7fa94015daa0 2024-03-18T19:10:05.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.490+0000 7fa944fc8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa940159b10 0x7fa940159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:05.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.490+0000 7fa944fc8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa940159b10 0x7fa940159f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50838/0 (socket says 172.21.15.1:50838) 2024-03-18T19:10:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.490+0000 7fa944fc8700 1 -- 172.21.15.1:0/3847001750 learned_addr learned my addr 172.21.15.1:0/3847001750 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.490+0000 7fa93f7fe700 1 --2- 172.21.15.1:0/3847001750 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa94015daa0 0x7fa94015df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.490+0000 7fa93ffff700 1 --2- 172.21.15.1:0/3847001750 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa92c004100 0x7fa940157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.490+0000 7fa944fc8700 1 -- 172.21.15.1:0/3847001750 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa94015daa0 msgr2=0x7fa94015df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.490+0000 7fa944fc8700 1 --2- 172.21.15.1:0/3847001750 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa94015daa0 0x7fa94015df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.490+0000 7fa944fc8700 1 -- 172.21.15.1:0/3847001750 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa92c004100 msgr2=0x7fa940157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.490+0000 7fa944fc8700 1 --2- 172.21.15.1:0/3847001750 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa92c004100 0x7fa940157590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.490+0000 7fa944fc8700 1 -- 172.21.15.1:0/3847001750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94015e610 con 0x7fa940159b10 2024-03-18T19:10:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.490+0000 7fa93ffff700 1 --2- 172.21.15.1:0/3847001750 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa92c004100 0x7fa940157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:10:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.491+0000 7fa944fc8700 1 --2- 172.21.15.1:0/3847001750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa940159b10 0x7fa940159f90 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fa94006f690 tx=0x7fa93800f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.491+0000 7fa93cff9700 1 -- 172.21.15.1:0/3847001750 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93800e040 con 0x7fa940159b10 2024-03-18T19:10:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.491+0000 7fa93cff9700 1 -- 172.21.15.1:0/3847001750 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93800f040 con 0x7fa940159b10 2024-03-18T19:10:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.491+0000 7fa946a2b700 1 -- 172.21.15.1:0/3847001750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa94015e8f0 con 0x7fa940159b10 2024-03-18T19:10:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.491+0000 7fa93cff9700 1 -- 172.21.15.1:0/3847001750 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa938003a20 con 0x7fa940159b10 2024-03-18T19:10:05.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.491+0000 7fa946a2b700 1 -- 172.21.15.1:0/3847001750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa94015eeb0 con 0x7fa940159b10 2024-03-18T19:10:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.492+0000 7fa946a2b700 1 -- 172.21.15.1:0/3847001750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9400cd740 con 0x7fa940159b10 2024-03-18T19:10:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.492+0000 7fa93cff9700 1 -- 172.21.15.1:0/3847001750 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa938003b80 con 0x7fa940159b10 2024-03-18T19:10:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.492+0000 7fa93cff9700 1 --2- 172.21.15.1:0/3847001750 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa92006adb0 0x7fa92006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.493+0000 7fa93cff9700 1 -- 172.21.15.1:0/3847001750 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa9380894c0 con 0x7fa940159b10 2024-03-18T19:10:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.493+0000 7fa93ffff700 1 --2- 172.21.15.1:0/3847001750 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa92006adb0 0x7fa92006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:05.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.493+0000 7fa93ffff700 1 --2- 172.21.15.1:0/3847001750 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa92006adb0 0x7fa92006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa928007fd0 tx=0x7fa928009040 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:05.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.497+0000 7fa93cff9700 1 -- 172.21.15.1:0/3847001750 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa938054210 con 0x7fa940159b10 2024-03-18T19:10:05.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:05.748+0000 7fa946a2b700 1 -- 172.21.15.1:0/3847001750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa9400ce320 con 0x7fa940159b10 2024-03-18T19:10:06.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.626+0000 7fa93cff9700 1 -- 172.21.15.1:0/3847001750 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v72) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa938057830 con 0x7fa940159b10 2024-03-18T19:10:06.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.629+0000 7fa946a2b700 1 -- 172.21.15.1:0/3847001750 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa92006adb0 msgr2=0x7fa92006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:06.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.629+0000 7fa946a2b700 1 --2- 172.21.15.1:0/3847001750 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa92006adb0 0x7fa92006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa928007fd0 tx=0x7fa928009040 comp rx=0 tx=0).stop 2024-03-18T19:10:06.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.629+0000 7fa946a2b700 1 -- 172.21.15.1:0/3847001750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa940159b10 msgr2=0x7fa940159f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:06.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.629+0000 7fa946a2b700 1 --2- 172.21.15.1:0/3847001750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa940159b10 0x7fa940159f90 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fa94006f690 tx=0x7fa93800f490 comp rx=0 tx=0).stop 2024-03-18T19:10:06.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.630+0000 7fa946a2b700 1 -- 172.21.15.1:0/3847001750 shutdown_connections 2024-03-18T19:10:06.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.630+0000 7fa946a2b700 1 --2- 172.21.15.1:0/3847001750 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa92006adb0 0x7fa92006d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:06.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.630+0000 7fa946a2b700 1 --2- 172.21.15.1:0/3847001750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa940159b10 0x7fa940159f90 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:06.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.630+0000 7fa946a2b700 1 --2- 172.21.15.1:0/3847001750 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa92c004100 0x7fa940157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:06.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.630+0000 7fa946a2b700 1 --2- 172.21.15.1:0/3847001750 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa94015daa0 0x7fa94015df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:06.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.630+0000 7fa946a2b700 1 -- 172.21.15.1:0/3847001750 >> 172.21.15.1:0/3847001750 conn(0x7fa9400cca00 msgr2=0x7fa9400b03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:06.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.630+0000 7fa946a2b700 1 -- 172.21.15.1:0/3847001750 shutdown_connections 2024-03-18T19:10:06.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.630+0000 7fa946a2b700 1 -- 172.21.15.1:0/3847001750 wait complete. 2024-03-18T19:10:06.640 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-18T19:10:06.641 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T19:10:06.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.780+0000 7f8f2125a700 1 Processor -- start 2024-03-18T19:10:06.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.780+0000 7f8f2125a700 1 -- start start 2024-03-18T19:10:06.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.781+0000 7f8f2125a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f1c06efb0 0x7f8f1c071da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:06.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.781+0000 7f8f2125a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c06da90 con 0x7f8f1c15d580 2024-03-18T19:10:06.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.781+0000 7f8f2125a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c06ab30 con 0x7f8f1c0722e0 2024-03-18T19:10:06.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.781+0000 7f8f2125a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c072770 con 0x7f8f1c06efb0 2024-03-18T19:10:06.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.781+0000 7f8f1ad9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8f1c15d580 0x7f8f1c06d380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52608/0 (socket says 172.21.15.1:52608) 2024-03-18T19:10:06.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.781+0000 7f8f1ad9d700 1 -- 172.21.15.1:0/757838270 learned_addr learned my addr 172.21.15.1:0/757838270 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:10:06.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.781+0000 7f8f19d9b700 1 -- 172.21.15.1:0/757838270 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843607911 0 0) 0x7f8f1c06da90 con 0x7f8f1c15d580 2024-03-18T19:10:06.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.781+0000 7f8f19d9b700 1 -- 172.21.15.1:0/757838270 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f080035b0 con 0x7f8f1c15d580 2024-03-18T19:10:06.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.781+0000 7f8f1a59c700 1 --2- 172.21.15.1:0/757838270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f1c06efb0 0x7f8f1c071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:06.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f19d9b700 1 -- 172.21.15.1:0/757838270 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1560805793 0 0) 0x7f8f080035b0 con 0x7f8f1c15d580 2024-03-18T19:10:06.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f19d9b700 1 -- 172.21.15.1:0/757838270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f1c06efb0 msgr2=0x7f8f1c071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:06.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f19d9b700 1 --2- 172.21.15.1:0/757838270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f1c06efb0 0x7f8f1c071da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:06.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f19d9b700 1 -- 172.21.15.1:0/757838270 >> v1:172.21.15.165:6789/0 conn(0x7f8f1c0722e0 legacy=0x7f8f1c0cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:06.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f19d9b700 1 -- 172.21.15.1:0/757838270 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c06ae10 con 0x7f8f1c15d580 2024-03-18T19:10:06.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f19d9b700 1 -- 172.21.15.1:0/757838270 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f8f140019f0 con 0x7f8f1c15d580 2024-03-18T19:10:06.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f19d9b700 1 -- 172.21.15.1:0/757838270 >> v1:172.21.15.1:6789/0 conn(0x7f8f1c15d580 legacy=0x7f8f1c06d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:06.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f19d9b700 1 --2- 172.21.15.1:0/757838270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f08003a30 0x7f8f08003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:06.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f19d9b700 1 --2- 172.21.15.1:0/757838270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f08004850 0x7f8f08004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:06.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f19d9b700 1 --2- 172.21.15.1:0/757838270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f1c06efb0 0x7f8f08005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:06.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f19d9b700 1 -- 172.21.15.1:0/757838270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0800bac0 con 0x7f8f08003a30 2024-03-18T19:10:06.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f19d9b700 1 -- 172.21.15.1:0/757838270 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0800bc30 con 0x7f8f08004850 2024-03-18T19:10:06.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f19d9b700 1 -- 172.21.15.1:0/757838270 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0800bdf0 con 0x7f8f1c06efb0 2024-03-18T19:10:06.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f1ad9d700 1 --2- 172.21.15.1:0/757838270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f08003a30 0x7f8f08003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:06.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f1a59c700 1 --2- 172.21.15.1:0/757838270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f08004850 0x7f8f08004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:06.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f1b59e700 1 --2- 172.21.15.1:0/757838270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f1c06efb0 0x7f8f08005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:06.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f1ad9d700 1 -- 172.21.15.1:0/757838270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f1c06efb0 msgr2=0x7f8f08005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:06.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f1ad9d700 1 --2- 172.21.15.1:0/757838270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f1c06efb0 0x7f8f08005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:06.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f1ad9d700 1 -- 172.21.15.1:0/757838270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f08004850 msgr2=0x7f8f08004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:06.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f1ad9d700 1 --2- 172.21.15.1:0/757838270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f08004850 0x7f8f08004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:06.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f1ad9d700 1 -- 172.21.15.1:0/757838270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f080035b0 con 0x7f8f08003a30 2024-03-18T19:10:06.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f1a59c700 1 --2- 172.21.15.1:0/757838270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f08004850 0x7f8f08004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:10:06.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f1ad9d700 1 --2- 172.21.15.1:0/757838270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f08003a30 0x7f8f08003e50 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f8f14002230 tx=0x7f8f08005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1333fc8aa74234e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:06.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f19d9b700 1 -- 172.21.15.1:0/757838270 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1400a250 con 0x7f8f08003a30 2024-03-18T19:10:06.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.782+0000 7f8f19d9b700 1 -- 172.21.15.1:0/757838270 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f140072d0 con 0x7f8f08003a30 2024-03-18T19:10:06.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.783+0000 7f8f19d9b700 1 -- 172.21.15.1:0/757838270 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f14007680 con 0x7f8f08003a30 2024-03-18T19:10:06.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.783+0000 7f8f2125a700 1 -- 172.21.15.1:0/757838270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f08003a30 msgr2=0x7f8f08003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:06.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.783+0000 7f8f2125a700 1 --2- 172.21.15.1:0/757838270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f08003a30 0x7f8f08003e50 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f8f14002230 tx=0x7f8f08005880 comp rx=0 tx=0).stop 2024-03-18T19:10:06.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.783+0000 7f8f1b59e700 1 -- 172.21.15.1:0/757838270 reap_dead start 2024-03-18T19:10:06.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.783+0000 7f8f2125a700 1 -- 172.21.15.1:0/757838270 shutdown_connections 2024-03-18T19:10:06.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.783+0000 7f8f2125a700 1 -- 172.21.15.1:0/757838270 >> 172.21.15.1:0/757838270 conn(0x7f8f1c0cca00 msgr2=0x7f8f1c0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:06.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.783+0000 7f8f2125a700 1 -- 172.21.15.1:0/757838270 shutdown_connections 2024-03-18T19:10:06.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.783+0000 7f8f2125a700 1 -- 172.21.15.1:0/757838270 wait complete. 2024-03-18T19:10:06.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.785+0000 7f8f2125a700 1 Processor -- start 2024-03-18T19:10:06.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.785+0000 7f8f2125a700 1 -- start start 2024-03-18T19:10:06.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.785+0000 7f8f2125a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f08003a30 0x7f8f1c15b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:06.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.785+0000 7f8f2125a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f1c15c030 0x7f8f1c1f34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:06.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.785+0000 7f8f2125a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f1c1603a0 0x7f8f1c0adf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:06.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.785+0000 7f8f2125a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c1f3a00 con 0x7f8f08003a30 2024-03-18T19:10:06.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.785+0000 7f8f2125a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c1e5d20 con 0x7f8f1c15c030 2024-03-18T19:10:06.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.785+0000 7f8f2125a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c1e6000 con 0x7f8f1c1603a0 2024-03-18T19:10:06.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.785+0000 7f8f1ad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f08003a30 0x7f8f1c15b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:06.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.786+0000 7f8f1ad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f08003a30 0x7f8f1c15b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50850/0 (socket says 172.21.15.1:50850) 2024-03-18T19:10:06.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.786+0000 7f8f1ad9d700 1 -- 172.21.15.1:0/1806026783 learned_addr learned my addr 172.21.15.1:0/1806026783 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:06.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.786+0000 7f8f1ad9d700 1 -- 172.21.15.1:0/1806026783 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f1c1603a0 msgr2=0x7f8f1c0adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:06.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.786+0000 7f8f1ad9d700 1 --2- 172.21.15.1:0/1806026783 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f1c1603a0 0x7f8f1c0adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:06.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.786+0000 7f8f1ad9d700 1 -- 172.21.15.1:0/1806026783 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f1c15c030 msgr2=0x7f8f1c1f34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:06.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.786+0000 7f8f1ad9d700 1 --2- 172.21.15.1:0/1806026783 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f1c15c030 0x7f8f1c1f34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:06.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.786+0000 7f8f1ad9d700 1 -- 172.21.15.1:0/1806026783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c1e62e0 con 0x7f8f08003a30 2024-03-18T19:10:06.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.786+0000 7f8f1ad9d700 1 --2- 172.21.15.1:0/1806026783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f08003a30 0x7f8f1c15b900 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f8f1400ad20 tx=0x7f8f14001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:06.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.786+0000 7f8f137fe700 1 -- 172.21.15.1:0/1806026783 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1400a250 con 0x7f8f08003a30 2024-03-18T19:10:06.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.786+0000 7f8f137fe700 1 -- 172.21.15.1:0/1806026783 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f14001250 con 0x7f8f08003a30 2024-03-18T19:10:06.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.786+0000 7f8f2125a700 1 -- 172.21.15.1:0/1806026783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1c1e65c0 con 0x7f8f08003a30 2024-03-18T19:10:06.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.786+0000 7f8f137fe700 1 -- 172.21.15.1:0/1806026783 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f14012610 con 0x7f8f08003a30 2024-03-18T19:10:06.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.786+0000 7f8f2125a700 1 -- 172.21.15.1:0/1806026783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1c160820 con 0x7f8f08003a30 2024-03-18T19:10:06.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.787+0000 7f8f2125a700 1 -- 172.21.15.1:0/1806026783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f1c1e6c10 con 0x7f8f08003a30 2024-03-18T19:10:06.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.788+0000 7f8f137fe700 1 -- 172.21.15.1:0/1806026783 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8f1401a070 con 0x7f8f08003a30 2024-03-18T19:10:06.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.788+0000 7f8f137fe700 1 --2- 172.21.15.1:0/1806026783 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8efc06adb0 0x7f8efc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:06.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.788+0000 7f8f137fe700 1 -- 172.21.15.1:0/1806026783 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f8f1408b680 con 0x7f8f08003a30 2024-03-18T19:10:06.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.788+0000 7f8f1a59c700 1 --2- 172.21.15.1:0/1806026783 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8efc06adb0 0x7f8efc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:06.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.788+0000 7f8f1a59c700 1 --2- 172.21.15.1:0/1806026783 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8efc06adb0 0x7f8efc06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c0aef10 tx=0x7f8f04008040 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:06.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:06.792+0000 7f8f137fe700 1 -- 172.21.15.1:0/1806026783 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8f140567a0 con 0x7f8f08003a30 2024-03-18T19:10:07.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.041+0000 7f8f2125a700 1 -- 172.21.15.1:0/1806026783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8f1c160a70 con 0x7f8f08003a30 2024-03-18T19:10:07.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.630+0000 7f8f137fe700 1 -- 172.21.15.1:0/1806026783 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v73) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8f14059dc0 con 0x7f8f08003a30 2024-03-18T19:10:07.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.633+0000 7f8f2125a700 1 -- 172.21.15.1:0/1806026783 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8efc06adb0 msgr2=0x7f8efc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:07.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.633+0000 7f8f2125a700 1 --2- 172.21.15.1:0/1806026783 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8efc06adb0 0x7f8efc06d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c0aef10 tx=0x7f8f04008040 comp rx=0 tx=0).stop 2024-03-18T19:10:07.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.633+0000 7f8f2125a700 1 -- 172.21.15.1:0/1806026783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f08003a30 msgr2=0x7f8f1c15b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:07.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.633+0000 7f8f2125a700 1 --2- 172.21.15.1:0/1806026783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f08003a30 0x7f8f1c15b900 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f8f1400ad20 tx=0x7f8f14001cb0 comp rx=0 tx=0).stop 2024-03-18T19:10:07.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.633+0000 7f8f2125a700 1 -- 172.21.15.1:0/1806026783 shutdown_connections 2024-03-18T19:10:07.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.633+0000 7f8f2125a700 1 --2- 172.21.15.1:0/1806026783 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8efc06adb0 0x7f8efc06d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:07.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.633+0000 7f8f2125a700 1 --2- 172.21.15.1:0/1806026783 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8f1c1603a0 0x7f8f1c0adf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:07.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.633+0000 7f8f2125a700 1 --2- 172.21.15.1:0/1806026783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f08003a30 0x7f8f1c15b900 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:07.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.633+0000 7f8f2125a700 1 --2- 172.21.15.1:0/1806026783 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f1c15c030 0x7f8f1c1f34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:07.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.633+0000 7f8f2125a700 1 -- 172.21.15.1:0/1806026783 >> 172.21.15.1:0/1806026783 conn(0x7f8f1c0cca00 msgr2=0x7f8f1c0cf280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:07.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.634+0000 7f8f2125a700 1 -- 172.21.15.1:0/1806026783 shutdown_connections 2024-03-18T19:10:07.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.634+0000 7f8f2125a700 1 -- 172.21.15.1:0/1806026783 wait complete. 2024-03-18T19:10:07.645 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T19:10:07.645 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:10:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.781+0000 7f8182920700 1 Processor -- start 2024-03-18T19:10:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.781+0000 7f8182920700 1 -- start start 2024-03-18T19:10:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.782+0000 7f8182920700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f817c15b500 0x7f817c06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.782+0000 7f8182920700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f817c072720 con 0x7f817c06ac90 2024-03-18T19:10:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.782+0000 7f8182920700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f817c06ffc0 con 0x7f817c15b120 2024-03-18T19:10:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.782+0000 7f8182920700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f817c06b100 con 0x7f817c15b500 2024-03-18T19:10:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.782+0000 7f817b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f817c15b500 0x7f817c06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.782+0000 7f8180ebd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f817c06ac90 0x7f817c0ca680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52616/0 (socket says 172.21.15.1:52616) 2024-03-18T19:10:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.782+0000 7f8180ebd700 1 -- 172.21.15.1:0/692784540 learned_addr learned my addr 172.21.15.1:0/692784540 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:10:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.782+0000 7f817b7fe700 1 -- 172.21.15.1:0/692784540 >> v1:172.21.15.165:6789/0 conn(0x7f817c15b120 legacy=0x7f817c072010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.782+0000 7f817b7fe700 1 -- 172.21.15.1:0/692784540 >> v1:172.21.15.1:6789/0 conn(0x7f817c06ac90 legacy=0x7f817c0ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.782+0000 7f817affd700 1 -- 172.21.15.1:0/692784540 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941457249 0 0) 0x7f8174001490 con 0x7f817c06ac90 2024-03-18T19:10:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.782+0000 7f817b7fe700 1 -- 172.21.15.1:0/692784540 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c06b520 con 0x7f817c15b500 2024-03-18T19:10:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.783+0000 7f817b7fe700 1 --2- 172.21.15.1:0/692784540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c15b500 0x7f817c06a750 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f8164009840 tx=0x7f8164009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3999cb1a5964038b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.783+0000 7f817affd700 1 -- 172.21.15.1:0/692784540 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8164011830 con 0x7f817c15b500 2024-03-18T19:10:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.783+0000 7f817affd700 1 -- 172.21.15.1:0/692784540 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8164011990 con 0x7f817c15b500 2024-03-18T19:10:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.783+0000 7f817affd700 1 -- 172.21.15.1:0/692784540 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8164011d20 con 0x7f817c15b500 2024-03-18T19:10:07.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.783+0000 7f8182920700 1 -- 172.21.15.1:0/692784540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c15b500 msgr2=0x7f817c06a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:07.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.783+0000 7f8182920700 1 --2- 172.21.15.1:0/692784540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c15b500 0x7f817c06a750 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f8164009840 tx=0x7f8164009b50 comp rx=0 tx=0).stop 2024-03-18T19:10:07.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.783+0000 7f8182920700 1 -- 172.21.15.1:0/692784540 shutdown_connections 2024-03-18T19:10:07.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.783+0000 7f8182920700 1 --2- 172.21.15.1:0/692784540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c15b500 0x7f817c06a750 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:07.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.783+0000 7f8182920700 1 -- 172.21.15.1:0/692784540 >> 172.21.15.1:0/692784540 conn(0x7f817c0ccab0 msgr2=0x7f817c0ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:07.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.783+0000 7f8182920700 1 -- 172.21.15.1:0/692784540 shutdown_connections 2024-03-18T19:10:07.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.783+0000 7f8182920700 1 -- 172.21.15.1:0/692784540 wait complete. 2024-03-18T19:10:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.784+0000 7f8182920700 1 Processor -- start 2024-03-18T19:10:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.785+0000 7f8182920700 1 -- start start 2024-03-18T19:10:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.785+0000 7f8182920700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c06ac90 0x7f817c1f0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.785+0000 7f8182920700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f817c15b120 0x7f817c0c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.785+0000 7f8182920700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f817c15b500 0x7f817c0c5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.785+0000 7f8182920700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c0b5340 con 0x7f817c06ac90 2024-03-18T19:10:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.785+0000 7f8182920700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c0b4730 con 0x7f817c15b120 2024-03-18T19:10:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.785+0000 7f8182920700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f817c0b4a10 con 0x7f817c15b500 2024-03-18T19:10:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.786+0000 7f817bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c06ac90 0x7f817c1f0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.786+0000 7f817bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c06ac90 0x7f817c1f0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50870/0 (socket says 172.21.15.1:50870) 2024-03-18T19:10:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.786+0000 7f817bfff700 1 -- 172.21.15.1:0/53270389 learned_addr learned my addr 172.21.15.1:0/53270389 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.786+0000 7f817bfff700 1 -- 172.21.15.1:0/53270389 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f817c15b500 msgr2=0x7f817c0c5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.786+0000 7f817bfff700 1 --2- 172.21.15.1:0/53270389 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f817c15b500 0x7f817c0c5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.786+0000 7f817bfff700 1 -- 172.21.15.1:0/53270389 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f817c15b120 msgr2=0x7f817c0c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.786+0000 7f817b7fe700 1 --2- 172.21.15.1:0/53270389 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f817c15b120 0x7f817c0c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:07.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.786+0000 7f817bfff700 1 --2- 172.21.15.1:0/53270389 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f817c15b120 0x7f817c0c57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:07.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.786+0000 7f817bfff700 1 -- 172.21.15.1:0/53270389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81640094a0 con 0x7f817c06ac90 2024-03-18T19:10:07.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.786+0000 7f817bfff700 1 --2- 172.21.15.1:0/53270389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c06ac90 0x7f817c1f0de0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f816c00c4e0 tx=0x7f816c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:07.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.786+0000 7f8178ff9700 1 -- 172.21.15.1:0/53270389 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816c0052d0 con 0x7f817c06ac90 2024-03-18T19:10:07.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.786+0000 7f8178ff9700 1 -- 172.21.15.1:0/53270389 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816c005430 con 0x7f817c06ac90 2024-03-18T19:10:07.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.786+0000 7f8178ff9700 1 -- 172.21.15.1:0/53270389 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816c0038c0 con 0x7f817c06ac90 2024-03-18T19:10:07.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.786+0000 7f8182920700 1 -- 172.21.15.1:0/53270389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f817c0b4cf0 con 0x7f817c06ac90 2024-03-18T19:10:07.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.786+0000 7f8182920700 1 -- 172.21.15.1:0/53270389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f817c0c6360 con 0x7f817c06ac90 2024-03-18T19:10:07.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.787+0000 7f8182920700 1 -- 172.21.15.1:0/53270389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f817c06dec0 con 0x7f817c06ac90 2024-03-18T19:10:07.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.788+0000 7f8178ff9700 1 -- 172.21.15.1:0/53270389 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f816c003a20 con 0x7f817c06ac90 2024-03-18T19:10:07.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.788+0000 7f8178ff9700 1 --2- 172.21.15.1:0/53270389 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f815c06adb0 0x7f815c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:07.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.788+0000 7f8178ff9700 1 -- 172.21.15.1:0/53270389 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f816c088a30 con 0x7f817c06ac90 2024-03-18T19:10:07.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.788+0000 7f817b7fe700 1 --2- 172.21.15.1:0/53270389 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f815c06adb0 0x7f815c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:07.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.789+0000 7f817b7fe700 1 --2- 172.21.15.1:0/53270389 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f815c06adb0 0x7f815c06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8164009840 tx=0x7f816401c040 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:07.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:07.793+0000 7f8178ff9700 1 -- 172.21.15.1:0/53270389 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f816c053a30 con 0x7f817c06ac90 2024-03-18T19:10:08.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.041+0000 7f8182920700 1 -- 172.21.15.1:0/53270389 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f817c0669d0 con 0x7f817c06ac90 2024-03-18T19:10:08.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.041+0000 7f8178ff9700 1 -- 172.21.15.1:0/53270389 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f816c017090 con 0x7f817c06ac90 2024-03-18T19:10:08.038 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:10:08.039 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8608,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3436643592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3436643592},{"type":"v1","addr":"172.21.15.165:6834","nonce":3436643592}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":8611,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1969816196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1969816196},{"type":"v1","addr":"172.21.15.165:6835","nonce":1969816196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":9375,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2671786416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2671786416},{"type":"v1","addr":"172.21.15.1:6836","nonce":2671786416}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:10:04.613212+0000","modified":"2024-03-18T19:10:07.625525+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9378":{"gid":9378,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1696108841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1696108841},{"type":"v1","addr":"172.21.15.1:6837","nonce":1696108841}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-03-18T19:10:08.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.044+0000 7f8182920700 1 -- 172.21.15.1:0/53270389 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f815c06adb0 msgr2=0x7f815c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:08.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.044+0000 7f8182920700 1 --2- 172.21.15.1:0/53270389 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f815c06adb0 0x7f815c06d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8164009840 tx=0x7f816401c040 comp rx=0 tx=0).stop 2024-03-18T19:10:08.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.044+0000 7f8182920700 1 -- 172.21.15.1:0/53270389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c06ac90 msgr2=0x7f817c1f0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:08.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.044+0000 7f8182920700 1 --2- 172.21.15.1:0/53270389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c06ac90 0x7f817c1f0de0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f816c00c4e0 tx=0x7f816c00c8a0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.044+0000 7f8182920700 1 -- 172.21.15.1:0/53270389 shutdown_connections 2024-03-18T19:10:08.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.044+0000 7f8182920700 1 --2- 172.21.15.1:0/53270389 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f815c06adb0 0x7f815c06d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.044+0000 7f8182920700 1 --2- 172.21.15.1:0/53270389 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f817c15b500 0x7f817c0c5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.044+0000 7f8182920700 1 --2- 172.21.15.1:0/53270389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f817c06ac90 0x7f817c1f0de0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.044+0000 7f8182920700 1 --2- 172.21.15.1:0/53270389 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f817c15b120 0x7f817c0c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.044+0000 7f8182920700 1 -- 172.21.15.1:0/53270389 >> 172.21.15.1:0/53270389 conn(0x7f817c0ccab0 msgr2=0x7f817c157ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:08.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.044+0000 7f8182920700 1 -- 172.21.15.1:0/53270389 shutdown_connections 2024-03-18T19:10:08.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.044+0000 7f8182920700 1 -- 172.21.15.1:0/53270389 wait complete. 2024-03-18T19:10:08.042 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 73 2024-03-18T19:10:08.053 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T19:10:08.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.181+0000 7fc821d83700 1 Processor -- start 2024-03-18T19:10:08.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.181+0000 7fc821d83700 1 -- start start 2024-03-18T19:10:08.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.182+0000 7fc821d83700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc81c144b70 0x7fc81c0ce980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.182+0000 7fc821d83700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc81c0cd110 0x7fc81c071a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.182+0000 7fc821d83700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81c0cf2d0 con 0x7fc81c0af610 2024-03-18T19:10:08.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.182+0000 7fc821d83700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc81c144f50 con 0x7fc81c144b70 2024-03-18T19:10:08.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.182+0000 7fc821d83700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc81c0cf5b0 con 0x7fc81c0cd110 2024-03-18T19:10:08.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.182+0000 7fc81b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc81c144b70 0x7fc81c0ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.182+0000 7fc81b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc81c144b70 0x7fc81c0ce980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50880/0 (socket says 172.21.15.1:50880) 2024-03-18T19:10:08.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.182+0000 7fc81b7fe700 1 -- 172.21.15.1:0/2729655213 learned_addr learned my addr 172.21.15.1:0/2729655213 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:08.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.182+0000 7fc81bfff700 1 --2- 172.21.15.1:0/2729655213 >> v2:172.21.15.165:3300/0 conn(0x7fc81c0cd110 0x7fc81c071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.182+0000 7fc81b7fe700 1 -- 172.21.15.1:0/2729655213 >> v2:172.21.15.165:3300/0 conn(0x7fc81c0cd110 msgr2=0x7fc81c071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:08.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.182+0000 7fc81b7fe700 1 --2- 172.21.15.1:0/2729655213 >> v2:172.21.15.165:3300/0 conn(0x7fc81c0cd110 0x7fc81c071a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.182+0000 7fc81b7fe700 1 -- 172.21.15.1:0/2729655213 >> v1:172.21.15.165:6789/0 conn(0x7fc81c0af610 legacy=0x7fc81c0cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:08.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.182+0000 7fc81b7fe700 1 -- 172.21.15.1:0/2729655213 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c0cf890 con 0x7fc81c144b70 2024-03-18T19:10:08.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.182+0000 7fc81b7fe700 1 --2- 172.21.15.1:0/2729655213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c144b70 0x7fc81c0ce980 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fc81c0715e0 tx=0x7fc80c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=936f82fa2e442d0a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:08.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.183+0000 7fc81a7fc700 1 -- 172.21.15.1:0/2729655213 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80c011830 con 0x7fc81c144b70 2024-03-18T19:10:08.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.183+0000 7fc81a7fc700 1 -- 172.21.15.1:0/2729655213 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc80c011990 con 0x7fc81c144b70 2024-03-18T19:10:08.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.183+0000 7fc81a7fc700 1 -- 172.21.15.1:0/2729655213 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80c011d20 con 0x7fc81c144b70 2024-03-18T19:10:08.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.183+0000 7fc821d83700 1 -- 172.21.15.1:0/2729655213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c144b70 msgr2=0x7fc81c0ce980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:08.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.183+0000 7fc821d83700 1 --2- 172.21.15.1:0/2729655213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c144b70 0x7fc81c0ce980 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fc81c0715e0 tx=0x7fc80c009bd0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.183+0000 7fc821d83700 1 -- 172.21.15.1:0/2729655213 shutdown_connections 2024-03-18T19:10:08.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.183+0000 7fc821d83700 1 --2- 172.21.15.1:0/2729655213 >> v2:172.21.15.165:3300/0 conn(0x7fc81c0cd110 0x7fc81c071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.183+0000 7fc821d83700 1 --2- 172.21.15.1:0/2729655213 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c144b70 0x7fc81c0ce980 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.183+0000 7fc821d83700 1 -- 172.21.15.1:0/2729655213 >> 172.21.15.1:0/2729655213 conn(0x7fc81c0b1d20 msgr2=0x7fc81c0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:08.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.183+0000 7fc821d83700 1 -- 172.21.15.1:0/2729655213 shutdown_connections 2024-03-18T19:10:08.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.183+0000 7fc821d83700 1 -- 172.21.15.1:0/2729655213 wait complete. 2024-03-18T19:10:08.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.184+0000 7fc821d83700 1 Processor -- start 2024-03-18T19:10:08.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.185+0000 7fc821d83700 1 -- start start 2024-03-18T19:10:08.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.185+0000 7fc821d83700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc81c0af610 0x7fc81c153610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.185+0000 7fc821d83700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c0cd110 0x7fc81c156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.185+0000 7fc821d83700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc81c144b70 0x7fc81c1437a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.185+0000 7fc821d83700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c12e050 con 0x7fc81c0cd110 2024-03-18T19:10:08.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.185+0000 7fc821d83700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c153b50 con 0x7fc81c0af610 2024-03-18T19:10:08.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.185+0000 7fc821d83700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c157620 con 0x7fc81c144b70 2024-03-18T19:10:08.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.185+0000 7fc81affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c0cd110 0x7fc81c156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.185+0000 7fc81affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c0cd110 0x7fc81c156cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50890/0 (socket says 172.21.15.1:50890) 2024-03-18T19:10:08.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.185+0000 7fc81affd700 1 -- 172.21.15.1:0/834423053 learned_addr learned my addr 172.21.15.1:0/834423053 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:08.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.185+0000 7fc81bfff700 1 --2- 172.21.15.1:0/834423053 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc81c144b70 0x7fc81c1437a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.186+0000 7fc81b7fe700 1 --2- 172.21.15.1:0/834423053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc81c0af610 0x7fc81c153610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.186+0000 7fc81affd700 1 -- 172.21.15.1:0/834423053 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc81c144b70 msgr2=0x7fc81c1437a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:08.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.186+0000 7fc81affd700 1 --2- 172.21.15.1:0/834423053 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc81c144b70 0x7fc81c1437a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.186+0000 7fc81affd700 1 -- 172.21.15.1:0/834423053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc81c0af610 msgr2=0x7fc81c153610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:08.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.186+0000 7fc81affd700 1 --2- 172.21.15.1:0/834423053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc81c0af610 0x7fc81c153610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.186+0000 7fc81affd700 1 -- 172.21.15.1:0/834423053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc80c009550 con 0x7fc81c0cd110 2024-03-18T19:10:08.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.186+0000 7fc81bfff700 1 --2- 172.21.15.1:0/834423053 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc81c144b70 0x7fc81c1437a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:10:08.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.186+0000 7fc81b7fe700 1 --2- 172.21.15.1:0/834423053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc81c0af610 0x7fc81c153610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:10:08.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.186+0000 7fc81affd700 1 --2- 172.21.15.1:0/834423053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c0cd110 0x7fc81c156cb0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fc81000c4e0 tx=0x7fc81000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:08.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.186+0000 7fc803fff700 1 -- 172.21.15.1:0/834423053 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8100052d0 con 0x7fc81c0cd110 2024-03-18T19:10:08.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.186+0000 7fc803fff700 1 -- 172.21.15.1:0/834423053 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc810005430 con 0x7fc81c0cd110 2024-03-18T19:10:08.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.186+0000 7fc821d83700 1 -- 172.21.15.1:0/834423053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc81c06ecb0 con 0x7fc81c0cd110 2024-03-18T19:10:08.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.186+0000 7fc803fff700 1 -- 172.21.15.1:0/834423053 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8100038c0 con 0x7fc81c0cd110 2024-03-18T19:10:08.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.186+0000 7fc821d83700 1 -- 172.21.15.1:0/834423053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc81c15bad0 con 0x7fc81c0cd110 2024-03-18T19:10:08.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.187+0000 7fc821d83700 1 -- 172.21.15.1:0/834423053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc81c0cda90 con 0x7fc81c0cd110 2024-03-18T19:10:08.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.187+0000 7fc803fff700 1 -- 172.21.15.1:0/834423053 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc810003a20 con 0x7fc81c0cd110 2024-03-18T19:10:08.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.188+0000 7fc803fff700 1 --2- 172.21.15.1:0/834423053 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc80806adb0 0x7fc80806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.188+0000 7fc803fff700 1 -- 172.21.15.1:0/834423053 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc810088a30 con 0x7fc81c0cd110 2024-03-18T19:10:08.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.188+0000 7fc81b7fe700 1 --2- 172.21.15.1:0/834423053 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc80806adb0 0x7fc80806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.188+0000 7fc81b7fe700 1 --2- 172.21.15.1:0/834423053 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc80806adb0 0x7fc80806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc80c014040 tx=0x7fc80c008030 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:08.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.192+0000 7fc803fff700 1 -- 172.21.15.1:0/834423053 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc810053a30 con 0x7fc81c0cd110 2024-03-18T19:10:08.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.395+0000 7fc821d83700 1 -- 172.21.15.1:0/834423053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc81c0676a0 con 0x7fc81c0cd110 2024-03-18T19:10:08.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.397+0000 7fc803fff700 1 -- 172.21.15.1:0/834423053 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) v1 ==== 74+0+16260 (secure 0 0 0) 0x7fc810017090 con 0x7fc81c0cd110 2024-03-18T19:10:08.394 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:10:08.394 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":161,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T19:10:05.614846+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:10:02.452367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-18T19:10:03.585045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T19:10:08.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.399+0000 7fc821d83700 1 -- 172.21.15.1:0/834423053 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc80806adb0 msgr2=0x7fc80806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:08.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.399+0000 7fc821d83700 1 --2- 172.21.15.1:0/834423053 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc80806adb0 0x7fc80806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc80c014040 tx=0x7fc80c008030 comp rx=0 tx=0).stop 2024-03-18T19:10:08.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.399+0000 7fc821d83700 1 -- 172.21.15.1:0/834423053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c0cd110 msgr2=0x7fc81c156cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:08.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.399+0000 7fc821d83700 1 --2- 172.21.15.1:0/834423053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c0cd110 0x7fc81c156cb0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fc81000c4e0 tx=0x7fc81000c8a0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.399+0000 7fc821d83700 1 -- 172.21.15.1:0/834423053 shutdown_connections 2024-03-18T19:10:08.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.399+0000 7fc821d83700 1 --2- 172.21.15.1:0/834423053 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc80806adb0 0x7fc80806d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.399+0000 7fc821d83700 1 --2- 172.21.15.1:0/834423053 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc81c144b70 0x7fc81c1437a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.399+0000 7fc821d83700 1 --2- 172.21.15.1:0/834423053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc81c0af610 0x7fc81c153610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.399+0000 7fc821d83700 1 --2- 172.21.15.1:0/834423053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc81c0cd110 0x7fc81c156cb0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.399+0000 7fc821d83700 1 -- 172.21.15.1:0/834423053 >> 172.21.15.1:0/834423053 conn(0x7fc81c0b1d20 msgr2=0x7fc81c140fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:08.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.399+0000 7fc821d83700 1 -- 172.21.15.1:0/834423053 shutdown_connections 2024-03-18T19:10:08.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.399+0000 7fc821d83700 1 -- 172.21.15.1:0/834423053 wait complete. 2024-03-18T19:10:08.408 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T19:10:08.409 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T19:10:08.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.537+0000 7ff166eae700 1 Processor -- start 2024-03-18T19:10:08.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.537+0000 7ff166eae700 1 -- start start 2024-03-18T19:10:08.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.538+0000 7ff166eae700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff16006f240 0x7ff160164ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.538+0000 7ff166eae700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff160144000 con 0x7ff1600b4110 2024-03-18T19:10:08.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.538+0000 7ff166eae700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1600cd8d0 con 0x7ff160143c20 2024-03-18T19:10:08.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.538+0000 7ff166eae700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1600b51b0 con 0x7ff16006f240 2024-03-18T19:10:08.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.538+0000 7ff16544b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff16006f240 0x7ff160164ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.538+0000 7ff15ffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff1600b4110 0x7ff16006eb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52624/0 (socket says 172.21.15.1:52624) 2024-03-18T19:10:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.538+0000 7ff15ffff700 1 -- 172.21.15.1:0/1446616236 learned_addr learned my addr 172.21.15.1:0/1446616236 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:10:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.538+0000 7ff16544b700 1 -- 172.21.15.1:0/1446616236 >> v1:172.21.15.165:6789/0 conn(0x7ff160143c20 legacy=0x7ff1600b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.538+0000 7ff16544b700 1 -- 172.21.15.1:0/1446616236 >> v1:172.21.15.1:6789/0 conn(0x7ff1600b4110 legacy=0x7ff16006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.538+0000 7ff15f7fe700 1 -- 172.21.15.1:0/1446616236 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356489295 0 0) 0x7ff160144000 con 0x7ff1600b4110 2024-03-18T19:10:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.538+0000 7ff16544b700 1 -- 172.21.15.1:0/1446616236 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff16006fdc0 con 0x7ff16006f240 2024-03-18T19:10:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.538+0000 7ff16544b700 1 --2- 172.21.15.1:0/1446616236 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff16006f240 0x7ff160164ea0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7ff158009d90 tx=0x7ff15800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a9faa45464ae0ae server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.539+0000 7ff15f7fe700 1 -- 172.21.15.1:0/1446616236 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15800e070 con 0x7ff16006f240 2024-03-18T19:10:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.539+0000 7ff15f7fe700 1 -- 172.21.15.1:0/1446616236 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff15800bb90 con 0x7ff16006f240 2024-03-18T19:10:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.539+0000 7ff15f7fe700 1 -- 172.21.15.1:0/1446616236 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff158003a20 con 0x7ff16006f240 2024-03-18T19:10:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.539+0000 7ff166eae700 1 -- 172.21.15.1:0/1446616236 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff16006f240 msgr2=0x7ff160164ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.539+0000 7ff166eae700 1 --2- 172.21.15.1:0/1446616236 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff16006f240 0x7ff160164ea0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7ff158009d90 tx=0x7ff15800b330 comp rx=0 tx=0).stop 2024-03-18T19:10:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.539+0000 7ff166eae700 1 -- 172.21.15.1:0/1446616236 shutdown_connections 2024-03-18T19:10:08.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.539+0000 7ff166eae700 1 --2- 172.21.15.1:0/1446616236 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff16006f240 0x7ff160164ea0 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.539+0000 7ff166eae700 1 -- 172.21.15.1:0/1446616236 >> 172.21.15.1:0/1446616236 conn(0x7ff1600b1d20 msgr2=0x7ff1600b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:08.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.539+0000 7ff166eae700 1 -- 172.21.15.1:0/1446616236 shutdown_connections 2024-03-18T19:10:08.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.539+0000 7ff166eae700 1 -- 172.21.15.1:0/1446616236 wait complete. 2024-03-18T19:10:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.540+0000 7ff166eae700 1 Processor -- start 2024-03-18T19:10:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.540+0000 7ff166eae700 1 -- start start 2024-03-18T19:10:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.541+0000 7ff166eae700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff16006f240 0x7ff16012ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.541+0000 7ff166eae700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff160143c20 0x7ff160142140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.541+0000 7ff166eae700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff160157900 0x7ff160158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.541+0000 7ff166eae700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16012e300 con 0x7ff160143c20 2024-03-18T19:10:08.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.541+0000 7ff166eae700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff160157d80 con 0x7ff16006f240 2024-03-18T19:10:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.541+0000 7ff166eae700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff160158060 con 0x7ff160157900 2024-03-18T19:10:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.541+0000 7ff15ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff160143c20 0x7ff160142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.541+0000 7ff15ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff160143c20 0x7ff160142140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50900/0 (socket says 172.21.15.1:50900) 2024-03-18T19:10:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.541+0000 7ff15ffff700 1 -- 172.21.15.1:0/3257457216 learned_addr learned my addr 172.21.15.1:0/3257457216 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.541+0000 7ff16544b700 1 --2- 172.21.15.1:0/3257457216 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff160157900 0x7ff160158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.541+0000 7ff164c4a700 1 --2- 172.21.15.1:0/3257457216 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff16006f240 0x7ff16012ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.542+0000 7ff15ffff700 1 -- 172.21.15.1:0/3257457216 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff160157900 msgr2=0x7ff160158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.542+0000 7ff15ffff700 1 --2- 172.21.15.1:0/3257457216 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff160157900 0x7ff160158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.542+0000 7ff15ffff700 1 -- 172.21.15.1:0/3257457216 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff16006f240 msgr2=0x7ff16012ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.542+0000 7ff15ffff700 1 --2- 172.21.15.1:0/3257457216 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff16006f240 0x7ff16012ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.542+0000 7ff15ffff700 1 -- 172.21.15.1:0/3257457216 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff158009a70 con 0x7ff160143c20 2024-03-18T19:10:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.542+0000 7ff15ffff700 1 --2- 172.21.15.1:0/3257457216 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff160143c20 0x7ff160142140 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7ff14c00c520 tx=0x7ff14c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.542+0000 7ff15d7fa700 1 -- 172.21.15.1:0/3257457216 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c004c80 con 0x7ff160143c20 2024-03-18T19:10:08.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.542+0000 7ff15d7fa700 1 -- 172.21.15.1:0/3257457216 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14c004e00 con 0x7ff160143c20 2024-03-18T19:10:08.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.542+0000 7ff15d7fa700 1 -- 172.21.15.1:0/3257457216 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c007310 con 0x7ff160143c20 2024-03-18T19:10:08.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.542+0000 7ff166eae700 1 -- 172.21.15.1:0/3257457216 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff160158340 con 0x7ff160143c20 2024-03-18T19:10:08.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.542+0000 7ff166eae700 1 -- 172.21.15.1:0/3257457216 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff160158900 con 0x7ff160143c20 2024-03-18T19:10:08.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.543+0000 7ff166eae700 1 -- 172.21.15.1:0/3257457216 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1600676a0 con 0x7ff160143c20 2024-03-18T19:10:08.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.543+0000 7ff15d7fa700 1 -- 172.21.15.1:0/3257457216 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff14c007470 con 0x7ff160143c20 2024-03-18T19:10:08.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.544+0000 7ff15d7fa700 1 --2- 172.21.15.1:0/3257457216 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff14406adb0 0x7ff14406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.544+0000 7ff15d7fa700 1 -- 172.21.15.1:0/3257457216 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff14c089310 con 0x7ff160143c20 2024-03-18T19:10:08.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.544+0000 7ff164c4a700 1 --2- 172.21.15.1:0/3257457216 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff14406adb0 0x7ff14406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.544+0000 7ff164c4a700 1 --2- 172.21.15.1:0/3257457216 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff14406adb0 0x7ff14406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff154009870 tx=0x7ff154008040 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:08.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.548+0000 7ff15d7fa700 1 -- 172.21.15.1:0/3257457216 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff14c054280 con 0x7ff160143c20 2024-03-18T19:10:08.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.748+0000 7ff166eae700 1 -- 172.21.15.1:0/3257457216 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff1600b4b60 con 0x7ff14406adb0 2024-03-18T19:10:08.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.757+0000 7ff15d7fa700 1 -- 172.21.15.1:0/3257457216 <== mgr.9153 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+279473 (secure 0 0 0) 0x7ff1540123e0 con 0x7ff14406adb0 2024-03-18T19:10:08.757 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:10:08.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.760+0000 7ff166eae700 1 -- 172.21.15.1:0/3257457216 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff14406adb0 msgr2=0x7ff14406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:08.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.760+0000 7ff166eae700 1 --2- 172.21.15.1:0/3257457216 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff14406adb0 0x7ff14406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff154009870 tx=0x7ff154008040 comp rx=0 tx=0).stop 2024-03-18T19:10:08.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.760+0000 7ff166eae700 1 -- 172.21.15.1:0/3257457216 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff160143c20 msgr2=0x7ff160142140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:08.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.760+0000 7ff166eae700 1 --2- 172.21.15.1:0/3257457216 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff160143c20 0x7ff160142140 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7ff14c00c520 tx=0x7ff14c00c8e0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.760+0000 7ff166eae700 1 -- 172.21.15.1:0/3257457216 shutdown_connections 2024-03-18T19:10:08.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.760+0000 7ff166eae700 1 --2- 172.21.15.1:0/3257457216 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff14406adb0 0x7ff14406d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.760+0000 7ff166eae700 1 --2- 172.21.15.1:0/3257457216 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff160157900 0x7ff160158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.760+0000 7ff166eae700 1 --2- 172.21.15.1:0/3257457216 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff16006f240 0x7ff16012ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.760+0000 7ff166eae700 1 --2- 172.21.15.1:0/3257457216 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff160143c20 0x7ff160142140 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.760+0000 7ff166eae700 1 -- 172.21.15.1:0/3257457216 >> 172.21.15.1:0/3257457216 conn(0x7ff1600b1d20 msgr2=0x7ff160071650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:08.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.760+0000 7ff166eae700 1 -- 172.21.15.1:0/3257457216 shutdown_connections 2024-03-18T19:10:08.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.760+0000 7ff166eae700 1 -- 172.21.15.1:0/3257457216 wait complete. 2024-03-18T19:10:08.760 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T19:10:08.769 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-18T19:10:07.082737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592092,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":428,"num_write_kb":2421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":754974720,"kb_used":352412,"kb_used_data":16548,"kb_used_omap":12,"kb_used_meta":335731,"kb_avail":754622308,"statfs":{"total":773094113280,"available":772733243392,"internally_reserved":0,"allocated":16945152,"data_stored":9106802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":343789113},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1724,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000913"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253750+0000","last_change":"2024-03-18T19:10:04.628148+0000","last_active":"2024-03-18T19:10:06.253750+0000","last_peered":"2024-03-18T19:10:06.253750+0000","last_clean":"2024-03-18T19:10:06.253750+0000","last_became_active":"2024-03-18T19:10:04.627881+0000","last_became_peered":"2024-03-18T19:10:04.627881+0000","last_unstale":"2024-03-18T19:10:06.253750+0000","last_undegraded":"2024-03-18T19:10:06.253750+0000","last_fullsized":"2024-03-18T19:10:06.253750+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:20:15.737771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253585+0000","last_change":"2024-03-18T19:10:04.629276+0000","last_active":"2024-03-18T19:10:06.253585+0000","last_peered":"2024-03-18T19:10:06.253585+0000","last_clean":"2024-03-18T19:10:06.253585+0000","last_became_active":"2024-03-18T19:10:04.628975+0000","last_became_peered":"2024-03-18T19:10:04.628975+0000","last_unstale":"2024-03-18T19:10:06.253585+0000","last_undegraded":"2024-03-18T19:10:06.253585+0000","last_fullsized":"2024-03-18T19:10:06.253585+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:02:40.471904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254977+0000","last_change":"2024-03-18T19:10:04.644642+0000","last_active":"2024-03-18T19:10:06.254977+0000","last_peered":"2024-03-18T19:10:06.254977+0000","last_clean":"2024-03-18T19:10:06.254977+0000","last_became_active":"2024-03-18T19:10:04.644457+0000","last_became_peered":"2024-03-18T19:10:04.644457+0000","last_unstale":"2024-03-18T19:10:06.254977+0000","last_undegraded":"2024-03-18T19:10:06.254977+0000","last_fullsized":"2024-03-18T19:10:06.254977+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:36.601312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254252+0000","last_change":"2024-03-18T19:10:04.630189+0000","last_active":"2024-03-18T19:10:06.254252+0000","last_peered":"2024-03-18T19:10:06.254252+0000","last_clean":"2024-03-18T19:10:06.254252+0000","last_became_active":"2024-03-18T19:10:04.629953+0000","last_became_peered":"2024-03-18T19:10:04.629953+0000","last_unstale":"2024-03-18T19:10:06.254252+0000","last_undegraded":"2024-03-18T19:10:06.254252+0000","last_fullsized":"2024-03-18T19:10:06.254252+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:11:37.835302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253565+0000","last_change":"2024-03-18T19:10:04.640038+0000","last_active":"2024-03-18T19:10:06.253565+0000","last_peered":"2024-03-18T19:10:06.253565+0000","last_clean":"2024-03-18T19:10:06.253565+0000","last_became_active":"2024-03-18T19:10:04.639621+0000","last_became_peered":"2024-03-18T19:10:04.639621+0000","last_unstale":"2024-03-18T19:10:06.253565+0000","last_undegraded":"2024-03-18T19:10:06.253565+0000","last_fullsized":"2024-03-18T19:10:06.253565+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:41.909132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.255069+0000","last_change":"2024-03-18T19:10:04.638251+0000","last_active":"2024-03-18T19:10:06.255069+0000","last_peered":"2024-03-18T19:10:06.255069+0000","last_clean":"2024-03-18T19:10:06.255069+0000","last_became_active":"2024-03-18T19:10:04.636850+0000","last_became_peered":"2024-03-18T19:10:04.636850+0000","last_unstale":"2024-03-18T19:10:06.255069+0000","last_undegraded":"2024-03-18T19:10:06.255069+0000","last_fullsized":"2024-03-18T19:10:06.255069+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:27:58.639356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254054+0000","last_change":"2024-03-18T19:10:04.636481+0000","last_active":"2024-03-18T19:10:06.254054+0000","last_peered":"2024-03-18T19:10:06.254054+0000","last_clean":"2024-03-18T19:10:06.254054+0000","last_became_active":"2024-03-18T19:10:04.632303+0000","last_became_peered":"2024-03-18T19:10:04.632303+0000","last_unstale":"2024-03-18T19:10:06.254054+0000","last_undegraded":"2024-03-18T19:10:06.254054+0000","last_fullsized":"2024-03-18T19:10:06.254054+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:47:30.693079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253673+0000","last_change":"2024-03-18T19:10:04.639879+0000","last_active":"2024-03-18T19:10:06.253673+0000","last_peered":"2024-03-18T19:10:06.253673+0000","last_clean":"2024-03-18T19:10:06.253673+0000","last_became_active":"2024-03-18T19:10:04.639333+0000","last_became_peered":"2024-03-18T19:10:04.639333+0000","last_unstale":"2024-03-18T19:10:06.253673+0000","last_undegraded":"2024-03-18T19:10:06.253673+0000","last_fullsized":"2024-03-18T19:10:06.253673+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:02:40.471904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253760+0000","last_change":"2024-03-18T19:10:04.623800+0000","last_active":"2024-03-18T19:10:06.253760+0000","last_peered":"2024-03-18T19:10:06.253760+0000","last_clean":"2024-03-18T19:10:06.253760+0000","last_became_active":"2024-03-18T19:10:04.623518+0000","last_became_peered":"2024-03-18T19:10:04.623518+0000","last_unstale":"2024-03-18T19:10:06.253760+0000","last_undegraded":"2024-03-18T19:10:06.253760+0000","last_fullsized":"2024-03-18T19:10:06.253760+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:20:15.737771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:03.170956+0000","last_change":"2024-03-18T19:10:03.170956+0000","last_active":"2024-03-18T19:10:03.170956+0000","last_peered":"2024-03-18T19:10:03.170956+0000","last_clean":"2024-03-18T19:10:03.170956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:03.170956+0000","last_undegraded":"2024-03-18T19:10:03.170956+0000","last_fullsized":"2024-03-18T19:10:03.170956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:03.170956+0000","last_change":"2024-03-18T19:10:03.170956+0000","last_active":"2024-03-18T19:10:03.170956+0000","last_peered":"2024-03-18T19:10:03.170956+0000","last_clean":"2024-03-18T19:10:03.170956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:03.170956+0000","last_undegraded":"2024-03-18T19:10:03.170956+0000","last_fullsized":"2024-03-18T19:10:03.170956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1d","version":"160'2","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254049+0000","last_change":"2024-03-18T19:10:04.200336+0000","last_active":"2024-03-18T19:10:06.254049+0000","last_peered":"2024-03-18T19:10:06.254049+0000","last_clean":"2024-03-18T19:10:06.254049+0000","last_became_active":"2024-03-18T19:10:04.200105+0000","last_became_peered":"2024-03-18T19:10:04.200105+0000","last_unstale":"2024-03-18T19:10:06.254049+0000","last_undegraded":"2024-03-18T19:10:06.254049+0000","last_fullsized":"2024-03-18T19:10:06.254049+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:09.616522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1c","version":"160'2","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253914+0000","last_change":"2024-03-18T19:10:04.197457+0000","last_active":"2024-03-18T19:10:06.253914+0000","last_peered":"2024-03-18T19:10:06.253914+0000","last_clean":"2024-03-18T19:10:06.253914+0000","last_became_active":"2024-03-18T19:10:04.197160+0000","last_became_peered":"2024-03-18T19:10:04.197160+0000","last_unstale":"2024-03-18T19:10:06.253914+0000","last_undegraded":"2024-03-18T19:10:06.253914+0000","last_fullsized":"2024-03-18T19:10:06.253914+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:57:05.031861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:03.170956+0000","last_change":"2024-03-18T19:10:03.170956+0000","last_active":"2024-03-18T19:10:03.170956+0000","last_peered":"2024-03-18T19:10:03.170956+0000","last_clean":"2024-03-18T19:10:03.170956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:03.170956+0000","last_undegraded":"2024-03-18T19:10:03.170956+0000","last_fullsized":"2024-03-18T19:10:03.170956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:03.170956+0000","last_change":"2024-03-18T19:10:03.170956+0000","last_active":"2024-03-18T19:10:03.170956+0000","last_peered":"2024-03-18T19:10:03.170956+0000","last_clean":"2024-03-18T19:10:03.170956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:03.170956+0000","last_undegraded":"2024-03-18T19:10:03.170956+0000","last_fullsized":"2024-03-18T19:10:03.170956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254126+0000","last_change":"2024-03-18T19:10:04.196553+0000","last_active":"2024-03-18T19:10:06.254126+0000","last_peered":"2024-03-18T19:10:06.254126+0000","last_clean":"2024-03-18T19:10:06.254126+0000","last_became_active":"2024-03-18T19:10:04.196259+0000","last_became_peered":"2024-03-18T19:10:04.196259+0000","last_unstale":"2024-03-18T19:10:06.254126+0000","last_undegraded":"2024-03-18T19:10:06.254126+0000","last_fullsized":"2024-03-18T19:10:06.254126+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:08:48.452699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254153+0000","last_change":"2024-03-18T19:10:04.624942+0000","last_active":"2024-03-18T19:10:06.254153+0000","last_peered":"2024-03-18T19:10:06.254153+0000","last_clean":"2024-03-18T19:10:06.254153+0000","last_became_active":"2024-03-18T19:10:04.624653+0000","last_became_peered":"2024-03-18T19:10:04.624653+0000","last_unstale":"2024-03-18T19:10:06.254153+0000","last_undegraded":"2024-03-18T19:10:06.254153+0000","last_fullsized":"2024-03-18T19:10:06.254153+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:41.506510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:03.170956+0000","last_change":"2024-03-18T19:10:03.170956+0000","last_active":"2024-03-18T19:10:03.170956+0000","last_peered":"2024-03-18T19:10:03.170956+0000","last_clean":"2024-03-18T19:10:03.170956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:03.170956+0000","last_undegraded":"2024-03-18T19:10:03.170956+0000","last_fullsized":"2024-03-18T19:10:03.170956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:03.170956+0000","last_change":"2024-03-18T19:10:03.170956+0000","last_active":"2024-03-18T19:10:03.170956+0000","last_peered":"2024-03-18T19:10:03.170956+0000","last_clean":"2024-03-18T19:10:03.170956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:03.170956+0000","last_undegraded":"2024-03-18T19:10:03.170956+0000","last_fullsized":"2024-03-18T19:10:03.170956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253841+0000","last_change":"2024-03-18T19:10:04.635217+0000","last_active":"2024-03-18T19:10:06.253841+0000","last_peered":"2024-03-18T19:10:06.253841+0000","last_clean":"2024-03-18T19:10:06.253841+0000","last_became_active":"2024-03-18T19:10:04.635017+0000","last_became_peered":"2024-03-18T19:10:04.635017+0000","last_unstale":"2024-03-18T19:10:06.253841+0000","last_undegraded":"2024-03-18T19:10:06.253841+0000","last_fullsized":"2024-03-18T19:10:06.253841+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:41:46.475185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:03.170956+0000","last_change":"2024-03-18T19:10:03.170956+0000","last_active":"2024-03-18T19:10:03.170956+0000","last_peered":"2024-03-18T19:10:03.170956+0000","last_clean":"2024-03-18T19:10:03.170956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:03.170956+0000","last_undegraded":"2024-03-18T19:10:03.170956+0000","last_fullsized":"2024-03-18T19:10:03.170956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.15","version":"160'3","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254165+0000","last_change":"2024-03-18T19:10:04.200205+0000","last_active":"2024-03-18T19:10:06.254165+0000","last_peered":"2024-03-18T19:10:06.254165+0000","last_clean":"2024-03-18T19:10:06.254165+0000","last_became_active":"2024-03-18T19:10:04.199937+0000","last_became_peered":"2024-03-18T19:10:04.199937+0000","last_unstale":"2024-03-18T19:10:06.254165+0000","last_undegraded":"2024-03-18T19:10:06.254165+0000","last_fullsized":"2024-03-18T19:10:06.254165+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:08:48.452699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.14","version":"160'6","reported_seq":17,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253947+0000","last_change":"2024-03-18T19:10:04.199066+0000","last_active":"2024-03-18T19:10:06.253947+0000","last_peered":"2024-03-18T19:10:06.253947+0000","last_clean":"2024-03-18T19:10:06.253947+0000","last_became_active":"2024-03-18T19:10:04.198773+0000","last_became_peered":"2024-03-18T19:10:04.198773+0000","last_unstale":"2024-03-18T19:10:06.253947+0000","last_undegraded":"2024-03-18T19:10:06.253947+0000","last_fullsized":"2024-03-18T19:10:06.253947+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:16.096174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:03.170956+0000","last_change":"2024-03-18T19:10:03.170956+0000","last_active":"2024-03-18T19:10:03.170956+0000","last_peered":"2024-03-18T19:10:03.170956+0000","last_clean":"2024-03-18T19:10:03.170956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:03.170956+0000","last_undegraded":"2024-03-18T19:10:03.170956+0000","last_fullsized":"2024-03-18T19:10:03.170956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254204+0000","last_change":"2024-03-18T19:10:04.637200+0000","last_active":"2024-03-18T19:10:06.254204+0000","last_peered":"2024-03-18T19:10:06.254204+0000","last_clean":"2024-03-18T19:10:06.254204+0000","last_became_active":"2024-03-18T19:10:04.636694+0000","last_became_peered":"2024-03-18T19:10:04.636694+0000","last_unstale":"2024-03-18T19:10:06.254204+0000","last_undegraded":"2024-03-18T19:10:06.254204+0000","last_fullsized":"2024-03-18T19:10:06.254204+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:46.740895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:03.170956+0000","last_change":"2024-03-18T19:10:03.170956+0000","last_active":"2024-03-18T19:10:03.170956+0000","last_peered":"2024-03-18T19:10:03.170956+0000","last_clean":"2024-03-18T19:10:03.170956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:03.170956+0000","last_undegraded":"2024-03-18T19:10:03.170956+0000","last_fullsized":"2024-03-18T19:10:03.170956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.10","version":"160'8","reported_seq":19,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253775+0000","last_change":"2024-03-18T19:10:04.202116+0000","last_active":"2024-03-18T19:10:06.253775+0000","last_peered":"2024-03-18T19:10:06.253775+0000","last_clean":"2024-03-18T19:10:06.253775+0000","last_became_active":"2024-03-18T19:10:04.201805+0000","last_became_peered":"2024-03-18T19:10:04.201805+0000","last_unstale":"2024-03-18T19:10:06.253775+0000","last_undegraded":"2024-03-18T19:10:06.253775+0000","last_fullsized":"2024-03-18T19:10:06.253775+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:55.480457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:03.170956+0000","last_change":"2024-03-18T19:10:03.170956+0000","last_active":"2024-03-18T19:10:03.170956+0000","last_peered":"2024-03-18T19:10:03.170956+0000","last_clean":"2024-03-18T19:10:03.170956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:03.170956+0000","last_undegraded":"2024-03-18T19:10:03.170956+0000","last_fullsized":"2024-03-18T19:10:03.170956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253824+0000","last_change":"2024-03-18T19:10:04.644492+0000","last_active":"2024-03-18T19:10:06.253824+0000","last_peered":"2024-03-18T19:10:06.253824+0000","last_clean":"2024-03-18T19:10:06.253824+0000","last_became_active":"2024-03-18T19:10:04.644009+0000","last_became_peered":"2024-03-18T19:10:04.644009+0000","last_unstale":"2024-03-18T19:10:06.253824+0000","last_undegraded":"2024-03-18T19:10:06.253824+0000","last_fullsized":"2024-03-18T19:10:06.253824+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:11:37.835302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:03.170956+0000","last_change":"2024-03-18T19:10:03.170956+0000","last_active":"2024-03-18T19:10:03.170956+0000","last_peered":"2024-03-18T19:10:03.170956+0000","last_clean":"2024-03-18T19:10:03.170956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:03.170956+0000","last_undegraded":"2024-03-18T19:10:03.170956+0000","last_fullsized":"2024-03-18T19:10:03.170956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253684+0000","last_change":"2024-03-18T19:10:04.629440+0000","last_active":"2024-03-18T19:10:06.253684+0000","last_peered":"2024-03-18T19:10:06.253684+0000","last_clean":"2024-03-18T19:10:06.253684+0000","last_became_active":"2024-03-18T19:10:04.629022+0000","last_became_peered":"2024-03-18T19:10:04.629022+0000","last_unstale":"2024-03-18T19:10:06.253684+0000","last_undegraded":"2024-03-18T19:10:06.253684+0000","last_fullsized":"2024-03-18T19:10:06.253684+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:10.396783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"160'1","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254223+0000","last_change":"2024-03-18T19:10:04.196572+0000","last_active":"2024-03-18T19:10:06.254223+0000","last_peered":"2024-03-18T19:10:06.254223+0000","last_clean":"2024-03-18T19:10:06.254223+0000","last_became_active":"2024-03-18T19:10:04.196281+0000","last_became_peered":"2024-03-18T19:10:04.196281+0000","last_unstale":"2024-03-18T19:10:06.254223+0000","last_undegraded":"2024-03-18T19:10:06.254223+0000","last_fullsized":"2024-03-18T19:10:06.254223+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:31:25.926313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:03.170956+0000","last_change":"2024-03-18T19:10:03.170956+0000","last_active":"2024-03-18T19:10:03.170956+0000","last_peered":"2024-03-18T19:10:03.170956+0000","last_clean":"2024-03-18T19:10:03.170956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:03.170956+0000","last_undegraded":"2024-03-18T19:10:03.170956+0000","last_fullsized":"2024-03-18T19:10:03.170956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:03.170956+0000","last_change":"2024-03-18T19:10:03.170956+0000","last_active":"2024-03-18T19:10:03.170956+0000","last_peered":"2024-03-18T19:10:03.170956+0000","last_clean":"2024-03-18T19:10:03.170956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:03.170956+0000","last_undegraded":"2024-03-18T19:10:03.170956+0000","last_fullsized":"2024-03-18T19:10:03.170956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:03.170956+0000","last_change":"2024-03-18T19:10:03.170956+0000","last_active":"2024-03-18T19:10:03.170956+0000","last_peered":"2024-03-18T19:10:03.170956+0000","last_clean":"2024-03-18T19:10:03.170956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:03.170956+0000","last_undegraded":"2024-03-18T19:10:03.170956+0000","last_fullsized":"2024-03-18T19:10:03.170956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253935+0000","last_change":"2024-03-18T19:10:04.202020+0000","last_active":"2024-03-18T19:10:06.253935+0000","last_peered":"2024-03-18T19:10:06.253935+0000","last_clean":"2024-03-18T19:10:06.253935+0000","last_became_active":"2024-03-18T19:10:04.201759+0000","last_became_peered":"2024-03-18T19:10:04.201759+0000","last_unstale":"2024-03-18T19:10:06.253935+0000","last_undegraded":"2024-03-18T19:10:06.253935+0000","last_fullsized":"2024-03-18T19:10:06.253935+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:08.645798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253864+0000","last_change":"2024-03-18T19:10:04.186249+0000","last_active":"2024-03-18T19:10:06.253864+0000","last_peered":"2024-03-18T19:10:06.253864+0000","last_clean":"2024-03-18T19:10:06.253864+0000","last_became_active":"2024-03-18T19:10:04.185337+0000","last_became_peered":"2024-03-18T19:10:04.185337+0000","last_unstale":"2024-03-18T19:10:06.253864+0000","last_undegraded":"2024-03-18T19:10:06.253864+0000","last_fullsized":"2024-03-18T19:10:06.253864+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:30:07.307315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:03.170956+0000","last_change":"2024-03-18T19:10:03.170956+0000","last_active":"2024-03-18T19:10:03.170956+0000","last_peered":"2024-03-18T19:10:03.170956+0000","last_clean":"2024-03-18T19:10:03.170956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:03.170956+0000","last_undegraded":"2024-03-18T19:10:03.170956+0000","last_fullsized":"2024-03-18T19:10:03.170956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:03.170956+0000","last_change":"2024-03-18T19:10:03.170956+0000","last_active":"2024-03-18T19:10:03.170956+0000","last_peered":"2024-03-18T19:10:03.170956+0000","last_clean":"2024-03-18T19:10:03.170956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:03.170956+0000","last_undegraded":"2024-03-18T19:10:03.170956+0000","last_fullsized":"2024-03-18T19:10:03.170956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253746+0000","last_change":"2024-03-18T19:10:04.629345+0000","last_active":"2024-03-18T19:10:06.253746+0000","last_peered":"2024-03-18T19:10:06.253746+0000","last_clean":"2024-03-18T19:10:06.253746+0000","last_became_active":"2024-03-18T19:10:04.629036+0000","last_became_peered":"2024-03-18T19:10:04.629036+0000","last_unstale":"2024-03-18T19:10:06.253746+0000","last_undegraded":"2024-03-18T19:10:06.253746+0000","last_fullsized":"2024-03-18T19:10:06.253746+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:36.601312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:03.170956+0000","last_change":"2024-03-18T19:10:03.170956+0000","last_active":"2024-03-18T19:10:03.170956+0000","last_peered":"2024-03-18T19:10:03.170956+0000","last_clean":"2024-03-18T19:10:03.170956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:03.170956+0000","last_undegraded":"2024-03-18T19:10:03.170956+0000","last_fullsized":"2024-03-18T19:10:03.170956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254831+0000","last_change":"2024-03-18T19:10:04.193415+0000","last_active":"2024-03-18T19:10:06.254831+0000","last_peered":"2024-03-18T19:10:06.254831+0000","last_clean":"2024-03-18T19:10:06.254831+0000","last_became_active":"2024-03-18T19:10:04.193131+0000","last_became_peered":"2024-03-18T19:10:04.193131+0000","last_unstale":"2024-03-18T19:10:06.254831+0000","last_undegraded":"2024-03-18T19:10:06.254831+0000","last_fullsized":"2024-03-18T19:10:06.254831+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:57:05.031861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254730+0000","last_change":"2024-03-18T19:10:04.632620+0000","last_active":"2024-03-18T19:10:06.254730+0000","last_peered":"2024-03-18T19:10:06.254730+0000","last_clean":"2024-03-18T19:10:06.254730+0000","last_became_active":"2024-03-18T19:10:04.632329+0000","last_became_peered":"2024-03-18T19:10:04.632329+0000","last_unstale":"2024-03-18T19:10:06.254730+0000","last_undegraded":"2024-03-18T19:10:06.254730+0000","last_fullsized":"2024-03-18T19:10:06.254730+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:41.506510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:03.170956+0000","last_change":"2024-03-18T19:10:03.170956+0000","last_active":"2024-03-18T19:10:03.170956+0000","last_peered":"2024-03-18T19:10:03.170956+0000","last_clean":"2024-03-18T19:10:03.170956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:03.170956+0000","last_undegraded":"2024-03-18T19:10:03.170956+0000","last_fullsized":"2024-03-18T19:10:03.170956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254128+0000","last_change":"2024-03-18T19:10:04.640929+0000","last_active":"2024-03-18T19:10:06.254128+0000","last_peered":"2024-03-18T19:10:06.254128+0000","last_clean":"2024-03-18T19:10:06.254128+0000","last_became_active":"2024-03-18T19:10:04.636895+0000","last_became_peered":"2024-03-18T19:10:04.636895+0000","last_unstale":"2024-03-18T19:10:06.254128+0000","last_undegraded":"2024-03-18T19:10:06.254128+0000","last_fullsized":"2024-03-18T19:10:06.254128+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:10.396783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253920+0000","last_change":"2024-03-18T19:10:04.197527+0000","last_active":"2024-03-18T19:10:06.253920+0000","last_peered":"2024-03-18T19:10:06.253920+0000","last_clean":"2024-03-18T19:10:06.253920+0000","last_became_active":"2024-03-18T19:10:04.197266+0000","last_became_peered":"2024-03-18T19:10:04.197266+0000","last_unstale":"2024-03-18T19:10:06.253920+0000","last_undegraded":"2024-03-18T19:10:06.253920+0000","last_fullsized":"2024-03-18T19:10:06.253920+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:30:07.307315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254277+0000","last_change":"2024-03-18T19:10:04.633688+0000","last_active":"2024-03-18T19:10:06.254277+0000","last_peered":"2024-03-18T19:10:06.254277+0000","last_clean":"2024-03-18T19:10:06.254277+0000","last_became_active":"2024-03-18T19:10:04.633394+0000","last_became_peered":"2024-03-18T19:10:04.633394+0000","last_unstale":"2024-03-18T19:10:06.254277+0000","last_undegraded":"2024-03-18T19:10:06.254277+0000","last_fullsized":"2024-03-18T19:10:06.254277+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:47:30.693079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254029+0000","last_change":"2024-03-18T19:10:04.190562+0000","last_active":"2024-03-18T19:10:06.254029+0000","last_peered":"2024-03-18T19:10:06.254029+0000","last_clean":"2024-03-18T19:10:06.254029+0000","last_became_active":"2024-03-18T19:10:04.190286+0000","last_became_peered":"2024-03-18T19:10:04.190286+0000","last_unstale":"2024-03-18T19:10:06.254029+0000","last_undegraded":"2024-03-18T19:10:06.254029+0000","last_fullsized":"2024-03-18T19:10:06.254029+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:16.096174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254222+0000","last_change":"2024-03-18T19:10:04.629407+0000","last_active":"2024-03-18T19:10:06.254222+0000","last_peered":"2024-03-18T19:10:06.254222+0000","last_clean":"2024-03-18T19:10:06.254222+0000","last_became_active":"2024-03-18T19:10:04.629093+0000","last_became_peered":"2024-03-18T19:10:04.629093+0000","last_unstale":"2024-03-18T19:10:06.254222+0000","last_undegraded":"2024-03-18T19:10:06.254222+0000","last_fullsized":"2024-03-18T19:10:06.254222+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:27:58.639356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:03.170956+0000","last_change":"2024-03-18T19:10:03.170956+0000","last_active":"2024-03-18T19:10:03.170956+0000","last_peered":"2024-03-18T19:10:03.170956+0000","last_clean":"2024-03-18T19:10:03.170956+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:03.170956+0000","last_undegraded":"2024-03-18T19:10:03.170956+0000","last_fullsized":"2024-03-18T19:10:03.170956+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T19:10:04.176211+0000","last_change":"2024-03-18T19:10:04.176211+0000","last_active":"2024-03-18T19:10:04.176211+0000","last_peered":"2024-03-18T19:10:04.176211+0000","last_clean":"2024-03-18T19:10:04.176211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T19:10:04.176211+0000","last_undegraded":"2024-03-18T19:10:04.176211+0000","last_fullsized":"2024-03-18T19:10:04.176211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253805+0000","last_change":"2024-03-18T19:10:04.201282+0000","last_active":"2024-03-18T19:10:06.253805+0000","last_peered":"2024-03-18T19:10:06.253805+0000","last_clean":"2024-03-18T19:10:06.253805+0000","last_became_active":"2024-03-18T19:10:04.200985+0000","last_became_peered":"2024-03-18T19:10:04.200985+0000","last_unstale":"2024-03-18T19:10:06.253805+0000","last_undegraded":"2024-03-18T19:10:06.253805+0000","last_fullsized":"2024-03-18T19:10:06.253805+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:55.480457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"150'249","reported_seq":754,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-18T19:10:04.619475+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T19:10:04.619475+0000","last_peered":"2024-03-18T19:10:04.619475+0000","last_clean":"2024-03-18T19:10:04.619475+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T19:10:04.619475+0000","last_undegraded":"2024-03-18T19:10:04.619475+0000","last_fullsized":"2024-03-18T19:10:04.619475+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":38,"acting":38,"num_store_stats":2},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":1724,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22,"ondisk_log_size":22,"up":26,"acting":26,"num_store_stats":2},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575084,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59924,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311916,"statfs":{"total":96636764160,"available":96575401984,"internally_reserved":0,"allocated":2617344,"data_stored":1603978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.55300000000000005,"15min":0.56599999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.27800000000000002},"max":{"1min":0.76200000000000001,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.436},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55800000000000005,"15min":0.57899999999999996},"min":{"1min":0.373,"5min":0.33700000000000002,"15min":0.318},"max":{"1min":0.78300000000000003,"5min":1.387,"15min":1.387},"last":0.57799999999999996}]},{"osd":1,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.53500000000000003,"15min":0.56799999999999995},"min":{"1min":0.34399999999999997,"5min":0.33200000000000002,"15min":0.307},"max":{"1min":0.73299999999999998,"5min":0.81699999999999995,"15min":1.484},"last":0.505},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55100000000000005,"15min":0.58599999999999997},"min":{"1min":0.39500000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.92000000000000004},"last":0.623}]},{"osd":2,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55200000000000005,"15min":0.57099999999999995},"min":{"1min":0.38200000000000001,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":0.68000000000000005,"5min":1.3600000000000001,"15min":1.5129999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57499999999999996,"15min":0.59099999999999997},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73499999999999999,"5min":1.327,"15min":1.536},"last":0.59799999999999998}]},{"osd":3,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55400000000000005,"15min":0.57599999999999996},"min":{"1min":0.38800000000000001,"5min":0.35599999999999998,"15min":0.318},"max":{"1min":0.745,"5min":0.79100000000000004,"15min":1.5489999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57399999999999995,"15min":0.59799999999999998},"min":{"1min":0.441,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.78300000000000003,"5min":0.81699999999999995,"15min":1.5660000000000001},"last":0.54400000000000004}]},{"osd":4,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.45800000000000002,"15min":0.45300000000000001},"min":{"1min":0.30299999999999999,"5min":0.27800000000000002,"15min":0.27100000000000002},"max":{"1min":0.748,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.46999999999999997,"15min":0.47299999999999998},"min":{"1min":0.35299999999999998,"5min":0.27500000000000002,"15min":0.25800000000000001},"max":{"1min":0.63900000000000001,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.41199999999999998}]},{"osd":5,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.48499999999999999,"15min":0.47699999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.26000000000000001},"max":{"1min":0.68600000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.51500000000000001,"15min":0.51400000000000001},"min":{"1min":0.33600000000000002,"5min":0.317,"15min":0.30299999999999999},"max":{"1min":0.70699999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.373}]},{"osd":6,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.54300000000000004},"min":{"1min":0.34000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.74099999999999999,"5min":0.81799999999999995,"15min":0.82799999999999996},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53600000000000003,"15min":0.53000000000000003},"min":{"1min":0.36399999999999999,"5min":0.311,"15min":0.29099999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.47499999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37124,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334716,"statfs":{"total":96636764160,"available":96598749184,"internally_reserved":0,"allocated":1957888,"data_stored":985405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52500000000000002,"15min":0.55500000000000005},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28499999999999998},"max":{"1min":0.67800000000000005,"5min":0.73699999999999999,"15min":0.76800000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53900000000000003,"15min":0.56499999999999995},"min":{"1min":0.36199999999999999,"5min":0.35199999999999998,"15min":0.27400000000000002},"max":{"1min":0.71099999999999997,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.58099999999999996}]},{"osd":1,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.56000000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.77400000000000002},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.56499999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.32000000000000001},"max":{"1min":0.78900000000000003,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.53400000000000003}]},{"osd":2,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55400000000000005,"15min":0.57299999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.30499999999999999},"max":{"1min":0.73699999999999999,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54500000000000004,"15min":0.57399999999999995},"min":{"1min":0.40999999999999998,"5min":0.35299999999999998,"15min":0.30299999999999999},"max":{"1min":0.73399999999999999,"5min":0.753,"15min":0.83199999999999996},"last":0.47099999999999997}]},{"osd":3,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.58399999999999996},"min":{"1min":0.40799999999999997,"5min":0.372,"15min":0.32100000000000001},"max":{"1min":0.76400000000000001,"5min":0.78400000000000003,"15min":0.89600000000000002},"last":0.624},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55400000000000005,"15min":0.57199999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72099999999999997,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.72099999999999997}]},{"osd":4,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46700000000000003,"15min":0.45000000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.48899999999999999,"15min":0.47699999999999998},"min":{"1min":0.32200000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.69399999999999995}]},{"osd":5,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.75,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.747},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53400000000000003,"15min":0.51600000000000001},"min":{"1min":0.34899999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.80400000000000005}]},{"osd":7,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54400000000000004,"15min":0.52900000000000003},"min":{"1min":0.371,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55600000000000005,"15min":0.53600000000000003},"min":{"1min":0.371,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.80500000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.79000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575084,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37176,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334664,"statfs":{"total":96636764160,"available":96598695936,"internally_reserved":0,"allocated":2011136,"data_stored":1012026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.35499999999999998,"15min":0.373},"min":{"1min":0.216,"5min":0.216,"15min":0.19600000000000001},"max":{"1min":0.58699999999999997,"5min":0.63300000000000001,"15min":0.66300000000000003},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.38600000000000001,"15min":0.41299999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.57099999999999995,"5min":0.66500000000000004,"15min":0.73699999999999999},"last":0.27900000000000003}]},{"osd":2,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38900000000000001,"15min":0.40400000000000003},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.59299999999999997,"5min":0.64200000000000002,"15min":0.65400000000000003},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.434,"15min":0.44500000000000001},"min":{"1min":0.29599999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.63300000000000001,"5min":0.66500000000000004,"15min":0.71699999999999997},"last":0.36599999999999999}]},{"osd":3,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.44500000000000001,"15min":0.46400000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.60399999999999998,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40200000000000002,"15min":0.41899999999999998},"min":{"1min":0.25900000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.61199999999999999,"5min":0.67800000000000005,"15min":0.77000000000000002},"last":0.38400000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60499999999999998,"15min":0.628},"min":{"1min":0.46200000000000002,"5min":0.33400000000000002,"15min":0.32800000000000001},"max":{"1min":0.89600000000000002,"5min":0.92300000000000004,"15min":0.93999999999999995},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.62,"15min":0.64100000000000001},"min":{"1min":0.443,"5min":0.439,"15min":0.36899999999999999},"max":{"1min":0.86499999999999999,"5min":0.97499999999999998,"15min":1.018},"last":0.59999999999999998}]},{"osd":5,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60399999999999998,"15min":0.63700000000000001},"min":{"1min":0.40500000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88700000000000001,"5min":0.89500000000000002,"15min":0.99199999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.63200000000000001,"15min":0.65600000000000003},"min":{"1min":0.47199999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":1.006},"last":0.57399999999999995}]},{"osd":6,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.60499999999999998,"15min":0.63400000000000001},"min":{"1min":0.42899999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.878,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.64600000000000002,"15min":0.67000000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.39100000000000001},"max":{"1min":0.91300000000000003,"5min":0.94899999999999995,"15min":0.98099999999999998},"last":0.66600000000000004}]},{"osd":7,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63200000000000001,"15min":0.65900000000000003},"min":{"1min":0.45100000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81899999999999995,"5min":0.90600000000000003,"15min":0.94799999999999995},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63400000000000001,"15min":0.66200000000000003},"min":{"1min":0.48299999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90800000000000003,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.58799999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575084,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37692,"kb_used_data":2480,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334148,"statfs":{"total":96636764160,"available":96598167552,"internally_reserved":0,"allocated":2539520,"data_stored":1569773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36299999999999999,"15min":0.376},"min":{"1min":0.218,"5min":0.19700000000000001,"15min":0.19400000000000001},"max":{"1min":0.59899999999999998,"5min":0.60499999999999998,"15min":0.621},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39400000000000002,"15min":0.40999999999999998},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20000000000000001},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.63200000000000001},"last":0.42599999999999999}]},{"osd":2,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40699999999999997,"15min":0.41699999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.21099999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.42299999999999999,"15min":0.432},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.59399999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.34000000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.40300000000000002,"15min":0.41299999999999998},"min":{"1min":0.27200000000000002,"5min":0.26500000000000001,"15min":0.217},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.64100000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.45500000000000002,"15min":0.46800000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.28799999999999998},"max":{"1min":0.64300000000000002,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.40999999999999998}]},{"osd":4,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.60099999999999998,"15min":0.63300000000000001},"min":{"1min":0.376,"5min":0.35699999999999998,"15min":0.27200000000000002},"max":{"1min":0.76900000000000002,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.60899999999999999,"15min":0.63500000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.315},"max":{"1min":0.75600000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.52300000000000002}]},{"osd":5,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.627,"15min":0.64300000000000002},"min":{"1min":0.432,"5min":0.376,"15min":0.34300000000000003},"max":{"1min":0.78100000000000003,"5min":1.248,"15min":1.248},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.63800000000000001,"15min":0.65800000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.31},"max":{"1min":0.84899999999999998,"5min":1.399,"15min":1.399},"last":0.55300000000000005}]},{"osd":6,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.63200000000000001,"15min":0.65500000000000003},"min":{"1min":0.44700000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.86099999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.64900000000000002,"15min":0.66700000000000004},"min":{"1min":0.47599999999999998,"5min":0.42399999999999999,"15min":0.40999999999999998},"max":{"1min":0.83499999999999996,"5min":1.284,"15min":1.284},"last":0.48799999999999999}]},{"osd":7,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.64100000000000001,"15min":0.66300000000000003},"min":{"1min":0.42399999999999999,"5min":0.39300000000000002,"15min":0.34899999999999998},"max":{"1min":0.878,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64000000000000001,"15min":0.66400000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.38500000000000001},"max":{"1min":0.86599999999999999,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.5}]}]},{"osd":2,"up_from":13,"seq":55834575084,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":59272,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312568,"statfs":{"total":96636764160,"available":96576069632,"internally_reserved":0,"allocated":1957888,"data_stored":985405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34999999999999998,"15min":0.36799999999999999},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.498,"5min":0.66600000000000004,"15min":0.78300000000000003},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.44,"5min":0.44400000000000001,"15min":0.45500000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.217},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.77200000000000002},"last":0.39500000000000002}]},{"osd":1,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.435,"15min":0.46400000000000002},"min":{"1min":0.31,"5min":0.27300000000000002,"15min":0.27100000000000002},"max":{"1min":0.62,"5min":0.69499999999999995,"15min":0.90500000000000003},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.378,"5min":0.38400000000000001,"15min":0.39600000000000002},"min":{"1min":0.26700000000000002,"5min":0.22800000000000001,"15min":0.216},"max":{"1min":0.54300000000000004,"5min":0.64500000000000002,"15min":0.66700000000000004},"last":0.433}]},{"osd":3,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.24199999999999999},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47599999999999998,"15min":0.499},"min":{"1min":0.33400000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.80100000000000005},"last":0.45500000000000002}]},{"osd":4,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58399999999999996,"15min":0.60999999999999999},"min":{"1min":0.33500000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76100000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.59799999999999998,"15min":0.624},"min":{"1min":0.38600000000000001,"5min":0.378,"15min":0.32100000000000001},"max":{"1min":1.355,"5min":1.448,"15min":1.4730000000000001},"last":0.72499999999999998}]},{"osd":5,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.61899999999999999,"15min":0.64100000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.34699999999999998},"max":{"1min":0.90600000000000003,"5min":1.327,"15min":1.8109999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.59999999999999998,"15min":0.63100000000000001},"min":{"1min":0.376,"5min":0.36899999999999999,"15min":0.33400000000000002},"max":{"1min":0.75,"5min":1.4630000000000001,"15min":1.948},"last":0.751}]},{"osd":6,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.624,"15min":0.65500000000000003},"min":{"1min":0.42599999999999999,"5min":0.39600000000000002,"15min":0.35899999999999999},"max":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":2.9470000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65600000000000003,"15min":0.67900000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.38100000000000001},"max":{"1min":0.94899999999999995,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.60199999999999998}]},{"osd":7,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.625,"15min":0.64700000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.89100000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64000000000000001,"15min":0.66100000000000003},"min":{"1min":0.437,"5min":0.38300000000000001,"15min":0.371},"max":{"1min":1.409,"5min":1.482,"15min":1.482},"last":0.67200000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575084,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37132,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334708,"statfs":{"total":96636764160,"available":96598740992,"internally_reserved":0,"allocated":1957888,"data_stored":985405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34699999999999998,"15min":0.374},"min":{"1min":0.23499999999999999,"5min":0.221,"15min":0.21099999999999999},"max":{"1min":0.442,"5min":0.63900000000000001,"15min":1.204},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38,"15min":0.39900000000000002},"min":{"1min":0.25700000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.52900000000000003,"5min":0.66900000000000004,"15min":1.137},"last":0.23999999999999999}]},{"osd":1,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.36899999999999999,"15min":0.39500000000000002},"min":{"1min":0.27700000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.502,"5min":0.629,"15min":1.1220000000000001},"last":0.218},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40699999999999997,"15min":0.441},"min":{"1min":0.29499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.57399999999999995,"5min":0.76600000000000001,"15min":1.1080000000000001},"last":0.29599999999999999}]},{"osd":2,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.41799999999999998,"15min":0.44600000000000001},"min":{"1min":0.308,"5min":0.29499999999999998,"15min":0.28399999999999997},"max":{"1min":0.61799999999999999,"5min":0.67500000000000004,"15min":1.1879999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.44500000000000001,"15min":0.47299999999999998},"min":{"1min":0.33400000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.55600000000000005,"5min":0.71499999999999997,"15min":1.2290000000000001},"last":0.32700000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.58099999999999996,"15min":0.62},"min":{"1min":0.441,"5min":0.38100000000000001,"15min":0.34300000000000003},"max":{"1min":0.82899999999999996,"5min":1.052,"15min":1.871},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.60199999999999998,"15min":0.629},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.39600000000000002},"max":{"1min":0.81000000000000005,"5min":0.90600000000000003,"15min":1.764},"last":0.42899999999999999}]},{"osd":5,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.63400000000000001,"15min":0.66200000000000003},"min":{"1min":0.47099999999999997,"5min":0.39200000000000002,"15min":0.379},"max":{"1min":0.84999999999999998,"5min":1.0249999999999999,"15min":2.5329999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.65500000000000003,"15min":0.68200000000000005},"min":{"1min":0.43099999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.92900000000000005,"5min":0.94599999999999995,"15min":1.794},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.63900000000000001,"15min":0.67300000000000004},"min":{"1min":0.496,"5min":0.42499999999999999,"15min":0.41499999999999998},"max":{"1min":0.89400000000000002,"5min":1.1060000000000001,"15min":2.7389999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.65000000000000002,"15min":0.67600000000000005},"min":{"1min":0.45000000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.91200000000000003,"5min":0.98299999999999998,"15min":1.8069999999999999},"last":0.49199999999999999}]},{"osd":7,"last update":"Mon Mar 18 19:09:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63100000000000001,"15min":0.66400000000000003},"min":{"1min":0.43099999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":1.732},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.63400000000000001,"15min":0.66900000000000004},"min":{"1min":0.51800000000000002,"5min":0.45900000000000002,"15min":0.40799999999999997},"max":{"1min":0.86299999999999999,"5min":1.085,"15min":2.7050000000000001},"last":0.53300000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575084,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33016,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":1945600,"data_stored":979405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.56699999999999995,"15min":0.57499999999999996},"min":{"1min":0.43099999999999999,"5min":0.31,"15min":0.246},"max":{"1min":2.3210000000000002,"5min":2.3210000000000002,"15min":2.3210000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.41699999999999998,"5min":0.35299999999999998,"15min":0.26000000000000001},"max":{"1min":1.7430000000000001,"5min":3.645,"15min":3.645},"last":0.66500000000000004}]},{"osd":1,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65000000000000002,"15min":0.61199999999999999},"min":{"1min":0.40699999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.6739999999999999,"5min":9.9499999999999993,"15min":9.9499999999999993},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54400000000000004,"15min":0.57399999999999995},"min":{"1min":0.40200000000000002,"5min":0.32300000000000001,"15min":0.308},"max":{"1min":0.80000000000000004,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.57099999999999995}]},{"osd":2,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56699999999999995,"15min":0.57799999999999996},"min":{"1min":0.439,"5min":0.35899999999999999,"15min":0.28999999999999998},"max":{"1min":2.431,"5min":2.431,"15min":2.431},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.61499999999999999,"15min":0.61299999999999999},"min":{"1min":0.45700000000000002,"5min":0.35099999999999998,"15min":0.32600000000000001},"max":{"1min":2.165,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.53200000000000003}]},{"osd":3,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.55800000000000005,"15min":0.58299999999999996},"min":{"1min":0.39700000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76800000000000002,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.58799999999999997,"15min":0.59999999999999998},"min":{"1min":0.42599999999999999,"5min":0.39300000000000002,"15min":0.33900000000000002},"max":{"1min":2.5510000000000002,"5min":2.5510000000000002,"15min":2.5510000000000002},"last":0.48699999999999999}]},{"osd":5,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46300000000000002,"15min":0.46000000000000002},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.245},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.77500000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.51600000000000001,"15min":0.49199999999999999},"min":{"1min":0.307,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.73599999999999999,"5min":5.4900000000000002,"15min":5.4900000000000002},"last":0.50900000000000001}]},{"osd":6,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.49199999999999999,"15min":0.48499999999999999},"min":{"1min":0.33200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.5190000000000001,"5min":2.5190000000000001,"15min":2.5190000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.55100000000000005,"15min":0.52900000000000003},"min":{"1min":0.32600000000000001,"5min":0.317,"15min":0.28999999999999998},"max":{"1min":1.2689999999999999,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":0.626}]},{"osd":7,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52200000000000002,"15min":0.51900000000000002},"min":{"1min":0.34999999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.94599999999999995,"5min":1.55,"15min":1.55},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.54800000000000004,"15min":0.54900000000000004},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.67800000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51076,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320764,"statfs":{"total":96636764160,"available":96584462336,"internally_reserved":0,"allocated":1957888,"data_stored":985405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55400000000000005,"15min":0.56599999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.26600000000000001},"max":{"1min":0.76400000000000001,"5min":0.86099999999999999,"15min":1.1679999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55200000000000005,"15min":0.56599999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.25800000000000001},"max":{"1min":0.69699999999999995,"5min":0.80600000000000005,"15min":1.1319999999999999},"last":0.497}]},{"osd":1,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54800000000000004,"15min":0.56599999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.29599999999999999},"max":{"1min":0.78300000000000003,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.442},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.56100000000000005,"15min":0.57899999999999996},"min":{"1min":0.34599999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71999999999999997,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.53700000000000003}]},{"osd":2,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55300000000000005,"15min":0.57399999999999995},"min":{"1min":0.371,"5min":0.30399999999999999,"15min":0.28399999999999997},"max":{"1min":0.79700000000000004,"5min":1.153,"15min":1.1579999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55100000000000005,"15min":0.56899999999999995},"min":{"1min":0.377,"5min":0.35599999999999998,"15min":0.33200000000000002},"max":{"1min":0.71199999999999997,"5min":0.79500000000000004,"15min":1.0569999999999999},"last":0.56200000000000006}]},{"osd":3,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.39700000000000002,"5min":0.35799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72499999999999998,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57399999999999995,"15min":0.59599999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77000000000000002,"5min":1.212,"15min":1.212},"last":0.58799999999999997}]},{"osd":4,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.46300000000000002},"min":{"1min":0.34799999999999998,"5min":0.255,"15min":0.24199999999999999},"max":{"1min":0.64700000000000002,"5min":1.248,"15min":1.248},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.496,"15min":0.48899999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.23100000000000001},"max":{"1min":0.82599999999999996,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.73199999999999998}]},{"osd":6,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.29399999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.68000000000000005,"5min":0.78200000000000003,"15min":1.0069999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.496,"5min":0.51100000000000001,"15min":0.50900000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.26800000000000002},"max":{"1min":0.71799999999999997,"5min":0.83199999999999996,"15min":0.94599999999999995},"last":0.60899999999999999}]},{"osd":7,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.32700000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.85699999999999998,"5min":1.282,"15min":1.282},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.34200000000000003,"5min":0.28799999999999998,"15min":0.28499999999999998},"max":{"1min":0.81000000000000005,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.72199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T19:10:08.775 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T19:10:08.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.938+0000 7fe2c6fa7700 1 Processor -- start 2024-03-18T19:10:08.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.938+0000 7fe2c6fa7700 1 -- start start 2024-03-18T19:10:08.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.938+0000 7fe2c6fa7700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe2c0143c20 0x7fe2c00b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.938+0000 7fe2c6fa7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2c00b3f40 0x7fe2c006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.938+0000 7fe2c6fa7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c0144000 con 0x7fe2c006f070 2024-03-18T19:10:08.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.938+0000 7fe2c6fa7700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c00b43c0 con 0x7fe2c0143c20 2024-03-18T19:10:08.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.938+0000 7fe2c6fa7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c00b51b0 con 0x7fe2c00b3f40 2024-03-18T19:10:08.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2c5544700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe2c006f070 0x7fe2c0071650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52626/0 (socket says 172.21.15.1:52626) 2024-03-18T19:10:08.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2c5544700 1 -- 172.21.15.1:0/898175596 learned_addr learned my addr 172.21.15.1:0/898175596 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:10:08.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2bf7fe700 1 -- 172.21.15.1:0/898175596 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179307670 0 0) 0x7fe2c0144000 con 0x7fe2c006f070 2024-03-18T19:10:08.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2bffff700 1 --2- 172.21.15.1:0/898175596 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2c00b3f40 0x7fe2c006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2c4d43700 1 --2- 172.21.15.1:0/898175596 >> v2:172.21.15.165:3300/0 conn(0x7fe2c0143c20 0x7fe2c00b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2bf7fe700 1 -- 172.21.15.1:0/898175596 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b00035b0 con 0x7fe2c006f070 2024-03-18T19:10:08.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2bf7fe700 1 -- 172.21.15.1:0/898175596 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2382665385 0 0) 0x7fe2b00035b0 con 0x7fe2c006f070 2024-03-18T19:10:08.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2bf7fe700 1 -- 172.21.15.1:0/898175596 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2c00b3f40 msgr2=0x7fe2c006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:08.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2bf7fe700 1 --2- 172.21.15.1:0/898175596 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2c00b3f40 0x7fe2c006eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2bf7fe700 1 -- 172.21.15.1:0/898175596 >> v2:172.21.15.165:3300/0 conn(0x7fe2c0143c20 msgr2=0x7fe2c00b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:08.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2bf7fe700 1 --2- 172.21.15.1:0/898175596 >> v2:172.21.15.165:3300/0 conn(0x7fe2c0143c20 0x7fe2c00b3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2bf7fe700 1 -- 172.21.15.1:0/898175596 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c00cd8d0 con 0x7fe2c006f070 2024-03-18T19:10:08.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2bf7fe700 1 -- 172.21.15.1:0/898175596 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fe2b80019f0 con 0x7fe2c006f070 2024-03-18T19:10:08.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2bf7fe700 1 -- 172.21.15.1:0/898175596 >> v1:172.21.15.1:6789/0 conn(0x7fe2c006f070 legacy=0x7fe2c0071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:08.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2bf7fe700 1 --2- 172.21.15.1:0/898175596 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2c00b3f40 0x7fe2b0003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2bf7fe700 1 --2- 172.21.15.1:0/898175596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2b0003fb0 0x7fe2b0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2bf7fe700 1 --2- 172.21.15.1:0/898175596 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2b0004d90 0x7fe2b0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2bf7fe700 1 -- 172.21.15.1:0/898175596 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c00b43c0 con 0x7fe2b0003fb0 2024-03-18T19:10:08.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2bf7fe700 1 -- 172.21.15.1:0/898175596 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b000ba10 con 0x7fe2b0004d90 2024-03-18T19:10:08.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2bf7fe700 1 -- 172.21.15.1:0/898175596 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b000bb50 con 0x7fe2c00b3f40 2024-03-18T19:10:08.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2bffff700 1 --2- 172.21.15.1:0/898175596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2b0003fb0 0x7fe2b0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2c4d43700 1 --2- 172.21.15.1:0/898175596 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2c00b3f40 0x7fe2b0003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.939+0000 7fe2c5544700 1 --2- 172.21.15.1:0/898175596 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2b0004d90 0x7fe2b0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.940+0000 7fe2bffff700 1 -- 172.21.15.1:0/898175596 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2c00b3f40 msgr2=0x7fe2b0003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:08.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.940+0000 7fe2bffff700 1 --2- 172.21.15.1:0/898175596 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2c00b3f40 0x7fe2b0003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.940+0000 7fe2bffff700 1 -- 172.21.15.1:0/898175596 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2b0004d90 msgr2=0x7fe2b0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:08.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.940+0000 7fe2bffff700 1 --2- 172.21.15.1:0/898175596 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2b0004d90 0x7fe2b0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.940+0000 7fe2bffff700 1 -- 172.21.15.1:0/898175596 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b00035b0 con 0x7fe2b0003fb0 2024-03-18T19:10:08.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.940+0000 7fe2bffff700 1 --2- 172.21.15.1:0/898175596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2b0003fb0 0x7fe2b0004850 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac009b30 tx=0x7fe2b0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770a714940ff1e19 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:08.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.940+0000 7fe2bf7fe700 1 -- 172.21.15.1:0/898175596 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac00e6c0 con 0x7fe2b0003fb0 2024-03-18T19:10:08.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.940+0000 7fe2bf7fe700 1 -- 172.21.15.1:0/898175596 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ac00e820 con 0x7fe2b0003fb0 2024-03-18T19:10:08.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.940+0000 7fe2bf7fe700 1 -- 172.21.15.1:0/898175596 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac00eb90 con 0x7fe2b0003fb0 2024-03-18T19:10:08.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.940+0000 7fe2c6fa7700 1 -- 172.21.15.1:0/898175596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2b0003fb0 msgr2=0x7fe2b0004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:08.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.940+0000 7fe2c6fa7700 1 --2- 172.21.15.1:0/898175596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2b0003fb0 0x7fe2b0004850 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac009b30 tx=0x7fe2b0005880 comp rx=0 tx=0).stop 2024-03-18T19:10:08.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.941+0000 7fe2c5544700 1 -- 172.21.15.1:0/898175596 reap_dead start 2024-03-18T19:10:08.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.941+0000 7fe2c6fa7700 1 -- 172.21.15.1:0/898175596 shutdown_connections 2024-03-18T19:10:08.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.941+0000 7fe2c6fa7700 1 -- 172.21.15.1:0/898175596 >> 172.21.15.1:0/898175596 conn(0x7fe2c00b1d20 msgr2=0x7fe2c00b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:08.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.941+0000 7fe2c6fa7700 1 -- 172.21.15.1:0/898175596 shutdown_connections 2024-03-18T19:10:08.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.941+0000 7fe2c6fa7700 1 -- 172.21.15.1:0/898175596 wait complete. 2024-03-18T19:10:08.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.942+0000 7fe2c6fa7700 1 Processor -- start 2024-03-18T19:10:08.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.942+0000 7fe2c6fa7700 1 -- start start 2024-03-18T19:10:08.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2c6fa7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2b0003fb0 0x7fe2c01e7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2c6fa7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2c015c3c0 0x7fe2c015a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2c6fa7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c01e55c0 0x7fe2c01e5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2c6fa7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c01e5f80 con 0x7fe2c01e55c0 2024-03-18T19:10:08.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2c6fa7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c015cb30 con 0x7fe2c015c3c0 2024-03-18T19:10:08.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2c6fa7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c015ce10 con 0x7fe2b0003fb0 2024-03-18T19:10:08.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2c5544700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c01e55c0 0x7fe2c01e5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2c5544700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c01e55c0 0x7fe2c01e5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50924/0 (socket says 172.21.15.1:50924) 2024-03-18T19:10:08.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2c5544700 1 -- 172.21.15.1:0/2297086100 learned_addr learned my addr 172.21.15.1:0/2297086100 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:08.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2c4d43700 1 --2- 172.21.15.1:0/2297086100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2b0003fb0 0x7fe2c01e7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2bffff700 1 --2- 172.21.15.1:0/2297086100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2c015c3c0 0x7fe2c015a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2c5544700 1 -- 172.21.15.1:0/2297086100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2b0003fb0 msgr2=0x7fe2c01e7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:08.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2c5544700 1 --2- 172.21.15.1:0/2297086100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2b0003fb0 0x7fe2c01e7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2c5544700 1 -- 172.21.15.1:0/2297086100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2c015c3c0 msgr2=0x7fe2c015a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:08.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2c5544700 1 --2- 172.21.15.1:0/2297086100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2c015c3c0 0x7fe2c015a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:08.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2c5544700 1 -- 172.21.15.1:0/2297086100 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c015d0f0 con 0x7fe2c01e55c0 2024-03-18T19:10:08.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2bffff700 1 --2- 172.21.15.1:0/2297086100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2c015c3c0 0x7fe2c015a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:10:08.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2c5544700 1 --2- 172.21.15.1:0/2297086100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c01e55c0 0x7fe2c01e5a40 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8000f80 tx=0x7fe2b800a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:08.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2bd7fa700 1 -- 172.21.15.1:0/2297086100 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b8007b60 con 0x7fe2c01e55c0 2024-03-18T19:10:08.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2bd7fa700 1 -- 172.21.15.1:0/2297086100 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b800ad60 con 0x7fe2c01e55c0 2024-03-18T19:10:08.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2c6fa7700 1 -- 172.21.15.1:0/2297086100 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c01e3000 con 0x7fe2c01e55c0 2024-03-18T19:10:08.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.943+0000 7fe2bd7fa700 1 -- 172.21.15.1:0/2297086100 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b8013630 con 0x7fe2c01e55c0 2024-03-18T19:10:08.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.944+0000 7fe2c6fa7700 1 -- 172.21.15.1:0/2297086100 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c015a6a0 con 0x7fe2c01e55c0 2024-03-18T19:10:08.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.944+0000 7fe2c6fa7700 1 -- 172.21.15.1:0/2297086100 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c00676a0 con 0x7fe2c01e55c0 2024-03-18T19:10:08.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.945+0000 7fe2bd7fa700 1 -- 172.21.15.1:0/2297086100 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe2b8013790 con 0x7fe2c01e55c0 2024-03-18T19:10:08.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.946+0000 7fe2bd7fa700 1 --2- 172.21.15.1:0/2297086100 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe2a406ada0 0x7fe2a406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:08.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.946+0000 7fe2bd7fa700 1 -- 172.21.15.1:0/2297086100 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe2b8089d10 con 0x7fe2c01e55c0 2024-03-18T19:10:08.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.946+0000 7fe2c4d43700 1 --2- 172.21.15.1:0/2297086100 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe2a406ada0 0x7fe2a406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:08.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.946+0000 7fe2c4d43700 1 --2- 172.21.15.1:0/2297086100 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe2a406ada0 0x7fe2a406d260 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4009ce0 tx=0x7fe2b4009210 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:08.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:08.950+0000 7fe2bd7fa700 1 -- 172.21.15.1:0/2297086100 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe2b8054be0 con 0x7fe2c01e55c0 2024-03-18T19:10:09.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.149+0000 7fe2c6fa7700 1 -- 172.21.15.1:0/2297086100 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe2c01e2e80 con 0x7fe2a406ada0 2024-03-18T19:10:09.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.157+0000 7fe2bd7fa700 1 -- 172.21.15.1:0/2297086100 <== mgr.9153 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289799 (secure 0 0 0) 0x7fe2b4019070 con 0x7fe2a406ada0 2024-03-18T19:10:09.158 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:10:09.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.160+0000 7fe2c6fa7700 1 -- 172.21.15.1:0/2297086100 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe2a406ada0 msgr2=0x7fe2a406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:09.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.160+0000 7fe2c6fa7700 1 --2- 172.21.15.1:0/2297086100 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe2a406ada0 0x7fe2a406d260 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4009ce0 tx=0x7fe2b4009210 comp rx=0 tx=0).stop 2024-03-18T19:10:09.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.160+0000 7fe2c6fa7700 1 -- 172.21.15.1:0/2297086100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c01e55c0 msgr2=0x7fe2c01e5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:09.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.160+0000 7fe2c6fa7700 1 --2- 172.21.15.1:0/2297086100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c01e55c0 0x7fe2c01e5a40 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8000f80 tx=0x7fe2b800a5f0 comp rx=0 tx=0).stop 2024-03-18T19:10:09.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.161+0000 7fe2c6fa7700 1 -- 172.21.15.1:0/2297086100 shutdown_connections 2024-03-18T19:10:09.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.161+0000 7fe2c6fa7700 1 --2- 172.21.15.1:0/2297086100 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe2a406ada0 0x7fe2a406d260 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:09.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.161+0000 7fe2c6fa7700 1 --2- 172.21.15.1:0/2297086100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2b0003fb0 0x7fe2c01e7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:09.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.161+0000 7fe2c6fa7700 1 --2- 172.21.15.1:0/2297086100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe2c01e55c0 0x7fe2c01e5a40 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:09.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.161+0000 7fe2c6fa7700 1 --2- 172.21.15.1:0/2297086100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe2c015c3c0 0x7fe2c015a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:09.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.161+0000 7fe2c6fa7700 1 -- 172.21.15.1:0/2297086100 >> 172.21.15.1:0/2297086100 conn(0x7fe2c00b1d20 msgr2=0x7fe2c0142bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:09.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.161+0000 7fe2c6fa7700 1 -- 172.21.15.1:0/2297086100 shutdown_connections 2024-03-18T19:10:09.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.161+0000 7fe2c6fa7700 1 -- 172.21.15.1:0/2297086100 wait complete. 2024-03-18T19:10:09.161 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T19:10:09.171 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-18T19:10:09.083074+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":443,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":352776,"kb_used_data":16892,"kb_used_omap":12,"kb_used_meta":335731,"kb_avail":754621944,"statfs":{"total":773094113280,"available":772732870656,"internally_reserved":0,"allocated":17297408,"data_stored":9274460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":343789113},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.906542"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254804+0000","last_change":"2024-03-18T19:10:04.636915+0000","last_active":"2024-03-18T19:10:06.254804+0000","last_peered":"2024-03-18T19:10:06.254804+0000","last_clean":"2024-03-18T19:10:06.254804+0000","last_became_active":"2024-03-18T19:10:04.636620+0000","last_became_peered":"2024-03-18T19:10:04.636620+0000","last_unstale":"2024-03-18T19:10:06.254804+0000","last_undegraded":"2024-03-18T19:10:06.254804+0000","last_fullsized":"2024-03-18T19:10:06.254804+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:06:29.481080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.623136+0000","last_change":"2024-03-18T19:10:04.637603+0000","last_active":"2024-03-18T19:10:05.623136+0000","last_peered":"2024-03-18T19:10:05.623136+0000","last_clean":"2024-03-18T19:10:05.623136+0000","last_became_active":"2024-03-18T19:10:04.637244+0000","last_became_peered":"2024-03-18T19:10:04.637244+0000","last_unstale":"2024-03-18T19:10:05.623136+0000","last_undegraded":"2024-03-18T19:10:05.623136+0000","last_fullsized":"2024-03-18T19:10:05.623136+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:33:21.436046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.621653+0000","last_change":"2024-03-18T19:10:04.638844+0000","last_active":"2024-03-18T19:10:05.621653+0000","last_peered":"2024-03-18T19:10:05.621653+0000","last_clean":"2024-03-18T19:10:05.621653+0000","last_became_active":"2024-03-18T19:10:04.638280+0000","last_became_peered":"2024-03-18T19:10:04.638280+0000","last_unstale":"2024-03-18T19:10:05.621653+0000","last_undegraded":"2024-03-18T19:10:05.621653+0000","last_fullsized":"2024-03-18T19:10:05.621653+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:33:21.436046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.622566+0000","last_change":"2024-03-18T19:10:04.643106+0000","last_active":"2024-03-18T19:10:05.622566+0000","last_peered":"2024-03-18T19:10:05.622566+0000","last_clean":"2024-03-18T19:10:05.622566+0000","last_became_active":"2024-03-18T19:10:04.642772+0000","last_became_peered":"2024-03-18T19:10:04.642772+0000","last_unstale":"2024-03-18T19:10:05.622566+0000","last_undegraded":"2024-03-18T19:10:05.622566+0000","last_fullsized":"2024-03-18T19:10:05.622566+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:09:30.161494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.753086+0000","last_change":"2024-03-18T19:10:04.635787+0000","last_active":"2024-03-18T19:10:05.753086+0000","last_peered":"2024-03-18T19:10:05.753086+0000","last_clean":"2024-03-18T19:10:05.753086+0000","last_became_active":"2024-03-18T19:10:04.635521+0000","last_became_peered":"2024-03-18T19:10:04.635521+0000","last_unstale":"2024-03-18T19:10:05.753086+0000","last_undegraded":"2024-03-18T19:10:05.753086+0000","last_fullsized":"2024-03-18T19:10:05.753086+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:40:28.267465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253750+0000","last_change":"2024-03-18T19:10:04.628148+0000","last_active":"2024-03-18T19:10:06.253750+0000","last_peered":"2024-03-18T19:10:06.253750+0000","last_clean":"2024-03-18T19:10:06.253750+0000","last_became_active":"2024-03-18T19:10:04.627881+0000","last_became_peered":"2024-03-18T19:10:04.627881+0000","last_unstale":"2024-03-18T19:10:06.253750+0000","last_undegraded":"2024-03-18T19:10:06.253750+0000","last_fullsized":"2024-03-18T19:10:06.253750+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:20:15.737771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253585+0000","last_change":"2024-03-18T19:10:04.629276+0000","last_active":"2024-03-18T19:10:06.253585+0000","last_peered":"2024-03-18T19:10:06.253585+0000","last_clean":"2024-03-18T19:10:06.253585+0000","last_became_active":"2024-03-18T19:10:04.628975+0000","last_became_peered":"2024-03-18T19:10:04.628975+0000","last_unstale":"2024-03-18T19:10:06.253585+0000","last_undegraded":"2024-03-18T19:10:06.253585+0000","last_fullsized":"2024-03-18T19:10:06.253585+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:02:40.471904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254942+0000","last_change":"2024-03-18T19:10:04.640491+0000","last_active":"2024-03-18T19:10:06.254942+0000","last_peered":"2024-03-18T19:10:06.254942+0000","last_clean":"2024-03-18T19:10:06.254942+0000","last_became_active":"2024-03-18T19:10:04.640151+0000","last_became_peered":"2024-03-18T19:10:04.640151+0000","last_unstale":"2024-03-18T19:10:06.254942+0000","last_undegraded":"2024-03-18T19:10:06.254942+0000","last_fullsized":"2024-03-18T19:10:06.254942+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:41.506510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254977+0000","last_change":"2024-03-18T19:10:04.644642+0000","last_active":"2024-03-18T19:10:06.254977+0000","last_peered":"2024-03-18T19:10:06.254977+0000","last_clean":"2024-03-18T19:10:06.254977+0000","last_became_active":"2024-03-18T19:10:04.644457+0000","last_became_peered":"2024-03-18T19:10:04.644457+0000","last_unstale":"2024-03-18T19:10:06.254977+0000","last_undegraded":"2024-03-18T19:10:06.254977+0000","last_fullsized":"2024-03-18T19:10:06.254977+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:36.601312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254503+0000","last_change":"2024-03-18T19:10:04.638633+0000","last_active":"2024-03-18T19:10:06.254503+0000","last_peered":"2024-03-18T19:10:06.254503+0000","last_clean":"2024-03-18T19:10:06.254503+0000","last_became_active":"2024-03-18T19:10:04.638417+0000","last_became_peered":"2024-03-18T19:10:04.638417+0000","last_unstale":"2024-03-18T19:10:06.254503+0000","last_undegraded":"2024-03-18T19:10:06.254503+0000","last_fullsized":"2024-03-18T19:10:06.254503+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:20:50.677531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.622315+0000","last_change":"2024-03-18T19:10:04.641920+0000","last_active":"2024-03-18T19:10:05.622315+0000","last_peered":"2024-03-18T19:10:05.622315+0000","last_clean":"2024-03-18T19:10:05.622315+0000","last_became_active":"2024-03-18T19:10:04.641313+0000","last_became_peered":"2024-03-18T19:10:04.641313+0000","last_unstale":"2024-03-18T19:10:05.622315+0000","last_undegraded":"2024-03-18T19:10:05.622315+0000","last_fullsized":"2024-03-18T19:10:05.622315+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:47:30.693079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254730+0000","last_change":"2024-03-18T19:10:04.640770+0000","last_active":"2024-03-18T19:10:06.254730+0000","last_peered":"2024-03-18T19:10:06.254730+0000","last_clean":"2024-03-18T19:10:06.254730+0000","last_became_active":"2024-03-18T19:10:04.640305+0000","last_became_peered":"2024-03-18T19:10:04.640305+0000","last_unstale":"2024-03-18T19:10:06.254730+0000","last_undegraded":"2024-03-18T19:10:06.254730+0000","last_fullsized":"2024-03-18T19:10:06.254730+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:18:09.517219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254252+0000","last_change":"2024-03-18T19:10:04.630189+0000","last_active":"2024-03-18T19:10:06.254252+0000","last_peered":"2024-03-18T19:10:06.254252+0000","last_clean":"2024-03-18T19:10:06.254252+0000","last_became_active":"2024-03-18T19:10:04.629953+0000","last_became_peered":"2024-03-18T19:10:04.629953+0000","last_unstale":"2024-03-18T19:10:06.254252+0000","last_undegraded":"2024-03-18T19:10:06.254252+0000","last_fullsized":"2024-03-18T19:10:06.254252+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:11:37.835302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254493+0000","last_change":"2024-03-18T19:10:04.633203+0000","last_active":"2024-03-18T19:10:06.254493+0000","last_peered":"2024-03-18T19:10:06.254493+0000","last_clean":"2024-03-18T19:10:06.254493+0000","last_became_active":"2024-03-18T19:10:04.632907+0000","last_became_peered":"2024-03-18T19:10:04.632907+0000","last_unstale":"2024-03-18T19:10:06.254493+0000","last_undegraded":"2024-03-18T19:10:06.254493+0000","last_fullsized":"2024-03-18T19:10:06.254493+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:45:59.271014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254843+0000","last_change":"2024-03-18T19:10:04.642468+0000","last_active":"2024-03-18T19:10:06.254843+0000","last_peered":"2024-03-18T19:10:06.254843+0000","last_clean":"2024-03-18T19:10:06.254843+0000","last_became_active":"2024-03-18T19:10:04.642233+0000","last_became_peered":"2024-03-18T19:10:04.642233+0000","last_unstale":"2024-03-18T19:10:06.254843+0000","last_undegraded":"2024-03-18T19:10:06.254843+0000","last_fullsized":"2024-03-18T19:10:06.254843+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:10:54.744144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253565+0000","last_change":"2024-03-18T19:10:04.640038+0000","last_active":"2024-03-18T19:10:06.253565+0000","last_peered":"2024-03-18T19:10:06.253565+0000","last_clean":"2024-03-18T19:10:06.253565+0000","last_became_active":"2024-03-18T19:10:04.639621+0000","last_became_peered":"2024-03-18T19:10:04.639621+0000","last_unstale":"2024-03-18T19:10:06.253565+0000","last_undegraded":"2024-03-18T19:10:06.253565+0000","last_fullsized":"2024-03-18T19:10:06.253565+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:41.909132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.255069+0000","last_change":"2024-03-18T19:10:04.638251+0000","last_active":"2024-03-18T19:10:06.255069+0000","last_peered":"2024-03-18T19:10:06.255069+0000","last_clean":"2024-03-18T19:10:06.255069+0000","last_became_active":"2024-03-18T19:10:04.636850+0000","last_became_peered":"2024-03-18T19:10:04.636850+0000","last_unstale":"2024-03-18T19:10:06.255069+0000","last_undegraded":"2024-03-18T19:10:06.255069+0000","last_fullsized":"2024-03-18T19:10:06.255069+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:27:58.639356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.753267+0000","last_change":"2024-03-18T19:10:04.635486+0000","last_active":"2024-03-18T19:10:05.753267+0000","last_peered":"2024-03-18T19:10:05.753267+0000","last_clean":"2024-03-18T19:10:05.753267+0000","last_became_active":"2024-03-18T19:10:04.635223+0000","last_became_peered":"2024-03-18T19:10:04.635223+0000","last_unstale":"2024-03-18T19:10:05.753267+0000","last_undegraded":"2024-03-18T19:10:05.753267+0000","last_fullsized":"2024-03-18T19:10:05.753267+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:20.649018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254054+0000","last_change":"2024-03-18T19:10:04.636481+0000","last_active":"2024-03-18T19:10:06.254054+0000","last_peered":"2024-03-18T19:10:06.254054+0000","last_clean":"2024-03-18T19:10:06.254054+0000","last_became_active":"2024-03-18T19:10:04.632303+0000","last_became_peered":"2024-03-18T19:10:04.632303+0000","last_unstale":"2024-03-18T19:10:06.254054+0000","last_undegraded":"2024-03-18T19:10:06.254054+0000","last_fullsized":"2024-03-18T19:10:06.254054+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:47:30.693079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.753149+0000","last_change":"2024-03-18T19:10:04.643639+0000","last_active":"2024-03-18T19:10:05.753149+0000","last_peered":"2024-03-18T19:10:05.753149+0000","last_clean":"2024-03-18T19:10:05.753149+0000","last_became_active":"2024-03-18T19:10:04.643444+0000","last_became_peered":"2024-03-18T19:10:04.643444+0000","last_unstale":"2024-03-18T19:10:05.753149+0000","last_undegraded":"2024-03-18T19:10:05.753149+0000","last_fullsized":"2024-03-18T19:10:05.753149+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:51:17.410504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.622103+0000","last_change":"2024-03-18T19:10:04.646026+0000","last_active":"2024-03-18T19:10:05.622103+0000","last_peered":"2024-03-18T19:10:05.622103+0000","last_clean":"2024-03-18T19:10:05.622103+0000","last_became_active":"2024-03-18T19:10:04.645809+0000","last_became_peered":"2024-03-18T19:10:04.645809+0000","last_unstale":"2024-03-18T19:10:05.622103+0000","last_undegraded":"2024-03-18T19:10:05.622103+0000","last_fullsized":"2024-03-18T19:10:05.622103+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:10.396783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254624+0000","last_change":"2024-03-18T19:10:04.636267+0000","last_active":"2024-03-18T19:10:06.254624+0000","last_peered":"2024-03-18T19:10:06.254624+0000","last_clean":"2024-03-18T19:10:06.254624+0000","last_became_active":"2024-03-18T19:10:04.636023+0000","last_became_peered":"2024-03-18T19:10:04.636023+0000","last_unstale":"2024-03-18T19:10:06.254624+0000","last_undegraded":"2024-03-18T19:10:06.254624+0000","last_fullsized":"2024-03-18T19:10:06.254624+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:03:58.941806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.622254+0000","last_change":"2024-03-18T19:10:04.643118+0000","last_active":"2024-03-18T19:10:05.622254+0000","last_peered":"2024-03-18T19:10:05.622254+0000","last_clean":"2024-03-18T19:10:05.622254+0000","last_became_active":"2024-03-18T19:10:04.642849+0000","last_became_peered":"2024-03-18T19:10:04.642849+0000","last_unstale":"2024-03-18T19:10:05.622254+0000","last_undegraded":"2024-03-18T19:10:05.622254+0000","last_fullsized":"2024-03-18T19:10:05.622254+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:47:12.822164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253673+0000","last_change":"2024-03-18T19:10:04.639879+0000","last_active":"2024-03-18T19:10:06.253673+0000","last_peered":"2024-03-18T19:10:06.253673+0000","last_clean":"2024-03-18T19:10:06.253673+0000","last_became_active":"2024-03-18T19:10:04.639333+0000","last_became_peered":"2024-03-18T19:10:04.639333+0000","last_unstale":"2024-03-18T19:10:06.253673+0000","last_undegraded":"2024-03-18T19:10:06.253673+0000","last_fullsized":"2024-03-18T19:10:06.253673+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:02:40.471904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.621837+0000","last_change":"2024-03-18T19:10:04.630104+0000","last_active":"2024-03-18T19:10:05.621837+0000","last_peered":"2024-03-18T19:10:05.621837+0000","last_clean":"2024-03-18T19:10:05.621837+0000","last_became_active":"2024-03-18T19:10:04.629803+0000","last_became_peered":"2024-03-18T19:10:04.629803+0000","last_unstale":"2024-03-18T19:10:05.621837+0000","last_undegraded":"2024-03-18T19:10:05.621837+0000","last_fullsized":"2024-03-18T19:10:05.621837+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:08:39.593581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.621670+0000","last_change":"2024-03-18T19:10:04.635213+0000","last_active":"2024-03-18T19:10:05.621670+0000","last_peered":"2024-03-18T19:10:05.621670+0000","last_clean":"2024-03-18T19:10:05.621670+0000","last_became_active":"2024-03-18T19:10:04.634945+0000","last_became_peered":"2024-03-18T19:10:04.634945+0000","last_unstale":"2024-03-18T19:10:05.621670+0000","last_undegraded":"2024-03-18T19:10:05.621670+0000","last_fullsized":"2024-03-18T19:10:05.621670+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:01:17.075265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253760+0000","last_change":"2024-03-18T19:10:04.623800+0000","last_active":"2024-03-18T19:10:06.253760+0000","last_peered":"2024-03-18T19:10:06.253760+0000","last_clean":"2024-03-18T19:10:06.253760+0000","last_became_active":"2024-03-18T19:10:04.623518+0000","last_became_peered":"2024-03-18T19:10:04.623518+0000","last_unstale":"2024-03-18T19:10:06.253760+0000","last_undegraded":"2024-03-18T19:10:06.253760+0000","last_fullsized":"2024-03-18T19:10:06.253760+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:20:15.737771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253996+0000","last_change":"2024-03-18T19:10:04.638493+0000","last_active":"2024-03-18T19:10:06.253996+0000","last_peered":"2024-03-18T19:10:06.253996+0000","last_clean":"2024-03-18T19:10:06.253996+0000","last_became_active":"2024-03-18T19:10:04.638236+0000","last_became_peered":"2024-03-18T19:10:04.638236+0000","last_unstale":"2024-03-18T19:10:06.253996+0000","last_undegraded":"2024-03-18T19:10:06.253996+0000","last_fullsized":"2024-03-18T19:10:06.253996+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:15:11.156679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.621500+0000","last_change":"2024-03-18T19:10:04.623483+0000","last_active":"2024-03-18T19:10:05.621500+0000","last_peered":"2024-03-18T19:10:05.621500+0000","last_clean":"2024-03-18T19:10:05.621500+0000","last_became_active":"2024-03-18T19:10:04.623032+0000","last_became_peered":"2024-03-18T19:10:04.623032+0000","last_unstale":"2024-03-18T19:10:05.621500+0000","last_undegraded":"2024-03-18T19:10:05.621500+0000","last_fullsized":"2024-03-18T19:10:05.621500+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:10.621778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.621555+0000","last_change":"2024-03-18T19:10:04.629481+0000","last_active":"2024-03-18T19:10:05.621555+0000","last_peered":"2024-03-18T19:10:05.621555+0000","last_clean":"2024-03-18T19:10:05.621555+0000","last_became_active":"2024-03-18T19:10:04.629254+0000","last_became_peered":"2024-03-18T19:10:04.629254+0000","last_unstale":"2024-03-18T19:10:05.621555+0000","last_undegraded":"2024-03-18T19:10:05.621555+0000","last_fullsized":"2024-03-18T19:10:05.621555+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:17.101429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254493+0000","last_change":"2024-03-18T19:10:04.633355+0000","last_active":"2024-03-18T19:10:06.254493+0000","last_peered":"2024-03-18T19:10:06.254493+0000","last_clean":"2024-03-18T19:10:06.254493+0000","last_became_active":"2024-03-18T19:10:04.633054+0000","last_became_peered":"2024-03-18T19:10:04.633054+0000","last_unstale":"2024-03-18T19:10:06.254493+0000","last_undegraded":"2024-03-18T19:10:06.254493+0000","last_fullsized":"2024-03-18T19:10:06.254493+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:47:30.693079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253629+0000","last_change":"2024-03-18T19:10:04.643440+0000","last_active":"2024-03-18T19:10:06.253629+0000","last_peered":"2024-03-18T19:10:06.253629+0000","last_clean":"2024-03-18T19:10:06.253629+0000","last_became_active":"2024-03-18T19:10:04.643163+0000","last_became_peered":"2024-03-18T19:10:04.643163+0000","last_unstale":"2024-03-18T19:10:06.253629+0000","last_undegraded":"2024-03-18T19:10:06.253629+0000","last_fullsized":"2024-03-18T19:10:06.253629+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:01:06.245452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254533+0000","last_change":"2024-03-18T19:10:04.632716+0000","last_active":"2024-03-18T19:10:06.254533+0000","last_peered":"2024-03-18T19:10:06.254533+0000","last_clean":"2024-03-18T19:10:06.254533+0000","last_became_active":"2024-03-18T19:10:04.632431+0000","last_became_peered":"2024-03-18T19:10:04.632431+0000","last_unstale":"2024-03-18T19:10:06.254533+0000","last_undegraded":"2024-03-18T19:10:06.254533+0000","last_fullsized":"2024-03-18T19:10:06.254533+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:47:12.822164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"160'4","reported_seq":15,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.622001+0000","last_change":"2024-03-18T19:10:04.201180+0000","last_active":"2024-03-18T19:10:05.622001+0000","last_peered":"2024-03-18T19:10:05.622001+0000","last_clean":"2024-03-18T19:10:05.622001+0000","last_became_active":"2024-03-18T19:10:04.200880+0000","last_became_peered":"2024-03-18T19:10:04.200880+0000","last_unstale":"2024-03-18T19:10:05.622001+0000","last_undegraded":"2024-03-18T19:10:05.622001+0000","last_fullsized":"2024-03-18T19:10:05.622001+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:01:05.807577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254470+0000","last_change":"2024-03-18T19:10:04.635706+0000","last_active":"2024-03-18T19:10:06.254470+0000","last_peered":"2024-03-18T19:10:06.254470+0000","last_clean":"2024-03-18T19:10:06.254470+0000","last_became_active":"2024-03-18T19:10:04.635458+0000","last_became_peered":"2024-03-18T19:10:04.635458+0000","last_unstale":"2024-03-18T19:10:06.254470+0000","last_undegraded":"2024-03-18T19:10:06.254470+0000","last_fullsized":"2024-03-18T19:10:06.254470+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:04.707796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"160'1","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.753366+0000","last_change":"2024-03-18T19:10:04.195353+0000","last_active":"2024-03-18T19:10:05.753366+0000","last_peered":"2024-03-18T19:10:05.753366+0000","last_clean":"2024-03-18T19:10:05.753366+0000","last_became_active":"2024-03-18T19:10:04.195083+0000","last_became_peered":"2024-03-18T19:10:04.195083+0000","last_unstale":"2024-03-18T19:10:05.753366+0000","last_undegraded":"2024-03-18T19:10:05.753366+0000","last_fullsized":"2024-03-18T19:10:05.753366+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:03:25.683638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.622403+0000","last_change":"2024-03-18T19:10:04.643370+0000","last_active":"2024-03-18T19:10:05.622403+0000","last_peered":"2024-03-18T19:10:05.622403+0000","last_clean":"2024-03-18T19:10:05.622403+0000","last_became_active":"2024-03-18T19:10:04.642828+0000","last_became_peered":"2024-03-18T19:10:04.642828+0000","last_unstale":"2024-03-18T19:10:05.622403+0000","last_undegraded":"2024-03-18T19:10:05.622403+0000","last_fullsized":"2024-03-18T19:10:05.622403+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:46.740895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"160'2","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254049+0000","last_change":"2024-03-18T19:10:04.200336+0000","last_active":"2024-03-18T19:10:06.254049+0000","last_peered":"2024-03-18T19:10:06.254049+0000","last_clean":"2024-03-18T19:10:06.254049+0000","last_became_active":"2024-03-18T19:10:04.200105+0000","last_became_peered":"2024-03-18T19:10:04.200105+0000","last_unstale":"2024-03-18T19:10:06.254049+0000","last_undegraded":"2024-03-18T19:10:06.254049+0000","last_fullsized":"2024-03-18T19:10:06.254049+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:09.616522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254768+0000","last_change":"2024-03-18T19:10:04.642958+0000","last_active":"2024-03-18T19:10:06.254768+0000","last_peered":"2024-03-18T19:10:06.254768+0000","last_clean":"2024-03-18T19:10:06.254768+0000","last_became_active":"2024-03-18T19:10:04.642715+0000","last_became_peered":"2024-03-18T19:10:04.642715+0000","last_unstale":"2024-03-18T19:10:06.254768+0000","last_undegraded":"2024-03-18T19:10:06.254768+0000","last_fullsized":"2024-03-18T19:10:06.254768+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:34:11.318435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"160'2","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253914+0000","last_change":"2024-03-18T19:10:04.197457+0000","last_active":"2024-03-18T19:10:06.253914+0000","last_peered":"2024-03-18T19:10:06.253914+0000","last_clean":"2024-03-18T19:10:06.253914+0000","last_became_active":"2024-03-18T19:10:04.197160+0000","last_became_peered":"2024-03-18T19:10:04.197160+0000","last_unstale":"2024-03-18T19:10:06.253914+0000","last_undegraded":"2024-03-18T19:10:06.253914+0000","last_fullsized":"2024-03-18T19:10:06.253914+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:57:05.031861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254872+0000","last_change":"2024-03-18T19:10:04.634798+0000","last_active":"2024-03-18T19:10:06.254872+0000","last_peered":"2024-03-18T19:10:06.254872+0000","last_clean":"2024-03-18T19:10:06.254872+0000","last_became_active":"2024-03-18T19:10:04.634537+0000","last_became_peered":"2024-03-18T19:10:04.634537+0000","last_unstale":"2024-03-18T19:10:06.254872+0000","last_undegraded":"2024-03-18T19:10:06.254872+0000","last_fullsized":"2024-03-18T19:10:06.254872+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:10.396783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"160'1","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.255022+0000","last_change":"2024-03-18T19:10:04.197699+0000","last_active":"2024-03-18T19:10:06.255022+0000","last_peered":"2024-03-18T19:10:06.255022+0000","last_clean":"2024-03-18T19:10:06.255022+0000","last_became_active":"2024-03-18T19:10:04.197422+0000","last_became_peered":"2024-03-18T19:10:04.197422+0000","last_unstale":"2024-03-18T19:10:06.255022+0000","last_undegraded":"2024-03-18T19:10:06.255022+0000","last_fullsized":"2024-03-18T19:10:06.255022+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:11:24.855403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254656+0000","last_change":"2024-03-18T19:10:04.640564+0000","last_active":"2024-03-18T19:10:06.254656+0000","last_peered":"2024-03-18T19:10:06.254656+0000","last_clean":"2024-03-18T19:10:06.254656+0000","last_became_active":"2024-03-18T19:10:04.640139+0000","last_became_peered":"2024-03-18T19:10:04.640139+0000","last_unstale":"2024-03-18T19:10:06.254656+0000","last_undegraded":"2024-03-18T19:10:06.254656+0000","last_fullsized":"2024-03-18T19:10:06.254656+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:46.740895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"160'1","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.753088+0000","last_change":"2024-03-18T19:10:04.198717+0000","last_active":"2024-03-18T19:10:05.753088+0000","last_peered":"2024-03-18T19:10:05.753088+0000","last_clean":"2024-03-18T19:10:05.753088+0000","last_became_active":"2024-03-18T19:10:04.198474+0000","last_became_peered":"2024-03-18T19:10:04.198474+0000","last_unstale":"2024-03-18T19:10:05.753088+0000","last_undegraded":"2024-03-18T19:10:05.753088+0000","last_fullsized":"2024-03-18T19:10:05.753088+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:45.449344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.753267+0000","last_change":"2024-03-18T19:10:04.646880+0000","last_active":"2024-03-18T19:10:05.753267+0000","last_peered":"2024-03-18T19:10:05.753267+0000","last_clean":"2024-03-18T19:10:05.753267+0000","last_became_active":"2024-03-18T19:10:04.646588+0000","last_became_peered":"2024-03-18T19:10:04.646588+0000","last_unstale":"2024-03-18T19:10:05.753267+0000","last_undegraded":"2024-03-18T19:10:05.753267+0000","last_fullsized":"2024-03-18T19:10:05.753267+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:52:15.112143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254126+0000","last_change":"2024-03-18T19:10:04.196553+0000","last_active":"2024-03-18T19:10:06.254126+0000","last_peered":"2024-03-18T19:10:06.254126+0000","last_clean":"2024-03-18T19:10:06.254126+0000","last_became_active":"2024-03-18T19:10:04.196259+0000","last_became_peered":"2024-03-18T19:10:04.196259+0000","last_unstale":"2024-03-18T19:10:06.254126+0000","last_undegraded":"2024-03-18T19:10:06.254126+0000","last_fullsized":"2024-03-18T19:10:06.254126+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:08:48.452699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254153+0000","last_change":"2024-03-18T19:10:04.624942+0000","last_active":"2024-03-18T19:10:06.254153+0000","last_peered":"2024-03-18T19:10:06.254153+0000","last_clean":"2024-03-18T19:10:06.254153+0000","last_became_active":"2024-03-18T19:10:04.624653+0000","last_became_peered":"2024-03-18T19:10:04.624653+0000","last_unstale":"2024-03-18T19:10:06.254153+0000","last_undegraded":"2024-03-18T19:10:06.254153+0000","last_fullsized":"2024-03-18T19:10:06.254153+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:41.506510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.622101+0000","last_change":"2024-03-18T19:10:04.198493+0000","last_active":"2024-03-18T19:10:05.622101+0000","last_peered":"2024-03-18T19:10:05.622101+0000","last_clean":"2024-03-18T19:10:05.622101+0000","last_became_active":"2024-03-18T19:10:04.198224+0000","last_became_peered":"2024-03-18T19:10:04.198224+0000","last_unstale":"2024-03-18T19:10:05.622101+0000","last_undegraded":"2024-03-18T19:10:05.622101+0000","last_fullsized":"2024-03-18T19:10:05.622101+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:45:58.265758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254378+0000","last_change":"2024-03-18T19:10:04.629789+0000","last_active":"2024-03-18T19:10:06.254378+0000","last_peered":"2024-03-18T19:10:06.254378+0000","last_clean":"2024-03-18T19:10:06.254378+0000","last_became_active":"2024-03-18T19:10:04.629518+0000","last_became_peered":"2024-03-18T19:10:04.629518+0000","last_unstale":"2024-03-18T19:10:06.254378+0000","last_undegraded":"2024-03-18T19:10:06.254378+0000","last_fullsized":"2024-03-18T19:10:06.254378+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:05.055829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.622272+0000","last_change":"2024-03-18T19:10:04.199577+0000","last_active":"2024-03-18T19:10:05.622272+0000","last_peered":"2024-03-18T19:10:05.622272+0000","last_clean":"2024-03-18T19:10:05.622272+0000","last_became_active":"2024-03-18T19:10:04.199351+0000","last_became_peered":"2024-03-18T19:10:04.199351+0000","last_unstale":"2024-03-18T19:10:05.622272+0000","last_undegraded":"2024-03-18T19:10:05.622272+0000","last_fullsized":"2024-03-18T19:10:05.622272+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:30:07.307315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253841+0000","last_change":"2024-03-18T19:10:04.635217+0000","last_active":"2024-03-18T19:10:06.253841+0000","last_peered":"2024-03-18T19:10:06.253841+0000","last_clean":"2024-03-18T19:10:06.253841+0000","last_became_active":"2024-03-18T19:10:04.635017+0000","last_became_peered":"2024-03-18T19:10:04.635017+0000","last_unstale":"2024-03-18T19:10:06.253841+0000","last_undegraded":"2024-03-18T19:10:06.253841+0000","last_fullsized":"2024-03-18T19:10:06.253841+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:41:46.475185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254468+0000","last_change":"2024-03-18T19:10:04.198076+0000","last_active":"2024-03-18T19:10:06.254468+0000","last_peered":"2024-03-18T19:10:06.254468+0000","last_clean":"2024-03-18T19:10:06.254468+0000","last_became_active":"2024-03-18T19:10:04.197819+0000","last_became_peered":"2024-03-18T19:10:04.197819+0000","last_unstale":"2024-03-18T19:10:06.254468+0000","last_undegraded":"2024-03-18T19:10:06.254468+0000","last_fullsized":"2024-03-18T19:10:06.254468+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:27.514031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.621667+0000","last_change":"2024-03-18T19:10:04.638287+0000","last_active":"2024-03-18T19:10:05.621667+0000","last_peered":"2024-03-18T19:10:05.621667+0000","last_clean":"2024-03-18T19:10:05.621667+0000","last_became_active":"2024-03-18T19:10:04.637773+0000","last_became_peered":"2024-03-18T19:10:04.637773+0000","last_unstale":"2024-03-18T19:10:05.621667+0000","last_undegraded":"2024-03-18T19:10:05.621667+0000","last_fullsized":"2024-03-18T19:10:05.621667+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:11:25.860659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"160'3","reported_seq":14,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254165+0000","last_change":"2024-03-18T19:10:04.200205+0000","last_active":"2024-03-18T19:10:06.254165+0000","last_peered":"2024-03-18T19:10:06.254165+0000","last_clean":"2024-03-18T19:10:06.254165+0000","last_became_active":"2024-03-18T19:10:04.199937+0000","last_became_peered":"2024-03-18T19:10:04.199937+0000","last_unstale":"2024-03-18T19:10:06.254165+0000","last_undegraded":"2024-03-18T19:10:06.254165+0000","last_fullsized":"2024-03-18T19:10:06.254165+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:08:48.452699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.621613+0000","last_change":"2024-03-18T19:10:04.644205+0000","last_active":"2024-03-18T19:10:05.621613+0000","last_peered":"2024-03-18T19:10:05.621613+0000","last_clean":"2024-03-18T19:10:05.621613+0000","last_became_active":"2024-03-18T19:10:04.643771+0000","last_became_peered":"2024-03-18T19:10:04.643771+0000","last_unstale":"2024-03-18T19:10:05.621613+0000","last_undegraded":"2024-03-18T19:10:05.621613+0000","last_fullsized":"2024-03-18T19:10:05.621613+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:48:40.236655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"160'6","reported_seq":17,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253947+0000","last_change":"2024-03-18T19:10:04.199066+0000","last_active":"2024-03-18T19:10:06.253947+0000","last_peered":"2024-03-18T19:10:06.253947+0000","last_clean":"2024-03-18T19:10:06.253947+0000","last_became_active":"2024-03-18T19:10:04.198773+0000","last_became_peered":"2024-03-18T19:10:04.198773+0000","last_unstale":"2024-03-18T19:10:06.253947+0000","last_undegraded":"2024-03-18T19:10:06.253947+0000","last_fullsized":"2024-03-18T19:10:06.253947+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:16.096174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254091+0000","last_change":"2024-03-18T19:10:04.643449+0000","last_active":"2024-03-18T19:10:06.254091+0000","last_peered":"2024-03-18T19:10:06.254091+0000","last_clean":"2024-03-18T19:10:06.254091+0000","last_became_active":"2024-03-18T19:10:04.643172+0000","last_became_peered":"2024-03-18T19:10:04.643172+0000","last_unstale":"2024-03-18T19:10:06.254091+0000","last_undegraded":"2024-03-18T19:10:06.254091+0000","last_fullsized":"2024-03-18T19:10:06.254091+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:46:24.784733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"160'2","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254849+0000","last_change":"2024-03-18T19:10:04.198591+0000","last_active":"2024-03-18T19:10:06.254849+0000","last_peered":"2024-03-18T19:10:06.254849+0000","last_clean":"2024-03-18T19:10:06.254849+0000","last_became_active":"2024-03-18T19:10:04.198318+0000","last_became_peered":"2024-03-18T19:10:04.198318+0000","last_unstale":"2024-03-18T19:10:06.254849+0000","last_undegraded":"2024-03-18T19:10:06.254849+0000","last_fullsized":"2024-03-18T19:10:06.254849+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:29:16.274406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254204+0000","last_change":"2024-03-18T19:10:04.637200+0000","last_active":"2024-03-18T19:10:06.254204+0000","last_peered":"2024-03-18T19:10:06.254204+0000","last_clean":"2024-03-18T19:10:06.254204+0000","last_became_active":"2024-03-18T19:10:04.636694+0000","last_became_peered":"2024-03-18T19:10:04.636694+0000","last_unstale":"2024-03-18T19:10:06.254204+0000","last_undegraded":"2024-03-18T19:10:06.254204+0000","last_fullsized":"2024-03-18T19:10:06.254204+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:46.740895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.621471+0000","last_change":"2024-03-18T19:10:04.194624+0000","last_active":"2024-03-18T19:10:05.621471+0000","last_peered":"2024-03-18T19:10:05.621471+0000","last_clean":"2024-03-18T19:10:05.621471+0000","last_became_active":"2024-03-18T19:10:04.194407+0000","last_became_peered":"2024-03-18T19:10:04.194407+0000","last_unstale":"2024-03-18T19:10:05.621471+0000","last_undegraded":"2024-03-18T19:10:05.621471+0000","last_fullsized":"2024-03-18T19:10:05.621471+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:18:08.511963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.753353+0000","last_change":"2024-03-18T19:10:04.638474+0000","last_active":"2024-03-18T19:10:05.753353+0000","last_peered":"2024-03-18T19:10:05.753353+0000","last_clean":"2024-03-18T19:10:05.753353+0000","last_became_active":"2024-03-18T19:10:04.638208+0000","last_became_peered":"2024-03-18T19:10:04.638208+0000","last_unstale":"2024-03-18T19:10:05.753353+0000","last_undegraded":"2024-03-18T19:10:05.753353+0000","last_fullsized":"2024-03-18T19:10:05.753353+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:44:05.711129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"160'8","reported_seq":19,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253775+0000","last_change":"2024-03-18T19:10:04.202116+0000","last_active":"2024-03-18T19:10:06.253775+0000","last_peered":"2024-03-18T19:10:06.253775+0000","last_clean":"2024-03-18T19:10:06.253775+0000","last_became_active":"2024-03-18T19:10:04.201805+0000","last_became_peered":"2024-03-18T19:10:04.201805+0000","last_unstale":"2024-03-18T19:10:06.253775+0000","last_undegraded":"2024-03-18T19:10:06.253775+0000","last_fullsized":"2024-03-18T19:10:06.253775+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:55.480457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.622021+0000","last_change":"2024-03-18T19:10:04.638960+0000","last_active":"2024-03-18T19:10:05.622021+0000","last_peered":"2024-03-18T19:10:05.622021+0000","last_clean":"2024-03-18T19:10:05.622021+0000","last_became_active":"2024-03-18T19:10:04.638721+0000","last_became_peered":"2024-03-18T19:10:04.638721+0000","last_unstale":"2024-03-18T19:10:05.622021+0000","last_undegraded":"2024-03-18T19:10:05.622021+0000","last_fullsized":"2024-03-18T19:10:05.622021+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:08:11.942546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"160'1","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.622378+0000","last_change":"2024-03-18T19:10:04.196903+0000","last_active":"2024-03-18T19:10:05.622378+0000","last_peered":"2024-03-18T19:10:05.622378+0000","last_clean":"2024-03-18T19:10:05.622378+0000","last_became_active":"2024-03-18T19:10:04.196613+0000","last_became_peered":"2024-03-18T19:10:04.196613+0000","last_unstale":"2024-03-18T19:10:05.622378+0000","last_undegraded":"2024-03-18T19:10:05.622378+0000","last_fullsized":"2024-03-18T19:10:05.622378+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:08:48.452699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253824+0000","last_change":"2024-03-18T19:10:04.644492+0000","last_active":"2024-03-18T19:10:06.253824+0000","last_peered":"2024-03-18T19:10:06.253824+0000","last_clean":"2024-03-18T19:10:06.253824+0000","last_became_active":"2024-03-18T19:10:04.644009+0000","last_became_peered":"2024-03-18T19:10:04.644009+0000","last_unstale":"2024-03-18T19:10:06.253824+0000","last_undegraded":"2024-03-18T19:10:06.253824+0000","last_fullsized":"2024-03-18T19:10:06.253824+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:11:37.835302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"160'2","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.621764+0000","last_change":"2024-03-18T19:10:04.198735+0000","last_active":"2024-03-18T19:10:05.621764+0000","last_peered":"2024-03-18T19:10:05.621764+0000","last_clean":"2024-03-18T19:10:05.621764+0000","last_became_active":"2024-03-18T19:10:04.198433+0000","last_became_peered":"2024-03-18T19:10:04.198433+0000","last_unstale":"2024-03-18T19:10:05.621764+0000","last_undegraded":"2024-03-18T19:10:05.621764+0000","last_fullsized":"2024-03-18T19:10:05.621764+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:17:46.072540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253684+0000","last_change":"2024-03-18T19:10:04.629440+0000","last_active":"2024-03-18T19:10:06.253684+0000","last_peered":"2024-03-18T19:10:06.253684+0000","last_clean":"2024-03-18T19:10:06.253684+0000","last_became_active":"2024-03-18T19:10:04.629022+0000","last_became_peered":"2024-03-18T19:10:04.629022+0000","last_unstale":"2024-03-18T19:10:06.253684+0000","last_undegraded":"2024-03-18T19:10:06.253684+0000","last_fullsized":"2024-03-18T19:10:06.253684+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:10.396783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"160'1","reported_seq":12,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254223+0000","last_change":"2024-03-18T19:10:04.196572+0000","last_active":"2024-03-18T19:10:06.254223+0000","last_peered":"2024-03-18T19:10:06.254223+0000","last_clean":"2024-03-18T19:10:06.254223+0000","last_became_active":"2024-03-18T19:10:04.196281+0000","last_became_peered":"2024-03-18T19:10:04.196281+0000","last_unstale":"2024-03-18T19:10:06.254223+0000","last_undegraded":"2024-03-18T19:10:06.254223+0000","last_fullsized":"2024-03-18T19:10:06.254223+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:31:25.926313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254617+0000","last_change":"2024-03-18T19:10:04.642903+0000","last_active":"2024-03-18T19:10:06.254617+0000","last_peered":"2024-03-18T19:10:06.254617+0000","last_clean":"2024-03-18T19:10:06.254617+0000","last_became_active":"2024-03-18T19:10:04.642439+0000","last_became_peered":"2024-03-18T19:10:04.642439+0000","last_unstale":"2024-03-18T19:10:06.254617+0000","last_undegraded":"2024-03-18T19:10:06.254617+0000","last_fullsized":"2024-03-18T19:10:06.254617+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:16:48.875986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.753284+0000","last_change":"2024-03-18T19:10:04.201479+0000","last_active":"2024-03-18T19:10:05.753284+0000","last_peered":"2024-03-18T19:10:05.753284+0000","last_clean":"2024-03-18T19:10:05.753284+0000","last_became_active":"2024-03-18T19:10:04.201141+0000","last_became_peered":"2024-03-18T19:10:04.201141+0000","last_unstale":"2024-03-18T19:10:05.753284+0000","last_undegraded":"2024-03-18T19:10:05.753284+0000","last_fullsized":"2024-03-18T19:10:05.753284+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:06:42.226922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.753260+0000","last_change":"2024-03-18T19:10:04.636420+0000","last_active":"2024-03-18T19:10:05.753260+0000","last_peered":"2024-03-18T19:10:05.753260+0000","last_clean":"2024-03-18T19:10:05.753260+0000","last_became_active":"2024-03-18T19:10:04.636185+0000","last_became_peered":"2024-03-18T19:10:04.636185+0000","last_unstale":"2024-03-18T19:10:05.753260+0000","last_undegraded":"2024-03-18T19:10:05.753260+0000","last_fullsized":"2024-03-18T19:10:05.753260+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:33:04.563157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"160'2","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.255030+0000","last_change":"2024-03-18T19:10:04.198929+0000","last_active":"2024-03-18T19:10:06.255030+0000","last_peered":"2024-03-18T19:10:06.255030+0000","last_clean":"2024-03-18T19:10:06.255030+0000","last_became_active":"2024-03-18T19:10:04.198677+0000","last_became_peered":"2024-03-18T19:10:04.198677+0000","last_unstale":"2024-03-18T19:10:06.255030+0000","last_undegraded":"2024-03-18T19:10:06.255030+0000","last_fullsized":"2024-03-18T19:10:06.255030+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:50:22.386468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254095+0000","last_change":"2024-03-18T19:10:04.640339+0000","last_active":"2024-03-18T19:10:06.254095+0000","last_peered":"2024-03-18T19:10:06.254095+0000","last_clean":"2024-03-18T19:10:06.254095+0000","last_became_active":"2024-03-18T19:10:04.640082+0000","last_became_peered":"2024-03-18T19:10:04.640082+0000","last_unstale":"2024-03-18T19:10:06.254095+0000","last_undegraded":"2024-03-18T19:10:06.254095+0000","last_fullsized":"2024-03-18T19:10:06.254095+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:35:01.750851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254775+0000","last_change":"2024-03-18T19:10:04.197990+0000","last_active":"2024-03-18T19:10:06.254775+0000","last_peered":"2024-03-18T19:10:06.254775+0000","last_clean":"2024-03-18T19:10:06.254775+0000","last_became_active":"2024-03-18T19:10:04.197707+0000","last_became_peered":"2024-03-18T19:10:04.197707+0000","last_unstale":"2024-03-18T19:10:06.254775+0000","last_undegraded":"2024-03-18T19:10:06.254775+0000","last_fullsized":"2024-03-18T19:10:06.254775+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:42.276524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.753344+0000","last_change":"2024-03-18T19:10:04.642146+0000","last_active":"2024-03-18T19:10:05.753344+0000","last_peered":"2024-03-18T19:10:05.753344+0000","last_clean":"2024-03-18T19:10:05.753344+0000","last_became_active":"2024-03-18T19:10:04.641801+0000","last_became_peered":"2024-03-18T19:10:04.641801+0000","last_unstale":"2024-03-18T19:10:05.753344+0000","last_undegraded":"2024-03-18T19:10:05.753344+0000","last_fullsized":"2024-03-18T19:10:05.753344+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:04:46.862151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253935+0000","last_change":"2024-03-18T19:10:04.202020+0000","last_active":"2024-03-18T19:10:06.253935+0000","last_peered":"2024-03-18T19:10:06.253935+0000","last_clean":"2024-03-18T19:10:06.253935+0000","last_became_active":"2024-03-18T19:10:04.201759+0000","last_became_peered":"2024-03-18T19:10:04.201759+0000","last_unstale":"2024-03-18T19:10:06.253935+0000","last_undegraded":"2024-03-18T19:10:06.253935+0000","last_fullsized":"2024-03-18T19:10:06.253935+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:08.645798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.622426+0000","last_change":"2024-03-18T19:10:04.642299+0000","last_active":"2024-03-18T19:10:05.622426+0000","last_peered":"2024-03-18T19:10:05.622426+0000","last_clean":"2024-03-18T19:10:05.622426+0000","last_became_active":"2024-03-18T19:10:04.642017+0000","last_became_peered":"2024-03-18T19:10:04.642017+0000","last_unstale":"2024-03-18T19:10:05.622426+0000","last_undegraded":"2024-03-18T19:10:05.622426+0000","last_fullsized":"2024-03-18T19:10:05.622426+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:41.506510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253864+0000","last_change":"2024-03-18T19:10:04.186249+0000","last_active":"2024-03-18T19:10:06.253864+0000","last_peered":"2024-03-18T19:10:06.253864+0000","last_clean":"2024-03-18T19:10:06.253864+0000","last_became_active":"2024-03-18T19:10:04.185337+0000","last_became_peered":"2024-03-18T19:10:04.185337+0000","last_unstale":"2024-03-18T19:10:06.253864+0000","last_undegraded":"2024-03-18T19:10:06.253864+0000","last_fullsized":"2024-03-18T19:10:06.253864+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:30:07.307315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.753085+0000","last_change":"2024-03-18T19:10:04.642897+0000","last_active":"2024-03-18T19:10:05.753085+0000","last_peered":"2024-03-18T19:10:05.753085+0000","last_clean":"2024-03-18T19:10:05.753085+0000","last_became_active":"2024-03-18T19:10:04.642530+0000","last_became_peered":"2024-03-18T19:10:04.642530+0000","last_unstale":"2024-03-18T19:10:05.753085+0000","last_undegraded":"2024-03-18T19:10:05.753085+0000","last_fullsized":"2024-03-18T19:10:05.753085+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:28:43.444029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"160'2","reported_seq":13,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254909+0000","last_change":"2024-03-18T19:10:04.200655+0000","last_active":"2024-03-18T19:10:06.254909+0000","last_peered":"2024-03-18T19:10:06.254909+0000","last_clean":"2024-03-18T19:10:06.254909+0000","last_became_active":"2024-03-18T19:10:04.199145+0000","last_became_peered":"2024-03-18T19:10:04.199145+0000","last_unstale":"2024-03-18T19:10:06.254909+0000","last_undegraded":"2024-03-18T19:10:06.254909+0000","last_fullsized":"2024-03-18T19:10:06.254909+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:32:56.217504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.621350+0000","last_change":"2024-03-18T19:10:04.639783+0000","last_active":"2024-03-18T19:10:05.621350+0000","last_peered":"2024-03-18T19:10:05.621350+0000","last_clean":"2024-03-18T19:10:05.621350+0000","last_became_active":"2024-03-18T19:10:04.639487+0000","last_became_peered":"2024-03-18T19:10:04.639487+0000","last_unstale":"2024-03-18T19:10:05.621350+0000","last_undegraded":"2024-03-18T19:10:05.621350+0000","last_fullsized":"2024-03-18T19:10:05.621350+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:56.485712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254919+0000","last_change":"2024-03-18T19:10:04.196048+0000","last_active":"2024-03-18T19:10:06.254919+0000","last_peered":"2024-03-18T19:10:06.254919+0000","last_clean":"2024-03-18T19:10:06.254919+0000","last_became_active":"2024-03-18T19:10:04.195774+0000","last_became_peered":"2024-03-18T19:10:04.195774+0000","last_unstale":"2024-03-18T19:10:06.254919+0000","last_undegraded":"2024-03-18T19:10:06.254919+0000","last_fullsized":"2024-03-18T19:10:06.254919+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:16.096174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253746+0000","last_change":"2024-03-18T19:10:04.629345+0000","last_active":"2024-03-18T19:10:06.253746+0000","last_peered":"2024-03-18T19:10:06.253746+0000","last_clean":"2024-03-18T19:10:06.253746+0000","last_became_active":"2024-03-18T19:10:04.629036+0000","last_became_peered":"2024-03-18T19:10:04.629036+0000","last_unstale":"2024-03-18T19:10:06.253746+0000","last_undegraded":"2024-03-18T19:10:06.253746+0000","last_fullsized":"2024-03-18T19:10:06.253746+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:36.601312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254713+0000","last_change":"2024-03-18T19:10:04.195060+0000","last_active":"2024-03-18T19:10:06.254713+0000","last_peered":"2024-03-18T19:10:06.254713+0000","last_clean":"2024-03-18T19:10:06.254713+0000","last_became_active":"2024-03-18T19:10:04.194781+0000","last_became_peered":"2024-03-18T19:10:04.194781+0000","last_unstale":"2024-03-18T19:10:06.254713+0000","last_undegraded":"2024-03-18T19:10:06.254713+0000","last_fullsized":"2024-03-18T19:10:06.254713+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:55.480457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253986+0000","last_change":"2024-03-18T19:10:04.642731+0000","last_active":"2024-03-18T19:10:06.253986+0000","last_peered":"2024-03-18T19:10:06.253986+0000","last_clean":"2024-03-18T19:10:06.253986+0000","last_became_active":"2024-03-18T19:10:04.642133+0000","last_became_peered":"2024-03-18T19:10:04.642133+0000","last_unstale":"2024-03-18T19:10:06.253986+0000","last_undegraded":"2024-03-18T19:10:06.253986+0000","last_fullsized":"2024-03-18T19:10:06.253986+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:05:19.787348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254831+0000","last_change":"2024-03-18T19:10:04.193415+0000","last_active":"2024-03-18T19:10:06.254831+0000","last_peered":"2024-03-18T19:10:06.254831+0000","last_clean":"2024-03-18T19:10:06.254831+0000","last_became_active":"2024-03-18T19:10:04.193131+0000","last_became_peered":"2024-03-18T19:10:04.193131+0000","last_unstale":"2024-03-18T19:10:06.254831+0000","last_undegraded":"2024-03-18T19:10:06.254831+0000","last_fullsized":"2024-03-18T19:10:06.254831+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:57:05.031861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254730+0000","last_change":"2024-03-18T19:10:04.632620+0000","last_active":"2024-03-18T19:10:06.254730+0000","last_peered":"2024-03-18T19:10:06.254730+0000","last_clean":"2024-03-18T19:10:06.254730+0000","last_became_active":"2024-03-18T19:10:04.632329+0000","last_became_peered":"2024-03-18T19:10:04.632329+0000","last_unstale":"2024-03-18T19:10:06.254730+0000","last_undegraded":"2024-03-18T19:10:06.254730+0000","last_fullsized":"2024-03-18T19:10:06.254730+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:41.506510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.622167+0000","last_change":"2024-03-18T19:10:04.199354+0000","last_active":"2024-03-18T19:10:05.622167+0000","last_peered":"2024-03-18T19:10:05.622167+0000","last_clean":"2024-03-18T19:10:05.622167+0000","last_became_active":"2024-03-18T19:10:04.199062+0000","last_became_peered":"2024-03-18T19:10:04.199062+0000","last_unstale":"2024-03-18T19:10:05.622167+0000","last_undegraded":"2024-03-18T19:10:05.622167+0000","last_fullsized":"2024-03-18T19:10:05.622167+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:16.096174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254128+0000","last_change":"2024-03-18T19:10:04.640929+0000","last_active":"2024-03-18T19:10:06.254128+0000","last_peered":"2024-03-18T19:10:06.254128+0000","last_clean":"2024-03-18T19:10:06.254128+0000","last_became_active":"2024-03-18T19:10:04.636895+0000","last_became_peered":"2024-03-18T19:10:04.636895+0000","last_unstale":"2024-03-18T19:10:06.254128+0000","last_undegraded":"2024-03-18T19:10:06.254128+0000","last_fullsized":"2024-03-18T19:10:06.254128+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:10.396783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253920+0000","last_change":"2024-03-18T19:10:04.197527+0000","last_active":"2024-03-18T19:10:06.253920+0000","last_peered":"2024-03-18T19:10:06.253920+0000","last_clean":"2024-03-18T19:10:06.253920+0000","last_became_active":"2024-03-18T19:10:04.197266+0000","last_became_peered":"2024-03-18T19:10:04.197266+0000","last_unstale":"2024-03-18T19:10:06.253920+0000","last_undegraded":"2024-03-18T19:10:06.253920+0000","last_fullsized":"2024-03-18T19:10:06.253920+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:30:07.307315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254277+0000","last_change":"2024-03-18T19:10:04.633688+0000","last_active":"2024-03-18T19:10:06.254277+0000","last_peered":"2024-03-18T19:10:06.254277+0000","last_clean":"2024-03-18T19:10:06.254277+0000","last_became_active":"2024-03-18T19:10:04.633394+0000","last_became_peered":"2024-03-18T19:10:04.633394+0000","last_unstale":"2024-03-18T19:10:06.254277+0000","last_undegraded":"2024-03-18T19:10:06.254277+0000","last_fullsized":"2024-03-18T19:10:06.254277+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:47:30.693079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254029+0000","last_change":"2024-03-18T19:10:04.190562+0000","last_active":"2024-03-18T19:10:06.254029+0000","last_peered":"2024-03-18T19:10:06.254029+0000","last_clean":"2024-03-18T19:10:06.254029+0000","last_became_active":"2024-03-18T19:10:04.190286+0000","last_became_peered":"2024-03-18T19:10:04.190286+0000","last_unstale":"2024-03-18T19:10:06.254029+0000","last_undegraded":"2024-03-18T19:10:06.254029+0000","last_fullsized":"2024-03-18T19:10:06.254029+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:16.096174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.254222+0000","last_change":"2024-03-18T19:10:04.629407+0000","last_active":"2024-03-18T19:10:06.254222+0000","last_peered":"2024-03-18T19:10:06.254222+0000","last_clean":"2024-03-18T19:10:06.254222+0000","last_became_active":"2024-03-18T19:10:04.629093+0000","last_became_peered":"2024-03-18T19:10:04.629093+0000","last_unstale":"2024-03-18T19:10:06.254222+0000","last_undegraded":"2024-03-18T19:10:06.254222+0000","last_fullsized":"2024-03-18T19:10:06.254222+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:27:58.639356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.753170+0000","last_change":"2024-03-18T19:10:04.201448+0000","last_active":"2024-03-18T19:10:05.753170+0000","last_peered":"2024-03-18T19:10:05.753170+0000","last_clean":"2024-03-18T19:10:05.753170+0000","last_became_active":"2024-03-18T19:10:04.201154+0000","last_became_peered":"2024-03-18T19:10:04.201154+0000","last_unstale":"2024-03-18T19:10:05.753170+0000","last_undegraded":"2024-03-18T19:10:05.753170+0000","last_fullsized":"2024-03-18T19:10:05.753170+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:08:09.388920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:05.621443+0000","last_change":"2024-03-18T19:10:04.638993+0000","last_active":"2024-03-18T19:10:05.621443+0000","last_peered":"2024-03-18T19:10:05.621443+0000","last_clean":"2024-03-18T19:10:05.621443+0000","last_became_active":"2024-03-18T19:10:04.638610+0000","last_became_peered":"2024-03-18T19:10:04.638610+0000","last_unstale":"2024-03-18T19:10:05.621443+0000","last_undegraded":"2024-03-18T19:10:05.621443+0000","last_fullsized":"2024-03-18T19:10:05.621443+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":159,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:04.176211+0000","last_clean_scrub_stamp":"2024-03-18T19:10:04.176211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:09:30.161494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-18T19:10:06.253805+0000","last_change":"2024-03-18T19:10:04.201282+0000","last_active":"2024-03-18T19:10:06.253805+0000","last_peered":"2024-03-18T19:10:06.253805+0000","last_clean":"2024-03-18T19:10:06.253805+0000","last_became_active":"2024-03-18T19:10:04.200985+0000","last_became_peered":"2024-03-18T19:10:04.200985+0000","last_unstale":"2024-03-18T19:10:06.253805+0000","last_undegraded":"2024-03-18T19:10:06.253805+0000","last_fullsized":"2024-03-18T19:10:06.253805+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T19:10:03.170956+0000","last_clean_scrub_stamp":"2024-03-18T19:10:03.170956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:55.480457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"150'249","reported_seq":754,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-18T19:10:04.619475+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T19:10:04.619475+0000","last_peered":"2024-03-18T19:10:04.619475+0000","last_clean":"2024-03-18T19:10:04.619475+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T19:10:04.619475+0000","last_undegraded":"2024-03-18T19:10:04.619475+0000","last_fullsized":"2024-03-18T19:10:04.619475+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575084,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59924,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311916,"statfs":{"total":96636764160,"available":96575401984,"internally_reserved":0,"allocated":2617344,"data_stored":1603978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.55300000000000005,"15min":0.56599999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.27800000000000002},"max":{"1min":0.76200000000000001,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.436},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55800000000000005,"15min":0.57899999999999996},"min":{"1min":0.373,"5min":0.33700000000000002,"15min":0.318},"max":{"1min":0.78300000000000003,"5min":1.387,"15min":1.387},"last":0.57799999999999996}]},{"osd":1,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.53500000000000003,"15min":0.56799999999999995},"min":{"1min":0.34399999999999997,"5min":0.33200000000000002,"15min":0.307},"max":{"1min":0.73299999999999998,"5min":0.81699999999999995,"15min":1.484},"last":0.505},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55100000000000005,"15min":0.58599999999999997},"min":{"1min":0.39500000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.92000000000000004},"last":0.623}]},{"osd":2,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55200000000000005,"15min":0.57099999999999995},"min":{"1min":0.38200000000000001,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":0.68000000000000005,"5min":1.3600000000000001,"15min":1.5129999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57499999999999996,"15min":0.59099999999999997},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73499999999999999,"5min":1.327,"15min":1.536},"last":0.59799999999999998}]},{"osd":3,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55400000000000005,"15min":0.57599999999999996},"min":{"1min":0.38800000000000001,"5min":0.35599999999999998,"15min":0.318},"max":{"1min":0.745,"5min":0.79100000000000004,"15min":1.5489999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57399999999999995,"15min":0.59799999999999998},"min":{"1min":0.441,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.78300000000000003,"5min":0.81699999999999995,"15min":1.5660000000000001},"last":0.54400000000000004}]},{"osd":4,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.45800000000000002,"15min":0.45300000000000001},"min":{"1min":0.30299999999999999,"5min":0.27800000000000002,"15min":0.27100000000000002},"max":{"1min":0.748,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.46999999999999997,"15min":0.47299999999999998},"min":{"1min":0.35299999999999998,"5min":0.27500000000000002,"15min":0.25800000000000001},"max":{"1min":0.63900000000000001,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.41199999999999998}]},{"osd":5,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.48499999999999999,"15min":0.47699999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.26000000000000001},"max":{"1min":0.68600000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.51500000000000001,"15min":0.51400000000000001},"min":{"1min":0.33600000000000002,"5min":0.317,"15min":0.30299999999999999},"max":{"1min":0.70699999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.373}]},{"osd":6,"last update":"Mon Mar 18 19:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.54300000000000004},"min":{"1min":0.34000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.74099999999999999,"5min":0.81799999999999995,"15min":0.82799999999999996},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53600000000000003,"15min":0.53000000000000003},"min":{"1min":0.36399999999999999,"5min":0.311,"15min":0.29099999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.47499999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575084,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37188,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334652,"statfs":{"total":96636764160,"available":96598683648,"internally_reserved":0,"allocated":2015232,"data_stored":1012628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52500000000000002,"15min":0.55500000000000005},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28499999999999998},"max":{"1min":0.67800000000000005,"5min":0.73699999999999999,"15min":0.76800000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53900000000000003,"15min":0.56499999999999995},"min":{"1min":0.36199999999999999,"5min":0.35199999999999998,"15min":0.27400000000000002},"max":{"1min":0.71099999999999997,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.51900000000000002}]},{"osd":1,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.56000000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.77400000000000002},"last":0.438},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.56499999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.32000000000000001},"max":{"1min":0.78900000000000003,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.54300000000000004}]},{"osd":2,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55400000000000005,"15min":0.57299999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.30499999999999999},"max":{"1min":0.73699999999999999,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54500000000000004,"15min":0.57399999999999995},"min":{"1min":0.40999999999999998,"5min":0.35299999999999998,"15min":0.30299999999999999},"max":{"1min":0.73399999999999999,"5min":0.753,"15min":0.83199999999999996},"last":0.55900000000000005}]},{"osd":3,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.58399999999999996},"min":{"1min":0.40799999999999997,"5min":0.372,"15min":0.32100000000000001},"max":{"1min":0.76400000000000001,"5min":0.78400000000000003,"15min":0.89600000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55400000000000005,"15min":0.57199999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72099999999999997,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.495}]},{"osd":4,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46700000000000003,"15min":0.45000000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.623},{"interface":"front","average":{"1min":0.495,"5min":0.48899999999999999,"15min":0.47699999999999998},"min":{"1min":0.32200000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.60199999999999998}]},{"osd":5,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.75,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53400000000000003,"15min":0.51600000000000001},"min":{"1min":0.34899999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.65600000000000003}]},{"osd":7,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54400000000000004,"15min":0.52900000000000003},"min":{"1min":0.371,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55600000000000005,"15min":0.53600000000000003},"min":{"1min":0.371,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.80500000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.57499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575084,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37176,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334664,"statfs":{"total":96636764160,"available":96598695936,"internally_reserved":0,"allocated":2011136,"data_stored":1012026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.35499999999999998,"15min":0.373},"min":{"1min":0.216,"5min":0.216,"15min":0.19600000000000001},"max":{"1min":0.58699999999999997,"5min":0.63300000000000001,"15min":0.66300000000000003},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.38600000000000001,"15min":0.41299999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.57099999999999995,"5min":0.66500000000000004,"15min":0.73699999999999999},"last":0.27900000000000003}]},{"osd":2,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38900000000000001,"15min":0.40400000000000003},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.59299999999999997,"5min":0.64200000000000002,"15min":0.65400000000000003},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.434,"15min":0.44500000000000001},"min":{"1min":0.29599999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.63300000000000001,"5min":0.66500000000000004,"15min":0.71699999999999997},"last":0.36599999999999999}]},{"osd":3,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.44500000000000001,"15min":0.46400000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.60399999999999998,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40200000000000002,"15min":0.41899999999999998},"min":{"1min":0.25900000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.61199999999999999,"5min":0.67800000000000005,"15min":0.77000000000000002},"last":0.38400000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60499999999999998,"15min":0.628},"min":{"1min":0.46200000000000002,"5min":0.33400000000000002,"15min":0.32800000000000001},"max":{"1min":0.89600000000000002,"5min":0.92300000000000004,"15min":0.93999999999999995},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.62,"15min":0.64100000000000001},"min":{"1min":0.443,"5min":0.439,"15min":0.36899999999999999},"max":{"1min":0.86499999999999999,"5min":0.97499999999999998,"15min":1.018},"last":0.59999999999999998}]},{"osd":5,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60399999999999998,"15min":0.63700000000000001},"min":{"1min":0.40500000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88700000000000001,"5min":0.89500000000000002,"15min":0.99199999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.63200000000000001,"15min":0.65600000000000003},"min":{"1min":0.47199999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":1.006},"last":0.57399999999999995}]},{"osd":6,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.60499999999999998,"15min":0.63400000000000001},"min":{"1min":0.42899999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.878,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.64600000000000002,"15min":0.67000000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.39100000000000001},"max":{"1min":0.91300000000000003,"5min":0.94899999999999995,"15min":0.98099999999999998},"last":0.66600000000000004}]},{"osd":7,"last update":"Mon Mar 18 19:09:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63200000000000001,"15min":0.65900000000000003},"min":{"1min":0.45100000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81899999999999995,"5min":0.90600000000000003,"15min":0.94799999999999995},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63400000000000001,"15min":0.66200000000000003},"min":{"1min":0.48299999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90800000000000003,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.58799999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575085,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37752,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334088,"statfs":{"total":96636764160,"available":96598106112,"internally_reserved":0,"allocated":2596864,"data_stored":1595732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36299999999999999,"15min":0.376},"min":{"1min":0.218,"5min":0.19700000000000001,"15min":0.19400000000000001},"max":{"1min":0.59899999999999998,"5min":0.60499999999999998,"15min":0.621},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39400000000000002,"15min":0.40999999999999998},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20000000000000001},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.63200000000000001},"last":0.436}]},{"osd":2,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40699999999999997,"15min":0.41699999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.21099999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.42299999999999999,"15min":0.432},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.59399999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.25800000000000001}]},{"osd":3,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.40300000000000002,"15min":0.41299999999999998},"min":{"1min":0.27200000000000002,"5min":0.26500000000000001,"15min":0.217},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.64100000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.45500000000000002,"15min":0.46800000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.28799999999999998},"max":{"1min":0.64300000000000002,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.39900000000000002}]},{"osd":4,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.60099999999999998,"15min":0.63300000000000001},"min":{"1min":0.376,"5min":0.35699999999999998,"15min":0.27200000000000002},"max":{"1min":0.76900000000000002,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.60899999999999999,"15min":0.63500000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.315},"max":{"1min":0.75600000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.52900000000000003}]},{"osd":5,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.627,"15min":0.64300000000000002},"min":{"1min":0.432,"5min":0.376,"15min":0.34300000000000003},"max":{"1min":0.78100000000000003,"5min":1.248,"15min":1.248},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.63800000000000001,"15min":0.65800000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.31},"max":{"1min":0.84899999999999998,"5min":1.399,"15min":1.399},"last":0.497}]},{"osd":6,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.63200000000000001,"15min":0.65500000000000003},"min":{"1min":0.44700000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.86099999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.64900000000000002,"15min":0.66700000000000004},"min":{"1min":0.47599999999999998,"5min":0.42399999999999999,"15min":0.40999999999999998},"max":{"1min":0.83499999999999996,"5min":1.284,"15min":1.284},"last":0.55700000000000005}]},{"osd":7,"last update":"Mon Mar 18 19:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.64100000000000001,"15min":0.66300000000000003},"min":{"1min":0.42399999999999999,"5min":0.39300000000000002,"15min":0.34899999999999998},"max":{"1min":0.878,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64000000000000001,"15min":0.66400000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.38500000000000001},"max":{"1min":0.86599999999999999,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.57799999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575085,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59332,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312508,"statfs":{"total":96636764160,"available":96576008192,"internally_reserved":0,"allocated":2015232,"data_stored":1012572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34999999999999998,"15min":0.36799999999999999},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.498,"5min":0.66600000000000004,"15min":0.78300000000000003},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.44400000000000001,"15min":0.45500000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.217},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.77200000000000002},"last":0.48999999999999999}]},{"osd":1,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.435,"15min":0.46400000000000002},"min":{"1min":0.31,"5min":0.27300000000000002,"15min":0.27100000000000002},"max":{"1min":0.62,"5min":0.69499999999999995,"15min":0.90500000000000003},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.38400000000000001,"15min":0.39600000000000002},"min":{"1min":0.26700000000000002,"5min":0.22800000000000001,"15min":0.216},"max":{"1min":0.54300000000000004,"5min":0.64500000000000002,"15min":0.66700000000000004},"last":0.33800000000000002}]},{"osd":3,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.40999999999999998,"15min":0.41299999999999998},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.24199999999999999},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47599999999999998,"15min":0.499},"min":{"1min":0.33400000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.80100000000000005},"last":0.51400000000000001}]},{"osd":4,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58399999999999996,"15min":0.60999999999999999},"min":{"1min":0.33500000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76100000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.59799999999999998,"15min":0.624},"min":{"1min":0.38600000000000001,"5min":0.378,"15min":0.32100000000000001},"max":{"1min":1.355,"5min":1.448,"15min":1.4730000000000001},"last":0.68000000000000005}]},{"osd":5,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.61899999999999999,"15min":0.64100000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.34699999999999998},"max":{"1min":0.90600000000000003,"5min":1.327,"15min":1.8109999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.59999999999999998,"15min":0.63100000000000001},"min":{"1min":0.376,"5min":0.36899999999999999,"15min":0.33400000000000002},"max":{"1min":0.75,"5min":1.4630000000000001,"15min":1.948},"last":0.69599999999999995}]},{"osd":6,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.624,"15min":0.65500000000000003},"min":{"1min":0.42599999999999999,"5min":0.39600000000000002,"15min":0.35899999999999999},"max":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":2.9470000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65600000000000003,"15min":0.67900000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.38100000000000001},"max":{"1min":0.94899999999999995,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.66500000000000004}]},{"osd":7,"last update":"Mon Mar 18 19:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.625,"15min":0.64700000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.89100000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64000000000000001,"15min":0.66100000000000003},"min":{"1min":0.437,"5min":0.38300000000000001,"15min":0.371},"max":{"1min":1.409,"5min":1.482,"15min":1.482},"last":0.623}]}]},{"osd":3,"up_from":13,"seq":55834575085,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37192,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334648,"statfs":{"total":96636764160,"available":96598679552,"internally_reserved":0,"allocated":2023424,"data_stored":1013590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:10:05 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.33700000000000002,"15min":0.371},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.45900000000000002,"5min":0.51400000000000001,"15min":1.204},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.36499999999999999,"15min":0.39400000000000002},"min":{"1min":0.23999999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54400000000000004,"5min":0.66900000000000004,"15min":1.137},"last":0.39800000000000002}]},{"osd":1,"last update":"Mon Mar 18 19:10:05 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.35399999999999998,"15min":0.39000000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.42699999999999999,"5min":0.629,"15min":1.1220000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.39600000000000002,"15min":0.436},"min":{"1min":0.29599999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.52100000000000002,"5min":0.76600000000000001,"15min":1.1080000000000001},"last":0.41899999999999998}]},{"osd":2,"last update":"Mon Mar 18 19:10:05 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.39900000000000002,"15min":0.44},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.48699999999999999,"5min":0.67500000000000004,"15min":1.1879999999999999},"last":0.44},{"interface":"front","average":{"1min":0.376,"5min":0.42099999999999999,"15min":0.46600000000000003},"min":{"1min":0.27200000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.47699999999999998,"5min":0.69199999999999995,"15min":1.2290000000000001},"last":0.46400000000000002}]},{"osd":4,"last update":"Mon Mar 18 19:10:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.57299999999999995,"15min":0.61299999999999999},"min":{"1min":0.39700000000000002,"5min":0.38100000000000001,"15min":0.34300000000000003},"max":{"1min":0.74199999999999999,"5min":1.052,"15min":1.871},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58999999999999997,"15min":0.623},"min":{"1min":0.42899999999999999,"5min":0.40699999999999997,"15min":0.39600000000000002},"max":{"1min":0.82499999999999996,"5min":0.88100000000000001,"15min":1.764},"last":1.3080000000000001}]},{"osd":5,"last update":"Mon Mar 18 19:10:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.623,"15min":0.65900000000000003},"min":{"1min":0.46500000000000002,"5min":0.39200000000000002,"15min":0.379},"max":{"1min":0.78400000000000003,"5min":1.0249999999999999,"15min":2.5329999999999999},"last":1.444},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.63600000000000001,"15min":0.67600000000000005},"min":{"1min":0.45400000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.80700000000000005,"5min":0.94599999999999995,"15min":1.794},"last":0.57399999999999995}]},{"osd":6,"last update":"Mon Mar 18 19:10:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.626,"15min":0.66500000000000004},"min":{"1min":0.42999999999999999,"5min":0.42499999999999999,"15min":0.41499999999999998},"max":{"1min":0.754,"5min":1.1060000000000001,"15min":2.7389999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.64100000000000001,"15min":0.67200000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.85899999999999999,"5min":0.98199999999999998,"15min":1.8069999999999999},"last":1.3999999999999999}]},{"osd":7,"last update":"Mon Mar 18 19:10:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61599999999999999,"15min":0.65700000000000003},"min":{"1min":0.41799999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.746,"5min":0.96099999999999997,"15min":1.732},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.61499999999999999,"15min":0.66300000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.79600000000000004,"5min":1.085,"15min":2.7050000000000001},"last":0.52700000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575085,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33076,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338764,"statfs":{"total":96636764160,"available":96602894336,"internally_reserved":0,"allocated":2007040,"data_stored":1011956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.56699999999999995,"15min":0.57499999999999996},"min":{"1min":0.43099999999999999,"5min":0.31,"15min":0.246},"max":{"1min":2.3210000000000002,"5min":2.3210000000000002,"15min":2.3210000000000002},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.41699999999999998,"5min":0.35299999999999998,"15min":0.26000000000000001},"max":{"1min":1.7430000000000001,"5min":3.645,"15min":3.645},"last":0.50600000000000001}]},{"osd":1,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65000000000000002,"15min":0.61199999999999999},"min":{"1min":0.40699999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.6739999999999999,"5min":9.9499999999999993,"15min":9.9499999999999993},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54400000000000004,"15min":0.57399999999999995},"min":{"1min":0.40200000000000002,"5min":0.32300000000000001,"15min":0.308},"max":{"1min":0.80000000000000004,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.374}]},{"osd":2,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56699999999999995,"15min":0.57799999999999996},"min":{"1min":0.439,"5min":0.35899999999999999,"15min":0.28999999999999998},"max":{"1min":2.431,"5min":2.431,"15min":2.431},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.61499999999999999,"15min":0.61299999999999999},"min":{"1min":0.45700000000000002,"5min":0.35099999999999998,"15min":0.32600000000000001},"max":{"1min":2.165,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.55400000000000005}]},{"osd":3,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.55800000000000005,"15min":0.58299999999999996},"min":{"1min":0.39700000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76800000000000002,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.58799999999999997,"15min":0.59999999999999998},"min":{"1min":0.42599999999999999,"5min":0.39300000000000002,"15min":0.33900000000000002},"max":{"1min":2.5510000000000002,"5min":2.5510000000000002,"15min":2.5510000000000002},"last":0.52900000000000003}]},{"osd":5,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46300000000000002,"15min":0.46000000000000002},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.245},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.77500000000000002},"last":0.624},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.51600000000000001,"15min":0.49199999999999999},"min":{"1min":0.307,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.73599999999999999,"5min":5.4900000000000002,"15min":5.4900000000000002},"last":0.49399999999999999}]},{"osd":6,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.49199999999999999,"15min":0.48499999999999999},"min":{"1min":0.33200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.5190000000000001,"5min":2.5190000000000001,"15min":2.5190000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.55100000000000005,"15min":0.52900000000000003},"min":{"1min":0.32600000000000001,"5min":0.317,"15min":0.28999999999999998},"max":{"1min":1.2689999999999999,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":0.65800000000000003}]},{"osd":7,"last update":"Mon Mar 18 19:09:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52200000000000002,"15min":0.51900000000000002},"min":{"1min":0.34999999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.94599999999999995,"5min":1.55,"15min":1.55},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.54800000000000004,"15min":0.54900000000000004},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.56999999999999995}]}]},{"osd":5,"up_from":13,"seq":55834575084,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51136,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320704,"statfs":{"total":96636764160,"available":96584400896,"internally_reserved":0,"allocated":2011136,"data_stored":1011978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55400000000000005,"15min":0.56599999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.26600000000000001},"max":{"1min":0.76400000000000001,"5min":0.86099999999999999,"15min":1.1679999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55200000000000005,"15min":0.56599999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.25800000000000001},"max":{"1min":0.69699999999999995,"5min":0.80600000000000005,"15min":1.1319999999999999},"last":0.621}]},{"osd":1,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54800000000000004,"15min":0.56599999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.29599999999999999},"max":{"1min":0.78300000000000003,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.56100000000000005,"15min":0.57899999999999996},"min":{"1min":0.34599999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71999999999999997,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.63200000000000001}]},{"osd":2,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55300000000000005,"15min":0.57399999999999995},"min":{"1min":0.371,"5min":0.30399999999999999,"15min":0.28399999999999997},"max":{"1min":0.79700000000000004,"5min":1.153,"15min":1.1579999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55100000000000005,"15min":0.56899999999999995},"min":{"1min":0.377,"5min":0.35599999999999998,"15min":0.33200000000000002},"max":{"1min":0.71199999999999997,"5min":0.79500000000000004,"15min":1.0569999999999999},"last":0.60899999999999999}]},{"osd":3,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.39700000000000002,"5min":0.35799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72499999999999998,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57399999999999995,"15min":0.59599999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77000000000000002,"5min":1.212,"15min":1.212},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.46300000000000002},"min":{"1min":0.34799999999999998,"5min":0.255,"15min":0.24199999999999999},"max":{"1min":0.64700000000000002,"5min":1.248,"15min":1.248},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.496,"15min":0.48899999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.23100000000000001},"max":{"1min":0.82599999999999996,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.499}]},{"osd":6,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.29399999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.68000000000000005,"5min":0.78200000000000003,"15min":1.0069999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.51100000000000001,"15min":0.50900000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.26800000000000002},"max":{"1min":0.71799999999999997,"5min":0.83199999999999996,"15min":0.94599999999999995},"last":0.51900000000000002}]},{"osd":7,"last update":"Mon Mar 18 19:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.32700000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.85699999999999998,"5min":1.282,"15min":1.282},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.34200000000000003,"5min":0.28799999999999998,"15min":0.28499999999999998},"max":{"1min":0.81000000000000005,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T19:10:09.176 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T19:10:09.177 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-18T19:10:09.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.347+0000 7f8a096cb700 1 Processor -- start 2024-03-18T19:10:09.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.347+0000 7f8a096cb700 1 -- start start 2024-03-18T19:10:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.347+0000 7f8a096cb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a040cfb50 0x7f8a040d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.347+0000 7f8a096cb700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8a040cff30 0x7f8a040b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.347+0000 7f8a096cb700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a040b3370 con 0x7f8a040b3d40 2024-03-18T19:10:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.347+0000 7f8a096cb700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a040b2760 con 0x7f8a040cff30 2024-03-18T19:10:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.347+0000 7f8a096cb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a040b2a40 con 0x7f8a040cfb50 2024-03-18T19:10:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.348+0000 7f8a02ffd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a040cfb50 0x7f8a040d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.348+0000 7f8a027fc700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8a040cff30 0x7f8a040b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.348+0000 7f8a027fc700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8a040cff30 0x7f8a040b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:50856/0 (socket says 172.21.15.1:50856) 2024-03-18T19:10:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.348+0000 7f8a02ffd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a040cfb50 0x7f8a040d3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:35302/0 (socket says 172.21.15.1:35302) 2024-03-18T19:10:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.348+0000 7f8a027fc700 1 -- 172.21.15.1:0/600624673 learned_addr learned my addr 172.21.15.1:0/600624673 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.348+0000 7f8a037fe700 1 --1- 172.21.15.1:0/600624673 >> v1:172.21.15.165:6789/0 conn(0x7f8a040b3d40 0x7f8a040b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:44944/0 (socket says 172.21.15.1:44944) 2024-03-18T19:10:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.348+0000 7f8a02ffd700 1 -- 172.21.15.1:0/600624673 >> v2:172.21.15.165:3300/0 conn(0x7f8a040cff30 msgr2=0x7f8a040b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:09.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.348+0000 7f8a02ffd700 1 --2- 172.21.15.1:0/600624673 >> v2:172.21.15.165:3300/0 conn(0x7f8a040cff30 0x7f8a040b3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.348+0000 7f8a02ffd700 1 -- 172.21.15.1:0/600624673 >> v1:172.21.15.165:6789/0 conn(0x7f8a040b3d40 legacy=0x7f8a040b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.348+0000 7f8a02ffd700 1 -- 172.21.15.1:0/600624673 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a040b2d20 con 0x7f8a040cfb50 2024-03-18T19:10:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.348+0000 7f8a01ffb700 1 -- 172.21.15.1:0/600624673 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949954828 0 0) 0x7f8a040b3370 con 0x7f8a040b3d40 2024-03-18T19:10:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.348+0000 7f8a027fc700 1 --2- 172.21.15.1:0/600624673 >> v2:172.21.15.165:3300/0 conn(0x7f8a040cff30 0x7f8a040b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:10:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.348+0000 7f8a02ffd700 1 --2- 172.21.15.1:0/600624673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a040cfb50 0x7f8a040d3c60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f89f4009820 tx=0x7f89f4009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f74389a708fa0f0a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.349+0000 7f8a01ffb700 1 -- 172.21.15.1:0/600624673 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f4005e30 con 0x7f8a040cfb50 2024-03-18T19:10:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.349+0000 7f8a01ffb700 1 -- 172.21.15.1:0/600624673 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f4006030 con 0x7f8a040cfb50 2024-03-18T19:10:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.349+0000 7f8a01ffb700 1 -- 172.21.15.1:0/600624673 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f40063c0 con 0x7f8a040cfb50 2024-03-18T19:10:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.349+0000 7f8a096cb700 1 -- 172.21.15.1:0/600624673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a040cfb50 msgr2=0x7f8a040d3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.349+0000 7f8a096cb700 1 --2- 172.21.15.1:0/600624673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a040cfb50 0x7f8a040d3c60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f89f4009820 tx=0x7f89f4009b30 comp rx=0 tx=0).stop 2024-03-18T19:10:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.349+0000 7f8a096cb700 1 -- 172.21.15.1:0/600624673 shutdown_connections 2024-03-18T19:10:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.349+0000 7f8a096cb700 1 --2- 172.21.15.1:0/600624673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a040cfb50 0x7f8a040d3c60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:09.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.349+0000 7f8a096cb700 1 --2- 172.21.15.1:0/600624673 >> v2:172.21.15.165:3300/0 conn(0x7f8a040cff30 0x7f8a040b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:09.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.349+0000 7f8a096cb700 1 -- 172.21.15.1:0/600624673 >> 172.21.15.1:0/600624673 conn(0x7f8a040d3140 msgr2=0x7f8a040d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:09.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.349+0000 7f8a096cb700 1 -- 172.21.15.1:0/600624673 shutdown_connections 2024-03-18T19:10:09.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.349+0000 7f8a096cb700 1 -- 172.21.15.1:0/600624673 wait complete. 2024-03-18T19:10:09.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.351+0000 7f8a096cb700 1 Processor -- start 2024-03-18T19:10:09.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.351+0000 7f8a096cb700 1 -- start start 2024-03-18T19:10:09.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.352+0000 7f8a096cb700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a040b3d40 0x7f8a0414a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:09.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.352+0000 7f8a096cb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a040cfb50 0x7f8a040c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:09.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.352+0000 7f8a096cb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a040cff30 0x7f8a040cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:09.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.352+0000 7f8a096cb700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a040c81d0 con 0x7f8a040cff30 2024-03-18T19:10:09.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.352+0000 7f8a096cb700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0414aa90 con 0x7f8a040b3d40 2024-03-18T19:10:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.352+0000 7f8a096cb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a040c7550 con 0x7f8a040cfb50 2024-03-18T19:10:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.352+0000 7f8a037fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a040cff30 0x7f8a040cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.352+0000 7f8a037fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a040cff30 0x7f8a040cc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50940/0 (socket says 172.21.15.1:50940) 2024-03-18T19:10:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.352+0000 7f8a02ffd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a040b3d40 0x7f8a0414a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.352+0000 7f8a027fc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a040cfb50 0x7f8a040c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.352+0000 7f8a037fe700 1 -- 172.21.15.1:0/4256263126 learned_addr learned my addr 172.21.15.1:0/4256263126 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.352+0000 7f8a037fe700 1 -- 172.21.15.1:0/4256263126 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a040cfb50 msgr2=0x7f8a040c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.352+0000 7f8a037fe700 1 --2- 172.21.15.1:0/4256263126 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a040cfb50 0x7f8a040c9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.352+0000 7f8a037fe700 1 -- 172.21.15.1:0/4256263126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a040b3d40 msgr2=0x7f8a0414a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.352+0000 7f8a037fe700 1 --2- 172.21.15.1:0/4256263126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a040b3d40 0x7f8a0414a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.352+0000 7f8a037fe700 1 -- 172.21.15.1:0/4256263126 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f8009550 con 0x7f8a040cff30 2024-03-18T19:10:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.353+0000 7f8a037fe700 1 --2- 172.21.15.1:0/4256263126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a040cff30 0x7f8a040cc1f0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00cca0 tx=0x7f89fc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:09.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.353+0000 7f89eb7fe700 1 -- 172.21.15.1:0/4256263126 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc00ab70 con 0x7f8a040cff30 2024-03-18T19:10:09.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.353+0000 7f8a096cb700 1 -- 172.21.15.1:0/4256263126 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f4009430 con 0x7f8a040cff30 2024-03-18T19:10:09.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.353+0000 7f89eb7fe700 1 -- 172.21.15.1:0/4256263126 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89fc00acd0 con 0x7f8a040cff30 2024-03-18T19:10:09.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.353+0000 7f89eb7fe700 1 -- 172.21.15.1:0/4256263126 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc007310 con 0x7f8a040cff30 2024-03-18T19:10:09.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.353+0000 7f8a096cb700 1 -- 172.21.15.1:0/4256263126 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a040c7b10 con 0x7f8a040cff30 2024-03-18T19:10:09.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.354+0000 7f8a096cb700 1 -- 172.21.15.1:0/4256263126 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a040c8310 con 0x7f8a040cff30 2024-03-18T19:10:09.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.354+0000 7f89eb7fe700 1 -- 172.21.15.1:0/4256263126 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f89fc007470 con 0x7f8a040cff30 2024-03-18T19:10:09.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.355+0000 7f89eb7fe700 1 --2- 172.21.15.1:0/4256263126 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f89f006adb0 0x7f89f006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:09.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.355+0000 7f89eb7fe700 1 -- 172.21.15.1:0/4256263126 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f89fc089210 con 0x7f8a040cff30 2024-03-18T19:10:09.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.355+0000 7f8a02ffd700 1 --2- 172.21.15.1:0/4256263126 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f89f006adb0 0x7f89f006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:09.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.355+0000 7f8a02ffd700 1 --2- 172.21.15.1:0/4256263126 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f89f006adb0 0x7f89f006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f89f4009b70 tx=0x7f89f4008070 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:09.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.359+0000 7f89eb7fe700 1 -- 172.21.15.1:0/4256263126 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8a040c8310 con 0x7f8a040cff30 2024-03-18T19:10:09.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.648+0000 7f8a096cb700 1 -- 172.21.15.1:0/4256263126 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f8a040c8310 con 0x7f8a040cff30 2024-03-18T19:10:09.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.651+0000 7f89eb7fe700 1 -- 172.21.15.1:0/4256263126 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+0 (secure 0 0 0) 0x7f8a040c8310 con 0x7f8a040cff30 2024-03-18T19:10:09.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.653+0000 7f8a096cb700 1 -- 172.21.15.1:0/4256263126 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f89f006adb0 msgr2=0x7f89f006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:09.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.653+0000 7f8a096cb700 1 --2- 172.21.15.1:0/4256263126 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f89f006adb0 0x7f89f006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f89f4009b70 tx=0x7f89f4008070 comp rx=0 tx=0).stop 2024-03-18T19:10:09.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.653+0000 7f8a096cb700 1 -- 172.21.15.1:0/4256263126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a040cff30 msgr2=0x7f8a040cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:09.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.653+0000 7f8a096cb700 1 --2- 172.21.15.1:0/4256263126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a040cff30 0x7f8a040cc1f0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00cca0 tx=0x7f89fc00a060 comp rx=0 tx=0).stop 2024-03-18T19:10:09.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.653+0000 7f8a096cb700 1 -- 172.21.15.1:0/4256263126 shutdown_connections 2024-03-18T19:10:09.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.653+0000 7f8a096cb700 1 --2- 172.21.15.1:0/4256263126 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f89f006adb0 0x7f89f006d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:09.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.653+0000 7f8a096cb700 1 --2- 172.21.15.1:0/4256263126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a040cff30 0x7f8a040cc1f0 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:09.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.653+0000 7f8a096cb700 1 --2- 172.21.15.1:0/4256263126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a040b3d40 0x7f8a0414a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:09.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.653+0000 7f8a096cb700 1 --2- 172.21.15.1:0/4256263126 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a040cfb50 0x7f8a040c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:09.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.653+0000 7f8a096cb700 1 -- 172.21.15.1:0/4256263126 >> 172.21.15.1:0/4256263126 conn(0x7f8a040d3140 msgr2=0x7f8a040d1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:09.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.653+0000 7f8a096cb700 1 -- 172.21.15.1:0/4256263126 shutdown_connections 2024-03-18T19:10:09.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.653+0000 7f8a096cb700 1 -- 172.21.15.1:0/4256263126 wait complete. 2024-03-18T19:10:09.652 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-18T19:10:09.663 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T19:10:09.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.790+0000 7fde84d32700 1 Processor -- start 2024-03-18T19:10:09.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.790+0000 7fde84d32700 1 -- start start 2024-03-18T19:10:09.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.791+0000 7fde84d32700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde8006dd50 0x7fde80072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:09.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.791+0000 7fde84d32700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde8006f490 con 0x7fde8006e290 2024-03-18T19:10:09.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.791+0000 7fde84d32700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde80071cf0 con 0x7fde8015d580 2024-03-18T19:10:09.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.791+0000 7fde84d32700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde8006f1b0 con 0x7fde8006dd50 2024-03-18T19:10:09.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.791+0000 7fde7dd9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fde8006e290 0x7fde800ca560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52640/0 (socket says 172.21.15.1:52640) 2024-03-18T19:10:09.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.791+0000 7fde7dd9b700 1 -- 172.21.15.1:0/5283430 learned_addr learned my addr 172.21.15.1:0/5283430 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:10:09.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7ed9d700 1 --2- 172.21.15.1:0/5283430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde8006dd50 0x7fde80072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:09.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7d59a700 1 -- 172.21.15.1:0/5283430 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2247155934 0 0) 0x7fde8006f490 con 0x7fde8006e290 2024-03-18T19:10:09.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7d59a700 1 -- 172.21.15.1:0/5283430 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde680035b0 con 0x7fde8006e290 2024-03-18T19:10:09.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7d59a700 1 -- 172.21.15.1:0/5283430 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 248900602 0 0) 0x7fde680035b0 con 0x7fde8006e290 2024-03-18T19:10:09.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7d59a700 1 -- 172.21.15.1:0/5283430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde8006dd50 msgr2=0x7fde80072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:09.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7d59a700 1 --2- 172.21.15.1:0/5283430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde8006dd50 0x7fde80072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:09.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7d59a700 1 -- 172.21.15.1:0/5283430 >> v1:172.21.15.165:6789/0 conn(0x7fde8015d580 legacy=0x7fde8006d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:09.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7d59a700 1 -- 172.21.15.1:0/5283430 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde800716a0 con 0x7fde8006e290 2024-03-18T19:10:09.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7d59a700 1 -- 172.21.15.1:0/5283430 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fde70001670 con 0x7fde8006e290 2024-03-18T19:10:09.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7d59a700 1 -- 172.21.15.1:0/5283430 >> v1:172.21.15.1:6789/0 conn(0x7fde8006e290 legacy=0x7fde800ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:09.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7d59a700 1 --2- 172.21.15.1:0/5283430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde68003a30 0x7fde68003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:09.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7d59a700 1 --2- 172.21.15.1:0/5283430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde8006dd50 0x7fde68004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:09.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7d59a700 1 --2- 172.21.15.1:0/5283430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde68004d90 0x7fde68005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:09.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7d59a700 1 -- 172.21.15.1:0/5283430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8006f1b0 con 0x7fde68004d90 2024-03-18T19:10:09.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7d59a700 1 -- 172.21.15.1:0/5283430 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6800bb80 con 0x7fde68003a30 2024-03-18T19:10:09.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7d59a700 1 -- 172.21.15.1:0/5283430 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde6800bd40 con 0x7fde8006dd50 2024-03-18T19:10:09.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7ed9d700 1 --2- 172.21.15.1:0/5283430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde68004d90 0x7fde68005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:09.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7e59c700 1 --2- 172.21.15.1:0/5283430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde68003a30 0x7fde68003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:09.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7dd9b700 1 --2- 172.21.15.1:0/5283430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde8006dd50 0x7fde68004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:09.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7ed9d700 1 -- 172.21.15.1:0/5283430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde8006dd50 msgr2=0x7fde68004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:09.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7ed9d700 1 --2- 172.21.15.1:0/5283430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde8006dd50 0x7fde68004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:09.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7ed9d700 1 -- 172.21.15.1:0/5283430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde68003a30 msgr2=0x7fde68003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:09.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7ed9d700 1 --2- 172.21.15.1:0/5283430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde68003a30 0x7fde68003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:09.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.792+0000 7fde7ed9d700 1 -- 172.21.15.1:0/5283430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde680035b0 con 0x7fde68004d90 2024-03-18T19:10:09.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.793+0000 7fde7ed9d700 1 --2- 172.21.15.1:0/5283430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde68004d90 0x7fde68005210 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fde800ca510 tx=0x7fde68005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68a927ccd0455d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:09.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.793+0000 7fde7d59a700 1 -- 172.21.15.1:0/5283430 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7800eba0 con 0x7fde68004d90 2024-03-18T19:10:09.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.793+0000 7fde7d59a700 1 -- 172.21.15.1:0/5283430 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7800f040 con 0x7fde68004d90 2024-03-18T19:10:09.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.793+0000 7fde7d59a700 1 -- 172.21.15.1:0/5283430 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78014480 con 0x7fde68004d90 2024-03-18T19:10:09.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.793+0000 7fde84d32700 1 -- 172.21.15.1:0/5283430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde68004d90 msgr2=0x7fde68005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:09.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.793+0000 7fde84d32700 1 --2- 172.21.15.1:0/5283430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde68004d90 0x7fde68005210 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fde800ca510 tx=0x7fde68005880 comp rx=0 tx=0).stop 2024-03-18T19:10:09.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.793+0000 7fde7ed9d700 1 -- 172.21.15.1:0/5283430 reap_dead start 2024-03-18T19:10:09.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.793+0000 7fde84d32700 1 -- 172.21.15.1:0/5283430 shutdown_connections 2024-03-18T19:10:09.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.793+0000 7fde84d32700 1 -- 172.21.15.1:0/5283430 >> 172.21.15.1:0/5283430 conn(0x7fde800cc990 msgr2=0x7fde800ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:09.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.793+0000 7fde84d32700 1 -- 172.21.15.1:0/5283430 shutdown_connections 2024-03-18T19:10:09.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.793+0000 7fde84d32700 1 -- 172.21.15.1:0/5283430 wait complete. 2024-03-18T19:10:09.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.795+0000 7fde84d32700 1 Processor -- start 2024-03-18T19:10:09.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.795+0000 7fde84d32700 1 -- start start 2024-03-18T19:10:09.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.795+0000 7fde84d32700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde68004d90 0x7fde80152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:09.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.795+0000 7fde84d32700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde800b2950 0x7fde8015f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:09.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.795+0000 7fde84d32700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde800b2dd0 0x7fde8015f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:09.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.795+0000 7fde84d32700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde800b32c0 con 0x7fde800b2dd0 2024-03-18T19:10:09.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.795+0000 7fde84d32700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde800b35a0 con 0x7fde68004d90 2024-03-18T19:10:09.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.795+0000 7fde84d32700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde800b3880 con 0x7fde800b2950 2024-03-18T19:10:09.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.796+0000 7fde7ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde800b2dd0 0x7fde8015f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:09.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.796+0000 7fde7ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde800b2dd0 0x7fde8015f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50950/0 (socket says 172.21.15.1:50950) 2024-03-18T19:10:09.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.796+0000 7fde7ed9d700 1 -- 172.21.15.1:0/1405266933 learned_addr learned my addr 172.21.15.1:0/1405266933 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:09.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.796+0000 7fde7e59c700 1 --2- 172.21.15.1:0/1405266933 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde68004d90 0x7fde80152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:09.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.796+0000 7fde7dd9b700 1 --2- 172.21.15.1:0/1405266933 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde800b2950 0x7fde8015f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:09.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.796+0000 7fde7ed9d700 1 -- 172.21.15.1:0/1405266933 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde800b2950 msgr2=0x7fde8015f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:09.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.796+0000 7fde7ed9d700 1 --2- 172.21.15.1:0/1405266933 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde800b2950 0x7fde8015f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:09.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.796+0000 7fde7ed9d700 1 -- 172.21.15.1:0/1405266933 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde68004d90 msgr2=0x7fde80152920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:09.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.796+0000 7fde7ed9d700 1 --2- 172.21.15.1:0/1405266933 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde68004d90 0x7fde80152920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:09.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.796+0000 7fde7ed9d700 1 -- 172.21.15.1:0/1405266933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde800b3b60 con 0x7fde800b2dd0 2024-03-18T19:10:09.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.796+0000 7fde7ed9d700 1 --2- 172.21.15.1:0/1405266933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde800b2dd0 0x7fde8015f6c0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fde78015000 tx=0x7fde78009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:09.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.796+0000 7fde6effd700 1 -- 172.21.15.1:0/1405266933 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78003c20 con 0x7fde800b2dd0 2024-03-18T19:10:09.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.796+0000 7fde6effd700 1 -- 172.21.15.1:0/1405266933 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde78003d80 con 0x7fde800b2dd0 2024-03-18T19:10:09.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.796+0000 7fde84d32700 1 -- 172.21.15.1:0/1405266933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde800b3e40 con 0x7fde800b2dd0 2024-03-18T19:10:09.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.796+0000 7fde6effd700 1 -- 172.21.15.1:0/1405266933 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78014c00 con 0x7fde800b2dd0 2024-03-18T19:10:09.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.796+0000 7fde84d32700 1 -- 172.21.15.1:0/1405266933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde800b4890 con 0x7fde800b2dd0 2024-03-18T19:10:09.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.797+0000 7fde84d32700 1 -- 172.21.15.1:0/1405266933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde800b4ab0 con 0x7fde800b2dd0 2024-03-18T19:10:09.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.798+0000 7fde6effd700 1 -- 172.21.15.1:0/1405266933 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fde78003f30 con 0x7fde800b2dd0 2024-03-18T19:10:09.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.798+0000 7fde6effd700 1 --2- 172.21.15.1:0/1405266933 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fde6006adb0 0x7fde6006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:09.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.798+0000 7fde6effd700 1 -- 172.21.15.1:0/1405266933 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fde7808af90 con 0x7fde800b2dd0 2024-03-18T19:10:09.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.798+0000 7fde7e59c700 1 --2- 172.21.15.1:0/1405266933 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fde6006adb0 0x7fde6006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:09.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.798+0000 7fde7e59c700 1 --2- 172.21.15.1:0/1405266933 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fde6006adb0 0x7fde6006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fde800aff60 tx=0x7fde74009040 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:09.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:09.803+0000 7fde6effd700 1 -- 172.21.15.1:0/1405266933 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fde800b4ab0 con 0x7fde800b2dd0 2024-03-18T19:10:10.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.049+0000 7fde84d32700 1 -- 172.21.15.1:0/1405266933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde800b4ab0 con 0x7fde800b2dd0 2024-03-18T19:10:10.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.050+0000 7fde6effd700 1 -- 172.21.15.1:0/1405266933 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4736 (secure 0 0 0) 0x7fde800b4ab0 con 0x7fde800b2dd0 2024-03-18T19:10:10.047 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:10:10.047 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8608,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3436643592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3436643592},{"type":"v1","addr":"172.21.15.165:6834","nonce":3436643592}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":8611,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1969816196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1969816196},{"type":"v1","addr":"172.21.15.165:6835","nonce":1969816196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":9375,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2671786416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2671786416},{"type":"v1","addr":"172.21.15.1:6836","nonce":2671786416}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:10:04.613212+0000","modified":"2024-03-18T19:10:07.625525+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9378":{"gid":9378,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1696108841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1696108841},{"type":"v1","addr":"172.21.15.1:6837","nonce":1696108841}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-03-18T19:10:10.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.052+0000 7fde84d32700 1 -- 172.21.15.1:0/1405266933 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fde6006adb0 msgr2=0x7fde6006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:10.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.052+0000 7fde84d32700 1 --2- 172.21.15.1:0/1405266933 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fde6006adb0 0x7fde6006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fde800aff60 tx=0x7fde74009040 comp rx=0 tx=0).stop 2024-03-18T19:10:10.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.052+0000 7fde84d32700 1 -- 172.21.15.1:0/1405266933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde800b2dd0 msgr2=0x7fde8015f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:10.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.052+0000 7fde84d32700 1 --2- 172.21.15.1:0/1405266933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde800b2dd0 0x7fde8015f6c0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fde78015000 tx=0x7fde78009750 comp rx=0 tx=0).stop 2024-03-18T19:10:10.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.052+0000 7fde84d32700 1 -- 172.21.15.1:0/1405266933 shutdown_connections 2024-03-18T19:10:10.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.052+0000 7fde84d32700 1 --2- 172.21.15.1:0/1405266933 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fde6006adb0 0x7fde6006d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.052+0000 7fde84d32700 1 --2- 172.21.15.1:0/1405266933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde800b2dd0 0x7fde8015f6c0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.052+0000 7fde84d32700 1 --2- 172.21.15.1:0/1405266933 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde68004d90 0x7fde80152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.052+0000 7fde84d32700 1 --2- 172.21.15.1:0/1405266933 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde800b2950 0x7fde8015f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.052+0000 7fde84d32700 1 -- 172.21.15.1:0/1405266933 >> 172.21.15.1:0/1405266933 conn(0x7fde800cc990 msgr2=0x7fde800cade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:10.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.052+0000 7fde84d32700 1 -- 172.21.15.1:0/1405266933 shutdown_connections 2024-03-18T19:10:10.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.052+0000 7fde84d32700 1 -- 172.21.15.1:0/1405266933 wait complete. 2024-03-18T19:10:10.050 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 73 2024-03-18T19:10:10.166 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-18T19:10:10.170+0000 7f91d1543700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T19:10:10.464 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T19:10:04.613212+0000', 'modified': '2024-03-18T19:10:07.625525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9378}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9378': {'gid': 9378, 'name': 'c', 'rank': 0, 'incarnation': 69, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6837/1696108841', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6835', 'nonce': 1696108841}, {'type': 'v1', 'addr': '172.21.15.1:6837', 'nonce': 1696108841}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T19:10:10.464 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T19:10:10.464 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T19:10:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.598+0000 7ff5c8132700 1 Processor -- start 2024-03-18T19:10:10.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.599+0000 7ff5c8132700 1 -- start start 2024-03-18T19:10:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.599+0000 7ff5c8132700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5c0143c60 0x7ff5c006eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.599+0000 7ff5c8132700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff5c00b4a50 0x7ff5c00ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.599+0000 7ff5c8132700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c006f430 con 0x7ff5c0072330 2024-03-18T19:10:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.599+0000 7ff5c8132700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5c006f710 con 0x7ff5c00b4a50 2024-03-18T19:10:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.599+0000 7ff5c8132700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c00cda00 con 0x7ff5c0143c60 2024-03-18T19:10:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.600+0000 7ff5c56cd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff5c0072330 0x7ff5c00b4340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52644/0 (socket says 172.21.15.1:52644) 2024-03-18T19:10:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.600+0000 7ff5c56cd700 1 -- 172.21.15.1:0/627404080 learned_addr learned my addr 172.21.15.1:0/627404080 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:10:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.600+0000 7ff5c66cf700 1 --2- 172.21.15.1:0/627404080 >> v2:172.21.15.1:3300/0 conn(0x7ff5c00b4a50 0x7ff5c00ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.600+0000 7ff5c5ece700 1 --2- 172.21.15.1:0/627404080 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5c0143c60 0x7ff5c006eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.600+0000 7ff5c4ecc700 1 -- 172.21.15.1:0/627404080 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938164135 0 0) 0x7ff5c006f430 con 0x7ff5c0072330 2024-03-18T19:10:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.600+0000 7ff5c4ecc700 1 -- 172.21.15.1:0/627404080 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5ac0035b0 con 0x7ff5c0072330 2024-03-18T19:10:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.600+0000 7ff5c66cf700 1 -- 172.21.15.1:0/627404080 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5c0143c60 msgr2=0x7ff5c006eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.600+0000 7ff5c66cf700 1 --2- 172.21.15.1:0/627404080 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5c0143c60 0x7ff5c006eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.600+0000 7ff5c66cf700 1 -- 172.21.15.1:0/627404080 >> v1:172.21.15.1:6789/0 conn(0x7ff5c0072330 legacy=0x7ff5c00b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.600+0000 7ff5c66cf700 1 -- 172.21.15.1:0/627404080 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c00cdce0 con 0x7ff5c00b4a50 2024-03-18T19:10:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.600+0000 7ff5c66cf700 1 --2- 172.21.15.1:0/627404080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5c00b4a50 0x7ff5c00ce4e0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8009d90 tx=0x7ff5b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdbace2c8f83c1e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:10.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.600+0000 7ff5c4ecc700 1 -- 172.21.15.1:0/627404080 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b800e070 con 0x7ff5c00b4a50 2024-03-18T19:10:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.600+0000 7ff5c4ecc700 1 -- 172.21.15.1:0/627404080 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b800bb20 con 0x7ff5c00b4a50 2024-03-18T19:10:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.600+0000 7ff5c4ecc700 1 -- 172.21.15.1:0/627404080 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b8012430 con 0x7ff5c00b4a50 2024-03-18T19:10:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.601+0000 7ff5c8132700 1 -- 172.21.15.1:0/627404080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5c00b4a50 msgr2=0x7ff5c00ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.601+0000 7ff5c8132700 1 --2- 172.21.15.1:0/627404080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5c00b4a50 0x7ff5c00ce4e0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8009d90 tx=0x7ff5b800b330 comp rx=0 tx=0).stop 2024-03-18T19:10:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.601+0000 7ff5c8132700 1 -- 172.21.15.1:0/627404080 shutdown_connections 2024-03-18T19:10:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.601+0000 7ff5c8132700 1 --2- 172.21.15.1:0/627404080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5c00b4a50 0x7ff5c00ce4e0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.601+0000 7ff5c8132700 1 --2- 172.21.15.1:0/627404080 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5c0143c60 0x7ff5c006eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.601+0000 7ff5c8132700 1 -- 172.21.15.1:0/627404080 >> 172.21.15.1:0/627404080 conn(0x7ff5c00b1d20 msgr2=0x7ff5c00b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.601+0000 7ff5c8132700 1 -- 172.21.15.1:0/627404080 shutdown_connections 2024-03-18T19:10:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.601+0000 7ff5c8132700 1 -- 172.21.15.1:0/627404080 wait complete. 2024-03-18T19:10:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.602+0000 7ff5c8132700 1 Processor -- start 2024-03-18T19:10:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.602+0000 7ff5c8132700 1 -- start start 2024-03-18T19:10:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.603+0000 7ff5c8132700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5c0072330 0x7ff5c013f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.603+0000 7ff5c8132700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5c00b4a50 0x7ff5c015c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.603+0000 7ff5c8132700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5c0143c60 0x7ff5c015c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.603+0000 7ff5c8132700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c00af6e0 con 0x7ff5c00b4a50 2024-03-18T19:10:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.603+0000 7ff5c8132700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c00ae870 con 0x7ff5c0072330 2024-03-18T19:10:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.603+0000 7ff5c8132700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c00aea60 con 0x7ff5c0143c60 2024-03-18T19:10:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.603+0000 7ff5c56cd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5c00b4a50 0x7ff5c015c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.603+0000 7ff5c56cd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5c00b4a50 0x7ff5c015c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50960/0 (socket says 172.21.15.1:50960) 2024-03-18T19:10:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.603+0000 7ff5c56cd700 1 -- 172.21.15.1:0/3229101935 learned_addr learned my addr 172.21.15.1:0/3229101935 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.603+0000 7ff5c66cf700 1 --2- 172.21.15.1:0/3229101935 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5c0143c60 0x7ff5c015c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.603+0000 7ff5c5ece700 1 --2- 172.21.15.1:0/3229101935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5c0072330 0x7ff5c013f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.603+0000 7ff5c56cd700 1 -- 172.21.15.1:0/3229101935 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5c0143c60 msgr2=0x7ff5c015c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.603+0000 7ff5c56cd700 1 --2- 172.21.15.1:0/3229101935 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5c0143c60 0x7ff5c015c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.603+0000 7ff5c56cd700 1 -- 172.21.15.1:0/3229101935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5c0072330 msgr2=0x7ff5c013f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.603+0000 7ff5c56cd700 1 --2- 172.21.15.1:0/3229101935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5c0072330 0x7ff5c013f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.603+0000 7ff5c56cd700 1 -- 172.21.15.1:0/3229101935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b8009a70 con 0x7ff5c00b4a50 2024-03-18T19:10:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.603+0000 7ff5c66cf700 1 --2- 172.21.15.1:0/3229101935 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5c0143c60 0x7ff5c015c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:10:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.603+0000 7ff5c5ece700 1 --2- 172.21.15.1:0/3229101935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5c0072330 0x7ff5c013f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:10:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.603+0000 7ff5c56cd700 1 --2- 172.21.15.1:0/3229101935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5c00b4a50 0x7ff5c015c1c0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc005910 tx=0x7ff5bc00ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.604+0000 7ff5b27fc700 1 -- 172.21.15.1:0/3229101935 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc005d70 con 0x7ff5c00b4a50 2024-03-18T19:10:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.604+0000 7ff5b27fc700 1 -- 172.21.15.1:0/3229101935 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5bc0070d0 con 0x7ff5c00b4a50 2024-03-18T19:10:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.604+0000 7ff5c8132700 1 -- 172.21.15.1:0/3229101935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c00aed40 con 0x7ff5c00b4a50 2024-03-18T19:10:10.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.604+0000 7ff5b27fc700 1 -- 172.21.15.1:0/3229101935 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc007460 con 0x7ff5c00b4a50 2024-03-18T19:10:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.604+0000 7ff5c8132700 1 -- 172.21.15.1:0/3229101935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c00af300 con 0x7ff5c00b4a50 2024-03-18T19:10:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.605+0000 7ff5b27fc700 1 -- 172.21.15.1:0/3229101935 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff5bc0075c0 con 0x7ff5c00b4a50 2024-03-18T19:10:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.605+0000 7ff5b27fc700 1 --2- 172.21.15.1:0/3229101935 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff5a406adb0 0x7ff5a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.606+0000 7ff5b27fc700 1 -- 172.21.15.1:0/3229101935 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff5bc088bd0 con 0x7ff5c00b4a50 2024-03-18T19:10:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.606+0000 7ff5c5ece700 1 --2- 172.21.15.1:0/3229101935 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff5a406adb0 0x7ff5a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.606+0000 7ff5c5ece700 1 --2- 172.21.15.1:0/3229101935 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff5a406adb0 0x7ff5a406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4009b90 tx=0x7ff5b4008040 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:10.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.608+0000 7ff5d42dbb80 1 Processor -- start 2024-03-18T19:10:10.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.608+0000 7ff5d42dbb80 1 -- start start 2024-03-18T19:10:10.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.608+0000 7ff5d42dbb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d6ec10 0x55cd87d6eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.608+0000 7ff5d42dbb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d400a0 0x55cd87d8b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.608+0000 7ff5d42dbb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d80630 0x55cd87d43160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.608+0000 7ff5d42dbb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd87cb9f20 con 0x55cd87d80630 2024-03-18T19:10:10.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.608+0000 7ff5d42dbb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd87c9bc30 con 0x55cd87d6ec10 2024-03-18T19:10:10.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.608+0000 7ff5d42dbb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd87c99430 con 0x55cd87d400a0 2024-03-18T19:10:10.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.609+0000 7ff5c66cf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d80630 0x55cd87d43160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.609+0000 7ff5c66cf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d80630 0x55cd87d43160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50976/0 (socket says 172.21.15.1:50976) 2024-03-18T19:10:10.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.609+0000 7ff5c66cf700 1 -- 172.21.15.1:0/2425751789 learned_addr learned my addr 172.21.15.1:0/2425751789 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:10.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.609+0000 7ff5c56cd700 1 --2- 172.21.15.1:0/2425751789 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d400a0 0x55cd87d8b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.609+0000 7ff5c5ece700 1 --2- 172.21.15.1:0/2425751789 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d6ec10 0x55cd87d6eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.609+0000 7ff5c66cf700 1 -- 172.21.15.1:0/2425751789 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d400a0 msgr2=0x55cd87d8b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:10.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.609+0000 7ff5c66cf700 1 --2- 172.21.15.1:0/2425751789 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d400a0 0x55cd87d8b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.609+0000 7ff5c66cf700 1 -- 172.21.15.1:0/2425751789 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d6ec10 msgr2=0x55cd87d6eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:10.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.609+0000 7ff5c66cf700 1 --2- 172.21.15.1:0/2425751789 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d6ec10 0x55cd87d6eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.609+0000 7ff5c66cf700 1 -- 172.21.15.1:0/2425751789 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd87a88790 con 0x55cd87d80630 2024-03-18T19:10:10.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.609+0000 7ff5c66cf700 1 --2- 172.21.15.1:0/2425751789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d80630 0x55cd87d43160 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7ff5b800b370 tx=0x7ff5b8010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79bc1f28f454c1f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:10.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.609+0000 7ff5a37fe700 1 -- 172.21.15.1:0/2425751789 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b8003a20 con 0x55cd87d80630 2024-03-18T19:10:10.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.609+0000 7ff5a37fe700 1 -- 172.21.15.1:0/2425751789 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b8003b80 con 0x55cd87d80630 2024-03-18T19:10:10.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.609+0000 7ff5a37fe700 1 -- 172.21.15.1:0/2425751789 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b80125e0 con 0x55cd87d80630 2024-03-18T19:10:10.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.610+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/2425751789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d80630 msgr2=0x55cd87d43160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:10.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.610+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/2425751789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d80630 0x55cd87d43160 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7ff5b800b370 tx=0x7ff5b8010000 comp rx=0 tx=0).stop 2024-03-18T19:10:10.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.610+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/2425751789 shutdown_connections 2024-03-18T19:10:10.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.610+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/2425751789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d80630 0x55cd87d43160 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.610+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/2425751789 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d6ec10 0x55cd87d6eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.610+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/2425751789 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d400a0 0x55cd87d8b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.610+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/2425751789 >> 172.21.15.1:0/2425751789 conn(0x55cd87ca2cf0 msgr2=0x55cd87d42a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:10.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.610+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/2425751789 shutdown_connections 2024-03-18T19:10:10.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.610+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/2425751789 wait complete. 2024-03-18T19:10:10.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.611+0000 7ff5d42dbb80 1 -- start start 2024-03-18T19:10:10.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.611+0000 7ff5d42dbb80 1 Processor -- start 2024-03-18T19:10:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5d42dbb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d400a0 0x55cd87d68670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5d42dbb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d6ec10 0x55cd87d68bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5d42dbb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d80630 0x55cd87d81630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5d42dbb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd87c94d20 con 0x55cd87d6ec10 2024-03-18T19:10:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5d42dbb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd87c30770 con 0x55cd87d400a0 2024-03-18T19:10:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5d42dbb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd87c536f0 con 0x55cd87d80630 2024-03-18T19:10:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5d42dbb80 10 client.0 _ll_register_callbacks cb 0x55cd87a9c5a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T19:10:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5c56cd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d6ec10 0x55cd87d68bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5c56cd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d6ec10 0x55cd87d68bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50982/0 (socket says 172.21.15.1:50982) 2024-03-18T19:10:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5c56cd700 1 -- 172.21.15.1:0/1200209987 learned_addr learned my addr 172.21.15.1:0/1200209987 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5c5ece700 1 --2- 172.21.15.1:0/1200209987 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d400a0 0x55cd87d68670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5c66cf700 1 --2- 172.21.15.1:0/1200209987 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d80630 0x55cd87d81630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5c56cd700 1 -- 172.21.15.1:0/1200209987 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d80630 msgr2=0x55cd87d81630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5c56cd700 1 --2- 172.21.15.1:0/1200209987 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d80630 0x55cd87d81630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5c56cd700 1 -- 172.21.15.1:0/1200209987 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d400a0 msgr2=0x55cd87d68670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5c56cd700 1 --2- 172.21.15.1:0/1200209987 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d400a0 0x55cd87d68670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5c56cd700 1 -- 172.21.15.1:0/1200209987 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c00cdce0 con 0x55cd87d6ec10 2024-03-18T19:10:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5c56cd700 1 --2- 172.21.15.1:0/1200209987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d6ec10 0x55cd87d68bb0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc0011e0 tx=0x7ff5bc058020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ff533e1b66e07c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5a0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T19:10:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5a0ff9700 1 -- 172.21.15.1:0/1200209987 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc05fbe0 con 0x55cd87d6ec10 2024-03-18T19:10:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5a0ff9700 1 -- 172.21.15.1:0/1200209987 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff5bc06f170 con 0x55cd87d6ec10 2024-03-18T19:10:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/1200209987 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cd87d36e90 con 0x55cd87d6ec10 2024-03-18T19:10:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5a0ff9700 1 -- 172.21.15.1:0/1200209987 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5bc05f750 con 0x55cd87d6ec10 2024-03-18T19:10:10.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5d42dbb80 10 client.9519 fetch_fsmap learned FSMap version 73 2024-03-18T19:10:10.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.612+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/1200209987 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x55cd87b750a0 con 0x55cd87d6ec10 2024-03-18T19:10:10.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.613+0000 7ff5a0ff9700 1 -- 172.21.15.1:0/1200209987 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55cd87d36e90 con 0x55cd87d6ec10 2024-03-18T19:10:10.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.613+0000 7ff5a0ff9700 1 -- 172.21.15.1:0/1200209987 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 73) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff5bc05b170 con 0x55cd87d6ec10 2024-03-18T19:10:10.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.613+0000 7ff5d42dbb80 10 client.9519 fetch_fsmap finished waiting for FSMap version 73 2024-03-18T19:10:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.613+0000 7ff5d42dbb80 10 client.9519 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T19:10:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.613+0000 7ff5d42dbb80 20 client.9519 populate_metadata read hostname 'smithi001' 2024-03-18T19:10:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.613+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/1200209987 >> [v2:172.21.15.1:6835/1696108841,v1:172.21.15.1:6837/1696108841] conn(0x55cd87d64fe0 0x55cd87d65bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.613+0000 7ff5d42dbb80 4 client.9519 mds_command: new command op to 9378 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T19:10:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.613+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/1200209987 --> [v2:172.21.15.1:6835/1696108841,v1:172.21.15.1:6837/1696108841] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cd87d14010 con 0x55cd87d64fe0 2024-03-18T19:10:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.613+0000 7ff5c5ece700 1 --2- 172.21.15.1:0/1200209987 >> [v2:172.21.15.1:6835/1696108841,v1:172.21.15.1:6837/1696108841] conn(0x55cd87d64fe0 0x55cd87d65bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.613+0000 7ff5a0ff9700 10 client.9519 ms_handle_connect on v2:172.21.15.1:6835/1696108841 2024-03-18T19:10:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.613+0000 7ff5c5ece700 1 --2- 172.21.15.1:0/1200209987 >> [v2:172.21.15.1:6835/1696108841,v1:172.21.15.1:6837/1696108841] conn(0x55cd87d64fe0 0x55cd87d65bf0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90eb7c9664d56733 server_cookie=1b82dadd69d2fa66 in_seq=0 out_seq=0 2024-03-18T19:10:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.614+0000 7ff5a0ff9700 1 -- 172.21.15.1:0/1200209987 <== mds.0 v2:172.21.15.1:6835/1696108841 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55cd87d14010 con 0x55cd87d64fe0 2024-03-18T19:10:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.614+0000 7ff5a0ff9700 10 client.9519 handle_command_reply: tid=0 2024-03-18T19:10:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.614+0000 7ff5d42dbb80 1 client.9519 shutdown 2024-03-18T19:10:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.614+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/1200209987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d6ec10 msgr2=0x55cd87d68bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:10.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.614+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/1200209987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d6ec10 0x55cd87d68bb0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc0011e0 tx=0x7ff5bc058020 comp rx=0 tx=0).stop 2024-03-18T19:10:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.614+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/1200209987 shutdown_connections 2024-03-18T19:10:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.614+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/1200209987 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d80630 0x55cd87d81630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.614+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/1200209987 >> [v2:172.21.15.1:6835/1696108841,v1:172.21.15.1:6837/1696108841] conn(0x55cd87d64fe0 0x55cd87d65bf0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.614+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/1200209987 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d400a0 0x55cd87d68670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.614+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/1200209987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d6ec10 0x55cd87d68bb0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.614+0000 7ff5a0ff9700 0 client.9519 ms_handle_reset on v2:172.21.15.1:6835/1696108841 2024-03-18T19:10:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.614+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/1200209987 >> 172.21.15.1:0/1200209987 conn(0x55cd87ca2cf0 msgr2=0x55cd87d3e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.614+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/1200209987 shutdown_connections 2024-03-18T19:10:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.614+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/1200209987 wait complete. 2024-03-18T19:10:10.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.614+0000 7ff5d42dbb80 20 client.9519 trim_cache size 0 max 16384 2024-03-18T19:10:10.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.632+0000 7ff5d42dbb80 1 Processor -- start 2024-03-18T19:10:10.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.632+0000 7ff5d42dbb80 1 -- start start 2024-03-18T19:10:10.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5d42dbb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d400a0 0x55cd87d69450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5d42dbb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d6ec10 0x55cd87d66c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5d42dbb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d80630 0x55cd87d67140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5d42dbb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd87a89390 con 0x55cd87d400a0 2024-03-18T19:10:10.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5d42dbb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd87d3dba0 con 0x55cd87d6ec10 2024-03-18T19:10:10.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5d42dbb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd87d68750 con 0x55cd87d80630 2024-03-18T19:10:10.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5c5ece700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d400a0 0x55cd87d69450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5c5ece700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d400a0 0x55cd87d69450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50998/0 (socket says 172.21.15.1:50998) 2024-03-18T19:10:10.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5c5ece700 1 -- 172.21.15.1:0/3521002142 learned_addr learned my addr 172.21.15.1:0/3521002142 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:10.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5c66cf700 1 --2- 172.21.15.1:0/3521002142 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d80630 0x55cd87d67140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5c56cd700 1 --2- 172.21.15.1:0/3521002142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d6ec10 0x55cd87d66c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5c5ece700 1 -- 172.21.15.1:0/3521002142 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d80630 msgr2=0x55cd87d67140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:10.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5c5ece700 1 --2- 172.21.15.1:0/3521002142 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d80630 0x55cd87d67140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5c5ece700 1 -- 172.21.15.1:0/3521002142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d6ec10 msgr2=0x55cd87d66c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:10.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5c5ece700 1 --2- 172.21.15.1:0/3521002142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d6ec10 0x55cd87d66c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5c5ece700 1 -- 172.21.15.1:0/3521002142 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd87c5b010 con 0x55cd87d400a0 2024-03-18T19:10:10.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5c66cf700 1 --2- 172.21.15.1:0/3521002142 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d80630 0x55cd87d67140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:10:10.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5c56cd700 1 --2- 172.21.15.1:0/3521002142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d6ec10 0x55cd87d66c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T19:10:10.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5c5ece700 1 --2- 172.21.15.1:0/3521002142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d400a0 0x55cd87d69450 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4009610 tx=0x7ff5b4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e382bcb5ac41c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:10.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5a3fff700 1 -- 172.21.15.1:0/3521002142 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b40044c0 con 0x55cd87d400a0 2024-03-18T19:10:10.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5a3fff700 1 -- 172.21.15.1:0/3521002142 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b4011b90 con 0x55cd87d400a0 2024-03-18T19:10:10.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.633+0000 7ff5a3fff700 1 -- 172.21.15.1:0/3521002142 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b4011d00 con 0x55cd87d400a0 2024-03-18T19:10:10.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.634+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/3521002142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d400a0 msgr2=0x55cd87d69450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:10.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.634+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/3521002142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d400a0 0x55cd87d69450 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4009610 tx=0x7ff5b4014000 comp rx=0 tx=0).stop 2024-03-18T19:10:10.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.634+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/3521002142 shutdown_connections 2024-03-18T19:10:10.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.634+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/3521002142 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d80630 0x55cd87d67140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.634+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/3521002142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d400a0 0x55cd87d69450 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.634+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/3521002142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d6ec10 0x55cd87d66c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.634+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/3521002142 >> 172.21.15.1:0/3521002142 conn(0x55cd87ca2cf0 msgr2=0x55cd87d44620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:10.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.634+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/3521002142 shutdown_connections 2024-03-18T19:10:10.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.634+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/3521002142 wait complete. 2024-03-18T19:10:10.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.635+0000 7ff5d42dbb80 1 -- start start 2024-03-18T19:10:10.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.635+0000 7ff5d42dbb80 1 Processor -- start 2024-03-18T19:10:10.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.636+0000 7ff5d42dbb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d400a0 0x55cd87d8d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.636+0000 7ff5d42dbb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d6ec10 0x55cd87dea8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.636+0000 7ff5d42dbb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d80630 0x55cd87d61ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.636+0000 7ff5d42dbb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd87deadf0 con 0x55cd87d6ec10 2024-03-18T19:10:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.636+0000 7ff5d42dbb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd87d625a0 con 0x55cd87d80630 2024-03-18T19:10:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.636+0000 7ff5d42dbb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd87d8a420 con 0x55cd87d400a0 2024-03-18T19:10:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.636+0000 7ff5d42dbb80 10 client.0 _ll_register_callbacks cb 0x7ff5bc057330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T19:10:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.636+0000 7ff5c56cd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d6ec10 0x55cd87dea8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.636+0000 7ff5c56cd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d6ec10 0x55cd87dea8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51014/0 (socket says 172.21.15.1:51014) 2024-03-18T19:10:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.636+0000 7ff5c56cd700 1 -- 172.21.15.1:0/548103877 learned_addr learned my addr 172.21.15.1:0/548103877 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.636+0000 7ff5c5ece700 1 --2- 172.21.15.1:0/548103877 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d400a0 0x55cd87d8d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.636+0000 7ff5c66cf700 1 --2- 172.21.15.1:0/548103877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d80630 0x55cd87d61ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.636+0000 7ff5c56cd700 1 -- 172.21.15.1:0/548103877 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d400a0 msgr2=0x55cd87d8d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:10.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.636+0000 7ff5c56cd700 1 --2- 172.21.15.1:0/548103877 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d400a0 0x55cd87d8d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.636+0000 7ff5c56cd700 1 -- 172.21.15.1:0/548103877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d80630 msgr2=0x55cd87d61ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.636+0000 7ff5c56cd700 1 --2- 172.21.15.1:0/548103877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d80630 0x55cd87d61ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.636+0000 7ff5c56cd700 1 -- 172.21.15.1:0/548103877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b401b040 con 0x55cd87d6ec10 2024-03-18T19:10:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.636+0000 7ff5c56cd700 1 --2- 172.21.15.1:0/548103877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d6ec10 0x55cd87dea8b0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc000f60 tx=0x7ff5bc007870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca0dc364f2fa89f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.636+0000 7ff5a0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T19:10:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.637+0000 7ff5a0ff9700 1 -- 172.21.15.1:0/548103877 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc00e040 con 0x55cd87d6ec10 2024-03-18T19:10:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.637+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/548103877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cd87d6bd00 con 0x55cd87d6ec10 2024-03-18T19:10:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.637+0000 7ff5a0ff9700 1 -- 172.21.15.1:0/548103877 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff5bc06f170 con 0x55cd87d6ec10 2024-03-18T19:10:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.637+0000 7ff5a0ff9700 1 -- 172.21.15.1:0/548103877 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5bc05fbe0 con 0x55cd87d6ec10 2024-03-18T19:10:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.637+0000 7ff5d42dbb80 10 client.9525 fetch_fsmap learned FSMap version 73 2024-03-18T19:10:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.637+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/548103877 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x55cd87ae91d0 con 0x55cd87d6ec10 2024-03-18T19:10:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.637+0000 7ff5a0ff9700 1 -- 172.21.15.1:0/548103877 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55cd87d6bd00 con 0x55cd87d6ec10 2024-03-18T19:10:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.637+0000 7ff5a0ff9700 1 -- 172.21.15.1:0/548103877 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 73) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff5bc05f750 con 0x55cd87d6ec10 2024-03-18T19:10:10.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.637+0000 7ff5d42dbb80 10 client.9525 fetch_fsmap finished waiting for FSMap version 73 2024-03-18T19:10:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.637+0000 7ff5d42dbb80 10 client.9525 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T19:10:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.637+0000 7ff5d42dbb80 20 client.9525 populate_metadata read hostname 'smithi001' 2024-03-18T19:10:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.637+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/548103877 >> [v2:172.21.15.1:6835/1696108841,v1:172.21.15.1:6837/1696108841] conn(0x55cd87d429a0 0x55cd87d42d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.637+0000 7ff5d42dbb80 4 client.9525 mds_command: new command op to 9378 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T19:10:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.637+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/548103877 --> [v2:172.21.15.1:6835/1696108841,v1:172.21.15.1:6837/1696108841] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7ff5b8003b80 con 0x55cd87d429a0 2024-03-18T19:10:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.637+0000 7ff5c5ece700 1 --2- 172.21.15.1:0/548103877 >> [v2:172.21.15.1:6835/1696108841,v1:172.21.15.1:6837/1696108841] conn(0x55cd87d429a0 0x55cd87d42d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.638+0000 7ff5a0ff9700 10 client.9525 ms_handle_connect on v2:172.21.15.1:6835/1696108841 2024-03-18T19:10:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.638+0000 7ff5c5ece700 1 --2- 172.21.15.1:0/548103877 >> [v2:172.21.15.1:6835/1696108841,v1:172.21.15.1:6837/1696108841] conn(0x55cd87d429a0 0x55cd87d42d80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab2a3fa518ca9175 server_cookie=86744c8be0cbd848 in_seq=0 out_seq=0 2024-03-18T19:10:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.638+0000 7ff5a0ff9700 1 -- 172.21.15.1:0/548103877 <== mds.0 v2:172.21.15.1:6835/1696108841 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x7ff5b8003b80 con 0x55cd87d429a0 2024-03-18T19:10:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.638+0000 7ff5a0ff9700 10 client.9525 handle_command_reply: tid=0 2024-03-18T19:10:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.638+0000 7ff5d42dbb80 1 client.9525 shutdown 2024-03-18T19:10:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.638+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/548103877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d6ec10 msgr2=0x55cd87dea8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.638+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/548103877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d6ec10 0x55cd87dea8b0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc000f60 tx=0x7ff5bc007870 comp rx=0 tx=0).stop 2024-03-18T19:10:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.638+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/548103877 shutdown_connections 2024-03-18T19:10:10.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.638+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/548103877 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55cd87d400a0 0x55cd87d8d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.638+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/548103877 >> [v2:172.21.15.1:6835/1696108841,v1:172.21.15.1:6837/1696108841] conn(0x55cd87d429a0 0x55cd87d42d80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.638+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/548103877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55cd87d80630 0x55cd87d61ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.638+0000 7ff5d42dbb80 1 --2- 172.21.15.1:0/548103877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55cd87d6ec10 0x55cd87dea8b0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.638+0000 7ff5a0ff9700 0 client.9525 ms_handle_reset on v2:172.21.15.1:6835/1696108841 2024-03-18T19:10:10.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.638+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/548103877 >> 172.21.15.1:0/548103877 conn(0x55cd87ca2cf0 msgr2=0x55cd87d42220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:10.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.638+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/548103877 shutdown_connections 2024-03-18T19:10:10.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.638+0000 7ff5d42dbb80 1 -- 172.21.15.1:0/548103877 wait complete. 2024-03-18T19:10:10.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.639+0000 7ff5d42dbb80 20 client.9525 trim_cache size 0 max 16384 2024-03-18T19:10:10.641 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:10:10.641 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "9c1c697f-8655-4ef3-8a13-da568d2cbe86", 2024-03-18T19:10:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-18T19:10:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 9378, 2024-03-18T19:10:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-18T19:10:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-18T19:10:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-18T19:10:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 6.025251591, 2024-03-18T19:10:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 73, 2024-03-18T19:10:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 161, 2024-03-18T19:10:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 161, 2024-03-18T19:10:10.642 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 11.339741396999999 2024-03-18T19:10:10.642 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:10:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.639+0000 7ff5c8132700 1 -- 172.21.15.1:0/3229101935 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff5a406adb0 msgr2=0x7ff5a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.639+0000 7ff5c8132700 1 --2- 172.21.15.1:0/3229101935 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff5a406adb0 0x7ff5a406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4009b90 tx=0x7ff5b4008040 comp rx=0 tx=0).stop 2024-03-18T19:10:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.639+0000 7ff5c8132700 1 -- 172.21.15.1:0/3229101935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5c00b4a50 msgr2=0x7ff5c015c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.639+0000 7ff5c8132700 1 --2- 172.21.15.1:0/3229101935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5c00b4a50 0x7ff5c015c1c0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc005910 tx=0x7ff5bc00ca90 comp rx=0 tx=0).stop 2024-03-18T19:10:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.639+0000 7ff5c8132700 1 -- 172.21.15.1:0/3229101935 shutdown_connections 2024-03-18T19:10:10.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.639+0000 7ff5c8132700 1 --2- 172.21.15.1:0/3229101935 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff5a406adb0 0x7ff5a406d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.639+0000 7ff5c8132700 1 --2- 172.21.15.1:0/3229101935 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff5c0143c60 0x7ff5c015c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.639+0000 7ff5c8132700 1 --2- 172.21.15.1:0/3229101935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5c0072330 0x7ff5c013f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.639+0000 7ff5c8132700 1 --2- 172.21.15.1:0/3229101935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5c00b4a50 0x7ff5c015c1c0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.639+0000 7ff5c8132700 1 -- 172.21.15.1:0/3229101935 >> 172.21.15.1:0/3229101935 conn(0x7ff5c00b1d20 msgr2=0x7ff5c006a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.639+0000 7ff5c8132700 1 -- 172.21.15.1:0/3229101935 shutdown_connections 2024-03-18T19:10:10.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.639+0000 7ff5c8132700 1 -- 172.21.15.1:0/3229101935 wait complete. 2024-03-18T19:10:10.648 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T19:10:10.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.776+0000 7f69e61d6700 1 Processor -- start 2024-03-18T19:10:10.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.776+0000 7f69e61d6700 1 -- start start 2024-03-18T19:10:10.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.777+0000 7f69e61d6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69e0144b30 0x7f69e00cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.777+0000 7f69e61d6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f69e00cfa90 0x7f69e00ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.777+0000 7f69e61d6700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e0144f10 con 0x7f69e006a750 2024-03-18T19:10:10.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.777+0000 7f69e61d6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e015fcc0 con 0x7f69e00cfa90 2024-03-18T19:10:10.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.777+0000 7f69e61d6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e00cea70 con 0x7f69e0144b30 2024-03-18T19:10:10.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.777+0000 7f69deffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f69e00cfa90 0x7f69e00ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.777+0000 7f69deffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f69e00cfa90 0x7f69e00ccf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51028/0 (socket says 172.21.15.1:51028) 2024-03-18T19:10:10.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.777+0000 7f69deffd700 1 -- 172.21.15.1:0/3823186717 learned_addr learned my addr 172.21.15.1:0/3823186717 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:10.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.777+0000 7f69df7fe700 1 --2- 172.21.15.1:0/3823186717 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69e0144b30 0x7f69e00cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.777+0000 7f69deffd700 1 -- 172.21.15.1:0/3823186717 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69e0144b30 msgr2=0x7f69e00cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:10.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.777+0000 7f69deffd700 1 --2- 172.21.15.1:0/3823186717 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69e0144b30 0x7f69e00cca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.777+0000 7f69deffd700 1 -- 172.21.15.1:0/3823186717 >> v1:172.21.15.165:6789/0 conn(0x7f69e006a750 legacy=0x7f69e006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:10.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.777+0000 7f69deffd700 1 -- 172.21.15.1:0/3823186717 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e00cecd0 con 0x7f69e00cfa90 2024-03-18T19:10:10.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.777+0000 7f69deffd700 1 --2- 172.21.15.1:0/3823186717 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69e00cfa90 0x7f69e00ccf40 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f69d00098f0 tx=0x7f69d0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c4f8878a1d774a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:10.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.778+0000 7f69de7fc700 1 -- 172.21.15.1:0/3823186717 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d0004230 con 0x7f69e00cfa90 2024-03-18T19:10:10.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.778+0000 7f69de7fc700 1 -- 172.21.15.1:0/3823186717 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d0004390 con 0x7f69e00cfa90 2024-03-18T19:10:10.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.778+0000 7f69de7fc700 1 -- 172.21.15.1:0/3823186717 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d00036a0 con 0x7f69e00cfa90 2024-03-18T19:10:10.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.778+0000 7f69e61d6700 1 -- 172.21.15.1:0/3823186717 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69e00cfa90 msgr2=0x7f69e00ccf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:10.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.778+0000 7f69e61d6700 1 --2- 172.21.15.1:0/3823186717 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69e00cfa90 0x7f69e00ccf40 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f69d00098f0 tx=0x7f69d0009c00 comp rx=0 tx=0).stop 2024-03-18T19:10:10.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.778+0000 7f69e61d6700 1 -- 172.21.15.1:0/3823186717 shutdown_connections 2024-03-18T19:10:10.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.778+0000 7f69e61d6700 1 --2- 172.21.15.1:0/3823186717 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69e0144b30 0x7f69e00cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.778+0000 7f69e61d6700 1 --2- 172.21.15.1:0/3823186717 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69e00cfa90 0x7f69e00ccf40 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.778+0000 7f69e61d6700 1 -- 172.21.15.1:0/3823186717 >> 172.21.15.1:0/3823186717 conn(0x7f69e00b1d20 msgr2=0x7f69e00b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:10.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.778+0000 7f69e61d6700 1 -- 172.21.15.1:0/3823186717 shutdown_connections 2024-03-18T19:10:10.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.778+0000 7f69e61d6700 1 -- 172.21.15.1:0/3823186717 wait complete. 2024-03-18T19:10:10.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.779+0000 7f69e61d6700 1 Processor -- start 2024-03-18T19:10:10.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.780+0000 7f69e61d6700 1 -- start start 2024-03-18T19:10:10.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.780+0000 7f69e61d6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69e006a750 0x7f69e015baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.780+0000 7f69e61d6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69e00cfa90 0x7f69e00c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.780+0000 7f69e61d6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69e0144b30 0x7f69e015d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.780+0000 7f69e61d6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e015c030 con 0x7f69e00cfa90 2024-03-18T19:10:10.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.780+0000 7f69e61d6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e00c7480 con 0x7f69e0144b30 2024-03-18T19:10:10.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.780+0000 7f69e61d6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e00c7760 con 0x7f69e006a750 2024-03-18T19:10:10.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.780+0000 7f69deffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69e00cfa90 0x7f69e00c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.780+0000 7f69deffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69e00cfa90 0x7f69e00c6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51030/0 (socket says 172.21.15.1:51030) 2024-03-18T19:10:10.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.780+0000 7f69deffd700 1 -- 172.21.15.1:0/3959402261 learned_addr learned my addr 172.21.15.1:0/3959402261 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:10.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.780+0000 7f69df7fe700 1 --2- 172.21.15.1:0/3959402261 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69e006a750 0x7f69e015baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.781+0000 7f69dffff700 1 --2- 172.21.15.1:0/3959402261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69e0144b30 0x7f69e015d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.781+0000 7f69deffd700 1 -- 172.21.15.1:0/3959402261 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69e006a750 msgr2=0x7f69e015baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:10.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.781+0000 7f69deffd700 1 --2- 172.21.15.1:0/3959402261 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69e006a750 0x7f69e015baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.781+0000 7f69deffd700 1 -- 172.21.15.1:0/3959402261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69e0144b30 msgr2=0x7f69e015d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:10.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.781+0000 7f69deffd700 1 --2- 172.21.15.1:0/3959402261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69e0144b30 0x7f69e015d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:10.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.781+0000 7f69deffd700 1 -- 172.21.15.1:0/3959402261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d0009550 con 0x7f69e00cfa90 2024-03-18T19:10:10.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.781+0000 7f69df7fe700 1 --2- 172.21.15.1:0/3959402261 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69e006a750 0x7f69e015baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:10:10.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.781+0000 7f69deffd700 1 --2- 172.21.15.1:0/3959402261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69e00cfa90 0x7f69e00c6aa0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f69d0004000 tx=0x7f69d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:10.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.781+0000 7f69d7fff700 1 -- 172.21.15.1:0/3959402261 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d0003cc0 con 0x7f69e00cfa90 2024-03-18T19:10:10.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.781+0000 7f69d7fff700 1 -- 172.21.15.1:0/3959402261 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d001f070 con 0x7f69e00cfa90 2024-03-18T19:10:10.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.781+0000 7f69d7fff700 1 -- 172.21.15.1:0/3959402261 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d000b8a0 con 0x7f69e00cfa90 2024-03-18T19:10:10.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.781+0000 7f69e61d6700 1 -- 172.21.15.1:0/3959402261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e00c7a40 con 0x7f69e00cfa90 2024-03-18T19:10:10.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.781+0000 7f69e61d6700 1 -- 172.21.15.1:0/3959402261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e00c8000 con 0x7f69e00cfa90 2024-03-18T19:10:10.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.782+0000 7f69e61d6700 1 -- 172.21.15.1:0/3959402261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e00676a0 con 0x7f69e00cfa90 2024-03-18T19:10:10.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.782+0000 7f69d7fff700 1 -- 172.21.15.1:0/3959402261 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f69d0021030 con 0x7f69e00cfa90 2024-03-18T19:10:10.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.783+0000 7f69d7fff700 1 --2- 172.21.15.1:0/3959402261 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f69c006adb0 0x7f69c006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:10.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.783+0000 7f69d7fff700 1 -- 172.21.15.1:0/3959402261 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f69d008aba0 con 0x7f69e00cfa90 2024-03-18T19:10:10.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.783+0000 7f69df7fe700 1 --2- 172.21.15.1:0/3959402261 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f69c006adb0 0x7f69c006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:10.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.783+0000 7f69df7fe700 1 --2- 172.21.15.1:0/3959402261 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f69c006adb0 0x7f69c006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f69c8007a50 tx=0x7f69c8008040 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:10.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:10.787+0000 7f69d7fff700 1 -- 172.21.15.1:0/3959402261 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f69d0055ba0 con 0x7f69e00cfa90 2024-03-18T19:10:11.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:11.036+0000 7f69e61d6700 1 -- 172.21.15.1:0/3959402261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f69e00676a0 con 0x7f69e00cfa90 2024-03-18T19:10:11.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:11.037+0000 7f69d7fff700 1 -- 172.21.15.1:0/3959402261 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v73) v1 ==== 53+0+73 (secure 0 0 0) 0x7f69d00591c0 con 0x7f69e00cfa90 2024-03-18T19:10:11.034 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T19:10:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:11.039+0000 7f69e61d6700 1 -- 172.21.15.1:0/3959402261 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f69c006adb0 msgr2=0x7f69c006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:11.039+0000 7f69e61d6700 1 --2- 172.21.15.1:0/3959402261 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f69c006adb0 0x7f69c006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f69c8007a50 tx=0x7f69c8008040 comp rx=0 tx=0).stop 2024-03-18T19:10:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:11.039+0000 7f69e61d6700 1 -- 172.21.15.1:0/3959402261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69e00cfa90 msgr2=0x7f69e00c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:11.039+0000 7f69e61d6700 1 --2- 172.21.15.1:0/3959402261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69e00cfa90 0x7f69e00c6aa0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f69d0004000 tx=0x7f69d0004750 comp rx=0 tx=0).stop 2024-03-18T19:10:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:11.039+0000 7f69e61d6700 1 -- 172.21.15.1:0/3959402261 shutdown_connections 2024-03-18T19:10:11.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:11.039+0000 7f69e61d6700 1 --2- 172.21.15.1:0/3959402261 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f69c006adb0 0x7f69c006d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:11.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:11.039+0000 7f69e61d6700 1 --2- 172.21.15.1:0/3959402261 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69e006a750 0x7f69e015baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:11.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:11.039+0000 7f69e61d6700 1 --2- 172.21.15.1:0/3959402261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69e0144b30 0x7f69e015d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:11.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:11.039+0000 7f69e61d6700 1 --2- 172.21.15.1:0/3959402261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69e00cfa90 0x7f69e00c6aa0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:11.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:11.039+0000 7f69e61d6700 1 -- 172.21.15.1:0/3959402261 >> 172.21.15.1:0/3959402261 conn(0x7f69e00b1d20 msgr2=0x7f69e00b01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:11.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:11.039+0000 7f69e61d6700 1 -- 172.21.15.1:0/3959402261 shutdown_connections 2024-03-18T19:10:11.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:11.039+0000 7f69e61d6700 1 -- 172.21.15.1:0/3959402261 wait complete. 2024-03-18T19:10:11.048 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T19:10:11.048 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T19:10:11.048 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-18T19:10:11.048 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T19:10:11.048 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T19:10:11.048 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T19:10:11.048 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T19:10:11.049 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T19:10:11.049 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:10:11.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:10:11.049 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-18T19:10:11.068 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T19:10:11.068 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T19:10:11.068 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T19:10:11.068 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:10:11.068 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T19:10:11.068 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:10:11.069 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:10:11.069 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-18T19:10:11.069 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-18T19:10:11.069 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:10:11.069 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-18T19:10:11.069 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-18T19:10:11.069 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T19:10:11.069 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:10:11.069 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-18T19:10:11.069 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T19:10:11.069 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 41358sec preferred_lft 41358sec 2024-03-18T19:10:11.069 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link 2024-03-18T19:10:11.069 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:10:11.070 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T19:10:11.070 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:10:11.070 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-18T19:10:11.070 INFO:teuthology.orchestra.run.smithi001.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T19:10:11.070 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff 2024-03-18T19:10:11.070 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T19:10:11.070 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:10:11.070 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::4ca9:f5ff:feb3:79a3/64 scope link 2024-03-18T19:10:11.070 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:10:11.070 INFO:teuthology.orchestra.run.smithi001.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T19:10:11.070 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T19:10:11.070 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::282f:a4ff:fe1a:9908/64 scope link 2024-03-18T19:10:11.071 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:10:11.071 INFO:teuthology.orchestra.run.smithi001.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T19:10:11.071 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2e:5c:60:78:b2:8c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T19:10:11.071 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::2c5c:60ff:fe78:b28c/64 scope link 2024-03-18T19:10:11.071 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T19:10:11.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:10:11.071 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T19:10:11.135 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-18T19:10:11.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:10:11.137 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-18T19:10:11.197 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-18T19:10:11.198 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T19:10:11.198 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T19:10:11.198 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T19:10:11.198 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T19:10:11.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:10:11.198 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T19:10:11.198 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T19:10:11.199 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T19:10:11.199 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T19:10:11.199 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T19:10:11.199 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T19:10:11.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:10:11.304 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T19:10:11.323 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T19:10:11.323 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T19:10:11.323 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:10:11.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:10:11.323 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-18T19:10:11.390 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T19:10:11.390 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T19:10:11.390 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T19:10:11.452 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T19:10:11.452 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T19:10:11.452 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T19:10:11.510 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T19:10:11.579 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T19:10:11.579 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T19:10:11.646 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T19:10:11.647 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T19:10:11.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:10:11.648 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T19:10:11.707 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T19:10:11.707 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T19:10:11.749 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T19:10:11.776 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T19:10:11.776 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T19:10:11.836 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-18T19:10:11.839+0000 7f6825f025c0 -1 init, newargv = 0x5596aea664e0 newargc=15 2024-03-18T19:10:11.836 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[134141]: starting ceph client 2024-03-18T19:10:11.841 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[134141]: starting fuse 2024-03-18T19:10:11.845 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T19:10:11.847 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T19:10:11.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:10:11.847 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T19:10:11.906 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T19:10:11.906 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-03-18T19:10:11.907 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T19:10:11.907 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-18T19:10:11.907 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-18T19:10:11.907 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-18T19:10:11.907 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-18T19:10:11.907 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:10:11.907 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-18T19:10:11.907 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T19:10:11.907 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-18T19:10:11.907 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T19:10:11.907 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:10:11.907 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-18T19:10:11.907 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-18T19:10:11.908 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-18T19:10:11.908 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-18T19:10:11.908 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:10:11.908 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-18T19:10:11.908 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T19:10:11.908 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T19:10:11.908 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T19:10:11.908 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-18T19:10:11.908 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-18T19:10:11.908 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-18T19:10:11.908 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T19:10:11.908 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T19:10:11.908 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-18T19:10:11.908 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-18T19:10:12.031 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.134141.asok 2024-03-18T19:10:12.213 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.134141.asok 2024-03-18T19:10:12.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:10:12.214 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.134141.asok status 2024-03-18T19:10:12.375 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T19:10:12.375 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-18T19:10:12.375 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T19:10:12.375 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T19:10:12.375 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-18T19:10:12.376 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-18T19:10:12.376 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T19:10:12.376 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "134141", 2024-03-18T19:10:12.376 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-18T19:10:12.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:12.376 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-18T19:10:12.376 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-18T19:10:12.376 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 9540, 2024-03-18T19:10:12.376 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-18T19:10:12.376 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-18T19:10:12.376 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-18T19:10:12.376 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 9540 2024-03-18T19:10:12.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:12.376 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T19:10:12.376 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T19:10:12.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T19:10:12.377 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 711223245 2024-03-18T19:10:12.377 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:12.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:12.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T19:10:12.377 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T19:10:12.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T19:10:12.377 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 711223245 2024-03-18T19:10:12.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T19:10:12.377 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.9540 192.168.0.1:0/711223245", 2024-03-18T19:10:12.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/711223245", 2024-03-18T19:10:12.377 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-18T19:10:12.377 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 73, 2024-03-18T19:10:12.378 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 161, 2024-03-18T19:10:12.378 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-18T19:10:12.378 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-18T19:10:12.378 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-18T19:10:12.378 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T19:10:12.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:10:12.385 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-18T19:10:12.385 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T19:10:12.404 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-18T19:10:12.404 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T19:10:12.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:10:12.404 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T19:10:12.472 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-18T19:10:12.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.643+0000 7f3f54322700 1 Processor -- start 2024-03-18T19:10:12.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.643+0000 7f3f54322700 1 -- start start 2024-03-18T19:10:12.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.644+0000 7f3f54322700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3f4c06d560 0x7f3f4c0aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:12.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.644+0000 7f3f54322700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f4c0af510 0x7f3f4c15b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:12.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.644+0000 7f3f54322700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f4c15ef00 con 0x7f3f4c06a750 2024-03-18T19:10:12.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.644+0000 7f3f54322700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f4c15f1e0 con 0x7f3f4c06d560 2024-03-18T19:10:12.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.644+0000 7f3f54322700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c15f4c0 con 0x7f3f4c0af510 2024-03-18T19:10:12.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.644+0000 7f3f520be700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3f4c06a750 0x7f3f4c06ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52648/0 (socket says 172.21.15.1:52648) 2024-03-18T19:10:12.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.644+0000 7f3f520be700 1 -- 172.21.15.1:0/2150495937 learned_addr learned my addr 172.21.15.1:0/2150495937 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:10:12.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.645+0000 7f3f528bf700 1 --2- 172.21.15.1:0/2150495937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f4c0af510 0x7f3f4c15b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:12.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.645+0000 7f3f518bd700 1 --2- 172.21.15.1:0/2150495937 >> v2:172.21.15.165:3300/0 conn(0x7f3f4c06d560 0x7f3f4c0aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:12.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.645+0000 7f3f510bc700 1 -- 172.21.15.1:0/2150495937 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518804602 0 0) 0x7f3f4c15ef00 con 0x7f3f4c06a750 2024-03-18T19:10:12.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.645+0000 7f3f510bc700 1 -- 172.21.15.1:0/2150495937 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f380035b0 con 0x7f3f4c06a750 2024-03-18T19:10:12.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.645+0000 7f3f518bd700 1 -- 172.21.15.1:0/2150495937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f4c0af510 msgr2=0x7f3f4c15b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:12.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.645+0000 7f3f518bd700 1 --2- 172.21.15.1:0/2150495937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f4c0af510 0x7f3f4c15b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:12.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.645+0000 7f3f510bc700 1 -- 172.21.15.1:0/2150495937 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2919053478 0 0) 0x7f3f380035b0 con 0x7f3f4c06a750 2024-03-18T19:10:12.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.645+0000 7f3f518bd700 1 -- 172.21.15.1:0/2150495937 >> v1:172.21.15.1:6789/0 conn(0x7f3f4c06a750 legacy=0x7f3f4c06ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:12.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.645+0000 7f3f518bd700 1 -- 172.21.15.1:0/2150495937 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4c15f7a0 con 0x7f3f4c06d560 2024-03-18T19:10:12.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.645+0000 7f3f528bf700 1 --2- 172.21.15.1:0/2150495937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f4c0af510 0x7f3f4c15b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:10:12.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.645+0000 7f3f518bd700 1 --2- 172.21.15.1:0/2150495937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f4c06d560 0x7f3f4c0aefd0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c009840 tx=0x7f3f3c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7000654e9432311 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:12.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.645+0000 7f3f510bc700 1 -- 172.21.15.1:0/2150495937 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f48001670 con 0x7f3f4c06d560 2024-03-18T19:10:12.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.645+0000 7f3f510bc700 1 -- 172.21.15.1:0/2150495937 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3c004030 con 0x7f3f4c06d560 2024-03-18T19:10:12.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.645+0000 7f3f510bc700 1 -- 172.21.15.1:0/2150495937 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c0043a0 con 0x7f3f4c06d560 2024-03-18T19:10:12.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.646+0000 7f3f54322700 1 -- 172.21.15.1:0/2150495937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f4c06d560 msgr2=0x7f3f4c0aefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:12.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.646+0000 7f3f54322700 1 --2- 172.21.15.1:0/2150495937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f4c06d560 0x7f3f4c0aefd0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c009840 tx=0x7f3f3c009b50 comp rx=0 tx=0).stop 2024-03-18T19:10:12.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.646+0000 7f3f54322700 1 -- 172.21.15.1:0/2150495937 shutdown_connections 2024-03-18T19:10:12.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.646+0000 7f3f54322700 1 --2- 172.21.15.1:0/2150495937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f4c0af510 0x7f3f4c15b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:12.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.646+0000 7f3f54322700 1 --2- 172.21.15.1:0/2150495937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f4c06d560 0x7f3f4c0aefd0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:12.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.646+0000 7f3f54322700 1 -- 172.21.15.1:0/2150495937 >> 172.21.15.1:0/2150495937 conn(0x7f3f4c0cff30 msgr2=0x7f3f4c0d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:12.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.646+0000 7f3f54322700 1 -- 172.21.15.1:0/2150495937 shutdown_connections 2024-03-18T19:10:12.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.646+0000 7f3f54322700 1 -- 172.21.15.1:0/2150495937 wait complete. 2024-03-18T19:10:12.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.647+0000 7f3f54322700 1 Processor -- start 2024-03-18T19:10:12.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.647+0000 7f3f54322700 1 -- start start 2024-03-18T19:10:12.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.648+0000 7f3f54322700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f4c06a750 0x7f3f4c149870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:12.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.648+0000 7f3f54322700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f4c06d560 0x7f3f4c1f0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:12.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.648+0000 7f3f54322700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4c0af510 0x7f3f4c15db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:12.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.648+0000 7f3f54322700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c0b2890 con 0x7f3f4c0af510 2024-03-18T19:10:12.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.648+0000 7f3f54322700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c0af940 con 0x7f3f4c06d560 2024-03-18T19:10:12.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.648+0000 7f3f54322700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c149db0 con 0x7f3f4c06a750 2024-03-18T19:10:12.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.648+0000 7f3f528bf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4c0af510 0x7f3f4c15db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:12.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.648+0000 7f3f528bf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4c0af510 0x7f3f4c15db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51046/0 (socket says 172.21.15.1:51046) 2024-03-18T19:10:12.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.648+0000 7f3f528bf700 1 -- 172.21.15.1:0/3071064281 learned_addr learned my addr 172.21.15.1:0/3071064281 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:12.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.648+0000 7f3f520be700 1 --2- 172.21.15.1:0/3071064281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f4c06a750 0x7f3f4c149870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:12.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.648+0000 7f3f518bd700 1 --2- 172.21.15.1:0/3071064281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f4c06d560 0x7f3f4c1f0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:12.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.649+0000 7f3f528bf700 1 -- 172.21.15.1:0/3071064281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f4c06a750 msgr2=0x7f3f4c149870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:12.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.649+0000 7f3f528bf700 1 --2- 172.21.15.1:0/3071064281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f4c06a750 0x7f3f4c149870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:12.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.649+0000 7f3f528bf700 1 -- 172.21.15.1:0/3071064281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f4c06d560 msgr2=0x7f3f4c1f0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:12.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.649+0000 7f3f528bf700 1 --2- 172.21.15.1:0/3071064281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f4c06d560 0x7f3f4c1f0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:12.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.649+0000 7f3f528bf700 1 -- 172.21.15.1:0/3071064281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f44009a70 con 0x7f3f4c0af510 2024-03-18T19:10:12.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.649+0000 7f3f518bd700 1 --2- 172.21.15.1:0/3071064281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f4c06d560 0x7f3f4c1f0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:10:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.649+0000 7f3f528bf700 1 --2- 172.21.15.1:0/3071064281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4c0af510 0x7f3f4c15db70 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f3f4400ef90 tx=0x7f3f4400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.649+0000 7f3f42ffd700 1 -- 172.21.15.1:0/3071064281 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4400cd40 con 0x7f3f4c0af510 2024-03-18T19:10:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.649+0000 7f3f42ffd700 1 -- 172.21.15.1:0/3071064281 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4400f040 con 0x7f3f4c0af510 2024-03-18T19:10:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.649+0000 7f3f54322700 1 -- 172.21.15.1:0/3071064281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3c0094a0 con 0x7f3f4c0af510 2024-03-18T19:10:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.649+0000 7f3f42ffd700 1 -- 172.21.15.1:0/3071064281 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f440146c0 con 0x7f3f4c0af510 2024-03-18T19:10:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.649+0000 7f3f54322700 1 -- 172.21.15.1:0/3071064281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4c146fb0 con 0x7f3f4c0af510 2024-03-18T19:10:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.650+0000 7f3f54322700 1 -- 172.21.15.1:0/3071064281 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f4c146e10 con 0x7f3f4c0af510 2024-03-18T19:10:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.651+0000 7f3f42ffd700 1 -- 172.21.15.1:0/3071064281 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3f44014820 con 0x7f3f4c0af510 2024-03-18T19:10:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.651+0000 7f3f42ffd700 1 --2- 172.21.15.1:0/3071064281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3f3006adb0 0x7f3f3006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:12.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.651+0000 7f3f42ffd700 1 -- 172.21.15.1:0/3071064281 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3f4408a280 con 0x7f3f4c0af510 2024-03-18T19:10:12.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.651+0000 7f3f520be700 1 --2- 172.21.15.1:0/3071064281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3f3006adb0 0x7f3f3006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:12.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.651+0000 7f3f520be700 1 --2- 172.21.15.1:0/3071064281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3f3006adb0 0x7f3f3006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3f48001eb0 tx=0x7f3f48001e40 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:12.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.656+0000 7f3f42ffd700 1 -- 172.21.15.1:0/3071064281 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3f4c146e10 con 0x7f3f4c0af510 2024-03-18T19:10:12.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.885+0000 7f3f54322700 1 -- 172.21.15.1:0/3071064281 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f4c15b480 con 0x7f3f3006adb0 2024-03-18T19:10:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.886+0000 7f3f42ffd700 1 -- 172.21.15.1:0/3071064281 <== mgr.9153 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f3f4800f690 con 0x7f3f3006adb0 2024-03-18T19:10:12.883 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T19:10:12.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T19:10:12.883 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephfs" 2024-03-18T19:10:12.883 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T19:10:12.884 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T19:10:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.889+0000 7f3f54322700 1 -- 172.21.15.1:0/3071064281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3f3006adb0 msgr2=0x7f3f3006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.889+0000 7f3f54322700 1 --2- 172.21.15.1:0/3071064281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3f3006adb0 0x7f3f3006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3f48001eb0 tx=0x7f3f48001e40 comp rx=0 tx=0).stop 2024-03-18T19:10:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.889+0000 7f3f54322700 1 -- 172.21.15.1:0/3071064281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4c0af510 msgr2=0x7f3f4c15db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.889+0000 7f3f54322700 1 --2- 172.21.15.1:0/3071064281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4c0af510 0x7f3f4c15db70 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f3f4400ef90 tx=0x7f3f4400c3a0 comp rx=0 tx=0).stop 2024-03-18T19:10:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.889+0000 7f3f54322700 1 -- 172.21.15.1:0/3071064281 shutdown_connections 2024-03-18T19:10:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.889+0000 7f3f54322700 1 --2- 172.21.15.1:0/3071064281 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3f3006adb0 0x7f3f3006d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.889+0000 7f3f54322700 1 --2- 172.21.15.1:0/3071064281 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f4c06a750 0x7f3f4c149870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.889+0000 7f3f54322700 1 --2- 172.21.15.1:0/3071064281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f4c0af510 0x7f3f4c15db70 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.889+0000 7f3f54322700 1 --2- 172.21.15.1:0/3071064281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f4c06d560 0x7f3f4c1f0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.889+0000 7f3f54322700 1 -- 172.21.15.1:0/3071064281 >> 172.21.15.1:0/3071064281 conn(0x7f3f4c0cff30 msgr2=0x7f3f4c06f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:12.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.889+0000 7f3f54322700 1 -- 172.21.15.1:0/3071064281 shutdown_connections 2024-03-18T19:10:12.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:12.889+0000 7f3f54322700 1 -- 172.21.15.1:0/3071064281 wait complete. 2024-03-18T19:10:12.897 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-18T19:10:13.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.027+0000 7f8abe1ce700 1 Processor -- start 2024-03-18T19:10:13.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.027+0000 7f8abe1ce700 1 -- start start 2024-03-18T19:10:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.028+0000 7f8abe1ce700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ab815d600 0x7f8ab809a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.028+0000 7f8abe1ce700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8ab80d0240 0x7f8ab80cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.028+0000 7f8abe1ce700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab80d06d0 con 0x7f8ab815d9e0 2024-03-18T19:10:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.028+0000 7f8abe1ce700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ab80cfdb0 con 0x7f8ab80d0240 2024-03-18T19:10:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.028+0000 7f8abe1ce700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab80b38d0 con 0x7f8ab815d600 2024-03-18T19:10:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.028+0000 7f8ab77fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ab815d600 0x7f8ab809a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.028+0000 7f8ab7fff700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8ab80d0240 0x7f8ab80cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:13.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.028+0000 7f8ab6ffd700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f8ab815d9e0 0x7f8ab80cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:44992/0 (socket says 172.21.15.1:44992) 2024-03-18T19:10:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.029+0000 7f8ab6ffd700 1 -- 172.21.15.1:0/1055147740 learned_addr learned my addr 172.21.15.1:0/1055147740 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:10:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.029+0000 7f8ab7fff700 1 -- 172.21.15.1:0/1055147740 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ab815d600 msgr2=0x7f8ab809a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.029+0000 7f8ab7fff700 1 --2- 172.21.15.1:0/1055147740 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ab815d600 0x7f8ab809a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.029+0000 7f8ab7fff700 1 -- 172.21.15.1:0/1055147740 >> v1:172.21.15.165:6789/0 conn(0x7f8ab815d9e0 legacy=0x7f8ab80cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.029+0000 7f8ab7fff700 1 -- 172.21.15.1:0/1055147740 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab809aa30 con 0x7f8ab80d0240 2024-03-18T19:10:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.029+0000 7f8ab77fe700 1 --2- 172.21.15.1:0/1055147740 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ab815d600 0x7f8ab809a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:10:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.029+0000 7f8ab7fff700 1 --2- 172.21.15.1:0/1055147740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ab80d0240 0x7f8ab80cbe00 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0009d90 tx=0x7f8ab000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2139ed053f08b7d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.029+0000 7f8ab67fc700 1 -- 172.21.15.1:0/1055147740 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab000e070 con 0x7f8ab80d0240 2024-03-18T19:10:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.029+0000 7f8ab67fc700 1 -- 172.21.15.1:0/1055147740 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ab000bb20 con 0x7f8ab80d0240 2024-03-18T19:10:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.029+0000 7f8ab67fc700 1 -- 172.21.15.1:0/1055147740 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab0012430 con 0x7f8ab80d0240 2024-03-18T19:10:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.030+0000 7f8abe1ce700 1 -- 172.21.15.1:0/1055147740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ab80d0240 msgr2=0x7f8ab80cbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.030+0000 7f8abe1ce700 1 --2- 172.21.15.1:0/1055147740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ab80d0240 0x7f8ab80cbe00 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0009d90 tx=0x7f8ab000b330 comp rx=0 tx=0).stop 2024-03-18T19:10:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.030+0000 7f8abe1ce700 1 -- 172.21.15.1:0/1055147740 shutdown_connections 2024-03-18T19:10:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.030+0000 7f8abe1ce700 1 --2- 172.21.15.1:0/1055147740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ab80d0240 0x7f8ab80cbe00 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.030+0000 7f8abe1ce700 1 --2- 172.21.15.1:0/1055147740 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ab815d600 0x7f8ab809a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.030+0000 7f8abe1ce700 1 -- 172.21.15.1:0/1055147740 >> 172.21.15.1:0/1055147740 conn(0x7f8ab80b2b90 msgr2=0x7f8ab80b2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.030+0000 7f8abe1ce700 1 -- 172.21.15.1:0/1055147740 shutdown_connections 2024-03-18T19:10:13.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.030+0000 7f8abe1ce700 1 -- 172.21.15.1:0/1055147740 wait complete. 2024-03-18T19:10:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.031+0000 7f8abe1ce700 1 Processor -- start 2024-03-18T19:10:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.031+0000 7f8abe1ce700 1 -- start start 2024-03-18T19:10:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.032+0000 7f8abe1ce700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ab80d0240 0x7f8ab815ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.032+0000 7f8abe1ce700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ab815d600 0x7f8ab80c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.032+0000 7f8abe1ce700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ab815d9e0 0x7f8ab815a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.032+0000 7f8abe1ce700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab815a5b0 con 0x7f8ab815d600 2024-03-18T19:10:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.032+0000 7f8abe1ce700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab815f1e0 con 0x7f8ab80d0240 2024-03-18T19:10:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.032+0000 7f8abe1ce700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab8160c30 con 0x7f8ab815d9e0 2024-03-18T19:10:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.032+0000 7f8ab6ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ab815d600 0x7f8ab80c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.032+0000 7f8ab6ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ab815d600 0x7f8ab80c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51056/0 (socket says 172.21.15.1:51056) 2024-03-18T19:10:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.032+0000 7f8ab6ffd700 1 -- 172.21.15.1:0/3820899944 learned_addr learned my addr 172.21.15.1:0/3820899944 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.032+0000 7f8ab7fff700 1 --2- 172.21.15.1:0/3820899944 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ab815d9e0 0x7f8ab815a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.032+0000 7f8ab77fe700 1 --2- 172.21.15.1:0/3820899944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ab80d0240 0x7f8ab815ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.033+0000 7f8ab6ffd700 1 -- 172.21.15.1:0/3820899944 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ab815d9e0 msgr2=0x7f8ab815a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.033+0000 7f8ab6ffd700 1 --2- 172.21.15.1:0/3820899944 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ab815d9e0 0x7f8ab815a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.033+0000 7f8ab6ffd700 1 -- 172.21.15.1:0/3820899944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ab80d0240 msgr2=0x7f8ab815ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.033+0000 7f8ab6ffd700 1 --2- 172.21.15.1:0/3820899944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ab80d0240 0x7f8ab815ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.033+0000 7f8ab6ffd700 1 -- 172.21.15.1:0/3820899944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa80094a0 con 0x7f8ab815d600 2024-03-18T19:10:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.033+0000 7f8ab7fff700 1 --2- 172.21.15.1:0/3820899944 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ab815d9e0 0x7f8ab815a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:10:13.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.033+0000 7f8ab6ffd700 1 --2- 172.21.15.1:0/3820899944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ab815d600 0x7f8ab80c8170 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f8aa000c430 tx=0x7f8aa000c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.033+0000 7f8aaffff700 1 -- 172.21.15.1:0/3820899944 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0005b70 con 0x7f8ab815d600 2024-03-18T19:10:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.033+0000 7f8aaffff700 1 -- 172.21.15.1:0/3820899944 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa0005cd0 con 0x7f8ab815d600 2024-03-18T19:10:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.033+0000 7f8aaffff700 1 -- 172.21.15.1:0/3820899944 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa00072f0 con 0x7f8ab815d600 2024-03-18T19:10:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.033+0000 7f8abe1ce700 1 -- 172.21.15.1:0/3820899944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab0009a70 con 0x7f8ab815d600 2024-03-18T19:10:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.033+0000 7f8abe1ce700 1 -- 172.21.15.1:0/3820899944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab8160950 con 0x7f8ab815d600 2024-03-18T19:10:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.034+0000 7f8abe1ce700 1 -- 172.21.15.1:0/3820899944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab815a6f0 con 0x7f8ab815d600 2024-03-18T19:10:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.034+0000 7f8aaffff700 1 -- 172.21.15.1:0/3820899944 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 82) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8aa0007450 con 0x7f8ab815d600 2024-03-18T19:10:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.035+0000 7f8aaffff700 1 --2- 172.21.15.1:0/3820899944 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8a9806adb0 0x7f8a9806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.035+0000 7f8aaffff700 1 -- 172.21.15.1:0/3820899944 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f8aa0088850 con 0x7f8ab815d600 2024-03-18T19:10:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.035+0000 7f8ab77fe700 1 --2- 172.21.15.1:0/3820899944 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8a9806adb0 0x7f8a9806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:13.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.035+0000 7f8ab77fe700 1 --2- 172.21.15.1:0/3820899944 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8a9806adb0 0x7f8a9806d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8aa8009d70 tx=0x7f8aa8009210 comp rx=0 tx=0).ready entity=mgr.9153 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:13.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.040+0000 7f8aaffff700 1 -- 172.21.15.1:0/3820899944 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8ab815a6f0 con 0x7f8ab815d600 2024-03-18T19:10:13.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:13.303+0000 7f8abe1ce700 1 -- 172.21.15.1:0/3820899944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f8ab815a6f0 con 0x7f8ab815d600 2024-03-18T19:10:14.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.133+0000 7f8aaffff700 1 -- 172.21.15.1:0/3820899944 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 83) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f8aa0053c20 con 0x7f8ab815d600 2024-03-18T19:10:14.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.135+0000 7f8aaffff700 1 -- 172.21.15.1:0/3820899944 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v83) v1 ==== 92+0+0 (secure 0 0 0) 0x7f8aa0072b60 con 0x7f8ab815d600 2024-03-18T19:10:14.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.137+0000 7f8abe1ce700 1 -- 172.21.15.1:0/3820899944 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8a9806adb0 msgr2=0x7f8a9806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:14.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.137+0000 7f8abe1ce700 1 --2- 172.21.15.1:0/3820899944 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8a9806adb0 0x7f8a9806d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8aa8009d70 tx=0x7f8aa8009210 comp rx=0 tx=0).stop 2024-03-18T19:10:14.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.137+0000 7f8abe1ce700 1 -- 172.21.15.1:0/3820899944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ab815d600 msgr2=0x7f8ab80c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:14.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.137+0000 7f8abe1ce700 1 --2- 172.21.15.1:0/3820899944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ab815d600 0x7f8ab80c8170 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f8aa000c430 tx=0x7f8aa000c740 comp rx=0 tx=0).stop 2024-03-18T19:10:14.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.138+0000 7f8abe1ce700 1 -- 172.21.15.1:0/3820899944 shutdown_connections 2024-03-18T19:10:14.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.138+0000 7f8abe1ce700 1 --2- 172.21.15.1:0/3820899944 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8a9806adb0 0x7f8a9806d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.138+0000 7f8abe1ce700 1 --2- 172.21.15.1:0/3820899944 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ab815d9e0 0x7f8ab815a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.138+0000 7f8abe1ce700 1 --2- 172.21.15.1:0/3820899944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ab80d0240 0x7f8ab815ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.138+0000 7f8abe1ce700 1 --2- 172.21.15.1:0/3820899944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ab815d600 0x7f8ab80c8170 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.138+0000 7f8abe1ce700 1 -- 172.21.15.1:0/3820899944 >> 172.21.15.1:0/3820899944 conn(0x7f8ab80b2b90 msgr2=0x7f8ab806e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:14.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.138+0000 7f8abe1ce700 1 -- 172.21.15.1:0/3820899944 shutdown_connections 2024-03-18T19:10:14.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.138+0000 7f8abe1ce700 1 -- 172.21.15.1:0/3820899944 wait complete. 2024-03-18T19:10:14.150 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-18T19:10:14.272 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:14.276+0000 7ff496682240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:10:14.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.290+0000 7ff43c8ad700 1 Processor -- start 2024-03-18T19:10:14.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.290+0000 7ff43c8ad700 1 -- start start 2024-03-18T19:10:14.288 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:14.291+0000 7f43b367b240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T19:10:14.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.291+0000 7ff43c8ad700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff438160a70 0x7ff4380d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:14.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.291+0000 7ff43c8ad700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff438160e50 0x7ff4380cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:14.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.291+0000 7ff43c8ad700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff438162170 con 0x7ff4381612d0 2024-03-18T19:10:14.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.291+0000 7ff43c8ad700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4380cef30 con 0x7ff438160a70 2024-03-18T19:10:14.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.291+0000 7ff43c8ad700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4380cf210 con 0x7ff438160e50 2024-03-18T19:10:14.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.292+0000 7ff43659c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff438160a70 0x7ff4380d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:14.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.292+0000 7ff436d9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff4381612d0 0x7ff4381616f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52678/0 (socket says 172.21.15.1:52678) 2024-03-18T19:10:14.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.292+0000 7ff436d9d700 1 -- 172.21.15.1:0/3078435496 learned_addr learned my addr 172.21.15.1:0/3078435496 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:10:14.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.292+0000 7ff43659c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff438160a70 0x7ff4380d3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51064/0 (socket says 172.21.15.1:51064) 2024-03-18T19:10:14.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.292+0000 7ff435d9b700 1 --2- 172.21.15.1:0/3078435496 >> v2:172.21.15.165:3300/0 conn(0x7ff438160e50 0x7ff4380cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:14.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.292+0000 7ff43659c700 1 -- 172.21.15.1:0/3078435496 >> v2:172.21.15.165:3300/0 conn(0x7ff438160e50 msgr2=0x7ff4380cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:14.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.292+0000 7ff43659c700 1 --2- 172.21.15.1:0/3078435496 >> v2:172.21.15.165:3300/0 conn(0x7ff438160e50 0x7ff4380cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.292+0000 7ff43659c700 1 -- 172.21.15.1:0/3078435496 >> v1:172.21.15.1:6789/0 conn(0x7ff4381612d0 legacy=0x7ff4381616f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:14.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.292+0000 7ff43559a700 1 -- 172.21.15.1:0/3078435496 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753308575 0 0) 0x7ff430001490 con 0x7ff4381612d0 2024-03-18T19:10:14.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.292+0000 7ff43659c700 1 -- 172.21.15.1:0/3078435496 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4380cf4f0 con 0x7ff438160a70 2024-03-18T19:10:14.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.292+0000 7ff43659c700 1 --2- 172.21.15.1:0/3078435496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff438160a70 0x7ff4380d3b70 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7ff4380c9d70 tx=0x7ff428009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=272e492802d1490d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:14.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.292+0000 7ff43559a700 1 -- 172.21.15.1:0/3078435496 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff428011830 con 0x7ff438160a70 2024-03-18T19:10:14.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.292+0000 7ff43559a700 1 -- 172.21.15.1:0/3078435496 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff428011990 con 0x7ff438160a70 2024-03-18T19:10:14.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.292+0000 7ff43559a700 1 -- 172.21.15.1:0/3078435496 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff428011d20 con 0x7ff438160a70 2024-03-18T19:10:14.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.293+0000 7ff43c8ad700 1 -- 172.21.15.1:0/3078435496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff438160a70 msgr2=0x7ff4380d3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:14.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.293+0000 7ff43c8ad700 1 --2- 172.21.15.1:0/3078435496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff438160a70 0x7ff4380d3b70 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7ff4380c9d70 tx=0x7ff428009b20 comp rx=0 tx=0).stop 2024-03-18T19:10:14.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.293+0000 7ff43c8ad700 1 -- 172.21.15.1:0/3078435496 shutdown_connections 2024-03-18T19:10:14.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.293+0000 7ff43c8ad700 1 --2- 172.21.15.1:0/3078435496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff438160a70 0x7ff4380d3b70 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.293+0000 7ff43c8ad700 1 --2- 172.21.15.1:0/3078435496 >> v2:172.21.15.165:3300/0 conn(0x7ff438160e50 0x7ff4380cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.293+0000 7ff43c8ad700 1 -- 172.21.15.1:0/3078435496 >> 172.21.15.1:0/3078435496 conn(0x7ff43809c560 msgr2=0x7ff4380d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:14.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.293+0000 7ff43c8ad700 1 -- 172.21.15.1:0/3078435496 shutdown_connections 2024-03-18T19:10:14.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.293+0000 7ff43c8ad700 1 -- 172.21.15.1:0/3078435496 wait complete. 2024-03-18T19:10:14.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.294+0000 7ff43c8ad700 1 Processor -- start 2024-03-18T19:10:14.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.295+0000 7ff43c8ad700 1 -- start start 2024-03-18T19:10:14.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.295+0000 7ff43c8ad700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff438160a70 0x7ff4380b8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:14.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.295+0000 7ff43c8ad700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff438160e50 0x7ff4380b1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:14.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.295+0000 7ff43c8ad700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4381612d0 0x7ff4380b5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:14.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.295+0000 7ff43c8ad700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4380b8900 con 0x7ff438160e50 2024-03-18T19:10:14.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.295+0000 7ff43c8ad700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4380b0900 con 0x7ff438160a70 2024-03-18T19:10:14.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.295+0000 7ff43c8ad700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4380b1680 con 0x7ff4381612d0 2024-03-18T19:10:14.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.295+0000 7ff435d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff438160e50 0x7ff4380b1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:14.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.295+0000 7ff435d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff438160e50 0x7ff4380b1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51072/0 (socket says 172.21.15.1:51072) 2024-03-18T19:10:14.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.295+0000 7ff435d9b700 1 -- 172.21.15.1:0/3788101778 learned_addr learned my addr 172.21.15.1:0/3788101778 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:14.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.295+0000 7ff43659c700 1 --2- 172.21.15.1:0/3788101778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff438160a70 0x7ff4380b8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:14.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.295+0000 7ff436d9d700 1 --2- 172.21.15.1:0/3788101778 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4381612d0 0x7ff4380b5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:14.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.296+0000 7ff435d9b700 1 -- 172.21.15.1:0/3788101778 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4381612d0 msgr2=0x7ff4380b5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:14.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.296+0000 7ff435d9b700 1 --2- 172.21.15.1:0/3788101778 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4381612d0 0x7ff4380b5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.296+0000 7ff435d9b700 1 -- 172.21.15.1:0/3788101778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff438160a70 msgr2=0x7ff4380b8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:14.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.296+0000 7ff435d9b700 1 --2- 172.21.15.1:0/3788101778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff438160a70 0x7ff4380b8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.296+0000 7ff435d9b700 1 -- 172.21.15.1:0/3788101778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4280094a0 con 0x7ff438160e50 2024-03-18T19:10:14.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.296+0000 7ff43659c700 1 --2- 172.21.15.1:0/3788101778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff438160a70 0x7ff4380b8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:10:14.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.296+0000 7ff435d9b700 1 --2- 172.21.15.1:0/3788101778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff438160e50 0x7ff4380b1c00 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7ff42c00e9b0 tx=0x7ff42c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:14.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.296+0000 7ff426ffd700 1 -- 172.21.15.1:0/3788101778 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c00ee50 con 0x7ff438160e50 2024-03-18T19:10:14.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.296+0000 7ff426ffd700 1 -- 172.21.15.1:0/3788101778 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42c00c7f0 con 0x7ff438160e50 2024-03-18T19:10:14.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.296+0000 7ff43c8ad700 1 -- 172.21.15.1:0/3788101778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4380b0ba0 con 0x7ff438160e50 2024-03-18T19:10:14.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.296+0000 7ff426ffd700 1 -- 172.21.15.1:0/3788101778 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c00cb60 con 0x7ff438160e50 2024-03-18T19:10:14.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.296+0000 7ff43c8ad700 1 -- 172.21.15.1:0/3788101778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4380b1160 con 0x7ff438160e50 2024-03-18T19:10:14.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.297+0000 7ff43c8ad700 1 -- 172.21.15.1:0/3788101778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4380b8ad0 con 0x7ff438160e50 2024-03-18T19:10:14.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.298+0000 7ff426ffd700 1 -- 172.21.15.1:0/3788101778 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 83) v1 ==== 89256+0+0 (secure 0 0 0) 0x7ff42c00ccc0 con 0x7ff438160e50 2024-03-18T19:10:14.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.298+0000 7ff426ffd700 1 --2- 172.21.15.1:0/3788101778 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff41806ae00 0x7ff41806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:14.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.298+0000 7ff43659c700 1 -- 172.21.15.1:0/3788101778 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff41806ae00 msgr2=0x7ff41806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:10:14.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.298+0000 7ff426ffd700 1 -- 172.21.15.1:0/3788101778 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff42c089530 con 0x7ff438160e50 2024-03-18T19:10:14.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.298+0000 7ff43659c700 1 --2- 172.21.15.1:0/3788101778 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff41806ae00 0x7ff41806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:10:14.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.303+0000 7ff426ffd700 1 -- 172.21.15.1:0/3788101778 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff4380b8ad0 con 0x7ff438160e50 2024-03-18T19:10:14.419 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:14.424+0000 7ff496682240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:10:14.457 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:14.460+0000 7f43b367b240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T19:10:14.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.499+0000 7ff43659c700 1 -- 172.21.15.1:0/3788101778 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff41806ae00 msgr2=0x7ff41806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:10:14.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.499+0000 7ff43659c700 1 --2- 172.21.15.1:0/3788101778 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff41806ae00 0x7ff41806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:10:14.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.553+0000 7ff43c8ad700 1 -- 172.21.15.1:0/3788101778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7ff4380b12f0 con 0x7ff438160e50 2024-03-18T19:10:14.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.557+0000 7ff426ffd700 1 -- 172.21.15.1:0/3788101778 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v53)=0 v53) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff42c054530 con 0x7ff438160e50 2024-03-18T19:10:14.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.559+0000 7ff43c8ad700 1 -- 172.21.15.1:0/3788101778 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff41806ae00 msgr2=0x7ff41806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:10:14.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.559+0000 7ff43c8ad700 1 --2- 172.21.15.1:0/3788101778 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff41806ae00 0x7ff41806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.559+0000 7ff43c8ad700 1 -- 172.21.15.1:0/3788101778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff438160e50 msgr2=0x7ff4380b1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:14.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.559+0000 7ff43c8ad700 1 --2- 172.21.15.1:0/3788101778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff438160e50 0x7ff4380b1c00 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7ff42c00e9b0 tx=0x7ff42c00ecc0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.559+0000 7ff43c8ad700 1 -- 172.21.15.1:0/3788101778 shutdown_connections 2024-03-18T19:10:14.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.559+0000 7ff43c8ad700 1 --2- 172.21.15.1:0/3788101778 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff41806ae00 0x7ff41806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.559+0000 7ff43c8ad700 1 --2- 172.21.15.1:0/3788101778 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4381612d0 0x7ff4380b5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.559+0000 7ff43c8ad700 1 --2- 172.21.15.1:0/3788101778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff438160a70 0x7ff4380b8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.559+0000 7ff43c8ad700 1 --2- 172.21.15.1:0/3788101778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff438160e50 0x7ff4380b1c00 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.559+0000 7ff43c8ad700 1 -- 172.21.15.1:0/3788101778 >> 172.21.15.1:0/3788101778 conn(0x7ff43809c560 msgr2=0x7ff4380ce100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:14.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.560+0000 7ff43c8ad700 1 -- 172.21.15.1:0/3788101778 shutdown_connections 2024-03-18T19:10:14.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.560+0000 7ff43c8ad700 1 -- 172.21.15.1:0/3788101778 wait complete. 2024-03-18T19:10:14.569 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-18T19:10:14.685 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:14.690+0000 7ff496682240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:10:14.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.702+0000 7efdd49bc700 1 Processor -- start 2024-03-18T19:10:14.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.702+0000 7efdd49bc700 1 -- start start 2024-03-18T19:10:14.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.703+0000 7efdd49bc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdd006efb0 0x7efdd0071da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:14.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.703+0000 7efdd49bc700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7efdd00722e0 0x7efdd00cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:14.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.703+0000 7efdd49bc700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd006da90 con 0x7efdd015d580 2024-03-18T19:10:14.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.703+0000 7efdd49bc700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdd0072770 con 0x7efdd00722e0 2024-03-18T19:10:14.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.703+0000 7efdd49bc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdd015d960 con 0x7efdd006efb0 2024-03-18T19:10:14.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.703+0000 7efdce59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efdd015d580 0x7efdd006d380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56934/0 (socket says 172.21.15.1:56934) 2024-03-18T19:10:14.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.703+0000 7efdce59c700 1 -- 172.21.15.1:0/544771287 learned_addr learned my addr 172.21.15.1:0/544771287 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:10:14.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.703+0000 7efdcdd9b700 1 --2- 172.21.15.1:0/544771287 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdd006efb0 0x7efdd0071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:14.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.703+0000 7efdced9d700 1 --2- 172.21.15.1:0/544771287 >> v2:172.21.15.165:3300/0 conn(0x7efdd00722e0 0x7efdd00cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:14.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.703+0000 7efdcd59a700 1 -- 172.21.15.1:0/544771287 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132794834 0 0) 0x7efdd006da90 con 0x7efdd015d580 2024-03-18T19:10:14.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.703+0000 7efdcd59a700 1 -- 172.21.15.1:0/544771287 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdb80035b0 con 0x7efdd015d580 2024-03-18T19:10:14.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdcd59a700 1 -- 172.21.15.1:0/544771287 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3233057038 0 0) 0x7efdb80035b0 con 0x7efdd015d580 2024-03-18T19:10:14.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdcd59a700 1 -- 172.21.15.1:0/544771287 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdd006efb0 msgr2=0x7efdd0071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:14.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdcd59a700 1 --2- 172.21.15.1:0/544771287 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdd006efb0 0x7efdd0071da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdcd59a700 1 -- 172.21.15.1:0/544771287 >> v2:172.21.15.165:3300/0 conn(0x7efdd00722e0 msgr2=0x7efdd00cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:14.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdcd59a700 1 --2- 172.21.15.1:0/544771287 >> v2:172.21.15.165:3300/0 conn(0x7efdd00722e0 0x7efdd00cf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdcd59a700 1 -- 172.21.15.1:0/544771287 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd006ab30 con 0x7efdd015d580 2024-03-18T19:10:14.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdcd59a700 1 -- 172.21.15.1:0/544771287 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7efdc4001670 con 0x7efdd015d580 2024-03-18T19:10:14.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdcdd9b700 1 --2- 172.21.15.1:0/544771287 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdd006efb0 0x7efdd0071da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:10:14.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdced9d700 1 --2- 172.21.15.1:0/544771287 >> v2:172.21.15.165:3300/0 conn(0x7efdd00722e0 0x7efdd00cf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:10:14.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdcd59a700 1 -- 172.21.15.1:0/544771287 >> v1:172.21.15.1:6789/0 conn(0x7efdd015d580 legacy=0x7efdd006d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:14.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdcd59a700 1 --2- 172.21.15.1:0/544771287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdb8003a30 0x7efdb8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:14.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdcd59a700 1 --2- 172.21.15.1:0/544771287 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdb8004850 0x7efdb8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:14.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdcd59a700 1 --2- 172.21.15.1:0/544771287 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdd006efb0 0x7efdb8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:14.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdcd59a700 1 -- 172.21.15.1:0/544771287 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd015d960 con 0x7efdb8003a30 2024-03-18T19:10:14.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdcd59a700 1 -- 172.21.15.1:0/544771287 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb80046c0 con 0x7efdb8004850 2024-03-18T19:10:14.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdcd59a700 1 -- 172.21.15.1:0/544771287 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb800ba90 con 0x7efdd006efb0 2024-03-18T19:10:14.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdce59c700 1 --2- 172.21.15.1:0/544771287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdb8003a30 0x7efdb8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:14.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdcdd9b700 1 --2- 172.21.15.1:0/544771287 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdb8004850 0x7efdb8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:14.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdced9d700 1 --2- 172.21.15.1:0/544771287 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdd006efb0 0x7efdb8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:14.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdce59c700 1 -- 172.21.15.1:0/544771287 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdd006efb0 msgr2=0x7efdb8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:14.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdce59c700 1 --2- 172.21.15.1:0/544771287 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdd006efb0 0x7efdb8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdce59c700 1 -- 172.21.15.1:0/544771287 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdb8004850 msgr2=0x7efdb8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:14.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdce59c700 1 --2- 172.21.15.1:0/544771287 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdb8004850 0x7efdb8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdce59c700 1 -- 172.21.15.1:0/544771287 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc00094a0 con 0x7efdb8003a30 2024-03-18T19:10:14.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdced9d700 1 --2- 172.21.15.1:0/544771287 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdd006efb0 0x7efdb8005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:10:14.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdce59c700 1 --2- 172.21.15.1:0/544771287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdb8003a30 0x7efdb8003e50 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7efdc400c240 tx=0x7efdb8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c64d9aaa08970f0a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:14.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.704+0000 7efdcd59a700 1 -- 172.21.15.1:0/544771287 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc4007c10 con 0x7efdb8003a30 2024-03-18T19:10:14.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.705+0000 7efdcd59a700 1 -- 172.21.15.1:0/544771287 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc400ce00 con 0x7efdb8003a30 2024-03-18T19:10:14.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.705+0000 7efdcd59a700 1 -- 172.21.15.1:0/544771287 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc4005ba0 con 0x7efdb8003a30 2024-03-18T19:10:14.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.705+0000 7efdd49bc700 1 -- 172.21.15.1:0/544771287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdb8003a30 msgr2=0x7efdb8003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:14.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.705+0000 7efdd49bc700 1 --2- 172.21.15.1:0/544771287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdb8003a30 0x7efdb8003e50 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7efdc400c240 tx=0x7efdb8005880 comp rx=0 tx=0).stop 2024-03-18T19:10:14.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.705+0000 7efdced9d700 1 -- 172.21.15.1:0/544771287 reap_dead start 2024-03-18T19:10:14.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.705+0000 7efdd49bc700 1 -- 172.21.15.1:0/544771287 shutdown_connections 2024-03-18T19:10:14.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.705+0000 7efdd49bc700 1 -- 172.21.15.1:0/544771287 >> 172.21.15.1:0/544771287 conn(0x7efdd00cca00 msgr2=0x7efdd00cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:14.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.706+0000 7efdd49bc700 1 -- 172.21.15.1:0/544771287 shutdown_connections 2024-03-18T19:10:14.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.706+0000 7efdd49bc700 1 -- 172.21.15.1:0/544771287 wait complete. 2024-03-18T19:10:14.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.707+0000 7efdd49bc700 1 Processor -- start 2024-03-18T19:10:14.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.707+0000 7efdd49bc700 1 -- start start 2024-03-18T19:10:14.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdd49bc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdb8003a30 0x7efdd00caa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:14.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdd49bc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdd0157530 0x7efdd015aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:14.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdd49bc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdd0160200 0x7efdd0160680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdd49bc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd015f160 con 0x7efdd0160200 2024-03-18T19:10:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdd49bc700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd015f440 con 0x7efdd0157530 2024-03-18T19:10:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdd49bc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdd015f720 con 0x7efdb8003a30 2024-03-18T19:10:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdced9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdd0160200 0x7efdd0160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdced9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdd0160200 0x7efdd0160680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51958/0 (socket says 172.21.15.1:51958) 2024-03-18T19:10:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdced9d700 1 -- 172.21.15.1:0/2320640830 learned_addr learned my addr 172.21.15.1:0/2320640830 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdce59c700 1 --2- 172.21.15.1:0/2320640830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdb8003a30 0x7efdd00caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdcdd9b700 1 --2- 172.21.15.1:0/2320640830 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdd0157530 0x7efdd015aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdced9d700 1 -- 172.21.15.1:0/2320640830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdb8003a30 msgr2=0x7efdd00caa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdced9d700 1 --2- 172.21.15.1:0/2320640830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdb8003a30 0x7efdd00caa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdced9d700 1 -- 172.21.15.1:0/2320640830 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdd0157530 msgr2=0x7efdd015aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdced9d700 1 --2- 172.21.15.1:0/2320640830 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdd0157530 0x7efdd015aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdced9d700 1 -- 172.21.15.1:0/2320640830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd015fa00 con 0x7efdd0160200 2024-03-18T19:10:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdced9d700 1 --2- 172.21.15.1:0/2320640830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdd0160200 0x7efdd0160680 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7efdd00674c0 tx=0x7efdc80114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdbeffd700 1 -- 172.21.15.1:0/2320640830 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc800e070 con 0x7efdd0160200 2024-03-18T19:10:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdbeffd700 1 -- 172.21.15.1:0/2320640830 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc8003c30 con 0x7efdd0160200 2024-03-18T19:10:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdd49bc700 1 -- 172.21.15.1:0/2320640830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd015fce0 con 0x7efdd0160200 2024-03-18T19:10:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.708+0000 7efdbeffd700 1 -- 172.21.15.1:0/2320640830 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc8003de0 con 0x7efdd0160200 2024-03-18T19:10:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.709+0000 7efdd49bc700 1 -- 172.21.15.1:0/2320640830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd01f3440 con 0x7efdd0160200 2024-03-18T19:10:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.710+0000 7efdd49bc700 1 -- 172.21.15.1:0/2320640830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd00ceea0 con 0x7efdd0160200 2024-03-18T19:10:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.710+0000 7efdbeffd700 1 -- 172.21.15.1:0/2320640830 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 83) v1 ==== 89256+0+0 (secure 0 0 0) 0x7efdc8011a20 con 0x7efdd0160200 2024-03-18T19:10:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.710+0000 7efdbeffd700 1 --2- 172.21.15.1:0/2320640830 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efdb006ae00 0x7efdb006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.710+0000 7efdce59c700 1 -- 172.21.15.1:0/2320640830 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efdb006ae00 msgr2=0x7efdb006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:10:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.710+0000 7efdbeffd700 1 -- 172.21.15.1:0/2320640830 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7efdc8089ad0 con 0x7efdd0160200 2024-03-18T19:10:14.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.710+0000 7efdce59c700 1 --2- 172.21.15.1:0/2320640830 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efdb006ae00 0x7efdb006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:10:14.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.715+0000 7efdbeffd700 1 -- 172.21.15.1:0/2320640830 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efdc80549c0 con 0x7efdd0160200 2024-03-18T19:10:14.757 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:14.761+0000 7f43b367b240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T19:10:14.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.911+0000 7efdce59c700 1 -- 172.21.15.1:0/2320640830 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efdb006ae00 msgr2=0x7efdb006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:10:14.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.911+0000 7efdce59c700 1 --2- 172.21.15.1:0/2320640830 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efdb006ae00 0x7efdb006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:10:14.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.934+0000 7efdd49bc700 1 -- 172.21.15.1:0/2320640830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7efdd00676a0 con 0x7efdd0160200 2024-03-18T19:10:14.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.938+0000 7efdbeffd700 1 -- 172.21.15.1:0/2320640830 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v54)=0 v54) v1 ==== 133+0+0 (secure 0 0 0) 0x7efdc8057fe0 con 0x7efdd0160200 2024-03-18T19:10:14.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.941+0000 7efdd49bc700 1 -- 172.21.15.1:0/2320640830 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efdb006ae00 msgr2=0x7efdb006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:10:14.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.941+0000 7efdd49bc700 1 --2- 172.21.15.1:0/2320640830 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efdb006ae00 0x7efdb006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.941+0000 7efdd49bc700 1 -- 172.21.15.1:0/2320640830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdd0160200 msgr2=0x7efdd0160680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:14.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.941+0000 7efdd49bc700 1 --2- 172.21.15.1:0/2320640830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdd0160200 0x7efdd0160680 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7efdd00674c0 tx=0x7efdc80114b0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.941+0000 7efdd49bc700 1 -- 172.21.15.1:0/2320640830 shutdown_connections 2024-03-18T19:10:14.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.941+0000 7efdd49bc700 1 --2- 172.21.15.1:0/2320640830 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efdb006ae00 0x7efdb006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.941+0000 7efdd49bc700 1 --2- 172.21.15.1:0/2320640830 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efdb8003a30 0x7efdd00caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.941+0000 7efdd49bc700 1 --2- 172.21.15.1:0/2320640830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efdd0160200 0x7efdd0160680 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.941+0000 7efdd49bc700 1 --2- 172.21.15.1:0/2320640830 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdd0157530 0x7efdd015aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:14.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.941+0000 7efdd49bc700 1 -- 172.21.15.1:0/2320640830 >> 172.21.15.1:0/2320640830 conn(0x7efdd00cca00 msgr2=0x7efdd00cf280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:14.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.941+0000 7efdd49bc700 1 -- 172.21.15.1:0/2320640830 shutdown_connections 2024-03-18T19:10:14.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:14.941+0000 7efdd49bc700 1 -- 172.21.15.1:0/2320640830 wait complete. 2024-03-18T19:10:14.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T19:10:14.951 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-03-18T19:10:14.973 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1 1m 2024-03-18T19:10:15.110 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:15.114+0000 7ff496682240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:10:15.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.162+0000 7f0fc4bb3700 1 Processor -- start 2024-03-18T19:10:15.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.162+0000 7f0fc4bb3700 1 -- start start 2024-03-18T19:10:15.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.163+0000 7f0fc4bb3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0fc0071550 0x7f0fc01661f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:15.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.163+0000 7f0fc4bb3700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc015b750 con 0x7f0fc015b370 2024-03-18T19:10:15.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.163+0000 7f0fc4bb3700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc00ccc20 con 0x7f0fc00aecb0 2024-03-18T19:10:15.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.163+0000 7f0fc4bb3700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc00cf510 con 0x7f0fc0071550 2024-03-18T19:10:15.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.163+0000 7f0fbed9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0fc0071550 0x7f0fc01661f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:15.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.163+0000 7f0fbe59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0fc015b370 0x7f0fc00cec70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56948/0 (socket says 172.21.15.1:56948) 2024-03-18T19:10:15.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.163+0000 7f0fbe59c700 1 -- 172.21.15.1:0/2648400954 learned_addr learned my addr 172.21.15.1:0/2648400954 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:10:15.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.163+0000 7f0fbd59a700 1 -- 172.21.15.1:0/2648400954 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469152468 0 0) 0x7f0fc015b750 con 0x7f0fc015b370 2024-03-18T19:10:15.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.164+0000 7f0fbd59a700 1 -- 172.21.15.1:0/2648400954 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa80035b0 con 0x7f0fc015b370 2024-03-18T19:10:15.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.164+0000 7f0fbed9d700 1 -- 172.21.15.1:0/2648400954 >> v1:172.21.15.165:6789/0 conn(0x7f0fc00aecb0 legacy=0x7f0fc0070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:15.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.164+0000 7f0fbed9d700 1 -- 172.21.15.1:0/2648400954 >> v1:172.21.15.1:6789/0 conn(0x7f0fc015b370 legacy=0x7f0fc00cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:15.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.164+0000 7f0fbed9d700 1 -- 172.21.15.1:0/2648400954 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc00cc2f0 con 0x7f0fc0071550 2024-03-18T19:10:15.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.164+0000 7f0fbed9d700 1 --2- 172.21.15.1:0/2648400954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc0071550 0x7f0fc01661f0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8009d90 tx=0x7f0fb800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6a64677a222f1b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:15.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.164+0000 7f0fbd59a700 1 -- 172.21.15.1:0/2648400954 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb800e070 con 0x7f0fc0071550 2024-03-18T19:10:15.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.164+0000 7f0fbd59a700 1 -- 172.21.15.1:0/2648400954 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb800bb20 con 0x7f0fc0071550 2024-03-18T19:10:15.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.164+0000 7f0fbd59a700 1 -- 172.21.15.1:0/2648400954 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8012430 con 0x7f0fc0071550 2024-03-18T19:10:15.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.165+0000 7f0fc4bb3700 1 -- 172.21.15.1:0/2648400954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc0071550 msgr2=0x7f0fc01661f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:15.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.165+0000 7f0fc4bb3700 1 --2- 172.21.15.1:0/2648400954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc0071550 0x7f0fc01661f0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8009d90 tx=0x7f0fb800b330 comp rx=0 tx=0).stop 2024-03-18T19:10:15.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.165+0000 7f0fc4bb3700 1 -- 172.21.15.1:0/2648400954 shutdown_connections 2024-03-18T19:10:15.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.165+0000 7f0fc4bb3700 1 --2- 172.21.15.1:0/2648400954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc0071550 0x7f0fc01661f0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:15.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.165+0000 7f0fc4bb3700 1 -- 172.21.15.1:0/2648400954 >> 172.21.15.1:0/2648400954 conn(0x7f0fc00b1350 msgr2=0x7f0fc00b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:15.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.165+0000 7f0fc4bb3700 1 -- 172.21.15.1:0/2648400954 shutdown_connections 2024-03-18T19:10:15.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.165+0000 7f0fc4bb3700 1 -- 172.21.15.1:0/2648400954 wait complete. 2024-03-18T19:10:15.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.167+0000 7f0fc4bb3700 1 Processor -- start 2024-03-18T19:10:15.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.167+0000 7f0fc4bb3700 1 -- start start 2024-03-18T19:10:15.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.168+0000 7f0fc4bb3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fc0071550 0x7f0fc0159a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:15.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.168+0000 7f0fc4bb3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fc00aecb0 0x7f0fc015f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:15.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.168+0000 7f0fc4bb3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc015b370 0x7f0fc0147b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:15.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.168+0000 7f0fc4bb3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc00ad350 con 0x7f0fc015b370 2024-03-18T19:10:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.168+0000 7f0fc4bb3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc00ad630 con 0x7f0fc0071550 2024-03-18T19:10:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.169+0000 7f0fc4bb3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc00ad910 con 0x7f0fc00aecb0 2024-03-18T19:10:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.169+0000 7f0fbed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc015b370 0x7f0fc0147b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.169+0000 7f0fbed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc015b370 0x7f0fc0147b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51968/0 (socket says 172.21.15.1:51968) 2024-03-18T19:10:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.169+0000 7f0fbed9d700 1 -- 172.21.15.1:0/3993387840 learned_addr learned my addr 172.21.15.1:0/3993387840 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.169+0000 7f0fbdd9b700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fc00aecb0 0x7f0fc015f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.169+0000 7f0fbe59c700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fc0071550 0x7f0fc0159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.169+0000 7f0fbed9d700 1 -- 172.21.15.1:0/3993387840 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fc00aecb0 msgr2=0x7f0fc015f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.169+0000 7f0fbed9d700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fc00aecb0 0x7f0fc015f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.169+0000 7f0fbed9d700 1 -- 172.21.15.1:0/3993387840 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fc0071550 msgr2=0x7f0fc0159a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.169+0000 7f0fbed9d700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fc0071550 0x7f0fc0159a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.169+0000 7f0fbed9d700 1 -- 172.21.15.1:0/3993387840 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb8009a70 con 0x7f0fc015b370 2024-03-18T19:10:15.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.169+0000 7f0fbed9d700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc015b370 0x7f0fc0147b00 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8013000 tx=0x7f0fb8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:15.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.169+0000 7f0faeffd700 1 -- 172.21.15.1:0/3993387840 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8012430 con 0x7f0fc015b370 2024-03-18T19:10:15.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.169+0000 7f0faeffd700 1 -- 172.21.15.1:0/3993387840 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb8020070 con 0x7f0fc015b370 2024-03-18T19:10:15.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.169+0000 7f0fc4bb3700 1 -- 172.21.15.1:0/3993387840 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc00adbf0 con 0x7f0fc015b370 2024-03-18T19:10:15.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.169+0000 7f0faeffd700 1 -- 172.21.15.1:0/3993387840 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8004280 con 0x7f0fc015b370 2024-03-18T19:10:15.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.169+0000 7f0fc4bb3700 1 -- 172.21.15.1:0/3993387840 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc0159f60 con 0x7f0fc015b370 2024-03-18T19:10:15.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.170+0000 7f0fc4bb3700 1 -- 172.21.15.1:0/3993387840 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc015a0f0 con 0x7f0fc015b370 2024-03-18T19:10:15.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.171+0000 7f0faeffd700 1 -- 172.21.15.1:0/3993387840 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 83) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f0fb80043e0 con 0x7f0fc015b370 2024-03-18T19:10:15.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.171+0000 7f0faeffd700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fa006ae00 0x7f0fa006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:15.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.171+0000 7f0fbe59c700 1 -- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fa006ae00 msgr2=0x7f0fa006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:10:15.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.171+0000 7f0faeffd700 1 -- 172.21.15.1:0/3993387840 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0fb8022070 con 0x7f0fc015b370 2024-03-18T19:10:15.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.172+0000 7f0fbe59c700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fa006ae00 0x7f0fa006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T19:10:15.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.176+0000 7f0faeffd700 1 -- 172.21.15.1:0/3993387840 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0fb8056670 con 0x7f0fc015b370 2024-03-18T19:10:15.233 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:15.236+0000 7f43b367b240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T19:10:15.321 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:15.326+0000 7ff496682240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:10:15.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.373+0000 7f0fbe59c700 1 -- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fa006ae00 msgr2=0x7f0fa006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:10:15.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.373+0000 7f0fbe59c700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fa006ae00 0x7f0fa006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T19:10:15.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.405+0000 7f0fc4bb3700 1 -- 172.21.15.1:0/3993387840 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fc00cc8b0 con 0x7f0fa006ae00 2024-03-18T19:10:15.450 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:15.454+0000 7f43b367b240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T19:10:15.692 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:15.696+0000 7ff496682240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:10:15.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.774+0000 7f0fbe59c700 1 -- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fa006ae00 msgr2=0x7f0fa006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:10:15.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:15.774+0000 7f0fbe59c700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fa006ae00 0x7f0fa006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T19:10:15.838 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:15.841+0000 7f43b367b240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T19:10:16.272 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:16.276+0000 7ff496682240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:10:16.343 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:16.348+0000 7ff496682240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:10:16.452 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:16.455+0000 7f43b367b240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T19:10:16.469 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:16.474+0000 7ff496682240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:10:16.528 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:16.531+0000 7f43b367b240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T19:10:16.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:16.575+0000 7f0fbe59c700 1 -- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fa006ae00 msgr2=0x7f0fa006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:10:16.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:16.575+0000 7f0fbe59c700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fa006ae00 0x7f0fa006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T19:10:16.646 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:16.651+0000 7ff496682240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:10:16.662 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:16.665+0000 7f43b367b240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T19:10:16.718 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:16.722+0000 7ff496682240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:10:16.795 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:16.800+0000 7ff496682240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:10:16.847 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:16.851+0000 7f43b367b240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T19:10:16.872 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:16.877+0000 7ff496682240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:10:16.923 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:16.926+0000 7f43b367b240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T19:10:16.949 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:16.954+0000 7ff496682240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:10:17.006 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:17.010+0000 7f43b367b240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T19:10:17.088 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:17.092+0000 7f43b367b240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T19:10:17.118 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:17.123+0000 7ff496682240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:10:17.173 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:17.175+0000 7f43b367b240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T19:10:17.332 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:17.337+0000 7ff496682240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:10:17.351 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:17.354+0000 7f43b367b240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T19:10:17.484 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:17.488+0000 7ff496682240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:10:17.556 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:17.561+0000 7ff496682240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:10:17.573 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:17.576+0000 7f43b367b240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T19:10:17.732 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:17.735+0000 7f43b367b240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T19:10:17.809 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:17.812+0000 7f43b367b240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T19:10:17.998 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:18.003+0000 7ff496682240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:10:18.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:18.176+0000 7f0fbe59c700 1 -- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fa006ae00 msgr2=0x7f0fa006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T19:10:18.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:18.176+0000 7f0fbe59c700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fa006ae00 0x7f0fa006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T19:10:18.283 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:18.286+0000 7f43b367b240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T19:10:18.332 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:18.337+0000 7ff496682240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:10:18.495 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:18.500+0000 7ff496682240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:10:18.635 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:18.638+0000 7f43b367b240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T19:10:18.810 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:18.812+0000 7f43b367b240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:10:18.882 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:18.887+0000 7ff496682240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:10:19.048 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:19.053+0000 7ff496682240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:10:19.220 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:19.223+0000 7f43b367b240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T19:10:19.397 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:19.400+0000 7f43b367b240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T19:10:19.408 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:19.413+0000 7ff496682240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:10:19.497 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T19:10:19.501+0000 7ff496682240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:10:19.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:19.570+0000 7f0faeffd700 1 -- 172.21.15.1:0/3993387840 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 84) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f0fb8056a40 con 0x7f0fc015b370 2024-03-18T19:10:19.776 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:19.780+0000 7f43b367b240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T19:10:19.871 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:19.874+0000 7f43b367b240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T19:10:19.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:19.892+0000 7f0faeffd700 1 -- 172.21.15.1:0/3993387840 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 85) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f0fb8089a00 con 0x7f0fc015b370 2024-03-18T19:10:19.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:19.893+0000 7f0faeffd700 1 -- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fa006ae00 msgr2=0x7f0fa006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T19:10:19.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:19.893+0000 7f0faeffd700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fa006ae00 0x7f0fa006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:20.031 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:20.034+0000 7f4326298700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:10:20.031 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:20.034+0000 7f4326298700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:10:20.033 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:20.034+0000 7f4326298700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:10:20.033 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:20.034+0000 7f4326298700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:10:20.033 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:20.034+0000 7f4326298700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:10:20.033 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:20.034+0000 7f4328a9d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:10:20.034 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:20.034+0000 7f4328a9d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:10:20.034 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:20.034+0000 7f4328a9d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:10:20.034 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:20.034+0000 7f4328a9d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:10:20.034 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:10:20.034+0000 7f4328a9d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:10:20.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:20.901+0000 7f0faeffd700 1 -- 172.21.15.1:0/3993387840 <== mon.0 v2:172.21.15.1:3300/0 9 ==== mgrmap(e 86) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f0fb8057310 con 0x7f0fc015b370 2024-03-18T19:10:20.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:20.902+0000 7f0faeffd700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fa006ae00 0x7f0fa006d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:20.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:20.902+0000 7f0faeffd700 1 -- 172.21.15.1:0/3993387840 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fa001d270 con 0x7f0fa006ae00 2024-03-18T19:10:20.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:20.905+0000 7f0fbe59c700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fa006ae00 0x7f0fa006d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:20.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:20.907+0000 7f0fbe59c700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fa006ae00 0x7f0fa006d240 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0007fd0 tx=0x7f0fb0007dc0 comp rx=0 tx=0).ready entity=mgr.9561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:20.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:20.967+0000 7f0faeffd700 1 -- 172.21.15.1:0/3993387840 <== mgr.9561 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f0fa001d270 con 0x7f0fa006ae00 2024-03-18T19:10:20.964 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path /snap_test_dir1 2024-03-18T19:10:20.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:20.969+0000 7f0fc4bb3700 1 -- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fa006ae00 msgr2=0x7f0fa006d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:20.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:20.970+0000 7f0fc4bb3700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fa006ae00 0x7f0fa006d240 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0007fd0 tx=0x7f0fb0007dc0 comp rx=0 tx=0).stop 2024-03-18T19:10:20.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:20.970+0000 7f0fc4bb3700 1 -- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc015b370 msgr2=0x7f0fc0147b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:20.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:20.970+0000 7f0fc4bb3700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc015b370 0x7f0fc0147b00 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8013000 tx=0x7f0fb8004ad0 comp rx=0 tx=0).stop 2024-03-18T19:10:20.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:20.970+0000 7f0fc4bb3700 1 -- 172.21.15.1:0/3993387840 shutdown_connections 2024-03-18T19:10:20.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:20.970+0000 7f0fc4bb3700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fa006ae00 0x7f0fa006d240 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:20.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:20.970+0000 7f0fc4bb3700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc015b370 0x7f0fc0147b00 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:20.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:20.970+0000 7f0fc4bb3700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fc0071550 0x7f0fc0159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:20.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:20.970+0000 7f0fc4bb3700 1 --2- 172.21.15.1:0/3993387840 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fc00aecb0 0x7f0fc015f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:20.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:20.970+0000 7f0fc4bb3700 1 -- 172.21.15.1:0/3993387840 >> 172.21.15.1:0/3993387840 conn(0x7f0fc00b1350 msgr2=0x7f0fc006d740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:20.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:20.970+0000 7f0fc4bb3700 1 -- 172.21.15.1:0/3993387840 shutdown_connections 2024-03-18T19:10:20.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:20.970+0000 7f0fc4bb3700 1 -- 172.21.15.1:0/3993387840 wait complete. 2024-03-18T19:10:20.980 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1 2024-03-18T19:10:20.980 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add /snap_test_dir1 55n 2024-03-18T19:10:21.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.115+0000 7f5df7a36700 1 Processor -- start 2024-03-18T19:10:21.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.115+0000 7f5df7a36700 1 -- start start 2024-03-18T19:10:21.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.116+0000 7f5df7a36700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5df0070920 0x7f5df006bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:21.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.116+0000 7f5df7a36700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5df006c0f0 0x7f5df015e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:21.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.116+0000 7f5df7a36700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df0161390 con 0x7f5df0146d70 2024-03-18T19:10:21.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.116+0000 7f5df7a36700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5df015dc60 con 0x7f5df006c0f0 2024-03-18T19:10:21.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.116+0000 7f5df7a36700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5df006d030 con 0x7f5df0070920 2024-03-18T19:10:21.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.116+0000 7f5df5fd3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5df006c0f0 0x7f5df015e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:21.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.116+0000 7f5df57d2700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5df0146d70 0x7f5df0070210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56962/0 (socket says 172.21.15.1:56962) 2024-03-18T19:10:21.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.116+0000 7f5df57d2700 1 -- 172.21.15.1:0/1631379703 learned_addr learned my addr 172.21.15.1:0/1631379703 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T19:10:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.117+0000 7f5df4fd1700 1 --2- 172.21.15.1:0/1631379703 >> v2:172.21.15.165:3300/0 conn(0x7f5df0070920 0x7f5df006bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.117+0000 7f5de7fff700 1 -- 172.21.15.1:0/1631379703 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531295397 0 0) 0x7f5de0001110 con 0x7f5df0146d70 2024-03-18T19:10:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.117+0000 7f5de7fff700 1 -- 172.21.15.1:0/1631379703 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ddc0035b0 con 0x7f5df0146d70 2024-03-18T19:10:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.117+0000 7f5df5fd3700 1 -- 172.21.15.1:0/1631379703 >> v2:172.21.15.165:3300/0 conn(0x7f5df0070920 msgr2=0x7f5df006bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.117+0000 7f5df5fd3700 1 --2- 172.21.15.1:0/1631379703 >> v2:172.21.15.165:3300/0 conn(0x7f5df0070920 0x7f5df006bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.117+0000 7f5df5fd3700 1 -- 172.21.15.1:0/1631379703 >> v1:172.21.15.1:6789/0 conn(0x7f5df0146d70 legacy=0x7f5df0070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.117+0000 7f5df5fd3700 1 -- 172.21.15.1:0/1631379703 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df006c520 con 0x7f5df006c0f0 2024-03-18T19:10:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.117+0000 7f5df5fd3700 1 --2- 172.21.15.1:0/1631379703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df006c0f0 0x7f5df015e0f0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f5df0162140 tx=0x7f5de800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434c9707a56dca4c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.117+0000 7f5de7fff700 1 -- 172.21.15.1:0/1631379703 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de800e070 con 0x7f5df006c0f0 2024-03-18T19:10:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.117+0000 7f5de7fff700 1 -- 172.21.15.1:0/1631379703 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de800bb20 con 0x7f5df006c0f0 2024-03-18T19:10:21.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.117+0000 7f5de7fff700 1 -- 172.21.15.1:0/1631379703 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de800be90 con 0x7f5df006c0f0 2024-03-18T19:10:21.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.117+0000 7f5df7a36700 1 -- 172.21.15.1:0/1631379703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df006c0f0 msgr2=0x7f5df015e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:21.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.117+0000 7f5df7a36700 1 --2- 172.21.15.1:0/1631379703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df006c0f0 0x7f5df015e0f0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f5df0162140 tx=0x7f5de800b330 comp rx=0 tx=0).stop 2024-03-18T19:10:21.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.118+0000 7f5df7a36700 1 -- 172.21.15.1:0/1631379703 shutdown_connections 2024-03-18T19:10:21.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.118+0000 7f5df7a36700 1 --2- 172.21.15.1:0/1631379703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df006c0f0 0x7f5df015e0f0 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:21.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.118+0000 7f5df7a36700 1 --2- 172.21.15.1:0/1631379703 >> v2:172.21.15.165:3300/0 conn(0x7f5df0070920 0x7f5df006bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:21.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.118+0000 7f5df7a36700 1 -- 172.21.15.1:0/1631379703 >> 172.21.15.1:0/1631379703 conn(0x7f5df00b2e80 msgr2=0x7f5df00b3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:21.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.118+0000 7f5df7a36700 1 -- 172.21.15.1:0/1631379703 shutdown_connections 2024-03-18T19:10:21.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.118+0000 7f5df7a36700 1 -- 172.21.15.1:0/1631379703 wait complete. 2024-03-18T19:10:21.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.119+0000 7f5df7a36700 1 Processor -- start 2024-03-18T19:10:21.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.119+0000 7f5df7a36700 1 -- start start 2024-03-18T19:10:21.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5df7a36700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5df006c0f0 0x7f5df0152c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:21.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5df7a36700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5df0070920 0x7f5df00cb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:21.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5df7a36700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df0146d70 0x7f5df00cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:21.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5df7a36700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df013b460 con 0x7f5df0146d70 2024-03-18T19:10:21.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5df7a36700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df0141c80 con 0x7f5df006c0f0 2024-03-18T19:10:21.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5df7a36700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5df01f4050 con 0x7f5df0070920 2024-03-18T19:10:21.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5df5fd3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df0146d70 0x7f5df00cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:21.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5df5fd3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df0146d70 0x7f5df00cba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52134/0 (socket says 172.21.15.1:52134) 2024-03-18T19:10:21.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5df5fd3700 1 -- 172.21.15.1:0/2598289300 learned_addr learned my addr 172.21.15.1:0/2598289300 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:21.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5df4fd1700 1 --2- 172.21.15.1:0/2598289300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5df0070920 0x7f5df00cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:21.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5df57d2700 1 --2- 172.21.15.1:0/2598289300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5df006c0f0 0x7f5df0152c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:21.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5df5fd3700 1 -- 172.21.15.1:0/2598289300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5df0070920 msgr2=0x7f5df00cb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:21.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5df5fd3700 1 --2- 172.21.15.1:0/2598289300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5df0070920 0x7f5df00cb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:21.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5df5fd3700 1 -- 172.21.15.1:0/2598289300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5df006c0f0 msgr2=0x7f5df0152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:21.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5df5fd3700 1 --2- 172.21.15.1:0/2598289300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5df006c0f0 0x7f5df0152c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:21.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5df5fd3700 1 -- 172.21.15.1:0/2598289300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de8009a70 con 0x7f5df0146d70 2024-03-18T19:10:21.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5df4fd1700 1 --2- 172.21.15.1:0/2598289300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5df0070920 0x7f5df00cb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T19:10:21.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5df5fd3700 1 --2- 172.21.15.1:0/2598289300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df0146d70 0x7f5df00cba90 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f5de8014040 tx=0x7f5de8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:21.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5de5ffb700 1 -- 172.21.15.1:0/2598289300 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de8004960 con 0x7f5df0146d70 2024-03-18T19:10:21.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5de5ffb700 1 -- 172.21.15.1:0/2598289300 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de8020070 con 0x7f5df0146d70 2024-03-18T19:10:21.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5df7a36700 1 -- 172.21.15.1:0/2598289300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df015d170 con 0x7f5df0146d70 2024-03-18T19:10:21.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.120+0000 7f5de5ffb700 1 -- 172.21.15.1:0/2598289300 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de80276a0 con 0x7f5df0146d70 2024-03-18T19:10:21.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.121+0000 7f5df7a36700 1 -- 172.21.15.1:0/2598289300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df01f3720 con 0x7f5df0146d70 2024-03-18T19:10:21.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.121+0000 7f5df7a36700 1 -- 172.21.15.1:0/2598289300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5df01f41c0 con 0x7f5df0146d70 2024-03-18T19:10:21.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.122+0000 7f5de5ffb700 1 -- 172.21.15.1:0/2598289300 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 86) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f5de8012430 con 0x7f5df0146d70 2024-03-18T19:10:21.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.122+0000 7f5de5ffb700 1 --2- 172.21.15.1:0/2598289300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5dd406ab30 0x7f5dd406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:21.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.122+0000 7f5de5ffb700 1 -- 172.21.15.1:0/2598289300 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6401+0+0 (secure 0 0 0) 0x7f5de8022070 con 0x7f5df0146d70 2024-03-18T19:10:21.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.122+0000 7f5df57d2700 1 --2- 172.21.15.1:0/2598289300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5dd406ab30 0x7f5dd406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:21.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.123+0000 7f5df57d2700 1 --2- 172.21.15.1:0/2598289300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5dd406ab30 0x7f5dd406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5de0005910 tx=0x7f5de0009040 comp rx=0 tx=0).ready entity=mgr.9561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:21.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.127+0000 7f5de5ffb700 1 -- 172.21.15.1:0/2598289300 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5de8056280 con 0x7f5df0146d70 2024-03-18T19:10:21.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.354+0000 7f5df7a36700 1 -- 172.21.15.1:0/2598289300 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "/snap_test_dir1", "retention_spec_or_period": "55n", "target": ["mon-mgr", ""]}) v1 -- 0x7f5df006ca70 con 0x7f5dd406ab30 2024-03-18T19:10:21.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.406+0000 7f5de5ffb700 1 -- 172.21.15.1:0/2598289300 <== mgr.9561 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+40 (secure 0 0 0) 0x7f5de0005940 con 0x7f5dd406ab30 2024-03-18T19:10:21.403 INFO:teuthology.orchestra.run.smithi001.stdout:Retention added to path /snap_test_dir1 2024-03-18T19:10:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.408+0000 7f5df7a36700 1 -- 172.21.15.1:0/2598289300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5dd406ab30 msgr2=0x7f5dd406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.408+0000 7f5df7a36700 1 --2- 172.21.15.1:0/2598289300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5dd406ab30 0x7f5dd406cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5de0005910 tx=0x7f5de0009040 comp rx=0 tx=0).stop 2024-03-18T19:10:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.408+0000 7f5df7a36700 1 -- 172.21.15.1:0/2598289300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df0146d70 msgr2=0x7f5df00cba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.408+0000 7f5df7a36700 1 --2- 172.21.15.1:0/2598289300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df0146d70 0x7f5df00cba90 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f5de8014040 tx=0x7f5de8004ad0 comp rx=0 tx=0).stop 2024-03-18T19:10:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.408+0000 7f5df7a36700 1 -- 172.21.15.1:0/2598289300 shutdown_connections 2024-03-18T19:10:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.408+0000 7f5df7a36700 1 --2- 172.21.15.1:0/2598289300 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5dd406ab30 0x7f5dd406cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.408+0000 7f5df7a36700 1 --2- 172.21.15.1:0/2598289300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5df0146d70 0x7f5df00cba90 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.408+0000 7f5df7a36700 1 --2- 172.21.15.1:0/2598289300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5df006c0f0 0x7f5df0152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.408+0000 7f5df7a36700 1 --2- 172.21.15.1:0/2598289300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5df0070920 0x7f5df00cb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.408+0000 7f5df7a36700 1 -- 172.21.15.1:0/2598289300 >> 172.21.15.1:0/2598289300 conn(0x7f5df00b2e80 msgr2=0x7f5df015e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:21.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.409+0000 7f5df7a36700 1 -- 172.21.15.1:0/2598289300 shutdown_connections 2024-03-18T19:10:21.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.409+0000 7f5df7a36700 1 -- 172.21.15.1:0/2598289300 wait complete. 2024-03-18T19:10:21.418 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path /snap_test_dir1 2024-03-18T19:10:21.418 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2024-03-18T19:10:21.418 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json /snap_test_dir1 2024-03-18T19:10:21.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.549+0000 7fc03045c700 1 Processor -- start 2024-03-18T19:10:21.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.549+0000 7fc03045c700 1 -- start start 2024-03-18T19:10:21.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.550+0000 7fc03045c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc028160a70 0x7fc0280d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:21.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.550+0000 7fc03045c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc028160e50 0x7fc0280cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:21.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.550+0000 7fc03045c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc028162170 con 0x7fc0281612d0 2024-03-18T19:10:21.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.550+0000 7fc03045c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0280cef30 con 0x7fc028160a70 2024-03-18T19:10:21.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.550+0000 7fc03045c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0280cf210 con 0x7fc028160e50 2024-03-18T19:10:21.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.550+0000 7fc02d9f7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc028160e50 0x7fc0280cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:21.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.550+0000 7fc02e1f8700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc028160a70 0x7fc0280d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:21.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.550+0000 7fc02d9f7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc028160e50 0x7fc0280cffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:41374/0 (socket says 172.21.15.1:41374) 2024-03-18T19:10:21.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.550+0000 7fc02e9f9700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fc0281612d0 0x7fc0280c6da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:35310/0 (socket says 172.21.15.1:35310) 2024-03-18T19:10:21.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.550+0000 7fc02d9f7700 1 -- 172.21.15.1:0/4078918023 learned_addr learned my addr 172.21.15.1:0/4078918023 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:21.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.551+0000 7fc02d9f7700 1 -- 172.21.15.1:0/4078918023 >> v2:172.21.15.165:3300/0 conn(0x7fc028160a70 msgr2=0x7fc0280d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T19:10:21.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.551+0000 7fc02d9f7700 1 --2- 172.21.15.1:0/4078918023 >> v2:172.21.15.165:3300/0 conn(0x7fc028160a70 0x7fc0280d3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:21.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.551+0000 7fc02d9f7700 1 -- 172.21.15.1:0/4078918023 >> v1:172.21.15.165:6789/0 conn(0x7fc0281612d0 legacy=0x7fc0280c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:21.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.551+0000 7fc02d9f7700 1 -- 172.21.15.1:0/4078918023 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0280cf4f0 con 0x7fc028160e50 2024-03-18T19:10:21.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.551+0000 7fc02d1f6700 1 -- 172.21.15.1:0/4078918023 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173279167 0 0) 0x7fc020001490 con 0x7fc0281612d0 2024-03-18T19:10:21.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.551+0000 7fc02e1f8700 1 --2- 172.21.15.1:0/4078918023 >> v2:172.21.15.165:3300/0 conn(0x7fc028160a70 0x7fc0280d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:10:21.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.551+0000 7fc02d9f7700 1 --2- 172.21.15.1:0/4078918023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc028160e50 0x7fc0280cffd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc0240098f0 tx=0x7fc024009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49b781f3368c3b74 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:21.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.551+0000 7fc02d1f6700 1 -- 172.21.15.1:0/4078918023 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc024011830 con 0x7fc028160e50 2024-03-18T19:10:21.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.551+0000 7fc02d1f6700 1 -- 172.21.15.1:0/4078918023 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc024011990 con 0x7fc028160e50 2024-03-18T19:10:21.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.551+0000 7fc02d1f6700 1 -- 172.21.15.1:0/4078918023 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc024011d20 con 0x7fc028160e50 2024-03-18T19:10:21.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.551+0000 7fc03045c700 1 -- 172.21.15.1:0/4078918023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc028160e50 msgr2=0x7fc0280cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:21.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.551+0000 7fc03045c700 1 --2- 172.21.15.1:0/4078918023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc028160e50 0x7fc0280cffd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc0240098f0 tx=0x7fc024009c00 comp rx=0 tx=0).stop 2024-03-18T19:10:21.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.552+0000 7fc03045c700 1 -- 172.21.15.1:0/4078918023 shutdown_connections 2024-03-18T19:10:21.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.552+0000 7fc03045c700 1 --2- 172.21.15.1:0/4078918023 >> v2:172.21.15.165:3300/0 conn(0x7fc028160a70 0x7fc0280d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:21.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.552+0000 7fc03045c700 1 --2- 172.21.15.1:0/4078918023 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc028160e50 0x7fc0280cffd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:21.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.552+0000 7fc03045c700 1 -- 172.21.15.1:0/4078918023 >> 172.21.15.1:0/4078918023 conn(0x7fc02809c560 msgr2=0x7fc0280d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:21.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.552+0000 7fc03045c700 1 -- 172.21.15.1:0/4078918023 shutdown_connections 2024-03-18T19:10:21.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.552+0000 7fc03045c700 1 -- 172.21.15.1:0/4078918023 wait complete. 2024-03-18T19:10:21.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.553+0000 7fc03045c700 1 Processor -- start 2024-03-18T19:10:21.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.553+0000 7fc03045c700 1 -- start start 2024-03-18T19:10:21.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc03045c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc028160a70 0x7fc028148a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:21.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc03045c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc028160e50 0x7fc0281f5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:21.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc03045c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0281612d0 0x7fc0280b7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:21.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc03045c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0280ccda0 con 0x7fc028160e50 2024-03-18T19:10:21.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc03045c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02815b250 con 0x7fc0281612d0 2024-03-18T19:10:21.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc03045c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02815b530 con 0x7fc028160a70 2024-03-18T19:10:21.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc02d9f7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc028160e50 0x7fc0281f5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:21.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc02e1f8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc028160a70 0x7fc028148a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:21.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc02d9f7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc028160e50 0x7fc0281f5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52138/0 (socket says 172.21.15.1:52138) 2024-03-18T19:10:21.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc02d9f7700 1 -- 172.21.15.1:0/1474904007 learned_addr learned my addr 172.21.15.1:0/1474904007 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T19:10:21.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc02e1f8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc028160a70 0x7fc028148a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:41382/0 (socket says 172.21.15.1:41382) 2024-03-18T19:10:21.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc02e9f9700 1 --2- 172.21.15.1:0/1474904007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0281612d0 0x7fc0280b7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:21.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc02d9f7700 1 -- 172.21.15.1:0/1474904007 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc028160a70 msgr2=0x7fc028148a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:21.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc02d9f7700 1 --2- 172.21.15.1:0/1474904007 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc028160a70 0x7fc028148a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:21.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc02d9f7700 1 -- 172.21.15.1:0/1474904007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0281612d0 msgr2=0x7fc0280b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:21.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc02d9f7700 1 --2- 172.21.15.1:0/1474904007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0281612d0 0x7fc0280b7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:21.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc02d9f7700 1 -- 172.21.15.1:0/1474904007 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0180094a0 con 0x7fc028160e50 2024-03-18T19:10:21.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc02e1f8700 1 --2- 172.21.15.1:0/1474904007 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc028160a70 0x7fc028148a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T19:10:21.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc02d9f7700 1 --2- 172.21.15.1:0/1474904007 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc028160e50 0x7fc0281f5f40 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fc024009c40 tx=0x7fc024004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:21.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.554+0000 7fc01effd700 1 -- 172.21.15.1:0/1474904007 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc024004550 con 0x7fc028160e50 2024-03-18T19:10:21.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.555+0000 7fc01effd700 1 -- 172.21.15.1:0/1474904007 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02401f070 con 0x7fc028160e50 2024-03-18T19:10:21.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.555+0000 7fc03045c700 1 -- 172.21.15.1:0/1474904007 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc024009550 con 0x7fc028160e50 2024-03-18T19:10:21.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.555+0000 7fc01effd700 1 -- 172.21.15.1:0/1474904007 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0240038e0 con 0x7fc028160e50 2024-03-18T19:10:21.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.555+0000 7fc03045c700 1 -- 172.21.15.1:0/1474904007 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0281f6480 con 0x7fc028160e50 2024-03-18T19:10:21.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.556+0000 7fc03045c700 1 -- 172.21.15.1:0/1474904007 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc02815ba50 con 0x7fc028160e50 2024-03-18T19:10:21.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.556+0000 7fc01effd700 1 -- 172.21.15.1:0/1474904007 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 86) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fc024003a40 con 0x7fc028160e50 2024-03-18T19:10:21.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.556+0000 7fc01effd700 1 --2- 172.21.15.1:0/1474904007 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc00c06ab60 0x7fc00c06d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T19:10:21.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.556+0000 7fc01effd700 1 -- 172.21.15.1:0/1474904007 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6401+0+0 (secure 0 0 0) 0x7fc024021070 con 0x7fc028160e50 2024-03-18T19:10:21.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.556+0000 7fc02e1f8700 1 --2- 172.21.15.1:0/1474904007 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc00c06ab60 0x7fc00c06d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T19:10:21.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.557+0000 7fc02e1f8700 1 --2- 172.21.15.1:0/1474904007 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc00c06ab60 0x7fc00c06d020 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc018011380 tx=0x7fc018009210 comp rx=0 tx=0).ready entity=mgr.9561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T19:10:21.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.561+0000 7fc01effd700 1 -- 172.21.15.1:0/1474904007 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc02815ba50 con 0x7fc028160e50 2024-03-18T19:10:21.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.789+0000 7fc03045c700 1 -- 172.21.15.1:0/1474904007 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc0280d40b0 con 0x7fc00c06ab60 2024-03-18T19:10:21.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.799+0000 7fc01effd700 1 -- 172.21.15.1:0/1474904007 <== mgr.9561 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+74 (secure 0 0 0) 0x7fc018011d40 con 0x7fc00c06ab60 2024-03-18T19:10:21.796 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T19:10:21.796 INFO:teuthology.orchestra.run.smithi001.stdout:{"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2024-03-18T19:10:21.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.801+0000 7fc03045c700 1 -- 172.21.15.1:0/1474904007 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc00c06ab60 msgr2=0x7fc00c06d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:21.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.801+0000 7fc03045c700 1 --2- 172.21.15.1:0/1474904007 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc00c06ab60 0x7fc00c06d020 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc018011380 tx=0x7fc018009210 comp rx=0 tx=0).stop 2024-03-18T19:10:21.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.801+0000 7fc03045c700 1 -- 172.21.15.1:0/1474904007 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc028160e50 msgr2=0x7fc0281f5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T19:10:21.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.801+0000 7fc03045c700 1 --2- 172.21.15.1:0/1474904007 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc028160e50 0x7fc0281f5f40 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fc024009c40 tx=0x7fc024004750 comp rx=0 tx=0).stop 2024-03-18T19:10:21.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.801+0000 7fc03045c700 1 -- 172.21.15.1:0/1474904007 shutdown_connections 2024-03-18T19:10:21.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.801+0000 7fc03045c700 1 --2- 172.21.15.1:0/1474904007 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc00c06ab60 0x7fc00c06d020 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:21.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.801+0000 7fc03045c700 1 --2- 172.21.15.1:0/1474904007 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc028160a70 0x7fc028148a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:21.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.801+0000 7fc03045c700 1 --2- 172.21.15.1:0/1474904007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0281612d0 0x7fc0280b7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:21.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.801+0000 7fc03045c700 1 --2- 172.21.15.1:0/1474904007 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc028160e50 0x7fc0281f5f40 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T19:10:21.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.801+0000 7fc03045c700 1 -- 172.21.15.1:0/1474904007 >> 172.21.15.1:0/1474904007 conn(0x7fc02809c560 msgr2=0x7fc02806b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T19:10:21.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.801+0000 7fc03045c700 1 -- 172.21.15.1:0/1474904007 shutdown_connections 2024-03-18T19:10:21.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T19:10:21.801+0000 7fc03045c700 1 -- 172.21.15.1:0/1474904007 wait complete. 2024-03-18T19:10:21.810 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2024-03-18T19:10:21.810 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': '/snap_test_dir1', 'schedule': ['1m'], 'retention': [{'n': 55}]} 2024-03-18T19:10:22.207 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:10:22.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:10:52.348 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:10:52.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:11:00.414 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:11:00.418+0000 7f431827c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:11:00.414 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:11:00.418+0000 7f431827c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:11:00.414 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:11:00.418+0000 7f431827c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:11:00.414 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:11:00.418+0000 7f431827c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:11:00.414 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T19:11:00.418+0000 7f431827c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T19:11:22.421 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:11:22.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:11:52.547 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:11:52.550 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:12:22.642 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:12:22.645 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:12:52.772 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:12:52.775 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:13:22.858 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:13:22.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:13:53.025 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:13:53.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:14:23.108 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:14:23.111 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:14:53.254 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:14:53.257 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:15:23.367 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:15:23.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:15:53.496 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:15:53.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:16:23.600 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:16:23.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:16:53.947 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:16:53.951 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:17:24.026 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:17:24.033 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:17:54.211 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:17:54.214 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:18:24.277 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:18:24.282 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:18:54.406 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:18:54.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:19:24.472 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:19:24.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:19:54.598 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:19:54.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:20:24.769 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:20:24.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:20:54.834 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:20:54.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:21:24.986 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:21:24.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:21:55.107 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:21:55.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:22:25.245 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:22:25.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:22:55.382 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:22:55.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:23:25.562 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:23:25.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:23:55.699 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:23:55.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:24:25.828 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:24:25.831 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:24:55.966 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:24:55.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:25:26.036 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:25:26.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:25:56.195 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:25:56.199 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:26:26.334 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:26:26.339 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:26:56.475 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:26:56.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:27:26.611 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:27:26.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:27:56.673 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:27:56.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:28:26.831 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:28:26.834 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:28:56.907 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:28:56.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:29:27.037 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:29:27.040 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:29:57.132 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:29:57.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:30:27.287 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:30:27.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:30:57.351 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:30:57.355 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:31:27.495 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:31:27.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:31:57.574 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:31:57.578 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:32:27.703 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:32:27.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:32:57.767 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:32:57.771 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:33:27.912 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:33:27.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:33:58.040 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:33:58.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:34:28.170 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:34:28.173 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:34:58.291 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:34:58.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:35:28.445 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:35:28.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:35:58.558 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:35:58.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:36:28.686 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:36:28.690 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:36:58.824 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:36:58.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:37:28.945 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:37:28.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:37:59.090 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:37:59.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:38:29.195 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:38:29.199 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:38:59.265 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:38:59.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:39:29.411 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:39:29.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:39:59.597 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:39:59.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:40:29.719 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:40:29.723 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:40:59.848 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:40:59.852 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:41:29.936 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:41:29.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:42:00.215 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:42:00.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:42:30.278 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:42:30.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:43:00.456 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:43:00.460 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:43:30.510 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:43:30.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:44:00.581 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:44:00.584 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:44:30.728 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:44:30.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:45:00.857 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:45:00.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:45:30.977 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:45:30.980 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:46:01.123 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:46:01.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:46:31.201 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:46:31.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:47:01.355 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:47:01.359 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:47:31.493 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:47:31.497 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:48:01.632 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:48:01.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:48:31.701 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:48:31.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:49:01.839 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:49:01.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:49:31.977 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:49:31.980 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:50:02.130 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:50:02.133 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:50:32.201 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:50:32.204 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:51:02.347 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:51:02.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:51:32.412 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:51:32.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:52:02.563 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:52:02.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:52:32.625 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:52:32.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:53:02.755 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:53:02.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:53:32.917 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:53:32.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:54:03.029 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:54:03.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:54:33.133 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:54:33.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:55:03.220 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:55:03.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:55:33.285 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:55:33.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:56:03.428 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:56:03.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:56:33.540 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:56:33.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:57:03.669 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:57:03.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:57:33.806 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:57:33.810 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:58:03.944 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:58:03.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:58:34.073 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:58:34.077 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:59:04.218 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:59:04.222 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:59:34.283 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T19:59:34.287 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:00:04.452 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:00:04.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:00:34.598 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:00:34.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:01:04.684 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:01:04.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:01:34.806 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:01:34.809 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:02:04.900 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:02:04.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:02:35.056 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:02:35.060 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:03:05.201 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:03:05.204 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:03:35.330 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:03:35.334 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:04:05.395 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:04:05.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:04:35.522 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:04:35.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:05:05.642 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:05:05.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:05:35.780 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:05:35.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:06:05.901 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:06:05.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:06:15.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:15.812 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:06:15.845 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_12_00_UTC 2024-03-18T20:06:15.845 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_13_00_UTC 2024-03-18T20:06:15.845 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_14_00_UTC 2024-03-18T20:06:15.845 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_15_00_UTC 2024-03-18T20:06:15.845 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_16_00_UTC 2024-03-18T20:06:15.845 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_17_00_UTC 2024-03-18T20:06:15.845 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_18_00_UTC 2024-03-18T20:06:15.846 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_19_00_UTC 2024-03-18T20:06:15.846 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_20_00_UTC 2024-03-18T20:06:15.846 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_21_00_UTC 2024-03-18T20:06:15.846 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_22_00_UTC 2024-03-18T20:06:15.846 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_23_00_UTC 2024-03-18T20:06:15.846 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_24_00_UTC 2024-03-18T20:06:15.846 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_25_00_UTC 2024-03-18T20:06:15.846 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_26_00_UTC 2024-03-18T20:06:15.846 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_27_00_UTC 2024-03-18T20:06:15.846 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_28_00_UTC 2024-03-18T20:06:15.847 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_29_00_UTC 2024-03-18T20:06:15.847 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_30_00_UTC 2024-03-18T20:06:15.847 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_31_00_UTC 2024-03-18T20:06:15.847 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_32_00_UTC 2024-03-18T20:06:15.847 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_33_00_UTC 2024-03-18T20:06:15.847 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_34_00_UTC 2024-03-18T20:06:15.847 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_35_00_UTC 2024-03-18T20:06:15.847 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_36_00_UTC 2024-03-18T20:06:15.847 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_37_00_UTC 2024-03-18T20:06:15.848 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_38_00_UTC 2024-03-18T20:06:15.848 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_39_00_UTC 2024-03-18T20:06:15.848 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_40_00_UTC 2024-03-18T20:06:15.848 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_41_00_UTC 2024-03-18T20:06:15.848 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_42_00_UTC 2024-03-18T20:06:15.848 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_43_00_UTC 2024-03-18T20:06:15.848 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_44_00_UTC 2024-03-18T20:06:15.848 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_45_00_UTC 2024-03-18T20:06:15.848 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_46_00_UTC 2024-03-18T20:06:15.849 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_47_00_UTC 2024-03-18T20:06:15.849 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_48_00_UTC 2024-03-18T20:06:15.849 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_49_00_UTC 2024-03-18T20:06:15.849 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_50_00_UTC 2024-03-18T20:06:15.849 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_51_00_UTC 2024-03-18T20:06:15.849 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_52_00_UTC 2024-03-18T20:06:15.849 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_53_00_UTC 2024-03-18T20:06:15.849 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_54_00_UTC 2024-03-18T20:06:15.849 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_55_00_UTC 2024-03-18T20:06:15.849 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_56_00_UTC 2024-03-18T20:06:15.849 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_57_00_UTC 2024-03-18T20:06:15.849 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_58_00_UTC 2024-03-18T20:06:15.850 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_59_00_UTC 2024-03-18T20:06:15.850 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_00_00_UTC 2024-03-18T20:06:15.850 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_01_00_UTC 2024-03-18T20:06:15.850 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_02_00_UTC 2024-03-18T20:06:15.850 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_03_00_UTC 2024-03-18T20:06:15.850 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_04_00_UTC 2024-03-18T20:06:15.850 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_05_00_UTC 2024-03-18T20:06:15.850 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_06_00_UTC 2024-03-18T20:06:15.852 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-18-19_12_00_UTC', 'scheduled-2024-03-18-19_13_00_UTC', 'scheduled-2024-03-18-19_14_00_UTC', 'scheduled-2024-03-18-19_15_00_UTC', 'scheduled-2024-03-18-19_16_00_UTC', 'scheduled-2024-03-18-19_17_00_UTC', 'scheduled-2024-03-18-19_18_00_UTC', 'scheduled-2024-03-18-19_19_00_UTC', 'scheduled-2024-03-18-19_20_00_UTC', 'scheduled-2024-03-18-19_21_00_UTC', 'scheduled-2024-03-18-19_22_00_UTC', 'scheduled-2024-03-18-19_23_00_UTC', 'scheduled-2024-03-18-19_24_00_UTC', 'scheduled-2024-03-18-19_25_00_UTC', 'scheduled-2024-03-18-19_26_00_UTC', 'scheduled-2024-03-18-19_27_00_UTC', 'scheduled-2024-03-18-19_28_00_UTC', 'scheduled-2024-03-18-19_29_00_UTC', 'scheduled-2024-03-18-19_30_00_UTC', 'scheduled-2024-03-18-19_31_00_UTC', 'scheduled-2024-03-18-19_32_00_UTC', 'scheduled-2024-03-18-19_33_00_UTC', 'scheduled-2024-03-18-19_34_00_UTC', 'scheduled-2024-03-18-19_35_00_UTC', 'scheduled-2024-03-18-19_36_00_UTC', 'scheduled-2024-03-18-19_37_00_UTC', 'scheduled-2024-03-18-19_38_00_UTC', 'scheduled-2024-03-18-19_39_00_UTC', 'scheduled-2024-03-18-19_40_00_UTC', 'scheduled-2024-03-18-19_41_00_UTC', 'scheduled-2024-03-18-19_42_00_UTC', 'scheduled-2024-03-18-19_43_00_UTC', 'scheduled-2024-03-18-19_44_00_UTC', 'scheduled-2024-03-18-19_45_00_UTC', 'scheduled-2024-03-18-19_46_00_UTC', 'scheduled-2024-03-18-19_47_00_UTC', 'scheduled-2024-03-18-19_48_00_UTC', 'scheduled-2024-03-18-19_49_00_UTC', 'scheduled-2024-03-18-19_50_00_UTC', 'scheduled-2024-03-18-19_51_00_UTC', 'scheduled-2024-03-18-19_52_00_UTC', 'scheduled-2024-03-18-19_53_00_UTC', 'scheduled-2024-03-18-19_54_00_UTC', 'scheduled-2024-03-18-19_55_00_UTC', 'scheduled-2024-03-18-19_56_00_UTC', 'scheduled-2024-03-18-19_57_00_UTC', 'scheduled-2024-03-18-19_58_00_UTC', 'scheduled-2024-03-18-19_59_00_UTC', 'scheduled-2024-03-18-20_00_00_UTC', 'scheduled-2024-03-18-20_01_00_UTC', 'scheduled-2024-03-18-20_02_00_UTC', 'scheduled-2024-03-18-20_03_00_UTC', 'scheduled-2024-03-18-20_04_00_UTC', 'scheduled-2024-03-18-20_05_00_UTC', 'scheduled-2024-03-18-20_06_00_UTC'] 2024-03-18T20:06:15.852 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1 2024-03-18T20:06:16.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.024+0000 7f5cf97ec700 1 Processor -- start 2024-03-18T20:06:16.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.024+0000 7f5cf97ec700 1 -- start start 2024-03-18T20:06:16.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.025+0000 7f5cf97ec700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5cf40d1ec0 0x7f5cf40b82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:16.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.025+0000 7f5cf97ec700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf40ce1e0 con 0x7f5cf415d7c0 2024-03-18T20:06:16.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.025+0000 7f5cf97ec700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf40ce4c0 con 0x7f5cf415d3e0 2024-03-18T20:06:16.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.025+0000 7f5cf97ec700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cf40cee60 con 0x7f5cf40d1ec0 2024-03-18T20:06:16.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.025+0000 7f5cf37fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5cf40d1ec0 0x7f5cf40b82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:16.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.025+0000 7f5cf27fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5cf415d7c0 0x7f5cf40d17b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33832/0 (socket says 172.21.15.1:33832) 2024-03-18T20:06:16.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.025+0000 7f5cf27fc700 1 -- 172.21.15.1:0/3858650601 learned_addr learned my addr 172.21.15.1:0/3858650601 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:16.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.025+0000 7f5cf37fe700 1 -- 172.21.15.1:0/3858650601 >> v1:172.21.15.165:6789/0 conn(0x7f5cf415d3e0 legacy=0x7f5cf40d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:16.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.025+0000 7f5cf37fe700 1 -- 172.21.15.1:0/3858650601 >> v1:172.21.15.1:6789/0 conn(0x7f5cf415d7c0 legacy=0x7f5cf40d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:16.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.025+0000 7f5cf37fe700 1 -- 172.21.15.1:0/3858650601 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf40ce7a0 con 0x7f5cf40d1ec0 2024-03-18T20:06:16.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.025+0000 7f5cf1ffb700 1 -- 172.21.15.1:0/3858650601 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115048949 0 0) 0x7f5cf40ce1e0 con 0x7f5cf415d7c0 2024-03-18T20:06:16.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.025+0000 7f5cf37fe700 1 --2- 172.21.15.1:0/3858650601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d1ec0 0x7f5cf40b82d0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f5cec009d90 tx=0x7f5cec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3ec2433b8dc739 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:16.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.025+0000 7f5cf1ffb700 1 -- 172.21.15.1:0/3858650601 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec00e070 con 0x7f5cf40d1ec0 2024-03-18T20:06:16.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.025+0000 7f5cf1ffb700 1 -- 172.21.15.1:0/3858650601 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cec00bb90 con 0x7f5cf40d1ec0 2024-03-18T20:06:16.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.025+0000 7f5cf1ffb700 1 -- 172.21.15.1:0/3858650601 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec003a20 con 0x7f5cf40d1ec0 2024-03-18T20:06:16.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.026+0000 7f5cf97ec700 1 -- 172.21.15.1:0/3858650601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d1ec0 msgr2=0x7f5cf40b82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:16.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.026+0000 7f5cf97ec700 1 --2- 172.21.15.1:0/3858650601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d1ec0 0x7f5cf40b82d0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f5cec009d90 tx=0x7f5cec00b330 comp rx=0 tx=0).stop 2024-03-18T20:06:16.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.026+0000 7f5cf97ec700 1 -- 172.21.15.1:0/3858650601 shutdown_connections 2024-03-18T20:06:16.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.026+0000 7f5cf97ec700 1 --2- 172.21.15.1:0/3858650601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d1ec0 0x7f5cf40b82d0 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:16.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.026+0000 7f5cf97ec700 1 -- 172.21.15.1:0/3858650601 >> 172.21.15.1:0/3858650601 conn(0x7f5cf409c540 msgr2=0x7f5cf40d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:16.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.026+0000 7f5cf97ec700 1 -- 172.21.15.1:0/3858650601 shutdown_connections 2024-03-18T20:06:16.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.026+0000 7f5cf97ec700 1 -- 172.21.15.1:0/3858650601 wait complete. 2024-03-18T20:06:16.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.027+0000 7f5cf97ec700 1 Processor -- start 2024-03-18T20:06:16.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.028+0000 7f5cf97ec700 1 -- start start 2024-03-18T20:06:16.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.028+0000 7f5cf97ec700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d1ec0 0x7f5cf4154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:16.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.028+0000 7f5cf97ec700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5cf415d3e0 0x7f5cf4160410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:16.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.028+0000 7f5cf97ec700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5cf40d0ae0 0x7f5cf40d0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:16.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.028+0000 7f5cf97ec700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf40d14b0 con 0x7f5cf40d1ec0 2024-03-18T20:06:16.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.028+0000 7f5cf97ec700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf415dcc0 con 0x7f5cf415d3e0 2024-03-18T20:06:16.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.028+0000 7f5cf97ec700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf415dfa0 con 0x7f5cf40d0ae0 2024-03-18T20:06:16.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.028+0000 7f5cf2ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d1ec0 0x7f5cf4154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:16.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.028+0000 7f5cf2ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d1ec0 0x7f5cf4154aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53036/0 (socket says 172.21.15.1:53036) 2024-03-18T20:06:16.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.028+0000 7f5cf2ffd700 1 -- 172.21.15.1:0/2877300899 learned_addr learned my addr 172.21.15.1:0/2877300899 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:16.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.029+0000 7f5cf37fe700 1 --2- 172.21.15.1:0/2877300899 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5cf40d0ae0 0x7f5cf40d0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:16.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.029+0000 7f5cf27fc700 1 --2- 172.21.15.1:0/2877300899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5cf415d3e0 0x7f5cf4160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:16.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.029+0000 7f5cf2ffd700 1 -- 172.21.15.1:0/2877300899 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5cf40d0ae0 msgr2=0x7f5cf40d0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:16.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.029+0000 7f5cf2ffd700 1 --2- 172.21.15.1:0/2877300899 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5cf40d0ae0 0x7f5cf40d0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:16.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.029+0000 7f5cf2ffd700 1 -- 172.21.15.1:0/2877300899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5cf415d3e0 msgr2=0x7f5cf4160410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:16.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.029+0000 7f5cf2ffd700 1 --2- 172.21.15.1:0/2877300899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5cf415d3e0 0x7f5cf4160410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:16.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.029+0000 7f5cf2ffd700 1 -- 172.21.15.1:0/2877300899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cec009a70 con 0x7f5cf40d1ec0 2024-03-18T20:06:16.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.029+0000 7f5cf27fc700 1 --2- 172.21.15.1:0/2877300899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5cf415d3e0 0x7f5cf4160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:16.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.029+0000 7f5cf2ffd700 1 --2- 172.21.15.1:0/2877300899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d1ec0 0x7f5cf4154aa0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f5ce800c4d0 tx=0x7f5ce800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:16.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.029+0000 7f5cdb7fe700 1 -- 172.21.15.1:0/2877300899 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8011790 con 0x7f5cf40d1ec0 2024-03-18T20:06:16.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.029+0000 7f5cdb7fe700 1 -- 172.21.15.1:0/2877300899 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce80118f0 con 0x7f5cf40d1ec0 2024-03-18T20:06:16.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.029+0000 7f5cf97ec700 1 -- 172.21.15.1:0/2877300899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cf415e280 con 0x7f5cf40d1ec0 2024-03-18T20:06:16.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.029+0000 7f5cdb7fe700 1 -- 172.21.15.1:0/2877300899 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8011ca0 con 0x7f5cf40d1ec0 2024-03-18T20:06:16.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.029+0000 7f5cf97ec700 1 -- 172.21.15.1:0/2877300899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cf415e840 con 0x7f5cf40d1ec0 2024-03-18T20:06:16.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.030+0000 7f5cf97ec700 1 -- 172.21.15.1:0/2877300899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cf4073350 con 0x7f5cf40d1ec0 2024-03-18T20:06:16.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.030+0000 7f5cdb7fe700 1 -- 172.21.15.1:0/2877300899 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 90) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f5ce801c460 con 0x7f5cf40d1ec0 2024-03-18T20:06:16.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.031+0000 7f5cdb7fe700 1 --2- 172.21.15.1:0/2877300899 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ce006ae30 0x7f5ce006d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:16.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.031+0000 7f5cdb7fe700 1 -- 172.21.15.1:0/2877300899 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f5ce80896e0 con 0x7f5cf40d1ec0 2024-03-18T20:06:16.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.031+0000 7f5cf27fc700 1 --2- 172.21.15.1:0/2877300899 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ce006ae30 0x7f5ce006d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:16.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.031+0000 7f5cf27fc700 1 --2- 172.21.15.1:0/2877300899 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ce006ae30 0x7f5ce006d2f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5cf41f6520 tx=0x7f5ce4009040 comp rx=0 tx=0).ready entity=mgr.9561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:16.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.036+0000 7f5cdb7fe700 1 -- 172.21.15.1:0/2877300899 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5ce80543c0 con 0x7f5cf40d1ec0 2024-03-18T20:06:16.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.264+0000 7f5cf97ec700 1 -- 172.21.15.1:0/2877300899 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5cf40b7030 con 0x7f5ce006ae30 2024-03-18T20:06:16.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.273+0000 7f5cdb7fe700 1 -- 172.21.15.1:0/2877300899 <== mgr.9561 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+373 (secure 0 0 0) 0x7f5ce4001cc0 con 0x7f5ce006ae30 2024-03-18T20:06:16.278 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:16.278 INFO:teuthology.orchestra.run.smithi001.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T19:10:20", "first": "2024-03-18T19:11:00", "last": "2024-03-18T20:06:00", "last_pruned": "2024-03-18T20:06:00", "created_count": 56, "pruned_count": 1, "active": true}] 2024-03-18T20:06:16.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.275+0000 7f5cf97ec700 1 -- 172.21.15.1:0/2877300899 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ce006ae30 msgr2=0x7f5ce006d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:16.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.275+0000 7f5cf97ec700 1 --2- 172.21.15.1:0/2877300899 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ce006ae30 0x7f5ce006d2f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5cf41f6520 tx=0x7f5ce4009040 comp rx=0 tx=0).stop 2024-03-18T20:06:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.275+0000 7f5cf97ec700 1 -- 172.21.15.1:0/2877300899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d1ec0 msgr2=0x7f5cf4154aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.275+0000 7f5cf97ec700 1 --2- 172.21.15.1:0/2877300899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d1ec0 0x7f5cf4154aa0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f5ce800c4d0 tx=0x7f5ce800c890 comp rx=0 tx=0).stop 2024-03-18T20:06:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.275+0000 7f5cf97ec700 1 -- 172.21.15.1:0/2877300899 shutdown_connections 2024-03-18T20:06:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.275+0000 7f5cf97ec700 1 --2- 172.21.15.1:0/2877300899 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5ce006ae30 0x7f5ce006d2f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.275+0000 7f5cf97ec700 1 --2- 172.21.15.1:0/2877300899 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5cf40d0ae0 0x7f5cf40d0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.275+0000 7f5cf97ec700 1 --2- 172.21.15.1:0/2877300899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cf40d1ec0 0x7f5cf4154aa0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.275+0000 7f5cf97ec700 1 --2- 172.21.15.1:0/2877300899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5cf415d3e0 0x7f5cf4160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.275+0000 7f5cf97ec700 1 -- 172.21.15.1:0/2877300899 >> 172.21.15.1:0/2877300899 conn(0x7f5cf409c540 msgr2=0x7f5cf40d17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.275+0000 7f5cf97ec700 1 -- 172.21.15.1:0/2877300899 shutdown_connections 2024-03-18T20:06:16.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.275+0000 7f5cf97ec700 1 -- 172.21.15.1:0/2877300899 wait complete. 2024-03-18T20:06:16.292 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T19:10:20", "first": "2024-03-18T19:11:00", "last": "2024-03-18T20:06:00", "last_pruned": "2024-03-18T20:06:00", "created_count": 56, "pruned_count": 1, "active": true}] 2024-03-18T20:06:16.292 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1', 'rel_path': '/snap_test_dir1', 'schedule': '1m', 'retention': {'n': 55}, 'start': '2024-03-18T00:00:00', 'created': '2024-03-18T19:10:20', 'first': '2024-03-18T19:11:00', 'last': '2024-03-18T20:06:00', 'last_pruned': '2024-03-18T20:06:00', 'created_count': 56, 'pruned_count': 1, 'active': True} 2024-03-18T20:06:16.293 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 55 2024-03-18T20:06:16.293 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 56 2024-03-18T20:06:16.293 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1 2024-03-18T20:06:16.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.415+0000 7f9afed61700 1 Processor -- start 2024-03-18T20:06:16.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.415+0000 7f9afed61700 1 -- start start 2024-03-18T20:06:16.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.415+0000 7f9afed61700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9af806f180 0x7f9af8161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:16.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.415+0000 7f9afed61700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9af806a750 0x7f9af806aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:16.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.415+0000 7f9afed61700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af806b0e0 con 0x7f9af806eda0 2024-03-18T20:06:16.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.415+0000 7f9afed61700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9af80cb320 con 0x7f9af806f180 2024-03-18T20:06:16.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.415+0000 7f9afed61700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af80ce010 con 0x7f9af806a750 2024-03-18T20:06:16.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.416+0000 7f9af7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9af806f180 0x7f9af8161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:16.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.416+0000 7f9af7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9af806f180 0x7f9af8161c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53050/0 (socket says 172.21.15.1:53050) 2024-03-18T20:06:16.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.416+0000 7f9af7fff700 1 -- 172.21.15.1:0/2977938190 learned_addr learned my addr 172.21.15.1:0/2977938190 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:16.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.416+0000 7f9afd2fe700 1 --2- 172.21.15.1:0/2977938190 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9af806a750 0x7f9af806aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:16.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.416+0000 7f9af7fff700 1 -- 172.21.15.1:0/2977938190 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9af806a750 msgr2=0x7f9af806aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:16.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.416+0000 7f9af7fff700 1 --2- 172.21.15.1:0/2977938190 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9af806a750 0x7f9af806aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:16.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.416+0000 7f9af7fff700 1 -- 172.21.15.1:0/2977938190 >> v1:172.21.15.165:6789/0 conn(0x7f9af806eda0 legacy=0x7f9af80ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:16.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.416+0000 7f9af7fff700 1 -- 172.21.15.1:0/2977938190 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af80cad60 con 0x7f9af806f180 2024-03-18T20:06:16.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.416+0000 7f9afd2fe700 1 --2- 172.21.15.1:0/2977938190 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9af806a750 0x7f9af806aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:16.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.416+0000 7f9af7fff700 1 --2- 172.21.15.1:0/2977938190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9af806f180 0x7f9af8161c60 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40098f0 tx=0x7f9ae4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e07243df59c673d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:16.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.416+0000 7f9af77fe700 1 -- 172.21.15.1:0/2977938190 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4004230 con 0x7f9af806f180 2024-03-18T20:06:16.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.416+0000 7f9af77fe700 1 -- 172.21.15.1:0/2977938190 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae4004390 con 0x7f9af806f180 2024-03-18T20:06:16.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.416+0000 7f9af77fe700 1 -- 172.21.15.1:0/2977938190 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae40036a0 con 0x7f9af806f180 2024-03-18T20:06:16.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.417+0000 7f9afed61700 1 -- 172.21.15.1:0/2977938190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9af806f180 msgr2=0x7f9af8161c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:16.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.417+0000 7f9afed61700 1 --2- 172.21.15.1:0/2977938190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9af806f180 0x7f9af8161c60 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40098f0 tx=0x7f9ae4009c00 comp rx=0 tx=0).stop 2024-03-18T20:06:16.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.417+0000 7f9afed61700 1 -- 172.21.15.1:0/2977938190 shutdown_connections 2024-03-18T20:06:16.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.417+0000 7f9afed61700 1 --2- 172.21.15.1:0/2977938190 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9af806a750 0x7f9af806aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:16.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.417+0000 7f9afed61700 1 --2- 172.21.15.1:0/2977938190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9af806f180 0x7f9af8161c60 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:16.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.417+0000 7f9afed61700 1 -- 172.21.15.1:0/2977938190 >> 172.21.15.1:0/2977938190 conn(0x7f9af80cff30 msgr2=0x7f9af80d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:16.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.417+0000 7f9afed61700 1 -- 172.21.15.1:0/2977938190 shutdown_connections 2024-03-18T20:06:16.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.417+0000 7f9afed61700 1 -- 172.21.15.1:0/2977938190 wait complete. 2024-03-18T20:06:16.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.418+0000 7f9afed61700 1 Processor -- start 2024-03-18T20:06:16.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.418+0000 7f9afed61700 1 -- start start 2024-03-18T20:06:16.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.419+0000 7f9afed61700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9af806a750 0x7f9af80c3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:16.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.419+0000 7f9afed61700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9af806eda0 0x7f9af815c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:16.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.419+0000 7f9afed61700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9af806f180 0x7f9af80cf260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:16.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.419+0000 7f9afed61700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af81f7080 con 0x7f9af806eda0 2024-03-18T20:06:16.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.419+0000 7f9afed61700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af81f6500 con 0x7f9af806a750 2024-03-18T20:06:16.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.419+0000 7f9afed61700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af81f67e0 con 0x7f9af806f180 2024-03-18T20:06:16.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.419+0000 7f9af7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9af806eda0 0x7f9af815c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:16.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.419+0000 7f9af7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9af806eda0 0x7f9af815c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53056/0 (socket says 172.21.15.1:53056) 2024-03-18T20:06:16.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.419+0000 7f9af7fff700 1 -- 172.21.15.1:0/2668477836 learned_addr learned my addr 172.21.15.1:0/2668477836 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:16.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.419+0000 7f9afd2fe700 1 --2- 172.21.15.1:0/2668477836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9af806f180 0x7f9af80cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:16.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.419+0000 7f9afcafd700 1 --2- 172.21.15.1:0/2668477836 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9af806a750 0x7f9af80c3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:16.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.419+0000 7f9af7fff700 1 -- 172.21.15.1:0/2668477836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9af806f180 msgr2=0x7f9af80cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.419+0000 7f9af7fff700 1 --2- 172.21.15.1:0/2668477836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9af806f180 0x7f9af80cf260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.419+0000 7f9af7fff700 1 -- 172.21.15.1:0/2668477836 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9af806a750 msgr2=0x7f9af80c3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.419+0000 7f9af7fff700 1 --2- 172.21.15.1:0/2668477836 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9af806a750 0x7f9af80c3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.419+0000 7f9af7fff700 1 -- 172.21.15.1:0/2668477836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae4009550 con 0x7f9af806eda0 2024-03-18T20:06:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.420+0000 7f9af7fff700 1 --2- 172.21.15.1:0/2668477836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9af806eda0 0x7f9af815c260 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4009130 tx=0x7f9ae4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.420+0000 7f9af57fa700 1 -- 172.21.15.1:0/2668477836 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae400b660 con 0x7f9af806eda0 2024-03-18T20:06:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.420+0000 7f9af57fa700 1 -- 172.21.15.1:0/2668477836 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae400b7c0 con 0x7f9af806eda0 2024-03-18T20:06:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.420+0000 7f9afed61700 1 -- 172.21.15.1:0/2668477836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af81f6ac0 con 0x7f9af806eda0 2024-03-18T20:06:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.420+0000 7f9af57fa700 1 -- 172.21.15.1:0/2668477836 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae400bb70 con 0x7f9af806eda0 2024-03-18T20:06:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.420+0000 7f9afed61700 1 -- 172.21.15.1:0/2668477836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af80c3780 con 0x7f9af806eda0 2024-03-18T20:06:16.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.421+0000 7f9afed61700 1 -- 172.21.15.1:0/2668477836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af80676a0 con 0x7f9af806eda0 2024-03-18T20:06:16.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.421+0000 7f9af57fa700 1 -- 172.21.15.1:0/2668477836 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 90) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f9ae400bcd0 con 0x7f9af806eda0 2024-03-18T20:06:16.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.422+0000 7f9af57fa700 1 --2- 172.21.15.1:0/2668477836 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9adc06ae00 0x7f9adc06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:16.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.422+0000 7f9af57fa700 1 -- 172.21.15.1:0/2668477836 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f9ae408aea0 con 0x7f9af806eda0 2024-03-18T20:06:16.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.422+0000 7f9afcafd700 1 --2- 172.21.15.1:0/2668477836 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9adc06ae00 0x7f9adc06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:16.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.422+0000 7f9afcafd700 1 --2- 172.21.15.1:0/2668477836 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9adc06ae00 0x7f9adc06d2c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9aec007fd0 tx=0x7f9aec007ec0 comp rx=0 tx=0).ready entity=mgr.9561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:16.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.426+0000 7f9af57fa700 1 -- 172.21.15.1:0/2668477836 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9ae4055d20 con 0x7f9af806eda0 2024-03-18T20:06:16.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.657+0000 7f9afed61700 1 -- 172.21.15.1:0/2668477836 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9af80ce500 con 0x7f9adc06ae00 2024-03-18T20:06:16.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.734+0000 7f9af57fa700 1 -- 172.21.15.1:0/2668477836 <== mgr.9561 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+42 (secure 0 0 0) 0x7f9aec003a90 con 0x7f9adc06ae00 2024-03-18T20:06:16.739 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule removed for path /snap_test_dir1 2024-03-18T20:06:16.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.736+0000 7f9afed61700 1 -- 172.21.15.1:0/2668477836 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9adc06ae00 msgr2=0x7f9adc06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:16.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.736+0000 7f9afed61700 1 --2- 172.21.15.1:0/2668477836 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9adc06ae00 0x7f9adc06d2c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9aec007fd0 tx=0x7f9aec007ec0 comp rx=0 tx=0).stop 2024-03-18T20:06:16.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.736+0000 7f9afed61700 1 -- 172.21.15.1:0/2668477836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9af806eda0 msgr2=0x7f9af815c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:16.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.736+0000 7f9afed61700 1 --2- 172.21.15.1:0/2668477836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9af806eda0 0x7f9af815c260 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4009130 tx=0x7f9ae4004750 comp rx=0 tx=0).stop 2024-03-18T20:06:16.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.736+0000 7f9afed61700 1 -- 172.21.15.1:0/2668477836 shutdown_connections 2024-03-18T20:06:16.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.736+0000 7f9afed61700 1 --2- 172.21.15.1:0/2668477836 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9adc06ae00 0x7f9adc06d2c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:16.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.736+0000 7f9afed61700 1 --2- 172.21.15.1:0/2668477836 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9af806f180 0x7f9af80cf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:16.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.736+0000 7f9afed61700 1 --2- 172.21.15.1:0/2668477836 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9af806a750 0x7f9af80c3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:16.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.736+0000 7f9afed61700 1 --2- 172.21.15.1:0/2668477836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9af806eda0 0x7f9af815c260 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:16.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.736+0000 7f9afed61700 1 -- 172.21.15.1:0/2668477836 >> 172.21.15.1:0/2668477836 conn(0x7f9af80cff30 msgr2=0x7f9af80afa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:16.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.736+0000 7f9afed61700 1 -- 172.21.15.1:0/2668477836 shutdown_connections 2024-03-18T20:06:16.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:16.736+0000 7f9afed61700 1 -- 172.21.15.1:0/2668477836 wait complete. 2024-03-18T20:06:16.753 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1 2024-03-18T20:06:16.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:16.754 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:06:16.778 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_12_00_UTC 2024-03-18T20:06:16.778 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_13_00_UTC 2024-03-18T20:06:16.779 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_14_00_UTC 2024-03-18T20:06:16.779 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_15_00_UTC 2024-03-18T20:06:16.779 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_16_00_UTC 2024-03-18T20:06:16.779 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_17_00_UTC 2024-03-18T20:06:16.779 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_18_00_UTC 2024-03-18T20:06:16.779 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_19_00_UTC 2024-03-18T20:06:16.779 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_20_00_UTC 2024-03-18T20:06:16.779 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_21_00_UTC 2024-03-18T20:06:16.779 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_22_00_UTC 2024-03-18T20:06:16.779 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_23_00_UTC 2024-03-18T20:06:16.779 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_24_00_UTC 2024-03-18T20:06:16.780 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_25_00_UTC 2024-03-18T20:06:16.780 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_26_00_UTC 2024-03-18T20:06:16.780 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_27_00_UTC 2024-03-18T20:06:16.780 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_28_00_UTC 2024-03-18T20:06:16.780 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_29_00_UTC 2024-03-18T20:06:16.780 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_30_00_UTC 2024-03-18T20:06:16.780 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_31_00_UTC 2024-03-18T20:06:16.780 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_32_00_UTC 2024-03-18T20:06:16.780 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_33_00_UTC 2024-03-18T20:06:16.780 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_34_00_UTC 2024-03-18T20:06:16.780 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_35_00_UTC 2024-03-18T20:06:16.780 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_36_00_UTC 2024-03-18T20:06:16.781 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_37_00_UTC 2024-03-18T20:06:16.781 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_38_00_UTC 2024-03-18T20:06:16.781 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_39_00_UTC 2024-03-18T20:06:16.781 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_40_00_UTC 2024-03-18T20:06:16.781 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_41_00_UTC 2024-03-18T20:06:16.781 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_42_00_UTC 2024-03-18T20:06:16.781 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_43_00_UTC 2024-03-18T20:06:16.781 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_44_00_UTC 2024-03-18T20:06:16.781 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_45_00_UTC 2024-03-18T20:06:16.781 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_46_00_UTC 2024-03-18T20:06:16.781 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_47_00_UTC 2024-03-18T20:06:16.781 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_48_00_UTC 2024-03-18T20:06:16.781 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_49_00_UTC 2024-03-18T20:06:16.782 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_50_00_UTC 2024-03-18T20:06:16.782 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_51_00_UTC 2024-03-18T20:06:16.782 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_52_00_UTC 2024-03-18T20:06:16.782 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_53_00_UTC 2024-03-18T20:06:16.782 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_54_00_UTC 2024-03-18T20:06:16.782 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_55_00_UTC 2024-03-18T20:06:16.782 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_56_00_UTC 2024-03-18T20:06:16.782 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_57_00_UTC 2024-03-18T20:06:16.782 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_58_00_UTC 2024-03-18T20:06:16.782 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-19_59_00_UTC 2024-03-18T20:06:16.782 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_00_00_UTC 2024-03-18T20:06:16.783 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_01_00_UTC 2024-03-18T20:06:16.783 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_02_00_UTC 2024-03-18T20:06:16.783 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_03_00_UTC 2024-03-18T20:06:16.783 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_04_00_UTC 2024-03-18T20:06:16.783 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_05_00_UTC 2024-03-18T20:06:16.783 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_06_00_UTC 2024-03-18T20:06:16.783 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_12_00_UTC 2024-03-18T20:06:16.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:16.783 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_12_00_UTC) 2024-03-18T20:06:16.846 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_13_00_UTC 2024-03-18T20:06:16.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:16.846 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_13_00_UTC) 2024-03-18T20:06:16.912 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_14_00_UTC 2024-03-18T20:06:16.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:16.913 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_14_00_UTC) 2024-03-18T20:06:16.978 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_15_00_UTC 2024-03-18T20:06:16.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:16.978 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_15_00_UTC) 2024-03-18T20:06:17.045 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_16_00_UTC 2024-03-18T20:06:17.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:17.045 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_16_00_UTC) 2024-03-18T20:06:17.111 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_17_00_UTC 2024-03-18T20:06:17.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:17.111 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_17_00_UTC) 2024-03-18T20:06:17.177 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_18_00_UTC 2024-03-18T20:06:17.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:17.177 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_18_00_UTC) 2024-03-18T20:06:17.243 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_19_00_UTC 2024-03-18T20:06:17.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:17.244 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_19_00_UTC) 2024-03-18T20:06:17.309 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_20_00_UTC 2024-03-18T20:06:17.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:17.310 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_20_00_UTC) 2024-03-18T20:06:17.375 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_21_00_UTC 2024-03-18T20:06:17.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:17.375 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_21_00_UTC) 2024-03-18T20:06:17.441 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_22_00_UTC 2024-03-18T20:06:17.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:17.441 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_22_00_UTC) 2024-03-18T20:06:17.507 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_23_00_UTC 2024-03-18T20:06:17.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:17.507 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_23_00_UTC) 2024-03-18T20:06:17.573 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_24_00_UTC 2024-03-18T20:06:17.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:17.573 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_24_00_UTC) 2024-03-18T20:06:17.639 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_25_00_UTC 2024-03-18T20:06:17.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:17.639 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_25_00_UTC) 2024-03-18T20:06:17.705 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_26_00_UTC 2024-03-18T20:06:17.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:17.706 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_26_00_UTC) 2024-03-18T20:06:17.771 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_27_00_UTC 2024-03-18T20:06:17.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:17.771 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_27_00_UTC) 2024-03-18T20:06:17.837 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_28_00_UTC 2024-03-18T20:06:17.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:17.837 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_28_00_UTC) 2024-03-18T20:06:17.903 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_29_00_UTC 2024-03-18T20:06:17.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:17.904 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_29_00_UTC) 2024-03-18T20:06:17.969 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_30_00_UTC 2024-03-18T20:06:17.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:17.970 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_30_00_UTC) 2024-03-18T20:06:18.036 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_31_00_UTC 2024-03-18T20:06:18.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:18.036 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_31_00_UTC) 2024-03-18T20:06:18.102 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_32_00_UTC 2024-03-18T20:06:18.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:18.102 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_32_00_UTC) 2024-03-18T20:06:18.169 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_33_00_UTC 2024-03-18T20:06:18.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:18.169 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_33_00_UTC) 2024-03-18T20:06:18.237 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_34_00_UTC 2024-03-18T20:06:18.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:18.237 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_34_00_UTC) 2024-03-18T20:06:18.304 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_35_00_UTC 2024-03-18T20:06:18.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:18.304 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_35_00_UTC) 2024-03-18T20:06:18.372 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_36_00_UTC 2024-03-18T20:06:18.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:18.372 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_36_00_UTC) 2024-03-18T20:06:18.438 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_37_00_UTC 2024-03-18T20:06:18.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:18.439 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_37_00_UTC) 2024-03-18T20:06:18.505 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_38_00_UTC 2024-03-18T20:06:18.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:18.505 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_38_00_UTC) 2024-03-18T20:06:18.570 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_39_00_UTC 2024-03-18T20:06:18.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:18.571 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_39_00_UTC) 2024-03-18T20:06:18.637 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_40_00_UTC 2024-03-18T20:06:18.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:18.637 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_40_00_UTC) 2024-03-18T20:06:18.704 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_41_00_UTC 2024-03-18T20:06:18.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:18.705 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_41_00_UTC) 2024-03-18T20:06:18.770 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_42_00_UTC 2024-03-18T20:06:18.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:18.771 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_42_00_UTC) 2024-03-18T20:06:18.836 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_43_00_UTC 2024-03-18T20:06:18.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:18.836 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_43_00_UTC) 2024-03-18T20:06:18.903 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_44_00_UTC 2024-03-18T20:06:18.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:18.903 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_44_00_UTC) 2024-03-18T20:06:18.970 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_45_00_UTC 2024-03-18T20:06:18.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:18.971 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_45_00_UTC) 2024-03-18T20:06:19.036 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_46_00_UTC 2024-03-18T20:06:19.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:19.036 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_46_00_UTC) 2024-03-18T20:06:19.100 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_47_00_UTC 2024-03-18T20:06:19.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:19.101 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_47_00_UTC) 2024-03-18T20:06:19.166 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_48_00_UTC 2024-03-18T20:06:19.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:19.166 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_48_00_UTC) 2024-03-18T20:06:19.232 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_49_00_UTC 2024-03-18T20:06:19.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:19.232 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_49_00_UTC) 2024-03-18T20:06:19.298 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_50_00_UTC 2024-03-18T20:06:19.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:19.299 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_50_00_UTC) 2024-03-18T20:06:19.364 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_51_00_UTC 2024-03-18T20:06:19.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:19.364 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_51_00_UTC) 2024-03-18T20:06:19.430 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_52_00_UTC 2024-03-18T20:06:19.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:19.430 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_52_00_UTC) 2024-03-18T20:06:19.495 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_53_00_UTC 2024-03-18T20:06:19.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:19.496 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_53_00_UTC) 2024-03-18T20:06:19.560 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_54_00_UTC 2024-03-18T20:06:19.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:19.561 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_54_00_UTC) 2024-03-18T20:06:19.627 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_55_00_UTC 2024-03-18T20:06:19.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:19.627 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_55_00_UTC) 2024-03-18T20:06:19.691 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_56_00_UTC 2024-03-18T20:06:19.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:19.692 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_56_00_UTC) 2024-03-18T20:06:19.756 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_57_00_UTC 2024-03-18T20:06:19.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:19.757 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_57_00_UTC) 2024-03-18T20:06:19.822 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_58_00_UTC 2024-03-18T20:06:19.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:19.822 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_58_00_UTC) 2024-03-18T20:06:19.939 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-19_59_00_UTC 2024-03-18T20:06:19.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:19.940 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-19_59_00_UTC) 2024-03-18T20:06:19.999 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-20_00_00_UTC 2024-03-18T20:06:19.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:19.999 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-20_00_00_UTC) 2024-03-18T20:06:20.030 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-20_01_00_UTC 2024-03-18T20:06:20.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:20.031 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-20_01_00_UTC) 2024-03-18T20:06:20.095 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-20_02_00_UTC 2024-03-18T20:06:20.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:20.096 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-20_02_00_UTC) 2024-03-18T20:06:20.173 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-20_03_00_UTC 2024-03-18T20:06:20.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:20.173 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-20_03_00_UTC) 2024-03-18T20:06:20.273 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-20_04_00_UTC 2024-03-18T20:06:20.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:20.273 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-20_04_00_UTC) 2024-03-18T20:06:20.326 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-20_05_00_UTC 2024-03-18T20:06:20.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:20.327 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-20_05_00_UTC) 2024-03-18T20:06:20.357 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-20_06_00_UTC 2024-03-18T20:06:20.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:20.357 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-20_06_00_UTC) 2024-03-18T20:06:20.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:20.422 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-03-18T20:06:20.484 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-18T20:06:20.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.646+0000 7f99f9ec7700 1 Processor -- start 2024-03-18T20:06:20.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.647+0000 7f99f9ec7700 1 -- start start 2024-03-18T20:06:20.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.647+0000 7f99f9ec7700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f99f406ca80 0x7f99f415d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.647+0000 7f99f9ec7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f99f409a150 0x7f99f40cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.647+0000 7f99f9ec7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f40b3b00 con 0x7f99f415d5b0 2024-03-18T20:06:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.647+0000 7f99f9ec7700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f40cbbb0 con 0x7f99f406ca80 2024-03-18T20:06:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.647+0000 7f99f9ec7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f40cf7f0 con 0x7f99f409a150 2024-03-18T20:06:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.647+0000 7f99f37fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f99f415d5b0 0x7f99f4099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33844/0 (socket says 172.21.15.1:33844) 2024-03-18T20:06:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.647+0000 7f99f37fe700 1 -- 172.21.15.1:0/1157809199 learned_addr learned my addr 172.21.15.1:0/1157809199 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.648+0000 7f99f3fff700 1 --2- 172.21.15.1:0/1157809199 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f99f409a150 0x7f99f40cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.648+0000 7f99f2ffd700 1 --2- 172.21.15.1:0/1157809199 >> v2:172.21.15.165:3300/0 conn(0x7f99f406ca80 0x7f99f415d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.648+0000 7f99f27fc700 1 -- 172.21.15.1:0/1157809199 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281233298 0 0) 0x7f99f40b3b00 con 0x7f99f415d5b0 2024-03-18T20:06:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.648+0000 7f99f27fc700 1 -- 172.21.15.1:0/1157809199 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99e00035b0 con 0x7f99f415d5b0 2024-03-18T20:06:20.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.648+0000 7f99f3fff700 1 -- 172.21.15.1:0/1157809199 >> v2:172.21.15.165:3300/0 conn(0x7f99f406ca80 msgr2=0x7f99f415d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.648+0000 7f99f3fff700 1 --2- 172.21.15.1:0/1157809199 >> v2:172.21.15.165:3300/0 conn(0x7f99f406ca80 0x7f99f415d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.648+0000 7f99f3fff700 1 -- 172.21.15.1:0/1157809199 >> v1:172.21.15.1:6789/0 conn(0x7f99f415d5b0 legacy=0x7f99f4099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.648+0000 7f99f27fc700 1 -- 172.21.15.1:0/1157809199 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1241459432 0 0) 0x7f99e00035b0 con 0x7f99f415d5b0 2024-03-18T20:06:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.648+0000 7f99f3fff700 1 -- 172.21.15.1:0/1157809199 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f40d04d0 con 0x7f99f409a150 2024-03-18T20:06:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.648+0000 7f99f2ffd700 1 --2- 172.21.15.1:0/1157809199 >> v2:172.21.15.165:3300/0 conn(0x7f99f406ca80 0x7f99f415d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.648+0000 7f99f3fff700 1 --2- 172.21.15.1:0/1157809199 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f99f409a150 0x7f99f40cbfd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f99ec009d90 tx=0x7f99ec00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21e79362cd5ca80c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.648+0000 7f99f27fc700 1 -- 172.21.15.1:0/1157809199 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec00e070 con 0x7f99f409a150 2024-03-18T20:06:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.648+0000 7f99f27fc700 1 -- 172.21.15.1:0/1157809199 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99ec003a20 con 0x7f99f409a150 2024-03-18T20:06:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.648+0000 7f99f27fc700 1 -- 172.21.15.1:0/1157809199 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99dc001670 con 0x7f99f409a150 2024-03-18T20:06:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.649+0000 7f99f9ec7700 1 -- 172.21.15.1:0/1157809199 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f99f409a150 msgr2=0x7f99f40cbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.649+0000 7f99f9ec7700 1 --2- 172.21.15.1:0/1157809199 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f99f409a150 0x7f99f40cbfd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f99ec009d90 tx=0x7f99ec00b330 comp rx=0 tx=0).stop 2024-03-18T20:06:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.649+0000 7f99f9ec7700 1 -- 172.21.15.1:0/1157809199 shutdown_connections 2024-03-18T20:06:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.649+0000 7f99f9ec7700 1 --2- 172.21.15.1:0/1157809199 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f99f409a150 0x7f99f40cbfd0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.649+0000 7f99f9ec7700 1 --2- 172.21.15.1:0/1157809199 >> v2:172.21.15.165:3300/0 conn(0x7f99f406ca80 0x7f99f415d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:20.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.649+0000 7f99f9ec7700 1 -- 172.21.15.1:0/1157809199 >> 172.21.15.1:0/1157809199 conn(0x7f99f40b2bd0 msgr2=0x7f99f40b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:20.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.649+0000 7f99f9ec7700 1 -- 172.21.15.1:0/1157809199 shutdown_connections 2024-03-18T20:06:20.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.649+0000 7f99f9ec7700 1 -- 172.21.15.1:0/1157809199 wait complete. 2024-03-18T20:06:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.650+0000 7f99f9ec7700 1 Processor -- start 2024-03-18T20:06:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.650+0000 7f99f9ec7700 1 -- start start 2024-03-18T20:06:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.651+0000 7f99f9ec7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f99f406ca80 0x7f99f40b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.651+0000 7f99f9ec7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99f409a150 0x7f99f414e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.651+0000 7f99f9ec7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f99f415d5b0 0x7f99f40b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.651+0000 7f99f9ec7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f40b4840 con 0x7f99f409a150 2024-03-18T20:06:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.651+0000 7f99f9ec7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f40b6040 con 0x7f99f406ca80 2024-03-18T20:06:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.651+0000 7f99f9ec7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f415b970 con 0x7f99f415d5b0 2024-03-18T20:06:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.651+0000 7f99f2ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99f409a150 0x7f99f414e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.651+0000 7f99f2ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99f409a150 0x7f99f414e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53072/0 (socket says 172.21.15.1:53072) 2024-03-18T20:06:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.651+0000 7f99f2ffd700 1 -- 172.21.15.1:0/2780379563 learned_addr learned my addr 172.21.15.1:0/2780379563 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.651+0000 7f99f37fe700 1 --2- 172.21.15.1:0/2780379563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f99f406ca80 0x7f99f40b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.651+0000 7f99f3fff700 1 --2- 172.21.15.1:0/2780379563 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f99f415d5b0 0x7f99f40b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.651+0000 7f99f2ffd700 1 -- 172.21.15.1:0/2780379563 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f99f415d5b0 msgr2=0x7f99f40b54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.651+0000 7f99f2ffd700 1 --2- 172.21.15.1:0/2780379563 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f99f415d5b0 0x7f99f40b54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.652+0000 7f99f2ffd700 1 -- 172.21.15.1:0/2780379563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f99f406ca80 msgr2=0x7f99f40b4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.652+0000 7f99f2ffd700 1 --2- 172.21.15.1:0/2780379563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f99f406ca80 0x7f99f40b4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.652+0000 7f99f2ffd700 1 -- 172.21.15.1:0/2780379563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e40094a0 con 0x7f99f409a150 2024-03-18T20:06:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.652+0000 7f99f2ffd700 1 --2- 172.21.15.1:0/2780379563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99f409a150 0x7f99f414e420 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f99e400ebb0 tx=0x7f99e400eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.652+0000 7f99ebfff700 1 -- 172.21.15.1:0/2780379563 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e400c850 con 0x7f99f409a150 2024-03-18T20:06:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.652+0000 7f99ebfff700 1 -- 172.21.15.1:0/2780379563 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e400c9b0 con 0x7f99f409a150 2024-03-18T20:06:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.652+0000 7f99ebfff700 1 -- 172.21.15.1:0/2780379563 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e400cd40 con 0x7f99f409a150 2024-03-18T20:06:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.652+0000 7f99f9ec7700 1 -- 172.21.15.1:0/2780379563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99ec009a70 con 0x7f99f409a150 2024-03-18T20:06:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.652+0000 7f99f9ec7700 1 -- 172.21.15.1:0/2780379563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f414e960 con 0x7f99f409a150 2024-03-18T20:06:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.653+0000 7f99f9ec7700 1 -- 172.21.15.1:0/2780379563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f40676a0 con 0x7f99f409a150 2024-03-18T20:06:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.653+0000 7f99ebfff700 1 -- 172.21.15.1:0/2780379563 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 90) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f99e4013470 con 0x7f99f409a150 2024-03-18T20:06:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.654+0000 7f99ebfff700 1 --2- 172.21.15.1:0/2780379563 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f99d406ae00 0x7f99d406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.654+0000 7f99ebfff700 1 -- 172.21.15.1:0/2780379563 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f99e4089ac0 con 0x7f99f409a150 2024-03-18T20:06:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.654+0000 7f99f37fe700 1 --2- 172.21.15.1:0/2780379563 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f99d406ae00 0x7f99d406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.654+0000 7f99f37fe700 1 --2- 172.21.15.1:0/2780379563 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f99d406ae00 0x7f99d406d2c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f99dc0011e0 tx=0x7f99dc001da0 comp rx=0 tx=0).ready entity=mgr.9561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.659+0000 7f99ebfff700 1 -- 172.21.15.1:0/2780379563 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f99e4054910 con 0x7f99f409a150 2024-03-18T20:06:20.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:20.935+0000 7f99f9ec7700 1 -- 172.21.15.1:0/2780379563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f99f4072cd0 con 0x7f99f409a150 2024-03-18T20:06:21.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:21.861+0000 7f99ebfff700 1 -- 172.21.15.1:0/2780379563 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 91) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f99e4054ce0 con 0x7f99f409a150 2024-03-18T20:06:21.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:21.862+0000 7f99ebfff700 1 -- 172.21.15.1:0/2780379563 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v91) v1 ==== 93+0+0 (secure 0 0 0) 0x7f99e4051550 con 0x7f99f409a150 2024-03-18T20:06:21.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:21.867+0000 7f99f9ec7700 1 -- 172.21.15.1:0/2780379563 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f99d406ae00 msgr2=0x7f99d406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:21.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:21.867+0000 7f99f9ec7700 1 --2- 172.21.15.1:0/2780379563 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f99d406ae00 0x7f99d406d2c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f99dc0011e0 tx=0x7f99dc001da0 comp rx=0 tx=0).stop 2024-03-18T20:06:21.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:21.867+0000 7f99f9ec7700 1 -- 172.21.15.1:0/2780379563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99f409a150 msgr2=0x7f99f414e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:21.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:21.867+0000 7f99f9ec7700 1 --2- 172.21.15.1:0/2780379563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99f409a150 0x7f99f414e420 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f99e400ebb0 tx=0x7f99e400eec0 comp rx=0 tx=0).stop 2024-03-18T20:06:21.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:21.867+0000 7f99f9ec7700 1 -- 172.21.15.1:0/2780379563 shutdown_connections 2024-03-18T20:06:21.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:21.867+0000 7f99f9ec7700 1 --2- 172.21.15.1:0/2780379563 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f99d406ae00 0x7f99d406d2c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:21.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:21.867+0000 7f99f9ec7700 1 --2- 172.21.15.1:0/2780379563 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f99f415d5b0 0x7f99f40b54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:21.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:21.867+0000 7f99f9ec7700 1 --2- 172.21.15.1:0/2780379563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f99f406ca80 0x7f99f40b4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:21.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:21.867+0000 7f99f9ec7700 1 --2- 172.21.15.1:0/2780379563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99f409a150 0x7f99f414e420 secure :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f99e400ebb0 tx=0x7f99e400eec0 comp rx=0 tx=0).stop 2024-03-18T20:06:21.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:21.867+0000 7f99f9ec7700 1 -- 172.21.15.1:0/2780379563 >> 172.21.15.1:0/2780379563 conn(0x7f99f40b2bd0 msgr2=0x7f99f4099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:21.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:21.867+0000 7f99f9ec7700 1 -- 172.21.15.1:0/2780379563 shutdown_connections 2024-03-18T20:06:21.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:21.868+0000 7f99f9ec7700 1 -- 172.21.15.1:0/2780379563 wait complete. 2024-03-18T20:06:21.895 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T20:06:21.895 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:06:21.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:06:21.985 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T20:06:21.985 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:06:21.985 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:06:22.006 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:06:22.013 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:22.008+0000 7f7b9e191240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:06:22.033 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:22.028+0000 7fc3c8580240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:06:22.160 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:22.155+0000 7f7b9e191240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:06:22.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.188+0000 7fe491078700 1 Processor -- start 2024-03-18T20:06:22.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.188+0000 7fe491078700 1 -- start start 2024-03-18T20:06:22.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.190+0000 7fe491078700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c06a750 0x7fe48c16b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:22.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.190+0000 7fe491078700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe48c08a270 0x7fe48c16a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:22.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.190+0000 7fe491078700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48c070f30 con 0x7fe48c08a7c0 2024-03-18T20:06:22.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.190+0000 7fe491078700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48c16dcc0 con 0x7fe48c08a270 2024-03-18T20:06:22.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.190+0000 7fe491078700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c16dfa0 con 0x7fe48c06a750 2024-03-18T20:06:22.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.190+0000 7fe48a59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe48c08a7c0 0x7fe48c06ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33852/0 (socket says 172.21.15.1:33852) 2024-03-18T20:06:22.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.190+0000 7fe48a59c700 1 -- 172.21.15.1:0/3257369060 learned_addr learned my addr 172.21.15.1:0/3257369060 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:22.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.192+0000 7fe48b59e700 1 --2- 172.21.15.1:0/3257369060 >> v2:172.21.15.165:3300/0 conn(0x7fe48c08a270 0x7fe48c16a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:22.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.192+0000 7fe489d9b700 1 -- 172.21.15.1:0/3257369060 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306400725 0 0) 0x7fe48c070f30 con 0x7fe48c08a7c0 2024-03-18T20:06:22.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.192+0000 7fe489d9b700 1 -- 172.21.15.1:0/3257369060 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe474003590 con 0x7fe48c08a7c0 2024-03-18T20:06:22.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.193+0000 7fe48b59e700 1 -- 172.21.15.1:0/3257369060 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c06a750 msgr2=0x7fe48c16b920 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T20:06:22.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.193+0000 7fe48b59e700 1 --2- 172.21.15.1:0/3257369060 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c06a750 0x7fe48c16b920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.193+0000 7fe48b59e700 1 -- 172.21.15.1:0/3257369060 >> v1:172.21.15.1:6789/0 conn(0x7fe48c08a7c0 legacy=0x7fe48c06ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:22.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.193+0000 7fe48b59e700 1 -- 172.21.15.1:0/3257369060 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c16e280 con 0x7fe48c08a270 2024-03-18T20:06:22.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.193+0000 7fe48b59e700 1 --2- 172.21.15.1:0/3257369060 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe48c08a270 0x7fe48c16a7d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fe480009910 tx=0x7fe480009c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cd0845ba80391d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:22.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.193+0000 7fe489d9b700 1 -- 172.21.15.1:0/3257369060 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48000e070 con 0x7fe48c08a270 2024-03-18T20:06:22.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.193+0000 7fe489d9b700 1 -- 172.21.15.1:0/3257369060 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe480009c90 con 0x7fe48c08a270 2024-03-18T20:06:22.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.193+0000 7fe489d9b700 1 -- 172.21.15.1:0/3257369060 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48000b9f0 con 0x7fe48c08a270 2024-03-18T20:06:22.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.195+0000 7fe491078700 1 -- 172.21.15.1:0/3257369060 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe48c08a270 msgr2=0x7fe48c16a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:22.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.195+0000 7fe491078700 1 --2- 172.21.15.1:0/3257369060 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe48c08a270 0x7fe48c16a7d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fe480009910 tx=0x7fe480009c20 comp rx=0 tx=0).stop 2024-03-18T20:06:22.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.195+0000 7fe491078700 1 -- 172.21.15.1:0/3257369060 shutdown_connections 2024-03-18T20:06:22.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.195+0000 7fe491078700 1 --2- 172.21.15.1:0/3257369060 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe48c08a270 0x7fe48c16a7d0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.195+0000 7fe491078700 1 --2- 172.21.15.1:0/3257369060 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c06a750 0x7fe48c16b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.195+0000 7fe491078700 1 -- 172.21.15.1:0/3257369060 >> 172.21.15.1:0/3257369060 conn(0x7fe48c0b0a00 msgr2=0x7fe48c0b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:22.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.195+0000 7fe491078700 1 -- 172.21.15.1:0/3257369060 shutdown_connections 2024-03-18T20:06:22.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.195+0000 7fe491078700 1 -- 172.21.15.1:0/3257369060 wait complete. 2024-03-18T20:06:22.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.198+0000 7fe491078700 1 Processor -- start 2024-03-18T20:06:22.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.198+0000 7fe491078700 1 -- start start 2024-03-18T20:06:22.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fe491078700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe48c06a750 0x7fe48c0cae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:22.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fe491078700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe48c08a270 0x7fe48c0cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:22.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fe491078700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c08a7c0 0x7fe48c0cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:22.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fe491078700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c09be50 con 0x7fe48c06a750 2024-03-18T20:06:22.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fe491078700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0cb7e0 con 0x7fe48c08a270 2024-03-18T20:06:22.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fe491078700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0cbac0 con 0x7fe48c08a7c0 2024-03-18T20:06:22.205 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fc3c8580240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:06:22.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fe48a59c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe48c08a270 0x7fe48c0cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:22.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fe48b59e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c08a7c0 0x7fe48c0cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:22.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fe48b59e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c08a7c0 0x7fe48c0cd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:47190/0 (socket says 172.21.15.1:47190) 2024-03-18T20:06:22.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fe48a59c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe48c08a270 0x7fe48c0cceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:39974/0 (socket says 172.21.15.1:39974) 2024-03-18T20:06:22.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fe48b59e700 1 -- 172.21.15.1:0/3865889908 learned_addr learned my addr 172.21.15.1:0/3865889908 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:22.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fe48ad9d700 1 --2- 172.21.15.1:0/3865889908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe48c06a750 0x7fe48c0cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:22.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fe48a59c700 1 -- 172.21.15.1:0/3865889908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c08a7c0 msgr2=0x7fe48c0cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:22.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fe48a59c700 1 --2- 172.21.15.1:0/3865889908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c08a7c0 0x7fe48c0cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fe48a59c700 1 -- 172.21.15.1:0/3865889908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe48c06a750 msgr2=0x7fe48c0cae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:22.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fe48a59c700 1 --2- 172.21.15.1:0/3865889908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe48c06a750 0x7fe48c0cae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fe48a59c700 1 -- 172.21.15.1:0/3865889908 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4800095f0 con 0x7fe48c08a270 2024-03-18T20:06:22.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fe48b59e700 1 --2- 172.21.15.1:0/3865889908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c08a7c0 0x7fe48c0cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:22.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.200+0000 7fe48ad9d700 1 --2- 172.21.15.1:0/3865889908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe48c06a750 0x7fe48c0cae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:22.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.201+0000 7fe48a59c700 1 --2- 172.21.15.1:0/3865889908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe48c08a270 0x7fe48c0cceb0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe47c00c430 tx=0x7fe47c00c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:22.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.201+0000 7fe4737fe700 1 -- 172.21.15.1:0/3865889908 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c005d70 con 0x7fe48c08a270 2024-03-18T20:06:22.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.201+0000 7fe4737fe700 1 -- 172.21.15.1:0/3865889908 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47c0070d0 con 0x7fe48c08a270 2024-03-18T20:06:22.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.201+0000 7fe491078700 1 -- 172.21.15.1:0/3865889908 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48c0cbda0 con 0x7fe48c08a270 2024-03-18T20:06:22.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.201+0000 7fe4737fe700 1 -- 172.21.15.1:0/3865889908 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c0074c0 con 0x7fe48c08a270 2024-03-18T20:06:22.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.201+0000 7fe491078700 1 -- 172.21.15.1:0/3865889908 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48c0cc360 con 0x7fe48c08a270 2024-03-18T20:06:22.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.202+0000 7fe491078700 1 -- 172.21.15.1:0/3865889908 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe48c0b4ff0 con 0x7fe48c08a270 2024-03-18T20:06:22.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.202+0000 7fe4737fe700 1 -- 172.21.15.1:0/3865889908 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 91) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fe47c007620 con 0x7fe48c08a270 2024-03-18T20:06:22.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.203+0000 7fe4737fe700 1 --2- 172.21.15.1:0/3865889908 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe47806adb0 0x7fe47806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:22.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.203+0000 7fe48ad9d700 1 -- 172.21.15.1:0/3865889908 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe47806adb0 msgr2=0x7fe47806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:22.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.203+0000 7fe4737fe700 1 -- 172.21.15.1:0/3865889908 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fe47c088f60 con 0x7fe48c08a270 2024-03-18T20:06:22.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.203+0000 7fe48ad9d700 1 --2- 172.21.15.1:0/3865889908 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe47806adb0 0x7fe47806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:06:22.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.207+0000 7fe4737fe700 1 -- 172.21.15.1:0/3865889908 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe47c053d20 con 0x7fe48c08a270 2024-03-18T20:06:22.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.404+0000 7fe48ad9d700 1 -- 172.21.15.1:0/3865889908 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe47806adb0 msgr2=0x7fe47806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.404+0000 7fe48ad9d700 1 --2- 172.21.15.1:0/3865889908 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe47806adb0 0x7fe47806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:06:22.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.449+0000 7fe491078700 1 -- 172.21.15.1:0/3865889908 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe48c073350 con 0x7fe48c08a270 2024-03-18T20:06:22.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.451+0000 7fe4737fe700 1 -- 172.21.15.1:0/3865889908 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v167) v1 ==== 74+0+16908 (secure 0 0 0) 0x7fe47c017020 con 0x7fe48c08a270 2024-03-18T20:06:22.456 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:22.457 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":167,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:06:21.844884+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:10:02.452367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-18T19:10:03.585045+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":112,"snap_epoch":167,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6833/76998":"2024-03-19T19:10:19.888890+0000","172.21.15.1:6832/76998":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/371341431":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/1433676792":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/4211592802":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/4117009712":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/2398817063":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/1643985667":"2024-03-19T19:10:19.888890+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":54},{"begin":59,"length":54}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":48,"length":9},{"begin":104,"length":9}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:06:22.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.453+0000 7fe491078700 1 -- 172.21.15.1:0/3865889908 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe47806adb0 msgr2=0x7fe47806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:06:22.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.453+0000 7fe491078700 1 --2- 172.21.15.1:0/3865889908 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe47806adb0 0x7fe47806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.453+0000 7fe491078700 1 -- 172.21.15.1:0/3865889908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe48c08a270 msgr2=0x7fe48c0cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:22.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.453+0000 7fe491078700 1 --2- 172.21.15.1:0/3865889908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe48c08a270 0x7fe48c0cceb0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe47c00c430 tx=0x7fe47c00c740 comp rx=0 tx=0).stop 2024-03-18T20:06:22.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.453+0000 7fe491078700 1 -- 172.21.15.1:0/3865889908 shutdown_connections 2024-03-18T20:06:22.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.453+0000 7fe491078700 1 --2- 172.21.15.1:0/3865889908 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe47806adb0 0x7fe47806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.453+0000 7fe491078700 1 --2- 172.21.15.1:0/3865889908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe48c08a7c0 0x7fe48c0cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.453+0000 7fe491078700 1 --2- 172.21.15.1:0/3865889908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe48c06a750 0x7fe48c0cae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.453+0000 7fe491078700 1 --2- 172.21.15.1:0/3865889908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe48c08a270 0x7fe48c0cceb0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.453+0000 7fe491078700 1 -- 172.21.15.1:0/3865889908 >> 172.21.15.1:0/3865889908 conn(0x7fe48c0b0a00 msgr2=0x7fe48c06c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:22.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.454+0000 7fe491078700 1 -- 172.21.15.1:0/3865889908 shutdown_connections 2024-03-18T20:06:22.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.454+0000 7fe491078700 1 -- 172.21.15.1:0/3865889908 wait complete. 2024-03-18T20:06:22.459 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:22.453+0000 7f7b9e191240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:06:22.471 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T19:10:02.452367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '160', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T19:10:03.585045+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '167', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 112, 'snap_epoch': 167, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:06:22.472 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:06:22.519 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:22.515+0000 7fc3c8580240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:06:22.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.600+0000 7f208ec7f700 1 Processor -- start 2024-03-18T20:06:22.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.600+0000 7f208ec7f700 1 -- start start 2024-03-18T20:06:22.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.601+0000 7f208ec7f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2088151150 0x7f208815b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:22.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.601+0000 7f208ec7f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f208806e2c0 0x7f208806e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:22.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.601+0000 7f208ec7f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208806f160 con 0x7f208806ed30 2024-03-18T20:06:22.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.601+0000 7f208ec7f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208815b670 con 0x7f2088151150 2024-03-18T20:06:22.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.601+0000 7f208ec7f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208806d220 con 0x7f208806e2c0 2024-03-18T20:06:22.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.601+0000 7f208ca1b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2088151150 0x7f208815b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:22.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.601+0000 7f208ca1b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2088151150 0x7f208815b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53088/0 (socket says 172.21.15.1:53088) 2024-03-18T20:06:22.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.601+0000 7f208ca1b700 1 -- 172.21.15.1:0/1806775807 learned_addr learned my addr 172.21.15.1:0/1806775807 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:22.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.601+0000 7f2087fff700 1 --2- 172.21.15.1:0/1806775807 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f208806e2c0 0x7f208806e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:22.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.601+0000 7f208ca1b700 1 -- 172.21.15.1:0/1806775807 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f208806e2c0 msgr2=0x7f208806e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:22.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.601+0000 7f208ca1b700 1 --2- 172.21.15.1:0/1806775807 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f208806e2c0 0x7f208806e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.601+0000 7f208ca1b700 1 -- 172.21.15.1:0/1806775807 >> v1:172.21.15.165:6789/0 conn(0x7f208806ed30 legacy=0x7f208815dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:22.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.601+0000 7f208ca1b700 1 -- 172.21.15.1:0/1806775807 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208806d500 con 0x7f2088151150 2024-03-18T20:06:22.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.601+0000 7f208ca1b700 1 --2- 172.21.15.1:0/1806775807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2088151150 0x7f208815b130 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f207c0098f0 tx=0x7f207c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ccfe88f58462c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:22.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.601+0000 7f20877fe700 1 -- 172.21.15.1:0/1806775807 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c011830 con 0x7f2088151150 2024-03-18T20:06:22.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.602+0000 7f20877fe700 1 -- 172.21.15.1:0/1806775807 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207c011990 con 0x7f2088151150 2024-03-18T20:06:22.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.602+0000 7f20877fe700 1 -- 172.21.15.1:0/1806775807 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c011d20 con 0x7f2088151150 2024-03-18T20:06:22.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.602+0000 7f208ec7f700 1 -- 172.21.15.1:0/1806775807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2088151150 msgr2=0x7f208815b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:22.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.602+0000 7f208ec7f700 1 --2- 172.21.15.1:0/1806775807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2088151150 0x7f208815b130 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f207c0098f0 tx=0x7f207c009c00 comp rx=0 tx=0).stop 2024-03-18T20:06:22.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.602+0000 7f208ec7f700 1 -- 172.21.15.1:0/1806775807 shutdown_connections 2024-03-18T20:06:22.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.602+0000 7f208ec7f700 1 --2- 172.21.15.1:0/1806775807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2088151150 0x7f208815b130 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.602+0000 7f208ec7f700 1 --2- 172.21.15.1:0/1806775807 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f208806e2c0 0x7f208806e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.602+0000 7f208ec7f700 1 -- 172.21.15.1:0/1806775807 >> 172.21.15.1:0/1806775807 conn(0x7f20880b1350 msgr2=0x7f20880b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:22.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.602+0000 7f208ec7f700 1 -- 172.21.15.1:0/1806775807 shutdown_connections 2024-03-18T20:06:22.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.602+0000 7f208ec7f700 1 -- 172.21.15.1:0/1806775807 wait complete. 2024-03-18T20:06:22.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.604+0000 7f208ec7f700 1 Processor -- start 2024-03-18T20:06:22.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.604+0000 7f208ec7f700 1 -- start start 2024-03-18T20:06:22.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.605+0000 7f208ec7f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f208806e2c0 0x7f2088154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:22.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.605+0000 7f208ec7f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f208806ed30 0x7f208814e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:22.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.605+0000 7f208ec7f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2088151150 0x7f208814f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:22.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.605+0000 7f208ec7f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2088158c30 con 0x7f2088151150 2024-03-18T20:06:22.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.605+0000 7f208ec7f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2088158300 con 0x7f208806e2c0 2024-03-18T20:06:22.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.605+0000 7f208ec7f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20881585e0 con 0x7f208806ed30 2024-03-18T20:06:22.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.605+0000 7f208d21c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2088151150 0x7f208814f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:22.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.605+0000 7f2087fff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f208806ed30 0x7f208814e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:22.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.605+0000 7f208d21c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2088151150 0x7f208814f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53102/0 (socket says 172.21.15.1:53102) 2024-03-18T20:06:22.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.605+0000 7f208d21c700 1 -- 172.21.15.1:0/1107264482 learned_addr learned my addr 172.21.15.1:0/1107264482 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:22.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.605+0000 7f208ca1b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f208806e2c0 0x7f2088154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:22.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.605+0000 7f208d21c700 1 -- 172.21.15.1:0/1107264482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f208806ed30 msgr2=0x7f208814e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:22.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.605+0000 7f208d21c700 1 --2- 172.21.15.1:0/1107264482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f208806ed30 0x7f208814e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.605+0000 7f208d21c700 1 -- 172.21.15.1:0/1107264482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f208806e2c0 msgr2=0x7f2088154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:22.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.605+0000 7f208d21c700 1 --2- 172.21.15.1:0/1107264482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f208806e2c0 0x7f2088154d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.606+0000 7f208d21c700 1 -- 172.21.15.1:0/1107264482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f207c009550 con 0x7f2088151150 2024-03-18T20:06:22.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.606+0000 7f208ca1b700 1 --2- 172.21.15.1:0/1107264482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f208806e2c0 0x7f2088154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:22.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.606+0000 7f208d21c700 1 --2- 172.21.15.1:0/1107264482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2088151150 0x7f208814f9a0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f208000cb50 tx=0x7f208000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:22.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.606+0000 7f20857fa700 1 -- 172.21.15.1:0/1107264482 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2080003c00 con 0x7f2088151150 2024-03-18T20:06:22.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.606+0000 7f20857fa700 1 -- 172.21.15.1:0/1107264482 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2080003d60 con 0x7f2088151150 2024-03-18T20:06:22.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.606+0000 7f208ec7f700 1 -- 172.21.15.1:0/1107264482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20881588c0 con 0x7f2088151150 2024-03-18T20:06:22.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.606+0000 7f20857fa700 1 -- 172.21.15.1:0/1107264482 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208000aa20 con 0x7f2088151150 2024-03-18T20:06:22.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.606+0000 7f208ec7f700 1 -- 172.21.15.1:0/1107264482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2088157fb0 con 0x7f2088151150 2024-03-18T20:06:22.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.607+0000 7f208ec7f700 1 -- 172.21.15.1:0/1107264482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2088158de0 con 0x7f2088151150 2024-03-18T20:06:22.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.608+0000 7f20857fa700 1 -- 172.21.15.1:0/1107264482 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 91) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f208000ab80 con 0x7f2088151150 2024-03-18T20:06:22.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.608+0000 7f20857fa700 1 --2- 172.21.15.1:0/1107264482 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f206c06adb0 0x7f206c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:22.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.608+0000 7f208ca1b700 1 -- 172.21.15.1:0/1107264482 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f206c06adb0 msgr2=0x7f206c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:22.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.608+0000 7f20857fa700 1 -- 172.21.15.1:0/1107264482 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f20800896d0 con 0x7f2088151150 2024-03-18T20:06:22.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.608+0000 7f208ca1b700 1 --2- 172.21.15.1:0/1107264482 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f206c06adb0 0x7f206c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:06:22.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.613+0000 7f20857fa700 1 -- 172.21.15.1:0/1107264482 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2080054520 con 0x7f2088151150 2024-03-18T20:06:22.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.809+0000 7f208ca1b700 1 -- 172.21.15.1:0/1107264482 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f206c06adb0 msgr2=0x7f206c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:22.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.809+0000 7f208ca1b700 1 --2- 172.21.15.1:0/1107264482 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f206c06adb0 0x7f206c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:06:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.815+0000 7f208ec7f700 1 -- 172.21.15.1:0/1107264482 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f20880676a0 con 0x7f2088151150 2024-03-18T20:06:22.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.815+0000 7f20857fa700 1 -- 172.21.15.1:0/1107264482 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v167) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2080057b40 con 0x7f2088151150 2024-03-18T20:06:22.821 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:06:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.818+0000 7f208ec7f700 1 -- 172.21.15.1:0/1107264482 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f206c06adb0 msgr2=0x7f206c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:06:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.818+0000 7f208ec7f700 1 --2- 172.21.15.1:0/1107264482 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f206c06adb0 0x7f206c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.818+0000 7f208ec7f700 1 -- 172.21.15.1:0/1107264482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2088151150 msgr2=0x7f208814f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.818+0000 7f208ec7f700 1 --2- 172.21.15.1:0/1107264482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2088151150 0x7f208814f9a0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f208000cb50 tx=0x7f208000ce60 comp rx=0 tx=0).stop 2024-03-18T20:06:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.818+0000 7f208ec7f700 1 -- 172.21.15.1:0/1107264482 shutdown_connections 2024-03-18T20:06:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.818+0000 7f208ec7f700 1 --2- 172.21.15.1:0/1107264482 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f206c06adb0 0x7f206c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.818+0000 7f208ec7f700 1 --2- 172.21.15.1:0/1107264482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2088151150 0x7f208814f9a0 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.818+0000 7f208ec7f700 1 --2- 172.21.15.1:0/1107264482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f208806e2c0 0x7f2088154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.818+0000 7f208ec7f700 1 --2- 172.21.15.1:0/1107264482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f208806ed30 0x7f208814e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.818+0000 7f208ec7f700 1 -- 172.21.15.1:0/1107264482 >> 172.21.15.1:0/1107264482 conn(0x7f20880b1350 msgr2=0x7f20880af2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:22.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.818+0000 7f208ec7f700 1 -- 172.21.15.1:0/1107264482 shutdown_connections 2024-03-18T20:06:22.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.818+0000 7f208ec7f700 1 -- 172.21.15.1:0/1107264482 wait complete. 2024-03-18T20:06:22.835 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:06:22.889 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:22.884+0000 7f7b9e191240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:06:22.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.961+0000 7fca9264e700 1 Processor -- start 2024-03-18T20:06:22.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.961+0000 7fca9264e700 1 -- start start 2024-03-18T20:06:22.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.962+0000 7fca9264e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca8c15b480 0x7fca8c0cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:22.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.962+0000 7fca9264e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca8c0c4c50 con 0x7fca8c0cfdd0 2024-03-18T20:06:22.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.962+0000 7fca9264e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca8c0c5210 con 0x7fca8c0cc7b0 2024-03-18T20:06:22.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.962+0000 7fca9264e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c0cccb0 con 0x7fca8c15b480 2024-03-18T20:06:22.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.962+0000 7fca90beb700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fca8c0cfdd0 0x7fca8c0c56a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33860/0 (socket says 172.21.15.1:33860) 2024-03-18T20:06:22.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.962+0000 7fca90beb700 1 -- 172.21.15.1:0/1519478743 learned_addr learned my addr 172.21.15.1:0/1519478743 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:22.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.962+0000 7fca8bfff700 1 --2- 172.21.15.1:0/1519478743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca8c15b480 0x7fca8c0cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:22.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.962+0000 7fca8b7fe700 1 -- 172.21.15.1:0/1519478743 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459791373 0 0) 0x7fca8c0c4c50 con 0x7fca8c0cfdd0 2024-03-18T20:06:22.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.962+0000 7fca8b7fe700 1 -- 172.21.15.1:0/1519478743 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca780035b0 con 0x7fca8c0cfdd0 2024-03-18T20:06:22.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.962+0000 7fca8b7fe700 1 -- 172.21.15.1:0/1519478743 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240670451 0 0) 0x7fca8c0c5210 con 0x7fca8c0cc7b0 2024-03-18T20:06:22.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.962+0000 7fca8b7fe700 1 -- 172.21.15.1:0/1519478743 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca8c0c4c50 con 0x7fca8c0cc7b0 2024-03-18T20:06:22.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.962+0000 7fca8bfff700 1 -- 172.21.15.1:0/1519478743 >> v1:172.21.15.165:6789/0 conn(0x7fca8c0cc7b0 legacy=0x7fca8c0cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:22.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.962+0000 7fca8bfff700 1 -- 172.21.15.1:0/1519478743 >> v1:172.21.15.1:6789/0 conn(0x7fca8c0cfdd0 legacy=0x7fca8c0c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:22.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.962+0000 7fca8bfff700 1 -- 172.21.15.1:0/1519478743 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8c0ccf90 con 0x7fca8c15b480 2024-03-18T20:06:22.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.962+0000 7fca8bfff700 1 --2- 172.21.15.1:0/1519478743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca8c15b480 0x7fca8c0cedd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fca7c009870 tx=0x7fca7c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=612dc68a83d5ef76 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:22.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.963+0000 7fca8b7fe700 1 -- 172.21.15.1:0/1519478743 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c011830 con 0x7fca8c15b480 2024-03-18T20:06:22.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.963+0000 7fca8b7fe700 1 -- 172.21.15.1:0/1519478743 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7c011990 con 0x7fca8c15b480 2024-03-18T20:06:22.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.963+0000 7fca8b7fe700 1 -- 172.21.15.1:0/1519478743 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c011d20 con 0x7fca8c15b480 2024-03-18T20:06:22.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.964+0000 7fca9264e700 1 -- 172.21.15.1:0/1519478743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca8c15b480 msgr2=0x7fca8c0cedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:22.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.964+0000 7fca9264e700 1 --2- 172.21.15.1:0/1519478743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca8c15b480 0x7fca8c0cedd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fca7c009870 tx=0x7fca7c009b80 comp rx=0 tx=0).stop 2024-03-18T20:06:22.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.964+0000 7fca9264e700 1 -- 172.21.15.1:0/1519478743 shutdown_connections 2024-03-18T20:06:22.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.964+0000 7fca9264e700 1 --2- 172.21.15.1:0/1519478743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca8c15b480 0x7fca8c0cedd0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.964+0000 7fca9264e700 1 -- 172.21.15.1:0/1519478743 >> 172.21.15.1:0/1519478743 conn(0x7fca8c0b2bf0 msgr2=0x7fca8c0b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:22.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.964+0000 7fca9264e700 1 -- 172.21.15.1:0/1519478743 shutdown_connections 2024-03-18T20:06:22.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.964+0000 7fca9264e700 1 -- 172.21.15.1:0/1519478743 wait complete. 2024-03-18T20:06:22.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.966+0000 7fca9264e700 1 Processor -- start 2024-03-18T20:06:22.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.966+0000 7fca9264e700 1 -- start start 2024-03-18T20:06:22.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.967+0000 7fca9264e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca8c0cc7b0 0x7fca8c0caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:22.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.967+0000 7fca9264e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca8c0cfdd0 0x7fca8c14a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:22.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.967+0000 7fca9264e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca8c15b480 0x7fca8c0cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:22.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.967+0000 7fca9264e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c0c9220 con 0x7fca8c15b480 2024-03-18T20:06:22.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.967+0000 7fca9264e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c1f2e40 con 0x7fca8c0cfdd0 2024-03-18T20:06:22.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.967+0000 7fca9264e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c1f6ee0 con 0x7fca8c0cc7b0 2024-03-18T20:06:22.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.967+0000 7fca90beb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca8c15b480 0x7fca8c0cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:22.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.967+0000 7fca8bfff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca8c0cc7b0 0x7fca8c0caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.967+0000 7fca90beb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca8c15b480 0x7fca8c0cd300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53110/0 (socket says 172.21.15.1:53110) 2024-03-18T20:06:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.967+0000 7fca90beb700 1 -- 172.21.15.1:0/3472407761 learned_addr learned my addr 172.21.15.1:0/3472407761 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.967+0000 7fca8bfff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca8c0cc7b0 0x7fca8c0caba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:47216/0 (socket says 172.21.15.1:47216) 2024-03-18T20:06:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.967+0000 7fca83fff700 1 --2- 172.21.15.1:0/3472407761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca8c0cfdd0 0x7fca8c14a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.967+0000 7fca90beb700 1 -- 172.21.15.1:0/3472407761 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca8c0cc7b0 msgr2=0x7fca8c0caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.967+0000 7fca90beb700 1 --2- 172.21.15.1:0/3472407761 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca8c0cc7b0 0x7fca8c0caba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.967+0000 7fca90beb700 1 -- 172.21.15.1:0/3472407761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca8c0cfdd0 msgr2=0x7fca8c14a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.967+0000 7fca90beb700 1 --2- 172.21.15.1:0/3472407761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca8c0cfdd0 0x7fca8c14a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.967+0000 7fca90beb700 1 -- 172.21.15.1:0/3472407761 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c0094a0 con 0x7fca8c15b480 2024-03-18T20:06:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.967+0000 7fca8bfff700 1 --2- 172.21.15.1:0/3472407761 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca8c0cc7b0 0x7fca8c0caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.968+0000 7fca90beb700 1 --2- 172.21.15.1:0/3472407761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca8c15b480 0x7fca8c0cd300 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fca8400cb40 tx=0x7fca8400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.968+0000 7fca897fa700 1 -- 172.21.15.1:0/3472407761 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8400a9e0 con 0x7fca8c15b480 2024-03-18T20:06:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.968+0000 7fca897fa700 1 -- 172.21.15.1:0/3472407761 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8400ab40 con 0x7fca8c15b480 2024-03-18T20:06:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.968+0000 7fca9264e700 1 -- 172.21.15.1:0/3472407761 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca8c1f7410 con 0x7fca8c15b480 2024-03-18T20:06:22.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.968+0000 7fca897fa700 1 -- 172.21.15.1:0/3472407761 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca84012400 con 0x7fca8c15b480 2024-03-18T20:06:22.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.968+0000 7fca9264e700 1 -- 172.21.15.1:0/3472407761 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8c1f7130 con 0x7fca8c15b480 2024-03-18T20:06:22.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.969+0000 7fca9264e700 1 -- 172.21.15.1:0/3472407761 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca8c0c9390 con 0x7fca8c15b480 2024-03-18T20:06:22.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.969+0000 7fca897fa700 1 -- 172.21.15.1:0/3472407761 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 91) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fca8401b3b0 con 0x7fca8c15b480 2024-03-18T20:06:22.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.970+0000 7fca897fa700 1 --2- 172.21.15.1:0/3472407761 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fca6c06adb0 0x7fca6c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:22.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.970+0000 7fca8bfff700 1 -- 172.21.15.1:0/3472407761 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fca6c06adb0 msgr2=0x7fca6c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:22.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.970+0000 7fca897fa700 1 -- 172.21.15.1:0/3472407761 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fca84089410 con 0x7fca8c15b480 2024-03-18T20:06:22.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.970+0000 7fca8bfff700 1 --2- 172.21.15.1:0/3472407761 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fca6c06adb0 0x7fca6c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:06:22.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:22.974+0000 7fca897fa700 1 -- 172.21.15.1:0/3472407761 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fca8c0c9390 con 0x7fca8c15b480 2024-03-18T20:06:22.981 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:22.977+0000 7fc3c8580240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:06:23.099 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:23.095+0000 7f7b9e191240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:06:23.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.171+0000 7fca8bfff700 1 -- 172.21.15.1:0/3472407761 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fca6c06adb0 msgr2=0x7fca6c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:23.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.171+0000 7fca8bfff700 1 --2- 172.21.15.1:0/3472407761 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fca6c06adb0 0x7fca6c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:06:23.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.177+0000 7fca9264e700 1 -- 172.21.15.1:0/3472407761 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fca8c0c9360 con 0x7fca8c15b480 2024-03-18T20:06:23.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.178+0000 7fca897fa700 1 -- 172.21.15.1:0/3472407761 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v167) v1 ==== 104+0+11 (secure 0 0 0) 0x7fca84054260 con 0x7fca8c15b480 2024-03-18T20:06:23.182 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:06:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.180+0000 7fca9264e700 1 -- 172.21.15.1:0/3472407761 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fca6c06adb0 msgr2=0x7fca6c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:06:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.180+0000 7fca9264e700 1 --2- 172.21.15.1:0/3472407761 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fca6c06adb0 0x7fca6c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.180+0000 7fca9264e700 1 -- 172.21.15.1:0/3472407761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca8c15b480 msgr2=0x7fca8c0cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.180+0000 7fca9264e700 1 --2- 172.21.15.1:0/3472407761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca8c15b480 0x7fca8c0cd300 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fca8400cb40 tx=0x7fca8400ce50 comp rx=0 tx=0).stop 2024-03-18T20:06:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.180+0000 7fca9264e700 1 -- 172.21.15.1:0/3472407761 shutdown_connections 2024-03-18T20:06:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.180+0000 7fca9264e700 1 --2- 172.21.15.1:0/3472407761 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fca6c06adb0 0x7fca6c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.180+0000 7fca9264e700 1 --2- 172.21.15.1:0/3472407761 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca8c0cc7b0 0x7fca8c0caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.180+0000 7fca9264e700 1 --2- 172.21.15.1:0/3472407761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca8c15b480 0x7fca8c0cd300 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.180+0000 7fca9264e700 1 --2- 172.21.15.1:0/3472407761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca8c0cfdd0 0x7fca8c14a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.180+0000 7fca9264e700 1 -- 172.21.15.1:0/3472407761 >> 172.21.15.1:0/3472407761 conn(0x7fca8c0b2bf0 msgr2=0x7fca8c0b5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.180+0000 7fca9264e700 1 -- 172.21.15.1:0/3472407761 shutdown_connections 2024-03-18T20:06:23.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.180+0000 7fca9264e700 1 -- 172.21.15.1:0/3472407761 wait complete. 2024-03-18T20:06:23.197 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:06:23.201 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:23.196+0000 7fc3c8580240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:06:23.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.322+0000 7ff191b5e700 1 Processor -- start 2024-03-18T20:06:23.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.322+0000 7ff191b5e700 1 -- start start 2024-03-18T20:06:23.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.323+0000 7ff191b5e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff18c06eda0 0x7ff18c161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:23.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.323+0000 7ff191b5e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff18c06f180 0x7ff18c0ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:23.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.323+0000 7ff191b5e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18c0caac0 con 0x7ff18c06a750 2024-03-18T20:06:23.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.323+0000 7ff191b5e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff18c0cb320 con 0x7ff18c06eda0 2024-03-18T20:06:23.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.323+0000 7ff191b5e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c0ce010 con 0x7ff18c06f180 2024-03-18T20:06:23.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.323+0000 7ff18b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff18c06eda0 0x7ff18c161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:23.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.323+0000 7ff18b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff18c06eda0 0x7ff18c161c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53124/0 (socket says 172.21.15.1:53124) 2024-03-18T20:06:23.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.323+0000 7ff18b7fe700 1 -- 172.21.15.1:0/3396026850 learned_addr learned my addr 172.21.15.1:0/3396026850 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:23.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.323+0000 7ff18affd700 1 --2- 172.21.15.1:0/3396026850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff18c06f180 0x7ff18c0ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:23.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.323+0000 7ff18b7fe700 1 -- 172.21.15.1:0/3396026850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff18c06f180 msgr2=0x7ff18c0ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:23.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.323+0000 7ff18b7fe700 1 --2- 172.21.15.1:0/3396026850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff18c06f180 0x7ff18c0ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.323+0000 7ff18b7fe700 1 -- 172.21.15.1:0/3396026850 >> v1:172.21.15.165:6789/0 conn(0x7ff18c06a750 legacy=0x7ff18c0715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:23.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.323+0000 7ff18b7fe700 1 -- 172.21.15.1:0/3396026850 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18c0cad60 con 0x7ff18c06eda0 2024-03-18T20:06:23.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.324+0000 7ff18b7fe700 1 --2- 172.21.15.1:0/3396026850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff18c06eda0 0x7ff18c161c60 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7ff17c0098f0 tx=0x7ff17c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=402af045855e4d02 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:23.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.324+0000 7ff18a7fc700 1 -- 172.21.15.1:0/3396026850 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c011830 con 0x7ff18c06eda0 2024-03-18T20:06:23.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.324+0000 7ff18a7fc700 1 -- 172.21.15.1:0/3396026850 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17c011990 con 0x7ff18c06eda0 2024-03-18T20:06:23.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.324+0000 7ff18a7fc700 1 -- 172.21.15.1:0/3396026850 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c011d20 con 0x7ff18c06eda0 2024-03-18T20:06:23.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.324+0000 7ff191b5e700 1 -- 172.21.15.1:0/3396026850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff18c06eda0 msgr2=0x7ff18c161c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:23.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.324+0000 7ff191b5e700 1 --2- 172.21.15.1:0/3396026850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff18c06eda0 0x7ff18c161c60 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7ff17c0098f0 tx=0x7ff17c009c00 comp rx=0 tx=0).stop 2024-03-18T20:06:23.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.324+0000 7ff191b5e700 1 -- 172.21.15.1:0/3396026850 shutdown_connections 2024-03-18T20:06:23.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.324+0000 7ff191b5e700 1 --2- 172.21.15.1:0/3396026850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff18c06eda0 0x7ff18c161c60 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.324+0000 7ff191b5e700 1 --2- 172.21.15.1:0/3396026850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff18c06f180 0x7ff18c0ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.324+0000 7ff191b5e700 1 -- 172.21.15.1:0/3396026850 >> 172.21.15.1:0/3396026850 conn(0x7ff18c0cff30 msgr2=0x7ff18c0d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:23.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.324+0000 7ff191b5e700 1 -- 172.21.15.1:0/3396026850 shutdown_connections 2024-03-18T20:06:23.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.324+0000 7ff191b5e700 1 -- 172.21.15.1:0/3396026850 wait complete. 2024-03-18T20:06:23.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.326+0000 7ff191b5e700 1 Processor -- start 2024-03-18T20:06:23.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.326+0000 7ff191b5e700 1 -- start start 2024-03-18T20:06:23.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.326+0000 7ff191b5e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff18c06a750 0x7ff18c150340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:23.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff191b5e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff18c06eda0 0x7ff18c0c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:23.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff191b5e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff18c06f180 0x7ff18c0adf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:23.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff191b5e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c0b5370 con 0x7ff18c06a750 2024-03-18T20:06:23.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff191b5e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c0c9160 con 0x7ff18c06eda0 2024-03-18T20:06:23.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff191b5e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c0c8550 con 0x7ff18c06f180 2024-03-18T20:06:23.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff18b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff18c06a750 0x7ff18c150340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:23.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff18b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff18c06a750 0x7ff18c150340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53136/0 (socket says 172.21.15.1:53136) 2024-03-18T20:06:23.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff18b7fe700 1 -- 172.21.15.1:0/4006292315 learned_addr learned my addr 172.21.15.1:0/4006292315 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:23.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff18bfff700 1 --2- 172.21.15.1:0/4006292315 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff18c06f180 0x7ff18c0adf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:23.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff18affd700 1 --2- 172.21.15.1:0/4006292315 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff18c06eda0 0x7ff18c0c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:23.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff18b7fe700 1 -- 172.21.15.1:0/4006292315 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff18c06f180 msgr2=0x7ff18c0adf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:23.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff18b7fe700 1 --2- 172.21.15.1:0/4006292315 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff18c06f180 0x7ff18c0adf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff18b7fe700 1 -- 172.21.15.1:0/4006292315 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff18c06eda0 msgr2=0x7ff18c0c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:23.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff18b7fe700 1 --2- 172.21.15.1:0/4006292315 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff18c06eda0 0x7ff18c0c9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff18b7fe700 1 -- 172.21.15.1:0/4006292315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17c009550 con 0x7ff18c06a750 2024-03-18T20:06:23.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff18b7fe700 1 --2- 172.21.15.1:0/4006292315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff18c06a750 0x7ff18c150340 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7ff17c009c40 tx=0x7ff17c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:23.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff183fff700 1 -- 172.21.15.1:0/4006292315 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c004550 con 0x7ff18c06a750 2024-03-18T20:06:23.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff183fff700 1 -- 172.21.15.1:0/4006292315 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17c01f070 con 0x7ff18c06a750 2024-03-18T20:06:23.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff183fff700 1 -- 172.21.15.1:0/4006292315 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c0038e0 con 0x7ff18c06a750 2024-03-18T20:06:23.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff191b5e700 1 -- 172.21.15.1:0/4006292315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff18c0c8830 con 0x7ff18c06a750 2024-03-18T20:06:23.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.327+0000 7ff191b5e700 1 -- 172.21.15.1:0/4006292315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18c0c8df0 con 0x7ff18c06a750 2024-03-18T20:06:23.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.328+0000 7ff191b5e700 1 -- 172.21.15.1:0/4006292315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18c0b2aa0 con 0x7ff18c06a750 2024-03-18T20:06:23.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.329+0000 7ff183fff700 1 -- 172.21.15.1:0/4006292315 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 91) v1 ==== 89251+0+0 (secure 0 0 0) 0x7ff17c003a40 con 0x7ff18c06a750 2024-03-18T20:06:23.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.330+0000 7ff183fff700 1 --2- 172.21.15.1:0/4006292315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff16c06adb0 0x7ff16c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:23.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.330+0000 7ff18affd700 1 -- 172.21.15.1:0/4006292315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff16c06adb0 msgr2=0x7ff16c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:23.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.330+0000 7ff183fff700 1 -- 172.21.15.1:0/4006292315 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6489+0+0 (secure 0 0 0) 0x7ff17c021070 con 0x7ff18c06a750 2024-03-18T20:06:23.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.330+0000 7ff18affd700 1 --2- 172.21.15.1:0/4006292315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff16c06adb0 0x7ff16c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:06:23.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.334+0000 7ff183fff700 1 -- 172.21.15.1:0/4006292315 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff17c055e00 con 0x7ff18c06a750 2024-03-18T20:06:23.470 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:23.465+0000 7f7b9e191240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:06:23.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.531+0000 7ff18affd700 1 -- 172.21.15.1:0/4006292315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff16c06adb0 msgr2=0x7ff16c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:23.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.531+0000 7ff18affd700 1 --2- 172.21.15.1:0/4006292315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff16c06adb0 0x7ff16c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:06:23.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.538+0000 7ff191b5e700 1 -- 172.21.15.1:0/4006292315 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff18c0676a0 con 0x7ff18c06a750 2024-03-18T20:06:23.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.539+0000 7ff183fff700 1 -- 172.21.15.1:0/4006292315 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v167) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff17c02b0a0 con 0x7ff18c06a750 2024-03-18T20:06:23.543 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:06:23.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.541+0000 7ff191b5e700 1 -- 172.21.15.1:0/4006292315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff16c06adb0 msgr2=0x7ff16c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:06:23.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.541+0000 7ff191b5e700 1 --2- 172.21.15.1:0/4006292315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff16c06adb0 0x7ff16c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.541+0000 7ff191b5e700 1 -- 172.21.15.1:0/4006292315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff18c06a750 msgr2=0x7ff18c150340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:23.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.541+0000 7ff191b5e700 1 --2- 172.21.15.1:0/4006292315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff18c06a750 0x7ff18c150340 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7ff17c009c40 tx=0x7ff17c004750 comp rx=0 tx=0).stop 2024-03-18T20:06:23.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.541+0000 7ff191b5e700 1 -- 172.21.15.1:0/4006292315 shutdown_connections 2024-03-18T20:06:23.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.541+0000 7ff191b5e700 1 --2- 172.21.15.1:0/4006292315 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff16c06adb0 0x7ff16c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.541+0000 7ff191b5e700 1 --2- 172.21.15.1:0/4006292315 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff18c06f180 0x7ff18c0adf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.541+0000 7ff191b5e700 1 --2- 172.21.15.1:0/4006292315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff18c06a750 0x7ff18c150340 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.541+0000 7ff191b5e700 1 --2- 172.21.15.1:0/4006292315 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff18c06eda0 0x7ff18c0c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.541+0000 7ff191b5e700 1 -- 172.21.15.1:0/4006292315 >> 172.21.15.1:0/4006292315 conn(0x7ff18c0cff30 msgr2=0x7ff18c0cf3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:23.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.541+0000 7ff191b5e700 1 -- 172.21.15.1:0/4006292315 shutdown_connections 2024-03-18T20:06:23.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.541+0000 7ff191b5e700 1 -- 172.21.15.1:0/4006292315 wait complete. 2024-03-18T20:06:23.559 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T20:06:23.594 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:23.590+0000 7fc3c8580240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:06:23.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.718+0000 7f04e2ba9700 1 Processor -- start 2024-03-18T20:06:23.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.718+0000 7f04e2ba9700 1 -- start start 2024-03-18T20:06:23.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.719+0000 7f04e2ba9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04dc070380 0x7f04dc166130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:23.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.719+0000 7f04e2ba9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04dc0cfcc0 con 0x7f04dc15b600 2024-03-18T20:06:23.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.719+0000 7f04e2ba9700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04dc0cd030 con 0x7f04dc0cc770 2024-03-18T20:06:23.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.719+0000 7f04e2ba9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc0b3a90 con 0x7f04dc070380 2024-03-18T20:06:23.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.719+0000 7f04e0945700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f04dc15b600 0x7f04dc0ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33874/0 (socket says 172.21.15.1:33874) 2024-03-18T20:06:23.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.719+0000 7f04e0945700 1 -- 172.21.15.1:0/812794670 learned_addr learned my addr 172.21.15.1:0/812794670 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.719+0000 7f04e1146700 1 --2- 172.21.15.1:0/812794670 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04dc070380 0x7f04dc166130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.719+0000 7f04db7fe700 1 -- 172.21.15.1:0/812794670 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2959425078 0 0) 0x7f04dc0cfcc0 con 0x7f04dc15b600 2024-03-18T20:06:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.719+0000 7f04db7fe700 1 -- 172.21.15.1:0/812794670 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04cc0035b0 con 0x7f04dc15b600 2024-03-18T20:06:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.719+0000 7f04db7fe700 1 -- 172.21.15.1:0/812794670 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683878326 0 0) 0x7f04dc0cd030 con 0x7f04dc0cc770 2024-03-18T20:06:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.719+0000 7f04db7fe700 1 -- 172.21.15.1:0/812794670 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04dc0cfcc0 con 0x7f04dc0cc770 2024-03-18T20:06:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.719+0000 7f04e1146700 1 -- 172.21.15.1:0/812794670 >> v1:172.21.15.165:6789/0 conn(0x7f04dc0cc770 legacy=0x7f04dc0cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.719+0000 7f04e1146700 1 -- 172.21.15.1:0/812794670 >> v1:172.21.15.1:6789/0 conn(0x7f04dc15b600 legacy=0x7f04dc0ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.719+0000 7f04e1146700 1 -- 172.21.15.1:0/812794670 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04dc15b9e0 con 0x7f04dc070380 2024-03-18T20:06:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.720+0000 7f04e1146700 1 --2- 172.21.15.1:0/812794670 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04dc070380 0x7f04dc166130 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f04d4009dc0 tx=0x7f04d400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba017174817bf39 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.720+0000 7f04db7fe700 1 -- 172.21.15.1:0/812794670 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d400e070 con 0x7f04dc070380 2024-03-18T20:06:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.720+0000 7f04db7fe700 1 -- 172.21.15.1:0/812794670 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d400bb20 con 0x7f04dc070380 2024-03-18T20:06:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.720+0000 7f04db7fe700 1 -- 172.21.15.1:0/812794670 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d4012430 con 0x7f04dc070380 2024-03-18T20:06:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.720+0000 7f04e2ba9700 1 -- 172.21.15.1:0/812794670 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04dc070380 msgr2=0x7f04dc166130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:23.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.720+0000 7f04e2ba9700 1 --2- 172.21.15.1:0/812794670 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04dc070380 0x7f04dc166130 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f04d4009dc0 tx=0x7f04d400b330 comp rx=0 tx=0).stop 2024-03-18T20:06:23.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.720+0000 7f04e2ba9700 1 -- 172.21.15.1:0/812794670 shutdown_connections 2024-03-18T20:06:23.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.720+0000 7f04e2ba9700 1 --2- 172.21.15.1:0/812794670 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04dc070380 0x7f04dc166130 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.720+0000 7f04e2ba9700 1 -- 172.21.15.1:0/812794670 >> 172.21.15.1:0/812794670 conn(0x7f04dc0b2bb0 msgr2=0x7f04dc0b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:23.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.721+0000 7f04e2ba9700 1 -- 172.21.15.1:0/812794670 shutdown_connections 2024-03-18T20:06:23.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.721+0000 7f04e2ba9700 1 -- 172.21.15.1:0/812794670 wait complete. 2024-03-18T20:06:23.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.722+0000 7f04e2ba9700 1 Processor -- start 2024-03-18T20:06:23.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.722+0000 7f04e2ba9700 1 -- start start 2024-03-18T20:06:23.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04e2ba9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04dc070380 0x7f04dc14e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:23.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04e2ba9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04dc0cc770 0x7f04dc0cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:23.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04e2ba9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04dc15b600 0x7f04dc0cf110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:23.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04e2ba9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc0c9550 con 0x7f04dc070380 2024-03-18T20:06:23.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04e2ba9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc0c9830 con 0x7f04dc15b600 2024-03-18T20:06:23.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04e2ba9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc0c9b10 con 0x7f04dc0cc770 2024-03-18T20:06:23.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04e0945700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04dc070380 0x7f04dc14e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:23.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04e0945700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04dc070380 0x7f04dc14e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53152/0 (socket says 172.21.15.1:53152) 2024-03-18T20:06:23.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04e0945700 1 -- 172.21.15.1:0/364376175 learned_addr learned my addr 172.21.15.1:0/364376175 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:23.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04dbfff700 1 --2- 172.21.15.1:0/364376175 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04dc0cc770 0x7f04dc0cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:23.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04e1146700 1 --2- 172.21.15.1:0/364376175 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04dc15b600 0x7f04dc0cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:23.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04e0945700 1 -- 172.21.15.1:0/364376175 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04dc0cc770 msgr2=0x7f04dc0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:23.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04e0945700 1 --2- 172.21.15.1:0/364376175 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04dc0cc770 0x7f04dc0cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04e0945700 1 -- 172.21.15.1:0/364376175 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04dc15b600 msgr2=0x7f04dc0cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:23.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04e0945700 1 --2- 172.21.15.1:0/364376175 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04dc15b600 0x7f04dc0cf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04e0945700 1 -- 172.21.15.1:0/364376175 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d4009a70 con 0x7f04dc070380 2024-03-18T20:06:23.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04e0945700 1 --2- 172.21.15.1:0/364376175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04dc070380 0x7f04dc14e390 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f04d0001eb0 tx=0x7f04d000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:23.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04d97fa700 1 -- 172.21.15.1:0/364376175 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0005d70 con 0x7f04dc070380 2024-03-18T20:06:23.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04d97fa700 1 -- 172.21.15.1:0/364376175 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d00070d0 con 0x7f04dc070380 2024-03-18T20:06:23.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04d97fa700 1 -- 172.21.15.1:0/364376175 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0007460 con 0x7f04dc070380 2024-03-18T20:06:23.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.723+0000 7f04e2ba9700 1 -- 172.21.15.1:0/364376175 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04dc0c9df0 con 0x7f04dc070380 2024-03-18T20:06:23.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.724+0000 7f04e2ba9700 1 -- 172.21.15.1:0/364376175 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04dc14e8d0 con 0x7f04dc070380 2024-03-18T20:06:23.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.724+0000 7f04e2ba9700 1 -- 172.21.15.1:0/364376175 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04dc0ca310 con 0x7f04dc070380 2024-03-18T20:06:23.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.726+0000 7f04d97fa700 1 -- 172.21.15.1:0/364376175 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 91) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f04d00075c0 con 0x7f04dc070380 2024-03-18T20:06:23.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.726+0000 7f04d97fa700 1 --2- 172.21.15.1:0/364376175 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f04c006adb0 0x7f04c006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:23.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.726+0000 7f04dbfff700 1 -- 172.21.15.1:0/364376175 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f04c006adb0 msgr2=0x7f04c006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:23.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.726+0000 7f04d97fa700 1 -- 172.21.15.1:0/364376175 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f04d0088b90 con 0x7f04dc070380 2024-03-18T20:06:23.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.726+0000 7f04dbfff700 1 --2- 172.21.15.1:0/364376175 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f04c006adb0 0x7f04c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:06:23.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.731+0000 7f04d97fa700 1 -- 172.21.15.1:0/364376175 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f04dc0ca310 con 0x7f04dc070380 2024-03-18T20:06:23.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.927+0000 7f04dbfff700 1 -- 172.21.15.1:0/364376175 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f04c006adb0 msgr2=0x7f04c006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:23.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.927+0000 7f04dbfff700 1 --2- 172.21.15.1:0/364376175 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f04c006adb0 0x7f04c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:06:23.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.953+0000 7f04e2ba9700 1 -- 172.21.15.1:0/364376175 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f04dc0ca310 con 0x7f04dc070380 2024-03-18T20:06:23.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.953+0000 7f04d97fa700 1 -- 172.21.15.1:0/364376175 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v167) v1 ==== 80+0+451 (secure 0 0 0) 0x7f04dc0ca310 con 0x7f04dc070380 2024-03-18T20:06:23.958 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6833/76998 2024-03-19T19:10:19.888890+0000 2024-03-18T20:06:23.958 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6832/76998 2024-03-19T19:10:19.888890+0000 2024-03-18T20:06:23.958 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/371341431 2024-03-19T19:10:19.888890+0000 2024-03-18T20:06:23.958 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1433676792 2024-03-19T19:10:19.888890+0000 2024-03-18T20:06:23.959 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/4211592802 2024-03-19T19:10:19.888890+0000 2024-03-18T20:06:23.959 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/4117009712 2024-03-19T19:10:19.888890+0000 2024-03-18T20:06:23.959 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2398817063 2024-03-19T19:10:19.888890+0000 2024-03-18T20:06:23.959 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1643985667 2024-03-19T19:10:19.888890+0000 2024-03-18T20:06:23.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.955+0000 7f04e2ba9700 1 -- 172.21.15.1:0/364376175 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f04c006adb0 msgr2=0x7f04c006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:06:23.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.955+0000 7f04e2ba9700 1 --2- 172.21.15.1:0/364376175 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f04c006adb0 0x7f04c006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.955+0000 7f04e2ba9700 1 -- 172.21.15.1:0/364376175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04dc070380 msgr2=0x7f04dc14e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:23.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.955+0000 7f04e2ba9700 1 --2- 172.21.15.1:0/364376175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04dc070380 0x7f04dc14e390 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f04d0001eb0 tx=0x7f04d000c7b0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.956+0000 7f04e2ba9700 1 -- 172.21.15.1:0/364376175 shutdown_connections 2024-03-18T20:06:23.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.956+0000 7f04e2ba9700 1 --2- 172.21.15.1:0/364376175 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f04c006adb0 0x7f04c006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.956+0000 7f04e2ba9700 1 --2- 172.21.15.1:0/364376175 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04dc15b600 0x7f04dc0cf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.956+0000 7f04e2ba9700 1 --2- 172.21.15.1:0/364376175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04dc070380 0x7f04dc14e390 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.956+0000 7f04e2ba9700 1 --2- 172.21.15.1:0/364376175 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f04dc0cc770 0x7f04dc0cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:23.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.956+0000 7f04e2ba9700 1 -- 172.21.15.1:0/364376175 >> 172.21.15.1:0/364376175 conn(0x7f04dc0b2bb0 msgr2=0x7f04dc0b2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:23.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.956+0000 7f04e2ba9700 1 -- 172.21.15.1:0/364376175 shutdown_connections 2024-03-18T20:06:23.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:23.956+0000 7f04e2ba9700 1 -- 172.21.15.1:0/364376175 wait complete. 2024-03-18T20:06:23.962 INFO:teuthology.orchestra.run.smithi001.stderr:listed 8 entries 2024-03-18T20:06:23.973 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-18T20:06:23.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:23.973 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T20:06:24.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[134141]: fuse finished with error 0 and tester_r 0 2024-03-18T20:06:24.012 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:06:24.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:24.013 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T20:06:24.058 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:24.053+0000 7f7b9e191240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:06:24.132 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:24.127+0000 7f7b9e191240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:06:24.185 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:06:24.185 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:06:24.203 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:06:24.203 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:06:24.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:24.204 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-18T20:06:24.226 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:24.221+0000 7fc3c8580240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:06:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:06:24.261 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-18T20:06:24.262 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:06:24.262 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:06:24.263 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:24.258+0000 7f7b9e191240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:06:24.281 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:06:24.281 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:06:24.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:24.281 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-18T20:06:24.302 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:24.297+0000 7fc3c8580240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:06:24.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:06:24.320 INFO:teuthology.orchestra.run.smithi001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-18T20:06:24.320 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:06:24.320 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:06:24.338 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:06:24.338 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:06:24.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:24.339 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-18T20:06:24.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:06:24.397 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-18T20:06:24.398 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:06:24.438 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:24.433+0000 7fc3c8580240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:06:24.450 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:24.445+0000 7f7b9e191240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:06:24.522 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:24.517+0000 7f7b9e191240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:06:24.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.520+0000 7f5c0933a700 1 Processor -- start 2024-03-18T20:06:24.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.520+0000 7f5c0933a700 1 -- start start 2024-03-18T20:06:24.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.521+0000 7f5c0933a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c0415d500 0x7f5c040d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:24.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.521+0000 7f5c0933a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c040d1ec0 0x7f5c040b83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:24.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.521+0000 7f5c0933a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c040cddc0 con 0x7f5c040d1300 2024-03-18T20:06:24.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.521+0000 7f5c0933a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c040d3dc0 con 0x7f5c0415d500 2024-03-18T20:06:24.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.521+0000 7f5c0933a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c040ce060 con 0x7f5c040d1ec0 2024-03-18T20:06:24.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.522+0000 7f5c02ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c0415d500 0x7f5c040d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:24.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.522+0000 7f5c02ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c0415d500 0x7f5c040d0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33346/0 (socket says 172.21.15.1:33346) 2024-03-18T20:06:24.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.522+0000 7f5c027fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5c040d1300 0x7f5c040d17b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59130/0 (socket says 172.21.15.1:59130) 2024-03-18T20:06:24.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.522+0000 7f5c02ffd700 1 -- 172.21.15.1:0/2470063036 learned_addr learned my addr 172.21.15.1:0/2470063036 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:24.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.522+0000 7f5c037fe700 1 --2- 172.21.15.1:0/2470063036 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c040d1ec0 0x7f5c040b83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:24.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.522+0000 7f5c02ffd700 1 -- 172.21.15.1:0/2470063036 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c040d1ec0 msgr2=0x7f5c040b83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:24.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.522+0000 7f5c02ffd700 1 --2- 172.21.15.1:0/2470063036 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c040d1ec0 0x7f5c040b83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:24.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.522+0000 7f5c02ffd700 1 -- 172.21.15.1:0/2470063036 >> v1:172.21.15.1:6789/0 conn(0x7f5c040d1300 legacy=0x7f5c040d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:24.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.522+0000 7f5c02ffd700 1 -- 172.21.15.1:0/2470063036 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c040ce340 con 0x7f5c0415d500 2024-03-18T20:06:24.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.522+0000 7f5c01ffb700 1 -- 172.21.15.1:0/2470063036 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727043123 0 0) 0x7f5c040cddc0 con 0x7f5c040d1300 2024-03-18T20:06:24.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.522+0000 7f5c02ffd700 1 --2- 172.21.15.1:0/2470063036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c0415d500 0x7f5c040d0dc0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f5bf40098f0 tx=0x7f5bf4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43eb003d630e94e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:24.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.522+0000 7f5c01ffb700 1 -- 172.21.15.1:0/2470063036 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4011830 con 0x7f5c0415d500 2024-03-18T20:06:24.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.522+0000 7f5c01ffb700 1 -- 172.21.15.1:0/2470063036 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf4011990 con 0x7f5c0415d500 2024-03-18T20:06:24.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.522+0000 7f5c01ffb700 1 -- 172.21.15.1:0/2470063036 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4011d20 con 0x7f5c0415d500 2024-03-18T20:06:24.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.523+0000 7f5c0933a700 1 -- 172.21.15.1:0/2470063036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c0415d500 msgr2=0x7f5c040d0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:24.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.523+0000 7f5c0933a700 1 --2- 172.21.15.1:0/2470063036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c0415d500 0x7f5c040d0dc0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f5bf40098f0 tx=0x7f5bf4009c00 comp rx=0 tx=0).stop 2024-03-18T20:06:24.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.523+0000 7f5c0933a700 1 -- 172.21.15.1:0/2470063036 shutdown_connections 2024-03-18T20:06:24.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.523+0000 7f5c0933a700 1 --2- 172.21.15.1:0/2470063036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c0415d500 0x7f5c040d0dc0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:24.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.523+0000 7f5c0933a700 1 --2- 172.21.15.1:0/2470063036 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c040d1ec0 0x7f5c040b83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:24.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.523+0000 7f5c0933a700 1 -- 172.21.15.1:0/2470063036 >> 172.21.15.1:0/2470063036 conn(0x7f5c0409c530 msgr2=0x7f5c040d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:24.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.523+0000 7f5c0933a700 1 -- 172.21.15.1:0/2470063036 shutdown_connections 2024-03-18T20:06:24.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.523+0000 7f5c0933a700 1 -- 172.21.15.1:0/2470063036 wait complete. 2024-03-18T20:06:24.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.524+0000 7f5c0933a700 1 Processor -- start 2024-03-18T20:06:24.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.524+0000 7f5c0933a700 1 -- start start 2024-03-18T20:06:24.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.525+0000 7f5c0933a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c040d1300 0x7f5c040c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:24.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.525+0000 7f5c0933a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c040d1ec0 0x7f5c040cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:24.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.525+0000 7f5c0933a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c0415d500 0x7f5c040b68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:24.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.525+0000 7f5c0933a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c040cc180 con 0x7f5c040d1300 2024-03-18T20:06:24.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.525+0000 7f5c0933a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c040cc460 con 0x7f5c040d1ec0 2024-03-18T20:06:24.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.525+0000 7f5c0933a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c040cc740 con 0x7f5c0415d500 2024-03-18T20:06:24.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.525+0000 7f5c02ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c040d1300 0x7f5c040c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:24.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.525+0000 7f5c02ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c040d1300 0x7f5c040c8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33354/0 (socket says 172.21.15.1:33354) 2024-03-18T20:06:24.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.525+0000 7f5c02ffd700 1 -- 172.21.15.1:0/1819269578 learned_addr learned my addr 172.21.15.1:0/1819269578 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:24.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.525+0000 7f5c037fe700 1 --2- 172.21.15.1:0/1819269578 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c0415d500 0x7f5c040b68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:24.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.525+0000 7f5c027fc700 1 --2- 172.21.15.1:0/1819269578 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c040d1ec0 0x7f5c040cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:24.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.526+0000 7f5c02ffd700 1 -- 172.21.15.1:0/1819269578 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c0415d500 msgr2=0x7f5c040b68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:24.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.526+0000 7f5c02ffd700 1 --2- 172.21.15.1:0/1819269578 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c0415d500 0x7f5c040b68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:24.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.526+0000 7f5c02ffd700 1 -- 172.21.15.1:0/1819269578 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c040d1ec0 msgr2=0x7f5c040cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:24.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.526+0000 7f5c02ffd700 1 --2- 172.21.15.1:0/1819269578 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c040d1ec0 0x7f5c040cb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:24.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.526+0000 7f5c02ffd700 1 -- 172.21.15.1:0/1819269578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf4009550 con 0x7f5c040d1300 2024-03-18T20:06:24.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.526+0000 7f5c02ffd700 1 --2- 172.21.15.1:0/1819269578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c040d1300 0x7f5c040c8790 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4009c40 tx=0x7f5bf4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:24.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.526+0000 7f5beb7fe700 1 -- 172.21.15.1:0/1819269578 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4004550 con 0x7f5c040d1300 2024-03-18T20:06:24.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.526+0000 7f5beb7fe700 1 -- 172.21.15.1:0/1819269578 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf401f070 con 0x7f5c040d1300 2024-03-18T20:06:24.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.526+0000 7f5beb7fe700 1 -- 172.21.15.1:0/1819269578 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf40038e0 con 0x7f5c040d1300 2024-03-18T20:06:24.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.526+0000 7f5c0933a700 1 -- 172.21.15.1:0/1819269578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c040cca20 con 0x7f5c040d1300 2024-03-18T20:06:24.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.526+0000 7f5c0933a700 1 -- 172.21.15.1:0/1819269578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c040c8cd0 con 0x7f5c040d1300 2024-03-18T20:06:24.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.527+0000 7f5c0933a700 1 -- 172.21.15.1:0/1819269578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c040c8e90 con 0x7f5c040d1300 2024-03-18T20:06:24.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.529+0000 7f5beb7fe700 1 -- 172.21.15.1:0/1819269578 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 91) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f5bf4003a40 con 0x7f5c040d1300 2024-03-18T20:06:24.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.529+0000 7f5beb7fe700 1 --2- 172.21.15.1:0/1819269578 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5bf006adb0 0x7f5bf006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:24.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.529+0000 7f5c027fc700 1 -- 172.21.15.1:0/1819269578 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5bf006adb0 msgr2=0x7f5bf006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:24.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.529+0000 7f5beb7fe700 1 -- 172.21.15.1:0/1819269578 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f5bf4021070 con 0x7f5c040d1300 2024-03-18T20:06:24.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.529+0000 7f5c027fc700 1 --2- 172.21.15.1:0/1819269578 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5bf006adb0 0x7f5bf006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:06:24.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.533+0000 7f5beb7fe700 1 -- 172.21.15.1:0/1819269578 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5bf4055ec0 con 0x7f5c040d1300 2024-03-18T20:06:24.601 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:24.596+0000 7f7b9e191240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:06:24.627 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:24.623+0000 7fc3c8580240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:06:24.678 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:24.673+0000 7f7b9e191240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:06:24.715 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:24.711+0000 7fc3c8580240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:06:24.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.730+0000 7f5c027fc700 1 -- 172.21.15.1:0/1819269578 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5bf006adb0 msgr2=0x7f5bf006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:24.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.730+0000 7f5c027fc700 1 --2- 172.21.15.1:0/1819269578 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5bf006adb0 0x7f5bf006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:06:24.756 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:24.751+0000 7f7b9e191240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:06:24.801 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:24.797+0000 7fc3c8580240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:06:24.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.807+0000 7f5c0933a700 1 -- 172.21.15.1:0/1819269578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c040676a0 con 0x7f5c040d1300 2024-03-18T20:06:24.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.807+0000 7f5beb7fe700 1 -- 172.21.15.1:0/1819269578 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f5bf4015070 con 0x7f5c040d1300 2024-03-18T20:06:24.812 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:24.812 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8608,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3436643592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3436643592},{"type":"v1","addr":"172.21.15.165:6834","nonce":3436643592}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":8611,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1969816196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1969816196},{"type":"v1","addr":"172.21.15.165:6835","nonce":1969816196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":9375,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2671786416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2671786416},{"type":"v1","addr":"172.21.15.1:6836","nonce":2671786416}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:10:04.613212+0000","modified":"2024-03-18T19:10:07.625525+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9378":{"gid":9378,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1696108841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1696108841},{"type":"v1","addr":"172.21.15.1:6837","nonce":1696108841}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-03-18T20:06:24.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.810+0000 7f5c0933a700 1 -- 172.21.15.1:0/1819269578 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5bf006adb0 msgr2=0x7f5bf006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:06:24.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.810+0000 7f5c0933a700 1 --2- 172.21.15.1:0/1819269578 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5bf006adb0 0x7f5bf006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:24.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.810+0000 7f5c0933a700 1 -- 172.21.15.1:0/1819269578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c040d1300 msgr2=0x7f5c040c8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:24.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.810+0000 7f5c0933a700 1 --2- 172.21.15.1:0/1819269578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c040d1300 0x7f5c040c8790 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4009c40 tx=0x7f5bf4004750 comp rx=0 tx=0).stop 2024-03-18T20:06:24.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.810+0000 7f5c0933a700 1 -- 172.21.15.1:0/1819269578 shutdown_connections 2024-03-18T20:06:24.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.810+0000 7f5c0933a700 1 --2- 172.21.15.1:0/1819269578 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5bf006adb0 0x7f5bf006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:24.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.810+0000 7f5c0933a700 1 --2- 172.21.15.1:0/1819269578 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c0415d500 0x7f5c040b68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:24.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.810+0000 7f5c0933a700 1 --2- 172.21.15.1:0/1819269578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c040d1300 0x7f5c040c8790 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:24.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.810+0000 7f5c0933a700 1 --2- 172.21.15.1:0/1819269578 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c040d1ec0 0x7f5c040cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:24.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.810+0000 7f5c0933a700 1 -- 172.21.15.1:0/1819269578 >> 172.21.15.1:0/1819269578 conn(0x7f5c0409c530 msgr2=0x7f5c0406b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:24.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.810+0000 7f5c0933a700 1 -- 172.21.15.1:0/1819269578 shutdown_connections 2024-03-18T20:06:24.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.810+0000 7f5c0933a700 1 -- 172.21.15.1:0/1819269578 wait complete. 2024-03-18T20:06:24.816 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 73 2024-03-18T20:06:24.827 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:06:24.887 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:24.882+0000 7fc3c8580240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:06:24.930 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:24.925+0000 7f7b9e191240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:06:24.972 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:24.967+0000 7fc3c8580240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:06:24.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.990+0000 7fa3790f6700 1 Processor -- start 2024-03-18T20:06:24.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.990+0000 7fa3790f6700 1 -- start start 2024-03-18T20:06:24.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.990+0000 7fa3790f6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3740716b0 0x7fa374071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:24.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.990+0000 7fa3790f6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa3740abd70 0x7fa37406d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:24.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.990+0000 7fa3790f6700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa374071fd0 con 0x7fa3740ae580 2024-03-18T20:06:24.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.990+0000 7fa3790f6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3740ac230 con 0x7fa3740abd70 2024-03-18T20:06:24.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.990+0000 7fa3790f6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa37406c620 con 0x7fa3740716b0 2024-03-18T20:06:24.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.991+0000 7fa37259c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa3740ae580 0x7fa37406d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59134/0 (socket says 172.21.15.1:59134) 2024-03-18T20:06:24.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.991+0000 7fa37259c700 1 -- 172.21.15.1:0/3696571152 learned_addr learned my addr 172.21.15.1:0/3696571152 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.991+0000 7fa37359e700 1 --2- 172.21.15.1:0/3696571152 >> v2:172.21.15.165:3300/0 conn(0x7fa3740abd70 0x7fa37406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.991+0000 7fa372d9d700 1 --2- 172.21.15.1:0/3696571152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3740716b0 0x7fa374071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.991+0000 7fa371d9b700 1 -- 172.21.15.1:0/3696571152 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077727338 0 0) 0x7fa374071fd0 con 0x7fa3740ae580 2024-03-18T20:06:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.991+0000 7fa371d9b700 1 -- 172.21.15.1:0/3696571152 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa35c0035b0 con 0x7fa3740ae580 2024-03-18T20:06:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.991+0000 7fa372d9d700 1 -- 172.21.15.1:0/3696571152 >> v2:172.21.15.165:3300/0 conn(0x7fa3740abd70 msgr2=0x7fa37406d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.991+0000 7fa372d9d700 1 --2- 172.21.15.1:0/3696571152 >> v2:172.21.15.165:3300/0 conn(0x7fa3740abd70 0x7fa37406d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.991+0000 7fa372d9d700 1 -- 172.21.15.1:0/3696571152 >> v1:172.21.15.1:6789/0 conn(0x7fa3740ae580 legacy=0x7fa37406d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.991+0000 7fa372d9d700 1 -- 172.21.15.1:0/3696571152 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37406c900 con 0x7fa3740716b0 2024-03-18T20:06:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.991+0000 7fa37359e700 1 --2- 172.21.15.1:0/3696571152 >> v2:172.21.15.165:3300/0 conn(0x7fa3740abd70 0x7fa37406d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.991+0000 7fa372d9d700 1 --2- 172.21.15.1:0/3696571152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3740716b0 0x7fa374071a90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa364009840 tx=0x7fa364009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38209598e5fc59f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.991+0000 7fa371d9b700 1 -- 172.21.15.1:0/3696571152 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa364011830 con 0x7fa3740716b0 2024-03-18T20:06:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.992+0000 7fa371d9b700 1 -- 172.21.15.1:0/3696571152 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa364011990 con 0x7fa3740716b0 2024-03-18T20:06:24.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.992+0000 7fa371d9b700 1 -- 172.21.15.1:0/3696571152 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa364011d20 con 0x7fa3740716b0 2024-03-18T20:06:24.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.992+0000 7fa3790f6700 1 -- 172.21.15.1:0/3696571152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3740716b0 msgr2=0x7fa374071a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:24.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.992+0000 7fa3790f6700 1 --2- 172.21.15.1:0/3696571152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3740716b0 0x7fa374071a90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa364009840 tx=0x7fa364009b50 comp rx=0 tx=0).stop 2024-03-18T20:06:24.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.992+0000 7fa3790f6700 1 -- 172.21.15.1:0/3696571152 shutdown_connections 2024-03-18T20:06:24.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.992+0000 7fa3790f6700 1 --2- 172.21.15.1:0/3696571152 >> v2:172.21.15.165:3300/0 conn(0x7fa3740abd70 0x7fa37406d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:24.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.992+0000 7fa3790f6700 1 --2- 172.21.15.1:0/3696571152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3740716b0 0x7fa374071a90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:24.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.992+0000 7fa3790f6700 1 -- 172.21.15.1:0/3696571152 >> 172.21.15.1:0/3696571152 conn(0x7fa3740cee10 msgr2=0x7fa3740cf220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:24.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.992+0000 7fa3790f6700 1 -- 172.21.15.1:0/3696571152 shutdown_connections 2024-03-18T20:06:24.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.992+0000 7fa3790f6700 1 -- 172.21.15.1:0/3696571152 wait complete. 2024-03-18T20:06:25.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.994+0000 7fa3790f6700 1 Processor -- start 2024-03-18T20:06:25.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.994+0000 7fa3790f6700 1 -- start start 2024-03-18T20:06:25.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.994+0000 7fa3790f6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3740716b0 0x7fa37415d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:25.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.994+0000 7fa3790f6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa3740abd70 0x7fa374157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:25.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.994+0000 7fa3790f6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa37409bf70 0x7fa37409c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:25.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.994+0000 7fa3790f6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37415cfd0 con 0x7fa37409bf70 2024-03-18T20:06:25.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.994+0000 7fa3790f6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa374157f40 con 0x7fa3740abd70 2024-03-18T20:06:25.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.994+0000 7fa3790f6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa37415c3c0 con 0x7fa3740716b0 2024-03-18T20:06:25.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.995+0000 7fa37359e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa37409bf70 0x7fa37409c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:25.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.995+0000 7fa37359e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa37409bf70 0x7fa37409c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33366/0 (socket says 172.21.15.1:33366) 2024-03-18T20:06:25.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.995+0000 7fa37359e700 1 -- 172.21.15.1:0/2173611238 learned_addr learned my addr 172.21.15.1:0/2173611238 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:25.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.995+0000 7fa372d9d700 1 --2- 172.21.15.1:0/2173611238 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3740716b0 0x7fa37415d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:25.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.995+0000 7fa37259c700 1 --2- 172.21.15.1:0/2173611238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa3740abd70 0x7fa374157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:25.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.995+0000 7fa372d9d700 1 -- 172.21.15.1:0/2173611238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa3740abd70 msgr2=0x7fa374157a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:25.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.995+0000 7fa372d9d700 1 --2- 172.21.15.1:0/2173611238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa3740abd70 0x7fa374157a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.995+0000 7fa372d9d700 1 -- 172.21.15.1:0/2173611238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa37409bf70 msgr2=0x7fa37409c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:25.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.995+0000 7fa372d9d700 1 --2- 172.21.15.1:0/2173611238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa37409bf70 0x7fa37409c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.995+0000 7fa372d9d700 1 -- 172.21.15.1:0/2173611238 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36c009a70 con 0x7fa3740716b0 2024-03-18T20:06:25.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.995+0000 7fa37259c700 1 --2- 172.21.15.1:0/2173611238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa3740abd70 0x7fa374157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:25.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.995+0000 7fa37359e700 1 --2- 172.21.15.1:0/2173611238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa37409bf70 0x7fa37409c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:25.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.995+0000 7fa372d9d700 1 --2- 172.21.15.1:0/2173611238 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3740716b0 0x7fa37415d460 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa364016000 tx=0x7fa364004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:25.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.995+0000 7fa35b7fe700 1 -- 172.21.15.1:0/2173611238 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa364004590 con 0x7fa3740716b0 2024-03-18T20:06:25.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.995+0000 7fa35b7fe700 1 -- 172.21.15.1:0/2173611238 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36401f070 con 0x7fa3740716b0 2024-03-18T20:06:25.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.995+0000 7fa35b7fe700 1 -- 172.21.15.1:0/2173611238 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3640038e0 con 0x7fa3740716b0 2024-03-18T20:06:25.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.995+0000 7fa3790f6700 1 -- 172.21.15.1:0/2173611238 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3640094a0 con 0x7fa3740716b0 2024-03-18T20:06:25.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.995+0000 7fa3790f6700 1 -- 172.21.15.1:0/2173611238 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa37415c980 con 0x7fa3740716b0 2024-03-18T20:06:25.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.996+0000 7fa3790f6700 1 -- 172.21.15.1:0/2173611238 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa374073350 con 0x7fa3740716b0 2024-03-18T20:06:25.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.997+0000 7fa35b7fe700 1 -- 172.21.15.1:0/2173611238 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 91) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fa364003a40 con 0x7fa3740716b0 2024-03-18T20:06:25.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.997+0000 7fa35b7fe700 1 --2- 172.21.15.1:0/2173611238 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa36006adb0 0x7fa36006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:25.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.997+0000 7fa37259c700 1 -- 172.21.15.1:0/2173611238 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa36006adb0 msgr2=0x7fa36006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:25.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.997+0000 7fa35b7fe700 1 -- 172.21.15.1:0/2173611238 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fa364021070 con 0x7fa3740716b0 2024-03-18T20:06:25.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:24.997+0000 7fa37259c700 1 --2- 172.21.15.1:0/2173611238 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa36006adb0 0x7fa36006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:06:25.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.002+0000 7fa35b7fe700 1 -- 172.21.15.1:0/2173611238 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa364055bd0 con 0x7fa3740716b0 2024-03-18T20:06:25.145 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:25.140+0000 7f7b9e191240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:06:25.154 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:25.150+0000 7fc3c8580240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:06:25.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.198+0000 7fa37259c700 1 -- 172.21.15.1:0/2173611238 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa36006adb0 msgr2=0x7fa36006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:25.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.198+0000 7fa37259c700 1 --2- 172.21.15.1:0/2173611238 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa36006adb0 0x7fa36006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:06:25.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.207+0000 7fa3790f6700 1 -- 172.21.15.1:0/2173611238 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa37409bd50 con 0x7fa3740716b0 2024-03-18T20:06:25.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.209+0000 7fa35b7fe700 1 -- 172.21.15.1:0/2173611238 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) v1 ==== 74+0+16908 (secure 0 0 0) 0x7fa364015090 con 0x7fa3740716b0 2024-03-18T20:06:25.215 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:25.215 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":168,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:06:24.855106+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:10:02.452367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-18T19:10:03.585045+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":168,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6833/76998":"2024-03-19T19:10:19.888890+0000","172.21.15.1:6832/76998":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/371341431":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/1433676792":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/4211592802":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/4117009712":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/2398817063":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/1643985667":"2024-03-19T19:10:19.888890+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:06:25.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.211+0000 7fa3790f6700 1 -- 172.21.15.1:0/2173611238 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa36006adb0 msgr2=0x7fa36006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:06:25.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.211+0000 7fa3790f6700 1 --2- 172.21.15.1:0/2173611238 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa36006adb0 0x7fa36006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.211+0000 7fa3790f6700 1 -- 172.21.15.1:0/2173611238 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3740716b0 msgr2=0x7fa37415d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:25.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.211+0000 7fa3790f6700 1 --2- 172.21.15.1:0/2173611238 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3740716b0 0x7fa37415d460 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa364016000 tx=0x7fa364004750 comp rx=0 tx=0).stop 2024-03-18T20:06:25.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.211+0000 7fa3790f6700 1 -- 172.21.15.1:0/2173611238 shutdown_connections 2024-03-18T20:06:25.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.211+0000 7fa3790f6700 1 --2- 172.21.15.1:0/2173611238 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa36006adb0 0x7fa36006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.211+0000 7fa3790f6700 1 --2- 172.21.15.1:0/2173611238 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3740716b0 0x7fa37415d460 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.211+0000 7fa3790f6700 1 --2- 172.21.15.1:0/2173611238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa37409bf70 0x7fa37409c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.211+0000 7fa3790f6700 1 --2- 172.21.15.1:0/2173611238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa3740abd70 0x7fa374157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.211+0000 7fa3790f6700 1 -- 172.21.15.1:0/2173611238 >> 172.21.15.1:0/2173611238 conn(0x7fa3740cee10 msgr2=0x7fa37406d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:25.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.211+0000 7fa3790f6700 1 -- 172.21.15.1:0/2173611238 shutdown_connections 2024-03-18T20:06:25.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.211+0000 7fa3790f6700 1 -- 172.21.15.1:0/2173611238 wait complete. 2024-03-18T20:06:25.229 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T19:10:02.452367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '160', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T19:10:03.585045+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '168', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 113, 'snap_epoch': 168, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:06:25.230 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:06:25.299 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:25.294+0000 7f7b9e191240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:06:25.372 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:25.367+0000 7f7b9e191240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:06:25.384 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:25.380+0000 7fc3c8580240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:06:25.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.395+0000 7f896f846700 1 Processor -- start 2024-03-18T20:06:25.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.396+0000 7f896f846700 1 -- start start 2024-03-18T20:06:25.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.396+0000 7f896f846700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f896806a750 0x7f896806ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:25.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.397+0000 7f896f846700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f896806d560 0x7f89680aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:25.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.397+0000 7f896f846700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f896806b070 con 0x7f89680af510 2024-03-18T20:06:25.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.397+0000 7f896f846700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f896815ef00 con 0x7f896806a750 2024-03-18T20:06:25.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.397+0000 7f896f846700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f896815f1e0 con 0x7f896806d560 2024-03-18T20:06:25.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.397+0000 7f896d5e2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f896806a750 0x7f896806ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:25.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.397+0000 7f896d5e2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f896806a750 0x7f896806ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33368/0 (socket says 172.21.15.1:33368) 2024-03-18T20:06:25.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.397+0000 7f896d5e2700 1 -- 172.21.15.1:0/1097868362 learned_addr learned my addr 172.21.15.1:0/1097868362 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.397+0000 7f896cde1700 1 --2- 172.21.15.1:0/1097868362 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f896806d560 0x7f89680aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.397+0000 7f896d5e2700 1 -- 172.21.15.1:0/1097868362 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f896806d560 msgr2=0x7f89680aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.397+0000 7f896d5e2700 1 --2- 172.21.15.1:0/1097868362 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f896806d560 0x7f89680aefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.397+0000 7f896d5e2700 1 -- 172.21.15.1:0/1097868362 >> v1:172.21.15.165:6789/0 conn(0x7f89680af510 legacy=0x7f89680715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.397+0000 7f896d5e2700 1 -- 172.21.15.1:0/1097868362 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896815f4c0 con 0x7f896806a750 2024-03-18T20:06:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.397+0000 7f896d5e2700 1 --2- 172.21.15.1:0/1097868362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f896806a750 0x7f896806ab30 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f89640098f0 tx=0x7f8964009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f23eafcc9a18413b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.397+0000 7f895ffff700 1 -- 172.21.15.1:0/1097868362 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8964011830 con 0x7f896806a750 2024-03-18T20:06:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.397+0000 7f895ffff700 1 -- 172.21.15.1:0/1097868362 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8964011990 con 0x7f896806a750 2024-03-18T20:06:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.397+0000 7f895ffff700 1 -- 172.21.15.1:0/1097868362 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8964011d20 con 0x7f896806a750 2024-03-18T20:06:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.398+0000 7f896f846700 1 -- 172.21.15.1:0/1097868362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f896806a750 msgr2=0x7f896806ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.398+0000 7f896f846700 1 --2- 172.21.15.1:0/1097868362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f896806a750 0x7f896806ab30 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f89640098f0 tx=0x7f8964009c00 comp rx=0 tx=0).stop 2024-03-18T20:06:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.398+0000 7f896f846700 1 -- 172.21.15.1:0/1097868362 shutdown_connections 2024-03-18T20:06:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.398+0000 7f896f846700 1 --2- 172.21.15.1:0/1097868362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f896806a750 0x7f896806ab30 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.398+0000 7f896f846700 1 --2- 172.21.15.1:0/1097868362 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f896806d560 0x7f89680aefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.398+0000 7f896f846700 1 -- 172.21.15.1:0/1097868362 >> 172.21.15.1:0/1097868362 conn(0x7f89680cff30 msgr2=0x7f89680d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:25.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.398+0000 7f896f846700 1 -- 172.21.15.1:0/1097868362 shutdown_connections 2024-03-18T20:06:25.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.398+0000 7f896f846700 1 -- 172.21.15.1:0/1097868362 wait complete. 2024-03-18T20:06:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.399+0000 7f896f846700 1 Processor -- start 2024-03-18T20:06:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.399+0000 7f896f846700 1 -- start start 2024-03-18T20:06:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.400+0000 7f896f846700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f896806a750 0x7f8968156450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.400+0000 7f896f846700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f896806d560 0x7f8968155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.400+0000 7f896f846700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89680af510 0x7f89680b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.400+0000 7f896f846700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89680b4520 con 0x7f89680af510 2024-03-18T20:06:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.400+0000 7f896f846700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89681562f0 con 0x7f896806a750 2024-03-18T20:06:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.400+0000 7f896f846700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89680b1010 con 0x7f896806d560 2024-03-18T20:06:25.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.400+0000 7f896dde3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89680af510 0x7f89680b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.400+0000 7f896dde3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89680af510 0x7f89680b2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33372/0 (socket says 172.21.15.1:33372) 2024-03-18T20:06:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.400+0000 7f896cde1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f896806d560 0x7f8968155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.400+0000 7f896dde3700 1 -- 172.21.15.1:0/3878832249 learned_addr learned my addr 172.21.15.1:0/3878832249 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.400+0000 7f896d5e2700 1 --2- 172.21.15.1:0/3878832249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f896806a750 0x7f8968156450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.401+0000 7f896cde1700 1 -- 172.21.15.1:0/3878832249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f896806a750 msgr2=0x7f8968156450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.401+0000 7f896cde1700 1 --2- 172.21.15.1:0/3878832249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f896806a750 0x7f8968156450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.401+0000 7f896cde1700 1 -- 172.21.15.1:0/3878832249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89680af510 msgr2=0x7f89680b2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.401+0000 7f896cde1700 1 --2- 172.21.15.1:0/3878832249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89680af510 0x7f89680b2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.401+0000 7f896cde1700 1 -- 172.21.15.1:0/3878832249 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8964009550 con 0x7f896806d560 2024-03-18T20:06:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.401+0000 7f896d5e2700 1 --2- 172.21.15.1:0/3878832249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f896806a750 0x7f8968156450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.401+0000 7f896dde3700 1 --2- 172.21.15.1:0/3878832249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89680af510 0x7f89680b2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.401+0000 7f896cde1700 1 --2- 172.21.15.1:0/3878832249 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f896806d560 0x7f8968155db0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8958009ca0 tx=0x7f8958010f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:25.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.401+0000 7f895dffb700 1 -- 172.21.15.1:0/3878832249 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8958015070 con 0x7f896806d560 2024-03-18T20:06:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.401+0000 7f895dffb700 1 -- 172.21.15.1:0/3878832249 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f895800b930 con 0x7f896806d560 2024-03-18T20:06:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.401+0000 7f896f846700 1 -- 172.21.15.1:0/3878832249 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89680b1df0 con 0x7f896806d560 2024-03-18T20:06:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.401+0000 7f895dffb700 1 -- 172.21.15.1:0/3878832249 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895800bca0 con 0x7f896806d560 2024-03-18T20:06:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.401+0000 7f896f846700 1 -- 172.21.15.1:0/3878832249 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89680b17a0 con 0x7f896806d560 2024-03-18T20:06:25.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.402+0000 7f896f846700 1 -- 172.21.15.1:0/3878832249 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f896815b480 con 0x7f896806d560 2024-03-18T20:06:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.403+0000 7f895dffb700 1 -- 172.21.15.1:0/3878832249 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 91) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f8958019410 con 0x7f896806d560 2024-03-18T20:06:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.403+0000 7f895dffb700 1 --2- 172.21.15.1:0/3878832249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f894c06adb0 0x7f894c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.403+0000 7f896d5e2700 1 -- 172.21.15.1:0/3878832249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f894c06adb0 msgr2=0x7f894c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.403+0000 7f895dffb700 1 -- 172.21.15.1:0/3878832249 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f895808cf20 con 0x7f896806d560 2024-03-18T20:06:25.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.403+0000 7f896d5e2700 1 --2- 172.21.15.1:0/3878832249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f894c06adb0 0x7f894c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:06:25.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.408+0000 7f895dffb700 1 -- 172.21.15.1:0/3878832249 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8958057d70 con 0x7f896806d560 2024-03-18T20:06:25.548 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:25.543+0000 7fc3c8580240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:06:25.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.604+0000 7f896d5e2700 1 -- 172.21.15.1:0/3878832249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f894c06adb0 msgr2=0x7f894c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:25.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.604+0000 7f896d5e2700 1 --2- 172.21.15.1:0/3878832249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f894c06adb0 0x7f894c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:06:25.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.613+0000 7f896f846700 1 -- 172.21.15.1:0/3878832249 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f89680676a0 con 0x7f896806d560 2024-03-18T20:06:25.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.613+0000 7f895dffb700 1 -- 172.21.15.1:0/3878832249 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v168) v1 ==== 93+0+10 (secure 0 0 0) 0x7f895801e020 con 0x7f896806d560 2024-03-18T20:06:25.618 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:06:25.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.616+0000 7f896f846700 1 -- 172.21.15.1:0/3878832249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f894c06adb0 msgr2=0x7f894c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:06:25.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.616+0000 7f896f846700 1 --2- 172.21.15.1:0/3878832249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f894c06adb0 0x7f894c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.616+0000 7f896f846700 1 -- 172.21.15.1:0/3878832249 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f896806d560 msgr2=0x7f8968155db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:25.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.616+0000 7f896f846700 1 --2- 172.21.15.1:0/3878832249 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f896806d560 0x7f8968155db0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8958009ca0 tx=0x7f8958010f60 comp rx=0 tx=0).stop 2024-03-18T20:06:25.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.616+0000 7f896f846700 1 -- 172.21.15.1:0/3878832249 shutdown_connections 2024-03-18T20:06:25.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.616+0000 7f896f846700 1 --2- 172.21.15.1:0/3878832249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f894c06adb0 0x7f894c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.616+0000 7f896f846700 1 --2- 172.21.15.1:0/3878832249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89680af510 0x7f89680b2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.616+0000 7f896f846700 1 --2- 172.21.15.1:0/3878832249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f896806a750 0x7f8968156450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.616+0000 7f896f846700 1 --2- 172.21.15.1:0/3878832249 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f896806d560 0x7f8968155db0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.616+0000 7f896f846700 1 -- 172.21.15.1:0/3878832249 >> 172.21.15.1:0/3878832249 conn(0x7f89680cff30 msgr2=0x7f8968159fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:25.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.616+0000 7f896f846700 1 -- 172.21.15.1:0/3878832249 shutdown_connections 2024-03-18T20:06:25.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.616+0000 7f896f846700 1 -- 172.21.15.1:0/3878832249 wait complete. 2024-03-18T20:06:25.626 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:25.621+0000 7fc3c8580240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:06:25.633 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:06:25.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.758+0000 7f01f6061700 1 Processor -- start 2024-03-18T20:06:25.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.758+0000 7f01f6061700 1 -- start start 2024-03-18T20:06:25.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.759+0000 7f01f6061700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01f015b4b0 0x7f01f00af3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:25.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.759+0000 7f01f6061700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f01f00cbf60 0x7f01f00cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:25.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.759+0000 7f01f6061700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f01f006a750 0x7f01f006abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:25.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.759+0000 7f01f6061700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f00cfbb0 con 0x7f01f006a750 2024-03-18T20:06:25.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.759+0000 7f01f6061700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f006b120 con 0x7f01f00cbf60 2024-03-18T20:06:25.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.759+0000 7f01f6061700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f00cf840 con 0x7f01f015b4b0 2024-03-18T20:06:25.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.759+0000 7f01effff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f01f006a750 0x7f01f006abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:25.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.759+0000 7f01effff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f01f006a750 0x7f01f006abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33374/0 (socket says 172.21.15.1:33374) 2024-03-18T20:06:25.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.759+0000 7f01effff700 1 -- 172.21.15.1:0/2675275816 learned_addr learned my addr 172.21.15.1:0/2675275816 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:25.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.759+0000 7f01ef7fe700 1 --2- 172.21.15.1:0/2675275816 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01f015b4b0 0x7f01f00af3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:25.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.759+0000 7f01eeffd700 1 --2- 172.21.15.1:0/2675275816 >> v2:172.21.15.165:3300/0 conn(0x7f01f00cbf60 0x7f01f00cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:25.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.760+0000 7f01effff700 1 -- 172.21.15.1:0/2675275816 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01f015b4b0 msgr2=0x7f01f00af3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:25.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.760+0000 7f01effff700 1 --2- 172.21.15.1:0/2675275816 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01f015b4b0 0x7f01f00af3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.760+0000 7f01effff700 1 -- 172.21.15.1:0/2675275816 >> v2:172.21.15.165:3300/0 conn(0x7f01f00cbf60 msgr2=0x7f01f00cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.760+0000 7f01effff700 1 --2- 172.21.15.1:0/2675275816 >> v2:172.21.15.165:3300/0 conn(0x7f01f00cbf60 0x7f01f00cc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.760+0000 7f01effff700 1 -- 172.21.15.1:0/2675275816 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f00cf5f0 con 0x7f01f006a750 2024-03-18T20:06:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.760+0000 7f01effff700 1 --2- 172.21.15.1:0/2675275816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01f006a750 0x7f01f006abe0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f01f006f590 tx=0x7f01e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4131feaeedc83369 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.760+0000 7f01ee7fc700 1 -- 172.21.15.1:0/2675275816 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e800e070 con 0x7f01f006a750 2024-03-18T20:06:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.760+0000 7f01ee7fc700 1 -- 172.21.15.1:0/2675275816 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e800b910 con 0x7f01f006a750 2024-03-18T20:06:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.760+0000 7f01ee7fc700 1 -- 172.21.15.1:0/2675275816 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e800bcc0 con 0x7f01f006a750 2024-03-18T20:06:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.761+0000 7f01f6061700 1 -- 172.21.15.1:0/2675275816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01f006a750 msgr2=0x7f01f006abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.761+0000 7f01f6061700 1 --2- 172.21.15.1:0/2675275816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01f006a750 0x7f01f006abe0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f01f006f590 tx=0x7f01e800b330 comp rx=0 tx=0).stop 2024-03-18T20:06:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.761+0000 7f01f6061700 1 -- 172.21.15.1:0/2675275816 shutdown_connections 2024-03-18T20:06:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.761+0000 7f01f6061700 1 --2- 172.21.15.1:0/2675275816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01f006a750 0x7f01f006abe0 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.761+0000 7f01f6061700 1 --2- 172.21.15.1:0/2675275816 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01f015b4b0 0x7f01f00af3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.761+0000 7f01f6061700 1 --2- 172.21.15.1:0/2675275816 >> v2:172.21.15.165:3300/0 conn(0x7f01f00cbf60 0x7f01f00cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.761+0000 7f01f6061700 1 -- 172.21.15.1:0/2675275816 >> 172.21.15.1:0/2675275816 conn(0x7f01f00cca50 msgr2=0x7f01f00cce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:25.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.761+0000 7f01f6061700 1 -- 172.21.15.1:0/2675275816 shutdown_connections 2024-03-18T20:06:25.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.761+0000 7f01f6061700 1 -- 172.21.15.1:0/2675275816 wait complete. 2024-03-18T20:06:25.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.762+0000 7f01f6061700 1 Processor -- start 2024-03-18T20:06:25.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.762+0000 7f01f6061700 1 -- start start 2024-03-18T20:06:25.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.763+0000 7f01f6061700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01f006a750 0x7f01f00b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:25.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.763+0000 7f01f6061700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01f00cbf60 0x7f01f01e60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:25.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.763+0000 7f01f6061700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01f015b4b0 0x7f01f01e65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:25.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.763+0000 7f01f6061700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f00b0c60 con 0x7f01f00cbf60 2024-03-18T20:06:25.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.763+0000 7f01f6061700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f00b0f40 con 0x7f01f015b4b0 2024-03-18T20:06:25.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.763+0000 7f01f6061700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f00b1220 con 0x7f01f006a750 2024-03-18T20:06:25.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.763+0000 7f01eeffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01f00cbf60 0x7f01f01e60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:25.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.763+0000 7f01ef7fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01f006a750 0x7f01f00b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:25.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.763+0000 7f01effff700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01f015b4b0 0x7f01f01e65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:25.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.763+0000 7f01eeffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01f00cbf60 0x7f01f01e60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33380/0 (socket says 172.21.15.1:33380) 2024-03-18T20:06:25.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.763+0000 7f01eeffd700 1 -- 172.21.15.1:0/886266272 learned_addr learned my addr 172.21.15.1:0/886266272 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:25.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.763+0000 7f01eeffd700 1 -- 172.21.15.1:0/886266272 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01f006a750 msgr2=0x7f01f00b0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:25.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.763+0000 7f01eeffd700 1 --2- 172.21.15.1:0/886266272 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01f006a750 0x7f01f00b0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.763+0000 7f01eeffd700 1 -- 172.21.15.1:0/886266272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01f015b4b0 msgr2=0x7f01f01e65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:25.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.763+0000 7f01eeffd700 1 --2- 172.21.15.1:0/886266272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01f015b4b0 0x7f01f01e65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.763+0000 7f01eeffd700 1 -- 172.21.15.1:0/886266272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e8009b30 con 0x7f01f00cbf60 2024-03-18T20:06:25.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.763+0000 7f01effff700 1 --2- 172.21.15.1:0/886266272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01f015b4b0 0x7f01f01e65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:25.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.763+0000 7f01eeffd700 1 --2- 172.21.15.1:0/886266272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01f00cbf60 0x7f01f01e60a0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f01d800e930 tx=0x7f01d800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:25.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.764+0000 7f01e7fff700 1 -- 172.21.15.1:0/886266272 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d8013680 con 0x7f01f00cbf60 2024-03-18T20:06:25.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.764+0000 7f01e7fff700 1 -- 172.21.15.1:0/886266272 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d80137e0 con 0x7f01f00cbf60 2024-03-18T20:06:25.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.764+0000 7f01f6061700 1 -- 172.21.15.1:0/886266272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f00b1500 con 0x7f01f00cbf60 2024-03-18T20:06:25.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.764+0000 7f01e7fff700 1 -- 172.21.15.1:0/886266272 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d8013b70 con 0x7f01f00cbf60 2024-03-18T20:06:25.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.764+0000 7f01f6061700 1 -- 172.21.15.1:0/886266272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f0156760 con 0x7f01f00cbf60 2024-03-18T20:06:25.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.765+0000 7f01f6061700 1 -- 172.21.15.1:0/886266272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f0067bc0 con 0x7f01f00cbf60 2024-03-18T20:06:25.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.765+0000 7f01e7fff700 1 -- 172.21.15.1:0/886266272 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 91) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f01d8013cd0 con 0x7f01f00cbf60 2024-03-18T20:06:25.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.766+0000 7f01e7fff700 1 --2- 172.21.15.1:0/886266272 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f01d006adb0 0x7f01d006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:25.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.766+0000 7f01ef7fe700 1 -- 172.21.15.1:0/886266272 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f01d006adb0 msgr2=0x7f01d006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:25.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.766+0000 7f01e7fff700 1 -- 172.21.15.1:0/886266272 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f01d80896e0 con 0x7f01f00cbf60 2024-03-18T20:06:25.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.766+0000 7f01ef7fe700 1 --2- 172.21.15.1:0/886266272 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f01d006adb0 0x7f01d006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:06:25.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.770+0000 7f01e7fff700 1 -- 172.21.15.1:0/886266272 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f01d80544a0 con 0x7f01f00cbf60 2024-03-18T20:06:25.820 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:25.815+0000 7f7b9e191240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:06:25.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.967+0000 7f01ef7fe700 1 -- 172.21.15.1:0/886266272 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f01d006adb0 msgr2=0x7f01d006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:25.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.967+0000 7f01ef7fe700 1 --2- 172.21.15.1:0/886266272 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f01d006adb0 0x7f01d006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:06:25.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.974+0000 7f01f6061700 1 -- 172.21.15.1:0/886266272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f01f0156950 con 0x7f01f00cbf60 2024-03-18T20:06:25.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.974+0000 7f01e7fff700 1 -- 172.21.15.1:0/886266272 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v168) v1 ==== 104+0+11 (secure 0 0 0) 0x7f01d8057ac0 con 0x7f01f00cbf60 2024-03-18T20:06:25.979 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:06:25.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.976+0000 7f01f6061700 1 -- 172.21.15.1:0/886266272 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f01d006adb0 msgr2=0x7f01d006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:06:25.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.976+0000 7f01f6061700 1 --2- 172.21.15.1:0/886266272 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f01d006adb0 0x7f01d006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.976+0000 7f01f6061700 1 -- 172.21.15.1:0/886266272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01f00cbf60 msgr2=0x7f01f01e60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:25.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.976+0000 7f01f6061700 1 --2- 172.21.15.1:0/886266272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01f00cbf60 0x7f01f01e60a0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f01d800e930 tx=0x7f01d800ec40 comp rx=0 tx=0).stop 2024-03-18T20:06:25.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.976+0000 7f01f6061700 1 -- 172.21.15.1:0/886266272 shutdown_connections 2024-03-18T20:06:25.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.976+0000 7f01f6061700 1 --2- 172.21.15.1:0/886266272 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f01d006adb0 0x7f01d006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.976+0000 7f01f6061700 1 --2- 172.21.15.1:0/886266272 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f01f006a750 0x7f01f00b0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.976+0000 7f01f6061700 1 --2- 172.21.15.1:0/886266272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01f015b4b0 0x7f01f01e65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.976+0000 7f01f6061700 1 --2- 172.21.15.1:0/886266272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01f00cbf60 0x7f01f01e60a0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:25.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.976+0000 7f01f6061700 1 -- 172.21.15.1:0/886266272 >> 172.21.15.1:0/886266272 conn(0x7f01f00cca50 msgr2=0x7f01f015cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:25.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.976+0000 7f01f6061700 1 -- 172.21.15.1:0/886266272 shutdown_connections 2024-03-18T20:06:25.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:25.977+0000 7f01f6061700 1 -- 172.21.15.1:0/886266272 wait complete. 2024-03-18T20:06:25.993 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:06:26.114 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:26.110+0000 7fc3c8580240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:06:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.118+0000 7fa6b3e7e700 1 Processor -- start 2024-03-18T20:06:26.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.118+0000 7fa6b3e7e700 1 -- start start 2024-03-18T20:06:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.119+0000 7fa6b3e7e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa6ac0cbd40 0x7fa6ac0cc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.119+0000 7fa6b3e7e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa6ac06b130 0x7fa6ac0cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.119+0000 7fa6b3e7e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ac0cf730 con 0x7fa6ac0c3670 2024-03-18T20:06:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.119+0000 7fa6b3e7e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6ac0cc660 con 0x7fa6ac0cbd40 2024-03-18T20:06:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.119+0000 7fa6b3e7e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6ac0cf980 con 0x7fa6ac06b130 2024-03-18T20:06:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.119+0000 7fa6b1c1a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa6ac0cbd40 0x7fa6ac0cc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.119+0000 7fa6b1c1a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa6ac0cbd40 0x7fa6ac0cc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33386/0 (socket says 172.21.15.1:33386) 2024-03-18T20:06:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.119+0000 7fa6b1c1a700 1 -- 172.21.15.1:0/884207623 learned_addr learned my addr 172.21.15.1:0/884207623 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:26.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.119+0000 7fa6b241b700 1 --2- 172.21.15.1:0/884207623 >> v2:172.21.15.165:3300/0 conn(0x7fa6ac06b130 0x7fa6ac0cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.120+0000 7fa6b1c1a700 1 -- 172.21.15.1:0/884207623 >> v2:172.21.15.165:3300/0 conn(0x7fa6ac06b130 msgr2=0x7fa6ac0cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.120+0000 7fa6b1c1a700 1 --2- 172.21.15.1:0/884207623 >> v2:172.21.15.165:3300/0 conn(0x7fa6ac06b130 0x7fa6ac0cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.120+0000 7fa6b1c1a700 1 -- 172.21.15.1:0/884207623 >> v1:172.21.15.165:6789/0 conn(0x7fa6ac0c3670 legacy=0x7fa6ac15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.120+0000 7fa6b1c1a700 1 -- 172.21.15.1:0/884207623 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac0cfc60 con 0x7fa6ac0cbd40 2024-03-18T20:06:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.120+0000 7fa6b1c1a700 1 --2- 172.21.15.1:0/884207623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac0cbd40 0x7fa6ac0cc120 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac0729e0 tx=0x7fa69c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425ccaecaaaaf888 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.120+0000 7fa6b0c18700 1 -- 172.21.15.1:0/884207623 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c00b870 con 0x7fa6ac0cbd40 2024-03-18T20:06:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.120+0000 7fa6b0c18700 1 -- 172.21.15.1:0/884207623 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69c00b9d0 con 0x7fa6ac0cbd40 2024-03-18T20:06:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.120+0000 7fa6b0c18700 1 -- 172.21.15.1:0/884207623 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c00bd60 con 0x7fa6ac0cbd40 2024-03-18T20:06:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.120+0000 7fa6b3e7e700 1 -- 172.21.15.1:0/884207623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac0cbd40 msgr2=0x7fa6ac0cc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.120+0000 7fa6b3e7e700 1 --2- 172.21.15.1:0/884207623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac0cbd40 0x7fa6ac0cc120 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac0729e0 tx=0x7fa69c009bd0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.121+0000 7fa6b3e7e700 1 -- 172.21.15.1:0/884207623 shutdown_connections 2024-03-18T20:06:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.121+0000 7fa6b3e7e700 1 --2- 172.21.15.1:0/884207623 >> v2:172.21.15.165:3300/0 conn(0x7fa6ac06b130 0x7fa6ac0cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.121+0000 7fa6b3e7e700 1 --2- 172.21.15.1:0/884207623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac0cbd40 0x7fa6ac0cc120 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.121+0000 7fa6b3e7e700 1 -- 172.21.15.1:0/884207623 >> 172.21.15.1:0/884207623 conn(0x7fa6ac0b1aa0 msgr2=0x7fa6ac0b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:26.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.121+0000 7fa6b3e7e700 1 -- 172.21.15.1:0/884207623 shutdown_connections 2024-03-18T20:06:26.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.121+0000 7fa6b3e7e700 1 -- 172.21.15.1:0/884207623 wait complete. 2024-03-18T20:06:26.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.122+0000 7fa6b3e7e700 1 Processor -- start 2024-03-18T20:06:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.122+0000 7fa6b3e7e700 1 -- start start 2024-03-18T20:06:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b3e7e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac06b130 0x7fa6ac15ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b3e7e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6ac0c3670 0x7fa6ac1f6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b3e7e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa6ac0cbd40 0x7fa6ac0c3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b3e7e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac15f970 con 0x7fa6ac06b130 2024-03-18T20:06:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b3e7e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac15f0d0 con 0x7fa6ac0c3670 2024-03-18T20:06:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b3e7e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac15f3b0 con 0x7fa6ac0cbd40 2024-03-18T20:06:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b1c1a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac06b130 0x7fa6ac15ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b1c1a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac06b130 0x7fa6ac15ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33388/0 (socket says 172.21.15.1:33388) 2024-03-18T20:06:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b1c1a700 1 -- 172.21.15.1:0/2857414991 learned_addr learned my addr 172.21.15.1:0/2857414991 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b1419700 1 --2- 172.21.15.1:0/2857414991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6ac0c3670 0x7fa6ac1f6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b241b700 1 --2- 172.21.15.1:0/2857414991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa6ac0cbd40 0x7fa6ac0c3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:26.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b1c1a700 1 -- 172.21.15.1:0/2857414991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa6ac0cbd40 msgr2=0x7fa6ac0c3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:26.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b1c1a700 1 --2- 172.21.15.1:0/2857414991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa6ac0cbd40 0x7fa6ac0c3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b1c1a700 1 -- 172.21.15.1:0/2857414991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6ac0c3670 msgr2=0x7fa6ac1f6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:26.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b1c1a700 1 --2- 172.21.15.1:0/2857414991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6ac0c3670 0x7fa6ac1f6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b1c1a700 1 -- 172.21.15.1:0/2857414991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69c009550 con 0x7fa6ac06b130 2024-03-18T20:06:26.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b241b700 1 --2- 172.21.15.1:0/2857414991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa6ac0cbd40 0x7fa6ac0c3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:26.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b1419700 1 --2- 172.21.15.1:0/2857414991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6ac0c3670 0x7fa6ac1f6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:26.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b1c1a700 1 --2- 172.21.15.1:0/2857414991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac06b130 0x7fa6ac15ea40 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fa69c000c00 tx=0x7fa69c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:26.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6a27fc700 1 -- 172.21.15.1:0/2857414991 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c004520 con 0x7fa6ac06b130 2024-03-18T20:06:26.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6a27fc700 1 -- 172.21.15.1:0/2857414991 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69c01e070 con 0x7fa6ac06b130 2024-03-18T20:06:26.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6a27fc700 1 -- 172.21.15.1:0/2857414991 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c0038a0 con 0x7fa6ac06b130 2024-03-18T20:06:26.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.123+0000 7fa6b3e7e700 1 -- 172.21.15.1:0/2857414991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ac15f690 con 0x7fa6ac06b130 2024-03-18T20:06:26.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.124+0000 7fa6b3e7e700 1 -- 172.21.15.1:0/2857414991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ac0b5130 con 0x7fa6ac06b130 2024-03-18T20:06:26.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.125+0000 7fa6b3e7e700 1 -- 172.21.15.1:0/2857414991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ac0cf580 con 0x7fa6ac06b130 2024-03-18T20:06:26.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.125+0000 7fa6a27fc700 1 -- 172.21.15.1:0/2857414991 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 91) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fa69c003a00 con 0x7fa6ac06b130 2024-03-18T20:06:26.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.125+0000 7fa6a27fc700 1 --2- 172.21.15.1:0/2857414991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa69006adb0 0x7fa69006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:26.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.126+0000 7fa6b1419700 1 -- 172.21.15.1:0/2857414991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa69006adb0 msgr2=0x7fa69006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:26.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.126+0000 7fa6a27fc700 1 -- 172.21.15.1:0/2857414991 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fa69c020070 con 0x7fa6ac06b130 2024-03-18T20:06:26.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.126+0000 7fa6b1419700 1 --2- 172.21.15.1:0/2857414991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa69006adb0 0x7fa69006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:06:26.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.130+0000 7fa6a27fc700 1 -- 172.21.15.1:0/2857414991 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa69c0bc580 con 0x7fa6ac06b130 2024-03-18T20:06:26.161 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:26.155+0000 7f7b9e191240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:06:26.324 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:26.319+0000 7f7b9e191240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:06:26.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.327+0000 7fa6b1419700 1 -- 172.21.15.1:0/2857414991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa69006adb0 msgr2=0x7fa69006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:26.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.327+0000 7fa6b1419700 1 --2- 172.21.15.1:0/2857414991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa69006adb0 0x7fa69006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:06:26.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.333+0000 7fa6b3e7e700 1 -- 172.21.15.1:0/2857414991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa6ac15e7e0 con 0x7fa6ac06b130 2024-03-18T20:06:26.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.334+0000 7fa6a27fc700 1 -- 172.21.15.1:0/2857414991 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v168) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa69c02a0a0 con 0x7fa6ac06b130 2024-03-18T20:06:26.338 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:06:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.336+0000 7fa6b3e7e700 1 -- 172.21.15.1:0/2857414991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa69006adb0 msgr2=0x7fa69006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:06:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.336+0000 7fa6b3e7e700 1 --2- 172.21.15.1:0/2857414991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa69006adb0 0x7fa69006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.336+0000 7fa6b3e7e700 1 -- 172.21.15.1:0/2857414991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac06b130 msgr2=0x7fa6ac15ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.336+0000 7fa6b3e7e700 1 --2- 172.21.15.1:0/2857414991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac06b130 0x7fa6ac15ea40 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fa69c000c00 tx=0x7fa69c004750 comp rx=0 tx=0).stop 2024-03-18T20:06:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.336+0000 7fa6b3e7e700 1 -- 172.21.15.1:0/2857414991 shutdown_connections 2024-03-18T20:06:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.336+0000 7fa6b3e7e700 1 --2- 172.21.15.1:0/2857414991 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa69006adb0 0x7fa69006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.336+0000 7fa6b3e7e700 1 --2- 172.21.15.1:0/2857414991 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa6ac0cbd40 0x7fa6ac0c3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.336+0000 7fa6b3e7e700 1 --2- 172.21.15.1:0/2857414991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6ac06b130 0x7fa6ac15ea40 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.336+0000 7fa6b3e7e700 1 --2- 172.21.15.1:0/2857414991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6ac0c3670 0x7fa6ac1f6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.336+0000 7fa6b3e7e700 1 -- 172.21.15.1:0/2857414991 >> 172.21.15.1:0/2857414991 conn(0x7fa6ac0b1aa0 msgr2=0x7fa6ac06d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.336+0000 7fa6b3e7e700 1 -- 172.21.15.1:0/2857414991 shutdown_connections 2024-03-18T20:06:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.336+0000 7fa6b3e7e700 1 -- 172.21.15.1:0/2857414991 wait complete. 2024-03-18T20:06:26.354 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:06:26.476 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:26.472+0000 7fc3c8580240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:06:26.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.476+0000 7f2c37d9f700 1 Processor -- start 2024-03-18T20:06:26.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.476+0000 7f2c37d9f700 1 -- start start 2024-03-18T20:06:26.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.477+0000 7f2c37d9f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2c30146190 0x7f2c3006a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:26.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.477+0000 7f2c37d9f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2c30146af0 0x7f2c3006ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:26.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.477+0000 7f2c37d9f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c3006ebd0 con 0x7f2c301465a0 2024-03-18T20:06:26.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.477+0000 7f2c37d9f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c3006f850 con 0x7f2c30146190 2024-03-18T20:06:26.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.477+0000 7f2c37d9f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c3006eeb0 con 0x7f2c30146af0 2024-03-18T20:06:26.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.477+0000 7f2c35b3b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2c30146190 0x7f2c3006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:26.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.477+0000 7f2c3533a700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2c301465a0 0x7f2c30071650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59150/0 (socket says 172.21.15.1:59150) 2024-03-18T20:06:26.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.477+0000 7f2c3533a700 1 -- 172.21.15.1:0/2843471534 learned_addr learned my addr 172.21.15.1:0/2843471534 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:26.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.477+0000 7f2c3633c700 1 --2- 172.21.15.1:0/2843471534 >> v2:172.21.15.165:3300/0 conn(0x7f2c30146af0 0x7f2c3006ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:26.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.478+0000 7f2c35b3b700 1 -- 172.21.15.1:0/2843471534 >> v2:172.21.15.165:3300/0 conn(0x7f2c30146af0 msgr2=0x7f2c3006ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:26.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.478+0000 7f2c35b3b700 1 --2- 172.21.15.1:0/2843471534 >> v2:172.21.15.165:3300/0 conn(0x7f2c30146af0 0x7f2c3006ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.478+0000 7f2c35b3b700 1 -- 172.21.15.1:0/2843471534 >> v1:172.21.15.1:6789/0 conn(0x7f2c301465a0 legacy=0x7f2c30071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:26.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.478+0000 7f2c35b3b700 1 -- 172.21.15.1:0/2843471534 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3006f190 con 0x7f2c30146190 2024-03-18T20:06:26.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.478+0000 7f2c35b3b700 1 --2- 172.21.15.1:0/2843471534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c30146190 0x7f2c3006a750 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f2c3006c6b0 tx=0x7f2c20009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e4853b9c3c2cfab server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:26.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.478+0000 7f2c34b39700 1 -- 172.21.15.1:0/2843471534 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c20011830 con 0x7f2c30146190 2024-03-18T20:06:26.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.478+0000 7f2c34b39700 1 -- 172.21.15.1:0/2843471534 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c20011990 con 0x7f2c30146190 2024-03-18T20:06:26.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.478+0000 7f2c34b39700 1 -- 172.21.15.1:0/2843471534 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c20011d20 con 0x7f2c30146190 2024-03-18T20:06:26.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.479+0000 7f2c37d9f700 1 -- 172.21.15.1:0/2843471534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c30146190 msgr2=0x7f2c3006a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:26.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.479+0000 7f2c37d9f700 1 --2- 172.21.15.1:0/2843471534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c30146190 0x7f2c3006a750 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f2c3006c6b0 tx=0x7f2c20009b20 comp rx=0 tx=0).stop 2024-03-18T20:06:26.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.479+0000 7f2c37d9f700 1 -- 172.21.15.1:0/2843471534 shutdown_connections 2024-03-18T20:06:26.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.479+0000 7f2c37d9f700 1 --2- 172.21.15.1:0/2843471534 >> v2:172.21.15.165:3300/0 conn(0x7f2c30146af0 0x7f2c3006ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.479+0000 7f2c37d9f700 1 --2- 172.21.15.1:0/2843471534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c30146190 0x7f2c3006a750 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.479+0000 7f2c37d9f700 1 -- 172.21.15.1:0/2843471534 >> 172.21.15.1:0/2843471534 conn(0x7f2c300b1d20 msgr2=0x7f2c300b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:26.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.479+0000 7f2c37d9f700 1 -- 172.21.15.1:0/2843471534 shutdown_connections 2024-03-18T20:06:26.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.479+0000 7f2c37d9f700 1 -- 172.21.15.1:0/2843471534 wait complete. 2024-03-18T20:06:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.480+0000 7f2c37d9f700 1 Processor -- start 2024-03-18T20:06:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.480+0000 7f2c37d9f700 1 -- start start 2024-03-18T20:06:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.481+0000 7f2c37d9f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c301465a0 0x7f2c301e2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.481+0000 7f2c37d9f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c30146af0 0x7f2c301e2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.481+0000 7f2c37d9f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c301f6410 0x7f2c300cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.481+0000 7f2c37d9f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c30142a40 con 0x7f2c301f6410 2024-03-18T20:06:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.481+0000 7f2c37d9f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3015d470 con 0x7f2c30146af0 2024-03-18T20:06:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.481+0000 7f2c37d9f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c300ce9a0 con 0x7f2c301465a0 2024-03-18T20:06:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.481+0000 7f2c3633c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c301f6410 0x7f2c300cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.481+0000 7f2c3633c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c301f6410 0x7f2c300cdcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33416/0 (socket says 172.21.15.1:33416) 2024-03-18T20:06:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.481+0000 7f2c3633c700 1 -- 172.21.15.1:0/1799192825 learned_addr learned my addr 172.21.15.1:0/1799192825 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.481+0000 7f2c3533a700 1 --2- 172.21.15.1:0/1799192825 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c30146af0 0x7f2c301e2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.481+0000 7f2c35b3b700 1 --2- 172.21.15.1:0/1799192825 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c301465a0 0x7f2c301e2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.481+0000 7f2c3633c700 1 -- 172.21.15.1:0/1799192825 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c301465a0 msgr2=0x7f2c301e2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.481+0000 7f2c3633c700 1 --2- 172.21.15.1:0/1799192825 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c301465a0 0x7f2c301e2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.481+0000 7f2c3633c700 1 -- 172.21.15.1:0/1799192825 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c30146af0 msgr2=0x7f2c301e2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.481+0000 7f2c3633c700 1 --2- 172.21.15.1:0/1799192825 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c30146af0 0x7f2c301e2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.481+0000 7f2c3633c700 1 -- 172.21.15.1:0/1799192825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c200094a0 con 0x7f2c301f6410 2024-03-18T20:06:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.481+0000 7f2c3633c700 1 --2- 172.21.15.1:0/1799192825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c301f6410 0x7f2c300cdcb0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f2c28007e60 tx=0x7f2c2800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.482+0000 7f2c267fc700 1 -- 172.21.15.1:0/1799192825 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2800ed30 con 0x7f2c301f6410 2024-03-18T20:06:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.482+0000 7f2c267fc700 1 -- 172.21.15.1:0/1799192825 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c28014420 con 0x7f2c301f6410 2024-03-18T20:06:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.482+0000 7f2c37d9f700 1 -- 172.21.15.1:0/1799192825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c300cec80 con 0x7f2c301f6410 2024-03-18T20:06:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.482+0000 7f2c267fc700 1 -- 172.21.15.1:0/1799192825 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c280147d0 con 0x7f2c301f6410 2024-03-18T20:06:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.482+0000 7f2c37d9f700 1 -- 172.21.15.1:0/1799192825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c300cf240 con 0x7f2c301f6410 2024-03-18T20:06:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.483+0000 7f2c37d9f700 1 -- 172.21.15.1:0/1799192825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c300b62e0 con 0x7f2c301f6410 2024-03-18T20:06:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.483+0000 7f2c267fc700 1 -- 172.21.15.1:0/1799192825 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 91) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f2c28014930 con 0x7f2c301f6410 2024-03-18T20:06:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.484+0000 7f2c267fc700 1 --2- 172.21.15.1:0/1799192825 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2c1406adb0 0x7f2c1406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.484+0000 7f2c35b3b700 1 -- 172.21.15.1:0/1799192825 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2c1406adb0 msgr2=0x7f2c1406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.484+0000 7f2c267fc700 1 -- 172.21.15.1:0/1799192825 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f2c28089dd0 con 0x7f2c301f6410 2024-03-18T20:06:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.484+0000 7f2c35b3b700 1 --2- 172.21.15.1:0/1799192825 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2c1406adb0 0x7f2c1406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:06:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.488+0000 7f2c267fc700 1 -- 172.21.15.1:0/1799192825 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2c28054c40 con 0x7f2c301f6410 2024-03-18T20:06:26.653 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:26.648+0000 7fc3c8580240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:06:26.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.685+0000 7f2c35b3b700 1 -- 172.21.15.1:0/1799192825 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2c1406adb0 msgr2=0x7f2c1406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:26.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.685+0000 7f2c35b3b700 1 --2- 172.21.15.1:0/1799192825 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2c1406adb0 0x7f2c1406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:06:26.713 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:26.707+0000 7f7b9e191240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:06:26.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.737+0000 7f2c37d9f700 1 -- 172.21.15.1:0/1799192825 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c300676a0 con 0x7f2c301f6410 2024-03-18T20:06:26.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.737+0000 7f2c267fc700 1 -- 172.21.15.1:0/1799192825 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f2c28058260 con 0x7f2c301f6410 2024-03-18T20:06:26.742 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:26.742 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8608,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3436643592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3436643592},{"type":"v1","addr":"172.21.15.165:6834","nonce":3436643592}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":8611,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1969816196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1969816196},{"type":"v1","addr":"172.21.15.165:6835","nonce":1969816196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":9375,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2671786416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2671786416},{"type":"v1","addr":"172.21.15.1:6836","nonce":2671786416}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:10:04.613212+0000","modified":"2024-03-18T19:10:07.625525+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9378":{"gid":9378,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1696108841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1696108841},{"type":"v1","addr":"172.21.15.1:6837","nonce":1696108841}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-03-18T20:06:26.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.739+0000 7f2c37d9f700 1 -- 172.21.15.1:0/1799192825 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2c1406adb0 msgr2=0x7f2c1406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:06:26.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.739+0000 7f2c37d9f700 1 --2- 172.21.15.1:0/1799192825 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2c1406adb0 0x7f2c1406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.739+0000 7f2c37d9f700 1 -- 172.21.15.1:0/1799192825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c301f6410 msgr2=0x7f2c300cdcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:26.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.739+0000 7f2c37d9f700 1 --2- 172.21.15.1:0/1799192825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c301f6410 0x7f2c300cdcb0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f2c28007e60 tx=0x7f2c2800e3c0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.739+0000 7f2c37d9f700 1 -- 172.21.15.1:0/1799192825 shutdown_connections 2024-03-18T20:06:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.739+0000 7f2c37d9f700 1 --2- 172.21.15.1:0/1799192825 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2c1406adb0 0x7f2c1406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.739+0000 7f2c37d9f700 1 --2- 172.21.15.1:0/1799192825 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c301465a0 0x7f2c301e2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.739+0000 7f2c37d9f700 1 --2- 172.21.15.1:0/1799192825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c301f6410 0x7f2c300cdcb0 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.739+0000 7f2c37d9f700 1 --2- 172.21.15.1:0/1799192825 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c30146af0 0x7f2c301e2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.739+0000 7f2c37d9f700 1 -- 172.21.15.1:0/1799192825 >> 172.21.15.1:0/1799192825 conn(0x7f2c300b1d20 msgr2=0x7f2c300b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.739+0000 7f2c37d9f700 1 -- 172.21.15.1:0/1799192825 shutdown_connections 2024-03-18T20:06:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.739+0000 7f2c37d9f700 1 -- 172.21.15.1:0/1799192825 wait complete. 2024-03-18T20:06:26.745 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 73 2024-03-18T20:06:26.757 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:06:26.877 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:26.872+0000 7f7b9e191240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:06:26.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.879+0000 7fc58a05c700 1 Processor -- start 2024-03-18T20:06:26.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.879+0000 7fc58a05c700 1 -- start start 2024-03-18T20:06:26.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.880+0000 7fc58a05c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc58415d500 0x7fc5840d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:26.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.880+0000 7fc58a05c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5840d1ec0 0x7fc5840b83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:26.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.880+0000 7fc58a05c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5840cddc0 con 0x7fc5840d1300 2024-03-18T20:06:26.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.880+0000 7fc58a05c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5840d3dc0 con 0x7fc58415d500 2024-03-18T20:06:26.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.880+0000 7fc58a05c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5840ce060 con 0x7fc5840d1ec0 2024-03-18T20:06:26.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.880+0000 7fc5837fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc58415d500 0x7fc5840d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:26.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.880+0000 7fc5837fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc58415d500 0x7fc5840d0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33422/0 (socket says 172.21.15.1:33422) 2024-03-18T20:06:26.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.880+0000 7fc582ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc5840d1300 0x7fc5840d17b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59152/0 (socket says 172.21.15.1:59152) 2024-03-18T20:06:26.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.880+0000 7fc5837fe700 1 -- 172.21.15.1:0/1701087545 learned_addr learned my addr 172.21.15.1:0/1701087545 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:26.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.880+0000 7fc583fff700 1 --2- 172.21.15.1:0/1701087545 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5840d1ec0 0x7fc5840b83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:26.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.880+0000 7fc5837fe700 1 -- 172.21.15.1:0/1701087545 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5840d1ec0 msgr2=0x7fc5840b83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:26.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.880+0000 7fc5837fe700 1 --2- 172.21.15.1:0/1701087545 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5840d1ec0 0x7fc5840b83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.880+0000 7fc5837fe700 1 -- 172.21.15.1:0/1701087545 >> v1:172.21.15.1:6789/0 conn(0x7fc5840d1300 legacy=0x7fc5840d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:26.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.880+0000 7fc5827fc700 1 -- 172.21.15.1:0/1701087545 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619086963 0 0) 0x7fc5840cddc0 con 0x7fc5840d1300 2024-03-18T20:06:26.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.880+0000 7fc5837fe700 1 -- 172.21.15.1:0/1701087545 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5840ce340 con 0x7fc58415d500 2024-03-18T20:06:26.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.880+0000 7fc5837fe700 1 --2- 172.21.15.1:0/1701087545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc58415d500 0x7fc5840d0dc0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7fc5740098f0 tx=0x7fc574009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=923ba4fe4aae4e9c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:26.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.881+0000 7fc5827fc700 1 -- 172.21.15.1:0/1701087545 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc574011830 con 0x7fc58415d500 2024-03-18T20:06:26.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.881+0000 7fc5827fc700 1 -- 172.21.15.1:0/1701087545 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc574011990 con 0x7fc58415d500 2024-03-18T20:06:26.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.881+0000 7fc5827fc700 1 -- 172.21.15.1:0/1701087545 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc574011d20 con 0x7fc58415d500 2024-03-18T20:06:26.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.881+0000 7fc58a05c700 1 -- 172.21.15.1:0/1701087545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc58415d500 msgr2=0x7fc5840d0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:26.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.881+0000 7fc58a05c700 1 --2- 172.21.15.1:0/1701087545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc58415d500 0x7fc5840d0dc0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7fc5740098f0 tx=0x7fc574009c00 comp rx=0 tx=0).stop 2024-03-18T20:06:26.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.881+0000 7fc58a05c700 1 -- 172.21.15.1:0/1701087545 shutdown_connections 2024-03-18T20:06:26.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.881+0000 7fc58a05c700 1 --2- 172.21.15.1:0/1701087545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc58415d500 0x7fc5840d0dc0 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.881+0000 7fc58a05c700 1 --2- 172.21.15.1:0/1701087545 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5840d1ec0 0x7fc5840b83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.881+0000 7fc58a05c700 1 -- 172.21.15.1:0/1701087545 >> 172.21.15.1:0/1701087545 conn(0x7fc58409c530 msgr2=0x7fc5840d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:26.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.881+0000 7fc58a05c700 1 -- 172.21.15.1:0/1701087545 shutdown_connections 2024-03-18T20:06:26.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.881+0000 7fc58a05c700 1 -- 172.21.15.1:0/1701087545 wait complete. 2024-03-18T20:06:26.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.882+0000 7fc58a05c700 1 Processor -- start 2024-03-18T20:06:26.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.883+0000 7fc58a05c700 1 -- start start 2024-03-18T20:06:26.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.883+0000 7fc58a05c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5840d1300 0x7fc5840c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:26.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.883+0000 7fc58a05c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5840d1ec0 0x7fc5840cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:26.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.883+0000 7fc58a05c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc58415d500 0x7fc5840b68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:26.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.883+0000 7fc58a05c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5840cc180 con 0x7fc5840d1ec0 2024-03-18T20:06:26.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.883+0000 7fc58a05c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5840cc460 con 0x7fc58415d500 2024-03-18T20:06:26.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.883+0000 7fc58a05c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5840cc740 con 0x7fc5840d1300 2024-03-18T20:06:26.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.884+0000 7fc582ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5840d1ec0 0x7fc5840cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:26.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.884+0000 7fc582ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5840d1ec0 0x7fc5840cb7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33438/0 (socket says 172.21.15.1:33438) 2024-03-18T20:06:26.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.884+0000 7fc582ffd700 1 -- 172.21.15.1:0/4269307351 learned_addr learned my addr 172.21.15.1:0/4269307351 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:26.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.884+0000 7fc5837fe700 1 --2- 172.21.15.1:0/4269307351 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5840d1300 0x7fc5840c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:26.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.884+0000 7fc583fff700 1 --2- 172.21.15.1:0/4269307351 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc58415d500 0x7fc5840b68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:26.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.884+0000 7fc582ffd700 1 -- 172.21.15.1:0/4269307351 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5840d1300 msgr2=0x7fc5840c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:26.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.884+0000 7fc582ffd700 1 --2- 172.21.15.1:0/4269307351 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5840d1300 0x7fc5840c8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.884+0000 7fc582ffd700 1 -- 172.21.15.1:0/4269307351 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc58415d500 msgr2=0x7fc5840b68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:26.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.884+0000 7fc582ffd700 1 --2- 172.21.15.1:0/4269307351 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc58415d500 0x7fc5840b68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:26.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.884+0000 7fc582ffd700 1 -- 172.21.15.1:0/4269307351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc574009550 con 0x7fc5840d1ec0 2024-03-18T20:06:26.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.884+0000 7fc582ffd700 1 --2- 172.21.15.1:0/4269307351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5840d1ec0 0x7fc5840cb7a0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7fc57800c550 tx=0x7fc57800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:26.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.884+0000 7fc56bfff700 1 -- 172.21.15.1:0/4269307351 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc578004c80 con 0x7fc5840d1ec0 2024-03-18T20:06:26.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.884+0000 7fc56bfff700 1 -- 172.21.15.1:0/4269307351 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc578004e00 con 0x7fc5840d1ec0 2024-03-18T20:06:26.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.884+0000 7fc58a05c700 1 -- 172.21.15.1:0/4269307351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5840cca20 con 0x7fc5840d1ec0 2024-03-18T20:06:26.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.884+0000 7fc56bfff700 1 -- 172.21.15.1:0/4269307351 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc578007310 con 0x7fc5840d1ec0 2024-03-18T20:06:26.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.884+0000 7fc58a05c700 1 -- 172.21.15.1:0/4269307351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5840c8cd0 con 0x7fc5840d1ec0 2024-03-18T20:06:26.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.885+0000 7fc58a05c700 1 -- 172.21.15.1:0/4269307351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5840676a0 con 0x7fc5840d1ec0 2024-03-18T20:06:26.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.886+0000 7fc56bfff700 1 -- 172.21.15.1:0/4269307351 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 91) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fc578007470 con 0x7fc5840d1ec0 2024-03-18T20:06:26.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.886+0000 7fc56bfff700 1 --2- 172.21.15.1:0/4269307351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc57006adb0 0x7fc57006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:26.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.887+0000 7fc5837fe700 1 -- 172.21.15.1:0/4269307351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc57006adb0 msgr2=0x7fc57006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:26.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.887+0000 7fc56bfff700 1 -- 172.21.15.1:0/4269307351 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fc578089900 con 0x7fc5840d1ec0 2024-03-18T20:06:26.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.887+0000 7fc5837fe700 1 --2- 172.21.15.1:0/4269307351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc57006adb0 0x7fc57006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:06:26.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:26.890+0000 7fc56bfff700 1 -- 172.21.15.1:0/4269307351 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc5780547a0 con 0x7fc5840d1ec0 2024-03-18T20:06:27.073 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:27.069+0000 7fc3c8580240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:06:27.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.088+0000 7fc5837fe700 1 -- 172.21.15.1:0/4269307351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc57006adb0 msgr2=0x7fc57006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:27.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.088+0000 7fc5837fe700 1 --2- 172.21.15.1:0/4269307351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc57006adb0 0x7fc57006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:06:27.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.094+0000 7fc58a05c700 1 -- 172.21.15.1:0/4269307351 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5840cd120 con 0x7fc5840d1ec0 2024-03-18T20:06:27.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.096+0000 7fc56bfff700 1 -- 172.21.15.1:0/4269307351 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) v1 ==== 74+0+16908 (secure 0 0 0) 0x7fc578057dc0 con 0x7fc5840d1ec0 2024-03-18T20:06:27.101 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:27.101 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":168,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:06:24.855106+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:10:02.452367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-18T19:10:03.585045+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":168,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6833/76998":"2024-03-19T19:10:19.888890+0000","172.21.15.1:6832/76998":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/371341431":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/1433676792":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/4211592802":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/4117009712":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/2398817063":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/1643985667":"2024-03-19T19:10:19.888890+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":20,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:06:27.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.098+0000 7fc58a05c700 1 -- 172.21.15.1:0/4269307351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc57006adb0 msgr2=0x7fc57006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:06:27.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.098+0000 7fc58a05c700 1 --2- 172.21.15.1:0/4269307351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc57006adb0 0x7fc57006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.098+0000 7fc58a05c700 1 -- 172.21.15.1:0/4269307351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5840d1ec0 msgr2=0x7fc5840cb7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:27.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.098+0000 7fc58a05c700 1 --2- 172.21.15.1:0/4269307351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5840d1ec0 0x7fc5840cb7a0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7fc57800c550 tx=0x7fc57800c860 comp rx=0 tx=0).stop 2024-03-18T20:06:27.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.098+0000 7fc58a05c700 1 -- 172.21.15.1:0/4269307351 shutdown_connections 2024-03-18T20:06:27.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.098+0000 7fc58a05c700 1 --2- 172.21.15.1:0/4269307351 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc57006adb0 0x7fc57006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.098+0000 7fc58a05c700 1 --2- 172.21.15.1:0/4269307351 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5840d1300 0x7fc5840c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.098+0000 7fc58a05c700 1 --2- 172.21.15.1:0/4269307351 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc58415d500 0x7fc5840b68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.098+0000 7fc58a05c700 1 --2- 172.21.15.1:0/4269307351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5840d1ec0 0x7fc5840cb7a0 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.098+0000 7fc58a05c700 1 -- 172.21.15.1:0/4269307351 >> 172.21.15.1:0/4269307351 conn(0x7fc58409c530 msgr2=0x7fc58406b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:27.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.098+0000 7fc58a05c700 1 -- 172.21.15.1:0/4269307351 shutdown_connections 2024-03-18T20:06:27.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.098+0000 7fc58a05c700 1 -- 172.21.15.1:0/4269307351 wait complete. 2024-03-18T20:06:27.116 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T20:06:27.116 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:06:27.241 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:27.236+0000 7f7b9e191240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:06:27.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.239+0000 7f0cab34f700 1 Processor -- start 2024-03-18T20:06:27.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.239+0000 7f0cab34f700 1 -- start start 2024-03-18T20:06:27.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.240+0000 7f0cab34f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ca4099a40 0x7f0ca406ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:27.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.240+0000 7f0cab34f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0ca40d05a0 0x7f0ca4099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:27.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.240+0000 7f0cab34f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca40cf7e0 con 0x7f0ca409a360 2024-03-18T20:06:27.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.240+0000 7f0cab34f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca40cfac0 con 0x7f0ca40d05a0 2024-03-18T20:06:27.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.240+0000 7f0cab34f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca40cfda0 con 0x7f0ca4099a40 2024-03-18T20:06:27.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.240+0000 7f0ca88ea700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0ca40d05a0 0x7f0ca4099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:27.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.240+0000 7f0ca88ea700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0ca40d05a0 0x7f0ca4099e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33454/0 (socket says 172.21.15.1:33454) 2024-03-18T20:06:27.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.240+0000 7f0ca88ea700 1 -- 172.21.15.1:0/1253561975 learned_addr learned my addr 172.21.15.1:0/1253561975 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:27.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.240+0000 7f0ca98ec700 1 --1- 172.21.15.1:0/1253561975 >> v1:172.21.15.1:6789/0 conn(0x7f0ca409a360 0x7f0ca406e880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59154/0 (socket says 172.21.15.1:59154) 2024-03-18T20:06:27.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.240+0000 7f0ca90eb700 1 --2- 172.21.15.1:0/1253561975 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ca4099a40 0x7f0ca406ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:27.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.241+0000 7f0ca88ea700 1 -- 172.21.15.1:0/1253561975 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ca4099a40 msgr2=0x7f0ca406ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:27.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.241+0000 7f0ca88ea700 1 --2- 172.21.15.1:0/1253561975 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ca4099a40 0x7f0ca406ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.241+0000 7f0ca88ea700 1 -- 172.21.15.1:0/1253561975 >> v1:172.21.15.1:6789/0 conn(0x7f0ca409a360 legacy=0x7f0ca406e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:27.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.241+0000 7f0ca88ea700 1 -- 172.21.15.1:0/1253561975 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca40d0080 con 0x7f0ca40d05a0 2024-03-18T20:06:27.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.241+0000 7f0c9bfff700 1 -- 172.21.15.1:0/1253561975 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233619144 0 0) 0x7f0ca40cf7e0 con 0x7f0ca409a360 2024-03-18T20:06:27.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.241+0000 7f0ca88ea700 1 --2- 172.21.15.1:0/1253561975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca40d05a0 0x7f0ca4099e20 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f0c940098f0 tx=0x7f0c94009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c82b0265df6ea3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:27.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.241+0000 7f0c9bfff700 1 -- 172.21.15.1:0/1253561975 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c94011830 con 0x7f0ca40d05a0 2024-03-18T20:06:27.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.241+0000 7f0c9bfff700 1 -- 172.21.15.1:0/1253561975 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c94011990 con 0x7f0ca40d05a0 2024-03-18T20:06:27.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.241+0000 7f0c9bfff700 1 -- 172.21.15.1:0/1253561975 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c94011d20 con 0x7f0ca40d05a0 2024-03-18T20:06:27.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.242+0000 7f0cab34f700 1 -- 172.21.15.1:0/1253561975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca40d05a0 msgr2=0x7f0ca4099e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:27.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.242+0000 7f0cab34f700 1 --2- 172.21.15.1:0/1253561975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca40d05a0 0x7f0ca4099e20 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f0c940098f0 tx=0x7f0c94009c00 comp rx=0 tx=0).stop 2024-03-18T20:06:27.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.242+0000 7f0cab34f700 1 -- 172.21.15.1:0/1253561975 shutdown_connections 2024-03-18T20:06:27.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.242+0000 7f0cab34f700 1 --2- 172.21.15.1:0/1253561975 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ca4099a40 0x7f0ca406ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.242+0000 7f0cab34f700 1 --2- 172.21.15.1:0/1253561975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca40d05a0 0x7f0ca4099e20 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.242+0000 7f0cab34f700 1 -- 172.21.15.1:0/1253561975 >> 172.21.15.1:0/1253561975 conn(0x7f0ca40b2cb0 msgr2=0x7f0ca40b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:27.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.242+0000 7f0cab34f700 1 -- 172.21.15.1:0/1253561975 shutdown_connections 2024-03-18T20:06:27.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.242+0000 7f0cab34f700 1 -- 172.21.15.1:0/1253561975 wait complete. 2024-03-18T20:06:27.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.243+0000 7f0cab34f700 1 Processor -- start 2024-03-18T20:06:27.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.243+0000 7f0cab34f700 1 -- start start 2024-03-18T20:06:27.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.244+0000 7f0cab34f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ca4099a40 0x7f0ca415c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:27.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.244+0000 7f0cab34f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca409a360 0x7f0ca41587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:27.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.244+0000 7f0cab34f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ca40d05a0 0x7f0ca4150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:27.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.244+0000 7f0cab34f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca41591f0 con 0x7f0ca409a360 2024-03-18T20:06:27.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.244+0000 7f0cab34f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca41594d0 con 0x7f0ca40d05a0 2024-03-18T20:06:27.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.244+0000 7f0cab34f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca41597b0 con 0x7f0ca4099a40 2024-03-18T20:06:27.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.244+0000 7f0ca88ea700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca409a360 0x7f0ca41587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:27.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.244+0000 7f0ca88ea700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca409a360 0x7f0ca41587a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33462/0 (socket says 172.21.15.1:33462) 2024-03-18T20:06:27.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.244+0000 7f0ca88ea700 1 -- 172.21.15.1:0/1454356150 learned_addr learned my addr 172.21.15.1:0/1454356150 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:27.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.244+0000 7f0ca90eb700 1 --2- 172.21.15.1:0/1454356150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ca4099a40 0x7f0ca415c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:27.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.244+0000 7f0ca98ec700 1 --2- 172.21.15.1:0/1454356150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ca40d05a0 0x7f0ca4150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:27.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.244+0000 7f0ca88ea700 1 -- 172.21.15.1:0/1454356150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ca4099a40 msgr2=0x7f0ca415c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:27.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.244+0000 7f0ca88ea700 1 --2- 172.21.15.1:0/1454356150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ca4099a40 0x7f0ca415c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.244+0000 7f0ca88ea700 1 -- 172.21.15.1:0/1454356150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ca40d05a0 msgr2=0x7f0ca4150190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:27.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.244+0000 7f0ca88ea700 1 --2- 172.21.15.1:0/1454356150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ca40d05a0 0x7f0ca4150190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.244+0000 7f0ca88ea700 1 -- 172.21.15.1:0/1454356150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c94009550 con 0x7f0ca409a360 2024-03-18T20:06:27.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.244+0000 7f0ca88ea700 1 --2- 172.21.15.1:0/1454356150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca409a360 0x7f0ca41587a0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f0c94009130 tx=0x7f0c94004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:27.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.244+0000 7f0c99ffb700 1 -- 172.21.15.1:0/1454356150 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c94004510 con 0x7f0ca409a360 2024-03-18T20:06:27.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.245+0000 7f0c99ffb700 1 -- 172.21.15.1:0/1454356150 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c9401e070 con 0x7f0ca409a360 2024-03-18T20:06:27.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.245+0000 7f0c99ffb700 1 -- 172.21.15.1:0/1454356150 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c940038c0 con 0x7f0ca409a360 2024-03-18T20:06:27.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.245+0000 7f0cab34f700 1 -- 172.21.15.1:0/1454356150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca409a950 con 0x7f0ca409a360 2024-03-18T20:06:27.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.245+0000 7f0cab34f700 1 -- 172.21.15.1:0/1454356150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca4159d70 con 0x7f0ca409a360 2024-03-18T20:06:27.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.246+0000 7f0cab34f700 1 -- 172.21.15.1:0/1454356150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca4068240 con 0x7f0ca409a360 2024-03-18T20:06:27.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.246+0000 7f0c99ffb700 1 -- 172.21.15.1:0/1454356150 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 91) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f0c94003a20 con 0x7f0ca409a360 2024-03-18T20:06:27.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.247+0000 7f0c99ffb700 1 --2- 172.21.15.1:0/1454356150 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c8806adb0 0x7f0c8806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:27.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.247+0000 7f0ca90eb700 1 -- 172.21.15.1:0/1454356150 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c8806adb0 msgr2=0x7f0c8806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:27.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.247+0000 7f0c99ffb700 1 -- 172.21.15.1:0/1454356150 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f0c9408a650 con 0x7f0ca409a360 2024-03-18T20:06:27.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.247+0000 7f0ca90eb700 1 --2- 172.21.15.1:0/1454356150 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c8806adb0 0x7f0c8806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:06:27.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.256+0000 7f0c99ffb700 1 -- 172.21.15.1:0/1454356150 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0c94055420 con 0x7f0ca409a360 2024-03-18T20:06:27.261 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:27.257+0000 7fc3c8580240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:06:27.329 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:27.324+0000 7f7b9e191240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:06:27.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.448+0000 7f0ca90eb700 1 -- 172.21.15.1:0/1454356150 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c8806adb0 msgr2=0x7f0c8806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:27.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.448+0000 7f0ca90eb700 1 --2- 172.21.15.1:0/1454356150 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c8806adb0 0x7f0c8806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:06:27.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.506+0000 7f0cab34f700 1 -- 172.21.15.1:0/1454356150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ca40cda40 con 0x7f0ca409a360 2024-03-18T20:06:27.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.506+0000 7f0c99ffb700 1 -- 172.21.15.1:0/1454356150 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f0c94058a40 con 0x7f0ca409a360 2024-03-18T20:06:27.511 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:27.511 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8608,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3436643592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3436643592},{"type":"v1","addr":"172.21.15.165:6834","nonce":3436643592}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":8611,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1969816196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1969816196},{"type":"v1","addr":"172.21.15.165:6835","nonce":1969816196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":9375,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2671786416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2671786416},{"type":"v1","addr":"172.21.15.1:6836","nonce":2671786416}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:10:04.613212+0000","modified":"2024-03-18T19:10:07.625525+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9378":{"gid":9378,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1696108841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1696108841},{"type":"v1","addr":"172.21.15.1:6837","nonce":1696108841}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-03-18T20:06:27.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.509+0000 7f0cab34f700 1 -- 172.21.15.1:0/1454356150 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c8806adb0 msgr2=0x7f0c8806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:06:27.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.509+0000 7f0cab34f700 1 --2- 172.21.15.1:0/1454356150 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c8806adb0 0x7f0c8806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.509+0000 7f0cab34f700 1 -- 172.21.15.1:0/1454356150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca409a360 msgr2=0x7f0ca41587a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:27.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.509+0000 7f0cab34f700 1 --2- 172.21.15.1:0/1454356150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca409a360 0x7f0ca41587a0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f0c94009130 tx=0x7f0c94004750 comp rx=0 tx=0).stop 2024-03-18T20:06:27.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.509+0000 7f0cab34f700 1 -- 172.21.15.1:0/1454356150 shutdown_connections 2024-03-18T20:06:27.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.509+0000 7f0cab34f700 1 --2- 172.21.15.1:0/1454356150 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0c8806adb0 0x7f0c8806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.509+0000 7f0cab34f700 1 --2- 172.21.15.1:0/1454356150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ca4099a40 0x7f0ca415c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.509+0000 7f0cab34f700 1 --2- 172.21.15.1:0/1454356150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ca40d05a0 0x7f0ca4150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.509+0000 7f0cab34f700 1 --2- 172.21.15.1:0/1454356150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ca409a360 0x7f0ca41587a0 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.509+0000 7f0cab34f700 1 -- 172.21.15.1:0/1454356150 >> 172.21.15.1:0/1454356150 conn(0x7f0ca40b2cb0 msgr2=0x7f0ca40b3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:27.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.509+0000 7f0cab34f700 1 -- 172.21.15.1:0/1454356150 shutdown_connections 2024-03-18T20:06:27.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.509+0000 7f0cab34f700 1 -- 172.21.15.1:0/1454356150 wait complete. 2024-03-18T20:06:27.515 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 73 2024-03-18T20:06:27.527 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:06:27.651 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:27.647+0000 7fc3c8580240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:06:27.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.652+0000 7f61d55ce700 1 Processor -- start 2024-03-18T20:06:27.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.652+0000 7f61d55ce700 1 -- start start 2024-03-18T20:06:27.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.653+0000 7f61d55ce700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d015b4d0 0x7f61d009a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:27.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.653+0000 7f61d55ce700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f61d00cf750 0x7f61d0099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:27.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.653+0000 7f61d55ce700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61d006f070 con 0x7f61d0099f80 2024-03-18T20:06:27.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.653+0000 7f61d55ce700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61d009a410 con 0x7f61d00cf750 2024-03-18T20:06:27.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.653+0000 7f61d55ce700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d00d0260 con 0x7f61d015b4d0 2024-03-18T20:06:27.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.653+0000 7f61ce7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f61d00cf750 0x7f61d0099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.653+0000 7f61cf7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f61d0099f80 0x7f61d00cbd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59170/0 (socket says 172.21.15.1:59170) 2024-03-18T20:06:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.653+0000 7f61cf7fe700 1 -- 172.21.15.1:0/3103110237 learned_addr learned my addr 172.21.15.1:0/3103110237 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.653+0000 7f61ce7fc700 1 --2- 172.21.15.1:0/3103110237 >> v2:172.21.15.1:3300/0 conn(0x7f61d00cf750 0x7f61d0099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33464/0 (socket says 172.21.15.1:33464) 2024-03-18T20:06:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.654+0000 7f61ceffd700 1 --2- 172.21.15.1:0/3103110237 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d015b4d0 0x7f61d009a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.654+0000 7f61ce7fc700 1 -- 172.21.15.1:0/3103110237 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d015b4d0 msgr2=0x7f61d009a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.654+0000 7f61ce7fc700 1 --2- 172.21.15.1:0/3103110237 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d015b4d0 0x7f61d009a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.654+0000 7f61ce7fc700 1 -- 172.21.15.1:0/3103110237 >> v1:172.21.15.1:6789/0 conn(0x7f61d0099f80 legacy=0x7f61d00cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.654+0000 7f61ce7fc700 1 -- 172.21.15.1:0/3103110237 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d00cff80 con 0x7f61d00cf750 2024-03-18T20:06:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.654+0000 7f61cdffb700 1 -- 172.21.15.1:0/3103110237 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287071708 0 0) 0x7f61d006f070 con 0x7f61d0099f80 2024-03-18T20:06:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.654+0000 7f61ceffd700 1 --2- 172.21.15.1:0/3103110237 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d015b4d0 0x7f61d009a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.654+0000 7f61ce7fc700 1 --2- 172.21.15.1:0/3103110237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d00cf750 0x7f61d0099a40 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f61c4009840 tx=0x7f61c4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9460341b1e7b448b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.654+0000 7f61cdffb700 1 -- 172.21.15.1:0/3103110237 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c4011830 con 0x7f61d00cf750 2024-03-18T20:06:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.654+0000 7f61cdffb700 1 -- 172.21.15.1:0/3103110237 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c4011990 con 0x7f61d00cf750 2024-03-18T20:06:27.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.654+0000 7f61cdffb700 1 -- 172.21.15.1:0/3103110237 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c4011d20 con 0x7f61d00cf750 2024-03-18T20:06:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.655+0000 7f61d55ce700 1 -- 172.21.15.1:0/3103110237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d00cf750 msgr2=0x7f61d0099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.655+0000 7f61d55ce700 1 --2- 172.21.15.1:0/3103110237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d00cf750 0x7f61d0099a40 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f61c4009840 tx=0x7f61c4009b50 comp rx=0 tx=0).stop 2024-03-18T20:06:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.655+0000 7f61d55ce700 1 -- 172.21.15.1:0/3103110237 shutdown_connections 2024-03-18T20:06:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.655+0000 7f61d55ce700 1 --2- 172.21.15.1:0/3103110237 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d015b4d0 0x7f61d009a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.655+0000 7f61d55ce700 1 --2- 172.21.15.1:0/3103110237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d00cf750 0x7f61d0099a40 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.655+0000 7f61d55ce700 1 -- 172.21.15.1:0/3103110237 >> 172.21.15.1:0/3103110237 conn(0x7f61d00b2cb0 msgr2=0x7f61d00b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.655+0000 7f61d55ce700 1 -- 172.21.15.1:0/3103110237 shutdown_connections 2024-03-18T20:06:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.655+0000 7f61d55ce700 1 -- 172.21.15.1:0/3103110237 wait complete. 2024-03-18T20:06:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.656+0000 7f61d55ce700 1 Processor -- start 2024-03-18T20:06:27.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.656+0000 7f61d55ce700 1 -- start start 2024-03-18T20:06:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.657+0000 7f61d55ce700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d0099f80 0x7f61d0158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.657+0000 7f61d55ce700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d00cf750 0x7f61d00ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.657+0000 7f61d55ce700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d015b4d0 0x7f61d015dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.657+0000 7f61d55ce700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d00c4c90 con 0x7f61d015b4d0 2024-03-18T20:06:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.657+0000 7f61d55ce700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d00cac40 con 0x7f61d00cf750 2024-03-18T20:06:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.657+0000 7f61d55ce700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d00caf20 con 0x7f61d0099f80 2024-03-18T20:06:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.657+0000 7f61cf7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d015b4d0 0x7f61d015dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.657+0000 7f61cf7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d015b4d0 0x7f61d015dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33466/0 (socket says 172.21.15.1:33466) 2024-03-18T20:06:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.657+0000 7f61cf7fe700 1 -- 172.21.15.1:0/838051934 learned_addr learned my addr 172.21.15.1:0/838051934 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.657+0000 7f61ceffd700 1 --2- 172.21.15.1:0/838051934 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d0099f80 0x7f61d0158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.657+0000 7f61ce7fc700 1 --2- 172.21.15.1:0/838051934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d00cf750 0x7f61d00ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.658+0000 7f61cf7fe700 1 -- 172.21.15.1:0/838051934 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d0099f80 msgr2=0x7f61d0158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.658+0000 7f61cf7fe700 1 --2- 172.21.15.1:0/838051934 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d0099f80 0x7f61d0158ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.658+0000 7f61cf7fe700 1 -- 172.21.15.1:0/838051934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d00cf750 msgr2=0x7f61d00ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.658+0000 7f61cf7fe700 1 --2- 172.21.15.1:0/838051934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d00cf750 0x7f61d00ca1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.658+0000 7f61cf7fe700 1 -- 172.21.15.1:0/838051934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c40094a0 con 0x7f61d015b4d0 2024-03-18T20:06:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.658+0000 7f61cf7fe700 1 --2- 172.21.15.1:0/838051934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d015b4d0 0x7f61d015dc00 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f61c800cca0 tx=0x7f61c800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.658+0000 7f61b77fe700 1 -- 172.21.15.1:0/838051934 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c800ab70 con 0x7f61d015b4d0 2024-03-18T20:06:27.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.658+0000 7f61b77fe700 1 -- 172.21.15.1:0/838051934 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c800acd0 con 0x7f61d015b4d0 2024-03-18T20:06:27.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.658+0000 7f61b77fe700 1 -- 172.21.15.1:0/838051934 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c8007310 con 0x7f61d015b4d0 2024-03-18T20:06:27.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.658+0000 7f61d55ce700 1 -- 172.21.15.1:0/838051934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d00cb200 con 0x7f61d015b4d0 2024-03-18T20:06:27.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.658+0000 7f61d55ce700 1 -- 172.21.15.1:0/838051934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d0159920 con 0x7f61d015b4d0 2024-03-18T20:06:27.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.659+0000 7f61d55ce700 1 -- 172.21.15.1:0/838051934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d0068240 con 0x7f61d015b4d0 2024-03-18T20:06:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.660+0000 7f61b77fe700 1 -- 172.21.15.1:0/838051934 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 91) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f61c8007470 con 0x7f61d015b4d0 2024-03-18T20:06:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.660+0000 7f61b77fe700 1 --2- 172.21.15.1:0/838051934 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f61bc06adb0 0x7f61bc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.660+0000 7f61ceffd700 1 -- 172.21.15.1:0/838051934 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f61bc06adb0 msgr2=0x7f61bc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.660+0000 7f61b77fe700 1 -- 172.21.15.1:0/838051934 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f61c80893c0 con 0x7f61d015b4d0 2024-03-18T20:06:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.660+0000 7f61ceffd700 1 --2- 172.21.15.1:0/838051934 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f61bc06adb0 0x7f61bc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:06:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.665+0000 7f61b77fe700 1 -- 172.21.15.1:0/838051934 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f61c80540e0 con 0x7f61d015b4d0 2024-03-18T20:06:27.749 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:27.744+0000 7fc3c8580240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:06:27.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.763+0000 7f61b77fe700 1 -- 172.21.15.1:0/838051934 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 92) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f61c80544b0 con 0x7f61d015b4d0 2024-03-18T20:06:27.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.763+0000 7f61b77fe700 1 -- 172.21.15.1:0/838051934 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f61bc06adb0 msgr2=0x7f61bc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:06:27.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.764+0000 7f61b77fe700 1 --2- 172.21.15.1:0/838051934 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f61bc06adb0 0x7f61bc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.836 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:27.832+0000 7fc3409a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:27.837 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:27.832+0000 7fc3409a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:27.837 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:27.832+0000 7fc3409a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:27.837 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:27.832+0000 7fc3409a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:27.837 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:27.832+0000 7fc3409a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:27.837 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:27.832+0000 7fc3439ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:27.837 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:27.832+0000 7fc3439ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:27.837 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:27.832+0000 7fc3439ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:27.837 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:27.832+0000 7fc3439ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:27.837 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:27.832+0000 7fc3439ae700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:27.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.935+0000 7f61d55ce700 1 -- 172.21.15.1:0/838051934 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61d00c50e0 con 0x7f61d015b4d0 2024-03-18T20:06:27.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.936+0000 7f61b77fe700 1 -- 172.21.15.1:0/838051934 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f61d00c50e0 con 0x7f61d015b4d0 2024-03-18T20:06:27.941 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:27.941 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8608,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3436643592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3436643592},{"type":"v1","addr":"172.21.15.165:6834","nonce":3436643592}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":8611,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1969816196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1969816196},{"type":"v1","addr":"172.21.15.165:6835","nonce":1969816196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":9375,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2671786416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2671786416},{"type":"v1","addr":"172.21.15.1:6836","nonce":2671786416}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T19:10:04.613212+0000","modified":"2024-03-18T19:10:07.625525+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9378":{"gid":9378,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1696108841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1696108841},{"type":"v1","addr":"172.21.15.1:6837","nonce":1696108841}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-03-18T20:06:27.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.938+0000 7f61d55ce700 1 -- 172.21.15.1:0/838051934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d015b4d0 msgr2=0x7f61d015dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:27.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.938+0000 7f61d55ce700 1 --2- 172.21.15.1:0/838051934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d015b4d0 0x7f61d015dc00 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f61c800cca0 tx=0x7f61c800a060 comp rx=0 tx=0).stop 2024-03-18T20:06:27.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.938+0000 7f61d55ce700 1 -- 172.21.15.1:0/838051934 shutdown_connections 2024-03-18T20:06:27.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.938+0000 7f61d55ce700 1 --2- 172.21.15.1:0/838051934 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f61bc06adb0 0x7f61bc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.938+0000 7f61d55ce700 1 --2- 172.21.15.1:0/838051934 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d0099f80 0x7f61d0158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.938+0000 7f61d55ce700 1 --2- 172.21.15.1:0/838051934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d015b4d0 0x7f61d015dc00 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.938+0000 7f61d55ce700 1 --2- 172.21.15.1:0/838051934 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d00cf750 0x7f61d00ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:27.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.938+0000 7f61d55ce700 1 -- 172.21.15.1:0/838051934 >> 172.21.15.1:0/838051934 conn(0x7f61d00b2cb0 msgr2=0x7f61d00d0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:27.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.938+0000 7f61d55ce700 1 -- 172.21.15.1:0/838051934 shutdown_connections 2024-03-18T20:06:27.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:27.938+0000 7f61d55ce700 1 -- 172.21.15.1:0/838051934 wait complete. 2024-03-18T20:06:27.944 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 73 2024-03-18T20:06:27.955 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:06:28.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.076+0000 7f963d498700 1 Processor -- start 2024-03-18T20:06:28.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.076+0000 7f963d498700 1 -- start start 2024-03-18T20:06:28.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.077+0000 7f963d498700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f963806a750 0x7f96380ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:28.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.077+0000 7f963d498700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963806ee90 con 0x7f963806ea10 2024-03-18T20:06:28.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.077+0000 7f963d498700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963806e1f0 con 0x7f96380ace80 2024-03-18T20:06:28.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.077+0000 7f963d498700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f963806e4d0 con 0x7f963806a750 2024-03-18T20:06:28.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.077+0000 7f9636ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f963806a750 0x7f96380ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:28.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.077+0000 7f96367fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f963806ea10 0x7f963806ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59178/0 (socket says 172.21.15.1:59178) 2024-03-18T20:06:28.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.077+0000 7f9636ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f963806a750 0x7f96380ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33580/0 (socket says 172.21.15.1:33580) 2024-03-18T20:06:28.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.077+0000 7f96367fc700 1 -- 172.21.15.1:0/3957758009 learned_addr learned my addr 172.21.15.1:0/3957758009 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:28.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.078+0000 7f9636ffd700 1 -- 172.21.15.1:0/3957758009 >> v1:172.21.15.165:6789/0 conn(0x7f96380ace80 legacy=0x7f96380ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:28.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.078+0000 7f9636ffd700 1 -- 172.21.15.1:0/3957758009 >> v1:172.21.15.1:6789/0 conn(0x7f963806ea10 legacy=0x7f963806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:28.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.078+0000 7f9636ffd700 1 -- 172.21.15.1:0/3957758009 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96380cc2a0 con 0x7f963806a750 2024-03-18T20:06:28.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.078+0000 7f9636ffd700 1 --2- 172.21.15.1:0/3957758009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f963806a750 0x7f96380ca5f0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f962c009840 tx=0x7f962c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75af37342de793a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:28.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.078+0000 7f9635ffb700 1 -- 172.21.15.1:0/3957758009 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c011830 con 0x7f963806a750 2024-03-18T20:06:28.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.078+0000 7f9635ffb700 1 -- 172.21.15.1:0/3957758009 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962c011990 con 0x7f963806a750 2024-03-18T20:06:28.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.078+0000 7f9635ffb700 1 -- 172.21.15.1:0/3957758009 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c011d20 con 0x7f963806a750 2024-03-18T20:06:28.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.079+0000 7f963d498700 1 -- 172.21.15.1:0/3957758009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f963806a750 msgr2=0x7f96380ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:28.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.079+0000 7f963d498700 1 --2- 172.21.15.1:0/3957758009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f963806a750 0x7f96380ca5f0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f962c009840 tx=0x7f962c009b50 comp rx=0 tx=0).stop 2024-03-18T20:06:28.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.079+0000 7f963d498700 1 -- 172.21.15.1:0/3957758009 shutdown_connections 2024-03-18T20:06:28.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.079+0000 7f963d498700 1 --2- 172.21.15.1:0/3957758009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f963806a750 0x7f96380ca5f0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.079+0000 7f963d498700 1 -- 172.21.15.1:0/3957758009 >> 172.21.15.1:0/3957758009 conn(0x7f96380cc990 msgr2=0x7f96380ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:28.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.079+0000 7f963d498700 1 -- 172.21.15.1:0/3957758009 shutdown_connections 2024-03-18T20:06:28.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.079+0000 7f963d498700 1 -- 172.21.15.1:0/3957758009 wait complete. 2024-03-18T20:06:28.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.080+0000 7f963d498700 1 Processor -- start 2024-03-18T20:06:28.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.080+0000 7f963d498700 1 -- start start 2024-03-18T20:06:28.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f963d498700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f963806a750 0x7f96380c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f963d498700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f963806ea10 0x7f96380c4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f963d498700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96380ace80 0x7f96380c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f963d498700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96380c9f00 con 0x7f96380ace80 2024-03-18T20:06:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f963d498700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96380c5b90 con 0x7f963806ea10 2024-03-18T20:06:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f963d498700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96380c5e70 con 0x7f963806a750 2024-03-18T20:06:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f96377fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96380ace80 0x7f96380c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f96377fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96380ace80 0x7f96380c54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33588/0 (socket says 172.21.15.1:33588) 2024-03-18T20:06:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f96377fe700 1 -- 172.21.15.1:0/2729840326 learned_addr learned my addr 172.21.15.1:0/2729840326 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f96367fc700 1 --2- 172.21.15.1:0/2729840326 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f963806ea10 0x7f96380c4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f9636ffd700 1 --2- 172.21.15.1:0/2729840326 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f963806a750 0x7f96380c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f96377fe700 1 -- 172.21.15.1:0/2729840326 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f963806a750 msgr2=0x7f96380c99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f96377fe700 1 --2- 172.21.15.1:0/2729840326 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f963806a750 0x7f96380c99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f96377fe700 1 -- 172.21.15.1:0/2729840326 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f963806ea10 msgr2=0x7f96380c4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f96377fe700 1 --2- 172.21.15.1:0/2729840326 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f963806ea10 0x7f96380c4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f96377fe700 1 -- 172.21.15.1:0/2729840326 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962c0094a0 con 0x7f96380ace80 2024-03-18T20:06:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f96377fe700 1 --2- 172.21.15.1:0/2729840326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96380ace80 0x7f96380c54a0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f963000cb50 tx=0x7f963000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f961f7fe700 1 -- 172.21.15.1:0/2729840326 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9630003c00 con 0x7f96380ace80 2024-03-18T20:06:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f961f7fe700 1 -- 172.21.15.1:0/2729840326 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9630003d60 con 0x7f96380ace80 2024-03-18T20:06:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f963d498700 1 -- 172.21.15.1:0/2729840326 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96380c6150 con 0x7f96380ace80 2024-03-18T20:06:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.081+0000 7f961f7fe700 1 -- 172.21.15.1:0/2729840326 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963000aa20 con 0x7f96380ace80 2024-03-18T20:06:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.082+0000 7f963d498700 1 -- 172.21.15.1:0/2729840326 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9638155040 con 0x7f96380ace80 2024-03-18T20:06:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.082+0000 7f963d498700 1 -- 172.21.15.1:0/2729840326 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9638155260 con 0x7f96380ace80 2024-03-18T20:06:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.083+0000 7f961f7fe700 1 -- 172.21.15.1:0/2729840326 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 92) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f963000ab80 con 0x7f96380ace80 2024-03-18T20:06:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.084+0000 7f961f7fe700 1 -- 172.21.15.1:0/2729840326 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6703+0+0 (secure 0 0 0) 0x7f9630089ba0 con 0x7f96380ace80 2024-03-18T20:06:28.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.088+0000 7f961f7fe700 1 -- 172.21.15.1:0/2729840326 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9630054e90 con 0x7f96380ace80 2024-03-18T20:06:28.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.292+0000 7f963d498700 1 -- 172.21.15.1:0/2729840326 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96380676a0 con 0x7f96380ace80 2024-03-18T20:06:28.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.293+0000 7f961f7fe700 1 -- 172.21.15.1:0/2729840326 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v169) v1 ==== 74+0+17201 (secure 0 0 0) 0x7f96300584b0 con 0x7f96380ace80 2024-03-18T20:06:28.298 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:28.298 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":169,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:06:27.758470+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-18T19:10:02.452367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-18T19:10:03.585045+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":168,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/4204704242":"2024-03-19T20:06:27.758434+0000","172.21.15.1:0/2988449518":"2024-03-19T20:06:27.758434+0000","172.21.15.1:0/1454712291":"2024-03-19T20:06:27.758434+0000","172.21.15.1:0/457749622":"2024-03-19T20:06:27.758434+0000","172.21.15.1:0/2246000724":"2024-03-19T20:06:27.758434+0000","172.21.15.1:6833/76998":"2024-03-19T19:10:19.888890+0000","172.21.15.1:6832/76998":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/371341431":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/1433676792":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/4211592802":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/4117009712":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/2398817063":"2024-03-19T19:10:19.888890+0000","172.21.15.1:0/99037403":"2024-03-19T20:06:27.758434+0000","172.21.15.1:0/1643985667":"2024-03-19T19:10:19.888890+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:06:28.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.295+0000 7f963d498700 1 -- 172.21.15.1:0/2729840326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96380ace80 msgr2=0x7f96380c54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:28.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.295+0000 7f963d498700 1 --2- 172.21.15.1:0/2729840326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96380ace80 0x7f96380c54a0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f963000cb50 tx=0x7f963000ce60 comp rx=0 tx=0).stop 2024-03-18T20:06:28.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.295+0000 7f963d498700 1 -- 172.21.15.1:0/2729840326 shutdown_connections 2024-03-18T20:06:28.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.295+0000 7f963d498700 1 --2- 172.21.15.1:0/2729840326 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f963806a750 0x7f96380c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.295+0000 7f963d498700 1 --2- 172.21.15.1:0/2729840326 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96380ace80 0x7f96380c54a0 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.295+0000 7f963d498700 1 --2- 172.21.15.1:0/2729840326 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f963806ea10 0x7f96380c4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.295+0000 7f963d498700 1 -- 172.21.15.1:0/2729840326 >> 172.21.15.1:0/2729840326 conn(0x7f96380cc990 msgr2=0x7f96380cdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:28.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.295+0000 7f963d498700 1 -- 172.21.15.1:0/2729840326 shutdown_connections 2024-03-18T20:06:28.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.295+0000 7f963d498700 1 -- 172.21.15.1:0/2729840326 wait complete. 2024-03-18T20:06:28.313 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T20:06:28.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.435+0000 7fc4fefdb700 1 Processor -- start 2024-03-18T20:06:28.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.435+0000 7fc4fefdb700 1 -- start start 2024-03-18T20:06:28.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.435+0000 7fc4fefdb700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc4f806ca80 0x7fc4f815d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:28.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.435+0000 7fc4fefdb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4f809a150 0x7fc4f80cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:28.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.436+0000 7fc4fefdb700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f80b3b00 con 0x7fc4f815d5b0 2024-03-18T20:06:28.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.436+0000 7fc4fefdb700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f80cbbb0 con 0x7fc4f806ca80 2024-03-18T20:06:28.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.436+0000 7fc4fefdb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f80cf7f0 con 0x7fc4f809a150 2024-03-18T20:06:28.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.436+0000 7fc4f7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc4f806ca80 0x7fc4f815d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:28.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.436+0000 7fc4f7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc4f806ca80 0x7fc4f815d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33592/0 (socket says 172.21.15.1:33592) 2024-03-18T20:06:28.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.436+0000 7fc4f7fff700 1 -- 172.21.15.1:0/1586301312 learned_addr learned my addr 172.21.15.1:0/1586301312 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:28.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.436+0000 7fc4fd578700 1 --2- 172.21.15.1:0/1586301312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4f809a150 0x7fc4f80cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:28.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.436+0000 7fc4f7fff700 1 -- 172.21.15.1:0/1586301312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4f809a150 msgr2=0x7fc4f80cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:28.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.436+0000 7fc4f7fff700 1 --2- 172.21.15.1:0/1586301312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4f809a150 0x7fc4f80cbfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.436+0000 7fc4f7fff700 1 -- 172.21.15.1:0/1586301312 >> v1:172.21.15.165:6789/0 conn(0x7fc4f815d5b0 legacy=0x7fc4f8099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:28.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.436+0000 7fc4f7fff700 1 -- 172.21.15.1:0/1586301312 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f80d04d0 con 0x7fc4f806ca80 2024-03-18T20:06:28.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.436+0000 7fc4f7fff700 1 --2- 172.21.15.1:0/1586301312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f806ca80 0x7fc4f815d990 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fc4e40098f0 tx=0x7fc4e4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d43b42f3ac7dae8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:28.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.436+0000 7fc4f77fe700 1 -- 172.21.15.1:0/1586301312 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4004230 con 0x7fc4f806ca80 2024-03-18T20:06:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.436+0000 7fc4f77fe700 1 -- 172.21.15.1:0/1586301312 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e4004390 con 0x7fc4f806ca80 2024-03-18T20:06:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.436+0000 7fc4f77fe700 1 -- 172.21.15.1:0/1586301312 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e40036a0 con 0x7fc4f806ca80 2024-03-18T20:06:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.437+0000 7fc4fefdb700 1 -- 172.21.15.1:0/1586301312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f806ca80 msgr2=0x7fc4f815d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.437+0000 7fc4fefdb700 1 --2- 172.21.15.1:0/1586301312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f806ca80 0x7fc4f815d990 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fc4e40098f0 tx=0x7fc4e4009c00 comp rx=0 tx=0).stop 2024-03-18T20:06:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.437+0000 7fc4fefdb700 1 -- 172.21.15.1:0/1586301312 shutdown_connections 2024-03-18T20:06:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.437+0000 7fc4fefdb700 1 --2- 172.21.15.1:0/1586301312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4f809a150 0x7fc4f80cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.437+0000 7fc4fefdb700 1 --2- 172.21.15.1:0/1586301312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f806ca80 0x7fc4f815d990 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.437+0000 7fc4fefdb700 1 -- 172.21.15.1:0/1586301312 >> 172.21.15.1:0/1586301312 conn(0x7fc4f80b2bd0 msgr2=0x7fc4f80b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.437+0000 7fc4fefdb700 1 -- 172.21.15.1:0/1586301312 shutdown_connections 2024-03-18T20:06:28.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.437+0000 7fc4fefdb700 1 -- 172.21.15.1:0/1586301312 wait complete. 2024-03-18T20:06:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.438+0000 7fc4fefdb700 1 Processor -- start 2024-03-18T20:06:28.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.439+0000 7fc4fefdb700 1 -- start start 2024-03-18T20:06:28.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4fefdb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4f806ca80 0x7fc4f8154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:28.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4fefdb700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4f809a150 0x7fc4f815b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:28.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4fefdb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f815d5b0 0x7fc4f815b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:28.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4fefdb700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f8154d80 con 0x7fc4f815d5b0 2024-03-18T20:06:28.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4fefdb700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f80b54d0 con 0x7fc4f809a150 2024-03-18T20:06:28.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4fefdb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f80b57b0 con 0x7fc4f806ca80 2024-03-18T20:06:28.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4fd578700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f815d5b0 0x7fc4f815b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:28.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4fd578700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f815d5b0 0x7fc4f815b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33598/0 (socket says 172.21.15.1:33598) 2024-03-18T20:06:28.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4fd578700 1 -- 172.21.15.1:0/946939969 learned_addr learned my addr 172.21.15.1:0/946939969 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:28.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4f7fff700 1 --2- 172.21.15.1:0/946939969 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4f809a150 0x7fc4f815b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:28.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4fcd77700 1 --2- 172.21.15.1:0/946939969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4f806ca80 0x7fc4f8154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:28.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4fd578700 1 -- 172.21.15.1:0/946939969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4f806ca80 msgr2=0x7fc4f8154840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:28.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4fd578700 1 --2- 172.21.15.1:0/946939969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4f806ca80 0x7fc4f8154840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4fd578700 1 -- 172.21.15.1:0/946939969 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4f809a150 msgr2=0x7fc4f815b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:28.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4fd578700 1 --2- 172.21.15.1:0/946939969 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4f809a150 0x7fc4f815b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4fd578700 1 -- 172.21.15.1:0/946939969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e4009550 con 0x7fc4f815d5b0 2024-03-18T20:06:28.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4f7fff700 1 --2- 172.21.15.1:0/946939969 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4f809a150 0x7fc4f815b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:28.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4fcd77700 1 --2- 172.21.15.1:0/946939969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4f806ca80 0x7fc4f8154840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:28.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4fd578700 1 --2- 172.21.15.1:0/946939969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f815d5b0 0x7fc4f815b8c0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7fc4f000cc30 tx=0x7fc4f000a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:28.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4f57fa700 1 -- 172.21.15.1:0/946939969 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f000ab70 con 0x7fc4f815d5b0 2024-03-18T20:06:28.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4f57fa700 1 -- 172.21.15.1:0/946939969 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f000acd0 con 0x7fc4f815d5b0 2024-03-18T20:06:28.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4fefdb700 1 -- 172.21.15.1:0/946939969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f80b5a90 con 0x7fc4f815d5b0 2024-03-18T20:06:28.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4f57fa700 1 -- 172.21.15.1:0/946939969 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f0003c40 con 0x7fc4f815d5b0 2024-03-18T20:06:28.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.440+0000 7fc4fefdb700 1 -- 172.21.15.1:0/946939969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f80c8700 con 0x7fc4f815d5b0 2024-03-18T20:06:28.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.441+0000 7fc4fefdb700 1 -- 172.21.15.1:0/946939969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f80b23b0 con 0x7fc4f815d5b0 2024-03-18T20:06:28.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.442+0000 7fc4f57fa700 1 -- 172.21.15.1:0/946939969 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 92) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fc4f000f3f0 con 0x7fc4f815d5b0 2024-03-18T20:06:28.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.442+0000 7fc4f57fa700 1 -- 172.21.15.1:0/946939969 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6703+0+0 (secure 0 0 0) 0x7fc4f0089370 con 0x7fc4f815d5b0 2024-03-18T20:06:28.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.447+0000 7fc4f57fa700 1 -- 172.21.15.1:0/946939969 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc4f00545c0 con 0x7fc4f815d5b0 2024-03-18T20:06:28.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.694+0000 7fc4fefdb700 1 -- 172.21.15.1:0/946939969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc4f80676a0 con 0x7fc4f815d5b0 2024-03-18T20:06:28.700 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-18T20:06:28.695+0000 7fbdbeb46700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T20:06:28.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.707+0000 7fc4f57fa700 1 -- 172.21.15.1:0/946939969 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v74) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc4f001d090 con 0x7fc4f815d5b0 2024-03-18T20:06:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.710+0000 7fc4fefdb700 1 -- 172.21.15.1:0/946939969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f815d5b0 msgr2=0x7fc4f815b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.710+0000 7fc4fefdb700 1 --2- 172.21.15.1:0/946939969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f815d5b0 0x7fc4f815b8c0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7fc4f000cc30 tx=0x7fc4f000a1f0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.711+0000 7fc4fefdb700 1 -- 172.21.15.1:0/946939969 shutdown_connections 2024-03-18T20:06:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.711+0000 7fc4fefdb700 1 --2- 172.21.15.1:0/946939969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4f806ca80 0x7fc4f8154840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.711+0000 7fc4fefdb700 1 --2- 172.21.15.1:0/946939969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4f815d5b0 0x7fc4f815b8c0 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.711+0000 7fc4fefdb700 1 --2- 172.21.15.1:0/946939969 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4f809a150 0x7fc4f815b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.711+0000 7fc4fefdb700 1 -- 172.21.15.1:0/946939969 >> 172.21.15.1:0/946939969 conn(0x7fc4f80b2bd0 msgr2=0x7fc4f806f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.711+0000 7fc4fefdb700 1 -- 172.21.15.1:0/946939969 shutdown_connections 2024-03-18T20:06:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.711+0000 7fc4fefdb700 1 -- 172.21.15.1:0/946939969 wait complete. 2024-03-18T20:06:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T20:06:28.732 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T20:06:28.758 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T20:06:28.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.861+0000 7fd31b8a3700 1 Processor -- start 2024-03-18T20:06:28.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.861+0000 7fd31b8a3700 1 -- start start 2024-03-18T20:06:28.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.862+0000 7fd31b8a3700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd314071d30 0x7fd3140add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:28.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.862+0000 7fd31b8a3700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3140ce8f0 con 0x7fd3140cb9e0 2024-03-18T20:06:28.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.862+0000 7fd31b8a3700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3140ae260 con 0x7fd31415a1d0 2024-03-18T20:06:28.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.862+0000 7fd31b8a3700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3140cdb00 con 0x7fd314071d30 2024-03-18T20:06:28.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.862+0000 7fd319e40700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd3140cb9e0 0x7fd3140cbe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59198/0 (socket says 172.21.15.1:59198) 2024-03-18T20:06:28.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.862+0000 7fd319e40700 1 -- 172.21.15.1:0/1020939963 learned_addr learned my addr 172.21.15.1:0/1020939963 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:28.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.862+0000 7fd318e3e700 1 --2- 172.21.15.1:0/1020939963 >> v2:172.21.15.165:3300/0 conn(0x7fd314071d30 0x7fd3140add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:28.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.862+0000 7fd30bfff700 1 -- 172.21.15.1:0/1020939963 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811948731 0 0) 0x7fd3140ce8f0 con 0x7fd3140cb9e0 2024-03-18T20:06:28.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.862+0000 7fd30bfff700 1 -- 172.21.15.1:0/1020939963 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3000035b0 con 0x7fd3140cb9e0 2024-03-18T20:06:28.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.862+0000 7fd30bfff700 1 -- 172.21.15.1:0/1020939963 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2300017737 0 0) 0x7fd3000035b0 con 0x7fd3140cb9e0 2024-03-18T20:06:28.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.862+0000 7fd30bfff700 1 -- 172.21.15.1:0/1020939963 >> v2:172.21.15.165:3300/0 conn(0x7fd314071d30 msgr2=0x7fd3140add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:28.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.862+0000 7fd30bfff700 1 --2- 172.21.15.1:0/1020939963 >> v2:172.21.15.165:3300/0 conn(0x7fd314071d30 0x7fd3140add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.862+0000 7fd30bfff700 1 -- 172.21.15.1:0/1020939963 >> v1:172.21.15.165:6789/0 conn(0x7fd31415a1d0 legacy=0x7fd314071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd30bfff700 1 -- 172.21.15.1:0/1020939963 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3140cdce0 con 0x7fd3140cb9e0 2024-03-18T20:06:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd30bfff700 1 -- 172.21.15.1:0/1020939963 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fd30c0019f0 con 0x7fd3140cb9e0 2024-03-18T20:06:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd30bfff700 1 -- 172.21.15.1:0/1020939963 >> v1:172.21.15.1:6789/0 conn(0x7fd3140cb9e0 legacy=0x7fd3140cbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd30bfff700 1 --2- 172.21.15.1:0/1020939963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd300003a00 0x7fd300003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd30bfff700 1 --2- 172.21.15.1:0/1020939963 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd300004850 0x7fd300004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd30bfff700 1 --2- 172.21.15.1:0/1020939963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd300005210 0x7fd30000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd30bfff700 1 -- 172.21.15.1:0/1020939963 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3140cdb00 con 0x7fd300003a00 2024-03-18T20:06:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd30bfff700 1 -- 172.21.15.1:0/1020939963 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30000c0b0 con 0x7fd300005210 2024-03-18T20:06:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd30bfff700 1 -- 172.21.15.1:0/1020939963 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd30000c270 con 0x7fd300004850 2024-03-18T20:06:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd31963f700 1 --2- 172.21.15.1:0/1020939963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd300003a00 0x7fd300003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd318e3e700 1 --2- 172.21.15.1:0/1020939963 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd300004850 0x7fd300004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd319e40700 1 --2- 172.21.15.1:0/1020939963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd300005210 0x7fd30000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd31963f700 1 -- 172.21.15.1:0/1020939963 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd300004850 msgr2=0x7fd300004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:28.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd31963f700 1 --2- 172.21.15.1:0/1020939963 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd300004850 0x7fd300004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd31963f700 1 -- 172.21.15.1:0/1020939963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd300005210 msgr2=0x7fd30000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd31963f700 1 --2- 172.21.15.1:0/1020939963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd300005210 0x7fd30000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd31963f700 1 -- 172.21.15.1:0/1020939963 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3000035b0 con 0x7fd300003a00 2024-03-18T20:06:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd319e40700 1 -- 172.21.15.1:0/1020939963 reap_dead start 2024-03-18T20:06:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd318e3e700 1 --2- 172.21.15.1:0/1020939963 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd300004850 0x7fd300004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd31963f700 1 --2- 172.21.15.1:0/1020939963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd300003a00 0x7fd300003e20 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fd310001860 tx=0x7fd300005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a586784da87125e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd30bfff700 1 -- 172.21.15.1:0/1020939963 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd310001320 con 0x7fd300003a00 2024-03-18T20:06:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd30bfff700 1 -- 172.21.15.1:0/1020939963 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd310005310 con 0x7fd300003a00 2024-03-18T20:06:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.863+0000 7fd30bfff700 1 -- 172.21.15.1:0/1020939963 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3100068b0 con 0x7fd300003a00 2024-03-18T20:06:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.864+0000 7fd31b8a3700 1 -- 172.21.15.1:0/1020939963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd300003a00 msgr2=0x7fd300003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.864+0000 7fd31b8a3700 1 --2- 172.21.15.1:0/1020939963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd300003a00 0x7fd300003e20 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fd310001860 tx=0x7fd300005880 comp rx=0 tx=0).stop 2024-03-18T20:06:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.864+0000 7fd31b8a3700 1 -- 172.21.15.1:0/1020939963 shutdown_connections 2024-03-18T20:06:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.864+0000 7fd31b8a3700 1 --2- 172.21.15.1:0/1020939963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd300003a00 0x7fd300003e20 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.864+0000 7fd31b8a3700 1 -- 172.21.15.1:0/1020939963 >> 172.21.15.1:0/1020939963 conn(0x7fd3140cc9b0 msgr2=0x7fd3140ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:28.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.864+0000 7fd31b8a3700 1 -- 172.21.15.1:0/1020939963 shutdown_connections 2024-03-18T20:06:28.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.864+0000 7fd31b8a3700 1 -- 172.21.15.1:0/1020939963 wait complete. 2024-03-18T20:06:28.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.865+0000 7fd31b8a3700 1 Processor -- start 2024-03-18T20:06:28.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.865+0000 7fd31b8a3700 1 -- start start 2024-03-18T20:06:28.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.866+0000 7fd31b8a3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd300003a00 0x7fd3140c6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.866+0000 7fd31b8a3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3141f2a70 0x7fd3141f2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.866+0000 7fd31b8a3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3141e3a90 0x7fd3141e3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.866+0000 7fd31b8a3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3141f3430 con 0x7fd3141f2a70 2024-03-18T20:06:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.866+0000 7fd31b8a3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3141e4ac0 con 0x7fd300003a00 2024-03-18T20:06:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.866+0000 7fd31b8a3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3140c3f80 con 0x7fd3141e3a90 2024-03-18T20:06:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.866+0000 7fd318e3e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3141f2a70 0x7fd3141f2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.866+0000 7fd318e3e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3141f2a70 0x7fd3141f2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33622/0 (socket says 172.21.15.1:33622) 2024-03-18T20:06:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.866+0000 7fd318e3e700 1 -- 172.21.15.1:0/275623264 learned_addr learned my addr 172.21.15.1:0/275623264 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.866+0000 7fd31963f700 1 --2- 172.21.15.1:0/275623264 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd300003a00 0x7fd3140c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.866+0000 7fd319e40700 1 --2- 172.21.15.1:0/275623264 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3141e3a90 0x7fd3141e3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.867+0000 7fd318e3e700 1 -- 172.21.15.1:0/275623264 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3141e3a90 msgr2=0x7fd3141e3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.867+0000 7fd318e3e700 1 --2- 172.21.15.1:0/275623264 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3141e3a90 0x7fd3141e3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.867+0000 7fd318e3e700 1 -- 172.21.15.1:0/275623264 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd300003a00 msgr2=0x7fd3140c6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.867+0000 7fd318e3e700 1 --2- 172.21.15.1:0/275623264 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd300003a00 0x7fd3140c6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.867+0000 7fd318e3e700 1 -- 172.21.15.1:0/275623264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3141e47e0 con 0x7fd3141f2a70 2024-03-18T20:06:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.867+0000 7fd318e3e700 1 --2- 172.21.15.1:0/275623264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3141f2a70 0x7fd3141f2ef0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fd3040115c0 tx=0x7fd3040118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.867+0000 7fd309ffb700 1 -- 172.21.15.1:0/275623264 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd304004220 con 0x7fd3141f2a70 2024-03-18T20:06:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.867+0000 7fd309ffb700 1 -- 172.21.15.1:0/275623264 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3040043a0 con 0x7fd3141f2a70 2024-03-18T20:06:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.867+0000 7fd309ffb700 1 -- 172.21.15.1:0/275623264 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3040038e0 con 0x7fd3141f2a70 2024-03-18T20:06:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.867+0000 7fd31b8a3700 1 -- 172.21.15.1:0/275623264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3140c4220 con 0x7fd3141f2a70 2024-03-18T20:06:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.867+0000 7fd31b8a3700 1 -- 172.21.15.1:0/275623264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3141e4450 con 0x7fd3141f2a70 2024-03-18T20:06:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.868+0000 7fd31b8a3700 1 -- 172.21.15.1:0/275623264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3140c3de0 con 0x7fd3141f2a70 2024-03-18T20:06:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.869+0000 7fd309ffb700 1 -- 172.21.15.1:0/275623264 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 93) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fd304003a40 con 0x7fd3141f2a70 2024-03-18T20:06:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.869+0000 7fd309ffb700 1 --2- 172.21.15.1:0/275623264 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd2f806aae0 0x7fd2f806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.870+0000 7fd31963f700 1 --2- 172.21.15.1:0/275623264 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd2f806aae0 0x7fd2f806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.870+0000 7fd309ffb700 1 -- 172.21.15.1:0/275623264 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6789+0+0 (secure 0 0 0) 0x7fd304089860 con 0x7fd3141f2a70 2024-03-18T20:06:28.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.870+0000 7fd31963f700 1 --2- 172.21.15.1:0/275623264 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd2f806aae0 0x7fd2f806cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd310001860 tx=0x7fd310009030 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:28.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:28.874+0000 7fd309ffb700 1 -- 172.21.15.1:0/275623264 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd3040548f0 con 0x7fd3141f2a70 2024-03-18T20:06:29.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.122+0000 7fd31b8a3700 1 -- 172.21.15.1:0/275623264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd3140c3de0 con 0x7fd3141f2a70 2024-03-18T20:06:29.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.773+0000 7fd309ffb700 1 -- 172.21.15.1:0/275623264 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v75) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd3140c3de0 con 0x7fd3141f2a70 2024-03-18T20:06:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.776+0000 7fd31b8a3700 1 -- 172.21.15.1:0/275623264 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd2f806aae0 msgr2=0x7fd2f806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.776+0000 7fd31b8a3700 1 --2- 172.21.15.1:0/275623264 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd2f806aae0 0x7fd2f806cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd310001860 tx=0x7fd310009030 comp rx=0 tx=0).stop 2024-03-18T20:06:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.776+0000 7fd31b8a3700 1 -- 172.21.15.1:0/275623264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3141f2a70 msgr2=0x7fd3141f2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:29.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.776+0000 7fd31b8a3700 1 --2- 172.21.15.1:0/275623264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3141f2a70 0x7fd3141f2ef0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fd3040115c0 tx=0x7fd3040118d0 comp rx=0 tx=0).stop 2024-03-18T20:06:29.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.776+0000 7fd31b8a3700 1 -- 172.21.15.1:0/275623264 shutdown_connections 2024-03-18T20:06:29.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.776+0000 7fd31b8a3700 1 --2- 172.21.15.1:0/275623264 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd2f806aae0 0x7fd2f806cfa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:29.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.776+0000 7fd31b8a3700 1 --2- 172.21.15.1:0/275623264 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3141e3a90 0x7fd3141e3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:29.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.776+0000 7fd31b8a3700 1 --2- 172.21.15.1:0/275623264 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd300003a00 0x7fd3140c6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:29.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.776+0000 7fd31b8a3700 1 --2- 172.21.15.1:0/275623264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3141f2a70 0x7fd3141f2ef0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:29.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.776+0000 7fd31b8a3700 1 -- 172.21.15.1:0/275623264 >> 172.21.15.1:0/275623264 conn(0x7fd3140cc9b0 msgr2=0x7fd31406bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:29.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.777+0000 7fd31b8a3700 1 -- 172.21.15.1:0/275623264 shutdown_connections 2024-03-18T20:06:29.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.777+0000 7fd31b8a3700 1 -- 172.21.15.1:0/275623264 wait complete. 2024-03-18T20:06:29.797 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T20:06:29.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.968+0000 7f6691b0b700 1 Processor -- start 2024-03-18T20:06:29.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.968+0000 7f6691b0b700 1 -- start start 2024-03-18T20:06:29.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.969+0000 7f6691b0b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f668c144b30 0x7f668c0723f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:29.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.969+0000 7f6691b0b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f668c15c050 0x7f668c0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:29.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.969+0000 7f6691b0b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f668c0d0ee0 con 0x7f668c0ccd50 2024-03-18T20:06:29.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.969+0000 7f6691b0b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f668c0cd720 con 0x7f668c144b30 2024-03-18T20:06:29.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.969+0000 7f6691b0b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f668c15c8b0 con 0x7f668c15c050 2024-03-18T20:06:29.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.969+0000 7f668affd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f668c0ccd50 0x7f668c15b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59208/0 (socket says 172.21.15.1:59208) 2024-03-18T20:06:29.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.969+0000 7f668affd700 1 -- 172.21.15.1:0/429793087 learned_addr learned my addr 172.21.15.1:0/429793087 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:29.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.969+0000 7f668b7fe700 1 --2- 172.21.15.1:0/429793087 >> v2:172.21.15.165:3300/0 conn(0x7f668c144b30 0x7f668c0723f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:29.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668bfff700 1 --2- 172.21.15.1:0/429793087 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f668c15c050 0x7f668c0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:29.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668a7fc700 1 -- 172.21.15.1:0/429793087 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443099303 0 0) 0x7f668c0d0ee0 con 0x7f668c0ccd50 2024-03-18T20:06:29.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668a7fc700 1 -- 172.21.15.1:0/429793087 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66780035b0 con 0x7f668c0ccd50 2024-03-18T20:06:29.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668a7fc700 1 -- 172.21.15.1:0/429793087 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 586521073 0 0) 0x7f66780035b0 con 0x7f668c0ccd50 2024-03-18T20:06:29.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668a7fc700 1 -- 172.21.15.1:0/429793087 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f668c15c050 msgr2=0x7f668c0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:29.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668a7fc700 1 --2- 172.21.15.1:0/429793087 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f668c15c050 0x7f668c0cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:29.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668a7fc700 1 -- 172.21.15.1:0/429793087 >> v2:172.21.15.165:3300/0 conn(0x7f668c144b30 msgr2=0x7f668c0723f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:29.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668a7fc700 1 --2- 172.21.15.1:0/429793087 >> v2:172.21.15.165:3300/0 conn(0x7f668c144b30 0x7f668c0723f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:29.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668a7fc700 1 -- 172.21.15.1:0/429793087 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668c15c5d0 con 0x7f668c0ccd50 2024-03-18T20:06:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668a7fc700 1 -- 172.21.15.1:0/429793087 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f667c001670 con 0x7f668c0ccd50 2024-03-18T20:06:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668a7fc700 1 -- 172.21.15.1:0/429793087 >> v1:172.21.15.1:6789/0 conn(0x7f668c0ccd50 legacy=0x7f668c15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668a7fc700 1 --2- 172.21.15.1:0/429793087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6678003a00 0x7f6678003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668a7fc700 1 --2- 172.21.15.1:0/429793087 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6678004850 0x7f6678004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668a7fc700 1 --2- 172.21.15.1:0/429793087 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f668c15c050 0x7f6678005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668a7fc700 1 -- 172.21.15.1:0/429793087 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668c0cd720 con 0x7f6678003a00 2024-03-18T20:06:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668a7fc700 1 -- 172.21.15.1:0/429793087 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668c15c8b0 con 0x7f6678004850 2024-03-18T20:06:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668a7fc700 1 -- 172.21.15.1:0/429793087 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f667800ba40 con 0x7f668c15c050 2024-03-18T20:06:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668b7fe700 1 --2- 172.21.15.1:0/429793087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6678003a00 0x7f6678003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668affd700 1 --2- 172.21.15.1:0/429793087 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6678004850 0x7f6678004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668bfff700 1 --2- 172.21.15.1:0/429793087 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f668c15c050 0x7f6678005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668b7fe700 1 -- 172.21.15.1:0/429793087 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f668c15c050 msgr2=0x7f6678005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668b7fe700 1 --2- 172.21.15.1:0/429793087 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f668c15c050 0x7f6678005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:29.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668b7fe700 1 -- 172.21.15.1:0/429793087 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6678004850 msgr2=0x7f6678004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668b7fe700 1 --2- 172.21.15.1:0/429793087 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6678004850 0x7f6678004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668b7fe700 1 -- 172.21.15.1:0/429793087 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66780035b0 con 0x7f6678003a00 2024-03-18T20:06:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.970+0000 7f668bfff700 1 --2- 172.21.15.1:0/429793087 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f668c15c050 0x7f6678005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.971+0000 7f668b7fe700 1 --2- 172.21.15.1:0/429793087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6678003a00 0x7f6678003e20 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f6674009b60 tx=0x7f6678005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9533cfbfa44b7071 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.971+0000 7f668a7fc700 1 -- 172.21.15.1:0/429793087 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667400e660 con 0x7f6678003a00 2024-03-18T20:06:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.971+0000 7f668a7fc700 1 -- 172.21.15.1:0/429793087 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667400e7c0 con 0x7f6678003a00 2024-03-18T20:06:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.971+0000 7f668a7fc700 1 -- 172.21.15.1:0/429793087 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667400eb30 con 0x7f6678003a00 2024-03-18T20:06:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.971+0000 7f6691b0b700 1 -- 172.21.15.1:0/429793087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6678003a00 msgr2=0x7f6678003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.971+0000 7f6691b0b700 1 --2- 172.21.15.1:0/429793087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6678003a00 0x7f6678003e20 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f6674009b60 tx=0x7f6678005880 comp rx=0 tx=0).stop 2024-03-18T20:06:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.971+0000 7f668bfff700 1 -- 172.21.15.1:0/429793087 reap_dead start 2024-03-18T20:06:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.971+0000 7f6691b0b700 1 -- 172.21.15.1:0/429793087 shutdown_connections 2024-03-18T20:06:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.971+0000 7f6691b0b700 1 -- 172.21.15.1:0/429793087 >> 172.21.15.1:0/429793087 conn(0x7f668c0d00c0 msgr2=0x7f668c0d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.971+0000 7f6691b0b700 1 -- 172.21.15.1:0/429793087 shutdown_connections 2024-03-18T20:06:29.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.971+0000 7f6691b0b700 1 -- 172.21.15.1:0/429793087 wait complete. 2024-03-18T20:06:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.972+0000 7f6691b0b700 1 Processor -- start 2024-03-18T20:06:29.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.973+0000 7f6691b0b700 1 -- start start 2024-03-18T20:06:29.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.973+0000 7f6691b0b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6678003a00 0x7f668c15afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:29.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.973+0000 7f6691b0b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f668c0cb890 0x7f668c0cbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:29.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.973+0000 7f6691b0b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f668c1417f0 0x7f668c141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.973+0000 7f6691b0b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668c1421e0 con 0x7f6678003a00 2024-03-18T20:06:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.973+0000 7f6691b0b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668c0ca170 con 0x7f668c0cb890 2024-03-18T20:06:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.973+0000 7f6691b0b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f668c15b4e0 con 0x7f668c1417f0 2024-03-18T20:06:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.973+0000 7f668b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6678003a00 0x7f668c15afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.973+0000 7f668b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6678003a00 0x7f668c15afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33662/0 (socket says 172.21.15.1:33662) 2024-03-18T20:06:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.973+0000 7f668b7fe700 1 -- 172.21.15.1:0/861910692 learned_addr learned my addr 172.21.15.1:0/861910692 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.974+0000 7f668affd700 1 --2- 172.21.15.1:0/861910692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f668c0cb890 0x7f668c0cbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.974+0000 7f668bfff700 1 --2- 172.21.15.1:0/861910692 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f668c1417f0 0x7f668c141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.974+0000 7f668b7fe700 1 -- 172.21.15.1:0/861910692 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f668c1417f0 msgr2=0x7f668c141ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.974+0000 7f668b7fe700 1 --2- 172.21.15.1:0/861910692 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f668c1417f0 0x7f668c141ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.974+0000 7f668b7fe700 1 -- 172.21.15.1:0/861910692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f668c0cb890 msgr2=0x7f668c0cbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.974+0000 7f668b7fe700 1 --2- 172.21.15.1:0/861910692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f668c0cb890 0x7f668c0cbd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.974+0000 7f668b7fe700 1 -- 172.21.15.1:0/861910692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668c0c9560 con 0x7f6678003a00 2024-03-18T20:06:29.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.974+0000 7f668affd700 1 --2- 172.21.15.1:0/861910692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f668c0cb890 0x7f668c0cbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.974+0000 7f668bfff700 1 --2- 172.21.15.1:0/861910692 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f668c1417f0 0x7f668c141ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.974+0000 7f668b7fe700 1 --2- 172.21.15.1:0/861910692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6678003a00 0x7f668c15afa0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f66740042f0 tx=0x7f66740047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.974+0000 7f6683fff700 1 -- 172.21.15.1:0/861910692 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667400e660 con 0x7f6678003a00 2024-03-18T20:06:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.974+0000 7f6683fff700 1 -- 172.21.15.1:0/861910692 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6674003ab0 con 0x7f6678003a00 2024-03-18T20:06:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.974+0000 7f6691b0b700 1 -- 172.21.15.1:0/861910692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f668c0c9840 con 0x7f6678003a00 2024-03-18T20:06:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.974+0000 7f6683fff700 1 -- 172.21.15.1:0/861910692 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6674003e60 con 0x7f6678003a00 2024-03-18T20:06:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.974+0000 7f6691b0b700 1 -- 172.21.15.1:0/861910692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f668c0c9e00 con 0x7f6678003a00 2024-03-18T20:06:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.975+0000 7f6691b0b700 1 -- 172.21.15.1:0/861910692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f668c0cd9a0 con 0x7f6678003a00 2024-03-18T20:06:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.976+0000 7f6683fff700 1 -- 172.21.15.1:0/861910692 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 94) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f6674013470 con 0x7f6678003a00 2024-03-18T20:06:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.976+0000 7f6683fff700 1 --2- 172.21.15.1:0/861910692 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f666c06ad20 0x7f666c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.976+0000 7f6683fff700 1 -- 172.21.15.1:0/861910692 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6789+0+0 (secure 0 0 0) 0x7f667408afa0 con 0x7f6678003a00 2024-03-18T20:06:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.976+0000 7f668affd700 1 --2- 172.21.15.1:0/861910692 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f666c06ad20 0x7f666c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:29.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.976+0000 7f668affd700 1 --2- 172.21.15.1:0/861910692 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f666c06ad20 0x7f666c06d1e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f668c1e4dc0 tx=0x7f667c009040 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:29.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:29.981+0000 7f6683fff700 1 -- 172.21.15.1:0/861910692 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6674055f00 con 0x7f6678003a00 2024-03-18T20:06:30.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:30.179+0000 7f6691b0b700 1 -- 172.21.15.1:0/861910692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f668c0734a0 con 0x7f6678003a00 2024-03-18T20:06:30.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:30.877+0000 7f6683fff700 1 -- 172.21.15.1:0/861910692 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v171) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6674059520 con 0x7f6678003a00 2024-03-18T20:06:30.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:30.880+0000 7f6691b0b700 1 -- 172.21.15.1:0/861910692 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f666c06ad20 msgr2=0x7f666c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:30.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:30.880+0000 7f6691b0b700 1 --2- 172.21.15.1:0/861910692 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f666c06ad20 0x7f666c06d1e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f668c1e4dc0 tx=0x7f667c009040 comp rx=0 tx=0).stop 2024-03-18T20:06:30.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:30.880+0000 7f6691b0b700 1 -- 172.21.15.1:0/861910692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6678003a00 msgr2=0x7f668c15afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:30.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:30.880+0000 7f6691b0b700 1 --2- 172.21.15.1:0/861910692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6678003a00 0x7f668c15afa0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f66740042f0 tx=0x7f66740047c0 comp rx=0 tx=0).stop 2024-03-18T20:06:30.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:30.880+0000 7f6691b0b700 1 -- 172.21.15.1:0/861910692 shutdown_connections 2024-03-18T20:06:30.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:30.880+0000 7f6691b0b700 1 --2- 172.21.15.1:0/861910692 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f666c06ad20 0x7f666c06d1e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:30.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:30.880+0000 7f6691b0b700 1 --2- 172.21.15.1:0/861910692 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f668c1417f0 0x7f668c141ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:30.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:30.880+0000 7f6691b0b700 1 --2- 172.21.15.1:0/861910692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6678003a00 0x7f668c15afa0 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:30.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:30.880+0000 7f6691b0b700 1 --2- 172.21.15.1:0/861910692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f668c0cb890 0x7f668c0cbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:30.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:30.880+0000 7f6691b0b700 1 -- 172.21.15.1:0/861910692 >> 172.21.15.1:0/861910692 conn(0x7f668c0d00c0 msgr2=0x7f668c06ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:30.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:30.880+0000 7f6691b0b700 1 -- 172.21.15.1:0/861910692 shutdown_connections 2024-03-18T20:06:30.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:30.880+0000 7f6691b0b700 1 -- 172.21.15.1:0/861910692 wait complete. 2024-03-18T20:06:30.886 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-18T20:06:30.903 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T20:06:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.035+0000 7f29e6e2f700 1 Processor -- start 2024-03-18T20:06:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.035+0000 7f29e6e2f700 1 -- start start 2024-03-18T20:06:31.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.036+0000 7f29e6e2f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29e0153f90 0x7f29e00b2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.036+0000 7f29e6e2f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f29e015ac20 0x7f29e00b3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.036+0000 7f29e6e2f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f29e00b2da0 0x7f29e00b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.036+0000 7f29e6e2f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e0157b80 con 0x7f29e00b2da0 2024-03-18T20:06:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.036+0000 7f29e6e2f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e015d5d0 con 0x7f29e015ac20 2024-03-18T20:06:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.036+0000 7f29e6e2f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e015d8b0 con 0x7f29e0153f90 2024-03-18T20:06:31.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.036+0000 7f29e53cc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f29e00b2da0 0x7f29e00b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:31.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.036+0000 7f29e53cc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f29e00b2da0 0x7f29e00b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33668/0 (socket says 172.21.15.1:33668) 2024-03-18T20:06:31.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.036+0000 7f29e53cc700 1 -- 172.21.15.1:0/4016326068 learned_addr learned my addr 172.21.15.1:0/4016326068 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:31.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.036+0000 7f29e4bcb700 1 --2- 172.21.15.1:0/4016326068 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29e0153f90 0x7f29e00b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:31.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.036+0000 7f29dffff700 1 --2- 172.21.15.1:0/4016326068 >> v2:172.21.15.165:3300/0 conn(0x7f29e015ac20 0x7f29e00b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:31.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.037+0000 7f29e53cc700 1 -- 172.21.15.1:0/4016326068 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29e0153f90 msgr2=0x7f29e00b2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:31.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.037+0000 7f29e53cc700 1 --2- 172.21.15.1:0/4016326068 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29e0153f90 0x7f29e00b2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:31.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.037+0000 7f29e53cc700 1 -- 172.21.15.1:0/4016326068 >> v2:172.21.15.165:3300/0 conn(0x7f29e015ac20 msgr2=0x7f29e00b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:31.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.037+0000 7f29e53cc700 1 --2- 172.21.15.1:0/4016326068 >> v2:172.21.15.165:3300/0 conn(0x7f29e015ac20 0x7f29e00b3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:31.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.037+0000 7f29e53cc700 1 -- 172.21.15.1:0/4016326068 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e015db90 con 0x7f29e00b2da0 2024-03-18T20:06:31.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.037+0000 7f29dffff700 1 --2- 172.21.15.1:0/4016326068 >> v2:172.21.15.165:3300/0 conn(0x7f29e015ac20 0x7f29e00b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:31.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.037+0000 7f29e53cc700 1 --2- 172.21.15.1:0/4016326068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e00b2da0 0x7f29e00b38b0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f29e0067220 tx=0x7f29d800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7dd20f88dce1f3e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.037+0000 7f29df7fe700 1 -- 172.21.15.1:0/4016326068 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d8010070 con 0x7f29e00b2da0 2024-03-18T20:06:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.037+0000 7f29df7fe700 1 -- 172.21.15.1:0/4016326068 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d800db80 con 0x7f29e00b2da0 2024-03-18T20:06:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.037+0000 7f29df7fe700 1 -- 172.21.15.1:0/4016326068 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d8016400 con 0x7f29e00b2da0 2024-03-18T20:06:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.037+0000 7f29e6e2f700 1 -- 172.21.15.1:0/4016326068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e00b2da0 msgr2=0x7f29e00b38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.037+0000 7f29e6e2f700 1 --2- 172.21.15.1:0/4016326068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e00b2da0 0x7f29e00b38b0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f29e0067220 tx=0x7f29d800d430 comp rx=0 tx=0).stop 2024-03-18T20:06:31.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.037+0000 7f29e6e2f700 1 -- 172.21.15.1:0/4016326068 shutdown_connections 2024-03-18T20:06:31.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.037+0000 7f29e6e2f700 1 --2- 172.21.15.1:0/4016326068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e00b2da0 0x7f29e00b38b0 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:31.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.037+0000 7f29e6e2f700 1 --2- 172.21.15.1:0/4016326068 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29e0153f90 0x7f29e00b2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:31.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.037+0000 7f29e6e2f700 1 --2- 172.21.15.1:0/4016326068 >> v2:172.21.15.165:3300/0 conn(0x7f29e015ac20 0x7f29e00b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:31.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.037+0000 7f29e6e2f700 1 -- 172.21.15.1:0/4016326068 >> 172.21.15.1:0/4016326068 conn(0x7f29e006a750 msgr2=0x7f29e006ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:31.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.038+0000 7f29e6e2f700 1 -- 172.21.15.1:0/4016326068 shutdown_connections 2024-03-18T20:06:31.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.038+0000 7f29e6e2f700 1 -- 172.21.15.1:0/4016326068 wait complete. 2024-03-18T20:06:31.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.039+0000 7f29e6e2f700 1 Processor -- start 2024-03-18T20:06:31.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.039+0000 7f29e6e2f700 1 -- start start 2024-03-18T20:06:31.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.040+0000 7f29e6e2f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29e00b2da0 0x7f29e014af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:31.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.040+0000 7f29e6e2f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0153f90 0x7f29e014f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:31.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.040+0000 7f29e6e2f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29e015ac20 0x7f29e0155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:31.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.040+0000 7f29e6e2f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e0155870 con 0x7f29e0153f90 2024-03-18T20:06:31.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.040+0000 7f29e6e2f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e014b440 con 0x7f29e00b2da0 2024-03-18T20:06:31.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.040+0000 7f29e6e2f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e0157310 con 0x7f29e015ac20 2024-03-18T20:06:31.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.040+0000 7f29dffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0153f90 0x7f29e014f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:31.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.040+0000 7f29dffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0153f90 0x7f29e014f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33680/0 (socket says 172.21.15.1:33680) 2024-03-18T20:06:31.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.040+0000 7f29dffff700 1 -- 172.21.15.1:0/2352348668 learned_addr learned my addr 172.21.15.1:0/2352348668 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:31.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.041+0000 7f29e4bcb700 1 --2- 172.21.15.1:0/2352348668 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29e00b2da0 0x7f29e014af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:31.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.041+0000 7f29e53cc700 1 --2- 172.21.15.1:0/2352348668 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29e015ac20 0x7f29e0155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:31.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.041+0000 7f29dffff700 1 -- 172.21.15.1:0/2352348668 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29e015ac20 msgr2=0x7f29e0155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:31.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.041+0000 7f29dffff700 1 --2- 172.21.15.1:0/2352348668 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29e015ac20 0x7f29e0155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:31.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.041+0000 7f29dffff700 1 -- 172.21.15.1:0/2352348668 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29e00b2da0 msgr2=0x7f29e014af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:31.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.041+0000 7f29dffff700 1 --2- 172.21.15.1:0/2352348668 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29e00b2da0 0x7f29e014af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:31.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.041+0000 7f29dffff700 1 -- 172.21.15.1:0/2352348668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d800bd40 con 0x7f29e0153f90 2024-03-18T20:06:31.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.041+0000 7f29dffff700 1 --2- 172.21.15.1:0/2352348668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0153f90 0x7f29e014f9e0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f29cc00c7f0 tx=0x7f29cc00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:31.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.041+0000 7f29dd7fa700 1 -- 172.21.15.1:0/2352348668 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29cc011800 con 0x7f29e0153f90 2024-03-18T20:06:31.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.041+0000 7f29dd7fa700 1 -- 172.21.15.1:0/2352348668 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29cc011960 con 0x7f29e0153f90 2024-03-18T20:06:31.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.041+0000 7f29dd7fa700 1 -- 172.21.15.1:0/2352348668 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29cc011d10 con 0x7f29e0153f90 2024-03-18T20:06:31.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.041+0000 7f29e6e2f700 1 -- 172.21.15.1:0/2352348668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e00b0570 con 0x7f29e0153f90 2024-03-18T20:06:31.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.041+0000 7f29e6e2f700 1 -- 172.21.15.1:0/2352348668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e00b0b30 con 0x7f29e0153f90 2024-03-18T20:06:31.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.042+0000 7f29e6e2f700 1 -- 172.21.15.1:0/2352348668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e00b0d50 con 0x7f29e0153f90 2024-03-18T20:06:31.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.043+0000 7f29dd7fa700 1 -- 172.21.15.1:0/2352348668 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 94) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f29cc018360 con 0x7f29e0153f90 2024-03-18T20:06:31.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.043+0000 7f29dd7fa700 1 --2- 172.21.15.1:0/2352348668 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f29c406ad20 0x7f29c406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:31.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.043+0000 7f29dd7fa700 1 -- 172.21.15.1:0/2352348668 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6351+0+0 (secure 0 0 0) 0x7f29cc089650 con 0x7f29e0153f90 2024-03-18T20:06:31.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.043+0000 7f29e4bcb700 1 --2- 172.21.15.1:0/2352348668 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f29c406ad20 0x7f29c406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:31.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.044+0000 7f29e4bcb700 1 --2- 172.21.15.1:0/2352348668 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f29c406ad20 0x7f29c406d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f29d400c900 tx=0x7f29d4008040 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:31.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.048+0000 7f29dd7fa700 1 -- 172.21.15.1:0/2352348668 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f29cc0544b0 con 0x7f29e0153f90 2024-03-18T20:06:31.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.252+0000 7f29e6e2f700 1 -- 172.21.15.1:0/2352348668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f29e00676a0 con 0x7f29e0153f90 2024-03-18T20:06:31.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.891+0000 7f29dd7fa700 1 -- 172.21.15.1:0/2352348668 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f29cc05e120 con 0x7f29e0153f90 2024-03-18T20:06:31.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.892+0000 7f29dd7fa700 1 -- 172.21.15.1:0/2352348668 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v172) v1 ==== 169+0+0 (secure 0 0 0) 0x7f29cc0544b0 con 0x7f29e0153f90 2024-03-18T20:06:31.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.895+0000 7f29e6e2f700 1 -- 172.21.15.1:0/2352348668 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f29c406ad20 msgr2=0x7f29c406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:31.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.895+0000 7f29e6e2f700 1 --2- 172.21.15.1:0/2352348668 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f29c406ad20 0x7f29c406d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f29d400c900 tx=0x7f29d4008040 comp rx=0 tx=0).stop 2024-03-18T20:06:31.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.895+0000 7f29e6e2f700 1 -- 172.21.15.1:0/2352348668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0153f90 msgr2=0x7f29e014f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:31.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.895+0000 7f29e6e2f700 1 --2- 172.21.15.1:0/2352348668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0153f90 0x7f29e014f9e0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f29cc00c7f0 tx=0x7f29cc00cbb0 comp rx=0 tx=0).stop 2024-03-18T20:06:31.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.895+0000 7f29e6e2f700 1 -- 172.21.15.1:0/2352348668 shutdown_connections 2024-03-18T20:06:31.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.895+0000 7f29e6e2f700 1 --2- 172.21.15.1:0/2352348668 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f29c406ad20 0x7f29c406d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:31.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.895+0000 7f29e6e2f700 1 --2- 172.21.15.1:0/2352348668 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29e015ac20 0x7f29e0155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:31.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.895+0000 7f29e6e2f700 1 --2- 172.21.15.1:0/2352348668 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29e00b2da0 0x7f29e014af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:31.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.895+0000 7f29e6e2f700 1 --2- 172.21.15.1:0/2352348668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e0153f90 0x7f29e014f9e0 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:31.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.895+0000 7f29e6e2f700 1 -- 172.21.15.1:0/2352348668 >> 172.21.15.1:0/2352348668 conn(0x7f29e006a750 msgr2=0x7f29e00b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:31.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.896+0000 7f29e6e2f700 1 -- 172.21.15.1:0/2352348668 shutdown_connections 2024-03-18T20:06:31.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:31.896+0000 7f29e6e2f700 1 -- 172.21.15.1:0/2352348668 wait complete. 2024-03-18T20:06:31.901 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-18T20:06:31.916 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-18T20:06:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.045+0000 7f238b8b8700 1 Processor -- start 2024-03-18T20:06:32.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.045+0000 7f238b8b8700 1 -- start start 2024-03-18T20:06:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.046+0000 7f238b8b8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f238406a750 0x7f238406abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.046+0000 7f238b8b8700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2384160ea0 con 0x7f23840727b0 2024-03-18T20:06:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.046+0000 7f238b8b8700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238415e090 con 0x7f238415ad70 2024-03-18T20:06:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.046+0000 7f238b8b8700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23840725b0 con 0x7f238406a750 2024-03-18T20:06:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.046+0000 7f2389e55700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f238406a750 0x7f238406abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.046+0000 7f2388e53700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f23840727b0 0x7f2384161610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59218/0 (socket says 172.21.15.1:59218) 2024-03-18T20:06:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.046+0000 7f2388e53700 1 -- 172.21.15.1:0/3549958079 learned_addr learned my addr 172.21.15.1:0/3549958079 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.046+0000 7f2389e55700 1 -- 172.21.15.1:0/3549958079 >> v1:172.21.15.165:6789/0 conn(0x7f238415ad70 legacy=0x7f238415d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.046+0000 7f2389e55700 1 -- 172.21.15.1:0/3549958079 >> v1:172.21.15.1:6789/0 conn(0x7f23840727b0 legacy=0x7f2384161610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.046+0000 7f2389e55700 1 -- 172.21.15.1:0/3549958079 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238415e370 con 0x7f238406a750 2024-03-18T20:06:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.046+0000 7f237bfff700 1 -- 172.21.15.1:0/3549958079 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454836232 0 0) 0x7f2384160ea0 con 0x7f23840727b0 2024-03-18T20:06:32.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.046+0000 7f2389e55700 1 --2- 172.21.15.1:0/3549958079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f238406a750 0x7f238406abc0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f237c009d90 tx=0x7f237c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2873ec420368bddf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:32.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.046+0000 7f237bfff700 1 -- 172.21.15.1:0/3549958079 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c00e070 con 0x7f238406a750 2024-03-18T20:06:32.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.047+0000 7f237bfff700 1 -- 172.21.15.1:0/3549958079 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237c00bb90 con 0x7f238406a750 2024-03-18T20:06:32.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.047+0000 7f237bfff700 1 -- 172.21.15.1:0/3549958079 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c003a20 con 0x7f238406a750 2024-03-18T20:06:32.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.047+0000 7f238b8b8700 1 -- 172.21.15.1:0/3549958079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f238406a750 msgr2=0x7f238406abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:32.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.047+0000 7f238b8b8700 1 --2- 172.21.15.1:0/3549958079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f238406a750 0x7f238406abc0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f237c009d90 tx=0x7f237c00b330 comp rx=0 tx=0).stop 2024-03-18T20:06:32.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.047+0000 7f238b8b8700 1 -- 172.21.15.1:0/3549958079 shutdown_connections 2024-03-18T20:06:32.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.047+0000 7f238b8b8700 1 --2- 172.21.15.1:0/3549958079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f238406a750 0x7f238406abc0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.047+0000 7f238b8b8700 1 -- 172.21.15.1:0/3549958079 >> 172.21.15.1:0/3549958079 conn(0x7f23840b1cb0 msgr2=0x7f23840b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:32.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.047+0000 7f238b8b8700 1 -- 172.21.15.1:0/3549958079 shutdown_connections 2024-03-18T20:06:32.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.047+0000 7f238b8b8700 1 -- 172.21.15.1:0/3549958079 wait complete. 2024-03-18T20:06:32.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.048+0000 7f238b8b8700 1 Processor -- start 2024-03-18T20:06:32.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.048+0000 7f238b8b8700 1 -- start start 2024-03-18T20:06:32.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.049+0000 7f238b8b8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f238406a750 0x7f23840ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:32.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.049+0000 7f238b8b8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23840727b0 0x7f23840acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:32.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.049+0000 7f238b8b8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f238415ad70 0x7f23840aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:32.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.049+0000 7f238b8b8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23841f32c0 con 0x7f23840727b0 2024-03-18T20:06:32.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.049+0000 7f238b8b8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23840ad700 con 0x7f238415ad70 2024-03-18T20:06:32.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.049+0000 7f238b8b8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23840ad420 con 0x7f238406a750 2024-03-18T20:06:32.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.049+0000 7f2388e53700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23840727b0 0x7f23840acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:32.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.049+0000 7f2388e53700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23840727b0 0x7f23840acdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33698/0 (socket says 172.21.15.1:33698) 2024-03-18T20:06:32.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.049+0000 7f2388e53700 1 -- 172.21.15.1:0/4282924965 learned_addr learned my addr 172.21.15.1:0/4282924965 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:32.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.049+0000 7f2389654700 1 --2- 172.21.15.1:0/4282924965 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f238406a750 0x7f23840ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:32.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.049+0000 7f2389e55700 1 --2- 172.21.15.1:0/4282924965 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f238415ad70 0x7f23840aebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:32.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.049+0000 7f2388e53700 1 -- 172.21.15.1:0/4282924965 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f238406a750 msgr2=0x7f23840ac8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:32.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.049+0000 7f2388e53700 1 --2- 172.21.15.1:0/4282924965 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f238406a750 0x7f23840ac8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.049+0000 7f2388e53700 1 -- 172.21.15.1:0/4282924965 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f238415ad70 msgr2=0x7f23840aebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:32.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.049+0000 7f2388e53700 1 --2- 172.21.15.1:0/4282924965 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f238415ad70 0x7f23840aebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.049+0000 7f2388e53700 1 -- 172.21.15.1:0/4282924965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c009a70 con 0x7f23840727b0 2024-03-18T20:06:32.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.050+0000 7f2389654700 1 --2- 172.21.15.1:0/4282924965 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f238406a750 0x7f23840ac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:32.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.050+0000 7f2389e55700 1 --2- 172.21.15.1:0/4282924965 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f238415ad70 0x7f23840aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:32.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.050+0000 7f2388e53700 1 --2- 172.21.15.1:0/4282924965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23840727b0 0x7f23840acdf0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f237400c520 tx=0x7f237400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:32.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.050+0000 7f2379ffb700 1 -- 172.21.15.1:0/4282924965 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2374004c80 con 0x7f23840727b0 2024-03-18T20:06:32.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.050+0000 7f2379ffb700 1 -- 172.21.15.1:0/4282924965 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2374004e00 con 0x7f23840727b0 2024-03-18T20:06:32.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.050+0000 7f2379ffb700 1 -- 172.21.15.1:0/4282924965 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2374007310 con 0x7f23840727b0 2024-03-18T20:06:32.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.050+0000 7f238b8b8700 1 -- 172.21.15.1:0/4282924965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23840cd400 con 0x7f23840727b0 2024-03-18T20:06:32.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.050+0000 7f238b8b8700 1 -- 172.21.15.1:0/4282924965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23840cd9c0 con 0x7f23840727b0 2024-03-18T20:06:32.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.051+0000 7f238b8b8700 1 -- 172.21.15.1:0/4282924965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23840cd260 con 0x7f23840727b0 2024-03-18T20:06:32.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.051+0000 7f2379ffb700 1 -- 172.21.15.1:0/4282924965 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2374007470 con 0x7f23840727b0 2024-03-18T20:06:32.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.052+0000 7f2379ffb700 1 --2- 172.21.15.1:0/4282924965 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f236806adb0 0x7f236806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:32.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.052+0000 7f2379ffb700 1 -- 172.21.15.1:0/4282924965 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f2374089240 con 0x7f23840727b0 2024-03-18T20:06:32.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.052+0000 7f2389654700 1 --2- 172.21.15.1:0/4282924965 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f236806adb0 0x7f236806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:32.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.052+0000 7f2389654700 1 --2- 172.21.15.1:0/4282924965 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f236806adb0 0x7f236806d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2380006fd0 tx=0x7f2380008040 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:32.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.056+0000 7f2379ffb700 1 -- 172.21.15.1:0/4282924965 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f23840cd260 con 0x7f23840727b0 2024-03-18T20:06:32.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.256+0000 7f238b8b8700 1 -- 172.21.15.1:0/4282924965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f23840cd260 con 0x7f23840727b0 2024-03-18T20:06:32.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.260+0000 7f2379ffb700 1 -- 172.21.15.1:0/4282924965 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v55) v1 ==== 115+0+0 (secure 0 0 0) 0x7f23840cd260 con 0x7f23840727b0 2024-03-18T20:06:32.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.263+0000 7f238b8b8700 1 -- 172.21.15.1:0/4282924965 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f236806adb0 msgr2=0x7f236806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:32.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.263+0000 7f238b8b8700 1 --2- 172.21.15.1:0/4282924965 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f236806adb0 0x7f236806d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2380006fd0 tx=0x7f2380008040 comp rx=0 tx=0).stop 2024-03-18T20:06:32.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.263+0000 7f238b8b8700 1 -- 172.21.15.1:0/4282924965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23840727b0 msgr2=0x7f23840acdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:32.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.263+0000 7f238b8b8700 1 --2- 172.21.15.1:0/4282924965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23840727b0 0x7f23840acdf0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f237400c520 tx=0x7f237400c8e0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.263+0000 7f238b8b8700 1 -- 172.21.15.1:0/4282924965 shutdown_connections 2024-03-18T20:06:32.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.263+0000 7f238b8b8700 1 --2- 172.21.15.1:0/4282924965 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f236806adb0 0x7f236806d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.263+0000 7f238b8b8700 1 --2- 172.21.15.1:0/4282924965 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f238406a750 0x7f23840ac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.263+0000 7f238b8b8700 1 --2- 172.21.15.1:0/4282924965 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f238415ad70 0x7f23840aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.263+0000 7f238b8b8700 1 --2- 172.21.15.1:0/4282924965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23840727b0 0x7f23840acdf0 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.263+0000 7f238b8b8700 1 -- 172.21.15.1:0/4282924965 >> 172.21.15.1:0/4282924965 conn(0x7f23840b1cb0 msgr2=0x7f23840afa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:32.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.263+0000 7f238b8b8700 1 -- 172.21.15.1:0/4282924965 shutdown_connections 2024-03-18T20:06:32.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.263+0000 7f238b8b8700 1 -- 172.21.15.1:0/4282924965 wait complete. 2024-03-18T20:06:32.280 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-18T20:06:32.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.403+0000 7f24228b6700 1 Processor -- start 2024-03-18T20:06:32.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.403+0000 7f24228b6700 1 -- start start 2024-03-18T20:06:32.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.404+0000 7f24228b6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f241c06ca80 0x7f241c15d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:32.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.404+0000 7f24228b6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f241c09a150 0x7f241c0cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:32.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.404+0000 7f24228b6700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f241c0b3b00 con 0x7f241c15d5b0 2024-03-18T20:06:32.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.404+0000 7f24228b6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f241c0cbbb0 con 0x7f241c06ca80 2024-03-18T20:06:32.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.404+0000 7f24228b6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241c0cf7f0 con 0x7f241c09a150 2024-03-18T20:06:32.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.404+0000 7f241b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f241c06ca80 0x7f241c15d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:32.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.404+0000 7f241b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f241c06ca80 0x7f241c15d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33714/0 (socket says 172.21.15.1:33714) 2024-03-18T20:06:32.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.404+0000 7f241b7fe700 1 -- 172.21.15.1:0/1319523456 learned_addr learned my addr 172.21.15.1:0/1319523456 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:32.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.404+0000 7f2420e53700 1 --2- 172.21.15.1:0/1319523456 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f241c09a150 0x7f241c0cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:32.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.404+0000 7f241b7fe700 1 -- 172.21.15.1:0/1319523456 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f241c09a150 msgr2=0x7f241c0cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:32.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.404+0000 7f241b7fe700 1 --2- 172.21.15.1:0/1319523456 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f241c09a150 0x7f241c0cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.404+0000 7f241b7fe700 1 -- 172.21.15.1:0/1319523456 >> v1:172.21.15.165:6789/0 conn(0x7f241c15d5b0 legacy=0x7f241c099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:32.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.404+0000 7f241b7fe700 1 -- 172.21.15.1:0/1319523456 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c0d04d0 con 0x7f241c06ca80 2024-03-18T20:06:32.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.404+0000 7f241b7fe700 1 --2- 172.21.15.1:0/1319523456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c06ca80 0x7f241c15d990 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f240c0098f0 tx=0x7f240c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=facaa0632ea91aa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:32.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.405+0000 7f241affd700 1 -- 172.21.15.1:0/1319523456 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c011830 con 0x7f241c06ca80 2024-03-18T20:06:32.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.405+0000 7f241affd700 1 -- 172.21.15.1:0/1319523456 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240c011990 con 0x7f241c06ca80 2024-03-18T20:06:32.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.405+0000 7f241affd700 1 -- 172.21.15.1:0/1319523456 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c011d20 con 0x7f241c06ca80 2024-03-18T20:06:32.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.405+0000 7f24228b6700 1 -- 172.21.15.1:0/1319523456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c06ca80 msgr2=0x7f241c15d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:32.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.405+0000 7f24228b6700 1 --2- 172.21.15.1:0/1319523456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c06ca80 0x7f241c15d990 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f240c0098f0 tx=0x7f240c009c00 comp rx=0 tx=0).stop 2024-03-18T20:06:32.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.405+0000 7f24228b6700 1 -- 172.21.15.1:0/1319523456 shutdown_connections 2024-03-18T20:06:32.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.405+0000 7f24228b6700 1 --2- 172.21.15.1:0/1319523456 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f241c09a150 0x7f241c0cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.405+0000 7f24228b6700 1 --2- 172.21.15.1:0/1319523456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c06ca80 0x7f241c15d990 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.405+0000 7f24228b6700 1 -- 172.21.15.1:0/1319523456 >> 172.21.15.1:0/1319523456 conn(0x7f241c0b2bd0 msgr2=0x7f241c0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:32.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.405+0000 7f24228b6700 1 -- 172.21.15.1:0/1319523456 shutdown_connections 2024-03-18T20:06:32.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.405+0000 7f24228b6700 1 -- 172.21.15.1:0/1319523456 wait complete. 2024-03-18T20:06:32.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.407+0000 7f24228b6700 1 Processor -- start 2024-03-18T20:06:32.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.407+0000 7f24228b6700 1 -- start start 2024-03-18T20:06:32.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.408+0000 7f24228b6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c06ca80 0x7f241c154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:32.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.408+0000 7f24228b6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f241c09a150 0x7f241c0c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:32.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.408+0000 7f24228b6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f241c15d5b0 0x7f241c0c4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:32.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.408+0000 7f24228b6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c144560 con 0x7f241c06ca80 2024-03-18T20:06:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.408+0000 7f24228b6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c0c9bc0 con 0x7f241c15d5b0 2024-03-18T20:06:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.408+0000 7f24228b6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241c0c9ea0 con 0x7f241c09a150 2024-03-18T20:06:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.408+0000 7f241bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c06ca80 0x7f241c154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.408+0000 7f241bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c06ca80 0x7f241c154350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33726/0 (socket says 172.21.15.1:33726) 2024-03-18T20:06:32.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.408+0000 7f241bfff700 1 -- 172.21.15.1:0/1151715332 learned_addr learned my addr 172.21.15.1:0/1151715332 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.408+0000 7f2420e53700 1 --2- 172.21.15.1:0/1151715332 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f241c15d5b0 0x7f241c0c4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.408+0000 7f241b7fe700 1 --2- 172.21.15.1:0/1151715332 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f241c09a150 0x7f241c0c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.408+0000 7f241bfff700 1 -- 172.21.15.1:0/1151715332 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f241c09a150 msgr2=0x7f241c0c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.408+0000 7f241bfff700 1 --2- 172.21.15.1:0/1151715332 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f241c09a150 0x7f241c0c9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.408+0000 7f241bfff700 1 -- 172.21.15.1:0/1151715332 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f241c15d5b0 msgr2=0x7f241c0c4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.408+0000 7f241bfff700 1 --2- 172.21.15.1:0/1151715332 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f241c15d5b0 0x7f241c0c4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.408+0000 7f241bfff700 1 -- 172.21.15.1:0/1151715332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c009550 con 0x7f241c06ca80 2024-03-18T20:06:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.408+0000 7f2420e53700 1 --2- 172.21.15.1:0/1151715332 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f241c15d5b0 0x7f241c0c4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.408+0000 7f241b7fe700 1 --2- 172.21.15.1:0/1151715332 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f241c09a150 0x7f241c0c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.408+0000 7f241bfff700 1 --2- 172.21.15.1:0/1151715332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c06ca80 0x7f241c154350 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f241000c450 tx=0x7f241000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.409+0000 7f2418ff9700 1 -- 172.21.15.1:0/1151715332 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24100052d0 con 0x7f241c06ca80 2024-03-18T20:06:32.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.409+0000 7f2418ff9700 1 -- 172.21.15.1:0/1151715332 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2410005430 con 0x7f241c06ca80 2024-03-18T20:06:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.409+0000 7f2418ff9700 1 -- 172.21.15.1:0/1151715332 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24100038c0 con 0x7f241c06ca80 2024-03-18T20:06:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.409+0000 7f24228b6700 1 -- 172.21.15.1:0/1151715332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241c0ca180 con 0x7f241c06ca80 2024-03-18T20:06:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.409+0000 7f24228b6700 1 -- 172.21.15.1:0/1151715332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241c0ca740 con 0x7f241c06ca80 2024-03-18T20:06:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.409+0000 7f24228b6700 1 -- 172.21.15.1:0/1151715332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f241c0676a0 con 0x7f241c06ca80 2024-03-18T20:06:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.410+0000 7f2418ff9700 1 -- 172.21.15.1:0/1151715332 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2410003a20 con 0x7f241c06ca80 2024-03-18T20:06:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.411+0000 7f2418ff9700 1 --2- 172.21.15.1:0/1151715332 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f240806adb0 0x7f240806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.411+0000 7f2418ff9700 1 -- 172.21.15.1:0/1151715332 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f24100888a0 con 0x7f241c06ca80 2024-03-18T20:06:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.411+0000 7f241b7fe700 1 --2- 172.21.15.1:0/1151715332 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f240806adb0 0x7f240806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:32.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.411+0000 7f241b7fe700 1 --2- 172.21.15.1:0/1151715332 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f240806adb0 0x7f240806d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f240c0098f0 tx=0x7f240c008030 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:32.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.415+0000 7f2418ff9700 1 -- 172.21.15.1:0/1151715332 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2410053910 con 0x7f241c06ca80 2024-03-18T20:06:32.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.616+0000 7f24228b6700 1 -- 172.21.15.1:0/1151715332 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f241c099be0 con 0x7f241c06ca80 2024-03-18T20:06:32.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.620+0000 7f2418ff9700 1 -- 172.21.15.1:0/1151715332 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v56) v1 ==== 120+0+0 (secure 0 0 0) 0x7f2410056f30 con 0x7f241c06ca80 2024-03-18T20:06:32.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.622+0000 7f24228b6700 1 -- 172.21.15.1:0/1151715332 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f240806adb0 msgr2=0x7f240806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:32.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.622+0000 7f24228b6700 1 --2- 172.21.15.1:0/1151715332 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f240806adb0 0x7f240806d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f240c0098f0 tx=0x7f240c008030 comp rx=0 tx=0).stop 2024-03-18T20:06:32.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.622+0000 7f24228b6700 1 -- 172.21.15.1:0/1151715332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c06ca80 msgr2=0x7f241c154350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:32.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.622+0000 7f24228b6700 1 --2- 172.21.15.1:0/1151715332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c06ca80 0x7f241c154350 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f241000c450 tx=0x7f241000c810 comp rx=0 tx=0).stop 2024-03-18T20:06:32.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.623+0000 7f24228b6700 1 -- 172.21.15.1:0/1151715332 shutdown_connections 2024-03-18T20:06:32.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.623+0000 7f24228b6700 1 --2- 172.21.15.1:0/1151715332 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f240806adb0 0x7f240806d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.623+0000 7f24228b6700 1 --2- 172.21.15.1:0/1151715332 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f241c15d5b0 0x7f241c0c4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.623+0000 7f24228b6700 1 --2- 172.21.15.1:0/1151715332 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c06ca80 0x7f241c154350 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.623+0000 7f24228b6700 1 --2- 172.21.15.1:0/1151715332 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f241c09a150 0x7f241c0c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.623+0000 7f24228b6700 1 -- 172.21.15.1:0/1151715332 >> 172.21.15.1:0/1151715332 conn(0x7f241c0b2bd0 msgr2=0x7f241c0b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:32.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.623+0000 7f24228b6700 1 -- 172.21.15.1:0/1151715332 shutdown_connections 2024-03-18T20:06:32.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.623+0000 7f24228b6700 1 -- 172.21.15.1:0/1151715332 wait complete. 2024-03-18T20:06:32.641 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-18T20:06:32.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.774+0000 7f429ce64700 1 Processor -- start 2024-03-18T20:06:32.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.774+0000 7f429ce64700 1 -- start start 2024-03-18T20:06:32.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.774+0000 7f429ce64700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4298073a60 0x7f42980ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:32.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.774+0000 7f429ce64700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f429806d300 0x7f42980acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:32.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.775+0000 7f429ce64700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f429806e160 con 0x7f429815d340 2024-03-18T20:06:32.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.775+0000 7f429ce64700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4298161150 con 0x7f429806d300 2024-03-18T20:06:32.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.775+0000 7f429ce64700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f429806d8c0 con 0x7f4298073a60 2024-03-18T20:06:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.775+0000 7f4295d9b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4298073a60 0x7f42980ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.775+0000 7f4296d9d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f429806d300 0x7f42980acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.775+0000 7f4296d9d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f429806d300 0x7f42980acdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:53358/0 (socket says 172.21.15.1:53358) 2024-03-18T20:06:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.775+0000 7f4296d9d700 1 -- 172.21.15.1:0/1680778093 learned_addr learned my addr 172.21.15.1:0/1680778093 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.775+0000 7f4295d9b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4298073a60 0x7f42980ac8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:43866/0 (socket says 172.21.15.1:43866) 2024-03-18T20:06:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.775+0000 7f429659c700 1 --1- 172.21.15.1:0/1680778093 >> v1:172.21.15.165:6789/0 conn(0x7f429815d340 0x7f42981615a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:35584/0 (socket says 172.21.15.1:35584) 2024-03-18T20:06:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.775+0000 7f4296d9d700 1 -- 172.21.15.1:0/1680778093 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4298073a60 msgr2=0x7f42980ac8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.775+0000 7f4296d9d700 1 --2- 172.21.15.1:0/1680778093 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4298073a60 0x7f42980ac8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.775+0000 7f4296d9d700 1 -- 172.21.15.1:0/1680778093 >> v1:172.21.15.165:6789/0 conn(0x7f429815d340 legacy=0x7f42981615a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.775+0000 7f4296d9d700 1 -- 172.21.15.1:0/1680778093 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429806dba0 con 0x7f429806d300 2024-03-18T20:06:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.775+0000 7f429559a700 1 -- 172.21.15.1:0/1680778093 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 227131993 0 0) 0x7f429806e160 con 0x7f429815d340 2024-03-18T20:06:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.775+0000 7f4295d9b700 1 --2- 172.21.15.1:0/1680778093 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4298073a60 0x7f42980ac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:32.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.776+0000 7f4296d9d700 1 --2- 172.21.15.1:0/1680778093 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f429806d300 0x7f42980acdf0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f4290009e50 tx=0x7f429000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70b680a6f8986122 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:32.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.776+0000 7f429559a700 1 -- 172.21.15.1:0/1680778093 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429000e070 con 0x7f429806d300 2024-03-18T20:06:32.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.776+0000 7f429559a700 1 -- 172.21.15.1:0/1680778093 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f429000b910 con 0x7f429806d300 2024-03-18T20:06:32.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.776+0000 7f429559a700 1 -- 172.21.15.1:0/1680778093 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429000bcc0 con 0x7f429806d300 2024-03-18T20:06:32.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.776+0000 7f429ce64700 1 -- 172.21.15.1:0/1680778093 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f429806d300 msgr2=0x7f42980acdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:32.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.776+0000 7f429ce64700 1 --2- 172.21.15.1:0/1680778093 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f429806d300 0x7f42980acdf0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f4290009e50 tx=0x7f429000b330 comp rx=0 tx=0).stop 2024-03-18T20:06:32.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.776+0000 7f429ce64700 1 -- 172.21.15.1:0/1680778093 shutdown_connections 2024-03-18T20:06:32.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.776+0000 7f429ce64700 1 --2- 172.21.15.1:0/1680778093 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f429806d300 0x7f42980acdf0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.776+0000 7f429ce64700 1 --2- 172.21.15.1:0/1680778093 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4298073a60 0x7f42980ac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.776+0000 7f429ce64700 1 -- 172.21.15.1:0/1680778093 >> 172.21.15.1:0/1680778093 conn(0x7f4298066b70 msgr2=0x7f429806a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:32.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.776+0000 7f429ce64700 1 -- 172.21.15.1:0/1680778093 shutdown_connections 2024-03-18T20:06:32.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.776+0000 7f429ce64700 1 -- 172.21.15.1:0/1680778093 wait complete. 2024-03-18T20:06:32.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.778+0000 7f429ce64700 1 Processor -- start 2024-03-18T20:06:32.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.778+0000 7f429ce64700 1 -- start start 2024-03-18T20:06:32.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.778+0000 7f429ce64700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f429806d300 0x7f42981512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:32.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.778+0000 7f429ce64700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4298073a60 0x7f4298153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:32.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.778+0000 7f429ce64700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f429815d340 0x7f429815ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:32.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.778+0000 7f429ce64700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4298152cb0 con 0x7f429806d300 2024-03-18T20:06:32.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.778+0000 7f429ce64700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4298152f90 con 0x7f429815d340 2024-03-18T20:06:32.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.778+0000 7f429ce64700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4298153270 con 0x7f4298073a60 2024-03-18T20:06:32.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.779+0000 7f429659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f429806d300 0x7f42981512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:32.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.779+0000 7f429659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f429806d300 0x7f42981512f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33734/0 (socket says 172.21.15.1:33734) 2024-03-18T20:06:32.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.779+0000 7f429659c700 1 -- 172.21.15.1:0/1978141702 learned_addr learned my addr 172.21.15.1:0/1978141702 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:32.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.779+0000 7f4295d9b700 1 --2- 172.21.15.1:0/1978141702 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4298073a60 0x7f4298153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:32.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.779+0000 7f4296d9d700 1 --2- 172.21.15.1:0/1978141702 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f429815d340 0x7f429815ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:32.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.779+0000 7f429659c700 1 -- 172.21.15.1:0/1978141702 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4298073a60 msgr2=0x7f4298153d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:32.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.779+0000 7f429659c700 1 --2- 172.21.15.1:0/1978141702 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4298073a60 0x7f4298153d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.779+0000 7f429659c700 1 -- 172.21.15.1:0/1978141702 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f429815d340 msgr2=0x7f429815ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:32.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.779+0000 7f429659c700 1 --2- 172.21.15.1:0/1978141702 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f429815d340 0x7f429815ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:32.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.779+0000 7f429659c700 1 -- 172.21.15.1:0/1978141702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42880094a0 con 0x7f429806d300 2024-03-18T20:06:32.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.779+0000 7f4296d9d700 1 --2- 172.21.15.1:0/1978141702 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f429815d340 0x7f429815ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:32.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.779+0000 7f429659c700 1 --2- 172.21.15.1:0/1978141702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f429806d300 0x7f42981512f0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f428c00c550 tx=0x7f428c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:32.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.779+0000 7f4286ffd700 1 -- 172.21.15.1:0/1978141702 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c004c80 con 0x7f429806d300 2024-03-18T20:06:32.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.779+0000 7f4286ffd700 1 -- 172.21.15.1:0/1978141702 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f428c004e00 con 0x7f429806d300 2024-03-18T20:06:32.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.779+0000 7f429ce64700 1 -- 172.21.15.1:0/1978141702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4290009b30 con 0x7f429806d300 2024-03-18T20:06:32.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.779+0000 7f4286ffd700 1 -- 172.21.15.1:0/1978141702 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c007310 con 0x7f429806d300 2024-03-18T20:06:32.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.779+0000 7f429ce64700 1 -- 172.21.15.1:0/1978141702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4298153830 con 0x7f429806d300 2024-03-18T20:06:32.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.780+0000 7f429ce64700 1 -- 172.21.15.1:0/1978141702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42980676a0 con 0x7f429806d300 2024-03-18T20:06:32.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.781+0000 7f4286ffd700 1 -- 172.21.15.1:0/1978141702 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f428c007470 con 0x7f429806d300 2024-03-18T20:06:32.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.781+0000 7f4286ffd700 1 --2- 172.21.15.1:0/1978141702 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f427806adb0 0x7f427806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:32.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.781+0000 7f4295d9b700 1 --2- 172.21.15.1:0/1978141702 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f427806adb0 0x7f427806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:32.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.781+0000 7f4286ffd700 1 -- 172.21.15.1:0/1978141702 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f428c089160 con 0x7f429806d300 2024-03-18T20:06:32.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.782+0000 7f4295d9b700 1 --2- 172.21.15.1:0/1978141702 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f427806adb0 0x7f427806d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4288009cd0 tx=0x7f4288009210 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:32.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.786+0000 7f4286ffd700 1 -- 172.21.15.1:0/1978141702 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f428c0541d0 con 0x7f429806d300 2024-03-18T20:06:32.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.988+0000 7f429ce64700 1 -- 172.21.15.1:0/1978141702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f42980cbd50 con 0x7f429806d300 2024-03-18T20:06:32.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.991+0000 7f4286ffd700 1 -- 172.21.15.1:0/1978141702 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v57) v1 ==== 104+0+0 (secure 0 0 0) 0x7f428c0577f0 con 0x7f429806d300 2024-03-18T20:06:32.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.993+0000 7f429ce64700 1 -- 172.21.15.1:0/1978141702 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f427806adb0 msgr2=0x7f427806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:32.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.993+0000 7f429ce64700 1 --2- 172.21.15.1:0/1978141702 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f427806adb0 0x7f427806d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4288009cd0 tx=0x7f4288009210 comp rx=0 tx=0).stop 2024-03-18T20:06:32.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.993+0000 7f429ce64700 1 -- 172.21.15.1:0/1978141702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f429806d300 msgr2=0x7f42981512f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:32.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.993+0000 7f429ce64700 1 --2- 172.21.15.1:0/1978141702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f429806d300 0x7f42981512f0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f428c00c550 tx=0x7f428c00c860 comp rx=0 tx=0).stop 2024-03-18T20:06:33.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.993+0000 7f429ce64700 1 -- 172.21.15.1:0/1978141702 shutdown_connections 2024-03-18T20:06:33.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.993+0000 7f429ce64700 1 --2- 172.21.15.1:0/1978141702 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f427806adb0 0x7f427806d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:33.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.993+0000 7f429ce64700 1 --2- 172.21.15.1:0/1978141702 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f429815d340 0x7f429815ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:33.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.993+0000 7f429ce64700 1 --2- 172.21.15.1:0/1978141702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f429806d300 0x7f42981512f0 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:33.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.993+0000 7f429ce64700 1 --2- 172.21.15.1:0/1978141702 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4298073a60 0x7f4298153d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:33.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.993+0000 7f429ce64700 1 -- 172.21.15.1:0/1978141702 >> 172.21.15.1:0/1978141702 conn(0x7f4298066b70 msgr2=0x7f42980cdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:33.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.994+0000 7f429ce64700 1 -- 172.21.15.1:0/1978141702 shutdown_connections 2024-03-18T20:06:33.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:32.994+0000 7f429ce64700 1 -- 172.21.15.1:0/1978141702 wait complete. 2024-03-18T20:06:33.010 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2024-03-18T20:06:33.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.131+0000 7f15b68b5700 1 Processor -- start 2024-03-18T20:06:33.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.131+0000 7f15b68b5700 1 -- start start 2024-03-18T20:06:33.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.131+0000 7f15b68b5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f15b00b2760 0x7f15b0160f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:33.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.131+0000 7f15b68b5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15b006d820 0x7f15b006dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:33.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.131+0000 7f15b68b5700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b0161e80 con 0x7f15b015d420 2024-03-18T20:06:33.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.131+0000 7f15b68b5700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b006e1f0 con 0x7f15b00b2760 2024-03-18T20:06:33.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.131+0000 7f15b68b5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b015d800 con 0x7f15b006d820 2024-03-18T20:06:33.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.132+0000 7f15af7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f15b00b2760 0x7f15b0160f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:33.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.132+0000 7f15af7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f15b00b2760 0x7f15b0160f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33738/0 (socket says 172.21.15.1:33738) 2024-03-18T20:06:33.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.132+0000 7f15af7fe700 1 -- 172.21.15.1:0/1493052241 learned_addr learned my addr 172.21.15.1:0/1493052241 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:33.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.132+0000 7f15b4e52700 1 --2- 172.21.15.1:0/1493052241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15b006d820 0x7f15b006dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:33.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.132+0000 7f15af7fe700 1 -- 172.21.15.1:0/1493052241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15b006d820 msgr2=0x7f15b006dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:33.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.132+0000 7f15af7fe700 1 --2- 172.21.15.1:0/1493052241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15b006d820 0x7f15b006dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:33.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.132+0000 7f15af7fe700 1 -- 172.21.15.1:0/1493052241 >> v1:172.21.15.165:6789/0 conn(0x7f15b015d420 legacy=0x7f15b00ac8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:33.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.132+0000 7f15af7fe700 1 -- 172.21.15.1:0/1493052241 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b00d0380 con 0x7f15b00b2760 2024-03-18T20:06:33.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.132+0000 7f15b4e52700 1 --2- 172.21.15.1:0/1493052241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15b006d820 0x7f15b006dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:33.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.132+0000 7f15af7fe700 1 --2- 172.21.15.1:0/1493052241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b00b2760 0x7f15b0160f90 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f15a4006fd0 tx=0x7f15a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bff49f43aa3dc8c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:33.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.132+0000 7f15aeffd700 1 -- 172.21.15.1:0/1493052241 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a4005e30 con 0x7f15b00b2760 2024-03-18T20:06:33.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.132+0000 7f15aeffd700 1 -- 172.21.15.1:0/1493052241 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a4006030 con 0x7f15b00b2760 2024-03-18T20:06:33.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.132+0000 7f15aeffd700 1 -- 172.21.15.1:0/1493052241 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a40063c0 con 0x7f15b00b2760 2024-03-18T20:06:33.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.133+0000 7f15b68b5700 1 -- 172.21.15.1:0/1493052241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b00b2760 msgr2=0x7f15b0160f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:33.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.133+0000 7f15b68b5700 1 --2- 172.21.15.1:0/1493052241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b00b2760 0x7f15b0160f90 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f15a4006fd0 tx=0x7f15a4009bd0 comp rx=0 tx=0).stop 2024-03-18T20:06:33.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.133+0000 7f15b68b5700 1 -- 172.21.15.1:0/1493052241 shutdown_connections 2024-03-18T20:06:33.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.133+0000 7f15b68b5700 1 --2- 172.21.15.1:0/1493052241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15b006d820 0x7f15b006dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:33.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.133+0000 7f15b68b5700 1 --2- 172.21.15.1:0/1493052241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b00b2760 0x7f15b0160f90 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:33.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.133+0000 7f15b68b5700 1 -- 172.21.15.1:0/1493052241 >> 172.21.15.1:0/1493052241 conn(0x7f15b006a750 msgr2=0x7f15b006ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:33.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.133+0000 7f15b68b5700 1 -- 172.21.15.1:0/1493052241 shutdown_connections 2024-03-18T20:06:33.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.133+0000 7f15b68b5700 1 -- 172.21.15.1:0/1493052241 wait complete. 2024-03-18T20:06:33.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.134+0000 7f15b68b5700 1 Processor -- start 2024-03-18T20:06:33.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.134+0000 7f15b68b5700 1 -- start start 2024-03-18T20:06:33.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.135+0000 7f15b68b5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15b006d820 0x7f15b01f2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:33.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.135+0000 7f15b68b5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15b00b2760 0x7f15b00ada30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:33.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.135+0000 7f15b68b5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b015d420 0x7f15b00adf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:33.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.135+0000 7f15b68b5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b00b0e90 con 0x7f15b015d420 2024-03-18T20:06:33.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.135+0000 7f15b68b5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b00b0280 con 0x7f15b00b2760 2024-03-18T20:06:33.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.135+0000 7f15b68b5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b00b0560 con 0x7f15b006d820 2024-03-18T20:06:33.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.135+0000 7f15b4e52700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b015d420 0x7f15b00adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:33.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.135+0000 7f15b4e52700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b015d420 0x7f15b00adf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33744/0 (socket says 172.21.15.1:33744) 2024-03-18T20:06:33.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.135+0000 7f15b4e52700 1 -- 172.21.15.1:0/2321836703 learned_addr learned my addr 172.21.15.1:0/2321836703 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:33.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.135+0000 7f15af7fe700 1 --2- 172.21.15.1:0/2321836703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15b00b2760 0x7f15b00ada30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:33.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.135+0000 7f15affff700 1 --2- 172.21.15.1:0/2321836703 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15b006d820 0x7f15b01f2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:33.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.135+0000 7f15b4e52700 1 -- 172.21.15.1:0/2321836703 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15b006d820 msgr2=0x7f15b01f2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:33.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.135+0000 7f15b4e52700 1 --2- 172.21.15.1:0/2321836703 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15b006d820 0x7f15b01f2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:33.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.135+0000 7f15b4e52700 1 -- 172.21.15.1:0/2321836703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15b00b2760 msgr2=0x7f15b00ada30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:33.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.135+0000 7f15b4e52700 1 --2- 172.21.15.1:0/2321836703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15b00b2760 0x7f15b00ada30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:33.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.135+0000 7f15b4e52700 1 -- 172.21.15.1:0/2321836703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a4009500 con 0x7f15b015d420 2024-03-18T20:06:33.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.136+0000 7f15b4e52700 1 --2- 172.21.15.1:0/2321836703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b015d420 0x7f15b00adf70 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f15a800cd50 tx=0x7f15a800a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:33.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.136+0000 7f15acff9700 1 -- 172.21.15.1:0/2321836703 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a800cea0 con 0x7f15b015d420 2024-03-18T20:06:33.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.136+0000 7f15acff9700 1 -- 172.21.15.1:0/2321836703 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a800f480 con 0x7f15b015d420 2024-03-18T20:06:33.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.136+0000 7f15b68b5700 1 -- 172.21.15.1:0/2321836703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b00b0840 con 0x7f15b015d420 2024-03-18T20:06:33.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.136+0000 7f15acff9700 1 -- 172.21.15.1:0/2321836703 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a800f830 con 0x7f15b015d420 2024-03-18T20:06:33.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.136+0000 7f15b68b5700 1 -- 172.21.15.1:0/2321836703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b01f2a40 con 0x7f15b015d420 2024-03-18T20:06:33.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.137+0000 7f15b68b5700 1 -- 172.21.15.1:0/2321836703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b00676a0 con 0x7f15b015d420 2024-03-18T20:06:33.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.137+0000 7f15acff9700 1 -- 172.21.15.1:0/2321836703 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f15a800b040 con 0x7f15b015d420 2024-03-18T20:06:33.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.138+0000 7f15acff9700 1 --2- 172.21.15.1:0/2321836703 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f159c06adb0 0x7f159c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:33.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.138+0000 7f15acff9700 1 -- 172.21.15.1:0/2321836703 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f15a8089ce0 con 0x7f15b015d420 2024-03-18T20:06:33.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.138+0000 7f15affff700 1 --2- 172.21.15.1:0/2321836703 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f159c06adb0 0x7f159c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:33.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.138+0000 7f15affff700 1 --2- 172.21.15.1:0/2321836703 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f159c06adb0 0x7f159c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f15a0007fd0 tx=0x7f15a0007ec0 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:33.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.142+0000 7f15acff9700 1 -- 172.21.15.1:0/2321836703 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f15a8054c40 con 0x7f15b015d420 2024-03-18T20:06:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.435+0000 7f15b68b5700 1 -- 172.21.15.1:0/2321836703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f15b0073af0 con 0x7f15b015d420 2024-03-18T20:06:33.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.889+0000 7f15acff9700 1 -- 172.21.15.1:0/2321836703 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v2491) v1 ==== 178+0+0 (secure 0 0 0) 0x7f15a8058260 con 0x7f15b015d420 2024-03-18T20:06:33.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.892+0000 7f15b68b5700 1 -- 172.21.15.1:0/2321836703 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f159c06adb0 msgr2=0x7f159c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:33.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.892+0000 7f15b68b5700 1 --2- 172.21.15.1:0/2321836703 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f159c06adb0 0x7f159c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f15a0007fd0 tx=0x7f15a0007ec0 comp rx=0 tx=0).stop 2024-03-18T20:06:33.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.892+0000 7f15b68b5700 1 -- 172.21.15.1:0/2321836703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b015d420 msgr2=0x7f15b00adf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:33.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.892+0000 7f15b68b5700 1 --2- 172.21.15.1:0/2321836703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b015d420 0x7f15b00adf70 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f15a800cd50 tx=0x7f15a800a560 comp rx=0 tx=0).stop 2024-03-18T20:06:33.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.892+0000 7f15b68b5700 1 -- 172.21.15.1:0/2321836703 shutdown_connections 2024-03-18T20:06:33.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.892+0000 7f15b68b5700 1 --2- 172.21.15.1:0/2321836703 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f159c06adb0 0x7f159c06d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:33.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.892+0000 7f15b68b5700 1 --2- 172.21.15.1:0/2321836703 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15b006d820 0x7f15b01f2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:33.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.892+0000 7f15b68b5700 1 --2- 172.21.15.1:0/2321836703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b015d420 0x7f15b00adf70 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:33.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.892+0000 7f15b68b5700 1 --2- 172.21.15.1:0/2321836703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15b00b2760 0x7f15b00ada30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:33.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.892+0000 7f15b68b5700 1 -- 172.21.15.1:0/2321836703 >> 172.21.15.1:0/2321836703 conn(0x7f15b006a750 msgr2=0x7f15b015d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:33.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.892+0000 7f15b68b5700 1 -- 172.21.15.1:0/2321836703 shutdown_connections 2024-03-18T20:06:33.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:33.892+0000 7f15b68b5700 1 -- 172.21.15.1:0/2321836703 wait complete. 2024-03-18T20:06:33.912 INFO:tasks.cephfs_test_runner:Test that number of snaps retained are as per user spec. ... ok 2024-03-18T20:06:33.912 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2024-03-18T20:06:33.912 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-18T20:06:33.912 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2024-03-18T20:06:34.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.038+0000 7f9f8f77f700 1 Processor -- start 2024-03-18T20:06:34.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.038+0000 7f9f8f77f700 1 -- start start 2024-03-18T20:06:34.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.039+0000 7f9f8f77f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9f88099a40 0x7f9f88099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:34.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.039+0000 7f9f8f77f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9f880cc310 0x7f9f880cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:34.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.039+0000 7f9f8f77f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8809a3e0 con 0x7f9f8815d680 2024-03-18T20:06:34.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.039+0000 7f9f8f77f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f880d0c80 con 0x7f9f880cc310 2024-03-18T20:06:34.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.039+0000 7f9f8f77f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f880d09a0 con 0x7f9f88099a40 2024-03-18T20:06:34.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.039+0000 7f9f8dd1c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9f880cc310 0x7f9f880cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:34.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.039+0000 7f9f8d51b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9f8815d680 0x7f9f880cff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59224/0 (socket says 172.21.15.1:59224) 2024-03-18T20:06:34.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.039+0000 7f9f8d51b700 1 -- 172.21.15.1:0/3765364319 learned_addr learned my addr 172.21.15.1:0/3765364319 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:34.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.039+0000 7f9f8cd1a700 1 --2- 172.21.15.1:0/3765364319 >> v2:172.21.15.165:3300/0 conn(0x7f9f88099a40 0x7f9f88099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:34.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.039+0000 7f9f8dd1c700 1 -- 172.21.15.1:0/3765364319 >> v2:172.21.15.165:3300/0 conn(0x7f9f88099a40 msgr2=0x7f9f88099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:34.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.039+0000 7f9f8dd1c700 1 --2- 172.21.15.1:0/3765364319 >> v2:172.21.15.165:3300/0 conn(0x7f9f88099a40 0x7f9f88099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:34.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.039+0000 7f9f8dd1c700 1 -- 172.21.15.1:0/3765364319 >> v1:172.21.15.1:6789/0 conn(0x7f9f8815d680 legacy=0x7f9f880cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:34.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.039+0000 7f9f7ffff700 1 -- 172.21.15.1:0/3765364319 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771748526 0 0) 0x7f9f8809a3e0 con 0x7f9f8815d680 2024-03-18T20:06:34.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.040+0000 7f9f8dd1c700 1 -- 172.21.15.1:0/3765364319 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f880cb830 con 0x7f9f880cc310 2024-03-18T20:06:34.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.040+0000 7f9f8dd1c700 1 --2- 172.21.15.1:0/3765364319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f880cc310 0x7f9f880cc7a0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f9f88071cb0 tx=0x7f9f8000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba6309a28ece01 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:34.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.040+0000 7f9f7ffff700 1 -- 172.21.15.1:0/3765364319 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8000e070 con 0x7f9f880cc310 2024-03-18T20:06:34.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.040+0000 7f9f7ffff700 1 -- 172.21.15.1:0/3765364319 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8000bbd0 con 0x7f9f880cc310 2024-03-18T20:06:34.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.040+0000 7f9f7ffff700 1 -- 172.21.15.1:0/3765364319 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f80012430 con 0x7f9f880cc310 2024-03-18T20:06:34.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.040+0000 7f9f8f77f700 1 -- 172.21.15.1:0/3765364319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f880cc310 msgr2=0x7f9f880cc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:34.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.040+0000 7f9f8f77f700 1 --2- 172.21.15.1:0/3765364319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f880cc310 0x7f9f880cc7a0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f9f88071cb0 tx=0x7f9f8000b330 comp rx=0 tx=0).stop 2024-03-18T20:06:34.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.040+0000 7f9f8f77f700 1 -- 172.21.15.1:0/3765364319 shutdown_connections 2024-03-18T20:06:34.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.040+0000 7f9f8f77f700 1 --2- 172.21.15.1:0/3765364319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f880cc310 0x7f9f880cc7a0 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:34.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.040+0000 7f9f8f77f700 1 --2- 172.21.15.1:0/3765364319 >> v2:172.21.15.165:3300/0 conn(0x7f9f88099a40 0x7f9f88099ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:34.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.040+0000 7f9f8f77f700 1 -- 172.21.15.1:0/3765364319 >> 172.21.15.1:0/3765364319 conn(0x7f9f880b2ba0 msgr2=0x7f9f880b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:34.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.040+0000 7f9f8f77f700 1 -- 172.21.15.1:0/3765364319 shutdown_connections 2024-03-18T20:06:34.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.041+0000 7f9f8f77f700 1 -- 172.21.15.1:0/3765364319 wait complete. 2024-03-18T20:06:34.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.042+0000 7f9f8f77f700 1 Processor -- start 2024-03-18T20:06:34.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.042+0000 7f9f8f77f700 1 -- start start 2024-03-18T20:06:34.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.043+0000 7f9f8f77f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f88099a40 0x7f9f8809e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:34.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.043+0000 7f9f8f77f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9f880cc310 0x7f9f8815a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:34.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.043+0000 7f9f8f77f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9f8815d680 0x7f9f880b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:34.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.043+0000 7f9f8f77f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f88160020 con 0x7f9f88099a40 2024-03-18T20:06:34.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.043+0000 7f9f8f77f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8815f410 con 0x7f9f880cc310 2024-03-18T20:06:34.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.043+0000 7f9f8f77f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8815f6f0 con 0x7f9f8815d680 2024-03-18T20:06:34.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.043+0000 7f9f8d51b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f88099a40 0x7f9f8809e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:34.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.043+0000 7f9f8d51b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f88099a40 0x7f9f8809e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33774/0 (socket says 172.21.15.1:33774) 2024-03-18T20:06:34.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.043+0000 7f9f8d51b700 1 -- 172.21.15.1:0/2470550272 learned_addr learned my addr 172.21.15.1:0/2470550272 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:34.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.043+0000 7f9f8d51b700 1 -- 172.21.15.1:0/2470550272 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9f8815d680 msgr2=0x7f9f880b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:34.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.043+0000 7f9f8d51b700 1 --2- 172.21.15.1:0/2470550272 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9f8815d680 0x7f9f880b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:34.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.043+0000 7f9f8d51b700 1 -- 172.21.15.1:0/2470550272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9f880cc310 msgr2=0x7f9f8815a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:34.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.043+0000 7f9f8d51b700 1 --2- 172.21.15.1:0/2470550272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9f880cc310 0x7f9f8815a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:34.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.043+0000 7f9f8d51b700 1 -- 172.21.15.1:0/2470550272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f80009a70 con 0x7f9f88099a40 2024-03-18T20:06:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.044+0000 7f9f8d51b700 1 --2- 172.21.15.1:0/2470550272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f88099a40 0x7f9f8809e630 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800c550 tx=0x7f9f7800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.044+0000 7f9f7dffb700 1 -- 172.21.15.1:0/2470550272 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f78004c80 con 0x7f9f88099a40 2024-03-18T20:06:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.044+0000 7f9f7dffb700 1 -- 172.21.15.1:0/2470550272 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f78004e00 con 0x7f9f88099a40 2024-03-18T20:06:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.044+0000 7f9f8f77f700 1 -- 172.21.15.1:0/2470550272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f8815f9d0 con 0x7f9f88099a40 2024-03-18T20:06:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.044+0000 7f9f7dffb700 1 -- 172.21.15.1:0/2470550272 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f78007310 con 0x7f9f88099a40 2024-03-18T20:06:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.044+0000 7f9f8f77f700 1 -- 172.21.15.1:0/2470550272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f8809eb70 con 0x7f9f88099a40 2024-03-18T20:06:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.045+0000 7f9f8f77f700 1 -- 172.21.15.1:0/2470550272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f88073350 con 0x7f9f88099a40 2024-03-18T20:06:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.045+0000 7f9f7dffb700 1 -- 172.21.15.1:0/2470550272 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9f78007470 con 0x7f9f88099a40 2024-03-18T20:06:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.046+0000 7f9f7dffb700 1 --2- 172.21.15.1:0/2470550272 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9f6c06adb0 0x7f9f6c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.046+0000 7f9f7dffb700 1 -- 172.21.15.1:0/2470550272 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f9f780891d0 con 0x7f9f88099a40 2024-03-18T20:06:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.046+0000 7f9f8cd1a700 1 --2- 172.21.15.1:0/2470550272 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9f6c06adb0 0x7f9f6c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:34.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.046+0000 7f9f8cd1a700 1 --2- 172.21.15.1:0/2470550272 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9f6c06adb0 0x7f9f6c06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9f880b33f0 tx=0x7f9f84008040 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:34.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.050+0000 7f9f7dffb700 1 -- 172.21.15.1:0/2470550272 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9f78054240 con 0x7f9f88099a40 2024-03-18T20:06:34.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.339+0000 7f9f8f77f700 1 -- 172.21.15.1:0/2470550272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f9f880676a0 con 0x7f9f88099a40 2024-03-18T20:06:34.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.891+0000 7f9f7dffb700 1 -- 172.21.15.1:0/2470550272 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2492) v1 ==== 177+0+0 (secure 0 0 0) 0x7f9f78057860 con 0x7f9f88099a40 2024-03-18T20:06:34.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.894+0000 7f9f8f77f700 1 -- 172.21.15.1:0/2470550272 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9f6c06adb0 msgr2=0x7f9f6c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:34.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.894+0000 7f9f8f77f700 1 --2- 172.21.15.1:0/2470550272 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9f6c06adb0 0x7f9f6c06d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9f880b33f0 tx=0x7f9f84008040 comp rx=0 tx=0).stop 2024-03-18T20:06:34.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.894+0000 7f9f8f77f700 1 -- 172.21.15.1:0/2470550272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f88099a40 msgr2=0x7f9f8809e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:34.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.894+0000 7f9f8f77f700 1 --2- 172.21.15.1:0/2470550272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f88099a40 0x7f9f8809e630 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800c550 tx=0x7f9f7800c910 comp rx=0 tx=0).stop 2024-03-18T20:06:34.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.894+0000 7f9f8f77f700 1 -- 172.21.15.1:0/2470550272 shutdown_connections 2024-03-18T20:06:34.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.894+0000 7f9f8f77f700 1 --2- 172.21.15.1:0/2470550272 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9f6c06adb0 0x7f9f6c06d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:34.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.894+0000 7f9f8f77f700 1 --2- 172.21.15.1:0/2470550272 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9f8815d680 0x7f9f880b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:34.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.894+0000 7f9f8f77f700 1 --2- 172.21.15.1:0/2470550272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f88099a40 0x7f9f8809e630 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:34.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.894+0000 7f9f8f77f700 1 --2- 172.21.15.1:0/2470550272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9f880cc310 0x7f9f8815a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:34.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.894+0000 7f9f8f77f700 1 -- 172.21.15.1:0/2470550272 >> 172.21.15.1:0/2470550272 conn(0x7f9f880b2ba0 msgr2=0x7f9f880b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:34.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.894+0000 7f9f8f77f700 1 -- 172.21.15.1:0/2470550272 shutdown_connections 2024-03-18T20:06:34.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:34.894+0000 7f9f8f77f700 1 -- 172.21.15.1:0/2470550272 wait complete. 2024-03-18T20:06:34.914 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-18T20:06:35.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.044+0000 7fb09f781700 1 Processor -- start 2024-03-18T20:06:35.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.044+0000 7fb09f781700 1 -- start start 2024-03-18T20:06:35.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.045+0000 7fb09f781700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb098160e50 0x7fb0980cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:35.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.045+0000 7fb09f781700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb0981612d0 0x7fb0981616f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:35.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.045+0000 7fb09f781700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb098162170 con 0x7fb098160a70 2024-03-18T20:06:35.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.045+0000 7fb09f781700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb098161c30 con 0x7fb0981612d0 2024-03-18T20:06:35.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.045+0000 7fb09f781700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0980cef30 con 0x7fb098160e50 2024-03-18T20:06:35.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.045+0000 7fb09dd1e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb0981612d0 0x7fb0981616f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:35.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.045+0000 7fb09dd1e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb0981612d0 0x7fb0981616f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57956/0 (socket says 172.21.15.1:57956) 2024-03-18T20:06:35.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.045+0000 7fb09dd1e700 1 -- 172.21.15.1:0/197619452 learned_addr learned my addr 172.21.15.1:0/197619452 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:35.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.045+0000 7fb09cd1c700 1 --2- 172.21.15.1:0/197619452 >> v2:172.21.15.165:3300/0 conn(0x7fb098160e50 0x7fb0980cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:35.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.045+0000 7fb09dd1e700 1 -- 172.21.15.1:0/197619452 >> v2:172.21.15.165:3300/0 conn(0x7fb098160e50 msgr2=0x7fb0980cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:35.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.045+0000 7fb09dd1e700 1 --2- 172.21.15.1:0/197619452 >> v2:172.21.15.165:3300/0 conn(0x7fb098160e50 0x7fb0980cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.045+0000 7fb09dd1e700 1 -- 172.21.15.1:0/197619452 >> v1:172.21.15.165:6789/0 conn(0x7fb098160a70 legacy=0x7fb0980d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:35.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.045+0000 7fb09dd1e700 1 -- 172.21.15.1:0/197619452 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0980cf210 con 0x7fb0981612d0 2024-03-18T20:06:35.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.045+0000 7fb09cd1c700 1 --2- 172.21.15.1:0/197619452 >> v2:172.21.15.165:3300/0 conn(0x7fb098160e50 0x7fb0980cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:35.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.046+0000 7fb09dd1e700 1 --2- 172.21.15.1:0/197619452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0981612d0 0x7fb0981616f0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fb0980c9d70 tx=0x7fb09000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c67e83cc202c25 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:35.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.046+0000 7fb08ffff700 1 -- 172.21.15.1:0/197619452 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09000e070 con 0x7fb0981612d0 2024-03-18T20:06:35.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.046+0000 7fb08ffff700 1 -- 172.21.15.1:0/197619452 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb090003a20 con 0x7fb0981612d0 2024-03-18T20:06:35.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.046+0000 7fb08ffff700 1 -- 172.21.15.1:0/197619452 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb090003dd0 con 0x7fb0981612d0 2024-03-18T20:06:35.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.046+0000 7fb09f781700 1 -- 172.21.15.1:0/197619452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0981612d0 msgr2=0x7fb0981616f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:35.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.046+0000 7fb09f781700 1 --2- 172.21.15.1:0/197619452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0981612d0 0x7fb0981616f0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fb0980c9d70 tx=0x7fb09000b330 comp rx=0 tx=0).stop 2024-03-18T20:06:35.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.046+0000 7fb09f781700 1 -- 172.21.15.1:0/197619452 shutdown_connections 2024-03-18T20:06:35.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.046+0000 7fb09f781700 1 --2- 172.21.15.1:0/197619452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0981612d0 0x7fb0981616f0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.046+0000 7fb09f781700 1 --2- 172.21.15.1:0/197619452 >> v2:172.21.15.165:3300/0 conn(0x7fb098160e50 0x7fb0980cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.046+0000 7fb09f781700 1 -- 172.21.15.1:0/197619452 >> 172.21.15.1:0/197619452 conn(0x7fb09809c560 msgr2=0x7fb0980d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:35.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.046+0000 7fb09f781700 1 -- 172.21.15.1:0/197619452 shutdown_connections 2024-03-18T20:06:35.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.047+0000 7fb09f781700 1 -- 172.21.15.1:0/197619452 wait complete. 2024-03-18T20:06:35.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.048+0000 7fb09f781700 1 Processor -- start 2024-03-18T20:06:35.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.048+0000 7fb09f781700 1 -- start start 2024-03-18T20:06:35.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.048+0000 7fb09f781700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb098160a70 0x7fb09815f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:35.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.048+0000 7fb09f781700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb098160e50 0x7fb098150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:35.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.048+0000 7fb09f781700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0981612d0 0x7fb098151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:35.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.048+0000 7fb09f781700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0980b80a0 con 0x7fb0981612d0 2024-03-18T20:06:35.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.048+0000 7fb09f781700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0981e7d30 con 0x7fb098160e50 2024-03-18T20:06:35.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.048+0000 7fb09f781700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0980c75e0 con 0x7fb098160a70 2024-03-18T20:06:35.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.049+0000 7fb09dd1e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0981612d0 0x7fb098151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:35.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.049+0000 7fb09dd1e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0981612d0 0x7fb098151320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57960/0 (socket says 172.21.15.1:57960) 2024-03-18T20:06:35.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.049+0000 7fb09dd1e700 1 -- 172.21.15.1:0/2553906231 learned_addr learned my addr 172.21.15.1:0/2553906231 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:35.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.049+0000 7fb09cd1c700 1 --2- 172.21.15.1:0/2553906231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb098160e50 0x7fb098150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:35.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.049+0000 7fb09d51d700 1 --2- 172.21.15.1:0/2553906231 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb098160a70 0x7fb09815f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:35.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.049+0000 7fb09dd1e700 1 -- 172.21.15.1:0/2553906231 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb098160a70 msgr2=0x7fb09815f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:35.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.049+0000 7fb09dd1e700 1 --2- 172.21.15.1:0/2553906231 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb098160a70 0x7fb09815f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.049+0000 7fb09dd1e700 1 -- 172.21.15.1:0/2553906231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb098160e50 msgr2=0x7fb098150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:35.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.049+0000 7fb09dd1e700 1 --2- 172.21.15.1:0/2553906231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb098160e50 0x7fb098150de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.049+0000 7fb09dd1e700 1 -- 172.21.15.1:0/2553906231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb090009b30 con 0x7fb0981612d0 2024-03-18T20:06:35.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.049+0000 7fb09d51d700 1 --2- 172.21.15.1:0/2553906231 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb098160a70 0x7fb09815f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:35.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.049+0000 7fb09dd1e700 1 --2- 172.21.15.1:0/2553906231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0981612d0 0x7fb098151320 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fb090015040 tx=0x7fb090004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:35.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.049+0000 7fb08dffb700 1 -- 172.21.15.1:0/2553906231 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb090004440 con 0x7fb0981612d0 2024-03-18T20:06:35.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.049+0000 7fb08dffb700 1 -- 172.21.15.1:0/2553906231 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0900045a0 con 0x7fb0981612d0 2024-03-18T20:06:35.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.049+0000 7fb09f781700 1 -- 172.21.15.1:0/2553906231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0980c78c0 con 0x7fb0981612d0 2024-03-18T20:06:35.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.049+0000 7fb08dffb700 1 -- 172.21.15.1:0/2553906231 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb090004950 con 0x7fb0981612d0 2024-03-18T20:06:35.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.050+0000 7fb09f781700 1 -- 172.21.15.1:0/2553906231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0980c7240 con 0x7fb0981612d0 2024-03-18T20:06:35.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.050+0000 7fb09f781700 1 -- 172.21.15.1:0/2553906231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0980c7de0 con 0x7fb0981612d0 2024-03-18T20:06:35.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.051+0000 7fb08dffb700 1 -- 172.21.15.1:0/2553906231 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb090012430 con 0x7fb0981612d0 2024-03-18T20:06:35.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.051+0000 7fb08dffb700 1 --2- 172.21.15.1:0/2553906231 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb07c06adb0 0x7fb07c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:35.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.052+0000 7fb08dffb700 1 -- 172.21.15.1:0/2553906231 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5937+0+0 (secure 0 0 0) 0x7fb09008aa50 con 0x7fb0981612d0 2024-03-18T20:06:35.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.052+0000 7fb09d51d700 1 --2- 172.21.15.1:0/2553906231 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb07c06adb0 0x7fb07c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:35.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.052+0000 7fb09d51d700 1 --2- 172.21.15.1:0/2553906231 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb07c06adb0 0x7fb07c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb088009930 tx=0x7fb088008040 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:35.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.056+0000 7fb08dffb700 1 -- 172.21.15.1:0/2553906231 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb0980c7de0 con 0x7fb0981612d0 2024-03-18T20:06:35.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.258+0000 7fb09f781700 1 -- 172.21.15.1:0/2553906231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb098162170 con 0x7fb0981612d0 2024-03-18T20:06:35.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.263+0000 7fb08dffb700 1 -- 172.21.15.1:0/2553906231 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v58)=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb098162170 con 0x7fb0981612d0 2024-03-18T20:06:35.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.265+0000 7fb09f781700 1 -- 172.21.15.1:0/2553906231 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb07c06adb0 msgr2=0x7fb07c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:35.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.265+0000 7fb09f781700 1 --2- 172.21.15.1:0/2553906231 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb07c06adb0 0x7fb07c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb088009930 tx=0x7fb088008040 comp rx=0 tx=0).stop 2024-03-18T20:06:35.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.265+0000 7fb09f781700 1 -- 172.21.15.1:0/2553906231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0981612d0 msgr2=0x7fb098151320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:35.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.265+0000 7fb09f781700 1 --2- 172.21.15.1:0/2553906231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0981612d0 0x7fb098151320 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fb090015040 tx=0x7fb090004ad0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.265+0000 7fb09f781700 1 -- 172.21.15.1:0/2553906231 shutdown_connections 2024-03-18T20:06:35.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.265+0000 7fb09f781700 1 --2- 172.21.15.1:0/2553906231 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb07c06adb0 0x7fb07c06d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.265+0000 7fb09f781700 1 --2- 172.21.15.1:0/2553906231 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb098160a70 0x7fb09815f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.265+0000 7fb09f781700 1 --2- 172.21.15.1:0/2553906231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0981612d0 0x7fb098151320 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.265+0000 7fb09f781700 1 --2- 172.21.15.1:0/2553906231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb098160e50 0x7fb098150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.266+0000 7fb09f781700 1 -- 172.21.15.1:0/2553906231 >> 172.21.15.1:0/2553906231 conn(0x7fb09809c560 msgr2=0x7fb0980d3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:35.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.266+0000 7fb09f781700 1 -- 172.21.15.1:0/2553906231 shutdown_connections 2024-03-18T20:06:35.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.266+0000 7fb09f781700 1 -- 172.21.15.1:0/2553906231 wait complete. 2024-03-18T20:06:35.282 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T20:06:35.283 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:06:35.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:06:35.374 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T20:06:35.375 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:06:35.375 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:06:35.393 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:06:35.393 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:06:35.393 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:06:35.414 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:06:35.414 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:06:35.414 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:06:35.452 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:06:35.452 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:06:35.453 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:06:35.474 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:06:35.474 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-18T20:06:35.474 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:06:35.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.612+0000 7f20f08db700 1 Processor -- start 2024-03-18T20:06:35.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.612+0000 7f20f08db700 1 -- start start 2024-03-18T20:06:35.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.613+0000 7f20f08db700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f20ec15dd80 0x7f20ec06d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:35.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.613+0000 7f20f08db700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ec0b4b20 con 0x7f20ec0d02f0 2024-03-18T20:06:35.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.613+0000 7f20f08db700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ec0b3f30 con 0x7f20ec1597a0 2024-03-18T20:06:35.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.613+0000 7f20f08db700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20ec0b41f0 con 0x7f20ec15dd80 2024-03-18T20:06:35.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.613+0000 7f20e9d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f20ec0d02f0 0x7f20ec06d0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42198/0 (socket says 172.21.15.1:42198) 2024-03-18T20:06:35.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.613+0000 7f20e9d9b700 1 -- 172.21.15.1:0/2306610636 learned_addr learned my addr 172.21.15.1:0/2306610636 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:35.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20e959a700 1 -- 172.21.15.1:0/2306610636 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701291648 0 0) 0x7f20ec0b4b20 con 0x7f20ec0d02f0 2024-03-18T20:06:35.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20ead9d700 1 --2- 172.21.15.1:0/2306610636 >> v2:172.21.15.165:3300/0 conn(0x7f20ec15dd80 0x7f20ec06d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:35.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20e959a700 1 -- 172.21.15.1:0/2306610636 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20d40035b0 con 0x7f20ec0d02f0 2024-03-18T20:06:35.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20e959a700 1 -- 172.21.15.1:0/2306610636 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3917089732 0 0) 0x7f20d40035b0 con 0x7f20ec0d02f0 2024-03-18T20:06:35.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20e959a700 1 -- 172.21.15.1:0/2306610636 >> v2:172.21.15.165:3300/0 conn(0x7f20ec15dd80 msgr2=0x7f20ec06d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:35.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20e959a700 1 --2- 172.21.15.1:0/2306610636 >> v2:172.21.15.165:3300/0 conn(0x7f20ec15dd80 0x7f20ec06d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20e959a700 1 -- 172.21.15.1:0/2306610636 >> v1:172.21.15.165:6789/0 conn(0x7f20ec1597a0 legacy=0x7f20ec0cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:35.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20e959a700 1 -- 172.21.15.1:0/2306610636 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ec0b44d0 con 0x7f20ec0d02f0 2024-03-18T20:06:35.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20e959a700 1 -- 172.21.15.1:0/2306610636 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f20e0001670 con 0x7f20ec0d02f0 2024-03-18T20:06:35.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20e959a700 1 -- 172.21.15.1:0/2306610636 >> v1:172.21.15.1:6789/0 conn(0x7f20ec0d02f0 legacy=0x7f20ec06d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:35.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20e959a700 1 --2- 172.21.15.1:0/2306610636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20d4003a00 0x7f20d4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:35.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20e959a700 1 --2- 172.21.15.1:0/2306610636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20d4004850 0x7f20d4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20e959a700 1 --2- 172.21.15.1:0/2306610636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20d4005210 0x7f20d400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20e959a700 1 -- 172.21.15.1:0/2306610636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec0b41f0 con 0x7f20d4003a00 2024-03-18T20:06:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20e959a700 1 -- 172.21.15.1:0/2306610636 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d400c0b0 con 0x7f20d4004850 2024-03-18T20:06:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20e959a700 1 -- 172.21.15.1:0/2306610636 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d400c270 con 0x7f20d4005210 2024-03-18T20:06:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20ea59c700 1 --2- 172.21.15.1:0/2306610636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20d4003a00 0x7f20d4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20e9d9b700 1 --2- 172.21.15.1:0/2306610636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20d4004850 0x7f20d4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20ead9d700 1 --2- 172.21.15.1:0/2306610636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20d4005210 0x7f20d400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20ea59c700 1 -- 172.21.15.1:0/2306610636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20d4005210 msgr2=0x7f20d400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20ea59c700 1 --2- 172.21.15.1:0/2306610636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20d4005210 0x7f20d400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20ea59c700 1 -- 172.21.15.1:0/2306610636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20d4004850 msgr2=0x7f20d4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20ea59c700 1 --2- 172.21.15.1:0/2306610636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20d4004850 0x7f20d4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20ea59c700 1 -- 172.21.15.1:0/2306610636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d40035b0 con 0x7f20d4003a00 2024-03-18T20:06:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.614+0000 7f20ead9d700 1 -- 172.21.15.1:0/2306610636 reap_dead start 2024-03-18T20:06:35.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.615+0000 7f20ea59c700 1 --2- 172.21.15.1:0/2306610636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20d4003a00 0x7f20d4003e20 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f20dc001800 tx=0x7f20d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f917e2c13a0a6fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:35.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.615+0000 7f20e959a700 1 -- 172.21.15.1:0/2306610636 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20dc0012f0 con 0x7f20d4003a00 2024-03-18T20:06:35.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.615+0000 7f20e959a700 1 -- 172.21.15.1:0/2306610636 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20dc005310 con 0x7f20d4003a00 2024-03-18T20:06:35.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.615+0000 7f20e959a700 1 -- 172.21.15.1:0/2306610636 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20dc006860 con 0x7f20d4003a00 2024-03-18T20:06:35.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.615+0000 7f20f08db700 1 -- 172.21.15.1:0/2306610636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20d4003a00 msgr2=0x7f20d4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:35.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.615+0000 7f20f08db700 1 --2- 172.21.15.1:0/2306610636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20d4003a00 0x7f20d4003e20 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f20dc001800 tx=0x7f20d4005880 comp rx=0 tx=0).stop 2024-03-18T20:06:35.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.615+0000 7f20f08db700 1 -- 172.21.15.1:0/2306610636 shutdown_connections 2024-03-18T20:06:35.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.615+0000 7f20f08db700 1 --2- 172.21.15.1:0/2306610636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20d4003a00 0x7f20d4003e20 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.615+0000 7f20f08db700 1 -- 172.21.15.1:0/2306610636 >> 172.21.15.1:0/2306610636 conn(0x7f20ec06ab70 msgr2=0x7f20ec06af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:35.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.615+0000 7f20f08db700 1 -- 172.21.15.1:0/2306610636 shutdown_connections 2024-03-18T20:06:35.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.615+0000 7f20f08db700 1 -- 172.21.15.1:0/2306610636 wait complete. 2024-03-18T20:06:35.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.617+0000 7f20f08db700 1 Processor -- start 2024-03-18T20:06:35.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.617+0000 7f20f08db700 1 -- start start 2024-03-18T20:06:35.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20f08db700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20d4003a00 0x7f20ec148240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:35.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20f08db700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec158420 0x7f20ec1588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:35.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20f08db700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20ec0b0bb0 0x7f20ec0b1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:35.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20f08db700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec158de0 con 0x7f20ec158420 2024-03-18T20:06:35.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20f08db700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec152500 con 0x7f20ec0b0bb0 2024-03-18T20:06:35.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20f08db700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec1563f0 con 0x7f20d4003a00 2024-03-18T20:06:35.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20e9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec158420 0x7f20ec1588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:35.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20e9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec158420 0x7f20ec1588a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57978/0 (socket says 172.21.15.1:57978) 2024-03-18T20:06:35.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20e9d9b700 1 -- 172.21.15.1:0/509420249 learned_addr learned my addr 172.21.15.1:0/509420249 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20ea59c700 1 --2- 172.21.15.1:0/509420249 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20d4003a00 0x7f20ec148240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20e9d9b700 1 -- 172.21.15.1:0/509420249 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20d4003a00 msgr2=0x7f20ec148240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20e9d9b700 1 --2- 172.21.15.1:0/509420249 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20d4003a00 0x7f20ec148240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20ead9d700 1 --2- 172.21.15.1:0/509420249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20ec0b0bb0 0x7f20ec0b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20e9d9b700 1 -- 172.21.15.1:0/509420249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20ec0b0bb0 msgr2=0x7f20ec0b1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20e9d9b700 1 --2- 172.21.15.1:0/509420249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20ec0b0bb0 0x7f20ec0b1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20e9d9b700 1 -- 172.21.15.1:0/509420249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ec0b1570 con 0x7f20ec158420 2024-03-18T20:06:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20ead9d700 1 --2- 172.21.15.1:0/509420249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20ec0b0bb0 0x7f20ec0b1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20e9d9b700 1 --2- 172.21.15.1:0/509420249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec158420 0x7f20ec1588a0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f20e00072c0 tx=0x7f20e0007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20daffd700 1 -- 172.21.15.1:0/509420249 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e00077a0 con 0x7f20ec158420 2024-03-18T20:06:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20daffd700 1 -- 172.21.15.1:0/509420249 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e000f7f0 con 0x7f20ec158420 2024-03-18T20:06:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20f08db700 1 -- 172.21.15.1:0/509420249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ec0b1ae0 con 0x7f20ec158420 2024-03-18T20:06:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20daffd700 1 -- 172.21.15.1:0/509420249 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e000fba0 con 0x7f20ec158420 2024-03-18T20:06:35.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.618+0000 7f20f08db700 1 -- 172.21.15.1:0/509420249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ec155da0 con 0x7f20ec158420 2024-03-18T20:06:35.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.619+0000 7f20f08db700 1 -- 172.21.15.1:0/509420249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ec0cedc0 con 0x7f20ec158420 2024-03-18T20:06:35.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.620+0000 7f20daffd700 1 -- 172.21.15.1:0/509420249 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f20e000fd00 con 0x7f20ec158420 2024-03-18T20:06:35.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.620+0000 7f20daffd700 1 --2- 172.21.15.1:0/509420249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f20cc06adb0 0x7f20cc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:35.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.620+0000 7f20daffd700 1 -- 172.21.15.1:0/509420249 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f20e00894e0 con 0x7f20ec158420 2024-03-18T20:06:35.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.620+0000 7f20ea59c700 1 --2- 172.21.15.1:0/509420249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f20cc06adb0 0x7f20cc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:35.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.621+0000 7f20ea59c700 1 --2- 172.21.15.1:0/509420249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f20cc06adb0 0x7f20cc06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f20dc001800 tx=0x7f20dc009030 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:35.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.625+0000 7f20daffd700 1 -- 172.21.15.1:0/509420249 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f20e0054550 con 0x7f20ec158420 2024-03-18T20:06:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.875+0000 7f20f08db700 1 -- 172.21.15.1:0/509420249 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20ec06f120 con 0x7f20ec158420 2024-03-18T20:06:35.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.876+0000 7f20daffd700 1 -- 172.21.15.1:0/509420249 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+3529 (secure 0 0 0) 0x7f20e0057b70 con 0x7f20ec158420 2024-03-18T20:06:35.881 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:35.881 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":75,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8608,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3436643592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3436643592},{"type":"v1","addr":"172.21.15.165:6834","nonce":3436643592}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":8611,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1969816196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1969816196},{"type":"v1","addr":"172.21.15.165:6835","nonce":1969816196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":9375,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2671786416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2671786416},{"type":"v1","addr":"172.21.15.1:6836","nonce":2671786416}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68},{"gid":9828,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1007347015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1007347015},{"type":"v1","addr":"172.21.15.1:6837","nonce":1007347015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[]} 2024-03-18T20:06:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.878+0000 7f20f08db700 1 -- 172.21.15.1:0/509420249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f20cc06adb0 msgr2=0x7f20cc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.878+0000 7f20f08db700 1 --2- 172.21.15.1:0/509420249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f20cc06adb0 0x7f20cc06d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f20dc001800 tx=0x7f20dc009030 comp rx=0 tx=0).stop 2024-03-18T20:06:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.878+0000 7f20f08db700 1 -- 172.21.15.1:0/509420249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec158420 msgr2=0x7f20ec1588a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.878+0000 7f20f08db700 1 --2- 172.21.15.1:0/509420249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec158420 0x7f20ec1588a0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f20e00072c0 tx=0x7f20e0007680 comp rx=0 tx=0).stop 2024-03-18T20:06:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.878+0000 7f20f08db700 1 -- 172.21.15.1:0/509420249 shutdown_connections 2024-03-18T20:06:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.878+0000 7f20f08db700 1 --2- 172.21.15.1:0/509420249 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f20cc06adb0 0x7f20cc06d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.878+0000 7f20f08db700 1 --2- 172.21.15.1:0/509420249 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f20d4003a00 0x7f20ec148240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.878+0000 7f20f08db700 1 --2- 172.21.15.1:0/509420249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20ec0b0bb0 0x7f20ec0b1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.878+0000 7f20f08db700 1 --2- 172.21.15.1:0/509420249 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec158420 0x7f20ec1588a0 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.878+0000 7f20f08db700 1 -- 172.21.15.1:0/509420249 >> 172.21.15.1:0/509420249 conn(0x7f20ec06ab70 msgr2=0x7f20ec06cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.878+0000 7f20f08db700 1 -- 172.21.15.1:0/509420249 shutdown_connections 2024-03-18T20:06:35.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:35.878+0000 7f20f08db700 1 -- 172.21.15.1:0/509420249 wait complete. 2024-03-18T20:06:35.884 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 75 2024-03-18T20:06:35.895 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T20:06:35.895 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-18T20:06:35.895 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T20:06:35.895 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:06:35.896 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T20:06:35.896 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-18T20:06:35.896 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-18T20:06:35.896 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:06:35.896 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T20:06:35.896 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-18T20:06:35.896 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-18T20:06:35.896 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:06:35.897 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T20:06:35.897 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-18T20:06:35.897 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-18T20:06:35.897 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:06:35.905 INFO:tasks.ceph.mds.a:Stopped 2024-03-18T20:06:35.905 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-18T20:06:35.906 INFO:tasks.ceph.mds.c:Stopped 2024-03-18T20:06:35.906 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-18T20:06:35.911 INFO:tasks.ceph.mds.b:Stopped 2024-03-18T20:06:35.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-18T20:06:35.912 INFO:tasks.ceph.mds.d:Stopped 2024-03-18T20:06:35.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-18T20:06:35.947 INFO:tasks.ceph.mds.a:Started 2024-03-18T20:06:35.947 DEBUG:teuthology.parallel:result is None 2024-03-18T20:06:35.948 INFO:tasks.ceph.mds.c:Started 2024-03-18T20:06:35.948 DEBUG:teuthology.parallel:result is None 2024-03-18T20:06:35.953 INFO:tasks.ceph.mds.b:Started 2024-03-18T20:06:35.953 DEBUG:teuthology.parallel:result is None 2024-03-18T20:06:35.954 INFO:tasks.ceph.mds.d:Started 2024-03-18T20:06:35.954 DEBUG:teuthology.parallel:result is None 2024-03-18T20:06:35.954 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-18T20:06:36.026 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:06:36.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:06:36.030 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-18T20:06:36.030 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T20:06:36.040 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-03-18T20:06:36.041 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-03-18T20:06:36.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.086+0000 7fd2b43f0700 1 Processor -- start 2024-03-18T20:06:36.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.086+0000 7fd2b43f0700 1 -- start start 2024-03-18T20:06:36.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.088+0000 7fd2b43f0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2ac099a40 0x7fd2ac0ade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:36.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.088+0000 7fd2b43f0700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd2ac06fa50 0x7fd2ac06fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:36.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.088+0000 7fd2b43f0700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ac0b3aa0 con 0x7fd2ac099e20 2024-03-18T20:06:36.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.088+0000 7fd2b43f0700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ac06e9b0 con 0x7fd2ac06fa50 2024-03-18T20:06:36.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.088+0000 7fd2b43f0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac06ec90 con 0x7fd2ac099a40 2024-03-18T20:06:36.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.088+0000 7fd2b198b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd2ac099e20 0x7fd2ac0cc6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42234/0 (socket says 172.21.15.1:42234) 2024-03-18T20:06:36.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.088+0000 7fd2b198b700 1 -- 172.21.15.1:0/4084241713 learned_addr learned my addr 172.21.15.1:0/4084241713 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:36.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.088+0000 7fd2b218c700 1 --2- 172.21.15.1:0/4084241713 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2ac099a40 0x7fd2ac0ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:36.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.088+0000 7fd2b298d700 1 --2- 172.21.15.1:0/4084241713 >> v2:172.21.15.165:3300/0 conn(0x7fd2ac06fa50 0x7fd2ac06fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:36.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.088+0000 7fd2b118a700 1 -- 172.21.15.1:0/4084241713 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786762011 0 0) 0x7fd2ac0b3aa0 con 0x7fd2ac099e20 2024-03-18T20:06:36.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.088+0000 7fd2b118a700 1 -- 172.21.15.1:0/4084241713 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2980035b0 con 0x7fd2ac099e20 2024-03-18T20:06:36.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.088+0000 7fd2b298d700 1 -- 172.21.15.1:0/4084241713 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2ac099a40 msgr2=0x7fd2ac0ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:36.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.088+0000 7fd2b298d700 1 --2- 172.21.15.1:0/4084241713 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2ac099a40 0x7fd2ac0ade00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:36.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.088+0000 7fd2b298d700 1 -- 172.21.15.1:0/4084241713 >> v1:172.21.15.1:6789/0 conn(0x7fd2ac099e20 legacy=0x7fd2ac0cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:36.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.088+0000 7fd2b298d700 1 -- 172.21.15.1:0/4084241713 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ac06ef70 con 0x7fd2ac06fa50 2024-03-18T20:06:36.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.088+0000 7fd2b218c700 1 --2- 172.21.15.1:0/4084241713 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2ac099a40 0x7fd2ac0ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:36.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.089+0000 7fd2b298d700 1 --2- 172.21.15.1:0/4084241713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2ac06fa50 0x7fd2ac06fee0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4009d90 tx=0x7fd2a400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=136b1cbd6caca38 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:36.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.089+0000 7fd2b118a700 1 -- 172.21.15.1:0/4084241713 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a400e070 con 0x7fd2ac06fa50 2024-03-18T20:06:36.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.089+0000 7fd2b118a700 1 -- 172.21.15.1:0/4084241713 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a400bb20 con 0x7fd2ac06fa50 2024-03-18T20:06:36.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.089+0000 7fd2b118a700 1 -- 172.21.15.1:0/4084241713 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a400be90 con 0x7fd2ac06fa50 2024-03-18T20:06:36.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.089+0000 7fd2b43f0700 1 -- 172.21.15.1:0/4084241713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2ac06fa50 msgr2=0x7fd2ac06fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:36.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.089+0000 7fd2b43f0700 1 --2- 172.21.15.1:0/4084241713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2ac06fa50 0x7fd2ac06fee0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4009d90 tx=0x7fd2a400b330 comp rx=0 tx=0).stop 2024-03-18T20:06:36.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.089+0000 7fd2b43f0700 1 -- 172.21.15.1:0/4084241713 shutdown_connections 2024-03-18T20:06:36.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.089+0000 7fd2b43f0700 1 --2- 172.21.15.1:0/4084241713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2ac06fa50 0x7fd2ac06fee0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:36.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.089+0000 7fd2b43f0700 1 --2- 172.21.15.1:0/4084241713 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2ac099a40 0x7fd2ac0ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:36.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.089+0000 7fd2b43f0700 1 -- 172.21.15.1:0/4084241713 >> 172.21.15.1:0/4084241713 conn(0x7fd2ac0b2c50 msgr2=0x7fd2ac0b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:36.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.090+0000 7fd2b43f0700 1 -- 172.21.15.1:0/4084241713 shutdown_connections 2024-03-18T20:06:36.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.090+0000 7fd2b43f0700 1 -- 172.21.15.1:0/4084241713 wait complete. 2024-03-18T20:06:36.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.091+0000 7fd2b43f0700 1 Processor -- start 2024-03-18T20:06:36.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.091+0000 7fd2b43f0700 1 -- start start 2024-03-18T20:06:36.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.091+0000 7fd2b43f0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2ac06fa50 0x7fd2ac155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:36.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b43f0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2ac099a40 0x7fd2ac0ca6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:36.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b43f0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2ac099e20 0x7fd2ac0cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:36.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b43f0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac0c9b70 con 0x7fd2ac099a40 2024-03-18T20:06:36.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b43f0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac0c8f60 con 0x7fd2ac06fa50 2024-03-18T20:06:36.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b43f0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac0c9240 con 0x7fd2ac099e20 2024-03-18T20:06:36.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b198b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2ac099a40 0x7fd2ac0ca6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b198b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2ac099a40 0x7fd2ac0ca6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58030/0 (socket says 172.21.15.1:58030) 2024-03-18T20:06:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b198b700 1 -- 172.21.15.1:0/2495592660 learned_addr learned my addr 172.21.15.1:0/2495592660 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b298d700 1 --2- 172.21.15.1:0/2495592660 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2ac099e20 0x7fd2ac0cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b218c700 1 --2- 172.21.15.1:0/2495592660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2ac06fa50 0x7fd2ac155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b198b700 1 -- 172.21.15.1:0/2495592660 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2ac099e20 msgr2=0x7fd2ac0cabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b198b700 1 --2- 172.21.15.1:0/2495592660 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2ac099e20 0x7fd2ac0cabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b198b700 1 -- 172.21.15.1:0/2495592660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2ac06fa50 msgr2=0x7fd2ac155840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b198b700 1 --2- 172.21.15.1:0/2495592660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2ac06fa50 0x7fd2ac155840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b198b700 1 -- 172.21.15.1:0/2495592660 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a80094a0 con 0x7fd2ac099a40 2024-03-18T20:06:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b218c700 1 --2- 172.21.15.1:0/2495592660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2ac06fa50 0x7fd2ac155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b298d700 1 --2- 172.21.15.1:0/2495592660 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2ac099e20 0x7fd2ac0cabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b198b700 1 --2- 172.21.15.1:0/2495592660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2ac099a40 0x7fd2ac0ca6b0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00c420 tx=0x7fd29c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2a2ffd700 1 -- 172.21.15.1:0/2495592660 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c005d70 con 0x7fd2ac099a40 2024-03-18T20:06:36.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2a2ffd700 1 -- 172.21.15.1:0/2495592660 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c0070d0 con 0x7fd2ac099a40 2024-03-18T20:06:36.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2a2ffd700 1 -- 172.21.15.1:0/2495592660 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c007440 con 0x7fd2ac099a40 2024-03-18T20:06:36.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b43f0700 1 -- 172.21.15.1:0/2495592660 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a4009a70 con 0x7fd2ac099a40 2024-03-18T20:06:36.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.092+0000 7fd2b43f0700 1 -- 172.21.15.1:0/2495592660 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ac0c9800 con 0x7fd2ac099a40 2024-03-18T20:06:36.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.093+0000 7fd2b43f0700 1 -- 172.21.15.1:0/2495592660 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ac073350 con 0x7fd2ac099a40 2024-03-18T20:06:36.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.094+0000 7fd2a2ffd700 1 -- 172.21.15.1:0/2495592660 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd29c0075a0 con 0x7fd2ac099a40 2024-03-18T20:06:36.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.094+0000 7fd2a2ffd700 1 --2- 172.21.15.1:0/2495592660 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd29006adb0 0x7fd29006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:36.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.094+0000 7fd2a2ffd700 1 -- 172.21.15.1:0/2495592660 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5937+0+0 (secure 0 0 0) 0x7fd29c088870 con 0x7fd2ac099a40 2024-03-18T20:06:36.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.094+0000 7fd2b218c700 1 --2- 172.21.15.1:0/2495592660 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd29006adb0 0x7fd29006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:36.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.095+0000 7fd2b218c700 1 --2- 172.21.15.1:0/2495592660 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd29006adb0 0x7fd29006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0b2b80 tx=0x7fd2a8008040 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:36.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.099+0000 7fd2a2ffd700 1 -- 172.21.15.1:0/2495592660 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd29c053990 con 0x7fd2ac099a40 2024-03-18T20:06:36.113 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-18T20:06:36.113 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-03-18T20:06:36.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:36.318+0000 7fd2b43f0700 1 -- 172.21.15.1:0/2495592660 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd2ac0cb490 con 0x7fd2ac099a40 2024-03-18T20:06:37.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.266+0000 7fd2a2ffd700 1 -- 172.21.15.1:0/2495592660 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v177) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd29c017020 con 0x7fd2ac099a40 2024-03-18T20:06:37.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.269+0000 7fd2b43f0700 1 -- 172.21.15.1:0/2495592660 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd29006adb0 msgr2=0x7fd29006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:37.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.269+0000 7fd2b43f0700 1 --2- 172.21.15.1:0/2495592660 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd29006adb0 0x7fd29006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0b2b80 tx=0x7fd2a8008040 comp rx=0 tx=0).stop 2024-03-18T20:06:37.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.269+0000 7fd2b43f0700 1 -- 172.21.15.1:0/2495592660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2ac099a40 msgr2=0x7fd2ac0ca6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:37.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.269+0000 7fd2b43f0700 1 --2- 172.21.15.1:0/2495592660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2ac099a40 0x7fd2ac0ca6b0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00c420 tx=0x7fd29c00c7e0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.269+0000 7fd2b43f0700 1 -- 172.21.15.1:0/2495592660 shutdown_connections 2024-03-18T20:06:37.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.269+0000 7fd2b43f0700 1 --2- 172.21.15.1:0/2495592660 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd29006adb0 0x7fd29006d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.269+0000 7fd2b43f0700 1 --2- 172.21.15.1:0/2495592660 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2ac099e20 0x7fd2ac0cabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.269+0000 7fd2b43f0700 1 --2- 172.21.15.1:0/2495592660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2ac06fa50 0x7fd2ac155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.269+0000 7fd2b43f0700 1 --2- 172.21.15.1:0/2495592660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2ac099a40 0x7fd2ac0ca6b0 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.269+0000 7fd2b43f0700 1 -- 172.21.15.1:0/2495592660 >> 172.21.15.1:0/2495592660 conn(0x7fd2ac0b2c50 msgr2=0x7fd2ac0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:37.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.269+0000 7fd2b43f0700 1 -- 172.21.15.1:0/2495592660 shutdown_connections 2024-03-18T20:06:37.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.269+0000 7fd2b43f0700 1 -- 172.21.15.1:0/2495592660 wait complete. 2024-03-18T20:06:37.276 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-18T20:06:37.289 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-18T20:06:37.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.458+0000 7fa705698700 1 Processor -- start 2024-03-18T20:06:37.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.458+0000 7fa705698700 1 -- start start 2024-03-18T20:06:37.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.458+0000 7fa705698700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa700160940 0x7fa7000d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:37.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.458+0000 7fa705698700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa7000cf090 0x7fa7000d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:37.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.458+0000 7fa705698700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7000cddc0 con 0x7fa700160d20 2024-03-18T20:06:37.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.458+0000 7fa705698700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7000cdff0 con 0x7fa7000cf090 2024-03-18T20:06:37.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.458+0000 7fa705698700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7000ce2d0 con 0x7fa700160940 2024-03-18T20:06:37.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.459+0000 7fa6feffd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa700160940 0x7fa7000d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:37.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.459+0000 7fa6fe7fc700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa7000cf090 0x7fa7000d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:37.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.459+0000 7fa6feffd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa700160940 0x7fa7000d0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:45176/0 (socket says 172.21.15.1:45176) 2024-03-18T20:06:37.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.459+0000 7fa6fe7fc700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa7000cf090 0x7fa7000d1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:33684/0 (socket says 172.21.15.1:33684) 2024-03-18T20:06:37.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.459+0000 7fa6feffd700 1 -- 172.21.15.1:0/2328608168 learned_addr learned my addr 172.21.15.1:0/2328608168 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:37.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.459+0000 7fa6ff7fe700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fa700160d20 0x7fa700161230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:48852/0 (socket says 172.21.15.1:48852) 2024-03-18T20:06:37.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.459+0000 7fa6fe7fc700 1 -- 172.21.15.1:0/2328608168 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa700160940 msgr2=0x7fa7000d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:37.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.459+0000 7fa6fe7fc700 1 --2- 172.21.15.1:0/2328608168 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa700160940 0x7fa7000d0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.459+0000 7fa6fe7fc700 1 -- 172.21.15.1:0/2328608168 >> v1:172.21.15.165:6789/0 conn(0x7fa700160d20 legacy=0x7fa700161230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:37.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.459+0000 7fa6fe7fc700 1 -- 172.21.15.1:0/2328608168 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7000ce5b0 con 0x7fa7000cf090 2024-03-18T20:06:37.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.459+0000 7fa6fdffb700 1 -- 172.21.15.1:0/2328608168 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165458564 0 0) 0x7fa7000cddc0 con 0x7fa700160d20 2024-03-18T20:06:37.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.459+0000 7fa6feffd700 1 --2- 172.21.15.1:0/2328608168 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa700160940 0x7fa7000d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:37.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.460+0000 7fa6fe7fc700 1 --2- 172.21.15.1:0/2328608168 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7000cf090 0x7fa7000d1790 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa6f4009840 tx=0x7fa6f4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc8c7b3ee3b0b266 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:37.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.460+0000 7fa6fdffb700 1 -- 172.21.15.1:0/2328608168 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f4011830 con 0x7fa7000cf090 2024-03-18T20:06:37.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.460+0000 7fa6fdffb700 1 -- 172.21.15.1:0/2328608168 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f4011990 con 0x7fa7000cf090 2024-03-18T20:06:37.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.460+0000 7fa6fdffb700 1 -- 172.21.15.1:0/2328608168 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f4011d20 con 0x7fa7000cf090 2024-03-18T20:06:37.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.460+0000 7fa705698700 1 -- 172.21.15.1:0/2328608168 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7000cf090 msgr2=0x7fa7000d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:37.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.460+0000 7fa705698700 1 --2- 172.21.15.1:0/2328608168 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7000cf090 0x7fa7000d1790 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa6f4009840 tx=0x7fa6f4009b50 comp rx=0 tx=0).stop 2024-03-18T20:06:37.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.460+0000 7fa705698700 1 -- 172.21.15.1:0/2328608168 shutdown_connections 2024-03-18T20:06:37.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.460+0000 7fa705698700 1 --2- 172.21.15.1:0/2328608168 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa700160940 0x7fa7000d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.460+0000 7fa705698700 1 --2- 172.21.15.1:0/2328608168 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7000cf090 0x7fa7000d1790 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.460+0000 7fa705698700 1 -- 172.21.15.1:0/2328608168 >> 172.21.15.1:0/2328608168 conn(0x7fa70009c520 msgr2=0x7fa7000d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:37.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.460+0000 7fa705698700 1 -- 172.21.15.1:0/2328608168 shutdown_connections 2024-03-18T20:06:37.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.460+0000 7fa705698700 1 -- 172.21.15.1:0/2328608168 wait complete. 2024-03-18T20:06:37.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.462+0000 7fa705698700 1 Processor -- start 2024-03-18T20:06:37.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.462+0000 7fa705698700 1 -- start start 2024-03-18T20:06:37.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.463+0000 7fa705698700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7000cf090 0x7fa7001556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:37.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.463+0000 7fa705698700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa700160940 0x7fa7000b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:37.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.463+0000 7fa705698700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa700160d20 0x7fa7000b4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:37.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.463+0000 7fa705698700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa700155c20 con 0x7fa7000cf090 2024-03-18T20:06:37.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.463+0000 7fa705698700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70015f070 con 0x7fa700160940 2024-03-18T20:06:37.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.463+0000 7fa705698700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa70015f350 con 0x7fa700160d20 2024-03-18T20:06:37.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.463+0000 7fa6feffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7000cf090 0x7fa7001556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:37.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.463+0000 7fa6feffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7000cf090 0x7fa7001556e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58036/0 (socket says 172.21.15.1:58036) 2024-03-18T20:06:37.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.463+0000 7fa6feffd700 1 -- 172.21.15.1:0/1103793996 learned_addr learned my addr 172.21.15.1:0/1103793996 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.463+0000 7fa6fe7fc700 1 --2- 172.21.15.1:0/1103793996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa700160940 0x7fa7000b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.463+0000 7fa6ff7fe700 1 --2- 172.21.15.1:0/1103793996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa700160d20 0x7fa7000b4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.463+0000 7fa6feffd700 1 -- 172.21.15.1:0/1103793996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa700160d20 msgr2=0x7fa7000b4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.463+0000 7fa6feffd700 1 --2- 172.21.15.1:0/1103793996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa700160d20 0x7fa7000b4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.464+0000 7fa6feffd700 1 -- 172.21.15.1:0/1103793996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa700160940 msgr2=0x7fa7000b3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.464+0000 7fa6feffd700 1 --2- 172.21.15.1:0/1103793996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa700160940 0x7fa7000b3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.464+0000 7fa6feffd700 1 -- 172.21.15.1:0/1103793996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f0009550 con 0x7fa7000cf090 2024-03-18T20:06:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.464+0000 7fa6feffd700 1 --2- 172.21.15.1:0/1103793996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7000cf090 0x7fa7001556e0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7fa6f000ebe0 tx=0x7fa6f000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.464+0000 7fa6e77fe700 1 -- 172.21.15.1:0/1103793996 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f000c850 con 0x7fa7000cf090 2024-03-18T20:06:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.464+0000 7fa6e77fe700 1 -- 172.21.15.1:0/1103793996 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f000c9b0 con 0x7fa7000cf090 2024-03-18T20:06:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.464+0000 7fa705698700 1 -- 172.21.15.1:0/1103793996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f40094a0 con 0x7fa7000cf090 2024-03-18T20:06:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.464+0000 7fa6e77fe700 1 -- 172.21.15.1:0/1103793996 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f000cd40 con 0x7fa7000cf090 2024-03-18T20:06:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.464+0000 7fa705698700 1 -- 172.21.15.1:0/1103793996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa70015f910 con 0x7fa7000cf090 2024-03-18T20:06:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.465+0000 7fa705698700 1 -- 172.21.15.1:0/1103793996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa700073350 con 0x7fa7000cf090 2024-03-18T20:06:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.465+0000 7fa6e77fe700 1 -- 172.21.15.1:0/1103793996 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa6f0013470 con 0x7fa7000cf090 2024-03-18T20:06:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.466+0000 7fa6e77fe700 1 --2- 172.21.15.1:0/1103793996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa6ec06adb0 0x7fa6ec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.466+0000 7fa6e77fe700 1 -- 172.21.15.1:0/1103793996 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fa6f0089730 con 0x7fa7000cf090 2024-03-18T20:06:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.466+0000 7fa6fe7fc700 1 --2- 172.21.15.1:0/1103793996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa6ec06adb0 0x7fa6ec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:37.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.466+0000 7fa6fe7fc700 1 --2- 172.21.15.1:0/1103793996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa6ec06adb0 0x7fa6ec06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa6f40090b0 tx=0x7fa6f4008030 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:37.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.470+0000 7fa6e77fe700 1 -- 172.21.15.1:0/1103793996 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa6f0054a50 con 0x7fa7000cf090 2024-03-18T20:06:37.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.753+0000 7fa705698700 1 -- 172.21.15.1:0/1103793996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa7000d0980 con 0x7fa7000cf090 2024-03-18T20:06:37.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.754+0000 7fa6e77fe700 1 -- 172.21.15.1:0/1103793996 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v41) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fa6f0058070 con 0x7fa7000cf090 2024-03-18T20:06:37.758 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:37.759 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:06:37.759 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-18T20:06:37.759 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:37.759 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-18T20:06:37.759 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlR93/CxAAbpe+5n3wHRur9xi8QzdCnQ==", 2024-03-18T20:06:37.759 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:06:37.759 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:06:37.760 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:06:37.760 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:06:37.760 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:06:37.760 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.760 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:37.760 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:37.760 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-18T20:06:37.760 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJ3OmGBAA37u227nxrMsh/JHt71HBxQ==", 2024-03-18T20:06:37.760 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:06:37.760 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:06:37.760 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:06:37.760 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:06:37.760 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:06:37.761 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:37.761 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:37.761 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-18T20:06:37.761 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJZFoExAAmI16oxwNKbf8dKetPfyk+g==", 2024-03-18T20:06:37.761 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:06:37.761 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:06:37.761 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:06:37.761 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:06:37.761 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:06:37.762 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:37.762 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:37.762 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-18T20:06:37.762 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlQrcvIBAA1LMv+EgC7S6wspX6QF49Pg==", 2024-03-18T20:06:37.762 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:06:37.762 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:06:37.762 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:06:37.762 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:06:37.763 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:06:37.763 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:37.763 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:37.763 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-18T20:06:37.763 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCUjPhlYOVHChAARFQrksZmMIdTlntd8TgvrQ==", 2024-03-18T20:06:37.763 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:06:37.763 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:06:37.763 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:06:37.763 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:06:37.764 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:37.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:37.764 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-18T20:06:37.764 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCXjPhl2zSDCBAA6Al1gkWVB79ZI9djGpVChA==", 2024-03-18T20:06:37.764 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:06:37.764 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:06:37.764 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:06:37.764 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:06:37.764 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:37.764 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:37.765 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-18T20:06:37.765 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCajPhlSoG/BhAAR93USDe6DCQW+hREOsnTlw==", 2024-03-18T20:06:37.765 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:06:37.765 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:06:37.765 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:06:37.765 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:06:37.765 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:37.765 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:37.765 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-18T20:06:37.765 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCdjPhlhJg5BRAAWuQhcnxyjR5UOiM13yKdAQ==", 2024-03-18T20:06:37.765 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:06:37.766 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:06:37.766 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:06:37.766 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:06:37.766 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:37.766 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:37.766 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-18T20:06:37.766 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCjjPhlCFnpCRAAKIyOQyVdjXZUTnAS6Nw/ng==", 2024-03-18T20:06:37.766 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:06:37.766 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:06:37.766 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:06:37.767 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:06:37.767 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:37.767 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:37.767 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-18T20:06:37.767 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCmjPhlAeFkCBAAJcEFpRk/03lQJxj+l2hEPA==", 2024-03-18T20:06:37.767 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:06:37.767 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:06:37.767 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:06:37.767 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:06:37.768 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:37.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:37.768 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-18T20:06:37.768 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCpjPhl/xuuBhAA0LgKfXEvN2O11ZlRQJVOkw==", 2024-03-18T20:06:37.768 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:06:37.768 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:06:37.768 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:06:37.768 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:06:37.768 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:37.768 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:37.769 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-18T20:06:37.769 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCsjPhldiR4BRAA571oh6U8PzOv/4LzjYLF/A==", 2024-03-18T20:06:37.769 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:06:37.769 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:06:37.769 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:06:37.769 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:06:37.769 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.769 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:37.769 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:37.769 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-18T20:06:37.770 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlu+2xJBAANnXGGjfeqP6S2FTecsfOsg==", 2024-03-18T20:06:37.770 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:06:37.770 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:06:37.770 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-18T20:06:37.770 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-18T20:06:37.770 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:37.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:37.771 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-18T20:06:37.771 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlhJ9+KxAAEnrQrCrgJR6xI+pucwzAXA==", 2024-03-18T20:06:37.771 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:06:37.771 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:06:37.771 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:06:37.771 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:06:37.771 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:06:37.771 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:37.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:37.772 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.2", 2024-03-18T20:06:37.772 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlxBUpKBAAIGTwmLWyj+P2Yvl55QVcWA==", 2024-03-18T20:06:37.772 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:06:37.772 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:06:37.772 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:06:37.772 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:06:37.772 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:06:37.772 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:37.772 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:37.772 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.3", 2024-03-18T20:06:37.772 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlmao3MRAAHeiQ6eyBh9QjjKPCtxkC8g==", 2024-03-18T20:06:37.772 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:06:37.773 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:06:37.773 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:06:37.773 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:06:37.773 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:06:37.773 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:37.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:37.773 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-18T20:06:37.773 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCRjPhlemOuKRAAVUggmtB7+EiVgyfJZ8TCCA==", 2024-03-18T20:06:37.773 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:06:37.773 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:06:37.773 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:06:37.773 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:06:37.773 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:06:37.774 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:37.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:37.774 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-18T20:06:37.774 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhleIuOCBAA9CH+AHewQbn+gpVr+6lJyw==", 2024-03-18T20:06:37.774 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:06:37.774 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:06:37.774 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T20:06:37.774 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:06:37.774 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:37.774 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:37.774 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-18T20:06:37.774 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlL0pTBBAAdFiPcdwwz514cuLAvioxYA==", 2024-03-18T20:06:37.775 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:06:37.775 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:06:37.775 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T20:06:37.775 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:06:37.775 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.775 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:37.775 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-18T20:06:37.775 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:06:37.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.756+0000 7fa705698700 1 -- 172.21.15.1:0/1103793996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa6ec06adb0 msgr2=0x7fa6ec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.756+0000 7fa705698700 1 --2- 172.21.15.1:0/1103793996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa6ec06adb0 0x7fa6ec06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa6f40090b0 tx=0x7fa6f4008030 comp rx=0 tx=0).stop 2024-03-18T20:06:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.756+0000 7fa705698700 1 -- 172.21.15.1:0/1103793996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7000cf090 msgr2=0x7fa7001556e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.756+0000 7fa705698700 1 --2- 172.21.15.1:0/1103793996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7000cf090 0x7fa7001556e0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7fa6f000ebe0 tx=0x7fa6f000eef0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.756+0000 7fa705698700 1 -- 172.21.15.1:0/1103793996 shutdown_connections 2024-03-18T20:06:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.756+0000 7fa705698700 1 --2- 172.21.15.1:0/1103793996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa6ec06adb0 0x7fa6ec06d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.756+0000 7fa705698700 1 --2- 172.21.15.1:0/1103793996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa700160d20 0x7fa7000b4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.756+0000 7fa705698700 1 --2- 172.21.15.1:0/1103793996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa7000cf090 0x7fa7001556e0 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.756+0000 7fa705698700 1 --2- 172.21.15.1:0/1103793996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa700160940 0x7fa7000b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.756+0000 7fa705698700 1 -- 172.21.15.1:0/1103793996 >> 172.21.15.1:0/1103793996 conn(0x7fa70009c520 msgr2=0x7fa70006bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.756+0000 7fa705698700 1 -- 172.21.15.1:0/1103793996 shutdown_connections 2024-03-18T20:06:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.756+0000 7fa705698700 1 -- 172.21.15.1:0/1103793996 wait complete. 2024-03-18T20:06:37.778 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:06:37.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.900+0000 7fdf159b6700 1 Processor -- start 2024-03-18T20:06:37.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.900+0000 7fdf159b6700 1 -- start start 2024-03-18T20:06:37.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.901+0000 7fdf159b6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf1015a320 0x7fdf100ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:37.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.901+0000 7fdf159b6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdf100cea30 0x7fdf10071600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:37.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.901+0000 7fdf159b6700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf100acd80 con 0x7fdf10071b40 2024-03-18T20:06:37.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.901+0000 7fdf159b6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf10071ff0 con 0x7fdf100cea30 2024-03-18T20:06:37.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.901+0000 7fdf159b6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf100cf220 con 0x7fdf1015a320 2024-03-18T20:06:37.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.901+0000 7fdf0e7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdf100cea30 0x7fdf10071600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:37.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.901+0000 7fdf0f7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdf10071b40 0x7fdf100cad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42240/0 (socket says 172.21.15.1:42240) 2024-03-18T20:06:37.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.901+0000 7fdf0f7fe700 1 -- 172.21.15.1:0/251638016 learned_addr learned my addr 172.21.15.1:0/251638016 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:37.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.901+0000 7fdf0effd700 1 --2- 172.21.15.1:0/251638016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf1015a320 0x7fdf100ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:37.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.901+0000 7fdf0e7fc700 1 -- 172.21.15.1:0/251638016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf1015a320 msgr2=0x7fdf100ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:37.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.901+0000 7fdf0e7fc700 1 --2- 172.21.15.1:0/251638016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf1015a320 0x7fdf100ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.901+0000 7fdf0e7fc700 1 -- 172.21.15.1:0/251638016 >> v1:172.21.15.1:6789/0 conn(0x7fdf10071b40 legacy=0x7fdf100cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:37.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.901+0000 7fdf0e7fc700 1 -- 172.21.15.1:0/251638016 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf100cef40 con 0x7fdf100cea30 2024-03-18T20:06:37.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.901+0000 7fdf0dffb700 1 -- 172.21.15.1:0/251638016 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159420436 0 0) 0x7fdf100acd80 con 0x7fdf10071b40 2024-03-18T20:06:37.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.902+0000 7fdf0e7fc700 1 --2- 172.21.15.1:0/251638016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf100cea30 0x7fdf10071600 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fdf04009870 tx=0x7fdf04009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be45144b595d50af server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:37.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.902+0000 7fdf0dffb700 1 -- 172.21.15.1:0/251638016 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf04011830 con 0x7fdf100cea30 2024-03-18T20:06:37.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.902+0000 7fdf0dffb700 1 -- 172.21.15.1:0/251638016 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf04011990 con 0x7fdf100cea30 2024-03-18T20:06:37.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.902+0000 7fdf0dffb700 1 -- 172.21.15.1:0/251638016 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf04011d20 con 0x7fdf100cea30 2024-03-18T20:06:37.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.902+0000 7fdf159b6700 1 -- 172.21.15.1:0/251638016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf100cea30 msgr2=0x7fdf10071600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:37.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.902+0000 7fdf159b6700 1 --2- 172.21.15.1:0/251638016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf100cea30 0x7fdf10071600 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fdf04009870 tx=0x7fdf04009b80 comp rx=0 tx=0).stop 2024-03-18T20:06:37.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.902+0000 7fdf159b6700 1 -- 172.21.15.1:0/251638016 shutdown_connections 2024-03-18T20:06:37.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.902+0000 7fdf159b6700 1 --2- 172.21.15.1:0/251638016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf1015a320 0x7fdf100ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.902+0000 7fdf159b6700 1 --2- 172.21.15.1:0/251638016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf100cea30 0x7fdf10071600 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.902+0000 7fdf159b6700 1 -- 172.21.15.1:0/251638016 >> 172.21.15.1:0/251638016 conn(0x7fdf100cc990 msgr2=0x7fdf100ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:37.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.902+0000 7fdf159b6700 1 -- 172.21.15.1:0/251638016 shutdown_connections 2024-03-18T20:06:37.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.902+0000 7fdf159b6700 1 -- 172.21.15.1:0/251638016 wait complete. 2024-03-18T20:06:37.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.903+0000 7fdf159b6700 1 Processor -- start 2024-03-18T20:06:37.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.904+0000 7fdf159b6700 1 -- start start 2024-03-18T20:06:37.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.904+0000 7fdf159b6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf10071b40 0x7fdf1014c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:37.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.904+0000 7fdf159b6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf100cea30 0x7fdf1014ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:37.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.904+0000 7fdf159b6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf1014f110 0x7fdf1014d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:37.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.904+0000 7fdf159b6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1014e740 con 0x7fdf1014f110 2024-03-18T20:06:37.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.904+0000 7fdf159b6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1014db30 con 0x7fdf10071b40 2024-03-18T20:06:37.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.904+0000 7fdf159b6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1014de10 con 0x7fdf100cea30 2024-03-18T20:06:37.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.905+0000 7fdf0f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf1014f110 0x7fdf1014d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:37.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.905+0000 7fdf0f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf1014f110 0x7fdf1014d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58060/0 (socket says 172.21.15.1:58060) 2024-03-18T20:06:37.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.905+0000 7fdf0f7fe700 1 -- 172.21.15.1:0/1173953639 learned_addr learned my addr 172.21.15.1:0/1173953639 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:37.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.905+0000 7fdf0effd700 1 --2- 172.21.15.1:0/1173953639 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf10071b40 0x7fdf1014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:37.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.905+0000 7fdf0e7fc700 1 --2- 172.21.15.1:0/1173953639 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf100cea30 0x7fdf1014ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:37.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.905+0000 7fdf0f7fe700 1 -- 172.21.15.1:0/1173953639 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf100cea30 msgr2=0x7fdf1014ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:37.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.905+0000 7fdf0f7fe700 1 --2- 172.21.15.1:0/1173953639 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf100cea30 0x7fdf1014ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.905+0000 7fdf0f7fe700 1 -- 172.21.15.1:0/1173953639 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf10071b40 msgr2=0x7fdf1014c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:37.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.905+0000 7fdf0f7fe700 1 --2- 172.21.15.1:0/1173953639 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf10071b40 0x7fdf1014c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:37.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.905+0000 7fdf0f7fe700 1 -- 172.21.15.1:0/1173953639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf040094a0 con 0x7fdf1014f110 2024-03-18T20:06:37.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.905+0000 7fdf0f7fe700 1 --2- 172.21.15.1:0/1173953639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf1014f110 0x7fdf1014d030 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fdf0800cba0 tx=0x7fdf0800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:37.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.905+0000 7fdef7fff700 1 -- 172.21.15.1:0/1173953639 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0800ab80 con 0x7fdf1014f110 2024-03-18T20:06:37.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.905+0000 7fdef7fff700 1 -- 172.21.15.1:0/1173953639 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0800ad00 con 0x7fdf1014f110 2024-03-18T20:06:37.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.905+0000 7fdf159b6700 1 -- 172.21.15.1:0/1173953639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1014e0f0 con 0x7fdf1014f110 2024-03-18T20:06:37.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.905+0000 7fdef7fff700 1 -- 172.21.15.1:0/1173953639 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf08007310 con 0x7fdf1014f110 2024-03-18T20:06:37.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.905+0000 7fdf159b6700 1 -- 172.21.15.1:0/1173953639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1014cd90 con 0x7fdf1014f110 2024-03-18T20:06:37.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.906+0000 7fdf159b6700 1 -- 172.21.15.1:0/1173953639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf100676a0 con 0x7fdf1014f110 2024-03-18T20:06:37.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.907+0000 7fdef7fff700 1 -- 172.21.15.1:0/1173953639 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fdf08007470 con 0x7fdf1014f110 2024-03-18T20:06:37.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.907+0000 7fdef7fff700 1 --2- 172.21.15.1:0/1173953639 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdefc06adb0 0x7fdefc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:37.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.907+0000 7fdef7fff700 1 -- 172.21.15.1:0/1173953639 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fdf08089070 con 0x7fdf1014f110 2024-03-18T20:06:37.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.907+0000 7fdf0effd700 1 --2- 172.21.15.1:0/1173953639 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdefc06adb0 0x7fdefc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:37.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.907+0000 7fdf0effd700 1 --2- 172.21.15.1:0/1173953639 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdefc06adb0 0x7fdefc06d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009b70 tx=0x7fdf00008040 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:37.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:37.912+0000 7fdef7fff700 1 -- 172.21.15.1:0/1173953639 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdf08054390 con 0x7fdf1014f110 2024-03-18T20:06:38.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.117+0000 7fdf159b6700 1 -- 172.21.15.1:0/1173953639 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf1014e880 con 0x7fdf1014f110 2024-03-18T20:06:38.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.118+0000 7fdef7fff700 1 -- 172.21.15.1:0/1173953639 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v177) v1 ==== 74+0+12671 (secure 0 0 0) 0x7fdf080579b0 con 0x7fdf1014f110 2024-03-18T20:06:38.123 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:38.123 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":177,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:06:37.263037+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:06:38.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.120+0000 7fdf159b6700 1 -- 172.21.15.1:0/1173953639 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdefc06adb0 msgr2=0x7fdefc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.120+0000 7fdf159b6700 1 --2- 172.21.15.1:0/1173953639 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdefc06adb0 0x7fdefc06d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009b70 tx=0x7fdf00008040 comp rx=0 tx=0).stop 2024-03-18T20:06:38.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.120+0000 7fdf159b6700 1 -- 172.21.15.1:0/1173953639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf1014f110 msgr2=0x7fdf1014d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.120+0000 7fdf159b6700 1 --2- 172.21.15.1:0/1173953639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf1014f110 0x7fdf1014d030 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fdf0800cba0 tx=0x7fdf0800cf60 comp rx=0 tx=0).stop 2024-03-18T20:06:38.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.120+0000 7fdf159b6700 1 -- 172.21.15.1:0/1173953639 shutdown_connections 2024-03-18T20:06:38.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.120+0000 7fdf159b6700 1 --2- 172.21.15.1:0/1173953639 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdefc06adb0 0x7fdefc06d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.120+0000 7fdf159b6700 1 --2- 172.21.15.1:0/1173953639 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf1014f110 0x7fdf1014d030 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.120+0000 7fdf159b6700 1 --2- 172.21.15.1:0/1173953639 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf10071b40 0x7fdf1014c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.120+0000 7fdf159b6700 1 --2- 172.21.15.1:0/1173953639 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdf100cea30 0x7fdf1014ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.120+0000 7fdf159b6700 1 -- 172.21.15.1:0/1173953639 >> 172.21.15.1:0/1173953639 conn(0x7fdf100cc990 msgr2=0x7fdf1006a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:38.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.120+0000 7fdf159b6700 1 -- 172.21.15.1:0/1173953639 shutdown_connections 2024-03-18T20:06:38.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.120+0000 7fdf159b6700 1 -- 172.21.15.1:0/1173953639 wait complete. 2024-03-18T20:06:38.137 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:06:38.138 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:06:38.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.260+0000 7fb2381af700 1 Processor -- start 2024-03-18T20:06:38.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.260+0000 7fb2381af700 1 -- start start 2024-03-18T20:06:38.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.261+0000 7fb2381af700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb23015ad70 0x7fb23015d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.261+0000 7fb2381af700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb230160ea0 con 0x7fb23006a750 2024-03-18T20:06:38.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.261+0000 7fb2381af700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb23015ddf0 con 0x7fb2300727b0 2024-03-18T20:06:38.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.261+0000 7fb2381af700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2300725b0 con 0x7fb23015ad70 2024-03-18T20:06:38.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.261+0000 7fb23674c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb23006a750 0x7fb23006eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42250/0 (socket says 172.21.15.1:42250) 2024-03-18T20:06:38.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.261+0000 7fb23674c700 1 -- 172.21.15.1:0/2402340772 learned_addr learned my addr 172.21.15.1:0/2402340772 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:38.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.262+0000 7fb235f4b700 1 --2- 172.21.15.1:0/2402340772 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb23015ad70 0x7fb23015d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.262+0000 7fb234f49700 1 -- 172.21.15.1:0/2402340772 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950274749 0 0) 0x7fb230160ea0 con 0x7fb23006a750 2024-03-18T20:06:38.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.262+0000 7fb234f49700 1 -- 172.21.15.1:0/2402340772 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb21c0035b0 con 0x7fb23006a750 2024-03-18T20:06:38.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.262+0000 7fb234f49700 1 -- 172.21.15.1:0/2402340772 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824861525 0 0) 0x7fb23015ddf0 con 0x7fb2300727b0 2024-03-18T20:06:38.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.262+0000 7fb234f49700 1 -- 172.21.15.1:0/2402340772 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb230160ea0 con 0x7fb2300727b0 2024-03-18T20:06:38.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.262+0000 7fb235f4b700 1 -- 172.21.15.1:0/2402340772 >> v1:172.21.15.165:6789/0 conn(0x7fb2300727b0 legacy=0x7fb230161610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.262+0000 7fb234f49700 1 -- 172.21.15.1:0/2402340772 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2025985880 0 0) 0x7fb21c0035b0 con 0x7fb23006a750 2024-03-18T20:06:38.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.262+0000 7fb235f4b700 1 -- 172.21.15.1:0/2402340772 >> v1:172.21.15.1:6789/0 conn(0x7fb23006a750 legacy=0x7fb23006eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.262+0000 7fb235f4b700 1 -- 172.21.15.1:0/2402340772 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23015e090 con 0x7fb23015ad70 2024-03-18T20:06:38.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.262+0000 7fb235f4b700 1 --2- 172.21.15.1:0/2402340772 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb23015ad70 0x7fb23015d8b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb220009870 tx=0x7fb220009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d70b4946efcf2bf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:38.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.262+0000 7fb234f49700 1 -- 172.21.15.1:0/2402340772 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2280019f0 con 0x7fb23015ad70 2024-03-18T20:06:38.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.262+0000 7fb234f49700 1 -- 172.21.15.1:0/2402340772 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb220004030 con 0x7fb23015ad70 2024-03-18T20:06:38.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.262+0000 7fb234f49700 1 -- 172.21.15.1:0/2402340772 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2200043c0 con 0x7fb23015ad70 2024-03-18T20:06:38.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.263+0000 7fb2381af700 1 -- 172.21.15.1:0/2402340772 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb23015ad70 msgr2=0x7fb23015d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.263+0000 7fb2381af700 1 --2- 172.21.15.1:0/2402340772 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb23015ad70 0x7fb23015d8b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb220009870 tx=0x7fb220009b80 comp rx=0 tx=0).stop 2024-03-18T20:06:38.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.263+0000 7fb2381af700 1 -- 172.21.15.1:0/2402340772 shutdown_connections 2024-03-18T20:06:38.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.263+0000 7fb2381af700 1 --2- 172.21.15.1:0/2402340772 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb23015ad70 0x7fb23015d8b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.263+0000 7fb2381af700 1 -- 172.21.15.1:0/2402340772 >> 172.21.15.1:0/2402340772 conn(0x7fb2300b1cb0 msgr2=0x7fb2300b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:38.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.263+0000 7fb2381af700 1 -- 172.21.15.1:0/2402340772 shutdown_connections 2024-03-18T20:06:38.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.263+0000 7fb2381af700 1 -- 172.21.15.1:0/2402340772 wait complete. 2024-03-18T20:06:38.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.265+0000 7fb2381af700 1 Processor -- start 2024-03-18T20:06:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.265+0000 7fb2381af700 1 -- start start 2024-03-18T20:06:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb2381af700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23006a750 0x7fb2301533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb2381af700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2300727b0 0x7fb23015c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb2381af700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb23015ad70 0x7fb23015c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb2381af700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb230154090 con 0x7fb23006a750 2024-03-18T20:06:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb2381af700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2300ae8c0 con 0x7fb23015ad70 2024-03-18T20:06:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb2381af700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2300aeba0 con 0x7fb2300727b0 2024-03-18T20:06:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb235f4b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23006a750 0x7fb2301533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb235f4b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23006a750 0x7fb2301533e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58064/0 (socket says 172.21.15.1:58064) 2024-03-18T20:06:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb235f4b700 1 -- 172.21.15.1:0/1535976996 learned_addr learned my addr 172.21.15.1:0/1535976996 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:38.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb23574a700 1 --2- 172.21.15.1:0/1535976996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2300727b0 0x7fb23015c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb23674c700 1 --2- 172.21.15.1:0/1535976996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb23015ad70 0x7fb23015c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb235f4b700 1 -- 172.21.15.1:0/1535976996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2300727b0 msgr2=0x7fb23015c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb235f4b700 1 --2- 172.21.15.1:0/1535976996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2300727b0 0x7fb23015c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb235f4b700 1 -- 172.21.15.1:0/1535976996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb23015ad70 msgr2=0x7fb23015c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb235f4b700 1 --2- 172.21.15.1:0/1535976996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb23015ad70 0x7fb23015c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb235f4b700 1 -- 172.21.15.1:0/1535976996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2200094a0 con 0x7fb23006a750 2024-03-18T20:06:38.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb23674c700 1 --2- 172.21.15.1:0/1535976996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb23015ad70 0x7fb23015c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:38.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb235f4b700 1 --2- 172.21.15.1:0/1535976996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23006a750 0x7fb2301533e0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fb220000c00 tx=0x7fb220004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:38.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb2267fc700 1 -- 172.21.15.1:0/1535976996 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb220003d80 con 0x7fb23006a750 2024-03-18T20:06:38.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb2267fc700 1 -- 172.21.15.1:0/1535976996 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22001f070 con 0x7fb23006a750 2024-03-18T20:06:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb2381af700 1 -- 172.21.15.1:0/1535976996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2300aee80 con 0x7fb23006a750 2024-03-18T20:06:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb2267fc700 1 -- 172.21.15.1:0/1535976996 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb220011870 con 0x7fb23006a750 2024-03-18T20:06:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.266+0000 7fb2381af700 1 -- 172.21.15.1:0/1535976996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2300af440 con 0x7fb23006a750 2024-03-18T20:06:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.268+0000 7fb2381af700 1 -- 172.21.15.1:0/1535976996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2300af630 con 0x7fb23006a750 2024-03-18T20:06:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.268+0000 7fb2267fc700 1 -- 172.21.15.1:0/1535976996 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb220021030 con 0x7fb23006a750 2024-03-18T20:06:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.268+0000 7fb2267fc700 1 --2- 172.21.15.1:0/1535976996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb21406adb0 0x7fb21406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.268+0000 7fb2267fc700 1 -- 172.21.15.1:0/1535976996 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb22008a6a0 con 0x7fb23006a750 2024-03-18T20:06:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.268+0000 7fb23574a700 1 --2- 172.21.15.1:0/1535976996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb21406adb0 0x7fb21406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.269+0000 7fb23574a700 1 --2- 172.21.15.1:0/1535976996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb21406adb0 0x7fb21406d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb2300adc70 tx=0x7fb22c009040 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:38.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.273+0000 7fb2267fc700 1 -- 172.21.15.1:0/1535976996 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb220055ae0 con 0x7fb23006a750 2024-03-18T20:06:38.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.476+0000 7fb2381af700 1 -- 172.21.15.1:0/1535976996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb2300676a0 con 0x7fb23006a750 2024-03-18T20:06:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.476+0000 7fb2267fc700 1 -- 172.21.15.1:0/1535976996 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v177) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb220059100 con 0x7fb23006a750 2024-03-18T20:06:38.481 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:06:38.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.478+0000 7fb2381af700 1 -- 172.21.15.1:0/1535976996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb21406adb0 msgr2=0x7fb21406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.478+0000 7fb2381af700 1 --2- 172.21.15.1:0/1535976996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb21406adb0 0x7fb21406d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb2300adc70 tx=0x7fb22c009040 comp rx=0 tx=0).stop 2024-03-18T20:06:38.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.478+0000 7fb2381af700 1 -- 172.21.15.1:0/1535976996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23006a750 msgr2=0x7fb2301533e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.478+0000 7fb2381af700 1 --2- 172.21.15.1:0/1535976996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23006a750 0x7fb2301533e0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fb220000c00 tx=0x7fb220004750 comp rx=0 tx=0).stop 2024-03-18T20:06:38.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.478+0000 7fb2381af700 1 -- 172.21.15.1:0/1535976996 shutdown_connections 2024-03-18T20:06:38.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.478+0000 7fb2381af700 1 --2- 172.21.15.1:0/1535976996 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb21406adb0 0x7fb21406d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.478+0000 7fb2381af700 1 --2- 172.21.15.1:0/1535976996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb23015ad70 0x7fb23015c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.478+0000 7fb2381af700 1 --2- 172.21.15.1:0/1535976996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23006a750 0x7fb2301533e0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.478+0000 7fb2381af700 1 --2- 172.21.15.1:0/1535976996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2300727b0 0x7fb23015c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.478+0000 7fb2381af700 1 -- 172.21.15.1:0/1535976996 >> 172.21.15.1:0/1535976996 conn(0x7fb2300b1cb0 msgr2=0x7fb2300b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:38.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.478+0000 7fb2381af700 1 -- 172.21.15.1:0/1535976996 shutdown_connections 2024-03-18T20:06:38.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.478+0000 7fb2381af700 1 -- 172.21.15.1:0/1535976996 wait complete. 2024-03-18T20:06:38.495 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T20:06:38.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.616+0000 7f8e87622700 1 Processor -- start 2024-03-18T20:06:38.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.616+0000 7f8e87622700 1 -- start start 2024-03-18T20:06:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e87622700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e8006a750 0x7f8e800cf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e87622700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8e800cf960 0x7f8e8006ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e87622700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8006d640 con 0x7f8e800cc6d0 2024-03-18T20:06:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e87622700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e8006de50 con 0x7f8e800cf960 2024-03-18T20:06:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e87622700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e800acd80 con 0x7f8e8006a750 2024-03-18T20:06:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e85bbf700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8e800cc6d0 0x7f8e800ccb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42252/0 (socket says 172.21.15.1:42252) 2024-03-18T20:06:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e85bbf700 1 -- 172.21.15.1:0/1447586251 learned_addr learned my addr 172.21.15.1:0/1447586251 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e77fff700 1 -- 172.21.15.1:0/1447586251 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076178566 0 0) 0x7f8e8006d640 con 0x7f8e800cc6d0 2024-03-18T20:06:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e853be700 1 --2- 172.21.15.1:0/1447586251 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e8006a750 0x7f8e800cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e84bbd700 1 --2- 172.21.15.1:0/1447586251 >> v2:172.21.15.165:3300/0 conn(0x7f8e800cf960 0x7f8e8006ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e77fff700 1 -- 172.21.15.1:0/1447586251 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e6c0035b0 con 0x7f8e800cc6d0 2024-03-18T20:06:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e77fff700 1 -- 172.21.15.1:0/1447586251 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2479160832 0 0) 0x7f8e6c0035b0 con 0x7f8e800cc6d0 2024-03-18T20:06:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e77fff700 1 -- 172.21.15.1:0/1447586251 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e8006a750 msgr2=0x7f8e800cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e77fff700 1 --2- 172.21.15.1:0/1447586251 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e8006a750 0x7f8e800cf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e77fff700 1 -- 172.21.15.1:0/1447586251 >> v2:172.21.15.165:3300/0 conn(0x7f8e800cf960 msgr2=0x7f8e8006ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e77fff700 1 --2- 172.21.15.1:0/1447586251 >> v2:172.21.15.165:3300/0 conn(0x7f8e800cf960 0x7f8e8006ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e77fff700 1 -- 172.21.15.1:0/1447586251 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8006b070 con 0x7f8e800cc6d0 2024-03-18T20:06:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e77fff700 1 -- 172.21.15.1:0/1447586251 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f8e780019f0 con 0x7f8e800cc6d0 2024-03-18T20:06:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e77fff700 1 -- 172.21.15.1:0/1447586251 >> v1:172.21.15.1:6789/0 conn(0x7f8e800cc6d0 legacy=0x7f8e800ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e77fff700 1 --2- 172.21.15.1:0/1447586251 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e6c003a00 0x7f8e6c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.617+0000 7f8e77fff700 1 --2- 172.21.15.1:0/1447586251 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e8006a750 0x7f8e6c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.618+0000 7f8e77fff700 1 --2- 172.21.15.1:0/1447586251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c004d90 0x7f8e6c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.618+0000 7f8e77fff700 1 -- 172.21.15.1:0/1447586251 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8006de50 con 0x7f8e6c004d90 2024-03-18T20:06:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.618+0000 7f8e77fff700 1 -- 172.21.15.1:0/1447586251 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e800acd80 con 0x7f8e6c003a00 2024-03-18T20:06:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.618+0000 7f8e77fff700 1 -- 172.21.15.1:0/1447586251 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c00ba10 con 0x7f8e8006a750 2024-03-18T20:06:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.618+0000 7f8e85bbf700 1 --2- 172.21.15.1:0/1447586251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c004d90 0x7f8e6c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.618+0000 7f8e853be700 1 --2- 172.21.15.1:0/1447586251 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e6c003a00 0x7f8e6c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.618+0000 7f8e84bbd700 1 --2- 172.21.15.1:0/1447586251 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e8006a750 0x7f8e6c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.618+0000 7f8e85bbf700 1 -- 172.21.15.1:0/1447586251 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e8006a750 msgr2=0x7f8e6c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.618+0000 7f8e85bbf700 1 --2- 172.21.15.1:0/1447586251 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e8006a750 0x7f8e6c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.618+0000 7f8e85bbf700 1 -- 172.21.15.1:0/1447586251 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e6c003a00 msgr2=0x7f8e6c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.618+0000 7f8e85bbf700 1 --2- 172.21.15.1:0/1447586251 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e6c003a00 0x7f8e6c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.618+0000 7f8e85bbf700 1 -- 172.21.15.1:0/1447586251 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6c0035b0 con 0x7f8e6c004d90 2024-03-18T20:06:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.618+0000 7f8e853be700 1 --2- 172.21.15.1:0/1447586251 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e6c003a00 0x7f8e6c003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.618+0000 7f8e84bbd700 1 --2- 172.21.15.1:0/1447586251 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e8006a750 0x7f8e6c004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:38.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.618+0000 7f8e85bbf700 1 --2- 172.21.15.1:0/1447586251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c004d90 0x7f8e6c005210 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f8e7800a8d0 tx=0x7f8e6c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=432e38859249f299 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.618+0000 7f8e77fff700 1 -- 172.21.15.1:0/1447586251 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7800a340 con 0x7f8e6c004d90 2024-03-18T20:06:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.618+0000 7f8e77fff700 1 -- 172.21.15.1:0/1447586251 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e78012600 con 0x7f8e6c004d90 2024-03-18T20:06:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.618+0000 7f8e77fff700 1 -- 172.21.15.1:0/1447586251 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e780129b0 con 0x7f8e6c004d90 2024-03-18T20:06:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.619+0000 7f8e87622700 1 -- 172.21.15.1:0/1447586251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c004d90 msgr2=0x7f8e6c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.619+0000 7f8e87622700 1 --2- 172.21.15.1:0/1447586251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c004d90 0x7f8e6c005210 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f8e7800a8d0 tx=0x7f8e6c005880 comp rx=0 tx=0).stop 2024-03-18T20:06:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.619+0000 7f8e85bbf700 1 -- 172.21.15.1:0/1447586251 reap_dead start 2024-03-18T20:06:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.619+0000 7f8e87622700 1 -- 172.21.15.1:0/1447586251 shutdown_connections 2024-03-18T20:06:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.619+0000 7f8e87622700 1 -- 172.21.15.1:0/1447586251 >> 172.21.15.1:0/1447586251 conn(0x7f8e8006b860 msgr2=0x7f8e8006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.619+0000 7f8e87622700 1 -- 172.21.15.1:0/1447586251 shutdown_connections 2024-03-18T20:06:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.619+0000 7f8e87622700 1 -- 172.21.15.1:0/1447586251 wait complete. 2024-03-18T20:06:38.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.620+0000 7f8e87622700 1 Processor -- start 2024-03-18T20:06:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.620+0000 7f8e87622700 1 -- start start 2024-03-18T20:06:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e87622700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c004d90 0x7f8e80157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e87622700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e801f3940 0x7f8e800c9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e87622700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e800c96a0 0x7f8e800b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e87622700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e80157fe0 con 0x7f8e6c004d90 2024-03-18T20:06:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e87622700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8015c680 con 0x7f8e800c96a0 2024-03-18T20:06:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e87622700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8015c960 con 0x7f8e801f3940 2024-03-18T20:06:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e853be700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c004d90 0x7f8e80157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e853be700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c004d90 0x7f8e80157aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58086/0 (socket says 172.21.15.1:58086) 2024-03-18T20:06:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e853be700 1 -- 172.21.15.1:0/1437472 learned_addr learned my addr 172.21.15.1:0/1437472 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e85bbf700 1 --2- 172.21.15.1:0/1437472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e800c96a0 0x7f8e800b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e84bbd700 1 --2- 172.21.15.1:0/1437472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e801f3940 0x7f8e800c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e853be700 1 -- 172.21.15.1:0/1437472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e801f3940 msgr2=0x7f8e800c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e853be700 1 --2- 172.21.15.1:0/1437472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e801f3940 0x7f8e800c9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e853be700 1 -- 172.21.15.1:0/1437472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e800c96a0 msgr2=0x7f8e800b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e853be700 1 --2- 172.21.15.1:0/1437472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e800c96a0 0x7f8e800b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e853be700 1 -- 172.21.15.1:0/1437472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8015cc40 con 0x7f8e6c004d90 2024-03-18T20:06:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e85bbf700 1 --2- 172.21.15.1:0/1437472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e800c96a0 0x7f8e800b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e853be700 1 --2- 172.21.15.1:0/1437472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c004d90 0x7f8e80157aa0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c010850 tx=0x7f8e7c010c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:38.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e75ffb700 1 -- 172.21.15.1:0/1437472 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c010e50 con 0x7f8e6c004d90 2024-03-18T20:06:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.621+0000 7f8e75ffb700 1 -- 172.21.15.1:0/1437472 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7c004020 con 0x7f8e6c004d90 2024-03-18T20:06:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.622+0000 7f8e75ffb700 1 -- 172.21.15.1:0/1437472 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c0043b0 con 0x7f8e6c004d90 2024-03-18T20:06:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.622+0000 7f8e87622700 1 -- 172.21.15.1:0/1437472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e8015cf20 con 0x7f8e6c004d90 2024-03-18T20:06:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.622+0000 7f8e87622700 1 -- 172.21.15.1:0/1437472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e800b35d0 con 0x7f8e6c004d90 2024-03-18T20:06:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.623+0000 7f8e87622700 1 -- 172.21.15.1:0/1437472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e800b2b60 con 0x7f8e6c004d90 2024-03-18T20:06:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.623+0000 7f8e75ffb700 1 -- 172.21.15.1:0/1437472 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8e7c00e8d0 con 0x7f8e6c004d90 2024-03-18T20:06:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.624+0000 7f8e75ffb700 1 --2- 172.21.15.1:0/1437472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8e6406adb0 0x7f8e6406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.624+0000 7f8e75ffb700 1 -- 172.21.15.1:0/1437472 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8e7c0895d0 con 0x7f8e6c004d90 2024-03-18T20:06:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.624+0000 7f8e84bbd700 1 --2- 172.21.15.1:0/1437472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8e6406adb0 0x7f8e6406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.624+0000 7f8e84bbd700 1 --2- 172.21.15.1:0/1437472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8e6406adb0 0x7f8e6406d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8e80070070 tx=0x7f8e70008040 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:38.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.628+0000 7f8e75ffb700 1 -- 172.21.15.1:0/1437472 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8e7c0548f0 con 0x7f8e6c004d90 2024-03-18T20:06:38.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.835+0000 7f8e87622700 1 -- 172.21.15.1:0/1437472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8e800676a0 con 0x7f8e6c004d90 2024-03-18T20:06:38.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.836+0000 7f8e75ffb700 1 -- 172.21.15.1:0/1437472 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v177) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8e7c057f10 con 0x7f8e6c004d90 2024-03-18T20:06:38.841 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:38.841 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T20:06:38.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:38.841 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-18T20:06:38.841 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-18T20:06:38.841 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:38.842 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T20:06:38.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.838+0000 7f8e87622700 1 -- 172.21.15.1:0/1437472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8e6406adb0 msgr2=0x7f8e6406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.838+0000 7f8e87622700 1 --2- 172.21.15.1:0/1437472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8e6406adb0 0x7f8e6406d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8e80070070 tx=0x7f8e70008040 comp rx=0 tx=0).stop 2024-03-18T20:06:38.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.838+0000 7f8e87622700 1 -- 172.21.15.1:0/1437472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c004d90 msgr2=0x7f8e80157aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.838+0000 7f8e87622700 1 --2- 172.21.15.1:0/1437472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c004d90 0x7f8e80157aa0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c010850 tx=0x7f8e7c010c10 comp rx=0 tx=0).stop 2024-03-18T20:06:38.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.838+0000 7f8e87622700 1 -- 172.21.15.1:0/1437472 shutdown_connections 2024-03-18T20:06:38.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.838+0000 7f8e87622700 1 --2- 172.21.15.1:0/1437472 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8e6406adb0 0x7f8e6406d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.838+0000 7f8e87622700 1 --2- 172.21.15.1:0/1437472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e800c96a0 0x7f8e800b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.838+0000 7f8e87622700 1 --2- 172.21.15.1:0/1437472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e6c004d90 0x7f8e80157aa0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.838+0000 7f8e87622700 1 --2- 172.21.15.1:0/1437472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e801f3940 0x7f8e800c9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.838+0000 7f8e87622700 1 -- 172.21.15.1:0/1437472 >> 172.21.15.1:0/1437472 conn(0x7f8e8006b860 msgr2=0x7f8e80158300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:38.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.838+0000 7f8e87622700 1 -- 172.21.15.1:0/1437472 shutdown_connections 2024-03-18T20:06:38.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.838+0000 7f8e87622700 1 -- 172.21.15.1:0/1437472 wait complete. 2024-03-18T20:06:38.855 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T20:06:38.855 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T20:06:38.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.980+0000 7f9dfc1fe700 1 Processor -- start 2024-03-18T20:06:38.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.980+0000 7f9dfc1fe700 1 -- start start 2024-03-18T20:06:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.981+0000 7f9dfc1fe700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9df4070920 0x7f9df406bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.981+0000 7f9dfc1fe700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df4161390 con 0x7f9df406c0f0 2024-03-18T20:06:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.981+0000 7f9dfc1fe700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df415dc60 con 0x7f9df4146d70 2024-03-18T20:06:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.981+0000 7f9dfc1fe700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df406d030 con 0x7f9df4070920 2024-03-18T20:06:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.981+0000 7f9dfa79b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9df406c0f0 0x7f9df415e0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42262/0 (socket says 172.21.15.1:42262) 2024-03-18T20:06:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.981+0000 7f9dfa79b700 1 -- 172.21.15.1:0/2798485722 learned_addr learned my addr 172.21.15.1:0/2798485722 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df9799700 1 --2- 172.21.15.1:0/2798485722 >> v2:172.21.15.165:3300/0 conn(0x7f9df4070920 0x7f9df406bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df8f98700 1 -- 172.21.15.1:0/2798485722 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1141348324 0 0) 0x7f9dec001490 con 0x7f9df406c0f0 2024-03-18T20:06:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df8f98700 1 -- 172.21.15.1:0/2798485722 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9de00035b0 con 0x7f9df406c0f0 2024-03-18T20:06:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df8f98700 1 -- 172.21.15.1:0/2798485722 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 376781324 0 0) 0x7f9de00035b0 con 0x7f9df406c0f0 2024-03-18T20:06:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df8f98700 1 -- 172.21.15.1:0/2798485722 >> v2:172.21.15.165:3300/0 conn(0x7f9df4070920 msgr2=0x7f9df406bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df8f98700 1 --2- 172.21.15.1:0/2798485722 >> v2:172.21.15.165:3300/0 conn(0x7f9df4070920 0x7f9df406bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df8f98700 1 -- 172.21.15.1:0/2798485722 >> v1:172.21.15.165:6789/0 conn(0x7f9df4146d70 legacy=0x7f9df4070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df8f98700 1 -- 172.21.15.1:0/2798485722 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df406c520 con 0x7f9df406c0f0 2024-03-18T20:06:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df8f98700 1 -- 172.21.15.1:0/2798485722 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f9dec001b80 con 0x7f9df406c0f0 2024-03-18T20:06:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df8f98700 1 -- 172.21.15.1:0/2798485722 >> v1:172.21.15.1:6789/0 conn(0x7f9df406c0f0 legacy=0x7f9df415e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df8f98700 1 --2- 172.21.15.1:0/2798485722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de0003a00 0x7f9de0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df8f98700 1 --2- 172.21.15.1:0/2798485722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9de0004850 0x7f9de0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df8f98700 1 --2- 172.21.15.1:0/2798485722 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9de0005210 0x7f9de000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df8f98700 1 -- 172.21.15.1:0/2798485722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df406d030 con 0x7f9de0003a00 2024-03-18T20:06:38.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df8f98700 1 -- 172.21.15.1:0/2798485722 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de000c0b0 con 0x7f9de0004850 2024-03-18T20:06:38.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df8f98700 1 -- 172.21.15.1:0/2798485722 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de000c270 con 0x7f9de0005210 2024-03-18T20:06:38.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df9f9a700 1 --2- 172.21.15.1:0/2798485722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de0003a00 0x7f9de0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df9799700 1 --2- 172.21.15.1:0/2798485722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9de0004850 0x7f9de0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9dfa79b700 1 --2- 172.21.15.1:0/2798485722 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9de0005210 0x7f9de000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df9f9a700 1 -- 172.21.15.1:0/2798485722 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9de0005210 msgr2=0x7f9de000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:38.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df9f9a700 1 --2- 172.21.15.1:0/2798485722 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9de0005210 0x7f9de000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df9f9a700 1 -- 172.21.15.1:0/2798485722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9de0004850 msgr2=0x7f9de0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:38.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df9f9a700 1 --2- 172.21.15.1:0/2798485722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9de0004850 0x7f9de0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9df9f9a700 1 -- 172.21.15.1:0/2798485722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de00035b0 con 0x7f9de0003a00 2024-03-18T20:06:38.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.982+0000 7f9dfa79b700 1 -- 172.21.15.1:0/2798485722 reap_dead start 2024-03-18T20:06:38.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.983+0000 7f9df9f9a700 1 --2- 172.21.15.1:0/2798485722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de0003a00 0x7f9de0003e20 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f9df00017c0 tx=0x7f9de0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=680f39efdf0e00b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:38.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.983+0000 7f9df8f98700 1 -- 172.21.15.1:0/2798485722 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df00052d0 con 0x7f9de0003a00 2024-03-18T20:06:38.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.983+0000 7f9df8f98700 1 -- 172.21.15.1:0/2798485722 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df0005430 con 0x7f9de0003a00 2024-03-18T20:06:38.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.983+0000 7f9df8f98700 1 -- 172.21.15.1:0/2798485722 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df00038c0 con 0x7f9de0003a00 2024-03-18T20:06:38.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.983+0000 7f9dfc1fe700 1 -- 172.21.15.1:0/2798485722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de0003a00 msgr2=0x7f9de0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.983+0000 7f9dfc1fe700 1 --2- 172.21.15.1:0/2798485722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de0003a00 0x7f9de0003e20 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f9df00017c0 tx=0x7f9de0005880 comp rx=0 tx=0).stop 2024-03-18T20:06:38.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.983+0000 7f9dfc1fe700 1 -- 172.21.15.1:0/2798485722 shutdown_connections 2024-03-18T20:06:38.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.983+0000 7f9dfc1fe700 1 --2- 172.21.15.1:0/2798485722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de0003a00 0x7f9de0003e20 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.983+0000 7f9dfc1fe700 1 -- 172.21.15.1:0/2798485722 >> 172.21.15.1:0/2798485722 conn(0x7f9df40b2e80 msgr2=0x7f9df40b3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:38.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.983+0000 7f9dfc1fe700 1 -- 172.21.15.1:0/2798485722 shutdown_connections 2024-03-18T20:06:38.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.983+0000 7f9dfc1fe700 1 -- 172.21.15.1:0/2798485722 wait complete. 2024-03-18T20:06:38.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.985+0000 7f9dfc1fe700 1 Processor -- start 2024-03-18T20:06:38.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.985+0000 7f9dfc1fe700 1 -- start start 2024-03-18T20:06:38.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.985+0000 7f9dfc1fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9de0003a00 0x7f9df413c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.985+0000 7f9dfc1fe700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9df415c690 0x7f9df41e88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.985+0000 7f9dfc1fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9df413ca50 0x7f9df413ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.985+0000 7f9dfc1fe700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df413d410 con 0x7f9df413ca50 2024-03-18T20:06:38.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.985+0000 7f9dfc1fe700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df413c850 con 0x7f9df415c690 2024-03-18T20:06:38.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.985+0000 7f9dfc1fe700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df41e91e0 con 0x7f9de0003a00 2024-03-18T20:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.985+0000 7f9dfa79b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9df413ca50 0x7f9df413ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.985+0000 7f9dfa79b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9df413ca50 0x7f9df413ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58096/0 (socket says 172.21.15.1:58096) 2024-03-18T20:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.985+0000 7f9dfa79b700 1 -- 172.21.15.1:0/1304787230 learned_addr learned my addr 172.21.15.1:0/1304787230 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.986+0000 7f9df9799700 1 --2- 172.21.15.1:0/1304787230 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9df415c690 0x7f9df41e88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.986+0000 7f9df9f9a700 1 --2- 172.21.15.1:0/1304787230 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9de0003a00 0x7f9df413c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.986+0000 7f9dfa79b700 1 -- 172.21.15.1:0/1304787230 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9de0003a00 msgr2=0x7f9df413c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.986+0000 7f9dfa79b700 1 --2- 172.21.15.1:0/1304787230 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9de0003a00 0x7f9df413c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.986+0000 7f9dfa79b700 1 -- 172.21.15.1:0/1304787230 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9df415c690 msgr2=0x7f9df41e88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.986+0000 7f9dfa79b700 1 --2- 172.21.15.1:0/1304787230 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9df415c690 0x7f9df41e88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.986+0000 7f9dfa79b700 1 -- 172.21.15.1:0/1304787230 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df41e94c0 con 0x7f9df413ca50 2024-03-18T20:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.986+0000 7f9dfa79b700 1 --2- 172.21.15.1:0/1304787230 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9df413ca50 0x7f9df413ced0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f9dec000f80 tx=0x7f9dec007340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.986+0000 7f9dea7fc700 1 -- 172.21.15.1:0/1304787230 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec001b70 con 0x7f9df413ca50 2024-03-18T20:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.986+0000 7f9dea7fc700 1 -- 172.21.15.1:0/1304787230 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dec007ab0 con 0x7f9df413ca50 2024-03-18T20:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.986+0000 7f9dfc1fe700 1 -- 172.21.15.1:0/1304787230 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df41e97a0 con 0x7f9df413ca50 2024-03-18T20:06:38.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.986+0000 7f9dea7fc700 1 -- 172.21.15.1:0/1304787230 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec013630 con 0x7f9df413ca50 2024-03-18T20:06:38.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.986+0000 7f9dfc1fe700 1 -- 172.21.15.1:0/1304787230 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df413a1f0 con 0x7f9df413ca50 2024-03-18T20:06:38.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.987+0000 7f9dfc1fe700 1 -- 172.21.15.1:0/1304787230 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df4073350 con 0x7f9df413ca50 2024-03-18T20:06:38.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.987+0000 7f9dea7fc700 1 -- 172.21.15.1:0/1304787230 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9dec013790 con 0x7f9df413ca50 2024-03-18T20:06:38.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.988+0000 7f9dea7fc700 1 --2- 172.21.15.1:0/1304787230 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9dd806adb0 0x7f9dd806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:38.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.988+0000 7f9dea7fc700 1 -- 172.21.15.1:0/1304787230 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9dec089880 con 0x7f9df413ca50 2024-03-18T20:06:38.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.988+0000 7f9df9f9a700 1 --2- 172.21.15.1:0/1304787230 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9dd806adb0 0x7f9dd806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:38.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.988+0000 7f9df9f9a700 1 --2- 172.21.15.1:0/1304787230 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9dd806adb0 0x7f9dd806d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9df0005040 tx=0x7f9df0009030 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:38.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:38.992+0000 7f9dea7fc700 1 -- 172.21.15.1:0/1304787230 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9dec054b10 con 0x7f9df413ca50 2024-03-18T20:06:39.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:39.194+0000 7f9dfc1fe700 1 -- 172.21.15.1:0/1304787230 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9df4161390 con 0x7f9df413ca50 2024-03-18T20:06:39.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:39.907+0000 7f9dea7fc700 1 -- 172.21.15.1:0/1304787230 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v178) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9df4161390 con 0x7f9df413ca50 2024-03-18T20:06:39.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:39.911+0000 7f9dfc1fe700 1 -- 172.21.15.1:0/1304787230 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9dd806adb0 msgr2=0x7f9dd806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:39.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:39.911+0000 7f9dfc1fe700 1 --2- 172.21.15.1:0/1304787230 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9dd806adb0 0x7f9dd806d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9df0005040 tx=0x7f9df0009030 comp rx=0 tx=0).stop 2024-03-18T20:06:39.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:39.911+0000 7f9dfc1fe700 1 -- 172.21.15.1:0/1304787230 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9df413ca50 msgr2=0x7f9df413ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:39.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:39.911+0000 7f9dfc1fe700 1 --2- 172.21.15.1:0/1304787230 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9df413ca50 0x7f9df413ced0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f9dec000f80 tx=0x7f9dec007340 comp rx=0 tx=0).stop 2024-03-18T20:06:39.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:39.911+0000 7f9dfc1fe700 1 -- 172.21.15.1:0/1304787230 shutdown_connections 2024-03-18T20:06:39.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:39.911+0000 7f9dfc1fe700 1 --2- 172.21.15.1:0/1304787230 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9dd806adb0 0x7f9dd806d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:39.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:39.911+0000 7f9dfc1fe700 1 --2- 172.21.15.1:0/1304787230 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9de0003a00 0x7f9df413c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:39.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:39.911+0000 7f9dfc1fe700 1 --2- 172.21.15.1:0/1304787230 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9df413ca50 0x7f9df413ced0 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:39.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:39.911+0000 7f9dfc1fe700 1 --2- 172.21.15.1:0/1304787230 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9df415c690 0x7f9df41e88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:39.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:39.911+0000 7f9dfc1fe700 1 -- 172.21.15.1:0/1304787230 >> 172.21.15.1:0/1304787230 conn(0x7f9df40b2e80 msgr2=0x7f9df4070210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:39.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:39.912+0000 7f9dfc1fe700 1 -- 172.21.15.1:0/1304787230 shutdown_connections 2024-03-18T20:06:39.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:39.912+0000 7f9dfc1fe700 1 -- 172.21.15.1:0/1304787230 wait complete. 2024-03-18T20:06:39.918 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-18T20:06:39.932 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T20:06:40.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.067+0000 7f1df40a4700 1 Processor -- start 2024-03-18T20:06:40.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.067+0000 7f1df40a4700 1 -- start start 2024-03-18T20:06:40.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.068+0000 7f1df40a4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec15d5b0 0x7f1dec15d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:40.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.068+0000 7f1df40a4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1dec099a40 0x7f1dec099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:40.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.068+0000 7f1df40a4700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dec09a830 con 0x7f1dec09a3e0 2024-03-18T20:06:40.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.068+0000 7f1df40a4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dec0cbbb0 con 0x7f1dec099a40 2024-03-18T20:06:40.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.068+0000 7f1df40a4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec0b3b00 con 0x7f1dec15d5b0 2024-03-18T20:06:40.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.068+0000 7f1df163f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1dec099a40 0x7f1dec099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:40.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.068+0000 7f1df163f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1dec099a40 0x7f1dec099ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58104/0 (socket says 172.21.15.1:58104) 2024-03-18T20:06:40.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.068+0000 7f1df2641700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1dec09a3e0 0x7f1dec06ba70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42268/0 (socket says 172.21.15.1:42268) 2024-03-18T20:06:40.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.068+0000 7f1df163f700 1 -- 172.21.15.1:0/1062778174 learned_addr learned my addr 172.21.15.1:0/1062778174 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:40.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.068+0000 7f1df1e40700 1 --2- 172.21.15.1:0/1062778174 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec15d5b0 0x7f1dec15d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:40.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.068+0000 7f1df163f700 1 -- 172.21.15.1:0/1062778174 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec15d5b0 msgr2=0x7f1dec15d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:40.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.068+0000 7f1df163f700 1 --2- 172.21.15.1:0/1062778174 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec15d5b0 0x7f1dec15d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:40.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.068+0000 7f1df163f700 1 -- 172.21.15.1:0/1062778174 >> v1:172.21.15.1:6789/0 conn(0x7f1dec09a3e0 legacy=0x7f1dec06ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:40.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.068+0000 7f1df163f700 1 -- 172.21.15.1:0/1062778174 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dec0cf7f0 con 0x7f1dec099a40 2024-03-18T20:06:40.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.068+0000 7f1df0e3e700 1 -- 172.21.15.1:0/1062778174 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735953703 0 0) 0x7f1dec09a830 con 0x7f1dec09a3e0 2024-03-18T20:06:40.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.068+0000 7f1df1e40700 1 --2- 172.21.15.1:0/1062778174 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec15d5b0 0x7f1dec15d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:40.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.068+0000 7f1df163f700 1 --2- 172.21.15.1:0/1062778174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec099a40 0x7f1dec099ea0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f1de80098f0 tx=0x7f1de8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7583461ba4d4173b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:40.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.069+0000 7f1df0e3e700 1 -- 172.21.15.1:0/1062778174 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de800b8a0 con 0x7f1dec099a40 2024-03-18T20:06:40.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.069+0000 7f1df0e3e700 1 -- 172.21.15.1:0/1062778174 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de800ba00 con 0x7f1dec099a40 2024-03-18T20:06:40.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.069+0000 7f1df0e3e700 1 -- 172.21.15.1:0/1062778174 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de800bd90 con 0x7f1dec099a40 2024-03-18T20:06:40.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.069+0000 7f1df40a4700 1 -- 172.21.15.1:0/1062778174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec099a40 msgr2=0x7f1dec099ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:40.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.069+0000 7f1df40a4700 1 --2- 172.21.15.1:0/1062778174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec099a40 0x7f1dec099ea0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f1de80098f0 tx=0x7f1de8009c00 comp rx=0 tx=0).stop 2024-03-18T20:06:40.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.069+0000 7f1df40a4700 1 -- 172.21.15.1:0/1062778174 shutdown_connections 2024-03-18T20:06:40.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.069+0000 7f1df40a4700 1 --2- 172.21.15.1:0/1062778174 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec15d5b0 0x7f1dec15d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:40.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.069+0000 7f1df40a4700 1 --2- 172.21.15.1:0/1062778174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec099a40 0x7f1dec099ea0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:40.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.069+0000 7f1df40a4700 1 -- 172.21.15.1:0/1062778174 >> 172.21.15.1:0/1062778174 conn(0x7f1dec0b2bd0 msgr2=0x7f1dec0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:40.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.069+0000 7f1df40a4700 1 -- 172.21.15.1:0/1062778174 shutdown_connections 2024-03-18T20:06:40.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.069+0000 7f1df40a4700 1 -- 172.21.15.1:0/1062778174 wait complete. 2024-03-18T20:06:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.070+0000 7f1df40a4700 1 Processor -- start 2024-03-18T20:06:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.071+0000 7f1df40a4700 1 -- start start 2024-03-18T20:06:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.071+0000 7f1df40a4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dec099a40 0x7f1dec0b5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.071+0000 7f1df40a4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec09a3e0 0x7f1dec1490f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.071+0000 7f1df40a4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec15d5b0 0x7f1dec149630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.071+0000 7f1df40a4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec0c81b0 con 0x7f1dec09a3e0 2024-03-18T20:06:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.071+0000 7f1df40a4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec0c8400 con 0x7f1dec099a40 2024-03-18T20:06:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.071+0000 7f1df40a4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec0c86e0 con 0x7f1dec15d5b0 2024-03-18T20:06:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.071+0000 7f1df163f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec09a3e0 0x7f1dec1490f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.071+0000 7f1df163f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec09a3e0 0x7f1dec1490f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58112/0 (socket says 172.21.15.1:58112) 2024-03-18T20:06:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.071+0000 7f1df163f700 1 -- 172.21.15.1:0/2214197090 learned_addr learned my addr 172.21.15.1:0/2214197090 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.072+0000 7f1df2641700 1 --2- 172.21.15.1:0/2214197090 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec15d5b0 0x7f1dec149630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:40.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.072+0000 7f1df1e40700 1 --2- 172.21.15.1:0/2214197090 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dec099a40 0x7f1dec0b5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.072+0000 7f1df163f700 1 -- 172.21.15.1:0/2214197090 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec15d5b0 msgr2=0x7f1dec149630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.072+0000 7f1df163f700 1 --2- 172.21.15.1:0/2214197090 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec15d5b0 0x7f1dec149630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.072+0000 7f1df163f700 1 -- 172.21.15.1:0/2214197090 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dec099a40 msgr2=0x7f1dec0b5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.072+0000 7f1df163f700 1 --2- 172.21.15.1:0/2214197090 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dec099a40 0x7f1dec0b5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.072+0000 7f1df163f700 1 -- 172.21.15.1:0/2214197090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de8009550 con 0x7f1dec09a3e0 2024-03-18T20:06:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.072+0000 7f1df163f700 1 --2- 172.21.15.1:0/2214197090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec09a3e0 0x7f1dec1490f0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f1de8000c00 tx=0x7f1de8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.072+0000 7f1de27fc700 1 -- 172.21.15.1:0/2214197090 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de80036a0 con 0x7f1dec09a3e0 2024-03-18T20:06:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.072+0000 7f1de27fc700 1 -- 172.21.15.1:0/2214197090 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de801e070 con 0x7f1dec09a3e0 2024-03-18T20:06:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.072+0000 7f1df40a4700 1 -- 172.21.15.1:0/2214197090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dec0c89c0 con 0x7f1dec09a3e0 2024-03-18T20:06:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.072+0000 7f1de27fc700 1 -- 172.21.15.1:0/2214197090 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8003a20 con 0x7f1dec09a3e0 2024-03-18T20:06:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.072+0000 7f1df40a4700 1 -- 172.21.15.1:0/2214197090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dec0c8f80 con 0x7f1dec09a3e0 2024-03-18T20:06:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.073+0000 7f1df40a4700 1 -- 172.21.15.1:0/2214197090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dec0676a0 con 0x7f1dec09a3e0 2024-03-18T20:06:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.074+0000 7f1de27fc700 1 -- 172.21.15.1:0/2214197090 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1de8003b80 con 0x7f1dec09a3e0 2024-03-18T20:06:40.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.074+0000 7f1de27fc700 1 --2- 172.21.15.1:0/2214197090 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1dd006adb0 0x7f1dd006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:40.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.074+0000 7f1de27fc700 1 -- 172.21.15.1:0/2214197090 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f1de8020070 con 0x7f1dec09a3e0 2024-03-18T20:06:40.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.074+0000 7f1df1e40700 1 --2- 172.21.15.1:0/2214197090 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1dd006adb0 0x7f1dd006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:40.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.075+0000 7f1df1e40700 1 --2- 172.21.15.1:0/2214197090 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1dd006adb0 0x7f1dd006d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc006fd0 tx=0x7f1ddc008040 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:40.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.079+0000 7f1de27fc700 1 -- 172.21.15.1:0/2214197090 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1de8055a80 con 0x7f1dec09a3e0 2024-03-18T20:06:40.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.281+0000 7f1df40a4700 1 -- 172.21.15.1:0/2214197090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1dec0ce910 con 0x7f1dec09a3e0 2024-03-18T20:06:40.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.914+0000 7f1de27fc700 1 -- 172.21.15.1:0/2214197090 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v179) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1de80150d0 con 0x7f1dec09a3e0 2024-03-18T20:06:40.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.919+0000 7f1df40a4700 1 -- 172.21.15.1:0/2214197090 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1dd006adb0 msgr2=0x7f1dd006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:40.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.919+0000 7f1df40a4700 1 --2- 172.21.15.1:0/2214197090 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1dd006adb0 0x7f1dd006d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc006fd0 tx=0x7f1ddc008040 comp rx=0 tx=0).stop 2024-03-18T20:06:40.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.919+0000 7f1df40a4700 1 -- 172.21.15.1:0/2214197090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec09a3e0 msgr2=0x7f1dec1490f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:40.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.919+0000 7f1df40a4700 1 --2- 172.21.15.1:0/2214197090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec09a3e0 0x7f1dec1490f0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f1de8000c00 tx=0x7f1de8004750 comp rx=0 tx=0).stop 2024-03-18T20:06:40.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.920+0000 7f1df40a4700 1 -- 172.21.15.1:0/2214197090 shutdown_connections 2024-03-18T20:06:40.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.920+0000 7f1df40a4700 1 --2- 172.21.15.1:0/2214197090 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1dd006adb0 0x7f1dd006d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:40.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.920+0000 7f1df40a4700 1 --2- 172.21.15.1:0/2214197090 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec15d5b0 0x7f1dec149630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:40.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.920+0000 7f1df40a4700 1 --2- 172.21.15.1:0/2214197090 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dec099a40 0x7f1dec0b5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:40.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.920+0000 7f1df40a4700 1 --2- 172.21.15.1:0/2214197090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec09a3e0 0x7f1dec1490f0 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:40.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.920+0000 7f1df40a4700 1 -- 172.21.15.1:0/2214197090 >> 172.21.15.1:0/2214197090 conn(0x7f1dec0b2bd0 msgr2=0x7f1dec06fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:40.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.920+0000 7f1df40a4700 1 -- 172.21.15.1:0/2214197090 shutdown_connections 2024-03-18T20:06:40.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:40.920+0000 7f1df40a4700 1 -- 172.21.15.1:0/2214197090 wait complete. 2024-03-18T20:06:40.938 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-18T20:06:40.948 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T20:06:41.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.078+0000 7fe8a1ca9700 1 Processor -- start 2024-03-18T20:06:41.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.078+0000 7fe8a1ca9700 1 -- start start 2024-03-18T20:06:41.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.079+0000 7fe8a1ca9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe89c0ace80 0x7fe89c0ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:41.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.079+0000 7fe8a1ca9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe89c0ad850 0x7fe89c15ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:41.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.079+0000 7fe8a1ca9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c15c340 con 0x7fe89c06a750 2024-03-18T20:06:41.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.079+0000 7fe8a1ca9700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89c070090 con 0x7fe89c0ace80 2024-03-18T20:06:41.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.079+0000 7fe8a1ca9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c15b7a0 con 0x7fe89c0ad850 2024-03-18T20:06:41.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.079+0000 7fe89b7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe89c06a750 0x7fe89c06ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42276/0 (socket says 172.21.15.1:42276) 2024-03-18T20:06:41.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.079+0000 7fe89b7fe700 1 -- 172.21.15.1:0/224744074 learned_addr learned my addr 172.21.15.1:0/224744074 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.079+0000 7fe89bfff700 1 --2- 172.21.15.1:0/224744074 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe89c0ad850 0x7fe89c15ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.079+0000 7fe89affd700 1 --2- 172.21.15.1:0/224744074 >> v2:172.21.15.165:3300/0 conn(0x7fe89c0ace80 0x7fe89c0ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.079+0000 7fe89a7fc700 1 -- 172.21.15.1:0/224744074 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015526042 0 0) 0x7fe89c15c340 con 0x7fe89c06a750 2024-03-18T20:06:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.079+0000 7fe89a7fc700 1 -- 172.21.15.1:0/224744074 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8840035b0 con 0x7fe89c06a750 2024-03-18T20:06:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.079+0000 7fe89bfff700 1 -- 172.21.15.1:0/224744074 >> v2:172.21.15.165:3300/0 conn(0x7fe89c0ace80 msgr2=0x7fe89c0ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.079+0000 7fe89bfff700 1 --2- 172.21.15.1:0/224744074 >> v2:172.21.15.165:3300/0 conn(0x7fe89c0ace80 0x7fe89c0ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.079+0000 7fe89a7fc700 1 -- 172.21.15.1:0/224744074 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3333833642 0 0) 0x7fe8840035b0 con 0x7fe89c06a750 2024-03-18T20:06:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.079+0000 7fe89bfff700 1 -- 172.21.15.1:0/224744074 >> v1:172.21.15.1:6789/0 conn(0x7fe89c06a750 legacy=0x7fe89c06ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.079+0000 7fe89bfff700 1 -- 172.21.15.1:0/224744074 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c15ba10 con 0x7fe89c0ad850 2024-03-18T20:06:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.079+0000 7fe89affd700 1 --2- 172.21.15.1:0/224744074 >> v2:172.21.15.165:3300/0 conn(0x7fe89c0ace80 0x7fe89c0ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.079+0000 7fe89bfff700 1 --2- 172.21.15.1:0/224744074 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe89c0ad850 0x7fe89c15ec80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe894009d90 tx=0x7fe89400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8225cb0bf5d5e4dd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.080+0000 7fe89a7fc700 1 -- 172.21.15.1:0/224744074 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89400e070 con 0x7fe89c0ad850 2024-03-18T20:06:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.080+0000 7fe89a7fc700 1 -- 172.21.15.1:0/224744074 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe894003a20 con 0x7fe89c0ad850 2024-03-18T20:06:41.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.080+0000 7fe89a7fc700 1 -- 172.21.15.1:0/224744074 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c0017f0 con 0x7fe89c0ad850 2024-03-18T20:06:41.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.080+0000 7fe8a1ca9700 1 -- 172.21.15.1:0/224744074 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe89c0ad850 msgr2=0x7fe89c15ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:41.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.080+0000 7fe8a1ca9700 1 --2- 172.21.15.1:0/224744074 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe89c0ad850 0x7fe89c15ec80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe894009d90 tx=0x7fe89400b330 comp rx=0 tx=0).stop 2024-03-18T20:06:41.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.080+0000 7fe8a1ca9700 1 -- 172.21.15.1:0/224744074 shutdown_connections 2024-03-18T20:06:41.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.080+0000 7fe8a1ca9700 1 --2- 172.21.15.1:0/224744074 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe89c0ad850 0x7fe89c15ec80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.080+0000 7fe8a1ca9700 1 --2- 172.21.15.1:0/224744074 >> v2:172.21.15.165:3300/0 conn(0x7fe89c0ace80 0x7fe89c0ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.080+0000 7fe8a1ca9700 1 -- 172.21.15.1:0/224744074 >> 172.21.15.1:0/224744074 conn(0x7fe89c0cc9b0 msgr2=0x7fe89c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:41.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.080+0000 7fe8a1ca9700 1 -- 172.21.15.1:0/224744074 shutdown_connections 2024-03-18T20:06:41.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.080+0000 7fe8a1ca9700 1 -- 172.21.15.1:0/224744074 wait complete. 2024-03-18T20:06:41.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.081+0000 7fe8a1ca9700 1 Processor -- start 2024-03-18T20:06:41.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.082+0000 7fe8a1ca9700 1 -- start start 2024-03-18T20:06:41.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.082+0000 7fe8a1ca9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe89c06a750 0x7fe89c0b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:41.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.082+0000 7fe8a1ca9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe89c0ace80 0x7fe89c1f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:41.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.082+0000 7fe8a1ca9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe89c0ad850 0x7fe89c149080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:41.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.082+0000 7fe8a1ca9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0b30d0 con 0x7fe89c0ad850 2024-03-18T20:06:41.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.082+0000 7fe8a1ca9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0cec90 con 0x7fe89c06a750 2024-03-18T20:06:41.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.082+0000 7fe8a1ca9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0b5c10 con 0x7fe89c0ace80 2024-03-18T20:06:41.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.083+0000 7fe89bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe89c0ad850 0x7fe89c149080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:41.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.083+0000 7fe89bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe89c0ad850 0x7fe89c149080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58126/0 (socket says 172.21.15.1:58126) 2024-03-18T20:06:41.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.083+0000 7fe89bfff700 1 -- 172.21.15.1:0/1863414262 learned_addr learned my addr 172.21.15.1:0/1863414262 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:41.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.083+0000 7fe89b7fe700 1 --2- 172.21.15.1:0/1863414262 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe89c06a750 0x7fe89c0b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:41.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.083+0000 7fe89affd700 1 --2- 172.21.15.1:0/1863414262 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe89c0ace80 0x7fe89c1f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:41.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.083+0000 7fe89bfff700 1 -- 172.21.15.1:0/1863414262 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe89c0ace80 msgr2=0x7fe89c1f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:41.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.083+0000 7fe89bfff700 1 --2- 172.21.15.1:0/1863414262 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe89c0ace80 0x7fe89c1f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.083+0000 7fe89bfff700 1 -- 172.21.15.1:0/1863414262 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe89c06a750 msgr2=0x7fe89c0b2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:41.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.083+0000 7fe89bfff700 1 --2- 172.21.15.1:0/1863414262 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe89c06a750 0x7fe89c0b2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.083+0000 7fe89bfff700 1 -- 172.21.15.1:0/1863414262 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8900094a0 con 0x7fe89c0ad850 2024-03-18T20:06:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.083+0000 7fe89bfff700 1 --2- 172.21.15.1:0/1863414262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe89c0ad850 0x7fe89c149080 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fe894004000 tx=0x7fe894004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.083+0000 7fe883fff700 1 -- 172.21.15.1:0/1863414262 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8940044c0 con 0x7fe89c0ad850 2024-03-18T20:06:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.083+0000 7fe883fff700 1 -- 172.21.15.1:0/1863414262 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe894004640 con 0x7fe89c0ad850 2024-03-18T20:06:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.083+0000 7fe8a1ca9700 1 -- 172.21.15.1:0/1863414262 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe894009a70 con 0x7fe89c0ad850 2024-03-18T20:06:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.083+0000 7fe883fff700 1 -- 172.21.15.1:0/1863414262 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89401b430 con 0x7fe89c0ad850 2024-03-18T20:06:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.083+0000 7fe8a1ca9700 1 -- 172.21.15.1:0/1863414262 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89c0ce640 con 0x7fe89c0ad850 2024-03-18T20:06:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.084+0000 7fe8a1ca9700 1 -- 172.21.15.1:0/1863414262 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89c0676a0 con 0x7fe89c0ad850 2024-03-18T20:06:41.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.085+0000 7fe883fff700 1 -- 172.21.15.1:0/1863414262 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe89401b590 con 0x7fe89c0ad850 2024-03-18T20:06:41.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.085+0000 7fe883fff700 1 --2- 172.21.15.1:0/1863414262 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe88806ade0 0x7fe88806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:41.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.085+0000 7fe883fff700 1 -- 172.21.15.1:0/1863414262 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fe89408afc0 con 0x7fe89c0ad850 2024-03-18T20:06:41.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.085+0000 7fe89b7fe700 1 --2- 172.21.15.1:0/1863414262 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe88806ade0 0x7fe88806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:41.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.086+0000 7fe89b7fe700 1 --2- 172.21.15.1:0/1863414262 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe88806ade0 0x7fe88806d2a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe88c000ef0 tx=0x7fe88c0012f0 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:41.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.090+0000 7fe883fff700 1 -- 172.21.15.1:0/1863414262 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe894055fc0 con 0x7fe89c0ad850 2024-03-18T20:06:41.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.337+0000 7fe8a1ca9700 1 -- 172.21.15.1:0/1863414262 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe89c0cedd0 con 0x7fe89c0ad850 2024-03-18T20:06:41.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.343+0000 7fe883fff700 1 -- 172.21.15.1:0/1863414262 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 21 and data pool 22 v77) v1 ==== 174+0+0 (secure 0 0 0) 0x7fe8940210c0 con 0x7fe89c0ad850 2024-03-18T20:06:41.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.347+0000 7fe8a1ca9700 1 -- 172.21.15.1:0/1863414262 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe88806ade0 msgr2=0x7fe88806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:41.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.347+0000 7fe8a1ca9700 1 --2- 172.21.15.1:0/1863414262 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe88806ade0 0x7fe88806d2a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe88c000ef0 tx=0x7fe88c0012f0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.347+0000 7fe8a1ca9700 1 -- 172.21.15.1:0/1863414262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe89c0ad850 msgr2=0x7fe89c149080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:41.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.347+0000 7fe8a1ca9700 1 --2- 172.21.15.1:0/1863414262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe89c0ad850 0x7fe89c149080 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fe894004000 tx=0x7fe894004ad0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.347+0000 7fe8a1ca9700 1 -- 172.21.15.1:0/1863414262 shutdown_connections 2024-03-18T20:06:41.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.347+0000 7fe8a1ca9700 1 --2- 172.21.15.1:0/1863414262 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe88806ade0 0x7fe88806d2a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.347+0000 7fe8a1ca9700 1 --2- 172.21.15.1:0/1863414262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe89c0ad850 0x7fe89c149080 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.347+0000 7fe8a1ca9700 1 --2- 172.21.15.1:0/1863414262 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe89c06a750 0x7fe89c0b2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.347+0000 7fe8a1ca9700 1 --2- 172.21.15.1:0/1863414262 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe89c0ace80 0x7fe89c1f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.347+0000 7fe8a1ca9700 1 -- 172.21.15.1:0/1863414262 >> 172.21.15.1:0/1863414262 conn(0x7fe89c0cc9b0 msgr2=0x7fe89c06ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:41.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.347+0000 7fe8a1ca9700 1 -- 172.21.15.1:0/1863414262 shutdown_connections 2024-03-18T20:06:41.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.347+0000 7fe8a1ca9700 1 -- 172.21.15.1:0/1863414262 wait complete. 2024-03-18T20:06:41.361 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 21 and data pool 22 2024-03-18T20:06:41.378 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:06:41.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.498+0000 7f609188d700 1 Processor -- start 2024-03-18T20:06:41.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.498+0000 7f609188d700 1 -- start start 2024-03-18T20:06:41.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.499+0000 7f609188d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f608c0cee90 0x7f608c0cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:41.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.499+0000 7f609188d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f608c0cfbd0 0x7f608c0b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:41.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.499+0000 7f609188d700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608c0cd0d0 con 0x7f608c0b6b60 2024-03-18T20:06:41.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.499+0000 7f609188d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f608c0d0060 con 0x7f608c0cfbd0 2024-03-18T20:06:41.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.499+0000 7f609188d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f608c0b3f00 con 0x7f608c0cee90 2024-03-18T20:06:41.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.499+0000 7f608b7fe700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f608c0cfbd0 0x7f608c0b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:41.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.499+0000 7f608a7fc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f608c0cee90 0x7f608c0cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:41.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.499+0000 7f608b7fe700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f608c0cfbd0 0x7f608c0b4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:33792/0 (socket says 172.21.15.1:33792) 2024-03-18T20:06:41.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.499+0000 7f608affd700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f608c0b6b60 0x7f608c0cc830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:48890/0 (socket says 172.21.15.1:48890) 2024-03-18T20:06:41.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.499+0000 7f608b7fe700 1 -- 172.21.15.1:0/3494772370 learned_addr learned my addr 172.21.15.1:0/3494772370 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:41.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.499+0000 7f608a7fc700 1 -- 172.21.15.1:0/3494772370 >> v2:172.21.15.165:3300/0 conn(0x7f608c0cfbd0 msgr2=0x7f608c0b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:41.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.499+0000 7f608a7fc700 1 --2- 172.21.15.1:0/3494772370 >> v2:172.21.15.165:3300/0 conn(0x7f608c0cfbd0 0x7f608c0b4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.499+0000 7f608a7fc700 1 -- 172.21.15.1:0/3494772370 >> v1:172.21.15.165:6789/0 conn(0x7f608c0b6b60 legacy=0x7f608c0cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:41.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.499+0000 7f608a7fc700 1 -- 172.21.15.1:0/3494772370 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f608c0b41e0 con 0x7f608c0cee90 2024-03-18T20:06:41.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.499+0000 7f608b7fe700 1 --2- 172.21.15.1:0/3494772370 >> v2:172.21.15.165:3300/0 conn(0x7f608c0cfbd0 0x7f608c0b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:41.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.500+0000 7f608a7fc700 1 --2- 172.21.15.1:0/3494772370 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f608c0cee90 0x7f608c0cf690 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6080009840 tx=0x7f6080009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b66efbac06e08c0a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:41.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.500+0000 7f6089ffb700 1 -- 172.21.15.1:0/3494772370 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6080011830 con 0x7f608c0cee90 2024-03-18T20:06:41.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.500+0000 7f6089ffb700 1 -- 172.21.15.1:0/3494772370 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6080011990 con 0x7f608c0cee90 2024-03-18T20:06:41.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.500+0000 7f6089ffb700 1 -- 172.21.15.1:0/3494772370 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6080011d20 con 0x7f608c0cee90 2024-03-18T20:06:41.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.500+0000 7f609188d700 1 -- 172.21.15.1:0/3494772370 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f608c0cee90 msgr2=0x7f608c0cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:41.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.500+0000 7f609188d700 1 --2- 172.21.15.1:0/3494772370 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f608c0cee90 0x7f608c0cf690 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6080009840 tx=0x7f6080009b50 comp rx=0 tx=0).stop 2024-03-18T20:06:41.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.500+0000 7f609188d700 1 -- 172.21.15.1:0/3494772370 shutdown_connections 2024-03-18T20:06:41.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.500+0000 7f609188d700 1 --2- 172.21.15.1:0/3494772370 >> v2:172.21.15.165:3300/0 conn(0x7f608c0cfbd0 0x7f608c0b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.500+0000 7f609188d700 1 --2- 172.21.15.1:0/3494772370 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f608c0cee90 0x7f608c0cf690 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.500+0000 7f609188d700 1 -- 172.21.15.1:0/3494772370 >> 172.21.15.1:0/3494772370 conn(0x7f608c0b2ba0 msgr2=0x7f608c0b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:41.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.501+0000 7f609188d700 1 -- 172.21.15.1:0/3494772370 shutdown_connections 2024-03-18T20:06:41.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.501+0000 7f609188d700 1 -- 172.21.15.1:0/3494772370 wait complete. 2024-03-18T20:06:41.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.502+0000 7f609188d700 1 Processor -- start 2024-03-18T20:06:41.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.502+0000 7f609188d700 1 -- start start 2024-03-18T20:06:41.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.503+0000 7f609188d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f608c0b6b60 0x7f608c153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:41.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.503+0000 7f609188d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f608c0cee90 0x7f608c0c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:41.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.503+0000 7f609188d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f608c0cfbd0 0x7f608c0b0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:41.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.503+0000 7f609188d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608c0b0120 con 0x7f608c0b6b60 2024-03-18T20:06:41.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.503+0000 7f609188d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608c154170 con 0x7f608c0cee90 2024-03-18T20:06:41.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.503+0000 7f609188d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f608c0c7e70 con 0x7f608c0cfbd0 2024-03-18T20:06:41.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.503+0000 7f608affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f608c0b6b60 0x7f608c153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:41.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.503+0000 7f608affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f608c0b6b60 0x7f608c153c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58128/0 (socket says 172.21.15.1:58128) 2024-03-18T20:06:41.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.503+0000 7f608affd700 1 -- 172.21.15.1:0/688249896 learned_addr learned my addr 172.21.15.1:0/688249896 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:41.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.504+0000 7f608a7fc700 1 --2- 172.21.15.1:0/688249896 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f608c0cee90 0x7f608c0c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:41.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.504+0000 7f608b7fe700 1 --2- 172.21.15.1:0/688249896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f608c0cfbd0 0x7f608c0b0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:41.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.504+0000 7f608affd700 1 -- 172.21.15.1:0/688249896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f608c0cfbd0 msgr2=0x7f608c0b0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:41.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.504+0000 7f608affd700 1 --2- 172.21.15.1:0/688249896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f608c0cfbd0 0x7f608c0b0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.504+0000 7f608affd700 1 -- 172.21.15.1:0/688249896 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f608c0cee90 msgr2=0x7f608c0c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:41.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.504+0000 7f608affd700 1 --2- 172.21.15.1:0/688249896 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f608c0cee90 0x7f608c0c7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.504+0000 7f608affd700 1 -- 172.21.15.1:0/688249896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6084009b30 con 0x7f608c0b6b60 2024-03-18T20:06:41.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.504+0000 7f608affd700 1 --2- 172.21.15.1:0/688249896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f608c0b6b60 0x7f608c153c30 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f607c00c520 tx=0x7f607c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:41.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.504+0000 7f6073fff700 1 -- 172.21.15.1:0/688249896 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c005cb0 con 0x7f608c0b6b60 2024-03-18T20:06:41.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.504+0000 7f6073fff700 1 -- 172.21.15.1:0/688249896 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607c005e10 con 0x7f608c0b6b60 2024-03-18T20:06:41.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.504+0000 7f609188d700 1 -- 172.21.15.1:0/688249896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60800094a0 con 0x7f608c0b6b60 2024-03-18T20:06:41.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.504+0000 7f6073fff700 1 -- 172.21.15.1:0/688249896 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c0072d0 con 0x7f608c0b6b60 2024-03-18T20:06:41.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.504+0000 7f609188d700 1 -- 172.21.15.1:0/688249896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f608c0af780 con 0x7f608c0b6b60 2024-03-18T20:06:41.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.505+0000 7f609188d700 1 -- 172.21.15.1:0/688249896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f608c0676a0 con 0x7f608c0b6b60 2024-03-18T20:06:41.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.505+0000 7f6073fff700 1 -- 172.21.15.1:0/688249896 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f607c007430 con 0x7f608c0b6b60 2024-03-18T20:06:41.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.506+0000 7f6073fff700 1 --2- 172.21.15.1:0/688249896 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f607806adb0 0x7f607806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:41.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.506+0000 7f6073fff700 1 -- 172.21.15.1:0/688249896 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f607c088980 con 0x7f608c0b6b60 2024-03-18T20:06:41.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.506+0000 7f608a7fc700 1 --2- 172.21.15.1:0/688249896 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f607806adb0 0x7f607806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:41.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.506+0000 7f608a7fc700 1 --2- 172.21.15.1:0/688249896 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f607806adb0 0x7f607806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f608c0b5ce0 tx=0x7f6080008030 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:41.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.511+0000 7f6073fff700 1 -- 172.21.15.1:0/688249896 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f607c053a00 con 0x7f608c0b6b60 2024-03-18T20:06:41.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.713+0000 7f609188d700 1 -- 172.21.15.1:0/688249896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f608c073f30 con 0x7f608c0b6b60 2024-03-18T20:06:41.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.714+0000 7f6073fff700 1 -- 172.21.15.1:0/688249896 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+16336 (secure 0 0 0) 0x7f607c017020 con 0x7f608c0b6b60 2024-03-18T20:06:41.722 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:41.722 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":180,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:06:41.338959+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:06:39.228206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-18T20:06:40.315627+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:06:41.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.716+0000 7f609188d700 1 -- 172.21.15.1:0/688249896 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f607806adb0 msgr2=0x7f607806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:41.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.716+0000 7f609188d700 1 --2- 172.21.15.1:0/688249896 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f607806adb0 0x7f607806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f608c0b5ce0 tx=0x7f6080008030 comp rx=0 tx=0).stop 2024-03-18T20:06:41.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.716+0000 7f609188d700 1 -- 172.21.15.1:0/688249896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f608c0b6b60 msgr2=0x7f608c153c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:41.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.716+0000 7f609188d700 1 --2- 172.21.15.1:0/688249896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f608c0b6b60 0x7f608c153c30 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f607c00c520 tx=0x7f607c00c8e0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.716+0000 7f609188d700 1 -- 172.21.15.1:0/688249896 shutdown_connections 2024-03-18T20:06:41.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.716+0000 7f609188d700 1 --2- 172.21.15.1:0/688249896 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f607806adb0 0x7f607806d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.716+0000 7f609188d700 1 --2- 172.21.15.1:0/688249896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f608c0cfbd0 0x7f608c0b0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.716+0000 7f609188d700 1 --2- 172.21.15.1:0/688249896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f608c0b6b60 0x7f608c153c30 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.716+0000 7f609188d700 1 --2- 172.21.15.1:0/688249896 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f608c0cee90 0x7f608c0c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.716+0000 7f609188d700 1 -- 172.21.15.1:0/688249896 >> 172.21.15.1:0/688249896 conn(0x7f608c0b2ba0 msgr2=0x7f608c06f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.717+0000 7f609188d700 1 -- 172.21.15.1:0/688249896 shutdown_connections 2024-03-18T20:06:41.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.717+0000 7f609188d700 1 -- 172.21.15.1:0/688249896 wait complete. 2024-03-18T20:06:41.734 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:06:41.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.856+0000 7f9b129fa700 1 Processor -- start 2024-03-18T20:06:41.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.856+0000 7f9b129fa700 1 -- start start 2024-03-18T20:06:41.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.857+0000 7f9b129fa700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9b0c06a750 0x7f9b0c06ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:41.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.857+0000 7f9b129fa700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b0c15d6b0 0x7f9b0c160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:41.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.857+0000 7f9b129fa700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0c0acfb0 con 0x7f9b0c0ad790 2024-03-18T20:06:41.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.857+0000 7f9b129fa700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0c0af7a0 con 0x7f9b0c06a750 2024-03-18T20:06:41.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.857+0000 7f9b129fa700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c06b070 con 0x7f9b0c15d6b0 2024-03-18T20:06:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.857+0000 7f9b0bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9b0c06a750 0x7f9b0c06ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.857+0000 7f9b10f97700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9b0c0ad790 0x7f9b0c161530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42282/0 (socket says 172.21.15.1:42282) 2024-03-18T20:06:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.857+0000 7f9b10f97700 1 -- 172.21.15.1:0/2723055659 learned_addr learned my addr 172.21.15.1:0/2723055659 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.857+0000 7f9b0b7fe700 1 --2- 172.21.15.1:0/2723055659 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b0c15d6b0 0x7f9b0c160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.857+0000 7f9b0bfff700 1 -- 172.21.15.1:0/2723055659 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b0c15d6b0 msgr2=0x7f9b0c160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.857+0000 7f9b0bfff700 1 --2- 172.21.15.1:0/2723055659 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b0c15d6b0 0x7f9b0c160ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.857+0000 7f9b0bfff700 1 -- 172.21.15.1:0/2723055659 >> v1:172.21.15.1:6789/0 conn(0x7f9b0c0ad790 legacy=0x7f9b0c161530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.857+0000 7f9b0bfff700 1 -- 172.21.15.1:0/2723055659 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0c0b50a0 con 0x7f9b0c06a750 2024-03-18T20:06:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.857+0000 7f9b0bfff700 1 --2- 172.21.15.1:0/2723055659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b0c06a750 0x7f9b0c06ab30 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f9afc009840 tx=0x7f9afc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a04b684cec2dbba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.858+0000 7f9b0affd700 1 -- 172.21.15.1:0/2723055659 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc011830 con 0x7f9b0c06a750 2024-03-18T20:06:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.858+0000 7f9b0affd700 1 -- 172.21.15.1:0/2723055659 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9afc011990 con 0x7f9b0c06a750 2024-03-18T20:06:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.858+0000 7f9b0affd700 1 -- 172.21.15.1:0/2723055659 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc011d20 con 0x7f9b0c06a750 2024-03-18T20:06:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.858+0000 7f9b129fa700 1 -- 172.21.15.1:0/2723055659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b0c06a750 msgr2=0x7f9b0c06ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:41.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.858+0000 7f9b129fa700 1 --2- 172.21.15.1:0/2723055659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b0c06a750 0x7f9b0c06ab30 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f9afc009840 tx=0x7f9afc009b50 comp rx=0 tx=0).stop 2024-03-18T20:06:41.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.858+0000 7f9b129fa700 1 -- 172.21.15.1:0/2723055659 shutdown_connections 2024-03-18T20:06:41.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.858+0000 7f9b129fa700 1 --2- 172.21.15.1:0/2723055659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b0c06a750 0x7f9b0c06ab30 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.858+0000 7f9b129fa700 1 --2- 172.21.15.1:0/2723055659 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b0c15d6b0 0x7f9b0c160ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.858+0000 7f9b129fa700 1 -- 172.21.15.1:0/2723055659 >> 172.21.15.1:0/2723055659 conn(0x7f9b0c06b860 msgr2=0x7f9b0c06bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:41.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.858+0000 7f9b129fa700 1 -- 172.21.15.1:0/2723055659 shutdown_connections 2024-03-18T20:06:41.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.858+0000 7f9b129fa700 1 -- 172.21.15.1:0/2723055659 wait complete. 2024-03-18T20:06:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.859+0000 7f9b129fa700 1 Processor -- start 2024-03-18T20:06:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.860+0000 7f9b129fa700 1 -- start start 2024-03-18T20:06:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.860+0000 7f9b129fa700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b0c06a750 0x7f9b0c14ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.860+0000 7f9b129fa700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b0c0ad790 0x7f9b0c06fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.860+0000 7f9b129fa700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b0c15d6b0 0x7f9b0c15fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.860+0000 7f9b129fa700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c15dd20 con 0x7f9b0c06a750 2024-03-18T20:06:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.860+0000 7f9b129fa700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c1605a0 con 0x7f9b0c15d6b0 2024-03-18T20:06:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.860+0000 7f9b129fa700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c151280 con 0x7f9b0c0ad790 2024-03-18T20:06:41.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.861+0000 7f9b0bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b0c06a750 0x7f9b0c14ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:41.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.861+0000 7f9b0bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b0c06a750 0x7f9b0c14ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58144/0 (socket says 172.21.15.1:58144) 2024-03-18T20:06:41.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.861+0000 7f9b0bfff700 1 -- 172.21.15.1:0/659036032 learned_addr learned my addr 172.21.15.1:0/659036032 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:41.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.861+0000 7f9b0bfff700 1 -- 172.21.15.1:0/659036032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b0c0ad790 msgr2=0x7f9b0c06fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:41.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.861+0000 7f9b0bfff700 1 --2- 172.21.15.1:0/659036032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b0c0ad790 0x7f9b0c06fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.861+0000 7f9b0bfff700 1 -- 172.21.15.1:0/659036032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b0c15d6b0 msgr2=0x7f9b0c15fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:41.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.861+0000 7f9b0bfff700 1 --2- 172.21.15.1:0/659036032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b0c15d6b0 0x7f9b0c15fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:41.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.861+0000 7f9b0bfff700 1 -- 172.21.15.1:0/659036032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9afc0094a0 con 0x7f9b0c06a750 2024-03-18T20:06:41.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.861+0000 7f9b0bfff700 1 --2- 172.21.15.1:0/659036032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b0c06a750 0x7f9b0c14ef00 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f9afc0090b0 tx=0x7f9afc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:41.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.861+0000 7f9b08ff9700 1 -- 172.21.15.1:0/659036032 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc0036a0 con 0x7f9b0c06a750 2024-03-18T20:06:41.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.861+0000 7f9b08ff9700 1 -- 172.21.15.1:0/659036032 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9afc01f070 con 0x7f9b0c06a750 2024-03-18T20:06:41.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.861+0000 7f9b129fa700 1 -- 172.21.15.1:0/659036032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b0c1607f0 con 0x7f9b0c06a750 2024-03-18T20:06:41.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.861+0000 7f9b08ff9700 1 -- 172.21.15.1:0/659036032 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc003a20 con 0x7f9b0c06a750 2024-03-18T20:06:41.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.861+0000 7f9b129fa700 1 -- 172.21.15.1:0/659036032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b0c150fa0 con 0x7f9b0c06a750 2024-03-18T20:06:41.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.862+0000 7f9b129fa700 1 -- 172.21.15.1:0/659036032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b0c0ad420 con 0x7f9b0c06a750 2024-03-18T20:06:41.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.863+0000 7f9b08ff9700 1 -- 172.21.15.1:0/659036032 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9afc003b80 con 0x7f9b0c06a750 2024-03-18T20:06:41.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.863+0000 7f9b08ff9700 1 --2- 172.21.15.1:0/659036032 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9aec06adb0 0x7f9aec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:41.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.863+0000 7f9b08ff9700 1 -- 172.21.15.1:0/659036032 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9afc021070 con 0x7f9b0c06a750 2024-03-18T20:06:41.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.863+0000 7f9b0b7fe700 1 --2- 172.21.15.1:0/659036032 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9aec06adb0 0x7f9aec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:41.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.863+0000 7f9b0b7fe700 1 --2- 172.21.15.1:0/659036032 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9aec06adb0 0x7f9aec06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c1f6d70 tx=0x7f9af4008040 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:41.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:41.867+0000 7f9b08ff9700 1 -- 172.21.15.1:0/659036032 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9b0c0ad420 con 0x7f9b0c06a750 2024-03-18T20:06:42.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.075+0000 7f9b129fa700 1 -- 172.21.15.1:0/659036032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b0c15de60 con 0x7f9b0c06a750 2024-03-18T20:06:42.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.076+0000 7f9b08ff9700 1 -- 172.21.15.1:0/659036032 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+16336 (secure 0 0 0) 0x7f9b0c15de60 con 0x7f9b0c06a750 2024-03-18T20:06:42.081 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:42.081 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":180,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:06:41.338959+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:06:39.228206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-18T20:06:40.315627+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:06:42.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.078+0000 7f9b129fa700 1 -- 172.21.15.1:0/659036032 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9aec06adb0 msgr2=0x7f9aec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:42.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.078+0000 7f9b129fa700 1 --2- 172.21.15.1:0/659036032 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9aec06adb0 0x7f9aec06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c1f6d70 tx=0x7f9af4008040 comp rx=0 tx=0).stop 2024-03-18T20:06:42.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.078+0000 7f9b129fa700 1 -- 172.21.15.1:0/659036032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b0c06a750 msgr2=0x7f9b0c14ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:42.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.078+0000 7f9b129fa700 1 --2- 172.21.15.1:0/659036032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b0c06a750 0x7f9b0c14ef00 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f9afc0090b0 tx=0x7f9afc004750 comp rx=0 tx=0).stop 2024-03-18T20:06:42.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.078+0000 7f9b129fa700 1 -- 172.21.15.1:0/659036032 shutdown_connections 2024-03-18T20:06:42.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.078+0000 7f9b129fa700 1 --2- 172.21.15.1:0/659036032 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f9aec06adb0 0x7f9aec06d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:42.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.078+0000 7f9b129fa700 1 --2- 172.21.15.1:0/659036032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b0c15d6b0 0x7f9b0c15fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:42.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.078+0000 7f9b129fa700 1 --2- 172.21.15.1:0/659036032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9b0c06a750 0x7f9b0c14ef00 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:42.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.078+0000 7f9b129fa700 1 --2- 172.21.15.1:0/659036032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9b0c0ad790 0x7f9b0c06fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:42.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.078+0000 7f9b129fa700 1 -- 172.21.15.1:0/659036032 >> 172.21.15.1:0/659036032 conn(0x7f9b0c06b860 msgr2=0x7f9b0c14fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:42.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.079+0000 7f9b129fa700 1 -- 172.21.15.1:0/659036032 shutdown_connections 2024-03-18T20:06:42.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.079+0000 7f9b129fa700 1 -- 172.21.15.1:0/659036032 wait complete. 2024-03-18T20:06:42.096 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T20:06:42.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.219+0000 7f4cda3fd700 1 Processor -- start 2024-03-18T20:06:42.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.219+0000 7f4cda3fd700 1 -- start start 2024-03-18T20:06:42.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.220+0000 7f4cda3fd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4cd4073a60 0x7f4cd40ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:42.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.220+0000 7f4cda3fd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4cd406d300 0x7f4cd40acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:42.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.220+0000 7f4cda3fd700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd406e160 con 0x7f4cd415d340 2024-03-18T20:06:42.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.220+0000 7f4cda3fd700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd4161150 con 0x7f4cd4073a60 2024-03-18T20:06:42.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.220+0000 7f4cda3fd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd406d8c0 con 0x7f4cd406d300 2024-03-18T20:06:42.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.220+0000 7f4cd37fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4cd4073a60 0x7f4cd40ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:42.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.220+0000 7f4cd3fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4cd415d340 0x7f4cd41615a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42290/0 (socket says 172.21.15.1:42290) 2024-03-18T20:06:42.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.220+0000 7f4cd3fff700 1 -- 172.21.15.1:0/3926176175 learned_addr learned my addr 172.21.15.1:0/3926176175 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:42.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.221+0000 7f4cd37fe700 1 -- 172.21.15.1:0/3926176175 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4cd406d300 msgr2=0x7f4cd40acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:42.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.221+0000 7f4cd37fe700 1 --2- 172.21.15.1:0/3926176175 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4cd406d300 0x7f4cd40acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:42.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.221+0000 7f4cd37fe700 1 -- 172.21.15.1:0/3926176175 >> v1:172.21.15.1:6789/0 conn(0x7f4cd415d340 legacy=0x7f4cd41615a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:42.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.221+0000 7f4cd37fe700 1 -- 172.21.15.1:0/3926176175 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd406dba0 con 0x7f4cd4073a60 2024-03-18T20:06:42.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.221+0000 7f4cd2ffd700 1 -- 172.21.15.1:0/3926176175 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717194602 0 0) 0x7f4cd406e160 con 0x7f4cd415d340 2024-03-18T20:06:42.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.221+0000 7f4cd37fe700 1 --2- 172.21.15.1:0/3926176175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cd4073a60 0x7f4cd40ac8b0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8009840 tx=0x7f4cc8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70cd8c41ca320d08 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:42.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.221+0000 7f4cd2ffd700 1 -- 172.21.15.1:0/3926176175 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8011830 con 0x7f4cd4073a60 2024-03-18T20:06:42.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.221+0000 7f4cd2ffd700 1 -- 172.21.15.1:0/3926176175 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc8011990 con 0x7f4cd4073a60 2024-03-18T20:06:42.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.221+0000 7f4cd2ffd700 1 -- 172.21.15.1:0/3926176175 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8011d20 con 0x7f4cd4073a60 2024-03-18T20:06:42.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.221+0000 7f4cda3fd700 1 -- 172.21.15.1:0/3926176175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cd4073a60 msgr2=0x7f4cd40ac8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:42.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.221+0000 7f4cda3fd700 1 --2- 172.21.15.1:0/3926176175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cd4073a60 0x7f4cd40ac8b0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8009840 tx=0x7f4cc8009b50 comp rx=0 tx=0).stop 2024-03-18T20:06:42.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.221+0000 7f4cda3fd700 1 -- 172.21.15.1:0/3926176175 shutdown_connections 2024-03-18T20:06:42.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.221+0000 7f4cda3fd700 1 --2- 172.21.15.1:0/3926176175 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4cd406d300 0x7f4cd40acdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:42.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.221+0000 7f4cda3fd700 1 --2- 172.21.15.1:0/3926176175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cd4073a60 0x7f4cd40ac8b0 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:42.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.221+0000 7f4cda3fd700 1 -- 172.21.15.1:0/3926176175 >> 172.21.15.1:0/3926176175 conn(0x7f4cd4066b70 msgr2=0x7f4cd406a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:42.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.222+0000 7f4cda3fd700 1 -- 172.21.15.1:0/3926176175 shutdown_connections 2024-03-18T20:06:42.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.222+0000 7f4cda3fd700 1 -- 172.21.15.1:0/3926176175 wait complete. 2024-03-18T20:06:42.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.223+0000 7f4cda3fd700 1 Processor -- start 2024-03-18T20:06:42.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.223+0000 7f4cda3fd700 1 -- start start 2024-03-18T20:06:42.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cda3fd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4cd406d300 0x7f4cd415ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:42.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cda3fd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cd4073a60 0x7f4cd414e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:42.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cda3fd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cd415d340 0x7f4cd4153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:42.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cda3fd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd414e890 con 0x7f4cd4073a60 2024-03-18T20:06:42.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cda3fd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd41539a0 con 0x7f4cd415d340 2024-03-18T20:06:42.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cda3fd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd4153c80 con 0x7f4cd406d300 2024-03-18T20:06:42.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cd37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cd4073a60 0x7f4cd414e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:42.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cd37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cd4073a60 0x7f4cd414e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58164/0 (socket says 172.21.15.1:58164) 2024-03-18T20:06:42.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cd37fe700 1 -- 172.21.15.1:0/2564369323 learned_addr learned my addr 172.21.15.1:0/2564369323 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:42.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cd3fff700 1 --2- 172.21.15.1:0/2564369323 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4cd406d300 0x7f4cd415ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:42.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cd37fe700 1 -- 172.21.15.1:0/2564369323 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4cd406d300 msgr2=0x7f4cd415ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:42.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cd899a700 1 --2- 172.21.15.1:0/2564369323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cd415d340 0x7f4cd4153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:42.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cd37fe700 1 --2- 172.21.15.1:0/2564369323 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4cd406d300 0x7f4cd415ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:42.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cd37fe700 1 -- 172.21.15.1:0/2564369323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cd415d340 msgr2=0x7f4cd4153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:42.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cd37fe700 1 --2- 172.21.15.1:0/2564369323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cd415d340 0x7f4cd4153150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:42.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cd37fe700 1 -- 172.21.15.1:0/2564369323 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc80094a0 con 0x7f4cd4073a60 2024-03-18T20:06:42.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cd899a700 1 --2- 172.21.15.1:0/2564369323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cd415d340 0x7f4cd4153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:42.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cd37fe700 1 --2- 172.21.15.1:0/2564369323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cd4073a60 0x7f4cd414e350 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8014040 tx=0x7f4cc8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:42.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cd0ff9700 1 -- 172.21.15.1:0/2564369323 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc80045f0 con 0x7f4cd4073a60 2024-03-18T20:06:42.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cd0ff9700 1 -- 172.21.15.1:0/2564369323 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc801e070 con 0x7f4cd4073a60 2024-03-18T20:06:42.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cda3fd700 1 -- 172.21.15.1:0/2564369323 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd4153f60 con 0x7f4cd4073a60 2024-03-18T20:06:42.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cd0ff9700 1 -- 172.21.15.1:0/2564369323 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8003b00 con 0x7f4cd4073a60 2024-03-18T20:06:42.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.224+0000 7f4cda3fd700 1 -- 172.21.15.1:0/2564369323 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd4154520 con 0x7f4cd4073a60 2024-03-18T20:06:42.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.225+0000 7f4cda3fd700 1 -- 172.21.15.1:0/2564369323 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd4154710 con 0x7f4cd4073a60 2024-03-18T20:06:42.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.226+0000 7f4cd0ff9700 1 -- 172.21.15.1:0/2564369323 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4cc8003c60 con 0x7f4cd4073a60 2024-03-18T20:06:42.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.226+0000 7f4cd0ff9700 1 --2- 172.21.15.1:0/2564369323 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4cc006adb0 0x7f4cc006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:42.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.226+0000 7f4cd0ff9700 1 -- 172.21.15.1:0/2564369323 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4cc808a780 con 0x7f4cd4073a60 2024-03-18T20:06:42.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.226+0000 7f4cd3fff700 1 --2- 172.21.15.1:0/2564369323 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4cc006adb0 0x7f4cc006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:42.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.227+0000 7f4cd3fff700 1 --2- 172.21.15.1:0/2564369323 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4cc006adb0 0x7f4cc006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4000f60 tx=0x7f4cc4007c90 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:42.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.231+0000 7f4cd0ff9700 1 -- 172.21.15.1:0/2564369323 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4cc8055750 con 0x7f4cd4073a60 2024-03-18T20:06:42.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:42.485+0000 7f4cda3fd700 1 -- 172.21.15.1:0/2564369323 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4cd406e9f0 con 0x7f4cd4073a60 2024-03-18T20:06:42.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-18T20:06:42.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-03-18T20:06:43.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.349+0000 7f4cd0ff9700 1 -- 172.21.15.1:0/2564369323 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v80) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4cc80200b0 con 0x7f4cd4073a60 2024-03-18T20:06:43.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.352+0000 7f4cda3fd700 1 -- 172.21.15.1:0/2564369323 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4cc006adb0 msgr2=0x7f4cc006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:43.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.352+0000 7f4cda3fd700 1 --2- 172.21.15.1:0/2564369323 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4cc006adb0 0x7f4cc006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4000f60 tx=0x7f4cc4007c90 comp rx=0 tx=0).stop 2024-03-18T20:06:43.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.352+0000 7f4cda3fd700 1 -- 172.21.15.1:0/2564369323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cd4073a60 msgr2=0x7f4cd414e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:43.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.352+0000 7f4cda3fd700 1 --2- 172.21.15.1:0/2564369323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cd4073a60 0x7f4cd414e350 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8014040 tx=0x7f4cc8004750 comp rx=0 tx=0).stop 2024-03-18T20:06:43.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.352+0000 7f4cda3fd700 1 -- 172.21.15.1:0/2564369323 shutdown_connections 2024-03-18T20:06:43.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.352+0000 7f4cda3fd700 1 --2- 172.21.15.1:0/2564369323 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4cc006adb0 0x7f4cc006d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:43.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.352+0000 7f4cda3fd700 1 --2- 172.21.15.1:0/2564369323 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4cd406d300 0x7f4cd415ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:43.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.352+0000 7f4cda3fd700 1 --2- 172.21.15.1:0/2564369323 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4cd415d340 0x7f4cd4153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:43.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.352+0000 7f4cda3fd700 1 --2- 172.21.15.1:0/2564369323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4cd4073a60 0x7f4cd414e350 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:43.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.352+0000 7f4cda3fd700 1 -- 172.21.15.1:0/2564369323 >> 172.21.15.1:0/2564369323 conn(0x7f4cd4066b70 msgr2=0x7f4cd406a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:43.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.352+0000 7f4cda3fd700 1 -- 172.21.15.1:0/2564369323 shutdown_connections 2024-03-18T20:06:43.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.352+0000 7f4cda3fd700 1 -- 172.21.15.1:0/2564369323 wait complete. 2024-03-18T20:06:43.373 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-18T20:06:43.373 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T20:06:43.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.504+0000 7faafddb0700 1 Processor -- start 2024-03-18T20:06:43.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.504+0000 7faafddb0700 1 -- start start 2024-03-18T20:06:43.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.504+0000 7faafddb0700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7faaf80d0cd0 0x7faaf80cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:43.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.504+0000 7faafddb0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faaf809a530 0x7faaf80cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:43.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.504+0000 7faafddb0700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf80d0730 con 0x7faaf8099a40 2024-03-18T20:06:43.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.504+0000 7faafddb0700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf806bdf0 con 0x7faaf809a530 2024-03-18T20:06:43.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.504+0000 7faafddb0700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf815d9f0 con 0x7faaf80d0cd0 2024-03-18T20:06:43.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.505+0000 7faaf7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faaf809a530 0x7faaf80cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:43.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.505+0000 7faaf7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faaf809a530 0x7faaf80cc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58168/0 (socket says 172.21.15.1:58168) 2024-03-18T20:06:43.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.505+0000 7faaf7fff700 1 -- 172.21.15.1:0/1091985546 learned_addr learned my addr 172.21.15.1:0/1091985546 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:43.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.505+0000 7faaf6ffd700 1 --2- 172.21.15.1:0/1091985546 >> v2:172.21.15.165:3300/0 conn(0x7faaf80d0cd0 0x7faaf80cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:43.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.505+0000 7faaf7fff700 1 -- 172.21.15.1:0/1091985546 >> v2:172.21.15.165:3300/0 conn(0x7faaf80d0cd0 msgr2=0x7faaf80cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:43.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.505+0000 7faaf7fff700 1 --2- 172.21.15.1:0/1091985546 >> v2:172.21.15.165:3300/0 conn(0x7faaf80d0cd0 0x7faaf80cc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:43.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.505+0000 7faaf7fff700 1 -- 172.21.15.1:0/1091985546 >> v1:172.21.15.165:6789/0 conn(0x7faaf8099a40 legacy=0x7faaf8099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:43.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.505+0000 7faaf7fff700 1 -- 172.21.15.1:0/1091985546 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf806bb10 con 0x7faaf809a530 2024-03-18T20:06:43.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.505+0000 7faaf7fff700 1 --2- 172.21.15.1:0/1091985546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaf809a530 0x7faaf80cc710 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7faaf8071e50 tx=0x7faaf000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79f6cdc10ca30223 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:43.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.505+0000 7faaf67fc700 1 -- 172.21.15.1:0/1091985546 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf000e070 con 0x7faaf809a530 2024-03-18T20:06:43.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.505+0000 7faaf67fc700 1 -- 172.21.15.1:0/1091985546 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf0003a20 con 0x7faaf809a530 2024-03-18T20:06:43.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.505+0000 7faaf67fc700 1 -- 172.21.15.1:0/1091985546 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf0003dd0 con 0x7faaf809a530 2024-03-18T20:06:43.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.506+0000 7faafddb0700 1 -- 172.21.15.1:0/1091985546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaf809a530 msgr2=0x7faaf80cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:43.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.506+0000 7faafddb0700 1 --2- 172.21.15.1:0/1091985546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaf809a530 0x7faaf80cc710 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7faaf8071e50 tx=0x7faaf000b330 comp rx=0 tx=0).stop 2024-03-18T20:06:43.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.506+0000 7faafddb0700 1 -- 172.21.15.1:0/1091985546 shutdown_connections 2024-03-18T20:06:43.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.506+0000 7faafddb0700 1 --2- 172.21.15.1:0/1091985546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaf809a530 0x7faaf80cc710 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:43.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.506+0000 7faafddb0700 1 --2- 172.21.15.1:0/1091985546 >> v2:172.21.15.165:3300/0 conn(0x7faaf80d0cd0 0x7faaf80cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:43.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.506+0000 7faafddb0700 1 -- 172.21.15.1:0/1091985546 >> 172.21.15.1:0/1091985546 conn(0x7faaf80b2bf0 msgr2=0x7faaf80b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:43.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.506+0000 7faafddb0700 1 -- 172.21.15.1:0/1091985546 shutdown_connections 2024-03-18T20:06:43.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.506+0000 7faafddb0700 1 -- 172.21.15.1:0/1091985546 wait complete. 2024-03-18T20:06:43.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.507+0000 7faafddb0700 1 Processor -- start 2024-03-18T20:06:43.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.507+0000 7faafddb0700 1 -- start start 2024-03-18T20:06:43.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.508+0000 7faafddb0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaf8099a40 0x7faaf8160480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:43.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.508+0000 7faafddb0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faaf809a530 0x7faaf80aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:43.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.508+0000 7faafddb0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaf80d0cd0 0x7faaf80af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:43.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.508+0000 7faafddb0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf814a010 con 0x7faaf80d0cd0 2024-03-18T20:06:43.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.508+0000 7faafddb0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf8149d30 con 0x7faaf8099a40 2024-03-18T20:06:43.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.508+0000 7faafddb0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf80c8370 con 0x7faaf809a530 2024-03-18T20:06:43.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.508+0000 7faaf7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaf80d0cd0 0x7faaf80af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:43.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.508+0000 7faaf7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaf80d0cd0 0x7faaf80af320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58170/0 (socket says 172.21.15.1:58170) 2024-03-18T20:06:43.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.508+0000 7faaf7fff700 1 -- 172.21.15.1:0/2715947178 learned_addr learned my addr 172.21.15.1:0/2715947178 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:43.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.508+0000 7faaf77fe700 1 --2- 172.21.15.1:0/2715947178 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaf8099a40 0x7faaf8160480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:43.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.508+0000 7faaf6ffd700 1 --2- 172.21.15.1:0/2715947178 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faaf809a530 0x7faaf80aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:43.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.508+0000 7faaf7fff700 1 -- 172.21.15.1:0/2715947178 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faaf809a530 msgr2=0x7faaf80aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:43.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.508+0000 7faaf7fff700 1 --2- 172.21.15.1:0/2715947178 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faaf809a530 0x7faaf80aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:43.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.508+0000 7faaf7fff700 1 -- 172.21.15.1:0/2715947178 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaf8099a40 msgr2=0x7faaf8160480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:43.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.508+0000 7faaf7fff700 1 --2- 172.21.15.1:0/2715947178 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaf8099a40 0x7faaf8160480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.508+0000 7faaf7fff700 1 -- 172.21.15.1:0/2715947178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf0009b30 con 0x7faaf80d0cd0 2024-03-18T20:06:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.508+0000 7faaf7fff700 1 --2- 172.21.15.1:0/2715947178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaf80d0cd0 0x7faaf80af320 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7faaf0004000 tx=0x7faaf0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.509+0000 7faadffff700 1 -- 172.21.15.1:0/2715947178 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf0004440 con 0x7faaf80d0cd0 2024-03-18T20:06:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.509+0000 7faadffff700 1 -- 172.21.15.1:0/2715947178 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf00045c0 con 0x7faaf80d0cd0 2024-03-18T20:06:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.509+0000 7faafddb0700 1 -- 172.21.15.1:0/2715947178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf80c8650 con 0x7faaf80d0cd0 2024-03-18T20:06:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.509+0000 7faadffff700 1 -- 172.21.15.1:0/2715947178 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf0004970 con 0x7faaf80d0cd0 2024-03-18T20:06:43.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.509+0000 7faafddb0700 1 -- 172.21.15.1:0/2715947178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf81609c0 con 0x7faaf80d0cd0 2024-03-18T20:06:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.510+0000 7faafddb0700 1 -- 172.21.15.1:0/2715947178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf80c81d0 con 0x7faaf80d0cd0 2024-03-18T20:06:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.510+0000 7faadffff700 1 -- 172.21.15.1:0/2715947178 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7faaf0012430 con 0x7faaf80d0cd0 2024-03-18T20:06:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.511+0000 7faadffff700 1 --2- 172.21.15.1:0/2715947178 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faae406adb0 0x7faae406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.511+0000 7faadffff700 1 -- 172.21.15.1:0/2715947178 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6101+0+0 (secure 0 0 0) 0x7faaf008aa30 con 0x7faaf80d0cd0 2024-03-18T20:06:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.511+0000 7faaf77fe700 1 --2- 172.21.15.1:0/2715947178 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faae406adb0 0x7faae406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.511+0000 7faaf77fe700 1 --2- 172.21.15.1:0/2715947178 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faae406adb0 0x7faae406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faae8009960 tx=0x7faae80090e0 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:43.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.515+0000 7faadffff700 1 -- 172.21.15.1:0/2715947178 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faaf80c81d0 con 0x7faaf80d0cd0 2024-03-18T20:06:43.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:43.762+0000 7faafddb0700 1 -- 172.21.15.1:0/2715947178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7faaf80c81d0 con 0x7faaf80d0cd0 2024-03-18T20:06:44.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.357+0000 7faadffff700 1 -- 172.21.15.1:0/2715947178 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v81) v1 ==== 122+0+0 (secure 0 0 0) 0x7faaf80c81d0 con 0x7faaf80d0cd0 2024-03-18T20:06:44.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.360+0000 7faafddb0700 1 -- 172.21.15.1:0/2715947178 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faae406adb0 msgr2=0x7faae406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:44.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.360+0000 7faafddb0700 1 --2- 172.21.15.1:0/2715947178 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faae406adb0 0x7faae406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faae8009960 tx=0x7faae80090e0 comp rx=0 tx=0).stop 2024-03-18T20:06:44.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.360+0000 7faafddb0700 1 -- 172.21.15.1:0/2715947178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaf80d0cd0 msgr2=0x7faaf80af320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:44.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.360+0000 7faafddb0700 1 --2- 172.21.15.1:0/2715947178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaf80d0cd0 0x7faaf80af320 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7faaf0004000 tx=0x7faaf0004ad0 comp rx=0 tx=0).stop 2024-03-18T20:06:44.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.360+0000 7faafddb0700 1 -- 172.21.15.1:0/2715947178 shutdown_connections 2024-03-18T20:06:44.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.360+0000 7faafddb0700 1 --2- 172.21.15.1:0/2715947178 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faae406adb0 0x7faae406d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:44.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.360+0000 7faafddb0700 1 --2- 172.21.15.1:0/2715947178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaf80d0cd0 0x7faaf80af320 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:44.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.360+0000 7faafddb0700 1 --2- 172.21.15.1:0/2715947178 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaf8099a40 0x7faaf8160480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:44.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.360+0000 7faafddb0700 1 --2- 172.21.15.1:0/2715947178 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faaf809a530 0x7faaf80aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:44.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.360+0000 7faafddb0700 1 -- 172.21.15.1:0/2715947178 >> 172.21.15.1:0/2715947178 conn(0x7faaf80b2bf0 msgr2=0x7faaf80b2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:44.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.360+0000 7faafddb0700 1 -- 172.21.15.1:0/2715947178 shutdown_connections 2024-03-18T20:06:44.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.360+0000 7faafddb0700 1 -- 172.21.15.1:0/2715947178 wait complete. 2024-03-18T20:06:44.380 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T20:06:44.380 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:06:44.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.508+0000 7f85e1f83700 1 Processor -- start 2024-03-18T20:06:44.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.508+0000 7f85e1f83700 1 -- start start 2024-03-18T20:06:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.508+0000 7f85e1f83700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f85dc0cc600 0x7f85dc06dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.508+0000 7f85e1f83700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85dc06e9c0 con 0x7f85dc15cfc0 2024-03-18T20:06:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.508+0000 7f85e1f83700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85dc0cd640 con 0x7f85dc0ca110 2024-03-18T20:06:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.508+0000 7f85e1f83700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85dc06e1f0 con 0x7f85dc0cc600 2024-03-18T20:06:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.509+0000 7f85daffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f85dc0cc600 0x7f85dc06dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.509+0000 7f85db7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f85dc15cfc0 0x7f85dc0cbef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49160/0 (socket says 172.21.15.1:49160) 2024-03-18T20:06:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.509+0000 7f85db7fe700 1 -- 172.21.15.1:0/491751193 learned_addr learned my addr 172.21.15.1:0/491751193 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.509+0000 7f85daffd700 1 -- 172.21.15.1:0/491751193 >> v1:172.21.15.165:6789/0 conn(0x7f85dc0ca110 legacy=0x7f85dc0ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.509+0000 7f85daffd700 1 -- 172.21.15.1:0/491751193 >> v1:172.21.15.1:6789/0 conn(0x7f85dc15cfc0 legacy=0x7f85dc0cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:44.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.509+0000 7f85da7fc700 1 -- 172.21.15.1:0/491751193 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277032367 0 0) 0x7f85dc06e9c0 con 0x7f85dc15cfc0 2024-03-18T20:06:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.509+0000 7f85daffd700 1 -- 172.21.15.1:0/491751193 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc0cd920 con 0x7f85dc0cc600 2024-03-18T20:06:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.509+0000 7f85daffd700 1 --2- 172.21.15.1:0/491751193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85dc0cc600 0x7f85dc06dcb0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f85d000a9e0 tx=0x7f85d000acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf850e4112ecc85 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.509+0000 7f85da7fc700 1 -- 172.21.15.1:0/491751193 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d000cb30 con 0x7f85dc0cc600 2024-03-18T20:06:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.509+0000 7f85da7fc700 1 -- 172.21.15.1:0/491751193 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d000cc90 con 0x7f85dc0cc600 2024-03-18T20:06:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.509+0000 7f85da7fc700 1 -- 172.21.15.1:0/491751193 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d0004030 con 0x7f85dc0cc600 2024-03-18T20:06:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.510+0000 7f85e1f83700 1 -- 172.21.15.1:0/491751193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85dc0cc600 msgr2=0x7f85dc06dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.510+0000 7f85e1f83700 1 --2- 172.21.15.1:0/491751193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85dc0cc600 0x7f85dc06dcb0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f85d000a9e0 tx=0x7f85d000acf0 comp rx=0 tx=0).stop 2024-03-18T20:06:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.510+0000 7f85e1f83700 1 -- 172.21.15.1:0/491751193 shutdown_connections 2024-03-18T20:06:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.510+0000 7f85e1f83700 1 --2- 172.21.15.1:0/491751193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85dc0cc600 0x7f85dc06dcb0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.510+0000 7f85e1f83700 1 -- 172.21.15.1:0/491751193 >> 172.21.15.1:0/491751193 conn(0x7f85dc0b1270 msgr2=0x7f85dc0b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.510+0000 7f85e1f83700 1 -- 172.21.15.1:0/491751193 shutdown_connections 2024-03-18T20:06:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.510+0000 7f85e1f83700 1 -- 172.21.15.1:0/491751193 wait complete. 2024-03-18T20:06:44.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.511+0000 7f85e1f83700 1 Processor -- start 2024-03-18T20:06:44.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.511+0000 7f85e1f83700 1 -- start start 2024-03-18T20:06:44.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.512+0000 7f85e1f83700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85dc0ca110 0x7f85dc1f0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:44.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.512+0000 7f85e1f83700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85dc0cc600 0x7f85dc0a86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:44.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.512+0000 7f85e1f83700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85dc15cfc0 0x7f85dc0cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:44.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.512+0000 7f85e1f83700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc0c43f0 con 0x7f85dc0cc600 2024-03-18T20:06:44.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.512+0000 7f85e1f83700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc1f1440 con 0x7f85dc15cfc0 2024-03-18T20:06:44.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.512+0000 7f85e1f83700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc1f1720 con 0x7f85dc0ca110 2024-03-18T20:06:44.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.512+0000 7f85daffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85dc0cc600 0x7f85dc0a86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:44.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.512+0000 7f85daffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85dc0cc600 0x7f85dc0a86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56430/0 (socket says 172.21.15.1:56430) 2024-03-18T20:06:44.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.512+0000 7f85daffd700 1 -- 172.21.15.1:0/4017838076 learned_addr learned my addr 172.21.15.1:0/4017838076 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:44.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.512+0000 7f85dbfff700 1 --2- 172.21.15.1:0/4017838076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85dc15cfc0 0x7f85dc0cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:44.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.512+0000 7f85db7fe700 1 --2- 172.21.15.1:0/4017838076 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85dc0ca110 0x7f85dc1f0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:44.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.512+0000 7f85daffd700 1 -- 172.21.15.1:0/4017838076 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85dc0ca110 msgr2=0x7f85dc1f0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:44.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.512+0000 7f85daffd700 1 --2- 172.21.15.1:0/4017838076 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85dc0ca110 0x7f85dc1f0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:44.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.512+0000 7f85daffd700 1 -- 172.21.15.1:0/4017838076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85dc15cfc0 msgr2=0x7f85dc0cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:44.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.512+0000 7f85daffd700 1 --2- 172.21.15.1:0/4017838076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85dc15cfc0 0x7f85dc0cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:44.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.512+0000 7f85daffd700 1 -- 172.21.15.1:0/4017838076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d000a640 con 0x7f85dc0cc600 2024-03-18T20:06:44.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.512+0000 7f85dbfff700 1 --2- 172.21.15.1:0/4017838076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85dc15cfc0 0x7f85dc0cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:44.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.512+0000 7f85db7fe700 1 --2- 172.21.15.1:0/4017838076 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85dc0ca110 0x7f85dc1f0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:44.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.513+0000 7f85daffd700 1 --2- 172.21.15.1:0/4017838076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85dc0cc600 0x7f85dc0a86b0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f85d0006ec0 tx=0x7f85d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:44.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.513+0000 7f85c3fff700 1 -- 172.21.15.1:0/4017838076 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d001d070 con 0x7f85dc0cc600 2024-03-18T20:06:44.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.513+0000 7f85c3fff700 1 -- 172.21.15.1:0/4017838076 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d0003ab0 con 0x7f85dc0cc600 2024-03-18T20:06:44.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.513+0000 7f85e1f83700 1 -- 172.21.15.1:0/4017838076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85dc1f1a00 con 0x7f85dc0cc600 2024-03-18T20:06:44.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.513+0000 7f85c3fff700 1 -- 172.21.15.1:0/4017838076 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d0003e40 con 0x7f85dc0cc600 2024-03-18T20:06:44.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.513+0000 7f85e1f83700 1 -- 172.21.15.1:0/4017838076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85dc0a8bf0 con 0x7f85dc0cc600 2024-03-18T20:06:44.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.514+0000 7f85e1f83700 1 -- 172.21.15.1:0/4017838076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85dc0676a0 con 0x7f85dc0cc600 2024-03-18T20:06:44.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.514+0000 7f85c3fff700 1 -- 172.21.15.1:0/4017838076 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f85d00183f0 con 0x7f85dc0cc600 2024-03-18T20:06:44.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.515+0000 7f85c3fff700 1 --2- 172.21.15.1:0/4017838076 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f85c806adb0 0x7f85c806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:44.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.515+0000 7f85c3fff700 1 -- 172.21.15.1:0/4017838076 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f85d008ad40 con 0x7f85dc0cc600 2024-03-18T20:06:44.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.515+0000 7f85db7fe700 1 --2- 172.21.15.1:0/4017838076 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f85c806adb0 0x7f85c806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:44.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.515+0000 7f85db7fe700 1 --2- 172.21.15.1:0/4017838076 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f85c806adb0 0x7f85c806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f85cc007fd0 tx=0x7f85cc009040 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:44.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.519+0000 7f85c3fff700 1 -- 172.21.15.1:0/4017838076 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f85d0055c00 con 0x7f85dc0cc600 2024-03-18T20:06:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.767+0000 7f85e1f83700 1 -- 172.21.15.1:0/4017838076 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85dc15c070 con 0x7f85dc0cc600 2024-03-18T20:06:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.768+0000 7f85c3fff700 1 -- 172.21.15.1:0/4017838076 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f85d0059220 con 0x7f85dc0cc600 2024-03-18T20:06:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:44.773 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":81,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7874,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3186058752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3186058752},{"type":"v1","addr":"172.21.15.1:6836","nonce":3186058752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":7886,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1126668185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1126668185},{"type":"v1","addr":"172.21.15.165:6835","nonce":1126668185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9028,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3922981693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3922981693},{"type":"v1","addr":"172.21.15.165:6834","nonce":3922981693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:06:41.339378+0000","modified":"2024-03-18T20:06:44.348416+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9909},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9909":{"gid":9909,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/3096868087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3096868087},{"type":"v1","addr":"172.21.15.1:6837","nonce":3096868087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-03-18T20:06:44.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.770+0000 7f85e1f83700 1 -- 172.21.15.1:0/4017838076 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f85c806adb0 msgr2=0x7f85c806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:44.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.770+0000 7f85e1f83700 1 --2- 172.21.15.1:0/4017838076 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f85c806adb0 0x7f85c806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f85cc007fd0 tx=0x7f85cc009040 comp rx=0 tx=0).stop 2024-03-18T20:06:44.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.770+0000 7f85e1f83700 1 -- 172.21.15.1:0/4017838076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85dc0cc600 msgr2=0x7f85dc0a86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:44.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.770+0000 7f85e1f83700 1 --2- 172.21.15.1:0/4017838076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85dc0cc600 0x7f85dc0a86b0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f85d0006ec0 tx=0x7f85d0004750 comp rx=0 tx=0).stop 2024-03-18T20:06:44.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.770+0000 7f85e1f83700 1 -- 172.21.15.1:0/4017838076 shutdown_connections 2024-03-18T20:06:44.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.770+0000 7f85e1f83700 1 --2- 172.21.15.1:0/4017838076 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f85c806adb0 0x7f85c806d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:44.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.770+0000 7f85e1f83700 1 --2- 172.21.15.1:0/4017838076 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f85dc0ca110 0x7f85dc1f0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:44.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.770+0000 7f85e1f83700 1 --2- 172.21.15.1:0/4017838076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85dc15cfc0 0x7f85dc0cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:44.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.770+0000 7f85e1f83700 1 --2- 172.21.15.1:0/4017838076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85dc0cc600 0x7f85dc0a86b0 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:44.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.770+0000 7f85e1f83700 1 -- 172.21.15.1:0/4017838076 >> 172.21.15.1:0/4017838076 conn(0x7f85dc0b1270 msgr2=0x7f85dc0ca790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:44.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.770+0000 7f85e1f83700 1 -- 172.21.15.1:0/4017838076 shutdown_connections 2024-03-18T20:06:44.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.770+0000 7f85e1f83700 1 -- 172.21.15.1:0/4017838076 wait complete. 2024-03-18T20:06:44.777 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 81 2024-03-18T20:06:44.787 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:06:44.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.908+0000 7fbbb3e37700 1 Processor -- start 2024-03-18T20:06:44.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.908+0000 7fbbb3e37700 1 -- start start 2024-03-18T20:06:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.909+0000 7fbbb3e37700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbbac06ae60 0x7fbbac164f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.909+0000 7fbbb3e37700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbbac1465a0 0x7fbbac167480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.909+0000 7fbbb3e37700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbac072660 con 0x7fbbac146190 2024-03-18T20:06:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.909+0000 7fbbb3e37700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbac06f850 con 0x7fbbac1465a0 2024-03-18T20:06:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.909+0000 7fbbb3e37700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbac06ebd0 con 0x7fbbac06ae60 2024-03-18T20:06:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.909+0000 7fbbb23d4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbbac1465a0 0x7fbbac167480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:44.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.909+0000 7fbbb23d4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbbac1465a0 0x7fbbac167480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56438/0 (socket says 172.21.15.1:56438) 2024-03-18T20:06:44.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.909+0000 7fbbb23d4700 1 -- 172.21.15.1:0/2911747782 learned_addr learned my addr 172.21.15.1:0/2911747782 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:44.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.909+0000 7fbbb13d2700 1 --2- 172.21.15.1:0/2911747782 >> v2:172.21.15.165:3300/0 conn(0x7fbbac06ae60 0x7fbbac164f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:44.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.909+0000 7fbbb23d4700 1 -- 172.21.15.1:0/2911747782 >> v2:172.21.15.165:3300/0 conn(0x7fbbac06ae60 msgr2=0x7fbbac164f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:44.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.909+0000 7fbbb23d4700 1 --2- 172.21.15.1:0/2911747782 >> v2:172.21.15.165:3300/0 conn(0x7fbbac06ae60 0x7fbbac164f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:44.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.909+0000 7fbbb23d4700 1 -- 172.21.15.1:0/2911747782 >> v1:172.21.15.165:6789/0 conn(0x7fbbac146190 legacy=0x7fbbac06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:44.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.909+0000 7fbbb23d4700 1 -- 172.21.15.1:0/2911747782 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbac06eeb0 con 0x7fbbac1465a0 2024-03-18T20:06:44.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.909+0000 7fbbb23d4700 1 --2- 172.21.15.1:0/2911747782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbac1465a0 0x7fbbac167480 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fbbac0731a0 tx=0x7fbba400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed04e76185d58bf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:44.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.910+0000 7fbbb0bd1700 1 -- 172.21.15.1:0/2911747782 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba400e070 con 0x7fbbac1465a0 2024-03-18T20:06:44.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.910+0000 7fbbb0bd1700 1 -- 172.21.15.1:0/2911747782 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba400b910 con 0x7fbbac1465a0 2024-03-18T20:06:44.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.910+0000 7fbbb0bd1700 1 -- 172.21.15.1:0/2911747782 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba400bcc0 con 0x7fbbac1465a0 2024-03-18T20:06:44.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.910+0000 7fbbb3e37700 1 -- 172.21.15.1:0/2911747782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbac1465a0 msgr2=0x7fbbac167480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:44.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.910+0000 7fbbb3e37700 1 --2- 172.21.15.1:0/2911747782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbac1465a0 0x7fbbac167480 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fbbac0731a0 tx=0x7fbba400b330 comp rx=0 tx=0).stop 2024-03-18T20:06:44.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.910+0000 7fbbb3e37700 1 -- 172.21.15.1:0/2911747782 shutdown_connections 2024-03-18T20:06:44.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.910+0000 7fbbb3e37700 1 --2- 172.21.15.1:0/2911747782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbac1465a0 0x7fbbac167480 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.910+0000 7fbbb3e37700 1 --2- 172.21.15.1:0/2911747782 >> v2:172.21.15.165:3300/0 conn(0x7fbbac06ae60 0x7fbbac164f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.910+0000 7fbbb3e37700 1 -- 172.21.15.1:0/2911747782 >> 172.21.15.1:0/2911747782 conn(0x7fbbac0b1d20 msgr2=0x7fbbac0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.910+0000 7fbbb3e37700 1 -- 172.21.15.1:0/2911747782 shutdown_connections 2024-03-18T20:06:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.910+0000 7fbbb3e37700 1 -- 172.21.15.1:0/2911747782 wait complete. 2024-03-18T20:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.911+0000 7fbbb3e37700 1 Processor -- start 2024-03-18T20:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.912+0000 7fbbb3e37700 1 -- start start 2024-03-18T20:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.912+0000 7fbbb3e37700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbbac06ae60 0x7fbbac1e5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.912+0000 7fbbb3e37700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbac1465a0 0x7fbbac1416a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.912+0000 7fbbb3e37700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbac141be0 0x7fbbac0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.912+0000 7fbbb3e37700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac12f570 con 0x7fbbac141be0 2024-03-18T20:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.912+0000 7fbbb3e37700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac1e6240 con 0x7fbbac1465a0 2024-03-18T20:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.912+0000 7fbbb3e37700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac1421f0 con 0x7fbbac06ae60 2024-03-18T20:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.912+0000 7fbbb23d4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbac141be0 0x7fbbac0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.912+0000 7fbbb23d4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbac141be0 0x7fbbac0ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56446/0 (socket says 172.21.15.1:56446) 2024-03-18T20:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.912+0000 7fbbb23d4700 1 -- 172.21.15.1:0/3847467513 learned_addr learned my addr 172.21.15.1:0/3847467513 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.913+0000 7fbbb13d2700 1 --2- 172.21.15.1:0/3847467513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbac1465a0 0x7fbbac1416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.913+0000 7fbbb1bd3700 1 --2- 172.21.15.1:0/3847467513 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbbac06ae60 0x7fbbac1e5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.913+0000 7fbbb23d4700 1 -- 172.21.15.1:0/3847467513 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbbac06ae60 msgr2=0x7fbbac1e5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.913+0000 7fbbb23d4700 1 --2- 172.21.15.1:0/3847467513 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbbac06ae60 0x7fbbac1e5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.913+0000 7fbbb23d4700 1 -- 172.21.15.1:0/3847467513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbac1465a0 msgr2=0x7fbbac1416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.913+0000 7fbbb23d4700 1 --2- 172.21.15.1:0/3847467513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbac1465a0 0x7fbbac1416a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.913+0000 7fbbb23d4700 1 -- 172.21.15.1:0/3847467513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba4009b30 con 0x7fbbac141be0 2024-03-18T20:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.913+0000 7fbbb23d4700 1 --2- 172.21.15.1:0/3847467513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbac141be0 0x7fbbac0ce590 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fbbac067530 tx=0x7fbba4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.913+0000 7fbba27fc700 1 -- 172.21.15.1:0/3847467513 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba40099b0 con 0x7fbbac141be0 2024-03-18T20:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.913+0000 7fbba27fc700 1 -- 172.21.15.1:0/3847467513 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba40092e0 con 0x7fbbac141be0 2024-03-18T20:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.913+0000 7fbbb3e37700 1 -- 172.21.15.1:0/3847467513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbac1424d0 con 0x7fbbac141be0 2024-03-18T20:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.913+0000 7fbba27fc700 1 -- 172.21.15.1:0/3847467513 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba401c640 con 0x7fbbac141be0 2024-03-18T20:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.913+0000 7fbbb3e37700 1 -- 172.21.15.1:0/3847467513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbac13eda0 con 0x7fbbac141be0 2024-03-18T20:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.914+0000 7fbbb3e37700 1 -- 172.21.15.1:0/3847467513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbac12f060 con 0x7fbbac141be0 2024-03-18T20:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.915+0000 7fbba27fc700 1 -- 172.21.15.1:0/3847467513 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbba401c7a0 con 0x7fbbac141be0 2024-03-18T20:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.915+0000 7fbba27fc700 1 --2- 172.21.15.1:0/3847467513 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbb9006adb0 0x7fbb9006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.915+0000 7fbbb1bd3700 1 --2- 172.21.15.1:0/3847467513 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbb9006adb0 0x7fbb9006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.915+0000 7fbba27fc700 1 -- 172.21.15.1:0/3847467513 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbba408b410 con 0x7fbbac141be0 2024-03-18T20:06:44.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.916+0000 7fbbb1bd3700 1 --2- 172.21.15.1:0/3847467513 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbb9006adb0 0x7fbb9006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbba8007fd0 tx=0x7fbba8009040 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:44.920+0000 7fbba27fc700 1 -- 172.21.15.1:0/3847467513 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbbac12f060 con 0x7fbbac141be0 2024-03-18T20:06:45.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.123+0000 7fbbb3e37700 1 -- 172.21.15.1:0/3847467513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbac12f060 con 0x7fbbac141be0 2024-03-18T20:06:45.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.125+0000 7fbba27fc700 1 -- 172.21.15.1:0/3847467513 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) v1 ==== 74+0+16323 (secure 0 0 0) 0x7fbbac12f060 con 0x7fbbac141be0 2024-03-18T20:06:45.129 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:45.129 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":181,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:06:42.341320+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:06:39.228206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-18T20:06:40.315627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:06:45.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.127+0000 7fbbb3e37700 1 -- 172.21.15.1:0/3847467513 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbb9006adb0 msgr2=0x7fbb9006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:45.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.127+0000 7fbbb3e37700 1 --2- 172.21.15.1:0/3847467513 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbb9006adb0 0x7fbb9006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbba8007fd0 tx=0x7fbba8009040 comp rx=0 tx=0).stop 2024-03-18T20:06:45.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.127+0000 7fbbb3e37700 1 -- 172.21.15.1:0/3847467513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbac141be0 msgr2=0x7fbbac0ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:45.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.127+0000 7fbbb3e37700 1 --2- 172.21.15.1:0/3847467513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbac141be0 0x7fbbac0ce590 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fbbac067530 tx=0x7fbba4004ad0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.127+0000 7fbbb3e37700 1 -- 172.21.15.1:0/3847467513 shutdown_connections 2024-03-18T20:06:45.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.127+0000 7fbbb3e37700 1 --2- 172.21.15.1:0/3847467513 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbb9006adb0 0x7fbb9006d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.127+0000 7fbbb3e37700 1 --2- 172.21.15.1:0/3847467513 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbbac06ae60 0x7fbbac1e5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.127+0000 7fbbb3e37700 1 --2- 172.21.15.1:0/3847467513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbac141be0 0x7fbbac0ce590 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.127+0000 7fbbb3e37700 1 --2- 172.21.15.1:0/3847467513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbac1465a0 0x7fbbac1416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.127+0000 7fbbb3e37700 1 -- 172.21.15.1:0/3847467513 >> 172.21.15.1:0/3847467513 conn(0x7fbbac0b1d20 msgr2=0x7fbbac0ad080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:45.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.127+0000 7fbbb3e37700 1 -- 172.21.15.1:0/3847467513 shutdown_connections 2024-03-18T20:06:45.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.127+0000 7fbbb3e37700 1 -- 172.21.15.1:0/3847467513 wait complete. 2024-03-18T20:06:45.144 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T20:06:45.145 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:06:45.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.265+0000 7f18e8888700 1 Processor -- start 2024-03-18T20:06:45.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.265+0000 7f18e8888700 1 -- start start 2024-03-18T20:06:45.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.266+0000 7f18e8888700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f18e4099a40 0x7f18e4099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:45.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.266+0000 7f18e8888700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f18e40cc310 0x7f18e40cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:45.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.266+0000 7f18e8888700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e409a3e0 con 0x7f18e415d680 2024-03-18T20:06:45.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.266+0000 7f18e8888700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e40d0c80 con 0x7f18e40cc310 2024-03-18T20:06:45.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.266+0000 7f18e8888700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e40d09a0 con 0x7f18e4099a40 2024-03-18T20:06:45.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.266+0000 7f18e2d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f18e40cc310 0x7f18e40cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:45.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.266+0000 7f18e2d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f18e40cc310 0x7f18e40cc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56454/0 (socket says 172.21.15.1:56454) 2024-03-18T20:06:45.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.266+0000 7f18e2d9d700 1 -- 172.21.15.1:0/218757974 learned_addr learned my addr 172.21.15.1:0/218757974 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:45.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.267+0000 7f18e1d9b700 1 --2- 172.21.15.1:0/218757974 >> v2:172.21.15.165:3300/0 conn(0x7f18e4099a40 0x7f18e4099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:45.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.267+0000 7f18e2d9d700 1 -- 172.21.15.1:0/218757974 >> v2:172.21.15.165:3300/0 conn(0x7f18e4099a40 msgr2=0x7f18e4099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:45.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.267+0000 7f18e2d9d700 1 --2- 172.21.15.1:0/218757974 >> v2:172.21.15.165:3300/0 conn(0x7f18e4099a40 0x7f18e4099ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.267+0000 7f18e2d9d700 1 -- 172.21.15.1:0/218757974 >> v1:172.21.15.165:6789/0 conn(0x7f18e415d680 legacy=0x7f18e40cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:45.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.267+0000 7f18e2d9d700 1 -- 172.21.15.1:0/218757974 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e40cb830 con 0x7f18e40cc310 2024-03-18T20:06:45.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.267+0000 7f18e2d9d700 1 --2- 172.21.15.1:0/218757974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18e40cc310 0x7f18e40cc7a0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f18e4071cb0 tx=0x7f18dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c259fb76b9b3ab99 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:45.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.267+0000 7f18e159a700 1 -- 172.21.15.1:0/218757974 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc00e070 con 0x7f18e40cc310 2024-03-18T20:06:45.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.267+0000 7f18e159a700 1 -- 172.21.15.1:0/218757974 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18dc003a20 con 0x7f18e40cc310 2024-03-18T20:06:45.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.267+0000 7f18e159a700 1 -- 172.21.15.1:0/218757974 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc003dd0 con 0x7f18e40cc310 2024-03-18T20:06:45.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.268+0000 7f18e8888700 1 -- 172.21.15.1:0/218757974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18e40cc310 msgr2=0x7f18e40cc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:45.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.268+0000 7f18e8888700 1 --2- 172.21.15.1:0/218757974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18e40cc310 0x7f18e40cc7a0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f18e4071cb0 tx=0x7f18dc00b330 comp rx=0 tx=0).stop 2024-03-18T20:06:45.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.268+0000 7f18e8888700 1 -- 172.21.15.1:0/218757974 shutdown_connections 2024-03-18T20:06:45.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.268+0000 7f18e8888700 1 --2- 172.21.15.1:0/218757974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18e40cc310 0x7f18e40cc7a0 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.268+0000 7f18e8888700 1 --2- 172.21.15.1:0/218757974 >> v2:172.21.15.165:3300/0 conn(0x7f18e4099a40 0x7f18e4099ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.268+0000 7f18e8888700 1 -- 172.21.15.1:0/218757974 >> 172.21.15.1:0/218757974 conn(0x7f18e40b2ba0 msgr2=0x7f18e40b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:45.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.268+0000 7f18e8888700 1 -- 172.21.15.1:0/218757974 shutdown_connections 2024-03-18T20:06:45.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.268+0000 7f18e8888700 1 -- 172.21.15.1:0/218757974 wait complete. 2024-03-18T20:06:45.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.269+0000 7f18e8888700 1 Processor -- start 2024-03-18T20:06:45.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.269+0000 7f18e8888700 1 -- start start 2024-03-18T20:06:45.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.270+0000 7f18e8888700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18e4099a40 0x7f18e40b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:45.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.270+0000 7f18e8888700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f18e40cc310 0x7f18e40b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:45.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.270+0000 7f18e8888700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18e415d680 0x7f18e40ae690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:45.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.270+0000 7f18e8888700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e40c91e0 con 0x7f18e4099a40 2024-03-18T20:06:45.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.270+0000 7f18e8888700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e40c8560 con 0x7f18e415d680 2024-03-18T20:06:45.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.270+0000 7f18e8888700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e40c8840 con 0x7f18e40cc310 2024-03-18T20:06:45.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.270+0000 7f18e259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18e4099a40 0x7f18e40b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:45.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.270+0000 7f18e259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18e4099a40 0x7f18e40b5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56460/0 (socket says 172.21.15.1:56460) 2024-03-18T20:06:45.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.270+0000 7f18e259c700 1 -- 172.21.15.1:0/3541149200 learned_addr learned my addr 172.21.15.1:0/3541149200 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:45.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.270+0000 7f18e2d9d700 1 --2- 172.21.15.1:0/3541149200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18e415d680 0x7f18e40ae690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:45.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.270+0000 7f18e1d9b700 1 --2- 172.21.15.1:0/3541149200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f18e40cc310 0x7f18e40b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:45.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.270+0000 7f18e259c700 1 -- 172.21.15.1:0/3541149200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f18e40cc310 msgr2=0x7f18e40b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:45.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.270+0000 7f18e259c700 1 --2- 172.21.15.1:0/3541149200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f18e40cc310 0x7f18e40b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.270+0000 7f18e259c700 1 -- 172.21.15.1:0/3541149200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18e415d680 msgr2=0x7f18e40ae690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:45.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.270+0000 7f18e259c700 1 --2- 172.21.15.1:0/3541149200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18e415d680 0x7f18e40ae690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.270+0000 7f18e259c700 1 -- 172.21.15.1:0/3541149200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18dc009b30 con 0x7f18e4099a40 2024-03-18T20:06:45.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.270+0000 7f18e259c700 1 --2- 172.21.15.1:0/3541149200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18e4099a40 0x7f18e40b5ee0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f18d400b4a0 tx=0x7f18d400b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:45.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.271+0000 7f18daffd700 1 -- 172.21.15.1:0/3541149200 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d4011830 con 0x7f18e4099a40 2024-03-18T20:06:45.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.271+0000 7f18daffd700 1 -- 172.21.15.1:0/3541149200 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d4011990 con 0x7f18e4099a40 2024-03-18T20:06:45.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.271+0000 7f18e8888700 1 -- 172.21.15.1:0/3541149200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e40c8b20 con 0x7f18e4099a40 2024-03-18T20:06:45.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.271+0000 7f18daffd700 1 -- 172.21.15.1:0/3541149200 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d4011d40 con 0x7f18e4099a40 2024-03-18T20:06:45.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.271+0000 7f18e8888700 1 -- 172.21.15.1:0/3541149200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e40b6420 con 0x7f18e4099a40 2024-03-18T20:06:45.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.272+0000 7f18e8888700 1 -- 172.21.15.1:0/3541149200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e40b7250 con 0x7f18e4099a40 2024-03-18T20:06:45.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.272+0000 7f18daffd700 1 -- 172.21.15.1:0/3541149200 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f18d401a400 con 0x7f18e4099a40 2024-03-18T20:06:45.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.273+0000 7f18daffd700 1 --2- 172.21.15.1:0/3541149200 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f18c406adb0 0x7f18c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:45.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.273+0000 7f18daffd700 1 -- 172.21.15.1:0/3541149200 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f18d40891f0 con 0x7f18e4099a40 2024-03-18T20:06:45.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.273+0000 7f18e1d9b700 1 --2- 172.21.15.1:0/3541149200 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f18c406adb0 0x7f18c406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:45.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.273+0000 7f18e1d9b700 1 --2- 172.21.15.1:0/3541149200 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f18c406adb0 0x7f18c406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f18e40c8000 tx=0x7f18cc008040 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:45.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.277+0000 7f18daffd700 1 -- 172.21.15.1:0/3541149200 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f18d4054130 con 0x7f18e4099a40 2024-03-18T20:06:45.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.477+0000 7f18e8888700 1 -- 172.21.15.1:0/3541149200 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f18e40d0680 con 0x7f18c406adb0 2024-03-18T20:06:45.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.486+0000 7f18daffd700 1 -- 172.21.15.1:0/3541149200 <== mgr.7721 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+285077 (secure 0 0 0) 0x7f18cc00c040 con 0x7f18c406adb0 2024-03-18T20:06:45.494 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:45.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.489+0000 7f18e8888700 1 -- 172.21.15.1:0/3541149200 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f18c406adb0 msgr2=0x7f18c406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.489+0000 7f18e8888700 1 --2- 172.21.15.1:0/3541149200 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f18c406adb0 0x7f18c406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f18e40c8000 tx=0x7f18cc008040 comp rx=0 tx=0).stop 2024-03-18T20:06:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.489+0000 7f18e8888700 1 -- 172.21.15.1:0/3541149200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18e4099a40 msgr2=0x7f18e40b5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.489+0000 7f18e8888700 1 --2- 172.21.15.1:0/3541149200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18e4099a40 0x7f18e40b5ee0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f18d400b4a0 tx=0x7f18d400b7b0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.489+0000 7f18e8888700 1 -- 172.21.15.1:0/3541149200 shutdown_connections 2024-03-18T20:06:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.489+0000 7f18e8888700 1 --2- 172.21.15.1:0/3541149200 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f18c406adb0 0x7f18c406d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.489+0000 7f18e8888700 1 --2- 172.21.15.1:0/3541149200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18e415d680 0x7f18e40ae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.489+0000 7f18e8888700 1 --2- 172.21.15.1:0/3541149200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18e4099a40 0x7f18e40b5ee0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.489+0000 7f18e8888700 1 --2- 172.21.15.1:0/3541149200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f18e40cc310 0x7f18e40b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.489+0000 7f18e8888700 1 -- 172.21.15.1:0/3541149200 >> 172.21.15.1:0/3541149200 conn(0x7f18e40b2ba0 msgr2=0x7f18e40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.489+0000 7f18e8888700 1 -- 172.21.15.1:0/3541149200 shutdown_connections 2024-03-18T20:06:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.489+0000 7f18e8888700 1 -- 172.21.15.1:0/3541149200 wait complete. 2024-03-18T20:06:45.497 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:06:45.506 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-18T20:06:43.777754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592114,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":464,"num_write_kb":2422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":294,"ondisk_log_size":294,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":44,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":364440,"kb_used_data":18724,"kb_used_omap":12,"kb_used_meta":345587,"kb_avail":754610280,"statfs":{"total":773094113280,"available":772720926720,"internally_reserved":0,"allocated":19173376,"data_stored":10270320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":353881657},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1746,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000678"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.359676+0000","last_change":"2024-03-18T20:06:41.359220+0000","last_active":"2024-03-18T20:06:41.359676+0000","last_peered":"2024-03-18T20:06:41.359676+0000","last_clean":"2024-03-18T20:06:41.359676+0000","last_became_active":"2024-03-18T20:06:41.358926+0000","last_became_peered":"2024-03-18T20:06:41.358926+0000","last_unstale":"2024-03-18T20:06:41.359676+0000","last_undegraded":"2024-03-18T20:06:41.359676+0000","last_fullsized":"2024-03-18T20:06:41.359676+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:51:35.490802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.358313+0000","last_change":"2024-03-18T20:06:41.353598+0000","last_active":"2024-03-18T20:06:41.358313+0000","last_peered":"2024-03-18T20:06:41.358313+0000","last_clean":"2024-03-18T20:06:41.358313+0000","last_became_active":"2024-03-18T20:06:41.353326+0000","last_became_peered":"2024-03-18T20:06:41.353326+0000","last_unstale":"2024-03-18T20:06:41.358313+0000","last_undegraded":"2024-03-18T20:06:41.358313+0000","last_fullsized":"2024-03-18T20:06:41.358313+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:03:49.415865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.348785+0000","last_change":"2024-03-18T20:06:41.365681+0000","last_active":"2024-03-18T20:06:42.348785+0000","last_peered":"2024-03-18T20:06:42.348785+0000","last_clean":"2024-03-18T20:06:42.348785+0000","last_became_active":"2024-03-18T20:06:41.365366+0000","last_became_peered":"2024-03-18T20:06:41.365366+0000","last_unstale":"2024-03-18T20:06:42.348785+0000","last_undegraded":"2024-03-18T20:06:42.348785+0000","last_fullsized":"2024-03-18T20:06:42.348785+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:13.069617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.350053+0000","last_change":"2024-03-18T20:06:41.367557+0000","last_active":"2024-03-18T20:06:42.350053+0000","last_peered":"2024-03-18T20:06:42.350053+0000","last_clean":"2024-03-18T20:06:42.350053+0000","last_became_active":"2024-03-18T20:06:41.366600+0000","last_became_peered":"2024-03-18T20:06:41.366600+0000","last_unstale":"2024-03-18T20:06:42.350053+0000","last_undegraded":"2024-03-18T20:06:42.350053+0000","last_fullsized":"2024-03-18T20:06:42.350053+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:44:10.550192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.374015+0000","last_change":"2024-03-18T20:06:41.370507+0000","last_active":"2024-03-18T20:06:41.374015+0000","last_peered":"2024-03-18T20:06:41.374015+0000","last_clean":"2024-03-18T20:06:41.374015+0000","last_became_active":"2024-03-18T20:06:41.370284+0000","last_became_peered":"2024-03-18T20:06:41.370284+0000","last_unstale":"2024-03-18T20:06:41.374015+0000","last_undegraded":"2024-03-18T20:06:41.374015+0000","last_fullsized":"2024-03-18T20:06:41.374015+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:28:07.293987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.348876+0000","last_change":"2024-03-18T20:06:41.361828+0000","last_active":"2024-03-18T20:06:42.348876+0000","last_peered":"2024-03-18T20:06:42.348876+0000","last_clean":"2024-03-18T20:06:42.348876+0000","last_became_active":"2024-03-18T20:06:41.361368+0000","last_became_peered":"2024-03-18T20:06:41.361368+0000","last_unstale":"2024-03-18T20:06:42.348876+0000","last_undegraded":"2024-03-18T20:06:42.348876+0000","last_fullsized":"2024-03-18T20:06:42.348876+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:10.907762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.348759+0000","last_change":"2024-03-18T20:06:41.362947+0000","last_active":"2024-03-18T20:06:42.348759+0000","last_peered":"2024-03-18T20:06:42.348759+0000","last_clean":"2024-03-18T20:06:42.348759+0000","last_became_active":"2024-03-18T20:06:41.362074+0000","last_became_peered":"2024-03-18T20:06:41.362074+0000","last_unstale":"2024-03-18T20:06:42.348759+0000","last_undegraded":"2024-03-18T20:06:42.348759+0000","last_fullsized":"2024-03-18T20:06:42.348759+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:57.248985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.363605+0000","last_change":"2024-03-18T20:06:41.360933+0000","last_active":"2024-03-18T20:06:41.363605+0000","last_peered":"2024-03-18T20:06:41.363605+0000","last_clean":"2024-03-18T20:06:41.363605+0000","last_became_active":"2024-03-18T20:06:41.360696+0000","last_became_peered":"2024-03-18T20:06:41.360696+0000","last_unstale":"2024-03-18T20:06:41.363605+0000","last_undegraded":"2024-03-18T20:06:41.363605+0000","last_fullsized":"2024-03-18T20:06:41.363605+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:29:17.155958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.348899+0000","last_change":"2024-03-18T20:06:41.352072+0000","last_active":"2024-03-18T20:06:42.348899+0000","last_peered":"2024-03-18T20:06:42.348899+0000","last_clean":"2024-03-18T20:06:42.348899+0000","last_became_active":"2024-03-18T20:06:41.351789+0000","last_became_peered":"2024-03-18T20:06:41.351789+0000","last_unstale":"2024-03-18T20:06:42.348899+0000","last_undegraded":"2024-03-18T20:06:42.348899+0000","last_fullsized":"2024-03-18T20:06:42.348899+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:47:35.787872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.373869+0000","last_change":"2024-03-18T20:06:41.367241+0000","last_active":"2024-03-18T20:06:41.373869+0000","last_peered":"2024-03-18T20:06:41.373869+0000","last_clean":"2024-03-18T20:06:41.373869+0000","last_became_active":"2024-03-18T20:06:41.366895+0000","last_became_peered":"2024-03-18T20:06:41.366895+0000","last_unstale":"2024-03-18T20:06:41.373869+0000","last_undegraded":"2024-03-18T20:06:41.373869+0000","last_fullsized":"2024-03-18T20:06:41.373869+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:59:08.429068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.369646+0000","last_change":"2024-03-18T20:06:41.369243+0000","last_active":"2024-03-18T20:06:41.369646+0000","last_peered":"2024-03-18T20:06:41.369646+0000","last_clean":"2024-03-18T20:06:41.369646+0000","last_became_active":"2024-03-18T20:06:41.368507+0000","last_became_peered":"2024-03-18T20:06:41.368507+0000","last_unstale":"2024-03-18T20:06:41.369646+0000","last_undegraded":"2024-03-18T20:06:41.369646+0000","last_fullsized":"2024-03-18T20:06:41.369646+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:37:49.042419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.472565+0000","last_change":"2024-03-18T20:06:41.362015+0000","last_active":"2024-03-18T20:06:42.472565+0000","last_peered":"2024-03-18T20:06:42.472565+0000","last_clean":"2024-03-18T20:06:42.472565+0000","last_became_active":"2024-03-18T20:06:41.361780+0000","last_became_peered":"2024-03-18T20:06:41.361780+0000","last_unstale":"2024-03-18T20:06:42.472565+0000","last_undegraded":"2024-03-18T20:06:42.472565+0000","last_fullsized":"2024-03-18T20:06:42.472565+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:25:10.634219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.349812+0000","last_change":"2024-03-18T20:06:41.369369+0000","last_active":"2024-03-18T20:06:42.349812+0000","last_peered":"2024-03-18T20:06:42.349812+0000","last_clean":"2024-03-18T20:06:42.349812+0000","last_became_active":"2024-03-18T20:06:41.369103+0000","last_became_peered":"2024-03-18T20:06:41.369103+0000","last_unstale":"2024-03-18T20:06:42.349812+0000","last_undegraded":"2024-03-18T20:06:42.349812+0000","last_fullsized":"2024-03-18T20:06:42.349812+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:41:54.947369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.373733+0000","last_change":"2024-03-18T20:06:41.364531+0000","last_active":"2024-03-18T20:06:41.373733+0000","last_peered":"2024-03-18T20:06:41.373733+0000","last_clean":"2024-03-18T20:06:41.373733+0000","last_became_active":"2024-03-18T20:06:41.364242+0000","last_became_peered":"2024-03-18T20:06:41.364242+0000","last_unstale":"2024-03-18T20:06:41.373733+0000","last_undegraded":"2024-03-18T20:06:41.373733+0000","last_fullsized":"2024-03-18T20:06:41.373733+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:46.243166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.374090+0000","last_change":"2024-03-18T20:06:41.371281+0000","last_active":"2024-03-18T20:06:41.374090+0000","last_peered":"2024-03-18T20:06:41.374090+0000","last_clean":"2024-03-18T20:06:41.374090+0000","last_became_active":"2024-03-18T20:06:41.371018+0000","last_became_peered":"2024-03-18T20:06:41.371018+0000","last_unstale":"2024-03-18T20:06:41.374090+0000","last_undegraded":"2024-03-18T20:06:41.374090+0000","last_fullsized":"2024-03-18T20:06:41.374090+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:25:22.714322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.350116+0000","last_change":"2024-03-18T20:06:41.371738+0000","last_active":"2024-03-18T20:06:42.350116+0000","last_peered":"2024-03-18T20:06:42.350116+0000","last_clean":"2024-03-18T20:06:42.350116+0000","last_became_active":"2024-03-18T20:06:41.371403+0000","last_became_peered":"2024-03-18T20:06:41.371403+0000","last_unstale":"2024-03-18T20:06:42.350116+0000","last_undegraded":"2024-03-18T20:06:42.350116+0000","last_fullsized":"2024-03-18T20:06:42.350116+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:38:28.729138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.373695+0000","last_change":"2024-03-18T20:06:41.364374+0000","last_active":"2024-03-18T20:06:41.373695+0000","last_peered":"2024-03-18T20:06:41.373695+0000","last_clean":"2024-03-18T20:06:41.373695+0000","last_became_active":"2024-03-18T20:06:41.364038+0000","last_became_peered":"2024-03-18T20:06:41.364038+0000","last_unstale":"2024-03-18T20:06:41.373695+0000","last_undegraded":"2024-03-18T20:06:41.373695+0000","last_fullsized":"2024-03-18T20:06:41.373695+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:10.907762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.349822+0000","last_change":"2024-03-18T20:06:41.367842+0000","last_active":"2024-03-18T20:06:42.349822+0000","last_peered":"2024-03-18T20:06:42.349822+0000","last_clean":"2024-03-18T20:06:42.349822+0000","last_became_active":"2024-03-18T20:06:41.366940+0000","last_became_peered":"2024-03-18T20:06:41.366940+0000","last_unstale":"2024-03-18T20:06:42.349822+0000","last_undegraded":"2024-03-18T20:06:42.349822+0000","last_fullsized":"2024-03-18T20:06:42.349822+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:28:14.508789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.360255+0000","last_change":"2024-03-18T20:06:41.353818+0000","last_active":"2024-03-18T20:06:41.360255+0000","last_peered":"2024-03-18T20:06:41.360255+0000","last_clean":"2024-03-18T20:06:41.360255+0000","last_became_active":"2024-03-18T20:06:41.353373+0000","last_became_peered":"2024-03-18T20:06:41.353373+0000","last_unstale":"2024-03-18T20:06:41.360255+0000","last_undegraded":"2024-03-18T20:06:41.360255+0000","last_fullsized":"2024-03-18T20:06:41.360255+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:45:25.376374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:39.899482+0000","last_change":"2024-03-18T20:06:39.899482+0000","last_active":"2024-03-18T20:06:39.899482+0000","last_peered":"2024-03-18T20:06:39.899482+0000","last_clean":"2024-03-18T20:06:39.899482+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:39.899482+0000","last_undegraded":"2024-03-18T20:06:39.899482+0000","last_fullsized":"2024-03-18T20:06:39.899482+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.363668+0000","last_change":"2024-03-18T20:06:41.360831+0000","last_active":"2024-03-18T20:06:41.363668+0000","last_peered":"2024-03-18T20:06:41.363668+0000","last_clean":"2024-03-18T20:06:41.363668+0000","last_became_active":"2024-03-18T20:06:41.360534+0000","last_became_peered":"2024-03-18T20:06:41.360534+0000","last_unstale":"2024-03-18T20:06:41.363668+0000","last_undegraded":"2024-03-18T20:06:41.363668+0000","last_fullsized":"2024-03-18T20:06:41.363668+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:28:45.396835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"180'1","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.364826+0000","last_change":"2024-03-18T20:06:40.933779+0000","last_active":"2024-03-18T20:06:41.364826+0000","last_peered":"2024-03-18T20:06:41.364826+0000","last_clean":"2024-03-18T20:06:41.364826+0000","last_became_active":"2024-03-18T20:06:40.933440+0000","last_became_peered":"2024-03-18T20:06:40.933440+0000","last_unstale":"2024-03-18T20:06:41.364826+0000","last_undegraded":"2024-03-18T20:06:41.364826+0000","last_fullsized":"2024-03-18T20:06:41.364826+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:37:27.163834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.350097+0000","last_change":"2024-03-18T20:06:41.370745+0000","last_active":"2024-03-18T20:06:42.350097+0000","last_peered":"2024-03-18T20:06:42.350097+0000","last_clean":"2024-03-18T20:06:42.350097+0000","last_became_active":"2024-03-18T20:06:41.370430+0000","last_became_peered":"2024-03-18T20:06:41.370430+0000","last_unstale":"2024-03-18T20:06:42.350097+0000","last_undegraded":"2024-03-18T20:06:42.350097+0000","last_fullsized":"2024-03-18T20:06:42.350097+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:37:28.168864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"180'2","reported_seq":12,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.365199+0000","last_change":"2024-03-18T20:06:40.934062+0000","last_active":"2024-03-18T20:06:41.365199+0000","last_peered":"2024-03-18T20:06:41.365199+0000","last_clean":"2024-03-18T20:06:41.365199+0000","last_became_active":"2024-03-18T20:06:40.933790+0000","last_became_peered":"2024-03-18T20:06:40.933790+0000","last_unstale":"2024-03-18T20:06:41.365199+0000","last_undegraded":"2024-03-18T20:06:41.365199+0000","last_fullsized":"2024-03-18T20:06:41.365199+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:32.848530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.364875+0000","last_change":"2024-03-18T20:06:41.360059+0000","last_active":"2024-03-18T20:06:41.364875+0000","last_peered":"2024-03-18T20:06:41.364875+0000","last_clean":"2024-03-18T20:06:41.364875+0000","last_became_active":"2024-03-18T20:06:41.359618+0000","last_became_peered":"2024-03-18T20:06:41.359618+0000","last_unstale":"2024-03-18T20:06:41.364875+0000","last_undegraded":"2024-03-18T20:06:41.364875+0000","last_fullsized":"2024-03-18T20:06:41.364875+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:52:13.527547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"180'2","reported_seq":12,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.363049+0000","last_change":"2024-03-18T20:06:40.931067+0000","last_active":"2024-03-18T20:06:41.363049+0000","last_peered":"2024-03-18T20:06:41.363049+0000","last_clean":"2024-03-18T20:06:41.363049+0000","last_became_active":"2024-03-18T20:06:40.930770+0000","last_became_peered":"2024-03-18T20:06:40.930770+0000","last_unstale":"2024-03-18T20:06:41.363049+0000","last_undegraded":"2024-03-18T20:06:41.363049+0000","last_fullsized":"2024-03-18T20:06:41.363049+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:08.593608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.472597+0000","last_change":"2024-03-18T20:06:41.366528+0000","last_active":"2024-03-18T20:06:42.472597+0000","last_peered":"2024-03-18T20:06:42.472597+0000","last_clean":"2024-03-18T20:06:42.472597+0000","last_became_active":"2024-03-18T20:06:41.366307+0000","last_became_peered":"2024-03-18T20:06:41.366307+0000","last_unstale":"2024-03-18T20:06:42.472597+0000","last_undegraded":"2024-03-18T20:06:42.472597+0000","last_fullsized":"2024-03-18T20:06:42.472597+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:08.081957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"180'1","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.350413+0000","last_change":"2024-03-18T20:06:40.930762+0000","last_active":"2024-03-18T20:06:42.350413+0000","last_peered":"2024-03-18T20:06:42.350413+0000","last_clean":"2024-03-18T20:06:42.350413+0000","last_became_active":"2024-03-18T20:06:40.930511+0000","last_became_peered":"2024-03-18T20:06:40.930511+0000","last_unstale":"2024-03-18T20:06:42.350413+0000","last_undegraded":"2024-03-18T20:06:42.350413+0000","last_fullsized":"2024-03-18T20:06:42.350413+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:45.238136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.348563+0000","last_change":"2024-03-18T20:06:41.352339+0000","last_active":"2024-03-18T20:06:42.348563+0000","last_peered":"2024-03-18T20:06:42.348563+0000","last_clean":"2024-03-18T20:06:42.348563+0000","last_became_active":"2024-03-18T20:06:41.352038+0000","last_became_peered":"2024-03-18T20:06:41.352038+0000","last_unstale":"2024-03-18T20:06:42.348563+0000","last_undegraded":"2024-03-18T20:06:42.348563+0000","last_fullsized":"2024-03-18T20:06:42.348563+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:46.243166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:39.899482+0000","last_change":"2024-03-18T20:06:39.899482+0000","last_active":"2024-03-18T20:06:39.899482+0000","last_peered":"2024-03-18T20:06:39.899482+0000","last_clean":"2024-03-18T20:06:39.899482+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:39.899482+0000","last_undegraded":"2024-03-18T20:06:39.899482+0000","last_fullsized":"2024-03-18T20:06:39.899482+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.358219+0000","last_change":"2024-03-18T20:06:41.352440+0000","last_active":"2024-03-18T20:06:41.358219+0000","last_peered":"2024-03-18T20:06:41.358219+0000","last_clean":"2024-03-18T20:06:41.358219+0000","last_became_active":"2024-03-18T20:06:41.352185+0000","last_became_peered":"2024-03-18T20:06:41.352185+0000","last_unstale":"2024-03-18T20:06:41.358219+0000","last_undegraded":"2024-03-18T20:06:41.358219+0000","last_fullsized":"2024-03-18T20:06:41.358219+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:23:38.989856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:39.899482+0000","last_change":"2024-03-18T20:06:39.899482+0000","last_active":"2024-03-18T20:06:39.899482+0000","last_peered":"2024-03-18T20:06:39.899482+0000","last_clean":"2024-03-18T20:06:39.899482+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:39.899482+0000","last_undegraded":"2024-03-18T20:06:39.899482+0000","last_fullsized":"2024-03-18T20:06:39.899482+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:39.899482+0000","last_change":"2024-03-18T20:06:39.899482+0000","last_active":"2024-03-18T20:06:39.899482+0000","last_peered":"2024-03-18T20:06:39.899482+0000","last_clean":"2024-03-18T20:06:39.899482+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:39.899482+0000","last_undegraded":"2024-03-18T20:06:39.899482+0000","last_fullsized":"2024-03-18T20:06:39.899482+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.350309+0000","last_change":"2024-03-18T20:06:41.367875+0000","last_active":"2024-03-18T20:06:42.350309+0000","last_peered":"2024-03-18T20:06:42.350309+0000","last_clean":"2024-03-18T20:06:42.350309+0000","last_became_active":"2024-03-18T20:06:41.367637+0000","last_became_peered":"2024-03-18T20:06:41.367637+0000","last_unstale":"2024-03-18T20:06:42.350309+0000","last_undegraded":"2024-03-18T20:06:42.350309+0000","last_fullsized":"2024-03-18T20:06:42.350309+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:01:45.149612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.345339+0000","last_change":"2024-03-18T20:06:40.927739+0000","last_active":"2024-03-18T20:06:41.345339+0000","last_peered":"2024-03-18T20:06:41.345339+0000","last_clean":"2024-03-18T20:06:41.345339+0000","last_became_active":"2024-03-18T20:06:40.927424+0000","last_became_peered":"2024-03-18T20:06:40.927424+0000","last_unstale":"2024-03-18T20:06:41.345339+0000","last_undegraded":"2024-03-18T20:06:41.345339+0000","last_fullsized":"2024-03-18T20:06:41.345339+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:05:32.348554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.374003+0000","last_change":"2024-03-18T20:06:41.370086+0000","last_active":"2024-03-18T20:06:41.374003+0000","last_peered":"2024-03-18T20:06:41.374003+0000","last_clean":"2024-03-18T20:06:41.374003+0000","last_became_active":"2024-03-18T20:06:41.364823+0000","last_became_peered":"2024-03-18T20:06:41.364823+0000","last_unstale":"2024-03-18T20:06:41.374003+0000","last_undegraded":"2024-03-18T20:06:41.374003+0000","last_fullsized":"2024-03-18T20:06:41.374003+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:59:29.574799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:39.899482+0000","last_change":"2024-03-18T20:06:39.899482+0000","last_active":"2024-03-18T20:06:39.899482+0000","last_peered":"2024-03-18T20:06:39.899482+0000","last_clean":"2024-03-18T20:06:39.899482+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:39.899482+0000","last_undegraded":"2024-03-18T20:06:39.899482+0000","last_fullsized":"2024-03-18T20:06:39.899482+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.373798+0000","last_change":"2024-03-18T20:06:41.367037+0000","last_active":"2024-03-18T20:06:41.373798+0000","last_peered":"2024-03-18T20:06:41.373798+0000","last_clean":"2024-03-18T20:06:41.373798+0000","last_became_active":"2024-03-18T20:06:41.366675+0000","last_became_peered":"2024-03-18T20:06:41.366675+0000","last_unstale":"2024-03-18T20:06:41.373798+0000","last_undegraded":"2024-03-18T20:06:41.373798+0000","last_fullsized":"2024-03-18T20:06:41.373798+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:24:42.956982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"180'3","reported_seq":13,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.367778+0000","last_change":"2024-03-18T20:06:40.929699+0000","last_active":"2024-03-18T20:06:41.367778+0000","last_peered":"2024-03-18T20:06:41.367778+0000","last_clean":"2024-03-18T20:06:41.367778+0000","last_became_active":"2024-03-18T20:06:40.926591+0000","last_became_peered":"2024-03-18T20:06:40.926591+0000","last_unstale":"2024-03-18T20:06:41.367778+0000","last_undegraded":"2024-03-18T20:06:41.367778+0000","last_fullsized":"2024-03-18T20:06:41.367778+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:40.459736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.349975+0000","last_change":"2024-03-18T20:06:41.369513+0000","last_active":"2024-03-18T20:06:42.349975+0000","last_peered":"2024-03-18T20:06:42.349975+0000","last_clean":"2024-03-18T20:06:42.349975+0000","last_became_active":"2024-03-18T20:06:41.367288+0000","last_became_peered":"2024-03-18T20:06:41.367288+0000","last_unstale":"2024-03-18T20:06:42.349975+0000","last_undegraded":"2024-03-18T20:06:42.349975+0000","last_fullsized":"2024-03-18T20:06:42.349975+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:33.853559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:39.899482+0000","last_change":"2024-03-18T20:06:39.899482+0000","last_active":"2024-03-18T20:06:39.899482+0000","last_peered":"2024-03-18T20:06:39.899482+0000","last_clean":"2024-03-18T20:06:39.899482+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:39.899482+0000","last_undegraded":"2024-03-18T20:06:39.899482+0000","last_fullsized":"2024-03-18T20:06:39.899482+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.363586+0000","last_change":"2024-03-18T20:06:41.359898+0000","last_active":"2024-03-18T20:06:41.363586+0000","last_peered":"2024-03-18T20:06:41.363586+0000","last_clean":"2024-03-18T20:06:41.363586+0000","last_became_active":"2024-03-18T20:06:41.359646+0000","last_became_peered":"2024-03-18T20:06:41.359646+0000","last_unstale":"2024-03-18T20:06:41.363586+0000","last_undegraded":"2024-03-18T20:06:41.363586+0000","last_fullsized":"2024-03-18T20:06:41.363586+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:28:34.744775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.348858+0000","last_change":"2024-03-18T20:06:40.931811+0000","last_active":"2024-03-18T20:06:42.348858+0000","last_peered":"2024-03-18T20:06:42.348858+0000","last_clean":"2024-03-18T20:06:42.348858+0000","last_became_active":"2024-03-18T20:06:40.931556+0000","last_became_peered":"2024-03-18T20:06:40.931556+0000","last_unstale":"2024-03-18T20:06:42.348858+0000","last_undegraded":"2024-03-18T20:06:42.348858+0000","last_fullsized":"2024-03-18T20:06:42.348858+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:20.769001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.348733+0000","last_change":"2024-03-18T20:06:40.929730+0000","last_active":"2024-03-18T20:06:42.348733+0000","last_peered":"2024-03-18T20:06:42.348733+0000","last_clean":"2024-03-18T20:06:42.348733+0000","last_became_active":"2024-03-18T20:06:40.929455+0000","last_became_peered":"2024-03-18T20:06:40.929455+0000","last_unstale":"2024-03-18T20:06:42.348733+0000","last_undegraded":"2024-03-18T20:06:42.348733+0000","last_fullsized":"2024-03-18T20:06:42.348733+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:20:20.338062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.365419+0000","last_change":"2024-03-18T20:06:41.361564+0000","last_active":"2024-03-18T20:06:41.365419+0000","last_peered":"2024-03-18T20:06:41.365419+0000","last_clean":"2024-03-18T20:06:41.365419+0000","last_became_active":"2024-03-18T20:06:41.361155+0000","last_became_peered":"2024-03-18T20:06:41.361155+0000","last_unstale":"2024-03-18T20:06:41.365419+0000","last_undegraded":"2024-03-18T20:06:41.365419+0000","last_fullsized":"2024-03-18T20:06:41.365419+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:05.115393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:39.899482+0000","last_change":"2024-03-18T20:06:39.899482+0000","last_active":"2024-03-18T20:06:39.899482+0000","last_peered":"2024-03-18T20:06:39.899482+0000","last_clean":"2024-03-18T20:06:39.899482+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:39.899482+0000","last_undegraded":"2024-03-18T20:06:39.899482+0000","last_fullsized":"2024-03-18T20:06:39.899482+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.364902+0000","last_change":"2024-03-18T20:06:41.364174+0000","last_active":"2024-03-18T20:06:41.364902+0000","last_peered":"2024-03-18T20:06:41.364902+0000","last_clean":"2024-03-18T20:06:41.364902+0000","last_became_active":"2024-03-18T20:06:41.363875+0000","last_became_peered":"2024-03-18T20:06:41.363875+0000","last_unstale":"2024-03-18T20:06:41.364902+0000","last_undegraded":"2024-03-18T20:06:41.364902+0000","last_fullsized":"2024-03-18T20:06:41.364902+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:48.634167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"180'1","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.366559+0000","last_change":"2024-03-18T20:06:40.930476+0000","last_active":"2024-03-18T20:06:41.366559+0000","last_peered":"2024-03-18T20:06:41.366559+0000","last_clean":"2024-03-18T20:06:41.366559+0000","last_became_active":"2024-03-18T20:06:40.930198+0000","last_became_peered":"2024-03-18T20:06:40.930198+0000","last_unstale":"2024-03-18T20:06:41.366559+0000","last_undegraded":"2024-03-18T20:06:41.366559+0000","last_fullsized":"2024-03-18T20:06:41.366559+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:27:19.221024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.370516+0000","last_change":"2024-03-18T20:06:41.369419+0000","last_active":"2024-03-18T20:06:41.370516+0000","last_peered":"2024-03-18T20:06:41.370516+0000","last_clean":"2024-03-18T20:06:41.370516+0000","last_became_active":"2024-03-18T20:06:41.368840+0000","last_became_peered":"2024-03-18T20:06:41.368840+0000","last_unstale":"2024-03-18T20:06:41.370516+0000","last_undegraded":"2024-03-18T20:06:41.370516+0000","last_fullsized":"2024-03-18T20:06:41.370516+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:51:57.729525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"180'2","reported_seq":12,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.363953+0000","last_change":"2024-03-18T20:06:40.931433+0000","last_active":"2024-03-18T20:06:41.363953+0000","last_peered":"2024-03-18T20:06:41.363953+0000","last_clean":"2024-03-18T20:06:41.363953+0000","last_became_active":"2024-03-18T20:06:40.931159+0000","last_became_peered":"2024-03-18T20:06:40.931159+0000","last_unstale":"2024-03-18T20:06:41.363953+0000","last_undegraded":"2024-03-18T20:06:41.363953+0000","last_fullsized":"2024-03-18T20:06:41.363953+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:58.234326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.d","version":"180'1","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.360577+0000","last_change":"2024-03-18T20:06:40.932926+0000","last_active":"2024-03-18T20:06:41.360577+0000","last_peered":"2024-03-18T20:06:41.360577+0000","last_clean":"2024-03-18T20:06:41.360577+0000","last_became_active":"2024-03-18T20:06:40.932626+0000","last_became_peered":"2024-03-18T20:06:40.932626+0000","last_unstale":"2024-03-18T20:06:41.360577+0000","last_undegraded":"2024-03-18T20:06:41.360577+0000","last_fullsized":"2024-03-18T20:06:41.360577+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:47.629138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.348883+0000","last_change":"2024-03-18T20:06:40.932442+0000","last_active":"2024-03-18T20:06:42.348883+0000","last_peered":"2024-03-18T20:06:42.348883+0000","last_clean":"2024-03-18T20:06:42.348883+0000","last_became_active":"2024-03-18T20:06:40.932159+0000","last_became_peered":"2024-03-18T20:06:40.932159+0000","last_unstale":"2024-03-18T20:06:42.348883+0000","last_undegraded":"2024-03-18T20:06:42.348883+0000","last_fullsized":"2024-03-18T20:06:42.348883+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:52.804403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.374207+0000","last_change":"2024-03-18T20:06:41.373675+0000","last_active":"2024-03-18T20:06:41.374207+0000","last_peered":"2024-03-18T20:06:41.374207+0000","last_clean":"2024-03-18T20:06:41.374207+0000","last_became_active":"2024-03-18T20:06:41.373382+0000","last_became_peered":"2024-03-18T20:06:41.373382+0000","last_unstale":"2024-03-18T20:06:41.374207+0000","last_undegraded":"2024-03-18T20:06:41.374207+0000","last_fullsized":"2024-03-18T20:06:41.374207+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:06:22.367037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"180'2","reported_seq":13,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.472494+0000","last_change":"2024-03-18T20:06:40.937596+0000","last_active":"2024-03-18T20:06:42.472494+0000","last_peered":"2024-03-18T20:06:42.472494+0000","last_clean":"2024-03-18T20:06:42.472494+0000","last_became_active":"2024-03-18T20:06:40.935870+0000","last_became_peered":"2024-03-18T20:06:40.935870+0000","last_unstale":"2024-03-18T20:06:42.472494+0000","last_undegraded":"2024-03-18T20:06:42.472494+0000","last_fullsized":"2024-03-18T20:06:42.472494+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:37.006175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.372713+0000","last_change":"2024-03-18T20:06:41.364920+0000","last_active":"2024-03-18T20:06:41.372713+0000","last_peered":"2024-03-18T20:06:41.372713+0000","last_clean":"2024-03-18T20:06:41.372713+0000","last_became_active":"2024-03-18T20:06:41.364665+0000","last_became_peered":"2024-03-18T20:06:41.364665+0000","last_unstale":"2024-03-18T20:06:41.372713+0000","last_undegraded":"2024-03-18T20:06:41.372713+0000","last_fullsized":"2024-03-18T20:06:41.372713+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:26.254209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.349874+0000","last_change":"2024-03-18T20:06:40.930701+0000","last_active":"2024-03-18T20:06:42.349874+0000","last_peered":"2024-03-18T20:06:42.349874+0000","last_clean":"2024-03-18T20:06:42.349874+0000","last_became_active":"2024-03-18T20:06:40.930424+0000","last_became_peered":"2024-03-18T20:06:40.930424+0000","last_unstale":"2024-03-18T20:06:42.349874+0000","last_undegraded":"2024-03-18T20:06:42.349874+0000","last_fullsized":"2024-03-18T20:06:42.349874+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:24:41.951953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.367665+0000","last_change":"2024-03-18T20:06:41.365985+0000","last_active":"2024-03-18T20:06:41.367665+0000","last_peered":"2024-03-18T20:06:41.367665+0000","last_clean":"2024-03-18T20:06:41.367665+0000","last_became_active":"2024-03-18T20:06:41.365707+0000","last_became_peered":"2024-03-18T20:06:41.365707+0000","last_unstale":"2024-03-18T20:06:41.367665+0000","last_undegraded":"2024-03-18T20:06:41.367665+0000","last_fullsized":"2024-03-18T20:06:41.367665+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:24.751000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.472412+0000","last_change":"2024-03-18T20:06:40.937204+0000","last_active":"2024-03-18T20:06:42.472412+0000","last_peered":"2024-03-18T20:06:42.472412+0000","last_clean":"2024-03-18T20:06:42.472412+0000","last_became_active":"2024-03-18T20:06:40.936078+0000","last_became_peered":"2024-03-18T20:06:40.936078+0000","last_unstale":"2024-03-18T20:06:42.472412+0000","last_undegraded":"2024-03-18T20:06:42.472412+0000","last_fullsized":"2024-03-18T20:06:42.472412+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:17:41.767913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.365386+0000","last_change":"2024-03-18T20:06:41.361411+0000","last_active":"2024-03-18T20:06:41.365386+0000","last_peered":"2024-03-18T20:06:41.365386+0000","last_clean":"2024-03-18T20:06:41.365386+0000","last_became_active":"2024-03-18T20:06:41.360784+0000","last_became_peered":"2024-03-18T20:06:41.360784+0000","last_unstale":"2024-03-18T20:06:41.365386+0000","last_undegraded":"2024-03-18T20:06:41.365386+0000","last_fullsized":"2024-03-18T20:06:41.365386+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:16:52.447786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.349925+0000","last_change":"2024-03-18T20:06:40.939149+0000","last_active":"2024-03-18T20:06:42.349925+0000","last_peered":"2024-03-18T20:06:42.349925+0000","last_clean":"2024-03-18T20:06:42.349925+0000","last_became_active":"2024-03-18T20:06:40.938934+0000","last_became_peered":"2024-03-18T20:06:40.938934+0000","last_unstale":"2024-03-18T20:06:42.349925+0000","last_undegraded":"2024-03-18T20:06:42.349925+0000","last_fullsized":"2024-03-18T20:06:42.349925+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:25:21.709293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.7","version":"180'2","reported_seq":13,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.349024+0000","last_change":"2024-03-18T20:06:40.933408+0000","last_active":"2024-03-18T20:06:42.349024+0000","last_peered":"2024-03-18T20:06:42.349024+0000","last_clean":"2024-03-18T20:06:42.349024+0000","last_became_active":"2024-03-18T20:06:40.932950+0000","last_became_peered":"2024-03-18T20:06:40.932950+0000","last_unstale":"2024-03-18T20:06:42.349024+0000","last_undegraded":"2024-03-18T20:06:42.349024+0000","last_fullsized":"2024-03-18T20:06:42.349024+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:33.586280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.349042+0000","last_change":"2024-03-18T20:06:41.362325+0000","last_active":"2024-03-18T20:06:42.349042+0000","last_peered":"2024-03-18T20:06:42.349042+0000","last_clean":"2024-03-18T20:06:42.349042+0000","last_became_active":"2024-03-18T20:06:41.362006+0000","last_became_peered":"2024-03-18T20:06:41.362006+0000","last_unstale":"2024-03-18T20:06:42.349042+0000","last_undegraded":"2024-03-18T20:06:42.349042+0000","last_fullsized":"2024-03-18T20:06:42.349042+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:10:58.991209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:39.899482+0000","last_change":"2024-03-18T20:06:39.899482+0000","last_active":"2024-03-18T20:06:39.899482+0000","last_peered":"2024-03-18T20:06:39.899482+0000","last_clean":"2024-03-18T20:06:39.899482+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:39.899482+0000","last_undegraded":"2024-03-18T20:06:39.899482+0000","last_fullsized":"2024-03-18T20:06:39.899482+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.346768+0000","last_change":"2024-03-18T20:06:40.931183+0000","last_active":"2024-03-18T20:06:41.346768+0000","last_peered":"2024-03-18T20:06:41.346768+0000","last_clean":"2024-03-18T20:06:41.346768+0000","last_became_active":"2024-03-18T20:06:40.930896+0000","last_became_peered":"2024-03-18T20:06:40.930896+0000","last_unstale":"2024-03-18T20:06:41.346768+0000","last_undegraded":"2024-03-18T20:06:41.346768+0000","last_fullsized":"2024-03-18T20:06:41.346768+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:00:22.450064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:40.904511+0000","last_change":"2024-03-18T20:06:40.904511+0000","last_active":"2024-03-18T20:06:40.904511+0000","last_peered":"2024-03-18T20:06:40.904511+0000","last_clean":"2024-03-18T20:06:40.904511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:40.904511+0000","last_undegraded":"2024-03-18T20:06:40.904511+0000","last_fullsized":"2024-03-18T20:06:40.904511+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:39.899482+0000","last_change":"2024-03-18T20:06:39.899482+0000","last_active":"2024-03-18T20:06:39.899482+0000","last_peered":"2024-03-18T20:06:39.899482+0000","last_clean":"2024-03-18T20:06:39.899482+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:39.899482+0000","last_undegraded":"2024-03-18T20:06:39.899482+0000","last_fullsized":"2024-03-18T20:06:39.899482+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.349650+0000","last_change":"2024-03-18T20:06:41.370421+0000","last_active":"2024-03-18T20:06:42.349650+0000","last_peered":"2024-03-18T20:06:42.349650+0000","last_clean":"2024-03-18T20:06:42.349650+0000","last_became_active":"2024-03-18T20:06:41.370105+0000","last_became_peered":"2024-03-18T20:06:41.370105+0000","last_unstale":"2024-03-18T20:06:42.349650+0000","last_undegraded":"2024-03-18T20:06:42.349650+0000","last_fullsized":"2024-03-18T20:06:42.349650+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:09.598637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:39.899482+0000","last_change":"2024-03-18T20:06:39.899482+0000","last_active":"2024-03-18T20:06:39.899482+0000","last_peered":"2024-03-18T20:06:39.899482+0000","last_clean":"2024-03-18T20:06:39.899482+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:39.899482+0000","last_undegraded":"2024-03-18T20:06:39.899482+0000","last_fullsized":"2024-03-18T20:06:39.899482+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.348630+0000","last_change":"2024-03-18T20:06:41.357946+0000","last_active":"2024-03-18T20:06:42.348630+0000","last_peered":"2024-03-18T20:06:42.348630+0000","last_clean":"2024-03-18T20:06:42.348630+0000","last_became_active":"2024-03-18T20:06:41.357682+0000","last_became_peered":"2024-03-18T20:06:41.357682+0000","last_unstale":"2024-03-18T20:06:42.348630+0000","last_undegraded":"2024-03-18T20:06:42.348630+0000","last_fullsized":"2024-03-18T20:06:42.348630+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:59:08.429068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.356036+0000","last_change":"2024-03-18T20:06:40.930876+0000","last_active":"2024-03-18T20:06:41.356036+0000","last_peered":"2024-03-18T20:06:41.356036+0000","last_clean":"2024-03-18T20:06:41.356036+0000","last_became_active":"2024-03-18T20:06:40.930606+0000","last_became_peered":"2024-03-18T20:06:40.930606+0000","last_unstale":"2024-03-18T20:06:41.356036+0000","last_undegraded":"2024-03-18T20:06:41.356036+0000","last_fullsized":"2024-03-18T20:06:41.356036+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:20:20.338062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.372567+0000","last_change":"2024-03-18T20:06:41.369577+0000","last_active":"2024-03-18T20:06:41.372567+0000","last_peered":"2024-03-18T20:06:41.372567+0000","last_clean":"2024-03-18T20:06:41.372567+0000","last_became_active":"2024-03-18T20:06:41.369332+0000","last_became_peered":"2024-03-18T20:06:41.369332+0000","last_unstale":"2024-03-18T20:06:41.372567+0000","last_undegraded":"2024-03-18T20:06:41.372567+0000","last_fullsized":"2024-03-18T20:06:41.372567+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:17:59.998734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:06:39.899482+0000","last_change":"2024-03-18T20:06:39.899482+0000","last_active":"2024-03-18T20:06:39.899482+0000","last_peered":"2024-03-18T20:06:39.899482+0000","last_clean":"2024-03-18T20:06:39.899482+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:06:39.899482+0000","last_undegraded":"2024-03-18T20:06:39.899482+0000","last_fullsized":"2024-03-18T20:06:39.899482+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.350214+0000","last_change":"2024-03-18T20:06:41.371561+0000","last_active":"2024-03-18T20:06:42.350214+0000","last_peered":"2024-03-18T20:06:42.350214+0000","last_clean":"2024-03-18T20:06:42.350214+0000","last_became_active":"2024-03-18T20:06:41.371307+0000","last_became_peered":"2024-03-18T20:06:41.371307+0000","last_unstale":"2024-03-18T20:06:42.350214+0000","last_undegraded":"2024-03-18T20:06:42.350214+0000","last_fullsized":"2024-03-18T20:06:42.350214+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:23:31.228650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.345969+0000","last_change":"2024-03-18T20:06:40.927231+0000","last_active":"2024-03-18T20:06:41.345969+0000","last_peered":"2024-03-18T20:06:41.345969+0000","last_clean":"2024-03-18T20:06:41.345969+0000","last_became_active":"2024-03-18T20:06:40.926933+0000","last_became_peered":"2024-03-18T20:06:40.926933+0000","last_unstale":"2024-03-18T20:06:41.345969+0000","last_undegraded":"2024-03-18T20:06:41.345969+0000","last_fullsized":"2024-03-18T20:06:41.345969+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:01:44.144583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.350106+0000","last_change":"2024-03-18T20:06:41.367390+0000","last_active":"2024-03-18T20:06:42.350106+0000","last_peered":"2024-03-18T20:06:42.350106+0000","last_clean":"2024-03-18T20:06:42.350106+0000","last_became_active":"2024-03-18T20:06:41.367077+0000","last_became_peered":"2024-03-18T20:06:41.367077+0000","last_unstale":"2024-03-18T20:06:42.350106+0000","last_undegraded":"2024-03-18T20:06:42.350106+0000","last_fullsized":"2024-03-18T20:06:42.350106+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:59.239355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"180'6","reported_seq":17,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.472662+0000","last_change":"2024-03-18T20:06:40.937199+0000","last_active":"2024-03-18T20:06:42.472662+0000","last_peered":"2024-03-18T20:06:42.472662+0000","last_clean":"2024-03-18T20:06:42.472662+0000","last_became_active":"2024-03-18T20:06:40.935955+0000","last_became_peered":"2024-03-18T20:06:40.935955+0000","last_unstale":"2024-03-18T20:06:42.472662+0000","last_undegraded":"2024-03-18T20:06:42.472662+0000","last_fullsized":"2024-03-18T20:06:42.472662+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:25:32.038941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"170'271","reported_seq":875,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-18T20:06:40.920985+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:06:40.920985+0000","last_peered":"2024-03-18T20:06:40.920985+0000","last_clean":"2024-03-18T20:06:40.920985+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:06:40.920985+0000","last_undegraded":"2024-03-18T20:06:40.920985+0000","last_fullsized":"2024-03-18T20:06:40.920985+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":1746,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23,"ondisk_log_size":23,"up":42,"acting":42,"num_store_stats":6},{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":86,"acting":86,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47848,"kb_used_data":2780,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323992,"statfs":{"total":96636764160,"available":96587767808,"internally_reserved":0,"allocated":2846720,"data_stored":1730078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.32500000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.77000000000000002,"5min":0.83999999999999997,"15min":1.1259999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.61899999999999999},"min":{"1min":0.41099999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.77400000000000002,"5min":0.81699999999999995,"15min":1.1379999999999999},"last":0.39600000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.82799999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.61799999999999999,"15min":0.61599999999999999},"min":{"1min":0.37,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79900000000000004,"5min":0.83899999999999997,"15min":1.1719999999999999},"last":0.52300000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.38500000000000001,"5min":0.36499999999999999,"15min":0.33900000000000002},"max":{"1min":0.77700000000000002,"5min":0.872,"15min":0.872},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.42699999999999999,"5min":0.41899999999999998,"15min":0.39200000000000002},"max":{"1min":0.82199999999999995,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.372}]},{"osd":3,"last update":"Mon Mar 18 20:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.61699999999999999,"15min":0.61399999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81399999999999995,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.64000000000000001,"15min":0.63200000000000001},"min":{"1min":0.45100000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.89000000000000001},"last":0.435}]},{"osd":4,"last update":"Mon Mar 18 20:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47399999999999998,"15min":0.46999999999999997},"min":{"1min":0.33000000000000002,"5min":0.32500000000000001,"15min":0.315},"max":{"1min":0.73599999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.70099999999999996,"5min":0.76100000000000001,"15min":0.84699999999999998},"last":0.41599999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.502,"15min":0.50700000000000001},"min":{"1min":0.374,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.375,"5min":0.35299999999999998,"15min":0.32900000000000001},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.55800000000000005}]},{"osd":6,"last update":"Mon Mar 18 20:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56799999999999995,"15min":0.56000000000000005},"min":{"1min":0.41299999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80400000000000005,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.55700000000000005,"15min":0.55500000000000005},"min":{"1min":0.41099999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.91300000000000003},"last":0.54600000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575762,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37432,"kb_used_data":2212,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334408,"statfs":{"total":96636764160,"available":96598433792,"internally_reserved":0,"allocated":2265088,"data_stored":1146450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.61299999999999999,"15min":0.59599999999999997},"min":{"1min":0.40300000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73899999999999999,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.60499999999999998,"15min":0.59999999999999998},"min":{"1min":0.371,"5min":0.36099999999999999,"15min":0.30299999999999999},"max":{"1min":0.70499999999999996,"5min":1.054,"15min":1.054},"last":0.63500000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60799999999999998,"15min":0.59599999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.34000000000000002},"max":{"1min":0.76700000000000002,"5min":0.82799999999999996,"15min":0.84599999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.61399999999999999,"15min":0.59999999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.28299999999999997},"max":{"1min":0.72999999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.42899999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.64100000000000001,"15min":0.625},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.40600000000000003},"max":{"1min":0.74299999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60399999999999998,"15min":0.59599999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.71799999999999997,"5min":0.79800000000000004,"15min":0.89600000000000002},"last":0.66800000000000004}]},{"osd":3,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.64200000000000002,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73399999999999999,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60799999999999998,"15min":0.60099999999999998},"min":{"1min":0.42399999999999999,"5min":0.41599999999999998,"15min":0.39500000000000002},"max":{"1min":0.72099999999999997,"5min":1.448,"15min":1.448},"last":0.60199999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.47499999999999998},"min":{"1min":0.318,"5min":0.316,"15min":0.316},"max":{"1min":0.69299999999999995,"5min":0.78000000000000003,"15min":0.82999999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.505,"15min":0.50700000000000001},"min":{"1min":0.372,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.65900000000000003,"5min":0.84399999999999997,"15min":0.86499999999999999},"last":0.68999999999999995}]},{"osd":5,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.495,"15min":0.48999999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.63800000000000001,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54500000000000004,"15min":0.53300000000000003},"min":{"1min":0.43099999999999999,"5min":0.38200000000000001,"15min":0.33900000000000002},"max":{"1min":0.71199999999999997,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.624}]},{"osd":7,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.39600000000000002,"5min":0.378,"15min":0.34300000000000003},"max":{"1min":0.71899999999999997,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56200000000000006,"15min":0.54700000000000004},"min":{"1min":0.45000000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67900000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.68000000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575762,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41516,"kb_used_data":2200,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330324,"statfs":{"total":96636764160,"available":96594251776,"internally_reserved":0,"allocated":2252800,"data_stored":1139758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:06:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40500000000000003,"15min":0.40100000000000002},"min":{"1min":0.26000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.54500000000000004,"5min":0.59299999999999997,"15min":0.63600000000000001},"last":0.317},{"interface":"front","average":{"1min":0.432,"5min":0.434,"15min":0.44400000000000001},"min":{"1min":0.29599999999999999,"5min":0.27500000000000002,"15min":0.26200000000000001},"max":{"1min":0.56000000000000005,"5min":0.71099999999999997,"15min":0.73599999999999999},"last":0.30199999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:06:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.25600000000000001},"max":{"1min":0.56100000000000005,"5min":0.66500000000000004,"15min":0.70299999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.47599999999999998,"15min":0.47299999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.60899999999999999,"5min":0.77500000000000002,"15min":0.79300000000000004},"last":0.34300000000000003}]},{"osd":3,"last update":"Mon Mar 18 20:06:00 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.497,"15min":0.497},"min":{"1min":0.42699999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.64100000000000001,"5min":0.71999999999999997,"15min":0.754},"last":0.379},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.45300000000000001},"min":{"1min":0.32400000000000001,"5min":0.29999999999999999,"15min":0.28299999999999997},"max":{"1min":0.58799999999999997,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.27200000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:06:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.66200000000000003,"15min":0.65700000000000003},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.39400000000000002},"max":{"1min":1.0049999999999999,"5min":1.012,"15min":1.0509999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.66000000000000003,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.91500000000000004,"5min":1.081,"15min":1.081},"last":0.52300000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:06:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.66700000000000004,"15min":0.65300000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.42199999999999999},"max":{"1min":0.93100000000000005,"5min":1.149,"15min":1.149},"last":0.499},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70699999999999996,"15min":0.69799999999999995},"min":{"1min":0.54500000000000004,"5min":0.48399999999999999,"15min":0.45600000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.47599999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:06:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.66400000000000003,"15min":0.65700000000000003},"min":{"1min":0.498,"5min":0.49299999999999999,"15min":0.44700000000000001},"max":{"1min":0.90300000000000002,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71799999999999997,"15min":0.70199999999999996},"min":{"1min":0.57599999999999996,"5min":0.46600000000000003,"15min":0.434},"max":{"1min":1.046,"5min":1.101,"15min":1.101},"last":0.51400000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:06:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.69699999999999995,"15min":0.69899999999999995},"min":{"1min":0.54900000000000004,"5min":0.46600000000000003,"15min":0.44400000000000001},"max":{"1min":0.98799999999999999,"5min":1.141,"15min":1.141},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70999999999999996,"15min":0.69899999999999995},"min":{"1min":0.54600000000000004,"5min":0.46400000000000002,"15min":0.46000000000000002},"max":{"1min":0.97199999999999998,"5min":0.995,"15min":0.995},"last":0.46300000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575763,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42120,"kb_used_data":2812,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329720,"statfs":{"total":96636764160,"available":96593633280,"internally_reserved":0,"allocated":2879488,"data_stored":1739018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.27800000000000002,"5min":0.27500000000000002,"15min":0.24199999999999999},"max":{"1min":0.65600000000000003,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42799999999999999},"min":{"1min":0.28599999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64000000000000001,"5min":0.66800000000000004,"15min":0.67700000000000005},"last":0.32900000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.438,"15min":0.443},"min":{"1min":0.32000000000000001,"5min":0.307,"15min":0.29799999999999999},"max":{"1min":0.55800000000000005,"5min":0.63200000000000001,"15min":0.67300000000000004},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.45800000000000002,"15min":0.46000000000000002},"min":{"1min":0.35599999999999998,"5min":0.308,"15min":0.30599999999999999},"max":{"1min":0.623,"5min":0.65300000000000002,"15min":0.751},"last":0.38300000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.441,"15min":0.435},"min":{"1min":0.317,"5min":0.29199999999999998,"15min":0.27900000000000003},"max":{"1min":0.60899999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.38,"5min":0.307,"15min":0.307},"max":{"1min":0.70599999999999996,"5min":0.73399999999999999,"15min":0.75600000000000001},"last":0.39300000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.67400000000000004,"15min":0.66500000000000004},"min":{"1min":0.51400000000000001,"5min":0.45900000000000002,"15min":0.42899999999999999},"max":{"1min":0.96299999999999997,"5min":1.0109999999999999,"15min":1.044},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.69099999999999995,"15min":0.67800000000000005},"min":{"1min":0.497,"5min":0.45500000000000002,"15min":0.44500000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.78800000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.69699999999999995,"15min":0.68300000000000005},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.44600000000000001},"max":{"1min":0.99399999999999999,"5min":1.0229999999999999,"15min":1.0629999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.70499999999999996,"15min":0.68799999999999994},"min":{"1min":0.52400000000000002,"5min":0.48199999999999998,"15min":0.42299999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.095},"last":0.61499999999999999}]},{"osd":6,"last update":"Mon Mar 18 20:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.68899999999999995,"15min":0.67500000000000004},"min":{"1min":0.52300000000000002,"5min":0.502,"15min":0.46999999999999997},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.749,"5min":0.71799999999999997,"15min":0.71099999999999997},"min":{"1min":0.56200000000000006,"5min":0.50700000000000001,"15min":0.45500000000000002},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.76800000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.72999999999999998,"15min":0.70699999999999996},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.45200000000000001},"max":{"1min":1.008,"5min":1.0600000000000001,"15min":1.075},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.71999999999999997,"15min":0.71099999999999997},"min":{"1min":0.56999999999999995,"5min":0.53000000000000003,"15min":0.45900000000000002},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.85199999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575763,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47276,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324564,"statfs":{"total":96636764160,"available":96588353536,"internally_reserved":0,"allocated":2256896,"data_stored":1139800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:06:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39800000000000002,"15min":0.38900000000000001},"min":{"1min":0.26800000000000002,"5min":0.25600000000000001,"15min":0.23300000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48399999999999999,"15min":0.47599999999999998},"min":{"1min":0.32700000000000001,"5min":0.30499999999999999,"15min":0.218},"max":{"1min":0.71599999999999997,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.34100000000000003}]},{"osd":1,"last update":"Mon Mar 18 20:06:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.49399999999999999,"15min":0.48099999999999998},"min":{"1min":0.34699999999999998,"5min":0.32800000000000001,"15min":0.27900000000000003},"max":{"1min":0.62,"5min":0.749,"15min":0.80100000000000005},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42599999999999999,"15min":0.41799999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.58299999999999996,"5min":0.61699999999999999,"15min":0.70799999999999996},"last":0.254}]},{"osd":3,"last update":"Mon Mar 18 20:06:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.439,"15min":0.42499999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27300000000000002},"max":{"1min":0.60999999999999999,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54200000000000004,"15min":0.52700000000000002},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.33700000000000002},"max":{"1min":0.69799999999999995,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.38300000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:06:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.67200000000000004,"15min":0.66100000000000003},"min":{"1min":0.52400000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":1.042},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66300000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.439,"15min":0.41699999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":1.0529999999999999},"last":0.56999999999999995}]},{"osd":5,"last update":"Mon Mar 18 20:06:04 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.68100000000000005,"15min":0.67400000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.40799999999999997},"max":{"1min":0.98399999999999999,"5min":0.98999999999999999,"15min":1.0369999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.68799999999999994,"15min":0.66900000000000004},"min":{"1min":0.50700000000000001,"5min":0.45300000000000001,"15min":0.41599999999999998},"max":{"1min":0.84499999999999997,"5min":1.04,"15min":1.04},"last":0.60699999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:06:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.68400000000000005,"15min":0.67900000000000005},"min":{"1min":0.54100000000000004,"5min":0.502,"15min":0.39800000000000002},"max":{"1min":0.82499999999999996,"5min":1.004,"15min":1.073},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73999999999999999,"15min":0.72999999999999998},"min":{"1min":0.54800000000000004,"5min":0.52800000000000002,"15min":0.45100000000000001},"max":{"1min":1.036,"5min":1.036,"15min":1.0660000000000001},"last":0.55600000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:06:04 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.69799999999999995,"15min":0.69299999999999995},"min":{"1min":0.58099999999999996,"5min":0.48099999999999998,"15min":0.46000000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.0209999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.71399999999999997,"15min":0.69799999999999995},"min":{"1min":0.51400000000000001,"5min":0.49199999999999999,"15min":0.46700000000000003},"max":{"1min":1.002,"5min":1.002,"15min":1.0109999999999999},"last":0.66400000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575762,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":47216,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324624,"statfs":{"total":96636764160,"available":96588414976,"internally_reserved":0,"allocated":2199552,"data_stored":1113636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36799999999999999,"15min":0.38800000000000001},"min":{"1min":0.27100000000000002,"5min":0.248,"15min":0.23300000000000001},"max":{"1min":0.45400000000000001,"5min":0.66700000000000004,"15min":0.70599999999999996},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40000000000000002,"15min":0.41299999999999998},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.52200000000000002,"5min":0.61399999999999999,"15min":0.78200000000000003},"last":0.35399999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39600000000000002,"15min":0.41299999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.24199999999999999},"max":{"1min":0.58999999999999997,"5min":0.66600000000000004,"15min":0.66800000000000004},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.45200000000000001,"15min":0.46000000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.88900000000000001},"last":0.374}]},{"osd":2,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.44900000000000001,"15min":0.46500000000000002},"min":{"1min":0.31900000000000001,"5min":0.309,"15min":0.29399999999999998},"max":{"1min":0.57999999999999996,"5min":0.64100000000000001,"15min":0.71299999999999997},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48499999999999999,"15min":0.498},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.60599999999999998,"5min":0.72099999999999997,"15min":0.76200000000000001},"last":0.51300000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63900000000000001,"15min":0.64500000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.38800000000000001},"max":{"1min":0.84099999999999997,"5min":0.95599999999999996,"15min":1.0409999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.64500000000000002,"15min":0.65800000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41699999999999998},"max":{"1min":0.88800000000000001,"5min":1.214,"15min":1.214},"last":0.52200000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.67000000000000004,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.85899999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70799999999999996,"15min":0.71399999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.40500000000000003},"max":{"1min":0.90400000000000003,"5min":1.0349999999999999,"15min":1.0660000000000001},"last":0.60899999999999999}]},{"osd":6,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.433},"max":{"1min":0.88900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.70099999999999996,"15min":0.71399999999999997},"min":{"1min":0.48999999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.95199999999999996,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.75700000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.69199999999999995,"15min":0.69899999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":1.02},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.68400000000000005,"15min":0.69099999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92200000000000004,"5min":1.0049999999999999,"15min":1.012},"last":0.69499999999999995}]}]},{"osd":4,"up_from":13,"seq":55834575763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41528,"kb_used_data":2220,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330312,"statfs":{"total":96636764160,"available":96594239488,"internally_reserved":0,"allocated":2273280,"data_stored":1147944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.45400000000000001,"5min":0.36599999999999999,"15min":0.309},"max":{"1min":0.77000000000000002,"5min":1.212,"15min":1.212},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60999999999999999,"15min":0.60499999999999998},"min":{"1min":0.46999999999999997,"5min":0.44,"15min":0.42699999999999999},"max":{"1min":0.79700000000000004,"5min":1.079,"15min":1.079},"last":0.307}]},{"osd":1,"last update":"Mon Mar 18 20:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.63500000000000001,"15min":0.628},"min":{"1min":0.45800000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.83399999999999996,"5min":1.143,"15min":1.143},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60999999999999999,"15min":0.59999999999999998},"min":{"1min":0.498,"5min":0.435,"15min":0.34200000000000003},"max":{"1min":0.78100000000000003,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.40300000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61399999999999999,"15min":0.60799999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35199999999999998},"max":{"1min":0.72999999999999998,"5min":1.246,"15min":1.246},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.626},"min":{"1min":0.437,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75700000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.44700000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.61599999999999999,"15min":0.60499999999999998},"min":{"1min":0.48599999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.749,"5min":1.034,"15min":1.034},"last":0.502},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63800000000000001,"15min":0.629},"min":{"1min":0.48399999999999999,"5min":0.45200000000000001,"15min":0.39900000000000002},"max":{"1min":0.75600000000000001,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.51500000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.48799999999999999,"15min":0.47999999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.60499999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.51700000000000002,"15min":0.501},"min":{"1min":0.308,"5min":0.308,"15min":0.30599999999999999},"max":{"1min":0.58199999999999996,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.46100000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.504,"15min":0.48899999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.308},"max":{"1min":0.54000000000000004,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.53900000000000003,"15min":0.53200000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.60899999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.48899999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.55100000000000005,"15min":0.53900000000000003},"min":{"1min":0.375,"5min":0.35299999999999998,"15min":0.34499999999999997},"max":{"1min":0.63600000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.58299999999999996,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.34499999999999997},"max":{"1min":0.69499999999999995,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.55700000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575761,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":59504,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312336,"statfs":{"total":96636764160,"available":96575832064,"internally_reserved":0,"allocated":2199552,"data_stored":1113636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:06:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.61299999999999999,"15min":0.60199999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34300000000000003},"max":{"1min":0.79000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.61099999999999999,"15min":0.60699999999999998},"min":{"1min":0.46100000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.81799999999999995},"last":0.45200000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:06:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.60199999999999998,"15min":0.60099999999999998},"min":{"1min":0.45100000000000001,"5min":0.41499999999999998,"15min":0.39000000000000001},"max":{"1min":0.82999999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.49399999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73699999999999999,"5min":0.84199999999999997,"15min":0.95199999999999996},"last":0.35699999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:06:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59799999999999998,"15min":0.60099999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.34899999999999998},"max":{"1min":0.72999999999999998,"5min":0.81499999999999995,"15min":0.83099999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.60599999999999998,"15min":0.60899999999999999},"min":{"1min":0.47199999999999998,"5min":0.44500000000000001,"15min":0.36099999999999999},"max":{"1min":0.78200000000000003,"5min":0.82899999999999996,"15min":0.88200000000000001},"last":0.504}]},{"osd":3,"last update":"Mon Mar 18 20:06:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.60599999999999998,"15min":0.61499999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.374},"max":{"1min":0.82299999999999995,"5min":0.88200000000000001,"15min":0.91900000000000004},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65100000000000002,"15min":0.63800000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.79200000000000004,"5min":0.874,"15min":0.874},"last":0.51800000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:06:08 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.49099999999999999,"15min":0.48499999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.28000000000000003},"max":{"1min":0.60799999999999998,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.44},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52200000000000002,"15min":0.51900000000000002},"min":{"1min":0.371,"5min":0.34000000000000002,"15min":0.316},"max":{"1min":0.751,"5min":0.79700000000000004,"15min":0.879},"last":0.48999999999999999}]},{"osd":6,"last update":"Mon Mar 18 20:06:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51500000000000001,"15min":0.50900000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.29699999999999999},"max":{"1min":0.72799999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.33800000000000002},"max":{"1min":0.72799999999999998,"5min":0.76600000000000001,"15min":0.81699999999999995},"last":0.38500000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:06:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56200000000000006,"15min":0.55900000000000005},"min":{"1min":0.34899999999999998,"5min":0.33900000000000002,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.83399999999999996,"15min":0.84499999999999997},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.34300000000000003},"max":{"1min":0.76300000000000001,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.55500000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:06:45.512 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:06:45.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.667+0000 7f65ddbce700 1 Processor -- start 2024-03-18T20:06:45.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.667+0000 7f65ddbce700 1 -- start start 2024-03-18T20:06:45.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.668+0000 7f65ddbce700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65d8143c20 0x7f65d80b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:45.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.668+0000 7f65ddbce700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f65d80b3f40 0x7f65d806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:45.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.668+0000 7f65ddbce700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f65d806f070 0x7f65d8071650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.668+0000 7f65ddbce700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d80b43c0 con 0x7f65d806f070 2024-03-18T20:06:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.668+0000 7f65ddbce700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d80b51b0 con 0x7f65d80b3f40 2024-03-18T20:06:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.668+0000 7f65ddbce700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d806f4f0 con 0x7f65d8143c20 2024-03-18T20:06:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.668+0000 7f65d7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f65d806f070 0x7f65d8071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.668+0000 7f65d7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f65d806f070 0x7f65d8071650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56466/0 (socket says 172.21.15.1:56466) 2024-03-18T20:06:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.668+0000 7f65d7fff700 1 -- 172.21.15.1:0/3227414464 learned_addr learned my addr 172.21.15.1:0/3227414464 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65d6ffd700 1 --2- 172.21.15.1:0/3227414464 >> v2:172.21.15.165:3300/0 conn(0x7f65d80b3f40 0x7f65d806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65d77fe700 1 --2- 172.21.15.1:0/3227414464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65d8143c20 0x7f65d80b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65d7fff700 1 -- 172.21.15.1:0/3227414464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65d8143c20 msgr2=0x7f65d80b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65d7fff700 1 --2- 172.21.15.1:0/3227414464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65d8143c20 0x7f65d80b3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65d7fff700 1 -- 172.21.15.1:0/3227414464 >> v2:172.21.15.165:3300/0 conn(0x7f65d80b3f40 msgr2=0x7f65d806eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65d7fff700 1 --2- 172.21.15.1:0/3227414464 >> v2:172.21.15.165:3300/0 conn(0x7f65d80b3f40 0x7f65d806eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65d7fff700 1 -- 172.21.15.1:0/3227414464 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d8144000 con 0x7f65d806f070 2024-03-18T20:06:45.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65d77fe700 1 --2- 172.21.15.1:0/3227414464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65d8143c20 0x7f65d80b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:45.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65d6ffd700 1 --2- 172.21.15.1:0/3227414464 >> v2:172.21.15.165:3300/0 conn(0x7f65d80b3f40 0x7f65d806eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:45.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65d7fff700 1 --2- 172.21.15.1:0/3227414464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65d806f070 0x7f65d8071650 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f65d8073d40 tx=0x7f65d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70c42e4d7c68c6d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:45.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65d67fc700 1 -- 172.21.15.1:0/3227414464 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d000e070 con 0x7f65d806f070 2024-03-18T20:06:45.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65d67fc700 1 -- 172.21.15.1:0/3227414464 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d000b980 con 0x7f65d806f070 2024-03-18T20:06:45.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65d67fc700 1 -- 172.21.15.1:0/3227414464 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d000bd30 con 0x7f65d806f070 2024-03-18T20:06:45.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65ddbce700 1 -- 172.21.15.1:0/3227414464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65d806f070 msgr2=0x7f65d8071650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:45.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65ddbce700 1 --2- 172.21.15.1:0/3227414464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65d806f070 0x7f65d8071650 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f65d8073d40 tx=0x7f65d000b330 comp rx=0 tx=0).stop 2024-03-18T20:06:45.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65ddbce700 1 -- 172.21.15.1:0/3227414464 shutdown_connections 2024-03-18T20:06:45.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65ddbce700 1 --2- 172.21.15.1:0/3227414464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65d806f070 0x7f65d8071650 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65ddbce700 1 --2- 172.21.15.1:0/3227414464 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65d8143c20 0x7f65d80b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65ddbce700 1 --2- 172.21.15.1:0/3227414464 >> v2:172.21.15.165:3300/0 conn(0x7f65d80b3f40 0x7f65d806eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65ddbce700 1 -- 172.21.15.1:0/3227414464 >> 172.21.15.1:0/3227414464 conn(0x7f65d80b1d20 msgr2=0x7f65d80b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:45.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65ddbce700 1 -- 172.21.15.1:0/3227414464 shutdown_connections 2024-03-18T20:06:45.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.669+0000 7f65ddbce700 1 -- 172.21.15.1:0/3227414464 wait complete. 2024-03-18T20:06:45.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.671+0000 7f65ddbce700 1 Processor -- start 2024-03-18T20:06:45.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.671+0000 7f65ddbce700 1 -- start start 2024-03-18T20:06:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65ddbce700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65d806f070 0x7f65d81f2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65ddbce700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65d80b3f40 0x7f65d80c6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65ddbce700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65d8143c20 0x7f65d80c73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65ddbce700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d80abd90 con 0x7f65d8143c20 2024-03-18T20:06:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65ddbce700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d81f38a0 con 0x7f65d80b3f40 2024-03-18T20:06:45.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65ddbce700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d81f3b80 con 0x7f65d806f070 2024-03-18T20:06:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65d7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65d8143c20 0x7f65d80c73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65d7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65d8143c20 0x7f65d80c73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56476/0 (socket says 172.21.15.1:56476) 2024-03-18T20:06:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65d7fff700 1 -- 172.21.15.1:0/1322789640 learned_addr learned my addr 172.21.15.1:0/1322789640 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65d77fe700 1 --2- 172.21.15.1:0/1322789640 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65d806f070 0x7f65d81f2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65d6ffd700 1 --2- 172.21.15.1:0/1322789640 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65d80b3f40 0x7f65d80c6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65d7fff700 1 -- 172.21.15.1:0/1322789640 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65d806f070 msgr2=0x7f65d81f2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65d7fff700 1 --2- 172.21.15.1:0/1322789640 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65d806f070 0x7f65d81f2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65d7fff700 1 -- 172.21.15.1:0/1322789640 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65d80b3f40 msgr2=0x7f65d80c6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65d7fff700 1 --2- 172.21.15.1:0/1322789640 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65d80b3f40 0x7f65d80c6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65d7fff700 1 -- 172.21.15.1:0/1322789640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d0009b30 con 0x7f65d8143c20 2024-03-18T20:06:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65d7fff700 1 --2- 172.21.15.1:0/1322789640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65d8143c20 0x7f65d80c73b0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f65d0012000 tx=0x7f65d0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65cffff700 1 -- 172.21.15.1:0/1322789640 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d00042d0 con 0x7f65d8143c20 2024-03-18T20:06:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65cffff700 1 -- 172.21.15.1:0/1322789640 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d0004430 con 0x7f65d8143c20 2024-03-18T20:06:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65ddbce700 1 -- 172.21.15.1:0/1322789640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d80abfe0 con 0x7f65d8143c20 2024-03-18T20:06:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65cffff700 1 -- 172.21.15.1:0/1322789640 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d0004820 con 0x7f65d8143c20 2024-03-18T20:06:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.672+0000 7f65ddbce700 1 -- 172.21.15.1:0/1322789640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d80c6370 con 0x7f65d8143c20 2024-03-18T20:06:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.673+0000 7f65ddbce700 1 -- 172.21.15.1:0/1322789640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d80c6500 con 0x7f65d8143c20 2024-03-18T20:06:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.674+0000 7f65cffff700 1 -- 172.21.15.1:0/1322789640 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f65d001c440 con 0x7f65d8143c20 2024-03-18T20:06:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.674+0000 7f65cffff700 1 --2- 172.21.15.1:0/1322789640 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f65b806adb0 0x7f65b806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.674+0000 7f65cffff700 1 -- 172.21.15.1:0/1322789640 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f65d008ad80 con 0x7f65d8143c20 2024-03-18T20:06:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.674+0000 7f65d77fe700 1 --2- 172.21.15.1:0/1322789640 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f65b806adb0 0x7f65b806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:45.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.675+0000 7f65d77fe700 1 --2- 172.21.15.1:0/1322789640 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f65b806adb0 0x7f65b806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f65c0006fd0 tx=0x7f65c0008040 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:45.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.679+0000 7f65cffff700 1 -- 172.21.15.1:0/1322789640 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f65d0055c40 con 0x7f65d8143c20 2024-03-18T20:06:45.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.879+0000 7f65ddbce700 1 -- 172.21.15.1:0/1322789640 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f65d80b45a0 con 0x7f65b806adb0 2024-03-18T20:06:45.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.887+0000 7f65cffff700 1 -- 172.21.15.1:0/1322789640 <== mgr.7721 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289796 (secure 0 0 0) 0x7f65c000add0 con 0x7f65b806adb0 2024-03-18T20:06:45.895 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:45.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.890+0000 7f65ddbce700 1 -- 172.21.15.1:0/1322789640 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f65b806adb0 msgr2=0x7f65b806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:45.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.890+0000 7f65ddbce700 1 --2- 172.21.15.1:0/1322789640 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f65b806adb0 0x7f65b806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f65c0006fd0 tx=0x7f65c0008040 comp rx=0 tx=0).stop 2024-03-18T20:06:45.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.890+0000 7f65ddbce700 1 -- 172.21.15.1:0/1322789640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65d8143c20 msgr2=0x7f65d80c73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:45.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.890+0000 7f65ddbce700 1 --2- 172.21.15.1:0/1322789640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65d8143c20 0x7f65d80c73b0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f65d0012000 tx=0x7f65d0004ad0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.890+0000 7f65ddbce700 1 -- 172.21.15.1:0/1322789640 shutdown_connections 2024-03-18T20:06:45.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.890+0000 7f65ddbce700 1 --2- 172.21.15.1:0/1322789640 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f65b806adb0 0x7f65b806d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.890+0000 7f65ddbce700 1 --2- 172.21.15.1:0/1322789640 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65d806f070 0x7f65d81f2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.890+0000 7f65ddbce700 1 --2- 172.21.15.1:0/1322789640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65d8143c20 0x7f65d80c73b0 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.890+0000 7f65ddbce700 1 --2- 172.21.15.1:0/1322789640 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65d80b3f40 0x7f65d80c6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:45.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.890+0000 7f65ddbce700 1 -- 172.21.15.1:0/1322789640 >> 172.21.15.1:0/1322789640 conn(0x7f65d80b1d20 msgr2=0x7f65d80b03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:45.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.891+0000 7f65ddbce700 1 -- 172.21.15.1:0/1322789640 shutdown_connections 2024-03-18T20:06:45.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:45.891+0000 7f65ddbce700 1 -- 172.21.15.1:0/1322789640 wait complete. 2024-03-18T20:06:45.899 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:06:45.908 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-18T20:06:45.778117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":483,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":364552,"kb_used_data":18828,"kb_used_omap":12,"kb_used_meta":345587,"kb_avail":754610168,"statfs":{"total":773094113280,"available":772720812032,"internally_reserved":0,"allocated":19279872,"data_stored":10323040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":353881657},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.873651"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.373448+0000","last_change":"2024-03-18T20:06:41.370195+0000","last_active":"2024-03-18T20:06:41.373448+0000","last_peered":"2024-03-18T20:06:41.373448+0000","last_clean":"2024-03-18T20:06:41.373448+0000","last_became_active":"2024-03-18T20:06:41.369925+0000","last_became_peered":"2024-03-18T20:06:41.369925+0000","last_unstale":"2024-03-18T20:06:41.373448+0000","last_undegraded":"2024-03-18T20:06:41.373448+0000","last_fullsized":"2024-03-18T20:06:41.373448+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:23:31.228650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.359676+0000","last_change":"2024-03-18T20:06:41.359220+0000","last_active":"2024-03-18T20:06:41.359676+0000","last_peered":"2024-03-18T20:06:41.359676+0000","last_clean":"2024-03-18T20:06:41.359676+0000","last_became_active":"2024-03-18T20:06:41.358926+0000","last_became_peered":"2024-03-18T20:06:41.358926+0000","last_unstale":"2024-03-18T20:06:41.359676+0000","last_undegraded":"2024-03-18T20:06:41.359676+0000","last_fullsized":"2024-03-18T20:06:41.359676+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:51:35.490802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.358313+0000","last_change":"2024-03-18T20:06:41.353598+0000","last_active":"2024-03-18T20:06:41.358313+0000","last_peered":"2024-03-18T20:06:41.358313+0000","last_clean":"2024-03-18T20:06:41.358313+0000","last_became_active":"2024-03-18T20:06:41.353326+0000","last_became_peered":"2024-03-18T20:06:41.353326+0000","last_unstale":"2024-03-18T20:06:41.358313+0000","last_undegraded":"2024-03-18T20:06:41.358313+0000","last_fullsized":"2024-03-18T20:06:41.358313+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:03:49.415865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.348785+0000","last_change":"2024-03-18T20:06:41.365681+0000","last_active":"2024-03-18T20:06:42.348785+0000","last_peered":"2024-03-18T20:06:42.348785+0000","last_clean":"2024-03-18T20:06:42.348785+0000","last_became_active":"2024-03-18T20:06:41.365366+0000","last_became_peered":"2024-03-18T20:06:41.365366+0000","last_unstale":"2024-03-18T20:06:42.348785+0000","last_undegraded":"2024-03-18T20:06:42.348785+0000","last_fullsized":"2024-03-18T20:06:42.348785+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:13.069617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.365446+0000","last_change":"2024-03-18T20:06:41.362689+0000","last_active":"2024-03-18T20:06:41.365446+0000","last_peered":"2024-03-18T20:06:41.365446+0000","last_clean":"2024-03-18T20:06:41.365446+0000","last_became_active":"2024-03-18T20:06:41.361827+0000","last_became_peered":"2024-03-18T20:06:41.361827+0000","last_unstale":"2024-03-18T20:06:41.365446+0000","last_undegraded":"2024-03-18T20:06:41.365446+0000","last_fullsized":"2024-03-18T20:06:41.365446+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:03:52.514953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.363124+0000","last_change":"2024-03-18T20:06:41.361383+0000","last_active":"2024-03-18T20:06:41.363124+0000","last_peered":"2024-03-18T20:06:41.363124+0000","last_clean":"2024-03-18T20:06:41.363124+0000","last_became_active":"2024-03-18T20:06:41.361118+0000","last_became_peered":"2024-03-18T20:06:41.361118+0000","last_unstale":"2024-03-18T20:06:41.363124+0000","last_undegraded":"2024-03-18T20:06:41.363124+0000","last_fullsized":"2024-03-18T20:06:41.363124+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:18:05.961228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.368635+0000","last_change":"2024-03-18T20:06:41.365889+0000","last_active":"2024-03-18T20:06:41.368635+0000","last_peered":"2024-03-18T20:06:41.368635+0000","last_clean":"2024-03-18T20:06:41.368635+0000","last_became_active":"2024-03-18T20:06:41.365599+0000","last_became_peered":"2024-03-18T20:06:41.365599+0000","last_unstale":"2024-03-18T20:06:41.368635+0000","last_undegraded":"2024-03-18T20:06:41.368635+0000","last_fullsized":"2024-03-18T20:06:41.368635+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:01:45.149612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.350053+0000","last_change":"2024-03-18T20:06:41.367557+0000","last_active":"2024-03-18T20:06:42.350053+0000","last_peered":"2024-03-18T20:06:42.350053+0000","last_clean":"2024-03-18T20:06:42.350053+0000","last_became_active":"2024-03-18T20:06:41.366600+0000","last_became_peered":"2024-03-18T20:06:41.366600+0000","last_unstale":"2024-03-18T20:06:42.350053+0000","last_undegraded":"2024-03-18T20:06:42.350053+0000","last_fullsized":"2024-03-18T20:06:42.350053+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:44:10.550192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.372719+0000","last_change":"2024-03-18T20:06:41.370818+0000","last_active":"2024-03-18T20:06:41.372719+0000","last_peered":"2024-03-18T20:06:41.372719+0000","last_clean":"2024-03-18T20:06:41.372719+0000","last_became_active":"2024-03-18T20:06:41.365853+0000","last_became_peered":"2024-03-18T20:06:41.365853+0000","last_unstale":"2024-03-18T20:06:41.372719+0000","last_undegraded":"2024-03-18T20:06:41.372719+0000","last_fullsized":"2024-03-18T20:06:41.372719+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:51:13.127502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.365920+0000","last_change":"2024-03-18T20:06:41.364736+0000","last_active":"2024-03-18T20:06:41.365920+0000","last_peered":"2024-03-18T20:06:41.365920+0000","last_clean":"2024-03-18T20:06:41.365920+0000","last_became_active":"2024-03-18T20:06:41.364350+0000","last_became_peered":"2024-03-18T20:06:41.364350+0000","last_unstale":"2024-03-18T20:06:41.365920+0000","last_undegraded":"2024-03-18T20:06:41.365920+0000","last_fullsized":"2024-03-18T20:06:41.365920+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:31:01.233304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.374015+0000","last_change":"2024-03-18T20:06:41.370507+0000","last_active":"2024-03-18T20:06:41.374015+0000","last_peered":"2024-03-18T20:06:41.374015+0000","last_clean":"2024-03-18T20:06:41.374015+0000","last_became_active":"2024-03-18T20:06:41.370284+0000","last_became_peered":"2024-03-18T20:06:41.370284+0000","last_unstale":"2024-03-18T20:06:41.374015+0000","last_undegraded":"2024-03-18T20:06:41.374015+0000","last_fullsized":"2024-03-18T20:06:41.374015+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:28:07.293987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.348876+0000","last_change":"2024-03-18T20:06:41.361828+0000","last_active":"2024-03-18T20:06:42.348876+0000","last_peered":"2024-03-18T20:06:42.348876+0000","last_clean":"2024-03-18T20:06:42.348876+0000","last_became_active":"2024-03-18T20:06:41.361368+0000","last_became_peered":"2024-03-18T20:06:41.361368+0000","last_unstale":"2024-03-18T20:06:42.348876+0000","last_undegraded":"2024-03-18T20:06:42.348876+0000","last_fullsized":"2024-03-18T20:06:42.348876+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:10.907762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.365465+0000","last_change":"2024-03-18T20:06:41.362853+0000","last_active":"2024-03-18T20:06:41.365465+0000","last_peered":"2024-03-18T20:06:41.365465+0000","last_clean":"2024-03-18T20:06:41.365465+0000","last_became_active":"2024-03-18T20:06:41.362195+0000","last_became_peered":"2024-03-18T20:06:41.362195+0000","last_unstale":"2024-03-18T20:06:41.365465+0000","last_undegraded":"2024-03-18T20:06:41.365465+0000","last_fullsized":"2024-03-18T20:06:41.365465+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:14:38.492230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.348759+0000","last_change":"2024-03-18T20:06:41.362947+0000","last_active":"2024-03-18T20:06:42.348759+0000","last_peered":"2024-03-18T20:06:42.348759+0000","last_clean":"2024-03-18T20:06:42.348759+0000","last_became_active":"2024-03-18T20:06:41.362074+0000","last_became_peered":"2024-03-18T20:06:41.362074+0000","last_unstale":"2024-03-18T20:06:42.348759+0000","last_undegraded":"2024-03-18T20:06:42.348759+0000","last_fullsized":"2024-03-18T20:06:42.348759+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:57.248985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.363605+0000","last_change":"2024-03-18T20:06:41.360933+0000","last_active":"2024-03-18T20:06:41.363605+0000","last_peered":"2024-03-18T20:06:41.363605+0000","last_clean":"2024-03-18T20:06:41.363605+0000","last_became_active":"2024-03-18T20:06:41.360696+0000","last_became_peered":"2024-03-18T20:06:41.360696+0000","last_unstale":"2024-03-18T20:06:41.363605+0000","last_undegraded":"2024-03-18T20:06:41.363605+0000","last_fullsized":"2024-03-18T20:06:41.363605+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:29:17.155958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.371986+0000","last_change":"2024-03-18T20:06:41.370591+0000","last_active":"2024-03-18T20:06:41.371986+0000","last_peered":"2024-03-18T20:06:41.371986+0000","last_clean":"2024-03-18T20:06:41.371986+0000","last_became_active":"2024-03-18T20:06:41.370337+0000","last_became_peered":"2024-03-18T20:06:41.370337+0000","last_unstale":"2024-03-18T20:06:41.371986+0000","last_undegraded":"2024-03-18T20:06:41.371986+0000","last_fullsized":"2024-03-18T20:06:41.371986+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:00:05.960230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.348899+0000","last_change":"2024-03-18T20:06:41.352072+0000","last_active":"2024-03-18T20:06:42.348899+0000","last_peered":"2024-03-18T20:06:42.348899+0000","last_clean":"2024-03-18T20:06:42.348899+0000","last_became_active":"2024-03-18T20:06:41.351789+0000","last_became_peered":"2024-03-18T20:06:41.351789+0000","last_unstale":"2024-03-18T20:06:42.348899+0000","last_undegraded":"2024-03-18T20:06:42.348899+0000","last_fullsized":"2024-03-18T20:06:42.348899+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:47:35.787872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.373869+0000","last_change":"2024-03-18T20:06:41.367241+0000","last_active":"2024-03-18T20:06:41.373869+0000","last_peered":"2024-03-18T20:06:41.373869+0000","last_clean":"2024-03-18T20:06:41.373869+0000","last_became_active":"2024-03-18T20:06:41.366895+0000","last_became_peered":"2024-03-18T20:06:41.366895+0000","last_unstale":"2024-03-18T20:06:41.373869+0000","last_undegraded":"2024-03-18T20:06:41.373869+0000","last_fullsized":"2024-03-18T20:06:41.373869+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:59:08.429068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.369646+0000","last_change":"2024-03-18T20:06:41.369243+0000","last_active":"2024-03-18T20:06:41.369646+0000","last_peered":"2024-03-18T20:06:41.369646+0000","last_clean":"2024-03-18T20:06:41.369646+0000","last_became_active":"2024-03-18T20:06:41.368507+0000","last_became_peered":"2024-03-18T20:06:41.368507+0000","last_unstale":"2024-03-18T20:06:41.369646+0000","last_undegraded":"2024-03-18T20:06:41.369646+0000","last_fullsized":"2024-03-18T20:06:41.369646+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:37:49.042419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.472565+0000","last_change":"2024-03-18T20:06:41.362015+0000","last_active":"2024-03-18T20:06:42.472565+0000","last_peered":"2024-03-18T20:06:42.472565+0000","last_clean":"2024-03-18T20:06:42.472565+0000","last_became_active":"2024-03-18T20:06:41.361780+0000","last_became_peered":"2024-03-18T20:06:41.361780+0000","last_unstale":"2024-03-18T20:06:42.472565+0000","last_undegraded":"2024-03-18T20:06:42.472565+0000","last_fullsized":"2024-03-18T20:06:42.472565+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:25:10.634219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.362000+0000","last_change":"2024-03-18T20:06:41.357938+0000","last_active":"2024-03-18T20:06:41.362000+0000","last_peered":"2024-03-18T20:06:41.362000+0000","last_clean":"2024-03-18T20:06:41.362000+0000","last_became_active":"2024-03-18T20:06:41.357612+0000","last_became_peered":"2024-03-18T20:06:41.357612+0000","last_unstale":"2024-03-18T20:06:41.362000+0000","last_undegraded":"2024-03-18T20:06:41.362000+0000","last_fullsized":"2024-03-18T20:06:41.362000+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:36.230522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.363330+0000","last_change":"2024-03-18T20:06:41.358918+0000","last_active":"2024-03-18T20:06:41.363330+0000","last_peered":"2024-03-18T20:06:41.363330+0000","last_clean":"2024-03-18T20:06:41.363330+0000","last_became_active":"2024-03-18T20:06:41.358640+0000","last_became_peered":"2024-03-18T20:06:41.358640+0000","last_unstale":"2024-03-18T20:06:41.363330+0000","last_undegraded":"2024-03-18T20:06:41.363330+0000","last_fullsized":"2024-03-18T20:06:41.363330+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:05:33.353583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.349812+0000","last_change":"2024-03-18T20:06:41.369369+0000","last_active":"2024-03-18T20:06:42.349812+0000","last_peered":"2024-03-18T20:06:42.349812+0000","last_clean":"2024-03-18T20:06:42.349812+0000","last_became_active":"2024-03-18T20:06:41.369103+0000","last_became_peered":"2024-03-18T20:06:41.369103+0000","last_unstale":"2024-03-18T20:06:42.349812+0000","last_undegraded":"2024-03-18T20:06:42.349812+0000","last_fullsized":"2024-03-18T20:06:42.349812+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:41:54.947369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.373733+0000","last_change":"2024-03-18T20:06:41.364531+0000","last_active":"2024-03-18T20:06:41.373733+0000","last_peered":"2024-03-18T20:06:41.373733+0000","last_clean":"2024-03-18T20:06:41.373733+0000","last_became_active":"2024-03-18T20:06:41.364242+0000","last_became_peered":"2024-03-18T20:06:41.364242+0000","last_unstale":"2024-03-18T20:06:41.373733+0000","last_undegraded":"2024-03-18T20:06:41.373733+0000","last_fullsized":"2024-03-18T20:06:41.373733+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:46.243166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.368072+0000","last_change":"2024-03-18T20:06:41.364304+0000","last_active":"2024-03-18T20:06:41.368072+0000","last_peered":"2024-03-18T20:06:41.368072+0000","last_clean":"2024-03-18T20:06:41.368072+0000","last_became_active":"2024-03-18T20:06:41.364020+0000","last_became_peered":"2024-03-18T20:06:41.364020+0000","last_unstale":"2024-03-18T20:06:41.368072+0000","last_undegraded":"2024-03-18T20:06:41.368072+0000","last_fullsized":"2024-03-18T20:06:41.368072+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:38:28.729138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.373346+0000","last_change":"2024-03-18T20:06:41.369391+0000","last_active":"2024-03-18T20:06:41.373346+0000","last_peered":"2024-03-18T20:06:41.373346+0000","last_clean":"2024-03-18T20:06:41.373346+0000","last_became_active":"2024-03-18T20:06:41.368921+0000","last_became_peered":"2024-03-18T20:06:41.368921+0000","last_unstale":"2024-03-18T20:06:41.373346+0000","last_undegraded":"2024-03-18T20:06:41.373346+0000","last_fullsized":"2024-03-18T20:06:41.373346+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:37:28.168864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.370732+0000","last_change":"2024-03-18T20:06:41.368615+0000","last_active":"2024-03-18T20:06:41.370732+0000","last_peered":"2024-03-18T20:06:41.370732+0000","last_clean":"2024-03-18T20:06:41.370732+0000","last_became_active":"2024-03-18T20:06:41.365724+0000","last_became_peered":"2024-03-18T20:06:41.365724+0000","last_unstale":"2024-03-18T20:06:41.370732+0000","last_undegraded":"2024-03-18T20:06:41.370732+0000","last_fullsized":"2024-03-18T20:06:41.370732+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:45.950076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.374090+0000","last_change":"2024-03-18T20:06:41.371281+0000","last_active":"2024-03-18T20:06:41.374090+0000","last_peered":"2024-03-18T20:06:41.374090+0000","last_clean":"2024-03-18T20:06:41.374090+0000","last_became_active":"2024-03-18T20:06:41.371018+0000","last_became_peered":"2024-03-18T20:06:41.371018+0000","last_unstale":"2024-03-18T20:06:41.374090+0000","last_undegraded":"2024-03-18T20:06:41.374090+0000","last_fullsized":"2024-03-18T20:06:41.374090+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:25:22.714322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.350116+0000","last_change":"2024-03-18T20:06:41.371738+0000","last_active":"2024-03-18T20:06:42.350116+0000","last_peered":"2024-03-18T20:06:42.350116+0000","last_clean":"2024-03-18T20:06:42.350116+0000","last_became_active":"2024-03-18T20:06:41.371403+0000","last_became_peered":"2024-03-18T20:06:41.371403+0000","last_unstale":"2024-03-18T20:06:42.350116+0000","last_undegraded":"2024-03-18T20:06:42.350116+0000","last_fullsized":"2024-03-18T20:06:42.350116+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:38:28.729138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.373695+0000","last_change":"2024-03-18T20:06:41.364374+0000","last_active":"2024-03-18T20:06:41.373695+0000","last_peered":"2024-03-18T20:06:41.373695+0000","last_clean":"2024-03-18T20:06:41.373695+0000","last_became_active":"2024-03-18T20:06:41.364038+0000","last_became_peered":"2024-03-18T20:06:41.364038+0000","last_unstale":"2024-03-18T20:06:41.373695+0000","last_undegraded":"2024-03-18T20:06:41.373695+0000","last_fullsized":"2024-03-18T20:06:41.373695+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:10.907762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.370064+0000","last_change":"2024-03-18T20:06:41.369439+0000","last_active":"2024-03-18T20:06:41.370064+0000","last_peered":"2024-03-18T20:06:41.370064+0000","last_clean":"2024-03-18T20:06:41.370064+0000","last_became_active":"2024-03-18T20:06:41.369291+0000","last_became_peered":"2024-03-18T20:06:41.369291+0000","last_unstale":"2024-03-18T20:06:41.370064+0000","last_undegraded":"2024-03-18T20:06:41.370064+0000","last_fullsized":"2024-03-18T20:06:41.370064+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:03.236875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.349822+0000","last_change":"2024-03-18T20:06:41.367842+0000","last_active":"2024-03-18T20:06:42.349822+0000","last_peered":"2024-03-18T20:06:42.349822+0000","last_clean":"2024-03-18T20:06:42.349822+0000","last_became_active":"2024-03-18T20:06:41.366940+0000","last_became_peered":"2024-03-18T20:06:41.366940+0000","last_unstale":"2024-03-18T20:06:42.349822+0000","last_undegraded":"2024-03-18T20:06:42.349822+0000","last_fullsized":"2024-03-18T20:06:42.349822+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:28:14.508789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.360255+0000","last_change":"2024-03-18T20:06:41.353818+0000","last_active":"2024-03-18T20:06:41.360255+0000","last_peered":"2024-03-18T20:06:41.360255+0000","last_clean":"2024-03-18T20:06:41.360255+0000","last_became_active":"2024-03-18T20:06:41.353373+0000","last_became_peered":"2024-03-18T20:06:41.353373+0000","last_unstale":"2024-03-18T20:06:41.360255+0000","last_undegraded":"2024-03-18T20:06:41.360255+0000","last_fullsized":"2024-03-18T20:06:41.360255+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:45:25.376374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"180'4","reported_seq":14,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.372786+0000","last_change":"2024-03-18T20:06:40.929942+0000","last_active":"2024-03-18T20:06:41.372786+0000","last_peered":"2024-03-18T20:06:41.372786+0000","last_clean":"2024-03-18T20:06:41.372786+0000","last_became_active":"2024-03-18T20:06:40.929621+0000","last_became_peered":"2024-03-18T20:06:40.929621+0000","last_unstale":"2024-03-18T20:06:41.372786+0000","last_undegraded":"2024-03-18T20:06:41.372786+0000","last_fullsized":"2024-03-18T20:06:41.372786+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:33:49.296066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.363668+0000","last_change":"2024-03-18T20:06:41.360831+0000","last_active":"2024-03-18T20:06:41.363668+0000","last_peered":"2024-03-18T20:06:41.363668+0000","last_clean":"2024-03-18T20:06:41.363668+0000","last_became_active":"2024-03-18T20:06:41.360534+0000","last_became_peered":"2024-03-18T20:06:41.360534+0000","last_unstale":"2024-03-18T20:06:41.363668+0000","last_undegraded":"2024-03-18T20:06:41.363668+0000","last_fullsized":"2024-03-18T20:06:41.363668+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:28:45.396835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"180'1","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.364826+0000","last_change":"2024-03-18T20:06:40.933779+0000","last_active":"2024-03-18T20:06:41.364826+0000","last_peered":"2024-03-18T20:06:41.364826+0000","last_clean":"2024-03-18T20:06:41.364826+0000","last_became_active":"2024-03-18T20:06:40.933440+0000","last_became_peered":"2024-03-18T20:06:40.933440+0000","last_unstale":"2024-03-18T20:06:41.364826+0000","last_undegraded":"2024-03-18T20:06:41.364826+0000","last_fullsized":"2024-03-18T20:06:41.364826+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:37:27.163834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.350097+0000","last_change":"2024-03-18T20:06:41.370745+0000","last_active":"2024-03-18T20:06:42.350097+0000","last_peered":"2024-03-18T20:06:42.350097+0000","last_clean":"2024-03-18T20:06:42.350097+0000","last_became_active":"2024-03-18T20:06:41.370430+0000","last_became_peered":"2024-03-18T20:06:41.370430+0000","last_unstale":"2024-03-18T20:06:42.350097+0000","last_undegraded":"2024-03-18T20:06:42.350097+0000","last_fullsized":"2024-03-18T20:06:42.350097+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:37:28.168864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"180'2","reported_seq":12,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.365199+0000","last_change":"2024-03-18T20:06:40.934062+0000","last_active":"2024-03-18T20:06:41.365199+0000","last_peered":"2024-03-18T20:06:41.365199+0000","last_clean":"2024-03-18T20:06:41.365199+0000","last_became_active":"2024-03-18T20:06:40.933790+0000","last_became_peered":"2024-03-18T20:06:40.933790+0000","last_unstale":"2024-03-18T20:06:41.365199+0000","last_undegraded":"2024-03-18T20:06:41.365199+0000","last_fullsized":"2024-03-18T20:06:41.365199+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:32.848530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.364875+0000","last_change":"2024-03-18T20:06:41.360059+0000","last_active":"2024-03-18T20:06:41.364875+0000","last_peered":"2024-03-18T20:06:41.364875+0000","last_clean":"2024-03-18T20:06:41.364875+0000","last_became_active":"2024-03-18T20:06:41.359618+0000","last_became_peered":"2024-03-18T20:06:41.359618+0000","last_unstale":"2024-03-18T20:06:41.364875+0000","last_undegraded":"2024-03-18T20:06:41.364875+0000","last_fullsized":"2024-03-18T20:06:41.364875+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:52:13.527547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"180'2","reported_seq":12,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.363049+0000","last_change":"2024-03-18T20:06:40.931067+0000","last_active":"2024-03-18T20:06:41.363049+0000","last_peered":"2024-03-18T20:06:41.363049+0000","last_clean":"2024-03-18T20:06:41.363049+0000","last_became_active":"2024-03-18T20:06:40.930770+0000","last_became_peered":"2024-03-18T20:06:40.930770+0000","last_unstale":"2024-03-18T20:06:41.363049+0000","last_undegraded":"2024-03-18T20:06:41.363049+0000","last_fullsized":"2024-03-18T20:06:41.363049+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:08.593608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.472597+0000","last_change":"2024-03-18T20:06:41.366528+0000","last_active":"2024-03-18T20:06:42.472597+0000","last_peered":"2024-03-18T20:06:42.472597+0000","last_clean":"2024-03-18T20:06:42.472597+0000","last_became_active":"2024-03-18T20:06:41.366307+0000","last_became_peered":"2024-03-18T20:06:41.366307+0000","last_unstale":"2024-03-18T20:06:42.472597+0000","last_undegraded":"2024-03-18T20:06:42.472597+0000","last_fullsized":"2024-03-18T20:06:42.472597+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:08.081957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"180'1","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.350413+0000","last_change":"2024-03-18T20:06:40.930762+0000","last_active":"2024-03-18T20:06:42.350413+0000","last_peered":"2024-03-18T20:06:42.350413+0000","last_clean":"2024-03-18T20:06:42.350413+0000","last_became_active":"2024-03-18T20:06:40.930511+0000","last_became_peered":"2024-03-18T20:06:40.930511+0000","last_unstale":"2024-03-18T20:06:42.350413+0000","last_undegraded":"2024-03-18T20:06:42.350413+0000","last_fullsized":"2024-03-18T20:06:42.350413+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:45.238136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.348563+0000","last_change":"2024-03-18T20:06:41.352339+0000","last_active":"2024-03-18T20:06:42.348563+0000","last_peered":"2024-03-18T20:06:42.348563+0000","last_clean":"2024-03-18T20:06:42.348563+0000","last_became_active":"2024-03-18T20:06:41.352038+0000","last_became_peered":"2024-03-18T20:06:41.352038+0000","last_unstale":"2024-03-18T20:06:42.348563+0000","last_undegraded":"2024-03-18T20:06:42.348563+0000","last_fullsized":"2024-03-18T20:06:42.348563+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:46.243166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"180'1","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.370966+0000","last_change":"2024-03-18T20:06:40.932432+0000","last_active":"2024-03-18T20:06:41.370966+0000","last_peered":"2024-03-18T20:06:41.370966+0000","last_clean":"2024-03-18T20:06:41.370966+0000","last_became_active":"2024-03-18T20:06:40.932137+0000","last_became_peered":"2024-03-18T20:06:40.932137+0000","last_unstale":"2024-03-18T20:06:41.370966+0000","last_undegraded":"2024-03-18T20:06:41.370966+0000","last_fullsized":"2024-03-18T20:06:41.370966+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:41:23.342658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.358219+0000","last_change":"2024-03-18T20:06:41.352440+0000","last_active":"2024-03-18T20:06:41.358219+0000","last_peered":"2024-03-18T20:06:41.358219+0000","last_clean":"2024-03-18T20:06:41.358219+0000","last_became_active":"2024-03-18T20:06:41.352185+0000","last_became_peered":"2024-03-18T20:06:41.352185+0000","last_unstale":"2024-03-18T20:06:41.358219+0000","last_undegraded":"2024-03-18T20:06:41.358219+0000","last_fullsized":"2024-03-18T20:06:41.358219+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:23:38.989856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.355017+0000","last_change":"2024-03-18T20:06:40.933946+0000","last_active":"2024-03-18T20:06:41.355017+0000","last_peered":"2024-03-18T20:06:41.355017+0000","last_clean":"2024-03-18T20:06:41.355017+0000","last_became_active":"2024-03-18T20:06:40.933640+0000","last_became_peered":"2024-03-18T20:06:40.933640+0000","last_unstale":"2024-03-18T20:06:41.355017+0000","last_undegraded":"2024-03-18T20:06:41.355017+0000","last_fullsized":"2024-03-18T20:06:41.355017+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:24:41.951953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.369607+0000","last_change":"2024-03-18T20:06:41.367784+0000","last_active":"2024-03-18T20:06:41.369607+0000","last_peered":"2024-03-18T20:06:41.369607+0000","last_clean":"2024-03-18T20:06:41.369607+0000","last_became_active":"2024-03-18T20:06:41.367487+0000","last_became_peered":"2024-03-18T20:06:41.367487+0000","last_unstale":"2024-03-18T20:06:41.369607+0000","last_undegraded":"2024-03-18T20:06:41.369607+0000","last_fullsized":"2024-03-18T20:06:41.369607+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:28:14.508789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.350876+0000","last_change":"2024-03-18T20:06:40.925271+0000","last_active":"2024-03-18T20:06:41.350876+0000","last_peered":"2024-03-18T20:06:41.350876+0000","last_clean":"2024-03-18T20:06:41.350876+0000","last_became_active":"2024-03-18T20:06:40.924949+0000","last_became_peered":"2024-03-18T20:06:40.924949+0000","last_unstale":"2024-03-18T20:06:41.350876+0000","last_undegraded":"2024-03-18T20:06:41.350876+0000","last_fullsized":"2024-03-18T20:06:41.350876+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:24.778983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.350309+0000","last_change":"2024-03-18T20:06:41.367875+0000","last_active":"2024-03-18T20:06:42.350309+0000","last_peered":"2024-03-18T20:06:42.350309+0000","last_clean":"2024-03-18T20:06:42.350309+0000","last_became_active":"2024-03-18T20:06:41.367637+0000","last_became_peered":"2024-03-18T20:06:41.367637+0000","last_unstale":"2024-03-18T20:06:42.350309+0000","last_undegraded":"2024-03-18T20:06:42.350309+0000","last_fullsized":"2024-03-18T20:06:42.350309+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:01:45.149612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.345339+0000","last_change":"2024-03-18T20:06:40.927739+0000","last_active":"2024-03-18T20:06:41.345339+0000","last_peered":"2024-03-18T20:06:41.345339+0000","last_clean":"2024-03-18T20:06:41.345339+0000","last_became_active":"2024-03-18T20:06:40.927424+0000","last_became_peered":"2024-03-18T20:06:40.927424+0000","last_unstale":"2024-03-18T20:06:41.345339+0000","last_undegraded":"2024-03-18T20:06:41.345339+0000","last_fullsized":"2024-03-18T20:06:41.345339+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:05:32.348554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.374003+0000","last_change":"2024-03-18T20:06:41.370086+0000","last_active":"2024-03-18T20:06:41.374003+0000","last_peered":"2024-03-18T20:06:41.374003+0000","last_clean":"2024-03-18T20:06:41.374003+0000","last_became_active":"2024-03-18T20:06:41.364823+0000","last_became_peered":"2024-03-18T20:06:41.364823+0000","last_unstale":"2024-03-18T20:06:41.374003+0000","last_undegraded":"2024-03-18T20:06:41.374003+0000","last_fullsized":"2024-03-18T20:06:41.374003+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:59:29.574799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.358201+0000","last_change":"2024-03-18T20:06:40.934702+0000","last_active":"2024-03-18T20:06:41.358201+0000","last_peered":"2024-03-18T20:06:41.358201+0000","last_clean":"2024-03-18T20:06:41.358201+0000","last_became_active":"2024-03-18T20:06:40.934440+0000","last_became_peered":"2024-03-18T20:06:40.934440+0000","last_unstale":"2024-03-18T20:06:41.358201+0000","last_undegraded":"2024-03-18T20:06:41.358201+0000","last_fullsized":"2024-03-18T20:06:41.358201+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:06:24.101460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.373798+0000","last_change":"2024-03-18T20:06:41.367037+0000","last_active":"2024-03-18T20:06:41.373798+0000","last_peered":"2024-03-18T20:06:41.373798+0000","last_clean":"2024-03-18T20:06:41.373798+0000","last_became_active":"2024-03-18T20:06:41.366675+0000","last_became_peered":"2024-03-18T20:06:41.366675+0000","last_unstale":"2024-03-18T20:06:41.373798+0000","last_undegraded":"2024-03-18T20:06:41.373798+0000","last_fullsized":"2024-03-18T20:06:41.373798+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:24:42.956982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"180'3","reported_seq":13,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.367778+0000","last_change":"2024-03-18T20:06:40.929699+0000","last_active":"2024-03-18T20:06:41.367778+0000","last_peered":"2024-03-18T20:06:41.367778+0000","last_clean":"2024-03-18T20:06:41.367778+0000","last_became_active":"2024-03-18T20:06:40.926591+0000","last_became_peered":"2024-03-18T20:06:40.926591+0000","last_unstale":"2024-03-18T20:06:41.367778+0000","last_undegraded":"2024-03-18T20:06:41.367778+0000","last_fullsized":"2024-03-18T20:06:41.367778+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:40.459736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.349975+0000","last_change":"2024-03-18T20:06:41.369513+0000","last_active":"2024-03-18T20:06:42.349975+0000","last_peered":"2024-03-18T20:06:42.349975+0000","last_clean":"2024-03-18T20:06:42.349975+0000","last_became_active":"2024-03-18T20:06:41.367288+0000","last_became_peered":"2024-03-18T20:06:41.367288+0000","last_unstale":"2024-03-18T20:06:42.349975+0000","last_undegraded":"2024-03-18T20:06:42.349975+0000","last_fullsized":"2024-03-18T20:06:42.349975+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:33.853559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"180'2","reported_seq":12,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.369855+0000","last_change":"2024-03-18T20:06:40.924973+0000","last_active":"2024-03-18T20:06:41.369855+0000","last_peered":"2024-03-18T20:06:41.369855+0000","last_clean":"2024-03-18T20:06:41.369855+0000","last_became_active":"2024-03-18T20:06:40.924659+0000","last_became_peered":"2024-03-18T20:06:40.924659+0000","last_unstale":"2024-03-18T20:06:41.369855+0000","last_undegraded":"2024-03-18T20:06:41.369855+0000","last_fullsized":"2024-03-18T20:06:41.369855+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:42:48.989865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.363586+0000","last_change":"2024-03-18T20:06:41.359898+0000","last_active":"2024-03-18T20:06:41.363586+0000","last_peered":"2024-03-18T20:06:41.363586+0000","last_clean":"2024-03-18T20:06:41.363586+0000","last_became_active":"2024-03-18T20:06:41.359646+0000","last_became_peered":"2024-03-18T20:06:41.359646+0000","last_unstale":"2024-03-18T20:06:41.363586+0000","last_undegraded":"2024-03-18T20:06:41.363586+0000","last_fullsized":"2024-03-18T20:06:41.363586+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:28:34.744775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.348858+0000","last_change":"2024-03-18T20:06:40.931811+0000","last_active":"2024-03-18T20:06:42.348858+0000","last_peered":"2024-03-18T20:06:42.348858+0000","last_clean":"2024-03-18T20:06:42.348858+0000","last_became_active":"2024-03-18T20:06:40.931556+0000","last_became_peered":"2024-03-18T20:06:40.931556+0000","last_unstale":"2024-03-18T20:06:42.348858+0000","last_undegraded":"2024-03-18T20:06:42.348858+0000","last_fullsized":"2024-03-18T20:06:42.348858+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:20.769001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.372779+0000","last_change":"2024-03-18T20:06:41.370825+0000","last_active":"2024-03-18T20:06:41.372779+0000","last_peered":"2024-03-18T20:06:41.372779+0000","last_clean":"2024-03-18T20:06:41.372779+0000","last_became_active":"2024-03-18T20:06:41.365779+0000","last_became_peered":"2024-03-18T20:06:41.365779+0000","last_unstale":"2024-03-18T20:06:41.372779+0000","last_undegraded":"2024-03-18T20:06:41.372779+0000","last_fullsized":"2024-03-18T20:06:41.372779+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:26:55.229872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.348733+0000","last_change":"2024-03-18T20:06:40.929730+0000","last_active":"2024-03-18T20:06:42.348733+0000","last_peered":"2024-03-18T20:06:42.348733+0000","last_clean":"2024-03-18T20:06:42.348733+0000","last_became_active":"2024-03-18T20:06:40.929455+0000","last_became_peered":"2024-03-18T20:06:40.929455+0000","last_unstale":"2024-03-18T20:06:42.348733+0000","last_undegraded":"2024-03-18T20:06:42.348733+0000","last_fullsized":"2024-03-18T20:06:42.348733+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:20:20.338062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.365419+0000","last_change":"2024-03-18T20:06:41.361564+0000","last_active":"2024-03-18T20:06:41.365419+0000","last_peered":"2024-03-18T20:06:41.365419+0000","last_clean":"2024-03-18T20:06:41.365419+0000","last_became_active":"2024-03-18T20:06:41.361155+0000","last_became_peered":"2024-03-18T20:06:41.361155+0000","last_unstale":"2024-03-18T20:06:41.365419+0000","last_undegraded":"2024-03-18T20:06:41.365419+0000","last_fullsized":"2024-03-18T20:06:41.365419+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:05.115393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"180'8","reported_seq":18,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.370276+0000","last_change":"2024-03-18T20:06:40.932391+0000","last_active":"2024-03-18T20:06:41.370276+0000","last_peered":"2024-03-18T20:06:41.370276+0000","last_clean":"2024-03-18T20:06:41.370276+0000","last_became_active":"2024-03-18T20:06:40.932086+0000","last_became_peered":"2024-03-18T20:06:40.932086+0000","last_unstale":"2024-03-18T20:06:41.370276+0000","last_undegraded":"2024-03-18T20:06:41.370276+0000","last_fullsized":"2024-03-18T20:06:41.370276+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:58:29.971935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.364902+0000","last_change":"2024-03-18T20:06:41.364174+0000","last_active":"2024-03-18T20:06:41.364902+0000","last_peered":"2024-03-18T20:06:41.364902+0000","last_clean":"2024-03-18T20:06:41.364902+0000","last_became_active":"2024-03-18T20:06:41.363875+0000","last_became_peered":"2024-03-18T20:06:41.363875+0000","last_unstale":"2024-03-18T20:06:41.364902+0000","last_undegraded":"2024-03-18T20:06:41.364902+0000","last_fullsized":"2024-03-18T20:06:41.364902+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:48.634167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"180'1","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.366559+0000","last_change":"2024-03-18T20:06:40.930476+0000","last_active":"2024-03-18T20:06:41.366559+0000","last_peered":"2024-03-18T20:06:41.366559+0000","last_clean":"2024-03-18T20:06:41.366559+0000","last_became_active":"2024-03-18T20:06:40.930198+0000","last_became_peered":"2024-03-18T20:06:40.930198+0000","last_unstale":"2024-03-18T20:06:41.366559+0000","last_undegraded":"2024-03-18T20:06:41.366559+0000","last_fullsized":"2024-03-18T20:06:41.366559+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:27:19.221024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.370516+0000","last_change":"2024-03-18T20:06:41.369419+0000","last_active":"2024-03-18T20:06:41.370516+0000","last_peered":"2024-03-18T20:06:41.370516+0000","last_clean":"2024-03-18T20:06:41.370516+0000","last_became_active":"2024-03-18T20:06:41.368840+0000","last_became_peered":"2024-03-18T20:06:41.368840+0000","last_unstale":"2024-03-18T20:06:41.370516+0000","last_undegraded":"2024-03-18T20:06:41.370516+0000","last_fullsized":"2024-03-18T20:06:41.370516+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:51:57.729525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"180'2","reported_seq":12,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.363953+0000","last_change":"2024-03-18T20:06:40.931433+0000","last_active":"2024-03-18T20:06:41.363953+0000","last_peered":"2024-03-18T20:06:41.363953+0000","last_clean":"2024-03-18T20:06:41.363953+0000","last_became_active":"2024-03-18T20:06:40.931159+0000","last_became_peered":"2024-03-18T20:06:40.931159+0000","last_unstale":"2024-03-18T20:06:41.363953+0000","last_undegraded":"2024-03-18T20:06:41.363953+0000","last_fullsized":"2024-03-18T20:06:41.363953+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:58.234326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.369720+0000","last_change":"2024-03-18T20:06:41.366736+0000","last_active":"2024-03-18T20:06:41.369720+0000","last_peered":"2024-03-18T20:06:41.369720+0000","last_clean":"2024-03-18T20:06:41.369720+0000","last_became_active":"2024-03-18T20:06:41.366436+0000","last_became_peered":"2024-03-18T20:06:41.366436+0000","last_unstale":"2024-03-18T20:06:41.369720+0000","last_undegraded":"2024-03-18T20:06:41.369720+0000","last_fullsized":"2024-03-18T20:06:41.369720+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:32.684095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"180'1","reported_seq":11,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.360577+0000","last_change":"2024-03-18T20:06:40.932926+0000","last_active":"2024-03-18T20:06:41.360577+0000","last_peered":"2024-03-18T20:06:41.360577+0000","last_clean":"2024-03-18T20:06:41.360577+0000","last_became_active":"2024-03-18T20:06:40.932626+0000","last_became_peered":"2024-03-18T20:06:40.932626+0000","last_unstale":"2024-03-18T20:06:41.360577+0000","last_undegraded":"2024-03-18T20:06:41.360577+0000","last_fullsized":"2024-03-18T20:06:41.360577+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:47.629138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.373735+0000","last_change":"2024-03-18T20:06:41.372386+0000","last_active":"2024-03-18T20:06:41.373735+0000","last_peered":"2024-03-18T20:06:41.373735+0000","last_clean":"2024-03-18T20:06:41.373735+0000","last_became_active":"2024-03-18T20:06:41.372100+0000","last_became_peered":"2024-03-18T20:06:41.372100+0000","last_unstale":"2024-03-18T20:06:41.373735+0000","last_undegraded":"2024-03-18T20:06:41.373735+0000","last_fullsized":"2024-03-18T20:06:41.373735+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:59.239355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.348883+0000","last_change":"2024-03-18T20:06:40.932442+0000","last_active":"2024-03-18T20:06:42.348883+0000","last_peered":"2024-03-18T20:06:42.348883+0000","last_clean":"2024-03-18T20:06:42.348883+0000","last_became_active":"2024-03-18T20:06:40.932159+0000","last_became_peered":"2024-03-18T20:06:40.932159+0000","last_unstale":"2024-03-18T20:06:42.348883+0000","last_undegraded":"2024-03-18T20:06:42.348883+0000","last_fullsized":"2024-03-18T20:06:42.348883+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:52.804403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.374207+0000","last_change":"2024-03-18T20:06:41.373675+0000","last_active":"2024-03-18T20:06:41.374207+0000","last_peered":"2024-03-18T20:06:41.374207+0000","last_clean":"2024-03-18T20:06:41.374207+0000","last_became_active":"2024-03-18T20:06:41.373382+0000","last_became_peered":"2024-03-18T20:06:41.373382+0000","last_unstale":"2024-03-18T20:06:41.374207+0000","last_undegraded":"2024-03-18T20:06:41.374207+0000","last_fullsized":"2024-03-18T20:06:41.374207+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:06:22.367037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"180'2","reported_seq":13,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.472494+0000","last_change":"2024-03-18T20:06:40.937596+0000","last_active":"2024-03-18T20:06:42.472494+0000","last_peered":"2024-03-18T20:06:42.472494+0000","last_clean":"2024-03-18T20:06:42.472494+0000","last_became_active":"2024-03-18T20:06:40.935870+0000","last_became_peered":"2024-03-18T20:06:40.935870+0000","last_unstale":"2024-03-18T20:06:42.472494+0000","last_undegraded":"2024-03-18T20:06:42.472494+0000","last_fullsized":"2024-03-18T20:06:42.472494+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:37.006175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.372713+0000","last_change":"2024-03-18T20:06:41.364920+0000","last_active":"2024-03-18T20:06:41.372713+0000","last_peered":"2024-03-18T20:06:41.372713+0000","last_clean":"2024-03-18T20:06:41.372713+0000","last_became_active":"2024-03-18T20:06:41.364665+0000","last_became_peered":"2024-03-18T20:06:41.364665+0000","last_unstale":"2024-03-18T20:06:41.372713+0000","last_undegraded":"2024-03-18T20:06:41.372713+0000","last_fullsized":"2024-03-18T20:06:41.372713+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:26.254209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.349874+0000","last_change":"2024-03-18T20:06:40.930701+0000","last_active":"2024-03-18T20:06:42.349874+0000","last_peered":"2024-03-18T20:06:42.349874+0000","last_clean":"2024-03-18T20:06:42.349874+0000","last_became_active":"2024-03-18T20:06:40.930424+0000","last_became_peered":"2024-03-18T20:06:40.930424+0000","last_unstale":"2024-03-18T20:06:42.349874+0000","last_undegraded":"2024-03-18T20:06:42.349874+0000","last_fullsized":"2024-03-18T20:06:42.349874+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:24:41.951953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.367665+0000","last_change":"2024-03-18T20:06:41.365985+0000","last_active":"2024-03-18T20:06:41.367665+0000","last_peered":"2024-03-18T20:06:41.367665+0000","last_clean":"2024-03-18T20:06:41.367665+0000","last_became_active":"2024-03-18T20:06:41.365707+0000","last_became_peered":"2024-03-18T20:06:41.365707+0000","last_unstale":"2024-03-18T20:06:41.367665+0000","last_undegraded":"2024-03-18T20:06:41.367665+0000","last_fullsized":"2024-03-18T20:06:41.367665+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:24.751000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.472412+0000","last_change":"2024-03-18T20:06:40.937204+0000","last_active":"2024-03-18T20:06:42.472412+0000","last_peered":"2024-03-18T20:06:42.472412+0000","last_clean":"2024-03-18T20:06:42.472412+0000","last_became_active":"2024-03-18T20:06:40.936078+0000","last_became_peered":"2024-03-18T20:06:40.936078+0000","last_unstale":"2024-03-18T20:06:42.472412+0000","last_undegraded":"2024-03-18T20:06:42.472412+0000","last_fullsized":"2024-03-18T20:06:42.472412+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:17:41.767913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.365386+0000","last_change":"2024-03-18T20:06:41.361411+0000","last_active":"2024-03-18T20:06:41.365386+0000","last_peered":"2024-03-18T20:06:41.365386+0000","last_clean":"2024-03-18T20:06:41.365386+0000","last_became_active":"2024-03-18T20:06:41.360784+0000","last_became_peered":"2024-03-18T20:06:41.360784+0000","last_unstale":"2024-03-18T20:06:41.365386+0000","last_undegraded":"2024-03-18T20:06:41.365386+0000","last_fullsized":"2024-03-18T20:06:41.365386+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:16:52.447786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":11,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.349925+0000","last_change":"2024-03-18T20:06:40.939149+0000","last_active":"2024-03-18T20:06:42.349925+0000","last_peered":"2024-03-18T20:06:42.349925+0000","last_clean":"2024-03-18T20:06:42.349925+0000","last_became_active":"2024-03-18T20:06:40.938934+0000","last_became_peered":"2024-03-18T20:06:40.938934+0000","last_unstale":"2024-03-18T20:06:42.349925+0000","last_undegraded":"2024-03-18T20:06:42.349925+0000","last_fullsized":"2024-03-18T20:06:42.349925+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:25:21.709293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.370939+0000","last_change":"2024-03-18T20:06:41.370241+0000","last_active":"2024-03-18T20:06:41.370939+0000","last_peered":"2024-03-18T20:06:41.370939+0000","last_clean":"2024-03-18T20:06:41.370939+0000","last_became_active":"2024-03-18T20:06:41.365913+0000","last_became_peered":"2024-03-18T20:06:41.365913+0000","last_unstale":"2024-03-18T20:06:41.370939+0000","last_undegraded":"2024-03-18T20:06:41.370939+0000","last_fullsized":"2024-03-18T20:06:41.370939+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:14:21.434238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"180'2","reported_seq":13,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.349024+0000","last_change":"2024-03-18T20:06:40.933408+0000","last_active":"2024-03-18T20:06:42.349024+0000","last_peered":"2024-03-18T20:06:42.349024+0000","last_clean":"2024-03-18T20:06:42.349024+0000","last_became_active":"2024-03-18T20:06:40.932950+0000","last_became_peered":"2024-03-18T20:06:40.932950+0000","last_unstale":"2024-03-18T20:06:42.349024+0000","last_undegraded":"2024-03-18T20:06:42.349024+0000","last_fullsized":"2024-03-18T20:06:42.349024+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:33.586280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.349042+0000","last_change":"2024-03-18T20:06:41.362325+0000","last_active":"2024-03-18T20:06:42.349042+0000","last_peered":"2024-03-18T20:06:42.349042+0000","last_clean":"2024-03-18T20:06:42.349042+0000","last_became_active":"2024-03-18T20:06:41.362006+0000","last_became_peered":"2024-03-18T20:06:41.362006+0000","last_unstale":"2024-03-18T20:06:42.349042+0000","last_undegraded":"2024-03-18T20:06:42.349042+0000","last_fullsized":"2024-03-18T20:06:42.349042+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:10:58.991209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.358098+0000","last_change":"2024-03-18T20:06:40.931201+0000","last_active":"2024-03-18T20:06:41.358098+0000","last_peered":"2024-03-18T20:06:41.358098+0000","last_clean":"2024-03-18T20:06:41.358098+0000","last_became_active":"2024-03-18T20:06:40.930943+0000","last_became_peered":"2024-03-18T20:06:40.930943+0000","last_unstale":"2024-03-18T20:06:41.358098+0000","last_undegraded":"2024-03-18T20:06:41.358098+0000","last_fullsized":"2024-03-18T20:06:41.358098+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:40.307734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.369741+0000","last_change":"2024-03-18T20:06:41.367294+0000","last_active":"2024-03-18T20:06:41.369741+0000","last_peered":"2024-03-18T20:06:41.369741+0000","last_clean":"2024-03-18T20:06:41.369741+0000","last_became_active":"2024-03-18T20:06:41.367008+0000","last_became_peered":"2024-03-18T20:06:41.367008+0000","last_unstale":"2024-03-18T20:06:41.369741+0000","last_undegraded":"2024-03-18T20:06:41.369741+0000","last_fullsized":"2024-03-18T20:06:41.369741+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:33.853559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.346768+0000","last_change":"2024-03-18T20:06:40.931183+0000","last_active":"2024-03-18T20:06:41.346768+0000","last_peered":"2024-03-18T20:06:41.346768+0000","last_clean":"2024-03-18T20:06:41.346768+0000","last_became_active":"2024-03-18T20:06:40.930896+0000","last_became_peered":"2024-03-18T20:06:40.930896+0000","last_unstale":"2024-03-18T20:06:41.346768+0000","last_undegraded":"2024-03-18T20:06:41.346768+0000","last_fullsized":"2024-03-18T20:06:41.346768+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:00:22.450064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.364104+0000","last_change":"2024-03-18T20:06:41.359577+0000","last_active":"2024-03-18T20:06:41.364104+0000","last_peered":"2024-03-18T20:06:41.364104+0000","last_clean":"2024-03-18T20:06:41.364104+0000","last_became_active":"2024-03-18T20:06:41.359313+0000","last_became_peered":"2024-03-18T20:06:41.359313+0000","last_unstale":"2024-03-18T20:06:41.364104+0000","last_undegraded":"2024-03-18T20:06:41.364104+0000","last_fullsized":"2024-03-18T20:06:41.364104+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:31:02.619139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.356931+0000","last_change":"2024-03-18T20:06:40.931948+0000","last_active":"2024-03-18T20:06:41.356931+0000","last_peered":"2024-03-18T20:06:41.356931+0000","last_clean":"2024-03-18T20:06:41.356931+0000","last_became_active":"2024-03-18T20:06:40.931634+0000","last_became_peered":"2024-03-18T20:06:40.931634+0000","last_unstale":"2024-03-18T20:06:41.356931+0000","last_undegraded":"2024-03-18T20:06:41.356931+0000","last_fullsized":"2024-03-18T20:06:41.356931+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:36:40.784732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.349650+0000","last_change":"2024-03-18T20:06:41.370421+0000","last_active":"2024-03-18T20:06:42.349650+0000","last_peered":"2024-03-18T20:06:42.349650+0000","last_clean":"2024-03-18T20:06:42.349650+0000","last_became_active":"2024-03-18T20:06:41.370105+0000","last_became_peered":"2024-03-18T20:06:41.370105+0000","last_unstale":"2024-03-18T20:06:42.349650+0000","last_undegraded":"2024-03-18T20:06:42.349650+0000","last_fullsized":"2024-03-18T20:06:42.349650+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:09.598637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.352758+0000","last_change":"2024-03-18T20:06:40.934521+0000","last_active":"2024-03-18T20:06:41.352758+0000","last_peered":"2024-03-18T20:06:41.352758+0000","last_clean":"2024-03-18T20:06:41.352758+0000","last_became_active":"2024-03-18T20:06:40.934259+0000","last_became_peered":"2024-03-18T20:06:40.934259+0000","last_unstale":"2024-03-18T20:06:41.352758+0000","last_undegraded":"2024-03-18T20:06:41.352758+0000","last_fullsized":"2024-03-18T20:06:41.352758+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:25:21.709293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.348630+0000","last_change":"2024-03-18T20:06:41.357946+0000","last_active":"2024-03-18T20:06:42.348630+0000","last_peered":"2024-03-18T20:06:42.348630+0000","last_clean":"2024-03-18T20:06:42.348630+0000","last_became_active":"2024-03-18T20:06:41.357682+0000","last_became_peered":"2024-03-18T20:06:41.357682+0000","last_unstale":"2024-03-18T20:06:42.348630+0000","last_undegraded":"2024-03-18T20:06:42.348630+0000","last_fullsized":"2024-03-18T20:06:42.348630+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:59:08.429068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.356036+0000","last_change":"2024-03-18T20:06:40.930876+0000","last_active":"2024-03-18T20:06:41.356036+0000","last_peered":"2024-03-18T20:06:41.356036+0000","last_clean":"2024-03-18T20:06:41.356036+0000","last_became_active":"2024-03-18T20:06:40.930606+0000","last_became_peered":"2024-03-18T20:06:40.930606+0000","last_unstale":"2024-03-18T20:06:41.356036+0000","last_undegraded":"2024-03-18T20:06:41.356036+0000","last_fullsized":"2024-03-18T20:06:41.356036+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:20:20.338062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.372567+0000","last_change":"2024-03-18T20:06:41.369577+0000","last_active":"2024-03-18T20:06:41.372567+0000","last_peered":"2024-03-18T20:06:41.372567+0000","last_clean":"2024-03-18T20:06:41.372567+0000","last_became_active":"2024-03-18T20:06:41.369332+0000","last_became_peered":"2024-03-18T20:06:41.369332+0000","last_unstale":"2024-03-18T20:06:41.372567+0000","last_undegraded":"2024-03-18T20:06:41.372567+0000","last_fullsized":"2024-03-18T20:06:41.372567+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:17:59.998734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.355121+0000","last_change":"2024-03-18T20:06:40.934224+0000","last_active":"2024-03-18T20:06:41.355121+0000","last_peered":"2024-03-18T20:06:41.355121+0000","last_clean":"2024-03-18T20:06:41.355121+0000","last_became_active":"2024-03-18T20:06:40.933941+0000","last_became_peered":"2024-03-18T20:06:40.933941+0000","last_unstale":"2024-03-18T20:06:41.355121+0000","last_undegraded":"2024-03-18T20:06:41.355121+0000","last_fullsized":"2024-03-18T20:06:41.355121+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:05:42.283464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.350214+0000","last_change":"2024-03-18T20:06:41.371561+0000","last_active":"2024-03-18T20:06:42.350214+0000","last_peered":"2024-03-18T20:06:42.350214+0000","last_clean":"2024-03-18T20:06:42.350214+0000","last_became_active":"2024-03-18T20:06:41.371307+0000","last_became_peered":"2024-03-18T20:06:41.371307+0000","last_unstale":"2024-03-18T20:06:42.350214+0000","last_undegraded":"2024-03-18T20:06:42.350214+0000","last_fullsized":"2024-03-18T20:06:42.350214+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:23:31.228650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":10,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-18T20:06:41.345969+0000","last_change":"2024-03-18T20:06:40.927231+0000","last_active":"2024-03-18T20:06:41.345969+0000","last_peered":"2024-03-18T20:06:41.345969+0000","last_clean":"2024-03-18T20:06:41.345969+0000","last_became_active":"2024-03-18T20:06:40.926933+0000","last_became_peered":"2024-03-18T20:06:40.926933+0000","last_unstale":"2024-03-18T20:06:41.345969+0000","last_undegraded":"2024-03-18T20:06:41.345969+0000","last_fullsized":"2024-03-18T20:06:41.345969+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:01:44.144583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":10,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.350106+0000","last_change":"2024-03-18T20:06:41.367390+0000","last_active":"2024-03-18T20:06:42.350106+0000","last_peered":"2024-03-18T20:06:42.350106+0000","last_clean":"2024-03-18T20:06:42.350106+0000","last_became_active":"2024-03-18T20:06:41.367077+0000","last_became_peered":"2024-03-18T20:06:41.367077+0000","last_unstale":"2024-03-18T20:06:42.350106+0000","last_undegraded":"2024-03-18T20:06:42.350106+0000","last_fullsized":"2024-03-18T20:06:42.350106+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:40.904511+0000","last_clean_scrub_stamp":"2024-03-18T20:06:40.904511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:59.239355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"180'6","reported_seq":17,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-18T20:06:42.472662+0000","last_change":"2024-03-18T20:06:40.937199+0000","last_active":"2024-03-18T20:06:42.472662+0000","last_peered":"2024-03-18T20:06:42.472662+0000","last_clean":"2024-03-18T20:06:42.472662+0000","last_became_active":"2024-03-18T20:06:40.935955+0000","last_became_peered":"2024-03-18T20:06:40.935955+0000","last_unstale":"2024-03-18T20:06:42.472662+0000","last_undegraded":"2024-03-18T20:06:42.472662+0000","last_fullsized":"2024-03-18T20:06:42.472662+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:06:39.899482+0000","last_clean_scrub_stamp":"2024-03-18T20:06:39.899482+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:25:32.038941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"170'271","reported_seq":875,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-18T20:06:40.920985+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:06:40.920985+0000","last_peered":"2024-03-18T20:06:40.920985+0000","last_clean":"2024-03-18T20:06:40.920985+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:06:40.920985+0000","last_undegraded":"2024-03-18T20:06:40.920985+0000","last_fullsized":"2024-03-18T20:06:40.920985+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575762,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47848,"kb_used_data":2780,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323992,"statfs":{"total":96636764160,"available":96587767808,"internally_reserved":0,"allocated":2846720,"data_stored":1730078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.32500000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.77000000000000002,"5min":0.83999999999999997,"15min":1.1259999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.61899999999999999},"min":{"1min":0.41099999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.77400000000000002,"5min":0.81699999999999995,"15min":1.1379999999999999},"last":0.39600000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.82799999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.61799999999999999,"15min":0.61599999999999999},"min":{"1min":0.37,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79900000000000004,"5min":0.83899999999999997,"15min":1.1719999999999999},"last":0.52300000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.38500000000000001,"5min":0.36499999999999999,"15min":0.33900000000000002},"max":{"1min":0.77700000000000002,"5min":0.872,"15min":0.872},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.42699999999999999,"5min":0.41899999999999998,"15min":0.39200000000000002},"max":{"1min":0.82199999999999995,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.372}]},{"osd":3,"last update":"Mon Mar 18 20:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.61699999999999999,"15min":0.61399999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81399999999999995,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.64000000000000001,"15min":0.63200000000000001},"min":{"1min":0.45100000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.89000000000000001},"last":0.435}]},{"osd":4,"last update":"Mon Mar 18 20:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47399999999999998,"15min":0.46999999999999997},"min":{"1min":0.33000000000000002,"5min":0.32500000000000001,"15min":0.315},"max":{"1min":0.73599999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.70099999999999996,"5min":0.76100000000000001,"15min":0.84699999999999998},"last":0.41599999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.502,"15min":0.50700000000000001},"min":{"1min":0.374,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.375,"5min":0.35299999999999998,"15min":0.32900000000000001},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.55800000000000005}]},{"osd":6,"last update":"Mon Mar 18 20:06:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56799999999999995,"15min":0.56000000000000005},"min":{"1min":0.41299999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80400000000000005,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.55700000000000005,"15min":0.55500000000000005},"min":{"1min":0.41099999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.91300000000000003},"last":0.54600000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575762,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37432,"kb_used_data":2212,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334408,"statfs":{"total":96636764160,"available":96598433792,"internally_reserved":0,"allocated":2265088,"data_stored":1146450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.61299999999999999,"15min":0.59599999999999997},"min":{"1min":0.40300000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73899999999999999,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.60499999999999998,"15min":0.59999999999999998},"min":{"1min":0.371,"5min":0.36099999999999999,"15min":0.30299999999999999},"max":{"1min":0.70499999999999996,"5min":1.054,"15min":1.054},"last":0.63500000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60799999999999998,"15min":0.59599999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.34000000000000002},"max":{"1min":0.76700000000000002,"5min":0.82799999999999996,"15min":0.84599999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.61399999999999999,"15min":0.59999999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.28299999999999997},"max":{"1min":0.72999999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.42899999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.64100000000000001,"15min":0.625},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.40600000000000003},"max":{"1min":0.74299999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60399999999999998,"15min":0.59599999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.71799999999999997,"5min":0.79800000000000004,"15min":0.89600000000000002},"last":0.66800000000000004}]},{"osd":3,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.64200000000000002,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73399999999999999,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60799999999999998,"15min":0.60099999999999998},"min":{"1min":0.42399999999999999,"5min":0.41599999999999998,"15min":0.39500000000000002},"max":{"1min":0.72099999999999997,"5min":1.448,"15min":1.448},"last":0.60199999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.47499999999999998},"min":{"1min":0.318,"5min":0.316,"15min":0.316},"max":{"1min":0.69299999999999995,"5min":0.78000000000000003,"15min":0.82999999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.505,"15min":0.50700000000000001},"min":{"1min":0.372,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.65900000000000003,"5min":0.84399999999999997,"15min":0.86499999999999999},"last":0.68999999999999995}]},{"osd":5,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.495,"15min":0.48999999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.63800000000000001,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54500000000000004,"15min":0.53300000000000003},"min":{"1min":0.43099999999999999,"5min":0.38200000000000001,"15min":0.33900000000000002},"max":{"1min":0.71199999999999997,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.624}]},{"osd":7,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.39600000000000002,"5min":0.378,"15min":0.34300000000000003},"max":{"1min":0.71899999999999997,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56200000000000006,"15min":0.54700000000000004},"min":{"1min":0.45000000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67900000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.68000000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575762,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41516,"kb_used_data":2200,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330324,"statfs":{"total":96636764160,"available":96594251776,"internally_reserved":0,"allocated":2252800,"data_stored":1139758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:06:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40500000000000003,"15min":0.40100000000000002},"min":{"1min":0.26000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.54500000000000004,"5min":0.59299999999999997,"15min":0.63600000000000001},"last":0.317},{"interface":"front","average":{"1min":0.432,"5min":0.434,"15min":0.44400000000000001},"min":{"1min":0.29599999999999999,"5min":0.27500000000000002,"15min":0.26200000000000001},"max":{"1min":0.56000000000000005,"5min":0.71099999999999997,"15min":0.73599999999999999},"last":0.30199999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:06:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.25600000000000001},"max":{"1min":0.56100000000000005,"5min":0.66500000000000004,"15min":0.70299999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.47599999999999998,"15min":0.47299999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.60899999999999999,"5min":0.77500000000000002,"15min":0.79300000000000004},"last":0.34300000000000003}]},{"osd":3,"last update":"Mon Mar 18 20:06:00 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.497,"15min":0.497},"min":{"1min":0.42699999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.64100000000000001,"5min":0.71999999999999997,"15min":0.754},"last":0.379},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.45300000000000001},"min":{"1min":0.32400000000000001,"5min":0.29999999999999999,"15min":0.28299999999999997},"max":{"1min":0.58799999999999997,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.27200000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:06:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.66200000000000003,"15min":0.65700000000000003},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.39400000000000002},"max":{"1min":1.0049999999999999,"5min":1.012,"15min":1.0509999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.66000000000000003,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.91500000000000004,"5min":1.081,"15min":1.081},"last":0.52300000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:06:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.66700000000000004,"15min":0.65300000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.42199999999999999},"max":{"1min":0.93100000000000005,"5min":1.149,"15min":1.149},"last":0.499},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70699999999999996,"15min":0.69799999999999995},"min":{"1min":0.54500000000000004,"5min":0.48399999999999999,"15min":0.45600000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.47599999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:06:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.66400000000000003,"15min":0.65700000000000003},"min":{"1min":0.498,"5min":0.49299999999999999,"15min":0.44700000000000001},"max":{"1min":0.90300000000000002,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71799999999999997,"15min":0.70199999999999996},"min":{"1min":0.57599999999999996,"5min":0.46600000000000003,"15min":0.434},"max":{"1min":1.046,"5min":1.101,"15min":1.101},"last":0.51400000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:06:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.69699999999999995,"15min":0.69899999999999995},"min":{"1min":0.54900000000000004,"5min":0.46600000000000003,"15min":0.44400000000000001},"max":{"1min":0.98799999999999999,"5min":1.141,"15min":1.141},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70999999999999996,"15min":0.69899999999999995},"min":{"1min":0.54600000000000004,"5min":0.46400000000000002,"15min":0.46000000000000002},"max":{"1min":0.97199999999999998,"5min":0.995,"15min":0.995},"last":0.46300000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575763,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42120,"kb_used_data":2812,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329720,"statfs":{"total":96636764160,"available":96593633280,"internally_reserved":0,"allocated":2879488,"data_stored":1739018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.27800000000000002,"5min":0.27500000000000002,"15min":0.24199999999999999},"max":{"1min":0.65600000000000003,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42799999999999999},"min":{"1min":0.28599999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64000000000000001,"5min":0.66800000000000004,"15min":0.67700000000000005},"last":0.32900000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.438,"15min":0.443},"min":{"1min":0.32000000000000001,"5min":0.307,"15min":0.29799999999999999},"max":{"1min":0.55800000000000005,"5min":0.63200000000000001,"15min":0.67300000000000004},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.45800000000000002,"15min":0.46000000000000002},"min":{"1min":0.35599999999999998,"5min":0.308,"15min":0.30599999999999999},"max":{"1min":0.623,"5min":0.65300000000000002,"15min":0.751},"last":0.38300000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.441,"15min":0.435},"min":{"1min":0.317,"5min":0.29199999999999998,"15min":0.27900000000000003},"max":{"1min":0.60899999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.38,"5min":0.307,"15min":0.307},"max":{"1min":0.70599999999999996,"5min":0.73399999999999999,"15min":0.75600000000000001},"last":0.39300000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.67400000000000004,"15min":0.66500000000000004},"min":{"1min":0.51400000000000001,"5min":0.45900000000000002,"15min":0.42899999999999999},"max":{"1min":0.96299999999999997,"5min":1.0109999999999999,"15min":1.044},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.69099999999999995,"15min":0.67800000000000005},"min":{"1min":0.497,"5min":0.45500000000000002,"15min":0.44500000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.78800000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.69699999999999995,"15min":0.68300000000000005},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.44600000000000001},"max":{"1min":0.99399999999999999,"5min":1.0229999999999999,"15min":1.0629999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.70499999999999996,"15min":0.68799999999999994},"min":{"1min":0.52400000000000002,"5min":0.48199999999999998,"15min":0.42299999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.095},"last":0.61499999999999999}]},{"osd":6,"last update":"Mon Mar 18 20:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.68899999999999995,"15min":0.67500000000000004},"min":{"1min":0.52300000000000002,"5min":0.502,"15min":0.46999999999999997},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.749,"5min":0.71799999999999997,"15min":0.71099999999999997},"min":{"1min":0.56200000000000006,"5min":0.50700000000000001,"15min":0.45500000000000002},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.76800000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.72999999999999998,"15min":0.70699999999999996},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.45200000000000001},"max":{"1min":1.008,"5min":1.0600000000000001,"15min":1.075},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.71999999999999997,"15min":0.71099999999999997},"min":{"1min":0.56999999999999995,"5min":0.53000000000000003,"15min":0.45900000000000002},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.85199999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575763,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47276,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324564,"statfs":{"total":96636764160,"available":96588353536,"internally_reserved":0,"allocated":2256896,"data_stored":1139800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:06:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.39800000000000002,"15min":0.38900000000000001},"min":{"1min":0.26800000000000002,"5min":0.25600000000000001,"15min":0.23300000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48399999999999999,"15min":0.47599999999999998},"min":{"1min":0.32700000000000001,"5min":0.30499999999999999,"15min":0.218},"max":{"1min":0.71599999999999997,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.34100000000000003}]},{"osd":1,"last update":"Mon Mar 18 20:06:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.49399999999999999,"15min":0.48099999999999998},"min":{"1min":0.34699999999999998,"5min":0.32800000000000001,"15min":0.27900000000000003},"max":{"1min":0.62,"5min":0.749,"15min":0.80100000000000005},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42599999999999999,"15min":0.41799999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.58299999999999996,"5min":0.61699999999999999,"15min":0.70799999999999996},"last":0.254}]},{"osd":3,"last update":"Mon Mar 18 20:06:04 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.439,"15min":0.42499999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27300000000000002},"max":{"1min":0.60999999999999999,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54200000000000004,"15min":0.52700000000000002},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.33700000000000002},"max":{"1min":0.69799999999999995,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.38300000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:06:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.67200000000000004,"15min":0.66100000000000003},"min":{"1min":0.52400000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":1.042},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66300000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.439,"15min":0.41699999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":1.0529999999999999},"last":0.56999999999999995}]},{"osd":5,"last update":"Mon Mar 18 20:06:04 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.68100000000000005,"15min":0.67400000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.40799999999999997},"max":{"1min":0.98399999999999999,"5min":0.98999999999999999,"15min":1.0369999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.68799999999999994,"15min":0.66900000000000004},"min":{"1min":0.50700000000000001,"5min":0.45300000000000001,"15min":0.41599999999999998},"max":{"1min":0.84499999999999997,"5min":1.04,"15min":1.04},"last":0.60699999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:06:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.68400000000000005,"15min":0.67900000000000005},"min":{"1min":0.54100000000000004,"5min":0.502,"15min":0.39800000000000002},"max":{"1min":0.82499999999999996,"5min":1.004,"15min":1.073},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73999999999999999,"15min":0.72999999999999998},"min":{"1min":0.54800000000000004,"5min":0.52800000000000002,"15min":0.45100000000000001},"max":{"1min":1.036,"5min":1.036,"15min":1.0660000000000001},"last":0.55600000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:06:04 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.69799999999999995,"15min":0.69299999999999995},"min":{"1min":0.58099999999999996,"5min":0.48099999999999998,"15min":0.46000000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.0209999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.71399999999999997,"15min":0.69799999999999995},"min":{"1min":0.51400000000000001,"5min":0.49199999999999999,"15min":0.46700000000000003},"max":{"1min":1.002,"5min":1.002,"15min":1.0109999999999999},"last":0.66400000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575763,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47276,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324564,"statfs":{"total":96636764160,"available":96588353536,"internally_reserved":0,"allocated":2256896,"data_stored":1140304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36799999999999999,"15min":0.38800000000000001},"min":{"1min":0.27100000000000002,"5min":0.248,"15min":0.23300000000000001},"max":{"1min":0.45400000000000001,"5min":0.66700000000000004,"15min":0.70599999999999996},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40000000000000002,"15min":0.41299999999999998},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.52200000000000002,"5min":0.61399999999999999,"15min":0.78200000000000003},"last":0.33700000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39600000000000002,"15min":0.41299999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.24199999999999999},"max":{"1min":0.58999999999999997,"5min":0.66600000000000004,"15min":0.66800000000000004},"last":0.314},{"interface":"front","average":{"1min":0.438,"5min":0.45200000000000001,"15min":0.46000000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.88900000000000001},"last":0.433}]},{"osd":2,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.44900000000000001,"15min":0.46500000000000002},"min":{"1min":0.31900000000000001,"5min":0.309,"15min":0.29399999999999998},"max":{"1min":0.57999999999999996,"5min":0.64100000000000001,"15min":0.71299999999999997},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48499999999999999,"15min":0.498},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.60599999999999998,"5min":0.72099999999999997,"15min":0.76200000000000001},"last":0.40400000000000003}]},{"osd":4,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63900000000000001,"15min":0.64500000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.38800000000000001},"max":{"1min":0.84099999999999997,"5min":0.95599999999999996,"15min":1.0409999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.64500000000000002,"15min":0.65800000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41699999999999998},"max":{"1min":0.88800000000000001,"5min":1.214,"15min":1.214},"last":0.69199999999999995}]},{"osd":5,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.67000000000000004,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.85899999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.751},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70799999999999996,"15min":0.71399999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.40500000000000003},"max":{"1min":0.90400000000000003,"5min":1.0349999999999999,"15min":1.0660000000000001},"last":0.77300000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.433},"max":{"1min":0.88900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.70099999999999996,"15min":0.71399999999999997},"min":{"1min":0.48999999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.95199999999999996,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.78800000000000003}]},{"osd":7,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.69199999999999995,"15min":0.69899999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":1.02},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.68400000000000005,"15min":0.69099999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92200000000000004,"5min":1.0049999999999999,"15min":1.012},"last":0.68300000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41528,"kb_used_data":2220,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330312,"statfs":{"total":96636764160,"available":96594239488,"internally_reserved":0,"allocated":2273280,"data_stored":1147944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.45400000000000001,"5min":0.36599999999999999,"15min":0.309},"max":{"1min":0.77000000000000002,"5min":1.212,"15min":1.212},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60999999999999999,"15min":0.60499999999999998},"min":{"1min":0.46999999999999997,"5min":0.44,"15min":0.42699999999999999},"max":{"1min":0.79700000000000004,"5min":1.079,"15min":1.079},"last":0.307}]},{"osd":1,"last update":"Mon Mar 18 20:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.63500000000000001,"15min":0.628},"min":{"1min":0.45800000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.83399999999999996,"5min":1.143,"15min":1.143},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60999999999999999,"15min":0.59999999999999998},"min":{"1min":0.498,"5min":0.435,"15min":0.34200000000000003},"max":{"1min":0.78100000000000003,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.40300000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61399999999999999,"15min":0.60799999999999998},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35199999999999998},"max":{"1min":0.72999999999999998,"5min":1.246,"15min":1.246},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.626},"min":{"1min":0.437,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75700000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.44700000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.61599999999999999,"15min":0.60499999999999998},"min":{"1min":0.48599999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.749,"5min":1.034,"15min":1.034},"last":0.502},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63800000000000001,"15min":0.629},"min":{"1min":0.48399999999999999,"5min":0.45200000000000001,"15min":0.39900000000000002},"max":{"1min":0.75600000000000001,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.51500000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.48799999999999999,"15min":0.47999999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.60499999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.51700000000000002,"15min":0.501},"min":{"1min":0.308,"5min":0.308,"15min":0.30599999999999999},"max":{"1min":0.58199999999999996,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.46100000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.504,"15min":0.48899999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.308},"max":{"1min":0.54000000000000004,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.53900000000000003,"15min":0.53200000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.60899999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.48899999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.55100000000000005,"15min":0.53900000000000003},"min":{"1min":0.375,"5min":0.35299999999999998,"15min":0.34499999999999997},"max":{"1min":0.63600000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.58299999999999996,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.34499999999999997},"max":{"1min":0.69499999999999995,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.55700000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575762,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59556,"kb_used_data":2196,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312284,"statfs":{"total":96636764160,"available":96575778816,"internally_reserved":0,"allocated":2248704,"data_stored":1139688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:06:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.61299999999999999,"15min":0.60199999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.34300000000000003},"max":{"1min":0.79000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.61099999999999999,"15min":0.60699999999999998},"min":{"1min":0.46100000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.81799999999999995},"last":0.45200000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:06:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.60199999999999998,"15min":0.60099999999999998},"min":{"1min":0.45100000000000001,"5min":0.41499999999999998,"15min":0.39000000000000001},"max":{"1min":0.82999999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.621,"15min":0.61799999999999999},"min":{"1min":0.49399999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73699999999999999,"5min":0.84199999999999997,"15min":0.95199999999999996},"last":0.35699999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:06:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59799999999999998,"15min":0.60099999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.34899999999999998},"max":{"1min":0.72999999999999998,"5min":0.81499999999999995,"15min":0.83099999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.60599999999999998,"15min":0.60899999999999999},"min":{"1min":0.47199999999999998,"5min":0.44500000000000001,"15min":0.36099999999999999},"max":{"1min":0.78200000000000003,"5min":0.82899999999999996,"15min":0.88200000000000001},"last":0.504}]},{"osd":3,"last update":"Mon Mar 18 20:06:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.60599999999999998,"15min":0.61499999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.374},"max":{"1min":0.82299999999999995,"5min":0.88200000000000001,"15min":0.91900000000000004},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65100000000000002,"15min":0.63800000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.79200000000000004,"5min":0.874,"15min":0.874},"last":0.51800000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:06:08 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.49099999999999999,"15min":0.48499999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.28000000000000003},"max":{"1min":0.60799999999999998,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.44},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52200000000000002,"15min":0.51900000000000002},"min":{"1min":0.371,"5min":0.34000000000000002,"15min":0.316},"max":{"1min":0.751,"5min":0.79700000000000004,"15min":0.879},"last":0.48999999999999999}]},{"osd":6,"last update":"Mon Mar 18 20:06:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51500000000000001,"15min":0.50900000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.29699999999999999},"max":{"1min":0.72799999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.33800000000000002},"max":{"1min":0.72799999999999998,"5min":0.76600000000000001,"15min":0.81699999999999995},"last":0.38500000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:06:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56200000000000006,"15min":0.55900000000000005},"min":{"1min":0.34899999999999998,"5min":0.33900000000000002,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.83399999999999996,"15min":0.84499999999999997},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.34300000000000003},"max":{"1min":0.76300000000000001,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.55500000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:06:45.912 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T20:06:45.913 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-18T20:06:46.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.076+0000 7f59018ca700 1 Processor -- start 2024-03-18T20:06:46.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.076+0000 7f59018ca700 1 -- start start 2024-03-18T20:06:46.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.077+0000 7f59018ca700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f58fc0b4810 0x7f58fc0d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:46.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.077+0000 7f59018ca700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f58fc0cde70 0x7f58fc0ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:46.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.077+0000 7f59018ca700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58fc0b3370 con 0x7f58fc0cfb50 2024-03-18T20:06:46.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.077+0000 7f59018ca700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58fc0b2760 con 0x7f58fc0cde70 2024-03-18T20:06:46.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.077+0000 7f59018ca700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58fc0b2a40 con 0x7f58fc0b4810 2024-03-18T20:06:46.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.077+0000 7f58fb7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f58fc0cde70 0x7f58fc0ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:46.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.077+0000 7f58faffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f58fc0cfb50 0x7f58fc0cff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49172/0 (socket says 172.21.15.1:49172) 2024-03-18T20:06:46.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.077+0000 7f58fb7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f58fc0cde70 0x7f58fc0ce2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56488/0 (socket says 172.21.15.1:56488) 2024-03-18T20:06:46.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.077+0000 7f58faffd700 1 -- 172.21.15.1:0/74160628 learned_addr learned my addr 172.21.15.1:0/74160628 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:46.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.077+0000 7f58fa7fc700 1 --2- 172.21.15.1:0/74160628 >> v2:172.21.15.165:3300/0 conn(0x7f58fc0b4810 0x7f58fc0d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:46.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.077+0000 7f58fb7fe700 1 -- 172.21.15.1:0/74160628 >> v2:172.21.15.165:3300/0 conn(0x7f58fc0b4810 msgr2=0x7f58fc0d3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:46.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.077+0000 7f58fb7fe700 1 --2- 172.21.15.1:0/74160628 >> v2:172.21.15.165:3300/0 conn(0x7f58fc0b4810 0x7f58fc0d3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.077+0000 7f58fb7fe700 1 -- 172.21.15.1:0/74160628 >> v1:172.21.15.1:6789/0 conn(0x7f58fc0cfb50 legacy=0x7f58fc0cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:46.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.077+0000 7f58fb7fe700 1 -- 172.21.15.1:0/74160628 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58fc0b2d20 con 0x7f58fc0cde70 2024-03-18T20:06:46.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.077+0000 7f58f9ffb700 1 -- 172.21.15.1:0/74160628 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174385267 0 0) 0x7f58fc0b3370 con 0x7f58fc0cfb50 2024-03-18T20:06:46.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.077+0000 7f58fa7fc700 1 --2- 172.21.15.1:0/74160628 >> v2:172.21.15.165:3300/0 conn(0x7f58fc0b4810 0x7f58fc0d3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:46.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.077+0000 7f58fb7fe700 1 --2- 172.21.15.1:0/74160628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58fc0cde70 0x7f58fc0ce2c0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f58fc08d8b0 tx=0x7f58f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b12b605c68dbf329 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:46.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.077+0000 7f58f9ffb700 1 -- 172.21.15.1:0/74160628 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f400e070 con 0x7f58fc0cde70 2024-03-18T20:06:46.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.078+0000 7f58f9ffb700 1 -- 172.21.15.1:0/74160628 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f400bbd0 con 0x7f58fc0cde70 2024-03-18T20:06:46.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.078+0000 7f58f9ffb700 1 -- 172.21.15.1:0/74160628 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f4012430 con 0x7f58fc0cde70 2024-03-18T20:06:46.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.078+0000 7f59018ca700 1 -- 172.21.15.1:0/74160628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58fc0cde70 msgr2=0x7f58fc0ce2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:46.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.078+0000 7f59018ca700 1 --2- 172.21.15.1:0/74160628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58fc0cde70 0x7f58fc0ce2c0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f58fc08d8b0 tx=0x7f58f400b330 comp rx=0 tx=0).stop 2024-03-18T20:06:46.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.078+0000 7f59018ca700 1 -- 172.21.15.1:0/74160628 shutdown_connections 2024-03-18T20:06:46.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.078+0000 7f59018ca700 1 --2- 172.21.15.1:0/74160628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58fc0cde70 0x7f58fc0ce2c0 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.078+0000 7f59018ca700 1 --2- 172.21.15.1:0/74160628 >> v2:172.21.15.165:3300/0 conn(0x7f58fc0b4810 0x7f58fc0d3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.078+0000 7f59018ca700 1 -- 172.21.15.1:0/74160628 >> 172.21.15.1:0/74160628 conn(0x7f58fc0d3140 msgr2=0x7f58fc0d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:46.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.078+0000 7f59018ca700 1 -- 172.21.15.1:0/74160628 shutdown_connections 2024-03-18T20:06:46.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.078+0000 7f59018ca700 1 -- 172.21.15.1:0/74160628 wait complete. 2024-03-18T20:06:46.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.079+0000 7f59018ca700 1 Processor -- start 2024-03-18T20:06:46.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.079+0000 7f59018ca700 1 -- start start 2024-03-18T20:06:46.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.080+0000 7f59018ca700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58fc0b4810 0x7f58fc15a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:46.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.080+0000 7f59018ca700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f58fc0cde70 0x7f58fc0b0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:46.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.080+0000 7f59018ca700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58fc0cfb50 0x7f58fc0b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:46.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.080+0000 7f59018ca700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc0cbc70 con 0x7f58fc0cfb50 2024-03-18T20:06:46.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.080+0000 7f59018ca700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc0cbf50 con 0x7f58fc0b4810 2024-03-18T20:06:46.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.080+0000 7f59018ca700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc0cc230 con 0x7f58fc0cde70 2024-03-18T20:06:46.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.080+0000 7f58fb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58fc0cfb50 0x7f58fc0b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:46.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.080+0000 7f58fb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58fc0cfb50 0x7f58fc0b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56490/0 (socket says 172.21.15.1:56490) 2024-03-18T20:06:46.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.080+0000 7f58fb7fe700 1 -- 172.21.15.1:0/3104338455 learned_addr learned my addr 172.21.15.1:0/3104338455 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:46.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.080+0000 7f58faffd700 1 --2- 172.21.15.1:0/3104338455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58fc0b4810 0x7f58fc15a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:46.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.080+0000 7f58fa7fc700 1 --2- 172.21.15.1:0/3104338455 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f58fc0cde70 0x7f58fc0b0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:46.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.081+0000 7f58fb7fe700 1 -- 172.21.15.1:0/3104338455 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f58fc0cde70 msgr2=0x7f58fc0b0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:46.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.081+0000 7f58fb7fe700 1 --2- 172.21.15.1:0/3104338455 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f58fc0cde70 0x7f58fc0b0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.081+0000 7f58fb7fe700 1 -- 172.21.15.1:0/3104338455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58fc0b4810 msgr2=0x7f58fc15a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:46.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.081+0000 7f58fb7fe700 1 --2- 172.21.15.1:0/3104338455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58fc0b4810 0x7f58fc15a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.081+0000 7f58fb7fe700 1 -- 172.21.15.1:0/3104338455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f4009a70 con 0x7f58fc0cfb50 2024-03-18T20:06:46.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.081+0000 7f58fb7fe700 1 --2- 172.21.15.1:0/3104338455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58fc0cfb50 0x7f58fc0b12c0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f58fc08d8e0 tx=0x7f58f4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:46.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.081+0000 7f58e3fff700 1 -- 172.21.15.1:0/3104338455 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f4012430 con 0x7f58fc0cfb50 2024-03-18T20:06:46.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.081+0000 7f58e3fff700 1 -- 172.21.15.1:0/3104338455 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f4020070 con 0x7f58fc0cfb50 2024-03-18T20:06:46.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.081+0000 7f59018ca700 1 -- 172.21.15.1:0/3104338455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58fc0cc510 con 0x7f58fc0cfb50 2024-03-18T20:06:46.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.081+0000 7f58e3fff700 1 -- 172.21.15.1:0/3104338455 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f4004280 con 0x7f58fc0cfb50 2024-03-18T20:06:46.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.081+0000 7f59018ca700 1 -- 172.21.15.1:0/3104338455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58fc15ab70 con 0x7f58fc0cfb50 2024-03-18T20:06:46.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.082+0000 7f59018ca700 1 -- 172.21.15.1:0/3104338455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58fc0cca30 con 0x7f58fc0cfb50 2024-03-18T20:06:46.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.082+0000 7f58e3fff700 1 -- 172.21.15.1:0/3104338455 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f58f40043e0 con 0x7f58fc0cfb50 2024-03-18T20:06:46.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.083+0000 7f58e3fff700 1 --2- 172.21.15.1:0/3104338455 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f58e806adb0 0x7f58e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:46.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.083+0000 7f58e3fff700 1 -- 172.21.15.1:0/3104338455 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f58f4022070 con 0x7f58fc0cfb50 2024-03-18T20:06:46.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.083+0000 7f58faffd700 1 --2- 172.21.15.1:0/3104338455 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f58e806adb0 0x7f58e806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:46.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.083+0000 7f58faffd700 1 --2- 172.21.15.1:0/3104338455 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f58e806adb0 0x7f58e806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f58f0007fd0 tx=0x7f58f0009040 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:46.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.087+0000 7f58e3fff700 1 -- 172.21.15.1:0/3104338455 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f58fc0cca30 con 0x7f58fc0cfb50 2024-03-18T20:06:46.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.375+0000 7f59018ca700 1 -- 172.21.15.1:0/3104338455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f58fc0cca30 con 0x7f58fc0cfb50 2024-03-18T20:06:46.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.378+0000 7f58e3fff700 1 -- 172.21.15.1:0/3104338455 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v42) v1 ==== 190+0+0 (secure 0 0 0) 0x7f58fc0cca30 con 0x7f58fc0cfb50 2024-03-18T20:06:46.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.380+0000 7f59018ca700 1 -- 172.21.15.1:0/3104338455 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f58e806adb0 msgr2=0x7f58e806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:46.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.380+0000 7f59018ca700 1 --2- 172.21.15.1:0/3104338455 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f58e806adb0 0x7f58e806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f58f0007fd0 tx=0x7f58f0009040 comp rx=0 tx=0).stop 2024-03-18T20:06:46.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.380+0000 7f59018ca700 1 -- 172.21.15.1:0/3104338455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58fc0cfb50 msgr2=0x7f58fc0b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:46.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.380+0000 7f59018ca700 1 --2- 172.21.15.1:0/3104338455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58fc0cfb50 0x7f58fc0b12c0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f58fc08d8e0 tx=0x7f58f4004ad0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.380+0000 7f59018ca700 1 -- 172.21.15.1:0/3104338455 shutdown_connections 2024-03-18T20:06:46.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.380+0000 7f59018ca700 1 --2- 172.21.15.1:0/3104338455 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f58e806adb0 0x7f58e806d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.380+0000 7f59018ca700 1 --2- 172.21.15.1:0/3104338455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58fc0cfb50 0x7f58fc0b12c0 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.380+0000 7f59018ca700 1 --2- 172.21.15.1:0/3104338455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58fc0b4810 0x7f58fc15a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.381+0000 7f59018ca700 1 --2- 172.21.15.1:0/3104338455 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f58fc0cde70 0x7f58fc0b0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.381+0000 7f59018ca700 1 -- 172.21.15.1:0/3104338455 >> 172.21.15.1:0/3104338455 conn(0x7f58fc0d3140 msgr2=0x7f58fc0b54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:46.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.381+0000 7f59018ca700 1 -- 172.21.15.1:0/3104338455 shutdown_connections 2024-03-18T20:06:46.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.381+0000 7f59018ca700 1 -- 172.21.15.1:0/3104338455 wait complete. 2024-03-18T20:06:46.387 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-18T20:06:46.397 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:06:46.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.518+0000 7fcad6929700 1 Processor -- start 2024-03-18T20:06:46.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.518+0000 7fcad6929700 1 -- start start 2024-03-18T20:06:46.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.518+0000 7fcad6929700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcad0143c20 0x7fcad00b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:46.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.518+0000 7fcad6929700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcad006f240 0x7fcad00cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:46.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.518+0000 7fcad6929700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad0144000 con 0x7fcad00b3f40 2024-03-18T20:06:46.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.518+0000 7fcad6929700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad00b43c0 con 0x7fcad006f240 2024-03-18T20:06:46.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.518+0000 7fcad6929700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad00b51b0 con 0x7fcad0143c20 2024-03-18T20:06:46.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.519+0000 7fcacffff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcad0143c20 0x7fcad00b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:46.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.519+0000 7fcad4ec6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcad006f240 0x7fcad00cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:46.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.519+0000 7fcacf7fe700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fcad00b3f40 0x7fcad006eb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:53654/0 (socket says 172.21.15.1:53654) 2024-03-18T20:06:46.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.519+0000 7fcacffff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcad0143c20 0x7fcad00b3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:59950/0 (socket says 172.21.15.1:59950) 2024-03-18T20:06:46.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.519+0000 7fcacf7fe700 1 -- 172.21.15.1:0/3968782895 learned_addr learned my addr 172.21.15.1:0/3968782895 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:46.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.519+0000 7fcad4ec6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcad006f240 0x7fcad00cddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:56630/0 (socket says 172.21.15.1:56630) 2024-03-18T20:06:46.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.519+0000 7fcacffff700 1 -- 172.21.15.1:0/3968782895 >> v2:172.21.15.165:3300/0 conn(0x7fcad006f240 msgr2=0x7fcad00cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:46.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.519+0000 7fcacffff700 1 --2- 172.21.15.1:0/3968782895 >> v2:172.21.15.165:3300/0 conn(0x7fcad006f240 0x7fcad00cddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.519+0000 7fcacffff700 1 -- 172.21.15.1:0/3968782895 >> v1:172.21.15.165:6789/0 conn(0x7fcad00b3f40 legacy=0x7fcad006eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.519+0000 7fcacffff700 1 -- 172.21.15.1:0/3968782895 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad00cd8d0 con 0x7fcad0143c20 2024-03-18T20:06:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.519+0000 7fcad4ec6700 1 --2- 172.21.15.1:0/3968782895 >> v2:172.21.15.165:3300/0 conn(0x7fcad006f240 0x7fcad00cddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.520+0000 7fcacffff700 1 --2- 172.21.15.1:0/3968782895 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcad0143c20 0x7fcad00b3a00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcab8009840 tx=0x7fcab8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=745a693b65fd9d4b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.520+0000 7fcaceffd700 1 -- 172.21.15.1:0/3968782895 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab8011830 con 0x7fcad0143c20 2024-03-18T20:06:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.520+0000 7fcaceffd700 1 -- 172.21.15.1:0/3968782895 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab8011990 con 0x7fcad0143c20 2024-03-18T20:06:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.520+0000 7fcaceffd700 1 -- 172.21.15.1:0/3968782895 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab8011d20 con 0x7fcad0143c20 2024-03-18T20:06:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.520+0000 7fcad6929700 1 -- 172.21.15.1:0/3968782895 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcad0143c20 msgr2=0x7fcad00b3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.520+0000 7fcad6929700 1 --2- 172.21.15.1:0/3968782895 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcad0143c20 0x7fcad00b3a00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcab8009840 tx=0x7fcab8009b50 comp rx=0 tx=0).stop 2024-03-18T20:06:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.520+0000 7fcad6929700 1 -- 172.21.15.1:0/3968782895 shutdown_connections 2024-03-18T20:06:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.520+0000 7fcad6929700 1 --2- 172.21.15.1:0/3968782895 >> v2:172.21.15.165:3300/0 conn(0x7fcad006f240 0x7fcad00cddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.520+0000 7fcad6929700 1 --2- 172.21.15.1:0/3968782895 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcad0143c20 0x7fcad00b3a00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.520+0000 7fcad6929700 1 -- 172.21.15.1:0/3968782895 >> 172.21.15.1:0/3968782895 conn(0x7fcad00b1d20 msgr2=0x7fcad00b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.520+0000 7fcad6929700 1 -- 172.21.15.1:0/3968782895 shutdown_connections 2024-03-18T20:06:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.520+0000 7fcad6929700 1 -- 172.21.15.1:0/3968782895 wait complete. 2024-03-18T20:06:46.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.522+0000 7fcad6929700 1 Processor -- start 2024-03-18T20:06:46.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.522+0000 7fcad6929700 1 -- start start 2024-03-18T20:06:46.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.523+0000 7fcad6929700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcad006f240 0x7fcad0142a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:46.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.523+0000 7fcad6929700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcad00b3f40 0x7fcad0155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:46.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.523+0000 7fcad6929700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcad0143c20 0x7fcad006a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:46.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.523+0000 7fcad6929700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad0154930 con 0x7fcad0143c20 2024-03-18T20:06:46.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.523+0000 7fcad6929700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad0154d50 con 0x7fcad00b3f40 2024-03-18T20:06:46.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.523+0000 7fcad6929700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad0153f90 con 0x7fcad006f240 2024-03-18T20:06:46.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.523+0000 7fcad4ec6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcad0143c20 0x7fcad006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:46.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.523+0000 7fcacffff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcad006f240 0x7fcad0142a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:46.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.523+0000 7fcacf7fe700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcad00b3f40 0x7fcad0155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:46.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.523+0000 7fcad4ec6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcad0143c20 0x7fcad006a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56506/0 (socket says 172.21.15.1:56506) 2024-03-18T20:06:46.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.523+0000 7fcad4ec6700 1 -- 172.21.15.1:0/2131932310 learned_addr learned my addr 172.21.15.1:0/2131932310 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:46.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.523+0000 7fcad4ec6700 1 -- 172.21.15.1:0/2131932310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcad006f240 msgr2=0x7fcad0142a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:46.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.523+0000 7fcad4ec6700 1 --2- 172.21.15.1:0/2131932310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcad006f240 0x7fcad0142a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.523+0000 7fcad4ec6700 1 -- 172.21.15.1:0/2131932310 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcad00b3f40 msgr2=0x7fcad0155840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:46.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.523+0000 7fcad4ec6700 1 --2- 172.21.15.1:0/2131932310 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcad00b3f40 0x7fcad0155840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.523+0000 7fcad4ec6700 1 -- 172.21.15.1:0/2131932310 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac800bc80 con 0x7fcad0143c20 2024-03-18T20:06:46.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.523+0000 7fcacffff700 1 --2- 172.21.15.1:0/2131932310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcad006f240 0x7fcad0142a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:46.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.523+0000 7fcacf7fe700 1 --2- 172.21.15.1:0/2131932310 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcad00b3f40 0x7fcad0155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:46.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.524+0000 7fcad4ec6700 1 --2- 172.21.15.1:0/2131932310 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcad0143c20 0x7fcad006a750 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fcac800e0c0 tx=0x7fcac800e3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:46.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.524+0000 7fcaccff9700 1 -- 172.21.15.1:0/2131932310 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac8010040 con 0x7fcad0143c20 2024-03-18T20:06:46.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.524+0000 7fcaccff9700 1 -- 172.21.15.1:0/2131932310 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac800ebd0 con 0x7fcad0143c20 2024-03-18T20:06:46.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.524+0000 7fcad6929700 1 -- 172.21.15.1:0/2131932310 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab80094a0 con 0x7fcad0143c20 2024-03-18T20:06:46.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.524+0000 7fcaccff9700 1 -- 172.21.15.1:0/2131932310 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac800b3a0 con 0x7fcad0143c20 2024-03-18T20:06:46.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.524+0000 7fcad6929700 1 -- 172.21.15.1:0/2131932310 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad0154550 con 0x7fcad0143c20 2024-03-18T20:06:46.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.525+0000 7fcad6929700 1 -- 172.21.15.1:0/2131932310 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad00676a0 con 0x7fcad0143c20 2024-03-18T20:06:46.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.525+0000 7fcaccff9700 1 -- 172.21.15.1:0/2131932310 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcac8009540 con 0x7fcad0143c20 2024-03-18T20:06:46.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.525+0000 7fcaccff9700 1 --2- 172.21.15.1:0/2131932310 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcab006adb0 0x7fcab006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:46.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.526+0000 7fcaccff9700 1 -- 172.21.15.1:0/2131932310 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcac8089690 con 0x7fcad0143c20 2024-03-18T20:06:46.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.526+0000 7fcacffff700 1 --2- 172.21.15.1:0/2131932310 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcab006adb0 0x7fcab006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:46.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.526+0000 7fcacffff700 1 --2- 172.21.15.1:0/2131932310 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcab006adb0 0x7fcab006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcab8009fd0 tx=0x7fcab801c040 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:46.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.530+0000 7fcaccff9700 1 -- 172.21.15.1:0/2131932310 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcac8054600 con 0x7fcad0143c20 2024-03-18T20:06:46.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.779+0000 7fcad6929700 1 -- 172.21.15.1:0/2131932310 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcad00aff80 con 0x7fcad0143c20 2024-03-18T20:06:46.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.780+0000 7fcaccff9700 1 -- 172.21.15.1:0/2131932310 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4738 (secure 0 0 0) 0x7fcac8057c20 con 0x7fcad0143c20 2024-03-18T20:06:46.785 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:06:46.785 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":81,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7874,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3186058752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3186058752},{"type":"v1","addr":"172.21.15.1:6836","nonce":3186058752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":7886,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1126668185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1126668185},{"type":"v1","addr":"172.21.15.165:6835","nonce":1126668185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9028,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3922981693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3922981693},{"type":"v1","addr":"172.21.15.165:6834","nonce":3922981693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:06:41.339378+0000","modified":"2024-03-18T20:06:44.348416+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9909},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9909":{"gid":9909,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/3096868087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3096868087},{"type":"v1","addr":"172.21.15.1:6837","nonce":3096868087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-03-18T20:06:46.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.782+0000 7fcad6929700 1 -- 172.21.15.1:0/2131932310 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcab006adb0 msgr2=0x7fcab006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:46.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.782+0000 7fcad6929700 1 --2- 172.21.15.1:0/2131932310 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcab006adb0 0x7fcab006d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcab8009fd0 tx=0x7fcab801c040 comp rx=0 tx=0).stop 2024-03-18T20:06:46.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.782+0000 7fcad6929700 1 -- 172.21.15.1:0/2131932310 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcad0143c20 msgr2=0x7fcad006a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:46.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.782+0000 7fcad6929700 1 --2- 172.21.15.1:0/2131932310 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcad0143c20 0x7fcad006a750 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fcac800e0c0 tx=0x7fcac800e3d0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.782+0000 7fcad6929700 1 -- 172.21.15.1:0/2131932310 shutdown_connections 2024-03-18T20:06:46.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.782+0000 7fcad6929700 1 --2- 172.21.15.1:0/2131932310 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fcab006adb0 0x7fcab006d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.782+0000 7fcad6929700 1 --2- 172.21.15.1:0/2131932310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcad006f240 0x7fcad0142a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.782+0000 7fcad6929700 1 --2- 172.21.15.1:0/2131932310 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcad0143c20 0x7fcad006a750 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.782+0000 7fcad6929700 1 --2- 172.21.15.1:0/2131932310 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcad00b3f40 0x7fcad0155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:46.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.782+0000 7fcad6929700 1 -- 172.21.15.1:0/2131932310 >> 172.21.15.1:0/2131932310 conn(0x7fcad00b1d20 msgr2=0x7fcad00b13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:46.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.782+0000 7fcad6929700 1 -- 172.21.15.1:0/2131932310 shutdown_connections 2024-03-18T20:06:46.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:46.782+0000 7fcad6929700 1 -- 172.21.15.1:0/2131932310 wait complete. 2024-03-18T20:06:46.788 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 81 2024-03-18T20:06:46.903 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-18T20:06:46.899+0000 7f02478b7700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T20:06:47.203 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T20:06:41.339378+0000', 'modified': '2024-03-18T20:06:44.348416+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9909}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9909': {'gid': 9909, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6837/3096868087', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6835', 'nonce': 3096868087}, {'type': 'v1', 'addr': '172.21.15.1:6837', 'nonce': 3096868087}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T20:06:47.203 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T20:06:47.204 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T20:06:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.331+0000 7f6fd18e9700 1 Processor -- start 2024-03-18T20:06:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.331+0000 7f6fd18e9700 1 -- start start 2024-03-18T20:06:47.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.331+0000 7f6fd18e9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fcc099a40 0x7f6fcc15e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.331+0000 7f6fd18e9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6fcc15d6f0 0x7f6fcc099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.331+0000 7f6fd18e9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fcc15e9e0 con 0x7f6fcc06e530 2024-03-18T20:06:47.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.331+0000 7f6fd18e9700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fcc06ea40 con 0x7f6fcc15d6f0 2024-03-18T20:06:47.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.331+0000 7f6fd18e9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc09a360 con 0x7f6fcc099a40 2024-03-18T20:06:47.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fcb7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6fcc06e530 0x7f6fcc160fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49174/0 (socket says 172.21.15.1:49174) 2024-03-18T20:06:47.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fcb7fe700 1 -- 172.21.15.1:0/3030310286 learned_addr learned my addr 172.21.15.1:0/3030310286 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:47.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fc9ffb700 1 -- 172.21.15.1:0/3030310286 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000782704 0 0) 0x7f6fcc15e9e0 con 0x7f6fcc06e530 2024-03-18T20:06:47.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fca7fc700 1 --2- 172.21.15.1:0/3030310286 >> v2:172.21.15.165:3300/0 conn(0x7f6fcc15d6f0 0x7f6fcc099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fcaffd700 1 --2- 172.21.15.1:0/3030310286 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fcc099a40 0x7f6fcc15e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fc9ffb700 1 -- 172.21.15.1:0/3030310286 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fb40035b0 con 0x7f6fcc06e530 2024-03-18T20:06:47.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fc9ffb700 1 -- 172.21.15.1:0/3030310286 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3671804197 0 0) 0x7f6fb40035b0 con 0x7f6fcc06e530 2024-03-18T20:06:47.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fc9ffb700 1 -- 172.21.15.1:0/3030310286 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fcc099a40 msgr2=0x7f6fcc15e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:47.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fc9ffb700 1 --2- 172.21.15.1:0/3030310286 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fcc099a40 0x7f6fcc15e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fc9ffb700 1 -- 172.21.15.1:0/3030310286 >> v2:172.21.15.165:3300/0 conn(0x7f6fcc15d6f0 msgr2=0x7f6fcc099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fc9ffb700 1 --2- 172.21.15.1:0/3030310286 >> v2:172.21.15.165:3300/0 conn(0x7f6fcc15d6f0 0x7f6fcc099e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fc9ffb700 1 -- 172.21.15.1:0/3030310286 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fcc06ece0 con 0x7f6fcc06e530 2024-03-18T20:06:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fc9ffb700 1 -- 172.21.15.1:0/3030310286 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f6fc40019f0 con 0x7f6fcc06e530 2024-03-18T20:06:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fc9ffb700 1 -- 172.21.15.1:0/3030310286 >> v1:172.21.15.1:6789/0 conn(0x7f6fcc06e530 legacy=0x7f6fcc160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fc9ffb700 1 --2- 172.21.15.1:0/3030310286 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fcc099a40 0x7f6fb4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fc9ffb700 1 --2- 172.21.15.1:0/3030310286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fb4003f80 0x7f6fb4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fc9ffb700 1 --2- 172.21.15.1:0/3030310286 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fb4004d90 0x7f6fb4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fc9ffb700 1 -- 172.21.15.1:0/3030310286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc06ea40 con 0x7f6fb4003f80 2024-03-18T20:06:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fc9ffb700 1 -- 172.21.15.1:0/3030310286 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc09a360 con 0x7f6fb4004d90 2024-03-18T20:06:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fc9ffb700 1 -- 172.21.15.1:0/3030310286 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb400ba10 con 0x7f6fcc099a40 2024-03-18T20:06:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fca7fc700 1 --2- 172.21.15.1:0/3030310286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fb4003f80 0x7f6fb4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fcb7fe700 1 --2- 172.21.15.1:0/3030310286 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fb4004d90 0x7f6fb4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.332+0000 7f6fcaffd700 1 --2- 172.21.15.1:0/3030310286 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fcc099a40 0x7f6fb4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.333+0000 7f6fca7fc700 1 -- 172.21.15.1:0/3030310286 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fcc099a40 msgr2=0x7f6fb4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.333+0000 7f6fca7fc700 1 --2- 172.21.15.1:0/3030310286 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fcc099a40 0x7f6fb4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.333+0000 7f6fca7fc700 1 -- 172.21.15.1:0/3030310286 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fb4004d90 msgr2=0x7f6fb4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.333+0000 7f6fca7fc700 1 --2- 172.21.15.1:0/3030310286 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fb4004d90 0x7f6fb4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.333+0000 7f6fca7fc700 1 -- 172.21.15.1:0/3030310286 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb40035b0 con 0x7f6fb4003f80 2024-03-18T20:06:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.333+0000 7f6fcb7fe700 1 --2- 172.21.15.1:0/3030310286 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fb4004d90 0x7f6fb4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.333+0000 7f6fca7fc700 1 --2- 172.21.15.1:0/3030310286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fb4003f80 0x7f6fb4004850 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0009b60 tx=0x7f6fb4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2e1383954e32655 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.333+0000 7f6fc9ffb700 1 -- 172.21.15.1:0/3030310286 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc000e660 con 0x7f6fb4003f80 2024-03-18T20:06:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.333+0000 7f6fc9ffb700 1 -- 172.21.15.1:0/3030310286 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc000e7c0 con 0x7f6fb4003f80 2024-03-18T20:06:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.333+0000 7f6fc9ffb700 1 -- 172.21.15.1:0/3030310286 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc000eb30 con 0x7f6fb4003f80 2024-03-18T20:06:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.333+0000 7f6fd18e9700 1 -- 172.21.15.1:0/3030310286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fb4003f80 msgr2=0x7f6fb4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.333+0000 7f6fd18e9700 1 --2- 172.21.15.1:0/3030310286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fb4003f80 0x7f6fb4004850 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0009b60 tx=0x7f6fb4005880 comp rx=0 tx=0).stop 2024-03-18T20:06:47.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.333+0000 7f6fcb7fe700 1 -- 172.21.15.1:0/3030310286 reap_dead start 2024-03-18T20:06:47.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.334+0000 7f6fd18e9700 1 -- 172.21.15.1:0/3030310286 shutdown_connections 2024-03-18T20:06:47.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.334+0000 7f6fd18e9700 1 -- 172.21.15.1:0/3030310286 >> 172.21.15.1:0/3030310286 conn(0x7f6fcc0b2af0 msgr2=0x7f6fcc0b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:47.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.334+0000 7f6fd18e9700 1 -- 172.21.15.1:0/3030310286 shutdown_connections 2024-03-18T20:06:47.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.334+0000 7f6fd18e9700 1 -- 172.21.15.1:0/3030310286 wait complete. 2024-03-18T20:06:47.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.335+0000 7f6fd18e9700 1 Processor -- start 2024-03-18T20:06:47.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.335+0000 7f6fd18e9700 1 -- start start 2024-03-18T20:06:47.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.336+0000 7f6fd18e9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fb4003f80 0x7f6fcc0b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.336+0000 7f6fd18e9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fcc0b4140 0x7f6fcc143050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.336+0000 7f6fd18e9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fcc143590 0x7f6fcc0b5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.336+0000 7f6fd18e9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc0b45c0 con 0x7f6fb4003f80 2024-03-18T20:06:47.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.336+0000 7f6fd18e9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc1f6560 con 0x7f6fcc0b4140 2024-03-18T20:06:47.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.336+0000 7f6fd18e9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc1f67d0 con 0x7f6fcc143590 2024-03-18T20:06:47.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.336+0000 7f6fcaffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fb4003f80 0x7f6fcc0b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.336+0000 7f6fcaffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fb4003f80 0x7f6fcc0b3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56530/0 (socket says 172.21.15.1:56530) 2024-03-18T20:06:47.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.336+0000 7f6fcaffd700 1 -- 172.21.15.1:0/3682324640 learned_addr learned my addr 172.21.15.1:0/3682324640 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:47.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.336+0000 7f6fcb7fe700 1 --2- 172.21.15.1:0/3682324640 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fcc143590 0x7f6fcc0b5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.336+0000 7f6fca7fc700 1 --2- 172.21.15.1:0/3682324640 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fcc0b4140 0x7f6fcc143050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.336+0000 7f6fcaffd700 1 -- 172.21.15.1:0/3682324640 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fcc143590 msgr2=0x7f6fcc0b5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:47.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.336+0000 7f6fcaffd700 1 --2- 172.21.15.1:0/3682324640 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fcc143590 0x7f6fcc0b5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.336+0000 7f6fcaffd700 1 -- 172.21.15.1:0/3682324640 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fcc0b4140 msgr2=0x7f6fcc143050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:47.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.336+0000 7f6fcaffd700 1 --2- 172.21.15.1:0/3682324640 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fcc0b4140 0x7f6fcc143050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.336+0000 7f6fcaffd700 1 -- 172.21.15.1:0/3682324640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fcc1f6ab0 con 0x7f6fb4003f80 2024-03-18T20:06:47.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.336+0000 7f6fcaffd700 1 --2- 172.21.15.1:0/3682324640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fb4003f80 0x7f6fcc0b3c00 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc011ad0 tx=0x7f6fbc011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:47.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.336+0000 7f6fb3fff700 1 -- 172.21.15.1:0/3682324640 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc00f940 con 0x7f6fb4003f80 2024-03-18T20:06:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.336+0000 7f6fb3fff700 1 -- 172.21.15.1:0/3682324640 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc00faa0 con 0x7f6fb4003f80 2024-03-18T20:06:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.337+0000 7f6fd18e9700 1 -- 172.21.15.1:0/3682324640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fcc1f6d90 con 0x7f6fb4003f80 2024-03-18T20:06:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.337+0000 7f6fb3fff700 1 -- 172.21.15.1:0/3682324640 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc00fe50 con 0x7f6fb4003f80 2024-03-18T20:06:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.337+0000 7f6fd18e9700 1 -- 172.21.15.1:0/3682324640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fcc143a10 con 0x7f6fb4003f80 2024-03-18T20:06:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.339+0000 7f6fb3fff700 1 -- 172.21.15.1:0/3682324640 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6fbc014440 con 0x7f6fb4003f80 2024-03-18T20:06:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.339+0000 7f6fb3fff700 1 --2- 172.21.15.1:0/3682324640 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6fb806adb0 0x7f6fb806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.339+0000 7f6fb3fff700 1 -- 172.21.15.1:0/3682324640 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6fbc089bf0 con 0x7f6fb4003f80 2024-03-18T20:06:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.339+0000 7f6fca7fc700 1 --2- 172.21.15.1:0/3682324640 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6fb806adb0 0x7f6fb806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.340+0000 7f6fca7fc700 1 --2- 172.21.15.1:0/3682324640 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6fb806adb0 0x7f6fb806d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6fc00090b0 tx=0x7f6fc001f040 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.341+0000 7f6fdda92b80 1 Processor -- start 2024-03-18T20:06:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.341+0000 7f6fdda92b80 1 -- start start 2024-03-18T20:06:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.342+0000 7f6fdda92b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd614c10 0x55edfd614ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.342+0000 7f6fdda92b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd5e60a0 0x55edfd631580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.342+0000 7f6fdda92b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd626630 0x55edfd5e9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.342+0000 7f6fdda92b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edfd55ff20 con 0x55edfd614c10 2024-03-18T20:06:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.342+0000 7f6fdda92b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edfd541c30 con 0x55edfd5e60a0 2024-03-18T20:06:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.342+0000 7f6fdda92b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55edfd53f430 con 0x55edfd626630 2024-03-18T20:06:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.342+0000 7f6fcaffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd614c10 0x55edfd614ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.342+0000 7f6fcaffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd614c10 0x55edfd614ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56534/0 (socket says 172.21.15.1:56534) 2024-03-18T20:06:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.342+0000 7f6fcaffd700 1 -- 172.21.15.1:0/102138994 learned_addr learned my addr 172.21.15.1:0/102138994 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.342+0000 7f6fcaffd700 1 -- 172.21.15.1:0/102138994 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd626630 msgr2=0x55edfd5e9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.342+0000 7f6fcaffd700 1 --2- 172.21.15.1:0/102138994 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd626630 0x55edfd5e9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.342+0000 7f6fcaffd700 1 -- 172.21.15.1:0/102138994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd5e60a0 msgr2=0x55edfd631580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.342+0000 7f6fcaffd700 1 --2- 172.21.15.1:0/102138994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd5e60a0 0x55edfd631580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.342+0000 7f6fcaffd700 1 -- 172.21.15.1:0/102138994 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55edfd32e790 con 0x55edfd614c10 2024-03-18T20:06:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.342+0000 7f6fcaffd700 1 --2- 172.21.15.1:0/102138994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd614c10 0x55edfd614ff0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc010010 tx=0x7f6fbc058270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc65c4a9c916ee0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.342+0000 7f6fb17fa700 1 -- 172.21.15.1:0/102138994 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc05c640 con 0x55edfd614c10 2024-03-18T20:06:47.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.342+0000 7f6fb17fa700 1 -- 172.21.15.1:0/102138994 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc060c00 con 0x55edfd614c10 2024-03-18T20:06:47.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.342+0000 7f6fb17fa700 1 -- 172.21.15.1:0/102138994 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc058a70 con 0x55edfd614c10 2024-03-18T20:06:47.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.343+0000 7f6fdda92b80 1 -- 172.21.15.1:0/102138994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd614c10 msgr2=0x55edfd614ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:47.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.343+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/102138994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd614c10 0x55edfd614ff0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc010010 tx=0x7f6fbc058270 comp rx=0 tx=0).stop 2024-03-18T20:06:47.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.343+0000 7f6fdda92b80 1 -- 172.21.15.1:0/102138994 shutdown_connections 2024-03-18T20:06:47.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.343+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/102138994 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd626630 0x55edfd5e9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.343+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/102138994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd614c10 0x55edfd614ff0 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.343+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/102138994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd5e60a0 0x55edfd631580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.343+0000 7f6fdda92b80 1 -- 172.21.15.1:0/102138994 >> 172.21.15.1:0/102138994 conn(0x55edfd548cf0 msgr2=0x55edfd5e8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:47.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.343+0000 7f6fdda92b80 1 -- 172.21.15.1:0/102138994 shutdown_connections 2024-03-18T20:06:47.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.343+0000 7f6fdda92b80 1 -- 172.21.15.1:0/102138994 wait complete. 2024-03-18T20:06:47.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.344+0000 7f6fdda92b80 1 -- start start 2024-03-18T20:06:47.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.344+0000 7f6fdda92b80 1 Processor -- start 2024-03-18T20:06:47.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fdda92b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd5e60a0 0x55edfd627160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fdda92b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd614c10 0x55edfd609110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fdda92b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd626630 0x55edfd609650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fdda92b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edfd53ad20 con 0x55edfd626630 2024-03-18T20:06:47.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fdda92b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edfd4d6770 con 0x55edfd614c10 2024-03-18T20:06:47.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fdda92b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55edfd4f96f0 con 0x55edfd5e60a0 2024-03-18T20:06:47.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fdda92b80 10 client.0 _ll_register_callbacks cb 0x55edfd3425a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:06:47.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fcb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd626630 0x55edfd609650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fcb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd626630 0x55edfd609650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56544/0 (socket says 172.21.15.1:56544) 2024-03-18T20:06:47.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fcb7fe700 1 -- 172.21.15.1:0/3537286488 learned_addr learned my addr 172.21.15.1:0/3537286488 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:47.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fca7fc700 1 --2- 172.21.15.1:0/3537286488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd614c10 0x55edfd609110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fcaffd700 1 --2- 172.21.15.1:0/3537286488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd5e60a0 0x55edfd627160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fcb7fe700 1 -- 172.21.15.1:0/3537286488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd5e60a0 msgr2=0x55edfd627160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:47.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fcb7fe700 1 --2- 172.21.15.1:0/3537286488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd5e60a0 0x55edfd627160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fcb7fe700 1 -- 172.21.15.1:0/3537286488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd614c10 msgr2=0x55edfd609110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:47.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fcb7fe700 1 --2- 172.21.15.1:0/3537286488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd614c10 0x55edfd609110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fcb7fe700 1 -- 172.21.15.1:0/3537286488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fcc143a10 con 0x55edfd626630 2024-03-18T20:06:47.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fcaffd700 1 --2- 172.21.15.1:0/3537286488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd5e60a0 0x55edfd627160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:47.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fca7fc700 1 --2- 172.21.15.1:0/3537286488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd614c10 0x55edfd609110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:47.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fcb7fe700 1 --2- 172.21.15.1:0/3537286488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd626630 0x55edfd609650 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0c8040 tx=0x7f6fc4011530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9f3ae2df29f32d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6faa7fc700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T20:06:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6faa7fc700 1 -- 172.21.15.1:0/3537286488 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc4011ca0 con 0x55edfd626630 2024-03-18T20:06:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6faa7fc700 1 -- 172.21.15.1:0/3537286488 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(181..181 src has 1..181) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6fc4011e00 con 0x55edfd626630 2024-03-18T20:06:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fdda92b80 1 -- 172.21.15.1:0/3537286488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55edfd5dce90 con 0x55edfd626630 2024-03-18T20:06:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6faa7fc700 1 -- 172.21.15.1:0/3537286488 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fc4015470 con 0x55edfd626630 2024-03-18T20:06:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fdda92b80 10 client.10053 fetch_fsmap learned FSMap version 81 2024-03-18T20:06:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.345+0000 7f6fdda92b80 1 -- 172.21.15.1:0/3537286488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) v3 -- 0x55edfd41b0a0 con 0x55edfd626630 2024-03-18T20:06:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.346+0000 7f6faa7fc700 1 -- 172.21.15.1:0/3537286488 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55edfd5dce90 con 0x55edfd626630 2024-03-18T20:06:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.346+0000 7f6faa7fc700 1 -- 172.21.15.1:0/3537286488 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 81) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f6fc40155d0 con 0x55edfd626630 2024-03-18T20:06:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.346+0000 7f6fdda92b80 10 client.10053 fetch_fsmap finished waiting for FSMap version 81 2024-03-18T20:06:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.346+0000 7f6fdda92b80 10 client.10053 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T20:06:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.346+0000 7f6fdda92b80 20 client.10053 populate_metadata read hostname 'smithi001' 2024-03-18T20:06:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.346+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/3537286488 >> [v2:172.21.15.1:6835/3096868087,v1:172.21.15.1:6837/3096868087] conn(0x55edfd617d30 0x55edfd64c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.346+0000 7f6fdda92b80 4 client.10053 mds_command: new command op to 9909 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T20:06:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.346+0000 7f6fdda92b80 1 -- 172.21.15.1:0/3537286488 --> [v2:172.21.15.1:6835/3096868087,v1:172.21.15.1:6837/3096868087] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55edfd5ba010 con 0x55edfd617d30 2024-03-18T20:06:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.346+0000 7f6fcaffd700 1 --2- 172.21.15.1:0/3537286488 >> [v2:172.21.15.1:6835/3096868087,v1:172.21.15.1:6837/3096868087] conn(0x55edfd617d30 0x55edfd64c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.346+0000 7f6faa7fc700 10 client.10053 ms_handle_connect on v2:172.21.15.1:6835/3096868087 2024-03-18T20:06:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.346+0000 7f6fcaffd700 1 --2- 172.21.15.1:0/3537286488 >> [v2:172.21.15.1:6835/3096868087,v1:172.21.15.1:6837/3096868087] conn(0x55edfd617d30 0x55edfd64c900 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a39ef50023548706 server_cookie=6dfd8a688929fb in_seq=0 out_seq=0 2024-03-18T20:06:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.347+0000 7f6faa7fc700 1 -- 172.21.15.1:0/3537286488 <== mds.0 v2:172.21.15.1:6835/3096868087 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55edfd5ba010 con 0x55edfd617d30 2024-03-18T20:06:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.347+0000 7f6faa7fc700 10 client.10053 handle_command_reply: tid=0 2024-03-18T20:06:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.347+0000 7f6fdda92b80 1 client.10053 shutdown 2024-03-18T20:06:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.347+0000 7f6fdda92b80 1 -- 172.21.15.1:0/3537286488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd626630 msgr2=0x55edfd609650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.347+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/3537286488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd626630 0x55edfd609650 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0c8040 tx=0x7f6fc4011530 comp rx=0 tx=0).stop 2024-03-18T20:06:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.347+0000 7f6fdda92b80 1 -- 172.21.15.1:0/3537286488 shutdown_connections 2024-03-18T20:06:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.347+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/3537286488 >> [v2:172.21.15.1:6835/3096868087,v1:172.21.15.1:6837/3096868087] conn(0x55edfd617d30 0x55edfd64c900 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.347+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/3537286488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd5e60a0 0x55edfd627160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.347+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/3537286488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd626630 0x55edfd609650 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.347+0000 7f6faa7fc700 0 client.10053 ms_handle_reset on v2:172.21.15.1:6835/3096868087 2024-03-18T20:06:47.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.347+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/3537286488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd614c10 0x55edfd609110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.347+0000 7f6fdda92b80 1 -- 172.21.15.1:0/3537286488 >> 172.21.15.1:0/3537286488 conn(0x55edfd548cf0 msgr2=0x55edfd5e49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:47.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.347+0000 7f6fdda92b80 1 -- 172.21.15.1:0/3537286488 shutdown_connections 2024-03-18T20:06:47.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.347+0000 7f6fdda92b80 1 -- 172.21.15.1:0/3537286488 wait complete. 2024-03-18T20:06:47.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.347+0000 7f6fdda92b80 20 client.10053 trim_cache size 0 max 16384 2024-03-18T20:06:47.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.365+0000 7f6fdda92b80 1 Processor -- start 2024-03-18T20:06:47.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.365+0000 7f6fdda92b80 1 -- start start 2024-03-18T20:06:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.366+0000 7f6fdda92b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd5e60a0 0x55edfd5e7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.366+0000 7f6fdda92b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd614c10 0x55edfd5e3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.366+0000 7f6fdda92b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd626630 0x55edfd5e4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.366+0000 7f6fdda92b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edfd32f390 con 0x55edfd5e60a0 2024-03-18T20:06:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.366+0000 7f6fdda92b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edfd613130 con 0x55edfd614c10 2024-03-18T20:06:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.366+0000 7f6fdda92b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55edfd627890 con 0x55edfd626630 2024-03-18T20:06:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.366+0000 7f6fcaffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd5e60a0 0x55edfd5e7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.366+0000 7f6fcaffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd5e60a0 0x55edfd5e7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56548/0 (socket says 172.21.15.1:56548) 2024-03-18T20:06:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.366+0000 7f6fcaffd700 1 -- 172.21.15.1:0/2517152097 learned_addr learned my addr 172.21.15.1:0/2517152097 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.366+0000 7f6fca7fc700 1 --2- 172.21.15.1:0/2517152097 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd614c10 0x55edfd5e3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.366+0000 7f6fcb7fe700 1 --2- 172.21.15.1:0/2517152097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd626630 0x55edfd5e4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.366+0000 7f6fcaffd700 1 -- 172.21.15.1:0/2517152097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd626630 msgr2=0x55edfd5e4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.366+0000 7f6fcaffd700 1 --2- 172.21.15.1:0/2517152097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd626630 0x55edfd5e4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.366+0000 7f6fcaffd700 1 -- 172.21.15.1:0/2517152097 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd614c10 msgr2=0x55edfd5e3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.366+0000 7f6fcaffd700 1 --2- 172.21.15.1:0/2517152097 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd614c10 0x55edfd5e3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.366+0000 7f6fcaffd700 1 -- 172.21.15.1:0/2517152097 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55edfd501010 con 0x55edfd5e60a0 2024-03-18T20:06:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.366+0000 7f6fcaffd700 1 --2- 172.21.15.1:0/2517152097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd5e60a0 0x55edfd5e7f70 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc087ce0 tx=0x7f6fbc05c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c05bd6ba9993fa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.367+0000 7f6fb1ffb700 1 -- 172.21.15.1:0/2517152097 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc056bb0 con 0x55edfd5e60a0 2024-03-18T20:06:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.367+0000 7f6fb1ffb700 1 -- 172.21.15.1:0/2517152097 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc054f20 con 0x55edfd5e60a0 2024-03-18T20:06:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.367+0000 7f6fb1ffb700 1 -- 172.21.15.1:0/2517152097 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc057f40 con 0x55edfd5e60a0 2024-03-18T20:06:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.367+0000 7f6fdda92b80 1 -- 172.21.15.1:0/2517152097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd5e60a0 msgr2=0x55edfd5e7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.367+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/2517152097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd5e60a0 0x55edfd5e7f70 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc087ce0 tx=0x7f6fbc05c000 comp rx=0 tx=0).stop 2024-03-18T20:06:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.367+0000 7f6fdda92b80 1 -- 172.21.15.1:0/2517152097 shutdown_connections 2024-03-18T20:06:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.367+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/2517152097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd626630 0x55edfd5e4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.367+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/2517152097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd5e60a0 0x55edfd5e7f70 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.367+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/2517152097 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd614c10 0x55edfd5e3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.367+0000 7f6fdda92b80 1 -- 172.21.15.1:0/2517152097 >> 172.21.15.1:0/2517152097 conn(0x55edfd548cf0 msgr2=0x55edfd616110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:47.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.367+0000 7f6fdda92b80 1 -- 172.21.15.1:0/2517152097 shutdown_connections 2024-03-18T20:06:47.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.367+0000 7f6fdda92b80 1 -- 172.21.15.1:0/2517152097 wait complete. 2024-03-18T20:06:47.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.368+0000 7f6fdda92b80 1 -- start start 2024-03-18T20:06:47.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.368+0000 7f6fdda92b80 1 Processor -- start 2024-03-18T20:06:47.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fdda92b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd5e60a0 0x55edfd618780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fdda92b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd614c10 0x55edfd62fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fdda92b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd626630 0x55edfd5e7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fdda92b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edfd649500 con 0x55edfd614c10 2024-03-18T20:06:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fdda92b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edfd614490 con 0x55edfd626630 2024-03-18T20:06:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fdda92b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55edfd699ac0 con 0x55edfd5e60a0 2024-03-18T20:06:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fdda92b80 10 client.0 _ll_register_callbacks cb 0x7f6fc40073f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:06:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fca7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd614c10 0x55edfd62fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fca7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd614c10 0x55edfd62fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56556/0 (socket says 172.21.15.1:56556) 2024-03-18T20:06:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fca7fc700 1 -- 172.21.15.1:0/3019785674 learned_addr learned my addr 172.21.15.1:0/3019785674 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fcb7fe700 1 --2- 172.21.15.1:0/3019785674 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd626630 0x55edfd5e7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fcaffd700 1 --2- 172.21.15.1:0/3019785674 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd5e60a0 0x55edfd618780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fca7fc700 1 -- 172.21.15.1:0/3019785674 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd5e60a0 msgr2=0x55edfd618780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:47.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fca7fc700 1 --2- 172.21.15.1:0/3019785674 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd5e60a0 0x55edfd618780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fca7fc700 1 -- 172.21.15.1:0/3019785674 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd626630 msgr2=0x55edfd5e7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fca7fc700 1 --2- 172.21.15.1:0/3019785674 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd626630 0x55edfd5e7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fca7fc700 1 -- 172.21.15.1:0/3019785674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55edfd32e790 con 0x55edfd614c10 2024-03-18T20:06:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fcaffd700 1 --2- 172.21.15.1:0/3019785674 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd5e60a0 0x55edfd618780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fcb7fe700 1 --2- 172.21.15.1:0/3019785674 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd626630 0x55edfd5e7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6fca7fc700 1 --2- 172.21.15.1:0/3019785674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd614c10 0x55edfd62fd50 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f6fc000e4b0 tx=0x7f6fc00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f14a8dd2e63a111 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.369+0000 7f6faa7fc700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T20:06:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.370+0000 7f6faa7fc700 1 -- 172.21.15.1:0/3019785674 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0022040 con 0x55edfd614c10 2024-03-18T20:06:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.370+0000 7f6faa7fc700 1 -- 172.21.15.1:0/3019785674 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(181..181 src has 1..181) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6fc00038b0 con 0x55edfd614c10 2024-03-18T20:06:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.370+0000 7f6fdda92b80 1 -- 172.21.15.1:0/3019785674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6fbc060c00 con 0x55edfd614c10 2024-03-18T20:06:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.370+0000 7f6faa7fc700 1 -- 172.21.15.1:0/3019785674 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fc0003a70 con 0x55edfd614c10 2024-03-18T20:06:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.370+0000 7f6fdda92b80 10 client.10059 fetch_fsmap learned FSMap version 81 2024-03-18T20:06:47.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.370+0000 7f6fdda92b80 1 -- 172.21.15.1:0/3019785674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=81,monmap=2+}) v3 -- 0x55edfd38f1d0 con 0x55edfd614c10 2024-03-18T20:06:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.370+0000 7f6faa7fc700 1 -- 172.21.15.1:0/3019785674 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc0040070 con 0x55edfd614c10 2024-03-18T20:06:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.370+0000 7f6faa7fc700 1 -- 172.21.15.1:0/3019785674 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 81) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f6fc0003bd0 con 0x55edfd614c10 2024-03-18T20:06:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.370+0000 7f6fdda92b80 10 client.10059 fetch_fsmap finished waiting for FSMap version 81 2024-03-18T20:06:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.370+0000 7f6fdda92b80 10 client.10059 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T20:06:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.370+0000 7f6fdda92b80 20 client.10059 populate_metadata read hostname 'smithi001' 2024-03-18T20:06:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.370+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/3019785674 >> [v2:172.21.15.1:6835/3096868087,v1:172.21.15.1:6837/3096868087] conn(0x55edfd693690 0x55edfd5e84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.370+0000 7f6fdda92b80 4 client.10059 mds_command: new command op to 9909 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T20:06:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.370+0000 7f6fdda92b80 1 -- 172.21.15.1:0/3019785674 --> [v2:172.21.15.1:6835/3096868087,v1:172.21.15.1:6837/3096868087] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55edfd374040 con 0x55edfd693690 2024-03-18T20:06:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.370+0000 7f6fcaffd700 1 --2- 172.21.15.1:0/3019785674 >> [v2:172.21.15.1:6835/3096868087,v1:172.21.15.1:6837/3096868087] conn(0x55edfd693690 0x55edfd5e84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.370+0000 7f6faa7fc700 10 client.10059 ms_handle_connect on v2:172.21.15.1:6835/3096868087 2024-03-18T20:06:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.370+0000 7f6fcaffd700 1 --2- 172.21.15.1:0/3019785674 >> [v2:172.21.15.1:6835/3096868087,v1:172.21.15.1:6837/3096868087] conn(0x55edfd693690 0x55edfd5e84b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b8784d5ce2852f0 server_cookie=f1e9945bf473c0df in_seq=0 out_seq=0 2024-03-18T20:06:47.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6faa7fc700 1 -- 172.21.15.1:0/3019785674 <== mds.0 v2:172.21.15.1:6835/3096868087 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55edfd374040 con 0x55edfd693690 2024-03-18T20:06:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6faa7fc700 10 client.10059 handle_command_reply: tid=0 2024-03-18T20:06:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fdda92b80 1 client.10059 shutdown 2024-03-18T20:06:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fdda92b80 1 -- 172.21.15.1:0/3019785674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd614c10 msgr2=0x55edfd62fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/3019785674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd614c10 0x55edfd62fd50 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f6fc000e4b0 tx=0x7f6fc00047c0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fdda92b80 1 -- 172.21.15.1:0/3019785674 shutdown_connections 2024-03-18T20:06:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/3019785674 >> [v2:172.21.15.1:6835/3096868087,v1:172.21.15.1:6837/3096868087] conn(0x55edfd693690 0x55edfd5e84b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/3019785674 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55edfd5e60a0 0x55edfd618780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/3019785674 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55edfd626630 0x55edfd5e7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fdda92b80 1 --2- 172.21.15.1:0/3019785674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55edfd614c10 0x55edfd62fd50 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fdda92b80 1 -- 172.21.15.1:0/3019785674 >> 172.21.15.1:0/3019785674 conn(0x55edfd548cf0 msgr2=0x55edfd641c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6faa7fc700 0 client.10059 ms_handle_reset on v2:172.21.15.1:6835/3096868087 2024-03-18T20:06:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fdda92b80 1 -- 172.21.15.1:0/3019785674 shutdown_connections 2024-03-18T20:06:47.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fdda92b80 1 -- 172.21.15.1:0/3019785674 wait complete. 2024-03-18T20:06:47.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fdda92b80 20 client.10059 trim_cache size 0 max 16384 2024-03-18T20:06:47.384 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:06:47.384 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "9c1c697f-8655-4ef3-8a13-da568d2cbe86", 2024-03-18T20:06:47.384 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-18T20:06:47.384 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 9909, 2024-03-18T20:06:47.384 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-18T20:06:47.384 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-18T20:06:47.384 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-18T20:06:47.384 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 6.0313125510000001, 2024-03-18T20:06:47.384 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 81, 2024-03-18T20:06:47.385 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 181, 2024-03-18T20:06:47.385 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 181, 2024-03-18T20:06:47.385 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 11.345680654000001 2024-03-18T20:06:47.385 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:06:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fd18e9700 1 -- 172.21.15.1:0/3682324640 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6fb806adb0 msgr2=0x7f6fb806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fd18e9700 1 --2- 172.21.15.1:0/3682324640 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6fb806adb0 0x7f6fb806d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6fc00090b0 tx=0x7f6fc001f040 comp rx=0 tx=0).stop 2024-03-18T20:06:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fd18e9700 1 -- 172.21.15.1:0/3682324640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fb4003f80 msgr2=0x7f6fcc0b3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fd18e9700 1 --2- 172.21.15.1:0/3682324640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fb4003f80 0x7f6fcc0b3c00 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc011ad0 tx=0x7f6fbc011de0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fd18e9700 1 -- 172.21.15.1:0/3682324640 shutdown_connections 2024-03-18T20:06:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fd18e9700 1 --2- 172.21.15.1:0/3682324640 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6fb806adb0 0x7f6fb806d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fd18e9700 1 --2- 172.21.15.1:0/3682324640 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fcc143590 0x7f6fcc0b5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fd18e9700 1 --2- 172.21.15.1:0/3682324640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fb4003f80 0x7f6fcc0b3c00 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fd18e9700 1 --2- 172.21.15.1:0/3682324640 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fcc0b4140 0x7f6fcc143050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.371+0000 7f6fd18e9700 1 -- 172.21.15.1:0/3682324640 >> 172.21.15.1:0/3682324640 conn(0x7f6fcc0b2af0 msgr2=0x7f6fcc0ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.372+0000 7f6fd18e9700 1 -- 172.21.15.1:0/3682324640 shutdown_connections 2024-03-18T20:06:47.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.372+0000 7f6fd18e9700 1 -- 172.21.15.1:0/3682324640 wait complete. 2024-03-18T20:06:47.390 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T20:06:47.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.509+0000 7f182147b700 1 Processor -- start 2024-03-18T20:06:47.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.509+0000 7f182147b700 1 -- start start 2024-03-18T20:06:47.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.510+0000 7f182147b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f181c160940 0x7f181c0d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.510+0000 7f182147b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181c0cddc0 con 0x7f181c0d1ea0 2024-03-18T20:06:47.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.510+0000 7f182147b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181c0cdff0 con 0x7f181c0cf090 2024-03-18T20:06:47.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.510+0000 7f182147b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f181c0ce2d0 con 0x7f181c160940 2024-03-18T20:06:47.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.510+0000 7f181b7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f181c0d1ea0 0x7f181c0b25a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49188/0 (socket says 172.21.15.1:49188) 2024-03-18T20:06:47.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.510+0000 7f181b7fe700 1 -- 172.21.15.1:0/1782025688 learned_addr learned my addr 172.21.15.1:0/1782025688 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:47.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.510+0000 7f181affd700 1 --2- 172.21.15.1:0/1782025688 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f181c160940 0x7f181c0d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.510+0000 7f1819ffb700 1 -- 172.21.15.1:0/1782025688 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716155490 0 0) 0x7f181c0cddc0 con 0x7f181c0d1ea0 2024-03-18T20:06:47.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.510+0000 7f1819ffb700 1 -- 172.21.15.1:0/1782025688 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18040035b0 con 0x7f181c0d1ea0 2024-03-18T20:06:47.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.510+0000 7f1819ffb700 1 -- 172.21.15.1:0/1782025688 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191910397 0 0) 0x7f181c0cdff0 con 0x7f181c0cf090 2024-03-18T20:06:47.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.510+0000 7f1819ffb700 1 -- 172.21.15.1:0/1782025688 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f181c0cddc0 con 0x7f181c0cf090 2024-03-18T20:06:47.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.510+0000 7f181affd700 1 -- 172.21.15.1:0/1782025688 >> v1:172.21.15.165:6789/0 conn(0x7f181c0cf090 legacy=0x7f181c0d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:47.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.510+0000 7f1819ffb700 1 -- 172.21.15.1:0/1782025688 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1552107017 0 0) 0x7f18040035b0 con 0x7f181c0d1ea0 2024-03-18T20:06:47.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.510+0000 7f181affd700 1 -- 172.21.15.1:0/1782025688 >> v1:172.21.15.1:6789/0 conn(0x7f181c0d1ea0 legacy=0x7f181c0b25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:47.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.510+0000 7f181affd700 1 -- 172.21.15.1:0/1782025688 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c0ce5b0 con 0x7f181c160940 2024-03-18T20:06:47.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.511+0000 7f181affd700 1 --2- 172.21.15.1:0/1782025688 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f181c160940 0x7f181c0d0190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1810009870 tx=0x7f1810009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3dbb8704ce755a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:47.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.511+0000 7f1819ffb700 1 -- 172.21.15.1:0/1782025688 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18140019f0 con 0x7f181c160940 2024-03-18T20:06:47.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.511+0000 7f1819ffb700 1 -- 172.21.15.1:0/1782025688 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1810004030 con 0x7f181c160940 2024-03-18T20:06:47.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.511+0000 7f1819ffb700 1 -- 172.21.15.1:0/1782025688 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18100043a0 con 0x7f181c160940 2024-03-18T20:06:47.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.511+0000 7f182147b700 1 -- 172.21.15.1:0/1782025688 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f181c160940 msgr2=0x7f181c0d0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:47.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.511+0000 7f182147b700 1 --2- 172.21.15.1:0/1782025688 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f181c160940 0x7f181c0d0190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1810009870 tx=0x7f1810009b80 comp rx=0 tx=0).stop 2024-03-18T20:06:47.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.511+0000 7f182147b700 1 -- 172.21.15.1:0/1782025688 shutdown_connections 2024-03-18T20:06:47.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.511+0000 7f182147b700 1 --2- 172.21.15.1:0/1782025688 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f181c160940 0x7f181c0d0190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.511+0000 7f182147b700 1 -- 172.21.15.1:0/1782025688 >> 172.21.15.1:0/1782025688 conn(0x7f181c09c520 msgr2=0x7f181c0d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:47.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.511+0000 7f182147b700 1 -- 172.21.15.1:0/1782025688 shutdown_connections 2024-03-18T20:06:47.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.511+0000 7f182147b700 1 -- 172.21.15.1:0/1782025688 wait complete. 2024-03-18T20:06:47.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.513+0000 7f182147b700 1 Processor -- start 2024-03-18T20:06:47.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.513+0000 7f182147b700 1 -- start start 2024-03-18T20:06:47.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.513+0000 7f182147b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f181c0cf090 0x7f181c151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.513+0000 7f182147b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f181c0d1ea0 0x7f181c0cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.513+0000 7f182147b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f181c160940 0x7f181c0cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.513+0000 7f182147b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c0c7dd0 con 0x7f181c160940 2024-03-18T20:06:47.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.513+0000 7f182147b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c0cd100 con 0x7f181c0cf090 2024-03-18T20:06:47.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.513+0000 7f182147b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f181c151b70 con 0x7f181c0d1ea0 2024-03-18T20:06:47.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.514+0000 7f181b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f181c160940 0x7f181c0cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.514+0000 7f181b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f181c160940 0x7f181c0cf4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56570/0 (socket says 172.21.15.1:56570) 2024-03-18T20:06:47.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.514+0000 7f181b7fe700 1 -- 172.21.15.1:0/1304454218 learned_addr learned my addr 172.21.15.1:0/1304454218 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:47.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.514+0000 7f181affd700 1 --2- 172.21.15.1:0/1304454218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f181c0cf090 0x7f181c151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.514+0000 7f181a7fc700 1 --2- 172.21.15.1:0/1304454218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f181c0d1ea0 0x7f181c0cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.514+0000 7f181b7fe700 1 -- 172.21.15.1:0/1304454218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f181c0d1ea0 msgr2=0x7f181c0cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:47.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.514+0000 7f181b7fe700 1 --2- 172.21.15.1:0/1304454218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f181c0d1ea0 0x7f181c0cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.514+0000 7f181b7fe700 1 -- 172.21.15.1:0/1304454218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f181c0cf090 msgr2=0x7f181c151630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:47.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.514+0000 7f181b7fe700 1 --2- 172.21.15.1:0/1304454218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f181c0cf090 0x7f181c151630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.514+0000 7f181b7fe700 1 -- 172.21.15.1:0/1304454218 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18100094a0 con 0x7f181c160940 2024-03-18T20:06:47.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.514+0000 7f181a7fc700 1 --2- 172.21.15.1:0/1304454218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f181c0d1ea0 0x7f181c0cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:47.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.514+0000 7f181affd700 1 --2- 172.21.15.1:0/1304454218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f181c0cf090 0x7f181c151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:47.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.514+0000 7f181b7fe700 1 --2- 172.21.15.1:0/1304454218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f181c160940 0x7f181c0cf4b0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f181400cc50 tx=0x7f181400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:47.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.514+0000 7f18037fe700 1 -- 172.21.15.1:0/1304454218 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181400ade0 con 0x7f181c160940 2024-03-18T20:06:47.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.514+0000 7f18037fe700 1 -- 172.21.15.1:0/1304454218 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181400ce50 con 0x7f181c160940 2024-03-18T20:06:47.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.514+0000 7f18037fe700 1 -- 172.21.15.1:0/1304454218 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1814007310 con 0x7f181c160940 2024-03-18T20:06:47.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.514+0000 7f182147b700 1 -- 172.21.15.1:0/1304454218 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181c0cc900 con 0x7f181c160940 2024-03-18T20:06:47.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.514+0000 7f182147b700 1 -- 172.21.15.1:0/1304454218 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f181c0c7430 con 0x7f181c160940 2024-03-18T20:06:47.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.515+0000 7f182147b700 1 -- 172.21.15.1:0/1304454218 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181c15c850 con 0x7f181c160940 2024-03-18T20:06:47.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.516+0000 7f18037fe700 1 -- 172.21.15.1:0/1304454218 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1814007470 con 0x7f181c160940 2024-03-18T20:06:47.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.516+0000 7f18037fe700 1 --2- 172.21.15.1:0/1304454218 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f180806adb0 0x7f180806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:47.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.516+0000 7f18037fe700 1 -- 172.21.15.1:0/1304454218 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1814089250 con 0x7f181c160940 2024-03-18T20:06:47.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.516+0000 7f181affd700 1 --2- 172.21.15.1:0/1304454218 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f180806adb0 0x7f180806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:47.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.517+0000 7f181affd700 1 --2- 172.21.15.1:0/1304454218 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f180806adb0 0x7f180806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f18100090b0 tx=0x7f1810013040 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:47.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.521+0000 7f18037fe700 1 -- 172.21.15.1:0/1304454218 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1814054180 con 0x7f181c160940 2024-03-18T20:06:47.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.768+0000 7f182147b700 1 -- 172.21.15.1:0/1304454218 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f181c073350 con 0x7f181c160940 2024-03-18T20:06:47.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.768+0000 7f18037fe700 1 -- 172.21.15.1:0/1304454218 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v81) v1 ==== 53+0+73 (secure 0 0 0) 0x7f18140577a0 con 0x7f181c160940 2024-03-18T20:06:47.774 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T20:06:47.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.770+0000 7f182147b700 1 -- 172.21.15.1:0/1304454218 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f180806adb0 msgr2=0x7f180806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.770+0000 7f182147b700 1 --2- 172.21.15.1:0/1304454218 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f180806adb0 0x7f180806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f18100090b0 tx=0x7f1810013040 comp rx=0 tx=0).stop 2024-03-18T20:06:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.770+0000 7f182147b700 1 -- 172.21.15.1:0/1304454218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f181c160940 msgr2=0x7f181c0cf4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.770+0000 7f182147b700 1 --2- 172.21.15.1:0/1304454218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f181c160940 0x7f181c0cf4b0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f181400cc50 tx=0x7f181400a340 comp rx=0 tx=0).stop 2024-03-18T20:06:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.770+0000 7f182147b700 1 -- 172.21.15.1:0/1304454218 shutdown_connections 2024-03-18T20:06:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.770+0000 7f182147b700 1 --2- 172.21.15.1:0/1304454218 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f180806adb0 0x7f180806d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.770+0000 7f182147b700 1 --2- 172.21.15.1:0/1304454218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f181c160940 0x7f181c0cf4b0 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.770+0000 7f182147b700 1 --2- 172.21.15.1:0/1304454218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f181c0cf090 0x7f181c151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.770+0000 7f182147b700 1 --2- 172.21.15.1:0/1304454218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f181c0d1ea0 0x7f181c0cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.771+0000 7f182147b700 1 -- 172.21.15.1:0/1304454218 >> 172.21.15.1:0/1304454218 conn(0x7f181c09c520 msgr2=0x7f181c0b6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.771+0000 7f182147b700 1 -- 172.21.15.1:0/1304454218 shutdown_connections 2024-03-18T20:06:47.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:47.771+0000 7f182147b700 1 -- 172.21.15.1:0/1304454218 wait complete. 2024-03-18T20:06:47.789 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T20:06:47.789 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T20:06:47.789 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-18T20:06:47.789 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T20:06:47.789 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T20:06:47.789 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T20:06:47.789 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T20:06:47.789 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T20:06:47.789 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:06:47.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:47.790 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-18T20:06:47.809 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T20:06:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T20:06:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T20:06:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:06:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T20:06:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:06:47.809 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:06:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-18T20:06:47.809 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-18T20:06:47.810 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:06:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-18T20:06:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-18T20:06:47.810 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T20:06:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:06:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-18T20:06:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T20:06:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 37962sec preferred_lft 37962sec 2024-03-18T20:06:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link 2024-03-18T20:06:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:06:47.810 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:06:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:06:47.810 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-18T20:06:47.811 INFO:teuthology.orchestra.run.smithi001.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T20:06:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:06:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T20:06:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:06:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::4ca9:f5ff:feb3:79a3/64 scope link 2024-03-18T20:06:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:06:47.811 INFO:teuthology.orchestra.run.smithi001.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T20:06:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T20:06:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::282f:a4ff:fe1a:9908/64 scope link 2024-03-18T20:06:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:06:47.811 INFO:teuthology.orchestra.run.smithi001.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T20:06:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2e:5c:60:78:b2:8c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T20:06:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::2c5c:60ff:fe78:b28c/64 scope link 2024-03-18T20:06:47.811 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:06:47.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:47.812 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T20:06:47.876 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-18T20:06:47.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:47.878 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-18T20:06:47.939 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-18T20:06:47.939 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T20:06:47.940 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T20:06:47.940 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T20:06:47.940 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T20:06:47.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:47.940 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T20:06:47.940 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T20:06:47.940 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T20:06:47.940 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T20:06:47.940 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T20:06:47.941 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T20:06:48.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:48.046 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T20:06:48.064 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T20:06:48.064 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T20:06:48.065 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:06:48.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:48.065 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-18T20:06:48.133 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T20:06:48.133 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:06:48.133 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T20:06:48.190 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:06:48.191 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:06:48.191 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T20:06:48.248 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T20:06:48.317 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:06:48.317 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:06:48.385 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:06:48.386 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:06:48.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:48.387 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T20:06:48.445 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T20:06:48.445 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T20:06:48.488 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T20:06:48.515 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:06:48.515 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:06:48.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-18T20:06:48.569+0000 7f056a28e5c0 -1 init, newargv = 0x564b302a74a0 newargc=15 2024-03-18T20:06:48.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[142816]: starting ceph client 2024-03-18T20:06:48.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[142816]: starting fuse 2024-03-18T20:06:48.585 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:06:48.586 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:06:48.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:48.586 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T20:06:48.645 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T20:06:48.645 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-03-18T20:06:48.645 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T20:06:48.645 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-18T20:06:48.645 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-18T20:06:48.645 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-18T20:06:48.646 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-18T20:06:48.646 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:06:48.646 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-18T20:06:48.646 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T20:06:48.646 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-18T20:06:48.646 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T20:06:48.646 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:06:48.646 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-18T20:06:48.646 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-18T20:06:48.646 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-18T20:06:48.646 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-18T20:06:48.646 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:06:48.646 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-18T20:06:48.646 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T20:06:48.646 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T20:06:48.647 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T20:06:48.647 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-18T20:06:48.647 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-18T20:06:48.647 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-18T20:06:48.647 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T20:06:48.647 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:06:48.647 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-18T20:06:48.647 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-18T20:06:48.770 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.142816.asok 2024-03-18T20:06:48.951 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.142816.asok 2024-03-18T20:06:48.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:48.952 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.142816.asok status 2024-03-18T20:06:49.113 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:06:49.113 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-18T20:06:49.114 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T20:06:49.114 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T20:06:49.114 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-18T20:06:49.114 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-18T20:06:49.114 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T20:06:49.114 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "142816", 2024-03-18T20:06:49.114 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-18T20:06:49.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:49.115 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-18T20:06:49.115 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-18T20:06:49.115 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 10074, 2024-03-18T20:06:49.115 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-18T20:06:49.115 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-18T20:06:49.115 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-18T20:06:49.116 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 10074 2024-03-18T20:06:49.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:49.116 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T20:06:49.116 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T20:06:49.116 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:06:49.116 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1610184568 2024-03-18T20:06:49.116 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:49.116 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:49.117 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T20:06:49.117 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T20:06:49.117 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:06:49.117 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1610184568 2024-03-18T20:06:49.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:06:49.117 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.10074 192.168.0.1:0/1610184568", 2024-03-18T20:06:49.117 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/1610184568", 2024-03-18T20:06:49.117 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-18T20:06:49.117 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 81, 2024-03-18T20:06:49.117 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 181, 2024-03-18T20:06:49.117 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-18T20:06:49.117 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-18T20:06:49.117 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-18T20:06:49.118 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:06:49.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:49.124 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-18T20:06:49.124 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T20:06:49.142 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-18T20:06:49.142 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T20:06:49.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:49.143 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T20:06:49.211 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-18T20:06:49.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.373+0000 7f003a950700 1 Processor -- start 2024-03-18T20:06:49.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.373+0000 7f003a950700 1 -- start start 2024-03-18T20:06:49.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.374+0000 7f003a950700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0034071cc0 0x7f00340ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:49.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.374+0000 7f003a950700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00340cac80 con 0x7f003415a240 2024-03-18T20:06:49.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.374+0000 7f003a950700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00340caf60 con 0x7f003406d560 2024-03-18T20:06:49.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.374+0000 7f003a950700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00340adff0 con 0x7f0034071cc0 2024-03-18T20:06:49.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.374+0000 7f00337fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0034071cc0 0x7f00340ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:49.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.374+0000 7f0033fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f003415a240 0x7f00340715b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49192/0 (socket says 172.21.15.1:49192) 2024-03-18T20:06:49.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.374+0000 7f0033fff700 1 -- 172.21.15.1:0/2552941252 learned_addr learned my addr 172.21.15.1:0/2552941252 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:49.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.375+0000 7f00337fe700 1 -- 172.21.15.1:0/2552941252 >> v1:172.21.15.165:6789/0 conn(0x7f003406d560 legacy=0x7f00340ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:49.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.375+0000 7f00337fe700 1 -- 172.21.15.1:0/2552941252 >> v1:172.21.15.1:6789/0 conn(0x7f003415a240 legacy=0x7f00340715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:49.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.375+0000 7f0032ffd700 1 -- 172.21.15.1:0/2552941252 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882523247 0 0) 0x7f00340cac80 con 0x7f003415a240 2024-03-18T20:06:49.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.375+0000 7f00337fe700 1 -- 172.21.15.1:0/2552941252 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00340cb240 con 0x7f0034071cc0 2024-03-18T20:06:49.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.375+0000 7f00337fe700 1 --2- 172.21.15.1:0/2552941252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0034071cc0 0x7f00340ca580 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f0024009840 tx=0x7f0024009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c46545b2981a0cb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:49.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.375+0000 7f0032ffd700 1 -- 172.21.15.1:0/2552941252 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024004230 con 0x7f0034071cc0 2024-03-18T20:06:49.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.375+0000 7f0032ffd700 1 -- 172.21.15.1:0/2552941252 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0024004390 con 0x7f0034071cc0 2024-03-18T20:06:49.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.375+0000 7f0032ffd700 1 -- 172.21.15.1:0/2552941252 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00240036a0 con 0x7f0034071cc0 2024-03-18T20:06:49.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.375+0000 7f003a950700 1 -- 172.21.15.1:0/2552941252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0034071cc0 msgr2=0x7f00340ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:49.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.375+0000 7f003a950700 1 --2- 172.21.15.1:0/2552941252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0034071cc0 0x7f00340ca580 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f0024009840 tx=0x7f0024009b50 comp rx=0 tx=0).stop 2024-03-18T20:06:49.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.376+0000 7f003a950700 1 -- 172.21.15.1:0/2552941252 shutdown_connections 2024-03-18T20:06:49.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.376+0000 7f003a950700 1 --2- 172.21.15.1:0/2552941252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0034071cc0 0x7f00340ca580 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:49.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.376+0000 7f003a950700 1 -- 172.21.15.1:0/2552941252 >> 172.21.15.1:0/2552941252 conn(0x7f00340cff30 msgr2=0x7f00340d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:49.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.376+0000 7f003a950700 1 -- 172.21.15.1:0/2552941252 shutdown_connections 2024-03-18T20:06:49.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.376+0000 7f003a950700 1 -- 172.21.15.1:0/2552941252 wait complete. 2024-03-18T20:06:49.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.377+0000 7f003a950700 1 Processor -- start 2024-03-18T20:06:49.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.377+0000 7f003a950700 1 -- start start 2024-03-18T20:06:49.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f003a950700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f003406d560 0x7f0034150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:49.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f003a950700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0034071cc0 0x7f00340c9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f003a950700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f003415a240 0x7f00340c7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f003a950700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00340c9de0 con 0x7f0034071cc0 2024-03-18T20:06:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f003a950700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00340ca0c0 con 0x7f003406d560 2024-03-18T20:06:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f003a950700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00340c4a80 con 0x7f003415a240 2024-03-18T20:06:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f00337fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0034071cc0 0x7f00340c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f00337fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0034071cc0 0x7f00340c9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56590/0 (socket says 172.21.15.1:56590) 2024-03-18T20:06:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f00337fe700 1 -- 172.21.15.1:0/2138111074 learned_addr learned my addr 172.21.15.1:0/2138111074 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f0033fff700 1 --2- 172.21.15.1:0/2138111074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f003406d560 0x7f0034150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f0038eed700 1 --2- 172.21.15.1:0/2138111074 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f003415a240 0x7f00340c7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f00337fe700 1 -- 172.21.15.1:0/2138111074 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f003415a240 msgr2=0x7f00340c7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f00337fe700 1 --2- 172.21.15.1:0/2138111074 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f003415a240 0x7f00340c7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f00337fe700 1 -- 172.21.15.1:0/2138111074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f003406d560 msgr2=0x7f0034150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f00337fe700 1 --2- 172.21.15.1:0/2138111074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f003406d560 0x7f0034150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:49.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f00337fe700 1 -- 172.21.15.1:0/2138111074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00240094a0 con 0x7f0034071cc0 2024-03-18T20:06:49.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f00337fe700 1 --2- 172.21.15.1:0/2138111074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0034071cc0 0x7f00340c9450 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f00240090b0 tx=0x7f0024004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:49.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f0030ff9700 1 -- 172.21.15.1:0/2138111074 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024011630 con 0x7f0034071cc0 2024-03-18T20:06:49.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f0030ff9700 1 -- 172.21.15.1:0/2138111074 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002401e070 con 0x7f0034071cc0 2024-03-18T20:06:49.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f003a950700 1 -- 172.21.15.1:0/2138111074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00340c4d60 con 0x7f0034071cc0 2024-03-18T20:06:49.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.378+0000 7f0030ff9700 1 -- 172.21.15.1:0/2138111074 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00240119d0 con 0x7f0034071cc0 2024-03-18T20:06:49.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.379+0000 7f003a950700 1 -- 172.21.15.1:0/2138111074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00341511f0 con 0x7f0034071cc0 2024-03-18T20:06:49.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.380+0000 7f003a950700 1 -- 172.21.15.1:0/2138111074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00340c5280 con 0x7f0034071cc0 2024-03-18T20:06:49.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.380+0000 7f0030ff9700 1 -- 172.21.15.1:0/2138111074 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0024011b30 con 0x7f0034071cc0 2024-03-18T20:06:49.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.381+0000 7f0030ff9700 1 --2- 172.21.15.1:0/2138111074 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f002006adb0 0x7f002006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:49.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.381+0000 7f0030ff9700 1 -- 172.21.15.1:0/2138111074 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f002408a810 con 0x7f0034071cc0 2024-03-18T20:06:49.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.381+0000 7f0033fff700 1 --2- 172.21.15.1:0/2138111074 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f002006adb0 0x7f002006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:49.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.381+0000 7f0033fff700 1 --2- 172.21.15.1:0/2138111074 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f002006adb0 0x7f002006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0028000f60 tx=0x7f0028007e70 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:49.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.385+0000 7f0030ff9700 1 -- 172.21.15.1:0/2138111074 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f00340c5280 con 0x7f0034071cc0 2024-03-18T20:06:49.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.623+0000 7f003a950700 1 -- 172.21.15.1:0/2138111074 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f003415a620 con 0x7f002006adb0 2024-03-18T20:06:49.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.624+0000 7f0030ff9700 1 -- 172.21.15.1:0/2138111074 <== mgr.7721 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f00280133f0 con 0x7f002006adb0 2024-03-18T20:06:49.630 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T20:06:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:06:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephfs" 2024-03-18T20:06:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:06:49.630 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T20:06:49.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.626+0000 7f003a950700 1 -- 172.21.15.1:0/2138111074 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f002006adb0 msgr2=0x7f002006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:49.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.626+0000 7f003a950700 1 --2- 172.21.15.1:0/2138111074 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f002006adb0 0x7f002006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0028000f60 tx=0x7f0028007e70 comp rx=0 tx=0).stop 2024-03-18T20:06:49.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.626+0000 7f003a950700 1 -- 172.21.15.1:0/2138111074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0034071cc0 msgr2=0x7f00340c9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:49.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.626+0000 7f003a950700 1 --2- 172.21.15.1:0/2138111074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0034071cc0 0x7f00340c9450 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f00240090b0 tx=0x7f0024004750 comp rx=0 tx=0).stop 2024-03-18T20:06:49.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.626+0000 7f003a950700 1 -- 172.21.15.1:0/2138111074 shutdown_connections 2024-03-18T20:06:49.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.626+0000 7f003a950700 1 --2- 172.21.15.1:0/2138111074 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f002006adb0 0x7f002006d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.626+0000 7f003a950700 1 --2- 172.21.15.1:0/2138111074 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f003415a240 0x7f00340c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.626+0000 7f003a950700 1 --2- 172.21.15.1:0/2138111074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f003406d560 0x7f0034150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.626+0000 7f003a950700 1 --2- 172.21.15.1:0/2138111074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0034071cc0 0x7f00340c9450 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.626+0000 7f003a950700 1 -- 172.21.15.1:0/2138111074 >> 172.21.15.1:0/2138111074 conn(0x7f00340cff30 msgr2=0x7f00340b0420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.626+0000 7f003a950700 1 -- 172.21.15.1:0/2138111074 shutdown_connections 2024-03-18T20:06:49.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.626+0000 7f003a950700 1 -- 172.21.15.1:0/2138111074 wait complete. 2024-03-18T20:06:49.644 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-18T20:06:49.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.765+0000 7fb2c7eaf700 1 Processor -- start 2024-03-18T20:06:49.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.765+0000 7fb2c7eaf700 1 -- start start 2024-03-18T20:06:49.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.766+0000 7fb2c7eaf700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb2c00b1e20 0x7fb2c00cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:49.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.766+0000 7fb2c7eaf700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c00d0df0 con 0x7fb2c00cbce0 2024-03-18T20:06:49.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.766+0000 7fb2c7eaf700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c00d0550 con 0x7fb2c00cd870 2024-03-18T20:06:49.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.766+0000 7fb2c7eaf700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2c00d0830 con 0x7fb2c00b1e20 2024-03-18T20:06:49.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.766+0000 7fb2c644c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb2c00cbce0 0x7fb2c00cc150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49198/0 (socket says 172.21.15.1:49198) 2024-03-18T20:06:49.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.766+0000 7fb2c644c700 1 -- 172.21.15.1:0/3840205105 learned_addr learned my addr 172.21.15.1:0/3840205105 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:49.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.766+0000 7fb2c544a700 1 --2- 172.21.15.1:0/3840205105 >> v2:172.21.15.165:3300/0 conn(0x7fb2c00b1e20 0x7fb2c00cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:49.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.766+0000 7fb2c4c49700 1 -- 172.21.15.1:0/3840205105 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114921101 0 0) 0x7fb2c00d0df0 con 0x7fb2c00cbce0 2024-03-18T20:06:49.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.766+0000 7fb2c4c49700 1 -- 172.21.15.1:0/3840205105 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2ac0035b0 con 0x7fb2c00cbce0 2024-03-18T20:06:49.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.766+0000 7fb2c4c49700 1 -- 172.21.15.1:0/3840205105 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1208069927 0 0) 0x7fb2ac0035b0 con 0x7fb2c00cbce0 2024-03-18T20:06:49.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.766+0000 7fb2c4c49700 1 -- 172.21.15.1:0/3840205105 >> v2:172.21.15.165:3300/0 conn(0x7fb2c00b1e20 msgr2=0x7fb2c00cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:49.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.766+0000 7fb2c4c49700 1 --2- 172.21.15.1:0/3840205105 >> v2:172.21.15.165:3300/0 conn(0x7fb2c00b1e20 0x7fb2c00cff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:49.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.766+0000 7fb2c4c49700 1 -- 172.21.15.1:0/3840205105 >> v1:172.21.15.165:6789/0 conn(0x7fb2c00cd870 legacy=0x7fb2c00cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:49.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c4c49700 1 -- 172.21.15.1:0/3840205105 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c00d0b10 con 0x7fb2c00cbce0 2024-03-18T20:06:49.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c4c49700 1 -- 172.21.15.1:0/3840205105 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fb2b80019f0 con 0x7fb2c00cbce0 2024-03-18T20:06:49.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c544a700 1 --2- 172.21.15.1:0/3840205105 >> v2:172.21.15.165:3300/0 conn(0x7fb2c00b1e20 0x7fb2c00cff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:49.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c4c49700 1 -- 172.21.15.1:0/3840205105 >> v1:172.21.15.1:6789/0 conn(0x7fb2c00cbce0 legacy=0x7fb2c00cc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:49.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c4c49700 1 --2- 172.21.15.1:0/3840205105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2ac003a00 0x7fb2ac003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:49.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c4c49700 1 --2- 172.21.15.1:0/3840205105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2ac004850 0x7fb2ac004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:49.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c4c49700 1 --2- 172.21.15.1:0/3840205105 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2ac005210 0x7fb2ac00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:49.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c4c49700 1 -- 172.21.15.1:0/3840205105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c00d0830 con 0x7fb2ac003a00 2024-03-18T20:06:49.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c4c49700 1 -- 172.21.15.1:0/3840205105 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ac00c0b0 con 0x7fb2ac005210 2024-03-18T20:06:49.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c4c49700 1 -- 172.21.15.1:0/3840205105 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ac00c270 con 0x7fb2ac004850 2024-03-18T20:06:49.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c5c4b700 1 --2- 172.21.15.1:0/3840205105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2ac003a00 0x7fb2ac003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:49.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c644c700 1 --2- 172.21.15.1:0/3840205105 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2ac005210 0x7fb2ac00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:49.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c544a700 1 --2- 172.21.15.1:0/3840205105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2ac004850 0x7fb2ac004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:49.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c5c4b700 1 -- 172.21.15.1:0/3840205105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2ac004850 msgr2=0x7fb2ac004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:49.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c5c4b700 1 --2- 172.21.15.1:0/3840205105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2ac004850 0x7fb2ac004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:49.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c5c4b700 1 -- 172.21.15.1:0/3840205105 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2ac005210 msgr2=0x7fb2ac00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:49.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c5c4b700 1 --2- 172.21.15.1:0/3840205105 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2ac005210 0x7fb2ac00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:49.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c5c4b700 1 -- 172.21.15.1:0/3840205105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ac0035b0 con 0x7fb2ac003a00 2024-03-18T20:06:49.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c644c700 1 -- 172.21.15.1:0/3840205105 reap_dead start 2024-03-18T20:06:49.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c544a700 1 --2- 172.21.15.1:0/3840205105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2ac004850 0x7fb2ac004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:49.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c5c4b700 1 --2- 172.21.15.1:0/3840205105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2ac003a00 0x7fb2ac003e20 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fb2b00017c0 tx=0x7fb2ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968252d0b20f4958 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:49.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c4c49700 1 -- 172.21.15.1:0/3840205105 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b00052d0 con 0x7fb2ac003a00 2024-03-18T20:06:49.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c4c49700 1 -- 172.21.15.1:0/3840205105 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b0005450 con 0x7fb2ac003a00 2024-03-18T20:06:49.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.767+0000 7fb2c4c49700 1 -- 172.21.15.1:0/3840205105 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b00038c0 con 0x7fb2ac003a00 2024-03-18T20:06:49.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.768+0000 7fb2c7eaf700 1 -- 172.21.15.1:0/3840205105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2ac003a00 msgr2=0x7fb2ac003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:49.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.768+0000 7fb2c7eaf700 1 --2- 172.21.15.1:0/3840205105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2ac003a00 0x7fb2ac003e20 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fb2b00017c0 tx=0x7fb2ac005880 comp rx=0 tx=0).stop 2024-03-18T20:06:49.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.768+0000 7fb2c7eaf700 1 -- 172.21.15.1:0/3840205105 shutdown_connections 2024-03-18T20:06:49.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.768+0000 7fb2c7eaf700 1 --2- 172.21.15.1:0/3840205105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2ac003a00 0x7fb2ac003e20 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:49.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.768+0000 7fb2c7eaf700 1 -- 172.21.15.1:0/3840205105 >> 172.21.15.1:0/3840205105 conn(0x7fb2c00d0ff0 msgr2=0x7fb2c00d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:49.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.768+0000 7fb2c7eaf700 1 -- 172.21.15.1:0/3840205105 shutdown_connections 2024-03-18T20:06:49.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.768+0000 7fb2c7eaf700 1 -- 172.21.15.1:0/3840205105 wait complete. 2024-03-18T20:06:49.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.769+0000 7fb2c7eaf700 1 Processor -- start 2024-03-18T20:06:49.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.769+0000 7fb2c7eaf700 1 -- start start 2024-03-18T20:06:49.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.770+0000 7fb2c7eaf700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2ac003a00 0x7fb2c0147aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:49.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.770+0000 7fb2c7eaf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2c01f2af0 0x7fb2c0145fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:49.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.770+0000 7fb2c7eaf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2c0146510 0x7fb2c00c4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:49.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.770+0000 7fb2c7eaf700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c0146a00 con 0x7fb2c0146510 2024-03-18T20:06:49.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.770+0000 7fb2c7eaf700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c0146ce0 con 0x7fb2ac003a00 2024-03-18T20:06:49.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.770+0000 7fb2c7eaf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c0146fc0 con 0x7fb2c01f2af0 2024-03-18T20:06:49.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.770+0000 7fb2c644c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2c0146510 0x7fb2c00c4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:49.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.770+0000 7fb2c644c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2c0146510 0x7fb2c00c4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56596/0 (socket says 172.21.15.1:56596) 2024-03-18T20:06:49.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.770+0000 7fb2c644c700 1 -- 172.21.15.1:0/1221598761 learned_addr learned my addr 172.21.15.1:0/1221598761 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:49.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.770+0000 7fb2c5c4b700 1 --2- 172.21.15.1:0/1221598761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2ac003a00 0x7fb2c0147aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:49.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.770+0000 7fb2c544a700 1 --2- 172.21.15.1:0/1221598761 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2c01f2af0 0x7fb2c0145fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:49.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.770+0000 7fb2c644c700 1 -- 172.21.15.1:0/1221598761 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2c01f2af0 msgr2=0x7fb2c0145fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:49.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.770+0000 7fb2c644c700 1 --2- 172.21.15.1:0/1221598761 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2c01f2af0 0x7fb2c0145fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:49.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.770+0000 7fb2c644c700 1 -- 172.21.15.1:0/1221598761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2ac003a00 msgr2=0x7fb2c0147aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:49.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.770+0000 7fb2c644c700 1 --2- 172.21.15.1:0/1221598761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2ac003a00 0x7fb2c0147aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:49.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.770+0000 7fb2c644c700 1 -- 172.21.15.1:0/1221598761 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c01472a0 con 0x7fb2c0146510 2024-03-18T20:06:49.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.770+0000 7fb2c5c4b700 1 --2- 172.21.15.1:0/1221598761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2ac003a00 0x7fb2c0147aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:06:49.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.771+0000 7fb2c644c700 1 --2- 172.21.15.1:0/1221598761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2c0146510 0x7fb2c00c4b20 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fb2c006e650 tx=0x7fb2b800a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:49.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.771+0000 7fb2b67fc700 1 -- 172.21.15.1:0/1221598761 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b800ad40 con 0x7fb2c0146510 2024-03-18T20:06:49.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.771+0000 7fb2b67fc700 1 -- 172.21.15.1:0/1221598761 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b80133f0 con 0x7fb2c0146510 2024-03-18T20:06:49.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.771+0000 7fb2b67fc700 1 -- 172.21.15.1:0/1221598761 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b800aea0 con 0x7fb2c0146510 2024-03-18T20:06:49.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.771+0000 7fb2c7eaf700 1 -- 172.21.15.1:0/1221598761 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c0147580 con 0x7fb2c0146510 2024-03-18T20:06:49.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.771+0000 7fb2c7eaf700 1 -- 172.21.15.1:0/1221598761 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c01597b0 con 0x7fb2c0146510 2024-03-18T20:06:49.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.772+0000 7fb2c7eaf700 1 -- 172.21.15.1:0/1221598761 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c0159940 con 0x7fb2c0146510 2024-03-18T20:06:49.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.772+0000 7fb2b67fc700 1 -- 172.21.15.1:0/1221598761 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 95) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb2b8013760 con 0x7fb2c0146510 2024-03-18T20:06:49.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.773+0000 7fb2b67fc700 1 --2- 172.21.15.1:0/1221598761 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb2a406adb0 0x7fb2a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:49.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.773+0000 7fb2b67fc700 1 -- 172.21.15.1:0/1221598761 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb2b808a1b0 con 0x7fb2c0146510 2024-03-18T20:06:49.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.773+0000 7fb2c5c4b700 1 --2- 172.21.15.1:0/1221598761 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb2a406adb0 0x7fb2a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:49.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.773+0000 7fb2c5c4b700 1 --2- 172.21.15.1:0/1221598761 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb2a406adb0 0x7fb2a406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0015040 tx=0x7fb2b0009030 comp rx=0 tx=0).ready entity=mgr.7721 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:49.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:49.777+0000 7fb2b67fc700 1 -- 172.21.15.1:0/1221598761 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb2b80550f0 con 0x7fb2c0146510 2024-03-18T20:06:50.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.042+0000 7fb2c7eaf700 1 -- 172.21.15.1:0/1221598761 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fb2c00676a0 con 0x7fb2c0146510 2024-03-18T20:06:50.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.839+0000 7fb2b67fc700 1 -- 172.21.15.1:0/1221598761 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 96) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fb2b8088540 con 0x7fb2c0146510 2024-03-18T20:06:50.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.840+0000 7fb2b67fc700 1 -- 172.21.15.1:0/1221598761 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v96) v1 ==== 92+0+0 (secure 0 0 0) 0x7fb2b8087750 con 0x7fb2c0146510 2024-03-18T20:06:50.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.843+0000 7fb2c7eaf700 1 -- 172.21.15.1:0/1221598761 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb2a406adb0 msgr2=0x7fb2a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:50.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.843+0000 7fb2c7eaf700 1 --2- 172.21.15.1:0/1221598761 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb2a406adb0 0x7fb2a406d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0015040 tx=0x7fb2b0009030 comp rx=0 tx=0).stop 2024-03-18T20:06:50.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.843+0000 7fb2c7eaf700 1 -- 172.21.15.1:0/1221598761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2c0146510 msgr2=0x7fb2c00c4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:50.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.843+0000 7fb2c7eaf700 1 --2- 172.21.15.1:0/1221598761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2c0146510 0x7fb2c00c4b20 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fb2c006e650 tx=0x7fb2b800a5f0 comp rx=0 tx=0).stop 2024-03-18T20:06:50.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.843+0000 7fb2c7eaf700 1 -- 172.21.15.1:0/1221598761 shutdown_connections 2024-03-18T20:06:50.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.843+0000 7fb2c7eaf700 1 --2- 172.21.15.1:0/1221598761 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb2a406adb0 0x7fb2a406d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:50.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.843+0000 7fb2c7eaf700 1 --2- 172.21.15.1:0/1221598761 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2c0146510 0x7fb2c00c4b20 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:50.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.843+0000 7fb2c7eaf700 1 --2- 172.21.15.1:0/1221598761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2ac003a00 0x7fb2c0147aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:50.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.843+0000 7fb2c7eaf700 1 --2- 172.21.15.1:0/1221598761 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2c01f2af0 0x7fb2c0145fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:50.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.843+0000 7fb2c7eaf700 1 -- 172.21.15.1:0/1221598761 >> 172.21.15.1:0/1221598761 conn(0x7fb2c00d0ff0 msgr2=0x7fb2c006f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:50.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.843+0000 7fb2c7eaf700 1 -- 172.21.15.1:0/1221598761 shutdown_connections 2024-03-18T20:06:50.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.843+0000 7fb2c7eaf700 1 -- 172.21.15.1:0/1221598761 wait complete. 2024-03-18T20:06:50.865 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-18T20:06:50.988 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:50.983+0000 7f27cc459240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:06:51.002 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:50.996+0000 7f2a2119a240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:06:51.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.997+0000 7f2017e77700 1 Processor -- start 2024-03-18T20:06:51.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.997+0000 7f2017e77700 1 -- start start 2024-03-18T20:06:51.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.998+0000 7f2017e77700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f201015b750 0x7f201015bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:51.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.998+0000 7f2017e77700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2010070db0 0x7f2010071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:51.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.998+0000 7f2017e77700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20100cbcc0 con 0x7f201013e130 2024-03-18T20:06:51.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.998+0000 7f2017e77700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f201013eff0 con 0x7f201015b750 2024-03-18T20:06:51.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.998+0000 7f2017e77700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2010071760 con 0x7f2010070db0 2024-03-18T20:06:51.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.998+0000 7f2015412700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f201015b750 0x7f201015bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:51.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.998+0000 7f2015c13700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f201013e130 0x7f201013e8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49202/0 (socket says 172.21.15.1:49202) 2024-03-18T20:06:51.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.998+0000 7f2015412700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f201015b750 0x7f201015bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56626/0 (socket says 172.21.15.1:56626) 2024-03-18T20:06:51.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.998+0000 7f2015c13700 1 -- 172.21.15.1:0/2924104443 learned_addr learned my addr 172.21.15.1:0/2924104443 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:51.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.998+0000 7f2016414700 1 --2- 172.21.15.1:0/2924104443 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2010070db0 0x7f2010071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:51.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.999+0000 7f2015412700 1 -- 172.21.15.1:0/2924104443 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2010070db0 msgr2=0x7f2010071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:51.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.999+0000 7f2015412700 1 --2- 172.21.15.1:0/2924104443 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2010070db0 0x7f2010071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.999+0000 7f2015412700 1 -- 172.21.15.1:0/2924104443 >> v1:172.21.15.1:6789/0 conn(0x7f201013e130 legacy=0x7f201013e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:51.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.999+0000 7f2015412700 1 -- 172.21.15.1:0/2924104443 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201015c210 con 0x7f201015b750 2024-03-18T20:06:51.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.999+0000 7f2014c11700 1 -- 172.21.15.1:0/2924104443 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384014313 0 0) 0x7f20100cbcc0 con 0x7f201013e130 2024-03-18T20:06:51.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.999+0000 7f2015412700 1 --2- 172.21.15.1:0/2924104443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f201015b750 0x7f201015bbe0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f2000009840 tx=0x7f2000009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e989eca115ff77c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:51.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.999+0000 7f2014c11700 1 -- 172.21.15.1:0/2924104443 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2000011830 con 0x7f201015b750 2024-03-18T20:06:51.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.999+0000 7f2014c11700 1 -- 172.21.15.1:0/2924104443 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2000011990 con 0x7f201015b750 2024-03-18T20:06:51.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:50.999+0000 7f2014c11700 1 -- 172.21.15.1:0/2924104443 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2000011d20 con 0x7f201015b750 2024-03-18T20:06:51.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.000+0000 7f2017e77700 1 -- 172.21.15.1:0/2924104443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f201015b750 msgr2=0x7f201015bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:51.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.000+0000 7f2017e77700 1 --2- 172.21.15.1:0/2924104443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f201015b750 0x7f201015bbe0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f2000009840 tx=0x7f2000009b50 comp rx=0 tx=0).stop 2024-03-18T20:06:51.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.000+0000 7f2017e77700 1 -- 172.21.15.1:0/2924104443 shutdown_connections 2024-03-18T20:06:51.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.000+0000 7f2017e77700 1 --2- 172.21.15.1:0/2924104443 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2010070db0 0x7f2010071220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.000+0000 7f2017e77700 1 --2- 172.21.15.1:0/2924104443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f201015b750 0x7f201015bbe0 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.000+0000 7f2017e77700 1 -- 172.21.15.1:0/2924104443 >> 172.21.15.1:0/2924104443 conn(0x7f201006a750 msgr2=0x7f201006ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:51.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.000+0000 7f2017e77700 1 -- 172.21.15.1:0/2924104443 shutdown_connections 2024-03-18T20:06:51.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.000+0000 7f2017e77700 1 -- 172.21.15.1:0/2924104443 wait complete. 2024-03-18T20:06:51.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.001+0000 7f2017e77700 1 Processor -- start 2024-03-18T20:06:51.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.001+0000 7f2017e77700 1 -- start start 2024-03-18T20:06:51.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.002+0000 7f2017e77700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2010070db0 0x7f20100cac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:51.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.002+0000 7f2017e77700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f201013e130 0x7f20100cf090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:51.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.002+0000 7f2017e77700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f201015b750 0x7f20101528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:51.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.002+0000 7f2017e77700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20100b2860 con 0x7f2010070db0 2024-03-18T20:06:51.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.002+0000 7f2017e77700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20100b2b20 con 0x7f201015b750 2024-03-18T20:06:51.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.002+0000 7f2017e77700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20100b2e00 con 0x7f201013e130 2024-03-18T20:06:51.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.002+0000 7f2015c13700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2010070db0 0x7f20100cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:51.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.002+0000 7f2015c13700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2010070db0 0x7f20100cac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56642/0 (socket says 172.21.15.1:56642) 2024-03-18T20:06:51.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.003+0000 7f2015c13700 1 -- 172.21.15.1:0/587836137 learned_addr learned my addr 172.21.15.1:0/587836137 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:51.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.003+0000 7f2015412700 1 --2- 172.21.15.1:0/587836137 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f201013e130 0x7f20100cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:51.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.003+0000 7f2016414700 1 --2- 172.21.15.1:0/587836137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f201015b750 0x7f20101528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:51.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.003+0000 7f2015c13700 1 -- 172.21.15.1:0/587836137 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f201013e130 msgr2=0x7f20100cf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:51.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.003+0000 7f2015c13700 1 --2- 172.21.15.1:0/587836137 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f201013e130 0x7f20100cf090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.003+0000 7f2015c13700 1 -- 172.21.15.1:0/587836137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f201015b750 msgr2=0x7f20101528e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:51.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.003+0000 7f2015c13700 1 --2- 172.21.15.1:0/587836137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f201015b750 0x7f20101528e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.003+0000 7f2015c13700 1 -- 172.21.15.1:0/587836137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20000094a0 con 0x7f2010070db0 2024-03-18T20:06:51.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.003+0000 7f2016414700 1 --2- 172.21.15.1:0/587836137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f201015b750 0x7f20101528e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:51.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.003+0000 7f2015c13700 1 --2- 172.21.15.1:0/587836137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2010070db0 0x7f20100cac50 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f200c0011e0 tx=0x7f200c00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:51.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.003+0000 7f20067fc700 1 -- 172.21.15.1:0/587836137 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c004c80 con 0x7f2010070db0 2024-03-18T20:06:51.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.003+0000 7f20067fc700 1 -- 172.21.15.1:0/587836137 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200c004de0 con 0x7f2010070db0 2024-03-18T20:06:51.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.003+0000 7f2017e77700 1 -- 172.21.15.1:0/587836137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20100b30e0 con 0x7f2010070db0 2024-03-18T20:06:51.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.003+0000 7f20067fc700 1 -- 172.21.15.1:0/587836137 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c007310 con 0x7f2010070db0 2024-03-18T20:06:51.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.003+0000 7f2017e77700 1 -- 172.21.15.1:0/587836137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20100cf5d0 con 0x7f2010070db0 2024-03-18T20:06:51.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.004+0000 7f2017e77700 1 -- 172.21.15.1:0/587836137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20100cf7c0 con 0x7f2010070db0 2024-03-18T20:06:51.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.005+0000 7f20067fc700 1 -- 172.21.15.1:0/587836137 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 96) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f200c007470 con 0x7f2010070db0 2024-03-18T20:06:51.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.005+0000 7f20067fc700 1 --2- 172.21.15.1:0/587836137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1ff406ae00 0x7f1ff406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:51.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.005+0000 7f2015412700 1 -- 172.21.15.1:0/587836137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1ff406ae00 msgr2=0x7f1ff406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:51.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.005+0000 7f20067fc700 1 -- 172.21.15.1:0/587836137 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f200c089390 con 0x7f2010070db0 2024-03-18T20:06:51.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.005+0000 7f2015412700 1 --2- 172.21.15.1:0/587836137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1ff406ae00 0x7f1ff406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:06:51.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.010+0000 7f20067fc700 1 -- 172.21.15.1:0/587836137 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f200c0542d0 con 0x7f2010070db0 2024-03-18T20:06:51.142 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:51.133+0000 7f27cc459240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:06:51.172 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:51.166+0000 7f2a2119a240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:06:51.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.206+0000 7f2015412700 1 -- 172.21.15.1:0/587836137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1ff406ae00 msgr2=0x7f1ff406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:51.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.206+0000 7f2015412700 1 --2- 172.21.15.1:0/587836137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1ff406ae00 0x7f1ff406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:06:51.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.261+0000 7f2017e77700 1 -- 172.21.15.1:0/587836137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f20100676a0 con 0x7f2010070db0 2024-03-18T20:06:51.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.265+0000 7f20067fc700 1 -- 172.21.15.1:0/587836137 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v59)=0 v59) v1 ==== 138+0+0 (secure 0 0 0) 0x7f200c0578f0 con 0x7f2010070db0 2024-03-18T20:06:51.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.268+0000 7f2017e77700 1 -- 172.21.15.1:0/587836137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1ff406ae00 msgr2=0x7f1ff406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:06:51.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.268+0000 7f2017e77700 1 --2- 172.21.15.1:0/587836137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1ff406ae00 0x7f1ff406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.268+0000 7f2017e77700 1 -- 172.21.15.1:0/587836137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2010070db0 msgr2=0x7f20100cac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:51.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.268+0000 7f2017e77700 1 --2- 172.21.15.1:0/587836137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2010070db0 0x7f20100cac50 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f200c0011e0 tx=0x7f200c00ca30 comp rx=0 tx=0).stop 2024-03-18T20:06:51.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.268+0000 7f2017e77700 1 -- 172.21.15.1:0/587836137 shutdown_connections 2024-03-18T20:06:51.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.268+0000 7f2017e77700 1 --2- 172.21.15.1:0/587836137 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f1ff406ae00 0x7f1ff406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.268+0000 7f2017e77700 1 --2- 172.21.15.1:0/587836137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f201015b750 0x7f20101528e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.268+0000 7f2017e77700 1 --2- 172.21.15.1:0/587836137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2010070db0 0x7f20100cac50 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.268+0000 7f2017e77700 1 --2- 172.21.15.1:0/587836137 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f201013e130 0x7f20100cf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.268+0000 7f2017e77700 1 -- 172.21.15.1:0/587836137 >> 172.21.15.1:0/587836137 conn(0x7f201006a750 msgr2=0x7f201006ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:51.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.268+0000 7f2017e77700 1 -- 172.21.15.1:0/587836137 shutdown_connections 2024-03-18T20:06:51.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.268+0000 7f2017e77700 1 -- 172.21.15.1:0/587836137 wait complete. 2024-03-18T20:06:51.286 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-18T20:06:51.406 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:51.401+0000 7f27cc459240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:06:51.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.410+0000 7fda01644700 1 Processor -- start 2024-03-18T20:06:51.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.410+0000 7fda01644700 1 -- start start 2024-03-18T20:06:51.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.411+0000 7fda01644700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd9fc15d5a0 0x7fd9fc0ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:51.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.411+0000 7fda01644700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd9fc06ae60 0x7fd9fc167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:51.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.411+0000 7fda01644700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9fc0cf600 con 0x7fd9fc0ce550 2024-03-18T20:06:51.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.411+0000 7fda01644700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9fc0ce9f0 con 0x7fd9fc15d5a0 2024-03-18T20:06:51.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.411+0000 7fda01644700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9fc0cecd0 con 0x7fd9fc06ae60 2024-03-18T20:06:51.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.411+0000 7fd9faffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd9fc15d5a0 0x7fd9fc0ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:51.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.411+0000 7fd9faffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd9fc15d5a0 0x7fd9fc0ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56646/0 (socket says 172.21.15.1:56646) 2024-03-18T20:06:51.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.411+0000 7fd9fa7fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd9fc0ce550 0x7fd9fc06a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49208/0 (socket says 172.21.15.1:49208) 2024-03-18T20:06:51.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.411+0000 7fd9faffd700 1 -- 172.21.15.1:0/562305796 learned_addr learned my addr 172.21.15.1:0/562305796 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:51.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.411+0000 7fd9fb7fe700 1 --2- 172.21.15.1:0/562305796 >> v2:172.21.15.165:3300/0 conn(0x7fd9fc06ae60 0x7fd9fc167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:51.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.412+0000 7fd9faffd700 1 -- 172.21.15.1:0/562305796 >> v2:172.21.15.165:3300/0 conn(0x7fd9fc06ae60 msgr2=0x7fd9fc167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:51.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.412+0000 7fd9faffd700 1 --2- 172.21.15.1:0/562305796 >> v2:172.21.15.165:3300/0 conn(0x7fd9fc06ae60 0x7fd9fc167140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.412+0000 7fd9faffd700 1 -- 172.21.15.1:0/562305796 >> v1:172.21.15.1:6789/0 conn(0x7fd9fc0ce550 legacy=0x7fd9fc06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.412+0000 7fd9faffd700 1 -- 172.21.15.1:0/562305796 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9fc0cefb0 con 0x7fd9fc15d5a0 2024-03-18T20:06:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.412+0000 7fd9f9ffb700 1 -- 172.21.15.1:0/562305796 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844716214 0 0) 0x7fd9fc0cf600 con 0x7fd9fc0ce550 2024-03-18T20:06:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.412+0000 7fd9faffd700 1 --2- 172.21.15.1:0/562305796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9fc15d5a0 0x7fd9fc0ce010 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc0cb3d0 tx=0x7fd9ec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92710cefae4ead7c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.412+0000 7fd9f9ffb700 1 -- 172.21.15.1:0/562305796 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec005e30 con 0x7fd9fc15d5a0 2024-03-18T20:06:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.412+0000 7fd9f9ffb700 1 -- 172.21.15.1:0/562305796 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9ec004030 con 0x7fd9fc15d5a0 2024-03-18T20:06:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.412+0000 7fd9f9ffb700 1 -- 172.21.15.1:0/562305796 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec0043e0 con 0x7fd9fc15d5a0 2024-03-18T20:06:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.413+0000 7fda01644700 1 -- 172.21.15.1:0/562305796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9fc15d5a0 msgr2=0x7fd9fc0ce010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.413+0000 7fda01644700 1 --2- 172.21.15.1:0/562305796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9fc15d5a0 0x7fd9fc0ce010 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc0cb3d0 tx=0x7fd9ec009bd0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.413+0000 7fda01644700 1 -- 172.21.15.1:0/562305796 shutdown_connections 2024-03-18T20:06:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.413+0000 7fda01644700 1 --2- 172.21.15.1:0/562305796 >> v2:172.21.15.165:3300/0 conn(0x7fd9fc06ae60 0x7fd9fc167140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.413+0000 7fda01644700 1 --2- 172.21.15.1:0/562305796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9fc15d5a0 0x7fd9fc0ce010 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.413+0000 7fda01644700 1 -- 172.21.15.1:0/562305796 >> 172.21.15.1:0/562305796 conn(0x7fd9fc0cc9b0 msgr2=0x7fd9fc0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:51.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.413+0000 7fda01644700 1 -- 172.21.15.1:0/562305796 shutdown_connections 2024-03-18T20:06:51.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.413+0000 7fda01644700 1 -- 172.21.15.1:0/562305796 wait complete. 2024-03-18T20:06:51.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.415+0000 7fda01644700 1 Processor -- start 2024-03-18T20:06:51.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.415+0000 7fda01644700 1 -- start start 2024-03-18T20:06:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.416+0000 7fda01644700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9fc06ae60 0x7fd9fc0c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.416+0000 7fda01644700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9fc0ce550 0x7fd9fc14c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.416+0000 7fda01644700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9fc15d5a0 0x7fd9fc0c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.416+0000 7fda01644700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9fc0c7a70 con 0x7fd9fc15d5a0 2024-03-18T20:06:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.416+0000 7fda01644700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9fc0c90a0 con 0x7fd9fc06ae60 2024-03-18T20:06:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.416+0000 7fda01644700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9fc149f70 con 0x7fd9fc0ce550 2024-03-18T20:06:51.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.416+0000 7fd9fb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9fc15d5a0 0x7fd9fc0c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:51.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.416+0000 7fd9fb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9fc15d5a0 0x7fd9fc0c7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56650/0 (socket says 172.21.15.1:56650) 2024-03-18T20:06:51.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.416+0000 7fd9fb7fe700 1 -- 172.21.15.1:0/1342354867 learned_addr learned my addr 172.21.15.1:0/1342354867 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:51.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.416+0000 7fd9faffd700 1 --2- 172.21.15.1:0/1342354867 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9fc06ae60 0x7fd9fc0c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.416+0000 7fd9fa7fc700 1 --2- 172.21.15.1:0/1342354867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9fc0ce550 0x7fd9fc14c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.417+0000 7fd9fb7fe700 1 -- 172.21.15.1:0/1342354867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9fc0ce550 msgr2=0x7fd9fc14c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.417+0000 7fd9fb7fe700 1 --2- 172.21.15.1:0/1342354867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9fc0ce550 0x7fd9fc14c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.417+0000 7fd9fb7fe700 1 -- 172.21.15.1:0/1342354867 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9fc06ae60 msgr2=0x7fd9fc0c8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.417+0000 7fd9fb7fe700 1 --2- 172.21.15.1:0/1342354867 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9fc06ae60 0x7fd9fc0c8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.417+0000 7fd9fb7fe700 1 -- 172.21.15.1:0/1342354867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ec009500 con 0x7fd9fc15d5a0 2024-03-18T20:06:51.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.417+0000 7fd9fb7fe700 1 --2- 172.21.15.1:0/1342354867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9fc15d5a0 0x7fd9fc0c7000 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4007de0 tx=0x7fd9f400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:51.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.417+0000 7fd9e37fe700 1 -- 172.21.15.1:0/1342354867 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f400ed30 con 0x7fd9fc15d5a0 2024-03-18T20:06:51.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.417+0000 7fd9e37fe700 1 -- 172.21.15.1:0/1342354867 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9f4014420 con 0x7fd9fc15d5a0 2024-03-18T20:06:51.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.417+0000 7fda01644700 1 -- 172.21.15.1:0/1342354867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9fc14a250 con 0x7fd9fc15d5a0 2024-03-18T20:06:51.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.417+0000 7fd9e37fe700 1 -- 172.21.15.1:0/1342354867 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f40147d0 con 0x7fd9fc15d5a0 2024-03-18T20:06:51.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.417+0000 7fda01644700 1 -- 172.21.15.1:0/1342354867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9fc14aaf0 con 0x7fd9fc15d5a0 2024-03-18T20:06:51.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.418+0000 7fda01644700 1 -- 172.21.15.1:0/1342354867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9fc0676a0 con 0x7fd9fc15d5a0 2024-03-18T20:06:51.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.419+0000 7fd9e37fe700 1 -- 172.21.15.1:0/1342354867 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 96) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fd9f4014930 con 0x7fd9fc15d5a0 2024-03-18T20:06:51.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.420+0000 7fd9e37fe700 1 --2- 172.21.15.1:0/1342354867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd9e806ae00 0x7fd9e806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:51.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.420+0000 7fd9faffd700 1 -- 172.21.15.1:0/1342354867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd9e806ae00 msgr2=0x7fd9e806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:51.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.420+0000 7fd9e37fe700 1 -- 172.21.15.1:0/1342354867 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd9f408abf0 con 0x7fd9fc15d5a0 2024-03-18T20:06:51.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.420+0000 7fd9faffd700 1 --2- 172.21.15.1:0/1342354867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd9e806ae00 0x7fd9e806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:06:51.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.423+0000 7fd9e37fe700 1 -- 172.21.15.1:0/1342354867 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd9f4055ba0 con 0x7fd9fc15d5a0 2024-03-18T20:06:51.456 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:51.450+0000 7f2a2119a240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:06:51.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.621+0000 7fd9faffd700 1 -- 172.21.15.1:0/1342354867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd9e806ae00 msgr2=0x7fd9e806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:51.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.621+0000 7fd9faffd700 1 --2- 172.21.15.1:0/1342354867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd9e806ae00 0x7fd9e806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:06:51.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.627+0000 7fda01644700 1 -- 172.21.15.1:0/1342354867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fd9fc0b12f0 con 0x7fd9fc15d5a0 2024-03-18T20:06:51.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.631+0000 7fd9e37fe700 1 -- 172.21.15.1:0/1342354867 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v60)=0 v60) v1 ==== 133+0+0 (secure 0 0 0) 0x7fd9f40591c0 con 0x7fd9fc15d5a0 2024-03-18T20:06:51.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.633+0000 7fda01644700 1 -- 172.21.15.1:0/1342354867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd9e806ae00 msgr2=0x7fd9e806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:06:51.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.633+0000 7fda01644700 1 --2- 172.21.15.1:0/1342354867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd9e806ae00 0x7fd9e806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.633+0000 7fda01644700 1 -- 172.21.15.1:0/1342354867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9fc15d5a0 msgr2=0x7fd9fc0c7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:51.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.633+0000 7fda01644700 1 --2- 172.21.15.1:0/1342354867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9fc15d5a0 0x7fd9fc0c7000 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4007de0 tx=0x7fd9f400e3c0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.633+0000 7fda01644700 1 -- 172.21.15.1:0/1342354867 shutdown_connections 2024-03-18T20:06:51.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.633+0000 7fda01644700 1 --2- 172.21.15.1:0/1342354867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd9e806ae00 0x7fd9e806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.633+0000 7fda01644700 1 --2- 172.21.15.1:0/1342354867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9fc15d5a0 0x7fd9fc0c7000 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.633+0000 7fda01644700 1 --2- 172.21.15.1:0/1342354867 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9fc06ae60 0x7fd9fc0c8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.633+0000 7fda01644700 1 --2- 172.21.15.1:0/1342354867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9fc0ce550 0x7fd9fc14c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.633+0000 7fda01644700 1 -- 172.21.15.1:0/1342354867 >> 172.21.15.1:0/1342354867 conn(0x7fd9fc0cc9b0 msgr2=0x7fd9fc06e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:51.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.633+0000 7fda01644700 1 -- 172.21.15.1:0/1342354867 shutdown_connections 2024-03-18T20:06:51.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.633+0000 7fda01644700 1 -- 172.21.15.1:0/1342354867 wait complete. 2024-03-18T20:06:51.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:51.653 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-03-18T20:06:51.676 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1h 2024-03-18T20:06:51.837 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:51.832+0000 7f27cc459240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:06:51.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.841+0000 7f40d02f0700 1 Processor -- start 2024-03-18T20:06:51.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.841+0000 7f40d02f0700 1 -- start start 2024-03-18T20:06:51.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.842+0000 7f40d02f0700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f40c8160e50 0x7f40c80cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:51.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.842+0000 7f40d02f0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f40c81612d0 0x7f40c81616f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:51.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.842+0000 7f40d02f0700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c8162170 con 0x7f40c8160a70 2024-03-18T20:06:51.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.842+0000 7f40d02f0700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c8161c30 con 0x7f40c81612d0 2024-03-18T20:06:51.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.842+0000 7f40d02f0700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c80cef30 con 0x7f40c8160e50 2024-03-18T20:06:51.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.842+0000 7f40ce88d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f40c81612d0 0x7f40c81616f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:51.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.842+0000 7f40cd88b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f40c8160e50 0x7f40c80cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:51.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.842+0000 7f40ce08c700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f40c8160a70 0x7f40c80d3b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:53696/0 (socket says 172.21.15.1:53696) 2024-03-18T20:06:51.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.842+0000 7f40ce88d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f40c81612d0 0x7f40c81616f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56660/0 (socket says 172.21.15.1:56660) 2024-03-18T20:06:51.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.842+0000 7f40ce08c700 1 -- 172.21.15.1:0/566980232 learned_addr learned my addr 172.21.15.1:0/566980232 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:51.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.843+0000 7f40ce88d700 1 -- 172.21.15.1:0/566980232 >> v2:172.21.15.165:3300/0 conn(0x7f40c8160e50 msgr2=0x7f40c80cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:51.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.843+0000 7f40ce88d700 1 --2- 172.21.15.1:0/566980232 >> v2:172.21.15.165:3300/0 conn(0x7f40c8160e50 0x7f40c80cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.843+0000 7f40ce88d700 1 -- 172.21.15.1:0/566980232 >> v1:172.21.15.165:6789/0 conn(0x7f40c8160a70 legacy=0x7f40c80d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:51.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.843+0000 7f40ce88d700 1 -- 172.21.15.1:0/566980232 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c80cf210 con 0x7f40c81612d0 2024-03-18T20:06:51.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.843+0000 7f40ce88d700 1 --2- 172.21.15.1:0/566980232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c81612d0 0x7f40c81616f0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f40c80c9d70 tx=0x7f40c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4379644b9f6aec22 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:51.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.845+0000 7f40cd08a700 1 -- 172.21.15.1:0/566980232 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c000e070 con 0x7f40c81612d0 2024-03-18T20:06:51.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.845+0000 7f40cd08a700 1 -- 172.21.15.1:0/566980232 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c000bb20 con 0x7f40c81612d0 2024-03-18T20:06:51.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.845+0000 7f40cd08a700 1 -- 172.21.15.1:0/566980232 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c0012430 con 0x7f40c81612d0 2024-03-18T20:06:51.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.845+0000 7f40d02f0700 1 -- 172.21.15.1:0/566980232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c81612d0 msgr2=0x7f40c81616f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:51.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.845+0000 7f40d02f0700 1 --2- 172.21.15.1:0/566980232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c81612d0 0x7f40c81616f0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f40c80c9d70 tx=0x7f40c000b330 comp rx=0 tx=0).stop 2024-03-18T20:06:51.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.846+0000 7f40d02f0700 1 -- 172.21.15.1:0/566980232 shutdown_connections 2024-03-18T20:06:51.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.846+0000 7f40d02f0700 1 --2- 172.21.15.1:0/566980232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c81612d0 0x7f40c81616f0 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.846+0000 7f40d02f0700 1 --2- 172.21.15.1:0/566980232 >> v2:172.21.15.165:3300/0 conn(0x7f40c8160e50 0x7f40c80cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.846+0000 7f40d02f0700 1 -- 172.21.15.1:0/566980232 >> 172.21.15.1:0/566980232 conn(0x7f40c809c560 msgr2=0x7f40c80d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:51.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.846+0000 7f40d02f0700 1 -- 172.21.15.1:0/566980232 shutdown_connections 2024-03-18T20:06:51.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.846+0000 7f40d02f0700 1 -- 172.21.15.1:0/566980232 wait complete. 2024-03-18T20:06:51.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.847+0000 7f40d02f0700 1 Processor -- start 2024-03-18T20:06:51.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.847+0000 7f40d02f0700 1 -- start start 2024-03-18T20:06:51.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.848+0000 7f40d02f0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40c8160a70 0x7f40c80c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:51.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.848+0000 7f40d02f0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c8160e50 0x7f40c80b8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:51.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.848+0000 7f40d02f0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40c81612d0 0x7f40c80cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.848+0000 7f40d02f0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c80c6c40 con 0x7f40c8160e50 2024-03-18T20:06:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.848+0000 7f40d02f0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c80c7d00 con 0x7f40c8160a70 2024-03-18T20:06:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.848+0000 7f40d02f0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c80c7f30 con 0x7f40c81612d0 2024-03-18T20:06:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.848+0000 7f40cd88b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c8160e50 0x7f40c80b8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.848+0000 7f40cd88b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c8160e50 0x7f40c80b8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56666/0 (socket says 172.21.15.1:56666) 2024-03-18T20:06:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.848+0000 7f40cd88b700 1 -- 172.21.15.1:0/692609581 learned_addr learned my addr 172.21.15.1:0/692609581 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.849+0000 7f40ce08c700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40c8160a70 0x7f40c80c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.849+0000 7f40ce88d700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40c81612d0 0x7f40c80cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.849+0000 7f40cd88b700 1 -- 172.21.15.1:0/692609581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40c81612d0 msgr2=0x7f40c80cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.849+0000 7f40cd88b700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40c81612d0 0x7f40c80cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.849+0000 7f40cd88b700 1 -- 172.21.15.1:0/692609581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40c8160a70 msgr2=0x7f40c80c77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.849+0000 7f40cd88b700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40c8160a70 0x7f40c80c77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:51.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.849+0000 7f40cd88b700 1 -- 172.21.15.1:0/692609581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c0009a70 con 0x7f40c8160e50 2024-03-18T20:06:51.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.849+0000 7f40cd88b700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c8160e50 0x7f40c80b8630 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f40b800e870 tx=0x7f40b800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:51.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.849+0000 7f40beffd700 1 -- 172.21.15.1:0/692609581 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b8013680 con 0x7f40c8160e50 2024-03-18T20:06:51.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.849+0000 7f40beffd700 1 -- 172.21.15.1:0/692609581 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b80137e0 con 0x7f40c8160e50 2024-03-18T20:06:51.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.849+0000 7f40d02f0700 1 -- 172.21.15.1:0/692609581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c80c8210 con 0x7f40c8160e50 2024-03-18T20:06:51.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.849+0000 7f40beffd700 1 -- 172.21.15.1:0/692609581 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b8013b70 con 0x7f40c8160e50 2024-03-18T20:06:51.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.849+0000 7f40d02f0700 1 -- 172.21.15.1:0/692609581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c80c87d0 con 0x7f40c8160e50 2024-03-18T20:06:51.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.850+0000 7f40d02f0700 1 -- 172.21.15.1:0/692609581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c80c8e20 con 0x7f40c8160e50 2024-03-18T20:06:51.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.851+0000 7f40beffd700 1 -- 172.21.15.1:0/692609581 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 96) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f40b8013cd0 con 0x7f40c8160e50 2024-03-18T20:06:51.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.851+0000 7f40beffd700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f40ac06ae00 0x7f40ac06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:51.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.851+0000 7f40ce08c700 1 -- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f40ac06ae00 msgr2=0x7f40ac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:51.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.851+0000 7f40beffd700 1 -- 172.21.15.1:0/692609581 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f40b80894f0 con 0x7f40c8160e50 2024-03-18T20:06:51.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.851+0000 7f40ce08c700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f40ac06ae00 0x7f40ac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:06:51.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:51.856+0000 7f40beffd700 1 -- 172.21.15.1:0/692609581 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f40b80543a0 con 0x7f40c8160e50 2024-03-18T20:06:51.927 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:51.922+0000 7f2a2119a240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:06:52.052 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:52.047+0000 7f27cc459240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:06:52.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:52.053+0000 7f40ce08c700 1 -- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f40ac06ae00 msgr2=0x7f40ac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:52.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:52.053+0000 7f40ce08c700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f40ac06ae00 0x7f40ac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:06:52.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:52.098+0000 7f40d02f0700 1 -- 172.21.15.1:0/692609581 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f40c80cfab0 con 0x7f40ac06ae00 2024-03-18T20:06:52.147 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:52.141+0000 7f2a2119a240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:06:52.426 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:52.421+0000 7f27cc459240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:06:52.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:52.454+0000 7f40ce08c700 1 -- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f40ac06ae00 msgr2=0x7f40ac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:52.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:52.454+0000 7f40ce08c700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f40ac06ae00 0x7f40ac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T20:06:52.535 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:52.529+0000 7f2a2119a240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:06:53.017 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:53.012+0000 7f27cc459240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:06:53.090 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:53.085+0000 7f27cc459240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:06:53.155 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:53.149+0000 7f2a2119a240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:06:53.217 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:53.212+0000 7f27cc459240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:06:53.231 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:53.225+0000 7f2a2119a240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:06:53.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:53.255+0000 7f40ce08c700 1 -- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f40ac06ae00 msgr2=0x7f40ac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:53.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:53.255+0000 7f40ce08c700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f40ac06ae00 0x7f40ac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T20:06:53.365 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:53.359+0000 7f2a2119a240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:06:53.402 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:53.394+0000 7f27cc459240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:06:53.471 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:53.466+0000 7f27cc459240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:06:53.550 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:53.545+0000 7f27cc459240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:06:53.551 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:53.545+0000 7f2a2119a240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:06:53.627 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:53.622+0000 7f2a2119a240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:06:53.628 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:53.623+0000 7f27cc459240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:06:53.706 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:53.701+0000 7f27cc459240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:06:53.709 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:53.704+0000 7f2a2119a240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:06:53.791 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:53.786+0000 7f2a2119a240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:06:53.874 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:53.868+0000 7f2a2119a240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:06:53.877 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:53.872+0000 7f27cc459240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:06:54.054 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:54.048+0000 7f2a2119a240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:06:54.091 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:54.086+0000 7f27cc459240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:06:54.244 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:54.239+0000 7f27cc459240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:06:54.277 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:54.271+0000 7f2a2119a240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:06:54.317 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:54.312+0000 7f27cc459240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:06:54.437 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:54.431+0000 7f2a2119a240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:06:54.515 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:54.509+0000 7f2a2119a240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:06:54.763 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:54.758+0000 7f27cc459240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:06:54.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:54.856+0000 7f40ce08c700 1 -- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f40ac06ae00 msgr2=0x7f40ac06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:54.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:54.856+0000 7f40ce08c700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f40ac06ae00 0x7f40ac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T20:06:54.988 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:54.982+0000 7f2a2119a240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:06:55.102 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:55.097+0000 7f27cc459240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:06:55.267 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:55.262+0000 7f27cc459240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:06:55.341 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:55.335+0000 7f2a2119a240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:06:55.514 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:55.508+0000 7f2a2119a240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:06:55.658 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:55.653+0000 7f27cc459240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:06:55.824 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:55.820+0000 7f27cc459240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:06:55.928 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:55.922+0000 7f2a2119a240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:06:56.106 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:56.100+0000 7f2a2119a240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:06:56.190 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:56.185+0000 7f27cc459240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:06:56.278 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:56.273+0000 7f27cc459240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:06:56.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:56.342+0000 7f40beffd700 1 -- 172.21.15.1:0/692609581 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 97) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f40b8087880 con 0x7f40c8160e50 2024-03-18T20:06:56.488 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:56.482+0000 7f2a2119a240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:06:56.582 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:56.577+0000 7f2a2119a240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:06:56.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:56.595+0000 7f40beffd700 1 -- 172.21.15.1:0/692609581 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 98) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f40b8054770 con 0x7f40c8160e50 2024-03-18T20:06:56.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:56.596+0000 7f40beffd700 1 -- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f40ac06ae00 msgr2=0x7f40ac06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:06:56.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:56.596+0000 7f40beffd700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f40ac06ae00 0x7f40ac06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:56.743 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:56.738+0000 7f2996dbd700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:56.744 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:56.738+0000 7f2996dbd700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:56.744 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:56.738+0000 7f2996dbd700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:56.744 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:56.738+0000 7f2996dbd700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:56.744 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:56.738+0000 7f2996dbd700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:56.744 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:56.738+0000 7f2993db7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:56.744 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:56.738+0000 7f2993db7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:56.744 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:56.738+0000 7f2993db7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:56.744 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:56.738+0000 7f2993db7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:56.744 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:56.738+0000 7f2993db7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:06:57.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.600+0000 7f40beffd700 1 -- 172.21.15.1:0/692609581 <== mon.0 v2:172.21.15.1:3300/0 9 ==== mgrmap(e 99) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f40b8055040 con 0x7f40c8160e50 2024-03-18T20:06:57.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.601+0000 7f40beffd700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f40ac06ae00 0x7f40ac06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:57.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.601+0000 7f40beffd700 1 -- 172.21.15.1:0/692609581 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f40ac027a50 con 0x7f40ac06ae00 2024-03-18T20:06:57.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.603+0000 7f40ce08c700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f40ac06ae00 0x7f40ac06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:57.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.604+0000 7f40ce08c700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f40ac06ae00 0x7f40ac06d2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f40c4005910 tx=0x7f40c4007f50 comp rx=0 tx=0).ready entity=mgr.10104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:57.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.663+0000 7f40beffd700 1 -- 172.21.15.1:0/692609581 <== mgr.10104 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f40ac027a50 con 0x7f40ac06ae00 2024-03-18T20:06:57.669 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path snap_test_dir1 2024-03-18T20:06:57.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.666+0000 7f40d02f0700 1 -- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f40ac06ae00 msgr2=0x7f40ac06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:57.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.666+0000 7f40d02f0700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f40ac06ae00 0x7f40ac06d2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f40c4005910 tx=0x7f40c4007f50 comp rx=0 tx=0).stop 2024-03-18T20:06:57.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.666+0000 7f40d02f0700 1 -- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c8160e50 msgr2=0x7f40c80b8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:57.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.666+0000 7f40d02f0700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c8160e50 0x7f40c80b8630 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f40b800e870 tx=0x7f40b800eb80 comp rx=0 tx=0).stop 2024-03-18T20:06:57.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.666+0000 7f40d02f0700 1 -- 172.21.15.1:0/692609581 shutdown_connections 2024-03-18T20:06:57.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.666+0000 7f40d02f0700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f40ac06ae00 0x7f40ac06d2c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:57.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.666+0000 7f40d02f0700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40c81612d0 0x7f40c80cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:57.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.666+0000 7f40d02f0700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40c8160a70 0x7f40c80c77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:57.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.666+0000 7f40d02f0700 1 --2- 172.21.15.1:0/692609581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c8160e50 0x7f40c80b8630 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:57.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.666+0000 7f40d02f0700 1 -- 172.21.15.1:0/692609581 >> 172.21.15.1:0/692609581 conn(0x7f40c809c560 msgr2=0x7f40c80d31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:57.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.666+0000 7f40d02f0700 1 -- 172.21.15.1:0/692609581 shutdown_connections 2024-03-18T20:06:57.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.666+0000 7f40d02f0700 1 -- 172.21.15.1:0/692609581 wait complete. 2024-03-18T20:06:57.685 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-03-18T20:06:57.686 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-03-18T20:06:57.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.810+0000 7faf41d31700 1 Processor -- start 2024-03-18T20:06:57.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.810+0000 7faf41d31700 1 -- start start 2024-03-18T20:06:57.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.811+0000 7faf41d31700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7faf3c0b2650 0x7faf3c070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:57.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.811+0000 7faf41d31700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c0ac8b0 0x7faf3c071380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:57.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.811+0000 7faf41d31700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf3c0b2b80 con 0x7faf3c15d3b0 2024-03-18T20:06:57.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.811+0000 7faf41d31700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf3c0b3800 con 0x7faf3c0b2650 2024-03-18T20:06:57.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.811+0000 7faf41d31700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c0b2e60 con 0x7faf3c0ac8b0 2024-03-18T20:06:57.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.811+0000 7faf3affd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7faf3c0b2650 0x7faf3c070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:57.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.811+0000 7faf3bfff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c0ac8b0 0x7faf3c071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:57.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.811+0000 7faf3b7fe700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7faf3c15d3b0 0x7faf3c06cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:59948/0 (socket says 172.21.15.1:59948) 2024-03-18T20:06:57.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.811+0000 7faf3affd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7faf3c0b2650 0x7faf3c070e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:42646/0 (socket says 172.21.15.1:42646) 2024-03-18T20:06:57.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.811+0000 7faf3b7fe700 1 -- 172.21.15.1:0/2428518569 learned_addr learned my addr 172.21.15.1:0/2428518569 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:57.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.811+0000 7faf3bfff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c0ac8b0 0x7faf3c071380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:48170/0 (socket says 172.21.15.1:48170) 2024-03-18T20:06:57.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.811+0000 7faf3bfff700 1 -- 172.21.15.1:0/2428518569 >> v2:172.21.15.165:3300/0 conn(0x7faf3c0b2650 msgr2=0x7faf3c070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:57.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.811+0000 7faf3bfff700 1 --2- 172.21.15.1:0/2428518569 >> v2:172.21.15.165:3300/0 conn(0x7faf3c0b2650 0x7faf3c070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:57.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.811+0000 7faf3bfff700 1 -- 172.21.15.1:0/2428518569 >> v1:172.21.15.165:6789/0 conn(0x7faf3c15d3b0 legacy=0x7faf3c06cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:57.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.811+0000 7faf3bfff700 1 -- 172.21.15.1:0/2428518569 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3c0b3140 con 0x7faf3c0ac8b0 2024-03-18T20:06:57.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.811+0000 7faf3a7fc700 1 -- 172.21.15.1:0/2428518569 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158696379 0 0) 0x7faf3c0b2b80 con 0x7faf3c15d3b0 2024-03-18T20:06:57.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.811+0000 7faf3affd700 1 --2- 172.21.15.1:0/2428518569 >> v2:172.21.15.165:3300/0 conn(0x7faf3c0b2650 0x7faf3c070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:57.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.812+0000 7faf3bfff700 1 --2- 172.21.15.1:0/2428518569 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c0ac8b0 0x7faf3c071380 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7faf34009d90 tx=0x7faf3400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8361e08b90349d22 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:57.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.812+0000 7faf3a7fc700 1 -- 172.21.15.1:0/2428518569 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3400e070 con 0x7faf3c0ac8b0 2024-03-18T20:06:57.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.812+0000 7faf3a7fc700 1 -- 172.21.15.1:0/2428518569 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf3400bb20 con 0x7faf3c0ac8b0 2024-03-18T20:06:57.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.812+0000 7faf3a7fc700 1 -- 172.21.15.1:0/2428518569 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf34012430 con 0x7faf3c0ac8b0 2024-03-18T20:06:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.812+0000 7faf41d31700 1 -- 172.21.15.1:0/2428518569 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c0ac8b0 msgr2=0x7faf3c071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.812+0000 7faf41d31700 1 --2- 172.21.15.1:0/2428518569 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c0ac8b0 0x7faf3c071380 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7faf34009d90 tx=0x7faf3400b330 comp rx=0 tx=0).stop 2024-03-18T20:06:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.812+0000 7faf41d31700 1 -- 172.21.15.1:0/2428518569 shutdown_connections 2024-03-18T20:06:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.812+0000 7faf41d31700 1 --2- 172.21.15.1:0/2428518569 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c0ac8b0 0x7faf3c071380 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.812+0000 7faf41d31700 1 --2- 172.21.15.1:0/2428518569 >> v2:172.21.15.165:3300/0 conn(0x7faf3c0b2650 0x7faf3c070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.812+0000 7faf41d31700 1 -- 172.21.15.1:0/2428518569 >> 172.21.15.1:0/2428518569 conn(0x7faf3c0b1350 msgr2=0x7faf3c0b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.812+0000 7faf41d31700 1 -- 172.21.15.1:0/2428518569 shutdown_connections 2024-03-18T20:06:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.812+0000 7faf41d31700 1 -- 172.21.15.1:0/2428518569 wait complete. 2024-03-18T20:06:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.813+0000 7faf41d31700 1 Processor -- start 2024-03-18T20:06:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.814+0000 7faf41d31700 1 -- start start 2024-03-18T20:06:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.814+0000 7faf41d31700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf3c0ac8b0 0x7faf3c1f3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:57.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.814+0000 7faf41d31700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf3c0b2650 0x7faf3c1564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:57.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.814+0000 7faf41d31700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c15d3b0 0x7faf3c158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:57.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.814+0000 7faf41d31700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c157880 con 0x7faf3c0b2650 2024-03-18T20:06:57.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.814+0000 7faf41d31700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c1569f0 con 0x7faf3c0ac8b0 2024-03-18T20:06:57.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.814+0000 7faf41d31700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c156c70 con 0x7faf3c15d3b0 2024-03-18T20:06:57.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.815+0000 7faf3affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf3c0b2650 0x7faf3c1564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:57.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.815+0000 7faf3affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf3c0b2650 0x7faf3c1564b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34242/0 (socket says 172.21.15.1:34242) 2024-03-18T20:06:57.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.815+0000 7faf3affd700 1 -- 172.21.15.1:0/3165836488 learned_addr learned my addr 172.21.15.1:0/3165836488 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:57.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.815+0000 7faf3b7fe700 1 --2- 172.21.15.1:0/3165836488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf3c0ac8b0 0x7faf3c1f3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:57.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.815+0000 7faf3bfff700 1 --2- 172.21.15.1:0/3165836488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c15d3b0 0x7faf3c158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:57.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.815+0000 7faf3affd700 1 -- 172.21.15.1:0/3165836488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c15d3b0 msgr2=0x7faf3c158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:57.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.815+0000 7faf3affd700 1 --2- 172.21.15.1:0/3165836488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c15d3b0 0x7faf3c158a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:57.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.815+0000 7faf3affd700 1 -- 172.21.15.1:0/3165836488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf3c0ac8b0 msgr2=0x7faf3c1f3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:57.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.815+0000 7faf3affd700 1 --2- 172.21.15.1:0/3165836488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf3c0ac8b0 0x7faf3c1f3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.815+0000 7faf3affd700 1 -- 172.21.15.1:0/3165836488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf300094a0 con 0x7faf3c0b2650 2024-03-18T20:06:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.815+0000 7faf3affd700 1 --2- 172.21.15.1:0/3165836488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf3c0b2650 0x7faf3c1564b0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7faf3000ebe0 tx=0x7faf3000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.815+0000 7faf23fff700 1 -- 172.21.15.1:0/3165836488 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3000c850 con 0x7faf3c0b2650 2024-03-18T20:06:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.815+0000 7faf23fff700 1 -- 172.21.15.1:0/3165836488 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf3000c9b0 con 0x7faf3c0b2650 2024-03-18T20:06:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.815+0000 7faf23fff700 1 -- 172.21.15.1:0/3165836488 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3000cd40 con 0x7faf3c0b2650 2024-03-18T20:06:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.815+0000 7faf41d31700 1 -- 172.21.15.1:0/3165836488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf34009a70 con 0x7faf3c0b2650 2024-03-18T20:06:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.815+0000 7faf41d31700 1 -- 172.21.15.1:0/3165836488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf3c157230 con 0x7faf3c0b2650 2024-03-18T20:06:57.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.816+0000 7faf41d31700 1 -- 172.21.15.1:0/3165836488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf3c1579f0 con 0x7faf3c0b2650 2024-03-18T20:06:57.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.817+0000 7faf23fff700 1 -- 172.21.15.1:0/3165836488 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 99) v1 ==== 88992+0+0 (secure 0 0 0) 0x7faf30013470 con 0x7faf3c0b2650 2024-03-18T20:06:57.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.817+0000 7faf23fff700 1 --2- 172.21.15.1:0/3165836488 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf2806ab30 0x7faf2806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:57.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.817+0000 7faf23fff700 1 -- 172.21.15.1:0/3165836488 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6445+0+0 (secure 0 0 0) 0x7faf30089350 con 0x7faf3c0b2650 2024-03-18T20:06:57.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.817+0000 7faf3b7fe700 1 --2- 172.21.15.1:0/3165836488 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf2806ab30 0x7faf2806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:57.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.818+0000 7faf3b7fe700 1 --2- 172.21.15.1:0/3165836488 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf2806ab30 0x7faf2806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faf2c000f60 tx=0x7faf2c007e70 comp rx=0 tx=0).ready entity=mgr.10104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:57.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:57.822+0000 7faf23fff700 1 -- 172.21.15.1:0/3165836488 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faf30054590 con 0x7faf3c0b2650 2024-03-18T20:06:58.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.048+0000 7faf41d31700 1 -- 172.21.15.1:0/3165836488 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7faf3c06d600 con 0x7faf2806ab30 2024-03-18T20:06:58.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.105+0000 7faf23fff700 1 -- 172.21.15.1:0/3165836488 <== mgr.10104 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7faf2c004ad0 con 0x7faf2806ab30 2024-03-18T20:06:58.111 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule removed for path snap_test_dir1 2024-03-18T20:06:58.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.108+0000 7faf41d31700 1 -- 172.21.15.1:0/3165836488 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf2806ab30 msgr2=0x7faf2806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:58.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.108+0000 7faf41d31700 1 --2- 172.21.15.1:0/3165836488 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf2806ab30 0x7faf2806cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faf2c000f60 tx=0x7faf2c007e70 comp rx=0 tx=0).stop 2024-03-18T20:06:58.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.108+0000 7faf41d31700 1 -- 172.21.15.1:0/3165836488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf3c0b2650 msgr2=0x7faf3c1564b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:58.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.108+0000 7faf41d31700 1 --2- 172.21.15.1:0/3165836488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf3c0b2650 0x7faf3c1564b0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7faf3000ebe0 tx=0x7faf3000eef0 comp rx=0 tx=0).stop 2024-03-18T20:06:58.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.108+0000 7faf41d31700 1 -- 172.21.15.1:0/3165836488 shutdown_connections 2024-03-18T20:06:58.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.108+0000 7faf41d31700 1 --2- 172.21.15.1:0/3165836488 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7faf2806ab30 0x7faf2806cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:58.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.108+0000 7faf41d31700 1 --2- 172.21.15.1:0/3165836488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf3c15d3b0 0x7faf3c158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:58.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.108+0000 7faf41d31700 1 --2- 172.21.15.1:0/3165836488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf3c0ac8b0 0x7faf3c1f3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:58.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.108+0000 7faf41d31700 1 --2- 172.21.15.1:0/3165836488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf3c0b2650 0x7faf3c1564b0 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:58.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.108+0000 7faf41d31700 1 -- 172.21.15.1:0/3165836488 >> 172.21.15.1:0/3165836488 conn(0x7faf3c0b1350 msgr2=0x7faf3c0b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:58.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.108+0000 7faf41d31700 1 -- 172.21.15.1:0/3165836488 shutdown_connections 2024-03-18T20:06:58.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.108+0000 7faf41d31700 1 -- 172.21.15.1:0/3165836488 wait complete. 2024-03-18T20:06:58.126 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-03-18T20:06:58.127 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-03-18T20:06:58.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.247+0000 7fc714dee700 1 Processor -- start 2024-03-18T20:06:58.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.247+0000 7fc714dee700 1 -- start start 2024-03-18T20:06:58.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.248+0000 7fc714dee700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc71006eda0 0x7fc710161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:58.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.248+0000 7fc714dee700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc71006f180 0x7fc7100ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:58.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.248+0000 7fc714dee700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7100caac0 con 0x7fc71006a750 2024-03-18T20:06:58.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.248+0000 7fc714dee700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7100cb320 con 0x7fc71006f180 2024-03-18T20:06:58.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.248+0000 7fc714dee700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7100ce010 con 0x7fc71006eda0 2024-03-18T20:06:58.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.249+0000 7fc70dd9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc71006f180 0x7fc7100ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:58.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.249+0000 7fc70dd9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc71006f180 0x7fc7100ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34244/0 (socket says 172.21.15.1:34244) 2024-03-18T20:06:58.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.249+0000 7fc70dd9b700 1 -- 172.21.15.1:0/3344956792 learned_addr learned my addr 172.21.15.1:0/3344956792 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:58.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.249+0000 7fc70e59c700 1 --2- 172.21.15.1:0/3344956792 >> v2:172.21.15.165:3300/0 conn(0x7fc71006eda0 0x7fc710161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.249+0000 7fc70dd9b700 1 -- 172.21.15.1:0/3344956792 >> v2:172.21.15.165:3300/0 conn(0x7fc71006eda0 msgr2=0x7fc710161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.249+0000 7fc70dd9b700 1 --2- 172.21.15.1:0/3344956792 >> v2:172.21.15.165:3300/0 conn(0x7fc71006eda0 0x7fc710161c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.249+0000 7fc70dd9b700 1 -- 172.21.15.1:0/3344956792 >> v1:172.21.15.165:6789/0 conn(0x7fc71006a750 legacy=0x7fc7100715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.249+0000 7fc70dd9b700 1 -- 172.21.15.1:0/3344956792 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7100cad60 con 0x7fc71006f180 2024-03-18T20:06:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.249+0000 7fc70d59a700 1 -- 172.21.15.1:0/3344956792 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646705156 0 0) 0x7fc7100caac0 con 0x7fc71006a750 2024-03-18T20:06:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.249+0000 7fc70dd9b700 1 --2- 172.21.15.1:0/3344956792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc71006f180 0x7fc7100ca580 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fc71006d3d0 tx=0x7fc704009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d429ef0cc990d65 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.249+0000 7fc70d59a700 1 -- 172.21.15.1:0/3344956792 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc704011830 con 0x7fc71006f180 2024-03-18T20:06:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.249+0000 7fc70d59a700 1 -- 172.21.15.1:0/3344956792 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc704011990 con 0x7fc71006f180 2024-03-18T20:06:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.249+0000 7fc70d59a700 1 -- 172.21.15.1:0/3344956792 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc704011d20 con 0x7fc71006f180 2024-03-18T20:06:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.250+0000 7fc714dee700 1 -- 172.21.15.1:0/3344956792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc71006f180 msgr2=0x7fc7100ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.250+0000 7fc714dee700 1 --2- 172.21.15.1:0/3344956792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc71006f180 0x7fc7100ca580 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fc71006d3d0 tx=0x7fc704009bd0 comp rx=0 tx=0).stop 2024-03-18T20:06:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.250+0000 7fc714dee700 1 -- 172.21.15.1:0/3344956792 shutdown_connections 2024-03-18T20:06:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.250+0000 7fc714dee700 1 --2- 172.21.15.1:0/3344956792 >> v2:172.21.15.165:3300/0 conn(0x7fc71006eda0 0x7fc710161c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:58.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.250+0000 7fc714dee700 1 --2- 172.21.15.1:0/3344956792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc71006f180 0x7fc7100ca580 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:58.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.250+0000 7fc714dee700 1 -- 172.21.15.1:0/3344956792 >> 172.21.15.1:0/3344956792 conn(0x7fc7100cff30 msgr2=0x7fc7100d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:58.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.250+0000 7fc714dee700 1 -- 172.21.15.1:0/3344956792 shutdown_connections 2024-03-18T20:06:58.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.250+0000 7fc714dee700 1 -- 172.21.15.1:0/3344956792 wait complete. 2024-03-18T20:06:58.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.251+0000 7fc714dee700 1 Processor -- start 2024-03-18T20:06:58.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.251+0000 7fc714dee700 1 -- start start 2024-03-18T20:06:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.252+0000 7fc714dee700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc71006a750 0x7fc71014f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.252+0000 7fc714dee700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc71006eda0 0x7fc7100714b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.252+0000 7fc714dee700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc71006f180 0x7fc7100719f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.252+0000 7fc714dee700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7101f3050 con 0x7fc71006f180 2024-03-18T20:06:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.252+0000 7fc714dee700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71014fb70 con 0x7fc71006eda0 2024-03-18T20:06:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.252+0000 7fc714dee700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7100c4b70 con 0x7fc71006a750 2024-03-18T20:06:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.252+0000 7fc70ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc71006f180 0x7fc7100719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.252+0000 7fc70ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc71006f180 0x7fc7100719f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34246/0 (socket says 172.21.15.1:34246) 2024-03-18T20:06:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.252+0000 7fc70ed9d700 1 -- 172.21.15.1:0/3170700187 learned_addr learned my addr 172.21.15.1:0/3170700187 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.252+0000 7fc70dd9b700 1 --2- 172.21.15.1:0/3170700187 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc71006eda0 0x7fc7100714b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.252+0000 7fc70e59c700 1 --2- 172.21.15.1:0/3170700187 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc71006a750 0x7fc71014f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.253+0000 7fc70dd9b700 1 -- 172.21.15.1:0/3170700187 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc71006a750 msgr2=0x7fc71014f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.253+0000 7fc70dd9b700 1 --2- 172.21.15.1:0/3170700187 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc71006a750 0x7fc71014f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:58.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.253+0000 7fc70dd9b700 1 -- 172.21.15.1:0/3170700187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc71006f180 msgr2=0x7fc7100719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:58.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.253+0000 7fc70dd9b700 1 --2- 172.21.15.1:0/3170700187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc71006f180 0x7fc7100719f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:58.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.253+0000 7fc70dd9b700 1 -- 172.21.15.1:0/3170700187 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc704009550 con 0x7fc71006eda0 2024-03-18T20:06:58.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.253+0000 7fc70ed9d700 1 --2- 172.21.15.1:0/3170700187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc71006f180 0x7fc7100719f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:06:58.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.253+0000 7fc70dd9b700 1 --2- 172.21.15.1:0/3170700187 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc71006eda0 0x7fc7100714b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fc704016000 tx=0x7fc704004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:58.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.253+0000 7fc6feffd700 1 -- 172.21.15.1:0/3170700187 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc704004510 con 0x7fc71006eda0 2024-03-18T20:06:58.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.253+0000 7fc6feffd700 1 -- 172.21.15.1:0/3170700187 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70401e070 con 0x7fc71006eda0 2024-03-18T20:06:58.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.253+0000 7fc714dee700 1 -- 172.21.15.1:0/3170700187 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71015d430 con 0x7fc71006eda0 2024-03-18T20:06:58.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.253+0000 7fc6feffd700 1 -- 172.21.15.1:0/3170700187 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7040038e0 con 0x7fc71006eda0 2024-03-18T20:06:58.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.253+0000 7fc714dee700 1 -- 172.21.15.1:0/3170700187 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71015d6d0 con 0x7fc71006eda0 2024-03-18T20:06:58.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.254+0000 7fc714dee700 1 -- 172.21.15.1:0/3170700187 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc71015d290 con 0x7fc71006eda0 2024-03-18T20:06:58.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.254+0000 7fc6feffd700 1 -- 172.21.15.1:0/3170700187 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 99) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fc704003a40 con 0x7fc71006eda0 2024-03-18T20:06:58.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.255+0000 7fc6feffd700 1 --2- 172.21.15.1:0/3170700187 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc6f006ab30 0x7fc6f006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:58.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.255+0000 7fc6feffd700 1 -- 172.21.15.1:0/3170700187 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6445+0+0 (secure 0 0 0) 0x7fc704089fe0 con 0x7fc71006eda0 2024-03-18T20:06:58.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.255+0000 7fc70e59c700 1 --2- 172.21.15.1:0/3170700187 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc6f006ab30 0x7fc6f006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.255+0000 7fc70e59c700 1 --2- 172.21.15.1:0/3170700187 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc6f006ab30 0x7fc6f006cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc7100720d0 tx=0x7fc700009210 comp rx=0 tx=0).ready entity=mgr.10104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:58.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.260+0000 7fc6feffd700 1 -- 172.21.15.1:0/3170700187 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc71015d290 con 0x7fc71006eda0 2024-03-18T20:06:58.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.491+0000 7fc714dee700 1 -- 172.21.15.1:0/3170700187 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7100ce500 con 0x7fc6f006ab30 2024-03-18T20:06:58.502 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:58.497+0000 7f299ddcb700 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-03-18T20:06:58.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.497+0000 7fc6feffd700 1 -- 172.21.15.1:0/3170700187 <== mgr.10104 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7fc700011860 con 0x7fc6f006ab30 2024-03-18T20:06:58.505 INFO:teuthology.orchestra.run.smithi001.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-03-18T20:06:58.509 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-18T20:06:58.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:06:58.510 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-03-18T20:06:58.549 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-18T20:06:58.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.718+0000 7fe86c70a700 1 Processor -- start 2024-03-18T20:06:58.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.718+0000 7fe86c70a700 1 -- start start 2024-03-18T20:06:58.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.719+0000 7fe86c70a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe86415dbe0 0x7fe8640b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:58.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.719+0000 7fe86c70a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe86415a5f0 0x7fe8640cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:58.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.719+0000 7fe86c70a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe8640cc4d0 0x7fe86406d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:58.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.719+0000 7fe86c70a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe86413f750 con 0x7fe86415dbe0 2024-03-18T20:06:58.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.719+0000 7fe86c70a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8640cc930 con 0x7fe8640cc4d0 2024-03-18T20:06:58.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.719+0000 7fe86c70a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe86415aad0 con 0x7fe86415a5f0 2024-03-18T20:06:58.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.719+0000 7fe86a4a6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe86415dbe0 0x7fe8640b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:58.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.720+0000 7fe86a4a6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe86415dbe0 0x7fe8640b2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34276/0 (socket says 172.21.15.1:34276) 2024-03-18T20:06:58.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.720+0000 7fe86a4a6700 1 -- 172.21.15.1:0/3945089001 learned_addr learned my addr 172.21.15.1:0/3945089001 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:58.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.720+0000 7fe86aca7700 1 --2- 172.21.15.1:0/3945089001 >> v2:172.21.15.165:3300/0 conn(0x7fe8640cc4d0 0x7fe86406d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:58.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.720+0000 7fe869ca5700 1 --2- 172.21.15.1:0/3945089001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe86415a5f0 0x7fe8640cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:58.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.720+0000 7fe86a4a6700 1 -- 172.21.15.1:0/3945089001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe86415a5f0 msgr2=0x7fe8640cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:58.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.720+0000 7fe86a4a6700 1 --2- 172.21.15.1:0/3945089001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe86415a5f0 0x7fe8640cbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:58.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.720+0000 7fe86a4a6700 1 -- 172.21.15.1:0/3945089001 >> v2:172.21.15.165:3300/0 conn(0x7fe8640cc4d0 msgr2=0x7fe86406d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:58.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.720+0000 7fe86a4a6700 1 --2- 172.21.15.1:0/3945089001 >> v2:172.21.15.165:3300/0 conn(0x7fe8640cc4d0 0x7fe86406d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:58.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.720+0000 7fe86a4a6700 1 -- 172.21.15.1:0/3945089001 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86415adb0 con 0x7fe86415dbe0 2024-03-18T20:06:58.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.720+0000 7fe86a4a6700 1 --2- 172.21.15.1:0/3945089001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe86415dbe0 0x7fe8640b2fb0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fe8640afbc0 tx=0x7fe85800be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c61304edfd089276 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:58.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.720+0000 7fe8694a4700 1 -- 172.21.15.1:0/3945089001 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85800dc70 con 0x7fe86415dbe0 2024-03-18T20:06:58.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.720+0000 7fe8694a4700 1 -- 172.21.15.1:0/3945089001 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85800ddd0 con 0x7fe86415dbe0 2024-03-18T20:06:58.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.720+0000 7fe8694a4700 1 -- 172.21.15.1:0/3945089001 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe858010690 con 0x7fe86415dbe0 2024-03-18T20:06:58.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.721+0000 7fe86c70a700 1 -- 172.21.15.1:0/3945089001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe86415dbe0 msgr2=0x7fe8640b2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:58.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.721+0000 7fe86c70a700 1 --2- 172.21.15.1:0/3945089001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe86415dbe0 0x7fe8640b2fb0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fe8640afbc0 tx=0x7fe85800be70 comp rx=0 tx=0).stop 2024-03-18T20:06:58.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.721+0000 7fe86c70a700 1 -- 172.21.15.1:0/3945089001 shutdown_connections 2024-03-18T20:06:58.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.721+0000 7fe86c70a700 1 --2- 172.21.15.1:0/3945089001 >> v2:172.21.15.165:3300/0 conn(0x7fe8640cc4d0 0x7fe86406d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:58.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.721+0000 7fe86c70a700 1 --2- 172.21.15.1:0/3945089001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe86415dbe0 0x7fe8640b2fb0 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:58.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.721+0000 7fe86c70a700 1 --2- 172.21.15.1:0/3945089001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe86415a5f0 0x7fe8640cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:58.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.721+0000 7fe86c70a700 1 -- 172.21.15.1:0/3945089001 >> 172.21.15.1:0/3945089001 conn(0x7fe8640b1cb0 msgr2=0x7fe8640b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:58.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.721+0000 7fe86c70a700 1 -- 172.21.15.1:0/3945089001 shutdown_connections 2024-03-18T20:06:58.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.721+0000 7fe86c70a700 1 -- 172.21.15.1:0/3945089001 wait complete. 2024-03-18T20:06:58.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.722+0000 7fe86c70a700 1 Processor -- start 2024-03-18T20:06:58.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.722+0000 7fe86c70a700 1 -- start start 2024-03-18T20:06:58.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.723+0000 7fe86c70a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8640cc4d0 0x7fe86415c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:58.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.723+0000 7fe86c70a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe86415a5f0 0x7fe8641574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:58.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.723+0000 7fe86c70a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe86415dbe0 0x7fe86412b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:58.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.723+0000 7fe86c70a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe864151cc0 con 0x7fe86415a5f0 2024-03-18T20:06:58.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.723+0000 7fe86c70a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86415d6c0 con 0x7fe8640cc4d0 2024-03-18T20:06:58.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.723+0000 7fe86c70a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe86415d100 con 0x7fe86415dbe0 2024-03-18T20:06:58.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.723+0000 7fe869ca5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe86415a5f0 0x7fe8641574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:58.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.723+0000 7fe869ca5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe86415a5f0 0x7fe8641574b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34288/0 (socket says 172.21.15.1:34288) 2024-03-18T20:06:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.723+0000 7fe869ca5700 1 -- 172.21.15.1:0/809661937 learned_addr learned my addr 172.21.15.1:0/809661937 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.723+0000 7fe86a4a6700 1 --2- 172.21.15.1:0/809661937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8640cc4d0 0x7fe86415c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.723+0000 7fe86aca7700 1 --2- 172.21.15.1:0/809661937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe86415dbe0 0x7fe86412b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.723+0000 7fe869ca5700 1 -- 172.21.15.1:0/809661937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe86415dbe0 msgr2=0x7fe86412b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.723+0000 7fe869ca5700 1 --2- 172.21.15.1:0/809661937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe86415dbe0 0x7fe86412b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.723+0000 7fe869ca5700 1 -- 172.21.15.1:0/809661937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8640cc4d0 msgr2=0x7fe86415c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.723+0000 7fe869ca5700 1 --2- 172.21.15.1:0/809661937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8640cc4d0 0x7fe86415c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.723+0000 7fe869ca5700 1 -- 172.21.15.1:0/809661937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85800b7f0 con 0x7fe86415a5f0 2024-03-18T20:06:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.723+0000 7fe869ca5700 1 --2- 172.21.15.1:0/809661937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe86415a5f0 0x7fe8641574b0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fe86000e9b0 tx=0x7fe86000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.723+0000 7fe856ffd700 1 -- 172.21.15.1:0/809661937 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe860009710 con 0x7fe86415a5f0 2024-03-18T20:06:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.723+0000 7fe856ffd700 1 -- 172.21.15.1:0/809661937 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe86000ee50 con 0x7fe86415a5f0 2024-03-18T20:06:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.724+0000 7fe86c70a700 1 -- 172.21.15.1:0/809661937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe86415d3e0 con 0x7fe86415a5f0 2024-03-18T20:06:58.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.724+0000 7fe856ffd700 1 -- 172.21.15.1:0/809661937 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86000ca50 con 0x7fe86415a5f0 2024-03-18T20:06:58.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.724+0000 7fe86c70a700 1 -- 172.21.15.1:0/809661937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8641517a0 con 0x7fe86415a5f0 2024-03-18T20:06:58.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.725+0000 7fe86c70a700 1 -- 172.21.15.1:0/809661937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe864151b10 con 0x7fe86415a5f0 2024-03-18T20:06:58.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.725+0000 7fe856ffd700 1 -- 172.21.15.1:0/809661937 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 99) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fe86000cbb0 con 0x7fe86415a5f0 2024-03-18T20:06:58.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.725+0000 7fe856ffd700 1 --2- 172.21.15.1:0/809661937 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe84806ab30 0x7fe84806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:58.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.725+0000 7fe856ffd700 1 -- 172.21.15.1:0/809661937 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6445+0+0 (secure 0 0 0) 0x7fe8600892c0 con 0x7fe86415a5f0 2024-03-18T20:06:58.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.725+0000 7fe86a4a6700 1 --2- 172.21.15.1:0/809661937 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe84806ab30 0x7fe84806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:58.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.726+0000 7fe86a4a6700 1 --2- 172.21.15.1:0/809661937 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe84806ab30 0x7fe84806cff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe85800beb0 tx=0x7fe858013000 comp rx=0 tx=0).ready entity=mgr.10104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:58.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.730+0000 7fe856ffd700 1 -- 172.21.15.1:0/809661937 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe860054430 con 0x7fe86415a5f0 2024-03-18T20:06:58.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:58.991+0000 7fe86c70a700 1 -- 172.21.15.1:0/809661937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fe8640676a0 con 0x7fe86415a5f0 2024-03-18T20:06:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.614+0000 7fe856ffd700 1 -- 172.21.15.1:0/809661937 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 100) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fe86005e0d0 con 0x7fe86415a5f0 2024-03-18T20:06:59.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.616+0000 7fe856ffd700 1 -- 172.21.15.1:0/809661937 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v100) v1 ==== 93+0+0 (secure 0 0 0) 0x7fe8600873d0 con 0x7fe86415a5f0 2024-03-18T20:06:59.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.620+0000 7fe86c70a700 1 -- 172.21.15.1:0/809661937 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe84806ab30 msgr2=0x7fe84806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:59.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.620+0000 7fe86c70a700 1 --2- 172.21.15.1:0/809661937 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe84806ab30 0x7fe84806cff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe85800beb0 tx=0x7fe858013000 comp rx=0 tx=0).stop 2024-03-18T20:06:59.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.620+0000 7fe86c70a700 1 -- 172.21.15.1:0/809661937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe86415a5f0 msgr2=0x7fe8641574b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:59.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.620+0000 7fe86c70a700 1 --2- 172.21.15.1:0/809661937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe86415a5f0 0x7fe8641574b0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fe86000e9b0 tx=0x7fe86000ecc0 comp rx=0 tx=0).stop 2024-03-18T20:06:59.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.620+0000 7fe86c70a700 1 -- 172.21.15.1:0/809661937 shutdown_connections 2024-03-18T20:06:59.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.620+0000 7fe86c70a700 1 --2- 172.21.15.1:0/809661937 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe84806ab30 0x7fe84806cff0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:59.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.620+0000 7fe86c70a700 1 --2- 172.21.15.1:0/809661937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe86415dbe0 0x7fe86412b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:59.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.620+0000 7fe86c70a700 1 --2- 172.21.15.1:0/809661937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8640cc4d0 0x7fe86415c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:59.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.620+0000 7fe86c70a700 1 --2- 172.21.15.1:0/809661937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe86415a5f0 0x7fe8641574b0 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:59.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.620+0000 7fe86c70a700 1 -- 172.21.15.1:0/809661937 >> 172.21.15.1:0/809661937 conn(0x7fe8640b1cb0 msgr2=0x7fe8640b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:59.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.620+0000 7fe86c70a700 1 -- 172.21.15.1:0/809661937 shutdown_connections 2024-03-18T20:06:59.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.620+0000 7fe86c70a700 1 -- 172.21.15.1:0/809661937 wait complete. 2024-03-18T20:06:59.641 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T20:06:59.641 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:06:59.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:06:59.728 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T20:06:59.729 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:06:59.729 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:06:59.748 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:06:59.773 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:59.768+0000 7f3dc9f9a240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:06:59.777 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:59.772+0000 7f78be4a7240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:06:59.918 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:06:59.913+0000 7f3dc9f9a240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:06:59.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.941+0000 7f009d6a7700 1 Processor -- start 2024-03-18T20:06:59.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.941+0000 7f009d6a7700 1 -- start start 2024-03-18T20:06:59.947 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:06:59.941+0000 7f78be4a7240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:06:59.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.942+0000 7f009d6a7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0098160940 0x7f00980d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:59.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.942+0000 7f009d6a7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00980cddc0 con 0x7f00980cf090 2024-03-18T20:06:59.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.942+0000 7f009d6a7700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00980cdff0 con 0x7f00980d1ea0 2024-03-18T20:06:59.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.942+0000 7f009d6a7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00980ce2d0 con 0x7f0098160940 2024-03-18T20:06:59.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.942+0000 7f00967fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f00980cf090 0x7f00980d1790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36064/0 (socket says 172.21.15.1:36064) 2024-03-18T20:06:59.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.942+0000 7f00967fc700 1 -- 172.21.15.1:0/2660089498 learned_addr learned my addr 172.21.15.1:0/2660089498 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:06:59.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.942+0000 7f0096ffd700 1 --2- 172.21.15.1:0/2660089498 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0098160940 0x7f00980d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:59.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.942+0000 7f0095ffb700 1 -- 172.21.15.1:0/2660089498 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889487283 0 0) 0x7f00980cddc0 con 0x7f00980cf090 2024-03-18T20:06:59.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.942+0000 7f0095ffb700 1 -- 172.21.15.1:0/2660089498 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00800035b0 con 0x7f00980cf090 2024-03-18T20:06:59.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0095ffb700 1 -- 172.21.15.1:0/2660089498 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271101387 0 0) 0x7f00980cdff0 con 0x7f00980d1ea0 2024-03-18T20:06:59.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0095ffb700 1 -- 172.21.15.1:0/2660089498 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00980cddc0 con 0x7f00980d1ea0 2024-03-18T20:06:59.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0095ffb700 1 -- 172.21.15.1:0/2660089498 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1670575113 0 0) 0x7f00800035b0 con 0x7f00980cf090 2024-03-18T20:06:59.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0095ffb700 1 -- 172.21.15.1:0/2660089498 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0098160940 msgr2=0x7f00980d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:59.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0095ffb700 1 --2- 172.21.15.1:0/2660089498 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0098160940 0x7f00980d0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:59.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0095ffb700 1 -- 172.21.15.1:0/2660089498 >> v1:172.21.15.165:6789/0 conn(0x7f00980d1ea0 legacy=0x7f00980b25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:59.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0095ffb700 1 -- 172.21.15.1:0/2660089498 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00980ce5b0 con 0x7f00980cf090 2024-03-18T20:06:59.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0095ffb700 1 -- 172.21.15.1:0/2660089498 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f00880015f0 con 0x7f00980cf090 2024-03-18T20:06:59.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0095ffb700 1 -- 172.21.15.1:0/2660089498 >> v1:172.21.15.1:6789/0 conn(0x7f00980cf090 legacy=0x7f00980d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:59.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0095ffb700 1 --2- 172.21.15.1:0/2660089498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0080003bc0 0x7f0080003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:59.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0095ffb700 1 --2- 172.21.15.1:0/2660089498 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0098160940 0x7f0080004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:59.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0095ffb700 1 --2- 172.21.15.1:0/2660089498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0080004d90 0x7f0080005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:59.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0095ffb700 1 -- 172.21.15.1:0/2660089498 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00980ce2d0 con 0x7f0080003bc0 2024-03-18T20:06:59.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0095ffb700 1 -- 172.21.15.1:0/2660089498 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008000bac0 con 0x7f0080004d90 2024-03-18T20:06:59.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0095ffb700 1 -- 172.21.15.1:0/2660089498 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f008000bc80 con 0x7f0098160940 2024-03-18T20:06:59.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0096ffd700 1 --2- 172.21.15.1:0/2660089498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0080003bc0 0x7f0080003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:59.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f00967fc700 1 --2- 172.21.15.1:0/2660089498 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0098160940 0x7f0080004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:59.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f00977fe700 1 --2- 172.21.15.1:0/2660089498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0080004d90 0x7f0080005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:59.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0096ffd700 1 -- 172.21.15.1:0/2660089498 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0098160940 msgr2=0x7f0080004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:59.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0096ffd700 1 --2- 172.21.15.1:0/2660089498 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0098160940 0x7f0080004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:59.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0096ffd700 1 -- 172.21.15.1:0/2660089498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0080004d90 msgr2=0x7f0080005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:06:59.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0096ffd700 1 --2- 172.21.15.1:0/2660089498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0080004d90 0x7f0080005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:59.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0096ffd700 1 -- 172.21.15.1:0/2660089498 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00800035b0 con 0x7f0080003bc0 2024-03-18T20:06:59.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f00967fc700 1 --2- 172.21.15.1:0/2660089498 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0098160940 0x7f0080004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:59.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f00977fe700 1 --2- 172.21.15.1:0/2660089498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0080004d90 0x7f0080005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:06:59.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0096ffd700 1 --2- 172.21.15.1:0/2660089498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0080003bc0 0x7f0080003fc0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f0090009fc0 tx=0x7f0080005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ffdb22d9184745c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:59.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0095ffb700 1 -- 172.21.15.1:0/2660089498 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009000ec30 con 0x7f0080003bc0 2024-03-18T20:06:59.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0095ffb700 1 -- 172.21.15.1:0/2660089498 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009000f040 con 0x7f0080003bc0 2024-03-18T20:06:59.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.943+0000 7f0095ffb700 1 -- 172.21.15.1:0/2660089498 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0090014480 con 0x7f0080003bc0 2024-03-18T20:06:59.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.944+0000 7f009d6a7700 1 -- 172.21.15.1:0/2660089498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0080003bc0 msgr2=0x7f0080003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:59.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.944+0000 7f009d6a7700 1 --2- 172.21.15.1:0/2660089498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0080003bc0 0x7f0080003fc0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f0090009fc0 tx=0x7f0080005880 comp rx=0 tx=0).stop 2024-03-18T20:06:59.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.944+0000 7f00977fe700 1 -- 172.21.15.1:0/2660089498 reap_dead start 2024-03-18T20:06:59.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.944+0000 7f009d6a7700 1 -- 172.21.15.1:0/2660089498 shutdown_connections 2024-03-18T20:06:59.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.944+0000 7f009d6a7700 1 -- 172.21.15.1:0/2660089498 >> 172.21.15.1:0/2660089498 conn(0x7f009809c520 msgr2=0x7f00980d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:06:59.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.944+0000 7f009d6a7700 1 -- 172.21.15.1:0/2660089498 shutdown_connections 2024-03-18T20:06:59.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.944+0000 7f009d6a7700 1 -- 172.21.15.1:0/2660089498 wait complete. 2024-03-18T20:06:59.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.945+0000 7f009d6a7700 1 Processor -- start 2024-03-18T20:06:59.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.946+0000 7f009d6a7700 1 -- start start 2024-03-18T20:06:59.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.946+0000 7f009d6a7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0080003bc0 0x7f009814c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:59.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.946+0000 7f009d6a7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00980caf10 0x7f0098156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:59.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.946+0000 7f009d6a7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00980cb390 0x7f0098156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:59.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.946+0000 7f009d6a7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009814cc20 con 0x7f0080003bc0 2024-03-18T20:06:59.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.946+0000 7f009d6a7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00980cc430 con 0x7f00980caf10 2024-03-18T20:06:59.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.946+0000 7f009d6a7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00980cc710 con 0x7f00980cb390 2024-03-18T20:06:59.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.946+0000 7f0096ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0080003bc0 0x7f009814c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:59.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.946+0000 7f0096ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0080003bc0 0x7f009814c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34316/0 (socket says 172.21.15.1:34316) 2024-03-18T20:06:59.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.946+0000 7f0096ffd700 1 -- 172.21.15.1:0/4015438731 learned_addr learned my addr 172.21.15.1:0/4015438731 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:06:59.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.946+0000 7f00977fe700 1 --2- 172.21.15.1:0/4015438731 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00980cb390 0x7f0098156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:59.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.947+0000 7f00967fc700 1 --2- 172.21.15.1:0/4015438731 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00980caf10 0x7f0098156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:06:59.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.947+0000 7f0096ffd700 1 -- 172.21.15.1:0/4015438731 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00980cb390 msgr2=0x7f0098156ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:59.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.947+0000 7f0096ffd700 1 --2- 172.21.15.1:0/4015438731 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00980cb390 0x7f0098156ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:59.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.947+0000 7f0096ffd700 1 -- 172.21.15.1:0/4015438731 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00980caf10 msgr2=0x7f0098156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:06:59.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.947+0000 7f0096ffd700 1 --2- 172.21.15.1:0/4015438731 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00980caf10 0x7f0098156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:06:59.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.947+0000 7f0096ffd700 1 -- 172.21.15.1:0/4015438731 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00980cc9f0 con 0x7f0080003bc0 2024-03-18T20:06:59.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.947+0000 7f0096ffd700 1 --2- 172.21.15.1:0/4015438731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0080003bc0 0x7f009814c6e0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f009000e700 tx=0x7f0090009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:06:59.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.947+0000 7f007f7fe700 1 -- 172.21.15.1:0/4015438731 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0090004450 con 0x7f0080003bc0 2024-03-18T20:06:59.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.947+0000 7f007f7fe700 1 -- 172.21.15.1:0/4015438731 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00900045d0 con 0x7f0080003bc0 2024-03-18T20:06:59.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.947+0000 7f007f7fe700 1 -- 172.21.15.1:0/4015438731 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0090003e10 con 0x7f0080003bc0 2024-03-18T20:06:59.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.948+0000 7f009d6a7700 1 -- 172.21.15.1:0/4015438731 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00980cccd0 con 0x7f0080003bc0 2024-03-18T20:06:59.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.948+0000 7f009d6a7700 1 -- 172.21.15.1:0/4015438731 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00981559c0 con 0x7f0080003bc0 2024-03-18T20:06:59.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.948+0000 7f009d6a7700 1 -- 172.21.15.1:0/4015438731 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0098073350 con 0x7f0080003bc0 2024-03-18T20:06:59.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.949+0000 7f007f7fe700 1 -- 172.21.15.1:0/4015438731 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 100) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f0090024070 con 0x7f0080003bc0 2024-03-18T20:06:59.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.950+0000 7f007f7fe700 1 --2- 172.21.15.1:0/4015438731 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f008406adb0 0x7f008406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:06:59.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.950+0000 7f00967fc700 1 -- 172.21.15.1:0/4015438731 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f008406adb0 msgr2=0x7f008406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:06:59.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.950+0000 7f007f7fe700 1 -- 172.21.15.1:0/4015438731 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f009008b5b0 con 0x7f0080003bc0 2024-03-18T20:06:59.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.950+0000 7f00967fc700 1 --2- 172.21.15.1:0/4015438731 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f008406adb0 0x7f008406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:06:59.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:06:59.954+0000 7f007f7fe700 1 -- 172.21.15.1:0/4015438731 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0090056550 con 0x7f0080003bc0 2024-03-18T20:07:00.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.151+0000 7f00967fc700 1 -- 172.21.15.1:0/4015438731 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f008406adb0 msgr2=0x7f008406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:00.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.151+0000 7f00967fc700 1 --2- 172.21.15.1:0/4015438731 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f008406adb0 0x7f008406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:07:00.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.159+0000 7f009d6a7700 1 -- 172.21.15.1:0/4015438731 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00980d0980 con 0x7f0080003bc0 2024-03-18T20:07:00.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.160+0000 7f007f7fe700 1 -- 172.21.15.1:0/4015438731 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v182) v1 ==== 74+0+16804 (secure 0 0 0) 0x7f0090059b70 con 0x7f0080003bc0 2024-03-18T20:07:00.166 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:07:00.167 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":182,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:06:56.591970+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:06:39.228206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-18T20:06:40.315627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6833/76998":"2024-03-19T20:06:56.591909+0000","172.21.15.1:6832/76998":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/2464679874":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/251298994":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/1615725209":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/1979178380":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/855901535":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/2343543091":"2024-03-19T20:06:56.591909+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:07:00.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.163+0000 7f009d6a7700 1 -- 172.21.15.1:0/4015438731 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f008406adb0 msgr2=0x7f008406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:07:00.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.163+0000 7f009d6a7700 1 --2- 172.21.15.1:0/4015438731 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f008406adb0 0x7f008406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:00.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.163+0000 7f009d6a7700 1 -- 172.21.15.1:0/4015438731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0080003bc0 msgr2=0x7f009814c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:00.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.163+0000 7f009d6a7700 1 --2- 172.21.15.1:0/4015438731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0080003bc0 0x7f009814c6e0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f009000e700 tx=0x7f0090009750 comp rx=0 tx=0).stop 2024-03-18T20:07:00.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.163+0000 7f009d6a7700 1 -- 172.21.15.1:0/4015438731 shutdown_connections 2024-03-18T20:07:00.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.163+0000 7f009d6a7700 1 --2- 172.21.15.1:0/4015438731 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f008406adb0 0x7f008406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:00.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.163+0000 7f009d6a7700 1 --2- 172.21.15.1:0/4015438731 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00980cb390 0x7f0098156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:00.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.163+0000 7f009d6a7700 1 --2- 172.21.15.1:0/4015438731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0080003bc0 0x7f009814c6e0 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:00.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.163+0000 7f009d6a7700 1 --2- 172.21.15.1:0/4015438731 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00980caf10 0x7f0098156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:00.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.163+0000 7f009d6a7700 1 -- 172.21.15.1:0/4015438731 >> 172.21.15.1:0/4015438731 conn(0x7f009809c520 msgr2=0x7f00980b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:00.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.163+0000 7f009d6a7700 1 -- 172.21.15.1:0/4015438731 shutdown_connections 2024-03-18T20:07:00.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.163+0000 7f009d6a7700 1 -- 172.21.15.1:0/4015438731 wait complete. 2024-03-18T20:07:00.182 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:06:39.228206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '180', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:06:40.315627+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '181', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:07:00.182 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:07:00.183 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:00.179+0000 7f3dc9f9a240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:07:00.227 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:00.221+0000 7f78be4a7240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:07:00.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.308+0000 7f6ca9977700 1 Processor -- start 2024-03-18T20:07:00.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.308+0000 7f6ca9977700 1 -- start start 2024-03-18T20:07:00.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.308+0000 7f6ca9977700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6ca40cd870 0x7f6ca40cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:00.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.308+0000 7f6ca9977700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca40d0df0 con 0x7f6ca40b1e20 2024-03-18T20:07:00.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.308+0000 7f6ca9977700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca40d0550 con 0x7f6ca40cbce0 2024-03-18T20:07:00.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.308+0000 7f6ca9977700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca40d0830 con 0x7f6ca40cd870 2024-03-18T20:07:00.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.309+0000 7f6ca27fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6ca40b1e20 0x7f6ca40cdc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36066/0 (socket says 172.21.15.1:36066) 2024-03-18T20:07:00.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.309+0000 7f6ca27fc700 1 -- 172.21.15.1:0/1163427927 learned_addr learned my addr 172.21.15.1:0/1163427927 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:00.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.309+0000 7f6ca2ffd700 1 --2- 172.21.15.1:0/1163427927 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40cd870 0x7f6ca40cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:00.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.309+0000 7f6ca1ffb700 1 -- 172.21.15.1:0/1163427927 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822369704 0 0) 0x7f6ca40d0df0 con 0x7f6ca40b1e20 2024-03-18T20:07:00.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.309+0000 7f6ca1ffb700 1 -- 172.21.15.1:0/1163427927 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c8c0035b0 con 0x7f6ca40b1e20 2024-03-18T20:07:00.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.309+0000 7f6ca2ffd700 1 -- 172.21.15.1:0/1163427927 >> v1:172.21.15.165:6789/0 conn(0x7f6ca40cbce0 legacy=0x7f6ca40cc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:00.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.309+0000 7f6ca2ffd700 1 -- 172.21.15.1:0/1163427927 >> v1:172.21.15.1:6789/0 conn(0x7f6ca40b1e20 legacy=0x7f6ca40cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:00.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.309+0000 7f6ca2ffd700 1 -- 172.21.15.1:0/1163427927 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca40d0b10 con 0x7f6ca40cd870 2024-03-18T20:07:00.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.309+0000 7f6ca2ffd700 1 --2- 172.21.15.1:0/1163427927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ca40cd870 0x7f6ca40cff70 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6c94009840 tx=0x7f6c94009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f1816f690b7cc6b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:00.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.309+0000 7f6ca1ffb700 1 -- 172.21.15.1:0/1163427927 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c94011830 con 0x7f6ca40cd870 2024-03-18T20:07:00.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.309+0000 7f6ca1ffb700 1 -- 172.21.15.1:0/1163427927 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c94011990 con 0x7f6ca40cd870 2024-03-18T20:07:00.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.309+0000 7f6ca1ffb700 1 -- 172.21.15.1:0/1163427927 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c94011d20 con 0x7f6ca40cd870 2024-03-18T20:07:00.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.310+0000 7f6ca9977700 1 -- 172.21.15.1:0/1163427927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ca40cd870 msgr2=0x7f6ca40cff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:00.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.310+0000 7f6ca9977700 1 --2- 172.21.15.1:0/1163427927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ca40cd870 0x7f6ca40cff70 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6c94009840 tx=0x7f6c94009b50 comp rx=0 tx=0).stop 2024-03-18T20:07:00.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.310+0000 7f6ca9977700 1 -- 172.21.15.1:0/1163427927 shutdown_connections 2024-03-18T20:07:00.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.310+0000 7f6ca9977700 1 --2- 172.21.15.1:0/1163427927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ca40cd870 0x7f6ca40cff70 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:00.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.310+0000 7f6ca9977700 1 -- 172.21.15.1:0/1163427927 >> 172.21.15.1:0/1163427927 conn(0x7f6ca40d0ff0 msgr2=0x7f6ca40d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:00.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.310+0000 7f6ca9977700 1 -- 172.21.15.1:0/1163427927 shutdown_connections 2024-03-18T20:07:00.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.310+0000 7f6ca9977700 1 -- 172.21.15.1:0/1163427927 wait complete. 2024-03-18T20:07:00.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.311+0000 7f6ca9977700 1 Processor -- start 2024-03-18T20:07:00.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.311+0000 7f6ca9977700 1 -- start start 2024-03-18T20:07:00.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.312+0000 7f6ca9977700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ca40b1e20 0x7f6ca41e2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:00.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.312+0000 7f6ca9977700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca40cbce0 0x7f6ca4147460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:00.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.312+0000 7f6ca9977700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6ca40cd870 0x7f6ca41e54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:00.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.312+0000 7f6ca9977700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca41479a0 con 0x7f6ca40cbce0 2024-03-18T20:07:00.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.312+0000 7f6ca9977700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca41e5d50 con 0x7f6ca40b1e20 2024-03-18T20:07:00.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.312+0000 7f6ca9977700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca41e6030 con 0x7f6ca40cd870 2024-03-18T20:07:00.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.312+0000 7f6ca27fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca40cbce0 0x7f6ca4147460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:00.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.312+0000 7f6ca27fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca40cbce0 0x7f6ca4147460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34326/0 (socket says 172.21.15.1:34326) 2024-03-18T20:07:00.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.312+0000 7f6ca27fc700 1 -- 172.21.15.1:0/1298042885 learned_addr learned my addr 172.21.15.1:0/1298042885 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:00.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.312+0000 7f6ca2ffd700 1 --2- 172.21.15.1:0/1298042885 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ca40b1e20 0x7f6ca41e2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:00.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.312+0000 7f6ca37fe700 1 --2- 172.21.15.1:0/1298042885 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6ca40cd870 0x7f6ca41e54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:00.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.313+0000 7f6ca27fc700 1 -- 172.21.15.1:0/1298042885 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6ca40cd870 msgr2=0x7f6ca41e54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:00.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.313+0000 7f6ca27fc700 1 --2- 172.21.15.1:0/1298042885 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6ca40cd870 0x7f6ca41e54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:00.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.313+0000 7f6ca27fc700 1 -- 172.21.15.1:0/1298042885 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ca40b1e20 msgr2=0x7f6ca41e2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:00.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.313+0000 7f6ca27fc700 1 --2- 172.21.15.1:0/1298042885 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ca40b1e20 0x7f6ca41e2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:00.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.313+0000 7f6ca27fc700 1 -- 172.21.15.1:0/1298042885 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c940094a0 con 0x7f6ca40cbce0 2024-03-18T20:07:00.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.313+0000 7f6ca37fe700 1 --2- 172.21.15.1:0/1298042885 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6ca40cd870 0x7f6ca41e54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:07:00.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.313+0000 7f6ca27fc700 1 --2- 172.21.15.1:0/1298042885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca40cbce0 0x7f6ca4147460 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800c700 tx=0x7f6c9800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:00.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.313+0000 7f6c8bfff700 1 -- 172.21.15.1:0/1298042885 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98005d70 con 0x7f6ca40cbce0 2024-03-18T20:07:00.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.313+0000 7f6c8bfff700 1 -- 172.21.15.1:0/1298042885 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c980070d0 con 0x7f6ca40cbce0 2024-03-18T20:07:00.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.313+0000 7f6ca9977700 1 -- 172.21.15.1:0/1298042885 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca41e6310 con 0x7f6ca40cbce0 2024-03-18T20:07:00.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.313+0000 7f6c8bfff700 1 -- 172.21.15.1:0/1298042885 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98007460 con 0x7f6ca40cbce0 2024-03-18T20:07:00.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.313+0000 7f6ca9977700 1 -- 172.21.15.1:0/1298042885 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca41e68d0 con 0x7f6ca40cbce0 2024-03-18T20:07:00.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.314+0000 7f6ca9977700 1 -- 172.21.15.1:0/1298042885 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca4159830 con 0x7f6ca40cbce0 2024-03-18T20:07:00.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.314+0000 7f6c8bfff700 1 -- 172.21.15.1:0/1298042885 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 100) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f6c980075c0 con 0x7f6ca40cbce0 2024-03-18T20:07:00.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.315+0000 7f6c8bfff700 1 --2- 172.21.15.1:0/1298042885 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c9006adb0 0x7f6c9006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:00.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.315+0000 7f6ca2ffd700 1 -- 172.21.15.1:0/1298042885 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c9006adb0 msgr2=0x7f6c9006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:00.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.315+0000 7f6c8bfff700 1 -- 172.21.15.1:0/1298042885 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f6c98088c90 con 0x7f6ca40cbce0 2024-03-18T20:07:00.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.315+0000 7f6ca2ffd700 1 --2- 172.21.15.1:0/1298042885 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c9006adb0 0x7f6c9006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:07:00.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.320+0000 7f6c8bfff700 1 -- 172.21.15.1:0/1298042885 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6c98053a80 con 0x7f6ca40cbce0 2024-03-18T20:07:00.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.516+0000 7f6ca2ffd700 1 -- 172.21.15.1:0/1298042885 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c9006adb0 msgr2=0x7f6c9006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:00.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.516+0000 7f6ca2ffd700 1 --2- 172.21.15.1:0/1298042885 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c9006adb0 0x7f6c9006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:07:00.609 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:00.604+0000 7f3dc9f9a240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:07:00.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.640+0000 7f6ca9977700 1 -- 172.21.15.1:0/1298042885 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6ca40676a0 con 0x7f6ca40cbce0 2024-03-18T20:07:00.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.640+0000 7f6c8bfff700 1 -- 172.21.15.1:0/1298042885 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v182) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6c980170a0 con 0x7f6ca40cbce0 2024-03-18T20:07:00.646 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:07:00.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.643+0000 7f6ca9977700 1 -- 172.21.15.1:0/1298042885 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c9006adb0 msgr2=0x7f6c9006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:07:00.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.643+0000 7f6ca9977700 1 --2- 172.21.15.1:0/1298042885 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c9006adb0 0x7f6c9006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:00.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.643+0000 7f6ca9977700 1 -- 172.21.15.1:0/1298042885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca40cbce0 msgr2=0x7f6ca4147460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:00.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.643+0000 7f6ca9977700 1 --2- 172.21.15.1:0/1298042885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca40cbce0 0x7f6ca4147460 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800c700 tx=0x7f6c9800ca10 comp rx=0 tx=0).stop 2024-03-18T20:07:00.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.643+0000 7f6ca9977700 1 -- 172.21.15.1:0/1298042885 shutdown_connections 2024-03-18T20:07:00.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.643+0000 7f6ca9977700 1 --2- 172.21.15.1:0/1298042885 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6c9006adb0 0x7f6c9006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:00.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.643+0000 7f6ca9977700 1 --2- 172.21.15.1:0/1298042885 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6ca40cd870 0x7f6ca41e54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:00.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.643+0000 7f6ca9977700 1 --2- 172.21.15.1:0/1298042885 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ca40b1e20 0x7f6ca41e2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:00.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.643+0000 7f6ca9977700 1 --2- 172.21.15.1:0/1298042885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca40cbce0 0x7f6ca4147460 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:00.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.643+0000 7f6ca9977700 1 -- 172.21.15.1:0/1298042885 >> 172.21.15.1:0/1298042885 conn(0x7f6ca40d0ff0 msgr2=0x7f6ca40c8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:00.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.643+0000 7f6ca9977700 1 -- 172.21.15.1:0/1298042885 shutdown_connections 2024-03-18T20:07:00.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.643+0000 7f6ca9977700 1 -- 172.21.15.1:0/1298042885 wait complete. 2024-03-18T20:07:00.661 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:07:00.778 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:00.769+0000 7f78be4a7240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:07:00.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.785+0000 7f27d46c7700 1 Processor -- start 2024-03-18T20:07:00.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.785+0000 7f27d46c7700 1 -- start start 2024-03-18T20:07:00.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.786+0000 7f27d46c7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27cc06efb0 0x7f27cc071da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.786+0000 7f27d46c7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc06da90 con 0x7f27cc15d580 2024-03-18T20:07:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.786+0000 7f27d46c7700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc06ab30 con 0x7f27cc0722e0 2024-03-18T20:07:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.786+0000 7f27d46c7700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27cc072770 con 0x7f27cc06efb0 2024-03-18T20:07:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.786+0000 7f27d1c62700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27cc06efb0 0x7f27cc071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.786+0000 7f27d2463700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f27cc15d580 0x7f27cc06d380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36080/0 (socket says 172.21.15.1:36080) 2024-03-18T20:07:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.786+0000 7f27d2463700 1 -- 172.21.15.1:0/3895451002 learned_addr learned my addr 172.21.15.1:0/3895451002 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.787+0000 7f27d1c62700 1 -- 172.21.15.1:0/3895451002 >> v1:172.21.15.165:6789/0 conn(0x7f27cc0722e0 legacy=0x7f27cc0cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.787+0000 7f27d1c62700 1 -- 172.21.15.1:0/3895451002 >> v1:172.21.15.1:6789/0 conn(0x7f27cc15d580 legacy=0x7f27cc06d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.787+0000 7f27d1c62700 1 -- 172.21.15.1:0/3895451002 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc06ae10 con 0x7f27cc06efb0 2024-03-18T20:07:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.787+0000 7f27d1c62700 1 --2- 172.21.15.1:0/3895451002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27cc06efb0 0x7f27cc071da0 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f27c0009870 tx=0x7f27c0009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7726d627510c59 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.787+0000 7f27d1461700 1 -- 172.21.15.1:0/3895451002 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c0011830 con 0x7f27cc06efb0 2024-03-18T20:07:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.787+0000 7f27d1461700 1 -- 172.21.15.1:0/3895451002 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c0011990 con 0x7f27cc06efb0 2024-03-18T20:07:00.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.787+0000 7f27d1461700 1 -- 172.21.15.1:0/3895451002 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c0011d20 con 0x7f27cc06efb0 2024-03-18T20:07:00.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.788+0000 7f27d46c7700 1 -- 172.21.15.1:0/3895451002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27cc06efb0 msgr2=0x7f27cc071da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:00.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.788+0000 7f27d46c7700 1 --2- 172.21.15.1:0/3895451002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27cc06efb0 0x7f27cc071da0 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f27c0009870 tx=0x7f27c0009b80 comp rx=0 tx=0).stop 2024-03-18T20:07:00.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.788+0000 7f27d46c7700 1 -- 172.21.15.1:0/3895451002 shutdown_connections 2024-03-18T20:07:00.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.788+0000 7f27d46c7700 1 --2- 172.21.15.1:0/3895451002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27cc06efb0 0x7f27cc071da0 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:00.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.788+0000 7f27d46c7700 1 -- 172.21.15.1:0/3895451002 >> 172.21.15.1:0/3895451002 conn(0x7f27cc0cca00 msgr2=0x7f27cc0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:00.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.788+0000 7f27d46c7700 1 -- 172.21.15.1:0/3895451002 shutdown_connections 2024-03-18T20:07:00.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.788+0000 7f27d46c7700 1 -- 172.21.15.1:0/3895451002 wait complete. 2024-03-18T20:07:00.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.789+0000 7f27d46c7700 1 Processor -- start 2024-03-18T20:07:00.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.789+0000 7f27d46c7700 1 -- start start 2024-03-18T20:07:00.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.790+0000 7f27d46c7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27cc06efb0 0x7f27cc0c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:00.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.790+0000 7f27d46c7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27cc0722e0 0x7f27cc14f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:00.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.790+0000 7f27d46c7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27cc15d580 0x7f27cc0b4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:00.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.790+0000 7f27d46c7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc0c8930 con 0x7f27cc15d580 2024-03-18T20:07:00.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.790+0000 7f27d46c7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc0c7a60 con 0x7f27cc06efb0 2024-03-18T20:07:00.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.790+0000 7f27d46c7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc0cac00 con 0x7f27cc0722e0 2024-03-18T20:07:00.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.790+0000 7f27d2c64700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27cc15d580 0x7f27cc0b4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:00.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.790+0000 7f27d2c64700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27cc15d580 0x7f27cc0b4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34344/0 (socket says 172.21.15.1:34344) 2024-03-18T20:07:00.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.790+0000 7f27d2c64700 1 -- 172.21.15.1:0/2564866548 learned_addr learned my addr 172.21.15.1:0/2564866548 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:00.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.790+0000 7f27d2463700 1 --2- 172.21.15.1:0/2564866548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27cc06efb0 0x7f27cc0c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:00.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.790+0000 7f27d1c62700 1 --2- 172.21.15.1:0/2564866548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27cc0722e0 0x7f27cc14f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:00.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.790+0000 7f27d2c64700 1 -- 172.21.15.1:0/2564866548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27cc0722e0 msgr2=0x7f27cc14f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:00.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.790+0000 7f27d2c64700 1 --2- 172.21.15.1:0/2564866548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27cc0722e0 0x7f27cc14f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:00.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.790+0000 7f27d2c64700 1 -- 172.21.15.1:0/2564866548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27cc06efb0 msgr2=0x7f27cc0c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:00.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.790+0000 7f27d2c64700 1 --2- 172.21.15.1:0/2564866548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27cc06efb0 0x7f27cc0c8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:00.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.790+0000 7f27d2c64700 1 -- 172.21.15.1:0/2564866548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c00094a0 con 0x7f27cc15d580 2024-03-18T20:07:00.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.790+0000 7f27d1c62700 1 --2- 172.21.15.1:0/2564866548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27cc0722e0 0x7f27cc14f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:07:00.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.790+0000 7f27d2c64700 1 --2- 172.21.15.1:0/2564866548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27cc15d580 0x7f27cc0b4db0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f27c400cb80 tx=0x7f27c400ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:00.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.790+0000 7f27beffd700 1 -- 172.21.15.1:0/2564866548 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c4003c00 con 0x7f27cc15d580 2024-03-18T20:07:00.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.791+0000 7f27beffd700 1 -- 172.21.15.1:0/2564866548 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c4003d60 con 0x7f27cc15d580 2024-03-18T20:07:00.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.791+0000 7f27d46c7700 1 -- 172.21.15.1:0/2564866548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27cc0c7cb0 con 0x7f27cc15d580 2024-03-18T20:07:00.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.791+0000 7f27beffd700 1 -- 172.21.15.1:0/2564866548 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c400ab00 con 0x7f27cc15d580 2024-03-18T20:07:00.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.791+0000 7f27d46c7700 1 -- 172.21.15.1:0/2564866548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27cc0c8270 con 0x7f27cc15d580 2024-03-18T20:07:00.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.792+0000 7f27d46c7700 1 -- 172.21.15.1:0/2564866548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27cc0c8a70 con 0x7f27cc15d580 2024-03-18T20:07:00.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.792+0000 7f27beffd700 1 -- 172.21.15.1:0/2564866548 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 100) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f27c400ac60 con 0x7f27cc15d580 2024-03-18T20:07:00.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.792+0000 7f27beffd700 1 --2- 172.21.15.1:0/2564866548 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f27b006adb0 0x7f27b006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:00.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.792+0000 7f27d2463700 1 -- 172.21.15.1:0/2564866548 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f27b006adb0 msgr2=0x7f27b006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:00.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.793+0000 7f27beffd700 1 -- 172.21.15.1:0/2564866548 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f27c40896b0 con 0x7f27cc15d580 2024-03-18T20:07:00.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.793+0000 7f27d2463700 1 --2- 172.21.15.1:0/2564866548 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f27b006adb0 0x7f27b006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:07:00.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.797+0000 7f27beffd700 1 -- 172.21.15.1:0/2564866548 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f27cc0c8a70 con 0x7f27cc15d580 2024-03-18T20:07:00.818 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:00.814+0000 7f3dc9f9a240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:07:00.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.994+0000 7f27d2463700 1 -- 172.21.15.1:0/2564866548 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f27b006adb0 msgr2=0x7f27b006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:00.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:00.994+0000 7f27d2463700 1 --2- 172.21.15.1:0/2564866548 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f27b006adb0 0x7f27b006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:07:01.013 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:01.007+0000 7f78be4a7240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:07:01.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.018+0000 7f27d46c7700 1 -- 172.21.15.1:0/2564866548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f27cc0c8a70 con 0x7f27cc15d580 2024-03-18T20:07:01.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.018+0000 7f27beffd700 1 -- 172.21.15.1:0/2564866548 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v182) v1 ==== 104+0+11 (secure 0 0 0) 0x7f27cc0c8a70 con 0x7f27cc15d580 2024-03-18T20:07:01.024 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:07:01.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.020+0000 7f27d46c7700 1 -- 172.21.15.1:0/2564866548 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f27b006adb0 msgr2=0x7f27b006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:07:01.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.020+0000 7f27d46c7700 1 --2- 172.21.15.1:0/2564866548 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f27b006adb0 0x7f27b006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.020+0000 7f27d46c7700 1 -- 172.21.15.1:0/2564866548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27cc15d580 msgr2=0x7f27cc0b4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:01.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.020+0000 7f27d46c7700 1 --2- 172.21.15.1:0/2564866548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27cc15d580 0x7f27cc0b4db0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f27c400cb80 tx=0x7f27c400ce90 comp rx=0 tx=0).stop 2024-03-18T20:07:01.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.020+0000 7f27d46c7700 1 -- 172.21.15.1:0/2564866548 shutdown_connections 2024-03-18T20:07:01.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.020+0000 7f27d46c7700 1 --2- 172.21.15.1:0/2564866548 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f27b006adb0 0x7f27b006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.020+0000 7f27d46c7700 1 --2- 172.21.15.1:0/2564866548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27cc15d580 0x7f27cc0b4db0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.020+0000 7f27d46c7700 1 --2- 172.21.15.1:0/2564866548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27cc06efb0 0x7f27cc0c8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.020+0000 7f27d46c7700 1 --2- 172.21.15.1:0/2564866548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27cc0722e0 0x7f27cc14f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.020+0000 7f27d46c7700 1 -- 172.21.15.1:0/2564866548 >> 172.21.15.1:0/2564866548 conn(0x7f27cc0cca00 msgr2=0x7f27cc15c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:01.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.021+0000 7f27d46c7700 1 -- 172.21.15.1:0/2564866548 shutdown_connections 2024-03-18T20:07:01.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.021+0000 7f27d46c7700 1 -- 172.21.15.1:0/2564866548 wait complete. 2024-03-18T20:07:01.039 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:07:01.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.164+0000 7efe95b19700 1 Processor -- start 2024-03-18T20:07:01.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.164+0000 7efe95b19700 1 -- start start 2024-03-18T20:07:01.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.165+0000 7efe95b19700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7efe9006a9c0 0x7efe900ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:01.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.165+0000 7efe95b19700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efe9006d310 0x7efe9006f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:01.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.165+0000 7efe95b19700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9006f010 con 0x7efe900cc3b0 2024-03-18T20:07:01.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.165+0000 7efe95b19700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe9006d880 con 0x7efe9006d310 2024-03-18T20:07:01.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.165+0000 7efe95b19700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe90071ea0 con 0x7efe9006a9c0 2024-03-18T20:07:01.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.165+0000 7efe8effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efe9006d310 0x7efe9006f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:01.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.165+0000 7efe8ffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efe900cc3b0 0x7efe9015e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36090/0 (socket says 172.21.15.1:36090) 2024-03-18T20:07:01.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.165+0000 7efe8ffff700 1 -- 172.21.15.1:0/718442661 learned_addr learned my addr 172.21.15.1:0/718442661 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:01.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.165+0000 7efe8f7fe700 1 --2- 172.21.15.1:0/718442661 >> v2:172.21.15.165:3300/0 conn(0x7efe9006a9c0 0x7efe900ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:01.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.165+0000 7efe8e7fc700 1 -- 172.21.15.1:0/718442661 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3346206498 0 0) 0x7efe9006f010 con 0x7efe900cc3b0 2024-03-18T20:07:01.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.165+0000 7efe8e7fc700 1 -- 172.21.15.1:0/718442661 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe780035b0 con 0x7efe900cc3b0 2024-03-18T20:07:01.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.165+0000 7efe8effd700 1 -- 172.21.15.1:0/718442661 >> v2:172.21.15.165:3300/0 conn(0x7efe9006a9c0 msgr2=0x7efe900ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:01.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.165+0000 7efe8effd700 1 --2- 172.21.15.1:0/718442661 >> v2:172.21.15.165:3300/0 conn(0x7efe9006a9c0 0x7efe900ca5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.165+0000 7efe8effd700 1 -- 172.21.15.1:0/718442661 >> v1:172.21.15.1:6789/0 conn(0x7efe900cc3b0 legacy=0x7efe9015e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:01.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.165+0000 7efe8effd700 1 -- 172.21.15.1:0/718442661 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9006ecb0 con 0x7efe9006d310 2024-03-18T20:07:01.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.166+0000 7efe8effd700 1 --2- 172.21.15.1:0/718442661 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe9006d310 0x7efe9006f4a0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7efe900cfed0 tx=0x7efe84009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e11f65c4255531 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:01.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.166+0000 7efe8e7fc700 1 -- 172.21.15.1:0/718442661 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe84011830 con 0x7efe9006d310 2024-03-18T20:07:01.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.166+0000 7efe8e7fc700 1 -- 172.21.15.1:0/718442661 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe84011990 con 0x7efe9006d310 2024-03-18T20:07:01.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.166+0000 7efe8e7fc700 1 -- 172.21.15.1:0/718442661 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe84011d20 con 0x7efe9006d310 2024-03-18T20:07:01.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.166+0000 7efe95b19700 1 -- 172.21.15.1:0/718442661 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe9006d310 msgr2=0x7efe9006f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:01.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.166+0000 7efe95b19700 1 --2- 172.21.15.1:0/718442661 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe9006d310 0x7efe9006f4a0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7efe900cfed0 tx=0x7efe84009b20 comp rx=0 tx=0).stop 2024-03-18T20:07:01.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.166+0000 7efe95b19700 1 -- 172.21.15.1:0/718442661 shutdown_connections 2024-03-18T20:07:01.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.166+0000 7efe95b19700 1 --2- 172.21.15.1:0/718442661 >> v2:172.21.15.165:3300/0 conn(0x7efe9006a9c0 0x7efe900ca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.166+0000 7efe95b19700 1 --2- 172.21.15.1:0/718442661 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe9006d310 0x7efe9006f4a0 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.166+0000 7efe95b19700 1 -- 172.21.15.1:0/718442661 >> 172.21.15.1:0/718442661 conn(0x7efe900b1aa0 msgr2=0x7efe900b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:01.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.166+0000 7efe95b19700 1 -- 172.21.15.1:0/718442661 shutdown_connections 2024-03-18T20:07:01.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.166+0000 7efe95b19700 1 -- 172.21.15.1:0/718442661 wait complete. 2024-03-18T20:07:01.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.168+0000 7efe95b19700 1 Processor -- start 2024-03-18T20:07:01.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.168+0000 7efe95b19700 1 -- start start 2024-03-18T20:07:01.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe95b19700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe9006a9c0 0x7efe900c2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:01.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe95b19700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe9006d310 0x7efe900cf060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe95b19700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe900cc3b0 0x7efe900c9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe95b19700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe900ce5f0 con 0x7efe9006d310 2024-03-18T20:07:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe95b19700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe90157710 con 0x7efe900cc3b0 2024-03-18T20:07:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe95b19700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe901579f0 con 0x7efe9006a9c0 2024-03-18T20:07:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe8effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe9006d310 0x7efe900cf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe8effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe9006d310 0x7efe900cf060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34374/0 (socket says 172.21.15.1:34374) 2024-03-18T20:07:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe8effd700 1 -- 172.21.15.1:0/3886795295 learned_addr learned my addr 172.21.15.1:0/3886795295 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe8f7fe700 1 --2- 172.21.15.1:0/3886795295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe9006a9c0 0x7efe900c2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe8ffff700 1 --2- 172.21.15.1:0/3886795295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe900cc3b0 0x7efe900c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe8effd700 1 -- 172.21.15.1:0/3886795295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe9006a9c0 msgr2=0x7efe900c2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe8effd700 1 --2- 172.21.15.1:0/3886795295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe9006a9c0 0x7efe900c2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe8effd700 1 -- 172.21.15.1:0/3886795295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe900cc3b0 msgr2=0x7efe900c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:01.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe8effd700 1 --2- 172.21.15.1:0/3886795295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe900cc3b0 0x7efe900c9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe8effd700 1 -- 172.21.15.1:0/3886795295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe840094a0 con 0x7efe9006d310 2024-03-18T20:07:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe8effd700 1 --2- 172.21.15.1:0/3886795295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe9006d310 0x7efe900cf060 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7efe84009b60 tx=0x7efe84004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe77fff700 1 -- 172.21.15.1:0/3886795295 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe84004590 con 0x7efe9006d310 2024-03-18T20:07:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe77fff700 1 -- 172.21.15.1:0/3886795295 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8401f070 con 0x7efe9006d310 2024-03-18T20:07:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe95b19700 1 -- 172.21.15.1:0/3886795295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe900ce860 con 0x7efe9006d310 2024-03-18T20:07:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.169+0000 7efe77fff700 1 -- 172.21.15.1:0/3886795295 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe840038e0 con 0x7efe9006d310 2024-03-18T20:07:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.170+0000 7efe95b19700 1 -- 172.21.15.1:0/3886795295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe90157cd0 con 0x7efe9006d310 2024-03-18T20:07:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.170+0000 7efe95b19700 1 -- 172.21.15.1:0/3886795295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe900b0fd0 con 0x7efe9006d310 2024-03-18T20:07:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.171+0000 7efe77fff700 1 -- 172.21.15.1:0/3886795295 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 100) v1 ==== 89240+0+0 (secure 0 0 0) 0x7efe84003a40 con 0x7efe9006d310 2024-03-18T20:07:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.171+0000 7efe77fff700 1 --2- 172.21.15.1:0/3886795295 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efe7c06adb0 0x7efe7c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.172+0000 7efe8f7fe700 1 -- 172.21.15.1:0/3886795295 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efe7c06adb0 msgr2=0x7efe7c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.172+0000 7efe77fff700 1 -- 172.21.15.1:0/3886795295 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6445+0+0 (secure 0 0 0) 0x7efe84021070 con 0x7efe9006d310 2024-03-18T20:07:01.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.172+0000 7efe8f7fe700 1 --2- 172.21.15.1:0/3886795295 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efe7c06adb0 0x7efe7c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:07:01.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.180+0000 7efe77fff700 1 -- 172.21.15.1:0/3886795295 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efe84090020 con 0x7efe9006d310 2024-03-18T20:07:01.191 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:01.186+0000 7f3dc9f9a240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:07:01.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.373+0000 7efe8f7fe700 1 -- 172.21.15.1:0/3886795295 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efe7c06adb0 msgr2=0x7efe7c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:01.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.373+0000 7efe8f7fe700 1 --2- 172.21.15.1:0/3886795295 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efe7c06adb0 0x7efe7c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:07:01.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.382+0000 7efe95b19700 1 -- 172.21.15.1:0/3886795295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efe900c36a0 con 0x7efe9006d310 2024-03-18T20:07:01.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.382+0000 7efe77fff700 1 -- 172.21.15.1:0/3886795295 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v182) v1 ==== 100+0+11 (secure 0 0 0) 0x7efe84055c10 con 0x7efe9006d310 2024-03-18T20:07:01.388 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:07:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.385+0000 7efe95b19700 1 -- 172.21.15.1:0/3886795295 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efe7c06adb0 msgr2=0x7efe7c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:07:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.385+0000 7efe95b19700 1 --2- 172.21.15.1:0/3886795295 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efe7c06adb0 0x7efe7c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.385+0000 7efe95b19700 1 -- 172.21.15.1:0/3886795295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe9006d310 msgr2=0x7efe900cf060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.385+0000 7efe95b19700 1 --2- 172.21.15.1:0/3886795295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe9006d310 0x7efe900cf060 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7efe84009b60 tx=0x7efe84004750 comp rx=0 tx=0).stop 2024-03-18T20:07:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.385+0000 7efe95b19700 1 -- 172.21.15.1:0/3886795295 shutdown_connections 2024-03-18T20:07:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.385+0000 7efe95b19700 1 --2- 172.21.15.1:0/3886795295 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7efe7c06adb0 0x7efe7c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.385+0000 7efe95b19700 1 --2- 172.21.15.1:0/3886795295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe9006a9c0 0x7efe900c2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.385+0000 7efe95b19700 1 --2- 172.21.15.1:0/3886795295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe900cc3b0 0x7efe900c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.385+0000 7efe95b19700 1 --2- 172.21.15.1:0/3886795295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe9006d310 0x7efe900cf060 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.385+0000 7efe95b19700 1 -- 172.21.15.1:0/3886795295 >> 172.21.15.1:0/3886795295 conn(0x7efe900b1aa0 msgr2=0x7efe900b08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:01.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.385+0000 7efe95b19700 1 -- 172.21.15.1:0/3886795295 shutdown_connections 2024-03-18T20:07:01.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.385+0000 7efe95b19700 1 -- 172.21.15.1:0/3886795295 wait complete. 2024-03-18T20:07:01.403 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T20:07:01.417 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:01.411+0000 7f78be4a7240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:07:01.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.525+0000 7f32a80a0700 1 Processor -- start 2024-03-18T20:07:01.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.525+0000 7f32a80a0700 1 -- start start 2024-03-18T20:07:01.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.526+0000 7f32a80a0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32a015b6c0 0x7f32a00b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:01.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.526+0000 7f32a80a0700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f32a00c87d0 0x7f32a00cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:01.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.526+0000 7f32a80a0700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a00d01f0 con 0x7f32a00cd430 2024-03-18T20:07:01.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.526+0000 7f32a80a0700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a00ccfa0 con 0x7f32a015b6c0 2024-03-18T20:07:01.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.526+0000 7f32a80a0700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a00d1db0 con 0x7f32a00c87d0 2024-03-18T20:07:01.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.526+0000 7f32a5e3c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32a015b6c0 0x7f32a00b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:01.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.526+0000 7f32a5e3c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32a015b6c0 0x7f32a00b2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34384/0 (socket says 172.21.15.1:34384) 2024-03-18T20:07:01.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.526+0000 7f32a5e3c700 1 -- 172.21.15.1:0/2870518474 learned_addr learned my addr 172.21.15.1:0/2870518474 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.526+0000 7f32a663d700 1 --2- 172.21.15.1:0/2870518474 >> v2:172.21.15.165:3300/0 conn(0x7f32a00c87d0 0x7f32a00cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.526+0000 7f32a5e3c700 1 -- 172.21.15.1:0/2870518474 >> v2:172.21.15.165:3300/0 conn(0x7f32a00c87d0 msgr2=0x7f32a00cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.526+0000 7f32a5e3c700 1 --2- 172.21.15.1:0/2870518474 >> v2:172.21.15.165:3300/0 conn(0x7f32a00c87d0 0x7f32a00cd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.526+0000 7f32a5e3c700 1 -- 172.21.15.1:0/2870518474 >> v1:172.21.15.165:6789/0 conn(0x7f32a00cd430 legacy=0x7f32a00c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.526+0000 7f32a5e3c700 1 -- 172.21.15.1:0/2870518474 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a00cff10 con 0x7f32a015b6c0 2024-03-18T20:07:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.526+0000 7f32a5e3c700 1 --2- 172.21.15.1:0/2870518474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a015b6c0 0x7f32a00b2160 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f32a00d0f90 tx=0x7f329c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d080391de24e4308 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.527+0000 7f32a4e3a700 1 -- 172.21.15.1:0/2870518474 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c00b870 con 0x7f32a015b6c0 2024-03-18T20:07:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.527+0000 7f32a4e3a700 1 -- 172.21.15.1:0/2870518474 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329c00b9d0 con 0x7f32a015b6c0 2024-03-18T20:07:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.527+0000 7f32a4e3a700 1 -- 172.21.15.1:0/2870518474 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c00bd60 con 0x7f32a015b6c0 2024-03-18T20:07:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.527+0000 7f32a80a0700 1 -- 172.21.15.1:0/2870518474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a015b6c0 msgr2=0x7f32a00b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.527+0000 7f32a80a0700 1 --2- 172.21.15.1:0/2870518474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a015b6c0 0x7f32a00b2160 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f32a00d0f90 tx=0x7f329c009bd0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.527+0000 7f32a80a0700 1 -- 172.21.15.1:0/2870518474 shutdown_connections 2024-03-18T20:07:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.527+0000 7f32a80a0700 1 --2- 172.21.15.1:0/2870518474 >> v2:172.21.15.165:3300/0 conn(0x7f32a00c87d0 0x7f32a00cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.527+0000 7f32a80a0700 1 --2- 172.21.15.1:0/2870518474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a015b6c0 0x7f32a00b2160 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.527+0000 7f32a80a0700 1 -- 172.21.15.1:0/2870518474 >> 172.21.15.1:0/2870518474 conn(0x7f32a00d0ff0 msgr2=0x7f32a00d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.527+0000 7f32a80a0700 1 -- 172.21.15.1:0/2870518474 shutdown_connections 2024-03-18T20:07:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.527+0000 7f32a80a0700 1 -- 172.21.15.1:0/2870518474 wait complete. 2024-03-18T20:07:01.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.529+0000 7f32a80a0700 1 Processor -- start 2024-03-18T20:07:01.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.529+0000 7f32a80a0700 1 -- start start 2024-03-18T20:07:01.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.529+0000 7f32a80a0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32a00c87d0 0x7f32a0155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:01.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.529+0000 7f32a80a0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32a00cd430 0x7f32a0155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:01.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.529+0000 7f32a80a0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a015b6c0 0x7f32a0156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:01.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.529+0000 7f32a80a0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a009c4f0 con 0x7f32a015b6c0 2024-03-18T20:07:01.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.529+0000 7f32a80a0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a009b870 con 0x7f32a00c87d0 2024-03-18T20:07:01.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.529+0000 7f32a80a0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a009bb50 con 0x7f32a00cd430 2024-03-18T20:07:01.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.529+0000 7f32a663d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a015b6c0 0x7f32a0156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:01.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.530+0000 7f32a5e3c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32a00c87d0 0x7f32a0155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:01.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.530+0000 7f32a663d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a015b6c0 0x7f32a0156200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34398/0 (socket says 172.21.15.1:34398) 2024-03-18T20:07:01.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.530+0000 7f32a663d700 1 -- 172.21.15.1:0/1841492762 learned_addr learned my addr 172.21.15.1:0/1841492762 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:01.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.530+0000 7f32a563b700 1 --2- 172.21.15.1:0/1841492762 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32a00cd430 0x7f32a0155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:01.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.530+0000 7f32a663d700 1 -- 172.21.15.1:0/1841492762 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32a00cd430 msgr2=0x7f32a0155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:01.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.530+0000 7f32a663d700 1 --2- 172.21.15.1:0/1841492762 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32a00cd430 0x7f32a0155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.530+0000 7f32a663d700 1 -- 172.21.15.1:0/1841492762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32a00c87d0 msgr2=0x7f32a0155640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:01.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.530+0000 7f32a663d700 1 --2- 172.21.15.1:0/1841492762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32a00c87d0 0x7f32a0155640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.530+0000 7f32a663d700 1 -- 172.21.15.1:0/1841492762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329c009550 con 0x7f32a015b6c0 2024-03-18T20:07:01.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.530+0000 7f32a5e3c700 1 --2- 172.21.15.1:0/1841492762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32a00c87d0 0x7f32a0155640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:07:01.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.530+0000 7f32a663d700 1 --2- 172.21.15.1:0/1841492762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a015b6c0 0x7f32a0156200 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f3298007e40 tx=0x7f329800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:01.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.530+0000 7f32967fc700 1 -- 172.21.15.1:0/1841492762 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329800ed30 con 0x7f32a015b6c0 2024-03-18T20:07:01.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.530+0000 7f32967fc700 1 -- 172.21.15.1:0/1841492762 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3298014420 con 0x7f32a015b6c0 2024-03-18T20:07:01.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.530+0000 7f32a80a0700 1 -- 172.21.15.1:0/1841492762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a009be30 con 0x7f32a015b6c0 2024-03-18T20:07:01.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.530+0000 7f32967fc700 1 -- 172.21.15.1:0/1841492762 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32980147d0 con 0x7f32a015b6c0 2024-03-18T20:07:01.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.530+0000 7f32a80a0700 1 -- 172.21.15.1:0/1841492762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a0157970 con 0x7f32a015b6c0 2024-03-18T20:07:01.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.531+0000 7f32a80a0700 1 -- 172.21.15.1:0/1841492762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a00676a0 con 0x7f32a015b6c0 2024-03-18T20:07:01.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.532+0000 7f32967fc700 1 -- 172.21.15.1:0/1841492762 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 100) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f3298014930 con 0x7f32a015b6c0 2024-03-18T20:07:01.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.532+0000 7f32967fc700 1 --2- 172.21.15.1:0/1841492762 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f328406adb0 0x7f328406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:01.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.532+0000 7f32a5e3c700 1 -- 172.21.15.1:0/1841492762 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f328406adb0 msgr2=0x7f328406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:01.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.532+0000 7f32967fc700 1 -- 172.21.15.1:0/1841492762 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f3298089e30 con 0x7f32a015b6c0 2024-03-18T20:07:01.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.532+0000 7f32a5e3c700 1 --2- 172.21.15.1:0/1841492762 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f328406adb0 0x7f328406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:07:01.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.537+0000 7f32967fc700 1 -- 172.21.15.1:0/1841492762 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3298054c20 con 0x7f32a015b6c0 2024-03-18T20:07:01.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.733+0000 7f32a5e3c700 1 -- 172.21.15.1:0/1841492762 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f328406adb0 msgr2=0x7f328406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:01.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.733+0000 7f32a5e3c700 1 --2- 172.21.15.1:0/1841492762 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f328406adb0 0x7f328406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:07:01.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.756+0000 7f32a80a0700 1 -- 172.21.15.1:0/1841492762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f32a0073560 con 0x7f32a015b6c0 2024-03-18T20:07:01.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.756+0000 7f32967fc700 1 -- 172.21.15.1:0/1841492762 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v182) v1 ==== 80+0+450 (secure 0 0 0) 0x7f3298058240 con 0x7f32a015b6c0 2024-03-18T20:07:01.763 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6833/76998 2024-03-19T20:06:56.591909+0000 2024-03-18T20:07:01.763 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6832/76998 2024-03-19T20:06:56.591909+0000 2024-03-18T20:07:01.763 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2464679874 2024-03-19T20:06:56.591909+0000 2024-03-18T20:07:01.763 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/251298994 2024-03-19T20:06:56.591909+0000 2024-03-18T20:07:01.764 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1615725209 2024-03-19T20:06:56.591909+0000 2024-03-18T20:07:01.764 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1979178380 2024-03-19T20:06:56.591909+0000 2024-03-18T20:07:01.764 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/855901535 2024-03-19T20:06:56.591909+0000 2024-03-18T20:07:01.764 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2343543091 2024-03-19T20:06:56.591909+0000 2024-03-18T20:07:01.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.758+0000 7f32a80a0700 1 -- 172.21.15.1:0/1841492762 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f328406adb0 msgr2=0x7f328406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:07:01.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.758+0000 7f32a80a0700 1 --2- 172.21.15.1:0/1841492762 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f328406adb0 0x7f328406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.758+0000 7f32a80a0700 1 -- 172.21.15.1:0/1841492762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a015b6c0 msgr2=0x7f32a0156200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:01.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.758+0000 7f32a80a0700 1 --2- 172.21.15.1:0/1841492762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a015b6c0 0x7f32a0156200 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f3298007e40 tx=0x7f329800e3c0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.758+0000 7f32a80a0700 1 -- 172.21.15.1:0/1841492762 shutdown_connections 2024-03-18T20:07:01.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.758+0000 7f32a80a0700 1 --2- 172.21.15.1:0/1841492762 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f328406adb0 0x7f328406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.758+0000 7f32a80a0700 1 --2- 172.21.15.1:0/1841492762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32a015b6c0 0x7f32a0156200 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.758+0000 7f32a80a0700 1 --2- 172.21.15.1:0/1841492762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32a00c87d0 0x7f32a0155640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.758+0000 7f32a80a0700 1 --2- 172.21.15.1:0/1841492762 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f32a00cd430 0x7f32a0155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:01.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.759+0000 7f32a80a0700 1 -- 172.21.15.1:0/1841492762 >> 172.21.15.1:0/1841492762 conn(0x7f32a00d0ff0 msgr2=0x7f32a00ca5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:01.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.759+0000 7f32a80a0700 1 -- 172.21.15.1:0/1841492762 shutdown_connections 2024-03-18T20:07:01.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:01.759+0000 7f32a80a0700 1 -- 172.21.15.1:0/1841492762 wait complete. 2024-03-18T20:07:01.766 INFO:teuthology.orchestra.run.smithi001.stderr:listed 8 entries 2024-03-18T20:07:01.775 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:01.770+0000 7f3dc9f9a240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:07:01.777 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-18T20:07:01.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:07:01.777 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T20:07:01.846 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:01.841+0000 7f3dc9f9a240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:07:01.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[142816]: fuse finished with error 0 and tester_r 0 2024-03-18T20:07:01.854 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:07:01.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:07:01.854 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T20:07:01.973 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:01.968+0000 7f3dc9f9a240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:07:01.979 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:07:01.979 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:07:02.000 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:07:02.000 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:07:02.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:07:02.000 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-18T20:07:02.042 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:02.037+0000 7f78be4a7240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:07:02.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:07:02.058 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-18T20:07:02.058 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:07:02.058 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:07:02.076 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:07:02.077 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:07:02.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:07:02.077 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-18T20:07:02.119 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:02.113+0000 7f78be4a7240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:07:02.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:07:02.135 INFO:teuthology.orchestra.run.smithi001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-18T20:07:02.135 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:07:02.135 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:07:02.154 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:07:02.154 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:07:02.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:07:02.154 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-18T20:07:02.157 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:02.153+0000 7f3dc9f9a240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:07:02.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:07:02.174 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-18T20:07:02.175 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:07:02.232 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:02.227+0000 7f3dc9f9a240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:07:02.255 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:02.249+0000 7f78be4a7240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:07:02.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.295+0000 7f892ffa4700 1 Processor -- start 2024-03-18T20:07:02.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.295+0000 7f892ffa4700 1 -- start start 2024-03-18T20:07:02.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.296+0000 7f892ffa4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89280cf960 0x7f89280cf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:02.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.296+0000 7f892ffa4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f89280cdda0 0x7f89280ce220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:02.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.296+0000 7f892ffa4700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892806d640 con 0x7f892806a750 2024-03-18T20:07:02.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.296+0000 7f892ffa4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f892806de50 con 0x7f89280cdda0 2024-03-18T20:07:02.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.296+0000 7f892ffa4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89280acd80 con 0x7f89280cf960 2024-03-18T20:07:02.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.297+0000 7f892dd40700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f892806a750 0x7f892806ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36096/0 (socket says 172.21.15.1:36096) 2024-03-18T20:07:02.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.297+0000 7f892dd40700 1 -- 172.21.15.1:0/54451007 learned_addr learned my addr 172.21.15.1:0/54451007 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:02.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.297+0000 7f892d53f700 1 --2- 172.21.15.1:0/54451007 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89280cf960 0x7f89280cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:02.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.297+0000 7f892e541700 1 --2- 172.21.15.1:0/54451007 >> v2:172.21.15.165:3300/0 conn(0x7f89280cdda0 0x7f89280ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:02.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.297+0000 7f892cd3e700 1 -- 172.21.15.1:0/54451007 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348083582 0 0) 0x7f892806d640 con 0x7f892806a750 2024-03-18T20:07:02.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.297+0000 7f892cd3e700 1 -- 172.21.15.1:0/54451007 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89140035b0 con 0x7f892806a750 2024-03-18T20:07:02.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.297+0000 7f892cd3e700 1 -- 172.21.15.1:0/54451007 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2322321076 0 0) 0x7f89140035b0 con 0x7f892806a750 2024-03-18T20:07:02.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.297+0000 7f892d53f700 1 -- 172.21.15.1:0/54451007 >> v2:172.21.15.165:3300/0 conn(0x7f89280cdda0 msgr2=0x7f89280ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:02.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.297+0000 7f892d53f700 1 --2- 172.21.15.1:0/54451007 >> v2:172.21.15.165:3300/0 conn(0x7f89280cdda0 0x7f89280ce220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.297+0000 7f892d53f700 1 -- 172.21.15.1:0/54451007 >> v1:172.21.15.1:6789/0 conn(0x7f892806a750 legacy=0x7f892806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:02.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.297+0000 7f892d53f700 1 -- 172.21.15.1:0/54451007 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892806d890 con 0x7f89280cf960 2024-03-18T20:07:02.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.297+0000 7f892e541700 1 --2- 172.21.15.1:0/54451007 >> v2:172.21.15.165:3300/0 conn(0x7f89280cdda0 0x7f89280ce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:07:02.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.297+0000 7f892d53f700 1 --2- 172.21.15.1:0/54451007 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89280cf960 0x7f89280cf280 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8918009840 tx=0x7f8918009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb55e5cebb707a40 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:02.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.297+0000 7f892cd3e700 1 -- 172.21.15.1:0/54451007 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924001670 con 0x7f89280cf960 2024-03-18T20:07:02.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.297+0000 7f892cd3e700 1 -- 172.21.15.1:0/54451007 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8918004030 con 0x7f89280cf960 2024-03-18T20:07:02.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.298+0000 7f892cd3e700 1 -- 172.21.15.1:0/54451007 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89180043c0 con 0x7f89280cf960 2024-03-18T20:07:02.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.298+0000 7f892ffa4700 1 -- 172.21.15.1:0/54451007 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89280cf960 msgr2=0x7f89280cf280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:02.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.298+0000 7f892ffa4700 1 --2- 172.21.15.1:0/54451007 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89280cf960 0x7f89280cf280 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8918009840 tx=0x7f8918009b50 comp rx=0 tx=0).stop 2024-03-18T20:07:02.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.298+0000 7f892ffa4700 1 -- 172.21.15.1:0/54451007 shutdown_connections 2024-03-18T20:07:02.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.298+0000 7f892ffa4700 1 --2- 172.21.15.1:0/54451007 >> v2:172.21.15.165:3300/0 conn(0x7f89280cdda0 0x7f89280ce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.298+0000 7f892ffa4700 1 --2- 172.21.15.1:0/54451007 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89280cf960 0x7f89280cf280 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.298+0000 7f892ffa4700 1 -- 172.21.15.1:0/54451007 >> 172.21.15.1:0/54451007 conn(0x7f892806b860 msgr2=0x7f892806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:02.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.298+0000 7f892ffa4700 1 -- 172.21.15.1:0/54451007 shutdown_connections 2024-03-18T20:07:02.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.298+0000 7f892ffa4700 1 -- 172.21.15.1:0/54451007 wait complete. 2024-03-18T20:07:02.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.300+0000 7f892ffa4700 1 Processor -- start 2024-03-18T20:07:02.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.300+0000 7f892ffa4700 1 -- start start 2024-03-18T20:07:02.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.301+0000 7f892ffa4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f892806a750 0x7f892815b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:02.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.301+0000 7f892ffa4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89280cdda0 0x7f89280b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:02.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.301+0000 7f892ffa4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89280cf960 0x7f892814ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:02.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.301+0000 7f892ffa4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89280b4300 con 0x7f892806a750 2024-03-18T20:07:02.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.301+0000 7f892ffa4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89280b45a0 con 0x7f89280cf960 2024-03-18T20:07:02.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.301+0000 7f892ffa4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8928153e10 con 0x7f89280cdda0 2024-03-18T20:07:02.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.301+0000 7f892dd40700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f892806a750 0x7f892815b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:02.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.301+0000 7f892dd40700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f892806a750 0x7f892815b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34414/0 (socket says 172.21.15.1:34414) 2024-03-18T20:07:02.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.301+0000 7f892dd40700 1 -- 172.21.15.1:0/1858023952 learned_addr learned my addr 172.21.15.1:0/1858023952 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:02.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.301+0000 7f892e541700 1 --2- 172.21.15.1:0/1858023952 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89280cf960 0x7f892814ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:02.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.301+0000 7f892d53f700 1 --2- 172.21.15.1:0/1858023952 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89280cdda0 0x7f89280b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:02.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.301+0000 7f892dd40700 1 -- 172.21.15.1:0/1858023952 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89280cdda0 msgr2=0x7f89280b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:02.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.301+0000 7f892dd40700 1 --2- 172.21.15.1:0/1858023952 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89280cdda0 0x7f89280b3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.301+0000 7f892dd40700 1 -- 172.21.15.1:0/1858023952 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89280cf960 msgr2=0x7f892814ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:02.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.301+0000 7f892dd40700 1 --2- 172.21.15.1:0/1858023952 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89280cf960 0x7f892814ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.301+0000 7f892dd40700 1 -- 172.21.15.1:0/1858023952 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8920009a70 con 0x7f892806a750 2024-03-18T20:07:02.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.301+0000 7f892dd40700 1 --2- 172.21.15.1:0/1858023952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f892806a750 0x7f892815b640 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f89240011e0 tx=0x7f892400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:02.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.302+0000 7f891e7fc700 1 -- 172.21.15.1:0/1858023952 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924004c80 con 0x7f892806a750 2024-03-18T20:07:02.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.302+0000 7f891e7fc700 1 -- 172.21.15.1:0/1858023952 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8924004e00 con 0x7f892806a750 2024-03-18T20:07:02.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.302+0000 7f892ffa4700 1 -- 172.21.15.1:0/1858023952 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89180094a0 con 0x7f892806a750 2024-03-18T20:07:02.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.302+0000 7f891e7fc700 1 -- 172.21.15.1:0/1858023952 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924007310 con 0x7f892806a750 2024-03-18T20:07:02.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.302+0000 7f892ffa4700 1 -- 172.21.15.1:0/1858023952 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89281543d0 con 0x7f892806a750 2024-03-18T20:07:02.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.303+0000 7f892ffa4700 1 -- 172.21.15.1:0/1858023952 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89280b0000 con 0x7f892806a750 2024-03-18T20:07:02.311 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:02.305+0000 7f3dc9f9a240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:07:02.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.303+0000 7f891e7fc700 1 -- 172.21.15.1:0/1858023952 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 100) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f8924007470 con 0x7f892806a750 2024-03-18T20:07:02.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.304+0000 7f891e7fc700 1 --2- 172.21.15.1:0/1858023952 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f890c06adb0 0x7f890c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:02.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.304+0000 7f892d53f700 1 -- 172.21.15.1:0/1858023952 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f890c06adb0 msgr2=0x7f890c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:02.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.304+0000 7f891e7fc700 1 -- 172.21.15.1:0/1858023952 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f8924089430 con 0x7f892806a750 2024-03-18T20:07:02.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.304+0000 7f892d53f700 1 --2- 172.21.15.1:0/1858023952 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f890c06adb0 0x7f890c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:07:02.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.309+0000 7f891e7fc700 1 -- 172.21.15.1:0/1858023952 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8924054210 con 0x7f892806a750 2024-03-18T20:07:02.390 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:02.385+0000 7f3dc9f9a240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:07:02.454 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:02.449+0000 7f78be4a7240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:07:02.467 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:02.462+0000 7f3dc9f9a240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:07:02.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.505+0000 7f892d53f700 1 -- 172.21.15.1:0/1858023952 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f890c06adb0 msgr2=0x7f890c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:02.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.505+0000 7f892d53f700 1 --2- 172.21.15.1:0/1858023952 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f890c06adb0 0x7f890c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:07:02.530 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:02.525+0000 7f78be4a7240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:07:02.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.568+0000 7f892ffa4700 1 -- 172.21.15.1:0/1858023952 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89280cc060 con 0x7f892806a750 2024-03-18T20:07:02.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.568+0000 7f891e7fc700 1 -- 172.21.15.1:0/1858023952 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f8924057830 con 0x7f892806a750 2024-03-18T20:07:02.574 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:07:02.574 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":81,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7874,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3186058752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3186058752},{"type":"v1","addr":"172.21.15.1:6836","nonce":3186058752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":7886,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1126668185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1126668185},{"type":"v1","addr":"172.21.15.165:6835","nonce":1126668185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9028,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3922981693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3922981693},{"type":"v1","addr":"172.21.15.165:6834","nonce":3922981693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:06:41.339378+0000","modified":"2024-03-18T20:06:44.348416+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9909},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9909":{"gid":9909,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/3096868087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3096868087},{"type":"v1","addr":"172.21.15.1:6837","nonce":3096868087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-03-18T20:07:02.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.571+0000 7f892ffa4700 1 -- 172.21.15.1:0/1858023952 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f890c06adb0 msgr2=0x7f890c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:07:02.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.571+0000 7f892ffa4700 1 --2- 172.21.15.1:0/1858023952 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f890c06adb0 0x7f890c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.571+0000 7f892ffa4700 1 -- 172.21.15.1:0/1858023952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f892806a750 msgr2=0x7f892815b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:02.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.571+0000 7f892ffa4700 1 --2- 172.21.15.1:0/1858023952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f892806a750 0x7f892815b640 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f89240011e0 tx=0x7f892400c9e0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.571+0000 7f892ffa4700 1 -- 172.21.15.1:0/1858023952 shutdown_connections 2024-03-18T20:07:02.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.571+0000 7f892ffa4700 1 --2- 172.21.15.1:0/1858023952 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f890c06adb0 0x7f890c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.571+0000 7f892ffa4700 1 --2- 172.21.15.1:0/1858023952 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89280cf960 0x7f892814ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.571+0000 7f892ffa4700 1 --2- 172.21.15.1:0/1858023952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f892806a750 0x7f892815b640 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.571+0000 7f892ffa4700 1 --2- 172.21.15.1:0/1858023952 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89280cdda0 0x7f89280b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.571+0000 7f892ffa4700 1 -- 172.21.15.1:0/1858023952 >> 172.21.15.1:0/1858023952 conn(0x7f892806b860 msgr2=0x7f89280af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:02.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.571+0000 7f892ffa4700 1 -- 172.21.15.1:0/1858023952 shutdown_connections 2024-03-18T20:07:02.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.571+0000 7f892ffa4700 1 -- 172.21.15.1:0/1858023952 wait complete. 2024-03-18T20:07:02.578 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 81 2024-03-18T20:07:02.590 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:07:02.618 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:02.611+0000 7f78be4a7240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:07:02.638 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:02.633+0000 7f3dc9f9a240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:07:02.702 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:02.696+0000 7f78be4a7240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:07:02.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.710+0000 7fdaf3a52700 1 Processor -- start 2024-03-18T20:07:02.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.711+0000 7fdaf3a52700 1 -- start start 2024-03-18T20:07:02.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.711+0000 7fdaf3a52700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdaec06a750 0x7fdaec0cf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:02.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.711+0000 7fdaf3a52700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaec0cf960 0x7fdaec06ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:02.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.711+0000 7fdaf3a52700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec06d640 con 0x7fdaec0cc6d0 2024-03-18T20:07:02.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.711+0000 7fdaf3a52700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaec06de50 con 0x7fdaec06a750 2024-03-18T20:07:02.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.711+0000 7fdaf3a52700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec0acd80 con 0x7fdaec0cf960 2024-03-18T20:07:02.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.712+0000 7fdaf0fed700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaec0cf960 0x7fdaec06ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:02.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.712+0000 7fdaf17ee700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdaec06a750 0x7fdaec0cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:02.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.712+0000 7fdaf1fef700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fdaec0cc6d0 0x7fdaec0ccb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:60004/0 (socket says 172.21.15.1:60004) 2024-03-18T20:07:02.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.712+0000 7fdaf1fef700 1 -- 172.21.15.1:0/2328872132 learned_addr learned my addr 172.21.15.1:0/2328872132 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:02.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.712+0000 7fdaf17ee700 1 -- 172.21.15.1:0/2328872132 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaec0cf960 msgr2=0x7fdaec06ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:02.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.712+0000 7fdaf17ee700 1 --2- 172.21.15.1:0/2328872132 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaec0cf960 0x7fdaec06ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.712+0000 7fdaf17ee700 1 -- 172.21.15.1:0/2328872132 >> v1:172.21.15.165:6789/0 conn(0x7fdaec0cc6d0 legacy=0x7fdaec0ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:02.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.712+0000 7fdaf17ee700 1 -- 172.21.15.1:0/2328872132 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec06b070 con 0x7fdaec06a750 2024-03-18T20:07:02.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.712+0000 7fdaf0fed700 1 --2- 172.21.15.1:0/2328872132 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaec0cf960 0x7fdaec06ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:07:02.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.712+0000 7fdaf17ee700 1 --2- 172.21.15.1:0/2328872132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaec06a750 0x7fdaec0cf280 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fdae8009840 tx=0x7fdae8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82488024c28a093b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:02.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.712+0000 7fdae3fff700 1 -- 172.21.15.1:0/2328872132 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae8011830 con 0x7fdaec06a750 2024-03-18T20:07:02.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.712+0000 7fdae3fff700 1 -- 172.21.15.1:0/2328872132 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae8011990 con 0x7fdaec06a750 2024-03-18T20:07:02.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.712+0000 7fdae3fff700 1 -- 172.21.15.1:0/2328872132 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae8011d20 con 0x7fdaec06a750 2024-03-18T20:07:02.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.713+0000 7fdaf3a52700 1 -- 172.21.15.1:0/2328872132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaec06a750 msgr2=0x7fdaec0cf280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:02.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.713+0000 7fdaf3a52700 1 --2- 172.21.15.1:0/2328872132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaec06a750 0x7fdaec0cf280 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fdae8009840 tx=0x7fdae8009b50 comp rx=0 tx=0).stop 2024-03-18T20:07:02.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.713+0000 7fdaf3a52700 1 -- 172.21.15.1:0/2328872132 shutdown_connections 2024-03-18T20:07:02.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.713+0000 7fdaf3a52700 1 --2- 172.21.15.1:0/2328872132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaec06a750 0x7fdaec0cf280 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.713+0000 7fdaf3a52700 1 --2- 172.21.15.1:0/2328872132 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaec0cf960 0x7fdaec06ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.713+0000 7fdaf3a52700 1 -- 172.21.15.1:0/2328872132 >> 172.21.15.1:0/2328872132 conn(0x7fdaec06b860 msgr2=0x7fdaec06bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:02.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.713+0000 7fdaf3a52700 1 -- 172.21.15.1:0/2328872132 shutdown_connections 2024-03-18T20:07:02.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.713+0000 7fdaf3a52700 1 -- 172.21.15.1:0/2328872132 wait complete. 2024-03-18T20:07:02.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.714+0000 7fdaf3a52700 1 Processor -- start 2024-03-18T20:07:02.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.714+0000 7fdaf3a52700 1 -- start start 2024-03-18T20:07:02.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.715+0000 7fdaf3a52700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaec06a750 0x7fdaec148b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:02.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.715+0000 7fdaf3a52700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaec0cc6d0 0x7fdaec0b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:02.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.715+0000 7fdaf3a52700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaec0cf960 0x7fdaec15ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:02.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.715+0000 7fdaf3a52700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec15b970 con 0x7fdaec0cc6d0 2024-03-18T20:07:02.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.715+0000 7fdaf3a52700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec15d310 con 0x7fdaec0cf960 2024-03-18T20:07:02.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.715+0000 7fdaf3a52700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec15d5f0 con 0x7fdaec06a750 2024-03-18T20:07:02.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.716+0000 7fdaf0fed700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaec0cc6d0 0x7fdaec0b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:02.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.716+0000 7fdaf0fed700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaec0cc6d0 0x7fdaec0b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34428/0 (socket says 172.21.15.1:34428) 2024-03-18T20:07:02.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.716+0000 7fdaf0fed700 1 -- 172.21.15.1:0/728502505 learned_addr learned my addr 172.21.15.1:0/728502505 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:02.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.716+0000 7fdaf1fef700 1 --2- 172.21.15.1:0/728502505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaec0cf960 0x7fdaec15ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:02.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.716+0000 7fdaf17ee700 1 --2- 172.21.15.1:0/728502505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaec06a750 0x7fdaec148b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:02.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.716+0000 7fdaf0fed700 1 -- 172.21.15.1:0/728502505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaec06a750 msgr2=0x7fdaec148b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:02.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.716+0000 7fdaf0fed700 1 --2- 172.21.15.1:0/728502505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaec06a750 0x7fdaec148b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.716+0000 7fdaf0fed700 1 -- 172.21.15.1:0/728502505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaec0cf960 msgr2=0x7fdaec15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:02.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.716+0000 7fdaf0fed700 1 --2- 172.21.15.1:0/728502505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaec0cf960 0x7fdaec15ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.716+0000 7fdaf0fed700 1 -- 172.21.15.1:0/728502505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdadc0094a0 con 0x7fdaec0cc6d0 2024-03-18T20:07:02.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.716+0000 7fdaf0fed700 1 --2- 172.21.15.1:0/728502505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaec0cc6d0 0x7fdaec0b38b0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00ebb0 tx=0x7fdadc00eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:02.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.716+0000 7fdae1ffb700 1 -- 172.21.15.1:0/728502505 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc00c850 con 0x7fdaec0cc6d0 2024-03-18T20:07:02.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.716+0000 7fdae1ffb700 1 -- 172.21.15.1:0/728502505 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdadc00c9b0 con 0x7fdaec0cc6d0 2024-03-18T20:07:02.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.716+0000 7fdaf3a52700 1 -- 172.21.15.1:0/728502505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae80094a0 con 0x7fdaec0cc6d0 2024-03-18T20:07:02.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.716+0000 7fdae1ffb700 1 -- 172.21.15.1:0/728502505 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc00cd40 con 0x7fdaec0cc6d0 2024-03-18T20:07:02.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.717+0000 7fdaf3a52700 1 -- 172.21.15.1:0/728502505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaec15bc50 con 0x7fdaec0cc6d0 2024-03-18T20:07:02.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.717+0000 7fdaf3a52700 1 -- 172.21.15.1:0/728502505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaec0af8f0 con 0x7fdaec0cc6d0 2024-03-18T20:07:02.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.718+0000 7fdae1ffb700 1 -- 172.21.15.1:0/728502505 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 100) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fdadc013470 con 0x7fdaec0cc6d0 2024-03-18T20:07:02.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.718+0000 7fdae1ffb700 1 --2- 172.21.15.1:0/728502505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdad006adb0 0x7fdad006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:02.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.718+0000 7fdaf17ee700 1 -- 172.21.15.1:0/728502505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdad006adb0 msgr2=0x7fdad006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:02.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.718+0000 7fdae1ffb700 1 -- 172.21.15.1:0/728502505 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6445+0+0 (secure 0 0 0) 0x7fdadc089800 con 0x7fdaec0cc6d0 2024-03-18T20:07:02.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.718+0000 7fdaf17ee700 1 --2- 172.21.15.1:0/728502505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdad006adb0 0x7fdad006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:07:02.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.723+0000 7fdae1ffb700 1 -- 172.21.15.1:0/728502505 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdadc054550 con 0x7fdaec0cc6d0 2024-03-18T20:07:02.787 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:02.781+0000 7f78be4a7240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:07:02.853 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:02.848+0000 7f3dc9f9a240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:07:02.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.919+0000 7fdaf17ee700 1 -- 172.21.15.1:0/728502505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdad006adb0 msgr2=0x7fdad006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:02.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.919+0000 7fdaf17ee700 1 --2- 172.21.15.1:0/728502505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdad006adb0 0x7fdad006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:07:02.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.928+0000 7fdaf3a52700 1 -- 172.21.15.1:0/728502505 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdaec0cc060 con 0x7fdaec0cc6d0 2024-03-18T20:07:02.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.930+0000 7fdae1ffb700 1 -- 172.21.15.1:0/728502505 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v182) v1 ==== 74+0+16804 (secure 0 0 0) 0x7fdadc057b70 con 0x7fdaec0cc6d0 2024-03-18T20:07:02.936 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:07:02.936 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":182,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:06:56.591970+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:06:39.228206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-18T20:06:40.315627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6833/76998":"2024-03-19T20:06:56.591909+0000","172.21.15.1:6832/76998":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/2464679874":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/251298994":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/1615725209":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/1979178380":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/855901535":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/2343543091":"2024-03-19T20:06:56.591909+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:07:02.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.932+0000 7fdaf3a52700 1 -- 172.21.15.1:0/728502505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdad006adb0 msgr2=0x7fdad006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:07:02.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.932+0000 7fdaf3a52700 1 --2- 172.21.15.1:0/728502505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdad006adb0 0x7fdad006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.932+0000 7fdaf3a52700 1 -- 172.21.15.1:0/728502505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaec0cc6d0 msgr2=0x7fdaec0b38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:02.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.932+0000 7fdaf3a52700 1 --2- 172.21.15.1:0/728502505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaec0cc6d0 0x7fdaec0b38b0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00ebb0 tx=0x7fdadc00eec0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.932+0000 7fdaf3a52700 1 -- 172.21.15.1:0/728502505 shutdown_connections 2024-03-18T20:07:02.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.932+0000 7fdaf3a52700 1 --2- 172.21.15.1:0/728502505 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fdad006adb0 0x7fdad006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.932+0000 7fdaf3a52700 1 --2- 172.21.15.1:0/728502505 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaec06a750 0x7fdaec148b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.932+0000 7fdaf3a52700 1 --2- 172.21.15.1:0/728502505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaec0cf960 0x7fdaec15ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.932+0000 7fdaf3a52700 1 --2- 172.21.15.1:0/728502505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaec0cc6d0 0x7fdaec0b38b0 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:02.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.932+0000 7fdaf3a52700 1 -- 172.21.15.1:0/728502505 >> 172.21.15.1:0/728502505 conn(0x7fdaec06b860 msgr2=0x7fdaec0afe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:02.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.932+0000 7fdaf3a52700 1 -- 172.21.15.1:0/728502505 shutdown_connections 2024-03-18T20:07:02.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:02.932+0000 7fdaf3a52700 1 -- 172.21.15.1:0/728502505 wait complete. 2024-03-18T20:07:02.950 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:06:39.228206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '180', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:06:40.315627+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '181', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:07:02.951 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:07:02.968 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:02.962+0000 7f78be4a7240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:07:03.004 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:02.999+0000 7f3dc9f9a240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:07:03.076 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:03.071+0000 7f3dc9f9a240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:07:03.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.094+0000 7f339c30c700 1 Processor -- start 2024-03-18T20:07:03.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.094+0000 7f339c30c700 1 -- start start 2024-03-18T20:07:03.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.095+0000 7f339c30c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f339415d580 0x7f339406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:03.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.095+0000 7f339c30c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f339406d380 0x7f3394071da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:03.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.095+0000 7f339c30c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f339406f4f0 con 0x7f339406d7e0 2024-03-18T20:07:03.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.095+0000 7f339c30c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f339415d960 con 0x7f339415d580 2024-03-18T20:07:03.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.095+0000 7f339c30c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f339406ab30 con 0x7f339406d380 2024-03-18T20:07:03.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.095+0000 7f339a0a8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f339415d580 0x7f339406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:03.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.095+0000 7f339a0a8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f339415d580 0x7f339406efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34434/0 (socket says 172.21.15.1:34434) 2024-03-18T20:07:03.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.095+0000 7f339a0a8700 1 -- 172.21.15.1:0/1203344285 learned_addr learned my addr 172.21.15.1:0/1203344285 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:03.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.095+0000 7f33998a7700 1 --2- 172.21.15.1:0/1203344285 >> v2:172.21.15.165:3300/0 conn(0x7f339406d380 0x7f3394071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:03.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.095+0000 7f339a0a8700 1 -- 172.21.15.1:0/1203344285 >> v2:172.21.15.165:3300/0 conn(0x7f339406d380 msgr2=0x7f3394071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:03.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.095+0000 7f339a0a8700 1 --2- 172.21.15.1:0/1203344285 >> v2:172.21.15.165:3300/0 conn(0x7f339406d380 0x7f3394071da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.095+0000 7f339a0a8700 1 -- 172.21.15.1:0/1203344285 >> v1:172.21.15.165:6789/0 conn(0x7f339406d7e0 legacy=0x7f33940cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:03.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.095+0000 7f33990a6700 1 -- 172.21.15.1:0/1203344285 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000534107 0 0) 0x7f338c001490 con 0x7f339406d7e0 2024-03-18T20:07:03.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.096+0000 7f339a0a8700 1 -- 172.21.15.1:0/1203344285 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339406ae10 con 0x7f339415d580 2024-03-18T20:07:03.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.096+0000 7f339a0a8700 1 --2- 172.21.15.1:0/1203344285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339415d580 0x7f339406efb0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f33900098f0 tx=0x7f3390009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47d356c1e87daa5a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:03.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.096+0000 7f33990a6700 1 -- 172.21.15.1:0/1203344285 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3390011830 con 0x7f339415d580 2024-03-18T20:07:03.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.096+0000 7f33990a6700 1 -- 172.21.15.1:0/1203344285 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3390011990 con 0x7f339415d580 2024-03-18T20:07:03.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.096+0000 7f33990a6700 1 -- 172.21.15.1:0/1203344285 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3390011d20 con 0x7f339415d580 2024-03-18T20:07:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.097+0000 7f339c30c700 1 -- 172.21.15.1:0/1203344285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339415d580 msgr2=0x7f339406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.097+0000 7f339c30c700 1 --2- 172.21.15.1:0/1203344285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339415d580 0x7f339406efb0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f33900098f0 tx=0x7f3390009c00 comp rx=0 tx=0).stop 2024-03-18T20:07:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.097+0000 7f339c30c700 1 -- 172.21.15.1:0/1203344285 shutdown_connections 2024-03-18T20:07:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.097+0000 7f339c30c700 1 --2- 172.21.15.1:0/1203344285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339415d580 0x7f339406efb0 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.097+0000 7f339c30c700 1 --2- 172.21.15.1:0/1203344285 >> v2:172.21.15.165:3300/0 conn(0x7f339406d380 0x7f3394071da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.097+0000 7f339c30c700 1 -- 172.21.15.1:0/1203344285 >> 172.21.15.1:0/1203344285 conn(0x7f33940cca00 msgr2=0x7f33940cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.097+0000 7f339c30c700 1 -- 172.21.15.1:0/1203344285 shutdown_connections 2024-03-18T20:07:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.097+0000 7f339c30c700 1 -- 172.21.15.1:0/1203344285 wait complete. 2024-03-18T20:07:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.099+0000 7f339c30c700 1 Processor -- start 2024-03-18T20:07:03.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.099+0000 7f339c30c700 1 -- start start 2024-03-18T20:07:03.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.100+0000 7f339c30c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339406d380 0x7f339415be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:03.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.100+0000 7f339c30c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f339406d7e0 0x7f33941f3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:03.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.100+0000 7f339c30c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f339415d580 0x7f33940b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:03.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.100+0000 7f339c30c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339415c5f0 con 0x7f339406d380 2024-03-18T20:07:03.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.100+0000 7f339c30c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3394160080 con 0x7f339415d580 2024-03-18T20:07:03.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.100+0000 7f339c30c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33940c70f0 con 0x7f339406d7e0 2024-03-18T20:07:03.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.100+0000 7f339a0a8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339406d380 0x7f339415be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:03.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.100+0000 7f339a0a8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339406d380 0x7f339415be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34444/0 (socket says 172.21.15.1:34444) 2024-03-18T20:07:03.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.100+0000 7f339a0a8700 1 -- 172.21.15.1:0/3485202052 learned_addr learned my addr 172.21.15.1:0/3485202052 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.100+0000 7f339a8a9700 1 --2- 172.21.15.1:0/3485202052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f339415d580 0x7f33940b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.100+0000 7f33998a7700 1 --2- 172.21.15.1:0/3485202052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f339406d7e0 0x7f33941f3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.100+0000 7f339a0a8700 1 -- 172.21.15.1:0/3485202052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f339406d7e0 msgr2=0x7f33941f3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.100+0000 7f339a0a8700 1 --2- 172.21.15.1:0/3485202052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f339406d7e0 0x7f33941f3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.100+0000 7f339a0a8700 1 -- 172.21.15.1:0/3485202052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f339415d580 msgr2=0x7f33940b4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.100+0000 7f339a0a8700 1 --2- 172.21.15.1:0/3485202052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f339415d580 0x7f33940b4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.100+0000 7f339a0a8700 1 -- 172.21.15.1:0/3485202052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3390009550 con 0x7f339406d380 2024-03-18T20:07:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.100+0000 7f339a0a8700 1 --2- 172.21.15.1:0/3485202052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339406d380 0x7f339415be80 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f3394067310 tx=0x7f3390004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.101+0000 7f338affd700 1 -- 172.21.15.1:0/3485202052 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339001b840 con 0x7f339406d380 2024-03-18T20:07:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.101+0000 7f338affd700 1 -- 172.21.15.1:0/3485202052 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f339001f070 con 0x7f339406d380 2024-03-18T20:07:03.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.101+0000 7f339c30c700 1 -- 172.21.15.1:0/3485202052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33940c73d0 con 0x7f339406d380 2024-03-18T20:07:03.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.101+0000 7f338affd700 1 -- 172.21.15.1:0/3485202052 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339001bbc0 con 0x7f339406d380 2024-03-18T20:07:03.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.101+0000 7f339c30c700 1 -- 172.21.15.1:0/3485202052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3394160360 con 0x7f339406d380 2024-03-18T20:07:03.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.102+0000 7f339c30c700 1 -- 172.21.15.1:0/3485202052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f339415cd90 con 0x7f339406d380 2024-03-18T20:07:03.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.102+0000 7f338affd700 1 -- 172.21.15.1:0/3485202052 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 100) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f339001bd20 con 0x7f339406d380 2024-03-18T20:07:03.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.103+0000 7f338affd700 1 --2- 172.21.15.1:0/3485202052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f337806adb0 0x7f337806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:03.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.103+0000 7f33998a7700 1 -- 172.21.15.1:0/3485202052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f337806adb0 msgr2=0x7f337806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:03.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.103+0000 7f338affd700 1 -- 172.21.15.1:0/3485202052 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f3390021070 con 0x7f339406d380 2024-03-18T20:07:03.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.103+0000 7f33998a7700 1 --2- 172.21.15.1:0/3485202052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f337806adb0 0x7f337806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:07:03.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.111+0000 7f338affd700 1 -- 172.21.15.1:0/3485202052 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3390055e00 con 0x7f339406d380 2024-03-18T20:07:03.199 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:03.194+0000 7f78be4a7240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:07:03.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.304+0000 7f33998a7700 1 -- 172.21.15.1:0/3485202052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f337806adb0 msgr2=0x7f337806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:03.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.304+0000 7f33998a7700 1 --2- 172.21.15.1:0/3485202052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f337806adb0 0x7f337806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:07:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.316+0000 7f339c30c700 1 -- 172.21.15.1:0/3485202052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f33940ce8e0 con 0x7f339406d380 2024-03-18T20:07:03.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.317+0000 7f338affd700 1 -- 172.21.15.1:0/3485202052 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v182) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3390015020 con 0x7f339406d380 2024-03-18T20:07:03.322 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:07:03.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.319+0000 7f339c30c700 1 -- 172.21.15.1:0/3485202052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f337806adb0 msgr2=0x7f337806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:07:03.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.319+0000 7f339c30c700 1 --2- 172.21.15.1:0/3485202052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f337806adb0 0x7f337806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.319+0000 7f339c30c700 1 -- 172.21.15.1:0/3485202052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339406d380 msgr2=0x7f339415be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:03.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.319+0000 7f339c30c700 1 --2- 172.21.15.1:0/3485202052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339406d380 0x7f339415be80 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f3394067310 tx=0x7f3390004750 comp rx=0 tx=0).stop 2024-03-18T20:07:03.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.319+0000 7f339c30c700 1 -- 172.21.15.1:0/3485202052 shutdown_connections 2024-03-18T20:07:03.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.319+0000 7f339c30c700 1 --2- 172.21.15.1:0/3485202052 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f337806adb0 0x7f337806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.319+0000 7f339c30c700 1 --2- 172.21.15.1:0/3485202052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f339415d580 0x7f33940b4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.319+0000 7f339c30c700 1 --2- 172.21.15.1:0/3485202052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f339406d380 0x7f339415be80 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.319+0000 7f339c30c700 1 --2- 172.21.15.1:0/3485202052 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f339406d7e0 0x7f33941f3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.319+0000 7f339c30c700 1 -- 172.21.15.1:0/3485202052 >> 172.21.15.1:0/3485202052 conn(0x7f33940cca00 msgr2=0x7f33940b12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:03.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.319+0000 7f339c30c700 1 -- 172.21.15.1:0/3485202052 shutdown_connections 2024-03-18T20:07:03.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.319+0000 7f339c30c700 1 -- 172.21.15.1:0/3485202052 wait complete. 2024-03-18T20:07:03.338 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:07:03.363 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:03.357+0000 7f78be4a7240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:07:03.443 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:03.437+0000 7f78be4a7240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:07:03.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.461+0000 7f84c3322700 1 Processor -- start 2024-03-18T20:07:03.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.461+0000 7f84c3322700 1 -- start start 2024-03-18T20:07:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.462+0000 7f84c3322700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f84bc160a70 0x7f84bc0d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.462+0000 7f84c3322700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f84bc160e50 0x7f84bc0cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.462+0000 7f84c3322700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f84bc1612d0 0x7f84bc161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.462+0000 7f84c3322700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84bc162170 con 0x7f84bc1612d0 2024-03-18T20:07:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.462+0000 7f84c3322700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84bc161c60 con 0x7f84bc160e50 2024-03-18T20:07:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.462+0000 7f84c3322700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc0cef30 con 0x7f84bc160a70 2024-03-18T20:07:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.462+0000 7f84c18bf700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f84bc1612d0 0x7f84bc161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.462+0000 7f84c18bf700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f84bc1612d0 0x7f84bc161720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34452/0 (socket says 172.21.15.1:34452) 2024-03-18T20:07:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.462+0000 7f84c18bf700 1 -- 172.21.15.1:0/3486535649 learned_addr learned my addr 172.21.15.1:0/3486535649 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:03.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.463+0000 7f84c08bd700 1 --2- 172.21.15.1:0/3486535649 >> v2:172.21.15.165:3300/0 conn(0x7f84bc160e50 0x7f84bc0cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.463+0000 7f84c10be700 1 --2- 172.21.15.1:0/3486535649 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f84bc160a70 0x7f84bc0d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.463+0000 7f84c18bf700 1 -- 172.21.15.1:0/3486535649 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f84bc160a70 msgr2=0x7f84bc0d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.463+0000 7f84c18bf700 1 --2- 172.21.15.1:0/3486535649 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f84bc160a70 0x7f84bc0d3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.463+0000 7f84c18bf700 1 -- 172.21.15.1:0/3486535649 >> v2:172.21.15.165:3300/0 conn(0x7f84bc160e50 msgr2=0x7f84bc0cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.463+0000 7f84c18bf700 1 --2- 172.21.15.1:0/3486535649 >> v2:172.21.15.165:3300/0 conn(0x7f84bc160e50 0x7f84bc0cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.463+0000 7f84c18bf700 1 -- 172.21.15.1:0/3486535649 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc0cf210 con 0x7f84bc1612d0 2024-03-18T20:07:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.463+0000 7f84c18bf700 1 --2- 172.21.15.1:0/3486535649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84bc1612d0 0x7f84bc161720 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f84bc09cc90 tx=0x7f84b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c296a22554ac133 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.463+0000 7f84b3fff700 1 -- 172.21.15.1:0/3486535649 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b400e070 con 0x7f84bc1612d0 2024-03-18T20:07:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.463+0000 7f84b3fff700 1 -- 172.21.15.1:0/3486535649 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b400b910 con 0x7f84bc1612d0 2024-03-18T20:07:03.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.463+0000 7f84b3fff700 1 -- 172.21.15.1:0/3486535649 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b400bcc0 con 0x7f84bc1612d0 2024-03-18T20:07:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.464+0000 7f84c3322700 1 -- 172.21.15.1:0/3486535649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84bc1612d0 msgr2=0x7f84bc161720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.464+0000 7f84c3322700 1 --2- 172.21.15.1:0/3486535649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84bc1612d0 0x7f84bc161720 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f84bc09cc90 tx=0x7f84b400b330 comp rx=0 tx=0).stop 2024-03-18T20:07:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.464+0000 7f84c3322700 1 -- 172.21.15.1:0/3486535649 shutdown_connections 2024-03-18T20:07:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.464+0000 7f84c3322700 1 --2- 172.21.15.1:0/3486535649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84bc1612d0 0x7f84bc161720 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.464+0000 7f84c3322700 1 --2- 172.21.15.1:0/3486535649 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f84bc160a70 0x7f84bc0d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.464+0000 7f84c3322700 1 --2- 172.21.15.1:0/3486535649 >> v2:172.21.15.165:3300/0 conn(0x7f84bc160e50 0x7f84bc0cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.464+0000 7f84c3322700 1 -- 172.21.15.1:0/3486535649 >> 172.21.15.1:0/3486535649 conn(0x7f84bc09c560 msgr2=0x7f84bc0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.464+0000 7f84c3322700 1 -- 172.21.15.1:0/3486535649 shutdown_connections 2024-03-18T20:07:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.464+0000 7f84c3322700 1 -- 172.21.15.1:0/3486535649 wait complete. 2024-03-18T20:07:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.466+0000 7f84c3322700 1 Processor -- start 2024-03-18T20:07:03.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.466+0000 7f84c3322700 1 -- start start 2024-03-18T20:07:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.467+0000 7f84c3322700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84bc160a70 0x7f84bc159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.467+0000 7f84c3322700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f84bc160e50 0x7f84bc15d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.467+0000 7f84c3322700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84bc1612d0 0x7f84bc15d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.467+0000 7f84c3322700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc0b78b0 con 0x7f84bc160a70 2024-03-18T20:07:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.467+0000 7f84c3322700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc0b72d0 con 0x7f84bc1612d0 2024-03-18T20:07:03.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.467+0000 7f84c3322700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc15de20 con 0x7f84bc160e50 2024-03-18T20:07:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.468+0000 7f84c10be700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84bc160a70 0x7f84bc159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.468+0000 7f84c10be700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84bc160a70 0x7f84bc159da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34464/0 (socket says 172.21.15.1:34464) 2024-03-18T20:07:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.468+0000 7f84c10be700 1 -- 172.21.15.1:0/2304323925 learned_addr learned my addr 172.21.15.1:0/2304323925 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.468+0000 7f84c08bd700 1 --2- 172.21.15.1:0/2304323925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f84bc160e50 0x7f84bc15d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.468+0000 7f84c18bf700 1 --2- 172.21.15.1:0/2304323925 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84bc1612d0 0x7f84bc15d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.468+0000 7f84c10be700 1 -- 172.21.15.1:0/2304323925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f84bc160e50 msgr2=0x7f84bc15d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.468+0000 7f84c10be700 1 --2- 172.21.15.1:0/2304323925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f84bc160e50 0x7f84bc15d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.468+0000 7f84c10be700 1 -- 172.21.15.1:0/2304323925 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84bc1612d0 msgr2=0x7f84bc15d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.468+0000 7f84c10be700 1 --2- 172.21.15.1:0/2304323925 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84bc1612d0 0x7f84bc15d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.468+0000 7f84c10be700 1 -- 172.21.15.1:0/2304323925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b4009b30 con 0x7f84bc160a70 2024-03-18T20:07:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.468+0000 7f84c10be700 1 --2- 172.21.15.1:0/2304323925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84bc160a70 0x7f84bc159da0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f84b800e870 tx=0x7f84b800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.468+0000 7f84b1ffb700 1 -- 172.21.15.1:0/2304323925 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8013680 con 0x7f84bc160a70 2024-03-18T20:07:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.468+0000 7f84b1ffb700 1 -- 172.21.15.1:0/2304323925 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b80137e0 con 0x7f84bc160a70 2024-03-18T20:07:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.468+0000 7f84c3322700 1 -- 172.21.15.1:0/2304323925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84bc0b7540 con 0x7f84bc160a70 2024-03-18T20:07:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.468+0000 7f84b1ffb700 1 -- 172.21.15.1:0/2304323925 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8013b90 con 0x7f84bc160a70 2024-03-18T20:07:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.468+0000 7f84c3322700 1 -- 172.21.15.1:0/2304323925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84bc15a2e0 con 0x7f84bc160a70 2024-03-18T20:07:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.470+0000 7f84c3322700 1 -- 172.21.15.1:0/2304323925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84bc067390 con 0x7f84bc160a70 2024-03-18T20:07:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.470+0000 7f84b1ffb700 1 -- 172.21.15.1:0/2304323925 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 100) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f84b8013cf0 con 0x7f84bc160a70 2024-03-18T20:07:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.470+0000 7f84b1ffb700 1 --2- 172.21.15.1:0/2304323925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f84a006adb0 0x7f84a006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.470+0000 7f84c08bd700 1 -- 172.21.15.1:0/2304323925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f84a006adb0 msgr2=0x7f84a006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.471+0000 7f84b1ffb700 1 -- 172.21.15.1:0/2304323925 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f84b80894d0 con 0x7f84bc160a70 2024-03-18T20:07:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.471+0000 7f84c08bd700 1 --2- 172.21.15.1:0/2304323925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f84a006adb0 0x7f84a006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:07:03.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.475+0000 7f84b1ffb700 1 -- 172.21.15.1:0/2304323925 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f84b80542b0 con 0x7f84bc160a70 2024-03-18T20:07:03.522 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:03.517+0000 7f3dc9f9a240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:07:03.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.672+0000 7f84c08bd700 1 -- 172.21.15.1:0/2304323925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f84a006adb0 msgr2=0x7f84a006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:03.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.672+0000 7f84c08bd700 1 --2- 172.21.15.1:0/2304323925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f84a006adb0 0x7f84a006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:07:03.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.677+0000 7f84c3322700 1 -- 172.21.15.1:0/2304323925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f84bc0676a0 con 0x7f84bc160a70 2024-03-18T20:07:03.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.677+0000 7f84b1ffb700 1 -- 172.21.15.1:0/2304323925 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v182) v1 ==== 104+0+11 (secure 0 0 0) 0x7f84b80578d0 con 0x7f84bc160a70 2024-03-18T20:07:03.683 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:07:03.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.679+0000 7f84c3322700 1 -- 172.21.15.1:0/2304323925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f84a006adb0 msgr2=0x7f84a006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:07:03.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.679+0000 7f84c3322700 1 --2- 172.21.15.1:0/2304323925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f84a006adb0 0x7f84a006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.679+0000 7f84c3322700 1 -- 172.21.15.1:0/2304323925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84bc160a70 msgr2=0x7f84bc159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:03.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.679+0000 7f84c3322700 1 --2- 172.21.15.1:0/2304323925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84bc160a70 0x7f84bc159da0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f84b800e870 tx=0x7f84b800eb80 comp rx=0 tx=0).stop 2024-03-18T20:07:03.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.680+0000 7f84c3322700 1 -- 172.21.15.1:0/2304323925 shutdown_connections 2024-03-18T20:07:03.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.680+0000 7f84c3322700 1 --2- 172.21.15.1:0/2304323925 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f84a006adb0 0x7f84a006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.680+0000 7f84c3322700 1 --2- 172.21.15.1:0/2304323925 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84bc1612d0 0x7f84bc15d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.680+0000 7f84c3322700 1 --2- 172.21.15.1:0/2304323925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84bc160a70 0x7f84bc159da0 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.680+0000 7f84c3322700 1 --2- 172.21.15.1:0/2304323925 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f84bc160e50 0x7f84bc15d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.680+0000 7f84c3322700 1 -- 172.21.15.1:0/2304323925 >> 172.21.15.1:0/2304323925 conn(0x7f84bc09c560 msgr2=0x7f84bc15f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:03.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.680+0000 7f84c3322700 1 -- 172.21.15.1:0/2304323925 shutdown_connections 2024-03-18T20:07:03.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.680+0000 7f84c3322700 1 -- 172.21.15.1:0/2304323925 wait complete. 2024-03-18T20:07:03.698 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:07:03.858 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:03.853+0000 7f3dc9f9a240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:07:03.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.861+0000 7f59f8725700 1 Processor -- start 2024-03-18T20:07:03.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.861+0000 7f59f8725700 1 -- start start 2024-03-18T20:07:03.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.862+0000 7f59f8725700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59f0071500 0x7f59f006a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:03.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.862+0000 7f59f8725700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f015c500 con 0x7f59f006ac90 2024-03-18T20:07:03.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.862+0000 7f59f8725700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f00af8b0 con 0x7f59f0158e30 2024-03-18T20:07:03.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.862+0000 7f59f8725700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59f006b120 con 0x7f59f0071500 2024-03-18T20:07:03.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.862+0000 7f59f6cc2700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f59f006ac90 0x7f59f00cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36102/0 (socket says 172.21.15.1:36102) 2024-03-18T20:07:03.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.862+0000 7f59f6cc2700 1 -- 172.21.15.1:0/1040610084 learned_addr learned my addr 172.21.15.1:0/1040610084 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:03.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.862+0000 7f59f5cc0700 1 --2- 172.21.15.1:0/1040610084 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59f0071500 0x7f59f006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:03.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.863+0000 7f59f54bf700 1 -- 172.21.15.1:0/1040610084 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455333303 0 0) 0x7f59f015c500 con 0x7f59f006ac90 2024-03-18T20:07:03.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.863+0000 7f59f54bf700 1 -- 172.21.15.1:0/1040610084 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59dc0035b0 con 0x7f59f006ac90 2024-03-18T20:07:03.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.863+0000 7f59f54bf700 1 -- 172.21.15.1:0/1040610084 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395098666 0 0) 0x7f59f00af8b0 con 0x7f59f0158e30 2024-03-18T20:07:03.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.863+0000 7f59f54bf700 1 -- 172.21.15.1:0/1040610084 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59f015c500 con 0x7f59f0158e30 2024-03-18T20:07:03.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.863+0000 7f59f5cc0700 1 -- 172.21.15.1:0/1040610084 >> v1:172.21.15.165:6789/0 conn(0x7f59f0158e30 legacy=0x7f59f00cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:03.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.863+0000 7f59f5cc0700 1 -- 172.21.15.1:0/1040610084 >> v1:172.21.15.1:6789/0 conn(0x7f59f006ac90 legacy=0x7f59f00cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:03.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.863+0000 7f59f5cc0700 1 -- 172.21.15.1:0/1040610084 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f015b8f0 con 0x7f59f0071500 2024-03-18T20:07:03.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.863+0000 7f59f5cc0700 1 --2- 172.21.15.1:0/1040610084 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59f0071500 0x7f59f006a750 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f59e0009870 tx=0x7f59e0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8e8b4e147d71cab server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:03.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.863+0000 7f59f54bf700 1 -- 172.21.15.1:0/1040610084 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e0011830 con 0x7f59f0071500 2024-03-18T20:07:03.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.863+0000 7f59f54bf700 1 -- 172.21.15.1:0/1040610084 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e0011990 con 0x7f59f0071500 2024-03-18T20:07:03.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.863+0000 7f59f54bf700 1 -- 172.21.15.1:0/1040610084 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e0011d20 con 0x7f59f0071500 2024-03-18T20:07:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.864+0000 7f59f8725700 1 -- 172.21.15.1:0/1040610084 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59f0071500 msgr2=0x7f59f006a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.864+0000 7f59f8725700 1 --2- 172.21.15.1:0/1040610084 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59f0071500 0x7f59f006a750 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f59e0009870 tx=0x7f59e0009b80 comp rx=0 tx=0).stop 2024-03-18T20:07:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.864+0000 7f59f8725700 1 -- 172.21.15.1:0/1040610084 shutdown_connections 2024-03-18T20:07:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.864+0000 7f59f8725700 1 --2- 172.21.15.1:0/1040610084 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59f0071500 0x7f59f006a750 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.864+0000 7f59f8725700 1 -- 172.21.15.1:0/1040610084 >> 172.21.15.1:0/1040610084 conn(0x7f59f00cca00 msgr2=0x7f59f00cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.864+0000 7f59f8725700 1 -- 172.21.15.1:0/1040610084 shutdown_connections 2024-03-18T20:07:03.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.864+0000 7f59f8725700 1 -- 172.21.15.1:0/1040610084 wait complete. 2024-03-18T20:07:03.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.865+0000 7f59f8725700 1 Processor -- start 2024-03-18T20:07:03.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.865+0000 7f59f8725700 1 -- start start 2024-03-18T20:07:03.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.866+0000 7f59f8725700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59f006ac90 0x7f59f00c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:03.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.866+0000 7f59f8725700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59f0071500 0x7f59f006e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:03.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.866+0000 7f59f8725700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59f0158e30 0x7f59f0156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:03.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.866+0000 7f59f8725700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f00c41b0 con 0x7f59f0071500 2024-03-18T20:07:03.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.866+0000 7f59f8725700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f00c4490 con 0x7f59f0158e30 2024-03-18T20:07:03.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.866+0000 7f59f8725700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59f00acf00 con 0x7f59f006ac90 2024-03-18T20:07:03.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.866+0000 7f59f5cc0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59f0071500 0x7f59f006e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:03.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.866+0000 7f59f5cc0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59f0071500 0x7f59f006e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34478/0 (socket says 172.21.15.1:34478) 2024-03-18T20:07:03.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.866+0000 7f59f64c1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59f006ac90 0x7f59f00c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:03.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.866+0000 7f59f5cc0700 1 -- 172.21.15.1:0/1097328358 learned_addr learned my addr 172.21.15.1:0/1097328358 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:03.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.866+0000 7f59f6cc2700 1 --2- 172.21.15.1:0/1097328358 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59f0158e30 0x7f59f0156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:03.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.866+0000 7f59f5cc0700 1 -- 172.21.15.1:0/1097328358 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59f006ac90 msgr2=0x7f59f00c3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:03.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.866+0000 7f59f5cc0700 1 --2- 172.21.15.1:0/1097328358 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59f006ac90 0x7f59f00c3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.866+0000 7f59f5cc0700 1 -- 172.21.15.1:0/1097328358 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59f0158e30 msgr2=0x7f59f0156480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:03.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.866+0000 7f59f5cc0700 1 --2- 172.21.15.1:0/1097328358 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59f0158e30 0x7f59f0156480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:03.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.866+0000 7f59f5cc0700 1 -- 172.21.15.1:0/1097328358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e00094a0 con 0x7f59f0071500 2024-03-18T20:07:03.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.866+0000 7f59f64c1700 1 --2- 172.21.15.1:0/1097328358 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59f006ac90 0x7f59f00c3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:07:03.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.866+0000 7f59f5cc0700 1 --2- 172.21.15.1:0/1097328358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59f0071500 0x7f59f006e560 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f59e00090b0 tx=0x7f59e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:03.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.867+0000 7f59e6ffd700 1 -- 172.21.15.1:0/1097328358 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e001a6e0 con 0x7f59f0071500 2024-03-18T20:07:03.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.867+0000 7f59f8725700 1 -- 172.21.15.1:0/1097328358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f00ad1e0 con 0x7f59f0071500 2024-03-18T20:07:03.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.867+0000 7f59e6ffd700 1 -- 172.21.15.1:0/1097328358 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e001a840 con 0x7f59f0071500 2024-03-18T20:07:03.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.867+0000 7f59e6ffd700 1 -- 172.21.15.1:0/1097328358 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e001abb0 con 0x7f59f0071500 2024-03-18T20:07:03.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.867+0000 7f59f8725700 1 -- 172.21.15.1:0/1097328358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f01569c0 con 0x7f59f0071500 2024-03-18T20:07:03.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.868+0000 7f59f8725700 1 -- 172.21.15.1:0/1097328358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f0073940 con 0x7f59f0071500 2024-03-18T20:07:03.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.868+0000 7f59e6ffd700 1 -- 172.21.15.1:0/1097328358 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 100) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f59e0022070 con 0x7f59f0071500 2024-03-18T20:07:03.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.869+0000 7f59e6ffd700 1 --2- 172.21.15.1:0/1097328358 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f59d406adb0 0x7f59d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:03.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.869+0000 7f59f64c1700 1 -- 172.21.15.1:0/1097328358 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f59d406adb0 msgr2=0x7f59d406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:03.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.869+0000 7f59e6ffd700 1 -- 172.21.15.1:0/1097328358 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f59e008a280 con 0x7f59f0071500 2024-03-18T20:07:03.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.869+0000 7f59f64c1700 1 --2- 172.21.15.1:0/1097328358 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f59d406adb0 0x7f59d406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:07:03.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:03.878+0000 7f59e6ffd700 1 -- 172.21.15.1:0/1097328358 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f59e0055130 con 0x7f59f0071500 2024-03-18T20:07:03.933 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:03.927+0000 7f78be4a7240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:07:04.024 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:04.019+0000 7f3dc9f9a240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:07:04.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.070+0000 7f59f64c1700 1 -- 172.21.15.1:0/1097328358 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f59d406adb0 msgr2=0x7f59d406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:04.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.070+0000 7f59f64c1700 1 --2- 172.21.15.1:0/1097328358 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f59d406adb0 0x7f59d406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:07:04.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.079+0000 7f59f8725700 1 -- 172.21.15.1:0/1097328358 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f59f00676a0 con 0x7f59f0071500 2024-03-18T20:07:04.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.079+0000 7f59e6ffd700 1 -- 172.21.15.1:0/1097328358 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v182) v1 ==== 100+0+11 (secure 0 0 0) 0x7f59e0015020 con 0x7f59f0071500 2024-03-18T20:07:04.085 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:07:04.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.082+0000 7f59f8725700 1 -- 172.21.15.1:0/1097328358 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f59d406adb0 msgr2=0x7f59d406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:07:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.082+0000 7f59f8725700 1 --2- 172.21.15.1:0/1097328358 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f59d406adb0 0x7f59d406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.082+0000 7f59f8725700 1 -- 172.21.15.1:0/1097328358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59f0071500 msgr2=0x7f59f006e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.082+0000 7f59f8725700 1 --2- 172.21.15.1:0/1097328358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59f0071500 0x7f59f006e560 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f59e00090b0 tx=0x7f59e0004750 comp rx=0 tx=0).stop 2024-03-18T20:07:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.082+0000 7f59f8725700 1 -- 172.21.15.1:0/1097328358 shutdown_connections 2024-03-18T20:07:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.082+0000 7f59f8725700 1 --2- 172.21.15.1:0/1097328358 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f59d406adb0 0x7f59d406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.082+0000 7f59f8725700 1 --2- 172.21.15.1:0/1097328358 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59f006ac90 0x7f59f00c3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.082+0000 7f59f8725700 1 --2- 172.21.15.1:0/1097328358 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59f0158e30 0x7f59f0156480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.082+0000 7f59f8725700 1 --2- 172.21.15.1:0/1097328358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59f0071500 0x7f59f006e560 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.082+0000 7f59f8725700 1 -- 172.21.15.1:0/1097328358 >> 172.21.15.1:0/1097328358 conn(0x7f59f00cca00 msgr2=0x7f59f0156da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.082+0000 7f59f8725700 1 -- 172.21.15.1:0/1097328358 shutdown_connections 2024-03-18T20:07:04.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.082+0000 7f59f8725700 1 -- 172.21.15.1:0/1097328358 wait complete. 2024-03-18T20:07:04.100 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:07:04.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.221+0000 7f5012a72700 1 Processor -- start 2024-03-18T20:07:04.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.221+0000 7f5012a72700 1 -- start start 2024-03-18T20:07:04.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.222+0000 7f5012a72700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f500c0d17b0 0x7f500c0d1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:04.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.222+0000 7f5012a72700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500c0b83f0 0x7f500c0b8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:04.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.222+0000 7f5012a72700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c0cddc0 con 0x7f500c15d500 2024-03-18T20:07:04.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.222+0000 7f5012a72700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f500c0d21a0 con 0x7f500c0d17b0 2024-03-18T20:07:04.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.222+0000 7f5012a72700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f500c0d3dc0 con 0x7f500c0b83f0 2024-03-18T20:07:04.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.222+0000 7f500bfff700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f500c0d17b0 0x7f500c0d1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:04.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.222+0000 7f501100f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500c0b83f0 0x7f500c0b8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:04.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.222+0000 7f501080e700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f500c15d500 0x7f500c0d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:60036/0 (socket says 172.21.15.1:60036) 2024-03-18T20:07:04.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.222+0000 7f501080e700 1 -- 172.21.15.1:0/1218953843 learned_addr learned my addr 172.21.15.1:0/1218953843 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:04.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.222+0000 7f500b7fe700 1 -- 172.21.15.1:0/1218953843 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620815271 0 0) 0x7f500c0cddc0 con 0x7f500c15d500 2024-03-18T20:07:04.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.222+0000 7f500b7fe700 1 -- 172.21.15.1:0/1218953843 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ffc0035b0 con 0x7f500c15d500 2024-03-18T20:07:04.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.222+0000 7f500bfff700 1 -- 172.21.15.1:0/1218953843 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500c0b83f0 msgr2=0x7f500c0b8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:04.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.222+0000 7f500bfff700 1 --2- 172.21.15.1:0/1218953843 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500c0b83f0 0x7f500c0b8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.222+0000 7f500bfff700 1 -- 172.21.15.1:0/1218953843 >> v1:172.21.15.165:6789/0 conn(0x7f500c15d500 legacy=0x7f500c0d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:04.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.222+0000 7f500bfff700 1 -- 172.21.15.1:0/1218953843 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500c0ce060 con 0x7f500c0d17b0 2024-03-18T20:07:04.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.222+0000 7f501100f700 1 --2- 172.21.15.1:0/1218953843 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500c0b83f0 0x7f500c0b8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:07:04.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.223+0000 7f500bfff700 1 --2- 172.21.15.1:0/1218953843 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f500c0d17b0 0x7f500c0d1c60 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8009840 tx=0x7f4ff8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f05abb13e7f8539 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:04.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.223+0000 7f500b7fe700 1 -- 172.21.15.1:0/1218953843 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8011830 con 0x7f500c0d17b0 2024-03-18T20:07:04.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.223+0000 7f500b7fe700 1 -- 172.21.15.1:0/1218953843 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff8011990 con 0x7f500c0d17b0 2024-03-18T20:07:04.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.223+0000 7f500b7fe700 1 -- 172.21.15.1:0/1218953843 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8011d20 con 0x7f500c0d17b0 2024-03-18T20:07:04.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.223+0000 7f5012a72700 1 -- 172.21.15.1:0/1218953843 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f500c0d17b0 msgr2=0x7f500c0d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:04.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.223+0000 7f5012a72700 1 --2- 172.21.15.1:0/1218953843 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f500c0d17b0 0x7f500c0d1c60 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8009840 tx=0x7f4ff8009b50 comp rx=0 tx=0).stop 2024-03-18T20:07:04.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.223+0000 7f5012a72700 1 -- 172.21.15.1:0/1218953843 shutdown_connections 2024-03-18T20:07:04.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.223+0000 7f5012a72700 1 --2- 172.21.15.1:0/1218953843 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500c0b83f0 0x7f500c0b8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.223+0000 7f5012a72700 1 --2- 172.21.15.1:0/1218953843 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f500c0d17b0 0x7f500c0d1c60 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.223+0000 7f5012a72700 1 -- 172.21.15.1:0/1218953843 >> 172.21.15.1:0/1218953843 conn(0x7f500c09c530 msgr2=0x7f500c0d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:04.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.223+0000 7f5012a72700 1 -- 172.21.15.1:0/1218953843 shutdown_connections 2024-03-18T20:07:04.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.223+0000 7f5012a72700 1 -- 172.21.15.1:0/1218953843 wait complete. 2024-03-18T20:07:04.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.225+0000 7f5012a72700 1 Processor -- start 2024-03-18T20:07:04.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.225+0000 7f5012a72700 1 -- start start 2024-03-18T20:07:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.225+0000 7f5012a72700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f500c0b83f0 0x7f500c15d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.225+0000 7f5012a72700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f500c0d17b0 0x7f500c150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.225+0000 7f5012a72700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500c15d500 0x7f500c0ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.225+0000 7f5012a72700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c15ead0 con 0x7f500c0b83f0 2024-03-18T20:07:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.225+0000 7f5012a72700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c15dec0 con 0x7f500c0d17b0 2024-03-18T20:07:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.225+0000 7f5012a72700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f500c15e1a0 con 0x7f500c15d500 2024-03-18T20:07:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.226+0000 7f501080e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f500c0b83f0 0x7f500c15d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.226+0000 7f501080e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f500c0b83f0 0x7f500c15d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34484/0 (socket says 172.21.15.1:34484) 2024-03-18T20:07:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.226+0000 7f501080e700 1 -- 172.21.15.1:0/1556572874 learned_addr learned my addr 172.21.15.1:0/1556572874 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.226+0000 7f500bfff700 1 --2- 172.21.15.1:0/1556572874 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f500c0d17b0 0x7f500c150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.226+0000 7f501100f700 1 --2- 172.21.15.1:0/1556572874 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500c15d500 0x7f500c0ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.226+0000 7f501080e700 1 -- 172.21.15.1:0/1556572874 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500c15d500 msgr2=0x7f500c0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:04.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.226+0000 7f501080e700 1 --2- 172.21.15.1:0/1556572874 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500c15d500 0x7f500c0ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.226+0000 7f501080e700 1 -- 172.21.15.1:0/1556572874 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f500c0d17b0 msgr2=0x7f500c150ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:04.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.226+0000 7f501080e700 1 --2- 172.21.15.1:0/1556572874 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f500c0d17b0 0x7f500c150ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.226+0000 7f501080e700 1 -- 172.21.15.1:0/1556572874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5004009a70 con 0x7f500c0b83f0 2024-03-18T20:07:04.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.226+0000 7f500bfff700 1 --2- 172.21.15.1:0/1556572874 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f500c0d17b0 0x7f500c150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:07:04.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.226+0000 7f501080e700 1 --2- 172.21.15.1:0/1556572874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f500c0b83f0 0x7f500c15d8e0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f5000001eb0 tx=0x7f500000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:04.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.226+0000 7f50097fa700 1 -- 172.21.15.1:0/1556572874 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500000cea0 con 0x7f500c0b83f0 2024-03-18T20:07:04.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.226+0000 7f50097fa700 1 -- 172.21.15.1:0/1556572874 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5000005d70 con 0x7f500c0b83f0 2024-03-18T20:07:04.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.226+0000 7f5012a72700 1 -- 172.21.15.1:0/1556572874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff80094a0 con 0x7f500c0b83f0 2024-03-18T20:07:04.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.226+0000 7f50097fa700 1 -- 172.21.15.1:0/1556572874 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50000072f0 con 0x7f500c0b83f0 2024-03-18T20:07:04.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.226+0000 7f5012a72700 1 -- 172.21.15.1:0/1556572874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f500c15e760 con 0x7f500c0b83f0 2024-03-18T20:07:04.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.227+0000 7f5012a72700 1 -- 172.21.15.1:0/1556572874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f500c0676a0 con 0x7f500c0b83f0 2024-03-18T20:07:04.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.228+0000 7f50097fa700 1 -- 172.21.15.1:0/1556572874 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 100) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f5000007450 con 0x7f500c0b83f0 2024-03-18T20:07:04.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.228+0000 7f50097fa700 1 --2- 172.21.15.1:0/1556572874 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4ff006adb0 0x7f4ff006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:04.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.229+0000 7f500bfff700 1 -- 172.21.15.1:0/1556572874 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4ff006adb0 msgr2=0x7f4ff006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:04.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.229+0000 7f50097fa700 1 -- 172.21.15.1:0/1556572874 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f5000088c30 con 0x7f500c0b83f0 2024-03-18T20:07:04.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.229+0000 7f500bfff700 1 --2- 172.21.15.1:0/1556572874 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4ff006adb0 0x7f4ff006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:07:04.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.233+0000 7f50097fa700 1 -- 172.21.15.1:0/1556572874 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f50000539a0 con 0x7f500c0b83f0 2024-03-18T20:07:04.294 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:04.288+0000 7f78be4a7240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:07:04.414 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:04.409+0000 7f3dc9f9a240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:07:04.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.430+0000 7f500bfff700 1 -- 172.21.15.1:0/1556572874 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4ff006adb0 msgr2=0x7f4ff006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:04.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.430+0000 7f500bfff700 1 --2- 172.21.15.1:0/1556572874 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4ff006adb0 0x7f4ff006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:07:04.471 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:04.465+0000 7f78be4a7240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:07:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.480+0000 7f5012a72700 1 -- 172.21.15.1:0/1556572874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f500c0d0950 con 0x7f500c0b83f0 2024-03-18T20:07:04.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.481+0000 7f50097fa700 1 -- 172.21.15.1:0/1556572874 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f5000056fc0 con 0x7f500c0b83f0 2024-03-18T20:07:04.487 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:07:04.487 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":81,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7874,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3186058752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3186058752},{"type":"v1","addr":"172.21.15.1:6836","nonce":3186058752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":7886,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1126668185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1126668185},{"type":"v1","addr":"172.21.15.165:6835","nonce":1126668185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9028,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3922981693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3922981693},{"type":"v1","addr":"172.21.15.165:6834","nonce":3922981693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:06:41.339378+0000","modified":"2024-03-18T20:06:44.348416+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9909},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9909":{"gid":9909,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/3096868087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3096868087},{"type":"v1","addr":"172.21.15.1:6837","nonce":3096868087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-03-18T20:07:04.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.483+0000 7f5012a72700 1 -- 172.21.15.1:0/1556572874 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4ff006adb0 msgr2=0x7f4ff006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:07:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.483+0000 7f5012a72700 1 --2- 172.21.15.1:0/1556572874 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4ff006adb0 0x7f4ff006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.483+0000 7f5012a72700 1 -- 172.21.15.1:0/1556572874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f500c0b83f0 msgr2=0x7f500c15d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.483+0000 7f5012a72700 1 --2- 172.21.15.1:0/1556572874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f500c0b83f0 0x7f500c15d8e0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f5000001eb0 tx=0x7f500000c7b0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.483+0000 7f5012a72700 1 -- 172.21.15.1:0/1556572874 shutdown_connections 2024-03-18T20:07:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.483+0000 7f5012a72700 1 --2- 172.21.15.1:0/1556572874 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4ff006adb0 0x7f4ff006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.483+0000 7f5012a72700 1 --2- 172.21.15.1:0/1556572874 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500c15d500 0x7f500c0ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.483+0000 7f5012a72700 1 --2- 172.21.15.1:0/1556572874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f500c0b83f0 0x7f500c15d8e0 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.483+0000 7f5012a72700 1 --2- 172.21.15.1:0/1556572874 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f500c0d17b0 0x7f500c150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.484+0000 7f5012a72700 1 -- 172.21.15.1:0/1556572874 >> 172.21.15.1:0/1556572874 conn(0x7f500c09c530 msgr2=0x7f500c06ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.484+0000 7f5012a72700 1 -- 172.21.15.1:0/1556572874 shutdown_connections 2024-03-18T20:07:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.484+0000 7f5012a72700 1 -- 172.21.15.1:0/1556572874 wait complete. 2024-03-18T20:07:04.491 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 81 2024-03-18T20:07:04.502 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:07:04.579 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:04.574+0000 7f3dc9f9a240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:07:04.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.640+0000 7f4bafccf700 1 Processor -- start 2024-03-18T20:07:04.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.640+0000 7f4bafccf700 1 -- start start 2024-03-18T20:07:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.641+0000 7f4bafccf700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4ba80cee90 0x7f4ba80cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.641+0000 7f4bafccf700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4ba80cfbd0 0x7f4ba80b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.641+0000 7f4bafccf700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba80cd0d0 con 0x7f4ba80b6b60 2024-03-18T20:07:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.641+0000 7f4bafccf700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba80d0040 con 0x7f4ba80cee90 2024-03-18T20:07:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.641+0000 7f4bafccf700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba80b3f00 con 0x7f4ba80cfbd0 2024-03-18T20:07:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.641+0000 7f4bad26a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4ba80cee90 0x7f4ba80cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:04.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.642+0000 7f4bad26a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4ba80cee90 0x7f4ba80cf690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46208/0 (socket says 172.21.15.1:46208) 2024-03-18T20:07:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.642+0000 7f4bad26a700 1 -- 172.21.15.1:0/1272680898 learned_addr learned my addr 172.21.15.1:0/1272680898 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.642+0000 7f4bae26c700 1 --2- 172.21.15.1:0/1272680898 >> v2:172.21.15.165:3300/0 conn(0x7f4ba80cfbd0 0x7f4ba80b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.642+0000 7f4bad26a700 1 -- 172.21.15.1:0/1272680898 >> v2:172.21.15.165:3300/0 conn(0x7f4ba80cfbd0 msgr2=0x7f4ba80b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.642+0000 7f4bad26a700 1 --2- 172.21.15.1:0/1272680898 >> v2:172.21.15.165:3300/0 conn(0x7f4ba80cfbd0 0x7f4ba80b4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.642+0000 7f4bad26a700 1 -- 172.21.15.1:0/1272680898 >> v1:172.21.15.165:6789/0 conn(0x7f4ba80b6b60 legacy=0x7f4ba80cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.642+0000 7f4bad26a700 1 -- 172.21.15.1:0/1272680898 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba80b41e0 con 0x7f4ba80cee90 2024-03-18T20:07:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.642+0000 7f4bad26a700 1 --2- 172.21.15.1:0/1272680898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ba80cee90 0x7f4ba80cf690 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8071f80 tx=0x7f4b98009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73f8fdc2efd5e292 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.642+0000 7f4baca69700 1 -- 172.21.15.1:0/1272680898 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b98011830 con 0x7f4ba80cee90 2024-03-18T20:07:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.642+0000 7f4baca69700 1 -- 172.21.15.1:0/1272680898 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b98011990 con 0x7f4ba80cee90 2024-03-18T20:07:04.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.642+0000 7f4baca69700 1 -- 172.21.15.1:0/1272680898 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b98011d20 con 0x7f4ba80cee90 2024-03-18T20:07:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.643+0000 7f4bafccf700 1 -- 172.21.15.1:0/1272680898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ba80cee90 msgr2=0x7f4ba80cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.643+0000 7f4bafccf700 1 --2- 172.21.15.1:0/1272680898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ba80cee90 0x7f4ba80cf690 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8071f80 tx=0x7f4b98009bd0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.643+0000 7f4bafccf700 1 -- 172.21.15.1:0/1272680898 shutdown_connections 2024-03-18T20:07:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.643+0000 7f4bafccf700 1 --2- 172.21.15.1:0/1272680898 >> v2:172.21.15.165:3300/0 conn(0x7f4ba80cfbd0 0x7f4ba80b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.643+0000 7f4bafccf700 1 --2- 172.21.15.1:0/1272680898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ba80cee90 0x7f4ba80cf690 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.643+0000 7f4bafccf700 1 -- 172.21.15.1:0/1272680898 >> 172.21.15.1:0/1272680898 conn(0x7f4ba80b2ba0 msgr2=0x7f4ba80b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.643+0000 7f4bafccf700 1 -- 172.21.15.1:0/1272680898 shutdown_connections 2024-03-18T20:07:04.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.643+0000 7f4bafccf700 1 -- 172.21.15.1:0/1272680898 wait complete. 2024-03-18T20:07:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.646+0000 7f4bafccf700 1 Processor -- start 2024-03-18T20:07:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.646+0000 7f4bafccf700 1 -- start start 2024-03-18T20:07:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4bafccf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ba80b6b60 0x7f4ba81494d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4bafccf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ba80cee90 0x7f4ba81f4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4bafccf700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ba80cfbd0 0x7f4ba81f4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4bafccf700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba8149a10 con 0x7f4ba80cee90 2024-03-18T20:07:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4bafccf700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba80c6cd0 con 0x7f4ba80cfbd0 2024-03-18T20:07:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4bafccf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba80c6fb0 con 0x7f4ba80b6b60 2024-03-18T20:07:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4bad26a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ba80cee90 0x7f4ba81f4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4bad26a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ba80cee90 0x7f4ba81f4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46222/0 (socket says 172.21.15.1:46222) 2024-03-18T20:07:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4bad26a700 1 -- 172.21.15.1:0/2639577165 learned_addr learned my addr 172.21.15.1:0/2639577165 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4bada6b700 1 --2- 172.21.15.1:0/2639577165 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ba80b6b60 0x7f4ba81494d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:04.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4bae26c700 1 --2- 172.21.15.1:0/2639577165 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ba80cfbd0 0x7f4ba81f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4bad26a700 1 -- 172.21.15.1:0/2639577165 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ba80b6b60 msgr2=0x7f4ba81494d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4bad26a700 1 --2- 172.21.15.1:0/2639577165 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ba80b6b60 0x7f4ba81494d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4bad26a700 1 -- 172.21.15.1:0/2639577165 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ba80cfbd0 msgr2=0x7f4ba81f4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4bad26a700 1 --2- 172.21.15.1:0/2639577165 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ba80cfbd0 0x7f4ba81f4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4bad26a700 1 -- 172.21.15.1:0/2639577165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b98009550 con 0x7f4ba80cee90 2024-03-18T20:07:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4bad26a700 1 --2- 172.21.15.1:0/2639577165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ba80cee90 0x7f4ba81f4220 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f4b98000c00 tx=0x7f4b98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4b9e7fc700 1 -- 172.21.15.1:0/2639577165 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b98004380 con 0x7f4ba80cee90 2024-03-18T20:07:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4b9e7fc700 1 -- 172.21.15.1:0/2639577165 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9801f070 con 0x7f4ba80cee90 2024-03-18T20:07:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4bafccf700 1 -- 172.21.15.1:0/2639577165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba80c7290 con 0x7f4ba80cee90 2024-03-18T20:07:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4b9e7fc700 1 -- 172.21.15.1:0/2639577165 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b980036a0 con 0x7f4ba80cee90 2024-03-18T20:07:04.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.647+0000 7f4bafccf700 1 -- 172.21.15.1:0/2639577165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba80c7850 con 0x7f4ba80cee90 2024-03-18T20:07:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.648+0000 7f4bafccf700 1 -- 172.21.15.1:0/2639577165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba8149cf0 con 0x7f4ba80cee90 2024-03-18T20:07:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.648+0000 7f4b9e7fc700 1 -- 172.21.15.1:0/2639577165 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 100) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f4b98021030 con 0x7f4ba80cee90 2024-03-18T20:07:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.649+0000 7f4b9e7fc700 1 --2- 172.21.15.1:0/2639577165 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4b8c06adb0 0x7f4b8c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.649+0000 7f4bada6b700 1 -- 172.21.15.1:0/2639577165 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4b8c06adb0 msgr2=0x7f4b8c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:04.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.649+0000 7f4b9e7fc700 1 -- 172.21.15.1:0/2639577165 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f4b9808aab0 con 0x7f4ba80cee90 2024-03-18T20:07:04.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.649+0000 7f4bada6b700 1 --2- 172.21.15.1:0/2639577165 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4b8c06adb0 0x7f4b8c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:07:04.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.653+0000 7f4b9e7fc700 1 -- 172.21.15.1:0/2639577165 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4ba8149cf0 con 0x7f4ba80cee90 2024-03-18T20:07:04.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.850+0000 7f4bada6b700 1 -- 172.21.15.1:0/2639577165 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4b8c06adb0 msgr2=0x7f4b8c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:04.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.850+0000 7f4bada6b700 1 --2- 172.21.15.1:0/2639577165 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4b8c06adb0 0x7f4b8c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:07:04.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.881+0000 7f4bafccf700 1 -- 172.21.15.1:0/2639577165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ba8149cf0 con 0x7f4ba80cee90 2024-03-18T20:07:04.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.882+0000 7f4b9e7fc700 1 -- 172.21.15.1:0/2639577165 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v182) v1 ==== 74+0+16804 (secure 0 0 0) 0x7f4ba8149cf0 con 0x7f4ba80cee90 2024-03-18T20:07:04.888 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:07:04.888 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":182,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:06:56.591970+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:06:39.228206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-18T20:06:40.315627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6833/76998":"2024-03-19T20:06:56.591909+0000","172.21.15.1:6832/76998":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/2464679874":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/251298994":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/1615725209":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/1979178380":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/855901535":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/2343543091":"2024-03-19T20:06:56.591909+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:07:04.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.884+0000 7f4bafccf700 1 -- 172.21.15.1:0/2639577165 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4b8c06adb0 msgr2=0x7f4b8c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:07:04.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.884+0000 7f4bafccf700 1 --2- 172.21.15.1:0/2639577165 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4b8c06adb0 0x7f4b8c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.884+0000 7f4bafccf700 1 -- 172.21.15.1:0/2639577165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ba80cee90 msgr2=0x7f4ba81f4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:04.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.884+0000 7f4bafccf700 1 --2- 172.21.15.1:0/2639577165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ba80cee90 0x7f4ba81f4220 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f4b98000c00 tx=0x7f4b98004750 comp rx=0 tx=0).stop 2024-03-18T20:07:04.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.884+0000 7f4bafccf700 1 -- 172.21.15.1:0/2639577165 shutdown_connections 2024-03-18T20:07:04.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.884+0000 7f4bafccf700 1 --2- 172.21.15.1:0/2639577165 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4b8c06adb0 0x7f4b8c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.884+0000 7f4bafccf700 1 --2- 172.21.15.1:0/2639577165 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ba80b6b60 0x7f4ba81494d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.884+0000 7f4bafccf700 1 --2- 172.21.15.1:0/2639577165 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ba80cfbd0 0x7f4ba81f4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.884+0000 7f4bafccf700 1 --2- 172.21.15.1:0/2639577165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ba80cee90 0x7f4ba81f4220 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:04.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.884+0000 7f4bafccf700 1 -- 172.21.15.1:0/2639577165 >> 172.21.15.1:0/2639577165 conn(0x7f4ba80b2ba0 msgr2=0x7f4ba80cbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:04.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.884+0000 7f4bafccf700 1 -- 172.21.15.1:0/2639577165 shutdown_connections 2024-03-18T20:07:04.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:04.884+0000 7f4bafccf700 1 -- 172.21.15.1:0/2639577165 wait complete. 2024-03-18T20:07:04.903 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T20:07:04.903 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:07:04.932 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:04.926+0000 7f78be4a7240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:07:04.942 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:04.937+0000 7f3dc9f9a240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:07:05.029 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:05.024+0000 7f3dc9f9a240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:07:05.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.024+0000 7fbd8560e700 1 Processor -- start 2024-03-18T20:07:05.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.024+0000 7fbd8560e700 1 -- start start 2024-03-18T20:07:05.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.024+0000 7fbd8560e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbd8015b4d0 0x7fbd8009a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:05.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.024+0000 7fbd8560e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8006f070 con 0x7fbd800cf750 2024-03-18T20:07:05.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.024+0000 7fbd8560e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd800d0260 con 0x7fbd8009a150 2024-03-18T20:07:05.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.024+0000 7fbd8560e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd800cff80 con 0x7fbd8015b4d0 2024-03-18T20:07:05.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.025+0000 7fbd7effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbd8015b4d0 0x7fbd8009a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:05.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.025+0000 7fbd7effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbd8015b4d0 0x7fbd8009a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46234/0 (socket says 172.21.15.1:46234) 2024-03-18T20:07:05.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.025+0000 7fbd7effd700 1 -- 172.21.15.1:0/3503249120 learned_addr learned my addr 172.21.15.1:0/3503249120 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:05.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.025+0000 7fbd7effd700 1 -- 172.21.15.1:0/3503249120 >> v1:172.21.15.165:6789/0 conn(0x7fbd8009a150 legacy=0x7fbd8006f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:05.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.025+0000 7fbd7effd700 1 -- 172.21.15.1:0/3503249120 >> v1:172.21.15.1:6789/0 conn(0x7fbd800cf750 legacy=0x7fbd80099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:05.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.025+0000 7fbd7dffb700 1 -- 172.21.15.1:0/3503249120 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144844418 0 0) 0x7fbd800d0260 con 0x7fbd8009a150 2024-03-18T20:07:05.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.025+0000 7fbd7effd700 1 -- 172.21.15.1:0/3503249120 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd8006ea20 con 0x7fbd8015b4d0 2024-03-18T20:07:05.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.025+0000 7fbd7effd700 1 --2- 172.21.15.1:0/3503249120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8015b4d0 0x7fbd8009a650 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fbd700098f0 tx=0x7fbd70009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b479d61083c7eac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:05.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.025+0000 7fbd7dffb700 1 -- 172.21.15.1:0/3503249120 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd70004230 con 0x7fbd8015b4d0 2024-03-18T20:07:05.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.025+0000 7fbd7dffb700 1 -- 172.21.15.1:0/3503249120 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd70004390 con 0x7fbd8015b4d0 2024-03-18T20:07:05.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.025+0000 7fbd7dffb700 1 -- 172.21.15.1:0/3503249120 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd700036a0 con 0x7fbd8015b4d0 2024-03-18T20:07:05.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.026+0000 7fbd8560e700 1 -- 172.21.15.1:0/3503249120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8015b4d0 msgr2=0x7fbd8009a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:05.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.026+0000 7fbd8560e700 1 --2- 172.21.15.1:0/3503249120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8015b4d0 0x7fbd8009a650 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fbd700098f0 tx=0x7fbd70009c00 comp rx=0 tx=0).stop 2024-03-18T20:07:05.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.026+0000 7fbd8560e700 1 -- 172.21.15.1:0/3503249120 shutdown_connections 2024-03-18T20:07:05.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.026+0000 7fbd8560e700 1 --2- 172.21.15.1:0/3503249120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8015b4d0 0x7fbd8009a650 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.026+0000 7fbd8560e700 1 -- 172.21.15.1:0/3503249120 >> 172.21.15.1:0/3503249120 conn(0x7fbd800b2cb0 msgr2=0x7fbd800b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:05.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.026+0000 7fbd8560e700 1 -- 172.21.15.1:0/3503249120 shutdown_connections 2024-03-18T20:07:05.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.026+0000 7fbd8560e700 1 -- 172.21.15.1:0/3503249120 wait complete. 2024-03-18T20:07:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.028+0000 7fbd8560e700 1 Processor -- start 2024-03-18T20:07:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.028+0000 7fbd8560e700 1 -- start start 2024-03-18T20:07:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd8560e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8009a150 0x7fbd800c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd8560e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd800cf750 0x7fbd800b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd8560e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd8015b4d0 0x7fbd800b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd8560e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd800c6e60 con 0x7fbd8009a150 2024-03-18T20:07:05.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd8560e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd800c6250 con 0x7fbd8015b4d0 2024-03-18T20:07:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd8560e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd800c6530 con 0x7fbd800cf750 2024-03-18T20:07:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd7effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8009a150 0x7fbd800c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd7effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8009a150 0x7fbd800c72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46244/0 (socket says 172.21.15.1:46244) 2024-03-18T20:07:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd7effd700 1 -- 172.21.15.1:0/3774909822 learned_addr learned my addr 172.21.15.1:0/3774909822 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd7f7fe700 1 --2- 172.21.15.1:0/3774909822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd8015b4d0 0x7fbd800b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd7e7fc700 1 --2- 172.21.15.1:0/3774909822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd800cf750 0x7fbd800b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd7effd700 1 -- 172.21.15.1:0/3774909822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd800cf750 msgr2=0x7fbd800b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd7effd700 1 --2- 172.21.15.1:0/3774909822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd800cf750 0x7fbd800b3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd7effd700 1 -- 172.21.15.1:0/3774909822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd8015b4d0 msgr2=0x7fbd800b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd7effd700 1 --2- 172.21.15.1:0/3774909822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd8015b4d0 0x7fbd800b4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd7effd700 1 -- 172.21.15.1:0/3774909822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd70009550 con 0x7fbd8009a150 2024-03-18T20:07:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd7e7fc700 1 --2- 172.21.15.1:0/3774909822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd800cf750 0x7fbd800b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:07:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd7effd700 1 --2- 172.21.15.1:0/3774909822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8009a150 0x7fbd800c72f0 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fbd70009130 tx=0x7fbd70004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:05.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd677fe700 1 -- 172.21.15.1:0/3774909822 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd70011630 con 0x7fbd8009a150 2024-03-18T20:07:05.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd677fe700 1 -- 172.21.15.1:0/3774909822 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd70011790 con 0x7fbd8009a150 2024-03-18T20:07:05.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.029+0000 7fbd8560e700 1 -- 172.21.15.1:0/3774909822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd800c6810 con 0x7fbd8009a150 2024-03-18T20:07:05.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.030+0000 7fbd677fe700 1 -- 172.21.15.1:0/3774909822 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd70011b40 con 0x7fbd8009a150 2024-03-18T20:07:05.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.030+0000 7fbd8560e700 1 -- 172.21.15.1:0/3774909822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd800c7830 con 0x7fbd8009a150 2024-03-18T20:07:05.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.031+0000 7fbd8560e700 1 -- 172.21.15.1:0/3774909822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd80068240 con 0x7fbd8009a150 2024-03-18T20:07:05.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.031+0000 7fbd677fe700 1 -- 172.21.15.1:0/3774909822 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 100) v1 ==== 89240+0+0 (secure 0 0 0) 0x7fbd70021070 con 0x7fbd8009a150 2024-03-18T20:07:05.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.032+0000 7fbd677fe700 1 --2- 172.21.15.1:0/3774909822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbd6c06adb0 0x7fbd6c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:05.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.032+0000 7fbd7e7fc700 1 -- 172.21.15.1:0/3774909822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbd6c06adb0 msgr2=0x7fbd6c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:05.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.032+0000 7fbd677fe700 1 -- 172.21.15.1:0/3774909822 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6445+0+0 (secure 0 0 0) 0x7fbd7008aeb0 con 0x7fbd8009a150 2024-03-18T20:07:05.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.032+0000 7fbd7e7fc700 1 --2- 172.21.15.1:0/3774909822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbd6c06adb0 0x7fbd6c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:07:05.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.036+0000 7fbd677fe700 1 -- 172.21.15.1:0/3774909822 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbd70055d30 con 0x7fbd8009a150 2024-03-18T20:07:05.115 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:05.109+0000 7f78be4a7240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:07:05.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.233+0000 7fbd7e7fc700 1 -- 172.21.15.1:0/3774909822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbd6c06adb0 msgr2=0x7fbd6c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:05.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.233+0000 7fbd7e7fc700 1 --2- 172.21.15.1:0/3774909822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbd6c06adb0 0x7fbd6c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:07:05.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.285+0000 7fbd8560e700 1 -- 172.21.15.1:0/3774909822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd800c7a50 con 0x7fbd8009a150 2024-03-18T20:07:05.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.285+0000 7fbd677fe700 1 -- 172.21.15.1:0/3774909822 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4738 (secure 0 0 0) 0x7fbd70059350 con 0x7fbd8009a150 2024-03-18T20:07:05.291 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:07:05.291 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":81,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7874,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3186058752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3186058752},{"type":"v1","addr":"172.21.15.1:6836","nonce":3186058752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":7886,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1126668185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1126668185},{"type":"v1","addr":"172.21.15.165:6835","nonce":1126668185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9028,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3922981693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3922981693},{"type":"v1","addr":"172.21.15.165:6834","nonce":3922981693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:06:41.339378+0000","modified":"2024-03-18T20:06:44.348416+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9909},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9909":{"gid":9909,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/3096868087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3096868087},{"type":"v1","addr":"172.21.15.1:6837","nonce":3096868087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-03-18T20:07:05.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.288+0000 7fbd8560e700 1 -- 172.21.15.1:0/3774909822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbd6c06adb0 msgr2=0x7fbd6c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:07:05.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.288+0000 7fbd8560e700 1 --2- 172.21.15.1:0/3774909822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbd6c06adb0 0x7fbd6c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.288+0000 7fbd8560e700 1 -- 172.21.15.1:0/3774909822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8009a150 msgr2=0x7fbd800c72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:05.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.288+0000 7fbd8560e700 1 --2- 172.21.15.1:0/3774909822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8009a150 0x7fbd800c72f0 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fbd70009130 tx=0x7fbd70004750 comp rx=0 tx=0).stop 2024-03-18T20:07:05.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.288+0000 7fbd8560e700 1 -- 172.21.15.1:0/3774909822 shutdown_connections 2024-03-18T20:07:05.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.288+0000 7fbd8560e700 1 --2- 172.21.15.1:0/3774909822 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbd6c06adb0 0x7fbd6c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.288+0000 7fbd8560e700 1 --2- 172.21.15.1:0/3774909822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd8015b4d0 0x7fbd800b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.288+0000 7fbd8560e700 1 --2- 172.21.15.1:0/3774909822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8009a150 0x7fbd800c72f0 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.288+0000 7fbd8560e700 1 --2- 172.21.15.1:0/3774909822 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd800cf750 0x7fbd800b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.288+0000 7fbd8560e700 1 -- 172.21.15.1:0/3774909822 >> 172.21.15.1:0/3774909822 conn(0x7fbd800b2cb0 msgr2=0x7fbd800d00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:05.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.288+0000 7fbd8560e700 1 -- 172.21.15.1:0/3774909822 shutdown_connections 2024-03-18T20:07:05.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.288+0000 7fbd8560e700 1 -- 172.21.15.1:0/3774909822 wait complete. 2024-03-18T20:07:05.295 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 81 2024-03-18T20:07:05.308 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:07:05.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.473+0000 7f0f8a0f6700 1 Processor -- start 2024-03-18T20:07:05.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.473+0000 7f0f8a0f6700 1 -- start start 2024-03-18T20:07:05.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.474+0000 7f0f8a0f6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f84161360 0x7f0f8406b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.474+0000 7f0f8a0f6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0f840cdc00 0x7f0f84161740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.474+0000 7f0f8a0f6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0f84161c80 0x7f0f84071c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.474+0000 7f0f8a0f6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f840717c0 con 0x7f0f84161c80 2024-03-18T20:07:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.474+0000 7f0f8a0f6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f8415da00 con 0x7f0f840cdc00 2024-03-18T20:07:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.474+0000 7f0f8a0f6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8406b670 con 0x7f0f84161360 2024-03-18T20:07:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.474+0000 7f0f83fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0f84161c80 0x7f0f84071c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.474+0000 7f0f83fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0f84161c80 0x7f0f84071c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46258/0 (socket says 172.21.15.1:46258) 2024-03-18T20:07:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.474+0000 7f0f83fff700 1 -- 172.21.15.1:0/1094432721 learned_addr learned my addr 172.21.15.1:0/1094432721 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.475+0000 7f0f7bfff700 1 --2- 172.21.15.1:0/1094432721 >> v2:172.21.15.165:3300/0 conn(0x7f0f840cdc00 0x7f0f84161740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.475+0000 7f0f837fe700 1 --2- 172.21.15.1:0/1094432721 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f84161360 0x7f0f8406b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.475+0000 7f0f83fff700 1 -- 172.21.15.1:0/1094432721 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f84161360 msgr2=0x7f0f8406b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.475+0000 7f0f83fff700 1 --2- 172.21.15.1:0/1094432721 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f84161360 0x7f0f8406b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.475+0000 7f0f83fff700 1 -- 172.21.15.1:0/1094432721 >> v2:172.21.15.165:3300/0 conn(0x7f0f840cdc00 msgr2=0x7f0f84161740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.475+0000 7f0f83fff700 1 --2- 172.21.15.1:0/1094432721 >> v2:172.21.15.165:3300/0 conn(0x7f0f840cdc00 0x7f0f84161740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.475+0000 7f0f83fff700 1 -- 172.21.15.1:0/1094432721 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f840cc8b0 con 0x7f0f84161c80 2024-03-18T20:07:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.475+0000 7f0f83fff700 1 --2- 172.21.15.1:0/1094432721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f84161c80 0x7f0f84071c50 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f0f84067530 tx=0x7f0f7c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49011bb22b7f68ef server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.475+0000 7f0f82ffd700 1 -- 172.21.15.1:0/1094432721 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c00e070 con 0x7f0f84161c80 2024-03-18T20:07:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.475+0000 7f0f82ffd700 1 -- 172.21.15.1:0/1094432721 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7c00b9c0 con 0x7f0f84161c80 2024-03-18T20:07:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.475+0000 7f0f82ffd700 1 -- 172.21.15.1:0/1094432721 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c00bd70 con 0x7f0f84161c80 2024-03-18T20:07:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.476+0000 7f0f8a0f6700 1 -- 172.21.15.1:0/1094432721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f84161c80 msgr2=0x7f0f84071c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.476+0000 7f0f8a0f6700 1 --2- 172.21.15.1:0/1094432721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f84161c80 0x7f0f84071c50 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f0f84067530 tx=0x7f0f7c00b330 comp rx=0 tx=0).stop 2024-03-18T20:07:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.476+0000 7f0f8a0f6700 1 -- 172.21.15.1:0/1094432721 shutdown_connections 2024-03-18T20:07:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.476+0000 7f0f8a0f6700 1 --2- 172.21.15.1:0/1094432721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f84161c80 0x7f0f84071c50 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.476+0000 7f0f8a0f6700 1 --2- 172.21.15.1:0/1094432721 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f84161360 0x7f0f8406b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.476+0000 7f0f8a0f6700 1 --2- 172.21.15.1:0/1094432721 >> v2:172.21.15.165:3300/0 conn(0x7f0f840cdc00 0x7f0f84161740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.476+0000 7f0f8a0f6700 1 -- 172.21.15.1:0/1094432721 >> 172.21.15.1:0/1094432721 conn(0x7f0f840ccab0 msgr2=0x7f0f840ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:05.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.476+0000 7f0f8a0f6700 1 -- 172.21.15.1:0/1094432721 shutdown_connections 2024-03-18T20:07:05.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.476+0000 7f0f8a0f6700 1 -- 172.21.15.1:0/1094432721 wait complete. 2024-03-18T20:07:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.477+0000 7f0f8a0f6700 1 Processor -- start 2024-03-18T20:07:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.478+0000 7f0f8a0f6700 1 -- start start 2024-03-18T20:07:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.478+0000 7f0f8a0f6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f840cdc00 0x7f0f8415be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.478+0000 7f0f8a0f6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f84161360 0x7f0f8409faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f8a0f6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f84161c80 0x7f0f840a0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f8a0f6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f840c42c0 con 0x7f0f84161360 2024-03-18T20:07:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f8a0f6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f840b48a0 con 0x7f0f840cdc00 2024-03-18T20:07:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f8a0f6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f840b3d20 con 0x7f0f84161c80 2024-03-18T20:07:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f7bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f84161360 0x7f0f8409faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:05.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f7bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f84161360 0x7f0f8409faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46268/0 (socket says 172.21.15.1:46268) 2024-03-18T20:07:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f7bfff700 1 -- 172.21.15.1:0/2182673585 learned_addr learned my addr 172.21.15.1:0/2182673585 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f837fe700 1 --2- 172.21.15.1:0/2182673585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f840cdc00 0x7f0f8415be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f83fff700 1 --2- 172.21.15.1:0/2182673585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f84161c80 0x7f0f840a0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f7bfff700 1 -- 172.21.15.1:0/2182673585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f84161c80 msgr2=0x7f0f840a0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f7bfff700 1 --2- 172.21.15.1:0/2182673585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f84161c80 0x7f0f840a0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f7bfff700 1 -- 172.21.15.1:0/2182673585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f840cdc00 msgr2=0x7f0f8415be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f7bfff700 1 --2- 172.21.15.1:0/2182673585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f840cdc00 0x7f0f8415be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f7bfff700 1 -- 172.21.15.1:0/2182673585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f7c009b30 con 0x7f0f84161360 2024-03-18T20:07:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f837fe700 1 --2- 172.21.15.1:0/2182673585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f840cdc00 0x7f0f8415be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:07:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f83fff700 1 --2- 172.21.15.1:0/2182673585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f84161c80 0x7f0f840a0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:07:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f7bfff700 1 --2- 172.21.15.1:0/2182673585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f84161360 0x7f0f8409faf0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c00e810 tx=0x7f0f6c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f80ff9700 1 -- 172.21.15.1:0/2182673585 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c013610 con 0x7f0f84161360 2024-03-18T20:07:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f80ff9700 1 -- 172.21.15.1:0/2182673585 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6c013770 con 0x7f0f84161360 2024-03-18T20:07:05.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f80ff9700 1 -- 172.21.15.1:0/2182673585 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c013b00 con 0x7f0f84161360 2024-03-18T20:07:05.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f8a0f6700 1 -- 172.21.15.1:0/2182673585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f840b4000 con 0x7f0f84161360 2024-03-18T20:07:05.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.479+0000 7f0f8a0f6700 1 -- 172.21.15.1:0/2182673585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f840b45c0 con 0x7f0f84161360 2024-03-18T20:07:05.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.481+0000 7f0f80ff9700 1 -- 172.21.15.1:0/2182673585 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 100) v1 ==== 89240+0+0 (secure 0 0 0) 0x7f0f6c013c60 con 0x7f0f84161360 2024-03-18T20:07:05.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.481+0000 7f0f8a0f6700 1 -- 172.21.15.1:0/2182673585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f840c4430 con 0x7f0f84161360 2024-03-18T20:07:05.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.481+0000 7f0f80ff9700 1 --2- 172.21.15.1:0/2182673585 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0f6406adb0 0x7f0f6406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:05.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.481+0000 7f0f837fe700 1 -- 172.21.15.1:0/2182673585 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0f6406adb0 msgr2=0x7f0f6406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:05.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.481+0000 7f0f80ff9700 1 -- 172.21.15.1:0/2182673585 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f0f6c089580 con 0x7f0f84161360 2024-03-18T20:07:05.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.481+0000 7f0f837fe700 1 --2- 172.21.15.1:0/2182673585 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0f6406adb0 0x7f0f6406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:07:05.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.486+0000 7f0f80ff9700 1 -- 172.21.15.1:0/2182673585 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0f6c0541d0 con 0x7f0f84161360 2024-03-18T20:07:05.505 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:05.501+0000 7f78be4a7240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:07:05.602 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:05.598+0000 7f78be4a7240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:07:05.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.620+0000 7f0f80ff9700 1 -- 172.21.15.1:0/2182673585 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 101) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f0f6c0877c0 con 0x7f0f84161360 2024-03-18T20:07:05.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.620+0000 7f0f80ff9700 1 -- 172.21.15.1:0/2182673585 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0f6406adb0 msgr2=0x7f0f6406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:07:05.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.620+0000 7f0f80ff9700 1 --2- 172.21.15.1:0/2182673585 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0f6406adb0 0x7f0f6406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.686 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:05.681+0000 7f78368cf700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:05.686 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:05.681+0000 7f78368cf700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:05.686 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:05.681+0000 7f78368cf700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:05.686 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:05.681+0000 7f78368cf700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:05.686 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:05.681+0000 7f78368cf700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:05.686 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:05.681+0000 7f78398d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:05.687 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:05.681+0000 7f78398d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:05.687 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:05.681+0000 7f78398d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:05.687 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:05.681+0000 7f78398d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:05.687 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:05.682+0000 7f78398d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:05.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.751+0000 7f0f8a0f6700 1 -- 172.21.15.1:0/2182673585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f840676a0 con 0x7f0f84161360 2024-03-18T20:07:05.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.752+0000 7f0f80ff9700 1 -- 172.21.15.1:0/2182673585 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f0f6c084d00 con 0x7f0f84161360 2024-03-18T20:07:05.757 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:07:05.757 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":81,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7874,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3186058752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3186058752},{"type":"v1","addr":"172.21.15.1:6836","nonce":3186058752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":7886,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1126668185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1126668185},{"type":"v1","addr":"172.21.15.165:6835","nonce":1126668185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9028,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3922981693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3922981693},{"type":"v1","addr":"172.21.15.165:6834","nonce":3922981693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:06:41.339378+0000","modified":"2024-03-18T20:06:44.348416+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9909},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9909":{"gid":9909,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/3096868087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":3096868087},{"type":"v1","addr":"172.21.15.1:6837","nonce":3096868087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-03-18T20:07:05.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.754+0000 7f0f8a0f6700 1 -- 172.21.15.1:0/2182673585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f84161360 msgr2=0x7f0f8409faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:05.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.754+0000 7f0f8a0f6700 1 --2- 172.21.15.1:0/2182673585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f84161360 0x7f0f8409faf0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c00e810 tx=0x7f0f6c00ebd0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.754+0000 7f0f8a0f6700 1 -- 172.21.15.1:0/2182673585 shutdown_connections 2024-03-18T20:07:05.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.754+0000 7f0f8a0f6700 1 --2- 172.21.15.1:0/2182673585 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0f6406adb0 0x7f0f6406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.754+0000 7f0f8a0f6700 1 --2- 172.21.15.1:0/2182673585 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f84161c80 0x7f0f840a0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.754+0000 7f0f8a0f6700 1 --2- 172.21.15.1:0/2182673585 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f840cdc00 0x7f0f8415be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.754+0000 7f0f8a0f6700 1 --2- 172.21.15.1:0/2182673585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f84161360 0x7f0f8409faf0 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.754+0000 7f0f8a0f6700 1 -- 172.21.15.1:0/2182673585 >> 172.21.15.1:0/2182673585 conn(0x7f0f840ccab0 msgr2=0x7f0f840ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:05.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.754+0000 7f0f8a0f6700 1 -- 172.21.15.1:0/2182673585 shutdown_connections 2024-03-18T20:07:05.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.754+0000 7f0f8a0f6700 1 -- 172.21.15.1:0/2182673585 wait complete. 2024-03-18T20:07:05.760 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 81 2024-03-18T20:07:05.771 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:07:05.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.892+0000 7f27aff82700 1 Processor -- start 2024-03-18T20:07:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.892+0000 7f27aff82700 1 -- start start 2024-03-18T20:07:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.893+0000 7f27aff82700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27a815b4d0 0x7f27a809a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.893+0000 7f27aff82700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27a809a150 0x7f27a806f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.893+0000 7f27aff82700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a806f070 con 0x7f27a80cf750 2024-03-18T20:07:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.893+0000 7f27aff82700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a80d0260 con 0x7f27a809a150 2024-03-18T20:07:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.893+0000 7f27aff82700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a80cff80 con 0x7f27a815b4d0 2024-03-18T20:07:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.893+0000 7f27ae51f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27a809a150 0x7f27a806f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.893+0000 7f27ae51f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27a809a150 0x7f27a806f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46390/0 (socket says 172.21.15.1:46390) 2024-03-18T20:07:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.893+0000 7f27ae51f700 1 -- 172.21.15.1:0/2268084893 learned_addr learned my addr 172.21.15.1:0/2268084893 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.894+0000 7f27add1e700 1 --2- 172.21.15.1:0/2268084893 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27a815b4d0 0x7f27a809a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.894+0000 7f27ae51f700 1 -- 172.21.15.1:0/2268084893 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27a815b4d0 msgr2=0x7f27a809a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.894+0000 7f27ae51f700 1 --2- 172.21.15.1:0/2268084893 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27a815b4d0 0x7f27a809a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.894+0000 7f27ae51f700 1 -- 172.21.15.1:0/2268084893 >> v1:172.21.15.165:6789/0 conn(0x7f27a80cf750 legacy=0x7f27a8099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.894+0000 7f27ae51f700 1 -- 172.21.15.1:0/2268084893 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a806ea20 con 0x7f27a809a150 2024-03-18T20:07:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.894+0000 7f27ae51f700 1 --2- 172.21.15.1:0/2268084893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27a809a150 0x7f27a806f500 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f27a0009e50 tx=0x7f27a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e7e05d06d72d80 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.894+0000 7f27acd1c700 1 -- 172.21.15.1:0/2268084893 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a000e070 con 0x7f27a809a150 2024-03-18T20:07:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.894+0000 7f27acd1c700 1 -- 172.21.15.1:0/2268084893 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a0003a20 con 0x7f27a809a150 2024-03-18T20:07:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.894+0000 7f27acd1c700 1 -- 172.21.15.1:0/2268084893 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a0003dd0 con 0x7f27a809a150 2024-03-18T20:07:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.895+0000 7f27aff82700 1 -- 172.21.15.1:0/2268084893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27a809a150 msgr2=0x7f27a806f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.895+0000 7f27aff82700 1 --2- 172.21.15.1:0/2268084893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27a809a150 0x7f27a806f500 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f27a0009e50 tx=0x7f27a000b330 comp rx=0 tx=0).stop 2024-03-18T20:07:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.895+0000 7f27aff82700 1 -- 172.21.15.1:0/2268084893 shutdown_connections 2024-03-18T20:07:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.895+0000 7f27aff82700 1 --2- 172.21.15.1:0/2268084893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27a809a150 0x7f27a806f500 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.895+0000 7f27aff82700 1 --2- 172.21.15.1:0/2268084893 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27a815b4d0 0x7f27a809a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.895+0000 7f27aff82700 1 -- 172.21.15.1:0/2268084893 >> 172.21.15.1:0/2268084893 conn(0x7f27a80b2cb0 msgr2=0x7f27a80b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.895+0000 7f27aff82700 1 -- 172.21.15.1:0/2268084893 shutdown_connections 2024-03-18T20:07:05.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.895+0000 7f27aff82700 1 -- 172.21.15.1:0/2268084893 wait complete. 2024-03-18T20:07:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.896+0000 7f27aff82700 1 Processor -- start 2024-03-18T20:07:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.896+0000 7f27aff82700 1 -- start start 2024-03-18T20:07:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.897+0000 7f27aff82700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27a809a150 0x7f27a8147f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.897+0000 7f27aff82700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27a80cf750 0x7f27a80b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.897+0000 7f27aff82700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27a815b4d0 0x7f27a81f05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.897+0000 7f27aff82700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a80b6290 con 0x7f27a815b4d0 2024-03-18T20:07:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.897+0000 7f27aff82700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a81f0b00 con 0x7f27a809a150 2024-03-18T20:07:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.897+0000 7f27aff82700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a80b6560 con 0x7f27a80cf750 2024-03-18T20:07:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.897+0000 7f27ae51f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27a815b4d0 0x7f27a81f05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.897+0000 7f27ae51f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27a815b4d0 0x7f27a81f05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46392/0 (socket says 172.21.15.1:46392) 2024-03-18T20:07:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.897+0000 7f27ae51f700 1 -- 172.21.15.1:0/1781856992 learned_addr learned my addr 172.21.15.1:0/1781856992 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:05.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.898+0000 7f27ad51d700 1 --2- 172.21.15.1:0/1781856992 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27a80cf750 0x7f27a80b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:05.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.898+0000 7f27add1e700 1 --2- 172.21.15.1:0/1781856992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27a809a150 0x7f27a8147f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:05.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.898+0000 7f27ae51f700 1 -- 172.21.15.1:0/1781856992 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27a80cf750 msgr2=0x7f27a80b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:05.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.898+0000 7f27ae51f700 1 --2- 172.21.15.1:0/1781856992 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27a80cf750 0x7f27a80b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.898+0000 7f27ae51f700 1 -- 172.21.15.1:0/1781856992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27a809a150 msgr2=0x7f27a8147f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:05.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.898+0000 7f27ae51f700 1 --2- 172.21.15.1:0/1781856992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27a809a150 0x7f27a8147f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:05.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.898+0000 7f27ae51f700 1 -- 172.21.15.1:0/1781856992 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a0009b30 con 0x7f27a815b4d0 2024-03-18T20:07:05.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.898+0000 7f27add1e700 1 --2- 172.21.15.1:0/1781856992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27a809a150 0x7f27a8147f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:05.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.898+0000 7f27ad51d700 1 --2- 172.21.15.1:0/1781856992 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27a80cf750 0x7f27a80b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:05.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.898+0000 7f27ae51f700 1 --2- 172.21.15.1:0/1781856992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27a815b4d0 0x7f27a81f05c0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f27a0004000 tx=0x7f27a0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:05.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.898+0000 7f279e7fc700 1 -- 172.21.15.1:0/1781856992 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a0004240 con 0x7f27a815b4d0 2024-03-18T20:07:05.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.898+0000 7f279e7fc700 1 -- 172.21.15.1:0/1781856992 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a00043a0 con 0x7f27a815b4d0 2024-03-18T20:07:05.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.898+0000 7f27aff82700 1 -- 172.21.15.1:0/1781856992 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a80cfd50 con 0x7f27a815b4d0 2024-03-18T20:07:05.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.898+0000 7f279e7fc700 1 -- 172.21.15.1:0/1781856992 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a0004750 con 0x7f27a815b4d0 2024-03-18T20:07:05.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.898+0000 7f27aff82700 1 -- 172.21.15.1:0/1781856992 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a80b6b20 con 0x7f27a815b4d0 2024-03-18T20:07:05.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.899+0000 7f27aff82700 1 -- 172.21.15.1:0/1781856992 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a8068240 con 0x7f27a815b4d0 2024-03-18T20:07:05.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.899+0000 7f279e7fc700 1 -- 172.21.15.1:0/1781856992 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 101) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f27a0019070 con 0x7f27a815b4d0 2024-03-18T20:07:05.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.900+0000 7f279e7fc700 1 -- 172.21.15.1:0/1781856992 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6703+0+0 (secure 0 0 0) 0x7f27a008a240 con 0x7f27a815b4d0 2024-03-18T20:07:05.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:05.904+0000 7f279e7fc700 1 -- 172.21.15.1:0/1781856992 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f27a0055410 con 0x7f27a815b4d0 2024-03-18T20:07:06.093 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:07:06.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:07:06.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.105+0000 7f27aff82700 1 -- 172.21.15.1:0/1781856992 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27a80cfbb0 con 0x7f27a815b4d0 2024-03-18T20:07:06.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.107+0000 7f279e7fc700 1 -- 172.21.15.1:0/1781856992 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v183) v1 ==== 74+0+17168 (secure 0 0 0) 0x7f27a80cfbb0 con 0x7f27a815b4d0 2024-03-18T20:07:06.112 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:07:06.112 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":183,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:07:05.611622+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:06:39.228206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-18T20:06:40.315627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/4118629845":"2024-03-19T20:07:05.611568+0000","172.21.15.1:0/1512815787":"2024-03-19T20:07:05.611568+0000","172.21.15.1:0/2894574471":"2024-03-19T20:07:05.611568+0000","172.21.15.1:0/933460777":"2024-03-19T20:07:05.611568+0000","172.21.15.1:0/919901722":"2024-03-19T20:07:05.611568+0000","172.21.15.1:6833/76998":"2024-03-19T20:06:56.591909+0000","172.21.15.1:6832/76998":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/2464679874":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/3491089776":"2024-03-19T20:07:05.611568+0000","172.21.15.1:0/251298994":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/1615725209":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/1979178380":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/855901535":"2024-03-19T20:06:56.591909+0000","172.21.15.1:0/2343543091":"2024-03-19T20:06:56.591909+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:07:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.109+0000 7f27aff82700 1 -- 172.21.15.1:0/1781856992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27a815b4d0 msgr2=0x7f27a81f05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.109+0000 7f27aff82700 1 --2- 172.21.15.1:0/1781856992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27a815b4d0 0x7f27a81f05c0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f27a0004000 tx=0x7f27a0004ad0 comp rx=0 tx=0).stop 2024-03-18T20:07:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.109+0000 7f27aff82700 1 -- 172.21.15.1:0/1781856992 shutdown_connections 2024-03-18T20:07:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.109+0000 7f27aff82700 1 --2- 172.21.15.1:0/1781856992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27a815b4d0 0x7f27a81f05c0 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.109+0000 7f27aff82700 1 --2- 172.21.15.1:0/1781856992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27a809a150 0x7f27a8147f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.109+0000 7f27aff82700 1 --2- 172.21.15.1:0/1781856992 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27a80cf750 0x7f27a80b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.109+0000 7f27aff82700 1 -- 172.21.15.1:0/1781856992 >> 172.21.15.1:0/1781856992 conn(0x7f27a80b2cb0 msgr2=0x7f27a80ade00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.109+0000 7f27aff82700 1 -- 172.21.15.1:0/1781856992 shutdown_connections 2024-03-18T20:07:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.109+0000 7f27aff82700 1 -- 172.21.15.1:0/1781856992 wait complete. 2024-03-18T20:07:06.127 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T20:07:06.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.300+0000 7f045e5b1700 1 Processor -- start 2024-03-18T20:07:06.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.300+0000 7f045e5b1700 1 -- start start 2024-03-18T20:07:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.301+0000 7f045e5b1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f045806fd80 0x7f045815a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.301+0000 7f045e5b1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f04580aec20 0x7f045806e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.301+0000 7f045e5b1700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04580acd80 con 0x7f04581548b0 2024-03-18T20:07:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.301+0000 7f045e5b1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04580adbe0 con 0x7f04580aec20 2024-03-18T20:07:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.301+0000 7f045e5b1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04580acfd0 con 0x7f045806fd80 2024-03-18T20:07:06.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.301+0000 7f045cb4e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f04580aec20 0x7f045806e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:06.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.301+0000 7f045cb4e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f04580aec20 0x7f045806e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46404/0 (socket says 172.21.15.1:46404) 2024-03-18T20:07:06.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.301+0000 7f045cb4e700 1 -- 172.21.15.1:0/191616172 learned_addr learned my addr 172.21.15.1:0/191616172 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:06.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.301+0000 7f04577fe700 1 --2- 172.21.15.1:0/191616172 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f045806fd80 0x7f045815a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:06.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.302+0000 7f045cb4e700 1 -- 172.21.15.1:0/191616172 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f045806fd80 msgr2=0x7f045815a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:06.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.302+0000 7f045cb4e700 1 --2- 172.21.15.1:0/191616172 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f045806fd80 0x7f045815a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:06.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.302+0000 7f045cb4e700 1 -- 172.21.15.1:0/191616172 >> v1:172.21.15.1:6789/0 conn(0x7f04581548b0 legacy=0x7f04580ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:06.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.302+0000 7f045cb4e700 1 -- 172.21.15.1:0/191616172 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04580ad2b0 con 0x7f04580aec20 2024-03-18T20:07:06.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.302+0000 7f04577fe700 1 --2- 172.21.15.1:0/191616172 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f045806fd80 0x7f045815a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:06.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.302+0000 7f045cb4e700 1 --2- 172.21.15.1:0/191616172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04580aec20 0x7f045806e150 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f0450009e50 tx=0x7f045000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb6eb7d71652129 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:06.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.302+0000 7f0456ffd700 1 -- 172.21.15.1:0/191616172 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045000e070 con 0x7f04580aec20 2024-03-18T20:07:06.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.302+0000 7f0456ffd700 1 -- 172.21.15.1:0/191616172 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045000b9c0 con 0x7f04580aec20 2024-03-18T20:07:06.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.302+0000 7f0456ffd700 1 -- 172.21.15.1:0/191616172 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045000bd70 con 0x7f04580aec20 2024-03-18T20:07:06.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.302+0000 7f045e5b1700 1 -- 172.21.15.1:0/191616172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04580aec20 msgr2=0x7f045806e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:06.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.302+0000 7f045e5b1700 1 --2- 172.21.15.1:0/191616172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04580aec20 0x7f045806e150 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f0450009e50 tx=0x7f045000b330 comp rx=0 tx=0).stop 2024-03-18T20:07:06.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.302+0000 7f045e5b1700 1 -- 172.21.15.1:0/191616172 shutdown_connections 2024-03-18T20:07:06.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.302+0000 7f045e5b1700 1 --2- 172.21.15.1:0/191616172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04580aec20 0x7f045806e150 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:06.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.302+0000 7f045e5b1700 1 --2- 172.21.15.1:0/191616172 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f045806fd80 0x7f045815a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:06.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.302+0000 7f045e5b1700 1 -- 172.21.15.1:0/191616172 >> 172.21.15.1:0/191616172 conn(0x7f045806b860 msgr2=0x7f045806bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:06.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.303+0000 7f045e5b1700 1 -- 172.21.15.1:0/191616172 shutdown_connections 2024-03-18T20:07:06.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.303+0000 7f045e5b1700 1 -- 172.21.15.1:0/191616172 wait complete. 2024-03-18T20:07:06.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.304+0000 7f045e5b1700 1 Processor -- start 2024-03-18T20:07:06.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.304+0000 7f045e5b1700 1 -- start start 2024-03-18T20:07:06.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f045e5b1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f045806fd80 0x7f045813f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:06.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f045e5b1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04580aec20 0x7f0458159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:06.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f045e5b1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04581548b0 0x7f0458159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:06.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f045e5b1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045813fe80 con 0x7f04580aec20 2024-03-18T20:07:06.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f045e5b1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045809cf20 con 0x7f04581548b0 2024-03-18T20:07:06.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f045e5b1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f045809d200 con 0x7f045806fd80 2024-03-18T20:07:06.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f04577fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04580aec20 0x7f0458159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:06.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f04577fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04580aec20 0x7f0458159130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46420/0 (socket says 172.21.15.1:46420) 2024-03-18T20:07:06.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f04577fe700 1 -- 172.21.15.1:0/1822843006 learned_addr learned my addr 172.21.15.1:0/1822843006 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:06.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f0457fff700 1 --2- 172.21.15.1:0/1822843006 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f045806fd80 0x7f045813f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:06.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f04577fe700 1 -- 172.21.15.1:0/1822843006 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f045806fd80 msgr2=0x7f045813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:06.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f04577fe700 1 --2- 172.21.15.1:0/1822843006 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f045806fd80 0x7f045813f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:06.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f04577fe700 1 -- 172.21.15.1:0/1822843006 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04581548b0 msgr2=0x7f0458159670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f04577fe700 1 --2- 172.21.15.1:0/1822843006 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04581548b0 0x7f0458159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f04577fe700 1 -- 172.21.15.1:0/1822843006 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0450009b30 con 0x7f04580aec20 2024-03-18T20:07:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f04577fe700 1 --2- 172.21.15.1:0/1822843006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04580aec20 0x7f0458159130 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f044800c750 tx=0x7f044800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f0454ff9700 1 -- 172.21.15.1:0/1822843006 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0448011800 con 0x7f04580aec20 2024-03-18T20:07:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f0454ff9700 1 -- 172.21.15.1:0/1822843006 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0448011960 con 0x7f04580aec20 2024-03-18T20:07:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f045e5b1700 1 -- 172.21.15.1:0/1822843006 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f045809d4e0 con 0x7f04580aec20 2024-03-18T20:07:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f0454ff9700 1 -- 172.21.15.1:0/1822843006 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0448011cd0 con 0x7f04580aec20 2024-03-18T20:07:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.305+0000 7f045e5b1700 1 -- 172.21.15.1:0/1822843006 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f045809c130 con 0x7f04580aec20 2024-03-18T20:07:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.306+0000 7f045e5b1700 1 -- 172.21.15.1:0/1822843006 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0458066ff0 con 0x7f04580aec20 2024-03-18T20:07:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.307+0000 7f0454ff9700 1 -- 172.21.15.1:0/1822843006 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 101) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f0448021350 con 0x7f04580aec20 2024-03-18T20:07:06.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.307+0000 7f0454ff9700 1 -- 172.21.15.1:0/1822843006 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6703+0+0 (secure 0 0 0) 0x7f04480890f0 con 0x7f04580aec20 2024-03-18T20:07:06.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.312+0000 7f0454ff9700 1 -- 172.21.15.1:0/1822843006 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0448054340 con 0x7f04580aec20 2024-03-18T20:07:06.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.558+0000 7f045e5b1700 1 -- 172.21.15.1:0/1822843006 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f04580676a0 con 0x7f04580aec20 2024-03-18T20:07:06.563 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-18T20:07:06.559+0000 7fbdbeb46700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T20:07:06.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.566+0000 7f0454ff9700 1 -- 172.21.15.1:0/1822843006 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v82) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0448057960 con 0x7f04580aec20 2024-03-18T20:07:06.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.568+0000 7f045e5b1700 1 -- 172.21.15.1:0/1822843006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04580aec20 msgr2=0x7f0458159130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:06.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.568+0000 7f045e5b1700 1 --2- 172.21.15.1:0/1822843006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04580aec20 0x7f0458159130 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f044800c750 tx=0x7f044800cb10 comp rx=0 tx=0).stop 2024-03-18T20:07:06.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.568+0000 7f045e5b1700 1 -- 172.21.15.1:0/1822843006 shutdown_connections 2024-03-18T20:07:06.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.568+0000 7f045e5b1700 1 --2- 172.21.15.1:0/1822843006 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f045806fd80 0x7f045813f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:06.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.568+0000 7f045e5b1700 1 --2- 172.21.15.1:0/1822843006 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04581548b0 0x7f0458159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:06.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.568+0000 7f045e5b1700 1 --2- 172.21.15.1:0/1822843006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04580aec20 0x7f0458159130 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:06.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.568+0000 7f045e5b1700 1 -- 172.21.15.1:0/1822843006 >> 172.21.15.1:0/1822843006 conn(0x7f045806b860 msgr2=0x7f045806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:06.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.568+0000 7f045e5b1700 1 -- 172.21.15.1:0/1822843006 shutdown_connections 2024-03-18T20:07:06.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.568+0000 7f045e5b1700 1 -- 172.21.15.1:0/1822843006 wait complete. 2024-03-18T20:07:06.574 INFO:tasks.ceph.mds.c.smithi001.stderr: -2850> 2024-03-18T20:06:46.899+0000 7f02478b7700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T20:07:06.574 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T20:07:06.586 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T20:07:06.602 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T20:07:06.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.714+0000 7f5243418700 1 Processor -- start 2024-03-18T20:07:06.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.714+0000 7f5243418700 1 -- start start 2024-03-18T20:07:06.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.715+0000 7f5243418700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f523c0cfa90 0x7f523c06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:06.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.715+0000 7f5243418700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f523c06ac90 0x7f523c167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:06.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.715+0000 7f5243418700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523c0cf600 con 0x7f523c15d5a0 2024-03-18T20:07:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.715+0000 7f5243418700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f523c06b140 con 0x7f523c06ac90 2024-03-18T20:07:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.715+0000 7f5243418700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523c0ce720 con 0x7f523c0cfa90 2024-03-18T20:07:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.715+0000 7f52409b3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f523c0cfa90 0x7f523c06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.715+0000 7f52419b5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f523c06ac90 0x7f523c167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.715+0000 7f52411b4700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f523c15d5a0 0x7f523c0ce010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:56548/0 (socket says 172.21.15.1:56548) 2024-03-18T20:07:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.715+0000 7f52409b3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f523c0cfa90 0x7f523c06a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:36262/0 (socket says 172.21.15.1:36262) 2024-03-18T20:07:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.715+0000 7f52419b5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f523c06ac90 0x7f523c167140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:55566/0 (socket says 172.21.15.1:55566) 2024-03-18T20:07:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.715+0000 7f52411b4700 1 -- 172.21.15.1:0/2526278796 learned_addr learned my addr 172.21.15.1:0/2526278796 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.715+0000 7f52409b3700 1 -- 172.21.15.1:0/2526278796 >> v2:172.21.15.165:3300/0 conn(0x7f523c06ac90 msgr2=0x7f523c167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.715+0000 7f52409b3700 1 --2- 172.21.15.1:0/2526278796 >> v2:172.21.15.165:3300/0 conn(0x7f523c06ac90 0x7f523c167140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.715+0000 7f52409b3700 1 -- 172.21.15.1:0/2526278796 >> v1:172.21.15.165:6789/0 conn(0x7f523c15d5a0 legacy=0x7f523c0ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.715+0000 7f52409b3700 1 -- 172.21.15.1:0/2526278796 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523c0ce9f0 con 0x7f523c0cfa90 2024-03-18T20:07:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.715+0000 7f52419b5700 1 --2- 172.21.15.1:0/2526278796 >> v2:172.21.15.165:3300/0 conn(0x7f523c06ac90 0x7f523c167140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:07:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.716+0000 7f52409b3700 1 --2- 172.21.15.1:0/2526278796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f523c0cfa90 0x7f523c06a750 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f522c009870 tx=0x7f522c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47b2ad4da856110b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.716+0000 7f5233fff700 1 -- 172.21.15.1:0/2526278796 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522c011830 con 0x7f523c0cfa90 2024-03-18T20:07:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.716+0000 7f5233fff700 1 -- 172.21.15.1:0/2526278796 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f522c011990 con 0x7f523c0cfa90 2024-03-18T20:07:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.716+0000 7f5233fff700 1 -- 172.21.15.1:0/2526278796 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522c011d20 con 0x7f523c0cfa90 2024-03-18T20:07:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.716+0000 7f5243418700 1 -- 172.21.15.1:0/2526278796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f523c0cfa90 msgr2=0x7f523c06a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.716+0000 7f5243418700 1 --2- 172.21.15.1:0/2526278796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f523c0cfa90 0x7f523c06a750 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f522c009870 tx=0x7f522c009b80 comp rx=0 tx=0).stop 2024-03-18T20:07:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.716+0000 7f5243418700 1 -- 172.21.15.1:0/2526278796 shutdown_connections 2024-03-18T20:07:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.716+0000 7f5243418700 1 --2- 172.21.15.1:0/2526278796 >> v2:172.21.15.165:3300/0 conn(0x7f523c06ac90 0x7f523c167140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.716+0000 7f5243418700 1 --2- 172.21.15.1:0/2526278796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f523c0cfa90 0x7f523c06a750 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.716+0000 7f5243418700 1 -- 172.21.15.1:0/2526278796 >> 172.21.15.1:0/2526278796 conn(0x7f523c0cc9b0 msgr2=0x7f523c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.717+0000 7f5243418700 1 -- 172.21.15.1:0/2526278796 shutdown_connections 2024-03-18T20:07:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.717+0000 7f5243418700 1 -- 172.21.15.1:0/2526278796 wait complete. 2024-03-18T20:07:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.718+0000 7f5243418700 1 Processor -- start 2024-03-18T20:07:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.718+0000 7f5243418700 1 -- start start 2024-03-18T20:07:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.718+0000 7f5243418700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f523c06ac90 0x7f523c159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f5243418700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f523c0cfa90 0x7f523c15ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f5243418700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f523c15d5a0 0x7f523c1432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f5243418700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523c0c9ed0 con 0x7f523c15d5a0 2024-03-18T20:07:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f5243418700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523c15a1e0 con 0x7f523c0cfa90 2024-03-18T20:07:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f5243418700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523c15a3f0 con 0x7f523c06ac90 2024-03-18T20:07:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f52419b5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f523c15d5a0 0x7f523c1432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f52419b5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f523c15d5a0 0x7f523c1432a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46460/0 (socket says 172.21.15.1:46460) 2024-03-18T20:07:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f52419b5700 1 -- 172.21.15.1:0/2351172426 learned_addr learned my addr 172.21.15.1:0/2351172426 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f52411b4700 1 --2- 172.21.15.1:0/2351172426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f523c06ac90 0x7f523c159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:06.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f52409b3700 1 --2- 172.21.15.1:0/2351172426 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f523c0cfa90 0x7f523c15ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f52419b5700 1 -- 172.21.15.1:0/2351172426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f523c06ac90 msgr2=0x7f523c159ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f52419b5700 1 --2- 172.21.15.1:0/2351172426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f523c06ac90 0x7f523c159ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f52419b5700 1 -- 172.21.15.1:0/2351172426 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f523c0cfa90 msgr2=0x7f523c15ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f52419b5700 1 --2- 172.21.15.1:0/2351172426 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f523c0cfa90 0x7f523c15ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f52419b5700 1 -- 172.21.15.1:0/2351172426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5234009a70 con 0x7f523c15d5a0 2024-03-18T20:07:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f52411b4700 1 --2- 172.21.15.1:0/2351172426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f523c06ac90 0x7f523c159ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:07:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f52409b3700 1 --2- 172.21.15.1:0/2351172426 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f523c0cfa90 0x7f523c15ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:07:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f52419b5700 1 --2- 172.21.15.1:0/2351172426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f523c15d5a0 0x7f523c1432a0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f523400efc0 tx=0x7f523400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f5231ffb700 1 -- 172.21.15.1:0/2351172426 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5234009cd0 con 0x7f523c15d5a0 2024-03-18T20:07:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f5231ffb700 1 -- 172.21.15.1:0/2351172426 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523400f040 con 0x7f523c15d5a0 2024-03-18T20:07:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f5231ffb700 1 -- 172.21.15.1:0/2351172426 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523400cd60 con 0x7f523c15d5a0 2024-03-18T20:07:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.719+0000 7f5243418700 1 -- 172.21.15.1:0/2351172426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522c0094a0 con 0x7f523c15d5a0 2024-03-18T20:07:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.720+0000 7f5243418700 1 -- 172.21.15.1:0/2351172426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523c15ac90 con 0x7f523c15d5a0 2024-03-18T20:07:06.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.721+0000 7f5243418700 1 -- 172.21.15.1:0/2351172426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f523c072960 con 0x7f523c15d5a0 2024-03-18T20:07:06.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.721+0000 7f5231ffb700 1 -- 172.21.15.1:0/2351172426 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 102) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f5234014480 con 0x7f523c15d5a0 2024-03-18T20:07:06.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.722+0000 7f5231ffb700 1 --2- 172.21.15.1:0/2351172426 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f522006aae0 0x7f522006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:06.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.722+0000 7f52411b4700 1 --2- 172.21.15.1:0/2351172426 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f522006aae0 0x7f522006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:06.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.722+0000 7f5231ffb700 1 -- 172.21.15.1:0/2351172426 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6789+0+0 (secure 0 0 0) 0x7f5234089b10 con 0x7f523c15d5a0 2024-03-18T20:07:06.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.722+0000 7f52411b4700 1 --2- 172.21.15.1:0/2351172426 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f522006aae0 0x7f522006cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5238005960 tx=0x7f5238009040 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:06.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.726+0000 7f5231ffb700 1 -- 172.21.15.1:0/2351172426 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5234054b60 con 0x7f523c15d5a0 2024-03-18T20:07:06.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:06.975+0000 7f5243418700 1 -- 172.21.15.1:0/2351172426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f523c0676a0 con 0x7f523c15d5a0 2024-03-18T20:07:07.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.568+0000 7f5231ffb700 1 -- 172.21.15.1:0/2351172426 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v83) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5234058180 con 0x7f523c15d5a0 2024-03-18T20:07:07.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.571+0000 7f5243418700 1 -- 172.21.15.1:0/2351172426 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f522006aae0 msgr2=0x7f522006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.571+0000 7f5243418700 1 --2- 172.21.15.1:0/2351172426 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f522006aae0 0x7f522006cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5238005960 tx=0x7f5238009040 comp rx=0 tx=0).stop 2024-03-18T20:07:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.571+0000 7f5243418700 1 -- 172.21.15.1:0/2351172426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f523c15d5a0 msgr2=0x7f523c1432a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.571+0000 7f5243418700 1 --2- 172.21.15.1:0/2351172426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f523c15d5a0 0x7f523c1432a0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f523400efc0 tx=0x7f523400c3a0 comp rx=0 tx=0).stop 2024-03-18T20:07:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.572+0000 7f5243418700 1 -- 172.21.15.1:0/2351172426 shutdown_connections 2024-03-18T20:07:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.572+0000 7f5243418700 1 --2- 172.21.15.1:0/2351172426 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f522006aae0 0x7f522006cfa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.572+0000 7f5243418700 1 --2- 172.21.15.1:0/2351172426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f523c06ac90 0x7f523c159ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.572+0000 7f5243418700 1 --2- 172.21.15.1:0/2351172426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f523c15d5a0 0x7f523c1432a0 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.572+0000 7f5243418700 1 --2- 172.21.15.1:0/2351172426 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f523c0cfa90 0x7f523c15ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.572+0000 7f5243418700 1 -- 172.21.15.1:0/2351172426 >> 172.21.15.1:0/2351172426 conn(0x7f523c0cc9b0 msgr2=0x7f523c15c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.572+0000 7f5243418700 1 -- 172.21.15.1:0/2351172426 shutdown_connections 2024-03-18T20:07:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.572+0000 7f5243418700 1 -- 172.21.15.1:0/2351172426 wait complete. 2024-03-18T20:07:07.591 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T20:07:07.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.771+0000 7f54ed6e7700 1 Processor -- start 2024-03-18T20:07:07.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.771+0000 7f54ed6e7700 1 -- start start 2024-03-18T20:07:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.772+0000 7f54ed6e7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54e815d8b0 0x7f54e80d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.772+0000 7f54ed6e7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54e80d0040 0x7f54e80cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.772+0000 7f54ed6e7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e815dc90 con 0x7f54e80d1170 2024-03-18T20:07:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.772+0000 7f54ed6e7700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e815ec90 con 0x7f54e80d0040 2024-03-18T20:07:07.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.772+0000 7f54ed6e7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e815ded0 con 0x7f54e815d8b0 2024-03-18T20:07:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.772+0000 7f54e67fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f54e80d1170 0x7f54e80cf930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58702/0 (socket says 172.21.15.1:58702) 2024-03-18T20:07:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.772+0000 7f54e77fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54e80d0040 0x7f54e80cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.772+0000 7f54e67fc700 1 -- 172.21.15.1:0/117126701 learned_addr learned my addr 172.21.15.1:0/117126701 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.772+0000 7f54e6ffd700 1 --2- 172.21.15.1:0/117126701 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54e815d8b0 0x7f54e80d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.772+0000 7f54e5ffb700 1 -- 172.21.15.1:0/117126701 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3738033659 0 0) 0x7f54d0001110 con 0x7f54e80d1170 2024-03-18T20:07:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.772+0000 7f54e5ffb700 1 -- 172.21.15.1:0/117126701 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54d40035b0 con 0x7f54e80d1170 2024-03-18T20:07:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.772+0000 7f54e77fe700 1 -- 172.21.15.1:0/117126701 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54e815d8b0 msgr2=0x7f54e80d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.772+0000 7f54e77fe700 1 --2- 172.21.15.1:0/117126701 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54e815d8b0 0x7f54e80d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.772+0000 7f54e77fe700 1 -- 172.21.15.1:0/117126701 >> v1:172.21.15.1:6789/0 conn(0x7f54e80d1170 legacy=0x7f54e80cf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.772+0000 7f54e77fe700 1 -- 172.21.15.1:0/117126701 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e815e1b0 con 0x7f54e80d0040 2024-03-18T20:07:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.773+0000 7f54e77fe700 1 --2- 172.21.15.1:0/117126701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e80d0040 0x7f54e80cde10 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f54d8009d90 tx=0x7f54d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d30234b29f9dfc07 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.773+0000 7f54e5ffb700 1 -- 172.21.15.1:0/117126701 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d800e070 con 0x7f54e80d0040 2024-03-18T20:07:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.773+0000 7f54e5ffb700 1 -- 172.21.15.1:0/117126701 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d800bb20 con 0x7f54e80d0040 2024-03-18T20:07:07.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.773+0000 7f54e5ffb700 1 -- 172.21.15.1:0/117126701 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d8012430 con 0x7f54e80d0040 2024-03-18T20:07:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.773+0000 7f54ed6e7700 1 -- 172.21.15.1:0/117126701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e80d0040 msgr2=0x7f54e80cde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.773+0000 7f54ed6e7700 1 --2- 172.21.15.1:0/117126701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e80d0040 0x7f54e80cde10 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f54d8009d90 tx=0x7f54d800b330 comp rx=0 tx=0).stop 2024-03-18T20:07:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.773+0000 7f54ed6e7700 1 -- 172.21.15.1:0/117126701 shutdown_connections 2024-03-18T20:07:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.773+0000 7f54ed6e7700 1 --2- 172.21.15.1:0/117126701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e80d0040 0x7f54e80cde10 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.773+0000 7f54ed6e7700 1 --2- 172.21.15.1:0/117126701 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54e815d8b0 0x7f54e80d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.773+0000 7f54ed6e7700 1 -- 172.21.15.1:0/117126701 >> 172.21.15.1:0/117126701 conn(0x7f54e809c640 msgr2=0x7f54e80d3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.773+0000 7f54ed6e7700 1 -- 172.21.15.1:0/117126701 shutdown_connections 2024-03-18T20:07:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.773+0000 7f54ed6e7700 1 -- 172.21.15.1:0/117126701 wait complete. 2024-03-18T20:07:07.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.775+0000 7f54ed6e7700 1 Processor -- start 2024-03-18T20:07:07.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.775+0000 7f54ed6e7700 1 -- start start 2024-03-18T20:07:07.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.775+0000 7f54ed6e7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54e80d0040 0x7f54e80b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:07.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.775+0000 7f54ed6e7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54e80d1170 0x7f54e80c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:07.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.775+0000 7f54ed6e7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e815d8b0 0x7f54e806b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:07.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.775+0000 7f54ed6e7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e80c69c0 con 0x7f54e815d8b0 2024-03-18T20:07:07.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.775+0000 7f54ed6e7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e80b7cd0 con 0x7f54e80d0040 2024-03-18T20:07:07.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.775+0000 7f54ed6e7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e815c4b0 con 0x7f54e80d1170 2024-03-18T20:07:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.775+0000 7f54e77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e815d8b0 0x7f54e806b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.775+0000 7f54e77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e815d8b0 0x7f54e806b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46484/0 (socket says 172.21.15.1:46484) 2024-03-18T20:07:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.775+0000 7f54e77fe700 1 -- 172.21.15.1:0/2342245603 learned_addr learned my addr 172.21.15.1:0/2342245603 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.776+0000 7f54e6ffd700 1 --2- 172.21.15.1:0/2342245603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54e80d0040 0x7f54e80b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.776+0000 7f54e77fe700 1 -- 172.21.15.1:0/2342245603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54e80d1170 msgr2=0x7f54e80c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.776+0000 7f54e77fe700 1 --2- 172.21.15.1:0/2342245603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54e80d1170 0x7f54e80c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.776+0000 7f54e77fe700 1 -- 172.21.15.1:0/2342245603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54e80d0040 msgr2=0x7f54e80b89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.776+0000 7f54e77fe700 1 --2- 172.21.15.1:0/2342245603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54e80d0040 0x7f54e80b89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.776+0000 7f54e77fe700 1 -- 172.21.15.1:0/2342245603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d8009a70 con 0x7f54e815d8b0 2024-03-18T20:07:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.776+0000 7f54e77fe700 1 --2- 172.21.15.1:0/2342245603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e815d8b0 0x7f54e806b5a0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f54d8014040 tx=0x7f54d8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.776+0000 7f54df7fe700 1 -- 172.21.15.1:0/2342245603 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d80129f0 con 0x7f54e815d8b0 2024-03-18T20:07:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.776+0000 7f54df7fe700 1 -- 172.21.15.1:0/2342245603 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d8020070 con 0x7f54e815d8b0 2024-03-18T20:07:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.776+0000 7f54ed6e7700 1 -- 172.21.15.1:0/2342245603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e815c1d0 con 0x7f54e815d8b0 2024-03-18T20:07:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.776+0000 7f54df7fe700 1 -- 172.21.15.1:0/2342245603 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d8012db0 con 0x7f54e815d8b0 2024-03-18T20:07:07.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.776+0000 7f54ed6e7700 1 -- 172.21.15.1:0/2342245603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e80b81e0 con 0x7f54e815d8b0 2024-03-18T20:07:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.777+0000 7f54ed6e7700 1 -- 172.21.15.1:0/2342245603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e8073350 con 0x7f54e815d8b0 2024-03-18T20:07:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.778+0000 7f54df7fe700 1 -- 172.21.15.1:0/2342245603 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 102) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f54d8004040 con 0x7f54e815d8b0 2024-03-18T20:07:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.778+0000 7f54df7fe700 1 --2- 172.21.15.1:0/2342245603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f54c806aae0 0x7f54c806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.778+0000 7f54df7fe700 1 -- 172.21.15.1:0/2342245603 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6789+0+0 (secure 0 0 0) 0x7f54d8022070 con 0x7f54e815d8b0 2024-03-18T20:07:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.778+0000 7f54e6ffd700 1 --2- 172.21.15.1:0/2342245603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f54c806aae0 0x7f54c806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.778+0000 7f54e6ffd700 1 --2- 172.21.15.1:0/2342245603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f54c806aae0 0x7f54c806cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f54e0009a10 tx=0x7f54e0008040 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:07.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.783+0000 7f54df7fe700 1 -- 172.21.15.1:0/2342245603 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f54d8056160 con 0x7f54e815d8b0 2024-03-18T20:07:07.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:07.986+0000 7f54ed6e7700 1 -- 172.21.15.1:0/2342245603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f54e80d3a10 con 0x7f54e815d8b0 2024-03-18T20:07:08.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:08.633+0000 7f54df7fe700 1 -- 172.21.15.1:0/2342245603 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 103) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f54d8056530 con 0x7f54e815d8b0 2024-03-18T20:07:08.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:08.859+0000 7f54df7fe700 1 -- 172.21.15.1:0/2342245603 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v185) v1 ==== 181+0+0 (secure 0 0 0) 0x7f54d8059bf0 con 0x7f54e815d8b0 2024-03-18T20:07:08.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:08.864+0000 7f54ed6e7700 1 -- 172.21.15.1:0/2342245603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f54c806aae0 msgr2=0x7f54c806cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:08.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:08.864+0000 7f54ed6e7700 1 --2- 172.21.15.1:0/2342245603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f54c806aae0 0x7f54c806cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f54e0009a10 tx=0x7f54e0008040 comp rx=0 tx=0).stop 2024-03-18T20:07:08.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:08.864+0000 7f54ed6e7700 1 -- 172.21.15.1:0/2342245603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e815d8b0 msgr2=0x7f54e806b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:08.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:08.864+0000 7f54ed6e7700 1 --2- 172.21.15.1:0/2342245603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e815d8b0 0x7f54e806b5a0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f54d8014040 tx=0x7f54d8004ad0 comp rx=0 tx=0).stop 2024-03-18T20:07:08.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:08.864+0000 7f54ed6e7700 1 -- 172.21.15.1:0/2342245603 shutdown_connections 2024-03-18T20:07:08.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:08.864+0000 7f54ed6e7700 1 --2- 172.21.15.1:0/2342245603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f54c806aae0 0x7f54c806cfa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:08.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:08.864+0000 7f54ed6e7700 1 --2- 172.21.15.1:0/2342245603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54e815d8b0 0x7f54e806b5a0 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:08.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:08.864+0000 7f54ed6e7700 1 --2- 172.21.15.1:0/2342245603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54e80d0040 0x7f54e80b89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:08.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:08.864+0000 7f54ed6e7700 1 --2- 172.21.15.1:0/2342245603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54e80d1170 0x7f54e80c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:08.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:08.864+0000 7f54ed6e7700 1 -- 172.21.15.1:0/2342245603 >> 172.21.15.1:0/2342245603 conn(0x7f54e809c640 msgr2=0x7f54e80b5690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:08.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:08.864+0000 7f54ed6e7700 1 -- 172.21.15.1:0/2342245603 shutdown_connections 2024-03-18T20:07:08.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:08.864+0000 7f54ed6e7700 1 -- 172.21.15.1:0/2342245603 wait complete. 2024-03-18T20:07:08.882 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-18T20:07:08.886 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T20:07:09.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.019+0000 7f775212e700 1 Processor -- start 2024-03-18T20:07:09.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.019+0000 7f775212e700 1 -- start start 2024-03-18T20:07:09.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.019+0000 7f775212e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f774c0b2fa0 0x7f774c15c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:09.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.019+0000 7f775212e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f774c0d1dc0 0x7f774c0b60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:09.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.019+0000 7f775212e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f774c15c8a0 con 0x7f774c15fc10 2024-03-18T20:07:09.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.019+0000 7f775212e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f774c0cc0a0 con 0x7f774c0d1dc0 2024-03-18T20:07:09.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.019+0000 7f775212e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f774c15ef20 con 0x7f774c0b2fa0 2024-03-18T20:07:09.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.020+0000 7f774bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f774c0d1dc0 0x7f774c0b60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:09.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.020+0000 7f774bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f774c0d1dc0 0x7f774c0b60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46500/0 (socket says 172.21.15.1:46500) 2024-03-18T20:07:09.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.020+0000 7f774affd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f774c15fc10 0x7f774c0d16b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58704/0 (socket says 172.21.15.1:58704) 2024-03-18T20:07:09.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.020+0000 7f774bfff700 1 -- 172.21.15.1:0/1531187656 learned_addr learned my addr 172.21.15.1:0/1531187656 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:09.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.020+0000 7f774b7fe700 1 --2- 172.21.15.1:0/1531187656 >> v2:172.21.15.165:3300/0 conn(0x7f774c0b2fa0 0x7f774c15c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:09.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.020+0000 7f774bfff700 1 -- 172.21.15.1:0/1531187656 >> v2:172.21.15.165:3300/0 conn(0x7f774c0b2fa0 msgr2=0x7f774c15c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:09.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.020+0000 7f774bfff700 1 --2- 172.21.15.1:0/1531187656 >> v2:172.21.15.165:3300/0 conn(0x7f774c0b2fa0 0x7f774c15c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:09.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.020+0000 7f774bfff700 1 -- 172.21.15.1:0/1531187656 >> v1:172.21.15.1:6789/0 conn(0x7f774c15fc10 legacy=0x7f774c0d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:09.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.020+0000 7f774bfff700 1 -- 172.21.15.1:0/1531187656 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f774c0cbdc0 con 0x7f774c0d1dc0 2024-03-18T20:07:09.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.020+0000 7f774a7fc700 1 -- 172.21.15.1:0/1531187656 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389360581 0 0) 0x7f774c15c8a0 con 0x7f774c15fc10 2024-03-18T20:07:09.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.020+0000 7f774bfff700 1 --2- 172.21.15.1:0/1531187656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f774c0d1dc0 0x7f774c0b60c0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f774c06b400 tx=0x7f774400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=768287718d0bfd1d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:09.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.020+0000 7f774a7fc700 1 -- 172.21.15.1:0/1531187656 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774400e070 con 0x7f774c0d1dc0 2024-03-18T20:07:09.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.020+0000 7f774a7fc700 1 -- 172.21.15.1:0/1531187656 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f774400b9c0 con 0x7f774c0d1dc0 2024-03-18T20:07:09.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.020+0000 7f774a7fc700 1 -- 172.21.15.1:0/1531187656 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774400bd70 con 0x7f774c0d1dc0 2024-03-18T20:07:09.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.021+0000 7f775212e700 1 -- 172.21.15.1:0/1531187656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f774c0d1dc0 msgr2=0x7f774c0b60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:09.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.021+0000 7f775212e700 1 --2- 172.21.15.1:0/1531187656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f774c0d1dc0 0x7f774c0b60c0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f774c06b400 tx=0x7f774400b330 comp rx=0 tx=0).stop 2024-03-18T20:07:09.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.021+0000 7f775212e700 1 -- 172.21.15.1:0/1531187656 shutdown_connections 2024-03-18T20:07:09.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.021+0000 7f775212e700 1 --2- 172.21.15.1:0/1531187656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f774c0d1dc0 0x7f774c0b60c0 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:09.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.021+0000 7f775212e700 1 --2- 172.21.15.1:0/1531187656 >> v2:172.21.15.165:3300/0 conn(0x7f774c0b2fa0 0x7f774c15c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:09.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.021+0000 7f775212e700 1 -- 172.21.15.1:0/1531187656 >> 172.21.15.1:0/1531187656 conn(0x7f774c06a750 msgr2=0x7f774c06ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:09.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.021+0000 7f775212e700 1 -- 172.21.15.1:0/1531187656 shutdown_connections 2024-03-18T20:07:09.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.021+0000 7f775212e700 1 -- 172.21.15.1:0/1531187656 wait complete. 2024-03-18T20:07:09.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.022+0000 7f775212e700 1 Processor -- start 2024-03-18T20:07:09.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.022+0000 7f775212e700 1 -- start start 2024-03-18T20:07:09.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.023+0000 7f775212e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f774c0b2fa0 0x7f774c06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.023+0000 7f775212e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f774c0d1dc0 0x7f774c0b6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.023+0000 7f775212e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f774c15fc10 0x7f774c1546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.023+0000 7f775212e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f774c0b6d70 con 0x7f774c0b2fa0 2024-03-18T20:07:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.023+0000 7f775212e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f774c154be0 con 0x7f774c0d1dc0 2024-03-18T20:07:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.023+0000 7f775212e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f774c06d220 con 0x7f774c15fc10 2024-03-18T20:07:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.023+0000 7f774b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f774c0b2fa0 0x7f774c06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.023+0000 7f774b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f774c0b2fa0 0x7f774c06bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46506/0 (socket says 172.21.15.1:46506) 2024-03-18T20:07:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.023+0000 7f774b7fe700 1 -- 172.21.15.1:0/1233650319 learned_addr learned my addr 172.21.15.1:0/1233650319 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.023+0000 7f774affd700 1 --2- 172.21.15.1:0/1233650319 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f774c0d1dc0 0x7f774c0b6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.023+0000 7f774bfff700 1 --2- 172.21.15.1:0/1233650319 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f774c15fc10 0x7f774c1546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.024+0000 7f774b7fe700 1 -- 172.21.15.1:0/1233650319 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f774c15fc10 msgr2=0x7f774c1546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.024+0000 7f774b7fe700 1 --2- 172.21.15.1:0/1233650319 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f774c15fc10 0x7f774c1546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.024+0000 7f774b7fe700 1 -- 172.21.15.1:0/1233650319 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f774c0d1dc0 msgr2=0x7f774c0b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:09.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.024+0000 7f774b7fe700 1 --2- 172.21.15.1:0/1233650319 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f774c0d1dc0 0x7f774c0b6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.024+0000 7f774b7fe700 1 -- 172.21.15.1:0/1233650319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7744009b30 con 0x7f774c0b2fa0 2024-03-18T20:07:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.024+0000 7f774b7fe700 1 --2- 172.21.15.1:0/1233650319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f774c0b2fa0 0x7f774c06bf50 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f773c00e810 tx=0x7f773c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.024+0000 7f7733fff700 1 -- 172.21.15.1:0/1233650319 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c00c070 con 0x7f774c0b2fa0 2024-03-18T20:07:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.024+0000 7f7733fff700 1 -- 172.21.15.1:0/1233650319 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f773c013610 con 0x7f774c0b2fa0 2024-03-18T20:07:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.024+0000 7f775212e700 1 -- 172.21.15.1:0/1233650319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f774c06c610 con 0x7f774c0b2fa0 2024-03-18T20:07:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.024+0000 7f7733fff700 1 -- 172.21.15.1:0/1233650319 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c0139a0 con 0x7f774c0b2fa0 2024-03-18T20:07:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.024+0000 7f775212e700 1 -- 172.21.15.1:0/1233650319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f774c06cbd0 con 0x7f774c0b2fa0 2024-03-18T20:07:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.025+0000 7f775212e700 1 -- 172.21.15.1:0/1233650319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f774c0c9190 con 0x7f774c0b2fa0 2024-03-18T20:07:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.025+0000 7f7733fff700 1 -- 172.21.15.1:0/1233650319 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 103) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f773c013b00 con 0x7f774c0b2fa0 2024-03-18T20:07:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.026+0000 7f7733fff700 1 --2- 172.21.15.1:0/1233650319 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f773806ad20 0x7f773806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:09.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.026+0000 7f7733fff700 1 -- 172.21.15.1:0/1233650319 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6351+0+0 (secure 0 0 0) 0x7f773c089500 con 0x7f774c0b2fa0 2024-03-18T20:07:09.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.026+0000 7f774affd700 1 --2- 172.21.15.1:0/1233650319 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f773806ad20 0x7f773806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:09.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.026+0000 7f774affd700 1 --2- 172.21.15.1:0/1233650319 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f773806ad20 0x7f773806d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f774c0cb060 tx=0x7f7740007e70 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:09.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.031+0000 7f7733fff700 1 -- 172.21.15.1:0/1233650319 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f773c054270 con 0x7f774c0b2fa0 2024-03-18T20:07:09.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.233+0000 7f775212e700 1 -- 172.21.15.1:0/1233650319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f774c0676a0 con 0x7f774c0b2fa0 2024-03-18T20:07:09.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.871+0000 7f7733fff700 1 -- 172.21.15.1:0/1233650319 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f773c054640 con 0x7f774c0b2fa0 2024-03-18T20:07:09.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.874+0000 7f7733fff700 1 -- 172.21.15.1:0/1233650319 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v186) v1 ==== 169+0+0 (secure 0 0 0) 0x7f773c072f80 con 0x7f774c0b2fa0 2024-03-18T20:07:09.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.879+0000 7f775212e700 1 -- 172.21.15.1:0/1233650319 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f773806ad20 msgr2=0x7f773806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:09.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.879+0000 7f775212e700 1 --2- 172.21.15.1:0/1233650319 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f773806ad20 0x7f773806d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f774c0cb060 tx=0x7f7740007e70 comp rx=0 tx=0).stop 2024-03-18T20:07:09.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.879+0000 7f775212e700 1 -- 172.21.15.1:0/1233650319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f774c0b2fa0 msgr2=0x7f774c06bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:09.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.879+0000 7f775212e700 1 --2- 172.21.15.1:0/1233650319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f774c0b2fa0 0x7f774c06bf50 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f773c00e810 tx=0x7f773c00eb20 comp rx=0 tx=0).stop 2024-03-18T20:07:09.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.879+0000 7f775212e700 1 -- 172.21.15.1:0/1233650319 shutdown_connections 2024-03-18T20:07:09.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.879+0000 7f775212e700 1 --2- 172.21.15.1:0/1233650319 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f773806ad20 0x7f773806d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:09.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.879+0000 7f775212e700 1 --2- 172.21.15.1:0/1233650319 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f774c15fc10 0x7f774c1546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:09.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.879+0000 7f775212e700 1 --2- 172.21.15.1:0/1233650319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f774c0b2fa0 0x7f774c06bf50 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.879+0000 7f775212e700 1 --2- 172.21.15.1:0/1233650319 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f774c0d1dc0 0x7f774c0b6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.879+0000 7f775212e700 1 -- 172.21.15.1:0/1233650319 >> 172.21.15.1:0/1233650319 conn(0x7f774c06a750 msgr2=0x7f774c0c5da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.880+0000 7f775212e700 1 -- 172.21.15.1:0/1233650319 shutdown_connections 2024-03-18T20:07:09.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:09.880+0000 7f775212e700 1 -- 172.21.15.1:0/1233650319 wait complete. 2024-03-18T20:07:09.886 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-18T20:07:09.901 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-18T20:07:10.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.030+0000 7f4de91a6700 1 Processor -- start 2024-03-18T20:07:10.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.030+0000 7f4de91a6700 1 -- start start 2024-03-18T20:07:10.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de91a6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4de415a1d0 0x7f4de40add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de91a6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4de4071620 0x7f4de4071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de91a6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4de40cb9e0 0x7f4de40cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de91a6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de40cdb00 con 0x7f4de4071620 2024-03-18T20:07:10.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de91a6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de415a5b0 con 0x7f4de40cb9e0 2024-03-18T20:07:10.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de91a6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de4071fe0 con 0x7f4de415a1d0 2024-03-18T20:07:10.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de259c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4de4071620 0x7f4de4071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de259c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4de4071620 0x7f4de4071aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46512/0 (socket says 172.21.15.1:46512) 2024-03-18T20:07:10.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de259c700 1 -- 172.21.15.1:0/408585553 learned_addr learned my addr 172.21.15.1:0/408585553 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de2d9d700 1 --2- 172.21.15.1:0/408585553 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4de415a1d0 0x7f4de40add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de359e700 1 --2- 172.21.15.1:0/408585553 >> v2:172.21.15.165:3300/0 conn(0x7f4de40cb9e0 0x7f4de40cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de259c700 1 -- 172.21.15.1:0/408585553 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4de415a1d0 msgr2=0x7f4de40add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de259c700 1 --2- 172.21.15.1:0/408585553 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4de415a1d0 0x7f4de40add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de259c700 1 -- 172.21.15.1:0/408585553 >> v2:172.21.15.165:3300/0 conn(0x7f4de40cb9e0 msgr2=0x7f4de40cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de259c700 1 --2- 172.21.15.1:0/408585553 >> v2:172.21.15.165:3300/0 conn(0x7f4de40cb9e0 0x7f4de40cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de259c700 1 -- 172.21.15.1:0/408585553 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de40ce8f0 con 0x7f4de4071620 2024-03-18T20:07:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de259c700 1 --2- 172.21.15.1:0/408585553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4de4071620 0x7f4de4071aa0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f4de409df60 tx=0x7f4dd8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d048d6e37b6b2d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de1d9b700 1 -- 172.21.15.1:0/408585553 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd800b870 con 0x7f4de4071620 2024-03-18T20:07:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de1d9b700 1 -- 172.21.15.1:0/408585553 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd800b9d0 con 0x7f4de4071620 2024-03-18T20:07:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.031+0000 7f4de1d9b700 1 -- 172.21.15.1:0/408585553 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd800bd60 con 0x7f4de4071620 2024-03-18T20:07:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.032+0000 7f4de91a6700 1 -- 172.21.15.1:0/408585553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4de4071620 msgr2=0x7f4de4071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:10.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.032+0000 7f4de91a6700 1 --2- 172.21.15.1:0/408585553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4de4071620 0x7f4de4071aa0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f4de409df60 tx=0x7f4dd8009bd0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.032+0000 7f4de91a6700 1 -- 172.21.15.1:0/408585553 shutdown_connections 2024-03-18T20:07:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.032+0000 7f4de91a6700 1 --2- 172.21.15.1:0/408585553 >> v2:172.21.15.165:3300/0 conn(0x7f4de40cb9e0 0x7f4de40cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.032+0000 7f4de91a6700 1 --2- 172.21.15.1:0/408585553 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4de415a1d0 0x7f4de40add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.032+0000 7f4de91a6700 1 --2- 172.21.15.1:0/408585553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4de4071620 0x7f4de4071aa0 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.032+0000 7f4de91a6700 1 -- 172.21.15.1:0/408585553 >> 172.21.15.1:0/408585553 conn(0x7f4de40cc9b0 msgr2=0x7f4de40ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.032+0000 7f4de91a6700 1 -- 172.21.15.1:0/408585553 shutdown_connections 2024-03-18T20:07:10.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.032+0000 7f4de91a6700 1 -- 172.21.15.1:0/408585553 wait complete. 2024-03-18T20:07:10.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.033+0000 7f4de91a6700 1 Processor -- start 2024-03-18T20:07:10.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.034+0000 7f4de91a6700 1 -- start start 2024-03-18T20:07:10.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.034+0000 7f4de91a6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4de4071620 0x7f4de41569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.034+0000 7f4de91a6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4de40cb9e0 0x7f4de409caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.034+0000 7f4de91a6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4de415a1d0 0x7f4de40c64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.034+0000 7f4de91a6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de4099ec0 con 0x7f4de4071620 2024-03-18T20:07:10.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.034+0000 7f4de91a6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de409d9b0 con 0x7f4de415a1d0 2024-03-18T20:07:10.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.034+0000 7f4de91a6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de409d3f0 con 0x7f4de40cb9e0 2024-03-18T20:07:10.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.034+0000 7f4de2d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4de4071620 0x7f4de41569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.034+0000 7f4de2d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4de4071620 0x7f4de41569f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46524/0 (socket says 172.21.15.1:46524) 2024-03-18T20:07:10.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.034+0000 7f4de2d9d700 1 -- 172.21.15.1:0/3571685156 learned_addr learned my addr 172.21.15.1:0/3571685156 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:10.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.035+0000 7f4de259c700 1 --2- 172.21.15.1:0/3571685156 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4de40cb9e0 0x7f4de409caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.035+0000 7f4de359e700 1 --2- 172.21.15.1:0/3571685156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4de415a1d0 0x7f4de40c64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.035+0000 7f4de2d9d700 1 -- 172.21.15.1:0/3571685156 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4de40cb9e0 msgr2=0x7f4de409caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:10.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.035+0000 7f4de2d9d700 1 --2- 172.21.15.1:0/3571685156 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4de40cb9e0 0x7f4de409caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.035+0000 7f4de2d9d700 1 -- 172.21.15.1:0/3571685156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4de415a1d0 msgr2=0x7f4de40c64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:10.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.035+0000 7f4de2d9d700 1 --2- 172.21.15.1:0/3571685156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4de415a1d0 0x7f4de40c64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.035+0000 7f4de2d9d700 1 -- 172.21.15.1:0/3571685156 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd8009550 con 0x7f4de4071620 2024-03-18T20:07:10.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.035+0000 7f4de259c700 1 --2- 172.21.15.1:0/3571685156 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4de40cb9e0 0x7f4de409caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:10.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.035+0000 7f4de359e700 1 --2- 172.21.15.1:0/3571685156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4de415a1d0 0x7f4de40c64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:10.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.035+0000 7f4de2d9d700 1 --2- 172.21.15.1:0/3571685156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4de4071620 0x7f4de41569f0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f4dd400e760 tx=0x7f4dd400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:10.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.035+0000 7f4dcb7fe700 1 -- 172.21.15.1:0/3571685156 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd4013610 con 0x7f4de4071620 2024-03-18T20:07:10.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.035+0000 7f4dcb7fe700 1 -- 172.21.15.1:0/3571685156 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd4013770 con 0x7f4de4071620 2024-03-18T20:07:10.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.035+0000 7f4dcb7fe700 1 -- 172.21.15.1:0/3571685156 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd4013b20 con 0x7f4de4071620 2024-03-18T20:07:10.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.035+0000 7f4de91a6700 1 -- 172.21.15.1:0/3571685156 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de409d6d0 con 0x7f4de4071620 2024-03-18T20:07:10.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.035+0000 7f4de91a6700 1 -- 172.21.15.1:0/3571685156 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de4156f30 con 0x7f4de4071620 2024-03-18T20:07:10.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.036+0000 7f4de91a6700 1 -- 172.21.15.1:0/3571685156 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de4073350 con 0x7f4de4071620 2024-03-18T20:07:10.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.037+0000 7f4dcb7fe700 1 -- 172.21.15.1:0/3571685156 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4dd4013c80 con 0x7f4de4071620 2024-03-18T20:07:10.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.037+0000 7f4dcb7fe700 1 --2- 172.21.15.1:0/3571685156 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4dd006adb0 0x7f4dd006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.037+0000 7f4dcb7fe700 1 -- 172.21.15.1:0/3571685156 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f4dd4089100 con 0x7f4de4071620 2024-03-18T20:07:10.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.037+0000 7f4de259c700 1 --2- 172.21.15.1:0/3571685156 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4dd006adb0 0x7f4dd006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.038+0000 7f4de259c700 1 --2- 172.21.15.1:0/3571685156 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4dd006adb0 0x7f4dd006d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4de40c55c0 tx=0x7f4dd8008030 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:10.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.042+0000 7f4dcb7fe700 1 -- 172.21.15.1:0/3571685156 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4dd4053f50 con 0x7f4de4071620 2024-03-18T20:07:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.243+0000 7f4de91a6700 1 -- 172.21.15.1:0/3571685156 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f4de40cc380 con 0x7f4de4071620 2024-03-18T20:07:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.247+0000 7f4dcb7fe700 1 -- 172.21.15.1:0/3571685156 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v61) v1 ==== 115+0+0 (secure 0 0 0) 0x7f4dd4057570 con 0x7f4de4071620 2024-03-18T20:07:10.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.249+0000 7f4de91a6700 1 -- 172.21.15.1:0/3571685156 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4dd006adb0 msgr2=0x7f4dd006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:10.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.249+0000 7f4de91a6700 1 --2- 172.21.15.1:0/3571685156 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4dd006adb0 0x7f4dd006d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4de40c55c0 tx=0x7f4dd8008030 comp rx=0 tx=0).stop 2024-03-18T20:07:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.249+0000 7f4de91a6700 1 -- 172.21.15.1:0/3571685156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4de4071620 msgr2=0x7f4de41569f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.249+0000 7f4de91a6700 1 --2- 172.21.15.1:0/3571685156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4de4071620 0x7f4de41569f0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f4dd400e760 tx=0x7f4dd400eb20 comp rx=0 tx=0).stop 2024-03-18T20:07:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.249+0000 7f4de91a6700 1 -- 172.21.15.1:0/3571685156 shutdown_connections 2024-03-18T20:07:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.249+0000 7f4de91a6700 1 --2- 172.21.15.1:0/3571685156 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4dd006adb0 0x7f4dd006d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.249+0000 7f4de91a6700 1 --2- 172.21.15.1:0/3571685156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4de415a1d0 0x7f4de40c64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.249+0000 7f4de91a6700 1 --2- 172.21.15.1:0/3571685156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4de4071620 0x7f4de41569f0 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.249+0000 7f4de91a6700 1 --2- 172.21.15.1:0/3571685156 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4de40cb9e0 0x7f4de409caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.249+0000 7f4de91a6700 1 -- 172.21.15.1:0/3571685156 >> 172.21.15.1:0/3571685156 conn(0x7f4de40cc9b0 msgr2=0x7f4de40afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.249+0000 7f4de91a6700 1 -- 172.21.15.1:0/3571685156 shutdown_connections 2024-03-18T20:07:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.249+0000 7f4de91a6700 1 -- 172.21.15.1:0/3571685156 wait complete. 2024-03-18T20:07:10.267 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-18T20:07:10.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.391+0000 7fbdf4bd7700 1 Processor -- start 2024-03-18T20:07:10.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.391+0000 7fbdf4bd7700 1 -- start start 2024-03-18T20:07:10.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.392+0000 7fbdf4bd7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbdf015a1d0 0x7fbdf00add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.392+0000 7fbdf4bd7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdf00cc0f0 0x7fbdf0071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.392+0000 7fbdf4bd7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf00ce8f0 con 0x7fbdf0071620 2024-03-18T20:07:10.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.392+0000 7fbdf4bd7700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf00cdb00 con 0x7fbdf015a1d0 2024-03-18T20:07:10.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.392+0000 7fbdf4bd7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf015a5b0 con 0x7fbdf00cc0f0 2024-03-18T20:07:10.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.392+0000 7fbdee59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbdf015a1d0 0x7fbdf00add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.392+0000 7fbdedd9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbdf0071620 0x7fbdf00cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58710/0 (socket says 172.21.15.1:58710) 2024-03-18T20:07:10.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.392+0000 7fbdedd9b700 1 -- 172.21.15.1:0/919831001 learned_addr learned my addr 172.21.15.1:0/919831001 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:10.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.392+0000 7fbdeed9d700 1 --2- 172.21.15.1:0/919831001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdf00cc0f0 0x7fbdf0071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.392+0000 7fbdee59c700 1 -- 172.21.15.1:0/919831001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdf00cc0f0 msgr2=0x7fbdf0071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:10.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.392+0000 7fbdee59c700 1 --2- 172.21.15.1:0/919831001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdf00cc0f0 0x7fbdf0071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.392+0000 7fbdee59c700 1 -- 172.21.15.1:0/919831001 >> v1:172.21.15.1:6789/0 conn(0x7fbdf0071620 legacy=0x7fbdf00cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:10.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.392+0000 7fbdee59c700 1 -- 172.21.15.1:0/919831001 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf0071fc0 con 0x7fbdf015a1d0 2024-03-18T20:07:10.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.392+0000 7fbded59a700 1 -- 172.21.15.1:0/919831001 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506383531 0 0) 0x7fbdf00ce8f0 con 0x7fbdf0071620 2024-03-18T20:07:10.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.392+0000 7fbdee59c700 1 --2- 172.21.15.1:0/919831001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdf015a1d0 0x7fbdf00add20 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009820 tx=0x7fbde0009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9b6407331ad2bb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:10.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.393+0000 7fbded59a700 1 -- 172.21.15.1:0/919831001 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0005e30 con 0x7fbdf015a1d0 2024-03-18T20:07:10.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.393+0000 7fbded59a700 1 -- 172.21.15.1:0/919831001 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde0006030 con 0x7fbdf015a1d0 2024-03-18T20:07:10.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.393+0000 7fbded59a700 1 -- 172.21.15.1:0/919831001 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde00063e0 con 0x7fbdf015a1d0 2024-03-18T20:07:10.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.393+0000 7fbdf4bd7700 1 -- 172.21.15.1:0/919831001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdf015a1d0 msgr2=0x7fbdf00add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:10.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.393+0000 7fbdf4bd7700 1 --2- 172.21.15.1:0/919831001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdf015a1d0 0x7fbdf00add20 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009820 tx=0x7fbde0009b30 comp rx=0 tx=0).stop 2024-03-18T20:07:10.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.393+0000 7fbdf4bd7700 1 -- 172.21.15.1:0/919831001 shutdown_connections 2024-03-18T20:07:10.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.393+0000 7fbdf4bd7700 1 --2- 172.21.15.1:0/919831001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdf015a1d0 0x7fbdf00add20 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.393+0000 7fbdf4bd7700 1 --2- 172.21.15.1:0/919831001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdf00cc0f0 0x7fbdf0071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.393+0000 7fbdf4bd7700 1 -- 172.21.15.1:0/919831001 >> 172.21.15.1:0/919831001 conn(0x7fbdf00cc9b0 msgr2=0x7fbdf00ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:10.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.393+0000 7fbdf4bd7700 1 -- 172.21.15.1:0/919831001 shutdown_connections 2024-03-18T20:07:10.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.393+0000 7fbdf4bd7700 1 -- 172.21.15.1:0/919831001 wait complete. 2024-03-18T20:07:10.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.395+0000 7fbdf4bd7700 1 Processor -- start 2024-03-18T20:07:10.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.395+0000 7fbdf4bd7700 1 -- start start 2024-03-18T20:07:10.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.395+0000 7fbdf4bd7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdf0071620 0x7fbdf00c47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.395+0000 7fbdf4bd7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbdf00cc0f0 0x7fbdf014fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.395+0000 7fbdf4bd7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdf015a1d0 0x7fbdf006f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.395+0000 7fbdf4bd7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf015a700 con 0x7fbdf0071620 2024-03-18T20:07:10.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.395+0000 7fbdf4bd7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf006dfd0 con 0x7fbdf00cc0f0 2024-03-18T20:07:10.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.395+0000 7fbdf4bd7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf00c6130 con 0x7fbdf015a1d0 2024-03-18T20:07:10.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.396+0000 7fbdee59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdf0071620 0x7fbdf00c47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.396+0000 7fbdee59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdf0071620 0x7fbdf00c47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46542/0 (socket says 172.21.15.1:46542) 2024-03-18T20:07:10.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.396+0000 7fbdee59c700 1 -- 172.21.15.1:0/2785523426 learned_addr learned my addr 172.21.15.1:0/2785523426 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:10.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.396+0000 7fbdeed9d700 1 --2- 172.21.15.1:0/2785523426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdf015a1d0 0x7fbdf006f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.396+0000 7fbdedd9b700 1 --2- 172.21.15.1:0/2785523426 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbdf00cc0f0 0x7fbdf014fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.396+0000 7fbdee59c700 1 -- 172.21.15.1:0/2785523426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdf015a1d0 msgr2=0x7fbdf006f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:10.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.396+0000 7fbdee59c700 1 --2- 172.21.15.1:0/2785523426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdf015a1d0 0x7fbdf006f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.396+0000 7fbdee59c700 1 -- 172.21.15.1:0/2785523426 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbdf00cc0f0 msgr2=0x7fbdf014fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:10.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.396+0000 7fbdee59c700 1 --2- 172.21.15.1:0/2785523426 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbdf00cc0f0 0x7fbdf014fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.396+0000 7fbdee59c700 1 -- 172.21.15.1:0/2785523426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde0009430 con 0x7fbdf0071620 2024-03-18T20:07:10.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.396+0000 7fbdeed9d700 1 --2- 172.21.15.1:0/2785523426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdf015a1d0 0x7fbdf006f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:10.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.396+0000 7fbdee59c700 1 --2- 172.21.15.1:0/2785523426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdf0071620 0x7fbdf00c47b0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fbdf006cfd0 tx=0x7fbde0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:10.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.396+0000 7fbddeffd700 1 -- 172.21.15.1:0/2785523426 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0025630 con 0x7fbdf0071620 2024-03-18T20:07:10.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.396+0000 7fbddeffd700 1 -- 172.21.15.1:0/2785523426 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde0020070 con 0x7fbdf0071620 2024-03-18T20:07:10.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.396+0000 7fbdf4bd7700 1 -- 172.21.15.1:0/2785523426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf00c6410 con 0x7fbdf0071620 2024-03-18T20:07:10.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.396+0000 7fbddeffd700 1 -- 172.21.15.1:0/2785523426 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde00259f0 con 0x7fbdf0071620 2024-03-18T20:07:10.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.396+0000 7fbdf4bd7700 1 -- 172.21.15.1:0/2785523426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf01500b0 con 0x7fbdf0071620 2024-03-18T20:07:10.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.397+0000 7fbdf4bd7700 1 -- 172.21.15.1:0/2785523426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf006ddc0 con 0x7fbdf0071620 2024-03-18T20:07:10.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.398+0000 7fbddeffd700 1 -- 172.21.15.1:0/2785523426 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbde0022030 con 0x7fbdf0071620 2024-03-18T20:07:10.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.398+0000 7fbddeffd700 1 --2- 172.21.15.1:0/2785523426 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbdd006adb0 0x7fbdd006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.398+0000 7fbdedd9b700 1 --2- 172.21.15.1:0/2785523426 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbdd006adb0 0x7fbdd006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.398+0000 7fbddeffd700 1 -- 172.21.15.1:0/2785523426 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 5937+0+0 (secure 0 0 0) 0x7fbde008ac80 con 0x7fbdf0071620 2024-03-18T20:07:10.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.398+0000 7fbdedd9b700 1 --2- 172.21.15.1:0/2785523426 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbdd006adb0 0x7fbdd006d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbdf014e6f0 tx=0x7fbde4009040 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:10.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.403+0000 7fbddeffd700 1 -- 172.21.15.1:0/2785523426 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbdf006ddc0 con 0x7fbdf0071620 2024-03-18T20:07:10.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.603+0000 7fbdf4bd7700 1 -- 172.21.15.1:0/2785523426 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fbdf006ddc0 con 0x7fbdf0071620 2024-03-18T20:07:10.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.607+0000 7fbddeffd700 1 -- 172.21.15.1:0/2785523426 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v62) v1 ==== 120+0+0 (secure 0 0 0) 0x7fbdf006ddc0 con 0x7fbdf0071620 2024-03-18T20:07:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.609+0000 7fbdf4bd7700 1 -- 172.21.15.1:0/2785523426 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbdd006adb0 msgr2=0x7fbdd006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:10.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.609+0000 7fbdf4bd7700 1 --2- 172.21.15.1:0/2785523426 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbdd006adb0 0x7fbdd006d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbdf014e6f0 tx=0x7fbde4009040 comp rx=0 tx=0).stop 2024-03-18T20:07:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.609+0000 7fbdf4bd7700 1 -- 172.21.15.1:0/2785523426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdf0071620 msgr2=0x7fbdf00c47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.609+0000 7fbdf4bd7700 1 --2- 172.21.15.1:0/2785523426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdf0071620 0x7fbdf00c47b0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fbdf006cfd0 tx=0x7fbde0004750 comp rx=0 tx=0).stop 2024-03-18T20:07:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.609+0000 7fbdf4bd7700 1 -- 172.21.15.1:0/2785523426 shutdown_connections 2024-03-18T20:07:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.609+0000 7fbdf4bd7700 1 --2- 172.21.15.1:0/2785523426 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbdd006adb0 0x7fbdd006d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.609+0000 7fbdf4bd7700 1 --2- 172.21.15.1:0/2785523426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbdf015a1d0 0x7fbdf006f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.609+0000 7fbdf4bd7700 1 --2- 172.21.15.1:0/2785523426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbdf0071620 0x7fbdf00c47b0 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.609+0000 7fbdf4bd7700 1 --2- 172.21.15.1:0/2785523426 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbdf00cc0f0 0x7fbdf014fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.609+0000 7fbdf4bd7700 1 -- 172.21.15.1:0/2785523426 >> 172.21.15.1:0/2785523426 conn(0x7fbdf00cc9b0 msgr2=0x7fbdf0151340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.610+0000 7fbdf4bd7700 1 -- 172.21.15.1:0/2785523426 shutdown_connections 2024-03-18T20:07:10.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.610+0000 7fbdf4bd7700 1 -- 172.21.15.1:0/2785523426 wait complete. 2024-03-18T20:07:10.627 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-18T20:07:10.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.749+0000 7fa54876b700 1 Processor -- start 2024-03-18T20:07:10.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.749+0000 7fa54876b700 1 -- start start 2024-03-18T20:07:10.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.750+0000 7fa54876b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa5400ade00 0x7fa5400cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.750+0000 7fa54876b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5400ccc20 0x7fa54006fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.750+0000 7fa54876b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5400b3aa0 con 0x7fa540099a40 2024-03-18T20:07:10.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.750+0000 7fa54876b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5400ae280 con 0x7fa5400ade00 2024-03-18T20:07:10.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.750+0000 7fa54876b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54006e9b0 con 0x7fa5400ccc20 2024-03-18T20:07:10.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.750+0000 7fa546507700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa540099a40 0x7fa540099e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58718/0 (socket says 172.21.15.1:58718) 2024-03-18T20:07:10.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.750+0000 7fa546507700 1 -- 172.21.15.1:0/3249194478 learned_addr learned my addr 172.21.15.1:0/3249194478 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:10.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa546d08700 1 --2- 172.21.15.1:0/3249194478 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5400ccc20 0x7fa54006fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa545d06700 1 --2- 172.21.15.1:0/3249194478 >> v2:172.21.15.165:3300/0 conn(0x7fa5400ade00 0x7fa5400cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa545505700 1 -- 172.21.15.1:0/3249194478 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190531041 0 0) 0x7fa5400b3aa0 con 0x7fa540099a40 2024-03-18T20:07:10.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa545505700 1 -- 172.21.15.1:0/3249194478 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa52c0035b0 con 0x7fa540099a40 2024-03-18T20:07:10.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa545505700 1 -- 172.21.15.1:0/3249194478 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1078173302 0 0) 0x7fa52c0035b0 con 0x7fa540099a40 2024-03-18T20:07:10.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa545505700 1 -- 172.21.15.1:0/3249194478 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5400ccc20 msgr2=0x7fa54006fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:10.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa545505700 1 --2- 172.21.15.1:0/3249194478 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5400ccc20 0x7fa54006fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa545505700 1 -- 172.21.15.1:0/3249194478 >> v2:172.21.15.165:3300/0 conn(0x7fa5400ade00 msgr2=0x7fa5400cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:10.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa545505700 1 --2- 172.21.15.1:0/3249194478 >> v2:172.21.15.165:3300/0 conn(0x7fa5400ade00 0x7fa5400cc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa545505700 1 -- 172.21.15.1:0/3249194478 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54006ec90 con 0x7fa540099a40 2024-03-18T20:07:10.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa545505700 1 -- 172.21.15.1:0/3249194478 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fa530001670 con 0x7fa540099a40 2024-03-18T20:07:10.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa545505700 1 -- 172.21.15.1:0/3249194478 >> v1:172.21.15.1:6789/0 conn(0x7fa540099a40 legacy=0x7fa540099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:10.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa545505700 1 --2- 172.21.15.1:0/3249194478 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5400ccc20 0x7fa52c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa545505700 1 --2- 172.21.15.1:0/3249194478 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa52c003f80 0x7fa52c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa545505700 1 --2- 172.21.15.1:0/3249194478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa52c004d90 0x7fa52c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa545505700 1 -- 172.21.15.1:0/3249194478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400ae280 con 0x7fa52c004d90 2024-03-18T20:07:10.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa545505700 1 -- 172.21.15.1:0/3249194478 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54006e9b0 con 0x7fa52c003f80 2024-03-18T20:07:10.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa545505700 1 -- 172.21.15.1:0/3249194478 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c00ba60 con 0x7fa5400ccc20 2024-03-18T20:07:10.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa546d08700 1 --2- 172.21.15.1:0/3249194478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa52c004d90 0x7fa52c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa546507700 1 --2- 172.21.15.1:0/3249194478 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5400ccc20 0x7fa52c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa545d06700 1 --2- 172.21.15.1:0/3249194478 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa52c003f80 0x7fa52c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa546d08700 1 -- 172.21.15.1:0/3249194478 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5400ccc20 msgr2=0x7fa52c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:10.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa546d08700 1 --2- 172.21.15.1:0/3249194478 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5400ccc20 0x7fa52c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa546d08700 1 -- 172.21.15.1:0/3249194478 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa52c003f80 msgr2=0x7fa52c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:10.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa546d08700 1 --2- 172.21.15.1:0/3249194478 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa52c003f80 0x7fa52c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.751+0000 7fa546d08700 1 -- 172.21.15.1:0/3249194478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52c0035b0 con 0x7fa52c004d90 2024-03-18T20:07:10.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.752+0000 7fa546d08700 1 --2- 172.21.15.1:0/3249194478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa52c004d90 0x7fa52c005210 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7fa53800e170 tx=0x7fa52c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=795ad27d8df8d580 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:10.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.752+0000 7fa545505700 1 -- 172.21.15.1:0/3249194478 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53800ec90 con 0x7fa52c004d90 2024-03-18T20:07:10.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.752+0000 7fa545505700 1 -- 172.21.15.1:0/3249194478 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa53800f040 con 0x7fa52c004d90 2024-03-18T20:07:10.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.752+0000 7fa545505700 1 -- 172.21.15.1:0/3249194478 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa538014480 con 0x7fa52c004d90 2024-03-18T20:07:10.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.752+0000 7fa54876b700 1 -- 172.21.15.1:0/3249194478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa52c004d90 msgr2=0x7fa52c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:10.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.752+0000 7fa54876b700 1 --2- 172.21.15.1:0/3249194478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa52c004d90 0x7fa52c005210 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7fa53800e170 tx=0x7fa52c005880 comp rx=0 tx=0).stop 2024-03-18T20:07:10.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.752+0000 7fa546d08700 1 -- 172.21.15.1:0/3249194478 reap_dead start 2024-03-18T20:07:10.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.752+0000 7fa54876b700 1 -- 172.21.15.1:0/3249194478 shutdown_connections 2024-03-18T20:07:10.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.752+0000 7fa54876b700 1 -- 172.21.15.1:0/3249194478 >> 172.21.15.1:0/3249194478 conn(0x7fa5400b2c50 msgr2=0x7fa5400b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:10.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.752+0000 7fa54876b700 1 -- 172.21.15.1:0/3249194478 shutdown_connections 2024-03-18T20:07:10.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.752+0000 7fa54876b700 1 -- 172.21.15.1:0/3249194478 wait complete. 2024-03-18T20:07:10.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.754+0000 7fa54876b700 1 Processor -- start 2024-03-18T20:07:10.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.754+0000 7fa54876b700 1 -- start start 2024-03-18T20:07:10.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.754+0000 7fa54876b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa52c004d90 0x7fa54015a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.754+0000 7fa54876b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5400ca790 0x7fa5400cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.754+0000 7fa54876b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5400c4c70 0x7fa5400c5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.754+0000 7fa54876b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400c5660 con 0x7fa5400c4c70 2024-03-18T20:07:10.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.754+0000 7fa54876b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400c9290 con 0x7fa52c004d90 2024-03-18T20:07:10.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.754+0000 7fa54876b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400c9570 con 0x7fa5400ca790 2024-03-18T20:07:10.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.754+0000 7fa546d08700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5400c4c70 0x7fa5400c5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.754+0000 7fa546d08700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5400c4c70 0x7fa5400c5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46566/0 (socket says 172.21.15.1:46566) 2024-03-18T20:07:10.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.754+0000 7fa546d08700 1 -- 172.21.15.1:0/813188125 learned_addr learned my addr 172.21.15.1:0/813188125 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:10.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.755+0000 7fa546507700 1 --2- 172.21.15.1:0/813188125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa52c004d90 0x7fa54015a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.755+0000 7fa545d06700 1 --2- 172.21.15.1:0/813188125 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5400ca790 0x7fa5400cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.755+0000 7fa546d08700 1 -- 172.21.15.1:0/813188125 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5400ca790 msgr2=0x7fa5400cac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:10.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.755+0000 7fa546d08700 1 --2- 172.21.15.1:0/813188125 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5400ca790 0x7fa5400cac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.755+0000 7fa546d08700 1 -- 172.21.15.1:0/813188125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa52c004d90 msgr2=0x7fa54015a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:10.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.755+0000 7fa546d08700 1 --2- 172.21.15.1:0/813188125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa52c004d90 0x7fa54015a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.755+0000 7fa546d08700 1 -- 172.21.15.1:0/813188125 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5400c9850 con 0x7fa5400c4c70 2024-03-18T20:07:10.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.755+0000 7fa546507700 1 --2- 172.21.15.1:0/813188125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa52c004d90 0x7fa54015a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:07:10.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.755+0000 7fa545d06700 1 --2- 172.21.15.1:0/813188125 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5400ca790 0x7fa5400cac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:07:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.755+0000 7fa546d08700 1 --2- 172.21.15.1:0/813188125 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5400c4c70 0x7fa5400c5120 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fa52c004790 tx=0x7fa538009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.755+0000 7fa536ffd700 1 -- 172.21.15.1:0/813188125 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53800ee80 con 0x7fa5400c4c70 2024-03-18T20:07:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.755+0000 7fa536ffd700 1 -- 172.21.15.1:0/813188125 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa538004250 con 0x7fa5400c4c70 2024-03-18T20:07:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.755+0000 7fa536ffd700 1 -- 172.21.15.1:0/813188125 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa538004600 con 0x7fa5400c4c70 2024-03-18T20:07:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.755+0000 7fa54876b700 1 -- 172.21.15.1:0/813188125 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5400c9b30 con 0x7fa5400c4c70 2024-03-18T20:07:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.755+0000 7fa54876b700 1 -- 172.21.15.1:0/813188125 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54015a780 con 0x7fa5400c4c70 2024-03-18T20:07:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.756+0000 7fa54876b700 1 -- 172.21.15.1:0/813188125 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5400676a0 con 0x7fa5400c4c70 2024-03-18T20:07:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.756+0000 7fa536ffd700 1 -- 172.21.15.1:0/813188125 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa538024020 con 0x7fa5400c4c70 2024-03-18T20:07:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.757+0000 7fa536ffd700 1 --2- 172.21.15.1:0/813188125 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa52406adb0 0x7fa52406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.757+0000 7fa546507700 1 --2- 172.21.15.1:0/813188125 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa52406adb0 0x7fa52406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.757+0000 7fa536ffd700 1 -- 172.21.15.1:0/813188125 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 5937+0+0 (secure 0 0 0) 0x7fa53808ae40 con 0x7fa5400c4c70 2024-03-18T20:07:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.757+0000 7fa546507700 1 --2- 172.21.15.1:0/813188125 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa52406adb0 0x7fa52406d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa530000fb0 tx=0x7fa530009040 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:10.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.762+0000 7fa536ffd700 1 -- 172.21.15.1:0/813188125 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa538055fd0 con 0x7fa5400c4c70 2024-03-18T20:07:10.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.969+0000 7fa54876b700 1 -- 172.21.15.1:0/813188125 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa5400cd950 con 0x7fa5400c4c70 2024-03-18T20:07:10.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.972+0000 7fa536ffd700 1 -- 172.21.15.1:0/813188125 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v63) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa5380595f0 con 0x7fa5400c4c70 2024-03-18T20:07:10.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.974+0000 7fa54876b700 1 -- 172.21.15.1:0/813188125 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa52406adb0 msgr2=0x7fa52406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:10.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.974+0000 7fa54876b700 1 --2- 172.21.15.1:0/813188125 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa52406adb0 0x7fa52406d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa530000fb0 tx=0x7fa530009040 comp rx=0 tx=0).stop 2024-03-18T20:07:10.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.975+0000 7fa54876b700 1 -- 172.21.15.1:0/813188125 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5400c4c70 msgr2=0x7fa5400c5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:10.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.975+0000 7fa54876b700 1 --2- 172.21.15.1:0/813188125 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5400c4c70 0x7fa5400c5120 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fa52c004790 tx=0x7fa538009750 comp rx=0 tx=0).stop 2024-03-18T20:07:10.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.975+0000 7fa54876b700 1 -- 172.21.15.1:0/813188125 shutdown_connections 2024-03-18T20:07:10.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.975+0000 7fa54876b700 1 --2- 172.21.15.1:0/813188125 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fa52406adb0 0x7fa52406d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.975+0000 7fa54876b700 1 --2- 172.21.15.1:0/813188125 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5400c4c70 0x7fa5400c5120 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.975+0000 7fa54876b700 1 --2- 172.21.15.1:0/813188125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa52c004d90 0x7fa54015a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.975+0000 7fa54876b700 1 --2- 172.21.15.1:0/813188125 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5400ca790 0x7fa5400cac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:10.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.975+0000 7fa54876b700 1 -- 172.21.15.1:0/813188125 >> 172.21.15.1:0/813188125 conn(0x7fa5400b2c50 msgr2=0x7fa5400ae9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:10.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.975+0000 7fa54876b700 1 -- 172.21.15.1:0/813188125 shutdown_connections 2024-03-18T20:07:10.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:10.975+0000 7fa54876b700 1 -- 172.21.15.1:0/813188125 wait complete. 2024-03-18T20:07:10.991 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2024-03-18T20:07:11.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.112+0000 7f0fdef79700 1 Processor -- start 2024-03-18T20:07:11.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.112+0000 7f0fdef79700 1 -- start start 2024-03-18T20:07:11.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.113+0000 7f0fdef79700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0fd80cee90 0x7f0fd80cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:11.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.113+0000 7f0fdef79700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd80cd0d0 con 0x7f0fd80cfbd0 2024-03-18T20:07:11.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.113+0000 7f0fdef79700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd80b3f00 con 0x7f0fd80b6b60 2024-03-18T20:07:11.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.113+0000 7f0fdef79700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd80d0040 con 0x7f0fd80cee90 2024-03-18T20:07:11.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.113+0000 7f0fdd516700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0fd80cfbd0 0x7f0fd80b4cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58726/0 (socket says 172.21.15.1:58726) 2024-03-18T20:07:11.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.113+0000 7f0fdd516700 1 -- 172.21.15.1:0/4115621088 learned_addr learned my addr 172.21.15.1:0/4115621088 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:11.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.113+0000 7f0fd7fff700 1 --2- 172.21.15.1:0/4115621088 >> v2:172.21.15.165:3300/0 conn(0x7f0fd80cee90 0x7f0fd80cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:11.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd77fe700 1 -- 172.21.15.1:0/4115621088 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570197796 0 0) 0x7f0fd80cd0d0 con 0x7f0fd80cfbd0 2024-03-18T20:07:11.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd77fe700 1 -- 172.21.15.1:0/4115621088 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fc80035b0 con 0x7f0fd80cfbd0 2024-03-18T20:07:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd77fe700 1 -- 172.21.15.1:0/4115621088 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 154542426 0 0) 0x7f0fc80035b0 con 0x7f0fd80cfbd0 2024-03-18T20:07:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd77fe700 1 -- 172.21.15.1:0/4115621088 >> v2:172.21.15.165:3300/0 conn(0x7f0fd80cee90 msgr2=0x7f0fd80cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd77fe700 1 --2- 172.21.15.1:0/4115621088 >> v2:172.21.15.165:3300/0 conn(0x7f0fd80cee90 0x7f0fd80cf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd77fe700 1 -- 172.21.15.1:0/4115621088 >> v1:172.21.15.165:6789/0 conn(0x7f0fd80b6b60 legacy=0x7f0fd80cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd77fe700 1 -- 172.21.15.1:0/4115621088 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd80b41e0 con 0x7f0fd80cfbd0 2024-03-18T20:07:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd77fe700 1 -- 172.21.15.1:0/4115621088 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f0fd00019f0 con 0x7f0fd80cfbd0 2024-03-18T20:07:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd77fe700 1 -- 172.21.15.1:0/4115621088 >> v1:172.21.15.1:6789/0 conn(0x7f0fd80cfbd0 legacy=0x7f0fd80b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd77fe700 1 --2- 172.21.15.1:0/4115621088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fc8003a00 0x7f0fc8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd77fe700 1 --2- 172.21.15.1:0/4115621088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc8004850 0x7f0fc8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd77fe700 1 --2- 172.21.15.1:0/4115621088 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fc8005210 0x7f0fc800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:11.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd77fe700 1 -- 172.21.15.1:0/4115621088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80d0040 con 0x7f0fc8004850 2024-03-18T20:07:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd77fe700 1 -- 172.21.15.1:0/4115621088 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc800c0b0 con 0x7f0fc8003a00 2024-03-18T20:07:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd77fe700 1 -- 172.21.15.1:0/4115621088 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc800c270 con 0x7f0fc8005210 2024-03-18T20:07:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd7fff700 1 --2- 172.21.15.1:0/4115621088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc8004850 0x7f0fc8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fdcd15700 1 --2- 172.21.15.1:0/4115621088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fc8003a00 0x7f0fc8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fdd516700 1 --2- 172.21.15.1:0/4115621088 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fc8005210 0x7f0fc800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd7fff700 1 -- 172.21.15.1:0/4115621088 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fc8005210 msgr2=0x7f0fc800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd7fff700 1 --2- 172.21.15.1:0/4115621088 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fc8005210 0x7f0fc800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd7fff700 1 -- 172.21.15.1:0/4115621088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fc8003a00 msgr2=0x7f0fc8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd7fff700 1 --2- 172.21.15.1:0/4115621088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fc8003a00 0x7f0fc8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd7fff700 1 -- 172.21.15.1:0/4115621088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc80035b0 con 0x7f0fc8004850 2024-03-18T20:07:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fdd516700 1 -- 172.21.15.1:0/4115621088 reap_dead start 2024-03-18T20:07:11.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.114+0000 7f0fd7fff700 1 --2- 172.21.15.1:0/4115621088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc8004850 0x7f0fc8004cd0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009b60 tx=0x7f0fc8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc522888faa8608 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.115+0000 7f0fd77fe700 1 -- 172.21.15.1:0/4115621088 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc400e6c0 con 0x7f0fc8004850 2024-03-18T20:07:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.115+0000 7f0fd77fe700 1 -- 172.21.15.1:0/4115621088 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc400e820 con 0x7f0fc8004850 2024-03-18T20:07:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.115+0000 7f0fd77fe700 1 -- 172.21.15.1:0/4115621088 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc400eb90 con 0x7f0fc8004850 2024-03-18T20:07:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.115+0000 7f0fdef79700 1 -- 172.21.15.1:0/4115621088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc8004850 msgr2=0x7f0fc8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.115+0000 7f0fdef79700 1 --2- 172.21.15.1:0/4115621088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc8004850 0x7f0fc8004cd0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009b60 tx=0x7f0fc8005880 comp rx=0 tx=0).stop 2024-03-18T20:07:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.115+0000 7f0fdef79700 1 -- 172.21.15.1:0/4115621088 shutdown_connections 2024-03-18T20:07:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.115+0000 7f0fdef79700 1 --2- 172.21.15.1:0/4115621088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc8004850 0x7f0fc8004cd0 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.115+0000 7f0fdef79700 1 -- 172.21.15.1:0/4115621088 >> 172.21.15.1:0/4115621088 conn(0x7f0fd80b2ba0 msgr2=0x7f0fd80b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.115+0000 7f0fdef79700 1 -- 172.21.15.1:0/4115621088 shutdown_connections 2024-03-18T20:07:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.115+0000 7f0fdef79700 1 -- 172.21.15.1:0/4115621088 wait complete. 2024-03-18T20:07:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.116+0000 7f0fdef79700 1 Processor -- start 2024-03-18T20:07:11.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.116+0000 7f0fdef79700 1 -- start start 2024-03-18T20:07:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.117+0000 7f0fdef79700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fc8004850 0x7f0fd81504d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.117+0000 7f0fdef79700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fd80c93c0 0x7f0fd8150d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.117+0000 7f0fdef79700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fd80c9840 0x7f0fd81512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.117+0000 7f0fdef79700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80c8f30 con 0x7f0fd80c93c0 2024-03-18T20:07:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.117+0000 7f0fdef79700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80c8320 con 0x7f0fc8004850 2024-03-18T20:07:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.117+0000 7f0fdef79700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80c8600 con 0x7f0fd80c9840 2024-03-18T20:07:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.117+0000 7f0fd7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fd80c93c0 0x7f0fd8150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.117+0000 7f0fd7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fd80c93c0 0x7f0fd8150d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46584/0 (socket says 172.21.15.1:46584) 2024-03-18T20:07:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.117+0000 7f0fd7fff700 1 -- 172.21.15.1:0/1483319531 learned_addr learned my addr 172.21.15.1:0/1483319531 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.118+0000 7f0fdcd15700 1 --2- 172.21.15.1:0/1483319531 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fc8004850 0x7f0fd81504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:11.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.118+0000 7f0fdd516700 1 --2- 172.21.15.1:0/1483319531 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fd80c9840 0x7f0fd81512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:11.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.118+0000 7f0fd7fff700 1 -- 172.21.15.1:0/1483319531 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fd80c9840 msgr2=0x7f0fd81512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:11.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.118+0000 7f0fd7fff700 1 --2- 172.21.15.1:0/1483319531 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fd80c9840 0x7f0fd81512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:11.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.118+0000 7f0fd7fff700 1 -- 172.21.15.1:0/1483319531 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fc8004850 msgr2=0x7f0fd81504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:11.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.118+0000 7f0fd7fff700 1 --2- 172.21.15.1:0/1483319531 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fc8004850 0x7f0fd81504d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:11.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.118+0000 7f0fd7fff700 1 -- 172.21.15.1:0/1483319531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd80c88e0 con 0x7f0fd80c93c0 2024-03-18T20:07:11.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.118+0000 7f0fdcd15700 1 --2- 172.21.15.1:0/1483319531 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fc8004850 0x7f0fd81504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:11.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.118+0000 7f0fdd516700 1 --2- 172.21.15.1:0/1483319531 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fd80c9840 0x7f0fd81512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:11.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.118+0000 7f0fd7fff700 1 --2- 172.21.15.1:0/1483319531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fd80c93c0 0x7f0fd8150d60 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f0fc800c080 tx=0x7f0fc40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:11.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.118+0000 7f0fd57fa700 1 -- 172.21.15.1:0/1483319531 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc400e6c0 con 0x7f0fd80c93c0 2024-03-18T20:07:11.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.118+0000 7f0fd57fa700 1 -- 172.21.15.1:0/1483319531 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc4003ad0 con 0x7f0fd80c93c0 2024-03-18T20:07:11.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.118+0000 7f0fdef79700 1 -- 172.21.15.1:0/1483319531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd80c8bc0 con 0x7f0fd80c93c0 2024-03-18T20:07:11.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.118+0000 7f0fd57fa700 1 -- 172.21.15.1:0/1483319531 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4003e80 con 0x7f0fd80c93c0 2024-03-18T20:07:11.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.118+0000 7f0fdef79700 1 -- 172.21.15.1:0/1483319531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd815aad0 con 0x7f0fd80c93c0 2024-03-18T20:07:11.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.119+0000 7f0fdef79700 1 -- 172.21.15.1:0/1483319531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd80c8d50 con 0x7f0fd80c93c0 2024-03-18T20:07:11.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.120+0000 7f0fd57fa700 1 -- 172.21.15.1:0/1483319531 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0fc4013470 con 0x7f0fd80c93c0 2024-03-18T20:07:11.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.120+0000 7f0fd57fa700 1 --2- 172.21.15.1:0/1483319531 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fbc06ade0 0x7f0fbc06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:11.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.120+0000 7f0fd57fa700 1 -- 172.21.15.1:0/1483319531 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f0fc408ae20 con 0x7f0fd80c93c0 2024-03-18T20:07:11.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.120+0000 7f0fdcd15700 1 --2- 172.21.15.1:0/1483319531 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fbc06ade0 0x7f0fbc06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:11.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.120+0000 7f0fdcd15700 1 --2- 172.21.15.1:0/1483319531 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fbc06ade0 0x7f0fbc06d2a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc000c00 tx=0x7f0fcc009040 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:11.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.125+0000 7f0fd57fa700 1 -- 172.21.15.1:0/1483319531 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0fc4055fb0 con 0x7f0fd80c93c0 2024-03-18T20:07:11.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.413+0000 7f0fdef79700 1 -- 172.21.15.1:0/1483319531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f0fd80676a0 con 0x7f0fd80c93c0 2024-03-18T20:07:11.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.869+0000 7f0fd57fa700 1 -- 172.21.15.1:0/1483319531 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2523) v1 ==== 174+0+0 (secure 0 0 0) 0x7f0fc4017020 con 0x7f0fd80c93c0 2024-03-18T20:07:11.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.872+0000 7f0fdef79700 1 -- 172.21.15.1:0/1483319531 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fbc06ade0 msgr2=0x7f0fbc06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:11.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.872+0000 7f0fdef79700 1 --2- 172.21.15.1:0/1483319531 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fbc06ade0 0x7f0fbc06d2a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc000c00 tx=0x7f0fcc009040 comp rx=0 tx=0).stop 2024-03-18T20:07:11.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.872+0000 7f0fdef79700 1 -- 172.21.15.1:0/1483319531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fd80c93c0 msgr2=0x7f0fd8150d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:11.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.872+0000 7f0fdef79700 1 --2- 172.21.15.1:0/1483319531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fd80c93c0 0x7f0fd8150d60 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f0fc800c080 tx=0x7f0fc40047c0 comp rx=0 tx=0).stop 2024-03-18T20:07:11.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.872+0000 7f0fdef79700 1 -- 172.21.15.1:0/1483319531 shutdown_connections 2024-03-18T20:07:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.872+0000 7f0fdef79700 1 --2- 172.21.15.1:0/1483319531 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0fbc06ade0 0x7f0fbc06d2a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.872+0000 7f0fdef79700 1 --2- 172.21.15.1:0/1483319531 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0fd80c9840 0x7f0fd81512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.872+0000 7f0fdef79700 1 --2- 172.21.15.1:0/1483319531 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0fc8004850 0x7f0fd81504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.872+0000 7f0fdef79700 1 --2- 172.21.15.1:0/1483319531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fd80c93c0 0x7f0fd8150d60 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.872+0000 7f0fdef79700 1 -- 172.21.15.1:0/1483319531 >> 172.21.15.1:0/1483319531 conn(0x7f0fd80b2ba0 msgr2=0x7f0fd80cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.872+0000 7f0fdef79700 1 -- 172.21.15.1:0/1483319531 shutdown_connections 2024-03-18T20:07:11.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:11.872+0000 7f0fdef79700 1 -- 172.21.15.1:0/1483319531 wait complete. 2024-03-18T20:07:11.892 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2024-03-18T20:07:11.892 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2024-03-18T20:07:11.892 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-18T20:07:11.892 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2024-03-18T20:07:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.019+0000 7f867366a700 1 Processor -- start 2024-03-18T20:07:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.019+0000 7f867366a700 1 -- start start 2024-03-18T20:07:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.020+0000 7f867366a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f866c143c20 0x7f866c0b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.020+0000 7f867366a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f866c06f240 0x7f866c0cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.020+0000 7f867366a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866c144000 con 0x7f866c0b3f40 2024-03-18T20:07:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.020+0000 7f867366a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f866c0b43c0 con 0x7f866c06f240 2024-03-18T20:07:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.020+0000 7f867366a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f866c0b51b0 con 0x7f866c143c20 2024-03-18T20:07:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.020+0000 7f8671c07700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f866c06f240 0x7f866c0cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.020+0000 7f8670c05700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f866c0b3f40 0x7f866c06eb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58732/0 (socket says 172.21.15.1:58732) 2024-03-18T20:07:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.020+0000 7f8670c05700 1 -- 172.21.15.1:0/2428170893 learned_addr learned my addr 172.21.15.1:0/2428170893 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:12.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.020+0000 7f8671406700 1 --2- 172.21.15.1:0/2428170893 >> v2:172.21.15.165:3300/0 conn(0x7f866c143c20 0x7f866c0b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.020+0000 7f8671c07700 1 -- 172.21.15.1:0/2428170893 >> v2:172.21.15.165:3300/0 conn(0x7f866c143c20 msgr2=0x7f866c0b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.020+0000 7f8671c07700 1 --2- 172.21.15.1:0/2428170893 >> v2:172.21.15.165:3300/0 conn(0x7f866c143c20 0x7f866c0b3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.020+0000 7f8671c07700 1 -- 172.21.15.1:0/2428170893 >> v1:172.21.15.1:6789/0 conn(0x7f866c0b3f40 legacy=0x7f866c06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.020+0000 7f8671c07700 1 -- 172.21.15.1:0/2428170893 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f866c0cd8d0 con 0x7f866c06f240 2024-03-18T20:07:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.020+0000 7f8671406700 1 --2- 172.21.15.1:0/2428170893 >> v2:172.21.15.165:3300/0 conn(0x7f866c143c20 0x7f866c0b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:07:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.021+0000 7f8671c07700 1 --2- 172.21.15.1:0/2428170893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866c06f240 0x7f866c0cddb0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f866c06d080 tx=0x7f866400d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad68ae9bb19eca83 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.021+0000 7f8663fff700 1 -- 172.21.15.1:0/2428170893 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8664010070 con 0x7f866c06f240 2024-03-18T20:07:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.021+0000 7f8663fff700 1 -- 172.21.15.1:0/2428170893 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866400b2e0 con 0x7f866c06f240 2024-03-18T20:07:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.021+0000 7f8663fff700 1 -- 172.21.15.1:0/2428170893 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866400dcc0 con 0x7f866c06f240 2024-03-18T20:07:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.021+0000 7f867366a700 1 -- 172.21.15.1:0/2428170893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866c06f240 msgr2=0x7f866c0cddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.021+0000 7f867366a700 1 --2- 172.21.15.1:0/2428170893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866c06f240 0x7f866c0cddb0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f866c06d080 tx=0x7f866400d330 comp rx=0 tx=0).stop 2024-03-18T20:07:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.021+0000 7f867366a700 1 -- 172.21.15.1:0/2428170893 shutdown_connections 2024-03-18T20:07:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.021+0000 7f867366a700 1 --2- 172.21.15.1:0/2428170893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866c06f240 0x7f866c0cddb0 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:12.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.021+0000 7f867366a700 1 --2- 172.21.15.1:0/2428170893 >> v2:172.21.15.165:3300/0 conn(0x7f866c143c20 0x7f866c0b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:12.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.021+0000 7f867366a700 1 -- 172.21.15.1:0/2428170893 >> 172.21.15.1:0/2428170893 conn(0x7f866c0b1d20 msgr2=0x7f866c0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:12.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.021+0000 7f867366a700 1 -- 172.21.15.1:0/2428170893 shutdown_connections 2024-03-18T20:07:12.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.021+0000 7f867366a700 1 -- 172.21.15.1:0/2428170893 wait complete. 2024-03-18T20:07:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.023+0000 7f867366a700 1 Processor -- start 2024-03-18T20:07:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.023+0000 7f867366a700 1 -- start start 2024-03-18T20:07:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.023+0000 7f867366a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f866c06f240 0x7f866c1f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.023+0000 7f867366a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866c0b3f40 0x7f866c1e2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.023+0000 7f867366a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f866c143c20 0x7f866c0aa550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.023+0000 7f867366a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866c1f4000 con 0x7f866c0b3f40 2024-03-18T20:07:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.023+0000 7f867366a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866c0a9870 con 0x7f866c143c20 2024-03-18T20:07:12.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.023+0000 7f867366a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866c1e2b50 con 0x7f866c06f240 2024-03-18T20:07:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.024+0000 7f8670c05700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866c0b3f40 0x7f866c1e2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.024+0000 7f8670c05700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866c0b3f40 0x7f866c1e2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46604/0 (socket says 172.21.15.1:46604) 2024-03-18T20:07:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.024+0000 7f8670c05700 1 -- 172.21.15.1:0/3518024909 learned_addr learned my addr 172.21.15.1:0/3518024909 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.024+0000 7f8671406700 1 --2- 172.21.15.1:0/3518024909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f866c06f240 0x7f866c1f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.024+0000 7f8670c05700 1 -- 172.21.15.1:0/3518024909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f866c06f240 msgr2=0x7f866c1f3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.024+0000 7f8670c05700 1 --2- 172.21.15.1:0/3518024909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f866c06f240 0x7f866c1f3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.024+0000 7f8670c05700 1 -- 172.21.15.1:0/3518024909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f866c143c20 msgr2=0x7f866c0aa550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.024+0000 7f8671c07700 1 --2- 172.21.15.1:0/3518024909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f866c143c20 0x7f866c0aa550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.024+0000 7f8670c05700 1 --2- 172.21.15.1:0/3518024909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f866c143c20 0x7f866c0aa550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.024+0000 7f8670c05700 1 -- 172.21.15.1:0/3518024909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f866400bc80 con 0x7f866c0b3f40 2024-03-18T20:07:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.024+0000 7f8671406700 1 --2- 172.21.15.1:0/3518024909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f866c06f240 0x7f866c1f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.024+0000 7f8670c05700 1 --2- 172.21.15.1:0/3518024909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866c0b3f40 0x7f866c1e2610 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f866800c500 tx=0x7f866800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.024+0000 7f8661ffb700 1 -- 172.21.15.1:0/3518024909 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8668005d70 con 0x7f866c0b3f40 2024-03-18T20:07:12.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.024+0000 7f8661ffb700 1 -- 172.21.15.1:0/3518024909 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86680070d0 con 0x7f866c0b3f40 2024-03-18T20:07:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.024+0000 7f867366a700 1 -- 172.21.15.1:0/3518024909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f866c0a8c60 con 0x7f866c0b3f40 2024-03-18T20:07:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.024+0000 7f8661ffb700 1 -- 172.21.15.1:0/3518024909 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8668007440 con 0x7f866c0b3f40 2024-03-18T20:07:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.024+0000 7f867366a700 1 -- 172.21.15.1:0/3518024909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f866c0a9220 con 0x7f866c0b3f40 2024-03-18T20:07:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.025+0000 7f867366a700 1 -- 172.21.15.1:0/3518024909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f866c0676a0 con 0x7f866c0b3f40 2024-03-18T20:07:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.026+0000 7f8661ffb700 1 -- 172.21.15.1:0/3518024909 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f86680075a0 con 0x7f866c0b3f40 2024-03-18T20:07:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.026+0000 7f8661ffb700 1 --2- 172.21.15.1:0/3518024909 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f865006adb0 0x7f865006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.026+0000 7f8661ffb700 1 -- 172.21.15.1:0/3518024909 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f8668088990 con 0x7f866c0b3f40 2024-03-18T20:07:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.026+0000 7f8671406700 1 --2- 172.21.15.1:0/3518024909 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f865006adb0 0x7f865006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:12.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.027+0000 7f8671406700 1 --2- 172.21.15.1:0/3518024909 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f865006adb0 0x7f865006d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f865c009820 tx=0x7f865c008040 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:12.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.031+0000 7f8661ffb700 1 -- 172.21.15.1:0/3518024909 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8668053a20 con 0x7f866c0b3f40 2024-03-18T20:07:12.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.318+0000 7f867366a700 1 -- 172.21.15.1:0/3518024909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7f866c0a99b0 con 0x7f866c0b3f40 2024-03-18T20:07:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.870+0000 7f8661ffb700 1 -- 172.21.15.1:0/3518024909 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2524) v1 ==== 168+0+0 (secure 0 0 0) 0x7f8668057040 con 0x7f866c0b3f40 2024-03-18T20:07:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.872+0000 7f867366a700 1 -- 172.21.15.1:0/3518024909 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f865006adb0 msgr2=0x7f865006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.873+0000 7f867366a700 1 --2- 172.21.15.1:0/3518024909 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f865006adb0 0x7f865006d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f865c009820 tx=0x7f865c008040 comp rx=0 tx=0).stop 2024-03-18T20:07:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.873+0000 7f867366a700 1 -- 172.21.15.1:0/3518024909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866c0b3f40 msgr2=0x7f866c1e2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.873+0000 7f867366a700 1 --2- 172.21.15.1:0/3518024909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866c0b3f40 0x7f866c1e2610 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f866800c500 tx=0x7f866800c8c0 comp rx=0 tx=0).stop 2024-03-18T20:07:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.873+0000 7f867366a700 1 -- 172.21.15.1:0/3518024909 shutdown_connections 2024-03-18T20:07:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.873+0000 7f867366a700 1 --2- 172.21.15.1:0/3518024909 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f865006adb0 0x7f865006d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.873+0000 7f867366a700 1 --2- 172.21.15.1:0/3518024909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f866c06f240 0x7f866c1f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.873+0000 7f867366a700 1 --2- 172.21.15.1:0/3518024909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f866c143c20 0x7f866c0aa550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.873+0000 7f867366a700 1 --2- 172.21.15.1:0/3518024909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f866c0b3f40 0x7f866c1e2610 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.873+0000 7f867366a700 1 -- 172.21.15.1:0/3518024909 >> 172.21.15.1:0/3518024909 conn(0x7f866c0b1d20 msgr2=0x7f866c0afb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.873+0000 7f867366a700 1 -- 172.21.15.1:0/3518024909 shutdown_connections 2024-03-18T20:07:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:12.873+0000 7f867366a700 1 -- 172.21.15.1:0/3518024909 wait complete. 2024-03-18T20:07:12.892 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-18T20:07:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.023+0000 7f6f15d51700 1 Processor -- start 2024-03-18T20:07:13.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.023+0000 7f6f15d51700 1 -- start start 2024-03-18T20:07:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.024+0000 7f6f15d51700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6f1006d560 0x7f6f100ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.024+0000 7f6f15d51700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f100cac80 con 0x7f6f10071cc0 2024-03-18T20:07:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.024+0000 7f6f15d51700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f100caf60 con 0x7f6f1015a240 2024-03-18T20:07:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.024+0000 7f6f15d51700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f100adff0 con 0x7f6f1006d560 2024-03-18T20:07:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.024+0000 7f6f0ffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6f1006d560 0x7f6f100ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:13.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.024+0000 7f6f0effd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6f10071cc0 0x7f6f100ae410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58734/0 (socket says 172.21.15.1:58734) 2024-03-18T20:07:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.024+0000 7f6f0effd700 1 -- 172.21.15.1:0/1877865054 learned_addr learned my addr 172.21.15.1:0/1877865054 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.024+0000 7f6f0ffff700 1 -- 172.21.15.1:0/1877865054 >> v1:172.21.15.165:6789/0 conn(0x7f6f1015a240 legacy=0x7f6f100715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.024+0000 7f6f0ffff700 1 -- 172.21.15.1:0/1877865054 >> v1:172.21.15.1:6789/0 conn(0x7f6f10071cc0 legacy=0x7f6f100ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.024+0000 7f6f0e7fc700 1 -- 172.21.15.1:0/1877865054 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701590687 0 0) 0x7f6f100cac80 con 0x7f6f10071cc0 2024-03-18T20:07:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.024+0000 7f6f0ffff700 1 -- 172.21.15.1:0/1877865054 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f100cb240 con 0x7f6f1006d560 2024-03-18T20:07:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.024+0000 7f6f0ffff700 1 --2- 172.21.15.1:0/1877865054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f1006d560 0x7f6f100ca580 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f6f08009d90 tx=0x7f6f0800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3439838c86f1a4fb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.025+0000 7f6f0e7fc700 1 -- 172.21.15.1:0/1877865054 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0800e070 con 0x7f6f1006d560 2024-03-18T20:07:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.025+0000 7f6f0e7fc700 1 -- 172.21.15.1:0/1877865054 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f0800bbd0 con 0x7f6f1006d560 2024-03-18T20:07:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.025+0000 7f6f0e7fc700 1 -- 172.21.15.1:0/1877865054 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f08012430 con 0x7f6f1006d560 2024-03-18T20:07:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.025+0000 7f6f15d51700 1 -- 172.21.15.1:0/1877865054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f1006d560 msgr2=0x7f6f100ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:13.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.025+0000 7f6f15d51700 1 --2- 172.21.15.1:0/1877865054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f1006d560 0x7f6f100ca580 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f6f08009d90 tx=0x7f6f0800b330 comp rx=0 tx=0).stop 2024-03-18T20:07:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.025+0000 7f6f15d51700 1 -- 172.21.15.1:0/1877865054 shutdown_connections 2024-03-18T20:07:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.025+0000 7f6f15d51700 1 --2- 172.21.15.1:0/1877865054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f1006d560 0x7f6f100ca580 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.025+0000 7f6f15d51700 1 -- 172.21.15.1:0/1877865054 >> 172.21.15.1:0/1877865054 conn(0x7f6f100cff30 msgr2=0x7f6f100d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.025+0000 7f6f15d51700 1 -- 172.21.15.1:0/1877865054 shutdown_connections 2024-03-18T20:07:13.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.025+0000 7f6f15d51700 1 -- 172.21.15.1:0/1877865054 wait complete. 2024-03-18T20:07:13.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.026+0000 7f6f15d51700 1 Processor -- start 2024-03-18T20:07:13.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.027+0000 7f6f15d51700 1 -- start start 2024-03-18T20:07:13.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.027+0000 7f6f15d51700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f1006d560 0x7f6f1014c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:13.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.027+0000 7f6f15d51700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6f10071cc0 0x7f6f100c6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:13.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.027+0000 7f6f15d51700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f1015a240 0x7f6f100c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:13.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.027+0000 7f6f15d51700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1009d270 con 0x7f6f1006d560 2024-03-18T20:07:13.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.027+0000 7f6f15d51700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1009c5f0 con 0x7f6f1015a240 2024-03-18T20:07:13.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.027+0000 7f6f15d51700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1009c8d0 con 0x7f6f10071cc0 2024-03-18T20:07:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.027+0000 7f6f0f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f1006d560 0x7f6f1014c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.027+0000 7f6f0f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f1006d560 0x7f6f1014c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46626/0 (socket says 172.21.15.1:46626) 2024-03-18T20:07:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.027+0000 7f6f0f7fe700 1 -- 172.21.15.1:0/4031312609 learned_addr learned my addr 172.21.15.1:0/4031312609 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.028+0000 7f6f0ffff700 1 --2- 172.21.15.1:0/4031312609 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f1015a240 0x7f6f100c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.028+0000 7f6f0effd700 1 --2- 172.21.15.1:0/4031312609 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6f10071cc0 0x7f6f100c6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:13.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.028+0000 7f6f0f7fe700 1 -- 172.21.15.1:0/4031312609 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6f10071cc0 msgr2=0x7f6f100c6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:13.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.028+0000 7f6f0f7fe700 1 --2- 172.21.15.1:0/4031312609 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6f10071cc0 0x7f6f100c6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:13.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.028+0000 7f6f0f7fe700 1 -- 172.21.15.1:0/4031312609 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f1015a240 msgr2=0x7f6f100c94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:13.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.028+0000 7f6f0f7fe700 1 --2- 172.21.15.1:0/4031312609 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f1015a240 0x7f6f100c94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:13.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.028+0000 7f6f0f7fe700 1 -- 172.21.15.1:0/4031312609 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f08009a70 con 0x7f6f1006d560 2024-03-18T20:07:13.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.028+0000 7f6f0f7fe700 1 --2- 172.21.15.1:0/4031312609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f1006d560 0x7f6f1014c260 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800c450 tx=0x7f6ef800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:13.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.028+0000 7f6f07fff700 1 -- 172.21.15.1:0/4031312609 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef80052d0 con 0x7f6f1006d560 2024-03-18T20:07:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.028+0000 7f6f07fff700 1 -- 172.21.15.1:0/4031312609 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef8005430 con 0x7f6f1006d560 2024-03-18T20:07:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.028+0000 7f6f07fff700 1 -- 172.21.15.1:0/4031312609 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef80038c0 con 0x7f6f1006d560 2024-03-18T20:07:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.028+0000 7f6f15d51700 1 -- 172.21.15.1:0/4031312609 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f1009cbb0 con 0x7f6f1006d560 2024-03-18T20:07:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.028+0000 7f6f15d51700 1 -- 172.21.15.1:0/4031312609 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f1009d690 con 0x7f6f1006d560 2024-03-18T20:07:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.029+0000 7f6f15d51700 1 -- 172.21.15.1:0/4031312609 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f1009d4e0 con 0x7f6f1006d560 2024-03-18T20:07:13.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.030+0000 7f6f07fff700 1 -- 172.21.15.1:0/4031312609 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6ef8003a20 con 0x7f6f1006d560 2024-03-18T20:07:13.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.030+0000 7f6f07fff700 1 --2- 172.21.15.1:0/4031312609 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ef006adb0 0x7f6ef006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:13.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.030+0000 7f6f07fff700 1 -- 172.21.15.1:0/4031312609 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f6ef80888a0 con 0x7f6f1006d560 2024-03-18T20:07:13.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.030+0000 7f6f0effd700 1 --2- 172.21.15.1:0/4031312609 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ef006adb0 0x7f6ef006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:13.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.030+0000 7f6f0effd700 1 --2- 172.21.15.1:0/4031312609 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ef006adb0 0x7f6ef006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6f00007b10 tx=0x7f6f00007a60 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:13.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.035+0000 7f6f07fff700 1 -- 172.21.15.1:0/4031312609 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6ef8053880 con 0x7f6f1006d560 2024-03-18T20:07:13.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.236+0000 7f6f15d51700 1 -- 172.21.15.1:0/4031312609 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6f100676a0 con 0x7f6f1006d560 2024-03-18T20:07:13.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.239+0000 7f6f07fff700 1 -- 172.21.15.1:0/4031312609 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6ef80170c0 con 0x7f6f1006d560 2024-03-18T20:07:13.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.242+0000 7f6f15d51700 1 -- 172.21.15.1:0/4031312609 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ef006adb0 msgr2=0x7f6ef006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:13.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.242+0000 7f6f15d51700 1 --2- 172.21.15.1:0/4031312609 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ef006adb0 0x7f6ef006d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6f00007b10 tx=0x7f6f00007a60 comp rx=0 tx=0).stop 2024-03-18T20:07:13.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.242+0000 7f6f15d51700 1 -- 172.21.15.1:0/4031312609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f1006d560 msgr2=0x7f6f1014c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:13.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.242+0000 7f6f15d51700 1 --2- 172.21.15.1:0/4031312609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f1006d560 0x7f6f1014c260 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800c450 tx=0x7f6ef800c760 comp rx=0 tx=0).stop 2024-03-18T20:07:13.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.242+0000 7f6f15d51700 1 -- 172.21.15.1:0/4031312609 shutdown_connections 2024-03-18T20:07:13.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.242+0000 7f6f15d51700 1 --2- 172.21.15.1:0/4031312609 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f6ef006adb0 0x7f6ef006d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:13.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.242+0000 7f6f15d51700 1 --2- 172.21.15.1:0/4031312609 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f1015a240 0x7f6f100c94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:13.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.242+0000 7f6f15d51700 1 --2- 172.21.15.1:0/4031312609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f1006d560 0x7f6f1014c260 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:13.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.242+0000 7f6f15d51700 1 --2- 172.21.15.1:0/4031312609 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6f10071cc0 0x7f6f100c6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:13.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.242+0000 7f6f15d51700 1 -- 172.21.15.1:0/4031312609 >> 172.21.15.1:0/4031312609 conn(0x7f6f100cff30 msgr2=0x7f6f101593e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:13.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.242+0000 7f6f15d51700 1 -- 172.21.15.1:0/4031312609 shutdown_connections 2024-03-18T20:07:13.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.242+0000 7f6f15d51700 1 -- 172.21.15.1:0/4031312609 wait complete. 2024-03-18T20:07:13.259 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T20:07:13.259 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:07:13.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:07:13.348 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T20:07:13.348 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:07:13.349 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:07:13.367 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:07:13.368 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:07:13.368 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:07:13.388 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:07:13.388 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:07:13.388 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:07:13.427 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:07:13.427 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:07:13.427 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:07:13.448 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:07:13.448 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-18T20:07:13.449 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:07:13.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.588+0000 7f5a29297700 1 Processor -- start 2024-03-18T20:07:13.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.588+0000 7f5a29297700 1 -- start start 2024-03-18T20:07:13.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.589+0000 7f5a29297700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5a2415d620 0x7f5a24070d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:13.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.589+0000 7f5a29297700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5a2406ce90 0x7f5a2406d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:13.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.589+0000 7f5a29297700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a240cd5a0 con 0x7f5a2406e8f0 2024-03-18T20:07:13.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.589+0000 7f5a29297700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a2415da00 con 0x7f5a2406ce90 2024-03-18T20:07:13.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.589+0000 7f5a29297700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a2415e5f0 con 0x7f5a2415d620 2024-03-18T20:07:13.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.589+0000 7f5a237fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5a2406ce90 0x7f5a2406d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:13.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.589+0000 7f5a237fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5a2406ce90 0x7f5a2406d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46632/0 (socket says 172.21.15.1:46632) 2024-03-18T20:07:13.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.589+0000 7f5a237fe700 1 -- 172.21.15.1:0/3768904339 learned_addr learned my addr 172.21.15.1:0/3768904339 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:13.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.590+0000 7f5a22ffd700 1 --2- 172.21.15.1:0/3768904339 >> v2:172.21.15.165:3300/0 conn(0x7f5a2415d620 0x7f5a24070d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:13.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.590+0000 7f5a237fe700 1 -- 172.21.15.1:0/3768904339 >> v2:172.21.15.165:3300/0 conn(0x7f5a2415d620 msgr2=0x7f5a24070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:13.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.590+0000 7f5a237fe700 1 --2- 172.21.15.1:0/3768904339 >> v2:172.21.15.165:3300/0 conn(0x7f5a2415d620 0x7f5a24070d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:13.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.590+0000 7f5a237fe700 1 -- 172.21.15.1:0/3768904339 >> v1:172.21.15.165:6789/0 conn(0x7f5a2406e8f0 legacy=0x7f5a24071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:13.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.590+0000 7f5a237fe700 1 -- 172.21.15.1:0/3768904339 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2415e8d0 con 0x7f5a2406ce90 2024-03-18T20:07:13.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.590+0000 7f5a237fe700 1 --2- 172.21.15.1:0/3768904339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a2406ce90 0x7f5a2406d290 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f5a240b2330 tx=0x7f5a1c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50a9c9ae3349afa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:13.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.590+0000 7f5a21ffb700 1 -- 172.21.15.1:0/3768904339 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1c00e070 con 0x7f5a2406ce90 2024-03-18T20:07:13.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.590+0000 7f5a21ffb700 1 -- 172.21.15.1:0/3768904339 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1c00b9c0 con 0x7f5a2406ce90 2024-03-18T20:07:13.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.590+0000 7f5a21ffb700 1 -- 172.21.15.1:0/3768904339 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1c00bd70 con 0x7f5a2406ce90 2024-03-18T20:07:13.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.591+0000 7f5a29297700 1 -- 172.21.15.1:0/3768904339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a2406ce90 msgr2=0x7f5a2406d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:13.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.591+0000 7f5a29297700 1 --2- 172.21.15.1:0/3768904339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a2406ce90 0x7f5a2406d290 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f5a240b2330 tx=0x7f5a1c00b330 comp rx=0 tx=0).stop 2024-03-18T20:07:13.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.591+0000 7f5a29297700 1 -- 172.21.15.1:0/3768904339 shutdown_connections 2024-03-18T20:07:13.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.591+0000 7f5a29297700 1 --2- 172.21.15.1:0/3768904339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a2406ce90 0x7f5a2406d290 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:13.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.591+0000 7f5a29297700 1 --2- 172.21.15.1:0/3768904339 >> v2:172.21.15.165:3300/0 conn(0x7f5a2415d620 0x7f5a24070d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:13.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.591+0000 7f5a29297700 1 -- 172.21.15.1:0/3768904339 >> 172.21.15.1:0/3768904339 conn(0x7f5a240b1270 msgr2=0x7f5a240b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:13.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.591+0000 7f5a29297700 1 -- 172.21.15.1:0/3768904339 shutdown_connections 2024-03-18T20:07:13.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.591+0000 7f5a29297700 1 -- 172.21.15.1:0/3768904339 wait complete. 2024-03-18T20:07:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.592+0000 7f5a29297700 1 Processor -- start 2024-03-18T20:07:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.592+0000 7f5a29297700 1 -- start start 2024-03-18T20:07:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.593+0000 7f5a29297700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a2406ce90 0x7f5a24156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.593+0000 7f5a29297700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a2406e8f0 0x7f5a241e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.593+0000 7f5a29297700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a2415d620 0x7f5a241e8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.593+0000 7f5a29297700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a240c34a0 con 0x7f5a2406e8f0 2024-03-18T20:07:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.593+0000 7f5a29297700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a241f47f0 con 0x7f5a2406ce90 2024-03-18T20:07:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.593+0000 7f5a29297700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a24157020 con 0x7f5a2415d620 2024-03-18T20:07:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.593+0000 7f5a227fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a2406e8f0 0x7f5a241e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.593+0000 7f5a227fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a2406e8f0 0x7f5a241e8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46648/0 (socket says 172.21.15.1:46648) 2024-03-18T20:07:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.593+0000 7f5a227fc700 1 -- 172.21.15.1:0/2368924602 learned_addr learned my addr 172.21.15.1:0/2368924602 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:13.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.593+0000 7f5a22ffd700 1 --2- 172.21.15.1:0/2368924602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a2406ce90 0x7f5a24156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:13.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.593+0000 7f5a237fe700 1 --2- 172.21.15.1:0/2368924602 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a2415d620 0x7f5a241e8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:13.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.593+0000 7f5a227fc700 1 -- 172.21.15.1:0/2368924602 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a2415d620 msgr2=0x7f5a241e8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:13.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.593+0000 7f5a227fc700 1 --2- 172.21.15.1:0/2368924602 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a2415d620 0x7f5a241e8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:13.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.593+0000 7f5a227fc700 1 -- 172.21.15.1:0/2368924602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a2406ce90 msgr2=0x7f5a24156530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:13.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.593+0000 7f5a227fc700 1 --2- 172.21.15.1:0/2368924602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a2406ce90 0x7f5a24156530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:13.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.593+0000 7f5a227fc700 1 -- 172.21.15.1:0/2368924602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1c009b30 con 0x7f5a2406e8f0 2024-03-18T20:07:13.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.593+0000 7f5a22ffd700 1 --2- 172.21.15.1:0/2368924602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a2406ce90 0x7f5a24156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:07:13.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.593+0000 7f5a227fc700 1 --2- 172.21.15.1:0/2368924602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a2406e8f0 0x7f5a241e8a30 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f5a1800c510 tx=0x7f5a1800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:13.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.594+0000 7f5a0b7fe700 1 -- 172.21.15.1:0/2368924602 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a180052d0 con 0x7f5a2406e8f0 2024-03-18T20:07:13.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.594+0000 7f5a0b7fe700 1 -- 172.21.15.1:0/2368924602 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a18005430 con 0x7f5a2406e8f0 2024-03-18T20:07:13.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.594+0000 7f5a29297700 1 -- 172.21.15.1:0/2368924602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a241f4230 con 0x7f5a2406e8f0 2024-03-18T20:07:13.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.594+0000 7f5a0b7fe700 1 -- 172.21.15.1:0/2368924602 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a180038c0 con 0x7f5a2406e8f0 2024-03-18T20:07:13.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.594+0000 7f5a29297700 1 -- 172.21.15.1:0/2368924602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a240c30c0 con 0x7f5a2406e8f0 2024-03-18T20:07:13.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.595+0000 7f5a29297700 1 -- 172.21.15.1:0/2368924602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a240676a0 con 0x7f5a2406e8f0 2024-03-18T20:07:13.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.595+0000 7f5a0b7fe700 1 -- 172.21.15.1:0/2368924602 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5a18003a20 con 0x7f5a2406e8f0 2024-03-18T20:07:13.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.595+0000 7f5a0b7fe700 1 --2- 172.21.15.1:0/2368924602 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5a1006adb0 0x7f5a1006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:13.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.596+0000 7f5a22ffd700 1 --2- 172.21.15.1:0/2368924602 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5a1006adb0 0x7f5a1006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:13.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.596+0000 7f5a0b7fe700 1 -- 172.21.15.1:0/2368924602 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f5a18088a50 con 0x7f5a2406e8f0 2024-03-18T20:07:13.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.596+0000 7f5a22ffd700 1 --2- 172.21.15.1:0/2368924602 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5a1006adb0 0x7f5a1006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5a240c5670 tx=0x7f5a14008040 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:13.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.600+0000 7f5a0b7fe700 1 -- 172.21.15.1:0/2368924602 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5a18053a30 con 0x7f5a2406e8f0 2024-03-18T20:07:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.848+0000 7f5a29297700 1 -- 172.21.15.1:0/2368924602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a240cce70 con 0x7f5a2406e8f0 2024-03-18T20:07:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.848+0000 7f5a0b7fe700 1 -- 172.21.15.1:0/2368924602 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3530 (secure 0 0 0) 0x7f5a18017090 con 0x7f5a2406e8f0 2024-03-18T20:07:13.854 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:07:13.854 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":83,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7874,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3186058752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3186058752},{"type":"v1","addr":"172.21.15.1:6836","nonce":3186058752}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":7886,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1126668185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1126668185},{"type":"v1","addr":"172.21.15.165:6835","nonce":1126668185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":9028,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/3922981693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":3922981693},{"type":"v1","addr":"172.21.15.165:6834","nonce":3922981693}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":76},{"gid":10329,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1153293597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1153293597},{"type":"v1","addr":"172.21.15.1:6837","nonce":1153293597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":83}],"filesystems":[]} 2024-03-18T20:07:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.850+0000 7f5a29297700 1 -- 172.21.15.1:0/2368924602 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5a1006adb0 msgr2=0x7f5a1006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.850+0000 7f5a29297700 1 --2- 172.21.15.1:0/2368924602 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5a1006adb0 0x7f5a1006d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5a240c5670 tx=0x7f5a14008040 comp rx=0 tx=0).stop 2024-03-18T20:07:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.850+0000 7f5a29297700 1 -- 172.21.15.1:0/2368924602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a2406e8f0 msgr2=0x7f5a241e8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.850+0000 7f5a29297700 1 --2- 172.21.15.1:0/2368924602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a2406e8f0 0x7f5a241e8a30 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f5a1800c510 tx=0x7f5a1800c8d0 comp rx=0 tx=0).stop 2024-03-18T20:07:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.850+0000 7f5a29297700 1 -- 172.21.15.1:0/2368924602 shutdown_connections 2024-03-18T20:07:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.850+0000 7f5a29297700 1 --2- 172.21.15.1:0/2368924602 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5a1006adb0 0x7f5a1006d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.850+0000 7f5a29297700 1 --2- 172.21.15.1:0/2368924602 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a2415d620 0x7f5a241e8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.850+0000 7f5a29297700 1 --2- 172.21.15.1:0/2368924602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a2406ce90 0x7f5a24156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.850+0000 7f5a29297700 1 --2- 172.21.15.1:0/2368924602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a2406e8f0 0x7f5a241e8a30 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.850+0000 7f5a29297700 1 -- 172.21.15.1:0/2368924602 >> 172.21.15.1:0/2368924602 conn(0x7f5a240b1270 msgr2=0x7f5a241588b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.850+0000 7f5a29297700 1 -- 172.21.15.1:0/2368924602 shutdown_connections 2024-03-18T20:07:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:13.850+0000 7f5a29297700 1 -- 172.21.15.1:0/2368924602 wait complete. 2024-03-18T20:07:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 83 2024-03-18T20:07:13.869 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T20:07:13.869 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-18T20:07:13.869 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T20:07:13.869 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:07:13.870 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T20:07:13.870 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-18T20:07:13.870 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-18T20:07:13.870 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:07:13.871 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T20:07:13.871 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-18T20:07:13.871 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-18T20:07:13.871 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:07:13.872 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T20:07:13.872 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-18T20:07:13.872 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-18T20:07:13.872 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:07:13.879 INFO:tasks.ceph.mds.a:Stopped 2024-03-18T20:07:13.879 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-18T20:07:13.881 INFO:tasks.ceph.mds.c:Stopped 2024-03-18T20:07:13.881 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-18T20:07:13.883 INFO:tasks.ceph.mds.b:Stopped 2024-03-18T20:07:13.883 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-18T20:07:13.884 INFO:tasks.ceph.mds.d:Stopped 2024-03-18T20:07:13.884 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-18T20:07:13.923 INFO:tasks.ceph.mds.a:Started 2024-03-18T20:07:13.924 DEBUG:teuthology.parallel:result is None 2024-03-18T20:07:13.925 INFO:tasks.ceph.mds.c:Started 2024-03-18T20:07:13.925 DEBUG:teuthology.parallel:result is None 2024-03-18T20:07:13.926 INFO:tasks.ceph.mds.b:Started 2024-03-18T20:07:13.926 DEBUG:teuthology.parallel:result is None 2024-03-18T20:07:13.927 INFO:tasks.ceph.mds.d:Started 2024-03-18T20:07:13.927 DEBUG:teuthology.parallel:result is None 2024-03-18T20:07:13.928 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-18T20:07:14.005 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-18T20:07:14.005 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T20:07:14.015 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-03-18T20:07:14.015 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-03-18T20:07:14.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.057+0000 7f0ebdccb700 1 Processor -- start 2024-03-18T20:07:14.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.057+0000 7f0ebdccb700 1 -- start start 2024-03-18T20:07:14.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.057+0000 7f0ebdccb700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0eb8160e50 0x7f0eb80cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:14.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.057+0000 7f0ebdccb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0eb81612d0 0x7f0eb81616f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:14.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.057+0000 7f0ebdccb700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb8162170 con 0x7f0eb8160a70 2024-03-18T20:07:14.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.057+0000 7f0ebdccb700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb8161c30 con 0x7f0eb8160e50 2024-03-18T20:07:14.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.057+0000 7f0ebdccb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb80cef30 con 0x7f0eb81612d0 2024-03-18T20:07:14.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.058+0000 7f0eb6ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0eb8160e50 0x7f0eb80cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:14.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.058+0000 7f0eb6ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0eb8160e50 0x7f0eb80cffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46702/0 (socket says 172.21.15.1:46702) 2024-03-18T20:07:14.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.058+0000 7f0eb6ffd700 1 -- 172.21.15.1:0/2068025520 learned_addr learned my addr 172.21.15.1:0/2068025520 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:14.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.058+0000 7f0eb7fff700 1 --2- 172.21.15.1:0/2068025520 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0eb81612d0 0x7f0eb81616f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:14.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.058+0000 7f0eb6ffd700 1 -- 172.21.15.1:0/2068025520 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0eb81612d0 msgr2=0x7f0eb81616f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:14.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.058+0000 7f0eb6ffd700 1 --2- 172.21.15.1:0/2068025520 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0eb81612d0 0x7f0eb81616f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:14.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.058+0000 7f0eb6ffd700 1 -- 172.21.15.1:0/2068025520 >> v1:172.21.15.165:6789/0 conn(0x7f0eb8160a70 legacy=0x7f0eb80d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:14.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.058+0000 7f0eb6ffd700 1 -- 172.21.15.1:0/2068025520 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb80cf210 con 0x7f0eb8160e50 2024-03-18T20:07:14.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.058+0000 7f0eb6ffd700 1 --2- 172.21.15.1:0/2068025520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb8160e50 0x7f0eb80cffd0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f0ea00098f0 tx=0x7f0ea0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8391e1adaee037f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:14.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.058+0000 7f0eb67fc700 1 -- 172.21.15.1:0/2068025520 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0004230 con 0x7f0eb8160e50 2024-03-18T20:07:14.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.058+0000 7f0eb67fc700 1 -- 172.21.15.1:0/2068025520 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea0004390 con 0x7f0eb8160e50 2024-03-18T20:07:14.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.058+0000 7f0eb67fc700 1 -- 172.21.15.1:0/2068025520 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea00036a0 con 0x7f0eb8160e50 2024-03-18T20:07:14.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.059+0000 7f0ebdccb700 1 -- 172.21.15.1:0/2068025520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb8160e50 msgr2=0x7f0eb80cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:14.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.059+0000 7f0ebdccb700 1 --2- 172.21.15.1:0/2068025520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb8160e50 0x7f0eb80cffd0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f0ea00098f0 tx=0x7f0ea0009c00 comp rx=0 tx=0).stop 2024-03-18T20:07:14.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.059+0000 7f0ebdccb700 1 -- 172.21.15.1:0/2068025520 shutdown_connections 2024-03-18T20:07:14.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.059+0000 7f0ebdccb700 1 --2- 172.21.15.1:0/2068025520 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0eb81612d0 0x7f0eb81616f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:14.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.059+0000 7f0ebdccb700 1 --2- 172.21.15.1:0/2068025520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb8160e50 0x7f0eb80cffd0 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:14.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.059+0000 7f0ebdccb700 1 -- 172.21.15.1:0/2068025520 >> 172.21.15.1:0/2068025520 conn(0x7f0eb809c560 msgr2=0x7f0eb80d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:14.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.059+0000 7f0ebdccb700 1 -- 172.21.15.1:0/2068025520 shutdown_connections 2024-03-18T20:07:14.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.059+0000 7f0ebdccb700 1 -- 172.21.15.1:0/2068025520 wait complete. 2024-03-18T20:07:14.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.060+0000 7f0ebdccb700 1 Processor -- start 2024-03-18T20:07:14.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.060+0000 7f0ebdccb700 1 -- start start 2024-03-18T20:07:14.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.061+0000 7f0ebdccb700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0eb8160a70 0x7f0eb80b5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:14.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.061+0000 7f0ebdccb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0eb8160e50 0x7f0eb815f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:14.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.061+0000 7f0ebdccb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb81612d0 0x7f0eb80c7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:14.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.061+0000 7f0ebdccb700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb80b6610 con 0x7f0eb81612d0 2024-03-18T20:07:14.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.061+0000 7f0ebdccb700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb80b68f0 con 0x7f0eb8160a70 2024-03-18T20:07:14.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.061+0000 7f0ebdccb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb80b6bd0 con 0x7f0eb8160e50 2024-03-18T20:07:14.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.061+0000 7f0eb7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb81612d0 0x7f0eb80c7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:14.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.061+0000 7f0eb77fe700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0eb8160a70 0x7f0eb80b5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:14.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.061+0000 7f0eb7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb81612d0 0x7f0eb80c7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46704/0 (socket says 172.21.15.1:46704) 2024-03-18T20:07:14.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.061+0000 7f0eb7fff700 1 -- 172.21.15.1:0/2411044824 learned_addr learned my addr 172.21.15.1:0/2411044824 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:14.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.061+0000 7f0eb6ffd700 1 --2- 172.21.15.1:0/2411044824 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0eb8160e50 0x7f0eb815f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:14.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.061+0000 7f0eb7fff700 1 -- 172.21.15.1:0/2411044824 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0eb8160e50 msgr2=0x7f0eb815f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:14.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.061+0000 7f0eb7fff700 1 --2- 172.21.15.1:0/2411044824 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0eb8160e50 0x7f0eb815f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:14.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.061+0000 7f0eb7fff700 1 -- 172.21.15.1:0/2411044824 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0eb8160a70 msgr2=0x7f0eb80b5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:14.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.061+0000 7f0eb7fff700 1 --2- 172.21.15.1:0/2411044824 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0eb8160a70 0x7f0eb80b5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:14.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.061+0000 7f0eb7fff700 1 -- 172.21.15.1:0/2411044824 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea0009550 con 0x7f0eb81612d0 2024-03-18T20:07:14.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.061+0000 7f0eb7fff700 1 --2- 172.21.15.1:0/2411044824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb81612d0 0x7f0eb80c7200 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f0eb000cd70 tx=0x7f0eb000a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:14.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.062+0000 7f0eaffff700 1 -- 172.21.15.1:0/2411044824 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0003a20 con 0x7f0eb81612d0 2024-03-18T20:07:14.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.062+0000 7f0eaffff700 1 -- 172.21.15.1:0/2411044824 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb0003b80 con 0x7f0eb81612d0 2024-03-18T20:07:14.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.062+0000 7f0ebdccb700 1 -- 172.21.15.1:0/2411044824 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb80cc090 con 0x7f0eb81612d0 2024-03-18T20:07:14.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.062+0000 7f0eaffff700 1 -- 172.21.15.1:0/2411044824 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0004020 con 0x7f0eb81612d0 2024-03-18T20:07:14.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.062+0000 7f0ebdccb700 1 -- 172.21.15.1:0/2411044824 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb80b6320 con 0x7f0eb81612d0 2024-03-18T20:07:14.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.063+0000 7f0ebdccb700 1 -- 172.21.15.1:0/2411044824 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb80cc5b0 con 0x7f0eb81612d0 2024-03-18T20:07:14.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.064+0000 7f0eaffff700 1 -- 172.21.15.1:0/2411044824 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0eb0004180 con 0x7f0eb81612d0 2024-03-18T20:07:14.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.064+0000 7f0eaffff700 1 --2- 172.21.15.1:0/2411044824 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0e9806adb0 0x7f0e9806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:14.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.064+0000 7f0eaffff700 1 -- 172.21.15.1:0/2411044824 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f0eb0012070 con 0x7f0eb81612d0 2024-03-18T20:07:14.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.064+0000 7f0eb77fe700 1 --2- 172.21.15.1:0/2411044824 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0e9806adb0 0x7f0e9806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:14.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.065+0000 7f0eb77fe700 1 --2- 172.21.15.1:0/2411044824 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0e9806adb0 0x7f0e9806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8009960 tx=0x7f0ea80090e0 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:14.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.068+0000 7f0eaffff700 1 -- 172.21.15.1:0/2411044824 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0eb0054f70 con 0x7f0eb81612d0 2024-03-18T20:07:14.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:14.288+0000 7f0ebdccb700 1 -- 172.21.15.1:0/2411044824 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0eb80cc5b0 con 0x7f0eb81612d0 2024-03-18T20:07:15.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.243+0000 7f0eaffff700 1 -- 172.21.15.1:0/2411044824 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v191) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0eb80cc5b0 con 0x7f0eb81612d0 2024-03-18T20:07:15.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.246+0000 7f0ebdccb700 1 -- 172.21.15.1:0/2411044824 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0e9806adb0 msgr2=0x7f0e9806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:15.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.246+0000 7f0ebdccb700 1 --2- 172.21.15.1:0/2411044824 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0e9806adb0 0x7f0e9806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8009960 tx=0x7f0ea80090e0 comp rx=0 tx=0).stop 2024-03-18T20:07:15.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.246+0000 7f0ebdccb700 1 -- 172.21.15.1:0/2411044824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb81612d0 msgr2=0x7f0eb80c7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:15.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.246+0000 7f0ebdccb700 1 --2- 172.21.15.1:0/2411044824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb81612d0 0x7f0eb80c7200 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f0eb000cd70 tx=0x7f0eb000a570 comp rx=0 tx=0).stop 2024-03-18T20:07:15.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.246+0000 7f0ebdccb700 1 -- 172.21.15.1:0/2411044824 shutdown_connections 2024-03-18T20:07:15.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.246+0000 7f0ebdccb700 1 --2- 172.21.15.1:0/2411044824 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0e9806adb0 0x7f0e9806d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:15.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.246+0000 7f0ebdccb700 1 --2- 172.21.15.1:0/2411044824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eb81612d0 0x7f0eb80c7200 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:15.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.246+0000 7f0ebdccb700 1 --2- 172.21.15.1:0/2411044824 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0eb8160a70 0x7f0eb80b5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:15.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.246+0000 7f0ebdccb700 1 --2- 172.21.15.1:0/2411044824 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0eb8160e50 0x7f0eb815f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:15.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.246+0000 7f0ebdccb700 1 -- 172.21.15.1:0/2411044824 >> 172.21.15.1:0/2411044824 conn(0x7f0eb809c560 msgr2=0x7f0eb80d3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:15.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.246+0000 7f0ebdccb700 1 -- 172.21.15.1:0/2411044824 shutdown_connections 2024-03-18T20:07:15.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.246+0000 7f0ebdccb700 1 -- 172.21.15.1:0/2411044824 wait complete. 2024-03-18T20:07:15.253 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-18T20:07:15.266 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-18T20:07:15.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.394+0000 7f563ea9b700 1 Processor -- start 2024-03-18T20:07:15.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.394+0000 7f563ea9b700 1 -- start start 2024-03-18T20:07:15.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.395+0000 7f563ea9b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5638143c20 0x7f56380b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:15.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.395+0000 7f563ea9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f563806f240 0x7f56380cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:15.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.395+0000 7f563ea9b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5638144000 con 0x7f56380b3f40 2024-03-18T20:07:15.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.395+0000 7f563ea9b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56380b43c0 con 0x7f563806f240 2024-03-18T20:07:15.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.395+0000 7f563ea9b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56380b51b0 con 0x7f5638143c20 2024-03-18T20:07:15.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.395+0000 7f563d038700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f563806f240 0x7f56380cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:15.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.396+0000 7f563d038700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f563806f240 0x7f56380cddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37530/0 (socket says 172.21.15.1:37530) 2024-03-18T20:07:15.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.396+0000 7f563d038700 1 -- 172.21.15.1:0/1526844518 learned_addr learned my addr 172.21.15.1:0/1526844518 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:15.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.396+0000 7f563c837700 1 --2- 172.21.15.1:0/1526844518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5638143c20 0x7f56380b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:15.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.396+0000 7f563d038700 1 -- 172.21.15.1:0/1526844518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5638143c20 msgr2=0x7f56380b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:15.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.396+0000 7f563d038700 1 --2- 172.21.15.1:0/1526844518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5638143c20 0x7f56380b3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:15.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.396+0000 7f563d038700 1 -- 172.21.15.1:0/1526844518 >> v1:172.21.15.165:6789/0 conn(0x7f56380b3f40 legacy=0x7f563806eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:15.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.396+0000 7f563d038700 1 -- 172.21.15.1:0/1526844518 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56380cd8d0 con 0x7f563806f240 2024-03-18T20:07:15.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.396+0000 7f563d038700 1 --2- 172.21.15.1:0/1526844518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f563806f240 0x7f56380cddb0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f563000d180 tx=0x7f563000d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e729f08148e888 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:15.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.396+0000 7f56377fe700 1 -- 172.21.15.1:0/1526844518 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5630010070 con 0x7f563806f240 2024-03-18T20:07:15.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.396+0000 7f56377fe700 1 -- 172.21.15.1:0/1526844518 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563000ddb0 con 0x7f563806f240 2024-03-18T20:07:15.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.396+0000 7f56377fe700 1 -- 172.21.15.1:0/1526844518 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563000b310 con 0x7f563806f240 2024-03-18T20:07:15.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.397+0000 7f563ea9b700 1 -- 172.21.15.1:0/1526844518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f563806f240 msgr2=0x7f56380cddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:15.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.397+0000 7f563ea9b700 1 --2- 172.21.15.1:0/1526844518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f563806f240 0x7f56380cddb0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f563000d180 tx=0x7f563000d490 comp rx=0 tx=0).stop 2024-03-18T20:07:15.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.397+0000 7f563ea9b700 1 -- 172.21.15.1:0/1526844518 shutdown_connections 2024-03-18T20:07:15.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.397+0000 7f563ea9b700 1 --2- 172.21.15.1:0/1526844518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f563806f240 0x7f56380cddb0 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:15.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.397+0000 7f563ea9b700 1 --2- 172.21.15.1:0/1526844518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5638143c20 0x7f56380b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:15.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.397+0000 7f563ea9b700 1 -- 172.21.15.1:0/1526844518 >> 172.21.15.1:0/1526844518 conn(0x7f56380b1d20 msgr2=0x7f56380b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:15.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.397+0000 7f563ea9b700 1 -- 172.21.15.1:0/1526844518 shutdown_connections 2024-03-18T20:07:15.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.397+0000 7f563ea9b700 1 -- 172.21.15.1:0/1526844518 wait complete. 2024-03-18T20:07:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.398+0000 7f563ea9b700 1 Processor -- start 2024-03-18T20:07:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.398+0000 7f563ea9b700 1 -- start start 2024-03-18T20:07:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.399+0000 7f563ea9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f563806f240 0x7f5638159b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.399+0000 7f563ea9b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56380b3f40 0x7f56380ce900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.399+0000 7f563ea9b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5638143c20 0x7f56380cee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.399+0000 7f563ea9b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5638134630 con 0x7f563806f240 2024-03-18T20:07:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.399+0000 7f563ea9b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56381e5d60 con 0x7f5638143c20 2024-03-18T20:07:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.399+0000 7f563ea9b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563815a7e0 con 0x7f56380b3f40 2024-03-18T20:07:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.399+0000 7f563c837700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f563806f240 0x7f5638159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.399+0000 7f563c837700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f563806f240 0x7f5638159b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37546/0 (socket says 172.21.15.1:37546) 2024-03-18T20:07:15.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.399+0000 7f563c837700 1 -- 172.21.15.1:0/3570906283 learned_addr learned my addr 172.21.15.1:0/3570906283 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:15.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.399+0000 7f563d038700 1 --2- 172.21.15.1:0/3570906283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5638143c20 0x7f56380cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:15.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.399+0000 7f5637fff700 1 --2- 172.21.15.1:0/3570906283 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56380b3f40 0x7f56380ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:15.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.399+0000 7f563c837700 1 -- 172.21.15.1:0/3570906283 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56380b3f40 msgr2=0x7f56380ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:15.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.399+0000 7f563c837700 1 --2- 172.21.15.1:0/3570906283 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56380b3f40 0x7f56380ce900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:15.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.399+0000 7f563c837700 1 -- 172.21.15.1:0/3570906283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5638143c20 msgr2=0x7f56380cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:15.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.399+0000 7f563c837700 1 --2- 172.21.15.1:0/3570906283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5638143c20 0x7f56380cee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:15.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.399+0000 7f563c837700 1 -- 172.21.15.1:0/3570906283 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563000bd40 con 0x7f563806f240 2024-03-18T20:07:15.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.400+0000 7f563c837700 1 --2- 172.21.15.1:0/3570906283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f563806f240 0x7f5638159b60 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f562c00d800 tx=0x7f562c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:15.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.400+0000 7f56357fa700 1 -- 172.21.15.1:0/3570906283 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c0125f0 con 0x7f563806f240 2024-03-18T20:07:15.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.400+0000 7f56357fa700 1 -- 172.21.15.1:0/3570906283 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562c012750 con 0x7f563806f240 2024-03-18T20:07:15.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.400+0000 7f563ea9b700 1 -- 172.21.15.1:0/3570906283 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f563815aac0 con 0x7f563806f240 2024-03-18T20:07:15.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.400+0000 7f56357fa700 1 -- 172.21.15.1:0/3570906283 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c012b00 con 0x7f563806f240 2024-03-18T20:07:15.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.400+0000 7f563ea9b700 1 -- 172.21.15.1:0/3570906283 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f563815b080 con 0x7f563806f240 2024-03-18T20:07:15.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.401+0000 7f563ea9b700 1 -- 172.21.15.1:0/3570906283 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56380676a0 con 0x7f563806f240 2024-03-18T20:07:15.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.401+0000 7f56357fa700 1 -- 172.21.15.1:0/3570906283 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f562c012c60 con 0x7f563806f240 2024-03-18T20:07:15.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.402+0000 7f56357fa700 1 --2- 172.21.15.1:0/3570906283 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f561c06adb0 0x7f561c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:15.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.402+0000 7f56357fa700 1 -- 172.21.15.1:0/3570906283 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f562c0892b0 con 0x7f563806f240 2024-03-18T20:07:15.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.402+0000 7f5637fff700 1 --2- 172.21.15.1:0/3570906283 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f561c06adb0 0x7f561c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:15.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.402+0000 7f5637fff700 1 --2- 172.21.15.1:0/3570906283 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f561c06adb0 0x7f561c06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5624007fd0 tx=0x7f5624009040 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:15.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.406+0000 7f56357fa700 1 -- 172.21.15.1:0/3570906283 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f562c054680 con 0x7f563806f240 2024-03-18T20:07:15.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.692+0000 7f563ea9b700 1 -- 172.21.15.1:0/3570906283 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f563806ffd0 con 0x7f563806f240 2024-03-18T20:07:15.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.693+0000 7f56357fa700 1 -- 172.21.15.1:0/3570906283 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v42) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f562c057ca0 con 0x7f563806f240 2024-03-18T20:07:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:07:15.698 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:07:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-18T20:07:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-18T20:07:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlR93/CxAAbpe+5n3wHRur9xi8QzdCnQ==", 2024-03-18T20:07:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:07:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:07:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:07:15.698 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:07:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:07:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-18T20:07:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJ3OmGBAA37u227nxrMsh/JHt71HBxQ==", 2024-03-18T20:07:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:07:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:07:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:07:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:07:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:07:15.699 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-18T20:07:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJZFoExAAmI16oxwNKbf8dKetPfyk+g==", 2024-03-18T20:07:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:07:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:07:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:07:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:07:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:07:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-18T20:07:15.700 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlQrcvIBAA1LMv+EgC7S6wspX6QF49Pg==", 2024-03-18T20:07:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:07:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:07:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:07:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:07:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:07:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-18T20:07:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCUjPhlYOVHChAARFQrksZmMIdTlntd8TgvrQ==", 2024-03-18T20:07:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:07:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:07:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:07:15.701 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:07:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-18T20:07:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCXjPhl2zSDCBAA6Al1gkWVB79ZI9djGpVChA==", 2024-03-18T20:07:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:07:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:07:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:07:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:07:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-18T20:07:15.702 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCajPhlSoG/BhAAR93USDe6DCQW+hREOsnTlw==", 2024-03-18T20:07:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:07:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:07:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:07:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:07:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-18T20:07:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCdjPhlhJg5BRAAWuQhcnxyjR5UOiM13yKdAQ==", 2024-03-18T20:07:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:07:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:07:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:07:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:07:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-18T20:07:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCjjPhlCFnpCRAAKIyOQyVdjXZUTnAS6Nw/ng==", 2024-03-18T20:07:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:07:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:07:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:07:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:07:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-18T20:07:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCmjPhlAeFkCBAAJcEFpRk/03lQJxj+l2hEPA==", 2024-03-18T20:07:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:07:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:07:15.704 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:07:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:07:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-18T20:07:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCpjPhl/xuuBhAA0LgKfXEvN2O11ZlRQJVOkw==", 2024-03-18T20:07:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:07:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:07:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:07:15.705 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:07:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-18T20:07:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCsjPhldiR4BRAA571oh6U8PzOv/4LzjYLF/A==", 2024-03-18T20:07:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:07:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:07:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:07:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:07:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:15.706 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-18T20:07:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlu+2xJBAANnXGGjfeqP6S2FTecsfOsg==", 2024-03-18T20:07:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:07:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:07:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-18T20:07:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-18T20:07:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-18T20:07:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlhJ9+KxAAEnrQrCrgJR6xI+pucwzAXA==", 2024-03-18T20:07:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:07:15.707 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:07:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:07:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:07:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:07:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.2", 2024-03-18T20:07:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlxBUpKBAAIGTwmLWyj+P2Yvl55QVcWA==", 2024-03-18T20:07:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:07:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:07:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:07:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:07:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:07:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.3", 2024-03-18T20:07:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlmao3MRAAHeiQ6eyBh9QjjKPCtxkC8g==", 2024-03-18T20:07:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:07:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:07:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:07:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:07:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:07:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-18T20:07:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCRjPhlemOuKRAAVUggmtB7+EiVgyfJZ8TCCA==", 2024-03-18T20:07:15.709 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:07:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:07:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:07:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:07:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:07:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-18T20:07:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhleIuOCBAA9CH+AHewQbn+gpVr+6lJyw==", 2024-03-18T20:07:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:07:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:07:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T20:07:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:07:15.710 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-18T20:07:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlL0pTBBAAdFiPcdwwz514cuLAvioxYA==", 2024-03-18T20:07:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:07:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:07:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T20:07:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:07:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:15.711 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-18T20:07:15.711 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:07:15.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.695+0000 7f563ea9b700 1 -- 172.21.15.1:0/3570906283 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f561c06adb0 msgr2=0x7f561c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:15.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.695+0000 7f563ea9b700 1 --2- 172.21.15.1:0/3570906283 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f561c06adb0 0x7f561c06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5624007fd0 tx=0x7f5624009040 comp rx=0 tx=0).stop 2024-03-18T20:07:15.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.695+0000 7f563ea9b700 1 -- 172.21.15.1:0/3570906283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f563806f240 msgr2=0x7f5638159b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:15.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.695+0000 7f563ea9b700 1 --2- 172.21.15.1:0/3570906283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f563806f240 0x7f5638159b60 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f562c00d800 tx=0x7f562c00db10 comp rx=0 tx=0).stop 2024-03-18T20:07:15.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.695+0000 7f563ea9b700 1 -- 172.21.15.1:0/3570906283 shutdown_connections 2024-03-18T20:07:15.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.695+0000 7f563ea9b700 1 --2- 172.21.15.1:0/3570906283 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f561c06adb0 0x7f561c06d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:15.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.695+0000 7f563ea9b700 1 --2- 172.21.15.1:0/3570906283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5638143c20 0x7f56380cee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:15.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.695+0000 7f563ea9b700 1 --2- 172.21.15.1:0/3570906283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f563806f240 0x7f5638159b60 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:15.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.695+0000 7f563ea9b700 1 --2- 172.21.15.1:0/3570906283 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56380b3f40 0x7f56380ce900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:15.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.695+0000 7f563ea9b700 1 -- 172.21.15.1:0/3570906283 >> 172.21.15.1:0/3570906283 conn(0x7f56380b1d20 msgr2=0x7f56381e4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:15.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.695+0000 7f563ea9b700 1 -- 172.21.15.1:0/3570906283 shutdown_connections 2024-03-18T20:07:15.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.695+0000 7f563ea9b700 1 -- 172.21.15.1:0/3570906283 wait complete. 2024-03-18T20:07:15.714 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:07:15.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.834+0000 7f39be9a1700 1 Processor -- start 2024-03-18T20:07:15.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.834+0000 7f39be9a1700 1 -- start start 2024-03-18T20:07:15.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.835+0000 7f39be9a1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f39b815d6b0 0x7f39b80b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:15.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.835+0000 7f39be9a1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f39b80ad790 0x7f39b80b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:15.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.835+0000 7f39be9a1700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b80acfb0 con 0x7f39b806a750 2024-03-18T20:07:15.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.835+0000 7f39be9a1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b80af7a0 con 0x7f39b80ad790 2024-03-18T20:07:15.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.835+0000 7f39be9a1700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b80b50a0 con 0x7f39b815d6b0 2024-03-18T20:07:15.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.835+0000 7f39bcf3e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f39b80ad790 0x7f39b80b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:15.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.835+0000 7f39b7fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f39b806a750 0x7f39b806ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43332/0 (socket says 172.21.15.1:43332) 2024-03-18T20:07:15.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.835+0000 7f39b7fff700 1 -- 172.21.15.1:0/846074406 learned_addr learned my addr 172.21.15.1:0/846074406 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:15.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.835+0000 7f39b77fe700 1 --2- 172.21.15.1:0/846074406 >> v2:172.21.15.165:3300/0 conn(0x7f39b815d6b0 0x7f39b80b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:15.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.835+0000 7f39b6ffd700 1 -- 172.21.15.1:0/846074406 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2290278236 0 0) 0x7f39b80acfb0 con 0x7f39b806a750 2024-03-18T20:07:15.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.835+0000 7f39b6ffd700 1 -- 172.21.15.1:0/846074406 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a00035b0 con 0x7f39b806a750 2024-03-18T20:07:15.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.836+0000 7f39bcf3e700 1 -- 172.21.15.1:0/846074406 >> v2:172.21.15.165:3300/0 conn(0x7f39b815d6b0 msgr2=0x7f39b80b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:15.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.836+0000 7f39bcf3e700 1 --2- 172.21.15.1:0/846074406 >> v2:172.21.15.165:3300/0 conn(0x7f39b815d6b0 0x7f39b80b2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:15.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.836+0000 7f39bcf3e700 1 -- 172.21.15.1:0/846074406 >> v1:172.21.15.1:6789/0 conn(0x7f39b806a750 legacy=0x7f39b806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:15.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.836+0000 7f39bcf3e700 1 -- 172.21.15.1:0/846074406 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b80af1e0 con 0x7f39b80ad790 2024-03-18T20:07:15.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.836+0000 7f39b77fe700 1 --2- 172.21.15.1:0/846074406 >> v2:172.21.15.165:3300/0 conn(0x7f39b815d6b0 0x7f39b80b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:07:15.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.836+0000 7f39bcf3e700 1 --2- 172.21.15.1:0/846074406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b80ad790 0x7f39b80b32e0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f39b80711e0 tx=0x7f39b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94cf8c69b8db38d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:15.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.836+0000 7f39b6ffd700 1 -- 172.21.15.1:0/846074406 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b000e070 con 0x7f39b80ad790 2024-03-18T20:07:15.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.836+0000 7f39b6ffd700 1 -- 172.21.15.1:0/846074406 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b000bb20 con 0x7f39b80ad790 2024-03-18T20:07:15.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.836+0000 7f39b6ffd700 1 -- 172.21.15.1:0/846074406 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b0012430 con 0x7f39b80ad790 2024-03-18T20:07:15.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.836+0000 7f39be9a1700 1 -- 172.21.15.1:0/846074406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b80ad790 msgr2=0x7f39b80b32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:15.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.836+0000 7f39be9a1700 1 --2- 172.21.15.1:0/846074406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b80ad790 0x7f39b80b32e0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f39b80711e0 tx=0x7f39b000b330 comp rx=0 tx=0).stop 2024-03-18T20:07:15.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.836+0000 7f39be9a1700 1 -- 172.21.15.1:0/846074406 shutdown_connections 2024-03-18T20:07:15.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.836+0000 7f39be9a1700 1 --2- 172.21.15.1:0/846074406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b80ad790 0x7f39b80b32e0 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:15.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.836+0000 7f39be9a1700 1 --2- 172.21.15.1:0/846074406 >> v2:172.21.15.165:3300/0 conn(0x7f39b815d6b0 0x7f39b80b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:15.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.836+0000 7f39be9a1700 1 -- 172.21.15.1:0/846074406 >> 172.21.15.1:0/846074406 conn(0x7f39b806b860 msgr2=0x7f39b806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:15.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.836+0000 7f39be9a1700 1 -- 172.21.15.1:0/846074406 shutdown_connections 2024-03-18T20:07:15.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.836+0000 7f39be9a1700 1 -- 172.21.15.1:0/846074406 wait complete. 2024-03-18T20:07:15.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.838+0000 7f39be9a1700 1 Processor -- start 2024-03-18T20:07:15.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.838+0000 7f39be9a1700 1 -- start start 2024-03-18T20:07:15.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39be9a1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b806a750 0x7f39b80cca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:15.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39be9a1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f39b80ad790 0x7f39b81582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:15.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39be9a1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f39b815d6b0 0x7f39b815b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:15.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39be9a1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b81593f0 con 0x7f39b806a750 2024-03-18T20:07:15.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39be9a1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80ccfe0 con 0x7f39b80ad790 2024-03-18T20:07:15.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39be9a1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80cd2c0 con 0x7f39b815d6b0 2024-03-18T20:07:15.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39b7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b806a750 0x7f39b80cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:15.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39b7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b806a750 0x7f39b80cca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37558/0 (socket says 172.21.15.1:37558) 2024-03-18T20:07:15.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39b7fff700 1 -- 172.21.15.1:0/1316341814 learned_addr learned my addr 172.21.15.1:0/1316341814 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:15.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39b77fe700 1 --2- 172.21.15.1:0/1316341814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f39b80ad790 0x7f39b81582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:15.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39bcf3e700 1 --2- 172.21.15.1:0/1316341814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f39b815d6b0 0x7f39b815b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:15.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39b7fff700 1 -- 172.21.15.1:0/1316341814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f39b815d6b0 msgr2=0x7f39b815b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39b7fff700 1 --2- 172.21.15.1:0/1316341814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f39b815d6b0 0x7f39b815b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39b7fff700 1 -- 172.21.15.1:0/1316341814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f39b80ad790 msgr2=0x7f39b81582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39b7fff700 1 --2- 172.21.15.1:0/1316341814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f39b80ad790 0x7f39b81582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39b7fff700 1 -- 172.21.15.1:0/1316341814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b0009a70 con 0x7f39b806a750 2024-03-18T20:07:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39b77fe700 1 --2- 172.21.15.1:0/1316341814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f39b80ad790 0x7f39b81582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39bcf3e700 1 --2- 172.21.15.1:0/1316341814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f39b815d6b0 0x7f39b815b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39b7fff700 1 --2- 172.21.15.1:0/1316341814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b806a750 0x7f39b80cca00 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00c700 tx=0x7f39ac00cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39b4ff9700 1 -- 172.21.15.1:0/1316341814 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac005d70 con 0x7f39b806a750 2024-03-18T20:07:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39b4ff9700 1 -- 172.21.15.1:0/1316341814 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ac0070d0 con 0x7f39b806a750 2024-03-18T20:07:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39be9a1700 1 -- 172.21.15.1:0/1316341814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b80cd5a0 con 0x7f39b806a750 2024-03-18T20:07:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39b4ff9700 1 -- 172.21.15.1:0/1316341814 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac007460 con 0x7f39b806a750 2024-03-18T20:07:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.839+0000 7f39be9a1700 1 -- 172.21.15.1:0/1316341814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b80cbc70 con 0x7f39b806a750 2024-03-18T20:07:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.840+0000 7f39be9a1700 1 -- 172.21.15.1:0/1316341814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b8159610 con 0x7f39b806a750 2024-03-18T20:07:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.841+0000 7f39b4ff9700 1 -- 172.21.15.1:0/1316341814 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f39ac0075c0 con 0x7f39b806a750 2024-03-18T20:07:15.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.841+0000 7f39b4ff9700 1 --2- 172.21.15.1:0/1316341814 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f39a406adb0 0x7f39a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:15.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.841+0000 7f39b4ff9700 1 -- 172.21.15.1:0/1316341814 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f39ac088900 con 0x7f39b806a750 2024-03-18T20:07:15.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.841+0000 7f39b77fe700 1 --2- 172.21.15.1:0/1316341814 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f39a406adb0 0x7f39a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:15.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.842+0000 7f39b77fe700 1 --2- 172.21.15.1:0/1316341814 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f39a406adb0 0x7f39a406d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f39b80cfd00 tx=0x7f39a8008040 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:15.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:15.846+0000 7f39b4ff9700 1 -- 172.21.15.1:0/1316341814 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f39b8159610 con 0x7f39b806a750 2024-03-18T20:07:16.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.051+0000 7f39be9a1700 1 -- 172.21.15.1:0/1316341814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39b8159610 con 0x7f39b806a750 2024-03-18T20:07:16.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.051+0000 7f39b4ff9700 1 -- 172.21.15.1:0/1316341814 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v191) v1 ==== 74+0+12671 (secure 0 0 0) 0x7f39b8159610 con 0x7f39b806a750 2024-03-18T20:07:16.056 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:07:16.056 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":191,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:07:15.239589+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:07:16.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.053+0000 7f39be9a1700 1 -- 172.21.15.1:0/1316341814 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f39a406adb0 msgr2=0x7f39a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:16.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.053+0000 7f39be9a1700 1 --2- 172.21.15.1:0/1316341814 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f39a406adb0 0x7f39a406d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f39b80cfd00 tx=0x7f39a8008040 comp rx=0 tx=0).stop 2024-03-18T20:07:16.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.053+0000 7f39be9a1700 1 -- 172.21.15.1:0/1316341814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b806a750 msgr2=0x7f39b80cca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:16.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.053+0000 7f39be9a1700 1 --2- 172.21.15.1:0/1316341814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b806a750 0x7f39b80cca00 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00c700 tx=0x7f39ac00cac0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.053+0000 7f39be9a1700 1 -- 172.21.15.1:0/1316341814 shutdown_connections 2024-03-18T20:07:16.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.053+0000 7f39be9a1700 1 --2- 172.21.15.1:0/1316341814 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f39a406adb0 0x7f39a406d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.053+0000 7f39be9a1700 1 --2- 172.21.15.1:0/1316341814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f39b815d6b0 0x7f39b815b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.053+0000 7f39be9a1700 1 --2- 172.21.15.1:0/1316341814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f39b806a750 0x7f39b80cca00 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.053+0000 7f39be9a1700 1 --2- 172.21.15.1:0/1316341814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f39b80ad790 0x7f39b81582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.054+0000 7f39be9a1700 1 -- 172.21.15.1:0/1316341814 >> 172.21.15.1:0/1316341814 conn(0x7f39b806b860 msgr2=0x7f39b80b0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:16.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.054+0000 7f39be9a1700 1 -- 172.21.15.1:0/1316341814 shutdown_connections 2024-03-18T20:07:16.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.054+0000 7f39be9a1700 1 -- 172.21.15.1:0/1316341814 wait complete. 2024-03-18T20:07:16.071 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:07:16.071 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:07:16.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.194+0000 7fb627275700 1 Processor -- start 2024-03-18T20:07:16.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.194+0000 7fb627275700 1 -- start start 2024-03-18T20:07:16.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.195+0000 7fb627275700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb620071d30 0x7fb6200add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:16.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.195+0000 7fb627275700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb6200cb9e0 0x7fb6200cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:16.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.195+0000 7fb627275700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6200ce8f0 con 0x7fb62015a1d0 2024-03-18T20:07:16.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.195+0000 7fb627275700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6200cdb00 con 0x7fb6200cb9e0 2024-03-18T20:07:16.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.195+0000 7fb627275700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62015a5b0 con 0x7fb620071d30 2024-03-18T20:07:16.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.195+0000 7fb625011700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb62015a1d0 0x7fb620071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43340/0 (socket says 172.21.15.1:43340) 2024-03-18T20:07:16.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.195+0000 7fb625011700 1 -- 172.21.15.1:0/843376785 learned_addr learned my addr 172.21.15.1:0/843376785 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:16.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.196+0000 7fb625812700 1 --2- 172.21.15.1:0/843376785 >> v2:172.21.15.165:3300/0 conn(0x7fb6200cb9e0 0x7fb6200cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:16.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.196+0000 7fb624810700 1 --2- 172.21.15.1:0/843376785 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb620071d30 0x7fb6200add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:16.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.196+0000 7fb617fff700 1 -- 172.21.15.1:0/843376785 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298887137 0 0) 0x7fb6200ce8f0 con 0x7fb62015a1d0 2024-03-18T20:07:16.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.196+0000 7fb617fff700 1 -- 172.21.15.1:0/843376785 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb60c0035b0 con 0x7fb62015a1d0 2024-03-18T20:07:16.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.196+0000 7fb624810700 1 -- 172.21.15.1:0/843376785 >> v2:172.21.15.165:3300/0 conn(0x7fb6200cb9e0 msgr2=0x7fb6200cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:16.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.196+0000 7fb624810700 1 --2- 172.21.15.1:0/843376785 >> v2:172.21.15.165:3300/0 conn(0x7fb6200cb9e0 0x7fb6200cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.196+0000 7fb624810700 1 -- 172.21.15.1:0/843376785 >> v1:172.21.15.1:6789/0 conn(0x7fb62015a1d0 legacy=0x7fb620071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:16.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.196+0000 7fb617fff700 1 -- 172.21.15.1:0/843376785 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3018342206 0 0) 0x7fb60c0035b0 con 0x7fb62015a1d0 2024-03-18T20:07:16.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.196+0000 7fb624810700 1 -- 172.21.15.1:0/843376785 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6200ae260 con 0x7fb620071d30 2024-03-18T20:07:16.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.196+0000 7fb625812700 1 --2- 172.21.15.1:0/843376785 >> v2:172.21.15.165:3300/0 conn(0x7fb6200cb9e0 0x7fb6200cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:07:16.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.196+0000 7fb624810700 1 --2- 172.21.15.1:0/843376785 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb620071d30 0x7fb6200add20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009840 tx=0x7fb61c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba8bffdb9068563 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:16.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.196+0000 7fb617fff700 1 -- 172.21.15.1:0/843376785 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6100017f0 con 0x7fb620071d30 2024-03-18T20:07:16.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.196+0000 7fb617fff700 1 -- 172.21.15.1:0/843376785 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb61c004030 con 0x7fb620071d30 2024-03-18T20:07:16.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.197+0000 7fb617fff700 1 -- 172.21.15.1:0/843376785 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c0043a0 con 0x7fb620071d30 2024-03-18T20:07:16.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.197+0000 7fb627275700 1 -- 172.21.15.1:0/843376785 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb620071d30 msgr2=0x7fb6200add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:16.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.197+0000 7fb627275700 1 --2- 172.21.15.1:0/843376785 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb620071d30 0x7fb6200add20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009840 tx=0x7fb61c009b50 comp rx=0 tx=0).stop 2024-03-18T20:07:16.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.197+0000 7fb627275700 1 -- 172.21.15.1:0/843376785 shutdown_connections 2024-03-18T20:07:16.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.197+0000 7fb627275700 1 --2- 172.21.15.1:0/843376785 >> v2:172.21.15.165:3300/0 conn(0x7fb6200cb9e0 0x7fb6200cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.197+0000 7fb627275700 1 --2- 172.21.15.1:0/843376785 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb620071d30 0x7fb6200add20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.197+0000 7fb627275700 1 -- 172.21.15.1:0/843376785 >> 172.21.15.1:0/843376785 conn(0x7fb6200cc9b0 msgr2=0x7fb6200ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:16.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.197+0000 7fb627275700 1 -- 172.21.15.1:0/843376785 shutdown_connections 2024-03-18T20:07:16.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.197+0000 7fb627275700 1 -- 172.21.15.1:0/843376785 wait complete. 2024-03-18T20:07:16.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.198+0000 7fb627275700 1 Processor -- start 2024-03-18T20:07:16.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.198+0000 7fb627275700 1 -- start start 2024-03-18T20:07:16.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.199+0000 7fb627275700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb620071d30 0x7fb6200c5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:16.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.199+0000 7fb627275700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6200cb9e0 0x7fb6200b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:16.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.199+0000 7fb627275700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb62015a1d0 0x7fb6201e6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:16.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.199+0000 7fb627275700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6200b5530 con 0x7fb6200cb9e0 2024-03-18T20:07:16.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.199+0000 7fb627275700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62006e350 con 0x7fb620071d30 2024-03-18T20:07:16.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.199+0000 7fb627275700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62006e630 con 0x7fb62015a1d0 2024-03-18T20:07:16.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.199+0000 7fb624810700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6200cb9e0 0x7fb6200b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:16.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.199+0000 7fb624810700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6200cb9e0 0x7fb6200b2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37572/0 (socket says 172.21.15.1:37572) 2024-03-18T20:07:16.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.199+0000 7fb624810700 1 -- 172.21.15.1:0/3449139944 learned_addr learned my addr 172.21.15.1:0/3449139944 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:16.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.199+0000 7fb625011700 1 --2- 172.21.15.1:0/3449139944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb620071d30 0x7fb6200c5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:16.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.199+0000 7fb624810700 1 -- 172.21.15.1:0/3449139944 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb62015a1d0 msgr2=0x7fb6201e6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:16.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.199+0000 7fb625812700 1 --2- 172.21.15.1:0/3449139944 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb62015a1d0 0x7fb6201e6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:16.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.199+0000 7fb624810700 1 --2- 172.21.15.1:0/3449139944 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb62015a1d0 0x7fb6201e6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.200+0000 7fb624810700 1 -- 172.21.15.1:0/3449139944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb620071d30 msgr2=0x7fb6200c5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:16.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.200+0000 7fb624810700 1 --2- 172.21.15.1:0/3449139944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb620071d30 0x7fb6200c5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.200+0000 7fb624810700 1 -- 172.21.15.1:0/3449139944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb618009a70 con 0x7fb6200cb9e0 2024-03-18T20:07:16.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.200+0000 7fb625011700 1 --2- 172.21.15.1:0/3449139944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb620071d30 0x7fb6200c5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:16.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.200+0000 7fb624810700 1 --2- 172.21.15.1:0/3449139944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6200cb9e0 0x7fb6200b2ba0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009b90 tx=0x7fb61c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:16.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.200+0000 7fb615ffb700 1 -- 172.21.15.1:0/3449139944 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c004590 con 0x7fb6200cb9e0 2024-03-18T20:07:16.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.200+0000 7fb615ffb700 1 -- 172.21.15.1:0/3449139944 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb61c01f070 con 0x7fb6200cb9e0 2024-03-18T20:07:16.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.200+0000 7fb627275700 1 -- 172.21.15.1:0/3449139944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb61c0094a0 con 0x7fb6200cb9e0 2024-03-18T20:07:16.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.200+0000 7fb615ffb700 1 -- 172.21.15.1:0/3449139944 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c011870 con 0x7fb6200cb9e0 2024-03-18T20:07:16.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.200+0000 7fb627275700 1 -- 172.21.15.1:0/3449139944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb62006ebf0 con 0x7fb6200cb9e0 2024-03-18T20:07:16.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.201+0000 7fb627275700 1 -- 172.21.15.1:0/3449139944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6200676a0 con 0x7fb6200cb9e0 2024-03-18T20:07:16.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.202+0000 7fb615ffb700 1 -- 172.21.15.1:0/3449139944 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb61c0119d0 con 0x7fb6200cb9e0 2024-03-18T20:07:16.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.202+0000 7fb615ffb700 1 --2- 172.21.15.1:0/3449139944 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb60406adb0 0x7fb60406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:16.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.202+0000 7fb615ffb700 1 -- 172.21.15.1:0/3449139944 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb61c021070 con 0x7fb6200cb9e0 2024-03-18T20:07:16.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.202+0000 7fb625011700 1 --2- 172.21.15.1:0/3449139944 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb60406adb0 0x7fb60406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:16.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.203+0000 7fb625011700 1 --2- 172.21.15.1:0/3449139944 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb60406adb0 0x7fb60406d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb610000c00 tx=0x7fb610009040 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:16.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.207+0000 7fb615ffb700 1 -- 172.21.15.1:0/3449139944 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb61c055f70 con 0x7fb6200cb9e0 2024-03-18T20:07:16.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.409+0000 7fb627275700 1 -- 172.21.15.1:0/3449139944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb6200ad300 con 0x7fb6200cb9e0 2024-03-18T20:07:16.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.410+0000 7fb615ffb700 1 -- 172.21.15.1:0/3449139944 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v191) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb61c059590 con 0x7fb6200cb9e0 2024-03-18T20:07:16.415 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:07:16.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.412+0000 7fb627275700 1 -- 172.21.15.1:0/3449139944 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb60406adb0 msgr2=0x7fb60406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:16.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.412+0000 7fb627275700 1 --2- 172.21.15.1:0/3449139944 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb60406adb0 0x7fb60406d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb610000c00 tx=0x7fb610009040 comp rx=0 tx=0).stop 2024-03-18T20:07:16.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.412+0000 7fb627275700 1 -- 172.21.15.1:0/3449139944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6200cb9e0 msgr2=0x7fb6200b2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:16.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.412+0000 7fb627275700 1 --2- 172.21.15.1:0/3449139944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6200cb9e0 0x7fb6200b2ba0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009b90 tx=0x7fb61c004750 comp rx=0 tx=0).stop 2024-03-18T20:07:16.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.412+0000 7fb627275700 1 -- 172.21.15.1:0/3449139944 shutdown_connections 2024-03-18T20:07:16.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.412+0000 7fb627275700 1 --2- 172.21.15.1:0/3449139944 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb60406adb0 0x7fb60406d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.412+0000 7fb627275700 1 --2- 172.21.15.1:0/3449139944 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb62015a1d0 0x7fb6201e6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.412+0000 7fb627275700 1 --2- 172.21.15.1:0/3449139944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb620071d30 0x7fb6200c5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.412+0000 7fb627275700 1 --2- 172.21.15.1:0/3449139944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6200cb9e0 0x7fb6200b2ba0 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.412+0000 7fb627275700 1 -- 172.21.15.1:0/3449139944 >> 172.21.15.1:0/3449139944 conn(0x7fb6200cc9b0 msgr2=0x7fb6200afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:16.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.412+0000 7fb627275700 1 -- 172.21.15.1:0/3449139944 shutdown_connections 2024-03-18T20:07:16.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.412+0000 7fb627275700 1 -- 172.21.15.1:0/3449139944 wait complete. 2024-03-18T20:07:16.429 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T20:07:16.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.549+0000 7f672783f700 1 Processor -- start 2024-03-18T20:07:16.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.550+0000 7f672783f700 1 -- start start 2024-03-18T20:07:16.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.550+0000 7f672783f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f672006a750 0x7f672006ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:16.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.550+0000 7f672783f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f672015d6b0 0x7f6720160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:16.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.551+0000 7f672783f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67200acfb0 con 0x7f67200ad790 2024-03-18T20:07:16.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.551+0000 7f672783f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67200af7a0 con 0x7f672006a750 2024-03-18T20:07:16.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.551+0000 7f672783f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f672006b070 con 0x7f672015d6b0 2024-03-18T20:07:16.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.551+0000 7f67255db700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f672006a750 0x7f672006ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:16.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.551+0000 7f6725ddc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f67200ad790 0x7f6720161530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43348/0 (socket says 172.21.15.1:43348) 2024-03-18T20:07:16.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.551+0000 7f6725ddc700 1 -- 172.21.15.1:0/2735618609 learned_addr learned my addr 172.21.15.1:0/2735618609 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:16.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.551+0000 7f6724dda700 1 --2- 172.21.15.1:0/2735618609 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f672015d6b0 0x7f6720160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:16.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.551+0000 7f67255db700 1 -- 172.21.15.1:0/2735618609 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f672015d6b0 msgr2=0x7f6720160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:16.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.551+0000 7f67255db700 1 --2- 172.21.15.1:0/2735618609 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f672015d6b0 0x7f6720160ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.551+0000 7f67255db700 1 -- 172.21.15.1:0/2735618609 >> v1:172.21.15.1:6789/0 conn(0x7f67200ad790 legacy=0x7f6720161530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:16.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.551+0000 7f6717fff700 1 -- 172.21.15.1:0/2735618609 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292664073 0 0) 0x7f67200acfb0 con 0x7f67200ad790 2024-03-18T20:07:16.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.551+0000 7f67255db700 1 -- 172.21.15.1:0/2735618609 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67200b50a0 con 0x7f672006a750 2024-03-18T20:07:16.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.551+0000 7f67255db700 1 --2- 172.21.15.1:0/2735618609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f672006a750 0x7f672006ab30 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f6710009840 tx=0x7f6710009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af475bac2ead17c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:16.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.551+0000 7f6717fff700 1 -- 172.21.15.1:0/2735618609 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6710011830 con 0x7f672006a750 2024-03-18T20:07:16.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.551+0000 7f6717fff700 1 -- 172.21.15.1:0/2735618609 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6710011990 con 0x7f672006a750 2024-03-18T20:07:16.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.551+0000 7f6717fff700 1 -- 172.21.15.1:0/2735618609 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6710011d20 con 0x7f672006a750 2024-03-18T20:07:16.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.552+0000 7f672783f700 1 -- 172.21.15.1:0/2735618609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f672006a750 msgr2=0x7f672006ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:16.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.552+0000 7f672783f700 1 --2- 172.21.15.1:0/2735618609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f672006a750 0x7f672006ab30 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f6710009840 tx=0x7f6710009b50 comp rx=0 tx=0).stop 2024-03-18T20:07:16.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.552+0000 7f672783f700 1 -- 172.21.15.1:0/2735618609 shutdown_connections 2024-03-18T20:07:16.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.552+0000 7f672783f700 1 --2- 172.21.15.1:0/2735618609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f672006a750 0x7f672006ab30 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.552+0000 7f672783f700 1 --2- 172.21.15.1:0/2735618609 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f672015d6b0 0x7f6720160ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.552+0000 7f672783f700 1 -- 172.21.15.1:0/2735618609 >> 172.21.15.1:0/2735618609 conn(0x7f672006b860 msgr2=0x7f672006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:16.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.552+0000 7f672783f700 1 -- 172.21.15.1:0/2735618609 shutdown_connections 2024-03-18T20:07:16.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.552+0000 7f672783f700 1 -- 172.21.15.1:0/2735618609 wait complete. 2024-03-18T20:07:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.553+0000 7f672783f700 1 Processor -- start 2024-03-18T20:07:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.553+0000 7f672783f700 1 -- start start 2024-03-18T20:07:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.554+0000 7f672783f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f672006a750 0x7f6720152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.554+0000 7f672783f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67200ad790 0x7f6720156c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.554+0000 7f672783f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f672015d6b0 0x7f67201516c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.554+0000 7f672783f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67200b1660 con 0x7f672015d6b0 2024-03-18T20:07:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.554+0000 7f672783f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67201f3e90 con 0x7f67200ad790 2024-03-18T20:07:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.554+0000 7f672783f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6720152a90 con 0x7f672006a750 2024-03-18T20:07:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.554+0000 7f6725ddc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f672015d6b0 0x7f67201516c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.554+0000 7f6725ddc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f672015d6b0 0x7f67201516c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37586/0 (socket says 172.21.15.1:37586) 2024-03-18T20:07:16.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.554+0000 7f6725ddc700 1 -- 172.21.15.1:0/2409678073 learned_addr learned my addr 172.21.15.1:0/2409678073 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.554+0000 7f67255db700 1 --2- 172.21.15.1:0/2409678073 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f672006a750 0x7f6720152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.554+0000 7f6724dda700 1 --2- 172.21.15.1:0/2409678073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67200ad790 0x7f6720156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.554+0000 7f6725ddc700 1 -- 172.21.15.1:0/2409678073 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f672006a750 msgr2=0x7f6720152f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.554+0000 7f6725ddc700 1 --2- 172.21.15.1:0/2409678073 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f672006a750 0x7f6720152f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.555+0000 7f6725ddc700 1 -- 172.21.15.1:0/2409678073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67200ad790 msgr2=0x7f6720156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.555+0000 7f6725ddc700 1 --2- 172.21.15.1:0/2409678073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67200ad790 0x7f6720156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.555+0000 7f6725ddc700 1 -- 172.21.15.1:0/2409678073 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67100094a0 con 0x7f672015d6b0 2024-03-18T20:07:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.555+0000 7f6725ddc700 1 --2- 172.21.15.1:0/2409678073 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f672015d6b0 0x7f67201516c0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f6718002230 tx=0x7f671800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.555+0000 7f6715ffb700 1 -- 172.21.15.1:0/2409678073 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671800ac20 con 0x7f672015d6b0 2024-03-18T20:07:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.555+0000 7f6715ffb700 1 -- 172.21.15.1:0/2409678073 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f671800ada0 con 0x7f672015d6b0 2024-03-18T20:07:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.555+0000 7f672783f700 1 -- 172.21.15.1:0/2409678073 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67201f3bb0 con 0x7f672015d6b0 2024-03-18T20:07:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.555+0000 7f6715ffb700 1 -- 172.21.15.1:0/2409678073 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6718007310 con 0x7f672015d6b0 2024-03-18T20:07:16.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.555+0000 7f672783f700 1 -- 172.21.15.1:0/2409678073 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6720152720 con 0x7f672015d6b0 2024-03-18T20:07:16.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.556+0000 7f672783f700 1 -- 172.21.15.1:0/2409678073 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67200ad420 con 0x7f672015d6b0 2024-03-18T20:07:16.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.556+0000 7f6715ffb700 1 -- 172.21.15.1:0/2409678073 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6718007470 con 0x7f672015d6b0 2024-03-18T20:07:16.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.557+0000 7f6715ffb700 1 --2- 172.21.15.1:0/2409678073 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f670406adb0 0x7f670406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:16.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.557+0000 7f6715ffb700 1 -- 172.21.15.1:0/2409678073 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f6718089110 con 0x7f672015d6b0 2024-03-18T20:07:16.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.557+0000 7f67255db700 1 --2- 172.21.15.1:0/2409678073 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f670406adb0 0x7f670406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:16.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.557+0000 7f67255db700 1 --2- 172.21.15.1:0/2409678073 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f670406adb0 0x7f670406d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f67100090b0 tx=0x7f6710008030 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:16.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.561+0000 7f6715ffb700 1 -- 172.21.15.1:0/2409678073 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f67200ad420 con 0x7f672015d6b0 2024-03-18T20:07:16.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.767+0000 7f672783f700 1 -- 172.21.15.1:0/2409678073 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f67200ad420 con 0x7f672015d6b0 2024-03-18T20:07:16.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.768+0000 7f6715ffb700 1 -- 172.21.15.1:0/2409678073 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v191) v1 ==== 84+0+65 (secure 0 0 0) 0x7f67200ad420 con 0x7f672015d6b0 2024-03-18T20:07:16.773 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:07:16.773 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T20:07:16.773 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:16.773 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-18T20:07:16.773 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-18T20:07:16.773 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:16.773 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T20:07:16.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.770+0000 7f672783f700 1 -- 172.21.15.1:0/2409678073 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f670406adb0 msgr2=0x7f670406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:16.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.770+0000 7f672783f700 1 --2- 172.21.15.1:0/2409678073 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f670406adb0 0x7f670406d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f67100090b0 tx=0x7f6710008030 comp rx=0 tx=0).stop 2024-03-18T20:07:16.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.770+0000 7f672783f700 1 -- 172.21.15.1:0/2409678073 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f672015d6b0 msgr2=0x7f67201516c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:16.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.770+0000 7f672783f700 1 --2- 172.21.15.1:0/2409678073 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f672015d6b0 0x7f67201516c0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f6718002230 tx=0x7f671800a060 comp rx=0 tx=0).stop 2024-03-18T20:07:16.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.770+0000 7f672783f700 1 -- 172.21.15.1:0/2409678073 shutdown_connections 2024-03-18T20:07:16.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.770+0000 7f672783f700 1 --2- 172.21.15.1:0/2409678073 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f670406adb0 0x7f670406d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.770+0000 7f672783f700 1 --2- 172.21.15.1:0/2409678073 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f672006a750 0x7f6720152f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.770+0000 7f672783f700 1 --2- 172.21.15.1:0/2409678073 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f672015d6b0 0x7f67201516c0 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.770+0000 7f672783f700 1 --2- 172.21.15.1:0/2409678073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67200ad790 0x7f6720156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.770+0000 7f672783f700 1 -- 172.21.15.1:0/2409678073 >> 172.21.15.1:0/2409678073 conn(0x7f672006b860 msgr2=0x7f672015c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:16.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.770+0000 7f672783f700 1 -- 172.21.15.1:0/2409678073 shutdown_connections 2024-03-18T20:07:16.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.770+0000 7f672783f700 1 -- 172.21.15.1:0/2409678073 wait complete. 2024-03-18T20:07:16.787 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T20:07:16.787 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T20:07:16.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.912+0000 7f0d3fa4a700 1 Processor -- start 2024-03-18T20:07:16.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.912+0000 7f0d3fa4a700 1 -- start start 2024-03-18T20:07:16.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.913+0000 7f0d3fa4a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0d3815a160 0x7f0d380b3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:16.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.913+0000 7f0d3fa4a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0d3815e460 0x7f0d380717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:16.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.913+0000 7f0d3fa4a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d380b4d10 con 0x7f0d3806f7d0 2024-03-18T20:07:16.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.913+0000 7f0d3cfe5700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0d3806f7d0 0x7f0d3815dd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43364/0 (socket says 172.21.15.1:43364) 2024-03-18T20:07:16.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.913+0000 7f0d3fa4a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d3815ec10 con 0x7f0d3815a160 2024-03-18T20:07:16.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.913+0000 7f0d3cfe5700 1 -- 172.21.15.1:0/574944696 learned_addr learned my addr 172.21.15.1:0/574944696 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:16.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.913+0000 7f0d3fa4a700 1 -- 172.21.15.1:0/574944696 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d380b4a30 con 0x7f0d3815e460 2024-03-18T20:07:16.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.913+0000 7f0d3d7e6700 1 --2- 172.21.15.1:0/574944696 >> v2:172.21.15.1:3300/0 conn(0x7f0d3815a160 0x7f0d380b3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:16.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.913+0000 7f0d3dfe7700 1 --2- 172.21.15.1:0/574944696 >> v2:172.21.15.165:3300/0 conn(0x7f0d3815e460 0x7f0d380717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:16.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.913+0000 7f0d2ffff700 1 -- 172.21.15.1:0/574944696 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392654203 0 0) 0x7f0d380b4d10 con 0x7f0d3806f7d0 2024-03-18T20:07:16.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.913+0000 7f0d2ffff700 1 -- 172.21.15.1:0/574944696 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d240035b0 con 0x7f0d3806f7d0 2024-03-18T20:07:16.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.913+0000 7f0d3d7e6700 1 -- 172.21.15.1:0/574944696 >> v2:172.21.15.165:3300/0 conn(0x7f0d3815e460 msgr2=0x7f0d380717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:16.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.913+0000 7f0d3d7e6700 1 --2- 172.21.15.1:0/574944696 >> v2:172.21.15.165:3300/0 conn(0x7f0d3815e460 0x7f0d380717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.913+0000 7f0d3d7e6700 1 -- 172.21.15.1:0/574944696 >> v1:172.21.15.1:6789/0 conn(0x7f0d3806f7d0 legacy=0x7f0d3815dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:16.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.913+0000 7f0d3d7e6700 1 -- 172.21.15.1:0/574944696 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3806ecf0 con 0x7f0d3815a160 2024-03-18T20:07:16.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.914+0000 7f0d3d7e6700 1 --2- 172.21.15.1:0/574944696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d3815a160 0x7f0d380b3a90 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f0d380cc2d0 tx=0x7f0d3400d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0a9443a4f12a44 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:16.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.914+0000 7f0d2ffff700 1 -- 172.21.15.1:0/574944696 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d34010070 con 0x7f0d3815a160 2024-03-18T20:07:16.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.914+0000 7f0d2ffff700 1 -- 172.21.15.1:0/574944696 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3400db20 con 0x7f0d3815a160 2024-03-18T20:07:16.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.914+0000 7f0d2ffff700 1 -- 172.21.15.1:0/574944696 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3400de90 con 0x7f0d3815a160 2024-03-18T20:07:16.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.914+0000 7f0d3fa4a700 1 -- 172.21.15.1:0/574944696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d3815a160 msgr2=0x7f0d380b3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:16.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.914+0000 7f0d3fa4a700 1 --2- 172.21.15.1:0/574944696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d3815a160 0x7f0d380b3a90 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f0d380cc2d0 tx=0x7f0d3400d330 comp rx=0 tx=0).stop 2024-03-18T20:07:16.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.914+0000 7f0d3fa4a700 1 -- 172.21.15.1:0/574944696 shutdown_connections 2024-03-18T20:07:16.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.914+0000 7f0d3fa4a700 1 --2- 172.21.15.1:0/574944696 >> v2:172.21.15.165:3300/0 conn(0x7f0d3815e460 0x7f0d380717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.914+0000 7f0d3fa4a700 1 --2- 172.21.15.1:0/574944696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d3815a160 0x7f0d380b3a90 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.914+0000 7f0d3fa4a700 1 -- 172.21.15.1:0/574944696 >> 172.21.15.1:0/574944696 conn(0x7f0d380d00c0 msgr2=0x7f0d380d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:16.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.914+0000 7f0d3fa4a700 1 -- 172.21.15.1:0/574944696 shutdown_connections 2024-03-18T20:07:16.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.914+0000 7f0d3fa4a700 1 -- 172.21.15.1:0/574944696 wait complete. 2024-03-18T20:07:16.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.916+0000 7f0d3fa4a700 1 Processor -- start 2024-03-18T20:07:16.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.916+0000 7f0d3fa4a700 1 -- start start 2024-03-18T20:07:16.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.916+0000 7f0d3fa4a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d3806f7d0 0x7f0d3806df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:16.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.916+0000 7f0d3fa4a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d3815a160 0x7f0d38152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:16.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.916+0000 7f0d3fa4a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d3815e460 0x7f0d38152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:16.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.916+0000 7f0d3fa4a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d38158440 con 0x7f0d3806f7d0 2024-03-18T20:07:16.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.916+0000 7f0d3fa4a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d381eee00 con 0x7f0d3815a160 2024-03-18T20:07:16.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.916+0000 7f0d3fa4a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d380c5350 con 0x7f0d3815e460 2024-03-18T20:07:16.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.916+0000 7f0d3d7e6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d3806f7d0 0x7f0d3806df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:16.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.916+0000 7f0d3d7e6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d3806f7d0 0x7f0d3806df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37614/0 (socket says 172.21.15.1:37614) 2024-03-18T20:07:16.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.916+0000 7f0d3d7e6700 1 -- 172.21.15.1:0/2482697344 learned_addr learned my addr 172.21.15.1:0/2482697344 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:16.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.917+0000 7f0d3dfe7700 1 --2- 172.21.15.1:0/2482697344 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d3815e460 0x7f0d38152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:16.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.917+0000 7f0d3cfe5700 1 --2- 172.21.15.1:0/2482697344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d3815a160 0x7f0d38152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:16.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.917+0000 7f0d3d7e6700 1 -- 172.21.15.1:0/2482697344 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d3815e460 msgr2=0x7f0d38152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:16.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.917+0000 7f0d3d7e6700 1 --2- 172.21.15.1:0/2482697344 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d3815e460 0x7f0d38152d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.917+0000 7f0d3d7e6700 1 -- 172.21.15.1:0/2482697344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d3815a160 msgr2=0x7f0d38152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:16.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.917+0000 7f0d3d7e6700 1 --2- 172.21.15.1:0/2482697344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d3815a160 0x7f0d38152850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:16.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.917+0000 7f0d3d7e6700 1 -- 172.21.15.1:0/2482697344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3400ba70 con 0x7f0d3806f7d0 2024-03-18T20:07:16.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.917+0000 7f0d3cfe5700 1 --2- 172.21.15.1:0/2482697344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d3815a160 0x7f0d38152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:16.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.917+0000 7f0d3d7e6700 1 --2- 172.21.15.1:0/2482697344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d3806f7d0 0x7f0d3806df50 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400d370 tx=0x7f0d34004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:16.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.917+0000 7f0d2dffb700 1 -- 172.21.15.1:0/2482697344 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d34009000 con 0x7f0d3806f7d0 2024-03-18T20:07:16.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.917+0000 7f0d2dffb700 1 -- 172.21.15.1:0/2482697344 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d34014630 con 0x7f0d3806f7d0 2024-03-18T20:07:16.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.917+0000 7f0d3fa4a700 1 -- 172.21.15.1:0/2482697344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d380c5630 con 0x7f0d3806f7d0 2024-03-18T20:07:16.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.917+0000 7f0d2dffb700 1 -- 172.21.15.1:0/2482697344 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d340149e0 con 0x7f0d3806f7d0 2024-03-18T20:07:16.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.917+0000 7f0d3fa4a700 1 -- 172.21.15.1:0/2482697344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d380c5bf0 con 0x7f0d3806f7d0 2024-03-18T20:07:16.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.918+0000 7f0d3fa4a700 1 -- 172.21.15.1:0/2482697344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d380676a0 con 0x7f0d3806f7d0 2024-03-18T20:07:16.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.919+0000 7f0d2dffb700 1 -- 172.21.15.1:0/2482697344 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0d34014b40 con 0x7f0d3806f7d0 2024-03-18T20:07:16.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.919+0000 7f0d2dffb700 1 --2- 172.21.15.1:0/2482697344 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0d1c06adb0 0x7f0d1c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:16.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.919+0000 7f0d2dffb700 1 -- 172.21.15.1:0/2482697344 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f0d3408d1b0 con 0x7f0d3806f7d0 2024-03-18T20:07:16.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.919+0000 7f0d3cfe5700 1 --2- 172.21.15.1:0/2482697344 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0d1c06adb0 0x7f0d1c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:16.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.920+0000 7f0d3cfe5700 1 --2- 172.21.15.1:0/2482697344 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0d1c06adb0 0x7f0d1c06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0d380b1d20 tx=0x7f0d28009040 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:16.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:16.924+0000 7f0d2dffb700 1 -- 172.21.15.1:0/2482697344 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0d34058470 con 0x7f0d3806f7d0 2024-03-18T20:07:17.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:17.128+0000 7f0d3fa4a700 1 -- 172.21.15.1:0/2482697344 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0d38072a90 con 0x7f0d3806f7d0 2024-03-18T20:07:17.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:17.885+0000 7f0d2dffb700 1 -- 172.21.15.1:0/2482697344 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v192) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0d34003e60 con 0x7f0d3806f7d0 2024-03-18T20:07:17.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:17.890+0000 7f0d3fa4a700 1 -- 172.21.15.1:0/2482697344 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0d1c06adb0 msgr2=0x7f0d1c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:17.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:17.890+0000 7f0d3fa4a700 1 --2- 172.21.15.1:0/2482697344 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0d1c06adb0 0x7f0d1c06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0d380b1d20 tx=0x7f0d28009040 comp rx=0 tx=0).stop 2024-03-18T20:07:17.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:17.890+0000 7f0d3fa4a700 1 -- 172.21.15.1:0/2482697344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d3806f7d0 msgr2=0x7f0d3806df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:17.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:17.890+0000 7f0d3fa4a700 1 --2- 172.21.15.1:0/2482697344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d3806f7d0 0x7f0d3806df50 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400d370 tx=0x7f0d34004750 comp rx=0 tx=0).stop 2024-03-18T20:07:17.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:17.890+0000 7f0d3fa4a700 1 -- 172.21.15.1:0/2482697344 shutdown_connections 2024-03-18T20:07:17.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:17.890+0000 7f0d3fa4a700 1 --2- 172.21.15.1:0/2482697344 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0d1c06adb0 0x7f0d1c06d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:17.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:17.890+0000 7f0d3fa4a700 1 --2- 172.21.15.1:0/2482697344 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0d3815e460 0x7f0d38152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:17.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:17.890+0000 7f0d3fa4a700 1 --2- 172.21.15.1:0/2482697344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d3806f7d0 0x7f0d3806df50 secure :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400d370 tx=0x7f0d34004750 comp rx=0 tx=0).stop 2024-03-18T20:07:17.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:17.890+0000 7f0d3fa4a700 1 --2- 172.21.15.1:0/2482697344 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d3815a160 0x7f0d38152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:17.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:17.890+0000 7f0d3fa4a700 1 -- 172.21.15.1:0/2482697344 >> 172.21.15.1:0/2482697344 conn(0x7f0d380d00c0 msgr2=0x7f0d3815dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:17.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:17.893+0000 7f0d3fa4a700 1 -- 172.21.15.1:0/2482697344 shutdown_connections 2024-03-18T20:07:17.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:17.893+0000 7f0d3fa4a700 1 -- 172.21.15.1:0/2482697344 wait complete. 2024-03-18T20:07:17.899 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-18T20:07:17.913 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T20:07:18.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.048+0000 7fbf3efc7700 1 Processor -- start 2024-03-18T20:07:18.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.048+0000 7fbf3efc7700 1 -- start start 2024-03-18T20:07:18.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.048+0000 7fbf3efc7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf380b0500 0x7fbf380cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:18.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.048+0000 7fbf3efc7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf3815ef80 con 0x7fbf3815eba0 2024-03-18T20:07:18.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.049+0000 7fbf3efc7700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf380cc9a0 con 0x7fbf380b0e20 2024-03-18T20:07:18.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.049+0000 7fbf3efc7700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf380ccc80 con 0x7fbf380b0500 2024-03-18T20:07:18.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.049+0000 7fbf37fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf380b0500 0x7fbf380cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:18.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.049+0000 7fbf3cd63700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbf3815eba0 0x7fbf380cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43368/0 (socket says 172.21.15.1:43368) 2024-03-18T20:07:18.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.049+0000 7fbf3cd63700 1 -- 172.21.15.1:0/286872188 learned_addr learned my addr 172.21.15.1:0/286872188 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:18.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.049+0000 7fbf37fff700 1 -- 172.21.15.1:0/286872188 >> v1:172.21.15.165:6789/0 conn(0x7fbf380b0e20 legacy=0x7fbf380b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:18.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.049+0000 7fbf37fff700 1 -- 172.21.15.1:0/286872188 >> v1:172.21.15.1:6789/0 conn(0x7fbf3815eba0 legacy=0x7fbf380cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:18.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.049+0000 7fbf37fff700 1 -- 172.21.15.1:0/286872188 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf380ccf60 con 0x7fbf380b0500 2024-03-18T20:07:18.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.049+0000 7fbf37fff700 1 --2- 172.21.15.1:0/286872188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf380b0500 0x7fbf380cfa80 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fbf24009840 tx=0x7fbf24009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb9c394d0c7ebb12 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:18.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.049+0000 7fbf377fe700 1 -- 172.21.15.1:0/286872188 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf24004230 con 0x7fbf380b0500 2024-03-18T20:07:18.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.049+0000 7fbf377fe700 1 -- 172.21.15.1:0/286872188 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf24004390 con 0x7fbf380b0500 2024-03-18T20:07:18.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.049+0000 7fbf377fe700 1 -- 172.21.15.1:0/286872188 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf240036a0 con 0x7fbf380b0500 2024-03-18T20:07:18.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.050+0000 7fbf3efc7700 1 -- 172.21.15.1:0/286872188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf380b0500 msgr2=0x7fbf380cfa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:18.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.050+0000 7fbf3efc7700 1 --2- 172.21.15.1:0/286872188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf380b0500 0x7fbf380cfa80 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fbf24009840 tx=0x7fbf24009b50 comp rx=0 tx=0).stop 2024-03-18T20:07:18.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.050+0000 7fbf3efc7700 1 -- 172.21.15.1:0/286872188 shutdown_connections 2024-03-18T20:07:18.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.050+0000 7fbf3efc7700 1 --2- 172.21.15.1:0/286872188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf380b0500 0x7fbf380cfa80 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:18.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.050+0000 7fbf3efc7700 1 -- 172.21.15.1:0/286872188 >> 172.21.15.1:0/286872188 conn(0x7fbf380b2bd0 msgr2=0x7fbf380b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:18.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.050+0000 7fbf3efc7700 1 -- 172.21.15.1:0/286872188 shutdown_connections 2024-03-18T20:07:18.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.050+0000 7fbf3efc7700 1 -- 172.21.15.1:0/286872188 wait complete. 2024-03-18T20:07:18.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.051+0000 7fbf3efc7700 1 Processor -- start 2024-03-18T20:07:18.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.051+0000 7fbf3efc7700 1 -- start start 2024-03-18T20:07:18.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.052+0000 7fbf3efc7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf380b0500 0x7fbf38159870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.052+0000 7fbf3efc7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf380b0e20 0x7fbf380c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.052+0000 7fbf3efc7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf3815eba0 0x7fbf380c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.052+0000 7fbf3efc7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380c8640 con 0x7fbf380b0e20 2024-03-18T20:07:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.052+0000 7fbf3efc7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380c8920 con 0x7fbf3815eba0 2024-03-18T20:07:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.052+0000 7fbf3efc7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380c8c00 con 0x7fbf380b0500 2024-03-18T20:07:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.052+0000 7fbf37fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf380b0e20 0x7fbf380c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.052+0000 7fbf37fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf380b0e20 0x7fbf380c96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37638/0 (socket says 172.21.15.1:37638) 2024-03-18T20:07:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.052+0000 7fbf37fff700 1 -- 172.21.15.1:0/2763658603 learned_addr learned my addr 172.21.15.1:0/2763658603 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.052+0000 7fbf3d564700 1 --2- 172.21.15.1:0/2763658603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf3815eba0 0x7fbf380c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.052+0000 7fbf3cd63700 1 --2- 172.21.15.1:0/2763658603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf380b0500 0x7fbf38159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.052+0000 7fbf37fff700 1 -- 172.21.15.1:0/2763658603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf380b0500 msgr2=0x7fbf38159870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.052+0000 7fbf37fff700 1 --2- 172.21.15.1:0/2763658603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf380b0500 0x7fbf38159870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.052+0000 7fbf37fff700 1 -- 172.21.15.1:0/2763658603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf3815eba0 msgr2=0x7fbf380c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:18.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.052+0000 7fbf37fff700 1 --2- 172.21.15.1:0/2763658603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf3815eba0 0x7fbf380c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:18.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.052+0000 7fbf37fff700 1 -- 172.21.15.1:0/2763658603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf240094a0 con 0x7fbf380b0e20 2024-03-18T20:07:18.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.052+0000 7fbf3cd63700 1 --2- 172.21.15.1:0/2763658603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf380b0500 0x7fbf38159870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:18.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.053+0000 7fbf37fff700 1 --2- 172.21.15.1:0/2763658603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf380b0e20 0x7fbf380c96e0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fbf24009b90 tx=0x7fbf24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:18.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.053+0000 7fbf357fa700 1 -- 172.21.15.1:0/2763658603 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf24011630 con 0x7fbf380b0e20 2024-03-18T20:07:18.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.053+0000 7fbf357fa700 1 -- 172.21.15.1:0/2763658603 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2401e070 con 0x7fbf380b0e20 2024-03-18T20:07:18.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.053+0000 7fbf3efc7700 1 -- 172.21.15.1:0/2763658603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf380c8ee0 con 0x7fbf380b0e20 2024-03-18T20:07:18.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.053+0000 7fbf357fa700 1 -- 172.21.15.1:0/2763658603 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf240119f0 con 0x7fbf380b0e20 2024-03-18T20:07:18.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.053+0000 7fbf3efc7700 1 -- 172.21.15.1:0/2763658603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf38159db0 con 0x7fbf380b0e20 2024-03-18T20:07:18.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.054+0000 7fbf3efc7700 1 -- 172.21.15.1:0/2763658603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf380c9400 con 0x7fbf380b0e20 2024-03-18T20:07:18.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.055+0000 7fbf357fa700 1 -- 172.21.15.1:0/2763658603 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbf24011b50 con 0x7fbf380b0e20 2024-03-18T20:07:18.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.055+0000 7fbf357fa700 1 --2- 172.21.15.1:0/2763658603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf1c06adb0 0x7fbf1c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:18.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.055+0000 7fbf357fa700 1 -- 172.21.15.1:0/2763658603 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5619+0+0 (secure 0 0 0) 0x7fbf2408ad30 con 0x7fbf380b0e20 2024-03-18T20:07:18.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.055+0000 7fbf3cd63700 1 --2- 172.21.15.1:0/2763658603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf1c06adb0 0x7fbf1c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:18.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.056+0000 7fbf3cd63700 1 --2- 172.21.15.1:0/2763658603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf1c06adb0 0x7fbf1c06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c001800 tx=0x7fbf2c009040 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:18.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.060+0000 7fbf357fa700 1 -- 172.21.15.1:0/2763658603 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbf24055fc0 con 0x7fbf380b0e20 2024-03-18T20:07:18.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.261+0000 7fbf3efc7700 1 -- 172.21.15.1:0/2763658603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fbf380c9400 con 0x7fbf380b0e20 2024-03-18T20:07:18.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.890+0000 7fbf357fa700 1 -- 172.21.15.1:0/2763658603 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v193) v1 ==== 170+0+0 (secure 0 0 0) 0x7fbf380c9400 con 0x7fbf380b0e20 2024-03-18T20:07:18.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.904+0000 7fbf3efc7700 1 -- 172.21.15.1:0/2763658603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf1c06adb0 msgr2=0x7fbf1c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:18.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.904+0000 7fbf3efc7700 1 --2- 172.21.15.1:0/2763658603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf1c06adb0 0x7fbf1c06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c001800 tx=0x7fbf2c009040 comp rx=0 tx=0).stop 2024-03-18T20:07:18.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.905+0000 7fbf3efc7700 1 -- 172.21.15.1:0/2763658603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf380b0e20 msgr2=0x7fbf380c96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:18.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.905+0000 7fbf3efc7700 1 --2- 172.21.15.1:0/2763658603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf380b0e20 0x7fbf380c96e0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fbf24009b90 tx=0x7fbf24004750 comp rx=0 tx=0).stop 2024-03-18T20:07:18.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.910+0000 7fbf3efc7700 1 -- 172.21.15.1:0/2763658603 shutdown_connections 2024-03-18T20:07:18.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.910+0000 7fbf3efc7700 1 --2- 172.21.15.1:0/2763658603 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fbf1c06adb0 0x7fbf1c06d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:18.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.910+0000 7fbf3efc7700 1 --2- 172.21.15.1:0/2763658603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf380b0500 0x7fbf38159870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:18.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.910+0000 7fbf3efc7700 1 --2- 172.21.15.1:0/2763658603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf3815eba0 0x7fbf380c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:18.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.910+0000 7fbf3efc7700 1 --2- 172.21.15.1:0/2763658603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf380b0e20 0x7fbf380c96e0 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:18.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.910+0000 7fbf3efc7700 1 -- 172.21.15.1:0/2763658603 >> 172.21.15.1:0/2763658603 conn(0x7fbf380b2bd0 msgr2=0x7fbf3806ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:18.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.912+0000 7fbf3efc7700 1 -- 172.21.15.1:0/2763658603 shutdown_connections 2024-03-18T20:07:18.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:18.912+0000 7fbf3efc7700 1 -- 172.21.15.1:0/2763658603 wait complete. 2024-03-18T20:07:18.919 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-18T20:07:18.933 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T20:07:19.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.062+0000 7f8d23a18700 1 Processor -- start 2024-03-18T20:07:19.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.062+0000 7f8d23a18700 1 -- start start 2024-03-18T20:07:19.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.063+0000 7f8d23a18700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8d1c15eba0 0x7f8d1c0cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:19.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.063+0000 7f8d23a18700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8d1c0b0500 0x7f8d1c0cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:19.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.063+0000 7f8d23a18700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1c15ef80 con 0x7f8d1c0cfdf0 2024-03-18T20:07:19.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.063+0000 7f8d23a18700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d1c0cc9a0 con 0x7f8d1c0b0500 2024-03-18T20:07:19.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.063+0000 7f8d23a18700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d1c0ccc80 con 0x7f8d1c15eba0 2024-03-18T20:07:19.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.063+0000 7f8d20fb3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8d1c0b0500 0x7f8d1c0cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:19.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.063+0000 7f8d20fb3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8d1c0b0500 0x7f8d1c0cf8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37650/0 (socket says 172.21.15.1:37650) 2024-03-18T20:07:19.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.063+0000 7f8d20fb3700 1 -- 172.21.15.1:0/1944243392 learned_addr learned my addr 172.21.15.1:0/1944243392 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:19.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.063+0000 7f8d217b4700 1 --2- 172.21.15.1:0/1944243392 >> v2:172.21.15.165:3300/0 conn(0x7f8d1c15eba0 0x7f8d1c0cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:19.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.063+0000 7f8d20fb3700 1 -- 172.21.15.1:0/1944243392 >> v2:172.21.15.165:3300/0 conn(0x7f8d1c15eba0 msgr2=0x7f8d1c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:19.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.063+0000 7f8d20fb3700 1 --2- 172.21.15.1:0/1944243392 >> v2:172.21.15.165:3300/0 conn(0x7f8d1c15eba0 0x7f8d1c0cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.063+0000 7f8d20fb3700 1 -- 172.21.15.1:0/1944243392 >> v1:172.21.15.165:6789/0 conn(0x7f8d1c0cfdf0 legacy=0x7f8d1c099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:19.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.063+0000 7f8d20fb3700 1 -- 172.21.15.1:0/1944243392 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1c0ccf60 con 0x7f8d1c0b0500 2024-03-18T20:07:19.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.063+0000 7f8d217b4700 1 --2- 172.21.15.1:0/1944243392 >> v2:172.21.15.165:3300/0 conn(0x7f8d1c15eba0 0x7f8d1c0cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:19.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.063+0000 7f8d20fb3700 1 --2- 172.21.15.1:0/1944243392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d1c0b0500 0x7f8d1c0cf8b0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c072fb0 tx=0x7f8d18009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a325c43d7c7e15e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:19.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.064+0000 7f8d13fff700 1 -- 172.21.15.1:0/1944243392 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d18004230 con 0x7f8d1c0b0500 2024-03-18T20:07:19.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.064+0000 7f8d13fff700 1 -- 172.21.15.1:0/1944243392 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d18004390 con 0x7f8d1c0b0500 2024-03-18T20:07:19.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.064+0000 7f8d13fff700 1 -- 172.21.15.1:0/1944243392 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d180036a0 con 0x7f8d1c0b0500 2024-03-18T20:07:19.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.064+0000 7f8d23a18700 1 -- 172.21.15.1:0/1944243392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d1c0b0500 msgr2=0x7f8d1c0cf8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:19.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.064+0000 7f8d23a18700 1 --2- 172.21.15.1:0/1944243392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d1c0b0500 0x7f8d1c0cf8b0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c072fb0 tx=0x7f8d18009bd0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.064+0000 7f8d23a18700 1 -- 172.21.15.1:0/1944243392 shutdown_connections 2024-03-18T20:07:19.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.064+0000 7f8d23a18700 1 --2- 172.21.15.1:0/1944243392 >> v2:172.21.15.165:3300/0 conn(0x7f8d1c15eba0 0x7f8d1c0cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.064+0000 7f8d23a18700 1 --2- 172.21.15.1:0/1944243392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d1c0b0500 0x7f8d1c0cf8b0 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.064+0000 7f8d23a18700 1 -- 172.21.15.1:0/1944243392 >> 172.21.15.1:0/1944243392 conn(0x7f8d1c0b2bd0 msgr2=0x7f8d1c0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:19.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.064+0000 7f8d23a18700 1 -- 172.21.15.1:0/1944243392 shutdown_connections 2024-03-18T20:07:19.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.064+0000 7f8d23a18700 1 -- 172.21.15.1:0/1944243392 wait complete. 2024-03-18T20:07:19.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.065+0000 7f8d23a18700 1 Processor -- start 2024-03-18T20:07:19.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.065+0000 7f8d23a18700 1 -- start start 2024-03-18T20:07:19.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.066+0000 7f8d23a18700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d1c0b0500 0x7f8d1c157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:19.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.066+0000 7f8d23a18700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8d1c0cfdf0 0x7f8d1c0c49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:19.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.066+0000 7f8d23a18700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8d1c15eba0 0x7f8d1c0c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:19.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.066+0000 7f8d23a18700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c0caa20 con 0x7f8d1c0b0500 2024-03-18T20:07:19.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.066+0000 7f8d23a18700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c157fe0 con 0x7f8d1c0cfdf0 2024-03-18T20:07:19.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.066+0000 7f8d23a18700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c0cac30 con 0x7f8d1c15eba0 2024-03-18T20:07:19.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.066+0000 7f8d217b4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d1c0b0500 0x7f8d1c157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:19.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.066+0000 7f8d217b4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d1c0b0500 0x7f8d1c157aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37660/0 (socket says 172.21.15.1:37660) 2024-03-18T20:07:19.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.066+0000 7f8d217b4700 1 -- 172.21.15.1:0/2322870926 learned_addr learned my addr 172.21.15.1:0/2322870926 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:19.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.067+0000 7f8d20fb3700 1 --2- 172.21.15.1:0/2322870926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8d1c0cfdf0 0x7f8d1c0c49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:19.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.067+0000 7f8d21fb5700 1 --2- 172.21.15.1:0/2322870926 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8d1c15eba0 0x7f8d1c0c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:19.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.067+0000 7f8d217b4700 1 -- 172.21.15.1:0/2322870926 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8d1c15eba0 msgr2=0x7f8d1c0c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:19.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.067+0000 7f8d217b4700 1 --2- 172.21.15.1:0/2322870926 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8d1c15eba0 0x7f8d1c0c6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.067+0000 7f8d217b4700 1 -- 172.21.15.1:0/2322870926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8d1c0cfdf0 msgr2=0x7f8d1c0c49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:19.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.067+0000 7f8d217b4700 1 --2- 172.21.15.1:0/2322870926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8d1c0cfdf0 0x7f8d1c0c49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.067+0000 7f8d217b4700 1 -- 172.21.15.1:0/2322870926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d18009550 con 0x7f8d1c0b0500 2024-03-18T20:07:19.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.067+0000 7f8d217b4700 1 --2- 172.21.15.1:0/2322870926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d1c0b0500 0x7f8d1c157aa0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c00c820 tx=0x7f8d0c00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:19.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.067+0000 7f8d11ffb700 1 -- 172.21.15.1:0/2322870926 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c011800 con 0x7f8d1c0b0500 2024-03-18T20:07:19.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.067+0000 7f8d11ffb700 1 -- 172.21.15.1:0/2322870926 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0c011960 con 0x7f8d1c0b0500 2024-03-18T20:07:19.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.067+0000 7f8d23a18700 1 -- 172.21.15.1:0/2322870926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d1c0caf10 con 0x7f8d1c0b0500 2024-03-18T20:07:19.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.067+0000 7f8d11ffb700 1 -- 172.21.15.1:0/2322870926 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c011d10 con 0x7f8d1c0b0500 2024-03-18T20:07:19.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.067+0000 7f8d23a18700 1 -- 172.21.15.1:0/2322870926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d1c0c7640 con 0x7f8d1c0b0500 2024-03-18T20:07:19.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.068+0000 7f8d23a18700 1 -- 172.21.15.1:0/2322870926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d1c0c7830 con 0x7f8d1c0b0500 2024-03-18T20:07:19.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.069+0000 7f8d11ffb700 1 -- 172.21.15.1:0/2322870926 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8d0c024480 con 0x7f8d1c0b0500 2024-03-18T20:07:19.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.069+0000 7f8d11ffb700 1 --2- 172.21.15.1:0/2322870926 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8d0006adb0 0x7f8d0006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:19.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.070+0000 7f8d11ffb700 1 -- 172.21.15.1:0/2322870926 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f8d0c089b00 con 0x7f8d1c0b0500 2024-03-18T20:07:19.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.070+0000 7f8d20fb3700 1 --2- 172.21.15.1:0/2322870926 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8d0006adb0 0x7f8d0006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:19.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.070+0000 7f8d20fb3700 1 --2- 172.21.15.1:0/2322870926 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8d0006adb0 0x7f8d0006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8d18009130 tx=0x7f8d18008030 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:19.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.074+0000 7f8d11ffb700 1 -- 172.21.15.1:0/2322870926 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8d0c054980 con 0x7f8d1c0b0500 2024-03-18T20:07:19.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.321+0000 7f8d23a18700 1 -- 172.21.15.1:0/2322870926 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f8d1c0676a0 con 0x7f8d1c0b0500 2024-03-18T20:07:19.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.326+0000 7f8d11ffb700 1 -- 172.21.15.1:0/2322870926 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 23 and data pool 24 v85) v1 ==== 174+0+0 (secure 0 0 0) 0x7f8d0c01f0c0 con 0x7f8d1c0b0500 2024-03-18T20:07:19.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.333+0000 7f8d23a18700 1 -- 172.21.15.1:0/2322870926 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8d0006adb0 msgr2=0x7f8d0006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:19.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.333+0000 7f8d23a18700 1 --2- 172.21.15.1:0/2322870926 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8d0006adb0 0x7f8d0006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8d18009130 tx=0x7f8d18008030 comp rx=0 tx=0).stop 2024-03-18T20:07:19.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.333+0000 7f8d23a18700 1 -- 172.21.15.1:0/2322870926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d1c0b0500 msgr2=0x7f8d1c157aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:19.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.333+0000 7f8d23a18700 1 --2- 172.21.15.1:0/2322870926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d1c0b0500 0x7f8d1c157aa0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c00c820 tx=0x7f8d0c00cbe0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.337+0000 7f8d23a18700 1 -- 172.21.15.1:0/2322870926 shutdown_connections 2024-03-18T20:07:19.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.337+0000 7f8d23a18700 1 --2- 172.21.15.1:0/2322870926 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8d0006adb0 0x7f8d0006d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.337+0000 7f8d23a18700 1 --2- 172.21.15.1:0/2322870926 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8d1c15eba0 0x7f8d1c0c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.337+0000 7f8d23a18700 1 --2- 172.21.15.1:0/2322870926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d1c0b0500 0x7f8d1c157aa0 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.337+0000 7f8d23a18700 1 --2- 172.21.15.1:0/2322870926 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8d1c0cfdf0 0x7f8d1c0c49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.337+0000 7f8d23a18700 1 -- 172.21.15.1:0/2322870926 >> 172.21.15.1:0/2322870926 conn(0x7f8d1c0b2bd0 msgr2=0x7f8d1c15dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:19.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.337+0000 7f8d23a18700 1 -- 172.21.15.1:0/2322870926 shutdown_connections 2024-03-18T20:07:19.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.339+0000 7f8d23a18700 1 -- 172.21.15.1:0/2322870926 wait complete. 2024-03-18T20:07:19.350 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 23 and data pool 24 2024-03-18T20:07:19.366 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:07:19.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.487+0000 7f3561328700 1 Processor -- start 2024-03-18T20:07:19.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.487+0000 7f3561328700 1 -- start start 2024-03-18T20:07:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f3561328700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f355c0b02e0 0x7f355c0cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f3561328700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f355c06b860 0x7f355c0cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f3561328700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f355c099a40 0x7f355c099ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f3561328700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f355c06ea70 con 0x7f355c099a40 2024-03-18T20:07:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f3561328700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f355c09a410 con 0x7f355c06b860 2024-03-18T20:07:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f3561328700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355c0d06e0 con 0x7f355c0b02e0 2024-03-18T20:07:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f355b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f355c099a40 0x7f355c099ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f355b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f355c099a40 0x7f355c099ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37666/0 (socket says 172.21.15.1:37666) 2024-03-18T20:07:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f355b7fe700 1 -- 172.21.15.1:0/1713401766 learned_addr learned my addr 172.21.15.1:0/1713401766 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f355a7fc700 1 --2- 172.21.15.1:0/1713401766 >> v2:172.21.15.165:3300/0 conn(0x7f355c06b860 0x7f355c0cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f355affd700 1 --2- 172.21.15.1:0/1713401766 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f355c0b02e0 0x7f355c0cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f355b7fe700 1 -- 172.21.15.1:0/1713401766 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f355c0b02e0 msgr2=0x7f355c0cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f355b7fe700 1 --2- 172.21.15.1:0/1713401766 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f355c0b02e0 0x7f355c0cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f355b7fe700 1 -- 172.21.15.1:0/1713401766 >> v2:172.21.15.165:3300/0 conn(0x7f355c06b860 msgr2=0x7f355c0cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f355b7fe700 1 --2- 172.21.15.1:0/1713401766 >> v2:172.21.15.165:3300/0 conn(0x7f355c06b860 0x7f355c0cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f355b7fe700 1 -- 172.21.15.1:0/1713401766 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c0cfe40 con 0x7f355c099a40 2024-03-18T20:07:19.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f355a7fc700 1 --2- 172.21.15.1:0/1713401766 >> v2:172.21.15.165:3300/0 conn(0x7f355c06b860 0x7f355c0cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:07:19.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f355b7fe700 1 --2- 172.21.15.1:0/1713401766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c099a40 0x7f355c099ed0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f355c072170 tx=0x7f355400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728790ccfa4a7ffb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:19.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f3559ffb700 1 -- 172.21.15.1:0/1713401766 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355400e070 con 0x7f355c099a40 2024-03-18T20:07:19.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f3559ffb700 1 -- 172.21.15.1:0/1713401766 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355400b910 con 0x7f355c099a40 2024-03-18T20:07:19.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.488+0000 7f3559ffb700 1 -- 172.21.15.1:0/1713401766 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355400bcc0 con 0x7f355c099a40 2024-03-18T20:07:19.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.489+0000 7f3561328700 1 -- 172.21.15.1:0/1713401766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c099a40 msgr2=0x7f355c099ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:19.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.489+0000 7f3561328700 1 --2- 172.21.15.1:0/1713401766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c099a40 0x7f355c099ed0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f355c072170 tx=0x7f355400b330 comp rx=0 tx=0).stop 2024-03-18T20:07:19.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.489+0000 7f3561328700 1 -- 172.21.15.1:0/1713401766 shutdown_connections 2024-03-18T20:07:19.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.489+0000 7f3561328700 1 --2- 172.21.15.1:0/1713401766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c099a40 0x7f355c099ed0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.489+0000 7f3561328700 1 --2- 172.21.15.1:0/1713401766 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f355c0b02e0 0x7f355c0cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.489+0000 7f3561328700 1 --2- 172.21.15.1:0/1713401766 >> v2:172.21.15.165:3300/0 conn(0x7f355c06b860 0x7f355c0cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.489+0000 7f3561328700 1 -- 172.21.15.1:0/1713401766 >> 172.21.15.1:0/1713401766 conn(0x7f355c0b2ac0 msgr2=0x7f355c0b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:19.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.489+0000 7f3561328700 1 -- 172.21.15.1:0/1713401766 shutdown_connections 2024-03-18T20:07:19.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.489+0000 7f3561328700 1 -- 172.21.15.1:0/1713401766 wait complete. 2024-03-18T20:07:19.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.490+0000 7f3561328700 1 Processor -- start 2024-03-18T20:07:19.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.490+0000 7f3561328700 1 -- start start 2024-03-18T20:07:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.491+0000 7f3561328700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c06b860 0x7f355c0b6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:19.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.491+0000 7f3561328700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f355c099a40 0x7f355c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:19.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.491+0000 7f3561328700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f355c0b02e0 0x7f355c15b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:19.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.491+0000 7f3561328700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c0b5c80 con 0x7f355c06b860 2024-03-18T20:07:19.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.491+0000 7f3561328700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c0b4360 con 0x7f355c0b02e0 2024-03-18T20:07:19.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.491+0000 7f3561328700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355c15be50 con 0x7f355c099a40 2024-03-18T20:07:19.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.491+0000 7f355affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c06b860 0x7f355c0b6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:19.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.491+0000 7f355affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c06b860 0x7f355c0b6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37678/0 (socket says 172.21.15.1:37678) 2024-03-18T20:07:19.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.491+0000 7f355affd700 1 -- 172.21.15.1:0/1867225056 learned_addr learned my addr 172.21.15.1:0/1867225056 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:19.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.491+0000 7f355b7fe700 1 --2- 172.21.15.1:0/1867225056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f355c0b02e0 0x7f355c15b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:19.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.491+0000 7f355a7fc700 1 --2- 172.21.15.1:0/1867225056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f355c099a40 0x7f355c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.492+0000 7f355affd700 1 -- 172.21.15.1:0/1867225056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f355c099a40 msgr2=0x7f355c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.492+0000 7f355affd700 1 --2- 172.21.15.1:0/1867225056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f355c099a40 0x7f355c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.492+0000 7f355affd700 1 -- 172.21.15.1:0/1867225056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f355c0b02e0 msgr2=0x7f355c15b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.492+0000 7f355affd700 1 --2- 172.21.15.1:0/1867225056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f355c0b02e0 0x7f355c15b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.492+0000 7f355affd700 1 -- 172.21.15.1:0/1867225056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3554009b30 con 0x7f355c06b860 2024-03-18T20:07:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.492+0000 7f355affd700 1 --2- 172.21.15.1:0/1867225056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c06b860 0x7f355c0b6040 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f354c00e9e0 tx=0x7f354c00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.492+0000 7f35437fe700 1 -- 172.21.15.1:0/1867225056 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c009710 con 0x7f355c06b860 2024-03-18T20:07:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.492+0000 7f35437fe700 1 -- 172.21.15.1:0/1867225056 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354c00ee80 con 0x7f355c06b860 2024-03-18T20:07:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.492+0000 7f3561328700 1 -- 172.21.15.1:0/1867225056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355c15e3e0 con 0x7f355c06b860 2024-03-18T20:07:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.492+0000 7f35437fe700 1 -- 172.21.15.1:0/1867225056 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c00c9f0 con 0x7f355c06b860 2024-03-18T20:07:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.492+0000 7f3561328700 1 -- 172.21.15.1:0/1867225056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355c0b4870 con 0x7f355c06b860 2024-03-18T20:07:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.493+0000 7f3561328700 1 -- 172.21.15.1:0/1867225056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355c15e6c0 con 0x7f355c06b860 2024-03-18T20:07:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.494+0000 7f35437fe700 1 -- 172.21.15.1:0/1867225056 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f354c00cb50 con 0x7f355c06b860 2024-03-18T20:07:19.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.494+0000 7f35437fe700 1 --2- 172.21.15.1:0/1867225056 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f354806adb0 0x7f354806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:19.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.494+0000 7f35437fe700 1 -- 172.21.15.1:0/1867225056 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f354c089520 con 0x7f355c06b860 2024-03-18T20:07:19.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.494+0000 7f355a7fc700 1 --2- 172.21.15.1:0/1867225056 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f354806adb0 0x7f354806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:19.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.495+0000 7f355a7fc700 1 --2- 172.21.15.1:0/1867225056 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f354806adb0 0x7f354806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f355c15cb70 tx=0x7f3550008040 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:19.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.499+0000 7f35437fe700 1 -- 172.21.15.1:0/1867225056 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f354c0544f0 con 0x7f355c06b860 2024-03-18T20:07:19.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.701+0000 7f3561328700 1 -- 172.21.15.1:0/1867225056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f355c073350 con 0x7f355c06b860 2024-03-18T20:07:19.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.702+0000 7f35437fe700 1 -- 172.21.15.1:0/1867225056 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v194) v1 ==== 74+0+16328 (secure 0 0 0) 0x7f354c057b10 con 0x7f355c06b860 2024-03-18T20:07:19.707 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:07:19.707 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":194,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:07:19.322484+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:07:17.162084+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-18T20:07:18.295080+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:07:19.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.704+0000 7f3561328700 1 -- 172.21.15.1:0/1867225056 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f354806adb0 msgr2=0x7f354806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:19.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.704+0000 7f3561328700 1 --2- 172.21.15.1:0/1867225056 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f354806adb0 0x7f354806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f355c15cb70 tx=0x7f3550008040 comp rx=0 tx=0).stop 2024-03-18T20:07:19.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.704+0000 7f3561328700 1 -- 172.21.15.1:0/1867225056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c06b860 msgr2=0x7f355c0b6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:19.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.704+0000 7f3561328700 1 --2- 172.21.15.1:0/1867225056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c06b860 0x7f355c0b6040 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f354c00e9e0 tx=0x7f354c00ecf0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.704+0000 7f3561328700 1 -- 172.21.15.1:0/1867225056 shutdown_connections 2024-03-18T20:07:19.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.704+0000 7f3561328700 1 --2- 172.21.15.1:0/1867225056 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f354806adb0 0x7f354806d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.704+0000 7f3561328700 1 --2- 172.21.15.1:0/1867225056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f355c0b02e0 0x7f355c15b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.704+0000 7f3561328700 1 --2- 172.21.15.1:0/1867225056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c06b860 0x7f355c0b6040 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.704+0000 7f3561328700 1 --2- 172.21.15.1:0/1867225056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f355c099a40 0x7f355c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.704+0000 7f3561328700 1 -- 172.21.15.1:0/1867225056 >> 172.21.15.1:0/1867225056 conn(0x7f355c0b2ac0 msgr2=0x7f355c15c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:19.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.704+0000 7f3561328700 1 -- 172.21.15.1:0/1867225056 shutdown_connections 2024-03-18T20:07:19.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.704+0000 7f3561328700 1 -- 172.21.15.1:0/1867225056 wait complete. 2024-03-18T20:07:19.722 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:07:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.842+0000 7f23d115a700 1 Processor -- start 2024-03-18T20:07:19.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.842+0000 7f23d115a700 1 -- start start 2024-03-18T20:07:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.843+0000 7f23d115a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23cc15a320 0x7f23cc0ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.843+0000 7f23d115a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f23cc071d10 0x7f23cc06e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.843+0000 7f23d115a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23cc0acd80 con 0x7f23cc0cea30 2024-03-18T20:07:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.843+0000 7f23d115a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23cc0cf220 con 0x7f23cc071d10 2024-03-18T20:07:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.843+0000 7f23d115a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc0cef40 con 0x7f23cc15a320 2024-03-18T20:07:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.844+0000 7f23cb59e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f23cc071d10 0x7f23cc06e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.844+0000 7f23cad9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23cc15a320 0x7f23cc0ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.844+0000 7f23cad9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23cc15a320 0x7f23cc0ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:54774/0 (socket says 172.21.15.1:54774) 2024-03-18T20:07:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.844+0000 7f23cad9d700 1 -- 172.21.15.1:0/1009366507 learned_addr learned my addr 172.21.15.1:0/1009366507 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.844+0000 7f23cb59e700 1 -- 172.21.15.1:0/1009366507 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23cc15a320 msgr2=0x7f23cc0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.844+0000 7f23cb59e700 1 --2- 172.21.15.1:0/1009366507 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23cc15a320 0x7f23cc0ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.844+0000 7f23cb59e700 1 -- 172.21.15.1:0/1009366507 >> v1:172.21.15.165:6789/0 conn(0x7f23cc0cea30 legacy=0x7f23cc071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.844+0000 7f23cb59e700 1 -- 172.21.15.1:0/1009366507 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23cc06d7e0 con 0x7f23cc071d10 2024-03-18T20:07:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.844+0000 7f23cb59e700 1 --2- 172.21.15.1:0/1009366507 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23cc071d10 0x7f23cc06e2c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f23c4009d90 tx=0x7f23c400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37ae1251abb1fe1e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.844+0000 7f23c9d9b700 1 -- 172.21.15.1:0/1009366507 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c400e070 con 0x7f23cc071d10 2024-03-18T20:07:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.844+0000 7f23c9d9b700 1 -- 172.21.15.1:0/1009366507 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c400bb20 con 0x7f23cc071d10 2024-03-18T20:07:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.844+0000 7f23c9d9b700 1 -- 172.21.15.1:0/1009366507 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c4012430 con 0x7f23cc071d10 2024-03-18T20:07:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.845+0000 7f23d115a700 1 -- 172.21.15.1:0/1009366507 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23cc071d10 msgr2=0x7f23cc06e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.845+0000 7f23d115a700 1 --2- 172.21.15.1:0/1009366507 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23cc071d10 0x7f23cc06e2c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f23c4009d90 tx=0x7f23c400b330 comp rx=0 tx=0).stop 2024-03-18T20:07:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.845+0000 7f23d115a700 1 -- 172.21.15.1:0/1009366507 shutdown_connections 2024-03-18T20:07:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.845+0000 7f23d115a700 1 --2- 172.21.15.1:0/1009366507 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23cc071d10 0x7f23cc06e2c0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.845+0000 7f23d115a700 1 --2- 172.21.15.1:0/1009366507 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23cc15a320 0x7f23cc0ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.845+0000 7f23d115a700 1 -- 172.21.15.1:0/1009366507 >> 172.21.15.1:0/1009366507 conn(0x7f23cc0cc990 msgr2=0x7f23cc0ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.845+0000 7f23d115a700 1 -- 172.21.15.1:0/1009366507 shutdown_connections 2024-03-18T20:07:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.845+0000 7f23d115a700 1 -- 172.21.15.1:0/1009366507 wait complete. 2024-03-18T20:07:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.846+0000 7f23d115a700 1 Processor -- start 2024-03-18T20:07:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.847+0000 7f23d115a700 1 -- start start 2024-03-18T20:07:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.847+0000 7f23d115a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23cc071d10 0x7f23cc1513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.847+0000 7f23d115a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23cc0cea30 0x7f23cc0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.847+0000 7f23d115a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23cc0cb7a0 0x7f23cc15a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.847+0000 7f23d115a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc0cbc50 con 0x7f23cc0cb7a0 2024-03-18T20:07:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.847+0000 7f23d115a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc1f2550 con 0x7f23cc0cea30 2024-03-18T20:07:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.847+0000 7f23d115a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc15b5c0 con 0x7f23cc071d10 2024-03-18T20:07:19.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.847+0000 7f23cb59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23cc0cb7a0 0x7f23cc15a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:19.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.847+0000 7f23cb59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23cc0cb7a0 0x7f23cc15a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37680/0 (socket says 172.21.15.1:37680) 2024-03-18T20:07:19.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.847+0000 7f23cb59e700 1 -- 172.21.15.1:0/431597679 learned_addr learned my addr 172.21.15.1:0/431597679 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:19.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.848+0000 7f23ca59c700 1 --2- 172.21.15.1:0/431597679 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23cc0cea30 0x7f23cc0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:19.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.848+0000 7f23cad9d700 1 --2- 172.21.15.1:0/431597679 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23cc071d10 0x7f23cc1513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:19.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.848+0000 7f23cb59e700 1 -- 172.21.15.1:0/431597679 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23cc071d10 msgr2=0x7f23cc1513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:19.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.848+0000 7f23cb59e700 1 --2- 172.21.15.1:0/431597679 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23cc071d10 0x7f23cc1513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.848+0000 7f23cb59e700 1 -- 172.21.15.1:0/431597679 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23cc0cea30 msgr2=0x7f23cc0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:19.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.848+0000 7f23cb59e700 1 --2- 172.21.15.1:0/431597679 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23cc0cea30 0x7f23cc0cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:19.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.848+0000 7f23cb59e700 1 -- 172.21.15.1:0/431597679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c4009a70 con 0x7f23cc0cb7a0 2024-03-18T20:07:19.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.848+0000 7f23cb59e700 1 --2- 172.21.15.1:0/431597679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23cc0cb7a0 0x7f23cc15a920 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f23c4014040 tx=0x7f23c4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:19.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.848+0000 7f23c37fe700 1 -- 172.21.15.1:0/431597679 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c4012430 con 0x7f23cc0cb7a0 2024-03-18T20:07:19.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.848+0000 7f23d115a700 1 -- 172.21.15.1:0/431597679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23cc1f27a0 con 0x7f23cc0cb7a0 2024-03-18T20:07:19.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.848+0000 7f23c37fe700 1 -- 172.21.15.1:0/431597679 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c4020070 con 0x7f23cc0cb7a0 2024-03-18T20:07:19.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.848+0000 7f23c37fe700 1 -- 172.21.15.1:0/431597679 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c4003ea0 con 0x7f23cc0cb7a0 2024-03-18T20:07:19.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.848+0000 7f23d115a700 1 -- 172.21.15.1:0/431597679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23cc1f2d60 con 0x7f23cc0cb7a0 2024-03-18T20:07:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.849+0000 7f23d115a700 1 -- 172.21.15.1:0/431597679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23cc06d2d0 con 0x7f23cc0cb7a0 2024-03-18T20:07:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.850+0000 7f23c37fe700 1 -- 172.21.15.1:0/431597679 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f23c4004040 con 0x7f23cc0cb7a0 2024-03-18T20:07:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.850+0000 7f23c37fe700 1 --2- 172.21.15.1:0/431597679 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f23ac06adb0 0x7f23ac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.850+0000 7f23c37fe700 1 -- 172.21.15.1:0/431597679 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f23c4022070 con 0x7f23cc0cb7a0 2024-03-18T20:07:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.850+0000 7f23cad9d700 1 --2- 172.21.15.1:0/431597679 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f23ac06adb0 0x7f23ac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.850+0000 7f23cad9d700 1 --2- 172.21.15.1:0/431597679 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f23ac06adb0 0x7f23ac06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009bd0 tx=0x7f23bc0092c0 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:19.855+0000 7f23c37fe700 1 -- 172.21.15.1:0/431597679 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f23c40561a0 con 0x7f23cc0cb7a0 2024-03-18T20:07:20.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.060+0000 7f23d115a700 1 -- 172.21.15.1:0/431597679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23cc0cbe70 con 0x7f23cc0cb7a0 2024-03-18T20:07:20.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.061+0000 7f23c37fe700 1 -- 172.21.15.1:0/431597679 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v194) v1 ==== 74+0+16328 (secure 0 0 0) 0x7f23c40597c0 con 0x7f23cc0cb7a0 2024-03-18T20:07:20.066 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:07:20.066 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":194,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:07:19.322484+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:07:17.162084+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-18T20:07:18.295080+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:07:20.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.064+0000 7f23d115a700 1 -- 172.21.15.1:0/431597679 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f23ac06adb0 msgr2=0x7f23ac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:20.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.064+0000 7f23d115a700 1 --2- 172.21.15.1:0/431597679 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f23ac06adb0 0x7f23ac06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009bd0 tx=0x7f23bc0092c0 comp rx=0 tx=0).stop 2024-03-18T20:07:20.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.064+0000 7f23d115a700 1 -- 172.21.15.1:0/431597679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23cc0cb7a0 msgr2=0x7f23cc15a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:20.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.064+0000 7f23d115a700 1 --2- 172.21.15.1:0/431597679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23cc0cb7a0 0x7f23cc15a920 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f23c4014040 tx=0x7f23c4004ad0 comp rx=0 tx=0).stop 2024-03-18T20:07:20.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.064+0000 7f23d115a700 1 -- 172.21.15.1:0/431597679 shutdown_connections 2024-03-18T20:07:20.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.064+0000 7f23d115a700 1 --2- 172.21.15.1:0/431597679 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f23ac06adb0 0x7f23ac06d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:20.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.064+0000 7f23d115a700 1 --2- 172.21.15.1:0/431597679 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f23cc071d10 0x7f23cc1513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:20.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.064+0000 7f23d115a700 1 --2- 172.21.15.1:0/431597679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23cc0cb7a0 0x7f23cc15a920 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:20.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.064+0000 7f23d115a700 1 --2- 172.21.15.1:0/431597679 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23cc0cea30 0x7f23cc0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:20.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.064+0000 7f23d115a700 1 -- 172.21.15.1:0/431597679 >> 172.21.15.1:0/431597679 conn(0x7f23cc0cc990 msgr2=0x7f23cc0ccd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:20.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.064+0000 7f23d115a700 1 -- 172.21.15.1:0/431597679 shutdown_connections 2024-03-18T20:07:20.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.064+0000 7f23d115a700 1 -- 172.21.15.1:0/431597679 wait complete. 2024-03-18T20:07:20.081 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T20:07:20.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.205+0000 7f3637005700 1 Processor -- start 2024-03-18T20:07:20.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.205+0000 7f3637005700 1 -- start start 2024-03-18T20:07:20.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.205+0000 7f3637005700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f363006ca80 0x7f363015d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:20.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.205+0000 7f3637005700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f363009a150 0x7f36300cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:20.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.205+0000 7f3637005700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36300b3b00 con 0x7f363015d5b0 2024-03-18T20:07:20.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.205+0000 7f3637005700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36300cbbb0 con 0x7f363009a150 2024-03-18T20:07:20.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.205+0000 7f3637005700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36300cf7f0 con 0x7f363006ca80 2024-03-18T20:07:20.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.206+0000 7f36355a2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f363009a150 0x7f36300cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:20.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.206+0000 7f3634da1700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f363015d5b0 0x7f3630099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43370/0 (socket says 172.21.15.1:43370) 2024-03-18T20:07:20.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.206+0000 7f3634da1700 1 -- 172.21.15.1:0/2889775296 learned_addr learned my addr 172.21.15.1:0/2889775296 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:20.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.206+0000 7f362ffff700 1 --2- 172.21.15.1:0/2889775296 >> v2:172.21.15.165:3300/0 conn(0x7f363006ca80 0x7f363015d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:20.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.206+0000 7f36355a2700 1 -- 172.21.15.1:0/2889775296 >> v2:172.21.15.165:3300/0 conn(0x7f363006ca80 msgr2=0x7f363015d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:20.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.206+0000 7f36355a2700 1 --2- 172.21.15.1:0/2889775296 >> v2:172.21.15.165:3300/0 conn(0x7f363006ca80 0x7f363015d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:20.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.206+0000 7f36355a2700 1 -- 172.21.15.1:0/2889775296 >> v1:172.21.15.1:6789/0 conn(0x7f363015d5b0 legacy=0x7f3630099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:20.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.206+0000 7f36355a2700 1 -- 172.21.15.1:0/2889775296 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36300d04d0 con 0x7f363009a150 2024-03-18T20:07:20.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.206+0000 7f362f7fe700 1 -- 172.21.15.1:0/2889775296 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156067906 0 0) 0x7f36300b3b00 con 0x7f363015d5b0 2024-03-18T20:07:20.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.206+0000 7f362ffff700 1 --2- 172.21.15.1:0/2889775296 >> v2:172.21.15.165:3300/0 conn(0x7f363006ca80 0x7f363015d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:07:20.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.206+0000 7f36355a2700 1 --2- 172.21.15.1:0/2889775296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f363009a150 0x7f36300cbfd0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f3630072c10 tx=0x7f362800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1014e4595719777 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:20.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.206+0000 7f362f7fe700 1 -- 172.21.15.1:0/2889775296 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362800e070 con 0x7f363009a150 2024-03-18T20:07:20.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.206+0000 7f362f7fe700 1 -- 172.21.15.1:0/2889775296 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362800bbd0 con 0x7f363009a150 2024-03-18T20:07:20.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.206+0000 7f362f7fe700 1 -- 172.21.15.1:0/2889775296 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3628012430 con 0x7f363009a150 2024-03-18T20:07:20.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.207+0000 7f3637005700 1 -- 172.21.15.1:0/2889775296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f363009a150 msgr2=0x7f36300cbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:20.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.207+0000 7f3637005700 1 --2- 172.21.15.1:0/2889775296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f363009a150 0x7f36300cbfd0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f3630072c10 tx=0x7f362800b330 comp rx=0 tx=0).stop 2024-03-18T20:07:20.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.207+0000 7f3637005700 1 -- 172.21.15.1:0/2889775296 shutdown_connections 2024-03-18T20:07:20.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.207+0000 7f3637005700 1 --2- 172.21.15.1:0/2889775296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f363009a150 0x7f36300cbfd0 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:20.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.207+0000 7f3637005700 1 --2- 172.21.15.1:0/2889775296 >> v2:172.21.15.165:3300/0 conn(0x7f363006ca80 0x7f363015d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:20.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.207+0000 7f3637005700 1 -- 172.21.15.1:0/2889775296 >> 172.21.15.1:0/2889775296 conn(0x7f36300b2bd0 msgr2=0x7f36300b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:20.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.207+0000 7f3637005700 1 -- 172.21.15.1:0/2889775296 shutdown_connections 2024-03-18T20:07:20.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.207+0000 7f3637005700 1 -- 172.21.15.1:0/2889775296 wait complete. 2024-03-18T20:07:20.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.208+0000 7f3637005700 1 Processor -- start 2024-03-18T20:07:20.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.208+0000 7f3637005700 1 -- start start 2024-03-18T20:07:20.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.209+0000 7f3637005700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f363006ca80 0x7f36301f5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:20.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.209+0000 7f3637005700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f363009a150 0x7f36301f59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:20.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.209+0000 7f3637005700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f363015d5b0 0x7f3630145900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:20.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.209+0000 7f3637005700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36301f5f10 con 0x7f363015d5b0 2024-03-18T20:07:20.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.209+0000 7f3637005700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36300ae230 con 0x7f363006ca80 2024-03-18T20:07:20.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.209+0000 7f3637005700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3630143730 con 0x7f363009a150 2024-03-18T20:07:20.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.209+0000 7f36355a2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f363015d5b0 0x7f3630145900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:20.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.209+0000 7f36355a2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f363015d5b0 0x7f3630145900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37696/0 (socket says 172.21.15.1:37696) 2024-03-18T20:07:20.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.209+0000 7f36355a2700 1 -- 172.21.15.1:0/630649316 learned_addr learned my addr 172.21.15.1:0/630649316 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:20.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.209+0000 7f3634da1700 1 --2- 172.21.15.1:0/630649316 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f363006ca80 0x7f36301f5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:20.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.209+0000 7f362ffff700 1 --2- 172.21.15.1:0/630649316 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f363009a150 0x7f36301f59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:20.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.209+0000 7f36355a2700 1 -- 172.21.15.1:0/630649316 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f363009a150 msgr2=0x7f36301f59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:20.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.209+0000 7f36355a2700 1 --2- 172.21.15.1:0/630649316 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f363009a150 0x7f36301f59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:20.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.209+0000 7f36355a2700 1 -- 172.21.15.1:0/630649316 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f363006ca80 msgr2=0x7f36301f5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:20.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.209+0000 7f36355a2700 1 --2- 172.21.15.1:0/630649316 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f363006ca80 0x7f36301f5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:20.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.209+0000 7f36355a2700 1 -- 172.21.15.1:0/630649316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3628009a70 con 0x7f363015d5b0 2024-03-18T20:07:20.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.210+0000 7f36355a2700 1 --2- 172.21.15.1:0/630649316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f363015d5b0 0x7f3630145900 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f3630072150 tx=0x7f3628004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:20.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.210+0000 7f362d7fa700 1 -- 172.21.15.1:0/630649316 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3628012430 con 0x7f363015d5b0 2024-03-18T20:07:20.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.210+0000 7f362d7fa700 1 -- 172.21.15.1:0/630649316 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3628020070 con 0x7f363015d5b0 2024-03-18T20:07:20.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.210+0000 7f3637005700 1 -- 172.21.15.1:0/630649316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3630143a10 con 0x7f363015d5b0 2024-03-18T20:07:20.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.210+0000 7f362d7fa700 1 -- 172.21.15.1:0/630649316 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3628004280 con 0x7f363015d5b0 2024-03-18T20:07:20.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.210+0000 7f3637005700 1 -- 172.21.15.1:0/630649316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3630143fd0 con 0x7f363015d5b0 2024-03-18T20:07:20.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.211+0000 7f3637005700 1 -- 172.21.15.1:0/630649316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36301441c0 con 0x7f363015d5b0 2024-03-18T20:07:20.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.211+0000 7f362d7fa700 1 -- 172.21.15.1:0/630649316 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f36280043e0 con 0x7f363015d5b0 2024-03-18T20:07:20.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.212+0000 7f362d7fa700 1 --2- 172.21.15.1:0/630649316 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f361406adb0 0x7f361406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:20.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.212+0000 7f3634da1700 1 --2- 172.21.15.1:0/630649316 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f361406adb0 0x7f361406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:20.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.212+0000 7f362d7fa700 1 -- 172.21.15.1:0/630649316 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3628022070 con 0x7f363015d5b0 2024-03-18T20:07:20.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.212+0000 7f3634da1700 1 --2- 172.21.15.1:0/630649316 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f361406adb0 0x7f361406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3624007fd0 tx=0x7f3624007de0 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:20.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.216+0000 7f362d7fa700 1 -- 172.21.15.1:0/630649316 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3628056200 con 0x7f363015d5b0 2024-03-18T20:07:20.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:20.466+0000 7f3637005700 1 -- 172.21.15.1:0/630649316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f36300676a0 con 0x7f363015d5b0 2024-03-18T20:07:21.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.336+0000 7f362d7fa700 1 -- 172.21.15.1:0/630649316 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v88) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3628059820 con 0x7f363015d5b0 2024-03-18T20:07:21.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.339+0000 7f3637005700 1 -- 172.21.15.1:0/630649316 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f361406adb0 msgr2=0x7f361406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:21.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.339+0000 7f3637005700 1 --2- 172.21.15.1:0/630649316 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f361406adb0 0x7f361406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3624007fd0 tx=0x7f3624007de0 comp rx=0 tx=0).stop 2024-03-18T20:07:21.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.339+0000 7f3637005700 1 -- 172.21.15.1:0/630649316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f363015d5b0 msgr2=0x7f3630145900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:21.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.339+0000 7f3637005700 1 --2- 172.21.15.1:0/630649316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f363015d5b0 0x7f3630145900 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f3630072150 tx=0x7f3628004ad0 comp rx=0 tx=0).stop 2024-03-18T20:07:21.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.339+0000 7f3637005700 1 -- 172.21.15.1:0/630649316 shutdown_connections 2024-03-18T20:07:21.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.339+0000 7f3637005700 1 --2- 172.21.15.1:0/630649316 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f361406adb0 0x7f361406d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:21.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.339+0000 7f3637005700 1 --2- 172.21.15.1:0/630649316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f363015d5b0 0x7f3630145900 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:21.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.339+0000 7f3637005700 1 --2- 172.21.15.1:0/630649316 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f363006ca80 0x7f36301f5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:21.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.339+0000 7f3637005700 1 --2- 172.21.15.1:0/630649316 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f363009a150 0x7f36301f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:21.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.339+0000 7f3637005700 1 -- 172.21.15.1:0/630649316 >> 172.21.15.1:0/630649316 conn(0x7f36300b2bd0 msgr2=0x7f363006b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:21.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.339+0000 7f3637005700 1 -- 172.21.15.1:0/630649316 shutdown_connections 2024-03-18T20:07:21.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.339+0000 7f3637005700 1 -- 172.21.15.1:0/630649316 wait complete. 2024-03-18T20:07:21.359 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-18T20:07:21.360 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T20:07:21.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.490+0000 7fd4c4ba2700 1 Processor -- start 2024-03-18T20:07:21.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.490+0000 7fd4c4ba2700 1 -- start start 2024-03-18T20:07:21.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.491+0000 7fd4c4ba2700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd4c015b6c0 0x7fd4c00b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:21.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.491+0000 7fd4c4ba2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4c00c87d0 0x7fd4c00cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:21.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.491+0000 7fd4c4ba2700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c00d01f0 con 0x7fd4c00cd430 2024-03-18T20:07:21.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.491+0000 7fd4c4ba2700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4c00ccfa0 con 0x7fd4c015b6c0 2024-03-18T20:07:21.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.491+0000 7fd4c4ba2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c00d1db0 con 0x7fd4c00c87d0 2024-03-18T20:07:21.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.492+0000 7fd4be59c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd4c015b6c0 0x7fd4c00b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:21.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.492+0000 7fd4bed9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4c00c87d0 0x7fd4c00cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:21.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.492+0000 7fd4be59c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd4c015b6c0 0x7fd4c00b2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:57838/0 (socket says 172.21.15.1:57838) 2024-03-18T20:07:21.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.492+0000 7fd4be59c700 1 -- 172.21.15.1:0/3193193338 learned_addr learned my addr 172.21.15.1:0/3193193338 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:21.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.492+0000 7fd4bdd9b700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fd4c00cd430 0x7fd4c00c80c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:36038/0 (socket says 172.21.15.1:36038) 2024-03-18T20:07:21.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.492+0000 7fd4be59c700 1 -- 172.21.15.1:0/3193193338 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4c00c87d0 msgr2=0x7fd4c00cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:21.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.492+0000 7fd4be59c700 1 --2- 172.21.15.1:0/3193193338 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4c00c87d0 0x7fd4c00cd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:21.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.492+0000 7fd4be59c700 1 -- 172.21.15.1:0/3193193338 >> v1:172.21.15.165:6789/0 conn(0x7fd4c00cd430 legacy=0x7fd4c00c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:21.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.492+0000 7fd4be59c700 1 -- 172.21.15.1:0/3193193338 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c00cff10 con 0x7fd4c015b6c0 2024-03-18T20:07:21.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.492+0000 7fd4bed9d700 1 --2- 172.21.15.1:0/3193193338 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4c00c87d0 0x7fd4c00cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:07:21.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.492+0000 7fd4be59c700 1 --2- 172.21.15.1:0/3193193338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4c015b6c0 0x7fd4c00b2160 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00098f0 tx=0x7fd4b0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9c14eac5d267337 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:21.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.492+0000 7fd4bd59a700 1 -- 172.21.15.1:0/3193193338 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0011830 con 0x7fd4c015b6c0 2024-03-18T20:07:21.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.492+0000 7fd4bd59a700 1 -- 172.21.15.1:0/3193193338 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b0011990 con 0x7fd4c015b6c0 2024-03-18T20:07:21.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.492+0000 7fd4bd59a700 1 -- 172.21.15.1:0/3193193338 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0011d20 con 0x7fd4c015b6c0 2024-03-18T20:07:21.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.493+0000 7fd4c4ba2700 1 -- 172.21.15.1:0/3193193338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4c015b6c0 msgr2=0x7fd4c00b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:21.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.493+0000 7fd4c4ba2700 1 --2- 172.21.15.1:0/3193193338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4c015b6c0 0x7fd4c00b2160 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00098f0 tx=0x7fd4b0009c00 comp rx=0 tx=0).stop 2024-03-18T20:07:21.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.493+0000 7fd4c4ba2700 1 -- 172.21.15.1:0/3193193338 shutdown_connections 2024-03-18T20:07:21.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.493+0000 7fd4c4ba2700 1 --2- 172.21.15.1:0/3193193338 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4c00c87d0 0x7fd4c00cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:21.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.493+0000 7fd4c4ba2700 1 --2- 172.21.15.1:0/3193193338 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4c015b6c0 0x7fd4c00b2160 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:21.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.493+0000 7fd4c4ba2700 1 -- 172.21.15.1:0/3193193338 >> 172.21.15.1:0/3193193338 conn(0x7fd4c00d0ff0 msgr2=0x7fd4c00d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:21.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.493+0000 7fd4c4ba2700 1 -- 172.21.15.1:0/3193193338 shutdown_connections 2024-03-18T20:07:21.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.493+0000 7fd4c4ba2700 1 -- 172.21.15.1:0/3193193338 wait complete. 2024-03-18T20:07:21.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.494+0000 7fd4c4ba2700 1 Processor -- start 2024-03-18T20:07:21.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.494+0000 7fd4c4ba2700 1 -- start start 2024-03-18T20:07:21.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.495+0000 7fd4c4ba2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4c00c87d0 0x7fd4c014b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:21.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.495+0000 7fd4c4ba2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4c00cd430 0x7fd4c009af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:21.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.495+0000 7fd4c4ba2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4c015b6c0 0x7fd4c009bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:21.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.495+0000 7fd4c4ba2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c00b4c90 con 0x7fd4c015b6c0 2024-03-18T20:07:21.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.495+0000 7fd4c4ba2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c006cdc0 con 0x7fd4c00cd430 2024-03-18T20:07:21.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.495+0000 7fd4c4ba2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c006d0a0 con 0x7fd4c00c87d0 2024-03-18T20:07:21.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.495+0000 7fd4bed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4c015b6c0 0x7fd4c009bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:21.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.495+0000 7fd4bed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4c015b6c0 0x7fd4c009bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37710/0 (socket says 172.21.15.1:37710) 2024-03-18T20:07:21.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.495+0000 7fd4bed9d700 1 -- 172.21.15.1:0/884868763 learned_addr learned my addr 172.21.15.1:0/884868763 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:21.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.495+0000 7fd4be59c700 1 --2- 172.21.15.1:0/884868763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4c00c87d0 0x7fd4c014b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:21.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.495+0000 7fd4bdd9b700 1 --2- 172.21.15.1:0/884868763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4c00cd430 0x7fd4c009af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:21.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.496+0000 7fd4bed9d700 1 -- 172.21.15.1:0/884868763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4c00c87d0 msgr2=0x7fd4c014b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:21.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.496+0000 7fd4bed9d700 1 --2- 172.21.15.1:0/884868763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4c00c87d0 0x7fd4c014b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:21.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.496+0000 7fd4bed9d700 1 -- 172.21.15.1:0/884868763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4c00cd430 msgr2=0x7fd4c009af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:21.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.496+0000 7fd4bed9d700 1 --2- 172.21.15.1:0/884868763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4c00cd430 0x7fd4c009af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:21.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.496+0000 7fd4bed9d700 1 -- 172.21.15.1:0/884868763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b8009a70 con 0x7fd4c015b6c0 2024-03-18T20:07:21.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.496+0000 7fd4bed9d700 1 --2- 172.21.15.1:0/884868763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4c015b6c0 0x7fd4c009bc20 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800efc0 tx=0x7fd4b800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:21.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.496+0000 7fd4aeffd700 1 -- 172.21.15.1:0/884868763 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b800ca90 con 0x7fd4c015b6c0 2024-03-18T20:07:21.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.496+0000 7fd4c4ba2700 1 -- 172.21.15.1:0/884868763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b0009550 con 0x7fd4c015b6c0 2024-03-18T20:07:21.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.496+0000 7fd4aeffd700 1 -- 172.21.15.1:0/884868763 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b800f040 con 0x7fd4c015b6c0 2024-03-18T20:07:21.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.496+0000 7fd4aeffd700 1 -- 172.21.15.1:0/884868763 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b800ce30 con 0x7fd4c015b6c0 2024-03-18T20:07:21.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.496+0000 7fd4c4ba2700 1 -- 172.21.15.1:0/884868763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c006d660 con 0x7fd4c015b6c0 2024-03-18T20:07:21.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.497+0000 7fd4c4ba2700 1 -- 172.21.15.1:0/884868763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c00b4dd0 con 0x7fd4c015b6c0 2024-03-18T20:07:21.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.497+0000 7fd4aeffd700 1 -- 172.21.15.1:0/884868763 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd4b8014480 con 0x7fd4c015b6c0 2024-03-18T20:07:21.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.498+0000 7fd4aeffd700 1 --2- 172.21.15.1:0/884868763 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd4a006adb0 0x7fd4a006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:21.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.498+0000 7fd4aeffd700 1 -- 172.21.15.1:0/884868763 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd4b8089e80 con 0x7fd4c015b6c0 2024-03-18T20:07:21.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.498+0000 7fd4be59c700 1 --2- 172.21.15.1:0/884868763 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd4a006adb0 0x7fd4a006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:21.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.498+0000 7fd4be59c700 1 --2- 172.21.15.1:0/884868763 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd4a006adb0 0x7fd4a006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0009c40 tx=0x7fd4b0008030 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.503+0000 7fd4aeffd700 1 -- 172.21.15.1:0/884868763 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd4c00b4dd0 con 0x7fd4c015b6c0 2024-03-18T20:07:21.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:21.755+0000 7fd4c4ba2700 1 -- 172.21.15.1:0/884868763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd4c00b4dd0 con 0x7fd4c015b6c0 2024-03-18T20:07:22.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.341+0000 7fd4aeffd700 1 -- 172.21.15.1:0/884868763 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v89) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd4c00b4dd0 con 0x7fd4c015b6c0 2024-03-18T20:07:22.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.344+0000 7fd4c4ba2700 1 -- 172.21.15.1:0/884868763 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd4a006adb0 msgr2=0x7fd4a006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:22.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.344+0000 7fd4c4ba2700 1 --2- 172.21.15.1:0/884868763 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd4a006adb0 0x7fd4a006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0009c40 tx=0x7fd4b0008030 comp rx=0 tx=0).stop 2024-03-18T20:07:22.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.344+0000 7fd4c4ba2700 1 -- 172.21.15.1:0/884868763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4c015b6c0 msgr2=0x7fd4c009bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:22.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.344+0000 7fd4c4ba2700 1 --2- 172.21.15.1:0/884868763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4c015b6c0 0x7fd4c009bc20 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800efc0 tx=0x7fd4b800c3a0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.344+0000 7fd4c4ba2700 1 -- 172.21.15.1:0/884868763 shutdown_connections 2024-03-18T20:07:22.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.344+0000 7fd4c4ba2700 1 --2- 172.21.15.1:0/884868763 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd4a006adb0 0x7fd4a006d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.344+0000 7fd4c4ba2700 1 --2- 172.21.15.1:0/884868763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4c00c87d0 0x7fd4c014b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.344+0000 7fd4c4ba2700 1 --2- 172.21.15.1:0/884868763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4c015b6c0 0x7fd4c009bc20 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.344+0000 7fd4c4ba2700 1 --2- 172.21.15.1:0/884868763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4c00cd430 0x7fd4c009af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.344+0000 7fd4c4ba2700 1 -- 172.21.15.1:0/884868763 >> 172.21.15.1:0/884868763 conn(0x7fd4c00d0ff0 msgr2=0x7fd4c00ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:22.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.344+0000 7fd4c4ba2700 1 -- 172.21.15.1:0/884868763 shutdown_connections 2024-03-18T20:07:22.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.344+0000 7fd4c4ba2700 1 -- 172.21.15.1:0/884868763 wait complete. 2024-03-18T20:07:22.364 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T20:07:22.364 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:07:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.492+0000 7ff3772b2700 1 Processor -- start 2024-03-18T20:07:22.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.492+0000 7ff3772b2700 1 -- start start 2024-03-18T20:07:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.493+0000 7ff3772b2700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff37015d680 0x7ff3700cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.493+0000 7ff3772b2700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3700d0c80 con 0x7ff3700d04b0 2024-03-18T20:07:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.493+0000 7ff3772b2700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3700d09a0 con 0x7ff37009a150 2024-03-18T20:07:22.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.493+0000 7ff3772b2700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3700cb830 con 0x7ff37015d680 2024-03-18T20:07:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.493+0000 7ff37484d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff3700d04b0 0x7ff370099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43386/0 (socket says 172.21.15.1:43386) 2024-03-18T20:07:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.493+0000 7ff37484d700 1 -- 172.21.15.1:0/3007632167 learned_addr learned my addr 172.21.15.1:0/3007632167 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.493+0000 7ff37504e700 1 --2- 172.21.15.1:0/3007632167 >> v2:172.21.15.165:3300/0 conn(0x7ff37015d680 0x7ff3700cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.493+0000 7ff367fff700 1 -- 172.21.15.1:0/3007632167 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725030573 0 0) 0x7ff3700d0c80 con 0x7ff3700d04b0 2024-03-18T20:07:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.493+0000 7ff367fff700 1 -- 172.21.15.1:0/3007632167 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff35c0035b0 con 0x7ff3700d04b0 2024-03-18T20:07:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.493+0000 7ff367fff700 1 -- 172.21.15.1:0/3007632167 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919413332 0 0) 0x7ff3700d09a0 con 0x7ff37009a150 2024-03-18T20:07:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.493+0000 7ff367fff700 1 -- 172.21.15.1:0/3007632167 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3700d0c80 con 0x7ff37009a150 2024-03-18T20:07:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.493+0000 7ff367fff700 1 -- 172.21.15.1:0/3007632167 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1447820081 0 0) 0x7ff35c0035b0 con 0x7ff3700d04b0 2024-03-18T20:07:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.493+0000 7ff367fff700 1 -- 172.21.15.1:0/3007632167 >> v2:172.21.15.165:3300/0 conn(0x7ff37015d680 msgr2=0x7ff3700cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.493+0000 7ff367fff700 1 --2- 172.21.15.1:0/3007632167 >> v2:172.21.15.165:3300/0 conn(0x7ff37015d680 0x7ff3700cff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.493+0000 7ff367fff700 1 -- 172.21.15.1:0/3007632167 >> v1:172.21.15.165:6789/0 conn(0x7ff37009a150 legacy=0x7ff3700cc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.493+0000 7ff367fff700 1 -- 172.21.15.1:0/3007632167 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3700cbb10 con 0x7ff3700d04b0 2024-03-18T20:07:22.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.493+0000 7ff367fff700 1 -- 172.21.15.1:0/3007632167 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7ff3600015f0 con 0x7ff3700d04b0 2024-03-18T20:07:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.493+0000 7ff367fff700 1 -- 172.21.15.1:0/3007632167 >> v1:172.21.15.1:6789/0 conn(0x7ff3700d04b0 legacy=0x7ff370099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.494+0000 7ff367fff700 1 --2- 172.21.15.1:0/3007632167 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff35c003be0 0x7ff35c003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.494+0000 7ff367fff700 1 --2- 172.21.15.1:0/3007632167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c004850 0x7ff35c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.494+0000 7ff367fff700 1 --2- 172.21.15.1:0/3007632167 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff35c005210 0x7ff35c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.494+0000 7ff367fff700 1 -- 172.21.15.1:0/3007632167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3700cb830 con 0x7ff35c004850 2024-03-18T20:07:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.494+0000 7ff367fff700 1 -- 172.21.15.1:0/3007632167 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c00c0e0 con 0x7ff35c003be0 2024-03-18T20:07:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.494+0000 7ff367fff700 1 -- 172.21.15.1:0/3007632167 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c00c2a0 con 0x7ff35c005210 2024-03-18T20:07:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.494+0000 7ff37484d700 1 --2- 172.21.15.1:0/3007632167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c004850 0x7ff35c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.494+0000 7ff37504e700 1 --2- 172.21.15.1:0/3007632167 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff35c003be0 0x7ff35c003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.494+0000 7ff37584f700 1 --2- 172.21.15.1:0/3007632167 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff35c005210 0x7ff35c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.494+0000 7ff37484d700 1 -- 172.21.15.1:0/3007632167 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff35c005210 msgr2=0x7ff35c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:22.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.494+0000 7ff37484d700 1 --2- 172.21.15.1:0/3007632167 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff35c005210 0x7ff35c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.494+0000 7ff37484d700 1 -- 172.21.15.1:0/3007632167 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff35c003be0 msgr2=0x7ff35c003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.494+0000 7ff37484d700 1 --2- 172.21.15.1:0/3007632167 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff35c003be0 0x7ff35c003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.494+0000 7ff37584f700 1 -- 172.21.15.1:0/3007632167 reap_dead start 2024-03-18T20:07:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.494+0000 7ff37484d700 1 -- 172.21.15.1:0/3007632167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c0035b0 con 0x7ff35c004850 2024-03-18T20:07:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.494+0000 7ff37484d700 1 --2- 172.21.15.1:0/3007632167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c004850 0x7ff35c004cd0 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7ff360007fa0 tx=0x7ff35c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7edb1182c573261 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.494+0000 7ff367fff700 1 -- 172.21.15.1:0/3007632167 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36000cce0 con 0x7ff35c004850 2024-03-18T20:07:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.494+0000 7ff367fff700 1 -- 172.21.15.1:0/3007632167 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff36000ce40 con 0x7ff35c004850 2024-03-18T20:07:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.494+0000 7ff367fff700 1 -- 172.21.15.1:0/3007632167 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff360005ba0 con 0x7ff35c004850 2024-03-18T20:07:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.495+0000 7ff3772b2700 1 -- 172.21.15.1:0/3007632167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c004850 msgr2=0x7ff35c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.495+0000 7ff3772b2700 1 --2- 172.21.15.1:0/3007632167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c004850 0x7ff35c004cd0 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7ff360007fa0 tx=0x7ff35c005880 comp rx=0 tx=0).stop 2024-03-18T20:07:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.495+0000 7ff3772b2700 1 -- 172.21.15.1:0/3007632167 shutdown_connections 2024-03-18T20:07:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.495+0000 7ff3772b2700 1 --2- 172.21.15.1:0/3007632167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c004850 0x7ff35c004cd0 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.495+0000 7ff3772b2700 1 -- 172.21.15.1:0/3007632167 >> 172.21.15.1:0/3007632167 conn(0x7ff3700b2ba0 msgr2=0x7ff3700b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.495+0000 7ff3772b2700 1 -- 172.21.15.1:0/3007632167 shutdown_connections 2024-03-18T20:07:22.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.495+0000 7ff3772b2700 1 -- 172.21.15.1:0/3007632167 wait complete. 2024-03-18T20:07:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.496+0000 7ff3772b2700 1 Processor -- start 2024-03-18T20:07:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.496+0000 7ff3772b2700 1 -- start start 2024-03-18T20:07:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.497+0000 7ff3772b2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c004850 0x7ff37015dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:22.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.497+0000 7ff3772b2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff37015f3f0 0x7ff3700cac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.497+0000 7ff3772b2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff37014e140 0x7ff37015fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.497+0000 7ff3772b2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37015efd0 con 0x7ff35c004850 2024-03-18T20:07:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.497+0000 7ff3772b2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37015f870 con 0x7ff37014e140 2024-03-18T20:07:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.497+0000 7ff3772b2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3700cb1c0 con 0x7ff37015f3f0 2024-03-18T20:07:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.497+0000 7ff37504e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c004850 0x7ff37015dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.497+0000 7ff37504e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c004850 0x7ff37015dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37740/0 (socket says 172.21.15.1:37740) 2024-03-18T20:07:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.497+0000 7ff37504e700 1 -- 172.21.15.1:0/3318588956 learned_addr learned my addr 172.21.15.1:0/3318588956 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.497+0000 7ff37484d700 1 --2- 172.21.15.1:0/3318588956 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff37015f3f0 0x7ff3700cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.497+0000 7ff37584f700 1 --2- 172.21.15.1:0/3318588956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff37014e140 0x7ff37015fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.497+0000 7ff37504e700 1 -- 172.21.15.1:0/3318588956 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff37015f3f0 msgr2=0x7ff3700cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.497+0000 7ff37504e700 1 --2- 172.21.15.1:0/3318588956 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff37015f3f0 0x7ff3700cac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.497+0000 7ff37504e700 1 -- 172.21.15.1:0/3318588956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff37014e140 msgr2=0x7ff37015fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.497+0000 7ff37504e700 1 --2- 172.21.15.1:0/3318588956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff37014e140 0x7ff37015fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.497+0000 7ff37504e700 1 -- 172.21.15.1:0/3318588956 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff37015dbe0 con 0x7ff35c004850 2024-03-18T20:07:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.497+0000 7ff37504e700 1 --2- 172.21.15.1:0/3318588956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c004850 0x7ff37015dd70 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7ff36c0115d0 tx=0x7ff36c0118e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.498+0000 7ff365ffb700 1 -- 172.21.15.1:0/3318588956 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36c015870 con 0x7ff35c004850 2024-03-18T20:07:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.498+0000 7ff365ffb700 1 -- 172.21.15.1:0/3318588956 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff36c0159d0 con 0x7ff35c004850 2024-03-18T20:07:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.498+0000 7ff3772b2700 1 -- 172.21.15.1:0/3318588956 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37015e350 con 0x7ff35c004850 2024-03-18T20:07:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.498+0000 7ff365ffb700 1 -- 172.21.15.1:0/3318588956 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36c015d40 con 0x7ff35c004850 2024-03-18T20:07:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.498+0000 7ff3772b2700 1 -- 172.21.15.1:0/3318588956 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37015e910 con 0x7ff35c004850 2024-03-18T20:07:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.499+0000 7ff3772b2700 1 -- 172.21.15.1:0/3318588956 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff37015f240 con 0x7ff35c004850 2024-03-18T20:07:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.499+0000 7ff365ffb700 1 -- 172.21.15.1:0/3318588956 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff36c0143e0 con 0x7ff35c004850 2024-03-18T20:07:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.500+0000 7ff365ffb700 1 --2- 172.21.15.1:0/3318588956 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff35406adb0 0x7ff35406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.500+0000 7ff365ffb700 1 -- 172.21.15.1:0/3318588956 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff36c089740 con 0x7ff35c004850 2024-03-18T20:07:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.500+0000 7ff37484d700 1 --2- 172.21.15.1:0/3318588956 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff35406adb0 0x7ff35406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.500+0000 7ff37484d700 1 --2- 172.21.15.1:0/3318588956 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff35406adb0 0x7ff35406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff360007fa0 tx=0x7ff360007000 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.505+0000 7ff365ffb700 1 -- 172.21.15.1:0/3318588956 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff36c054710 con 0x7ff35c004850 2024-03-18T20:07:22.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.755+0000 7ff3772b2700 1 -- 172.21.15.1:0/3318588956 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3700b7250 con 0x7ff35c004850 2024-03-18T20:07:22.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.755+0000 7ff365ffb700 1 -- 172.21.15.1:0/3318588956 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4741 (secure 0 0 0) 0x7ff36c057d30 con 0x7ff35c004850 2024-03-18T20:07:22.760 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:07:22.761 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":89,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8204,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2572860438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2572860438},{"type":"v1","addr":"172.21.15.165:6835","nonce":2572860438}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9439,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3158632513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3158632513},{"type":"v1","addr":"172.21.15.1:6836","nonce":3158632513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9448,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/4005747445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4005747445},{"type":"v1","addr":"172.21.15.165:6834","nonce":4005747445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:07:19.322806+0000","modified":"2024-03-18T20:07:22.335335+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10422":{"gid":10422,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1610313086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1610313086},{"type":"v1","addr":"172.21.15.1:6837","nonce":1610313086}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-03-18T20:07:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.758+0000 7ff3772b2700 1 -- 172.21.15.1:0/3318588956 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff35406adb0 msgr2=0x7ff35406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.758+0000 7ff3772b2700 1 --2- 172.21.15.1:0/3318588956 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff35406adb0 0x7ff35406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff360007fa0 tx=0x7ff360007000 comp rx=0 tx=0).stop 2024-03-18T20:07:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.758+0000 7ff3772b2700 1 -- 172.21.15.1:0/3318588956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c004850 msgr2=0x7ff37015dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.758+0000 7ff3772b2700 1 --2- 172.21.15.1:0/3318588956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c004850 0x7ff37015dd70 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7ff36c0115d0 tx=0x7ff36c0118e0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.758+0000 7ff3772b2700 1 -- 172.21.15.1:0/3318588956 shutdown_connections 2024-03-18T20:07:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.758+0000 7ff3772b2700 1 --2- 172.21.15.1:0/3318588956 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7ff35406adb0 0x7ff35406d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.758+0000 7ff3772b2700 1 --2- 172.21.15.1:0/3318588956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff37014e140 0x7ff37015fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.758+0000 7ff3772b2700 1 --2- 172.21.15.1:0/3318588956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff35c004850 0x7ff37015dd70 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.758+0000 7ff3772b2700 1 --2- 172.21.15.1:0/3318588956 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff37015f3f0 0x7ff3700cac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.758+0000 7ff3772b2700 1 -- 172.21.15.1:0/3318588956 >> 172.21.15.1:0/3318588956 conn(0x7ff3700b2ba0 msgr2=0x7ff37006f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.758+0000 7ff3772b2700 1 -- 172.21.15.1:0/3318588956 shutdown_connections 2024-03-18T20:07:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.758+0000 7ff3772b2700 1 -- 172.21.15.1:0/3318588956 wait complete. 2024-03-18T20:07:22.764 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 89 2024-03-18T20:07:22.775 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:07:22.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.896+0000 7f49f8fce700 1 Processor -- start 2024-03-18T20:07:22.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.896+0000 7f49f8fce700 1 -- start start 2024-03-18T20:07:22.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.897+0000 7f49f8fce700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f415b4d0 0x7f49f409a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:22.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.897+0000 7f49f8fce700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f49f409a150 0x7f49f406f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:22.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.897+0000 7f49f8fce700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f406f070 con 0x7f49f40cf750 2024-03-18T20:07:22.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.897+0000 7f49f8fce700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49f40d0260 con 0x7f49f409a150 2024-03-18T20:07:22.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.897+0000 7f49f8fce700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f40cff80 con 0x7f49f415b4d0 2024-03-18T20:07:22.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.897+0000 7f49f2d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f49f409a150 0x7f49f406f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:22.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.897+0000 7f49f2d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f49f409a150 0x7f49f406f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37742/0 (socket says 172.21.15.1:37742) 2024-03-18T20:07:22.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.897+0000 7f49f2d9d700 1 -- 172.21.15.1:0/1851723034 learned_addr learned my addr 172.21.15.1:0/1851723034 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:22.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.897+0000 7f49f259c700 1 --2- 172.21.15.1:0/1851723034 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f415b4d0 0x7f49f409a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:22.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.897+0000 7f49f2d9d700 1 -- 172.21.15.1:0/1851723034 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f415b4d0 msgr2=0x7f49f409a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:22.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.897+0000 7f49f2d9d700 1 --2- 172.21.15.1:0/1851723034 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f415b4d0 0x7f49f409a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.897+0000 7f49f2d9d700 1 -- 172.21.15.1:0/1851723034 >> v1:172.21.15.165:6789/0 conn(0x7f49f40cf750 legacy=0x7f49f4099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:22.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.897+0000 7f49f2d9d700 1 -- 172.21.15.1:0/1851723034 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f406ea20 con 0x7f49f409a150 2024-03-18T20:07:22.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.897+0000 7f49f2d9d700 1 --2- 172.21.15.1:0/1851723034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f409a150 0x7f49f406f500 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f49ec009e50 tx=0x7f49ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edb5e6834b8db27b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:22.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.898+0000 7f49f159a700 1 -- 172.21.15.1:0/1851723034 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ec00e070 con 0x7f49f409a150 2024-03-18T20:07:22.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.898+0000 7f49f159a700 1 -- 172.21.15.1:0/1851723034 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49ec00b910 con 0x7f49f409a150 2024-03-18T20:07:22.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.898+0000 7f49f159a700 1 -- 172.21.15.1:0/1851723034 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ec00bcc0 con 0x7f49f409a150 2024-03-18T20:07:22.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.898+0000 7f49f8fce700 1 -- 172.21.15.1:0/1851723034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f409a150 msgr2=0x7f49f406f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:22.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.898+0000 7f49f8fce700 1 --2- 172.21.15.1:0/1851723034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f409a150 0x7f49f406f500 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f49ec009e50 tx=0x7f49ec00b330 comp rx=0 tx=0).stop 2024-03-18T20:07:22.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.898+0000 7f49f8fce700 1 -- 172.21.15.1:0/1851723034 shutdown_connections 2024-03-18T20:07:22.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.898+0000 7f49f8fce700 1 --2- 172.21.15.1:0/1851723034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f409a150 0x7f49f406f500 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.898+0000 7f49f8fce700 1 --2- 172.21.15.1:0/1851723034 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f415b4d0 0x7f49f409a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.898+0000 7f49f8fce700 1 -- 172.21.15.1:0/1851723034 >> 172.21.15.1:0/1851723034 conn(0x7f49f40b2cb0 msgr2=0x7f49f40b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:22.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.898+0000 7f49f8fce700 1 -- 172.21.15.1:0/1851723034 shutdown_connections 2024-03-18T20:07:22.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.898+0000 7f49f8fce700 1 -- 172.21.15.1:0/1851723034 wait complete. 2024-03-18T20:07:22.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.900+0000 7f49f8fce700 1 Processor -- start 2024-03-18T20:07:22.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.900+0000 7f49f8fce700 1 -- start start 2024-03-18T20:07:22.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.900+0000 7f49f8fce700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f409a150 0x7f49f41507e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:22.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.900+0000 7f49f8fce700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f40cf750 0x7f49f40c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:22.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.900+0000 7f49f8fce700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49f415b4d0 0x7f49f415c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:22.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.900+0000 7f49f8fce700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f40c5ff0 con 0x7f49f409a150 2024-03-18T20:07:22.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.900+0000 7f49f8fce700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f415cb10 con 0x7f49f415b4d0 2024-03-18T20:07:22.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.900+0000 7f49f8fce700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f40c5370 con 0x7f49f40cf750 2024-03-18T20:07:22.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.901+0000 7f49f259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f409a150 0x7f49f41507e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:22.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.901+0000 7f49f259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f409a150 0x7f49f41507e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37748/0 (socket says 172.21.15.1:37748) 2024-03-18T20:07:22.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.901+0000 7f49f259c700 1 -- 172.21.15.1:0/2037936057 learned_addr learned my addr 172.21.15.1:0/2037936057 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:22.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.901+0000 7f49f1d9b700 1 --2- 172.21.15.1:0/2037936057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f40cf750 0x7f49f40c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:22.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.901+0000 7f49f2d9d700 1 --2- 172.21.15.1:0/2037936057 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49f415b4d0 0x7f49f415c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:22.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.901+0000 7f49f259c700 1 -- 172.21.15.1:0/2037936057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f40cf750 msgr2=0x7f49f40c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:22.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.901+0000 7f49f259c700 1 --2- 172.21.15.1:0/2037936057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f40cf750 0x7f49f40c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.901+0000 7f49f259c700 1 -- 172.21.15.1:0/2037936057 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49f415b4d0 msgr2=0x7f49f415c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:22.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.901+0000 7f49f259c700 1 --2- 172.21.15.1:0/2037936057 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49f415b4d0 0x7f49f415c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:22.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.901+0000 7f49f259c700 1 -- 172.21.15.1:0/2037936057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ec009b30 con 0x7f49f409a150 2024-03-18T20:07:22.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.901+0000 7f49f259c700 1 --2- 172.21.15.1:0/2037936057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f409a150 0x7f49f41507e0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f49e400e870 tx=0x7f49e400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:22.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.901+0000 7f49e2ffd700 1 -- 172.21.15.1:0/2037936057 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e4009590 con 0x7f49f409a150 2024-03-18T20:07:22.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.901+0000 7f49e2ffd700 1 -- 172.21.15.1:0/2037936057 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e4013680 con 0x7f49f409a150 2024-03-18T20:07:22.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.901+0000 7f49f8fce700 1 -- 172.21.15.1:0/2037936057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f40cfd50 con 0x7f49f409a150 2024-03-18T20:07:22.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.901+0000 7f49e2ffd700 1 -- 172.21.15.1:0/2037936057 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e40139f0 con 0x7f49f409a150 2024-03-18T20:07:22.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.901+0000 7f49f8fce700 1 -- 172.21.15.1:0/2037936057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f40c5930 con 0x7f49f409a150 2024-03-18T20:07:22.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.902+0000 7f49f8fce700 1 -- 172.21.15.1:0/2037936057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f40c6260 con 0x7f49f409a150 2024-03-18T20:07:22.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.903+0000 7f49e2ffd700 1 -- 172.21.15.1:0/2037936057 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f49e4013b50 con 0x7f49f409a150 2024-03-18T20:07:22.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.903+0000 7f49e2ffd700 1 --2- 172.21.15.1:0/2037936057 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49d406adb0 0x7f49d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:22.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.903+0000 7f49e2ffd700 1 -- 172.21.15.1:0/2037936057 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f49e4089530 con 0x7f49f409a150 2024-03-18T20:07:22.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.903+0000 7f49f1d9b700 1 --2- 172.21.15.1:0/2037936057 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49d406adb0 0x7f49d406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:22.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.904+0000 7f49f1d9b700 1 --2- 172.21.15.1:0/2037936057 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49d406adb0 0x7f49d406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f49f40b1800 tx=0x7f49e8009040 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:22.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:22.908+0000 7f49e2ffd700 1 -- 172.21.15.1:0/2037936057 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f49e4054500 con 0x7f49f409a150 2024-03-18T20:07:23.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.109+0000 7f49f8fce700 1 -- 172.21.15.1:0/2037936057 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49f4068240 con 0x7f49f409a150 2024-03-18T20:07:23.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.110+0000 7f49e2ffd700 1 -- 172.21.15.1:0/2037936057 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) v1 ==== 74+0+16315 (secure 0 0 0) 0x7f49e4057b20 con 0x7f49f409a150 2024-03-18T20:07:23.115 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:07:23.115 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":195,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:07:20.324460+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:07:17.162084+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-18T20:07:18.295080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:07:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.113+0000 7f49f8fce700 1 -- 172.21.15.1:0/2037936057 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49d406adb0 msgr2=0x7f49d406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.113+0000 7f49f8fce700 1 --2- 172.21.15.1:0/2037936057 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49d406adb0 0x7f49d406d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f49f40b1800 tx=0x7f49e8009040 comp rx=0 tx=0).stop 2024-03-18T20:07:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.113+0000 7f49f8fce700 1 -- 172.21.15.1:0/2037936057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f409a150 msgr2=0x7f49f41507e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.113+0000 7f49f8fce700 1 --2- 172.21.15.1:0/2037936057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f409a150 0x7f49f41507e0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f49e400e870 tx=0x7f49e400eb80 comp rx=0 tx=0).stop 2024-03-18T20:07:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.113+0000 7f49f8fce700 1 -- 172.21.15.1:0/2037936057 shutdown_connections 2024-03-18T20:07:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.113+0000 7f49f8fce700 1 --2- 172.21.15.1:0/2037936057 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f49d406adb0 0x7f49d406d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.113+0000 7f49f8fce700 1 --2- 172.21.15.1:0/2037936057 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f49f415b4d0 0x7f49f415c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.113+0000 7f49f8fce700 1 --2- 172.21.15.1:0/2037936057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49f409a150 0x7f49f41507e0 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.113+0000 7f49f8fce700 1 --2- 172.21.15.1:0/2037936057 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f49f40cf750 0x7f49f40c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.113+0000 7f49f8fce700 1 -- 172.21.15.1:0/2037936057 >> 172.21.15.1:0/2037936057 conn(0x7f49f40b2cb0 msgr2=0x7f49f40b3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.113+0000 7f49f8fce700 1 -- 172.21.15.1:0/2037936057 shutdown_connections 2024-03-18T20:07:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.113+0000 7f49f8fce700 1 -- 172.21.15.1:0/2037936057 wait complete. 2024-03-18T20:07:23.130 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T20:07:23.130 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:07:23.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.250+0000 7fc1c71e6700 1 Processor -- start 2024-03-18T20:07:23.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.250+0000 7fc1c71e6700 1 -- start start 2024-03-18T20:07:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.251+0000 7fc1c71e6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1c0099a40 0x7fc1c0099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.251+0000 7fc1c71e6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc1c00cc310 0x7fc1c00cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.251+0000 7fc1c71e6700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c009a3e0 con 0x7fc1c015d680 2024-03-18T20:07:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.251+0000 7fc1c71e6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c00d0c80 con 0x7fc1c00cc310 2024-03-18T20:07:23.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.251+0000 7fc1c71e6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c00d09a0 con 0x7fc1c0099a40 2024-03-18T20:07:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.251+0000 7fc1c4f82700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc1c015d680 0x7fc1c00cff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43392/0 (socket says 172.21.15.1:43392) 2024-03-18T20:07:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.251+0000 7fc1c4f82700 1 -- 172.21.15.1:0/1998001212 learned_addr learned my addr 172.21.15.1:0/1998001212 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.251+0000 7fc1c5783700 1 --2- 172.21.15.1:0/1998001212 >> v2:172.21.15.165:3300/0 conn(0x7fc1c00cc310 0x7fc1c00cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.251+0000 7fc1bffff700 1 --2- 172.21.15.1:0/1998001212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1c0099a40 0x7fc1c0099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.251+0000 7fc1bf7fe700 1 -- 172.21.15.1:0/1998001212 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856745460 0 0) 0x7fc1c009a3e0 con 0x7fc1c015d680 2024-03-18T20:07:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.251+0000 7fc1bf7fe700 1 -- 172.21.15.1:0/1998001212 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b00035b0 con 0x7fc1c015d680 2024-03-18T20:07:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.251+0000 7fc1bf7fe700 1 -- 172.21.15.1:0/1998001212 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3215598726 0 0) 0x7fc1b00035b0 con 0x7fc1c015d680 2024-03-18T20:07:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.251+0000 7fc1bf7fe700 1 -- 172.21.15.1:0/1998001212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1c0099a40 msgr2=0x7fc1c0099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.251+0000 7fc1bf7fe700 1 --2- 172.21.15.1:0/1998001212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1c0099a40 0x7fc1c0099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.251+0000 7fc1bf7fe700 1 -- 172.21.15.1:0/1998001212 >> v2:172.21.15.165:3300/0 conn(0x7fc1c00cc310 msgr2=0x7fc1c00cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.251+0000 7fc1bf7fe700 1 --2- 172.21.15.1:0/1998001212 >> v2:172.21.15.165:3300/0 conn(0x7fc1c00cc310 0x7fc1c00cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.251+0000 7fc1bf7fe700 1 -- 172.21.15.1:0/1998001212 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c00cb830 con 0x7fc1c015d680 2024-03-18T20:07:23.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.251+0000 7fc1bf7fe700 1 -- 172.21.15.1:0/1998001212 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fc1b4001670 con 0x7fc1c015d680 2024-03-18T20:07:23.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.251+0000 7fc1bf7fe700 1 -- 172.21.15.1:0/1998001212 >> v1:172.21.15.1:6789/0 conn(0x7fc1c015d680 legacy=0x7fc1c00cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:23.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.252+0000 7fc1bf7fe700 1 --2- 172.21.15.1:0/1998001212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1b0003a00 0x7fc1b0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:23.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.252+0000 7fc1bf7fe700 1 --2- 172.21.15.1:0/1998001212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1b0004850 0x7fc1b0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:23.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.252+0000 7fc1bf7fe700 1 --2- 172.21.15.1:0/1998001212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1c0099a40 0x7fc1b0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:23.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.252+0000 7fc1bf7fe700 1 -- 172.21.15.1:0/1998001212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c00d0c80 con 0x7fc1b0004850 2024-03-18T20:07:23.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.252+0000 7fc1bf7fe700 1 -- 172.21.15.1:0/1998001212 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c00d09a0 con 0x7fc1b0003a00 2024-03-18T20:07:23.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.252+0000 7fc1bf7fe700 1 -- 172.21.15.1:0/1998001212 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b000ba10 con 0x7fc1c0099a40 2024-03-18T20:07:23.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.252+0000 7fc1bffff700 1 --2- 172.21.15.1:0/1998001212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1b0004850 0x7fc1b0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:23.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.252+0000 7fc1c4f82700 1 --2- 172.21.15.1:0/1998001212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1b0003a00 0x7fc1b0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:23.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.252+0000 7fc1c5783700 1 --2- 172.21.15.1:0/1998001212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1c0099a40 0x7fc1b0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:23.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.252+0000 7fc1bffff700 1 -- 172.21.15.1:0/1998001212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1c0099a40 msgr2=0x7fc1b0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:23.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.252+0000 7fc1bffff700 1 --2- 172.21.15.1:0/1998001212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1c0099a40 0x7fc1b0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.252+0000 7fc1bffff700 1 -- 172.21.15.1:0/1998001212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1b0003a00 msgr2=0x7fc1b0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:23.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.252+0000 7fc1bffff700 1 --2- 172.21.15.1:0/1998001212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1b0003a00 0x7fc1b0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.252+0000 7fc1bffff700 1 -- 172.21.15.1:0/1998001212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b00035b0 con 0x7fc1b0004850 2024-03-18T20:07:23.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.252+0000 7fc1c5783700 1 --2- 172.21.15.1:0/1998001212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1c0099a40 0x7fc1b0005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:07:23.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.252+0000 7fc1bffff700 1 --2- 172.21.15.1:0/1998001212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1b0004850 0x7fc1b0004cd0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac009b60 tx=0x7fc1b0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c47ef7f3c5b4115 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:23.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.252+0000 7fc1bf7fe700 1 -- 172.21.15.1:0/1998001212 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac00e660 con 0x7fc1b0004850 2024-03-18T20:07:23.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.252+0000 7fc1bf7fe700 1 -- 172.21.15.1:0/1998001212 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ac00e7c0 con 0x7fc1b0004850 2024-03-18T20:07:23.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.252+0000 7fc1bf7fe700 1 -- 172.21.15.1:0/1998001212 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac00eb30 con 0x7fc1b0004850 2024-03-18T20:07:23.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.253+0000 7fc1c71e6700 1 -- 172.21.15.1:0/1998001212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1b0004850 msgr2=0x7fc1b0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:23.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.253+0000 7fc1c71e6700 1 --2- 172.21.15.1:0/1998001212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1b0004850 0x7fc1b0004cd0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac009b60 tx=0x7fc1b0005880 comp rx=0 tx=0).stop 2024-03-18T20:07:23.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.253+0000 7fc1c5783700 1 -- 172.21.15.1:0/1998001212 reap_dead start 2024-03-18T20:07:23.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.253+0000 7fc1c71e6700 1 -- 172.21.15.1:0/1998001212 shutdown_connections 2024-03-18T20:07:23.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.253+0000 7fc1c71e6700 1 -- 172.21.15.1:0/1998001212 >> 172.21.15.1:0/1998001212 conn(0x7fc1c00b2ba0 msgr2=0x7fc1c00b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:23.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.253+0000 7fc1c71e6700 1 -- 172.21.15.1:0/1998001212 shutdown_connections 2024-03-18T20:07:23.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.253+0000 7fc1c71e6700 1 -- 172.21.15.1:0/1998001212 wait complete. 2024-03-18T20:07:23.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.254+0000 7fc1c71e6700 1 Processor -- start 2024-03-18T20:07:23.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.254+0000 7fc1c71e6700 1 -- start start 2024-03-18T20:07:23.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.255+0000 7fc1c71e6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1b0004850 0x7fc1c0160430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:23.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.255+0000 7fc1c71e6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1c014c340 0x7fc1c00b6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:23.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.255+0000 7fc1c71e6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1c00b69b0 0x7fc1c00b3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:23.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.255+0000 7fc1c71e6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c015f390 con 0x7fc1c014c340 2024-03-18T20:07:23.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.255+0000 7fc1c71e6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c015f670 con 0x7fc1c00b69b0 2024-03-18T20:07:23.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.255+0000 7fc1c71e6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c015f950 con 0x7fc1b0004850 2024-03-18T20:07:23.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.255+0000 7fc1bffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1c014c340 0x7fc1c00b6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:23.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.255+0000 7fc1bffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1c014c340 0x7fc1c00b6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37766/0 (socket says 172.21.15.1:37766) 2024-03-18T20:07:23.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.255+0000 7fc1bffff700 1 -- 172.21.15.1:0/537063403 learned_addr learned my addr 172.21.15.1:0/537063403 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:23.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.255+0000 7fc1c4f82700 1 --2- 172.21.15.1:0/537063403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1b0004850 0x7fc1c0160430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:23.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.255+0000 7fc1c5783700 1 --2- 172.21.15.1:0/537063403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1c00b69b0 0x7fc1c00b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:23.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.255+0000 7fc1bffff700 1 -- 172.21.15.1:0/537063403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1b0004850 msgr2=0x7fc1c0160430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:23.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.255+0000 7fc1bffff700 1 --2- 172.21.15.1:0/537063403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1b0004850 0x7fc1c0160430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.255+0000 7fc1bffff700 1 -- 172.21.15.1:0/537063403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1c00b69b0 msgr2=0x7fc1c00b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:23.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.255+0000 7fc1bffff700 1 --2- 172.21.15.1:0/537063403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1c00b69b0 0x7fc1c00b3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.255+0000 7fc1bffff700 1 -- 172.21.15.1:0/537063403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c015fc30 con 0x7fc1c014c340 2024-03-18T20:07:23.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.255+0000 7fc1c5783700 1 --2- 172.21.15.1:0/537063403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1c00b69b0 0x7fc1c00b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:07:23.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.256+0000 7fc1bffff700 1 --2- 172.21.15.1:0/537063403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1c014c340 0x7fc1c00b6470 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fc1b00046f0 tx=0x7fc1ac0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:23.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.256+0000 7fc1bd7fa700 1 -- 172.21.15.1:0/537063403 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac00e660 con 0x7fc1c014c340 2024-03-18T20:07:23.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.256+0000 7fc1bd7fa700 1 -- 172.21.15.1:0/537063403 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ac003ad0 con 0x7fc1c014c340 2024-03-18T20:07:23.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.256+0000 7fc1c71e6700 1 -- 172.21.15.1:0/537063403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c015ff10 con 0x7fc1c014c340 2024-03-18T20:07:23.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.256+0000 7fc1bd7fa700 1 -- 172.21.15.1:0/537063403 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac003e80 con 0x7fc1c014c340 2024-03-18T20:07:23.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.256+0000 7fc1c71e6700 1 -- 172.21.15.1:0/537063403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c015efb0 con 0x7fc1c014c340 2024-03-18T20:07:23.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.257+0000 7fc1c71e6700 1 -- 172.21.15.1:0/537063403 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c0160280 con 0x7fc1c014c340 2024-03-18T20:07:23.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.257+0000 7fc1bd7fa700 1 -- 172.21.15.1:0/537063403 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc1ac013470 con 0x7fc1c014c340 2024-03-18T20:07:23.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.258+0000 7fc1bd7fa700 1 --2- 172.21.15.1:0/537063403 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc1a406adb0 0x7fc1a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:23.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.258+0000 7fc1bd7fa700 1 -- 172.21.15.1:0/537063403 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc1ac08ade0 con 0x7fc1c014c340 2024-03-18T20:07:23.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.258+0000 7fc1c4f82700 1 --2- 172.21.15.1:0/537063403 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc1a406adb0 0x7fc1a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:23.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.258+0000 7fc1c4f82700 1 --2- 172.21.15.1:0/537063403 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc1a406adb0 0x7fc1a406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4000fb0 tx=0x7fc1b4001e00 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:23.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.262+0000 7fc1bd7fa700 1 -- 172.21.15.1:0/537063403 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc1ac055ed0 con 0x7fc1c014c340 2024-03-18T20:07:23.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.461+0000 7fc1c71e6700 1 -- 172.21.15.1:0/537063403 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc1c015f7e0 con 0x7fc1a406adb0 2024-03-18T20:07:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.469+0000 7fc1bd7fa700 1 -- 172.21.15.1:0/537063403 <== mgr.10200 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+282003 (secure 0 0 0) 0x7fc1c015f7e0 con 0x7fc1a406adb0 2024-03-18T20:07:23.477 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:07:23.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.473+0000 7fc1c71e6700 1 -- 172.21.15.1:0/537063403 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc1a406adb0 msgr2=0x7fc1a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:23.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.473+0000 7fc1c71e6700 1 --2- 172.21.15.1:0/537063403 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc1a406adb0 0x7fc1a406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4000fb0 tx=0x7fc1b4001e00 comp rx=0 tx=0).stop 2024-03-18T20:07:23.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.473+0000 7fc1c71e6700 1 -- 172.21.15.1:0/537063403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1c014c340 msgr2=0x7fc1c00b6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:23.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.473+0000 7fc1c71e6700 1 --2- 172.21.15.1:0/537063403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1c014c340 0x7fc1c00b6470 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fc1b00046f0 tx=0x7fc1ac0047c0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.473+0000 7fc1c71e6700 1 -- 172.21.15.1:0/537063403 shutdown_connections 2024-03-18T20:07:23.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.473+0000 7fc1c71e6700 1 --2- 172.21.15.1:0/537063403 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fc1a406adb0 0x7fc1a406d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.473+0000 7fc1c71e6700 1 --2- 172.21.15.1:0/537063403 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1b0004850 0x7fc1c0160430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.473+0000 7fc1c71e6700 1 --2- 172.21.15.1:0/537063403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1c00b69b0 0x7fc1c00b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.473+0000 7fc1c71e6700 1 --2- 172.21.15.1:0/537063403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1c014c340 0x7fc1c00b6470 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.473+0000 7fc1c71e6700 1 -- 172.21.15.1:0/537063403 >> 172.21.15.1:0/537063403 conn(0x7fc1c00b2ba0 msgr2=0x7fc1c00cc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:23.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.473+0000 7fc1c71e6700 1 -- 172.21.15.1:0/537063403 shutdown_connections 2024-03-18T20:07:23.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.473+0000 7fc1c71e6700 1 -- 172.21.15.1:0/537063403 wait complete. 2024-03-18T20:07:23.480 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:07:23.490 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-18T20:07:21.632013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591040,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":502,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":311,"ondisk_log_size":311,"up":88,"acting":88,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":95,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":373944,"kb_used_data":20032,"kb_used_omap":12,"kb_used_meta":353779,"kb_avail":754600776,"statfs":{"total":773094113280,"available":772711194624,"internally_reserved":0,"allocated":20512768,"data_stored":10966377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":362270265},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000733"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332710+0000","last_change":"2024-03-18T20:07:19.345101+0000","last_active":"2024-03-18T20:07:20.332710+0000","last_peered":"2024-03-18T20:07:20.332710+0000","last_clean":"2024-03-18T20:07:20.332710+0000","last_became_active":"2024-03-18T20:07:19.344798+0000","last_became_peered":"2024-03-18T20:07:19.344798+0000","last_unstale":"2024-03-18T20:07:20.332710+0000","last_undegraded":"2024-03-18T20:07:20.332710+0000","last_fullsized":"2024-03-18T20:07:20.332710+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:29:33.340151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454245+0000","last_change":"2024-03-18T20:07:19.345726+0000","last_active":"2024-03-18T20:07:20.454245+0000","last_peered":"2024-03-18T20:07:20.454245+0000","last_clean":"2024-03-18T20:07:20.454245+0000","last_became_active":"2024-03-18T20:07:19.345456+0000","last_became_peered":"2024-03-18T20:07:19.345456+0000","last_unstale":"2024-03-18T20:07:20.454245+0000","last_undegraded":"2024-03-18T20:07:20.454245+0000","last_fullsized":"2024-03-18T20:07:20.454245+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:00:37.910486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332715+0000","last_change":"2024-03-18T20:07:19.345435+0000","last_active":"2024-03-18T20:07:20.332715+0000","last_peered":"2024-03-18T20:07:20.332715+0000","last_clean":"2024-03-18T20:07:20.332715+0000","last_became_active":"2024-03-18T20:07:19.345015+0000","last_became_peered":"2024-03-18T20:07:19.345015+0000","last_unstale":"2024-03-18T20:07:20.332715+0000","last_undegraded":"2024-03-18T20:07:20.332715+0000","last_fullsized":"2024-03-18T20:07:20.332715+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:00:43.939604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332531+0000","last_change":"2024-03-18T20:07:19.353667+0000","last_active":"2024-03-18T20:07:20.332531+0000","last_peered":"2024-03-18T20:07:20.332531+0000","last_clean":"2024-03-18T20:07:20.332531+0000","last_became_active":"2024-03-18T20:07:19.353304+0000","last_became_peered":"2024-03-18T20:07:19.353304+0000","last_unstale":"2024-03-18T20:07:20.332531+0000","last_undegraded":"2024-03-18T20:07:20.332531+0000","last_fullsized":"2024-03-18T20:07:20.332531+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:41.216249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454036+0000","last_change":"2024-03-18T20:07:19.341565+0000","last_active":"2024-03-18T20:07:20.454036+0000","last_peered":"2024-03-18T20:07:20.454036+0000","last_clean":"2024-03-18T20:07:20.454036+0000","last_became_active":"2024-03-18T20:07:19.341293+0000","last_became_peered":"2024-03-18T20:07:19.341293+0000","last_unstale":"2024-03-18T20:07:20.454036+0000","last_undegraded":"2024-03-18T20:07:20.454036+0000","last_fullsized":"2024-03-18T20:07:20.454036+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:31:39.212677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617671+0000","last_change":"2024-03-18T20:07:19.344583+0000","last_active":"2024-03-18T20:07:20.617671+0000","last_peered":"2024-03-18T20:07:20.617671+0000","last_clean":"2024-03-18T20:07:20.617671+0000","last_became_active":"2024-03-18T20:07:19.344312+0000","last_became_peered":"2024-03-18T20:07:19.344312+0000","last_unstale":"2024-03-18T20:07:20.617671+0000","last_undegraded":"2024-03-18T20:07:20.617671+0000","last_fullsized":"2024-03-18T20:07:20.617671+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:56:21.591342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.616710+0000","last_change":"2024-03-18T20:07:19.348770+0000","last_active":"2024-03-18T20:07:20.616710+0000","last_peered":"2024-03-18T20:07:20.616710+0000","last_clean":"2024-03-18T20:07:20.616710+0000","last_became_active":"2024-03-18T20:07:19.345749+0000","last_became_peered":"2024-03-18T20:07:19.345749+0000","last_unstale":"2024-03-18T20:07:20.616710+0000","last_undegraded":"2024-03-18T20:07:20.616710+0000","last_fullsized":"2024-03-18T20:07:20.616710+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:34.547722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618596+0000","last_change":"2024-03-18T20:07:19.335107+0000","last_active":"2024-03-18T20:07:20.618596+0000","last_peered":"2024-03-18T20:07:20.618596+0000","last_clean":"2024-03-18T20:07:20.618596+0000","last_became_active":"2024-03-18T20:07:19.334865+0000","last_became_peered":"2024-03-18T20:07:19.334865+0000","last_unstale":"2024-03-18T20:07:20.618596+0000","last_undegraded":"2024-03-18T20:07:20.618596+0000","last_fullsized":"2024-03-18T20:07:20.618596+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:44.623779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618270+0000","last_change":"2024-03-18T20:07:19.341279+0000","last_active":"2024-03-18T20:07:20.618270+0000","last_peered":"2024-03-18T20:07:20.618270+0000","last_clean":"2024-03-18T20:07:20.618270+0000","last_became_active":"2024-03-18T20:07:19.341020+0000","last_became_peered":"2024-03-18T20:07:19.341020+0000","last_unstale":"2024-03-18T20:07:20.618270+0000","last_undegraded":"2024-03-18T20:07:20.618270+0000","last_fullsized":"2024-03-18T20:07:20.618270+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:24.003624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454130+0000","last_change":"2024-03-18T20:07:19.341875+0000","last_active":"2024-03-18T20:07:20.454130+0000","last_peered":"2024-03-18T20:07:20.454130+0000","last_clean":"2024-03-18T20:07:20.454130+0000","last_became_active":"2024-03-18T20:07:19.341594+0000","last_became_peered":"2024-03-18T20:07:19.341594+0000","last_unstale":"2024-03-18T20:07:20.454130+0000","last_undegraded":"2024-03-18T20:07:20.454130+0000","last_fullsized":"2024-03-18T20:07:20.454130+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:18:43.940601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332298+0000","last_change":"2024-03-18T20:07:19.354331+0000","last_active":"2024-03-18T20:07:20.332298+0000","last_peered":"2024-03-18T20:07:20.332298+0000","last_clean":"2024-03-18T20:07:20.332298+0000","last_became_active":"2024-03-18T20:07:19.354058+0000","last_became_peered":"2024-03-18T20:07:19.354058+0000","last_unstale":"2024-03-18T20:07:20.332298+0000","last_undegraded":"2024-03-18T20:07:20.332298+0000","last_fullsized":"2024-03-18T20:07:20.332298+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:42:23.929450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618687+0000","last_change":"2024-03-18T20:07:19.340616+0000","last_active":"2024-03-18T20:07:20.618687+0000","last_peered":"2024-03-18T20:07:20.618687+0000","last_clean":"2024-03-18T20:07:20.618687+0000","last_became_active":"2024-03-18T20:07:19.340399+0000","last_became_peered":"2024-03-18T20:07:19.340399+0000","last_unstale":"2024-03-18T20:07:20.618687+0000","last_undegraded":"2024-03-18T20:07:20.618687+0000","last_fullsized":"2024-03-18T20:07:20.618687+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:14:59.554787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.331336+0000","last_change":"2024-03-18T20:07:19.337516+0000","last_active":"2024-03-18T20:07:20.331336+0000","last_peered":"2024-03-18T20:07:20.331336+0000","last_clean":"2024-03-18T20:07:20.331336+0000","last_became_active":"2024-03-18T20:07:19.337251+0000","last_became_peered":"2024-03-18T20:07:19.337251+0000","last_unstale":"2024-03-18T20:07:20.331336+0000","last_undegraded":"2024-03-18T20:07:20.331336+0000","last_fullsized":"2024-03-18T20:07:20.331336+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:54:47.068491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.453972+0000","last_change":"2024-03-18T20:07:19.349446+0000","last_active":"2024-03-18T20:07:20.453972+0000","last_peered":"2024-03-18T20:07:20.453972+0000","last_clean":"2024-03-18T20:07:20.453972+0000","last_became_active":"2024-03-18T20:07:19.349155+0000","last_became_peered":"2024-03-18T20:07:19.349155+0000","last_unstale":"2024-03-18T20:07:20.453972+0000","last_undegraded":"2024-03-18T20:07:20.453972+0000","last_fullsized":"2024-03-18T20:07:20.453972+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:17:52.731361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332440+0000","last_change":"2024-03-18T20:07:19.344031+0000","last_active":"2024-03-18T20:07:20.332440+0000","last_peered":"2024-03-18T20:07:20.332440+0000","last_clean":"2024-03-18T20:07:20.332440+0000","last_became_active":"2024-03-18T20:07:19.341621+0000","last_became_peered":"2024-03-18T20:07:19.341621+0000","last_unstale":"2024-03-18T20:07:20.332440+0000","last_undegraded":"2024-03-18T20:07:20.332440+0000","last_fullsized":"2024-03-18T20:07:20.332440+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:27:33.209246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454074+0000","last_change":"2024-03-18T20:07:19.350206+0000","last_active":"2024-03-18T20:07:20.454074+0000","last_peered":"2024-03-18T20:07:20.454074+0000","last_clean":"2024-03-18T20:07:20.454074+0000","last_became_active":"2024-03-18T20:07:19.349950+0000","last_became_peered":"2024-03-18T20:07:19.349950+0000","last_unstale":"2024-03-18T20:07:20.454074+0000","last_undegraded":"2024-03-18T20:07:20.454074+0000","last_fullsized":"2024-03-18T20:07:20.454074+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:39:31.996325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454124+0000","last_change":"2024-03-18T20:07:19.342254+0000","last_active":"2024-03-18T20:07:20.454124+0000","last_peered":"2024-03-18T20:07:20.454124+0000","last_clean":"2024-03-18T20:07:20.454124+0000","last_became_active":"2024-03-18T20:07:19.341997+0000","last_became_peered":"2024-03-18T20:07:19.341997+0000","last_unstale":"2024-03-18T20:07:20.454124+0000","last_undegraded":"2024-03-18T20:07:20.454124+0000","last_fullsized":"2024-03-18T20:07:20.454124+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:15:16.471603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.330792+0000","last_change":"2024-03-18T20:07:19.343299+0000","last_active":"2024-03-18T20:07:20.330792+0000","last_peered":"2024-03-18T20:07:20.330792+0000","last_clean":"2024-03-18T20:07:20.330792+0000","last_became_active":"2024-03-18T20:07:19.343031+0000","last_became_peered":"2024-03-18T20:07:19.343031+0000","last_unstale":"2024-03-18T20:07:20.330792+0000","last_undegraded":"2024-03-18T20:07:20.330792+0000","last_fullsized":"2024-03-18T20:07:20.330792+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:20:46.798924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.330581+0000","last_change":"2024-03-18T20:07:19.334107+0000","last_active":"2024-03-18T20:07:20.330581+0000","last_peered":"2024-03-18T20:07:20.330581+0000","last_clean":"2024-03-18T20:07:20.330581+0000","last_became_active":"2024-03-18T20:07:19.333820+0000","last_became_peered":"2024-03-18T20:07:19.333820+0000","last_unstale":"2024-03-18T20:07:20.330581+0000","last_undegraded":"2024-03-18T20:07:20.330581+0000","last_fullsized":"2024-03-18T20:07:20.330581+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:28.280469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454281+0000","last_change":"2024-03-18T20:07:19.336692+0000","last_active":"2024-03-18T20:07:20.454281+0000","last_peered":"2024-03-18T20:07:20.454281+0000","last_clean":"2024-03-18T20:07:20.454281+0000","last_became_active":"2024-03-18T20:07:19.336414+0000","last_became_peered":"2024-03-18T20:07:19.336414+0000","last_unstale":"2024-03-18T20:07:20.454281+0000","last_undegraded":"2024-03-18T20:07:20.454281+0000","last_fullsized":"2024-03-18T20:07:20.454281+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:24:28.820103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"194'4","reported_seq":15,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332769+0000","last_change":"2024-03-18T20:07:18.912523+0000","last_active":"2024-03-18T20:07:20.332769+0000","last_peered":"2024-03-18T20:07:20.332769+0000","last_clean":"2024-03-18T20:07:20.332769+0000","last_became_active":"2024-03-18T20:07:18.912324+0000","last_became_peered":"2024-03-18T20:07:18.912324+0000","last_unstale":"2024-03-18T20:07:20.332769+0000","last_undegraded":"2024-03-18T20:07:20.332769+0000","last_fullsized":"2024-03-18T20:07:20.332769+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:22.044963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454094+0000","last_change":"2024-03-18T20:07:19.351678+0000","last_active":"2024-03-18T20:07:20.454094+0000","last_peered":"2024-03-18T20:07:20.454094+0000","last_clean":"2024-03-18T20:07:20.454094+0000","last_became_active":"2024-03-18T20:07:19.351423+0000","last_became_peered":"2024-03-18T20:07:19.351423+0000","last_unstale":"2024-03-18T20:07:20.454094+0000","last_undegraded":"2024-03-18T20:07:20.454094+0000","last_fullsized":"2024-03-18T20:07:20.454094+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:03:27.829819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617832+0000","last_change":"2024-03-18T20:07:19.342067+0000","last_active":"2024-03-18T20:07:20.617832+0000","last_peered":"2024-03-18T20:07:20.617832+0000","last_clean":"2024-03-18T20:07:20.617832+0000","last_became_active":"2024-03-18T20:07:19.341811+0000","last_became_peered":"2024-03-18T20:07:19.341811+0000","last_unstale":"2024-03-18T20:07:20.617832+0000","last_undegraded":"2024-03-18T20:07:20.617832+0000","last_fullsized":"2024-03-18T20:07:20.617832+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:31:06.816211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"194'1","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617793+0000","last_change":"2024-03-18T20:07:18.907542+0000","last_active":"2024-03-18T20:07:20.617793+0000","last_peered":"2024-03-18T20:07:20.617793+0000","last_clean":"2024-03-18T20:07:20.617793+0000","last_became_active":"2024-03-18T20:07:18.907272+0000","last_became_peered":"2024-03-18T20:07:18.907272+0000","last_unstale":"2024-03-18T20:07:20.617793+0000","last_undegraded":"2024-03-18T20:07:20.617793+0000","last_fullsized":"2024-03-18T20:07:20.617793+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:05:29.037731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332707+0000","last_change":"2024-03-18T20:07:19.340706+0000","last_active":"2024-03-18T20:07:20.332707+0000","last_peered":"2024-03-18T20:07:20.332707+0000","last_clean":"2024-03-18T20:07:20.332707+0000","last_became_active":"2024-03-18T20:07:19.340436+0000","last_became_peered":"2024-03-18T20:07:19.340436+0000","last_unstale":"2024-03-18T20:07:20.332707+0000","last_undegraded":"2024-03-18T20:07:20.332707+0000","last_fullsized":"2024-03-18T20:07:20.332707+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:31:40.598513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.616776+0000","last_change":"2024-03-18T20:07:18.902226+0000","last_active":"2024-03-18T20:07:20.616776+0000","last_peered":"2024-03-18T20:07:20.616776+0000","last_clean":"2024-03-18T20:07:20.616776+0000","last_became_active":"2024-03-18T20:07:18.901889+0000","last_became_peered":"2024-03-18T20:07:18.901889+0000","last_unstale":"2024-03-18T20:07:20.616776+0000","last_undegraded":"2024-03-18T20:07:20.616776+0000","last_fullsized":"2024-03-18T20:07:20.616776+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:44.817526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454131+0000","last_change":"2024-03-18T20:07:19.336872+0000","last_active":"2024-03-18T20:07:20.454131+0000","last_peered":"2024-03-18T20:07:20.454131+0000","last_clean":"2024-03-18T20:07:20.454131+0000","last_became_active":"2024-03-18T20:07:19.336598+0000","last_became_peered":"2024-03-18T20:07:19.336598+0000","last_unstale":"2024-03-18T20:07:20.454131+0000","last_undegraded":"2024-03-18T20:07:20.454131+0000","last_fullsized":"2024-03-18T20:07:20.454131+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:49:59.914780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.15","version":"194'3","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332818+0000","last_change":"2024-03-18T20:07:18.898929+0000","last_active":"2024-03-18T20:07:20.332818+0000","last_peered":"2024-03-18T20:07:20.332818+0000","last_clean":"2024-03-18T20:07:20.332818+0000","last_became_active":"2024-03-18T20:07:18.898644+0000","last_became_peered":"2024-03-18T20:07:18.898644+0000","last_unstale":"2024-03-18T20:07:20.332818+0000","last_undegraded":"2024-03-18T20:07:20.332818+0000","last_fullsized":"2024-03-18T20:07:20.332818+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:56:21.115285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.330786+0000","last_change":"2024-03-18T20:07:19.334165+0000","last_active":"2024-03-18T20:07:20.330786+0000","last_peered":"2024-03-18T20:07:20.330786+0000","last_clean":"2024-03-18T20:07:20.330786+0000","last_became_active":"2024-03-18T20:07:19.333882+0000","last_became_peered":"2024-03-18T20:07:19.333882+0000","last_unstale":"2024-03-18T20:07:20.330786+0000","last_undegraded":"2024-03-18T20:07:20.330786+0000","last_fullsized":"2024-03-18T20:07:20.330786+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:43:27.974268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618048+0000","last_change":"2024-03-18T20:07:19.343671+0000","last_active":"2024-03-18T20:07:20.618048+0000","last_peered":"2024-03-18T20:07:20.618048+0000","last_clean":"2024-03-18T20:07:20.618048+0000","last_became_active":"2024-03-18T20:07:19.343131+0000","last_became_peered":"2024-03-18T20:07:19.343131+0000","last_unstale":"2024-03-18T20:07:20.618048+0000","last_undegraded":"2024-03-18T20:07:20.618048+0000","last_fullsized":"2024-03-18T20:07:20.618048+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:02.622237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.10","version":"194'8","reported_seq":19,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617529+0000","last_change":"2024-03-18T20:07:18.902132+0000","last_active":"2024-03-18T20:07:20.617529+0000","last_peered":"2024-03-18T20:07:20.617529+0000","last_clean":"2024-03-18T20:07:20.617529+0000","last_became_active":"2024-03-18T20:07:18.901852+0000","last_became_peered":"2024-03-18T20:07:18.901852+0000","last_unstale":"2024-03-18T20:07:20.617529+0000","last_undegraded":"2024-03-18T20:07:20.617529+0000","last_fullsized":"2024-03-18T20:07:20.617529+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:39:30.990897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617433+0000","last_change":"2024-03-18T20:07:19.347506+0000","last_active":"2024-03-18T20:07:20.617433+0000","last_peered":"2024-03-18T20:07:20.617433+0000","last_clean":"2024-03-18T20:07:20.617433+0000","last_became_active":"2024-03-18T20:07:19.345389+0000","last_became_peered":"2024-03-18T20:07:19.345389+0000","last_unstale":"2024-03-18T20:07:20.617433+0000","last_undegraded":"2024-03-18T20:07:20.617433+0000","last_fullsized":"2024-03-18T20:07:20.617433+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:09:26.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.330943+0000","last_change":"2024-03-18T20:07:19.338909+0000","last_active":"2024-03-18T20:07:20.330943+0000","last_peered":"2024-03-18T20:07:20.330943+0000","last_clean":"2024-03-18T20:07:20.330943+0000","last_became_active":"2024-03-18T20:07:19.338001+0000","last_became_peered":"2024-03-18T20:07:19.338001+0000","last_unstale":"2024-03-18T20:07:20.330943+0000","last_undegraded":"2024-03-18T20:07:20.330943+0000","last_fullsized":"2024-03-18T20:07:20.330943+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:42:02.327061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"194'2","reported_seq":13,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618179+0000","last_change":"2024-03-18T20:07:18.908765+0000","last_active":"2024-03-18T20:07:20.618179+0000","last_peered":"2024-03-18T20:07:20.618179+0000","last_clean":"2024-03-18T20:07:20.618179+0000","last_became_active":"2024-03-18T20:07:18.908470+0000","last_became_peered":"2024-03-18T20:07:18.908470+0000","last_unstale":"2024-03-18T20:07:20.618179+0000","last_undegraded":"2024-03-18T20:07:20.618179+0000","last_fullsized":"2024-03-18T20:07:20.618179+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:23:53.048106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617914+0000","last_change":"2024-03-18T20:07:18.908241+0000","last_active":"2024-03-18T20:07:20.617914+0000","last_peered":"2024-03-18T20:07:20.617914+0000","last_clean":"2024-03-18T20:07:20.617914+0000","last_became_active":"2024-03-18T20:07:18.907935+0000","last_became_peered":"2024-03-18T20:07:18.907935+0000","last_unstale":"2024-03-18T20:07:20.617914+0000","last_undegraded":"2024-03-18T20:07:20.617914+0000","last_fullsized":"2024-03-18T20:07:20.617914+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:29.743674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454308+0000","last_change":"2024-03-18T20:07:19.345125+0000","last_active":"2024-03-18T20:07:20.454308+0000","last_peered":"2024-03-18T20:07:20.454308+0000","last_clean":"2024-03-18T20:07:20.454308+0000","last_became_active":"2024-03-18T20:07:19.344838+0000","last_became_peered":"2024-03-18T20:07:19.344838+0000","last_unstale":"2024-03-18T20:07:20.454308+0000","last_undegraded":"2024-03-18T20:07:20.454308+0000","last_fullsized":"2024-03-18T20:07:20.454308+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:28:49.494512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454282+0000","last_change":"2024-03-18T20:07:18.898269+0000","last_active":"2024-03-18T20:07:20.454282+0000","last_peered":"2024-03-18T20:07:20.454282+0000","last_clean":"2024-03-18T20:07:20.454282+0000","last_became_active":"2024-03-18T20:07:18.897985+0000","last_became_peered":"2024-03-18T20:07:18.897985+0000","last_unstale":"2024-03-18T20:07:20.454282+0000","last_undegraded":"2024-03-18T20:07:20.454282+0000","last_fullsized":"2024-03-18T20:07:20.454282+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:04.550348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617772+0000","last_change":"2024-03-18T20:07:19.344502+0000","last_active":"2024-03-18T20:07:20.617772+0000","last_peered":"2024-03-18T20:07:20.617772+0000","last_clean":"2024-03-18T20:07:20.617772+0000","last_became_active":"2024-03-18T20:07:19.344225+0000","last_became_peered":"2024-03-18T20:07:19.344225+0000","last_unstale":"2024-03-18T20:07:20.617772+0000","last_undegraded":"2024-03-18T20:07:20.617772+0000","last_fullsized":"2024-03-18T20:07:20.617772+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:15:04.952572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617712+0000","last_change":"2024-03-18T20:07:19.355229+0000","last_active":"2024-03-18T20:07:20.617712+0000","last_peered":"2024-03-18T20:07:20.617712+0000","last_clean":"2024-03-18T20:07:20.617712+0000","last_became_active":"2024-03-18T20:07:19.354939+0000","last_became_peered":"2024-03-18T20:07:19.354939+0000","last_unstale":"2024-03-18T20:07:20.617712+0000","last_undegraded":"2024-03-18T20:07:20.617712+0000","last_fullsized":"2024-03-18T20:07:20.617712+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:24.611484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332610+0000","last_change":"2024-03-18T20:07:19.350492+0000","last_active":"2024-03-18T20:07:20.332610+0000","last_peered":"2024-03-18T20:07:20.332610+0000","last_clean":"2024-03-18T20:07:20.332610+0000","last_became_active":"2024-03-18T20:07:19.350224+0000","last_became_peered":"2024-03-18T20:07:19.350224+0000","last_unstale":"2024-03-18T20:07:20.332610+0000","last_undegraded":"2024-03-18T20:07:20.332610+0000","last_fullsized":"2024-03-18T20:07:20.332610+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:14.209896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332460+0000","last_change":"2024-03-18T20:07:18.904697+0000","last_active":"2024-03-18T20:07:20.332460+0000","last_peered":"2024-03-18T20:07:20.332460+0000","last_clean":"2024-03-18T20:07:20.332460+0000","last_became_active":"2024-03-18T20:07:18.904431+0000","last_became_peered":"2024-03-18T20:07:18.904431+0000","last_unstale":"2024-03-18T20:07:20.332460+0000","last_undegraded":"2024-03-18T20:07:20.332460+0000","last_fullsized":"2024-03-18T20:07:20.332460+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:57.606338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454211+0000","last_change":"2024-03-18T20:07:19.337251+0000","last_active":"2024-03-18T20:07:20.454211+0000","last_peered":"2024-03-18T20:07:20.454211+0000","last_clean":"2024-03-18T20:07:20.454211+0000","last_became_active":"2024-03-18T20:07:19.336992+0000","last_became_peered":"2024-03-18T20:07:19.336992+0000","last_unstale":"2024-03-18T20:07:20.454211+0000","last_undegraded":"2024-03-18T20:07:20.454211+0000","last_fullsized":"2024-03-18T20:07:20.454211+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:01:40.097948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:18.883885+0000","last_change":"2024-03-18T20:07:18.883885+0000","last_active":"2024-03-18T20:07:18.883885+0000","last_peered":"2024-03-18T20:07:18.883885+0000","last_clean":"2024-03-18T20:07:18.883885+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:18.883885+0000","last_undegraded":"2024-03-18T20:07:18.883885+0000","last_fullsized":"2024-03-18T20:07:18.883885+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454000+0000","last_change":"2024-03-18T20:07:19.338944+0000","last_active":"2024-03-18T20:07:20.454000+0000","last_peered":"2024-03-18T20:07:20.454000+0000","last_clean":"2024-03-18T20:07:20.454000+0000","last_became_active":"2024-03-18T20:07:19.338578+0000","last_became_peered":"2024-03-18T20:07:19.338578+0000","last_unstale":"2024-03-18T20:07:20.454000+0000","last_undegraded":"2024-03-18T20:07:20.454000+0000","last_fullsized":"2024-03-18T20:07:20.454000+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:06.095092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:07:17.878457+0000","last_change":"2024-03-18T20:07:17.878457+0000","last_active":"2024-03-18T20:07:17.878457+0000","last_peered":"2024-03-18T20:07:17.878457+0000","last_clean":"2024-03-18T20:07:17.878457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:07:17.878457+0000","last_undegraded":"2024-03-18T20:07:17.878457+0000","last_fullsized":"2024-03-18T20:07:17.878457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"184'293","reported_seq":965,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-03-18T20:07:19.333396+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:07:19.333396+0000","last_peered":"2024-03-18T20:07:19.333396+0000","last_clean":"2024-03-18T20:07:19.333396+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:07:19.333396+0000","last_undegraded":"2024-03-18T20:07:19.333396+0000","last_fullsized":"2024-03-18T20:07:19.333396+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":68,"acting":68,"num_store_stats":4},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":672,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":18,"acting":18,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575768,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48032,"kb_used_data":2964,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323808,"statfs":{"total":96636764160,"available":96587579392,"internally_reserved":0,"allocated":3035136,"data_stored":1828899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.57699999999999996,"15min":0.58299999999999996},"min":{"1min":0.33200000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.70199999999999996,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.58199999999999996,"15min":0.60899999999999999},"min":{"1min":0.35199999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.72799999999999998,"5min":0.80100000000000005,"15min":0.81699999999999995},"last":0.48299999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.58599999999999997,"15min":0.59899999999999998},"min":{"1min":0.318,"5min":0.249,"15min":0.249},"max":{"1min":0.74199999999999999,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.59699999999999998,"15min":0.60799999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.75900000000000001,"5min":0.79900000000000004,"15min":0.83899999999999997},"last":0.58799999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.58599999999999997,"15min":0.60099999999999998},"min":{"1min":0.39700000000000002,"5min":0.36499999999999999,"15min":0.33900000000000002},"max":{"1min":0.77500000000000002,"5min":0.872,"15min":0.872},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78800000000000003,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.52400000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.60199999999999998,"15min":0.60799999999999998},"min":{"1min":0.40899999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80200000000000005,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.435},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.624,"15min":0.63},"min":{"1min":0.42399999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.77800000000000002,"5min":0.84699999999999998,"15min":0.89000000000000001},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Mar 18 20:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47199999999999998},"min":{"1min":0.34200000000000003,"5min":0.32500000000000001,"15min":0.315},"max":{"1min":0.73199999999999998,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.499,"15min":0.495},"min":{"1min":0.35599999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.84699999999999998},"last":0.59999999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51200000000000001,"15min":0.50700000000000001},"min":{"1min":0.35899999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73299999999999998,"5min":1.127,"15min":1.127},"last":0.623},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55100000000000005,"15min":0.54500000000000004},"min":{"1min":0.45000000000000001,"5min":0.35299999999999998,"15min":0.32900000000000001},"max":{"1min":0.80800000000000005,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.54700000000000004}]},{"osd":6,"last update":"Mon Mar 18 20:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56899999999999995,"15min":0.56100000000000005},"min":{"1min":0.42999999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.76500000000000001,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56699999999999995,"15min":0.55600000000000005},"min":{"1min":0.45500000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.91300000000000003},"last":0.55900000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575768,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41692,"kb_used_data":2384,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330148,"statfs":{"total":96636764160,"available":96594071552,"internally_reserved":0,"allocated":2441216,"data_stored":1238531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.61299999999999999,"15min":0.59599999999999997},"min":{"1min":0.40300000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73899999999999999,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.314},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.60499999999999998,"15min":0.59999999999999998},"min":{"1min":0.371,"5min":0.36099999999999999,"15min":0.30299999999999999},"max":{"1min":0.70499999999999996,"5min":1.054,"15min":1.054},"last":0.36099999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60799999999999998,"15min":0.59599999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.34000000000000002},"max":{"1min":0.76700000000000002,"5min":0.82799999999999996,"15min":0.84599999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.61399999999999999,"15min":0.59999999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.28299999999999997},"max":{"1min":0.72999999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.47899999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.64100000000000001,"15min":0.625},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.40600000000000003},"max":{"1min":0.74299999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60399999999999998,"15min":0.59599999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.71799999999999997,"5min":0.79800000000000004,"15min":0.89600000000000002},"last":0.52200000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.64200000000000002,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73399999999999999,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60799999999999998,"15min":0.60099999999999998},"min":{"1min":0.42399999999999999,"5min":0.41599999999999998,"15min":0.39500000000000002},"max":{"1min":0.72099999999999997,"5min":1.448,"15min":1.448},"last":0.55900000000000005}]},{"osd":4,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.47499999999999998},"min":{"1min":0.318,"5min":0.316,"15min":0.316},"max":{"1min":0.69299999999999995,"5min":0.78000000000000003,"15min":0.82999999999999996},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.505,"15min":0.50700000000000001},"min":{"1min":0.372,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.65900000000000003,"5min":0.84399999999999997,"15min":0.86499999999999999},"last":0.60999999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.495,"15min":0.48999999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.63800000000000001,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.505},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54500000000000004,"15min":0.53300000000000003},"min":{"1min":0.43099999999999999,"5min":0.38200000000000001,"15min":0.33900000000000002},"max":{"1min":0.71199999999999997,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.622}]},{"osd":7,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.39600000000000002,"5min":0.378,"15min":0.34300000000000003},"max":{"1min":0.71899999999999997,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56200000000000006,"15min":0.54700000000000004},"min":{"1min":0.45000000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67900000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.59999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575768,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41704,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330136,"statfs":{"total":96636764160,"available":96594059264,"internally_reserved":0,"allocated":2453504,"data_stored":1240095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.38800000000000001,"15min":0.39700000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.42599999999999999,"5min":0.58499999999999996,"15min":0.63600000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.42599999999999999,"15min":0.44},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.504,"5min":0.71099999999999997,"15min":0.73599999999999999},"last":0.35299999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.42199999999999999,"15min":0.42399999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.25600000000000001},"max":{"1min":0.57499999999999996,"5min":0.66500000000000004,"15min":0.70299999999999996},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.46300000000000002,"15min":0.46800000000000003},"min":{"1min":0.313,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59699999999999998,"5min":0.77500000000000002,"15min":0.79300000000000004},"last":0.41399999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.48499999999999999,"15min":0.49199999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.49299999999999999,"5min":0.71999999999999997,"15min":0.754},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.439,"15min":0.45100000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.55400000000000005,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.40000000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.39400000000000002},"max":{"1min":0.78700000000000003,"5min":1.012,"15min":1.0509999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.48399999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.77000000000000002,"5min":1.081,"15min":1.081},"last":0.53000000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.65400000000000003,"15min":0.64800000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.69799999999999995,"5min":1.149,"15min":1.149},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.69499999999999995,"15min":0.69299999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.45600000000000002},"max":{"1min":0.79000000000000004,"5min":1.069,"15min":1.069},"last":0.57999999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.65700000000000003,"15min":0.65400000000000003},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.44700000000000001},"max":{"1min":0.81200000000000006,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.502},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.70799999999999996,"15min":0.69899999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.77700000000000002,"5min":1.101,"15min":1.101},"last":0.55400000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.68700000000000006,"15min":0.69299999999999995},"min":{"1min":0.48199999999999998,"5min":0.46600000000000003,"15min":0.44400000000000001},"max":{"1min":0.82999999999999996,"5min":1.141,"15min":1.141},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.69799999999999995,"15min":0.69499999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46000000000000002},"max":{"1min":0.76400000000000001,"5min":0.995,"15min":0.995},"last":0.51700000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575768,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42212,"kb_used_data":2904,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329628,"statfs":{"total":96636764160,"available":96593539072,"internally_reserved":0,"allocated":2973696,"data_stored":1796163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.38,"15min":0.39300000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.40999999999999998,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.40500000000000003,"15min":0.42299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.438,"5min":0.66800000000000004,"15min":0.67700000000000005},"last":0.38600000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.42899999999999999,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.53600000000000003,"5min":0.63200000000000001,"15min":0.67300000000000004},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.44400000000000001,"15min":0.45500000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.55700000000000005,"5min":0.65200000000000002,"15min":0.751},"last":0.436}]},{"osd":3,"last update":"Mon Mar 18 20:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.42699999999999999,"15min":0.43099999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.27900000000000003},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.371},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.49399999999999999,"15min":0.501},"min":{"1min":0.308,"5min":0.308,"15min":0.307},"max":{"1min":0.59799999999999998,"5min":0.73399999999999999,"15min":0.75600000000000001},"last":0.40699999999999997}]},{"osd":4,"last update":"Mon Mar 18 20:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.64900000000000002,"15min":0.65800000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.74099999999999999,"5min":1.0109999999999999,"15min":1.044},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.67200000000000004,"15min":0.67100000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.84899999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.52000000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.67200000000000004,"15min":0.67800000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82199999999999995,"5min":1.0229999999999999,"15min":1.0629999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.68500000000000005,"15min":0.68200000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.42299999999999999},"max":{"1min":0.86499999999999999,"5min":1.042,"15min":1.095},"last":0.55500000000000005}]},{"osd":6,"last update":"Mon Mar 18 20:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.67500000000000004,"15min":0.67100000000000004},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.89200000000000002,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.69899999999999995,"15min":0.70399999999999996},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.80600000000000005,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.61899999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.69699999999999995,"15min":0.69999999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.81699999999999995,"5min":1.0600000000000001,"15min":1.075},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.70399999999999996,"15min":0.70799999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.98199999999999998,"5min":1.02,"15min":1.02},"last":0.54500000000000004}]}]},{"osd":2,"up_from":13,"seq":55834575768,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":47412,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324428,"statfs":{"total":96636764160,"available":96588214272,"internally_reserved":0,"allocated":2392064,"data_stored":1211804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.38900000000000001,"15min":0.38700000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.59999999999999998,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.46999999999999997,"15min":0.47399999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.218},"max":{"1min":0.63900000000000001,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.25600000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.47699999999999998,"15min":0.47899999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.27900000000000003},"max":{"1min":0.58399999999999996,"5min":0.749,"15min":0.77800000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.41099999999999998,"15min":0.41399999999999998},"min":{"1min":0.254,"5min":0.25,"15min":0.25},"max":{"1min":0.48699999999999999,"5min":0.58299999999999996,"15min":0.70799999999999996},"last":0.45400000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.41899999999999998,"15min":0.42199999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27300000000000002},"max":{"1min":0.52700000000000002,"5min":0.629,"15min":0.69399999999999995},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.52400000000000002,"15min":0.52500000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33700000000000002},"max":{"1min":0.66700000000000004,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.44}]},{"osd":4,"last update":"Mon Mar 18 20:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66600000000000004,"15min":0.65900000000000003},"min":{"1min":0.42599999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.65000000000000002,"15min":0.65200000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":1.0529999999999999},"last":0.75600000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.67100000000000004,"15min":0.67400000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.40799999999999997},"max":{"1min":0.88,"5min":0.98999999999999999,"15min":1.0369999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.67900000000000005,"15min":0.66700000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.84199999999999997}]},{"osd":6,"last update":"Mon Mar 18 20:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67600000000000005,"15min":0.67700000000000005},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.39800000000000002},"max":{"1min":0.998,"5min":1.004,"15min":1.073},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.72399999999999998,"15min":0.72699999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.45100000000000001},"max":{"1min":0.94699999999999995,"5min":1.036,"15min":1.0660000000000001},"last":0.68300000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.69099999999999995,"15min":0.68999999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.69599999999999995,"15min":0.69399999999999995},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.97099999999999997,"5min":1.002,"15min":1.0109999999999999},"last":0.78900000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575768,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":47396,"kb_used_data":2324,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324444,"statfs":{"total":96636764160,"available":96588230656,"internally_reserved":0,"allocated":2379776,"data_stored":1205795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36799999999999999,"15min":0.38800000000000001},"min":{"1min":0.27100000000000002,"5min":0.248,"15min":0.23300000000000001},"max":{"1min":0.45400000000000001,"5min":0.66700000000000004,"15min":0.70599999999999996},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40000000000000002,"15min":0.41299999999999998},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.52200000000000002,"5min":0.61399999999999999,"15min":0.78200000000000003},"last":0.38300000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39600000000000002,"15min":0.41299999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.24199999999999999},"max":{"1min":0.58999999999999997,"5min":0.66600000000000004,"15min":0.66800000000000004},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.45200000000000001,"15min":0.46000000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.88900000000000001},"last":0.42999999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.44900000000000001,"15min":0.46500000000000002},"min":{"1min":0.31900000000000001,"5min":0.309,"15min":0.29399999999999998},"max":{"1min":0.57999999999999996,"5min":0.64100000000000001,"15min":0.71299999999999997},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48499999999999999,"15min":0.498},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.60599999999999998,"5min":0.72099999999999997,"15min":0.76200000000000001},"last":0.39900000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63900000000000001,"15min":0.64500000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.38800000000000001},"max":{"1min":0.84099999999999997,"5min":0.95599999999999996,"15min":1.0409999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.64500000000000002,"15min":0.65800000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41699999999999998},"max":{"1min":0.88800000000000001,"5min":1.214,"15min":1.214},"last":0.76400000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.67000000000000004,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.85899999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70799999999999996,"15min":0.71399999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.40500000000000003},"max":{"1min":0.90400000000000003,"5min":1.0349999999999999,"15min":1.0660000000000001},"last":0.71899999999999997}]},{"osd":6,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.433},"max":{"1min":0.88900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.70099999999999996,"15min":0.71399999999999997},"min":{"1min":0.48999999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.95199999999999996,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.57499999999999996}]},{"osd":7,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.69199999999999995,"15min":0.69899999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":1.02},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.68400000000000005,"15min":0.69099999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92200000000000004,"5min":1.0049999999999999,"15min":1.012},"last":0.59999999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575769,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41716,"kb_used_data":2400,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330124,"statfs":{"total":96636764160,"available":96594046976,"internally_reserved":0,"allocated":2457600,"data_stored":1239295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.57599999999999996,"15min":0.59899999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.309},"max":{"1min":0.67400000000000004,"5min":0.78700000000000003,"15min":1.212},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.78100000000000003,"5min":0.85399999999999998,"15min":1.079},"last":0.47399999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.60999999999999999,"15min":0.622},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73599999999999999,"5min":0.83399999999999996,"15min":1.143},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.58399999999999996,"15min":0.59399999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.34200000000000003},"max":{"1min":0.76900000000000002,"5min":0.78400000000000003,"15min":1.0669999999999999},"last":0.42399999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.58399999999999996,"15min":0.60299999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35199999999999998},"max":{"1min":0.79600000000000004,"5min":0.80900000000000005,"15min":1.246},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.59999999999999998,"15min":0.61799999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.68400000000000005,"5min":0.78200000000000003,"15min":1.1140000000000001},"last":0.70599999999999996}]},{"osd":3,"last update":"Mon Mar 18 20:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.59499999999999997,"15min":0.59999999999999998},"min":{"1min":0.41299999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73199999999999998,"5min":0.80500000000000005,"15min":1.034},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.60799999999999998,"15min":0.623},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.751,"5min":0.82799999999999996,"15min":1.1559999999999999},"last":0.50900000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67700000000000005,"5min":0.68600000000000005,"15min":0.94299999999999995},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.505,"15min":0.498},"min":{"1min":0.317,"5min":0.308,"15min":0.30599999999999999},"max":{"1min":0.66400000000000003,"5min":0.81699999999999995,"15min":0.92400000000000004},"last":0.64400000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48899999999999999,"15min":0.48799999999999999},"min":{"1min":0.33000000000000002,"5min":0.32500000000000001,"15min":0.308},"max":{"1min":0.65000000000000002,"5min":0.70999999999999996,"15min":0.95999999999999996},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.52700000000000002,"15min":0.52800000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.63400000000000001,"5min":0.80500000000000005,"15min":0.99299999999999999},"last":0.74099999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":0.60799999999999998,"5min":0.76100000000000001,"15min":0.97699999999999998},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.56899999999999995,"15min":0.55700000000000005},"min":{"1min":0.375,"5min":0.36499999999999999,"15min":0.34499999999999997},"max":{"1min":0.68799999999999994,"5min":0.91500000000000004,"15min":0.96599999999999997},"last":0.72699999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575767,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":63780,"kb_used_data":2324,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94308060,"statfs":{"total":96636764160,"available":96571453440,"internally_reserved":0,"allocated":2379776,"data_stored":1205795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":62912967},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59699999999999998,"15min":0.59799999999999998},"min":{"1min":0.39300000000000002,"5min":0.35799999999999998,"15min":0.34300000000000003},"max":{"1min":0.85599999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.59299999999999997,"15min":0.60299999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.72299999999999998,"5min":0.80000000000000004,"15min":0.81799999999999995},"last":0.501}]},{"osd":1,"last update":"Mon Mar 18 20:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39000000000000001},"max":{"1min":0.749,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.60199999999999998,"15min":0.61099999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.74299999999999999,"5min":0.84199999999999997,"15min":0.95199999999999996},"last":0.72399999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.34899999999999998},"max":{"1min":0.76500000000000001,"5min":0.77800000000000002,"15min":0.83099999999999996},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.59099999999999997,"15min":0.60299999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.36099999999999999},"max":{"1min":0.70699999999999996,"5min":0.82899999999999996,"15min":0.88200000000000001},"last":0.53900000000000003}]},{"osd":3,"last update":"Mon Mar 18 20:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.59299999999999997,"15min":0.60899999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.374},"max":{"1min":0.81000000000000005,"5min":0.88200000000000001,"15min":0.91900000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.63700000000000001,"15min":0.63400000000000001},"min":{"1min":0.442,"5min":0.375,"15min":0.375},"max":{"1min":0.82099999999999995,"5min":0.874,"15min":0.874},"last":0.66100000000000003}]},{"osd":4,"last update":"Mon Mar 18 20:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.48499999999999999,"15min":0.48099999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67000000000000004,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.499,"5min":0.52600000000000002,"15min":0.51500000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71899999999999997,"5min":0.79700000000000004,"15min":0.879},"last":0.68400000000000005}]},{"osd":6,"last update":"Mon Mar 18 20:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52000000000000002,"15min":0.505},"min":{"1min":0.311,"5min":0.311,"15min":0.29699999999999999},"max":{"1min":0.71199999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.53800000000000003,"15min":0.53600000000000003},"min":{"1min":0.38400000000000001,"5min":0.36299999999999999,"15min":0.33800000000000002},"max":{"1min":0.69799999999999995,"5min":0.76600000000000001,"15min":0.81699999999999995},"last":0.63800000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56200000000000006,"15min":0.55500000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32000000000000001},"max":{"1min":0.754,"5min":0.83399999999999996,"15min":0.84499999999999997},"last":0.751},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.44400000000000001,"5min":0.38300000000000001,"15min":0.34300000000000003},"max":{"1min":0.83699999999999997,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.73599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:07:23.496 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:07:23.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.651+0000 7fd5ad62d700 1 Processor -- start 2024-03-18T20:07:23.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.651+0000 7fd5ad62d700 1 -- start start 2024-03-18T20:07:23.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.652+0000 7fd5ad62d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd5a80d1ea0 0x7fd5a8168560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.652+0000 7fd5ad62d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a80cddc0 con 0x7fd5a8160940 2024-03-18T20:07:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.652+0000 7fd5ad62d700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a80cdff0 con 0x7fd5a80cf090 2024-03-18T20:07:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.652+0000 7fd5ad62d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a80ce2d0 con 0x7fd5a80d1ea0 2024-03-18T20:07:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.652+0000 7fd5a6ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd5a8160940 0x7fd5a80d0190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43398/0 (socket says 172.21.15.1:43398) 2024-03-18T20:07:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.652+0000 7fd5a6ffd700 1 -- 172.21.15.1:0/316311386 learned_addr learned my addr 172.21.15.1:0/316311386 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.652+0000 7fd5a77fe700 1 --2- 172.21.15.1:0/316311386 >> v2:172.21.15.165:3300/0 conn(0x7fd5a80d1ea0 0x7fd5a8168560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.652+0000 7fd5a5ffb700 1 -- 172.21.15.1:0/316311386 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538602166 0 0) 0x7fd5a80cddc0 con 0x7fd5a8160940 2024-03-18T20:07:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.652+0000 7fd5a5ffb700 1 -- 172.21.15.1:0/316311386 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5940035b0 con 0x7fd5a8160940 2024-03-18T20:07:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a5ffb700 1 -- 172.21.15.1:0/316311386 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 220098995 0 0) 0x7fd5940035b0 con 0x7fd5a8160940 2024-03-18T20:07:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a5ffb700 1 -- 172.21.15.1:0/316311386 >> v2:172.21.15.165:3300/0 conn(0x7fd5a80d1ea0 msgr2=0x7fd5a8168560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a5ffb700 1 --2- 172.21.15.1:0/316311386 >> v2:172.21.15.165:3300/0 conn(0x7fd5a80d1ea0 0x7fd5a8168560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a5ffb700 1 -- 172.21.15.1:0/316311386 >> v1:172.21.15.165:6789/0 conn(0x7fd5a80cf090 legacy=0x7fd5a80d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:23.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a5ffb700 1 -- 172.21.15.1:0/316311386 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a80ce5b0 con 0x7fd5a8160940 2024-03-18T20:07:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a5ffb700 1 -- 172.21.15.1:0/316311386 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fd5a00019f0 con 0x7fd5a8160940 2024-03-18T20:07:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a5ffb700 1 -- 172.21.15.1:0/316311386 >> v1:172.21.15.1:6789/0 conn(0x7fd5a8160940 legacy=0x7fd5a80d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a5ffb700 1 --2- 172.21.15.1:0/316311386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd594003a00 0x7fd594003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a5ffb700 1 --2- 172.21.15.1:0/316311386 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd594004850 0x7fd594004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a5ffb700 1 --2- 172.21.15.1:0/316311386 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd594005210 0x7fd59400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a5ffb700 1 -- 172.21.15.1:0/316311386 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a80ce2d0 con 0x7fd594003a00 2024-03-18T20:07:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a5ffb700 1 -- 172.21.15.1:0/316311386 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59400c0b0 con 0x7fd594004850 2024-03-18T20:07:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a5ffb700 1 -- 172.21.15.1:0/316311386 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59400c270 con 0x7fd594005210 2024-03-18T20:07:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a6ffd700 1 --2- 172.21.15.1:0/316311386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd594003a00 0x7fd594003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a67fc700 1 --2- 172.21.15.1:0/316311386 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd594004850 0x7fd594004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a77fe700 1 --2- 172.21.15.1:0/316311386 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd594005210 0x7fd59400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a6ffd700 1 -- 172.21.15.1:0/316311386 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd594005210 msgr2=0x7fd59400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a6ffd700 1 --2- 172.21.15.1:0/316311386 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd594005210 0x7fd59400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a6ffd700 1 -- 172.21.15.1:0/316311386 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd594004850 msgr2=0x7fd594004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a6ffd700 1 --2- 172.21.15.1:0/316311386 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd594004850 0x7fd594004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a77fe700 1 -- 172.21.15.1:0/316311386 reap_dead start 2024-03-18T20:07:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a6ffd700 1 -- 172.21.15.1:0/316311386 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5940035b0 con 0x7fd594003a00 2024-03-18T20:07:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a6ffd700 1 --2- 172.21.15.1:0/316311386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd594003a00 0x7fd594003e20 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000a830 tx=0x7fd594005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b3cf4aeb0ea81c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a5ffb700 1 -- 172.21.15.1:0/316311386 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a000a2f0 con 0x7fd594003a00 2024-03-18T20:07:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a5ffb700 1 -- 172.21.15.1:0/316311386 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a0012600 con 0x7fd594003a00 2024-03-18T20:07:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.653+0000 7fd5a5ffb700 1 -- 172.21.15.1:0/316311386 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a00129b0 con 0x7fd594003a00 2024-03-18T20:07:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.654+0000 7fd5ad62d700 1 -- 172.21.15.1:0/316311386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd594003a00 msgr2=0x7fd594003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.654+0000 7fd5ad62d700 1 --2- 172.21.15.1:0/316311386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd594003a00 0x7fd594003e20 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000a830 tx=0x7fd594005880 comp rx=0 tx=0).stop 2024-03-18T20:07:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.654+0000 7fd5ad62d700 1 -- 172.21.15.1:0/316311386 shutdown_connections 2024-03-18T20:07:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.654+0000 7fd5ad62d700 1 --2- 172.21.15.1:0/316311386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd594003a00 0x7fd594003e20 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.654+0000 7fd5ad62d700 1 -- 172.21.15.1:0/316311386 >> 172.21.15.1:0/316311386 conn(0x7fd5a809c520 msgr2=0x7fd5a80d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:23.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.654+0000 7fd5ad62d700 1 -- 172.21.15.1:0/316311386 shutdown_connections 2024-03-18T20:07:23.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.654+0000 7fd5ad62d700 1 -- 172.21.15.1:0/316311386 wait complete. 2024-03-18T20:07:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.655+0000 7fd5ad62d700 1 Processor -- start 2024-03-18T20:07:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.655+0000 7fd5ad62d700 1 -- start start 2024-03-18T20:07:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.656+0000 7fd5ad62d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd594003a00 0x7fd5a80cc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.656+0000 7fd5ad62d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5a80b5dd0 0x7fd5a80b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.656+0000 7fd5ad62d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5a80afd80 0x7fd5a80b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.656+0000 7fd5ad62d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a80b6790 con 0x7fd5a80b5dd0 2024-03-18T20:07:23.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.656+0000 7fd5ad62d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a80cc6a0 con 0x7fd5a80afd80 2024-03-18T20:07:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.656+0000 7fd5ad62d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a80b0770 con 0x7fd594003a00 2024-03-18T20:07:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.656+0000 7fd5a67fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5a80b5dd0 0x7fd5a80b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.656+0000 7fd5a67fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5a80b5dd0 0x7fd5a80b6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37780/0 (socket says 172.21.15.1:37780) 2024-03-18T20:07:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.656+0000 7fd5a67fc700 1 -- 172.21.15.1:0/2622672130 learned_addr learned my addr 172.21.15.1:0/2622672130 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.656+0000 7fd5a6ffd700 1 --2- 172.21.15.1:0/2622672130 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd594003a00 0x7fd5a80cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.657+0000 7fd5a77fe700 1 --2- 172.21.15.1:0/2622672130 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5a80afd80 0x7fd5a80b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.657+0000 7fd5a67fc700 1 -- 172.21.15.1:0/2622672130 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd594003a00 msgr2=0x7fd5a80cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.657+0000 7fd5a67fc700 1 --2- 172.21.15.1:0/2622672130 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd594003a00 0x7fd5a80cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.657+0000 7fd5a67fc700 1 -- 172.21.15.1:0/2622672130 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5a80afd80 msgr2=0x7fd5a80b0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.657+0000 7fd5a67fc700 1 --2- 172.21.15.1:0/2622672130 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5a80afd80 0x7fd5a80b0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.657+0000 7fd5a67fc700 1 -- 172.21.15.1:0/2622672130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a80b0a00 con 0x7fd5a80b5dd0 2024-03-18T20:07:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.657+0000 7fd5a67fc700 1 --2- 172.21.15.1:0/2622672130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5a80b5dd0 0x7fd5a80b6250 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7fd59000f610 tx=0x7fd59000f9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.657+0000 7fd59f7fe700 1 -- 172.21.15.1:0/2622672130 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5900052e0 con 0x7fd5a80b5dd0 2024-03-18T20:07:23.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.657+0000 7fd59f7fe700 1 -- 172.21.15.1:0/2622672130 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd590005440 con 0x7fd5a80b5dd0 2024-03-18T20:07:23.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.657+0000 7fd5ad62d700 1 -- 172.21.15.1:0/2622672130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a80b0ce0 con 0x7fd5a80b5dd0 2024-03-18T20:07:23.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.657+0000 7fd59f7fe700 1 -- 172.21.15.1:0/2622672130 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5900038a0 con 0x7fd5a80b5dd0 2024-03-18T20:07:23.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.657+0000 7fd5ad62d700 1 -- 172.21.15.1:0/2622672130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a80b12a0 con 0x7fd5a80b5dd0 2024-03-18T20:07:23.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.658+0000 7fd5ad62d700 1 -- 172.21.15.1:0/2622672130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a8073350 con 0x7fd5a80b5dd0 2024-03-18T20:07:23.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.658+0000 7fd59f7fe700 1 -- 172.21.15.1:0/2622672130 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd590003a00 con 0x7fd5a80b5dd0 2024-03-18T20:07:23.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.659+0000 7fd59f7fe700 1 --2- 172.21.15.1:0/2622672130 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd58806adb0 0x7fd58806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:23.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.659+0000 7fd59f7fe700 1 -- 172.21.15.1:0/2622672130 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd590089330 con 0x7fd5a80b5dd0 2024-03-18T20:07:23.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.659+0000 7fd5a6ffd700 1 --2- 172.21.15.1:0/2622672130 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd58806adb0 0x7fd58806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:23.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.659+0000 7fd5a6ffd700 1 --2- 172.21.15.1:0/2622672130 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd58806adb0 0x7fd58806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000afd0 tx=0x7fd5a0007000 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:23.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.663+0000 7fd59f7fe700 1 -- 172.21.15.1:0/2622672130 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd590054300 con 0x7fd5a80b5dd0 2024-03-18T20:07:23.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.865+0000 7fd5ad62d700 1 -- 172.21.15.1:0/2622672130 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd5a80ceb70 con 0x7fd58806adb0 2024-03-18T20:07:23.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.874+0000 7fd59f7fe700 1 -- 172.21.15.1:0/2622672130 <== mgr.10200 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290233 (secure 0 0 0) 0x7fd5a000a210 con 0x7fd58806adb0 2024-03-18T20:07:23.882 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:07:23.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.877+0000 7fd5ad62d700 1 -- 172.21.15.1:0/2622672130 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd58806adb0 msgr2=0x7fd58806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:23.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.877+0000 7fd5ad62d700 1 --2- 172.21.15.1:0/2622672130 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd58806adb0 0x7fd58806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000afd0 tx=0x7fd5a0007000 comp rx=0 tx=0).stop 2024-03-18T20:07:23.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.877+0000 7fd5ad62d700 1 -- 172.21.15.1:0/2622672130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5a80b5dd0 msgr2=0x7fd5a80b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:23.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.877+0000 7fd5ad62d700 1 --2- 172.21.15.1:0/2622672130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5a80b5dd0 0x7fd5a80b6250 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7fd59000f610 tx=0x7fd59000f9d0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.877+0000 7fd5ad62d700 1 -- 172.21.15.1:0/2622672130 shutdown_connections 2024-03-18T20:07:23.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.877+0000 7fd5ad62d700 1 --2- 172.21.15.1:0/2622672130 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fd58806adb0 0x7fd58806d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.877+0000 7fd5ad62d700 1 --2- 172.21.15.1:0/2622672130 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd594003a00 0x7fd5a80cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.877+0000 7fd5ad62d700 1 --2- 172.21.15.1:0/2622672130 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5a80afd80 0x7fd5a80b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.877+0000 7fd5ad62d700 1 --2- 172.21.15.1:0/2622672130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5a80b5dd0 0x7fd5a80b6250 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:23.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.877+0000 7fd5ad62d700 1 -- 172.21.15.1:0/2622672130 >> 172.21.15.1:0/2622672130 conn(0x7fd5a809c520 msgr2=0x7fd5a80b2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:23.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.877+0000 7fd5ad62d700 1 -- 172.21.15.1:0/2622672130 shutdown_connections 2024-03-18T20:07:23.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:23.877+0000 7fd5ad62d700 1 -- 172.21.15.1:0/2622672130 wait complete. 2024-03-18T20:07:23.885 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:07:23.894 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-18T20:07:23.632476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":523,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":331,"ondisk_log_size":331,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":374192,"kb_used_data":20296,"kb_used_omap":12,"kb_used_meta":353779,"kb_avail":754600528,"statfs":{"total":773094113280,"available":772710940672,"internally_reserved":0,"allocated":20783104,"data_stored":11093708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":362270265},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.748177"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332710+0000","last_change":"2024-03-18T20:07:19.345101+0000","last_active":"2024-03-18T20:07:20.332710+0000","last_peered":"2024-03-18T20:07:20.332710+0000","last_clean":"2024-03-18T20:07:20.332710+0000","last_became_active":"2024-03-18T20:07:19.344798+0000","last_became_peered":"2024-03-18T20:07:19.344798+0000","last_unstale":"2024-03-18T20:07:20.332710+0000","last_undegraded":"2024-03-18T20:07:20.332710+0000","last_fullsized":"2024-03-18T20:07:20.332710+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:29:33.340151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454245+0000","last_change":"2024-03-18T20:07:19.345726+0000","last_active":"2024-03-18T20:07:20.454245+0000","last_peered":"2024-03-18T20:07:20.454245+0000","last_clean":"2024-03-18T20:07:20.454245+0000","last_became_active":"2024-03-18T20:07:19.345456+0000","last_became_peered":"2024-03-18T20:07:19.345456+0000","last_unstale":"2024-03-18T20:07:20.454245+0000","last_undegraded":"2024-03-18T20:07:20.454245+0000","last_fullsized":"2024-03-18T20:07:20.454245+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:00:37.910486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332715+0000","last_change":"2024-03-18T20:07:19.345435+0000","last_active":"2024-03-18T20:07:20.332715+0000","last_peered":"2024-03-18T20:07:20.332715+0000","last_clean":"2024-03-18T20:07:20.332715+0000","last_became_active":"2024-03-18T20:07:19.345015+0000","last_became_peered":"2024-03-18T20:07:19.345015+0000","last_unstale":"2024-03-18T20:07:20.332715+0000","last_undegraded":"2024-03-18T20:07:20.332715+0000","last_fullsized":"2024-03-18T20:07:20.332715+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:00:43.939604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332531+0000","last_change":"2024-03-18T20:07:19.353667+0000","last_active":"2024-03-18T20:07:20.332531+0000","last_peered":"2024-03-18T20:07:20.332531+0000","last_clean":"2024-03-18T20:07:20.332531+0000","last_became_active":"2024-03-18T20:07:19.353304+0000","last_became_peered":"2024-03-18T20:07:19.353304+0000","last_unstale":"2024-03-18T20:07:20.332531+0000","last_undegraded":"2024-03-18T20:07:20.332531+0000","last_fullsized":"2024-03-18T20:07:20.332531+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:41.216249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618472+0000","last_change":"2024-03-18T20:07:19.342896+0000","last_active":"2024-03-18T20:07:20.618472+0000","last_peered":"2024-03-18T20:07:20.618472+0000","last_clean":"2024-03-18T20:07:20.618472+0000","last_became_active":"2024-03-18T20:07:19.342664+0000","last_became_peered":"2024-03-18T20:07:19.342664+0000","last_unstale":"2024-03-18T20:07:20.618472+0000","last_undegraded":"2024-03-18T20:07:20.618472+0000","last_fullsized":"2024-03-18T20:07:20.618472+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:34.547722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454036+0000","last_change":"2024-03-18T20:07:19.341565+0000","last_active":"2024-03-18T20:07:20.454036+0000","last_peered":"2024-03-18T20:07:20.454036+0000","last_clean":"2024-03-18T20:07:20.454036+0000","last_became_active":"2024-03-18T20:07:19.341293+0000","last_became_peered":"2024-03-18T20:07:19.341293+0000","last_unstale":"2024-03-18T20:07:20.454036+0000","last_undegraded":"2024-03-18T20:07:20.454036+0000","last_fullsized":"2024-03-18T20:07:20.454036+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:31:39.212677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617671+0000","last_change":"2024-03-18T20:07:19.344583+0000","last_active":"2024-03-18T20:07:20.617671+0000","last_peered":"2024-03-18T20:07:20.617671+0000","last_clean":"2024-03-18T20:07:20.617671+0000","last_became_active":"2024-03-18T20:07:19.344312+0000","last_became_peered":"2024-03-18T20:07:19.344312+0000","last_unstale":"2024-03-18T20:07:20.617671+0000","last_undegraded":"2024-03-18T20:07:20.617671+0000","last_fullsized":"2024-03-18T20:07:20.617671+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:56:21.591342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.616710+0000","last_change":"2024-03-18T20:07:19.348770+0000","last_active":"2024-03-18T20:07:20.616710+0000","last_peered":"2024-03-18T20:07:20.616710+0000","last_clean":"2024-03-18T20:07:20.616710+0000","last_became_active":"2024-03-18T20:07:19.345749+0000","last_became_peered":"2024-03-18T20:07:19.345749+0000","last_unstale":"2024-03-18T20:07:20.616710+0000","last_undegraded":"2024-03-18T20:07:20.616710+0000","last_fullsized":"2024-03-18T20:07:20.616710+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:43:34.547722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618596+0000","last_change":"2024-03-18T20:07:19.335107+0000","last_active":"2024-03-18T20:07:20.618596+0000","last_peered":"2024-03-18T20:07:20.618596+0000","last_clean":"2024-03-18T20:07:20.618596+0000","last_became_active":"2024-03-18T20:07:19.334865+0000","last_became_peered":"2024-03-18T20:07:19.334865+0000","last_unstale":"2024-03-18T20:07:20.618596+0000","last_undegraded":"2024-03-18T20:07:20.618596+0000","last_fullsized":"2024-03-18T20:07:20.618596+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:50:44.623779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618270+0000","last_change":"2024-03-18T20:07:19.341279+0000","last_active":"2024-03-18T20:07:20.618270+0000","last_peered":"2024-03-18T20:07:20.618270+0000","last_clean":"2024-03-18T20:07:20.618270+0000","last_became_active":"2024-03-18T20:07:19.341020+0000","last_became_peered":"2024-03-18T20:07:19.341020+0000","last_unstale":"2024-03-18T20:07:20.618270+0000","last_undegraded":"2024-03-18T20:07:20.618270+0000","last_fullsized":"2024-03-18T20:07:20.618270+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:24.003624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617332+0000","last_change":"2024-03-18T20:07:19.344078+0000","last_active":"2024-03-18T20:07:20.617332+0000","last_peered":"2024-03-18T20:07:20.617332+0000","last_clean":"2024-03-18T20:07:20.617332+0000","last_became_active":"2024-03-18T20:07:19.343837+0000","last_became_peered":"2024-03-18T20:07:19.343837+0000","last_unstale":"2024-03-18T20:07:20.617332+0000","last_undegraded":"2024-03-18T20:07:20.617332+0000","last_fullsized":"2024-03-18T20:07:20.617332+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:18:43.940601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454130+0000","last_change":"2024-03-18T20:07:19.341875+0000","last_active":"2024-03-18T20:07:20.454130+0000","last_peered":"2024-03-18T20:07:20.454130+0000","last_clean":"2024-03-18T20:07:20.454130+0000","last_became_active":"2024-03-18T20:07:19.341594+0000","last_became_peered":"2024-03-18T20:07:19.341594+0000","last_unstale":"2024-03-18T20:07:20.454130+0000","last_undegraded":"2024-03-18T20:07:20.454130+0000","last_fullsized":"2024-03-18T20:07:20.454130+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:18:43.940601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617224+0000","last_change":"2024-03-18T20:07:19.352452+0000","last_active":"2024-03-18T20:07:20.617224+0000","last_peered":"2024-03-18T20:07:20.617224+0000","last_clean":"2024-03-18T20:07:20.617224+0000","last_became_active":"2024-03-18T20:07:19.350437+0000","last_became_peered":"2024-03-18T20:07:19.350437+0000","last_unstale":"2024-03-18T20:07:20.617224+0000","last_undegraded":"2024-03-18T20:07:20.617224+0000","last_fullsized":"2024-03-18T20:07:20.617224+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:06:51.389693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618839+0000","last_change":"2024-03-18T20:07:19.355171+0000","last_active":"2024-03-18T20:07:20.618839+0000","last_peered":"2024-03-18T20:07:20.618839+0000","last_clean":"2024-03-18T20:07:20.618839+0000","last_became_active":"2024-03-18T20:07:19.354935+0000","last_became_peered":"2024-03-18T20:07:19.354935+0000","last_unstale":"2024-03-18T20:07:20.618839+0000","last_undegraded":"2024-03-18T20:07:20.618839+0000","last_fullsized":"2024-03-18T20:07:20.618839+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:36:53.574323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617000+0000","last_change":"2024-03-18T20:07:19.347079+0000","last_active":"2024-03-18T20:07:20.617000+0000","last_peered":"2024-03-18T20:07:20.617000+0000","last_clean":"2024-03-18T20:07:20.617000+0000","last_became_active":"2024-03-18T20:07:19.346434+0000","last_became_peered":"2024-03-18T20:07:19.346434+0000","last_unstale":"2024-03-18T20:07:20.617000+0000","last_undegraded":"2024-03-18T20:07:20.617000+0000","last_fullsized":"2024-03-18T20:07:20.617000+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:04:30.494327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332298+0000","last_change":"2024-03-18T20:07:19.354331+0000","last_active":"2024-03-18T20:07:20.332298+0000","last_peered":"2024-03-18T20:07:20.332298+0000","last_clean":"2024-03-18T20:07:20.332298+0000","last_became_active":"2024-03-18T20:07:19.354058+0000","last_became_peered":"2024-03-18T20:07:19.354058+0000","last_unstale":"2024-03-18T20:07:20.332298+0000","last_undegraded":"2024-03-18T20:07:20.332298+0000","last_fullsized":"2024-03-18T20:07:20.332298+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:42:23.929450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618687+0000","last_change":"2024-03-18T20:07:19.340616+0000","last_active":"2024-03-18T20:07:20.618687+0000","last_peered":"2024-03-18T20:07:20.618687+0000","last_clean":"2024-03-18T20:07:20.618687+0000","last_became_active":"2024-03-18T20:07:19.340399+0000","last_became_peered":"2024-03-18T20:07:19.340399+0000","last_unstale":"2024-03-18T20:07:20.618687+0000","last_undegraded":"2024-03-18T20:07:20.618687+0000","last_fullsized":"2024-03-18T20:07:20.618687+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:14:59.554787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.331336+0000","last_change":"2024-03-18T20:07:19.337516+0000","last_active":"2024-03-18T20:07:20.331336+0000","last_peered":"2024-03-18T20:07:20.331336+0000","last_clean":"2024-03-18T20:07:20.331336+0000","last_became_active":"2024-03-18T20:07:19.337251+0000","last_became_peered":"2024-03-18T20:07:19.337251+0000","last_unstale":"2024-03-18T20:07:20.331336+0000","last_undegraded":"2024-03-18T20:07:20.331336+0000","last_fullsized":"2024-03-18T20:07:20.331336+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:54:47.068491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617216+0000","last_change":"2024-03-18T20:07:19.346742+0000","last_active":"2024-03-18T20:07:20.617216+0000","last_peered":"2024-03-18T20:07:20.617216+0000","last_clean":"2024-03-18T20:07:20.617216+0000","last_became_active":"2024-03-18T20:07:19.346467+0000","last_became_peered":"2024-03-18T20:07:19.346467+0000","last_unstale":"2024-03-18T20:07:20.617216+0000","last_undegraded":"2024-03-18T20:07:20.617216+0000","last_fullsized":"2024-03-18T20:07:20.617216+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:51:51.106876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617159+0000","last_change":"2024-03-18T20:07:19.350822+0000","last_active":"2024-03-18T20:07:20.617159+0000","last_peered":"2024-03-18T20:07:20.617159+0000","last_clean":"2024-03-18T20:07:20.617159+0000","last_became_active":"2024-03-18T20:07:19.350431+0000","last_became_peered":"2024-03-18T20:07:19.350431+0000","last_unstale":"2024-03-18T20:07:20.617159+0000","last_undegraded":"2024-03-18T20:07:20.617159+0000","last_fullsized":"2024-03-18T20:07:20.617159+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:05.555776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617316+0000","last_change":"2024-03-18T20:07:19.343817+0000","last_active":"2024-03-18T20:07:20.617316+0000","last_peered":"2024-03-18T20:07:20.617316+0000","last_clean":"2024-03-18T20:07:20.617316+0000","last_became_active":"2024-03-18T20:07:19.343588+0000","last_became_peered":"2024-03-18T20:07:19.343588+0000","last_unstale":"2024-03-18T20:07:20.617316+0000","last_undegraded":"2024-03-18T20:07:20.617316+0000","last_fullsized":"2024-03-18T20:07:20.617316+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:14.848323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.453972+0000","last_change":"2024-03-18T20:07:19.349446+0000","last_active":"2024-03-18T20:07:20.453972+0000","last_peered":"2024-03-18T20:07:20.453972+0000","last_clean":"2024-03-18T20:07:20.453972+0000","last_became_active":"2024-03-18T20:07:19.349155+0000","last_became_peered":"2024-03-18T20:07:19.349155+0000","last_unstale":"2024-03-18T20:07:20.453972+0000","last_undegraded":"2024-03-18T20:07:20.453972+0000","last_fullsized":"2024-03-18T20:07:20.453972+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:17:52.731361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618267+0000","last_change":"2024-03-18T20:07:19.340035+0000","last_active":"2024-03-18T20:07:20.618267+0000","last_peered":"2024-03-18T20:07:20.618267+0000","last_clean":"2024-03-18T20:07:20.618267+0000","last_became_active":"2024-03-18T20:07:19.339766+0000","last_became_peered":"2024-03-18T20:07:19.339766+0000","last_unstale":"2024-03-18T20:07:20.618267+0000","last_undegraded":"2024-03-18T20:07:20.618267+0000","last_fullsized":"2024-03-18T20:07:20.618267+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:09:26.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332440+0000","last_change":"2024-03-18T20:07:19.344031+0000","last_active":"2024-03-18T20:07:20.332440+0000","last_peered":"2024-03-18T20:07:20.332440+0000","last_clean":"2024-03-18T20:07:20.332440+0000","last_became_active":"2024-03-18T20:07:19.341621+0000","last_became_peered":"2024-03-18T20:07:19.341621+0000","last_unstale":"2024-03-18T20:07:20.332440+0000","last_undegraded":"2024-03-18T20:07:20.332440+0000","last_fullsized":"2024-03-18T20:07:20.332440+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:27:33.209246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617276+0000","last_change":"2024-03-18T20:07:19.350128+0000","last_active":"2024-03-18T20:07:20.617276+0000","last_peered":"2024-03-18T20:07:20.617276+0000","last_clean":"2024-03-18T20:07:20.617276+0000","last_became_active":"2024-03-18T20:07:19.345040+0000","last_became_peered":"2024-03-18T20:07:19.345040+0000","last_unstale":"2024-03-18T20:07:20.617276+0000","last_undegraded":"2024-03-18T20:07:20.617276+0000","last_fullsized":"2024-03-18T20:07:20.617276+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:09:28.038651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.616904+0000","last_change":"2024-03-18T20:07:19.349161+0000","last_active":"2024-03-18T20:07:20.616904+0000","last_peered":"2024-03-18T20:07:20.616904+0000","last_clean":"2024-03-18T20:07:20.616904+0000","last_became_active":"2024-03-18T20:07:19.348881+0000","last_became_peered":"2024-03-18T20:07:19.348881+0000","last_unstale":"2024-03-18T20:07:20.616904+0000","last_undegraded":"2024-03-18T20:07:20.616904+0000","last_fullsized":"2024-03-18T20:07:20.616904+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:58.611766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454074+0000","last_change":"2024-03-18T20:07:19.350206+0000","last_active":"2024-03-18T20:07:20.454074+0000","last_peered":"2024-03-18T20:07:20.454074+0000","last_clean":"2024-03-18T20:07:20.454074+0000","last_became_active":"2024-03-18T20:07:19.349950+0000","last_became_peered":"2024-03-18T20:07:19.349950+0000","last_unstale":"2024-03-18T20:07:20.454074+0000","last_undegraded":"2024-03-18T20:07:20.454074+0000","last_fullsized":"2024-03-18T20:07:20.454074+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:39:31.996325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.616923+0000","last_change":"2024-03-18T20:07:19.356428+0000","last_active":"2024-03-18T20:07:20.616923+0000","last_peered":"2024-03-18T20:07:20.616923+0000","last_clean":"2024-03-18T20:07:20.616923+0000","last_became_active":"2024-03-18T20:07:19.352909+0000","last_became_peered":"2024-03-18T20:07:19.352909+0000","last_unstale":"2024-03-18T20:07:20.616923+0000","last_undegraded":"2024-03-18T20:07:20.616923+0000","last_fullsized":"2024-03-18T20:07:20.616923+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:55:24.661146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454124+0000","last_change":"2024-03-18T20:07:19.342254+0000","last_active":"2024-03-18T20:07:20.454124+0000","last_peered":"2024-03-18T20:07:20.454124+0000","last_clean":"2024-03-18T20:07:20.454124+0000","last_became_active":"2024-03-18T20:07:19.341997+0000","last_became_peered":"2024-03-18T20:07:19.341997+0000","last_unstale":"2024-03-18T20:07:20.454124+0000","last_undegraded":"2024-03-18T20:07:20.454124+0000","last_fullsized":"2024-03-18T20:07:20.454124+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:15:16.471603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.330792+0000","last_change":"2024-03-18T20:07:19.343299+0000","last_active":"2024-03-18T20:07:20.330792+0000","last_peered":"2024-03-18T20:07:20.330792+0000","last_clean":"2024-03-18T20:07:20.330792+0000","last_became_active":"2024-03-18T20:07:19.343031+0000","last_became_peered":"2024-03-18T20:07:19.343031+0000","last_unstale":"2024-03-18T20:07:20.330792+0000","last_undegraded":"2024-03-18T20:07:20.330792+0000","last_fullsized":"2024-03-18T20:07:20.330792+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:20:46.798924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617139+0000","last_change":"2024-03-18T20:07:19.337751+0000","last_active":"2024-03-18T20:07:20.617139+0000","last_peered":"2024-03-18T20:07:20.617139+0000","last_clean":"2024-03-18T20:07:20.617139+0000","last_became_active":"2024-03-18T20:07:19.337464+0000","last_became_peered":"2024-03-18T20:07:19.337464+0000","last_unstale":"2024-03-18T20:07:20.617139+0000","last_undegraded":"2024-03-18T20:07:20.617139+0000","last_fullsized":"2024-03-18T20:07:20.617139+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:42:23.929450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.330581+0000","last_change":"2024-03-18T20:07:19.334107+0000","last_active":"2024-03-18T20:07:20.330581+0000","last_peered":"2024-03-18T20:07:20.330581+0000","last_clean":"2024-03-18T20:07:20.330581+0000","last_became_active":"2024-03-18T20:07:19.333820+0000","last_became_peered":"2024-03-18T20:07:19.333820+0000","last_unstale":"2024-03-18T20:07:20.330581+0000","last_undegraded":"2024-03-18T20:07:20.330581+0000","last_fullsized":"2024-03-18T20:07:20.330581+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:28.280469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454281+0000","last_change":"2024-03-18T20:07:19.336692+0000","last_active":"2024-03-18T20:07:20.454281+0000","last_peered":"2024-03-18T20:07:20.454281+0000","last_clean":"2024-03-18T20:07:20.454281+0000","last_became_active":"2024-03-18T20:07:19.336414+0000","last_became_peered":"2024-03-18T20:07:19.336414+0000","last_unstale":"2024-03-18T20:07:20.454281+0000","last_undegraded":"2024-03-18T20:07:20.454281+0000","last_fullsized":"2024-03-18T20:07:20.454281+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:24:28.820103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"194'4","reported_seq":15,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332769+0000","last_change":"2024-03-18T20:07:18.912523+0000","last_active":"2024-03-18T20:07:20.332769+0000","last_peered":"2024-03-18T20:07:20.332769+0000","last_clean":"2024-03-18T20:07:20.332769+0000","last_became_active":"2024-03-18T20:07:18.912324+0000","last_became_peered":"2024-03-18T20:07:18.912324+0000","last_unstale":"2024-03-18T20:07:20.332769+0000","last_undegraded":"2024-03-18T20:07:20.332769+0000","last_fullsized":"2024-03-18T20:07:20.332769+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:22.044963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.616710+0000","last_change":"2024-03-18T20:07:19.352221+0000","last_active":"2024-03-18T20:07:20.616710+0000","last_peered":"2024-03-18T20:07:20.616710+0000","last_clean":"2024-03-18T20:07:20.616710+0000","last_became_active":"2024-03-18T20:07:19.345171+0000","last_became_peered":"2024-03-18T20:07:19.345171+0000","last_unstale":"2024-03-18T20:07:20.616710+0000","last_undegraded":"2024-03-18T20:07:20.616710+0000","last_fullsized":"2024-03-18T20:07:20.616710+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:38:58.705427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"194'1","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618621+0000","last_change":"2024-03-18T20:07:18.910344+0000","last_active":"2024-03-18T20:07:20.618621+0000","last_peered":"2024-03-18T20:07:20.618621+0000","last_clean":"2024-03-18T20:07:20.618621+0000","last_became_active":"2024-03-18T20:07:18.909526+0000","last_became_peered":"2024-03-18T20:07:18.909526+0000","last_unstale":"2024-03-18T20:07:20.618621+0000","last_undegraded":"2024-03-18T20:07:20.618621+0000","last_fullsized":"2024-03-18T20:07:20.618621+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:30:26.326348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618886+0000","last_change":"2024-03-18T20:07:19.351713+0000","last_active":"2024-03-18T20:07:20.618886+0000","last_peered":"2024-03-18T20:07:20.618886+0000","last_clean":"2024-03-18T20:07:20.618886+0000","last_became_active":"2024-03-18T20:07:19.351456+0000","last_became_peered":"2024-03-18T20:07:19.351456+0000","last_unstale":"2024-03-18T20:07:20.618886+0000","last_undegraded":"2024-03-18T20:07:20.618886+0000","last_fullsized":"2024-03-18T20:07:20.618886+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:02.622237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"194'2","reported_seq":13,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617373+0000","last_change":"2024-03-18T20:07:18.902085+0000","last_active":"2024-03-18T20:07:20.617373+0000","last_peered":"2024-03-18T20:07:20.617373+0000","last_clean":"2024-03-18T20:07:20.617373+0000","last_became_active":"2024-03-18T20:07:18.901871+0000","last_became_peered":"2024-03-18T20:07:18.901871+0000","last_unstale":"2024-03-18T20:07:20.617373+0000","last_undegraded":"2024-03-18T20:07:20.617373+0000","last_fullsized":"2024-03-18T20:07:20.617373+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:22.044963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454094+0000","last_change":"2024-03-18T20:07:19.351678+0000","last_active":"2024-03-18T20:07:20.454094+0000","last_peered":"2024-03-18T20:07:20.454094+0000","last_clean":"2024-03-18T20:07:20.454094+0000","last_became_active":"2024-03-18T20:07:19.351423+0000","last_became_peered":"2024-03-18T20:07:19.351423+0000","last_unstale":"2024-03-18T20:07:20.454094+0000","last_undegraded":"2024-03-18T20:07:20.454094+0000","last_fullsized":"2024-03-18T20:07:20.454094+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:03:27.829819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"194'2","reported_seq":13,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617248+0000","last_change":"2024-03-18T20:07:18.905863+0000","last_active":"2024-03-18T20:07:20.617248+0000","last_peered":"2024-03-18T20:07:20.617248+0000","last_clean":"2024-03-18T20:07:20.617248+0000","last_became_active":"2024-03-18T20:07:18.905556+0000","last_became_peered":"2024-03-18T20:07:18.905556+0000","last_unstale":"2024-03-18T20:07:20.617248+0000","last_undegraded":"2024-03-18T20:07:20.617248+0000","last_fullsized":"2024-03-18T20:07:20.617248+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:45.055903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617832+0000","last_change":"2024-03-18T20:07:19.342067+0000","last_active":"2024-03-18T20:07:20.617832+0000","last_peered":"2024-03-18T20:07:20.617832+0000","last_clean":"2024-03-18T20:07:20.617832+0000","last_became_active":"2024-03-18T20:07:19.341811+0000","last_became_peered":"2024-03-18T20:07:19.341811+0000","last_unstale":"2024-03-18T20:07:20.617832+0000","last_undegraded":"2024-03-18T20:07:20.617832+0000","last_fullsized":"2024-03-18T20:07:20.617832+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:31:06.816211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"194'1","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617793+0000","last_change":"2024-03-18T20:07:18.907542+0000","last_active":"2024-03-18T20:07:20.617793+0000","last_peered":"2024-03-18T20:07:20.617793+0000","last_clean":"2024-03-18T20:07:20.617793+0000","last_became_active":"2024-03-18T20:07:18.907272+0000","last_became_peered":"2024-03-18T20:07:18.907272+0000","last_unstale":"2024-03-18T20:07:20.617793+0000","last_undegraded":"2024-03-18T20:07:20.617793+0000","last_fullsized":"2024-03-18T20:07:20.617793+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:05:29.037731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617488+0000","last_change":"2024-03-18T20:07:19.358359+0000","last_active":"2024-03-18T20:07:20.617488+0000","last_peered":"2024-03-18T20:07:20.617488+0000","last_clean":"2024-03-18T20:07:20.617488+0000","last_became_active":"2024-03-18T20:07:19.358043+0000","last_became_peered":"2024-03-18T20:07:19.358043+0000","last_unstale":"2024-03-18T20:07:20.617488+0000","last_undegraded":"2024-03-18T20:07:20.617488+0000","last_fullsized":"2024-03-18T20:07:20.617488+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:23.050391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"194'1","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618887+0000","last_change":"2024-03-18T20:07:18.914311+0000","last_active":"2024-03-18T20:07:20.618887+0000","last_peered":"2024-03-18T20:07:20.618887+0000","last_clean":"2024-03-18T20:07:20.618887+0000","last_became_active":"2024-03-18T20:07:18.914059+0000","last_became_peered":"2024-03-18T20:07:18.914059+0000","last_unstale":"2024-03-18T20:07:20.618887+0000","last_undegraded":"2024-03-18T20:07:20.618887+0000","last_fullsized":"2024-03-18T20:07:20.618887+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:08:51.354537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617005+0000","last_change":"2024-03-18T20:07:19.347886+0000","last_active":"2024-03-18T20:07:20.617005+0000","last_peered":"2024-03-18T20:07:20.617005+0000","last_clean":"2024-03-18T20:07:20.617005+0000","last_became_active":"2024-03-18T20:07:19.347627+0000","last_became_peered":"2024-03-18T20:07:19.347627+0000","last_unstale":"2024-03-18T20:07:20.617005+0000","last_undegraded":"2024-03-18T20:07:20.617005+0000","last_fullsized":"2024-03-18T20:07:20.617005+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:54:47.068491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617723+0000","last_change":"2024-03-18T20:07:18.911192+0000","last_active":"2024-03-18T20:07:20.617723+0000","last_peered":"2024-03-18T20:07:20.617723+0000","last_clean":"2024-03-18T20:07:20.617723+0000","last_became_active":"2024-03-18T20:07:18.910806+0000","last_became_peered":"2024-03-18T20:07:18.910806+0000","last_unstale":"2024-03-18T20:07:20.617723+0000","last_undegraded":"2024-03-18T20:07:20.617723+0000","last_fullsized":"2024-03-18T20:07:20.617723+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:25:36.075820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332707+0000","last_change":"2024-03-18T20:07:19.340706+0000","last_active":"2024-03-18T20:07:20.332707+0000","last_peered":"2024-03-18T20:07:20.332707+0000","last_clean":"2024-03-18T20:07:20.332707+0000","last_became_active":"2024-03-18T20:07:19.340436+0000","last_became_peered":"2024-03-18T20:07:19.340436+0000","last_unstale":"2024-03-18T20:07:20.332707+0000","last_undegraded":"2024-03-18T20:07:20.332707+0000","last_fullsized":"2024-03-18T20:07:20.332707+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:31:40.598513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.616776+0000","last_change":"2024-03-18T20:07:18.902226+0000","last_active":"2024-03-18T20:07:20.616776+0000","last_peered":"2024-03-18T20:07:20.616776+0000","last_clean":"2024-03-18T20:07:20.616776+0000","last_became_active":"2024-03-18T20:07:18.901889+0000","last_became_peered":"2024-03-18T20:07:18.901889+0000","last_unstale":"2024-03-18T20:07:20.616776+0000","last_undegraded":"2024-03-18T20:07:20.616776+0000","last_fullsized":"2024-03-18T20:07:20.616776+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:44.817526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454131+0000","last_change":"2024-03-18T20:07:19.336872+0000","last_active":"2024-03-18T20:07:20.454131+0000","last_peered":"2024-03-18T20:07:20.454131+0000","last_clean":"2024-03-18T20:07:20.454131+0000","last_became_active":"2024-03-18T20:07:19.336598+0000","last_became_peered":"2024-03-18T20:07:19.336598+0000","last_unstale":"2024-03-18T20:07:20.454131+0000","last_undegraded":"2024-03-18T20:07:20.454131+0000","last_fullsized":"2024-03-18T20:07:20.454131+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:49:59.914780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617356+0000","last_change":"2024-03-18T20:07:18.911125+0000","last_active":"2024-03-18T20:07:20.617356+0000","last_peered":"2024-03-18T20:07:20.617356+0000","last_clean":"2024-03-18T20:07:20.617356+0000","last_became_active":"2024-03-18T20:07:18.910844+0000","last_became_peered":"2024-03-18T20:07:18.910844+0000","last_unstale":"2024-03-18T20:07:20.617356+0000","last_undegraded":"2024-03-18T20:07:20.617356+0000","last_fullsized":"2024-03-18T20:07:20.617356+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:56:53.759606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.616720+0000","last_change":"2024-03-18T20:07:19.354104+0000","last_active":"2024-03-18T20:07:20.616720+0000","last_peered":"2024-03-18T20:07:20.616720+0000","last_clean":"2024-03-18T20:07:20.616720+0000","last_became_active":"2024-03-18T20:07:19.353940+0000","last_became_peered":"2024-03-18T20:07:19.353940+0000","last_unstale":"2024-03-18T20:07:20.616720+0000","last_undegraded":"2024-03-18T20:07:20.616720+0000","last_fullsized":"2024-03-18T20:07:20.616720+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:50:42.144860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"194'3","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332818+0000","last_change":"2024-03-18T20:07:18.898929+0000","last_active":"2024-03-18T20:07:20.332818+0000","last_peered":"2024-03-18T20:07:20.332818+0000","last_clean":"2024-03-18T20:07:20.332818+0000","last_became_active":"2024-03-18T20:07:18.898644+0000","last_became_peered":"2024-03-18T20:07:18.898644+0000","last_unstale":"2024-03-18T20:07:20.332818+0000","last_undegraded":"2024-03-18T20:07:20.332818+0000","last_fullsized":"2024-03-18T20:07:20.332818+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:56:21.115285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617052+0000","last_change":"2024-03-18T20:07:19.349242+0000","last_active":"2024-03-18T20:07:20.617052+0000","last_peered":"2024-03-18T20:07:20.617052+0000","last_clean":"2024-03-18T20:07:20.617052+0000","last_became_active":"2024-03-18T20:07:19.348995+0000","last_became_peered":"2024-03-18T20:07:19.348995+0000","last_unstale":"2024-03-18T20:07:20.617052+0000","last_undegraded":"2024-03-18T20:07:20.617052+0000","last_fullsized":"2024-03-18T20:07:20.617052+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:13:20.206058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"194'6","reported_seq":17,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617107+0000","last_change":"2024-03-18T20:07:18.912491+0000","last_active":"2024-03-18T20:07:20.617107+0000","last_peered":"2024-03-18T20:07:20.617107+0000","last_clean":"2024-03-18T20:07:20.617107+0000","last_became_active":"2024-03-18T20:07:18.912141+0000","last_became_peered":"2024-03-18T20:07:18.912141+0000","last_unstale":"2024-03-18T20:07:20.617107+0000","last_undegraded":"2024-03-18T20:07:20.617107+0000","last_fullsized":"2024-03-18T20:07:20.617107+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:39.286792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.330786+0000","last_change":"2024-03-18T20:07:19.334165+0000","last_active":"2024-03-18T20:07:20.330786+0000","last_peered":"2024-03-18T20:07:20.330786+0000","last_clean":"2024-03-18T20:07:20.330786+0000","last_became_active":"2024-03-18T20:07:19.333882+0000","last_became_peered":"2024-03-18T20:07:19.333882+0000","last_unstale":"2024-03-18T20:07:20.330786+0000","last_undegraded":"2024-03-18T20:07:20.330786+0000","last_fullsized":"2024-03-18T20:07:20.330786+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:43:27.974268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"194'2","reported_seq":13,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618632+0000","last_change":"2024-03-18T20:07:18.903676+0000","last_active":"2024-03-18T20:07:20.618632+0000","last_peered":"2024-03-18T20:07:20.618632+0000","last_clean":"2024-03-18T20:07:20.618632+0000","last_became_active":"2024-03-18T20:07:18.903259+0000","last_became_peered":"2024-03-18T20:07:18.903259+0000","last_unstale":"2024-03-18T20:07:20.618632+0000","last_undegraded":"2024-03-18T20:07:20.618632+0000","last_fullsized":"2024-03-18T20:07:20.618632+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:53:32.007909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618048+0000","last_change":"2024-03-18T20:07:19.343671+0000","last_active":"2024-03-18T20:07:20.618048+0000","last_peered":"2024-03-18T20:07:20.618048+0000","last_clean":"2024-03-18T20:07:20.618048+0000","last_became_active":"2024-03-18T20:07:19.343131+0000","last_became_peered":"2024-03-18T20:07:19.343131+0000","last_unstale":"2024-03-18T20:07:20.618048+0000","last_undegraded":"2024-03-18T20:07:20.618048+0000","last_fullsized":"2024-03-18T20:07:20.618048+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:02.622237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.616960+0000","last_change":"2024-03-18T20:07:18.913826+0000","last_active":"2024-03-18T20:07:20.616960+0000","last_peered":"2024-03-18T20:07:20.616960+0000","last_clean":"2024-03-18T20:07:20.616960+0000","last_became_active":"2024-03-18T20:07:18.913488+0000","last_became_peered":"2024-03-18T20:07:18.913488+0000","last_unstale":"2024-03-18T20:07:20.616960+0000","last_undegraded":"2024-03-18T20:07:20.616960+0000","last_fullsized":"2024-03-18T20:07:20.616960+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:51:00.839318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617209+0000","last_change":"2024-03-18T20:07:19.335409+0000","last_active":"2024-03-18T20:07:20.617209+0000","last_peered":"2024-03-18T20:07:20.617209+0000","last_clean":"2024-03-18T20:07:20.617209+0000","last_became_active":"2024-03-18T20:07:19.335143+0000","last_became_peered":"2024-03-18T20:07:19.335143+0000","last_unstale":"2024-03-18T20:07:20.617209+0000","last_undegraded":"2024-03-18T20:07:20.617209+0000","last_fullsized":"2024-03-18T20:07:20.617209+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:27:33.209246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617055+0000","last_change":"2024-03-18T20:07:18.903796+0000","last_active":"2024-03-18T20:07:20.617055+0000","last_peered":"2024-03-18T20:07:20.617055+0000","last_clean":"2024-03-18T20:07:20.617055+0000","last_became_active":"2024-03-18T20:07:18.903420+0000","last_became_peered":"2024-03-18T20:07:18.903420+0000","last_unstale":"2024-03-18T20:07:20.617055+0000","last_undegraded":"2024-03-18T20:07:20.617055+0000","last_fullsized":"2024-03-18T20:07:20.617055+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:16.631254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617673+0000","last_change":"2024-03-18T20:07:19.351972+0000","last_active":"2024-03-18T20:07:20.617673+0000","last_peered":"2024-03-18T20:07:20.617673+0000","last_clean":"2024-03-18T20:07:20.617673+0000","last_became_active":"2024-03-18T20:07:19.351721+0000","last_became_peered":"2024-03-18T20:07:19.351721+0000","last_unstale":"2024-03-18T20:07:20.617673+0000","last_undegraded":"2024-03-18T20:07:20.617673+0000","last_fullsized":"2024-03-18T20:07:20.617673+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:17.636682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"194'8","reported_seq":19,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617529+0000","last_change":"2024-03-18T20:07:18.902132+0000","last_active":"2024-03-18T20:07:20.617529+0000","last_peered":"2024-03-18T20:07:20.617529+0000","last_clean":"2024-03-18T20:07:20.617529+0000","last_became_active":"2024-03-18T20:07:18.901852+0000","last_became_peered":"2024-03-18T20:07:18.901852+0000","last_unstale":"2024-03-18T20:07:20.617529+0000","last_undegraded":"2024-03-18T20:07:20.617529+0000","last_fullsized":"2024-03-18T20:07:20.617529+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:39:30.990897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617433+0000","last_change":"2024-03-18T20:07:19.347506+0000","last_active":"2024-03-18T20:07:20.617433+0000","last_peered":"2024-03-18T20:07:20.617433+0000","last_clean":"2024-03-18T20:07:20.617433+0000","last_became_active":"2024-03-18T20:07:19.345389+0000","last_became_peered":"2024-03-18T20:07:19.345389+0000","last_unstale":"2024-03-18T20:07:20.617433+0000","last_undegraded":"2024-03-18T20:07:20.617433+0000","last_fullsized":"2024-03-18T20:07:20.617433+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:09:26.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"194'1","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617438+0000","last_change":"2024-03-18T20:07:18.910342+0000","last_active":"2024-03-18T20:07:20.617438+0000","last_peered":"2024-03-18T20:07:20.617438+0000","last_clean":"2024-03-18T20:07:20.617438+0000","last_became_active":"2024-03-18T20:07:18.909729+0000","last_became_peered":"2024-03-18T20:07:18.909729+0000","last_unstale":"2024-03-18T20:07:20.617438+0000","last_undegraded":"2024-03-18T20:07:20.617438+0000","last_fullsized":"2024-03-18T20:07:20.617438+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:06.134491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.330943+0000","last_change":"2024-03-18T20:07:19.338909+0000","last_active":"2024-03-18T20:07:20.330943+0000","last_peered":"2024-03-18T20:07:20.330943+0000","last_clean":"2024-03-18T20:07:20.330943+0000","last_became_active":"2024-03-18T20:07:19.338001+0000","last_became_peered":"2024-03-18T20:07:19.338001+0000","last_unstale":"2024-03-18T20:07:20.330943+0000","last_undegraded":"2024-03-18T20:07:20.330943+0000","last_fullsized":"2024-03-18T20:07:20.330943+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:42:02.327061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"194'2","reported_seq":13,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618179+0000","last_change":"2024-03-18T20:07:18.908765+0000","last_active":"2024-03-18T20:07:20.618179+0000","last_peered":"2024-03-18T20:07:20.618179+0000","last_clean":"2024-03-18T20:07:20.618179+0000","last_became_active":"2024-03-18T20:07:18.908470+0000","last_became_peered":"2024-03-18T20:07:18.908470+0000","last_unstale":"2024-03-18T20:07:20.618179+0000","last_undegraded":"2024-03-18T20:07:20.618179+0000","last_fullsized":"2024-03-18T20:07:20.618179+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:23:53.048106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617007+0000","last_change":"2024-03-18T20:07:19.340612+0000","last_active":"2024-03-18T20:07:20.617007+0000","last_peered":"2024-03-18T20:07:20.617007+0000","last_clean":"2024-03-18T20:07:20.617007+0000","last_became_active":"2024-03-18T20:07:19.340316+0000","last_became_peered":"2024-03-18T20:07:19.340316+0000","last_unstale":"2024-03-18T20:07:20.617007+0000","last_undegraded":"2024-03-18T20:07:20.617007+0000","last_fullsized":"2024-03-18T20:07:20.617007+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:56:41.216249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"194'1","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618752+0000","last_change":"2024-03-18T20:07:18.903509+0000","last_active":"2024-03-18T20:07:20.618752+0000","last_peered":"2024-03-18T20:07:20.618752+0000","last_clean":"2024-03-18T20:07:20.618752+0000","last_became_active":"2024-03-18T20:07:18.903235+0000","last_became_peered":"2024-03-18T20:07:18.903235+0000","last_unstale":"2024-03-18T20:07:20.618752+0000","last_undegraded":"2024-03-18T20:07:20.618752+0000","last_fullsized":"2024-03-18T20:07:20.618752+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:51:20.136124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617214+0000","last_change":"2024-03-18T20:07:19.344458+0000","last_active":"2024-03-18T20:07:20.617214+0000","last_peered":"2024-03-18T20:07:20.617214+0000","last_clean":"2024-03-18T20:07:20.617214+0000","last_became_active":"2024-03-18T20:07:19.344162+0000","last_became_peered":"2024-03-18T20:07:19.344162+0000","last_unstale":"2024-03-18T20:07:20.617214+0000","last_undegraded":"2024-03-18T20:07:20.617214+0000","last_fullsized":"2024-03-18T20:07:20.617214+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:14:59.413612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617914+0000","last_change":"2024-03-18T20:07:18.908241+0000","last_active":"2024-03-18T20:07:20.617914+0000","last_peered":"2024-03-18T20:07:20.617914+0000","last_clean":"2024-03-18T20:07:20.617914+0000","last_became_active":"2024-03-18T20:07:18.907935+0000","last_became_peered":"2024-03-18T20:07:18.907935+0000","last_unstale":"2024-03-18T20:07:20.617914+0000","last_undegraded":"2024-03-18T20:07:20.617914+0000","last_fullsized":"2024-03-18T20:07:20.617914+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:29.743674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617018+0000","last_change":"2024-03-18T20:07:19.353026+0000","last_active":"2024-03-18T20:07:20.617018+0000","last_peered":"2024-03-18T20:07:20.617018+0000","last_clean":"2024-03-18T20:07:20.617018+0000","last_became_active":"2024-03-18T20:07:19.352472+0000","last_became_peered":"2024-03-18T20:07:19.352472+0000","last_unstale":"2024-03-18T20:07:20.617018+0000","last_undegraded":"2024-03-18T20:07:20.617018+0000","last_fullsized":"2024-03-18T20:07:20.617018+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:08.668881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"194'2","reported_seq":13,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618567+0000","last_change":"2024-03-18T20:07:18.904376+0000","last_active":"2024-03-18T20:07:20.618567+0000","last_peered":"2024-03-18T20:07:20.618567+0000","last_clean":"2024-03-18T20:07:20.618567+0000","last_became_active":"2024-03-18T20:07:18.904047+0000","last_became_peered":"2024-03-18T20:07:18.904047+0000","last_unstale":"2024-03-18T20:07:20.618567+0000","last_undegraded":"2024-03-18T20:07:20.618567+0000","last_fullsized":"2024-03-18T20:07:20.618567+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:00:06.952190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618613+0000","last_change":"2024-03-18T20:07:19.340358+0000","last_active":"2024-03-18T20:07:20.618613+0000","last_peered":"2024-03-18T20:07:20.618613+0000","last_clean":"2024-03-18T20:07:20.618613+0000","last_became_active":"2024-03-18T20:07:19.339093+0000","last_became_peered":"2024-03-18T20:07:19.339093+0000","last_unstale":"2024-03-18T20:07:20.618613+0000","last_undegraded":"2024-03-18T20:07:20.618613+0000","last_fullsized":"2024-03-18T20:07:20.618613+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:56.777263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617058+0000","last_change":"2024-03-18T20:07:18.914624+0000","last_active":"2024-03-18T20:07:20.617058+0000","last_peered":"2024-03-18T20:07:20.617058+0000","last_clean":"2024-03-18T20:07:20.617058+0000","last_became_active":"2024-03-18T20:07:18.914330+0000","last_became_peered":"2024-03-18T20:07:18.914330+0000","last_unstale":"2024-03-18T20:07:20.617058+0000","last_undegraded":"2024-03-18T20:07:20.617058+0000","last_fullsized":"2024-03-18T20:07:20.617058+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:57.606338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618547+0000","last_change":"2024-03-18T20:07:19.340080+0000","last_active":"2024-03-18T20:07:20.618547+0000","last_peered":"2024-03-18T20:07:20.618547+0000","last_clean":"2024-03-18T20:07:20.618547+0000","last_became_active":"2024-03-18T20:07:19.339301+0000","last_became_peered":"2024-03-18T20:07:19.339301+0000","last_unstale":"2024-03-18T20:07:20.618547+0000","last_undegraded":"2024-03-18T20:07:20.618547+0000","last_fullsized":"2024-03-18T20:07:20.618547+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:35:11.568151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617808+0000","last_change":"2024-03-18T20:07:18.907216+0000","last_active":"2024-03-18T20:07:20.617808+0000","last_peered":"2024-03-18T20:07:20.617808+0000","last_clean":"2024-03-18T20:07:20.617808+0000","last_became_active":"2024-03-18T20:07:18.906898+0000","last_became_peered":"2024-03-18T20:07:18.906898+0000","last_unstale":"2024-03-18T20:07:20.617808+0000","last_undegraded":"2024-03-18T20:07:20.617808+0000","last_fullsized":"2024-03-18T20:07:20.617808+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:11:04.433179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454308+0000","last_change":"2024-03-18T20:07:19.345125+0000","last_active":"2024-03-18T20:07:20.454308+0000","last_peered":"2024-03-18T20:07:20.454308+0000","last_clean":"2024-03-18T20:07:20.454308+0000","last_became_active":"2024-03-18T20:07:19.344838+0000","last_became_peered":"2024-03-18T20:07:19.344838+0000","last_unstale":"2024-03-18T20:07:20.454308+0000","last_undegraded":"2024-03-18T20:07:20.454308+0000","last_fullsized":"2024-03-18T20:07:20.454308+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:28:49.494512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454282+0000","last_change":"2024-03-18T20:07:18.898269+0000","last_active":"2024-03-18T20:07:20.454282+0000","last_peered":"2024-03-18T20:07:20.454282+0000","last_clean":"2024-03-18T20:07:20.454282+0000","last_became_active":"2024-03-18T20:07:18.897985+0000","last_became_peered":"2024-03-18T20:07:18.897985+0000","last_unstale":"2024-03-18T20:07:20.454282+0000","last_undegraded":"2024-03-18T20:07:20.454282+0000","last_fullsized":"2024-03-18T20:07:20.454282+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:04.550348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617038+0000","last_change":"2024-03-18T20:07:19.336135+0000","last_active":"2024-03-18T20:07:20.617038+0000","last_peered":"2024-03-18T20:07:20.617038+0000","last_clean":"2024-03-18T20:07:20.617038+0000","last_became_active":"2024-03-18T20:07:19.335867+0000","last_became_peered":"2024-03-18T20:07:19.335867+0000","last_unstale":"2024-03-18T20:07:20.617038+0000","last_undegraded":"2024-03-18T20:07:20.617038+0000","last_fullsized":"2024-03-18T20:07:20.617038+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:00:43.939604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"194'2","reported_seq":13,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618640+0000","last_change":"2024-03-18T20:07:18.903804+0000","last_active":"2024-03-18T20:07:20.618640+0000","last_peered":"2024-03-18T20:07:20.618640+0000","last_clean":"2024-03-18T20:07:20.618640+0000","last_became_active":"2024-03-18T20:07:18.903200+0000","last_became_peered":"2024-03-18T20:07:18.903200+0000","last_unstale":"2024-03-18T20:07:20.618640+0000","last_undegraded":"2024-03-18T20:07:20.618640+0000","last_fullsized":"2024-03-18T20:07:20.618640+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:55:21.280803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617772+0000","last_change":"2024-03-18T20:07:19.344502+0000","last_active":"2024-03-18T20:07:20.617772+0000","last_peered":"2024-03-18T20:07:20.617772+0000","last_clean":"2024-03-18T20:07:20.617772+0000","last_became_active":"2024-03-18T20:07:19.344225+0000","last_became_peered":"2024-03-18T20:07:19.344225+0000","last_unstale":"2024-03-18T20:07:20.617772+0000","last_undegraded":"2024-03-18T20:07:20.617772+0000","last_fullsized":"2024-03-18T20:07:20.617772+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:15:04.952572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617275+0000","last_change":"2024-03-18T20:07:18.912301+0000","last_active":"2024-03-18T20:07:20.617275+0000","last_peered":"2024-03-18T20:07:20.617275+0000","last_clean":"2024-03-18T20:07:20.617275+0000","last_became_active":"2024-03-18T20:07:18.911927+0000","last_became_peered":"2024-03-18T20:07:18.911927+0000","last_unstale":"2024-03-18T20:07:20.617275+0000","last_undegraded":"2024-03-18T20:07:20.617275+0000","last_fullsized":"2024-03-18T20:07:20.617275+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:11.254210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617383+0000","last_change":"2024-03-18T20:07:19.348618+0000","last_active":"2024-03-18T20:07:20.617383+0000","last_peered":"2024-03-18T20:07:20.617383+0000","last_clean":"2024-03-18T20:07:20.617383+0000","last_became_active":"2024-03-18T20:07:19.348370+0000","last_became_peered":"2024-03-18T20:07:19.348370+0000","last_unstale":"2024-03-18T20:07:20.617383+0000","last_undegraded":"2024-03-18T20:07:20.617383+0000","last_fullsized":"2024-03-18T20:07:20.617383+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:56:22.120713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.616990+0000","last_change":"2024-03-18T20:07:18.905261+0000","last_active":"2024-03-18T20:07:20.616990+0000","last_peered":"2024-03-18T20:07:20.616990+0000","last_clean":"2024-03-18T20:07:20.616990+0000","last_became_active":"2024-03-18T20:07:18.904983+0000","last_became_peered":"2024-03-18T20:07:18.904983+0000","last_unstale":"2024-03-18T20:07:20.616990+0000","last_undegraded":"2024-03-18T20:07:20.616990+0000","last_fullsized":"2024-03-18T20:07:20.616990+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:03:41.544459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617712+0000","last_change":"2024-03-18T20:07:19.355229+0000","last_active":"2024-03-18T20:07:20.617712+0000","last_peered":"2024-03-18T20:07:20.617712+0000","last_clean":"2024-03-18T20:07:20.617712+0000","last_became_active":"2024-03-18T20:07:19.354939+0000","last_became_peered":"2024-03-18T20:07:19.354939+0000","last_unstale":"2024-03-18T20:07:20.617712+0000","last_undegraded":"2024-03-18T20:07:20.617712+0000","last_fullsized":"2024-03-18T20:07:20.617712+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:24.611484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617327+0000","last_change":"2024-03-18T20:07:18.907490+0000","last_active":"2024-03-18T20:07:20.617327+0000","last_peered":"2024-03-18T20:07:20.617327+0000","last_clean":"2024-03-18T20:07:20.617327+0000","last_became_active":"2024-03-18T20:07:18.907278+0000","last_became_peered":"2024-03-18T20:07:18.907278+0000","last_unstale":"2024-03-18T20:07:20.617327+0000","last_undegraded":"2024-03-18T20:07:20.617327+0000","last_fullsized":"2024-03-18T20:07:20.617327+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:36:25.885460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332610+0000","last_change":"2024-03-18T20:07:19.350492+0000","last_active":"2024-03-18T20:07:20.332610+0000","last_peered":"2024-03-18T20:07:20.332610+0000","last_clean":"2024-03-18T20:07:20.332610+0000","last_became_active":"2024-03-18T20:07:19.350224+0000","last_became_peered":"2024-03-18T20:07:19.350224+0000","last_unstale":"2024-03-18T20:07:20.332610+0000","last_undegraded":"2024-03-18T20:07:20.332610+0000","last_fullsized":"2024-03-18T20:07:20.332610+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:14.209896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.332460+0000","last_change":"2024-03-18T20:07:18.904697+0000","last_active":"2024-03-18T20:07:20.332460+0000","last_peered":"2024-03-18T20:07:20.332460+0000","last_clean":"2024-03-18T20:07:20.332460+0000","last_became_active":"2024-03-18T20:07:18.904431+0000","last_became_peered":"2024-03-18T20:07:18.904431+0000","last_unstale":"2024-03-18T20:07:20.332460+0000","last_undegraded":"2024-03-18T20:07:20.332460+0000","last_fullsized":"2024-03-18T20:07:20.332460+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:04:57.606338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454211+0000","last_change":"2024-03-18T20:07:19.337251+0000","last_active":"2024-03-18T20:07:20.454211+0000","last_peered":"2024-03-18T20:07:20.454211+0000","last_clean":"2024-03-18T20:07:20.454211+0000","last_became_active":"2024-03-18T20:07:19.336992+0000","last_became_peered":"2024-03-18T20:07:19.336992+0000","last_unstale":"2024-03-18T20:07:20.454211+0000","last_undegraded":"2024-03-18T20:07:20.454211+0000","last_fullsized":"2024-03-18T20:07:20.454211+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:01:40.097948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617052+0000","last_change":"2024-03-18T20:07:18.913624+0000","last_active":"2024-03-18T20:07:20.617052+0000","last_peered":"2024-03-18T20:07:20.617052+0000","last_clean":"2024-03-18T20:07:20.617052+0000","last_became_active":"2024-03-18T20:07:18.913173+0000","last_became_peered":"2024-03-18T20:07:18.913173+0000","last_unstale":"2024-03-18T20:07:20.617052+0000","last_undegraded":"2024-03-18T20:07:20.617052+0000","last_fullsized":"2024-03-18T20:07:20.617052+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:13:28.019732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617103+0000","last_change":"2024-03-18T20:07:19.344487+0000","last_active":"2024-03-18T20:07:20.617103+0000","last_peered":"2024-03-18T20:07:20.617103+0000","last_clean":"2024-03-18T20:07:20.617103+0000","last_became_active":"2024-03-18T20:07:19.343558+0000","last_became_peered":"2024-03-18T20:07:19.343558+0000","last_unstale":"2024-03-18T20:07:20.617103+0000","last_undegraded":"2024-03-18T20:07:20.617103+0000","last_fullsized":"2024-03-18T20:07:20.617103+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:56:33.413150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617181+0000","last_change":"2024-03-18T20:07:18.902224+0000","last_active":"2024-03-18T20:07:20.617181+0000","last_peered":"2024-03-18T20:07:20.617181+0000","last_clean":"2024-03-18T20:07:20.617181+0000","last_became_active":"2024-03-18T20:07:18.901937+0000","last_became_peered":"2024-03-18T20:07:18.901937+0000","last_unstale":"2024-03-18T20:07:20.617181+0000","last_undegraded":"2024-03-18T20:07:20.617181+0000","last_fullsized":"2024-03-18T20:07:20.617181+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:13:19.200630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617766+0000","last_change":"2024-03-18T20:07:19.351506+0000","last_active":"2024-03-18T20:07:20.617766+0000","last_peered":"2024-03-18T20:07:20.617766+0000","last_clean":"2024-03-18T20:07:20.617766+0000","last_became_active":"2024-03-18T20:07:19.351225+0000","last_became_peered":"2024-03-18T20:07:19.351225+0000","last_unstale":"2024-03-18T20:07:20.617766+0000","last_undegraded":"2024-03-18T20:07:20.617766+0000","last_fullsized":"2024-03-18T20:07:20.617766+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:27:03.104090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.618169+0000","last_change":"2024-03-18T20:07:18.904470+0000","last_active":"2024-03-18T20:07:20.618169+0000","last_peered":"2024-03-18T20:07:20.618169+0000","last_clean":"2024-03-18T20:07:20.618169+0000","last_became_active":"2024-03-18T20:07:18.904177+0000","last_became_peered":"2024-03-18T20:07:18.904177+0000","last_unstale":"2024-03-18T20:07:20.618169+0000","last_undegraded":"2024-03-18T20:07:20.618169+0000","last_fullsized":"2024-03-18T20:07:20.618169+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:30:59.566195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":10,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.454000+0000","last_change":"2024-03-18T20:07:19.338944+0000","last_active":"2024-03-18T20:07:20.454000+0000","last_peered":"2024-03-18T20:07:20.454000+0000","last_clean":"2024-03-18T20:07:20.454000+0000","last_became_active":"2024-03-18T20:07:19.338578+0000","last_became_peered":"2024-03-18T20:07:19.338578+0000","last_unstale":"2024-03-18T20:07:20.454000+0000","last_undegraded":"2024-03-18T20:07:20.454000+0000","last_fullsized":"2024-03-18T20:07:20.454000+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:18.883885+0000","last_clean_scrub_stamp":"2024-03-18T20:07:18.883885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:06.095092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":11,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-18T20:07:20.617189+0000","last_change":"2024-03-18T20:07:18.912669+0000","last_active":"2024-03-18T20:07:20.617189+0000","last_peered":"2024-03-18T20:07:20.617189+0000","last_clean":"2024-03-18T20:07:20.617189+0000","last_became_active":"2024-03-18T20:07:18.912382+0000","last_became_peered":"2024-03-18T20:07:18.912382+0000","last_unstale":"2024-03-18T20:07:20.617189+0000","last_undegraded":"2024-03-18T20:07:20.617189+0000","last_fullsized":"2024-03-18T20:07:20.617189+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:07:17.878457+0000","last_clean_scrub_stamp":"2024-03-18T20:07:17.878457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:10:06.566917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"184'293","reported_seq":965,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-03-18T20:07:19.333396+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:07:19.333396+0000","last_peered":"2024-03-18T20:07:19.333396+0000","last_clean":"2024-03-18T20:07:19.333396+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:07:19.333396+0000","last_undegraded":"2024-03-18T20:07:19.333396+0000","last_fullsized":"2024-03-18T20:07:19.333396+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575768,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48032,"kb_used_data":2964,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323808,"statfs":{"total":96636764160,"available":96587579392,"internally_reserved":0,"allocated":3035136,"data_stored":1828899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":46135755},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.57699999999999996,"15min":0.58299999999999996},"min":{"1min":0.33200000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.70199999999999996,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.58199999999999996,"15min":0.60899999999999999},"min":{"1min":0.35199999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.72799999999999998,"5min":0.80100000000000005,"15min":0.81699999999999995},"last":0.48299999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.58599999999999997,"15min":0.59899999999999998},"min":{"1min":0.318,"5min":0.249,"15min":0.249},"max":{"1min":0.74199999999999999,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.59699999999999998,"15min":0.60799999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.75900000000000001,"5min":0.79900000000000004,"15min":0.83899999999999997},"last":0.58799999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.58599999999999997,"15min":0.60099999999999998},"min":{"1min":0.39700000000000002,"5min":0.36499999999999999,"15min":0.33900000000000002},"max":{"1min":0.77500000000000002,"5min":0.872,"15min":0.872},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.60999999999999999,"15min":0.625},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78800000000000003,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.52400000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.60199999999999998,"15min":0.60799999999999998},"min":{"1min":0.40899999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80200000000000005,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.435},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.624,"15min":0.63},"min":{"1min":0.42399999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.77800000000000002,"5min":0.84699999999999998,"15min":0.89000000000000001},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Mar 18 20:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47199999999999998},"min":{"1min":0.34200000000000003,"5min":0.32500000000000001,"15min":0.315},"max":{"1min":0.73199999999999998,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.499,"15min":0.495},"min":{"1min":0.35599999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.84699999999999998},"last":0.59999999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51200000000000001,"15min":0.50700000000000001},"min":{"1min":0.35899999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73299999999999998,"5min":1.127,"15min":1.127},"last":0.623},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55100000000000005,"15min":0.54500000000000004},"min":{"1min":0.45000000000000001,"5min":0.35299999999999998,"15min":0.32900000000000001},"max":{"1min":0.80800000000000005,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.54700000000000004}]},{"osd":6,"last update":"Mon Mar 18 20:07:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56899999999999995,"15min":0.56100000000000005},"min":{"1min":0.42999999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.76500000000000001,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56699999999999995,"15min":0.55600000000000005},"min":{"1min":0.45500000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.91300000000000003},"last":0.55900000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575768,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41692,"kb_used_data":2384,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330148,"statfs":{"total":96636764160,"available":96594071552,"internally_reserved":0,"allocated":2441216,"data_stored":1238531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.61299999999999999,"15min":0.59599999999999997},"min":{"1min":0.40300000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73899999999999999,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.314},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.60499999999999998,"15min":0.59999999999999998},"min":{"1min":0.371,"5min":0.36099999999999999,"15min":0.30299999999999999},"max":{"1min":0.70499999999999996,"5min":1.054,"15min":1.054},"last":0.36099999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.60799999999999998,"15min":0.59599999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.34000000000000002},"max":{"1min":0.76700000000000002,"5min":0.82799999999999996,"15min":0.84599999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.61399999999999999,"15min":0.59999999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.28299999999999997},"max":{"1min":0.72999999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.47899999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.64100000000000001,"15min":0.625},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.40600000000000003},"max":{"1min":0.74299999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60399999999999998,"15min":0.59599999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.71799999999999997,"5min":0.79800000000000004,"15min":0.89600000000000002},"last":0.52200000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.64200000000000002,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73399999999999999,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60799999999999998,"15min":0.60099999999999998},"min":{"1min":0.42399999999999999,"5min":0.41599999999999998,"15min":0.39500000000000002},"max":{"1min":0.72099999999999997,"5min":1.448,"15min":1.448},"last":0.55900000000000005}]},{"osd":4,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.47499999999999998},"min":{"1min":0.318,"5min":0.316,"15min":0.316},"max":{"1min":0.69299999999999995,"5min":0.78000000000000003,"15min":0.82999999999999996},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.505,"15min":0.50700000000000001},"min":{"1min":0.372,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.65900000000000003,"5min":0.84399999999999997,"15min":0.86499999999999999},"last":0.60999999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.495,"15min":0.48999999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.63800000000000001,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.505},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54500000000000004,"15min":0.53300000000000003},"min":{"1min":0.43099999999999999,"5min":0.38200000000000001,"15min":0.33900000000000002},"max":{"1min":0.71199999999999997,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.622}]},{"osd":7,"last update":"Mon Mar 18 20:06:27 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.39600000000000002,"5min":0.378,"15min":0.34300000000000003},"max":{"1min":0.71899999999999997,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56200000000000006,"15min":0.54700000000000004},"min":{"1min":0.45000000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67900000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.59999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575768,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41704,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330136,"statfs":{"total":96636764160,"available":96594059264,"internally_reserved":0,"allocated":2453504,"data_stored":1240095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.38800000000000001,"15min":0.39700000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.42599999999999999,"5min":0.58499999999999996,"15min":0.63600000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.42599999999999999,"15min":0.44},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.504,"5min":0.71099999999999997,"15min":0.73599999999999999},"last":0.35299999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.42199999999999999,"15min":0.42399999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.25600000000000001},"max":{"1min":0.57499999999999996,"5min":0.66500000000000004,"15min":0.70299999999999996},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.46300000000000002,"15min":0.46800000000000003},"min":{"1min":0.313,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59699999999999998,"5min":0.77500000000000002,"15min":0.79300000000000004},"last":0.41399999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.48499999999999999,"15min":0.49199999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.49299999999999999,"5min":0.71999999999999997,"15min":0.754},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.439,"15min":0.45100000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.55400000000000005,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.40000000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.39400000000000002},"max":{"1min":0.78700000000000003,"5min":1.012,"15min":1.0509999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.48399999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.77000000000000002,"5min":1.081,"15min":1.081},"last":0.53000000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.65400000000000003,"15min":0.64800000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.69799999999999995,"5min":1.149,"15min":1.149},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.69499999999999995,"15min":0.69299999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.45600000000000002},"max":{"1min":0.79000000000000004,"5min":1.069,"15min":1.069},"last":0.57999999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.65700000000000003,"15min":0.65400000000000003},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.44700000000000001},"max":{"1min":0.81200000000000006,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.502},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.70799999999999996,"15min":0.69899999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.77700000000000002,"5min":1.101,"15min":1.101},"last":0.55400000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.68700000000000006,"15min":0.69299999999999995},"min":{"1min":0.48199999999999998,"5min":0.46600000000000003,"15min":0.44400000000000001},"max":{"1min":0.82999999999999996,"5min":1.141,"15min":1.141},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.69799999999999995,"15min":0.69499999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46000000000000002},"max":{"1min":0.76400000000000001,"5min":0.995,"15min":0.995},"last":0.51700000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575769,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42280,"kb_used_data":2972,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329560,"statfs":{"total":96636764160,"available":96593469440,"internally_reserved":0,"allocated":3043328,"data_stored":1829003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.38,"15min":0.39300000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.40999999999999998,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.40500000000000003,"15min":0.42299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.438,"5min":0.66800000000000004,"15min":0.67700000000000005},"last":0.42499999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.42899999999999999,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.53600000000000003,"5min":0.63200000000000001,"15min":0.67300000000000004},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.44400000000000001,"15min":0.45500000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.55700000000000005,"5min":0.65200000000000002,"15min":0.751},"last":0.46800000000000003}]},{"osd":3,"last update":"Mon Mar 18 20:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.42699999999999999,"15min":0.43099999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.27900000000000003},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.49399999999999999,"15min":0.501},"min":{"1min":0.308,"5min":0.308,"15min":0.307},"max":{"1min":0.59799999999999998,"5min":0.73399999999999999,"15min":0.75600000000000001},"last":0.54100000000000004}]},{"osd":4,"last update":"Mon Mar 18 20:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.64900000000000002,"15min":0.65800000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.74099999999999999,"5min":1.0109999999999999,"15min":1.044},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.67200000000000004,"15min":0.67100000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.84899999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.83299999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.67200000000000004,"15min":0.67800000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82199999999999995,"5min":1.0229999999999999,"15min":1.0629999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.68500000000000005,"15min":0.68200000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.42299999999999999},"max":{"1min":0.86499999999999999,"5min":1.042,"15min":1.095},"last":0.81699999999999995}]},{"osd":6,"last update":"Mon Mar 18 20:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.67500000000000004,"15min":0.67100000000000004},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.89200000000000002,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.69899999999999995,"15min":0.70399999999999996},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.80600000000000005,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.84899999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.69699999999999995,"15min":0.69999999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.81699999999999995,"5min":1.0600000000000001,"15min":1.075},"last":0.87},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.70399999999999996,"15min":0.70799999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.98199999999999998,"5min":1.02,"15min":1.02},"last":0.68999999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575769,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47452,"kb_used_data":2384,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324388,"statfs":{"total":96636764160,"available":96588173312,"internally_reserved":0,"allocated":2441216,"data_stored":1238531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.38900000000000001,"15min":0.38700000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.59999999999999998,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.46999999999999997,"15min":0.47399999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.218},"max":{"1min":0.63900000000000001,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.50800000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.47699999999999998,"15min":0.47899999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.27900000000000003},"max":{"1min":0.58399999999999996,"5min":0.749,"15min":0.77800000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.41099999999999998,"15min":0.41399999999999998},"min":{"1min":0.254,"5min":0.25,"15min":0.25},"max":{"1min":0.48699999999999999,"5min":0.58299999999999996,"15min":0.70799999999999996},"last":0.48899999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.41899999999999998,"15min":0.42199999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27300000000000002},"max":{"1min":0.52700000000000002,"5min":0.629,"15min":0.69399999999999995},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.52400000000000002,"15min":0.52500000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33700000000000002},"max":{"1min":0.66700000000000004,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.46400000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66600000000000004,"15min":0.65900000000000003},"min":{"1min":0.42599999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.65000000000000002,"15min":0.65200000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":1.0529999999999999},"last":0.69699999999999995}]},{"osd":5,"last update":"Mon Mar 18 20:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.67100000000000004,"15min":0.67400000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.40799999999999997},"max":{"1min":0.88,"5min":0.98999999999999999,"15min":1.0369999999999999},"last":0.63},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.67900000000000005,"15min":0.66700000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.68600000000000005}]},{"osd":6,"last update":"Mon Mar 18 20:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.67600000000000005,"15min":0.67700000000000005},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.39800000000000002},"max":{"1min":0.998,"5min":1.004,"15min":1.073},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.72399999999999998,"15min":0.72699999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.45100000000000001},"max":{"1min":0.94699999999999995,"5min":1.036,"15min":1.0660000000000001},"last":0.76500000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:07:05 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.69099999999999995,"15min":0.68999999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.748},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.69599999999999995,"15min":0.69399999999999995},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.97099999999999997,"5min":1.002,"15min":1.0109999999999999},"last":0.65300000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575769,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47472,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324368,"statfs":{"total":96636764160,"available":96588152832,"internally_reserved":0,"allocated":2461696,"data_stored":1240699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":46135752},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36799999999999999,"15min":0.38800000000000001},"min":{"1min":0.27100000000000002,"5min":0.248,"15min":0.23300000000000001},"max":{"1min":0.45400000000000001,"5min":0.66700000000000004,"15min":0.70599999999999996},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40000000000000002,"15min":0.41299999999999998},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.52200000000000002,"5min":0.61399999999999999,"15min":0.78200000000000003},"last":0.45800000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39600000000000002,"15min":0.41299999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.24199999999999999},"max":{"1min":0.58999999999999997,"5min":0.66600000000000004,"15min":0.66800000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.45200000000000001,"15min":0.46000000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.88900000000000001},"last":0.438}]},{"osd":2,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.44900000000000001,"15min":0.46500000000000002},"min":{"1min":0.31900000000000001,"5min":0.309,"15min":0.29399999999999998},"max":{"1min":0.57999999999999996,"5min":0.64100000000000001,"15min":0.71299999999999997},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48499999999999999,"15min":0.498},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.60599999999999998,"5min":0.72099999999999997,"15min":0.76200000000000001},"last":0.39300000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63900000000000001,"15min":0.64500000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.38800000000000001},"max":{"1min":0.84099999999999997,"5min":0.95599999999999996,"15min":1.0409999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.64500000000000002,"15min":0.65800000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41699999999999998},"max":{"1min":0.88800000000000001,"5min":1.214,"15min":1.214},"last":0.70399999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.67000000000000004,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.85899999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.745},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70799999999999996,"15min":0.71399999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.40500000000000003},"max":{"1min":0.90400000000000003,"5min":1.0349999999999999,"15min":1.0660000000000001},"last":0.80700000000000005}]},{"osd":6,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.433},"max":{"1min":0.88900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.70099999999999996,"15min":0.71399999999999997},"min":{"1min":0.48999999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.95199999999999996,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.78300000000000003}]},{"osd":7,"last update":"Mon Mar 18 20:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.69199999999999995,"15min":0.69899999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":1.02},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.68400000000000005,"15min":0.69099999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92200000000000004,"5min":1.0049999999999999,"15min":1.012},"last":0.71399999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575769,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41716,"kb_used_data":2400,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330124,"statfs":{"total":96636764160,"available":96594046976,"internally_reserved":0,"allocated":2457600,"data_stored":1239295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.57599999999999996,"15min":0.59899999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.309},"max":{"1min":0.67400000000000004,"5min":0.78700000000000003,"15min":1.212},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.78100000000000003,"5min":0.85399999999999998,"15min":1.079},"last":0.47399999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.60999999999999999,"15min":0.622},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73599999999999999,"5min":0.83399999999999996,"15min":1.143},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.58399999999999996,"15min":0.59399999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.34200000000000003},"max":{"1min":0.76900000000000002,"5min":0.78400000000000003,"15min":1.0669999999999999},"last":0.42399999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.58399999999999996,"15min":0.60299999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35199999999999998},"max":{"1min":0.79600000000000004,"5min":0.80900000000000005,"15min":1.246},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.59999999999999998,"15min":0.61799999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.68400000000000005,"5min":0.78200000000000003,"15min":1.1140000000000001},"last":0.70599999999999996}]},{"osd":3,"last update":"Mon Mar 18 20:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.59499999999999997,"15min":0.59999999999999998},"min":{"1min":0.41299999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73199999999999998,"5min":0.80500000000000005,"15min":1.034},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.60799999999999998,"15min":0.623},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.751,"5min":0.82799999999999996,"15min":1.1559999999999999},"last":0.50900000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67700000000000005,"5min":0.68600000000000005,"15min":0.94299999999999995},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.505,"15min":0.498},"min":{"1min":0.317,"5min":0.308,"15min":0.30599999999999999},"max":{"1min":0.66400000000000003,"5min":0.81699999999999995,"15min":0.92400000000000004},"last":0.64400000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48899999999999999,"15min":0.48799999999999999},"min":{"1min":0.33000000000000002,"5min":0.32500000000000001,"15min":0.308},"max":{"1min":0.65000000000000002,"5min":0.70999999999999996,"15min":0.95999999999999996},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.52700000000000002,"15min":0.52800000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.63400000000000001,"5min":0.80500000000000005,"15min":0.99299999999999999},"last":0.74099999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.34499999999999997},"max":{"1min":0.60799999999999998,"5min":0.76100000000000001,"15min":0.97699999999999998},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.56899999999999995,"15min":0.55700000000000005},"min":{"1min":0.375,"5min":0.36499999999999999,"15min":0.34499999999999997},"max":{"1min":0.68799999999999994,"5min":0.91500000000000004,"15min":0.96599999999999997},"last":0.72699999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575768,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63844,"kb_used_data":2392,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307996,"statfs":{"total":96636764160,"available":96571387904,"internally_reserved":0,"allocated":2449408,"data_stored":1238655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":62912967},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59699999999999998,"15min":0.59799999999999998},"min":{"1min":0.39300000000000002,"5min":0.35799999999999998,"15min":0.34300000000000003},"max":{"1min":0.85599999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.59299999999999997,"15min":0.60299999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.72299999999999998,"5min":0.80000000000000004,"15min":0.81799999999999995},"last":0.56899999999999995}]},{"osd":1,"last update":"Mon Mar 18 20:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39000000000000001},"max":{"1min":0.749,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.60199999999999998,"15min":0.61099999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.74299999999999999,"5min":0.84199999999999997,"15min":0.95199999999999996},"last":0.70099999999999996}]},{"osd":2,"last update":"Mon Mar 18 20:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.57899999999999996,"15min":0.59599999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.34899999999999998},"max":{"1min":0.76500000000000001,"5min":0.77800000000000002,"15min":0.83099999999999996},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.59099999999999997,"15min":0.60299999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.36099999999999999},"max":{"1min":0.70699999999999996,"5min":0.82899999999999996,"15min":0.88200000000000001},"last":0.54400000000000004}]},{"osd":3,"last update":"Mon Mar 18 20:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.59299999999999997,"15min":0.60899999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.374},"max":{"1min":0.81000000000000005,"5min":0.88200000000000001,"15min":0.91900000000000004},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.63700000000000001,"15min":0.63400000000000001},"min":{"1min":0.442,"5min":0.375,"15min":0.375},"max":{"1min":0.82099999999999995,"5min":0.874,"15min":0.874},"last":0.77600000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.48499999999999999,"15min":0.48099999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67000000000000004,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.52600000000000002,"15min":0.51500000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71899999999999997,"5min":0.79700000000000004,"15min":0.879},"last":0.79900000000000004}]},{"osd":6,"last update":"Mon Mar 18 20:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52000000000000002,"15min":0.505},"min":{"1min":0.311,"5min":0.311,"15min":0.29699999999999999},"max":{"1min":0.71199999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.53800000000000003,"15min":0.53600000000000003},"min":{"1min":0.38400000000000001,"5min":0.36299999999999999,"15min":0.33800000000000002},"max":{"1min":0.69799999999999995,"5min":0.76600000000000001,"15min":0.81699999999999995},"last":0.68799999999999994}]},{"osd":7,"last update":"Mon Mar 18 20:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56200000000000006,"15min":0.55500000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32000000000000001},"max":{"1min":0.754,"5min":0.83399999999999996,"15min":0.84499999999999997},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.44400000000000001,"5min":0.38300000000000001,"15min":0.34300000000000003},"max":{"1min":0.83699999999999997,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.81000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:07:23.900 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T20:07:23.900 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-18T20:07:24.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.062+0000 7f500c1e0700 1 Processor -- start 2024-03-18T20:07:24.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.063+0000 7f500c1e0700 1 -- start start 2024-03-18T20:07:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.063+0000 7f500c1e0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f500415b460 0x7f50040b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.063+0000 7f500c1e0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500409a150 0x7f50040c5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.063+0000 7f500c1e0700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50040c4c10 con 0x7f50040cc6e0 2024-03-18T20:07:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.063+0000 7f500c1e0700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50040c51d0 con 0x7f500415b460 2024-03-18T20:07:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.063+0000 7f500c1e0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50040ccbe0 con 0x7f500409a150 2024-03-18T20:07:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.063+0000 7f5009f7c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f500415b460 0x7f50040b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.064+0000 7f5009f7c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f500415b460 0x7f50040b36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37796/0 (socket says 172.21.15.1:37796) 2024-03-18T20:07:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.064+0000 7f5009f7c700 1 -- 172.21.15.1:0/2578812139 learned_addr learned my addr 172.21.15.1:0/2578812139 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.064+0000 7f500a77d700 1 --2- 172.21.15.1:0/2578812139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500409a150 0x7f50040c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.064+0000 7f5009f7c700 1 -- 172.21.15.1:0/2578812139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500409a150 msgr2=0x7f50040c5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.064+0000 7f5009f7c700 1 --2- 172.21.15.1:0/2578812139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500409a150 0x7f50040c5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.064+0000 7f5009f7c700 1 -- 172.21.15.1:0/2578812139 >> v1:172.21.15.165:6789/0 conn(0x7f50040cc6e0 legacy=0x7f5004099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.064+0000 7f5009f7c700 1 -- 172.21.15.1:0/2578812139 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50040ccec0 con 0x7f500415b460 2024-03-18T20:07:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.064+0000 7f500a77d700 1 --2- 172.21.15.1:0/2578812139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500409a150 0x7f50040c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.064+0000 7f5009f7c700 1 --2- 172.21.15.1:0/2578812139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f500415b460 0x7f50040b36d0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40098f0 tx=0x7f4ff4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e3d0c9c9b1a6d77 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.064+0000 7f5008f7a700 1 -- 172.21.15.1:0/2578812139 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4004230 con 0x7f500415b460 2024-03-18T20:07:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.064+0000 7f5008f7a700 1 -- 172.21.15.1:0/2578812139 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff4004390 con 0x7f500415b460 2024-03-18T20:07:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.064+0000 7f5008f7a700 1 -- 172.21.15.1:0/2578812139 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff40036a0 con 0x7f500415b460 2024-03-18T20:07:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.065+0000 7f500c1e0700 1 -- 172.21.15.1:0/2578812139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f500415b460 msgr2=0x7f50040b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.065+0000 7f500c1e0700 1 --2- 172.21.15.1:0/2578812139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f500415b460 0x7f50040b36d0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40098f0 tx=0x7f4ff4009c00 comp rx=0 tx=0).stop 2024-03-18T20:07:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.065+0000 7f500c1e0700 1 -- 172.21.15.1:0/2578812139 shutdown_connections 2024-03-18T20:07:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.065+0000 7f500c1e0700 1 --2- 172.21.15.1:0/2578812139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500409a150 0x7f50040c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.065+0000 7f500c1e0700 1 --2- 172.21.15.1:0/2578812139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f500415b460 0x7f50040b36d0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.065+0000 7f500c1e0700 1 -- 172.21.15.1:0/2578812139 >> 172.21.15.1:0/2578812139 conn(0x7f50040b2bb0 msgr2=0x7f50040b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.065+0000 7f500c1e0700 1 -- 172.21.15.1:0/2578812139 shutdown_connections 2024-03-18T20:07:24.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.065+0000 7f500c1e0700 1 -- 172.21.15.1:0/2578812139 wait complete. 2024-03-18T20:07:24.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.066+0000 7f500c1e0700 1 Processor -- start 2024-03-18T20:07:24.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.066+0000 7f500c1e0700 1 -- start start 2024-03-18T20:07:24.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.067+0000 7f500c1e0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500409a150 0x7f5004156f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:24.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.067+0000 7f500c1e0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50040cc6e0 0x7f50040b6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:24.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.067+0000 7f500c1e0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f500415b460 0x7f5004154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:24.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.067+0000 7f500c1e0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50040b6ed0 con 0x7f500415b460 2024-03-18T20:07:24.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.067+0000 7f500c1e0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5004155550 con 0x7f50040cc6e0 2024-03-18T20:07:24.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.067+0000 7f500c1e0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5004154f40 con 0x7f500409a150 2024-03-18T20:07:24.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.067+0000 7f500a77d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f500415b460 0x7f5004154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.068+0000 7f500a77d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f500415b460 0x7f5004154100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37798/0 (socket says 172.21.15.1:37798) 2024-03-18T20:07:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.068+0000 7f500a77d700 1 -- 172.21.15.1:0/1890474001 learned_addr learned my addr 172.21.15.1:0/1890474001 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.068+0000 7f5009f7c700 1 --2- 172.21.15.1:0/1890474001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500409a150 0x7f5004156f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.068+0000 7f500977b700 1 --2- 172.21.15.1:0/1890474001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50040cc6e0 0x7f50040b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.068+0000 7f500a77d700 1 -- 172.21.15.1:0/1890474001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500409a150 msgr2=0x7f5004156f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.068+0000 7f500a77d700 1 --2- 172.21.15.1:0/1890474001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500409a150 0x7f5004156f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.068+0000 7f500a77d700 1 -- 172.21.15.1:0/1890474001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50040cc6e0 msgr2=0x7f50040b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.068+0000 7f500a77d700 1 --2- 172.21.15.1:0/1890474001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50040cc6e0 0x7f50040b6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.068+0000 7f500a77d700 1 -- 172.21.15.1:0/1890474001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff4009550 con 0x7f500415b460 2024-03-18T20:07:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.068+0000 7f500977b700 1 --2- 172.21.15.1:0/1890474001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50040cc6e0 0x7f50040b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.068+0000 7f500a77d700 1 --2- 172.21.15.1:0/1890474001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f500415b460 0x7f5004154100 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc00cef0 tx=0x7f4ffc00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.068+0000 7f4ffa7fc700 1 -- 172.21.15.1:0/1890474001 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc00ae50 con 0x7f500415b460 2024-03-18T20:07:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.068+0000 7f4ffa7fc700 1 -- 172.21.15.1:0/1890474001 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ffc015440 con 0x7f500415b460 2024-03-18T20:07:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.068+0000 7f500c1e0700 1 -- 172.21.15.1:0/1890474001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5004157680 con 0x7f500415b460 2024-03-18T20:07:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.068+0000 7f4ffa7fc700 1 -- 172.21.15.1:0/1890474001 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc0157f0 con 0x7f500415b460 2024-03-18T20:07:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.068+0000 7f500c1e0700 1 -- 172.21.15.1:0/1890474001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50041551e0 con 0x7f500415b460 2024-03-18T20:07:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.069+0000 7f500c1e0700 1 -- 172.21.15.1:0/1890474001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5004155700 con 0x7f500415b460 2024-03-18T20:07:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.070+0000 7f4ffa7fc700 1 -- 172.21.15.1:0/1890474001 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4ffc00d430 con 0x7f500415b460 2024-03-18T20:07:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.071+0000 7f4ffa7fc700 1 --2- 172.21.15.1:0/1890474001 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4fe806adb0 0x7f4fe806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.071+0000 7f4ffa7fc700 1 -- 172.21.15.1:0/1890474001 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4ffc089e80 con 0x7f500415b460 2024-03-18T20:07:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.071+0000 7f5009f7c700 1 --2- 172.21.15.1:0/1890474001 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4fe806adb0 0x7f4fe806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.071+0000 7f5009f7c700 1 --2- 172.21.15.1:0/1890474001 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4fe806adb0 0x7f4fe806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4009840 tx=0x7f4ff4008030 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.074+0000 7f4ffa7fc700 1 -- 172.21.15.1:0/1890474001 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4ffc054ea0 con 0x7f500415b460 2024-03-18T20:07:24.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.358+0000 7f500c1e0700 1 -- 172.21.15.1:0/1890474001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f50040cb7b0 con 0x7f500415b460 2024-03-18T20:07:24.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.361+0000 7f4ffa7fc700 1 -- 172.21.15.1:0/1890474001 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v43) v1 ==== 190+0+0 (secure 0 0 0) 0x7f4ffc01a020 con 0x7f500415b460 2024-03-18T20:07:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.363+0000 7f500c1e0700 1 -- 172.21.15.1:0/1890474001 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4fe806adb0 msgr2=0x7f4fe806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.363+0000 7f500c1e0700 1 --2- 172.21.15.1:0/1890474001 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4fe806adb0 0x7f4fe806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4009840 tx=0x7f4ff4008030 comp rx=0 tx=0).stop 2024-03-18T20:07:24.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.363+0000 7f500c1e0700 1 -- 172.21.15.1:0/1890474001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f500415b460 msgr2=0x7f5004154100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:24.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.363+0000 7f500c1e0700 1 --2- 172.21.15.1:0/1890474001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f500415b460 0x7f5004154100 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc00cef0 tx=0x7f4ffc00a340 comp rx=0 tx=0).stop 2024-03-18T20:07:24.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.363+0000 7f500c1e0700 1 -- 172.21.15.1:0/1890474001 shutdown_connections 2024-03-18T20:07:24.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.363+0000 7f500c1e0700 1 --2- 172.21.15.1:0/1890474001 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f4fe806adb0 0x7f4fe806d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:24.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.363+0000 7f500c1e0700 1 --2- 172.21.15.1:0/1890474001 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f500409a150 0x7f5004156f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:24.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.363+0000 7f500c1e0700 1 --2- 172.21.15.1:0/1890474001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f500415b460 0x7f5004154100 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:24.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.363+0000 7f500c1e0700 1 --2- 172.21.15.1:0/1890474001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50040cc6e0 0x7f50040b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:24.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.363+0000 7f500c1e0700 1 -- 172.21.15.1:0/1890474001 >> 172.21.15.1:0/1890474001 conn(0x7f50040b2bb0 msgr2=0x7f50040cbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:24.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.363+0000 7f500c1e0700 1 -- 172.21.15.1:0/1890474001 shutdown_connections 2024-03-18T20:07:24.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.363+0000 7f500c1e0700 1 -- 172.21.15.1:0/1890474001 wait complete. 2024-03-18T20:07:24.370 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-18T20:07:24.380 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:07:24.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.502+0000 7f1065064700 1 Processor -- start 2024-03-18T20:07:24.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.502+0000 7f1065064700 1 -- start start 2024-03-18T20:07:24.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.503+0000 7f1065064700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1060159620 0x7f10600b3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:24.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.503+0000 7f1065064700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106015dc60 con 0x7f1060159a00 2024-03-18T20:07:24.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.503+0000 7f1065064700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1060071c40 con 0x7f106015e0f0 2024-03-18T20:07:24.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.503+0000 7f1065064700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106015d400 con 0x7f1060159620 2024-03-18T20:07:24.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.503+0000 7f105f59e700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1060159a00 0x7f1060071530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33998/0 (socket says 172.21.15.1:33998) 2024-03-18T20:07:24.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.503+0000 7f105f59e700 1 -- 172.21.15.1:0/3904660146 learned_addr learned my addr 172.21.15.1:0/3904660146 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:24.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.503+0000 7f105ed9d700 1 --2- 172.21.15.1:0/3904660146 >> v2:172.21.15.165:3300/0 conn(0x7f1060159620 0x7f10600b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:24.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.503+0000 7f105dd9b700 1 -- 172.21.15.1:0/3904660146 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172122901 0 0) 0x7f106015dc60 con 0x7f1060159a00 2024-03-18T20:07:24.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.503+0000 7f105dd9b700 1 -- 172.21.15.1:0/3904660146 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10480035b0 con 0x7f1060159a00 2024-03-18T20:07:24.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.503+0000 7f105dd9b700 1 -- 172.21.15.1:0/3904660146 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2420430049 0 0) 0x7f10480035b0 con 0x7f1060159a00 2024-03-18T20:07:24.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.503+0000 7f105dd9b700 1 -- 172.21.15.1:0/3904660146 >> v2:172.21.15.165:3300/0 conn(0x7f1060159620 msgr2=0x7f10600b3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:24.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.503+0000 7f105dd9b700 1 --2- 172.21.15.1:0/3904660146 >> v2:172.21.15.165:3300/0 conn(0x7f1060159620 0x7f10600b3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:24.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.503+0000 7f105dd9b700 1 -- 172.21.15.1:0/3904660146 >> v1:172.21.15.165:6789/0 conn(0x7f106015e0f0 legacy=0x7f1060070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:24.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105dd9b700 1 -- 172.21.15.1:0/3904660146 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106015ac20 con 0x7f1060159a00 2024-03-18T20:07:24.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105dd9b700 1 -- 172.21.15.1:0/3904660146 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f10580019f0 con 0x7f1060159a00 2024-03-18T20:07:24.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105dd9b700 1 -- 172.21.15.1:0/3904660146 >> v1:172.21.15.1:6789/0 conn(0x7f1060159a00 legacy=0x7f1060071530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:24.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105dd9b700 1 --2- 172.21.15.1:0/3904660146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1048003a00 0x7f1048003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:24.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105dd9b700 1 --2- 172.21.15.1:0/3904660146 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1048004850 0x7f1048004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:24.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105dd9b700 1 --2- 172.21.15.1:0/3904660146 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1048005210 0x7f104800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:24.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105dd9b700 1 -- 172.21.15.1:0/3904660146 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106015d400 con 0x7f1048003a00 2024-03-18T20:07:24.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105dd9b700 1 -- 172.21.15.1:0/3904660146 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104800c0b0 con 0x7f1048005210 2024-03-18T20:07:24.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105dd9b700 1 -- 172.21.15.1:0/3904660146 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f104800c270 con 0x7f1048004850 2024-03-18T20:07:24.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105ed9d700 1 --2- 172.21.15.1:0/3904660146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1048003a00 0x7f1048003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:24.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105f59e700 1 --2- 172.21.15.1:0/3904660146 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1048005210 0x7f104800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:24.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105e59c700 1 --2- 172.21.15.1:0/3904660146 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1048004850 0x7f1048004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:24.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105ed9d700 1 -- 172.21.15.1:0/3904660146 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1048004850 msgr2=0x7f1048004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:24.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105ed9d700 1 --2- 172.21.15.1:0/3904660146 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1048004850 0x7f1048004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:24.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105ed9d700 1 -- 172.21.15.1:0/3904660146 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1048005210 msgr2=0x7f104800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:24.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105ed9d700 1 --2- 172.21.15.1:0/3904660146 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1048005210 0x7f104800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:24.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105ed9d700 1 -- 172.21.15.1:0/3904660146 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10480035b0 con 0x7f1048003a00 2024-03-18T20:07:24.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105f59e700 1 --2- 172.21.15.1:0/3904660146 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1048005210 0x7f104800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:07:24.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105f59e700 1 -- 172.21.15.1:0/3904660146 reap_dead start 2024-03-18T20:07:24.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105ed9d700 1 --2- 172.21.15.1:0/3904660146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1048003a00 0x7f1048003e20 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f1054009b60 tx=0x7f1048005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a9e1a969980cfd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:24.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105dd9b700 1 -- 172.21.15.1:0/3904660146 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105400e6c0 con 0x7f1048003a00 2024-03-18T20:07:24.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105dd9b700 1 -- 172.21.15.1:0/3904660146 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105400e820 con 0x7f1048003a00 2024-03-18T20:07:24.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.504+0000 7f105dd9b700 1 -- 172.21.15.1:0/3904660146 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105400eb90 con 0x7f1048003a00 2024-03-18T20:07:24.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.505+0000 7f1065064700 1 -- 172.21.15.1:0/3904660146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1048003a00 msgr2=0x7f1048003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:24.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.505+0000 7f1065064700 1 --2- 172.21.15.1:0/3904660146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1048003a00 0x7f1048003e20 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f1054009b60 tx=0x7f1048005880 comp rx=0 tx=0).stop 2024-03-18T20:07:24.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.505+0000 7f1065064700 1 -- 172.21.15.1:0/3904660146 shutdown_connections 2024-03-18T20:07:24.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.505+0000 7f1065064700 1 --2- 172.21.15.1:0/3904660146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1048003a00 0x7f1048003e20 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:24.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.505+0000 7f1065064700 1 -- 172.21.15.1:0/3904660146 >> 172.21.15.1:0/3904660146 conn(0x7f106006a750 msgr2=0x7f106006ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:24.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.505+0000 7f1065064700 1 -- 172.21.15.1:0/3904660146 shutdown_connections 2024-03-18T20:07:24.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.505+0000 7f1065064700 1 -- 172.21.15.1:0/3904660146 wait complete. 2024-03-18T20:07:24.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.506+0000 7f1065064700 1 Processor -- start 2024-03-18T20:07:24.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.506+0000 7f1065064700 1 -- start start 2024-03-18T20:07:24.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.507+0000 7f1065064700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1048003a00 0x7f10601456b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:24.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.507+0000 7f1065064700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1060131640 0x7f1060146bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:24.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.507+0000 7f1065064700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10600cf190 0x7f106006d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:24.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.507+0000 7f1065064700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10600cfb80 con 0x7f10600cf190 2024-03-18T20:07:24.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.507+0000 7f1065064700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1060146110 con 0x7f1048003a00 2024-03-18T20:07:24.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.507+0000 7f1065064700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10601463f0 con 0x7f1060131640 2024-03-18T20:07:24.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.507+0000 7f105f59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10600cf190 0x7f106006d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:24.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.507+0000 7f105f59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10600cf190 0x7f106006d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49626/0 (socket says 172.21.15.1:49626) 2024-03-18T20:07:24.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.507+0000 7f105f59e700 1 -- 172.21.15.1:0/2987575447 learned_addr learned my addr 172.21.15.1:0/2987575447 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:24.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.507+0000 7f105e59c700 1 --2- 172.21.15.1:0/2987575447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1060131640 0x7f1060146bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:24.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.507+0000 7f105ed9d700 1 --2- 172.21.15.1:0/2987575447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1048003a00 0x7f10601456b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:24.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.508+0000 7f105f59e700 1 -- 172.21.15.1:0/2987575447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1060131640 msgr2=0x7f1060146bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:24.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.508+0000 7f105f59e700 1 --2- 172.21.15.1:0/2987575447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1060131640 0x7f1060146bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:24.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.508+0000 7f105f59e700 1 -- 172.21.15.1:0/2987575447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1048003a00 msgr2=0x7f10601456b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:24.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.508+0000 7f105f59e700 1 --2- 172.21.15.1:0/2987575447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1048003a00 0x7f10601456b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:24.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.508+0000 7f105f59e700 1 -- 172.21.15.1:0/2987575447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10601466d0 con 0x7f10600cf190 2024-03-18T20:07:24.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.508+0000 7f105f59e700 1 --2- 172.21.15.1:0/2987575447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10600cf190 0x7f106006d690 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f1058002000 tx=0x7f105800e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:24.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.508+0000 7f10477fe700 1 -- 172.21.15.1:0/2987575447 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105800eb00 con 0x7f10600cf190 2024-03-18T20:07:24.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.508+0000 7f10477fe700 1 -- 172.21.15.1:0/2987575447 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105800ec80 con 0x7f10600cf190 2024-03-18T20:07:24.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.508+0000 7f1065064700 1 -- 172.21.15.1:0/2987575447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1060145bf0 con 0x7f10600cf190 2024-03-18T20:07:24.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.508+0000 7f10477fe700 1 -- 172.21.15.1:0/2987575447 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1058013420 con 0x7f10600cf190 2024-03-18T20:07:24.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.508+0000 7f1065064700 1 -- 172.21.15.1:0/2987575447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1060147310 con 0x7f10600cf190 2024-03-18T20:07:24.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.509+0000 7f1065064700 1 -- 172.21.15.1:0/2987575447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1060145f70 con 0x7f10600cf190 2024-03-18T20:07:24.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.510+0000 7f10477fe700 1 -- 172.21.15.1:0/2987575447 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f10580135a0 con 0x7f10600cf190 2024-03-18T20:07:24.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.510+0000 7f10477fe700 1 --2- 172.21.15.1:0/2987575447 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f104c06adb0 0x7f104c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:24.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.510+0000 7f10477fe700 1 -- 172.21.15.1:0/2987575447 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1058089770 con 0x7f10600cf190 2024-03-18T20:07:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.510+0000 7f105ed9d700 1 --2- 172.21.15.1:0/2987575447 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f104c06adb0 0x7f104c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.510+0000 7f105ed9d700 1 --2- 172.21.15.1:0/2987575447 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f104c06adb0 0x7f104c06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f10540090b0 tx=0x7f105401e040 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.515+0000 7f10477fe700 1 -- 172.21.15.1:0/2987575447 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1060145f70 con 0x7f10600cf190 2024-03-18T20:07:24.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.759+0000 7f1065064700 1 -- 172.21.15.1:0/2987575447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1060147310 con 0x7f10600cf190 2024-03-18T20:07:24.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.760+0000 7f10477fe700 1 -- 172.21.15.1:0/2987575447 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4741 (secure 0 0 0) 0x7f1060147310 con 0x7f10600cf190 2024-03-18T20:07:24.765 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:07:24.765 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":89,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8204,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2572860438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2572860438},{"type":"v1","addr":"172.21.15.165:6835","nonce":2572860438}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9439,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3158632513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3158632513},{"type":"v1","addr":"172.21.15.1:6836","nonce":3158632513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9448,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/4005747445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4005747445},{"type":"v1","addr":"172.21.15.165:6834","nonce":4005747445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:07:19.322806+0000","modified":"2024-03-18T20:07:22.335335+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10422":{"gid":10422,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1610313086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1610313086},{"type":"v1","addr":"172.21.15.1:6837","nonce":1610313086}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-03-18T20:07:24.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.762+0000 7f1065064700 1 -- 172.21.15.1:0/2987575447 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f104c06adb0 msgr2=0x7f104c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:24.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.762+0000 7f1065064700 1 --2- 172.21.15.1:0/2987575447 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f104c06adb0 0x7f104c06d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f10540090b0 tx=0x7f105401e040 comp rx=0 tx=0).stop 2024-03-18T20:07:24.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.762+0000 7f1065064700 1 -- 172.21.15.1:0/2987575447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10600cf190 msgr2=0x7f106006d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.762+0000 7f1065064700 1 --2- 172.21.15.1:0/2987575447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10600cf190 0x7f106006d690 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f1058002000 tx=0x7f105800e440 comp rx=0 tx=0).stop 2024-03-18T20:07:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.762+0000 7f1065064700 1 -- 172.21.15.1:0/2987575447 shutdown_connections 2024-03-18T20:07:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.762+0000 7f1065064700 1 --2- 172.21.15.1:0/2987575447 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f104c06adb0 0x7f104c06d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.762+0000 7f1065064700 1 --2- 172.21.15.1:0/2987575447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10600cf190 0x7f106006d690 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.762+0000 7f1065064700 1 --2- 172.21.15.1:0/2987575447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1048003a00 0x7f10601456b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.762+0000 7f1065064700 1 --2- 172.21.15.1:0/2987575447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1060131640 0x7f1060146bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.762+0000 7f1065064700 1 -- 172.21.15.1:0/2987575447 >> 172.21.15.1:0/2987575447 conn(0x7f106006a750 msgr2=0x7f106015c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.762+0000 7f1065064700 1 -- 172.21.15.1:0/2987575447 shutdown_connections 2024-03-18T20:07:24.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:24.762+0000 7f1065064700 1 -- 172.21.15.1:0/2987575447 wait complete. 2024-03-18T20:07:24.769 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 89 2024-03-18T20:07:24.886 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-18T20:07:24.882+0000 7f152709d700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T20:07:25.183 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T20:07:19.322806+0000', 'modified': '2024-03-18T20:07:22.335335+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10422}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10422': {'gid': 10422, 'name': 'c', 'rank': 0, 'incarnation': 85, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6837/1610313086', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6835', 'nonce': 1610313086}, {'type': 'v1', 'addr': '172.21.15.1:6837', 'nonce': 1610313086}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T20:07:25.184 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T20:07:25.184 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T20:07:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.312+0000 7fb3284aa700 1 Processor -- start 2024-03-18T20:07:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.312+0000 7fb3284aa700 1 -- start start 2024-03-18T20:07:25.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.313+0000 7fb3284aa700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb320071490 0x7fb32015b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.313+0000 7fb3284aa700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb32015a380 0x7fb320071870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.313+0000 7fb3284aa700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32015b6f0 con 0x7fb3200724e0 2024-03-18T20:07:25.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.313+0000 7fb3284aa700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3200cb510 con 0x7fb32015a380 2024-03-18T20:07:25.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.313+0000 7fb3284aa700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb320071db0 con 0x7fb320071490 2024-03-18T20:07:25.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.313+0000 7fb325a45700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb32015a380 0x7fb320071870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.313+0000 7fb325a45700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb32015a380 0x7fb320071870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49642/0 (socket says 172.21.15.1:49642) 2024-03-18T20:07:25.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.313+0000 7fb325a45700 1 -- 172.21.15.1:0/2281606055 learned_addr learned my addr 172.21.15.1:0/2281606055 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:25.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.313+0000 7fb326246700 1 --2- 172.21.15.1:0/2281606055 >> v2:172.21.15.165:3300/0 conn(0x7fb320071490 0x7fb32015b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.313+0000 7fb325a45700 1 -- 172.21.15.1:0/2281606055 >> v2:172.21.15.165:3300/0 conn(0x7fb320071490 msgr2=0x7fb32015b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:25.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.313+0000 7fb325a45700 1 --2- 172.21.15.1:0/2281606055 >> v2:172.21.15.165:3300/0 conn(0x7fb320071490 0x7fb32015b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.313+0000 7fb325a45700 1 -- 172.21.15.1:0/2281606055 >> v1:172.21.15.165:6789/0 conn(0x7fb3200724e0 legacy=0x7fb32015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:25.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.313+0000 7fb325a45700 1 -- 172.21.15.1:0/2281606055 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3200ca900 con 0x7fb32015a380 2024-03-18T20:07:25.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.313+0000 7fb326246700 1 --2- 172.21.15.1:0/2281606055 >> v2:172.21.15.165:3300/0 conn(0x7fb320071490 0x7fb32015b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:25.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.314+0000 7fb325a45700 1 --2- 172.21.15.1:0/2281606055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb32015a380 0x7fb320071870 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fb32006c7d0 tx=0x7fb31c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40c82771a60bc45b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:25.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.314+0000 7fb325244700 1 -- 172.21.15.1:0/2281606055 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c004230 con 0x7fb32015a380 2024-03-18T20:07:25.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.314+0000 7fb325244700 1 -- 172.21.15.1:0/2281606055 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31c004390 con 0x7fb32015a380 2024-03-18T20:07:25.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.314+0000 7fb325244700 1 -- 172.21.15.1:0/2281606055 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c0036a0 con 0x7fb32015a380 2024-03-18T20:07:25.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.314+0000 7fb3284aa700 1 -- 172.21.15.1:0/2281606055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb32015a380 msgr2=0x7fb320071870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:25.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.314+0000 7fb3284aa700 1 --2- 172.21.15.1:0/2281606055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb32015a380 0x7fb320071870 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fb32006c7d0 tx=0x7fb31c009bd0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.314+0000 7fb3284aa700 1 -- 172.21.15.1:0/2281606055 shutdown_connections 2024-03-18T20:07:25.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.314+0000 7fb3284aa700 1 --2- 172.21.15.1:0/2281606055 >> v2:172.21.15.165:3300/0 conn(0x7fb320071490 0x7fb32015b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.314+0000 7fb3284aa700 1 --2- 172.21.15.1:0/2281606055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb32015a380 0x7fb320071870 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.314+0000 7fb3284aa700 1 -- 172.21.15.1:0/2281606055 >> 172.21.15.1:0/2281606055 conn(0x7fb3200b1aa0 msgr2=0x7fb3200b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:25.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.314+0000 7fb3284aa700 1 -- 172.21.15.1:0/2281606055 shutdown_connections 2024-03-18T20:07:25.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.314+0000 7fb3284aa700 1 -- 172.21.15.1:0/2281606055 wait complete. 2024-03-18T20:07:25.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.316+0000 7fb3284aa700 1 Processor -- start 2024-03-18T20:07:25.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.316+0000 7fb3284aa700 1 -- start start 2024-03-18T20:07:25.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.316+0000 7fb3284aa700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb320071490 0x7fb3200aff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.316+0000 7fb3284aa700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3200724e0 0x7fb320154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.316+0000 7fb3284aa700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb32015a380 0x7fb3200a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.316+0000 7fb3284aa700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3200c6670 con 0x7fb32015a380 2024-03-18T20:07:25.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.316+0000 7fb3284aa700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32009fed0 con 0x7fb320071490 2024-03-18T20:07:25.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.316+0000 7fb3284aa700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3200c52a0 con 0x7fb3200724e0 2024-03-18T20:07:25.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.316+0000 7fb326a47700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb32015a380 0x7fb3200a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.316+0000 7fb326a47700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb32015a380 0x7fb3200a0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49654/0 (socket says 172.21.15.1:49654) 2024-03-18T20:07:25.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.316+0000 7fb326a47700 1 -- 172.21.15.1:0/1571827458 learned_addr learned my addr 172.21.15.1:0/1571827458 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:25.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.317+0000 7fb326a47700 1 -- 172.21.15.1:0/1571827458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3200724e0 msgr2=0x7fb320154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:25.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.317+0000 7fb326a47700 1 --2- 172.21.15.1:0/1571827458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3200724e0 0x7fb320154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.317+0000 7fb326246700 1 --2- 172.21.15.1:0/1571827458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb320071490 0x7fb3200aff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.317+0000 7fb326a47700 1 -- 172.21.15.1:0/1571827458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb320071490 msgr2=0x7fb3200aff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:25.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.317+0000 7fb326a47700 1 --2- 172.21.15.1:0/1571827458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb320071490 0x7fb3200aff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.317+0000 7fb326a47700 1 -- 172.21.15.1:0/1571827458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c009550 con 0x7fb32015a380 2024-03-18T20:07:25.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.317+0000 7fb326246700 1 --2- 172.21.15.1:0/1571827458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb320071490 0x7fb3200aff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:25.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.317+0000 7fb326a47700 1 --2- 172.21.15.1:0/1571827458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb32015a380 0x7fb3200a0970 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fb31800b970 tx=0x7fb31800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:25.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.317+0000 7fb316ffd700 1 -- 172.21.15.1:0/1571827458 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31800c9b0 con 0x7fb32015a380 2024-03-18T20:07:25.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.317+0000 7fb316ffd700 1 -- 172.21.15.1:0/1571827458 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31800cb10 con 0x7fb32015a380 2024-03-18T20:07:25.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.317+0000 7fb316ffd700 1 -- 172.21.15.1:0/1571827458 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb318014440 con 0x7fb32015a380 2024-03-18T20:07:25.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.317+0000 7fb3284aa700 1 -- 172.21.15.1:0/1571827458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3200c4fc0 con 0x7fb32015a380 2024-03-18T20:07:25.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.317+0000 7fb3284aa700 1 -- 172.21.15.1:0/1571827458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3200a0450 con 0x7fb32015a380 2024-03-18T20:07:25.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.319+0000 7fb316ffd700 1 -- 172.21.15.1:0/1571827458 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb31801d3b0 con 0x7fb32015a380 2024-03-18T20:07:25.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.319+0000 7fb316ffd700 1 --2- 172.21.15.1:0/1571827458 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb30406adb0 0x7fb30406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.319+0000 7fb326246700 1 --2- 172.21.15.1:0/1571827458 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb30406adb0 0x7fb30406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.319+0000 7fb316ffd700 1 -- 172.21.15.1:0/1571827458 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb318089310 con 0x7fb32015a380 2024-03-18T20:07:25.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.320+0000 7fb326246700 1 --2- 172.21.15.1:0/1571827458 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb30406adb0 0x7fb30406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb310008910 tx=0x7fb310008040 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:25.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.321+0000 7fb334653b80 1 Processor -- start 2024-03-18T20:07:25.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.321+0000 7fb334653b80 1 -- start start 2024-03-18T20:07:25.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.322+0000 7fb334653b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0eea3fb0 0x55ee0ee7f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.322+0000 7fb334653b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0ee7f6a0 0x55ee0eec7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.322+0000 7fb334653b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0eea4390 0x55ee0eeaaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.322+0000 7fb334653b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee0edf5f20 con 0x55ee0ee7f6a0 2024-03-18T20:07:25.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.322+0000 7fb334653b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee0edd7c30 con 0x55ee0eea3fb0 2024-03-18T20:07:25.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.322+0000 7fb334653b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ee0edd5430 con 0x55ee0eea4390 2024-03-18T20:07:25.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.322+0000 7fb325a45700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0ee7f6a0 0x55ee0eec7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.323+0000 7fb325a45700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0ee7f6a0 0x55ee0eec7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49660/0 (socket says 172.21.15.1:49660) 2024-03-18T20:07:25.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.323+0000 7fb325a45700 1 -- 172.21.15.1:0/3553087509 learned_addr learned my addr 172.21.15.1:0/3553087509 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:25.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.323+0000 7fb326a47700 1 --2- 172.21.15.1:0/3553087509 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0eea4390 0x55ee0eeaaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.323+0000 7fb325a45700 1 -- 172.21.15.1:0/3553087509 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0eea4390 msgr2=0x55ee0eeaaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:25.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.323+0000 7fb325a45700 1 --2- 172.21.15.1:0/3553087509 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0eea4390 0x55ee0eeaaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.323+0000 7fb325a45700 1 -- 172.21.15.1:0/3553087509 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0eea3fb0 msgr2=0x55ee0ee7f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:25.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.323+0000 7fb326246700 1 --2- 172.21.15.1:0/3553087509 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0eea3fb0 0x55ee0ee7f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.323+0000 7fb325a45700 1 --2- 172.21.15.1:0/3553087509 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0eea3fb0 0x55ee0ee7f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.323+0000 7fb325a45700 1 -- 172.21.15.1:0/3553087509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee0ebc4790 con 0x55ee0ee7f6a0 2024-03-18T20:07:25.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.323+0000 7fb326a47700 1 --2- 172.21.15.1:0/3553087509 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0eea4390 0x55ee0eeaaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:25.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.323+0000 7fb325a45700 1 --2- 172.21.15.1:0/3553087509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0ee7f6a0 0x55ee0eec7550 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fb320067530 tx=0x7fb31c00e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6adb6df62cb460f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:25.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.323+0000 7fb303fff700 1 -- 172.21.15.1:0/3553087509 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c004230 con 0x55ee0ee7f6a0 2024-03-18T20:07:25.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.323+0000 7fb303fff700 1 -- 172.21.15.1:0/3553087509 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31c004580 con 0x55ee0ee7f6a0 2024-03-18T20:07:25.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.323+0000 7fb303fff700 1 -- 172.21.15.1:0/3553087509 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c003a50 con 0x55ee0ee7f6a0 2024-03-18T20:07:25.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.324+0000 7fb334653b80 1 -- 172.21.15.1:0/3553087509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0ee7f6a0 msgr2=0x55ee0eec7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:25.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.324+0000 7fb334653b80 1 --2- 172.21.15.1:0/3553087509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0ee7f6a0 0x55ee0eec7550 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fb320067530 tx=0x7fb31c00e000 comp rx=0 tx=0).stop 2024-03-18T20:07:25.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.324+0000 7fb334653b80 1 -- 172.21.15.1:0/3553087509 shutdown_connections 2024-03-18T20:07:25.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.324+0000 7fb334653b80 1 --2- 172.21.15.1:0/3553087509 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0eea4390 0x55ee0eeaaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.324+0000 7fb334653b80 1 --2- 172.21.15.1:0/3553087509 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0eea3fb0 0x55ee0ee7f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.324+0000 7fb334653b80 1 --2- 172.21.15.1:0/3553087509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0ee7f6a0 0x55ee0eec7550 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.324+0000 7fb334653b80 1 -- 172.21.15.1:0/3553087509 >> 172.21.15.1:0/3553087509 conn(0x55ee0eddecf0 msgr2=0x55ee0ee7ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:25.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.324+0000 7fb334653b80 1 -- 172.21.15.1:0/3553087509 shutdown_connections 2024-03-18T20:07:25.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.324+0000 7fb334653b80 1 -- 172.21.15.1:0/3553087509 wait complete. 2024-03-18T20:07:25.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.324+0000 7fb334653b80 1 -- start start 2024-03-18T20:07:25.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.325+0000 7fb334653b80 1 Processor -- start 2024-03-18T20:07:25.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.325+0000 7fb334653b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0ee7f6a0 0x55ee0eeac5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.325+0000 7fb334653b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0eea3fb0 0x55ee0ee7a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.325+0000 7fb334653b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0eea4390 0x55ee0eea0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.325+0000 7fb334653b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee0edd0d20 con 0x55ee0ee7f6a0 2024-03-18T20:07:25.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb334653b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee0ed6c770 con 0x55ee0eea4390 2024-03-18T20:07:25.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb334653b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ee0ed8f6f0 con 0x55ee0eea3fb0 2024-03-18T20:07:25.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb334653b80 10 client.0 _ll_register_callbacks cb 0x55ee0ebd85a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:07:25.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb326246700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0ee7f6a0 0x55ee0eeac5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb326246700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0ee7f6a0 0x55ee0eeac5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49674/0 (socket says 172.21.15.1:49674) 2024-03-18T20:07:25.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb326246700 1 -- 172.21.15.1:0/411195967 learned_addr learned my addr 172.21.15.1:0/411195967 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:25.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb325a45700 1 --2- 172.21.15.1:0/411195967 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0eea3fb0 0x55ee0ee7a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb326246700 1 -- 172.21.15.1:0/411195967 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0eea3fb0 msgr2=0x55ee0ee7a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:25.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb326246700 1 --2- 172.21.15.1:0/411195967 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0eea3fb0 0x55ee0ee7a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb326246700 1 -- 172.21.15.1:0/411195967 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0eea4390 msgr2=0x55ee0eea0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:25.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb326246700 1 --2- 172.21.15.1:0/411195967 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0eea4390 0x55ee0eea0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb326246700 1 -- 172.21.15.1:0/411195967 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3200ca900 con 0x55ee0ee7f6a0 2024-03-18T20:07:25.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb326246700 1 --2- 172.21.15.1:0/411195967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0ee7f6a0 0x55ee0eeac5d0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fb310008f60 tx=0x7fb3100047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f334ea4c615998a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:25.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb3017fa700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T20:07:25.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb3017fa700 1 -- 172.21.15.1:0/411195967 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31000b050 con 0x55ee0ee7f6a0 2024-03-18T20:07:25.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb3017fa700 1 -- 172.21.15.1:0/411195967 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(195..195 src has 1..195) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb31001f610 con 0x55ee0ee7f6a0 2024-03-18T20:07:25.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb334653b80 1 -- 172.21.15.1:0/411195967 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ee0ee72e90 con 0x55ee0ee7f6a0 2024-03-18T20:07:25.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb3017fa700 1 -- 172.21.15.1:0/411195967 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb31001f7d0 con 0x55ee0ee7f6a0 2024-03-18T20:07:25.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb334653b80 10 client.10563 fetch_fsmap learned FSMap version 89 2024-03-18T20:07:25.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.326+0000 7fb334653b80 1 -- 172.21.15.1:0/411195967 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) v3 -- 0x55ee0ecb10a0 con 0x55ee0ee7f6a0 2024-03-18T20:07:25.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.327+0000 7fb3017fa700 1 -- 172.21.15.1:0/411195967 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ee0ee72e90 con 0x55ee0ee7f6a0 2024-03-18T20:07:25.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.327+0000 7fb3017fa700 1 -- 172.21.15.1:0/411195967 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 89) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb31001f930 con 0x55ee0ee7f6a0 2024-03-18T20:07:25.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.327+0000 7fb334653b80 10 client.10563 fetch_fsmap finished waiting for FSMap version 89 2024-03-18T20:07:25.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.327+0000 7fb334653b80 10 client.10563 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T20:07:25.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.327+0000 7fb334653b80 20 client.10563 populate_metadata read hostname 'smithi001' 2024-03-18T20:07:25.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.327+0000 7fb334653b80 1 --2- 172.21.15.1:0/411195967 >> [v2:172.21.15.1:6835/1610313086,v1:172.21.15.1:6837/1610313086] conn(0x55ee0eea64b0 0x55ee0eea6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.327+0000 7fb334653b80 4 client.10563 mds_command: new command op to 10422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T20:07:25.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.327+0000 7fb334653b80 1 -- 172.21.15.1:0/411195967 --> [v2:172.21.15.1:6835/1610313086,v1:172.21.15.1:6837/1610313086] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ee0ee50010 con 0x55ee0eea64b0 2024-03-18T20:07:25.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.327+0000 7fb325a45700 1 --2- 172.21.15.1:0/411195967 >> [v2:172.21.15.1:6835/1610313086,v1:172.21.15.1:6837/1610313086] conn(0x55ee0eea64b0 0x55ee0eea6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.327+0000 7fb3017fa700 10 client.10563 ms_handle_connect on v2:172.21.15.1:6835/1610313086 2024-03-18T20:07:25.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.327+0000 7fb325a45700 1 --2- 172.21.15.1:0/411195967 >> [v2:172.21.15.1:6835/1610313086,v1:172.21.15.1:6837/1610313086] conn(0x55ee0eea64b0 0x55ee0eea6890 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f44961fa2a72d92 server_cookie=ecb3399d224dc350 in_seq=0 out_seq=0 2024-03-18T20:07:25.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.328+0000 7fb3017fa700 1 -- 172.21.15.1:0/411195967 <== mds.0 v2:172.21.15.1:6835/1610313086 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ee0ee50010 con 0x55ee0eea64b0 2024-03-18T20:07:25.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.328+0000 7fb3017fa700 10 client.10563 handle_command_reply: tid=0 2024-03-18T20:07:25.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.328+0000 7fb334653b80 1 client.10563 shutdown 2024-03-18T20:07:25.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.328+0000 7fb334653b80 1 -- 172.21.15.1:0/411195967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0ee7f6a0 msgr2=0x55ee0eeac5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:25.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.328+0000 7fb334653b80 1 --2- 172.21.15.1:0/411195967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0ee7f6a0 0x55ee0eeac5d0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fb310008f60 tx=0x7fb3100047c0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.328+0000 7fb334653b80 1 -- 172.21.15.1:0/411195967 shutdown_connections 2024-03-18T20:07:25.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.328+0000 7fb334653b80 1 --2- 172.21.15.1:0/411195967 >> [v2:172.21.15.1:6835/1610313086,v1:172.21.15.1:6837/1610313086] conn(0x55ee0eea64b0 0x55ee0eea6890 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.328+0000 7fb334653b80 1 --2- 172.21.15.1:0/411195967 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0eea4390 0x55ee0eea0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.328+0000 7fb334653b80 1 --2- 172.21.15.1:0/411195967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0ee7f6a0 0x55ee0eeac5d0 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.328+0000 7fb3017fa700 0 client.10563 ms_handle_reset on v2:172.21.15.1:6835/1610313086 2024-03-18T20:07:25.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.328+0000 7fb334653b80 1 --2- 172.21.15.1:0/411195967 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0eea3fb0 0x55ee0ee7a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.328+0000 7fb334653b80 1 -- 172.21.15.1:0/411195967 >> 172.21.15.1:0/411195967 conn(0x55ee0eddecf0 msgr2=0x55ee0eeab1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:25.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.328+0000 7fb334653b80 1 -- 172.21.15.1:0/411195967 shutdown_connections 2024-03-18T20:07:25.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.328+0000 7fb334653b80 1 -- 172.21.15.1:0/411195967 wait complete. 2024-03-18T20:07:25.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.328+0000 7fb334653b80 20 client.10563 trim_cache size 0 max 16384 2024-03-18T20:07:25.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.346+0000 7fb334653b80 1 Processor -- start 2024-03-18T20:07:25.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.346+0000 7fb334653b80 1 -- start start 2024-03-18T20:07:25.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.347+0000 7fb334653b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0ee7f6a0 0x55ee0ee7fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.347+0000 7fb334653b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0eea3fb0 0x55ee0ee7d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.347+0000 7fb334653b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0eea4390 0x55ee0ef2a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.347+0000 7fb334653b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee0ebc5390 con 0x55ee0eea4390 2024-03-18T20:07:25.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.347+0000 7fb334653b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee0eea81b0 con 0x55ee0ee7f6a0 2024-03-18T20:07:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.347+0000 7fb334653b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ee0eea7bc0 con 0x55ee0eea3fb0 2024-03-18T20:07:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.347+0000 7fb326a47700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0eea4390 0x55ee0ef2a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.347+0000 7fb326a47700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0eea4390 0x55ee0ef2a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49680/0 (socket says 172.21.15.1:49680) 2024-03-18T20:07:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.347+0000 7fb326a47700 1 -- 172.21.15.1:0/2952205747 learned_addr learned my addr 172.21.15.1:0/2952205747 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.347+0000 7fb325a45700 1 --2- 172.21.15.1:0/2952205747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0eea3fb0 0x55ee0ee7d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.347+0000 7fb326246700 1 --2- 172.21.15.1:0/2952205747 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0ee7f6a0 0x55ee0ee7fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.347+0000 7fb326a47700 1 -- 172.21.15.1:0/2952205747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0eea3fb0 msgr2=0x55ee0ee7d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.347+0000 7fb326a47700 1 --2- 172.21.15.1:0/2952205747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0eea3fb0 0x55ee0ee7d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.347+0000 7fb326a47700 1 -- 172.21.15.1:0/2952205747 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0ee7f6a0 msgr2=0x55ee0ee7fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.347+0000 7fb326a47700 1 --2- 172.21.15.1:0/2952205747 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0ee7f6a0 0x55ee0ee7fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.347+0000 7fb326a47700 1 -- 172.21.15.1:0/2952205747 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee0ed97010 con 0x55ee0eea4390 2024-03-18T20:07:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.347+0000 7fb326246700 1 --2- 172.21.15.1:0/2952205747 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0ee7f6a0 0x55ee0ee7fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:07:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.348+0000 7fb326a47700 1 --2- 172.21.15.1:0/2952205747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0eea4390 0x55ee0ef2a2f0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7fb318057bf0 tx=0x7fb3180585f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=635becbcc2b9158 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.348+0000 7fb314da8700 1 -- 172.21.15.1:0/2952205747 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3180602e0 con 0x55ee0eea4390 2024-03-18T20:07:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.348+0000 7fb314da8700 1 -- 172.21.15.1:0/2952205747 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb318056290 con 0x55ee0eea4390 2024-03-18T20:07:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.348+0000 7fb314da8700 1 -- 172.21.15.1:0/2952205747 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb318058150 con 0x55ee0eea4390 2024-03-18T20:07:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.348+0000 7fb334653b80 1 -- 172.21.15.1:0/2952205747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0eea4390 msgr2=0x55ee0ef2a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.348+0000 7fb334653b80 1 --2- 172.21.15.1:0/2952205747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0eea4390 0x55ee0ef2a2f0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7fb318057bf0 tx=0x7fb3180585f0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.348+0000 7fb334653b80 1 -- 172.21.15.1:0/2952205747 shutdown_connections 2024-03-18T20:07:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.348+0000 7fb334653b80 1 --2- 172.21.15.1:0/2952205747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0eea4390 0x55ee0ef2a2f0 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.348+0000 7fb334653b80 1 --2- 172.21.15.1:0/2952205747 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0ee7f6a0 0x55ee0ee7fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.348+0000 7fb334653b80 1 --2- 172.21.15.1:0/2952205747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0eea3fb0 0x55ee0ee7d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.348+0000 7fb334653b80 1 -- 172.21.15.1:0/2952205747 >> 172.21.15.1:0/2952205747 conn(0x55ee0eddecf0 msgr2=0x55ee0eeaa9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.349+0000 7fb334653b80 1 -- 172.21.15.1:0/2952205747 shutdown_connections 2024-03-18T20:07:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.349+0000 7fb334653b80 1 -- 172.21.15.1:0/2952205747 wait complete. 2024-03-18T20:07:25.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.349+0000 7fb334653b80 1 -- start start 2024-03-18T20:07:25.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.349+0000 7fb334653b80 1 Processor -- start 2024-03-18T20:07:25.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.350+0000 7fb334653b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0ee7f6a0 0x55ee0ef23d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.350+0000 7fb334653b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0eea3fb0 0x55ee0ee7a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.350+0000 7fb334653b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0eea4390 0x55ee0ee7ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.350+0000 7fb334653b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee0eec1b30 con 0x55ee0eea4390 2024-03-18T20:07:25.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.350+0000 7fb334653b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee0eead590 con 0x55ee0eea3fb0 2024-03-18T20:07:25.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.350+0000 7fb334653b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ee0eead7e0 con 0x55ee0ee7f6a0 2024-03-18T20:07:25.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.350+0000 7fb334653b80 10 client.0 _ll_register_callbacks cb 0x7fb31000da50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:07:25.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.350+0000 7fb326a47700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0eea4390 0x55ee0ee7ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.350+0000 7fb326a47700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0eea4390 0x55ee0ee7ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49690/0 (socket says 172.21.15.1:49690) 2024-03-18T20:07:25.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.350+0000 7fb326a47700 1 -- 172.21.15.1:0/77277006 learned_addr learned my addr 172.21.15.1:0/77277006 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:25.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.350+0000 7fb326246700 1 --2- 172.21.15.1:0/77277006 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0ee7f6a0 0x55ee0ef23d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.350+0000 7fb325a45700 1 --2- 172.21.15.1:0/77277006 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0eea3fb0 0x55ee0ee7a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.350+0000 7fb326a47700 1 -- 172.21.15.1:0/77277006 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0ee7f6a0 msgr2=0x55ee0ef23d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:25.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.350+0000 7fb326a47700 1 --2- 172.21.15.1:0/77277006 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0ee7f6a0 0x55ee0ef23d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.350+0000 7fb326a47700 1 -- 172.21.15.1:0/77277006 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0eea3fb0 msgr2=0x55ee0ee7a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:25.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.350+0000 7fb326a47700 1 --2- 172.21.15.1:0/77277006 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0eea3fb0 0x55ee0ee7a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.350+0000 7fb326a47700 1 -- 172.21.15.1:0/77277006 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3200a0450 con 0x55ee0eea4390 2024-03-18T20:07:25.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.351+0000 7fb326a47700 1 --2- 172.21.15.1:0/77277006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0eea4390 0x55ee0ee7ac20 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7fb318009a00 tx=0x7fb31800e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=289ade3d7c7e633a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:25.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.351+0000 7fb3017fa700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T20:07:25.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.351+0000 7fb3017fa700 1 -- 172.21.15.1:0/77277006 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3180575e0 con 0x55ee0eea4390 2024-03-18T20:07:25.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.351+0000 7fb3017fa700 1 -- 172.21.15.1:0/77277006 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(195..195 src has 1..195) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb3180602e0 con 0x55ee0eea4390 2024-03-18T20:07:25.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.351+0000 7fb334653b80 1 -- 172.21.15.1:0/77277006 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb31c004580 con 0x55ee0eea4390 2024-03-18T20:07:25.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.351+0000 7fb3017fa700 1 -- 172.21.15.1:0/77277006 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb318058150 con 0x55ee0eea4390 2024-03-18T20:07:25.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.351+0000 7fb334653b80 10 client.10569 fetch_fsmap learned FSMap version 89 2024-03-18T20:07:25.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.351+0000 7fb334653b80 1 -- 172.21.15.1:0/77277006 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) v3 -- 0x55ee0ec251d0 con 0x55ee0eea4390 2024-03-18T20:07:25.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.351+0000 7fb3017fa700 1 -- 172.21.15.1:0/77277006 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31c004580 con 0x55ee0eea4390 2024-03-18T20:07:25.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.351+0000 7fb3017fa700 1 -- 172.21.15.1:0/77277006 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 89) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb31805fe50 con 0x55ee0eea4390 2024-03-18T20:07:25.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.351+0000 7fb334653b80 10 client.10569 fetch_fsmap finished waiting for FSMap version 89 2024-03-18T20:07:25.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.351+0000 7fb334653b80 10 client.10569 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T20:07:25.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.351+0000 7fb334653b80 20 client.10569 populate_metadata read hostname 'smithi001' 2024-03-18T20:07:25.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.351+0000 7fb334653b80 1 --2- 172.21.15.1:0/77277006 >> [v2:172.21.15.1:6835/1610313086,v1:172.21.15.1:6837/1610313086] conn(0x55ee0ee7df10 0x55ee0ee7e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.351+0000 7fb334653b80 4 client.10569 mds_command: new command op to 10422 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T20:07:25.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.351+0000 7fb334653b80 1 -- 172.21.15.1:0/77277006 --> [v2:172.21.15.1:6835/1610313086,v1:172.21.15.1:6837/1610313086] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ee0ec0a040 con 0x55ee0ee7df10 2024-03-18T20:07:25.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.351+0000 7fb326246700 1 --2- 172.21.15.1:0/77277006 >> [v2:172.21.15.1:6835/1610313086,v1:172.21.15.1:6837/1610313086] conn(0x55ee0ee7df10 0x55ee0ee7e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.352+0000 7fb3017fa700 10 client.10569 ms_handle_connect on v2:172.21.15.1:6835/1610313086 2024-03-18T20:07:25.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.352+0000 7fb326246700 1 --2- 172.21.15.1:0/77277006 >> [v2:172.21.15.1:6835/1610313086,v1:172.21.15.1:6837/1610313086] conn(0x55ee0ee7df10 0x55ee0ee7e2f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9dfb05531e88ba9 server_cookie=a4a89bd3607cba14 in_seq=0 out_seq=0 2024-03-18T20:07:25.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.352+0000 7fb3017fa700 1 -- 172.21.15.1:0/77277006 <== mds.0 v2:172.21.15.1:6835/1610313086 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55ee0ec0a040 con 0x55ee0ee7df10 2024-03-18T20:07:25.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.352+0000 7fb3017fa700 10 client.10569 handle_command_reply: tid=0 2024-03-18T20:07:25.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.352+0000 7fb334653b80 1 client.10569 shutdown 2024-03-18T20:07:25.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.352+0000 7fb334653b80 1 -- 172.21.15.1:0/77277006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0eea4390 msgr2=0x55ee0ee7ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:25.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.352+0000 7fb334653b80 1 --2- 172.21.15.1:0/77277006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0eea4390 0x55ee0ee7ac20 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7fb318009a00 tx=0x7fb31800e040 comp rx=0 tx=0).stop 2024-03-18T20:07:25.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.352+0000 7fb334653b80 1 -- 172.21.15.1:0/77277006 shutdown_connections 2024-03-18T20:07:25.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.352+0000 7fb334653b80 1 --2- 172.21.15.1:0/77277006 >> [v2:172.21.15.1:6835/1610313086,v1:172.21.15.1:6837/1610313086] conn(0x55ee0ee7df10 0x55ee0ee7e2f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.352+0000 7fb334653b80 1 --2- 172.21.15.1:0/77277006 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55ee0ee7f6a0 0x55ee0ef23d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.352+0000 7fb3017fa700 0 client.10569 ms_handle_reset on v2:172.21.15.1:6835/1610313086 2024-03-18T20:07:25.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.352+0000 7fb334653b80 1 --2- 172.21.15.1:0/77277006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ee0eea4390 0x55ee0ee7ac20 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.352+0000 7fb334653b80 1 --2- 172.21.15.1:0/77277006 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ee0eea3fb0 0x55ee0ee7a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.352+0000 7fb334653b80 1 -- 172.21.15.1:0/77277006 >> 172.21.15.1:0/77277006 conn(0x55ee0eddecf0 msgr2=0x55ee0eeaa9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:25.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.352+0000 7fb334653b80 1 -- 172.21.15.1:0/77277006 shutdown_connections 2024-03-18T20:07:25.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.352+0000 7fb334653b80 1 -- 172.21.15.1:0/77277006 wait complete. 2024-03-18T20:07:25.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.352+0000 7fb334653b80 20 client.10569 trim_cache size 0 max 16384 2024-03-18T20:07:25.364 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:07:25.364 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "9c1c697f-8655-4ef3-8a13-da568d2cbe86", 2024-03-18T20:07:25.364 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-18T20:07:25.364 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 10422, 2024-03-18T20:07:25.364 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-18T20:07:25.364 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-18T20:07:25.364 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-18T20:07:25.364 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 6.0284105720000003, 2024-03-18T20:07:25.364 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 89, 2024-03-18T20:07:25.364 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 195, 2024-03-18T20:07:25.364 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 195, 2024-03-18T20:07:25.364 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 11.350226399 2024-03-18T20:07:25.364 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:07:25.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.353+0000 7fb3284aa700 1 -- 172.21.15.1:0/1571827458 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb30406adb0 msgr2=0x7fb30406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:25.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.353+0000 7fb3284aa700 1 --2- 172.21.15.1:0/1571827458 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb30406adb0 0x7fb30406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb310008910 tx=0x7fb310008040 comp rx=0 tx=0).stop 2024-03-18T20:07:25.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.353+0000 7fb3284aa700 1 -- 172.21.15.1:0/1571827458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb32015a380 msgr2=0x7fb3200a0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:25.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.353+0000 7fb3284aa700 1 --2- 172.21.15.1:0/1571827458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb32015a380 0x7fb3200a0970 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fb31800b970 tx=0x7fb31800bd30 comp rx=0 tx=0).stop 2024-03-18T20:07:25.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.353+0000 7fb3284aa700 1 -- 172.21.15.1:0/1571827458 shutdown_connections 2024-03-18T20:07:25.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.353+0000 7fb3284aa700 1 --2- 172.21.15.1:0/1571827458 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fb30406adb0 0x7fb30406d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.353+0000 7fb3284aa700 1 --2- 172.21.15.1:0/1571827458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb32015a380 0x7fb3200a0970 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.353+0000 7fb3284aa700 1 --2- 172.21.15.1:0/1571827458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb320071490 0x7fb3200aff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.353+0000 7fb3284aa700 1 --2- 172.21.15.1:0/1571827458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb3200724e0 0x7fb320154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.353+0000 7fb3284aa700 1 -- 172.21.15.1:0/1571827458 >> 172.21.15.1:0/1571827458 conn(0x7fb3200b1aa0 msgr2=0x7fb32006a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:25.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.353+0000 7fb3284aa700 1 -- 172.21.15.1:0/1571827458 shutdown_connections 2024-03-18T20:07:25.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.353+0000 7fb3284aa700 1 -- 172.21.15.1:0/1571827458 wait complete. 2024-03-18T20:07:25.370 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T20:07:25.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.490+0000 7fe893ddf700 1 Processor -- start 2024-03-18T20:07:25.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.490+0000 7fe893ddf700 1 -- start start 2024-03-18T20:07:25.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.491+0000 7fe893ddf700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe88c15b4d0 0x7fe88c09a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.491+0000 7fe893ddf700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe88c0cf750 0x7fe88c099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.491+0000 7fe893ddf700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe88c06f070 con 0x7fe88c099f80 2024-03-18T20:07:25.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.491+0000 7fe893ddf700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe88c09a3f0 con 0x7fe88c0cf750 2024-03-18T20:07:25.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.491+0000 7fe893ddf700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe88c0d0260 con 0x7fe88c15b4d0 2024-03-18T20:07:25.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.491+0000 7fe89137a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe88c0cf750 0x7fe88c099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.491+0000 7fe89137a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe88c0cf750 0x7fe88c099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49694/0 (socket says 172.21.15.1:49694) 2024-03-18T20:07:25.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.491+0000 7fe89137a700 1 -- 172.21.15.1:0/1722318266 learned_addr learned my addr 172.21.15.1:0/1722318266 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:25.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.491+0000 7fe891b7b700 1 --2- 172.21.15.1:0/1722318266 >> v2:172.21.15.165:3300/0 conn(0x7fe88c15b4d0 0x7fe88c09a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.491+0000 7fe89137a700 1 -- 172.21.15.1:0/1722318266 >> v2:172.21.15.165:3300/0 conn(0x7fe88c15b4d0 msgr2=0x7fe88c09a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:25.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.491+0000 7fe89137a700 1 --2- 172.21.15.1:0/1722318266 >> v2:172.21.15.165:3300/0 conn(0x7fe88c15b4d0 0x7fe88c09a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.491+0000 7fe89137a700 1 -- 172.21.15.1:0/1722318266 >> v1:172.21.15.165:6789/0 conn(0x7fe88c099f80 legacy=0x7fe88c0cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:25.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.491+0000 7fe89137a700 1 -- 172.21.15.1:0/1722318266 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88c0cff80 con 0x7fe88c0cf750 2024-03-18T20:07:25.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.492+0000 7fe89137a700 1 --2- 172.21.15.1:0/1722318266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe88c0cf750 0x7fe88c099a40 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fe88c071290 tx=0x7fe87c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f26202ae37ca537 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:25.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.492+0000 7fe890b79700 1 -- 172.21.15.1:0/1722318266 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c011830 con 0x7fe88c0cf750 2024-03-18T20:07:25.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.492+0000 7fe890b79700 1 -- 172.21.15.1:0/1722318266 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87c011990 con 0x7fe88c0cf750 2024-03-18T20:07:25.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.492+0000 7fe890b79700 1 -- 172.21.15.1:0/1722318266 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c011d20 con 0x7fe88c0cf750 2024-03-18T20:07:25.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.492+0000 7fe893ddf700 1 -- 172.21.15.1:0/1722318266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe88c0cf750 msgr2=0x7fe88c099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:25.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.492+0000 7fe893ddf700 1 --2- 172.21.15.1:0/1722318266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe88c0cf750 0x7fe88c099a40 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fe88c071290 tx=0x7fe87c009bd0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.492+0000 7fe893ddf700 1 -- 172.21.15.1:0/1722318266 shutdown_connections 2024-03-18T20:07:25.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.492+0000 7fe893ddf700 1 --2- 172.21.15.1:0/1722318266 >> v2:172.21.15.165:3300/0 conn(0x7fe88c15b4d0 0x7fe88c09a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.492+0000 7fe893ddf700 1 --2- 172.21.15.1:0/1722318266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe88c0cf750 0x7fe88c099a40 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.492+0000 7fe893ddf700 1 -- 172.21.15.1:0/1722318266 >> 172.21.15.1:0/1722318266 conn(0x7fe88c0b2cb0 msgr2=0x7fe88c0b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:25.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.492+0000 7fe893ddf700 1 -- 172.21.15.1:0/1722318266 shutdown_connections 2024-03-18T20:07:25.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.492+0000 7fe893ddf700 1 -- 172.21.15.1:0/1722318266 wait complete. 2024-03-18T20:07:25.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.494+0000 7fe893ddf700 1 Processor -- start 2024-03-18T20:07:25.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.494+0000 7fe893ddf700 1 -- start start 2024-03-18T20:07:25.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.494+0000 7fe893ddf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe88c099f80 0x7fe88c0c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.494+0000 7fe893ddf700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe88c0cf750 0x7fe88c15da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.494+0000 7fe893ddf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe88c15b4d0 0x7fe88c157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.494+0000 7fe893ddf700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88c06bc80 con 0x7fe88c099f80 2024-03-18T20:07:25.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.494+0000 7fe893ddf700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88c15df70 con 0x7fe88c0cf750 2024-03-18T20:07:25.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.494+0000 7fe893ddf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe88c15e240 con 0x7fe88c15b4d0 2024-03-18T20:07:25.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.494+0000 7fe891b7b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe88c099f80 0x7fe88c0c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.494+0000 7fe891b7b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe88c099f80 0x7fe88c0c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49704/0 (socket says 172.21.15.1:49704) 2024-03-18T20:07:25.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.495+0000 7fe891b7b700 1 -- 172.21.15.1:0/3752228867 learned_addr learned my addr 172.21.15.1:0/3752228867 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:25.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.495+0000 7fe89237c700 1 --2- 172.21.15.1:0/3752228867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe88c15b4d0 0x7fe88c157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.495+0000 7fe89137a700 1 --2- 172.21.15.1:0/3752228867 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe88c0cf750 0x7fe88c15da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.495+0000 7fe891b7b700 1 -- 172.21.15.1:0/3752228867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe88c15b4d0 msgr2=0x7fe88c157950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:25.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.495+0000 7fe891b7b700 1 --2- 172.21.15.1:0/3752228867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe88c15b4d0 0x7fe88c157950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.495+0000 7fe891b7b700 1 -- 172.21.15.1:0/3752228867 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe88c0cf750 msgr2=0x7fe88c15da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:25.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.495+0000 7fe891b7b700 1 --2- 172.21.15.1:0/3752228867 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe88c0cf750 0x7fe88c15da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.495+0000 7fe891b7b700 1 -- 172.21.15.1:0/3752228867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c009550 con 0x7fe88c099f80 2024-03-18T20:07:25.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.495+0000 7fe891b7b700 1 --2- 172.21.15.1:0/3752228867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe88c099f80 0x7fe88c0c7850 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fe88800e870 tx=0x7fe88800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:25.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.495+0000 7fe8827fc700 1 -- 172.21.15.1:0/3752228867 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe888009590 con 0x7fe88c099f80 2024-03-18T20:07:25.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.495+0000 7fe8827fc700 1 -- 172.21.15.1:0/3752228867 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe888013680 con 0x7fe88c099f80 2024-03-18T20:07:25.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.495+0000 7fe8827fc700 1 -- 172.21.15.1:0/3752228867 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8880139f0 con 0x7fe88c099f80 2024-03-18T20:07:25.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.495+0000 7fe893ddf700 1 -- 172.21.15.1:0/3752228867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe88c15e520 con 0x7fe88c099f80 2024-03-18T20:07:25.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.495+0000 7fe893ddf700 1 -- 172.21.15.1:0/3752228867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe88c06bed0 con 0x7fe88c099f80 2024-03-18T20:07:25.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.497+0000 7fe893ddf700 1 -- 172.21.15.1:0/3752228867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe88c06c240 con 0x7fe88c099f80 2024-03-18T20:07:25.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.497+0000 7fe8827fc700 1 -- 172.21.15.1:0/3752228867 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe888013b50 con 0x7fe88c099f80 2024-03-18T20:07:25.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.497+0000 7fe8827fc700 1 --2- 172.21.15.1:0/3752228867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe87006adb0 0x7fe87006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:25.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.497+0000 7fe8827fc700 1 -- 172.21.15.1:0/3752228867 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe888089480 con 0x7fe88c099f80 2024-03-18T20:07:25.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.497+0000 7fe89137a700 1 --2- 172.21.15.1:0/3752228867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe87006adb0 0x7fe87006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:25.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.498+0000 7fe89137a700 1 --2- 172.21.15.1:0/3752228867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe87006adb0 0x7fe87006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009c10 tx=0x7fe87c008030 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:25.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.502+0000 7fe8827fc700 1 -- 172.21.15.1:0/3752228867 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe888054450 con 0x7fe88c099f80 2024-03-18T20:07:25.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.749+0000 7fe893ddf700 1 -- 172.21.15.1:0/3752228867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe88c068240 con 0x7fe88c099f80 2024-03-18T20:07:25.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.750+0000 7fe8827fc700 1 -- 172.21.15.1:0/3752228867 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v89) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe88805ce60 con 0x7fe88c099f80 2024-03-18T20:07:25.754 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T20:07:25.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.752+0000 7fe893ddf700 1 -- 172.21.15.1:0/3752228867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe87006adb0 msgr2=0x7fe87006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.752+0000 7fe893ddf700 1 --2- 172.21.15.1:0/3752228867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe87006adb0 0x7fe87006d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009c10 tx=0x7fe87c008030 comp rx=0 tx=0).stop 2024-03-18T20:07:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.752+0000 7fe893ddf700 1 -- 172.21.15.1:0/3752228867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe88c099f80 msgr2=0x7fe88c0c7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.752+0000 7fe893ddf700 1 --2- 172.21.15.1:0/3752228867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe88c099f80 0x7fe88c0c7850 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fe88800e870 tx=0x7fe88800eb80 comp rx=0 tx=0).stop 2024-03-18T20:07:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.752+0000 7fe893ddf700 1 -- 172.21.15.1:0/3752228867 shutdown_connections 2024-03-18T20:07:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.752+0000 7fe893ddf700 1 --2- 172.21.15.1:0/3752228867 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe87006adb0 0x7fe87006d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.752+0000 7fe893ddf700 1 --2- 172.21.15.1:0/3752228867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe88c15b4d0 0x7fe88c157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.752+0000 7fe893ddf700 1 --2- 172.21.15.1:0/3752228867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe88c099f80 0x7fe88c0c7850 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.752+0000 7fe893ddf700 1 --2- 172.21.15.1:0/3752228867 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe88c0cf750 0x7fe88c15da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.752+0000 7fe893ddf700 1 -- 172.21.15.1:0/3752228867 >> 172.21.15.1:0/3752228867 conn(0x7fe88c0b2cb0 msgr2=0x7fe88c0afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.752+0000 7fe893ddf700 1 -- 172.21.15.1:0/3752228867 shutdown_connections 2024-03-18T20:07:25.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:25.752+0000 7fe893ddf700 1 -- 172.21.15.1:0/3752228867 wait complete. 2024-03-18T20:07:25.768 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T20:07:25.769 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T20:07:25.769 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-18T20:07:25.769 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T20:07:25.769 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T20:07:25.769 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T20:07:25.769 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T20:07:25.769 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T20:07:25.769 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:07:25.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:07:25.769 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-18T20:07:25.788 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T20:07:25.788 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T20:07:25.788 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T20:07:25.788 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:07:25.789 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T20:07:25.789 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:07:25.789 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:07:25.789 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-18T20:07:25.789 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-18T20:07:25.789 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:07:25.789 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-18T20:07:25.789 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-18T20:07:25.789 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T20:07:25.789 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:07:25.789 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-18T20:07:25.789 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T20:07:25.789 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 37924sec preferred_lft 37924sec 2024-03-18T20:07:25.789 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link 2024-03-18T20:07:25.790 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:07:25.790 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:07:25.790 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:07:25.790 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-18T20:07:25.790 INFO:teuthology.orchestra.run.smithi001.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T20:07:25.790 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:07:25.790 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T20:07:25.790 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:07:25.790 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::4ca9:f5ff:feb3:79a3/64 scope link 2024-03-18T20:07:25.790 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:07:25.790 INFO:teuthology.orchestra.run.smithi001.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T20:07:25.790 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T20:07:25.790 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::282f:a4ff:fe1a:9908/64 scope link 2024-03-18T20:07:25.790 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:07:25.791 INFO:teuthology.orchestra.run.smithi001.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T20:07:25.791 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2e:5c:60:78:b2:8c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T20:07:25.791 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::2c5c:60ff:fe78:b28c/64 scope link 2024-03-18T20:07:25.791 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:07:25.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:07:25.791 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T20:07:25.854 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-18T20:07:25.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:07:25.856 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-18T20:07:25.917 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-18T20:07:25.917 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T20:07:25.917 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T20:07:25.917 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T20:07:25.917 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T20:07:25.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:07:25.918 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T20:07:25.918 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T20:07:25.918 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T20:07:25.918 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T20:07:25.918 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T20:07:25.918 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T20:07:26.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:07:26.024 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T20:07:26.042 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T20:07:26.042 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T20:07:26.042 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:07:26.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:07:26.043 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-18T20:07:26.111 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T20:07:26.111 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:07:26.111 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T20:07:26.170 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:07:26.170 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:07:26.170 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T20:07:26.227 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T20:07:26.295 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:07:26.295 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:07:26.364 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:07:26.364 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:07:26.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:07:26.365 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T20:07:26.424 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T20:07:26.424 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T20:07:26.466 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T20:07:26.494 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:07:26.494 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:07:26.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-18T20:07:26.550+0000 7f73c28895c0 -1 init, newargv = 0x55f580c7c030 newargc=15 2024-03-18T20:07:26.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[146826]: starting ceph client 2024-03-18T20:07:26.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[146826]: starting fuse 2024-03-18T20:07:26.564 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:07:26.564 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:07:26.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:07:26.565 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T20:07:26.625 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T20:07:26.625 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-03-18T20:07:26.625 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T20:07:26.625 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-18T20:07:26.625 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-18T20:07:26.625 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-18T20:07:26.625 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-18T20:07:26.625 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:07:26.626 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-18T20:07:26.626 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T20:07:26.626 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-18T20:07:26.626 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T20:07:26.626 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:07:26.626 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-18T20:07:26.626 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-18T20:07:26.626 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-18T20:07:26.626 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-18T20:07:26.626 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:07:26.626 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-18T20:07:26.626 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T20:07:26.626 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T20:07:26.626 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T20:07:26.626 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-18T20:07:26.627 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-18T20:07:26.627 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-18T20:07:26.627 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T20:07:26.627 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:07:26.627 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-18T20:07:26.627 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-18T20:07:26.749 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.146826.asok 2024-03-18T20:07:26.931 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.146826.asok 2024-03-18T20:07:26.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:07:26.932 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.146826.asok status 2024-03-18T20:07:27.094 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:07:27.094 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-18T20:07:27.094 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T20:07:27.094 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T20:07:27.095 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-18T20:07:27.095 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-18T20:07:27.095 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T20:07:27.095 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "146826", 2024-03-18T20:07:27.095 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-18T20:07:27.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:27.095 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-18T20:07:27.095 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-18T20:07:27.095 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 10581, 2024-03-18T20:07:27.095 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-18T20:07:27.095 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-18T20:07:27.095 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-18T20:07:27.095 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 10581 2024-03-18T20:07:27.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:27.096 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T20:07:27.096 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T20:07:27.096 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:07:27.096 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1899754441 2024-03-18T20:07:27.096 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:27.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:27.096 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T20:07:27.096 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T20:07:27.096 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:07:27.096 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1899754441 2024-03-18T20:07:27.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:07:27.096 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.10581 192.168.0.1:0/1899754441", 2024-03-18T20:07:27.096 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/1899754441", 2024-03-18T20:07:27.096 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-18T20:07:27.097 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 89, 2024-03-18T20:07:27.097 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 195, 2024-03-18T20:07:27.097 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-18T20:07:27.097 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-18T20:07:27.097 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-18T20:07:27.097 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:07:27.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:07:27.104 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-18T20:07:27.104 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T20:07:27.123 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-18T20:07:27.123 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T20:07:27.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:07:27.123 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T20:07:27.192 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-18T20:07:27.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.355+0000 7fea04f14700 1 Processor -- start 2024-03-18T20:07:27.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.355+0000 7fea04f14700 1 -- start start 2024-03-18T20:07:27.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.356+0000 7fea04f14700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea00144bb0 0x7fea0015b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:27.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.356+0000 7fea04f14700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fea000cefe0 0x7fea000ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:27.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.356+0000 7fea04f14700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea000cd330 con 0x7fea000cc990 2024-03-18T20:07:27.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.356+0000 7fea04f14700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea0006eb50 con 0x7fea000cefe0 2024-03-18T20:07:27.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.356+0000 7fea04f14700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea00144f90 con 0x7fea00144bb0 2024-03-18T20:07:27.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.356+0000 7fe9fed9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fea000cefe0 0x7fea000ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:27.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.356+0000 7fe9fdd9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fea000cc990 0x7fea000ce8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34012/0 (socket says 172.21.15.1:34012) 2024-03-18T20:07:27.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.356+0000 7fe9fdd9b700 1 -- 172.21.15.1:0/3585331097 learned_addr learned my addr 172.21.15.1:0/3585331097 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:27.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.356+0000 7fe9fe59c700 1 --2- 172.21.15.1:0/3585331097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea00144bb0 0x7fea0015b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:27.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.356+0000 7fe9fed9d700 1 -- 172.21.15.1:0/3585331097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea00144bb0 msgr2=0x7fea0015b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:27.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.356+0000 7fe9fed9d700 1 --2- 172.21.15.1:0/3585331097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea00144bb0 0x7fea0015b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:27.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.356+0000 7fe9fed9d700 1 -- 172.21.15.1:0/3585331097 >> v1:172.21.15.1:6789/0 conn(0x7fea000cc990 legacy=0x7fea000ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:27.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.356+0000 7fe9fed9d700 1 -- 172.21.15.1:0/3585331097 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea0006edc0 con 0x7fea000cefe0 2024-03-18T20:07:27.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.356+0000 7fe9fd59a700 1 -- 172.21.15.1:0/3585331097 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199247128 0 0) 0x7fea000cd330 con 0x7fea000cc990 2024-03-18T20:07:27.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.357+0000 7fe9fed9d700 1 --2- 172.21.15.1:0/3585331097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea000cefe0 0x7fea000ccdf0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8009d90 tx=0x7fe9f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c433cf843a2de1e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:27.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.357+0000 7fe9fd59a700 1 -- 172.21.15.1:0/3585331097 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f800e070 con 0x7fea000cefe0 2024-03-18T20:07:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.357+0000 7fe9fd59a700 1 -- 172.21.15.1:0/3585331097 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f800bb20 con 0x7fea000cefe0 2024-03-18T20:07:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.357+0000 7fe9fd59a700 1 -- 172.21.15.1:0/3585331097 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8012430 con 0x7fea000cefe0 2024-03-18T20:07:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.357+0000 7fea04f14700 1 -- 172.21.15.1:0/3585331097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea000cefe0 msgr2=0x7fea000ccdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.357+0000 7fea04f14700 1 --2- 172.21.15.1:0/3585331097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea000cefe0 0x7fea000ccdf0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8009d90 tx=0x7fe9f800b330 comp rx=0 tx=0).stop 2024-03-18T20:07:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.357+0000 7fea04f14700 1 -- 172.21.15.1:0/3585331097 shutdown_connections 2024-03-18T20:07:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.357+0000 7fea04f14700 1 --2- 172.21.15.1:0/3585331097 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea000cefe0 0x7fea000ccdf0 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.357+0000 7fea04f14700 1 --2- 172.21.15.1:0/3585331097 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea00144bb0 0x7fea0015b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.357+0000 7fea04f14700 1 -- 172.21.15.1:0/3585331097 >> 172.21.15.1:0/3585331097 conn(0x7fea000b1cb0 msgr2=0x7fea000b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.357+0000 7fea04f14700 1 -- 172.21.15.1:0/3585331097 shutdown_connections 2024-03-18T20:07:27.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.357+0000 7fea04f14700 1 -- 172.21.15.1:0/3585331097 wait complete. 2024-03-18T20:07:27.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.359+0000 7fea04f14700 1 Processor -- start 2024-03-18T20:07:27.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.359+0000 7fea04f14700 1 -- start start 2024-03-18T20:07:27.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fea04f14700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea000cc990 0x7fea000cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:27.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fea04f14700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea000cefe0 0x7fea000c9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:27.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fea04f14700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea00144bb0 0x7fea000c9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:27.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fea04f14700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea000b3a70 con 0x7fea000cefe0 2024-03-18T20:07:27.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fea04f14700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea000cbe20 con 0x7fea000cc990 2024-03-18T20:07:27.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fea04f14700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea0015aad0 con 0x7fea00144bb0 2024-03-18T20:07:27.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fe9fdd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea000cefe0 0x7fea000c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:27.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fe9fdd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea000cefe0 0x7fea000c9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49732/0 (socket says 172.21.15.1:49732) 2024-03-18T20:07:27.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fe9fdd9b700 1 -- 172.21.15.1:0/1459675462 learned_addr learned my addr 172.21.15.1:0/1459675462 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:27.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fe9fed9d700 1 --2- 172.21.15.1:0/1459675462 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea00144bb0 0x7fea000c9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:27.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fe9fdd9b700 1 -- 172.21.15.1:0/1459675462 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea00144bb0 msgr2=0x7fea000c9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:27.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fe9fdd9b700 1 --2- 172.21.15.1:0/1459675462 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea00144bb0 0x7fea000c9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:27.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fe9fe59c700 1 --2- 172.21.15.1:0/1459675462 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea000cc990 0x7fea000cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:27.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fe9fdd9b700 1 -- 172.21.15.1:0/1459675462 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea000cc990 msgr2=0x7fea000cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:27.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fe9fdd9b700 1 --2- 172.21.15.1:0/1459675462 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea000cc990 0x7fea000cb8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:27.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fe9fdd9b700 1 -- 172.21.15.1:0/1459675462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f8009a70 con 0x7fea000cefe0 2024-03-18T20:07:27.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fe9fed9d700 1 --2- 172.21.15.1:0/1459675462 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea00144bb0 0x7fea000c9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:27.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fe9fdd9b700 1 --2- 172.21.15.1:0/1459675462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea000cefe0 0x7fea000c9330 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fe9f40011e0 tx=0x7fe9f400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:27.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fe9eeffd700 1 -- 172.21.15.1:0/1459675462 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f4004c80 con 0x7fea000cefe0 2024-03-18T20:07:27.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fe9eeffd700 1 -- 172.21.15.1:0/1459675462 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f4004e00 con 0x7fea000cefe0 2024-03-18T20:07:27.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fe9eeffd700 1 -- 172.21.15.1:0/1459675462 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f4007310 con 0x7fea000cefe0 2024-03-18T20:07:27.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.360+0000 7fea04f14700 1 -- 172.21.15.1:0/1459675462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea000b3c60 con 0x7fea000cefe0 2024-03-18T20:07:27.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.361+0000 7fea04f14700 1 -- 172.21.15.1:0/1459675462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea0013ab50 con 0x7fea000cefe0 2024-03-18T20:07:27.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.362+0000 7fea04f14700 1 -- 172.21.15.1:0/1459675462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea0006efe0 con 0x7fea000cefe0 2024-03-18T20:07:27.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.362+0000 7fe9eeffd700 1 -- 172.21.15.1:0/1459675462 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe9f4007470 con 0x7fea000cefe0 2024-03-18T20:07:27.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.362+0000 7fe9eeffd700 1 --2- 172.21.15.1:0/1459675462 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe9e006ade0 0x7fe9e006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:27.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.362+0000 7fe9eeffd700 1 -- 172.21.15.1:0/1459675462 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe9f4089350 con 0x7fea000cefe0 2024-03-18T20:07:27.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.363+0000 7fe9fe59c700 1 --2- 172.21.15.1:0/1459675462 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe9e006ade0 0x7fe9e006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:27.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.363+0000 7fe9fe59c700 1 --2- 172.21.15.1:0/1459675462 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe9e006ade0 0x7fe9e006d2a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0009a10 tx=0x7fe9f0008040 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:27.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.367+0000 7fe9eeffd700 1 -- 172.21.15.1:0/1459675462 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe9f4054320 con 0x7fea000cefe0 2024-03-18T20:07:27.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.600+0000 7fea04f14700 1 -- 172.21.15.1:0/1459675462 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fea000cf870 con 0x7fe9e006ade0 2024-03-18T20:07:27.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.601+0000 7fe9eeffd700 1 -- 172.21.15.1:0/1459675462 <== mgr.10200 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe9f0009660 con 0x7fe9e006ade0 2024-03-18T20:07:27.606 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T20:07:27.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:07:27.606 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephfs" 2024-03-18T20:07:27.606 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:07:27.606 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T20:07:27.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.603+0000 7fea04f14700 1 -- 172.21.15.1:0/1459675462 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe9e006ade0 msgr2=0x7fe9e006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:27.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.603+0000 7fea04f14700 1 --2- 172.21.15.1:0/1459675462 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe9e006ade0 0x7fe9e006d2a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0009a10 tx=0x7fe9f0008040 comp rx=0 tx=0).stop 2024-03-18T20:07:27.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.603+0000 7fea04f14700 1 -- 172.21.15.1:0/1459675462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea000cefe0 msgr2=0x7fea000c9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:27.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.603+0000 7fea04f14700 1 --2- 172.21.15.1:0/1459675462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea000cefe0 0x7fea000c9330 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fe9f40011e0 tx=0x7fe9f400cae0 comp rx=0 tx=0).stop 2024-03-18T20:07:27.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.603+0000 7fea04f14700 1 -- 172.21.15.1:0/1459675462 shutdown_connections 2024-03-18T20:07:27.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.603+0000 7fea04f14700 1 --2- 172.21.15.1:0/1459675462 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7fe9e006ade0 0x7fe9e006d2a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:27.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.603+0000 7fea04f14700 1 --2- 172.21.15.1:0/1459675462 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea00144bb0 0x7fea000c9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:27.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.603+0000 7fea04f14700 1 --2- 172.21.15.1:0/1459675462 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea000cc990 0x7fea000cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:27.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.603+0000 7fea04f14700 1 --2- 172.21.15.1:0/1459675462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea000cefe0 0x7fea000c9330 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:27.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.603+0000 7fea04f14700 1 -- 172.21.15.1:0/1459675462 >> 172.21.15.1:0/1459675462 conn(0x7fea000b1cb0 msgr2=0x7fea000b08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:27.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.603+0000 7fea04f14700 1 -- 172.21.15.1:0/1459675462 shutdown_connections 2024-03-18T20:07:27.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.603+0000 7fea04f14700 1 -- 172.21.15.1:0/1459675462 wait complete. 2024-03-18T20:07:27.620 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-18T20:07:27.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.744+0000 7f2e3f857700 1 Processor -- start 2024-03-18T20:07:27.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.744+0000 7f2e3f857700 1 -- start start 2024-03-18T20:07:27.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e3f857700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2e38160a70 0x7f2e380d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:27.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e3f857700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e38160e50 0x7f2e380cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:27.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e3f857700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e381612d0 0x7f2e381616f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:27.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e3f857700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e38162170 con 0x7f2e38160e50 2024-03-18T20:07:27.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e3f857700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e38161c30 con 0x7f2e38160a70 2024-03-18T20:07:27.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e3f857700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e380cef30 con 0x7f2e381612d0 2024-03-18T20:07:27.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e3cdf2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e38160e50 0x7f2e380cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:27.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e3cdf2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e38160e50 0x7f2e380cffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49744/0 (socket says 172.21.15.1:49744) 2024-03-18T20:07:27.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e3cdf2700 1 -- 172.21.15.1:0/3424077273 learned_addr learned my addr 172.21.15.1:0/3424077273 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:27.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e3d5f3700 1 --2- 172.21.15.1:0/3424077273 >> v2:172.21.15.165:3300/0 conn(0x7f2e38160a70 0x7f2e380d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:27.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e3ddf4700 1 --2- 172.21.15.1:0/3424077273 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e381612d0 0x7f2e381616f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:27.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e3cdf2700 1 -- 172.21.15.1:0/3424077273 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e381612d0 msgr2=0x7f2e381616f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:27.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e3cdf2700 1 --2- 172.21.15.1:0/3424077273 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e381612d0 0x7f2e381616f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:27.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e3cdf2700 1 -- 172.21.15.1:0/3424077273 >> v2:172.21.15.165:3300/0 conn(0x7f2e38160a70 msgr2=0x7f2e380d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:27.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e3cdf2700 1 --2- 172.21.15.1:0/3424077273 >> v2:172.21.15.165:3300/0 conn(0x7f2e38160a70 0x7f2e380d3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:27.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e3cdf2700 1 -- 172.21.15.1:0/3424077273 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e380cf210 con 0x7f2e38160e50 2024-03-18T20:07:27.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e3ddf4700 1 --2- 172.21.15.1:0/3424077273 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e381612d0 0x7f2e381616f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:27.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e3d5f3700 1 --2- 172.21.15.1:0/3424077273 >> v2:172.21.15.165:3300/0 conn(0x7f2e38160a70 0x7f2e380d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:27.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e3cdf2700 1 --2- 172.21.15.1:0/3424077273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38160e50 0x7f2e380cffd0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f2e3808d8f0 tx=0x7f2e28009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501dab1bd72422df server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:27.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e2ffff700 1 -- 172.21.15.1:0/3424077273 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2800b870 con 0x7f2e38160e50 2024-03-18T20:07:27.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.745+0000 7f2e2ffff700 1 -- 172.21.15.1:0/3424077273 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2800b9d0 con 0x7f2e38160e50 2024-03-18T20:07:27.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.746+0000 7f2e2ffff700 1 -- 172.21.15.1:0/3424077273 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2800bd60 con 0x7f2e38160e50 2024-03-18T20:07:27.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.746+0000 7f2e3f857700 1 -- 172.21.15.1:0/3424077273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38160e50 msgr2=0x7f2e380cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:27.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.746+0000 7f2e3f857700 1 --2- 172.21.15.1:0/3424077273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38160e50 0x7f2e380cffd0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f2e3808d8f0 tx=0x7f2e28009bd0 comp rx=0 tx=0).stop 2024-03-18T20:07:27.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.746+0000 7f2e3f857700 1 -- 172.21.15.1:0/3424077273 shutdown_connections 2024-03-18T20:07:27.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.746+0000 7f2e3f857700 1 --2- 172.21.15.1:0/3424077273 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e381612d0 0x7f2e381616f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:27.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.746+0000 7f2e3f857700 1 --2- 172.21.15.1:0/3424077273 >> v2:172.21.15.165:3300/0 conn(0x7f2e38160a70 0x7f2e380d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:27.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.746+0000 7f2e3f857700 1 --2- 172.21.15.1:0/3424077273 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38160e50 0x7f2e380cffd0 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:27.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.746+0000 7f2e3f857700 1 -- 172.21.15.1:0/3424077273 >> 172.21.15.1:0/3424077273 conn(0x7f2e3809c560 msgr2=0x7f2e380d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:27.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.746+0000 7f2e3f857700 1 -- 172.21.15.1:0/3424077273 shutdown_connections 2024-03-18T20:07:27.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.746+0000 7f2e3f857700 1 -- 172.21.15.1:0/3424077273 wait complete. 2024-03-18T20:07:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.748+0000 7f2e3f857700 1 Processor -- start 2024-03-18T20:07:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.748+0000 7f2e3f857700 1 -- start start 2024-03-18T20:07:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e3f857700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38160a70 0x7f2e38151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e3f857700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e38160e50 0x7f2e3815b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e3f857700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e381612d0 0x7f2e380cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e3f857700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3815a0e0 con 0x7f2e38160a70 2024-03-18T20:07:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e3f857700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3815a3c0 con 0x7f2e38160e50 2024-03-18T20:07:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e3f857700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3815a6a0 con 0x7f2e381612d0 2024-03-18T20:07:27.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e3d5f3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38160a70 0x7f2e38151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:27.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e3d5f3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38160a70 0x7f2e38151050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49760/0 (socket says 172.21.15.1:49760) 2024-03-18T20:07:27.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e3d5f3700 1 -- 172.21.15.1:0/2235558768 learned_addr learned my addr 172.21.15.1:0/2235558768 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:27.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e3cdf2700 1 --2- 172.21.15.1:0/2235558768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e38160e50 0x7f2e3815b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:27.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e3d5f3700 1 -- 172.21.15.1:0/2235558768 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e381612d0 msgr2=0x7f2e380cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:27.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e3ddf4700 1 --2- 172.21.15.1:0/2235558768 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e381612d0 0x7f2e380cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:27.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e3d5f3700 1 --2- 172.21.15.1:0/2235558768 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e381612d0 0x7f2e380cc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:27.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e3d5f3700 1 -- 172.21.15.1:0/2235558768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e38160e50 msgr2=0x7f2e3815b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:27.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e3d5f3700 1 --2- 172.21.15.1:0/2235558768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e38160e50 0x7f2e3815b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:27.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e3d5f3700 1 -- 172.21.15.1:0/2235558768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e28009550 con 0x7f2e38160a70 2024-03-18T20:07:27.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e3cdf2700 1 --2- 172.21.15.1:0/2235558768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e38160e50 0x7f2e3815b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:07:27.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e3d5f3700 1 --2- 172.21.15.1:0/2235558768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38160a70 0x7f2e38151050 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400c7f0 tx=0x7f2e3400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:27.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e2dffb700 1 -- 172.21.15.1:0/2235558768 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e34011800 con 0x7f2e38160a70 2024-03-18T20:07:27.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e2dffb700 1 -- 172.21.15.1:0/2235558768 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e34011960 con 0x7f2e38160a70 2024-03-18T20:07:27.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.749+0000 7f2e3f857700 1 -- 172.21.15.1:0/2235558768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e3815a980 con 0x7f2e38160a70 2024-03-18T20:07:27.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.750+0000 7f2e2dffb700 1 -- 172.21.15.1:0/2235558768 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e34011d10 con 0x7f2e38160a70 2024-03-18T20:07:27.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.750+0000 7f2e3f857700 1 -- 172.21.15.1:0/2235558768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e3815b6c0 con 0x7f2e38160a70 2024-03-18T20:07:27.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.751+0000 7f2e3f857700 1 -- 172.21.15.1:0/2235558768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e38067390 con 0x7f2e38160a70 2024-03-18T20:07:27.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.751+0000 7f2e2dffb700 1 -- 172.21.15.1:0/2235558768 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 104) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2e34018360 con 0x7f2e38160a70 2024-03-18T20:07:27.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.752+0000 7f2e2dffb700 1 --2- 172.21.15.1:0/2235558768 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2e1c06adb0 0x7f2e1c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:27.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.752+0000 7f2e3cdf2700 1 --2- 172.21.15.1:0/2235558768 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2e1c06adb0 0x7f2e1c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:27.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.752+0000 7f2e2dffb700 1 -- 172.21.15.1:0/2235558768 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2e34089710 con 0x7f2e38160a70 2024-03-18T20:07:27.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.752+0000 7f2e3cdf2700 1 --2- 172.21.15.1:0/2235558768 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2e1c06adb0 0x7f2e1c06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2e3808d8f0 tx=0x7f2e28008030 comp rx=0 tx=0).ready entity=mgr.10200 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:27.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:27.756+0000 7f2e2dffb700 1 -- 172.21.15.1:0/2235558768 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2e340544b0 con 0x7f2e38160a70 2024-03-18T20:07:28.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:28.019+0000 7f2e3f857700 1 -- 172.21.15.1:0/2235558768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f2e3815b8b0 con 0x7f2e38160a70 2024-03-18T20:07:28.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:28.865+0000 7f2e2dffb700 1 -- 172.21.15.1:0/2235558768 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 105) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f2e34087aa0 con 0x7f2e38160a70 2024-03-18T20:07:28.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:28.867+0000 7f2e2dffb700 1 -- 172.21.15.1:0/2235558768 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v105) v1 ==== 92+0+0 (secure 0 0 0) 0x7f2e34070f60 con 0x7f2e38160a70 2024-03-18T20:07:28.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:28.870+0000 7f2e3f857700 1 -- 172.21.15.1:0/2235558768 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2e1c06adb0 msgr2=0x7f2e1c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:28.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:28.870+0000 7f2e3f857700 1 --2- 172.21.15.1:0/2235558768 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2e1c06adb0 0x7f2e1c06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2e3808d8f0 tx=0x7f2e28008030 comp rx=0 tx=0).stop 2024-03-18T20:07:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:28.870+0000 7f2e3f857700 1 -- 172.21.15.1:0/2235558768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38160a70 msgr2=0x7f2e38151050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:28.870+0000 7f2e3f857700 1 --2- 172.21.15.1:0/2235558768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38160a70 0x7f2e38151050 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400c7f0 tx=0x7f2e3400cbb0 comp rx=0 tx=0).stop 2024-03-18T20:07:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:28.870+0000 7f2e3f857700 1 -- 172.21.15.1:0/2235558768 shutdown_connections 2024-03-18T20:07:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:28.870+0000 7f2e3f857700 1 --2- 172.21.15.1:0/2235558768 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f2e1c06adb0 0x7f2e1c06d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:28.870+0000 7f2e3f857700 1 --2- 172.21.15.1:0/2235558768 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e381612d0 0x7f2e380cc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:28.870+0000 7f2e3f857700 1 --2- 172.21.15.1:0/2235558768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38160a70 0x7f2e38151050 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:28.870+0000 7f2e3f857700 1 --2- 172.21.15.1:0/2235558768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e38160e50 0x7f2e3815b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:28.870+0000 7f2e3f857700 1 -- 172.21.15.1:0/2235558768 >> 172.21.15.1:0/2235558768 conn(0x7f2e3809c560 msgr2=0x7f2e380d3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:28.870+0000 7f2e3f857700 1 -- 172.21.15.1:0/2235558768 shutdown_connections 2024-03-18T20:07:28.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:28.870+0000 7f2e3f857700 1 -- 172.21.15.1:0/2235558768 wait complete. 2024-03-18T20:07:28.891 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-18T20:07:29.005 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:29.000+0000 7ff2fadc1240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:07:29.027 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:29.022+0000 7f7966304240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:07:29.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.024+0000 7f8e806bd700 1 Processor -- start 2024-03-18T20:07:29.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.024+0000 7f8e806bd700 1 -- start start 2024-03-18T20:07:29.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.025+0000 7f8e806bd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8e7806f180 0x7f8e78161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:29.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.025+0000 7f8e806bd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8e7806a750 0x7f8e7806ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:29.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.025+0000 7f8e806bd700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7806b0a0 con 0x7f8e7806eda0 2024-03-18T20:07:29.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.025+0000 7f8e806bd700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e780cb320 con 0x7f8e7806a750 2024-03-18T20:07:29.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.025+0000 7f8e806bd700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e780ce010 con 0x7f8e7806f180 2024-03-18T20:07:29.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.025+0000 7f8e7ec5a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8e7806a750 0x7f8e7806ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:29.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.025+0000 7f8e7ec5a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8e7806a750 0x7f8e7806ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49792/0 (socket says 172.21.15.1:49792) 2024-03-18T20:07:29.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.025+0000 7f8e7ec5a700 1 -- 172.21.15.1:0/4062465175 learned_addr learned my addr 172.21.15.1:0/4062465175 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:29.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.025+0000 7f8e7dc58700 1 --2- 172.21.15.1:0/4062465175 >> v2:172.21.15.165:3300/0 conn(0x7f8e7806f180 0x7f8e78161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:29.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.026+0000 7f8e7ec5a700 1 -- 172.21.15.1:0/4062465175 >> v2:172.21.15.165:3300/0 conn(0x7f8e7806f180 msgr2=0x7f8e78161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:29.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.026+0000 7f8e7ec5a700 1 --2- 172.21.15.1:0/4062465175 >> v2:172.21.15.165:3300/0 conn(0x7f8e7806f180 0x7f8e78161c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.026+0000 7f8e7ec5a700 1 -- 172.21.15.1:0/4062465175 >> v1:172.21.15.165:6789/0 conn(0x7f8e7806eda0 legacy=0x7f8e780ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:29.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.026+0000 7f8e7ec5a700 1 -- 172.21.15.1:0/4062465175 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e780cad60 con 0x7f8e7806a750 2024-03-18T20:07:29.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.026+0000 7f8e7ec5a700 1 --2- 172.21.15.1:0/4062465175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7806a750 0x7f8e7806ab60 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f8e780b19d0 tx=0x7f8e7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1f116a82e4c3e75 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:29.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.026+0000 7f8e7d457700 1 -- 172.21.15.1:0/4062465175 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7000e070 con 0x7f8e7806a750 2024-03-18T20:07:29.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.026+0000 7f8e7d457700 1 -- 172.21.15.1:0/4062465175 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7000b910 con 0x7f8e7806a750 2024-03-18T20:07:29.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.026+0000 7f8e7d457700 1 -- 172.21.15.1:0/4062465175 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7000bcc0 con 0x7f8e7806a750 2024-03-18T20:07:29.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.027+0000 7f8e806bd700 1 -- 172.21.15.1:0/4062465175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7806a750 msgr2=0x7f8e7806ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:29.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.027+0000 7f8e806bd700 1 --2- 172.21.15.1:0/4062465175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7806a750 0x7f8e7806ab60 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f8e780b19d0 tx=0x7f8e7000b330 comp rx=0 tx=0).stop 2024-03-18T20:07:29.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.027+0000 7f8e806bd700 1 -- 172.21.15.1:0/4062465175 shutdown_connections 2024-03-18T20:07:29.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.027+0000 7f8e806bd700 1 --2- 172.21.15.1:0/4062465175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7806a750 0x7f8e7806ab60 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.027+0000 7f8e806bd700 1 --2- 172.21.15.1:0/4062465175 >> v2:172.21.15.165:3300/0 conn(0x7f8e7806f180 0x7f8e78161c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.027+0000 7f8e806bd700 1 -- 172.21.15.1:0/4062465175 >> 172.21.15.1:0/4062465175 conn(0x7f8e780cff30 msgr2=0x7f8e780d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:29.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.027+0000 7f8e806bd700 1 -- 172.21.15.1:0/4062465175 shutdown_connections 2024-03-18T20:07:29.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.027+0000 7f8e806bd700 1 -- 172.21.15.1:0/4062465175 wait complete. 2024-03-18T20:07:29.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.028+0000 7f8e806bd700 1 Processor -- start 2024-03-18T20:07:29.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.028+0000 7f8e806bd700 1 -- start start 2024-03-18T20:07:29.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e806bd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7806a750 0x7f8e78153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:29.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e806bd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e7806eda0 0x7f8e780c33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:29.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e806bd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e7806f180 0x7f8e780cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:29.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e806bd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7815d230 con 0x7f8e7806a750 2024-03-18T20:07:29.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e806bd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e780b59d0 con 0x7f8e7806eda0 2024-03-18T20:07:29.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e806bd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e781f7450 con 0x7f8e7806f180 2024-03-18T20:07:29.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e7e459700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7806a750 0x7f8e78153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:29.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e7e459700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7806a750 0x7f8e78153500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49804/0 (socket says 172.21.15.1:49804) 2024-03-18T20:07:29.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e7e459700 1 -- 172.21.15.1:0/2995123750 learned_addr learned my addr 172.21.15.1:0/2995123750 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:29.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e7dc58700 1 --2- 172.21.15.1:0/2995123750 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e7806eda0 0x7f8e780c33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:29.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e7ec5a700 1 --2- 172.21.15.1:0/2995123750 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e7806f180 0x7f8e780cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:29.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e7e459700 1 -- 172.21.15.1:0/2995123750 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e7806f180 msgr2=0x7f8e780cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:29.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e7e459700 1 --2- 172.21.15.1:0/2995123750 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e7806f180 0x7f8e780cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e7e459700 1 -- 172.21.15.1:0/2995123750 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e7806eda0 msgr2=0x7f8e780c33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:29.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e7e459700 1 --2- 172.21.15.1:0/2995123750 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e7806eda0 0x7f8e780c33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e7e459700 1 -- 172.21.15.1:0/2995123750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e70009b30 con 0x7f8e7806a750 2024-03-18T20:07:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e7e459700 1 --2- 172.21.15.1:0/2995123750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7806a750 0x7f8e78153500 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f8e6800c4e0 tx=0x7f8e6800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e6effd700 1 -- 172.21.15.1:0/2995123750 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e680052d0 con 0x7f8e7806a750 2024-03-18T20:07:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e6effd700 1 -- 172.21.15.1:0/2995123750 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e68005430 con 0x7f8e7806a750 2024-03-18T20:07:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e806bd700 1 -- 172.21.15.1:0/2995123750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e781f7730 con 0x7f8e7806a750 2024-03-18T20:07:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e6effd700 1 -- 172.21.15.1:0/2995123750 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e680038c0 con 0x7f8e7806a750 2024-03-18T20:07:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.029+0000 7f8e806bd700 1 -- 172.21.15.1:0/2995123750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e780b5660 con 0x7f8e7806a750 2024-03-18T20:07:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.031+0000 7f8e806bd700 1 -- 172.21.15.1:0/2995123750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e780b57f0 con 0x7f8e7806a750 2024-03-18T20:07:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.031+0000 7f8e6effd700 1 -- 172.21.15.1:0/2995123750 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 105) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f8e68003a20 con 0x7f8e7806a750 2024-03-18T20:07:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.031+0000 7f8e6effd700 1 --2- 172.21.15.1:0/2995123750 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8e5c06ae00 0x7f8e5c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.031+0000 7f8e7dc58700 1 -- 172.21.15.1:0/2995123750 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8e5c06ae00 msgr2=0x7f8e5c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.031+0000 7f8e6effd700 1 -- 172.21.15.1:0/2995123750 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8e68088990 con 0x7f8e7806a750 2024-03-18T20:07:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.031+0000 7f8e7dc58700 1 --2- 172.21.15.1:0/2995123750 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8e5c06ae00 0x7f8e5c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:07:29.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.036+0000 7f8e6effd700 1 -- 172.21.15.1:0/2995123750 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8e68053960 con 0x7f8e7806a750 2024-03-18T20:07:29.149 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:29.144+0000 7ff2fadc1240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:07:29.197 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:29.192+0000 7f7966304240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:07:29.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.232+0000 7f8e7dc58700 1 -- 172.21.15.1:0/2995123750 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8e5c06ae00 msgr2=0x7f8e5c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:29.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.232+0000 7f8e7dc58700 1 --2- 172.21.15.1:0/2995123750 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8e5c06ae00 0x7f8e5c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:07:29.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.285+0000 7f8e806bd700 1 -- 172.21.15.1:0/2995123750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f8e78070000 con 0x7f8e7806a750 2024-03-18T20:07:29.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.290+0000 7f8e6effd700 1 -- 172.21.15.1:0/2995123750 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v65)=0 v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8e680170d0 con 0x7f8e7806a750 2024-03-18T20:07:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.292+0000 7f8e806bd700 1 -- 172.21.15.1:0/2995123750 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8e5c06ae00 msgr2=0x7f8e5c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:07:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.292+0000 7f8e806bd700 1 --2- 172.21.15.1:0/2995123750 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8e5c06ae00 0x7f8e5c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.292+0000 7f8e806bd700 1 -- 172.21.15.1:0/2995123750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7806a750 msgr2=0x7f8e78153500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.292+0000 7f8e806bd700 1 --2- 172.21.15.1:0/2995123750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7806a750 0x7f8e78153500 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f8e6800c4e0 tx=0x7f8e6800c7f0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.292+0000 7f8e806bd700 1 -- 172.21.15.1:0/2995123750 shutdown_connections 2024-03-18T20:07:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.292+0000 7f8e806bd700 1 --2- 172.21.15.1:0/2995123750 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f8e5c06ae00 0x7f8e5c06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.292+0000 7f8e806bd700 1 --2- 172.21.15.1:0/2995123750 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e7806f180 0x7f8e780cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.292+0000 7f8e806bd700 1 --2- 172.21.15.1:0/2995123750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e7806a750 0x7f8e78153500 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.292+0000 7f8e806bd700 1 --2- 172.21.15.1:0/2995123750 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e7806eda0 0x7f8e780c33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.292+0000 7f8e806bd700 1 -- 172.21.15.1:0/2995123750 >> 172.21.15.1:0/2995123750 conn(0x7f8e780cff30 msgr2=0x7f8e7815c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.292+0000 7f8e806bd700 1 -- 172.21.15.1:0/2995123750 shutdown_connections 2024-03-18T20:07:29.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.292+0000 7f8e806bd700 1 -- 172.21.15.1:0/2995123750 wait complete. 2024-03-18T20:07:29.310 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-18T20:07:29.413 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:29.408+0000 7ff2fadc1240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:07:29.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.435+0000 7f0b74228700 1 Processor -- start 2024-03-18T20:07:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.435+0000 7f0b74228700 1 -- start start 2024-03-18T20:07:29.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.436+0000 7f0b74228700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0b6c15eba0 0x7f0b6c0cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:29.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.436+0000 7f0b74228700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0b6c0b0e20 0x7f0b6c0b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:29.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.436+0000 7f0b74228700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b6c15ef80 con 0x7f0b6c0b0500 2024-03-18T20:07:29.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.436+0000 7f0b74228700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b6c0cc9a0 con 0x7f0b6c0b0e20 2024-03-18T20:07:29.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.436+0000 7f0b74228700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b6c0ccc80 con 0x7f0b6c15eba0 2024-03-18T20:07:29.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.436+0000 7f0b727c5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0b6c0b0e20 0x7f0b6c0b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:29.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.436+0000 7f0b717c3700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0b6c0b0500 0x7f0b6c0cf8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34032/0 (socket says 172.21.15.1:34032) 2024-03-18T20:07:29.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.436+0000 7f0b717c3700 1 -- 172.21.15.1:0/948105125 learned_addr learned my addr 172.21.15.1:0/948105125 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:29.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.436+0000 7f0b71fc4700 1 --2- 172.21.15.1:0/948105125 >> v2:172.21.15.165:3300/0 conn(0x7f0b6c15eba0 0x7f0b6c0cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:29.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.437+0000 7f0b727c5700 1 -- 172.21.15.1:0/948105125 >> v2:172.21.15.165:3300/0 conn(0x7f0b6c15eba0 msgr2=0x7f0b6c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:29.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.437+0000 7f0b727c5700 1 --2- 172.21.15.1:0/948105125 >> v2:172.21.15.165:3300/0 conn(0x7f0b6c15eba0 0x7f0b6c0cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.437+0000 7f0b727c5700 1 -- 172.21.15.1:0/948105125 >> v1:172.21.15.1:6789/0 conn(0x7f0b6c0b0500 legacy=0x7f0b6c0cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:29.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.437+0000 7f0b727c5700 1 -- 172.21.15.1:0/948105125 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6c0ccf60 con 0x7f0b6c0b0e20 2024-03-18T20:07:29.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.437+0000 7f0b70fc2700 1 -- 172.21.15.1:0/948105125 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549733111 0 0) 0x7f0b6c15ef80 con 0x7f0b6c0b0500 2024-03-18T20:07:29.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.437+0000 7f0b727c5700 1 --2- 172.21.15.1:0/948105125 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b6c0b0e20 0x7f0b6c0b1290 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c072fb0 tx=0x7f0b6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ceed217528c89b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:29.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.437+0000 7f0b70fc2700 1 -- 172.21.15.1:0/948105125 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6400e070 con 0x7f0b6c0b0e20 2024-03-18T20:07:29.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.437+0000 7f0b70fc2700 1 -- 172.21.15.1:0/948105125 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6400bb20 con 0x7f0b6c0b0e20 2024-03-18T20:07:29.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.437+0000 7f0b70fc2700 1 -- 172.21.15.1:0/948105125 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b64012430 con 0x7f0b6c0b0e20 2024-03-18T20:07:29.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.438+0000 7f0b74228700 1 -- 172.21.15.1:0/948105125 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b6c0b0e20 msgr2=0x7f0b6c0b1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:29.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.438+0000 7f0b74228700 1 --2- 172.21.15.1:0/948105125 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b6c0b0e20 0x7f0b6c0b1290 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c072fb0 tx=0x7f0b6400b330 comp rx=0 tx=0).stop 2024-03-18T20:07:29.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.438+0000 7f0b74228700 1 -- 172.21.15.1:0/948105125 shutdown_connections 2024-03-18T20:07:29.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.438+0000 7f0b74228700 1 --2- 172.21.15.1:0/948105125 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b6c0b0e20 0x7f0b6c0b1290 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.438+0000 7f0b74228700 1 --2- 172.21.15.1:0/948105125 >> v2:172.21.15.165:3300/0 conn(0x7f0b6c15eba0 0x7f0b6c0cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.438+0000 7f0b74228700 1 -- 172.21.15.1:0/948105125 >> 172.21.15.1:0/948105125 conn(0x7f0b6c0b2bd0 msgr2=0x7f0b6c0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:29.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.438+0000 7f0b74228700 1 -- 172.21.15.1:0/948105125 shutdown_connections 2024-03-18T20:07:29.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.438+0000 7f0b74228700 1 -- 172.21.15.1:0/948105125 wait complete. 2024-03-18T20:07:29.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.439+0000 7f0b74228700 1 Processor -- start 2024-03-18T20:07:29.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.439+0000 7f0b74228700 1 -- start start 2024-03-18T20:07:29.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.440+0000 7f0b74228700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b6c0b0500 0x7f0b6c0c71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:29.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.440+0000 7f0b74228700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b6c0b0e20 0x7f0b6c1f26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:29.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.440+0000 7f0b74228700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b6c15eba0 0x7f0b6c1f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:29.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.440+0000 7f0b74228700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6c06b860 con 0x7f0b6c15eba0 2024-03-18T20:07:29.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.440+0000 7f0b74228700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6c0c5280 con 0x7f0b6c0b0e20 2024-03-18T20:07:29.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.440+0000 7f0b74228700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6c1f3a90 con 0x7f0b6c0b0500 2024-03-18T20:07:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.441+0000 7f0b727c5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b6c15eba0 0x7f0b6c1f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.441+0000 7f0b727c5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b6c15eba0 0x7f0b6c1f2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49814/0 (socket says 172.21.15.1:49814) 2024-03-18T20:07:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.441+0000 7f0b727c5700 1 -- 172.21.15.1:0/2397251935 learned_addr learned my addr 172.21.15.1:0/2397251935 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.441+0000 7f0b717c3700 1 --2- 172.21.15.1:0/2397251935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b6c0b0e20 0x7f0b6c1f26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.441+0000 7f0b71fc4700 1 --2- 172.21.15.1:0/2397251935 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b6c0b0500 0x7f0b6c0c71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.441+0000 7f0b727c5700 1 -- 172.21.15.1:0/2397251935 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b6c0b0500 msgr2=0x7f0b6c0c71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.441+0000 7f0b727c5700 1 --2- 172.21.15.1:0/2397251935 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b6c0b0500 0x7f0b6c0c71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.441+0000 7f0b727c5700 1 -- 172.21.15.1:0/2397251935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b6c0b0e20 msgr2=0x7f0b6c1f26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.441+0000 7f0b727c5700 1 --2- 172.21.15.1:0/2397251935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b6c0b0e20 0x7f0b6c1f26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.441+0000 7f0b727c5700 1 -- 172.21.15.1:0/2397251935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b64009a70 con 0x7f0b6c15eba0 2024-03-18T20:07:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.441+0000 7f0b727c5700 1 --2- 172.21.15.1:0/2397251935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b6c15eba0 0x7f0b6c1f2c00 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c072150 tx=0x7f0b64004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.441+0000 7f0b627fc700 1 -- 172.21.15.1:0/2397251935 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b64012430 con 0x7f0b6c15eba0 2024-03-18T20:07:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.441+0000 7f0b627fc700 1 -- 172.21.15.1:0/2397251935 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b64020070 con 0x7f0b6c15eba0 2024-03-18T20:07:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.441+0000 7f0b627fc700 1 -- 172.21.15.1:0/2397251935 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b64004280 con 0x7f0b6c15eba0 2024-03-18T20:07:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.441+0000 7f0b74228700 1 -- 172.21.15.1:0/2397251935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b6c0c4cc0 con 0x7f0b6c15eba0 2024-03-18T20:07:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.441+0000 7f0b74228700 1 -- 172.21.15.1:0/2397251935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b6c06ba40 con 0x7f0b6c15eba0 2024-03-18T20:07:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.442+0000 7f0b74228700 1 -- 172.21.15.1:0/2397251935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b6c0676a0 con 0x7f0b6c15eba0 2024-03-18T20:07:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.443+0000 7f0b627fc700 1 -- 172.21.15.1:0/2397251935 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 105) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f0b640043e0 con 0x7f0b6c15eba0 2024-03-18T20:07:29.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.443+0000 7f0b627fc700 1 --2- 172.21.15.1:0/2397251935 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0b5006ae30 0x7f0b5006d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:29.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.444+0000 7f0b71fc4700 1 -- 172.21.15.1:0/2397251935 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0b5006ae30 msgr2=0x7f0b5006d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:29.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.444+0000 7f0b627fc700 1 -- 172.21.15.1:0/2397251935 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0b64022070 con 0x7f0b6c15eba0 2024-03-18T20:07:29.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.444+0000 7f0b71fc4700 1 --2- 172.21.15.1:0/2397251935 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0b5006ae30 0x7f0b5006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:07:29.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.448+0000 7f0b627fc700 1 -- 172.21.15.1:0/2397251935 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0b64055d80 con 0x7f0b6c15eba0 2024-03-18T20:07:29.479 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:29.474+0000 7f7966304240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:07:29.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.645+0000 7f0b71fc4700 1 -- 172.21.15.1:0/2397251935 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0b5006ae30 msgr2=0x7f0b5006d2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:29.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.645+0000 7f0b71fc4700 1 --2- 172.21.15.1:0/2397251935 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0b5006ae30 0x7f0b5006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:07:29.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.652+0000 7f0b74228700 1 -- 172.21.15.1:0/2397251935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f0b6c06e9f0 con 0x7f0b6c15eba0 2024-03-18T20:07:29.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.657+0000 7f0b627fc700 1 -- 172.21.15.1:0/2397251935 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v66)=0 v66) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0b640593a0 con 0x7f0b6c15eba0 2024-03-18T20:07:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.659+0000 7f0b74228700 1 -- 172.21.15.1:0/2397251935 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0b5006ae30 msgr2=0x7f0b5006d2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:07:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.659+0000 7f0b74228700 1 --2- 172.21.15.1:0/2397251935 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0b5006ae30 0x7f0b5006d2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.659+0000 7f0b74228700 1 -- 172.21.15.1:0/2397251935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b6c15eba0 msgr2=0x7f0b6c1f2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.659+0000 7f0b74228700 1 --2- 172.21.15.1:0/2397251935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b6c15eba0 0x7f0b6c1f2c00 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c072150 tx=0x7f0b64004ad0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.659+0000 7f0b74228700 1 -- 172.21.15.1:0/2397251935 shutdown_connections 2024-03-18T20:07:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.659+0000 7f0b74228700 1 --2- 172.21.15.1:0/2397251935 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f0b5006ae30 0x7f0b5006d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.659+0000 7f0b74228700 1 --2- 172.21.15.1:0/2397251935 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b6c0b0500 0x7f0b6c0c71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.659+0000 7f0b74228700 1 --2- 172.21.15.1:0/2397251935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b6c15eba0 0x7f0b6c1f2c00 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.659+0000 7f0b74228700 1 --2- 172.21.15.1:0/2397251935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b6c0b0e20 0x7f0b6c1f26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.659+0000 7f0b74228700 1 -- 172.21.15.1:0/2397251935 >> 172.21.15.1:0/2397251935 conn(0x7f0b6c0b2bd0 msgr2=0x7f0b6c151710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:29.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.659+0000 7f0b74228700 1 -- 172.21.15.1:0/2397251935 shutdown_connections 2024-03-18T20:07:29.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.659+0000 7f0b74228700 1 -- 172.21.15.1:0/2397251935 wait complete. 2024-03-18T20:07:29.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:07:29.677 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-03-18T20:07:29.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:07:29.700 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2024-03-18T20:07:29.763 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/test_restart 1m 2024-03-18T20:07:29.834 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:29.830+0000 7ff2fadc1240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:07:29.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.930+0000 7f5b74447700 1 Processor -- start 2024-03-18T20:07:29.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.930+0000 7f5b74447700 1 -- start start 2024-03-18T20:07:29.936 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:29.930+0000 7f7966304240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:07:29.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.931+0000 7f5b74447700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5b6c06d560 0x7f5b6c0aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:29.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.931+0000 7f5b74447700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5b6c0af510 0x7f5b6c15b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.931+0000 7f5b74447700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b6c15ef00 con 0x7f5b6c06a750 2024-03-18T20:07:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.931+0000 7f5b74447700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b6c15f1e0 con 0x7f5b6c06d560 2024-03-18T20:07:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.931+0000 7f5b74447700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b6c15f4c0 con 0x7f5b6c0af510 2024-03-18T20:07:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.931+0000 7f5b719e2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5b6c06d560 0x7f5b6c0aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.931+0000 7f5b721e3700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5b6c06a750 0x7f5b6c06ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34034/0 (socket says 172.21.15.1:34034) 2024-03-18T20:07:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.931+0000 7f5b721e3700 1 -- 172.21.15.1:0/918283432 learned_addr learned my addr 172.21.15.1:0/918283432 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:07:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.931+0000 7f5b729e4700 1 --2- 172.21.15.1:0/918283432 >> v2:172.21.15.165:3300/0 conn(0x7f5b6c0af510 0x7f5b6c15b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.931+0000 7f5b719e2700 1 -- 172.21.15.1:0/918283432 >> v2:172.21.15.165:3300/0 conn(0x7f5b6c0af510 msgr2=0x7f5b6c15b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.931+0000 7f5b719e2700 1 --2- 172.21.15.1:0/918283432 >> v2:172.21.15.165:3300/0 conn(0x7f5b6c0af510 0x7f5b6c15b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.931+0000 7f5b719e2700 1 -- 172.21.15.1:0/918283432 >> v1:172.21.15.1:6789/0 conn(0x7f5b6c06a750 legacy=0x7f5b6c06ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.931+0000 7f5b719e2700 1 -- 172.21.15.1:0/918283432 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c15f7a0 con 0x7f5b6c06d560 2024-03-18T20:07:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.931+0000 7f5b711e1700 1 -- 172.21.15.1:0/918283432 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975079623 0 0) 0x7f5b6c15ef00 con 0x7f5b6c06a750 2024-03-18T20:07:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.932+0000 7f5b719e2700 1 --2- 172.21.15.1:0/918283432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b6c06d560 0x7f5b6c0aefd0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c06d270 tx=0x7f5b5c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71f1a86199064dce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:29.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.932+0000 7f5b711e1700 1 -- 172.21.15.1:0/918283432 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c011830 con 0x7f5b6c06d560 2024-03-18T20:07:29.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.932+0000 7f5b711e1700 1 -- 172.21.15.1:0/918283432 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5c011990 con 0x7f5b6c06d560 2024-03-18T20:07:29.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.932+0000 7f5b711e1700 1 -- 172.21.15.1:0/918283432 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c011d20 con 0x7f5b6c06d560 2024-03-18T20:07:29.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.932+0000 7f5b74447700 1 -- 172.21.15.1:0/918283432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b6c06d560 msgr2=0x7f5b6c0aefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:29.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.932+0000 7f5b74447700 1 --2- 172.21.15.1:0/918283432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b6c06d560 0x7f5b6c0aefd0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c06d270 tx=0x7f5b5c009b20 comp rx=0 tx=0).stop 2024-03-18T20:07:29.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.932+0000 7f5b74447700 1 -- 172.21.15.1:0/918283432 shutdown_connections 2024-03-18T20:07:29.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.932+0000 7f5b74447700 1 --2- 172.21.15.1:0/918283432 >> v2:172.21.15.165:3300/0 conn(0x7f5b6c0af510 0x7f5b6c15b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.932+0000 7f5b74447700 1 --2- 172.21.15.1:0/918283432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b6c06d560 0x7f5b6c0aefd0 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.932+0000 7f5b74447700 1 -- 172.21.15.1:0/918283432 >> 172.21.15.1:0/918283432 conn(0x7f5b6c0cff30 msgr2=0x7f5b6c0d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:29.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.933+0000 7f5b74447700 1 -- 172.21.15.1:0/918283432 shutdown_connections 2024-03-18T20:07:29.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.933+0000 7f5b74447700 1 -- 172.21.15.1:0/918283432 wait complete. 2024-03-18T20:07:29.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.934+0000 7f5b74447700 1 Processor -- start 2024-03-18T20:07:29.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.934+0000 7f5b74447700 1 -- start start 2024-03-18T20:07:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b74447700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b6c06a750 0x7f5b6c0ce510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b74447700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b6c06d560 0x7f5b6c14a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b74447700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b6c0af510 0x7f5b6c0c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b74447700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c0b3ca0 con 0x7f5b6c06d560 2024-03-18T20:07:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b74447700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c0b4620 con 0x7f5b6c0af510 2024-03-18T20:07:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b74447700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c0b3090 con 0x7f5b6c06a750 2024-03-18T20:07:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b719e2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b6c06d560 0x7f5b6c14a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:29.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b719e2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b6c06d560 0x7f5b6c14a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49828/0 (socket says 172.21.15.1:49828) 2024-03-18T20:07:29.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b719e2700 1 -- 172.21.15.1:0/3655603415 learned_addr learned my addr 172.21.15.1:0/3655603415 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:29.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b721e3700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b6c06a750 0x7f5b6c0ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:29.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b729e4700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b6c0af510 0x7f5b6c0c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:29.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b719e2700 1 -- 172.21.15.1:0/3655603415 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b6c06a750 msgr2=0x7f5b6c0ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:29.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b719e2700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b6c06a750 0x7f5b6c0ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b719e2700 1 -- 172.21.15.1:0/3655603415 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b6c0af510 msgr2=0x7f5b6c0c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:29.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b719e2700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b6c0af510 0x7f5b6c0c7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:29.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b719e2700 1 -- 172.21.15.1:0/3655603415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5c0094a0 con 0x7f5b6c06d560 2024-03-18T20:07:29.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b719e2700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b6c06d560 0x7f5b6c14a240 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c067310 tx=0x7f5b5c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:29.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b62ffd700 1 -- 172.21.15.1:0/3655603415 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c0036a0 con 0x7f5b6c06d560 2024-03-18T20:07:29.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b62ffd700 1 -- 172.21.15.1:0/3655603415 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5c01f070 con 0x7f5b6c06d560 2024-03-18T20:07:29.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b74447700 1 -- 172.21.15.1:0/3655603415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6c0b3370 con 0x7f5b6c06d560 2024-03-18T20:07:29.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b62ffd700 1 -- 172.21.15.1:0/3655603415 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c003a60 con 0x7f5b6c06d560 2024-03-18T20:07:29.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.935+0000 7f5b74447700 1 -- 172.21.15.1:0/3655603415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6c0b3930 con 0x7f5b6c06d560 2024-03-18T20:07:29.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.936+0000 7f5b74447700 1 -- 172.21.15.1:0/3655603415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b6c0b3e50 con 0x7f5b6c06d560 2024-03-18T20:07:29.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.937+0000 7f5b62ffd700 1 -- 172.21.15.1:0/3655603415 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 105) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f5b5c003bc0 con 0x7f5b6c06d560 2024-03-18T20:07:29.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.937+0000 7f5b62ffd700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b5006ae00 0x7f5b5006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:29.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.937+0000 7f5b721e3700 1 -- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b5006ae00 msgr2=0x7f5b5006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:29.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.937+0000 7f5b62ffd700 1 -- 172.21.15.1:0/3655603415 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5b5c021070 con 0x7f5b6c06d560 2024-03-18T20:07:29.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.937+0000 7f5b721e3700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b5006ae00 0x7f5b5006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:07:29.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:29.942+0000 7f5b62ffd700 1 -- 172.21.15.1:0/3655603415 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5b6c0b3e50 con 0x7f5b6c06d560 2024-03-18T20:07:30.044 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:30.040+0000 7ff2fadc1240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:07:30.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:30.138+0000 7f5b721e3700 1 -- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b5006ae00 msgr2=0x7f5b5006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:30.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:30.138+0000 7f5b721e3700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b5006ae00 0x7f5b5006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:07:30.157 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:30.152+0000 7f7966304240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:07:30.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:30.172+0000 7f5b74447700 1 -- 172.21.15.1:0/3655603415 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b6c15b480 con 0x7f5b5006ae00 2024-03-18T20:07:30.412 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:30.407+0000 7ff2fadc1240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:07:30.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:30.539+0000 7f5b721e3700 1 -- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b5006ae00 msgr2=0x7f5b5006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:30.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:30.539+0000 7f5b721e3700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b5006ae00 0x7f5b5006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T20:07:30.544 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:30.540+0000 7f7966304240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:07:30.989 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:30.984+0000 7ff2fadc1240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:07:31.061 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:31.055+0000 7ff2fadc1240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:07:31.158 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:31.153+0000 7f7966304240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:07:31.187 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:31.181+0000 7ff2fadc1240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:07:31.233 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:31.228+0000 7f7966304240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:07:31.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:31.340+0000 7f5b721e3700 1 -- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b5006ae00 msgr2=0x7f5b5006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:31.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:31.340+0000 7f5b721e3700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b5006ae00 0x7f5b5006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T20:07:31.364 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:31.359+0000 7ff2fadc1240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:07:31.366 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:31.362+0000 7f7966304240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:07:31.435 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:31.430+0000 7ff2fadc1240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:07:31.512 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:31.507+0000 7ff2fadc1240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:07:31.553 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:31.548+0000 7f7966304240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:07:31.588 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:31.583+0000 7ff2fadc1240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:07:31.628 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:31.624+0000 7f7966304240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:07:31.665 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:31.660+0000 7ff2fadc1240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:07:31.710 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:31.706+0000 7f7966304240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:07:31.791 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:31.787+0000 7f7966304240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:07:31.833 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:31.828+0000 7ff2fadc1240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:07:31.874 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:31.868+0000 7f7966304240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:07:32.046 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:32.041+0000 7ff2fadc1240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:07:32.051 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:32.046+0000 7f7966304240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:07:32.196 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:32.191+0000 7ff2fadc1240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:07:32.268 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:32.263+0000 7ff2fadc1240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:07:32.274 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:32.270+0000 7f7966304240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:07:32.433 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:32.428+0000 7f7966304240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:07:32.510 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:32.505+0000 7f7966304240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:07:32.709 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:32.704+0000 7ff2fadc1240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:07:32.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:32.941+0000 7f5b721e3700 1 -- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b5006ae00 msgr2=0x7f5b5006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/76998 2024-03-18T20:07:32.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:32.941+0000 7f5b721e3700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b5006ae00 0x7f5b5006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T20:07:32.982 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:32.978+0000 7f7966304240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:07:33.045 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:33.040+0000 7ff2fadc1240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:07:33.206 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:33.201+0000 7ff2fadc1240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:07:33.336 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:33.332+0000 7f7966304240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:07:33.509 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:33.504+0000 7f7966304240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:07:33.589 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:33.584+0000 7ff2fadc1240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:07:33.752 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:33.747+0000 7ff2fadc1240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:07:33.923 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:33.918+0000 7f7966304240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:07:34.099 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:34.094+0000 7f7966304240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:07:34.111 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:34.106+0000 7ff2fadc1240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:07:34.199 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:07:34.194+0000 7ff2fadc1240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:07:34.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:34.264+0000 7f5b62ffd700 1 -- 172.21.15.1:0/3655603415 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 106) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f5b5c089200 con 0x7f5b6c06d560 2024-03-18T20:07:34.484 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:34.479+0000 7f7966304240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:07:34.578 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:34.573+0000 7f7966304240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:07:34.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:34.595+0000 7f5b62ffd700 1 -- 172.21.15.1:0/3655603415 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 107) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f5b5c056180 con 0x7f5b6c06d560 2024-03-18T20:07:34.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:34.595+0000 7f5b62ffd700 1 -- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b5006ae00 msgr2=0x7f5b5006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:07:34.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:34.595+0000 7f5b62ffd700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b5006ae00 0x7f5b5006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:34.738 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:34.733+0000 7f78dbf27700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:34.738 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:34.733+0000 7f78dbf27700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:34.738 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:34.733+0000 7f78dbf27700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:34.738 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:34.733+0000 7f78dbf27700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:34.738 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:34.733+0000 7f78dbf27700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:34.738 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:34.733+0000 7f78d8f21700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:34.738 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:34.733+0000 7f78d8f21700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:34.738 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:34.733+0000 7f78d8f21700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:34.738 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:34.733+0000 7f78d8f21700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:34.739 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:07:34.733+0000 7f78d8f21700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:07:35.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.600+0000 7f5b62ffd700 1 -- 172.21.15.1:0/3655603415 <== mon.0 v2:172.21.15.1:3300/0 9 ==== mgrmap(e 108) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f5b5c056a50 con 0x7f5b6c06d560 2024-03-18T20:07:35.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.600+0000 7f5b62ffd700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b5006ae00 0x7f5b5006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:35.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.600+0000 7f5b62ffd700 1 -- 172.21.15.1:0/3655603415 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b50027a50 con 0x7f5b5006ae00 2024-03-18T20:07:35.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.603+0000 7f5b721e3700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b5006ae00 0x7f5b5006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:35.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.605+0000 7f5b721e3700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b5006ae00 0x7f5b5006d2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5b68007fd0 tx=0x7f5b68007de0 comp rx=0 tx=0).ready entity=mgr.10614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:35.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.665+0000 7f5b62ffd700 1 -- 172.21.15.1:0/3655603415 <== mgr.10614 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f5b50027a50 con 0x7f5b5006ae00 2024-03-18T20:07:35.670 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path /snap_test_dir1/test_restart 2024-03-18T20:07:35.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.668+0000 7f5b74447700 1 -- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b5006ae00 msgr2=0x7f5b5006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:35.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.668+0000 7f5b74447700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b5006ae00 0x7f5b5006d2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5b68007fd0 tx=0x7f5b68007de0 comp rx=0 tx=0).stop 2024-03-18T20:07:35.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.668+0000 7f5b74447700 1 -- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b6c06d560 msgr2=0x7f5b6c14a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:35.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.668+0000 7f5b74447700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b6c06d560 0x7f5b6c14a240 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c067310 tx=0x7f5b5c004750 comp rx=0 tx=0).stop 2024-03-18T20:07:35.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.668+0000 7f5b74447700 1 -- 172.21.15.1:0/3655603415 shutdown_connections 2024-03-18T20:07:35.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.668+0000 7f5b74447700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f5b5006ae00 0x7f5b5006d2c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:35.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.668+0000 7f5b74447700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5b6c06a750 0x7f5b6c0ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:35.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.668+0000 7f5b74447700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b6c0af510 0x7f5b6c0c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:35.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.668+0000 7f5b74447700 1 --2- 172.21.15.1:0/3655603415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b6c06d560 0x7f5b6c14a240 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:35.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.668+0000 7f5b74447700 1 -- 172.21.15.1:0/3655603415 >> 172.21.15.1:0/3655603415 conn(0x7f5b6c0cff30 msgr2=0x7f5b6c0cb450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:35.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.668+0000 7f5b74447700 1 -- 172.21.15.1:0/3655603415 shutdown_connections 2024-03-18T20:07:35.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.668+0000 7f5b74447700 1 -- 172.21.15.1:0/3655603415 wait complete. 2024-03-18T20:07:35.686 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2024-03-18T20:07:35.686 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/test_restart 1m 2024-03-18T20:07:35.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.813+0000 7f9681fef700 1 Processor -- start 2024-03-18T20:07:35.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.813+0000 7f9681fef700 1 -- start start 2024-03-18T20:07:35.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.814+0000 7f9681fef700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f967c099a40 0x7f967c0ade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:35.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.814+0000 7f9681fef700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f967c099e20 0x7f967c0cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.814+0000 7f9681fef700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f967c0ccc20 0x7f967c06fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.814+0000 7f9681fef700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f967c0b3aa0 con 0x7f967c0ccc20 2024-03-18T20:07:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.814+0000 7f9681fef700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f967c06e9b0 con 0x7f967c099e20 2024-03-18T20:07:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.814+0000 7f9681fef700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967c06ec90 con 0x7f967c099a40 2024-03-18T20:07:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.814+0000 7f967bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f967c0ccc20 0x7f967c06fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.814+0000 7f967bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f967c0ccc20 0x7f967c06fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41960/0 (socket says 172.21.15.1:41960) 2024-03-18T20:07:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.814+0000 7f967bfff700 1 -- 172.21.15.1:0/4226232647 learned_addr learned my addr 172.21.15.1:0/4226232647 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.814+0000 7f967b7fe700 1 --2- 172.21.15.1:0/4226232647 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f967c099a40 0x7f967c0ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.814+0000 7f967affd700 1 --2- 172.21.15.1:0/4226232647 >> v2:172.21.15.165:3300/0 conn(0x7f967c099e20 0x7f967c0cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.814+0000 7f967bfff700 1 -- 172.21.15.1:0/4226232647 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f967c099a40 msgr2=0x7f967c0ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.814+0000 7f967bfff700 1 --2- 172.21.15.1:0/4226232647 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f967c099a40 0x7f967c0ade00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.814+0000 7f967bfff700 1 -- 172.21.15.1:0/4226232647 >> v2:172.21.15.165:3300/0 conn(0x7f967c099e20 msgr2=0x7f967c0cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:07:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.814+0000 7f967bfff700 1 --2- 172.21.15.1:0/4226232647 >> v2:172.21.15.165:3300/0 conn(0x7f967c099e20 0x7f967c0cc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.814+0000 7f967bfff700 1 -- 172.21.15.1:0/4226232647 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c06ef70 con 0x7f967c0ccc20 2024-03-18T20:07:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.814+0000 7f967bfff700 1 --2- 172.21.15.1:0/4226232647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f967c0ccc20 0x7f967c06fa50 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f967c0d2260 tx=0x7f967400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54d53755a0f41cd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.815+0000 7f967a7fc700 1 -- 172.21.15.1:0/4226232647 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967400e070 con 0x7f967c0ccc20 2024-03-18T20:07:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.815+0000 7f967a7fc700 1 -- 172.21.15.1:0/4226232647 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967400b910 con 0x7f967c0ccc20 2024-03-18T20:07:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.815+0000 7f967a7fc700 1 -- 172.21.15.1:0/4226232647 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967400bcc0 con 0x7f967c0ccc20 2024-03-18T20:07:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.815+0000 7f9681fef700 1 -- 172.21.15.1:0/4226232647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f967c0ccc20 msgr2=0x7f967c06fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.815+0000 7f9681fef700 1 --2- 172.21.15.1:0/4226232647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f967c0ccc20 0x7f967c06fa50 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f967c0d2260 tx=0x7f967400b330 comp rx=0 tx=0).stop 2024-03-18T20:07:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.815+0000 7f9681fef700 1 -- 172.21.15.1:0/4226232647 shutdown_connections 2024-03-18T20:07:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.815+0000 7f9681fef700 1 --2- 172.21.15.1:0/4226232647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f967c0ccc20 0x7f967c06fa50 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.815+0000 7f9681fef700 1 --2- 172.21.15.1:0/4226232647 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f967c099a40 0x7f967c0ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.815+0000 7f9681fef700 1 --2- 172.21.15.1:0/4226232647 >> v2:172.21.15.165:3300/0 conn(0x7f967c099e20 0x7f967c0cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.815+0000 7f9681fef700 1 -- 172.21.15.1:0/4226232647 >> 172.21.15.1:0/4226232647 conn(0x7f967c0b2c50 msgr2=0x7f967c0b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.815+0000 7f9681fef700 1 -- 172.21.15.1:0/4226232647 shutdown_connections 2024-03-18T20:07:35.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.815+0000 7f9681fef700 1 -- 172.21.15.1:0/4226232647 wait complete. 2024-03-18T20:07:35.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.816+0000 7f9681fef700 1 Processor -- start 2024-03-18T20:07:35.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.817+0000 7f9681fef700 1 -- start start 2024-03-18T20:07:35.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.817+0000 7f9681fef700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f967c099a40 0x7f967c154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:35.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.817+0000 7f9681fef700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f967c099e20 0x7f967c154de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:35.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.817+0000 7f9681fef700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f967c0ccc20 0x7f967c0c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:35.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.817+0000 7f9681fef700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c15d810 con 0x7f967c099e20 2024-03-18T20:07:35.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.817+0000 7f9681fef700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c15cc00 con 0x7f967c0ccc20 2024-03-18T20:07:35.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.817+0000 7f9681fef700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967c15cee0 con 0x7f967c099a40 2024-03-18T20:07:35.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.817+0000 7f967affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f967c099e20 0x7f967c154de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:35.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.817+0000 7f967affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f967c099e20 0x7f967c154de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41972/0 (socket says 172.21.15.1:41972) 2024-03-18T20:07:35.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.818+0000 7f967affd700 1 -- 172.21.15.1:0/3952098473 learned_addr learned my addr 172.21.15.1:0/3952098473 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:07:35.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.818+0000 7f967b7fe700 1 --2- 172.21.15.1:0/3952098473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f967c099a40 0x7f967c154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:35.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.818+0000 7f967bfff700 1 --2- 172.21.15.1:0/3952098473 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f967c0ccc20 0x7f967c0c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:35.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.818+0000 7f967affd700 1 -- 172.21.15.1:0/3952098473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f967c099a40 msgr2=0x7f967c154630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:35.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.818+0000 7f967affd700 1 --2- 172.21.15.1:0/3952098473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f967c099a40 0x7f967c154630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:35.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.818+0000 7f967affd700 1 -- 172.21.15.1:0/3952098473 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f967c0ccc20 msgr2=0x7f967c0c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:35.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.818+0000 7f967affd700 1 --2- 172.21.15.1:0/3952098473 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f967c0ccc20 0x7f967c0c60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:35.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.818+0000 7f967affd700 1 -- 172.21.15.1:0/3952098473 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9674009b30 con 0x7f967c099e20 2024-03-18T20:07:35.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.818+0000 7f967b7fe700 1 --2- 172.21.15.1:0/3952098473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f967c099a40 0x7f967c154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:07:35.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.818+0000 7f967affd700 1 --2- 172.21.15.1:0/3952098473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f967c099e20 0x7f967c154de0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f966c00e870 tx=0x7f966c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:35.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.818+0000 7f9673fff700 1 -- 172.21.15.1:0/3952098473 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c009590 con 0x7f967c099e20 2024-03-18T20:07:35.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.818+0000 7f9673fff700 1 -- 172.21.15.1:0/3952098473 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966c00ee70 con 0x7f967c099e20 2024-03-18T20:07:35.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.818+0000 7f9681fef700 1 -- 172.21.15.1:0/3952098473 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967c15d1c0 con 0x7f967c099e20 2024-03-18T20:07:35.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.818+0000 7f9673fff700 1 -- 172.21.15.1:0/3952098473 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c00ca50 con 0x7f967c099e20 2024-03-18T20:07:35.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.818+0000 7f9681fef700 1 -- 172.21.15.1:0/3952098473 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967c154b70 con 0x7f967c099e20 2024-03-18T20:07:35.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.819+0000 7f9681fef700 1 -- 172.21.15.1:0/3952098473 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f967c0676a0 con 0x7f967c099e20 2024-03-18T20:07:35.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.819+0000 7f9673fff700 1 -- 172.21.15.1:0/3952098473 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 108) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f966c00cbb0 con 0x7f967c099e20 2024-03-18T20:07:35.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.820+0000 7f9673fff700 1 --2- 172.21.15.1:0/3952098473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f965c06ab30 0x7f965c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:07:35.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.820+0000 7f9673fff700 1 -- 172.21.15.1:0/3952098473 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f966c089230 con 0x7f967c099e20 2024-03-18T20:07:35.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.820+0000 7f967b7fe700 1 --2- 172.21.15.1:0/3952098473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f965c06ab30 0x7f965c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:07:35.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.820+0000 7f967b7fe700 1 --2- 172.21.15.1:0/3952098473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f965c06ab30 0x7f965c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9664009b10 tx=0x7f9664008040 comp rx=0 tx=0).ready entity=mgr.10614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:07:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:35.825+0000 7f9673fff700 1 -- 172.21.15.1:0/3952098473 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f966c054470 con 0x7f967c099e20 2024-03-18T20:07:36.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:36.056+0000 7f9681fef700 1 -- 172.21.15.1:0/3952098473 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f967c09a2a0 con 0x7f965c06ab30 2024-03-18T20:07:36.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:36.095+0000 7f9673fff700 1 -- 172.21.15.1:0/3952098473 <== mgr.10614 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7f9664011730 con 0x7f965c06ab30 2024-03-18T20:07:36.100 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule activated for path /snap_test_dir1/test_restart 2024-03-18T20:07:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:36.098+0000 7f9681fef700 1 -- 172.21.15.1:0/3952098473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f965c06ab30 msgr2=0x7f965c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:36.098+0000 7f9681fef700 1 --2- 172.21.15.1:0/3952098473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f965c06ab30 0x7f965c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9664009b10 tx=0x7f9664008040 comp rx=0 tx=0).stop 2024-03-18T20:07:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:36.098+0000 7f9681fef700 1 -- 172.21.15.1:0/3952098473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f967c099e20 msgr2=0x7f967c154de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:07:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:36.098+0000 7f9681fef700 1 --2- 172.21.15.1:0/3952098473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f967c099e20 0x7f967c154de0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f966c00e870 tx=0x7f966c00ec30 comp rx=0 tx=0).stop 2024-03-18T20:07:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:36.098+0000 7f9681fef700 1 -- 172.21.15.1:0/3952098473 shutdown_connections 2024-03-18T20:07:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:36.098+0000 7f9681fef700 1 --2- 172.21.15.1:0/3952098473 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f965c06ab30 0x7f965c06cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:36.098+0000 7f9681fef700 1 --2- 172.21.15.1:0/3952098473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f967c099a40 0x7f967c154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:36.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:36.098+0000 7f9681fef700 1 --2- 172.21.15.1:0/3952098473 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f967c0ccc20 0x7f967c0c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:36.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:36.098+0000 7f9681fef700 1 --2- 172.21.15.1:0/3952098473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f967c099e20 0x7f967c154de0 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:07:36.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:36.098+0000 7f9681fef700 1 -- 172.21.15.1:0/3952098473 >> 172.21.15.1:0/3952098473 conn(0x7f967c0b2c50 msgr2=0x7f967c0cf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:07:36.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:36.098+0000 7f9681fef700 1 -- 172.21.15.1:0/3952098473 shutdown_connections 2024-03-18T20:07:36.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:07:36.098+0000 7f9681fef700 1 -- 172.21.15.1:0/3952098473 wait complete. 2024-03-18T20:07:36.115 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2024-03-18T20:07:36.247 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:07:36.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:08:00.110 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:08:00.104+0000 7f78caec5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:08:00.111 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:08:00.104+0000 7f78caec5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:08:00.111 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:08:00.104+0000 7f78caec5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:08:00.111 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:08:00.104+0000 7f78caec5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:08:00.111 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:08:00.104+0000 7f78caec5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:08:06.352 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:08:06.356 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:08:36.498 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:08:36.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:09:06.602 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:09:06.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:09:36.748 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:09:36.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:10:06.836 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:10:06.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:10:36.907 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:10:36.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:11:07.044 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:11:07.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:11:37.181 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:11:37.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:12:07.302 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:12:07.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:12:37.440 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:12:37.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:13:07.505 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:13:07.509 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:13:37.648 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:13:37.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:14:07.795 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:14:07.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:14:38.090 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:14:38.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:15:08.159 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:15:08.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:15:38.298 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:15:38.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:16:08.419 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:16:08.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:16:38.583 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:16:38.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:17:08.646 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:17:08.649 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:17:38.780 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:17:38.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:18:08.868 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:18:08.871 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:18:16.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:18:16.116 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-03-18T20:18:16.147 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_08_00_UTC 2024-03-18T20:18:16.147 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_09_00_UTC 2024-03-18T20:18:16.147 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_10_00_UTC 2024-03-18T20:18:16.147 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_11_00_UTC 2024-03-18T20:18:16.147 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_12_00_UTC 2024-03-18T20:18:16.148 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_13_00_UTC 2024-03-18T20:18:16.148 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_14_00_UTC 2024-03-18T20:18:16.148 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_15_00_UTC 2024-03-18T20:18:16.148 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_16_00_UTC 2024-03-18T20:18:16.148 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_17_00_UTC 2024-03-18T20:18:16.148 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_18_00_UTC 2024-03-18T20:18:16.148 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-18-20_08_00_UTC', 'scheduled-2024-03-18-20_09_00_UTC', 'scheduled-2024-03-18-20_10_00_UTC', 'scheduled-2024-03-18-20_11_00_UTC', 'scheduled-2024-03-18-20_12_00_UTC', 'scheduled-2024-03-18-20_13_00_UTC', 'scheduled-2024-03-18-20_14_00_UTC', 'scheduled-2024-03-18-20_15_00_UTC', 'scheduled-2024-03-18-20_16_00_UTC', 'scheduled-2024-03-18-20_17_00_UTC', 'scheduled-2024-03-18-20_18_00_UTC'] 2024-03-18T20:18:16.148 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2024-03-18T20:18:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.321+0000 7f405bd27700 1 Processor -- start 2024-03-18T20:18:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.322+0000 7f405bd27700 1 -- start start 2024-03-18T20:18:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.322+0000 7f405bd27700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f405406dae0 0x7f4054070a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:18:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.322+0000 7f405bd27700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40540b38d0 con 0x7f4054070580 2024-03-18T20:18:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.322+0000 7f405bd27700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40540b4470 con 0x7f40540ce8d0 2024-03-18T20:18:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f405bd27700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40540cecb0 con 0x7f405406dae0 2024-03-18T20:18:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f40592c2700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4054070580 0x7f405406d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52026/0 (socket says 172.21.15.1:52026) 2024-03-18T20:18:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f40592c2700 1 -- 172.21.15.1:0/1405236575 learned_addr learned my addr 172.21.15.1:0/1405236575 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:18:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f4058ac1700 1 -- 172.21.15.1:0/1405236575 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153219115 0 0) 0x7f40540b38d0 con 0x7f4054070580 2024-03-18T20:18:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f405a2c4700 1 --2- 172.21.15.1:0/1405236575 >> v2:172.21.15.165:3300/0 conn(0x7f405406dae0 0x7f4054070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:18:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f4058ac1700 1 -- 172.21.15.1:0/1405236575 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40400035b0 con 0x7f4054070580 2024-03-18T20:18:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f4058ac1700 1 -- 172.21.15.1:0/1405236575 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1107928035 0 0) 0x7f40400035b0 con 0x7f4054070580 2024-03-18T20:18:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f4058ac1700 1 -- 172.21.15.1:0/1405236575 >> v2:172.21.15.165:3300/0 conn(0x7f405406dae0 msgr2=0x7f4054070a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:18:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f4058ac1700 1 --2- 172.21.15.1:0/1405236575 >> v2:172.21.15.165:3300/0 conn(0x7f405406dae0 0x7f4054070a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f4058ac1700 1 -- 172.21.15.1:0/1405236575 >> v1:172.21.15.165:6789/0 conn(0x7f40540ce8d0 legacy=0x7f40540b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:18:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f4058ac1700 1 -- 172.21.15.1:0/1405236575 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4054071020 con 0x7f4054070580 2024-03-18T20:18:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f4058ac1700 1 -- 172.21.15.1:0/1405236575 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f4050001670 con 0x7f4054070580 2024-03-18T20:18:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f4058ac1700 1 -- 172.21.15.1:0/1405236575 >> v1:172.21.15.1:6789/0 conn(0x7f4054070580 legacy=0x7f405406d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:18:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f4058ac1700 1 --2- 172.21.15.1:0/1405236575 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4040003a00 0x7f4040003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:18:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f4058ac1700 1 --2- 172.21.15.1:0/1405236575 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4040004850 0x7f4040004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:18:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f4058ac1700 1 --2- 172.21.15.1:0/1405236575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4040005210 0x7f404000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:18:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f4058ac1700 1 -- 172.21.15.1:0/1405236575 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40540cecb0 con 0x7f4040005210 2024-03-18T20:18:16.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f4058ac1700 1 -- 172.21.15.1:0/1405236575 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404000c1b0 con 0x7f4040003a00 2024-03-18T20:18:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f4058ac1700 1 -- 172.21.15.1:0/1405236575 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f404000c370 con 0x7f4040004850 2024-03-18T20:18:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f405a2c4700 1 --2- 172.21.15.1:0/1405236575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4040005210 0x7f404000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:18:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.323+0000 7f4059ac3700 1 --2- 172.21.15.1:0/1405236575 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4040003a00 0x7f4040003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:18:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.324+0000 7f40592c2700 1 --2- 172.21.15.1:0/1405236575 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4040004850 0x7f4040004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:18:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.324+0000 7f405a2c4700 1 -- 172.21.15.1:0/1405236575 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4040004850 msgr2=0x7f4040004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:18:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.324+0000 7f405a2c4700 1 --2- 172.21.15.1:0/1405236575 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4040004850 0x7f4040004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.324+0000 7f405a2c4700 1 -- 172.21.15.1:0/1405236575 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4040003a00 msgr2=0x7f4040003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:18:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.324+0000 7f405a2c4700 1 --2- 172.21.15.1:0/1405236575 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4040003a00 0x7f4040003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.324+0000 7f405a2c4700 1 -- 172.21.15.1:0/1405236575 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40400035b0 con 0x7f4040005210 2024-03-18T20:18:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.324+0000 7f40592c2700 1 --2- 172.21.15.1:0/1405236575 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4040004850 0x7f4040004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:18:16.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.324+0000 7f405a2c4700 1 -- 172.21.15.1:0/1405236575 reap_dead start 2024-03-18T20:18:16.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.324+0000 7f405a2c4700 1 --2- 172.21.15.1:0/1405236575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4040005210 0x7f404000ba10 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f4054070360 tx=0x7f4040005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1153899ec1927863 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:18:16.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.324+0000 7f4058ac1700 1 -- 172.21.15.1:0/1405236575 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c00a2d0 con 0x7f4040005210 2024-03-18T20:18:16.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.324+0000 7f4058ac1700 1 -- 172.21.15.1:0/1405236575 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404c010040 con 0x7f4040005210 2024-03-18T20:18:16.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.324+0000 7f4058ac1700 1 -- 172.21.15.1:0/1405236575 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c00fd10 con 0x7f4040005210 2024-03-18T20:18:16.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.325+0000 7f405bd27700 1 -- 172.21.15.1:0/1405236575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4040005210 msgr2=0x7f404000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:18:16.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.325+0000 7f405bd27700 1 --2- 172.21.15.1:0/1405236575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4040005210 0x7f404000ba10 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f4054070360 tx=0x7f4040005880 comp rx=0 tx=0).stop 2024-03-18T20:18:16.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.325+0000 7f405bd27700 1 -- 172.21.15.1:0/1405236575 shutdown_connections 2024-03-18T20:18:16.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.325+0000 7f405bd27700 1 --2- 172.21.15.1:0/1405236575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4040005210 0x7f404000ba10 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:16.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.325+0000 7f405bd27700 1 -- 172.21.15.1:0/1405236575 >> 172.21.15.1:0/1405236575 conn(0x7f405406ab70 msgr2=0x7f405406af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:18:16.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.325+0000 7f405bd27700 1 -- 172.21.15.1:0/1405236575 shutdown_connections 2024-03-18T20:18:16.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.325+0000 7f405bd27700 1 -- 172.21.15.1:0/1405236575 wait complete. 2024-03-18T20:18:16.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.326+0000 7f405bd27700 1 Processor -- start 2024-03-18T20:18:16.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.326+0000 7f405bd27700 1 -- start start 2024-03-18T20:18:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.327+0000 7f405bd27700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4040005210 0x7f4054140330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:18:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.327+0000 7f405bd27700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40541f6dc0 0x7f4054156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:18:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.327+0000 7f405bd27700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40541f7240 0x7f4054156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:18:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.327+0000 7f405bd27700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4054140a50 con 0x7f40541f7240 2024-03-18T20:18:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.327+0000 7f405bd27700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40541429b0 con 0x7f40541f6dc0 2024-03-18T20:18:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.327+0000 7f405bd27700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4054142c80 con 0x7f4040005210 2024-03-18T20:18:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.327+0000 7f405a2c4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40541f7240 0x7f4054156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:18:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.327+0000 7f405a2c4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40541f7240 0x7f4054156580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51346/0 (socket says 172.21.15.1:51346) 2024-03-18T20:18:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.327+0000 7f405a2c4700 1 -- 172.21.15.1:0/1458227600 learned_addr learned my addr 172.21.15.1:0/1458227600 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:18:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.327+0000 7f40592c2700 1 --2- 172.21.15.1:0/1458227600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40541f6dc0 0x7f4054156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:18:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.327+0000 7f4059ac3700 1 --2- 172.21.15.1:0/1458227600 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4040005210 0x7f4054140330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:18:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.327+0000 7f405a2c4700 1 -- 172.21.15.1:0/1458227600 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4040005210 msgr2=0x7f4054140330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:18:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.327+0000 7f405a2c4700 1 --2- 172.21.15.1:0/1458227600 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4040005210 0x7f4054140330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:16.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.327+0000 7f405a2c4700 1 -- 172.21.15.1:0/1458227600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40541f6dc0 msgr2=0x7f4054156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:18:16.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.327+0000 7f405a2c4700 1 --2- 172.21.15.1:0/1458227600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40541f6dc0 0x7f4054156040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:16.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.327+0000 7f405a2c4700 1 -- 172.21.15.1:0/1458227600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4054142f60 con 0x7f40541f7240 2024-03-18T20:18:16.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.327+0000 7f405a2c4700 1 --2- 172.21.15.1:0/1458227600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40541f7240 0x7f4054156580 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f405406e100 tx=0x7f404c00a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:18:16.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.327+0000 7f404a7fc700 1 -- 172.21.15.1:0/1458227600 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c003c30 con 0x7f40541f7240 2024-03-18T20:18:16.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.328+0000 7f404a7fc700 1 -- 172.21.15.1:0/1458227600 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404c003db0 con 0x7f40541f7240 2024-03-18T20:18:16.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.328+0000 7f405bd27700 1 -- 172.21.15.1:0/1458227600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4054143240 con 0x7f40541f7240 2024-03-18T20:18:16.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.328+0000 7f404a7fc700 1 -- 172.21.15.1:0/1458227600 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c004260 con 0x7f40541f7240 2024-03-18T20:18:16.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.328+0000 7f405bd27700 1 -- 172.21.15.1:0/1458227600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4054143800 con 0x7f40541f7240 2024-03-18T20:18:16.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.328+0000 7f405bd27700 1 -- 172.21.15.1:0/1458227600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40540cbfc0 con 0x7f40541f7240 2024-03-18T20:18:16.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.329+0000 7f404a7fc700 1 -- 172.21.15.1:0/1458227600 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 112) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f404c0043c0 con 0x7f40541f7240 2024-03-18T20:18:16.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.329+0000 7f404a7fc700 1 --2- 172.21.15.1:0/1458227600 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f403806ae00 0x7f403806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:18:16.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.329+0000 7f404a7fc700 1 -- 172.21.15.1:0/1458227600 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f404c08a8d0 con 0x7f40541f7240 2024-03-18T20:18:16.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.330+0000 7f4059ac3700 1 --2- 172.21.15.1:0/1458227600 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f403806ae00 0x7f403806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:18:16.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.330+0000 7f4059ac3700 1 --2- 172.21.15.1:0/1458227600 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f403806ae00 0x7f403806d2c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f40540b3e60 tx=0x7f4044009040 comp rx=0 tx=0).ready entity=mgr.10614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:18:16.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.334+0000 7f404a7fc700 1 -- 172.21.15.1:0/1458227600 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f404c055760 con 0x7f40541f7240 2024-03-18T20:18:16.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.564+0000 7f405bd27700 1 -- 172.21.15.1:0/1458227600 --> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4054159fa0 con 0x7f403806ae00 2024-03-18T20:18:16.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.570+0000 7f404a7fc700 1 -- 172.21.15.1:0/1458227600 <== mgr.10614 v2:172.21.15.1:6832/76998 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+375 (secure 0 0 0) 0x7f4044012500 con 0x7f403806ae00 2024-03-18T20:18:16.574 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:18:16.574 INFO:teuthology.orchestra.run.smithi001.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T20:07:35", "first": "2024-03-18T20:08:00", "last": "2024-03-18T20:18:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2024-03-18T20:18:16.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.572+0000 7f405bd27700 1 -- 172.21.15.1:0/1458227600 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f403806ae00 msgr2=0x7f403806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:18:16.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.572+0000 7f405bd27700 1 --2- 172.21.15.1:0/1458227600 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f403806ae00 0x7f403806d2c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f40540b3e60 tx=0x7f4044009040 comp rx=0 tx=0).stop 2024-03-18T20:18:16.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.572+0000 7f405bd27700 1 -- 172.21.15.1:0/1458227600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40541f7240 msgr2=0x7f4054156580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:18:16.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.572+0000 7f405bd27700 1 --2- 172.21.15.1:0/1458227600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40541f7240 0x7f4054156580 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f405406e100 tx=0x7f404c00a860 comp rx=0 tx=0).stop 2024-03-18T20:18:16.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.572+0000 7f405bd27700 1 -- 172.21.15.1:0/1458227600 shutdown_connections 2024-03-18T20:18:16.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.572+0000 7f405bd27700 1 --2- 172.21.15.1:0/1458227600 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f403806ae00 0x7f403806d2c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:16.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.572+0000 7f405bd27700 1 --2- 172.21.15.1:0/1458227600 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4040005210 0x7f4054140330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:16.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.572+0000 7f405bd27700 1 --2- 172.21.15.1:0/1458227600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40541f7240 0x7f4054156580 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:16.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.572+0000 7f405bd27700 1 --2- 172.21.15.1:0/1458227600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40541f6dc0 0x7f4054156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:16.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.572+0000 7f405bd27700 1 -- 172.21.15.1:0/1458227600 >> 172.21.15.1:0/1458227600 conn(0x7f405406ab70 msgr2=0x7f405414db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:18:16.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.572+0000 7f405bd27700 1 -- 172.21.15.1:0/1458227600 shutdown_connections 2024-03-18T20:18:16.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.572+0000 7f405bd27700 1 -- 172.21.15.1:0/1458227600 wait complete. 2024-03-18T20:18:16.588 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T20:07:35", "first": "2024-03-18T20:08:00", "last": "2024-03-18T20:18:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2024-03-18T20:18:16.589 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2024-03-18T00:00:00', 'created': '2024-03-18T20:07:35', 'first': '2024-03-18T20:08:00', 'last': '2024-03-18T20:18:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2024-03-18T20:18:16.589 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-03-18T20:18:16.589 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-03-18T20:18:16.589 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-18T20:18:16.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.710+0000 7f4009c70700 1 Processor -- start 2024-03-18T20:18:16.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.710+0000 7f4009c70700 1 -- start start 2024-03-18T20:18:16.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.711+0000 7f4009c70700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f40040cc170 0x7f40040ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:18:16.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.711+0000 7f4009c70700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400406e270 con 0x7f400415a2e0 2024-03-18T20:18:16.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.711+0000 7f4009c70700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40040ae050 con 0x7f40040ae600 2024-03-18T20:18:16.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.711+0000 7f4009c70700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f400406f690 con 0x7f40040cc170 2024-03-18T20:18:16.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.711+0000 7f4003fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f40040cc170 0x7f40040ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:18:16.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.711+0000 7f4002ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f400415a2e0 0x7f400406ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52040/0 (socket says 172.21.15.1:52040) 2024-03-18T20:18:16.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.711+0000 7f4002ffd700 1 -- 172.21.15.1:0/1346948196 learned_addr learned my addr 172.21.15.1:0/1346948196 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:18:16.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.711+0000 7f4003fff700 1 -- 172.21.15.1:0/1346948196 >> v1:172.21.15.165:6789/0 conn(0x7f40040ae600 legacy=0x7f40040ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:18:16.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.711+0000 7f4003fff700 1 -- 172.21.15.1:0/1346948196 >> v1:172.21.15.1:6789/0 conn(0x7f400415a2e0 legacy=0x7f400406ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:18:16.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.711+0000 7f4003fff700 1 -- 172.21.15.1:0/1346948196 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40040add70 con 0x7f40040cc170 2024-03-18T20:18:16.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.711+0000 7f4003fff700 1 --2- 172.21.15.1:0/1346948196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40040cc170 0x7f40040ad790 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009d90 tx=0x7f3ffc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8133e460e71a2d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:18:16.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.712+0000 7f40027fc700 1 -- 172.21.15.1:0/1346948196 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc00bdd0 con 0x7f40040cc170 2024-03-18T20:18:16.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.712+0000 7f40027fc700 1 -- 172.21.15.1:0/1346948196 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ffc003a20 con 0x7f40040cc170 2024-03-18T20:18:16.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.712+0000 7f40027fc700 1 -- 172.21.15.1:0/1346948196 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc003dd0 con 0x7f40040cc170 2024-03-18T20:18:16.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.712+0000 7f4009c70700 1 -- 172.21.15.1:0/1346948196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40040cc170 msgr2=0x7f40040ad790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:18:16.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.712+0000 7f4009c70700 1 --2- 172.21.15.1:0/1346948196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40040cc170 0x7f40040ad790 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009d90 tx=0x7f3ffc00b330 comp rx=0 tx=0).stop 2024-03-18T20:18:16.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.712+0000 7f4009c70700 1 -- 172.21.15.1:0/1346948196 shutdown_connections 2024-03-18T20:18:16.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.712+0000 7f4009c70700 1 --2- 172.21.15.1:0/1346948196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40040cc170 0x7f40040ad790 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:16.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.712+0000 7f4009c70700 1 -- 172.21.15.1:0/1346948196 >> 172.21.15.1:0/1346948196 conn(0x7f400406b860 msgr2=0x7f400406bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:18:16.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.712+0000 7f4009c70700 1 -- 172.21.15.1:0/1346948196 shutdown_connections 2024-03-18T20:18:16.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.712+0000 7f4009c70700 1 -- 172.21.15.1:0/1346948196 wait complete. 2024-03-18T20:18:16.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.713+0000 7f4009c70700 1 Processor -- start 2024-03-18T20:18:16.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.714+0000 7f4009c70700 1 -- start start 2024-03-18T20:18:16.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.714+0000 7f4009c70700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40040ae600 0x7f4004146780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:18:16.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.714+0000 7f4009c70700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40040cc170 0x7f40041e6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:18:16.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.714+0000 7f4009c70700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f400415a2e0 0x7f40041566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:18:16.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.714+0000 7f4009c70700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40041e78e0 con 0x7f40040ae600 2024-03-18T20:18:16.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.714+0000 7f4009c70700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40041e6c60 con 0x7f400415a2e0 2024-03-18T20:18:16.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.714+0000 7f4009c70700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40041e6f40 con 0x7f40040cc170 2024-03-18T20:18:16.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.714+0000 7f40037fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40040ae600 0x7f4004146780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:18:16.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.714+0000 7f40037fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40040ae600 0x7f4004146780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51354/0 (socket says 172.21.15.1:51354) 2024-03-18T20:18:16.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.714+0000 7f40037fe700 1 -- 172.21.15.1:0/2758483405 learned_addr learned my addr 172.21.15.1:0/2758483405 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:18:16.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.714+0000 7f4002ffd700 1 --2- 172.21.15.1:0/2758483405 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40040cc170 0x7f40041e6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:18:16.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.714+0000 7f4003fff700 1 --2- 172.21.15.1:0/2758483405 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f400415a2e0 0x7f40041566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:18:16.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.715+0000 7f40037fe700 1 -- 172.21.15.1:0/2758483405 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40040cc170 msgr2=0x7f40041e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:18:16.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.715+0000 7f40037fe700 1 --2- 172.21.15.1:0/2758483405 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40040cc170 0x7f40041e6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:16.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.715+0000 7f40037fe700 1 -- 172.21.15.1:0/2758483405 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f400415a2e0 msgr2=0x7f40041566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:18:16.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.715+0000 7f40037fe700 1 --2- 172.21.15.1:0/2758483405 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f400415a2e0 0x7f40041566c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:16.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.715+0000 7f40037fe700 1 -- 172.21.15.1:0/2758483405 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ffc009a70 con 0x7f40040ae600 2024-03-18T20:18:16.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.715+0000 7f40037fe700 1 --2- 172.21.15.1:0/2758483405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40040ae600 0x7f4004146780 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f3ff400c4a0 tx=0x7f3ff400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:18:16.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.715+0000 7f3ffbfff700 1 -- 172.21.15.1:0/2758483405 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff40054e0 con 0x7f40040ae600 2024-03-18T20:18:16.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.715+0000 7f3ffbfff700 1 -- 172.21.15.1:0/2758483405 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff4003680 con 0x7f40040ae600 2024-03-18T20:18:16.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.715+0000 7f4009c70700 1 -- 172.21.15.1:0/2758483405 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40041e7220 con 0x7f40040ae600 2024-03-18T20:18:16.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.715+0000 7f3ffbfff700 1 -- 172.21.15.1:0/2758483405 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4003a30 con 0x7f40040ae600 2024-03-18T20:18:16.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.715+0000 7f4009c70700 1 -- 172.21.15.1:0/2758483405 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4004153af0 con 0x7f40040ae600 2024-03-18T20:18:16.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.716+0000 7f4009c70700 1 -- 172.21.15.1:0/2758483405 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40040676a0 con 0x7f40040ae600 2024-03-18T20:18:16.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.716+0000 7f3ffbfff700 1 -- 172.21.15.1:0/2758483405 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 112) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f3ff4003b90 con 0x7f40040ae600 2024-03-18T20:18:16.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.717+0000 7f3ffbfff700 1 --2- 172.21.15.1:0/2758483405 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3fe406ae00 0x7f3fe406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:18:16.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.717+0000 7f3ffbfff700 1 -- 172.21.15.1:0/2758483405 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6445+0+0 (secure 0 0 0) 0x7f3ff4088b00 con 0x7f40040ae600 2024-03-18T20:18:16.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.717+0000 7f4002ffd700 1 --2- 172.21.15.1:0/2758483405 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3fe406ae00 0x7f3fe406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:18:16.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.717+0000 7f4002ffd700 1 --2- 172.21.15.1:0/2758483405 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3fe406ae00 0x7f3fe406d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f400409dff0 tx=0x7f3fec009040 comp rx=0 tx=0).ready entity=mgr.10614 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:18:16.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.722+0000 7f3ffbfff700 1 -- 172.21.15.1:0/2758483405 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3ff4053980 con 0x7f40040ae600 2024-03-18T20:18:16.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.981+0000 7f4009c70700 1 -- 172.21.15.1:0/2758483405 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f40040b1880 con 0x7f40040ae600 2024-03-18T20:18:16.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.984+0000 7f3ffbfff700 1 -- 172.21.15.1:0/2758483405 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v112) v1 ==== 74+0+171126 (secure 0 0 0) 0x7f3ff4017020 con 0x7f40040ae600 2024-03-18T20:18:16.991 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:18:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.987+0000 7f4009c70700 1 -- 172.21.15.1:0/2758483405 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3fe406ae00 msgr2=0x7f3fe406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:18:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.987+0000 7f4009c70700 1 --2- 172.21.15.1:0/2758483405 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3fe406ae00 0x7f3fe406d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f400409dff0 tx=0x7f3fec009040 comp rx=0 tx=0).stop 2024-03-18T20:18:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.987+0000 7f4009c70700 1 -- 172.21.15.1:0/2758483405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40040ae600 msgr2=0x7f4004146780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:18:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.987+0000 7f4009c70700 1 --2- 172.21.15.1:0/2758483405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40040ae600 0x7f4004146780 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f3ff400c4a0 tx=0x7f3ff400c7b0 comp rx=0 tx=0).stop 2024-03-18T20:18:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.987+0000 7f4009c70700 1 -- 172.21.15.1:0/2758483405 shutdown_connections 2024-03-18T20:18:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.987+0000 7f4009c70700 1 --2- 172.21.15.1:0/2758483405 >> [v2:172.21.15.1:6832/76998,v1:172.21.15.1:6833/76998] conn(0x7f3fe406ae00 0x7f3fe406d2c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.987+0000 7f4009c70700 1 --2- 172.21.15.1:0/2758483405 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f400415a2e0 0x7f40041566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.987+0000 7f4009c70700 1 --2- 172.21.15.1:0/2758483405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40040ae600 0x7f4004146780 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.987+0000 7f4009c70700 1 --2- 172.21.15.1:0/2758483405 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f40040cc170 0x7f40041e6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:18:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.987+0000 7f4009c70700 1 -- 172.21.15.1:0/2758483405 >> 172.21.15.1:0/2758483405 conn(0x7f400406b860 msgr2=0x7f40040b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:18:16.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.987+0000 7f4009c70700 1 -- 172.21.15.1:0/2758483405 shutdown_connections 2024-03-18T20:18:16.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:18:16.987+0000 7f4009c70700 1 -- 172.21.15.1:0/2758483405 wait complete. 2024-03-18T20:18:17.003 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":112,"active_gid":10614,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":76998},{"type":"v1","addr":"172.21.15.1:6833","nonce":76998}]},"active_addr":"172.21.15.1:6833/76998","active_change":"2024-03-18T20:07:34.587824+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":9568,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":196,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":3201942299}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":181301955}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":2850855126}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":2331829563}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":2032368612}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":1513627829}]}]} 2024-03-18T20:18:17.046 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: y 2024-03-18T20:18:17.046 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-18T20:18:17.046 INFO:tasks.ceph.mgr.y:Stopping old one... 2024-03-18T20:18:17.047 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-18T20:18:17.047 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:18:17.076 INFO:tasks.ceph.mgr.y:Stopped 2024-03-18T20:18:17.076 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-18T20:18:17.119 INFO:tasks.ceph.mgr.y:Started 2024-03-18T20:18:17.300 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:17.295+0000 7ff511e4d240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:18:17.469 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:17.465+0000 7ff511e4d240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:18:17.743 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:17.739+0000 7ff511e4d240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:18:18.190 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:18.185+0000 7ff511e4d240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:18:18.408 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:18.404+0000 7ff511e4d240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:18:18.796 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:18.791+0000 7ff511e4d240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:18:19.409 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:19.405+0000 7ff511e4d240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:18:19.485 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:19.481+0000 7ff511e4d240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:18:19.621 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:19.617+0000 7ff511e4d240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:18:19.808 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:19.803+0000 7ff511e4d240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:18:19.884 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:19.880+0000 7ff511e4d240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:18:19.967 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:19.962+0000 7ff511e4d240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:18:20.049 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:20.044+0000 7ff511e4d240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:18:20.131 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:20.127+0000 7ff511e4d240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:18:20.309 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:20.305+0000 7ff511e4d240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:18:20.531 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:20.527+0000 7ff511e4d240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:18:20.692 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:20.687+0000 7ff511e4d240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:18:20.769 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:20.764+0000 7ff511e4d240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:18:21.243 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:21.238+0000 7ff511e4d240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:18:21.594 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:21.590+0000 7ff511e4d240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:18:21.769 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:21.764+0000 7ff511e4d240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:18:22.182 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:22.177+0000 7ff511e4d240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:18:22.359 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:22.354+0000 7ff511e4d240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:18:22.737 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:22.733+0000 7ff511e4d240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:18:22.832 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:22.828+0000 7ff511e4d240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:18:22.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:22.989+0000 7ff48722f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:18:22.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:22.989+0000 7ff48722f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:18:22.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:22.989+0000 7ff48722f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:18:22.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:22.989+0000 7ff48722f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:18:22.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:22.989+0000 7ff48722f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:18:22.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:22.989+0000 7ff484229700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:18:22.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:22.989+0000 7ff484229700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:18:22.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:22.989+0000 7ff484229700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:18:22.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:22.989+0000 7ff484229700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:18:22.994 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:18:22.989+0000 7ff484229700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:18:38.997 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:18:39.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:19:00.991 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:19:00.986+0000 7ff48f23f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:19:00.992 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:19:00.986+0000 7ff48f23f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:19:00.992 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:19:00.986+0000 7ff48f23f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:19:00.992 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:19:00.986+0000 7ff48f23f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:19:00.992 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:19:00.986+0000 7ff48f23f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:19:09.101 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:19:09.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:19:39.181 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:19:39.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:20:09.318 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:20:09.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:20:39.389 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:20:39.394 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:21:09.535 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:21:09.538 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:21:39.594 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:21:39.598 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:22:09.735 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:22:09.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:22:39.822 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:22:39.825 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:23:10.152 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:23:10.156 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:23:17.120 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/test_restart 1m 2024-03-18T20:23:17.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.259+0000 7fd179782700 1 Processor -- start 2024-03-18T20:23:17.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.259+0000 7fd179782700 1 -- start start 2024-03-18T20:23:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd179782700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1740cfb50 0x7fd1740d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd179782700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1740cee80 0x7fd1740b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd179782700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1740b3370 con 0x7fd1740cff30 2024-03-18T20:23:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd179782700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1740cf310 con 0x7fd1740cee80 2024-03-18T20:23:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd179782700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1740b2760 con 0x7fd1740cfb50 2024-03-18T20:23:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd1737fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1740cee80 0x7fd1740b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd1737fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1740cee80 0x7fd1740b3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57482/0 (socket says 172.21.15.1:57482) 2024-03-18T20:23:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd1737fe700 1 -- 172.21.15.1:0/2146450248 learned_addr learned my addr 172.21.15.1:0/2146450248 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd1727fc700 1 --1- 172.21.15.1:0/2146450248 >> v1:172.21.15.1:6789/0 conn(0x7fd1740cff30 0x7fd1740b3800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43148/0 (socket says 172.21.15.1:43148) 2024-03-18T20:23:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd172ffd700 1 --2- 172.21.15.1:0/2146450248 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1740cfb50 0x7fd1740d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:17.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd1737fe700 1 -- 172.21.15.1:0/2146450248 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1740cfb50 msgr2=0x7fd1740d3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd1737fe700 1 --2- 172.21.15.1:0/2146450248 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1740cfb50 0x7fd1740d3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd1737fe700 1 -- 172.21.15.1:0/2146450248 >> v1:172.21.15.1:6789/0 conn(0x7fd1740cff30 legacy=0x7fd1740b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd171ffb700 1 -- 172.21.15.1:0/2146450248 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948960242 0 0) 0x7fd1740b3370 con 0x7fd1740cff30 2024-03-18T20:23:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd1737fe700 1 -- 172.21.15.1:0/2146450248 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1740b2a40 con 0x7fd1740cee80 2024-03-18T20:23:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd172ffd700 1 --2- 172.21.15.1:0/2146450248 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1740cfb50 0x7fd1740d3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd1737fe700 1 --2- 172.21.15.1:0/2146450248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1740cee80 0x7fd1740b3f10 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fd16c009e50 tx=0x7fd16c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c2cff67abc9f52 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd171ffb700 1 -- 172.21.15.1:0/2146450248 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c00e070 con 0x7fd1740cee80 2024-03-18T20:23:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd171ffb700 1 -- 172.21.15.1:0/2146450248 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16c00b9c0 con 0x7fd1740cee80 2024-03-18T20:23:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.260+0000 7fd171ffb700 1 -- 172.21.15.1:0/2146450248 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c00bd70 con 0x7fd1740cee80 2024-03-18T20:23:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.261+0000 7fd179782700 1 -- 172.21.15.1:0/2146450248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1740cee80 msgr2=0x7fd1740b3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.261+0000 7fd179782700 1 --2- 172.21.15.1:0/2146450248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1740cee80 0x7fd1740b3f10 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fd16c009e50 tx=0x7fd16c00b330 comp rx=0 tx=0).stop 2024-03-18T20:23:17.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.261+0000 7fd179782700 1 -- 172.21.15.1:0/2146450248 shutdown_connections 2024-03-18T20:23:17.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.261+0000 7fd179782700 1 --2- 172.21.15.1:0/2146450248 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1740cee80 0x7fd1740b3f10 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:17.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.261+0000 7fd179782700 1 --2- 172.21.15.1:0/2146450248 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1740cfb50 0x7fd1740d3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:17.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.261+0000 7fd179782700 1 -- 172.21.15.1:0/2146450248 >> 172.21.15.1:0/2146450248 conn(0x7fd1740d3140 msgr2=0x7fd1740d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:17.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.261+0000 7fd179782700 1 -- 172.21.15.1:0/2146450248 shutdown_connections 2024-03-18T20:23:17.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.261+0000 7fd179782700 1 -- 172.21.15.1:0/2146450248 wait complete. 2024-03-18T20:23:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.262+0000 7fd179782700 1 Processor -- start 2024-03-18T20:23:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.262+0000 7fd179782700 1 -- start start 2024-03-18T20:23:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.263+0000 7fd179782700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1740cee80 0x7fd1740ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.263+0000 7fd179782700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1740cfb50 0x7fd1740ca210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.263+0000 7fd179782700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1740cff30 0x7fd1740c8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.263+0000 7fd179782700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1740caa60 con 0x7fd1740cfb50 2024-03-18T20:23:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.263+0000 7fd179782700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1740cad40 con 0x7fd1740cee80 2024-03-18T20:23:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.263+0000 7fd179782700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1740cb020 con 0x7fd1740cff30 2024-03-18T20:23:17.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.263+0000 7fd1727fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1740cfb50 0x7fd1740ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.263+0000 7fd1727fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1740cfb50 0x7fd1740ca210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57490/0 (socket says 172.21.15.1:57490) 2024-03-18T20:23:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.263+0000 7fd1727fc700 1 -- 172.21.15.1:0/1893155330 learned_addr learned my addr 172.21.15.1:0/1893155330 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.264+0000 7fd172ffd700 1 --2- 172.21.15.1:0/1893155330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1740cee80 0x7fd1740ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.264+0000 7fd1737fe700 1 --2- 172.21.15.1:0/1893155330 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1740cff30 0x7fd1740c8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.264+0000 7fd1727fc700 1 -- 172.21.15.1:0/1893155330 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1740cff30 msgr2=0x7fd1740c8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.264+0000 7fd1727fc700 1 --2- 172.21.15.1:0/1893155330 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1740cff30 0x7fd1740c8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.264+0000 7fd1727fc700 1 -- 172.21.15.1:0/1893155330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1740cee80 msgr2=0x7fd1740ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.264+0000 7fd1727fc700 1 --2- 172.21.15.1:0/1893155330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1740cee80 0x7fd1740ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.264+0000 7fd1727fc700 1 -- 172.21.15.1:0/1893155330 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c009b30 con 0x7fd1740cfb50 2024-03-18T20:23:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.264+0000 7fd1727fc700 1 --2- 172.21.15.1:0/1893155330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1740cfb50 0x7fd1740ca210 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7fd16400c550 tx=0x7fd16400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:17.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.264+0000 7fd15b7fe700 1 -- 172.21.15.1:0/1893155330 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164004c80 con 0x7fd1740cfb50 2024-03-18T20:23:17.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.264+0000 7fd15b7fe700 1 -- 172.21.15.1:0/1893155330 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd164004e00 con 0x7fd1740cfb50 2024-03-18T20:23:17.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.264+0000 7fd179782700 1 -- 172.21.15.1:0/1893155330 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1740cb300 con 0x7fd1740cfb50 2024-03-18T20:23:17.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.264+0000 7fd15b7fe700 1 -- 172.21.15.1:0/1893155330 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164007310 con 0x7fd1740cfb50 2024-03-18T20:23:17.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.264+0000 7fd179782700 1 -- 172.21.15.1:0/1893155330 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1740cbb00 con 0x7fd1740cfb50 2024-03-18T20:23:17.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.265+0000 7fd179782700 1 -- 172.21.15.1:0/1893155330 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd174073350 con 0x7fd1740cfb50 2024-03-18T20:23:17.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.265+0000 7fd15b7fe700 1 -- 172.21.15.1:0/1893155330 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 118) v1 ==== 89268+0+0 (secure 0 0 0) 0x7fd164007470 con 0x7fd1740cfb50 2024-03-18T20:23:17.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.266+0000 7fd15b7fe700 1 --2- 172.21.15.1:0/1893155330 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd16006ae00 0x7fd16006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:17.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.266+0000 7fd15b7fe700 1 -- 172.21.15.1:0/1893155330 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6703+0+0 (secure 0 0 0) 0x7fd1640896b0 con 0x7fd1740cfb50 2024-03-18T20:23:17.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.266+0000 7fd172ffd700 1 --2- 172.21.15.1:0/1893155330 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd16006ae00 0x7fd16006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:17.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.266+0000 7fd172ffd700 1 --2- 172.21.15.1:0/1893155330 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd16006ae00 0x7fd16006d2c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd1680088d0 tx=0x7fd168008040 comp rx=0 tx=0).ready entity=mgr.10713 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:17.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.271+0000 7fd15b7fe700 1 -- 172.21.15.1:0/1893155330 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd164054310 con 0x7fd1740cfb50 2024-03-18T20:23:17.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.501+0000 7fd179782700 1 -- 172.21.15.1:0/1893155330 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd174160d90 con 0x7fd16006ae00 2024-03-18T20:23:17.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.546+0000 7fd15b7fe700 1 -- 172.21.15.1:0/1893155330 <== mgr.10713 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fd16800c040 con 0x7fd16006ae00 2024-03-18T20:23:17.551 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2024-03-18T20:23:17.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.549+0000 7fd179782700 1 -- 172.21.15.1:0/1893155330 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd16006ae00 msgr2=0x7fd16006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:17.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.549+0000 7fd179782700 1 --2- 172.21.15.1:0/1893155330 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd16006ae00 0x7fd16006d2c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd1680088d0 tx=0x7fd168008040 comp rx=0 tx=0).stop 2024-03-18T20:23:17.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.549+0000 7fd179782700 1 -- 172.21.15.1:0/1893155330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1740cfb50 msgr2=0x7fd1740ca210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:17.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.549+0000 7fd179782700 1 --2- 172.21.15.1:0/1893155330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1740cfb50 0x7fd1740ca210 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7fd16400c550 tx=0x7fd16400c910 comp rx=0 tx=0).stop 2024-03-18T20:23:17.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.549+0000 7fd179782700 1 -- 172.21.15.1:0/1893155330 shutdown_connections 2024-03-18T20:23:17.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.549+0000 7fd179782700 1 --2- 172.21.15.1:0/1893155330 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd16006ae00 0x7fd16006d2c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:17.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.549+0000 7fd179782700 1 --2- 172.21.15.1:0/1893155330 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1740cff30 0x7fd1740c8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:17.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.549+0000 7fd179782700 1 --2- 172.21.15.1:0/1893155330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1740cee80 0x7fd1740ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:17.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.549+0000 7fd179782700 1 --2- 172.21.15.1:0/1893155330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1740cfb50 0x7fd1740ca210 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:17.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.549+0000 7fd179782700 1 -- 172.21.15.1:0/1893155330 >> 172.21.15.1:0/1893155330 conn(0x7fd1740d3140 msgr2=0x7fd1740b5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:17.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.549+0000 7fd179782700 1 -- 172.21.15.1:0/1893155330 shutdown_connections 2024-03-18T20:23:17.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.549+0000 7fd179782700 1 -- 172.21.15.1:0/1893155330 wait complete. 2024-03-18T20:23:17.565 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2024-03-18T20:23:17.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:17.566 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-03-18T20:23:17.589 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_08_00_UTC 2024-03-18T20:23:17.589 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_09_00_UTC 2024-03-18T20:23:17.589 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_10_00_UTC 2024-03-18T20:23:17.589 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_11_00_UTC 2024-03-18T20:23:17.589 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_12_00_UTC 2024-03-18T20:23:17.589 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_13_00_UTC 2024-03-18T20:23:17.589 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_14_00_UTC 2024-03-18T20:23:17.589 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_15_00_UTC 2024-03-18T20:23:17.589 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_16_00_UTC 2024-03-18T20:23:17.589 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_17_00_UTC 2024-03-18T20:23:17.590 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_18_00_UTC 2024-03-18T20:23:17.590 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_19_00_UTC 2024-03-18T20:23:17.590 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_20_00_UTC 2024-03-18T20:23:17.590 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_21_00_UTC 2024-03-18T20:23:17.590 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_22_00_UTC 2024-03-18T20:23:17.590 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_23_00_UTC 2024-03-18T20:23:17.590 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-03-18-20_08_00_UTC', 'scheduled-2024-03-18-20_09_00_UTC', 'scheduled-2024-03-18-20_10_00_UTC', 'scheduled-2024-03-18-20_11_00_UTC', 'scheduled-2024-03-18-20_12_00_UTC', 'scheduled-2024-03-18-20_13_00_UTC', 'scheduled-2024-03-18-20_14_00_UTC', 'scheduled-2024-03-18-20_15_00_UTC', 'scheduled-2024-03-18-20_16_00_UTC', 'scheduled-2024-03-18-20_17_00_UTC', 'scheduled-2024-03-18-20_18_00_UTC', 'scheduled-2024-03-18-20_19_00_UTC', 'scheduled-2024-03-18-20_20_00_UTC', 'scheduled-2024-03-18-20_21_00_UTC', 'scheduled-2024-03-18-20_22_00_UTC', 'scheduled-2024-03-18-20_23_00_UTC'] 2024-03-18T20:23:17.590 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2024-03-18T20:23:17.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.752+0000 7f6741cf4700 1 Processor -- start 2024-03-18T20:23:17.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.752+0000 7f6741cf4700 1 -- start start 2024-03-18T20:23:17.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.753+0000 7f6741cf4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f673c15d5b0 0x7f673c15d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.753+0000 7f6741cf4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f673c099a40 0x7f673c099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.753+0000 7f6741cf4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f673c09a3e0 0x7f673c06ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.753+0000 7f6741cf4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673c0cbbb0 con 0x7f673c099a40 2024-03-18T20:23:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.753+0000 7f6741cf4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673c09a830 con 0x7f673c09a3e0 2024-03-18T20:23:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.753+0000 7f6741cf4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673c0b3b00 con 0x7f673c15d5b0 2024-03-18T20:23:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.753+0000 7f673affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f673c099a40 0x7f673c099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.753+0000 7f673affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f673c099a40 0x7f673c099ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57500/0 (socket says 172.21.15.1:57500) 2024-03-18T20:23:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.753+0000 7f673affd700 1 -- 172.21.15.1:0/3045987267 learned_addr learned my addr 172.21.15.1:0/3045987267 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.754+0000 7f673b7fe700 1 --2- 172.21.15.1:0/3045987267 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f673c15d5b0 0x7f673c15d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:17.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.754+0000 7f673bfff700 1 --2- 172.21.15.1:0/3045987267 >> v2:172.21.15.165:3300/0 conn(0x7f673c09a3e0 0x7f673c06ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:17.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.754+0000 7f673affd700 1 -- 172.21.15.1:0/3045987267 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f673c15d5b0 msgr2=0x7f673c15d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:17.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.754+0000 7f673affd700 1 --2- 172.21.15.1:0/3045987267 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f673c15d5b0 0x7f673c15d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:17.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.754+0000 7f673affd700 1 -- 172.21.15.1:0/3045987267 >> v2:172.21.15.165:3300/0 conn(0x7f673c09a3e0 msgr2=0x7f673c06ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:17.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.754+0000 7f673affd700 1 --2- 172.21.15.1:0/3045987267 >> v2:172.21.15.165:3300/0 conn(0x7f673c09a3e0 0x7f673c06ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:17.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.754+0000 7f673affd700 1 -- 172.21.15.1:0/3045987267 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673c0cf7f0 con 0x7f673c099a40 2024-03-18T20:23:17.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.754+0000 7f673b7fe700 1 --2- 172.21.15.1:0/3045987267 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f673c15d5b0 0x7f673c15d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:17.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.754+0000 7f673bfff700 1 --2- 172.21.15.1:0/3045987267 >> v2:172.21.15.165:3300/0 conn(0x7f673c09a3e0 0x7f673c06ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:17.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.754+0000 7f673affd700 1 --2- 172.21.15.1:0/3045987267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f673c099a40 0x7f673c099ea0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f673c0720b0 tx=0x7f6724009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa816ef725368d23 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:17.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.754+0000 7f673a7fc700 1 -- 172.21.15.1:0/3045987267 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672400b870 con 0x7f673c099a40 2024-03-18T20:23:17.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.754+0000 7f673a7fc700 1 -- 172.21.15.1:0/3045987267 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672400b9d0 con 0x7f673c099a40 2024-03-18T20:23:17.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.754+0000 7f673a7fc700 1 -- 172.21.15.1:0/3045987267 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672400bd60 con 0x7f673c099a40 2024-03-18T20:23:17.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.755+0000 7f6741cf4700 1 -- 172.21.15.1:0/3045987267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f673c099a40 msgr2=0x7f673c099ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:17.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.755+0000 7f6741cf4700 1 --2- 172.21.15.1:0/3045987267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f673c099a40 0x7f673c099ea0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f673c0720b0 tx=0x7f6724009bd0 comp rx=0 tx=0).stop 2024-03-18T20:23:17.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.755+0000 7f6741cf4700 1 -- 172.21.15.1:0/3045987267 shutdown_connections 2024-03-18T20:23:17.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.755+0000 7f6741cf4700 1 --2- 172.21.15.1:0/3045987267 >> v2:172.21.15.165:3300/0 conn(0x7f673c09a3e0 0x7f673c06ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:17.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.755+0000 7f6741cf4700 1 --2- 172.21.15.1:0/3045987267 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f673c15d5b0 0x7f673c15d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:17.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.755+0000 7f6741cf4700 1 --2- 172.21.15.1:0/3045987267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f673c099a40 0x7f673c099ea0 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:17.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.755+0000 7f6741cf4700 1 -- 172.21.15.1:0/3045987267 >> 172.21.15.1:0/3045987267 conn(0x7f673c0b2bd0 msgr2=0x7f673c0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:17.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.755+0000 7f6741cf4700 1 -- 172.21.15.1:0/3045987267 shutdown_connections 2024-03-18T20:23:17.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.755+0000 7f6741cf4700 1 -- 172.21.15.1:0/3045987267 wait complete. 2024-03-18T20:23:17.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.756+0000 7f6741cf4700 1 Processor -- start 2024-03-18T20:23:17.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.756+0000 7f6741cf4700 1 -- start start 2024-03-18T20:23:17.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.757+0000 7f6741cf4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f673c099a40 0x7f673c0b5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:17.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.757+0000 7f6741cf4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f673c09a3e0 0x7f673c1493b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.757+0000 7f6741cf4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f673c15d5b0 0x7f673c0ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.757+0000 7f6741cf4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c0b6150 con 0x7f673c099a40 2024-03-18T20:23:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.757+0000 7f6741cf4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c0b40e0 con 0x7f673c09a3e0 2024-03-18T20:23:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.757+0000 7f6741cf4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673c0b43c0 con 0x7f673c15d5b0 2024-03-18T20:23:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.757+0000 7f673b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f673c099a40 0x7f673c0b5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.757+0000 7f673b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f673c099a40 0x7f673c0b5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57504/0 (socket says 172.21.15.1:57504) 2024-03-18T20:23:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.757+0000 7f673b7fe700 1 -- 172.21.15.1:0/2135170946 learned_addr learned my addr 172.21.15.1:0/2135170946 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.757+0000 7f673bfff700 1 --2- 172.21.15.1:0/2135170946 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f673c15d5b0 0x7f673c0ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.757+0000 7f673affd700 1 --2- 172.21.15.1:0/2135170946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f673c09a3e0 0x7f673c1493b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.757+0000 7f673b7fe700 1 -- 172.21.15.1:0/2135170946 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f673c15d5b0 msgr2=0x7f673c0ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.757+0000 7f673b7fe700 1 --2- 172.21.15.1:0/2135170946 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f673c15d5b0 0x7f673c0ccf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:17.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.757+0000 7f673b7fe700 1 -- 172.21.15.1:0/2135170946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f673c09a3e0 msgr2=0x7f673c1493b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:17.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.757+0000 7f673b7fe700 1 --2- 172.21.15.1:0/2135170946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f673c09a3e0 0x7f673c1493b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:17.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.757+0000 7f673b7fe700 1 -- 172.21.15.1:0/2135170946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6724009550 con 0x7f673c099a40 2024-03-18T20:23:17.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.757+0000 7f673bfff700 1 --2- 172.21.15.1:0/2135170946 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f673c15d5b0 0x7f673c0ccf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:23:17.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.757+0000 7f673affd700 1 --2- 172.21.15.1:0/2135170946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f673c09a3e0 0x7f673c1493b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:23:17.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.757+0000 7f673b7fe700 1 --2- 172.21.15.1:0/2135170946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f673c099a40 0x7f673c0b5c10 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f672c00c750 tx=0x7f672c00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:17.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.758+0000 7f6733fff700 1 -- 172.21.15.1:0/2135170946 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c011800 con 0x7f673c099a40 2024-03-18T20:23:17.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.758+0000 7f6733fff700 1 -- 172.21.15.1:0/2135170946 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672c011960 con 0x7f673c099a40 2024-03-18T20:23:17.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.758+0000 7f6733fff700 1 -- 172.21.15.1:0/2135170946 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c011cd0 con 0x7f673c099a40 2024-03-18T20:23:17.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.758+0000 7f6741cf4700 1 -- 172.21.15.1:0/2135170946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f673c0b46a0 con 0x7f673c099a40 2024-03-18T20:23:17.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.758+0000 7f6741cf4700 1 -- 172.21.15.1:0/2135170946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f673c0b4c60 con 0x7f673c099a40 2024-03-18T20:23:17.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.758+0000 7f6741cf4700 1 -- 172.21.15.1:0/2135170946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673c0cb7b0 con 0x7f673c099a40 2024-03-18T20:23:17.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.759+0000 7f6733fff700 1 -- 172.21.15.1:0/2135170946 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 118) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f672c021350 con 0x7f673c099a40 2024-03-18T20:23:17.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.759+0000 7f6733fff700 1 --2- 172.21.15.1:0/2135170946 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f671c06ae00 0x7f671c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:17.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.759+0000 7f6733fff700 1 -- 172.21.15.1:0/2135170946 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6703+0+0 (secure 0 0 0) 0x7f672c089a40 con 0x7f673c099a40 2024-03-18T20:23:17.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.759+0000 7f673affd700 1 --2- 172.21.15.1:0/2135170946 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f671c06ae00 0x7f671c06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:17.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.760+0000 7f673affd700 1 --2- 172.21.15.1:0/2135170946 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f671c06ae00 0x7f671c06d2c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6724014040 tx=0x7f6724008030 comp rx=0 tx=0).ready entity=mgr.10713 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:17.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.764+0000 7f6733fff700 1 -- 172.21.15.1:0/2135170946 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f673c0cb7b0 con 0x7f673c099a40 2024-03-18T20:23:17.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.990+0000 7f6741cf4700 1 -- 172.21.15.1:0/2135170946 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f673c0d07b0 con 0x7f671c06ae00 2024-03-18T20:23:18.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.996+0000 7f6733fff700 1 -- 172.21.15.1:0/2135170946 <== mgr.10713 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+376 (secure 0 0 0) 0x7f672400bda0 con 0x7f671c06ae00 2024-03-18T20:23:18.000 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:18.000 INFO:teuthology.orchestra.run.smithi001.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T20:07:35", "first": "2024-03-18T20:08:00", "last": "2024-03-18T20:23:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2024-03-18T20:23:18.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.998+0000 7f6741cf4700 1 -- 172.21.15.1:0/2135170946 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f671c06ae00 msgr2=0x7f671c06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:18.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.998+0000 7f6741cf4700 1 --2- 172.21.15.1:0/2135170946 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f671c06ae00 0x7f671c06d2c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6724014040 tx=0x7f6724008030 comp rx=0 tx=0).stop 2024-03-18T20:23:18.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.998+0000 7f6741cf4700 1 -- 172.21.15.1:0/2135170946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f673c099a40 msgr2=0x7f673c0b5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:18.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.998+0000 7f6741cf4700 1 --2- 172.21.15.1:0/2135170946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f673c099a40 0x7f673c0b5c10 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f672c00c750 tx=0x7f672c00cb10 comp rx=0 tx=0).stop 2024-03-18T20:23:18.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.998+0000 7f6741cf4700 1 -- 172.21.15.1:0/2135170946 shutdown_connections 2024-03-18T20:23:18.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.998+0000 7f6741cf4700 1 --2- 172.21.15.1:0/2135170946 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f671c06ae00 0x7f671c06d2c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:18.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.998+0000 7f6741cf4700 1 --2- 172.21.15.1:0/2135170946 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f673c15d5b0 0x7f673c0ccf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:18.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.998+0000 7f6741cf4700 1 --2- 172.21.15.1:0/2135170946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f673c099a40 0x7f673c0b5c10 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:18.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.998+0000 7f6741cf4700 1 --2- 172.21.15.1:0/2135170946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f673c09a3e0 0x7f673c1493b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:18.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.998+0000 7f6741cf4700 1 -- 172.21.15.1:0/2135170946 >> 172.21.15.1:0/2135170946 conn(0x7f673c0b2bd0 msgr2=0x7f673c06fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:18.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.998+0000 7f6741cf4700 1 -- 172.21.15.1:0/2135170946 shutdown_connections 2024-03-18T20:23:18.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:17.998+0000 7f6741cf4700 1 -- 172.21.15.1:0/2135170946 wait complete. 2024-03-18T20:23:18.014 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T20:07:35", "first": "2024-03-18T20:08:00", "last": "2024-03-18T20:23:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2024-03-18T20:23:18.015 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2024-03-18T00:00:00', 'created': '2024-03-18T20:07:35', 'first': '2024-03-18T20:08:00', 'last': '2024-03-18T20:23:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2024-03-18T20:23:18.015 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 16 2024-03-18T20:23:18.015 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 16 2024-03-18T20:23:18.015 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/test_restart 1m 2024-03-18T20:23:18.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.139+0000 7f7e9ed6d700 1 Processor -- start 2024-03-18T20:23:18.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.139+0000 7f7e9ed6d700 1 -- start start 2024-03-18T20:23:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.140+0000 7f7e9ed6d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7e980b4590 0x7f7e980ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.140+0000 7f7e9ed6d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e9815e250 0x7f7e980ad800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.140+0000 7f7e9ed6d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9815e870 con 0x7f7e980b4970 2024-03-18T20:23:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.140+0000 7f7e9ed6d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e98161d60 con 0x7f7e980b4590 2024-03-18T20:23:18.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.140+0000 7f7e9ed6d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e98160fa0 con 0x7f7e9815e250 2024-03-18T20:23:18.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.140+0000 7f7e97fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7e980b4970 0x7f7e9815db40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43164/0 (socket says 172.21.15.1:43164) 2024-03-18T20:23:18.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.140+0000 7f7e97fff700 1 -- 172.21.15.1:0/1094960687 learned_addr learned my addr 172.21.15.1:0/1094960687 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:18.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.140+0000 7f7e9cb09700 1 --2- 172.21.15.1:0/1094960687 >> v2:172.21.15.165:3300/0 conn(0x7f7e980b4590 0x7f7e980ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:18.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.140+0000 7f7e9d30a700 1 --2- 172.21.15.1:0/1094960687 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e9815e250 0x7f7e980ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:18.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.140+0000 7f7e977fe700 1 -- 172.21.15.1:0/1094960687 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3945064156 0 0) 0x7f7e9815e870 con 0x7f7e980b4970 2024-03-18T20:23:18.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e977fe700 1 -- 172.21.15.1:0/1094960687 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e880035b0 con 0x7f7e980b4970 2024-03-18T20:23:18.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e977fe700 1 -- 172.21.15.1:0/1094960687 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1186052528 0 0) 0x7f7e880035b0 con 0x7f7e980b4970 2024-03-18T20:23:18.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e977fe700 1 -- 172.21.15.1:0/1094960687 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e9815e250 msgr2=0x7f7e980ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:18.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e977fe700 1 --2- 172.21.15.1:0/1094960687 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e9815e250 0x7f7e980ad800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:18.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e977fe700 1 -- 172.21.15.1:0/1094960687 >> v2:172.21.15.165:3300/0 conn(0x7f7e980b4590 msgr2=0x7f7e980ad2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:18.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e977fe700 1 --2- 172.21.15.1:0/1094960687 >> v2:172.21.15.165:3300/0 conn(0x7f7e980b4590 0x7f7e980ad2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:18.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e977fe700 1 -- 172.21.15.1:0/1094960687 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e98161280 con 0x7f7e980b4970 2024-03-18T20:23:18.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e977fe700 1 -- 172.21.15.1:0/1094960687 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f7e84001670 con 0x7f7e980b4970 2024-03-18T20:23:18.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e977fe700 1 -- 172.21.15.1:0/1094960687 >> v1:172.21.15.1:6789/0 conn(0x7f7e980b4970 legacy=0x7f7e9815db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:18.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e977fe700 1 --2- 172.21.15.1:0/1094960687 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e88003a00 0x7f7e88003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:18.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e977fe700 1 --2- 172.21.15.1:0/1094960687 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e88004850 0x7f7e88004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:18.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e977fe700 1 --2- 172.21.15.1:0/1094960687 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e9815e250 0x7f7e88005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:18.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e977fe700 1 -- 172.21.15.1:0/1094960687 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e98161d60 con 0x7f7e88004850 2024-03-18T20:23:18.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e977fe700 1 -- 172.21.15.1:0/1094960687 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e98160fa0 con 0x7f7e88003a00 2024-03-18T20:23:18.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e977fe700 1 -- 172.21.15.1:0/1094960687 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8800ba40 con 0x7f7e9815e250 2024-03-18T20:23:18.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e97fff700 1 --2- 172.21.15.1:0/1094960687 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e88004850 0x7f7e88004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:18.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e9cb09700 1 --2- 172.21.15.1:0/1094960687 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e88003a00 0x7f7e88003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:18.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e9d30a700 1 --2- 172.21.15.1:0/1094960687 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e9815e250 0x7f7e88005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:18.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e97fff700 1 -- 172.21.15.1:0/1094960687 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e9815e250 msgr2=0x7f7e88005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:18.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e97fff700 1 --2- 172.21.15.1:0/1094960687 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e9815e250 0x7f7e88005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:18.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e97fff700 1 -- 172.21.15.1:0/1094960687 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e88003a00 msgr2=0x7f7e88003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:18.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e97fff700 1 --2- 172.21.15.1:0/1094960687 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e88003a00 0x7f7e88003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:18.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e97fff700 1 -- 172.21.15.1:0/1094960687 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e880035b0 con 0x7f7e88004850 2024-03-18T20:23:18.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.141+0000 7f7e97fff700 1 --2- 172.21.15.1:0/1094960687 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e88004850 0x7f7e88004cd0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f7e980cf670 tx=0x7f7e88005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f71611d03782a75b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:18.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.142+0000 7f7e977fe700 1 -- 172.21.15.1:0/1094960687 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e84007c40 con 0x7f7e88004850 2024-03-18T20:23:18.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.142+0000 7f7e977fe700 1 -- 172.21.15.1:0/1094960687 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8400cda0 con 0x7f7e88004850 2024-03-18T20:23:18.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.142+0000 7f7e977fe700 1 -- 172.21.15.1:0/1094960687 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e84005ba0 con 0x7f7e88004850 2024-03-18T20:23:18.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.142+0000 7f7e9ed6d700 1 -- 172.21.15.1:0/1094960687 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e88004850 msgr2=0x7f7e88004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:18.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.142+0000 7f7e9ed6d700 1 --2- 172.21.15.1:0/1094960687 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e88004850 0x7f7e88004cd0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f7e980cf670 tx=0x7f7e88005880 comp rx=0 tx=0).stop 2024-03-18T20:23:18.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.142+0000 7f7e9d30a700 1 -- 172.21.15.1:0/1094960687 reap_dead start 2024-03-18T20:23:18.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.142+0000 7f7e9ed6d700 1 -- 172.21.15.1:0/1094960687 shutdown_connections 2024-03-18T20:23:18.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.142+0000 7f7e9ed6d700 1 -- 172.21.15.1:0/1094960687 >> 172.21.15.1:0/1094960687 conn(0x7f7e9806ab70 msgr2=0x7f7e9806af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:18.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.142+0000 7f7e9ed6d700 1 -- 172.21.15.1:0/1094960687 shutdown_connections 2024-03-18T20:23:18.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.142+0000 7f7e9ed6d700 1 -- 172.21.15.1:0/1094960687 wait complete. 2024-03-18T20:23:18.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.144+0000 7f7e9ed6d700 1 Processor -- start 2024-03-18T20:23:18.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.144+0000 7f7e9ed6d700 1 -- start start 2024-03-18T20:23:18.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.145+0000 7f7e9ed6d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e88004850 0x7f7e98157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:18.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.145+0000 7f7e9ed6d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e9806f370 0x7f7e98150840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:18.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.145+0000 7f7e9ed6d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e981e8e80 0x7f7e981e9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:18.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.145+0000 7f7e9ed6d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e981e7eb0 con 0x7f7e88004850 2024-03-18T20:23:18.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.145+0000 7f7e9ed6d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9806f870 con 0x7f7e9806f370 2024-03-18T20:23:18.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.145+0000 7f7e9ed6d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e981e89f0 con 0x7f7e981e8e80 2024-03-18T20:23:18.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.145+0000 7f7e9cb09700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e88004850 0x7f7e98157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:18.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.145+0000 7f7e9cb09700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e88004850 0x7f7e98157f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57512/0 (socket says 172.21.15.1:57512) 2024-03-18T20:23:18.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.145+0000 7f7e9cb09700 1 -- 172.21.15.1:0/387232721 learned_addr learned my addr 172.21.15.1:0/387232721 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:18.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.145+0000 7f7e9d30a700 1 --2- 172.21.15.1:0/387232721 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e981e8e80 0x7f7e981e9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:18.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.145+0000 7f7e97fff700 1 --2- 172.21.15.1:0/387232721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e9806f370 0x7f7e98150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:18.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.145+0000 7f7e9cb09700 1 -- 172.21.15.1:0/387232721 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e981e8e80 msgr2=0x7f7e981e9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:18.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.145+0000 7f7e9cb09700 1 --2- 172.21.15.1:0/387232721 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e981e8e80 0x7f7e981e9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:18.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.145+0000 7f7e9cb09700 1 -- 172.21.15.1:0/387232721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e9806f370 msgr2=0x7f7e98150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:18.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.145+0000 7f7e9cb09700 1 --2- 172.21.15.1:0/387232721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e9806f370 0x7f7e98150840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:18.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.145+0000 7f7e9cb09700 1 -- 172.21.15.1:0/387232721 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e981e80c0 con 0x7f7e88004850 2024-03-18T20:23:18.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.145+0000 7f7e9cb09700 1 --2- 172.21.15.1:0/387232721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e88004850 0x7f7e98157f20 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c0119d0 tx=0x7f7e8c011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:18.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.145+0000 7f7e957fa700 1 -- 172.21.15.1:0/387232721 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c00f8e0 con 0x7f7e88004850 2024-03-18T20:23:18.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.146+0000 7f7e957fa700 1 -- 172.21.15.1:0/387232721 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8c00fa40 con 0x7f7e88004850 2024-03-18T20:23:18.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.146+0000 7f7e9ed6d700 1 -- 172.21.15.1:0/387232721 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e981e83a0 con 0x7f7e88004850 2024-03-18T20:23:18.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.146+0000 7f7e957fa700 1 -- 172.21.15.1:0/387232721 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c00fdd0 con 0x7f7e88004850 2024-03-18T20:23:18.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.146+0000 7f7e9ed6d700 1 -- 172.21.15.1:0/387232721 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e98158460 con 0x7f7e88004850 2024-03-18T20:23:18.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.147+0000 7f7e9ed6d700 1 -- 172.21.15.1:0/387232721 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e98071de0 con 0x7f7e88004850 2024-03-18T20:23:18.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.147+0000 7f7e957fa700 1 -- 172.21.15.1:0/387232721 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 118) v1 ==== 89268+0+0 (secure 0 0 0) 0x7f7e8c014440 con 0x7f7e88004850 2024-03-18T20:23:18.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.147+0000 7f7e957fa700 1 --2- 172.21.15.1:0/387232721 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7e7c06ae30 0x7f7e7c06d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:18.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.148+0000 7f7e957fa700 1 -- 172.21.15.1:0/387232721 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6703+0+0 (secure 0 0 0) 0x7f7e8c08a050 con 0x7f7e88004850 2024-03-18T20:23:18.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.148+0000 7f7e97fff700 1 --2- 172.21.15.1:0/387232721 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7e7c06ae30 0x7f7e7c06d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:18.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.148+0000 7f7e97fff700 1 --2- 172.21.15.1:0/387232721 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7e7c06ae30 0x7f7e7c06d2f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7e84008fd0 tx=0x7f7e8400a000 comp rx=0 tx=0).ready entity=mgr.10713 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:18.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.152+0000 7f7e957fa700 1 -- 172.21.15.1:0/387232721 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7e8c054d70 con 0x7f7e88004850 2024-03-18T20:23:18.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.382+0000 7f7e9ed6d700 1 -- 172.21.15.1:0/387232721 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e98161840 con 0x7f7e7c06ae30 2024-03-18T20:23:18.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.434+0000 7f7e957fa700 1 -- 172.21.15.1:0/387232721 <== mgr.10713 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f7e84007b60 con 0x7f7e7c06ae30 2024-03-18T20:23:18.438 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule removed for path /snap_test_dir1/test_restart 2024-03-18T20:23:18.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.436+0000 7f7e9ed6d700 1 -- 172.21.15.1:0/387232721 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7e7c06ae30 msgr2=0x7f7e7c06d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:18.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.436+0000 7f7e9ed6d700 1 --2- 172.21.15.1:0/387232721 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7e7c06ae30 0x7f7e7c06d2f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7e84008fd0 tx=0x7f7e8400a000 comp rx=0 tx=0).stop 2024-03-18T20:23:18.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.436+0000 7f7e9ed6d700 1 -- 172.21.15.1:0/387232721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e88004850 msgr2=0x7f7e98157f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:18.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.436+0000 7f7e9ed6d700 1 --2- 172.21.15.1:0/387232721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e88004850 0x7f7e98157f20 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c0119d0 tx=0x7f7e8c011d90 comp rx=0 tx=0).stop 2024-03-18T20:23:18.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.436+0000 7f7e9ed6d700 1 -- 172.21.15.1:0/387232721 shutdown_connections 2024-03-18T20:23:18.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.436+0000 7f7e9ed6d700 1 --2- 172.21.15.1:0/387232721 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7e7c06ae30 0x7f7e7c06d2f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:18.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.436+0000 7f7e9ed6d700 1 --2- 172.21.15.1:0/387232721 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e981e8e80 0x7f7e981e9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:18.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.436+0000 7f7e9ed6d700 1 --2- 172.21.15.1:0/387232721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e88004850 0x7f7e98157f20 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:18.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.436+0000 7f7e9ed6d700 1 --2- 172.21.15.1:0/387232721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e9806f370 0x7f7e98150840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:18.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.436+0000 7f7e9ed6d700 1 -- 172.21.15.1:0/387232721 >> 172.21.15.1:0/387232721 conn(0x7f7e9806ab70 msgr2=0x7f7e9815b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:18.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.436+0000 7f7e9ed6d700 1 -- 172.21.15.1:0/387232721 shutdown_connections 2024-03-18T20:23:18.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:18.436+0000 7f7e9ed6d700 1 -- 172.21.15.1:0/387232721 wait complete. 2024-03-18T20:23:18.452 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2024-03-18T20:23:18.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:18.453 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-03-18T20:23:18.475 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_08_00_UTC 2024-03-18T20:23:18.476 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_09_00_UTC 2024-03-18T20:23:18.476 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_10_00_UTC 2024-03-18T20:23:18.476 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_11_00_UTC 2024-03-18T20:23:18.476 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_12_00_UTC 2024-03-18T20:23:18.476 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_13_00_UTC 2024-03-18T20:23:18.476 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_14_00_UTC 2024-03-18T20:23:18.476 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_15_00_UTC 2024-03-18T20:23:18.476 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_16_00_UTC 2024-03-18T20:23:18.476 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_17_00_UTC 2024-03-18T20:23:18.476 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_18_00_UTC 2024-03-18T20:23:18.476 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_19_00_UTC 2024-03-18T20:23:18.476 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_20_00_UTC 2024-03-18T20:23:18.477 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_21_00_UTC 2024-03-18T20:23:18.477 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_22_00_UTC 2024-03-18T20:23:18.477 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_23_00_UTC 2024-03-18T20:23:18.477 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_08_00_UTC 2024-03-18T20:23:18.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:18.477 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_08_00_UTC) 2024-03-18T20:23:18.541 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_09_00_UTC 2024-03-18T20:23:18.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:18.541 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_09_00_UTC) 2024-03-18T20:23:18.605 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_10_00_UTC 2024-03-18T20:23:18.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:18.605 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_10_00_UTC) 2024-03-18T20:23:18.670 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_11_00_UTC 2024-03-18T20:23:18.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:18.670 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_11_00_UTC) 2024-03-18T20:23:18.734 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_12_00_UTC 2024-03-18T20:23:18.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:18.735 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_12_00_UTC) 2024-03-18T20:23:18.800 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_13_00_UTC 2024-03-18T20:23:18.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:18.800 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_13_00_UTC) 2024-03-18T20:23:18.865 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_14_00_UTC 2024-03-18T20:23:18.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:18.865 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_14_00_UTC) 2024-03-18T20:23:18.930 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_15_00_UTC 2024-03-18T20:23:18.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:18.930 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_15_00_UTC) 2024-03-18T20:23:18.995 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_16_00_UTC 2024-03-18T20:23:18.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:18.995 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_16_00_UTC) 2024-03-18T20:23:19.061 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_17_00_UTC 2024-03-18T20:23:19.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:19.061 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_17_00_UTC) 2024-03-18T20:23:19.125 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_18_00_UTC 2024-03-18T20:23:19.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:19.126 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_18_00_UTC) 2024-03-18T20:23:19.190 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_19_00_UTC 2024-03-18T20:23:19.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:19.191 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_19_00_UTC) 2024-03-18T20:23:19.281 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_20_00_UTC 2024-03-18T20:23:19.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:19.281 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_20_00_UTC) 2024-03-18T20:23:19.307 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_21_00_UTC 2024-03-18T20:23:19.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:19.307 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_21_00_UTC) 2024-03-18T20:23:19.371 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_22_00_UTC 2024-03-18T20:23:19.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:19.371 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_22_00_UTC) 2024-03-18T20:23:19.439 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_23_00_UTC 2024-03-18T20:23:19.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:19.439 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-03-18-20_23_00_UTC) 2024-03-18T20:23:19.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:19.505 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2024-03-18T20:23:19.567 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-18T20:23:19.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.729+0000 7fcf3f3e9700 1 Processor -- start 2024-03-18T20:23:19.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.729+0000 7fcf3f3e9700 1 -- start start 2024-03-18T20:23:19.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.730+0000 7fcf3f3e9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf3815b6c0 0x7fcf380b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:19.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.730+0000 7fcf3f3e9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcf380c87d0 0x7fcf380cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:19.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.730+0000 7fcf3f3e9700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf380d01f0 con 0x7fcf380cd430 2024-03-18T20:23:19.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.730+0000 7fcf3f3e9700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf380ccfa0 con 0x7fcf380c87d0 2024-03-18T20:23:19.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.730+0000 7fcf3f3e9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf380d1db0 con 0x7fcf3815b6c0 2024-03-18T20:23:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.731+0000 7fcf3d185700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf3815b6c0 0x7fcf380b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.731+0000 7fcf3d986700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcf380c87d0 0x7fcf380cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.731+0000 7fcf3c984700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fcf380cd430 0x7fcf380c80c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:43404/0 (socket says 172.21.15.1:43404) 2024-03-18T20:23:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.731+0000 7fcf3c984700 1 -- 172.21.15.1:0/2876732844 learned_addr learned my addr 172.21.15.1:0/2876732844 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.731+0000 7fcf3d185700 1 -- 172.21.15.1:0/2876732844 >> v2:172.21.15.165:3300/0 conn(0x7fcf380c87d0 msgr2=0x7fcf380cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.731+0000 7fcf3d185700 1 --2- 172.21.15.1:0/2876732844 >> v2:172.21.15.165:3300/0 conn(0x7fcf380c87d0 0x7fcf380cd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.731+0000 7fcf3d185700 1 -- 172.21.15.1:0/2876732844 >> v1:172.21.15.165:6789/0 conn(0x7fcf380cd430 legacy=0x7fcf380c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.731+0000 7fcf3d185700 1 -- 172.21.15.1:0/2876732844 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf380cff10 con 0x7fcf3815b6c0 2024-03-18T20:23:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.731+0000 7fcf3d986700 1 --2- 172.21.15.1:0/2876732844 >> v2:172.21.15.165:3300/0 conn(0x7fcf380c87d0 0x7fcf380cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:23:19.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.731+0000 7fcf3d185700 1 --2- 172.21.15.1:0/2876732844 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf3815b6c0 0x7fcf380b2160 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fcf28009840 tx=0x7fcf28009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27c1ab6398b02157 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.732+0000 7fcf2ffff700 1 -- 172.21.15.1:0/2876732844 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28011830 con 0x7fcf3815b6c0 2024-03-18T20:23:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.732+0000 7fcf2ffff700 1 -- 172.21.15.1:0/2876732844 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf28011990 con 0x7fcf3815b6c0 2024-03-18T20:23:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.732+0000 7fcf2ffff700 1 -- 172.21.15.1:0/2876732844 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28011d20 con 0x7fcf3815b6c0 2024-03-18T20:23:19.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.732+0000 7fcf3f3e9700 1 -- 172.21.15.1:0/2876732844 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf3815b6c0 msgr2=0x7fcf380b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:19.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.732+0000 7fcf3f3e9700 1 --2- 172.21.15.1:0/2876732844 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf3815b6c0 0x7fcf380b2160 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fcf28009840 tx=0x7fcf28009b50 comp rx=0 tx=0).stop 2024-03-18T20:23:19.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.732+0000 7fcf3f3e9700 1 -- 172.21.15.1:0/2876732844 shutdown_connections 2024-03-18T20:23:19.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.732+0000 7fcf3f3e9700 1 --2- 172.21.15.1:0/2876732844 >> v2:172.21.15.165:3300/0 conn(0x7fcf380c87d0 0x7fcf380cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:19.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.732+0000 7fcf3f3e9700 1 --2- 172.21.15.1:0/2876732844 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf3815b6c0 0x7fcf380b2160 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:19.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.732+0000 7fcf3f3e9700 1 -- 172.21.15.1:0/2876732844 >> 172.21.15.1:0/2876732844 conn(0x7fcf380d0ff0 msgr2=0x7fcf380d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:19.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.732+0000 7fcf3f3e9700 1 -- 172.21.15.1:0/2876732844 shutdown_connections 2024-03-18T20:23:19.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.732+0000 7fcf3f3e9700 1 -- 172.21.15.1:0/2876732844 wait complete. 2024-03-18T20:23:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.734+0000 7fcf3f3e9700 1 Processor -- start 2024-03-18T20:23:19.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.734+0000 7fcf3f3e9700 1 -- start start 2024-03-18T20:23:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.735+0000 7fcf3f3e9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf380c87d0 0x7fcf380b1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.735+0000 7fcf3f3e9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf380cd430 0x7fcf380aeb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.735+0000 7fcf3f3e9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf3815b6c0 0x7fcf381570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.735+0000 7fcf3f3e9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf380c4b20 con 0x7fcf3815b6c0 2024-03-18T20:23:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.735+0000 7fcf3f3e9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf380af600 con 0x7fcf380c87d0 2024-03-18T20:23:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.735+0000 7fcf3f3e9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf380af8e0 con 0x7fcf380cd430 2024-03-18T20:23:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.735+0000 7fcf3d986700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf3815b6c0 0x7fcf381570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.735+0000 7fcf3d986700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf3815b6c0 0x7fcf381570e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57514/0 (socket says 172.21.15.1:57514) 2024-03-18T20:23:19.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.735+0000 7fcf3d986700 1 -- 172.21.15.1:0/3602459473 learned_addr learned my addr 172.21.15.1:0/3602459473 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.735+0000 7fcf3c984700 1 --2- 172.21.15.1:0/3602459473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf380cd430 0x7fcf380aeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.735+0000 7fcf3d185700 1 --2- 172.21.15.1:0/3602459473 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf380c87d0 0x7fcf380b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.735+0000 7fcf3d986700 1 -- 172.21.15.1:0/3602459473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf380cd430 msgr2=0x7fcf380aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.735+0000 7fcf3d986700 1 --2- 172.21.15.1:0/3602459473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf380cd430 0x7fcf380aeb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.735+0000 7fcf3d986700 1 -- 172.21.15.1:0/3602459473 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf380c87d0 msgr2=0x7fcf380b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.735+0000 7fcf3d986700 1 --2- 172.21.15.1:0/3602459473 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf380c87d0 0x7fcf380b1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.735+0000 7fcf3d986700 1 -- 172.21.15.1:0/3602459473 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf30009a70 con 0x7fcf3815b6c0 2024-03-18T20:23:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.736+0000 7fcf3d986700 1 --2- 172.21.15.1:0/3602459473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf3815b6c0 0x7fcf381570e0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000efc0 tx=0x7fcf3000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.736+0000 7fcf2dffb700 1 -- 172.21.15.1:0/3602459473 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3000ca90 con 0x7fcf3815b6c0 2024-03-18T20:23:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.736+0000 7fcf2dffb700 1 -- 172.21.15.1:0/3602459473 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf3000f040 con 0x7fcf3815b6c0 2024-03-18T20:23:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.736+0000 7fcf3f3e9700 1 -- 172.21.15.1:0/3602459473 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf280094a0 con 0x7fcf3815b6c0 2024-03-18T20:23:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.736+0000 7fcf2dffb700 1 -- 172.21.15.1:0/3602459473 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3000ce30 con 0x7fcf3815b6c0 2024-03-18T20:23:19.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.736+0000 7fcf3f3e9700 1 -- 172.21.15.1:0/3602459473 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf380c5050 con 0x7fcf3815b6c0 2024-03-18T20:23:19.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.737+0000 7fcf3f3e9700 1 -- 172.21.15.1:0/3602459473 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf380676a0 con 0x7fcf3815b6c0 2024-03-18T20:23:19.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.737+0000 7fcf2dffb700 1 -- 172.21.15.1:0/3602459473 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 118) v1 ==== 89268+0+0 (secure 0 0 0) 0x7fcf30014480 con 0x7fcf3815b6c0 2024-03-18T20:23:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.738+0000 7fcf2dffb700 1 --2- 172.21.15.1:0/3602459473 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcf1c06ae00 0x7fcf1c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.738+0000 7fcf2dffb700 1 -- 172.21.15.1:0/3602459473 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6791+0+0 (secure 0 0 0) 0x7fcf3008a040 con 0x7fcf3815b6c0 2024-03-18T20:23:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.738+0000 7fcf3d185700 1 --2- 172.21.15.1:0/3602459473 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcf1c06ae00 0x7fcf1c06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:19.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.738+0000 7fcf3d185700 1 --2- 172.21.15.1:0/3602459473 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcf1c06ae00 0x7fcf1c06d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcf28000c00 tx=0x7fcf28008030 comp rx=0 tx=0).ready entity=mgr.10713 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:19.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:19.742+0000 7fcf2dffb700 1 -- 172.21.15.1:0/3602459473 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcf30054c50 con 0x7fcf3815b6c0 2024-03-18T20:23:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.009+0000 7fcf3f3e9700 1 -- 172.21.15.1:0/3602459473 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fcf3806f800 con 0x7fcf3815b6c0 2024-03-18T20:23:20.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.260+0000 7fcf2dffb700 1 -- 172.21.15.1:0/3602459473 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 119) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fcf30088120 con 0x7fcf3815b6c0 2024-03-18T20:23:20.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.263+0000 7fcf2dffb700 1 -- 172.21.15.1:0/3602459473 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v119) v1 ==== 93+0+0 (secure 0 0 0) 0x7fcf30087c60 con 0x7fcf3815b6c0 2024-03-18T20:23:20.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.267+0000 7fcf3f3e9700 1 -- 172.21.15.1:0/3602459473 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcf1c06ae00 msgr2=0x7fcf1c06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.267+0000 7fcf3f3e9700 1 --2- 172.21.15.1:0/3602459473 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcf1c06ae00 0x7fcf1c06d2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcf28000c00 tx=0x7fcf28008030 comp rx=0 tx=0).stop 2024-03-18T20:23:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.267+0000 7fcf3f3e9700 1 -- 172.21.15.1:0/3602459473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf3815b6c0 msgr2=0x7fcf381570e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.267+0000 7fcf3f3e9700 1 --2- 172.21.15.1:0/3602459473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf3815b6c0 0x7fcf381570e0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000efc0 tx=0x7fcf3000c3a0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.271+0000 7fcf3f3e9700 1 -- 172.21.15.1:0/3602459473 shutdown_connections 2024-03-18T20:23:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.271+0000 7fcf3f3e9700 1 --2- 172.21.15.1:0/3602459473 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcf1c06ae00 0x7fcf1c06d2c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.271+0000 7fcf3f3e9700 1 --2- 172.21.15.1:0/3602459473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf3815b6c0 0x7fcf381570e0 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.271+0000 7fcf3f3e9700 1 --2- 172.21.15.1:0/3602459473 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf380c87d0 0x7fcf380b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.271+0000 7fcf3f3e9700 1 --2- 172.21.15.1:0/3602459473 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf380cd430 0x7fcf380aeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.271+0000 7fcf3f3e9700 1 -- 172.21.15.1:0/3602459473 >> 172.21.15.1:0/3602459473 conn(0x7fcf380d0ff0 msgr2=0x7fcf380ca510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.271+0000 7fcf3f3e9700 1 -- 172.21.15.1:0/3602459473 shutdown_connections 2024-03-18T20:23:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.271+0000 7fcf3f3e9700 1 -- 172.21.15.1:0/3602459473 wait complete. 2024-03-18T20:23:20.292 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T20:23:20.293 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:23:20.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:23:20.388 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T20:23:20.388 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:23:20.388 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:23:20.412 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:23:20.422 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:20.416+0000 7f4e634a7240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:23:20.422 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:20.417+0000 7f5e66159240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:23:20.570 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:20.564+0000 7f4e634a7240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:23:20.593 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:20.588+0000 7f5e66159240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:23:20.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.588+0000 7f4feefa0700 1 Processor -- start 2024-03-18T20:23:20.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.589+0000 7f4feefa0700 1 -- start start 2024-03-18T20:23:20.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.590+0000 7f4feefa0700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4fe8070e70 0x7f4fe80b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:20.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.590+0000 7f4feefa0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fe80b22d0 0x7f4fe80ada80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:20.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.590+0000 7f4feefa0700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe80ad5f0 con 0x7f4fe80cb1a0 2024-03-18T20:23:20.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.590+0000 7f4feefa0700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe80acd00 con 0x7f4fe8070e70 2024-03-18T20:23:20.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.590+0000 7f4feefa0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe80717e0 con 0x7f4fe80b22d0 2024-03-18T20:23:20.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.590+0000 7f4fedf9e700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4fe80cb1a0 0x7f4fe8159630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43190/0 (socket says 172.21.15.1:43190) 2024-03-18T20:23:20.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.590+0000 7f4fedf9e700 1 -- 172.21.15.1:0/2707662857 learned_addr learned my addr 172.21.15.1:0/2707662857 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:20.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.590+0000 7f4fed79d700 1 --2- 172.21.15.1:0/2707662857 >> v2:172.21.15.165:3300/0 conn(0x7f4fe8070e70 0x7f4fe80b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:20.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.590+0000 7f4fee79f700 1 --2- 172.21.15.1:0/2707662857 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fe80b22d0 0x7f4fe80ada80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:20.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.590+0000 7f4fecf9c700 1 -- 172.21.15.1:0/2707662857 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282590856 0 0) 0x7f4fe80ad5f0 con 0x7f4fe80cb1a0 2024-03-18T20:23:20.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.591+0000 7f4fecf9c700 1 -- 172.21.15.1:0/2707662857 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd40035b0 con 0x7f4fe80cb1a0 2024-03-18T20:23:20.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.591+0000 7f4fed79d700 1 -- 172.21.15.1:0/2707662857 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fe80b22d0 msgr2=0x7f4fe80ada80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:20.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.591+0000 7f4fed79d700 1 --2- 172.21.15.1:0/2707662857 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fe80b22d0 0x7f4fe80ada80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.591+0000 7f4fed79d700 1 -- 172.21.15.1:0/2707662857 >> v1:172.21.15.1:6789/0 conn(0x7f4fe80cb1a0 legacy=0x7f4fe8159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:20.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.591+0000 7f4fed79d700 1 -- 172.21.15.1:0/2707662857 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe80acfa0 con 0x7f4fe8070e70 2024-03-18T20:23:20.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.591+0000 7f4fed79d700 1 --2- 172.21.15.1:0/2707662857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4fe8070e70 0x7f4fe80b1d90 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4009d90 tx=0x7f4fe400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f0d3010596909d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:20.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.592+0000 7f4fecf9c700 1 -- 172.21.15.1:0/2707662857 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe400e070 con 0x7f4fe8070e70 2024-03-18T20:23:20.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.592+0000 7f4fecf9c700 1 -- 172.21.15.1:0/2707662857 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe400bb20 con 0x7f4fe8070e70 2024-03-18T20:23:20.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.592+0000 7f4fecf9c700 1 -- 172.21.15.1:0/2707662857 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4012430 con 0x7f4fe8070e70 2024-03-18T20:23:20.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.593+0000 7f4feefa0700 1 -- 172.21.15.1:0/2707662857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4fe8070e70 msgr2=0x7f4fe80b1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:20.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.593+0000 7f4feefa0700 1 --2- 172.21.15.1:0/2707662857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4fe8070e70 0x7f4fe80b1d90 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4009d90 tx=0x7f4fe400b330 comp rx=0 tx=0).stop 2024-03-18T20:23:20.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.593+0000 7f4feefa0700 1 -- 172.21.15.1:0/2707662857 shutdown_connections 2024-03-18T20:23:20.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.593+0000 7f4feefa0700 1 --2- 172.21.15.1:0/2707662857 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fe80b22d0 0x7f4fe80ada80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.593+0000 7f4feefa0700 1 --2- 172.21.15.1:0/2707662857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4fe8070e70 0x7f4fe80b1d90 secure :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4009d90 tx=0x7f4fe400b330 comp rx=0 tx=0).stop 2024-03-18T20:23:20.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.593+0000 7f4feefa0700 1 -- 172.21.15.1:0/2707662857 >> 172.21.15.1:0/2707662857 conn(0x7f4fe80ceda0 msgr2=0x7f4fe80cf1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:20.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.594+0000 7f4feefa0700 1 -- 172.21.15.1:0/2707662857 shutdown_connections 2024-03-18T20:23:20.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.594+0000 7f4feefa0700 1 -- 172.21.15.1:0/2707662857 wait complete. 2024-03-18T20:23:20.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.596+0000 7f4feefa0700 1 Processor -- start 2024-03-18T20:23:20.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.596+0000 7f4feefa0700 1 -- start start 2024-03-18T20:23:20.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.597+0000 7f4feefa0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4fe80b22d0 0x7f4fe814c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:20.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.597+0000 7f4feefa0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fe80cb1a0 0x7f4fe8152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:20.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.597+0000 7f4feefa0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fe809b8f0 0x7f4fe80afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:20.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.597+0000 7f4feefa0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe809a850 con 0x7f4fe809b8f0 2024-03-18T20:23:20.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.597+0000 7f4feefa0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe809ab30 con 0x7f4fe80b22d0 2024-03-18T20:23:20.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.597+0000 7f4feefa0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe809ae10 con 0x7f4fe80cb1a0 2024-03-18T20:23:20.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.597+0000 7f4fee79f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fe809b8f0 0x7f4fe80afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:20.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.597+0000 7f4fee79f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fe809b8f0 0x7f4fe80afc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57532/0 (socket says 172.21.15.1:57532) 2024-03-18T20:23:20.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.597+0000 7f4fee79f700 1 -- 172.21.15.1:0/4086566803 learned_addr learned my addr 172.21.15.1:0/4086566803 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:20.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.597+0000 7f4fedf9e700 1 --2- 172.21.15.1:0/4086566803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4fe80b22d0 0x7f4fe814c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:20.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.597+0000 7f4fee79f700 1 -- 172.21.15.1:0/4086566803 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fe80cb1a0 msgr2=0x7f4fe8152550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:20.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.597+0000 7f4fee79f700 1 --2- 172.21.15.1:0/4086566803 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fe80cb1a0 0x7f4fe8152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.597+0000 7f4fee79f700 1 -- 172.21.15.1:0/4086566803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4fe80b22d0 msgr2=0x7f4fe814c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:20.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.597+0000 7f4fee79f700 1 --2- 172.21.15.1:0/4086566803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4fe80b22d0 0x7f4fe814c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.597+0000 7f4fee79f700 1 -- 172.21.15.1:0/4086566803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe4009a70 con 0x7f4fe809b8f0 2024-03-18T20:23:20.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.597+0000 7f4fee79f700 1 --2- 172.21.15.1:0/4086566803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fe809b8f0 0x7f4fe80afc50 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f4fd800bed0 tx=0x7f4fd800bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:20.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.597+0000 7f4fde7fc700 1 -- 172.21.15.1:0/4086566803 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd800db70 con 0x7f4fe809b8f0 2024-03-18T20:23:20.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.597+0000 7f4fde7fc700 1 -- 172.21.15.1:0/4086566803 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd800f7d0 con 0x7f4fe809b8f0 2024-03-18T20:23:20.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.598+0000 7f4fde7fc700 1 -- 172.21.15.1:0/4086566803 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd800fb60 con 0x7f4fe809b8f0 2024-03-18T20:23:20.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.598+0000 7f4feefa0700 1 -- 172.21.15.1:0/4086566803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe809b0f0 con 0x7f4fe809b8f0 2024-03-18T20:23:20.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.598+0000 7f4feefa0700 1 -- 172.21.15.1:0/4086566803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe814cc70 con 0x7f4fe809b8f0 2024-03-18T20:23:20.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.599+0000 7f4feefa0700 1 -- 172.21.15.1:0/4086566803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe809b610 con 0x7f4fe809b8f0 2024-03-18T20:23:20.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.603+0000 7f4fde7fc700 1 -- 172.21.15.1:0/4086566803 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 119) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f4fd8014070 con 0x7f4fe809b8f0 2024-03-18T20:23:20.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.604+0000 7f4fde7fc700 1 --2- 172.21.15.1:0/4086566803 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4fcc06ae80 0x7f4fcc06d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:20.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.604+0000 7f4fedf9e700 1 -- 172.21.15.1:0/4086566803 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4fcc06ae80 msgr2=0x7f4fcc06d340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:20.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.604+0000 7f4fde7fc700 1 -- 172.21.15.1:0/4086566803 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6791+0+0 (secure 0 0 0) 0x7f4fd8088d40 con 0x7f4fe809b8f0 2024-03-18T20:23:20.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.604+0000 7f4fedf9e700 1 --2- 172.21.15.1:0/4086566803 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4fcc06ae80 0x7f4fcc06d340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:23:20.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.605+0000 7f4fde7fc700 1 -- 172.21.15.1:0/4086566803 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4fd8053a40 con 0x7f4fe809b8f0 2024-03-18T20:23:20.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.805+0000 7f4fedf9e700 1 -- 172.21.15.1:0/4086566803 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4fcc06ae80 msgr2=0x7f4fcc06d340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:20.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.805+0000 7f4fedf9e700 1 --2- 172.21.15.1:0/4086566803 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4fcc06ae80 0x7f4fcc06d340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:23:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.817+0000 7f4feefa0700 1 -- 172.21.15.1:0/4086566803 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fe809b610 con 0x7f4fe809b8f0 2024-03-18T20:23:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.818+0000 7f4fde7fc700 1 -- 172.21.15.1:0/4086566803 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v199) v1 ==== 74+0+17325 (secure 0 0 0) 0x7f4fe809b610 con 0x7f4fe809b8f0 2024-03-18T20:23:20.823 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:20.823 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":199,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:23:20.248210+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:07:17.162084+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-18T20:07:18.295080+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"199","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":199,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2032368612":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/181301955":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/1513627829":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/2331829563":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/2988375791":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/666830041":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/2295652306":"2024-03-19T20:07:34.587648+0000","172.21.15.1:6833/76998":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/2850855126":"2024-03-19T20:18:22.841807+0000","172.21.15.1:6832/76998":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/3253696179":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/3201942299":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/2111434827":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/1313400628":"2024-03-19T20:07:34.587648+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":10},{"begin":18,"length":10}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":9,"length":3},{"begin":25,"length":3}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:23:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.821+0000 7f4feefa0700 1 -- 172.21.15.1:0/4086566803 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4fcc06ae80 msgr2=0x7f4fcc06d340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:23:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.821+0000 7f4feefa0700 1 --2- 172.21.15.1:0/4086566803 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4fcc06ae80 0x7f4fcc06d340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.821+0000 7f4feefa0700 1 -- 172.21.15.1:0/4086566803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fe809b8f0 msgr2=0x7f4fe80afc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.821+0000 7f4feefa0700 1 --2- 172.21.15.1:0/4086566803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fe809b8f0 0x7f4fe80afc50 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f4fd800bed0 tx=0x7f4fd800bf00 comp rx=0 tx=0).stop 2024-03-18T20:23:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.821+0000 7f4feefa0700 1 -- 172.21.15.1:0/4086566803 shutdown_connections 2024-03-18T20:23:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.821+0000 7f4feefa0700 1 --2- 172.21.15.1:0/4086566803 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4fcc06ae80 0x7f4fcc06d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.821+0000 7f4feefa0700 1 --2- 172.21.15.1:0/4086566803 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fe809b8f0 0x7f4fe80afc50 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.821+0000 7f4feefa0700 1 --2- 172.21.15.1:0/4086566803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4fe80b22d0 0x7f4fe814c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.821+0000 7f4feefa0700 1 --2- 172.21.15.1:0/4086566803 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4fe80cb1a0 0x7f4fe8152550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.821+0000 7f4feefa0700 1 -- 172.21.15.1:0/4086566803 >> 172.21.15.1:0/4086566803 conn(0x7f4fe80ceda0 msgr2=0x7f4fe80ae400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.821+0000 7f4feefa0700 1 -- 172.21.15.1:0/4086566803 shutdown_connections 2024-03-18T20:23:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.821+0000 7f4feefa0700 1 -- 172.21.15.1:0/4086566803 wait complete. 2024-03-18T20:23:20.838 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:07:17.162084+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '194', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:07:18.295080+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '199', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 27, 'snap_epoch': 199, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:23:20.839 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:23:20.839 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:20.833+0000 7f4e634a7240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:23:20.878 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:20.874+0000 7f5e66159240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:23:20.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.967+0000 7f895581e700 1 Processor -- start 2024-03-18T20:23:20.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.967+0000 7f895581e700 1 -- start start 2024-03-18T20:23:20.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.967+0000 7f895581e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f895006eda0 0x7f8950161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:20.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.967+0000 7f895581e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f895006f180 0x7f89500ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:20.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.967+0000 7f895581e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f895006a750 0x7f895006ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:20.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.967+0000 7f895581e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89500cb320 con 0x7f895006f180 2024-03-18T20:23:20.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.967+0000 7f895581e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89500ce010 con 0x7f895006eda0 2024-03-18T20:23:20.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.967+0000 7f895581e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f895006b0c0 con 0x7f895006a750 2024-03-18T20:23:20.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.968+0000 7f894e7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f895006f180 0x7f89500ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:20.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.968+0000 7f894e7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f895006f180 0x7f89500ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57538/0 (socket says 172.21.15.1:57538) 2024-03-18T20:23:20.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.968+0000 7f894e7fc700 1 -- 172.21.15.1:0/934020477 learned_addr learned my addr 172.21.15.1:0/934020477 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:20.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.968+0000 7f894f7fe700 1 --2- 172.21.15.1:0/934020477 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f895006a750 0x7f895006ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:20.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.968+0000 7f894effd700 1 --2- 172.21.15.1:0/934020477 >> v2:172.21.15.165:3300/0 conn(0x7f895006eda0 0x7f8950161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:20.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.968+0000 7f894e7fc700 1 -- 172.21.15.1:0/934020477 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f895006a750 msgr2=0x7f895006ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:20.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.968+0000 7f894e7fc700 1 --2- 172.21.15.1:0/934020477 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f895006a750 0x7f895006ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.968+0000 7f894e7fc700 1 -- 172.21.15.1:0/934020477 >> v2:172.21.15.165:3300/0 conn(0x7f895006eda0 msgr2=0x7f8950161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:20.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.968+0000 7f894e7fc700 1 --2- 172.21.15.1:0/934020477 >> v2:172.21.15.165:3300/0 conn(0x7f895006eda0 0x7f8950161c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.968+0000 7f894e7fc700 1 -- 172.21.15.1:0/934020477 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89500caac0 con 0x7f895006f180 2024-03-18T20:23:20.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.968+0000 7f894e7fc700 1 --2- 172.21.15.1:0/934020477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f895006f180 0x7f89500ca580 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f895006d3d0 tx=0x7f8944009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4bfc2b3a0134b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:20.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.968+0000 7f894dffb700 1 -- 172.21.15.1:0/934020477 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8944011830 con 0x7f895006f180 2024-03-18T20:23:20.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.968+0000 7f894dffb700 1 -- 172.21.15.1:0/934020477 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8944011990 con 0x7f895006f180 2024-03-18T20:23:20.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.968+0000 7f894dffb700 1 -- 172.21.15.1:0/934020477 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8944011d20 con 0x7f895006f180 2024-03-18T20:23:20.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.969+0000 7f895581e700 1 -- 172.21.15.1:0/934020477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f895006f180 msgr2=0x7f89500ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:20.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.969+0000 7f895581e700 1 --2- 172.21.15.1:0/934020477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f895006f180 0x7f89500ca580 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f895006d3d0 tx=0x7f8944009bd0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.969+0000 7f895581e700 1 -- 172.21.15.1:0/934020477 shutdown_connections 2024-03-18T20:23:20.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.969+0000 7f895581e700 1 --2- 172.21.15.1:0/934020477 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f895006a750 0x7f895006ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.969+0000 7f895581e700 1 --2- 172.21.15.1:0/934020477 >> v2:172.21.15.165:3300/0 conn(0x7f895006eda0 0x7f8950161c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.969+0000 7f895581e700 1 --2- 172.21.15.1:0/934020477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f895006f180 0x7f89500ca580 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.969+0000 7f895581e700 1 -- 172.21.15.1:0/934020477 >> 172.21.15.1:0/934020477 conn(0x7f89500cff30 msgr2=0x7f89500d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:20.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.969+0000 7f895581e700 1 -- 172.21.15.1:0/934020477 shutdown_connections 2024-03-18T20:23:20.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.969+0000 7f895581e700 1 -- 172.21.15.1:0/934020477 wait complete. 2024-03-18T20:23:20.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.970+0000 7f895581e700 1 Processor -- start 2024-03-18T20:23:20.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.970+0000 7f895581e700 1 -- start start 2024-03-18T20:23:20.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.971+0000 7f895581e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f895006a750 0x7f89500c7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:20.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.971+0000 7f895581e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f895006eda0 0x7f89500c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:20.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.971+0000 7f895581e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f895006f180 0x7f89500c9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:20.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.971+0000 7f895581e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89500b2b20 con 0x7f895006f180 2024-03-18T20:23:20.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.971+0000 7f895581e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89500b3380 con 0x7f895006eda0 2024-03-18T20:23:20.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.971+0000 7f895581e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89500c8820 con 0x7f895006a750 2024-03-18T20:23:20.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.971+0000 7f894f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f895006f180 0x7f89500c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.971+0000 7f894effd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f895006a750 0x7f89500c7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.971+0000 7f894e7fc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f895006eda0 0x7f89500c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.971+0000 7f894f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f895006f180 0x7f89500c9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57554/0 (socket says 172.21.15.1:57554) 2024-03-18T20:23:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.971+0000 7f894f7fe700 1 -- 172.21.15.1:0/4137581852 learned_addr learned my addr 172.21.15.1:0/4137581852 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.972+0000 7f894f7fe700 1 -- 172.21.15.1:0/4137581852 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f895006a750 msgr2=0x7f89500c7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.972+0000 7f894f7fe700 1 --2- 172.21.15.1:0/4137581852 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f895006a750 0x7f89500c7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.972+0000 7f894f7fe700 1 -- 172.21.15.1:0/4137581852 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f895006eda0 msgr2=0x7f89500c7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.972+0000 7f894f7fe700 1 --2- 172.21.15.1:0/4137581852 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f895006eda0 0x7f89500c7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.972+0000 7f894f7fe700 1 -- 172.21.15.1:0/4137581852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8944009550 con 0x7f895006f180 2024-03-18T20:23:20.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.972+0000 7f894f7fe700 1 --2- 172.21.15.1:0/4137581852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f895006f180 0x7f89500c9030 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f894800eed0 tx=0x7f894800ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:20.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.972+0000 7f8937fff700 1 -- 172.21.15.1:0/4137581852 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8948010040 con 0x7f895006f180 2024-03-18T20:23:20.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.972+0000 7f8937fff700 1 -- 172.21.15.1:0/4137581852 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f894800b9f0 con 0x7f895006f180 2024-03-18T20:23:20.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.972+0000 7f895581e700 1 -- 172.21.15.1:0/4137581852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89500b2d30 con 0x7f895006f180 2024-03-18T20:23:20.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.972+0000 7f8937fff700 1 -- 172.21.15.1:0/4137581852 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894800bda0 con 0x7f895006f180 2024-03-18T20:23:20.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.972+0000 7f895581e700 1 -- 172.21.15.1:0/4137581852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89500c8520 con 0x7f895006f180 2024-03-18T20:23:20.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.973+0000 7f895581e700 1 -- 172.21.15.1:0/4137581852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89500b3510 con 0x7f895006f180 2024-03-18T20:23:20.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.974+0000 7f8937fff700 1 -- 172.21.15.1:0/4137581852 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 119) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f8948014400 con 0x7f895006f180 2024-03-18T20:23:20.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.974+0000 7f8937fff700 1 --2- 172.21.15.1:0/4137581852 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f893c06adb0 0x7f893c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:20.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.974+0000 7f894effd700 1 -- 172.21.15.1:0/4137581852 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f893c06adb0 msgr2=0x7f893c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:20.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.974+0000 7f8937fff700 1 -- 172.21.15.1:0/4137581852 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 6791+0+0 (secure 0 0 0) 0x7f8948089c60 con 0x7f895006f180 2024-03-18T20:23:20.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.974+0000 7f894effd700 1 --2- 172.21.15.1:0/4137581852 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f893c06adb0 0x7f893c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:23:20.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:20.983+0000 7f8937fff700 1 -- 172.21.15.1:0/4137581852 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8948054870 con 0x7f895006f180 2024-03-18T20:23:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.175+0000 7f894effd700 1 -- 172.21.15.1:0/4137581852 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f893c06adb0 msgr2=0x7f893c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.175+0000 7f894effd700 1 --2- 172.21.15.1:0/4137581852 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f893c06adb0 0x7f893c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:23:21.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.188+0000 7f895581e700 1 -- 172.21.15.1:0/4137581852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f89500676a0 con 0x7f895006f180 2024-03-18T20:23:21.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.188+0000 7f8937fff700 1 -- 172.21.15.1:0/4137581852 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v199) v1 ==== 93+0+10 (secure 0 0 0) 0x7f89480180a0 con 0x7f895006f180 2024-03-18T20:23:21.193 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:23:21.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.190+0000 7f895581e700 1 -- 172.21.15.1:0/4137581852 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f893c06adb0 msgr2=0x7f893c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:23:21.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.190+0000 7f895581e700 1 --2- 172.21.15.1:0/4137581852 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f893c06adb0 0x7f893c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.190+0000 7f895581e700 1 -- 172.21.15.1:0/4137581852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f895006f180 msgr2=0x7f89500c9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:21.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.190+0000 7f895581e700 1 --2- 172.21.15.1:0/4137581852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f895006f180 0x7f89500c9030 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f894800eed0 tx=0x7f894800ef00 comp rx=0 tx=0).stop 2024-03-18T20:23:21.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.190+0000 7f895581e700 1 -- 172.21.15.1:0/4137581852 shutdown_connections 2024-03-18T20:23:21.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.190+0000 7f895581e700 1 --2- 172.21.15.1:0/4137581852 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f893c06adb0 0x7f893c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.190+0000 7f895581e700 1 --2- 172.21.15.1:0/4137581852 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f895006a750 0x7f89500c7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.190+0000 7f895581e700 1 --2- 172.21.15.1:0/4137581852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f895006f180 0x7f89500c9030 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.190+0000 7f895581e700 1 --2- 172.21.15.1:0/4137581852 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f895006eda0 0x7f89500c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.190+0000 7f895581e700 1 -- 172.21.15.1:0/4137581852 >> 172.21.15.1:0/4137581852 conn(0x7f89500cff30 msgr2=0x7f89500ae350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:21.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.191+0000 7f895581e700 1 -- 172.21.15.1:0/4137581852 shutdown_connections 2024-03-18T20:23:21.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.191+0000 7f895581e700 1 -- 172.21.15.1:0/4137581852 wait complete. 2024-03-18T20:23:21.208 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:23:21.269 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:21.264+0000 7f4e634a7240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:23:21.368 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:21.364+0000 7f5e66159240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:23:21.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.401+0000 7fb195852700 1 Processor -- start 2024-03-18T20:23:21.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.401+0000 7fb195852700 1 -- start start 2024-03-18T20:23:21.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.401+0000 7fb195852700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb190160a70 0x7fb1900d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:21.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.402+0000 7fb195852700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb190162170 con 0x7fb190160e50 2024-03-18T20:23:21.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.402+0000 7fb195852700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1900d06e0 con 0x7fb1901612d0 2024-03-18T20:23:21.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.402+0000 7fb195852700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1900cef30 con 0x7fb190160a70 2024-03-18T20:23:21.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.402+0000 7fb18effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb190160a70 0x7fb1900d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:21.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.402+0000 7fb18e7fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb190160e50 0x7fb1900cffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43196/0 (socket says 172.21.15.1:43196) 2024-03-18T20:23:21.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.402+0000 7fb18e7fc700 1 -- 172.21.15.1:0/786500348 learned_addr learned my addr 172.21.15.1:0/786500348 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:21.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.402+0000 7fb18effd700 1 --2- 172.21.15.1:0/786500348 >> v2:172.21.15.1:3300/0 conn(0x7fb190160a70 0x7fb1900d3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57562/0 (socket says 172.21.15.1:57562) 2024-03-18T20:23:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.402+0000 7fb18effd700 1 -- 172.21.15.1:0/786500348 >> v1:172.21.15.165:6789/0 conn(0x7fb1901612d0 legacy=0x7fb1901616f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.402+0000 7fb18effd700 1 -- 172.21.15.1:0/786500348 >> v1:172.21.15.1:6789/0 conn(0x7fb190160e50 legacy=0x7fb1900cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.402+0000 7fb18effd700 1 -- 172.21.15.1:0/786500348 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1900cf210 con 0x7fb190160a70 2024-03-18T20:23:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.402+0000 7fb18dffb700 1 -- 172.21.15.1:0/786500348 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038675330 0 0) 0x7fb184001110 con 0x7fb190160e50 2024-03-18T20:23:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.402+0000 7fb18effd700 1 --2- 172.21.15.1:0/786500348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb190160a70 0x7fb1900d3b70 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fb180009840 tx=0x7fb180009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8320e8973e8ad2b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.402+0000 7fb18dffb700 1 -- 172.21.15.1:0/786500348 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb180011830 con 0x7fb190160a70 2024-03-18T20:23:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.402+0000 7fb18dffb700 1 -- 172.21.15.1:0/786500348 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb180011990 con 0x7fb190160a70 2024-03-18T20:23:21.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.402+0000 7fb18dffb700 1 -- 172.21.15.1:0/786500348 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb180011d20 con 0x7fb190160a70 2024-03-18T20:23:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.403+0000 7fb195852700 1 -- 172.21.15.1:0/786500348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb190160a70 msgr2=0x7fb1900d3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.403+0000 7fb195852700 1 --2- 172.21.15.1:0/786500348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb190160a70 0x7fb1900d3b70 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fb180009840 tx=0x7fb180009b50 comp rx=0 tx=0).stop 2024-03-18T20:23:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.403+0000 7fb195852700 1 -- 172.21.15.1:0/786500348 shutdown_connections 2024-03-18T20:23:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.403+0000 7fb195852700 1 --2- 172.21.15.1:0/786500348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb190160a70 0x7fb1900d3b70 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.403+0000 7fb195852700 1 -- 172.21.15.1:0/786500348 >> 172.21.15.1:0/786500348 conn(0x7fb19009c560 msgr2=0x7fb1900d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.403+0000 7fb195852700 1 -- 172.21.15.1:0/786500348 shutdown_connections 2024-03-18T20:23:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.403+0000 7fb195852700 1 -- 172.21.15.1:0/786500348 wait complete. 2024-03-18T20:23:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.404+0000 7fb195852700 1 Processor -- start 2024-03-18T20:23:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.404+0000 7fb195852700 1 -- start start 2024-03-18T20:23:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.405+0000 7fb195852700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb190160a70 0x7fb19015eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:21.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.405+0000 7fb195852700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb190160e50 0x7fb1900cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.405+0000 7fb195852700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1901612d0 0x7fb1900cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.405+0000 7fb195852700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1900b87f0 con 0x7fb190160a70 2024-03-18T20:23:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.405+0000 7fb195852700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19015f670 con 0x7fb1901612d0 2024-03-18T20:23:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.405+0000 7fb195852700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1900b7be0 con 0x7fb190160e50 2024-03-18T20:23:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.405+0000 7fb18effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb190160a70 0x7fb19015eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.405+0000 7fb18effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb190160a70 0x7fb19015eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57568/0 (socket says 172.21.15.1:57568) 2024-03-18T20:23:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.405+0000 7fb18effd700 1 -- 172.21.15.1:0/644239382 learned_addr learned my addr 172.21.15.1:0/644239382 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.405+0000 7fb18e7fc700 1 --2- 172.21.15.1:0/644239382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb190160e50 0x7fb1900cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.405+0000 7fb18f7fe700 1 --2- 172.21.15.1:0/644239382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1901612d0 0x7fb1900cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.406+0000 7fb18effd700 1 -- 172.21.15.1:0/644239382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb190160e50 msgr2=0x7fb1900cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.406+0000 7fb18effd700 1 --2- 172.21.15.1:0/644239382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb190160e50 0x7fb1900cc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.406+0000 7fb18effd700 1 -- 172.21.15.1:0/644239382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1901612d0 msgr2=0x7fb1900cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:21.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.406+0000 7fb18effd700 1 --2- 172.21.15.1:0/644239382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1901612d0 0x7fb1900cccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.406+0000 7fb18effd700 1 -- 172.21.15.1:0/644239382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1800094a0 con 0x7fb190160a70 2024-03-18T20:23:21.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.406+0000 7fb18effd700 1 --2- 172.21.15.1:0/644239382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb190160a70 0x7fb19015eac0 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7fb1800090b0 tx=0x7fb180004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:21.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.406+0000 7fb177fff700 1 -- 172.21.15.1:0/644239382 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18001a860 con 0x7fb190160a70 2024-03-18T20:23:21.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.406+0000 7fb177fff700 1 -- 172.21.15.1:0/644239382 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18001a9e0 con 0x7fb190160a70 2024-03-18T20:23:21.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.406+0000 7fb195852700 1 -- 172.21.15.1:0/644239382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1900b7ec0 con 0x7fb190160a70 2024-03-18T20:23:21.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.406+0000 7fb177fff700 1 -- 172.21.15.1:0/644239382 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18001ad90 con 0x7fb190160a70 2024-03-18T20:23:21.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.406+0000 7fb195852700 1 -- 172.21.15.1:0/644239382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1900b8480 con 0x7fb190160a70 2024-03-18T20:23:21.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.407+0000 7fb195852700 1 -- 172.21.15.1:0/644239382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb190067390 con 0x7fb190160a70 2024-03-18T20:23:21.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.409+0000 7fb177fff700 1 -- 172.21.15.1:0/644239382 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 119) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fb180022070 con 0x7fb190160a70 2024-03-18T20:23:21.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.409+0000 7fb177fff700 1 --2- 172.21.15.1:0/644239382 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb17c06adb0 0x7fb17c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:21.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.409+0000 7fb18e7fc700 1 -- 172.21.15.1:0/644239382 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb17c06adb0 msgr2=0x7fb17c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:21.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.409+0000 7fb177fff700 1 -- 172.21.15.1:0/644239382 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6791+0+0 (secure 0 0 0) 0x7fb18008afb0 con 0x7fb190160a70 2024-03-18T20:23:21.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.409+0000 7fb18e7fc700 1 --2- 172.21.15.1:0/644239382 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb17c06adb0 0x7fb17c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:23:21.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.417+0000 7fb177fff700 1 -- 172.21.15.1:0/644239382 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb180055cb0 con 0x7fb190160a70 2024-03-18T20:23:21.500 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:21.495+0000 7f4e634a7240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:23:21.593 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:21.589+0000 7f5e66159240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:23:21.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.610+0000 7fb18e7fc700 1 -- 172.21.15.1:0/644239382 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb17c06adb0 msgr2=0x7fb17c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:21.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.610+0000 7fb18e7fc700 1 --2- 172.21.15.1:0/644239382 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb17c06adb0 0x7fb17c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:23:21.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.619+0000 7fb195852700 1 -- 172.21.15.1:0/644239382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb1900676a0 con 0x7fb190160a70 2024-03-18T20:23:21.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.619+0000 7fb177fff700 1 -- 172.21.15.1:0/644239382 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v200) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb180029020 con 0x7fb190160a70 2024-03-18T20:23:21.624 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:23:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.621+0000 7fb195852700 1 -- 172.21.15.1:0/644239382 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb17c06adb0 msgr2=0x7fb17c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:23:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.621+0000 7fb195852700 1 --2- 172.21.15.1:0/644239382 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb17c06adb0 0x7fb17c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.621+0000 7fb195852700 1 -- 172.21.15.1:0/644239382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb190160a70 msgr2=0x7fb19015eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.621+0000 7fb195852700 1 --2- 172.21.15.1:0/644239382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb190160a70 0x7fb19015eac0 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7fb1800090b0 tx=0x7fb180004750 comp rx=0 tx=0).stop 2024-03-18T20:23:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.621+0000 7fb195852700 1 -- 172.21.15.1:0/644239382 shutdown_connections 2024-03-18T20:23:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.621+0000 7fb195852700 1 --2- 172.21.15.1:0/644239382 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb17c06adb0 0x7fb17c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.621+0000 7fb195852700 1 --2- 172.21.15.1:0/644239382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1901612d0 0x7fb1900cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.621+0000 7fb195852700 1 --2- 172.21.15.1:0/644239382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb190160a70 0x7fb19015eac0 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.621+0000 7fb195852700 1 --2- 172.21.15.1:0/644239382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb190160e50 0x7fb1900cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.621+0000 7fb195852700 1 -- 172.21.15.1:0/644239382 >> 172.21.15.1:0/644239382 conn(0x7fb19009c560 msgr2=0x7fb1900d1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.621+0000 7fb195852700 1 -- 172.21.15.1:0/644239382 shutdown_connections 2024-03-18T20:23:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.621+0000 7fb195852700 1 -- 172.21.15.1:0/644239382 wait complete. 2024-03-18T20:23:21.638 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:23:21.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.765+0000 7fbe1ebcc700 1 Processor -- start 2024-03-18T20:23:21.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.765+0000 7fbe1ebcc700 1 -- start start 2024-03-18T20:23:21.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.766+0000 7fbe1ebcc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbe180c3670 0x7fbe1815e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:21.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.766+0000 7fbe1ebcc700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe180cf730 con 0x7fbe1815e660 2024-03-18T20:23:21.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.766+0000 7fbe1ebcc700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe180cf980 con 0x7fbe180cbd40 2024-03-18T20:23:21.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.766+0000 7fbe1ebcc700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe180cfc60 con 0x7fbe180c3670 2024-03-18T20:23:21.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.766+0000 7fbe17fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbe180c3670 0x7fbe1815e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:21.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.766+0000 7fbe1d169700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbe1815e660 0x7fbe180cdb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43200/0 (socket says 172.21.15.1:43200) 2024-03-18T20:23:21.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.766+0000 7fbe1d169700 1 -- 172.21.15.1:0/2503379879 learned_addr learned my addr 172.21.15.1:0/2503379879 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:21.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.766+0000 7fbe17fff700 1 -- 172.21.15.1:0/2503379879 >> v1:172.21.15.165:6789/0 conn(0x7fbe180cbd40 legacy=0x7fbe180cc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:21.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.766+0000 7fbe17fff700 1 -- 172.21.15.1:0/2503379879 >> v1:172.21.15.1:6789/0 conn(0x7fbe1815e660 legacy=0x7fbe180cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:21.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.766+0000 7fbe17fff700 1 -- 172.21.15.1:0/2503379879 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1815dc00 con 0x7fbe180c3670 2024-03-18T20:23:21.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.766+0000 7fbe17fff700 1 --2- 172.21.15.1:0/2503379879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe180c3670 0x7fbe1815e120 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7fbe04009840 tx=0x7fbe04009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a21e5b986963f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:21.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.767+0000 7fbe177fe700 1 -- 172.21.15.1:0/2503379879 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe04011830 con 0x7fbe180c3670 2024-03-18T20:23:21.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.767+0000 7fbe177fe700 1 -- 172.21.15.1:0/2503379879 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe04011990 con 0x7fbe180c3670 2024-03-18T20:23:21.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.767+0000 7fbe177fe700 1 -- 172.21.15.1:0/2503379879 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe04011d20 con 0x7fbe180c3670 2024-03-18T20:23:21.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.767+0000 7fbe1ebcc700 1 -- 172.21.15.1:0/2503379879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe180c3670 msgr2=0x7fbe1815e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:21.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.767+0000 7fbe1ebcc700 1 --2- 172.21.15.1:0/2503379879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe180c3670 0x7fbe1815e120 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7fbe04009840 tx=0x7fbe04009b50 comp rx=0 tx=0).stop 2024-03-18T20:23:21.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.767+0000 7fbe1ebcc700 1 -- 172.21.15.1:0/2503379879 shutdown_connections 2024-03-18T20:23:21.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.767+0000 7fbe1ebcc700 1 --2- 172.21.15.1:0/2503379879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe180c3670 0x7fbe1815e120 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.767+0000 7fbe1ebcc700 1 -- 172.21.15.1:0/2503379879 >> 172.21.15.1:0/2503379879 conn(0x7fbe180b1aa0 msgr2=0x7fbe180b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:21.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.767+0000 7fbe1ebcc700 1 -- 172.21.15.1:0/2503379879 shutdown_connections 2024-03-18T20:23:21.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.767+0000 7fbe1ebcc700 1 -- 172.21.15.1:0/2503379879 wait complete. 2024-03-18T20:23:21.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.769+0000 7fbe1ebcc700 1 Processor -- start 2024-03-18T20:23:21.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.769+0000 7fbe1ebcc700 1 -- start start 2024-03-18T20:23:21.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.769+0000 7fbe1ebcc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe180c3670 0x7fbe180b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:21.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.769+0000 7fbe1ebcc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe180cbd40 0x7fbe180ae720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:21.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.769+0000 7fbe1ebcc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe1815e660 0x7fbe180b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:21.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.769+0000 7fbe1ebcc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe180aec60 con 0x7fbe180cbd40 2024-03-18T20:23:21.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.770+0000 7fbe1ebcc700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe180b5c20 con 0x7fbe180c3670 2024-03-18T20:23:21.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.770+0000 7fbe1ebcc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe180b5940 con 0x7fbe1815e660 2024-03-18T20:23:21.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.770+0000 7fbe17fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe180cbd40 0x7fbe180ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:21.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.770+0000 7fbe17fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe180cbd40 0x7fbe180ae720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57574/0 (socket says 172.21.15.1:57574) 2024-03-18T20:23:21.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.770+0000 7fbe17fff700 1 -- 172.21.15.1:0/7549958 learned_addr learned my addr 172.21.15.1:0/7549958 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:21.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.770+0000 7fbe1c968700 1 --2- 172.21.15.1:0/7549958 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe180c3670 0x7fbe180b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:21.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.770+0000 7fbe1d169700 1 --2- 172.21.15.1:0/7549958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe1815e660 0x7fbe180b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:21.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.770+0000 7fbe17fff700 1 -- 172.21.15.1:0/7549958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe1815e660 msgr2=0x7fbe180b2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:21.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.770+0000 7fbe17fff700 1 --2- 172.21.15.1:0/7549958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe1815e660 0x7fbe180b2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.770+0000 7fbe17fff700 1 -- 172.21.15.1:0/7549958 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe180c3670 msgr2=0x7fbe180b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:21.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.770+0000 7fbe17fff700 1 --2- 172.21.15.1:0/7549958 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe180c3670 0x7fbe180b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.770+0000 7fbe17fff700 1 -- 172.21.15.1:0/7549958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe040094a0 con 0x7fbe180cbd40 2024-03-18T20:23:21.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.770+0000 7fbe17fff700 1 --2- 172.21.15.1:0/7549958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe180cbd40 0x7fbe180ae720 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7fbe040090b0 tx=0x7fbe04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:21.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.770+0000 7fbe157fa700 1 -- 172.21.15.1:0/7549958 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe040036a0 con 0x7fbe180cbd40 2024-03-18T20:23:21.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.770+0000 7fbe157fa700 1 -- 172.21.15.1:0/7549958 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe0401f070 con 0x7fbe180cbd40 2024-03-18T20:23:21.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.770+0000 7fbe1ebcc700 1 -- 172.21.15.1:0/7549958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe1806b3c0 con 0x7fbe180cbd40 2024-03-18T20:23:21.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.770+0000 7fbe157fa700 1 -- 172.21.15.1:0/7549958 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe04003a60 con 0x7fbe180cbd40 2024-03-18T20:23:21.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.770+0000 7fbe1ebcc700 1 -- 172.21.15.1:0/7549958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe180c47d0 con 0x7fbe180cbd40 2024-03-18T20:23:21.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.772+0000 7fbe1ebcc700 1 -- 172.21.15.1:0/7549958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe180ce200 con 0x7fbe180cbd40 2024-03-18T20:23:21.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.772+0000 7fbe157fa700 1 -- 172.21.15.1:0/7549958 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 119) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fbe04003bc0 con 0x7fbe180cbd40 2024-03-18T20:23:21.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.772+0000 7fbe157fa700 1 --2- 172.21.15.1:0/7549958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbdfc06adb0 0x7fbdfc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:21.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.772+0000 7fbe1c968700 1 -- 172.21.15.1:0/7549958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbdfc06adb0 msgr2=0x7fbdfc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:21.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.773+0000 7fbe157fa700 1 -- 172.21.15.1:0/7549958 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6791+0+0 (secure 0 0 0) 0x7fbe04021070 con 0x7fbe180cbd40 2024-03-18T20:23:21.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.773+0000 7fbe1c968700 1 --2- 172.21.15.1:0/7549958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbdfc06adb0 0x7fbdfc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:23:21.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.777+0000 7fbe157fa700 1 -- 172.21.15.1:0/7549958 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbe04055f80 con 0x7fbe180cbd40 2024-03-18T20:23:21.870 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:21.865+0000 7f4e634a7240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:23:21.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.974+0000 7fbe1c968700 1 -- 172.21.15.1:0/7549958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbdfc06adb0 msgr2=0x7fbdfc06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:21.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.974+0000 7fbe1c968700 1 --2- 172.21.15.1:0/7549958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbdfc06adb0 0x7fbdfc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:23:21.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.980+0000 7fbe1ebcc700 1 -- 172.21.15.1:0/7549958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbe180676a0 con 0x7fbe180cbd40 2024-03-18T20:23:21.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.980+0000 7fbe157fa700 1 -- 172.21.15.1:0/7549958 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v200) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbe040595a0 con 0x7fbe180cbd40 2024-03-18T20:23:21.985 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:23:21.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.982+0000 7fbe1ebcc700 1 -- 172.21.15.1:0/7549958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbdfc06adb0 msgr2=0x7fbdfc06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:23:21.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.982+0000 7fbe1ebcc700 1 --2- 172.21.15.1:0/7549958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbdfc06adb0 0x7fbdfc06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.982+0000 7fbe1ebcc700 1 -- 172.21.15.1:0/7549958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe180cbd40 msgr2=0x7fbe180ae720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:21.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.982+0000 7fbe1ebcc700 1 --2- 172.21.15.1:0/7549958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe180cbd40 0x7fbe180ae720 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7fbe040090b0 tx=0x7fbe04004750 comp rx=0 tx=0).stop 2024-03-18T20:23:21.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.982+0000 7fbe1ebcc700 1 -- 172.21.15.1:0/7549958 shutdown_connections 2024-03-18T20:23:21.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.982+0000 7fbe1ebcc700 1 --2- 172.21.15.1:0/7549958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbdfc06adb0 0x7fbdfc06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.983+0000 7fbe1ebcc700 1 --2- 172.21.15.1:0/7549958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbe1815e660 0x7fbe180b2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.983+0000 7fbe1ebcc700 1 --2- 172.21.15.1:0/7549958 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe180c3670 0x7fbe180b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.983+0000 7fbe1ebcc700 1 --2- 172.21.15.1:0/7549958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe180cbd40 0x7fbe180ae720 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:21.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.983+0000 7fbe1ebcc700 1 -- 172.21.15.1:0/7549958 >> 172.21.15.1:0/7549958 conn(0x7fbe180b1aa0 msgr2=0x7fbe180a0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:21.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.983+0000 7fbe1ebcc700 1 -- 172.21.15.1:0/7549958 shutdown_connections 2024-03-18T20:23:21.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:21.983+0000 7fbe1ebcc700 1 -- 172.21.15.1:0/7549958 wait complete. 2024-03-18T20:23:21.991 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:21.987+0000 7f5e66159240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:23:22.000 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T20:23:22.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.125+0000 7f51a3516700 1 Processor -- start 2024-03-18T20:23:22.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.125+0000 7f51a3516700 1 -- start start 2024-03-18T20:23:22.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.126+0000 7f51a3516700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f519c15d3b0 0x7f519c06cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:22.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.126+0000 7f51a3516700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f519c071550 0x7f519c0b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:22.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.126+0000 7f51a3516700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519c0b2b80 con 0x7f519c06d430 2024-03-18T20:23:22.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.126+0000 7f51a3516700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f519c0b3800 con 0x7f519c071550 2024-03-18T20:23:22.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.126+0000 7f51a3516700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f519c0b2e60 con 0x7f519c15d3b0 2024-03-18T20:23:22.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.126+0000 7f51a0ab1700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f519c06d430 0x7f519c070e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43202/0 (socket says 172.21.15.1:43202) 2024-03-18T20:23:22.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.126+0000 7f51a0ab1700 1 -- 172.21.15.1:0/1724021032 learned_addr learned my addr 172.21.15.1:0/1724021032 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:22.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.126+0000 7f51a12b2700 1 --2- 172.21.15.1:0/1724021032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f519c15d3b0 0x7f519c06cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:22.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.126+0000 7f51a1ab3700 1 --2- 172.21.15.1:0/1724021032 >> v2:172.21.15.165:3300/0 conn(0x7f519c071550 0x7f519c0b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:22.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.126+0000 7f5193fff700 1 -- 172.21.15.1:0/1724021032 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216355496 0 0) 0x7f519c0b2b80 con 0x7f519c06d430 2024-03-18T20:23:22.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.126+0000 7f5193fff700 1 -- 172.21.15.1:0/1724021032 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51880035b0 con 0x7f519c06d430 2024-03-18T20:23:22.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.127+0000 7f51a12b2700 1 -- 172.21.15.1:0/1724021032 >> v2:172.21.15.165:3300/0 conn(0x7f519c071550 msgr2=0x7f519c0b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:22.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.127+0000 7f51a12b2700 1 --2- 172.21.15.1:0/1724021032 >> v2:172.21.15.165:3300/0 conn(0x7f519c071550 0x7f519c0b3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:22.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.127+0000 7f51a12b2700 1 -- 172.21.15.1:0/1724021032 >> v1:172.21.15.1:6789/0 conn(0x7f519c06d430 legacy=0x7f519c070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:22.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.127+0000 7f51a12b2700 1 -- 172.21.15.1:0/1724021032 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519c0b3140 con 0x7f519c15d3b0 2024-03-18T20:23:22.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.127+0000 7f51a1ab3700 1 --2- 172.21.15.1:0/1724021032 >> v2:172.21.15.165:3300/0 conn(0x7f519c071550 0x7f519c0b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:23:22.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.127+0000 7f51a12b2700 1 --2- 172.21.15.1:0/1724021032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f519c15d3b0 0x7f519c06cef0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5198009870 tx=0x7f5198009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca5ed38585a9d529 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:22.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.127+0000 7f5193fff700 1 -- 172.21.15.1:0/1724021032 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5198011830 con 0x7f519c15d3b0 2024-03-18T20:23:22.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.127+0000 7f5193fff700 1 -- 172.21.15.1:0/1724021032 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5198011990 con 0x7f519c15d3b0 2024-03-18T20:23:22.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.127+0000 7f5193fff700 1 -- 172.21.15.1:0/1724021032 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5198011d00 con 0x7f519c15d3b0 2024-03-18T20:23:22.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.128+0000 7f51a3516700 1 -- 172.21.15.1:0/1724021032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f519c15d3b0 msgr2=0x7f519c06cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:22.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.128+0000 7f51a3516700 1 --2- 172.21.15.1:0/1724021032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f519c15d3b0 0x7f519c06cef0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5198009870 tx=0x7f5198009b80 comp rx=0 tx=0).stop 2024-03-18T20:23:22.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.128+0000 7f51a3516700 1 -- 172.21.15.1:0/1724021032 shutdown_connections 2024-03-18T20:23:22.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.128+0000 7f51a3516700 1 --2- 172.21.15.1:0/1724021032 >> v2:172.21.15.165:3300/0 conn(0x7f519c071550 0x7f519c0b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:22.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.128+0000 7f51a3516700 1 --2- 172.21.15.1:0/1724021032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f519c15d3b0 0x7f519c06cef0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:22.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.128+0000 7f51a3516700 1 -- 172.21.15.1:0/1724021032 >> 172.21.15.1:0/1724021032 conn(0x7f519c0b1350 msgr2=0x7f519c0b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:22.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.128+0000 7f51a3516700 1 -- 172.21.15.1:0/1724021032 shutdown_connections 2024-03-18T20:23:22.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.128+0000 7f51a3516700 1 -- 172.21.15.1:0/1724021032 wait complete. 2024-03-18T20:23:22.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.129+0000 7f51a3516700 1 Processor -- start 2024-03-18T20:23:22.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.129+0000 7f51a3516700 1 -- start start 2024-03-18T20:23:22.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f51a3516700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f519c06d430 0x7f519c154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:22.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f51a3516700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f519c071550 0x7f519c0af960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:22.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f51a3516700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f519c15d3b0 0x7f519c0afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:22.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f51a3516700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519c1f69e0 con 0x7f519c15d3b0 2024-03-18T20:23:22.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f51a3516700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519c1f6c30 con 0x7f519c06d430 2024-03-18T20:23:22.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f51a3516700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f519c1f6f10 con 0x7f519c071550 2024-03-18T20:23:22.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f51a1ab3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f519c15d3b0 0x7f519c0afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:22.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f51a1ab3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f519c15d3b0 0x7f519c0afea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57580/0 (socket says 172.21.15.1:57580) 2024-03-18T20:23:22.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f51a1ab3700 1 -- 172.21.15.1:0/45728215 learned_addr learned my addr 172.21.15.1:0/45728215 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:22.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f51a12b2700 1 --2- 172.21.15.1:0/45728215 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f519c06d430 0x7f519c154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:22.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f51a0ab1700 1 --2- 172.21.15.1:0/45728215 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f519c071550 0x7f519c0af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:22.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f51a1ab3700 1 -- 172.21.15.1:0/45728215 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f519c071550 msgr2=0x7f519c0af960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:22.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f51a1ab3700 1 --2- 172.21.15.1:0/45728215 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f519c071550 0x7f519c0af960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f51a1ab3700 1 -- 172.21.15.1:0/45728215 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f519c06d430 msgr2=0x7f519c154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f51a1ab3700 1 --2- 172.21.15.1:0/45728215 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f519c06d430 0x7f519c154ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f51a1ab3700 1 -- 172.21.15.1:0/45728215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5194009a70 con 0x7f519c15d3b0 2024-03-18T20:23:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f51a1ab3700 1 --2- 172.21.15.1:0/45728215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f519c15d3b0 0x7f519c0afea0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f519400efc0 tx=0x7f519400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f5191ffb700 1 -- 172.21.15.1:0/45728215 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519400ca90 con 0x7f519c15d3b0 2024-03-18T20:23:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f5191ffb700 1 -- 172.21.15.1:0/45728215 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f519400f040 con 0x7f519c15d3b0 2024-03-18T20:23:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f51a3516700 1 -- 172.21.15.1:0/45728215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51980094a0 con 0x7f519c15d3b0 2024-03-18T20:23:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.130+0000 7f5191ffb700 1 -- 172.21.15.1:0/45728215 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519400ce10 con 0x7f519c15d3b0 2024-03-18T20:23:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.131+0000 7f51a3516700 1 -- 172.21.15.1:0/45728215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f519c1f74d0 con 0x7f519c15d3b0 2024-03-18T20:23:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.132+0000 7f51a3516700 1 -- 172.21.15.1:0/45728215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f519c073350 con 0x7f519c15d3b0 2024-03-18T20:23:22.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.132+0000 7f5191ffb700 1 -- 172.21.15.1:0/45728215 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 119) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f5194014480 con 0x7f519c15d3b0 2024-03-18T20:23:22.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.132+0000 7f5191ffb700 1 --2- 172.21.15.1:0/45728215 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f518006adb0 0x7f518006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:22.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.133+0000 7f51a12b2700 1 -- 172.21.15.1:0/45728215 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f518006adb0 msgr2=0x7f518006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:22.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.133+0000 7f5191ffb700 1 -- 172.21.15.1:0/45728215 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6791+0+0 (secure 0 0 0) 0x7f5194089f90 con 0x7f519c15d3b0 2024-03-18T20:23:22.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.133+0000 7f51a12b2700 1 --2- 172.21.15.1:0/45728215 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f518006adb0 0x7f518006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:23:22.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.137+0000 7f5191ffb700 1 -- 172.21.15.1:0/45728215 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f51940bd580 con 0x7f519c15d3b0 2024-03-18T20:23:22.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.334+0000 7f51a12b2700 1 -- 172.21.15.1:0/45728215 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f518006adb0 msgr2=0x7f518006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:22.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.334+0000 7f51a12b2700 1 --2- 172.21.15.1:0/45728215 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f518006adb0 0x7f518006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:23:22.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.361+0000 7f51a3516700 1 -- 172.21.15.1:0/45728215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f519c0676a0 con 0x7f519c15d3b0 2024-03-18T20:23:22.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.361+0000 7f5191ffb700 1 -- 172.21.15.1:0/45728215 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 14 entries v200) v1 ==== 81+0+792 (secure 0 0 0) 0x7f5194018330 con 0x7f519c15d3b0 2024-03-18T20:23:22.366 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2032368612 2024-03-19T20:18:22.841807+0000 2024-03-18T20:23:22.366 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/181301955 2024-03-19T20:18:22.841807+0000 2024-03-18T20:23:22.366 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1513627829 2024-03-19T20:18:22.841807+0000 2024-03-18T20:23:22.366 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2331829563 2024-03-19T20:18:22.841807+0000 2024-03-18T20:23:22.366 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2988375791 2024-03-19T20:07:34.587648+0000 2024-03-18T20:23:22.366 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/666830041 2024-03-19T20:07:34.587648+0000 2024-03-18T20:23:22.366 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2295652306 2024-03-19T20:07:34.587648+0000 2024-03-18T20:23:22.366 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6833/76998 2024-03-19T20:07:34.587648+0000 2024-03-18T20:23:22.366 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2850855126 2024-03-19T20:18:22.841807+0000 2024-03-18T20:23:22.366 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6832/76998 2024-03-19T20:07:34.587648+0000 2024-03-18T20:23:22.367 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3253696179 2024-03-19T20:07:34.587648+0000 2024-03-18T20:23:22.367 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3201942299 2024-03-19T20:18:22.841807+0000 2024-03-18T20:23:22.367 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2111434827 2024-03-19T20:07:34.587648+0000 2024-03-18T20:23:22.367 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1313400628 2024-03-19T20:07:34.587648+0000 2024-03-18T20:23:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.363+0000 7f51a3516700 1 -- 172.21.15.1:0/45728215 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f518006adb0 msgr2=0x7f518006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:23:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.363+0000 7f51a3516700 1 --2- 172.21.15.1:0/45728215 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f518006adb0 0x7f518006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.363+0000 7f51a3516700 1 -- 172.21.15.1:0/45728215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f519c15d3b0 msgr2=0x7f519c0afea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.364+0000 7f51a3516700 1 --2- 172.21.15.1:0/45728215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f519c15d3b0 0x7f519c0afea0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f519400efc0 tx=0x7f519400c3a0 comp rx=0 tx=0).stop 2024-03-18T20:23:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.364+0000 7f51a3516700 1 -- 172.21.15.1:0/45728215 shutdown_connections 2024-03-18T20:23:22.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.364+0000 7f51a3516700 1 --2- 172.21.15.1:0/45728215 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f518006adb0 0x7f518006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.364+0000 7f51a3516700 1 --2- 172.21.15.1:0/45728215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f519c15d3b0 0x7f519c0afea0 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.364+0000 7f51a3516700 1 --2- 172.21.15.1:0/45728215 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f519c06d430 0x7f519c154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.364+0000 7f51a3516700 1 --2- 172.21.15.1:0/45728215 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f519c071550 0x7f519c0af960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.364+0000 7f51a3516700 1 -- 172.21.15.1:0/45728215 >> 172.21.15.1:0/45728215 conn(0x7f519c0b1350 msgr2=0x7f519c0ad740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.364+0000 7f51a3516700 1 -- 172.21.15.1:0/45728215 shutdown_connections 2024-03-18T20:23:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.364+0000 7f51a3516700 1 -- 172.21.15.1:0/45728215 wait complete. 2024-03-18T20:23:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:listed 14 entries 2024-03-18T20:23:22.381 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-18T20:23:22.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:22.381 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T20:23:22.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[146826]: fuse finished with error 0 and tester_r 0 2024-03-18T20:23:22.422 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:23:22.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:22.422 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T20:23:22.455 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:22.450+0000 7f4e634a7240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:23:22.526 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:22.522+0000 7f4e634a7240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:23:22.594 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:23:22.595 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:23:22.612 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:23:22.612 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:23:22.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:22.613 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-18T20:23:22.628 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:22.623+0000 7f5e66159240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:23:22.656 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:22.651+0000 7f4e634a7240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:23:22.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:23:22.672 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-18T20:23:22.673 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:23:22.673 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:23:22.692 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:23:22.692 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:23:22.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:22.692 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-18T20:23:22.705 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:22.701+0000 7f5e66159240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:23:22.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:23:22.722 INFO:teuthology.orchestra.run.smithi001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-18T20:23:22.723 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:23:22.723 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:23:22.745 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:23:22.746 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:23:22.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:22.746 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-18T20:23:22.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:23:22.806 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-18T20:23:22.806 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:23:22.841 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:22.836+0000 7f5e66159240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:23:22.845 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:22.840+0000 7f4e634a7240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:23:22.917 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:22.912+0000 7f4e634a7240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:23:22.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.928+0000 7f2acfd28700 1 Processor -- start 2024-03-18T20:23:22.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.928+0000 7f2acfd28700 1 -- start start 2024-03-18T20:23:22.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.930+0000 7f2acfd28700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2ac80cc170 0x7f2ac80ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:22.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.930+0000 7f2acfd28700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac806e270 con 0x7f2ac815a2e0 2024-03-18T20:23:22.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.930+0000 7f2acfd28700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac80ae050 con 0x7f2ac80ae600 2024-03-18T20:23:22.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.930+0000 7f2acfd28700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac806f690 con 0x7f2ac80cc170 2024-03-18T20:23:22.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.930+0000 7f2ace2c5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2ac80cc170 0x7f2ac80ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:22.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.930+0000 7f2acd2c3700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2ac815a2e0 0x7f2ac806ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43204/0 (socket says 172.21.15.1:43204) 2024-03-18T20:23:22.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.930+0000 7f2acd2c3700 1 -- 172.21.15.1:0/3335133940 learned_addr learned my addr 172.21.15.1:0/3335133940 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:22.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.930+0000 7f2ace2c5700 1 --2- 172.21.15.1:0/3335133940 >> v2:172.21.15.1:3300/0 conn(0x7f2ac80cc170 0x7f2ac80ad790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57596/0 (socket says 172.21.15.1:57596) 2024-03-18T20:23:22.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.930+0000 7f2ace2c5700 1 -- 172.21.15.1:0/3335133940 >> v1:172.21.15.165:6789/0 conn(0x7f2ac80ae600 legacy=0x7f2ac80ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:22.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.930+0000 7f2ace2c5700 1 -- 172.21.15.1:0/3335133940 >> v1:172.21.15.1:6789/0 conn(0x7f2ac815a2e0 legacy=0x7f2ac806ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:22.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.930+0000 7f2ace2c5700 1 -- 172.21.15.1:0/3335133940 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac80add70 con 0x7f2ac80cc170 2024-03-18T20:23:22.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.930+0000 7f2ace2c5700 1 --2- 172.21.15.1:0/3335133940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac80cc170 0x7f2ac80ad790 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0009d90 tx=0x7f2ac000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=313379120fd5eaf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:22.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.930+0000 7f2accac2700 1 -- 172.21.15.1:0/3335133940 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac000e070 con 0x7f2ac80cc170 2024-03-18T20:23:22.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.930+0000 7f2accac2700 1 -- 172.21.15.1:0/3335133940 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac000bbd0 con 0x7f2ac80cc170 2024-03-18T20:23:22.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.930+0000 7f2accac2700 1 -- 172.21.15.1:0/3335133940 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0012430 con 0x7f2ac80cc170 2024-03-18T20:23:22.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.931+0000 7f2acfd28700 1 -- 172.21.15.1:0/3335133940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac80cc170 msgr2=0x7f2ac80ad790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:22.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.931+0000 7f2acfd28700 1 --2- 172.21.15.1:0/3335133940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac80cc170 0x7f2ac80ad790 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0009d90 tx=0x7f2ac000b330 comp rx=0 tx=0).stop 2024-03-18T20:23:22.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.931+0000 7f2acfd28700 1 -- 172.21.15.1:0/3335133940 shutdown_connections 2024-03-18T20:23:22.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.931+0000 7f2acfd28700 1 --2- 172.21.15.1:0/3335133940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac80cc170 0x7f2ac80ad790 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:22.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.931+0000 7f2acfd28700 1 -- 172.21.15.1:0/3335133940 >> 172.21.15.1:0/3335133940 conn(0x7f2ac806b860 msgr2=0x7f2ac806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:22.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.931+0000 7f2acfd28700 1 -- 172.21.15.1:0/3335133940 shutdown_connections 2024-03-18T20:23:22.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.931+0000 7f2acfd28700 1 -- 172.21.15.1:0/3335133940 wait complete. 2024-03-18T20:23:22.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.932+0000 7f2acfd28700 1 Processor -- start 2024-03-18T20:23:22.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.932+0000 7f2acfd28700 1 -- start start 2024-03-18T20:23:22.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.933+0000 7f2acfd28700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac80ae600 0x7f2ac81e3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:22.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.933+0000 7f2acfd28700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ac80cc170 0x7f2ac8159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:22.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.933+0000 7f2acfd28700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ac815a2e0 0x7f2ac80c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:22.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.933+0000 7f2acfd28700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac81596c0 con 0x7f2ac80ae600 2024-03-18T20:23:22.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.933+0000 7f2acfd28700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac80ca240 con 0x7f2ac815a2e0 2024-03-18T20:23:22.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.933+0000 7f2acfd28700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac8156f00 con 0x7f2ac80cc170 2024-03-18T20:23:22.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.933+0000 7f2acdac4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac80ae600 0x7f2ac81e3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:22.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.933+0000 7f2acdac4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac80ae600 0x7f2ac81e3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57606/0 (socket says 172.21.15.1:57606) 2024-03-18T20:23:22.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.933+0000 7f2acdac4700 1 -- 172.21.15.1:0/17294469 learned_addr learned my addr 172.21.15.1:0/17294469 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:22.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.933+0000 7f2acd2c3700 1 --2- 172.21.15.1:0/17294469 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ac80cc170 0x7f2ac8159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:22.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.933+0000 7f2ace2c5700 1 --2- 172.21.15.1:0/17294469 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ac815a2e0 0x7f2ac80c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:22.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.934+0000 7f2acdac4700 1 -- 172.21.15.1:0/17294469 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ac80cc170 msgr2=0x7f2ac8159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:22.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.934+0000 7f2acdac4700 1 --2- 172.21.15.1:0/17294469 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ac80cc170 0x7f2ac8159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.934+0000 7f2acdac4700 1 -- 172.21.15.1:0/17294469 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ac815a2e0 msgr2=0x7f2ac80c9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.934+0000 7f2acdac4700 1 --2- 172.21.15.1:0/17294469 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ac815a2e0 0x7f2ac80c9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.934+0000 7f2acdac4700 1 -- 172.21.15.1:0/17294469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac0009a70 con 0x7f2ac80ae600 2024-03-18T20:23:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.934+0000 7f2ace2c5700 1 --2- 172.21.15.1:0/17294469 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ac815a2e0 0x7f2ac80c9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:23:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.934+0000 7f2acd2c3700 1 --2- 172.21.15.1:0/17294469 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ac80cc170 0x7f2ac8159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:23:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.934+0000 7f2acdac4700 1 --2- 172.21.15.1:0/17294469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac80ae600 0x7f2ac81e3ef0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f2ac400c730 tx=0x7f2ac400caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:22.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.934+0000 7f2abe7fc700 1 -- 172.21.15.1:0/17294469 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac40052d0 con 0x7f2ac80ae600 2024-03-18T20:23:22.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.934+0000 7f2acfd28700 1 -- 172.21.15.1:0/17294469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac81e4580 con 0x7f2ac80ae600 2024-03-18T20:23:22.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.934+0000 7f2abe7fc700 1 -- 172.21.15.1:0/17294469 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac4005430 con 0x7f2ac80ae600 2024-03-18T20:23:22.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.934+0000 7f2abe7fc700 1 -- 172.21.15.1:0/17294469 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac40038a0 con 0x7f2ac80ae600 2024-03-18T20:23:22.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.934+0000 7f2acfd28700 1 -- 172.21.15.1:0/17294469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac81e4b40 con 0x7f2ac80ae600 2024-03-18T20:23:22.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.935+0000 7f2acfd28700 1 -- 172.21.15.1:0/17294469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac80b18c0 con 0x7f2ac80ae600 2024-03-18T20:23:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.936+0000 7f2abe7fc700 1 -- 172.21.15.1:0/17294469 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 119) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f2ac4003a00 con 0x7f2ac80ae600 2024-03-18T20:23:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.936+0000 7f2abe7fc700 1 --2- 172.21.15.1:0/17294469 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2aac06adb0 0x7f2aac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.936+0000 7f2acd2c3700 1 -- 172.21.15.1:0/17294469 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2aac06adb0 msgr2=0x7f2aac06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.936+0000 7f2abe7fc700 1 -- 172.21.15.1:0/17294469 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6791+0+0 (secure 0 0 0) 0x7f2ac4088e20 con 0x7f2ac80ae600 2024-03-18T20:23:22.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.936+0000 7f2acd2c3700 1 --2- 172.21.15.1:0/17294469 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2aac06adb0 0x7f2aac06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:23:22.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:22.941+0000 7f2abe7fc700 1 -- 172.21.15.1:0/17294469 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2ac4053a30 con 0x7f2ac80ae600 2024-03-18T20:23:22.997 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:22.992+0000 7f4e634a7240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:23:23.031 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:23.026+0000 7f5e66159240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:23:23.074 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:23.069+0000 7f4e634a7240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:23:23.108 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:23.103+0000 7f5e66159240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:23:23.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.137+0000 7f2acd2c3700 1 -- 172.21.15.1:0/17294469 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2aac06adb0 msgr2=0x7f2aac06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:23.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.137+0000 7f2acd2c3700 1 --2- 172.21.15.1:0/17294469 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2aac06adb0 0x7f2aac06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:23:23.152 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:23.147+0000 7f4e634a7240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:23:23.190 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:23.186+0000 7f5e66159240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:23:23.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.193+0000 7f2acfd28700 1 -- 172.21.15.1:0/17294469 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ac80676a0 con 0x7f2ac80ae600 2024-03-18T20:23:23.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.193+0000 7f2abe7fc700 1 -- 172.21.15.1:0/17294469 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4741 (secure 0 0 0) 0x7f2ac4003cb0 con 0x7f2ac80ae600 2024-03-18T20:23:23.198 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:23.198 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":89,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8204,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2572860438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2572860438},{"type":"v1","addr":"172.21.15.165:6835","nonce":2572860438}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9439,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3158632513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3158632513},{"type":"v1","addr":"172.21.15.1:6836","nonce":3158632513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9448,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/4005747445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4005747445},{"type":"v1","addr":"172.21.15.165:6834","nonce":4005747445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:07:19.322806+0000","modified":"2024-03-18T20:07:22.335335+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10422":{"gid":10422,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1610313086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1610313086},{"type":"v1","addr":"172.21.15.1:6837","nonce":1610313086}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-03-18T20:23:23.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.195+0000 7f2acfd28700 1 -- 172.21.15.1:0/17294469 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2aac06adb0 msgr2=0x7f2aac06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:23:23.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.195+0000 7f2acfd28700 1 --2- 172.21.15.1:0/17294469 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2aac06adb0 0x7f2aac06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:23.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.195+0000 7f2acfd28700 1 -- 172.21.15.1:0/17294469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac80ae600 msgr2=0x7f2ac81e3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:23.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.195+0000 7f2acfd28700 1 --2- 172.21.15.1:0/17294469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac80ae600 0x7f2ac81e3ef0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f2ac400c730 tx=0x7f2ac400caf0 comp rx=0 tx=0).stop 2024-03-18T20:23:23.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.195+0000 7f2acfd28700 1 -- 172.21.15.1:0/17294469 shutdown_connections 2024-03-18T20:23:23.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.195+0000 7f2acfd28700 1 --2- 172.21.15.1:0/17294469 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2aac06adb0 0x7f2aac06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:23.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.195+0000 7f2acfd28700 1 --2- 172.21.15.1:0/17294469 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ac815a2e0 0x7f2ac80c9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:23.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.195+0000 7f2acfd28700 1 --2- 172.21.15.1:0/17294469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac80ae600 0x7f2ac81e3ef0 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:23.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.195+0000 7f2acfd28700 1 --2- 172.21.15.1:0/17294469 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2ac80cc170 0x7f2ac8159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:23.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.195+0000 7f2acfd28700 1 -- 172.21.15.1:0/17294469 >> 172.21.15.1:0/17294469 conn(0x7f2ac806b860 msgr2=0x7f2ac80b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:23.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.196+0000 7f2acfd28700 1 -- 172.21.15.1:0/17294469 shutdown_connections 2024-03-18T20:23:23.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.196+0000 7f2acfd28700 1 -- 172.21.15.1:0/17294469 wait complete. 2024-03-18T20:23:23.201 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 89 2024-03-18T20:23:23.213 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:23:23.276 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:23.272+0000 7f5e66159240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:23:23.323 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:23.318+0000 7f4e634a7240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:23:23.361 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:23.357+0000 7f5e66159240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:23:23.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.376+0000 7fc16e072700 1 Processor -- start 2024-03-18T20:23:23.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.376+0000 7fc16e072700 1 -- start start 2024-03-18T20:23:23.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.377+0000 7fc16e072700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc168160940 0x7fc1680d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.377+0000 7fc16e072700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc1680cf090 0x7fc1680d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.377+0000 7fc16e072700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1680cddc0 con 0x7fc168160d20 2024-03-18T20:23:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.377+0000 7fc16e072700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1680cdff0 con 0x7fc1680cf090 2024-03-18T20:23:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.377+0000 7fc16e072700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1680ce2d0 con 0x7fc168160940 2024-03-18T20:23:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.377+0000 7fc167fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc168160d20 0x7fc168161230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43220/0 (socket says 172.21.15.1:43220) 2024-03-18T20:23:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.377+0000 7fc167fff700 1 -- 172.21.15.1:0/1284578378 learned_addr learned my addr 172.21.15.1:0/1284578378 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.377+0000 7fc1677fe700 1 --2- 172.21.15.1:0/1284578378 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc168160940 0x7fc1680d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.377+0000 7fc166ffd700 1 --2- 172.21.15.1:0/1284578378 >> v2:172.21.15.165:3300/0 conn(0x7fc1680cf090 0x7fc1680d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.377+0000 7fc1667fc700 1 -- 172.21.15.1:0/1284578378 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808515028 0 0) 0x7fc1680cddc0 con 0x7fc168160d20 2024-03-18T20:23:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.378+0000 7fc1667fc700 1 -- 172.21.15.1:0/1284578378 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1500035b0 con 0x7fc168160d20 2024-03-18T20:23:23.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.378+0000 7fc166ffd700 1 -- 172.21.15.1:0/1284578378 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc168160940 msgr2=0x7fc1680d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:23.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.378+0000 7fc1667fc700 1 -- 172.21.15.1:0/1284578378 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3159585317 0 0) 0x7fc1500035b0 con 0x7fc168160d20 2024-03-18T20:23:23.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.378+0000 7fc166ffd700 1 --2- 172.21.15.1:0/1284578378 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc168160940 0x7fc1680d0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:23.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.378+0000 7fc166ffd700 1 -- 172.21.15.1:0/1284578378 >> v1:172.21.15.1:6789/0 conn(0x7fc168160d20 legacy=0x7fc168161230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:23.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.378+0000 7fc166ffd700 1 -- 172.21.15.1:0/1284578378 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1680ce5b0 con 0x7fc1680cf090 2024-03-18T20:23:23.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.378+0000 7fc1677fe700 1 --2- 172.21.15.1:0/1284578378 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc168160940 0x7fc1680d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:23:23.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.378+0000 7fc166ffd700 1 --2- 172.21.15.1:0/1284578378 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1680cf090 0x7fc1680d1790 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc15c009840 tx=0x7fc15c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96f4114987680221 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:23.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.378+0000 7fc1667fc700 1 -- 172.21.15.1:0/1284578378 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1600019f0 con 0x7fc1680cf090 2024-03-18T20:23:23.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.378+0000 7fc1667fc700 1 -- 172.21.15.1:0/1284578378 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15c004030 con 0x7fc1680cf090 2024-03-18T20:23:23.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.378+0000 7fc1667fc700 1 -- 172.21.15.1:0/1284578378 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c0043c0 con 0x7fc1680cf090 2024-03-18T20:23:23.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.379+0000 7fc16e072700 1 -- 172.21.15.1:0/1284578378 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1680cf090 msgr2=0x7fc1680d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:23.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.379+0000 7fc16e072700 1 --2- 172.21.15.1:0/1284578378 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1680cf090 0x7fc1680d1790 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc15c009840 tx=0x7fc15c009b50 comp rx=0 tx=0).stop 2024-03-18T20:23:23.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.379+0000 7fc16e072700 1 -- 172.21.15.1:0/1284578378 shutdown_connections 2024-03-18T20:23:23.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.379+0000 7fc16e072700 1 --2- 172.21.15.1:0/1284578378 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc168160940 0x7fc1680d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:23.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.379+0000 7fc16e072700 1 --2- 172.21.15.1:0/1284578378 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1680cf090 0x7fc1680d1790 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:23.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.379+0000 7fc16e072700 1 -- 172.21.15.1:0/1284578378 >> 172.21.15.1:0/1284578378 conn(0x7fc16809c520 msgr2=0x7fc1680d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:23.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.379+0000 7fc16e072700 1 -- 172.21.15.1:0/1284578378 shutdown_connections 2024-03-18T20:23:23.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.379+0000 7fc16e072700 1 -- 172.21.15.1:0/1284578378 wait complete. 2024-03-18T20:23:23.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.380+0000 7fc16e072700 1 Processor -- start 2024-03-18T20:23:23.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.380+0000 7fc16e072700 1 -- start start 2024-03-18T20:23:23.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.381+0000 7fc16e072700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1680cf090 0x7fc1680b4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:23.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.381+0000 7fc16e072700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc168160940 0x7fc1680ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:23.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.381+0000 7fc16e072700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc168160d20 0x7fc168152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:23.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.381+0000 7fc16e072700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1681533f0 con 0x7fc168160940 2024-03-18T20:23:23.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.381+0000 7fc16e072700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc168153890 con 0x7fc168160d20 2024-03-18T20:23:23.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.381+0000 7fc16e072700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1680b6cb0 con 0x7fc1680cf090 2024-03-18T20:23:23.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.381+0000 7fc166ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc168160940 0x7fc1680ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.381+0000 7fc1677fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1680cf090 0x7fc1680b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.381+0000 7fc166ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc168160940 0x7fc1680ccc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57614/0 (socket says 172.21.15.1:57614) 2024-03-18T20:23:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.381+0000 7fc166ffd700 1 -- 172.21.15.1:0/4202828650 learned_addr learned my addr 172.21.15.1:0/4202828650 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.381+0000 7fc167fff700 1 --2- 172.21.15.1:0/4202828650 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc168160d20 0x7fc168152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.381+0000 7fc166ffd700 1 -- 172.21.15.1:0/4202828650 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1680cf090 msgr2=0x7fc1680b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.381+0000 7fc166ffd700 1 --2- 172.21.15.1:0/4202828650 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1680cf090 0x7fc1680b4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.381+0000 7fc166ffd700 1 -- 172.21.15.1:0/4202828650 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc168160d20 msgr2=0x7fc168152eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.381+0000 7fc166ffd700 1 --2- 172.21.15.1:0/4202828650 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc168160d20 0x7fc168152eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.381+0000 7fc166ffd700 1 -- 172.21.15.1:0/4202828650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1580094a0 con 0x7fc168160940 2024-03-18T20:23:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.381+0000 7fc167fff700 1 --2- 172.21.15.1:0/4202828650 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc168160d20 0x7fc168152eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:23:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.382+0000 7fc166ffd700 1 --2- 172.21.15.1:0/4202828650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc168160940 0x7fc1680ccc70 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fc16806b110 tx=0x7fc15c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:23.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.382+0000 7fc14ffff700 1 -- 172.21.15.1:0/4202828650 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c003c70 con 0x7fc168160940 2024-03-18T20:23:23.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.382+0000 7fc14ffff700 1 -- 172.21.15.1:0/4202828650 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15c01f070 con 0x7fc168160940 2024-03-18T20:23:23.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.382+0000 7fc16e072700 1 -- 172.21.15.1:0/4202828650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc15c0094a0 con 0x7fc168160940 2024-03-18T20:23:23.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.382+0000 7fc14ffff700 1 -- 172.21.15.1:0/4202828650 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c011630 con 0x7fc168160940 2024-03-18T20:23:23.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.382+0000 7fc16e072700 1 -- 172.21.15.1:0/4202828650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1680b6760 con 0x7fc168160940 2024-03-18T20:23:23.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.383+0000 7fc16e072700 1 -- 172.21.15.1:0/4202828650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc168073350 con 0x7fc168160940 2024-03-18T20:23:23.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.383+0000 7fc14ffff700 1 -- 172.21.15.1:0/4202828650 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 119) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fc15c011790 con 0x7fc168160940 2024-03-18T20:23:23.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.384+0000 7fc14ffff700 1 --2- 172.21.15.1:0/4202828650 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc15406adb0 0x7fc15406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:23.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.384+0000 7fc1677fe700 1 -- 172.21.15.1:0/4202828650 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc15406adb0 msgr2=0x7fc15406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:23.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.384+0000 7fc14ffff700 1 -- 172.21.15.1:0/4202828650 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6791+0+0 (secure 0 0 0) 0x7fc15c021070 con 0x7fc168160940 2024-03-18T20:23:23.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.384+0000 7fc1677fe700 1 --2- 172.21.15.1:0/4202828650 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc15406adb0 0x7fc15406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:23:23.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.389+0000 7fc14ffff700 1 -- 172.21.15.1:0/4202828650 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc15c055d60 con 0x7fc168160940 2024-03-18T20:23:23.537 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:23.533+0000 7f4e634a7240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:23:23.544 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:23.539+0000 7f5e66159240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:23:23.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.585+0000 7fc1677fe700 1 -- 172.21.15.1:0/4202828650 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc15406adb0 msgr2=0x7fc15406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:23.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.585+0000 7fc1677fe700 1 --2- 172.21.15.1:0/4202828650 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc15406adb0 0x7fc15406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:23:23.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.595+0000 7fc16e072700 1 -- 172.21.15.1:0/4202828650 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1680d0980 con 0x7fc168160940 2024-03-18T20:23:23.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.596+0000 7fc14ffff700 1 -- 172.21.15.1:0/4202828650 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v200) v1 ==== 74+0+17326 (secure 0 0 0) 0x7fc15c02b090 con 0x7fc168160940 2024-03-18T20:23:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:23.601 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":200,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:23:21.256018+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:07:17.162084+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-18T20:07:18.295080+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"200","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":32,"snap_epoch":200,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2032368612":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/181301955":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/1513627829":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/2331829563":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/2988375791":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/666830041":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/2295652306":"2024-03-19T20:07:34.587648+0000","172.21.15.1:6833/76998":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/2850855126":"2024-03-19T20:18:22.841807+0000","172.21.15.1:6832/76998":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/3253696179":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/3201942299":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/2111434827":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/1313400628":"2024-03-19T20:07:34.587648+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":15},{"begin":18,"length":15}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":12,"length":5},{"begin":28,"length":5}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:23:23.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.598+0000 7fc16e072700 1 -- 172.21.15.1:0/4202828650 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc15406adb0 msgr2=0x7fc15406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:23:23.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.598+0000 7fc16e072700 1 --2- 172.21.15.1:0/4202828650 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc15406adb0 0x7fc15406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:23.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.598+0000 7fc16e072700 1 -- 172.21.15.1:0/4202828650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc168160940 msgr2=0x7fc1680ccc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:23.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.598+0000 7fc16e072700 1 --2- 172.21.15.1:0/4202828650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc168160940 0x7fc1680ccc70 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fc16806b110 tx=0x7fc15c004750 comp rx=0 tx=0).stop 2024-03-18T20:23:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.598+0000 7fc16e072700 1 -- 172.21.15.1:0/4202828650 shutdown_connections 2024-03-18T20:23:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.598+0000 7fc16e072700 1 --2- 172.21.15.1:0/4202828650 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc15406adb0 0x7fc15406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.598+0000 7fc16e072700 1 --2- 172.21.15.1:0/4202828650 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc1680cf090 0x7fc1680b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.598+0000 7fc16e072700 1 --2- 172.21.15.1:0/4202828650 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc168160d20 0x7fc168152eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.598+0000 7fc16e072700 1 --2- 172.21.15.1:0/4202828650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc168160940 0x7fc1680ccc70 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.598+0000 7fc16e072700 1 -- 172.21.15.1:0/4202828650 >> 172.21.15.1:0/4202828650 conn(0x7fc16809c520 msgr2=0x7fc1680b1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.599+0000 7fc16e072700 1 -- 172.21.15.1:0/4202828650 shutdown_connections 2024-03-18T20:23:23.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.599+0000 7fc16e072700 1 -- 172.21.15.1:0/4202828650 wait complete. 2024-03-18T20:23:23.616 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:07:17.162084+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '194', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:07:18.295080+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '200', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 32, 'snap_epoch': 200, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:23:23.617 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:23:23.688 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:23.683+0000 7f4e634a7240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:23:23.761 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:23.756+0000 7f4e634a7240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:23:23.773 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:23.769+0000 7f5e66159240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:23:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.782+0000 7f4023562700 1 Processor -- start 2024-03-18T20:23:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.783+0000 7f4023562700 1 -- start start 2024-03-18T20:23:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.783+0000 7f4023562700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f401c15eb40 0x7f401c15d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.783+0000 7f4023562700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f401c06d300 con 0x7f401c157be0 2024-03-18T20:23:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.783+0000 7f4023562700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f401c071600 con 0x7f401c0b25e0 2024-03-18T20:23:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.783+0000 7f4023562700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f401c0b3450 con 0x7f401c15eb40 2024-03-18T20:23:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.784+0000 7f40212fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f401c157be0 0x7f401c070db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43226/0 (socket says 172.21.15.1:43226) 2024-03-18T20:23:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.784+0000 7f40212fe700 1 -- 172.21.15.1:0/2526895873 learned_addr learned my addr 172.21.15.1:0/2526895873 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.784+0000 7f4020afd700 1 --2- 172.21.15.1:0/2526895873 >> v2:172.21.15.165:3300/0 conn(0x7f401c15eb40 0x7f401c15d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.784+0000 7f400ffff700 1 -- 172.21.15.1:0/2526895873 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720718577 0 0) 0x7f401c06d300 con 0x7f401c157be0 2024-03-18T20:23:23.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.784+0000 7f400ffff700 1 -- 172.21.15.1:0/2526895873 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40080035b0 con 0x7f401c157be0 2024-03-18T20:23:23.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.784+0000 7f400ffff700 1 -- 172.21.15.1:0/2526895873 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243849628 0 0) 0x7f401c071600 con 0x7f401c0b25e0 2024-03-18T20:23:23.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.784+0000 7f400ffff700 1 -- 172.21.15.1:0/2526895873 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f401c06d300 con 0x7f401c0b25e0 2024-03-18T20:23:23.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.784+0000 7f4020afd700 1 -- 172.21.15.1:0/2526895873 >> v1:172.21.15.165:6789/0 conn(0x7f401c0b25e0 legacy=0x7f401c06da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:23.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.784+0000 7f4020afd700 1 -- 172.21.15.1:0/2526895873 >> v1:172.21.15.1:6789/0 conn(0x7f401c157be0 legacy=0x7f401c070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:23.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.784+0000 7f4020afd700 1 -- 172.21.15.1:0/2526895873 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c0718e0 con 0x7f401c15eb40 2024-03-18T20:23:23.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.784+0000 7f4020afd700 1 --2- 172.21.15.1:0/2526895873 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f401c15eb40 0x7f401c15d9e0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f4018009840 tx=0x7f4018009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b50fcbf86bf3e63f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:23.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.784+0000 7f400ffff700 1 -- 172.21.15.1:0/2526895873 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4018011830 con 0x7f401c15eb40 2024-03-18T20:23:23.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.785+0000 7f400ffff700 1 -- 172.21.15.1:0/2526895873 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4018011990 con 0x7f401c15eb40 2024-03-18T20:23:23.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.785+0000 7f400ffff700 1 -- 172.21.15.1:0/2526895873 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4018011d00 con 0x7f401c15eb40 2024-03-18T20:23:23.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.785+0000 7f4023562700 1 -- 172.21.15.1:0/2526895873 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f401c15eb40 msgr2=0x7f401c15d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:23.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.785+0000 7f4023562700 1 --2- 172.21.15.1:0/2526895873 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f401c15eb40 0x7f401c15d9e0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f4018009840 tx=0x7f4018009b50 comp rx=0 tx=0).stop 2024-03-18T20:23:23.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.785+0000 7f4023562700 1 -- 172.21.15.1:0/2526895873 shutdown_connections 2024-03-18T20:23:23.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.785+0000 7f4023562700 1 --2- 172.21.15.1:0/2526895873 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f401c15eb40 0x7f401c15d9e0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:23.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.785+0000 7f4023562700 1 -- 172.21.15.1:0/2526895873 >> 172.21.15.1:0/2526895873 conn(0x7f401c06a750 msgr2=0x7f401c06ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:23.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.785+0000 7f4023562700 1 -- 172.21.15.1:0/2526895873 shutdown_connections 2024-03-18T20:23:23.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.785+0000 7f4023562700 1 -- 172.21.15.1:0/2526895873 wait complete. 2024-03-18T20:23:23.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.786+0000 7f4023562700 1 Processor -- start 2024-03-18T20:23:23.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.787+0000 7f4023562700 1 -- start start 2024-03-18T20:23:23.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.787+0000 7f4023562700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f401c0b25e0 0x7f401c14a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:23.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.787+0000 7f4023562700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f401c157be0 0x7f401c0af910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:23.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.787+0000 7f4023562700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f401c15eb40 0x7f401c0c7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:23.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.787+0000 7f4023562700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c0afe50 con 0x7f401c157be0 2024-03-18T20:23:23.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.787+0000 7f4023562700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c0b0040 con 0x7f401c15eb40 2024-03-18T20:23:23.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.787+0000 7f4023562700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f401c0b0320 con 0x7f401c0b25e0 2024-03-18T20:23:23.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.787+0000 7f4020afd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f401c157be0 0x7f401c0af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:23.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.787+0000 7f4020afd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f401c157be0 0x7f401c0af910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57620/0 (socket says 172.21.15.1:57620) 2024-03-18T20:23:23.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.787+0000 7f4020afd700 1 -- 172.21.15.1:0/4256801160 learned_addr learned my addr 172.21.15.1:0/4256801160 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:23.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.788+0000 7f4021aff700 1 --2- 172.21.15.1:0/4256801160 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f401c15eb40 0x7f401c0c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:23.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.788+0000 7f40212fe700 1 --2- 172.21.15.1:0/4256801160 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f401c0b25e0 0x7f401c14a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:23.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.788+0000 7f4020afd700 1 -- 172.21.15.1:0/4256801160 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f401c0b25e0 msgr2=0x7f401c14a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:23.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.788+0000 7f4020afd700 1 --2- 172.21.15.1:0/4256801160 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f401c0b25e0 0x7f401c14a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:23.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.788+0000 7f4020afd700 1 -- 172.21.15.1:0/4256801160 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f401c15eb40 msgr2=0x7f401c0c7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:23.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.788+0000 7f4020afd700 1 --2- 172.21.15.1:0/4256801160 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f401c15eb40 0x7f401c0c7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:23.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.788+0000 7f4020afd700 1 -- 172.21.15.1:0/4256801160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40180094a0 con 0x7f401c157be0 2024-03-18T20:23:23.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.788+0000 7f4020afd700 1 --2- 172.21.15.1:0/4256801160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f401c157be0 0x7f401c0af910 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f4018009fd0 tx=0x7f4018004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:23.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.788+0000 7f400dffb700 1 -- 172.21.15.1:0/4256801160 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4018004550 con 0x7f401c157be0 2024-03-18T20:23:23.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.788+0000 7f400dffb700 1 -- 172.21.15.1:0/4256801160 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f401801f070 con 0x7f401c157be0 2024-03-18T20:23:23.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.788+0000 7f4023562700 1 -- 172.21.15.1:0/4256801160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f401c0b0600 con 0x7f401c157be0 2024-03-18T20:23:23.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.788+0000 7f400dffb700 1 -- 172.21.15.1:0/4256801160 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40180038e0 con 0x7f401c157be0 2024-03-18T20:23:23.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.788+0000 7f4023562700 1 -- 172.21.15.1:0/4256801160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401c0b0bc0 con 0x7f401c157be0 2024-03-18T20:23:23.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.789+0000 7f4023562700 1 -- 172.21.15.1:0/4256801160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f401c06b6e0 con 0x7f401c157be0 2024-03-18T20:23:23.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.790+0000 7f400dffb700 1 -- 172.21.15.1:0/4256801160 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 119) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f4018003a40 con 0x7f401c157be0 2024-03-18T20:23:23.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.790+0000 7f400dffb700 1 --2- 172.21.15.1:0/4256801160 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f400006adb0 0x7f400006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:23.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.791+0000 7f40212fe700 1 -- 172.21.15.1:0/4256801160 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f400006adb0 msgr2=0x7f400006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:23.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.791+0000 7f400dffb700 1 -- 172.21.15.1:0/4256801160 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6791+0+0 (secure 0 0 0) 0x7f4018021070 con 0x7f401c157be0 2024-03-18T20:23:23.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.791+0000 7f40212fe700 1 --2- 172.21.15.1:0/4256801160 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f400006adb0 0x7f400006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:23:23.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.794+0000 7f400dffb700 1 -- 172.21.15.1:0/4256801160 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4018055f30 con 0x7f401c157be0 2024-03-18T20:23:23.937 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:23.933+0000 7f5e66159240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:23:23.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.992+0000 7f40212fe700 1 -- 172.21.15.1:0/4256801160 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f400006adb0 msgr2=0x7f400006d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:23.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.992+0000 7f40212fe700 1 --2- 172.21.15.1:0/4256801160 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f400006adb0 0x7f400006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:23:24.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.997+0000 7f4023562700 1 -- 172.21.15.1:0/4256801160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f401c0676a0 con 0x7f401c157be0 2024-03-18T20:23:24.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:23.997+0000 7f400dffb700 1 -- 172.21.15.1:0/4256801160 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v200) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4018059550 con 0x7f401c157be0 2024-03-18T20:23:24.002 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:23:24.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.000+0000 7f4023562700 1 -- 172.21.15.1:0/4256801160 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f400006adb0 msgr2=0x7f400006d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:23:24.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.000+0000 7f4023562700 1 --2- 172.21.15.1:0/4256801160 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f400006adb0 0x7f400006d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.000+0000 7f4023562700 1 -- 172.21.15.1:0/4256801160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f401c157be0 msgr2=0x7f401c0af910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:24.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.000+0000 7f4023562700 1 --2- 172.21.15.1:0/4256801160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f401c157be0 0x7f401c0af910 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f4018009fd0 tx=0x7f4018004750 comp rx=0 tx=0).stop 2024-03-18T20:23:24.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.000+0000 7f4023562700 1 -- 172.21.15.1:0/4256801160 shutdown_connections 2024-03-18T20:23:24.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.000+0000 7f4023562700 1 --2- 172.21.15.1:0/4256801160 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f400006adb0 0x7f400006d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.000+0000 7f4023562700 1 --2- 172.21.15.1:0/4256801160 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f401c0b25e0 0x7f401c14a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.000+0000 7f4023562700 1 --2- 172.21.15.1:0/4256801160 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f401c15eb40 0x7f401c0c7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.000+0000 7f4023562700 1 --2- 172.21.15.1:0/4256801160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f401c157be0 0x7f401c0af910 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.000+0000 7f4023562700 1 -- 172.21.15.1:0/4256801160 >> 172.21.15.1:0/4256801160 conn(0x7f401c06a750 msgr2=0x7f401c157fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:24.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.000+0000 7f4023562700 1 -- 172.21.15.1:0/4256801160 shutdown_connections 2024-03-18T20:23:24.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.000+0000 7f4023562700 1 -- 172.21.15.1:0/4256801160 wait complete. 2024-03-18T20:23:24.015 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:24.011+0000 7f5e66159240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:23:24.016 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:23:24.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.143+0000 7fc9fdb79700 1 Processor -- start 2024-03-18T20:23:24.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.143+0000 7fc9fdb79700 1 -- start start 2024-03-18T20:23:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.144+0000 7fc9fdb79700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc9f815a5f0 0x7fc9f80cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.144+0000 7fc9fdb79700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9f80cc4d0 0x7fc9f806d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.144+0000 7fc9fdb79700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f815aad0 con 0x7fc9f815dbe0 2024-03-18T20:23:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.144+0000 7fc9fdb79700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9f80cc960 con 0x7fc9f815a5f0 2024-03-18T20:23:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.144+0000 7fc9fdb79700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f813f750 con 0x7fc9f80cc4d0 2024-03-18T20:23:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.144+0000 7fc9f77fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc9f815dbe0 0x7fc9f80b2fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43230/0 (socket says 172.21.15.1:43230) 2024-03-18T20:23:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.144+0000 7fc9f77fe700 1 -- 172.21.15.1:0/2535354448 learned_addr learned my addr 172.21.15.1:0/2535354448 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.145+0000 7fc9effff700 1 --2- 172.21.15.1:0/2535354448 >> v2:172.21.15.165:3300/0 conn(0x7fc9f815a5f0 0x7fc9f80cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.145+0000 7fc9f7fff700 1 --2- 172.21.15.1:0/2535354448 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9f80cc4d0 0x7fc9f806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.145+0000 7fc9f6ffd700 1 -- 172.21.15.1:0/2535354448 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882193614 0 0) 0x7fc9f815aad0 con 0x7fc9f815dbe0 2024-03-18T20:23:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.145+0000 7fc9f6ffd700 1 -- 172.21.15.1:0/2535354448 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e40035b0 con 0x7fc9f815dbe0 2024-03-18T20:23:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.145+0000 7fc9f7fff700 1 -- 172.21.15.1:0/2535354448 >> v2:172.21.15.165:3300/0 conn(0x7fc9f815a5f0 msgr2=0x7fc9f80cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.145+0000 7fc9f7fff700 1 --2- 172.21.15.1:0/2535354448 >> v2:172.21.15.165:3300/0 conn(0x7fc9f815a5f0 0x7fc9f80cbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.145+0000 7fc9f7fff700 1 -- 172.21.15.1:0/2535354448 >> v1:172.21.15.1:6789/0 conn(0x7fc9f815dbe0 legacy=0x7fc9f80b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.145+0000 7fc9f7fff700 1 -- 172.21.15.1:0/2535354448 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f815adb0 con 0x7fc9f80cc4d0 2024-03-18T20:23:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.145+0000 7fc9effff700 1 --2- 172.21.15.1:0/2535354448 >> v2:172.21.15.165:3300/0 conn(0x7fc9f815a5f0 0x7fc9f80cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:23:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.145+0000 7fc9f7fff700 1 --2- 172.21.15.1:0/2535354448 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9f80cc4d0 0x7fc9f806d500 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0009d90 tx=0x7fc9f000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c30c590833242cdf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.145+0000 7fc9f6ffd700 1 -- 172.21.15.1:0/2535354448 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f000e070 con 0x7fc9f80cc4d0 2024-03-18T20:23:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.145+0000 7fc9f6ffd700 1 -- 172.21.15.1:0/2535354448 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f000bb20 con 0x7fc9f80cc4d0 2024-03-18T20:23:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.145+0000 7fc9f6ffd700 1 -- 172.21.15.1:0/2535354448 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f0012430 con 0x7fc9f80cc4d0 2024-03-18T20:23:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.146+0000 7fc9fdb79700 1 -- 172.21.15.1:0/2535354448 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9f80cc4d0 msgr2=0x7fc9f806d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.146+0000 7fc9fdb79700 1 --2- 172.21.15.1:0/2535354448 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9f80cc4d0 0x7fc9f806d500 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0009d90 tx=0x7fc9f000b330 comp rx=0 tx=0).stop 2024-03-18T20:23:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.146+0000 7fc9fdb79700 1 -- 172.21.15.1:0/2535354448 shutdown_connections 2024-03-18T20:23:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.146+0000 7fc9fdb79700 1 --2- 172.21.15.1:0/2535354448 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9f80cc4d0 0x7fc9f806d500 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.146+0000 7fc9fdb79700 1 --2- 172.21.15.1:0/2535354448 >> v2:172.21.15.165:3300/0 conn(0x7fc9f815a5f0 0x7fc9f80cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.146+0000 7fc9fdb79700 1 -- 172.21.15.1:0/2535354448 >> 172.21.15.1:0/2535354448 conn(0x7fc9f80b1cb0 msgr2=0x7fc9f80b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.146+0000 7fc9fdb79700 1 -- 172.21.15.1:0/2535354448 shutdown_connections 2024-03-18T20:23:24.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.146+0000 7fc9fdb79700 1 -- 172.21.15.1:0/2535354448 wait complete. 2024-03-18T20:23:24.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.148+0000 7fc9fdb79700 1 Processor -- start 2024-03-18T20:23:24.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.148+0000 7fc9fdb79700 1 -- start start 2024-03-18T20:23:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.148+0000 7fc9fdb79700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9f80cc4d0 0x7fc9f8150490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.148+0000 7fc9fdb79700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f815a5f0 0x7fc9f8152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.148+0000 7fc9fdb79700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9f815dbe0 0x7fc9f8158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.148+0000 7fc9fdb79700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f81542a0 con 0x7fc9f815a5f0 2024-03-18T20:23:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.148+0000 7fc9fdb79700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f8153620 con 0x7fc9f80cc4d0 2024-03-18T20:23:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.148+0000 7fc9fdb79700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f8153900 con 0x7fc9f815dbe0 2024-03-18T20:23:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.148+0000 7fc9effff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f815a5f0 0x7fc9f8152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.149+0000 7fc9effff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f815a5f0 0x7fc9f8152cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57632/0 (socket says 172.21.15.1:57632) 2024-03-18T20:23:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.149+0000 7fc9effff700 1 -- 172.21.15.1:0/55874710 learned_addr learned my addr 172.21.15.1:0/55874710 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.149+0000 7fc9f77fe700 1 --2- 172.21.15.1:0/55874710 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9f80cc4d0 0x7fc9f8150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.149+0000 7fc9f7fff700 1 --2- 172.21.15.1:0/55874710 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9f815dbe0 0x7fc9f8158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.149+0000 7fc9effff700 1 -- 172.21.15.1:0/55874710 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9f815dbe0 msgr2=0x7fc9f8158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.149+0000 7fc9effff700 1 --2- 172.21.15.1:0/55874710 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9f815dbe0 0x7fc9f8158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.149+0000 7fc9effff700 1 -- 172.21.15.1:0/55874710 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9f80cc4d0 msgr2=0x7fc9f8150490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:24.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.149+0000 7fc9effff700 1 --2- 172.21.15.1:0/55874710 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9f80cc4d0 0x7fc9f8150490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.149+0000 7fc9effff700 1 -- 172.21.15.1:0/55874710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e00094a0 con 0x7fc9f815a5f0 2024-03-18T20:23:24.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.149+0000 7fc9effff700 1 --2- 172.21.15.1:0/55874710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f815a5f0 0x7fc9f8152cd0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000ebe0 tx=0x7fc9e000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:24.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.149+0000 7fc9f4ff9700 1 -- 172.21.15.1:0/55874710 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e00097a0 con 0x7fc9f815a5f0 2024-03-18T20:23:24.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.149+0000 7fc9f4ff9700 1 -- 172.21.15.1:0/55874710 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e000c850 con 0x7fc9f815a5f0 2024-03-18T20:23:24.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.149+0000 7fc9fdb79700 1 -- 172.21.15.1:0/55874710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f0009a70 con 0x7fc9f815a5f0 2024-03-18T20:23:24.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.149+0000 7fc9f4ff9700 1 -- 172.21.15.1:0/55874710 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e000cbe0 con 0x7fc9f815a5f0 2024-03-18T20:23:24.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.149+0000 7fc9fdb79700 1 -- 172.21.15.1:0/55874710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f8153ec0 con 0x7fc9f815a5f0 2024-03-18T20:23:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.150+0000 7fc9fdb79700 1 -- 172.21.15.1:0/55874710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f8073350 con 0x7fc9f815a5f0 2024-03-18T20:23:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.151+0000 7fc9f4ff9700 1 -- 172.21.15.1:0/55874710 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 119) v1 ==== 89251+0+0 (secure 0 0 0) 0x7fc9e000cd40 con 0x7fc9f815a5f0 2024-03-18T20:23:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.151+0000 7fc9f4ff9700 1 --2- 172.21.15.1:0/55874710 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc9d806adb0 0x7fc9d806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.151+0000 7fc9f77fe700 1 -- 172.21.15.1:0/55874710 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc9d806adb0 msgr2=0x7fc9d806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.151+0000 7fc9f4ff9700 1 -- 172.21.15.1:0/55874710 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6791+0+0 (secure 0 0 0) 0x7fc9e0089570 con 0x7fc9f815a5f0 2024-03-18T20:23:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.151+0000 7fc9f77fe700 1 --2- 172.21.15.1:0/55874710 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc9d806adb0 0x7fc9d806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:23:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.156+0000 7fc9f4ff9700 1 -- 172.21.15.1:0/55874710 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc9e0054290 con 0x7fc9f815a5f0 2024-03-18T20:23:24.209 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:24.205+0000 7f4e634a7240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:23:24.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.352+0000 7fc9f77fe700 1 -- 172.21.15.1:0/55874710 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc9d806adb0 msgr2=0x7fc9d806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:24.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.352+0000 7fc9f77fe700 1 --2- 172.21.15.1:0/55874710 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc9d806adb0 0x7fc9d806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:23:24.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.360+0000 7fc9fdb79700 1 -- 172.21.15.1:0/55874710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc9f80cc2c0 con 0x7fc9f815a5f0 2024-03-18T20:23:24.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.361+0000 7fc9f4ff9700 1 -- 172.21.15.1:0/55874710 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v201) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc9e00578b0 con 0x7fc9f815a5f0 2024-03-18T20:23:24.365 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:23:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.363+0000 7fc9fdb79700 1 -- 172.21.15.1:0/55874710 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc9d806adb0 msgr2=0x7fc9d806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:23:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.363+0000 7fc9fdb79700 1 --2- 172.21.15.1:0/55874710 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc9d806adb0 0x7fc9d806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.363+0000 7fc9fdb79700 1 -- 172.21.15.1:0/55874710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f815a5f0 msgr2=0x7fc9f8152cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.363+0000 7fc9fdb79700 1 --2- 172.21.15.1:0/55874710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f815a5f0 0x7fc9f8152cd0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000ebe0 tx=0x7fc9e000eef0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.363+0000 7fc9fdb79700 1 -- 172.21.15.1:0/55874710 shutdown_connections 2024-03-18T20:23:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.363+0000 7fc9fdb79700 1 --2- 172.21.15.1:0/55874710 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc9d806adb0 0x7fc9d806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.363+0000 7fc9fdb79700 1 --2- 172.21.15.1:0/55874710 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc9f815dbe0 0x7fc9f8158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.363+0000 7fc9fdb79700 1 --2- 172.21.15.1:0/55874710 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9f80cc4d0 0x7fc9f8150490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.363+0000 7fc9fdb79700 1 --2- 172.21.15.1:0/55874710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9f815a5f0 0x7fc9f8152cd0 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.363+0000 7fc9fdb79700 1 -- 172.21.15.1:0/55874710 >> 172.21.15.1:0/55874710 conn(0x7fc9f80b1cb0 msgr2=0x7fc9f806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:24.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.363+0000 7fc9fdb79700 1 -- 172.21.15.1:0/55874710 shutdown_connections 2024-03-18T20:23:24.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.363+0000 7fc9fdb79700 1 -- 172.21.15.1:0/55874710 wait complete. 2024-03-18T20:23:24.380 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:23:24.511 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:24.506+0000 7f5e66159240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:23:24.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.507+0000 7f9e98dee700 1 Processor -- start 2024-03-18T20:23:24.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.507+0000 7f9e98dee700 1 -- start start 2024-03-18T20:23:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.508+0000 7f9e98dee700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e940b2160 0x7f9e94166300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.508+0000 7f9e98dee700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e940d01f0 con 0x7f9e9415b6c0 2024-03-18T20:23:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.508+0000 7f9e98dee700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e940ccfa0 con 0x7f9e940cd430 2024-03-18T20:23:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.508+0000 7f9e98dee700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e940d1db0 con 0x7f9e940b2160 2024-03-18T20:23:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.509+0000 7f9e9259c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9e9415b6c0 0x7f9e940c80c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42368/0 (socket says 172.21.15.1:42368) 2024-03-18T20:23:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.509+0000 7f9e9259c700 1 -- 172.21.15.1:0/2035888407 learned_addr learned my addr 172.21.15.1:0/2035888407 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.509+0000 7f9e92d9d700 1 --2- 172.21.15.1:0/2035888407 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e940b2160 0x7f9e94166300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.509+0000 7f9e9159a700 1 -- 172.21.15.1:0/2035888407 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071083856 0 0) 0x7f9e940d01f0 con 0x7f9e9415b6c0 2024-03-18T20:23:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.509+0000 7f9e9159a700 1 -- 172.21.15.1:0/2035888407 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e7c0035b0 con 0x7f9e9415b6c0 2024-03-18T20:23:24.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.509+0000 7f9e9159a700 1 -- 172.21.15.1:0/2035888407 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863545125 0 0) 0x7f9e940ccfa0 con 0x7f9e940cd430 2024-03-18T20:23:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.509+0000 7f9e9159a700 1 -- 172.21.15.1:0/2035888407 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e940d01f0 con 0x7f9e940cd430 2024-03-18T20:23:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.509+0000 7f9e9159a700 1 -- 172.21.15.1:0/2035888407 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 11205084 0 0) 0x7f9e7c0035b0 con 0x7f9e9415b6c0 2024-03-18T20:23:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.509+0000 7f9e92d9d700 1 -- 172.21.15.1:0/2035888407 >> v1:172.21.15.165:6789/0 conn(0x7f9e940cd430 legacy=0x7f9e940c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.509+0000 7f9e92d9d700 1 -- 172.21.15.1:0/2035888407 >> v1:172.21.15.1:6789/0 conn(0x7f9e9415b6c0 legacy=0x7f9e940c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.509+0000 7f9e92d9d700 1 -- 172.21.15.1:0/2035888407 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e940cff10 con 0x7f9e940b2160 2024-03-18T20:23:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.509+0000 7f9e92d9d700 1 --2- 172.21.15.1:0/2035888407 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e940b2160 0x7f9e94166300 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009dc0 tx=0x7f9e8c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f45b745eef000d3f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.509+0000 7f9e9159a700 1 -- 172.21.15.1:0/2035888407 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c00e070 con 0x7f9e940b2160 2024-03-18T20:23:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.510+0000 7f9e9159a700 1 -- 172.21.15.1:0/2035888407 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8c003a20 con 0x7f9e940b2160 2024-03-18T20:23:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.510+0000 7f9e9159a700 1 -- 172.21.15.1:0/2035888407 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e84001670 con 0x7f9e940b2160 2024-03-18T20:23:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.510+0000 7f9e98dee700 1 -- 172.21.15.1:0/2035888407 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e940b2160 msgr2=0x7f9e94166300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.510+0000 7f9e98dee700 1 --2- 172.21.15.1:0/2035888407 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e940b2160 0x7f9e94166300 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009dc0 tx=0x7f9e8c00b330 comp rx=0 tx=0).stop 2024-03-18T20:23:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.510+0000 7f9e98dee700 1 -- 172.21.15.1:0/2035888407 shutdown_connections 2024-03-18T20:23:24.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.510+0000 7f9e98dee700 1 --2- 172.21.15.1:0/2035888407 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e940b2160 0x7f9e94166300 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.510+0000 7f9e98dee700 1 -- 172.21.15.1:0/2035888407 >> 172.21.15.1:0/2035888407 conn(0x7f9e940d0ff0 msgr2=0x7f9e940d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:24.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.510+0000 7f9e98dee700 1 -- 172.21.15.1:0/2035888407 shutdown_connections 2024-03-18T20:23:24.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.510+0000 7f9e98dee700 1 -- 172.21.15.1:0/2035888407 wait complete. 2024-03-18T20:23:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.512+0000 7f9e98dee700 1 Processor -- start 2024-03-18T20:23:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.512+0000 7f9e98dee700 1 -- start start 2024-03-18T20:23:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.512+0000 7f9e98dee700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e940b2160 0x7f9e94152e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.512+0000 7f9e98dee700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e940cd430 0x7f9e941e5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.512+0000 7f9e98dee700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9415b6c0 0x7f9e941567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.512+0000 7f9e98dee700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e94153370 con 0x7f9e940cd430 2024-03-18T20:23:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.512+0000 7f9e98dee700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e941e56a0 con 0x7f9e9415b6c0 2024-03-18T20:23:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.512+0000 7f9e98dee700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e94156cf0 con 0x7f9e940b2160 2024-03-18T20:23:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.512+0000 7f9e91d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e940cd430 0x7f9e941e5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.512+0000 7f9e9259c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e940b2160 0x7f9e94152e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.512+0000 7f9e91d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e940cd430 0x7f9e941e5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36836/0 (socket says 172.21.15.1:36836) 2024-03-18T20:23:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.512+0000 7f9e91d9b700 1 -- 172.21.15.1:0/313303768 learned_addr learned my addr 172.21.15.1:0/313303768 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.513+0000 7f9e92d9d700 1 --2- 172.21.15.1:0/313303768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9415b6c0 0x7f9e941567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.513+0000 7f9e91d9b700 1 -- 172.21.15.1:0/313303768 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e940b2160 msgr2=0x7f9e94152e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.513+0000 7f9e91d9b700 1 --2- 172.21.15.1:0/313303768 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e940b2160 0x7f9e94152e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.513+0000 7f9e91d9b700 1 -- 172.21.15.1:0/313303768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9415b6c0 msgr2=0x7f9e941567b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.513+0000 7f9e91d9b700 1 --2- 172.21.15.1:0/313303768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9415b6c0 0x7f9e941567b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.513+0000 7f9e91d9b700 1 -- 172.21.15.1:0/313303768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8c009a70 con 0x7f9e940cd430 2024-03-18T20:23:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.513+0000 7f9e9259c700 1 --2- 172.21.15.1:0/313303768 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e940b2160 0x7f9e94152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:23:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.513+0000 7f9e91d9b700 1 --2- 172.21.15.1:0/313303768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e940cd430 0x7f9e941e5160 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f9e8800c520 tx=0x7f9e8800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.513+0000 7f9e82ffd700 1 -- 172.21.15.1:0/313303768 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e880052d0 con 0x7f9e940cd430 2024-03-18T20:23:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.513+0000 7f9e82ffd700 1 -- 172.21.15.1:0/313303768 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e88005430 con 0x7f9e940cd430 2024-03-18T20:23:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.513+0000 7f9e82ffd700 1 -- 172.21.15.1:0/313303768 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e880038c0 con 0x7f9e940cd430 2024-03-18T20:23:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.513+0000 7f9e98dee700 1 -- 172.21.15.1:0/313303768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e941e5920 con 0x7f9e940cd430 2024-03-18T20:23:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.513+0000 7f9e98dee700 1 -- 172.21.15.1:0/313303768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e941e5ee0 con 0x7f9e940cd430 2024-03-18T20:23:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.514+0000 7f9e98dee700 1 -- 172.21.15.1:0/313303768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e940676a0 con 0x7f9e940cd430 2024-03-18T20:23:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.515+0000 7f9e82ffd700 1 -- 172.21.15.1:0/313303768 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 119) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f9e88003a20 con 0x7f9e940cd430 2024-03-18T20:23:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.515+0000 7f9e82ffd700 1 --2- 172.21.15.1:0/313303768 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e7406adb0 0x7f9e7406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.515+0000 7f9e9259c700 1 -- 172.21.15.1:0/313303768 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e7406adb0 msgr2=0x7f9e7406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.515+0000 7f9e82ffd700 1 -- 172.21.15.1:0/313303768 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6775+0+0 (secure 0 0 0) 0x7f9e88088d50 con 0x7f9e940cd430 2024-03-18T20:23:24.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.515+0000 7f9e9259c700 1 --2- 172.21.15.1:0/313303768 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e7406adb0 0x7f9e7406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:23:24.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.525+0000 7f9e82ffd700 1 -- 172.21.15.1:0/313303768 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9e8808c030 con 0x7f9e940cd430 2024-03-18T20:23:24.552 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:24.548+0000 7f4e634a7240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:23:24.717 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:24.713+0000 7f4e634a7240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:23:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.716+0000 7f9e9259c700 1 -- 172.21.15.1:0/313303768 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e7406adb0 msgr2=0x7f9e7406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:24.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.716+0000 7f9e9259c700 1 --2- 172.21.15.1:0/313303768 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e7406adb0 0x7f9e7406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:23:24.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.733+0000 7f9e98dee700 1 -- 172.21.15.1:0/313303768 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9e940c8a00 con 0x7f9e940cd430 2024-03-18T20:23:24.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.734+0000 7f9e82ffd700 1 -- 172.21.15.1:0/313303768 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v201) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9e88017020 con 0x7f9e940cd430 2024-03-18T20:23:24.738 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:23:24.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.736+0000 7f9e98dee700 1 -- 172.21.15.1:0/313303768 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e7406adb0 msgr2=0x7f9e7406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:23:24.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.736+0000 7f9e98dee700 1 --2- 172.21.15.1:0/313303768 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e7406adb0 0x7f9e7406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.736+0000 7f9e98dee700 1 -- 172.21.15.1:0/313303768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e940cd430 msgr2=0x7f9e941e5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:24.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.736+0000 7f9e98dee700 1 --2- 172.21.15.1:0/313303768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e940cd430 0x7f9e941e5160 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f9e8800c520 tx=0x7f9e8800c830 comp rx=0 tx=0).stop 2024-03-18T20:23:24.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.736+0000 7f9e98dee700 1 -- 172.21.15.1:0/313303768 shutdown_connections 2024-03-18T20:23:24.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.736+0000 7f9e98dee700 1 --2- 172.21.15.1:0/313303768 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e7406adb0 0x7f9e7406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.736+0000 7f9e98dee700 1 --2- 172.21.15.1:0/313303768 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e940b2160 0x7f9e94152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.736+0000 7f9e98dee700 1 --2- 172.21.15.1:0/313303768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9415b6c0 0x7f9e941567b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.736+0000 7f9e98dee700 1 --2- 172.21.15.1:0/313303768 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e940cd430 0x7f9e941e5160 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.736+0000 7f9e98dee700 1 -- 172.21.15.1:0/313303768 >> 172.21.15.1:0/313303768 conn(0x7f9e940d0ff0 msgr2=0x7f9e9415a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:24.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.736+0000 7f9e98dee700 1 -- 172.21.15.1:0/313303768 shutdown_connections 2024-03-18T20:23:24.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.736+0000 7f9e98dee700 1 -- 172.21.15.1:0/313303768 wait complete. 2024-03-18T20:23:24.754 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:23:24.885 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:24.880+0000 7f5e66159240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:23:24.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.885+0000 7f59b63dc700 1 Processor -- start 2024-03-18T20:23:24.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.885+0000 7f59b63dc700 1 -- start start 2024-03-18T20:23:24.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.886+0000 7f59b63dc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59b015b3d0 0x7f59b00cf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:24.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.886+0000 7f59b63dc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f59b0073b00 0x7f59b00cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:24.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.886+0000 7f59b63dc700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b00ce200 con 0x7f59b015aff0 2024-03-18T20:23:24.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.886+0000 7f59b63dc700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59b00ce4a0 con 0x7f59b0073b00 2024-03-18T20:23:24.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.886+0000 7f59b63dc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b00ce780 con 0x7f59b015b3d0 2024-03-18T20:23:24.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.886+0000 7f59b4979700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f59b0073b00 0x7f59b00cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:24.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.886+0000 7f59b4979700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f59b0073b00 0x7f59b00cfa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36852/0 (socket says 172.21.15.1:36852) 2024-03-18T20:23:24.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.886+0000 7f59affff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f59b015aff0 0x7f59b015f780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42376/0 (socket says 172.21.15.1:42376) 2024-03-18T20:23:24.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.886+0000 7f59b4979700 1 -- 172.21.15.1:0/3965855099 learned_addr learned my addr 172.21.15.1:0/3965855099 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:24.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.886+0000 7f59af7fe700 1 --2- 172.21.15.1:0/3965855099 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59b015b3d0 0x7f59b00cf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:24.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.886+0000 7f59b4979700 1 -- 172.21.15.1:0/3965855099 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59b015b3d0 msgr2=0x7f59b00cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:24.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.886+0000 7f59b4979700 1 --2- 172.21.15.1:0/3965855099 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59b015b3d0 0x7f59b00cf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.886+0000 7f59b4979700 1 -- 172.21.15.1:0/3965855099 >> v1:172.21.15.1:6789/0 conn(0x7f59b015aff0 legacy=0x7f59b015f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:24.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.886+0000 7f59b4979700 1 -- 172.21.15.1:0/3965855099 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b00cea60 con 0x7f59b0073b00 2024-03-18T20:23:24.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.886+0000 7f59af7fe700 1 --2- 172.21.15.1:0/3965855099 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59b015b3d0 0x7f59b00cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:23:24.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.887+0000 7f59b4979700 1 --2- 172.21.15.1:0/3965855099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59b0073b00 0x7f59b00cfa80 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f59a8009e50 tx=0x7f59a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=887a38427e20a242 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:24.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.887+0000 7f59aeffd700 1 -- 172.21.15.1:0/3965855099 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a800e070 con 0x7f59b0073b00 2024-03-18T20:23:24.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.887+0000 7f59aeffd700 1 -- 172.21.15.1:0/3965855099 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59a800b9c0 con 0x7f59b0073b00 2024-03-18T20:23:24.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.887+0000 7f59aeffd700 1 -- 172.21.15.1:0/3965855099 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a800bd70 con 0x7f59b0073b00 2024-03-18T20:23:24.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.888+0000 7f59b63dc700 1 -- 172.21.15.1:0/3965855099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59b0073b00 msgr2=0x7f59b00cfa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:24.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.888+0000 7f59b63dc700 1 --2- 172.21.15.1:0/3965855099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59b0073b00 0x7f59b00cfa80 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f59a8009e50 tx=0x7f59a800b330 comp rx=0 tx=0).stop 2024-03-18T20:23:24.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.888+0000 7f59b63dc700 1 -- 172.21.15.1:0/3965855099 shutdown_connections 2024-03-18T20:23:24.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.888+0000 7f59b63dc700 1 --2- 172.21.15.1:0/3965855099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59b0073b00 0x7f59b00cfa80 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.888+0000 7f59b63dc700 1 --2- 172.21.15.1:0/3965855099 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59b015b3d0 0x7f59b00cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.888+0000 7f59b63dc700 1 -- 172.21.15.1:0/3965855099 >> 172.21.15.1:0/3965855099 conn(0x7f59b006a750 msgr2=0x7f59b006ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:24.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.888+0000 7f59b63dc700 1 -- 172.21.15.1:0/3965855099 shutdown_connections 2024-03-18T20:23:24.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.888+0000 7f59b63dc700 1 -- 172.21.15.1:0/3965855099 wait complete. 2024-03-18T20:23:24.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.889+0000 7f59b63dc700 1 Processor -- start 2024-03-18T20:23:24.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.889+0000 7f59b63dc700 1 -- start start 2024-03-18T20:23:24.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.890+0000 7f59b63dc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59b0073b00 0x7f59b00b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:24.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.890+0000 7f59b63dc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59b015aff0 0x7f59b0138db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:24.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.890+0000 7f59b63dc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59b015b3d0 0x7f59b0158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:24.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.890+0000 7f59b63dc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b00b4810 con 0x7f59b0073b00 2024-03-18T20:23:24.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.890+0000 7f59b63dc700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b00b3890 con 0x7f59b015b3d0 2024-03-18T20:23:24.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.890+0000 7f59b63dc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b01392f0 con 0x7f59b015aff0 2024-03-18T20:23:24.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.890+0000 7f59affff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59b0073b00 0x7f59b00b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:24.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.890+0000 7f59affff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59b0073b00 0x7f59b00b3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36864/0 (socket says 172.21.15.1:36864) 2024-03-18T20:23:24.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.890+0000 7f59affff700 1 -- 172.21.15.1:0/3418567262 learned_addr learned my addr 172.21.15.1:0/3418567262 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:24.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.890+0000 7f59af7fe700 1 --2- 172.21.15.1:0/3418567262 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59b015aff0 0x7f59b0138db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:24.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.890+0000 7f59b4979700 1 --2- 172.21.15.1:0/3418567262 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59b015b3d0 0x7f59b0158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:24.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.890+0000 7f59affff700 1 -- 172.21.15.1:0/3418567262 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59b015aff0 msgr2=0x7f59b0138db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:24.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.890+0000 7f59affff700 1 --2- 172.21.15.1:0/3418567262 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59b015aff0 0x7f59b0138db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.890+0000 7f59affff700 1 -- 172.21.15.1:0/3418567262 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59b015b3d0 msgr2=0x7f59b0158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:24.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.890+0000 7f59affff700 1 --2- 172.21.15.1:0/3418567262 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59b015b3d0 0x7f59b0158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:24.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.890+0000 7f59affff700 1 -- 172.21.15.1:0/3418567262 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a8009b30 con 0x7f59b0073b00 2024-03-18T20:23:24.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.890+0000 7f59affff700 1 --2- 172.21.15.1:0/3418567262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59b0073b00 0x7f59b00b3d20 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f59a000c520 tx=0x7f59a000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:24.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.891+0000 7f59acff9700 1 -- 172.21.15.1:0/3418567262 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a0005cb0 con 0x7f59b0073b00 2024-03-18T20:23:24.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.891+0000 7f59acff9700 1 -- 172.21.15.1:0/3418567262 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59a0005e10 con 0x7f59b0073b00 2024-03-18T20:23:24.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.891+0000 7f59b63dc700 1 -- 172.21.15.1:0/3418567262 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b00b2c80 con 0x7f59b0073b00 2024-03-18T20:23:24.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.891+0000 7f59acff9700 1 -- 172.21.15.1:0/3418567262 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a00072d0 con 0x7f59b0073b00 2024-03-18T20:23:24.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.891+0000 7f59b63dc700 1 -- 172.21.15.1:0/3418567262 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b00b3240 con 0x7f59b0073b00 2024-03-18T20:23:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.892+0000 7f59b63dc700 1 -- 172.21.15.1:0/3418567262 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b00676a0 con 0x7f59b0073b00 2024-03-18T20:23:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.892+0000 7f59acff9700 1 -- 172.21.15.1:0/3418567262 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 119) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f59a0007430 con 0x7f59b0073b00 2024-03-18T20:23:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.892+0000 7f59acff9700 1 --2- 172.21.15.1:0/3418567262 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f599c06adb0 0x7f599c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.892+0000 7f59af7fe700 1 -- 172.21.15.1:0/3418567262 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f599c06adb0 msgr2=0x7f599c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.893+0000 7f59acff9700 1 -- 172.21.15.1:0/3418567262 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6775+0+0 (secure 0 0 0) 0x7f59a0088bb0 con 0x7f59b0073b00 2024-03-18T20:23:24.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.893+0000 7f59af7fe700 1 --2- 172.21.15.1:0/3418567262 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f599c06adb0 0x7f599c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:23:24.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:24.897+0000 7f59acff9700 1 -- 172.21.15.1:0/3418567262 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f59a0053900 con 0x7f59b0073b00 2024-03-18T20:23:25.063 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:25.058+0000 7f5e66159240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:23:25.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.094+0000 7f59af7fe700 1 -- 172.21.15.1:0/3418567262 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f599c06adb0 msgr2=0x7f599c06d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:25.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.094+0000 7f59af7fe700 1 --2- 172.21.15.1:0/3418567262 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f599c06adb0 0x7f599c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:23:25.109 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:25.104+0000 7f4e634a7240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:23:25.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.146+0000 7f59b63dc700 1 -- 172.21.15.1:0/3418567262 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59b00ccba0 con 0x7f59b0073b00 2024-03-18T20:23:25.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.147+0000 7f59acff9700 1 -- 172.21.15.1:0/3418567262 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4741 (secure 0 0 0) 0x7f59a0017020 con 0x7f59b0073b00 2024-03-18T20:23:25.151 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:25.152 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":89,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8204,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2572860438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2572860438},{"type":"v1","addr":"172.21.15.165:6835","nonce":2572860438}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9439,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3158632513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3158632513},{"type":"v1","addr":"172.21.15.1:6836","nonce":3158632513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9448,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/4005747445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4005747445},{"type":"v1","addr":"172.21.15.165:6834","nonce":4005747445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:07:19.322806+0000","modified":"2024-03-18T20:07:22.335335+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10422":{"gid":10422,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1610313086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1610313086},{"type":"v1","addr":"172.21.15.1:6837","nonce":1610313086}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-03-18T20:23:25.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.150+0000 7f59b63dc700 1 -- 172.21.15.1:0/3418567262 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f599c06adb0 msgr2=0x7f599c06d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:23:25.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.150+0000 7f59b63dc700 1 --2- 172.21.15.1:0/3418567262 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f599c06adb0 0x7f599c06d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.150+0000 7f59b63dc700 1 -- 172.21.15.1:0/3418567262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59b0073b00 msgr2=0x7f59b00b3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:25.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.150+0000 7f59b63dc700 1 --2- 172.21.15.1:0/3418567262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59b0073b00 0x7f59b00b3d20 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f59a000c520 tx=0x7f59a000c830 comp rx=0 tx=0).stop 2024-03-18T20:23:25.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.151+0000 7f59b63dc700 1 -- 172.21.15.1:0/3418567262 shutdown_connections 2024-03-18T20:23:25.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.151+0000 7f59b63dc700 1 --2- 172.21.15.1:0/3418567262 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f599c06adb0 0x7f599c06d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.151+0000 7f59b63dc700 1 --2- 172.21.15.1:0/3418567262 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59b015b3d0 0x7f59b0158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.151+0000 7f59b63dc700 1 --2- 172.21.15.1:0/3418567262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59b0073b00 0x7f59b00b3d20 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.151+0000 7f59b63dc700 1 --2- 172.21.15.1:0/3418567262 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59b015aff0 0x7f59b0138db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.151+0000 7f59b63dc700 1 -- 172.21.15.1:0/3418567262 >> 172.21.15.1:0/3418567262 conn(0x7f59b006a750 msgr2=0x7f59b006ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:25.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.151+0000 7f59b63dc700 1 -- 172.21.15.1:0/3418567262 shutdown_connections 2024-03-18T20:23:25.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.151+0000 7f59b63dc700 1 -- 172.21.15.1:0/3418567262 wait complete. 2024-03-18T20:23:25.156 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 89 2024-03-18T20:23:25.168 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:23:25.277 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:25.272+0000 7f4e634a7240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:23:25.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.291+0000 7f4692569700 1 Processor -- start 2024-03-18T20:23:25.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.291+0000 7f4692569700 1 -- start start 2024-03-18T20:23:25.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.292+0000 7f4692569700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f468c0ae600 0x7f468c0ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:25.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.292+0000 7f4692569700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468c06e270 con 0x7f468c15a2e0 2024-03-18T20:23:25.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.292+0000 7f4692569700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468c0ae050 con 0x7f468c0cc170 2024-03-18T20:23:25.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.292+0000 7f4692569700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f468c06f690 con 0x7f468c0ae600 2024-03-18T20:23:25.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.292+0000 7f468b7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f468c15a2e0 0x7f468c0ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42388/0 (socket says 172.21.15.1:42388) 2024-03-18T20:23:25.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.292+0000 7f468b7fe700 1 -- 172.21.15.1:0/21713137 learned_addr learned my addr 172.21.15.1:0/21713137 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:25.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.293+0000 7f468bfff700 1 --2- 172.21.15.1:0/21713137 >> v2:172.21.15.165:3300/0 conn(0x7f468c0ae600 0x7f468c0ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:25.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.293+0000 7f468affd700 1 -- 172.21.15.1:0/21713137 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338854957 0 0) 0x7f468c06e270 con 0x7f468c15a2e0 2024-03-18T20:23:25.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.293+0000 7f468affd700 1 -- 172.21.15.1:0/21713137 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46740035b0 con 0x7f468c15a2e0 2024-03-18T20:23:25.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.293+0000 7f468affd700 1 -- 172.21.15.1:0/21713137 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630595302 0 0) 0x7f468c0ae050 con 0x7f468c0cc170 2024-03-18T20:23:25.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.293+0000 7f468affd700 1 -- 172.21.15.1:0/21713137 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f468c06e270 con 0x7f468c0cc170 2024-03-18T20:23:25.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.293+0000 7f468affd700 1 -- 172.21.15.1:0/21713137 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1260855755 0 0) 0x7f46740035b0 con 0x7f468c15a2e0 2024-03-18T20:23:25.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.293+0000 7f468bfff700 1 -- 172.21.15.1:0/21713137 >> v1:172.21.15.165:6789/0 conn(0x7f468c0cc170 legacy=0x7f468c06ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.293+0000 7f468bfff700 1 -- 172.21.15.1:0/21713137 >> v1:172.21.15.1:6789/0 conn(0x7f468c15a2e0 legacy=0x7f468c0ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.293+0000 7f468bfff700 1 -- 172.21.15.1:0/21713137 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468c0add70 con 0x7f468c0ae600 2024-03-18T20:23:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.293+0000 7f468bfff700 1 --2- 172.21.15.1:0/21713137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f468c0ae600 0x7f468c0ad790 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f467c009840 tx=0x7f467c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6c49b38f046d8bc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.293+0000 7f468affd700 1 -- 172.21.15.1:0/21713137 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4680001670 con 0x7f468c0ae600 2024-03-18T20:23:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.293+0000 7f468affd700 1 -- 172.21.15.1:0/21713137 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467c004030 con 0x7f468c0ae600 2024-03-18T20:23:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.293+0000 7f468affd700 1 -- 172.21.15.1:0/21713137 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467c0043c0 con 0x7f468c0ae600 2024-03-18T20:23:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.294+0000 7f4692569700 1 -- 172.21.15.1:0/21713137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f468c0ae600 msgr2=0x7f468c0ad790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.294+0000 7f4692569700 1 --2- 172.21.15.1:0/21713137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f468c0ae600 0x7f468c0ad790 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f467c009840 tx=0x7f467c009b50 comp rx=0 tx=0).stop 2024-03-18T20:23:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.294+0000 7f4692569700 1 -- 172.21.15.1:0/21713137 shutdown_connections 2024-03-18T20:23:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.294+0000 7f4692569700 1 --2- 172.21.15.1:0/21713137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f468c0ae600 0x7f468c0ad790 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.294+0000 7f4692569700 1 -- 172.21.15.1:0/21713137 >> 172.21.15.1:0/21713137 conn(0x7f468c06b860 msgr2=0x7f468c06bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.294+0000 7f4692569700 1 -- 172.21.15.1:0/21713137 shutdown_connections 2024-03-18T20:23:25.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.294+0000 7f4692569700 1 -- 172.21.15.1:0/21713137 wait complete. 2024-03-18T20:23:25.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.295+0000 7f4692569700 1 Processor -- start 2024-03-18T20:23:25.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.295+0000 7f4692569700 1 -- start start 2024-03-18T20:23:25.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.296+0000 7f4692569700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f468c0ae600 0x7f468c14ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:25.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.296+0000 7f4692569700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f468c0cc170 0x7f468c157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:25.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.296+0000 7f4692569700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f468c15a2e0 0x7f468c157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:25.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.296+0000 7f4692569700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c154db0 con 0x7f468c0ae600 2024-03-18T20:23:25.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.296+0000 7f4692569700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c155090 con 0x7f468c0cc170 2024-03-18T20:23:25.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.296+0000 7f4692569700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468c155370 con 0x7f468c15a2e0 2024-03-18T20:23:25.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.296+0000 7f468bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f468c0ae600 0x7f468c14ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:25.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.296+0000 7f468bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f468c0ae600 0x7f468c14ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36866/0 (socket says 172.21.15.1:36866) 2024-03-18T20:23:25.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.296+0000 7f468bfff700 1 -- 172.21.15.1:0/2315635913 learned_addr learned my addr 172.21.15.1:0/2315635913 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:25.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.296+0000 7f468b7fe700 1 --2- 172.21.15.1:0/2315635913 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f468c0cc170 0x7f468c157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:25.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.296+0000 7f4690b06700 1 --2- 172.21.15.1:0/2315635913 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f468c15a2e0 0x7f468c157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:25.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.296+0000 7f468bfff700 1 -- 172.21.15.1:0/2315635913 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f468c15a2e0 msgr2=0x7f468c157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:25.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.296+0000 7f468bfff700 1 --2- 172.21.15.1:0/2315635913 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f468c15a2e0 0x7f468c157560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.296+0000 7f468bfff700 1 -- 172.21.15.1:0/2315635913 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f468c0cc170 msgr2=0x7f468c157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:25.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.296+0000 7f468bfff700 1 --2- 172.21.15.1:0/2315635913 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f468c0cc170 0x7f468c157020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.296+0000 7f468bfff700 1 -- 172.21.15.1:0/2315635913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467c0094a0 con 0x7f468c0ae600 2024-03-18T20:23:25.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.297+0000 7f468bfff700 1 --2- 172.21.15.1:0/2315635913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f468c0ae600 0x7f468c14ea30 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f467c0090b0 tx=0x7f467c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:25.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.297+0000 7f4688ff9700 1 -- 172.21.15.1:0/2315635913 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467c004570 con 0x7f468c0ae600 2024-03-18T20:23:25.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.297+0000 7f4688ff9700 1 -- 172.21.15.1:0/2315635913 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467c01f070 con 0x7f468c0ae600 2024-03-18T20:23:25.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.297+0000 7f4692569700 1 -- 172.21.15.1:0/2315635913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f468c155650 con 0x7f468c0ae600 2024-03-18T20:23:25.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.297+0000 7f4688ff9700 1 -- 172.21.15.1:0/2315635913 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467c011870 con 0x7f468c0ae600 2024-03-18T20:23:25.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.297+0000 7f4692569700 1 -- 172.21.15.1:0/2315635913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468c14ef70 con 0x7f468c0ae600 2024-03-18T20:23:25.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.298+0000 7f4692569700 1 -- 172.21.15.1:0/2315635913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f468c15e9f0 con 0x7f468c0ae600 2024-03-18T20:23:25.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.298+0000 7f4688ff9700 1 -- 172.21.15.1:0/2315635913 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 119) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f467c0119d0 con 0x7f468c0ae600 2024-03-18T20:23:25.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.299+0000 7f4688ff9700 1 --2- 172.21.15.1:0/2315635913 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f467806adb0 0x7f467806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:25.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.299+0000 7f468b7fe700 1 -- 172.21.15.1:0/2315635913 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f467806adb0 msgr2=0x7f467806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:25.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.299+0000 7f4688ff9700 1 -- 172.21.15.1:0/2315635913 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6775+0+0 (secure 0 0 0) 0x7f467c021070 con 0x7f468c0ae600 2024-03-18T20:23:25.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.299+0000 7f468b7fe700 1 --2- 172.21.15.1:0/2315635913 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f467806adb0 0x7f467806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:23:25.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.308+0000 7f4688ff9700 1 -- 172.21.15.1:0/2315635913 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f467c055e00 con 0x7f468c0ae600 2024-03-18T20:23:25.492 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:25.487+0000 7f5e66159240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:23:25.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.500+0000 7f468b7fe700 1 -- 172.21.15.1:0/2315635913 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f467806adb0 msgr2=0x7f467806d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:25.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.500+0000 7f468b7fe700 1 --2- 172.21.15.1:0/2315635913 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f467806adb0 0x7f467806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:23:25.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.511+0000 7f4692569700 1 -- 172.21.15.1:0/2315635913 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f468c155b70 con 0x7f468c0ae600 2024-03-18T20:23:25.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.513+0000 7f4688ff9700 1 -- 172.21.15.1:0/2315635913 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v201) v1 ==== 74+0+17301 (secure 0 0 0) 0x7f467c02b090 con 0x7f468c0ae600 2024-03-18T20:23:25.517 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:25.517 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":201,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:23:24.264472+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:07:17.162084+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-18T20:07:18.295080+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"201","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":201,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2032368612":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/181301955":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/1513627829":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/2331829563":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/2988375791":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/666830041":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/2295652306":"2024-03-19T20:07:34.587648+0000","172.21.15.1:6833/76998":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/2850855126":"2024-03-19T20:18:22.841807+0000","172.21.15.1:6832/76998":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/3253696179":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/3201942299":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/2111434827":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/1313400628":"2024-03-19T20:07:34.587648+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":24,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:23:25.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.515+0000 7f4692569700 1 -- 172.21.15.1:0/2315635913 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f467806adb0 msgr2=0x7f467806d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:23:25.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.515+0000 7f4692569700 1 --2- 172.21.15.1:0/2315635913 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f467806adb0 0x7f467806d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.515+0000 7f4692569700 1 -- 172.21.15.1:0/2315635913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f468c0ae600 msgr2=0x7f468c14ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:25.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.515+0000 7f4692569700 1 --2- 172.21.15.1:0/2315635913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f468c0ae600 0x7f468c14ea30 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f467c0090b0 tx=0x7f467c004750 comp rx=0 tx=0).stop 2024-03-18T20:23:25.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.515+0000 7f4692569700 1 -- 172.21.15.1:0/2315635913 shutdown_connections 2024-03-18T20:23:25.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.515+0000 7f4692569700 1 --2- 172.21.15.1:0/2315635913 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f467806adb0 0x7f467806d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.515+0000 7f4692569700 1 --2- 172.21.15.1:0/2315635913 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f468c15a2e0 0x7f468c157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.515+0000 7f4692569700 1 --2- 172.21.15.1:0/2315635913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f468c0ae600 0x7f468c14ea30 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.515+0000 7f4692569700 1 --2- 172.21.15.1:0/2315635913 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f468c0cc170 0x7f468c157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.515+0000 7f4692569700 1 -- 172.21.15.1:0/2315635913 >> 172.21.15.1:0/2315635913 conn(0x7f468c06b860 msgr2=0x7f468c0b0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:25.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.515+0000 7f4692569700 1 -- 172.21.15.1:0/2315635913 shutdown_connections 2024-03-18T20:23:25.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.515+0000 7f4692569700 1 -- 172.21.15.1:0/2315635913 wait complete. 2024-03-18T20:23:25.534 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T20:23:25.534 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:23:25.642 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:25.637+0000 7f4e634a7240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:23:25.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.656+0000 7f6a26b22700 1 Processor -- start 2024-03-18T20:23:25.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.656+0000 7f6a26b22700 1 -- start start 2024-03-18T20:23:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.657+0000 7f6a26b22700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a201614f0 0x7f6a2015d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.657+0000 7f6a26b22700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a200d0330 con 0x7f6a2006d440 2024-03-18T20:23:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.657+0000 7f6a26b22700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a200cf720 con 0x7f6a20161090 2024-03-18T20:23:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.657+0000 7f6a26b22700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a200d07c0 con 0x7f6a201614f0 2024-03-18T20:23:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.657+0000 7f6a248be700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6a2006d440 0x7f6a2006f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42404/0 (socket says 172.21.15.1:42404) 2024-03-18T20:23:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.657+0000 7f6a248be700 1 -- 172.21.15.1:0/3900706930 learned_addr learned my addr 172.21.15.1:0/3900706930 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.657+0000 7f6a250bf700 1 --2- 172.21.15.1:0/3900706930 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a201614f0 0x7f6a2015d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.657+0000 7f6a1f7fe700 1 -- 172.21.15.1:0/3900706930 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93794950 0 0) 0x7f6a200d0330 con 0x7f6a2006d440 2024-03-18T20:23:25.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.657+0000 7f6a1f7fe700 1 -- 172.21.15.1:0/3900706930 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a100035b0 con 0x7f6a2006d440 2024-03-18T20:23:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.657+0000 7f6a1f7fe700 1 -- 172.21.15.1:0/3900706930 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99927000 0 0) 0x7f6a200cf720 con 0x7f6a20161090 2024-03-18T20:23:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.657+0000 7f6a1f7fe700 1 -- 172.21.15.1:0/3900706930 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a200d0330 con 0x7f6a20161090 2024-03-18T20:23:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.657+0000 7f6a250bf700 1 -- 172.21.15.1:0/3900706930 >> v1:172.21.15.165:6789/0 conn(0x7f6a20161090 legacy=0x7f6a2015cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.657+0000 7f6a250bf700 1 -- 172.21.15.1:0/3900706930 >> v1:172.21.15.1:6789/0 conn(0x7f6a2006d440 legacy=0x7f6a2006f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.657+0000 7f6a250bf700 1 -- 172.21.15.1:0/3900706930 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a200cfa00 con 0x7f6a201614f0 2024-03-18T20:23:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.657+0000 7f6a250bf700 1 --2- 172.21.15.1:0/3900706930 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a201614f0 0x7f6a2015d350 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6a18009dc0 tx=0x7f6a1800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e254660b9ca11c1e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.658+0000 7f6a1f7fe700 1 -- 172.21.15.1:0/3900706930 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1800e070 con 0x7f6a201614f0 2024-03-18T20:23:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.658+0000 7f6a1f7fe700 1 -- 172.21.15.1:0/3900706930 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1800bb20 con 0x7f6a201614f0 2024-03-18T20:23:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.658+0000 7f6a1f7fe700 1 -- 172.21.15.1:0/3900706930 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a18012430 con 0x7f6a201614f0 2024-03-18T20:23:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.658+0000 7f6a26b22700 1 -- 172.21.15.1:0/3900706930 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a201614f0 msgr2=0x7f6a2015d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.658+0000 7f6a26b22700 1 --2- 172.21.15.1:0/3900706930 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a201614f0 0x7f6a2015d350 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6a18009dc0 tx=0x7f6a1800b330 comp rx=0 tx=0).stop 2024-03-18T20:23:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.658+0000 7f6a26b22700 1 -- 172.21.15.1:0/3900706930 shutdown_connections 2024-03-18T20:23:25.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.658+0000 7f6a26b22700 1 --2- 172.21.15.1:0/3900706930 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a201614f0 0x7f6a2015d350 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.658+0000 7f6a26b22700 1 -- 172.21.15.1:0/3900706930 >> 172.21.15.1:0/3900706930 conn(0x7f6a200b2ac0 msgr2=0x7f6a200b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:25.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.658+0000 7f6a26b22700 1 -- 172.21.15.1:0/3900706930 shutdown_connections 2024-03-18T20:23:25.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.658+0000 7f6a26b22700 1 -- 172.21.15.1:0/3900706930 wait complete. 2024-03-18T20:23:25.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.660+0000 7f6a26b22700 1 Processor -- start 2024-03-18T20:23:25.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.660+0000 7f6a26b22700 1 -- start start 2024-03-18T20:23:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.660+0000 7f6a26b22700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a2006d440 0x7f6a200ae710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.660+0000 7f6a26b22700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a20161090 0x7f6a2014b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.660+0000 7f6a26b22700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a201614f0 0x7f6a200c5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.661+0000 7f6a26b22700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2014b9c0 con 0x7f6a201614f0 2024-03-18T20:23:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.661+0000 7f6a26b22700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a20155630 con 0x7f6a20161090 2024-03-18T20:23:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.661+0000 7f6a26b22700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a200af780 con 0x7f6a2006d440 2024-03-18T20:23:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.661+0000 7f6a250bf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a201614f0 0x7f6a200c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.661+0000 7f6a250bf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a201614f0 0x7f6a200c5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36874/0 (socket says 172.21.15.1:36874) 2024-03-18T20:23:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.661+0000 7f6a250bf700 1 -- 172.21.15.1:0/161052039 learned_addr learned my addr 172.21.15.1:0/161052039 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.661+0000 7f6a248be700 1 --2- 172.21.15.1:0/161052039 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a2006d440 0x7f6a200ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.661+0000 7f6a1ffff700 1 --2- 172.21.15.1:0/161052039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a20161090 0x7f6a2014b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.661+0000 7f6a250bf700 1 -- 172.21.15.1:0/161052039 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a2006d440 msgr2=0x7f6a200ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:25.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.661+0000 7f6a250bf700 1 --2- 172.21.15.1:0/161052039 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a2006d440 0x7f6a200ae710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.661+0000 7f6a250bf700 1 -- 172.21.15.1:0/161052039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a20161090 msgr2=0x7f6a2014b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.661+0000 7f6a250bf700 1 --2- 172.21.15.1:0/161052039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a20161090 0x7f6a2014b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.661+0000 7f6a250bf700 1 -- 172.21.15.1:0/161052039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a18009a70 con 0x7f6a201614f0 2024-03-18T20:23:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.661+0000 7f6a1ffff700 1 --2- 172.21.15.1:0/161052039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a20161090 0x7f6a2014b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:23:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.661+0000 7f6a250bf700 1 --2- 172.21.15.1:0/161052039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a201614f0 0x7f6a200c5430 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f6a1800b370 tx=0x7f6a18004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.661+0000 7f6a1d7fa700 1 -- 172.21.15.1:0/161052039 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1801b810 con 0x7f6a201614f0 2024-03-18T20:23:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.661+0000 7f6a1d7fa700 1 -- 172.21.15.1:0/161052039 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a18020070 con 0x7f6a201614f0 2024-03-18T20:23:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.661+0000 7f6a26b22700 1 -- 172.21.15.1:0/161052039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a200afa60 con 0x7f6a201614f0 2024-03-18T20:23:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.661+0000 7f6a1d7fa700 1 -- 172.21.15.1:0/161052039 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1801bbd0 con 0x7f6a201614f0 2024-03-18T20:23:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.662+0000 7f6a26b22700 1 -- 172.21.15.1:0/161052039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a200c4f10 con 0x7f6a201614f0 2024-03-18T20:23:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.663+0000 7f6a1d7fa700 1 -- 172.21.15.1:0/161052039 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 119) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f6a1801bd30 con 0x7f6a201614f0 2024-03-18T20:23:25.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.663+0000 7f6a26b22700 1 -- 172.21.15.1:0/161052039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a20073350 con 0x7f6a201614f0 2024-03-18T20:23:25.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.664+0000 7f6a1d7fa700 1 --2- 172.21.15.1:0/161052039 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6a0406adb0 0x7f6a0406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:25.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.664+0000 7f6a248be700 1 -- 172.21.15.1:0/161052039 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6a0406adb0 msgr2=0x7f6a0406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:25.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.664+0000 7f6a1d7fa700 1 -- 172.21.15.1:0/161052039 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6775+0+0 (secure 0 0 0) 0x7f6a18022070 con 0x7f6a201614f0 2024-03-18T20:23:25.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.664+0000 7f6a248be700 1 --2- 172.21.15.1:0/161052039 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6a0406adb0 0x7f6a0406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:23:25.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.668+0000 7f6a1d7fa700 1 -- 172.21.15.1:0/161052039 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6a18056450 con 0x7f6a201614f0 2024-03-18T20:23:25.676 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:25.672+0000 7f5e66159240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:23:25.733 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:25.728+0000 7f4e634a7240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:23:25.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.865+0000 7f6a248be700 1 -- 172.21.15.1:0/161052039 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6a0406adb0 msgr2=0x7f6a0406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:25.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.865+0000 7f6a248be700 1 --2- 172.21.15.1:0/161052039 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6a0406adb0 0x7f6a0406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:23:25.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.917+0000 7f6a26b22700 1 -- 172.21.15.1:0/161052039 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a200b6f50 con 0x7f6a201614f0 2024-03-18T20:23:25.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.918+0000 7f6a1d7fa700 1 -- 172.21.15.1:0/161052039 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4741 (secure 0 0 0) 0x7f6a1805ee60 con 0x7f6a201614f0 2024-03-18T20:23:25.923 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:25.924 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":89,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8204,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2572860438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2572860438},{"type":"v1","addr":"172.21.15.165:6835","nonce":2572860438}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9439,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3158632513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3158632513},{"type":"v1","addr":"172.21.15.1:6836","nonce":3158632513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9448,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/4005747445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4005747445},{"type":"v1","addr":"172.21.15.165:6834","nonce":4005747445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:07:19.322806+0000","modified":"2024-03-18T20:07:22.335335+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10422":{"gid":10422,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1610313086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1610313086},{"type":"v1","addr":"172.21.15.1:6837","nonce":1610313086}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-03-18T20:23:25.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.920+0000 7f6a26b22700 1 -- 172.21.15.1:0/161052039 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6a0406adb0 msgr2=0x7f6a0406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:23:25.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.920+0000 7f6a26b22700 1 --2- 172.21.15.1:0/161052039 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6a0406adb0 0x7f6a0406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.920+0000 7f6a26b22700 1 -- 172.21.15.1:0/161052039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a201614f0 msgr2=0x7f6a200c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:25.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.920+0000 7f6a26b22700 1 --2- 172.21.15.1:0/161052039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a201614f0 0x7f6a200c5430 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f6a1800b370 tx=0x7f6a18004ad0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.920+0000 7f6a26b22700 1 -- 172.21.15.1:0/161052039 shutdown_connections 2024-03-18T20:23:25.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.920+0000 7f6a26b22700 1 --2- 172.21.15.1:0/161052039 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6a0406adb0 0x7f6a0406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.920+0000 7f6a26b22700 1 --2- 172.21.15.1:0/161052039 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a2006d440 0x7f6a200ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.920+0000 7f6a26b22700 1 --2- 172.21.15.1:0/161052039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a201614f0 0x7f6a200c5430 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.920+0000 7f6a26b22700 1 --2- 172.21.15.1:0/161052039 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a20161090 0x7f6a2014b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:25.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.920+0000 7f6a26b22700 1 -- 172.21.15.1:0/161052039 >> 172.21.15.1:0/161052039 conn(0x7f6a200b2ac0 msgr2=0x7f6a200b2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:25.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.920+0000 7f6a26b22700 1 -- 172.21.15.1:0/161052039 shutdown_connections 2024-03-18T20:23:25.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:25.920+0000 7f6a26b22700 1 -- 172.21.15.1:0/161052039 wait complete. 2024-03-18T20:23:25.935 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 89 2024-03-18T20:23:25.938 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:23:26.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.061+0000 7f251a3f2700 1 Processor -- start 2024-03-18T20:23:26.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.061+0000 7f251a3f2700 1 -- start start 2024-03-18T20:23:26.066 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:26.062+0000 7f5e66159240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:23:26.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.062+0000 7f251a3f2700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f251415d6b0 0x7f25140b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:26.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.062+0000 7f251a3f2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25140ad790 0x7f25140b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:26.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.062+0000 7f251a3f2700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25140acfb0 con 0x7f251406a750 2024-03-18T20:23:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.062+0000 7f251a3f2700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25140af7a0 con 0x7f251415d6b0 2024-03-18T20:23:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.062+0000 7f251a3f2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25140b50a0 con 0x7f25140ad790 2024-03-18T20:23:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.062+0000 7f251898f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25140ad790 0x7f25140b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.062+0000 7f25137fe700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f251415d6b0 0x7f25140b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.062+0000 7f2513fff700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f251406a750 0x7f251406ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:52882/0 (socket says 172.21.15.1:52882) 2024-03-18T20:23:26.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.062+0000 7f2513fff700 1 -- 172.21.15.1:0/1901095467 learned_addr learned my addr 172.21.15.1:0/1901095467 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:26.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.063+0000 7f25137fe700 1 -- 172.21.15.1:0/1901095467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25140ad790 msgr2=0x7f25140b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:26.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.063+0000 7f25137fe700 1 --2- 172.21.15.1:0/1901095467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25140ad790 0x7f25140b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.063+0000 7f25137fe700 1 -- 172.21.15.1:0/1901095467 >> v1:172.21.15.165:6789/0 conn(0x7f251406a750 legacy=0x7f251406ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:26.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.063+0000 7f25137fe700 1 -- 172.21.15.1:0/1901095467 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25140af1e0 con 0x7f251415d6b0 2024-03-18T20:23:26.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.063+0000 7f2512ffd700 1 -- 172.21.15.1:0/1901095467 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322333980 0 0) 0x7f25140acfb0 con 0x7f251406a750 2024-03-18T20:23:26.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.063+0000 7f251898f700 1 --2- 172.21.15.1:0/1901095467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25140ad790 0x7f25140b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:23:26.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.063+0000 7f25137fe700 1 --2- 172.21.15.1:0/1901095467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f251415d6b0 0x7f25140b2da0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009840 tx=0x7f24fc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae01a8ada5ce4a54 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.063+0000 7f2512ffd700 1 -- 172.21.15.1:0/1901095467 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc011830 con 0x7f251415d6b0 2024-03-18T20:23:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.063+0000 7f2512ffd700 1 -- 172.21.15.1:0/1901095467 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24fc011990 con 0x7f251415d6b0 2024-03-18T20:23:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.063+0000 7f2512ffd700 1 -- 172.21.15.1:0/1901095467 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc011d20 con 0x7f251415d6b0 2024-03-18T20:23:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.064+0000 7f251a3f2700 1 -- 172.21.15.1:0/1901095467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f251415d6b0 msgr2=0x7f25140b2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.064+0000 7f251a3f2700 1 --2- 172.21.15.1:0/1901095467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f251415d6b0 0x7f25140b2da0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009840 tx=0x7f24fc009b50 comp rx=0 tx=0).stop 2024-03-18T20:23:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.064+0000 7f251a3f2700 1 -- 172.21.15.1:0/1901095467 shutdown_connections 2024-03-18T20:23:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.064+0000 7f251a3f2700 1 --2- 172.21.15.1:0/1901095467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25140ad790 0x7f25140b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.064+0000 7f251a3f2700 1 --2- 172.21.15.1:0/1901095467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f251415d6b0 0x7f25140b2da0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.064+0000 7f251a3f2700 1 -- 172.21.15.1:0/1901095467 >> 172.21.15.1:0/1901095467 conn(0x7f251406b860 msgr2=0x7f251406bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.064+0000 7f251a3f2700 1 -- 172.21.15.1:0/1901095467 shutdown_connections 2024-03-18T20:23:26.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.064+0000 7f251a3f2700 1 -- 172.21.15.1:0/1901095467 wait complete. 2024-03-18T20:23:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.065+0000 7f251a3f2700 1 Processor -- start 2024-03-18T20:23:26.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.066+0000 7f251a3f2700 1 -- start start 2024-03-18T20:23:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f251a3f2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f251406a750 0x7f2514145800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f251a3f2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25140ad790 0x7f25141e8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f251a3f2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f251415d6b0 0x7f25140cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f251a3f2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251415e200 con 0x7f251406a750 2024-03-18T20:23:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f251a3f2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25141e92a0 con 0x7f251415d6b0 2024-03-18T20:23:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f251a3f2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25141e9580 con 0x7f25140ad790 2024-03-18T20:23:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f2513fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f251406a750 0x7f2514145800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f2513fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f251406a750 0x7f2514145800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36884/0 (socket says 172.21.15.1:36884) 2024-03-18T20:23:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f2513fff700 1 -- 172.21.15.1:0/3201856093 learned_addr learned my addr 172.21.15.1:0/3201856093 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f251898f700 1 --2- 172.21.15.1:0/3201856093 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f251415d6b0 0x7f25140cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f25137fe700 1 --2- 172.21.15.1:0/3201856093 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25140ad790 0x7f25141e8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f2513fff700 1 -- 172.21.15.1:0/3201856093 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25140ad790 msgr2=0x7f25141e8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:26.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f2513fff700 1 --2- 172.21.15.1:0/3201856093 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25140ad790 0x7f25141e8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f2513fff700 1 -- 172.21.15.1:0/3201856093 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f251415d6b0 msgr2=0x7f25140cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f2513fff700 1 --2- 172.21.15.1:0/3201856093 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f251415d6b0 0x7f25140cbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f2513fff700 1 -- 172.21.15.1:0/3201856093 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c009a70 con 0x7f251406a750 2024-03-18T20:23:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f2513fff700 1 --2- 172.21.15.1:0/3201856093 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f251406a750 0x7f2514145800 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f25040011e0 tx=0x7f250400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f2510ff9700 1 -- 172.21.15.1:0/3201856093 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504004c80 con 0x7f251406a750 2024-03-18T20:23:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f2510ff9700 1 -- 172.21.15.1:0/3201856093 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2504004de0 con 0x7f251406a750 2024-03-18T20:23:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f2510ff9700 1 -- 172.21.15.1:0/3201856093 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504007310 con 0x7f251406a750 2024-03-18T20:23:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.067+0000 7f251a3f2700 1 -- 172.21.15.1:0/3201856093 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24fc0094a0 con 0x7f251406a750 2024-03-18T20:23:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.068+0000 7f251a3f2700 1 -- 172.21.15.1:0/3201856093 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25141e9860 con 0x7f251406a750 2024-03-18T20:23:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.069+0000 7f251a3f2700 1 -- 172.21.15.1:0/3201856093 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25140ad420 con 0x7f251406a750 2024-03-18T20:23:26.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.070+0000 7f2510ff9700 1 -- 172.21.15.1:0/3201856093 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 119) v1 ==== 89251+0+0 (secure 0 0 0) 0x7f2504007470 con 0x7f251406a750 2024-03-18T20:23:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.070+0000 7f2510ff9700 1 --2- 172.21.15.1:0/3201856093 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f24f406adb0 0x7f24f406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.070+0000 7f25137fe700 1 -- 172.21.15.1:0/3201856093 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f24f406adb0 msgr2=0x7f24f406d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.070+0000 7f2510ff9700 1 -- 172.21.15.1:0/3201856093 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6775+0+0 (secure 0 0 0) 0x7f2504089470 con 0x7f251406a750 2024-03-18T20:23:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.070+0000 7f25137fe700 1 --2- 172.21.15.1:0/3201856093 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f24f406adb0 0x7f24f406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:23:26.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.075+0000 7f2510ff9700 1 -- 172.21.15.1:0/3201856093 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f25140ad420 con 0x7f251406a750 2024-03-18T20:23:26.164 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:26.159+0000 7f5e66159240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:23:26.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.177+0000 7f2510ff9700 1 -- 172.21.15.1:0/3201856093 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 120) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f2504054570 con 0x7f251406a750 2024-03-18T20:23:26.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.178+0000 7f2510ff9700 1 -- 172.21.15.1:0/3201856093 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f24f406adb0 msgr2=0x7f24f406d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:23:26.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.178+0000 7f2510ff9700 1 --2- 172.21.15.1:0/3201856093 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f24f406adb0 0x7f24f406d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.250 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:26.245+0000 7f5dddd80700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:26.250 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:26.245+0000 7f5dddd80700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:26.250 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:26.245+0000 7f5dddd80700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:26.250 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:26.245+0000 7f5dddd80700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:26.250 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:26.245+0000 7f5dddd80700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:26.250 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:26.246+0000 7f5de1587700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:26.251 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:26.246+0000 7f5de1587700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:26.251 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:26.246+0000 7f5de1587700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:26.251 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:26.246+0000 7f5de1587700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:26.251 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:26.246+0000 7f5de1587700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:26.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.347+0000 7f251a3f2700 1 -- 172.21.15.1:0/3201856093 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f251415e6b0 con 0x7f251406a750 2024-03-18T20:23:26.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.347+0000 7f2510ff9700 1 -- 172.21.15.1:0/3201856093 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4741 (secure 0 0 0) 0x7f251415e6b0 con 0x7f251406a750 2024-03-18T20:23:26.352 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:26.352 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":89,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8204,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2572860438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2572860438},{"type":"v1","addr":"172.21.15.165:6835","nonce":2572860438}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9439,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3158632513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3158632513},{"type":"v1","addr":"172.21.15.1:6836","nonce":3158632513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9448,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/4005747445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4005747445},{"type":"v1","addr":"172.21.15.165:6834","nonce":4005747445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:07:19.322806+0000","modified":"2024-03-18T20:07:22.335335+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10422":{"gid":10422,"name":"c","rank":0,"incarnation":85,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/1610313086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1610313086},{"type":"v1","addr":"172.21.15.1:6837","nonce":1610313086}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-03-18T20:23:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.349+0000 7f251a3f2700 1 -- 172.21.15.1:0/3201856093 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f251406a750 msgr2=0x7f2514145800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.349+0000 7f251a3f2700 1 --2- 172.21.15.1:0/3201856093 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f251406a750 0x7f2514145800 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f25040011e0 tx=0x7f250400ca30 comp rx=0 tx=0).stop 2024-03-18T20:23:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.349+0000 7f251a3f2700 1 -- 172.21.15.1:0/3201856093 shutdown_connections 2024-03-18T20:23:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.349+0000 7f251a3f2700 1 --2- 172.21.15.1:0/3201856093 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f24f406adb0 0x7f24f406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.349+0000 7f251a3f2700 1 --2- 172.21.15.1:0/3201856093 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f251415d6b0 0x7f25140cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.349+0000 7f251a3f2700 1 --2- 172.21.15.1:0/3201856093 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f251406a750 0x7f2514145800 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.349+0000 7f251a3f2700 1 --2- 172.21.15.1:0/3201856093 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25140ad790 0x7f25141e8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.349+0000 7f251a3f2700 1 -- 172.21.15.1:0/3201856093 >> 172.21.15.1:0/3201856093 conn(0x7f251406b860 msgr2=0x7f25140b2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:26.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.349+0000 7f251a3f2700 1 -- 172.21.15.1:0/3201856093 shutdown_connections 2024-03-18T20:23:26.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.349+0000 7f251a3f2700 1 -- 172.21.15.1:0/3201856093 wait complete. 2024-03-18T20:23:26.355 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 89 2024-03-18T20:23:26.366 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:23:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.486+0000 7f504fb1c700 1 Processor -- start 2024-03-18T20:23:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.487+0000 7f504fb1c700 1 -- start start 2024-03-18T20:23:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.487+0000 7f504fb1c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f504815afc0 0x7f5048161c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.487+0000 7f504fb1c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f504815da00 0x7f504815ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.487+0000 7f504fb1c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504806d090 con 0x7f504815de60 2024-03-18T20:23:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.487+0000 7f504fb1c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f504806d430 con 0x7f504815da00 2024-03-18T20:23:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.487+0000 7f504fb1c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f504815e310 con 0x7f504815afc0 2024-03-18T20:23:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.488+0000 7f504d0b7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f504815da00 0x7f504815ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.488+0000 7f504d0b7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f504815da00 0x7f504815ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36996/0 (socket says 172.21.15.1:36996) 2024-03-18T20:23:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.488+0000 7f504d0b7700 1 -- 172.21.15.1:0/1678786952 learned_addr learned my addr 172.21.15.1:0/1678786952 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.488+0000 7f504d8b8700 1 --2- 172.21.15.1:0/1678786952 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f504815afc0 0x7f5048161c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.488+0000 7f504d0b7700 1 -- 172.21.15.1:0/1678786952 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f504815afc0 msgr2=0x7f5048161c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.488+0000 7f504d0b7700 1 --2- 172.21.15.1:0/1678786952 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f504815afc0 0x7f5048161c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.488+0000 7f504d0b7700 1 -- 172.21.15.1:0/1678786952 >> v1:172.21.15.165:6789/0 conn(0x7f504815de60 legacy=0x7f5048070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.488+0000 7f504d0b7700 1 -- 172.21.15.1:0/1678786952 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504815e5f0 con 0x7f504815da00 2024-03-18T20:23:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.488+0000 7f504d0b7700 1 --2- 172.21.15.1:0/1678786952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f504815da00 0x7f504815ec10 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f503c0098f0 tx=0x7f503c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=493af5d556d86048 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.488+0000 7f504c8b6700 1 -- 172.21.15.1:0/1678786952 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c011830 con 0x7f504815da00 2024-03-18T20:23:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.488+0000 7f504c8b6700 1 -- 172.21.15.1:0/1678786952 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503c011990 con 0x7f504815da00 2024-03-18T20:23:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.488+0000 7f504c8b6700 1 -- 172.21.15.1:0/1678786952 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c011d20 con 0x7f504815da00 2024-03-18T20:23:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.489+0000 7f504fb1c700 1 -- 172.21.15.1:0/1678786952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f504815da00 msgr2=0x7f504815ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.489+0000 7f504fb1c700 1 --2- 172.21.15.1:0/1678786952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f504815da00 0x7f504815ec10 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f503c0098f0 tx=0x7f503c009c00 comp rx=0 tx=0).stop 2024-03-18T20:23:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.489+0000 7f504fb1c700 1 -- 172.21.15.1:0/1678786952 shutdown_connections 2024-03-18T20:23:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.489+0000 7f504fb1c700 1 --2- 172.21.15.1:0/1678786952 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f504815afc0 0x7f5048161c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.489+0000 7f504fb1c700 1 --2- 172.21.15.1:0/1678786952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f504815da00 0x7f504815ec10 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.489+0000 7f504fb1c700 1 -- 172.21.15.1:0/1678786952 >> 172.21.15.1:0/1678786952 conn(0x7f504806a750 msgr2=0x7f504806ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.489+0000 7f504fb1c700 1 -- 172.21.15.1:0/1678786952 shutdown_connections 2024-03-18T20:23:26.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.489+0000 7f504fb1c700 1 -- 172.21.15.1:0/1678786952 wait complete. 2024-03-18T20:23:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.490+0000 7f504fb1c700 1 Processor -- start 2024-03-18T20:23:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.490+0000 7f504fb1c700 1 -- start start 2024-03-18T20:23:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.491+0000 7f504fb1c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f504815afc0 0x7f5048155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:26.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.491+0000 7f504fb1c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f504815da00 0x7f50481e75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.491+0000 7f504fb1c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f504815de60 0x7f50481e7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.491+0000 7f504fb1c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50480cc350 con 0x7f504815de60 2024-03-18T20:23:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.491+0000 7f504fb1c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50481e8310 con 0x7f504815afc0 2024-03-18T20:23:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.491+0000 7f504fb1c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50481e85f0 con 0x7f504815da00 2024-03-18T20:23:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.491+0000 7f504e0b9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f504815de60 0x7f50481e7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.492+0000 7f504e0b9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f504815de60 0x7f50481e7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37008/0 (socket says 172.21.15.1:37008) 2024-03-18T20:23:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.492+0000 7f504e0b9700 1 -- 172.21.15.1:0/629134455 learned_addr learned my addr 172.21.15.1:0/629134455 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.492+0000 7f504d8b8700 1 --2- 172.21.15.1:0/629134455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f504815afc0 0x7f5048155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.492+0000 7f504d0b7700 1 --2- 172.21.15.1:0/629134455 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f504815da00 0x7f50481e75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.492+0000 7f504e0b9700 1 -- 172.21.15.1:0/629134455 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f504815da00 msgr2=0x7f50481e75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.492+0000 7f504e0b9700 1 --2- 172.21.15.1:0/629134455 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f504815da00 0x7f50481e75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.492+0000 7f504e0b9700 1 -- 172.21.15.1:0/629134455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f504815afc0 msgr2=0x7f5048155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.492+0000 7f504e0b9700 1 --2- 172.21.15.1:0/629134455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f504815afc0 0x7f5048155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.492+0000 7f504e0b9700 1 -- 172.21.15.1:0/629134455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c009550 con 0x7f504815de60 2024-03-18T20:23:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.492+0000 7f504e0b9700 1 --2- 172.21.15.1:0/629134455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f504815de60 0x7f50481e7ae0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f504000cc50 tx=0x7f504000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.492+0000 7f503a7fc700 1 -- 172.21.15.1:0/629134455 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5040003c00 con 0x7f504815de60 2024-03-18T20:23:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.492+0000 7f503a7fc700 1 -- 172.21.15.1:0/629134455 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5040003d60 con 0x7f504815de60 2024-03-18T20:23:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.492+0000 7f504fb1c700 1 -- 172.21.15.1:0/629134455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50481e88d0 con 0x7f504815de60 2024-03-18T20:23:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.492+0000 7f503a7fc700 1 -- 172.21.15.1:0/629134455 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504000aab0 con 0x7f504815de60 2024-03-18T20:23:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.492+0000 7f504fb1c700 1 -- 172.21.15.1:0/629134455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50481e8e90 con 0x7f504815de60 2024-03-18T20:23:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.493+0000 7f504fb1c700 1 -- 172.21.15.1:0/629134455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50481e9020 con 0x7f504815de60 2024-03-18T20:23:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.494+0000 7f503a7fc700 1 -- 172.21.15.1:0/629134455 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 120) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f504000ac10 con 0x7f504815de60 2024-03-18T20:23:26.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.494+0000 7f503a7fc700 1 -- 172.21.15.1:0/629134455 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 7075+0+0 (secure 0 0 0) 0x7f5040089340 con 0x7f504815de60 2024-03-18T20:23:26.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.499+0000 7f503a7fc700 1 -- 172.21.15.1:0/629134455 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5040054360 con 0x7f504815de60 2024-03-18T20:23:26.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.700+0000 7f504fb1c700 1 -- 172.21.15.1:0/629134455 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5048068240 con 0x7f504815de60 2024-03-18T20:23:26.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.702+0000 7f503a7fc700 1 -- 172.21.15.1:0/629134455 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v202) v1 ==== 74+0+17718 (secure 0 0 0) 0x7f5040057980 con 0x7f504815de60 2024-03-18T20:23:26.706 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:26.706 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":202,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:23:26.172681+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:07:17.162084+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"194","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-18T20:07:18.295080+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"201","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":201,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2944340419":"2024-03-19T20:23:26.172646+0000","172.21.15.1:6833/148356":"2024-03-19T20:23:26.172646+0000","172.21.15.1:0/1758629203":"2024-03-19T20:23:26.172646+0000","172.21.15.1:6832/148356":"2024-03-19T20:23:26.172646+0000","172.21.15.1:0/1045548864":"2024-03-19T20:23:26.172646+0000","172.21.15.1:0/2520336386":"2024-03-19T20:23:26.172646+0000","172.21.15.1:0/2032368612":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/181301955":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/1768205806":"2024-03-19T20:23:26.172646+0000","172.21.15.1:0/1513627829":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/2331829563":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/2988375791":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/666830041":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/2295652306":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/3212520246":"2024-03-19T20:23:26.172646+0000","172.21.15.1:6833/76998":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/2850855126":"2024-03-19T20:18:22.841807+0000","172.21.15.1:6832/76998":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/3253696179":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/3201942299":"2024-03-19T20:18:22.841807+0000","172.21.15.1:0/2111434827":"2024-03-19T20:07:34.587648+0000","172.21.15.1:0/1313400628":"2024-03-19T20:07:34.587648+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:23:26.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.704+0000 7f504fb1c700 1 -- 172.21.15.1:0/629134455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f504815de60 msgr2=0x7f50481e7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:26.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.704+0000 7f504fb1c700 1 --2- 172.21.15.1:0/629134455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f504815de60 0x7f50481e7ae0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f504000cc50 tx=0x7f504000cf60 comp rx=0 tx=0).stop 2024-03-18T20:23:26.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.704+0000 7f504fb1c700 1 -- 172.21.15.1:0/629134455 shutdown_connections 2024-03-18T20:23:26.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.704+0000 7f504fb1c700 1 --2- 172.21.15.1:0/629134455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f504815de60 0x7f50481e7ae0 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.704+0000 7f504fb1c700 1 --2- 172.21.15.1:0/629134455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f504815afc0 0x7f5048155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.704+0000 7f504fb1c700 1 --2- 172.21.15.1:0/629134455 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f504815da00 0x7f50481e75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.704+0000 7f504fb1c700 1 -- 172.21.15.1:0/629134455 >> 172.21.15.1:0/629134455 conn(0x7f504806a750 msgr2=0x7f50480ae8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:26.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.704+0000 7f504fb1c700 1 -- 172.21.15.1:0/629134455 shutdown_connections 2024-03-18T20:23:26.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.704+0000 7f504fb1c700 1 -- 172.21.15.1:0/629134455 wait complete. 2024-03-18T20:23:26.721 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T20:23:26.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.842+0000 7ff4c58ef700 1 Processor -- start 2024-03-18T20:23:26.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.842+0000 7ff4c58ef700 1 -- start start 2024-03-18T20:23:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.843+0000 7ff4c58ef700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff4c00cfb50 0x7ff4c00d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.843+0000 7ff4c58ef700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff4c00cff30 0x7ff4c00b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.843+0000 7ff4c58ef700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c00b3370 con 0x7ff4c00b3d40 2024-03-18T20:23:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.843+0000 7ff4c58ef700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c00b2760 con 0x7ff4c00cfb50 2024-03-18T20:23:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.843+0000 7ff4c58ef700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c00b2a40 con 0x7ff4c00cff30 2024-03-18T20:23:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.843+0000 7ff4beffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff4c00cfb50 0x7ff4c00d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.843+0000 7ff4beffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff4c00cfb50 0x7ff4c00d3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37012/0 (socket says 172.21.15.1:37012) 2024-03-18T20:23:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.843+0000 7ff4beffd700 1 -- 172.21.15.1:0/1496281779 learned_addr learned my addr 172.21.15.1:0/1496281779 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.843+0000 7ff4be7fc700 1 --2- 172.21.15.1:0/1496281779 >> v2:172.21.15.165:3300/0 conn(0x7ff4c00cff30 0x7ff4c00b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.843+0000 7ff4beffd700 1 -- 172.21.15.1:0/1496281779 >> v2:172.21.15.165:3300/0 conn(0x7ff4c00cff30 msgr2=0x7ff4c00b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.843+0000 7ff4beffd700 1 --2- 172.21.15.1:0/1496281779 >> v2:172.21.15.165:3300/0 conn(0x7ff4c00cff30 0x7ff4c00b3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.843+0000 7ff4beffd700 1 -- 172.21.15.1:0/1496281779 >> v1:172.21.15.165:6789/0 conn(0x7ff4c00b3d40 legacy=0x7ff4c00b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.843+0000 7ff4beffd700 1 -- 172.21.15.1:0/1496281779 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c00b2d20 con 0x7ff4c00cfb50 2024-03-18T20:23:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.843+0000 7ff4beffd700 1 --2- 172.21.15.1:0/1496281779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c00cfb50 0x7ff4c00d3c60 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7ff4c008d8b0 tx=0x7ff4b4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6693a825b5bb7ccd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.844+0000 7ff4bdffb700 1 -- 172.21.15.1:0/1496281779 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4005e30 con 0x7ff4c00cfb50 2024-03-18T20:23:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.844+0000 7ff4bdffb700 1 -- 172.21.15.1:0/1496281779 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b4004030 con 0x7ff4c00cfb50 2024-03-18T20:23:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.844+0000 7ff4bdffb700 1 -- 172.21.15.1:0/1496281779 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b40043e0 con 0x7ff4c00cfb50 2024-03-18T20:23:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.844+0000 7ff4c58ef700 1 -- 172.21.15.1:0/1496281779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c00cfb50 msgr2=0x7ff4c00d3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.844+0000 7ff4c58ef700 1 --2- 172.21.15.1:0/1496281779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c00cfb50 0x7ff4c00d3c60 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7ff4c008d8b0 tx=0x7ff4b4009bd0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.844+0000 7ff4c58ef700 1 -- 172.21.15.1:0/1496281779 shutdown_connections 2024-03-18T20:23:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.844+0000 7ff4c58ef700 1 --2- 172.21.15.1:0/1496281779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c00cfb50 0x7ff4c00d3c60 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.844+0000 7ff4c58ef700 1 --2- 172.21.15.1:0/1496281779 >> v2:172.21.15.165:3300/0 conn(0x7ff4c00cff30 0x7ff4c00b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.844+0000 7ff4c58ef700 1 -- 172.21.15.1:0/1496281779 >> 172.21.15.1:0/1496281779 conn(0x7ff4c00d3140 msgr2=0x7ff4c00d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:26.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.844+0000 7ff4c58ef700 1 -- 172.21.15.1:0/1496281779 shutdown_connections 2024-03-18T20:23:26.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.844+0000 7ff4c58ef700 1 -- 172.21.15.1:0/1496281779 wait complete. 2024-03-18T20:23:26.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.845+0000 7ff4c58ef700 1 Processor -- start 2024-03-18T20:23:26.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.846+0000 7ff4c58ef700 1 -- start start 2024-03-18T20:23:26.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.846+0000 7ff4c58ef700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4c00b3d40 0x7ff4c0158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:26.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.846+0000 7ff4c58ef700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c00cfb50 0x7ff4c00c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:26.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.846+0000 7ff4c58ef700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4c00cff30 0x7ff4c0157a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:26.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.846+0000 7ff4c58ef700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c00c7b90 con 0x7ff4c00cfb50 2024-03-18T20:23:26.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.846+0000 7ff4c58ef700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c00c9320 con 0x7ff4c00cff30 2024-03-18T20:23:26.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.846+0000 7ff4c58ef700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c00c9600 con 0x7ff4c00b3d40 2024-03-18T20:23:26.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.846+0000 7ff4be7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c00cfb50 0x7ff4c00c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:26.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.846+0000 7ff4be7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c00cfb50 0x7ff4c00c8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37018/0 (socket says 172.21.15.1:37018) 2024-03-18T20:23:26.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.846+0000 7ff4be7fc700 1 -- 172.21.15.1:0/2970311162 learned_addr learned my addr 172.21.15.1:0/2970311162 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:26.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.847+0000 7ff4beffd700 1 --2- 172.21.15.1:0/2970311162 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4c00b3d40 0x7ff4c0158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:26.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.847+0000 7ff4bf7fe700 1 --2- 172.21.15.1:0/2970311162 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4c00cff30 0x7ff4c0157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:26.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.847+0000 7ff4be7fc700 1 -- 172.21.15.1:0/2970311162 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4c00b3d40 msgr2=0x7ff4c0158250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:26.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.847+0000 7ff4be7fc700 1 --2- 172.21.15.1:0/2970311162 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4c00b3d40 0x7ff4c0158250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.847+0000 7ff4be7fc700 1 -- 172.21.15.1:0/2970311162 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4c00cff30 msgr2=0x7ff4c0157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:26.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.847+0000 7ff4be7fc700 1 --2- 172.21.15.1:0/2970311162 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4c00cff30 0x7ff4c0157a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:26.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.847+0000 7ff4be7fc700 1 -- 172.21.15.1:0/2970311162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b4009500 con 0x7ff4c00cfb50 2024-03-18T20:23:26.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.847+0000 7ff4be7fc700 1 --2- 172.21.15.1:0/2970311162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c00cfb50 0x7ff4c00c8c90 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7ff4b000e870 tx=0x7ff4b000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:26.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.847+0000 7ff4a7fff700 1 -- 172.21.15.1:0/2970311162 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b0009590 con 0x7ff4c00cfb50 2024-03-18T20:23:26.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.847+0000 7ff4a7fff700 1 -- 172.21.15.1:0/2970311162 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b0013680 con 0x7ff4c00cfb50 2024-03-18T20:23:26.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.847+0000 7ff4c58ef700 1 -- 172.21.15.1:0/2970311162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c00c7e30 con 0x7ff4c00cfb50 2024-03-18T20:23:26.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.847+0000 7ff4a7fff700 1 -- 172.21.15.1:0/2970311162 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b00139f0 con 0x7ff4c00cfb50 2024-03-18T20:23:26.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.847+0000 7ff4c58ef700 1 -- 172.21.15.1:0/2970311162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c00c98e0 con 0x7ff4c00cfb50 2024-03-18T20:23:26.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.848+0000 7ff4c58ef700 1 -- 172.21.15.1:0/2970311162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c0073350 con 0x7ff4c00cfb50 2024-03-18T20:23:26.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.848+0000 7ff4a7fff700 1 -- 172.21.15.1:0/2970311162 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 120) v1 ==== 88848+0+0 (secure 0 0 0) 0x7ff4b0013b50 con 0x7ff4c00cfb50 2024-03-18T20:23:26.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.849+0000 7ff4a7fff700 1 -- 172.21.15.1:0/2970311162 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 7075+0+0 (secure 0 0 0) 0x7ff4b0089120 con 0x7ff4c00cfb50 2024-03-18T20:23:26.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:26.853+0000 7ff4a7fff700 1 -- 172.21.15.1:0/2970311162 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff4b00541f0 con 0x7ff4c00cfb50 2024-03-18T20:23:27.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.107+0000 7ff4c58ef700 1 -- 172.21.15.1:0/2970311162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff4c0067b30 con 0x7ff4c00cfb50 2024-03-18T20:23:27.112 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-18T20:23:27.108+0000 7fbdbeb46700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T20:23:27.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.114+0000 7ff4a7fff700 1 -- 172.21.15.1:0/2970311162 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v90) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff4b0057810 con 0x7ff4c00cfb50 2024-03-18T20:23:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.116+0000 7ff4c58ef700 1 -- 172.21.15.1:0/2970311162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c00cfb50 msgr2=0x7ff4c00c8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.116+0000 7ff4c58ef700 1 --2- 172.21.15.1:0/2970311162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c00cfb50 0x7ff4c00c8c90 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7ff4b000e870 tx=0x7ff4b000eb80 comp rx=0 tx=0).stop 2024-03-18T20:23:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.116+0000 7ff4c58ef700 1 -- 172.21.15.1:0/2970311162 shutdown_connections 2024-03-18T20:23:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.116+0000 7ff4c58ef700 1 --2- 172.21.15.1:0/2970311162 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff4c00b3d40 0x7ff4c0158250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.116+0000 7ff4c58ef700 1 --2- 172.21.15.1:0/2970311162 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4c00cff30 0x7ff4c0157a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:27.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.116+0000 7ff4c58ef700 1 --2- 172.21.15.1:0/2970311162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4c00cfb50 0x7ff4c00c8c90 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:27.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.116+0000 7ff4c58ef700 1 -- 172.21.15.1:0/2970311162 >> 172.21.15.1:0/2970311162 conn(0x7ff4c00d3140 msgr2=0x7ff4c015e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:27.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.117+0000 7ff4c58ef700 1 -- 172.21.15.1:0/2970311162 shutdown_connections 2024-03-18T20:23:27.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.117+0000 7ff4c58ef700 1 -- 172.21.15.1:0/2970311162 wait complete. 2024-03-18T20:23:27.122 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T20:23:27.134 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T20:23:27.159 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T20:23:27.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.263+0000 7f97290df700 1 Processor -- start 2024-03-18T20:23:27.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.263+0000 7f97290df700 1 -- start start 2024-03-18T20:23:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.263+0000 7f97290df700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f972415d780 0x7f97240cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.263+0000 7f97290df700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f97240ce5a0 0x7f97240cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.263+0000 7f97290df700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97240ce090 con 0x7f972409dd40 2024-03-18T20:23:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.263+0000 7f97290df700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97240cb480 con 0x7f97240ce5a0 2024-03-18T20:23:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.263+0000 7f97290df700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f972406db30 con 0x7f972415d780 2024-03-18T20:23:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.264+0000 7f9722d9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f972415d780 0x7f97240cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.264+0000 7f972359e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f97240ce5a0 0x7f97240cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.264+0000 7f9722d9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f972415d780 0x7f97240cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:51976/0 (socket says 172.21.15.1:51976) 2024-03-18T20:23:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.264+0000 7f972259c700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f972409dd40 0x7f97240cad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:52910/0 (socket says 172.21.15.1:52910) 2024-03-18T20:23:27.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.264+0000 7f9722d9d700 1 -- 172.21.15.1:0/1868997430 learned_addr learned my addr 172.21.15.1:0/1868997430 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:27.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.264+0000 7f9722d9d700 1 -- 172.21.15.1:0/1868997430 >> v2:172.21.15.165:3300/0 conn(0x7f97240ce5a0 msgr2=0x7f97240cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:27.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.264+0000 7f9722d9d700 1 --2- 172.21.15.1:0/1868997430 >> v2:172.21.15.165:3300/0 conn(0x7f97240ce5a0 0x7f97240cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:27.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.264+0000 7f9722d9d700 1 -- 172.21.15.1:0/1868997430 >> v1:172.21.15.165:6789/0 conn(0x7f972409dd40 legacy=0x7f97240cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.264+0000 7f9722d9d700 1 -- 172.21.15.1:0/1868997430 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972406d520 con 0x7f972415d780 2024-03-18T20:23:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.264+0000 7f9721d9b700 1 -- 172.21.15.1:0/1868997430 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460310812 0 0) 0x7f97240ce090 con 0x7f972409dd40 2024-03-18T20:23:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.264+0000 7f972359e700 1 --2- 172.21.15.1:0/1868997430 >> v2:172.21.15.165:3300/0 conn(0x7f97240ce5a0 0x7f97240cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:23:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.264+0000 7f9722d9d700 1 --2- 172.21.15.1:0/1868997430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f972415d780 0x7f97240cdb50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f971800bb90 tx=0x7f971800bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70b2518253431311 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.265+0000 7f9721d9b700 1 -- 172.21.15.1:0/1868997430 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971800dd30 con 0x7f972415d780 2024-03-18T20:23:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.265+0000 7f9721d9b700 1 -- 172.21.15.1:0/1868997430 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f971800de90 con 0x7f972415d780 2024-03-18T20:23:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.265+0000 7f9721d9b700 1 -- 172.21.15.1:0/1868997430 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9718010690 con 0x7f972415d780 2024-03-18T20:23:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.265+0000 7f97290df700 1 -- 172.21.15.1:0/1868997430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f972415d780 msgr2=0x7f97240cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.265+0000 7f97290df700 1 --2- 172.21.15.1:0/1868997430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f972415d780 0x7f97240cdb50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f971800bb90 tx=0x7f971800bea0 comp rx=0 tx=0).stop 2024-03-18T20:23:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.265+0000 7f97290df700 1 -- 172.21.15.1:0/1868997430 shutdown_connections 2024-03-18T20:23:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.265+0000 7f97290df700 1 --2- 172.21.15.1:0/1868997430 >> v2:172.21.15.165:3300/0 conn(0x7f97240ce5a0 0x7f97240cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:27.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.265+0000 7f97290df700 1 --2- 172.21.15.1:0/1868997430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f972415d780 0x7f97240cdb50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:27.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.265+0000 7f97290df700 1 -- 172.21.15.1:0/1868997430 >> 172.21.15.1:0/1868997430 conn(0x7f97240cca00 msgr2=0x7f97240cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:27.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.265+0000 7f97290df700 1 -- 172.21.15.1:0/1868997430 shutdown_connections 2024-03-18T20:23:27.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.265+0000 7f97290df700 1 -- 172.21.15.1:0/1868997430 wait complete. 2024-03-18T20:23:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.267+0000 7f97290df700 1 Processor -- start 2024-03-18T20:23:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.267+0000 7f97290df700 1 -- start start 2024-03-18T20:23:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.267+0000 7f97290df700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f972409dd40 0x7f97240c7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.267+0000 7f97290df700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97240ce5a0 0x7f972414e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.267+0000 7f97290df700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f972415d780 0x7f972415b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.267+0000 7f97290df700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97240c4ac0 con 0x7f97240ce5a0 2024-03-18T20:23:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.267+0000 7f97290df700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97240c7590 con 0x7f972415d780 2024-03-18T20:23:27.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.267+0000 7f97290df700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97240ca210 con 0x7f972409dd40 2024-03-18T20:23:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.268+0000 7f972259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97240ce5a0 0x7f972414e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.268+0000 7f972259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97240ce5a0 0x7f972414e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37050/0 (socket says 172.21.15.1:37050) 2024-03-18T20:23:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.268+0000 7f972259c700 1 -- 172.21.15.1:0/1116767881 learned_addr learned my addr 172.21.15.1:0/1116767881 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.268+0000 7f9722d9d700 1 --2- 172.21.15.1:0/1116767881 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f972409dd40 0x7f97240c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.268+0000 7f972359e700 1 --2- 172.21.15.1:0/1116767881 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f972415d780 0x7f972415b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.268+0000 7f972259c700 1 -- 172.21.15.1:0/1116767881 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f972409dd40 msgr2=0x7f97240c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.268+0000 7f972259c700 1 --2- 172.21.15.1:0/1116767881 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f972409dd40 0x7f97240c7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.268+0000 7f972259c700 1 -- 172.21.15.1:0/1116767881 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f972415d780 msgr2=0x7f972415b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.268+0000 7f972259c700 1 --2- 172.21.15.1:0/1116767881 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f972415d780 0x7f972415b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.268+0000 7f972259c700 1 -- 172.21.15.1:0/1116767881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971c009a70 con 0x7f97240ce5a0 2024-03-18T20:23:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.268+0000 7f972259c700 1 --2- 172.21.15.1:0/1116767881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97240ce5a0 0x7f972414e390 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f971400c550 tx=0x7f971400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.268+0000 7f970b7fe700 1 -- 172.21.15.1:0/1116767881 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9714004c80 con 0x7f97240ce5a0 2024-03-18T20:23:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.268+0000 7f970b7fe700 1 -- 172.21.15.1:0/1116767881 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f971400ce90 con 0x7f97240ce5a0 2024-03-18T20:23:27.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.268+0000 7f970b7fe700 1 -- 172.21.15.1:0/1116767881 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97140070d0 con 0x7f97240ce5a0 2024-03-18T20:23:27.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.268+0000 7f97290df700 1 -- 172.21.15.1:0/1116767881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971800b7f0 con 0x7f97240ce5a0 2024-03-18T20:23:27.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.268+0000 7f97290df700 1 -- 172.21.15.1:0/1116767881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97240c3f40 con 0x7f97240ce5a0 2024-03-18T20:23:27.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.269+0000 7f97290df700 1 -- 172.21.15.1:0/1116767881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9724070e00 con 0x7f97240ce5a0 2024-03-18T20:23:27.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.270+0000 7f970b7fe700 1 -- 172.21.15.1:0/1116767881 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 121) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f9714007230 con 0x7f97240ce5a0 2024-03-18T20:23:27.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.270+0000 7f970b7fe700 1 --2- 172.21.15.1:0/1116767881 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f971006aae0 0x7f971006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:27.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.270+0000 7f970b7fe700 1 -- 172.21.15.1:0/1116767881 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 7161+0+0 (secure 0 0 0) 0x7f97140890a0 con 0x7f97240ce5a0 2024-03-18T20:23:27.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.270+0000 7f9722d9d700 1 --2- 172.21.15.1:0/1116767881 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f971006aae0 0x7f971006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:27.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.271+0000 7f9722d9d700 1 --2- 172.21.15.1:0/1116767881 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f971006aae0 0x7f971006cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f971800bee0 tx=0x7f9718008040 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:27.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.275+0000 7f970b7fe700 1 -- 172.21.15.1:0/1116767881 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9714054010 con 0x7f97240ce5a0 2024-03-18T20:23:27.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:27.525+0000 7f97290df700 1 -- 172.21.15.1:0/1116767881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f97240c4f70 con 0x7f97240ce5a0 2024-03-18T20:23:28.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.116+0000 7f970b7fe700 1 -- 172.21.15.1:0/1116767881 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v91) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9714057630 con 0x7f97240ce5a0 2024-03-18T20:23:28.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.119+0000 7f97290df700 1 -- 172.21.15.1:0/1116767881 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f971006aae0 msgr2=0x7f971006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:28.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.119+0000 7f97290df700 1 --2- 172.21.15.1:0/1116767881 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f971006aae0 0x7f971006cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f971800bee0 tx=0x7f9718008040 comp rx=0 tx=0).stop 2024-03-18T20:23:28.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.119+0000 7f97290df700 1 -- 172.21.15.1:0/1116767881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97240ce5a0 msgr2=0x7f972414e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:28.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.119+0000 7f97290df700 1 --2- 172.21.15.1:0/1116767881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97240ce5a0 0x7f972414e390 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f971400c550 tx=0x7f971400c860 comp rx=0 tx=0).stop 2024-03-18T20:23:28.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.119+0000 7f97290df700 1 -- 172.21.15.1:0/1116767881 shutdown_connections 2024-03-18T20:23:28.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.119+0000 7f97290df700 1 --2- 172.21.15.1:0/1116767881 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f971006aae0 0x7f971006cfa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:28.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.119+0000 7f97290df700 1 --2- 172.21.15.1:0/1116767881 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f972409dd40 0x7f97240c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:28.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.119+0000 7f97290df700 1 --2- 172.21.15.1:0/1116767881 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f972415d780 0x7f972415b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:28.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.119+0000 7f97290df700 1 --2- 172.21.15.1:0/1116767881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97240ce5a0 0x7f972414e390 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:28.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.119+0000 7f97290df700 1 -- 172.21.15.1:0/1116767881 >> 172.21.15.1:0/1116767881 conn(0x7f97240cca00 msgr2=0x7f9724159980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:28.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.119+0000 7f97290df700 1 -- 172.21.15.1:0/1116767881 shutdown_connections 2024-03-18T20:23:28.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.119+0000 7f97290df700 1 -- 172.21.15.1:0/1116767881 wait complete. 2024-03-18T20:23:28.139 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T20:23:28.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.322+0000 7f8785c63700 1 Processor -- start 2024-03-18T20:23:28.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.322+0000 7f8785c63700 1 -- start start 2024-03-18T20:23:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f8785c63700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87800cd460 0x7f87800723f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f8785c63700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f878015b940 0x7f878015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f8785c63700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87800d0ee0 con 0x7f8780144b30 2024-03-18T20:23:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f8785c63700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f878015c8b0 con 0x7f878015b940 2024-03-18T20:23:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f8785c63700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f878015c5d0 con 0x7f87800cd460 2024-03-18T20:23:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f877f7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8780144b30 0x7f87800ccd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42416/0 (socket says 172.21.15.1:42416) 2024-03-18T20:23:28.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f877f7fe700 1 -- 172.21.15.1:0/1173314037 learned_addr learned my addr 172.21.15.1:0/1173314037 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f877e7fc700 1 -- 172.21.15.1:0/1173314037 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904718916 0 0) 0x7f87800d0ee0 con 0x7f8780144b30 2024-03-18T20:23:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f877effd700 1 --2- 172.21.15.1:0/1173314037 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87800cd460 0x7f87800723f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f877ffff700 1 --2- 172.21.15.1:0/1173314037 >> v2:172.21.15.165:3300/0 conn(0x7f878015b940 0x7f878015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f877e7fc700 1 -- 172.21.15.1:0/1173314037 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87680035b0 con 0x7f8780144b30 2024-03-18T20:23:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f877e7fc700 1 -- 172.21.15.1:0/1173314037 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3224888392 0 0) 0x7f87680035b0 con 0x7f8780144b30 2024-03-18T20:23:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f877e7fc700 1 -- 172.21.15.1:0/1173314037 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87800cd460 msgr2=0x7f87800723f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f877e7fc700 1 --2- 172.21.15.1:0/1173314037 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87800cd460 0x7f87800723f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f877e7fc700 1 -- 172.21.15.1:0/1173314037 >> v2:172.21.15.165:3300/0 conn(0x7f878015b940 msgr2=0x7f878015bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f877e7fc700 1 --2- 172.21.15.1:0/1173314037 >> v2:172.21.15.165:3300/0 conn(0x7f878015b940 0x7f878015bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f877e7fc700 1 -- 172.21.15.1:0/1173314037 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8780071910 con 0x7f8780144b30 2024-03-18T20:23:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f877e7fc700 1 -- 172.21.15.1:0/1173314037 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f8774001670 con 0x7f8780144b30 2024-03-18T20:23:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f877e7fc700 1 -- 172.21.15.1:0/1173314037 >> v1:172.21.15.1:6789/0 conn(0x7f8780144b30 legacy=0x7f87800ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f877e7fc700 1 --2- 172.21.15.1:0/1173314037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8768003a00 0x7f8768003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:28.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f877e7fc700 1 --2- 172.21.15.1:0/1173314037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8768004850 0x7f8768004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.323+0000 7f877e7fc700 1 --2- 172.21.15.1:0/1173314037 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87800cd460 0x7f8768005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.324+0000 7f877e7fc700 1 -- 172.21.15.1:0/1173314037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878015c8b0 con 0x7f8768003a00 2024-03-18T20:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.324+0000 7f877e7fc700 1 -- 172.21.15.1:0/1173314037 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878015c5d0 con 0x7f8768004850 2024-03-18T20:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.324+0000 7f877e7fc700 1 -- 172.21.15.1:0/1173314037 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f876800ba40 con 0x7f87800cd460 2024-03-18T20:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.324+0000 7f877f7fe700 1 --2- 172.21.15.1:0/1173314037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8768003a00 0x7f8768003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.324+0000 7f877effd700 1 --2- 172.21.15.1:0/1173314037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8768004850 0x7f8768004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.324+0000 7f877ffff700 1 --2- 172.21.15.1:0/1173314037 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87800cd460 0x7f8768005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.324+0000 7f877f7fe700 1 -- 172.21.15.1:0/1173314037 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87800cd460 msgr2=0x7f8768005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.324+0000 7f877f7fe700 1 --2- 172.21.15.1:0/1173314037 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87800cd460 0x7f8768005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.324+0000 7f877f7fe700 1 -- 172.21.15.1:0/1173314037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8768004850 msgr2=0x7f8768004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.324+0000 7f877f7fe700 1 --2- 172.21.15.1:0/1173314037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8768004850 0x7f8768004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.324+0000 7f877f7fe700 1 -- 172.21.15.1:0/1173314037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87680035b0 con 0x7f8768003a00 2024-03-18T20:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.324+0000 7f877ffff700 1 --2- 172.21.15.1:0/1173314037 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87800cd460 0x7f8768005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:23:28.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.324+0000 7f877f7fe700 1 --2- 172.21.15.1:0/1173314037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8768003a00 0x7f8768003e20 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f877400c370 tx=0x7f8768005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bca3e456655eb86 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.324+0000 7f877e7fc700 1 -- 172.21.15.1:0/1173314037 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8774007c60 con 0x7f8768003a00 2024-03-18T20:23:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.324+0000 7f877e7fc700 1 -- 172.21.15.1:0/1173314037 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877400ce50 con 0x7f8768003a00 2024-03-18T20:23:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.324+0000 7f877e7fc700 1 -- 172.21.15.1:0/1173314037 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8774005ba0 con 0x7f8768003a00 2024-03-18T20:23:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.325+0000 7f8785c63700 1 -- 172.21.15.1:0/1173314037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8768003a00 msgr2=0x7f8768003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.325+0000 7f8785c63700 1 --2- 172.21.15.1:0/1173314037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8768003a00 0x7f8768003e20 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f877400c370 tx=0x7f8768005880 comp rx=0 tx=0).stop 2024-03-18T20:23:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.325+0000 7f877ffff700 1 -- 172.21.15.1:0/1173314037 reap_dead start 2024-03-18T20:23:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.325+0000 7f8785c63700 1 -- 172.21.15.1:0/1173314037 shutdown_connections 2024-03-18T20:23:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.325+0000 7f8785c63700 1 -- 172.21.15.1:0/1173314037 >> 172.21.15.1:0/1173314037 conn(0x7f87800d00c0 msgr2=0x7f87800d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.325+0000 7f8785c63700 1 -- 172.21.15.1:0/1173314037 shutdown_connections 2024-03-18T20:23:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.325+0000 7f8785c63700 1 -- 172.21.15.1:0/1173314037 wait complete. 2024-03-18T20:23:28.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.326+0000 7f8785c63700 1 Processor -- start 2024-03-18T20:23:28.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.326+0000 7f8785c63700 1 -- start start 2024-03-18T20:23:28.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.327+0000 7f8785c63700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8768003a00 0x7f878013f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.327+0000 7f8785c63700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87800b4270 0x7f87800b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.327+0000 7f8785c63700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87801e48d0 0x7f87801e4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.327+0000 7f8785c63700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87800c9430 con 0x7f87801e48d0 2024-03-18T20:23:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.327+0000 7f8785c63700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87800b4d30 con 0x7f8768003a00 2024-03-18T20:23:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.327+0000 7f8785c63700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87800b5010 con 0x7f87800b4270 2024-03-18T20:23:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.327+0000 7f877ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87801e48d0 0x7f87801e4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.327+0000 7f877ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87801e48d0 0x7f87801e4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37082/0 (socket says 172.21.15.1:37082) 2024-03-18T20:23:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.327+0000 7f877ffff700 1 -- 172.21.15.1:0/1399475574 learned_addr learned my addr 172.21.15.1:0/1399475574 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.327+0000 7f877effd700 1 --2- 172.21.15.1:0/1399475574 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87800b4270 0x7f87800b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.327+0000 7f877f7fe700 1 --2- 172.21.15.1:0/1399475574 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8768003a00 0x7f878013f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.327+0000 7f877ffff700 1 -- 172.21.15.1:0/1399475574 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87800b4270 msgr2=0x7f87800b46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.327+0000 7f877ffff700 1 --2- 172.21.15.1:0/1399475574 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87800b4270 0x7f87800b46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:28.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.327+0000 7f877ffff700 1 -- 172.21.15.1:0/1399475574 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8768003a00 msgr2=0x7f878013f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.327+0000 7f877ffff700 1 --2- 172.21.15.1:0/1399475574 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8768003a00 0x7f878013f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.327+0000 7f877ffff700 1 -- 172.21.15.1:0/1399475574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87800c9640 con 0x7f87801e48d0 2024-03-18T20:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.327+0000 7f877effd700 1 --2- 172.21.15.1:0/1399475574 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87800b4270 0x7f87800b46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.327+0000 7f877ffff700 1 --2- 172.21.15.1:0/1399475574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87801e48d0 0x7f87801e4d60 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f8780073ab0 tx=0x7f87780104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.328+0000 7f8767fff700 1 -- 172.21.15.1:0/1399475574 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877800e050 con 0x7f87801e48d0 2024-03-18T20:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.328+0000 7f8767fff700 1 -- 172.21.15.1:0/1399475574 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877800f040 con 0x7f87801e48d0 2024-03-18T20:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.328+0000 7f8785c63700 1 -- 172.21.15.1:0/1399475574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87800c9920 con 0x7f87801e48d0 2024-03-18T20:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.328+0000 7f8767fff700 1 -- 172.21.15.1:0/1399475574 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8778003a20 con 0x7f87801e48d0 2024-03-18T20:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.328+0000 7f8785c63700 1 -- 172.21.15.1:0/1399475574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87800c9e40 con 0x7f87801e48d0 2024-03-18T20:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.329+0000 7f8785c63700 1 -- 172.21.15.1:0/1399475574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87800c9b40 con 0x7f87801e48d0 2024-03-18T20:23:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.329+0000 7f8767fff700 1 -- 172.21.15.1:0/1399475574 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 121) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f8778003b80 con 0x7f87801e48d0 2024-03-18T20:23:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.330+0000 7f8767fff700 1 --2- 172.21.15.1:0/1399475574 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f876c06aae0 0x7f876c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.330+0000 7f8767fff700 1 -- 172.21.15.1:0/1399475574 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 7161+0+0 (secure 0 0 0) 0x7f8778089930 con 0x7f87801e48d0 2024-03-18T20:23:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.330+0000 7f877f7fe700 1 --2- 172.21.15.1:0/1399475574 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f876c06aae0 0x7f876c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.330+0000 7f877f7fe700 1 --2- 172.21.15.1:0/1399475574 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f876c06aae0 0x7f876c06cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8774015040 tx=0x7f877400a000 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.334+0000 7f8767fff700 1 -- 172.21.15.1:0/1399475574 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f87780548a0 con 0x7f87801e48d0 2024-03-18T20:23:28.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:28.535+0000 7f8785c63700 1 -- 172.21.15.1:0/1399475574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f87800b51d0 con 0x7f87801e48d0 2024-03-18T20:23:29.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.194+0000 7f8767fff700 1 -- 172.21.15.1:0/1399475574 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 122) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f8778054c70 con 0x7f87801e48d0 2024-03-18T20:23:29.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.279+0000 7f8767fff700 1 -- 172.21.15.1:0/1399475574 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v204) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8778070340 con 0x7f87801e48d0 2024-03-18T20:23:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.286+0000 7f8785c63700 1 -- 172.21.15.1:0/1399475574 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f876c06aae0 msgr2=0x7f876c06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.286+0000 7f8785c63700 1 --2- 172.21.15.1:0/1399475574 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f876c06aae0 0x7f876c06cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8774015040 tx=0x7f877400a000 comp rx=0 tx=0).stop 2024-03-18T20:23:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.286+0000 7f8785c63700 1 -- 172.21.15.1:0/1399475574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87801e48d0 msgr2=0x7f87801e4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.286+0000 7f8785c63700 1 --2- 172.21.15.1:0/1399475574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87801e48d0 0x7f87801e4d60 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f8780073ab0 tx=0x7f87780104a0 comp rx=0 tx=0).stop 2024-03-18T20:23:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.286+0000 7f8785c63700 1 -- 172.21.15.1:0/1399475574 shutdown_connections 2024-03-18T20:23:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.286+0000 7f8785c63700 1 --2- 172.21.15.1:0/1399475574 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f876c06aae0 0x7f876c06cfa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.286+0000 7f8785c63700 1 --2- 172.21.15.1:0/1399475574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87801e48d0 0x7f87801e4d60 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.286+0000 7f8785c63700 1 --2- 172.21.15.1:0/1399475574 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8768003a00 0x7f878013f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.286+0000 7f8785c63700 1 --2- 172.21.15.1:0/1399475574 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87800b4270 0x7f87800b46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:29.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.286+0000 7f8785c63700 1 -- 172.21.15.1:0/1399475574 >> 172.21.15.1:0/1399475574 conn(0x7f87800d00c0 msgr2=0x7f87800ce2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:29.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.286+0000 7f8785c63700 1 -- 172.21.15.1:0/1399475574 shutdown_connections 2024-03-18T20:23:29.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.286+0000 7f8785c63700 1 -- 172.21.15.1:0/1399475574 wait complete. 2024-03-18T20:23:29.292 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-18T20:23:29.304 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T20:23:29.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.433+0000 7fcc0a956700 1 Processor -- start 2024-03-18T20:23:29.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.433+0000 7fcc0a956700 1 -- start start 2024-03-18T20:23:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.434+0000 7fcc0a956700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcc0415d590 0x7fcc040b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.434+0000 7fcc0a956700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcc0406fe10 0x7fcc0406b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.434+0000 7fcc0a956700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0415d970 con 0x7fcc0406a750 2024-03-18T20:23:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.434+0000 7fcc0a956700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc04072320 con 0x7fcc0415d590 2024-03-18T20:23:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.434+0000 7fcc0a956700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc04072600 con 0x7fcc0406fe10 2024-03-18T20:23:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.434+0000 7fcc03fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcc0415d590 0x7fcc040b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:29.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.434+0000 7fcc037fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcc0406a750 0x7fcc0406abd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42420/0 (socket says 172.21.15.1:42420) 2024-03-18T20:23:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.434+0000 7fcc037fe700 1 -- 172.21.15.1:0/2046321118 learned_addr learned my addr 172.21.15.1:0/2046321118 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.434+0000 7fcc03fff700 1 -- 172.21.15.1:0/2046321118 >> v2:172.21.15.165:3300/0 conn(0x7fcc0406fe10 msgr2=0x7fcc0406b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.434+0000 7fcc03fff700 1 --2- 172.21.15.1:0/2046321118 >> v2:172.21.15.165:3300/0 conn(0x7fcc0406fe10 0x7fcc0406b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.434+0000 7fcc03fff700 1 -- 172.21.15.1:0/2046321118 >> v1:172.21.15.1:6789/0 conn(0x7fcc0406a750 legacy=0x7fcc0406abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.434+0000 7fcc03fff700 1 -- 172.21.15.1:0/2046321118 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc040b30b0 con 0x7fcc0415d590 2024-03-18T20:23:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.434+0000 7fcc02ffd700 1 -- 172.21.15.1:0/2046321118 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235879995 0 0) 0x7fcc0415d970 con 0x7fcc0406a750 2024-03-18T20:23:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.435+0000 7fcc03fff700 1 --2- 172.21.15.1:0/2046321118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc0415d590 0x7fcc040b3bc0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fcc040b6350 tx=0x7fcbf4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a0c83566f05986 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.435+0000 7fcc02ffd700 1 -- 172.21.15.1:0/2046321118 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4011830 con 0x7fcc0415d590 2024-03-18T20:23:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.435+0000 7fcc02ffd700 1 -- 172.21.15.1:0/2046321118 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf4011990 con 0x7fcc0415d590 2024-03-18T20:23:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.435+0000 7fcc02ffd700 1 -- 172.21.15.1:0/2046321118 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4011d20 con 0x7fcc0415d590 2024-03-18T20:23:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.435+0000 7fcc0a956700 1 -- 172.21.15.1:0/2046321118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc0415d590 msgr2=0x7fcc040b3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.435+0000 7fcc0a956700 1 --2- 172.21.15.1:0/2046321118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc0415d590 0x7fcc040b3bc0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fcc040b6350 tx=0x7fcbf4009b20 comp rx=0 tx=0).stop 2024-03-18T20:23:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.435+0000 7fcc0a956700 1 -- 172.21.15.1:0/2046321118 shutdown_connections 2024-03-18T20:23:29.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.435+0000 7fcc0a956700 1 --2- 172.21.15.1:0/2046321118 >> v2:172.21.15.165:3300/0 conn(0x7fcc0406fe10 0x7fcc0406b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:29.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.435+0000 7fcc0a956700 1 --2- 172.21.15.1:0/2046321118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc0415d590 0x7fcc040b3bc0 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:29.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.435+0000 7fcc0a956700 1 -- 172.21.15.1:0/2046321118 >> 172.21.15.1:0/2046321118 conn(0x7fcc040d00c0 msgr2=0x7fcc040d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:29.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.435+0000 7fcc0a956700 1 -- 172.21.15.1:0/2046321118 shutdown_connections 2024-03-18T20:23:29.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.435+0000 7fcc0a956700 1 -- 172.21.15.1:0/2046321118 wait complete. 2024-03-18T20:23:29.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.437+0000 7fcc0a956700 1 Processor -- start 2024-03-18T20:23:29.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.437+0000 7fcc0a956700 1 -- start start 2024-03-18T20:23:29.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.437+0000 7fcc0a956700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc0406a750 0x7fcc041475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:29.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.437+0000 7fcc0a956700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc0406fe10 0x7fcc04152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:29.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.437+0000 7fcc0a956700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc0415d590 0x7fcc04152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:29.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.438+0000 7fcc0a956700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0414c560 con 0x7fcc0406fe10 2024-03-18T20:23:29.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.438+0000 7fcc0a956700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc04147b30 con 0x7fcc0406a750 2024-03-18T20:23:29.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.438+0000 7fcc0a956700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0414db50 con 0x7fcc0415d590 2024-03-18T20:23:29.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.438+0000 7fcc037fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc0406fe10 0x7fcc04152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:29.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.438+0000 7fcc037fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc0406fe10 0x7fcc04152590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37102/0 (socket says 172.21.15.1:37102) 2024-03-18T20:23:29.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.438+0000 7fcc037fe700 1 -- 172.21.15.1:0/2511868603 learned_addr learned my addr 172.21.15.1:0/2511868603 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:29.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.438+0000 7fcc03fff700 1 --2- 172.21.15.1:0/2511868603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc0406a750 0x7fcc041475f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:29.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.438+0000 7fcc08ef3700 1 --2- 172.21.15.1:0/2511868603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc0415d590 0x7fcc04152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:29.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.438+0000 7fcc037fe700 1 -- 172.21.15.1:0/2511868603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc0415d590 msgr2=0x7fcc04152ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:29.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.438+0000 7fcc037fe700 1 --2- 172.21.15.1:0/2511868603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc0415d590 0x7fcc04152ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:29.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.438+0000 7fcc037fe700 1 -- 172.21.15.1:0/2511868603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc0406a750 msgr2=0x7fcc041475f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:29.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.438+0000 7fcc037fe700 1 --2- 172.21.15.1:0/2511868603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc0406a750 0x7fcc041475f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.438+0000 7fcc037fe700 1 -- 172.21.15.1:0/2511868603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf40094a0 con 0x7fcc0406fe10 2024-03-18T20:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.438+0000 7fcc037fe700 1 --2- 172.21.15.1:0/2511868603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc0406fe10 0x7fcc04152590 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7fcbec00c550 tx=0x7fcbec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.438+0000 7fcc00ff9700 1 -- 172.21.15.1:0/2511868603 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec004c80 con 0x7fcc0406fe10 2024-03-18T20:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.438+0000 7fcc00ff9700 1 -- 172.21.15.1:0/2511868603 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbec004e00 con 0x7fcc0406fe10 2024-03-18T20:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.438+0000 7fcc00ff9700 1 -- 172.21.15.1:0/2511868603 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec007310 con 0x7fcc0406fe10 2024-03-18T20:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.438+0000 7fcc0a956700 1 -- 172.21.15.1:0/2511868603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0414bbc0 con 0x7fcc0406fe10 2024-03-18T20:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.438+0000 7fcc0a956700 1 -- 172.21.15.1:0/2511868603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0414c180 con 0x7fcc0406fe10 2024-03-18T20:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.439+0000 7fcc0a956700 1 -- 172.21.15.1:0/2511868603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0414c6a0 con 0x7fcc0406fe10 2024-03-18T20:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.440+0000 7fcc00ff9700 1 -- 172.21.15.1:0/2511868603 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 122) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fcbec007470 con 0x7fcc0406fe10 2024-03-18T20:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.440+0000 7fcc00ff9700 1 --2- 172.21.15.1:0/2511868603 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcbe406ad20 0x7fcbe406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.440+0000 7fcc00ff9700 1 -- 172.21.15.1:0/2511868603 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 6723+0+0 (secure 0 0 0) 0x7fcbec089400 con 0x7fcc0406fe10 2024-03-18T20:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.440+0000 7fcc03fff700 1 --2- 172.21.15.1:0/2511868603 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcbe406ad20 0x7fcbe406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:29.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.441+0000 7fcc03fff700 1 --2- 172.21.15.1:0/2511868603 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcbe406ad20 0x7fcbe406d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40090b0 tx=0x7fcbf4008030 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.445+0000 7fcc00ff9700 1 -- 172.21.15.1:0/2511868603 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcc0414c6a0 con 0x7fcc0406fe10 2024-03-18T20:23:29.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:29.646+0000 7fcc0a956700 1 -- 172.21.15.1:0/2511868603 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcc0414c6a0 con 0x7fcc0406fe10 2024-03-18T20:23:30.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.286+0000 7fcc00ff9700 1 -- 172.21.15.1:0/2511868603 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v205) v1 ==== 169+0+0 (secure 0 0 0) 0x7fcc0414c6a0 con 0x7fcc0406fe10 2024-03-18T20:23:30.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.298+0000 7fcc00ff9700 1 -- 172.21.15.1:0/2511868603 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcbec0545d0 con 0x7fcc0406fe10 2024-03-18T20:23:30.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.300+0000 7fcc0a956700 1 -- 172.21.15.1:0/2511868603 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcbe406ad20 msgr2=0x7fcbe406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:30.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.300+0000 7fcc0a956700 1 --2- 172.21.15.1:0/2511868603 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcbe406ad20 0x7fcbe406d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40090b0 tx=0x7fcbf4008030 comp rx=0 tx=0).stop 2024-03-18T20:23:30.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.300+0000 7fcc0a956700 1 -- 172.21.15.1:0/2511868603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc0406fe10 msgr2=0x7fcc04152590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:30.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.300+0000 7fcc0a956700 1 --2- 172.21.15.1:0/2511868603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc0406fe10 0x7fcc04152590 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7fcbec00c550 tx=0x7fcbec00c910 comp rx=0 tx=0).stop 2024-03-18T20:23:30.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.300+0000 7fcc0a956700 1 -- 172.21.15.1:0/2511868603 shutdown_connections 2024-03-18T20:23:30.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.300+0000 7fcc0a956700 1 --2- 172.21.15.1:0/2511868603 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcbe406ad20 0x7fcbe406d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:30.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.300+0000 7fcc0a956700 1 --2- 172.21.15.1:0/2511868603 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc0415d590 0x7fcc04152ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:30.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.300+0000 7fcc0a956700 1 --2- 172.21.15.1:0/2511868603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc0406a750 0x7fcc041475f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:30.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.300+0000 7fcc0a956700 1 --2- 172.21.15.1:0/2511868603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc0406fe10 0x7fcc04152590 secure :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7fcbec00c550 tx=0x7fcbec00c910 comp rx=0 tx=0).stop 2024-03-18T20:23:30.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.300+0000 7fcc0a956700 1 -- 172.21.15.1:0/2511868603 >> 172.21.15.1:0/2511868603 conn(0x7fcc040d00c0 msgr2=0x7fcc040ac8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:30.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.302+0000 7fcc0a956700 1 -- 172.21.15.1:0/2511868603 shutdown_connections 2024-03-18T20:23:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.302+0000 7fcc0a956700 1 -- 172.21.15.1:0/2511868603 wait complete. 2024-03-18T20:23:30.310 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-18T20:23:30.325 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-18T20:23:30.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.453+0000 7f29ed726700 1 Processor -- start 2024-03-18T20:23:30.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.453+0000 7f29ed726700 1 -- start start 2024-03-18T20:23:30.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.454+0000 7f29ed726700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f29e80d3c60 0x7f29e80ce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:30.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.454+0000 7f29ed726700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e80b3370 con 0x7f29e80b4810 2024-03-18T20:23:30.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.454+0000 7f29ed726700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e80b2760 con 0x7f29e80cfb50 2024-03-18T20:23:30.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.454+0000 7f29ed726700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e80d40b0 con 0x7f29e80d3c60 2024-03-18T20:23:30.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.454+0000 7f29e77fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f29e80d3c60 0x7f29e80ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:30.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.454+0000 7f29e67fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f29e80b4810 0x7f29e80cde70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42432/0 (socket says 172.21.15.1:42432) 2024-03-18T20:23:30.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.454+0000 7f29e77fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f29e80d3c60 0x7f29e80ce580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37104/0 (socket says 172.21.15.1:37104) 2024-03-18T20:23:30.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.454+0000 7f29e67fc700 1 -- 172.21.15.1:0/3461508032 learned_addr learned my addr 172.21.15.1:0/3461508032 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:30.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.454+0000 7f29e77fe700 1 -- 172.21.15.1:0/3461508032 >> v1:172.21.15.165:6789/0 conn(0x7f29e80cfb50 legacy=0x7f29e80cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:30.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.454+0000 7f29e77fe700 1 -- 172.21.15.1:0/3461508032 >> v1:172.21.15.1:6789/0 conn(0x7f29e80b4810 legacy=0x7f29e80cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:30.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.454+0000 7f29e77fe700 1 -- 172.21.15.1:0/3461508032 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e80b2a40 con 0x7f29e80d3c60 2024-03-18T20:23:30.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.454+0000 7f29e77fe700 1 --2- 172.21.15.1:0/3461508032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e80d3c60 0x7f29e80ce580 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f29e0009d90 tx=0x7f29e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4008c9da01d62f4c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:30.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.455+0000 7f29e5ffb700 1 -- 172.21.15.1:0/3461508032 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e000e070 con 0x7f29e80d3c60 2024-03-18T20:23:30.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.455+0000 7f29e5ffb700 1 -- 172.21.15.1:0/3461508032 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e000bbd0 con 0x7f29e80d3c60 2024-03-18T20:23:30.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.455+0000 7f29e5ffb700 1 -- 172.21.15.1:0/3461508032 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e0003a20 con 0x7f29e80d3c60 2024-03-18T20:23:30.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.455+0000 7f29ed726700 1 -- 172.21.15.1:0/3461508032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e80d3c60 msgr2=0x7f29e80ce580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:30.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.455+0000 7f29ed726700 1 --2- 172.21.15.1:0/3461508032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e80d3c60 0x7f29e80ce580 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f29e0009d90 tx=0x7f29e000b330 comp rx=0 tx=0).stop 2024-03-18T20:23:30.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.455+0000 7f29ed726700 1 -- 172.21.15.1:0/3461508032 shutdown_connections 2024-03-18T20:23:30.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.455+0000 7f29ed726700 1 --2- 172.21.15.1:0/3461508032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e80d3c60 0x7f29e80ce580 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:30.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.455+0000 7f29ed726700 1 -- 172.21.15.1:0/3461508032 >> 172.21.15.1:0/3461508032 conn(0x7f29e80d3140 msgr2=0x7f29e80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:30.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.455+0000 7f29ed726700 1 -- 172.21.15.1:0/3461508032 shutdown_connections 2024-03-18T20:23:30.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.455+0000 7f29ed726700 1 -- 172.21.15.1:0/3461508032 wait complete. 2024-03-18T20:23:30.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.457+0000 7f29ed726700 1 Processor -- start 2024-03-18T20:23:30.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.457+0000 7f29ed726700 1 -- start start 2024-03-18T20:23:30.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.458+0000 7f29ed726700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29e80b4810 0x7f29e81571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:30.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.458+0000 7f29ed726700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29e80cfb50 0x7f29e80cc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:30.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.458+0000 7f29ed726700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e80d3c60 0x7f29e80c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:30.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.458+0000 7f29ed726700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e81f3a10 con 0x7f29e80d3c60 2024-03-18T20:23:30.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.458+0000 7f29ed726700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e80c83f0 con 0x7f29e80b4810 2024-03-18T20:23:30.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.458+0000 7f29ed726700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e80c86d0 con 0x7f29e80cfb50 2024-03-18T20:23:30.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.458+0000 7f29e77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e80d3c60 0x7f29e80c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:30.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.458+0000 7f29e77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e80d3c60 0x7f29e80c6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37110/0 (socket says 172.21.15.1:37110) 2024-03-18T20:23:30.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.458+0000 7f29e77fe700 1 -- 172.21.15.1:0/3244457245 learned_addr learned my addr 172.21.15.1:0/3244457245 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:30.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.458+0000 7f29e6ffd700 1 --2- 172.21.15.1:0/3244457245 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29e80b4810 0x7f29e81571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:30.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.458+0000 7f29e67fc700 1 --2- 172.21.15.1:0/3244457245 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29e80cfb50 0x7f29e80cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:30.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.458+0000 7f29e77fe700 1 -- 172.21.15.1:0/3244457245 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29e80cfb50 msgr2=0x7f29e80cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:30.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.458+0000 7f29e77fe700 1 --2- 172.21.15.1:0/3244457245 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29e80cfb50 0x7f29e80cc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:30.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.458+0000 7f29e77fe700 1 -- 172.21.15.1:0/3244457245 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29e80b4810 msgr2=0x7f29e81571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:30.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.458+0000 7f29e77fe700 1 --2- 172.21.15.1:0/3244457245 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29e80b4810 0x7f29e81571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:30.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.458+0000 7f29e77fe700 1 -- 172.21.15.1:0/3244457245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e0009a70 con 0x7f29e80d3c60 2024-03-18T20:23:30.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.458+0000 7f29e77fe700 1 --2- 172.21.15.1:0/3244457245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e80d3c60 0x7f29e80c6aa0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f29e808d8e0 tx=0x7f29e0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:30.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.459+0000 7f29cf7fe700 1 -- 172.21.15.1:0/3244457245 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e0012630 con 0x7f29e80d3c60 2024-03-18T20:23:30.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.459+0000 7f29cf7fe700 1 -- 172.21.15.1:0/3244457245 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e00127b0 con 0x7f29e80d3c60 2024-03-18T20:23:30.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.459+0000 7f29ed726700 1 -- 172.21.15.1:0/3244457245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e80c89b0 con 0x7f29e80d3c60 2024-03-18T20:23:30.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.459+0000 7f29cf7fe700 1 -- 172.21.15.1:0/3244457245 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e0012b60 con 0x7f29e80d3c60 2024-03-18T20:23:30.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.459+0000 7f29ed726700 1 -- 172.21.15.1:0/3244457245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e80c8030 con 0x7f29e80d3c60 2024-03-18T20:23:30.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.460+0000 7f29ed726700 1 -- 172.21.15.1:0/3244457245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e81f3bc0 con 0x7f29e80d3c60 2024-03-18T20:23:30.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.460+0000 7f29cf7fe700 1 -- 172.21.15.1:0/3244457245 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f29e0012cc0 con 0x7f29e80d3c60 2024-03-18T20:23:30.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.460+0000 7f29cf7fe700 1 --2- 172.21.15.1:0/3244457245 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f29d406adb0 0x7f29d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:30.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.461+0000 7f29cf7fe700 1 -- 172.21.15.1:0/3244457245 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6309+0+0 (secure 0 0 0) 0x7f29e008bc70 con 0x7f29e80d3c60 2024-03-18T20:23:30.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.461+0000 7f29e6ffd700 1 --2- 172.21.15.1:0/3244457245 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f29d406adb0 0x7f29d406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:30.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.461+0000 7f29e6ffd700 1 --2- 172.21.15.1:0/3244457245 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f29d406adb0 0x7f29d406d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f29dc004f80 tx=0x7f29dc009040 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:30.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.465+0000 7f29cf7fe700 1 -- 172.21.15.1:0/3244457245 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f29e81f3bc0 con 0x7f29e80d3c60 2024-03-18T20:23:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.667+0000 7f29ed726700 1 -- 172.21.15.1:0/3244457245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f29e81f3850 con 0x7f29e80d3c60 2024-03-18T20:23:30.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.670+0000 7f29cf7fe700 1 -- 172.21.15.1:0/3244457245 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v67) v1 ==== 115+0+0 (secure 0 0 0) 0x7f29e00569c0 con 0x7f29e80d3c60 2024-03-18T20:23:30.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.672+0000 7f29ed726700 1 -- 172.21.15.1:0/3244457245 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f29d406adb0 msgr2=0x7f29d406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:30.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.672+0000 7f29ed726700 1 --2- 172.21.15.1:0/3244457245 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f29d406adb0 0x7f29d406d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f29dc004f80 tx=0x7f29dc009040 comp rx=0 tx=0).stop 2024-03-18T20:23:30.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.672+0000 7f29ed726700 1 -- 172.21.15.1:0/3244457245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e80d3c60 msgr2=0x7f29e80c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:30.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.672+0000 7f29ed726700 1 --2- 172.21.15.1:0/3244457245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e80d3c60 0x7f29e80c6aa0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f29e808d8e0 tx=0x7f29e0004ad0 comp rx=0 tx=0).stop 2024-03-18T20:23:30.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.672+0000 7f29ed726700 1 -- 172.21.15.1:0/3244457245 shutdown_connections 2024-03-18T20:23:30.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.672+0000 7f29ed726700 1 --2- 172.21.15.1:0/3244457245 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f29d406adb0 0x7f29d406d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:30.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.672+0000 7f29ed726700 1 --2- 172.21.15.1:0/3244457245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29e80d3c60 0x7f29e80c6aa0 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:30.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.672+0000 7f29ed726700 1 --2- 172.21.15.1:0/3244457245 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29e80b4810 0x7f29e81571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:30.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.672+0000 7f29ed726700 1 --2- 172.21.15.1:0/3244457245 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f29e80cfb50 0x7f29e80cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:30.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.672+0000 7f29ed726700 1 -- 172.21.15.1:0/3244457245 >> 172.21.15.1:0/3244457245 conn(0x7f29e80d3140 msgr2=0x7f29e815e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:30.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.673+0000 7f29ed726700 1 -- 172.21.15.1:0/3244457245 shutdown_connections 2024-03-18T20:23:30.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.673+0000 7f29ed726700 1 -- 172.21.15.1:0/3244457245 wait complete. 2024-03-18T20:23:30.690 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-18T20:23:30.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.813+0000 7f6c440bb700 1 Processor -- start 2024-03-18T20:23:30.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.813+0000 7f6c440bb700 1 -- start start 2024-03-18T20:23:30.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.814+0000 7f6c440bb700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6c3c15d780 0x7f6c3c0cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:30.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.814+0000 7f6c440bb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c3c0ce5a0 0x7f6c3c0cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:30.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.814+0000 7f6c440bb700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c0ce090 con 0x7f6c3c09dd40 2024-03-18T20:23:30.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.814+0000 7f6c440bb700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3c0cb480 con 0x7f6c3c15d780 2024-03-18T20:23:30.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.814+0000 7f6c440bb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c06db30 con 0x7f6c3c0ce5a0 2024-03-18T20:23:30.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.814+0000 7f6c41e57700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6c3c15d780 0x7f6c3c0cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:30.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.814+0000 7f6c41656700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6c3c09dd40 0x7f6c3c0cad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42446/0 (socket says 172.21.15.1:42446) 2024-03-18T20:23:30.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.814+0000 7f6c41656700 1 -- 172.21.15.1:0/540393010 learned_addr learned my addr 172.21.15.1:0/540393010 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:30.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.814+0000 7f6c42658700 1 --2- 172.21.15.1:0/540393010 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c3c0ce5a0 0x7f6c3c0cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:30.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.814+0000 7f6c41e57700 1 -- 172.21.15.1:0/540393010 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c3c0ce5a0 msgr2=0x7f6c3c0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:30.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.814+0000 7f6c41e57700 1 --2- 172.21.15.1:0/540393010 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c3c0ce5a0 0x7f6c3c0cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:30.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.814+0000 7f6c41e57700 1 -- 172.21.15.1:0/540393010 >> v1:172.21.15.1:6789/0 conn(0x7f6c3c09dd40 legacy=0x7f6c3c0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:30.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.814+0000 7f6c41e57700 1 -- 172.21.15.1:0/540393010 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c06d520 con 0x7f6c3c15d780 2024-03-18T20:23:30.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.814+0000 7f6c40e55700 1 -- 172.21.15.1:0/540393010 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993577265 0 0) 0x7f6c3c0ce090 con 0x7f6c3c09dd40 2024-03-18T20:23:30.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.815+0000 7f6c41e57700 1 --2- 172.21.15.1:0/540393010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c3c15d780 0x7f6c3c0cdb50 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f6c38009840 tx=0x7f6c38009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a6a9a4befff6cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:30.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.815+0000 7f6c40e55700 1 -- 172.21.15.1:0/540393010 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c38011830 con 0x7f6c3c15d780 2024-03-18T20:23:30.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.815+0000 7f6c40e55700 1 -- 172.21.15.1:0/540393010 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c38011990 con 0x7f6c3c15d780 2024-03-18T20:23:30.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.815+0000 7f6c40e55700 1 -- 172.21.15.1:0/540393010 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c38011d20 con 0x7f6c3c15d780 2024-03-18T20:23:30.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.815+0000 7f6c440bb700 1 -- 172.21.15.1:0/540393010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c3c15d780 msgr2=0x7f6c3c0cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:30.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.815+0000 7f6c440bb700 1 --2- 172.21.15.1:0/540393010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c3c15d780 0x7f6c3c0cdb50 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f6c38009840 tx=0x7f6c38009b50 comp rx=0 tx=0).stop 2024-03-18T20:23:30.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.815+0000 7f6c440bb700 1 -- 172.21.15.1:0/540393010 shutdown_connections 2024-03-18T20:23:30.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.815+0000 7f6c440bb700 1 --2- 172.21.15.1:0/540393010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c3c15d780 0x7f6c3c0cdb50 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:30.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.815+0000 7f6c440bb700 1 --2- 172.21.15.1:0/540393010 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c3c0ce5a0 0x7f6c3c0cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:30.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.815+0000 7f6c440bb700 1 -- 172.21.15.1:0/540393010 >> 172.21.15.1:0/540393010 conn(0x7f6c3c0cca00 msgr2=0x7f6c3c0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:30.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.815+0000 7f6c440bb700 1 -- 172.21.15.1:0/540393010 shutdown_connections 2024-03-18T20:23:30.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.815+0000 7f6c440bb700 1 -- 172.21.15.1:0/540393010 wait complete. 2024-03-18T20:23:30.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.817+0000 7f6c440bb700 1 Processor -- start 2024-03-18T20:23:30.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.817+0000 7f6c440bb700 1 -- start start 2024-03-18T20:23:30.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.817+0000 7f6c440bb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c3c09dd40 0x7f6c3c1f2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:30.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.817+0000 7f6c440bb700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c3c0ce5a0 0x7f6c3c0c7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:30.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.817+0000 7f6c440bb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c3c15d780 0x7f6c3c06a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:30.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.817+0000 7f6c440bb700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c0c7aa0 con 0x7f6c3c09dd40 2024-03-18T20:23:30.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.817+0000 7f6c440bb700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c0c7d80 con 0x7f6c3c0ce5a0 2024-03-18T20:23:30.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.817+0000 7f6c440bb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c0c8060 con 0x7f6c3c15d780 2024-03-18T20:23:30.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.817+0000 7f6c41e57700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c3c09dd40 0x7f6c3c1f2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:30.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.817+0000 7f6c41e57700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c3c09dd40 0x7f6c3c1f2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37134/0 (socket says 172.21.15.1:37134) 2024-03-18T20:23:30.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.817+0000 7f6c41e57700 1 -- 172.21.15.1:0/2456198509 learned_addr learned my addr 172.21.15.1:0/2456198509 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:30.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.818+0000 7f6c41e57700 1 -- 172.21.15.1:0/2456198509 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c3c15d780 msgr2=0x7f6c3c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:30.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.818+0000 7f6c41e57700 1 --2- 172.21.15.1:0/2456198509 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c3c15d780 0x7f6c3c06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:30.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.818+0000 7f6c41e57700 1 -- 172.21.15.1:0/2456198509 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c3c0ce5a0 msgr2=0x7f6c3c0c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.818+0000 7f6c41656700 1 --2- 172.21.15.1:0/2456198509 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c3c0ce5a0 0x7f6c3c0c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.818+0000 7f6c41e57700 1 --2- 172.21.15.1:0/2456198509 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c3c0ce5a0 0x7f6c3c0c7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.818+0000 7f6c41e57700 1 -- 172.21.15.1:0/2456198509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c380094a0 con 0x7f6c3c09dd40 2024-03-18T20:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.818+0000 7f6c41656700 1 --2- 172.21.15.1:0/2456198509 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c3c0ce5a0 0x7f6c3c0c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.818+0000 7f6c41e57700 1 --2- 172.21.15.1:0/2456198509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c3c09dd40 0x7f6c3c1f2d90 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f6c380090b0 tx=0x7f6c38004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.818+0000 7f6c327fc700 1 -- 172.21.15.1:0/2456198509 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c380038a0 con 0x7f6c3c09dd40 2024-03-18T20:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.818+0000 7f6c327fc700 1 -- 172.21.15.1:0/2456198509 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c3801f070 con 0x7f6c3c09dd40 2024-03-18T20:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.818+0000 7f6c440bb700 1 -- 172.21.15.1:0/2456198509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c3c0c8340 con 0x7f6c3c09dd40 2024-03-18T20:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.818+0000 7f6c327fc700 1 -- 172.21.15.1:0/2456198509 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c38003c20 con 0x7f6c3c09dd40 2024-03-18T20:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.818+0000 7f6c440bb700 1 -- 172.21.15.1:0/2456198509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c3c1e5ed0 con 0x7f6c3c09dd40 2024-03-18T20:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.819+0000 7f6c440bb700 1 -- 172.21.15.1:0/2456198509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c3c0c8550 con 0x7f6c3c09dd40 2024-03-18T20:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.820+0000 7f6c327fc700 1 -- 172.21.15.1:0/2456198509 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6c3801a430 con 0x7f6c3c09dd40 2024-03-18T20:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.820+0000 7f6c327fc700 1 --2- 172.21.15.1:0/2456198509 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6c2006adb0 0x7f6c2006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:30.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.820+0000 7f6c327fc700 1 -- 172.21.15.1:0/2456198509 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6309+0+0 (secure 0 0 0) 0x7f6c38021070 con 0x7f6c3c09dd40 2024-03-18T20:23:30.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.820+0000 7f6c41656700 1 --2- 172.21.15.1:0/2456198509 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6c2006adb0 0x7f6c2006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:30.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.821+0000 7f6c41656700 1 --2- 172.21.15.1:0/2456198509 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6c2006adb0 0x7f6c2006d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c007fd0 tx=0x7f6c2c009040 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:30.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:30.825+0000 7f6c327fc700 1 -- 172.21.15.1:0/2456198509 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6c38055fa0 con 0x7f6c3c09dd40 2024-03-18T20:23:31.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.027+0000 7f6c440bb700 1 -- 172.21.15.1:0/2456198509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f6c3c0676a0 con 0x7f6c3c09dd40 2024-03-18T20:23:31.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.031+0000 7f6c327fc700 1 -- 172.21.15.1:0/2456198509 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v68) v1 ==== 120+0+0 (secure 0 0 0) 0x7f6c38010380 con 0x7f6c3c09dd40 2024-03-18T20:23:31.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.034+0000 7f6c440bb700 1 -- 172.21.15.1:0/2456198509 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6c2006adb0 msgr2=0x7f6c2006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:31.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.034+0000 7f6c440bb700 1 --2- 172.21.15.1:0/2456198509 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6c2006adb0 0x7f6c2006d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c007fd0 tx=0x7f6c2c009040 comp rx=0 tx=0).stop 2024-03-18T20:23:31.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.034+0000 7f6c440bb700 1 -- 172.21.15.1:0/2456198509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c3c09dd40 msgr2=0x7f6c3c1f2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:31.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.034+0000 7f6c440bb700 1 --2- 172.21.15.1:0/2456198509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c3c09dd40 0x7f6c3c1f2d90 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f6c380090b0 tx=0x7f6c38004750 comp rx=0 tx=0).stop 2024-03-18T20:23:31.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.034+0000 7f6c440bb700 1 -- 172.21.15.1:0/2456198509 shutdown_connections 2024-03-18T20:23:31.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.034+0000 7f6c440bb700 1 --2- 172.21.15.1:0/2456198509 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6c2006adb0 0x7f6c2006d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.034+0000 7f6c440bb700 1 --2- 172.21.15.1:0/2456198509 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c3c15d780 0x7f6c3c06a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.034+0000 7f6c440bb700 1 --2- 172.21.15.1:0/2456198509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c3c09dd40 0x7f6c3c1f2d90 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.034+0000 7f6c440bb700 1 --2- 172.21.15.1:0/2456198509 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c3c0ce5a0 0x7f6c3c0c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.034+0000 7f6c440bb700 1 -- 172.21.15.1:0/2456198509 >> 172.21.15.1:0/2456198509 conn(0x7f6c3c0cca00 msgr2=0x7f6c3c06dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:31.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.034+0000 7f6c440bb700 1 -- 172.21.15.1:0/2456198509 shutdown_connections 2024-03-18T20:23:31.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.034+0000 7f6c440bb700 1 -- 172.21.15.1:0/2456198509 wait complete. 2024-03-18T20:23:31.050 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-18T20:23:31.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.180+0000 7fbf09b62700 1 Processor -- start 2024-03-18T20:23:31.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.180+0000 7fbf09b62700 1 -- start start 2024-03-18T20:23:31.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.181+0000 7fbf09b62700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf04071cc0 0x7fbf040ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:31.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.181+0000 7fbf09b62700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf040cac80 con 0x7fbf0406d560 2024-03-18T20:23:31.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.181+0000 7fbf09b62700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf040caf60 con 0x7fbf0415a240 2024-03-18T20:23:31.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.181+0000 7fbf09b62700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf040adff0 con 0x7fbf04071cc0 2024-03-18T20:23:31.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.181+0000 7fbf03fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbf0406d560 0x7fbf040ae410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42450/0 (socket says 172.21.15.1:42450) 2024-03-18T20:23:31.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.181+0000 7fbf03fff700 1 -- 172.21.15.1:0/1930469417 learned_addr learned my addr 172.21.15.1:0/1930469417 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:31.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.181+0000 7fbf02ffd700 1 --2- 172.21.15.1:0/1930469417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf04071cc0 0x7fbf040ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:31.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.181+0000 7fbf027fc700 1 -- 172.21.15.1:0/1930469417 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167911033 0 0) 0x7fbf040cac80 con 0x7fbf0406d560 2024-03-18T20:23:31.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.181+0000 7fbf027fc700 1 -- 172.21.15.1:0/1930469417 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeec0035b0 con 0x7fbf0406d560 2024-03-18T20:23:31.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.181+0000 7fbf027fc700 1 -- 172.21.15.1:0/1930469417 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3913813464 0 0) 0x7fbeec0035b0 con 0x7fbf0406d560 2024-03-18T20:23:31.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.181+0000 7fbf027fc700 1 -- 172.21.15.1:0/1930469417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf04071cc0 msgr2=0x7fbf040ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:31.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.181+0000 7fbf027fc700 1 --2- 172.21.15.1:0/1930469417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf04071cc0 0x7fbf040ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.181+0000 7fbf027fc700 1 -- 172.21.15.1:0/1930469417 >> v1:172.21.15.165:6789/0 conn(0x7fbf0415a240 legacy=0x7fbf040715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:31.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.181+0000 7fbf027fc700 1 -- 172.21.15.1:0/1930469417 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf040cb240 con 0x7fbf0406d560 2024-03-18T20:23:31.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.181+0000 7fbf027fc700 1 -- 172.21.15.1:0/1930469417 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fbefc0019f0 con 0x7fbf0406d560 2024-03-18T20:23:31.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.181+0000 7fbf027fc700 1 -- 172.21.15.1:0/1930469417 >> v1:172.21.15.1:6789/0 conn(0x7fbf0406d560 legacy=0x7fbf040ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:31.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.182+0000 7fbf027fc700 1 --2- 172.21.15.1:0/1930469417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf04071cc0 0x7fbeec003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:31.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.182+0000 7fbf027fc700 1 --2- 172.21.15.1:0/1930469417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbeec003fb0 0x7fbeec004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:31.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.182+0000 7fbf027fc700 1 --2- 172.21.15.1:0/1930469417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbeec004d90 0x7fbeec005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:31.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.182+0000 7fbf027fc700 1 -- 172.21.15.1:0/1930469417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf040adff0 con 0x7fbeec004d90 2024-03-18T20:23:31.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.182+0000 7fbf027fc700 1 -- 172.21.15.1:0/1930469417 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec00baf0 con 0x7fbeec003fb0 2024-03-18T20:23:31.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.182+0000 7fbf027fc700 1 -- 172.21.15.1:0/1930469417 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec00bcb0 con 0x7fbf04071cc0 2024-03-18T20:23:31.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.182+0000 7fbf03fff700 1 --2- 172.21.15.1:0/1930469417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbeec004d90 0x7fbeec005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:31.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.182+0000 7fbf037fe700 1 --2- 172.21.15.1:0/1930469417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf04071cc0 0x7fbeec003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:31.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.182+0000 7fbf02ffd700 1 --2- 172.21.15.1:0/1930469417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbeec003fb0 0x7fbeec004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:31.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.182+0000 7fbf03fff700 1 -- 172.21.15.1:0/1930469417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf04071cc0 msgr2=0x7fbeec003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:31.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.182+0000 7fbf03fff700 1 --2- 172.21.15.1:0/1930469417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf04071cc0 0x7fbeec003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.182+0000 7fbf03fff700 1 -- 172.21.15.1:0/1930469417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbeec003fb0 msgr2=0x7fbeec004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:31.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.182+0000 7fbf03fff700 1 --2- 172.21.15.1:0/1930469417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbeec003fb0 0x7fbeec004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.182+0000 7fbf03fff700 1 -- 172.21.15.1:0/1930469417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeec0035b0 con 0x7fbeec004d90 2024-03-18T20:23:31.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.182+0000 7fbf03fff700 1 --2- 172.21.15.1:0/1930469417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbeec004d90 0x7fbeec005210 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00a800 tx=0x7fbeec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34ab7912f88671d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:31.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.182+0000 7fbf027fc700 1 -- 172.21.15.1:0/1930469417 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc00a2f0 con 0x7fbeec004d90 2024-03-18T20:23:31.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.182+0000 7fbf027fc700 1 -- 172.21.15.1:0/1930469417 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbefc012620 con 0x7fbeec004d90 2024-03-18T20:23:31.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.182+0000 7fbf027fc700 1 -- 172.21.15.1:0/1930469417 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc0129d0 con 0x7fbeec004d90 2024-03-18T20:23:31.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.183+0000 7fbf09b62700 1 -- 172.21.15.1:0/1930469417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbeec004d90 msgr2=0x7fbeec005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:31.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.183+0000 7fbf09b62700 1 --2- 172.21.15.1:0/1930469417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbeec004d90 0x7fbeec005210 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00a800 tx=0x7fbeec005880 comp rx=0 tx=0).stop 2024-03-18T20:23:31.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.183+0000 7fbf03fff700 1 -- 172.21.15.1:0/1930469417 reap_dead start 2024-03-18T20:23:31.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.183+0000 7fbf09b62700 1 -- 172.21.15.1:0/1930469417 shutdown_connections 2024-03-18T20:23:31.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.183+0000 7fbf09b62700 1 -- 172.21.15.1:0/1930469417 >> 172.21.15.1:0/1930469417 conn(0x7fbf040cff30 msgr2=0x7fbf040d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:31.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.183+0000 7fbf09b62700 1 -- 172.21.15.1:0/1930469417 shutdown_connections 2024-03-18T20:23:31.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.183+0000 7fbf09b62700 1 -- 172.21.15.1:0/1930469417 wait complete. 2024-03-18T20:23:31.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.184+0000 7fbf09b62700 1 Processor -- start 2024-03-18T20:23:31.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.185+0000 7fbf09b62700 1 -- start start 2024-03-18T20:23:31.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbf09b62700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf041f0f10 0x7fbf040b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:31.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbf09b62700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf040b55c0 0x7fbf040c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:31.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbf09b62700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf040c5fc0 0x7fbf0406a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:31.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbf09b62700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf040b5a40 con 0x7fbf041f0f10 2024-03-18T20:23:31.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbf09b62700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf040c6440 con 0x7fbf040b55c0 2024-03-18T20:23:31.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbf09b62700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf041f4bb0 con 0x7fbf040c5fc0 2024-03-18T20:23:31.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbf037fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf041f0f10 0x7fbf040b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:31.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbf037fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf041f0f10 0x7fbf040b5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37164/0 (socket says 172.21.15.1:37164) 2024-03-18T20:23:31.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbf037fe700 1 -- 172.21.15.1:0/3287165472 learned_addr learned my addr 172.21.15.1:0/3287165472 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:31.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbf02ffd700 1 --2- 172.21.15.1:0/3287165472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf040b55c0 0x7fbf040c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:31.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbf03fff700 1 --2- 172.21.15.1:0/3287165472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf040c5fc0 0x7fbf0406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:31.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbf037fe700 1 -- 172.21.15.1:0/3287165472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf040c5fc0 msgr2=0x7fbf0406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:31.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbf037fe700 1 --2- 172.21.15.1:0/3287165472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf040c5fc0 0x7fbf0406a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbf037fe700 1 -- 172.21.15.1:0/3287165472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf040b55c0 msgr2=0x7fbf040c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:31.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbf037fe700 1 --2- 172.21.15.1:0/3287165472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf040b55c0 0x7fbf040c5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbf037fe700 1 -- 172.21.15.1:0/3287165472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf041f4e70 con 0x7fbf041f0f10 2024-03-18T20:23:31.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbf037fe700 1 --2- 172.21.15.1:0/3287165472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf041f0f10 0x7fbf040b5080 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fbef400f7f0 tx=0x7fbef400fb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:31.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbeebfff700 1 -- 172.21.15.1:0/3287165472 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef40052e0 con 0x7fbf041f0f10 2024-03-18T20:23:31.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbeebfff700 1 -- 172.21.15.1:0/3287165472 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef4005440 con 0x7fbf041f0f10 2024-03-18T20:23:31.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbeebfff700 1 -- 172.21.15.1:0/3287165472 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef40038c0 con 0x7fbf041f0f10 2024-03-18T20:23:31.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbf09b62700 1 -- 172.21.15.1:0/3287165472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf041f5150 con 0x7fbf041f0f10 2024-03-18T20:23:31.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.186+0000 7fbf09b62700 1 -- 172.21.15.1:0/3287165472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf041f5710 con 0x7fbf041f0f10 2024-03-18T20:23:31.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.187+0000 7fbf09b62700 1 -- 172.21.15.1:0/3287165472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf040cfd30 con 0x7fbf041f0f10 2024-03-18T20:23:31.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.188+0000 7fbeebfff700 1 -- 172.21.15.1:0/3287165472 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbef4003a20 con 0x7fbf041f0f10 2024-03-18T20:23:31.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.188+0000 7fbeebfff700 1 --2- 172.21.15.1:0/3287165472 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbef006ade0 0x7fbef006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:31.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.188+0000 7fbeebfff700 1 -- 172.21.15.1:0/3287165472 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6309+0+0 (secure 0 0 0) 0x7fbef4089370 con 0x7fbf041f0f10 2024-03-18T20:23:31.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.188+0000 7fbf02ffd700 1 --2- 172.21.15.1:0/3287165472 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbef006ade0 0x7fbef006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:31.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.189+0000 7fbf02ffd700 1 --2- 172.21.15.1:0/3287165472 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbef006ade0 0x7fbef006d2a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbf040b05e0 tx=0x7fbef8009210 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:31.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.193+0000 7fbeebfff700 1 -- 172.21.15.1:0/3287165472 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbef4054290 con 0x7fbf041f0f10 2024-03-18T20:23:31.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.394+0000 7fbf09b62700 1 -- 172.21.15.1:0/3287165472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fbf0409dc90 con 0x7fbf041f0f10 2024-03-18T20:23:31.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.398+0000 7fbeebfff700 1 -- 172.21.15.1:0/3287165472 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v69) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbef40578b0 con 0x7fbf041f0f10 2024-03-18T20:23:31.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.400+0000 7fbf09b62700 1 -- 172.21.15.1:0/3287165472 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbef006ade0 msgr2=0x7fbef006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:31.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.400+0000 7fbf09b62700 1 --2- 172.21.15.1:0/3287165472 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbef006ade0 0x7fbef006d2a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbf040b05e0 tx=0x7fbef8009210 comp rx=0 tx=0).stop 2024-03-18T20:23:31.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.400+0000 7fbf09b62700 1 -- 172.21.15.1:0/3287165472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf041f0f10 msgr2=0x7fbf040b5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:31.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.400+0000 7fbf09b62700 1 --2- 172.21.15.1:0/3287165472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf041f0f10 0x7fbf040b5080 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fbef400f7f0 tx=0x7fbef400fb00 comp rx=0 tx=0).stop 2024-03-18T20:23:31.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.400+0000 7fbf09b62700 1 -- 172.21.15.1:0/3287165472 shutdown_connections 2024-03-18T20:23:31.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.400+0000 7fbf09b62700 1 --2- 172.21.15.1:0/3287165472 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbef006ade0 0x7fbef006d2a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.400+0000 7fbf09b62700 1 --2- 172.21.15.1:0/3287165472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf040c5fc0 0x7fbf0406a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.400+0000 7fbf09b62700 1 --2- 172.21.15.1:0/3287165472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf041f0f10 0x7fbf040b5080 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.400+0000 7fbf09b62700 1 --2- 172.21.15.1:0/3287165472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf040b55c0 0x7fbf040c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.400+0000 7fbf09b62700 1 -- 172.21.15.1:0/3287165472 >> 172.21.15.1:0/3287165472 conn(0x7fbf040cff30 msgr2=0x7fbf040b1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:31.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.400+0000 7fbf09b62700 1 -- 172.21.15.1:0/3287165472 shutdown_connections 2024-03-18T20:23:31.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.400+0000 7fbf09b62700 1 -- 172.21.15.1:0/3287165472 wait complete. 2024-03-18T20:23:31.417 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2024-03-18T20:23:31.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.541+0000 7fd570220700 1 Processor -- start 2024-03-18T20:23:31.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.541+0000 7fd570220700 1 -- start start 2024-03-18T20:23:31.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.542+0000 7fd570220700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd56815b4d0 0x7fd56809a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:31.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.542+0000 7fd570220700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd5680cf750 0x7fd568099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:31.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.542+0000 7fd570220700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd568099f80 0x7fd5680cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:31.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.542+0000 7fd570220700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd56809a410 con 0x7fd568099f80 2024-03-18T20:23:31.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.542+0000 7fd570220700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd56806f070 con 0x7fd5680cf750 2024-03-18T20:23:31.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.542+0000 7fd570220700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5680d0260 con 0x7fd56815b4d0 2024-03-18T20:23:31.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.542+0000 7fd56e7bd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd568099f80 0x7fd5680cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:31.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.542+0000 7fd56e7bd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd568099f80 0x7fd5680cbd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37168/0 (socket says 172.21.15.1:37168) 2024-03-18T20:23:31.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.542+0000 7fd56e7bd700 1 -- 172.21.15.1:0/1792771697 learned_addr learned my addr 172.21.15.1:0/1792771697 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:31.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.542+0000 7fd56dfbc700 1 --2- 172.21.15.1:0/1792771697 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd56815b4d0 0x7fd56809a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:31.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.542+0000 7fd56d7bb700 1 --2- 172.21.15.1:0/1792771697 >> v2:172.21.15.165:3300/0 conn(0x7fd5680cf750 0x7fd568099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:31.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.542+0000 7fd56e7bd700 1 -- 172.21.15.1:0/1792771697 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd56815b4d0 msgr2=0x7fd56809a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:31.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.542+0000 7fd56e7bd700 1 --2- 172.21.15.1:0/1792771697 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd56815b4d0 0x7fd56809a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.542+0000 7fd56e7bd700 1 -- 172.21.15.1:0/1792771697 >> v2:172.21.15.165:3300/0 conn(0x7fd5680cf750 msgr2=0x7fd568099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:31.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.542+0000 7fd56e7bd700 1 --2- 172.21.15.1:0/1792771697 >> v2:172.21.15.165:3300/0 conn(0x7fd5680cf750 0x7fd568099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.542+0000 7fd56e7bd700 1 -- 172.21.15.1:0/1792771697 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5680cff80 con 0x7fd568099f80 2024-03-18T20:23:31.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.542+0000 7fd56e7bd700 1 --2- 172.21.15.1:0/1792771697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd568099f80 0x7fd5680cbd60 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7fd5680d22c0 tx=0x7fd56000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6815d0a6c01536a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:31.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.542+0000 7fd56cfba700 1 -- 172.21.15.1:0/1792771697 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56000e070 con 0x7fd568099f80 2024-03-18T20:23:31.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.542+0000 7fd56cfba700 1 -- 172.21.15.1:0/1792771697 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56000b9c0 con 0x7fd568099f80 2024-03-18T20:23:31.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.543+0000 7fd56cfba700 1 -- 172.21.15.1:0/1792771697 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56000bd70 con 0x7fd568099f80 2024-03-18T20:23:31.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.543+0000 7fd570220700 1 -- 172.21.15.1:0/1792771697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd568099f80 msgr2=0x7fd5680cbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:31.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.543+0000 7fd570220700 1 --2- 172.21.15.1:0/1792771697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd568099f80 0x7fd5680cbd60 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7fd5680d22c0 tx=0x7fd56000b330 comp rx=0 tx=0).stop 2024-03-18T20:23:31.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.543+0000 7fd570220700 1 -- 172.21.15.1:0/1792771697 shutdown_connections 2024-03-18T20:23:31.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.543+0000 7fd570220700 1 --2- 172.21.15.1:0/1792771697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd568099f80 0x7fd5680cbd60 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.543+0000 7fd570220700 1 --2- 172.21.15.1:0/1792771697 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd56815b4d0 0x7fd56809a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.543+0000 7fd570220700 1 --2- 172.21.15.1:0/1792771697 >> v2:172.21.15.165:3300/0 conn(0x7fd5680cf750 0x7fd568099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.543+0000 7fd570220700 1 -- 172.21.15.1:0/1792771697 >> 172.21.15.1:0/1792771697 conn(0x7fd5680b2cb0 msgr2=0x7fd5680b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:31.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.543+0000 7fd570220700 1 -- 172.21.15.1:0/1792771697 shutdown_connections 2024-03-18T20:23:31.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.543+0000 7fd570220700 1 -- 172.21.15.1:0/1792771697 wait complete. 2024-03-18T20:23:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.544+0000 7fd570220700 1 Processor -- start 2024-03-18T20:23:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.544+0000 7fd570220700 1 -- start start 2024-03-18T20:23:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.545+0000 7fd570220700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd568099f80 0x7fd56814c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.545+0000 7fd570220700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5680cf750 0x7fd56815d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.545+0000 7fd570220700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd56815b4d0 0x7fd5680aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.545+0000 7fd570220700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56814f800 con 0x7fd568099f80 2024-03-18T20:23:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.545+0000 7fd570220700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56815dc50 con 0x7fd5680cf750 2024-03-18T20:23:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.545+0000 7fd570220700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56815df20 con 0x7fd56815b4d0 2024-03-18T20:23:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.545+0000 7fd56dfbc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd568099f80 0x7fd56814c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.545+0000 7fd56dfbc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd568099f80 0x7fd56814c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37182/0 (socket says 172.21.15.1:37182) 2024-03-18T20:23:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.545+0000 7fd56dfbc700 1 -- 172.21.15.1:0/3054071664 learned_addr learned my addr 172.21.15.1:0/3054071664 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:31.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.545+0000 7fd56e7bd700 1 --2- 172.21.15.1:0/3054071664 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd56815b4d0 0x7fd5680aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.545+0000 7fd56d7bb700 1 --2- 172.21.15.1:0/3054071664 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5680cf750 0x7fd56815d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.545+0000 7fd56dfbc700 1 -- 172.21.15.1:0/3054071664 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd56815b4d0 msgr2=0x7fd5680aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.545+0000 7fd56dfbc700 1 --2- 172.21.15.1:0/3054071664 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd56815b4d0 0x7fd5680aed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.545+0000 7fd56dfbc700 1 -- 172.21.15.1:0/3054071664 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5680cf750 msgr2=0x7fd56815d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.545+0000 7fd56dfbc700 1 --2- 172.21.15.1:0/3054071664 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5680cf750 0x7fd56815d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.545+0000 7fd56dfbc700 1 -- 172.21.15.1:0/3054071664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd560009b30 con 0x7fd568099f80 2024-03-18T20:23:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.546+0000 7fd56dfbc700 1 --2- 172.21.15.1:0/3054071664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd568099f80 0x7fd56814c080 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fd55800d800 tx=0x7fd55800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.546+0000 7fd55e7fc700 1 -- 172.21.15.1:0/3054071664 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5580125f0 con 0x7fd568099f80 2024-03-18T20:23:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.546+0000 7fd55e7fc700 1 -- 172.21.15.1:0/3054071664 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd558012750 con 0x7fd568099f80 2024-03-18T20:23:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.546+0000 7fd570220700 1 -- 172.21.15.1:0/3054071664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd56815e200 con 0x7fd568099f80 2024-03-18T20:23:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.546+0000 7fd55e7fc700 1 -- 172.21.15.1:0/3054071664 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd558012b00 con 0x7fd568099f80 2024-03-18T20:23:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.546+0000 7fd570220700 1 -- 172.21.15.1:0/3054071664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd56815e7c0 con 0x7fd568099f80 2024-03-18T20:23:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.547+0000 7fd570220700 1 -- 172.21.15.1:0/3054071664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd568068240 con 0x7fd568099f80 2024-03-18T20:23:31.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.547+0000 7fd55e7fc700 1 -- 172.21.15.1:0/3054071664 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd558012c60 con 0x7fd568099f80 2024-03-18T20:23:31.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.548+0000 7fd55e7fc700 1 --2- 172.21.15.1:0/3054071664 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd54c06adb0 0x7fd54c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:31.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.548+0000 7fd55e7fc700 1 -- 172.21.15.1:0/3054071664 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6309+0+0 (secure 0 0 0) 0x7fd558089760 con 0x7fd568099f80 2024-03-18T20:23:31.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.548+0000 7fd56d7bb700 1 --2- 172.21.15.1:0/3054071664 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd54c06adb0 0x7fd54c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:31.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.548+0000 7fd56d7bb700 1 --2- 172.21.15.1:0/3054071664 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd54c06adb0 0x7fd54c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd5680c4fb0 tx=0x7fd564008040 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:31.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.552+0000 7fd55e7fc700 1 -- 172.21.15.1:0/3054071664 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd558054680 con 0x7fd568099f80 2024-03-18T20:23:31.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:31.838+0000 7fd570220700 1 -- 172.21.15.1:0/3054071664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7fd5680cb940 con 0x7fd568099f80 2024-03-18T20:23:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.295+0000 7fd55e7fc700 1 -- 172.21.15.1:0/3054071664 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v3034) v1 ==== 165+0+0 (secure 0 0 0) 0x7fd558057ca0 con 0x7fd568099f80 2024-03-18T20:23:32.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.298+0000 7fd570220700 1 -- 172.21.15.1:0/3054071664 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd54c06adb0 msgr2=0x7fd54c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:32.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.298+0000 7fd570220700 1 --2- 172.21.15.1:0/3054071664 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd54c06adb0 0x7fd54c06d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd5680c4fb0 tx=0x7fd564008040 comp rx=0 tx=0).stop 2024-03-18T20:23:32.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.298+0000 7fd570220700 1 -- 172.21.15.1:0/3054071664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd568099f80 msgr2=0x7fd56814c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:32.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.298+0000 7fd570220700 1 --2- 172.21.15.1:0/3054071664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd568099f80 0x7fd56814c080 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fd55800d800 tx=0x7fd55800db10 comp rx=0 tx=0).stop 2024-03-18T20:23:32.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.298+0000 7fd570220700 1 -- 172.21.15.1:0/3054071664 shutdown_connections 2024-03-18T20:23:32.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.298+0000 7fd570220700 1 --2- 172.21.15.1:0/3054071664 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd54c06adb0 0x7fd54c06d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:32.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.298+0000 7fd570220700 1 --2- 172.21.15.1:0/3054071664 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd56815b4d0 0x7fd5680aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:32.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.298+0000 7fd570220700 1 --2- 172.21.15.1:0/3054071664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd568099f80 0x7fd56814c080 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:32.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.298+0000 7fd570220700 1 --2- 172.21.15.1:0/3054071664 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5680cf750 0x7fd56815d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:32.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.298+0000 7fd570220700 1 -- 172.21.15.1:0/3054071664 >> 172.21.15.1:0/3054071664 conn(0x7fd5680b2cb0 msgr2=0x7fd5680b3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:32.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.298+0000 7fd570220700 1 -- 172.21.15.1:0/3054071664 shutdown_connections 2024-03-18T20:23:32.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.298+0000 7fd570220700 1 -- 172.21.15.1:0/3054071664 wait complete. 2024-03-18T20:23:32.316 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2024-03-18T20:23:32.317 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2024-03-18T20:23:32.317 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-03-18T20:23:32.317 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2024-03-18T20:23:32.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.443+0000 7fb9e8221700 1 Processor -- start 2024-03-18T20:23:32.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.443+0000 7fb9e8221700 1 -- start start 2024-03-18T20:23:32.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.444+0000 7fb9e8221700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9e015a2e0 0x7fb9e00ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:32.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.444+0000 7fb9e8221700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e006e270 con 0x7fb9e00ae600 2024-03-18T20:23:32.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.444+0000 7fb9e8221700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e00ae050 con 0x7fb9e00cc170 2024-03-18T20:23:32.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.444+0000 7fb9e8221700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e006f690 con 0x7fb9e015a2e0 2024-03-18T20:23:32.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.444+0000 7fb9e5fbd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb9e00ae600 0x7fb9e00ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42452/0 (socket says 172.21.15.1:42452) 2024-03-18T20:23:32.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.444+0000 7fb9e5fbd700 1 -- 172.21.15.1:0/3355969829 learned_addr learned my addr 172.21.15.1:0/3355969829 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:32.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.444+0000 7fb9e57bc700 1 --2- 172.21.15.1:0/3355969829 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9e015a2e0 0x7fb9e00ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:32.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e4fbb700 1 -- 172.21.15.1:0/3355969829 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549132459 0 0) 0x7fb9e006e270 con 0x7fb9e00ae600 2024-03-18T20:23:32.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e4fbb700 1 -- 172.21.15.1:0/3355969829 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9cc0035b0 con 0x7fb9e00ae600 2024-03-18T20:23:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e4fbb700 1 -- 172.21.15.1:0/3355969829 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809892711 0 0) 0x7fb9e00ae050 con 0x7fb9e00cc170 2024-03-18T20:23:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e4fbb700 1 -- 172.21.15.1:0/3355969829 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9e006e270 con 0x7fb9e00cc170 2024-03-18T20:23:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e4fbb700 1 -- 172.21.15.1:0/3355969829 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 732692298 0 0) 0x7fb9cc0035b0 con 0x7fb9e00ae600 2024-03-18T20:23:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e4fbb700 1 -- 172.21.15.1:0/3355969829 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9e015a2e0 msgr2=0x7fb9e00ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e4fbb700 1 --2- 172.21.15.1:0/3355969829 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9e015a2e0 0x7fb9e00ad790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e4fbb700 1 -- 172.21.15.1:0/3355969829 >> v1:172.21.15.165:6789/0 conn(0x7fb9e00cc170 legacy=0x7fb9e006ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e4fbb700 1 -- 172.21.15.1:0/3355969829 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e00add70 con 0x7fb9e00ae600 2024-03-18T20:23:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e4fbb700 1 -- 172.21.15.1:0/3355969829 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fb9d0001670 con 0x7fb9e00ae600 2024-03-18T20:23:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e4fbb700 1 -- 172.21.15.1:0/3355969829 >> v1:172.21.15.1:6789/0 conn(0x7fb9e00ae600 legacy=0x7fb9e00ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:32.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e4fbb700 1 --2- 172.21.15.1:0/3355969829 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9e015a2e0 0x7fb9cc003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e4fbb700 1 --2- 172.21.15.1:0/3355969829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc004120 0x7fb9cc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e4fbb700 1 --2- 172.21.15.1:0/3355969829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9cc004d90 0x7fb9cc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e4fbb700 1 -- 172.21.15.1:0/3355969829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e006f690 con 0x7fb9cc004120 2024-03-18T20:23:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e4fbb700 1 -- 172.21.15.1:0/3355969829 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc00bba0 con 0x7fb9cc004d90 2024-03-18T20:23:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e4fbb700 1 -- 172.21.15.1:0/3355969829 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc00bd30 con 0x7fb9e015a2e0 2024-03-18T20:23:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e57bc700 1 --2- 172.21.15.1:0/3355969829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc004120 0x7fb9cc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e5fbd700 1 --2- 172.21.15.1:0/3355969829 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9e015a2e0 0x7fb9cc003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e67be700 1 --2- 172.21.15.1:0/3355969829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9cc004d90 0x7fb9cc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e57bc700 1 -- 172.21.15.1:0/3355969829 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9e015a2e0 msgr2=0x7fb9cc003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e57bc700 1 --2- 172.21.15.1:0/3355969829 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9e015a2e0 0x7fb9cc003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e57bc700 1 -- 172.21.15.1:0/3355969829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9cc004d90 msgr2=0x7fb9cc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:32.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e57bc700 1 --2- 172.21.15.1:0/3355969829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9cc004d90 0x7fb9cc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:32.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e57bc700 1 -- 172.21.15.1:0/3355969829 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9cc0035b0 con 0x7fb9cc004120 2024-03-18T20:23:32.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e67be700 1 --2- 172.21.15.1:0/3355969829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9cc004d90 0x7fb9cc005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:32.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.445+0000 7fb9e57bc700 1 --2- 172.21.15.1:0/3355969829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc004120 0x7fb9cc004850 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc009b30 tx=0x7fb9cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d9031ce954adc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:32.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.446+0000 7fb9e4fbb700 1 -- 172.21.15.1:0/3355969829 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc00e6c0 con 0x7fb9cc004120 2024-03-18T20:23:32.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.446+0000 7fb9e4fbb700 1 -- 172.21.15.1:0/3355969829 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9dc00e820 con 0x7fb9cc004120 2024-03-18T20:23:32.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.446+0000 7fb9e4fbb700 1 -- 172.21.15.1:0/3355969829 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc00eb90 con 0x7fb9cc004120 2024-03-18T20:23:32.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.446+0000 7fb9e8221700 1 -- 172.21.15.1:0/3355969829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc004120 msgr2=0x7fb9cc004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:32.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.446+0000 7fb9e8221700 1 --2- 172.21.15.1:0/3355969829 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc004120 0x7fb9cc004850 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc009b30 tx=0x7fb9cc005880 comp rx=0 tx=0).stop 2024-03-18T20:23:32.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.446+0000 7fb9e67be700 1 -- 172.21.15.1:0/3355969829 reap_dead start 2024-03-18T20:23:32.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.446+0000 7fb9e8221700 1 -- 172.21.15.1:0/3355969829 shutdown_connections 2024-03-18T20:23:32.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.446+0000 7fb9e8221700 1 -- 172.21.15.1:0/3355969829 >> 172.21.15.1:0/3355969829 conn(0x7fb9e006b860 msgr2=0x7fb9e006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:32.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.446+0000 7fb9e8221700 1 -- 172.21.15.1:0/3355969829 shutdown_connections 2024-03-18T20:23:32.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.446+0000 7fb9e8221700 1 -- 172.21.15.1:0/3355969829 wait complete. 2024-03-18T20:23:32.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.447+0000 7fb9e8221700 1 Processor -- start 2024-03-18T20:23:32.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.448+0000 7fb9e8221700 1 -- start start 2024-03-18T20:23:32.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.448+0000 7fb9e8221700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc004120 0x7fb9e00ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:32.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.448+0000 7fb9e8221700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9e01584b0 0x7fb9e0158930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:32.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.448+0000 7fb9e8221700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9e01435e0 0x7fb9e0143a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:32.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.448+0000 7fb9e8221700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e0158e70 con 0x7fb9cc004120 2024-03-18T20:23:32.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.448+0000 7fb9e8221700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e00cfcd0 con 0x7fb9e01584b0 2024-03-18T20:23:32.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.448+0000 7fb9e8221700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e0143fa0 con 0x7fb9e01435e0 2024-03-18T20:23:32.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.448+0000 7fb9e5fbd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc004120 0x7fb9e00ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:32.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.448+0000 7fb9e5fbd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc004120 0x7fb9e00ccaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37208/0 (socket says 172.21.15.1:37208) 2024-03-18T20:23:32.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.448+0000 7fb9e5fbd700 1 -- 172.21.15.1:0/1399170633 learned_addr learned my addr 172.21.15.1:0/1399170633 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:32.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.448+0000 7fb9e67be700 1 --2- 172.21.15.1:0/1399170633 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9e01435e0 0x7fb9e0143a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:32.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.448+0000 7fb9e57bc700 1 --2- 172.21.15.1:0/1399170633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9e01584b0 0x7fb9e0158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:32.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.449+0000 7fb9e5fbd700 1 -- 172.21.15.1:0/1399170633 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9e01435e0 msgr2=0x7fb9e0143a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:32.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.449+0000 7fb9e5fbd700 1 --2- 172.21.15.1:0/1399170633 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9e01435e0 0x7fb9e0143a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:32.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.449+0000 7fb9e5fbd700 1 -- 172.21.15.1:0/1399170633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9e01584b0 msgr2=0x7fb9e0158930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:32.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.449+0000 7fb9e5fbd700 1 --2- 172.21.15.1:0/1399170633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9e01584b0 0x7fb9e0158930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:32.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.449+0000 7fb9e5fbd700 1 -- 172.21.15.1:0/1399170633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e0150710 con 0x7fb9cc004120 2024-03-18T20:23:32.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.449+0000 7fb9e67be700 1 --2- 172.21.15.1:0/1399170633 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9e01435e0 0x7fb9e0143a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:23:32.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.449+0000 7fb9e57bc700 1 --2- 172.21.15.1:0/1399170633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9e01584b0 0x7fb9e0158930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:23:32.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.449+0000 7fb9e5fbd700 1 --2- 172.21.15.1:0/1399170633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc004120 0x7fb9e00ccaf0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0007fd0 tx=0x7fb9d00073f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:32.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.449+0000 7fb9d67fc700 1 -- 172.21.15.1:0/1399170633 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0011580 con 0x7fb9cc004120 2024-03-18T20:23:32.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.449+0000 7fb9d67fc700 1 -- 172.21.15.1:0/1399170633 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d0011700 con 0x7fb9cc004120 2024-03-18T20:23:32.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.449+0000 7fb9d67fc700 1 -- 172.21.15.1:0/1399170633 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0011ab0 con 0x7fb9cc004120 2024-03-18T20:23:32.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.449+0000 7fb9e8221700 1 -- 172.21.15.1:0/1399170633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e0159090 con 0x7fb9cc004120 2024-03-18T20:23:32.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.449+0000 7fb9e8221700 1 -- 172.21.15.1:0/1399170633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e01500c0 con 0x7fb9cc004120 2024-03-18T20:23:32.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.450+0000 7fb9e8221700 1 -- 172.21.15.1:0/1399170633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e00676a0 con 0x7fb9cc004120 2024-03-18T20:23:32.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.450+0000 7fb9d67fc700 1 -- 172.21.15.1:0/1399170633 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb9d0011c10 con 0x7fb9cc004120 2024-03-18T20:23:32.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.451+0000 7fb9d67fc700 1 --2- 172.21.15.1:0/1399170633 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb9c406adb0 0x7fb9c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:32.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.451+0000 7fb9d67fc700 1 -- 172.21.15.1:0/1399170633 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6309+0+0 (secure 0 0 0) 0x7fb9d0088d70 con 0x7fb9cc004120 2024-03-18T20:23:32.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.451+0000 7fb9e57bc700 1 --2- 172.21.15.1:0/1399170633 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb9c406adb0 0x7fb9c406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:32.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.451+0000 7fb9e57bc700 1 --2- 172.21.15.1:0/1399170633 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb9c406adb0 0x7fb9c406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc0090b0 tx=0x7fb9dc008030 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:32.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.455+0000 7fb9d67fc700 1 -- 172.21.15.1:0/1399170633 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb9d0053b40 con 0x7fb9cc004120 2024-03-18T20:23:32.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:32.741+0000 7fb9e8221700 1 -- 172.21.15.1:0/1399170633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7fb9e006c420 con 0x7fb9cc004120 2024-03-18T20:23:33.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.296+0000 7fb9d67fc700 1 -- 172.21.15.1:0/1399170633 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3035) v1 ==== 166+0+0 (secure 0 0 0) 0x7fb9d001a020 con 0x7fb9cc004120 2024-03-18T20:23:33.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.299+0000 7fb9e8221700 1 -- 172.21.15.1:0/1399170633 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb9c406adb0 msgr2=0x7fb9c406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:33.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.299+0000 7fb9e8221700 1 --2- 172.21.15.1:0/1399170633 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb9c406adb0 0x7fb9c406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc0090b0 tx=0x7fb9dc008030 comp rx=0 tx=0).stop 2024-03-18T20:23:33.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.299+0000 7fb9e8221700 1 -- 172.21.15.1:0/1399170633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc004120 msgr2=0x7fb9e00ccaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:33.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.299+0000 7fb9e8221700 1 --2- 172.21.15.1:0/1399170633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc004120 0x7fb9e00ccaf0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0007fd0 tx=0x7fb9d00073f0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.299+0000 7fb9e8221700 1 -- 172.21.15.1:0/1399170633 shutdown_connections 2024-03-18T20:23:33.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.299+0000 7fb9e8221700 1 --2- 172.21.15.1:0/1399170633 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb9c406adb0 0x7fb9c406d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.299+0000 7fb9e8221700 1 --2- 172.21.15.1:0/1399170633 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb9e01435e0 0x7fb9e0143a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.299+0000 7fb9e8221700 1 --2- 172.21.15.1:0/1399170633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9cc004120 0x7fb9e00ccaf0 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.299+0000 7fb9e8221700 1 --2- 172.21.15.1:0/1399170633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb9e01584b0 0x7fb9e0158930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.299+0000 7fb9e8221700 1 -- 172.21.15.1:0/1399170633 >> 172.21.15.1:0/1399170633 conn(0x7fb9e006b860 msgr2=0x7fb9e006ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:33.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.299+0000 7fb9e8221700 1 -- 172.21.15.1:0/1399170633 shutdown_connections 2024-03-18T20:23:33.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.299+0000 7fb9e8221700 1 -- 172.21.15.1:0/1399170633 wait complete. 2024-03-18T20:23:33.319 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-18T20:23:33.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.450+0000 7f4485319700 1 Processor -- start 2024-03-18T20:23:33.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.450+0000 7f4485319700 1 -- start start 2024-03-18T20:23:33.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.450+0000 7f4485319700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f44800cfb50 0x7f44800d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:33.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.450+0000 7f4485319700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f44800cee80 0x7f44800b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.450+0000 7f4485319700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44800b3370 con 0x7f44800cff30 2024-03-18T20:23:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.450+0000 7f4485319700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44800cf2d0 con 0x7f44800cfb50 2024-03-18T20:23:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.450+0000 7f4485319700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44800b2760 con 0x7f44800cee80 2024-03-18T20:23:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.451+0000 7f447effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f44800cfb50 0x7f44800d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.451+0000 7f447e7fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f44800cff30 0x7f44800b3800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42460/0 (socket says 172.21.15.1:42460) 2024-03-18T20:23:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.451+0000 7f447e7fc700 1 -- 172.21.15.1:0/3119537528 learned_addr learned my addr 172.21.15.1:0/3119537528 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.451+0000 7f447effd700 1 -- 172.21.15.1:0/3119537528 >> v2:172.21.15.165:3300/0 conn(0x7f44800cee80 msgr2=0x7f44800b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.451+0000 7f447effd700 1 --2- 172.21.15.1:0/3119537528 >> v2:172.21.15.165:3300/0 conn(0x7f44800cee80 0x7f44800b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.451+0000 7f447effd700 1 -- 172.21.15.1:0/3119537528 >> v1:172.21.15.1:6789/0 conn(0x7f44800cff30 legacy=0x7f44800b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.451+0000 7f447effd700 1 -- 172.21.15.1:0/3119537528 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44800b2a40 con 0x7f44800cfb50 2024-03-18T20:23:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.451+0000 7f447effd700 1 --2- 172.21.15.1:0/3119537528 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44800cfb50 0x7f44800d3c60 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f44800ca1a0 tx=0x7f4470009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a5ca737888fb1d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.451+0000 7f447dffb700 1 -- 172.21.15.1:0/3119537528 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4470005e30 con 0x7f44800cfb50 2024-03-18T20:23:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.451+0000 7f447dffb700 1 -- 172.21.15.1:0/3119537528 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4470006030 con 0x7f44800cfb50 2024-03-18T20:23:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.451+0000 7f447dffb700 1 -- 172.21.15.1:0/3119537528 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44700063c0 con 0x7f44800cfb50 2024-03-18T20:23:33.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.452+0000 7f4485319700 1 -- 172.21.15.1:0/3119537528 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44800cfb50 msgr2=0x7f44800d3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:33.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.452+0000 7f4485319700 1 --2- 172.21.15.1:0/3119537528 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44800cfb50 0x7f44800d3c60 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f44800ca1a0 tx=0x7f4470009b00 comp rx=0 tx=0).stop 2024-03-18T20:23:33.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.452+0000 7f4485319700 1 -- 172.21.15.1:0/3119537528 shutdown_connections 2024-03-18T20:23:33.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.452+0000 7f4485319700 1 --2- 172.21.15.1:0/3119537528 >> v2:172.21.15.165:3300/0 conn(0x7f44800cee80 0x7f44800b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.452+0000 7f4485319700 1 --2- 172.21.15.1:0/3119537528 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44800cfb50 0x7f44800d3c60 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.452+0000 7f4485319700 1 -- 172.21.15.1:0/3119537528 >> 172.21.15.1:0/3119537528 conn(0x7f44800d3140 msgr2=0x7f44800d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:33.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.452+0000 7f4485319700 1 -- 172.21.15.1:0/3119537528 shutdown_connections 2024-03-18T20:23:33.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.452+0000 7f4485319700 1 -- 172.21.15.1:0/3119537528 wait complete. 2024-03-18T20:23:33.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.454+0000 7f4485319700 1 Processor -- start 2024-03-18T20:23:33.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.454+0000 7f4485319700 1 -- start start 2024-03-18T20:23:33.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.454+0000 7f4485319700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f44800cee80 0x7f44800c8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:33.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f4485319700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44800cfb50 0x7f44800b5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:33.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f4485319700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44800cff30 0x7f44800b6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:33.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f4485319700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44800cc7e0 con 0x7f44800cfb50 2024-03-18T20:23:33.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f4485319700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44800ca340 con 0x7f44800cee80 2024-03-18T20:23:33.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f4485319700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44800c8cb0 con 0x7f44800cff30 2024-03-18T20:23:33.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f447e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44800cfb50 0x7f44800b5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:33.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f447e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44800cfb50 0x7f44800b5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37216/0 (socket says 172.21.15.1:37216) 2024-03-18T20:23:33.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f447e7fc700 1 -- 172.21.15.1:0/1309659350 learned_addr learned my addr 172.21.15.1:0/1309659350 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:33.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f447effd700 1 --2- 172.21.15.1:0/1309659350 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f44800cee80 0x7f44800c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:33.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f447e7fc700 1 -- 172.21.15.1:0/1309659350 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44800cff30 msgr2=0x7f44800b6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:33.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f447f7fe700 1 --2- 172.21.15.1:0/1309659350 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44800cff30 0x7f44800b6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:33.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f447e7fc700 1 --2- 172.21.15.1:0/1309659350 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44800cff30 0x7f44800b6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f447e7fc700 1 -- 172.21.15.1:0/1309659350 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f44800cee80 msgr2=0x7f44800c8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:33.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f447e7fc700 1 --2- 172.21.15.1:0/1309659350 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f44800cee80 0x7f44800c8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f447e7fc700 1 -- 172.21.15.1:0/1309659350 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4470009430 con 0x7f44800cfb50 2024-03-18T20:23:33.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f447effd700 1 --2- 172.21.15.1:0/1309659350 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f44800cee80 0x7f44800c8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:33.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f447f7fe700 1 --2- 172.21.15.1:0/1309659350 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44800cff30 0x7f44800b6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:33.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f447e7fc700 1 --2- 172.21.15.1:0/1309659350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44800cfb50 0x7f44800b5cd0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f447400c480 tx=0x7f447400c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:33.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f44677fe700 1 -- 172.21.15.1:0/1309659350 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4474005d70 con 0x7f44800cfb50 2024-03-18T20:23:33.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f44677fe700 1 -- 172.21.15.1:0/1309659350 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44740070d0 con 0x7f44800cfb50 2024-03-18T20:23:33.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f44677fe700 1 -- 172.21.15.1:0/1309659350 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4474007440 con 0x7f44800cfb50 2024-03-18T20:23:33.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f4485319700 1 -- 172.21.15.1:0/1309659350 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44800c8f90 con 0x7f44800cfb50 2024-03-18T20:23:33.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.455+0000 7f4485319700 1 -- 172.21.15.1:0/1309659350 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44800c9550 con 0x7f44800cfb50 2024-03-18T20:23:33.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.456+0000 7f4485319700 1 -- 172.21.15.1:0/1309659350 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4480073350 con 0x7f44800cfb50 2024-03-18T20:23:33.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.457+0000 7f44677fe700 1 -- 172.21.15.1:0/1309659350 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f44740075a0 con 0x7f44800cfb50 2024-03-18T20:23:33.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.457+0000 7f44677fe700 1 --2- 172.21.15.1:0/1309659350 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f446c06adb0 0x7f446c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:33.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.457+0000 7f44677fe700 1 -- 172.21.15.1:0/1309659350 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6309+0+0 (secure 0 0 0) 0x7f4474088aa0 con 0x7f44800cfb50 2024-03-18T20:23:33.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.457+0000 7f447effd700 1 --2- 172.21.15.1:0/1309659350 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f446c06adb0 0x7f446c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:33.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.458+0000 7f447effd700 1 --2- 172.21.15.1:0/1309659350 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f446c06adb0 0x7f446c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4470013040 tx=0x7f4470008070 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:33.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.462+0000 7f44677fe700 1 -- 172.21.15.1:0/1309659350 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4474053a50 con 0x7f44800cfb50 2024-03-18T20:23:33.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.663+0000 7f4485319700 1 -- 172.21.15.1:0/1309659350 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f44800ca480 con 0x7f44800cfb50 2024-03-18T20:23:33.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.667+0000 7f44677fe700 1 -- 172.21.15.1:0/1309659350 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v70)=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7f44740170c0 con 0x7f44800cfb50 2024-03-18T20:23:33.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.669+0000 7f4485319700 1 -- 172.21.15.1:0/1309659350 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f446c06adb0 msgr2=0x7f446c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:33.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.669+0000 7f4485319700 1 --2- 172.21.15.1:0/1309659350 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f446c06adb0 0x7f446c06d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4470013040 tx=0x7f4470008070 comp rx=0 tx=0).stop 2024-03-18T20:23:33.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.669+0000 7f4485319700 1 -- 172.21.15.1:0/1309659350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44800cfb50 msgr2=0x7f44800b5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:33.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.669+0000 7f4485319700 1 --2- 172.21.15.1:0/1309659350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44800cfb50 0x7f44800b5cd0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f447400c480 tx=0x7f447400c840 comp rx=0 tx=0).stop 2024-03-18T20:23:33.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.669+0000 7f4485319700 1 -- 172.21.15.1:0/1309659350 shutdown_connections 2024-03-18T20:23:33.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.669+0000 7f4485319700 1 --2- 172.21.15.1:0/1309659350 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f446c06adb0 0x7f446c06d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.669+0000 7f4485319700 1 --2- 172.21.15.1:0/1309659350 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44800cff30 0x7f44800b6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.669+0000 7f4485319700 1 --2- 172.21.15.1:0/1309659350 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f44800cee80 0x7f44800c8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.669+0000 7f4485319700 1 --2- 172.21.15.1:0/1309659350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44800cfb50 0x7f44800b5cd0 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:33.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.669+0000 7f4485319700 1 -- 172.21.15.1:0/1309659350 >> 172.21.15.1:0/1309659350 conn(0x7f44800d3140 msgr2=0x7f44800b1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:33.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.669+0000 7f4485319700 1 -- 172.21.15.1:0/1309659350 shutdown_connections 2024-03-18T20:23:33.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:33.669+0000 7f4485319700 1 -- 172.21.15.1:0/1309659350 wait complete. 2024-03-18T20:23:33.686 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T20:23:33.687 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:23:33.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:23:33.778 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T20:23:33.779 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:23:33.779 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:23:33.798 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:23:33.799 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:23:33.799 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:23:33.819 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:23:33.820 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:23:33.820 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:23:33.857 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:23:33.857 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:23:33.857 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:23:33.879 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:23:33.880 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-18T20:23:33.880 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:23:34.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.018+0000 7f8b58add700 1 Processor -- start 2024-03-18T20:23:34.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.018+0000 7f8b58add700 1 -- start start 2024-03-18T20:23:34.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.019+0000 7f8b58add700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8b54099a40 0x7f8b54099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:34.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.019+0000 7f8b58add700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5409a3e0 con 0x7f8b540cc310 2024-03-18T20:23:34.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.019+0000 7f8b58add700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b540d0c80 con 0x7f8b5415d680 2024-03-18T20:23:34.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.019+0000 7f8b58add700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b540d09a0 con 0x7f8b54099a40 2024-03-18T20:23:34.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.019+0000 7f8b52d9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8b540cc310 0x7f8b540cc7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42462/0 (socket says 172.21.15.1:42462) 2024-03-18T20:23:34.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.019+0000 7f8b52d9d700 1 -- 172.21.15.1:0/2291043241 learned_addr learned my addr 172.21.15.1:0/2291043241 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:34.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b51d9b700 1 --2- 172.21.15.1:0/2291043241 >> v2:172.21.15.165:3300/0 conn(0x7f8b54099a40 0x7f8b54099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:34.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b5159a700 1 -- 172.21.15.1:0/2291043241 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219665130 0 0) 0x7f8b5409a3e0 con 0x7f8b540cc310 2024-03-18T20:23:34.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b5159a700 1 -- 172.21.15.1:0/2291043241 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b3c0035b0 con 0x7f8b540cc310 2024-03-18T20:23:34.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b5159a700 1 -- 172.21.15.1:0/2291043241 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281247252 0 0) 0x7f8b540d0c80 con 0x7f8b5415d680 2024-03-18T20:23:34.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b5159a700 1 -- 172.21.15.1:0/2291043241 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b5409a3e0 con 0x7f8b5415d680 2024-03-18T20:23:34.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b5159a700 1 -- 172.21.15.1:0/2291043241 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 561698530 0 0) 0x7f8b3c0035b0 con 0x7f8b540cc310 2024-03-18T20:23:34.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b5159a700 1 -- 172.21.15.1:0/2291043241 >> v2:172.21.15.165:3300/0 conn(0x7f8b54099a40 msgr2=0x7f8b54099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:34.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b5159a700 1 --2- 172.21.15.1:0/2291043241 >> v2:172.21.15.165:3300/0 conn(0x7f8b54099a40 0x7f8b54099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:34.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b5159a700 1 -- 172.21.15.1:0/2291043241 >> v1:172.21.15.165:6789/0 conn(0x7f8b5415d680 legacy=0x7f8b540cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:34.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b5159a700 1 -- 172.21.15.1:0/2291043241 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b540cb830 con 0x7f8b540cc310 2024-03-18T20:23:34.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b5159a700 1 -- 172.21.15.1:0/2291043241 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f8b4c0019f0 con 0x7f8b540cc310 2024-03-18T20:23:34.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b5159a700 1 -- 172.21.15.1:0/2291043241 >> v1:172.21.15.1:6789/0 conn(0x7f8b540cc310 legacy=0x7f8b540cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:34.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b5159a700 1 --2- 172.21.15.1:0/2291043241 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8b3c003b90 0x7f8b3c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:34.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b5159a700 1 --2- 172.21.15.1:0/2291043241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b3c004850 0x7f8b3c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:34.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b5159a700 1 --2- 172.21.15.1:0/2291043241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8b3c005210 0x7f8b3c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:34.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b5159a700 1 -- 172.21.15.1:0/2291043241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b540d09a0 con 0x7f8b3c004850 2024-03-18T20:23:34.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b5159a700 1 -- 172.21.15.1:0/2291043241 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c00c090 con 0x7f8b3c003b90 2024-03-18T20:23:34.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b5159a700 1 -- 172.21.15.1:0/2291043241 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c00c250 con 0x7f8b3c005210 2024-03-18T20:23:34.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b51d9b700 1 --2- 172.21.15.1:0/2291043241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b3c004850 0x7f8b3c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:34.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b5259c700 1 --2- 172.21.15.1:0/2291043241 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8b3c003b90 0x7f8b3c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:34.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b52d9d700 1 --2- 172.21.15.1:0/2291043241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8b3c005210 0x7f8b3c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:34.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b51d9b700 1 -- 172.21.15.1:0/2291043241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8b3c005210 msgr2=0x7f8b3c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:34.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b51d9b700 1 --2- 172.21.15.1:0/2291043241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8b3c005210 0x7f8b3c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:34.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b51d9b700 1 -- 172.21.15.1:0/2291043241 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8b3c003b90 msgr2=0x7f8b3c003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:34.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b51d9b700 1 --2- 172.21.15.1:0/2291043241 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8b3c003b90 0x7f8b3c003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:34.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b51d9b700 1 -- 172.21.15.1:0/2291043241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c0035b0 con 0x7f8b3c004850 2024-03-18T20:23:34.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.020+0000 7f8b52d9d700 1 -- 172.21.15.1:0/2291043241 reap_dead start 2024-03-18T20:23:34.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.021+0000 7f8b51d9b700 1 --2- 172.21.15.1:0/2291043241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b3c004850 0x7f8b3c004cd0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009b60 tx=0x7f8b3c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2d57ac44fb070e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:34.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.021+0000 7f8b5159a700 1 -- 172.21.15.1:0/2291043241 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4400e6c0 con 0x7f8b3c004850 2024-03-18T20:23:34.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.021+0000 7f8b5159a700 1 -- 172.21.15.1:0/2291043241 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b4400e820 con 0x7f8b3c004850 2024-03-18T20:23:34.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.021+0000 7f8b5159a700 1 -- 172.21.15.1:0/2291043241 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4400eb90 con 0x7f8b3c004850 2024-03-18T20:23:34.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.021+0000 7f8b58add700 1 -- 172.21.15.1:0/2291043241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b3c004850 msgr2=0x7f8b3c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:34.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.021+0000 7f8b58add700 1 --2- 172.21.15.1:0/2291043241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b3c004850 0x7f8b3c004cd0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009b60 tx=0x7f8b3c005880 comp rx=0 tx=0).stop 2024-03-18T20:23:34.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.021+0000 7f8b58add700 1 -- 172.21.15.1:0/2291043241 shutdown_connections 2024-03-18T20:23:34.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.021+0000 7f8b58add700 1 --2- 172.21.15.1:0/2291043241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b3c004850 0x7f8b3c004cd0 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:34.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.021+0000 7f8b58add700 1 -- 172.21.15.1:0/2291043241 >> 172.21.15.1:0/2291043241 conn(0x7f8b540b2ba0 msgr2=0x7f8b540b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:34.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.021+0000 7f8b58add700 1 -- 172.21.15.1:0/2291043241 shutdown_connections 2024-03-18T20:23:34.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.021+0000 7f8b58add700 1 -- 172.21.15.1:0/2291043241 wait complete. 2024-03-18T20:23:34.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.023+0000 7f8b58add700 1 Processor -- start 2024-03-18T20:23:34.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.023+0000 7f8b58add700 1 -- start start 2024-03-18T20:23:34.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.023+0000 7f8b58add700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8b3c004850 0x7f8b5414ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:34.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.023+0000 7f8b58add700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b540c7470 0x7f8b540c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:34.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.023+0000 7f8b58add700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8b540c4d50 0x7f8b540c51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:34.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.023+0000 7f8b58add700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b540c5710 con 0x7f8b540c7470 2024-03-18T20:23:34.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.023+0000 7f8b58add700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5414f040 con 0x7f8b540c4d50 2024-03-18T20:23:34.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.023+0000 7f8b58add700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5414e450 con 0x7f8b3c004850 2024-03-18T20:23:34.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.024+0000 7f8b51d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b540c7470 0x7f8b540c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:34.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.024+0000 7f8b51d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b540c7470 0x7f8b540c78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37224/0 (socket says 172.21.15.1:37224) 2024-03-18T20:23:34.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.024+0000 7f8b51d9b700 1 -- 172.21.15.1:0/2028251775 learned_addr learned my addr 172.21.15.1:0/2028251775 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:34.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.024+0000 7f8b5259c700 1 --2- 172.21.15.1:0/2028251775 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8b3c004850 0x7f8b5414ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:34.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.024+0000 7f8b52d9d700 1 --2- 172.21.15.1:0/2028251775 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8b540c4d50 0x7f8b540c51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:34.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.024+0000 7f8b51d9b700 1 -- 172.21.15.1:0/2028251775 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8b3c004850 msgr2=0x7f8b5414ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:34.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.024+0000 7f8b51d9b700 1 --2- 172.21.15.1:0/2028251775 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8b3c004850 0x7f8b5414ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:34.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.024+0000 7f8b51d9b700 1 -- 172.21.15.1:0/2028251775 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8b540c4d50 msgr2=0x7f8b540c51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:34.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.024+0000 7f8b51d9b700 1 --2- 172.21.15.1:0/2028251775 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8b540c4d50 0x7f8b540c51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:34.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.024+0000 7f8b51d9b700 1 -- 172.21.15.1:0/2028251775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5415dbe0 con 0x7f8b540c7470 2024-03-18T20:23:34.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.024+0000 7f8b52d9d700 1 --2- 172.21.15.1:0/2028251775 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8b540c4d50 0x7f8b540c51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:34.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.024+0000 7f8b5259c700 1 --2- 172.21.15.1:0/2028251775 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8b3c004850 0x7f8b5414ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:34.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.024+0000 7f8b51d9b700 1 --2- 172.21.15.1:0/2028251775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b540c7470 0x7f8b540c78f0 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c00c060 tx=0x7f8b440047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:34.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.024+0000 7f8b42ffd700 1 -- 172.21.15.1:0/2028251775 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4400e6c0 con 0x7f8b540c7470 2024-03-18T20:23:34.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.024+0000 7f8b42ffd700 1 -- 172.21.15.1:0/2028251775 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b44003ad0 con 0x7f8b540c7470 2024-03-18T20:23:34.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.024+0000 7f8b42ffd700 1 -- 172.21.15.1:0/2028251775 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b44003e80 con 0x7f8b540c7470 2024-03-18T20:23:34.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.024+0000 7f8b58add700 1 -- 172.21.15.1:0/2028251775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b5414e710 con 0x7f8b540c7470 2024-03-18T20:23:34.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.024+0000 7f8b58add700 1 -- 172.21.15.1:0/2028251775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b5414ecd0 con 0x7f8b540c7470 2024-03-18T20:23:34.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.025+0000 7f8b58add700 1 -- 172.21.15.1:0/2028251775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b5414f320 con 0x7f8b540c7470 2024-03-18T20:23:34.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.026+0000 7f8b42ffd700 1 -- 172.21.15.1:0/2028251775 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8b44013470 con 0x7f8b540c7470 2024-03-18T20:23:34.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.026+0000 7f8b42ffd700 1 --2- 172.21.15.1:0/2028251775 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8b3406adb0 0x7f8b3406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:34.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.026+0000 7f8b42ffd700 1 -- 172.21.15.1:0/2028251775 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 6309+0+0 (secure 0 0 0) 0x7f8b4408ad40 con 0x7f8b540c7470 2024-03-18T20:23:34.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.026+0000 7f8b5259c700 1 --2- 172.21.15.1:0/2028251775 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8b3406adb0 0x7f8b3406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:34.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.027+0000 7f8b5259c700 1 --2- 172.21.15.1:0/2028251775 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8b3406adb0 0x7f8b3406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8b48007fd0 tx=0x7f8b48009040 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:34.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.031+0000 7f8b42ffd700 1 -- 172.21.15.1:0/2028251775 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8b44055d60 con 0x7f8b540c7470 2024-03-18T20:23:34.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.278+0000 7f8b58add700 1 -- 172.21.15.1:0/2028251775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b540676a0 con 0x7f8b540c7470 2024-03-18T20:23:34.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.278+0000 7f8b42ffd700 1 -- 172.21.15.1:0/2028251775 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+3530 (secure 0 0 0) 0x7f8b44017020 con 0x7f8b540c7470 2024-03-18T20:23:34.283 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:34.283 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":91,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8204,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2572860438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2572860438},{"type":"v1","addr":"172.21.15.165:6835","nonce":2572860438}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9439,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/3158632513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":3158632513},{"type":"v1","addr":"172.21.15.1:6836","nonce":3158632513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":9448,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/4005747445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4005747445},{"type":"v1","addr":"172.21.15.165:6834","nonce":4005747445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":10938,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2922067079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2922067079},{"type":"v1","addr":"172.21.15.1:6837","nonce":2922067079}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":91}],"filesystems":[]} 2024-03-18T20:23:34.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.280+0000 7f8b58add700 1 -- 172.21.15.1:0/2028251775 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8b3406adb0 msgr2=0x7f8b3406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:34.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.280+0000 7f8b58add700 1 --2- 172.21.15.1:0/2028251775 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8b3406adb0 0x7f8b3406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8b48007fd0 tx=0x7f8b48009040 comp rx=0 tx=0).stop 2024-03-18T20:23:34.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.280+0000 7f8b58add700 1 -- 172.21.15.1:0/2028251775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b540c7470 msgr2=0x7f8b540c78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:34.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.280+0000 7f8b58add700 1 --2- 172.21.15.1:0/2028251775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b540c7470 0x7f8b540c78f0 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c00c060 tx=0x7f8b440047c0 comp rx=0 tx=0).stop 2024-03-18T20:23:34.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.281+0000 7f8b58add700 1 -- 172.21.15.1:0/2028251775 shutdown_connections 2024-03-18T20:23:34.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.281+0000 7f8b58add700 1 --2- 172.21.15.1:0/2028251775 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8b3406adb0 0x7f8b3406d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:34.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.281+0000 7f8b58add700 1 --2- 172.21.15.1:0/2028251775 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8b3c004850 0x7f8b5414ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:34.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.281+0000 7f8b58add700 1 --2- 172.21.15.1:0/2028251775 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8b540c4d50 0x7f8b540c51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:34.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.281+0000 7f8b58add700 1 --2- 172.21.15.1:0/2028251775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b540c7470 0x7f8b540c78f0 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:34.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.281+0000 7f8b58add700 1 -- 172.21.15.1:0/2028251775 >> 172.21.15.1:0/2028251775 conn(0x7f8b540b2ba0 msgr2=0x7f8b540cd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:34.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.281+0000 7f8b58add700 1 -- 172.21.15.1:0/2028251775 shutdown_connections 2024-03-18T20:23:34.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.281+0000 7f8b58add700 1 -- 172.21.15.1:0/2028251775 wait complete. 2024-03-18T20:23:34.286 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 91 2024-03-18T20:23:34.297 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T20:23:34.297 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-18T20:23:34.298 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T20:23:34.298 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:23:34.298 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T20:23:34.298 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-18T20:23:34.298 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-18T20:23:34.298 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:23:34.299 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T20:23:34.299 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-18T20:23:34.299 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-18T20:23:34.299 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:23:34.299 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T20:23:34.299 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-18T20:23:34.299 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-18T20:23:34.299 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:23:34.307 INFO:tasks.ceph.mds.a:Stopped 2024-03-18T20:23:34.308 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-18T20:23:34.308 INFO:tasks.ceph.mds.c:Stopped 2024-03-18T20:23:34.308 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-18T20:23:34.310 INFO:tasks.ceph.mds.b:Stopped 2024-03-18T20:23:34.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-18T20:23:34.311 INFO:tasks.ceph.mds.d:Stopped 2024-03-18T20:23:34.311 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-18T20:23:34.350 INFO:tasks.ceph.mds.a:Started 2024-03-18T20:23:34.350 DEBUG:teuthology.parallel:result is None 2024-03-18T20:23:34.351 INFO:tasks.ceph.mds.c:Started 2024-03-18T20:23:34.351 DEBUG:teuthology.parallel:result is None 2024-03-18T20:23:34.352 INFO:tasks.ceph.mds.b:Started 2024-03-18T20:23:34.352 DEBUG:teuthology.parallel:result is None 2024-03-18T20:23:34.352 INFO:tasks.ceph.mds.d:Started 2024-03-18T20:23:34.353 DEBUG:teuthology.parallel:result is None 2024-03-18T20:23:34.353 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-18T20:23:34.433 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-18T20:23:34.433 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T20:23:34.436 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-03-18T20:23:34.436 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-03-18T20:23:34.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.483+0000 7f44c7d4d700 1 Processor -- start 2024-03-18T20:23:34.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.483+0000 7f44c7d4d700 1 -- start start 2024-03-18T20:23:34.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.484+0000 7f44c7d4d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f44c00d1ec0 0x7f44c00b82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:34.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.484+0000 7f44c7d4d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c00ce1e0 con 0x7f44c015d7c0 2024-03-18T20:23:34.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.484+0000 7f44c7d4d700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c00ce4c0 con 0x7f44c015d3e0 2024-03-18T20:23:34.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.484+0000 7f44c7d4d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c00cee60 con 0x7f44c00d1ec0 2024-03-18T20:23:34.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.484+0000 7f44c52e8700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f44c015d7c0 0x7f44c00d17b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45456/0 (socket says 172.21.15.1:45456) 2024-03-18T20:23:34.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.484+0000 7f44c52e8700 1 -- 172.21.15.1:0/126024939 learned_addr learned my addr 172.21.15.1:0/126024939 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:34.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.484+0000 7f44c62ea700 1 --2- 172.21.15.1:0/126024939 >> v2:172.21.15.165:3300/0 conn(0x7f44c00d1ec0 0x7f44c00b82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:34.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.484+0000 7f44c4ae7700 1 -- 172.21.15.1:0/126024939 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639153662 0 0) 0x7f44c00ce1e0 con 0x7f44c015d7c0 2024-03-18T20:23:34.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.484+0000 7f44c4ae7700 1 -- 172.21.15.1:0/126024939 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44ac0035b0 con 0x7f44c015d7c0 2024-03-18T20:23:34.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.484+0000 7f44c62ea700 1 -- 172.21.15.1:0/126024939 >> v1:172.21.15.165:6789/0 conn(0x7f44c015d3e0 legacy=0x7f44c00d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:34.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.485+0000 7f44c4ae7700 1 -- 172.21.15.1:0/126024939 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311858238 0 0) 0x7f44c00ce4c0 con 0x7f44c015d3e0 2024-03-18T20:23:34.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.485+0000 7f44c62ea700 1 -- 172.21.15.1:0/126024939 >> v1:172.21.15.1:6789/0 conn(0x7f44c015d7c0 legacy=0x7f44c00d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:34.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.485+0000 7f44c62ea700 1 -- 172.21.15.1:0/126024939 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c00ce7a0 con 0x7f44c00d1ec0 2024-03-18T20:23:34.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.485+0000 7f44c62ea700 1 --2- 172.21.15.1:0/126024939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f44c00d1ec0 0x7f44c00b82d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f44b8009d90 tx=0x7f44b800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1e94737cce301df server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:34.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.485+0000 7f44c4ae7700 1 -- 172.21.15.1:0/126024939 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b800e070 con 0x7f44c00d1ec0 2024-03-18T20:23:34.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.485+0000 7f44c4ae7700 1 -- 172.21.15.1:0/126024939 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b800bb20 con 0x7f44c00d1ec0 2024-03-18T20:23:34.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.485+0000 7f44c4ae7700 1 -- 172.21.15.1:0/126024939 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b800bed0 con 0x7f44c00d1ec0 2024-03-18T20:23:34.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.485+0000 7f44c7d4d700 1 -- 172.21.15.1:0/126024939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f44c00d1ec0 msgr2=0x7f44c00b82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:34.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.485+0000 7f44c7d4d700 1 --2- 172.21.15.1:0/126024939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f44c00d1ec0 0x7f44c00b82d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f44b8009d90 tx=0x7f44b800b330 comp rx=0 tx=0).stop 2024-03-18T20:23:34.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.485+0000 7f44c7d4d700 1 -- 172.21.15.1:0/126024939 shutdown_connections 2024-03-18T20:23:34.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.485+0000 7f44c7d4d700 1 --2- 172.21.15.1:0/126024939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f44c00d1ec0 0x7f44c00b82d0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:34.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.485+0000 7f44c7d4d700 1 -- 172.21.15.1:0/126024939 >> 172.21.15.1:0/126024939 conn(0x7f44c009c540 msgr2=0x7f44c00d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:34.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.486+0000 7f44c7d4d700 1 -- 172.21.15.1:0/126024939 shutdown_connections 2024-03-18T20:23:34.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.486+0000 7f44c7d4d700 1 -- 172.21.15.1:0/126024939 wait complete. 2024-03-18T20:23:34.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.487+0000 7f44c7d4d700 1 Processor -- start 2024-03-18T20:23:34.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.487+0000 7f44c7d4d700 1 -- start start 2024-03-18T20:23:34.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.487+0000 7f44c7d4d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44c00d1ec0 0x7f44c015dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:34.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.487+0000 7f44c7d4d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f44c015d3e0 0x7f44c0154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:34.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.487+0000 7f44c7d4d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44c00b4230 0x7f44c015f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:34.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.487+0000 7f44c7d4d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c00c8b60 con 0x7f44c00b4230 2024-03-18T20:23:34.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.487+0000 7f44c7d4d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c015e0f0 con 0x7f44c015d3e0 2024-03-18T20:23:34.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.487+0000 7f44c7d4d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c015e320 con 0x7f44c00d1ec0 2024-03-18T20:23:34.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.488+0000 7f44c62ea700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44c00b4230 0x7f44c015f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:34.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.488+0000 7f44c62ea700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44c00b4230 0x7f44c015f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46362/0 (socket says 172.21.15.1:46362) 2024-03-18T20:23:34.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.488+0000 7f44c62ea700 1 -- 172.21.15.1:0/1894293366 learned_addr learned my addr 172.21.15.1:0/1894293366 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:34.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.488+0000 7f44c52e8700 1 --2- 172.21.15.1:0/1894293366 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f44c015d3e0 0x7f44c0154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:34.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.488+0000 7f44c5ae9700 1 --2- 172.21.15.1:0/1894293366 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44c00d1ec0 0x7f44c015dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:34.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.488+0000 7f44c62ea700 1 -- 172.21.15.1:0/1894293366 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44c00d1ec0 msgr2=0x7f44c015dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:34.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.488+0000 7f44c62ea700 1 --2- 172.21.15.1:0/1894293366 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44c00d1ec0 0x7f44c015dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:34.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.488+0000 7f44c62ea700 1 -- 172.21.15.1:0/1894293366 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f44c015d3e0 msgr2=0x7f44c0154930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:34.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.488+0000 7f44c62ea700 1 --2- 172.21.15.1:0/1894293366 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f44c015d3e0 0x7f44c0154930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:34.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.488+0000 7f44c62ea700 1 -- 172.21.15.1:0/1894293366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b8009a70 con 0x7f44c00b4230 2024-03-18T20:23:34.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.488+0000 7f44c62ea700 1 --2- 172.21.15.1:0/1894293366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44c00b4230 0x7f44c015f3c0 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7f44b8014040 tx=0x7f44b8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:34.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.488+0000 7f44b67fc700 1 -- 172.21.15.1:0/1894293366 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b801b810 con 0x7f44c00b4230 2024-03-18T20:23:34.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.488+0000 7f44b67fc700 1 -- 172.21.15.1:0/1894293366 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b8020070 con 0x7f44c00b4230 2024-03-18T20:23:34.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.488+0000 7f44b67fc700 1 -- 172.21.15.1:0/1894293366 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b801bbb0 con 0x7f44c00b4230 2024-03-18T20:23:34.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.488+0000 7f44c7d4d700 1 -- 172.21.15.1:0/1894293366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c015e600 con 0x7f44c00b4230 2024-03-18T20:23:34.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.488+0000 7f44c7d4d700 1 -- 172.21.15.1:0/1894293366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c015ebc0 con 0x7f44c00b4230 2024-03-18T20:23:34.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.489+0000 7f44c7d4d700 1 -- 172.21.15.1:0/1894293366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c0073350 con 0x7f44c00b4230 2024-03-18T20:23:34.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.490+0000 7f44b67fc700 1 -- 172.21.15.1:0/1894293366 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f44b801bd10 con 0x7f44c00b4230 2024-03-18T20:23:34.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.490+0000 7f44b67fc700 1 --2- 172.21.15.1:0/1894293366 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f44a406adb0 0x7f44a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:34.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.490+0000 7f44b67fc700 1 -- 172.21.15.1:0/1894293366 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6309+0+0 (secure 0 0 0) 0x7f44b8022070 con 0x7f44c00b4230 2024-03-18T20:23:34.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.490+0000 7f44c5ae9700 1 --2- 172.21.15.1:0/1894293366 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f44a406adb0 0x7f44a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:34.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.491+0000 7f44c5ae9700 1 --2- 172.21.15.1:0/1894293366 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f44a406adb0 0x7f44a406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f44b0000f60 tx=0x7f44b0009040 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:34.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.495+0000 7f44b67fc700 1 -- 172.21.15.1:0/1894293366 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f44b805a0d0 con 0x7f44c00b4230 2024-03-18T20:23:34.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:34.714+0000 7f44c7d4d700 1 -- 172.21.15.1:0/1894293366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f44c00d08c0 con 0x7f44c00b4230 2024-03-18T20:23:35.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.670+0000 7f44b67fc700 1 -- 172.21.15.1:0/1894293366 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v210) v1 ==== 97+0+0 (secure 0 0 0) 0x7f44b8059c60 con 0x7f44c00b4230 2024-03-18T20:23:35.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.674+0000 7f44c7d4d700 1 -- 172.21.15.1:0/1894293366 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f44a406adb0 msgr2=0x7f44a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:35.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.674+0000 7f44c7d4d700 1 --2- 172.21.15.1:0/1894293366 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f44a406adb0 0x7f44a406d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f44b0000f60 tx=0x7f44b0009040 comp rx=0 tx=0).stop 2024-03-18T20:23:35.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.674+0000 7f44c7d4d700 1 -- 172.21.15.1:0/1894293366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44c00b4230 msgr2=0x7f44c015f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:35.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.674+0000 7f44c7d4d700 1 --2- 172.21.15.1:0/1894293366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44c00b4230 0x7f44c015f3c0 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7f44b8014040 tx=0x7f44b8004ad0 comp rx=0 tx=0).stop 2024-03-18T20:23:35.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.674+0000 7f44c7d4d700 1 -- 172.21.15.1:0/1894293366 shutdown_connections 2024-03-18T20:23:35.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.674+0000 7f44c7d4d700 1 --2- 172.21.15.1:0/1894293366 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f44a406adb0 0x7f44a406d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:35.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.674+0000 7f44c7d4d700 1 --2- 172.21.15.1:0/1894293366 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44c00d1ec0 0x7f44c015dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:35.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.674+0000 7f44c7d4d700 1 --2- 172.21.15.1:0/1894293366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44c00b4230 0x7f44c015f3c0 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:35.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.674+0000 7f44c7d4d700 1 --2- 172.21.15.1:0/1894293366 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f44c015d3e0 0x7f44c0154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:35.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.674+0000 7f44c7d4d700 1 -- 172.21.15.1:0/1894293366 >> 172.21.15.1:0/1894293366 conn(0x7f44c009c540 msgr2=0x7f44c00d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:35.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.674+0000 7f44c7d4d700 1 -- 172.21.15.1:0/1894293366 shutdown_connections 2024-03-18T20:23:35.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.674+0000 7f44c7d4d700 1 -- 172.21.15.1:0/1894293366 wait complete. 2024-03-18T20:23:35.680 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-18T20:23:35.693 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-18T20:23:35.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.821+0000 7fc842f93700 1 Processor -- start 2024-03-18T20:23:35.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.821+0000 7fc842f93700 1 -- start start 2024-03-18T20:23:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.822+0000 7fc842f93700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc83c15d680 0x7fc83c0cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.822+0000 7fc842f93700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc83c09a150 0x7fc83c0cc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.822+0000 7fc842f93700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83c0d0c80 con 0x7fc83c0d04b0 2024-03-18T20:23:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.822+0000 7fc842f93700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc83c0d09a0 con 0x7fc83c09a150 2024-03-18T20:23:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.822+0000 7fc842f93700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc83c0cb830 con 0x7fc83c15d680 2024-03-18T20:23:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.822+0000 7fc841530700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc83c09a150 0x7fc83c0cc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:35.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.822+0000 7fc841530700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc83c09a150 0x7fc83c0cc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46376/0 (socket says 172.21.15.1:46376) 2024-03-18T20:23:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.822+0000 7fc841530700 1 -- 172.21.15.1:0/3740208605 learned_addr learned my addr 172.21.15.1:0/3740208605 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.822+0000 7fc841530700 1 -- 172.21.15.1:0/3740208605 >> v2:172.21.15.165:3300/0 conn(0x7fc83c15d680 msgr2=0x7fc83c0cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.822+0000 7fc841530700 1 --2- 172.21.15.1:0/3740208605 >> v2:172.21.15.165:3300/0 conn(0x7fc83c15d680 0x7fc83c0cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.822+0000 7fc841530700 1 -- 172.21.15.1:0/3740208605 >> v1:172.21.15.165:6789/0 conn(0x7fc83c0d04b0 legacy=0x7fc83c099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.822+0000 7fc841530700 1 -- 172.21.15.1:0/3740208605 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83c0cbb10 con 0x7fc83c09a150 2024-03-18T20:23:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.823+0000 7fc841530700 1 --2- 172.21.15.1:0/3740208605 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc83c09a150 0x7fc83c0cc310 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7fc83c072070 tx=0x7fc83400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e45cc36b7aef6ab server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.823+0000 7fc83b7fe700 1 -- 172.21.15.1:0/3740208605 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83400e070 con 0x7fc83c09a150 2024-03-18T20:23:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.823+0000 7fc83b7fe700 1 -- 172.21.15.1:0/3740208605 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc834003a20 con 0x7fc83c09a150 2024-03-18T20:23:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.823+0000 7fc83b7fe700 1 -- 172.21.15.1:0/3740208605 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc834003dd0 con 0x7fc83c09a150 2024-03-18T20:23:35.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.823+0000 7fc842f93700 1 -- 172.21.15.1:0/3740208605 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc83c09a150 msgr2=0x7fc83c0cc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.823+0000 7fc842f93700 1 --2- 172.21.15.1:0/3740208605 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc83c09a150 0x7fc83c0cc310 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7fc83c072070 tx=0x7fc83400b330 comp rx=0 tx=0).stop 2024-03-18T20:23:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.823+0000 7fc842f93700 1 -- 172.21.15.1:0/3740208605 shutdown_connections 2024-03-18T20:23:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.823+0000 7fc842f93700 1 --2- 172.21.15.1:0/3740208605 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc83c09a150 0x7fc83c0cc310 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.823+0000 7fc842f93700 1 --2- 172.21.15.1:0/3740208605 >> v2:172.21.15.165:3300/0 conn(0x7fc83c15d680 0x7fc83c0cff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.823+0000 7fc842f93700 1 -- 172.21.15.1:0/3740208605 >> 172.21.15.1:0/3740208605 conn(0x7fc83c0b2ba0 msgr2=0x7fc83c0b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.823+0000 7fc842f93700 1 -- 172.21.15.1:0/3740208605 shutdown_connections 2024-03-18T20:23:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.823+0000 7fc842f93700 1 -- 172.21.15.1:0/3740208605 wait complete. 2024-03-18T20:23:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.825+0000 7fc842f93700 1 Processor -- start 2024-03-18T20:23:35.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.825+0000 7fc842f93700 1 -- start start 2024-03-18T20:23:35.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.825+0000 7fc842f93700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc83c09a150 0x7fc83c06f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:35.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.825+0000 7fc842f93700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc83c0d04b0 0x7fc83c15a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:35.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.825+0000 7fc842f93700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc83c15d680 0x7fc83c15a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:35.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.825+0000 7fc842f93700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c15e0e0 con 0x7fc83c09a150 2024-03-18T20:23:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.825+0000 7fc842f93700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c15e3c0 con 0x7fc83c15d680 2024-03-18T20:23:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.825+0000 7fc842f93700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c15e6a0 con 0x7fc83c0d04b0 2024-03-18T20:23:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.826+0000 7fc840d2f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc83c09a150 0x7fc83c06f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.826+0000 7fc840d2f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc83c09a150 0x7fc83c06f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46392/0 (socket says 172.21.15.1:46392) 2024-03-18T20:23:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.826+0000 7fc840d2f700 1 -- 172.21.15.1:0/2175010504 learned_addr learned my addr 172.21.15.1:0/2175010504 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.826+0000 7fc841530700 1 --2- 172.21.15.1:0/2175010504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc83c15d680 0x7fc83c15a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.826+0000 7fc83bfff700 1 --2- 172.21.15.1:0/2175010504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc83c0d04b0 0x7fc83c15a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.826+0000 7fc840d2f700 1 -- 172.21.15.1:0/2175010504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc83c0d04b0 msgr2=0x7fc83c15a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.826+0000 7fc840d2f700 1 --2- 172.21.15.1:0/2175010504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc83c0d04b0 0x7fc83c15a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.826+0000 7fc840d2f700 1 -- 172.21.15.1:0/2175010504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc83c15d680 msgr2=0x7fc83c15a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:35.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.826+0000 7fc840d2f700 1 --2- 172.21.15.1:0/2175010504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc83c15d680 0x7fc83c15a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:35.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.826+0000 7fc840d2f700 1 -- 172.21.15.1:0/2175010504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc834009b30 con 0x7fc83c09a150 2024-03-18T20:23:35.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.826+0000 7fc841530700 1 --2- 172.21.15.1:0/2175010504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc83c15d680 0x7fc83c15a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:35.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.826+0000 7fc840d2f700 1 --2- 172.21.15.1:0/2175010504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc83c09a150 0x7fc83c06f7c0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7fc83000b4a0 tx=0x7fc83000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:35.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.826+0000 7fc8397fa700 1 -- 172.21.15.1:0/2175010504 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc830011830 con 0x7fc83c09a150 2024-03-18T20:23:35.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.826+0000 7fc8397fa700 1 -- 172.21.15.1:0/2175010504 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc830011990 con 0x7fc83c09a150 2024-03-18T20:23:35.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.826+0000 7fc8397fa700 1 -- 172.21.15.1:0/2175010504 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc830011d40 con 0x7fc83c09a150 2024-03-18T20:23:35.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.826+0000 7fc842f93700 1 -- 172.21.15.1:0/2175010504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc83c15dbe0 con 0x7fc83c09a150 2024-03-18T20:23:35.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.826+0000 7fc842f93700 1 -- 172.21.15.1:0/2175010504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc83c15ec60 con 0x7fc83c09a150 2024-03-18T20:23:35.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.827+0000 7fc842f93700 1 -- 172.21.15.1:0/2175010504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc83c0676a0 con 0x7fc83c09a150 2024-03-18T20:23:35.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.828+0000 7fc8397fa700 1 -- 172.21.15.1:0/2175010504 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc83001a400 con 0x7fc83c09a150 2024-03-18T20:23:35.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.828+0000 7fc8397fa700 1 --2- 172.21.15.1:0/2175010504 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc82006adb0 0x7fc82006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:35.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.828+0000 7fc8397fa700 1 -- 172.21.15.1:0/2175010504 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5277+0+0 (secure 0 0 0) 0x7fc830089340 con 0x7fc83c09a150 2024-03-18T20:23:35.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.828+0000 7fc83bfff700 1 --2- 172.21.15.1:0/2175010504 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc82006adb0 0x7fc82006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:35.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.829+0000 7fc83bfff700 1 --2- 172.21.15.1:0/2175010504 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc82006adb0 0x7fc82006d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc83c066b00 tx=0x7fc828009040 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:35.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:35.833+0000 7fc8397fa700 1 -- 172.21.15.1:0/2175010504 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc830054540 con 0x7fc83c09a150 2024-03-18T20:23:36.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.118+0000 7fc842f93700 1 -- 172.21.15.1:0/2175010504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc83c0cffd0 con 0x7fc83c09a150 2024-03-18T20:23:36.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.119+0000 7fc8397fa700 1 -- 172.21.15.1:0/2175010504 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v48) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fc830057b60 con 0x7fc83c09a150 2024-03-18T20:23:36.123 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:36.124 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:23:36.124 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-18T20:23:36.124 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:36.124 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-18T20:23:36.124 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlR93/CxAAbpe+5n3wHRur9xi8QzdCnQ==", 2024-03-18T20:23:36.124 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:23:36.124 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:23:36.124 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:23:36.124 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:23:36.124 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:23:36.124 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.124 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:36.124 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:36.124 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-18T20:23:36.125 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJ3OmGBAA37u227nxrMsh/JHt71HBxQ==", 2024-03-18T20:23:36.125 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:23:36.125 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:23:36.125 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:23:36.125 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:23:36.125 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:23:36.125 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.125 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:36.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:36.126 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-18T20:23:36.126 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJZFoExAAmI16oxwNKbf8dKetPfyk+g==", 2024-03-18T20:23:36.126 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:23:36.126 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:23:36.126 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:23:36.126 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:23:36.126 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:23:36.126 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:36.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:36.127 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-18T20:23:36.127 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlQrcvIBAA1LMv+EgC7S6wspX6QF49Pg==", 2024-03-18T20:23:36.127 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:23:36.127 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:23:36.127 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:23:36.127 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:23:36.127 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:23:36.127 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:36.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:36.128 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-18T20:23:36.128 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCUjPhlYOVHChAARFQrksZmMIdTlntd8TgvrQ==", 2024-03-18T20:23:36.128 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:23:36.128 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:23:36.128 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:23:36.128 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:23:36.128 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:36.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:36.128 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-18T20:23:36.129 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCXjPhl2zSDCBAA6Al1gkWVB79ZI9djGpVChA==", 2024-03-18T20:23:36.129 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:23:36.129 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:23:36.129 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:23:36.129 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:23:36.129 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:36.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:36.129 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-18T20:23:36.129 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCajPhlSoG/BhAAR93USDe6DCQW+hREOsnTlw==", 2024-03-18T20:23:36.129 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:23:36.129 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:23:36.129 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:23:36.130 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:23:36.130 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:36.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:36.130 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-18T20:23:36.130 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCdjPhlhJg5BRAAWuQhcnxyjR5UOiM13yKdAQ==", 2024-03-18T20:23:36.130 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:23:36.130 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:23:36.130 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:23:36.130 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:23:36.130 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:36.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:36.131 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-18T20:23:36.131 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCjjPhlCFnpCRAAKIyOQyVdjXZUTnAS6Nw/ng==", 2024-03-18T20:23:36.131 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:23:36.131 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:23:36.131 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:23:36.131 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:23:36.131 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:36.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:36.131 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-18T20:23:36.131 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCmjPhlAeFkCBAAJcEFpRk/03lQJxj+l2hEPA==", 2024-03-18T20:23:36.131 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:23:36.132 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:23:36.132 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:23:36.132 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:23:36.132 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:36.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:36.132 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-18T20:23:36.132 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCpjPhl/xuuBhAA0LgKfXEvN2O11ZlRQJVOkw==", 2024-03-18T20:23:36.132 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:23:36.132 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:23:36.132 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:23:36.132 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:23:36.132 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:36.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:36.133 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-18T20:23:36.133 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCsjPhldiR4BRAA571oh6U8PzOv/4LzjYLF/A==", 2024-03-18T20:23:36.133 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:23:36.133 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:23:36.133 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:23:36.133 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:23:36.133 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:36.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:36.133 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-18T20:23:36.133 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlu+2xJBAANnXGGjfeqP6S2FTecsfOsg==", 2024-03-18T20:23:36.133 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:23:36.133 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:23:36.134 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-18T20:23:36.134 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-18T20:23:36.134 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:36.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:36.134 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-18T20:23:36.134 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlhJ9+KxAAEnrQrCrgJR6xI+pucwzAXA==", 2024-03-18T20:23:36.134 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:23:36.134 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:23:36.134 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:23:36.134 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:23:36.134 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:23:36.134 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:36.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:36.135 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.2", 2024-03-18T20:23:36.135 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlxBUpKBAAIGTwmLWyj+P2Yvl55QVcWA==", 2024-03-18T20:23:36.135 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:23:36.135 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:23:36.135 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:23:36.135 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:23:36.135 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:23:36.135 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:36.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:36.135 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.3", 2024-03-18T20:23:36.135 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlmao3MRAAHeiQ6eyBh9QjjKPCtxkC8g==", 2024-03-18T20:23:36.135 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:23:36.136 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:23:36.136 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:23:36.136 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:23:36.136 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:23:36.136 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:36.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:36.136 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-18T20:23:36.136 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCRjPhlemOuKRAAVUggmtB7+EiVgyfJZ8TCCA==", 2024-03-18T20:23:36.136 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:23:36.136 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:23:36.136 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:23:36.136 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:23:36.136 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:23:36.137 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:36.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:36.137 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-18T20:23:36.137 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhleIuOCBAA9CH+AHewQbn+gpVr+6lJyw==", 2024-03-18T20:23:36.137 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:23:36.137 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:23:36.137 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T20:23:36.137 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:23:36.137 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:36.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:36.137 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-18T20:23:36.138 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlL0pTBBAAdFiPcdwwz514cuLAvioxYA==", 2024-03-18T20:23:36.138 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:23:36.138 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:23:36.138 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T20:23:36.138 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:23:36.138 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.138 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:36.138 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-18T20:23:36.138 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:23:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.121+0000 7fc842f93700 1 -- 172.21.15.1:0/2175010504 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc82006adb0 msgr2=0x7fc82006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.121+0000 7fc842f93700 1 --2- 172.21.15.1:0/2175010504 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc82006adb0 0x7fc82006d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc83c066b00 tx=0x7fc828009040 comp rx=0 tx=0).stop 2024-03-18T20:23:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.121+0000 7fc842f93700 1 -- 172.21.15.1:0/2175010504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc83c09a150 msgr2=0x7fc83c06f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.121+0000 7fc842f93700 1 --2- 172.21.15.1:0/2175010504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc83c09a150 0x7fc83c06f7c0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7fc83000b4a0 tx=0x7fc83000b860 comp rx=0 tx=0).stop 2024-03-18T20:23:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.121+0000 7fc842f93700 1 -- 172.21.15.1:0/2175010504 shutdown_connections 2024-03-18T20:23:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.121+0000 7fc842f93700 1 --2- 172.21.15.1:0/2175010504 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc82006adb0 0x7fc82006d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.121+0000 7fc842f93700 1 --2- 172.21.15.1:0/2175010504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc83c15d680 0x7fc83c15a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.121+0000 7fc842f93700 1 --2- 172.21.15.1:0/2175010504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc83c09a150 0x7fc83c06f7c0 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.121+0000 7fc842f93700 1 --2- 172.21.15.1:0/2175010504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc83c0d04b0 0x7fc83c15a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.121+0000 7fc842f93700 1 -- 172.21.15.1:0/2175010504 >> 172.21.15.1:0/2175010504 conn(0x7fc83c0b2ba0 msgr2=0x7fc83c0b2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:36.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.121+0000 7fc842f93700 1 -- 172.21.15.1:0/2175010504 shutdown_connections 2024-03-18T20:23:36.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.121+0000 7fc842f93700 1 -- 172.21.15.1:0/2175010504 wait complete. 2024-03-18T20:23:36.141 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:23:36.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.263+0000 7f3f5faba700 1 Processor -- start 2024-03-18T20:23:36.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.263+0000 7f3f5faba700 1 -- start start 2024-03-18T20:23:36.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5faba700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f580cfab0 0x7f3f5815c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:36.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5faba700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3f58140b20 0x7f3f580cfe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:36.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5faba700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3f580cdf80 0x7f3f58070db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:36.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5faba700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f5815c660 con 0x7f3f580cdf80 2024-03-18T20:23:36.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5faba700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f580d03d0 con 0x7f3f58140b20 2024-03-18T20:23:36.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5faba700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5815fbe0 con 0x7f3f580cfab0 2024-03-18T20:23:36.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5e057700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3f580cdf80 0x7f3f58070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:36.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5e057700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3f580cdf80 0x7f3f58070db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46394/0 (socket says 172.21.15.1:46394) 2024-03-18T20:23:36.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5e057700 1 -- 172.21.15.1:0/124841865 learned_addr learned my addr 172.21.15.1:0/124841865 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:36.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5d856700 1 --2- 172.21.15.1:0/124841865 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f580cfab0 0x7f3f5815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:36.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5d055700 1 --2- 172.21.15.1:0/124841865 >> v2:172.21.15.165:3300/0 conn(0x7f3f58140b20 0x7f3f580cfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:36.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5e057700 1 -- 172.21.15.1:0/124841865 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f580cfab0 msgr2=0x7f3f5815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:36.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5e057700 1 --2- 172.21.15.1:0/124841865 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f580cfab0 0x7f3f5815c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5e057700 1 -- 172.21.15.1:0/124841865 >> v2:172.21.15.165:3300/0 conn(0x7f3f58140b20 msgr2=0x7f3f580cfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:36.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5e057700 1 --2- 172.21.15.1:0/124841865 >> v2:172.21.15.165:3300/0 conn(0x7f3f58140b20 0x7f3f580cfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5e057700 1 -- 172.21.15.1:0/124841865 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5815f060 con 0x7f3f580cdf80 2024-03-18T20:23:36.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5d856700 1 --2- 172.21.15.1:0/124841865 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f580cfab0 0x7f3f5815c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:23:36.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5d055700 1 --2- 172.21.15.1:0/124841865 >> v2:172.21.15.165:3300/0 conn(0x7f3f58140b20 0x7f3f580cfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:23:36.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5e057700 1 --2- 172.21.15.1:0/124841865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f580cdf80 0x7f3f58070db0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f3f580d0730 tx=0x7f3f5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ba521e5fe212ba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:36.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.264+0000 7f3f5c854700 1 -- 172.21.15.1:0/124841865 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5000e070 con 0x7f3f580cdf80 2024-03-18T20:23:36.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.265+0000 7f3f5c854700 1 -- 172.21.15.1:0/124841865 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5000b9c0 con 0x7f3f580cdf80 2024-03-18T20:23:36.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.265+0000 7f3f5c854700 1 -- 172.21.15.1:0/124841865 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5000bd70 con 0x7f3f580cdf80 2024-03-18T20:23:36.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.265+0000 7f3f5faba700 1 -- 172.21.15.1:0/124841865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f580cdf80 msgr2=0x7f3f58070db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:36.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.265+0000 7f3f5faba700 1 --2- 172.21.15.1:0/124841865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f580cdf80 0x7f3f58070db0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f3f580d0730 tx=0x7f3f5000b330 comp rx=0 tx=0).stop 2024-03-18T20:23:36.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.265+0000 7f3f5faba700 1 -- 172.21.15.1:0/124841865 shutdown_connections 2024-03-18T20:23:36.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.265+0000 7f3f5faba700 1 --2- 172.21.15.1:0/124841865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f580cdf80 0x7f3f58070db0 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.265+0000 7f3f5faba700 1 --2- 172.21.15.1:0/124841865 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f580cfab0 0x7f3f5815c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.265+0000 7f3f5faba700 1 --2- 172.21.15.1:0/124841865 >> v2:172.21.15.165:3300/0 conn(0x7f3f58140b20 0x7f3f580cfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.265+0000 7f3f5faba700 1 -- 172.21.15.1:0/124841865 >> 172.21.15.1:0/124841865 conn(0x7f3f5806ab70 msgr2=0x7f3f5806af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:36.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.265+0000 7f3f5faba700 1 -- 172.21.15.1:0/124841865 shutdown_connections 2024-03-18T20:23:36.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.265+0000 7f3f5faba700 1 -- 172.21.15.1:0/124841865 wait complete. 2024-03-18T20:23:36.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.267+0000 7f3f5faba700 1 Processor -- start 2024-03-18T20:23:36.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.267+0000 7f3f5faba700 1 -- start start 2024-03-18T20:23:36.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.268+0000 7f3f5faba700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f580cdf80 0x7f3f581544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:36.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.268+0000 7f3f5faba700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f580cfab0 0x7f3f5815c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:36.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.268+0000 7f3f5faba700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f581400f0 0x7f3f5815cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:36.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.268+0000 7f3f5faba700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5813fc60 con 0x7f3f581400f0 2024-03-18T20:23:36.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.268+0000 7f3f5faba700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5813f650 con 0x7f3f580cdf80 2024-03-18T20:23:36.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.268+0000 7f3f5faba700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f58154e40 con 0x7f3f580cfab0 2024-03-18T20:23:36.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.268+0000 7f3f5e057700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f581400f0 0x7f3f5815cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:36.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.268+0000 7f3f5e057700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f581400f0 0x7f3f5815cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46406/0 (socket says 172.21.15.1:46406) 2024-03-18T20:23:36.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.268+0000 7f3f5e057700 1 -- 172.21.15.1:0/1143183847 learned_addr learned my addr 172.21.15.1:0/1143183847 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:36.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.268+0000 7f3f5d856700 1 --2- 172.21.15.1:0/1143183847 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f580cdf80 0x7f3f581544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:36.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.268+0000 7f3f5d055700 1 --2- 172.21.15.1:0/1143183847 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f580cfab0 0x7f3f5815c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:36.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.268+0000 7f3f5e057700 1 -- 172.21.15.1:0/1143183847 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f580cfab0 msgr2=0x7f3f5815c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:36.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.268+0000 7f3f5e057700 1 --2- 172.21.15.1:0/1143183847 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f580cfab0 0x7f3f5815c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.268+0000 7f3f5e057700 1 -- 172.21.15.1:0/1143183847 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f580cdf80 msgr2=0x7f3f581544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:36.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.268+0000 7f3f5e057700 1 --2- 172.21.15.1:0/1143183847 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f580cdf80 0x7f3f581544d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.268+0000 7f3f5e057700 1 -- 172.21.15.1:0/1143183847 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f50009b30 con 0x7f3f581400f0 2024-03-18T20:23:36.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.268+0000 7f3f5d055700 1 --2- 172.21.15.1:0/1143183847 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f580cfab0 0x7f3f5815c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:36.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.268+0000 7f3f5e057700 1 --2- 172.21.15.1:0/1143183847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f581400f0 0x7f3f5815cce0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f3f50012000 tx=0x7f3f50004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:36.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.268+0000 7f3f4e7fc700 1 -- 172.21.15.1:0/1143183847 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f50004250 con 0x7f3f581400f0 2024-03-18T20:23:36.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.269+0000 7f3f4e7fc700 1 -- 172.21.15.1:0/1143183847 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f500043d0 con 0x7f3f581400f0 2024-03-18T20:23:36.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.269+0000 7f3f4e7fc700 1 -- 172.21.15.1:0/1143183847 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f50004780 con 0x7f3f581400f0 2024-03-18T20:23:36.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.269+0000 7f3f5faba700 1 -- 172.21.15.1:0/1143183847 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f5815d670 con 0x7f3f581400f0 2024-03-18T20:23:36.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.269+0000 7f3f5faba700 1 -- 172.21.15.1:0/1143183847 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f5815d950 con 0x7f3f581400f0 2024-03-18T20:23:36.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.270+0000 7f3f5faba700 1 -- 172.21.15.1:0/1143183847 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f5815eca0 con 0x7f3f581400f0 2024-03-18T20:23:36.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.270+0000 7f3f4e7fc700 1 -- 172.21.15.1:0/1143183847 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3f5001c440 con 0x7f3f581400f0 2024-03-18T20:23:36.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.270+0000 7f3f4e7fc700 1 --2- 172.21.15.1:0/1143183847 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3f3c06adb0 0x7f3f3c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:36.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.271+0000 7f3f4e7fc700 1 -- 172.21.15.1:0/1143183847 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f3f5008a8c0 con 0x7f3f581400f0 2024-03-18T20:23:36.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.271+0000 7f3f5d856700 1 --2- 172.21.15.1:0/1143183847 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3f3c06adb0 0x7f3f3c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:36.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.271+0000 7f3f5d856700 1 --2- 172.21.15.1:0/1143183847 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3f3c06adb0 0x7f3f3c06d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3f540038f0 tx=0x7f3f54008040 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:36.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.275+0000 7f3f4e7fc700 1 -- 172.21.15.1:0/1143183847 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3f5815eca0 con 0x7f3f581400f0 2024-03-18T20:23:36.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.479+0000 7f3f5faba700 1 -- 172.21.15.1:0/1143183847 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f5815eca0 con 0x7f3f581400f0 2024-03-18T20:23:36.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.479+0000 7f3f4e7fc700 1 -- 172.21.15.1:0/1143183847 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v210) v1 ==== 74+0+12717 (secure 0 0 0) 0x7f3f5815eca0 con 0x7f3f581400f0 2024-03-18T20:23:36.484 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:36.484 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":210,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:23:35.666991+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:23:36.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.482+0000 7f3f5faba700 1 -- 172.21.15.1:0/1143183847 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3f3c06adb0 msgr2=0x7f3f3c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:36.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.482+0000 7f3f5faba700 1 --2- 172.21.15.1:0/1143183847 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3f3c06adb0 0x7f3f3c06d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3f540038f0 tx=0x7f3f54008040 comp rx=0 tx=0).stop 2024-03-18T20:23:36.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.482+0000 7f3f5faba700 1 -- 172.21.15.1:0/1143183847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f581400f0 msgr2=0x7f3f5815cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:36.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.482+0000 7f3f5faba700 1 --2- 172.21.15.1:0/1143183847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f581400f0 0x7f3f5815cce0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f3f50012000 tx=0x7f3f50004ad0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.482+0000 7f3f5faba700 1 -- 172.21.15.1:0/1143183847 shutdown_connections 2024-03-18T20:23:36.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.482+0000 7f3f5faba700 1 --2- 172.21.15.1:0/1143183847 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3f3c06adb0 0x7f3f3c06d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.482+0000 7f3f5faba700 1 --2- 172.21.15.1:0/1143183847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f581400f0 0x7f3f5815cce0 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.482+0000 7f3f5faba700 1 --2- 172.21.15.1:0/1143183847 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f580cdf80 0x7f3f581544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.482+0000 7f3f5faba700 1 --2- 172.21.15.1:0/1143183847 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3f580cfab0 0x7f3f5815c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.482+0000 7f3f5faba700 1 -- 172.21.15.1:0/1143183847 >> 172.21.15.1:0/1143183847 conn(0x7f3f5806ab70 msgr2=0x7f3f580cb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:36.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.482+0000 7f3f5faba700 1 -- 172.21.15.1:0/1143183847 shutdown_connections 2024-03-18T20:23:36.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.482+0000 7f3f5faba700 1 -- 172.21.15.1:0/1143183847 wait complete. 2024-03-18T20:23:36.499 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:23:36.499 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:23:36.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.623+0000 7fcb60f92700 1 Processor -- start 2024-03-18T20:23:36.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.623+0000 7fcb60f92700 1 -- start start 2024-03-18T20:23:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.624+0000 7fcb60f92700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcb5c06efb0 0x7fcb5c071da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.624+0000 7fcb60f92700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb5c0722e0 0x7fcb5c0cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.624+0000 7fcb60f92700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5c06da90 con 0x7fcb5c15d580 2024-03-18T20:23:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.624+0000 7fcb60f92700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb5c072770 con 0x7fcb5c06efb0 2024-03-18T20:23:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.624+0000 7fcb60f92700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c15d960 con 0x7fcb5c0722e0 2024-03-18T20:23:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.624+0000 7fcb59d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcb5c06efb0 0x7fcb5c071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.624+0000 7fcb59d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcb5c06efb0 0x7fcb5c071da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46420/0 (socket says 172.21.15.1:46420) 2024-03-18T20:23:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.624+0000 7fcb59d9b700 1 -- 172.21.15.1:0/2418119115 learned_addr learned my addr 172.21.15.1:0/2418119115 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.624+0000 7fcb5ad9d700 1 --2- 172.21.15.1:0/2418119115 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb5c0722e0 0x7fcb5c0cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.624+0000 7fcb59d9b700 1 -- 172.21.15.1:0/2418119115 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb5c0722e0 msgr2=0x7fcb5c0cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.624+0000 7fcb59d9b700 1 --2- 172.21.15.1:0/2418119115 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb5c0722e0 0x7fcb5c0cf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.624+0000 7fcb59d9b700 1 -- 172.21.15.1:0/2418119115 >> v1:172.21.15.165:6789/0 conn(0x7fcb5c15d580 legacy=0x7fcb5c06d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:36.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.624+0000 7fcb59d9b700 1 -- 172.21.15.1:0/2418119115 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5c06ab30 con 0x7fcb5c06efb0 2024-03-18T20:23:36.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.624+0000 7fcb59d9b700 1 --2- 172.21.15.1:0/2418119115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb5c06efb0 0x7fcb5c071da0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c009920 tx=0x7fcb4c009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f758fe7a0b4eb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:36.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.624+0000 7fcb5959a700 1 -- 172.21.15.1:0/2418119115 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c00ba20 con 0x7fcb5c06efb0 2024-03-18T20:23:36.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.624+0000 7fcb5959a700 1 -- 172.21.15.1:0/2418119115 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4c00bb80 con 0x7fcb5c06efb0 2024-03-18T20:23:36.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.624+0000 7fcb5959a700 1 -- 172.21.15.1:0/2418119115 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c009e70 con 0x7fcb5c06efb0 2024-03-18T20:23:36.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.625+0000 7fcb60f92700 1 -- 172.21.15.1:0/2418119115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb5c06efb0 msgr2=0x7fcb5c071da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:36.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.625+0000 7fcb60f92700 1 --2- 172.21.15.1:0/2418119115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb5c06efb0 0x7fcb5c071da0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c009920 tx=0x7fcb4c009c30 comp rx=0 tx=0).stop 2024-03-18T20:23:36.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.625+0000 7fcb60f92700 1 -- 172.21.15.1:0/2418119115 shutdown_connections 2024-03-18T20:23:36.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.625+0000 7fcb60f92700 1 --2- 172.21.15.1:0/2418119115 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb5c0722e0 0x7fcb5c0cf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.625+0000 7fcb60f92700 1 --2- 172.21.15.1:0/2418119115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb5c06efb0 0x7fcb5c071da0 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.625+0000 7fcb60f92700 1 -- 172.21.15.1:0/2418119115 >> 172.21.15.1:0/2418119115 conn(0x7fcb5c0cca00 msgr2=0x7fcb5c0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:36.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.625+0000 7fcb60f92700 1 -- 172.21.15.1:0/2418119115 shutdown_connections 2024-03-18T20:23:36.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.625+0000 7fcb60f92700 1 -- 172.21.15.1:0/2418119115 wait complete. 2024-03-18T20:23:36.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.626+0000 7fcb60f92700 1 Processor -- start 2024-03-18T20:23:36.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.626+0000 7fcb60f92700 1 -- start start 2024-03-18T20:23:36.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.627+0000 7fcb60f92700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb5c06efb0 0x7fcb5c1f3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:36.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.627+0000 7fcb60f92700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb5c0722e0 0x7fcb5c156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:36.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.627+0000 7fcb60f92700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb5c15d580 0x7fcb5c160000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:36.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.627+0000 7fcb60f92700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c15a1c0 con 0x7fcb5c06efb0 2024-03-18T20:23:36.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.627+0000 7fcb60f92700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c159890 con 0x7fcb5c15d580 2024-03-18T20:23:36.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.627+0000 7fcb60f92700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c159b70 con 0x7fcb5c0722e0 2024-03-18T20:23:36.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.627+0000 7fcb5a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb5c06efb0 0x7fcb5c1f3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:36.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.627+0000 7fcb5a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb5c06efb0 0x7fcb5c1f3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46432/0 (socket says 172.21.15.1:46432) 2024-03-18T20:23:36.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.627+0000 7fcb5a59c700 1 -- 172.21.15.1:0/3567966282 learned_addr learned my addr 172.21.15.1:0/3567966282 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:36.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.627+0000 7fcb59d9b700 1 --2- 172.21.15.1:0/3567966282 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb5c0722e0 0x7fcb5c156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:36.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.627+0000 7fcb5ad9d700 1 --2- 172.21.15.1:0/3567966282 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb5c15d580 0x7fcb5c160000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:36.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.628+0000 7fcb5a59c700 1 -- 172.21.15.1:0/3567966282 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb5c0722e0 msgr2=0x7fcb5c156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:36.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.628+0000 7fcb5a59c700 1 --2- 172.21.15.1:0/3567966282 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb5c0722e0 0x7fcb5c156ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.628+0000 7fcb5a59c700 1 -- 172.21.15.1:0/3567966282 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb5c15d580 msgr2=0x7fcb5c160000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:36.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.628+0000 7fcb5a59c700 1 --2- 172.21.15.1:0/3567966282 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb5c15d580 0x7fcb5c160000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.628+0000 7fcb5a59c700 1 -- 172.21.15.1:0/3567966282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c009550 con 0x7fcb5c06efb0 2024-03-18T20:23:36.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.628+0000 7fcb59d9b700 1 --2- 172.21.15.1:0/3567966282 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb5c0722e0 0x7fcb5c156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:36.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.628+0000 7fcb5a59c700 1 --2- 172.21.15.1:0/3567966282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb5c06efb0 0x7fcb5c1f3340 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000c4e0 tx=0x7fcb5000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:36.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.628+0000 7fcb4affd700 1 -- 172.21.15.1:0/3567966282 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb500052d0 con 0x7fcb5c06efb0 2024-03-18T20:23:36.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.628+0000 7fcb4affd700 1 -- 172.21.15.1:0/3567966282 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb50005430 con 0x7fcb5c06efb0 2024-03-18T20:23:36.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.628+0000 7fcb60f92700 1 -- 172.21.15.1:0/3567966282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5c159e50 con 0x7fcb5c06efb0 2024-03-18T20:23:36.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.628+0000 7fcb4affd700 1 -- 172.21.15.1:0/3567966282 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb500038c0 con 0x7fcb5c06efb0 2024-03-18T20:23:36.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.628+0000 7fcb60f92700 1 -- 172.21.15.1:0/3567966282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb5c1f3880 con 0x7fcb5c06efb0 2024-03-18T20:23:36.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.629+0000 7fcb60f92700 1 -- 172.21.15.1:0/3567966282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb5c0ceea0 con 0x7fcb5c06efb0 2024-03-18T20:23:36.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.629+0000 7fcb4affd700 1 -- 172.21.15.1:0/3567966282 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcb50003a20 con 0x7fcb5c06efb0 2024-03-18T20:23:36.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.630+0000 7fcb4affd700 1 --2- 172.21.15.1:0/3567966282 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcb3c06adb0 0x7fcb3c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:36.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.630+0000 7fcb4affd700 1 -- 172.21.15.1:0/3567966282 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5277+0+0 (secure 0 0 0) 0x7fcb50088710 con 0x7fcb5c06efb0 2024-03-18T20:23:36.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.630+0000 7fcb59d9b700 1 --2- 172.21.15.1:0/3567966282 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcb3c06adb0 0x7fcb3c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:36.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.630+0000 7fcb59d9b700 1 --2- 172.21.15.1:0/3567966282 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcb3c06adb0 0x7fcb3c06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c0b0530 tx=0x7fcb4c008030 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:36.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.634+0000 7fcb4affd700 1 -- 172.21.15.1:0/3567966282 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcb50053a20 con 0x7fcb5c06efb0 2024-03-18T20:23:36.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.838+0000 7fcb60f92700 1 -- 172.21.15.1:0/3567966282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcb5c070e00 con 0x7fcb5c06efb0 2024-03-18T20:23:36.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.838+0000 7fcb4affd700 1 -- 172.21.15.1:0/3567966282 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v210) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcb50017020 con 0x7fcb5c06efb0 2024-03-18T20:23:36.844 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:23:36.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.840+0000 7fcb60f92700 1 -- 172.21.15.1:0/3567966282 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcb3c06adb0 msgr2=0x7fcb3c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.840+0000 7fcb60f92700 1 --2- 172.21.15.1:0/3567966282 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcb3c06adb0 0x7fcb3c06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c0b0530 tx=0x7fcb4c008030 comp rx=0 tx=0).stop 2024-03-18T20:23:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.841+0000 7fcb60f92700 1 -- 172.21.15.1:0/3567966282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb5c06efb0 msgr2=0x7fcb5c1f3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.841+0000 7fcb60f92700 1 --2- 172.21.15.1:0/3567966282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb5c06efb0 0x7fcb5c1f3340 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000c4e0 tx=0x7fcb5000c8a0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.841+0000 7fcb60f92700 1 -- 172.21.15.1:0/3567966282 shutdown_connections 2024-03-18T20:23:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.841+0000 7fcb60f92700 1 --2- 172.21.15.1:0/3567966282 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcb3c06adb0 0x7fcb3c06d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.841+0000 7fcb60f92700 1 --2- 172.21.15.1:0/3567966282 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb5c15d580 0x7fcb5c160000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.841+0000 7fcb60f92700 1 --2- 172.21.15.1:0/3567966282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb5c06efb0 0x7fcb5c1f3340 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.841+0000 7fcb60f92700 1 --2- 172.21.15.1:0/3567966282 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcb5c0722e0 0x7fcb5c156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.841+0000 7fcb60f92700 1 -- 172.21.15.1:0/3567966282 >> 172.21.15.1:0/3567966282 conn(0x7fcb5c0cca00 msgr2=0x7fcb5c0cf280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.841+0000 7fcb60f92700 1 -- 172.21.15.1:0/3567966282 shutdown_connections 2024-03-18T20:23:36.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.841+0000 7fcb60f92700 1 -- 172.21.15.1:0/3567966282 wait complete. 2024-03-18T20:23:36.859 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T20:23:36.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.982+0000 7f880627a700 1 Processor -- start 2024-03-18T20:23:36.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.982+0000 7f880627a700 1 -- start start 2024-03-18T20:23:36.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.982+0000 7f880627a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88000cf0c0 0x7f880015c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:36.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.982+0000 7f880627a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f880006cd90 0x7f880006d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:36.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.982+0000 7f880627a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f880015c8b0 con 0x7f88000aebe0 2024-03-18T20:23:36.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.982+0000 7f880627a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f880006d760 con 0x7f88000cf0c0 2024-03-18T20:23:36.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.982+0000 7f880627a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88000aefc0 con 0x7f880006cd90 2024-03-18T20:23:36.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.982+0000 7f87ff7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88000cf0c0 0x7f880015c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:36.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.983+0000 7f87ff7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88000cf0c0 0x7f880015c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46444/0 (socket says 172.21.15.1:46444) 2024-03-18T20:23:36.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.983+0000 7f87ff7fe700 1 -- 172.21.15.1:0/3447444327 learned_addr learned my addr 172.21.15.1:0/3447444327 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:36.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.983+0000 7f8804817700 1 --2- 172.21.15.1:0/3447444327 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f880006cd90 0x7f880006d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:36.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.983+0000 7f87ff7fe700 1 -- 172.21.15.1:0/3447444327 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f880006cd90 msgr2=0x7f880006d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:36.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.983+0000 7f87ff7fe700 1 --2- 172.21.15.1:0/3447444327 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f880006cd90 0x7f880006d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.983+0000 7f87ff7fe700 1 -- 172.21.15.1:0/3447444327 >> v1:172.21.15.165:6789/0 conn(0x7f88000aebe0 legacy=0x7f88000cc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:36.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.983+0000 7f87ff7fe700 1 -- 172.21.15.1:0/3447444327 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88000cd740 con 0x7f88000cf0c0 2024-03-18T20:23:36.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.983+0000 7f87ff7fe700 1 --2- 172.21.15.1:0/3447444327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88000cf0c0 0x7f880015c370 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f87f40098f0 tx=0x7f87f4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f5728d329346a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:36.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.983+0000 7f87feffd700 1 -- 172.21.15.1:0/3447444327 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f400b8a0 con 0x7f88000cf0c0 2024-03-18T20:23:36.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.983+0000 7f87feffd700 1 -- 172.21.15.1:0/3447444327 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f400ba00 con 0x7f88000cf0c0 2024-03-18T20:23:36.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.983+0000 7f87feffd700 1 -- 172.21.15.1:0/3447444327 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f400bd90 con 0x7f88000cf0c0 2024-03-18T20:23:36.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.984+0000 7f880627a700 1 -- 172.21.15.1:0/3447444327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88000cf0c0 msgr2=0x7f880015c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:36.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.984+0000 7f880627a700 1 --2- 172.21.15.1:0/3447444327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88000cf0c0 0x7f880015c370 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f87f40098f0 tx=0x7f87f4009c00 comp rx=0 tx=0).stop 2024-03-18T20:23:36.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.984+0000 7f880627a700 1 -- 172.21.15.1:0/3447444327 shutdown_connections 2024-03-18T20:23:36.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.984+0000 7f880627a700 1 --2- 172.21.15.1:0/3447444327 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f880006cd90 0x7f880006d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.984+0000 7f880627a700 1 --2- 172.21.15.1:0/3447444327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88000cf0c0 0x7f880015c370 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.984+0000 7f880627a700 1 -- 172.21.15.1:0/3447444327 >> 172.21.15.1:0/3447444327 conn(0x7f88000b1270 msgr2=0x7f88000b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:36.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.984+0000 7f880627a700 1 -- 172.21.15.1:0/3447444327 shutdown_connections 2024-03-18T20:23:36.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.984+0000 7f880627a700 1 -- 172.21.15.1:0/3447444327 wait complete. 2024-03-18T20:23:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.985+0000 7f880627a700 1 Processor -- start 2024-03-18T20:23:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.986+0000 7f880627a700 1 -- start start 2024-03-18T20:23:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.986+0000 7f880627a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f880006cd90 0x7f880015db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.986+0000 7f880627a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88000aebe0 0x7f88000c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.986+0000 7f880627a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88000cf0c0 0x7f8800153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.986+0000 7f880627a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880015e0c0 con 0x7f88000cf0c0 2024-03-18T20:23:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.986+0000 7f880627a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88000c8b20 con 0x7f880006cd90 2024-03-18T20:23:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.986+0000 7f880627a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88000c8de0 con 0x7f88000aebe0 2024-03-18T20:23:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.986+0000 7f8804817700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88000cf0c0 0x7f8800153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.987+0000 7f8804817700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88000cf0c0 0x7f8800153820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46456/0 (socket says 172.21.15.1:46456) 2024-03-18T20:23:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.987+0000 7f8804817700 1 -- 172.21.15.1:0/2051298065 learned_addr learned my addr 172.21.15.1:0/2051298065 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.987+0000 7f87fffff700 1 --2- 172.21.15.1:0/2051298065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f880006cd90 0x7f880015db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.987+0000 7f8804817700 1 -- 172.21.15.1:0/2051298065 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88000aebe0 msgr2=0x7f88000c85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.987+0000 7f87ff7fe700 1 --2- 172.21.15.1:0/2051298065 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88000aebe0 0x7f88000c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.987+0000 7f8804817700 1 --2- 172.21.15.1:0/2051298065 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88000aebe0 0x7f88000c85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.987+0000 7f8804817700 1 -- 172.21.15.1:0/2051298065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f880006cd90 msgr2=0x7f880015db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.987+0000 7f8804817700 1 --2- 172.21.15.1:0/2051298065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f880006cd90 0x7f880015db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.987+0000 7f8804817700 1 -- 172.21.15.1:0/2051298065 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f4009550 con 0x7f88000cf0c0 2024-03-18T20:23:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.987+0000 7f87fffff700 1 --2- 172.21.15.1:0/2051298065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f880006cd90 0x7f880015db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.987+0000 7f8804817700 1 --2- 172.21.15.1:0/2051298065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88000cf0c0 0x7f8800153820 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f87f8007e40 tx=0x7f87f800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.987+0000 7f87fcff9700 1 -- 172.21.15.1:0/2051298065 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f800eea0 con 0x7f88000cf0c0 2024-03-18T20:23:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.987+0000 7f87fcff9700 1 -- 172.21.15.1:0/2051298065 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f8014420 con 0x7f88000cf0c0 2024-03-18T20:23:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.987+0000 7f880627a700 1 -- 172.21.15.1:0/2051298065 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88000c90c0 con 0x7f88000cf0c0 2024-03-18T20:23:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.987+0000 7f87fcff9700 1 -- 172.21.15.1:0/2051298065 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f80147b0 con 0x7f88000cf0c0 2024-03-18T20:23:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.987+0000 7f880627a700 1 -- 172.21.15.1:0/2051298065 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88000c9680 con 0x7f88000cf0c0 2024-03-18T20:23:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.988+0000 7f880627a700 1 -- 172.21.15.1:0/2051298065 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88000676a0 con 0x7f88000cf0c0 2024-03-18T20:23:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.989+0000 7f87fcff9700 1 -- 172.21.15.1:0/2051298065 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f87f8014910 con 0x7f88000cf0c0 2024-03-18T20:23:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.989+0000 7f87fcff9700 1 --2- 172.21.15.1:0/2051298065 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f87ec06adb0 0x7f87ec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.989+0000 7f87fcff9700 1 -- 172.21.15.1:0/2051298065 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f87f8089b00 con 0x7f88000cf0c0 2024-03-18T20:23:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.989+0000 7f87fffff700 1 --2- 172.21.15.1:0/2051298065 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f87ec06adb0 0x7f87ec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:36.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.989+0000 7f87fffff700 1 --2- 172.21.15.1:0/2051298065 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f87ec06adb0 0x7f87ec06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f88000b1ee0 tx=0x7f87f0009040 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:36.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:36.994+0000 7f87fcff9700 1 -- 172.21.15.1:0/2051298065 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f87f8054d80 con 0x7f88000cf0c0 2024-03-18T20:23:37.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.199+0000 7f880627a700 1 -- 172.21.15.1:0/2051298065 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f88000b16f0 con 0x7f88000cf0c0 2024-03-18T20:23:37.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.200+0000 7f87fcff9700 1 -- 172.21.15.1:0/2051298065 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v210) v1 ==== 84+0+65 (secure 0 0 0) 0x7f87f80583a0 con 0x7f88000cf0c0 2024-03-18T20:23:37.204 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:37.204 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T20:23:37.204 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:37.204 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-18T20:23:37.204 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-18T20:23:37.205 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:37.205 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T20:23:37.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.202+0000 7f880627a700 1 -- 172.21.15.1:0/2051298065 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f87ec06adb0 msgr2=0x7f87ec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:37.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.202+0000 7f880627a700 1 --2- 172.21.15.1:0/2051298065 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f87ec06adb0 0x7f87ec06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f88000b1ee0 tx=0x7f87f0009040 comp rx=0 tx=0).stop 2024-03-18T20:23:37.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.202+0000 7f880627a700 1 -- 172.21.15.1:0/2051298065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88000cf0c0 msgr2=0x7f8800153820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:37.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.202+0000 7f880627a700 1 --2- 172.21.15.1:0/2051298065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88000cf0c0 0x7f8800153820 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f87f8007e40 tx=0x7f87f800e3c0 comp rx=0 tx=0).stop 2024-03-18T20:23:37.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.202+0000 7f880627a700 1 -- 172.21.15.1:0/2051298065 shutdown_connections 2024-03-18T20:23:37.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.202+0000 7f880627a700 1 --2- 172.21.15.1:0/2051298065 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f87ec06adb0 0x7f87ec06d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:37.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.202+0000 7f880627a700 1 --2- 172.21.15.1:0/2051298065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88000cf0c0 0x7f8800153820 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:37.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.202+0000 7f880627a700 1 --2- 172.21.15.1:0/2051298065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f880006cd90 0x7f880015db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:37.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.202+0000 7f880627a700 1 --2- 172.21.15.1:0/2051298065 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88000aebe0 0x7f88000c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:37.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.202+0000 7f880627a700 1 -- 172.21.15.1:0/2051298065 >> 172.21.15.1:0/2051298065 conn(0x7f88000b1270 msgr2=0x7f88000cc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:37.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.202+0000 7f880627a700 1 -- 172.21.15.1:0/2051298065 shutdown_connections 2024-03-18T20:23:37.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.202+0000 7f880627a700 1 -- 172.21.15.1:0/2051298065 wait complete. 2024-03-18T20:23:37.219 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T20:23:37.219 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T20:23:37.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.345+0000 7f8e2439f700 1 Processor -- start 2024-03-18T20:23:37.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.345+0000 7f8e2439f700 1 -- start start 2024-03-18T20:23:37.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.346+0000 7f8e2439f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8e1c15d590 0x7f8e1c0b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:37.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.346+0000 7f8e2439f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8e1c06fe10 0x7f8e1c06b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:37.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.346+0000 7f8e2439f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c15d970 con 0x7f8e1c06a750 2024-03-18T20:23:37.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.346+0000 7f8e2439f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e1c072320 con 0x7f8e1c15d590 2024-03-18T20:23:37.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.346+0000 7f8e2439f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e1c072600 con 0x7f8e1c06fe10 2024-03-18T20:23:37.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.346+0000 7f8e2213b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8e1c15d590 0x7f8e1c0b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:37.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.346+0000 7f8e2193a700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8e1c06a750 0x7f8e1c06abd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45472/0 (socket says 172.21.15.1:45472) 2024-03-18T20:23:37.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.346+0000 7f8e2193a700 1 -- 172.21.15.1:0/3484251117 learned_addr learned my addr 172.21.15.1:0/3484251117 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:37.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.346+0000 7f8e2293c700 1 --2- 172.21.15.1:0/3484251117 >> v2:172.21.15.165:3300/0 conn(0x7f8e1c06fe10 0x7f8e1c06b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:37.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.346+0000 7f8e2213b700 1 -- 172.21.15.1:0/3484251117 >> v2:172.21.15.165:3300/0 conn(0x7f8e1c06fe10 msgr2=0x7f8e1c06b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:37.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.346+0000 7f8e2213b700 1 --2- 172.21.15.1:0/3484251117 >> v2:172.21.15.165:3300/0 conn(0x7f8e1c06fe10 0x7f8e1c06b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:37.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.346+0000 7f8e2213b700 1 -- 172.21.15.1:0/3484251117 >> v1:172.21.15.1:6789/0 conn(0x7f8e1c06a750 legacy=0x7f8e1c06abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:37.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.346+0000 7f8e2213b700 1 -- 172.21.15.1:0/3484251117 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1c0b30b0 con 0x7f8e1c15d590 2024-03-18T20:23:37.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.346+0000 7f8e21139700 1 -- 172.21.15.1:0/3484251117 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503229575 0 0) 0x7f8e1c15d970 con 0x7f8e1c06a750 2024-03-18T20:23:37.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.346+0000 7f8e2213b700 1 --2- 172.21.15.1:0/3484251117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e1c15d590 0x7f8e1c0b3bc0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c0b6350 tx=0x7f8e10009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f9465c67136d59 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:37.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.347+0000 7f8e21139700 1 -- 172.21.15.1:0/3484251117 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e10011830 con 0x7f8e1c15d590 2024-03-18T20:23:37.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.347+0000 7f8e21139700 1 -- 172.21.15.1:0/3484251117 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e10011990 con 0x7f8e1c15d590 2024-03-18T20:23:37.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.347+0000 7f8e21139700 1 -- 172.21.15.1:0/3484251117 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e10011d20 con 0x7f8e1c15d590 2024-03-18T20:23:37.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.347+0000 7f8e2439f700 1 -- 172.21.15.1:0/3484251117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e1c15d590 msgr2=0x7f8e1c0b3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:37.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.347+0000 7f8e2439f700 1 --2- 172.21.15.1:0/3484251117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e1c15d590 0x7f8e1c0b3bc0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c0b6350 tx=0x7f8e10009b20 comp rx=0 tx=0).stop 2024-03-18T20:23:37.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.347+0000 7f8e2439f700 1 -- 172.21.15.1:0/3484251117 shutdown_connections 2024-03-18T20:23:37.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.347+0000 7f8e2439f700 1 --2- 172.21.15.1:0/3484251117 >> v2:172.21.15.165:3300/0 conn(0x7f8e1c06fe10 0x7f8e1c06b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:37.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.347+0000 7f8e2439f700 1 --2- 172.21.15.1:0/3484251117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e1c15d590 0x7f8e1c0b3bc0 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:37.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.347+0000 7f8e2439f700 1 -- 172.21.15.1:0/3484251117 >> 172.21.15.1:0/3484251117 conn(0x7f8e1c0d00c0 msgr2=0x7f8e1c0d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:37.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.347+0000 7f8e2439f700 1 -- 172.21.15.1:0/3484251117 shutdown_connections 2024-03-18T20:23:37.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.347+0000 7f8e2439f700 1 -- 172.21.15.1:0/3484251117 wait complete. 2024-03-18T20:23:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.348+0000 7f8e2439f700 1 Processor -- start 2024-03-18T20:23:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.349+0000 7f8e2439f700 1 -- start start 2024-03-18T20:23:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.349+0000 7f8e2439f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e1c06a750 0x7f8e1c147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.349+0000 7f8e2439f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e1c06fe10 0x7f8e1c155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.349+0000 7f8e2439f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e1c15d590 0x7f8e1c14a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.349+0000 7f8e2439f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c153d10 con 0x7f8e1c15d590 2024-03-18T20:23:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.349+0000 7f8e2439f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c148110 con 0x7f8e1c06fe10 2024-03-18T20:23:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.349+0000 7f8e2439f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c153750 con 0x7f8e1c06a750 2024-03-18T20:23:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.349+0000 7f8e2293c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e1c15d590 0x7f8e1c14a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.349+0000 7f8e2293c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e1c15d590 0x7f8e1c14a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46470/0 (socket says 172.21.15.1:46470) 2024-03-18T20:23:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.349+0000 7f8e2293c700 1 -- 172.21.15.1:0/720604216 learned_addr learned my addr 172.21.15.1:0/720604216 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.349+0000 7f8e2193a700 1 --2- 172.21.15.1:0/720604216 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e1c06fe10 0x7f8e1c155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.350+0000 7f8e2213b700 1 --2- 172.21.15.1:0/720604216 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e1c06a750 0x7f8e1c147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.350+0000 7f8e2293c700 1 -- 172.21.15.1:0/720604216 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e1c06a750 msgr2=0x7f8e1c147bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.350+0000 7f8e2293c700 1 --2- 172.21.15.1:0/720604216 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e1c06a750 0x7f8e1c147bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.350+0000 7f8e2293c700 1 -- 172.21.15.1:0/720604216 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e1c06fe10 msgr2=0x7f8e1c155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.350+0000 7f8e2293c700 1 --2- 172.21.15.1:0/720604216 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e1c06fe10 0x7f8e1c155130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.350+0000 7f8e2293c700 1 -- 172.21.15.1:0/720604216 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e100094a0 con 0x7f8e1c15d590 2024-03-18T20:23:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.350+0000 7f8e2213b700 1 --2- 172.21.15.1:0/720604216 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e1c06a750 0x7f8e1c147bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:23:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.350+0000 7f8e2293c700 1 --2- 172.21.15.1:0/720604216 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e1c15d590 0x7f8e1c14a8d0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f8e14007e40 tx=0x7f8e1400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.350+0000 7f8e0effd700 1 -- 172.21.15.1:0/720604216 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1400ede0 con 0x7f8e1c15d590 2024-03-18T20:23:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.350+0000 7f8e0effd700 1 -- 172.21.15.1:0/720604216 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e14014420 con 0x7f8e1c15d590 2024-03-18T20:23:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.350+0000 7f8e0effd700 1 -- 172.21.15.1:0/720604216 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e140147d0 con 0x7f8e1c15d590 2024-03-18T20:23:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.350+0000 7f8e2439f700 1 -- 172.21.15.1:0/720604216 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1c153a30 con 0x7f8e1c15d590 2024-03-18T20:23:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.350+0000 7f8e2439f700 1 -- 172.21.15.1:0/720604216 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e1c1487e0 con 0x7f8e1c15d590 2024-03-18T20:23:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.351+0000 7f8e2439f700 1 -- 172.21.15.1:0/720604216 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e1c148360 con 0x7f8e1c15d590 2024-03-18T20:23:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.351+0000 7f8e0effd700 1 -- 172.21.15.1:0/720604216 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f8e14014930 con 0x7f8e1c15d590 2024-03-18T20:23:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.352+0000 7f8e0effd700 1 --2- 172.21.15.1:0/720604216 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8e0006adb0 0x7f8e0006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.352+0000 7f8e0effd700 1 -- 172.21.15.1:0/720604216 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f8e140899f0 con 0x7f8e1c15d590 2024-03-18T20:23:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.352+0000 7f8e2213b700 1 --2- 172.21.15.1:0/720604216 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8e0006adb0 0x7f8e0006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.352+0000 7f8e2213b700 1 --2- 172.21.15.1:0/720604216 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8e0006adb0 0x7f8e0006d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8e100090b0 tx=0x7f8e10008030 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:37.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.356+0000 7f8e0effd700 1 -- 172.21.15.1:0/720604216 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8e1c148360 con 0x7f8e1c15d590 2024-03-18T20:23:37.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:37.557+0000 7f8e2439f700 1 -- 172.21.15.1:0/720604216 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8e1c148360 con 0x7f8e1c15d590 2024-03-18T20:23:38.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.312+0000 7f8e0effd700 1 -- 172.21.15.1:0/720604216 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v211) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8e1c148360 con 0x7f8e1c15d590 2024-03-18T20:23:38.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.317+0000 7f8e2439f700 1 -- 172.21.15.1:0/720604216 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8e0006adb0 msgr2=0x7f8e0006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:38.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.317+0000 7f8e2439f700 1 --2- 172.21.15.1:0/720604216 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8e0006adb0 0x7f8e0006d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8e100090b0 tx=0x7f8e10008030 comp rx=0 tx=0).stop 2024-03-18T20:23:38.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.317+0000 7f8e2439f700 1 -- 172.21.15.1:0/720604216 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e1c15d590 msgr2=0x7f8e1c14a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:38.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.317+0000 7f8e2439f700 1 --2- 172.21.15.1:0/720604216 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e1c15d590 0x7f8e1c14a8d0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f8e14007e40 tx=0x7f8e1400e3c0 comp rx=0 tx=0).stop 2024-03-18T20:23:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.317+0000 7f8e2439f700 1 -- 172.21.15.1:0/720604216 shutdown_connections 2024-03-18T20:23:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.317+0000 7f8e2439f700 1 --2- 172.21.15.1:0/720604216 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8e0006adb0 0x7f8e0006d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.317+0000 7f8e2439f700 1 --2- 172.21.15.1:0/720604216 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e1c06a750 0x7f8e1c147bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.317+0000 7f8e2439f700 1 --2- 172.21.15.1:0/720604216 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e1c15d590 0x7f8e1c14a8d0 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.317+0000 7f8e2439f700 1 --2- 172.21.15.1:0/720604216 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e1c06fe10 0x7f8e1c155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.317+0000 7f8e2439f700 1 -- 172.21.15.1:0/720604216 >> 172.21.15.1:0/720604216 conn(0x7f8e1c0d00c0 msgr2=0x7f8e1c0b3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.318+0000 7f8e2439f700 1 -- 172.21.15.1:0/720604216 shutdown_connections 2024-03-18T20:23:38.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.319+0000 7f8e2439f700 1 -- 172.21.15.1:0/720604216 wait complete. 2024-03-18T20:23:38.324 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-18T20:23:38.339 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T20:23:38.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.473+0000 7fa1b004e700 1 Processor -- start 2024-03-18T20:23:38.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.473+0000 7fa1b004e700 1 -- start start 2024-03-18T20:23:38.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.474+0000 7fa1b004e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa1a8144500 0x7fa1a80717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.474+0000 7fa1b004e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1a80cfdb0 0x7fa1a8141010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.474+0000 7fa1b004e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a8071d20 con 0x7fa1a80cf9d0 2024-03-18T20:23:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.474+0000 7fa1b004e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a815ed40 con 0x7fa1a8144500 2024-03-18T20:23:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.474+0000 7fa1b004e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a815ef90 con 0x7fa1a80cfdb0 2024-03-18T20:23:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.474+0000 7fa1ad5e9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa1a8144500 0x7fa1a80717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.474+0000 7fa1ae5eb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1a80cfdb0 0x7fa1a8141010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.474+0000 7fa1ad5e9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa1a8144500 0x7fa1a80717e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:42660/0 (socket says 172.21.15.1:42660) 2024-03-18T20:23:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.474+0000 7fa1ad5e9700 1 -- 172.21.15.1:0/3704403452 learned_addr learned my addr 172.21.15.1:0/3704403452 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.474+0000 7fa1addea700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fa1a80cf9d0 0x7fa1a80ce4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:39712/0 (socket says 172.21.15.1:39712) 2024-03-18T20:23:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.475+0000 7fa1ae5eb700 1 -- 172.21.15.1:0/3704403452 >> v2:172.21.15.165:3300/0 conn(0x7fa1a8144500 msgr2=0x7fa1a80717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.475+0000 7fa1ae5eb700 1 --2- 172.21.15.1:0/3704403452 >> v2:172.21.15.165:3300/0 conn(0x7fa1a8144500 0x7fa1a80717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:38.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.475+0000 7fa1ae5eb700 1 -- 172.21.15.1:0/3704403452 >> v1:172.21.15.165:6789/0 conn(0x7fa1a80cf9d0 legacy=0x7fa1a80ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:38.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.475+0000 7fa1ae5eb700 1 -- 172.21.15.1:0/3704403452 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a815f270 con 0x7fa1a80cfdb0 2024-03-18T20:23:38.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.475+0000 7fa1ad5e9700 1 --2- 172.21.15.1:0/3704403452 >> v2:172.21.15.165:3300/0 conn(0x7fa1a8144500 0x7fa1a80717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:23:38.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.475+0000 7fa1ae5eb700 1 --2- 172.21.15.1:0/3704403452 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1a80cfdb0 0x7fa1a8141010 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0009dc0 tx=0x7fa1a000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5471791179c250c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:38.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.475+0000 7fa1acde8700 1 -- 172.21.15.1:0/3704403452 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a000e070 con 0x7fa1a80cfdb0 2024-03-18T20:23:38.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.475+0000 7fa1acde8700 1 -- 172.21.15.1:0/3704403452 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a000bae0 con 0x7fa1a80cfdb0 2024-03-18T20:23:38.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.475+0000 7fa1acde8700 1 -- 172.21.15.1:0/3704403452 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a000be90 con 0x7fa1a80cfdb0 2024-03-18T20:23:38.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.476+0000 7fa1b004e700 1 -- 172.21.15.1:0/3704403452 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1a80cfdb0 msgr2=0x7fa1a8141010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:38.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.476+0000 7fa1b004e700 1 --2- 172.21.15.1:0/3704403452 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1a80cfdb0 0x7fa1a8141010 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0009dc0 tx=0x7fa1a000b330 comp rx=0 tx=0).stop 2024-03-18T20:23:38.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.476+0000 7fa1b004e700 1 -- 172.21.15.1:0/3704403452 shutdown_connections 2024-03-18T20:23:38.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.476+0000 7fa1b004e700 1 --2- 172.21.15.1:0/3704403452 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1a80cfdb0 0x7fa1a8141010 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:38.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.476+0000 7fa1b004e700 1 --2- 172.21.15.1:0/3704403452 >> v2:172.21.15.165:3300/0 conn(0x7fa1a8144500 0x7fa1a80717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:38.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.476+0000 7fa1b004e700 1 -- 172.21.15.1:0/3704403452 >> 172.21.15.1:0/3704403452 conn(0x7fa1a8066b70 msgr2=0x7fa1a806a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:38.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.476+0000 7fa1b004e700 1 -- 172.21.15.1:0/3704403452 shutdown_connections 2024-03-18T20:23:38.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.476+0000 7fa1b004e700 1 -- 172.21.15.1:0/3704403452 wait complete. 2024-03-18T20:23:38.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.477+0000 7fa1b004e700 1 Processor -- start 2024-03-18T20:23:38.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.477+0000 7fa1b004e700 1 -- start start 2024-03-18T20:23:38.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.478+0000 7fa1b004e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1a80cf9d0 0x7fa1a80b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:38.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.478+0000 7fa1b004e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1a80cfdb0 0x7fa1a813c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:38.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.478+0000 7fa1b004e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1a8144500 0x7fa1a815dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:38.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.478+0000 7fa1b004e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a80b2010 con 0x7fa1a80cf9d0 2024-03-18T20:23:38.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.478+0000 7fa1b004e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a815eb70 con 0x7fa1a8144500 2024-03-18T20:23:38.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.478+0000 7fa1b004e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a815e890 con 0x7fa1a80cfdb0 2024-03-18T20:23:38.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.478+0000 7fa1addea700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1a80cf9d0 0x7fa1a80b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:38.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.478+0000 7fa1addea700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1a80cf9d0 0x7fa1a80b1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46474/0 (socket says 172.21.15.1:46474) 2024-03-18T20:23:38.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.478+0000 7fa1addea700 1 -- 172.21.15.1:0/714682744 learned_addr learned my addr 172.21.15.1:0/714682744 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:38.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.478+0000 7fa1ad5e9700 1 --2- 172.21.15.1:0/714682744 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1a80cfdb0 0x7fa1a813c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:38.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.478+0000 7fa1ae5eb700 1 --2- 172.21.15.1:0/714682744 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1a8144500 0x7fa1a815dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:38.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.478+0000 7fa1addea700 1 -- 172.21.15.1:0/714682744 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1a80cfdb0 msgr2=0x7fa1a813c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:38.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.478+0000 7fa1addea700 1 --2- 172.21.15.1:0/714682744 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1a80cfdb0 0x7fa1a813c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:38.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.478+0000 7fa1addea700 1 -- 172.21.15.1:0/714682744 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1a8144500 msgr2=0x7fa1a815dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:38.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.478+0000 7fa1addea700 1 --2- 172.21.15.1:0/714682744 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1a8144500 0x7fa1a815dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:38.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.478+0000 7fa1addea700 1 -- 172.21.15.1:0/714682744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a4009550 con 0x7fa1a80cf9d0 2024-03-18T20:23:38.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.478+0000 7fa1addea700 1 --2- 172.21.15.1:0/714682744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1a80cf9d0 0x7fa1a80b1ad0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7fa19800c550 tx=0x7fa19800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:38.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.479+0000 7fa19e7fc700 1 -- 172.21.15.1:0/714682744 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa198005cb0 con 0x7fa1a80cf9d0 2024-03-18T20:23:38.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.479+0000 7fa19e7fc700 1 -- 172.21.15.1:0/714682744 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa198005e10 con 0x7fa1a80cf9d0 2024-03-18T20:23:38.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.479+0000 7fa1b004e700 1 -- 172.21.15.1:0/714682744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a0009a70 con 0x7fa1a80cf9d0 2024-03-18T20:23:38.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.479+0000 7fa19e7fc700 1 -- 172.21.15.1:0/714682744 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1980072d0 con 0x7fa1a80cf9d0 2024-03-18T20:23:38.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.479+0000 7fa1b004e700 1 -- 172.21.15.1:0/714682744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a80acca0 con 0x7fa1a80cf9d0 2024-03-18T20:23:38.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.480+0000 7fa1b004e700 1 -- 172.21.15.1:0/714682744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a80676a0 con 0x7fa1a80cf9d0 2024-03-18T20:23:38.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.480+0000 7fa19e7fc700 1 -- 172.21.15.1:0/714682744 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa198007430 con 0x7fa1a80cf9d0 2024-03-18T20:23:38.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.481+0000 7fa19e7fc700 1 --2- 172.21.15.1:0/714682744 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa18c06adb0 0x7fa18c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:38.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.481+0000 7fa19e7fc700 1 -- 172.21.15.1:0/714682744 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 5647+0+0 (secure 0 0 0) 0x7fa198088650 con 0x7fa1a80cf9d0 2024-03-18T20:23:38.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.481+0000 7fa1ad5e9700 1 --2- 172.21.15.1:0/714682744 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa18c06adb0 0x7fa18c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:38.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.481+0000 7fa1ad5e9700 1 --2- 172.21.15.1:0/714682744 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa18c06adb0 0x7fa18c06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa1a81f72e0 tx=0x7fa1a4008040 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:38.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.486+0000 7fa19e7fc700 1 -- 172.21.15.1:0/714682744 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa198053810 con 0x7fa1a80cf9d0 2024-03-18T20:23:38.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:38.689+0000 7fa1b004e700 1 -- 172.21.15.1:0/714682744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa1a8071d20 con 0x7fa1a80cf9d0 2024-03-18T20:23:39.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.315+0000 7fa19e7fc700 1 -- 172.21.15.1:0/714682744 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v212) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa1a8071d20 con 0x7fa1a80cf9d0 2024-03-18T20:23:39.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.324+0000 7fa1b004e700 1 -- 172.21.15.1:0/714682744 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa18c06adb0 msgr2=0x7fa18c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:39.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.324+0000 7fa1b004e700 1 --2- 172.21.15.1:0/714682744 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa18c06adb0 0x7fa18c06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa1a81f72e0 tx=0x7fa1a4008040 comp rx=0 tx=0).stop 2024-03-18T20:23:39.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.324+0000 7fa1b004e700 1 -- 172.21.15.1:0/714682744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1a80cf9d0 msgr2=0x7fa1a80b1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:39.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.324+0000 7fa1b004e700 1 --2- 172.21.15.1:0/714682744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1a80cf9d0 0x7fa1a80b1ad0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7fa19800c550 tx=0x7fa19800c910 comp rx=0 tx=0).stop 2024-03-18T20:23:39.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.326+0000 7fa1b004e700 1 -- 172.21.15.1:0/714682744 shutdown_connections 2024-03-18T20:23:39.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.326+0000 7fa1b004e700 1 --2- 172.21.15.1:0/714682744 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa18c06adb0 0x7fa18c06d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.326+0000 7fa1b004e700 1 --2- 172.21.15.1:0/714682744 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1a8144500 0x7fa1a815dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.326+0000 7fa1b004e700 1 --2- 172.21.15.1:0/714682744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1a80cf9d0 0x7fa1a80b1ad0 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.326+0000 7fa1b004e700 1 --2- 172.21.15.1:0/714682744 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1a80cfdb0 0x7fa1a813c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.326+0000 7fa1b004e700 1 -- 172.21.15.1:0/714682744 >> 172.21.15.1:0/714682744 conn(0x7fa1a8066b70 msgr2=0x7fa1a806a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:39.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.330+0000 7fa1b004e700 1 -- 172.21.15.1:0/714682744 shutdown_connections 2024-03-18T20:23:39.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.331+0000 7fa1b004e700 1 -- 172.21.15.1:0/714682744 wait complete. 2024-03-18T20:23:39.339 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-18T20:23:39.355 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T20:23:39.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.484+0000 7ffa868e8700 1 Processor -- start 2024-03-18T20:23:39.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.484+0000 7ffa868e8700 1 -- start start 2024-03-18T20:23:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.485+0000 7ffa868e8700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ffa8015ad70 0x7ffa8015d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.485+0000 7ffa868e8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa8006a750 0x7ffa8006abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.485+0000 7ffa868e8700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa80160ea0 con 0x7ffa800727b0 2024-03-18T20:23:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.485+0000 7ffa868e8700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa800725b0 con 0x7ffa8015ad70 2024-03-18T20:23:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.485+0000 7ffa868e8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8006b100 con 0x7ffa8006a750 2024-03-18T20:23:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.485+0000 7ffa7f7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ffa800727b0 0x7ffa80161610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45476/0 (socket says 172.21.15.1:45476) 2024-03-18T20:23:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.485+0000 7ffa7f7fe700 1 -- 172.21.15.1:0/1050311816 learned_addr learned my addr 172.21.15.1:0/1050311816 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7ffff700 1 --2- 172.21.15.1:0/1050311816 >> v2:172.21.15.165:3300/0 conn(0x7ffa8015ad70 0x7ffa8015d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa84e85700 1 --2- 172.21.15.1:0/1050311816 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa8006a750 0x7ffa8006abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7effd700 1 -- 172.21.15.1:0/1050311816 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493490329 0 0) 0x7ffa80160ea0 con 0x7ffa800727b0 2024-03-18T20:23:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7effd700 1 -- 172.21.15.1:0/1050311816 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa6c0035b0 con 0x7ffa800727b0 2024-03-18T20:23:39.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7effd700 1 -- 172.21.15.1:0/1050311816 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 514350683 0 0) 0x7ffa6c0035b0 con 0x7ffa800727b0 2024-03-18T20:23:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7effd700 1 -- 172.21.15.1:0/1050311816 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa8006a750 msgr2=0x7ffa8006abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7effd700 1 --2- 172.21.15.1:0/1050311816 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa8006a750 0x7ffa8006abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7effd700 1 -- 172.21.15.1:0/1050311816 >> v2:172.21.15.165:3300/0 conn(0x7ffa8015ad70 msgr2=0x7ffa8015d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7effd700 1 --2- 172.21.15.1:0/1050311816 >> v2:172.21.15.165:3300/0 conn(0x7ffa8015ad70 0x7ffa8015d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7effd700 1 -- 172.21.15.1:0/1050311816 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa8015ddf0 con 0x7ffa800727b0 2024-03-18T20:23:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7effd700 1 -- 172.21.15.1:0/1050311816 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7ffa70001670 con 0x7ffa800727b0 2024-03-18T20:23:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7effd700 1 -- 172.21.15.1:0/1050311816 >> v1:172.21.15.1:6789/0 conn(0x7ffa800727b0 legacy=0x7ffa80161610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7effd700 1 --2- 172.21.15.1:0/1050311816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6c003a00 0x7ffa6c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7effd700 1 --2- 172.21.15.1:0/1050311816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa6c004850 0x7ffa6c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7effd700 1 --2- 172.21.15.1:0/1050311816 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa8006a750 0x7ffa6c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7effd700 1 -- 172.21.15.1:0/1050311816 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8006b100 con 0x7ffa6c003a00 2024-03-18T20:23:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7effd700 1 -- 172.21.15.1:0/1050311816 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c00ba10 con 0x7ffa6c004850 2024-03-18T20:23:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7effd700 1 -- 172.21.15.1:0/1050311816 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c00bb50 con 0x7ffa8006a750 2024-03-18T20:23:39.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7ffff700 1 --2- 172.21.15.1:0/1050311816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6c003a00 0x7ffa6c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7f7fe700 1 --2- 172.21.15.1:0/1050311816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa6c004850 0x7ffa6c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa84e85700 1 --2- 172.21.15.1:0/1050311816 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa8006a750 0x7ffa6c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7ffff700 1 -- 172.21.15.1:0/1050311816 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa8006a750 msgr2=0x7ffa6c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7ffff700 1 --2- 172.21.15.1:0/1050311816 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa8006a750 0x7ffa6c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7ffff700 1 -- 172.21.15.1:0/1050311816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa6c004850 msgr2=0x7ffa6c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7ffff700 1 --2- 172.21.15.1:0/1050311816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa6c004850 0x7ffa6c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.486+0000 7ffa7ffff700 1 -- 172.21.15.1:0/1050311816 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6c0035b0 con 0x7ffa6c003a00 2024-03-18T20:23:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.487+0000 7ffa7ffff700 1 --2- 172.21.15.1:0/1050311816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6c003a00 0x7ffa6c003e20 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7ffa68009b60 tx=0x7ffa6c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53018ceec99265d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.487+0000 7ffa7effd700 1 -- 172.21.15.1:0/1050311816 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6800e6c0 con 0x7ffa6c003a00 2024-03-18T20:23:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.487+0000 7ffa7effd700 1 -- 172.21.15.1:0/1050311816 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6800e820 con 0x7ffa6c003a00 2024-03-18T20:23:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.487+0000 7ffa7effd700 1 -- 172.21.15.1:0/1050311816 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6800eb90 con 0x7ffa6c003a00 2024-03-18T20:23:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.487+0000 7ffa868e8700 1 -- 172.21.15.1:0/1050311816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6c003a00 msgr2=0x7ffa6c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:39.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.487+0000 7ffa868e8700 1 --2- 172.21.15.1:0/1050311816 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6c003a00 0x7ffa6c003e20 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7ffa68009b60 tx=0x7ffa6c005880 comp rx=0 tx=0).stop 2024-03-18T20:23:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.487+0000 7ffa84e85700 1 -- 172.21.15.1:0/1050311816 reap_dead start 2024-03-18T20:23:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.487+0000 7ffa868e8700 1 -- 172.21.15.1:0/1050311816 shutdown_connections 2024-03-18T20:23:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.487+0000 7ffa868e8700 1 -- 172.21.15.1:0/1050311816 >> 172.21.15.1:0/1050311816 conn(0x7ffa800b1cb0 msgr2=0x7ffa800b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.487+0000 7ffa868e8700 1 -- 172.21.15.1:0/1050311816 shutdown_connections 2024-03-18T20:23:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.487+0000 7ffa868e8700 1 -- 172.21.15.1:0/1050311816 wait complete. 2024-03-18T20:23:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.489+0000 7ffa868e8700 1 Processor -- start 2024-03-18T20:23:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.489+0000 7ffa868e8700 1 -- start start 2024-03-18T20:23:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.490+0000 7ffa868e8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6c003a00 0x7ffa80147890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.490+0000 7ffa868e8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa800af0c0 0x7ffa800cc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.490+0000 7ffa868e8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa800cced0 0x7ffa801496f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.490+0000 7ffa868e8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa80148650 con 0x7ffa6c003a00 2024-03-18T20:23:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.490+0000 7ffa868e8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa80148930 con 0x7ffa800cced0 2024-03-18T20:23:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.490+0000 7ffa868e8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa80148c10 con 0x7ffa800af0c0 2024-03-18T20:23:39.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.490+0000 7ffa7ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6c003a00 0x7ffa80147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.490+0000 7ffa7ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6c003a00 0x7ffa80147890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46490/0 (socket says 172.21.15.1:46490) 2024-03-18T20:23:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.490+0000 7ffa7ffff700 1 -- 172.21.15.1:0/2800332049 learned_addr learned my addr 172.21.15.1:0/2800332049 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.490+0000 7ffa84e85700 1 --2- 172.21.15.1:0/2800332049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa800cced0 0x7ffa801496f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.490+0000 7ffa7f7fe700 1 --2- 172.21.15.1:0/2800332049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa800af0c0 0x7ffa800cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.490+0000 7ffa7ffff700 1 -- 172.21.15.1:0/2800332049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa800af0c0 msgr2=0x7ffa800cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.490+0000 7ffa7ffff700 1 --2- 172.21.15.1:0/2800332049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa800af0c0 0x7ffa800cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.490+0000 7ffa7ffff700 1 -- 172.21.15.1:0/2800332049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa800cced0 msgr2=0x7ffa801496f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.490+0000 7ffa7ffff700 1 --2- 172.21.15.1:0/2800332049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa800cced0 0x7ffa801496f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.490+0000 7ffa7ffff700 1 -- 172.21.15.1:0/2800332049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa80148ef0 con 0x7ffa6c003a00 2024-03-18T20:23:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.491+0000 7ffa7ffff700 1 --2- 172.21.15.1:0/2800332049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6c003a00 0x7ffa80147890 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7ffa680090b0 tx=0x7ffa680047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.491+0000 7ffa7cff9700 1 -- 172.21.15.1:0/2800332049 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6800e6c0 con 0x7ffa6c003a00 2024-03-18T20:23:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.491+0000 7ffa7cff9700 1 -- 172.21.15.1:0/2800332049 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa680038c0 con 0x7ffa6c003a00 2024-03-18T20:23:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.491+0000 7ffa868e8700 1 -- 172.21.15.1:0/2800332049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa801491d0 con 0x7ffa6c003a00 2024-03-18T20:23:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.491+0000 7ffa7cff9700 1 -- 172.21.15.1:0/2800332049 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa68003c70 con 0x7ffa6c003a00 2024-03-18T20:23:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.491+0000 7ffa868e8700 1 -- 172.21.15.1:0/2800332049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa80147dd0 con 0x7ffa6c003a00 2024-03-18T20:23:39.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.492+0000 7ffa868e8700 1 -- 172.21.15.1:0/2800332049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa80149540 con 0x7ffa6c003a00 2024-03-18T20:23:39.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.492+0000 7ffa7cff9700 1 -- 172.21.15.1:0/2800332049 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ffa68013470 con 0x7ffa6c003a00 2024-03-18T20:23:39.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.493+0000 7ffa7cff9700 1 --2- 172.21.15.1:0/2800332049 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffa6006adb0 0x7ffa6006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:39.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.493+0000 7ffa7cff9700 1 -- 172.21.15.1:0/2800332049 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ffa6808ade0 con 0x7ffa6c003a00 2024-03-18T20:23:39.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.493+0000 7ffa7f7fe700 1 --2- 172.21.15.1:0/2800332049 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffa6006adb0 0x7ffa6006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:39.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.493+0000 7ffa7f7fe700 1 --2- 172.21.15.1:0/2800332049 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffa6006adb0 0x7ffa6006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffa80158280 tx=0x7ffa70007980 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:39.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.498+0000 7ffa7cff9700 1 -- 172.21.15.1:0/2800332049 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffa68055e00 con 0x7ffa6c003a00 2024-03-18T20:23:39.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.745+0000 7ffa868e8700 1 -- 172.21.15.1:0/2800332049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ffa800676a0 con 0x7ffa6c003a00 2024-03-18T20:23:39.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.751+0000 7ffa7cff9700 1 -- 172.21.15.1:0/2800332049 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 25 and data pool 26 v93) v1 ==== 174+0+0 (secure 0 0 0) 0x7ffa680170c0 con 0x7ffa6c003a00 2024-03-18T20:23:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.757+0000 7ffa868e8700 1 -- 172.21.15.1:0/2800332049 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffa6006adb0 msgr2=0x7ffa6006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:39.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.757+0000 7ffa868e8700 1 --2- 172.21.15.1:0/2800332049 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffa6006adb0 0x7ffa6006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffa80158280 tx=0x7ffa70007980 comp rx=0 tx=0).stop 2024-03-18T20:23:39.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.757+0000 7ffa868e8700 1 -- 172.21.15.1:0/2800332049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6c003a00 msgr2=0x7ffa80147890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:39.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.757+0000 7ffa868e8700 1 --2- 172.21.15.1:0/2800332049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6c003a00 0x7ffa80147890 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7ffa680090b0 tx=0x7ffa680047c0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.773+0000 7ffa868e8700 1 -- 172.21.15.1:0/2800332049 shutdown_connections 2024-03-18T20:23:39.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.773+0000 7ffa868e8700 1 --2- 172.21.15.1:0/2800332049 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffa6006adb0 0x7ffa6006d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.773+0000 7ffa868e8700 1 --2- 172.21.15.1:0/2800332049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa800cced0 0x7ffa801496f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.773+0000 7ffa868e8700 1 --2- 172.21.15.1:0/2800332049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6c003a00 0x7ffa80147890 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.773+0000 7ffa868e8700 1 --2- 172.21.15.1:0/2800332049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa800af0c0 0x7ffa800cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.773+0000 7ffa868e8700 1 -- 172.21.15.1:0/2800332049 >> 172.21.15.1:0/2800332049 conn(0x7ffa800b1cb0 msgr2=0x7ffa80159bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:39.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.773+0000 7ffa868e8700 1 -- 172.21.15.1:0/2800332049 shutdown_connections 2024-03-18T20:23:39.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.774+0000 7ffa868e8700 1 -- 172.21.15.1:0/2800332049 wait complete. 2024-03-18T20:23:39.786 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 25 and data pool 26 2024-03-18T20:23:39.795 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:23:39.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.915+0000 7f0563d90700 1 Processor -- start 2024-03-18T20:23:39.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.916+0000 7f0563d90700 1 -- start start 2024-03-18T20:23:39.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.916+0000 7f0563d90700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f055c0ae070 0x7f055c15a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:39.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.916+0000 7f0563d90700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f055c0ad140 0x7f055c0b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:39.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.916+0000 7f0563d90700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f055c072100 con 0x7f055c15a1b0 2024-03-18T20:23:39.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.916+0000 7f0563d90700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f055c0b3560 con 0x7f055c0ae070 2024-03-18T20:23:39.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.916+0000 7f0563d90700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f055c070610 con 0x7f055c0ad140 2024-03-18T20:23:39.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.917+0000 7f056132b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f055c0ae070 0x7f055c15a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:39.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.917+0000 7f056132b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f055c0ae070 0x7f055c15a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46492/0 (socket says 172.21.15.1:46492) 2024-03-18T20:23:39.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.917+0000 7f056132b700 1 -- 172.21.15.1:0/3795277170 learned_addr learned my addr 172.21.15.1:0/3795277170 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:39.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.917+0000 7f056232d700 1 --2- 172.21.15.1:0/3795277170 >> v2:172.21.15.165:3300/0 conn(0x7f055c0ad140 0x7f055c0b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:39.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.917+0000 7f056132b700 1 -- 172.21.15.1:0/3795277170 >> v2:172.21.15.165:3300/0 conn(0x7f055c0ad140 msgr2=0x7f055c0b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:39.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.917+0000 7f056132b700 1 --2- 172.21.15.1:0/3795277170 >> v2:172.21.15.165:3300/0 conn(0x7f055c0ad140 0x7f055c0b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.917+0000 7f056132b700 1 -- 172.21.15.1:0/3795277170 >> v1:172.21.15.165:6789/0 conn(0x7f055c15a1b0 legacy=0x7f055c0aca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:39.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.917+0000 7f056132b700 1 -- 172.21.15.1:0/3795277170 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055c0708f0 con 0x7f055c0ae070 2024-03-18T20:23:39.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.917+0000 7f056132b700 1 --2- 172.21.15.1:0/3795277170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f055c0ae070 0x7f055c15a590 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f055c06e980 tx=0x7f055000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=800a83ee648f3bad server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:39.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.917+0000 7f0560b2a700 1 -- 172.21.15.1:0/3795277170 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055000dcf0 con 0x7f055c0ae070 2024-03-18T20:23:39.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.917+0000 7f0560b2a700 1 -- 172.21.15.1:0/3795277170 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f055000de50 con 0x7f055c0ae070 2024-03-18T20:23:39.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.917+0000 7f0560b2a700 1 -- 172.21.15.1:0/3795277170 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0550004250 con 0x7f055c0ae070 2024-03-18T20:23:39.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.918+0000 7f0563d90700 1 -- 172.21.15.1:0/3795277170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f055c0ae070 msgr2=0x7f055c15a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:39.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.918+0000 7f0563d90700 1 --2- 172.21.15.1:0/3795277170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f055c0ae070 0x7f055c15a590 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f055c06e980 tx=0x7f055000be70 comp rx=0 tx=0).stop 2024-03-18T20:23:39.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.918+0000 7f0563d90700 1 -- 172.21.15.1:0/3795277170 shutdown_connections 2024-03-18T20:23:39.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.918+0000 7f0563d90700 1 --2- 172.21.15.1:0/3795277170 >> v2:172.21.15.165:3300/0 conn(0x7f055c0ad140 0x7f055c0b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.918+0000 7f0563d90700 1 --2- 172.21.15.1:0/3795277170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f055c0ae070 0x7f055c15a590 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.918+0000 7f0563d90700 1 -- 172.21.15.1:0/3795277170 >> 172.21.15.1:0/3795277170 conn(0x7f055c0cee10 msgr2=0x7f055c0cf220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:39.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.918+0000 7f0563d90700 1 -- 172.21.15.1:0/3795277170 shutdown_connections 2024-03-18T20:23:39.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.918+0000 7f0563d90700 1 -- 172.21.15.1:0/3795277170 wait complete. 2024-03-18T20:23:39.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.919+0000 7f0563d90700 1 Processor -- start 2024-03-18T20:23:39.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.920+0000 7f0563d90700 1 -- start start 2024-03-18T20:23:39.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.920+0000 7f0563d90700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f055c0ad140 0x7f055c1f6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:39.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.920+0000 7f0563d90700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f055c0ae070 0x7f055c1f2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.920+0000 7f0563d90700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f055c15a1b0 0x7f055c14ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.920+0000 7f0563d90700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055c1f6b00 con 0x7f055c0ae070 2024-03-18T20:23:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.920+0000 7f0563d90700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055c1f6520 con 0x7f055c0ad140 2024-03-18T20:23:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.920+0000 7f0563d90700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f055c14d500 con 0x7f055c15a1b0 2024-03-18T20:23:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.921+0000 7f056132b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f055c0ae070 0x7f055c1f2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.921+0000 7f056132b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f055c0ae070 0x7f055c1f2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46506/0 (socket says 172.21.15.1:46506) 2024-03-18T20:23:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.921+0000 7f056132b700 1 -- 172.21.15.1:0/3795932345 learned_addr learned my addr 172.21.15.1:0/3795932345 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.921+0000 7f0561b2c700 1 --2- 172.21.15.1:0/3795932345 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f055c0ad140 0x7f055c1f6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.921+0000 7f056232d700 1 --2- 172.21.15.1:0/3795932345 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f055c15a1b0 0x7f055c14ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.921+0000 7f056132b700 1 -- 172.21.15.1:0/3795932345 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f055c15a1b0 msgr2=0x7f055c14ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.921+0000 7f056132b700 1 --2- 172.21.15.1:0/3795932345 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f055c15a1b0 0x7f055c14ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.921+0000 7f056132b700 1 -- 172.21.15.1:0/3795932345 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f055c0ad140 msgr2=0x7f055c1f6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.921+0000 7f056132b700 1 --2- 172.21.15.1:0/3795932345 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f055c0ad140 0x7f055c1f6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:39.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.921+0000 7f056132b700 1 -- 172.21.15.1:0/3795932345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055000b7f0 con 0x7f055c0ae070 2024-03-18T20:23:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.921+0000 7f056232d700 1 --2- 172.21.15.1:0/3795932345 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f055c15a1b0 0x7f055c14ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.921+0000 7f056132b700 1 --2- 172.21.15.1:0/3795932345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f055c0ae070 0x7f055c1f2950 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f055000beb0 tx=0x7f0550004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.921+0000 7f054e7fc700 1 -- 172.21.15.1:0/3795932345 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0550004230 con 0x7f055c0ae070 2024-03-18T20:23:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.921+0000 7f054e7fc700 1 -- 172.21.15.1:0/3795932345 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0550010470 con 0x7f055c0ae070 2024-03-18T20:23:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.921+0000 7f0563d90700 1 -- 172.21.15.1:0/3795932345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f055c1f6790 con 0x7f055c0ae070 2024-03-18T20:23:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.921+0000 7f054e7fc700 1 -- 172.21.15.1:0/3795932345 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0550010820 con 0x7f055c0ae070 2024-03-18T20:23:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.921+0000 7f0563d90700 1 -- 172.21.15.1:0/3795932345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f055c09a210 con 0x7f055c0ae070 2024-03-18T20:23:39.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.922+0000 7f0563d90700 1 -- 172.21.15.1:0/3795932345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f055c1f6c70 con 0x7f055c0ae070 2024-03-18T20:23:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.923+0000 7f054e7fc700 1 -- 172.21.15.1:0/3795932345 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f05500193f0 con 0x7f055c0ae070 2024-03-18T20:23:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.923+0000 7f054e7fc700 1 --2- 172.21.15.1:0/3795932345 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f054006adb0 0x7f054006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.923+0000 7f054e7fc700 1 -- 172.21.15.1:0/3795932345 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f055008b670 con 0x7f055c0ae070 2024-03-18T20:23:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.923+0000 7f0561b2c700 1 --2- 172.21.15.1:0/3795932345 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f054006adb0 0x7f054006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:39.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.924+0000 7f0561b2c700 1 --2- 172.21.15.1:0/3795932345 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f054006adb0 0x7f054006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f055c09a800 tx=0x7f0558008040 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:39.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:39.928+0000 7f054e7fc700 1 -- 172.21.15.1:0/3795932345 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0550059fa0 con 0x7f055c0ae070 2024-03-18T20:23:40.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.129+0000 7f0563d90700 1 -- 172.21.15.1:0/3795932345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f055c073350 con 0x7f055c0ae070 2024-03-18T20:23:40.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.130+0000 7f054e7fc700 1 -- 172.21.15.1:0/3795932345 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+16378 (secure 0 0 0) 0x7f0550059b30 con 0x7f055c0ae070 2024-03-18T20:23:40.135 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:40.135 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:23:39.746976+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:23:37.591406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-03-18T20:23:38.723169+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:23:40.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.132+0000 7f0563d90700 1 -- 172.21.15.1:0/3795932345 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f054006adb0 msgr2=0x7f054006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:40.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.132+0000 7f0563d90700 1 --2- 172.21.15.1:0/3795932345 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f054006adb0 0x7f054006d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f055c09a800 tx=0x7f0558008040 comp rx=0 tx=0).stop 2024-03-18T20:23:40.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.132+0000 7f0563d90700 1 -- 172.21.15.1:0/3795932345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f055c0ae070 msgr2=0x7f055c1f2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.132+0000 7f0563d90700 1 --2- 172.21.15.1:0/3795932345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f055c0ae070 0x7f055c1f2950 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f055000beb0 tx=0x7f0550004750 comp rx=0 tx=0).stop 2024-03-18T20:23:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.132+0000 7f0563d90700 1 -- 172.21.15.1:0/3795932345 shutdown_connections 2024-03-18T20:23:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.132+0000 7f0563d90700 1 --2- 172.21.15.1:0/3795932345 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f054006adb0 0x7f054006d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.132+0000 7f0563d90700 1 --2- 172.21.15.1:0/3795932345 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f055c15a1b0 0x7f055c14ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.132+0000 7f0563d90700 1 --2- 172.21.15.1:0/3795932345 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f055c0ad140 0x7f055c1f6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.132+0000 7f0563d90700 1 --2- 172.21.15.1:0/3795932345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f055c0ae070 0x7f055c1f2950 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.133+0000 7f0563d90700 1 -- 172.21.15.1:0/3795932345 >> 172.21.15.1:0/3795932345 conn(0x7f055c0cee10 msgr2=0x7f055c1587b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.133+0000 7f0563d90700 1 -- 172.21.15.1:0/3795932345 shutdown_connections 2024-03-18T20:23:40.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.133+0000 7f0563d90700 1 -- 172.21.15.1:0/3795932345 wait complete. 2024-03-18T20:23:40.150 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:23:40.214 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:23:40.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:23:40.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.274+0000 7f776fd10700 1 Processor -- start 2024-03-18T20:23:40.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.274+0000 7f776fd10700 1 -- start start 2024-03-18T20:23:40.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.275+0000 7f776fd10700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680cfab0 0x7f776815c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:40.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.275+0000 7f776fd10700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f77680cdf80 0x7f77680cba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:40.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.275+0000 7f776fd10700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776815c660 con 0x7f7768140b20 2024-03-18T20:23:40.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.275+0000 7f776fd10700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f776815fbe0 con 0x7f77680cdf80 2024-03-18T20:23:40.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.275+0000 7f776fd10700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f776815f060 con 0x7f77680cfab0 2024-03-18T20:23:40.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.275+0000 7f776d2ab700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7768140b20 0x7f77680cfe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45488/0 (socket says 172.21.15.1:45488) 2024-03-18T20:23:40.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.275+0000 7f776d2ab700 1 -- 172.21.15.1:0/1499640183 learned_addr learned my addr 172.21.15.1:0/1499640183 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:40.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776caaa700 1 -- 172.21.15.1:0/1499640183 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075616689 0 0) 0x7f776815c660 con 0x7f7768140b20 2024-03-18T20:23:40.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776caaa700 1 -- 172.21.15.1:0/1499640183 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77540035b0 con 0x7f7768140b20 2024-03-18T20:23:40.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776e2ad700 1 --2- 172.21.15.1:0/1499640183 >> v2:172.21.15.165:3300/0 conn(0x7f77680cdf80 0x7f77680cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:40.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776daac700 1 --2- 172.21.15.1:0/1499640183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680cfab0 0x7f776815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:40.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776caaa700 1 -- 172.21.15.1:0/1499640183 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2779005195 0 0) 0x7f77540035b0 con 0x7f7768140b20 2024-03-18T20:23:40.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776caaa700 1 -- 172.21.15.1:0/1499640183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680cfab0 msgr2=0x7f776815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:40.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776caaa700 1 --2- 172.21.15.1:0/1499640183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680cfab0 0x7f776815c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:40.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776caaa700 1 -- 172.21.15.1:0/1499640183 >> v2:172.21.15.165:3300/0 conn(0x7f77680cdf80 msgr2=0x7f77680cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:40.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776caaa700 1 --2- 172.21.15.1:0/1499640183 >> v2:172.21.15.165:3300/0 conn(0x7f77680cdf80 0x7f77680cba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:40.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776caaa700 1 -- 172.21.15.1:0/1499640183 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776815f340 con 0x7f7768140b20 2024-03-18T20:23:40.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776caaa700 1 -- 172.21.15.1:0/1499640183 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f7764001670 con 0x7f7768140b20 2024-03-18T20:23:40.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776daac700 1 --2- 172.21.15.1:0/1499640183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680cfab0 0x7f776815c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:23:40.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776caaa700 1 -- 172.21.15.1:0/1499640183 >> v1:172.21.15.1:6789/0 conn(0x7f7768140b20 legacy=0x7f77680cfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:40.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776caaa700 1 --2- 172.21.15.1:0/1499640183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7754003a00 0x7f7754003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:40.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776caaa700 1 --2- 172.21.15.1:0/1499640183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7754004850 0x7f7754004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:40.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776caaa700 1 --2- 172.21.15.1:0/1499640183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680cfab0 0x7f7754005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:40.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776caaa700 1 -- 172.21.15.1:0/1499640183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776815fbe0 con 0x7f7754003a00 2024-03-18T20:23:40.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776caaa700 1 -- 172.21.15.1:0/1499640183 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776815f060 con 0x7f7754004850 2024-03-18T20:23:40.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776caaa700 1 -- 172.21.15.1:0/1499640183 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f775400ba10 con 0x7f77680cfab0 2024-03-18T20:23:40.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776daac700 1 --2- 172.21.15.1:0/1499640183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7754003a00 0x7f7754003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:40.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776d2ab700 1 --2- 172.21.15.1:0/1499640183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7754004850 0x7f7754004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:40.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776e2ad700 1 --2- 172.21.15.1:0/1499640183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680cfab0 0x7f7754005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:40.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776daac700 1 -- 172.21.15.1:0/1499640183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680cfab0 msgr2=0x7f7754005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:40.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776daac700 1 --2- 172.21.15.1:0/1499640183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77680cfab0 0x7f7754005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:40.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776daac700 1 -- 172.21.15.1:0/1499640183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7754004850 msgr2=0x7f7754004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:40.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776daac700 1 --2- 172.21.15.1:0/1499640183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7754004850 0x7f7754004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:40.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.276+0000 7f776daac700 1 -- 172.21.15.1:0/1499640183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77540035b0 con 0x7f7754003a00 2024-03-18T20:23:40.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.277+0000 7f776daac700 1 --2- 172.21.15.1:0/1499640183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7754003a00 0x7f7754003e20 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f7758009980 tx=0x7f7754005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac41f7648cacae6f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:40.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.277+0000 7f776caaa700 1 -- 172.21.15.1:0/1499640183 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775800c070 con 0x7f7754003a00 2024-03-18T20:23:40.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.277+0000 7f776caaa700 1 -- 172.21.15.1:0/1499640183 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7758013610 con 0x7f7754003a00 2024-03-18T20:23:40.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.277+0000 7f776caaa700 1 -- 172.21.15.1:0/1499640183 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77580139a0 con 0x7f7754003a00 2024-03-18T20:23:40.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.277+0000 7f776fd10700 1 -- 172.21.15.1:0/1499640183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7754003a00 msgr2=0x7f7754003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:40.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.277+0000 7f776fd10700 1 --2- 172.21.15.1:0/1499640183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7754003a00 0x7f7754003e20 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f7758009980 tx=0x7f7754005880 comp rx=0 tx=0).stop 2024-03-18T20:23:40.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.277+0000 7f776e2ad700 1 -- 172.21.15.1:0/1499640183 reap_dead start 2024-03-18T20:23:40.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.277+0000 7f776fd10700 1 -- 172.21.15.1:0/1499640183 shutdown_connections 2024-03-18T20:23:40.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.277+0000 7f776fd10700 1 -- 172.21.15.1:0/1499640183 >> 172.21.15.1:0/1499640183 conn(0x7f776806ab70 msgr2=0x7f776806af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:40.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.277+0000 7f776fd10700 1 -- 172.21.15.1:0/1499640183 shutdown_connections 2024-03-18T20:23:40.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.277+0000 7f776fd10700 1 -- 172.21.15.1:0/1499640183 wait complete. 2024-03-18T20:23:40.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.279+0000 7f776fd10700 1 Processor -- start 2024-03-18T20:23:40.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.279+0000 7f776fd10700 1 -- start start 2024-03-18T20:23:40.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.280+0000 7f776fd10700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7754003a00 0x7f77680aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:40.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.280+0000 7f776fd10700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7768142df0 0x7f776813d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:40.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.280+0000 7f776fd10700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f776806e180 0x7f776806e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:40.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.280+0000 7f776fd10700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776806eb40 con 0x7f7768142df0 2024-03-18T20:23:40.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.280+0000 7f776fd10700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7768143560 con 0x7f776806e180 2024-03-18T20:23:40.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.280+0000 7f776fd10700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7768143840 con 0x7f7754003a00 2024-03-18T20:23:40.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.280+0000 7f776d2ab700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7768142df0 0x7f776813d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:40.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.280+0000 7f776d2ab700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7768142df0 0x7f776813d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46522/0 (socket says 172.21.15.1:46522) 2024-03-18T20:23:40.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.280+0000 7f776d2ab700 1 -- 172.21.15.1:0/731594105 learned_addr learned my addr 172.21.15.1:0/731594105 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:40.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.280+0000 7f776daac700 1 --2- 172.21.15.1:0/731594105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7754003a00 0x7f77680aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:40.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.280+0000 7f776e2ad700 1 --2- 172.21.15.1:0/731594105 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f776806e180 0x7f776806e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:40.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.280+0000 7f776d2ab700 1 -- 172.21.15.1:0/731594105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7754003a00 msgr2=0x7f77680aeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:40.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.280+0000 7f776d2ab700 1 --2- 172.21.15.1:0/731594105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7754003a00 0x7f77680aeb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:40.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.280+0000 7f776d2ab700 1 -- 172.21.15.1:0/731594105 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f776806e180 msgr2=0x7f776806e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:40.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.280+0000 7f776d2ab700 1 --2- 172.21.15.1:0/731594105 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f776806e180 0x7f776806e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:40.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.280+0000 7f776d2ab700 1 -- 172.21.15.1:0/731594105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7768143b20 con 0x7f7768142df0 2024-03-18T20:23:40.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.281+0000 7f776d2ab700 1 --2- 172.21.15.1:0/731594105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7768142df0 0x7f776813d020 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f7764007340 tx=0x7f7764007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:40.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.281+0000 7f775e7fc700 1 -- 172.21.15.1:0/731594105 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776400f660 con 0x7f7768142df0 2024-03-18T20:23:40.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.281+0000 7f775e7fc700 1 -- 172.21.15.1:0/731594105 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776400f800 con 0x7f7768142df0 2024-03-18T20:23:40.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.281+0000 7f776fd10700 1 -- 172.21.15.1:0/731594105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7768143e00 con 0x7f7768142df0 2024-03-18T20:23:40.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.281+0000 7f775e7fc700 1 -- 172.21.15.1:0/731594105 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776400fbb0 con 0x7f7768142df0 2024-03-18T20:23:40.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.281+0000 7f776fd10700 1 -- 172.21.15.1:0/731594105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7768143270 con 0x7f7768142df0 2024-03-18T20:23:40.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.282+0000 7f776fd10700 1 -- 172.21.15.1:0/731594105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77680676a0 con 0x7f7768142df0 2024-03-18T20:23:40.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.282+0000 7f775e7fc700 1 -- 172.21.15.1:0/731594105 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f776400fd10 con 0x7f7768142df0 2024-03-18T20:23:40.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.283+0000 7f775e7fc700 1 --2- 172.21.15.1:0/731594105 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f774c06adb0 0x7f774c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:40.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.283+0000 7f775e7fc700 1 -- 172.21.15.1:0/731594105 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f7764089630 con 0x7f7768142df0 2024-03-18T20:23:40.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.283+0000 7f776daac700 1 --2- 172.21.15.1:0/731594105 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f774c06adb0 0x7f774c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:40.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.283+0000 7f776daac700 1 --2- 172.21.15.1:0/731594105 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f774c06adb0 0x7f774c06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f77580098a0 tx=0x7f775801e040 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:40.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.287+0000 7f775e7fc700 1 -- 172.21.15.1:0/731594105 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7764054550 con 0x7f7768142df0 2024-03-18T20:23:40.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.491+0000 7f776fd10700 1 -- 172.21.15.1:0/731594105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f776806cf10 con 0x7f7768142df0 2024-03-18T20:23:40.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.492+0000 7f775e7fc700 1 -- 172.21.15.1:0/731594105 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+16378 (secure 0 0 0) 0x7f7764057b70 con 0x7f7768142df0 2024-03-18T20:23:40.497 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:40.497 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":213,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:23:39.746976+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:23:37.591406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-03-18T20:23:38.723169+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:23:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.494+0000 7f776fd10700 1 -- 172.21.15.1:0/731594105 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f774c06adb0 msgr2=0x7f774c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.494+0000 7f776fd10700 1 --2- 172.21.15.1:0/731594105 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f774c06adb0 0x7f774c06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f77580098a0 tx=0x7f775801e040 comp rx=0 tx=0).stop 2024-03-18T20:23:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.494+0000 7f776fd10700 1 -- 172.21.15.1:0/731594105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7768142df0 msgr2=0x7f776813d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.494+0000 7f776fd10700 1 --2- 172.21.15.1:0/731594105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7768142df0 0x7f776813d020 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f7764007340 tx=0x7f7764007650 comp rx=0 tx=0).stop 2024-03-18T20:23:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.494+0000 7f776fd10700 1 -- 172.21.15.1:0/731594105 shutdown_connections 2024-03-18T20:23:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.494+0000 7f776fd10700 1 --2- 172.21.15.1:0/731594105 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f774c06adb0 0x7f774c06d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.494+0000 7f776fd10700 1 --2- 172.21.15.1:0/731594105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7754003a00 0x7f77680aeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.494+0000 7f776fd10700 1 --2- 172.21.15.1:0/731594105 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f776806e180 0x7f776806e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.494+0000 7f776fd10700 1 --2- 172.21.15.1:0/731594105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7768142df0 0x7f776813d020 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:40.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.494+0000 7f776fd10700 1 -- 172.21.15.1:0/731594105 >> 172.21.15.1:0/731594105 conn(0x7f776806ab70 msgr2=0x7f7768157570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:40.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.494+0000 7f776fd10700 1 -- 172.21.15.1:0/731594105 shutdown_connections 2024-03-18T20:23:40.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.494+0000 7f776fd10700 1 -- 172.21.15.1:0/731594105 wait complete. 2024-03-18T20:23:40.511 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T20:23:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.675+0000 7f45791f1700 1 Processor -- start 2024-03-18T20:23:40.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.675+0000 7f45791f1700 1 -- start start 2024-03-18T20:23:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f45791f1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f457406d560 0x7f45740aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f45791f1700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457415ef00 con 0x7f45740af510 2024-03-18T20:23:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f45791f1700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457415f1e0 con 0x7f457406a750 2024-03-18T20:23:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f45791f1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f457415f4c0 con 0x7f457406d560 2024-03-18T20:23:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f457359e700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f45740af510 0x7f457415b900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45492/0 (socket says 172.21.15.1:45492) 2024-03-18T20:23:40.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f457359e700 1 -- 172.21.15.1:0/3429605964 learned_addr learned my addr 172.21.15.1:0/3429605964 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:40.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f457259c700 1 --2- 172.21.15.1:0/3429605964 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f457406d560 0x7f45740aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:40.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f4571d9b700 1 -- 172.21.15.1:0/3429605964 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806122279 0 0) 0x7f457415ef00 con 0x7f45740af510 2024-03-18T20:23:40.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f4571d9b700 1 -- 172.21.15.1:0/3429605964 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f455c0035b0 con 0x7f45740af510 2024-03-18T20:23:40.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f4571d9b700 1 -- 172.21.15.1:0/3429605964 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220266242 0 0) 0x7f457415f1e0 con 0x7f457406a750 2024-03-18T20:23:40.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f4571d9b700 1 -- 172.21.15.1:0/3429605964 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f457415ef00 con 0x7f457406a750 2024-03-18T20:23:40.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f4571d9b700 1 -- 172.21.15.1:0/3429605964 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 191906399 0 0) 0x7f455c0035b0 con 0x7f45740af510 2024-03-18T20:23:40.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f4571d9b700 1 -- 172.21.15.1:0/3429605964 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f457406d560 msgr2=0x7f45740aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:40.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f4571d9b700 1 --2- 172.21.15.1:0/3429605964 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f457406d560 0x7f45740aefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:40.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f4571d9b700 1 -- 172.21.15.1:0/3429605964 >> v1:172.21.15.165:6789/0 conn(0x7f457406a750 legacy=0x7f457406ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:40.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f4571d9b700 1 -- 172.21.15.1:0/3429605964 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457415f7a0 con 0x7f45740af510 2024-03-18T20:23:40.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f457259c700 1 --2- 172.21.15.1:0/3429605964 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f457406d560 0x7f45740aefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:23:40.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f4571d9b700 1 -- 172.21.15.1:0/3429605964 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f456c0019f0 con 0x7f45740af510 2024-03-18T20:23:40.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f4571d9b700 1 -- 172.21.15.1:0/3429605964 >> v1:172.21.15.1:6789/0 conn(0x7f45740af510 legacy=0x7f457415b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:40.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f4571d9b700 1 --2- 172.21.15.1:0/3429605964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f455c003bc0 0x7f455c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:40.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f4571d9b700 1 --2- 172.21.15.1:0/3429605964 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f457406d560 0x7f455c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:40.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f4571d9b700 1 --2- 172.21.15.1:0/3429605964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c004d90 0x7f455c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:40.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f4571d9b700 1 -- 172.21.15.1:0/3429605964 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457415f4c0 con 0x7f455c004d90 2024-03-18T20:23:40.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f4571d9b700 1 -- 172.21.15.1:0/3429605964 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455c00bac0 con 0x7f455c003bc0 2024-03-18T20:23:40.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.676+0000 7f4571d9b700 1 -- 172.21.15.1:0/3429605964 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f455c00bc00 con 0x7f457406d560 2024-03-18T20:23:40.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.677+0000 7f457359e700 1 --2- 172.21.15.1:0/3429605964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c004d90 0x7f455c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:40.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.677+0000 7f4572d9d700 1 --2- 172.21.15.1:0/3429605964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f455c003bc0 0x7f455c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:40.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.677+0000 7f457259c700 1 --2- 172.21.15.1:0/3429605964 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f457406d560 0x7f455c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:40.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.677+0000 7f457359e700 1 -- 172.21.15.1:0/3429605964 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f457406d560 msgr2=0x7f455c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:40.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.677+0000 7f457359e700 1 --2- 172.21.15.1:0/3429605964 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f457406d560 0x7f455c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:40.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.677+0000 7f457359e700 1 -- 172.21.15.1:0/3429605964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f455c003bc0 msgr2=0x7f455c003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:40.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.677+0000 7f457359e700 1 --2- 172.21.15.1:0/3429605964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f455c003bc0 0x7f455c003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:40.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.677+0000 7f457359e700 1 -- 172.21.15.1:0/3429605964 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45680094a0 con 0x7f455c004d90 2024-03-18T20:23:40.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.677+0000 7f457359e700 1 --2- 172.21.15.1:0/3429605964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c004d90 0x7f455c005210 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f45740d1030 tx=0x7f455c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=333e227f9b9def37 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:40.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.677+0000 7f4571d9b700 1 -- 172.21.15.1:0/3429605964 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456c0072d0 con 0x7f455c004d90 2024-03-18T20:23:40.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.677+0000 7f4571d9b700 1 -- 172.21.15.1:0/3429605964 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f456c007450 con 0x7f455c004d90 2024-03-18T20:23:40.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.677+0000 7f4571d9b700 1 -- 172.21.15.1:0/3429605964 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456c007800 con 0x7f455c004d90 2024-03-18T20:23:40.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.678+0000 7f45791f1700 1 -- 172.21.15.1:0/3429605964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c004d90 msgr2=0x7f455c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:40.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.678+0000 7f45791f1700 1 --2- 172.21.15.1:0/3429605964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f455c004d90 0x7f455c005210 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f45740d1030 tx=0x7f455c005880 comp rx=0 tx=0).stop 2024-03-18T20:23:40.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.678+0000 7f457359e700 1 -- 172.21.15.1:0/3429605964 reap_dead start 2024-03-18T20:23:40.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.678+0000 7f45791f1700 1 -- 172.21.15.1:0/3429605964 shutdown_connections 2024-03-18T20:23:40.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.678+0000 7f45791f1700 1 -- 172.21.15.1:0/3429605964 >> 172.21.15.1:0/3429605964 conn(0x7f45740cff30 msgr2=0x7f45740d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:40.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.678+0000 7f45791f1700 1 -- 172.21.15.1:0/3429605964 shutdown_connections 2024-03-18T20:23:40.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.678+0000 7f45791f1700 1 -- 172.21.15.1:0/3429605964 wait complete. 2024-03-18T20:23:40.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.679+0000 7f45791f1700 1 Processor -- start 2024-03-18T20:23:40.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.679+0000 7f45791f1700 1 -- start start 2024-03-18T20:23:40.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.680+0000 7f45791f1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4574155fe0 0x7f45740b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:40.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.680+0000 7f45791f1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45740b5380 0x7f45740cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:40.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.680+0000 7f45791f1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45740cba00 0x7f45740b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:40.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.680+0000 7f45791f1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45740b1d70 con 0x7f45740b5380 2024-03-18T20:23:40.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.680+0000 7f45791f1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4574156930 con 0x7f45740cba00 2024-03-18T20:23:40.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.680+0000 7f45791f1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45740cbe80 con 0x7f4574155fe0 2024-03-18T20:23:40.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.680+0000 7f457259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45740b5380 0x7f45740cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:40.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.680+0000 7f457259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45740b5380 0x7f45740cb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46538/0 (socket says 172.21.15.1:46538) 2024-03-18T20:23:40.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.680+0000 7f457259c700 1 -- 172.21.15.1:0/731309180 learned_addr learned my addr 172.21.15.1:0/731309180 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:40.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.680+0000 7f4572d9d700 1 --2- 172.21.15.1:0/731309180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4574155fe0 0x7f45740b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:40.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.680+0000 7f457359e700 1 --2- 172.21.15.1:0/731309180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45740cba00 0x7f45740b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:40.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.680+0000 7f457259c700 1 -- 172.21.15.1:0/731309180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4574155fe0 msgr2=0x7f45740b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:40.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.680+0000 7f457259c700 1 --2- 172.21.15.1:0/731309180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4574155fe0 0x7f45740b0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:40.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.680+0000 7f457259c700 1 -- 172.21.15.1:0/731309180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45740cba00 msgr2=0x7f45740b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:40.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.680+0000 7f457259c700 1 --2- 172.21.15.1:0/731309180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45740cba00 0x7f45740b3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:40.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.680+0000 7f457259c700 1 -- 172.21.15.1:0/731309180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45741563c0 con 0x7f45740b5380 2024-03-18T20:23:40.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.681+0000 7f457259c700 1 --2- 172.21.15.1:0/731309180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45740b5380 0x7f45740cb4c0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f4568011cd0 tx=0x7f4568009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:40.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.681+0000 7f455b7fe700 1 -- 172.21.15.1:0/731309180 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456800f9a0 con 0x7f45740b5380 2024-03-18T20:23:40.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.681+0000 7f455b7fe700 1 -- 172.21.15.1:0/731309180 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f456800fb00 con 0x7f45740b5380 2024-03-18T20:23:40.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.681+0000 7f45791f1700 1 -- 172.21.15.1:0/731309180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457415e480 con 0x7f45740b5380 2024-03-18T20:23:40.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.681+0000 7f455b7fe700 1 -- 172.21.15.1:0/731309180 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456800fe90 con 0x7f45740b5380 2024-03-18T20:23:40.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.681+0000 7f45791f1700 1 -- 172.21.15.1:0/731309180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45740b13d0 con 0x7f45740b5380 2024-03-18T20:23:40.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.682+0000 7f45791f1700 1 -- 172.21.15.1:0/731309180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45740676a0 con 0x7f45740b5380 2024-03-18T20:23:40.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.682+0000 7f455b7fe700 1 -- 172.21.15.1:0/731309180 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4568004020 con 0x7f45740b5380 2024-03-18T20:23:40.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.683+0000 7f455b7fe700 1 --2- 172.21.15.1:0/731309180 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f456006adb0 0x7f456006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:40.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.683+0000 7f455b7fe700 1 -- 172.21.15.1:0/731309180 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f4568089f60 con 0x7f45740b5380 2024-03-18T20:23:40.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.683+0000 7f4572d9d700 1 --2- 172.21.15.1:0/731309180 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f456006adb0 0x7f456006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:40.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.683+0000 7f4572d9d700 1 --2- 172.21.15.1:0/731309180 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f456006adb0 0x7f456006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4564007fd0 tx=0x7f4564009040 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:40.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.687+0000 7f455b7fe700 1 -- 172.21.15.1:0/731309180 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4568054ff0 con 0x7f45740b5380 2024-03-18T20:23:40.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:40.939+0000 7f45791f1700 1 -- 172.21.15.1:0/731309180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f45740b5eb0 con 0x7f45740b5380 2024-03-18T20:23:41.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.768+0000 7f455b7fe700 1 -- 172.21.15.1:0/731309180 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v96) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4568058610 con 0x7f45740b5380 2024-03-18T20:23:41.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.771+0000 7f45791f1700 1 -- 172.21.15.1:0/731309180 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f456006adb0 msgr2=0x7f456006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:41.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.771+0000 7f45791f1700 1 --2- 172.21.15.1:0/731309180 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f456006adb0 0x7f456006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4564007fd0 tx=0x7f4564009040 comp rx=0 tx=0).stop 2024-03-18T20:23:41.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.771+0000 7f45791f1700 1 -- 172.21.15.1:0/731309180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45740b5380 msgr2=0x7f45740cb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:41.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.771+0000 7f45791f1700 1 --2- 172.21.15.1:0/731309180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45740b5380 0x7f45740cb4c0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f4568011cd0 tx=0x7f4568009ed0 comp rx=0 tx=0).stop 2024-03-18T20:23:41.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.771+0000 7f45791f1700 1 -- 172.21.15.1:0/731309180 shutdown_connections 2024-03-18T20:23:41.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.771+0000 7f45791f1700 1 --2- 172.21.15.1:0/731309180 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f456006adb0 0x7f456006d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:41.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.771+0000 7f45791f1700 1 --2- 172.21.15.1:0/731309180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4574155fe0 0x7f45740b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:41.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.771+0000 7f45791f1700 1 --2- 172.21.15.1:0/731309180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45740cba00 0x7f45740b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:41.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.771+0000 7f45791f1700 1 --2- 172.21.15.1:0/731309180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45740b5380 0x7f45740cb4c0 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:41.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.771+0000 7f45791f1700 1 -- 172.21.15.1:0/731309180 >> 172.21.15.1:0/731309180 conn(0x7f45740cff30 msgr2=0x7f4574159af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:41.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.771+0000 7f45791f1700 1 -- 172.21.15.1:0/731309180 shutdown_connections 2024-03-18T20:23:41.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.771+0000 7f45791f1700 1 -- 172.21.15.1:0/731309180 wait complete. 2024-03-18T20:23:41.790 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-18T20:23:41.791 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T20:23:41.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.921+0000 7f27cb673700 1 Processor -- start 2024-03-18T20:23:41.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.921+0000 7f27cb673700 1 -- start start 2024-03-18T20:23:41.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.922+0000 7f27cb673700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f27c415d5a0 0x7f27c40ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:41.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.922+0000 7f27cb673700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27c406ae60 0x7f27c4167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:41.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.922+0000 7f27cb673700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c40cf600 con 0x7f27c40ce550 2024-03-18T20:23:41.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.922+0000 7f27cb673700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27c40ce9f0 con 0x7f27c415d5a0 2024-03-18T20:23:41.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.922+0000 7f27cb673700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c40cecd0 con 0x7f27c406ae60 2024-03-18T20:23:41.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.922+0000 7f27c8c0e700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f27c40ce550 0x7f27c406a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45502/0 (socket says 172.21.15.1:45502) 2024-03-18T20:23:41.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.922+0000 7f27c8c0e700 1 -- 172.21.15.1:0/676831492 learned_addr learned my addr 172.21.15.1:0/676831492 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:41.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27c9c10700 1 --2- 172.21.15.1:0/676831492 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27c406ae60 0x7f27c4167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:41.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27c940f700 1 --2- 172.21.15.1:0/676831492 >> v2:172.21.15.165:3300/0 conn(0x7f27c415d5a0 0x7f27c40ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:41.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27bbfff700 1 -- 172.21.15.1:0/676831492 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70261429 0 0) 0x7f27c40cf600 con 0x7f27c40ce550 2024-03-18T20:23:41.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27bbfff700 1 -- 172.21.15.1:0/676831492 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b00035b0 con 0x7f27c40ce550 2024-03-18T20:23:41.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27bbfff700 1 -- 172.21.15.1:0/676831492 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1341073571 0 0) 0x7f27b00035b0 con 0x7f27c40ce550 2024-03-18T20:23:41.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27bbfff700 1 -- 172.21.15.1:0/676831492 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27c406ae60 msgr2=0x7f27c4167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:41.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27bbfff700 1 --2- 172.21.15.1:0/676831492 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27c406ae60 0x7f27c4167140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:41.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27bbfff700 1 -- 172.21.15.1:0/676831492 >> v2:172.21.15.165:3300/0 conn(0x7f27c415d5a0 msgr2=0x7f27c40ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:41.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27bbfff700 1 --2- 172.21.15.1:0/676831492 >> v2:172.21.15.165:3300/0 conn(0x7f27c415d5a0 0x7f27c40ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:41.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27bbfff700 1 -- 172.21.15.1:0/676831492 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c40cefb0 con 0x7f27c40ce550 2024-03-18T20:23:41.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27bbfff700 1 -- 172.21.15.1:0/676831492 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f27c0001670 con 0x7f27c40ce550 2024-03-18T20:23:41.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27bbfff700 1 -- 172.21.15.1:0/676831492 >> v1:172.21.15.1:6789/0 conn(0x7f27c40ce550 legacy=0x7f27c406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:41.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27bbfff700 1 --2- 172.21.15.1:0/676831492 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27b0003a00 0x7f27b0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:41.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27bbfff700 1 --2- 172.21.15.1:0/676831492 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27c406ae60 0x7f27b0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27bbfff700 1 --2- 172.21.15.1:0/676831492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b0004d90 0x7f27b0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27bbfff700 1 -- 172.21.15.1:0/676831492 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c40ce9f0 con 0x7f27b0004d90 2024-03-18T20:23:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27bbfff700 1 -- 172.21.15.1:0/676831492 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c40cecd0 con 0x7f27b0003a00 2024-03-18T20:23:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27bbfff700 1 -- 172.21.15.1:0/676831492 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27b000ba10 con 0x7f27c406ae60 2024-03-18T20:23:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27c9c10700 1 --2- 172.21.15.1:0/676831492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b0004d90 0x7f27b0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27c940f700 1 --2- 172.21.15.1:0/676831492 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27b0003a00 0x7f27b0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27c8c0e700 1 --2- 172.21.15.1:0/676831492 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27c406ae60 0x7f27b0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27c9c10700 1 -- 172.21.15.1:0/676831492 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27c406ae60 msgr2=0x7f27b0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27c9c10700 1 --2- 172.21.15.1:0/676831492 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27c406ae60 0x7f27b0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27c9c10700 1 -- 172.21.15.1:0/676831492 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27b0003a00 msgr2=0x7f27b0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27c9c10700 1 --2- 172.21.15.1:0/676831492 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27b0003a00 0x7f27b0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27c9c10700 1 -- 172.21.15.1:0/676831492 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b00035b0 con 0x7f27b0004d90 2024-03-18T20:23:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.923+0000 7f27c8c0e700 1 --2- 172.21.15.1:0/676831492 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27c406ae60 0x7f27b0004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:23:41.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.924+0000 7f27c9c10700 1 --2- 172.21.15.1:0/676831492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b0004d90 0x7f27b0005210 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00e170 tx=0x7f27b0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b59509f5c14b360 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:41.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.924+0000 7f27bbfff700 1 -- 172.21.15.1:0/676831492 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc00ec90 con 0x7f27b0004d90 2024-03-18T20:23:41.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.924+0000 7f27bbfff700 1 -- 172.21.15.1:0/676831492 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc00f040 con 0x7f27b0004d90 2024-03-18T20:23:41.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.924+0000 7f27bbfff700 1 -- 172.21.15.1:0/676831492 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc014480 con 0x7f27b0004d90 2024-03-18T20:23:41.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.924+0000 7f27cb673700 1 -- 172.21.15.1:0/676831492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b0004d90 msgr2=0x7f27b0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:41.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.924+0000 7f27cb673700 1 --2- 172.21.15.1:0/676831492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b0004d90 0x7f27b0005210 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00e170 tx=0x7f27b0005880 comp rx=0 tx=0).stop 2024-03-18T20:23:41.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.924+0000 7f27c9c10700 1 -- 172.21.15.1:0/676831492 reap_dead start 2024-03-18T20:23:41.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.924+0000 7f27cb673700 1 -- 172.21.15.1:0/676831492 shutdown_connections 2024-03-18T20:23:41.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.924+0000 7f27cb673700 1 -- 172.21.15.1:0/676831492 >> 172.21.15.1:0/676831492 conn(0x7f27c40cc9b0 msgr2=0x7f27c40ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:41.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.924+0000 7f27cb673700 1 -- 172.21.15.1:0/676831492 shutdown_connections 2024-03-18T20:23:41.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.924+0000 7f27cb673700 1 -- 172.21.15.1:0/676831492 wait complete. 2024-03-18T20:23:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.926+0000 7f27cb673700 1 Processor -- start 2024-03-18T20:23:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.926+0000 7f27cb673700 1 -- start start 2024-03-18T20:23:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.926+0000 7f27cb673700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b0004d90 0x7f27c415ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.926+0000 7f27cb673700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27c414d4a0 0x7f27c4156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.926+0000 7f27cb673700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27c41563d0 0x7f27c41571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.926+0000 7f27cb673700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c4160040 con 0x7f27b0004d90 2024-03-18T20:23:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.926+0000 7f27cb673700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c40b34b0 con 0x7f27c414d4a0 2024-03-18T20:23:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.926+0000 7f27cb673700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c415f4d0 con 0x7f27c41563d0 2024-03-18T20:23:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.926+0000 7f27c940f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b0004d90 0x7f27c415ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:41.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.926+0000 7f27c940f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b0004d90 0x7f27c415ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46548/0 (socket says 172.21.15.1:46548) 2024-03-18T20:23:41.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.927+0000 7f27c940f700 1 -- 172.21.15.1:0/537569321 learned_addr learned my addr 172.21.15.1:0/537569321 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:41.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.927+0000 7f27c940f700 1 -- 172.21.15.1:0/537569321 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27c41563d0 msgr2=0x7f27c41571b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:41.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.927+0000 7f27c940f700 1 --2- 172.21.15.1:0/537569321 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27c41563d0 0x7f27c41571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:41.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.927+0000 7f27c940f700 1 -- 172.21.15.1:0/537569321 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27c414d4a0 msgr2=0x7f27c4156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:41.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.927+0000 7f27c8c0e700 1 --2- 172.21.15.1:0/537569321 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27c414d4a0 0x7f27c4156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:41.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.927+0000 7f27c940f700 1 --2- 172.21.15.1:0/537569321 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27c414d4a0 0x7f27c4156c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:41.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.927+0000 7f27c940f700 1 -- 172.21.15.1:0/537569321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c415f7a0 con 0x7f27b0004d90 2024-03-18T20:23:41.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.927+0000 7f27c940f700 1 --2- 172.21.15.1:0/537569321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b0004d90 0x7f27c415ef90 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f27b40119f0 tx=0x7f27b4011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:41.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.927+0000 7f27b9ffb700 1 -- 172.21.15.1:0/537569321 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b400f8e0 con 0x7f27b0004d90 2024-03-18T20:23:41.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.927+0000 7f27b9ffb700 1 -- 172.21.15.1:0/537569321 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b400fa60 con 0x7f27b0004d90 2024-03-18T20:23:41.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.927+0000 7f27cb673700 1 -- 172.21.15.1:0/537569321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c415fa80 con 0x7f27b0004d90 2024-03-18T20:23:41.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.927+0000 7f27b9ffb700 1 -- 172.21.15.1:0/537569321 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b400fe10 con 0x7f27b0004d90 2024-03-18T20:23:41.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.927+0000 7f27cb673700 1 -- 172.21.15.1:0/537569321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c4158cd0 con 0x7f27b0004d90 2024-03-18T20:23:41.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.929+0000 7f27cb673700 1 -- 172.21.15.1:0/537569321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27c40b3790 con 0x7f27b0004d90 2024-03-18T20:23:41.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.929+0000 7f27b9ffb700 1 -- 172.21.15.1:0/537569321 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f27b4014440 con 0x7f27b0004d90 2024-03-18T20:23:41.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.929+0000 7f27b9ffb700 1 --2- 172.21.15.1:0/537569321 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f27a806adb0 0x7f27a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:41.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.929+0000 7f27b9ffb700 1 -- 172.21.15.1:0/537569321 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f27b4089d00 con 0x7f27b0004d90 2024-03-18T20:23:41.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.929+0000 7f27c8c0e700 1 --2- 172.21.15.1:0/537569321 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f27a806adb0 0x7f27a806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:41.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.930+0000 7f27c8c0e700 1 --2- 172.21.15.1:0/537569321 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f27a806adb0 0x7f27a806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f27c0000c00 tx=0x7f27c0001df0 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:41.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:41.934+0000 7f27b9ffb700 1 -- 172.21.15.1:0/537569321 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f27b4054c20 con 0x7f27b0004d90 2024-03-18T20:23:42.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.183+0000 7f27cb673700 1 -- 172.21.15.1:0/537569321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f27c40676a0 con 0x7f27b0004d90 2024-03-18T20:23:42.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.773+0000 7f27b9ffb700 1 -- 172.21.15.1:0/537569321 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v97) v1 ==== 122+0+0 (secure 0 0 0) 0x7f27b4058240 con 0x7f27b0004d90 2024-03-18T20:23:42.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.775+0000 7f27cb673700 1 -- 172.21.15.1:0/537569321 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f27a806adb0 msgr2=0x7f27a806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:42.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.775+0000 7f27cb673700 1 --2- 172.21.15.1:0/537569321 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f27a806adb0 0x7f27a806d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f27c0000c00 tx=0x7f27c0001df0 comp rx=0 tx=0).stop 2024-03-18T20:23:42.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.776+0000 7f27cb673700 1 -- 172.21.15.1:0/537569321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b0004d90 msgr2=0x7f27c415ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:42.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.776+0000 7f27cb673700 1 --2- 172.21.15.1:0/537569321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b0004d90 0x7f27c415ef90 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f27b40119f0 tx=0x7f27b4011db0 comp rx=0 tx=0).stop 2024-03-18T20:23:42.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.776+0000 7f27cb673700 1 -- 172.21.15.1:0/537569321 shutdown_connections 2024-03-18T20:23:42.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.776+0000 7f27cb673700 1 --2- 172.21.15.1:0/537569321 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f27a806adb0 0x7f27a806d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:42.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.776+0000 7f27cb673700 1 --2- 172.21.15.1:0/537569321 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f27c41563d0 0x7f27c41571b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:42.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.776+0000 7f27cb673700 1 --2- 172.21.15.1:0/537569321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27b0004d90 0x7f27c415ef90 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:42.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.776+0000 7f27cb673700 1 --2- 172.21.15.1:0/537569321 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27c414d4a0 0x7f27c4156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:42.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.776+0000 7f27cb673700 1 -- 172.21.15.1:0/537569321 >> 172.21.15.1:0/537569321 conn(0x7f27c40cc9b0 msgr2=0x7f27c40b0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:42.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.776+0000 7f27cb673700 1 -- 172.21.15.1:0/537569321 shutdown_connections 2024-03-18T20:23:42.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.776+0000 7f27cb673700 1 -- 172.21.15.1:0/537569321 wait complete. 2024-03-18T20:23:42.795 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T20:23:42.796 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:23:42.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.923+0000 7f4c790f8700 1 Processor -- start 2024-03-18T20:23:42.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.923+0000 7f4c790f8700 1 -- start start 2024-03-18T20:23:42.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.924+0000 7f4c790f8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c7406a750 0x7f4c740cf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:42.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.924+0000 7f4c790f8700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4c740cf960 0x7f4c7406ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:42.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.924+0000 7f4c790f8700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c7406d640 con 0x7f4c740cc6d0 2024-03-18T20:23:42.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.924+0000 7f4c790f8700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7406de50 con 0x7f4c7406a750 2024-03-18T20:23:42.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.924+0000 7f4c790f8700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c740acd80 con 0x7f4c740cf960 2024-03-18T20:23:42.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.924+0000 7f4c72d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c7406a750 0x7f4c740cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:42.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.924+0000 7f4c72d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c7406a750 0x7f4c740cf280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46560/0 (socket says 172.21.15.1:46560) 2024-03-18T20:23:42.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.924+0000 7f4c72d9d700 1 -- 172.21.15.1:0/1899080500 learned_addr learned my addr 172.21.15.1:0/1899080500 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:42.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.924+0000 7f4c7259c700 1 --2- 172.21.15.1:0/1899080500 >> v2:172.21.15.165:3300/0 conn(0x7f4c740cf960 0x7f4c7406ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:42.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.925+0000 7f4c72d9d700 1 -- 172.21.15.1:0/1899080500 >> v2:172.21.15.165:3300/0 conn(0x7f4c740cf960 msgr2=0x7f4c7406ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:42.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.925+0000 7f4c72d9d700 1 --2- 172.21.15.1:0/1899080500 >> v2:172.21.15.165:3300/0 conn(0x7f4c740cf960 0x7f4c7406ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:42.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.925+0000 7f4c72d9d700 1 -- 172.21.15.1:0/1899080500 >> v1:172.21.15.165:6789/0 conn(0x7f4c740cc6d0 legacy=0x7f4c740ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:42.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.925+0000 7f4c72d9d700 1 -- 172.21.15.1:0/1899080500 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7406b070 con 0x7f4c7406a750 2024-03-18T20:23:42.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.925+0000 7f4c7259c700 1 --2- 172.21.15.1:0/1899080500 >> v2:172.21.15.165:3300/0 conn(0x7f4c740cf960 0x7f4c7406ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:23:42.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.925+0000 7f4c72d9d700 1 --2- 172.21.15.1:0/1899080500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c7406a750 0x7f4c740cf280 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f4c740715b0 tx=0x7f4c68009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e542e75b82249 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:42.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.925+0000 7f4c71d9b700 1 -- 172.21.15.1:0/1899080500 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6800b870 con 0x7f4c7406a750 2024-03-18T20:23:42.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.925+0000 7f4c71d9b700 1 -- 172.21.15.1:0/1899080500 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6800b9d0 con 0x7f4c7406a750 2024-03-18T20:23:42.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.925+0000 7f4c71d9b700 1 -- 172.21.15.1:0/1899080500 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6800bd60 con 0x7f4c7406a750 2024-03-18T20:23:42.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.925+0000 7f4c790f8700 1 -- 172.21.15.1:0/1899080500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c7406a750 msgr2=0x7f4c740cf280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:42.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.925+0000 7f4c790f8700 1 --2- 172.21.15.1:0/1899080500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c7406a750 0x7f4c740cf280 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f4c740715b0 tx=0x7f4c68009bd0 comp rx=0 tx=0).stop 2024-03-18T20:23:42.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.925+0000 7f4c790f8700 1 -- 172.21.15.1:0/1899080500 shutdown_connections 2024-03-18T20:23:42.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.925+0000 7f4c790f8700 1 --2- 172.21.15.1:0/1899080500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c7406a750 0x7f4c740cf280 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:42.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.925+0000 7f4c790f8700 1 --2- 172.21.15.1:0/1899080500 >> v2:172.21.15.165:3300/0 conn(0x7f4c740cf960 0x7f4c7406ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:42.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.925+0000 7f4c790f8700 1 -- 172.21.15.1:0/1899080500 >> 172.21.15.1:0/1899080500 conn(0x7f4c7406b860 msgr2=0x7f4c7406bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:42.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.926+0000 7f4c790f8700 1 -- 172.21.15.1:0/1899080500 shutdown_connections 2024-03-18T20:23:42.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.926+0000 7f4c790f8700 1 -- 172.21.15.1:0/1899080500 wait complete. 2024-03-18T20:23:42.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.927+0000 7f4c790f8700 1 Processor -- start 2024-03-18T20:23:42.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.927+0000 7f4c790f8700 1 -- start start 2024-03-18T20:23:42.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.928+0000 7f4c790f8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c7406a750 0x7f4c7415d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:42.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.928+0000 7f4c790f8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c740cc6d0 0x7f4c740b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:42.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.928+0000 7f4c790f8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c740cf960 0x7f4c740b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:42.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.928+0000 7f4c790f8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7415d400 con 0x7f4c740cf960 2024-03-18T20:23:42.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.928+0000 7f4c790f8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7415c780 con 0x7f4c740cc6d0 2024-03-18T20:23:42.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.928+0000 7f4c790f8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7415ca60 con 0x7f4c7406a750 2024-03-18T20:23:42.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.928+0000 7f4c7359e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c740cf960 0x7f4c740b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:42.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.928+0000 7f4c7359e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c740cf960 0x7f4c740b48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46570/0 (socket says 172.21.15.1:46570) 2024-03-18T20:23:42.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.928+0000 7f4c7359e700 1 -- 172.21.15.1:0/3362087855 learned_addr learned my addr 172.21.15.1:0/3362087855 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:42.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.928+0000 7f4c7259c700 1 --2- 172.21.15.1:0/3362087855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c740cc6d0 0x7f4c740b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:42.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.928+0000 7f4c72d9d700 1 --2- 172.21.15.1:0/3362087855 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c7406a750 0x7f4c7415d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:42.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.928+0000 7f4c7359e700 1 -- 172.21.15.1:0/3362087855 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c7406a750 msgr2=0x7f4c7415d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:42.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.928+0000 7f4c7359e700 1 --2- 172.21.15.1:0/3362087855 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c7406a750 0x7f4c7415d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:42.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.928+0000 7f4c7359e700 1 -- 172.21.15.1:0/3362087855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c740cc6d0 msgr2=0x7f4c740b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:42.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.928+0000 7f4c7359e700 1 --2- 172.21.15.1:0/3362087855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c740cc6d0 0x7f4c740b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:42.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.928+0000 7f4c7359e700 1 -- 172.21.15.1:0/3362087855 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c68009550 con 0x7f4c740cf960 2024-03-18T20:23:42.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.928+0000 7f4c7359e700 1 --2- 172.21.15.1:0/3362087855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c740cf960 0x7f4c740b48a0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c00cc50 tx=0x7f4c6c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:42.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.929+0000 7f4c5b7fe700 1 -- 172.21.15.1:0/3362087855 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c00ad80 con 0x7f4c740cf960 2024-03-18T20:23:42.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.929+0000 7f4c5b7fe700 1 -- 172.21.15.1:0/3362087855 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6c00ce50 con 0x7f4c740cf960 2024-03-18T20:23:42.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.929+0000 7f4c790f8700 1 -- 172.21.15.1:0/3362087855 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c7415cd40 con 0x7f4c740cf960 2024-03-18T20:23:42.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.929+0000 7f4c5b7fe700 1 -- 172.21.15.1:0/3362087855 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c003c40 con 0x7f4c740cf960 2024-03-18T20:23:42.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.929+0000 7f4c790f8700 1 -- 172.21.15.1:0/3362087855 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c7415dd60 con 0x7f4c740cf960 2024-03-18T20:23:42.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.930+0000 7f4c790f8700 1 -- 172.21.15.1:0/3362087855 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c740676a0 con 0x7f4c740cf960 2024-03-18T20:23:42.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.930+0000 7f4c5b7fe700 1 -- 172.21.15.1:0/3362087855 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4c6c0050d0 con 0x7f4c740cf960 2024-03-18T20:23:42.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.931+0000 7f4c5b7fe700 1 --2- 172.21.15.1:0/3362087855 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c6006adb0 0x7f4c6006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:42.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.931+0000 7f4c72d9d700 1 --2- 172.21.15.1:0/3362087855 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c6006adb0 0x7f4c6006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:42.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.931+0000 7f4c5b7fe700 1 -- 172.21.15.1:0/3362087855 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f4c6c0896e0 con 0x7f4c740cf960 2024-03-18T20:23:42.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.931+0000 7f4c72d9d700 1 --2- 172.21.15.1:0/3362087855 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c6006adb0 0x7f4c6006d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4c68000c00 tx=0x7f4c68008030 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:42.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:42.935+0000 7f4c5b7fe700 1 -- 172.21.15.1:0/3362087855 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4c6c054690 con 0x7f4c740cf960 2024-03-18T20:23:43.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.182+0000 7f4c790f8700 1 -- 172.21.15.1:0/3362087855 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c740b4e90 con 0x7f4c740cf960 2024-03-18T20:23:43.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.182+0000 7f4c5b7fe700 1 -- 172.21.15.1:0/3362087855 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f4c6c057cb0 con 0x7f4c740cf960 2024-03-18T20:23:43.189 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:43.190 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":97,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2241819619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2241819619},{"type":"v1","addr":"172.21.15.165:6835","nonce":2241819619}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/320492236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":320492236},{"type":"v1","addr":"172.21.15.165:6834","nonce":320492236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":11031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/1035910531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1035910531},{"type":"v1","addr":"172.21.15.1:6836","nonce":1035910531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:23:39.747292+0000","modified":"2024-03-18T20:23:42.766952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11034":{"gid":11034,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2388480799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2388480799},{"type":"v1","addr":"172.21.15.1:6837","nonce":2388480799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-03-18T20:23:43.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.185+0000 7f4c790f8700 1 -- 172.21.15.1:0/3362087855 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c6006adb0 msgr2=0x7f4c6006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:43.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.185+0000 7f4c790f8700 1 --2- 172.21.15.1:0/3362087855 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c6006adb0 0x7f4c6006d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4c68000c00 tx=0x7f4c68008030 comp rx=0 tx=0).stop 2024-03-18T20:23:43.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.185+0000 7f4c790f8700 1 -- 172.21.15.1:0/3362087855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c740cf960 msgr2=0x7f4c740b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:43.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.185+0000 7f4c790f8700 1 --2- 172.21.15.1:0/3362087855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c740cf960 0x7f4c740b48a0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c00cc50 tx=0x7f4c6c00a340 comp rx=0 tx=0).stop 2024-03-18T20:23:43.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.185+0000 7f4c790f8700 1 -- 172.21.15.1:0/3362087855 shutdown_connections 2024-03-18T20:23:43.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.185+0000 7f4c790f8700 1 --2- 172.21.15.1:0/3362087855 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c6006adb0 0x7f4c6006d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.185+0000 7f4c790f8700 1 --2- 172.21.15.1:0/3362087855 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c7406a750 0x7f4c7415d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.185+0000 7f4c790f8700 1 --2- 172.21.15.1:0/3362087855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c740cf960 0x7f4c740b48a0 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.185+0000 7f4c790f8700 1 --2- 172.21.15.1:0/3362087855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c740cc6d0 0x7f4c740b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.185+0000 7f4c790f8700 1 -- 172.21.15.1:0/3362087855 >> 172.21.15.1:0/3362087855 conn(0x7f4c7406b860 msgr2=0x7f4c7406bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:43.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.185+0000 7f4c790f8700 1 -- 172.21.15.1:0/3362087855 shutdown_connections 2024-03-18T20:23:43.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.185+0000 7f4c790f8700 1 -- 172.21.15.1:0/3362087855 wait complete. 2024-03-18T20:23:43.192 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 97 2024-03-18T20:23:43.202 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:23:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.322+0000 7fa50be96700 1 Processor -- start 2024-03-18T20:23:43.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.322+0000 7fa50be96700 1 -- start start 2024-03-18T20:23:43.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.323+0000 7fa50be96700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa504146190 0x7fa50406a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:43.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.323+0000 7fa50be96700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa504146af0 0x7fa50406ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:43.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.323+0000 7fa50be96700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa50406ebd0 con 0x7fa5041465a0 2024-03-18T20:23:43.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.323+0000 7fa50be96700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa50406f850 con 0x7fa504146190 2024-03-18T20:23:43.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.323+0000 7fa50be96700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa50406eeb0 con 0x7fa504146af0 2024-03-18T20:23:43.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.323+0000 7fa509c32700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa504146190 0x7fa50406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:43.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.323+0000 7fa509c32700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa504146190 0x7fa50406a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46574/0 (socket says 172.21.15.1:46574) 2024-03-18T20:23:43.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.323+0000 7fa509c32700 1 -- 172.21.15.1:0/3771726210 learned_addr learned my addr 172.21.15.1:0/3771726210 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:43.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.323+0000 7fa509431700 1 --1- 172.21.15.1:0/3771726210 >> v1:172.21.15.1:6789/0 conn(0x7fa5041465a0 0x7fa504071650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45508/0 (socket says 172.21.15.1:45508) 2024-03-18T20:23:43.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.324+0000 7fa50a433700 1 --2- 172.21.15.1:0/3771726210 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa504146af0 0x7fa50406ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:43.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.324+0000 7fa509c32700 1 -- 172.21.15.1:0/3771726210 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa504146af0 msgr2=0x7fa50406ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:43.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.324+0000 7fa509c32700 1 --2- 172.21.15.1:0/3771726210 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa504146af0 0x7fa50406ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.324+0000 7fa509c32700 1 -- 172.21.15.1:0/3771726210 >> v1:172.21.15.1:6789/0 conn(0x7fa5041465a0 legacy=0x7fa504071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.324+0000 7fa508c30700 1 -- 172.21.15.1:0/3771726210 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046637061 0 0) 0x7fa50406ebd0 con 0x7fa5041465a0 2024-03-18T20:23:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.324+0000 7fa509c32700 1 -- 172.21.15.1:0/3771726210 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50406f190 con 0x7fa504146190 2024-03-18T20:23:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.324+0000 7fa50a433700 1 --2- 172.21.15.1:0/3771726210 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa504146af0 0x7fa50406ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:23:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.324+0000 7fa509c32700 1 --2- 172.21.15.1:0/3771726210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa504146190 0x7fa50406a750 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7fa4f40098f0 tx=0x7fa4f4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b294b929b9d8603 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.324+0000 7fa508c30700 1 -- 172.21.15.1:0/3771726210 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f400b8a0 con 0x7fa504146190 2024-03-18T20:23:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.324+0000 7fa508c30700 1 -- 172.21.15.1:0/3771726210 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f400ba00 con 0x7fa504146190 2024-03-18T20:23:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.324+0000 7fa508c30700 1 -- 172.21.15.1:0/3771726210 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f400bd90 con 0x7fa504146190 2024-03-18T20:23:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.325+0000 7fa50be96700 1 -- 172.21.15.1:0/3771726210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa504146190 msgr2=0x7fa50406a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.325+0000 7fa50be96700 1 --2- 172.21.15.1:0/3771726210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa504146190 0x7fa50406a750 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7fa4f40098f0 tx=0x7fa4f4009c00 comp rx=0 tx=0).stop 2024-03-18T20:23:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.325+0000 7fa50be96700 1 -- 172.21.15.1:0/3771726210 shutdown_connections 2024-03-18T20:23:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.325+0000 7fa50be96700 1 --2- 172.21.15.1:0/3771726210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa504146190 0x7fa50406a750 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.325+0000 7fa50be96700 1 --2- 172.21.15.1:0/3771726210 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa504146af0 0x7fa50406ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.325+0000 7fa50be96700 1 -- 172.21.15.1:0/3771726210 >> 172.21.15.1:0/3771726210 conn(0x7fa5040b1d20 msgr2=0x7fa5040b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:43.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.325+0000 7fa50be96700 1 -- 172.21.15.1:0/3771726210 shutdown_connections 2024-03-18T20:23:43.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.325+0000 7fa50be96700 1 -- 172.21.15.1:0/3771726210 wait complete. 2024-03-18T20:23:43.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.326+0000 7fa50be96700 1 Processor -- start 2024-03-18T20:23:43.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.326+0000 7fa50be96700 1 -- start start 2024-03-18T20:23:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa50be96700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5041465a0 0x7fa5040cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa50be96700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa504146af0 0x7fa504144a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa50be96700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5040cb850 0x7fa5040afc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa50be96700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5041f3cc0 con 0x7fa5040cb850 2024-03-18T20:23:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa50be96700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5041f2e30 con 0x7fa5041465a0 2024-03-18T20:23:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa50be96700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5040c5230 con 0x7fa504146af0 2024-03-18T20:23:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa50a433700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5040cb850 0x7fa5040afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa50a433700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5040cb850 0x7fa5040afc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46586/0 (socket says 172.21.15.1:46586) 2024-03-18T20:23:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa50a433700 1 -- 172.21.15.1:0/4019414205 learned_addr learned my addr 172.21.15.1:0/4019414205 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa509c32700 1 --2- 172.21.15.1:0/4019414205 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5041465a0 0x7fa5040cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa509431700 1 --2- 172.21.15.1:0/4019414205 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa504146af0 0x7fa504144a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa50a433700 1 -- 172.21.15.1:0/4019414205 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa504146af0 msgr2=0x7fa504144a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:43.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa50a433700 1 --2- 172.21.15.1:0/4019414205 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa504146af0 0x7fa504144a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa50a433700 1 -- 172.21.15.1:0/4019414205 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5041465a0 msgr2=0x7fa5040cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa50a433700 1 --2- 172.21.15.1:0/4019414205 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5041465a0 0x7fa5040cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa50a433700 1 -- 172.21.15.1:0/4019414205 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f4009550 con 0x7fa5040cb850 2024-03-18T20:23:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa50a433700 1 --2- 172.21.15.1:0/4019414205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5040cb850 0x7fa5040afc70 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc007f20 tx=0x7fa4fc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa4fa7fc700 1 -- 172.21.15.1:0/4019414205 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc00de90 con 0x7fa5040cb850 2024-03-18T20:23:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa4fa7fc700 1 -- 172.21.15.1:0/4019414205 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4fc00f040 con 0x7fa5040cb850 2024-03-18T20:23:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa50be96700 1 -- 172.21.15.1:0/4019414205 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5041f30b0 con 0x7fa5040cb850 2024-03-18T20:23:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.327+0000 7fa4fa7fc700 1 -- 172.21.15.1:0/4019414205 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc003c60 con 0x7fa5040cb850 2024-03-18T20:23:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.328+0000 7fa50be96700 1 -- 172.21.15.1:0/4019414205 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5041f3670 con 0x7fa5040cb850 2024-03-18T20:23:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.328+0000 7fa50be96700 1 -- 172.21.15.1:0/4019414205 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5040676a0 con 0x7fa5040cb850 2024-03-18T20:23:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.329+0000 7fa4fa7fc700 1 -- 172.21.15.1:0/4019414205 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa4fc004030 con 0x7fa5040cb850 2024-03-18T20:23:43.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.329+0000 7fa4fa7fc700 1 --2- 172.21.15.1:0/4019414205 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa4e806adb0 0x7fa4e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:43.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.329+0000 7fa4fa7fc700 1 -- 172.21.15.1:0/4019414205 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fa4fc089930 con 0x7fa5040cb850 2024-03-18T20:23:43.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.329+0000 7fa509c32700 1 --2- 172.21.15.1:0/4019414205 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa4e806adb0 0x7fa4e806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:43.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.330+0000 7fa509c32700 1 --2- 172.21.15.1:0/4019414205 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa4e806adb0 0x7fa4e806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4014040 tx=0x7fa4f4008030 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:43.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.334+0000 7fa4fa7fc700 1 -- 172.21.15.1:0/4019414205 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa4fc054740 con 0x7fa5040cb850 2024-03-18T20:23:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.537+0000 7fa50be96700 1 -- 172.21.15.1:0/4019414205 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5041f3e00 con 0x7fa5040cb850 2024-03-18T20:23:43.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.538+0000 7fa4fa7fc700 1 -- 172.21.15.1:0/4019414205 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v214) v1 ==== 74+0+16365 (secure 0 0 0) 0x7fa5041f3e00 con 0x7fa5040cb850 2024-03-18T20:23:43.543 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:43.543 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":214,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:23:40.748873+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:23:37.591406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-03-18T20:23:38.723169+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:23:43.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.540+0000 7fa50be96700 1 -- 172.21.15.1:0/4019414205 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa4e806adb0 msgr2=0x7fa4e806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:43.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.541+0000 7fa50be96700 1 --2- 172.21.15.1:0/4019414205 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa4e806adb0 0x7fa4e806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4014040 tx=0x7fa4f4008030 comp rx=0 tx=0).stop 2024-03-18T20:23:43.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.541+0000 7fa50be96700 1 -- 172.21.15.1:0/4019414205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5040cb850 msgr2=0x7fa5040afc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:43.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.541+0000 7fa50be96700 1 --2- 172.21.15.1:0/4019414205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5040cb850 0x7fa5040afc70 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc007f20 tx=0x7fa4fc00d3b0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.541+0000 7fa50be96700 1 -- 172.21.15.1:0/4019414205 shutdown_connections 2024-03-18T20:23:43.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.541+0000 7fa50be96700 1 --2- 172.21.15.1:0/4019414205 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa4e806adb0 0x7fa4e806d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.541+0000 7fa50be96700 1 --2- 172.21.15.1:0/4019414205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5040cb850 0x7fa5040afc70 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.541+0000 7fa50be96700 1 --2- 172.21.15.1:0/4019414205 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa5041465a0 0x7fa5040cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.541+0000 7fa50be96700 1 --2- 172.21.15.1:0/4019414205 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa504146af0 0x7fa504144a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.541+0000 7fa50be96700 1 -- 172.21.15.1:0/4019414205 >> 172.21.15.1:0/4019414205 conn(0x7fa5040b1d20 msgr2=0x7fa5040ae700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:43.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.541+0000 7fa50be96700 1 -- 172.21.15.1:0/4019414205 shutdown_connections 2024-03-18T20:23:43.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.541+0000 7fa50be96700 1 -- 172.21.15.1:0/4019414205 wait complete. 2024-03-18T20:23:43.558 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T20:23:43.558 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:23:43.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.678+0000 7f30f829e700 1 Processor -- start 2024-03-18T20:23:43.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.678+0000 7f30f829e700 1 -- start start 2024-03-18T20:23:43.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.680+0000 7f30f829e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f30f00cc170 0x7f30f00ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:43.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.680+0000 7f30f829e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f006e270 con 0x7f30f00ae600 2024-03-18T20:23:43.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.680+0000 7f30f829e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f00ae050 con 0x7f30f015a2e0 2024-03-18T20:23:43.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.680+0000 7f30f829e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30f006f690 con 0x7f30f00cc170 2024-03-18T20:23:43.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.680+0000 7f30f603a700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f30f00ae600 0x7f30f00ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45522/0 (socket says 172.21.15.1:45522) 2024-03-18T20:23:43.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.680+0000 7f30f683b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f30f00cc170 0x7f30f00ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:43.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.680+0000 7f30f603a700 1 -- 172.21.15.1:0/3336862793 learned_addr learned my addr 172.21.15.1:0/3336862793 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:43.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.680+0000 7f30f683b700 1 -- 172.21.15.1:0/3336862793 >> v1:172.21.15.165:6789/0 conn(0x7f30f015a2e0 legacy=0x7f30f006ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:43.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.680+0000 7f30f683b700 1 -- 172.21.15.1:0/3336862793 >> v1:172.21.15.1:6789/0 conn(0x7f30f00ae600 legacy=0x7f30f00ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:43.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.680+0000 7f30f683b700 1 -- 172.21.15.1:0/3336862793 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f00add70 con 0x7f30f00cc170 2024-03-18T20:23:43.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.680+0000 7f30f5038700 1 -- 172.21.15.1:0/3336862793 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549777188 0 0) 0x7f30f006e270 con 0x7f30f00ae600 2024-03-18T20:23:43.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.680+0000 7f30f683b700 1 --2- 172.21.15.1:0/3336862793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f00cc170 0x7f30f00ad790 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f30e8009d90 tx=0x7f30e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291dc1651c9143ab server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:43.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.680+0000 7f30f5038700 1 -- 172.21.15.1:0/3336862793 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e800e070 con 0x7f30f00cc170 2024-03-18T20:23:43.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.681+0000 7f30f5038700 1 -- 172.21.15.1:0/3336862793 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e800bbd0 con 0x7f30f00cc170 2024-03-18T20:23:43.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.681+0000 7f30f5038700 1 -- 172.21.15.1:0/3336862793 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e8003a20 con 0x7f30f00cc170 2024-03-18T20:23:43.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.681+0000 7f30f829e700 1 -- 172.21.15.1:0/3336862793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f00cc170 msgr2=0x7f30f00ad790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:43.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.681+0000 7f30f829e700 1 --2- 172.21.15.1:0/3336862793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f00cc170 0x7f30f00ad790 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f30e8009d90 tx=0x7f30e800b330 comp rx=0 tx=0).stop 2024-03-18T20:23:43.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.681+0000 7f30f829e700 1 -- 172.21.15.1:0/3336862793 shutdown_connections 2024-03-18T20:23:43.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.681+0000 7f30f829e700 1 --2- 172.21.15.1:0/3336862793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f00cc170 0x7f30f00ad790 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.681+0000 7f30f829e700 1 -- 172.21.15.1:0/3336862793 >> 172.21.15.1:0/3336862793 conn(0x7f30f006b860 msgr2=0x7f30f006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:43.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.681+0000 7f30f829e700 1 -- 172.21.15.1:0/3336862793 shutdown_connections 2024-03-18T20:23:43.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.681+0000 7f30f829e700 1 -- 172.21.15.1:0/3336862793 wait complete. 2024-03-18T20:23:43.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.682+0000 7f30f829e700 1 Processor -- start 2024-03-18T20:23:43.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.682+0000 7f30f829e700 1 -- start start 2024-03-18T20:23:43.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.683+0000 7f30f829e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f00ae600 0x7f30f014c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:43.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.683+0000 7f30f829e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30f00cc170 0x7f30f00ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:43.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.683+0000 7f30f829e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30f015a2e0 0x7f30f014be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:43.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.683+0000 7f30f829e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f014cf00 con 0x7f30f00ae600 2024-03-18T20:23:43.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.683+0000 7f30f829e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f00cecd0 con 0x7f30f00cc170 2024-03-18T20:23:43.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.683+0000 7f30f829e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f00cefb0 con 0x7f30f015a2e0 2024-03-18T20:23:43.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.683+0000 7f30f603a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f00ae600 0x7f30f014c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:43.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.683+0000 7f30f603a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f00ae600 0x7f30f014c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46606/0 (socket says 172.21.15.1:46606) 2024-03-18T20:23:43.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.683+0000 7f30f603a700 1 -- 172.21.15.1:0/1055066523 learned_addr learned my addr 172.21.15.1:0/1055066523 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:43.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.683+0000 7f30f5839700 1 --2- 172.21.15.1:0/1055066523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30f00cc170 0x7f30f00ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:43.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.683+0000 7f30f683b700 1 --2- 172.21.15.1:0/1055066523 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30f015a2e0 0x7f30f014be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:43.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.684+0000 7f30f603a700 1 -- 172.21.15.1:0/1055066523 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30f015a2e0 msgr2=0x7f30f014be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:43.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.684+0000 7f30f603a700 1 --2- 172.21.15.1:0/1055066523 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30f015a2e0 0x7f30f014be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.684+0000 7f30f603a700 1 -- 172.21.15.1:0/1055066523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30f00cc170 msgr2=0x7f30f00ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:43.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.684+0000 7f30f603a700 1 --2- 172.21.15.1:0/1055066523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30f00cc170 0x7f30f00ce5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.684+0000 7f30f603a700 1 -- 172.21.15.1:0/1055066523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e8009a70 con 0x7f30f00ae600 2024-03-18T20:23:43.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.684+0000 7f30f603a700 1 --2- 172.21.15.1:0/1055066523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f00ae600 0x7f30f014c600 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f30ec0011e0 tx=0x7f30ec00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:43.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.684+0000 7f30e2ffd700 1 -- 172.21.15.1:0/1055066523 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec004c80 con 0x7f30f00ae600 2024-03-18T20:23:43.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.684+0000 7f30e2ffd700 1 -- 172.21.15.1:0/1055066523 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ec004e00 con 0x7f30f00ae600 2024-03-18T20:23:43.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.684+0000 7f30f829e700 1 -- 172.21.15.1:0/1055066523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f00cc630 con 0x7f30f00ae600 2024-03-18T20:23:43.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.684+0000 7f30e2ffd700 1 -- 172.21.15.1:0/1055066523 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec007310 con 0x7f30f00ae600 2024-03-18T20:23:43.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.684+0000 7f30f829e700 1 -- 172.21.15.1:0/1055066523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f00ccbf0 con 0x7f30f00ae600 2024-03-18T20:23:43.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.685+0000 7f30f829e700 1 -- 172.21.15.1:0/1055066523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f00cce10 con 0x7f30f00ae600 2024-03-18T20:23:43.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.685+0000 7f30e2ffd700 1 -- 172.21.15.1:0/1055066523 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f30ec007470 con 0x7f30f00ae600 2024-03-18T20:23:43.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.686+0000 7f30e2ffd700 1 --2- 172.21.15.1:0/1055066523 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30d406adb0 0x7f30d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:43.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.686+0000 7f30e2ffd700 1 -- 172.21.15.1:0/1055066523 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f30ec089340 con 0x7f30f00ae600 2024-03-18T20:23:43.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.686+0000 7f30f5839700 1 --2- 172.21.15.1:0/1055066523 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30d406adb0 0x7f30d406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:43.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.686+0000 7f30f5839700 1 --2- 172.21.15.1:0/1055066523 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30d406adb0 0x7f30d406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f30e4009870 tx=0x7f30e4008040 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:43.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.691+0000 7f30e2ffd700 1 -- 172.21.15.1:0/1055066523 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f30ec0542f0 con 0x7f30f00ae600 2024-03-18T20:23:43.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.889+0000 7f30f829e700 1 -- 172.21.15.1:0/1055066523 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f30f006ea60 con 0x7f30d406adb0 2024-03-18T20:23:43.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.898+0000 7f30e2ffd700 1 -- 172.21.15.1:0/1055066523 <== mgr.10797 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290017 (secure 0 0 0) 0x7f30e40123e0 con 0x7f30d406adb0 2024-03-18T20:23:43.906 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:43.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.901+0000 7f30f829e700 1 -- 172.21.15.1:0/1055066523 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30d406adb0 msgr2=0x7f30d406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.901+0000 7f30f829e700 1 --2- 172.21.15.1:0/1055066523 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30d406adb0 0x7f30d406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f30e4009870 tx=0x7f30e4008040 comp rx=0 tx=0).stop 2024-03-18T20:23:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.901+0000 7f30f829e700 1 -- 172.21.15.1:0/1055066523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f00ae600 msgr2=0x7f30f014c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.901+0000 7f30f829e700 1 --2- 172.21.15.1:0/1055066523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f00ae600 0x7f30f014c600 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f30ec0011e0 tx=0x7f30ec00cae0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.901+0000 7f30f829e700 1 -- 172.21.15.1:0/1055066523 shutdown_connections 2024-03-18T20:23:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.901+0000 7f30f829e700 1 --2- 172.21.15.1:0/1055066523 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30d406adb0 0x7f30d406d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.901+0000 7f30f829e700 1 --2- 172.21.15.1:0/1055066523 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30f015a2e0 0x7f30f014be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.901+0000 7f30f829e700 1 --2- 172.21.15.1:0/1055066523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f00ae600 0x7f30f014c600 unknown :-1 s=CLOSED pgs=2346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.901+0000 7f30f829e700 1 --2- 172.21.15.1:0/1055066523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30f00cc170 0x7f30f00ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.901+0000 7f30f829e700 1 -- 172.21.15.1:0/1055066523 >> 172.21.15.1:0/1055066523 conn(0x7f30f006b860 msgr2=0x7f30f01578d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.901+0000 7f30f829e700 1 -- 172.21.15.1:0/1055066523 shutdown_connections 2024-03-18T20:23:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:43.901+0000 7f30f829e700 1 -- 172.21.15.1:0/1055066523 wait complete. 2024-03-18T20:23:43.909 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:23:43.918 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-18T20:23:42.189956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":131,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":395768,"kb_used_data":22136,"kb_used_omap":12,"kb_used_meta":373491,"kb_avail":754578952,"statfs":{"total":773094113280,"available":772688846848,"internally_reserved":0,"allocated":22667264,"data_stored":12126748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":382455353},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000798"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590951+0000","last_change":"2024-03-18T20:23:39.775368+0000","last_active":"2024-03-18T20:23:41.590951+0000","last_peered":"2024-03-18T20:23:41.590951+0000","last_clean":"2024-03-18T20:23:41.590951+0000","last_became_active":"2024-03-18T20:23:39.775103+0000","last_became_peered":"2024-03-18T20:23:39.775103+0000","last_unstale":"2024-03-18T20:23:41.590951+0000","last_undegraded":"2024-03-18T20:23:41.590951+0000","last_fullsized":"2024-03-18T20:23:41.590951+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:56.346368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590813+0000","last_change":"2024-03-18T20:23:39.767233+0000","last_active":"2024-03-18T20:23:41.590813+0000","last_peered":"2024-03-18T20:23:41.590813+0000","last_clean":"2024-03-18T20:23:41.590813+0000","last_became_active":"2024-03-18T20:23:39.766984+0000","last_became_peered":"2024-03-18T20:23:39.766984+0000","last_unstale":"2024-03-18T20:23:41.590813+0000","last_undegraded":"2024-03-18T20:23:41.590813+0000","last_fullsized":"2024-03-18T20:23:41.590813+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:02:54.102503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763218+0000","last_change":"2024-03-18T20:23:39.761925+0000","last_active":"2024-03-18T20:23:40.763218+0000","last_peered":"2024-03-18T20:23:40.763218+0000","last_clean":"2024-03-18T20:23:40.763218+0000","last_became_active":"2024-03-18T20:23:39.761679+0000","last_became_peered":"2024-03-18T20:23:39.761679+0000","last_unstale":"2024-03-18T20:23:40.763218+0000","last_undegraded":"2024-03-18T20:23:40.763218+0000","last_fullsized":"2024-03-18T20:23:40.763218+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:55:17.412986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.774748+0000","last_change":"2024-03-18T20:23:39.772902+0000","last_active":"2024-03-18T20:23:39.774748+0000","last_peered":"2024-03-18T20:23:39.774748+0000","last_clean":"2024-03-18T20:23:39.774748+0000","last_became_active":"2024-03-18T20:23:39.772675+0000","last_became_peered":"2024-03-18T20:23:39.772675+0000","last_unstale":"2024-03-18T20:23:39.774748+0000","last_undegraded":"2024-03-18T20:23:39.774748+0000","last_fullsized":"2024-03-18T20:23:39.774748+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:12:18.174454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780861+0000","last_change":"2024-03-18T20:23:39.772041+0000","last_active":"2024-03-18T20:23:39.780861+0000","last_peered":"2024-03-18T20:23:39.780861+0000","last_clean":"2024-03-18T20:23:39.780861+0000","last_became_active":"2024-03-18T20:23:39.771843+0000","last_became_peered":"2024-03-18T20:23:39.771843+0000","last_unstale":"2024-03-18T20:23:39.780861+0000","last_undegraded":"2024-03-18T20:23:39.780861+0000","last_fullsized":"2024-03-18T20:23:39.780861+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:24:28.800911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780896+0000","last_change":"2024-03-18T20:23:39.775650+0000","last_active":"2024-03-18T20:23:39.780896+0000","last_peered":"2024-03-18T20:23:39.780896+0000","last_clean":"2024-03-18T20:23:39.780896+0000","last_became_active":"2024-03-18T20:23:39.775190+0000","last_became_peered":"2024-03-18T20:23:39.775190+0000","last_unstale":"2024-03-18T20:23:39.780896+0000","last_undegraded":"2024-03-18T20:23:39.780896+0000","last_fullsized":"2024-03-18T20:23:39.780896+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:48.403628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763368+0000","last_change":"2024-03-18T20:23:39.772541+0000","last_active":"2024-03-18T20:23:40.763368+0000","last_peered":"2024-03-18T20:23:40.763368+0000","last_clean":"2024-03-18T20:23:40.763368+0000","last_became_active":"2024-03-18T20:23:39.764037+0000","last_became_peered":"2024-03-18T20:23:39.764037+0000","last_unstale":"2024-03-18T20:23:40.763368+0000","last_undegraded":"2024-03-18T20:23:40.763368+0000","last_fullsized":"2024-03-18T20:23:40.763368+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:12.307651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:42.006874+0000","last_change":"2024-03-18T20:23:39.774083+0000","last_active":"2024-03-18T20:23:42.006874+0000","last_peered":"2024-03-18T20:23:42.006874+0000","last_clean":"2024-03-18T20:23:42.006874+0000","last_became_active":"2024-03-18T20:23:39.773016+0000","last_became_peered":"2024-03-18T20:23:39.773016+0000","last_unstale":"2024-03-18T20:23:42.006874+0000","last_undegraded":"2024-03-18T20:23:42.006874+0000","last_fullsized":"2024-03-18T20:23:42.006874+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:01:47.194750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590887+0000","last_change":"2024-03-18T20:23:39.767854+0000","last_active":"2024-03-18T20:23:41.590887+0000","last_peered":"2024-03-18T20:23:41.590887+0000","last_clean":"2024-03-18T20:23:41.590887+0000","last_became_active":"2024-03-18T20:23:39.767614+0000","last_became_peered":"2024-03-18T20:23:39.767614+0000","last_unstale":"2024-03-18T20:23:41.590887+0000","last_undegraded":"2024-03-18T20:23:41.590887+0000","last_fullsized":"2024-03-18T20:23:41.590887+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:00:42.414528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.781163+0000","last_change":"2024-03-18T20:23:39.778000+0000","last_active":"2024-03-18T20:23:39.781163+0000","last_peered":"2024-03-18T20:23:39.781163+0000","last_clean":"2024-03-18T20:23:39.781163+0000","last_became_active":"2024-03-18T20:23:39.777388+0000","last_became_peered":"2024-03-18T20:23:39.777388+0000","last_unstale":"2024-03-18T20:23:39.781163+0000","last_undegraded":"2024-03-18T20:23:39.781163+0000","last_fullsized":"2024-03-18T20:23:39.781163+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:14.173247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.006906+0000","last_change":"2024-03-18T20:23:39.763882+0000","last_active":"2024-03-18T20:23:41.006906+0000","last_peered":"2024-03-18T20:23:41.006906+0000","last_clean":"2024-03-18T20:23:41.006906+0000","last_became_active":"2024-03-18T20:23:39.763608+0000","last_became_peered":"2024-03-18T20:23:39.763608+0000","last_unstale":"2024-03-18T20:23:41.006906+0000","last_undegraded":"2024-03-18T20:23:41.006906+0000","last_fullsized":"2024-03-18T20:23:41.006906+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:59.000708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.770769+0000","last_change":"2024-03-18T20:23:39.766381+0000","last_active":"2024-03-18T20:23:39.770769+0000","last_peered":"2024-03-18T20:23:39.770769+0000","last_clean":"2024-03-18T20:23:39.770769+0000","last_became_active":"2024-03-18T20:23:39.766058+0000","last_became_peered":"2024-03-18T20:23:39.766058+0000","last_unstale":"2024-03-18T20:23:39.770769+0000","last_undegraded":"2024-03-18T20:23:39.770769+0000","last_fullsized":"2024-03-18T20:23:39.770769+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:26.058894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970640+0000","last_change":"2024-03-18T20:23:39.758470+0000","last_active":"2024-03-18T20:23:40.970640+0000","last_peered":"2024-03-18T20:23:40.970640+0000","last_clean":"2024-03-18T20:23:40.970640+0000","last_became_active":"2024-03-18T20:23:39.758170+0000","last_became_peered":"2024-03-18T20:23:39.758170+0000","last_unstale":"2024-03-18T20:23:40.970640+0000","last_undegraded":"2024-03-18T20:23:40.970640+0000","last_fullsized":"2024-03-18T20:23:40.970640+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:41.949264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763446+0000","last_change":"2024-03-18T20:23:39.773025+0000","last_active":"2024-03-18T20:23:40.763446+0000","last_peered":"2024-03-18T20:23:40.763446+0000","last_clean":"2024-03-18T20:23:40.763446+0000","last_became_active":"2024-03-18T20:23:39.772723+0000","last_became_peered":"2024-03-18T20:23:39.772723+0000","last_unstale":"2024-03-18T20:23:40.763446+0000","last_undegraded":"2024-03-18T20:23:40.763446+0000","last_fullsized":"2024-03-18T20:23:40.763446+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:02:35.258826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590515+0000","last_change":"2024-03-18T20:23:39.767200+0000","last_active":"2024-03-18T20:23:41.590515+0000","last_peered":"2024-03-18T20:23:41.590515+0000","last_clean":"2024-03-18T20:23:41.590515+0000","last_became_active":"2024-03-18T20:23:39.767035+0000","last_became_peered":"2024-03-18T20:23:39.767035+0000","last_unstale":"2024-03-18T20:23:41.590515+0000","last_undegraded":"2024-03-18T20:23:41.590515+0000","last_fullsized":"2024-03-18T20:23:41.590515+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:35.116397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780705+0000","last_change":"2024-03-18T20:23:39.771329+0000","last_active":"2024-03-18T20:23:39.780705+0000","last_peered":"2024-03-18T20:23:39.780705+0000","last_clean":"2024-03-18T20:23:39.780705+0000","last_became_active":"2024-03-18T20:23:39.768752+0000","last_became_peered":"2024-03-18T20:23:39.768752+0000","last_unstale":"2024-03-18T20:23:39.780705+0000","last_undegraded":"2024-03-18T20:23:39.780705+0000","last_fullsized":"2024-03-18T20:23:39.780705+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:33:12.771579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780905+0000","last_change":"2024-03-18T20:23:39.775925+0000","last_active":"2024-03-18T20:23:39.780905+0000","last_peered":"2024-03-18T20:23:39.780905+0000","last_clean":"2024-03-18T20:23:39.780905+0000","last_became_active":"2024-03-18T20:23:39.775636+0000","last_became_peered":"2024-03-18T20:23:39.775636+0000","last_unstale":"2024-03-18T20:23:39.780905+0000","last_undegraded":"2024-03-18T20:23:39.780905+0000","last_fullsized":"2024-03-18T20:23:39.780905+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:34:32.364043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.971006+0000","last_change":"2024-03-18T20:23:39.765350+0000","last_active":"2024-03-18T20:23:40.971006+0000","last_peered":"2024-03-18T20:23:40.971006+0000","last_clean":"2024-03-18T20:23:40.971006+0000","last_became_active":"2024-03-18T20:23:39.765082+0000","last_became_peered":"2024-03-18T20:23:39.765082+0000","last_unstale":"2024-03-18T20:23:40.971006+0000","last_undegraded":"2024-03-18T20:23:40.971006+0000","last_fullsized":"2024-03-18T20:23:40.971006+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:59.000708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970906+0000","last_change":"2024-03-18T20:23:39.762644+0000","last_active":"2024-03-18T20:23:40.970906+0000","last_peered":"2024-03-18T20:23:40.970906+0000","last_clean":"2024-03-18T20:23:40.970906+0000","last_became_active":"2024-03-18T20:23:39.760016+0000","last_became_peered":"2024-03-18T20:23:39.760016+0000","last_unstale":"2024-03-18T20:23:40.970906+0000","last_undegraded":"2024-03-18T20:23:40.970906+0000","last_fullsized":"2024-03-18T20:23:40.970906+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:21:09.949066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780772+0000","last_change":"2024-03-18T20:23:39.772736+0000","last_active":"2024-03-18T20:23:39.780772+0000","last_peered":"2024-03-18T20:23:39.780772+0000","last_clean":"2024-03-18T20:23:39.780772+0000","last_became_active":"2024-03-18T20:23:39.772317+0000","last_became_peered":"2024-03-18T20:23:39.772317+0000","last_unstale":"2024-03-18T20:23:39.780772+0000","last_undegraded":"2024-03-18T20:23:39.780772+0000","last_fullsized":"2024-03-18T20:23:39.780772+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:46:57.873851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590894+0000","last_change":"2024-03-18T20:23:39.765153+0000","last_active":"2024-03-18T20:23:41.590894+0000","last_peered":"2024-03-18T20:23:41.590894+0000","last_clean":"2024-03-18T20:23:41.590894+0000","last_became_active":"2024-03-18T20:23:39.764882+0000","last_became_peered":"2024-03-18T20:23:39.764882+0000","last_unstale":"2024-03-18T20:23:41.590894+0000","last_undegraded":"2024-03-18T20:23:41.590894+0000","last_fullsized":"2024-03-18T20:23:41.590894+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:26.058894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590827+0000","last_change":"2024-03-18T20:23:39.766755+0000","last_active":"2024-03-18T20:23:41.590827+0000","last_peered":"2024-03-18T20:23:41.590827+0000","last_clean":"2024-03-18T20:23:41.590827+0000","last_became_active":"2024-03-18T20:23:39.766368+0000","last_became_peered":"2024-03-18T20:23:39.766368+0000","last_unstale":"2024-03-18T20:23:41.590827+0000","last_undegraded":"2024-03-18T20:23:41.590827+0000","last_fullsized":"2024-03-18T20:23:41.590827+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:51:11.509782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:42.006588+0000","last_change":"2024-03-18T20:23:39.771477+0000","last_active":"2024-03-18T20:23:42.006588+0000","last_peered":"2024-03-18T20:23:42.006588+0000","last_clean":"2024-03-18T20:23:42.006588+0000","last_became_active":"2024-03-18T20:23:39.771245+0000","last_became_peered":"2024-03-18T20:23:39.771245+0000","last_unstale":"2024-03-18T20:23:42.006588+0000","last_undegraded":"2024-03-18T20:23:42.006588+0000","last_fullsized":"2024-03-18T20:23:42.006588+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:10.588668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590402+0000","last_change":"2024-03-18T20:23:39.764714+0000","last_active":"2024-03-18T20:23:41.590402+0000","last_peered":"2024-03-18T20:23:41.590402+0000","last_clean":"2024-03-18T20:23:41.590402+0000","last_became_active":"2024-03-18T20:23:39.764385+0000","last_became_peered":"2024-03-18T20:23:39.764385+0000","last_unstale":"2024-03-18T20:23:41.590402+0000","last_undegraded":"2024-03-18T20:23:41.590402+0000","last_fullsized":"2024-03-18T20:23:41.590402+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:26:24.225550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763632+0000","last_change":"2024-03-18T20:23:39.778928+0000","last_active":"2024-03-18T20:23:40.763632+0000","last_peered":"2024-03-18T20:23:40.763632+0000","last_clean":"2024-03-18T20:23:40.763632+0000","last_became_active":"2024-03-18T20:23:39.778655+0000","last_became_peered":"2024-03-18T20:23:39.778655+0000","last_unstale":"2024-03-18T20:23:40.763632+0000","last_undegraded":"2024-03-18T20:23:40.763632+0000","last_fullsized":"2024-03-18T20:23:40.763632+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:03:54.384457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.007060+0000","last_change":"2024-03-18T20:23:39.771221+0000","last_active":"2024-03-18T20:23:41.007060+0000","last_peered":"2024-03-18T20:23:41.007060+0000","last_clean":"2024-03-18T20:23:41.007060+0000","last_became_active":"2024-03-18T20:23:39.768932+0000","last_became_peered":"2024-03-18T20:23:39.768932+0000","last_unstale":"2024-03-18T20:23:41.007060+0000","last_undegraded":"2024-03-18T20:23:41.007060+0000","last_fullsized":"2024-03-18T20:23:41.007060+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:41.949264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:42.006688+0000","last_change":"2024-03-18T20:23:39.764476+0000","last_active":"2024-03-18T20:23:42.006688+0000","last_peered":"2024-03-18T20:23:42.006688+0000","last_clean":"2024-03-18T20:23:42.006688+0000","last_became_active":"2024-03-18T20:23:39.764132+0000","last_became_peered":"2024-03-18T20:23:39.764132+0000","last_unstale":"2024-03-18T20:23:42.006688+0000","last_undegraded":"2024-03-18T20:23:42.006688+0000","last_fullsized":"2024-03-18T20:23:42.006688+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:04:48.691324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780654+0000","last_change":"2024-03-18T20:23:39.768930+0000","last_active":"2024-03-18T20:23:39.780654+0000","last_peered":"2024-03-18T20:23:39.780654+0000","last_clean":"2024-03-18T20:23:39.780654+0000","last_became_active":"2024-03-18T20:23:39.768653+0000","last_became_peered":"2024-03-18T20:23:39.768653+0000","last_unstale":"2024-03-18T20:23:39.780654+0000","last_undegraded":"2024-03-18T20:23:39.780654+0000","last_fullsized":"2024-03-18T20:23:39.780654+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:51:42.338719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.006883+0000","last_change":"2024-03-18T20:23:39.768945+0000","last_active":"2024-03-18T20:23:41.006883+0000","last_peered":"2024-03-18T20:23:41.006883+0000","last_clean":"2024-03-18T20:23:41.006883+0000","last_became_active":"2024-03-18T20:23:39.768666+0000","last_became_peered":"2024-03-18T20:23:39.768666+0000","last_unstale":"2024-03-18T20:23:41.006883+0000","last_undegraded":"2024-03-18T20:23:41.006883+0000","last_fullsized":"2024-03-18T20:23:41.006883+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:01:47.194750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.006745+0000","last_change":"2024-03-18T20:23:39.768712+0000","last_active":"2024-03-18T20:23:41.006745+0000","last_peered":"2024-03-18T20:23:41.006745+0000","last_clean":"2024-03-18T20:23:41.006745+0000","last_became_active":"2024-03-18T20:23:39.768441+0000","last_became_peered":"2024-03-18T20:23:39.768441+0000","last_unstale":"2024-03-18T20:23:41.006745+0000","last_undegraded":"2024-03-18T20:23:41.006745+0000","last_fullsized":"2024-03-18T20:23:41.006745+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:10.588668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763495+0000","last_change":"2024-03-18T20:23:39.775405+0000","last_active":"2024-03-18T20:23:40.763495+0000","last_peered":"2024-03-18T20:23:40.763495+0000","last_clean":"2024-03-18T20:23:40.763495+0000","last_became_active":"2024-03-18T20:23:39.775096+0000","last_became_peered":"2024-03-18T20:23:39.775096+0000","last_unstale":"2024-03-18T20:23:40.763495+0000","last_undegraded":"2024-03-18T20:23:40.763495+0000","last_fullsized":"2024-03-18T20:23:40.763495+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:56.694065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.781335+0000","last_change":"2024-03-18T20:23:39.779447+0000","last_active":"2024-03-18T20:23:39.781335+0000","last_peered":"2024-03-18T20:23:39.781335+0000","last_clean":"2024-03-18T20:23:39.781335+0000","last_became_active":"2024-03-18T20:23:39.769075+0000","last_became_peered":"2024-03-18T20:23:39.769075+0000","last_unstale":"2024-03-18T20:23:39.781335+0000","last_undegraded":"2024-03-18T20:23:39.781335+0000","last_fullsized":"2024-03-18T20:23:39.781335+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:11.412667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:42.006835+0000","last_change":"2024-03-18T20:23:39.763471+0000","last_active":"2024-03-18T20:23:42.006835+0000","last_peered":"2024-03-18T20:23:42.006835+0000","last_clean":"2024-03-18T20:23:42.006835+0000","last_became_active":"2024-03-18T20:23:39.763142+0000","last_became_peered":"2024-03-18T20:23:39.763142+0000","last_unstale":"2024-03-18T20:23:42.006835+0000","last_undegraded":"2024-03-18T20:23:42.006835+0000","last_fullsized":"2024-03-18T20:23:42.006835+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:37.066466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"213'4","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.778347+0000","last_change":"2024-03-18T20:23:39.337683+0000","last_active":"2024-03-18T20:23:39.778347+0000","last_peered":"2024-03-18T20:23:39.778347+0000","last_clean":"2024-03-18T20:23:39.778347+0000","last_became_active":"2024-03-18T20:23:39.337383+0000","last_became_peered":"2024-03-18T20:23:39.337383+0000","last_unstale":"2024-03-18T20:23:39.778347+0000","last_undegraded":"2024-03-18T20:23:39.778347+0000","last_fullsized":"2024-03-18T20:23:39.778347+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:02:01.706948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763702+0000","last_change":"2024-03-18T20:23:39.778651+0000","last_active":"2024-03-18T20:23:40.763702+0000","last_peered":"2024-03-18T20:23:40.763702+0000","last_clean":"2024-03-18T20:23:40.763702+0000","last_became_active":"2024-03-18T20:23:39.773401+0000","last_became_peered":"2024-03-18T20:23:39.773401+0000","last_unstale":"2024-03-18T20:23:40.763702+0000","last_undegraded":"2024-03-18T20:23:40.763702+0000","last_fullsized":"2024-03-18T20:23:40.763702+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:27:50.812568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"213'1","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.781104+0000","last_change":"2024-03-18T20:23:39.336840+0000","last_active":"2024-03-18T20:23:39.781104+0000","last_peered":"2024-03-18T20:23:39.781104+0000","last_clean":"2024-03-18T20:23:39.781104+0000","last_became_active":"2024-03-18T20:23:39.336484+0000","last_became_peered":"2024-03-18T20:23:39.336484+0000","last_unstale":"2024-03-18T20:23:39.781104+0000","last_undegraded":"2024-03-18T20:23:39.781104+0000","last_fullsized":"2024-03-18T20:23:39.781104+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:06.114833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.007113+0000","last_change":"2024-03-18T20:23:39.765405+0000","last_active":"2024-03-18T20:23:41.007113+0000","last_peered":"2024-03-18T20:23:41.007113+0000","last_clean":"2024-03-18T20:23:41.007113+0000","last_became_active":"2024-03-18T20:23:39.764143+0000","last_became_peered":"2024-03-18T20:23:39.764143+0000","last_unstale":"2024-03-18T20:23:41.007113+0000","last_undegraded":"2024-03-18T20:23:41.007113+0000","last_fullsized":"2024-03-18T20:23:41.007113+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:35.217366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"213'2","reported_seq":13,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.006962+0000","last_change":"2024-03-18T20:23:39.328337+0000","last_active":"2024-03-18T20:23:41.006962+0000","last_peered":"2024-03-18T20:23:41.006962+0000","last_clean":"2024-03-18T20:23:41.006962+0000","last_became_active":"2024-03-18T20:23:39.328070+0000","last_became_peered":"2024-03-18T20:23:39.328070+0000","last_unstale":"2024-03-18T20:23:41.006962+0000","last_undegraded":"2024-03-18T20:23:41.006962+0000","last_fullsized":"2024-03-18T20:23:41.006962+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:11.301777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763429+0000","last_change":"2024-03-18T20:23:39.762894+0000","last_active":"2024-03-18T20:23:40.763429+0000","last_peered":"2024-03-18T20:23:40.763429+0000","last_clean":"2024-03-18T20:23:40.763429+0000","last_became_active":"2024-03-18T20:23:39.762628+0000","last_became_peered":"2024-03-18T20:23:39.762628+0000","last_unstale":"2024-03-18T20:23:40.763429+0000","last_undegraded":"2024-03-18T20:23:40.763429+0000","last_fullsized":"2024-03-18T20:23:40.763429+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:55:47.781505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"213'2","reported_seq":13,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970851+0000","last_change":"2024-03-18T20:23:39.333280+0000","last_active":"2024-03-18T20:23:40.970851+0000","last_peered":"2024-03-18T20:23:40.970851+0000","last_clean":"2024-03-18T20:23:40.970851+0000","last_became_active":"2024-03-18T20:23:39.333041+0000","last_became_peered":"2024-03-18T20:23:39.333041+0000","last_unstale":"2024-03-18T20:23:40.970851+0000","last_undegraded":"2024-03-18T20:23:40.970851+0000","last_fullsized":"2024-03-18T20:23:40.970851+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:06.137232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970560+0000","last_change":"2024-03-18T20:23:39.774714+0000","last_active":"2024-03-18T20:23:40.970560+0000","last_peered":"2024-03-18T20:23:40.970560+0000","last_clean":"2024-03-18T20:23:40.970560+0000","last_became_active":"2024-03-18T20:23:39.774461+0000","last_became_peered":"2024-03-18T20:23:39.774461+0000","last_unstale":"2024-03-18T20:23:40.970560+0000","last_undegraded":"2024-03-18T20:23:40.970560+0000","last_fullsized":"2024-03-18T20:23:40.970560+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:01:47.194750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"213'1","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780779+0000","last_change":"2024-03-18T20:23:39.338440+0000","last_active":"2024-03-18T20:23:39.780779+0000","last_peered":"2024-03-18T20:23:39.780779+0000","last_clean":"2024-03-18T20:23:39.780779+0000","last_became_active":"2024-03-18T20:23:39.338138+0000","last_became_peered":"2024-03-18T20:23:39.338138+0000","last_unstale":"2024-03-18T20:23:39.780779+0000","last_undegraded":"2024-03-18T20:23:39.780779+0000","last_fullsized":"2024-03-18T20:23:39.780779+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:01:47.102858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590655+0000","last_change":"2024-03-18T20:23:39.763657+0000","last_active":"2024-03-18T20:23:41.590655+0000","last_peered":"2024-03-18T20:23:41.590655+0000","last_clean":"2024-03-18T20:23:41.590655+0000","last_became_active":"2024-03-18T20:23:39.763463+0000","last_became_peered":"2024-03-18T20:23:39.763463+0000","last_unstale":"2024-03-18T20:23:41.590655+0000","last_undegraded":"2024-03-18T20:23:41.590655+0000","last_fullsized":"2024-03-18T20:23:41.590655+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:12:18.174454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"213'1","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.781173+0000","last_change":"2024-03-18T20:23:39.339137+0000","last_active":"2024-03-18T20:23:39.781173+0000","last_peered":"2024-03-18T20:23:39.781173+0000","last_clean":"2024-03-18T20:23:39.781173+0000","last_became_active":"2024-03-18T20:23:39.338816+0000","last_became_peered":"2024-03-18T20:23:39.338816+0000","last_unstale":"2024-03-18T20:23:39.781173+0000","last_undegraded":"2024-03-18T20:23:39.781173+0000","last_fullsized":"2024-03-18T20:23:39.781173+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:39.916823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590719+0000","last_change":"2024-03-18T20:23:39.766598+0000","last_active":"2024-03-18T20:23:41.590719+0000","last_peered":"2024-03-18T20:23:41.590719+0000","last_clean":"2024-03-18T20:23:41.590719+0000","last_became_active":"2024-03-18T20:23:39.766094+0000","last_became_peered":"2024-03-18T20:23:39.766094+0000","last_unstale":"2024-03-18T20:23:41.590719+0000","last_undegraded":"2024-03-18T20:23:41.590719+0000","last_fullsized":"2024-03-18T20:23:41.590719+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:41:40.231425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970740+0000","last_change":"2024-03-18T20:23:39.327614+0000","last_active":"2024-03-18T20:23:40.970740+0000","last_peered":"2024-03-18T20:23:40.970740+0000","last_clean":"2024-03-18T20:23:40.970740+0000","last_became_active":"2024-03-18T20:23:39.327310+0000","last_became_peered":"2024-03-18T20:23:39.327310+0000","last_unstale":"2024-03-18T20:23:40.970740+0000","last_undegraded":"2024-03-18T20:23:40.970740+0000","last_fullsized":"2024-03-18T20:23:40.970740+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:27:49.806694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:42.006794+0000","last_change":"2024-03-18T20:23:39.775571+0000","last_active":"2024-03-18T20:23:42.006794+0000","last_peered":"2024-03-18T20:23:42.006794+0000","last_clean":"2024-03-18T20:23:42.006794+0000","last_became_active":"2024-03-18T20:23:39.775299+0000","last_became_peered":"2024-03-18T20:23:39.775299+0000","last_unstale":"2024-03-18T20:23:42.006794+0000","last_undegraded":"2024-03-18T20:23:42.006794+0000","last_fullsized":"2024-03-18T20:23:42.006794+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:33:55.665965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.757193+0000","last_change":"2024-03-18T20:23:39.338549+0000","last_active":"2024-03-18T20:23:39.757193+0000","last_peered":"2024-03-18T20:23:39.757193+0000","last_clean":"2024-03-18T20:23:39.757193+0000","last_became_active":"2024-03-18T20:23:39.338300+0000","last_became_peered":"2024-03-18T20:23:39.338300+0000","last_unstale":"2024-03-18T20:23:39.757193+0000","last_undegraded":"2024-03-18T20:23:39.757193+0000","last_fullsized":"2024-03-18T20:23:39.757193+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:44.978976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.781024+0000","last_change":"2024-03-18T20:23:39.777131+0000","last_active":"2024-03-18T20:23:39.781024+0000","last_peered":"2024-03-18T20:23:39.781024+0000","last_clean":"2024-03-18T20:23:39.781024+0000","last_became_active":"2024-03-18T20:23:39.776409+0000","last_became_peered":"2024-03-18T20:23:39.776409+0000","last_unstale":"2024-03-18T20:23:39.781024+0000","last_undegraded":"2024-03-18T20:23:39.781024+0000","last_fullsized":"2024-03-18T20:23:39.781024+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:40:00.542040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590937+0000","last_change":"2024-03-18T20:23:39.332818+0000","last_active":"2024-03-18T20:23:41.590937+0000","last_peered":"2024-03-18T20:23:41.590937+0000","last_clean":"2024-03-18T20:23:41.590937+0000","last_became_active":"2024-03-18T20:23:39.332328+0000","last_became_peered":"2024-03-18T20:23:39.332328+0000","last_unstale":"2024-03-18T20:23:41.590937+0000","last_undegraded":"2024-03-18T20:23:41.590937+0000","last_fullsized":"2024-03-18T20:23:41.590937+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:41:36.473564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.971098+0000","last_change":"2024-03-18T20:23:39.759713+0000","last_active":"2024-03-18T20:23:40.971098+0000","last_peered":"2024-03-18T20:23:40.971098+0000","last_clean":"2024-03-18T20:23:40.971098+0000","last_became_active":"2024-03-18T20:23:39.759422+0000","last_became_peered":"2024-03-18T20:23:39.759422+0000","last_unstale":"2024-03-18T20:23:40.971098+0000","last_undegraded":"2024-03-18T20:23:40.971098+0000","last_fullsized":"2024-03-18T20:23:40.971098+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:55:04.135410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"213'3","reported_seq":13,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.781025+0000","last_change":"2024-03-18T20:23:39.339071+0000","last_active":"2024-03-18T20:23:39.781025+0000","last_peered":"2024-03-18T20:23:39.781025+0000","last_clean":"2024-03-18T20:23:39.781025+0000","last_became_active":"2024-03-18T20:23:39.338717+0000","last_became_peered":"2024-03-18T20:23:39.338717+0000","last_unstale":"2024-03-18T20:23:39.781025+0000","last_undegraded":"2024-03-18T20:23:39.781025+0000","last_fullsized":"2024-03-18T20:23:39.781025+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:22:55.858384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.591030+0000","last_change":"2024-03-18T20:23:39.774364+0000","last_active":"2024-03-18T20:23:41.591030+0000","last_peered":"2024-03-18T20:23:41.591030+0000","last_clean":"2024-03-18T20:23:41.591030+0000","last_became_active":"2024-03-18T20:23:39.774095+0000","last_became_peered":"2024-03-18T20:23:39.774095+0000","last_unstale":"2024-03-18T20:23:41.591030+0000","last_undegraded":"2024-03-18T20:23:41.591030+0000","last_fullsized":"2024-03-18T20:23:41.591030+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:32:20.890795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"213'6","reported_seq":17,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970935+0000","last_change":"2024-03-18T20:23:39.329381+0000","last_active":"2024-03-18T20:23:40.970935+0000","last_peered":"2024-03-18T20:23:40.970935+0000","last_clean":"2024-03-18T20:23:40.970935+0000","last_became_active":"2024-03-18T20:23:39.329110+0000","last_became_peered":"2024-03-18T20:23:39.329110+0000","last_unstale":"2024-03-18T20:23:40.970935+0000","last_undegraded":"2024-03-18T20:23:40.970935+0000","last_fullsized":"2024-03-18T20:23:40.970935+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:30.833726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763343+0000","last_change":"2024-03-18T20:23:39.772880+0000","last_active":"2024-03-18T20:23:40.763343+0000","last_peered":"2024-03-18T20:23:40.763343+0000","last_clean":"2024-03-18T20:23:40.763343+0000","last_became_active":"2024-03-18T20:23:39.764158+0000","last_became_peered":"2024-03-18T20:23:39.764158+0000","last_unstale":"2024-03-18T20:23:40.763343+0000","last_undegraded":"2024-03-18T20:23:40.763343+0000","last_fullsized":"2024-03-18T20:23:40.763343+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:07.143106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"213'2","reported_seq":13,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970837+0000","last_change":"2024-03-18T20:23:39.332410+0000","last_active":"2024-03-18T20:23:40.970837+0000","last_peered":"2024-03-18T20:23:40.970837+0000","last_clean":"2024-03-18T20:23:40.970837+0000","last_became_active":"2024-03-18T20:23:39.332168+0000","last_became_peered":"2024-03-18T20:23:39.332168+0000","last_unstale":"2024-03-18T20:23:40.970837+0000","last_undegraded":"2024-03-18T20:23:40.970837+0000","last_fullsized":"2024-03-18T20:23:40.970837+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:28:59.985890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780722+0000","last_change":"2024-03-18T20:23:39.772098+0000","last_active":"2024-03-18T20:23:39.780722+0000","last_peered":"2024-03-18T20:23:39.780722+0000","last_clean":"2024-03-18T20:23:39.780722+0000","last_became_active":"2024-03-18T20:23:39.771829+0000","last_became_peered":"2024-03-18T20:23:39.771829+0000","last_unstale":"2024-03-18T20:23:39.780722+0000","last_undegraded":"2024-03-18T20:23:39.780722+0000","last_fullsized":"2024-03-18T20:23:39.780722+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:45.947401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590955+0000","last_change":"2024-03-18T20:23:39.335648+0000","last_active":"2024-03-18T20:23:41.590955+0000","last_peered":"2024-03-18T20:23:41.590955+0000","last_clean":"2024-03-18T20:23:41.590955+0000","last_became_active":"2024-03-18T20:23:39.335381+0000","last_became_peered":"2024-03-18T20:23:39.335381+0000","last_unstale":"2024-03-18T20:23:41.590955+0000","last_undegraded":"2024-03-18T20:23:41.590955+0000","last_fullsized":"2024-03-18T20:23:41.590955+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:42.078249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970795+0000","last_change":"2024-03-18T20:23:39.772922+0000","last_active":"2024-03-18T20:23:40.970795+0000","last_peered":"2024-03-18T20:23:40.970795+0000","last_clean":"2024-03-18T20:23:40.970795+0000","last_became_active":"2024-03-18T20:23:39.772605+0000","last_became_peered":"2024-03-18T20:23:39.772605+0000","last_unstale":"2024-03-18T20:23:40.970795+0000","last_undegraded":"2024-03-18T20:23:40.970795+0000","last_fullsized":"2024-03-18T20:23:40.970795+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:10.588668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.760976+0000","last_change":"2024-03-18T20:23:39.335354+0000","last_active":"2024-03-18T20:23:39.760976+0000","last_peered":"2024-03-18T20:23:39.760976+0000","last_clean":"2024-03-18T20:23:39.760976+0000","last_became_active":"2024-03-18T20:23:39.335084+0000","last_became_peered":"2024-03-18T20:23:39.335084+0000","last_unstale":"2024-03-18T20:23:39.760976+0000","last_undegraded":"2024-03-18T20:23:39.760976+0000","last_fullsized":"2024-03-18T20:23:39.760976+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:58:57.300557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970793+0000","last_change":"2024-03-18T20:23:39.766941+0000","last_active":"2024-03-18T20:23:40.970793+0000","last_peered":"2024-03-18T20:23:40.970793+0000","last_clean":"2024-03-18T20:23:40.970793+0000","last_became_active":"2024-03-18T20:23:39.766689+0000","last_became_peered":"2024-03-18T20:23:39.766689+0000","last_unstale":"2024-03-18T20:23:40.970793+0000","last_undegraded":"2024-03-18T20:23:40.970793+0000","last_fullsized":"2024-03-18T20:23:40.970793+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:30.491107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"213'8","reported_seq":19,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.006742+0000","last_change":"2024-03-18T20:23:39.331204+0000","last_active":"2024-03-18T20:23:41.006742+0000","last_peered":"2024-03-18T20:23:41.006742+0000","last_clean":"2024-03-18T20:23:41.006742+0000","last_became_active":"2024-03-18T20:23:39.330947+0000","last_became_peered":"2024-03-18T20:23:39.330947+0000","last_unstale":"2024-03-18T20:23:41.006742+0000","last_undegraded":"2024-03-18T20:23:41.006742+0000","last_fullsized":"2024-03-18T20:23:41.006742+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:39.419825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763671+0000","last_change":"2024-03-18T20:23:39.775343+0000","last_active":"2024-03-18T20:23:40.763671+0000","last_peered":"2024-03-18T20:23:40.763671+0000","last_clean":"2024-03-18T20:23:40.763671+0000","last_became_active":"2024-03-18T20:23:39.774987+0000","last_became_peered":"2024-03-18T20:23:39.774987+0000","last_unstale":"2024-03-18T20:23:40.763671+0000","last_undegraded":"2024-03-18T20:23:40.763671+0000","last_fullsized":"2024-03-18T20:23:40.763671+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:40.425699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"213'1","reported_seq":12,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.007142+0000","last_change":"2024-03-18T20:23:39.335549+0000","last_active":"2024-03-18T20:23:41.007142+0000","last_peered":"2024-03-18T20:23:41.007142+0000","last_clean":"2024-03-18T20:23:41.007142+0000","last_became_active":"2024-03-18T20:23:39.335279+0000","last_became_peered":"2024-03-18T20:23:39.335279+0000","last_unstale":"2024-03-18T20:23:41.007142+0000","last_undegraded":"2024-03-18T20:23:41.007142+0000","last_fullsized":"2024-03-18T20:23:41.007142+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:02:34.252952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780852+0000","last_change":"2024-03-18T20:23:39.774804+0000","last_active":"2024-03-18T20:23:39.780852+0000","last_peered":"2024-03-18T20:23:39.780852+0000","last_clean":"2024-03-18T20:23:39.780852+0000","last_became_active":"2024-03-18T20:23:39.774539+0000","last_became_peered":"2024-03-18T20:23:39.774539+0000","last_unstale":"2024-03-18T20:23:39.780852+0000","last_undegraded":"2024-03-18T20:23:39.780852+0000","last_fullsized":"2024-03-18T20:23:39.780852+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:43.947623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"213'2","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.773318+0000","last_change":"2024-03-18T20:23:39.335330+0000","last_active":"2024-03-18T20:23:39.773318+0000","last_peered":"2024-03-18T20:23:39.773318+0000","last_clean":"2024-03-18T20:23:39.773318+0000","last_became_active":"2024-03-18T20:23:39.335061+0000","last_became_peered":"2024-03-18T20:23:39.335061+0000","last_unstale":"2024-03-18T20:23:39.773318+0000","last_undegraded":"2024-03-18T20:23:39.773318+0000","last_fullsized":"2024-03-18T20:23:39.773318+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:42:50.684286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.007142+0000","last_change":"2024-03-18T20:23:39.770963+0000","last_active":"2024-03-18T20:23:41.007142+0000","last_peered":"2024-03-18T20:23:41.007142+0000","last_clean":"2024-03-18T20:23:41.007142+0000","last_became_active":"2024-03-18T20:23:39.768768+0000","last_became_peered":"2024-03-18T20:23:39.768768+0000","last_unstale":"2024-03-18T20:23:41.007142+0000","last_undegraded":"2024-03-18T20:23:41.007142+0000","last_fullsized":"2024-03-18T20:23:41.007142+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:30.491107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"213'1","reported_seq":12,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763319+0000","last_change":"2024-03-18T20:23:39.332256+0000","last_active":"2024-03-18T20:23:40.763319+0000","last_peered":"2024-03-18T20:23:40.763319+0000","last_clean":"2024-03-18T20:23:40.763319+0000","last_became_active":"2024-03-18T20:23:39.331959+0000","last_became_peered":"2024-03-18T20:23:39.331959+0000","last_unstale":"2024-03-18T20:23:40.763319+0000","last_undegraded":"2024-03-18T20:23:40.763319+0000","last_fullsized":"2024-03-18T20:23:40.763319+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:09:15.620959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.591020+0000","last_change":"2024-03-18T20:23:39.781943+0000","last_active":"2024-03-18T20:23:41.591020+0000","last_peered":"2024-03-18T20:23:41.591020+0000","last_clean":"2024-03-18T20:23:41.591020+0000","last_became_active":"2024-03-18T20:23:39.781660+0000","last_became_peered":"2024-03-18T20:23:39.781660+0000","last_unstale":"2024-03-18T20:23:41.591020+0000","last_undegraded":"2024-03-18T20:23:41.591020+0000","last_fullsized":"2024-03-18T20:23:41.591020+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:43:05.508887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.006967+0000","last_change":"2024-03-18T20:23:39.327794+0000","last_active":"2024-03-18T20:23:41.006967+0000","last_peered":"2024-03-18T20:23:41.006967+0000","last_clean":"2024-03-18T20:23:41.006967+0000","last_became_active":"2024-03-18T20:23:39.327508+0000","last_became_peered":"2024-03-18T20:23:39.327508+0000","last_unstale":"2024-03-18T20:23:41.006967+0000","last_undegraded":"2024-03-18T20:23:41.006967+0000","last_fullsized":"2024-03-18T20:23:41.006967+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:56:24.251796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590494+0000","last_change":"2024-03-18T20:23:39.774457+0000","last_active":"2024-03-18T20:23:41.590494+0000","last_peered":"2024-03-18T20:23:41.590494+0000","last_clean":"2024-03-18T20:23:41.590494+0000","last_became_active":"2024-03-18T20:23:39.774203+0000","last_became_peered":"2024-03-18T20:23:39.774203+0000","last_unstale":"2024-03-18T20:23:41.590494+0000","last_undegraded":"2024-03-18T20:23:41.590494+0000","last_fullsized":"2024-03-18T20:23:41.590494+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:20:52.271630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"213'2","reported_seq":13,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970910+0000","last_change":"2024-03-18T20:23:39.331954+0000","last_active":"2024-03-18T20:23:40.970910+0000","last_peered":"2024-03-18T20:23:40.970910+0000","last_clean":"2024-03-18T20:23:40.970910+0000","last_became_active":"2024-03-18T20:23:39.331741+0000","last_became_peered":"2024-03-18T20:23:39.331741+0000","last_unstale":"2024-03-18T20:23:40.970910+0000","last_undegraded":"2024-03-18T20:23:40.970910+0000","last_fullsized":"2024-03-18T20:23:40.970910+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:56:24.251796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.770845+0000","last_change":"2024-03-18T20:23:39.766629+0000","last_active":"2024-03-18T20:23:39.770845+0000","last_peered":"2024-03-18T20:23:39.770845+0000","last_clean":"2024-03-18T20:23:39.770845+0000","last_became_active":"2024-03-18T20:23:39.766415+0000","last_became_peered":"2024-03-18T20:23:39.766415+0000","last_unstale":"2024-03-18T20:23:39.770845+0000","last_undegraded":"2024-03-18T20:23:39.770845+0000","last_fullsized":"2024-03-18T20:23:39.770845+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:41:40.231425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.765952+0000","last_change":"2024-03-18T20:23:39.334672+0000","last_active":"2024-03-18T20:23:39.765952+0000","last_peered":"2024-03-18T20:23:39.765952+0000","last_clean":"2024-03-18T20:23:39.765952+0000","last_became_active":"2024-03-18T20:23:39.334148+0000","last_became_peered":"2024-03-18T20:23:39.334148+0000","last_unstale":"2024-03-18T20:23:39.765952+0000","last_undegraded":"2024-03-18T20:23:39.765952+0000","last_fullsized":"2024-03-18T20:23:39.765952+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:21:46.632822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.770841+0000","last_change":"2024-03-18T20:23:39.766505+0000","last_active":"2024-03-18T20:23:39.770841+0000","last_peered":"2024-03-18T20:23:39.770841+0000","last_clean":"2024-03-18T20:23:39.770841+0000","last_became_active":"2024-03-18T20:23:39.766223+0000","last_became_peered":"2024-03-18T20:23:39.766223+0000","last_unstale":"2024-03-18T20:23:39.770841+0000","last_undegraded":"2024-03-18T20:23:39.770841+0000","last_fullsized":"2024-03-18T20:23:39.770841+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:35.116397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763352+0000","last_change":"2024-03-18T20:23:39.332289+0000","last_active":"2024-03-18T20:23:40.763352+0000","last_peered":"2024-03-18T20:23:40.763352+0000","last_clean":"2024-03-18T20:23:40.763352+0000","last_became_active":"2024-03-18T20:23:39.332014+0000","last_became_peered":"2024-03-18T20:23:39.332014+0000","last_unstale":"2024-03-18T20:23:40.763352+0000","last_undegraded":"2024-03-18T20:23:40.763352+0000","last_fullsized":"2024-03-18T20:23:40.763352+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:45:21.252973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780958+0000","last_change":"2024-03-18T20:23:39.776164+0000","last_active":"2024-03-18T20:23:39.780958+0000","last_peered":"2024-03-18T20:23:39.780958+0000","last_clean":"2024-03-18T20:23:39.780958+0000","last_became_active":"2024-03-18T20:23:39.775917+0000","last_became_peered":"2024-03-18T20:23:39.775917+0000","last_unstale":"2024-03-18T20:23:39.780958+0000","last_undegraded":"2024-03-18T20:23:39.780958+0000","last_fullsized":"2024-03-18T20:23:39.780958+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:59:24.493213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.762843+0000","last_change":"2024-03-18T20:23:39.324353+0000","last_active":"2024-03-18T20:23:40.762843+0000","last_peered":"2024-03-18T20:23:40.762843+0000","last_clean":"2024-03-18T20:23:40.762843+0000","last_became_active":"2024-03-18T20:23:39.324037+0000","last_became_peered":"2024-03-18T20:23:39.324037+0000","last_unstale":"2024-03-18T20:23:40.762843+0000","last_undegraded":"2024-03-18T20:23:40.762843+0000","last_fullsized":"2024-03-18T20:23:40.762843+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:32:02.472421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763748+0000","last_change":"2024-03-18T20:23:39.762706+0000","last_active":"2024-03-18T20:23:40.763748+0000","last_peered":"2024-03-18T20:23:40.763748+0000","last_clean":"2024-03-18T20:23:40.763748+0000","last_became_active":"2024-03-18T20:23:39.762461+0000","last_became_peered":"2024-03-18T20:23:39.762461+0000","last_unstale":"2024-03-18T20:23:40.763748+0000","last_undegraded":"2024-03-18T20:23:40.763748+0000","last_fullsized":"2024-03-18T20:23:40.763748+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:29:00.991763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"213'2","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.776518+0000","last_change":"2024-03-18T20:23:39.334632+0000","last_active":"2024-03-18T20:23:39.776518+0000","last_peered":"2024-03-18T20:23:39.776518+0000","last_clean":"2024-03-18T20:23:39.776518+0000","last_became_active":"2024-03-18T20:23:39.334345+0000","last_became_peered":"2024-03-18T20:23:39.334345+0000","last_unstale":"2024-03-18T20:23:39.776518+0000","last_undegraded":"2024-03-18T20:23:39.776518+0000","last_fullsized":"2024-03-18T20:23:39.776518+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:05:56.973792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590754+0000","last_change":"2024-03-18T20:23:39.762505+0000","last_active":"2024-03-18T20:23:41.590754+0000","last_peered":"2024-03-18T20:23:41.590754+0000","last_clean":"2024-03-18T20:23:41.590754+0000","last_became_active":"2024-03-18T20:23:39.762227+0000","last_became_peered":"2024-03-18T20:23:39.762227+0000","last_unstale":"2024-03-18T20:23:41.590754+0000","last_undegraded":"2024-03-18T20:23:41.590754+0000","last_fullsized":"2024-03-18T20:23:41.590754+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:57:49.050649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763604+0000","last_change":"2024-03-18T20:23:39.326895+0000","last_active":"2024-03-18T20:23:40.763604+0000","last_peered":"2024-03-18T20:23:40.763604+0000","last_clean":"2024-03-18T20:23:40.763604+0000","last_became_active":"2024-03-18T20:23:39.326605+0000","last_became_peered":"2024-03-18T20:23:39.326605+0000","last_unstale":"2024-03-18T20:23:40.763604+0000","last_undegraded":"2024-03-18T20:23:40.763604+0000","last_fullsized":"2024-03-18T20:23:40.763604+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:36:04.660134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:42.006839+0000","last_change":"2024-03-18T20:23:39.773684+0000","last_active":"2024-03-18T20:23:42.006839+0000","last_peered":"2024-03-18T20:23:42.006839+0000","last_clean":"2024-03-18T20:23:42.006839+0000","last_became_active":"2024-03-18T20:23:39.773341+0000","last_became_peered":"2024-03-18T20:23:39.773341+0000","last_unstale":"2024-03-18T20:23:42.006839+0000","last_undegraded":"2024-03-18T20:23:42.006839+0000","last_fullsized":"2024-03-18T20:23:42.006839+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:30.491107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.756377+0000","last_change":"2024-03-18T20:23:39.338114+0000","last_active":"2024-03-18T20:23:39.756377+0000","last_peered":"2024-03-18T20:23:39.756377+0000","last_clean":"2024-03-18T20:23:39.756377+0000","last_became_active":"2024-03-18T20:23:39.337837+0000","last_became_peered":"2024-03-18T20:23:39.337837+0000","last_unstale":"2024-03-18T20:23:39.756377+0000","last_undegraded":"2024-03-18T20:23:39.756377+0000","last_fullsized":"2024-03-18T20:23:39.756377+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:07:20.730496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763713+0000","last_change":"2024-03-18T20:23:39.763620+0000","last_active":"2024-03-18T20:23:40.763713+0000","last_peered":"2024-03-18T20:23:40.763713+0000","last_clean":"2024-03-18T20:23:40.763713+0000","last_became_active":"2024-03-18T20:23:39.763361+0000","last_became_peered":"2024-03-18T20:23:39.763361+0000","last_unstale":"2024-03-18T20:23:40.763713+0000","last_undegraded":"2024-03-18T20:23:40.763713+0000","last_fullsized":"2024-03-18T20:23:40.763713+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:56:25.257670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.756531+0000","last_change":"2024-03-18T20:23:39.329506+0000","last_active":"2024-03-18T20:23:39.756531+0000","last_peered":"2024-03-18T20:23:39.756531+0000","last_clean":"2024-03-18T20:23:39.756531+0000","last_became_active":"2024-03-18T20:23:39.329028+0000","last_became_peered":"2024-03-18T20:23:39.329028+0000","last_unstale":"2024-03-18T20:23:39.756531+0000","last_undegraded":"2024-03-18T20:23:39.756531+0000","last_fullsized":"2024-03-18T20:23:39.756531+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:49:53.935705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.762793+0000","last_change":"2024-03-18T20:23:39.765543+0000","last_active":"2024-03-18T20:23:40.762793+0000","last_peered":"2024-03-18T20:23:40.762793+0000","last_clean":"2024-03-18T20:23:40.762793+0000","last_became_active":"2024-03-18T20:23:39.764942+0000","last_became_peered":"2024-03-18T20:23:39.764942+0000","last_unstale":"2024-03-18T20:23:40.762793+0000","last_undegraded":"2024-03-18T20:23:40.762793+0000","last_fullsized":"2024-03-18T20:23:40.762793+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:53:15.836495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:42.006583+0000","last_change":"2024-03-18T20:23:39.333831+0000","last_active":"2024-03-18T20:23:42.006583+0000","last_peered":"2024-03-18T20:23:42.006583+0000","last_clean":"2024-03-18T20:23:42.006583+0000","last_became_active":"2024-03-18T20:23:39.333541+0000","last_became_peered":"2024-03-18T20:23:39.333541+0000","last_unstale":"2024-03-18T20:23:42.006583+0000","last_undegraded":"2024-03-18T20:23:42.006583+0000","last_fullsized":"2024-03-18T20:23:42.006583+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:55.688191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970755+0000","last_change":"2024-03-18T20:23:39.762015+0000","last_active":"2024-03-18T20:23:40.970755+0000","last_peered":"2024-03-18T20:23:40.970755+0000","last_clean":"2024-03-18T20:23:40.970755+0000","last_became_active":"2024-03-18T20:23:39.761733+0000","last_became_peered":"2024-03-18T20:23:39.761733+0000","last_unstale":"2024-03-18T20:23:40.970755+0000","last_undegraded":"2024-03-18T20:23:40.970755+0000","last_fullsized":"2024-03-18T20:23:40.970755+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:35.217366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:42.006715+0000","last_change":"2024-03-18T20:23:39.330955+0000","last_active":"2024-03-18T20:23:42.006715+0000","last_peered":"2024-03-18T20:23:42.006715+0000","last_clean":"2024-03-18T20:23:42.006715+0000","last_became_active":"2024-03-18T20:23:39.330635+0000","last_became_peered":"2024-03-18T20:23:39.330635+0000","last_unstale":"2024-03-18T20:23:42.006715+0000","last_undegraded":"2024-03-18T20:23:42.006715+0000","last_fullsized":"2024-03-18T20:23:42.006715+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:39.419825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590979+0000","last_change":"2024-03-18T20:23:39.769949+0000","last_active":"2024-03-18T20:23:41.590979+0000","last_peered":"2024-03-18T20:23:41.590979+0000","last_clean":"2024-03-18T20:23:41.590979+0000","last_became_active":"2024-03-18T20:23:39.769687+0000","last_became_peered":"2024-03-18T20:23:39.769687+0000","last_unstale":"2024-03-18T20:23:41.590979+0000","last_undegraded":"2024-03-18T20:23:41.590979+0000","last_fullsized":"2024-03-18T20:23:41.590979+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:04:55.872080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763517+0000","last_change":"2024-03-18T20:23:39.323965+0000","last_active":"2024-03-18T20:23:40.763517+0000","last_peered":"2024-03-18T20:23:40.763517+0000","last_clean":"2024-03-18T20:23:40.763517+0000","last_became_active":"2024-03-18T20:23:39.323731+0000","last_became_peered":"2024-03-18T20:23:39.323731+0000","last_unstale":"2024-03-18T20:23:40.763517+0000","last_undegraded":"2024-03-18T20:23:40.763517+0000","last_fullsized":"2024-03-18T20:23:40.763517+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:57:50.281821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:42.006517+0000","last_change":"2024-03-18T20:23:39.771006+0000","last_active":"2024-03-18T20:23:42.006517+0000","last_peered":"2024-03-18T20:23:42.006517+0000","last_clean":"2024-03-18T20:23:42.006517+0000","last_became_active":"2024-03-18T20:23:39.770741+0000","last_became_peered":"2024-03-18T20:23:39.770741+0000","last_unstale":"2024-03-18T20:23:42.006517+0000","last_undegraded":"2024-03-18T20:23:42.006517+0000","last_fullsized":"2024-03-18T20:23:42.006517+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:00.540282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.762903+0000","last_change":"2024-03-18T20:23:39.327798+0000","last_active":"2024-03-18T20:23:40.762903+0000","last_peered":"2024-03-18T20:23:40.762903+0000","last_clean":"2024-03-18T20:23:40.762903+0000","last_became_active":"2024-03-18T20:23:39.327543+0000","last_became_peered":"2024-03-18T20:23:39.327543+0000","last_unstale":"2024-03-18T20:23:40.762903+0000","last_undegraded":"2024-03-18T20:23:40.762903+0000","last_fullsized":"2024-03-18T20:23:40.762903+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:41.616808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763041+0000","last_change":"2024-03-18T20:23:39.776313+0000","last_active":"2024-03-18T20:23:40.763041+0000","last_peered":"2024-03-18T20:23:40.763041+0000","last_clean":"2024-03-18T20:23:40.763041+0000","last_became_active":"2024-03-18T20:23:39.776057+0000","last_became_peered":"2024-03-18T20:23:39.776057+0000","last_unstale":"2024-03-18T20:23:40.763041+0000","last_undegraded":"2024-03-18T20:23:40.763041+0000","last_fullsized":"2024-03-18T20:23:40.763041+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:59:57.901328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.759917+0000","last_change":"2024-03-18T20:23:39.330747+0000","last_active":"2024-03-18T20:23:39.759917+0000","last_peered":"2024-03-18T20:23:39.759917+0000","last_clean":"2024-03-18T20:23:39.759917+0000","last_became_active":"2024-03-18T20:23:39.330391+0000","last_became_peered":"2024-03-18T20:23:39.330391+0000","last_unstale":"2024-03-18T20:23:39.759917+0000","last_undegraded":"2024-03-18T20:23:39.759917+0000","last_fullsized":"2024-03-18T20:23:39.759917+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:52.293500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"203'326","reported_seq":1107,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.762106+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:23:39.762106+0000","last_peered":"2024-03-18T20:23:39.762106+0000","last_clean":"2024-03-18T20:23:39.762106+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:23:39.762106+0000","last_undegraded":"2024-03-18T20:23:39.762106+0000","last_fullsized":"2024-03-18T20:23:39.762106+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575962,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52364,"kb_used_data":3188,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319476,"statfs":{"total":96636764160,"available":96583143424,"internally_reserved":0,"allocated":3264512,"data_stored":1954669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.622,"15min":0.60099999999999998},"min":{"1min":0.40400000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.75800000000000001,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61399999999999999,"15min":0.61099999999999999},"min":{"1min":0.42699999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.84499999999999997,"5min":1.093,"15min":1.093},"last":0.75}]},{"osd":1,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.63600000000000001,"15min":0.61299999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.35999999999999999},"max":{"1min":0.83099999999999996,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.629,"15min":0.61799999999999999},"min":{"1min":0.40300000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78600000000000003,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.66600000000000004}]},{"osd":2,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.63,"15min":0.61299999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.77900000000000003,"5min":1.216,"15min":1.216},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63200000000000001,"15min":0.623},"min":{"1min":0.48499999999999999,"5min":0.41699999999999998,"15min":0.39000000000000001},"max":{"1min":0.73199999999999998,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.52500000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.63,"15min":0.61499999999999999},"min":{"1min":0.47199999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.69799999999999995,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.495},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66700000000000004,"15min":0.64300000000000002},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.38600000000000001},"max":{"1min":0.79200000000000004,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.59899999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47799999999999998,"15min":0.47699999999999998},"min":{"1min":0.32600000000000001,"5min":0.316,"15min":0.28899999999999998},"max":{"1min":0.67400000000000004,"5min":0.78700000000000003,"15min":0.86099999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.497,"15min":0.502},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.63,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.76600000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52300000000000002,"15min":0.51500000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.316},"max":{"1min":0.65500000000000003,"5min":0.86599999999999999,"15min":0.89400000000000002},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.54600000000000004,"15min":0.55100000000000005},"min":{"1min":0.377,"5min":0.36899999999999999,"15min":0.34699999999999998},"max":{"1min":0.625,"5min":0.76500000000000001,"15min":1.042},"last":0.81599999999999995}]},{"osd":6,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.36599999999999999,"5min":0.36199999999999999,"15min":0.34999999999999998},"max":{"1min":0.752,"5min":0.85599999999999998,"15min":1.129},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.34000000000000002},"max":{"1min":0.68500000000000005,"5min":0.85299999999999998,"15min":1.0609999999999999},"last":0.71599999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575962,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41932,"kb_used_data":2624,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329908,"statfs":{"total":96636764160,"available":96593825792,"internally_reserved":0,"allocated":2686976,"data_stored":1371091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.437,"5min":0.38200000000000001,"15min":0.375},"max":{"1min":0.73399999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61099999999999999,"15min":0.61799999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.318},"max":{"1min":0.72199999999999998,"5min":0.81499999999999995,"15min":0.83299999999999996},"last":0.54300000000000004}]},{"osd":1,"last update":"Mon Mar 18 20:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.60699999999999998,"15min":0.60899999999999999},"min":{"1min":0.45300000000000001,"5min":0.36499999999999999,"15min":0.35799999999999998},"max":{"1min":0.72099999999999997,"5min":1.244,"15min":1.244},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61899999999999999,"15min":0.61199999999999999},"min":{"1min":0.45000000000000001,"5min":0.41799999999999998,"15min":0.33200000000000002},"max":{"1min":0.755,"5min":1.286,"15min":1.286},"last":0.55500000000000005}]},{"osd":2,"last update":"Mon Mar 18 20:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.64500000000000002,"15min":0.63800000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77000000000000002,"5min":1.276,"15min":1.276},"last":0.501},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.61699999999999999,"15min":0.61599999999999999},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.374},"max":{"1min":0.67100000000000004,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.58899999999999997}]},{"osd":3,"last update":"Mon Mar 18 20:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.63,"15min":0.63},"min":{"1min":0.41399999999999998,"5min":0.39000000000000001,"15min":0.32700000000000001},"max":{"1min":0.66800000000000004,"5min":0.85399999999999998,"15min":1.02},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.63500000000000001,"15min":0.623},"min":{"1min":0.46800000000000003,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.73699999999999999,"5min":1.337,"15min":1.337},"last":0.57399999999999995}]},{"osd":4,"last update":"Mon Mar 18 20:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.48699999999999999,"15min":0.48499999999999999},"min":{"1min":0.34599999999999997,"5min":0.32600000000000001,"15min":0.312},"max":{"1min":0.80100000000000005,"5min":0.86499999999999999,"15min":0.88300000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.52200000000000002,"15min":0.51200000000000001},"min":{"1min":0.40400000000000003,"5min":0.34599999999999997,"15min":0.32600000000000001},"max":{"1min":0.69299999999999995,"5min":0.94899999999999995,"15min":0.95499999999999996},"last":0.35299999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.505,"15min":0.50700000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.31},"max":{"1min":0.76000000000000001,"5min":0.82699999999999996,"15min":0.84799999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.39300000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.94199999999999995},"last":0.40699999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56299999999999994,"15min":0.55500000000000005},"min":{"1min":0.38,"5min":0.35799999999999998,"15min":0.34499999999999997},"max":{"1min":0.73599999999999999,"5min":0.95599999999999996,"15min":1.0429999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.34599999999999997},"max":{"1min":0.79000000000000004,"5min":1.028,"15min":1.028},"last":0.438}]}]},{"osd":1,"up_from":13,"seq":55834575962,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47680,"kb_used_data":2608,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324160,"statfs":{"total":96636764160,"available":96587939840,"internally_reserved":0,"allocated":2670592,"data_stored":1364301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.40699999999999997,"15min":0.39600000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.23999999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.501},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.42099999999999999,"15min":0.42999999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.59899999999999998,"5min":0.69699999999999995,"15min":0.72299999999999998},"last":0.54000000000000004}]},{"osd":2,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.438,"15min":0.435},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.58199999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.47999999999999998,"15min":0.47599999999999998},"min":{"1min":0.32100000000000001,"5min":0.313,"15min":0.29899999999999999},"max":{"1min":0.65300000000000002,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.70099999999999996}]},{"osd":3,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.495,"15min":0.49299999999999999},"min":{"1min":0.33000000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.62,"5min":0.75900000000000001,"15min":0.76100000000000001},"last":0.625},{"interface":"front","average":{"1min":0.437,"5min":0.443,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.59199999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.68200000000000005}]},{"osd":4,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.69799999999999995,"15min":0.67500000000000004},"min":{"1min":0.498,"5min":0.496,"15min":0.44700000000000001},"max":{"1min":1.0680000000000001,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67500000000000004,"15min":0.68200000000000005},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":1.746},"last":0.84499999999999997}]},{"osd":5,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.68500000000000005,"15min":0.67700000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.42099999999999999},"max":{"1min":0.90300000000000002,"5min":0.96099999999999997,"15min":1.6910000000000001},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73599999999999999,"15min":0.71599999999999997},"min":{"1min":0.53400000000000003,"5min":0.47399999999999998,"15min":0.46899999999999997},"max":{"1min":1.109,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.89400000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.69899999999999995,"15min":0.68899999999999995},"min":{"1min":0.54700000000000004,"5min":0.437,"15min":0.437},"max":{"1min":0.872,"5min":0.98899999999999999,"15min":1.637},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.71999999999999997,"15min":0.71399999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.40899999999999997},"max":{"1min":1.014,"5min":1.014,"15min":1.722},"last":0.82899999999999996}]},{"osd":7,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.72099999999999997,"15min":0.71499999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.44600000000000001},"max":{"1min":0.99199999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71899999999999997,"15min":0.71099999999999997},"min":{"1min":0.495,"5min":0.495,"15min":0.42799999999999999},"max":{"1min":1.0329999999999999,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.88100000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575963,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46608,"kb_used_data":3204,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94325232,"statfs":{"total":96636764160,"available":96589037568,"internally_reserved":0,"allocated":3280896,"data_stored":1955397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44431814},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.40000000000000002,"15min":0.39600000000000002},"min":{"1min":0.25600000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.373},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.27200000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.59299999999999997,"5min":0.64700000000000002,"15min":1.1850000000000001},"last":0.34000000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.435,"15min":0.439},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.317},{"interface":"front","average":{"1min":0.436,"5min":0.44900000000000001,"15min":0.45700000000000002},"min":{"1min":0.311,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.54400000000000004,"5min":0.64300000000000002,"15min":0.70999999999999996},"last":0.29899999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.434,"15min":0.435},"min":{"1min":0.317,"5min":0.28100000000000003,"15min":0.26100000000000001},"max":{"1min":0.56100000000000005,"5min":0.61299999999999999,"15min":0.628},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.496,"15min":0.5},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.63800000000000001,"5min":0.73999999999999999,"15min":0.91500000000000004},"last":0.40699999999999997}]},{"osd":4,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65500000000000003,"15min":0.66100000000000003},"min":{"1min":0.47499999999999998,"5min":0.45400000000000001,"15min":0.42999999999999999},"max":{"1min":0.79800000000000004,"5min":0.96599999999999997,"15min":1.0029999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.67400000000000004,"15min":0.66600000000000004},"min":{"1min":0.48299999999999998,"5min":0.47099999999999997,"15min":0.39700000000000002},"max":{"1min":0.88300000000000001,"5min":0.99199999999999999,"15min":1.0469999999999999},"last":0.55100000000000005}]},{"osd":5,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.68400000000000005,"15min":0.67400000000000004},"min":{"1min":0.51300000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.89300000000000002,"5min":0.97099999999999997,"15min":1.0129999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.68700000000000006},"min":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.42399999999999999},"max":{"1min":0.89400000000000002,"5min":0.92900000000000005,"15min":1.518},"last":0.442}]},{"osd":6,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67200000000000004,"15min":0.67000000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45000000000000001},"max":{"1min":0.90700000000000003,"5min":0.97399999999999998,"15min":1.1279999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.71299999999999997,"15min":0.71199999999999997},"min":{"1min":0.48999999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.85399999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.60399999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.71199999999999997,"15min":0.70899999999999996},"min":{"1min":0.51900000000000002,"5min":0.46700000000000003,"15min":0.44900000000000001},"max":{"1min":0.84599999999999997,"5min":0.99099999999999999,"15min":1.464},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71799999999999997,"15min":0.71099999999999997},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.47599999999999998},"max":{"1min":0.85299999999999998,"5min":1.042,"15min":1.2350000000000001},"last":0.51900000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575963,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47692,"kb_used_data":2624,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324148,"statfs":{"total":96636764160,"available":96587927552,"internally_reserved":0,"allocated":2686976,"data_stored":1371603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.38300000000000001,"15min":0.38100000000000001},"min":{"1min":0.26400000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.44900000000000001,"5min":0.55300000000000005,"15min":0.70799999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.495,"15min":0.48999999999999999},"min":{"1min":0.32700000000000001,"5min":0.313,"15min":0.24199999999999999},"max":{"1min":0.63800000000000001,"5min":0.72899999999999998,"15min":0.83399999999999996},"last":0.47599999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.48799999999999999,"15min":0.48199999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.30299999999999999},"max":{"1min":0.58599999999999997,"5min":0.749,"15min":0.77100000000000002},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.42399999999999999,"15min":0.41999999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27600000000000002},"max":{"1min":0.55000000000000004,"5min":0.59299999999999997,"15min":0.71099999999999997},"last":0.52000000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.443,"15min":0.438},"min":{"1min":0.34699999999999998,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":0.57299999999999995,"5min":0.64700000000000002,"15min":0.66300000000000003},"last":0.501},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.313},"max":{"1min":0.56699999999999995,"5min":0.752,"15min":0.752},"last":0.57399999999999995}]},{"osd":4,"last update":"Mon Mar 18 20:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.66800000000000004,"15min":0.66100000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":0.80900000000000005,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.753},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.68400000000000005,"15min":0.66900000000000004},"min":{"1min":0.46700000000000003,"5min":0.443,"15min":0.443},"max":{"1min":0.75,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.79800000000000004}]},{"osd":5,"last update":"Mon Mar 18 20:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.68000000000000005,"15min":0.68500000000000005},"min":{"1min":0.48399999999999999,"5min":0.48199999999999998,"15min":0.45300000000000001},"max":{"1min":0.81000000000000005,"5min":1.028,"15min":1.028},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.624,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.44600000000000001},"max":{"1min":0.83399999999999996,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.88200000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.69499999999999995,"15min":0.69399999999999995},"min":{"1min":0.55000000000000004,"5min":0.48899999999999999,"15min":0.46700000000000003},"max":{"1min":0.80100000000000005,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.73999999999999999,"15min":0.73399999999999999},"min":{"1min":0.54900000000000004,"5min":0.51900000000000002,"15min":0.51000000000000001},"max":{"1min":0.83199999999999996,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.90200000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.69999999999999996,"15min":0.70399999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.42199999999999999},"max":{"1min":0.80400000000000005,"5min":0.99199999999999999,"15min":1.0469999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.71399999999999997,"15min":0.71099999999999997},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.49099999999999999},"max":{"1min":0.81499999999999995,"5min":1.371,"15min":1.371},"last":0.92400000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51788,"kb_used_data":2616,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320052,"statfs":{"total":96636764160,"available":96583733248,"internally_reserved":0,"allocated":2678784,"data_stored":1364461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:23:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.27800000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54900000000000004,"5min":0.65500000000000003,"15min":0.98099999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42199999999999999,"15min":0.42099999999999999},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.24199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.78800000000000003},"last":0.40100000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:23:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40799999999999997,"15min":0.41399999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.254},"max":{"1min":0.52700000000000002,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.375},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48199999999999998,"15min":0.47199999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.29099999999999998},"max":{"1min":0.65500000000000003,"5min":0.72599999999999998,"15min":0.73099999999999998},"last":0.435}]},{"osd":2,"last update":"Mon Mar 18 20:23:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47399999999999998,"15min":0.47099999999999997},"min":{"1min":0.36899999999999999,"5min":0.33100000000000002,"15min":0.32900000000000001},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.502,"15min":0.504},"min":{"1min":0.39900000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64400000000000002,"5min":0.70099999999999996,"15min":0.73099999999999998},"last":0.51700000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:23:09 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.66000000000000003,"15min":0.65600000000000003},"min":{"1min":0.45900000000000002,"5min":0.436,"15min":0.42499999999999999},"max":{"1min":0.68999999999999995,"5min":0.91700000000000004,"15min":1.097},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.68999999999999995,"15min":0.67900000000000005},"min":{"1min":0.53100000000000003,"5min":0.46899999999999997,"15min":0.42899999999999999},"max":{"1min":0.86799999999999999,"5min":0.99299999999999999,"15min":1.0569999999999999},"last":0.79100000000000004}]},{"osd":5,"last update":"Mon Mar 18 20:23:09 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.70499999999999996,"15min":0.69899999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.441},"max":{"1min":0.91800000000000004,"5min":1.0489999999999999,"15min":1.1599999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.748,"15min":0.73399999999999999},"min":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.46700000000000003},"max":{"1min":0.90400000000000003,"5min":1.171,"15min":1.171},"last":0.81200000000000006}]},{"osd":6,"last update":"Mon Mar 18 20:23:09 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.70699999999999996,"15min":0.71599999999999997},"min":{"1min":0.56599999999999995,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81100000000000005,"5min":1.012,"15min":1.194},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.45100000000000001},"max":{"1min":0.83999999999999997,"5min":1.0409999999999999,"15min":1.2410000000000001},"last":0.88700000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:23:09 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72899999999999998,"15min":0.72699999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85299999999999998,"5min":1.079,"15min":1.079},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.71699999999999997,"15min":0.71099999999999997},"min":{"1min":0.48799999999999999,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.78200000000000003,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.752}]}]},{"osd":4,"up_from":13,"seq":55834575963,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60000,"kb_used_data":2636,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311840,"statfs":{"total":96636764160,"available":96575324160,"internally_reserved":0,"allocated":2699264,"data_stored":1372619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.59299999999999997,"15min":0.60399999999999998},"min":{"1min":0.40999999999999998,"5min":0.36899999999999999,"15min":0.34300000000000003},"max":{"1min":0.75600000000000001,"5min":0.76100000000000001,"15min":0.81799999999999995},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60199999999999998,"15min":0.60899999999999999},"min":{"1min":0.44500000000000001,"5min":0.40699999999999997,"15min":0.371},"max":{"1min":0.75,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.66400000000000003}]},{"osd":1,"last update":"Mon Mar 18 20:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.61699999999999999,"15min":0.622},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70299999999999996,"5min":0.81899999999999995,"15min":0.88300000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58899999999999997,"15min":0.59399999999999997},"min":{"1min":0.47599999999999998,"5min":0.38900000000000001,"15min":0.38400000000000001},"max":{"1min":0.76800000000000002,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.72499999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59299999999999997,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42599999999999999,"15min":0.39200000000000002},"max":{"1min":0.76000000000000001,"5min":0.80500000000000005,"15min":0.82599999999999996},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.622,"15min":0.627},"min":{"1min":0.44800000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78000000000000003,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.70599999999999996}]},{"osd":3,"last update":"Mon Mar 18 20:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.60699999999999998,"15min":0.60899999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.32900000000000001},"max":{"1min":0.72499999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.60599999999999998,"15min":0.627},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73499999999999999,"5min":0.79100000000000004,"15min":0.86199999999999999},"last":0.64700000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45600000000000002,"15min":0.47699999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.315},"max":{"1min":0.64600000000000002,"5min":0.70299999999999996,"15min":0.76000000000000001},"last":0.442},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48699999999999999,"15min":0.503},"min":{"1min":0.36399999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.64800000000000002,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.47499999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48199999999999998,"15min":0.503},"min":{"1min":0.375,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68200000000000005,"5min":0.71399999999999997,"15min":0.77100000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53200000000000003,"15min":0.54800000000000004},"min":{"1min":0.41499999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.754,"5min":0.76200000000000001,"15min":0.80900000000000005},"last":0.49199999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52400000000000002,"15min":0.54300000000000004},"min":{"1min":0.433,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64300000000000002,"5min":0.87,"15min":0.87},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55800000000000005,"15min":0.57999999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.78200000000000003,"5min":0.81200000000000006,"15min":0.85399999999999998},"last":0.46000000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575962,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47704,"kb_used_data":2636,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324136,"statfs":{"total":96636764160,"available":96587915264,"internally_reserved":0,"allocated":2699264,"data_stored":1372607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.61799999999999999,"15min":0.60899999999999999},"min":{"1min":0.438,"5min":0.35999999999999999,"15min":0.31900000000000001},"max":{"1min":0.81100000000000005,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.62,"15min":0.60799999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.33100000000000002},"max":{"1min":0.82599999999999996,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.58199999999999996}]},{"osd":1,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.629,"15min":0.60699999999999998},"min":{"1min":0.437,"5min":0.373,"15min":0.373},"max":{"1min":0.77700000000000002,"5min":1.056,"15min":1.056},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61699999999999999,"15min":0.60599999999999998},"min":{"1min":0.48099999999999998,"5min":0.436,"15min":0.36899999999999999},"max":{"1min":0.77700000000000002,"5min":1.018,"15min":1.018},"last":0.46800000000000003}]},{"osd":2,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61599999999999999,"15min":0.60799999999999998},"min":{"1min":0.45500000000000002,"5min":0.38900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80200000000000005,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.64000000000000001,"15min":0.61399999999999999},"min":{"1min":0.48999999999999999,"5min":0.44500000000000001,"15min":0.434},"max":{"1min":0.74099999999999999,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.496}]},{"osd":3,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.48599999999999999,"5min":0.43099999999999999,"15min":0.40799999999999997},"max":{"1min":0.754,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.66700000000000004,"15min":0.65200000000000002},"min":{"1min":0.47499999999999998,"5min":0.46999999999999997,"15min":0.46100000000000002},"max":{"1min":0.78500000000000003,"5min":1.161,"15min":1.161},"last":0.69499999999999995}]},{"osd":4,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.47999999999999998},"min":{"1min":0.41799999999999998,"5min":0.34100000000000003,"15min":0.32600000000000001},"max":{"1min":0.68799999999999994,"5min":0.74099999999999999,"15min":0.77500000000000002},"last":0.627},{"interface":"front","average":{"1min":0.502,"5min":0.51300000000000001,"15min":0.51000000000000001},"min":{"1min":0.33000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.78900000000000003,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.60799999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.34999999999999998,"5min":0.32800000000000001,"15min":0.32100000000000001},"max":{"1min":0.67200000000000004,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.53800000000000003,"15min":0.53700000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":0.63800000000000001,"5min":0.82399999999999995,"15min":0.86599999999999999},"last":0.67900000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.36899999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76300000000000001,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58299999999999996,"15min":0.57499999999999996},"min":{"1min":0.38,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73099999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.749}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:23:43.923 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:23:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.078+0000 7f799305b700 1 Processor -- start 2024-03-18T20:23:44.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.078+0000 7f799305b700 1 -- start start 2024-03-18T20:23:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.079+0000 7f799305b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f798c15d500 0x7f798c0d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.079+0000 7f799305b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f798c0d1300 0x7f798c0d17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.079+0000 7f799305b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798c0cddc0 con 0x7f798c0d1cf0 2024-03-18T20:23:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.079+0000 7f799305b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f798c0d2180 con 0x7f798c15d500 2024-03-18T20:23:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.079+0000 7f799305b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798c0d3dc0 con 0x7f798c0d1300 2024-03-18T20:23:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.079+0000 7f7990df7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f798c15d500 0x7f798c0d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.079+0000 7f7990df7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f798c15d500 0x7f798c0d0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46616/0 (socket says 172.21.15.1:46616) 2024-03-18T20:23:44.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.079+0000 7f7990df7700 1 -- 172.21.15.1:0/4102082427 learned_addr learned my addr 172.21.15.1:0/4102082427 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.079+0000 7f798bfff700 1 --2- 172.21.15.1:0/4102082427 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f798c0d1300 0x7f798c0d17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.079+0000 7f7990df7700 1 -- 172.21.15.1:0/4102082427 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f798c0d1300 msgr2=0x7f798c0d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.079+0000 7f7990df7700 1 --2- 172.21.15.1:0/4102082427 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f798c0d1300 0x7f798c0d17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.079+0000 7f7990df7700 1 -- 172.21.15.1:0/4102082427 >> v1:172.21.15.165:6789/0 conn(0x7f798c0d1cf0 legacy=0x7f798c0b83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.079+0000 7f7990df7700 1 -- 172.21.15.1:0/4102082427 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798c0ce060 con 0x7f798c15d500 2024-03-18T20:23:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.079+0000 7f798bfff700 1 --2- 172.21.15.1:0/4102082427 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f798c0d1300 0x7f798c0d17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.079+0000 7f7990df7700 1 --2- 172.21.15.1:0/4102082427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f798c15d500 0x7f798c0d0dc0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f79800098f0 tx=0x7f7980009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d88d209bd07eafb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.080+0000 7f798b7fe700 1 -- 172.21.15.1:0/4102082427 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7980004230 con 0x7f798c15d500 2024-03-18T20:23:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.080+0000 7f798b7fe700 1 -- 172.21.15.1:0/4102082427 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7980004390 con 0x7f798c15d500 2024-03-18T20:23:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.080+0000 7f798b7fe700 1 -- 172.21.15.1:0/4102082427 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79800036a0 con 0x7f798c15d500 2024-03-18T20:23:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.080+0000 7f799305b700 1 -- 172.21.15.1:0/4102082427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f798c15d500 msgr2=0x7f798c0d0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.080+0000 7f799305b700 1 --2- 172.21.15.1:0/4102082427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f798c15d500 0x7f798c0d0dc0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f79800098f0 tx=0x7f7980009c00 comp rx=0 tx=0).stop 2024-03-18T20:23:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.080+0000 7f799305b700 1 -- 172.21.15.1:0/4102082427 shutdown_connections 2024-03-18T20:23:44.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.080+0000 7f799305b700 1 --2- 172.21.15.1:0/4102082427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f798c15d500 0x7f798c0d0dc0 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.080+0000 7f799305b700 1 --2- 172.21.15.1:0/4102082427 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f798c0d1300 0x7f798c0d17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.080+0000 7f799305b700 1 -- 172.21.15.1:0/4102082427 >> 172.21.15.1:0/4102082427 conn(0x7f798c09c530 msgr2=0x7f798c0d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:44.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.080+0000 7f799305b700 1 -- 172.21.15.1:0/4102082427 shutdown_connections 2024-03-18T20:23:44.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.080+0000 7f799305b700 1 -- 172.21.15.1:0/4102082427 wait complete. 2024-03-18T20:23:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.081+0000 7f799305b700 1 Processor -- start 2024-03-18T20:23:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.081+0000 7f799305b700 1 -- start start 2024-03-18T20:23:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.082+0000 7f799305b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f798c0d1300 0x7f798c0b5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.082+0000 7f799305b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f798c0d1cf0 0x7f798c0cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.082+0000 7f799305b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f798c15d500 0x7f798c0c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.082+0000 7f799305b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798c0b72c0 con 0x7f798c15d500 2024-03-18T20:23:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.082+0000 7f799305b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798c0b6640 con 0x7f798c0d1cf0 2024-03-18T20:23:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.082+0000 7f799305b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798c0b6920 con 0x7f798c0d1300 2024-03-18T20:23:44.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.082+0000 7f79915f8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f798c15d500 0x7f798c0c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.082+0000 7f79915f8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f798c15d500 0x7f798c0c7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46630/0 (socket says 172.21.15.1:46630) 2024-03-18T20:23:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.082+0000 7f79915f8700 1 -- 172.21.15.1:0/1266669689 learned_addr learned my addr 172.21.15.1:0/1266669689 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.083+0000 7f798bfff700 1 --2- 172.21.15.1:0/1266669689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f798c0d1cf0 0x7f798c0cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.083+0000 7f7990df7700 1 --2- 172.21.15.1:0/1266669689 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f798c0d1300 0x7f798c0b5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.083+0000 7f79915f8700 1 -- 172.21.15.1:0/1266669689 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f798c0d1300 msgr2=0x7f798c0b5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.083+0000 7f79915f8700 1 --2- 172.21.15.1:0/1266669689 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f798c0d1300 0x7f798c0b5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.083+0000 7f79915f8700 1 -- 172.21.15.1:0/1266669689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f798c0d1cf0 msgr2=0x7f798c0cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.083+0000 7f79915f8700 1 --2- 172.21.15.1:0/1266669689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f798c0d1cf0 0x7f798c0cac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.083+0000 7f79915f8700 1 -- 172.21.15.1:0/1266669689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7980009550 con 0x7f798c15d500 2024-03-18T20:23:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.083+0000 7f79915f8700 1 --2- 172.21.15.1:0/1266669689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f798c15d500 0x7f798c0c7d70 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f798400cc10 tx=0x7f798400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.083+0000 7f79897fa700 1 -- 172.21.15.1:0/1266669689 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798400aa60 con 0x7f798c15d500 2024-03-18T20:23:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.083+0000 7f79897fa700 1 -- 172.21.15.1:0/1266669689 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798400abc0 con 0x7f798c15d500 2024-03-18T20:23:44.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.083+0000 7f799305b700 1 -- 172.21.15.1:0/1266669689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f798c0b6c00 con 0x7f798c15d500 2024-03-18T20:23:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.083+0000 7f79897fa700 1 -- 172.21.15.1:0/1266669689 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7984012420 con 0x7f798c15d500 2024-03-18T20:23:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.083+0000 7f799305b700 1 -- 172.21.15.1:0/1266669689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f798c0b6350 con 0x7f798c15d500 2024-03-18T20:23:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.084+0000 7f799305b700 1 -- 172.21.15.1:0/1266669689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f798c0676a0 con 0x7f798c15d500 2024-03-18T20:23:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.085+0000 7f79897fa700 1 -- 172.21.15.1:0/1266669689 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7984012580 con 0x7f798c15d500 2024-03-18T20:23:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.085+0000 7f79897fa700 1 --2- 172.21.15.1:0/1266669689 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f797006adb0 0x7f797006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:44.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.085+0000 7f79897fa700 1 -- 172.21.15.1:0/1266669689 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f7984089930 con 0x7f798c15d500 2024-03-18T20:23:44.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.085+0000 7f7990df7700 1 --2- 172.21.15.1:0/1266669689 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f797006adb0 0x7f797006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:44.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.085+0000 7f7990df7700 1 --2- 172.21.15.1:0/1266669689 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f797006adb0 0x7f797006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7980009130 tx=0x7f7980008030 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:44.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.090+0000 7f79897fa700 1 -- 172.21.15.1:0/1266669689 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7984054850 con 0x7f798c15d500 2024-03-18T20:23:44.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.287+0000 7f799305b700 1 -- 172.21.15.1:0/1266669689 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f798c0ce900 con 0x7f797006adb0 2024-03-18T20:23:44.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.295+0000 7f79897fa700 1 -- 172.21.15.1:0/1266669689 <== mgr.10797 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290017 (secure 0 0 0) 0x7f7980003720 con 0x7f797006adb0 2024-03-18T20:23:44.304 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.298+0000 7f799305b700 1 -- 172.21.15.1:0/1266669689 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f797006adb0 msgr2=0x7f797006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.298+0000 7f799305b700 1 --2- 172.21.15.1:0/1266669689 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f797006adb0 0x7f797006d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7980009130 tx=0x7f7980008030 comp rx=0 tx=0).stop 2024-03-18T20:23:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.298+0000 7f799305b700 1 -- 172.21.15.1:0/1266669689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f798c15d500 msgr2=0x7f798c0c7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.298+0000 7f799305b700 1 --2- 172.21.15.1:0/1266669689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f798c15d500 0x7f798c0c7d70 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f798400cc10 tx=0x7f798400cf20 comp rx=0 tx=0).stop 2024-03-18T20:23:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.298+0000 7f799305b700 1 -- 172.21.15.1:0/1266669689 shutdown_connections 2024-03-18T20:23:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.298+0000 7f799305b700 1 --2- 172.21.15.1:0/1266669689 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f797006adb0 0x7f797006d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.298+0000 7f799305b700 1 --2- 172.21.15.1:0/1266669689 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f798c0d1300 0x7f798c0b5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.298+0000 7f799305b700 1 --2- 172.21.15.1:0/1266669689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f798c15d500 0x7f798c0c7d70 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.298+0000 7f799305b700 1 --2- 172.21.15.1:0/1266669689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f798c0d1cf0 0x7f798c0cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.298+0000 7f799305b700 1 -- 172.21.15.1:0/1266669689 >> 172.21.15.1:0/1266669689 conn(0x7f798c09c530 msgr2=0x7f798c160430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:44.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.298+0000 7f799305b700 1 -- 172.21.15.1:0/1266669689 shutdown_connections 2024-03-18T20:23:44.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.299+0000 7f799305b700 1 -- 172.21.15.1:0/1266669689 wait complete. 2024-03-18T20:23:44.307 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:23:44.316 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-18T20:23:44.190308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":131,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":395768,"kb_used_data":22136,"kb_used_omap":12,"kb_used_meta":373491,"kb_avail":754578952,"statfs":{"total":773094113280,"available":772688846848,"internally_reserved":0,"allocated":22667264,"data_stored":12126748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":382455353},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.879646"},"pg_stats":[{"pgid":"26.3f","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590951+0000","last_change":"2024-03-18T20:23:39.775368+0000","last_active":"2024-03-18T20:23:41.590951+0000","last_peered":"2024-03-18T20:23:41.590951+0000","last_clean":"2024-03-18T20:23:41.590951+0000","last_became_active":"2024-03-18T20:23:39.775103+0000","last_became_peered":"2024-03-18T20:23:39.775103+0000","last_unstale":"2024-03-18T20:23:41.590951+0000","last_undegraded":"2024-03-18T20:23:41.590951+0000","last_fullsized":"2024-03-18T20:23:41.590951+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:56.346368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590813+0000","last_change":"2024-03-18T20:23:39.767233+0000","last_active":"2024-03-18T20:23:41.590813+0000","last_peered":"2024-03-18T20:23:41.590813+0000","last_clean":"2024-03-18T20:23:41.590813+0000","last_became_active":"2024-03-18T20:23:39.766984+0000","last_became_peered":"2024-03-18T20:23:39.766984+0000","last_unstale":"2024-03-18T20:23:41.590813+0000","last_undegraded":"2024-03-18T20:23:41.590813+0000","last_fullsized":"2024-03-18T20:23:41.590813+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:02:54.102503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3d","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763218+0000","last_change":"2024-03-18T20:23:39.761925+0000","last_active":"2024-03-18T20:23:40.763218+0000","last_peered":"2024-03-18T20:23:40.763218+0000","last_clean":"2024-03-18T20:23:40.763218+0000","last_became_active":"2024-03-18T20:23:39.761679+0000","last_became_peered":"2024-03-18T20:23:39.761679+0000","last_unstale":"2024-03-18T20:23:40.763218+0000","last_undegraded":"2024-03-18T20:23:40.763218+0000","last_fullsized":"2024-03-18T20:23:40.763218+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:55:17.412986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.774748+0000","last_change":"2024-03-18T20:23:39.772902+0000","last_active":"2024-03-18T20:23:39.774748+0000","last_peered":"2024-03-18T20:23:39.774748+0000","last_clean":"2024-03-18T20:23:39.774748+0000","last_became_active":"2024-03-18T20:23:39.772675+0000","last_became_peered":"2024-03-18T20:23:39.772675+0000","last_unstale":"2024-03-18T20:23:39.774748+0000","last_undegraded":"2024-03-18T20:23:39.774748+0000","last_fullsized":"2024-03-18T20:23:39.774748+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:12:18.174454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780861+0000","last_change":"2024-03-18T20:23:39.772041+0000","last_active":"2024-03-18T20:23:39.780861+0000","last_peered":"2024-03-18T20:23:39.780861+0000","last_clean":"2024-03-18T20:23:39.780861+0000","last_became_active":"2024-03-18T20:23:39.771843+0000","last_became_peered":"2024-03-18T20:23:39.771843+0000","last_unstale":"2024-03-18T20:23:39.780861+0000","last_undegraded":"2024-03-18T20:23:39.780861+0000","last_fullsized":"2024-03-18T20:23:39.780861+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:24:28.800911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780896+0000","last_change":"2024-03-18T20:23:39.775650+0000","last_active":"2024-03-18T20:23:39.780896+0000","last_peered":"2024-03-18T20:23:39.780896+0000","last_clean":"2024-03-18T20:23:39.780896+0000","last_became_active":"2024-03-18T20:23:39.775190+0000","last_became_peered":"2024-03-18T20:23:39.775190+0000","last_unstale":"2024-03-18T20:23:39.780896+0000","last_undegraded":"2024-03-18T20:23:39.780896+0000","last_fullsized":"2024-03-18T20:23:39.780896+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:48.403628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763368+0000","last_change":"2024-03-18T20:23:39.772541+0000","last_active":"2024-03-18T20:23:40.763368+0000","last_peered":"2024-03-18T20:23:40.763368+0000","last_clean":"2024-03-18T20:23:40.763368+0000","last_became_active":"2024-03-18T20:23:39.764037+0000","last_became_peered":"2024-03-18T20:23:39.764037+0000","last_unstale":"2024-03-18T20:23:40.763368+0000","last_undegraded":"2024-03-18T20:23:40.763368+0000","last_fullsized":"2024-03-18T20:23:40.763368+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:12.307651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:42.006874+0000","last_change":"2024-03-18T20:23:39.774083+0000","last_active":"2024-03-18T20:23:42.006874+0000","last_peered":"2024-03-18T20:23:42.006874+0000","last_clean":"2024-03-18T20:23:42.006874+0000","last_became_active":"2024-03-18T20:23:39.773016+0000","last_became_peered":"2024-03-18T20:23:39.773016+0000","last_unstale":"2024-03-18T20:23:42.006874+0000","last_undegraded":"2024-03-18T20:23:42.006874+0000","last_fullsized":"2024-03-18T20:23:42.006874+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:01:47.194750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590887+0000","last_change":"2024-03-18T20:23:39.767854+0000","last_active":"2024-03-18T20:23:41.590887+0000","last_peered":"2024-03-18T20:23:41.590887+0000","last_clean":"2024-03-18T20:23:41.590887+0000","last_became_active":"2024-03-18T20:23:39.767614+0000","last_became_peered":"2024-03-18T20:23:39.767614+0000","last_unstale":"2024-03-18T20:23:41.590887+0000","last_undegraded":"2024-03-18T20:23:41.590887+0000","last_fullsized":"2024-03-18T20:23:41.590887+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:00:42.414528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.781163+0000","last_change":"2024-03-18T20:23:39.778000+0000","last_active":"2024-03-18T20:23:39.781163+0000","last_peered":"2024-03-18T20:23:39.781163+0000","last_clean":"2024-03-18T20:23:39.781163+0000","last_became_active":"2024-03-18T20:23:39.777388+0000","last_became_peered":"2024-03-18T20:23:39.777388+0000","last_unstale":"2024-03-18T20:23:39.781163+0000","last_undegraded":"2024-03-18T20:23:39.781163+0000","last_fullsized":"2024-03-18T20:23:39.781163+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:14.173247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.006906+0000","last_change":"2024-03-18T20:23:39.763882+0000","last_active":"2024-03-18T20:23:41.006906+0000","last_peered":"2024-03-18T20:23:41.006906+0000","last_clean":"2024-03-18T20:23:41.006906+0000","last_became_active":"2024-03-18T20:23:39.763608+0000","last_became_peered":"2024-03-18T20:23:39.763608+0000","last_unstale":"2024-03-18T20:23:41.006906+0000","last_undegraded":"2024-03-18T20:23:41.006906+0000","last_fullsized":"2024-03-18T20:23:41.006906+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:59.000708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.770769+0000","last_change":"2024-03-18T20:23:39.766381+0000","last_active":"2024-03-18T20:23:39.770769+0000","last_peered":"2024-03-18T20:23:39.770769+0000","last_clean":"2024-03-18T20:23:39.770769+0000","last_became_active":"2024-03-18T20:23:39.766058+0000","last_became_peered":"2024-03-18T20:23:39.766058+0000","last_unstale":"2024-03-18T20:23:39.770769+0000","last_undegraded":"2024-03-18T20:23:39.770769+0000","last_fullsized":"2024-03-18T20:23:39.770769+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:26.058894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970640+0000","last_change":"2024-03-18T20:23:39.758470+0000","last_active":"2024-03-18T20:23:40.970640+0000","last_peered":"2024-03-18T20:23:40.970640+0000","last_clean":"2024-03-18T20:23:40.970640+0000","last_became_active":"2024-03-18T20:23:39.758170+0000","last_became_peered":"2024-03-18T20:23:39.758170+0000","last_unstale":"2024-03-18T20:23:40.970640+0000","last_undegraded":"2024-03-18T20:23:40.970640+0000","last_fullsized":"2024-03-18T20:23:40.970640+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:41.949264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763446+0000","last_change":"2024-03-18T20:23:39.773025+0000","last_active":"2024-03-18T20:23:40.763446+0000","last_peered":"2024-03-18T20:23:40.763446+0000","last_clean":"2024-03-18T20:23:40.763446+0000","last_became_active":"2024-03-18T20:23:39.772723+0000","last_became_peered":"2024-03-18T20:23:39.772723+0000","last_unstale":"2024-03-18T20:23:40.763446+0000","last_undegraded":"2024-03-18T20:23:40.763446+0000","last_fullsized":"2024-03-18T20:23:40.763446+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:02:35.258826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590515+0000","last_change":"2024-03-18T20:23:39.767200+0000","last_active":"2024-03-18T20:23:41.590515+0000","last_peered":"2024-03-18T20:23:41.590515+0000","last_clean":"2024-03-18T20:23:41.590515+0000","last_became_active":"2024-03-18T20:23:39.767035+0000","last_became_peered":"2024-03-18T20:23:39.767035+0000","last_unstale":"2024-03-18T20:23:41.590515+0000","last_undegraded":"2024-03-18T20:23:41.590515+0000","last_fullsized":"2024-03-18T20:23:41.590515+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:35.116397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780705+0000","last_change":"2024-03-18T20:23:39.771329+0000","last_active":"2024-03-18T20:23:39.780705+0000","last_peered":"2024-03-18T20:23:39.780705+0000","last_clean":"2024-03-18T20:23:39.780705+0000","last_became_active":"2024-03-18T20:23:39.768752+0000","last_became_peered":"2024-03-18T20:23:39.768752+0000","last_unstale":"2024-03-18T20:23:39.780705+0000","last_undegraded":"2024-03-18T20:23:39.780705+0000","last_fullsized":"2024-03-18T20:23:39.780705+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:33:12.771579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780905+0000","last_change":"2024-03-18T20:23:39.775925+0000","last_active":"2024-03-18T20:23:39.780905+0000","last_peered":"2024-03-18T20:23:39.780905+0000","last_clean":"2024-03-18T20:23:39.780905+0000","last_became_active":"2024-03-18T20:23:39.775636+0000","last_became_peered":"2024-03-18T20:23:39.775636+0000","last_unstale":"2024-03-18T20:23:39.780905+0000","last_undegraded":"2024-03-18T20:23:39.780905+0000","last_fullsized":"2024-03-18T20:23:39.780905+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:34:32.364043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.971006+0000","last_change":"2024-03-18T20:23:39.765350+0000","last_active":"2024-03-18T20:23:40.971006+0000","last_peered":"2024-03-18T20:23:40.971006+0000","last_clean":"2024-03-18T20:23:40.971006+0000","last_became_active":"2024-03-18T20:23:39.765082+0000","last_became_peered":"2024-03-18T20:23:39.765082+0000","last_unstale":"2024-03-18T20:23:40.971006+0000","last_undegraded":"2024-03-18T20:23:40.971006+0000","last_fullsized":"2024-03-18T20:23:40.971006+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:59.000708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970906+0000","last_change":"2024-03-18T20:23:39.762644+0000","last_active":"2024-03-18T20:23:40.970906+0000","last_peered":"2024-03-18T20:23:40.970906+0000","last_clean":"2024-03-18T20:23:40.970906+0000","last_became_active":"2024-03-18T20:23:39.760016+0000","last_became_peered":"2024-03-18T20:23:39.760016+0000","last_unstale":"2024-03-18T20:23:40.970906+0000","last_undegraded":"2024-03-18T20:23:40.970906+0000","last_fullsized":"2024-03-18T20:23:40.970906+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:21:09.949066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780772+0000","last_change":"2024-03-18T20:23:39.772736+0000","last_active":"2024-03-18T20:23:39.780772+0000","last_peered":"2024-03-18T20:23:39.780772+0000","last_clean":"2024-03-18T20:23:39.780772+0000","last_became_active":"2024-03-18T20:23:39.772317+0000","last_became_peered":"2024-03-18T20:23:39.772317+0000","last_unstale":"2024-03-18T20:23:39.780772+0000","last_undegraded":"2024-03-18T20:23:39.780772+0000","last_fullsized":"2024-03-18T20:23:39.780772+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:46:57.873851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590894+0000","last_change":"2024-03-18T20:23:39.765153+0000","last_active":"2024-03-18T20:23:41.590894+0000","last_peered":"2024-03-18T20:23:41.590894+0000","last_clean":"2024-03-18T20:23:41.590894+0000","last_became_active":"2024-03-18T20:23:39.764882+0000","last_became_peered":"2024-03-18T20:23:39.764882+0000","last_unstale":"2024-03-18T20:23:41.590894+0000","last_undegraded":"2024-03-18T20:23:41.590894+0000","last_fullsized":"2024-03-18T20:23:41.590894+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:26.058894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590827+0000","last_change":"2024-03-18T20:23:39.766755+0000","last_active":"2024-03-18T20:23:41.590827+0000","last_peered":"2024-03-18T20:23:41.590827+0000","last_clean":"2024-03-18T20:23:41.590827+0000","last_became_active":"2024-03-18T20:23:39.766368+0000","last_became_peered":"2024-03-18T20:23:39.766368+0000","last_unstale":"2024-03-18T20:23:41.590827+0000","last_undegraded":"2024-03-18T20:23:41.590827+0000","last_fullsized":"2024-03-18T20:23:41.590827+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:51:11.509782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:42.006588+0000","last_change":"2024-03-18T20:23:39.771477+0000","last_active":"2024-03-18T20:23:42.006588+0000","last_peered":"2024-03-18T20:23:42.006588+0000","last_clean":"2024-03-18T20:23:42.006588+0000","last_became_active":"2024-03-18T20:23:39.771245+0000","last_became_peered":"2024-03-18T20:23:39.771245+0000","last_unstale":"2024-03-18T20:23:42.006588+0000","last_undegraded":"2024-03-18T20:23:42.006588+0000","last_fullsized":"2024-03-18T20:23:42.006588+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:10.588668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590402+0000","last_change":"2024-03-18T20:23:39.764714+0000","last_active":"2024-03-18T20:23:41.590402+0000","last_peered":"2024-03-18T20:23:41.590402+0000","last_clean":"2024-03-18T20:23:41.590402+0000","last_became_active":"2024-03-18T20:23:39.764385+0000","last_became_peered":"2024-03-18T20:23:39.764385+0000","last_unstale":"2024-03-18T20:23:41.590402+0000","last_undegraded":"2024-03-18T20:23:41.590402+0000","last_fullsized":"2024-03-18T20:23:41.590402+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:26:24.225550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763632+0000","last_change":"2024-03-18T20:23:39.778928+0000","last_active":"2024-03-18T20:23:40.763632+0000","last_peered":"2024-03-18T20:23:40.763632+0000","last_clean":"2024-03-18T20:23:40.763632+0000","last_became_active":"2024-03-18T20:23:39.778655+0000","last_became_peered":"2024-03-18T20:23:39.778655+0000","last_unstale":"2024-03-18T20:23:40.763632+0000","last_undegraded":"2024-03-18T20:23:40.763632+0000","last_fullsized":"2024-03-18T20:23:40.763632+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:03:54.384457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.007060+0000","last_change":"2024-03-18T20:23:39.771221+0000","last_active":"2024-03-18T20:23:41.007060+0000","last_peered":"2024-03-18T20:23:41.007060+0000","last_clean":"2024-03-18T20:23:41.007060+0000","last_became_active":"2024-03-18T20:23:39.768932+0000","last_became_peered":"2024-03-18T20:23:39.768932+0000","last_unstale":"2024-03-18T20:23:41.007060+0000","last_undegraded":"2024-03-18T20:23:41.007060+0000","last_fullsized":"2024-03-18T20:23:41.007060+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:41.949264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:42.006688+0000","last_change":"2024-03-18T20:23:39.764476+0000","last_active":"2024-03-18T20:23:42.006688+0000","last_peered":"2024-03-18T20:23:42.006688+0000","last_clean":"2024-03-18T20:23:42.006688+0000","last_became_active":"2024-03-18T20:23:39.764132+0000","last_became_peered":"2024-03-18T20:23:39.764132+0000","last_unstale":"2024-03-18T20:23:42.006688+0000","last_undegraded":"2024-03-18T20:23:42.006688+0000","last_fullsized":"2024-03-18T20:23:42.006688+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:04:48.691324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780654+0000","last_change":"2024-03-18T20:23:39.768930+0000","last_active":"2024-03-18T20:23:39.780654+0000","last_peered":"2024-03-18T20:23:39.780654+0000","last_clean":"2024-03-18T20:23:39.780654+0000","last_became_active":"2024-03-18T20:23:39.768653+0000","last_became_peered":"2024-03-18T20:23:39.768653+0000","last_unstale":"2024-03-18T20:23:39.780654+0000","last_undegraded":"2024-03-18T20:23:39.780654+0000","last_fullsized":"2024-03-18T20:23:39.780654+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:51:42.338719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.006883+0000","last_change":"2024-03-18T20:23:39.768945+0000","last_active":"2024-03-18T20:23:41.006883+0000","last_peered":"2024-03-18T20:23:41.006883+0000","last_clean":"2024-03-18T20:23:41.006883+0000","last_became_active":"2024-03-18T20:23:39.768666+0000","last_became_peered":"2024-03-18T20:23:39.768666+0000","last_unstale":"2024-03-18T20:23:41.006883+0000","last_undegraded":"2024-03-18T20:23:41.006883+0000","last_fullsized":"2024-03-18T20:23:41.006883+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:01:47.194750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.006745+0000","last_change":"2024-03-18T20:23:39.768712+0000","last_active":"2024-03-18T20:23:41.006745+0000","last_peered":"2024-03-18T20:23:41.006745+0000","last_clean":"2024-03-18T20:23:41.006745+0000","last_became_active":"2024-03-18T20:23:39.768441+0000","last_became_peered":"2024-03-18T20:23:39.768441+0000","last_unstale":"2024-03-18T20:23:41.006745+0000","last_undegraded":"2024-03-18T20:23:41.006745+0000","last_fullsized":"2024-03-18T20:23:41.006745+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:10.588668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763495+0000","last_change":"2024-03-18T20:23:39.775405+0000","last_active":"2024-03-18T20:23:40.763495+0000","last_peered":"2024-03-18T20:23:40.763495+0000","last_clean":"2024-03-18T20:23:40.763495+0000","last_became_active":"2024-03-18T20:23:39.775096+0000","last_became_peered":"2024-03-18T20:23:39.775096+0000","last_unstale":"2024-03-18T20:23:40.763495+0000","last_undegraded":"2024-03-18T20:23:40.763495+0000","last_fullsized":"2024-03-18T20:23:40.763495+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:56.694065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.781335+0000","last_change":"2024-03-18T20:23:39.779447+0000","last_active":"2024-03-18T20:23:39.781335+0000","last_peered":"2024-03-18T20:23:39.781335+0000","last_clean":"2024-03-18T20:23:39.781335+0000","last_became_active":"2024-03-18T20:23:39.769075+0000","last_became_peered":"2024-03-18T20:23:39.769075+0000","last_unstale":"2024-03-18T20:23:39.781335+0000","last_undegraded":"2024-03-18T20:23:39.781335+0000","last_fullsized":"2024-03-18T20:23:39.781335+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:11.412667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:42.006835+0000","last_change":"2024-03-18T20:23:39.763471+0000","last_active":"2024-03-18T20:23:42.006835+0000","last_peered":"2024-03-18T20:23:42.006835+0000","last_clean":"2024-03-18T20:23:42.006835+0000","last_became_active":"2024-03-18T20:23:39.763142+0000","last_became_peered":"2024-03-18T20:23:39.763142+0000","last_unstale":"2024-03-18T20:23:42.006835+0000","last_undegraded":"2024-03-18T20:23:42.006835+0000","last_fullsized":"2024-03-18T20:23:42.006835+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:37.066466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"213'4","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.778347+0000","last_change":"2024-03-18T20:23:39.337683+0000","last_active":"2024-03-18T20:23:39.778347+0000","last_peered":"2024-03-18T20:23:39.778347+0000","last_clean":"2024-03-18T20:23:39.778347+0000","last_became_active":"2024-03-18T20:23:39.337383+0000","last_became_peered":"2024-03-18T20:23:39.337383+0000","last_unstale":"2024-03-18T20:23:39.778347+0000","last_undegraded":"2024-03-18T20:23:39.778347+0000","last_fullsized":"2024-03-18T20:23:39.778347+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:02:01.706948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763702+0000","last_change":"2024-03-18T20:23:39.778651+0000","last_active":"2024-03-18T20:23:40.763702+0000","last_peered":"2024-03-18T20:23:40.763702+0000","last_clean":"2024-03-18T20:23:40.763702+0000","last_became_active":"2024-03-18T20:23:39.773401+0000","last_became_peered":"2024-03-18T20:23:39.773401+0000","last_unstale":"2024-03-18T20:23:40.763702+0000","last_undegraded":"2024-03-18T20:23:40.763702+0000","last_fullsized":"2024-03-18T20:23:40.763702+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:27:50.812568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"213'1","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.781104+0000","last_change":"2024-03-18T20:23:39.336840+0000","last_active":"2024-03-18T20:23:39.781104+0000","last_peered":"2024-03-18T20:23:39.781104+0000","last_clean":"2024-03-18T20:23:39.781104+0000","last_became_active":"2024-03-18T20:23:39.336484+0000","last_became_peered":"2024-03-18T20:23:39.336484+0000","last_unstale":"2024-03-18T20:23:39.781104+0000","last_undegraded":"2024-03-18T20:23:39.781104+0000","last_fullsized":"2024-03-18T20:23:39.781104+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:06.114833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.007113+0000","last_change":"2024-03-18T20:23:39.765405+0000","last_active":"2024-03-18T20:23:41.007113+0000","last_peered":"2024-03-18T20:23:41.007113+0000","last_clean":"2024-03-18T20:23:41.007113+0000","last_became_active":"2024-03-18T20:23:39.764143+0000","last_became_peered":"2024-03-18T20:23:39.764143+0000","last_unstale":"2024-03-18T20:23:41.007113+0000","last_undegraded":"2024-03-18T20:23:41.007113+0000","last_fullsized":"2024-03-18T20:23:41.007113+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:35.217366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"213'2","reported_seq":13,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.006962+0000","last_change":"2024-03-18T20:23:39.328337+0000","last_active":"2024-03-18T20:23:41.006962+0000","last_peered":"2024-03-18T20:23:41.006962+0000","last_clean":"2024-03-18T20:23:41.006962+0000","last_became_active":"2024-03-18T20:23:39.328070+0000","last_became_peered":"2024-03-18T20:23:39.328070+0000","last_unstale":"2024-03-18T20:23:41.006962+0000","last_undegraded":"2024-03-18T20:23:41.006962+0000","last_fullsized":"2024-03-18T20:23:41.006962+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:11.301777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763429+0000","last_change":"2024-03-18T20:23:39.762894+0000","last_active":"2024-03-18T20:23:40.763429+0000","last_peered":"2024-03-18T20:23:40.763429+0000","last_clean":"2024-03-18T20:23:40.763429+0000","last_became_active":"2024-03-18T20:23:39.762628+0000","last_became_peered":"2024-03-18T20:23:39.762628+0000","last_unstale":"2024-03-18T20:23:40.763429+0000","last_undegraded":"2024-03-18T20:23:40.763429+0000","last_fullsized":"2024-03-18T20:23:40.763429+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:55:47.781505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"213'2","reported_seq":13,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970851+0000","last_change":"2024-03-18T20:23:39.333280+0000","last_active":"2024-03-18T20:23:40.970851+0000","last_peered":"2024-03-18T20:23:40.970851+0000","last_clean":"2024-03-18T20:23:40.970851+0000","last_became_active":"2024-03-18T20:23:39.333041+0000","last_became_peered":"2024-03-18T20:23:39.333041+0000","last_unstale":"2024-03-18T20:23:40.970851+0000","last_undegraded":"2024-03-18T20:23:40.970851+0000","last_fullsized":"2024-03-18T20:23:40.970851+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:06.137232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970560+0000","last_change":"2024-03-18T20:23:39.774714+0000","last_active":"2024-03-18T20:23:40.970560+0000","last_peered":"2024-03-18T20:23:40.970560+0000","last_clean":"2024-03-18T20:23:40.970560+0000","last_became_active":"2024-03-18T20:23:39.774461+0000","last_became_peered":"2024-03-18T20:23:39.774461+0000","last_unstale":"2024-03-18T20:23:40.970560+0000","last_undegraded":"2024-03-18T20:23:40.970560+0000","last_fullsized":"2024-03-18T20:23:40.970560+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:01:47.194750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"213'1","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780779+0000","last_change":"2024-03-18T20:23:39.338440+0000","last_active":"2024-03-18T20:23:39.780779+0000","last_peered":"2024-03-18T20:23:39.780779+0000","last_clean":"2024-03-18T20:23:39.780779+0000","last_became_active":"2024-03-18T20:23:39.338138+0000","last_became_peered":"2024-03-18T20:23:39.338138+0000","last_unstale":"2024-03-18T20:23:39.780779+0000","last_undegraded":"2024-03-18T20:23:39.780779+0000","last_fullsized":"2024-03-18T20:23:39.780779+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:01:47.102858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590655+0000","last_change":"2024-03-18T20:23:39.763657+0000","last_active":"2024-03-18T20:23:41.590655+0000","last_peered":"2024-03-18T20:23:41.590655+0000","last_clean":"2024-03-18T20:23:41.590655+0000","last_became_active":"2024-03-18T20:23:39.763463+0000","last_became_peered":"2024-03-18T20:23:39.763463+0000","last_unstale":"2024-03-18T20:23:41.590655+0000","last_undegraded":"2024-03-18T20:23:41.590655+0000","last_fullsized":"2024-03-18T20:23:41.590655+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:12:18.174454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"213'1","reported_seq":11,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.781173+0000","last_change":"2024-03-18T20:23:39.339137+0000","last_active":"2024-03-18T20:23:39.781173+0000","last_peered":"2024-03-18T20:23:39.781173+0000","last_clean":"2024-03-18T20:23:39.781173+0000","last_became_active":"2024-03-18T20:23:39.338816+0000","last_became_peered":"2024-03-18T20:23:39.338816+0000","last_unstale":"2024-03-18T20:23:39.781173+0000","last_undegraded":"2024-03-18T20:23:39.781173+0000","last_fullsized":"2024-03-18T20:23:39.781173+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:39.916823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590719+0000","last_change":"2024-03-18T20:23:39.766598+0000","last_active":"2024-03-18T20:23:41.590719+0000","last_peered":"2024-03-18T20:23:41.590719+0000","last_clean":"2024-03-18T20:23:41.590719+0000","last_became_active":"2024-03-18T20:23:39.766094+0000","last_became_peered":"2024-03-18T20:23:39.766094+0000","last_unstale":"2024-03-18T20:23:41.590719+0000","last_undegraded":"2024-03-18T20:23:41.590719+0000","last_fullsized":"2024-03-18T20:23:41.590719+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:41:40.231425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970740+0000","last_change":"2024-03-18T20:23:39.327614+0000","last_active":"2024-03-18T20:23:40.970740+0000","last_peered":"2024-03-18T20:23:40.970740+0000","last_clean":"2024-03-18T20:23:40.970740+0000","last_became_active":"2024-03-18T20:23:39.327310+0000","last_became_peered":"2024-03-18T20:23:39.327310+0000","last_unstale":"2024-03-18T20:23:40.970740+0000","last_undegraded":"2024-03-18T20:23:40.970740+0000","last_fullsized":"2024-03-18T20:23:40.970740+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:27:49.806694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:42.006794+0000","last_change":"2024-03-18T20:23:39.775571+0000","last_active":"2024-03-18T20:23:42.006794+0000","last_peered":"2024-03-18T20:23:42.006794+0000","last_clean":"2024-03-18T20:23:42.006794+0000","last_became_active":"2024-03-18T20:23:39.775299+0000","last_became_peered":"2024-03-18T20:23:39.775299+0000","last_unstale":"2024-03-18T20:23:42.006794+0000","last_undegraded":"2024-03-18T20:23:42.006794+0000","last_fullsized":"2024-03-18T20:23:42.006794+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:33:55.665965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.757193+0000","last_change":"2024-03-18T20:23:39.338549+0000","last_active":"2024-03-18T20:23:39.757193+0000","last_peered":"2024-03-18T20:23:39.757193+0000","last_clean":"2024-03-18T20:23:39.757193+0000","last_became_active":"2024-03-18T20:23:39.338300+0000","last_became_peered":"2024-03-18T20:23:39.338300+0000","last_unstale":"2024-03-18T20:23:39.757193+0000","last_undegraded":"2024-03-18T20:23:39.757193+0000","last_fullsized":"2024-03-18T20:23:39.757193+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:44.978976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.781024+0000","last_change":"2024-03-18T20:23:39.777131+0000","last_active":"2024-03-18T20:23:39.781024+0000","last_peered":"2024-03-18T20:23:39.781024+0000","last_clean":"2024-03-18T20:23:39.781024+0000","last_became_active":"2024-03-18T20:23:39.776409+0000","last_became_peered":"2024-03-18T20:23:39.776409+0000","last_unstale":"2024-03-18T20:23:39.781024+0000","last_undegraded":"2024-03-18T20:23:39.781024+0000","last_fullsized":"2024-03-18T20:23:39.781024+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:40:00.542040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590937+0000","last_change":"2024-03-18T20:23:39.332818+0000","last_active":"2024-03-18T20:23:41.590937+0000","last_peered":"2024-03-18T20:23:41.590937+0000","last_clean":"2024-03-18T20:23:41.590937+0000","last_became_active":"2024-03-18T20:23:39.332328+0000","last_became_peered":"2024-03-18T20:23:39.332328+0000","last_unstale":"2024-03-18T20:23:41.590937+0000","last_undegraded":"2024-03-18T20:23:41.590937+0000","last_fullsized":"2024-03-18T20:23:41.590937+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:41:36.473564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.971098+0000","last_change":"2024-03-18T20:23:39.759713+0000","last_active":"2024-03-18T20:23:40.971098+0000","last_peered":"2024-03-18T20:23:40.971098+0000","last_clean":"2024-03-18T20:23:40.971098+0000","last_became_active":"2024-03-18T20:23:39.759422+0000","last_became_peered":"2024-03-18T20:23:39.759422+0000","last_unstale":"2024-03-18T20:23:40.971098+0000","last_undegraded":"2024-03-18T20:23:40.971098+0000","last_fullsized":"2024-03-18T20:23:40.971098+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:55:04.135410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"213'3","reported_seq":13,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.781025+0000","last_change":"2024-03-18T20:23:39.339071+0000","last_active":"2024-03-18T20:23:39.781025+0000","last_peered":"2024-03-18T20:23:39.781025+0000","last_clean":"2024-03-18T20:23:39.781025+0000","last_became_active":"2024-03-18T20:23:39.338717+0000","last_became_peered":"2024-03-18T20:23:39.338717+0000","last_unstale":"2024-03-18T20:23:39.781025+0000","last_undegraded":"2024-03-18T20:23:39.781025+0000","last_fullsized":"2024-03-18T20:23:39.781025+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:22:55.858384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.591030+0000","last_change":"2024-03-18T20:23:39.774364+0000","last_active":"2024-03-18T20:23:41.591030+0000","last_peered":"2024-03-18T20:23:41.591030+0000","last_clean":"2024-03-18T20:23:41.591030+0000","last_became_active":"2024-03-18T20:23:39.774095+0000","last_became_peered":"2024-03-18T20:23:39.774095+0000","last_unstale":"2024-03-18T20:23:41.591030+0000","last_undegraded":"2024-03-18T20:23:41.591030+0000","last_fullsized":"2024-03-18T20:23:41.591030+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:32:20.890795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"213'6","reported_seq":17,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970935+0000","last_change":"2024-03-18T20:23:39.329381+0000","last_active":"2024-03-18T20:23:40.970935+0000","last_peered":"2024-03-18T20:23:40.970935+0000","last_clean":"2024-03-18T20:23:40.970935+0000","last_became_active":"2024-03-18T20:23:39.329110+0000","last_became_peered":"2024-03-18T20:23:39.329110+0000","last_unstale":"2024-03-18T20:23:40.970935+0000","last_undegraded":"2024-03-18T20:23:40.970935+0000","last_fullsized":"2024-03-18T20:23:40.970935+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:30.833726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763343+0000","last_change":"2024-03-18T20:23:39.772880+0000","last_active":"2024-03-18T20:23:40.763343+0000","last_peered":"2024-03-18T20:23:40.763343+0000","last_clean":"2024-03-18T20:23:40.763343+0000","last_became_active":"2024-03-18T20:23:39.764158+0000","last_became_peered":"2024-03-18T20:23:39.764158+0000","last_unstale":"2024-03-18T20:23:40.763343+0000","last_undegraded":"2024-03-18T20:23:40.763343+0000","last_fullsized":"2024-03-18T20:23:40.763343+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:07.143106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"213'2","reported_seq":13,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970837+0000","last_change":"2024-03-18T20:23:39.332410+0000","last_active":"2024-03-18T20:23:40.970837+0000","last_peered":"2024-03-18T20:23:40.970837+0000","last_clean":"2024-03-18T20:23:40.970837+0000","last_became_active":"2024-03-18T20:23:39.332168+0000","last_became_peered":"2024-03-18T20:23:39.332168+0000","last_unstale":"2024-03-18T20:23:40.970837+0000","last_undegraded":"2024-03-18T20:23:40.970837+0000","last_fullsized":"2024-03-18T20:23:40.970837+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:28:59.985890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780722+0000","last_change":"2024-03-18T20:23:39.772098+0000","last_active":"2024-03-18T20:23:39.780722+0000","last_peered":"2024-03-18T20:23:39.780722+0000","last_clean":"2024-03-18T20:23:39.780722+0000","last_became_active":"2024-03-18T20:23:39.771829+0000","last_became_peered":"2024-03-18T20:23:39.771829+0000","last_unstale":"2024-03-18T20:23:39.780722+0000","last_undegraded":"2024-03-18T20:23:39.780722+0000","last_fullsized":"2024-03-18T20:23:39.780722+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:45.947401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590955+0000","last_change":"2024-03-18T20:23:39.335648+0000","last_active":"2024-03-18T20:23:41.590955+0000","last_peered":"2024-03-18T20:23:41.590955+0000","last_clean":"2024-03-18T20:23:41.590955+0000","last_became_active":"2024-03-18T20:23:39.335381+0000","last_became_peered":"2024-03-18T20:23:39.335381+0000","last_unstale":"2024-03-18T20:23:41.590955+0000","last_undegraded":"2024-03-18T20:23:41.590955+0000","last_fullsized":"2024-03-18T20:23:41.590955+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:42.078249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970795+0000","last_change":"2024-03-18T20:23:39.772922+0000","last_active":"2024-03-18T20:23:40.970795+0000","last_peered":"2024-03-18T20:23:40.970795+0000","last_clean":"2024-03-18T20:23:40.970795+0000","last_became_active":"2024-03-18T20:23:39.772605+0000","last_became_peered":"2024-03-18T20:23:39.772605+0000","last_unstale":"2024-03-18T20:23:40.970795+0000","last_undegraded":"2024-03-18T20:23:40.970795+0000","last_fullsized":"2024-03-18T20:23:40.970795+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:10.588668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.760976+0000","last_change":"2024-03-18T20:23:39.335354+0000","last_active":"2024-03-18T20:23:39.760976+0000","last_peered":"2024-03-18T20:23:39.760976+0000","last_clean":"2024-03-18T20:23:39.760976+0000","last_became_active":"2024-03-18T20:23:39.335084+0000","last_became_peered":"2024-03-18T20:23:39.335084+0000","last_unstale":"2024-03-18T20:23:39.760976+0000","last_undegraded":"2024-03-18T20:23:39.760976+0000","last_fullsized":"2024-03-18T20:23:39.760976+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:58:57.300557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970793+0000","last_change":"2024-03-18T20:23:39.766941+0000","last_active":"2024-03-18T20:23:40.970793+0000","last_peered":"2024-03-18T20:23:40.970793+0000","last_clean":"2024-03-18T20:23:40.970793+0000","last_became_active":"2024-03-18T20:23:39.766689+0000","last_became_peered":"2024-03-18T20:23:39.766689+0000","last_unstale":"2024-03-18T20:23:40.970793+0000","last_undegraded":"2024-03-18T20:23:40.970793+0000","last_fullsized":"2024-03-18T20:23:40.970793+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:30.491107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"213'8","reported_seq":19,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.006742+0000","last_change":"2024-03-18T20:23:39.331204+0000","last_active":"2024-03-18T20:23:41.006742+0000","last_peered":"2024-03-18T20:23:41.006742+0000","last_clean":"2024-03-18T20:23:41.006742+0000","last_became_active":"2024-03-18T20:23:39.330947+0000","last_became_peered":"2024-03-18T20:23:39.330947+0000","last_unstale":"2024-03-18T20:23:41.006742+0000","last_undegraded":"2024-03-18T20:23:41.006742+0000","last_fullsized":"2024-03-18T20:23:41.006742+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:39.419825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763671+0000","last_change":"2024-03-18T20:23:39.775343+0000","last_active":"2024-03-18T20:23:40.763671+0000","last_peered":"2024-03-18T20:23:40.763671+0000","last_clean":"2024-03-18T20:23:40.763671+0000","last_became_active":"2024-03-18T20:23:39.774987+0000","last_became_peered":"2024-03-18T20:23:39.774987+0000","last_unstale":"2024-03-18T20:23:40.763671+0000","last_undegraded":"2024-03-18T20:23:40.763671+0000","last_fullsized":"2024-03-18T20:23:40.763671+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:40.425699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"213'1","reported_seq":12,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.007142+0000","last_change":"2024-03-18T20:23:39.335549+0000","last_active":"2024-03-18T20:23:41.007142+0000","last_peered":"2024-03-18T20:23:41.007142+0000","last_clean":"2024-03-18T20:23:41.007142+0000","last_became_active":"2024-03-18T20:23:39.335279+0000","last_became_peered":"2024-03-18T20:23:39.335279+0000","last_unstale":"2024-03-18T20:23:41.007142+0000","last_undegraded":"2024-03-18T20:23:41.007142+0000","last_fullsized":"2024-03-18T20:23:41.007142+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:02:34.252952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780852+0000","last_change":"2024-03-18T20:23:39.774804+0000","last_active":"2024-03-18T20:23:39.780852+0000","last_peered":"2024-03-18T20:23:39.780852+0000","last_clean":"2024-03-18T20:23:39.780852+0000","last_became_active":"2024-03-18T20:23:39.774539+0000","last_became_peered":"2024-03-18T20:23:39.774539+0000","last_unstale":"2024-03-18T20:23:39.780852+0000","last_undegraded":"2024-03-18T20:23:39.780852+0000","last_fullsized":"2024-03-18T20:23:39.780852+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:43.947623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"213'2","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.773318+0000","last_change":"2024-03-18T20:23:39.335330+0000","last_active":"2024-03-18T20:23:39.773318+0000","last_peered":"2024-03-18T20:23:39.773318+0000","last_clean":"2024-03-18T20:23:39.773318+0000","last_became_active":"2024-03-18T20:23:39.335061+0000","last_became_peered":"2024-03-18T20:23:39.335061+0000","last_unstale":"2024-03-18T20:23:39.773318+0000","last_undegraded":"2024-03-18T20:23:39.773318+0000","last_fullsized":"2024-03-18T20:23:39.773318+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:42:50.684286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.007142+0000","last_change":"2024-03-18T20:23:39.770963+0000","last_active":"2024-03-18T20:23:41.007142+0000","last_peered":"2024-03-18T20:23:41.007142+0000","last_clean":"2024-03-18T20:23:41.007142+0000","last_became_active":"2024-03-18T20:23:39.768768+0000","last_became_peered":"2024-03-18T20:23:39.768768+0000","last_unstale":"2024-03-18T20:23:41.007142+0000","last_undegraded":"2024-03-18T20:23:41.007142+0000","last_fullsized":"2024-03-18T20:23:41.007142+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:30.491107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"213'1","reported_seq":12,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763319+0000","last_change":"2024-03-18T20:23:39.332256+0000","last_active":"2024-03-18T20:23:40.763319+0000","last_peered":"2024-03-18T20:23:40.763319+0000","last_clean":"2024-03-18T20:23:40.763319+0000","last_became_active":"2024-03-18T20:23:39.331959+0000","last_became_peered":"2024-03-18T20:23:39.331959+0000","last_unstale":"2024-03-18T20:23:40.763319+0000","last_undegraded":"2024-03-18T20:23:40.763319+0000","last_fullsized":"2024-03-18T20:23:40.763319+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:09:15.620959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.591020+0000","last_change":"2024-03-18T20:23:39.781943+0000","last_active":"2024-03-18T20:23:41.591020+0000","last_peered":"2024-03-18T20:23:41.591020+0000","last_clean":"2024-03-18T20:23:41.591020+0000","last_became_active":"2024-03-18T20:23:39.781660+0000","last_became_peered":"2024-03-18T20:23:39.781660+0000","last_unstale":"2024-03-18T20:23:41.591020+0000","last_undegraded":"2024-03-18T20:23:41.591020+0000","last_fullsized":"2024-03-18T20:23:41.591020+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:43:05.508887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.006967+0000","last_change":"2024-03-18T20:23:39.327794+0000","last_active":"2024-03-18T20:23:41.006967+0000","last_peered":"2024-03-18T20:23:41.006967+0000","last_clean":"2024-03-18T20:23:41.006967+0000","last_became_active":"2024-03-18T20:23:39.327508+0000","last_became_peered":"2024-03-18T20:23:39.327508+0000","last_unstale":"2024-03-18T20:23:41.006967+0000","last_undegraded":"2024-03-18T20:23:41.006967+0000","last_fullsized":"2024-03-18T20:23:41.006967+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:56:24.251796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590494+0000","last_change":"2024-03-18T20:23:39.774457+0000","last_active":"2024-03-18T20:23:41.590494+0000","last_peered":"2024-03-18T20:23:41.590494+0000","last_clean":"2024-03-18T20:23:41.590494+0000","last_became_active":"2024-03-18T20:23:39.774203+0000","last_became_peered":"2024-03-18T20:23:39.774203+0000","last_unstale":"2024-03-18T20:23:41.590494+0000","last_undegraded":"2024-03-18T20:23:41.590494+0000","last_fullsized":"2024-03-18T20:23:41.590494+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:20:52.271630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"213'2","reported_seq":13,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970910+0000","last_change":"2024-03-18T20:23:39.331954+0000","last_active":"2024-03-18T20:23:40.970910+0000","last_peered":"2024-03-18T20:23:40.970910+0000","last_clean":"2024-03-18T20:23:40.970910+0000","last_became_active":"2024-03-18T20:23:39.331741+0000","last_became_peered":"2024-03-18T20:23:39.331741+0000","last_unstale":"2024-03-18T20:23:40.970910+0000","last_undegraded":"2024-03-18T20:23:40.970910+0000","last_fullsized":"2024-03-18T20:23:40.970910+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:56:24.251796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.770845+0000","last_change":"2024-03-18T20:23:39.766629+0000","last_active":"2024-03-18T20:23:39.770845+0000","last_peered":"2024-03-18T20:23:39.770845+0000","last_clean":"2024-03-18T20:23:39.770845+0000","last_became_active":"2024-03-18T20:23:39.766415+0000","last_became_peered":"2024-03-18T20:23:39.766415+0000","last_unstale":"2024-03-18T20:23:39.770845+0000","last_undegraded":"2024-03-18T20:23:39.770845+0000","last_fullsized":"2024-03-18T20:23:39.770845+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:41:40.231425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.765952+0000","last_change":"2024-03-18T20:23:39.334672+0000","last_active":"2024-03-18T20:23:39.765952+0000","last_peered":"2024-03-18T20:23:39.765952+0000","last_clean":"2024-03-18T20:23:39.765952+0000","last_became_active":"2024-03-18T20:23:39.334148+0000","last_became_peered":"2024-03-18T20:23:39.334148+0000","last_unstale":"2024-03-18T20:23:39.765952+0000","last_undegraded":"2024-03-18T20:23:39.765952+0000","last_fullsized":"2024-03-18T20:23:39.765952+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:21:46.632822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.770841+0000","last_change":"2024-03-18T20:23:39.766505+0000","last_active":"2024-03-18T20:23:39.770841+0000","last_peered":"2024-03-18T20:23:39.770841+0000","last_clean":"2024-03-18T20:23:39.770841+0000","last_became_active":"2024-03-18T20:23:39.766223+0000","last_became_peered":"2024-03-18T20:23:39.766223+0000","last_unstale":"2024-03-18T20:23:39.770841+0000","last_undegraded":"2024-03-18T20:23:39.770841+0000","last_fullsized":"2024-03-18T20:23:39.770841+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:35.116397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763352+0000","last_change":"2024-03-18T20:23:39.332289+0000","last_active":"2024-03-18T20:23:40.763352+0000","last_peered":"2024-03-18T20:23:40.763352+0000","last_clean":"2024-03-18T20:23:40.763352+0000","last_became_active":"2024-03-18T20:23:39.332014+0000","last_became_peered":"2024-03-18T20:23:39.332014+0000","last_unstale":"2024-03-18T20:23:40.763352+0000","last_undegraded":"2024-03-18T20:23:40.763352+0000","last_fullsized":"2024-03-18T20:23:40.763352+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:45:21.252973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":9,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.780958+0000","last_change":"2024-03-18T20:23:39.776164+0000","last_active":"2024-03-18T20:23:39.780958+0000","last_peered":"2024-03-18T20:23:39.780958+0000","last_clean":"2024-03-18T20:23:39.780958+0000","last_became_active":"2024-03-18T20:23:39.775917+0000","last_became_peered":"2024-03-18T20:23:39.775917+0000","last_unstale":"2024-03-18T20:23:39.780958+0000","last_undegraded":"2024-03-18T20:23:39.780958+0000","last_fullsized":"2024-03-18T20:23:39.780958+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:59:24.493213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.762843+0000","last_change":"2024-03-18T20:23:39.324353+0000","last_active":"2024-03-18T20:23:40.762843+0000","last_peered":"2024-03-18T20:23:40.762843+0000","last_clean":"2024-03-18T20:23:40.762843+0000","last_became_active":"2024-03-18T20:23:39.324037+0000","last_became_peered":"2024-03-18T20:23:39.324037+0000","last_unstale":"2024-03-18T20:23:40.762843+0000","last_undegraded":"2024-03-18T20:23:40.762843+0000","last_fullsized":"2024-03-18T20:23:40.762843+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:32:02.472421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763748+0000","last_change":"2024-03-18T20:23:39.762706+0000","last_active":"2024-03-18T20:23:40.763748+0000","last_peered":"2024-03-18T20:23:40.763748+0000","last_clean":"2024-03-18T20:23:40.763748+0000","last_became_active":"2024-03-18T20:23:39.762461+0000","last_became_peered":"2024-03-18T20:23:39.762461+0000","last_unstale":"2024-03-18T20:23:40.763748+0000","last_undegraded":"2024-03-18T20:23:40.763748+0000","last_fullsized":"2024-03-18T20:23:40.763748+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:29:00.991763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"213'2","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.776518+0000","last_change":"2024-03-18T20:23:39.334632+0000","last_active":"2024-03-18T20:23:39.776518+0000","last_peered":"2024-03-18T20:23:39.776518+0000","last_clean":"2024-03-18T20:23:39.776518+0000","last_became_active":"2024-03-18T20:23:39.334345+0000","last_became_peered":"2024-03-18T20:23:39.334345+0000","last_unstale":"2024-03-18T20:23:39.776518+0000","last_undegraded":"2024-03-18T20:23:39.776518+0000","last_fullsized":"2024-03-18T20:23:39.776518+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:05:56.973792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590754+0000","last_change":"2024-03-18T20:23:39.762505+0000","last_active":"2024-03-18T20:23:41.590754+0000","last_peered":"2024-03-18T20:23:41.590754+0000","last_clean":"2024-03-18T20:23:41.590754+0000","last_became_active":"2024-03-18T20:23:39.762227+0000","last_became_peered":"2024-03-18T20:23:39.762227+0000","last_unstale":"2024-03-18T20:23:41.590754+0000","last_undegraded":"2024-03-18T20:23:41.590754+0000","last_fullsized":"2024-03-18T20:23:41.590754+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:57:49.050649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763604+0000","last_change":"2024-03-18T20:23:39.326895+0000","last_active":"2024-03-18T20:23:40.763604+0000","last_peered":"2024-03-18T20:23:40.763604+0000","last_clean":"2024-03-18T20:23:40.763604+0000","last_became_active":"2024-03-18T20:23:39.326605+0000","last_became_peered":"2024-03-18T20:23:39.326605+0000","last_unstale":"2024-03-18T20:23:40.763604+0000","last_undegraded":"2024-03-18T20:23:40.763604+0000","last_fullsized":"2024-03-18T20:23:40.763604+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:36:04.660134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:42.006839+0000","last_change":"2024-03-18T20:23:39.773684+0000","last_active":"2024-03-18T20:23:42.006839+0000","last_peered":"2024-03-18T20:23:42.006839+0000","last_clean":"2024-03-18T20:23:42.006839+0000","last_became_active":"2024-03-18T20:23:39.773341+0000","last_became_peered":"2024-03-18T20:23:39.773341+0000","last_unstale":"2024-03-18T20:23:42.006839+0000","last_undegraded":"2024-03-18T20:23:42.006839+0000","last_fullsized":"2024-03-18T20:23:42.006839+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:30.491107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.756377+0000","last_change":"2024-03-18T20:23:39.338114+0000","last_active":"2024-03-18T20:23:39.756377+0000","last_peered":"2024-03-18T20:23:39.756377+0000","last_clean":"2024-03-18T20:23:39.756377+0000","last_became_active":"2024-03-18T20:23:39.337837+0000","last_became_peered":"2024-03-18T20:23:39.337837+0000","last_unstale":"2024-03-18T20:23:39.756377+0000","last_undegraded":"2024-03-18T20:23:39.756377+0000","last_fullsized":"2024-03-18T20:23:39.756377+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:07:20.730496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763713+0000","last_change":"2024-03-18T20:23:39.763620+0000","last_active":"2024-03-18T20:23:40.763713+0000","last_peered":"2024-03-18T20:23:40.763713+0000","last_clean":"2024-03-18T20:23:40.763713+0000","last_became_active":"2024-03-18T20:23:39.763361+0000","last_became_peered":"2024-03-18T20:23:39.763361+0000","last_unstale":"2024-03-18T20:23:40.763713+0000","last_undegraded":"2024-03-18T20:23:40.763713+0000","last_fullsized":"2024-03-18T20:23:40.763713+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:56:25.257670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.756531+0000","last_change":"2024-03-18T20:23:39.329506+0000","last_active":"2024-03-18T20:23:39.756531+0000","last_peered":"2024-03-18T20:23:39.756531+0000","last_clean":"2024-03-18T20:23:39.756531+0000","last_became_active":"2024-03-18T20:23:39.329028+0000","last_became_peered":"2024-03-18T20:23:39.329028+0000","last_unstale":"2024-03-18T20:23:39.756531+0000","last_undegraded":"2024-03-18T20:23:39.756531+0000","last_fullsized":"2024-03-18T20:23:39.756531+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:49:53.935705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.762793+0000","last_change":"2024-03-18T20:23:39.765543+0000","last_active":"2024-03-18T20:23:40.762793+0000","last_peered":"2024-03-18T20:23:40.762793+0000","last_clean":"2024-03-18T20:23:40.762793+0000","last_became_active":"2024-03-18T20:23:39.764942+0000","last_became_peered":"2024-03-18T20:23:39.764942+0000","last_unstale":"2024-03-18T20:23:40.762793+0000","last_undegraded":"2024-03-18T20:23:40.762793+0000","last_fullsized":"2024-03-18T20:23:40.762793+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:53:15.836495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:42.006583+0000","last_change":"2024-03-18T20:23:39.333831+0000","last_active":"2024-03-18T20:23:42.006583+0000","last_peered":"2024-03-18T20:23:42.006583+0000","last_clean":"2024-03-18T20:23:42.006583+0000","last_became_active":"2024-03-18T20:23:39.333541+0000","last_became_peered":"2024-03-18T20:23:39.333541+0000","last_unstale":"2024-03-18T20:23:42.006583+0000","last_undegraded":"2024-03-18T20:23:42.006583+0000","last_fullsized":"2024-03-18T20:23:42.006583+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:55.688191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.970755+0000","last_change":"2024-03-18T20:23:39.762015+0000","last_active":"2024-03-18T20:23:40.970755+0000","last_peered":"2024-03-18T20:23:40.970755+0000","last_clean":"2024-03-18T20:23:40.970755+0000","last_became_active":"2024-03-18T20:23:39.761733+0000","last_became_peered":"2024-03-18T20:23:39.761733+0000","last_unstale":"2024-03-18T20:23:40.970755+0000","last_undegraded":"2024-03-18T20:23:40.970755+0000","last_fullsized":"2024-03-18T20:23:40.970755+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:35.217366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:42.006715+0000","last_change":"2024-03-18T20:23:39.330955+0000","last_active":"2024-03-18T20:23:42.006715+0000","last_peered":"2024-03-18T20:23:42.006715+0000","last_clean":"2024-03-18T20:23:42.006715+0000","last_became_active":"2024-03-18T20:23:39.330635+0000","last_became_peered":"2024-03-18T20:23:39.330635+0000","last_unstale":"2024-03-18T20:23:42.006715+0000","last_undegraded":"2024-03-18T20:23:42.006715+0000","last_fullsized":"2024-03-18T20:23:42.006715+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:39.419825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:41.590979+0000","last_change":"2024-03-18T20:23:39.769949+0000","last_active":"2024-03-18T20:23:41.590979+0000","last_peered":"2024-03-18T20:23:41.590979+0000","last_clean":"2024-03-18T20:23:41.590979+0000","last_became_active":"2024-03-18T20:23:39.769687+0000","last_became_peered":"2024-03-18T20:23:39.769687+0000","last_unstale":"2024-03-18T20:23:41.590979+0000","last_undegraded":"2024-03-18T20:23:41.590979+0000","last_fullsized":"2024-03-18T20:23:41.590979+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:04:55.872080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763517+0000","last_change":"2024-03-18T20:23:39.323965+0000","last_active":"2024-03-18T20:23:40.763517+0000","last_peered":"2024-03-18T20:23:40.763517+0000","last_clean":"2024-03-18T20:23:40.763517+0000","last_became_active":"2024-03-18T20:23:39.323731+0000","last_became_peered":"2024-03-18T20:23:39.323731+0000","last_unstale":"2024-03-18T20:23:40.763517+0000","last_undegraded":"2024-03-18T20:23:40.763517+0000","last_fullsized":"2024-03-18T20:23:40.763517+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:57:50.281821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:42.006517+0000","last_change":"2024-03-18T20:23:39.771006+0000","last_active":"2024-03-18T20:23:42.006517+0000","last_peered":"2024-03-18T20:23:42.006517+0000","last_clean":"2024-03-18T20:23:42.006517+0000","last_became_active":"2024-03-18T20:23:39.770741+0000","last_became_peered":"2024-03-18T20:23:39.770741+0000","last_unstale":"2024-03-18T20:23:42.006517+0000","last_undegraded":"2024-03-18T20:23:42.006517+0000","last_fullsized":"2024-03-18T20:23:42.006517+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:00.540282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":11,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.762903+0000","last_change":"2024-03-18T20:23:39.327798+0000","last_active":"2024-03-18T20:23:40.762903+0000","last_peered":"2024-03-18T20:23:40.762903+0000","last_clean":"2024-03-18T20:23:40.762903+0000","last_became_active":"2024-03-18T20:23:39.327543+0000","last_became_peered":"2024-03-18T20:23:39.327543+0000","last_unstale":"2024-03-18T20:23:40.762903+0000","last_undegraded":"2024-03-18T20:23:40.762903+0000","last_fullsized":"2024-03-18T20:23:40.762903+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:41.616808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":10,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-03-18T20:23:40.763041+0000","last_change":"2024-03-18T20:23:39.776313+0000","last_active":"2024-03-18T20:23:40.763041+0000","last_peered":"2024-03-18T20:23:40.763041+0000","last_clean":"2024-03-18T20:23:40.763041+0000","last_became_active":"2024-03-18T20:23:39.776057+0000","last_became_peered":"2024-03-18T20:23:39.776057+0000","last_unstale":"2024-03-18T20:23:40.763041+0000","last_undegraded":"2024-03-18T20:23:40.763041+0000","last_fullsized":"2024-03-18T20:23:40.763041+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:39.310311+0000","last_clean_scrub_stamp":"2024-03-18T20:23:39.310311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:59:57.901328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":10,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.759917+0000","last_change":"2024-03-18T20:23:39.330747+0000","last_active":"2024-03-18T20:23:39.759917+0000","last_peered":"2024-03-18T20:23:39.759917+0000","last_clean":"2024-03-18T20:23:39.759917+0000","last_became_active":"2024-03-18T20:23:39.330391+0000","last_became_peered":"2024-03-18T20:23:39.330391+0000","last_unstale":"2024-03-18T20:23:39.759917+0000","last_undegraded":"2024-03-18T20:23:39.759917+0000","last_fullsized":"2024-03-18T20:23:39.759917+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:23:38.304437+0000","last_clean_scrub_stamp":"2024-03-18T20:23:38.304437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:52.293500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"203'326","reported_seq":1107,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-18T20:23:39.762106+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:23:39.762106+0000","last_peered":"2024-03-18T20:23:39.762106+0000","last_clean":"2024-03-18T20:23:39.762106+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:23:39.762106+0000","last_undegraded":"2024-03-18T20:23:39.762106+0000","last_fullsized":"2024-03-18T20:23:39.762106+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575962,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52364,"kb_used_data":3188,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319476,"statfs":{"total":96636764160,"available":96583143424,"internally_reserved":0,"allocated":3264512,"data_stored":1954669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.622,"15min":0.60099999999999998},"min":{"1min":0.40400000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.75800000000000001,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61399999999999999,"15min":0.61099999999999999},"min":{"1min":0.42699999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.84499999999999997,"5min":1.093,"15min":1.093},"last":0.75}]},{"osd":1,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.63600000000000001,"15min":0.61299999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.35999999999999999},"max":{"1min":0.83099999999999996,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.629,"15min":0.61799999999999999},"min":{"1min":0.40300000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78600000000000003,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.66600000000000004}]},{"osd":2,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.63,"15min":0.61299999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.77900000000000003,"5min":1.216,"15min":1.216},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.63200000000000001,"15min":0.623},"min":{"1min":0.48499999999999999,"5min":0.41699999999999998,"15min":0.39000000000000001},"max":{"1min":0.73199999999999998,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.52500000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.63,"15min":0.61499999999999999},"min":{"1min":0.47199999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.69799999999999995,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.495},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66700000000000004,"15min":0.64300000000000002},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.38600000000000001},"max":{"1min":0.79200000000000004,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.59899999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47799999999999998,"15min":0.47699999999999998},"min":{"1min":0.32600000000000001,"5min":0.316,"15min":0.28899999999999998},"max":{"1min":0.67400000000000004,"5min":0.78700000000000003,"15min":0.86099999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.497,"15min":0.502},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.63,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.76600000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52300000000000002,"15min":0.51500000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.316},"max":{"1min":0.65500000000000003,"5min":0.86599999999999999,"15min":0.89400000000000002},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.54600000000000004,"15min":0.55100000000000005},"min":{"1min":0.377,"5min":0.36899999999999999,"15min":0.34699999999999998},"max":{"1min":0.625,"5min":0.76500000000000001,"15min":1.042},"last":0.81599999999999995}]},{"osd":6,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.36599999999999999,"5min":0.36199999999999999,"15min":0.34999999999999998},"max":{"1min":0.752,"5min":0.85599999999999998,"15min":1.129},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.34000000000000002},"max":{"1min":0.68500000000000005,"5min":0.85299999999999998,"15min":1.0609999999999999},"last":0.71599999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575962,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41932,"kb_used_data":2624,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329908,"statfs":{"total":96636764160,"available":96593825792,"internally_reserved":0,"allocated":2686976,"data_stored":1371091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.437,"5min":0.38200000000000001,"15min":0.375},"max":{"1min":0.73399999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61099999999999999,"15min":0.61799999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.318},"max":{"1min":0.72199999999999998,"5min":0.81499999999999995,"15min":0.83299999999999996},"last":0.54300000000000004}]},{"osd":1,"last update":"Mon Mar 18 20:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.60699999999999998,"15min":0.60899999999999999},"min":{"1min":0.45300000000000001,"5min":0.36499999999999999,"15min":0.35799999999999998},"max":{"1min":0.72099999999999997,"5min":1.244,"15min":1.244},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61899999999999999,"15min":0.61199999999999999},"min":{"1min":0.45000000000000001,"5min":0.41799999999999998,"15min":0.33200000000000002},"max":{"1min":0.755,"5min":1.286,"15min":1.286},"last":0.55500000000000005}]},{"osd":2,"last update":"Mon Mar 18 20:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.64500000000000002,"15min":0.63800000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77000000000000002,"5min":1.276,"15min":1.276},"last":0.501},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.61699999999999999,"15min":0.61599999999999999},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.374},"max":{"1min":0.67100000000000004,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.58899999999999997}]},{"osd":3,"last update":"Mon Mar 18 20:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.63,"15min":0.63},"min":{"1min":0.41399999999999998,"5min":0.39000000000000001,"15min":0.32700000000000001},"max":{"1min":0.66800000000000004,"5min":0.85399999999999998,"15min":1.02},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.63500000000000001,"15min":0.623},"min":{"1min":0.46800000000000003,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.73699999999999999,"5min":1.337,"15min":1.337},"last":0.57399999999999995}]},{"osd":4,"last update":"Mon Mar 18 20:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.48699999999999999,"15min":0.48499999999999999},"min":{"1min":0.34599999999999997,"5min":0.32600000000000001,"15min":0.312},"max":{"1min":0.80100000000000005,"5min":0.86499999999999999,"15min":0.88300000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.52200000000000002,"15min":0.51200000000000001},"min":{"1min":0.40400000000000003,"5min":0.34599999999999997,"15min":0.32600000000000001},"max":{"1min":0.69299999999999995,"5min":0.94899999999999995,"15min":0.95499999999999996},"last":0.35299999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.505,"15min":0.50700000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.31},"max":{"1min":0.76000000000000001,"5min":0.82699999999999996,"15min":0.84799999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.39300000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.94199999999999995},"last":0.40699999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56299999999999994,"15min":0.55500000000000005},"min":{"1min":0.38,"5min":0.35799999999999998,"15min":0.34499999999999997},"max":{"1min":0.73599999999999999,"5min":0.95599999999999996,"15min":1.0429999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.34599999999999997},"max":{"1min":0.79000000000000004,"5min":1.028,"15min":1.028},"last":0.438}]}]},{"osd":1,"up_from":13,"seq":55834575962,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47680,"kb_used_data":2608,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324160,"statfs":{"total":96636764160,"available":96587939840,"internally_reserved":0,"allocated":2670592,"data_stored":1364301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.40699999999999997,"15min":0.39600000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.23999999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.501},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.42099999999999999,"15min":0.42999999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.59899999999999998,"5min":0.69699999999999995,"15min":0.72299999999999998},"last":0.54000000000000004}]},{"osd":2,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.438,"15min":0.435},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.58199999999999996,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.47999999999999998,"15min":0.47599999999999998},"min":{"1min":0.32100000000000001,"5min":0.313,"15min":0.29899999999999999},"max":{"1min":0.65300000000000002,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.70099999999999996}]},{"osd":3,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.495,"15min":0.49299999999999999},"min":{"1min":0.33000000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.62,"5min":0.75900000000000001,"15min":0.76100000000000001},"last":0.625},{"interface":"front","average":{"1min":0.437,"5min":0.443,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.59199999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.68200000000000005}]},{"osd":4,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.69799999999999995,"15min":0.67500000000000004},"min":{"1min":0.498,"5min":0.496,"15min":0.44700000000000001},"max":{"1min":1.0680000000000001,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67500000000000004,"15min":0.68200000000000005},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":1.746},"last":0.84499999999999997}]},{"osd":5,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.68500000000000005,"15min":0.67700000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.42099999999999999},"max":{"1min":0.90300000000000002,"5min":0.96099999999999997,"15min":1.6910000000000001},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73599999999999999,"15min":0.71599999999999997},"min":{"1min":0.53400000000000003,"5min":0.47399999999999998,"15min":0.46899999999999997},"max":{"1min":1.109,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.89400000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.69899999999999995,"15min":0.68899999999999995},"min":{"1min":0.54700000000000004,"5min":0.437,"15min":0.437},"max":{"1min":0.872,"5min":0.98899999999999999,"15min":1.637},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.71999999999999997,"15min":0.71399999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.40899999999999997},"max":{"1min":1.014,"5min":1.014,"15min":1.722},"last":0.82899999999999996}]},{"osd":7,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.72099999999999997,"15min":0.71499999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.44600000000000001},"max":{"1min":0.99199999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71899999999999997,"15min":0.71099999999999997},"min":{"1min":0.495,"5min":0.495,"15min":0.42799999999999999},"max":{"1min":1.0329999999999999,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.88100000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575963,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46608,"kb_used_data":3204,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94325232,"statfs":{"total":96636764160,"available":96589037568,"internally_reserved":0,"allocated":3280896,"data_stored":1955397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44431814},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.40000000000000002,"15min":0.39600000000000002},"min":{"1min":0.25600000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.373},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.27200000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.59299999999999997,"5min":0.64700000000000002,"15min":1.1850000000000001},"last":0.34000000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.435,"15min":0.439},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.317},{"interface":"front","average":{"1min":0.436,"5min":0.44900000000000001,"15min":0.45700000000000002},"min":{"1min":0.311,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.54400000000000004,"5min":0.64300000000000002,"15min":0.70999999999999996},"last":0.29899999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.434,"15min":0.435},"min":{"1min":0.317,"5min":0.28100000000000003,"15min":0.26100000000000001},"max":{"1min":0.56100000000000005,"5min":0.61299999999999999,"15min":0.628},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.496,"15min":0.5},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.63800000000000001,"5min":0.73999999999999999,"15min":0.91500000000000004},"last":0.40699999999999997}]},{"osd":4,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65500000000000003,"15min":0.66100000000000003},"min":{"1min":0.47499999999999998,"5min":0.45400000000000001,"15min":0.42999999999999999},"max":{"1min":0.79800000000000004,"5min":0.96599999999999997,"15min":1.0029999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.67400000000000004,"15min":0.66600000000000004},"min":{"1min":0.48299999999999998,"5min":0.47099999999999997,"15min":0.39700000000000002},"max":{"1min":0.88300000000000001,"5min":0.99199999999999999,"15min":1.0469999999999999},"last":0.55100000000000005}]},{"osd":5,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.68400000000000005,"15min":0.67400000000000004},"min":{"1min":0.51300000000000001,"5min":0.439,"15min":0.439},"max":{"1min":0.89300000000000002,"5min":0.97099999999999997,"15min":1.0129999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.68700000000000006},"min":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.42399999999999999},"max":{"1min":0.89400000000000002,"5min":0.92900000000000005,"15min":1.518},"last":0.442}]},{"osd":6,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67200000000000004,"15min":0.67000000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45000000000000001},"max":{"1min":0.90700000000000003,"5min":0.97399999999999998,"15min":1.1279999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.71299999999999997,"15min":0.71199999999999997},"min":{"1min":0.48999999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.85399999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.60399999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.71199999999999997,"15min":0.70899999999999996},"min":{"1min":0.51900000000000002,"5min":0.46700000000000003,"15min":0.44900000000000001},"max":{"1min":0.84599999999999997,"5min":0.99099999999999999,"15min":1.464},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71799999999999997,"15min":0.71099999999999997},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.47599999999999998},"max":{"1min":0.85299999999999998,"5min":1.042,"15min":1.2350000000000001},"last":0.51900000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575963,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47692,"kb_used_data":2624,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324148,"statfs":{"total":96636764160,"available":96587927552,"internally_reserved":0,"allocated":2686976,"data_stored":1371603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.38300000000000001,"15min":0.38100000000000001},"min":{"1min":0.26400000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.44900000000000001,"5min":0.55300000000000005,"15min":0.70799999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.495,"15min":0.48999999999999999},"min":{"1min":0.32700000000000001,"5min":0.313,"15min":0.24199999999999999},"max":{"1min":0.63800000000000001,"5min":0.72899999999999998,"15min":0.83399999999999996},"last":0.47599999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.48799999999999999,"15min":0.48199999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.30299999999999999},"max":{"1min":0.58599999999999997,"5min":0.749,"15min":0.77100000000000002},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.42399999999999999,"15min":0.41999999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27600000000000002},"max":{"1min":0.55000000000000004,"5min":0.59299999999999997,"15min":0.71099999999999997},"last":0.52000000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.443,"15min":0.438},"min":{"1min":0.34699999999999998,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":0.57299999999999995,"5min":0.64700000000000002,"15min":0.66300000000000003},"last":0.501},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.313},"max":{"1min":0.56699999999999995,"5min":0.752,"15min":0.752},"last":0.57399999999999995}]},{"osd":4,"last update":"Mon Mar 18 20:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.66800000000000004,"15min":0.66100000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":0.80900000000000005,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.753},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.68400000000000005,"15min":0.66900000000000004},"min":{"1min":0.46700000000000003,"5min":0.443,"15min":0.443},"max":{"1min":0.75,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.79800000000000004}]},{"osd":5,"last update":"Mon Mar 18 20:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.68000000000000005,"15min":0.68500000000000005},"min":{"1min":0.48399999999999999,"5min":0.48199999999999998,"15min":0.45300000000000001},"max":{"1min":0.81000000000000005,"5min":1.028,"15min":1.028},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.624,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.44600000000000001},"max":{"1min":0.83399999999999996,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.88200000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.69499999999999995,"15min":0.69399999999999995},"min":{"1min":0.55000000000000004,"5min":0.48899999999999999,"15min":0.46700000000000003},"max":{"1min":0.80100000000000005,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.73999999999999999,"15min":0.73399999999999999},"min":{"1min":0.54900000000000004,"5min":0.51900000000000002,"15min":0.51000000000000001},"max":{"1min":0.83199999999999996,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.90200000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.69999999999999996,"15min":0.70399999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.42199999999999999},"max":{"1min":0.80400000000000005,"5min":0.99199999999999999,"15min":1.0469999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.71399999999999997,"15min":0.71099999999999997},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.49099999999999999},"max":{"1min":0.81499999999999995,"5min":1.371,"15min":1.371},"last":0.92400000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51788,"kb_used_data":2616,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320052,"statfs":{"total":96636764160,"available":96583733248,"internally_reserved":0,"allocated":2678784,"data_stored":1364461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:23:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.27800000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54900000000000004,"5min":0.65500000000000003,"15min":0.98099999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42199999999999999,"15min":0.42099999999999999},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.24199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.78800000000000003},"last":0.40100000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:23:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40799999999999997,"15min":0.41399999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.254},"max":{"1min":0.52700000000000002,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.375},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48199999999999998,"15min":0.47199999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.29099999999999998},"max":{"1min":0.65500000000000003,"5min":0.72599999999999998,"15min":0.73099999999999998},"last":0.435}]},{"osd":2,"last update":"Mon Mar 18 20:23:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47399999999999998,"15min":0.47099999999999997},"min":{"1min":0.36899999999999999,"5min":0.33100000000000002,"15min":0.32900000000000001},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.502,"15min":0.504},"min":{"1min":0.39900000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64400000000000002,"5min":0.70099999999999996,"15min":0.73099999999999998},"last":0.51700000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:23:09 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.66000000000000003,"15min":0.65600000000000003},"min":{"1min":0.45900000000000002,"5min":0.436,"15min":0.42499999999999999},"max":{"1min":0.68999999999999995,"5min":0.91700000000000004,"15min":1.097},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.68999999999999995,"15min":0.67900000000000005},"min":{"1min":0.53100000000000003,"5min":0.46899999999999997,"15min":0.42899999999999999},"max":{"1min":0.86799999999999999,"5min":0.99299999999999999,"15min":1.0569999999999999},"last":0.79100000000000004}]},{"osd":5,"last update":"Mon Mar 18 20:23:09 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.70499999999999996,"15min":0.69899999999999995},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.441},"max":{"1min":0.91800000000000004,"5min":1.0489999999999999,"15min":1.1599999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.748,"15min":0.73399999999999999},"min":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.46700000000000003},"max":{"1min":0.90400000000000003,"5min":1.171,"15min":1.171},"last":0.81200000000000006}]},{"osd":6,"last update":"Mon Mar 18 20:23:09 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.70699999999999996,"15min":0.71599999999999997},"min":{"1min":0.56599999999999995,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81100000000000005,"5min":1.012,"15min":1.194},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.45100000000000001},"max":{"1min":0.83999999999999997,"5min":1.0409999999999999,"15min":1.2410000000000001},"last":0.88700000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:23:09 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72899999999999998,"15min":0.72699999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85299999999999998,"5min":1.079,"15min":1.079},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.71699999999999997,"15min":0.71099999999999997},"min":{"1min":0.48799999999999999,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.78200000000000003,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.752}]}]},{"osd":4,"up_from":13,"seq":55834575963,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60000,"kb_used_data":2636,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311840,"statfs":{"total":96636764160,"available":96575324160,"internally_reserved":0,"allocated":2699264,"data_stored":1372619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.59299999999999997,"15min":0.60399999999999998},"min":{"1min":0.40999999999999998,"5min":0.36899999999999999,"15min":0.34300000000000003},"max":{"1min":0.75600000000000001,"5min":0.76100000000000001,"15min":0.81799999999999995},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60199999999999998,"15min":0.60899999999999999},"min":{"1min":0.44500000000000001,"5min":0.40699999999999997,"15min":0.371},"max":{"1min":0.75,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.66400000000000003}]},{"osd":1,"last update":"Mon Mar 18 20:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.61699999999999999,"15min":0.622},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70299999999999996,"5min":0.81899999999999995,"15min":0.88300000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58899999999999997,"15min":0.59399999999999997},"min":{"1min":0.47599999999999998,"5min":0.38900000000000001,"15min":0.38400000000000001},"max":{"1min":0.76800000000000002,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.72499999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59299999999999997,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42599999999999999,"15min":0.39200000000000002},"max":{"1min":0.76000000000000001,"5min":0.80500000000000005,"15min":0.82599999999999996},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.622,"15min":0.627},"min":{"1min":0.44800000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78000000000000003,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.70599999999999996}]},{"osd":3,"last update":"Mon Mar 18 20:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.60699999999999998,"15min":0.60899999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.32900000000000001},"max":{"1min":0.72499999999999998,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.60599999999999998,"15min":0.627},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73499999999999999,"5min":0.79100000000000004,"15min":0.86199999999999999},"last":0.64700000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45600000000000002,"15min":0.47699999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.315},"max":{"1min":0.64600000000000002,"5min":0.70299999999999996,"15min":0.76000000000000001},"last":0.442},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48699999999999999,"15min":0.503},"min":{"1min":0.36399999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.64800000000000002,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.47499999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48199999999999998,"15min":0.503},"min":{"1min":0.375,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68200000000000005,"5min":0.71399999999999997,"15min":0.77100000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53200000000000003,"15min":0.54800000000000004},"min":{"1min":0.41499999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.754,"5min":0.76200000000000001,"15min":0.80900000000000005},"last":0.49199999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52400000000000002,"15min":0.54300000000000004},"min":{"1min":0.433,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64300000000000002,"5min":0.87,"15min":0.87},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55800000000000005,"15min":0.57999999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.78200000000000003,"5min":0.81200000000000006,"15min":0.85399999999999998},"last":0.46000000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575962,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47704,"kb_used_data":2636,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324136,"statfs":{"total":96636764160,"available":96587915264,"internally_reserved":0,"allocated":2699264,"data_stored":1372607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.61799999999999999,"15min":0.60899999999999999},"min":{"1min":0.438,"5min":0.35999999999999999,"15min":0.31900000000000001},"max":{"1min":0.81100000000000005,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.62,"15min":0.60799999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.33100000000000002},"max":{"1min":0.82599999999999996,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.58199999999999996}]},{"osd":1,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.629,"15min":0.60699999999999998},"min":{"1min":0.437,"5min":0.373,"15min":0.373},"max":{"1min":0.77700000000000002,"5min":1.056,"15min":1.056},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61699999999999999,"15min":0.60599999999999998},"min":{"1min":0.48099999999999998,"5min":0.436,"15min":0.36899999999999999},"max":{"1min":0.77700000000000002,"5min":1.018,"15min":1.018},"last":0.46800000000000003}]},{"osd":2,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61599999999999999,"15min":0.60799999999999998},"min":{"1min":0.45500000000000002,"5min":0.38900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80200000000000005,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.64000000000000001,"15min":0.61399999999999999},"min":{"1min":0.48999999999999999,"5min":0.44500000000000001,"15min":0.434},"max":{"1min":0.74099999999999999,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.496}]},{"osd":3,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.63500000000000001,"15min":0.61599999999999999},"min":{"1min":0.48599999999999999,"5min":0.43099999999999999,"15min":0.40799999999999997},"max":{"1min":0.754,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.66700000000000004,"15min":0.65200000000000002},"min":{"1min":0.47499999999999998,"5min":0.46999999999999997,"15min":0.46100000000000002},"max":{"1min":0.78500000000000003,"5min":1.161,"15min":1.161},"last":0.69499999999999995}]},{"osd":4,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.47999999999999998},"min":{"1min":0.41799999999999998,"5min":0.34100000000000003,"15min":0.32600000000000001},"max":{"1min":0.68799999999999994,"5min":0.74099999999999999,"15min":0.77500000000000002},"last":0.627},{"interface":"front","average":{"1min":0.502,"5min":0.51300000000000001,"15min":0.51000000000000001},"min":{"1min":0.33000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.78900000000000003,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.60799999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.34999999999999998,"5min":0.32800000000000001,"15min":0.32100000000000001},"max":{"1min":0.67200000000000004,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.53800000000000003,"15min":0.53700000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":0.63800000000000001,"5min":0.82399999999999995,"15min":0.86599999999999999},"last":0.67900000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.36899999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76300000000000001,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58299999999999996,"15min":0.57499999999999996},"min":{"1min":0.38,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73099999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.749}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:23:44.320 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T20:23:44.321 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-18T20:23:44.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.483+0000 7f0ba933b700 1 Processor -- start 2024-03-18T20:23:44.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.483+0000 7f0ba933b700 1 -- start start 2024-03-18T20:23:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.484+0000 7f0ba933b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0ba406ca80 0x7f0ba415d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:44.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.484+0000 7f0ba933b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ba409a150 0x7f0ba40cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.484+0000 7f0ba933b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba40b3b00 con 0x7f0ba415d5b0 2024-03-18T20:23:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.484+0000 7f0ba933b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ba40cbbb0 con 0x7f0ba406ca80 2024-03-18T20:23:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.484+0000 7f0ba933b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba40cf7f0 con 0x7f0ba409a150 2024-03-18T20:23:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.485+0000 7f0ba27fc700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0ba406ca80 0x7f0ba415d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.485+0000 7f0ba37fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ba409a150 0x7f0ba40cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.485+0000 7f0ba2ffd700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f0ba415d5b0 0x7f0ba4099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:38756/0 (socket says 172.21.15.1:38756) 2024-03-18T20:23:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.485+0000 7f0ba2ffd700 1 -- 172.21.15.1:0/3160864132 learned_addr learned my addr 172.21.15.1:0/3160864132 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.485+0000 7f0ba1ffb700 1 -- 172.21.15.1:0/3160864132 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408062300 0 0) 0x7f0ba40b3b00 con 0x7f0ba415d5b0 2024-03-18T20:23:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.485+0000 7f0ba1ffb700 1 -- 172.21.15.1:0/3160864132 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b8c0035b0 con 0x7f0ba415d5b0 2024-03-18T20:23:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.485+0000 7f0ba27fc700 1 -- 172.21.15.1:0/3160864132 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ba409a150 msgr2=0x7f0ba40cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.485+0000 7f0ba27fc700 1 --2- 172.21.15.1:0/3160864132 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ba409a150 0x7f0ba40cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.485+0000 7f0ba27fc700 1 -- 172.21.15.1:0/3160864132 >> v1:172.21.15.165:6789/0 conn(0x7f0ba415d5b0 legacy=0x7f0ba4099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:44.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.485+0000 7f0ba27fc700 1 -- 172.21.15.1:0/3160864132 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba40d04d0 con 0x7f0ba406ca80 2024-03-18T20:23:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.485+0000 7f0ba37fe700 1 --2- 172.21.15.1:0/3160864132 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ba409a150 0x7f0ba40cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:23:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.485+0000 7f0ba27fc700 1 --2- 172.21.15.1:0/3160864132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba406ca80 0x7f0ba415d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f0b94009840 tx=0x7f0b94009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b69619b3969dbc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.485+0000 7f0ba1ffb700 1 -- 172.21.15.1:0/3160864132 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b94011830 con 0x7f0ba406ca80 2024-03-18T20:23:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.485+0000 7f0ba1ffb700 1 -- 172.21.15.1:0/3160864132 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b94011990 con 0x7f0ba406ca80 2024-03-18T20:23:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.485+0000 7f0ba1ffb700 1 -- 172.21.15.1:0/3160864132 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b94011d20 con 0x7f0ba406ca80 2024-03-18T20:23:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.486+0000 7f0ba933b700 1 -- 172.21.15.1:0/3160864132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba406ca80 msgr2=0x7f0ba415d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.486+0000 7f0ba933b700 1 --2- 172.21.15.1:0/3160864132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba406ca80 0x7f0ba415d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f0b94009840 tx=0x7f0b94009b50 comp rx=0 tx=0).stop 2024-03-18T20:23:44.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.486+0000 7f0ba933b700 1 -- 172.21.15.1:0/3160864132 shutdown_connections 2024-03-18T20:23:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.486+0000 7f0ba933b700 1 --2- 172.21.15.1:0/3160864132 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ba409a150 0x7f0ba40cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.486+0000 7f0ba933b700 1 --2- 172.21.15.1:0/3160864132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba406ca80 0x7f0ba415d990 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.486+0000 7f0ba933b700 1 -- 172.21.15.1:0/3160864132 >> 172.21.15.1:0/3160864132 conn(0x7f0ba40b2bd0 msgr2=0x7f0ba40b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.486+0000 7f0ba933b700 1 -- 172.21.15.1:0/3160864132 shutdown_connections 2024-03-18T20:23:44.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.486+0000 7f0ba933b700 1 -- 172.21.15.1:0/3160864132 wait complete. 2024-03-18T20:23:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.487+0000 7f0ba933b700 1 Processor -- start 2024-03-18T20:23:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.487+0000 7f0ba933b700 1 -- start start 2024-03-18T20:23:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.488+0000 7f0ba933b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba406ca80 0x7f0ba40b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.488+0000 7f0ba933b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ba409a150 0x7f0ba4149260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.488+0000 7f0ba933b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ba415d5b0 0x7f0ba40b6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.488+0000 7f0ba933b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba40b4cb0 con 0x7f0ba415d5b0 2024-03-18T20:23:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.488+0000 7f0ba933b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba40b40a0 con 0x7f0ba406ca80 2024-03-18T20:23:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.488+0000 7f0ba933b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba40b4380 con 0x7f0ba409a150 2024-03-18T20:23:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.488+0000 7f0ba37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ba415d5b0 0x7f0ba40b6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:44.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.488+0000 7f0ba37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ba415d5b0 0x7f0ba40b6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57808/0 (socket says 172.21.15.1:57808) 2024-03-18T20:23:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.488+0000 7f0ba2ffd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba406ca80 0x7f0ba40b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.488+0000 7f0ba37fe700 1 -- 172.21.15.1:0/3452655641 learned_addr learned my addr 172.21.15.1:0/3452655641 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.488+0000 7f0ba27fc700 1 --2- 172.21.15.1:0/3452655641 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ba409a150 0x7f0ba4149260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.489+0000 7f0ba37fe700 1 -- 172.21.15.1:0/3452655641 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ba409a150 msgr2=0x7f0ba4149260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.489+0000 7f0ba37fe700 1 --2- 172.21.15.1:0/3452655641 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ba409a150 0x7f0ba4149260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.489+0000 7f0ba37fe700 1 -- 172.21.15.1:0/3452655641 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba406ca80 msgr2=0x7f0ba40b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.489+0000 7f0ba37fe700 1 --2- 172.21.15.1:0/3452655641 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba406ca80 0x7f0ba40b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.489+0000 7f0ba37fe700 1 -- 172.21.15.1:0/3452655641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c009a70 con 0x7f0ba415d5b0 2024-03-18T20:23:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.489+0000 7f0ba2ffd700 1 --2- 172.21.15.1:0/3452655641 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba406ca80 0x7f0ba40b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:23:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.489+0000 7f0ba37fe700 1 --2- 172.21.15.1:0/3452655641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ba415d5b0 0x7f0ba40b6280 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c00efc0 tx=0x7f0b9c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.489+0000 7f0b8b7fe700 1 -- 172.21.15.1:0/3452655641 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c00ca90 con 0x7f0ba415d5b0 2024-03-18T20:23:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.489+0000 7f0b8b7fe700 1 -- 172.21.15.1:0/3452655641 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b9c00f040 con 0x7f0ba415d5b0 2024-03-18T20:23:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.489+0000 7f0ba933b700 1 -- 172.21.15.1:0/3452655641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b940094a0 con 0x7f0ba415d5b0 2024-03-18T20:23:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.489+0000 7f0b8b7fe700 1 -- 172.21.15.1:0/3452655641 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c00ce30 con 0x7f0ba415d5b0 2024-03-18T20:23:44.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.489+0000 7f0ba933b700 1 -- 172.21.15.1:0/3452655641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba40b4940 con 0x7f0ba415d5b0 2024-03-18T20:23:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.490+0000 7f0ba933b700 1 -- 172.21.15.1:0/3452655641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba40b4e20 con 0x7f0ba415d5b0 2024-03-18T20:23:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.491+0000 7f0b8b7fe700 1 -- 172.21.15.1:0/3452655641 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0b9c014480 con 0x7f0ba415d5b0 2024-03-18T20:23:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.491+0000 7f0b8b7fe700 1 --2- 172.21.15.1:0/3452655641 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b9006adb0 0x7f0b9006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.491+0000 7f0b8b7fe700 1 -- 172.21.15.1:0/3452655641 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f0b9c089ee0 con 0x7f0ba415d5b0 2024-03-18T20:23:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.491+0000 7f0ba2ffd700 1 --2- 172.21.15.1:0/3452655641 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b9006adb0 0x7f0b9006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:44.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.491+0000 7f0ba2ffd700 1 --2- 172.21.15.1:0/3452655641 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b9006adb0 0x7f0b9006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0b98007fd0 tx=0x7f0b98007dc0 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:44.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.496+0000 7f0b8b7fe700 1 -- 172.21.15.1:0/3452655641 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0b9c054c60 con 0x7f0ba415d5b0 2024-03-18T20:23:44.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.789+0000 7f0ba933b700 1 -- 172.21.15.1:0/3452655641 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f0ba40676a0 con 0x7f0ba415d5b0 2024-03-18T20:23:44.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.792+0000 7f0b8b7fe700 1 -- 172.21.15.1:0/3452655641 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v49) v1 ==== 190+0+0 (secure 0 0 0) 0x7f0b9c0180e0 con 0x7f0ba415d5b0 2024-03-18T20:23:44.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.795+0000 7f0ba933b700 1 -- 172.21.15.1:0/3452655641 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b9006adb0 msgr2=0x7f0b9006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:44.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.795+0000 7f0ba933b700 1 --2- 172.21.15.1:0/3452655641 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b9006adb0 0x7f0b9006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0b98007fd0 tx=0x7f0b98007dc0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.795+0000 7f0ba933b700 1 -- 172.21.15.1:0/3452655641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ba415d5b0 msgr2=0x7f0ba40b6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:44.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.795+0000 7f0ba933b700 1 --2- 172.21.15.1:0/3452655641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ba415d5b0 0x7f0ba40b6280 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c00efc0 tx=0x7f0b9c00c3a0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.795+0000 7f0ba933b700 1 -- 172.21.15.1:0/3452655641 shutdown_connections 2024-03-18T20:23:44.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.795+0000 7f0ba933b700 1 --2- 172.21.15.1:0/3452655641 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b9006adb0 0x7f0b9006d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.795+0000 7f0ba933b700 1 --2- 172.21.15.1:0/3452655641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ba415d5b0 0x7f0ba40b6280 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.795+0000 7f0ba933b700 1 --2- 172.21.15.1:0/3452655641 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba406ca80 0x7f0ba40b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.795+0000 7f0ba933b700 1 --2- 172.21.15.1:0/3452655641 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ba409a150 0x7f0ba4149260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.795+0000 7f0ba933b700 1 -- 172.21.15.1:0/3452655641 >> 172.21.15.1:0/3452655641 conn(0x7f0ba40b2bd0 msgr2=0x7f0ba40b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:44.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.795+0000 7f0ba933b700 1 -- 172.21.15.1:0/3452655641 shutdown_connections 2024-03-18T20:23:44.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.795+0000 7f0ba933b700 1 -- 172.21.15.1:0/3452655641 wait complete. 2024-03-18T20:23:44.801 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-18T20:23:44.812 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:23:44.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.933+0000 7f50c2724700 1 Processor -- start 2024-03-18T20:23:44.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.933+0000 7f50c2724700 1 -- start start 2024-03-18T20:23:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.933+0000 7f50c2724700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f50bc0b6b60 0x7f50bc0cc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.933+0000 7f50c2724700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f50bc0cee90 0x7f50bc0cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:44.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.933+0000 7f50c2724700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50bc0cfbd0 0x7f50bc099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.933+0000 7f50c2724700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50bc0d0060 con 0x7f50bc0cee90 2024-03-18T20:23:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.933+0000 7f50c2724700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50bc0cd0d0 con 0x7f50bc0b6b60 2024-03-18T20:23:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.933+0000 7f50c2724700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc0b3f00 con 0x7f50bc0cfbd0 2024-03-18T20:23:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.933+0000 7f50bb7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f50bc0cee90 0x7f50bc0cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.934+0000 7f50bb7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f50bc0cee90 0x7f50bc0cf690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57818/0 (socket says 172.21.15.1:57818) 2024-03-18T20:23:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.934+0000 7f50bb7fe700 1 -- 172.21.15.1:0/2777750888 learned_addr learned my addr 172.21.15.1:0/2777750888 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.934+0000 7f50bbfff700 1 --2- 172.21.15.1:0/2777750888 >> v2:172.21.15.165:3300/0 conn(0x7f50bc0b6b60 0x7f50bc0cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.934+0000 7f50c0cc1700 1 --2- 172.21.15.1:0/2777750888 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50bc0cfbd0 0x7f50bc099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.934+0000 7f50bb7fe700 1 -- 172.21.15.1:0/2777750888 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50bc0cfbd0 msgr2=0x7f50bc099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.934+0000 7f50bb7fe700 1 --2- 172.21.15.1:0/2777750888 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50bc0cfbd0 0x7f50bc099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.934+0000 7f50bb7fe700 1 -- 172.21.15.1:0/2777750888 >> v2:172.21.15.165:3300/0 conn(0x7f50bc0b6b60 msgr2=0x7f50bc0cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.934+0000 7f50bb7fe700 1 --2- 172.21.15.1:0/2777750888 >> v2:172.21.15.165:3300/0 conn(0x7f50bc0b6b60 0x7f50bc0cc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.934+0000 7f50bb7fe700 1 -- 172.21.15.1:0/2777750888 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50bc0b41e0 con 0x7f50bc0cee90 2024-03-18T20:23:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.934+0000 7f50c0cc1700 1 --2- 172.21.15.1:0/2777750888 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50bc0cfbd0 0x7f50bc099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.934+0000 7f50bb7fe700 1 --2- 172.21.15.1:0/2777750888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50bc0cee90 0x7f50bc0cf690 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7f50bc06e730 tx=0x7f50b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3347f144a7874 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.934+0000 7f50baffd700 1 -- 172.21.15.1:0/2777750888 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b000b870 con 0x7f50bc0cee90 2024-03-18T20:23:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.934+0000 7f50baffd700 1 -- 172.21.15.1:0/2777750888 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b000b9d0 con 0x7f50bc0cee90 2024-03-18T20:23:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.934+0000 7f50baffd700 1 -- 172.21.15.1:0/2777750888 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b000bd60 con 0x7f50bc0cee90 2024-03-18T20:23:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.935+0000 7f50c2724700 1 -- 172.21.15.1:0/2777750888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50bc0cee90 msgr2=0x7f50bc0cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.935+0000 7f50c2724700 1 --2- 172.21.15.1:0/2777750888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50bc0cee90 0x7f50bc0cf690 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7f50bc06e730 tx=0x7f50b0009bd0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.935+0000 7f50c2724700 1 -- 172.21.15.1:0/2777750888 shutdown_connections 2024-03-18T20:23:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.935+0000 7f50c2724700 1 --2- 172.21.15.1:0/2777750888 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50bc0cfbd0 0x7f50bc099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.935+0000 7f50c2724700 1 --2- 172.21.15.1:0/2777750888 >> v2:172.21.15.165:3300/0 conn(0x7f50bc0b6b60 0x7f50bc0cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.935+0000 7f50c2724700 1 --2- 172.21.15.1:0/2777750888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50bc0cee90 0x7f50bc0cf690 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.935+0000 7f50c2724700 1 -- 172.21.15.1:0/2777750888 >> 172.21.15.1:0/2777750888 conn(0x7f50bc0b2ba0 msgr2=0x7f50bc0b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:44.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.935+0000 7f50c2724700 1 -- 172.21.15.1:0/2777750888 shutdown_connections 2024-03-18T20:23:44.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.935+0000 7f50c2724700 1 -- 172.21.15.1:0/2777750888 wait complete. 2024-03-18T20:23:44.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.936+0000 7f50c2724700 1 Processor -- start 2024-03-18T20:23:44.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.936+0000 7f50c2724700 1 -- start start 2024-03-18T20:23:44.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.937+0000 7f50c2724700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50bc0b6b60 0x7f50bc14bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:44.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.937+0000 7f50c2724700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50bc0cee90 0x7f50bc0cae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:44.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.937+0000 7f50c2724700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50bc0cfbd0 0x7f50bc06f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:44.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.937+0000 7f50c2724700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc0ca1a0 con 0x7f50bc0cfbd0 2024-03-18T20:23:44.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.937+0000 7f50c2724700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc14f370 con 0x7f50bc0b6b60 2024-03-18T20:23:44.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.937+0000 7f50c2724700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc0ca3b0 con 0x7f50bc0cee90 2024-03-18T20:23:44.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.937+0000 7f50c0cc1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50bc0cfbd0 0x7f50bc06f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:44.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.937+0000 7f50c0cc1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50bc0cfbd0 0x7f50bc06f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57832/0 (socket says 172.21.15.1:57832) 2024-03-18T20:23:44.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.937+0000 7f50c0cc1700 1 -- 172.21.15.1:0/319387587 learned_addr learned my addr 172.21.15.1:0/319387587 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:44.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.937+0000 7f50bb7fe700 1 --2- 172.21.15.1:0/319387587 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50bc0cee90 0x7f50bc0cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:44.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.937+0000 7f50bbfff700 1 --2- 172.21.15.1:0/319387587 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50bc0b6b60 0x7f50bc14bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:44.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.937+0000 7f50c0cc1700 1 -- 172.21.15.1:0/319387587 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50bc0cee90 msgr2=0x7f50bc0cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:44.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.937+0000 7f50c0cc1700 1 --2- 172.21.15.1:0/319387587 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50bc0cee90 0x7f50bc0cae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.937+0000 7f50c0cc1700 1 -- 172.21.15.1:0/319387587 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50bc0b6b60 msgr2=0x7f50bc14bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:44.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.937+0000 7f50c0cc1700 1 --2- 172.21.15.1:0/319387587 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50bc0b6b60 0x7f50bc14bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:44.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.937+0000 7f50c0cc1700 1 -- 172.21.15.1:0/319387587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b0009550 con 0x7f50bc0cfbd0 2024-03-18T20:23:44.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.937+0000 7f50c0cc1700 1 --2- 172.21.15.1:0/319387587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50bc0cfbd0 0x7f50bc06f820 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7f50b400cd50 tx=0x7f50b400a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:44.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.938+0000 7f50b8ff9700 1 -- 172.21.15.1:0/319387587 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b400cea0 con 0x7f50bc0cfbd0 2024-03-18T20:23:44.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.938+0000 7f50b8ff9700 1 -- 172.21.15.1:0/319387587 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b400f480 con 0x7f50bc0cfbd0 2024-03-18T20:23:44.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.938+0000 7f50c2724700 1 -- 172.21.15.1:0/319387587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50bc0ca690 con 0x7f50bc0cfbd0 2024-03-18T20:23:44.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.938+0000 7f50b8ff9700 1 -- 172.21.15.1:0/319387587 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b400f830 con 0x7f50bc0cfbd0 2024-03-18T20:23:44.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.938+0000 7f50c2724700 1 -- 172.21.15.1:0/319387587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50bc0cb3d0 con 0x7f50bc0cfbd0 2024-03-18T20:23:44.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.939+0000 7f50c2724700 1 -- 172.21.15.1:0/319387587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50bc0cabb0 con 0x7f50bc0cfbd0 2024-03-18T20:23:44.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.939+0000 7f50b8ff9700 1 -- 172.21.15.1:0/319387587 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f50b400b040 con 0x7f50bc0cfbd0 2024-03-18T20:23:44.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.939+0000 7f50b8ff9700 1 --2- 172.21.15.1:0/319387587 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f50a806adb0 0x7f50a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:44.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.940+0000 7f50b8ff9700 1 -- 172.21.15.1:0/319387587 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f50b4089da0 con 0x7f50bc0cfbd0 2024-03-18T20:23:44.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.940+0000 7f50bbfff700 1 --2- 172.21.15.1:0/319387587 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f50a806adb0 0x7f50a806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:44.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.940+0000 7f50bbfff700 1 --2- 172.21.15.1:0/319387587 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f50a806adb0 0x7f50a806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f50ac009950 tx=0x7f50ac008040 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:44.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:44.944+0000 7f50b8ff9700 1 -- 172.21.15.1:0/319387587 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f50b4054c40 con 0x7f50bc0cfbd0 2024-03-18T20:23:45.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.192+0000 7f50c2724700 1 -- 172.21.15.1:0/319387587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50bc0676a0 con 0x7f50bc0cfbd0 2024-03-18T20:23:45.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.192+0000 7f50b8ff9700 1 -- 172.21.15.1:0/319387587 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f50b4058260 con 0x7f50bc0cfbd0 2024-03-18T20:23:45.198 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:45.198 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":97,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2241819619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2241819619},{"type":"v1","addr":"172.21.15.165:6835","nonce":2241819619}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/320492236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":320492236},{"type":"v1","addr":"172.21.15.165:6834","nonce":320492236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":11031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/1035910531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1035910531},{"type":"v1","addr":"172.21.15.1:6836","nonce":1035910531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:23:39.747292+0000","modified":"2024-03-18T20:23:42.766952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11034":{"gid":11034,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2388480799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2388480799},{"type":"v1","addr":"172.21.15.1:6837","nonce":2388480799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-03-18T20:23:45.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.194+0000 7f50c2724700 1 -- 172.21.15.1:0/319387587 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f50a806adb0 msgr2=0x7f50a806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:45.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.194+0000 7f50c2724700 1 --2- 172.21.15.1:0/319387587 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f50a806adb0 0x7f50a806d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f50ac009950 tx=0x7f50ac008040 comp rx=0 tx=0).stop 2024-03-18T20:23:45.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.194+0000 7f50c2724700 1 -- 172.21.15.1:0/319387587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50bc0cfbd0 msgr2=0x7f50bc06f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:45.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.194+0000 7f50c2724700 1 --2- 172.21.15.1:0/319387587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50bc0cfbd0 0x7f50bc06f820 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7f50b400cd50 tx=0x7f50b400a560 comp rx=0 tx=0).stop 2024-03-18T20:23:45.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.194+0000 7f50c2724700 1 -- 172.21.15.1:0/319387587 shutdown_connections 2024-03-18T20:23:45.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.194+0000 7f50c2724700 1 --2- 172.21.15.1:0/319387587 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f50a806adb0 0x7f50a806d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.194+0000 7f50c2724700 1 --2- 172.21.15.1:0/319387587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50bc0cfbd0 0x7f50bc06f820 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.194+0000 7f50c2724700 1 --2- 172.21.15.1:0/319387587 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50bc0b6b60 0x7f50bc14bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.194+0000 7f50c2724700 1 --2- 172.21.15.1:0/319387587 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50bc0cee90 0x7f50bc0cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.194+0000 7f50c2724700 1 -- 172.21.15.1:0/319387587 >> 172.21.15.1:0/319387587 conn(0x7f50bc0b2ba0 msgr2=0x7f50bc0b4bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:45.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.195+0000 7f50c2724700 1 -- 172.21.15.1:0/319387587 shutdown_connections 2024-03-18T20:23:45.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.195+0000 7f50c2724700 1 -- 172.21.15.1:0/319387587 wait complete. 2024-03-18T20:23:45.201 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 97 2024-03-18T20:23:45.308 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-18T20:23:45.303+0000 7efcc4a30700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T20:23:45.615 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T20:23:39.747292+0000', 'modified': '2024-03-18T20:23:42.766952+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11034}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11034': {'gid': 11034, 'name': 'c', 'rank': 0, 'incarnation': 93, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6837/2388480799', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6835', 'nonce': 2388480799}, {'type': 'v1', 'addr': '172.21.15.1:6837', 'nonce': 2388480799}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T20:23:45.615 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T20:23:45.616 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T20:23:45.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.741+0000 7fe7c8bf0700 1 Processor -- start 2024-03-18T20:23:45.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.741+0000 7fe7c8bf0700 1 -- start start 2024-03-18T20:23:45.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.742+0000 7fe7c8bf0700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe7c406a9c0 0x7fe7c40ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.742+0000 7fe7c8bf0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7c406d310 0x7fe7c406f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.742+0000 7fe7c8bf0700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c406f010 con 0x7fe7c40cc3b0 2024-03-18T20:23:45.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.742+0000 7fe7c8bf0700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7c406d880 con 0x7fe7c406a9c0 2024-03-18T20:23:45.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.742+0000 7fe7c8bf0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c4071ea0 con 0x7fe7c406d310 2024-03-18T20:23:45.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.742+0000 7fe7c1d9b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7c406d310 0x7fe7c406f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.742+0000 7fe7c1d9b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7c406d310 0x7fe7c406f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:48738/0 (socket says 172.21.15.1:48738) 2024-03-18T20:23:45.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.742+0000 7fe7c1d9b700 1 -- 172.21.15.1:0/81852076 learned_addr learned my addr 172.21.15.1:0/81852076 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:45.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.742+0000 7fe7c259c700 1 --2- 172.21.15.1:0/81852076 >> v2:172.21.15.165:3300/0 conn(0x7fe7c406a9c0 0x7fe7c40ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.743+0000 7fe7c259c700 1 -- 172.21.15.1:0/81852076 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7c406d310 msgr2=0x7fe7c406f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:45.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.743+0000 7fe7c259c700 1 --2- 172.21.15.1:0/81852076 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7c406d310 0x7fe7c406f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.743+0000 7fe7c259c700 1 -- 172.21.15.1:0/81852076 >> v1:172.21.15.165:6789/0 conn(0x7fe7c40cc3b0 legacy=0x7fe7c415e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:45.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.743+0000 7fe7c259c700 1 -- 172.21.15.1:0/81852076 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c406ecb0 con 0x7fe7c406a9c0 2024-03-18T20:23:45.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.743+0000 7fe7c1d9b700 1 --2- 172.21.15.1:0/81852076 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7c406d310 0x7fe7c406f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:23:45.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.743+0000 7fe7c259c700 1 --2- 172.21.15.1:0/81852076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7c406a9c0 0x7fe7c40ca5c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4009840 tx=0x7fe7b4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20e263e2da8e6c00 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:45.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.743+0000 7fe7c159a700 1 -- 172.21.15.1:0/81852076 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4011830 con 0x7fe7c406a9c0 2024-03-18T20:23:45.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.743+0000 7fe7c159a700 1 -- 172.21.15.1:0/81852076 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b4011990 con 0x7fe7c406a9c0 2024-03-18T20:23:45.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.743+0000 7fe7c159a700 1 -- 172.21.15.1:0/81852076 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4011d20 con 0x7fe7c406a9c0 2024-03-18T20:23:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.743+0000 7fe7c8bf0700 1 -- 172.21.15.1:0/81852076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7c406a9c0 msgr2=0x7fe7c40ca5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.743+0000 7fe7c8bf0700 1 --2- 172.21.15.1:0/81852076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7c406a9c0 0x7fe7c40ca5c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4009840 tx=0x7fe7b4009b50 comp rx=0 tx=0).stop 2024-03-18T20:23:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.744+0000 7fe7c8bf0700 1 -- 172.21.15.1:0/81852076 shutdown_connections 2024-03-18T20:23:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.744+0000 7fe7c8bf0700 1 --2- 172.21.15.1:0/81852076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7c406a9c0 0x7fe7c40ca5c0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.744+0000 7fe7c8bf0700 1 --2- 172.21.15.1:0/81852076 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7c406d310 0x7fe7c406f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.744+0000 7fe7c8bf0700 1 -- 172.21.15.1:0/81852076 >> 172.21.15.1:0/81852076 conn(0x7fe7c40b1aa0 msgr2=0x7fe7c40b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.744+0000 7fe7c8bf0700 1 -- 172.21.15.1:0/81852076 shutdown_connections 2024-03-18T20:23:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.744+0000 7fe7c8bf0700 1 -- 172.21.15.1:0/81852076 wait complete. 2024-03-18T20:23:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.745+0000 7fe7c8bf0700 1 Processor -- start 2024-03-18T20:23:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.745+0000 7fe7c8bf0700 1 -- start start 2024-03-18T20:23:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.746+0000 7fe7c8bf0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7c406a9c0 0x7fe7c40c6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.746+0000 7fe7c8bf0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7c406d310 0x7fe7c4155e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.746+0000 7fe7c8bf0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7c40cc3b0 0x7fe7c40c55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.746+0000 7fe7c8bf0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c41563a0 con 0x7fe7c406d310 2024-03-18T20:23:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.746+0000 7fe7c8bf0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c409e970 con 0x7fe7c40cc3b0 2024-03-18T20:23:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.746+0000 7fe7c8bf0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c409e3b0 con 0x7fe7c406a9c0 2024-03-18T20:23:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.746+0000 7fe7c1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7c406d310 0x7fe7c4155e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.746+0000 7fe7c1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7c406d310 0x7fe7c4155e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57844/0 (socket says 172.21.15.1:57844) 2024-03-18T20:23:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.746+0000 7fe7c1d9b700 1 -- 172.21.15.1:0/2719908032 learned_addr learned my addr 172.21.15.1:0/2719908032 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.746+0000 7fe7c2d9d700 1 --2- 172.21.15.1:0/2719908032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7c40cc3b0 0x7fe7c40c55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.746+0000 7fe7c259c700 1 --2- 172.21.15.1:0/2719908032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7c406a9c0 0x7fe7c40c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.746+0000 7fe7c1d9b700 1 -- 172.21.15.1:0/2719908032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7c406a9c0 msgr2=0x7fe7c40c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.746+0000 7fe7c1d9b700 1 --2- 172.21.15.1:0/2719908032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7c406a9c0 0x7fe7c40c6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.746+0000 7fe7c1d9b700 1 -- 172.21.15.1:0/2719908032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7c40cc3b0 msgr2=0x7fe7c40c55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.746+0000 7fe7c1d9b700 1 --2- 172.21.15.1:0/2719908032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7c40cc3b0 0x7fe7c40c55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.746+0000 7fe7c1d9b700 1 -- 172.21.15.1:0/2719908032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b8009550 con 0x7fe7c406d310 2024-03-18T20:23:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.746+0000 7fe7c1d9b700 1 --2- 172.21.15.1:0/2719908032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7c406d310 0x7fe7c4155e60 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800ebe0 tx=0x7fe7b800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.747+0000 7fe7b2ffd700 1 -- 172.21.15.1:0/2719908032 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b800c850 con 0x7fe7c406d310 2024-03-18T20:23:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.747+0000 7fe7b2ffd700 1 -- 172.21.15.1:0/2719908032 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b800c9b0 con 0x7fe7c406d310 2024-03-18T20:23:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.747+0000 7fe7c8bf0700 1 -- 172.21.15.1:0/2719908032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b40094a0 con 0x7fe7c406d310 2024-03-18T20:23:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.747+0000 7fe7b2ffd700 1 -- 172.21.15.1:0/2719908032 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b800cd40 con 0x7fe7c406d310 2024-03-18T20:23:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.747+0000 7fe7c8bf0700 1 -- 172.21.15.1:0/2719908032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c40c9b20 con 0x7fe7c406d310 2024-03-18T20:23:45.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.748+0000 7fe7b2ffd700 1 -- 172.21.15.1:0/2719908032 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe7b8013470 con 0x7fe7c406d310 2024-03-18T20:23:45.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.749+0000 7fe7b2ffd700 1 --2- 172.21.15.1:0/2719908032 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe7a406adb0 0x7fe7a406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.749+0000 7fe7b2ffd700 1 -- 172.21.15.1:0/2719908032 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fe7b80897f0 con 0x7fe7c406d310 2024-03-18T20:23:45.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.749+0000 7fe7c259c700 1 --2- 172.21.15.1:0/2719908032 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe7a406adb0 0x7fe7a406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.749+0000 7fe7c259c700 1 --2- 172.21.15.1:0/2719908032 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe7a406adb0 0x7fe7a406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4000c00 tx=0x7fe7b4008030 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:45.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.750+0000 7fe7d4d99b80 1 Processor -- start 2024-03-18T20:23:45.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.751+0000 7fe7d4d99b80 1 -- start start 2024-03-18T20:23:45.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.751+0000 7fe7d4d99b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e8487c10 0x5653e8487ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.751+0000 7fe7d4d99b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e84590a0 0x5653e84a4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.751+0000 7fe7d4d99b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8499630 0x5653e845c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.751+0000 7fe7d4d99b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e83d2f20 con 0x5653e8499630 2024-03-18T20:23:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.751+0000 7fe7d4d99b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e83b4c30 con 0x5653e84590a0 2024-03-18T20:23:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.751+0000 7fe7d4d99b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x5653e83b2430 con 0x5653e8487c10 2024-03-18T20:23:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.751+0000 7fe7c2d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8499630 0x5653e845c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.752+0000 7fe7c2d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8499630 0x5653e845c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57856/0 (socket says 172.21.15.1:57856) 2024-03-18T20:23:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.752+0000 7fe7c2d9d700 1 -- 172.21.15.1:0/1625436701 learned_addr learned my addr 172.21.15.1:0/1625436701 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.752+0000 7fe7c1d9b700 1 --2- 172.21.15.1:0/1625436701 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e84590a0 0x5653e84a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.752+0000 7fe7c259c700 1 --2- 172.21.15.1:0/1625436701 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e8487c10 0x5653e8487ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.752+0000 7fe7c2d9d700 1 -- 172.21.15.1:0/1625436701 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e8487c10 msgr2=0x5653e8487ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.752+0000 7fe7c2d9d700 1 --2- 172.21.15.1:0/1625436701 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e8487c10 0x5653e8487ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.752+0000 7fe7c2d9d700 1 -- 172.21.15.1:0/1625436701 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e84590a0 msgr2=0x5653e84a4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.752+0000 7fe7c2d9d700 1 --2- 172.21.15.1:0/1625436701 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e84590a0 0x5653e84a4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.752+0000 7fe7c2d9d700 1 -- 172.21.15.1:0/1625436701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653e81a1790 con 0x5653e8499630 2024-03-18T20:23:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.752+0000 7fe7c2d9d700 1 --2- 172.21.15.1:0/1625436701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8499630 0x5653e845c160 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc0021a0 tx=0x7fe7bc001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52fffcb27c4fbed5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.752+0000 7fe7a3fff700 1 -- 172.21.15.1:0/1625436701 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc00fcc0 con 0x5653e8499630 2024-03-18T20:23:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.752+0000 7fe7a3fff700 1 -- 172.21.15.1:0/1625436701 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7bc00fe20 con 0x5653e8499630 2024-03-18T20:23:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.752+0000 7fe7a3fff700 1 -- 172.21.15.1:0/1625436701 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc003c20 con 0x5653e8499630 2024-03-18T20:23:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.753+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/1625436701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8499630 msgr2=0x5653e845c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.753+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/1625436701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8499630 0x5653e845c160 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc0021a0 tx=0x7fe7bc001a70 comp rx=0 tx=0).stop 2024-03-18T20:23:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.753+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/1625436701 shutdown_connections 2024-03-18T20:23:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.753+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/1625436701 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e8487c10 0x5653e8487ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.753+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/1625436701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8499630 0x5653e845c160 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.753+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/1625436701 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e84590a0 0x5653e84a4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.753+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/1625436701 >> 172.21.15.1:0/1625436701 conn(0x5653e83bbcf0 msgr2=0x5653e845ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.753+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/1625436701 shutdown_connections 2024-03-18T20:23:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.753+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/1625436701 wait complete. 2024-03-18T20:23:45.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.753+0000 7fe7d4d99b80 1 -- start start 2024-03-18T20:23:45.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.754+0000 7fe7d4d99b80 1 Processor -- start 2024-03-18T20:23:45.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.754+0000 7fe7d4d99b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e84590a0 0x5653e849c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.754+0000 7fe7d4d99b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 0x5653e848b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.754+0000 7fe7d4d99b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e8499630 0x5653e847d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.754+0000 7fe7d4d99b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e83add20 con 0x5653e8487c10 2024-03-18T20:23:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.754+0000 7fe7d4d99b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e8349770 con 0x5653e8499630 2024-03-18T20:23:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.754+0000 7fe7d4d99b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x5653e836c6f0 con 0x5653e84590a0 2024-03-18T20:23:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7d4d99b80 10 client.0 _ll_register_callbacks cb 0x5653e81b55a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:23:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7c1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 0x5653e848b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7c1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 0x5653e848b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57868/0 (socket says 172.21.15.1:57868) 2024-03-18T20:23:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7c1d9b700 1 -- 172.21.15.1:0/1572904692 learned_addr learned my addr 172.21.15.1:0/1572904692 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7c2d9d700 1 --2- 172.21.15.1:0/1572904692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e8499630 0x5653e847d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7c259c700 1 --2- 172.21.15.1:0/1572904692 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e84590a0 0x5653e849c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7c1d9b700 1 -- 172.21.15.1:0/1572904692 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e84590a0 msgr2=0x5653e849c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7c1d9b700 1 --2- 172.21.15.1:0/1572904692 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e84590a0 0x5653e849c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7c1d9b700 1 -- 172.21.15.1:0/1572904692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e8499630 msgr2=0x5653e847d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7c1d9b700 1 --2- 172.21.15.1:0/1572904692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e8499630 0x5653e847d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7c1d9b700 1 -- 172.21.15.1:0/1572904692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c406f010 con 0x5653e8487c10 2024-03-18T20:23:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7c2d9d700 1 --2- 172.21.15.1:0/1572904692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e8499630 0x5653e847d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7c1d9b700 1 --2- 172.21.15.1:0/1572904692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 0x5653e848b670 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800ebb0 tx=0x7fe7b8058b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4abac4bbacaa23f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7a17fa700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T20:23:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7a17fa700 1 -- 172.21.15.1:0/1572904692 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b805c200 con 0x5653e8487c10 2024-03-18T20:23:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7a17fa700 1 -- 172.21.15.1:0/1572904692 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(214..214 src has 1..214) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fe7b80607c0 con 0x5653e8487c10 2024-03-18T20:23:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/1572904692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5653e844fe90 con 0x5653e8487c10 2024-03-18T20:23:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7a17fa700 1 -- 172.21.15.1:0/1572904692 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7b80578c0 con 0x5653e8487c10 2024-03-18T20:23:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7d4d99b80 10 client.11157 fetch_fsmap learned FSMap version 97 2024-03-18T20:23:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.755+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/1572904692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=2+}) v3 -- 0x5653e828e0a0 con 0x5653e8487c10 2024-03-18T20:23:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.756+0000 7fe7a17fa700 1 -- 172.21.15.1:0/1572904692 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5653e844fe90 con 0x5653e8487c10 2024-03-18T20:23:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.756+0000 7fe7a17fa700 1 -- 172.21.15.1:0/1572904692 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 97) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe7b8060330 con 0x5653e8487c10 2024-03-18T20:23:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.756+0000 7fe7d4d99b80 10 client.11157 fetch_fsmap finished waiting for FSMap version 97 2024-03-18T20:23:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.756+0000 7fe7d4d99b80 10 client.11157 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T20:23:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.756+0000 7fe7d4d99b80 20 client.11157 populate_metadata read hostname 'smithi001' 2024-03-18T20:23:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.756+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/1572904692 >> [v2:172.21.15.1:6835/2388480799,v1:172.21.15.1:6837/2388480799] conn(0x5653e8489410 0x5653e84897f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.756+0000 7fe7d4d99b80 4 client.11157 mds_command: new command op to 11034 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T20:23:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.756+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/1572904692 --> [v2:172.21.15.1:6835/2388480799,v1:172.21.15.1:6837/2388480799] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5653e842d010 con 0x5653e8489410 2024-03-18T20:23:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.756+0000 7fe7c259c700 1 --2- 172.21.15.1:0/1572904692 >> [v2:172.21.15.1:6835/2388480799,v1:172.21.15.1:6837/2388480799] conn(0x5653e8489410 0x5653e84897f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.756+0000 7fe7a17fa700 10 client.11157 ms_handle_connect on v2:172.21.15.1:6835/2388480799 2024-03-18T20:23:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.756+0000 7fe7c259c700 1 --2- 172.21.15.1:0/1572904692 >> [v2:172.21.15.1:6835/2388480799,v1:172.21.15.1:6837/2388480799] conn(0x5653e8489410 0x5653e84897f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e983405fa9edf3d server_cookie=ec68a56de2eae06e in_seq=0 out_seq=0 2024-03-18T20:23:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.757+0000 7fe7a17fa700 1 -- 172.21.15.1:0/1572904692 <== mds.0 v2:172.21.15.1:6835/2388480799 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5653e842d010 con 0x5653e8489410 2024-03-18T20:23:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.757+0000 7fe7a17fa700 10 client.11157 handle_command_reply: tid=0 2024-03-18T20:23:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.757+0000 7fe7d4d99b80 1 client.11157 shutdown 2024-03-18T20:23:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.757+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/1572904692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 msgr2=0x5653e848b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.757+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/1572904692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 0x5653e848b670 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800ebb0 tx=0x7fe7b8058b80 comp rx=0 tx=0).stop 2024-03-18T20:23:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.757+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/1572904692 shutdown_connections 2024-03-18T20:23:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.757+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/1572904692 >> [v2:172.21.15.1:6835/2388480799,v1:172.21.15.1:6837/2388480799] conn(0x5653e8489410 0x5653e84897f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.757+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/1572904692 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e84590a0 0x5653e849c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.757+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/1572904692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e8499630 0x5653e847d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.757+0000 7fe7a17fa700 0 client.11157 ms_handle_reset on v2:172.21.15.1:6835/2388480799 2024-03-18T20:23:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.757+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/1572904692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 0x5653e848b670 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.757+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/1572904692 >> 172.21.15.1:0/1572904692 conn(0x5653e83bbcf0 msgr2=0x5653e8488330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.757+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/1572904692 shutdown_connections 2024-03-18T20:23:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.757+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/1572904692 wait complete. 2024-03-18T20:23:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.757+0000 7fe7d4d99b80 20 client.11157 trim_cache size 0 max 16384 2024-03-18T20:23:45.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.775+0000 7fe7d4d99b80 1 Processor -- start 2024-03-18T20:23:45.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.775+0000 7fe7d4d99b80 1 -- start start 2024-03-18T20:23:45.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.776+0000 7fe7d4d99b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e84590a0 0x5653e847e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.776+0000 7fe7d4d99b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 0x5653e849a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.776+0000 7fe7d4d99b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e8499630 0x5653e84a2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.776+0000 7fe7d4d99b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e81a2390 con 0x5653e8487c10 2024-03-18T20:23:45.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.776+0000 7fe7d4d99b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e847cf10 con 0x5653e84590a0 2024-03-18T20:23:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.776+0000 7fe7d4d99b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x5653e849a690 con 0x5653e8499630 2024-03-18T20:23:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.776+0000 7fe7c1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 0x5653e849a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.776+0000 7fe7c1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 0x5653e849a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57872/0 (socket says 172.21.15.1:57872) 2024-03-18T20:23:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.776+0000 7fe7c1d9b700 1 -- 172.21.15.1:0/1589370363 learned_addr learned my addr 172.21.15.1:0/1589370363 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.776+0000 7fe7c259c700 1 --2- 172.21.15.1:0/1589370363 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e84590a0 0x5653e847e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.776+0000 7fe7c2d9d700 1 --2- 172.21.15.1:0/1589370363 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e8499630 0x5653e84a2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.776+0000 7fe7c1d9b700 1 -- 172.21.15.1:0/1589370363 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e8499630 msgr2=0x5653e84a2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.776+0000 7fe7c1d9b700 1 --2- 172.21.15.1:0/1589370363 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e8499630 0x5653e84a2d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.776+0000 7fe7c1d9b700 1 -- 172.21.15.1:0/1589370363 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e84590a0 msgr2=0x5653e847e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.776+0000 7fe7c1d9b700 1 --2- 172.21.15.1:0/1589370363 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e84590a0 0x5653e847e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.776+0000 7fe7c1d9b700 1 -- 172.21.15.1:0/1589370363 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653e8374010 con 0x5653e8487c10 2024-03-18T20:23:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.776+0000 7fe7c1d9b700 1 --2- 172.21.15.1:0/1589370363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 0x5653e849a150 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800cfd0 tx=0x7fe7b8089000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9440ee8d14d97f5f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:45.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.778+0000 7fe7b0ff9700 1 -- 172.21.15.1:0/1589370363 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b80578c0 con 0x5653e8487c10 2024-03-18T20:23:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.778+0000 7fe7b0ff9700 1 -- 172.21.15.1:0/1589370363 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b8057b10 con 0x5653e8487c10 2024-03-18T20:23:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.779+0000 7fe7b0ff9700 1 -- 172.21.15.1:0/1589370363 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b80607c0 con 0x5653e8487c10 2024-03-18T20:23:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.779+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/1589370363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 msgr2=0x5653e849a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.779+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/1589370363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 0x5653e849a150 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800cfd0 tx=0x7fe7b8089000 comp rx=0 tx=0).stop 2024-03-18T20:23:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.779+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/1589370363 shutdown_connections 2024-03-18T20:23:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.779+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/1589370363 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e8499630 0x5653e84a2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.779+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/1589370363 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e84590a0 0x5653e847e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.779+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/1589370363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 0x5653e849a150 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.779+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/1589370363 >> 172.21.15.1:0/1589370363 conn(0x5653e83bbcf0 msgr2=0x5653e849a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.779+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/1589370363 shutdown_connections 2024-03-18T20:23:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.779+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/1589370363 wait complete. 2024-03-18T20:23:45.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.780+0000 7fe7d4d99b80 1 -- start start 2024-03-18T20:23:45.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.780+0000 7fe7d4d99b80 1 Processor -- start 2024-03-18T20:23:45.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.781+0000 7fe7d4d99b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e84590a0 0x5653e8509440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.781+0000 7fe7d4d99b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 0x5653e8498dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.781+0000 7fe7d4d99b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e8499630 0x5653e8485c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.781+0000 7fe7d4d99b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e84864a0 con 0x5653e8487c10 2024-03-18T20:23:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.781+0000 7fe7d4d99b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e8482e20 con 0x5653e84590a0 2024-03-18T20:23:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.781+0000 7fe7d4d99b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x5653e847f1e0 con 0x5653e8499630 2024-03-18T20:23:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.781+0000 7fe7d4d99b80 10 client.0 _ll_register_callbacks cb 0x7fe7b8057f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:23:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.781+0000 7fe7c1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 0x5653e8498dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.781+0000 7fe7c1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 0x5653e8498dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57880/0 (socket says 172.21.15.1:57880) 2024-03-18T20:23:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.781+0000 7fe7c1d9b700 1 -- 172.21.15.1:0/3529126122 learned_addr learned my addr 172.21.15.1:0/3529126122 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.781+0000 7fe7c259c700 1 --2- 172.21.15.1:0/3529126122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e84590a0 0x5653e8509440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.781+0000 7fe7c2d9d700 1 --2- 172.21.15.1:0/3529126122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e8499630 0x5653e8485c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.781+0000 7fe7c1d9b700 1 -- 172.21.15.1:0/3529126122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e8499630 msgr2=0x5653e8485c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.781+0000 7fe7c1d9b700 1 --2- 172.21.15.1:0/3529126122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e8499630 0x5653e8485c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.781+0000 7fe7c1d9b700 1 -- 172.21.15.1:0/3529126122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e84590a0 msgr2=0x5653e8509440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.781+0000 7fe7c1d9b700 1 --2- 172.21.15.1:0/3529126122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e84590a0 0x5653e8509440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.781+0000 7fe7c1d9b700 1 -- 172.21.15.1:0/3529126122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5653e828e0a0 con 0x5653e8487c10 2024-03-18T20:23:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.782+0000 7fe7c1d9b700 1 --2- 172.21.15.1:0/3529126122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 0x5653e8498dc0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7fe7b80098e0 tx=0x7fe7b8087f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ba92824ce88e1e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.782+0000 7fe7a17fa700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T20:23:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.782+0000 7fe7a17fa700 1 -- 172.21.15.1:0/3529126122 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b805e500 con 0x5653e8487c10 2024-03-18T20:23:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.782+0000 7fe7a17fa700 1 -- 172.21.15.1:0/3529126122 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(214..214 src has 1..214) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fe7b805c200 con 0x5653e8487c10 2024-03-18T20:23:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.782+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/3529126122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe7bc00fe20 con 0x5653e8487c10 2024-03-18T20:23:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.782+0000 7fe7a17fa700 1 -- 172.21.15.1:0/3529126122 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7b80607c0 con 0x5653e8487c10 2024-03-18T20:23:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.782+0000 7fe7d4d99b80 10 client.11163 fetch_fsmap learned FSMap version 97 2024-03-18T20:23:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.782+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/3529126122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=2+}) v3 -- 0x5653e82021d0 con 0x5653e8487c10 2024-03-18T20:23:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.782+0000 7fe7a17fa700 1 -- 172.21.15.1:0/3529126122 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7bc00fe20 con 0x5653e8487c10 2024-03-18T20:23:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.782+0000 7fe7a17fa700 1 -- 172.21.15.1:0/3529126122 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 97) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe7b8060330 con 0x5653e8487c10 2024-03-18T20:23:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.782+0000 7fe7d4d99b80 10 client.11163 fetch_fsmap finished waiting for FSMap version 97 2024-03-18T20:23:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.782+0000 7fe7d4d99b80 10 client.11163 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T20:23:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.782+0000 7fe7d4d99b80 20 client.11163 populate_metadata read hostname 'smithi001' 2024-03-18T20:23:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.782+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/3529126122 >> [v2:172.21.15.1:6835/2388480799,v1:172.21.15.1:6837/2388480799] conn(0x5653e84821f0 0x5653e848a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.782+0000 7fe7d4d99b80 4 client.11163 mds_command: new command op to 11034 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T20:23:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.782+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/3529126122 --> [v2:172.21.15.1:6835/2388480799,v1:172.21.15.1:6837/2388480799] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5653e81e7040 con 0x5653e84821f0 2024-03-18T20:23:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.782+0000 7fe7c259c700 1 --2- 172.21.15.1:0/3529126122 >> [v2:172.21.15.1:6835/2388480799,v1:172.21.15.1:6837/2388480799] conn(0x5653e84821f0 0x5653e848a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.783+0000 7fe7a17fa700 10 client.11163 ms_handle_connect on v2:172.21.15.1:6835/2388480799 2024-03-18T20:23:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.783+0000 7fe7c259c700 1 --2- 172.21.15.1:0/3529126122 >> [v2:172.21.15.1:6835/2388480799,v1:172.21.15.1:6837/2388480799] conn(0x5653e84821f0 0x5653e848a090 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1540f8a8644067 server_cookie=fc1011d195c77165 in_seq=0 out_seq=0 2024-03-18T20:23:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.783+0000 7fe7a17fa700 1 -- 172.21.15.1:0/3529126122 <== mds.0 v2:172.21.15.1:6835/2388480799 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x5653e81e7040 con 0x5653e84821f0 2024-03-18T20:23:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.783+0000 7fe7a17fa700 10 client.11163 handle_command_reply: tid=0 2024-03-18T20:23:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.783+0000 7fe7d4d99b80 1 client.11163 shutdown 2024-03-18T20:23:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.783+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/3529126122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 msgr2=0x5653e8498dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.783+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/3529126122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 0x5653e8498dc0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7fe7b80098e0 tx=0x7fe7b8087f90 comp rx=0 tx=0).stop 2024-03-18T20:23:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.783+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/3529126122 shutdown_connections 2024-03-18T20:23:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.783+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/3529126122 >> [v2:172.21.15.1:6835/2388480799,v1:172.21.15.1:6837/2388480799] conn(0x5653e84821f0 0x5653e848a090 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.783+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/3529126122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5653e8499630 0x5653e8485c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.783+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/3529126122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5653e84590a0 0x5653e8509440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.783+0000 7fe7a17fa700 0 client.11163 ms_handle_reset on v2:172.21.15.1:6835/2388480799 2024-03-18T20:23:45.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.783+0000 7fe7d4d99b80 1 --2- 172.21.15.1:0/3529126122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5653e8487c10 0x5653e8498dc0 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.783+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/3529126122 >> 172.21.15.1:0/3529126122 conn(0x5653e83bbcf0 msgr2=0x5653e847e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:45.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.783+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/3529126122 shutdown_connections 2024-03-18T20:23:45.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.783+0000 7fe7d4d99b80 1 -- 172.21.15.1:0/3529126122 wait complete. 2024-03-18T20:23:45.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.783+0000 7fe7d4d99b80 20 client.11163 trim_cache size 0 max 16384 2024-03-18T20:23:45.794 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:23:45.794 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "9c1c697f-8655-4ef3-8a13-da568d2cbe86", 2024-03-18T20:23:45.794 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-18T20:23:45.795 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 11034, 2024-03-18T20:23:45.795 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-18T20:23:45.795 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-18T20:23:45.795 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-18T20:23:45.795 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 6.0359784059999999, 2024-03-18T20:23:45.795 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 97, 2024-03-18T20:23:45.795 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 214, 2024-03-18T20:23:45.795 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 214, 2024-03-18T20:23:45.795 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 11.353701143 2024-03-18T20:23:45.795 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:23:45.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.784+0000 7fe7c8bf0700 1 -- 172.21.15.1:0/2719908032 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe7a406adb0 msgr2=0x7fe7a406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:45.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.784+0000 7fe7c8bf0700 1 --2- 172.21.15.1:0/2719908032 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe7a406adb0 0x7fe7a406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4000c00 tx=0x7fe7b4008030 comp rx=0 tx=0).stop 2024-03-18T20:23:45.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.784+0000 7fe7c8bf0700 1 -- 172.21.15.1:0/2719908032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7c406d310 msgr2=0x7fe7c4155e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:45.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.784+0000 7fe7c8bf0700 1 --2- 172.21.15.1:0/2719908032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7c406d310 0x7fe7c4155e60 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800ebe0 tx=0x7fe7b800eef0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.784+0000 7fe7c8bf0700 1 -- 172.21.15.1:0/2719908032 shutdown_connections 2024-03-18T20:23:45.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.784+0000 7fe7c8bf0700 1 --2- 172.21.15.1:0/2719908032 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe7a406adb0 0x7fe7a406d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.784+0000 7fe7c8bf0700 1 --2- 172.21.15.1:0/2719908032 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7c406a9c0 0x7fe7c40c6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.784+0000 7fe7c8bf0700 1 --2- 172.21.15.1:0/2719908032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7c40cc3b0 0x7fe7c40c55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.784+0000 7fe7c8bf0700 1 --2- 172.21.15.1:0/2719908032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7c406d310 0x7fe7c4155e60 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.784+0000 7fe7c8bf0700 1 -- 172.21.15.1:0/2719908032 >> 172.21.15.1:0/2719908032 conn(0x7fe7c40b1aa0 msgr2=0x7fe7c415e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:45.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.784+0000 7fe7c8bf0700 1 -- 172.21.15.1:0/2719908032 shutdown_connections 2024-03-18T20:23:45.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.784+0000 7fe7c8bf0700 1 -- 172.21.15.1:0/2719908032 wait complete. 2024-03-18T20:23:45.801 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T20:23:45.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.921+0000 7fa0af759700 1 Processor -- start 2024-03-18T20:23:45.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.921+0000 7fa0af759700 1 -- start start 2024-03-18T20:23:45.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.922+0000 7fa0af759700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa0a815d580 0x7fa0a8072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.922+0000 7fa0af759700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa0a806e290 0x7fa0a80cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.922+0000 7fa0af759700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a806f490 con 0x7fa0a806d640 2024-03-18T20:23:45.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.922+0000 7fa0af759700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a8071cf0 con 0x7fa0a815d580 2024-03-18T20:23:45.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.922+0000 7fa0af759700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a806f1b0 con 0x7fa0a806e290 2024-03-18T20:23:45.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.922+0000 7fa0ad4f5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa0a815d580 0x7fa0a8072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.922+0000 7fa0adcf6700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa0a806d640 0x7fa0a80ca560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55710/0 (socket says 172.21.15.1:55710) 2024-03-18T20:23:45.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.922+0000 7fa0ad4f5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa0a815d580 0x7fa0a8072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57890/0 (socket says 172.21.15.1:57890) 2024-03-18T20:23:45.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.922+0000 7fa0adcf6700 1 -- 172.21.15.1:0/2008209694 learned_addr learned my addr 172.21.15.1:0/2008209694 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:45.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.922+0000 7fa0accf4700 1 --2- 172.21.15.1:0/2008209694 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa0a806e290 0x7fa0a80cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.922+0000 7fa0ad4f5700 1 -- 172.21.15.1:0/2008209694 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa0a806e290 msgr2=0x7fa0a80cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:45.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.922+0000 7fa0ad4f5700 1 --2- 172.21.15.1:0/2008209694 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa0a806e290 0x7fa0a80cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.922+0000 7fa0ad4f5700 1 -- 172.21.15.1:0/2008209694 >> v1:172.21.15.1:6789/0 conn(0x7fa0a806d640 legacy=0x7fa0a80ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:45.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.922+0000 7fa0ad4f5700 1 -- 172.21.15.1:0/2008209694 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a80716a0 con 0x7fa0a815d580 2024-03-18T20:23:45.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.922+0000 7fa09ffff700 1 -- 172.21.15.1:0/2008209694 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189479819 0 0) 0x7fa0a806f490 con 0x7fa0a806d640 2024-03-18T20:23:45.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.922+0000 7fa0ad4f5700 1 --2- 172.21.15.1:0/2008209694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0a815d580 0x7fa0a8072180 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4009840 tx=0x7fa0a4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a940665961001d2c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:45.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.922+0000 7fa09ffff700 1 -- 172.21.15.1:0/2008209694 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4011830 con 0x7fa0a815d580 2024-03-18T20:23:45.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.923+0000 7fa09ffff700 1 -- 172.21.15.1:0/2008209694 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a4011990 con 0x7fa0a815d580 2024-03-18T20:23:45.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.923+0000 7fa09ffff700 1 -- 172.21.15.1:0/2008209694 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4011d20 con 0x7fa0a815d580 2024-03-18T20:23:45.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.923+0000 7fa0af759700 1 -- 172.21.15.1:0/2008209694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0a815d580 msgr2=0x7fa0a8072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:45.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.923+0000 7fa0af759700 1 --2- 172.21.15.1:0/2008209694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0a815d580 0x7fa0a8072180 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4009840 tx=0x7fa0a4009b50 comp rx=0 tx=0).stop 2024-03-18T20:23:45.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.923+0000 7fa0af759700 1 -- 172.21.15.1:0/2008209694 shutdown_connections 2024-03-18T20:23:45.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.923+0000 7fa0af759700 1 --2- 172.21.15.1:0/2008209694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0a815d580 0x7fa0a8072180 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.923+0000 7fa0af759700 1 --2- 172.21.15.1:0/2008209694 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa0a806e290 0x7fa0a80cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.923+0000 7fa0af759700 1 -- 172.21.15.1:0/2008209694 >> 172.21.15.1:0/2008209694 conn(0x7fa0a80cc990 msgr2=0x7fa0a80ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:45.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.923+0000 7fa0af759700 1 -- 172.21.15.1:0/2008209694 shutdown_connections 2024-03-18T20:23:45.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.923+0000 7fa0af759700 1 -- 172.21.15.1:0/2008209694 wait complete. 2024-03-18T20:23:45.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.925+0000 7fa0af759700 1 Processor -- start 2024-03-18T20:23:45.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.925+0000 7fa0af759700 1 -- start start 2024-03-18T20:23:45.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa0af759700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa0a806d640 0x7fa0a81f6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa0af759700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa0a815d580 0x7fa0a815aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa0af759700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0a81524f0 0x7fa0a815b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa0af759700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a80b3f20 con 0x7fa0a81524f0 2024-03-18T20:23:45.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa0af759700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a81f7000 con 0x7fa0a806d640 2024-03-18T20:23:45.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa0af759700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a81f72e0 con 0x7fa0a815d580 2024-03-18T20:23:45.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa0adcf6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0a81524f0 0x7fa0a815b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa0adcf6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0a81524f0 0x7fa0a815b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57902/0 (socket says 172.21.15.1:57902) 2024-03-18T20:23:45.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa0adcf6700 1 -- 172.21.15.1:0/1151006695 learned_addr learned my addr 172.21.15.1:0/1151006695 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:45.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa0accf4700 1 --2- 172.21.15.1:0/1151006695 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa0a815d580 0x7fa0a815aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa0ad4f5700 1 --2- 172.21.15.1:0/1151006695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa0a806d640 0x7fa0a81f6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa0adcf6700 1 -- 172.21.15.1:0/1151006695 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa0a815d580 msgr2=0x7fa0a815aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:45.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa0adcf6700 1 --2- 172.21.15.1:0/1151006695 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa0a815d580 0x7fa0a815aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa0adcf6700 1 -- 172.21.15.1:0/1151006695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa0a806d640 msgr2=0x7fa0a81f6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:45.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa0adcf6700 1 --2- 172.21.15.1:0/1151006695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa0a806d640 0x7fa0a81f6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:45.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa0adcf6700 1 -- 172.21.15.1:0/1151006695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a40094a0 con 0x7fa0a81524f0 2024-03-18T20:23:45.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa0accf4700 1 --2- 172.21.15.1:0/1151006695 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa0a815d580 0x7fa0a815aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:45.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa0adcf6700 1 --2- 172.21.15.1:0/1151006695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0a81524f0 0x7fa0a815b420 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0002230 tx=0x7fa0a000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:45.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa09dffb700 1 -- 172.21.15.1:0/1151006695 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a000ac20 con 0x7fa0a81524f0 2024-03-18T20:23:45.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa09dffb700 1 -- 172.21.15.1:0/1151006695 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a000ada0 con 0x7fa0a81524f0 2024-03-18T20:23:45.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa09dffb700 1 -- 172.21.15.1:0/1151006695 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0007310 con 0x7fa0a81524f0 2024-03-18T20:23:45.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.926+0000 7fa0af759700 1 -- 172.21.15.1:0/1151006695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a80b3b40 con 0x7fa0a81524f0 2024-03-18T20:23:45.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.927+0000 7fa0af759700 1 -- 172.21.15.1:0/1151006695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a815d960 con 0x7fa0a81524f0 2024-03-18T20:23:45.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.927+0000 7fa0af759700 1 -- 172.21.15.1:0/1151006695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a80b39a0 con 0x7fa0a81524f0 2024-03-18T20:23:45.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.928+0000 7fa09dffb700 1 -- 172.21.15.1:0/1151006695 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa0a0007470 con 0x7fa0a81524f0 2024-03-18T20:23:45.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.928+0000 7fa09dffb700 1 --2- 172.21.15.1:0/1151006695 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa08c06adb0 0x7fa08c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:45.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.929+0000 7fa09dffb700 1 -- 172.21.15.1:0/1151006695 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fa0a00892c0 con 0x7fa0a81524f0 2024-03-18T20:23:45.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.929+0000 7fa0ad4f5700 1 --2- 172.21.15.1:0/1151006695 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa08c06adb0 0x7fa08c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:45.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.929+0000 7fa0ad4f5700 1 --2- 172.21.15.1:0/1151006695 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa08c06adb0 0x7fa08c06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4009b90 tx=0x7fa0a4008030 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:45.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:45.933+0000 7fa09dffb700 1 -- 172.21.15.1:0/1151006695 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa0a80b39a0 con 0x7fa0a81524f0 2024-03-18T20:23:46.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:46.181+0000 7fa0af759700 1 -- 172.21.15.1:0/1151006695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa0a80b4060 con 0x7fa0a81524f0 2024-03-18T20:23:46.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:46.182+0000 7fa09dffb700 1 -- 172.21.15.1:0/1151006695 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v97) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa0a80b4060 con 0x7fa0a81524f0 2024-03-18T20:23:46.187 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T20:23:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:46.184+0000 7fa0af759700 1 -- 172.21.15.1:0/1151006695 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa08c06adb0 msgr2=0x7fa08c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:46.184+0000 7fa0af759700 1 --2- 172.21.15.1:0/1151006695 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa08c06adb0 0x7fa08c06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4009b90 tx=0x7fa0a4008030 comp rx=0 tx=0).stop 2024-03-18T20:23:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:46.184+0000 7fa0af759700 1 -- 172.21.15.1:0/1151006695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0a81524f0 msgr2=0x7fa0a815b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:46.184+0000 7fa0af759700 1 --2- 172.21.15.1:0/1151006695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0a81524f0 0x7fa0a815b420 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0002230 tx=0x7fa0a000a060 comp rx=0 tx=0).stop 2024-03-18T20:23:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:46.184+0000 7fa0af759700 1 -- 172.21.15.1:0/1151006695 shutdown_connections 2024-03-18T20:23:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:46.184+0000 7fa0af759700 1 --2- 172.21.15.1:0/1151006695 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa08c06adb0 0x7fa08c06d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:46.184+0000 7fa0af759700 1 --2- 172.21.15.1:0/1151006695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0a81524f0 0x7fa0a815b420 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:46.184+0000 7fa0af759700 1 --2- 172.21.15.1:0/1151006695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa0a806d640 0x7fa0a81f6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:46.184+0000 7fa0af759700 1 --2- 172.21.15.1:0/1151006695 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa0a815d580 0x7fa0a815aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:46.184+0000 7fa0af759700 1 -- 172.21.15.1:0/1151006695 >> 172.21.15.1:0/1151006695 conn(0x7fa0a80cc990 msgr2=0x7fa0a815c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:46.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:46.184+0000 7fa0af759700 1 -- 172.21.15.1:0/1151006695 shutdown_connections 2024-03-18T20:23:46.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:46.184+0000 7fa0af759700 1 -- 172.21.15.1:0/1151006695 wait complete. 2024-03-18T20:23:46.202 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T20:23:46.202 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T20:23:46.202 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-18T20:23:46.202 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T20:23:46.202 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T20:23:46.202 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T20:23:46.202 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T20:23:46.202 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T20:23:46.202 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:23:46.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:46.203 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-18T20:23:46.222 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T20:23:46.222 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T20:23:46.222 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T20:23:46.222 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:23:46.222 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T20:23:46.222 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:23:46.222 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:23:46.222 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-18T20:23:46.222 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-18T20:23:46.223 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:23:46.223 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-18T20:23:46.223 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-18T20:23:46.223 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T20:23:46.223 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:23:46.223 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-18T20:23:46.223 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T20:23:46.223 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 36943sec preferred_lft 36943sec 2024-03-18T20:23:46.223 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link 2024-03-18T20:23:46.223 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:23:46.223 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:23:46.223 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:23:46.223 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-18T20:23:46.224 INFO:teuthology.orchestra.run.smithi001.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T20:23:46.224 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:23:46.224 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T20:23:46.224 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:23:46.224 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::4ca9:f5ff:feb3:79a3/64 scope link 2024-03-18T20:23:46.224 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:23:46.225 INFO:teuthology.orchestra.run.smithi001.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T20:23:46.225 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T20:23:46.225 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::282f:a4ff:fe1a:9908/64 scope link 2024-03-18T20:23:46.225 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:23:46.225 INFO:teuthology.orchestra.run.smithi001.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T20:23:46.225 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2e:5c:60:78:b2:8c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T20:23:46.225 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::2c5c:60ff:fe78:b28c/64 scope link 2024-03-18T20:23:46.225 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:23:46.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:46.226 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T20:23:46.288 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-18T20:23:46.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:46.290 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-18T20:23:46.350 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-18T20:23:46.351 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T20:23:46.351 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T20:23:46.351 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T20:23:46.351 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T20:23:46.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:46.351 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T20:23:46.351 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T20:23:46.351 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T20:23:46.352 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T20:23:46.352 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T20:23:46.352 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T20:23:46.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:46.458 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T20:23:46.476 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T20:23:46.477 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T20:23:46.477 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:23:46.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:46.477 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-18T20:23:46.545 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T20:23:46.545 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:23:46.545 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T20:23:46.604 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:23:46.604 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:23:46.604 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T20:23:46.662 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T20:23:46.730 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:23:46.730 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:23:46.803 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:23:46.804 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:23:46.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:46.804 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T20:23:46.864 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T20:23:46.864 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T20:23:46.908 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T20:23:46.937 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:23:46.937 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:23:46.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-18T20:23:46.990+0000 7fe76cd045c0 -1 init, newargv = 0x559cfab89e90 newargc=15 2024-03-18T20:23:46.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[152682]: starting ceph client 2024-03-18T20:23:47.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[152682]: starting fuse 2024-03-18T20:23:47.007 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:23:47.008 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:23:47.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:47.008 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T20:23:47.069 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T20:23:47.069 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-03-18T20:23:47.069 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T20:23:47.069 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-18T20:23:47.069 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-18T20:23:47.070 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-18T20:23:47.070 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-18T20:23:47.070 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:23:47.070 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-18T20:23:47.070 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T20:23:47.070 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-18T20:23:47.070 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T20:23:47.070 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:23:47.070 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-18T20:23:47.070 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-18T20:23:47.070 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-18T20:23:47.070 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-18T20:23:47.070 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:23:47.070 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-18T20:23:47.070 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T20:23:47.071 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T20:23:47.071 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T20:23:47.071 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-18T20:23:47.071 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-18T20:23:47.071 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-18T20:23:47.071 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T20:23:47.071 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:23:47.071 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-18T20:23:47.071 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-18T20:23:47.196 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.152682.asok 2024-03-18T20:23:47.377 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.152682.asok 2024-03-18T20:23:47.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:47.377 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.152682.asok status 2024-03-18T20:23:47.539 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:23:47.539 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-18T20:23:47.539 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T20:23:47.539 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T20:23:47.539 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-18T20:23:47.539 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-18T20:23:47.539 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T20:23:47.539 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "152682", 2024-03-18T20:23:47.540 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-18T20:23:47.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:47.540 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-18T20:23:47.540 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-18T20:23:47.540 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 11181, 2024-03-18T20:23:47.540 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-18T20:23:47.540 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-18T20:23:47.540 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-18T20:23:47.540 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 11181 2024-03-18T20:23:47.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:47.540 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T20:23:47.540 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T20:23:47.540 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:23:47.540 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4201818502 2024-03-18T20:23:47.540 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:47.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:47.541 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T20:23:47.541 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T20:23:47.541 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:23:47.541 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4201818502 2024-03-18T20:23:47.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:23:47.541 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.11181 192.168.0.1:0/4201818502", 2024-03-18T20:23:47.541 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/4201818502", 2024-03-18T20:23:47.541 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-18T20:23:47.541 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 97, 2024-03-18T20:23:47.541 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 214, 2024-03-18T20:23:47.541 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-18T20:23:47.542 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-18T20:23:47.542 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-18T20:23:47.542 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:23:47.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:47.549 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-18T20:23:47.549 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T20:23:47.568 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-18T20:23:47.568 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T20:23:47.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:47.568 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T20:23:47.636 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-18T20:23:47.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.799+0000 7f87cce35700 1 Processor -- start 2024-03-18T20:23:47.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.799+0000 7f87cce35700 1 -- start start 2024-03-18T20:23:47.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.800+0000 7f87cce35700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87c815d780 0x7f87c80cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:47.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.800+0000 7f87cce35700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87c809dd40 0x7f87c80cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:47.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.800+0000 7f87cce35700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f87c80ce5a0 0x7f87c80cea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:47.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.800+0000 7f87cce35700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87c80ce090 con 0x7f87c815d780 2024-03-18T20:23:47.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.800+0000 7f87cce35700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87c806db30 con 0x7f87c80ce5a0 2024-03-18T20:23:47.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.800+0000 7f87cce35700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c806d520 con 0x7f87c809dd40 2024-03-18T20:23:47.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.800+0000 7f87c659c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87c815d780 0x7f87c80cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:47.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.800+0000 7f87c659c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f87c815d780 0x7f87c80cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57912/0 (socket says 172.21.15.1:57912) 2024-03-18T20:23:47.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.800+0000 7f87c659c700 1 -- 172.21.15.1:0/2971080090 learned_addr learned my addr 172.21.15.1:0/2971080090 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:47.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.801+0000 7f87c5d9b700 1 --2- 172.21.15.1:0/2971080090 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87c809dd40 0x7f87c80cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:47.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.801+0000 7f87c6d9d700 1 --2- 172.21.15.1:0/2971080090 >> v2:172.21.15.165:3300/0 conn(0x7f87c80ce5a0 0x7f87c80cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:47.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.801+0000 7f87c659c700 1 -- 172.21.15.1:0/2971080090 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87c809dd40 msgr2=0x7f87c80cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:47.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.801+0000 7f87c659c700 1 --2- 172.21.15.1:0/2971080090 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87c809dd40 0x7f87c80cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:47.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.801+0000 7f87c659c700 1 -- 172.21.15.1:0/2971080090 >> v2:172.21.15.165:3300/0 conn(0x7f87c80ce5a0 msgr2=0x7f87c80cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:47.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.801+0000 7f87c659c700 1 --2- 172.21.15.1:0/2971080090 >> v2:172.21.15.165:3300/0 conn(0x7f87c80ce5a0 0x7f87c80cea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:47.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.801+0000 7f87c659c700 1 -- 172.21.15.1:0/2971080090 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c806f460 con 0x7f87c815d780 2024-03-18T20:23:47.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.801+0000 7f87c5d9b700 1 --2- 172.21.15.1:0/2971080090 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87c809dd40 0x7f87c80cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:47.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.801+0000 7f87c6d9d700 1 --2- 172.21.15.1:0/2971080090 >> v2:172.21.15.165:3300/0 conn(0x7f87c80ce5a0 0x7f87c80cea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:47.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.801+0000 7f87c659c700 1 --2- 172.21.15.1:0/2971080090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87c815d780 0x7f87c80cdb50 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f87c815ed20 tx=0x7f87bc00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e230629227cfda server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:47.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.801+0000 7f87c559a700 1 -- 172.21.15.1:0/2971080090 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc00dd30 con 0x7f87c815d780 2024-03-18T20:23:47.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.801+0000 7f87c559a700 1 -- 172.21.15.1:0/2971080090 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87bc00de90 con 0x7f87c815d780 2024-03-18T20:23:47.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.801+0000 7f87c559a700 1 -- 172.21.15.1:0/2971080090 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc010690 con 0x7f87c815d780 2024-03-18T20:23:47.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.802+0000 7f87cce35700 1 -- 172.21.15.1:0/2971080090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87c815d780 msgr2=0x7f87c80cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:47.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.802+0000 7f87cce35700 1 --2- 172.21.15.1:0/2971080090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87c815d780 0x7f87c80cdb50 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f87c815ed20 tx=0x7f87bc00be70 comp rx=0 tx=0).stop 2024-03-18T20:23:47.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.802+0000 7f87cce35700 1 -- 172.21.15.1:0/2971080090 shutdown_connections 2024-03-18T20:23:47.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.802+0000 7f87cce35700 1 --2- 172.21.15.1:0/2971080090 >> v2:172.21.15.165:3300/0 conn(0x7f87c80ce5a0 0x7f87c80cea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:47.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.802+0000 7f87cce35700 1 --2- 172.21.15.1:0/2971080090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87c815d780 0x7f87c80cdb50 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:47.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.802+0000 7f87cce35700 1 --2- 172.21.15.1:0/2971080090 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87c809dd40 0x7f87c80cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:47.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.802+0000 7f87cce35700 1 -- 172.21.15.1:0/2971080090 >> 172.21.15.1:0/2971080090 conn(0x7f87c80cca00 msgr2=0x7f87c80cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:47.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.802+0000 7f87cce35700 1 -- 172.21.15.1:0/2971080090 shutdown_connections 2024-03-18T20:23:47.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.802+0000 7f87cce35700 1 -- 172.21.15.1:0/2971080090 wait complete. 2024-03-18T20:23:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.803+0000 7f87cce35700 1 Processor -- start 2024-03-18T20:23:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.803+0000 7f87cce35700 1 -- start start 2024-03-18T20:23:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.804+0000 7f87cce35700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87c809dd40 0x7f87c80c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.804+0000 7f87cce35700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87c80ce5a0 0x7f87c80acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.804+0000 7f87cce35700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87c815d780 0x7f87c814b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.804+0000 7f87cce35700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c80ad330 con 0x7f87c80ce5a0 2024-03-18T20:23:47.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.804+0000 7f87cce35700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c80ad5b0 con 0x7f87c809dd40 2024-03-18T20:23:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.804+0000 7f87cce35700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c80ad890 con 0x7f87c815d780 2024-03-18T20:23:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.804+0000 7f87c5d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87c80ce5a0 0x7f87c80acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.805+0000 7f87c5d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87c80ce5a0 0x7f87c80acdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57926/0 (socket says 172.21.15.1:57926) 2024-03-18T20:23:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.805+0000 7f87c5d9b700 1 -- 172.21.15.1:0/2347680259 learned_addr learned my addr 172.21.15.1:0/2347680259 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.805+0000 7f87c659c700 1 --2- 172.21.15.1:0/2347680259 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87c809dd40 0x7f87c80c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.805+0000 7f87c6d9d700 1 --2- 172.21.15.1:0/2347680259 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87c815d780 0x7f87c814b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.805+0000 7f87c5d9b700 1 -- 172.21.15.1:0/2347680259 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87c815d780 msgr2=0x7f87c814b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.805+0000 7f87c5d9b700 1 --2- 172.21.15.1:0/2347680259 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87c815d780 0x7f87c814b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.805+0000 7f87c5d9b700 1 -- 172.21.15.1:0/2347680259 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87c809dd40 msgr2=0x7f87c80c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.805+0000 7f87c5d9b700 1 --2- 172.21.15.1:0/2347680259 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87c809dd40 0x7f87c80c7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.805+0000 7f87c5d9b700 1 -- 172.21.15.1:0/2347680259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87bc00b7f0 con 0x7f87c80ce5a0 2024-03-18T20:23:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.805+0000 7f87c6d9d700 1 --2- 172.21.15.1:0/2347680259 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87c815d780 0x7f87c814b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:23:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.805+0000 7f87c5d9b700 1 --2- 172.21.15.1:0/2347680259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87c80ce5a0 0x7f87c80acdf0 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f87b800c7f0 tx=0x7f87b800cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.805+0000 7f87b6ffd700 1 -- 172.21.15.1:0/2347680259 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b8011800 con 0x7f87c80ce5a0 2024-03-18T20:23:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.805+0000 7f87b6ffd700 1 -- 172.21.15.1:0/2347680259 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b8011960 con 0x7f87c80ce5a0 2024-03-18T20:23:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.805+0000 7f87cce35700 1 -- 172.21.15.1:0/2347680259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c80adb70 con 0x7f87c80ce5a0 2024-03-18T20:23:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.805+0000 7f87b6ffd700 1 -- 172.21.15.1:0/2347680259 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b8011cf0 con 0x7f87c80ce5a0 2024-03-18T20:23:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.805+0000 7f87cce35700 1 -- 172.21.15.1:0/2347680259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c80ae130 con 0x7f87c80ce5a0 2024-03-18T20:23:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.806+0000 7f87cce35700 1 -- 172.21.15.1:0/2347680259 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c80ae350 con 0x7f87c80ce5a0 2024-03-18T20:23:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.807+0000 7f87b6ffd700 1 -- 172.21.15.1:0/2347680259 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f87b8018360 con 0x7f87c80ce5a0 2024-03-18T20:23:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.807+0000 7f87b6ffd700 1 --2- 172.21.15.1:0/2347680259 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f87a806ade0 0x7f87a806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.807+0000 7f87b6ffd700 1 -- 172.21.15.1:0/2347680259 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f87b8089660 con 0x7f87c80ce5a0 2024-03-18T20:23:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.807+0000 7f87c659c700 1 --2- 172.21.15.1:0/2347680259 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f87a806ade0 0x7f87a806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:47.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.808+0000 7f87c659c700 1 --2- 172.21.15.1:0/2347680259 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f87a806ade0 0x7f87a806d2a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f87bc00beb0 tx=0x7f87bc012000 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:47.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:47.812+0000 7f87b6ffd700 1 -- 172.21.15.1:0/2347680259 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f87b8054480 con 0x7f87c80ce5a0 2024-03-18T20:23:48.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.043+0000 7f87cce35700 1 -- 172.21.15.1:0/2347680259 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f87c80cef90 con 0x7f87a806ade0 2024-03-18T20:23:48.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.044+0000 7f87b6ffd700 1 -- 172.21.15.1:0/2347680259 <== mgr.10797 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f87bc020040 con 0x7f87a806ade0 2024-03-18T20:23:48.049 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T20:23:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:23:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephfs" 2024-03-18T20:23:48.049 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:23:48.050 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T20:23:48.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.046+0000 7f87cce35700 1 -- 172.21.15.1:0/2347680259 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f87a806ade0 msgr2=0x7f87a806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:48.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.046+0000 7f87cce35700 1 --2- 172.21.15.1:0/2347680259 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f87a806ade0 0x7f87a806d2a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f87bc00beb0 tx=0x7f87bc012000 comp rx=0 tx=0).stop 2024-03-18T20:23:48.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.046+0000 7f87cce35700 1 -- 172.21.15.1:0/2347680259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87c80ce5a0 msgr2=0x7f87c80acdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:48.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.046+0000 7f87cce35700 1 --2- 172.21.15.1:0/2347680259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87c80ce5a0 0x7f87c80acdf0 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f87b800c7f0 tx=0x7f87b800cb00 comp rx=0 tx=0).stop 2024-03-18T20:23:48.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.046+0000 7f87cce35700 1 -- 172.21.15.1:0/2347680259 shutdown_connections 2024-03-18T20:23:48.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.046+0000 7f87cce35700 1 --2- 172.21.15.1:0/2347680259 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f87a806ade0 0x7f87a806d2a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:48.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.046+0000 7f87cce35700 1 --2- 172.21.15.1:0/2347680259 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f87c815d780 0x7f87c814b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:48.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.046+0000 7f87cce35700 1 --2- 172.21.15.1:0/2347680259 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87c809dd40 0x7f87c80c7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:48.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.046+0000 7f87cce35700 1 --2- 172.21.15.1:0/2347680259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f87c80ce5a0 0x7f87c80acdf0 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:48.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.046+0000 7f87cce35700 1 -- 172.21.15.1:0/2347680259 >> 172.21.15.1:0/2347680259 conn(0x7f87c80cca00 msgr2=0x7f87c80c2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:48.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.046+0000 7f87cce35700 1 -- 172.21.15.1:0/2347680259 shutdown_connections 2024-03-18T20:23:48.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.046+0000 7f87cce35700 1 -- 172.21.15.1:0/2347680259 wait complete. 2024-03-18T20:23:48.064 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-18T20:23:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.185+0000 7f1e5542c700 1 Processor -- start 2024-03-18T20:23:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.185+0000 7f1e5542c700 1 -- start start 2024-03-18T20:23:48.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.186+0000 7f1e5542c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e500cfa10 0x7f1e50168490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.186+0000 7f1e5542c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e50142fc0 con 0x7f1e500b1e80 2024-03-18T20:23:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.186+0000 7f1e5542c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e500cfff0 con 0x7f1e500cb450 2024-03-18T20:23:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.186+0000 7f1e5542c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e50140cb0 con 0x7f1e500cfa10 2024-03-18T20:23:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.186+0000 7f1e4effd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1e500b1e80 0x7f1e500cad40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55724/0 (socket says 172.21.15.1:55724) 2024-03-18T20:23:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.186+0000 7f1e4effd700 1 -- 172.21.15.1:0/537637975 learned_addr learned my addr 172.21.15.1:0/537637975 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.186+0000 7f1e4f7fe700 1 --2- 172.21.15.1:0/537637975 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e500cfa10 0x7f1e50168490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.186+0000 7f1e4dffb700 1 -- 172.21.15.1:0/537637975 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2333279783 0 0) 0x7f1e50142fc0 con 0x7f1e500b1e80 2024-03-18T20:23:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.187+0000 7f1e4dffb700 1 -- 172.21.15.1:0/537637975 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e380035b0 con 0x7f1e500b1e80 2024-03-18T20:23:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.187+0000 7f1e4dffb700 1 -- 172.21.15.1:0/537637975 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121086934 0 0) 0x7f1e500cfff0 con 0x7f1e500cb450 2024-03-18T20:23:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.187+0000 7f1e4dffb700 1 -- 172.21.15.1:0/537637975 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e50142fc0 con 0x7f1e500cb450 2024-03-18T20:23:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.187+0000 7f1e4f7fe700 1 -- 172.21.15.1:0/537637975 >> v1:172.21.15.165:6789/0 conn(0x7f1e500cb450 legacy=0x7f1e500cf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.187+0000 7f1e4dffb700 1 -- 172.21.15.1:0/537637975 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 979414219 0 0) 0x7f1e380035b0 con 0x7f1e500b1e80 2024-03-18T20:23:48.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.187+0000 7f1e4f7fe700 1 -- 172.21.15.1:0/537637975 >> v1:172.21.15.1:6789/0 conn(0x7f1e500b1e80 legacy=0x7f1e500cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.187+0000 7f1e4f7fe700 1 -- 172.21.15.1:0/537637975 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e50143210 con 0x7f1e500cfa10 2024-03-18T20:23:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.187+0000 7f1e4f7fe700 1 --2- 172.21.15.1:0/537637975 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e500cfa10 0x7f1e50168490 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1e48009dc0 tx=0x7f1e4800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8efcec5fc2139051 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.187+0000 7f1e4dffb700 1 -- 172.21.15.1:0/537637975 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4800e070 con 0x7f1e500cfa10 2024-03-18T20:23:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.187+0000 7f1e4dffb700 1 -- 172.21.15.1:0/537637975 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e48003a20 con 0x7f1e500cfa10 2024-03-18T20:23:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.187+0000 7f1e4dffb700 1 -- 172.21.15.1:0/537637975 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e40001670 con 0x7f1e500cfa10 2024-03-18T20:23:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.188+0000 7f1e5542c700 1 -- 172.21.15.1:0/537637975 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e500cfa10 msgr2=0x7f1e50168490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.188+0000 7f1e5542c700 1 --2- 172.21.15.1:0/537637975 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e500cfa10 0x7f1e50168490 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1e48009dc0 tx=0x7f1e4800b330 comp rx=0 tx=0).stop 2024-03-18T20:23:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.188+0000 7f1e5542c700 1 -- 172.21.15.1:0/537637975 shutdown_connections 2024-03-18T20:23:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.188+0000 7f1e5542c700 1 --2- 172.21.15.1:0/537637975 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e500cfa10 0x7f1e50168490 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.188+0000 7f1e5542c700 1 -- 172.21.15.1:0/537637975 >> 172.21.15.1:0/537637975 conn(0x7f1e50066c80 msgr2=0x7f1e500b2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.188+0000 7f1e5542c700 1 -- 172.21.15.1:0/537637975 shutdown_connections 2024-03-18T20:23:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.188+0000 7f1e5542c700 1 -- 172.21.15.1:0/537637975 wait complete. 2024-03-18T20:23:48.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.189+0000 7f1e5542c700 1 Processor -- start 2024-03-18T20:23:48.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.189+0000 7f1e5542c700 1 -- start start 2024-03-18T20:23:48.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.190+0000 7f1e5542c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e500b1e80 0x7f1e501453f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.190+0000 7f1e5542c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e500cb450 0x7f1e5013f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.190+0000 7f1e5542c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e500cfa10 0x7f1e5013b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.190+0000 7f1e5542c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e500cd970 con 0x7f1e500cb450 2024-03-18T20:23:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.190+0000 7f1e5542c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e500cdc50 con 0x7f1e500cfa10 2024-03-18T20:23:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.190+0000 7f1e5542c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e500cdf30 con 0x7f1e500b1e80 2024-03-18T20:23:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.190+0000 7f1e4e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e500cb450 0x7f1e5013f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.190+0000 7f1e4e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e500cb450 0x7f1e5013f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57928/0 (socket says 172.21.15.1:57928) 2024-03-18T20:23:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.190+0000 7f1e4e7fc700 1 -- 172.21.15.1:0/2126181441 learned_addr learned my addr 172.21.15.1:0/2126181441 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.190+0000 7f1e4effd700 1 --2- 172.21.15.1:0/2126181441 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e500b1e80 0x7f1e501453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.190+0000 7f1e4f7fe700 1 --2- 172.21.15.1:0/2126181441 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e500cfa10 0x7f1e5013b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.191+0000 7f1e4e7fc700 1 -- 172.21.15.1:0/2126181441 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e500b1e80 msgr2=0x7f1e501453f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.191+0000 7f1e4e7fc700 1 --2- 172.21.15.1:0/2126181441 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e500b1e80 0x7f1e501453f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.191+0000 7f1e4e7fc700 1 -- 172.21.15.1:0/2126181441 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e500cfa10 msgr2=0x7f1e5013b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:48.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.191+0000 7f1e4e7fc700 1 --2- 172.21.15.1:0/2126181441 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e500cfa10 0x7f1e5013b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:48.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.191+0000 7f1e4e7fc700 1 -- 172.21.15.1:0/2126181441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e48009a70 con 0x7f1e500cb450 2024-03-18T20:23:48.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.191+0000 7f1e4e7fc700 1 --2- 172.21.15.1:0/2126181441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e500cb450 0x7f1e5013f1e0 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f1e4400c520 tx=0x7f1e4400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:48.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.191+0000 7f1e377fe700 1 -- 172.21.15.1:0/2126181441 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e440052d0 con 0x7f1e500cb450 2024-03-18T20:23:48.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.191+0000 7f1e377fe700 1 -- 172.21.15.1:0/2126181441 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e44005430 con 0x7f1e500cb450 2024-03-18T20:23:48.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.191+0000 7f1e5542c700 1 -- 172.21.15.1:0/2126181441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e500ce210 con 0x7f1e500cb450 2024-03-18T20:23:48.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.191+0000 7f1e377fe700 1 -- 172.21.15.1:0/2126181441 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e440038c0 con 0x7f1e500cb450 2024-03-18T20:23:48.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.191+0000 7f1e5542c700 1 -- 172.21.15.1:0/2126181441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e5013f720 con 0x7f1e500cb450 2024-03-18T20:23:48.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.192+0000 7f1e5542c700 1 -- 172.21.15.1:0/2126181441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e500ce450 con 0x7f1e500cb450 2024-03-18T20:23:48.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.192+0000 7f1e377fe700 1 -- 172.21.15.1:0/2126181441 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 123) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1e44003a20 con 0x7f1e500cb450 2024-03-18T20:23:48.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.193+0000 7f1e377fe700 1 --2- 172.21.15.1:0/2126181441 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e3c06adb0 0x7f1e3c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:48.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.193+0000 7f1e377fe700 1 -- 172.21.15.1:0/2126181441 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f1e44088ac0 con 0x7f1e500cb450 2024-03-18T20:23:48.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.193+0000 7f1e4effd700 1 --2- 172.21.15.1:0/2126181441 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e3c06adb0 0x7f1e3c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:48.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.193+0000 7f1e4effd700 1 --2- 172.21.15.1:0/2126181441 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e3c06adb0 0x7f1e3c06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1e40000f60 tx=0x7f1e40000bc0 comp rx=0 tx=0).ready entity=mgr.10797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:48.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.197+0000 7f1e377fe700 1 -- 172.21.15.1:0/2126181441 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1e500ce450 con 0x7f1e500cb450 2024-03-18T20:23:48.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:48.459+0000 7f1e5542c700 1 -- 172.21.15.1:0/2126181441 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f1e500ce450 con 0x7f1e500cb450 2024-03-18T20:23:49.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.080+0000 7f1e377fe700 1 -- 172.21.15.1:0/2126181441 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 124) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f1e44086e30 con 0x7f1e500cb450 2024-03-18T20:23:49.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.081+0000 7f1e377fe700 1 -- 172.21.15.1:0/2126181441 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v124) v1 ==== 92+0+0 (secure 0 0 0) 0x7f1e44073210 con 0x7f1e500cb450 2024-03-18T20:23:49.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.085+0000 7f1e5542c700 1 -- 172.21.15.1:0/2126181441 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e3c06adb0 msgr2=0x7f1e3c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:49.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.085+0000 7f1e5542c700 1 --2- 172.21.15.1:0/2126181441 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e3c06adb0 0x7f1e3c06d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1e40000f60 tx=0x7f1e40000bc0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.085+0000 7f1e5542c700 1 -- 172.21.15.1:0/2126181441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e500cb450 msgr2=0x7f1e5013f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:49.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.085+0000 7f1e5542c700 1 --2- 172.21.15.1:0/2126181441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e500cb450 0x7f1e5013f1e0 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f1e4400c520 tx=0x7f1e4400c830 comp rx=0 tx=0).stop 2024-03-18T20:23:49.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.086+0000 7f1e5542c700 1 -- 172.21.15.1:0/2126181441 shutdown_connections 2024-03-18T20:23:49.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.086+0000 7f1e5542c700 1 --2- 172.21.15.1:0/2126181441 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e3c06adb0 0x7f1e3c06d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.086+0000 7f1e5542c700 1 --2- 172.21.15.1:0/2126181441 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e500b1e80 0x7f1e501453f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.086+0000 7f1e5542c700 1 --2- 172.21.15.1:0/2126181441 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e500cfa10 0x7f1e5013b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.086+0000 7f1e5542c700 1 --2- 172.21.15.1:0/2126181441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e500cb450 0x7f1e5013f1e0 unknown :-1 s=CLOSED pgs=2366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.086+0000 7f1e5542c700 1 -- 172.21.15.1:0/2126181441 >> 172.21.15.1:0/2126181441 conn(0x7f1e50066c80 msgr2=0x7f1e50141a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:49.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.086+0000 7f1e5542c700 1 -- 172.21.15.1:0/2126181441 shutdown_connections 2024-03-18T20:23:49.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.086+0000 7f1e5542c700 1 -- 172.21.15.1:0/2126181441 wait complete. 2024-03-18T20:23:49.111 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-18T20:23:49.229 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:49.224+0000 7f80b98a8240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:23:49.247 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:49.242+0000 7f4705b3b240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:23:49.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.244+0000 7f3746728700 1 Processor -- start 2024-03-18T20:23:49.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.244+0000 7f3746728700 1 -- start start 2024-03-18T20:23:49.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.244+0000 7f3746728700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f374015ad70 0x7f374015d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:49.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.244+0000 7f3746728700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f37400727b0 0x7f3740161610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:49.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.244+0000 7f3746728700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3740160ea0 con 0x7f3740161b50 2024-03-18T20:23:49.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.244+0000 7f3746728700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37400725b0 con 0x7f374015ad70 2024-03-18T20:23:49.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.244+0000 7f3746728700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f374015ddf0 con 0x7f37400727b0 2024-03-18T20:23:49.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.245+0000 7f373ffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f374015ad70 0x7f374015d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:49.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.245+0000 7f3744cc5700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3740161b50 0x7f374006a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55736/0 (socket says 172.21.15.1:55736) 2024-03-18T20:23:49.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.245+0000 7f3744cc5700 1 -- 172.21.15.1:0/858843927 learned_addr learned my addr 172.21.15.1:0/858843927 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:49.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.245+0000 7f373f7fe700 1 --2- 172.21.15.1:0/858843927 >> v2:172.21.15.165:3300/0 conn(0x7f37400727b0 0x7f3740161610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:49.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.245+0000 7f373ffff700 1 -- 172.21.15.1:0/858843927 >> v2:172.21.15.165:3300/0 conn(0x7f37400727b0 msgr2=0x7f3740161610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:49.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.245+0000 7f373ffff700 1 --2- 172.21.15.1:0/858843927 >> v2:172.21.15.165:3300/0 conn(0x7f37400727b0 0x7f3740161610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.245+0000 7f373ffff700 1 -- 172.21.15.1:0/858843927 >> v1:172.21.15.1:6789/0 conn(0x7f3740161b50 legacy=0x7f374006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:49.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.245+0000 7f373ffff700 1 -- 172.21.15.1:0/858843927 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374015e090 con 0x7f374015ad70 2024-03-18T20:23:49.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.245+0000 7f373ffff700 1 --2- 172.21.15.1:0/858843927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f374015ad70 0x7f374015d8b0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f3740071600 tx=0x7f3730009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708fe9f42e53383c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:49.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.245+0000 7f373effd700 1 -- 172.21.15.1:0/858843927 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3730011830 con 0x7f374015ad70 2024-03-18T20:23:49.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.245+0000 7f373effd700 1 -- 172.21.15.1:0/858843927 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3730011990 con 0x7f374015ad70 2024-03-18T20:23:49.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.245+0000 7f373effd700 1 -- 172.21.15.1:0/858843927 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3730011d20 con 0x7f374015ad70 2024-03-18T20:23:49.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.246+0000 7f3746728700 1 -- 172.21.15.1:0/858843927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f374015ad70 msgr2=0x7f374015d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:49.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.246+0000 7f3746728700 1 --2- 172.21.15.1:0/858843927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f374015ad70 0x7f374015d8b0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f3740071600 tx=0x7f3730009b20 comp rx=0 tx=0).stop 2024-03-18T20:23:49.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.246+0000 7f3746728700 1 -- 172.21.15.1:0/858843927 shutdown_connections 2024-03-18T20:23:49.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.246+0000 7f3746728700 1 --2- 172.21.15.1:0/858843927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f374015ad70 0x7f374015d8b0 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.246+0000 7f3746728700 1 --2- 172.21.15.1:0/858843927 >> v2:172.21.15.165:3300/0 conn(0x7f37400727b0 0x7f3740161610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.246+0000 7f3746728700 1 -- 172.21.15.1:0/858843927 >> 172.21.15.1:0/858843927 conn(0x7f37400b1cb0 msgr2=0x7f37400b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:49.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.246+0000 7f3746728700 1 -- 172.21.15.1:0/858843927 shutdown_connections 2024-03-18T20:23:49.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.246+0000 7f3746728700 1 -- 172.21.15.1:0/858843927 wait complete. 2024-03-18T20:23:49.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.248+0000 7f3746728700 1 Processor -- start 2024-03-18T20:23:49.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.248+0000 7f3746728700 1 -- start start 2024-03-18T20:23:49.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.249+0000 7f3746728700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f37400727b0 0x7f3740145570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:49.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.249+0000 7f3746728700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f374015ad70 0x7f3740145ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:49.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.249+0000 7f3746728700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3740161b50 0x7f374015f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.249+0000 7f3746728700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37400af850 con 0x7f374015ad70 2024-03-18T20:23:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.249+0000 7f3746728700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37400aecd0 con 0x7f3740161b50 2024-03-18T20:23:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.249+0000 7f3746728700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37400aefb0 con 0x7f37400727b0 2024-03-18T20:23:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.250+0000 7f373f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f374015ad70 0x7f3740145ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.250+0000 7f373f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f374015ad70 0x7f3740145ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57962/0 (socket says 172.21.15.1:57962) 2024-03-18T20:23:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.250+0000 7f373f7fe700 1 -- 172.21.15.1:0/4100118874 learned_addr learned my addr 172.21.15.1:0/4100118874 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.250+0000 7f3744cc5700 1 --2- 172.21.15.1:0/4100118874 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3740161b50 0x7f374015f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.250+0000 7f373ffff700 1 --2- 172.21.15.1:0/4100118874 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f37400727b0 0x7f3740145570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.250+0000 7f373f7fe700 1 -- 172.21.15.1:0/4100118874 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f37400727b0 msgr2=0x7f3740145570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.250+0000 7f373f7fe700 1 --2- 172.21.15.1:0/4100118874 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f37400727b0 0x7f3740145570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.250+0000 7f373f7fe700 1 -- 172.21.15.1:0/4100118874 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3740161b50 msgr2=0x7f374015f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.250+0000 7f373f7fe700 1 --2- 172.21.15.1:0/4100118874 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3740161b50 0x7f374015f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.250+0000 7f373f7fe700 1 -- 172.21.15.1:0/4100118874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37300094a0 con 0x7f374015ad70 2024-03-18T20:23:49.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.250+0000 7f373f7fe700 1 --2- 172.21.15.1:0/4100118874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f374015ad70 0x7f3740145ab0 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f373400e9b0 tx=0x7f373400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:49.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.250+0000 7f373cff9700 1 -- 172.21.15.1:0/4100118874 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3734009710 con 0x7f374015ad70 2024-03-18T20:23:49.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.250+0000 7f373cff9700 1 -- 172.21.15.1:0/4100118874 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373400ee50 con 0x7f374015ad70 2024-03-18T20:23:49.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.250+0000 7f3746728700 1 -- 172.21.15.1:0/4100118874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37400af290 con 0x7f374015ad70 2024-03-18T20:23:49.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.250+0000 7f373cff9700 1 -- 172.21.15.1:0/4100118874 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373400ca50 con 0x7f374015ad70 2024-03-18T20:23:49.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.250+0000 7f3746728700 1 -- 172.21.15.1:0/4100118874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3740159d40 con 0x7f374015ad70 2024-03-18T20:23:49.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.251+0000 7f3746728700 1 -- 172.21.15.1:0/4100118874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37400676a0 con 0x7f374015ad70 2024-03-18T20:23:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.252+0000 7f373cff9700 1 -- 172.21.15.1:0/4100118874 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 124) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f373400cbb0 con 0x7f374015ad70 2024-03-18T20:23:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.252+0000 7f373cff9700 1 --2- 172.21.15.1:0/4100118874 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f372c06ae00 0x7f372c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.252+0000 7f373ffff700 1 -- 172.21.15.1:0/4100118874 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f372c06ae00 msgr2=0x7f372c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.252+0000 7f373cff9700 1 -- 172.21.15.1:0/4100118874 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f3734089650 con 0x7f374015ad70 2024-03-18T20:23:49.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.253+0000 7f373ffff700 1 --2- 172.21.15.1:0/4100118874 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f372c06ae00 0x7f372c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:23:49.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.257+0000 7f373cff9700 1 -- 172.21.15.1:0/4100118874 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3734054570 con 0x7f374015ad70 2024-03-18T20:23:49.374 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:49.369+0000 7f80b98a8240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:23:49.417 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:49.412+0000 7f4705b3b240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:23:49.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.454+0000 7f373ffff700 1 -- 172.21.15.1:0/4100118874 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f372c06ae00 msgr2=0x7f372c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:49.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.454+0000 7f373ffff700 1 --2- 172.21.15.1:0/4100118874 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f372c06ae00 0x7f372c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:23:49.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.506+0000 7f3746728700 1 -- 172.21.15.1:0/4100118874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f3740071da0 con 0x7f374015ad70 2024-03-18T20:23:49.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.510+0000 7f373cff9700 1 -- 172.21.15.1:0/4100118874 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v71)=0 v71) v1 ==== 138+0+0 (secure 0 0 0) 0x7f373400ce60 con 0x7f374015ad70 2024-03-18T20:23:49.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.513+0000 7f3746728700 1 -- 172.21.15.1:0/4100118874 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f372c06ae00 msgr2=0x7f372c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:23:49.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.513+0000 7f3746728700 1 --2- 172.21.15.1:0/4100118874 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f372c06ae00 0x7f372c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.513+0000 7f3746728700 1 -- 172.21.15.1:0/4100118874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f374015ad70 msgr2=0x7f3740145ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:49.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.513+0000 7f3746728700 1 --2- 172.21.15.1:0/4100118874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f374015ad70 0x7f3740145ab0 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f373400e9b0 tx=0x7f373400ecc0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.513+0000 7f3746728700 1 -- 172.21.15.1:0/4100118874 shutdown_connections 2024-03-18T20:23:49.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.513+0000 7f3746728700 1 --2- 172.21.15.1:0/4100118874 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f372c06ae00 0x7f372c06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.513+0000 7f3746728700 1 --2- 172.21.15.1:0/4100118874 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f37400727b0 0x7f3740145570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.513+0000 7f3746728700 1 --2- 172.21.15.1:0/4100118874 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3740161b50 0x7f374015f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.513+0000 7f3746728700 1 --2- 172.21.15.1:0/4100118874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f374015ad70 0x7f3740145ab0 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.513+0000 7f3746728700 1 -- 172.21.15.1:0/4100118874 >> 172.21.15.1:0/4100118874 conn(0x7f37400b1cb0 msgr2=0x7f374006fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:49.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.513+0000 7f3746728700 1 -- 172.21.15.1:0/4100118874 shutdown_connections 2024-03-18T20:23:49.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.513+0000 7f3746728700 1 -- 172.21.15.1:0/4100118874 wait complete. 2024-03-18T20:23:49.536 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-18T20:23:49.639 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:49.634+0000 7f80b98a8240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:23:49.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.660+0000 7f36f8d90700 1 Processor -- start 2024-03-18T20:23:49.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.660+0000 7f36f8d90700 1 -- start start 2024-03-18T20:23:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.661+0000 7f36f8d90700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f36f4099a40 0x7f36f40ade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.661+0000 7f36f8d90700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f36f406fa50 0x7f36f406fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.661+0000 7f36f8d90700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f40b3aa0 con 0x7f36f4099e20 2024-03-18T20:23:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.661+0000 7f36f8d90700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36f406e9b0 con 0x7f36f4099a40 2024-03-18T20:23:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.661+0000 7f36f8d90700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36f406ec90 con 0x7f36f406fa50 2024-03-18T20:23:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.661+0000 7f36f259c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f36f4099a40 0x7f36f40ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.661+0000 7f36f1d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f36f4099e20 0x7f36f40cc6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55738/0 (socket says 172.21.15.1:55738) 2024-03-18T20:23:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.661+0000 7f36f1d9b700 1 -- 172.21.15.1:0/3932803549 learned_addr learned my addr 172.21.15.1:0/3932803549 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:49.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.661+0000 7f36f2d9d700 1 --2- 172.21.15.1:0/3932803549 >> v2:172.21.15.165:3300/0 conn(0x7f36f406fa50 0x7f36f406fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:49.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.661+0000 7f36f259c700 1 -- 172.21.15.1:0/3932803549 >> v2:172.21.15.165:3300/0 conn(0x7f36f406fa50 msgr2=0x7f36f406fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:23:49.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.661+0000 7f36f259c700 1 --2- 172.21.15.1:0/3932803549 >> v2:172.21.15.165:3300/0 conn(0x7f36f406fa50 0x7f36f406fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.661+0000 7f36f259c700 1 -- 172.21.15.1:0/3932803549 >> v1:172.21.15.1:6789/0 conn(0x7f36f4099e20 legacy=0x7f36f40cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:49.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.661+0000 7f36f259c700 1 -- 172.21.15.1:0/3932803549 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f406ef70 con 0x7f36f4099a40 2024-03-18T20:23:49.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.661+0000 7f36f159a700 1 -- 172.21.15.1:0/3932803549 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774473454 0 0) 0x7f36f40b3aa0 con 0x7f36f4099e20 2024-03-18T20:23:49.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.661+0000 7f36f259c700 1 --2- 172.21.15.1:0/3932803549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36f4099a40 0x7f36f40ade00 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f36f4067140 tx=0x7f36e4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f4875fbff5366c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:49.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.661+0000 7f36f159a700 1 -- 172.21.15.1:0/3932803549 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e4011830 con 0x7f36f4099a40 2024-03-18T20:23:49.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.662+0000 7f36f159a700 1 -- 172.21.15.1:0/3932803549 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e4011990 con 0x7f36f4099a40 2024-03-18T20:23:49.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.662+0000 7f36f159a700 1 -- 172.21.15.1:0/3932803549 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e4011d20 con 0x7f36f4099a40 2024-03-18T20:23:49.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.662+0000 7f36f8d90700 1 -- 172.21.15.1:0/3932803549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36f4099a40 msgr2=0x7f36f40ade00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:49.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.662+0000 7f36f8d90700 1 --2- 172.21.15.1:0/3932803549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36f4099a40 0x7f36f40ade00 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f36f4067140 tx=0x7f36e4009b20 comp rx=0 tx=0).stop 2024-03-18T20:23:49.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.662+0000 7f36f8d90700 1 -- 172.21.15.1:0/3932803549 shutdown_connections 2024-03-18T20:23:49.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.662+0000 7f36f8d90700 1 --2- 172.21.15.1:0/3932803549 >> v2:172.21.15.165:3300/0 conn(0x7f36f406fa50 0x7f36f406fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.662+0000 7f36f8d90700 1 --2- 172.21.15.1:0/3932803549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36f4099a40 0x7f36f40ade00 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.662+0000 7f36f8d90700 1 -- 172.21.15.1:0/3932803549 >> 172.21.15.1:0/3932803549 conn(0x7f36f40b2c50 msgr2=0x7f36f40b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:49.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.662+0000 7f36f8d90700 1 -- 172.21.15.1:0/3932803549 shutdown_connections 2024-03-18T20:23:49.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.662+0000 7f36f8d90700 1 -- 172.21.15.1:0/3932803549 wait complete. 2024-03-18T20:23:49.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.664+0000 7f36f8d90700 1 Processor -- start 2024-03-18T20:23:49.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.664+0000 7f36f8d90700 1 -- start start 2024-03-18T20:23:49.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.664+0000 7f36f8d90700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f36f406fa50 0x7f36f40b67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:49.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.664+0000 7f36f8d90700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36f4099a40 0x7f36f40c6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:49.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.664+0000 7f36f8d90700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36f4099e20 0x7f36f40ca650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:49.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.664+0000 7f36f8d90700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f406cf50 con 0x7f36f4099a40 2024-03-18T20:23:49.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.664+0000 7f36f8d90700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f415a1f0 con 0x7f36f4099e20 2024-03-18T20:23:49.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.664+0000 7f36f8d90700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f40c74c0 con 0x7f36f406fa50 2024-03-18T20:23:49.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.665+0000 7f36f1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36f4099a40 0x7f36f40c6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:49.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.665+0000 7f36f1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36f4099a40 0x7f36f40c6f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57972/0 (socket says 172.21.15.1:57972) 2024-03-18T20:23:49.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.665+0000 7f36f1d9b700 1 -- 172.21.15.1:0/1036542878 learned_addr learned my addr 172.21.15.1:0/1036542878 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:49.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.665+0000 7f36f259c700 1 --2- 172.21.15.1:0/1036542878 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f36f406fa50 0x7f36f40b67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:49.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.665+0000 7f36f2d9d700 1 --2- 172.21.15.1:0/1036542878 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36f4099e20 0x7f36f40ca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:49.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.665+0000 7f36f1d9b700 1 -- 172.21.15.1:0/1036542878 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f36f406fa50 msgr2=0x7f36f40b67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:49.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.665+0000 7f36f1d9b700 1 --2- 172.21.15.1:0/1036542878 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f36f406fa50 0x7f36f40b67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.665+0000 7f36f1d9b700 1 -- 172.21.15.1:0/1036542878 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36f4099e20 msgr2=0x7f36f40ca650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:49.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.665+0000 7f36f1d9b700 1 --2- 172.21.15.1:0/1036542878 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36f4099e20 0x7f36f40ca650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.665+0000 7f36f1d9b700 1 -- 172.21.15.1:0/1036542878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e40094a0 con 0x7f36f4099a40 2024-03-18T20:23:49.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.665+0000 7f36f1d9b700 1 --2- 172.21.15.1:0/1036542878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36f4099a40 0x7f36f40c6f80 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f36e80011e0 tx=0x7f36e800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:49.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.665+0000 7f36e2ffd700 1 -- 172.21.15.1:0/1036542878 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e8004c80 con 0x7f36f4099a40 2024-03-18T20:23:49.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.665+0000 7f36e2ffd700 1 -- 172.21.15.1:0/1036542878 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e8004e00 con 0x7f36f4099a40 2024-03-18T20:23:49.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.665+0000 7f36f8d90700 1 -- 172.21.15.1:0/1036542878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f406c310 con 0x7f36f4099a40 2024-03-18T20:23:49.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.665+0000 7f36e2ffd700 1 -- 172.21.15.1:0/1036542878 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e8007310 con 0x7f36f4099a40 2024-03-18T20:23:49.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.665+0000 7f36f8d90700 1 -- 172.21.15.1:0/1036542878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f406c890 con 0x7f36f4099a40 2024-03-18T20:23:49.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.667+0000 7f36f8d90700 1 -- 172.21.15.1:0/1036542878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f406d090 con 0x7f36f4099a40 2024-03-18T20:23:49.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.667+0000 7f36e2ffd700 1 -- 172.21.15.1:0/1036542878 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 124) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f36e8007470 con 0x7f36f4099a40 2024-03-18T20:23:49.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.667+0000 7f36e2ffd700 1 --2- 172.21.15.1:0/1036542878 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f36d406ae00 0x7f36d406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:49.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.668+0000 7f36f259c700 1 -- 172.21.15.1:0/1036542878 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f36d406ae00 msgr2=0x7f36d406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:49.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.668+0000 7f36e2ffd700 1 -- 172.21.15.1:0/1036542878 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f36e80892e0 con 0x7f36f4099a40 2024-03-18T20:23:49.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.668+0000 7f36f259c700 1 --2- 172.21.15.1:0/1036542878 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f36d406ae00 0x7f36d406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:23:49.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.672+0000 7f36e2ffd700 1 -- 172.21.15.1:0/1036542878 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f36f406d090 con 0x7f36f4099a40 2024-03-18T20:23:49.710 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:49.705+0000 7f4705b3b240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:23:49.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.869+0000 7f36f259c700 1 -- 172.21.15.1:0/1036542878 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f36d406ae00 msgr2=0x7f36d406d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:49.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.869+0000 7f36f259c700 1 --2- 172.21.15.1:0/1036542878 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f36d406ae00 0x7f36d406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:23:49.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.881+0000 7f36f8d90700 1 -- 172.21.15.1:0/1036542878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f36f406d090 con 0x7f36f4099a40 2024-03-18T20:23:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.885+0000 7f36e2ffd700 1 -- 172.21.15.1:0/1036542878 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v72)=0 v72) v1 ==== 133+0+0 (secure 0 0 0) 0x7f36f406d090 con 0x7f36f4099a40 2024-03-18T20:23:49.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.887+0000 7f36f8d90700 1 -- 172.21.15.1:0/1036542878 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f36d406ae00 msgr2=0x7f36d406d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:23:49.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.887+0000 7f36f8d90700 1 --2- 172.21.15.1:0/1036542878 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f36d406ae00 0x7f36d406d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.887+0000 7f36f8d90700 1 -- 172.21.15.1:0/1036542878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36f4099a40 msgr2=0x7f36f40c6f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:49.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.887+0000 7f36f8d90700 1 --2- 172.21.15.1:0/1036542878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36f4099a40 0x7f36f40c6f80 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f36e80011e0 tx=0x7f36e800ca30 comp rx=0 tx=0).stop 2024-03-18T20:23:49.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.887+0000 7f36f8d90700 1 -- 172.21.15.1:0/1036542878 shutdown_connections 2024-03-18T20:23:49.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.887+0000 7f36f8d90700 1 --2- 172.21.15.1:0/1036542878 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f36d406ae00 0x7f36d406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.887+0000 7f36f8d90700 1 --2- 172.21.15.1:0/1036542878 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f36f406fa50 0x7f36f40b67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.887+0000 7f36f8d90700 1 --2- 172.21.15.1:0/1036542878 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36f4099e20 0x7f36f40ca650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.887+0000 7f36f8d90700 1 --2- 172.21.15.1:0/1036542878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36f4099a40 0x7f36f40c6f80 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:49.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.887+0000 7f36f8d90700 1 -- 172.21.15.1:0/1036542878 >> 172.21.15.1:0/1036542878 conn(0x7f36f40b2c50 msgr2=0x7f36f406fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:49.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.887+0000 7f36f8d90700 1 -- 172.21.15.1:0/1036542878 shutdown_connections 2024-03-18T20:23:49.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:49.887+0000 7f36f8d90700 1 -- 172.21.15.1:0/1036542878 wait complete. 2024-03-18T20:23:49.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:49.906 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-03-18T20:23:49.930 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-03-18T20:23:50.068 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:50.064+0000 7f80b98a8240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:23:50.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.098+0000 7fb8f4ad0700 1 Processor -- start 2024-03-18T20:23:50.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.098+0000 7fb8f4ad0700 1 -- start start 2024-03-18T20:23:50.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.099+0000 7fb8f4ad0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb8f0070580 0x7fb8f0070a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:50.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.099+0000 7fb8f4ad0700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f00b38d0 con 0x7fb8f00ce8d0 2024-03-18T20:23:50.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.099+0000 7fb8f4ad0700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f00b4470 con 0x7fb8f006d3d0 2024-03-18T20:23:50.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.099+0000 7fb8f4ad0700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f00cecb0 con 0x7fb8f0070580 2024-03-18T20:23:50.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.100+0000 7fb8edd9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb8f0070580 0x7fb8f0070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:50.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.100+0000 7fb8ee59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb8f00ce8d0 0x7fb8f00b3d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55748/0 (socket says 172.21.15.1:55748) 2024-03-18T20:23:50.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.100+0000 7fb8edd9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb8f0070580 0x7fb8f0070a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57984/0 (socket says 172.21.15.1:57984) 2024-03-18T20:23:50.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.100+0000 7fb8ee59c700 1 -- 172.21.15.1:0/3108471012 learned_addr learned my addr 172.21.15.1:0/3108471012 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:23:50.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.100+0000 7fb8edd9b700 1 -- 172.21.15.1:0/3108471012 >> v1:172.21.15.165:6789/0 conn(0x7fb8f006d3d0 legacy=0x7fb8f00b1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:50.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.100+0000 7fb8edd9b700 1 -- 172.21.15.1:0/3108471012 >> v1:172.21.15.1:6789/0 conn(0x7fb8f00ce8d0 legacy=0x7fb8f00b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:50.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.100+0000 7fb8edd9b700 1 -- 172.21.15.1:0/3108471012 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f0071020 con 0x7fb8f0070580 2024-03-18T20:23:50.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.100+0000 7fb8ed59a700 1 -- 172.21.15.1:0/3108471012 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690477440 0 0) 0x7fb8f00b38d0 con 0x7fb8f00ce8d0 2024-03-18T20:23:50.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.100+0000 7fb8edd9b700 1 --2- 172.21.15.1:0/3108471012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8f0070580 0x7fb8f0070a00 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4009840 tx=0x7fb8e4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=923bd93b65789f3e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:50.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.100+0000 7fb8ed59a700 1 -- 172.21.15.1:0/3108471012 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4011830 con 0x7fb8f0070580 2024-03-18T20:23:50.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.100+0000 7fb8ed59a700 1 -- 172.21.15.1:0/3108471012 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e4011990 con 0x7fb8f0070580 2024-03-18T20:23:50.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.100+0000 7fb8ed59a700 1 -- 172.21.15.1:0/3108471012 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4011d20 con 0x7fb8f0070580 2024-03-18T20:23:50.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.101+0000 7fb8f4ad0700 1 -- 172.21.15.1:0/3108471012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8f0070580 msgr2=0x7fb8f0070a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:50.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.101+0000 7fb8f4ad0700 1 --2- 172.21.15.1:0/3108471012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8f0070580 0x7fb8f0070a00 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4009840 tx=0x7fb8e4009b50 comp rx=0 tx=0).stop 2024-03-18T20:23:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.101+0000 7fb8f4ad0700 1 -- 172.21.15.1:0/3108471012 shutdown_connections 2024-03-18T20:23:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.101+0000 7fb8f4ad0700 1 --2- 172.21.15.1:0/3108471012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8f0070580 0x7fb8f0070a00 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.101+0000 7fb8f4ad0700 1 -- 172.21.15.1:0/3108471012 >> 172.21.15.1:0/3108471012 conn(0x7fb8f006ab70 msgr2=0x7fb8f006af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.101+0000 7fb8f4ad0700 1 -- 172.21.15.1:0/3108471012 shutdown_connections 2024-03-18T20:23:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.101+0000 7fb8f4ad0700 1 -- 172.21.15.1:0/3108471012 wait complete. 2024-03-18T20:23:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.102+0000 7fb8f4ad0700 1 Processor -- start 2024-03-18T20:23:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.103+0000 7fb8f4ad0700 1 -- start start 2024-03-18T20:23:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.103+0000 7fb8f4ad0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8f006d3d0 0x7fb8f01f08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.103+0000 7fb8f4ad0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8f0070580 0x7fb8f0157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.103+0000 7fb8f4ad0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8f00ce8d0 0x7fb8f013d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.103+0000 7fb8f4ad0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f0158f40 con 0x7fb8f0070580 2024-03-18T20:23:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.103+0000 7fb8f4ad0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f01f0df0 con 0x7fb8f006d3d0 2024-03-18T20:23:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.103+0000 7fb8f4ad0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f01593d0 con 0x7fb8f00ce8d0 2024-03-18T20:23:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.103+0000 7fb8edd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8f0070580 0x7fb8f0157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.103+0000 7fb8edd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8f0070580 0x7fb8f0157870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57986/0 (socket says 172.21.15.1:57986) 2024-03-18T20:23:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.103+0000 7fb8edd9b700 1 -- 172.21.15.1:0/621406680 learned_addr learned my addr 172.21.15.1:0/621406680 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.104+0000 7fb8ee59c700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8f006d3d0 0x7fb8f01f08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.104+0000 7fb8eed9d700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8f00ce8d0 0x7fb8f013d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.104+0000 7fb8edd9b700 1 -- 172.21.15.1:0/621406680 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8f00ce8d0 msgr2=0x7fb8f013d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.104+0000 7fb8edd9b700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8f00ce8d0 0x7fb8f013d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.104+0000 7fb8edd9b700 1 -- 172.21.15.1:0/621406680 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8f006d3d0 msgr2=0x7fb8f01f08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.104+0000 7fb8edd9b700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8f006d3d0 0x7fb8f01f08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.104+0000 7fb8edd9b700 1 -- 172.21.15.1:0/621406680 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e40094a0 con 0x7fb8f0070580 2024-03-18T20:23:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.104+0000 7fb8edd9b700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8f0070580 0x7fb8f0157870 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40090b0 tx=0x7fb8e4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.104+0000 7fb8deffd700 1 -- 172.21.15.1:0/621406680 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e401a860 con 0x7fb8f0070580 2024-03-18T20:23:50.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.104+0000 7fb8deffd700 1 -- 172.21.15.1:0/621406680 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e401a9e0 con 0x7fb8f0070580 2024-03-18T20:23:50.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.104+0000 7fb8f4ad0700 1 -- 172.21.15.1:0/621406680 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f0158330 con 0x7fb8f0070580 2024-03-18T20:23:50.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.104+0000 7fb8deffd700 1 -- 172.21.15.1:0/621406680 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e401ad90 con 0x7fb8f0070580 2024-03-18T20:23:50.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.104+0000 7fb8f4ad0700 1 -- 172.21.15.1:0/621406680 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f01588f0 con 0x7fb8f0070580 2024-03-18T20:23:50.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.105+0000 7fb8f4ad0700 1 -- 172.21.15.1:0/621406680 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f00b25c0 con 0x7fb8f0070580 2024-03-18T20:23:50.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.105+0000 7fb8deffd700 1 -- 172.21.15.1:0/621406680 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 124) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fb8e4022070 con 0x7fb8f0070580 2024-03-18T20:23:50.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.106+0000 7fb8deffd700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb8d006ae00 0x7fb8d006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:50.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.106+0000 7fb8ee59c700 1 -- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb8d006ae00 msgr2=0x7fb8d006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:50.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.106+0000 7fb8deffd700 1 -- 172.21.15.1:0/621406680 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fb8e408a890 con 0x7fb8f0070580 2024-03-18T20:23:50.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.106+0000 7fb8ee59c700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb8d006ae00 0x7fb8d006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:23:50.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.110+0000 7fb8deffd700 1 -- 172.21.15.1:0/621406680 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb8e4055840 con 0x7fb8f0070580 2024-03-18T20:23:50.176 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:50.171+0000 7f4705b3b240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:23:50.280 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:50.275+0000 7f80b98a8240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:23:50.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.307+0000 7fb8ee59c700 1 -- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb8d006ae00 msgr2=0x7fb8d006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:50.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.307+0000 7fb8ee59c700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb8d006ae00 0x7fb8d006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:23:50.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.339+0000 7fb8f4ad0700 1 -- 172.21.15.1:0/621406680 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8f0159fa0 con 0x7fb8d006ae00 2024-03-18T20:23:50.396 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:50.391+0000 7f4705b3b240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:23:50.652 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:50.647+0000 7f80b98a8240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:23:50.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.708+0000 7fb8ee59c700 1 -- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb8d006ae00 msgr2=0x7fb8d006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:50.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:50.708+0000 7fb8ee59c700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb8d006ae00 0x7fb8d006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T20:23:50.787 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:50.782+0000 7f4705b3b240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:23:51.234 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:51.230+0000 7f80b98a8240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:23:51.305 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:51.302+0000 7f80b98a8240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:23:51.405 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:51.400+0000 7f4705b3b240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:23:51.432 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:51.428+0000 7f80b98a8240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:23:51.481 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:51.476+0000 7f4705b3b240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:23:51.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:51.509+0000 7fb8ee59c700 1 -- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb8d006ae00 msgr2=0x7fb8d006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:51.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:51.509+0000 7fb8ee59c700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb8d006ae00 0x7fb8d006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T20:23:51.609 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:51.606+0000 7f80b98a8240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:23:51.615 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:51.610+0000 7f4705b3b240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:23:51.681 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:51.678+0000 7f80b98a8240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:23:51.759 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:51.756+0000 7f80b98a8240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:23:51.802 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:51.797+0000 7f4705b3b240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:23:51.837 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:51.833+0000 7f80b98a8240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:23:51.878 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:51.873+0000 7f4705b3b240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:23:51.914 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:51.910+0000 7f80b98a8240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:23:51.961 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:51.956+0000 7f4705b3b240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:23:52.043 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:52.038+0000 7f4705b3b240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:23:52.082 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:52.079+0000 7f80b98a8240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:23:52.126 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:52.121+0000 7f4705b3b240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:23:52.297 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:52.293+0000 7f80b98a8240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:23:52.306 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:52.300+0000 7f4705b3b240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:23:52.449 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:52.445+0000 7f80b98a8240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:23:52.522 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:52.518+0000 7f80b98a8240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:23:52.529 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:52.524+0000 7f4705b3b240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:23:52.688 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:52.683+0000 7f4705b3b240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:23:52.765 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:52.760+0000 7f4705b3b240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:23:52.966 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:52.962+0000 7f80b98a8240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:23:53.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:53.110+0000 7fb8ee59c700 1 -- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb8d006ae00 msgr2=0x7fb8d006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:23:53.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:53.110+0000 7fb8ee59c700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb8d006ae00 0x7fb8d006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T20:23:53.241 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:53.235+0000 7f4705b3b240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:23:53.303 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:53.298+0000 7f80b98a8240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:23:53.465 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:53.461+0000 7f80b98a8240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:23:53.594 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:53.589+0000 7f4705b3b240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:23:53.767 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:53.761+0000 7f4705b3b240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:23:53.852 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:53.848+0000 7f80b98a8240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:23:54.016 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:54.012+0000 7f80b98a8240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:23:54.181 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:54.176+0000 7f4705b3b240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:23:54.358 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:54.352+0000 7f4705b3b240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:23:54.377 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:54.373+0000 7f80b98a8240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:23:54.464 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:23:54.461+0000 7f80b98a8240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:23:54.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:54.529+0000 7fb8deffd700 1 -- 172.21.15.1:0/621406680 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 125) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fb8e4055c10 con 0x7fb8f0070580 2024-03-18T20:23:54.741 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:54.736+0000 7f4705b3b240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:23:54.835 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:54.830+0000 7f4705b3b240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:23:54.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:54.848+0000 7fb8deffd700 1 -- 172.21.15.1:0/621406680 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 126) v1 ==== 88865+0+0 (secure 0 0 0) 0x7fb8e4026da0 con 0x7fb8f0070580 2024-03-18T20:23:54.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:54.849+0000 7fb8deffd700 1 -- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb8d006ae00 msgr2=0x7fb8d006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:23:54.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:54.849+0000 7fb8deffd700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb8d006ae00 0x7fb8d006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:55.018 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:55.012+0000 7f4678758700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:55.019 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:55.012+0000 7f4678758700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:55.019 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:55.012+0000 7f4678758700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:55.019 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:55.012+0000 7f4678758700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:55.019 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:55.012+0000 7f4678758700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:55.019 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:55.012+0000 7f467b75e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:55.019 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:55.012+0000 7f467b75e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:55.019 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:55.012+0000 7f467b75e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:55.019 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:55.012+0000 7f467b75e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:55.019 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:23:55.012+0000 7f467b75e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:23:55.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:55.854+0000 7fb8deffd700 1 -- 172.21.15.1:0/621406680 <== mon.0 v2:172.21.15.1:3300/0 9 ==== mgrmap(e 127) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fb8e40564e0 con 0x7fb8f0070580 2024-03-18T20:23:55.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:55.855+0000 7fb8deffd700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb8d006ae00 0x7fb8d006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:55.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:55.855+0000 7fb8deffd700 1 -- 172.21.15.1:0/621406680 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8d001cd40 con 0x7fb8d006ae00 2024-03-18T20:23:55.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:55.856+0000 7fb8ee59c700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb8d006ae00 0x7fb8d006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:55.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:55.860+0000 7fb8ee59c700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb8d006ae00 0x7fb8d006d2c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0001210 tx=0x7fb8e00011a0 comp rx=0 tx=0).ready entity=mgr.11205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:55.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:55.918+0000 7fb8deffd700 1 -- 172.21.15.1:0/621406680 <== mgr.11205 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb8d001cd40 con 0x7fb8d006ae00 2024-03-18T20:23:55.923 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path snap_test_dir1 2024-03-18T20:23:55.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:55.920+0000 7fb8f4ad0700 1 -- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb8d006ae00 msgr2=0x7fb8d006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:55.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:55.920+0000 7fb8f4ad0700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb8d006ae00 0x7fb8d006d2c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0001210 tx=0x7fb8e00011a0 comp rx=0 tx=0).stop 2024-03-18T20:23:55.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:55.920+0000 7fb8f4ad0700 1 -- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8f0070580 msgr2=0x7fb8f0157870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:55.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:55.920+0000 7fb8f4ad0700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8f0070580 0x7fb8f0157870 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40090b0 tx=0x7fb8e4004750 comp rx=0 tx=0).stop 2024-03-18T20:23:55.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:55.920+0000 7fb8f4ad0700 1 -- 172.21.15.1:0/621406680 shutdown_connections 2024-03-18T20:23:55.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:55.920+0000 7fb8f4ad0700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb8d006ae00 0x7fb8d006d2c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:55.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:55.920+0000 7fb8f4ad0700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8f00ce8d0 0x7fb8f013d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:55.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:55.920+0000 7fb8f4ad0700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8f006d3d0 0x7fb8f01f08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:55.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:55.920+0000 7fb8f4ad0700 1 --2- 172.21.15.1:0/621406680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8f0070580 0x7fb8f0157870 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:55.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:55.920+0000 7fb8f4ad0700 1 -- 172.21.15.1:0/621406680 >> 172.21.15.1:0/621406680 conn(0x7fb8f006ab70 msgr2=0x7fb8f014a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:55.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:55.920+0000 7fb8f4ad0700 1 -- 172.21.15.1:0/621406680 shutdown_connections 2024-03-18T20:23:55.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:55.920+0000 7fb8f4ad0700 1 -- 172.21.15.1:0/621406680 wait complete. 2024-03-18T20:23:55.938 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-03-18T20:23:55.939 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2024-03-18T20:23:56.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.065+0000 7f4c42982700 1 Processor -- start 2024-03-18T20:23:56.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.065+0000 7f4c42982700 1 -- start start 2024-03-18T20:23:56.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.066+0000 7f4c42982700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c3c06ae60 0x7f4c3c0af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:56.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.066+0000 7f4c42982700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3c15dd70 con 0x7f4c3c15d540 2024-03-18T20:23:56.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.066+0000 7f4c42982700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3c15eaf0 con 0x7f4c3c161b60 2024-03-18T20:23:56.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.066+0000 7f4c42982700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c3c162100 con 0x7f4c3c06ae60 2024-03-18T20:23:56.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.066+0000 7f4c3b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c3c06ae60 0x7f4c3c0af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:56.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.066+0000 7f4c3b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c3c06ae60 0x7f4c3c0af0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37200/0 (socket says 172.21.15.1:37200) 2024-03-18T20:23:56.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.066+0000 7f4c3b7fe700 1 -- 172.21.15.1:0/3891670439 learned_addr learned my addr 172.21.15.1:0/3891670439 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:56.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.066+0000 7f4c3bfff700 1 --1- 172.21.15.1:0/3891670439 >> v1:172.21.15.1:6789/0 conn(0x7f4c3c15d540 0x7f4c3c06a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55548/0 (socket says 172.21.15.1:55548) 2024-03-18T20:23:56.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.066+0000 7f4c3b7fe700 1 -- 172.21.15.1:0/3891670439 >> v1:172.21.15.165:6789/0 conn(0x7f4c3c161b60 legacy=0x7f4c3c0b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:56.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.066+0000 7f4c3b7fe700 1 -- 172.21.15.1:0/3891670439 >> v1:172.21.15.1:6789/0 conn(0x7f4c3c15d540 legacy=0x7f4c3c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:56.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.066+0000 7f4c3b7fe700 1 -- 172.21.15.1:0/3891670439 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3c15e010 con 0x7f4c3c06ae60 2024-03-18T20:23:56.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.067+0000 7f4c3b7fe700 1 --2- 172.21.15.1:0/3891670439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3c06ae60 0x7f4c3c0af0f0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f4c240098f0 tx=0x7f4c24009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83f8094912755d03 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:56.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.067+0000 7f4c3affd700 1 -- 172.21.15.1:0/3891670439 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2400b8a0 con 0x7f4c3c06ae60 2024-03-18T20:23:56.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.067+0000 7f4c3affd700 1 -- 172.21.15.1:0/3891670439 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2400ba00 con 0x7f4c3c06ae60 2024-03-18T20:23:56.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.067+0000 7f4c3affd700 1 -- 172.21.15.1:0/3891670439 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2400bd90 con 0x7f4c3c06ae60 2024-03-18T20:23:56.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.067+0000 7f4c42982700 1 -- 172.21.15.1:0/3891670439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3c06ae60 msgr2=0x7f4c3c0af0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:56.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.067+0000 7f4c42982700 1 --2- 172.21.15.1:0/3891670439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3c06ae60 0x7f4c3c0af0f0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f4c240098f0 tx=0x7f4c24009c00 comp rx=0 tx=0).stop 2024-03-18T20:23:56.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.067+0000 7f4c42982700 1 -- 172.21.15.1:0/3891670439 shutdown_connections 2024-03-18T20:23:56.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.067+0000 7f4c42982700 1 --2- 172.21.15.1:0/3891670439 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3c06ae60 0x7f4c3c0af0f0 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:56.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.067+0000 7f4c42982700 1 -- 172.21.15.1:0/3891670439 >> 172.21.15.1:0/3891670439 conn(0x7f4c3c0b1cb0 msgr2=0x7f4c3c0b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:56.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.067+0000 7f4c42982700 1 -- 172.21.15.1:0/3891670439 shutdown_connections 2024-03-18T20:23:56.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.067+0000 7f4c42982700 1 -- 172.21.15.1:0/3891670439 wait complete. 2024-03-18T20:23:56.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.068+0000 7f4c42982700 1 Processor -- start 2024-03-18T20:23:56.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.069+0000 7f4c42982700 1 -- start start 2024-03-18T20:23:56.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.069+0000 7f4c42982700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c3c06ae60 0x7f4c3c1e7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:56.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.069+0000 7f4c42982700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c3c15d540 0x7f4c3c149af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:56.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.069+0000 7f4c42982700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3c161b60 0x7f4c3c0acbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:56.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.069+0000 7f4c42982700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c0ae050 con 0x7f4c3c161b60 2024-03-18T20:23:56.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.069+0000 7f4c42982700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c0ad2c0 con 0x7f4c3c15d540 2024-03-18T20:23:56.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.069+0000 7f4c42982700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c0ad6f0 con 0x7f4c3c06ae60 2024-03-18T20:23:56.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.069+0000 7f4c40f1f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3c161b60 0x7f4c3c0acbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:56.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.070+0000 7f4c40f1f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3c161b60 0x7f4c3c0acbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37220/0 (socket says 172.21.15.1:37220) 2024-03-18T20:23:56.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.070+0000 7f4c40f1f700 1 -- 172.21.15.1:0/3539577367 learned_addr learned my addr 172.21.15.1:0/3539577367 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.070+0000 7f4c3bfff700 1 --2- 172.21.15.1:0/3539577367 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c3c06ae60 0x7f4c3c1e7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.070+0000 7f4c3b7fe700 1 --2- 172.21.15.1:0/3539577367 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c3c15d540 0x7f4c3c149af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.070+0000 7f4c40f1f700 1 -- 172.21.15.1:0/3539577367 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c3c06ae60 msgr2=0x7f4c3c1e7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.070+0000 7f4c40f1f700 1 --2- 172.21.15.1:0/3539577367 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c3c06ae60 0x7f4c3c1e7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.070+0000 7f4c40f1f700 1 -- 172.21.15.1:0/3539577367 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c3c15d540 msgr2=0x7f4c3c149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.070+0000 7f4c40f1f700 1 --2- 172.21.15.1:0/3539577367 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c3c15d540 0x7f4c3c149af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.070+0000 7f4c40f1f700 1 -- 172.21.15.1:0/3539577367 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c24009550 con 0x7f4c3c161b60 2024-03-18T20:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.070+0000 7f4c40f1f700 1 --2- 172.21.15.1:0/3539577367 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3c161b60 0x7f4c3c0acbc0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f4c3400cc80 tx=0x7f4c3400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.070+0000 7f4c38ff9700 1 -- 172.21.15.1:0/3539577367 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c34003c00 con 0x7f4c3c161b60 2024-03-18T20:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.070+0000 7f4c38ff9700 1 -- 172.21.15.1:0/3539577367 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c34003d60 con 0x7f4c3c161b60 2024-03-18T20:23:56.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.070+0000 7f4c38ff9700 1 -- 172.21.15.1:0/3539577367 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3400aaa0 con 0x7f4c3c161b60 2024-03-18T20:23:56.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.070+0000 7f4c42982700 1 -- 172.21.15.1:0/3539577367 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c3c0ad990 con 0x7f4c3c161b60 2024-03-18T20:23:56.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.070+0000 7f4c42982700 1 -- 172.21.15.1:0/3539577367 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c3c0ae470 con 0x7f4c3c161b60 2024-03-18T20:23:56.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.071+0000 7f4c42982700 1 -- 172.21.15.1:0/3539577367 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c3c0ae190 con 0x7f4c3c161b60 2024-03-18T20:23:56.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.072+0000 7f4c38ff9700 1 -- 172.21.15.1:0/3539577367 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 127) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f4c3400ac00 con 0x7f4c3c161b60 2024-03-18T20:23:56.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.072+0000 7f4c38ff9700 1 --2- 172.21.15.1:0/3539577367 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c1c06ab30 0x7f4c1c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:56.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.072+0000 7f4c3bfff700 1 --2- 172.21.15.1:0/3539577367 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c1c06ab30 0x7f4c1c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:56.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.072+0000 7f4c38ff9700 1 -- 172.21.15.1:0/3539577367 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f4c34089200 con 0x7f4c3c161b60 2024-03-18T20:23:56.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.073+0000 7f4c3bfff700 1 --2- 172.21.15.1:0/3539577367 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c1c06ab30 0x7f4c1c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c000f60 tx=0x7f4c2c007ed0 comp rx=0 tx=0).ready entity=mgr.11205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:56.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.077+0000 7f4c38ff9700 1 -- 172.21.15.1:0/3539577367 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4c3c0ae190 con 0x7f4c3c161b60 2024-03-18T20:23:56.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.309+0000 7f4c42982700 1 -- 172.21.15.1:0/3539577367 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c3c15e5d0 con 0x7f4c1c06ab30 2024-03-18T20:23:56.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.355+0000 7f4c38ff9700 1 -- 172.21.15.1:0/3539577367 <== mgr.11205 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7f4c2c0133f0 con 0x7f4c1c06ab30 2024-03-18T20:23:56.361 INFO:teuthology.orchestra.run.smithi001.stdout:Retention added to path snap_test_dir1 2024-03-18T20:23:56.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.357+0000 7f4c42982700 1 -- 172.21.15.1:0/3539577367 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c1c06ab30 msgr2=0x7f4c1c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:56.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.357+0000 7f4c42982700 1 --2- 172.21.15.1:0/3539577367 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c1c06ab30 0x7f4c1c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c000f60 tx=0x7f4c2c007ed0 comp rx=0 tx=0).stop 2024-03-18T20:23:56.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.357+0000 7f4c42982700 1 -- 172.21.15.1:0/3539577367 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3c161b60 msgr2=0x7f4c3c0acbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:56.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.357+0000 7f4c42982700 1 --2- 172.21.15.1:0/3539577367 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3c161b60 0x7f4c3c0acbc0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f4c3400cc80 tx=0x7f4c3400cf90 comp rx=0 tx=0).stop 2024-03-18T20:23:56.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.357+0000 7f4c42982700 1 -- 172.21.15.1:0/3539577367 shutdown_connections 2024-03-18T20:23:56.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.357+0000 7f4c42982700 1 --2- 172.21.15.1:0/3539577367 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c1c06ab30 0x7f4c1c06cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:56.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.357+0000 7f4c42982700 1 --2- 172.21.15.1:0/3539577367 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c3c06ae60 0x7f4c3c1e7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:56.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.357+0000 7f4c42982700 1 --2- 172.21.15.1:0/3539577367 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3c161b60 0x7f4c3c0acbc0 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:56.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.357+0000 7f4c42982700 1 --2- 172.21.15.1:0/3539577367 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c3c15d540 0x7f4c3c149af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:56.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.357+0000 7f4c42982700 1 -- 172.21.15.1:0/3539577367 >> 172.21.15.1:0/3539577367 conn(0x7f4c3c0b1cb0 msgr2=0x7f4c3c071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:56.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.358+0000 7f4c42982700 1 -- 172.21.15.1:0/3539577367 shutdown_connections 2024-03-18T20:23:56.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.358+0000 7f4c42982700 1 -- 172.21.15.1:0/3539577367 wait complete. 2024-03-18T20:23:56.375 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2024-03-18T20:23:56.375 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-03-18-20_24_00 in ~4s... 2024-03-18T20:23:56.375 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2024-03-18T20:23:56.375 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-03-18T20:23:56.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.496+0000 7fc74421c700 1 Processor -- start 2024-03-18T20:23:56.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.497+0000 7fc74421c700 1 -- start start 2024-03-18T20:23:56.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.498+0000 7fc74421c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc73c0714e0 0x7fc73c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:56.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.498+0000 7fc74421c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73c0aedf0 con 0x7fc73c06e310 2024-03-18T20:23:56.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.498+0000 7fc74421c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73c15a940 con 0x7fc73c15a270 2024-03-18T20:23:56.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.498+0000 7fc74421c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc73c15ac20 con 0x7fc73c0714e0 2024-03-18T20:23:56.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.498+0000 7fc7417b7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc73c0714e0 0x7fc73c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:56.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.498+0000 7fc7417b7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc73c0714e0 0x7fc73c15e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37222/0 (socket says 172.21.15.1:37222) 2024-03-18T20:23:56.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.498+0000 7fc7417b7700 1 -- 172.21.15.1:0/1200714551 learned_addr learned my addr 172.21.15.1:0/1200714551 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:56.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.498+0000 7fc7417b7700 1 -- 172.21.15.1:0/1200714551 >> v1:172.21.15.165:6789/0 conn(0x7fc73c15a270 legacy=0x7fc73c070dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:56.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.498+0000 7fc7417b7700 1 -- 172.21.15.1:0/1200714551 >> v1:172.21.15.1:6789/0 conn(0x7fc73c06e310 legacy=0x7fc73c15bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:56.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.498+0000 7fc7417b7700 1 -- 172.21.15.1:0/1200714551 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73c15af00 con 0x7fc73c0714e0 2024-03-18T20:23:56.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.499+0000 7fc7417b7700 1 --2- 172.21.15.1:0/1200714551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc73c0714e0 0x7fc73c15e680 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7fc72c0098f0 tx=0x7fc72c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7133de744412afe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:56.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.499+0000 7fc740fb6700 1 -- 172.21.15.1:0/1200714551 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c00b8a0 con 0x7fc73c0714e0 2024-03-18T20:23:56.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.499+0000 7fc740fb6700 1 -- 172.21.15.1:0/1200714551 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72c00ba00 con 0x7fc73c0714e0 2024-03-18T20:23:56.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.499+0000 7fc740fb6700 1 -- 172.21.15.1:0/1200714551 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c00bd90 con 0x7fc73c0714e0 2024-03-18T20:23:56.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.499+0000 7fc74421c700 1 -- 172.21.15.1:0/1200714551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc73c0714e0 msgr2=0x7fc73c15e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:56.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.499+0000 7fc74421c700 1 --2- 172.21.15.1:0/1200714551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc73c0714e0 0x7fc73c15e680 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7fc72c0098f0 tx=0x7fc72c009c00 comp rx=0 tx=0).stop 2024-03-18T20:23:56.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.499+0000 7fc74421c700 1 -- 172.21.15.1:0/1200714551 shutdown_connections 2024-03-18T20:23:56.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.499+0000 7fc74421c700 1 --2- 172.21.15.1:0/1200714551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc73c0714e0 0x7fc73c15e680 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:56.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.499+0000 7fc74421c700 1 -- 172.21.15.1:0/1200714551 >> 172.21.15.1:0/1200714551 conn(0x7fc73c066b70 msgr2=0x7fc73c06a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:56.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.499+0000 7fc74421c700 1 -- 172.21.15.1:0/1200714551 shutdown_connections 2024-03-18T20:23:56.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.499+0000 7fc74421c700 1 -- 172.21.15.1:0/1200714551 wait complete. 2024-03-18T20:23:56.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.501+0000 7fc74421c700 1 Processor -- start 2024-03-18T20:23:56.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.501+0000 7fc74421c700 1 -- start start 2024-03-18T20:23:56.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.502+0000 7fc74421c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc73c06e310 0x7fc73c14a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:56.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.502+0000 7fc74421c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc73c0714e0 0x7fc73c1e5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:56.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.502+0000 7fc74421c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc73c15a270 0x7fc73c0a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:56.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.502+0000 7fc74421c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c0aa2a0 con 0x7fc73c0714e0 2024-03-18T20:23:56.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.502+0000 7fc74421c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c0a93f0 con 0x7fc73c15a270 2024-03-18T20:23:56.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.502+0000 7fc74421c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c0a9690 con 0x7fc73c06e310 2024-03-18T20:23:56.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.502+0000 7fc7417b7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc73c0714e0 0x7fc73c1e5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:56.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.502+0000 7fc7417b7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc73c0714e0 0x7fc73c1e5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37226/0 (socket says 172.21.15.1:37226) 2024-03-18T20:23:56.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.502+0000 7fc7417b7700 1 -- 172.21.15.1:0/1273515754 learned_addr learned my addr 172.21.15.1:0/1273515754 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:23:56.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.502+0000 7fc741fb8700 1 --2- 172.21.15.1:0/1273515754 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc73c06e310 0x7fc73c14a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:56.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.502+0000 7fc7427b9700 1 --2- 172.21.15.1:0/1273515754 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc73c15a270 0x7fc73c0a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:56.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.502+0000 7fc7417b7700 1 -- 172.21.15.1:0/1273515754 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc73c06e310 msgr2=0x7fc73c14a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:56.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.502+0000 7fc7417b7700 1 --2- 172.21.15.1:0/1273515754 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc73c06e310 0x7fc73c14a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:56.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.502+0000 7fc7417b7700 1 -- 172.21.15.1:0/1273515754 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc73c15a270 msgr2=0x7fc73c0a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:56.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.502+0000 7fc7417b7700 1 --2- 172.21.15.1:0/1273515754 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc73c15a270 0x7fc73c0a8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:56.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.502+0000 7fc7417b7700 1 -- 172.21.15.1:0/1273515754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72c009550 con 0x7fc73c0714e0 2024-03-18T20:23:56.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.502+0000 7fc7427b9700 1 --2- 172.21.15.1:0/1273515754 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc73c15a270 0x7fc73c0a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:56.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.502+0000 7fc741fb8700 1 --2- 172.21.15.1:0/1273515754 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc73c06e310 0x7fc73c14a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:23:56.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.502+0000 7fc7417b7700 1 --2- 172.21.15.1:0/1273515754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc73c0714e0 0x7fc73c1e5ea0 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7fc72c000c00 tx=0x7fc72c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:56.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.503+0000 7fc7327fc700 1 -- 172.21.15.1:0/1273515754 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c004570 con 0x7fc73c0714e0 2024-03-18T20:23:56.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.503+0000 7fc7327fc700 1 -- 172.21.15.1:0/1273515754 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72c01f070 con 0x7fc73c0714e0 2024-03-18T20:23:56.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.503+0000 7fc74421c700 1 -- 172.21.15.1:0/1273515754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc73c0a9970 con 0x7fc73c0714e0 2024-03-18T20:23:56.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.503+0000 7fc7327fc700 1 -- 172.21.15.1:0/1273515754 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c0038e0 con 0x7fc73c0714e0 2024-03-18T20:23:56.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.503+0000 7fc74421c700 1 -- 172.21.15.1:0/1273515754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc73c0a9f30 con 0x7fc73c0714e0 2024-03-18T20:23:56.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.504+0000 7fc74421c700 1 -- 172.21.15.1:0/1273515754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc73c0676a0 con 0x7fc73c0714e0 2024-03-18T20:23:56.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.504+0000 7fc7327fc700 1 -- 172.21.15.1:0/1273515754 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 127) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fc72c021030 con 0x7fc73c0714e0 2024-03-18T20:23:56.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.505+0000 7fc7327fc700 1 --2- 172.21.15.1:0/1273515754 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc72006ab30 0x7fc72006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:23:56.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.505+0000 7fc7327fc700 1 -- 172.21.15.1:0/1273515754 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6473+0+0 (secure 0 0 0) 0x7fc72c08a850 con 0x7fc73c0714e0 2024-03-18T20:23:56.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.505+0000 7fc741fb8700 1 --2- 172.21.15.1:0/1273515754 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc72006ab30 0x7fc72006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:23:56.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.505+0000 7fc741fb8700 1 --2- 172.21.15.1:0/1273515754 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc72006ab30 0x7fc72006cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc734000f60 tx=0x7fc734009040 comp rx=0 tx=0).ready entity=mgr.11205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:23:56.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.509+0000 7fc7327fc700 1 -- 172.21.15.1:0/1273515754 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc72c055ac0 con 0x7fc73c0714e0 2024-03-18T20:23:56.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.737+0000 7fc74421c700 1 -- 172.21.15.1:0/1273515754 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc73c15a650 con 0x7fc72006ab30 2024-03-18T20:23:56.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.745+0000 7fc7327fc700 1 -- 172.21.15.1:0/1273515754 <== mgr.11205 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7fc734001340 con 0x7fc72006ab30 2024-03-18T20:23:56.751 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:23:56.751 INFO:teuthology.orchestra.run.smithi001.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-03-18T20:23:56.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.747+0000 7fc74421c700 1 -- 172.21.15.1:0/1273515754 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc72006ab30 msgr2=0x7fc72006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:56.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.747+0000 7fc74421c700 1 --2- 172.21.15.1:0/1273515754 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc72006ab30 0x7fc72006cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc734000f60 tx=0x7fc734009040 comp rx=0 tx=0).stop 2024-03-18T20:23:56.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.747+0000 7fc74421c700 1 -- 172.21.15.1:0/1273515754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc73c0714e0 msgr2=0x7fc73c1e5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:23:56.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.747+0000 7fc74421c700 1 --2- 172.21.15.1:0/1273515754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc73c0714e0 0x7fc73c1e5ea0 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7fc72c000c00 tx=0x7fc72c004750 comp rx=0 tx=0).stop 2024-03-18T20:23:56.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.748+0000 7fc74421c700 1 -- 172.21.15.1:0/1273515754 shutdown_connections 2024-03-18T20:23:56.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.748+0000 7fc74421c700 1 --2- 172.21.15.1:0/1273515754 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc72006ab30 0x7fc72006cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:56.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.748+0000 7fc74421c700 1 --2- 172.21.15.1:0/1273515754 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc73c06e310 0x7fc73c14a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:56.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.748+0000 7fc74421c700 1 --2- 172.21.15.1:0/1273515754 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc73c15a270 0x7fc73c0a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:56.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.748+0000 7fc74421c700 1 --2- 172.21.15.1:0/1273515754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc73c0714e0 0x7fc73c1e5ea0 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:23:56.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.748+0000 7fc74421c700 1 -- 172.21.15.1:0/1273515754 >> 172.21.15.1:0/1273515754 conn(0x7fc73c066b70 msgr2=0x7fc73c15e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:23:56.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.748+0000 7fc74421c700 1 -- 172.21.15.1:0/1273515754 shutdown_connections 2024-03-18T20:23:56.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:23:56.748+0000 7fc74421c700 1 -- 172.21.15.1:0/1273515754 wait complete. 2024-03-18T20:23:56.765 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-03-18T20:23:56.765 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2024-03-18T20:23:56.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:56.765 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:23:56.788 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T20:23:56.788 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:23:56.788 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:23:57.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:57.789 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:23:57.817 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T20:23:57.817 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:23:57.817 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:23:58.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:58.818 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:23:58.845 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T20:23:58.846 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:23:58.846 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:23:59.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:23:59.847 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:23:59.875 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-03-18T20:23:59.875 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:23:59.875 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:00.371 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:24:00.365+0000 7f466a73c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:24:00.371 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:24:00.365+0000 7f466a73c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:24:00.371 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:24:00.365+0000 7f466a73c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:24:00.371 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:24:00.365+0000 7f466a73c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:24:00.371 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:24:00.365+0000 7f466a73c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:24:00.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:00.876 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:00.903 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:00.904 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:00.904 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:00.904 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:00.904 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-03-18-20_24_00_UTC'] 2024-03-18T20:24:00.904 DEBUG:tasks.cephfs.test_snap_schedules:exec=1710793436.3754241, now = 1710793440.904444, timo = 4 2024-03-18T20:24:01.904 DEBUG:tasks.cephfs.test_snap_schedules:expecting removal of snap snap_test_dir1/.snap/scheduled-2024-03-18-20_24_00 in ~64s... 2024-03-18T20:24:01.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:01.905 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:01.932 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:01.933 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:01.933 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:01.933 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:02.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:02.933 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:02.961 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:02.961 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:02.961 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:02.961 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:03.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:03.962 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:03.990 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:03.990 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:03.990 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:03.990 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:04.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:04.991 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:05.019 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:05.019 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:05.019 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:05.019 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:06.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:06.021 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:06.048 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:06.048 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:06.048 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:06.048 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:07.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:07.049 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:07.077 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:07.077 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:07.077 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:07.077 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:08.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:08.079 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:08.105 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:08.106 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:08.106 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:08.106 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:09.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:09.108 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:09.135 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:09.136 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:09.136 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:09.136 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:10.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:10.137 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:10.164 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:10.165 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:10.165 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:10.165 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:10.335 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:24:10.338 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:24:11.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:11.166 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:11.193 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:11.193 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:11.193 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:11.194 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:12.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:12.195 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:12.221 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:12.221 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:12.221 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:12.221 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:13.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:13.223 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:13.250 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:13.250 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:13.250 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:13.250 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:14.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:14.252 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:14.278 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:14.279 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:14.280 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:14.280 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:15.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:15.280 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:15.307 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:15.308 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:15.308 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:15.308 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:16.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:16.309 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:16.336 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:16.337 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:16.337 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:16.337 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:17.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:17.338 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:17.366 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:17.366 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:17.366 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:17.366 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:18.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:18.367 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:18.395 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:18.395 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:18.395 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:18.395 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:19.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:19.396 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:19.424 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:19.424 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:19.424 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:19.424 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:20.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:20.425 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:20.453 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:20.454 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:20.454 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:20.455 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:21.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:21.455 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:21.483 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:21.484 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:21.484 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:21.484 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:22.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:22.485 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:22.512 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:22.513 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:22.513 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:22.513 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:23.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:23.514 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:23.541 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:23.542 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:23.542 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:23.542 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:24.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:24.543 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:24.570 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:24.571 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:24.571 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:24.571 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:25.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:25.572 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:25.599 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:25.600 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:25.600 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:25.600 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:26.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:26.601 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:26.629 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:26.629 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:26.629 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:26.629 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:27.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:27.630 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:27.658 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:27.658 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:27.658 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:27.658 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:28.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:28.659 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:28.686 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:28.687 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:28.687 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:28.687 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:29.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:29.688 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:29.715 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:29.716 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:29.716 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:29.716 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:30.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:30.717 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:30.745 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:30.745 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:30.745 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:30.745 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:31.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:31.746 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:31.773 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:31.774 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:31.774 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:31.774 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:32.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:32.775 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:32.802 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:32.803 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:32.803 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:32.803 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:33.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:33.804 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:33.830 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:33.831 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:33.832 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:33.832 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:34.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:34.833 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:34.862 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:34.862 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:34.862 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:34.863 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:35.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:35.864 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:35.891 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:35.892 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:35.892 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:35.892 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:36.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:36.893 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:36.921 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:36.921 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:36.921 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:36.921 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:37.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:37.923 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:37.950 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:37.950 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:37.950 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:37.950 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:38.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:38.951 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:38.979 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:38.980 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:38.980 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:38.980 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:39.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:39.982 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:40.009 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:40.009 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:40.010 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:40.010 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:40.448 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:24:40.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:24:41.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:41.011 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:41.039 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:41.039 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:41.039 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:41.039 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:42.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:42.041 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:42.067 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:42.068 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:42.068 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:42.068 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:43.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:43.069 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:43.097 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:43.097 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:43.097 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:43.097 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:44.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:44.099 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:44.125 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:44.126 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:44.127 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:44.127 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:45.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:45.128 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:45.155 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:45.156 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:45.156 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:45.156 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:46.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:46.157 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:46.184 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:46.184 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:46.185 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:46.185 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:47.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:47.186 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:47.214 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:47.214 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:47.214 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:47.214 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:48.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:48.216 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:48.243 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:48.243 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:48.244 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:48.244 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:49.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:49.245 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:49.272 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:49.272 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:49.272 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:49.272 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:50.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:50.273 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:50.300 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:50.301 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:50.301 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:50.301 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:51.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:51.302 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:51.329 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:51.329 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:51.329 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:51.329 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:52.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:52.330 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:52.357 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:52.358 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:52.358 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:52.358 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:53.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:53.359 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:53.386 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:53.387 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:53.387 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:53.387 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:54.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:54.389 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:54.416 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:54.417 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:54.417 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:54.417 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:55.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:55.418 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:55.446 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:55.446 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:55.446 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:55.446 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:56.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:56.447 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:56.475 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:56.475 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:56.475 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:56.475 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:57.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:57.476 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:57.504 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:57.504 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:57.504 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:57.504 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:58.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:58.505 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:58.533 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:58.533 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:58.533 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:58.533 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:24:59.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:24:59.534 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:24:59.562 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_24_00_UTC 2024-03-18T20:24:59.563 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:24:59.563 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-03-18T20:24:59.563 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-03-18T20:25:00.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:00.564 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:25:00.589 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_25_00_UTC 2024-03-18T20:25:00.589 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-03-18-20_25_00_UTC'} 2024-03-18T20:25:00.590 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-03-18-20_25_00_UTC'} 2024-03-18T20:25:00.590 INFO:tasks.cephfs.test_snap_schedules:removed: {'scheduled-2024-03-18-20_24_00_UTC'} 2024-03-18T20:25:00.590 DEBUG:tasks.cephfs.test_snap_schedules:snapshots removed=['scheduled-2024-03-18-20_24_00_UTC'] 2024-03-18T20:25:00.590 DEBUG:tasks.cephfs.test_snap_schedules:exec=1710793436.3754241, now = 1710793500.5903294, timo = 64 2024-03-18T20:25:01.591 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-03-18T20:25:01.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.727+0000 7f8ff9b45700 1 Processor -- start 2024-03-18T20:25:01.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.728+0000 7f8ff9b45700 1 -- start start 2024-03-18T20:25:01.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.728+0000 7f8ff9b45700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8ff4160a70 0x7f8ff40d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:01.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.728+0000 7f8ff9b45700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff4162170 con 0x7f8ff4160e50 2024-03-18T20:25:01.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.728+0000 7f8ff9b45700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff40d06e0 con 0x7f8ff41612d0 2024-03-18T20:25:01.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.728+0000 7f8ff9b45700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff40cef30 con 0x7f8ff4160a70 2024-03-18T20:25:01.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.728+0000 7f8ff37fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8ff4160a70 0x7f8ff40d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:01.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.728+0000 7f8ff2ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8ff4160e50 0x7f8ff40cffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58104/0 (socket says 172.21.15.1:58104) 2024-03-18T20:25:01.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.728+0000 7f8ff2ffd700 1 -- 172.21.15.1:0/4271978060 learned_addr learned my addr 172.21.15.1:0/4271978060 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:01.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.729+0000 7f8ff37fe700 1 -- 172.21.15.1:0/4271978060 >> v1:172.21.15.165:6789/0 conn(0x7f8ff41612d0 legacy=0x7f8ff41616f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:01.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.729+0000 7f8ff37fe700 1 -- 172.21.15.1:0/4271978060 >> v1:172.21.15.1:6789/0 conn(0x7f8ff4160e50 legacy=0x7f8ff40cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:01.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.729+0000 7f8ff27fc700 1 -- 172.21.15.1:0/4271978060 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989531718 0 0) 0x7f8fe4001110 con 0x7f8ff4160e50 2024-03-18T20:25:01.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.729+0000 7f8ff37fe700 1 -- 172.21.15.1:0/4271978060 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff40cf210 con 0x7f8ff4160a70 2024-03-18T20:25:01.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.729+0000 7f8ff37fe700 1 --2- 172.21.15.1:0/4271978060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff4160a70 0x7f8ff40d3b70 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc009840 tx=0x7f8fdc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce3d9a9da3f0ae11 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:01.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.729+0000 7f8ff27fc700 1 -- 172.21.15.1:0/4271978060 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc011830 con 0x7f8ff4160a70 2024-03-18T20:25:01.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.729+0000 7f8ff27fc700 1 -- 172.21.15.1:0/4271978060 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc011990 con 0x7f8ff4160a70 2024-03-18T20:25:01.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.729+0000 7f8ff27fc700 1 -- 172.21.15.1:0/4271978060 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc011d20 con 0x7f8ff4160a70 2024-03-18T20:25:01.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.730+0000 7f8ff9b45700 1 -- 172.21.15.1:0/4271978060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff4160a70 msgr2=0x7f8ff40d3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:01.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.730+0000 7f8ff9b45700 1 --2- 172.21.15.1:0/4271978060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff4160a70 0x7f8ff40d3b70 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc009840 tx=0x7f8fdc009b50 comp rx=0 tx=0).stop 2024-03-18T20:25:01.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.730+0000 7f8ff9b45700 1 -- 172.21.15.1:0/4271978060 shutdown_connections 2024-03-18T20:25:01.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.730+0000 7f8ff9b45700 1 --2- 172.21.15.1:0/4271978060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff4160a70 0x7f8ff40d3b70 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.730+0000 7f8ff9b45700 1 -- 172.21.15.1:0/4271978060 >> 172.21.15.1:0/4271978060 conn(0x7f8ff409c560 msgr2=0x7f8ff40d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:01.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.730+0000 7f8ff9b45700 1 -- 172.21.15.1:0/4271978060 shutdown_connections 2024-03-18T20:25:01.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.730+0000 7f8ff9b45700 1 -- 172.21.15.1:0/4271978060 wait complete. 2024-03-18T20:25:01.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.731+0000 7f8ff9b45700 1 Processor -- start 2024-03-18T20:25:01.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.731+0000 7f8ff9b45700 1 -- start start 2024-03-18T20:25:01.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.732+0000 7f8ff9b45700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff4160a70 0x7f8ff415eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:01.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.732+0000 7f8ff9b45700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff4160e50 0x7f8ff40cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:01.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.732+0000 7f8ff9b45700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ff41612d0 0x7f8ff40cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:01.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.732+0000 7f8ff9b45700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40b87f0 con 0x7f8ff4160a70 2024-03-18T20:25:01.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.732+0000 7f8ff9b45700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff415f670 con 0x7f8ff41612d0 2024-03-18T20:25:01.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.732+0000 7f8ff9b45700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40b7be0 con 0x7f8ff4160e50 2024-03-18T20:25:01.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.732+0000 7f8ff37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff4160a70 0x7f8ff415eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:01.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.732+0000 7f8ff37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff4160a70 0x7f8ff415eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33682/0 (socket says 172.21.15.1:33682) 2024-03-18T20:25:01.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.732+0000 7f8ff37fe700 1 -- 172.21.15.1:0/3397807534 learned_addr learned my addr 172.21.15.1:0/3397807534 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:01.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.732+0000 7f8ff2ffd700 1 --2- 172.21.15.1:0/3397807534 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff4160e50 0x7f8ff40cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:01.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.732+0000 7f8ff3fff700 1 --2- 172.21.15.1:0/3397807534 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ff41612d0 0x7f8ff40cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.732+0000 7f8ff37fe700 1 -- 172.21.15.1:0/3397807534 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff4160e50 msgr2=0x7f8ff40cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.732+0000 7f8ff37fe700 1 --2- 172.21.15.1:0/3397807534 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff4160e50 0x7f8ff40cc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.732+0000 7f8ff37fe700 1 -- 172.21.15.1:0/3397807534 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ff41612d0 msgr2=0x7f8ff40cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.732+0000 7f8ff37fe700 1 --2- 172.21.15.1:0/3397807534 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ff41612d0 0x7f8ff40cccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.732+0000 7f8ff37fe700 1 -- 172.21.15.1:0/3397807534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fdc0094a0 con 0x7f8ff4160a70 2024-03-18T20:25:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.732+0000 7f8ff3fff700 1 --2- 172.21.15.1:0/3397807534 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ff41612d0 0x7f8ff40cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.733+0000 7f8ff37fe700 1 --2- 172.21.15.1:0/3397807534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff4160a70 0x7f8ff415eac0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f8ff406afe0 tx=0x7f8fdc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.733+0000 7f8febfff700 1 -- 172.21.15.1:0/3397807534 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc0036a0 con 0x7f8ff4160a70 2024-03-18T20:25:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.733+0000 7f8febfff700 1 -- 172.21.15.1:0/3397807534 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc01f070 con 0x7f8ff4160a70 2024-03-18T20:25:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.733+0000 7f8febfff700 1 -- 172.21.15.1:0/3397807534 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc003a40 con 0x7f8ff4160a70 2024-03-18T20:25:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.733+0000 7f8ff9b45700 1 -- 172.21.15.1:0/3397807534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff40b7ec0 con 0x7f8ff4160a70 2024-03-18T20:25:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.733+0000 7f8ff9b45700 1 -- 172.21.15.1:0/3397807534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff40b8480 con 0x7f8ff4160a70 2024-03-18T20:25:01.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.734+0000 7f8ff9b45700 1 -- 172.21.15.1:0/3397807534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff4067390 con 0x7f8ff4160a70 2024-03-18T20:25:01.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.734+0000 7f8febfff700 1 -- 172.21.15.1:0/3397807534 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 130) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f8fdc003ba0 con 0x7f8ff4160a70 2024-03-18T20:25:01.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.735+0000 7f8febfff700 1 --2- 172.21.15.1:0/3397807534 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8fd406af20 0x7f8fd406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:01.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.735+0000 7f8febfff700 1 -- 172.21.15.1:0/3397807534 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f8fdc021070 con 0x7f8ff4160a70 2024-03-18T20:25:01.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.735+0000 7f8ff2ffd700 1 --2- 172.21.15.1:0/3397807534 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8fd406af20 0x7f8fd406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:01.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.735+0000 7f8ff2ffd700 1 --2- 172.21.15.1:0/3397807534 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8fd406af20 0x7f8fd406d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8ff40cc760 tx=0x7f8fe4007aa0 comp rx=0 tx=0).ready entity=mgr.11205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:01.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.740+0000 7f8febfff700 1 -- 172.21.15.1:0/3397807534 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8fdc056210 con 0x7f8ff4160a70 2024-03-18T20:25:01.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:01.967+0000 7f8ff9b45700 1 -- 172.21.15.1:0/3397807534 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ff40b1170 con 0x7f8fd406af20 2024-03-18T20:25:02.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.024+0000 7f8febfff700 1 -- 172.21.15.1:0/3397807534 <== mgr.11205 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f8ff40b1170 con 0x7f8fd406af20 2024-03-18T20:25:02.030 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule removed for path snap_test_dir1 2024-03-18T20:25:02.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.026+0000 7f8ff9b45700 1 -- 172.21.15.1:0/3397807534 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8fd406af20 msgr2=0x7f8fd406d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:02.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.026+0000 7f8ff9b45700 1 --2- 172.21.15.1:0/3397807534 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8fd406af20 0x7f8fd406d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8ff40cc760 tx=0x7f8fe4007aa0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.026+0000 7f8ff9b45700 1 -- 172.21.15.1:0/3397807534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff4160a70 msgr2=0x7f8ff415eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:02.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.026+0000 7f8ff9b45700 1 --2- 172.21.15.1:0/3397807534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff4160a70 0x7f8ff415eac0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f8ff406afe0 tx=0x7f8fdc004750 comp rx=0 tx=0).stop 2024-03-18T20:25:02.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.026+0000 7f8ff9b45700 1 -- 172.21.15.1:0/3397807534 shutdown_connections 2024-03-18T20:25:02.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.026+0000 7f8ff9b45700 1 --2- 172.21.15.1:0/3397807534 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8fd406af20 0x7f8fd406d3e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.026+0000 7f8ff9b45700 1 --2- 172.21.15.1:0/3397807534 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ff41612d0 0x7f8ff40cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.026+0000 7f8ff9b45700 1 --2- 172.21.15.1:0/3397807534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff4160a70 0x7f8ff415eac0 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.026+0000 7f8ff9b45700 1 --2- 172.21.15.1:0/3397807534 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ff4160e50 0x7f8ff40cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.026+0000 7f8ff9b45700 1 -- 172.21.15.1:0/3397807534 >> 172.21.15.1:0/3397807534 conn(0x7f8ff409c560 msgr2=0x7f8ff40d1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:02.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.027+0000 7f8ff9b45700 1 -- 172.21.15.1:0/3397807534 shutdown_connections 2024-03-18T20:25:02.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.027+0000 7f8ff9b45700 1 -- 172.21.15.1:0/3397807534 wait complete. 2024-03-18T20:25:02.044 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-03-18T20:25:02.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:02.044 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:25:02.065 INFO:teuthology.orchestra.run.smithi001.stdout:scheduled-2024-03-18-20_25_00_UTC 2024-03-18T20:25:02.066 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-03-18-20_25_00_UTC 2024-03-18T20:25:02.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:02.066 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-03-18-20_25_00_UTC) 2024-03-18T20:25:02.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:02.131 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-03-18T20:25:02.194 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-18T20:25:02.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.357+0000 7fd3b9814700 1 Processor -- start 2024-03-18T20:25:02.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.357+0000 7fd3b9814700 1 -- start start 2024-03-18T20:25:02.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.357+0000 7fd3b9814700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3b40cfb50 0x7fd3b40d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:02.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.358+0000 7fd3b9814700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd3b40cff30 0x7fd3b40b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:02.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.358+0000 7fd3b9814700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b40b3370 con 0x7fd3b40b3d40 2024-03-18T20:25:02.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.358+0000 7fd3b9814700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3b40b2760 con 0x7fd3b40cff30 2024-03-18T20:25:02.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.358+0000 7fd3b9814700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b40b2a40 con 0x7fd3b40cfb50 2024-03-18T20:25:02.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.358+0000 7fd3b37fe700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fd3b40b3d40 0x7fd3b40b41d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:43124/0 (socket says 172.21.15.1:43124) 2024-03-18T20:25:02.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.358+0000 7fd3b37fe700 1 -- 172.21.15.1:0/767449667 learned_addr learned my addr 172.21.15.1:0/767449667 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:02.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.358+0000 7fd3b2ffd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3b40cfb50 0x7fd3b40d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:02.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.358+0000 7fd3b27fc700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd3b40cff30 0x7fd3b40b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:02.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.358+0000 7fd3b1ffb700 1 -- 172.21.15.1:0/767449667 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3751729492 0 0) 0x7fd3b40b3370 con 0x7fd3b40b3d40 2024-03-18T20:25:02.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.358+0000 7fd3b1ffb700 1 -- 172.21.15.1:0/767449667 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd39c0035b0 con 0x7fd3b40b3d40 2024-03-18T20:25:02.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.358+0000 7fd3b27fc700 1 -- 172.21.15.1:0/767449667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3b40cfb50 msgr2=0x7fd3b40d3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:02.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.358+0000 7fd3b27fc700 1 --2- 172.21.15.1:0/767449667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3b40cfb50 0x7fd3b40d3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.358+0000 7fd3b27fc700 1 -- 172.21.15.1:0/767449667 >> v1:172.21.15.165:6789/0 conn(0x7fd3b40b3d40 legacy=0x7fd3b40b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:02.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.358+0000 7fd3b27fc700 1 -- 172.21.15.1:0/767449667 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b40b2d20 con 0x7fd3b40cff30 2024-03-18T20:25:02.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.358+0000 7fd3b2ffd700 1 --2- 172.21.15.1:0/767449667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3b40cfb50 0x7fd3b40d3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:02.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.359+0000 7fd3b27fc700 1 --2- 172.21.15.1:0/767449667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3b40cff30 0x7fd3b40b3800 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8009840 tx=0x7fd3a8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea11108be46a2eb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:02.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.359+0000 7fd3b1ffb700 1 -- 172.21.15.1:0/767449667 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8011830 con 0x7fd3b40cff30 2024-03-18T20:25:02.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.359+0000 7fd3b1ffb700 1 -- 172.21.15.1:0/767449667 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a8011990 con 0x7fd3b40cff30 2024-03-18T20:25:02.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.359+0000 7fd3b1ffb700 1 -- 172.21.15.1:0/767449667 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8011d00 con 0x7fd3b40cff30 2024-03-18T20:25:02.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.359+0000 7fd3b9814700 1 -- 172.21.15.1:0/767449667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3b40cff30 msgr2=0x7fd3b40b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:02.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.359+0000 7fd3b9814700 1 --2- 172.21.15.1:0/767449667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3b40cff30 0x7fd3b40b3800 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8009840 tx=0x7fd3a8009b50 comp rx=0 tx=0).stop 2024-03-18T20:25:02.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.359+0000 7fd3b9814700 1 -- 172.21.15.1:0/767449667 shutdown_connections 2024-03-18T20:25:02.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.359+0000 7fd3b9814700 1 --2- 172.21.15.1:0/767449667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3b40cfb50 0x7fd3b40d3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.359+0000 7fd3b9814700 1 --2- 172.21.15.1:0/767449667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3b40cff30 0x7fd3b40b3800 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.359+0000 7fd3b9814700 1 -- 172.21.15.1:0/767449667 >> 172.21.15.1:0/767449667 conn(0x7fd3b40d3140 msgr2=0x7fd3b40d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:02.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.359+0000 7fd3b9814700 1 -- 172.21.15.1:0/767449667 shutdown_connections 2024-03-18T20:25:02.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.359+0000 7fd3b9814700 1 -- 172.21.15.1:0/767449667 wait complete. 2024-03-18T20:25:02.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.361+0000 7fd3b9814700 1 Processor -- start 2024-03-18T20:25:02.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.361+0000 7fd3b9814700 1 -- start start 2024-03-18T20:25:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.362+0000 7fd3b9814700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3b40b3d40 0x7fd3b414d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.362+0000 7fd3b9814700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3b40cfb50 0x7fd3b415df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.362+0000 7fd3b9814700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b40cff30 0x7fd3b415e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.362+0000 7fd3b9814700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b40c77c0 con 0x7fd3b40cff30 2024-03-18T20:25:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.362+0000 7fd3b9814700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b40c7be0 con 0x7fd3b40b3d40 2024-03-18T20:25:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.362+0000 7fd3b9814700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b40c6b40 con 0x7fd3b40cfb50 2024-03-18T20:25:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.362+0000 7fd3b37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b40cff30 0x7fd3b415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.362+0000 7fd3b2ffd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3b40b3d40 0x7fd3b414d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.362+0000 7fd3b37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b40cff30 0x7fd3b415e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33698/0 (socket says 172.21.15.1:33698) 2024-03-18T20:25:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.362+0000 7fd3b37fe700 1 -- 172.21.15.1:0/3387664088 learned_addr learned my addr 172.21.15.1:0/3387664088 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.362+0000 7fd3b27fc700 1 --2- 172.21.15.1:0/3387664088 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3b40cfb50 0x7fd3b415df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.363+0000 7fd3b37fe700 1 -- 172.21.15.1:0/3387664088 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3b40cfb50 msgr2=0x7fd3b415df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.363+0000 7fd3b37fe700 1 --2- 172.21.15.1:0/3387664088 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3b40cfb50 0x7fd3b415df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.363+0000 7fd3b37fe700 1 -- 172.21.15.1:0/3387664088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3b40b3d40 msgr2=0x7fd3b414d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:02.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.363+0000 7fd3b37fe700 1 --2- 172.21.15.1:0/3387664088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3b40b3d40 0x7fd3b414d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:02.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.363+0000 7fd3b37fe700 1 -- 172.21.15.1:0/3387664088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a4009430 con 0x7fd3b40cff30 2024-03-18T20:25:02.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.363+0000 7fd3b2ffd700 1 --2- 172.21.15.1:0/3387664088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3b40b3d40 0x7fd3b414d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:02.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.363+0000 7fd3b37fe700 1 --2- 172.21.15.1:0/3387664088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b40cff30 0x7fd3b415e460 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac00cb40 tx=0x7fd3ac00ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:02.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.363+0000 7fd39bfff700 1 -- 172.21.15.1:0/3387664088 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac00a9e0 con 0x7fd3b40cff30 2024-03-18T20:25:02.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.363+0000 7fd39bfff700 1 -- 172.21.15.1:0/3387664088 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ac00ab40 con 0x7fd3b40cff30 2024-03-18T20:25:02.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.363+0000 7fd3b9814700 1 -- 172.21.15.1:0/3387664088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a80094a0 con 0x7fd3b40cff30 2024-03-18T20:25:02.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.363+0000 7fd39bfff700 1 -- 172.21.15.1:0/3387664088 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac012400 con 0x7fd3b40cff30 2024-03-18T20:25:02.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.363+0000 7fd3b9814700 1 -- 172.21.15.1:0/3387664088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b40c7100 con 0x7fd3b40cff30 2024-03-18T20:25:02.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.364+0000 7fd3b9814700 1 -- 172.21.15.1:0/3387664088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b40c7900 con 0x7fd3b40cff30 2024-03-18T20:25:02.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.365+0000 7fd39bfff700 1 -- 172.21.15.1:0/3387664088 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 130) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fd3ac01b3b0 con 0x7fd3b40cff30 2024-03-18T20:25:02.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.365+0000 7fd39bfff700 1 --2- 172.21.15.1:0/3387664088 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd3a006af20 0x7fd3a006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:02.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.365+0000 7fd39bfff700 1 -- 172.21.15.1:0/3387664088 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6560+0+0 (secure 0 0 0) 0x7fd3ac0896a0 con 0x7fd3b40cff30 2024-03-18T20:25:02.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.365+0000 7fd3b2ffd700 1 --2- 172.21.15.1:0/3387664088 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd3a006af20 0x7fd3a006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:02.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.365+0000 7fd3b2ffd700 1 --2- 172.21.15.1:0/3387664088 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd3a006af20 0x7fd3a006d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40114b0 tx=0x7fd3a40091c0 comp rx=0 tx=0).ready entity=mgr.11205 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:02.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.370+0000 7fd39bfff700 1 -- 172.21.15.1:0/3387664088 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd3b40c7900 con 0x7fd3b40cff30 2024-03-18T20:25:02.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:02.635+0000 7fd3b9814700 1 -- 172.21.15.1:0/3387664088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fd3b40c7900 con 0x7fd3b40cff30 2024-03-18T20:25:03.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.179+0000 7fd39bfff700 1 -- 172.21.15.1:0/3387664088 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 131) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fd3ac0545d0 con 0x7fd3b40cff30 2024-03-18T20:25:03.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.180+0000 7fd39bfff700 1 -- 172.21.15.1:0/3387664088 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v131) v1 ==== 93+0+0 (secure 0 0 0) 0x7fd3ac0844e0 con 0x7fd3b40cff30 2024-03-18T20:25:03.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.183+0000 7fd3b9814700 1 -- 172.21.15.1:0/3387664088 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd3a006af20 msgr2=0x7fd3a006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.183+0000 7fd3b9814700 1 --2- 172.21.15.1:0/3387664088 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd3a006af20 0x7fd3a006d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40114b0 tx=0x7fd3a40091c0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.183+0000 7fd3b9814700 1 -- 172.21.15.1:0/3387664088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b40cff30 msgr2=0x7fd3b415e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.183+0000 7fd3b9814700 1 --2- 172.21.15.1:0/3387664088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b40cff30 0x7fd3b415e460 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac00cb40 tx=0x7fd3ac00ce50 comp rx=0 tx=0).stop 2024-03-18T20:25:03.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.183+0000 7fd3b9814700 1 -- 172.21.15.1:0/3387664088 shutdown_connections 2024-03-18T20:25:03.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.183+0000 7fd3b9814700 1 --2- 172.21.15.1:0/3387664088 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd3a006af20 0x7fd3a006d3e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.183+0000 7fd3b9814700 1 --2- 172.21.15.1:0/3387664088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b40cff30 0x7fd3b415e460 unknown :-1 s=CLOSED pgs=2404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.183+0000 7fd3b9814700 1 --2- 172.21.15.1:0/3387664088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3b40b3d40 0x7fd3b414d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.183+0000 7fd3b9814700 1 --2- 172.21.15.1:0/3387664088 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd3b40cfb50 0x7fd3b415df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.183+0000 7fd3b9814700 1 -- 172.21.15.1:0/3387664088 >> 172.21.15.1:0/3387664088 conn(0x7fd3b40d3140 msgr2=0x7fd3b40d1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:03.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.183+0000 7fd3b9814700 1 -- 172.21.15.1:0/3387664088 shutdown_connections 2024-03-18T20:25:03.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.183+0000 7fd3b9814700 1 -- 172.21.15.1:0/3387664088 wait complete. 2024-03-18T20:25:03.203 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T20:25:03.203 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:25:03.257 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:25:03.294 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T20:25:03.294 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:25:03.294 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:25:03.313 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:25:03.330 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:03.327+0000 7f7b493a2240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:25:03.339 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:03.334+0000 7f25953da240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:25:03.477 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:03.473+0000 7f7b493a2240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:25:03.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.496+0000 7f30acf79700 1 Processor -- start 2024-03-18T20:25:03.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.497+0000 7f30acf79700 1 -- start start 2024-03-18T20:25:03.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.499+0000 7f30acf79700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f30a8154730 0x7f30a80aeb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.499+0000 7f30acf79700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f30a80b35f0 0x7f30a8071250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.499+0000 7f30acf79700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a8071790 con 0x7f30a8070dd0 2024-03-18T20:25:03.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.499+0000 7f30acf79700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a80b49d0 con 0x7f30a80b35f0 2024-03-18T20:25:03.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.499+0000 7f30acf79700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a80af070 con 0x7f30a8154730 2024-03-18T20:25:03.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.500+0000 7f30a659c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f30a8154730 0x7f30a80aeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.500+0000 7f30a659c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f30a8154730 0x7f30a80aeb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:55648/0 (socket says 172.21.15.1:55648) 2024-03-18T20:25:03.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.500+0000 7f30a659c700 1 -- 172.21.15.1:0/3461957301 learned_addr learned my addr 172.21.15.1:0/3461957301 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:03.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.500+0000 7f30a6d9d700 1 --2- 172.21.15.1:0/3461957301 >> v2:172.21.15.1:3300/0 conn(0x7f30a80b35f0 0x7f30a8071250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.500+0000 7f30a659c700 1 -- 172.21.15.1:0/3461957301 >> v2:172.21.15.1:3300/0 conn(0x7f30a80b35f0 msgr2=0x7f30a8071250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:03.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.500+0000 7f30a659c700 1 --2- 172.21.15.1:0/3461957301 >> v2:172.21.15.1:3300/0 conn(0x7f30a80b35f0 0x7f30a8071250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.500+0000 7f30a659c700 1 -- 172.21.15.1:0/3461957301 >> v1:172.21.15.165:6789/0 conn(0x7f30a8070dd0 legacy=0x7f30a815db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.500+0000 7f30a559a700 1 -- 172.21.15.1:0/3461957301 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963780914 0 0) 0x7f30a8071790 con 0x7f30a8070dd0 2024-03-18T20:25:03.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.500+0000 7f30a659c700 1 -- 172.21.15.1:0/3461957301 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a806d140 con 0x7f30a8154730 2024-03-18T20:25:03.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.500+0000 7f30a6d9d700 1 --2- 172.21.15.1:0/3461957301 >> v2:172.21.15.1:3300/0 conn(0x7f30a80b35f0 0x7f30a8071250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:03.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.501+0000 7f30a659c700 1 --2- 172.21.15.1:0/3461957301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30a8154730 0x7f30a80aeb30 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f30a8068860 tx=0x7f3098009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14f79e74c5aa03fb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:03.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.501+0000 7f30a559a700 1 -- 172.21.15.1:0/3461957301 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3098011830 con 0x7f30a8154730 2024-03-18T20:25:03.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.501+0000 7f30a559a700 1 -- 172.21.15.1:0/3461957301 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3098011990 con 0x7f30a8154730 2024-03-18T20:25:03.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.501+0000 7f30a559a700 1 -- 172.21.15.1:0/3461957301 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3098011d00 con 0x7f30a8154730 2024-03-18T20:25:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.502+0000 7f30acf79700 1 -- 172.21.15.1:0/3461957301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30a8154730 msgr2=0x7f30a80aeb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.502+0000 7f30acf79700 1 --2- 172.21.15.1:0/3461957301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30a8154730 0x7f30a80aeb30 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f30a8068860 tx=0x7f3098009bd0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.502+0000 7f30acf79700 1 -- 172.21.15.1:0/3461957301 shutdown_connections 2024-03-18T20:25:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.502+0000 7f30acf79700 1 --2- 172.21.15.1:0/3461957301 >> v2:172.21.15.1:3300/0 conn(0x7f30a80b35f0 0x7f30a8071250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.502+0000 7f30acf79700 1 --2- 172.21.15.1:0/3461957301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30a8154730 0x7f30a80aeb30 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.502+0000 7f30acf79700 1 -- 172.21.15.1:0/3461957301 >> 172.21.15.1:0/3461957301 conn(0x7f30a8066960 msgr2=0x7f30a806a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.502+0000 7f30acf79700 1 -- 172.21.15.1:0/3461957301 shutdown_connections 2024-03-18T20:25:03.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.503+0000 7f30acf79700 1 -- 172.21.15.1:0/3461957301 wait complete. 2024-03-18T20:25:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.506+0000 7f30acf79700 1 Processor -- start 2024-03-18T20:25:03.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.506+0000 7f30acf79700 1 -- start start 2024-03-18T20:25:03.511 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:03.507+0000 7f25953da240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:25:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.507+0000 7f30acf79700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30a8070dd0 0x7f30a81580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.507+0000 7f30acf79700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30a80b35f0 0x7f30a815aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.507+0000 7f30acf79700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30a8154730 0x7f30a8144430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.507+0000 7f30acf79700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a815afa0 con 0x7f30a80b35f0 2024-03-18T20:25:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.507+0000 7f30acf79700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a81496a0 con 0x7f30a8154730 2024-03-18T20:25:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.507+0000 7f30acf79700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a8158960 con 0x7f30a8070dd0 2024-03-18T20:25:03.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.507+0000 7f30a5d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30a80b35f0 0x7f30a815aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.507+0000 7f30a659c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30a8070dd0 0x7f30a81580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.507+0000 7f30a5d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30a80b35f0 0x7f30a815aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33716/0 (socket says 172.21.15.1:33716) 2024-03-18T20:25:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.507+0000 7f30a5d9b700 1 -- 172.21.15.1:0/2213028466 learned_addr learned my addr 172.21.15.1:0/2213028466 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.507+0000 7f30a6d9d700 1 --2- 172.21.15.1:0/2213028466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30a8154730 0x7f30a8144430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.508+0000 7f30a5d9b700 1 -- 172.21.15.1:0/2213028466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30a8070dd0 msgr2=0x7f30a81580f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.508+0000 7f30a5d9b700 1 --2- 172.21.15.1:0/2213028466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30a8070dd0 0x7f30a81580f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.508+0000 7f30a5d9b700 1 -- 172.21.15.1:0/2213028466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30a8154730 msgr2=0x7f30a8144430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.508+0000 7f30a5d9b700 1 --2- 172.21.15.1:0/2213028466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30a8154730 0x7f30a8144430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.508+0000 7f30a5d9b700 1 -- 172.21.15.1:0/2213028466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a0009a70 con 0x7f30a80b35f0 2024-03-18T20:25:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.508+0000 7f30a6d9d700 1 --2- 172.21.15.1:0/2213028466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30a8154730 0x7f30a8144430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.508+0000 7f30a5d9b700 1 --2- 172.21.15.1:0/2213028466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30a80b35f0 0x7f30a815aa60 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f309c00c550 tx=0x7f309c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.508+0000 7f3096ffd700 1 -- 172.21.15.1:0/2213028466 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c0054e0 con 0x7f30a80b35f0 2024-03-18T20:25:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.508+0000 7f3096ffd700 1 -- 172.21.15.1:0/2213028466 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c003680 con 0x7f30a80b35f0 2024-03-18T20:25:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.508+0000 7f30acf79700 1 -- 172.21.15.1:0/2213028466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3098009550 con 0x7f30a80b35f0 2024-03-18T20:25:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.508+0000 7f3096ffd700 1 -- 172.21.15.1:0/2213028466 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c003a30 con 0x7f30a80b35f0 2024-03-18T20:25:03.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.508+0000 7f30acf79700 1 -- 172.21.15.1:0/2213028466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a80b1e30 con 0x7f30a80b35f0 2024-03-18T20:25:03.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.509+0000 7f30acf79700 1 -- 172.21.15.1:0/2213028466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a806b6e0 con 0x7f30a80b35f0 2024-03-18T20:25:03.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.510+0000 7f3096ffd700 1 -- 172.21.15.1:0/2213028466 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 131) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f309c003b90 con 0x7f30a80b35f0 2024-03-18T20:25:03.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.510+0000 7f3096ffd700 1 --2- 172.21.15.1:0/2213028466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f308806aed0 0x7f308806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.510+0000 7f3096ffd700 1 -- 172.21.15.1:0/2213028466 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f309c088f40 con 0x7f30a80b35f0 2024-03-18T20:25:03.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.510+0000 7f30a659c700 1 -- 172.21.15.1:0/2213028466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f308806aed0 msgr2=0x7f308806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:03.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.510+0000 7f30a659c700 1 --2- 172.21.15.1:0/2213028466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f308806aed0 0x7f308806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:03.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.515+0000 7f3096ffd700 1 -- 172.21.15.1:0/2213028466 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f309c0539f0 con 0x7f30a80b35f0 2024-03-18T20:25:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.711+0000 7f30a659c700 1 -- 172.21.15.1:0/2213028466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f308806aed0 msgr2=0x7f308806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.711+0000 7f30a659c700 1 --2- 172.21.15.1:0/2213028466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f308806aed0 0x7f308806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.718+0000 7f30acf79700 1 -- 172.21.15.1:0/2213028466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30a8071b50 con 0x7f30a80b35f0 2024-03-18T20:25:03.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.719+0000 7f3096ffd700 1 -- 172.21.15.1:0/2213028466 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v217) v1 ==== 74+0+16990 (secure 0 0 0) 0x7f309c017090 con 0x7f30a80b35f0 2024-03-18T20:25:03.725 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:03.725 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":217,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:24:56.101238+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:23:37.591406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-03-18T20:23:38.723169+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"214","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.c","mappings":[{"from":1,"to":0}]},{"pgid":"26.f","mappings":[{"from":1,"to":0}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2462726896":"2024-03-19T20:23:54.844816+0000","172.21.15.1:6833/148356":"2024-03-19T20:23:54.844816+0000","172.21.15.1:6832/148356":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/4177404804":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/1122038130":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/2416229963":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/2662947140":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/2110350357":"2024-03-19T20:23:54.844816+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:03.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.721+0000 7f30acf79700 1 -- 172.21.15.1:0/2213028466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f308806aed0 msgr2=0x7f308806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:03.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.721+0000 7f30acf79700 1 --2- 172.21.15.1:0/2213028466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f308806aed0 0x7f308806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.721+0000 7f30acf79700 1 -- 172.21.15.1:0/2213028466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30a80b35f0 msgr2=0x7f30a815aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.721+0000 7f30acf79700 1 --2- 172.21.15.1:0/2213028466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30a80b35f0 0x7f30a815aa60 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f309c00c550 tx=0x7f309c00c860 comp rx=0 tx=0).stop 2024-03-18T20:25:03.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.721+0000 7f30acf79700 1 -- 172.21.15.1:0/2213028466 shutdown_connections 2024-03-18T20:25:03.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.721+0000 7f30acf79700 1 --2- 172.21.15.1:0/2213028466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f308806aed0 0x7f308806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.721+0000 7f30acf79700 1 --2- 172.21.15.1:0/2213028466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30a8070dd0 0x7f30a81580f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.721+0000 7f30acf79700 1 --2- 172.21.15.1:0/2213028466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30a8154730 0x7f30a8144430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.721+0000 7f30acf79700 1 --2- 172.21.15.1:0/2213028466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30a80b35f0 0x7f30a815aa60 unknown :-1 s=CLOSED pgs=2407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.721+0000 7f30acf79700 1 -- 172.21.15.1:0/2213028466 >> 172.21.15.1:0/2213028466 conn(0x7f30a8066960 msgr2=0x7f30a815cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.722+0000 7f30acf79700 1 -- 172.21.15.1:0/2213028466 shutdown_connections 2024-03-18T20:25:03.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.722+0000 7f30acf79700 1 -- 172.21.15.1:0/2213028466 wait complete. 2024-03-18T20:25:03.739 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:23:37.591406+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '213', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:23:38.723169+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '214', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:25:03.740 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:25:03.742 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:03.738+0000 7f7b493a2240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:25:03.794 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:03.789+0000 7f25953da240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:25:03.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.865+0000 7f0ccf9ac700 1 Processor -- start 2024-03-18T20:25:03.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.866+0000 7f0ccf9ac700 1 -- start start 2024-03-18T20:25:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.866+0000 7f0ccf9ac700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0cc815ad70 0x7f0cc815d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.866+0000 7f0ccf9ac700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc8160ea0 con 0x7f0cc80727b0 2024-03-18T20:25:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.866+0000 7f0ccf9ac700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc815ddf0 con 0x7f0cc806a750 2024-03-18T20:25:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.866+0000 7f0ccf9ac700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc80725b0 con 0x7f0cc815ad70 2024-03-18T20:25:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0cccf47700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0cc80727b0 0x7f0cc8161610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58112/0 (socket says 172.21.15.1:58112) 2024-03-18T20:25:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0cccf47700 1 -- 172.21.15.1:0/2803714122 learned_addr learned my addr 172.21.15.1:0/2803714122 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0ccd748700 1 --2- 172.21.15.1:0/2803714122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0cc815ad70 0x7f0cc815d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0cbbfff700 1 -- 172.21.15.1:0/2803714122 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786744858 0 0) 0x7f0cc8160ea0 con 0x7f0cc80727b0 2024-03-18T20:25:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0cbbfff700 1 -- 172.21.15.1:0/2803714122 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cb40035b0 con 0x7f0cc80727b0 2024-03-18T20:25:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0cbbfff700 1 -- 172.21.15.1:0/2803714122 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006736122 0 0) 0x7f0cc815ddf0 con 0x7f0cc806a750 2024-03-18T20:25:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0cbbfff700 1 -- 172.21.15.1:0/2803714122 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cc8160ea0 con 0x7f0cc806a750 2024-03-18T20:25:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0cbbfff700 1 -- 172.21.15.1:0/2803714122 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 669066058 0 0) 0x7f0cb40035b0 con 0x7f0cc80727b0 2024-03-18T20:25:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0cbbfff700 1 -- 172.21.15.1:0/2803714122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0cc815ad70 msgr2=0x7f0cc815d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0cbbfff700 1 --2- 172.21.15.1:0/2803714122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0cc815ad70 0x7f0cc815d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0cbbfff700 1 -- 172.21.15.1:0/2803714122 >> v1:172.21.15.165:6789/0 conn(0x7f0cc806a750 legacy=0x7f0cc806eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0cbbfff700 1 -- 172.21.15.1:0/2803714122 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc815e090 con 0x7f0cc80727b0 2024-03-18T20:25:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0cbbfff700 1 -- 172.21.15.1:0/2803714122 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f0cbc001670 con 0x7f0cc80727b0 2024-03-18T20:25:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0ccd748700 1 --2- 172.21.15.1:0/2803714122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0cc815ad70 0x7f0cc815d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:03.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0cbbfff700 1 -- 172.21.15.1:0/2803714122 >> v1:172.21.15.1:6789/0 conn(0x7f0cc80727b0 legacy=0x7f0cc8161610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0cbbfff700 1 --2- 172.21.15.1:0/2803714122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cb4003bc0 0x7f0cb4003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0cbbfff700 1 --2- 172.21.15.1:0/2803714122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0cc815ad70 0x7f0cb4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0cbbfff700 1 --2- 172.21.15.1:0/2803714122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cb4004d90 0x7f0cb4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0cbbfff700 1 -- 172.21.15.1:0/2803714122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb400bac0 con 0x7f0cb4004d90 2024-03-18T20:25:03.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0cbbfff700 1 -- 172.21.15.1:0/2803714122 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb400bc30 con 0x7f0cb4003bc0 2024-03-18T20:25:03.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0cbbfff700 1 -- 172.21.15.1:0/2803714122 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb400bdf0 con 0x7f0cc815ad70 2024-03-18T20:25:03.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.867+0000 7f0ccd748700 1 --2- 172.21.15.1:0/2803714122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cb4003bc0 0x7f0cb4003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.868+0000 7f0cccf47700 1 --2- 172.21.15.1:0/2803714122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0cc815ad70 0x7f0cb4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.868+0000 7f0ccdf49700 1 --2- 172.21.15.1:0/2803714122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cb4004d90 0x7f0cb4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.868+0000 7f0ccd748700 1 -- 172.21.15.1:0/2803714122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0cc815ad70 msgr2=0x7f0cb4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:03.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.868+0000 7f0ccd748700 1 --2- 172.21.15.1:0/2803714122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0cc815ad70 0x7f0cb4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.868+0000 7f0ccd748700 1 -- 172.21.15.1:0/2803714122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cb4004d90 msgr2=0x7f0cb4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:03.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.868+0000 7f0ccd748700 1 --2- 172.21.15.1:0/2803714122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cb4004d90 0x7f0cb4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.868+0000 7f0ccd748700 1 -- 172.21.15.1:0/2803714122 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc40094a0 con 0x7f0cb4003bc0 2024-03-18T20:25:03.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.868+0000 7f0ccd748700 1 --2- 172.21.15.1:0/2803714122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cb4003bc0 0x7f0cb4003fc0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4009d30 tx=0x7f0cb4005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1609663eae0a2378 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:03.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.868+0000 7f0cbbfff700 1 -- 172.21.15.1:0/2803714122 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4009e80 con 0x7f0cb4003bc0 2024-03-18T20:25:03.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.868+0000 7f0cbbfff700 1 -- 172.21.15.1:0/2803714122 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc400e9a0 con 0x7f0cb4003bc0 2024-03-18T20:25:03.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.868+0000 7f0cbbfff700 1 -- 172.21.15.1:0/2803714122 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc400ed10 con 0x7f0cb4003bc0 2024-03-18T20:25:03.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.869+0000 7f0ccf9ac700 1 -- 172.21.15.1:0/2803714122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cb4003bc0 msgr2=0x7f0cb4003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.869+0000 7f0ccf9ac700 1 --2- 172.21.15.1:0/2803714122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cb4003bc0 0x7f0cb4003fc0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4009d30 tx=0x7f0cb4005880 comp rx=0 tx=0).stop 2024-03-18T20:25:03.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.869+0000 7f0ccdf49700 1 -- 172.21.15.1:0/2803714122 reap_dead start 2024-03-18T20:25:03.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.869+0000 7f0ccf9ac700 1 -- 172.21.15.1:0/2803714122 shutdown_connections 2024-03-18T20:25:03.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.869+0000 7f0ccf9ac700 1 -- 172.21.15.1:0/2803714122 >> 172.21.15.1:0/2803714122 conn(0x7f0cc80b1cb0 msgr2=0x7f0cc80b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:03.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.869+0000 7f0ccf9ac700 1 -- 172.21.15.1:0/2803714122 shutdown_connections 2024-03-18T20:25:03.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.869+0000 7f0ccf9ac700 1 -- 172.21.15.1:0/2803714122 wait complete. 2024-03-18T20:25:03.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.870+0000 7f0ccf9ac700 1 Processor -- start 2024-03-18T20:25:03.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.870+0000 7f0ccf9ac700 1 -- start start 2024-03-18T20:25:03.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.871+0000 7f0ccf9ac700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cb4003bc0 0x7f0cc81442f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.871+0000 7f0ccf9ac700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0cc815f6a0 0x7f0cc815fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.871+0000 7f0ccf9ac700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cc80afa70 0x7f0cc80afef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.871+0000 7f0ccf9ac700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc81f77a0 con 0x7f0cb4003bc0 2024-03-18T20:25:03.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.871+0000 7f0ccf9ac700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc80b0430 con 0x7f0cc80afa70 2024-03-18T20:25:03.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.871+0000 7f0ccf9ac700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc81f6840 con 0x7f0cc815f6a0 2024-03-18T20:25:03.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.871+0000 7f0ccd748700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cb4003bc0 0x7f0cc81442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.871+0000 7f0ccd748700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cb4003bc0 0x7f0cc81442f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33738/0 (socket says 172.21.15.1:33738) 2024-03-18T20:25:03.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.871+0000 7f0ccd748700 1 -- 172.21.15.1:0/4215387502 learned_addr learned my addr 172.21.15.1:0/4215387502 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:03.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.871+0000 7f0cccf47700 1 --2- 172.21.15.1:0/4215387502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0cc815f6a0 0x7f0cc815fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.871+0000 7f0ccdf49700 1 --2- 172.21.15.1:0/4215387502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cc80afa70 0x7f0cc80afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:03.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.871+0000 7f0ccd748700 1 -- 172.21.15.1:0/4215387502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0cc815f6a0 msgr2=0x7f0cc815fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.871+0000 7f0ccd748700 1 --2- 172.21.15.1:0/4215387502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0cc815f6a0 0x7f0cc815fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.871+0000 7f0ccd748700 1 -- 172.21.15.1:0/4215387502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cc80afa70 msgr2=0x7f0cc80afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:03.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.871+0000 7f0ccd748700 1 --2- 172.21.15.1:0/4215387502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cc80afa70 0x7f0cc80afef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:03.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.871+0000 7f0ccd748700 1 -- 172.21.15.1:0/4215387502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc81f6b20 con 0x7f0cb4003bc0 2024-03-18T20:25:03.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.872+0000 7f0ccd748700 1 --2- 172.21.15.1:0/4215387502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cb4003bc0 0x7f0cc81442f0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f0cb400bdc0 tx=0x7f0cc40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:03.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.872+0000 7f0cb9ffb700 1 -- 172.21.15.1:0/4215387502 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4003ab0 con 0x7f0cb4003bc0 2024-03-18T20:25:03.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.872+0000 7f0cb9ffb700 1 -- 172.21.15.1:0/4215387502 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc4021070 con 0x7f0cb4003bc0 2024-03-18T20:25:03.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.872+0000 7f0ccf9ac700 1 -- 172.21.15.1:0/4215387502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc81f6e00 con 0x7f0cb4003bc0 2024-03-18T20:25:03.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.872+0000 7f0cb9ffb700 1 -- 172.21.15.1:0/4215387502 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4003e70 con 0x7f0cb4003bc0 2024-03-18T20:25:03.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.872+0000 7f0ccf9ac700 1 -- 172.21.15.1:0/4215387502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc81f73c0 con 0x7f0cb4003bc0 2024-03-18T20:25:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.873+0000 7f0ccf9ac700 1 -- 172.21.15.1:0/4215387502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc81f7730 con 0x7f0cb4003bc0 2024-03-18T20:25:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.874+0000 7f0cb9ffb700 1 -- 172.21.15.1:0/4215387502 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 131) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f0cc4028480 con 0x7f0cb4003bc0 2024-03-18T20:25:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.874+0000 7f0cb9ffb700 1 --2- 172.21.15.1:0/4215387502 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0cac06aed0 0x7f0cac06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.874+0000 7f0cccf47700 1 -- 172.21.15.1:0/4215387502 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0cac06aed0 msgr2=0x7f0cac06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.874+0000 7f0cb9ffb700 1 -- 172.21.15.1:0/4215387502 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f0cc4023070 con 0x7f0cb4003bc0 2024-03-18T20:25:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.874+0000 7f0cccf47700 1 --2- 172.21.15.1:0/4215387502 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0cac06aed0 0x7f0cac06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:03.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:03.879+0000 7f0cb9ffb700 1 -- 172.21.15.1:0/4215387502 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0cc4055f00 con 0x7f0cb4003bc0 2024-03-18T20:25:04.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.075+0000 7f0cccf47700 1 -- 172.21.15.1:0/4215387502 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0cac06aed0 msgr2=0x7f0cac06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:04.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.075+0000 7f0cccf47700 1 --2- 172.21.15.1:0/4215387502 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0cac06aed0 0x7f0cac06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:04.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.080+0000 7f0ccf9ac700 1 -- 172.21.15.1:0/4215387502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0cc80676a0 con 0x7f0cb4003bc0 2024-03-18T20:25:04.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.080+0000 7f0cb9ffb700 1 -- 172.21.15.1:0/4215387502 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v217) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0cc4059520 con 0x7f0cb4003bc0 2024-03-18T20:25:04.085 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:25:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.084+0000 7f0ccf9ac700 1 -- 172.21.15.1:0/4215387502 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0cac06aed0 msgr2=0x7f0cac06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.084+0000 7f0ccf9ac700 1 --2- 172.21.15.1:0/4215387502 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0cac06aed0 0x7f0cac06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.084+0000 7f0ccf9ac700 1 -- 172.21.15.1:0/4215387502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cb4003bc0 msgr2=0x7f0cc81442f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.084+0000 7f0ccf9ac700 1 --2- 172.21.15.1:0/4215387502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cb4003bc0 0x7f0cc81442f0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f0cb400bdc0 tx=0x7f0cc40047c0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.084+0000 7f0ccf9ac700 1 -- 172.21.15.1:0/4215387502 shutdown_connections 2024-03-18T20:25:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.084+0000 7f0ccf9ac700 1 --2- 172.21.15.1:0/4215387502 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0cac06aed0 0x7f0cac06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.084+0000 7f0ccf9ac700 1 --2- 172.21.15.1:0/4215387502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0cc80afa70 0x7f0cc80afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.084+0000 7f0ccf9ac700 1 --2- 172.21.15.1:0/4215387502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cb4003bc0 0x7f0cc81442f0 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.084+0000 7f0ccf9ac700 1 --2- 172.21.15.1:0/4215387502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0cc815f6a0 0x7f0cc815fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.084+0000 7f0ccf9ac700 1 -- 172.21.15.1:0/4215387502 >> 172.21.15.1:0/4215387502 conn(0x7f0cc80b1cb0 msgr2=0x7f0cc80b11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.084+0000 7f0ccf9ac700 1 -- 172.21.15.1:0/4215387502 shutdown_connections 2024-03-18T20:25:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.084+0000 7f0ccf9ac700 1 -- 172.21.15.1:0/4215387502 wait complete. 2024-03-18T20:25:04.107 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:25:04.166 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:04.162+0000 7f7b493a2240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:25:04.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.232+0000 7f72321c2700 1 Processor -- start 2024-03-18T20:25:04.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.233+0000 7f72321c2700 1 -- start start 2024-03-18T20:25:04.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.233+0000 7f72321c2700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f722c0b1e20 0x7f722c0cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:04.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.233+0000 7f72321c2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c0cbce0 0x7f722c0cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:04.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.233+0000 7f72321c2700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f722c0d0df0 con 0x7f722c0cd870 2024-03-18T20:25:04.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.233+0000 7f72321c2700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f722c0d0550 con 0x7f722c0b1e20 2024-03-18T20:25:04.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.233+0000 7f72321c2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f722c0d0830 con 0x7f722c0cbce0 2024-03-18T20:25:04.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.234+0000 7f722b7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f722c0cd870 0x7f722c0cdc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58122/0 (socket says 172.21.15.1:58122) 2024-03-18T20:25:04.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.234+0000 7f722b7fe700 1 -- 172.21.15.1:0/3022646550 learned_addr learned my addr 172.21.15.1:0/3022646550 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:04.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.234+0000 7f722affd700 1 --2- 172.21.15.1:0/3022646550 >> v2:172.21.15.165:3300/0 conn(0x7f722c0b1e20 0x7f722c0cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:04.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.234+0000 7f722bfff700 1 --2- 172.21.15.1:0/3022646550 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c0cbce0 0x7f722c0cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:04.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.234+0000 7f722a7fc700 1 -- 172.21.15.1:0/3022646550 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592554607 0 0) 0x7f722c0d0df0 con 0x7f722c0cd870 2024-03-18T20:25:04.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.234+0000 7f722a7fc700 1 -- 172.21.15.1:0/3022646550 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72140035b0 con 0x7f722c0cd870 2024-03-18T20:25:04.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.234+0000 7f722bfff700 1 -- 172.21.15.1:0/3022646550 >> v2:172.21.15.165:3300/0 conn(0x7f722c0b1e20 msgr2=0x7f722c0cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:04.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.234+0000 7f722bfff700 1 --2- 172.21.15.1:0/3022646550 >> v2:172.21.15.165:3300/0 conn(0x7f722c0b1e20 0x7f722c0cff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.234+0000 7f722bfff700 1 -- 172.21.15.1:0/3022646550 >> v1:172.21.15.1:6789/0 conn(0x7f722c0cd870 legacy=0x7f722c0cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:04.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.234+0000 7f722bfff700 1 -- 172.21.15.1:0/3022646550 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722c0d0b10 con 0x7f722c0cbce0 2024-03-18T20:25:04.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.234+0000 7f722affd700 1 --2- 172.21.15.1:0/3022646550 >> v2:172.21.15.165:3300/0 conn(0x7f722c0b1e20 0x7f722c0cff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:04.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.234+0000 7f722bfff700 1 --2- 172.21.15.1:0/3022646550 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c0cbce0 0x7f722c0cc170 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7224009d90 tx=0x7f722400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c0c75babbd979f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:04.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.234+0000 7f722a7fc700 1 -- 172.21.15.1:0/3022646550 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722400e070 con 0x7f722c0cbce0 2024-03-18T20:25:04.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.234+0000 7f722a7fc700 1 -- 172.21.15.1:0/3022646550 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722400bb20 con 0x7f722c0cbce0 2024-03-18T20:25:04.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.234+0000 7f722a7fc700 1 -- 172.21.15.1:0/3022646550 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7224012430 con 0x7f722c0cbce0 2024-03-18T20:25:04.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.235+0000 7f72321c2700 1 -- 172.21.15.1:0/3022646550 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c0cbce0 msgr2=0x7f722c0cc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:04.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.235+0000 7f72321c2700 1 --2- 172.21.15.1:0/3022646550 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c0cbce0 0x7f722c0cc170 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7224009d90 tx=0x7f722400b330 comp rx=0 tx=0).stop 2024-03-18T20:25:04.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.235+0000 7f72321c2700 1 -- 172.21.15.1:0/3022646550 shutdown_connections 2024-03-18T20:25:04.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.235+0000 7f72321c2700 1 --2- 172.21.15.1:0/3022646550 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c0cbce0 0x7f722c0cc170 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.235+0000 7f72321c2700 1 --2- 172.21.15.1:0/3022646550 >> v2:172.21.15.165:3300/0 conn(0x7f722c0b1e20 0x7f722c0cff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.235+0000 7f72321c2700 1 -- 172.21.15.1:0/3022646550 >> 172.21.15.1:0/3022646550 conn(0x7f722c0d0ff0 msgr2=0x7f722c0d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:04.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.235+0000 7f72321c2700 1 -- 172.21.15.1:0/3022646550 shutdown_connections 2024-03-18T20:25:04.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.235+0000 7f72321c2700 1 -- 172.21.15.1:0/3022646550 wait complete. 2024-03-18T20:25:04.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.236+0000 7f72321c2700 1 Processor -- start 2024-03-18T20:25:04.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.236+0000 7f72321c2700 1 -- start start 2024-03-18T20:25:04.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.237+0000 7f72321c2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f722c0b1e20 0x7f722c1e4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:04.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.237+0000 7f72321c2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c0cbce0 0x7f722c1579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.237+0000 7f72321c2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f722c0cd870 0x7f722c15a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.237+0000 7f72321c2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c1e4db0 con 0x7f722c0cd870 2024-03-18T20:25:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.237+0000 7f72321c2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c0b1400 con 0x7f722c0b1e20 2024-03-18T20:25:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.237+0000 7f72321c2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f722c1e5170 con 0x7f722c0cbce0 2024-03-18T20:25:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.237+0000 7f722bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f722c0cd870 0x7f722c15a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.237+0000 7f722bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f722c0cd870 0x7f722c15a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33748/0 (socket says 172.21.15.1:33748) 2024-03-18T20:25:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.237+0000 7f722bfff700 1 -- 172.21.15.1:0/4258024280 learned_addr learned my addr 172.21.15.1:0/4258024280 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.237+0000 7f722b7fe700 1 --2- 172.21.15.1:0/4258024280 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f722c0b1e20 0x7f722c1e4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.237+0000 7f722affd700 1 --2- 172.21.15.1:0/4258024280 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c0cbce0 0x7f722c1579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.238+0000 7f722bfff700 1 -- 172.21.15.1:0/4258024280 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c0cbce0 msgr2=0x7f722c1579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.238+0000 7f722bfff700 1 --2- 172.21.15.1:0/4258024280 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c0cbce0 0x7f722c1579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.238+0000 7f722bfff700 1 -- 172.21.15.1:0/4258024280 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f722c0b1e20 msgr2=0x7f722c1e4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.238+0000 7f722bfff700 1 --2- 172.21.15.1:0/4258024280 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f722c0b1e20 0x7f722c1e4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.238+0000 7f722bfff700 1 -- 172.21.15.1:0/4258024280 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72200094a0 con 0x7f722c0cd870 2024-03-18T20:25:04.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.238+0000 7f722bfff700 1 --2- 172.21.15.1:0/4258024280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f722c0cd870 0x7f722c15a050 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f722c0d0970 tx=0x7f7224004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:04.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.238+0000 7f7213fff700 1 -- 172.21.15.1:0/4258024280 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72240129f0 con 0x7f722c0cd870 2024-03-18T20:25:04.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.238+0000 7f72321c2700 1 -- 172.21.15.1:0/4258024280 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7224009a70 con 0x7f722c0cd870 2024-03-18T20:25:04.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.238+0000 7f7213fff700 1 -- 172.21.15.1:0/4258024280 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7224020070 con 0x7f722c0cd870 2024-03-18T20:25:04.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.238+0000 7f7213fff700 1 -- 172.21.15.1:0/4258024280 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7224012db0 con 0x7f722c0cd870 2024-03-18T20:25:04.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.238+0000 7f72321c2700 1 -- 172.21.15.1:0/4258024280 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f722c0b0e40 con 0x7f722c0cd870 2024-03-18T20:25:04.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.239+0000 7f72321c2700 1 -- 172.21.15.1:0/4258024280 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f722c1e4fd0 con 0x7f722c0cd870 2024-03-18T20:25:04.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.240+0000 7f7213fff700 1 -- 172.21.15.1:0/4258024280 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 131) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f7224004040 con 0x7f722c0cd870 2024-03-18T20:25:04.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.241+0000 7f7213fff700 1 --2- 172.21.15.1:0/4258024280 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f721806aed0 0x7f721806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:04.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.241+0000 7f722b7fe700 1 -- 172.21.15.1:0/4258024280 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f721806aed0 msgr2=0x7f721806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:04.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.241+0000 7f7213fff700 1 -- 172.21.15.1:0/4258024280 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f7224022070 con 0x7f722c0cd870 2024-03-18T20:25:04.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.241+0000 7f722b7fe700 1 --2- 172.21.15.1:0/4258024280 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f721806aed0 0x7f721806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:04.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.249+0000 7f7213fff700 1 -- 172.21.15.1:0/4258024280 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f72240566b0 con 0x7f722c0cd870 2024-03-18T20:25:04.268 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:04.263+0000 7f25953da240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:25:04.378 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:04.375+0000 7f7b493a2240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:25:04.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.442+0000 7f722b7fe700 1 -- 172.21.15.1:0/4258024280 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f721806aed0 msgr2=0x7f721806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:04.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.442+0000 7f722b7fe700 1 --2- 172.21.15.1:0/4258024280 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f721806aed0 0x7f721806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:04.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.451+0000 7f72321c2700 1 -- 172.21.15.1:0/4258024280 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f722c1e4fd0 con 0x7f722c0cd870 2024-03-18T20:25:04.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.452+0000 7f7213fff700 1 -- 172.21.15.1:0/4258024280 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v217) v1 ==== 104+0+11 (secure 0 0 0) 0x7f722c1e4fd0 con 0x7f722c0cd870 2024-03-18T20:25:04.457 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:25:04.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.454+0000 7f72321c2700 1 -- 172.21.15.1:0/4258024280 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f721806aed0 msgr2=0x7f721806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:04.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.454+0000 7f72321c2700 1 --2- 172.21.15.1:0/4258024280 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f721806aed0 0x7f721806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.454+0000 7f72321c2700 1 -- 172.21.15.1:0/4258024280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f722c0cd870 msgr2=0x7f722c15a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:04.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.454+0000 7f72321c2700 1 --2- 172.21.15.1:0/4258024280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f722c0cd870 0x7f722c15a050 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f722c0d0970 tx=0x7f7224004ad0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.454+0000 7f72321c2700 1 -- 172.21.15.1:0/4258024280 shutdown_connections 2024-03-18T20:25:04.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.454+0000 7f72321c2700 1 --2- 172.21.15.1:0/4258024280 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f721806aed0 0x7f721806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.454+0000 7f72321c2700 1 --2- 172.21.15.1:0/4258024280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f722c0cd870 0x7f722c15a050 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.454+0000 7f72321c2700 1 --2- 172.21.15.1:0/4258024280 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f722c0b1e20 0x7f722c1e4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.454+0000 7f72321c2700 1 --2- 172.21.15.1:0/4258024280 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f722c0cbce0 0x7f722c1579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.454+0000 7f72321c2700 1 -- 172.21.15.1:0/4258024280 >> 172.21.15.1:0/4258024280 conn(0x7f722c0d0ff0 msgr2=0x7f722c0cc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:04.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.454+0000 7f72321c2700 1 -- 172.21.15.1:0/4258024280 shutdown_connections 2024-03-18T20:25:04.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.454+0000 7f72321c2700 1 -- 172.21.15.1:0/4258024280 wait complete. 2024-03-18T20:25:04.472 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:25:04.491 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:04.487+0000 7f25953da240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:25:04.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.597+0000 7f7fd83a0700 1 Processor -- start 2024-03-18T20:25:04.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.597+0000 7f7fd83a0700 1 -- start start 2024-03-18T20:25:04.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.598+0000 7f7fd83a0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd0071cc0 0x7f7fd00ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:04.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.598+0000 7f7fd83a0700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd00cac80 con 0x7f7fd006d560 2024-03-18T20:25:04.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.598+0000 7f7fd83a0700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd00caf60 con 0x7f7fd015a240 2024-03-18T20:25:04.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.598+0000 7f7fd83a0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd00adff0 con 0x7f7fd0071cc0 2024-03-18T20:25:04.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.598+0000 7f7fd693d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7fd006d560 0x7f7fd00ae410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52996/0 (socket says 172.21.15.1:52996) 2024-03-18T20:25:04.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.598+0000 7f7fd693d700 1 -- 172.21.15.1:0/1115336736 learned_addr learned my addr 172.21.15.1:0/1115336736 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:04.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.598+0000 7f7fd593b700 1 --2- 172.21.15.1:0/1115336736 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd0071cc0 0x7f7fd00ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:04.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.599+0000 7f7fd513a700 1 -- 172.21.15.1:0/1115336736 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356358959 0 0) 0x7f7fd00cac80 con 0x7f7fd006d560 2024-03-18T20:25:04.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.599+0000 7f7fd513a700 1 -- 172.21.15.1:0/1115336736 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fbc0035b0 con 0x7f7fd006d560 2024-03-18T20:25:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.599+0000 7f7fd513a700 1 -- 172.21.15.1:0/1115336736 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481826782 0 0) 0x7f7fd00caf60 con 0x7f7fd015a240 2024-03-18T20:25:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.599+0000 7f7fd513a700 1 -- 172.21.15.1:0/1115336736 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fd00cac80 con 0x7f7fd015a240 2024-03-18T20:25:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.599+0000 7f7fd593b700 1 -- 172.21.15.1:0/1115336736 >> v1:172.21.15.165:6789/0 conn(0x7f7fd015a240 legacy=0x7f7fd00715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.599+0000 7f7fd593b700 1 -- 172.21.15.1:0/1115336736 >> v1:172.21.15.1:6789/0 conn(0x7f7fd006d560 legacy=0x7f7fd00ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.599+0000 7f7fd593b700 1 -- 172.21.15.1:0/1115336736 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd00cb240 con 0x7f7fd0071cc0 2024-03-18T20:25:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.599+0000 7f7fd593b700 1 --2- 172.21.15.1:0/1115336736 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd0071cc0 0x7f7fd00ca580 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc009870 tx=0x7f7fcc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be6fa53d2dd70ccf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.599+0000 7f7fd513a700 1 -- 172.21.15.1:0/1115336736 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc011830 con 0x7f7fd0071cc0 2024-03-18T20:25:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.599+0000 7f7fd513a700 1 -- 172.21.15.1:0/1115336736 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fcc011990 con 0x7f7fd0071cc0 2024-03-18T20:25:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.599+0000 7f7fd513a700 1 -- 172.21.15.1:0/1115336736 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc011d20 con 0x7f7fd0071cc0 2024-03-18T20:25:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.600+0000 7f7fd83a0700 1 -- 172.21.15.1:0/1115336736 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd0071cc0 msgr2=0x7f7fd00ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.600+0000 7f7fd83a0700 1 --2- 172.21.15.1:0/1115336736 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd0071cc0 0x7f7fd00ca580 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc009870 tx=0x7f7fcc009b80 comp rx=0 tx=0).stop 2024-03-18T20:25:04.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.600+0000 7f7fd83a0700 1 -- 172.21.15.1:0/1115336736 shutdown_connections 2024-03-18T20:25:04.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.600+0000 7f7fd83a0700 1 --2- 172.21.15.1:0/1115336736 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd0071cc0 0x7f7fd00ca580 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.600+0000 7f7fd83a0700 1 -- 172.21.15.1:0/1115336736 >> 172.21.15.1:0/1115336736 conn(0x7f7fd00cff30 msgr2=0x7f7fd00d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:04.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.600+0000 7f7fd83a0700 1 -- 172.21.15.1:0/1115336736 shutdown_connections 2024-03-18T20:25:04.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.600+0000 7f7fd83a0700 1 -- 172.21.15.1:0/1115336736 wait complete. 2024-03-18T20:25:04.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.601+0000 7f7fd83a0700 1 Processor -- start 2024-03-18T20:25:04.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.601+0000 7f7fd83a0700 1 -- start start 2024-03-18T20:25:04.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.602+0000 7f7fd83a0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fd006d560 0x7f7fd00c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:04.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.602+0000 7f7fd83a0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd0071cc0 0x7f7fd00b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:04.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.602+0000 7f7fd83a0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd015a240 0x7f7fd00b3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:04.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.602+0000 7f7fd83a0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd00c7d80 con 0x7f7fd0071cc0 2024-03-18T20:25:04.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.602+0000 7f7fd83a0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd00c8060 con 0x7f7fd006d560 2024-03-18T20:25:04.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.602+0000 7f7fd83a0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd00c8340 con 0x7f7fd015a240 2024-03-18T20:25:04.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.602+0000 7f7fd593b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd0071cc0 0x7f7fd00b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:04.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.602+0000 7f7fd593b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd0071cc0 0x7f7fd00b2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60550/0 (socket says 172.21.15.1:60550) 2024-03-18T20:25:04.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.602+0000 7f7fd593b700 1 -- 172.21.15.1:0/3209244121 learned_addr learned my addr 172.21.15.1:0/3209244121 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:04.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.602+0000 7f7fd693d700 1 --2- 172.21.15.1:0/3209244121 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd015a240 0x7f7fd00b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:04.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.602+0000 7f7fd613c700 1 --2- 172.21.15.1:0/3209244121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fd006d560 0x7f7fd00c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:04.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.602+0000 7f7fd593b700 1 -- 172.21.15.1:0/3209244121 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd015a240 msgr2=0x7f7fd00b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:04.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.602+0000 7f7fd593b700 1 --2- 172.21.15.1:0/3209244121 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd015a240 0x7f7fd00b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.602+0000 7f7fd593b700 1 -- 172.21.15.1:0/3209244121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fd006d560 msgr2=0x7f7fd00c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:04.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.602+0000 7f7fd593b700 1 --2- 172.21.15.1:0/3209244121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fd006d560 0x7f7fd00c61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.602+0000 7f7fd593b700 1 -- 172.21.15.1:0/3209244121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fcc0094a0 con 0x7f7fd0071cc0 2024-03-18T20:25:04.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.602+0000 7f7fd593b700 1 --2- 172.21.15.1:0/3209244121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd0071cc0 0x7f7fd00b2b20 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc0090b0 tx=0x7f7fcc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:04.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.603+0000 7f7fc6ffd700 1 -- 172.21.15.1:0/3209244121 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc01a6e0 con 0x7f7fd0071cc0 2024-03-18T20:25:04.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.603+0000 7f7fc6ffd700 1 -- 172.21.15.1:0/3209244121 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fcc01a840 con 0x7f7fd0071cc0 2024-03-18T20:25:04.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.603+0000 7f7fc6ffd700 1 -- 172.21.15.1:0/3209244121 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc01abb0 con 0x7f7fd0071cc0 2024-03-18T20:25:04.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.603+0000 7f7fd83a0700 1 -- 172.21.15.1:0/3209244121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd00c8620 con 0x7f7fd0071cc0 2024-03-18T20:25:04.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.603+0000 7f7fd83a0700 1 -- 172.21.15.1:0/3209244121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd00c7a10 con 0x7f7fd0071cc0 2024-03-18T20:25:04.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.604+0000 7f7fd83a0700 1 -- 172.21.15.1:0/3209244121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd00c43f0 con 0x7f7fd0071cc0 2024-03-18T20:25:04.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.605+0000 7f7fc6ffd700 1 -- 172.21.15.1:0/3209244121 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 131) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f7fcc022070 con 0x7f7fd0071cc0 2024-03-18T20:25:04.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.605+0000 7f7fc6ffd700 1 --2- 172.21.15.1:0/3209244121 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fb406aed0 0x7f7fb406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:04.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.605+0000 7f7fd613c700 1 -- 172.21.15.1:0/3209244121 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fb406aed0 msgr2=0x7f7fb406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:04.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.605+0000 7f7fc6ffd700 1 -- 172.21.15.1:0/3209244121 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f7fcc08aa00 con 0x7f7fd0071cc0 2024-03-18T20:25:04.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.605+0000 7f7fd613c700 1 --2- 172.21.15.1:0/3209244121 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fb406aed0 0x7f7fb406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:04.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.610+0000 7f7fc6ffd700 1 -- 172.21.15.1:0/3209244121 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7fcc055770 con 0x7f7fd0071cc0 2024-03-18T20:25:04.752 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:04.749+0000 7f7b493a2240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:25:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.806+0000 7f7fd613c700 1 -- 172.21.15.1:0/3209244121 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fb406aed0 msgr2=0x7f7fb406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:04.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.806+0000 7f7fd613c700 1 --2- 172.21.15.1:0/3209244121 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fb406aed0 0x7f7fb406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.814+0000 7f7fd83a0700 1 -- 172.21.15.1:0/3209244121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7fd00676a0 con 0x7f7fd0071cc0 2024-03-18T20:25:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.815+0000 7f7fc6ffd700 1 -- 172.21.15.1:0/3209244121 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v217) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7fcc0150a0 con 0x7f7fd0071cc0 2024-03-18T20:25:04.820 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:25:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.817+0000 7f7fd83a0700 1 -- 172.21.15.1:0/3209244121 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fb406aed0 msgr2=0x7f7fb406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.817+0000 7f7fd83a0700 1 --2- 172.21.15.1:0/3209244121 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fb406aed0 0x7f7fb406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.817+0000 7f7fd83a0700 1 -- 172.21.15.1:0/3209244121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd0071cc0 msgr2=0x7f7fd00b2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.817+0000 7f7fd83a0700 1 --2- 172.21.15.1:0/3209244121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd0071cc0 0x7f7fd00b2b20 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc0090b0 tx=0x7f7fcc004750 comp rx=0 tx=0).stop 2024-03-18T20:25:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.817+0000 7f7fd83a0700 1 -- 172.21.15.1:0/3209244121 shutdown_connections 2024-03-18T20:25:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.817+0000 7f7fd83a0700 1 --2- 172.21.15.1:0/3209244121 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fb406aed0 0x7f7fb406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.817+0000 7f7fd83a0700 1 --2- 172.21.15.1:0/3209244121 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd015a240 0x7f7fd00b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.817+0000 7f7fd83a0700 1 --2- 172.21.15.1:0/3209244121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fd006d560 0x7f7fd00c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.817+0000 7f7fd83a0700 1 --2- 172.21.15.1:0/3209244121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd0071cc0 0x7f7fd00b2b20 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.817+0000 7f7fd83a0700 1 -- 172.21.15.1:0/3209244121 >> 172.21.15.1:0/3209244121 conn(0x7f7fd00cff30 msgr2=0x7f7fd0159330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.817+0000 7f7fd83a0700 1 -- 172.21.15.1:0/3209244121 shutdown_connections 2024-03-18T20:25:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.817+0000 7f7fd83a0700 1 -- 172.21.15.1:0/3209244121 wait complete. 2024-03-18T20:25:04.836 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T20:25:04.890 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:04.885+0000 7f25953da240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:25:05.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.998+0000 7f1d53a6e700 1 Processor -- start 2024-03-18T20:25:05.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.998+0000 7f1d53a6e700 1 -- start start 2024-03-18T20:25:05.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.999+0000 7f1d53a6e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1d4c160a70 0x7f1d4c0d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:05.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.999+0000 7f1d53a6e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4c160e50 0x7f1d4c0cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:05.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.999+0000 7f1d53a6e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1d4c1612d0 0x7f1d4c161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:05.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.999+0000 7f1d53a6e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d4c162170 con 0x7f1d4c1612d0 2024-03-18T20:25:05.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.999+0000 7f1d53a6e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d4c161c60 con 0x7f1d4c160a70 2024-03-18T20:25:05.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.999+0000 7f1d53a6e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4c0cef30 con 0x7f1d4c160e50 2024-03-18T20:25:05.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.999+0000 7f1d5200b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1d4c1612d0 0x7f1d4c161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:05.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.999+0000 7f1d5200b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1d4c1612d0 0x7f1d4c161720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60562/0 (socket says 172.21.15.1:60562) 2024-03-18T20:25:05.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.999+0000 7f1d5200b700 1 -- 172.21.15.1:0/3368327129 learned_addr learned my addr 172.21.15.1:0/3368327129 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.999+0000 7f1d5180a700 1 --2- 172.21.15.1:0/3368327129 >> v2:172.21.15.165:3300/0 conn(0x7f1d4c160a70 0x7f1d4c0d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:04.999+0000 7f1d51009700 1 --2- 172.21.15.1:0/3368327129 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4c160e50 0x7f1d4c0cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.000+0000 7f1d5200b700 1 -- 172.21.15.1:0/3368327129 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4c160e50 msgr2=0x7f1d4c0cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.000+0000 7f1d5200b700 1 --2- 172.21.15.1:0/3368327129 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4c160e50 0x7f1d4c0cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.000+0000 7f1d5200b700 1 -- 172.21.15.1:0/3368327129 >> v2:172.21.15.165:3300/0 conn(0x7f1d4c160a70 msgr2=0x7f1d4c0d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.000+0000 7f1d5200b700 1 --2- 172.21.15.1:0/3368327129 >> v2:172.21.15.165:3300/0 conn(0x7f1d4c160a70 0x7f1d4c0d3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.000+0000 7f1d5200b700 1 -- 172.21.15.1:0/3368327129 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d4c0cf210 con 0x7f1d4c1612d0 2024-03-18T20:25:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.000+0000 7f1d5200b700 1 --2- 172.21.15.1:0/3368327129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d4c1612d0 0x7f1d4c161720 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c08d8f0 tx=0x7f1d4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826ab8fecf860662 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.000+0000 7f1d50808700 1 -- 172.21.15.1:0/3368327129 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4400e070 con 0x7f1d4c1612d0 2024-03-18T20:25:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.000+0000 7f1d50808700 1 -- 172.21.15.1:0/3368327129 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d4400b910 con 0x7f1d4c1612d0 2024-03-18T20:25:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.000+0000 7f1d50808700 1 -- 172.21.15.1:0/3368327129 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4400bcc0 con 0x7f1d4c1612d0 2024-03-18T20:25:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.001+0000 7f1d53a6e700 1 -- 172.21.15.1:0/3368327129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d4c1612d0 msgr2=0x7f1d4c161720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.001+0000 7f1d53a6e700 1 --2- 172.21.15.1:0/3368327129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d4c1612d0 0x7f1d4c161720 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c08d8f0 tx=0x7f1d4400b330 comp rx=0 tx=0).stop 2024-03-18T20:25:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.001+0000 7f1d53a6e700 1 -- 172.21.15.1:0/3368327129 shutdown_connections 2024-03-18T20:25:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.001+0000 7f1d53a6e700 1 --2- 172.21.15.1:0/3368327129 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d4c1612d0 0x7f1d4c161720 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.001+0000 7f1d53a6e700 1 --2- 172.21.15.1:0/3368327129 >> v2:172.21.15.165:3300/0 conn(0x7f1d4c160a70 0x7f1d4c0d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.001+0000 7f1d53a6e700 1 --2- 172.21.15.1:0/3368327129 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4c160e50 0x7f1d4c0cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.001+0000 7f1d53a6e700 1 -- 172.21.15.1:0/3368327129 >> 172.21.15.1:0/3368327129 conn(0x7f1d4c09c560 msgr2=0x7f1d4c0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.001+0000 7f1d53a6e700 1 -- 172.21.15.1:0/3368327129 shutdown_connections 2024-03-18T20:25:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.001+0000 7f1d53a6e700 1 -- 172.21.15.1:0/3368327129 wait complete. 2024-03-18T20:25:05.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.002+0000 7f1d53a6e700 1 Processor -- start 2024-03-18T20:25:05.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.002+0000 7f1d53a6e700 1 -- start start 2024-03-18T20:25:05.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.003+0000 7f1d53a6e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d4c160a70 0x7f1d4c06bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:05.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.003+0000 7f1d53a6e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4c160e50 0x7f1d4c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:05.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.003+0000 7f1d53a6e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d4c1612d0 0x7f1d4c0a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:05.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.003+0000 7f1d53a6e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4c0ca320 con 0x7f1d4c160a70 2024-03-18T20:25:05.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.003+0000 7f1d53a6e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4c0ca570 con 0x7f1d4c1612d0 2024-03-18T20:25:05.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.003+0000 7f1d53a6e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4c0ca850 con 0x7f1d4c160e50 2024-03-18T20:25:05.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.003+0000 7f1d5180a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d4c160a70 0x7f1d4c06bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:05.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.003+0000 7f1d5180a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d4c160a70 0x7f1d4c06bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60564/0 (socket says 172.21.15.1:60564) 2024-03-18T20:25:05.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.003+0000 7f1d5180a700 1 -- 172.21.15.1:0/595022616 learned_addr learned my addr 172.21.15.1:0/595022616 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:05.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.003+0000 7f1d51009700 1 --2- 172.21.15.1:0/595022616 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4c160e50 0x7f1d4c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:05.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.003+0000 7f1d5200b700 1 --2- 172.21.15.1:0/595022616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d4c1612d0 0x7f1d4c0a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:05.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.003+0000 7f1d5180a700 1 -- 172.21.15.1:0/595022616 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4c160e50 msgr2=0x7f1d4c0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:05.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.003+0000 7f1d5180a700 1 --2- 172.21.15.1:0/595022616 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4c160e50 0x7f1d4c0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.003+0000 7f1d5180a700 1 -- 172.21.15.1:0/595022616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d4c1612d0 msgr2=0x7f1d4c0a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:05.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.003+0000 7f1d5180a700 1 --2- 172.21.15.1:0/595022616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d4c1612d0 0x7f1d4c0a0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.003+0000 7f1d5180a700 1 -- 172.21.15.1:0/595022616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d44009b30 con 0x7f1d4c160a70 2024-03-18T20:25:05.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.003+0000 7f1d5180a700 1 --2- 172.21.15.1:0/595022616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d4c160a70 0x7f1d4c06bfe0 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f1d4800e870 tx=0x7f1d4800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:05.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.004+0000 7f1d427fc700 1 -- 172.21.15.1:0/595022616 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d48013680 con 0x7f1d4c160a70 2024-03-18T20:25:05.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.004+0000 7f1d427fc700 1 -- 172.21.15.1:0/595022616 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d480137e0 con 0x7f1d4c160a70 2024-03-18T20:25:05.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.004+0000 7f1d53a6e700 1 -- 172.21.15.1:0/595022616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d4c0cab30 con 0x7f1d4c160a70 2024-03-18T20:25:05.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.004+0000 7f1d427fc700 1 -- 172.21.15.1:0/595022616 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d48013b90 con 0x7f1d4c160a70 2024-03-18T20:25:05.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.004+0000 7f1d53a6e700 1 -- 172.21.15.1:0/595022616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d4c0cb0f0 con 0x7f1d4c160a70 2024-03-18T20:25:05.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.005+0000 7f1d53a6e700 1 -- 172.21.15.1:0/595022616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d4c067390 con 0x7f1d4c160a70 2024-03-18T20:25:05.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.006+0000 7f1d427fc700 1 -- 172.21.15.1:0/595022616 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 131) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f1d48013cf0 con 0x7f1d4c160a70 2024-03-18T20:25:05.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.006+0000 7f1d427fc700 1 --2- 172.21.15.1:0/595022616 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1d3006aed0 0x7f1d3006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:05.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.006+0000 7f1d51009700 1 -- 172.21.15.1:0/595022616 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1d3006aed0 msgr2=0x7f1d3006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:05.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.006+0000 7f1d427fc700 1 -- 172.21.15.1:0/595022616 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f1d48089fd0 con 0x7f1d4c160a70 2024-03-18T20:25:05.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.006+0000 7f1d51009700 1 --2- 172.21.15.1:0/595022616 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1d3006aed0 0x7f1d3006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:05.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.011+0000 7f1d427fc700 1 -- 172.21.15.1:0/595022616 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1d48054c00 con 0x7f1d4c160a70 2024-03-18T20:25:05.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.207+0000 7f1d51009700 1 -- 172.21.15.1:0/595022616 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1d3006aed0 msgr2=0x7f1d3006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:05.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.207+0000 7f1d51009700 1 --2- 172.21.15.1:0/595022616 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1d3006aed0 0x7f1d3006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:05.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.234+0000 7f1d53a6e700 1 -- 172.21.15.1:0/595022616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1d4c0676a0 con 0x7f1d4c160a70 2024-03-18T20:25:05.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.235+0000 7f1d427fc700 1 -- 172.21.15.1:0/595022616 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v218) v1 ==== 80+0+454 (secure 0 0 0) 0x7f1d480211e0 con 0x7f1d4c160a70 2024-03-18T20:25:05.240 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2462726896 2024-03-19T20:23:54.844816+0000 2024-03-18T20:25:05.240 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6833/148356 2024-03-19T20:23:54.844816+0000 2024-03-18T20:25:05.240 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6832/148356 2024-03-19T20:23:54.844816+0000 2024-03-18T20:25:05.240 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/4177404804 2024-03-19T20:23:54.844816+0000 2024-03-18T20:25:05.240 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1122038130 2024-03-19T20:23:54.844816+0000 2024-03-18T20:25:05.240 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2416229963 2024-03-19T20:23:54.844816+0000 2024-03-18T20:25:05.240 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2662947140 2024-03-19T20:23:54.844816+0000 2024-03-18T20:25:05.240 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2110350357 2024-03-19T20:23:54.844816+0000 2024-03-18T20:25:05.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.237+0000 7f1d53a6e700 1 -- 172.21.15.1:0/595022616 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1d3006aed0 msgr2=0x7f1d3006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:05.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.237+0000 7f1d53a6e700 1 --2- 172.21.15.1:0/595022616 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1d3006aed0 0x7f1d3006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.237+0000 7f1d53a6e700 1 -- 172.21.15.1:0/595022616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d4c160a70 msgr2=0x7f1d4c06bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:05.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.237+0000 7f1d53a6e700 1 --2- 172.21.15.1:0/595022616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d4c160a70 0x7f1d4c06bfe0 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f1d4800e870 tx=0x7f1d4800eb80 comp rx=0 tx=0).stop 2024-03-18T20:25:05.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.237+0000 7f1d53a6e700 1 -- 172.21.15.1:0/595022616 shutdown_connections 2024-03-18T20:25:05.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.237+0000 7f1d53a6e700 1 --2- 172.21.15.1:0/595022616 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1d3006aed0 0x7f1d3006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.237+0000 7f1d53a6e700 1 --2- 172.21.15.1:0/595022616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d4c1612d0 0x7f1d4c0a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.237+0000 7f1d53a6e700 1 --2- 172.21.15.1:0/595022616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d4c160a70 0x7f1d4c06bfe0 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.237+0000 7f1d53a6e700 1 --2- 172.21.15.1:0/595022616 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4c160e50 0x7f1d4c0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.237+0000 7f1d53a6e700 1 -- 172.21.15.1:0/595022616 >> 172.21.15.1:0/595022616 conn(0x7f1d4c09c560 msgr2=0x7f1d4c15f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:05.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.237+0000 7f1d53a6e700 1 -- 172.21.15.1:0/595022616 shutdown_connections 2024-03-18T20:25:05.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.237+0000 7f1d53a6e700 1 -- 172.21.15.1:0/595022616 wait complete. 2024-03-18T20:25:05.244 INFO:teuthology.orchestra.run.smithi001.stderr:listed 8 entries 2024-03-18T20:25:05.255 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-18T20:25:05.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:05.255 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T20:25:05.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[152682]: fuse finished with error 0 and tester_r 0 2024-03-18T20:25:05.333 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:25:05.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:05.334 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T20:25:05.344 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:05.341+0000 7f7b493a2240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:25:05.420 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:05.416+0000 7f7b493a2240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:25:05.493 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:25:05.493 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:25:05.510 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:25:05.511 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:25:05.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:05.511 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-18T20:25:05.523 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:05.518+0000 7f25953da240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:25:05.550 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:05.546+0000 7f7b493a2240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:25:05.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:25:05.566 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-18T20:25:05.567 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:25:05.567 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:25:05.585 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:25:05.586 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:25:05.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:05.586 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-18T20:25:05.602 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:05.596+0000 7f25953da240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:25:05.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:25:05.619 INFO:teuthology.orchestra.run.smithi001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-18T20:25:05.620 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:25:05.620 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:25:05.638 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:25:05.638 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:25:05.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:05.639 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-18T20:25:05.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:25:05.696 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-18T20:25:05.697 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:25:05.736 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:05.732+0000 7f7b493a2240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:25:05.745 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:05.740+0000 7f25953da240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:25:05.809 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:05.805+0000 7f7b493a2240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:25:05.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.820+0000 7f93d8eb5700 1 Processor -- start 2024-03-18T20:25:05.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.820+0000 7f93d8eb5700 1 -- start start 2024-03-18T20:25:05.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.821+0000 7f93d8eb5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f93d415a320 0x7f93d40ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:05.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.821+0000 7f93d8eb5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f93d40cea30 0x7f93d4071600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:05.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.821+0000 7f93d8eb5700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d40acd80 con 0x7f93d4071b40 2024-03-18T20:25:05.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.821+0000 7f93d8eb5700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d4071ff0 con 0x7f93d415a320 2024-03-18T20:25:05.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.821+0000 7f93d8eb5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d40cf220 con 0x7f93d40cea30 2024-03-18T20:25:05.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.821+0000 7f93d259c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f93d415a320 0x7f93d40ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:05.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.821+0000 7f93d2d9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f93d4071b40 0x7f93d40cad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53000/0 (socket says 172.21.15.1:53000) 2024-03-18T20:25:05.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.821+0000 7f93d2d9d700 1 -- 172.21.15.1:0/2644853105 learned_addr learned my addr 172.21.15.1:0/2644853105 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:05.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.821+0000 7f93d1d9b700 1 --2- 172.21.15.1:0/2644853105 >> v2:172.21.15.165:3300/0 conn(0x7f93d40cea30 0x7f93d4071600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:05.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.822+0000 7f93d259c700 1 -- 172.21.15.1:0/2644853105 >> v2:172.21.15.165:3300/0 conn(0x7f93d40cea30 msgr2=0x7f93d4071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:05.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.822+0000 7f93d259c700 1 --2- 172.21.15.1:0/2644853105 >> v2:172.21.15.165:3300/0 conn(0x7f93d40cea30 0x7f93d4071600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.822+0000 7f93d259c700 1 -- 172.21.15.1:0/2644853105 >> v1:172.21.15.1:6789/0 conn(0x7f93d4071b40 legacy=0x7f93d40cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:05.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.822+0000 7f93d259c700 1 -- 172.21.15.1:0/2644853105 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d40cef40 con 0x7f93d415a320 2024-03-18T20:25:05.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.822+0000 7f93d159a700 1 -- 172.21.15.1:0/2644853105 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53187768 0 0) 0x7f93d40acd80 con 0x7f93d4071b40 2024-03-18T20:25:05.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.822+0000 7f93d259c700 1 --2- 172.21.15.1:0/2644853105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d415a320 0x7f93d40ce4f0 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f93d40672b0 tx=0x7f93c800bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97301d6952f96d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:05.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.822+0000 7f93d159a700 1 -- 172.21.15.1:0/2644853105 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c800dc70 con 0x7f93d415a320 2024-03-18T20:25:05.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.822+0000 7f93d159a700 1 -- 172.21.15.1:0/2644853105 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c800ddd0 con 0x7f93d415a320 2024-03-18T20:25:05.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.822+0000 7f93d159a700 1 -- 172.21.15.1:0/2644853105 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c8010690 con 0x7f93d415a320 2024-03-18T20:25:05.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.823+0000 7f93d8eb5700 1 -- 172.21.15.1:0/2644853105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d415a320 msgr2=0x7f93d40ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:05.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.823+0000 7f93d8eb5700 1 --2- 172.21.15.1:0/2644853105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d415a320 0x7f93d40ce4f0 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f93d40672b0 tx=0x7f93c800bdc0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.823+0000 7f93d8eb5700 1 -- 172.21.15.1:0/2644853105 shutdown_connections 2024-03-18T20:25:05.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.823+0000 7f93d8eb5700 1 --2- 172.21.15.1:0/2644853105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d415a320 0x7f93d40ce4f0 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.823+0000 7f93d8eb5700 1 --2- 172.21.15.1:0/2644853105 >> v2:172.21.15.165:3300/0 conn(0x7f93d40cea30 0x7f93d4071600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.823+0000 7f93d8eb5700 1 -- 172.21.15.1:0/2644853105 >> 172.21.15.1:0/2644853105 conn(0x7f93d40cc990 msgr2=0x7f93d40ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:05.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.823+0000 7f93d8eb5700 1 -- 172.21.15.1:0/2644853105 shutdown_connections 2024-03-18T20:25:05.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.823+0000 7f93d8eb5700 1 -- 172.21.15.1:0/2644853105 wait complete. 2024-03-18T20:25:05.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.824+0000 7f93d8eb5700 1 Processor -- start 2024-03-18T20:25:05.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.824+0000 7f93d8eb5700 1 -- start start 2024-03-18T20:25:05.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.825+0000 7f93d8eb5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d4071b40 0x7f93d415d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:05.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.825+0000 7f93d8eb5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93d40cea30 0x7f93d40ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:05.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.825+0000 7f93d8eb5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93d40c79f0 0x7f93d40c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:05.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.825+0000 7f93d8eb5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d40c8b70 con 0x7f93d4071b40 2024-03-18T20:25:05.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.825+0000 7f93d8eb5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d415d5a0 con 0x7f93d40cea30 2024-03-18T20:25:05.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.825+0000 7f93d8eb5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d40c7ef0 con 0x7f93d40c79f0 2024-03-18T20:25:05.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.825+0000 7f93d259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d4071b40 0x7f93d415d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:05.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.825+0000 7f93d259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d4071b40 0x7f93d415d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60580/0 (socket says 172.21.15.1:60580) 2024-03-18T20:25:05.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.825+0000 7f93d259c700 1 -- 172.21.15.1:0/1402535763 learned_addr learned my addr 172.21.15.1:0/1402535763 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:05.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.825+0000 7f93d1d9b700 1 --2- 172.21.15.1:0/1402535763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93d40cea30 0x7f93d40ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:05.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.825+0000 7f93d2d9d700 1 --2- 172.21.15.1:0/1402535763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93d40c79f0 0x7f93d40c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:05.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.825+0000 7f93d259c700 1 -- 172.21.15.1:0/1402535763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93d40c79f0 msgr2=0x7f93d40c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:05.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.825+0000 7f93d259c700 1 --2- 172.21.15.1:0/1402535763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93d40c79f0 0x7f93d40c8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.825+0000 7f93d259c700 1 -- 172.21.15.1:0/1402535763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93d40cea30 msgr2=0x7f93d40ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:05.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.825+0000 7f93d259c700 1 --2- 172.21.15.1:0/1402535763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93d40cea30 0x7f93d40ae040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:05.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.825+0000 7f93d259c700 1 -- 172.21.15.1:0/1402535763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c800b740 con 0x7f93d4071b40 2024-03-18T20:25:05.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.826+0000 7f93d259c700 1 --2- 172.21.15.1:0/1402535763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d4071b40 0x7f93d415d060 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f93c800b2c0 tx=0x7f93c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:05.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.826+0000 7f93c2ffd700 1 -- 172.21.15.1:0/1402535763 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c801d070 con 0x7f93d4071b40 2024-03-18T20:25:05.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.826+0000 7f93c2ffd700 1 -- 172.21.15.1:0/1402535763 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c801e040 con 0x7f93d4071b40 2024-03-18T20:25:05.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.826+0000 7f93d8eb5700 1 -- 172.21.15.1:0/1402535763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d40c81d0 con 0x7f93d4071b40 2024-03-18T20:25:05.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.826+0000 7f93c2ffd700 1 -- 172.21.15.1:0/1402535763 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c8010c60 con 0x7f93d4071b40 2024-03-18T20:25:05.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.826+0000 7f93d8eb5700 1 -- 172.21.15.1:0/1402535763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d40c8790 con 0x7f93d4071b40 2024-03-18T20:25:05.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.827+0000 7f93d8eb5700 1 -- 172.21.15.1:0/1402535763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d40676a0 con 0x7f93d4071b40 2024-03-18T20:25:05.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.827+0000 7f93c2ffd700 1 -- 172.21.15.1:0/1402535763 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 131) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f93c80036a0 con 0x7f93d4071b40 2024-03-18T20:25:05.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.827+0000 7f93c2ffd700 1 --2- 172.21.15.1:0/1402535763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93b406aed0 0x7f93b406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:05.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.828+0000 7f93d1d9b700 1 -- 172.21.15.1:0/1402535763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93b406aed0 msgr2=0x7f93b406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:05.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.828+0000 7f93c2ffd700 1 -- 172.21.15.1:0/1402535763 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6648+0+0 (secure 0 0 0) 0x7f93c808dce0 con 0x7f93d4071b40 2024-03-18T20:25:05.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.828+0000 7f93d1d9b700 1 --2- 172.21.15.1:0/1402535763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93b406aed0 0x7f93b406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:05.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:05.832+0000 7f93c2ffd700 1 -- 172.21.15.1:0/1402535763 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f93c8058730 con 0x7f93d4071b40 2024-03-18T20:25:05.888 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:05.885+0000 7f7b493a2240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:25:05.938 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:05.933+0000 7f25953da240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:25:05.966 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:05.962+0000 7f7b493a2240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:25:06.014 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:06.009+0000 7f25953da240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:25:06.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.029+0000 7f93d1d9b700 1 -- 172.21.15.1:0/1402535763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93b406aed0 msgr2=0x7f93b406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:06.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.029+0000 7f93d1d9b700 1 --2- 172.21.15.1:0/1402535763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93b406aed0 0x7f93b406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:06.044 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:06.040+0000 7f7b493a2240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:25:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.082+0000 7f93d8eb5700 1 -- 172.21.15.1:0/1402535763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93d40cdc40 con 0x7f93d4071b40 2024-03-18T20:25:06.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.083+0000 7f93c2ffd700 1 -- 172.21.15.1:0/1402535763 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f93c805bd50 con 0x7f93d4071b40 2024-03-18T20:25:06.088 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:06.088 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":97,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2241819619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2241819619},{"type":"v1","addr":"172.21.15.165:6835","nonce":2241819619}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/320492236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":320492236},{"type":"v1","addr":"172.21.15.165:6834","nonce":320492236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":11031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/1035910531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1035910531},{"type":"v1","addr":"172.21.15.1:6836","nonce":1035910531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:23:39.747292+0000","modified":"2024-03-18T20:23:42.766952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11034":{"gid":11034,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2388480799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2388480799},{"type":"v1","addr":"172.21.15.1:6837","nonce":2388480799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-03-18T20:25:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.085+0000 7f93d8eb5700 1 -- 172.21.15.1:0/1402535763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93b406aed0 msgr2=0x7f93b406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.085+0000 7f93d8eb5700 1 --2- 172.21.15.1:0/1402535763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93b406aed0 0x7f93b406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.085+0000 7f93d8eb5700 1 -- 172.21.15.1:0/1402535763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d4071b40 msgr2=0x7f93d415d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.085+0000 7f93d8eb5700 1 --2- 172.21.15.1:0/1402535763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d4071b40 0x7f93d415d060 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f93c800b2c0 tx=0x7f93c8004750 comp rx=0 tx=0).stop 2024-03-18T20:25:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.085+0000 7f93d8eb5700 1 -- 172.21.15.1:0/1402535763 shutdown_connections 2024-03-18T20:25:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.085+0000 7f93d8eb5700 1 --2- 172.21.15.1:0/1402535763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93b406aed0 0x7f93b406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.085+0000 7f93d8eb5700 1 --2- 172.21.15.1:0/1402535763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93d40c79f0 0x7f93d40c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.085+0000 7f93d8eb5700 1 --2- 172.21.15.1:0/1402535763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93d4071b40 0x7f93d415d060 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.085+0000 7f93d8eb5700 1 --2- 172.21.15.1:0/1402535763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93d40cea30 0x7f93d40ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.085+0000 7f93d8eb5700 1 -- 172.21.15.1:0/1402535763 >> 172.21.15.1:0/1402535763 conn(0x7f93d40cc990 msgr2=0x7f93d40ccd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.085+0000 7f93d8eb5700 1 -- 172.21.15.1:0/1402535763 shutdown_connections 2024-03-18T20:25:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.085+0000 7f93d8eb5700 1 -- 172.21.15.1:0/1402535763 wait complete. 2024-03-18T20:25:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 97 2024-03-18T20:25:06.097 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:06.092+0000 7f25953da240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:25:06.103 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:25:06.185 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:06.179+0000 7f25953da240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:25:06.219 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:06.216+0000 7f7b493a2240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:25:06.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.227+0000 7ff9b7a59700 1 Processor -- start 2024-03-18T20:25:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.228+0000 7ff9b7a59700 1 -- start start 2024-03-18T20:25:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.229+0000 7ff9b7a59700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff9b00d1c60 0x7ff9b00b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.229+0000 7ff9b7a59700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b00cddc0 con 0x7ff9b00d17b0 2024-03-18T20:25:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.229+0000 7ff9b7a59700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b00d3dc0 con 0x7ff9b015d500 2024-03-18T20:25:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.229+0000 7ff9b7a59700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b00ce060 con 0x7ff9b00d1c60 2024-03-18T20:25:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.229+0000 7ff9b5ff6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff9b00d1c60 0x7ff9b00b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.229+0000 7ff9b4ff4700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff9b00d17b0 0x7ff9b00b83f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53008/0 (socket says 172.21.15.1:53008) 2024-03-18T20:25:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.229+0000 7ff9b5ff6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff9b00d1c60 0x7ff9b00b8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60592/0 (socket says 172.21.15.1:60592) 2024-03-18T20:25:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.229+0000 7ff9b4ff4700 1 -- 172.21.15.1:0/3067350802 learned_addr learned my addr 172.21.15.1:0/3067350802 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.229+0000 7ff9b5ff6700 1 -- 172.21.15.1:0/3067350802 >> v1:172.21.15.165:6789/0 conn(0x7ff9b015d500 legacy=0x7ff9b00d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.229+0000 7ff9b5ff6700 1 -- 172.21.15.1:0/3067350802 >> v1:172.21.15.1:6789/0 conn(0x7ff9b00d17b0 legacy=0x7ff9b00b83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.230+0000 7ff9b5ff6700 1 -- 172.21.15.1:0/3067350802 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b00ce340 con 0x7ff9b00d1c60 2024-03-18T20:25:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.230+0000 7ff9a7fff700 1 -- 172.21.15.1:0/3067350802 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410977257 0 0) 0x7ff9b00cddc0 con 0x7ff9b00d17b0 2024-03-18T20:25:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.230+0000 7ff9b5ff6700 1 --2- 172.21.15.1:0/3067350802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9b00d1c60 0x7ff9b00b8b00 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8009d90 tx=0x7ff9a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=830333fd828a11b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.230+0000 7ff9a7fff700 1 -- 172.21.15.1:0/3067350802 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a800e070 con 0x7ff9b00d1c60 2024-03-18T20:25:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.230+0000 7ff9a7fff700 1 -- 172.21.15.1:0/3067350802 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a800bbd0 con 0x7ff9b00d1c60 2024-03-18T20:25:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.230+0000 7ff9a7fff700 1 -- 172.21.15.1:0/3067350802 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8012430 con 0x7ff9b00d1c60 2024-03-18T20:25:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.230+0000 7ff9b7a59700 1 -- 172.21.15.1:0/3067350802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9b00d1c60 msgr2=0x7ff9b00b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.231+0000 7ff9b7a59700 1 --2- 172.21.15.1:0/3067350802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9b00d1c60 0x7ff9b00b8b00 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8009d90 tx=0x7ff9a800b330 comp rx=0 tx=0).stop 2024-03-18T20:25:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.231+0000 7ff9b7a59700 1 -- 172.21.15.1:0/3067350802 shutdown_connections 2024-03-18T20:25:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.231+0000 7ff9b7a59700 1 --2- 172.21.15.1:0/3067350802 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9b00d1c60 0x7ff9b00b8b00 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.231+0000 7ff9b7a59700 1 -- 172.21.15.1:0/3067350802 >> 172.21.15.1:0/3067350802 conn(0x7ff9b009c530 msgr2=0x7ff9b00d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.231+0000 7ff9b7a59700 1 -- 172.21.15.1:0/3067350802 shutdown_connections 2024-03-18T20:25:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.231+0000 7ff9b7a59700 1 -- 172.21.15.1:0/3067350802 wait complete. 2024-03-18T20:25:06.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.232+0000 7ff9b7a59700 1 Processor -- start 2024-03-18T20:25:06.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.232+0000 7ff9b7a59700 1 -- start start 2024-03-18T20:25:06.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9b7a59700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9b00d17b0 0x7ff9b00cb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:06.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9b7a59700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9b00d1c60 0x7ff9b00b7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:06.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9b7a59700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9b015d500 0x7ff9b00b8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:06.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9b7a59700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b00cb000 con 0x7ff9b00d1c60 2024-03-18T20:25:06.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9b7a59700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b00c9370 con 0x7ff9b015d500 2024-03-18T20:25:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9b7a59700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b00c9650 con 0x7ff9b00d17b0 2024-03-18T20:25:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9b4ff4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9b00d1c60 0x7ff9b00b7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9b4ff4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9b00d1c60 0x7ff9b00b7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60606/0 (socket says 172.21.15.1:60606) 2024-03-18T20:25:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9b4ff4700 1 -- 172.21.15.1:0/1018659620 learned_addr learned my addr 172.21.15.1:0/1018659620 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9b57f5700 1 --2- 172.21.15.1:0/1018659620 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9b00d17b0 0x7ff9b00cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9b5ff6700 1 --2- 172.21.15.1:0/1018659620 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9b015d500 0x7ff9b00b8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9b4ff4700 1 -- 172.21.15.1:0/1018659620 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9b00d17b0 msgr2=0x7ff9b00cb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9b4ff4700 1 --2- 172.21.15.1:0/1018659620 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9b00d17b0 0x7ff9b00cb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9b4ff4700 1 -- 172.21.15.1:0/1018659620 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9b015d500 msgr2=0x7ff9b00b8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9b4ff4700 1 --2- 172.21.15.1:0/1018659620 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9b015d500 0x7ff9b00b8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9b4ff4700 1 -- 172.21.15.1:0/1018659620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a8009a70 con 0x7ff9b00d1c60 2024-03-18T20:25:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9b4ff4700 1 --2- 172.21.15.1:0/1018659620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9b00d1c60 0x7ff9b00b7fe0 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7ff9a000c520 tx=0x7ff9a000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9a5ffb700 1 -- 172.21.15.1:0/1018659620 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a0004c80 con 0x7ff9b00d1c60 2024-03-18T20:25:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9a5ffb700 1 -- 172.21.15.1:0/1018659620 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a0004e00 con 0x7ff9b00d1c60 2024-03-18T20:25:06.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9b7a59700 1 -- 172.21.15.1:0/1018659620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b00c9930 con 0x7ff9b00d1c60 2024-03-18T20:25:06.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.233+0000 7ff9a5ffb700 1 -- 172.21.15.1:0/1018659620 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a0007310 con 0x7ff9b00d1c60 2024-03-18T20:25:06.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.234+0000 7ff9b7a59700 1 -- 172.21.15.1:0/1018659620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b00c9ef0 con 0x7ff9b00d1c60 2024-03-18T20:25:06.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.234+0000 7ff9b7a59700 1 -- 172.21.15.1:0/1018659620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b015c580 con 0x7ff9b00d1c60 2024-03-18T20:25:06.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.235+0000 7ff9a5ffb700 1 -- 172.21.15.1:0/1018659620 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 131) v1 ==== 89353+0+0 (secure 0 0 0) 0x7ff9a0007470 con 0x7ff9b00d1c60 2024-03-18T20:25:06.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.236+0000 7ff9a5ffb700 1 --2- 172.21.15.1:0/1018659620 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff99406aed0 0x7ff99406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:06.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.236+0000 7ff9b57f5700 1 -- 172.21.15.1:0/1018659620 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff99406aed0 msgr2=0x7ff99406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:06.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.236+0000 7ff9a5ffb700 1 -- 172.21.15.1:0/1018659620 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6632+0+0 (secure 0 0 0) 0x7ff9a00896e0 con 0x7ff9b00d1c60 2024-03-18T20:25:06.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.236+0000 7ff9b57f5700 1 --2- 172.21.15.1:0/1018659620 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff99406aed0 0x7ff99406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:06.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.240+0000 7ff9a5ffb700 1 -- 172.21.15.1:0/1018659620 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff9a00541c0 con 0x7ff9b00d1c60 2024-03-18T20:25:06.273 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:06.268+0000 7f25953da240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:25:06.437 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:06.433+0000 7f7b493a2240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:25:06.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.437+0000 7ff9b57f5700 1 -- 172.21.15.1:0/1018659620 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff99406aed0 msgr2=0x7ff99406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:06.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.437+0000 7ff9b57f5700 1 --2- 172.21.15.1:0/1018659620 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff99406aed0 0x7ff99406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.447+0000 7ff9b7a59700 1 -- 172.21.15.1:0/1018659620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9b00676a0 con 0x7ff9b00d1c60 2024-03-18T20:25:06.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.449+0000 7ff9a5ffb700 1 -- 172.21.15.1:0/1018659620 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v219) v1 ==== 74+0+17125 (secure 0 0 0) 0x7ff9a00577e0 con 0x7ff9b00d1c60 2024-03-18T20:25:06.454 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:06.454 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":219,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:25:06.176019+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:23:37.591406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-03-18T20:23:38.723169+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":219,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.c","mappings":[{"from":1,"to":0}]},{"pgid":"26.f","mappings":[{"from":1,"to":0}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2462726896":"2024-03-19T20:23:54.844816+0000","172.21.15.1:6833/148356":"2024-03-19T20:23:54.844816+0000","172.21.15.1:6832/148356":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/4177404804":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/1122038130":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/2416229963":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/2662947140":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/2110350357":"2024-03-19T20:23:54.844816+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":26,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:06.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.451+0000 7ff9b7a59700 1 -- 172.21.15.1:0/1018659620 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff99406aed0 msgr2=0x7ff99406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.451+0000 7ff9b7a59700 1 --2- 172.21.15.1:0/1018659620 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff99406aed0 0x7ff99406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.451+0000 7ff9b7a59700 1 -- 172.21.15.1:0/1018659620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9b00d1c60 msgr2=0x7ff9b00b7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.451+0000 7ff9b7a59700 1 --2- 172.21.15.1:0/1018659620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9b00d1c60 0x7ff9b00b7fe0 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7ff9a000c520 tx=0x7ff9a000c830 comp rx=0 tx=0).stop 2024-03-18T20:25:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.451+0000 7ff9b7a59700 1 -- 172.21.15.1:0/1018659620 shutdown_connections 2024-03-18T20:25:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.451+0000 7ff9b7a59700 1 --2- 172.21.15.1:0/1018659620 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff99406aed0 0x7ff99406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.451+0000 7ff9b7a59700 1 --2- 172.21.15.1:0/1018659620 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff9b00d17b0 0x7ff9b00cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.451+0000 7ff9b7a59700 1 --2- 172.21.15.1:0/1018659620 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9b015d500 0x7ff9b00b8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.451+0000 7ff9b7a59700 1 --2- 172.21.15.1:0/1018659620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff9b00d1c60 0x7ff9b00b7fe0 unknown :-1 s=CLOSED pgs=2421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.451+0000 7ff9b7a59700 1 -- 172.21.15.1:0/1018659620 >> 172.21.15.1:0/1018659620 conn(0x7ff9b009c530 msgr2=0x7ff9b006c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.451+0000 7ff9b7a59700 1 -- 172.21.15.1:0/1018659620 shutdown_connections 2024-03-18T20:25:06.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.451+0000 7ff9b7a59700 1 -- 172.21.15.1:0/1018659620 wait complete. 2024-03-18T20:25:06.458 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:06.451+0000 7f25953da240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:25:06.469 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:23:37.591406+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '213', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:23:38.723169+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '219', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 219, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:25:06.469 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:25:06.589 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:06.585+0000 7f7b493a2240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:25:06.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.599+0000 7f6e5d960700 1 Processor -- start 2024-03-18T20:25:06.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.599+0000 7f6e5d960700 1 -- start start 2024-03-18T20:25:06.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.600+0000 7f6e5d960700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6e5815a240 0x7f6e580ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:06.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.600+0000 7f6e5d960700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6e5806d560 0x7f6e5806fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:06.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.600+0000 7f6e5d960700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e58070000 con 0x7f6e580715b0 2024-03-18T20:25:06.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.600+0000 7f6e5d960700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e580adff0 con 0x7f6e5806d560 2024-03-18T20:25:06.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.600+0000 7f6e5d960700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e580cac80 con 0x7f6e5815a240 2024-03-18T20:25:06.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.600+0000 7f6e567fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6e5806d560 0x7f6e5806fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:06.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.600+0000 7f6e577fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6e580715b0 0x7f6e5815de10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53010/0 (socket says 172.21.15.1:53010) 2024-03-18T20:25:06.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.600+0000 7f6e577fe700 1 -- 172.21.15.1:0/3482948913 learned_addr learned my addr 172.21.15.1:0/3482948913 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:06.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.600+0000 7f6e56ffd700 1 --2- 172.21.15.1:0/3482948913 >> v2:172.21.15.165:3300/0 conn(0x7f6e5815a240 0x7f6e580ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:06.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.600+0000 7f6e567fc700 1 -- 172.21.15.1:0/3482948913 >> v2:172.21.15.165:3300/0 conn(0x7f6e5815a240 msgr2=0x7f6e580ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:06.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.600+0000 7f6e567fc700 1 --2- 172.21.15.1:0/3482948913 >> v2:172.21.15.165:3300/0 conn(0x7f6e5815a240 0x7f6e580ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.600+0000 7f6e567fc700 1 -- 172.21.15.1:0/3482948913 >> v1:172.21.15.1:6789/0 conn(0x7f6e580715b0 legacy=0x7f6e5815de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:06.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.600+0000 7f6e55ffb700 1 -- 172.21.15.1:0/3482948913 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767617922 0 0) 0x7f6e58070000 con 0x7f6e580715b0 2024-03-18T20:25:06.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.600+0000 7f6e567fc700 1 -- 172.21.15.1:0/3482948913 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e580caf60 con 0x7f6e5806d560 2024-03-18T20:25:06.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.601+0000 7f6e567fc700 1 --2- 172.21.15.1:0/3482948913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e5806d560 0x7f6e5806fac0 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c00bae0 tx=0x7f6e4c00bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e19642c4999d93d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:06.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.601+0000 7f6e55ffb700 1 -- 172.21.15.1:0/3482948913 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c00dc30 con 0x7f6e5806d560 2024-03-18T20:25:06.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.601+0000 7f6e55ffb700 1 -- 172.21.15.1:0/3482948913 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e4c00dd90 con 0x7f6e5806d560 2024-03-18T20:25:06.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.601+0000 7f6e55ffb700 1 -- 172.21.15.1:0/3482948913 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c010690 con 0x7f6e5806d560 2024-03-18T20:25:06.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.601+0000 7f6e5d960700 1 -- 172.21.15.1:0/3482948913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e5806d560 msgr2=0x7f6e5806fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:06.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.601+0000 7f6e5d960700 1 --2- 172.21.15.1:0/3482948913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e5806d560 0x7f6e5806fac0 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c00bae0 tx=0x7f6e4c00bdf0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.602+0000 7f6e5d960700 1 -- 172.21.15.1:0/3482948913 shutdown_connections 2024-03-18T20:25:06.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.602+0000 7f6e5d960700 1 --2- 172.21.15.1:0/3482948913 >> v2:172.21.15.165:3300/0 conn(0x7f6e5815a240 0x7f6e580ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.602+0000 7f6e5d960700 1 --2- 172.21.15.1:0/3482948913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e5806d560 0x7f6e5806fac0 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.602+0000 7f6e5d960700 1 -- 172.21.15.1:0/3482948913 >> 172.21.15.1:0/3482948913 conn(0x7f6e580cff30 msgr2=0x7f6e580d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:06.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.602+0000 7f6e5d960700 1 -- 172.21.15.1:0/3482948913 shutdown_connections 2024-03-18T20:25:06.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.602+0000 7f6e5d960700 1 -- 172.21.15.1:0/3482948913 wait complete. 2024-03-18T20:25:06.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.603+0000 7f6e5d960700 1 Processor -- start 2024-03-18T20:25:06.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.603+0000 7f6e5d960700 1 -- start start 2024-03-18T20:25:06.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.604+0000 7f6e5d960700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e5806d560 0x7f6e580ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:06.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.604+0000 7f6e5d960700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e580715b0 0x7f6e580c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:06.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.604+0000 7f6e5d960700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e5815a240 0x7f6e581565e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:06.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.604+0000 7f6e5d960700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e580c85e0 con 0x7f6e580715b0 2024-03-18T20:25:06.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.604+0000 7f6e5d960700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e580c9eb0 con 0x7f6e5815a240 2024-03-18T20:25:06.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.604+0000 7f6e5d960700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e58157150 con 0x7f6e5806d560 2024-03-18T20:25:06.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.604+0000 7f6e567fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e580715b0 0x7f6e580c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:06.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.604+0000 7f6e567fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e580715b0 0x7f6e580c8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60614/0 (socket says 172.21.15.1:60614) 2024-03-18T20:25:06.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.604+0000 7f6e567fc700 1 -- 172.21.15.1:0/2185561206 learned_addr learned my addr 172.21.15.1:0/2185561206 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:06.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.604+0000 7f6e577fe700 1 --2- 172.21.15.1:0/2185561206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e5815a240 0x7f6e581565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:06.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.604+0000 7f6e56ffd700 1 --2- 172.21.15.1:0/2185561206 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e5806d560 0x7f6e580ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:06.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.604+0000 7f6e567fc700 1 -- 172.21.15.1:0/2185561206 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e5806d560 msgr2=0x7f6e580ceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:06.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.604+0000 7f6e567fc700 1 --2- 172.21.15.1:0/2185561206 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e5806d560 0x7f6e580ceed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.604+0000 7f6e567fc700 1 -- 172.21.15.1:0/2185561206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e5815a240 msgr2=0x7f6e581565e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:06.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.604+0000 7f6e567fc700 1 --2- 172.21.15.1:0/2185561206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e5815a240 0x7f6e581565e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.604+0000 7f6e567fc700 1 -- 172.21.15.1:0/2185561206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c00b740 con 0x7f6e580715b0 2024-03-18T20:25:06.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.604+0000 7f6e577fe700 1 --2- 172.21.15.1:0/2185561206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e5815a240 0x7f6e581565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:06.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.604+0000 7f6e567fc700 1 --2- 172.21.15.1:0/2185561206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e580715b0 0x7f6e580c8a70 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c00b2c0 tx=0x7f6e4c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:06.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.605+0000 7f6e3ffff700 1 -- 172.21.15.1:0/2185561206 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c003b70 con 0x7f6e580715b0 2024-03-18T20:25:06.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.605+0000 7f6e3ffff700 1 -- 172.21.15.1:0/2185561206 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e4c003cd0 con 0x7f6e580715b0 2024-03-18T20:25:06.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.605+0000 7f6e5d960700 1 -- 172.21.15.1:0/2185561206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e58157430 con 0x7f6e580715b0 2024-03-18T20:25:06.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.605+0000 7f6e3ffff700 1 -- 172.21.15.1:0/2185561206 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c0106b0 con 0x7f6e580715b0 2024-03-18T20:25:06.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.605+0000 7f6e5d960700 1 -- 172.21.15.1:0/2185561206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e580c8270 con 0x7f6e580715b0 2024-03-18T20:25:06.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.606+0000 7f6e5d960700 1 -- 172.21.15.1:0/2185561206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e580c8400 con 0x7f6e580715b0 2024-03-18T20:25:06.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.607+0000 7f6e3ffff700 1 -- 172.21.15.1:0/2185561206 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 131) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f6e4c023440 con 0x7f6e580715b0 2024-03-18T20:25:06.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.607+0000 7f6e3ffff700 1 --2- 172.21.15.1:0/2185561206 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6e4406aed0 0x7f6e4406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:06.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.607+0000 7f6e56ffd700 1 -- 172.21.15.1:0/2185561206 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6e4406aed0 msgr2=0x7f6e4406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:06.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.607+0000 7f6e3ffff700 1 -- 172.21.15.1:0/2185561206 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f6e4c08b2b0 con 0x7f6e580715b0 2024-03-18T20:25:06.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.607+0000 7f6e56ffd700 1 --2- 172.21.15.1:0/2185561206 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6e4406aed0 0x7f6e4406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:06.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.612+0000 7f6e3ffff700 1 -- 172.21.15.1:0/2185561206 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6e4c055d10 con 0x7f6e580715b0 2024-03-18T20:25:06.663 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:06.660+0000 7f7b493a2240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:25:06.687 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:06.682+0000 7f25953da240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:25:06.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.808+0000 7f6e56ffd700 1 -- 172.21.15.1:0/2185561206 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6e4406aed0 msgr2=0x7f6e4406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:06.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.808+0000 7f6e56ffd700 1 --2- 172.21.15.1:0/2185561206 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6e4406aed0 0x7f6e4406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:06.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.816+0000 7f6e5d960700 1 -- 172.21.15.1:0/2185561206 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6e580c7df0 con 0x7f6e580715b0 2024-03-18T20:25:06.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.817+0000 7f6e3ffff700 1 -- 172.21.15.1:0/2185561206 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v219) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6e580c7df0 con 0x7f6e580715b0 2024-03-18T20:25:06.822 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:25:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.819+0000 7f6e5d960700 1 -- 172.21.15.1:0/2185561206 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6e4406aed0 msgr2=0x7f6e4406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:06.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.819+0000 7f6e5d960700 1 --2- 172.21.15.1:0/2185561206 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6e4406aed0 0x7f6e4406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.819+0000 7f6e5d960700 1 -- 172.21.15.1:0/2185561206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e580715b0 msgr2=0x7f6e580c8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:06.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.819+0000 7f6e5d960700 1 --2- 172.21.15.1:0/2185561206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e580715b0 0x7f6e580c8a70 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c00b2c0 tx=0x7f6e4c004750 comp rx=0 tx=0).stop 2024-03-18T20:25:06.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.819+0000 7f6e5d960700 1 -- 172.21.15.1:0/2185561206 shutdown_connections 2024-03-18T20:25:06.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.819+0000 7f6e5d960700 1 --2- 172.21.15.1:0/2185561206 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6e4406aed0 0x7f6e4406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.819+0000 7f6e5d960700 1 --2- 172.21.15.1:0/2185561206 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e5806d560 0x7f6e580ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.819+0000 7f6e5d960700 1 --2- 172.21.15.1:0/2185561206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e5815a240 0x7f6e581565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.819+0000 7f6e5d960700 1 --2- 172.21.15.1:0/2185561206 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e580715b0 0x7f6e580c8a70 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.819+0000 7f6e5d960700 1 -- 172.21.15.1:0/2185561206 >> 172.21.15.1:0/2185561206 conn(0x7f6e580cff30 msgr2=0x7f6e580d0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:06.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.819+0000 7f6e5d960700 1 -- 172.21.15.1:0/2185561206 shutdown_connections 2024-03-18T20:25:06.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.819+0000 7f6e5d960700 1 -- 172.21.15.1:0/2185561206 wait complete. 2024-03-18T20:25:06.837 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:25:06.855 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:06.850+0000 7f25953da240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:25:06.935 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:06.930+0000 7f25953da240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:25:06.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.963+0000 7faf321ae700 1 Processor -- start 2024-03-18T20:25:06.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.963+0000 7faf321ae700 1 -- start start 2024-03-18T20:25:06.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.964+0000 7faf321ae700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf2c160a70 0x7faf2c0d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:06.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.964+0000 7faf321ae700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7faf2c160e50 0x7faf2c0cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:06.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.964+0000 7faf321ae700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf2c162170 con 0x7faf2c1612d0 2024-03-18T20:25:06.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.964+0000 7faf321ae700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf2c0cef30 con 0x7faf2c160e50 2024-03-18T20:25:06.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.964+0000 7faf321ae700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c0cf210 con 0x7faf2c160a70 2024-03-18T20:25:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.964+0000 7faf2bfff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7faf2c1612d0 0x7faf2c0c6da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53024/0 (socket says 172.21.15.1:53024) 2024-03-18T20:25:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.964+0000 7faf2bfff700 1 -- 172.21.15.1:0/1060062741 learned_addr learned my addr 172.21.15.1:0/1060062741 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.964+0000 7faf2b7fe700 1 --2- 172.21.15.1:0/1060062741 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf2c160a70 0x7faf2c0d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.964+0000 7faf2affd700 1 --2- 172.21.15.1:0/1060062741 >> v2:172.21.15.165:3300/0 conn(0x7faf2c160e50 0x7faf2c0cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.964+0000 7faf2a7fc700 1 -- 172.21.15.1:0/1060062741 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923669002 0 0) 0x7faf1c001490 con 0x7faf2c1612d0 2024-03-18T20:25:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.964+0000 7faf2a7fc700 1 -- 172.21.15.1:0/1060062741 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf180035b0 con 0x7faf2c1612d0 2024-03-18T20:25:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.965+0000 7faf2affd700 1 -- 172.21.15.1:0/1060062741 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf2c160a70 msgr2=0x7faf2c0d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.965+0000 7faf2affd700 1 --2- 172.21.15.1:0/1060062741 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf2c160a70 0x7faf2c0d3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.965+0000 7faf2affd700 1 -- 172.21.15.1:0/1060062741 >> v1:172.21.15.1:6789/0 conn(0x7faf2c1612d0 legacy=0x7faf2c0c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.965+0000 7faf2affd700 1 -- 172.21.15.1:0/1060062741 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2c0cf4f0 con 0x7faf2c160e50 2024-03-18T20:25:06.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.965+0000 7faf2b7fe700 1 --2- 172.21.15.1:0/1060062741 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf2c160a70 0x7faf2c0d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.965+0000 7faf2affd700 1 --2- 172.21.15.1:0/1060062741 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf2c160e50 0x7faf2c0cffd0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7faf14009840 tx=0x7faf14009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0af791e5db76756 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.965+0000 7faf2a7fc700 1 -- 172.21.15.1:0/1060062741 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf14011830 con 0x7faf2c160e50 2024-03-18T20:25:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.965+0000 7faf2a7fc700 1 -- 172.21.15.1:0/1060062741 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf14011990 con 0x7faf2c160e50 2024-03-18T20:25:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.965+0000 7faf2a7fc700 1 -- 172.21.15.1:0/1060062741 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf14011d20 con 0x7faf2c160e50 2024-03-18T20:25:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.966+0000 7faf321ae700 1 -- 172.21.15.1:0/1060062741 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf2c160e50 msgr2=0x7faf2c0cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.966+0000 7faf321ae700 1 --2- 172.21.15.1:0/1060062741 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf2c160e50 0x7faf2c0cffd0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7faf14009840 tx=0x7faf14009b50 comp rx=0 tx=0).stop 2024-03-18T20:25:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.966+0000 7faf321ae700 1 -- 172.21.15.1:0/1060062741 shutdown_connections 2024-03-18T20:25:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.966+0000 7faf321ae700 1 --2- 172.21.15.1:0/1060062741 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf2c160a70 0x7faf2c0d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.966+0000 7faf321ae700 1 --2- 172.21.15.1:0/1060062741 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf2c160e50 0x7faf2c0cffd0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.966+0000 7faf321ae700 1 -- 172.21.15.1:0/1060062741 >> 172.21.15.1:0/1060062741 conn(0x7faf2c09c560 msgr2=0x7faf2c0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:06.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.966+0000 7faf321ae700 1 -- 172.21.15.1:0/1060062741 shutdown_connections 2024-03-18T20:25:06.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.966+0000 7faf321ae700 1 -- 172.21.15.1:0/1060062741 wait complete. 2024-03-18T20:25:06.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.967+0000 7faf321ae700 1 Processor -- start 2024-03-18T20:25:06.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.967+0000 7faf321ae700 1 -- start start 2024-03-18T20:25:06.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.968+0000 7faf321ae700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf2c160a70 0x7faf2c15c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:06.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.968+0000 7faf321ae700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf2c160e50 0x7faf2c0d3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:06.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.968+0000 7faf321ae700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf2c1612d0 0x7faf2c0cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:06.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.968+0000 7faf321ae700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c1f42b0 con 0x7faf2c160a70 2024-03-18T20:25:06.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.968+0000 7faf321ae700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c1f3630 con 0x7faf2c1612d0 2024-03-18T20:25:06.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.968+0000 7faf321ae700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c1f3910 con 0x7faf2c160e50 2024-03-18T20:25:06.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.968+0000 7faf2b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf2c160a70 0x7faf2c15c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:06.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.968+0000 7faf2b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf2c160a70 0x7faf2c15c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60618/0 (socket says 172.21.15.1:60618) 2024-03-18T20:25:06.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.968+0000 7faf2b7fe700 1 -- 172.21.15.1:0/3492184717 learned_addr learned my addr 172.21.15.1:0/3492184717 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:06.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.968+0000 7faf2bfff700 1 --2- 172.21.15.1:0/3492184717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf2c1612d0 0x7faf2c0cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:06.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.968+0000 7faf2affd700 1 --2- 172.21.15.1:0/3492184717 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf2c160e50 0x7faf2c0d3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:06.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.968+0000 7faf2b7fe700 1 -- 172.21.15.1:0/3492184717 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf2c160e50 msgr2=0x7faf2c0d3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:06.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.968+0000 7faf2b7fe700 1 --2- 172.21.15.1:0/3492184717 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf2c160e50 0x7faf2c0d3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.968+0000 7faf2b7fe700 1 -- 172.21.15.1:0/3492184717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf2c1612d0 msgr2=0x7faf2c0cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:06.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.968+0000 7faf2b7fe700 1 --2- 172.21.15.1:0/3492184717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf2c1612d0 0x7faf2c0cc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.968+0000 7faf2b7fe700 1 -- 172.21.15.1:0/3492184717 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf240094a0 con 0x7faf2c160a70 2024-03-18T20:25:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.968+0000 7faf2bfff700 1 --2- 172.21.15.1:0/3492184717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf2c1612d0 0x7faf2c0cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.968+0000 7faf2b7fe700 1 --2- 172.21.15.1:0/3492184717 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf2c160a70 0x7faf2c15c9a0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7faf2400ebe0 tx=0x7faf2400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.969+0000 7faf23fff700 1 -- 172.21.15.1:0/3492184717 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf24009760 con 0x7faf2c160a70 2024-03-18T20:25:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.969+0000 7faf23fff700 1 -- 172.21.15.1:0/3492184717 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2400c850 con 0x7faf2c160a70 2024-03-18T20:25:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.969+0000 7faf23fff700 1 -- 172.21.15.1:0/3492184717 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2400cbe0 con 0x7faf2c160a70 2024-03-18T20:25:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.969+0000 7faf321ae700 1 -- 172.21.15.1:0/3492184717 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf140094a0 con 0x7faf2c160a70 2024-03-18T20:25:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.969+0000 7faf321ae700 1 -- 172.21.15.1:0/3492184717 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf2c1f3ed0 con 0x7faf2c160a70 2024-03-18T20:25:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.970+0000 7faf321ae700 1 -- 172.21.15.1:0/3492184717 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf2c067390 con 0x7faf2c160a70 2024-03-18T20:25:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.970+0000 7faf23fff700 1 -- 172.21.15.1:0/3492184717 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 131) v1 ==== 89353+0+0 (secure 0 0 0) 0x7faf2400cd40 con 0x7faf2c160a70 2024-03-18T20:25:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.971+0000 7faf23fff700 1 --2- 172.21.15.1:0/3492184717 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf0c06aed0 0x7faf0c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.971+0000 7faf2affd700 1 -- 172.21.15.1:0/3492184717 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf0c06aed0 msgr2=0x7faf0c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.971+0000 7faf23fff700 1 -- 172.21.15.1:0/3492184717 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6632+0+0 (secure 0 0 0) 0x7faf24089a60 con 0x7faf2c160a70 2024-03-18T20:25:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.971+0000 7faf2affd700 1 --2- 172.21.15.1:0/3492184717 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf0c06aed0 0x7faf0c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:06.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:06.975+0000 7faf23fff700 1 -- 172.21.15.1:0/3492184717 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faf240545d0 con 0x7faf2c160a70 2024-03-18T20:25:07.113 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:07.109+0000 7f7b493a2240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:25:07.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.172+0000 7faf2affd700 1 -- 172.21.15.1:0/3492184717 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf0c06aed0 msgr2=0x7faf0c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:07.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.172+0000 7faf2affd700 1 --2- 172.21.15.1:0/3492184717 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf0c06aed0 0x7faf0c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:07.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.175+0000 7faf321ae700 1 -- 172.21.15.1:0/3492184717 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faf2c0676a0 con 0x7faf2c160a70 2024-03-18T20:25:07.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.176+0000 7faf23fff700 1 -- 172.21.15.1:0/3492184717 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v219) v1 ==== 104+0+11 (secure 0 0 0) 0x7faf24057bf0 con 0x7faf2c160a70 2024-03-18T20:25:07.181 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:25:07.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.178+0000 7faf321ae700 1 -- 172.21.15.1:0/3492184717 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf0c06aed0 msgr2=0x7faf0c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:07.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.178+0000 7faf321ae700 1 --2- 172.21.15.1:0/3492184717 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf0c06aed0 0x7faf0c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.178+0000 7faf321ae700 1 -- 172.21.15.1:0/3492184717 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf2c160a70 msgr2=0x7faf2c15c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:07.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.178+0000 7faf321ae700 1 --2- 172.21.15.1:0/3492184717 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf2c160a70 0x7faf2c15c9a0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7faf2400ebe0 tx=0x7faf2400eef0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.178+0000 7faf321ae700 1 -- 172.21.15.1:0/3492184717 shutdown_connections 2024-03-18T20:25:07.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.178+0000 7faf321ae700 1 --2- 172.21.15.1:0/3492184717 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf0c06aed0 0x7faf0c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.178+0000 7faf321ae700 1 --2- 172.21.15.1:0/3492184717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf2c1612d0 0x7faf2c0cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.178+0000 7faf321ae700 1 --2- 172.21.15.1:0/3492184717 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf2c160a70 0x7faf2c15c9a0 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.178+0000 7faf321ae700 1 --2- 172.21.15.1:0/3492184717 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf2c160e50 0x7faf2c0d3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.178+0000 7faf321ae700 1 -- 172.21.15.1:0/3492184717 >> 172.21.15.1:0/3492184717 conn(0x7faf2c09c560 msgr2=0x7faf2c0c9610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:07.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.178+0000 7faf321ae700 1 -- 172.21.15.1:0/3492184717 shutdown_connections 2024-03-18T20:25:07.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.178+0000 7faf321ae700 1 -- 172.21.15.1:0/3492184717 wait complete. 2024-03-18T20:25:07.196 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:25:07.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.362+0000 7f9e9adc5700 1 Processor -- start 2024-03-18T20:25:07.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.362+0000 7f9e9adc5700 1 -- start start 2024-03-18T20:25:07.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.363+0000 7f9e9adc5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9e9415b6c0 0x7f9e940b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:07.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.363+0000 7f9e9adc5700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e940d01f0 con 0x7f9e940c87d0 2024-03-18T20:25:07.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.363+0000 7f9e9adc5700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e940ccfa0 con 0x7f9e940cd430 2024-03-18T20:25:07.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.363+0000 7f9e9adc5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e940d1db0 con 0x7f9e9415b6c0 2024-03-18T20:25:07.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.364+0000 7f9e99362700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9e940c87d0 0x7f9e940cd9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53034/0 (socket says 172.21.15.1:53034) 2024-03-18T20:25:07.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.364+0000 7f9e98b61700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9e9415b6c0 0x7f9e940b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:07.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.364+0000 7f9e99362700 1 -- 172.21.15.1:0/897826062 learned_addr learned my addr 172.21.15.1:0/897826062 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:07.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.364+0000 7f9e937fe700 1 -- 172.21.15.1:0/897826062 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161265051 0 0) 0x7f9e940d01f0 con 0x7f9e940c87d0 2024-03-18T20:25:07.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.364+0000 7f9e937fe700 1 -- 172.21.15.1:0/897826062 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e840035b0 con 0x7f9e940c87d0 2024-03-18T20:25:07.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.364+0000 7f9e937fe700 1 -- 172.21.15.1:0/897826062 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537019361 0 0) 0x7f9e940ccfa0 con 0x7f9e940cd430 2024-03-18T20:25:07.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.364+0000 7f9e937fe700 1 -- 172.21.15.1:0/897826062 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e940d01f0 con 0x7f9e940cd430 2024-03-18T20:25:07.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.364+0000 7f9e98b61700 1 -- 172.21.15.1:0/897826062 >> v1:172.21.15.165:6789/0 conn(0x7f9e940cd430 legacy=0x7f9e940c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:07.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.364+0000 7f9e98b61700 1 -- 172.21.15.1:0/897826062 >> v1:172.21.15.1:6789/0 conn(0x7f9e940c87d0 legacy=0x7f9e940cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:07.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.364+0000 7f9e98b61700 1 -- 172.21.15.1:0/897826062 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e940cff10 con 0x7f9e9415b6c0 2024-03-18T20:25:07.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.364+0000 7f9e98b61700 1 --2- 172.21.15.1:0/897826062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9415b6c0 0x7f9e940b2160 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9e88009840 tx=0x7f9e88009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50fa2138674edd7e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:07.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.364+0000 7f9e937fe700 1 -- 172.21.15.1:0/897826062 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e88011830 con 0x7f9e9415b6c0 2024-03-18T20:25:07.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.364+0000 7f9e937fe700 1 -- 172.21.15.1:0/897826062 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e88011990 con 0x7f9e9415b6c0 2024-03-18T20:25:07.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.364+0000 7f9e937fe700 1 -- 172.21.15.1:0/897826062 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e88011d20 con 0x7f9e9415b6c0 2024-03-18T20:25:07.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.365+0000 7f9e9adc5700 1 -- 172.21.15.1:0/897826062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9415b6c0 msgr2=0x7f9e940b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:07.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.365+0000 7f9e9adc5700 1 --2- 172.21.15.1:0/897826062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9415b6c0 0x7f9e940b2160 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9e88009840 tx=0x7f9e88009b50 comp rx=0 tx=0).stop 2024-03-18T20:25:07.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.365+0000 7f9e9adc5700 1 -- 172.21.15.1:0/897826062 shutdown_connections 2024-03-18T20:25:07.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.365+0000 7f9e9adc5700 1 --2- 172.21.15.1:0/897826062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9415b6c0 0x7f9e940b2160 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.365+0000 7f9e9adc5700 1 -- 172.21.15.1:0/897826062 >> 172.21.15.1:0/897826062 conn(0x7f9e940d0ff0 msgr2=0x7f9e940d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:07.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.365+0000 7f9e9adc5700 1 -- 172.21.15.1:0/897826062 shutdown_connections 2024-03-18T20:25:07.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.365+0000 7f9e9adc5700 1 -- 172.21.15.1:0/897826062 wait complete. 2024-03-18T20:25:07.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.367+0000 7f9e9adc5700 1 Processor -- start 2024-03-18T20:25:07.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.367+0000 7f9e9adc5700 1 -- start start 2024-03-18T20:25:07.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.368+0000 7f9e9adc5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e940c87d0 0x7f9e940b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:07.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.368+0000 7f9e9adc5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e940cd430 0x7f9e940b5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:07.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.368+0000 7f9e9adc5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9415b6c0 0x7f9e940b5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:07.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.368+0000 7f9e9adc5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e94147770 con 0x7f9e940c87d0 2024-03-18T20:25:07.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.368+0000 7f9e9adc5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e94149fb0 con 0x7f9e9415b6c0 2024-03-18T20:25:07.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.368+0000 7f9e9adc5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e940b24d0 con 0x7f9e940cd430 2024-03-18T20:25:07.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.368+0000 7f9e98b61700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e940c87d0 0x7f9e940b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:07.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.368+0000 7f9e98b61700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e940c87d0 0x7f9e940b1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60622/0 (socket says 172.21.15.1:60622) 2024-03-18T20:25:07.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.368+0000 7f9e98b61700 1 -- 172.21.15.1:0/3309163511 learned_addr learned my addr 172.21.15.1:0/3309163511 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:07.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.368+0000 7f9e99362700 1 --2- 172.21.15.1:0/3309163511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9415b6c0 0x7f9e940b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:07.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.368+0000 7f9e93fff700 1 --2- 172.21.15.1:0/3309163511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e940cd430 0x7f9e940b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:07.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.368+0000 7f9e98b61700 1 -- 172.21.15.1:0/3309163511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e940cd430 msgr2=0x7f9e940b5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:07.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.368+0000 7f9e98b61700 1 --2- 172.21.15.1:0/3309163511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e940cd430 0x7f9e940b5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.368+0000 7f9e98b61700 1 -- 172.21.15.1:0/3309163511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9415b6c0 msgr2=0x7f9e940b5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:07.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.368+0000 7f9e98b61700 1 --2- 172.21.15.1:0/3309163511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9415b6c0 0x7f9e940b5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.368+0000 7f9e98b61700 1 -- 172.21.15.1:0/3309163511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e880094a0 con 0x7f9e940c87d0 2024-03-18T20:25:07.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.368+0000 7f9e99362700 1 --2- 172.21.15.1:0/3309163511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9415b6c0 0x7f9e940b5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:07.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.368+0000 7f9e98b61700 1 --2- 172.21.15.1:0/3309163511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e940c87d0 0x7f9e940b1f90 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7f9e88014040 tx=0x7f9e88004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:07.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.369+0000 7f9e917fa700 1 -- 172.21.15.1:0/3309163511 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8801a6e0 con 0x7f9e940c87d0 2024-03-18T20:25:07.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.369+0000 7f9e917fa700 1 -- 172.21.15.1:0/3309163511 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8801a840 con 0x7f9e940c87d0 2024-03-18T20:25:07.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.369+0000 7f9e917fa700 1 -- 172.21.15.1:0/3309163511 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8801abb0 con 0x7f9e940c87d0 2024-03-18T20:25:07.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.369+0000 7f9e9adc5700 1 -- 172.21.15.1:0/3309163511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e941470f0 con 0x7f9e940c87d0 2024-03-18T20:25:07.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.369+0000 7f9e9adc5700 1 -- 172.21.15.1:0/3309163511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e94147390 con 0x7f9e940c87d0 2024-03-18T20:25:07.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.370+0000 7f9e9adc5700 1 -- 172.21.15.1:0/3309163511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e94146f50 con 0x7f9e940c87d0 2024-03-18T20:25:07.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.370+0000 7f9e917fa700 1 -- 172.21.15.1:0/3309163511 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 131) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f9e88022070 con 0x7f9e940c87d0 2024-03-18T20:25:07.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.371+0000 7f9e917fa700 1 --2- 172.21.15.1:0/3309163511 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e7806aed0 0x7f9e7806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:07.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.371+0000 7f9e93fff700 1 -- 172.21.15.1:0/3309163511 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e7806aed0 msgr2=0x7f9e7806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:07.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.371+0000 7f9e917fa700 1 -- 172.21.15.1:0/3309163511 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f9e8808a590 con 0x7f9e940c87d0 2024-03-18T20:25:07.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.371+0000 7f9e93fff700 1 --2- 172.21.15.1:0/3309163511 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e7806aed0 0x7f9e7806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:07.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.379+0000 7f9e917fa700 1 -- 172.21.15.1:0/3309163511 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9e94146f50 con 0x7f9e940c87d0 2024-03-18T20:25:07.426 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:07.421+0000 7f25953da240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:25:07.454 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:07.450+0000 7f7b493a2240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:25:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.572+0000 7f9e93fff700 1 -- 172.21.15.1:0/3309163511 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e7806aed0 msgr2=0x7f9e7806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.572+0000 7f9e93fff700 1 --2- 172.21.15.1:0/3309163511 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e7806aed0 0x7f9e7806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:07.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.585+0000 7f9e9adc5700 1 -- 172.21.15.1:0/3309163511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9e94146f50 con 0x7f9e940c87d0 2024-03-18T20:25:07.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.586+0000 7f9e917fa700 1 -- 172.21.15.1:0/3309163511 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v219) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9e94146f50 con 0x7f9e940c87d0 2024-03-18T20:25:07.591 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:25:07.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.588+0000 7f9e9adc5700 1 -- 172.21.15.1:0/3309163511 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e7806aed0 msgr2=0x7f9e7806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:07.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.588+0000 7f9e9adc5700 1 --2- 172.21.15.1:0/3309163511 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e7806aed0 0x7f9e7806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.588+0000 7f9e9adc5700 1 -- 172.21.15.1:0/3309163511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e940c87d0 msgr2=0x7f9e940b1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:07.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.588+0000 7f9e9adc5700 1 --2- 172.21.15.1:0/3309163511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e940c87d0 0x7f9e940b1f90 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7f9e88014040 tx=0x7f9e88004750 comp rx=0 tx=0).stop 2024-03-18T20:25:07.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.588+0000 7f9e9adc5700 1 -- 172.21.15.1:0/3309163511 shutdown_connections 2024-03-18T20:25:07.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.588+0000 7f9e9adc5700 1 --2- 172.21.15.1:0/3309163511 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e7806aed0 0x7f9e7806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.588+0000 7f9e9adc5700 1 --2- 172.21.15.1:0/3309163511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e9415b6c0 0x7f9e940b5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.588+0000 7f9e9adc5700 1 --2- 172.21.15.1:0/3309163511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e940c87d0 0x7f9e940b1f90 unknown :-1 s=CLOSED pgs=2428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.588+0000 7f9e9adc5700 1 --2- 172.21.15.1:0/3309163511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e940cd430 0x7f9e940b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.588+0000 7f9e9adc5700 1 -- 172.21.15.1:0/3309163511 >> 172.21.15.1:0/3309163511 conn(0x7f9e940d0ff0 msgr2=0x7f9e940cd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:07.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.588+0000 7f9e9adc5700 1 -- 172.21.15.1:0/3309163511 shutdown_connections 2024-03-18T20:25:07.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.588+0000 7f9e9adc5700 1 -- 172.21.15.1:0/3309163511 wait complete. 2024-03-18T20:25:07.606 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:25:07.618 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:07.614+0000 7f7b493a2240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:25:07.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.735+0000 7fb21cccd700 1 Processor -- start 2024-03-18T20:25:07.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.735+0000 7fb21cccd700 1 -- start start 2024-03-18T20:25:07.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.736+0000 7fb21cccd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2180b2350 0x7fb2180b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:07.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.736+0000 7fb21cccd700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb21815d790 con 0x7fb2180aef90 2024-03-18T20:25:07.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.736+0000 7fb21cccd700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb218072680 con 0x7fb21815d3b0 2024-03-18T20:25:07.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.736+0000 7fb21cccd700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb218160f20 con 0x7fb2180b2350 2024-03-18T20:25:07.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.736+0000 7fb216d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2180b2350 0x7fb2180b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:07.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.736+0000 7fb215d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb2180aef90 0x7fb2180b1c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53048/0 (socket says 172.21.15.1:53048) 2024-03-18T20:25:07.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.736+0000 7fb216d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2180b2350 0x7fb2180b39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60630/0 (socket says 172.21.15.1:60630) 2024-03-18T20:25:07.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.736+0000 7fb215d9b700 1 -- 172.21.15.1:0/400862688 learned_addr learned my addr 172.21.15.1:0/400862688 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:07.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.736+0000 7fb216d9d700 1 -- 172.21.15.1:0/400862688 >> v1:172.21.15.165:6789/0 conn(0x7fb21815d3b0 legacy=0x7fb218161930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:07.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.736+0000 7fb216d9d700 1 -- 172.21.15.1:0/400862688 >> v1:172.21.15.1:6789/0 conn(0x7fb2180aef90 legacy=0x7fb2180b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:07.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.736+0000 7fb216d9d700 1 -- 172.21.15.1:0/400862688 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb218072960 con 0x7fb2180b2350 2024-03-18T20:25:07.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.736+0000 7fb21559a700 1 -- 172.21.15.1:0/400862688 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50130605 0 0) 0x7fb21815d790 con 0x7fb2180aef90 2024-03-18T20:25:07.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.737+0000 7fb216d9d700 1 --2- 172.21.15.1:0/400862688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2180b2350 0x7fb2180b39c0 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7fb210009d90 tx=0x7fb21000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f27e954e5f021f11 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:07.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.737+0000 7fb21559a700 1 -- 172.21.15.1:0/400862688 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21000e070 con 0x7fb2180b2350 2024-03-18T20:25:07.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.737+0000 7fb21559a700 1 -- 172.21.15.1:0/400862688 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21000bbd0 con 0x7fb2180b2350 2024-03-18T20:25:07.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.737+0000 7fb21559a700 1 -- 172.21.15.1:0/400862688 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb210012430 con 0x7fb2180b2350 2024-03-18T20:25:07.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.737+0000 7fb21cccd700 1 -- 172.21.15.1:0/400862688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2180b2350 msgr2=0x7fb2180b39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:07.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.737+0000 7fb21cccd700 1 --2- 172.21.15.1:0/400862688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2180b2350 0x7fb2180b39c0 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7fb210009d90 tx=0x7fb21000b330 comp rx=0 tx=0).stop 2024-03-18T20:25:07.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.738+0000 7fb21cccd700 1 -- 172.21.15.1:0/400862688 shutdown_connections 2024-03-18T20:25:07.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.738+0000 7fb21cccd700 1 --2- 172.21.15.1:0/400862688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2180b2350 0x7fb2180b39c0 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.738+0000 7fb21cccd700 1 -- 172.21.15.1:0/400862688 >> 172.21.15.1:0/400862688 conn(0x7fb21806d8d0 msgr2=0x7fb21806dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:07.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.738+0000 7fb21cccd700 1 -- 172.21.15.1:0/400862688 shutdown_connections 2024-03-18T20:25:07.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.738+0000 7fb21cccd700 1 -- 172.21.15.1:0/400862688 wait complete. 2024-03-18T20:25:07.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.739+0000 7fb21cccd700 1 Processor -- start 2024-03-18T20:25:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.739+0000 7fb21cccd700 1 -- start start 2024-03-18T20:25:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.740+0000 7fb21cccd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2180aef90 0x7fb218158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.740+0000 7fb21cccd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2180b2350 0x7fb21815eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.740+0000 7fb21cccd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb21815d3b0 0x7fb218155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.740+0000 7fb21cccd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21815f1e0 con 0x7fb2180aef90 2024-03-18T20:25:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.740+0000 7fb21cccd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21815f410 con 0x7fb21815d3b0 2024-03-18T20:25:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.740+0000 7fb21cccd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21815f6f0 con 0x7fb2180b2350 2024-03-18T20:25:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.740+0000 7fb21659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2180aef90 0x7fb218158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.740+0000 7fb21659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2180aef90 0x7fb218158e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60632/0 (socket says 172.21.15.1:60632) 2024-03-18T20:25:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.740+0000 7fb21659c700 1 -- 172.21.15.1:0/1402398460 learned_addr learned my addr 172.21.15.1:0/1402398460 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.740+0000 7fb216d9d700 1 --2- 172.21.15.1:0/1402398460 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb21815d3b0 0x7fb218155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:07.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.740+0000 7fb215d9b700 1 --2- 172.21.15.1:0/1402398460 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2180b2350 0x7fb21815eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:07.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.740+0000 7fb21659c700 1 -- 172.21.15.1:0/1402398460 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2180b2350 msgr2=0x7fb21815eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:07.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.740+0000 7fb21659c700 1 --2- 172.21.15.1:0/1402398460 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2180b2350 0x7fb21815eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.740+0000 7fb21659c700 1 -- 172.21.15.1:0/1402398460 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb21815d3b0 msgr2=0x7fb218155f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:07.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.740+0000 7fb21659c700 1 --2- 172.21.15.1:0/1402398460 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb21815d3b0 0x7fb218155f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:07.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.740+0000 7fb21659c700 1 -- 172.21.15.1:0/1402398460 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb210009a70 con 0x7fb2180aef90 2024-03-18T20:25:07.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.741+0000 7fb21659c700 1 --2- 172.21.15.1:0/1402398460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2180aef90 0x7fb218158e50 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7fb20800c450 tx=0x7fb20800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:07.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.741+0000 7fb206ffd700 1 -- 172.21.15.1:0/1402398460 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2080052d0 con 0x7fb2180aef90 2024-03-18T20:25:07.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.741+0000 7fb206ffd700 1 -- 172.21.15.1:0/1402398460 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb208005430 con 0x7fb2180aef90 2024-03-18T20:25:07.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.741+0000 7fb21cccd700 1 -- 172.21.15.1:0/1402398460 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb21815f9d0 con 0x7fb2180aef90 2024-03-18T20:25:07.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.741+0000 7fb206ffd700 1 -- 172.21.15.1:0/1402398460 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2080038c0 con 0x7fb2180aef90 2024-03-18T20:25:07.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.741+0000 7fb21cccd700 1 -- 172.21.15.1:0/1402398460 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21815ff90 con 0x7fb2180aef90 2024-03-18T20:25:07.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.742+0000 7fb21cccd700 1 -- 172.21.15.1:0/1402398460 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2180722d0 con 0x7fb2180aef90 2024-03-18T20:25:07.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.742+0000 7fb206ffd700 1 -- 172.21.15.1:0/1402398460 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 131) v1 ==== 89353+0+0 (secure 0 0 0) 0x7fb208003a20 con 0x7fb2180aef90 2024-03-18T20:25:07.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.743+0000 7fb206ffd700 1 --2- 172.21.15.1:0/1402398460 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb1f806aed0 0x7fb1f806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:07.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.743+0000 7fb215d9b700 1 -- 172.21.15.1:0/1402398460 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb1f806aed0 msgr2=0x7fb1f806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:07.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.743+0000 7fb206ffd700 1 -- 172.21.15.1:0/1402398460 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6632+0+0 (secure 0 0 0) 0x7fb208089040 con 0x7fb2180aef90 2024-03-18T20:25:07.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.743+0000 7fb215d9b700 1 --2- 172.21.15.1:0/1402398460 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb1f806aed0 0x7fb1f806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:07.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.747+0000 7fb206ffd700 1 -- 172.21.15.1:0/1402398460 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb208053d60 con 0x7fb2180aef90 2024-03-18T20:25:07.800 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:07.795+0000 7f25953da240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:25:07.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.944+0000 7fb215d9b700 1 -- 172.21.15.1:0/1402398460 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb1f806aed0 msgr2=0x7fb1f806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:07.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.944+0000 7fb215d9b700 1 --2- 172.21.15.1:0/1402398460 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb1f806aed0 0x7fb1f806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:07.976 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:07.971+0000 7f25953da240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:25:08.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.998+0000 7fb21cccd700 1 -- 172.21.15.1:0/1402398460 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb218073350 con 0x7fb2180aef90 2024-03-18T20:25:08.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:07.999+0000 7fb206ffd700 1 -- 172.21.15.1:0/1402398460 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4739 (secure 0 0 0) 0x7fb208017090 con 0x7fb2180aef90 2024-03-18T20:25:08.004 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:08.004 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":97,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2241819619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2241819619},{"type":"v1","addr":"172.21.15.165:6835","nonce":2241819619}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/320492236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":320492236},{"type":"v1","addr":"172.21.15.165:6834","nonce":320492236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":11031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/1035910531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1035910531},{"type":"v1","addr":"172.21.15.1:6836","nonce":1035910531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:23:39.747292+0000","modified":"2024-03-18T20:23:42.766952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11034":{"gid":11034,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2388480799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2388480799},{"type":"v1","addr":"172.21.15.1:6837","nonce":2388480799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-03-18T20:25:08.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.001+0000 7fb21cccd700 1 -- 172.21.15.1:0/1402398460 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb1f806aed0 msgr2=0x7fb1f806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:08.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.001+0000 7fb21cccd700 1 --2- 172.21.15.1:0/1402398460 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb1f806aed0 0x7fb1f806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.001+0000 7fb21cccd700 1 -- 172.21.15.1:0/1402398460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2180aef90 msgr2=0x7fb218158e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:08.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.001+0000 7fb21cccd700 1 --2- 172.21.15.1:0/1402398460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2180aef90 0x7fb218158e50 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7fb20800c450 tx=0x7fb20800c760 comp rx=0 tx=0).stop 2024-03-18T20:25:08.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.001+0000 7fb21cccd700 1 -- 172.21.15.1:0/1402398460 shutdown_connections 2024-03-18T20:25:08.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.001+0000 7fb21cccd700 1 --2- 172.21.15.1:0/1402398460 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb1f806aed0 0x7fb1f806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.001+0000 7fb21cccd700 1 --2- 172.21.15.1:0/1402398460 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb21815d3b0 0x7fb218155f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.001+0000 7fb21cccd700 1 --2- 172.21.15.1:0/1402398460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2180aef90 0x7fb218158e50 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.001+0000 7fb21cccd700 1 --2- 172.21.15.1:0/1402398460 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2180b2350 0x7fb21815eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.001+0000 7fb21cccd700 1 -- 172.21.15.1:0/1402398460 >> 172.21.15.1:0/1402398460 conn(0x7fb21806d8d0 msgr2=0x7fb21806eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:08.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.001+0000 7fb21cccd700 1 -- 172.21.15.1:0/1402398460 shutdown_connections 2024-03-18T20:25:08.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.001+0000 7fb21cccd700 1 -- 172.21.15.1:0/1402398460 wait complete. 2024-03-18T20:25:08.008 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 97 2024-03-18T20:25:08.010 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:08.007+0000 7f7b493a2240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:25:08.019 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:25:08.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.142+0000 7f4c72f10700 1 Processor -- start 2024-03-18T20:25:08.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.142+0000 7f4c72f10700 1 -- start start 2024-03-18T20:25:08.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c72f10700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c6c15b4d0 0x7f4c6c09a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:08.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c72f10700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4c6c0cf750 0x7f4c6c099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:08.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c72f10700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c6c099f80 0x7f4c6c0cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:08.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c72f10700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c6c09a410 con 0x7f4c6c15b4d0 2024-03-18T20:25:08.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c72f10700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c6c06f070 con 0x7f4c6c0cf750 2024-03-18T20:25:08.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c72f10700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6c0d0260 con 0x7f4c6c099f80 2024-03-18T20:25:08.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c70cac700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c6c15b4d0 0x7f4c6c09a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:08.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c70cac700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c6c15b4d0 0x7f4c6c09a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60642/0 (socket says 172.21.15.1:60642) 2024-03-18T20:25:08.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c70cac700 1 -- 172.21.15.1:0/543348415 learned_addr learned my addr 172.21.15.1:0/543348415 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:08.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c714ad700 1 --2- 172.21.15.1:0/543348415 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c6c099f80 0x7f4c6c0cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:08.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c6bfff700 1 --2- 172.21.15.1:0/543348415 >> v2:172.21.15.165:3300/0 conn(0x7f4c6c0cf750 0x7f4c6c099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:08.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c70cac700 1 -- 172.21.15.1:0/543348415 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c6c099f80 msgr2=0x7f4c6c0cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:08.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c70cac700 1 --2- 172.21.15.1:0/543348415 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c6c099f80 0x7f4c6c0cbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c70cac700 1 -- 172.21.15.1:0/543348415 >> v2:172.21.15.165:3300/0 conn(0x7f4c6c0cf750 msgr2=0x7f4c6c099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c70cac700 1 --2- 172.21.15.1:0/543348415 >> v2:172.21.15.165:3300/0 conn(0x7f4c6c0cf750 0x7f4c6c099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c70cac700 1 -- 172.21.15.1:0/543348415 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c6c0cff80 con 0x7f4c6c15b4d0 2024-03-18T20:25:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c714ad700 1 --2- 172.21.15.1:0/543348415 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c6c099f80 0x7f4c6c0cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c70cac700 1 --2- 172.21.15.1:0/543348415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c6c15b4d0 0x7f4c6c09a650 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c071d40 tx=0x7f4c60009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d4e875cc7f82802 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c6b7fe700 1 -- 172.21.15.1:0/543348415 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6000b870 con 0x7f4c6c15b4d0 2024-03-18T20:25:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c6b7fe700 1 -- 172.21.15.1:0/543348415 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6000b9d0 con 0x7f4c6c15b4d0 2024-03-18T20:25:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.143+0000 7f4c6b7fe700 1 -- 172.21.15.1:0/543348415 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6000bd60 con 0x7f4c6c15b4d0 2024-03-18T20:25:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.144+0000 7f4c72f10700 1 -- 172.21.15.1:0/543348415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c6c15b4d0 msgr2=0x7f4c6c09a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.144+0000 7f4c72f10700 1 --2- 172.21.15.1:0/543348415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c6c15b4d0 0x7f4c6c09a650 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c071d40 tx=0x7f4c60009bd0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.144+0000 7f4c72f10700 1 -- 172.21.15.1:0/543348415 shutdown_connections 2024-03-18T20:25:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.144+0000 7f4c72f10700 1 --2- 172.21.15.1:0/543348415 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c6c099f80 0x7f4c6c0cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.144+0000 7f4c72f10700 1 --2- 172.21.15.1:0/543348415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c6c15b4d0 0x7f4c6c09a650 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.144+0000 7f4c72f10700 1 --2- 172.21.15.1:0/543348415 >> v2:172.21.15.165:3300/0 conn(0x7f4c6c0cf750 0x7f4c6c099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.144+0000 7f4c72f10700 1 -- 172.21.15.1:0/543348415 >> 172.21.15.1:0/543348415 conn(0x7f4c6c0b2cb0 msgr2=0x7f4c6c0b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:08.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.144+0000 7f4c72f10700 1 -- 172.21.15.1:0/543348415 shutdown_connections 2024-03-18T20:25:08.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.144+0000 7f4c72f10700 1 -- 172.21.15.1:0/543348415 wait complete. 2024-03-18T20:25:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.146+0000 7f4c72f10700 1 Processor -- start 2024-03-18T20:25:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.146+0000 7f4c72f10700 1 -- start start 2024-03-18T20:25:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.147+0000 7f4c72f10700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c6c099f80 0x7f4c6c14c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.147+0000 7f4c72f10700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c6c0cf750 0x7f4c6c15bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.147+0000 7f4c72f10700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c6c15b4d0 0x7f4c6c0c9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.147+0000 7f4c72f10700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6c0c5d30 con 0x7f4c6c099f80 2024-03-18T20:25:08.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.147+0000 7f4c72f10700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6c14cbc0 con 0x7f4c6c0cf750 2024-03-18T20:25:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.147+0000 7f4c72f10700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6c0c5120 con 0x7f4c6c15b4d0 2024-03-18T20:25:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.147+0000 7f4c70cac700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c6c099f80 0x7f4c6c14c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.147+0000 7f4c70cac700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c6c099f80 0x7f4c6c14c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60656/0 (socket says 172.21.15.1:60656) 2024-03-18T20:25:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.147+0000 7f4c70cac700 1 -- 172.21.15.1:0/2032155996 learned_addr learned my addr 172.21.15.1:0/2032155996 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.147+0000 7f4c714ad700 1 --2- 172.21.15.1:0/2032155996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c6c15b4d0 0x7f4c6c0c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.147+0000 7f4c6bfff700 1 --2- 172.21.15.1:0/2032155996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c6c0cf750 0x7f4c6c15bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.147+0000 7f4c70cac700 1 -- 172.21.15.1:0/2032155996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c6c15b4d0 msgr2=0x7f4c6c0c9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.147+0000 7f4c70cac700 1 --2- 172.21.15.1:0/2032155996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c6c15b4d0 0x7f4c6c0c9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.147+0000 7f4c70cac700 1 -- 172.21.15.1:0/2032155996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c6c0cf750 msgr2=0x7f4c6c15bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.147+0000 7f4c70cac700 1 --2- 172.21.15.1:0/2032155996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c6c0cf750 0x7f4c6c15bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.147+0000 7f4c70cac700 1 -- 172.21.15.1:0/2032155996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c60009550 con 0x7f4c6c099f80 2024-03-18T20:25:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.147+0000 7f4c714ad700 1 --2- 172.21.15.1:0/2032155996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c6c15b4d0 0x7f4c6c0c9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.147+0000 7f4c6bfff700 1 --2- 172.21.15.1:0/2032155996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c6c0cf750 0x7f4c6c15bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:08.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.147+0000 7f4c70cac700 1 --2- 172.21.15.1:0/2032155996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c6c099f80 0x7f4c6c14c680 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f4c60014040 tx=0x7f4c60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.148+0000 7f4c697fa700 1 -- 172.21.15.1:0/2032155996 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60004520 con 0x7f4c6c099f80 2024-03-18T20:25:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.148+0000 7f4c697fa700 1 -- 172.21.15.1:0/2032155996 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6001e070 con 0x7f4c6c099f80 2024-03-18T20:25:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.148+0000 7f4c72f10700 1 -- 172.21.15.1:0/2032155996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c6c0c5400 con 0x7f4c6c099f80 2024-03-18T20:25:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.148+0000 7f4c697fa700 1 -- 172.21.15.1:0/2032155996 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c600038c0 con 0x7f4c6c099f80 2024-03-18T20:25:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.148+0000 7f4c72f10700 1 -- 172.21.15.1:0/2032155996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c6c0c59c0 con 0x7f4c6c099f80 2024-03-18T20:25:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.149+0000 7f4c72f10700 1 -- 172.21.15.1:0/2032155996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c6c068240 con 0x7f4c6c099f80 2024-03-18T20:25:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.149+0000 7f4c697fa700 1 -- 172.21.15.1:0/2032155996 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 131) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f4c60003a20 con 0x7f4c6c099f80 2024-03-18T20:25:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.149+0000 7f4c697fa700 1 --2- 172.21.15.1:0/2032155996 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c5006aed0 0x7f4c5006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.150+0000 7f4c6bfff700 1 -- 172.21.15.1:0/2032155996 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c5006aed0 msgr2=0x7f4c5006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.150+0000 7f4c697fa700 1 -- 172.21.15.1:0/2032155996 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f4c60020070 con 0x7f4c6c099f80 2024-03-18T20:25:08.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.150+0000 7f4c6bfff700 1 --2- 172.21.15.1:0/2032155996 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c5006aed0 0x7f4c5006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:08.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.154+0000 7f4c697fa700 1 -- 172.21.15.1:0/2032155996 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4c60055ac0 con 0x7f4c6c099f80 2024-03-18T20:25:08.177 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:08.173+0000 7f7b493a2240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:25:08.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.351+0000 7f4c6bfff700 1 -- 172.21.15.1:0/2032155996 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c5006aed0 msgr2=0x7f4c5006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:08.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.351+0000 7f4c6bfff700 1 --2- 172.21.15.1:0/2032155996 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c5006aed0 0x7f4c5006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:08.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.358+0000 7f4c72f10700 1 -- 172.21.15.1:0/2032155996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c6c0676a0 con 0x7f4c6c099f80 2024-03-18T20:25:08.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.360+0000 7f4c697fa700 1 -- 172.21.15.1:0/2032155996 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v219) v1 ==== 74+0+17125 (secure 0 0 0) 0x7f4c6002a090 con 0x7f4c6c099f80 2024-03-18T20:25:08.365 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:08.366 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":219,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:25:06.176019+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:23:37.591406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-03-18T20:23:38.723169+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":219,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.c","mappings":[{"from":1,"to":0}]},{"pgid":"26.f","mappings":[{"from":1,"to":0}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2462726896":"2024-03-19T20:23:54.844816+0000","172.21.15.1:6833/148356":"2024-03-19T20:23:54.844816+0000","172.21.15.1:6832/148356":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/4177404804":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/1122038130":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/2416229963":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/2662947140":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/2110350357":"2024-03-19T20:23:54.844816+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":26,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:08.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.362+0000 7f4c72f10700 1 -- 172.21.15.1:0/2032155996 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c5006aed0 msgr2=0x7f4c5006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:08.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.362+0000 7f4c72f10700 1 --2- 172.21.15.1:0/2032155996 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c5006aed0 0x7f4c5006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.362+0000 7f4c72f10700 1 -- 172.21.15.1:0/2032155996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c6c099f80 msgr2=0x7f4c6c14c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:08.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.362+0000 7f4c72f10700 1 --2- 172.21.15.1:0/2032155996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c6c099f80 0x7f4c6c14c680 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f4c60014040 tx=0x7f4c60004750 comp rx=0 tx=0).stop 2024-03-18T20:25:08.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.362+0000 7f4c72f10700 1 -- 172.21.15.1:0/2032155996 shutdown_connections 2024-03-18T20:25:08.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.362+0000 7f4c72f10700 1 --2- 172.21.15.1:0/2032155996 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c5006aed0 0x7f4c5006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.362+0000 7f4c72f10700 1 --2- 172.21.15.1:0/2032155996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c6c15b4d0 0x7f4c6c0c9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.362+0000 7f4c72f10700 1 --2- 172.21.15.1:0/2032155996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c6c099f80 0x7f4c6c14c680 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.362+0000 7f4c72f10700 1 --2- 172.21.15.1:0/2032155996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c6c0cf750 0x7f4c6c15bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.362+0000 7f4c72f10700 1 -- 172.21.15.1:0/2032155996 >> 172.21.15.1:0/2032155996 conn(0x7f4c6c0b2cb0 msgr2=0x7f4c6c0cc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:08.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.362+0000 7f4c72f10700 1 -- 172.21.15.1:0/2032155996 shutdown_connections 2024-03-18T20:25:08.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.362+0000 7f4c72f10700 1 -- 172.21.15.1:0/2032155996 wait complete. 2024-03-18T20:25:08.380 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T20:25:08.380 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:25:08.403 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:08.398+0000 7f25953da240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:25:08.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.501+0000 7f0700338700 1 Processor -- start 2024-03-18T20:25:08.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.501+0000 7f0700338700 1 -- start start 2024-03-18T20:25:08.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.502+0000 7f0700338700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f06f8099a40 0x7f06f806ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:08.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.502+0000 7f0700338700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f06f80d05a0 0x7f06f8099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:08.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.502+0000 7f0700338700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f80cf7e0 con 0x7f06f809a360 2024-03-18T20:25:08.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.502+0000 7f0700338700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f80cfac0 con 0x7f06f8099a40 2024-03-18T20:25:08.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.502+0000 7f0700338700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f80cfda0 con 0x7f06f80d05a0 2024-03-18T20:25:08.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.502+0000 7f06fe0d4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f06f8099a40 0x7f06f806ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:08.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.502+0000 7f06fe8d5700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f06f809a360 0x7f06f806e880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53052/0 (socket says 172.21.15.1:53052) 2024-03-18T20:25:08.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.502+0000 7f06fe8d5700 1 -- 172.21.15.1:0/1995151778 learned_addr learned my addr 172.21.15.1:0/1995151778 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:08.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.502+0000 7f06fd8d3700 1 --2- 172.21.15.1:0/1995151778 >> v2:172.21.15.165:3300/0 conn(0x7f06f80d05a0 0x7f06f8099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:08.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.502+0000 7f06fe0d4700 1 -- 172.21.15.1:0/1995151778 >> v2:172.21.15.165:3300/0 conn(0x7f06f80d05a0 msgr2=0x7f06f8099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:08.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.502+0000 7f06fe0d4700 1 --2- 172.21.15.1:0/1995151778 >> v2:172.21.15.165:3300/0 conn(0x7f06f80d05a0 0x7f06f8099e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.502+0000 7f06fe0d4700 1 -- 172.21.15.1:0/1995151778 >> v1:172.21.15.1:6789/0 conn(0x7f06f809a360 legacy=0x7f06f806e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:08.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.502+0000 7f06fe0d4700 1 -- 172.21.15.1:0/1995151778 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f80d0080 con 0x7f06f8099a40 2024-03-18T20:25:08.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.502+0000 7f06fd0d2700 1 -- 172.21.15.1:0/1995151778 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749375816 0 0) 0x7f06f80cf7e0 con 0x7f06f809a360 2024-03-18T20:25:08.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.502+0000 7f06fe0d4700 1 --2- 172.21.15.1:0/1995151778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06f8099a40 0x7f06f806ba70 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f06f8071d40 tx=0x7f06f4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73adece37081a9d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:08.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.503+0000 7f06fd0d2700 1 -- 172.21.15.1:0/1995151778 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f4011830 con 0x7f06f8099a40 2024-03-18T20:25:08.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.503+0000 7f06fd0d2700 1 -- 172.21.15.1:0/1995151778 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f4011990 con 0x7f06f8099a40 2024-03-18T20:25:08.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.503+0000 7f06fd0d2700 1 -- 172.21.15.1:0/1995151778 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f4011d20 con 0x7f06f8099a40 2024-03-18T20:25:08.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.503+0000 7f0700338700 1 -- 172.21.15.1:0/1995151778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06f8099a40 msgr2=0x7f06f806ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:08.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.503+0000 7f0700338700 1 --2- 172.21.15.1:0/1995151778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06f8099a40 0x7f06f806ba70 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f06f8071d40 tx=0x7f06f4009b20 comp rx=0 tx=0).stop 2024-03-18T20:25:08.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.503+0000 7f0700338700 1 -- 172.21.15.1:0/1995151778 shutdown_connections 2024-03-18T20:25:08.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.503+0000 7f0700338700 1 --2- 172.21.15.1:0/1995151778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06f8099a40 0x7f06f806ba70 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.503+0000 7f0700338700 1 --2- 172.21.15.1:0/1995151778 >> v2:172.21.15.165:3300/0 conn(0x7f06f80d05a0 0x7f06f8099e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.503+0000 7f0700338700 1 -- 172.21.15.1:0/1995151778 >> 172.21.15.1:0/1995151778 conn(0x7f06f80b2cb0 msgr2=0x7f06f80b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:08.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.503+0000 7f0700338700 1 -- 172.21.15.1:0/1995151778 shutdown_connections 2024-03-18T20:25:08.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.503+0000 7f0700338700 1 -- 172.21.15.1:0/1995151778 wait complete. 2024-03-18T20:25:08.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.505+0000 7f0700338700 1 Processor -- start 2024-03-18T20:25:08.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.505+0000 7f0700338700 1 -- start start 2024-03-18T20:25:08.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.505+0000 7f0700338700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06f8099a40 0x7f06f80c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:08.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.505+0000 7f0700338700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06f809a360 0x7f06f80c6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:08.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.505+0000 7f0700338700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f06f80d05a0 0x7f06f80c6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:08.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.505+0000 7f0700338700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80c52d0 con 0x7f06f809a360 2024-03-18T20:25:08.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.505+0000 7f0700338700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80c55b0 con 0x7f06f80d05a0 2024-03-18T20:25:08.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.505+0000 7f0700338700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80c5890 con 0x7f06f8099a40 2024-03-18T20:25:08.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.505+0000 7f06fd8d3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06f809a360 0x7f06f80c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:08.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.505+0000 7f06fd8d3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06f809a360 0x7f06f80c6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60682/0 (socket says 172.21.15.1:60682) 2024-03-18T20:25:08.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.505+0000 7f06fd8d3700 1 -- 172.21.15.1:0/3815793776 learned_addr learned my addr 172.21.15.1:0/3815793776 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:08.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.506+0000 7f06fe0d4700 1 --2- 172.21.15.1:0/3815793776 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06f8099a40 0x7f06f80c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:08.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.506+0000 7f06fe8d5700 1 --2- 172.21.15.1:0/3815793776 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f06f80d05a0 0x7f06f80c6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:08.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.506+0000 7f06fd8d3700 1 -- 172.21.15.1:0/3815793776 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06f8099a40 msgr2=0x7f06f80c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:08.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.506+0000 7f06fd8d3700 1 --2- 172.21.15.1:0/3815793776 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06f8099a40 0x7f06f80c4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.506+0000 7f06fd8d3700 1 -- 172.21.15.1:0/3815793776 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f06f80d05a0 msgr2=0x7f06f80c6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:08.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.506+0000 7f06fd8d3700 1 --2- 172.21.15.1:0/3815793776 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f06f80d05a0 0x7f06f80c6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.506+0000 7f06fd8d3700 1 -- 172.21.15.1:0/3815793776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f40094a0 con 0x7f06f809a360 2024-03-18T20:25:08.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.506+0000 7f06fe8d5700 1 --2- 172.21.15.1:0/3815793776 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f06f80d05a0 0x7f06f80c6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:08.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.506+0000 7f06fe0d4700 1 --2- 172.21.15.1:0/3815793776 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06f8099a40 0x7f06f80c4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:08.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.506+0000 7f06fd8d3700 1 --2- 172.21.15.1:0/3815793776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06f809a360 0x7f06f80c6730 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f06e800e840 tx=0x7f06e800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:08.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.506+0000 7f06eeffd700 1 -- 172.21.15.1:0/3815793776 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e8009590 con 0x7f06f809a360 2024-03-18T20:25:08.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.506+0000 7f06eeffd700 1 -- 172.21.15.1:0/3815793776 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e8013680 con 0x7f06f809a360 2024-03-18T20:25:08.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.506+0000 7f0700338700 1 -- 172.21.15.1:0/3815793776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f80c5b70 con 0x7f06f809a360 2024-03-18T20:25:08.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.506+0000 7f06eeffd700 1 -- 172.21.15.1:0/3815793776 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e8013a30 con 0x7f06f809a360 2024-03-18T20:25:08.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.506+0000 7f0700338700 1 -- 172.21.15.1:0/3815793776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f80c9fb0 con 0x7f06f809a360 2024-03-18T20:25:08.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.507+0000 7f0700338700 1 -- 172.21.15.1:0/3815793776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f8068240 con 0x7f06f809a360 2024-03-18T20:25:08.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.508+0000 7f06eeffd700 1 -- 172.21.15.1:0/3815793776 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 131) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f06e8013b90 con 0x7f06f809a360 2024-03-18T20:25:08.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.508+0000 7f06eeffd700 1 --2- 172.21.15.1:0/3815793776 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f06dc06aed0 0x7f06dc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:08.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.508+0000 7f06fe0d4700 1 -- 172.21.15.1:0/3815793776 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f06dc06aed0 msgr2=0x7f06dc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:08.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.508+0000 7f06eeffd700 1 -- 172.21.15.1:0/3815793776 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f06e8089a10 con 0x7f06f809a360 2024-03-18T20:25:08.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.508+0000 7f06fe0d4700 1 --2- 172.21.15.1:0/3815793776 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f06dc06aed0 0x7f06dc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:08.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.513+0000 7f06eeffd700 1 -- 172.21.15.1:0/3815793776 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f06e8054580 con 0x7f06f809a360 2024-03-18T20:25:08.539 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:08.536+0000 7f7b493a2240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:25:08.587 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:08.582+0000 7f25953da240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:25:08.628 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:08.625+0000 7f7b493a2240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:25:08.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.709+0000 7f06fe0d4700 1 -- 172.21.15.1:0/3815793776 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f06dc06aed0 msgr2=0x7f06dc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:08.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.709+0000 7f06fe0d4700 1 --2- 172.21.15.1:0/3815793776 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f06dc06aed0 0x7f06dc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:08.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.765+0000 7f0700338700 1 -- 172.21.15.1:0/3815793776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06f80cda40 con 0x7f06f809a360 2024-03-18T20:25:08.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.766+0000 7f06eeffd700 1 -- 172.21.15.1:0/3815793776 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f06e8057ba0 con 0x7f06f809a360 2024-03-18T20:25:08.771 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:08.771 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":97,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2241819619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2241819619},{"type":"v1","addr":"172.21.15.165:6835","nonce":2241819619}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/320492236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":320492236},{"type":"v1","addr":"172.21.15.165:6834","nonce":320492236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":11031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/1035910531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1035910531},{"type":"v1","addr":"172.21.15.1:6836","nonce":1035910531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:23:39.747292+0000","modified":"2024-03-18T20:23:42.766952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11034":{"gid":11034,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2388480799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2388480799},{"type":"v1","addr":"172.21.15.1:6837","nonce":2388480799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-03-18T20:25:08.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.768+0000 7f0700338700 1 -- 172.21.15.1:0/3815793776 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f06dc06aed0 msgr2=0x7f06dc06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:08.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.768+0000 7f0700338700 1 --2- 172.21.15.1:0/3815793776 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f06dc06aed0 0x7f06dc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.768+0000 7f0700338700 1 -- 172.21.15.1:0/3815793776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06f809a360 msgr2=0x7f06f80c6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:08.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.768+0000 7f0700338700 1 --2- 172.21.15.1:0/3815793776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06f809a360 0x7f06f80c6730 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f06e800e840 tx=0x7f06e800ec00 comp rx=0 tx=0).stop 2024-03-18T20:25:08.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.768+0000 7f0700338700 1 -- 172.21.15.1:0/3815793776 shutdown_connections 2024-03-18T20:25:08.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.768+0000 7f0700338700 1 --2- 172.21.15.1:0/3815793776 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f06dc06aed0 0x7f06dc06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.768+0000 7f0700338700 1 --2- 172.21.15.1:0/3815793776 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06f8099a40 0x7f06f80c4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.768+0000 7f0700338700 1 --2- 172.21.15.1:0/3815793776 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f06f80d05a0 0x7f06f80c6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.768+0000 7f0700338700 1 --2- 172.21.15.1:0/3815793776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06f809a360 0x7f06f80c6730 unknown :-1 s=CLOSED pgs=2436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.768+0000 7f0700338700 1 -- 172.21.15.1:0/3815793776 >> 172.21.15.1:0/3815793776 conn(0x7f06f80b2cb0 msgr2=0x7f06f80b01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:08.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.768+0000 7f0700338700 1 -- 172.21.15.1:0/3815793776 shutdown_connections 2024-03-18T20:25:08.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.768+0000 7f0700338700 1 -- 172.21.15.1:0/3815793776 wait complete. 2024-03-18T20:25:08.775 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 97 2024-03-18T20:25:08.787 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:25:08.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.952+0000 7f0cec07c700 1 Processor -- start 2024-03-18T20:25:08.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.952+0000 7f0cec07c700 1 -- start start 2024-03-18T20:25:08.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.953+0000 7f0cec07c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ce40cf960 0x7f0ce40cf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:08.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.953+0000 7f0cec07c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0ce40cdda0 0x7f0ce40ce220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:08.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.953+0000 7f0cec07c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce406d640 con 0x7f0ce406a750 2024-03-18T20:25:08.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.953+0000 7f0cec07c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce406de50 con 0x7f0ce40cdda0 2024-03-18T20:25:08.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.953+0000 7f0cec07c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce40acd80 con 0x7f0ce40cf960 2024-03-18T20:25:08.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.953+0000 7f0ce9e18700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0ce406a750 0x7f0ce406ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53068/0 (socket says 172.21.15.1:53068) 2024-03-18T20:25:08.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.953+0000 7f0ce9e18700 1 -- 172.21.15.1:0/4097564878 learned_addr learned my addr 172.21.15.1:0/4097564878 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:08.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.953+0000 7f0ce9617700 1 --2- 172.21.15.1:0/4097564878 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ce40cf960 0x7f0ce40cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:08.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.953+0000 7f0cea619700 1 --2- 172.21.15.1:0/4097564878 >> v2:172.21.15.165:3300/0 conn(0x7f0ce40cdda0 0x7f0ce40ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:08.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.953+0000 7f0ce8e16700 1 -- 172.21.15.1:0/4097564878 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891892069 0 0) 0x7f0ce406d640 con 0x7f0ce406a750 2024-03-18T20:25:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.953+0000 7f0ce8e16700 1 -- 172.21.15.1:0/4097564878 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cd00035b0 con 0x7f0ce406a750 2024-03-18T20:25:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.954+0000 7f0ce9617700 1 -- 172.21.15.1:0/4097564878 >> v2:172.21.15.165:3300/0 conn(0x7f0ce40cdda0 msgr2=0x7f0ce40ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.954+0000 7f0ce9617700 1 --2- 172.21.15.1:0/4097564878 >> v2:172.21.15.165:3300/0 conn(0x7f0ce40cdda0 0x7f0ce40ce220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.954+0000 7f0ce9617700 1 -- 172.21.15.1:0/4097564878 >> v1:172.21.15.1:6789/0 conn(0x7f0ce406a750 legacy=0x7f0ce406ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.954+0000 7f0ce9617700 1 -- 172.21.15.1:0/4097564878 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce406d890 con 0x7f0ce40cf960 2024-03-18T20:25:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.954+0000 7f0cea619700 1 --2- 172.21.15.1:0/4097564878 >> v2:172.21.15.165:3300/0 conn(0x7f0ce40cdda0 0x7f0ce40ce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.954+0000 7f0ce9617700 1 --2- 172.21.15.1:0/4097564878 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ce40cf960 0x7f0ce40cf280 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4009840 tx=0x7f0cd4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5763d54e5904346e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.954+0000 7f0ce8e16700 1 -- 172.21.15.1:0/4097564878 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4011830 con 0x7f0ce40cf960 2024-03-18T20:25:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.954+0000 7f0ce8e16700 1 -- 172.21.15.1:0/4097564878 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd4011990 con 0x7f0ce40cf960 2024-03-18T20:25:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.954+0000 7f0ce8e16700 1 -- 172.21.15.1:0/4097564878 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4011d00 con 0x7f0ce40cf960 2024-03-18T20:25:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.955+0000 7f0cec07c700 1 -- 172.21.15.1:0/4097564878 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ce40cf960 msgr2=0x7f0ce40cf280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.955+0000 7f0cec07c700 1 --2- 172.21.15.1:0/4097564878 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ce40cf960 0x7f0ce40cf280 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4009840 tx=0x7f0cd4009b50 comp rx=0 tx=0).stop 2024-03-18T20:25:08.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.955+0000 7f0cec07c700 1 -- 172.21.15.1:0/4097564878 shutdown_connections 2024-03-18T20:25:08.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.955+0000 7f0cec07c700 1 --2- 172.21.15.1:0/4097564878 >> v2:172.21.15.165:3300/0 conn(0x7f0ce40cdda0 0x7f0ce40ce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.955+0000 7f0cec07c700 1 --2- 172.21.15.1:0/4097564878 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ce40cf960 0x7f0ce40cf280 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.955+0000 7f0cec07c700 1 -- 172.21.15.1:0/4097564878 >> 172.21.15.1:0/4097564878 conn(0x7f0ce406b860 msgr2=0x7f0ce406bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:08.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.955+0000 7f0cec07c700 1 -- 172.21.15.1:0/4097564878 shutdown_connections 2024-03-18T20:25:08.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.955+0000 7f0cec07c700 1 -- 172.21.15.1:0/4097564878 wait complete. 2024-03-18T20:25:08.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.956+0000 7f0cec07c700 1 Processor -- start 2024-03-18T20:25:08.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.956+0000 7f0cec07c700 1 -- start start 2024-03-18T20:25:08.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.957+0000 7f0cec07c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ce406a750 0x7f0ce41432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:08.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.957+0000 7f0cec07c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ce40cdda0 0x7f0ce40adfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:08.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.957+0000 7f0cec07c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ce40cf960 0x7f0ce40ae4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:08.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.957+0000 7f0cec07c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce41437e0 con 0x7f0ce40cf960 2024-03-18T20:25:08.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.957+0000 7f0cec07c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce41f6b20 con 0x7f0ce406a750 2024-03-18T20:25:08.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.957+0000 7f0cec07c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce41f6e00 con 0x7f0ce40cdda0 2024-03-18T20:25:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.957+0000 7f0cea619700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ce40cf960 0x7f0ce40ae4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.957+0000 7f0cea619700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ce40cf960 0x7f0ce40ae4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60694/0 (socket says 172.21.15.1:60694) 2024-03-18T20:25:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.957+0000 7f0cea619700 1 -- 172.21.15.1:0/3316967762 learned_addr learned my addr 172.21.15.1:0/3316967762 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.957+0000 7f0ce9e18700 1 --2- 172.21.15.1:0/3316967762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ce406a750 0x7f0ce41432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.957+0000 7f0ce9617700 1 --2- 172.21.15.1:0/3316967762 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ce40cdda0 0x7f0ce40adfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.958+0000 7f0cea619700 1 -- 172.21.15.1:0/3316967762 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ce40cdda0 msgr2=0x7f0ce40adfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.958+0000 7f0cea619700 1 --2- 172.21.15.1:0/3316967762 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ce40cdda0 0x7f0ce40adfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.958+0000 7f0cea619700 1 -- 172.21.15.1:0/3316967762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ce406a750 msgr2=0x7f0ce41432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.958+0000 7f0cea619700 1 --2- 172.21.15.1:0/3316967762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ce406a750 0x7f0ce41432a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.958+0000 7f0cea619700 1 -- 172.21.15.1:0/3316967762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc009a70 con 0x7f0ce40cf960 2024-03-18T20:25:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.958+0000 7f0cea619700 1 --2- 172.21.15.1:0/3316967762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ce40cf960 0x7f0ce40ae4e0 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc00efc0 tx=0x7f0cdc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:08.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.958+0000 7f0cda7fc700 1 -- 172.21.15.1:0/3316967762 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc00ca90 con 0x7f0ce40cf960 2024-03-18T20:25:08.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.958+0000 7f0cda7fc700 1 -- 172.21.15.1:0/3316967762 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cdc00f040 con 0x7f0ce40cf960 2024-03-18T20:25:08.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.958+0000 7f0cec07c700 1 -- 172.21.15.1:0/3316967762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd40094a0 con 0x7f0ce40cf960 2024-03-18T20:25:08.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.958+0000 7f0cda7fc700 1 -- 172.21.15.1:0/3316967762 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc00ce10 con 0x7f0ce40cf960 2024-03-18T20:25:08.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.958+0000 7f0cec07c700 1 -- 172.21.15.1:0/3316967762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce41f73c0 con 0x7f0ce40cf960 2024-03-18T20:25:08.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.959+0000 7f0cec07c700 1 -- 172.21.15.1:0/3316967762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce4072490 con 0x7f0ce40cf960 2024-03-18T20:25:08.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.959+0000 7f0cda7fc700 1 -- 172.21.15.1:0/3316967762 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 131) v1 ==== 89353+0+0 (secure 0 0 0) 0x7f0cdc014480 con 0x7f0ce40cf960 2024-03-18T20:25:08.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.960+0000 7f0cda7fc700 1 --2- 172.21.15.1:0/3316967762 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0cc806aed0 0x7f0cc806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:08.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.960+0000 7f0ce9e18700 1 -- 172.21.15.1:0/3316967762 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0cc806aed0 msgr2=0x7f0cc806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:08.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.960+0000 7f0cda7fc700 1 -- 172.21.15.1:0/3316967762 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f0cdc08a2b0 con 0x7f0ce40cf960 2024-03-18T20:25:08.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.960+0000 7f0ce9e18700 1 --2- 172.21.15.1:0/3316967762 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0cc806aed0 0x7f0cc806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:08.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:08.969+0000 7f0cda7fc700 1 -- 172.21.15.1:0/3316967762 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0cdc054c00 con 0x7f0ce40cf960 2024-03-18T20:25:08.977 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:08.972+0000 7f25953da240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:25:09.074 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:09.069+0000 7f25953da240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:25:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.092+0000 7f0cda7fc700 1 -- 172.21.15.1:0/3316967762 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 132) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f0cdc054fd0 con 0x7f0ce40cf960 2024-03-18T20:25:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.092+0000 7f0cda7fc700 1 -- 172.21.15.1:0/3316967762 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0cc806aed0 msgr2=0x7f0cc806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:09.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.092+0000 7f0cda7fc700 1 --2- 172.21.15.1:0/3316967762 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0cc806aed0 0x7f0cc806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.155 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:09.150+0000 7f2510808700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:09.155 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:09.150+0000 7f2510808700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:09.155 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:09.150+0000 7f2510808700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:09.156 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:09.150+0000 7f2510808700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:09.156 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:09.150+0000 7f2510808700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:09.156 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:09.151+0000 7f250d802700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:09.156 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:09.151+0000 7f250d802700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:09.156 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:09.151+0000 7f250d802700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:09.156 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:09.151+0000 7f250d802700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:09.157 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:09.151+0000 7f250d802700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:09.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.233+0000 7f0cec07c700 1 -- 172.21.15.1:0/3316967762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ce40676a0 con 0x7f0ce40cf960 2024-03-18T20:25:09.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.233+0000 7f0cda7fc700 1 -- 172.21.15.1:0/3316967762 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f0cdc075cd0 con 0x7f0ce40cf960 2024-03-18T20:25:09.239 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:09.239 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":97,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2241819619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2241819619},{"type":"v1","addr":"172.21.15.165:6835","nonce":2241819619}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/320492236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":320492236},{"type":"v1","addr":"172.21.15.165:6834","nonce":320492236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":11031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/1035910531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1035910531},{"type":"v1","addr":"172.21.15.1:6836","nonce":1035910531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:23:39.747292+0000","modified":"2024-03-18T20:23:42.766952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11034},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11034":{"gid":11034,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/2388480799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":2388480799},{"type":"v1","addr":"172.21.15.1:6837","nonce":2388480799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-03-18T20:25:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.235+0000 7f0cec07c700 1 -- 172.21.15.1:0/3316967762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ce40cf960 msgr2=0x7f0ce40ae4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.235+0000 7f0cec07c700 1 --2- 172.21.15.1:0/3316967762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ce40cf960 0x7f0ce40ae4e0 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc00efc0 tx=0x7f0cdc00c3a0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.235+0000 7f0cec07c700 1 -- 172.21.15.1:0/3316967762 shutdown_connections 2024-03-18T20:25:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.235+0000 7f0cec07c700 1 --2- 172.21.15.1:0/3316967762 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0cc806aed0 0x7f0cc806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.235+0000 7f0cec07c700 1 --2- 172.21.15.1:0/3316967762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ce40cf960 0x7f0ce40ae4e0 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.235+0000 7f0cec07c700 1 --2- 172.21.15.1:0/3316967762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ce406a750 0x7f0ce41432a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.235+0000 7f0cec07c700 1 --2- 172.21.15.1:0/3316967762 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ce40cdda0 0x7f0ce40adfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.235+0000 7f0cec07c700 1 -- 172.21.15.1:0/3316967762 >> 172.21.15.1:0/3316967762 conn(0x7f0ce406b860 msgr2=0x7f0ce40cc2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.236+0000 7f0cec07c700 1 -- 172.21.15.1:0/3316967762 shutdown_connections 2024-03-18T20:25:09.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.236+0000 7f0cec07c700 1 -- 172.21.15.1:0/3316967762 wait complete. 2024-03-18T20:25:09.242 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 97 2024-03-18T20:25:09.253 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:25:09.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.372+0000 7faaf0d95700 1 Processor -- start 2024-03-18T20:25:09.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.372+0000 7faaf0d95700 1 -- start start 2024-03-18T20:25:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.373+0000 7faaf0d95700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faaec140b20 0x7faaec0cfe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.373+0000 7faaf0d95700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faaec0cdf80 0x7faaec0cba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.373+0000 7faaf0d95700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaec0d03d0 con 0x7faaec0cfab0 2024-03-18T20:25:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.373+0000 7faaf0d95700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaec15fbe0 con 0x7faaec0cdf80 2024-03-18T20:25:09.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.373+0000 7faaf0d95700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaec15f060 con 0x7faaec140b20 2024-03-18T20:25:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.373+0000 7faaead9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faaec0cdf80 0x7faaec0cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.373+0000 7faaead9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faaec0cdf80 0x7faaec0cba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60830/0 (socket says 172.21.15.1:60830) 2024-03-18T20:25:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.374+0000 7faaead9d700 1 -- 172.21.15.1:0/1930515496 learned_addr learned my addr 172.21.15.1:0/1930515496 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.374+0000 7faae9d9b700 1 --2- 172.21.15.1:0/1930515496 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faaec140b20 0x7faaec0cfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.374+0000 7faaead9d700 1 -- 172.21.15.1:0/1930515496 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faaec140b20 msgr2=0x7faaec0cfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.374+0000 7faaead9d700 1 --2- 172.21.15.1:0/1930515496 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faaec140b20 0x7faaec0cfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.374+0000 7faaead9d700 1 -- 172.21.15.1:0/1930515496 >> v1:172.21.15.165:6789/0 conn(0x7faaec0cfab0 legacy=0x7faaec15c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.374+0000 7faae959a700 1 -- 172.21.15.1:0/1930515496 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939180569 0 0) 0x7faaec0d03d0 con 0x7faaec0cfab0 2024-03-18T20:25:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.374+0000 7faaead9d700 1 -- 172.21.15.1:0/1930515496 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaec15f340 con 0x7faaec0cdf80 2024-03-18T20:25:09.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.374+0000 7faaead9d700 1 --2- 172.21.15.1:0/1930515496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0cdf80 0x7faaec0cba70 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7faae4009e50 tx=0x7faae400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b047ac5fa1f1c30 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.374+0000 7faae959a700 1 -- 172.21.15.1:0/1930515496 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae400e070 con 0x7faaec0cdf80 2024-03-18T20:25:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.374+0000 7faae959a700 1 -- 172.21.15.1:0/1930515496 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae400b910 con 0x7faaec0cdf80 2024-03-18T20:25:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.374+0000 7faae959a700 1 -- 172.21.15.1:0/1930515496 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae400bcc0 con 0x7faaec0cdf80 2024-03-18T20:25:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.375+0000 7faaf0d95700 1 -- 172.21.15.1:0/1930515496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0cdf80 msgr2=0x7faaec0cba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.375+0000 7faaf0d95700 1 --2- 172.21.15.1:0/1930515496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0cdf80 0x7faaec0cba70 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7faae4009e50 tx=0x7faae400b330 comp rx=0 tx=0).stop 2024-03-18T20:25:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.375+0000 7faaf0d95700 1 -- 172.21.15.1:0/1930515496 shutdown_connections 2024-03-18T20:25:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.375+0000 7faaf0d95700 1 --2- 172.21.15.1:0/1930515496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0cdf80 0x7faaec0cba70 unknown :-1 s=CLOSED pgs=2445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.375+0000 7faaf0d95700 1 --2- 172.21.15.1:0/1930515496 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faaec140b20 0x7faaec0cfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.375+0000 7faaf0d95700 1 -- 172.21.15.1:0/1930515496 >> 172.21.15.1:0/1930515496 conn(0x7faaec06ab70 msgr2=0x7faaec06af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.375+0000 7faaf0d95700 1 -- 172.21.15.1:0/1930515496 shutdown_connections 2024-03-18T20:25:09.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.375+0000 7faaf0d95700 1 -- 172.21.15.1:0/1930515496 wait complete. 2024-03-18T20:25:09.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.376+0000 7faaf0d95700 1 Processor -- start 2024-03-18T20:25:09.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.376+0000 7faaf0d95700 1 -- start start 2024-03-18T20:25:09.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.377+0000 7faaf0d95700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0cdf80 0x7faaec13ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:09.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.377+0000 7faaf0d95700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaec0cfab0 0x7faaec13ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:09.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.377+0000 7faaf0d95700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faaec0b01a0 0x7faaec0b17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:09.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.377+0000 7faaf0d95700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaec0b13c0 con 0x7faaec0cdf80 2024-03-18T20:25:09.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.377+0000 7faaf0d95700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaec13b220 con 0x7faaec0cfab0 2024-03-18T20:25:09.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.377+0000 7faaf0d95700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaec0b0740 con 0x7faaec0b01a0 2024-03-18T20:25:09.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.377+0000 7faaea59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0cdf80 0x7faaec13ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:09.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.377+0000 7faaea59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0cdf80 0x7faaec13ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60836/0 (socket says 172.21.15.1:60836) 2024-03-18T20:25:09.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.377+0000 7faaea59c700 1 -- 172.21.15.1:0/50475643 learned_addr learned my addr 172.21.15.1:0/50475643 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:09.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.377+0000 7faae9d9b700 1 --2- 172.21.15.1:0/50475643 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaec0cfab0 0x7faaec13ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:09.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.377+0000 7faaead9d700 1 --2- 172.21.15.1:0/50475643 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faaec0b01a0 0x7faaec0b17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:09.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.377+0000 7faaea59c700 1 -- 172.21.15.1:0/50475643 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faaec0b01a0 msgr2=0x7faaec0b17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:09.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.377+0000 7faaea59c700 1 --2- 172.21.15.1:0/50475643 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faaec0b01a0 0x7faaec0b17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.377+0000 7faaea59c700 1 -- 172.21.15.1:0/50475643 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaec0cfab0 msgr2=0x7faaec13ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:09.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.377+0000 7faaea59c700 1 --2- 172.21.15.1:0/50475643 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaec0cfab0 0x7faaec13ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.377+0000 7faaea59c700 1 -- 172.21.15.1:0/50475643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae4009b30 con 0x7faaec0cdf80 2024-03-18T20:25:09.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.377+0000 7faae9d9b700 1 --2- 172.21.15.1:0/50475643 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaec0cfab0 0x7faaec13ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:09.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.377+0000 7faaea59c700 1 --2- 172.21.15.1:0/50475643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0cdf80 0x7faaec13ace0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7faadc0011e0 tx=0x7faadc00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:09.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.377+0000 7faadaffd700 1 -- 172.21.15.1:0/50475643 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc0054e0 con 0x7faaec0cdf80 2024-03-18T20:25:09.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.378+0000 7faadaffd700 1 -- 172.21.15.1:0/50475643 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faadc003680 con 0x7faaec0cdf80 2024-03-18T20:25:09.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.378+0000 7faadaffd700 1 -- 172.21.15.1:0/50475643 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc003a30 con 0x7faaec0cdf80 2024-03-18T20:25:09.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.378+0000 7faaf0d95700 1 -- 172.21.15.1:0/50475643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaec0b0a20 con 0x7faaec0cdf80 2024-03-18T20:25:09.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.378+0000 7faaf0d95700 1 -- 172.21.15.1:0/50475643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaec0b0fe0 con 0x7faaec0cdf80 2024-03-18T20:25:09.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.379+0000 7faaf0d95700 1 -- 172.21.15.1:0/50475643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaec0676a0 con 0x7faaec0cdf80 2024-03-18T20:25:09.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.379+0000 7faadaffd700 1 -- 172.21.15.1:0/50475643 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 132) v1 ==== 88848+0+0 (secure 0 0 0) 0x7faadc003b90 con 0x7faaec0cdf80 2024-03-18T20:25:09.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.379+0000 7faadaffd700 1 -- 172.21.15.1:0/50475643 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 6932+0+0 (secure 0 0 0) 0x7faadc088740 con 0x7faaec0cdf80 2024-03-18T20:25:09.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.384+0000 7faadaffd700 1 -- 172.21.15.1:0/50475643 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faadc0538a0 con 0x7faaec0cdf80 2024-03-18T20:25:09.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.586+0000 7faaf0d95700 1 -- 172.21.15.1:0/50475643 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faaec06cf10 con 0x7faaec0cdf80 2024-03-18T20:25:09.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.587+0000 7faadaffd700 1 -- 172.21.15.1:0/50475643 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v220) v1 ==== 74+0+17542 (secure 0 0 0) 0x7faadc017090 con 0x7faaec0cdf80 2024-03-18T20:25:09.593 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:09.593 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":220,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:25:09.083557+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:23:37.591406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_data","create_time":"2024-03-18T20:23:38.723169+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":219,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.c","mappings":[{"from":1,"to":0}]},{"pgid":"26.f","mappings":[{"from":1,"to":0}]},{"pgid":"26.28","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3517854457":"2024-03-19T20:25:09.083518+0000","172.21.15.1:0/205456073":"2024-03-19T20:25:09.083518+0000","172.21.15.1:0/3415079095":"2024-03-19T20:25:09.083518+0000","172.21.15.1:0/3206244524":"2024-03-19T20:25:09.083518+0000","172.21.15.1:0/248746907":"2024-03-19T20:25:09.083518+0000","172.21.15.1:0/2651488689":"2024-03-19T20:25:09.083518+0000","172.21.15.1:0/238941017":"2024-03-19T20:25:09.083518+0000","172.21.15.1:0/2462726896":"2024-03-19T20:23:54.844816+0000","172.21.15.1:6833/148356":"2024-03-19T20:23:54.844816+0000","172.21.15.1:6832/148356":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/232905881":"2024-03-19T20:25:09.083518+0000","172.21.15.1:0/4177404804":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/1122038130":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/2416229963":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/2662947140":"2024-03-19T20:23:54.844816+0000","172.21.15.1:0/2110350357":"2024-03-19T20:23:54.844816+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]},{"pool":26,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:09.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.590+0000 7faaf0d95700 1 -- 172.21.15.1:0/50475643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0cdf80 msgr2=0x7faaec13ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:09.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.590+0000 7faaf0d95700 1 --2- 172.21.15.1:0/50475643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0cdf80 0x7faaec13ace0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7faadc0011e0 tx=0x7faadc00ca30 comp rx=0 tx=0).stop 2024-03-18T20:25:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.590+0000 7faaf0d95700 1 -- 172.21.15.1:0/50475643 shutdown_connections 2024-03-18T20:25:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.590+0000 7faaf0d95700 1 --2- 172.21.15.1:0/50475643 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faaec0b01a0 0x7faaec0b17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.590+0000 7faaf0d95700 1 --2- 172.21.15.1:0/50475643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaec0cdf80 0x7faaec13ace0 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.590+0000 7faaf0d95700 1 --2- 172.21.15.1:0/50475643 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaec0cfab0 0x7faaec13ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.590+0000 7faaf0d95700 1 -- 172.21.15.1:0/50475643 >> 172.21.15.1:0/50475643 conn(0x7faaec06ab70 msgr2=0x7faaec15abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.590+0000 7faaf0d95700 1 -- 172.21.15.1:0/50475643 shutdown_connections 2024-03-18T20:25:09.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.590+0000 7faaf0d95700 1 -- 172.21.15.1:0/50475643 wait complete. 2024-03-18T20:25:09.608 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T20:25:09.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.728+0000 7f00d861f700 1 Processor -- start 2024-03-18T20:25:09.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.728+0000 7f00d861f700 1 -- start start 2024-03-18T20:25:09.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.729+0000 7f00d861f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f00d015a5f0 0x7f00d00cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:09.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.729+0000 7f00d861f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00d00cc4d0 0x7f00d006d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:09.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.729+0000 7f00d861f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d015aad0 con 0x7f00d015dbe0 2024-03-18T20:25:09.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.729+0000 7f00d861f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d00cc960 con 0x7f00d015a5f0 2024-03-18T20:25:09.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.729+0000 7f00d861f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d013f750 con 0x7f00d00cc4d0 2024-03-18T20:25:09.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.729+0000 7f00d5bba700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f00d015a5f0 0x7f00d00cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:09.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.729+0000 7f00d5bba700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f00d015a5f0 0x7f00d00cbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60852/0 (socket says 172.21.15.1:60852) 2024-03-18T20:25:09.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.729+0000 7f00d5bba700 1 -- 172.21.15.1:0/1977615790 learned_addr learned my addr 172.21.15.1:0/1977615790 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:09.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.729+0000 7f00d6bbc700 1 --2- 172.21.15.1:0/1977615790 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00d00cc4d0 0x7f00d006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:09.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.729+0000 7f00d5bba700 1 -- 172.21.15.1:0/1977615790 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00d00cc4d0 msgr2=0x7f00d006d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:09.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.729+0000 7f00d5bba700 1 --2- 172.21.15.1:0/1977615790 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00d00cc4d0 0x7f00d006d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.729+0000 7f00d5bba700 1 -- 172.21.15.1:0/1977615790 >> v1:172.21.15.1:6789/0 conn(0x7f00d015dbe0 legacy=0x7f00d00b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:09.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.729+0000 7f00d5bba700 1 -- 172.21.15.1:0/1977615790 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d015adb0 con 0x7f00d015a5f0 2024-03-18T20:25:09.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.729+0000 7f00d5bba700 1 --2- 172.21.15.1:0/1977615790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d015a5f0 0x7f00d00cbf90 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f00c00098f0 tx=0x7f00c0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f218d6f2b1300410 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:09.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.730+0000 7f00d53b9700 1 -- 172.21.15.1:0/1977615790 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c0011830 con 0x7f00d015a5f0 2024-03-18T20:25:09.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.730+0000 7f00d53b9700 1 -- 172.21.15.1:0/1977615790 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00c0011990 con 0x7f00d015a5f0 2024-03-18T20:25:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.730+0000 7f00d53b9700 1 -- 172.21.15.1:0/1977615790 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c0011d20 con 0x7f00d015a5f0 2024-03-18T20:25:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.730+0000 7f00d861f700 1 -- 172.21.15.1:0/1977615790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d015a5f0 msgr2=0x7f00d00cbf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.730+0000 7f00d861f700 1 --2- 172.21.15.1:0/1977615790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d015a5f0 0x7f00d00cbf90 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f00c00098f0 tx=0x7f00c0009c00 comp rx=0 tx=0).stop 2024-03-18T20:25:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.730+0000 7f00d861f700 1 -- 172.21.15.1:0/1977615790 shutdown_connections 2024-03-18T20:25:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.730+0000 7f00d861f700 1 --2- 172.21.15.1:0/1977615790 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00d00cc4d0 0x7f00d006d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.730+0000 7f00d861f700 1 --2- 172.21.15.1:0/1977615790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d015a5f0 0x7f00d00cbf90 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.730+0000 7f00d861f700 1 -- 172.21.15.1:0/1977615790 >> 172.21.15.1:0/1977615790 conn(0x7f00d00b1cb0 msgr2=0x7f00d00b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.730+0000 7f00d861f700 1 -- 172.21.15.1:0/1977615790 shutdown_connections 2024-03-18T20:25:09.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.730+0000 7f00d861f700 1 -- 172.21.15.1:0/1977615790 wait complete. 2024-03-18T20:25:09.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.731+0000 7f00d861f700 1 Processor -- start 2024-03-18T20:25:09.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.732+0000 7f00d861f700 1 -- start start 2024-03-18T20:25:09.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.732+0000 7f00d861f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00d00cc4d0 0x7f00d00cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:09.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.732+0000 7f00d861f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00d015a5f0 0x7f00d00aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:09.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.732+0000 7f00d861f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d015dbe0 0x7f00d006a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:09.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.732+0000 7f00d861f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d00a84a0 con 0x7f00d015dbe0 2024-03-18T20:25:09.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.732+0000 7f00d861f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d01f3380 con 0x7f00d015a5f0 2024-03-18T20:25:09.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.732+0000 7f00d861f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d01f3660 con 0x7f00d00cc4d0 2024-03-18T20:25:09.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.732+0000 7f00d6bbc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d015dbe0 0x7f00d006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:09.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.732+0000 7f00d6bbc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d015dbe0 0x7f00d006a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60856/0 (socket says 172.21.15.1:60856) 2024-03-18T20:25:09.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.732+0000 7f00d6bbc700 1 -- 172.21.15.1:0/3537556777 learned_addr learned my addr 172.21.15.1:0/3537556777 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:09.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.733+0000 7f00d63bb700 1 --2- 172.21.15.1:0/3537556777 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00d00cc4d0 0x7f00d00cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:09.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.733+0000 7f00d6bbc700 1 -- 172.21.15.1:0/3537556777 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00d00cc4d0 msgr2=0x7f00d00cb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:09.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.733+0000 7f00d6bbc700 1 --2- 172.21.15.1:0/3537556777 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00d00cc4d0 0x7f00d00cb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.733+0000 7f00d5bba700 1 --2- 172.21.15.1:0/3537556777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00d015a5f0 0x7f00d00aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:09.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.733+0000 7f00d6bbc700 1 -- 172.21.15.1:0/3537556777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00d015a5f0 msgr2=0x7f00d00aecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:09.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.733+0000 7f00d6bbc700 1 --2- 172.21.15.1:0/3537556777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00d015a5f0 0x7f00d00aecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:09.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.733+0000 7f00d6bbc700 1 -- 172.21.15.1:0/3537556777 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c0009550 con 0x7f00d015dbe0 2024-03-18T20:25:09.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.733+0000 7f00d6bbc700 1 --2- 172.21.15.1:0/3537556777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d015dbe0 0x7f00d006a750 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f00c800efc0 tx=0x7f00c800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:09.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.733+0000 7f00c6ffd700 1 -- 172.21.15.1:0/3537556777 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c800cd40 con 0x7f00d015dbe0 2024-03-18T20:25:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.733+0000 7f00c6ffd700 1 -- 172.21.15.1:0/3537556777 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00c800f040 con 0x7f00d015dbe0 2024-03-18T20:25:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.733+0000 7f00d861f700 1 -- 172.21.15.1:0/3537556777 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d01f3940 con 0x7f00d015dbe0 2024-03-18T20:25:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.733+0000 7f00c6ffd700 1 -- 172.21.15.1:0/3537556777 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c80146c0 con 0x7f00d015dbe0 2024-03-18T20:25:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.733+0000 7f00d861f700 1 -- 172.21.15.1:0/3537556777 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d00cbbf0 con 0x7f00d015dbe0 2024-03-18T20:25:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.734+0000 7f00d861f700 1 -- 172.21.15.1:0/3537556777 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d00676a0 con 0x7f00d015dbe0 2024-03-18T20:25:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.735+0000 7f00c6ffd700 1 -- 172.21.15.1:0/3537556777 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 132) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f00c8014820 con 0x7f00d015dbe0 2024-03-18T20:25:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.735+0000 7f00c6ffd700 1 -- 172.21.15.1:0/3537556777 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 6932+0+0 (secure 0 0 0) 0x7f00c8089bd0 con 0x7f00d015dbe0 2024-03-18T20:25:09.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.740+0000 7f00c6ffd700 1 -- 172.21.15.1:0/3537556777 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f00c8054d30 con 0x7f00d015dbe0 2024-03-18T20:25:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.988+0000 7f00d861f700 1 -- 172.21.15.1:0/3537556777 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f00d0151420 con 0x7f00d015dbe0 2024-03-18T20:25:09.994 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-18T20:25:09.989+0000 7fbdbeb46700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T20:25:10.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:09.999+0000 7f00c6ffd700 1 -- 172.21.15.1:0/3537556777 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v98) v1 ==== 161+0+0 (secure 0 0 0) 0x7f00c8018020 con 0x7f00d015dbe0 2024-03-18T20:25:10.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.001+0000 7f00d861f700 1 -- 172.21.15.1:0/3537556777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d015dbe0 msgr2=0x7f00d006a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:10.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.001+0000 7f00d861f700 1 --2- 172.21.15.1:0/3537556777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d015dbe0 0x7f00d006a750 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f00c800efc0 tx=0x7f00c800c3a0 comp rx=0 tx=0).stop 2024-03-18T20:25:10.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.001+0000 7f00d861f700 1 -- 172.21.15.1:0/3537556777 shutdown_connections 2024-03-18T20:25:10.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.001+0000 7f00d861f700 1 --2- 172.21.15.1:0/3537556777 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00d00cc4d0 0x7f00d00cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:10.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.001+0000 7f00d861f700 1 --2- 172.21.15.1:0/3537556777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d015dbe0 0x7f00d006a750 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:10.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.001+0000 7f00d861f700 1 --2- 172.21.15.1:0/3537556777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00d015a5f0 0x7f00d00aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:10.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.001+0000 7f00d861f700 1 -- 172.21.15.1:0/3537556777 >> 172.21.15.1:0/3537556777 conn(0x7f00d00b1cb0 msgr2=0x7f00d013c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:10.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.001+0000 7f00d861f700 1 -- 172.21.15.1:0/3537556777 shutdown_connections 2024-03-18T20:25:10.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.001+0000 7f00d861f700 1 -- 172.21.15.1:0/3537556777 wait complete. 2024-03-18T20:25:10.008 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T20:25:10.019 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T20:25:10.042 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T20:25:10.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.148+0000 7fbf5a429700 1 Processor -- start 2024-03-18T20:25:10.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.148+0000 7fbf5a429700 1 -- start start 2024-03-18T20:25:10.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.149+0000 7fbf5a429700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf540cfa90 0x7fbf5406a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:10.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.149+0000 7fbf5a429700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf5406ac90 0x7fbf54167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:10.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.149+0000 7fbf5a429700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf540cf600 con 0x7fbf5415d5a0 2024-03-18T20:25:10.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.149+0000 7fbf5a429700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf5406b140 con 0x7fbf540cfa90 2024-03-18T20:25:10.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.149+0000 7fbf5a429700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf540ce720 con 0x7fbf5406ac90 2024-03-18T20:25:10.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.149+0000 7fbf537fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf540cfa90 0x7fbf5406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:10.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.149+0000 7fbf537fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf540cfa90 0x7fbf5406a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60882/0 (socket says 172.21.15.1:60882) 2024-03-18T20:25:10.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.149+0000 7fbf537fe700 1 -- 172.21.15.1:0/2856488258 learned_addr learned my addr 172.21.15.1:0/2856488258 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:10.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.149+0000 7fbf589c6700 1 --2- 172.21.15.1:0/2856488258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf5406ac90 0x7fbf54167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:10.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.149+0000 7fbf537fe700 1 -- 172.21.15.1:0/2856488258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf5406ac90 msgr2=0x7fbf54167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:10.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.149+0000 7fbf537fe700 1 --2- 172.21.15.1:0/2856488258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf5406ac90 0x7fbf54167140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:10.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.149+0000 7fbf537fe700 1 -- 172.21.15.1:0/2856488258 >> v1:172.21.15.165:6789/0 conn(0x7fbf5415d5a0 legacy=0x7fbf540ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:10.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.149+0000 7fbf537fe700 1 -- 172.21.15.1:0/2856488258 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf540ce9f0 con 0x7fbf540cfa90 2024-03-18T20:25:10.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.149+0000 7fbf537fe700 1 --2- 172.21.15.1:0/2856488258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf540cfa90 0x7fbf5406a750 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c0098f0 tx=0x7fbf3c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ff9ffd5cc2f2a5f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:10.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.149+0000 7fbf52ffd700 1 -- 172.21.15.1:0/2856488258 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c00b8a0 con 0x7fbf540cfa90 2024-03-18T20:25:10.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.150+0000 7fbf52ffd700 1 -- 172.21.15.1:0/2856488258 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3c00ba00 con 0x7fbf540cfa90 2024-03-18T20:25:10.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.150+0000 7fbf52ffd700 1 -- 172.21.15.1:0/2856488258 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c00bd90 con 0x7fbf540cfa90 2024-03-18T20:25:10.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.150+0000 7fbf5a429700 1 -- 172.21.15.1:0/2856488258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf540cfa90 msgr2=0x7fbf5406a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:10.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.150+0000 7fbf5a429700 1 --2- 172.21.15.1:0/2856488258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf540cfa90 0x7fbf5406a750 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c0098f0 tx=0x7fbf3c009c00 comp rx=0 tx=0).stop 2024-03-18T20:25:10.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.150+0000 7fbf5a429700 1 -- 172.21.15.1:0/2856488258 shutdown_connections 2024-03-18T20:25:10.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.150+0000 7fbf5a429700 1 --2- 172.21.15.1:0/2856488258 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf5406ac90 0x7fbf54167140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:10.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.150+0000 7fbf5a429700 1 --2- 172.21.15.1:0/2856488258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf540cfa90 0x7fbf5406a750 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:10.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.150+0000 7fbf5a429700 1 -- 172.21.15.1:0/2856488258 >> 172.21.15.1:0/2856488258 conn(0x7fbf540cc9b0 msgr2=0x7fbf540ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:10.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.150+0000 7fbf5a429700 1 -- 172.21.15.1:0/2856488258 shutdown_connections 2024-03-18T20:25:10.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.150+0000 7fbf5a429700 1 -- 172.21.15.1:0/2856488258 wait complete. 2024-03-18T20:25:10.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.151+0000 7fbf5a429700 1 Processor -- start 2024-03-18T20:25:10.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.152+0000 7fbf5a429700 1 -- start start 2024-03-18T20:25:10.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.152+0000 7fbf5a429700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf5406ac90 0x7fbf540c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:10.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.152+0000 7fbf5a429700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf540cfa90 0x7fbf5415ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:10.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.152+0000 7fbf5a429700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf5415d5a0 0x7fbf540c8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:10.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.152+0000 7fbf5a429700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf540c8360 con 0x7fbf5406ac90 2024-03-18T20:25:10.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.152+0000 7fbf5a429700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf540c76e0 con 0x7fbf5415d5a0 2024-03-18T20:25:10.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.152+0000 7fbf5a429700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf540c79c0 con 0x7fbf540cfa90 2024-03-18T20:25:10.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.152+0000 7fbf53fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf5406ac90 0x7fbf540c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:10.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.152+0000 7fbf53fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf5406ac90 0x7fbf540c7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60888/0 (socket says 172.21.15.1:60888) 2024-03-18T20:25:10.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.152+0000 7fbf53fff700 1 -- 172.21.15.1:0/2096729972 learned_addr learned my addr 172.21.15.1:0/2096729972 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:10.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.153+0000 7fbf589c6700 1 --2- 172.21.15.1:0/2096729972 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf5415d5a0 0x7fbf540c8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:10.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.153+0000 7fbf537fe700 1 --2- 172.21.15.1:0/2096729972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf540cfa90 0x7fbf5415ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:10.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.153+0000 7fbf53fff700 1 -- 172.21.15.1:0/2096729972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf540cfa90 msgr2=0x7fbf5415ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:10.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.153+0000 7fbf53fff700 1 --2- 172.21.15.1:0/2096729972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf540cfa90 0x7fbf5415ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:10.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.153+0000 7fbf53fff700 1 -- 172.21.15.1:0/2096729972 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf5415d5a0 msgr2=0x7fbf540c8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:10.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.153+0000 7fbf53fff700 1 --2- 172.21.15.1:0/2096729972 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf5415d5a0 0x7fbf540c8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:10.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.153+0000 7fbf53fff700 1 -- 172.21.15.1:0/2096729972 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3c009550 con 0x7fbf5406ac90 2024-03-18T20:25:10.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.153+0000 7fbf53fff700 1 --2- 172.21.15.1:0/2096729972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf5406ac90 0x7fbf540c7000 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7fbf4400c4d0 tx=0x7fbf4400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:10.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.153+0000 7fbf50ff9700 1 -- 172.21.15.1:0/2096729972 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf440052d0 con 0x7fbf5406ac90 2024-03-18T20:25:10.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.153+0000 7fbf50ff9700 1 -- 172.21.15.1:0/2096729972 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf44005430 con 0x7fbf5406ac90 2024-03-18T20:25:10.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.153+0000 7fbf5a429700 1 -- 172.21.15.1:0/2096729972 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf540c7ca0 con 0x7fbf5406ac90 2024-03-18T20:25:10.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.153+0000 7fbf50ff9700 1 -- 172.21.15.1:0/2096729972 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf440038f0 con 0x7fbf5406ac90 2024-03-18T20:25:10.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.153+0000 7fbf5a429700 1 -- 172.21.15.1:0/2096729972 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf541492d0 con 0x7fbf5406ac90 2024-03-18T20:25:10.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.154+0000 7fbf5a429700 1 -- 172.21.15.1:0/2096729972 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf540676a0 con 0x7fbf5406ac90 2024-03-18T20:25:10.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.154+0000 7fbf50ff9700 1 -- 172.21.15.1:0/2096729972 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 133) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fbf44003a50 con 0x7fbf5406ac90 2024-03-18T20:25:10.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.155+0000 7fbf50ff9700 1 --2- 172.21.15.1:0/2096729972 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf3406aae0 0x7fbf3406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:10.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.155+0000 7fbf50ff9700 1 -- 172.21.15.1:0/2096729972 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7018+0+0 (secure 0 0 0) 0x7fbf440888e0 con 0x7fbf5406ac90 2024-03-18T20:25:10.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.155+0000 7fbf537fe700 1 --2- 172.21.15.1:0/2096729972 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf3406aae0 0x7fbf3406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:10.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.155+0000 7fbf537fe700 1 --2- 172.21.15.1:0/2096729972 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf3406aae0 0x7fbf3406cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c009c40 tx=0x7fbf3c008030 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:10.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.160+0000 7fbf50ff9700 1 -- 172.21.15.1:0/2096729972 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbf44053810 con 0x7fbf5406ac90 2024-03-18T20:25:10.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:10.407+0000 7fbf5a429700 1 -- 172.21.15.1:0/2096729972 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fbf5406df80 con 0x7fbf5406ac90 2024-03-18T20:25:10.586 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:25:10.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:25:11.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.001+0000 7fbf50ff9700 1 -- 172.21.15.1:0/2096729972 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v99) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbf44056e30 con 0x7fbf5406ac90 2024-03-18T20:25:11.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.003+0000 7fbf5a429700 1 -- 172.21.15.1:0/2096729972 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf3406aae0 msgr2=0x7fbf3406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:11.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.003+0000 7fbf5a429700 1 --2- 172.21.15.1:0/2096729972 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf3406aae0 0x7fbf3406cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c009c40 tx=0x7fbf3c008030 comp rx=0 tx=0).stop 2024-03-18T20:25:11.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.003+0000 7fbf5a429700 1 -- 172.21.15.1:0/2096729972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf5406ac90 msgr2=0x7fbf540c7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:11.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.003+0000 7fbf5a429700 1 --2- 172.21.15.1:0/2096729972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf5406ac90 0x7fbf540c7000 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7fbf4400c4d0 tx=0x7fbf4400c7e0 comp rx=0 tx=0).stop 2024-03-18T20:25:11.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.003+0000 7fbf5a429700 1 -- 172.21.15.1:0/2096729972 shutdown_connections 2024-03-18T20:25:11.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.003+0000 7fbf5a429700 1 --2- 172.21.15.1:0/2096729972 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf3406aae0 0x7fbf3406cfa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:11.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.003+0000 7fbf5a429700 1 --2- 172.21.15.1:0/2096729972 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf5415d5a0 0x7fbf540c8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:11.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.003+0000 7fbf5a429700 1 --2- 172.21.15.1:0/2096729972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf5406ac90 0x7fbf540c7000 unknown :-1 s=CLOSED pgs=2453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:11.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.003+0000 7fbf5a429700 1 --2- 172.21.15.1:0/2096729972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf540cfa90 0x7fbf5415ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:11.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.003+0000 7fbf5a429700 1 -- 172.21.15.1:0/2096729972 >> 172.21.15.1:0/2096729972 conn(0x7fbf540cc9b0 msgr2=0x7fbf540ce010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:11.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.003+0000 7fbf5a429700 1 -- 172.21.15.1:0/2096729972 shutdown_connections 2024-03-18T20:25:11.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.003+0000 7fbf5a429700 1 -- 172.21.15.1:0/2096729972 wait complete. 2024-03-18T20:25:11.023 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T20:25:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.155+0000 7f5084a5a700 1 Processor -- start 2024-03-18T20:25:11.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.155+0000 7f5084a5a700 1 -- start start 2024-03-18T20:25:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.156+0000 7f5084a5a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f508015b550 0x7f5080099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.156+0000 7f5084a5a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f508009a600 0x7f5080099f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.156+0000 7f5084a5a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50800ae260 con 0x7f50800ae6f0 2024-03-18T20:25:11.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.156+0000 7f5084a5a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50800d0150 con 0x7f508009a600 2024-03-18T20:25:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.156+0000 7f5084a5a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50800cfb90 con 0x7f508015b550 2024-03-18T20:25:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.156+0000 7f507dd9b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f508009a600 0x7f5080099f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.156+0000 7f507e59c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f508015b550 0x7f5080099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.156+0000 7f507ed9d700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f50800ae6f0 0x7f50800aeb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:47002/0 (socket says 172.21.15.1:47002) 2024-03-18T20:25:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.156+0000 7f507ed9d700 1 -- 172.21.15.1:0/91124030 learned_addr learned my addr 172.21.15.1:0/91124030 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.156+0000 7f507dd9b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f508009a600 0x7f5080099f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:55246/0 (socket says 172.21.15.1:55246) 2024-03-18T20:25:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.157+0000 7f507d59a700 1 -- 172.21.15.1:0/91124030 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924968884 0 0) 0x7f50800ae260 con 0x7f50800ae6f0 2024-03-18T20:25:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.157+0000 7f507e59c700 1 -- 172.21.15.1:0/91124030 >> v2:172.21.15.165:3300/0 conn(0x7f508009a600 msgr2=0x7f5080099f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.157+0000 7f507e59c700 1 --2- 172.21.15.1:0/91124030 >> v2:172.21.15.165:3300/0 conn(0x7f508009a600 0x7f5080099f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:11.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.157+0000 7f507e59c700 1 -- 172.21.15.1:0/91124030 >> v1:172.21.15.165:6789/0 conn(0x7f50800ae6f0 legacy=0x7f50800aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:11.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.157+0000 7f507e59c700 1 -- 172.21.15.1:0/91124030 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50800cfe70 con 0x7f508015b550 2024-03-18T20:25:11.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.157+0000 7f507dd9b700 1 --2- 172.21.15.1:0/91124030 >> v2:172.21.15.165:3300/0 conn(0x7f508009a600 0x7f5080099f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:11.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.157+0000 7f507e59c700 1 --2- 172.21.15.1:0/91124030 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f508015b550 0x7f5080099a40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5070009840 tx=0x7f5070009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=157951a749daa899 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:11.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.157+0000 7f507d59a700 1 -- 172.21.15.1:0/91124030 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070011830 con 0x7f508015b550 2024-03-18T20:25:11.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.157+0000 7f507d59a700 1 -- 172.21.15.1:0/91124030 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5070011990 con 0x7f508015b550 2024-03-18T20:25:11.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.157+0000 7f507d59a700 1 -- 172.21.15.1:0/91124030 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070011d20 con 0x7f508015b550 2024-03-18T20:25:11.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.158+0000 7f5084a5a700 1 -- 172.21.15.1:0/91124030 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f508015b550 msgr2=0x7f5080099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:11.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.158+0000 7f5084a5a700 1 --2- 172.21.15.1:0/91124030 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f508015b550 0x7f5080099a40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5070009840 tx=0x7f5070009b50 comp rx=0 tx=0).stop 2024-03-18T20:25:11.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.158+0000 7f5084a5a700 1 -- 172.21.15.1:0/91124030 shutdown_connections 2024-03-18T20:25:11.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.158+0000 7f5084a5a700 1 --2- 172.21.15.1:0/91124030 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f508015b550 0x7f5080099a40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:11.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.158+0000 7f5084a5a700 1 --2- 172.21.15.1:0/91124030 >> v2:172.21.15.165:3300/0 conn(0x7f508009a600 0x7f5080099f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:11.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.158+0000 7f5084a5a700 1 -- 172.21.15.1:0/91124030 >> 172.21.15.1:0/91124030 conn(0x7f50800b2c30 msgr2=0x7f50800b3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:11.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.158+0000 7f5084a5a700 1 -- 172.21.15.1:0/91124030 shutdown_connections 2024-03-18T20:25:11.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.158+0000 7f5084a5a700 1 -- 172.21.15.1:0/91124030 wait complete. 2024-03-18T20:25:11.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.159+0000 7f5084a5a700 1 Processor -- start 2024-03-18T20:25:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.159+0000 7f5084a5a700 1 -- start start 2024-03-18T20:25:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.160+0000 7f5084a5a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f508009a600 0x7f50801f0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.160+0000 7f5084a5a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50800ae6f0 0x7f50800b6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.160+0000 7f5084a5a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f508015b550 0x7f50800ca120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.160+0000 7f5084a5a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50800c83b0 con 0x7f508009a600 2024-03-18T20:25:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.160+0000 7f5084a5a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50800c8690 con 0x7f508015b550 2024-03-18T20:25:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.160+0000 7f5084a5a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50800c8970 con 0x7f50800ae6f0 2024-03-18T20:25:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.160+0000 7f507e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f508009a600 0x7f50801f0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.160+0000 7f507e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f508009a600 0x7f50801f0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60920/0 (socket says 172.21.15.1:60920) 2024-03-18T20:25:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.160+0000 7f507e59c700 1 -- 172.21.15.1:0/2726946169 learned_addr learned my addr 172.21.15.1:0/2726946169 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.160+0000 7f507ed9d700 1 --2- 172.21.15.1:0/2726946169 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f508015b550 0x7f50800ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.160+0000 7f507dd9b700 1 --2- 172.21.15.1:0/2726946169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50800ae6f0 0x7f50800b6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:11.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.161+0000 7f507e59c700 1 -- 172.21.15.1:0/2726946169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50800ae6f0 msgr2=0x7f50800b6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:11.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.161+0000 7f507e59c700 1 --2- 172.21.15.1:0/2726946169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50800ae6f0 0x7f50800b6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:11.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.161+0000 7f507e59c700 1 -- 172.21.15.1:0/2726946169 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f508015b550 msgr2=0x7f50800ca120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:11.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.161+0000 7f507e59c700 1 --2- 172.21.15.1:0/2726946169 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f508015b550 0x7f50800ca120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:11.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.161+0000 7f507e59c700 1 -- 172.21.15.1:0/2726946169 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50740094a0 con 0x7f508009a600 2024-03-18T20:25:11.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.161+0000 7f507e59c700 1 --2- 172.21.15.1:0/2726946169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f508009a600 0x7f50801f0270 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7f5070009b90 tx=0x7f5070004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:11.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.161+0000 7f506effd700 1 -- 172.21.15.1:0/2726946169 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50700036a0 con 0x7f508009a600 2024-03-18T20:25:11.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.161+0000 7f506effd700 1 -- 172.21.15.1:0/2726946169 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507001e070 con 0x7f508009a600 2024-03-18T20:25:11.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.161+0000 7f5084a5a700 1 -- 172.21.15.1:0/2726946169 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50700094a0 con 0x7f508009a600 2024-03-18T20:25:11.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.161+0000 7f506effd700 1 -- 172.21.15.1:0/2726946169 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070003a20 con 0x7f508009a600 2024-03-18T20:25:11.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.161+0000 7f5084a5a700 1 -- 172.21.15.1:0/2726946169 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50800c8f30 con 0x7f508009a600 2024-03-18T20:25:11.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.162+0000 7f5084a5a700 1 -- 172.21.15.1:0/2726946169 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50800c9150 con 0x7f508009a600 2024-03-18T20:25:11.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.162+0000 7f506effd700 1 -- 172.21.15.1:0/2726946169 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 133) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f5070003b80 con 0x7f508009a600 2024-03-18T20:25:11.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.163+0000 7f506effd700 1 --2- 172.21.15.1:0/2726946169 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f506006aae0 0x7f506006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:11.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.163+0000 7f506effd700 1 -- 172.21.15.1:0/2726946169 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 7018+0+0 (secure 0 0 0) 0x7f507008a320 con 0x7f508009a600 2024-03-18T20:25:11.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.163+0000 7f507dd9b700 1 --2- 172.21.15.1:0/2726946169 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f506006aae0 0x7f506006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:11.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.164+0000 7f507dd9b700 1 --2- 172.21.15.1:0/2726946169 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f506006aae0 0x7f506006cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f508009aa20 tx=0x7f5074009210 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:11.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.168+0000 7f506effd700 1 -- 172.21.15.1:0/2726946169 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5070055320 con 0x7f508009a600 2024-03-18T20:25:11.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:11.373+0000 7f5084a5a700 1 -- 172.21.15.1:0/2726946169 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f50800676a0 con 0x7f508009a600 2024-03-18T20:25:12.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.104+0000 7f506effd700 1 -- 172.21.15.1:0/2726946169 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 134) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f50700556f0 con 0x7f508009a600 2024-03-18T20:25:12.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.198+0000 7f506effd700 1 -- 172.21.15.1:0/2726946169 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v222) v1 ==== 181+0+0 (secure 0 0 0) 0x7f507008a0d0 con 0x7f508009a600 2024-03-18T20:25:12.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.201+0000 7f5084a5a700 1 -- 172.21.15.1:0/2726946169 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f506006aae0 msgr2=0x7f506006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:12.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.201+0000 7f5084a5a700 1 --2- 172.21.15.1:0/2726946169 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f506006aae0 0x7f506006cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f508009aa20 tx=0x7f5074009210 comp rx=0 tx=0).stop 2024-03-18T20:25:12.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.201+0000 7f5084a5a700 1 -- 172.21.15.1:0/2726946169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f508009a600 msgr2=0x7f50801f0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:12.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.201+0000 7f5084a5a700 1 --2- 172.21.15.1:0/2726946169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f508009a600 0x7f50801f0270 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7f5070009b90 tx=0x7f5070004750 comp rx=0 tx=0).stop 2024-03-18T20:25:12.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.201+0000 7f5084a5a700 1 -- 172.21.15.1:0/2726946169 shutdown_connections 2024-03-18T20:25:12.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.201+0000 7f5084a5a700 1 --2- 172.21.15.1:0/2726946169 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f506006aae0 0x7f506006cfa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:12.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.201+0000 7f5084a5a700 1 --2- 172.21.15.1:0/2726946169 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f508015b550 0x7f50800ca120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:12.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.201+0000 7f5084a5a700 1 --2- 172.21.15.1:0/2726946169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f508009a600 0x7f50801f0270 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:12.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.201+0000 7f5084a5a700 1 --2- 172.21.15.1:0/2726946169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50800ae6f0 0x7f50800b6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:12.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.201+0000 7f5084a5a700 1 -- 172.21.15.1:0/2726946169 >> 172.21.15.1:0/2726946169 conn(0x7f50800b2c30 msgr2=0x7f50800cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:12.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.201+0000 7f5084a5a700 1 -- 172.21.15.1:0/2726946169 shutdown_connections 2024-03-18T20:25:12.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.202+0000 7f5084a5a700 1 -- 172.21.15.1:0/2726946169 wait complete. 2024-03-18T20:25:12.209 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-18T20:25:12.224 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T20:25:12.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.354+0000 7fa84c2b2700 1 Processor -- start 2024-03-18T20:25:12.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.354+0000 7fa84c2b2700 1 -- start start 2024-03-18T20:25:12.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.354+0000 7fa84c2b2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa84415af20 0x7fa844072480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:12.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.354+0000 7fa84c2b2700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa8440af6b0 0x7fa84415dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:12.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.354+0000 7fa84c2b2700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa84415e860 con 0x7fa84415b300 2024-03-18T20:25:12.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.354+0000 7fa84c2b2700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8440719a0 con 0x7fa84415af20 2024-03-18T20:25:12.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.354+0000 7fa84c2b2700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa844071c80 con 0x7fa8440af6b0 2024-03-18T20:25:12.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.355+0000 7fa84a04e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa84415af20 0x7fa844072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:12.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.355+0000 7fa84a04e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa84415af20 0x7fa844072480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60934/0 (socket says 172.21.15.1:60934) 2024-03-18T20:25:12.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.355+0000 7fa84a04e700 1 -- 172.21.15.1:0/114302864 learned_addr learned my addr 172.21.15.1:0/114302864 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:12.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.355+0000 7fa84a84f700 1 --2- 172.21.15.1:0/114302864 >> v2:172.21.15.165:3300/0 conn(0x7fa8440af6b0 0x7fa84415dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:12.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.355+0000 7fa84a04e700 1 -- 172.21.15.1:0/114302864 >> v2:172.21.15.165:3300/0 conn(0x7fa8440af6b0 msgr2=0x7fa84415dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:12.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.355+0000 7fa84a04e700 1 --2- 172.21.15.1:0/114302864 >> v2:172.21.15.165:3300/0 conn(0x7fa8440af6b0 0x7fa84415dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:12.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.355+0000 7fa84a04e700 1 -- 172.21.15.1:0/114302864 >> v1:172.21.15.165:6789/0 conn(0x7fa84415b300 legacy=0x7fa8440aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:12.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.355+0000 7fa84a04e700 1 -- 172.21.15.1:0/114302864 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa844071f60 con 0x7fa84415af20 2024-03-18T20:25:12.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.355+0000 7fa84a04e700 1 --2- 172.21.15.1:0/114302864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa84415af20 0x7fa844072480 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7fa8400098f0 tx=0x7fa840009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d53728a2dbeebd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:12.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.355+0000 7fa84904c700 1 -- 172.21.15.1:0/114302864 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84000b860 con 0x7fa84415af20 2024-03-18T20:25:12.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.355+0000 7fa84904c700 1 -- 172.21.15.1:0/114302864 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa84000b9c0 con 0x7fa84415af20 2024-03-18T20:25:12.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.355+0000 7fa84904c700 1 -- 172.21.15.1:0/114302864 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84000bd50 con 0x7fa84415af20 2024-03-18T20:25:12.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.356+0000 7fa84c2b2700 1 -- 172.21.15.1:0/114302864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa84415af20 msgr2=0x7fa844072480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:12.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.356+0000 7fa84c2b2700 1 --2- 172.21.15.1:0/114302864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa84415af20 0x7fa844072480 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7fa8400098f0 tx=0x7fa840009c00 comp rx=0 tx=0).stop 2024-03-18T20:25:12.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.356+0000 7fa84c2b2700 1 -- 172.21.15.1:0/114302864 shutdown_connections 2024-03-18T20:25:12.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.356+0000 7fa84c2b2700 1 --2- 172.21.15.1:0/114302864 >> v2:172.21.15.165:3300/0 conn(0x7fa8440af6b0 0x7fa84415dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:12.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.356+0000 7fa84c2b2700 1 --2- 172.21.15.1:0/114302864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa84415af20 0x7fa844072480 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:12.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.356+0000 7fa84c2b2700 1 -- 172.21.15.1:0/114302864 >> 172.21.15.1:0/114302864 conn(0x7fa8440d00a0 msgr2=0x7fa8440d04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:12.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.356+0000 7fa84c2b2700 1 -- 172.21.15.1:0/114302864 shutdown_connections 2024-03-18T20:25:12.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.356+0000 7fa84c2b2700 1 -- 172.21.15.1:0/114302864 wait complete. 2024-03-18T20:25:12.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.357+0000 7fa84c2b2700 1 Processor -- start 2024-03-18T20:25:12.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.357+0000 7fa84c2b2700 1 -- start start 2024-03-18T20:25:12.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.358+0000 7fa84c2b2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8440af6b0 0x7fa844147cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:12.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.358+0000 7fa84c2b2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa84415af20 0x7fa8440ada60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:12.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.358+0000 7fa84c2b2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa84415b300 0x7fa8440adfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:12.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.358+0000 7fa84c2b2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa844149630 con 0x7fa8440af6b0 2024-03-18T20:25:12.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.358+0000 7fa84c2b2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa844149910 con 0x7fa84415b300 2024-03-18T20:25:12.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.358+0000 7fa84c2b2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa844149bf0 con 0x7fa84415af20 2024-03-18T20:25:12.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.358+0000 7fa84a04e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8440af6b0 0x7fa844147cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:12.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.358+0000 7fa84a04e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8440af6b0 0x7fa844147cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60948/0 (socket says 172.21.15.1:60948) 2024-03-18T20:25:12.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.358+0000 7fa84a04e700 1 -- 172.21.15.1:0/2181938778 learned_addr learned my addr 172.21.15.1:0/2181938778 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:12.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.358+0000 7fa84984d700 1 --2- 172.21.15.1:0/2181938778 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa84415af20 0x7fa8440ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:12.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.358+0000 7fa84a84f700 1 --2- 172.21.15.1:0/2181938778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa84415b300 0x7fa8440adfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:12.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.358+0000 7fa84a04e700 1 -- 172.21.15.1:0/2181938778 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa84415af20 msgr2=0x7fa8440ada60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:12.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.358+0000 7fa84a04e700 1 --2- 172.21.15.1:0/2181938778 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa84415af20 0x7fa8440ada60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:12.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.358+0000 7fa84a04e700 1 -- 172.21.15.1:0/2181938778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa84415b300 msgr2=0x7fa8440adfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:12.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.358+0000 7fa84a04e700 1 --2- 172.21.15.1:0/2181938778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa84415b300 0x7fa8440adfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:12.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.358+0000 7fa84a04e700 1 -- 172.21.15.1:0/2181938778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa840009550 con 0x7fa8440af6b0 2024-03-18T20:25:12.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.358+0000 7fa84a84f700 1 --2- 172.21.15.1:0/2181938778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa84415b300 0x7fa8440adfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:12.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.358+0000 7fa84a04e700 1 --2- 172.21.15.1:0/2181938778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8440af6b0 0x7fa844147cd0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7fa840009c40 tx=0x7fa840004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:12.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.359+0000 7fa836ffd700 1 -- 172.21.15.1:0/2181938778 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa840004380 con 0x7fa8440af6b0 2024-03-18T20:25:12.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.359+0000 7fa836ffd700 1 -- 172.21.15.1:0/2181938778 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa84001f070 con 0x7fa8440af6b0 2024-03-18T20:25:12.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.359+0000 7fa84c2b2700 1 -- 172.21.15.1:0/2181938778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa844149ed0 con 0x7fa8440af6b0 2024-03-18T20:25:12.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.359+0000 7fa836ffd700 1 -- 172.21.15.1:0/2181938778 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8400036a0 con 0x7fa8440af6b0 2024-03-18T20:25:12.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.359+0000 7fa84c2b2700 1 -- 172.21.15.1:0/2181938778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa844148210 con 0x7fa8440af6b0 2024-03-18T20:25:12.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.360+0000 7fa84c2b2700 1 -- 172.21.15.1:0/2181938778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa844070270 con 0x7fa8440af6b0 2024-03-18T20:25:12.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.360+0000 7fa836ffd700 1 -- 172.21.15.1:0/2181938778 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 134) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fa840021030 con 0x7fa8440af6b0 2024-03-18T20:25:12.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.360+0000 7fa836ffd700 1 --2- 172.21.15.1:0/2181938778 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa82806ad20 0x7fa82806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:12.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.360+0000 7fa836ffd700 1 -- 172.21.15.1:0/2181938778 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 6580+0+0 (secure 0 0 0) 0x7fa84008aa00 con 0x7fa8440af6b0 2024-03-18T20:25:12.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.361+0000 7fa84984d700 1 --2- 172.21.15.1:0/2181938778 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa82806ad20 0x7fa82806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:12.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.361+0000 7fa84984d700 1 --2- 172.21.15.1:0/2181938778 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa82806ad20 0x7fa82806d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa84415a0e0 tx=0x7fa838009040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:12.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.366+0000 7fa836ffd700 1 -- 172.21.15.1:0/2181938778 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa840055910 con 0x7fa8440af6b0 2024-03-18T20:25:12.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:12.567+0000 7fa84c2b2700 1 -- 172.21.15.1:0/2181938778 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa8440676a0 con 0x7fa8440af6b0 2024-03-18T20:25:13.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.087+0000 7fa836ffd700 1 -- 172.21.15.1:0/2181938778 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v223) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa8400150d0 con 0x7fa8440af6b0 2024-03-18T20:25:13.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.093+0000 7fa84c2b2700 1 -- 172.21.15.1:0/2181938778 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa82806ad20 msgr2=0x7fa82806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:13.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.093+0000 7fa84c2b2700 1 --2- 172.21.15.1:0/2181938778 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa82806ad20 0x7fa82806d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa84415a0e0 tx=0x7fa838009040 comp rx=0 tx=0).stop 2024-03-18T20:25:13.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.093+0000 7fa84c2b2700 1 -- 172.21.15.1:0/2181938778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8440af6b0 msgr2=0x7fa844147cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:13.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.093+0000 7fa84c2b2700 1 --2- 172.21.15.1:0/2181938778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8440af6b0 0x7fa844147cd0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7fa840009c40 tx=0x7fa840004750 comp rx=0 tx=0).stop 2024-03-18T20:25:13.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.093+0000 7fa84c2b2700 1 -- 172.21.15.1:0/2181938778 shutdown_connections 2024-03-18T20:25:13.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.093+0000 7fa84c2b2700 1 --2- 172.21.15.1:0/2181938778 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa82806ad20 0x7fa82806d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.093+0000 7fa84c2b2700 1 --2- 172.21.15.1:0/2181938778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa84415b300 0x7fa8440adfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.093+0000 7fa84c2b2700 1 --2- 172.21.15.1:0/2181938778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8440af6b0 0x7fa844147cd0 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.093+0000 7fa84c2b2700 1 --2- 172.21.15.1:0/2181938778 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa84415af20 0x7fa8440ada60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.093+0000 7fa84c2b2700 1 -- 172.21.15.1:0/2181938778 >> 172.21.15.1:0/2181938778 conn(0x7fa8440d00a0 msgr2=0x7fa8440b1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:13.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.093+0000 7fa84c2b2700 1 -- 172.21.15.1:0/2181938778 shutdown_connections 2024-03-18T20:25:13.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.094+0000 7fa84c2b2700 1 -- 172.21.15.1:0/2181938778 wait complete. 2024-03-18T20:25:13.103 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-18T20:25:13.116 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-18T20:25:13.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.244+0000 7fe339e6c700 1 Processor -- start 2024-03-18T20:25:13.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.245+0000 7fe339e6c700 1 -- start start 2024-03-18T20:25:13.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.245+0000 7fe339e6c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe3340714c0 0x7fe33415bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.245+0000 7fe339e6c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3340cbcc0 con 0x7fe33415b750 2024-03-18T20:25:13.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.245+0000 7fe339e6c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe33415c210 con 0x7fe33413e130 2024-03-18T20:25:13.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.245+0000 7fe339e6c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe33413eff0 con 0x7fe3340714c0 2024-03-18T20:25:13.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe332ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe33415b750 0x7fe334070db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53096/0 (socket says 172.21.15.1:53096) 2024-03-18T20:25:13.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe332ffd700 1 -- 172.21.15.1:0/252366748 learned_addr learned my addr 172.21.15.1:0/252366748 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:13.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe333fff700 1 --2- 172.21.15.1:0/252366748 >> v2:172.21.15.165:3300/0 conn(0x7fe3340714c0 0x7fe33415bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe3327fc700 1 -- 172.21.15.1:0/252366748 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505994908 0 0) 0x7fe3340cbcc0 con 0x7fe33415b750 2024-03-18T20:25:13.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe3327fc700 1 -- 172.21.15.1:0/252366748 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3200035b0 con 0x7fe33415b750 2024-03-18T20:25:13.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe3327fc700 1 -- 172.21.15.1:0/252366748 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3538368795 0 0) 0x7fe3200035b0 con 0x7fe33415b750 2024-03-18T20:25:13.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe3327fc700 1 -- 172.21.15.1:0/252366748 >> v2:172.21.15.165:3300/0 conn(0x7fe3340714c0 msgr2=0x7fe33415bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:13.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe3327fc700 1 --2- 172.21.15.1:0/252366748 >> v2:172.21.15.165:3300/0 conn(0x7fe3340714c0 0x7fe33415bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe3327fc700 1 -- 172.21.15.1:0/252366748 >> v1:172.21.15.165:6789/0 conn(0x7fe33413e130 legacy=0x7fe33413e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe3327fc700 1 -- 172.21.15.1:0/252366748 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33415c4f0 con 0x7fe33415b750 2024-03-18T20:25:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe3327fc700 1 -- 172.21.15.1:0/252366748 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fe31c001670 con 0x7fe33415b750 2024-03-18T20:25:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe3327fc700 1 -- 172.21.15.1:0/252366748 >> v1:172.21.15.1:6789/0 conn(0x7fe33415b750 legacy=0x7fe334070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:13.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe3327fc700 1 --2- 172.21.15.1:0/252366748 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe320003a00 0x7fe320003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe3327fc700 1 --2- 172.21.15.1:0/252366748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe320004400 0x7fe3200048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe3327fc700 1 --2- 172.21.15.1:0/252366748 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe320005170 0x7fe32000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe3327fc700 1 -- 172.21.15.1:0/252366748 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33413eff0 con 0x7fe320004400 2024-03-18T20:25:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe3327fc700 1 -- 172.21.15.1:0/252366748 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32000c0b0 con 0x7fe320003a00 2024-03-18T20:25:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe3327fc700 1 -- 172.21.15.1:0/252366748 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe32000c270 con 0x7fe320005170 2024-03-18T20:25:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe332ffd700 1 --2- 172.21.15.1:0/252366748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe320004400 0x7fe3200048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe3337fe700 1 --2- 172.21.15.1:0/252366748 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe320003a00 0x7fe320003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.246+0000 7fe333fff700 1 --2- 172.21.15.1:0/252366748 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe320005170 0x7fe32000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.247+0000 7fe332ffd700 1 -- 172.21.15.1:0/252366748 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe320005170 msgr2=0x7fe32000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.247+0000 7fe332ffd700 1 --2- 172.21.15.1:0/252366748 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe320005170 0x7fe32000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.247+0000 7fe332ffd700 1 -- 172.21.15.1:0/252366748 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe320003a00 msgr2=0x7fe320003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.247+0000 7fe332ffd700 1 --2- 172.21.15.1:0/252366748 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe320003a00 0x7fe320003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.247+0000 7fe332ffd700 1 -- 172.21.15.1:0/252366748 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3200035b0 con 0x7fe320004400 2024-03-18T20:25:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.247+0000 7fe333fff700 1 -- 172.21.15.1:0/252366748 reap_dead start 2024-03-18T20:25:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.247+0000 7fe332ffd700 1 --2- 172.21.15.1:0/252366748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe320004400 0x7fe3200048d0 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00c2f0 tx=0x7fe320005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c6a7703d1ae2404 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:13.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.247+0000 7fe3327fc700 1 -- 172.21.15.1:0/252366748 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c00ce30 con 0x7fe320004400 2024-03-18T20:25:13.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.247+0000 7fe3327fc700 1 -- 172.21.15.1:0/252366748 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31c005960 con 0x7fe320004400 2024-03-18T20:25:13.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.247+0000 7fe3327fc700 1 -- 172.21.15.1:0/252366748 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c005d10 con 0x7fe320004400 2024-03-18T20:25:13.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.247+0000 7fe339e6c700 1 -- 172.21.15.1:0/252366748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe320004400 msgr2=0x7fe3200048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:13.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.247+0000 7fe339e6c700 1 --2- 172.21.15.1:0/252366748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe320004400 0x7fe3200048d0 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00c2f0 tx=0x7fe320005880 comp rx=0 tx=0).stop 2024-03-18T20:25:13.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.248+0000 7fe339e6c700 1 -- 172.21.15.1:0/252366748 shutdown_connections 2024-03-18T20:25:13.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.248+0000 7fe339e6c700 1 --2- 172.21.15.1:0/252366748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe320004400 0x7fe3200048d0 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.248+0000 7fe339e6c700 1 -- 172.21.15.1:0/252366748 >> 172.21.15.1:0/252366748 conn(0x7fe33406a750 msgr2=0x7fe33406ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:13.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.248+0000 7fe339e6c700 1 -- 172.21.15.1:0/252366748 shutdown_connections 2024-03-18T20:25:13.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.248+0000 7fe339e6c700 1 -- 172.21.15.1:0/252366748 wait complete. 2024-03-18T20:25:13.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.250+0000 7fe339e6c700 1 Processor -- start 2024-03-18T20:25:13.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.251+0000 7fe339e6c700 1 -- start start 2024-03-18T20:25:13.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.252+0000 7fe339e6c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe320004400 0x7fe33415ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.252+0000 7fe339e6c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe334140b10 0x7fe334140f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.252+0000 7fe339e6c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3340b1100 0x7fe3340b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.252+0000 7fe339e6c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3341414d0 con 0x7fe334140b10 2024-03-18T20:25:13.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.252+0000 7fe339e6c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340b0c70 con 0x7fe320004400 2024-03-18T20:25:13.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.252+0000 7fe339e6c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340afdf0 con 0x7fe3340b1100 2024-03-18T20:25:13.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.252+0000 7fe332ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe334140b10 0x7fe334140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.252+0000 7fe332ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe334140b10 0x7fe334140f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60964/0 (socket says 172.21.15.1:60964) 2024-03-18T20:25:13.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.252+0000 7fe332ffd700 1 -- 172.21.15.1:0/2482117763 learned_addr learned my addr 172.21.15.1:0/2482117763 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:13.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.252+0000 7fe3337fe700 1 --2- 172.21.15.1:0/2482117763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe320004400 0x7fe33415ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.252+0000 7fe333fff700 1 --2- 172.21.15.1:0/2482117763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3340b1100 0x7fe3340b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.252+0000 7fe332ffd700 1 -- 172.21.15.1:0/2482117763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3340b1100 msgr2=0x7fe3340b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:13.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.252+0000 7fe332ffd700 1 --2- 172.21.15.1:0/2482117763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3340b1100 0x7fe3340b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.252+0000 7fe332ffd700 1 -- 172.21.15.1:0/2482117763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe320004400 msgr2=0x7fe33415ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:13.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.252+0000 7fe332ffd700 1 --2- 172.21.15.1:0/2482117763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe320004400 0x7fe33415ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.252+0000 7fe332ffd700 1 -- 172.21.15.1:0/2482117763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33415b1a0 con 0x7fe334140b10 2024-03-18T20:25:13.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.253+0000 7fe332ffd700 1 --2- 172.21.15.1:0/2482117763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe334140b10 0x7fe334140f90 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7fe31c0016c0 tx=0x7fe31c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:13.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.253+0000 7fe32bfff700 1 -- 172.21.15.1:0/2482117763 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c005ce0 con 0x7fe334140b10 2024-03-18T20:25:13.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.253+0000 7fe32bfff700 1 -- 172.21.15.1:0/2482117763 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31c005e40 con 0x7fe334140b10 2024-03-18T20:25:13.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.253+0000 7fe32bfff700 1 -- 172.21.15.1:0/2482117763 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c004a90 con 0x7fe334140b10 2024-03-18T20:25:13.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.253+0000 7fe339e6c700 1 -- 172.21.15.1:0/2482117763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3340b0060 con 0x7fe334140b10 2024-03-18T20:25:13.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.253+0000 7fe339e6c700 1 -- 172.21.15.1:0/2482117763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3340b0620 con 0x7fe334140b10 2024-03-18T20:25:13.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.254+0000 7fe339e6c700 1 -- 172.21.15.1:0/2482117763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3340676a0 con 0x7fe334140b10 2024-03-18T20:25:13.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.255+0000 7fe32bfff700 1 -- 172.21.15.1:0/2482117763 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 134) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fe31c004bf0 con 0x7fe334140b10 2024-03-18T20:25:13.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.256+0000 7fe32bfff700 1 --2- 172.21.15.1:0/2482117763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe31406ad20 0x7fe31406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.256+0000 7fe32bfff700 1 -- 172.21.15.1:0/2482117763 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6079+0+0 (secure 0 0 0) 0x7fe31c089a80 con 0x7fe334140b10 2024-03-18T20:25:13.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.256+0000 7fe3337fe700 1 --2- 172.21.15.1:0/2482117763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe31406ad20 0x7fe31406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.256+0000 7fe3337fe700 1 --2- 172.21.15.1:0/2482117763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe31406ad20 0x7fe31406d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe33413b250 tx=0x7fe324009040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:13.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.263+0000 7fe32bfff700 1 -- 172.21.15.1:0/2482117763 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe31c054c40 con 0x7fe334140b10 2024-03-18T20:25:13.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.468+0000 7fe339e6c700 1 -- 172.21.15.1:0/2482117763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fe3340b0db0 con 0x7fe334140b10 2024-03-18T20:25:13.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.472+0000 7fe32bfff700 1 -- 172.21.15.1:0/2482117763 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v73) v1 ==== 115+0+0 (secure 0 0 0) 0x7fe31c007da0 con 0x7fe334140b10 2024-03-18T20:25:13.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.474+0000 7fe339e6c700 1 -- 172.21.15.1:0/2482117763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe31406ad20 msgr2=0x7fe31406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:13.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.474+0000 7fe339e6c700 1 --2- 172.21.15.1:0/2482117763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe31406ad20 0x7fe31406d1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe33413b250 tx=0x7fe324009040 comp rx=0 tx=0).stop 2024-03-18T20:25:13.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.474+0000 7fe339e6c700 1 -- 172.21.15.1:0/2482117763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe334140b10 msgr2=0x7fe334140f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:13.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.474+0000 7fe339e6c700 1 --2- 172.21.15.1:0/2482117763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe334140b10 0x7fe334140f90 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7fe31c0016c0 tx=0x7fe31c001a00 comp rx=0 tx=0).stop 2024-03-18T20:25:13.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.474+0000 7fe339e6c700 1 -- 172.21.15.1:0/2482117763 shutdown_connections 2024-03-18T20:25:13.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.475+0000 7fe339e6c700 1 --2- 172.21.15.1:0/2482117763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe31406ad20 0x7fe31406d1e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.475+0000 7fe339e6c700 1 --2- 172.21.15.1:0/2482117763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3340b1100 0x7fe3340b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.475+0000 7fe339e6c700 1 --2- 172.21.15.1:0/2482117763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe320004400 0x7fe33415ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.475+0000 7fe339e6c700 1 --2- 172.21.15.1:0/2482117763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe334140b10 0x7fe334140f90 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.475+0000 7fe339e6c700 1 -- 172.21.15.1:0/2482117763 >> 172.21.15.1:0/2482117763 conn(0x7fe33406a750 msgr2=0x7fe3340cc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:13.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.475+0000 7fe339e6c700 1 -- 172.21.15.1:0/2482117763 shutdown_connections 2024-03-18T20:25:13.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.475+0000 7fe339e6c700 1 -- 172.21.15.1:0/2482117763 wait complete. 2024-03-18T20:25:13.491 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-18T20:25:13.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.614+0000 7f076ae6d700 1 Processor -- start 2024-03-18T20:25:13.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.614+0000 7f076ae6d700 1 -- start start 2024-03-18T20:25:13.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.615+0000 7f076ae6d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f07640cc9c0 0x7f0764071570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.615+0000 7f076ae6d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f076406a750 0x7f076406ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.615+0000 7f076ae6d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076406ef60 con 0x7f0764144ac0 2024-03-18T20:25:13.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.615+0000 7f076ae6d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07640cfdc0 con 0x7f07640cc9c0 2024-03-18T20:25:13.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.615+0000 7f076ae6d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f076406b0b0 con 0x7f076406a750 2024-03-18T20:25:13.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.615+0000 7f0763fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f07640cc9c0 0x7f0764071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.615+0000 7f0768c09700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0764144ac0 0x7f07640cf6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53108/0 (socket says 172.21.15.1:53108) 2024-03-18T20:25:13.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.615+0000 7f0768c09700 1 -- 172.21.15.1:0/1132424600 learned_addr learned my addr 172.21.15.1:0/1132424600 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:13.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.615+0000 7f076940a700 1 --2- 172.21.15.1:0/1132424600 >> v2:172.21.15.165:3300/0 conn(0x7f076406a750 0x7f076406ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.615+0000 7f0763fff700 1 -- 172.21.15.1:0/1132424600 >> v2:172.21.15.165:3300/0 conn(0x7f076406a750 msgr2=0x7f076406ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:13.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.615+0000 7f0763fff700 1 --2- 172.21.15.1:0/1132424600 >> v2:172.21.15.165:3300/0 conn(0x7f076406a750 0x7f076406ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.615+0000 7f0763fff700 1 -- 172.21.15.1:0/1132424600 >> v1:172.21.15.1:6789/0 conn(0x7f0764144ac0 legacy=0x7f07640cf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:13.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.615+0000 7f0763fff700 1 -- 172.21.15.1:0/1132424600 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0764144ea0 con 0x7f07640cc9c0 2024-03-18T20:25:13.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.615+0000 7f07637fe700 1 -- 172.21.15.1:0/1132424600 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803422507 0 0) 0x7f076406ef60 con 0x7f0764144ac0 2024-03-18T20:25:13.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.616+0000 7f0763fff700 1 --2- 172.21.15.1:0/1132424600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640cc9c0 0x7f0764071570 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f07640afc30 tx=0x7f0750009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=940f2de090df87cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:13.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.616+0000 7f07637fe700 1 -- 172.21.15.1:0/1132424600 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0750011830 con 0x7f07640cc9c0 2024-03-18T20:25:13.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.616+0000 7f07637fe700 1 -- 172.21.15.1:0/1132424600 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0750011990 con 0x7f07640cc9c0 2024-03-18T20:25:13.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.616+0000 7f07637fe700 1 -- 172.21.15.1:0/1132424600 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0750011d20 con 0x7f07640cc9c0 2024-03-18T20:25:13.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.616+0000 7f076ae6d700 1 -- 172.21.15.1:0/1132424600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640cc9c0 msgr2=0x7f0764071570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:13.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.616+0000 7f076ae6d700 1 --2- 172.21.15.1:0/1132424600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640cc9c0 0x7f0764071570 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f07640afc30 tx=0x7f0750009b20 comp rx=0 tx=0).stop 2024-03-18T20:25:13.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.616+0000 7f076ae6d700 1 -- 172.21.15.1:0/1132424600 shutdown_connections 2024-03-18T20:25:13.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.616+0000 7f076ae6d700 1 --2- 172.21.15.1:0/1132424600 >> v2:172.21.15.165:3300/0 conn(0x7f076406a750 0x7f076406ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.616+0000 7f076ae6d700 1 --2- 172.21.15.1:0/1132424600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07640cc9c0 0x7f0764071570 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.616+0000 7f076ae6d700 1 -- 172.21.15.1:0/1132424600 >> 172.21.15.1:0/1132424600 conn(0x7f07640b1d20 msgr2=0x7f07640b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:13.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.616+0000 7f076ae6d700 1 -- 172.21.15.1:0/1132424600 shutdown_connections 2024-03-18T20:25:13.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.616+0000 7f076ae6d700 1 -- 172.21.15.1:0/1132424600 wait complete. 2024-03-18T20:25:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.617+0000 7f076ae6d700 1 Processor -- start 2024-03-18T20:25:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.618+0000 7f076ae6d700 1 -- start start 2024-03-18T20:25:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.618+0000 7f076ae6d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f076406a750 0x7f0764142140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.618+0000 7f076ae6d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07640cc9c0 0x7f0764134940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.618+0000 7f076ae6d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0764144ac0 0x7f076415ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.618+0000 7f076ae6d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07641347b0 con 0x7f0764144ac0 2024-03-18T20:25:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.618+0000 7f076ae6d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0764134e80 con 0x7f076406a750 2024-03-18T20:25:13.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.618+0000 7f076ae6d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07640a8540 con 0x7f07640cc9c0 2024-03-18T20:25:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.619+0000 7f076940a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0764144ac0 0x7f076415ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.619+0000 7f076940a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0764144ac0 0x7f076415ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60982/0 (socket says 172.21.15.1:60982) 2024-03-18T20:25:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.619+0000 7f076940a700 1 -- 172.21.15.1:0/929695237 learned_addr learned my addr 172.21.15.1:0/929695237 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.619+0000 7f0763fff700 1 --2- 172.21.15.1:0/929695237 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07640cc9c0 0x7f0764134940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.619+0000 7f076940a700 1 -- 172.21.15.1:0/929695237 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07640cc9c0 msgr2=0x7f0764134940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.619+0000 7f0768c09700 1 --2- 172.21.15.1:0/929695237 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f076406a750 0x7f0764142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.619+0000 7f076940a700 1 --2- 172.21.15.1:0/929695237 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07640cc9c0 0x7f0764134940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.619+0000 7f076940a700 1 -- 172.21.15.1:0/929695237 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f076406a750 msgr2=0x7f0764142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.619+0000 7f076940a700 1 --2- 172.21.15.1:0/929695237 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f076406a750 0x7f0764142140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.619+0000 7f076940a700 1 -- 172.21.15.1:0/929695237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07500094a0 con 0x7f0764144ac0 2024-03-18T20:25:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.619+0000 7f076940a700 1 --2- 172.21.15.1:0/929695237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0764144ac0 0x7f076415ad70 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f075c007e70 tx=0x7f075c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.619+0000 7f07617fa700 1 -- 172.21.15.1:0/929695237 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c00ee50 con 0x7f0764144ac0 2024-03-18T20:25:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.619+0000 7f07617fa700 1 -- 172.21.15.1:0/929695237 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c014420 con 0x7f0764144ac0 2024-03-18T20:25:13.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.619+0000 7f076ae6d700 1 -- 172.21.15.1:0/929695237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0764142720 con 0x7f0764144ac0 2024-03-18T20:25:13.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.619+0000 7f07617fa700 1 -- 172.21.15.1:0/929695237 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c0147d0 con 0x7f0764144ac0 2024-03-18T20:25:13.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.619+0000 7f076ae6d700 1 -- 172.21.15.1:0/929695237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0764142ce0 con 0x7f0764144ac0 2024-03-18T20:25:13.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.620+0000 7f076ae6d700 1 -- 172.21.15.1:0/929695237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07640676a0 con 0x7f0764144ac0 2024-03-18T20:25:13.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.621+0000 7f07617fa700 1 -- 172.21.15.1:0/929695237 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 134) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f075c014930 con 0x7f0764144ac0 2024-03-18T20:25:13.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.621+0000 7f07617fa700 1 --2- 172.21.15.1:0/929695237 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f074806ad20 0x7f074806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.621+0000 7f07617fa700 1 -- 172.21.15.1:0/929695237 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f075c089c00 con 0x7f0764144ac0 2024-03-18T20:25:13.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.621+0000 7f0768c09700 1 --2- 172.21.15.1:0/929695237 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f074806ad20 0x7f074806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.622+0000 7f0768c09700 1 --2- 172.21.15.1:0/929695237 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f074806ad20 0x7f074806d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0758000f60 tx=0x7f0758009040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:13.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.626+0000 7f07617fa700 1 -- 172.21.15.1:0/929695237 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f075c08f020 con 0x7f0764144ac0 2024-03-18T20:25:13.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.825+0000 7f076ae6d700 1 -- 172.21.15.1:0/929695237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f07640b2ae0 con 0x7f0764144ac0 2024-03-18T20:25:13.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.829+0000 7f07617fa700 1 -- 172.21.15.1:0/929695237 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v74) v1 ==== 120+0+0 (secure 0 0 0) 0x7f075c054d10 con 0x7f0764144ac0 2024-03-18T20:25:13.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.831+0000 7f076ae6d700 1 -- 172.21.15.1:0/929695237 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f074806ad20 msgr2=0x7f074806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:13.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.831+0000 7f076ae6d700 1 --2- 172.21.15.1:0/929695237 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f074806ad20 0x7f074806d1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0758000f60 tx=0x7f0758009040 comp rx=0 tx=0).stop 2024-03-18T20:25:13.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.831+0000 7f076ae6d700 1 -- 172.21.15.1:0/929695237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0764144ac0 msgr2=0x7f076415ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:13.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.831+0000 7f076ae6d700 1 --2- 172.21.15.1:0/929695237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0764144ac0 0x7f076415ad70 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f075c007e70 tx=0x7f075c00e3c0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.831+0000 7f076ae6d700 1 -- 172.21.15.1:0/929695237 shutdown_connections 2024-03-18T20:25:13.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.831+0000 7f076ae6d700 1 --2- 172.21.15.1:0/929695237 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f074806ad20 0x7f074806d1e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.831+0000 7f076ae6d700 1 --2- 172.21.15.1:0/929695237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0764144ac0 0x7f076415ad70 unknown :-1 s=CLOSED pgs=2464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.831+0000 7f076ae6d700 1 --2- 172.21.15.1:0/929695237 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f076406a750 0x7f0764142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.831+0000 7f076ae6d700 1 --2- 172.21.15.1:0/929695237 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07640cc9c0 0x7f0764134940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.831+0000 7f076ae6d700 1 -- 172.21.15.1:0/929695237 >> 172.21.15.1:0/929695237 conn(0x7f07640b1d20 msgr2=0x7f07640ce9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:13.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.831+0000 7f076ae6d700 1 -- 172.21.15.1:0/929695237 shutdown_connections 2024-03-18T20:25:13.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.831+0000 7f076ae6d700 1 -- 172.21.15.1:0/929695237 wait complete. 2024-03-18T20:25:13.848 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-18T20:25:13.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.971+0000 7f1d4aaf0700 1 Processor -- start 2024-03-18T20:25:13.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.971+0000 7f1d4aaf0700 1 -- start start 2024-03-18T20:25:13.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.971+0000 7f1d4aaf0700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1d4415d500 0x7f1d4406c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.971+0000 7f1d4aaf0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4406c7b0 0x7f1d44099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.971+0000 7f1d4aaf0700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d4406f490 con 0x7f1d440cb7b0 2024-03-18T20:25:13.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.971+0000 7f1d4aaf0700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d4406e880 con 0x7f1d4415d500 2024-03-18T20:25:13.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.971+0000 7f1d4aaf0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4406eb60 con 0x7f1d4406c7b0 2024-03-18T20:25:13.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d4908d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1d440cb7b0 0x7f1d440cccd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53114/0 (socket says 172.21.15.1:53114) 2024-03-18T20:25:13.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d4908d700 1 -- 172.21.15.1:0/1292273187 learned_addr learned my addr 172.21.15.1:0/1292273187 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:13.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d4888c700 1 --2- 172.21.15.1:0/1292273187 >> v2:172.21.15.165:3300/0 conn(0x7f1d4415d500 0x7f1d4406c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d43fff700 1 --2- 172.21.15.1:0/1292273187 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4406c7b0 0x7f1d44099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d437fe700 1 -- 172.21.15.1:0/1292273187 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 676930205 0 0) 0x7f1d4406f490 con 0x7f1d440cb7b0 2024-03-18T20:25:13.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d437fe700 1 -- 172.21.15.1:0/1292273187 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d340035b0 con 0x7f1d440cb7b0 2024-03-18T20:25:13.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d437fe700 1 -- 172.21.15.1:0/1292273187 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3523195542 0 0) 0x7f1d340035b0 con 0x7f1d440cb7b0 2024-03-18T20:25:13.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d437fe700 1 -- 172.21.15.1:0/1292273187 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4406c7b0 msgr2=0x7f1d44099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:13.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d437fe700 1 --2- 172.21.15.1:0/1292273187 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4406c7b0 0x7f1d44099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d437fe700 1 -- 172.21.15.1:0/1292273187 >> v2:172.21.15.165:3300/0 conn(0x7f1d4415d500 msgr2=0x7f1d4406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:13.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d437fe700 1 --2- 172.21.15.1:0/1292273187 >> v2:172.21.15.165:3300/0 conn(0x7f1d4415d500 0x7f1d4406c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d437fe700 1 -- 172.21.15.1:0/1292273187 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d4406ee40 con 0x7f1d440cb7b0 2024-03-18T20:25:13.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d437fe700 1 -- 172.21.15.1:0/1292273187 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f1d3c0019f0 con 0x7f1d440cb7b0 2024-03-18T20:25:13.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d437fe700 1 -- 172.21.15.1:0/1292273187 >> v1:172.21.15.1:6789/0 conn(0x7f1d440cb7b0 legacy=0x7f1d440cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:13.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d437fe700 1 --2- 172.21.15.1:0/1292273187 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d34003a00 0x7f1d34003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d437fe700 1 --2- 172.21.15.1:0/1292273187 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4406c7b0 0x7f1d34004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d437fe700 1 --2- 172.21.15.1:0/1292273187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d34004d90 0x7f1d34005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d437fe700 1 -- 172.21.15.1:0/1292273187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4406e880 con 0x7f1d34004d90 2024-03-18T20:25:13.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d437fe700 1 -- 172.21.15.1:0/1292273187 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4406eb60 con 0x7f1d34003a00 2024-03-18T20:25:13.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d437fe700 1 -- 172.21.15.1:0/1292273187 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3400ba10 con 0x7f1d4406c7b0 2024-03-18T20:25:13.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d4908d700 1 --2- 172.21.15.1:0/1292273187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d34004d90 0x7f1d34005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.972+0000 7f1d4888c700 1 --2- 172.21.15.1:0/1292273187 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d34003a00 0x7f1d34003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.973+0000 7f1d43fff700 1 --2- 172.21.15.1:0/1292273187 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4406c7b0 0x7f1d34004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.973+0000 7f1d4908d700 1 -- 172.21.15.1:0/1292273187 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4406c7b0 msgr2=0x7f1d34004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:13.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.973+0000 7f1d4908d700 1 --2- 172.21.15.1:0/1292273187 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4406c7b0 0x7f1d34004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.973+0000 7f1d4908d700 1 -- 172.21.15.1:0/1292273187 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d34003a00 msgr2=0x7f1d34003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:13.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.973+0000 7f1d4908d700 1 --2- 172.21.15.1:0/1292273187 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d34003a00 0x7f1d34003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.973+0000 7f1d4908d700 1 -- 172.21.15.1:0/1292273187 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d340035b0 con 0x7f1d34004d90 2024-03-18T20:25:13.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.973+0000 7f1d4908d700 1 --2- 172.21.15.1:0/1292273187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d34004d90 0x7f1d34005210 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c00a780 tx=0x7f1d34005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dde454197b0ccff7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:13.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.973+0000 7f1d437fe700 1 -- 172.21.15.1:0/1292273187 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c0072d0 con 0x7f1d34004d90 2024-03-18T20:25:13.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.973+0000 7f1d437fe700 1 -- 172.21.15.1:0/1292273187 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3c007430 con 0x7f1d34004d90 2024-03-18T20:25:13.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.973+0000 7f1d437fe700 1 -- 172.21.15.1:0/1292273187 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c0077e0 con 0x7f1d34004d90 2024-03-18T20:25:13.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.974+0000 7f1d4aaf0700 1 -- 172.21.15.1:0/1292273187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d34004d90 msgr2=0x7f1d34005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:13.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.974+0000 7f1d4aaf0700 1 --2- 172.21.15.1:0/1292273187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d34004d90 0x7f1d34005210 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c00a780 tx=0x7f1d34005880 comp rx=0 tx=0).stop 2024-03-18T20:25:13.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.974+0000 7f1d4908d700 1 -- 172.21.15.1:0/1292273187 reap_dead start 2024-03-18T20:25:13.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.974+0000 7f1d4aaf0700 1 -- 172.21.15.1:0/1292273187 shutdown_connections 2024-03-18T20:25:13.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.974+0000 7f1d4aaf0700 1 -- 172.21.15.1:0/1292273187 >> 172.21.15.1:0/1292273187 conn(0x7f1d440b2c50 msgr2=0x7f1d440b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:13.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.974+0000 7f1d4aaf0700 1 -- 172.21.15.1:0/1292273187 shutdown_connections 2024-03-18T20:25:13.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.974+0000 7f1d4aaf0700 1 -- 172.21.15.1:0/1292273187 wait complete. 2024-03-18T20:25:13.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.975+0000 7f1d4aaf0700 1 Processor -- start 2024-03-18T20:25:13.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.975+0000 7f1d4aaf0700 1 -- start start 2024-03-18T20:25:13.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.976+0000 7f1d4aaf0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d34004d90 0x7f1d4415da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.976+0000 7f1d4aaf0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d44147c80 0x7f1d44148100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.976+0000 7f1d4aaf0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4415f740 0x7f1d4415fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.976+0000 7f1d4aaf0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d440c9e60 con 0x7f1d34004d90 2024-03-18T20:25:13.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.976+0000 7f1d4aaf0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d440ca280 con 0x7f1d44147c80 2024-03-18T20:25:13.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.976+0000 7f1d4aaf0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d440c94c0 con 0x7f1d4415f740 2024-03-18T20:25:13.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.976+0000 7f1d4888c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d34004d90 0x7f1d4415da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.976+0000 7f1d4888c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d34004d90 0x7f1d4415da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32776/0 (socket says 172.21.15.1:32776) 2024-03-18T20:25:13.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.976+0000 7f1d4888c700 1 -- 172.21.15.1:0/533999553 learned_addr learned my addr 172.21.15.1:0/533999553 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:13.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.976+0000 7f1d4888c700 1 -- 172.21.15.1:0/533999553 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4415f740 msgr2=0x7f1d4415fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:13.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.976+0000 7f1d4888c700 1 --2- 172.21.15.1:0/533999553 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4415f740 0x7f1d4415fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.976+0000 7f1d4888c700 1 -- 172.21.15.1:0/533999553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d44147c80 msgr2=0x7f1d44148100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:13.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.976+0000 7f1d4888c700 1 --2- 172.21.15.1:0/533999553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d44147c80 0x7f1d44148100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:13.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.976+0000 7f1d4888c700 1 -- 172.21.15.1:0/533999553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d440c97a0 con 0x7f1d34004d90 2024-03-18T20:25:13.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.976+0000 7f1d4888c700 1 --2- 172.21.15.1:0/533999553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d34004d90 0x7f1d4415da60 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f1d38011a50 tx=0x7f1d38011e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:13.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.976+0000 7f1d417fa700 1 -- 172.21.15.1:0/533999553 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3800f940 con 0x7f1d34004d90 2024-03-18T20:25:13.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.977+0000 7f1d417fa700 1 -- 172.21.15.1:0/533999553 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3800fac0 con 0x7f1d34004d90 2024-03-18T20:25:13.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.977+0000 7f1d417fa700 1 -- 172.21.15.1:0/533999553 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3800fe70 con 0x7f1d34004d90 2024-03-18T20:25:13.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.977+0000 7f1d4aaf0700 1 -- 172.21.15.1:0/533999553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d440c9a80 con 0x7f1d34004d90 2024-03-18T20:25:13.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.977+0000 7f1d4aaf0700 1 -- 172.21.15.1:0/533999553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d4415dfa0 con 0x7f1d34004d90 2024-03-18T20:25:13.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.977+0000 7f1d4aaf0700 1 -- 172.21.15.1:0/533999553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d4406e5a0 con 0x7f1d34004d90 2024-03-18T20:25:13.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.978+0000 7f1d417fa700 1 -- 172.21.15.1:0/533999553 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 134) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f1d38014440 con 0x7f1d34004d90 2024-03-18T20:25:13.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.978+0000 7f1d417fa700 1 --2- 172.21.15.1:0/533999553 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1d2806ad20 0x7f1d2806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:13.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.978+0000 7f1d417fa700 1 -- 172.21.15.1:0/533999553 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f1d38089c80 con 0x7f1d34004d90 2024-03-18T20:25:13.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.978+0000 7f1d43fff700 1 --2- 172.21.15.1:0/533999553 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1d2806ad20 0x7f1d2806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:13.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.979+0000 7f1d43fff700 1 --2- 172.21.15.1:0/533999553 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1d2806ad20 0x7f1d2806d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1d30009b50 tx=0x7f1d30009670 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:13.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:13.983+0000 7f1d417fa700 1 -- 172.21.15.1:0/533999553 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1d38054d90 con 0x7f1d34004d90 2024-03-18T20:25:14.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.100+0000 7f1d417fa700 1 -- 172.21.15.1:0/533999553 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1d38055160 con 0x7f1d34004d90 2024-03-18T20:25:14.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.188+0000 7f1d4aaf0700 1 -- 172.21.15.1:0/533999553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1d440ca0d0 con 0x7f1d34004d90 2024-03-18T20:25:14.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.191+0000 7f1d417fa700 1 -- 172.21.15.1:0/533999553 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v75) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1d380890a0 con 0x7f1d34004d90 2024-03-18T20:25:14.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.194+0000 7f1d4aaf0700 1 -- 172.21.15.1:0/533999553 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1d2806ad20 msgr2=0x7f1d2806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:14.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.194+0000 7f1d4aaf0700 1 --2- 172.21.15.1:0/533999553 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1d2806ad20 0x7f1d2806d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1d30009b50 tx=0x7f1d30009670 comp rx=0 tx=0).stop 2024-03-18T20:25:14.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.194+0000 7f1d4aaf0700 1 -- 172.21.15.1:0/533999553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d34004d90 msgr2=0x7f1d4415da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:14.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.194+0000 7f1d4aaf0700 1 --2- 172.21.15.1:0/533999553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d34004d90 0x7f1d4415da60 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f1d38011a50 tx=0x7f1d38011e10 comp rx=0 tx=0).stop 2024-03-18T20:25:14.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.194+0000 7f1d4aaf0700 1 -- 172.21.15.1:0/533999553 shutdown_connections 2024-03-18T20:25:14.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.194+0000 7f1d4aaf0700 1 --2- 172.21.15.1:0/533999553 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1d2806ad20 0x7f1d2806d1e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.194+0000 7f1d4aaf0700 1 --2- 172.21.15.1:0/533999553 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1d4415f740 0x7f1d4415fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.194+0000 7f1d4aaf0700 1 --2- 172.21.15.1:0/533999553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d34004d90 0x7f1d4415da60 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.194+0000 7f1d4aaf0700 1 --2- 172.21.15.1:0/533999553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d44147c80 0x7f1d44148100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.194+0000 7f1d4aaf0700 1 -- 172.21.15.1:0/533999553 >> 172.21.15.1:0/533999553 conn(0x7f1d440b2c50 msgr2=0x7f1d440b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:14.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.194+0000 7f1d4aaf0700 1 -- 172.21.15.1:0/533999553 shutdown_connections 2024-03-18T20:25:14.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.194+0000 7f1d4aaf0700 1 -- 172.21.15.1:0/533999553 wait complete. 2024-03-18T20:25:14.211 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2024-03-18T20:25:14.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.332+0000 7f2cc981b700 1 Processor -- start 2024-03-18T20:25:14.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.332+0000 7f2cc981b700 1 -- start start 2024-03-18T20:25:14.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.333+0000 7f2cc981b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2cc40cf090 0x7f2cc40d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:14.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.333+0000 7f2cc981b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2cc4160d20 0x7f2cc40d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:14.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.333+0000 7f2cc981b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc40cddc0 con 0x7f2cc4160940 2024-03-18T20:25:14.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.333+0000 7f2cc981b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc40cdff0 con 0x7f2cc4160d20 2024-03-18T20:25:14.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.333+0000 7f2cc981b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc40ce2d0 con 0x7f2cc40cf090 2024-03-18T20:25:14.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.333+0000 7f2cc37fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2cc4160d20 0x7f2cc40d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:14.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.333+0000 7f2cc2ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2cc4160940 0x7f2cc40d0190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53118/0 (socket says 172.21.15.1:53118) 2024-03-18T20:25:14.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.333+0000 7f2cc2ffd700 1 -- 172.21.15.1:0/1032009126 learned_addr learned my addr 172.21.15.1:0/1032009126 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:14.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.333+0000 7f2cc27fc700 1 --2- 172.21.15.1:0/1032009126 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2cc40cf090 0x7f2cc40d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:14.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.333+0000 7f2cc37fe700 1 -- 172.21.15.1:0/1032009126 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2cc40cf090 msgr2=0x7f2cc40d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:14.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.333+0000 7f2cc37fe700 1 --2- 172.21.15.1:0/1032009126 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2cc40cf090 0x7f2cc40d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.333+0000 7f2cc37fe700 1 -- 172.21.15.1:0/1032009126 >> v1:172.21.15.1:6789/0 conn(0x7f2cc4160940 legacy=0x7f2cc40d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:14.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.333+0000 7f2cc37fe700 1 -- 172.21.15.1:0/1032009126 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc40ce5b0 con 0x7f2cc4160d20 2024-03-18T20:25:14.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.333+0000 7f2cc37fe700 1 --2- 172.21.15.1:0/1032009126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc4160d20 0x7f2cc40d1cd0 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc009d90 tx=0x7f2cbc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd576cc177ef43d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:14.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.333+0000 7f2cc1ffb700 1 -- 172.21.15.1:0/1032009126 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc00e070 con 0x7f2cc4160d20 2024-03-18T20:25:14.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.333+0000 7f2cc1ffb700 1 -- 172.21.15.1:0/1032009126 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cbc00bbd0 con 0x7f2cc4160d20 2024-03-18T20:25:14.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.333+0000 7f2cc1ffb700 1 -- 172.21.15.1:0/1032009126 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc012430 con 0x7f2cc4160d20 2024-03-18T20:25:14.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.334+0000 7f2cc981b700 1 -- 172.21.15.1:0/1032009126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc4160d20 msgr2=0x7f2cc40d1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:14.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.334+0000 7f2cc981b700 1 --2- 172.21.15.1:0/1032009126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc4160d20 0x7f2cc40d1cd0 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc009d90 tx=0x7f2cbc00b330 comp rx=0 tx=0).stop 2024-03-18T20:25:14.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.334+0000 7f2cc981b700 1 -- 172.21.15.1:0/1032009126 shutdown_connections 2024-03-18T20:25:14.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.334+0000 7f2cc981b700 1 --2- 172.21.15.1:0/1032009126 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc4160d20 0x7f2cc40d1cd0 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.334+0000 7f2cc981b700 1 --2- 172.21.15.1:0/1032009126 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2cc40cf090 0x7f2cc40d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.334+0000 7f2cc981b700 1 -- 172.21.15.1:0/1032009126 >> 172.21.15.1:0/1032009126 conn(0x7f2cc409c520 msgr2=0x7f2cc40d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:14.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.334+0000 7f2cc981b700 1 -- 172.21.15.1:0/1032009126 shutdown_connections 2024-03-18T20:25:14.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.334+0000 7f2cc981b700 1 -- 172.21.15.1:0/1032009126 wait complete. 2024-03-18T20:25:14.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.335+0000 7f2cc981b700 1 Processor -- start 2024-03-18T20:25:14.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.335+0000 7f2cc981b700 1 -- start start 2024-03-18T20:25:14.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.336+0000 7f2cc981b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2cc40cf090 0x7f2cc40c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:14.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.336+0000 7f2cc981b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cc4160940 0x7f2cc414e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:14.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.336+0000 7f2cc981b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc4160d20 0x7f2cc4152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:14.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.336+0000 7f2cc981b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc40b7110 con 0x7f2cc4160d20 2024-03-18T20:25:14.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.336+0000 7f2cc981b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc4160170 con 0x7f2cc4160940 2024-03-18T20:25:14.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.336+0000 7f2cc981b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc414ea10 con 0x7f2cc40cf090 2024-03-18T20:25:14.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.336+0000 7f2cc37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc4160d20 0x7f2cc4152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:14.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.336+0000 7f2cc37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc4160d20 0x7f2cc4152f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32790/0 (socket says 172.21.15.1:32790) 2024-03-18T20:25:14.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.336+0000 7f2cc37fe700 1 -- 172.21.15.1:0/1536513572 learned_addr learned my addr 172.21.15.1:0/1536513572 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:14.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.336+0000 7f2cc2ffd700 1 --2- 172.21.15.1:0/1536513572 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2cc40cf090 0x7f2cc40c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:14.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.336+0000 7f2cc27fc700 1 --2- 172.21.15.1:0/1536513572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cc4160940 0x7f2cc414e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:14.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.336+0000 7f2cc37fe700 1 -- 172.21.15.1:0/1536513572 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2cc40cf090 msgr2=0x7f2cc40c6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:14.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.336+0000 7f2cc37fe700 1 --2- 172.21.15.1:0/1536513572 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2cc40cf090 0x7f2cc40c6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.336+0000 7f2cc37fe700 1 -- 172.21.15.1:0/1536513572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cc4160940 msgr2=0x7f2cc414e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:14.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.336+0000 7f2cc37fe700 1 --2- 172.21.15.1:0/1536513572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cc4160940 0x7f2cc414e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:14.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.336+0000 7f2cc37fe700 1 -- 172.21.15.1:0/1536513572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cbc009a70 con 0x7f2cc4160d20 2024-03-18T20:25:14.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.336+0000 7f2cc27fc700 1 --2- 172.21.15.1:0/1536513572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cc4160940 0x7f2cc414e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:14.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.337+0000 7f2cc37fe700 1 --2- 172.21.15.1:0/1536513572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc4160d20 0x7f2cc4152f70 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc014040 tx=0x7f2cbc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:14.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.337+0000 7f2cabfff700 1 -- 172.21.15.1:0/1536513572 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc003ea0 con 0x7f2cc4160d20 2024-03-18T20:25:14.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.337+0000 7f2cabfff700 1 -- 172.21.15.1:0/1536513572 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cbc020070 con 0x7f2cc4160d20 2024-03-18T20:25:14.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.337+0000 7f2cc981b700 1 -- 172.21.15.1:0/1536513572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc40c7370 con 0x7f2cc4160d20 2024-03-18T20:25:14.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.337+0000 7f2cabfff700 1 -- 172.21.15.1:0/1536513572 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc004280 con 0x7f2cc4160d20 2024-03-18T20:25:14.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.337+0000 7f2cc981b700 1 -- 172.21.15.1:0/1536513572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc40b6ac0 con 0x7f2cc4160d20 2024-03-18T20:25:14.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.338+0000 7f2cc981b700 1 -- 172.21.15.1:0/1536513572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc4073350 con 0x7f2cc4160d20 2024-03-18T20:25:14.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.338+0000 7f2cabfff700 1 -- 172.21.15.1:0/1536513572 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2cbc0043e0 con 0x7f2cc4160d20 2024-03-18T20:25:14.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.339+0000 7f2cabfff700 1 --2- 172.21.15.1:0/1536513572 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2cb006adb0 0x7f2cb006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:14.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.339+0000 7f2cc2ffd700 1 --2- 172.21.15.1:0/1536513572 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2cb006adb0 0x7f2cb006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:14.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.339+0000 7f2cabfff700 1 -- 172.21.15.1:0/1536513572 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f2cbc022070 con 0x7f2cc4160d20 2024-03-18T20:25:14.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.339+0000 7f2cc2ffd700 1 --2- 172.21.15.1:0/1536513572 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2cb006adb0 0x7f2cb006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4005910 tx=0x7f2cb4009040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:14.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.343+0000 7f2cabfff700 1 -- 172.21.15.1:0/1536513572 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2cbc0566f0 con 0x7f2cc4160d20 2024-03-18T20:25:14.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:14.630+0000 7f2cc981b700 1 -- 172.21.15.1:0/1536513572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f2cc40b7250 con 0x7f2cc4160d20 2024-03-18T20:25:15.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.206+0000 7f2cabfff700 1 -- 172.21.15.1:0/1536513572 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3103) v1 ==== 163+0+0 (secure 0 0 0) 0x7f2cbc059d10 con 0x7f2cc4160d20 2024-03-18T20:25:15.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.209+0000 7f2cc981b700 1 -- 172.21.15.1:0/1536513572 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2cb006adb0 msgr2=0x7f2cb006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:15.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.209+0000 7f2cc981b700 1 --2- 172.21.15.1:0/1536513572 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2cb006adb0 0x7f2cb006d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4005910 tx=0x7f2cb4009040 comp rx=0 tx=0).stop 2024-03-18T20:25:15.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.209+0000 7f2cc981b700 1 -- 172.21.15.1:0/1536513572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc4160d20 msgr2=0x7f2cc4152f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:15.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.209+0000 7f2cc981b700 1 --2- 172.21.15.1:0/1536513572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc4160d20 0x7f2cc4152f70 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc014040 tx=0x7f2cbc004ad0 comp rx=0 tx=0).stop 2024-03-18T20:25:15.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.209+0000 7f2cc981b700 1 -- 172.21.15.1:0/1536513572 shutdown_connections 2024-03-18T20:25:15.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.209+0000 7f2cc981b700 1 --2- 172.21.15.1:0/1536513572 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2cb006adb0 0x7f2cb006d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:15.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.209+0000 7f2cc981b700 1 --2- 172.21.15.1:0/1536513572 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2cc40cf090 0x7f2cc40c6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:15.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.209+0000 7f2cc981b700 1 --2- 172.21.15.1:0/1536513572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc4160d20 0x7f2cc4152f70 unknown :-1 s=CLOSED pgs=2470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:15.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.209+0000 7f2cc981b700 1 --2- 172.21.15.1:0/1536513572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cc4160940 0x7f2cc414e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:15.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.209+0000 7f2cc981b700 1 -- 172.21.15.1:0/1536513572 >> 172.21.15.1:0/1536513572 conn(0x7f2cc409c520 msgr2=0x7f2cc40b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:15.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.209+0000 7f2cc981b700 1 -- 172.21.15.1:0/1536513572 shutdown_connections 2024-03-18T20:25:15.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.209+0000 7f2cc981b700 1 -- 172.21.15.1:0/1536513572 wait complete. 2024-03-18T20:25:15.228 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... ok 2024-03-18T20:25:15.228 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command succes with --fs argument for non-default fs 2024-03-18T20:25:15.228 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2024-03-18T20:25:15.228 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2024-03-18T20:25:15.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.357+0000 7fbf67236700 1 Processor -- start 2024-03-18T20:25:15.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.357+0000 7fbf67236700 1 -- start start 2024-03-18T20:25:15.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.358+0000 7fbf67236700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf60146190 0x7fbf6006a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:15.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.358+0000 7fbf67236700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6006ebd0 con 0x7fbf60146af0 2024-03-18T20:25:15.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.358+0000 7fbf67236700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6006eeb0 con 0x7fbf601465a0 2024-03-18T20:25:15.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.358+0000 7fbf67236700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf6006f850 con 0x7fbf60146190 2024-03-18T20:25:15.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.358+0000 7fbf64fd2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf60146190 0x7fbf6006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:15.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.358+0000 7fbf64fd2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf60146190 0x7fbf6006a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34136/0 (socket says 172.21.15.1:34136) 2024-03-18T20:25:15.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.358+0000 7fbf64fd2700 1 -- 172.21.15.1:0/3058258186 learned_addr learned my addr 172.21.15.1:0/3058258186 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:15.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.358+0000 7fbf657d3700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbf60146af0 0x7fbf60071d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35102/0 (socket says 172.21.15.1:35102) 2024-03-18T20:25:15.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.358+0000 7fbf64fd2700 1 -- 172.21.15.1:0/3058258186 >> v1:172.21.15.165:6789/0 conn(0x7fbf601465a0 legacy=0x7fbf60071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:15.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.358+0000 7fbf64fd2700 1 -- 172.21.15.1:0/3058258186 >> v1:172.21.15.1:6789/0 conn(0x7fbf60146af0 legacy=0x7fbf60071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:15.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.358+0000 7fbf64fd2700 1 -- 172.21.15.1:0/3058258186 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6006f190 con 0x7fbf60146190 2024-03-18T20:25:15.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.358+0000 7fbf64fd2700 1 --2- 172.21.15.1:0/3058258186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf60146190 0x7fbf6006a750 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7fbf540098f0 tx=0x7fbf54009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c6156e463c882a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:15.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.359+0000 7fbf5f7fe700 1 -- 172.21.15.1:0/3058258186 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5400b8a0 con 0x7fbf60146190 2024-03-18T20:25:15.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.359+0000 7fbf5f7fe700 1 -- 172.21.15.1:0/3058258186 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5400ba00 con 0x7fbf60146190 2024-03-18T20:25:15.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.359+0000 7fbf5f7fe700 1 -- 172.21.15.1:0/3058258186 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5400bd90 con 0x7fbf60146190 2024-03-18T20:25:15.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.359+0000 7fbf67236700 1 -- 172.21.15.1:0/3058258186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf60146190 msgr2=0x7fbf6006a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:15.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.359+0000 7fbf67236700 1 --2- 172.21.15.1:0/3058258186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf60146190 0x7fbf6006a750 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7fbf540098f0 tx=0x7fbf54009c00 comp rx=0 tx=0).stop 2024-03-18T20:25:15.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.359+0000 7fbf67236700 1 -- 172.21.15.1:0/3058258186 shutdown_connections 2024-03-18T20:25:15.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.359+0000 7fbf67236700 1 --2- 172.21.15.1:0/3058258186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf60146190 0x7fbf6006a750 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:15.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.359+0000 7fbf67236700 1 -- 172.21.15.1:0/3058258186 >> 172.21.15.1:0/3058258186 conn(0x7fbf600b1d20 msgr2=0x7fbf600b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:15.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.359+0000 7fbf67236700 1 -- 172.21.15.1:0/3058258186 shutdown_connections 2024-03-18T20:25:15.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.359+0000 7fbf67236700 1 -- 172.21.15.1:0/3058258186 wait complete. 2024-03-18T20:25:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.360+0000 7fbf67236700 1 Processor -- start 2024-03-18T20:25:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.360+0000 7fbf67236700 1 -- start start 2024-03-18T20:25:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.361+0000 7fbf67236700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf601465a0 0x7fbf6013ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.361+0000 7fbf67236700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf60146af0 0x7fbf600ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.361+0000 7fbf67236700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf600ca860 0x7fbf600c5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.361+0000 7fbf67236700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600cad10 con 0x7fbf60146af0 2024-03-18T20:25:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.361+0000 7fbf67236700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6013f010 con 0x7fbf600ca860 2024-03-18T20:25:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.361+0000 7fbf67236700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf600c79a0 con 0x7fbf601465a0 2024-03-18T20:25:15.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.361+0000 7fbf5ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf60146af0 0x7fbf600ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:15.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.361+0000 7fbf5ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf60146af0 0x7fbf600ca320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34142/0 (socket says 172.21.15.1:34142) 2024-03-18T20:25:15.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.361+0000 7fbf5ffff700 1 -- 172.21.15.1:0/976022563 learned_addr learned my addr 172.21.15.1:0/976022563 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:15.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.362+0000 7fbf64fd2700 1 --2- 172.21.15.1:0/976022563 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf601465a0 0x7fbf6013ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:15.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.362+0000 7fbf657d3700 1 --2- 172.21.15.1:0/976022563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf600ca860 0x7fbf600c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:15.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.362+0000 7fbf5ffff700 1 -- 172.21.15.1:0/976022563 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf601465a0 msgr2=0x7fbf6013ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:15.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.362+0000 7fbf5ffff700 1 --2- 172.21.15.1:0/976022563 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf601465a0 0x7fbf6013ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:15.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.362+0000 7fbf5ffff700 1 -- 172.21.15.1:0/976022563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf600ca860 msgr2=0x7fbf600c5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:15.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.362+0000 7fbf5ffff700 1 --2- 172.21.15.1:0/976022563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf600ca860 0x7fbf600c5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:15.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.362+0000 7fbf5ffff700 1 -- 172.21.15.1:0/976022563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf54009550 con 0x7fbf60146af0 2024-03-18T20:25:15.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.362+0000 7fbf657d3700 1 --2- 172.21.15.1:0/976022563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf600ca860 0x7fbf600c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:15.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.362+0000 7fbf5ffff700 1 --2- 172.21.15.1:0/976022563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf60146af0 0x7fbf600ca320 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c00c4e0 tx=0x7fbf4c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:15.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.362+0000 7fbf5d7fa700 1 -- 172.21.15.1:0/976022563 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c0052d0 con 0x7fbf60146af0 2024-03-18T20:25:15.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.362+0000 7fbf5d7fa700 1 -- 172.21.15.1:0/976022563 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4c005430 con 0x7fbf60146af0 2024-03-18T20:25:15.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.362+0000 7fbf67236700 1 -- 172.21.15.1:0/976022563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf600c5ec0 con 0x7fbf60146af0 2024-03-18T20:25:15.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.362+0000 7fbf5d7fa700 1 -- 172.21.15.1:0/976022563 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c0038c0 con 0x7fbf60146af0 2024-03-18T20:25:15.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.362+0000 7fbf67236700 1 -- 172.21.15.1:0/976022563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf600c6480 con 0x7fbf60146af0 2024-03-18T20:25:15.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.363+0000 7fbf67236700 1 -- 172.21.15.1:0/976022563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf600c7b50 con 0x7fbf60146af0 2024-03-18T20:25:15.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.364+0000 7fbf5d7fa700 1 -- 172.21.15.1:0/976022563 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbf4c003a20 con 0x7fbf60146af0 2024-03-18T20:25:15.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.365+0000 7fbf5d7fa700 1 --2- 172.21.15.1:0/976022563 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf4406adb0 0x7fbf4406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:15.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.365+0000 7fbf5d7fa700 1 -- 172.21.15.1:0/976022563 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6079+0+0 (secure 0 0 0) 0x7fbf4c088b80 con 0x7fbf60146af0 2024-03-18T20:25:15.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.365+0000 7fbf64fd2700 1 --2- 172.21.15.1:0/976022563 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf4406adb0 0x7fbf4406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:15.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.365+0000 7fbf64fd2700 1 --2- 172.21.15.1:0/976022563 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf4406adb0 0x7fbf4406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbf54014040 tx=0x7fbf54008030 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:15.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.368+0000 7fbf5d7fa700 1 -- 172.21.15.1:0/976022563 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbf4c053c50 con 0x7fbf60146af0 2024-03-18T20:25:15.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:15.659+0000 7fbf67236700 1 -- 172.21.15.1:0/976022563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7fbf600b62e0 con 0x7fbf60146af0 2024-03-18T20:25:16.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.208+0000 7fbf5d7fa700 1 -- 172.21.15.1:0/976022563 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3104) v1 ==== 189+0+0 (secure 0 0 0) 0x7fbf4c057270 con 0x7fbf60146af0 2024-03-18T20:25:16.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.211+0000 7fbf67236700 1 -- 172.21.15.1:0/976022563 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf4406adb0 msgr2=0x7fbf4406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:16.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.211+0000 7fbf67236700 1 --2- 172.21.15.1:0/976022563 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf4406adb0 0x7fbf4406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbf54014040 tx=0x7fbf54008030 comp rx=0 tx=0).stop 2024-03-18T20:25:16.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.211+0000 7fbf67236700 1 -- 172.21.15.1:0/976022563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf60146af0 msgr2=0x7fbf600ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:16.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.211+0000 7fbf67236700 1 --2- 172.21.15.1:0/976022563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf60146af0 0x7fbf600ca320 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c00c4e0 tx=0x7fbf4c00c8a0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.211+0000 7fbf67236700 1 -- 172.21.15.1:0/976022563 shutdown_connections 2024-03-18T20:25:16.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.211+0000 7fbf67236700 1 --2- 172.21.15.1:0/976022563 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf4406adb0 0x7fbf4406d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.211+0000 7fbf67236700 1 --2- 172.21.15.1:0/976022563 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf601465a0 0x7fbf6013ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.211+0000 7fbf67236700 1 --2- 172.21.15.1:0/976022563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf600ca860 0x7fbf600c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.211+0000 7fbf67236700 1 --2- 172.21.15.1:0/976022563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf60146af0 0x7fbf600ca320 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.211+0000 7fbf67236700 1 -- 172.21.15.1:0/976022563 >> 172.21.15.1:0/976022563 conn(0x7fbf600b1d20 msgr2=0x7fbf60144c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:16.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.211+0000 7fbf67236700 1 -- 172.21.15.1:0/976022563 shutdown_connections 2024-03-18T20:25:16.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.211+0000 7fbf67236700 1 -- 172.21.15.1:0/976022563 wait complete. 2024-03-18T20:25:16.230 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-18T20:25:16.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.361+0000 7f6fa12b7700 1 Processor -- start 2024-03-18T20:25:16.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.361+0000 7f6fa12b7700 1 -- start start 2024-03-18T20:25:16.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.362+0000 7f6fa12b7700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6f9c071d30 0x7f6f9c0add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:16.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.362+0000 7f6fa12b7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6f9c0cb9e0 0x7f6f9c0cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:16.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.362+0000 7f6fa12b7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f9c0ce8f0 con 0x7f6f9c15a1d0 2024-03-18T20:25:16.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.362+0000 7f6fa12b7700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f9c0cdb00 con 0x7f6f9c071d30 2024-03-18T20:25:16.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.362+0000 7f6fa12b7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c15a5b0 con 0x7f6f9c0cb9e0 2024-03-18T20:25:16.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.362+0000 7f6f9affd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6f9c15a1d0 0x7f6f9c071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35104/0 (socket says 172.21.15.1:35104) 2024-03-18T20:25:16.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.362+0000 7f6f9affd700 1 -- 172.21.15.1:0/3370663682 learned_addr learned my addr 172.21.15.1:0/3370663682 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:16.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f9b7fe700 1 --2- 172.21.15.1:0/3370663682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6f9c0cb9e0 0x7f6f9c0cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:16.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f9a7fc700 1 --2- 172.21.15.1:0/3370663682 >> v2:172.21.15.165:3300/0 conn(0x7f6f9c071d30 0x7f6f9c0add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:16.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f99ffb700 1 -- 172.21.15.1:0/3370663682 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093692950 0 0) 0x7f6f9c0ce8f0 con 0x7f6f9c15a1d0 2024-03-18T20:25:16.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f99ffb700 1 -- 172.21.15.1:0/3370663682 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f840035b0 con 0x7f6f9c15a1d0 2024-03-18T20:25:16.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f99ffb700 1 -- 172.21.15.1:0/3370663682 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 924997632 0 0) 0x7f6f840035b0 con 0x7f6f9c15a1d0 2024-03-18T20:25:16.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f99ffb700 1 -- 172.21.15.1:0/3370663682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6f9c0cb9e0 msgr2=0x7f6f9c0cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:16.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f99ffb700 1 --2- 172.21.15.1:0/3370663682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6f9c0cb9e0 0x7f6f9c0cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f99ffb700 1 -- 172.21.15.1:0/3370663682 >> v2:172.21.15.165:3300/0 conn(0x7f6f9c071d30 msgr2=0x7f6f9c0add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:16.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f99ffb700 1 --2- 172.21.15.1:0/3370663682 >> v2:172.21.15.165:3300/0 conn(0x7f6f9c071d30 0x7f6f9c0add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f99ffb700 1 -- 172.21.15.1:0/3370663682 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9c0ae260 con 0x7f6f9c15a1d0 2024-03-18T20:25:16.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f99ffb700 1 -- 172.21.15.1:0/3370663682 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f6f900017f0 con 0x7f6f9c15a1d0 2024-03-18T20:25:16.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f99ffb700 1 -- 172.21.15.1:0/3370663682 >> v1:172.21.15.1:6789/0 conn(0x7f6f9c15a1d0 legacy=0x7f6f9c071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:16.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f99ffb700 1 --2- 172.21.15.1:0/3370663682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6f9c0cb9e0 0x7f6f84003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:16.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f99ffb700 1 --2- 172.21.15.1:0/3370663682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f84003f80 0x7f6f84004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:16.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f99ffb700 1 --2- 172.21.15.1:0/3370663682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f84004d90 0x7f6f84005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:16.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f99ffb700 1 -- 172.21.15.1:0/3370663682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c0cdb00 con 0x7f6f84003f80 2024-03-18T20:25:16.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f99ffb700 1 -- 172.21.15.1:0/3370663682 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8400ba10 con 0x7f6f84004d90 2024-03-18T20:25:16.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f99ffb700 1 -- 172.21.15.1:0/3370663682 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8400bb50 con 0x7f6f9c0cb9e0 2024-03-18T20:25:16.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f9a7fc700 1 --2- 172.21.15.1:0/3370663682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f84003f80 0x7f6f84004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:16.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f9affd700 1 --2- 172.21.15.1:0/3370663682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6f9c0cb9e0 0x7f6f84003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:16.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f9b7fe700 1 --2- 172.21.15.1:0/3370663682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f84004d90 0x7f6f84005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:16.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f9a7fc700 1 -- 172.21.15.1:0/3370663682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6f9c0cb9e0 msgr2=0x7f6f84003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:16.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f9a7fc700 1 --2- 172.21.15.1:0/3370663682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6f9c0cb9e0 0x7f6f84003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f9a7fc700 1 -- 172.21.15.1:0/3370663682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f84004d90 msgr2=0x7f6f84005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:16.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f9a7fc700 1 --2- 172.21.15.1:0/3370663682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f84004d90 0x7f6f84005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f9a7fc700 1 -- 172.21.15.1:0/3370663682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f840035b0 con 0x7f6f84003f80 2024-03-18T20:25:16.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.363+0000 7f6f9b7fe700 1 --2- 172.21.15.1:0/3370663682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f84004d90 0x7f6f84005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:16.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.364+0000 7f6f9a7fc700 1 --2- 172.21.15.1:0/3370663682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f84003f80 0x7f6f84004850 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c009b60 tx=0x7f6f84005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dbb9e0fba1e5130 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:16.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.364+0000 7f6f99ffb700 1 -- 172.21.15.1:0/3370663682 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c00e660 con 0x7f6f84003f80 2024-03-18T20:25:16.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.364+0000 7f6f99ffb700 1 -- 172.21.15.1:0/3370663682 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8c00e7c0 con 0x7f6f84003f80 2024-03-18T20:25:16.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.364+0000 7f6f99ffb700 1 -- 172.21.15.1:0/3370663682 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c00eb30 con 0x7f6f84003f80 2024-03-18T20:25:16.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.364+0000 7f6fa12b7700 1 -- 172.21.15.1:0/3370663682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f84003f80 msgr2=0x7f6f84004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:16.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.364+0000 7f6fa12b7700 1 --2- 172.21.15.1:0/3370663682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f84003f80 0x7f6f84004850 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c009b60 tx=0x7f6f84005880 comp rx=0 tx=0).stop 2024-03-18T20:25:16.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.364+0000 7f6f9b7fe700 1 -- 172.21.15.1:0/3370663682 reap_dead start 2024-03-18T20:25:16.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.364+0000 7f6fa12b7700 1 -- 172.21.15.1:0/3370663682 shutdown_connections 2024-03-18T20:25:16.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.364+0000 7f6fa12b7700 1 -- 172.21.15.1:0/3370663682 >> 172.21.15.1:0/3370663682 conn(0x7f6f9c0cc9b0 msgr2=0x7f6f9c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:16.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.364+0000 7f6fa12b7700 1 -- 172.21.15.1:0/3370663682 shutdown_connections 2024-03-18T20:25:16.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.364+0000 7f6fa12b7700 1 -- 172.21.15.1:0/3370663682 wait complete. 2024-03-18T20:25:16.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.366+0000 7f6fa12b7700 1 Processor -- start 2024-03-18T20:25:16.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.366+0000 7f6fa12b7700 1 -- start start 2024-03-18T20:25:16.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.367+0000 7f6fa12b7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6f84003f80 0x7f6f9c158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:16.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.367+0000 7f6fa12b7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f9c09d280 0x7f6f9c0c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:16.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.367+0000 7f6fa12b7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f9c0c91d0 0x7f6f9c0c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:16.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.367+0000 7f6fa12b7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c1f38c0 con 0x7f6f9c09d280 2024-03-18T20:25:16.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.367+0000 7f6fa12b7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c0c9b90 con 0x7f6f9c0c91d0 2024-03-18T20:25:16.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.367+0000 7f6fa12b7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c09d9d0 con 0x7f6f84003f80 2024-03-18T20:25:16.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.367+0000 7f6f9a7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f9c09d280 0x7f6f9c0c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:16.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.367+0000 7f6f9a7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f9c09d280 0x7f6f9c0c6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34148/0 (socket says 172.21.15.1:34148) 2024-03-18T20:25:16.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.367+0000 7f6f9a7fc700 1 -- 172.21.15.1:0/819142937 learned_addr learned my addr 172.21.15.1:0/819142937 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:16.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.367+0000 7f6f9affd700 1 --2- 172.21.15.1:0/819142937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6f84003f80 0x7f6f9c158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:16.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.367+0000 7f6f9b7fe700 1 --2- 172.21.15.1:0/819142937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f9c0c91d0 0x7f6f9c0c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:16.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.367+0000 7f6f9a7fc700 1 -- 172.21.15.1:0/819142937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6f84003f80 msgr2=0x7f6f9c158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:16.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.367+0000 7f6f9a7fc700 1 --2- 172.21.15.1:0/819142937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6f84003f80 0x7f6f9c158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.367+0000 7f6f9a7fc700 1 -- 172.21.15.1:0/819142937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f9c0c91d0 msgr2=0x7f6f9c0c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:16.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.367+0000 7f6f9a7fc700 1 --2- 172.21.15.1:0/819142937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f9c0c91d0 0x7f6f9c0c9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.367+0000 7f6f9a7fc700 1 -- 172.21.15.1:0/819142937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9c09d700 con 0x7f6f9c09d280 2024-03-18T20:25:16.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.367+0000 7f6f9affd700 1 --2- 172.21.15.1:0/819142937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6f84003f80 0x7f6f9c158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:16.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.368+0000 7f6f9a7fc700 1 --2- 172.21.15.1:0/819142937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f9c09d280 0x7f6f9c0c6860 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c0090b0 tx=0x7f6f8c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:16.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.368+0000 7f6f837fe700 1 -- 172.21.15.1:0/819142937 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c021070 con 0x7f6f9c09d280 2024-03-18T20:25:16.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.368+0000 7f6f837fe700 1 -- 172.21.15.1:0/819142937 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8c0045d0 con 0x7f6f9c09d280 2024-03-18T20:25:16.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.368+0000 7f6fa12b7700 1 -- 172.21.15.1:0/819142937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f9c1f2f20 con 0x7f6f9c09d280 2024-03-18T20:25:16.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.368+0000 7f6f837fe700 1 -- 172.21.15.1:0/819142937 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c00e660 con 0x7f6f9c09d280 2024-03-18T20:25:16.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.368+0000 7f6fa12b7700 1 -- 172.21.15.1:0/819142937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f9c1f34e0 con 0x7f6f9c09d280 2024-03-18T20:25:16.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.369+0000 7f6fa12b7700 1 -- 172.21.15.1:0/819142937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f9c0676a0 con 0x7f6f9c09d280 2024-03-18T20:25:16.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.369+0000 7f6f837fe700 1 -- 172.21.15.1:0/819142937 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6f8c025070 con 0x7f6f9c09d280 2024-03-18T20:25:16.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.370+0000 7f6f837fe700 1 --2- 172.21.15.1:0/819142937 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6f8806adb0 0x7f6f8806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:16.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.370+0000 7f6f837fe700 1 -- 172.21.15.1:0/819142937 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f6f8c08a860 con 0x7f6f9c09d280 2024-03-18T20:25:16.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.370+0000 7f6f9affd700 1 --2- 172.21.15.1:0/819142937 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6f8806adb0 0x7f6f8806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:16.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.370+0000 7f6f9affd700 1 --2- 172.21.15.1:0/819142937 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6f8806adb0 0x7f6f8806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6f90000ef0 tx=0x7f6f90009040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:16.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.374+0000 7f6f837fe700 1 -- 172.21.15.1:0/819142937 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6f8c0558f0 con 0x7f6f9c09d280 2024-03-18T20:25:16.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.579+0000 7f6fa12b7700 1 -- 172.21.15.1:0/819142937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6f9c0cb590 con 0x7f6f9c09d280 2024-03-18T20:25:16.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.582+0000 7f6f837fe700 1 -- 172.21.15.1:0/819142937 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v76)=0 v76) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6f8c02c0c0 con 0x7f6f9c09d280 2024-03-18T20:25:16.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.584+0000 7f6fa12b7700 1 -- 172.21.15.1:0/819142937 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6f8806adb0 msgr2=0x7f6f8806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:16.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.584+0000 7f6fa12b7700 1 --2- 172.21.15.1:0/819142937 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6f8806adb0 0x7f6f8806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6f90000ef0 tx=0x7f6f90009040 comp rx=0 tx=0).stop 2024-03-18T20:25:16.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.584+0000 7f6fa12b7700 1 -- 172.21.15.1:0/819142937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f9c09d280 msgr2=0x7f6f9c0c6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:16.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.584+0000 7f6fa12b7700 1 --2- 172.21.15.1:0/819142937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f9c09d280 0x7f6f9c0c6860 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c0090b0 tx=0x7f6f8c0047c0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.584+0000 7f6fa12b7700 1 -- 172.21.15.1:0/819142937 shutdown_connections 2024-03-18T20:25:16.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.584+0000 7f6fa12b7700 1 --2- 172.21.15.1:0/819142937 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6f8806adb0 0x7f6f8806d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.584+0000 7f6fa12b7700 1 --2- 172.21.15.1:0/819142937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6f84003f80 0x7f6f9c158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.584+0000 7f6fa12b7700 1 --2- 172.21.15.1:0/819142937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f9c0c91d0 0x7f6f9c0c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.584+0000 7f6fa12b7700 1 --2- 172.21.15.1:0/819142937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6f9c09d280 0x7f6f9c0c6860 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.584+0000 7f6fa12b7700 1 -- 172.21.15.1:0/819142937 >> 172.21.15.1:0/819142937 conn(0x7f6f9c0cc9b0 msgr2=0x7f6f9c0cadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:16.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.584+0000 7f6fa12b7700 1 -- 172.21.15.1:0/819142937 shutdown_connections 2024-03-18T20:25:16.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.584+0000 7f6fa12b7700 1 -- 172.21.15.1:0/819142937 wait complete. 2024-03-18T20:25:16.601 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T20:25:16.601 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:25:16.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:25:16.693 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T20:25:16.693 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:25:16.693 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:25:16.712 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:25:16.713 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:25:16.713 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:25:16.734 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:25:16.734 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:25:16.734 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:25:16.771 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:25:16.771 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:25:16.771 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:25:16.794 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:25:16.794 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-18T20:25:16.795 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:25:16.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.932+0000 7f42abc87700 1 Processor -- start 2024-03-18T20:25:16.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.933+0000 7f42abc87700 1 -- start start 2024-03-18T20:25:16.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.934+0000 7f42abc87700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42a4146190 0x7f42a406a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:16.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.934+0000 7f42abc87700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f42a41465a0 0x7f42a406ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:16.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.934+0000 7f42abc87700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42a406ebd0 con 0x7f42a4146af0 2024-03-18T20:25:16.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.934+0000 7f42abc87700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a406f850 con 0x7f42a41465a0 2024-03-18T20:25:16.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.934+0000 7f42abc87700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a406eeb0 con 0x7f42a4146190 2024-03-18T20:25:16.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.934+0000 7f42a9222700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f42a41465a0 0x7f42a406ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:16.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.934+0000 7f42a9a23700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42a4146190 0x7f42a406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:16.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.934+0000 7f42a9222700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f42a41465a0 0x7f42a406ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:33214/0 (socket says 172.21.15.1:33214) 2024-03-18T20:25:16.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.934+0000 7f42aa224700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f42a4146af0 0x7f42a4071650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:41534/0 (socket says 172.21.15.1:41534) 2024-03-18T20:25:16.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.934+0000 7f42a9a23700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42a4146190 0x7f42a406a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:59534/0 (socket says 172.21.15.1:59534) 2024-03-18T20:25:16.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.934+0000 7f42a9222700 1 -- 172.21.15.1:0/3718661708 learned_addr learned my addr 172.21.15.1:0/3718661708 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:16.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.934+0000 7f42a9222700 1 -- 172.21.15.1:0/3718661708 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42a4146190 msgr2=0x7f42a406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:16.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.934+0000 7f42a9222700 1 --2- 172.21.15.1:0/3718661708 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42a4146190 0x7f42a406a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.934+0000 7f42a9222700 1 -- 172.21.15.1:0/3718661708 >> v1:172.21.15.165:6789/0 conn(0x7f42a4146af0 legacy=0x7f42a4071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:16.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.934+0000 7f42a9222700 1 -- 172.21.15.1:0/3718661708 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a406f190 con 0x7f42a41465a0 2024-03-18T20:25:16.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.934+0000 7f42a9a23700 1 --2- 172.21.15.1:0/3718661708 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42a4146190 0x7f42a406a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:16.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.935+0000 7f42a9222700 1 --2- 172.21.15.1:0/3718661708 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42a41465a0 0x7f42a406ac90 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f42a00098f0 tx=0x7f42a0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cb7a2f9257ce2dc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:16.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.935+0000 7f42a8a21700 1 -- 172.21.15.1:0/3718661708 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0011830 con 0x7f42a41465a0 2024-03-18T20:25:16.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.935+0000 7f42a8a21700 1 -- 172.21.15.1:0/3718661708 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a0011990 con 0x7f42a41465a0 2024-03-18T20:25:16.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.935+0000 7f42a8a21700 1 -- 172.21.15.1:0/3718661708 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0011d20 con 0x7f42a41465a0 2024-03-18T20:25:16.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.935+0000 7f42abc87700 1 -- 172.21.15.1:0/3718661708 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42a41465a0 msgr2=0x7f42a406ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:16.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.935+0000 7f42abc87700 1 --2- 172.21.15.1:0/3718661708 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42a41465a0 0x7f42a406ac90 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f42a00098f0 tx=0x7f42a0009c00 comp rx=0 tx=0).stop 2024-03-18T20:25:16.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.935+0000 7f42abc87700 1 -- 172.21.15.1:0/3718661708 shutdown_connections 2024-03-18T20:25:16.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.935+0000 7f42abc87700 1 --2- 172.21.15.1:0/3718661708 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42a4146190 0x7f42a406a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.935+0000 7f42abc87700 1 --2- 172.21.15.1:0/3718661708 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42a41465a0 0x7f42a406ac90 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.935+0000 7f42abc87700 1 -- 172.21.15.1:0/3718661708 >> 172.21.15.1:0/3718661708 conn(0x7f42a40b1d20 msgr2=0x7f42a40b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:16.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.935+0000 7f42abc87700 1 -- 172.21.15.1:0/3718661708 shutdown_connections 2024-03-18T20:25:16.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.935+0000 7f42abc87700 1 -- 172.21.15.1:0/3718661708 wait complete. 2024-03-18T20:25:16.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.937+0000 7f42abc87700 1 Processor -- start 2024-03-18T20:25:16.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.937+0000 7f42abc87700 1 -- start start 2024-03-18T20:25:16.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.937+0000 7f42abc87700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42a41465a0 0x7f42a41f5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:16.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.937+0000 7f42abc87700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42a4146af0 0x7f42a40ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:16.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.937+0000 7f42abc87700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42a41f4360 0x7f42a40ae870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:16.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.937+0000 7f42abc87700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a40c5130 con 0x7f42a41f4360 2024-03-18T20:25:16.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.937+0000 7f42abc87700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a40c5410 con 0x7f42a4146af0 2024-03-18T20:25:16.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.937+0000 7f42abc87700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a40c56f0 con 0x7f42a41465a0 2024-03-18T20:25:16.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.937+0000 7f42aa224700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42a41f4360 0x7f42a40ae870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:16.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.938+0000 7f42aa224700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42a41f4360 0x7f42a40ae870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34160/0 (socket says 172.21.15.1:34160) 2024-03-18T20:25:16.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.938+0000 7f42aa224700 1 -- 172.21.15.1:0/1110899514 learned_addr learned my addr 172.21.15.1:0/1110899514 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:16.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.938+0000 7f42a9222700 1 --2- 172.21.15.1:0/1110899514 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42a4146af0 0x7f42a40ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:16.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.938+0000 7f42a9a23700 1 --2- 172.21.15.1:0/1110899514 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42a41465a0 0x7f42a41f5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:16.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.938+0000 7f42aa224700 1 -- 172.21.15.1:0/1110899514 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42a41465a0 msgr2=0x7f42a41f5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:16.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.938+0000 7f42aa224700 1 --2- 172.21.15.1:0/1110899514 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42a41465a0 0x7f42a41f5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.938+0000 7f42aa224700 1 -- 172.21.15.1:0/1110899514 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42a4146af0 msgr2=0x7f42a40ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:16.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.938+0000 7f42aa224700 1 --2- 172.21.15.1:0/1110899514 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42a4146af0 0x7f42a40ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:16.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.938+0000 7f42aa224700 1 -- 172.21.15.1:0/1110899514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42940094a0 con 0x7f42a41f4360 2024-03-18T20:25:16.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.938+0000 7f42a9a23700 1 --2- 172.21.15.1:0/1110899514 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42a41465a0 0x7f42a41f5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:16.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.938+0000 7f42aa224700 1 --2- 172.21.15.1:0/1110899514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42a41f4360 0x7f42a40ae870 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7f429c00cbc0 tx=0x7f429c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:16.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.938+0000 7f429a7fc700 1 -- 172.21.15.1:0/1110899514 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429c00aaa0 con 0x7f42a41f4360 2024-03-18T20:25:16.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.938+0000 7f429a7fc700 1 -- 172.21.15.1:0/1110899514 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f429c00ac00 con 0x7f42a41f4360 2024-03-18T20:25:16.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.938+0000 7f42abc87700 1 -- 172.21.15.1:0/1110899514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a0009550 con 0x7f42a41f4360 2024-03-18T20:25:16.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.938+0000 7f429a7fc700 1 -- 172.21.15.1:0/1110899514 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429c007070 con 0x7f42a41f4360 2024-03-18T20:25:16.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.938+0000 7f42abc87700 1 -- 172.21.15.1:0/1110899514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a40c5cb0 con 0x7f42a41f4360 2024-03-18T20:25:16.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.939+0000 7f42abc87700 1 -- 172.21.15.1:0/1110899514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a40676a0 con 0x7f42a41f4360 2024-03-18T20:25:16.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.940+0000 7f429a7fc700 1 -- 172.21.15.1:0/1110899514 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f429c0071d0 con 0x7f42a41f4360 2024-03-18T20:25:16.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.940+0000 7f429a7fc700 1 --2- 172.21.15.1:0/1110899514 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f428806adb0 0x7f428806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:16.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.940+0000 7f429a7fc700 1 -- 172.21.15.1:0/1110899514 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 6079+0+0 (secure 0 0 0) 0x7f429c089180 con 0x7f42a41f4360 2024-03-18T20:25:16.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.940+0000 7f42a9a23700 1 --2- 172.21.15.1:0/1110899514 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f428806adb0 0x7f428806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:16.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.941+0000 7f42a9a23700 1 --2- 172.21.15.1:0/1110899514 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f428806adb0 0x7f428806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4294009dc0 tx=0x7f4294008040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:16.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:16.945+0000 7f429a7fc700 1 -- 172.21.15.1:0/1110899514 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f429c054220 con 0x7f42a41f4360 2024-03-18T20:25:17.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.193+0000 7f42abc87700 1 -- 172.21.15.1:0/1110899514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42a40b14c0 con 0x7f42a41f4360 2024-03-18T20:25:17.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.193+0000 7f429a7fc700 1 -- 172.21.15.1:0/1110899514 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3525 (secure 0 0 0) 0x7f429c017de0 con 0x7f42a41f4360 2024-03-18T20:25:17.197 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:17.198 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":99,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2241819619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":2241819619},{"type":"v1","addr":"172.21.15.165:6835","nonce":2241819619}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":9922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/320492236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":320492236},{"type":"v1","addr":"172.21.15.165:6834","nonce":320492236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":11031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/1035910531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":1035910531},{"type":"v1","addr":"172.21.15.1:6836","nonce":1035910531}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":92},{"gid":11466,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/883710967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":883710967},{"type":"v1","addr":"172.21.15.1:6837","nonce":883710967}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":99}],"filesystems":[]} 2024-03-18T20:25:17.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.195+0000 7f42abc87700 1 -- 172.21.15.1:0/1110899514 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f428806adb0 msgr2=0x7f428806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:17.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.195+0000 7f42abc87700 1 --2- 172.21.15.1:0/1110899514 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f428806adb0 0x7f428806d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4294009dc0 tx=0x7f4294008040 comp rx=0 tx=0).stop 2024-03-18T20:25:17.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.195+0000 7f42abc87700 1 -- 172.21.15.1:0/1110899514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42a41f4360 msgr2=0x7f42a40ae870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:17.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.195+0000 7f42abc87700 1 --2- 172.21.15.1:0/1110899514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42a41f4360 0x7f42a40ae870 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7f429c00cbc0 tx=0x7f429c00ced0 comp rx=0 tx=0).stop 2024-03-18T20:25:17.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.195+0000 7f42abc87700 1 -- 172.21.15.1:0/1110899514 shutdown_connections 2024-03-18T20:25:17.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.195+0000 7f42abc87700 1 --2- 172.21.15.1:0/1110899514 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f428806adb0 0x7f428806d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:17.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.195+0000 7f42abc87700 1 --2- 172.21.15.1:0/1110899514 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42a41465a0 0x7f42a41f5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:17.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.195+0000 7f42abc87700 1 --2- 172.21.15.1:0/1110899514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42a41f4360 0x7f42a40ae870 unknown :-1 s=CLOSED pgs=2477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:17.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.195+0000 7f42abc87700 1 --2- 172.21.15.1:0/1110899514 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42a4146af0 0x7f42a40ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:17.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.195+0000 7f42abc87700 1 -- 172.21.15.1:0/1110899514 >> 172.21.15.1:0/1110899514 conn(0x7f42a40b1d20 msgr2=0x7f42a41453e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:17.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.195+0000 7f42abc87700 1 -- 172.21.15.1:0/1110899514 shutdown_connections 2024-03-18T20:25:17.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.195+0000 7f42abc87700 1 -- 172.21.15.1:0/1110899514 wait complete. 2024-03-18T20:25:17.201 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 99 2024-03-18T20:25:17.212 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T20:25:17.212 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-18T20:25:17.212 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T20:25:17.212 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:25:17.213 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T20:25:17.213 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-18T20:25:17.213 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-18T20:25:17.213 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:25:17.213 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T20:25:17.213 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-18T20:25:17.214 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-18T20:25:17.214 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:25:17.214 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T20:25:17.214 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-18T20:25:17.214 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-18T20:25:17.214 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:25:17.221 INFO:tasks.ceph.mds.a:Stopped 2024-03-18T20:25:17.221 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-18T20:25:17.222 INFO:tasks.ceph.mds.c:Stopped 2024-03-18T20:25:17.222 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-18T20:25:17.224 INFO:tasks.ceph.mds.b:Stopped 2024-03-18T20:25:17.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-18T20:25:17.225 INFO:tasks.ceph.mds.d:Stopped 2024-03-18T20:25:17.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-18T20:25:17.263 INFO:tasks.ceph.mds.a:Started 2024-03-18T20:25:17.264 DEBUG:teuthology.parallel:result is None 2024-03-18T20:25:17.264 INFO:tasks.ceph.mds.c:Started 2024-03-18T20:25:17.264 DEBUG:teuthology.parallel:result is None 2024-03-18T20:25:17.267 INFO:tasks.ceph.mds.b:Started 2024-03-18T20:25:17.267 DEBUG:teuthology.parallel:result is None 2024-03-18T20:25:17.267 INFO:tasks.ceph.mds.d:Started 2024-03-18T20:25:17.268 DEBUG:teuthology.parallel:result is None 2024-03-18T20:25:17.268 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-18T20:25:17.345 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-18T20:25:17.345 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T20:25:17.356 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-03-18T20:25:17.356 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-03-18T20:25:17.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.401+0000 7fb88bd43700 1 Processor -- start 2024-03-18T20:25:17.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.401+0000 7fb88bd43700 1 -- start start 2024-03-18T20:25:17.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.402+0000 7fb88bd43700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb88415b6c0 0x7fb8840b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:17.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.402+0000 7fb88bd43700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8840c87d0 0x7fb8840cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:17.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.402+0000 7fb88bd43700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8840d01f0 con 0x7fb8840cd430 2024-03-18T20:25:17.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.402+0000 7fb88bd43700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8840ccfa0 con 0x7fb88415b6c0 2024-03-18T20:25:17.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.402+0000 7fb88bd43700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8840d1db0 con 0x7fb8840c87d0 2024-03-18T20:25:17.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.402+0000 7fb8892de700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb8840cd430 0x7fb8840c80c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35116/0 (socket says 172.21.15.1:35116) 2024-03-18T20:25:17.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.402+0000 7fb889adf700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb88415b6c0 0x7fb8840b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:17.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.402+0000 7fb8892de700 1 -- 172.21.15.1:0/3747686037 learned_addr learned my addr 172.21.15.1:0/3747686037 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:17.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.402+0000 7fb88a2e0700 1 --2- 172.21.15.1:0/3747686037 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8840c87d0 0x7fb8840cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:17.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.403+0000 7fb889adf700 1 -- 172.21.15.1:0/3747686037 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8840c87d0 msgr2=0x7fb8840cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:17.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.403+0000 7fb889adf700 1 --2- 172.21.15.1:0/3747686037 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8840c87d0 0x7fb8840cd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:17.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.403+0000 7fb889adf700 1 -- 172.21.15.1:0/3747686037 >> v1:172.21.15.1:6789/0 conn(0x7fb8840cd430 legacy=0x7fb8840c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:17.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.403+0000 7fb889adf700 1 -- 172.21.15.1:0/3747686037 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8840cff10 con 0x7fb88415b6c0 2024-03-18T20:25:17.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.403+0000 7fb888add700 1 -- 172.21.15.1:0/3747686037 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224563826 0 0) 0x7fb8840d01f0 con 0x7fb8840cd430 2024-03-18T20:25:17.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.403+0000 7fb889adf700 1 --2- 172.21.15.1:0/3747686037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb88415b6c0 0x7fb8840b2160 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009d90 tx=0x7fb87c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5788fee06842250 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:17.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.403+0000 7fb888add700 1 -- 172.21.15.1:0/3747686037 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c00e070 con 0x7fb88415b6c0 2024-03-18T20:25:17.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.403+0000 7fb888add700 1 -- 172.21.15.1:0/3747686037 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87c00bb20 con 0x7fb88415b6c0 2024-03-18T20:25:17.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.403+0000 7fb888add700 1 -- 172.21.15.1:0/3747686037 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c012430 con 0x7fb88415b6c0 2024-03-18T20:25:17.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.403+0000 7fb88bd43700 1 -- 172.21.15.1:0/3747686037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb88415b6c0 msgr2=0x7fb8840b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:17.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.403+0000 7fb88bd43700 1 --2- 172.21.15.1:0/3747686037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb88415b6c0 0x7fb8840b2160 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009d90 tx=0x7fb87c00b330 comp rx=0 tx=0).stop 2024-03-18T20:25:17.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.403+0000 7fb88bd43700 1 -- 172.21.15.1:0/3747686037 shutdown_connections 2024-03-18T20:25:17.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.403+0000 7fb88bd43700 1 --2- 172.21.15.1:0/3747686037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb88415b6c0 0x7fb8840b2160 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:17.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.403+0000 7fb88bd43700 1 --2- 172.21.15.1:0/3747686037 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8840c87d0 0x7fb8840cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:17.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.403+0000 7fb88bd43700 1 -- 172.21.15.1:0/3747686037 >> 172.21.15.1:0/3747686037 conn(0x7fb8840d0ff0 msgr2=0x7fb8840d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:17.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.404+0000 7fb88bd43700 1 -- 172.21.15.1:0/3747686037 shutdown_connections 2024-03-18T20:25:17.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.404+0000 7fb88bd43700 1 -- 172.21.15.1:0/3747686037 wait complete. 2024-03-18T20:25:17.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.405+0000 7fb88bd43700 1 Processor -- start 2024-03-18T20:25:17.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.405+0000 7fb88bd43700 1 -- start start 2024-03-18T20:25:17.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.405+0000 7fb88bd43700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8840c87d0 0x7fb8841496c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:17.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.405+0000 7fb88bd43700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8840cd430 0x7fb88409d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:17.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.405+0000 7fb88bd43700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb88415b6c0 0x7fb88409d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:17.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.405+0000 7fb88bd43700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb884149c00 con 0x7fb8840c87d0 2024-03-18T20:25:17.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.405+0000 7fb88bd43700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8841f1790 con 0x7fb88415b6c0 2024-03-18T20:25:17.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.405+0000 7fb88bd43700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8841f1ac0 con 0x7fb8840cd430 2024-03-18T20:25:17.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.406+0000 7fb889adf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8840c87d0 0x7fb8841496c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:17.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.406+0000 7fb889adf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8840c87d0 0x7fb8841496c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34208/0 (socket says 172.21.15.1:34208) 2024-03-18T20:25:17.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.406+0000 7fb889adf700 1 -- 172.21.15.1:0/1632540672 learned_addr learned my addr 172.21.15.1:0/1632540672 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:17.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.406+0000 7fb889adf700 1 -- 172.21.15.1:0/1632540672 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8840cd430 msgr2=0x7fb88409d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:17.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.406+0000 7fb8892de700 1 --2- 172.21.15.1:0/1632540672 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8840cd430 0x7fb88409d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:17.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.406+0000 7fb889adf700 1 --2- 172.21.15.1:0/1632540672 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8840cd430 0x7fb88409d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:17.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.406+0000 7fb88a2e0700 1 --2- 172.21.15.1:0/1632540672 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb88415b6c0 0x7fb88409d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:17.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.406+0000 7fb889adf700 1 -- 172.21.15.1:0/1632540672 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb88415b6c0 msgr2=0x7fb88409d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:17.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.406+0000 7fb889adf700 1 --2- 172.21.15.1:0/1632540672 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb88415b6c0 0x7fb88409d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:17.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.406+0000 7fb889adf700 1 -- 172.21.15.1:0/1632540672 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c009a70 con 0x7fb8840c87d0 2024-03-18T20:25:17.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.406+0000 7fb8892de700 1 --2- 172.21.15.1:0/1632540672 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8840cd430 0x7fb88409d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:17.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.406+0000 7fb88a2e0700 1 --2- 172.21.15.1:0/1632540672 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb88415b6c0 0x7fb88409d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:17.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.406+0000 7fb889adf700 1 --2- 172.21.15.1:0/1632540672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8840c87d0 0x7fb8841496c0 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7fb87c014040 tx=0x7fb87c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:17.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.406+0000 7fb87a7fc700 1 -- 172.21.15.1:0/1632540672 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c012430 con 0x7fb8840c87d0 2024-03-18T20:25:17.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.406+0000 7fb87a7fc700 1 -- 172.21.15.1:0/1632540672 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87c020070 con 0x7fb8840c87d0 2024-03-18T20:25:17.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.406+0000 7fb88bd43700 1 -- 172.21.15.1:0/1632540672 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8841f1da0 con 0x7fb8840c87d0 2024-03-18T20:25:17.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.406+0000 7fb87a7fc700 1 -- 172.21.15.1:0/1632540672 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c004260 con 0x7fb8840c87d0 2024-03-18T20:25:17.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.406+0000 7fb88bd43700 1 -- 172.21.15.1:0/1632540672 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8841f2360 con 0x7fb8840c87d0 2024-03-18T20:25:17.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.407+0000 7fb88bd43700 1 -- 172.21.15.1:0/1632540672 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8840676a0 con 0x7fb8840c87d0 2024-03-18T20:25:17.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.408+0000 7fb87a7fc700 1 -- 172.21.15.1:0/1632540672 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb87c0043c0 con 0x7fb8840c87d0 2024-03-18T20:25:17.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.408+0000 7fb87a7fc700 1 --2- 172.21.15.1:0/1632540672 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb86806adb0 0x7fb86806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:17.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.408+0000 7fb87a7fc700 1 -- 172.21.15.1:0/1632540672 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 6051+0+0 (secure 0 0 0) 0x7fb87c022070 con 0x7fb8840c87d0 2024-03-18T20:25:17.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.408+0000 7fb8892de700 1 --2- 172.21.15.1:0/1632540672 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb86806adb0 0x7fb86806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:17.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.408+0000 7fb8892de700 1 --2- 172.21.15.1:0/1632540672 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb86806adb0 0x7fb86806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb8840c7490 tx=0x7fb874009040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:17.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.413+0000 7fb87a7fc700 1 -- 172.21.15.1:0/1632540672 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb87c056330 con 0x7fb8840c87d0 2024-03-18T20:25:17.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:17.631+0000 7fb88bd43700 1 -- 172.21.15.1:0/1632540672 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb884073560 con 0x7fb8840c87d0 2024-03-18T20:25:18.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.585+0000 7fb87a7fc700 1 -- 172.21.15.1:0/1632540672 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v228) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb87c059950 con 0x7fb8840c87d0 2024-03-18T20:25:18.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.588+0000 7fb88bd43700 1 -- 172.21.15.1:0/1632540672 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb86806adb0 msgr2=0x7fb86806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:18.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.588+0000 7fb88bd43700 1 --2- 172.21.15.1:0/1632540672 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb86806adb0 0x7fb86806d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb8840c7490 tx=0x7fb874009040 comp rx=0 tx=0).stop 2024-03-18T20:25:18.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.588+0000 7fb88bd43700 1 -- 172.21.15.1:0/1632540672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8840c87d0 msgr2=0x7fb8841496c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:18.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.588+0000 7fb88bd43700 1 --2- 172.21.15.1:0/1632540672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8840c87d0 0x7fb8841496c0 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7fb87c014040 tx=0x7fb87c004ad0 comp rx=0 tx=0).stop 2024-03-18T20:25:18.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.588+0000 7fb88bd43700 1 -- 172.21.15.1:0/1632540672 shutdown_connections 2024-03-18T20:25:18.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.588+0000 7fb88bd43700 1 --2- 172.21.15.1:0/1632540672 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb86806adb0 0x7fb86806d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:18.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.588+0000 7fb88bd43700 1 --2- 172.21.15.1:0/1632540672 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb88415b6c0 0x7fb88409d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:18.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.588+0000 7fb88bd43700 1 --2- 172.21.15.1:0/1632540672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8840c87d0 0x7fb8841496c0 unknown :-1 s=CLOSED pgs=2486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:18.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.588+0000 7fb88bd43700 1 --2- 172.21.15.1:0/1632540672 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb8840cd430 0x7fb88409d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:18.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.588+0000 7fb88bd43700 1 -- 172.21.15.1:0/1632540672 >> 172.21.15.1:0/1632540672 conn(0x7fb8840d0ff0 msgr2=0x7fb88406e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:18.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.588+0000 7fb88bd43700 1 -- 172.21.15.1:0/1632540672 shutdown_connections 2024-03-18T20:25:18.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.588+0000 7fb88bd43700 1 -- 172.21.15.1:0/1632540672 wait complete. 2024-03-18T20:25:18.593 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-18T20:25:18.609 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-18T20:25:18.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.739+0000 7fee0f71a700 1 Processor -- start 2024-03-18T20:25:18.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.740+0000 7fee0f71a700 1 -- start start 2024-03-18T20:25:18.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.741+0000 7fee0f71a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fee0815d6b0 0x7fee080b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.741+0000 7fee0f71a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fee080ad790 0x7fee080b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.741+0000 7fee0f71a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee080acfb0 con 0x7fee0806a750 2024-03-18T20:25:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.741+0000 7fee0f71a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee080af7a0 con 0x7fee0815d6b0 2024-03-18T20:25:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.741+0000 7fee0f71a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee080b50a0 con 0x7fee080ad790 2024-03-18T20:25:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.741+0000 7fee0ccb5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fee0815d6b0 0x7fee080b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.741+0000 7fee0ccb5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fee0815d6b0 0x7fee080b2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34212/0 (socket says 172.21.15.1:34212) 2024-03-18T20:25:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.741+0000 7fee0ccb5700 1 -- 172.21.15.1:0/1097984001 learned_addr learned my addr 172.21.15.1:0/1097984001 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.741+0000 7fee0dcb7700 1 --2- 172.21.15.1:0/1097984001 >> v2:172.21.15.165:3300/0 conn(0x7fee080ad790 0x7fee080b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.741+0000 7fee0ccb5700 1 -- 172.21.15.1:0/1097984001 >> v2:172.21.15.165:3300/0 conn(0x7fee080ad790 msgr2=0x7fee080b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.741+0000 7fee0ccb5700 1 --2- 172.21.15.1:0/1097984001 >> v2:172.21.15.165:3300/0 conn(0x7fee080ad790 0x7fee080b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.741+0000 7fee0ccb5700 1 -- 172.21.15.1:0/1097984001 >> v1:172.21.15.165:6789/0 conn(0x7fee0806a750 legacy=0x7fee0806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.741+0000 7fee0ccb5700 1 -- 172.21.15.1:0/1097984001 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee080af1e0 con 0x7fee0815d6b0 2024-03-18T20:25:18.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.741+0000 7fee0dcb7700 1 --2- 172.21.15.1:0/1097984001 >> v2:172.21.15.165:3300/0 conn(0x7fee080ad790 0x7fee080b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.741+0000 7fee0ccb5700 1 --2- 172.21.15.1:0/1097984001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee0815d6b0 0x7fee080b2da0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7fee080726a0 tx=0x7fedfc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b2febc8c183152d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.741+0000 7fedfbfff700 1 -- 172.21.15.1:0/1097984001 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc004230 con 0x7fee0815d6b0 2024-03-18T20:25:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.741+0000 7fedfbfff700 1 -- 172.21.15.1:0/1097984001 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedfc004390 con 0x7fee0815d6b0 2024-03-18T20:25:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.741+0000 7fedfbfff700 1 -- 172.21.15.1:0/1097984001 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc0036a0 con 0x7fee0815d6b0 2024-03-18T20:25:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.742+0000 7fee0f71a700 1 -- 172.21.15.1:0/1097984001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee0815d6b0 msgr2=0x7fee080b2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.742+0000 7fee0f71a700 1 --2- 172.21.15.1:0/1097984001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee0815d6b0 0x7fee080b2da0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7fee080726a0 tx=0x7fedfc009bd0 comp rx=0 tx=0).stop 2024-03-18T20:25:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.742+0000 7fee0f71a700 1 -- 172.21.15.1:0/1097984001 shutdown_connections 2024-03-18T20:25:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.742+0000 7fee0f71a700 1 --2- 172.21.15.1:0/1097984001 >> v2:172.21.15.165:3300/0 conn(0x7fee080ad790 0x7fee080b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.742+0000 7fee0f71a700 1 --2- 172.21.15.1:0/1097984001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee0815d6b0 0x7fee080b2da0 unknown :-1 s=CLOSED pgs=2487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.742+0000 7fee0f71a700 1 -- 172.21.15.1:0/1097984001 >> 172.21.15.1:0/1097984001 conn(0x7fee0806b860 msgr2=0x7fee0806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.742+0000 7fee0f71a700 1 -- 172.21.15.1:0/1097984001 shutdown_connections 2024-03-18T20:25:18.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.742+0000 7fee0f71a700 1 -- 172.21.15.1:0/1097984001 wait complete. 2024-03-18T20:25:18.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.744+0000 7fee0f71a700 1 Processor -- start 2024-03-18T20:25:18.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.744+0000 7fee0f71a700 1 -- start start 2024-03-18T20:25:18.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0f71a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee0806a750 0x7fee0814b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:18.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0f71a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee080ad790 0x7fee080cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:18.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0f71a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee0815d6b0 0x7fee0815c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:18.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0f71a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee080cd5e0 con 0x7fee080ad790 2024-03-18T20:25:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0f71a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee080cd8c0 con 0x7fee0806a750 2024-03-18T20:25:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0f71a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee080cdba0 con 0x7fee0815d6b0 2024-03-18T20:25:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0ccb5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee080ad790 0x7fee080cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0ccb5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee080ad790 0x7fee080cb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34214/0 (socket says 172.21.15.1:34214) 2024-03-18T20:25:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0ccb5700 1 -- 172.21.15.1:0/1304335700 learned_addr learned my addr 172.21.15.1:0/1304335700 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0d4b6700 1 --2- 172.21.15.1:0/1304335700 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee0806a750 0x7fee0814b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0dcb7700 1 --2- 172.21.15.1:0/1304335700 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee0815d6b0 0x7fee0815c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0ccb5700 1 -- 172.21.15.1:0/1304335700 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee0815d6b0 msgr2=0x7fee0815c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0ccb5700 1 --2- 172.21.15.1:0/1304335700 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee0815d6b0 0x7fee0815c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0ccb5700 1 -- 172.21.15.1:0/1304335700 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee0806a750 msgr2=0x7fee0814b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0ccb5700 1 --2- 172.21.15.1:0/1304335700 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee0806a750 0x7fee0814b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0ccb5700 1 -- 172.21.15.1:0/1304335700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedfc009550 con 0x7fee080ad790 2024-03-18T20:25:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0d4b6700 1 --2- 172.21.15.1:0/1304335700 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee0806a750 0x7fee0814b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:18.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0dcb7700 1 --2- 172.21.15.1:0/1304335700 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee0815d6b0 0x7fee0815c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0ccb5700 1 --2- 172.21.15.1:0/1304335700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee080ad790 0x7fee080cb750 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7fedfc009130 tx=0x7fedfc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fedf9ffb700 1 -- 172.21.15.1:0/1304335700 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc011630 con 0x7fee080ad790 2024-03-18T20:25:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fedf9ffb700 1 -- 172.21.15.1:0/1304335700 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedfc011790 con 0x7fee080ad790 2024-03-18T20:25:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.745+0000 7fee0f71a700 1 -- 172.21.15.1:0/1304335700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee080cde80 con 0x7fee080ad790 2024-03-18T20:25:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.746+0000 7fedf9ffb700 1 -- 172.21.15.1:0/1304335700 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc011b40 con 0x7fee080ad790 2024-03-18T20:25:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.746+0000 7fee0f71a700 1 -- 172.21.15.1:0/1304335700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee0814b740 con 0x7fee080ad790 2024-03-18T20:25:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.747+0000 7fee0f71a700 1 -- 172.21.15.1:0/1304335700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee08066960 con 0x7fee080ad790 2024-03-18T20:25:18.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.747+0000 7fedf9ffb700 1 -- 172.21.15.1:0/1304335700 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fedfc011ca0 con 0x7fee080ad790 2024-03-18T20:25:18.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.748+0000 7fedf9ffb700 1 --2- 172.21.15.1:0/1304335700 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fedec06adb0 0x7fedec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:18.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.748+0000 7fee0d4b6700 1 --2- 172.21.15.1:0/1304335700 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fedec06adb0 0x7fedec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:18.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.748+0000 7fedf9ffb700 1 -- 172.21.15.1:0/1304335700 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 5277+0+0 (secure 0 0 0) 0x7fedfc08a8c0 con 0x7fee080ad790 2024-03-18T20:25:18.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.748+0000 7fee0d4b6700 1 --2- 172.21.15.1:0/1304335700 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fedec06adb0 0x7fedec06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fee04003f40 tx=0x7fee04009040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:18.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:18.753+0000 7fedf9ffb700 1 -- 172.21.15.1:0/1304335700 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fedfc055c80 con 0x7fee080ad790 2024-03-18T20:25:19.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.036+0000 7fee0f71a700 1 -- 172.21.15.1:0/1304335700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fee080ad420 con 0x7fee080ad790 2024-03-18T20:25:19.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.036+0000 7fedf9ffb700 1 -- 172.21.15.1:0/1304335700 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v49) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fee080ad420 con 0x7fee080ad790 2024-03-18T20:25:19.041 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:19.041 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:25:19.041 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-18T20:25:19.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:19.041 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-18T20:25:19.041 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlR93/CxAAbpe+5n3wHRur9xi8QzdCnQ==", 2024-03-18T20:25:19.041 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:25:19.042 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:25:19.042 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:25:19.042 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:25:19.042 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:25:19.042 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:19.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:19.042 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-18T20:25:19.042 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJ3OmGBAA37u227nxrMsh/JHt71HBxQ==", 2024-03-18T20:25:19.042 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:25:19.043 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:25:19.043 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:25:19.043 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:25:19.043 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:25:19.043 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:19.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:19.043 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-18T20:25:19.044 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJZFoExAAmI16oxwNKbf8dKetPfyk+g==", 2024-03-18T20:25:19.044 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:25:19.044 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:25:19.044 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:25:19.044 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:25:19.044 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:25:19.044 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.044 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:19.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:19.044 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-18T20:25:19.044 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlQrcvIBAA1LMv+EgC7S6wspX6QF49Pg==", 2024-03-18T20:25:19.044 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:25:19.044 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:25:19.044 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:25:19.045 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:25:19.045 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:25:19.045 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:19.045 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:19.045 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-18T20:25:19.045 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCUjPhlYOVHChAARFQrksZmMIdTlntd8TgvrQ==", 2024-03-18T20:25:19.045 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:25:19.045 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:25:19.045 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:25:19.045 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:25:19.045 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:19.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:19.046 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-18T20:25:19.046 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCXjPhl2zSDCBAA6Al1gkWVB79ZI9djGpVChA==", 2024-03-18T20:25:19.046 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:25:19.046 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:25:19.046 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:25:19.046 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:25:19.046 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:19.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:19.046 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-18T20:25:19.047 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCajPhlSoG/BhAAR93USDe6DCQW+hREOsnTlw==", 2024-03-18T20:25:19.047 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:25:19.047 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:25:19.047 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:25:19.047 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:25:19.047 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:19.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:19.047 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-18T20:25:19.047 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCdjPhlhJg5BRAAWuQhcnxyjR5UOiM13yKdAQ==", 2024-03-18T20:25:19.047 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:25:19.047 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:25:19.047 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:25:19.047 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:25:19.048 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:19.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:19.048 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-18T20:25:19.048 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCjjPhlCFnpCRAAKIyOQyVdjXZUTnAS6Nw/ng==", 2024-03-18T20:25:19.048 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:25:19.048 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:25:19.048 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:25:19.048 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:25:19.048 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:19.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:19.048 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-18T20:25:19.048 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCmjPhlAeFkCBAAJcEFpRk/03lQJxj+l2hEPA==", 2024-03-18T20:25:19.049 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:25:19.049 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:25:19.049 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:25:19.049 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:25:19.049 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:19.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:19.049 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-18T20:25:19.049 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCpjPhl/xuuBhAA0LgKfXEvN2O11ZlRQJVOkw==", 2024-03-18T20:25:19.049 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:25:19.050 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:25:19.050 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:25:19.050 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:25:19.050 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.050 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:19.050 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:19.050 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-18T20:25:19.050 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCsjPhldiR4BRAA571oh6U8PzOv/4LzjYLF/A==", 2024-03-18T20:25:19.051 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:25:19.051 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:25:19.051 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:25:19.051 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:25:19.051 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:19.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:19.051 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-18T20:25:19.051 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlu+2xJBAANnXGGjfeqP6S2FTecsfOsg==", 2024-03-18T20:25:19.051 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:25:19.051 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:25:19.051 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-18T20:25:19.051 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-18T20:25:19.052 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:19.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:19.052 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-18T20:25:19.052 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlhJ9+KxAAEnrQrCrgJR6xI+pucwzAXA==", 2024-03-18T20:25:19.052 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:25:19.052 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:25:19.052 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:25:19.052 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:25:19.052 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:25:19.052 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:19.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:19.053 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.2", 2024-03-18T20:25:19.053 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlxBUpKBAAIGTwmLWyj+P2Yvl55QVcWA==", 2024-03-18T20:25:19.053 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:25:19.053 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:25:19.053 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:25:19.053 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:25:19.053 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:25:19.054 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:19.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:19.054 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.3", 2024-03-18T20:25:19.054 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlmao3MRAAHeiQ6eyBh9QjjKPCtxkC8g==", 2024-03-18T20:25:19.054 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:25:19.055 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:25:19.055 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:25:19.055 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:25:19.055 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:25:19.055 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:19.055 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:19.055 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-18T20:25:19.055 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCRjPhlemOuKRAAVUggmtB7+EiVgyfJZ8TCCA==", 2024-03-18T20:25:19.055 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:25:19.055 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:25:19.055 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:25:19.055 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:25:19.055 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:25:19.055 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:19.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:19.056 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-18T20:25:19.056 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhleIuOCBAA9CH+AHewQbn+gpVr+6lJyw==", 2024-03-18T20:25:19.056 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:25:19.056 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:25:19.056 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T20:25:19.056 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:25:19.056 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:19.056 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:19.056 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-18T20:25:19.056 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlL0pTBBAAdFiPcdwwz514cuLAvioxYA==", 2024-03-18T20:25:19.056 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:25:19.057 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:25:19.057 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T20:25:19.057 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:25:19.057 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.057 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:19.057 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-18T20:25:19.057 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:25:19.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.039+0000 7fee0f71a700 1 -- 172.21.15.1:0/1304335700 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fedec06adb0 msgr2=0x7fedec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.039+0000 7fee0f71a700 1 --2- 172.21.15.1:0/1304335700 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fedec06adb0 0x7fedec06d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fee04003f40 tx=0x7fee04009040 comp rx=0 tx=0).stop 2024-03-18T20:25:19.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.039+0000 7fee0f71a700 1 -- 172.21.15.1:0/1304335700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee080ad790 msgr2=0x7fee080cb750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.039+0000 7fee0f71a700 1 --2- 172.21.15.1:0/1304335700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee080ad790 0x7fee080cb750 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7fedfc009130 tx=0x7fedfc004750 comp rx=0 tx=0).stop 2024-03-18T20:25:19.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.039+0000 7fee0f71a700 1 -- 172.21.15.1:0/1304335700 shutdown_connections 2024-03-18T20:25:19.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.039+0000 7fee0f71a700 1 --2- 172.21.15.1:0/1304335700 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fedec06adb0 0x7fedec06d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.039+0000 7fee0f71a700 1 --2- 172.21.15.1:0/1304335700 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fee0815d6b0 0x7fee0815c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.039+0000 7fee0f71a700 1 --2- 172.21.15.1:0/1304335700 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee0806a750 0x7fee0814b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.039+0000 7fee0f71a700 1 --2- 172.21.15.1:0/1304335700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fee080ad790 0x7fee080cb750 unknown :-1 s=CLOSED pgs=2488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.039+0000 7fee0f71a700 1 -- 172.21.15.1:0/1304335700 >> 172.21.15.1:0/1304335700 conn(0x7fee0806b860 msgr2=0x7fee0806da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:19.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.039+0000 7fee0f71a700 1 -- 172.21.15.1:0/1304335700 shutdown_connections 2024-03-18T20:25:19.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.039+0000 7fee0f71a700 1 -- 172.21.15.1:0/1304335700 wait complete. 2024-03-18T20:25:19.060 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:25:19.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.181+0000 7faaf00b2700 1 Processor -- start 2024-03-18T20:25:19.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.182+0000 7faaf00b2700 1 -- start start 2024-03-18T20:25:19.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.183+0000 7faaf00b2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faae815a2e0 0x7faae80ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.183+0000 7faaf00b2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faae80cc170 0x7faae806ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.183+0000 7faaf00b2700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae806e270 con 0x7faae80ae600 2024-03-18T20:25:19.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.183+0000 7faaf00b2700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faae80ae050 con 0x7faae80cc170 2024-03-18T20:25:19.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.183+0000 7faaf00b2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae80add70 con 0x7faae815a2e0 2024-03-18T20:25:19.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.183+0000 7faaee64f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faae80cc170 0x7faae806ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.183+0000 7faaede4e700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7faae80ae600 0x7faae80ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35120/0 (socket says 172.21.15.1:35120) 2024-03-18T20:25:19.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.183+0000 7faaede4e700 1 -- 172.21.15.1:0/700288419 learned_addr learned my addr 172.21.15.1:0/700288419 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:19.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.183+0000 7faaed64d700 1 --2- 172.21.15.1:0/700288419 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faae815a2e0 0x7faae80ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.183+0000 7faaee64f700 1 -- 172.21.15.1:0/700288419 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faae815a2e0 msgr2=0x7faae80ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:19.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.183+0000 7faaee64f700 1 --2- 172.21.15.1:0/700288419 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faae815a2e0 0x7faae80ad790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.183+0000 7faaee64f700 1 -- 172.21.15.1:0/700288419 >> v1:172.21.15.1:6789/0 conn(0x7faae80ae600 legacy=0x7faae80ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.183+0000 7faaee64f700 1 -- 172.21.15.1:0/700288419 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae806e4a0 con 0x7faae80cc170 2024-03-18T20:25:19.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.183+0000 7faaece4c700 1 -- 172.21.15.1:0/700288419 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749418914 0 0) 0x7faae806e270 con 0x7faae80ae600 2024-03-18T20:25:19.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.183+0000 7faaee64f700 1 --2- 172.21.15.1:0/700288419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faae80cc170 0x7faae806ef80 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7faae0009d90 tx=0x7faae000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2939546640c34511 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:19.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.183+0000 7faaece4c700 1 -- 172.21.15.1:0/700288419 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae000e070 con 0x7faae80cc170 2024-03-18T20:25:19.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.183+0000 7faaece4c700 1 -- 172.21.15.1:0/700288419 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae000bbd0 con 0x7faae80cc170 2024-03-18T20:25:19.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.183+0000 7faaece4c700 1 -- 172.21.15.1:0/700288419 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae0012430 con 0x7faae80cc170 2024-03-18T20:25:19.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.184+0000 7faaf00b2700 1 -- 172.21.15.1:0/700288419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faae80cc170 msgr2=0x7faae806ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.184+0000 7faaf00b2700 1 --2- 172.21.15.1:0/700288419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faae80cc170 0x7faae806ef80 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7faae0009d90 tx=0x7faae000b330 comp rx=0 tx=0).stop 2024-03-18T20:25:19.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.184+0000 7faaf00b2700 1 -- 172.21.15.1:0/700288419 shutdown_connections 2024-03-18T20:25:19.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.184+0000 7faaf00b2700 1 --2- 172.21.15.1:0/700288419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faae80cc170 0x7faae806ef80 unknown :-1 s=CLOSED pgs=2490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.184+0000 7faaf00b2700 1 --2- 172.21.15.1:0/700288419 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faae815a2e0 0x7faae80ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.184+0000 7faaf00b2700 1 -- 172.21.15.1:0/700288419 >> 172.21.15.1:0/700288419 conn(0x7faae806b860 msgr2=0x7faae806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:19.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.184+0000 7faaf00b2700 1 -- 172.21.15.1:0/700288419 shutdown_connections 2024-03-18T20:25:19.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.184+0000 7faaf00b2700 1 -- 172.21.15.1:0/700288419 wait complete. 2024-03-18T20:25:19.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.185+0000 7faaf00b2700 1 Processor -- start 2024-03-18T20:25:19.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.185+0000 7faaf00b2700 1 -- start start 2024-03-18T20:25:19.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.186+0000 7faaf00b2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faae80ae600 0x7faae80cf880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.186+0000 7faaf00b2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faae80cc170 0x7faae815df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.186+0000 7faaf00b2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faae815a2e0 0x7faae81489d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.186+0000 7faaf00b2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae80cc780 con 0x7faae80cc170 2024-03-18T20:25:19.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.186+0000 7faaf00b2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae80cca60 con 0x7faae815a2e0 2024-03-18T20:25:19.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.186+0000 7faaf00b2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae80ccd40 con 0x7faae80ae600 2024-03-18T20:25:19.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.186+0000 7faaed64d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faae80cc170 0x7faae815df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.186+0000 7faaed64d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faae80cc170 0x7faae815df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34246/0 (socket says 172.21.15.1:34246) 2024-03-18T20:25:19.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.186+0000 7faaed64d700 1 -- 172.21.15.1:0/3629073035 learned_addr learned my addr 172.21.15.1:0/3629073035 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:19.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.186+0000 7faaed64d700 1 -- 172.21.15.1:0/3629073035 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faae80ae600 msgr2=0x7faae80cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.186+0000 7faaed64d700 1 --2- 172.21.15.1:0/3629073035 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faae80ae600 0x7faae80cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.186+0000 7faaed64d700 1 -- 172.21.15.1:0/3629073035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faae815a2e0 msgr2=0x7faae81489d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T20:25:19.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.186+0000 7faaed64d700 1 --2- 172.21.15.1:0/3629073035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faae815a2e0 0x7faae81489d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.186+0000 7faaed64d700 1 -- 172.21.15.1:0/3629073035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae0009a70 con 0x7faae80cc170 2024-03-18T20:25:19.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.187+0000 7faaed64d700 1 --2- 172.21.15.1:0/3629073035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faae80cc170 0x7faae815df90 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7faadc00e810 tx=0x7faadc00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:19.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.187+0000 7faada7fc700 1 -- 172.21.15.1:0/3629073035 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc013610 con 0x7faae80cc170 2024-03-18T20:25:19.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.187+0000 7faada7fc700 1 -- 172.21.15.1:0/3629073035 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faadc013770 con 0x7faae80cc170 2024-03-18T20:25:19.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.187+0000 7faaf00b2700 1 -- 172.21.15.1:0/3629073035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae80cd020 con 0x7faae80cc170 2024-03-18T20:25:19.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.187+0000 7faada7fc700 1 -- 172.21.15.1:0/3629073035 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc013b00 con 0x7faae80cc170 2024-03-18T20:25:19.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.187+0000 7faaf00b2700 1 -- 172.21.15.1:0/3629073035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae815e4d0 con 0x7faae80cc170 2024-03-18T20:25:19.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.188+0000 7faaf00b2700 1 -- 172.21.15.1:0/3629073035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae80ccba0 con 0x7faae80cc170 2024-03-18T20:25:19.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.188+0000 7faada7fc700 1 -- 172.21.15.1:0/3629073035 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7faadc013c60 con 0x7faae80cc170 2024-03-18T20:25:19.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.188+0000 7faada7fc700 1 --2- 172.21.15.1:0/3629073035 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faacc06adb0 0x7faacc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.189+0000 7faada7fc700 1 -- 172.21.15.1:0/3629073035 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 5277+0+0 (secure 0 0 0) 0x7faadc089090 con 0x7faae80cc170 2024-03-18T20:25:19.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.189+0000 7faaede4e700 1 --2- 172.21.15.1:0/3629073035 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faacc06adb0 0x7faacc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.189+0000 7faaede4e700 1 --2- 172.21.15.1:0/3629073035 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faacc06adb0 0x7faacc06d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7faae4001610 tx=0x7faae40079f0 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:19.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.193+0000 7faada7fc700 1 -- 172.21.15.1:0/3629073035 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faae80ccba0 con 0x7faae80cc170 2024-03-18T20:25:19.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.398+0000 7faaf00b2700 1 -- 172.21.15.1:0/3629073035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faae80ccba0 con 0x7faae80cc170 2024-03-18T20:25:19.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.399+0000 7faada7fc700 1 -- 172.21.15.1:0/3629073035 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v228) v1 ==== 74+0+12717 (secure 0 0 0) 0x7faae80ccba0 con 0x7faae80cc170 2024-03-18T20:25:19.403 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:19.403 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":228,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:25:18.582198+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:19.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.402+0000 7faaf00b2700 1 -- 172.21.15.1:0/3629073035 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faacc06adb0 msgr2=0x7faacc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.402+0000 7faaf00b2700 1 --2- 172.21.15.1:0/3629073035 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faacc06adb0 0x7faacc06d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7faae4001610 tx=0x7faae40079f0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.402+0000 7faaf00b2700 1 -- 172.21.15.1:0/3629073035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faae80cc170 msgr2=0x7faae815df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.402+0000 7faaf00b2700 1 --2- 172.21.15.1:0/3629073035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faae80cc170 0x7faae815df90 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7faadc00e810 tx=0x7faadc00ebd0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.402+0000 7faaf00b2700 1 -- 172.21.15.1:0/3629073035 shutdown_connections 2024-03-18T20:25:19.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.402+0000 7faaf00b2700 1 --2- 172.21.15.1:0/3629073035 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faacc06adb0 0x7faacc06d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.402+0000 7faaf00b2700 1 --2- 172.21.15.1:0/3629073035 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faae80ae600 0x7faae80cf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.402+0000 7faaf00b2700 1 --2- 172.21.15.1:0/3629073035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faae815a2e0 0x7faae81489d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.402+0000 7faaf00b2700 1 --2- 172.21.15.1:0/3629073035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faae80cc170 0x7faae815df90 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.402+0000 7faaf00b2700 1 -- 172.21.15.1:0/3629073035 >> 172.21.15.1:0/3629073035 conn(0x7faae806b860 msgr2=0x7faae80afe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:19.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.402+0000 7faaf00b2700 1 -- 172.21.15.1:0/3629073035 shutdown_connections 2024-03-18T20:25:19.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.402+0000 7faaf00b2700 1 -- 172.21.15.1:0/3629073035 wait complete. 2024-03-18T20:25:19.419 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:25:19.420 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:25:19.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.544+0000 7f3ecd3b4700 1 Processor -- start 2024-03-18T20:25:19.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.544+0000 7f3ecd3b4700 1 -- start start 2024-03-18T20:25:19.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.545+0000 7f3ecd3b4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ec806c270 0x7f3ec806c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.545+0000 7f3ecd3b4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3ec8160f70 0x7f3ec80cf350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.545+0000 7f3ecd3b4700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec8161e80 con 0x7f3ec80b32b0 2024-03-18T20:25:19.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.545+0000 7f3ecd3b4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec80b20e0 con 0x7f3ec8160f70 2024-03-18T20:25:19.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.545+0000 7f3ecd3b4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec806cb90 con 0x7f3ec806c270 2024-03-18T20:25:19.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.545+0000 7f3ec6ffd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ec806c270 0x7f3ec806c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.545+0000 7f3ec6ffd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ec806c270 0x7f3ec806c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:59628/0 (socket says 172.21.15.1:59628) 2024-03-18T20:25:19.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.545+0000 7f3ec77fe700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3ec8160f70 0x7f3ec80cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.545+0000 7f3ec6ffd700 1 -- 172.21.15.1:0/3463008424 learned_addr learned my addr 172.21.15.1:0/3463008424 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:19.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.546+0000 7f3ec6ffd700 1 -- 172.21.15.1:0/3463008424 >> v2:172.21.15.165:3300/0 conn(0x7f3ec8160f70 msgr2=0x7f3ec80cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:19.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.546+0000 7f3ec6ffd700 1 --2- 172.21.15.1:0/3463008424 >> v2:172.21.15.165:3300/0 conn(0x7f3ec8160f70 0x7f3ec80cf350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.546+0000 7f3ec6ffd700 1 -- 172.21.15.1:0/3463008424 >> v1:172.21.15.165:6789/0 conn(0x7f3ec80b32b0 legacy=0x7f3ec80b19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.546+0000 7f3ec6ffd700 1 -- 172.21.15.1:0/3463008424 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec81618c0 con 0x7f3ec806c270 2024-03-18T20:25:19.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.546+0000 7f3ec77fe700 1 --2- 172.21.15.1:0/3463008424 >> v2:172.21.15.165:3300/0 conn(0x7f3ec8160f70 0x7f3ec80cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:19.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.546+0000 7f3ec6ffd700 1 --2- 172.21.15.1:0/3463008424 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ec806c270 0x7f3ec806c650 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3eb80098f0 tx=0x7f3eb8009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb975e4b2f05b888 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:19.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.546+0000 7f3ec5ffb700 1 -- 172.21.15.1:0/3463008424 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8011830 con 0x7f3ec806c270 2024-03-18T20:25:19.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.546+0000 7f3ec5ffb700 1 -- 172.21.15.1:0/3463008424 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb8011990 con 0x7f3ec806c270 2024-03-18T20:25:19.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.546+0000 7f3ec5ffb700 1 -- 172.21.15.1:0/3463008424 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8011d20 con 0x7f3ec806c270 2024-03-18T20:25:19.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.547+0000 7f3ecd3b4700 1 -- 172.21.15.1:0/3463008424 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ec806c270 msgr2=0x7f3ec806c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.547+0000 7f3ecd3b4700 1 --2- 172.21.15.1:0/3463008424 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ec806c270 0x7f3ec806c650 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3eb80098f0 tx=0x7f3eb8009c00 comp rx=0 tx=0).stop 2024-03-18T20:25:19.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.547+0000 7f3ecd3b4700 1 -- 172.21.15.1:0/3463008424 shutdown_connections 2024-03-18T20:25:19.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.547+0000 7f3ecd3b4700 1 --2- 172.21.15.1:0/3463008424 >> v2:172.21.15.165:3300/0 conn(0x7f3ec8160f70 0x7f3ec80cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.547+0000 7f3ecd3b4700 1 --2- 172.21.15.1:0/3463008424 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ec806c270 0x7f3ec806c650 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.547+0000 7f3ecd3b4700 1 -- 172.21.15.1:0/3463008424 >> 172.21.15.1:0/3463008424 conn(0x7f3ec8099a40 msgr2=0x7f3ec8099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:19.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.547+0000 7f3ecd3b4700 1 -- 172.21.15.1:0/3463008424 shutdown_connections 2024-03-18T20:25:19.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.547+0000 7f3ecd3b4700 1 -- 172.21.15.1:0/3463008424 wait complete. 2024-03-18T20:25:19.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.548+0000 7f3ecd3b4700 1 Processor -- start 2024-03-18T20:25:19.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.548+0000 7f3ecd3b4700 1 -- start start 2024-03-18T20:25:19.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.549+0000 7f3ecd3b4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec806c270 0x7f3ec815a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.549+0000 7f3ecd3b4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ec80b32b0 0x7f3ec80c9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.549+0000 7f3ecd3b4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec8160f70 0x7f3ec815f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.549+0000 7f3ecd3b4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec80c5bd0 con 0x7f3ec8160f70 2024-03-18T20:25:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.549+0000 7f3ecd3b4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec80c4f50 con 0x7f3ec806c270 2024-03-18T20:25:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.549+0000 7f3ecd3b4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec80c5230 con 0x7f3ec80b32b0 2024-03-18T20:25:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.549+0000 7f3ec77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec8160f70 0x7f3ec815f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.549+0000 7f3ec77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec8160f70 0x7f3ec815f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34262/0 (socket says 172.21.15.1:34262) 2024-03-18T20:25:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.549+0000 7f3ec77fe700 1 -- 172.21.15.1:0/1484763713 learned_addr learned my addr 172.21.15.1:0/1484763713 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.549+0000 7f3ec6ffd700 1 --2- 172.21.15.1:0/1484763713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec806c270 0x7f3ec815a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.549+0000 7f3ec67fc700 1 --2- 172.21.15.1:0/1484763713 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ec80b32b0 0x7f3ec80c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.549+0000 7f3ec77fe700 1 -- 172.21.15.1:0/1484763713 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ec80b32b0 msgr2=0x7f3ec80c9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.549+0000 7f3ec77fe700 1 --2- 172.21.15.1:0/1484763713 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ec80b32b0 0x7f3ec80c9240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.549+0000 7f3ec77fe700 1 -- 172.21.15.1:0/1484763713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec806c270 msgr2=0x7f3ec815a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.549+0000 7f3ec77fe700 1 --2- 172.21.15.1:0/1484763713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec806c270 0x7f3ec815a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.549+0000 7f3ec77fe700 1 -- 172.21.15.1:0/1484763713 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec0009a70 con 0x7f3ec8160f70 2024-03-18T20:25:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.549+0000 7f3ec6ffd700 1 --2- 172.21.15.1:0/1484763713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec806c270 0x7f3ec815a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.550+0000 7f3ec77fe700 1 --2- 172.21.15.1:0/1484763713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec8160f70 0x7f3ec815f440 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000efc0 tx=0x7f3ec000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.550+0000 7f3ebf7fe700 1 -- 172.21.15.1:0/1484763713 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec000ca90 con 0x7f3ec8160f70 2024-03-18T20:25:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.550+0000 7f3ebf7fe700 1 -- 172.21.15.1:0/1484763713 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec000f040 con 0x7f3ec8160f70 2024-03-18T20:25:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.550+0000 7f3ecd3b4700 1 -- 172.21.15.1:0/1484763713 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb8009550 con 0x7f3ec8160f70 2024-03-18T20:25:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.550+0000 7f3ebf7fe700 1 -- 172.21.15.1:0/1484763713 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec000ce10 con 0x7f3ec8160f70 2024-03-18T20:25:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.550+0000 7f3ecd3b4700 1 -- 172.21.15.1:0/1484763713 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec80c57f0 con 0x7f3ec8160f70 2024-03-18T20:25:19.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.551+0000 7f3ecd3b4700 1 -- 172.21.15.1:0/1484763713 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec8067b90 con 0x7f3ec8160f70 2024-03-18T20:25:19.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.551+0000 7f3ebf7fe700 1 -- 172.21.15.1:0/1484763713 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3ec0014480 con 0x7f3ec8160f70 2024-03-18T20:25:19.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.552+0000 7f3ebf7fe700 1 --2- 172.21.15.1:0/1484763713 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3ea806adb0 0x7f3ea806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.552+0000 7f3ebf7fe700 1 -- 172.21.15.1:0/1484763713 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f3ec0089ac0 con 0x7f3ec8160f70 2024-03-18T20:25:19.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.552+0000 7f3ec6ffd700 1 --2- 172.21.15.1:0/1484763713 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3ea806adb0 0x7f3ea806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.552+0000 7f3ec6ffd700 1 --2- 172.21.15.1:0/1484763713 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3ea806adb0 0x7f3ea806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8014040 tx=0x7f3eb8008030 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:19.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.556+0000 7f3ebf7fe700 1 -- 172.21.15.1:0/1484763713 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3ec0054b90 con 0x7f3ec8160f70 2024-03-18T20:25:19.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.761+0000 7f3ecd3b4700 1 -- 172.21.15.1:0/1484763713 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3ec80c5d10 con 0x7f3ec8160f70 2024-03-18T20:25:19.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.761+0000 7f3ebf7fe700 1 -- 172.21.15.1:0/1484763713 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v228) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3ec0009d70 con 0x7f3ec8160f70 2024-03-18T20:25:19.765 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:25:19.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.763+0000 7f3ecd3b4700 1 -- 172.21.15.1:0/1484763713 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3ea806adb0 msgr2=0x7f3ea806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.763+0000 7f3ecd3b4700 1 --2- 172.21.15.1:0/1484763713 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3ea806adb0 0x7f3ea806d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8014040 tx=0x7f3eb8008030 comp rx=0 tx=0).stop 2024-03-18T20:25:19.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.763+0000 7f3ecd3b4700 1 -- 172.21.15.1:0/1484763713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec8160f70 msgr2=0x7f3ec815f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.763+0000 7f3ecd3b4700 1 --2- 172.21.15.1:0/1484763713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec8160f70 0x7f3ec815f440 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000efc0 tx=0x7f3ec000c3a0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.763+0000 7f3ecd3b4700 1 -- 172.21.15.1:0/1484763713 shutdown_connections 2024-03-18T20:25:19.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.763+0000 7f3ecd3b4700 1 --2- 172.21.15.1:0/1484763713 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3ea806adb0 0x7f3ea806d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.763+0000 7f3ecd3b4700 1 --2- 172.21.15.1:0/1484763713 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec8160f70 0x7f3ec815f440 unknown :-1 s=CLOSED pgs=2492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.763+0000 7f3ecd3b4700 1 --2- 172.21.15.1:0/1484763713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec806c270 0x7f3ec815a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.763+0000 7f3ecd3b4700 1 --2- 172.21.15.1:0/1484763713 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ec80b32b0 0x7f3ec80c9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.763+0000 7f3ecd3b4700 1 -- 172.21.15.1:0/1484763713 >> 172.21.15.1:0/1484763713 conn(0x7f3ec8099a40 msgr2=0x7f3ec809a030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:19.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.763+0000 7f3ecd3b4700 1 -- 172.21.15.1:0/1484763713 shutdown_connections 2024-03-18T20:25:19.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.763+0000 7f3ecd3b4700 1 -- 172.21.15.1:0/1484763713 wait complete. 2024-03-18T20:25:19.780 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T20:25:19.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.900+0000 7fabc4686700 1 Processor -- start 2024-03-18T20:25:19.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.900+0000 7fabc4686700 1 -- start start 2024-03-18T20:25:19.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.901+0000 7fabc4686700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fabbc0ad940 0x7fabbc160e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.901+0000 7fabc4686700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fabbc0ac8b0 0x7fabbc0b34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.901+0000 7fabc4686700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabbc161e30 con 0x7fabbc15d370 2024-03-18T20:25:19.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.901+0000 7fabc4686700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabbc0b3050 con 0x7fabbc0ad940 2024-03-18T20:25:19.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.901+0000 7fabc4686700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabbc15d750 con 0x7fabbc0ac8b0 2024-03-18T20:25:19.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.901+0000 7fabc1c21700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fabbc0ad940 0x7fabbc160e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.901+0000 7fabc2422700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fabbc15d370 0x7fabbc0ad230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35136/0 (socket says 172.21.15.1:35136) 2024-03-18T20:25:19.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.901+0000 7fabc1c21700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fabbc0ad940 0x7fabbc160e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34268/0 (socket says 172.21.15.1:34268) 2024-03-18T20:25:19.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.901+0000 7fabc2422700 1 -- 172.21.15.1:0/4015629428 learned_addr learned my addr 172.21.15.1:0/4015629428 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:19.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.901+0000 7fabc2c23700 1 --2- 172.21.15.1:0/4015629428 >> v2:172.21.15.165:3300/0 conn(0x7fabbc0ac8b0 0x7fabbc0b34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.901+0000 7fabc1c21700 1 -- 172.21.15.1:0/4015629428 >> v2:172.21.15.165:3300/0 conn(0x7fabbc0ac8b0 msgr2=0x7fabbc0b34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:19.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.901+0000 7fabc1c21700 1 --2- 172.21.15.1:0/4015629428 >> v2:172.21.15.165:3300/0 conn(0x7fabbc0ac8b0 0x7fabbc0b34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.901+0000 7fabc1c21700 1 -- 172.21.15.1:0/4015629428 >> v1:172.21.15.1:6789/0 conn(0x7fabbc15d370 legacy=0x7fabbc0ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.901+0000 7fabc1c21700 1 -- 172.21.15.1:0/4015629428 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabbc06f730 con 0x7fabbc0ad940 2024-03-18T20:25:19.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.901+0000 7fabc1420700 1 -- 172.21.15.1:0/4015629428 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686341886 0 0) 0x7fabbc161e30 con 0x7fabbc15d370 2024-03-18T20:25:19.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.901+0000 7fabc1c21700 1 --2- 172.21.15.1:0/4015629428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabbc0ad940 0x7fabbc160e90 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7fabbc06f2b0 tx=0x7fabac009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=850867eb419d9eee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:19.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.901+0000 7fabc1420700 1 -- 172.21.15.1:0/4015629428 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac011830 con 0x7fabbc0ad940 2024-03-18T20:25:19.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.901+0000 7fabc1420700 1 -- 172.21.15.1:0/4015629428 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabac011990 con 0x7fabbc0ad940 2024-03-18T20:25:19.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.901+0000 7fabc1420700 1 -- 172.21.15.1:0/4015629428 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac011d20 con 0x7fabbc0ad940 2024-03-18T20:25:19.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.902+0000 7fabc4686700 1 -- 172.21.15.1:0/4015629428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabbc0ad940 msgr2=0x7fabbc160e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.902+0000 7fabc4686700 1 --2- 172.21.15.1:0/4015629428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabbc0ad940 0x7fabbc160e90 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7fabbc06f2b0 tx=0x7fabac009b20 comp rx=0 tx=0).stop 2024-03-18T20:25:19.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.902+0000 7fabc4686700 1 -- 172.21.15.1:0/4015629428 shutdown_connections 2024-03-18T20:25:19.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.902+0000 7fabc4686700 1 --2- 172.21.15.1:0/4015629428 >> v2:172.21.15.165:3300/0 conn(0x7fabbc0ac8b0 0x7fabbc0b34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.902+0000 7fabc4686700 1 --2- 172.21.15.1:0/4015629428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabbc0ad940 0x7fabbc160e90 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.902+0000 7fabc4686700 1 -- 172.21.15.1:0/4015629428 >> 172.21.15.1:0/4015629428 conn(0x7fabbc06ab70 msgr2=0x7fabbc06af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:19.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.902+0000 7fabc4686700 1 -- 172.21.15.1:0/4015629428 shutdown_connections 2024-03-18T20:25:19.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.902+0000 7fabc4686700 1 -- 172.21.15.1:0/4015629428 wait complete. 2024-03-18T20:25:19.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.903+0000 7fabc4686700 1 Processor -- start 2024-03-18T20:25:19.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.903+0000 7fabc4686700 1 -- start start 2024-03-18T20:25:19.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.904+0000 7fabc4686700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabbc0ac8b0 0x7fabbc0b4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.904+0000 7fabc4686700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabbc0ad940 0x7fabbc149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.904+0000 7fabc4686700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabbc15d370 0x7fabbc14a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.904+0000 7fabc4686700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabbc0cd080 con 0x7fabbc15d370 2024-03-18T20:25:19.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.904+0000 7fabc4686700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabbc0cd360 con 0x7fabbc0ac8b0 2024-03-18T20:25:19.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.904+0000 7fabc4686700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabbc0cd640 con 0x7fabbc0ad940 2024-03-18T20:25:19.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.904+0000 7fabc2c23700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabbc15d370 0x7fabbc14a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.904+0000 7fabc2c23700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabbc15d370 0x7fabbc14a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34284/0 (socket says 172.21.15.1:34284) 2024-03-18T20:25:19.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.904+0000 7fabc2c23700 1 -- 172.21.15.1:0/914220626 learned_addr learned my addr 172.21.15.1:0/914220626 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:19.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.904+0000 7fabc2422700 1 --2- 172.21.15.1:0/914220626 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabbc0ac8b0 0x7fabbc0b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.904+0000 7fabc2c23700 1 -- 172.21.15.1:0/914220626 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabbc0ad940 msgr2=0x7fabbc149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.904+0000 7fabc1c21700 1 --2- 172.21.15.1:0/914220626 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabbc0ad940 0x7fabbc149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.904+0000 7fabc2c23700 1 --2- 172.21.15.1:0/914220626 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabbc0ad940 0x7fabbc149d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.904+0000 7fabc2c23700 1 -- 172.21.15.1:0/914220626 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabbc0ac8b0 msgr2=0x7fabbc0b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:19.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.904+0000 7fabc2c23700 1 --2- 172.21.15.1:0/914220626 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabbc0ac8b0 0x7fabbc0b4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:19.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.904+0000 7fabc2c23700 1 -- 172.21.15.1:0/914220626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabac0094a0 con 0x7fabbc15d370 2024-03-18T20:25:19.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.904+0000 7fabc2422700 1 --2- 172.21.15.1:0/914220626 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabbc0ac8b0 0x7fabbc0b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:19.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.905+0000 7fabc2c23700 1 --2- 172.21.15.1:0/914220626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabbc15d370 0x7fabbc14a270 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7fabb4007e40 tx=0x7fabb400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:19.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.905+0000 7fabb2ffd700 1 -- 172.21.15.1:0/914220626 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb400eea0 con 0x7fabbc15d370 2024-03-18T20:25:19.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.905+0000 7fabb2ffd700 1 -- 172.21.15.1:0/914220626 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb4014420 con 0x7fabbc15d370 2024-03-18T20:25:19.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.905+0000 7fabc4686700 1 -- 172.21.15.1:0/914220626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabbc0cd920 con 0x7fabbc15d370 2024-03-18T20:25:19.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.905+0000 7fabb2ffd700 1 -- 172.21.15.1:0/914220626 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb40147b0 con 0x7fabbc15d370 2024-03-18T20:25:19.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.905+0000 7fabc4686700 1 -- 172.21.15.1:0/914220626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabbc0b4560 con 0x7fabbc15d370 2024-03-18T20:25:19.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.906+0000 7fabc4686700 1 -- 172.21.15.1:0/914220626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabbc0b4720 con 0x7fabbc15d370 2024-03-18T20:25:19.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.906+0000 7fabb2ffd700 1 -- 172.21.15.1:0/914220626 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fabb4014910 con 0x7fabbc15d370 2024-03-18T20:25:19.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.907+0000 7fabb2ffd700 1 --2- 172.21.15.1:0/914220626 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faba006adb0 0x7faba006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:19.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.907+0000 7fabc2422700 1 --2- 172.21.15.1:0/914220626 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faba006adb0 0x7faba006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:19.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.907+0000 7fabb2ffd700 1 -- 172.21.15.1:0/914220626 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 5277+0+0 (secure 0 0 0) 0x7fabb4089bb0 con 0x7fabbc15d370 2024-03-18T20:25:19.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.907+0000 7fabc2422700 1 --2- 172.21.15.1:0/914220626 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faba006adb0 0x7faba006d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fabb8007fd0 tx=0x7fabb8009040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:19.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:19.911+0000 7fabb2ffd700 1 -- 172.21.15.1:0/914220626 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fabb4054d90 con 0x7fabbc15d370 2024-03-18T20:25:20.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.116+0000 7fabc4686700 1 -- 172.21.15.1:0/914220626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fabbc0cde40 con 0x7fabbc15d370 2024-03-18T20:25:20.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.117+0000 7fabb2ffd700 1 -- 172.21.15.1:0/914220626 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v228) v1 ==== 84+0+65 (secure 0 0 0) 0x7fabb40583b0 con 0x7fabbc15d370 2024-03-18T20:25:20.122 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:20.122 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T20:25:20.122 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:20.123 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-18T20:25:20.123 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-18T20:25:20.123 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:20.123 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T20:25:20.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.119+0000 7fabc4686700 1 -- 172.21.15.1:0/914220626 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faba006adb0 msgr2=0x7faba006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.119+0000 7fabc4686700 1 --2- 172.21.15.1:0/914220626 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faba006adb0 0x7faba006d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fabb8007fd0 tx=0x7fabb8009040 comp rx=0 tx=0).stop 2024-03-18T20:25:20.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.119+0000 7fabc4686700 1 -- 172.21.15.1:0/914220626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabbc15d370 msgr2=0x7fabbc14a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.119+0000 7fabc4686700 1 --2- 172.21.15.1:0/914220626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabbc15d370 0x7fabbc14a270 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7fabb4007e40 tx=0x7fabb400e3c0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.119+0000 7fabc4686700 1 -- 172.21.15.1:0/914220626 shutdown_connections 2024-03-18T20:25:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.119+0000 7fabc4686700 1 --2- 172.21.15.1:0/914220626 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faba006adb0 0x7faba006d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.119+0000 7fabc4686700 1 --2- 172.21.15.1:0/914220626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabbc15d370 0x7fabbc14a270 unknown :-1 s=CLOSED pgs=2495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.119+0000 7fabc4686700 1 --2- 172.21.15.1:0/914220626 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabbc0ac8b0 0x7fabbc0b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.119+0000 7fabc4686700 1 --2- 172.21.15.1:0/914220626 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabbc0ad940 0x7fabbc149d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.119+0000 7fabc4686700 1 -- 172.21.15.1:0/914220626 >> 172.21.15.1:0/914220626 conn(0x7fabbc06ab70 msgr2=0x7fabbc1583d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.119+0000 7fabc4686700 1 -- 172.21.15.1:0/914220626 shutdown_connections 2024-03-18T20:25:20.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.119+0000 7fabc4686700 1 -- 172.21.15.1:0/914220626 wait complete. 2024-03-18T20:25:20.135 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T20:25:20.136 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T20:25:20.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.261+0000 7f48f887a700 1 Processor -- start 2024-03-18T20:25:20.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.261+0000 7f48f887a700 1 -- start start 2024-03-18T20:25:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.262+0000 7f48f887a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f48f40ceef0 0x7f48f415c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.262+0000 7f48f887a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f48f40cf9e0 0x7f48f40cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.262+0000 7f48f887a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f40b4250 con 0x7f48f40b3a70 2024-03-18T20:25:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.262+0000 7f48f887a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48f40cb970 con 0x7f48f40cf9e0 2024-03-18T20:25:20.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.262+0000 7f48f887a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f415ca70 con 0x7f48f40ceef0 2024-03-18T20:25:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.262+0000 7f48f2d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f48f40cf9e0 0x7f48f40cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.262+0000 7f48f2d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f48f40cf9e0 0x7f48f40cbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34290/0 (socket says 172.21.15.1:34290) 2024-03-18T20:25:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.262+0000 7f48f2d9d700 1 -- 172.21.15.1:0/2682618615 learned_addr learned my addr 172.21.15.1:0/2682618615 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.263+0000 7f48f259c700 1 --2- 172.21.15.1:0/2682618615 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f48f40ceef0 0x7f48f415c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.263+0000 7f48f2d9d700 1 -- 172.21.15.1:0/2682618615 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f48f40ceef0 msgr2=0x7f48f415c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.263+0000 7f48f2d9d700 1 --2- 172.21.15.1:0/2682618615 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f48f40ceef0 0x7f48f415c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.263+0000 7f48f2d9d700 1 -- 172.21.15.1:0/2682618615 >> v1:172.21.15.165:6789/0 conn(0x7f48f40b3a70 legacy=0x7f48f40cf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.263+0000 7f48f2d9d700 1 -- 172.21.15.1:0/2682618615 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f40b4530 con 0x7f48f40cf9e0 2024-03-18T20:25:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.263+0000 7f48f259c700 1 --2- 172.21.15.1:0/2682618615 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f48f40ceef0 0x7f48f415c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.263+0000 7f48f2d9d700 1 --2- 172.21.15.1:0/2682618615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f40cf9e0 0x7f48f40cbdc0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f48ec009e50 tx=0x7f48ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc3deb67595f74d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.263+0000 7f48f159a700 1 -- 172.21.15.1:0/2682618615 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ec00e070 con 0x7f48f40cf9e0 2024-03-18T20:25:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.263+0000 7f48f159a700 1 -- 172.21.15.1:0/2682618615 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48ec003a20 con 0x7f48f40cf9e0 2024-03-18T20:25:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.263+0000 7f48f159a700 1 -- 172.21.15.1:0/2682618615 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ec003dd0 con 0x7f48f40cf9e0 2024-03-18T20:25:20.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.263+0000 7f48f887a700 1 -- 172.21.15.1:0/2682618615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f40cf9e0 msgr2=0x7f48f40cbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.263+0000 7f48f887a700 1 --2- 172.21.15.1:0/2682618615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f40cf9e0 0x7f48f40cbdc0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f48ec009e50 tx=0x7f48ec00b330 comp rx=0 tx=0).stop 2024-03-18T20:25:20.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.264+0000 7f48f887a700 1 -- 172.21.15.1:0/2682618615 shutdown_connections 2024-03-18T20:25:20.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.264+0000 7f48f887a700 1 --2- 172.21.15.1:0/2682618615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f40cf9e0 0x7f48f40cbdc0 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.264+0000 7f48f887a700 1 --2- 172.21.15.1:0/2682618615 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f48f40ceef0 0x7f48f415c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.264+0000 7f48f887a700 1 -- 172.21.15.1:0/2682618615 >> 172.21.15.1:0/2682618615 conn(0x7f48f406a750 msgr2=0x7f48f406ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:20.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.264+0000 7f48f887a700 1 -- 172.21.15.1:0/2682618615 shutdown_connections 2024-03-18T20:25:20.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.264+0000 7f48f887a700 1 -- 172.21.15.1:0/2682618615 wait complete. 2024-03-18T20:25:20.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.265+0000 7f48f887a700 1 Processor -- start 2024-03-18T20:25:20.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.265+0000 7f48f887a700 1 -- start start 2024-03-18T20:25:20.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.266+0000 7f48f887a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48f40b3a70 0x7f48f415c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.266+0000 7f48f887a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f40ceef0 0x7f48f413d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.266+0000 7f48f887a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f48f40cf9e0 0x7f48f413fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.266+0000 7f48f887a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f413e1b0 con 0x7f48f40ceef0 2024-03-18T20:25:20.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.266+0000 7f48f887a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f413de10 con 0x7f48f40b3a70 2024-03-18T20:25:20.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.266+0000 7f48f887a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f413dae0 con 0x7f48f40cf9e0 2024-03-18T20:25:20.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.266+0000 7f48f1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f40ceef0 0x7f48f413d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.266+0000 7f48f1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f40ceef0 0x7f48f413d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34296/0 (socket says 172.21.15.1:34296) 2024-03-18T20:25:20.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.266+0000 7f48f1d9b700 1 -- 172.21.15.1:0/2963177108 learned_addr learned my addr 172.21.15.1:0/2963177108 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:20.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.267+0000 7f48f2d9d700 1 --2- 172.21.15.1:0/2963177108 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f48f40cf9e0 0x7f48f413fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.267+0000 7f48f259c700 1 --2- 172.21.15.1:0/2963177108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48f40b3a70 0x7f48f415c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.267+0000 7f48f1d9b700 1 -- 172.21.15.1:0/2963177108 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f48f40cf9e0 msgr2=0x7f48f413fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.267+0000 7f48f1d9b700 1 --2- 172.21.15.1:0/2963177108 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f48f40cf9e0 0x7f48f413fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.267+0000 7f48f1d9b700 1 -- 172.21.15.1:0/2963177108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48f40b3a70 msgr2=0x7f48f415c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:20.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.267+0000 7f48f1d9b700 1 --2- 172.21.15.1:0/2963177108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48f40b3a70 0x7f48f415c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:20.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.267+0000 7f48f1d9b700 1 -- 172.21.15.1:0/2963177108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ec009b30 con 0x7f48f40ceef0 2024-03-18T20:25:20.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.267+0000 7f48f1d9b700 1 --2- 172.21.15.1:0/2963177108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f40ceef0 0x7f48f413d5a0 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f48e400c4d0 tx=0x7f48e400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:20.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.267+0000 7f48eaffd700 1 -- 172.21.15.1:0/2963177108 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e4011790 con 0x7f48f40ceef0 2024-03-18T20:25:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.267+0000 7f48eaffd700 1 -- 172.21.15.1:0/2963177108 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e40118f0 con 0x7f48f40ceef0 2024-03-18T20:25:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.267+0000 7f48f887a700 1 -- 172.21.15.1:0/2963177108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f41406f0 con 0x7f48f40ceef0 2024-03-18T20:25:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.267+0000 7f48eaffd700 1 -- 172.21.15.1:0/2963177108 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e4011ca0 con 0x7f48f40ceef0 2024-03-18T20:25:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.267+0000 7f48f887a700 1 -- 172.21.15.1:0/2963177108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f4140cb0 con 0x7f48f40ceef0 2024-03-18T20:25:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.268+0000 7f48f887a700 1 -- 172.21.15.1:0/2963177108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48f40676a0 con 0x7f48f40ceef0 2024-03-18T20:25:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.269+0000 7f48eaffd700 1 -- 172.21.15.1:0/2963177108 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f48e401c460 con 0x7f48f40ceef0 2024-03-18T20:25:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.269+0000 7f48eaffd700 1 --2- 172.21.15.1:0/2963177108 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f48d406adb0 0x7f48d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.269+0000 7f48eaffd700 1 -- 172.21.15.1:0/2963177108 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f48e4089100 con 0x7f48f40ceef0 2024-03-18T20:25:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.269+0000 7f48f259c700 1 --2- 172.21.15.1:0/2963177108 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f48d406adb0 0x7f48d406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.270+0000 7f48f259c700 1 --2- 172.21.15.1:0/2963177108 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f48d406adb0 0x7f48d406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f48dc006fd0 tx=0x7f48dc008040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.274+0000 7f48eaffd700 1 -- 172.21.15.1:0/2963177108 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f48e4054380 con 0x7f48f40ceef0 2024-03-18T20:25:20.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:20.477+0000 7f48f887a700 1 -- 172.21.15.1:0/2963177108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f48f40cc420 con 0x7f48f40ceef0 2024-03-18T20:25:21.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.225+0000 7f48eaffd700 1 -- 172.21.15.1:0/2963177108 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v229) v1 ==== 138+0+0 (secure 0 0 0) 0x7f48e40579a0 con 0x7f48f40ceef0 2024-03-18T20:25:21.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.228+0000 7f48f887a700 1 -- 172.21.15.1:0/2963177108 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f48d406adb0 msgr2=0x7f48d406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.228+0000 7f48f887a700 1 --2- 172.21.15.1:0/2963177108 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f48d406adb0 0x7f48d406d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f48dc006fd0 tx=0x7f48dc008040 comp rx=0 tx=0).stop 2024-03-18T20:25:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.228+0000 7f48f887a700 1 -- 172.21.15.1:0/2963177108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f40ceef0 msgr2=0x7f48f413d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.228+0000 7f48f887a700 1 --2- 172.21.15.1:0/2963177108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f40ceef0 0x7f48f413d5a0 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f48e400c4d0 tx=0x7f48e400c890 comp rx=0 tx=0).stop 2024-03-18T20:25:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.229+0000 7f48f887a700 1 -- 172.21.15.1:0/2963177108 shutdown_connections 2024-03-18T20:25:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.229+0000 7f48f887a700 1 --2- 172.21.15.1:0/2963177108 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f48d406adb0 0x7f48d406d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.229+0000 7f48f887a700 1 --2- 172.21.15.1:0/2963177108 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f48f40cf9e0 0x7f48f413fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.229+0000 7f48f887a700 1 --2- 172.21.15.1:0/2963177108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48f40b3a70 0x7f48f415c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.229+0000 7f48f887a700 1 --2- 172.21.15.1:0/2963177108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f40ceef0 0x7f48f413d5a0 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:21.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.229+0000 7f48f887a700 1 -- 172.21.15.1:0/2963177108 >> 172.21.15.1:0/2963177108 conn(0x7f48f406a750 msgr2=0x7f48f40cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:21.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.229+0000 7f48f887a700 1 -- 172.21.15.1:0/2963177108 shutdown_connections 2024-03-18T20:25:21.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.229+0000 7f48f887a700 1 -- 172.21.15.1:0/2963177108 wait complete. 2024-03-18T20:25:21.238 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-18T20:25:21.252 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T20:25:21.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.388+0000 7f1281cf5700 1 Processor -- start 2024-03-18T20:25:21.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.388+0000 7f1281cf5700 1 -- start start 2024-03-18T20:25:21.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.388+0000 7f1281cf5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f127c0cfa90 0x7f127c06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f1281cf5700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f127c0cf600 con 0x7f127c15d5a0 2024-03-18T20:25:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f1281cf5700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f127c0ce720 con 0x7f127c06ac90 2024-03-18T20:25:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f1281cf5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f127c06b140 con 0x7f127c0cfa90 2024-03-18T20:25:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f127b7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f127c15d5a0 0x7f127c0ce010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35144/0 (socket says 172.21.15.1:35144) 2024-03-18T20:25:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f127b7fe700 1 -- 172.21.15.1:0/2289593515 learned_addr learned my addr 172.21.15.1:0/2289593515 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f127affd700 1 --2- 172.21.15.1:0/2289593515 >> v2:172.21.15.165:3300/0 conn(0x7f127c0cfa90 0x7f127c06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f127a7fc700 1 -- 172.21.15.1:0/2289593515 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075274244 0 0) 0x7f127c0cf600 con 0x7f127c15d5a0 2024-03-18T20:25:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f127a7fc700 1 -- 172.21.15.1:0/2289593515 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12640035b0 con 0x7f127c15d5a0 2024-03-18T20:25:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f127a7fc700 1 -- 172.21.15.1:0/2289593515 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2064501616 0 0) 0x7f12640035b0 con 0x7f127c15d5a0 2024-03-18T20:25:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f127a7fc700 1 -- 172.21.15.1:0/2289593515 >> v2:172.21.15.165:3300/0 conn(0x7f127c0cfa90 msgr2=0x7f127c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:21.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f127a7fc700 1 --2- 172.21.15.1:0/2289593515 >> v2:172.21.15.165:3300/0 conn(0x7f127c0cfa90 0x7f127c06a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:21.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f127a7fc700 1 -- 172.21.15.1:0/2289593515 >> v1:172.21.15.165:6789/0 conn(0x7f127c06ac90 legacy=0x7f127c167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:21.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f127a7fc700 1 -- 172.21.15.1:0/2289593515 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f127c0ce9f0 con 0x7f127c15d5a0 2024-03-18T20:25:21.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f127a7fc700 1 -- 172.21.15.1:0/2289593515 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f126c0017f0 con 0x7f127c15d5a0 2024-03-18T20:25:21.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f127a7fc700 1 -- 172.21.15.1:0/2289593515 >> v1:172.21.15.1:6789/0 conn(0x7f127c15d5a0 legacy=0x7f127c0ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:21.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f127a7fc700 1 --2- 172.21.15.1:0/2289593515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1264003a00 0x7f1264003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:21.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f127a7fc700 1 --2- 172.21.15.1:0/2289593515 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1264004850 0x7f1264004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:21.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f127a7fc700 1 --2- 172.21.15.1:0/2289593515 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1264005210 0x7f126400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:21.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f127a7fc700 1 -- 172.21.15.1:0/2289593515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127c06b140 con 0x7f1264003a00 2024-03-18T20:25:21.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f127a7fc700 1 -- 172.21.15.1:0/2289593515 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126400c0b0 con 0x7f1264005210 2024-03-18T20:25:21.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f127a7fc700 1 -- 172.21.15.1:0/2289593515 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126400c270 con 0x7f1264004850 2024-03-18T20:25:21.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.389+0000 7f127b7fe700 1 --2- 172.21.15.1:0/2289593515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1264003a00 0x7f1264003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:21.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.390+0000 7f127bfff700 1 --2- 172.21.15.1:0/2289593515 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1264005210 0x7f126400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:21.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.390+0000 7f127affd700 1 --2- 172.21.15.1:0/2289593515 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1264004850 0x7f1264004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:21.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.390+0000 7f127b7fe700 1 -- 172.21.15.1:0/2289593515 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1264004850 msgr2=0x7f1264004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:21.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.390+0000 7f127b7fe700 1 --2- 172.21.15.1:0/2289593515 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1264004850 0x7f1264004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:21.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.390+0000 7f127b7fe700 1 -- 172.21.15.1:0/2289593515 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1264005210 msgr2=0x7f126400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:21.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.390+0000 7f127b7fe700 1 --2- 172.21.15.1:0/2289593515 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1264005210 0x7f126400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:21.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.390+0000 7f127b7fe700 1 -- 172.21.15.1:0/2289593515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12640035b0 con 0x7f1264003a00 2024-03-18T20:25:21.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.390+0000 7f127bfff700 1 -- 172.21.15.1:0/2289593515 reap_dead start 2024-03-18T20:25:21.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.390+0000 7f127b7fe700 1 --2- 172.21.15.1:0/2289593515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1264003a00 0x7f1264003e20 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f126c007fd0 tx=0x7f1264005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a90e2e5b1ff6084 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:21.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.390+0000 7f127a7fc700 1 -- 172.21.15.1:0/2289593515 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126c00cd80 con 0x7f1264003a00 2024-03-18T20:25:21.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.390+0000 7f127a7fc700 1 -- 172.21.15.1:0/2289593515 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f126c0070d0 con 0x7f1264003a00 2024-03-18T20:25:21.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.390+0000 7f127a7fc700 1 -- 172.21.15.1:0/2289593515 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126c007480 con 0x7f1264003a00 2024-03-18T20:25:21.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.391+0000 7f1281cf5700 1 -- 172.21.15.1:0/2289593515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1264003a00 msgr2=0x7f1264003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:21.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.391+0000 7f1281cf5700 1 --2- 172.21.15.1:0/2289593515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1264003a00 0x7f1264003e20 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f126c007fd0 tx=0x7f1264005880 comp rx=0 tx=0).stop 2024-03-18T20:25:21.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.391+0000 7f1281cf5700 1 -- 172.21.15.1:0/2289593515 shutdown_connections 2024-03-18T20:25:21.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.391+0000 7f1281cf5700 1 --2- 172.21.15.1:0/2289593515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1264003a00 0x7f1264003e20 unknown :-1 s=CLOSED pgs=2499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:21.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.391+0000 7f1281cf5700 1 -- 172.21.15.1:0/2289593515 >> 172.21.15.1:0/2289593515 conn(0x7f127c0cc9b0 msgr2=0x7f127c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:21.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.391+0000 7f1281cf5700 1 -- 172.21.15.1:0/2289593515 shutdown_connections 2024-03-18T20:25:21.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.391+0000 7f1281cf5700 1 -- 172.21.15.1:0/2289593515 wait complete. 2024-03-18T20:25:21.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.392+0000 7f1281cf5700 1 Processor -- start 2024-03-18T20:25:21.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.392+0000 7f1281cf5700 1 -- start start 2024-03-18T20:25:21.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.393+0000 7f1281cf5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1264003a00 0x7f127c0c3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:21.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.393+0000 7f1281cf5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f127c1f5ea0 0x7f127c1f6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:21.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.393+0000 7f1281cf5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f127c0b3b50 0x7f127c0b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:21.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.393+0000 7f1281cf5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127c0b5450 con 0x7f127c0b3b50 2024-03-18T20:25:21.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.393+0000 7f1281cf5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127c1f6860 con 0x7f1264003a00 2024-03-18T20:25:21.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.393+0000 7f1281cf5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f127c0b4840 con 0x7f127c1f5ea0 2024-03-18T20:25:21.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.393+0000 7f127bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f127c0b3b50 0x7f127c0b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:21.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.393+0000 7f127bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f127c0b3b50 0x7f127c0b3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34304/0 (socket says 172.21.15.1:34304) 2024-03-18T20:25:21.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.393+0000 7f127bfff700 1 -- 172.21.15.1:0/2263442122 learned_addr learned my addr 172.21.15.1:0/2263442122 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:21.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.393+0000 7f127b7fe700 1 --2- 172.21.15.1:0/2263442122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1264003a00 0x7f127c0c3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:21.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.393+0000 7f127affd700 1 --2- 172.21.15.1:0/2263442122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f127c1f5ea0 0x7f127c1f6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:21.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.393+0000 7f127bfff700 1 -- 172.21.15.1:0/2263442122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f127c1f5ea0 msgr2=0x7f127c1f6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:21.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.393+0000 7f127bfff700 1 --2- 172.21.15.1:0/2263442122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f127c1f5ea0 0x7f127c1f6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:21.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.393+0000 7f127bfff700 1 -- 172.21.15.1:0/2263442122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1264003a00 msgr2=0x7f127c0c3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:21.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.393+0000 7f127bfff700 1 --2- 172.21.15.1:0/2263442122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1264003a00 0x7f127c0c3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:21.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.393+0000 7f127bfff700 1 -- 172.21.15.1:0/2263442122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f127c0b4b20 con 0x7f127c0b3b50 2024-03-18T20:25:21.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.394+0000 7f127bfff700 1 --2- 172.21.15.1:0/2263442122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f127c0b3b50 0x7f127c0b3fd0 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f127c0b2a50 tx=0x7f127400fe70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:21.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.394+0000 7f1263fff700 1 -- 172.21.15.1:0/2263442122 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1274003cd0 con 0x7f127c0b3b50 2024-03-18T20:25:21.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.394+0000 7f1263fff700 1 -- 172.21.15.1:0/2263442122 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1274003e30 con 0x7f127c0b3b50 2024-03-18T20:25:21.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.394+0000 7f1281cf5700 1 -- 172.21.15.1:0/2263442122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f127c0b4e00 con 0x7f127c0b3b50 2024-03-18T20:25:21.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.394+0000 7f1263fff700 1 -- 172.21.15.1:0/2263442122 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1274005310 con 0x7f127c0b3b50 2024-03-18T20:25:21.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.394+0000 7f1281cf5700 1 -- 172.21.15.1:0/2263442122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f127c0b4510 con 0x7f127c0b3b50 2024-03-18T20:25:21.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.395+0000 7f1281cf5700 1 -- 172.21.15.1:0/2263442122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f127c073350 con 0x7f127c0b3b50 2024-03-18T20:25:21.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.395+0000 7f1263fff700 1 -- 172.21.15.1:0/2263442122 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1274005470 con 0x7f127c0b3b50 2024-03-18T20:25:21.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.396+0000 7f1263fff700 1 --2- 172.21.15.1:0/2263442122 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f126806adb0 0x7f126806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:21.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.396+0000 7f1263fff700 1 -- 172.21.15.1:0/2263442122 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5647+0+0 (secure 0 0 0) 0x7f12740894e0 con 0x7f127c0b3b50 2024-03-18T20:25:21.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.396+0000 7f127b7fe700 1 --2- 172.21.15.1:0/2263442122 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f126806adb0 0x7f126806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:21.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.396+0000 7f127b7fe700 1 --2- 172.21.15.1:0/2263442122 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f126806adb0 0x7f126806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f126c000c00 tx=0x7f126c00a000 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:21.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.401+0000 7f1263fff700 1 -- 172.21.15.1:0/2263442122 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1274054560 con 0x7f127c0b3b50 2024-03-18T20:25:21.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:21.603+0000 7f1281cf5700 1 -- 172.21.15.1:0/2263442122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f127c1402f0 con 0x7f127c0b3b50 2024-03-18T20:25:22.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.235+0000 7f1263fff700 1 -- 172.21.15.1:0/2263442122 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v230) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1274057b80 con 0x7f127c0b3b50 2024-03-18T20:25:22.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.241+0000 7f1281cf5700 1 -- 172.21.15.1:0/2263442122 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f126806adb0 msgr2=0x7f126806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:22.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.241+0000 7f1281cf5700 1 --2- 172.21.15.1:0/2263442122 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f126806adb0 0x7f126806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f126c000c00 tx=0x7f126c00a000 comp rx=0 tx=0).stop 2024-03-18T20:25:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.241+0000 7f1281cf5700 1 -- 172.21.15.1:0/2263442122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f127c0b3b50 msgr2=0x7f127c0b3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.241+0000 7f1281cf5700 1 --2- 172.21.15.1:0/2263442122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f127c0b3b50 0x7f127c0b3fd0 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f127c0b2a50 tx=0x7f127400fe70 comp rx=0 tx=0).stop 2024-03-18T20:25:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.241+0000 7f1281cf5700 1 -- 172.21.15.1:0/2263442122 shutdown_connections 2024-03-18T20:25:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.241+0000 7f1281cf5700 1 --2- 172.21.15.1:0/2263442122 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f126806adb0 0x7f126806d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.241+0000 7f1281cf5700 1 --2- 172.21.15.1:0/2263442122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f127c0b3b50 0x7f127c0b3fd0 unknown :-1 s=CLOSED pgs=2500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.241+0000 7f1281cf5700 1 --2- 172.21.15.1:0/2263442122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1264003a00 0x7f127c0c3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.241+0000 7f1281cf5700 1 --2- 172.21.15.1:0/2263442122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f127c1f5ea0 0x7f127c1f6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.241+0000 7f1281cf5700 1 -- 172.21.15.1:0/2263442122 >> 172.21.15.1:0/2263442122 conn(0x7f127c0cc9b0 msgr2=0x7f127c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:22.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.242+0000 7f1281cf5700 1 -- 172.21.15.1:0/2263442122 shutdown_connections 2024-03-18T20:25:22.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.242+0000 7f1281cf5700 1 -- 172.21.15.1:0/2263442122 wait complete. 2024-03-18T20:25:22.252 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-18T20:25:22.270 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T20:25:22.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.400+0000 7f2d4a072700 1 Processor -- start 2024-03-18T20:25:22.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.400+0000 7f2d4a072700 1 -- start start 2024-03-18T20:25:22.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d4a072700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2d440724e0 0x7f2d4415b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:22.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d4a072700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d4415b6f0 con 0x7f2d44071490 2024-03-18T20:25:22.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d4a072700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d440cb510 con 0x7f2d4415a380 2024-03-18T20:25:22.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d4a072700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d440ca900 con 0x7f2d440724e0 2024-03-18T20:25:22.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d437fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2d44071490 0x7f2d44071870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35148/0 (socket says 172.21.15.1:35148) 2024-03-18T20:25:22.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d437fe700 1 -- 172.21.15.1:0/1737143428 learned_addr learned my addr 172.21.15.1:0/1737143428 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:22.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d427fc700 1 -- 172.21.15.1:0/1737143428 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958164608 0 0) 0x7f2d34001110 con 0x7f2d44071490 2024-03-18T20:25:22.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d427fc700 1 -- 172.21.15.1:0/1737143428 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d2c0035b0 con 0x7f2d44071490 2024-03-18T20:25:22.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d427fc700 1 -- 172.21.15.1:0/1737143428 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2611943853 0 0) 0x7f2d2c0035b0 con 0x7f2d44071490 2024-03-18T20:25:22.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d427fc700 1 -- 172.21.15.1:0/1737143428 >> v2:172.21.15.165:3300/0 conn(0x7f2d440724e0 msgr2=0x7f2d4415b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:22.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d427fc700 1 --2- 172.21.15.1:0/1737143428 >> v2:172.21.15.165:3300/0 conn(0x7f2d440724e0 0x7f2d4415b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d427fc700 1 -- 172.21.15.1:0/1737143428 >> v1:172.21.15.165:6789/0 conn(0x7f2d4415a380 legacy=0x7f2d4415baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:22.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d427fc700 1 -- 172.21.15.1:0/1737143428 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d440cabe0 con 0x7f2d44071490 2024-03-18T20:25:22.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d427fc700 1 -- 172.21.15.1:0/1737143428 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f2d34001800 con 0x7f2d44071490 2024-03-18T20:25:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d427fc700 1 -- 172.21.15.1:0/1737143428 >> v1:172.21.15.1:6789/0 conn(0x7f2d44071490 legacy=0x7f2d44071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d427fc700 1 --2- 172.21.15.1:0/1737143428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d2c003a00 0x7f2d2c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d427fc700 1 --2- 172.21.15.1:0/1737143428 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d2c004850 0x7f2d2c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d427fc700 1 --2- 172.21.15.1:0/1737143428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d2c005410 0x7f2d2c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d427fc700 1 -- 172.21.15.1:0/1737143428 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d440ca900 con 0x7f2d2c005410 2024-03-18T20:25:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d427fc700 1 -- 172.21.15.1:0/1737143428 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c00c2d0 con 0x7f2d2c004850 2024-03-18T20:25:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.401+0000 7f2d427fc700 1 -- 172.21.15.1:0/1737143428 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c00c490 con 0x7f2d2c003a00 2024-03-18T20:25:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.402+0000 7f2d43fff700 1 --2- 172.21.15.1:0/1737143428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d2c005410 0x7f2d2c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.402+0000 7f2d42ffd700 1 --2- 172.21.15.1:0/1737143428 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d2c004850 0x7f2d2c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.402+0000 7f2d43fff700 1 -- 172.21.15.1:0/1737143428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d2c003a00 msgr2=0x7f2d2c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.402+0000 7f2d43fff700 1 --2- 172.21.15.1:0/1737143428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d2c003a00 0x7f2d2c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.402+0000 7f2d43fff700 1 -- 172.21.15.1:0/1737143428 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d2c004850 msgr2=0x7f2d2c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.402+0000 7f2d43fff700 1 --2- 172.21.15.1:0/1737143428 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d2c004850 0x7f2d2c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.402+0000 7f2d43fff700 1 -- 172.21.15.1:0/1737143428 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c0035b0 con 0x7f2d2c005410 2024-03-18T20:25:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.402+0000 7f2d43fff700 1 -- 172.21.15.1:0/1737143428 reap_dead start 2024-03-18T20:25:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.402+0000 7f2d43fff700 1 --2- 172.21.15.1:0/1737143428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d2c005410 0x7f2d2c00ba10 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c009e40 tx=0x7f2d2c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e21cf01c2d94b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.402+0000 7f2d427fc700 1 -- 172.21.15.1:0/1737143428 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c0098b0 con 0x7f2d2c005410 2024-03-18T20:25:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.402+0000 7f2d427fc700 1 -- 172.21.15.1:0/1737143428 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d3c003c20 con 0x7f2d2c005410 2024-03-18T20:25:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.402+0000 7f2d427fc700 1 -- 172.21.15.1:0/1737143428 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c00b8a0 con 0x7f2d2c005410 2024-03-18T20:25:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.403+0000 7f2d4a072700 1 -- 172.21.15.1:0/1737143428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d2c005410 msgr2=0x7f2d2c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.403+0000 7f2d4a072700 1 --2- 172.21.15.1:0/1737143428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d2c005410 0x7f2d2c00ba10 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c009e40 tx=0x7f2d2c005880 comp rx=0 tx=0).stop 2024-03-18T20:25:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.403+0000 7f2d4a072700 1 -- 172.21.15.1:0/1737143428 shutdown_connections 2024-03-18T20:25:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.403+0000 7f2d4a072700 1 --2- 172.21.15.1:0/1737143428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d2c005410 0x7f2d2c00ba10 unknown :-1 s=CLOSED pgs=2502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.403+0000 7f2d4a072700 1 -- 172.21.15.1:0/1737143428 >> 172.21.15.1:0/1737143428 conn(0x7f2d440b1aa0 msgr2=0x7f2d440b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.403+0000 7f2d4a072700 1 -- 172.21.15.1:0/1737143428 shutdown_connections 2024-03-18T20:25:22.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.403+0000 7f2d4a072700 1 -- 172.21.15.1:0/1737143428 wait complete. 2024-03-18T20:25:22.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.404+0000 7f2d4a072700 1 Processor -- start 2024-03-18T20:25:22.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.404+0000 7f2d4a072700 1 -- start start 2024-03-18T20:25:22.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.405+0000 7f2d4a072700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d2c005410 0x7f2d440c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:22.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.405+0000 7f2d4a072700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d440c51d0 0x7f2d441e5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:22.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.405+0000 7f2d4a072700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d44155910 0x7f2d440c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:22.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.405+0000 7f2d4a072700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d441e5ab0 con 0x7f2d44155910 2024-03-18T20:25:22.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.405+0000 7f2d4a072700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d440c3c40 con 0x7f2d2c005410 2024-03-18T20:25:22.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.405+0000 7f2d4a072700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d440c3ee0 con 0x7f2d440c51d0 2024-03-18T20:25:22.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.405+0000 7f2d43fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d44155910 0x7f2d440c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:22.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.405+0000 7f2d43fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d44155910 0x7f2d440c5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34332/0 (socket says 172.21.15.1:34332) 2024-03-18T20:25:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.405+0000 7f2d43fff700 1 -- 172.21.15.1:0/3940413232 learned_addr learned my addr 172.21.15.1:0/3940413232 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.405+0000 7f2d42ffd700 1 --2- 172.21.15.1:0/3940413232 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d440c51d0 0x7f2d441e5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.405+0000 7f2d43fff700 1 -- 172.21.15.1:0/3940413232 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d440c51d0 msgr2=0x7f2d441e5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.405+0000 7f2d43fff700 1 --2- 172.21.15.1:0/3940413232 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d440c51d0 0x7f2d441e5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.405+0000 7f2d43fff700 1 -- 172.21.15.1:0/3940413232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d2c005410 msgr2=0x7f2d440c49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.405+0000 7f2d43fff700 1 --2- 172.21.15.1:0/3940413232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d2c005410 0x7f2d440c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.405+0000 7f2d43fff700 1 -- 172.21.15.1:0/3940413232 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d440cbb20 con 0x7f2d44155910 2024-03-18T20:25:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.406+0000 7f2d43fff700 1 --2- 172.21.15.1:0/3940413232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d44155910 0x7f2d440c5b60 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f2d440731a0 tx=0x7f2d3c004be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.406+0000 7f2d2bfff700 1 -- 172.21.15.1:0/3940413232 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c015970 con 0x7f2d44155910 2024-03-18T20:25:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.406+0000 7f2d2bfff700 1 -- 172.21.15.1:0/3940413232 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d3c015af0 con 0x7f2d44155910 2024-03-18T20:25:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.406+0000 7f2d4a072700 1 -- 172.21.15.1:0/3940413232 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d440c41c0 con 0x7f2d44155910 2024-03-18T20:25:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.406+0000 7f2d2bfff700 1 -- 172.21.15.1:0/3940413232 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c015ea0 con 0x7f2d44155910 2024-03-18T20:25:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.406+0000 7f2d4a072700 1 -- 172.21.15.1:0/3940413232 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d441e57a0 con 0x7f2d44155910 2024-03-18T20:25:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.407+0000 7f2d4a072700 1 -- 172.21.15.1:0/3940413232 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d440c46e0 con 0x7f2d44155910 2024-03-18T20:25:22.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.408+0000 7f2d2bfff700 1 -- 172.21.15.1:0/3940413232 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2d3c02a440 con 0x7f2d44155910 2024-03-18T20:25:22.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.408+0000 7f2d2bfff700 1 --2- 172.21.15.1:0/3940413232 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2d3006ada0 0x7f2d3006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:22.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.408+0000 7f2d2bfff700 1 -- 172.21.15.1:0/3940413232 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2d3c011070 con 0x7f2d44155910 2024-03-18T20:25:22.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.408+0000 7f2d437fe700 1 --2- 172.21.15.1:0/3940413232 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2d3006ada0 0x7f2d3006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:22.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.408+0000 7f2d437fe700 1 --2- 172.21.15.1:0/3940413232 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2d3006ada0 0x7f2d3006d260 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2d34000ff0 tx=0x7f2d34001220 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:22.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.412+0000 7f2d2bfff700 1 -- 172.21.15.1:0/3940413232 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2d3c0557f0 con 0x7f2d44155910 2024-03-18T20:25:22.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.662+0000 7f2d4a072700 1 -- 172.21.15.1:0/3940413232 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2d440c46e0 con 0x7f2d44155910 2024-03-18T20:25:22.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.666+0000 7f2d2bfff700 1 -- 172.21.15.1:0/3940413232 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 27 and data pool 28 v101) v1 ==== 174+0+0 (secure 0 0 0) 0x7f2d440c46e0 con 0x7f2d44155910 2024-03-18T20:25:22.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.670+0000 7f2d4a072700 1 -- 172.21.15.1:0/3940413232 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2d3006ada0 msgr2=0x7f2d3006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:22.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.670+0000 7f2d4a072700 1 --2- 172.21.15.1:0/3940413232 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2d3006ada0 0x7f2d3006d260 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2d34000ff0 tx=0x7f2d34001220 comp rx=0 tx=0).stop 2024-03-18T20:25:22.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.670+0000 7f2d4a072700 1 -- 172.21.15.1:0/3940413232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d44155910 msgr2=0x7f2d440c5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:22.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.670+0000 7f2d4a072700 1 --2- 172.21.15.1:0/3940413232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d44155910 0x7f2d440c5b60 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f2d440731a0 tx=0x7f2d3c004be0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.670+0000 7f2d4a072700 1 -- 172.21.15.1:0/3940413232 shutdown_connections 2024-03-18T20:25:22.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.670+0000 7f2d4a072700 1 --2- 172.21.15.1:0/3940413232 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2d3006ada0 0x7f2d3006d260 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.670+0000 7f2d4a072700 1 --2- 172.21.15.1:0/3940413232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d44155910 0x7f2d440c5b60 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.670+0000 7f2d4a072700 1 --2- 172.21.15.1:0/3940413232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d2c005410 0x7f2d440c49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.670+0000 7f2d4a072700 1 --2- 172.21.15.1:0/3940413232 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2d440c51d0 0x7f2d441e5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.670+0000 7f2d4a072700 1 -- 172.21.15.1:0/3940413232 >> 172.21.15.1:0/3940413232 conn(0x7f2d440b1aa0 msgr2=0x7f2d440b0360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:22.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.670+0000 7f2d4a072700 1 -- 172.21.15.1:0/3940413232 shutdown_connections 2024-03-18T20:25:22.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.670+0000 7f2d4a072700 1 -- 172.21.15.1:0/3940413232 wait complete. 2024-03-18T20:25:22.675 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 27 and data pool 28 2024-03-18T20:25:22.695 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:25:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.817+0000 7faebbf29700 1 Processor -- start 2024-03-18T20:25:22.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.818+0000 7faebbf29700 1 -- start start 2024-03-18T20:25:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.818+0000 7faebbf29700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faeb415a2e0 0x7faeb40ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.818+0000 7faebbf29700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb406e270 con 0x7faeb40cc170 2024-03-18T20:25:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.818+0000 7faebbf29700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb40ae050 con 0x7faeb40ae600 2024-03-18T20:25:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.818+0000 7faebbf29700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb406f690 con 0x7faeb415a2e0 2024-03-18T20:25:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.818+0000 7faeba4c6700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7faeb40cc170 0x7faeb406ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35152/0 (socket says 172.21.15.1:35152) 2024-03-18T20:25:22.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.818+0000 7faeba4c6700 1 -- 172.21.15.1:0/681288152 learned_addr learned my addr 172.21.15.1:0/681288152 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.819+0000 7faeb94c4700 1 --2- 172.21.15.1:0/681288152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faeb415a2e0 0x7faeb40ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.819+0000 7faeb8cc3700 1 -- 172.21.15.1:0/681288152 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7874465 0 0) 0x7faeb406e270 con 0x7faeb40cc170 2024-03-18T20:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.819+0000 7faeb8cc3700 1 -- 172.21.15.1:0/681288152 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faea00035b0 con 0x7faeb40cc170 2024-03-18T20:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.819+0000 7faeb8cc3700 1 -- 172.21.15.1:0/681288152 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 109313737 0 0) 0x7faea00035b0 con 0x7faeb40cc170 2024-03-18T20:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.819+0000 7faeb8cc3700 1 -- 172.21.15.1:0/681288152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faeb415a2e0 msgr2=0x7faeb40ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.819+0000 7faeb8cc3700 1 --2- 172.21.15.1:0/681288152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faeb415a2e0 0x7faeb40ad790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.819+0000 7faeb8cc3700 1 -- 172.21.15.1:0/681288152 >> v1:172.21.15.165:6789/0 conn(0x7faeb40ae600 legacy=0x7faeb40ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.819+0000 7faeb8cc3700 1 -- 172.21.15.1:0/681288152 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb40add70 con 0x7faeb40cc170 2024-03-18T20:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.819+0000 7faeb8cc3700 1 -- 172.21.15.1:0/681288152 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7faeb0001670 con 0x7faeb40cc170 2024-03-18T20:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.819+0000 7faeb8cc3700 1 -- 172.21.15.1:0/681288152 >> v1:172.21.15.1:6789/0 conn(0x7faeb40cc170 legacy=0x7faeb406ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.819+0000 7faeb8cc3700 1 --2- 172.21.15.1:0/681288152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faeb415a2e0 0x7faea0003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.819+0000 7faeb8cc3700 1 --2- 172.21.15.1:0/681288152 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faea0003f70 0x7faea0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:22.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.819+0000 7faeb8cc3700 1 --2- 172.21.15.1:0/681288152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faea0004d90 0x7faea0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.819+0000 7faeb8cc3700 1 -- 172.21.15.1:0/681288152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb406f690 con 0x7faea0004d90 2024-03-18T20:25:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.819+0000 7faeb8cc3700 1 -- 172.21.15.1:0/681288152 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea000bb20 con 0x7faea0003f70 2024-03-18T20:25:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.819+0000 7faeb8cc3700 1 -- 172.21.15.1:0/681288152 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea000bce0 con 0x7faeb415a2e0 2024-03-18T20:25:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.819+0000 7faeba4c6700 1 --2- 172.21.15.1:0/681288152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faea0004d90 0x7faea0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.819+0000 7faeb94c4700 1 --2- 172.21.15.1:0/681288152 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faea0003f70 0x7faea0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.820+0000 7faeba4c6700 1 -- 172.21.15.1:0/681288152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faeb415a2e0 msgr2=0x7faea0003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.820+0000 7faeba4c6700 1 --2- 172.21.15.1:0/681288152 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faeb415a2e0 0x7faea0003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.820+0000 7faeba4c6700 1 -- 172.21.15.1:0/681288152 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faea0003f70 msgr2=0x7faea0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.820+0000 7faeba4c6700 1 --2- 172.21.15.1:0/681288152 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faea0003f70 0x7faea0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.820+0000 7faeba4c6700 1 -- 172.21.15.1:0/681288152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea00035b0 con 0x7faea0004d90 2024-03-18T20:25:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.820+0000 7faeba4c6700 1 --2- 172.21.15.1:0/681288152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faea0004d90 0x7faea0005210 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7faeb40cc620 tx=0x7faea0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3789b247040899ca server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.820+0000 7faeb8cc3700 1 -- 172.21.15.1:0/681288152 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb000ce00 con 0x7faea0004d90 2024-03-18T20:25:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.820+0000 7faeb8cc3700 1 -- 172.21.15.1:0/681288152 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb0005960 con 0x7faea0004d90 2024-03-18T20:25:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.820+0000 7faeb8cc3700 1 -- 172.21.15.1:0/681288152 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb0005d10 con 0x7faea0004d90 2024-03-18T20:25:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.820+0000 7faebbf29700 1 -- 172.21.15.1:0/681288152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faea0004d90 msgr2=0x7faea0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.820+0000 7faebbf29700 1 --2- 172.21.15.1:0/681288152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faea0004d90 0x7faea0005210 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7faeb40cc620 tx=0x7faea0005880 comp rx=0 tx=0).stop 2024-03-18T20:25:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.820+0000 7faeba4c6700 1 -- 172.21.15.1:0/681288152 reap_dead start 2024-03-18T20:25:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.820+0000 7faebbf29700 1 -- 172.21.15.1:0/681288152 shutdown_connections 2024-03-18T20:25:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.820+0000 7faebbf29700 1 -- 172.21.15.1:0/681288152 >> 172.21.15.1:0/681288152 conn(0x7faeb406b860 msgr2=0x7faeb406bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.821+0000 7faebbf29700 1 -- 172.21.15.1:0/681288152 shutdown_connections 2024-03-18T20:25:22.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.821+0000 7faebbf29700 1 -- 172.21.15.1:0/681288152 wait complete. 2024-03-18T20:25:22.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.822+0000 7faebbf29700 1 Processor -- start 2024-03-18T20:25:22.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.822+0000 7faebbf29700 1 -- start start 2024-03-18T20:25:22.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.822+0000 7faebbf29700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faeb4143530 0x7faeb4143d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:22.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.822+0000 7faebbf29700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faeb4144290 0x7faeb4145d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:22.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.822+0000 7faebbf29700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faeb41462c0 0x7faeb40cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:22.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.822+0000 7faebbf29700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb41454d0 con 0x7faeb4143530 2024-03-18T20:25:22.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.822+0000 7faebbf29700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb4146740 con 0x7faeb4144290 2024-03-18T20:25:22.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.822+0000 7faebbf29700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb4155130 con 0x7faeb41462c0 2024-03-18T20:25:22.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.823+0000 7faeb9cc5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faeb4143530 0x7faeb4143d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:22.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.823+0000 7faeb9cc5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faeb4143530 0x7faeb4143d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34340/0 (socket says 172.21.15.1:34340) 2024-03-18T20:25:22.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.823+0000 7faeb9cc5700 1 -- 172.21.15.1:0/1764355742 learned_addr learned my addr 172.21.15.1:0/1764355742 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:22.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.823+0000 7faeb94c4700 1 --2- 172.21.15.1:0/1764355742 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faeb4144290 0x7faeb4145d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:22.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.823+0000 7faeba4c6700 1 --2- 172.21.15.1:0/1764355742 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faeb41462c0 0x7faeb40cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:22.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.823+0000 7faeb9cc5700 1 -- 172.21.15.1:0/1764355742 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faeb41462c0 msgr2=0x7faeb40cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:22.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.823+0000 7faeb9cc5700 1 --2- 172.21.15.1:0/1764355742 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faeb41462c0 0x7faeb40cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.823+0000 7faeb9cc5700 1 -- 172.21.15.1:0/1764355742 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faeb4144290 msgr2=0x7faeb4145d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:22.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.823+0000 7faeb9cc5700 1 --2- 172.21.15.1:0/1764355742 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faeb4144290 0x7faeb4145d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:22.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.823+0000 7faeb9cc5700 1 -- 172.21.15.1:0/1764355742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb41448c0 con 0x7faeb4143530 2024-03-18T20:25:22.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.823+0000 7faeb94c4700 1 --2- 172.21.15.1:0/1764355742 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faeb4144290 0x7faeb4145d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:22.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.823+0000 7faeba4c6700 1 --2- 172.21.15.1:0/1764355742 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faeb41462c0 0x7faeb40cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:22.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.823+0000 7faeb9cc5700 1 --2- 172.21.15.1:0/1764355742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faeb4143530 0x7faeb4143d50 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7faeac00cb20 tx=0x7faeac00cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:22.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.823+0000 7faeaa7fc700 1 -- 172.21.15.1:0/1764355742 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac005070 con 0x7faeb4143530 2024-03-18T20:25:22.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.823+0000 7faeaa7fc700 1 -- 172.21.15.1:0/1764355742 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeac0051d0 con 0x7faeb4143530 2024-03-18T20:25:22.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.823+0000 7faeaa7fc700 1 -- 172.21.15.1:0/1764355742 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac005540 con 0x7faeb4143530 2024-03-18T20:25:22.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.823+0000 7faebbf29700 1 -- 172.21.15.1:0/1764355742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb4144ba0 con 0x7faeb4143530 2024-03-18T20:25:22.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.823+0000 7faebbf29700 1 -- 172.21.15.1:0/1764355742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb4145160 con 0x7faeb4143530 2024-03-18T20:25:22.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.824+0000 7faebbf29700 1 -- 172.21.15.1:0/1764355742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb4144670 con 0x7faeb4143530 2024-03-18T20:25:22.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.825+0000 7faeaa7fc700 1 -- 172.21.15.1:0/1764355742 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7faeac0056a0 con 0x7faeb4143530 2024-03-18T20:25:22.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.825+0000 7faeaa7fc700 1 --2- 172.21.15.1:0/1764355742 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fae9806adb0 0x7fae9806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:22.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.825+0000 7faeb94c4700 1 --2- 172.21.15.1:0/1764355742 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fae9806adb0 0x7fae9806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:22.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.825+0000 7faeaa7fc700 1 -- 172.21.15.1:0/1764355742 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6129+0+0 (secure 0 0 0) 0x7faeac088e70 con 0x7faeb4143530 2024-03-18T20:25:22.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.826+0000 7faeb94c4700 1 --2- 172.21.15.1:0/1764355742 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fae9806adb0 0x7fae9806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7faeb41f2430 tx=0x7faea4008040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:22.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:22.830+0000 7faeaa7fc700 1 -- 172.21.15.1:0/1764355742 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faeac053e40 con 0x7faeb4143530 2024-03-18T20:25:23.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.030+0000 7faebbf29700 1 -- 172.21.15.1:0/1764355742 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faeb40676a0 con 0x7faeb4143530 2024-03-18T20:25:23.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.032+0000 7faeaa7fc700 1 -- 172.21.15.1:0/1764355742 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v231) v1 ==== 74+0+16442 (secure 0 0 0) 0x7faeac017070 con 0x7faeb4143530 2024-03-18T20:25:23.036 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:23.036 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":231,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:25:22.663398+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:25:20.511297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-18T20:25:21.637693+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:23.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.034+0000 7faebbf29700 1 -- 172.21.15.1:0/1764355742 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fae9806adb0 msgr2=0x7fae9806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.034+0000 7faebbf29700 1 --2- 172.21.15.1:0/1764355742 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fae9806adb0 0x7fae9806d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7faeb41f2430 tx=0x7faea4008040 comp rx=0 tx=0).stop 2024-03-18T20:25:23.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.034+0000 7faebbf29700 1 -- 172.21.15.1:0/1764355742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faeb4143530 msgr2=0x7faeb4143d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.034+0000 7faebbf29700 1 --2- 172.21.15.1:0/1764355742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faeb4143530 0x7faeb4143d50 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7faeac00cb20 tx=0x7faeac00cee0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.034+0000 7faebbf29700 1 -- 172.21.15.1:0/1764355742 shutdown_connections 2024-03-18T20:25:23.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.034+0000 7faebbf29700 1 --2- 172.21.15.1:0/1764355742 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fae9806adb0 0x7fae9806d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.034+0000 7faebbf29700 1 --2- 172.21.15.1:0/1764355742 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faeb41462c0 0x7faeb40cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.034+0000 7faebbf29700 1 --2- 172.21.15.1:0/1764355742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faeb4143530 0x7faeb4143d50 unknown :-1 s=CLOSED pgs=2506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.034+0000 7faebbf29700 1 --2- 172.21.15.1:0/1764355742 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faeb4144290 0x7faeb4145d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.034+0000 7faebbf29700 1 -- 172.21.15.1:0/1764355742 >> 172.21.15.1:0/1764355742 conn(0x7faeb406b860 msgr2=0x7faeb406aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:23.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.034+0000 7faebbf29700 1 -- 172.21.15.1:0/1764355742 shutdown_connections 2024-03-18T20:25:23.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.034+0000 7faebbf29700 1 -- 172.21.15.1:0/1764355742 wait complete. 2024-03-18T20:25:23.051 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:25:23.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.173+0000 7f1e7e95c700 1 Processor -- start 2024-03-18T20:25:23.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.173+0000 7f1e7e95c700 1 -- start start 2024-03-18T20:25:23.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.174+0000 7f1e7e95c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e7806a750 0x7f1e7806ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.174+0000 7f1e7e95c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1e7815d6b0 0x7f1e78160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.174+0000 7f1e7e95c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e780acfb0 con 0x7f1e780ad790 2024-03-18T20:25:23.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.174+0000 7f1e7e95c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e780af7a0 con 0x7f1e7815d6b0 2024-03-18T20:25:23.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.174+0000 7f1e7e95c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7806b070 con 0x7f1e7806a750 2024-03-18T20:25:23.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.174+0000 7f1e777fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1e7815d6b0 0x7f1e78160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.174+0000 7f1e777fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1e7815d6b0 0x7f1e78160ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34344/0 (socket says 172.21.15.1:34344) 2024-03-18T20:25:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.174+0000 7f1e777fe700 1 -- 172.21.15.1:0/3561681208 learned_addr learned my addr 172.21.15.1:0/3561681208 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.175+0000 7f1e777fe700 1 -- 172.21.15.1:0/3561681208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e7806a750 msgr2=0x7f1e7806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.175+0000 7f1e77fff700 1 --2- 172.21.15.1:0/3561681208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e7806a750 0x7f1e7806ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.175+0000 7f1e777fe700 1 --2- 172.21.15.1:0/3561681208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e7806a750 0x7f1e7806ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.175+0000 7f1e777fe700 1 -- 172.21.15.1:0/3561681208 >> v1:172.21.15.165:6789/0 conn(0x7f1e780ad790 legacy=0x7f1e78161530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.175+0000 7f1e777fe700 1 -- 172.21.15.1:0/3561681208 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e780b50a0 con 0x7f1e7815d6b0 2024-03-18T20:25:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.175+0000 7f1e77fff700 1 --2- 172.21.15.1:0/3561681208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e7806a750 0x7f1e7806ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.175+0000 7f1e777fe700 1 --2- 172.21.15.1:0/3561681208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7815d6b0 0x7f1e78160ff0 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c0098f0 tx=0x7f1e6c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b109f4a94750cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.175+0000 7f1e76ffd700 1 -- 172.21.15.1:0/3561681208 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c004230 con 0x7f1e7815d6b0 2024-03-18T20:25:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.175+0000 7f1e76ffd700 1 -- 172.21.15.1:0/3561681208 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e6c004390 con 0x7f1e7815d6b0 2024-03-18T20:25:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.175+0000 7f1e76ffd700 1 -- 172.21.15.1:0/3561681208 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c0036a0 con 0x7f1e7815d6b0 2024-03-18T20:25:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.175+0000 7f1e7e95c700 1 -- 172.21.15.1:0/3561681208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7815d6b0 msgr2=0x7f1e78160ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.175+0000 7f1e7e95c700 1 --2- 172.21.15.1:0/3561681208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7815d6b0 0x7f1e78160ff0 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c0098f0 tx=0x7f1e6c009c00 comp rx=0 tx=0).stop 2024-03-18T20:25:23.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.175+0000 7f1e7e95c700 1 -- 172.21.15.1:0/3561681208 shutdown_connections 2024-03-18T20:25:23.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.175+0000 7f1e7e95c700 1 --2- 172.21.15.1:0/3561681208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e7806a750 0x7f1e7806ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.176+0000 7f1e7e95c700 1 --2- 172.21.15.1:0/3561681208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7815d6b0 0x7f1e78160ff0 unknown :-1 s=CLOSED pgs=2507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.176+0000 7f1e7e95c700 1 -- 172.21.15.1:0/3561681208 >> 172.21.15.1:0/3561681208 conn(0x7f1e7806b860 msgr2=0x7f1e7806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:23.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.176+0000 7f1e7e95c700 1 -- 172.21.15.1:0/3561681208 shutdown_connections 2024-03-18T20:25:23.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.176+0000 7f1e7e95c700 1 -- 172.21.15.1:0/3561681208 wait complete. 2024-03-18T20:25:23.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.177+0000 7f1e7e95c700 1 Processor -- start 2024-03-18T20:25:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.177+0000 7f1e7e95c700 1 -- start start 2024-03-18T20:25:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.178+0000 7f1e7e95c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7806a750 0x7f1e781523f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.178+0000 7f1e7e95c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e780ad790 0x7f1e780cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.178+0000 7f1e7e95c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e7815d6b0 0x7f1e78153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.178+0000 7f1e7e95c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7815f9b0 con 0x7f1e7806a750 2024-03-18T20:25:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.178+0000 7f1e7e95c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e780ccf20 con 0x7f1e7815d6b0 2024-03-18T20:25:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.178+0000 7f1e7e95c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7815eda0 con 0x7f1e780ad790 2024-03-18T20:25:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.178+0000 7f1e77fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7806a750 0x7f1e781523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.178+0000 7f1e77fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7806a750 0x7f1e781523f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34360/0 (socket says 172.21.15.1:34360) 2024-03-18T20:25:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.178+0000 7f1e77fff700 1 -- 172.21.15.1:0/3135367341 learned_addr learned my addr 172.21.15.1:0/3135367341 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.179+0000 7f1e777fe700 1 --2- 172.21.15.1:0/3135367341 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e780ad790 0x7f1e780cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.179+0000 7f1e7cef9700 1 --2- 172.21.15.1:0/3135367341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e7815d6b0 0x7f1e78153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:23.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.179+0000 7f1e77fff700 1 -- 172.21.15.1:0/3135367341 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e780ad790 msgr2=0x7f1e780cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.179+0000 7f1e77fff700 1 --2- 172.21.15.1:0/3135367341 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e780ad790 0x7f1e780cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.179+0000 7f1e77fff700 1 -- 172.21.15.1:0/3135367341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e7815d6b0 msgr2=0x7f1e78153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.179+0000 7f1e77fff700 1 --2- 172.21.15.1:0/3135367341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e7815d6b0 0x7f1e78153550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.179+0000 7f1e77fff700 1 -- 172.21.15.1:0/3135367341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c009550 con 0x7f1e7806a750 2024-03-18T20:25:23.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.179+0000 7f1e77fff700 1 --2- 172.21.15.1:0/3135367341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7806a750 0x7f1e781523f0 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800c750 tx=0x7f1e6800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:23.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.179+0000 7f1e74ff9700 1 -- 172.21.15.1:0/3135367341 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e68011800 con 0x7f1e7806a750 2024-03-18T20:25:23.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.179+0000 7f1e74ff9700 1 -- 172.21.15.1:0/3135367341 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e68011960 con 0x7f1e7806a750 2024-03-18T20:25:23.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.179+0000 7f1e7e95c700 1 -- 172.21.15.1:0/3135367341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e7815f080 con 0x7f1e7806a750 2024-03-18T20:25:23.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.179+0000 7f1e74ff9700 1 -- 172.21.15.1:0/3135367341 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e68011cd0 con 0x7f1e7806a750 2024-03-18T20:25:23.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.179+0000 7f1e7e95c700 1 -- 172.21.15.1:0/3135367341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e7815f640 con 0x7f1e7806a750 2024-03-18T20:25:23.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.180+0000 7f1e7e95c700 1 -- 172.21.15.1:0/3135367341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e780b4c60 con 0x7f1e7806a750 2024-03-18T20:25:23.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.181+0000 7f1e74ff9700 1 -- 172.21.15.1:0/3135367341 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1e68021350 con 0x7f1e7806a750 2024-03-18T20:25:23.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.181+0000 7f1e74ff9700 1 --2- 172.21.15.1:0/3135367341 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e6406ade0 0x7f1e6406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.181+0000 7f1e777fe700 1 --2- 172.21.15.1:0/3135367341 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e6406ade0 0x7f1e6406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:23.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.181+0000 7f1e74ff9700 1 -- 172.21.15.1:0/3135367341 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f1e68089740 con 0x7f1e7806a750 2024-03-18T20:25:23.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.182+0000 7f1e777fe700 1 --2- 172.21.15.1:0/3135367341 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e6406ade0 0x7f1e6406d2a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c009130 tx=0x7f1e6c008030 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:23.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.186+0000 7f1e74ff9700 1 -- 172.21.15.1:0/3135367341 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1e68054550 con 0x7f1e7806a750 2024-03-18T20:25:23.220 INFO:tasks.ceph.mds.d.smithi165.stderr:2024-03-18T20:25:23.216+0000 7f52f3512700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T20:25:23.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.390+0000 7f1e7e95c700 1 -- 172.21.15.1:0/3135367341 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e780676a0 con 0x7f1e7806a750 2024-03-18T20:25:23.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.391+0000 7f1e74ff9700 1 -- 172.21.15.1:0/3135367341 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v231) v1 ==== 74+0+16442 (secure 0 0 0) 0x7f1e68057b70 con 0x7f1e7806a750 2024-03-18T20:25:23.395 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:23.395 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":231,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:25:22.663398+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:25:20.511297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-18T20:25:21.637693+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.393+0000 7f1e7e95c700 1 -- 172.21.15.1:0/3135367341 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e6406ade0 msgr2=0x7f1e6406d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.393+0000 7f1e7e95c700 1 --2- 172.21.15.1:0/3135367341 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e6406ade0 0x7f1e6406d2a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c009130 tx=0x7f1e6c008030 comp rx=0 tx=0).stop 2024-03-18T20:25:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.393+0000 7f1e7e95c700 1 -- 172.21.15.1:0/3135367341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7806a750 msgr2=0x7f1e781523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.393+0000 7f1e7e95c700 1 --2- 172.21.15.1:0/3135367341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7806a750 0x7f1e781523f0 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800c750 tx=0x7f1e6800cb10 comp rx=0 tx=0).stop 2024-03-18T20:25:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.393+0000 7f1e7e95c700 1 -- 172.21.15.1:0/3135367341 shutdown_connections 2024-03-18T20:25:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.393+0000 7f1e7e95c700 1 --2- 172.21.15.1:0/3135367341 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e6406ade0 0x7f1e6406d2a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.393+0000 7f1e7e95c700 1 --2- 172.21.15.1:0/3135367341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e7815d6b0 0x7f1e78153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.393+0000 7f1e7e95c700 1 --2- 172.21.15.1:0/3135367341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e7806a750 0x7f1e781523f0 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.393+0000 7f1e7e95c700 1 --2- 172.21.15.1:0/3135367341 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e780ad790 0x7f1e780cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.393+0000 7f1e7e95c700 1 -- 172.21.15.1:0/3135367341 >> 172.21.15.1:0/3135367341 conn(0x7f1e7806b860 msgr2=0x7f1e780b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.393+0000 7f1e7e95c700 1 -- 172.21.15.1:0/3135367341 shutdown_connections 2024-03-18T20:25:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.393+0000 7f1e7e95c700 1 -- 172.21.15.1:0/3135367341 wait complete. 2024-03-18T20:25:23.410 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T20:25:23.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.534+0000 7f61ddbe5700 1 Processor -- start 2024-03-18T20:25:23.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.534+0000 7f61ddbe5700 1 -- start start 2024-03-18T20:25:23.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.535+0000 7f61ddbe5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d815d5b0 0x7f61d815d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.535+0000 7f61ddbe5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f61d8099a40 0x7f61d8099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.535+0000 7f61ddbe5700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61d809a830 con 0x7f61d809a3e0 2024-03-18T20:25:23.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.535+0000 7f61ddbe5700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61d80cbbb0 con 0x7f61d8099a40 2024-03-18T20:25:23.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.535+0000 7f61ddbe5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d80b3b00 con 0x7f61d815d5b0 2024-03-18T20:25:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.535+0000 7f61d6ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f61d8099a40 0x7f61d8099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.535+0000 7f61d7fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f61d809a3e0 0x7f61d806ba70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35166/0 (socket says 172.21.15.1:35166) 2024-03-18T20:25:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.535+0000 7f61d7fff700 1 -- 172.21.15.1:0/4275452503 learned_addr learned my addr 172.21.15.1:0/4275452503 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.535+0000 7f61d77fe700 1 --2- 172.21.15.1:0/4275452503 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d815d5b0 0x7f61d815d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.535+0000 7f61d6ffd700 1 -- 172.21.15.1:0/4275452503 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d815d5b0 msgr2=0x7f61d815d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.535+0000 7f61d6ffd700 1 --2- 172.21.15.1:0/4275452503 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d815d5b0 0x7f61d815d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.535+0000 7f61d6ffd700 1 -- 172.21.15.1:0/4275452503 >> v1:172.21.15.1:6789/0 conn(0x7f61d809a3e0 legacy=0x7f61d806ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.535+0000 7f61d67fc700 1 -- 172.21.15.1:0/4275452503 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620698934 0 0) 0x7f61d809a830 con 0x7f61d809a3e0 2024-03-18T20:25:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.535+0000 7f61d6ffd700 1 -- 172.21.15.1:0/4275452503 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d80cf7f0 con 0x7f61d8099a40 2024-03-18T20:25:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.535+0000 7f61d77fe700 1 --2- 172.21.15.1:0/4275452503 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d815d5b0 0x7f61d815d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.535+0000 7f61d6ffd700 1 --2- 172.21.15.1:0/4275452503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d8099a40 0x7f61d8099ea0 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f61cc009840 tx=0x7f61cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf9d8e5ca367e1b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.535+0000 7f61d67fc700 1 -- 172.21.15.1:0/4275452503 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc011830 con 0x7f61d8099a40 2024-03-18T20:25:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.535+0000 7f61d67fc700 1 -- 172.21.15.1:0/4275452503 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61cc011990 con 0x7f61d8099a40 2024-03-18T20:25:23.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.535+0000 7f61d67fc700 1 -- 172.21.15.1:0/4275452503 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc011d20 con 0x7f61d8099a40 2024-03-18T20:25:23.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.536+0000 7f61ddbe5700 1 -- 172.21.15.1:0/4275452503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d8099a40 msgr2=0x7f61d8099ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.536+0000 7f61ddbe5700 1 --2- 172.21.15.1:0/4275452503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d8099a40 0x7f61d8099ea0 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f61cc009840 tx=0x7f61cc009b50 comp rx=0 tx=0).stop 2024-03-18T20:25:23.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.536+0000 7f61ddbe5700 1 -- 172.21.15.1:0/4275452503 shutdown_connections 2024-03-18T20:25:23.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.536+0000 7f61ddbe5700 1 --2- 172.21.15.1:0/4275452503 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d815d5b0 0x7f61d815d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.536+0000 7f61ddbe5700 1 --2- 172.21.15.1:0/4275452503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d8099a40 0x7f61d8099ea0 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.536+0000 7f61ddbe5700 1 -- 172.21.15.1:0/4275452503 >> 172.21.15.1:0/4275452503 conn(0x7f61d80b2bd0 msgr2=0x7f61d80b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:23.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.536+0000 7f61ddbe5700 1 -- 172.21.15.1:0/4275452503 shutdown_connections 2024-03-18T20:25:23.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.536+0000 7f61ddbe5700 1 -- 172.21.15.1:0/4275452503 wait complete. 2024-03-18T20:25:23.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.538+0000 7f61ddbe5700 1 Processor -- start 2024-03-18T20:25:23.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.538+0000 7f61ddbe5700 1 -- start start 2024-03-18T20:25:23.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61ddbe5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d8099a40 0x7f61d80b5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61ddbe5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d809a3e0 0x7f61d81490f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61ddbe5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d815d5b0 0x7f61d8149630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61ddbe5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d80c81b0 con 0x7f61d8099a40 2024-03-18T20:25:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61ddbe5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d80c8400 con 0x7f61d815d5b0 2024-03-18T20:25:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61ddbe5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d80c86e0 con 0x7f61d809a3e0 2024-03-18T20:25:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61d77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d8099a40 0x7f61d80b5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61d77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d8099a40 0x7f61d80b5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34384/0 (socket says 172.21.15.1:34384) 2024-03-18T20:25:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61d77fe700 1 -- 172.21.15.1:0/78286475 learned_addr learned my addr 172.21.15.1:0/78286475 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61d6ffd700 1 --2- 172.21.15.1:0/78286475 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d809a3e0 0x7f61d81490f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61d7fff700 1 --2- 172.21.15.1:0/78286475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d815d5b0 0x7f61d8149630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61d77fe700 1 -- 172.21.15.1:0/78286475 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d809a3e0 msgr2=0x7f61d81490f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61d77fe700 1 --2- 172.21.15.1:0/78286475 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d809a3e0 0x7f61d81490f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61d77fe700 1 -- 172.21.15.1:0/78286475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d815d5b0 msgr2=0x7f61d8149630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:23.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61d77fe700 1 --2- 172.21.15.1:0/78286475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d815d5b0 0x7f61d8149630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:23.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61d77fe700 1 -- 172.21.15.1:0/78286475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61cc0094a0 con 0x7f61d8099a40 2024-03-18T20:25:23.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61d77fe700 1 --2- 172.21.15.1:0/78286475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d8099a40 0x7f61d80b5f80 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f61c800c750 tx=0x7f61c800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:23.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61bffff700 1 -- 172.21.15.1:0/78286475 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c8011800 con 0x7f61d8099a40 2024-03-18T20:25:23.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61bffff700 1 -- 172.21.15.1:0/78286475 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c8011960 con 0x7f61d8099a40 2024-03-18T20:25:23.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61bffff700 1 -- 172.21.15.1:0/78286475 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c8011cd0 con 0x7f61d8099a40 2024-03-18T20:25:23.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.539+0000 7f61ddbe5700 1 -- 172.21.15.1:0/78286475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d80c89c0 con 0x7f61d8099a40 2024-03-18T20:25:23.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.540+0000 7f61ddbe5700 1 -- 172.21.15.1:0/78286475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d80c8f80 con 0x7f61d8099a40 2024-03-18T20:25:23.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.540+0000 7f61ddbe5700 1 -- 172.21.15.1:0/78286475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d80676a0 con 0x7f61d8099a40 2024-03-18T20:25:23.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.541+0000 7f61bffff700 1 -- 172.21.15.1:0/78286475 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f61c8021350 con 0x7f61d8099a40 2024-03-18T20:25:23.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.541+0000 7f61bffff700 1 --2- 172.21.15.1:0/78286475 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f61c406adb0 0x7f61c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:23.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.541+0000 7f61bffff700 1 -- 172.21.15.1:0/78286475 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f61c80895b0 con 0x7f61d8099a40 2024-03-18T20:25:23.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.541+0000 7f61d6ffd700 1 --2- 172.21.15.1:0/78286475 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f61c406adb0 0x7f61c406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:23.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.542+0000 7f61d6ffd700 1 --2- 172.21.15.1:0/78286475 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f61c406adb0 0x7f61c406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f61cc0090b0 tx=0x7f61cc01c040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:23.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.546+0000 7f61bffff700 1 -- 172.21.15.1:0/78286475 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f61c8054450 con 0x7f61d8099a40 2024-03-18T20:25:23.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:23.796+0000 7f61ddbe5700 1 -- 172.21.15.1:0/78286475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f61d80ce910 con 0x7f61d8099a40 2024-03-18T20:25:24.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.670+0000 7f61bffff700 1 -- 172.21.15.1:0/78286475 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v103) v1 ==== 118+0+0 (secure 0 0 0) 0x7f61c805ce60 con 0x7f61d8099a40 2024-03-18T20:25:24.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.672+0000 7f61ddbe5700 1 -- 172.21.15.1:0/78286475 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f61c406adb0 msgr2=0x7f61c406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:24.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.672+0000 7f61ddbe5700 1 --2- 172.21.15.1:0/78286475 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f61c406adb0 0x7f61c406d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f61cc0090b0 tx=0x7f61cc01c040 comp rx=0 tx=0).stop 2024-03-18T20:25:24.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.672+0000 7f61ddbe5700 1 -- 172.21.15.1:0/78286475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d8099a40 msgr2=0x7f61d80b5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:24.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.672+0000 7f61ddbe5700 1 --2- 172.21.15.1:0/78286475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d8099a40 0x7f61d80b5f80 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f61c800c750 tx=0x7f61c800ca60 comp rx=0 tx=0).stop 2024-03-18T20:25:24.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.672+0000 7f61ddbe5700 1 -- 172.21.15.1:0/78286475 shutdown_connections 2024-03-18T20:25:24.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.672+0000 7f61ddbe5700 1 --2- 172.21.15.1:0/78286475 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f61c406adb0 0x7f61c406d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:24.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.673+0000 7f61ddbe5700 1 --2- 172.21.15.1:0/78286475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d815d5b0 0x7f61d8149630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:24.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.673+0000 7f61ddbe5700 1 --2- 172.21.15.1:0/78286475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61d8099a40 0x7f61d80b5f80 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:24.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.673+0000 7f61ddbe5700 1 --2- 172.21.15.1:0/78286475 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61d809a3e0 0x7f61d81490f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:24.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.673+0000 7f61ddbe5700 1 -- 172.21.15.1:0/78286475 >> 172.21.15.1:0/78286475 conn(0x7f61d80b2bd0 msgr2=0x7f61d806fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:24.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.673+0000 7f61ddbe5700 1 -- 172.21.15.1:0/78286475 shutdown_connections 2024-03-18T20:25:24.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.673+0000 7f61ddbe5700 1 -- 172.21.15.1:0/78286475 wait complete. 2024-03-18T20:25:24.692 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-18T20:25:24.692 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T20:25:24.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.822+0000 7fabc8cc7700 1 Processor -- start 2024-03-18T20:25:24.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.822+0000 7fabc8cc7700 1 -- start start 2024-03-18T20:25:24.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.823+0000 7fabc8cc7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fabc4158e30 0x7fabc4071500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:24.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.823+0000 7fabc8cc7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabc40cbf70 0x7fabc40cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:24.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.823+0000 7fabc8cc7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc415c500 con 0x7fabc406a750 2024-03-18T20:25:24.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.823+0000 7fabc8cc7700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc40af8b0 con 0x7fabc4158e30 2024-03-18T20:25:24.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.823+0000 7fabc8cc7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc415b8f0 con 0x7fabc40cbf70 2024-03-18T20:25:24.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.823+0000 7fabc259c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fabc4158e30 0x7fabc4071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:24.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.823+0000 7fabc2d9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fabc406a750 0x7fabc40cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39512/0 (socket says 172.21.15.1:39512) 2024-03-18T20:25:24.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.823+0000 7fabc2d9d700 1 -- 172.21.15.1:0/2666430144 learned_addr learned my addr 172.21.15.1:0/2666430144 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:24.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.823+0000 7fabc1d9b700 1 --2- 172.21.15.1:0/2666430144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabc40cbf70 0x7fabc40cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:24.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.823+0000 7fabc259c700 1 -- 172.21.15.1:0/2666430144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabc40cbf70 msgr2=0x7fabc40cc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:24.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.823+0000 7fabc259c700 1 --2- 172.21.15.1:0/2666430144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabc40cbf70 0x7fabc40cc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:24.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.823+0000 7fabc259c700 1 -- 172.21.15.1:0/2666430144 >> v1:172.21.15.1:6789/0 conn(0x7fabc406a750 legacy=0x7fabc40cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:24.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.823+0000 7fabc259c700 1 -- 172.21.15.1:0/2666430144 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc415bbd0 con 0x7fabc4158e30 2024-03-18T20:25:24.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.823+0000 7fabc159a700 1 -- 172.21.15.1:0/2666430144 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255206166 0 0) 0x7fabc415c500 con 0x7fabc406a750 2024-03-18T20:25:24.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.824+0000 7fabc259c700 1 --2- 172.21.15.1:0/2666430144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc4158e30 0x7fabc4071500 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009d90 tx=0x7fabb400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20bccc38931496b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:24.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.824+0000 7fabc159a700 1 -- 172.21.15.1:0/2666430144 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb400e070 con 0x7fabc4158e30 2024-03-18T20:25:24.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.824+0000 7fabc159a700 1 -- 172.21.15.1:0/2666430144 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb400bbd0 con 0x7fabc4158e30 2024-03-18T20:25:24.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.824+0000 7fabc159a700 1 -- 172.21.15.1:0/2666430144 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb4012430 con 0x7fabc4158e30 2024-03-18T20:25:24.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.824+0000 7fabc8cc7700 1 -- 172.21.15.1:0/2666430144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc4158e30 msgr2=0x7fabc4071500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:24.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.824+0000 7fabc8cc7700 1 --2- 172.21.15.1:0/2666430144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc4158e30 0x7fabc4071500 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009d90 tx=0x7fabb400b330 comp rx=0 tx=0).stop 2024-03-18T20:25:24.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.824+0000 7fabc8cc7700 1 -- 172.21.15.1:0/2666430144 shutdown_connections 2024-03-18T20:25:24.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.824+0000 7fabc8cc7700 1 --2- 172.21.15.1:0/2666430144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc4158e30 0x7fabc4071500 unknown :-1 s=CLOSED pgs=2513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:24.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.824+0000 7fabc8cc7700 1 --2- 172.21.15.1:0/2666430144 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabc40cbf70 0x7fabc40cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:24.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.824+0000 7fabc8cc7700 1 -- 172.21.15.1:0/2666430144 >> 172.21.15.1:0/2666430144 conn(0x7fabc40cca00 msgr2=0x7fabc40cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:24.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.824+0000 7fabc8cc7700 1 -- 172.21.15.1:0/2666430144 shutdown_connections 2024-03-18T20:25:24.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.824+0000 7fabc8cc7700 1 -- 172.21.15.1:0/2666430144 wait complete. 2024-03-18T20:25:24.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.826+0000 7fabc8cc7700 1 Processor -- start 2024-03-18T20:25:24.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.826+0000 7fabc8cc7700 1 -- start start 2024-03-18T20:25:24.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.827+0000 7fabc8cc7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc406a750 0x7fabc40b5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:24.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.827+0000 7fabc8cc7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabc40cbf70 0x7fabc4158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:24.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.827+0000 7fabc8cc7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabc4158e30 0x7fabc40b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:24.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.827+0000 7fabc8cc7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc40b4d70 con 0x7fabc406a750 2024-03-18T20:25:24.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.827+0000 7fabc8cc7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc40c4830 con 0x7fabc4158e30 2024-03-18T20:25:24.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.827+0000 7fabc8cc7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc40b4180 con 0x7fabc40cbf70 2024-03-18T20:25:24.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.827+0000 7fabc259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc406a750 0x7fabc40b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:24.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.827+0000 7fabc259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc406a750 0x7fabc40b5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59788/0 (socket says 172.21.15.1:59788) 2024-03-18T20:25:24.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.827+0000 7fabc259c700 1 -- 172.21.15.1:0/3198020481 learned_addr learned my addr 172.21.15.1:0/3198020481 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:24.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.827+0000 7fabc2d9d700 1 --2- 172.21.15.1:0/3198020481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabc4158e30 0x7fabc40b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:24.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.827+0000 7fabc1d9b700 1 --2- 172.21.15.1:0/3198020481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabc40cbf70 0x7fabc4158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:24.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.827+0000 7fabc259c700 1 -- 172.21.15.1:0/3198020481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabc40cbf70 msgr2=0x7fabc4158370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:24.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.827+0000 7fabc259c700 1 --2- 172.21.15.1:0/3198020481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabc40cbf70 0x7fabc4158370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:24.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.828+0000 7fabc259c700 1 -- 172.21.15.1:0/3198020481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabc4158e30 msgr2=0x7fabc40b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:24.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.828+0000 7fabc259c700 1 --2- 172.21.15.1:0/3198020481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabc4158e30 0x7fabc40b3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:24.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.828+0000 7fabc259c700 1 -- 172.21.15.1:0/3198020481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb4009a70 con 0x7fabc406a750 2024-03-18T20:25:24.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.828+0000 7fabc1d9b700 1 --2- 172.21.15.1:0/3198020481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabc40cbf70 0x7fabc4158370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:24.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.828+0000 7fabc259c700 1 --2- 172.21.15.1:0/3198020481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc406a750 0x7fabc40b5190 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7fabb4014040 tx=0x7fabb4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:24.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.828+0000 7fabb2ffd700 1 -- 172.21.15.1:0/3198020481 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb4012430 con 0x7fabc406a750 2024-03-18T20:25:24.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.828+0000 7fabb2ffd700 1 -- 172.21.15.1:0/3198020481 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb4020070 con 0x7fabc406a750 2024-03-18T20:25:24.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.828+0000 7fabb2ffd700 1 -- 172.21.15.1:0/3198020481 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb4004260 con 0x7fabc406a750 2024-03-18T20:25:24.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.828+0000 7fabc8cc7700 1 -- 172.21.15.1:0/3198020481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc41588b0 con 0x7fabc406a750 2024-03-18T20:25:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.828+0000 7fabc8cc7700 1 -- 172.21.15.1:0/3198020481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc40b46b0 con 0x7fabc406a750 2024-03-18T20:25:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.829+0000 7fabc8cc7700 1 -- 172.21.15.1:0/3198020481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc4073940 con 0x7fabc406a750 2024-03-18T20:25:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.829+0000 7fabb2ffd700 1 -- 172.21.15.1:0/3198020481 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fabb40043c0 con 0x7fabc406a750 2024-03-18T20:25:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.830+0000 7fabb2ffd700 1 --2- 172.21.15.1:0/3198020481 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faba406adb0 0x7faba406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.830+0000 7fabb2ffd700 1 -- 172.21.15.1:0/3198020481 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fabb4022070 con 0x7fabc406a750 2024-03-18T20:25:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.830+0000 7fabc1d9b700 1 --2- 172.21.15.1:0/3198020481 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faba406adb0 0x7faba406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.830+0000 7fabc1d9b700 1 --2- 172.21.15.1:0/3198020481 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faba406adb0 0x7faba406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fabc40cc930 tx=0x7fabb8008040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:24.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:24.834+0000 7fabb2ffd700 1 -- 172.21.15.1:0/3198020481 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fabb40563a0 con 0x7fabc406a750 2024-03-18T20:25:25.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.082+0000 7fabc8cc7700 1 -- 172.21.15.1:0/3198020481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fabc40676a0 con 0x7fabc406a750 2024-03-18T20:25:25.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.674+0000 7fabb2ffd700 1 -- 172.21.15.1:0/3198020481 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v104) v1 ==== 122+0+0 (secure 0 0 0) 0x7fabb40599c0 con 0x7fabc406a750 2024-03-18T20:25:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.677+0000 7fabc8cc7700 1 -- 172.21.15.1:0/3198020481 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faba406adb0 msgr2=0x7faba406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.677+0000 7fabc8cc7700 1 --2- 172.21.15.1:0/3198020481 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faba406adb0 0x7faba406d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fabc40cc930 tx=0x7fabb8008040 comp rx=0 tx=0).stop 2024-03-18T20:25:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.677+0000 7fabc8cc7700 1 -- 172.21.15.1:0/3198020481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc406a750 msgr2=0x7fabc40b5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.677+0000 7fabc8cc7700 1 --2- 172.21.15.1:0/3198020481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc406a750 0x7fabc40b5190 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7fabb4014040 tx=0x7fabb4004ad0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.677+0000 7fabc8cc7700 1 -- 172.21.15.1:0/3198020481 shutdown_connections 2024-03-18T20:25:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.677+0000 7fabc8cc7700 1 --2- 172.21.15.1:0/3198020481 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faba406adb0 0x7faba406d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.677+0000 7fabc8cc7700 1 --2- 172.21.15.1:0/3198020481 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fabc4158e30 0x7fabc40b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.677+0000 7fabc8cc7700 1 --2- 172.21.15.1:0/3198020481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fabc406a750 0x7fabc40b5190 unknown :-1 s=CLOSED pgs=2514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.677+0000 7fabc8cc7700 1 --2- 172.21.15.1:0/3198020481 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fabc40cbf70 0x7fabc4158370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.677+0000 7fabc8cc7700 1 -- 172.21.15.1:0/3198020481 >> 172.21.15.1:0/3198020481 conn(0x7fabc40cca00 msgr2=0x7fabc415d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.678+0000 7fabc8cc7700 1 -- 172.21.15.1:0/3198020481 shutdown_connections 2024-03-18T20:25:25.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.678+0000 7fabc8cc7700 1 -- 172.21.15.1:0/3198020481 wait complete. 2024-03-18T20:25:25.696 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T20:25:25.696 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:25:25.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.824+0000 7f7d09b8d700 1 Processor -- start 2024-03-18T20:25:25.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.824+0000 7f7d09b8d700 1 -- start start 2024-03-18T20:25:25.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.825+0000 7f7d09b8d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d0415d580 0x7f7d04072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:25.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.825+0000 7f7d09b8d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d0406dd50 0x7f7d040cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:25.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.825+0000 7f7d09b8d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0406f490 con 0x7f7d0406e290 2024-03-18T20:25:25.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.825+0000 7f7d09b8d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d04071cf0 con 0x7f7d0406dd50 2024-03-18T20:25:25.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.825+0000 7f7d09b8d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0406f1b0 con 0x7f7d0415d580 2024-03-18T20:25:25.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.825+0000 7f7d03fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d0406dd50 0x7f7d040cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:25.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.825+0000 7f7d02ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7d0406e290 0x7f7d0406d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39514/0 (socket says 172.21.15.1:39514) 2024-03-18T20:25:25.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.826+0000 7f7d02ffd700 1 -- 172.21.15.1:0/3164003864 learned_addr learned my addr 172.21.15.1:0/3164003864 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.826+0000 7f7d037fe700 1 --2- 172.21.15.1:0/3164003864 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d0415d580 0x7f7d04072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.826+0000 7f7d03fff700 1 -- 172.21.15.1:0/3164003864 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d0415d580 msgr2=0x7f7d04072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.826+0000 7f7d03fff700 1 --2- 172.21.15.1:0/3164003864 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d0415d580 0x7f7d04072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.826+0000 7f7d027fc700 1 -- 172.21.15.1:0/3164003864 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261556398 0 0) 0x7f7d0406f490 con 0x7f7d0406e290 2024-03-18T20:25:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.826+0000 7f7d03fff700 1 -- 172.21.15.1:0/3164003864 >> v1:172.21.15.1:6789/0 conn(0x7f7d0406e290 legacy=0x7f7d0406d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.826+0000 7f7d03fff700 1 -- 172.21.15.1:0/3164003864 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d040716a0 con 0x7f7d0406dd50 2024-03-18T20:25:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.826+0000 7f7d03fff700 1 --2- 172.21.15.1:0/3164003864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d0406dd50 0x7f7d040cd4b0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc009d90 tx=0x7f7cfc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e58cf12b7a50276 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.826+0000 7f7d027fc700 1 -- 172.21.15.1:0/3164003864 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc00e070 con 0x7f7d0406dd50 2024-03-18T20:25:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.826+0000 7f7d027fc700 1 -- 172.21.15.1:0/3164003864 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cfc00bb20 con 0x7f7d0406dd50 2024-03-18T20:25:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.826+0000 7f7d027fc700 1 -- 172.21.15.1:0/3164003864 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc012430 con 0x7f7d0406dd50 2024-03-18T20:25:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.827+0000 7f7d09b8d700 1 -- 172.21.15.1:0/3164003864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d0406dd50 msgr2=0x7f7d040cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.827+0000 7f7d09b8d700 1 --2- 172.21.15.1:0/3164003864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d0406dd50 0x7f7d040cd4b0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc009d90 tx=0x7f7cfc00b330 comp rx=0 tx=0).stop 2024-03-18T20:25:25.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.827+0000 7f7d09b8d700 1 -- 172.21.15.1:0/3164003864 shutdown_connections 2024-03-18T20:25:25.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.827+0000 7f7d09b8d700 1 --2- 172.21.15.1:0/3164003864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d0406dd50 0x7f7d040cd4b0 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.827+0000 7f7d09b8d700 1 --2- 172.21.15.1:0/3164003864 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d0415d580 0x7f7d04072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.827+0000 7f7d09b8d700 1 -- 172.21.15.1:0/3164003864 >> 172.21.15.1:0/3164003864 conn(0x7f7d040cc990 msgr2=0x7f7d040ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:25.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.827+0000 7f7d09b8d700 1 -- 172.21.15.1:0/3164003864 shutdown_connections 2024-03-18T20:25:25.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.827+0000 7f7d09b8d700 1 -- 172.21.15.1:0/3164003864 wait complete. 2024-03-18T20:25:25.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.828+0000 7f7d09b8d700 1 Processor -- start 2024-03-18T20:25:25.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.828+0000 7f7d09b8d700 1 -- start start 2024-03-18T20:25:25.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.829+0000 7f7d09b8d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d0406dd50 0x7f7d041526f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:25.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.829+0000 7f7d09b8d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d0415d580 0x7f7d0415e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:25.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.829+0000 7f7d09b8d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d0415edc0 0x7f7d0415fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:25.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.829+0000 7f7d09b8d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0415f250 con 0x7f7d0415d580 2024-03-18T20:25:25.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.829+0000 7f7d09b8d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d040c8530 con 0x7f7d0406dd50 2024-03-18T20:25:25.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.829+0000 7f7d09b8d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0415dc80 con 0x7f7d0415edc0 2024-03-18T20:25:25.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.829+0000 7f7d02ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d0415d580 0x7f7d0415e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:25.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.829+0000 7f7d02ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d0415d580 0x7f7d0415e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59796/0 (socket says 172.21.15.1:59796) 2024-03-18T20:25:25.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.829+0000 7f7d02ffd700 1 -- 172.21.15.1:0/2706759212 learned_addr learned my addr 172.21.15.1:0/2706759212 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:25.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.829+0000 7f7d03fff700 1 --2- 172.21.15.1:0/2706759212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d0415edc0 0x7f7d0415fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:25.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.829+0000 7f7d037fe700 1 --2- 172.21.15.1:0/2706759212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d0406dd50 0x7f7d041526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:25.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.829+0000 7f7d02ffd700 1 -- 172.21.15.1:0/2706759212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d0415edc0 msgr2=0x7f7d0415fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:25.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.829+0000 7f7d02ffd700 1 --2- 172.21.15.1:0/2706759212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d0415edc0 0x7f7d0415fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.829+0000 7f7d02ffd700 1 -- 172.21.15.1:0/2706759212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d0406dd50 msgr2=0x7f7d041526f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:25.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.829+0000 7f7d02ffd700 1 --2- 172.21.15.1:0/2706759212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d0406dd50 0x7f7d041526f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:25.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.829+0000 7f7d02ffd700 1 -- 172.21.15.1:0/2706759212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cfc009a70 con 0x7f7d0415d580 2024-03-18T20:25:25.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.829+0000 7f7d037fe700 1 --2- 172.21.15.1:0/2706759212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d0406dd50 0x7f7d041526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:25.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.829+0000 7f7d02ffd700 1 --2- 172.21.15.1:0/2706759212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d0415d580 0x7f7d0415e880 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f7cec0011e0 tx=0x7f7cec00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:25.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.830+0000 7f7cfbfff700 1 -- 172.21.15.1:0/2706759212 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec004c80 con 0x7f7d0415d580 2024-03-18T20:25:25.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.830+0000 7f7cfbfff700 1 -- 172.21.15.1:0/2706759212 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cec004e00 con 0x7f7d0415d580 2024-03-18T20:25:25.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.830+0000 7f7d09b8d700 1 -- 172.21.15.1:0/2706759212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d04152c30 con 0x7f7d0415d580 2024-03-18T20:25:25.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.830+0000 7f7cfbfff700 1 -- 172.21.15.1:0/2706759212 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec007310 con 0x7f7d0415d580 2024-03-18T20:25:25.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.830+0000 7f7d09b8d700 1 -- 172.21.15.1:0/2706759212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d040c7ee0 con 0x7f7d0415d580 2024-03-18T20:25:25.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.831+0000 7f7d09b8d700 1 -- 172.21.15.1:0/2706759212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d040c86a0 con 0x7f7d0415d580 2024-03-18T20:25:25.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.831+0000 7f7cfbfff700 1 -- 172.21.15.1:0/2706759212 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7cec007470 con 0x7f7d0415d580 2024-03-18T20:25:25.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.831+0000 7f7cfbfff700 1 --2- 172.21.15.1:0/2706759212 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7ce406adb0 0x7f7ce406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:25.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.832+0000 7f7d037fe700 1 --2- 172.21.15.1:0/2706759212 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7ce406adb0 0x7f7ce406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:25.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.832+0000 7f7cfbfff700 1 -- 172.21.15.1:0/2706759212 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f7cec089340 con 0x7f7d0415d580 2024-03-18T20:25:25.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.832+0000 7f7d037fe700 1 --2- 172.21.15.1:0/2706759212 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7ce406adb0 0x7f7ce406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4009ac0 tx=0x7f7cf4008040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:25.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:25.836+0000 7f7cfbfff700 1 -- 172.21.15.1:0/2706759212 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7cec0542f0 con 0x7f7d0415d580 2024-03-18T20:25:26.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.088+0000 7f7d09b8d700 1 -- 172.21.15.1:0/2706759212 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d040676a0 con 0x7f7d0415d580 2024-03-18T20:25:26.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.088+0000 7f7cfbfff700 1 -- 172.21.15.1:0/2706759212 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4747 (secure 0 0 0) 0x7f7cec057910 con 0x7f7d0415d580 2024-03-18T20:25:26.093 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:26.093 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":104,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10369,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/677258734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":677258734},{"type":"v1","addr":"172.21.15.165:6835","nonce":677258734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":11544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2018398457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2018398457},{"type":"v1","addr":"172.21.15.1:6836","nonce":2018398457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":11547,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1635684321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1635684321},{"type":"v1","addr":"172.21.15.1:6837","nonce":1635684321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:25:22.663717+0000","modified":"2024-03-18T20:25:25.668584+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11556":{"gid":11556,"name":"d","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6834/4024828630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4024828630},{"type":"v1","addr":"172.21.15.165:6834","nonce":4024828630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-03-18T20:25:26.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.090+0000 7f7d09b8d700 1 -- 172.21.15.1:0/2706759212 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7ce406adb0 msgr2=0x7f7ce406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:26.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.090+0000 7f7d09b8d700 1 --2- 172.21.15.1:0/2706759212 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7ce406adb0 0x7f7ce406d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4009ac0 tx=0x7f7cf4008040 comp rx=0 tx=0).stop 2024-03-18T20:25:26.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.090+0000 7f7d09b8d700 1 -- 172.21.15.1:0/2706759212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d0415d580 msgr2=0x7f7d0415e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:26.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.090+0000 7f7d09b8d700 1 --2- 172.21.15.1:0/2706759212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d0415d580 0x7f7d0415e880 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f7cec0011e0 tx=0x7f7cec00cae0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.091+0000 7f7d09b8d700 1 -- 172.21.15.1:0/2706759212 shutdown_connections 2024-03-18T20:25:26.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.091+0000 7f7d09b8d700 1 --2- 172.21.15.1:0/2706759212 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7ce406adb0 0x7f7ce406d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.091+0000 7f7d09b8d700 1 --2- 172.21.15.1:0/2706759212 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d0415edc0 0x7f7d0415fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.091+0000 7f7d09b8d700 1 --2- 172.21.15.1:0/2706759212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d0406dd50 0x7f7d041526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.091+0000 7f7d09b8d700 1 --2- 172.21.15.1:0/2706759212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d0415d580 0x7f7d0415e880 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.091+0000 7f7d09b8d700 1 -- 172.21.15.1:0/2706759212 >> 172.21.15.1:0/2706759212 conn(0x7f7d040cc990 msgr2=0x7f7d0406d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:26.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.091+0000 7f7d09b8d700 1 -- 172.21.15.1:0/2706759212 shutdown_connections 2024-03-18T20:25:26.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.091+0000 7f7d09b8d700 1 -- 172.21.15.1:0/2706759212 wait complete. 2024-03-18T20:25:26.096 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 104 2024-03-18T20:25:26.107 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:25:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.229+0000 7f3abe98a700 1 Processor -- start 2024-03-18T20:25:26.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.229+0000 7f3abe98a700 1 -- start start 2024-03-18T20:25:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.230+0000 7f3abe98a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3ab815aff0 0x7f3ab815f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.230+0000 7f3abe98a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ab8073b00 0x7f3ab80cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.230+0000 7f3abe98a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab815fcc0 con 0x7f3ab815b3d0 2024-03-18T20:25:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.230+0000 7f3abe98a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab80ce200 con 0x7f3ab815aff0 2024-03-18T20:25:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.230+0000 7f3abe98a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab80ce4a0 con 0x7f3ab8073b00 2024-03-18T20:25:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.230+0000 7f3ab7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3ab815aff0 0x7f3ab815f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.230+0000 7f3ab77fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3ab815b3d0 0x7f3ab80cf540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39516/0 (socket says 172.21.15.1:39516) 2024-03-18T20:25:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.230+0000 7f3ab77fe700 1 -- 172.21.15.1:0/3915129901 learned_addr learned my addr 172.21.15.1:0/3915129901 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.230+0000 7f3ab7fff700 1 -- 172.21.15.1:0/3915129901 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ab8073b00 msgr2=0x7f3ab80cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:26.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.230+0000 7f3abcf27700 1 --2- 172.21.15.1:0/3915129901 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ab8073b00 0x7f3ab80cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.230+0000 7f3ab6ffd700 1 -- 172.21.15.1:0/3915129901 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113773319 0 0) 0x7f3ab815fcc0 con 0x7f3ab815b3d0 2024-03-18T20:25:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.230+0000 7f3ab7fff700 1 --2- 172.21.15.1:0/3915129901 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ab8073b00 0x7f3ab80cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.230+0000 7f3ab7fff700 1 -- 172.21.15.1:0/3915129901 >> v1:172.21.15.1:6789/0 conn(0x7f3ab815b3d0 legacy=0x7f3ab80cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.230+0000 7f3ab7fff700 1 -- 172.21.15.1:0/3915129901 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab80ce780 con 0x7f3ab815aff0 2024-03-18T20:25:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.230+0000 7f3abcf27700 1 --2- 172.21.15.1:0/3915129901 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ab8073b00 0x7f3ab80cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.231+0000 7f3ab7fff700 1 --2- 172.21.15.1:0/3915129901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ab815aff0 0x7f3ab815f780 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0009840 tx=0x7f3aa0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8927986ee8da81 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.231+0000 7f3ab6ffd700 1 -- 172.21.15.1:0/3915129901 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0011830 con 0x7f3ab815aff0 2024-03-18T20:25:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.231+0000 7f3ab6ffd700 1 -- 172.21.15.1:0/3915129901 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa0011990 con 0x7f3ab815aff0 2024-03-18T20:25:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.231+0000 7f3ab6ffd700 1 -- 172.21.15.1:0/3915129901 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0011d20 con 0x7f3ab815aff0 2024-03-18T20:25:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.231+0000 7f3abe98a700 1 -- 172.21.15.1:0/3915129901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ab815aff0 msgr2=0x7f3ab815f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.231+0000 7f3abe98a700 1 --2- 172.21.15.1:0/3915129901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ab815aff0 0x7f3ab815f780 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0009840 tx=0x7f3aa0009b50 comp rx=0 tx=0).stop 2024-03-18T20:25:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.231+0000 7f3abe98a700 1 -- 172.21.15.1:0/3915129901 shutdown_connections 2024-03-18T20:25:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.231+0000 7f3abe98a700 1 --2- 172.21.15.1:0/3915129901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ab815aff0 0x7f3ab815f780 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.231+0000 7f3abe98a700 1 --2- 172.21.15.1:0/3915129901 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ab8073b00 0x7f3ab80cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.231+0000 7f3abe98a700 1 -- 172.21.15.1:0/3915129901 >> 172.21.15.1:0/3915129901 conn(0x7f3ab806a750 msgr2=0x7f3ab806ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:26.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.231+0000 7f3abe98a700 1 -- 172.21.15.1:0/3915129901 shutdown_connections 2024-03-18T20:25:26.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.231+0000 7f3abe98a700 1 -- 172.21.15.1:0/3915129901 wait complete. 2024-03-18T20:25:26.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.232+0000 7f3abe98a700 1 Processor -- start 2024-03-18T20:25:26.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.233+0000 7f3abe98a700 1 -- start start 2024-03-18T20:25:26.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.233+0000 7f3abe98a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ab8073b00 0x7f3ab81f2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.233+0000 7f3abe98a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ab815aff0 0x7f3ab80b0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.233+0000 7f3abe98a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ab815b3d0 0x7f3ab80b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.233+0000 7f3abe98a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab80b3590 con 0x7f3ab8073b00 2024-03-18T20:25:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.233+0000 7f3abe98a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab80b3870 con 0x7f3ab815b3d0 2024-03-18T20:25:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.233+0000 7f3abe98a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab80b3b50 con 0x7f3ab815aff0 2024-03-18T20:25:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.233+0000 7f3ab7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ab8073b00 0x7f3ab81f2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.234+0000 7f3ab7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ab8073b00 0x7f3ab81f2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59822/0 (socket says 172.21.15.1:59822) 2024-03-18T20:25:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.234+0000 7f3ab7fff700 1 -- 172.21.15.1:0/1134547456 learned_addr learned my addr 172.21.15.1:0/1134547456 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.234+0000 7f3ab77fe700 1 --2- 172.21.15.1:0/1134547456 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ab815aff0 0x7f3ab80b0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.234+0000 7f3abcf27700 1 --2- 172.21.15.1:0/1134547456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ab815b3d0 0x7f3ab80b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.234+0000 7f3ab7fff700 1 -- 172.21.15.1:0/1134547456 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ab815aff0 msgr2=0x7f3ab80b0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:26.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.234+0000 7f3ab7fff700 1 --2- 172.21.15.1:0/1134547456 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ab815aff0 0x7f3ab80b0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.234+0000 7f3ab7fff700 1 -- 172.21.15.1:0/1134547456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ab815b3d0 msgr2=0x7f3ab80b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.234+0000 7f3ab7fff700 1 --2- 172.21.15.1:0/1134547456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ab815b3d0 0x7f3ab80b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.234+0000 7f3ab7fff700 1 -- 172.21.15.1:0/1134547456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa00094a0 con 0x7f3ab8073b00 2024-03-18T20:25:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.234+0000 7f3abcf27700 1 --2- 172.21.15.1:0/1134547456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ab815b3d0 0x7f3ab80b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.234+0000 7f3ab7fff700 1 --2- 172.21.15.1:0/1134547456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ab8073b00 0x7f3ab81f2450 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f3aa00090b0 tx=0x7f3aa0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.234+0000 7f3ab4ff9700 1 -- 172.21.15.1:0/1134547456 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa00038a0 con 0x7f3ab8073b00 2024-03-18T20:25:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.234+0000 7f3ab4ff9700 1 -- 172.21.15.1:0/1134547456 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa001f070 con 0x7f3ab8073b00 2024-03-18T20:25:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.234+0000 7f3abe98a700 1 -- 172.21.15.1:0/1134547456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab80b3e30 con 0x7f3ab8073b00 2024-03-18T20:25:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.234+0000 7f3ab4ff9700 1 -- 172.21.15.1:0/1134547456 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0003c60 con 0x7f3ab8073b00 2024-03-18T20:25:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.234+0000 7f3abe98a700 1 -- 172.21.15.1:0/1134547456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab81f2990 con 0x7f3ab8073b00 2024-03-18T20:25:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.235+0000 7f3abe98a700 1 -- 172.21.15.1:0/1134547456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab81f2b20 con 0x7f3ab8073b00 2024-03-18T20:25:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.236+0000 7f3ab4ff9700 1 -- 172.21.15.1:0/1134547456 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3aa001a430 con 0x7f3ab8073b00 2024-03-18T20:25:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.236+0000 7f3ab4ff9700 1 --2- 172.21.15.1:0/1134547456 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3a9806adb0 0x7f3a9806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:26.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.236+0000 7f3ab4ff9700 1 -- 172.21.15.1:0/1134547456 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f3aa0021070 con 0x7f3ab8073b00 2024-03-18T20:25:26.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.236+0000 7f3ab77fe700 1 --2- 172.21.15.1:0/1134547456 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3a9806adb0 0x7f3a9806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:26.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.237+0000 7f3ab77fe700 1 --2- 172.21.15.1:0/1134547456 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3a9806adb0 0x7f3a9806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80010e0 tx=0x7f3aa8009040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:26.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.241+0000 7f3ab4ff9700 1 -- 172.21.15.1:0/1134547456 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3aa0056080 con 0x7f3ab8073b00 2024-03-18T20:25:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.444+0000 7f3abe98a700 1 -- 172.21.15.1:0/1134547456 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ab80b42f0 con 0x7f3ab8073b00 2024-03-18T20:25:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.445+0000 7f3ab4ff9700 1 -- 172.21.15.1:0/1134547456 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) v1 ==== 74+0+16429 (secure 0 0 0) 0x7f3aa0015090 con 0x7f3ab8073b00 2024-03-18T20:25:26.450 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:26.450 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":232,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:25:23.665001+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:25:20.511297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-18T20:25:21.637693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:26.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.448+0000 7f3abe98a700 1 -- 172.21.15.1:0/1134547456 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3a9806adb0 msgr2=0x7f3a9806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:26.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.448+0000 7f3abe98a700 1 --2- 172.21.15.1:0/1134547456 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3a9806adb0 0x7f3a9806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80010e0 tx=0x7f3aa8009040 comp rx=0 tx=0).stop 2024-03-18T20:25:26.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.448+0000 7f3abe98a700 1 -- 172.21.15.1:0/1134547456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ab8073b00 msgr2=0x7f3ab81f2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:26.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.448+0000 7f3abe98a700 1 --2- 172.21.15.1:0/1134547456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ab8073b00 0x7f3ab81f2450 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f3aa00090b0 tx=0x7f3aa0004750 comp rx=0 tx=0).stop 2024-03-18T20:25:26.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.448+0000 7f3abe98a700 1 -- 172.21.15.1:0/1134547456 shutdown_connections 2024-03-18T20:25:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.448+0000 7f3abe98a700 1 --2- 172.21.15.1:0/1134547456 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3a9806adb0 0x7f3a9806d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.448+0000 7f3abe98a700 1 --2- 172.21.15.1:0/1134547456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ab815b3d0 0x7f3ab80b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.448+0000 7f3abe98a700 1 --2- 172.21.15.1:0/1134547456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ab8073b00 0x7f3ab81f2450 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.448+0000 7f3abe98a700 1 --2- 172.21.15.1:0/1134547456 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ab815aff0 0x7f3ab80b0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.448+0000 7f3abe98a700 1 -- 172.21.15.1:0/1134547456 >> 172.21.15.1:0/1134547456 conn(0x7f3ab806a750 msgr2=0x7f3ab815f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.448+0000 7f3abe98a700 1 -- 172.21.15.1:0/1134547456 shutdown_connections 2024-03-18T20:25:26.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.448+0000 7f3abe98a700 1 -- 172.21.15.1:0/1134547456 wait complete. 2024-03-18T20:25:26.464 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T20:25:26.465 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:25:26.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.586+0000 7f483e250700 1 Processor -- start 2024-03-18T20:25:26.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.586+0000 7f483e250700 1 -- start start 2024-03-18T20:25:26.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.587+0000 7f483e250700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f483815d580 0x7f4838072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:26.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.587+0000 7f483e250700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f483806e290 0x7f48380cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:26.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.587+0000 7f483e250700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483806f490 con 0x7f483806d640 2024-03-18T20:25:26.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.587+0000 7f483e250700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4838071cf0 con 0x7f483806e290 2024-03-18T20:25:26.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.587+0000 7f483e250700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483806f1b0 con 0x7f483815d580 2024-03-18T20:25:26.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.587+0000 7f4836ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f483806e290 0x7f48380cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:26.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.587+0000 7f4836ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f483806e290 0x7f48380cd4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59826/0 (socket says 172.21.15.1:59826) 2024-03-18T20:25:26.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.587+0000 7f4836ffd700 1 -- 172.21.15.1:0/2996745366 learned_addr learned my addr 172.21.15.1:0/2996745366 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:26.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.587+0000 7f4836ffd700 1 -- 172.21.15.1:0/2996745366 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f483815d580 msgr2=0x7f4838072180 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T20:25:26.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.587+0000 7f4836ffd700 1 --2- 172.21.15.1:0/2996745366 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f483815d580 0x7f4838072180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.587+0000 7f4836ffd700 1 -- 172.21.15.1:0/2996745366 >> v1:172.21.15.165:6789/0 conn(0x7f483806d640 legacy=0x7f48380ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:26.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.587+0000 7f4836ffd700 1 -- 172.21.15.1:0/2996745366 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48380716a0 con 0x7f483806e290 2024-03-18T20:25:26.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.587+0000 7f4836ffd700 1 --2- 172.21.15.1:0/2996745366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f483806e290 0x7f48380cd4b0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f482c0098f0 tx=0x7f482c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359cb2c4aec7363d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:26.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.587+0000 7f48367fc700 1 -- 172.21.15.1:0/2996745366 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c00b8a0 con 0x7f483806e290 2024-03-18T20:25:26.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.588+0000 7f48367fc700 1 -- 172.21.15.1:0/2996745366 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482c00ba00 con 0x7f483806e290 2024-03-18T20:25:26.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.588+0000 7f48367fc700 1 -- 172.21.15.1:0/2996745366 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c00bd90 con 0x7f483806e290 2024-03-18T20:25:26.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.588+0000 7f483e250700 1 -- 172.21.15.1:0/2996745366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f483806e290 msgr2=0x7f48380cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:26.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.588+0000 7f483e250700 1 --2- 172.21.15.1:0/2996745366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f483806e290 0x7f48380cd4b0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f482c0098f0 tx=0x7f482c009c00 comp rx=0 tx=0).stop 2024-03-18T20:25:26.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.588+0000 7f483e250700 1 -- 172.21.15.1:0/2996745366 shutdown_connections 2024-03-18T20:25:26.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.588+0000 7f483e250700 1 --2- 172.21.15.1:0/2996745366 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f483815d580 0x7f4838072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.588+0000 7f483e250700 1 --2- 172.21.15.1:0/2996745366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f483806e290 0x7f48380cd4b0 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.588+0000 7f483e250700 1 -- 172.21.15.1:0/2996745366 >> 172.21.15.1:0/2996745366 conn(0x7f48380cc990 msgr2=0x7f48380ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:26.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.588+0000 7f483e250700 1 -- 172.21.15.1:0/2996745366 shutdown_connections 2024-03-18T20:25:26.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.588+0000 7f483e250700 1 -- 172.21.15.1:0/2996745366 wait complete. 2024-03-18T20:25:26.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.589+0000 7f483e250700 1 Processor -- start 2024-03-18T20:25:26.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.590+0000 7f483e250700 1 -- start start 2024-03-18T20:25:26.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.590+0000 7f483e250700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f483806d640 0x7f48381526f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:26.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.590+0000 7f483e250700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f483815d580 0x7f483815d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:26.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.590+0000 7f483e250700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f483815dea0 0x7f48380c9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:26.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.590+0000 7f483e250700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483815e340 con 0x7f483815dea0 2024-03-18T20:25:26.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.590+0000 7f483e250700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483815e620 con 0x7f483806d640 2024-03-18T20:25:26.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.590+0000 7f483e250700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483815e900 con 0x7f483815d580 2024-03-18T20:25:26.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.590+0000 7f4837fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f483815dea0 0x7f48380c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:26.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.590+0000 7f4837fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f483815dea0 0x7f48380c9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59842/0 (socket says 172.21.15.1:59842) 2024-03-18T20:25:26.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.590+0000 7f4837fff700 1 -- 172.21.15.1:0/503430871 learned_addr learned my addr 172.21.15.1:0/503430871 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:26.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.591+0000 7f4836ffd700 1 --2- 172.21.15.1:0/503430871 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f483815d580 0x7f483815d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:26.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.591+0000 7f48377fe700 1 --2- 172.21.15.1:0/503430871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f483806d640 0x7f48381526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:26.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.591+0000 7f4837fff700 1 -- 172.21.15.1:0/503430871 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f483815d580 msgr2=0x7f483815d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:26.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.591+0000 7f4837fff700 1 --2- 172.21.15.1:0/503430871 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f483815d580 0x7f483815d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.591+0000 7f4837fff700 1 -- 172.21.15.1:0/503430871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f483806d640 msgr2=0x7f48381526f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:26.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.591+0000 7f4837fff700 1 --2- 172.21.15.1:0/503430871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f483806d640 0x7f48381526f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.591+0000 7f4837fff700 1 -- 172.21.15.1:0/503430871 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482c009550 con 0x7f483815dea0 2024-03-18T20:25:26.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.591+0000 7f4837fff700 1 --2- 172.21.15.1:0/503430871 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f483815dea0 0x7f48380c9310 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f483000cc50 tx=0x7f483000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:26.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.591+0000 7f481ffff700 1 -- 172.21.15.1:0/503430871 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4830003c00 con 0x7f483815dea0 2024-03-18T20:25:26.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.591+0000 7f481ffff700 1 -- 172.21.15.1:0/503430871 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4830003d60 con 0x7f483815dea0 2024-03-18T20:25:26.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.591+0000 7f483e250700 1 -- 172.21.15.1:0/503430871 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483815ebe0 con 0x7f483815dea0 2024-03-18T20:25:26.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.591+0000 7f481ffff700 1 -- 172.21.15.1:0/503430871 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483000ab60 con 0x7f483815dea0 2024-03-18T20:25:26.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.591+0000 7f483e250700 1 -- 172.21.15.1:0/503430871 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4838152c30 con 0x7f483815dea0 2024-03-18T20:25:26.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.592+0000 7f483e250700 1 -- 172.21.15.1:0/503430871 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48380c7220 con 0x7f483815dea0 2024-03-18T20:25:26.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.593+0000 7f481ffff700 1 -- 172.21.15.1:0/503430871 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f483000acc0 con 0x7f483815dea0 2024-03-18T20:25:26.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.593+0000 7f481ffff700 1 --2- 172.21.15.1:0/503430871 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f482406adb0 0x7f482406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:26.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.593+0000 7f481ffff700 1 -- 172.21.15.1:0/503430871 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f4830089690 con 0x7f483815dea0 2024-03-18T20:25:26.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.593+0000 7f48377fe700 1 --2- 172.21.15.1:0/503430871 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f482406adb0 0x7f482406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:26.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.594+0000 7f48377fe700 1 --2- 172.21.15.1:0/503430871 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f482406adb0 0x7f482406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4828009870 tx=0x7f4828008040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:26.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.598+0000 7f481ffff700 1 -- 172.21.15.1:0/503430871 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f48300545b0 con 0x7f483815dea0 2024-03-18T20:25:26.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.797+0000 7f483e250700 1 -- 172.21.15.1:0/503430871 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f483806ee40 con 0x7f482406adb0 2024-03-18T20:25:26.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.806+0000 7f481ffff700 1 -- 172.21.15.1:0/503430871 <== mgr.11322 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278927 (secure 0 0 0) 0x7f48280123e0 con 0x7f482406adb0 2024-03-18T20:25:26.814 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.809+0000 7f483e250700 1 -- 172.21.15.1:0/503430871 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f482406adb0 msgr2=0x7f482406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.809+0000 7f483e250700 1 --2- 172.21.15.1:0/503430871 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f482406adb0 0x7f482406d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4828009870 tx=0x7f4828008040 comp rx=0 tx=0).stop 2024-03-18T20:25:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.809+0000 7f483e250700 1 -- 172.21.15.1:0/503430871 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f483815dea0 msgr2=0x7f48380c9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.809+0000 7f483e250700 1 --2- 172.21.15.1:0/503430871 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f483815dea0 0x7f48380c9310 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f483000cc50 tx=0x7f483000cf60 comp rx=0 tx=0).stop 2024-03-18T20:25:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.809+0000 7f483e250700 1 -- 172.21.15.1:0/503430871 shutdown_connections 2024-03-18T20:25:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.809+0000 7f483e250700 1 --2- 172.21.15.1:0/503430871 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f482406adb0 0x7f482406d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.809+0000 7f483e250700 1 --2- 172.21.15.1:0/503430871 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f483815dea0 0x7f48380c9310 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.809+0000 7f483e250700 1 --2- 172.21.15.1:0/503430871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f483806d640 0x7f48381526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.809+0000 7f483e250700 1 --2- 172.21.15.1:0/503430871 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f483815d580 0x7f483815d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.809+0000 7f483e250700 1 -- 172.21.15.1:0/503430871 >> 172.21.15.1:0/503430871 conn(0x7f48380cc990 msgr2=0x7f483806c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:26.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.809+0000 7f483e250700 1 -- 172.21.15.1:0/503430871 shutdown_connections 2024-03-18T20:25:26.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.809+0000 7f483e250700 1 -- 172.21.15.1:0/503430871 wait complete. 2024-03-18T20:25:26.817 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:25:26.825 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-18T20:25:25.099985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":401344,"kb_used_data":23616,"kb_used_omap":12,"kb_used_meta":377587,"kb_avail":754573376,"statfs":{"total":773094113280,"available":772683137024,"internally_reserved":0,"allocated":24182784,"data_stored":12953914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":386649657},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000678"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104328+0000","last_change":"2024-03-18T20:25:22.679426+0000","last_active":"2024-03-18T20:25:24.104328+0000","last_peered":"2024-03-18T20:25:24.104328+0000","last_clean":"2024-03-18T20:25:24.104328+0000","last_became_active":"2024-03-18T20:25:22.679159+0000","last_became_peered":"2024-03-18T20:25:22.679159+0000","last_unstale":"2024-03-18T20:25:24.104328+0000","last_undegraded":"2024-03-18T20:25:24.104328+0000","last_fullsized":"2024-03-18T20:25:24.104328+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:17:59.556803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.672828+0000","last_change":"2024-03-18T20:25:22.685766+0000","last_active":"2024-03-18T20:25:23.672828+0000","last_peered":"2024-03-18T20:25:23.672828+0000","last_clean":"2024-03-18T20:25:23.672828+0000","last_became_active":"2024-03-18T20:25:22.683799+0000","last_became_peered":"2024-03-18T20:25:22.683799+0000","last_unstale":"2024-03-18T20:25:23.672828+0000","last_undegraded":"2024-03-18T20:25:23.672828+0000","last_fullsized":"2024-03-18T20:25:23.672828+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:42:25.581359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673444+0000","last_change":"2024-03-18T20:25:22.693152+0000","last_active":"2024-03-18T20:25:23.673444+0000","last_peered":"2024-03-18T20:25:23.673444+0000","last_clean":"2024-03-18T20:25:23.673444+0000","last_became_active":"2024-03-18T20:25:22.686521+0000","last_became_peered":"2024-03-18T20:25:22.686521+0000","last_unstale":"2024-03-18T20:25:23.673444+0000","last_undegraded":"2024-03-18T20:25:23.673444+0000","last_fullsized":"2024-03-18T20:25:23.673444+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:40:18.302530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104395+0000","last_change":"2024-03-18T20:25:22.693546+0000","last_active":"2024-03-18T20:25:24.104395+0000","last_peered":"2024-03-18T20:25:24.104395+0000","last_clean":"2024-03-18T20:25:24.104395+0000","last_became_active":"2024-03-18T20:25:22.693267+0000","last_became_peered":"2024-03-18T20:25:22.693267+0000","last_unstale":"2024-03-18T20:25:24.104395+0000","last_undegraded":"2024-03-18T20:25:24.104395+0000","last_fullsized":"2024-03-18T20:25:24.104395+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:35.486064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.672936+0000","last_change":"2024-03-18T20:25:22.690898+0000","last_active":"2024-03-18T20:25:23.672936+0000","last_peered":"2024-03-18T20:25:23.672936+0000","last_clean":"2024-03-18T20:25:23.672936+0000","last_became_active":"2024-03-18T20:25:22.690461+0000","last_became_peered":"2024-03-18T20:25:22.690461+0000","last_unstale":"2024-03-18T20:25:23.672936+0000","last_undegraded":"2024-03-18T20:25:23.672936+0000","last_fullsized":"2024-03-18T20:25:23.672936+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:15.208122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104291+0000","last_change":"2024-03-18T20:25:22.681389+0000","last_active":"2024-03-18T20:25:24.104291+0000","last_peered":"2024-03-18T20:25:24.104291+0000","last_clean":"2024-03-18T20:25:24.104291+0000","last_became_active":"2024-03-18T20:25:22.681146+0000","last_became_peered":"2024-03-18T20:25:22.681146+0000","last_unstale":"2024-03-18T20:25:24.104291+0000","last_undegraded":"2024-03-18T20:25:24.104291+0000","last_fullsized":"2024-03-18T20:25:24.104291+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:05:57.372944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673199+0000","last_change":"2024-03-18T20:25:22.692857+0000","last_active":"2024-03-18T20:25:23.673199+0000","last_peered":"2024-03-18T20:25:23.673199+0000","last_clean":"2024-03-18T20:25:23.673199+0000","last_became_active":"2024-03-18T20:25:22.692615+0000","last_became_peered":"2024-03-18T20:25:22.692615+0000","last_unstale":"2024-03-18T20:25:23.673199+0000","last_undegraded":"2024-03-18T20:25:23.673199+0000","last_fullsized":"2024-03-18T20:25:23.673199+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:21:45.938034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673244+0000","last_change":"2024-03-18T20:25:22.693769+0000","last_active":"2024-03-18T20:25:23.673244+0000","last_peered":"2024-03-18T20:25:23.673244+0000","last_clean":"2024-03-18T20:25:23.673244+0000","last_became_active":"2024-03-18T20:25:22.691936+0000","last_became_peered":"2024-03-18T20:25:22.691936+0000","last_unstale":"2024-03-18T20:25:23.673244+0000","last_undegraded":"2024-03-18T20:25:23.673244+0000","last_fullsized":"2024-03-18T20:25:23.673244+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:29:49.401587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673379+0000","last_change":"2024-03-18T20:25:22.693670+0000","last_active":"2024-03-18T20:25:23.673379+0000","last_peered":"2024-03-18T20:25:23.673379+0000","last_clean":"2024-03-18T20:25:23.673379+0000","last_became_active":"2024-03-18T20:25:22.693178+0000","last_became_peered":"2024-03-18T20:25:22.693178+0000","last_unstale":"2024-03-18T20:25:23.673379+0000","last_undegraded":"2024-03-18T20:25:23.673379+0000","last_fullsized":"2024-03-18T20:25:23.673379+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:13.992732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673104+0000","last_change":"2024-03-18T20:25:22.691577+0000","last_active":"2024-03-18T20:25:23.673104+0000","last_peered":"2024-03-18T20:25:23.673104+0000","last_clean":"2024-03-18T20:25:23.673104+0000","last_became_active":"2024-03-18T20:25:22.691250+0000","last_became_peered":"2024-03-18T20:25:22.691250+0000","last_unstale":"2024-03-18T20:25:23.673104+0000","last_undegraded":"2024-03-18T20:25:23.673104+0000","last_fullsized":"2024-03-18T20:25:23.673104+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:55:21.946848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104453+0000","last_change":"2024-03-18T20:25:22.689262+0000","last_active":"2024-03-18T20:25:24.104453+0000","last_peered":"2024-03-18T20:25:24.104453+0000","last_clean":"2024-03-18T20:25:24.104453+0000","last_became_active":"2024-03-18T20:25:22.688913+0000","last_became_peered":"2024-03-18T20:25:22.688913+0000","last_unstale":"2024-03-18T20:25:24.104453+0000","last_undegraded":"2024-03-18T20:25:24.104453+0000","last_fullsized":"2024-03-18T20:25:24.104453+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:15:50.135885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673294+0000","last_change":"2024-03-18T20:25:22.691295+0000","last_active":"2024-03-18T20:25:23.673294+0000","last_peered":"2024-03-18T20:25:23.673294+0000","last_clean":"2024-03-18T20:25:23.673294+0000","last_became_active":"2024-03-18T20:25:22.690890+0000","last_became_peered":"2024-03-18T20:25:22.690890+0000","last_unstale":"2024-03-18T20:25:23.673294+0000","last_undegraded":"2024-03-18T20:25:23.673294+0000","last_fullsized":"2024-03-18T20:25:23.673294+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:23:16.320044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104245+0000","last_change":"2024-03-18T20:25:22.244727+0000","last_active":"2024-03-18T20:25:24.104245+0000","last_peered":"2024-03-18T20:25:24.104245+0000","last_clean":"2024-03-18T20:25:24.104245+0000","last_became_active":"2024-03-18T20:25:22.244417+0000","last_became_peered":"2024-03-18T20:25:22.244417+0000","last_unstale":"2024-03-18T20:25:24.104245+0000","last_undegraded":"2024-03-18T20:25:24.104245+0000","last_fullsized":"2024-03-18T20:25:24.104245+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:10.187509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104215+0000","last_change":"2024-03-18T20:25:22.684633+0000","last_active":"2024-03-18T20:25:24.104215+0000","last_peered":"2024-03-18T20:25:24.104215+0000","last_clean":"2024-03-18T20:25:24.104215+0000","last_became_active":"2024-03-18T20:25:22.684366+0000","last_became_peered":"2024-03-18T20:25:22.684366+0000","last_unstale":"2024-03-18T20:25:24.104215+0000","last_undegraded":"2024-03-18T20:25:24.104215+0000","last_fullsized":"2024-03-18T20:25:24.104215+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:15.299142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104504+0000","last_change":"2024-03-18T20:25:22.685090+0000","last_active":"2024-03-18T20:25:24.104504+0000","last_peered":"2024-03-18T20:25:24.104504+0000","last_clean":"2024-03-18T20:25:24.104504+0000","last_became_active":"2024-03-18T20:25:22.684824+0000","last_became_peered":"2024-03-18T20:25:22.684824+0000","last_unstale":"2024-03-18T20:25:24.104504+0000","last_undegraded":"2024-03-18T20:25:24.104504+0000","last_fullsized":"2024-03-18T20:25:24.104504+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:21.750701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104292+0000","last_change":"2024-03-18T20:25:22.249078+0000","last_active":"2024-03-18T20:25:24.104292+0000","last_peered":"2024-03-18T20:25:24.104292+0000","last_clean":"2024-03-18T20:25:24.104292+0000","last_became_active":"2024-03-18T20:25:22.248588+0000","last_became_peered":"2024-03-18T20:25:22.248588+0000","last_unstale":"2024-03-18T20:25:24.104292+0000","last_undegraded":"2024-03-18T20:25:24.104292+0000","last_fullsized":"2024-03-18T20:25:24.104292+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:34:42.893253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.672855+0000","last_change":"2024-03-18T20:25:22.686185+0000","last_active":"2024-03-18T20:25:23.672855+0000","last_peered":"2024-03-18T20:25:23.672855+0000","last_clean":"2024-03-18T20:25:23.672855+0000","last_became_active":"2024-03-18T20:25:22.685939+0000","last_became_peered":"2024-03-18T20:25:22.685939+0000","last_unstale":"2024-03-18T20:25:23.672855+0000","last_undegraded":"2024-03-18T20:25:23.672855+0000","last_fullsized":"2024-03-18T20:25:23.672855+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:48:48.961872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673112+0000","last_change":"2024-03-18T20:25:22.692895+0000","last_active":"2024-03-18T20:25:23.673112+0000","last_peered":"2024-03-18T20:25:23.673112+0000","last_clean":"2024-03-18T20:25:23.673112+0000","last_became_active":"2024-03-18T20:25:22.685766+0000","last_became_peered":"2024-03-18T20:25:22.685766+0000","last_unstale":"2024-03-18T20:25:23.673112+0000","last_undegraded":"2024-03-18T20:25:23.673112+0000","last_fullsized":"2024-03-18T20:25:23.673112+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:16:08.020891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673490+0000","last_change":"2024-03-18T20:25:22.695653+0000","last_active":"2024-03-18T20:25:23.673490+0000","last_peered":"2024-03-18T20:25:23.673490+0000","last_clean":"2024-03-18T20:25:23.673490+0000","last_became_active":"2024-03-18T20:25:22.694092+0000","last_became_peered":"2024-03-18T20:25:22.694092+0000","last_unstale":"2024-03-18T20:25:23.673490+0000","last_undegraded":"2024-03-18T20:25:23.673490+0000","last_fullsized":"2024-03-18T20:25:23.673490+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:20.319345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.672984+0000","last_change":"2024-03-18T20:25:22.693170+0000","last_active":"2024-03-18T20:25:23.672984+0000","last_peered":"2024-03-18T20:25:23.672984+0000","last_clean":"2024-03-18T20:25:23.672984+0000","last_became_active":"2024-03-18T20:25:22.691592+0000","last_became_peered":"2024-03-18T20:25:22.691592+0000","last_unstale":"2024-03-18T20:25:23.672984+0000","last_undegraded":"2024-03-18T20:25:23.672984+0000","last_fullsized":"2024-03-18T20:25:23.672984+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:23.897085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104222+0000","last_change":"2024-03-18T20:25:22.689011+0000","last_active":"2024-03-18T20:25:24.104222+0000","last_peered":"2024-03-18T20:25:24.104222+0000","last_clean":"2024-03-18T20:25:24.104222+0000","last_became_active":"2024-03-18T20:25:22.688750+0000","last_became_peered":"2024-03-18T20:25:22.688750+0000","last_unstale":"2024-03-18T20:25:24.104222+0000","last_undegraded":"2024-03-18T20:25:24.104222+0000","last_fullsized":"2024-03-18T20:25:24.104222+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:57.345369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673442+0000","last_change":"2024-03-18T20:25:22.247183+0000","last_active":"2024-03-18T20:25:23.673442+0000","last_peered":"2024-03-18T20:25:23.673442+0000","last_clean":"2024-03-18T20:25:23.673442+0000","last_became_active":"2024-03-18T20:25:22.246890+0000","last_became_peered":"2024-03-18T20:25:22.246890+0000","last_unstale":"2024-03-18T20:25:23.673442+0000","last_undegraded":"2024-03-18T20:25:23.673442+0000","last_fullsized":"2024-03-18T20:25:23.673442+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:40:21.886849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673372+0000","last_change":"2024-03-18T20:25:22.683574+0000","last_active":"2024-03-18T20:25:23.673372+0000","last_peered":"2024-03-18T20:25:23.673372+0000","last_clean":"2024-03-18T20:25:23.673372+0000","last_became_active":"2024-03-18T20:25:22.683015+0000","last_became_peered":"2024-03-18T20:25:22.683015+0000","last_unstale":"2024-03-18T20:25:23.673372+0000","last_undegraded":"2024-03-18T20:25:23.673372+0000","last_fullsized":"2024-03-18T20:25:23.673372+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:11.606826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673204+0000","last_change":"2024-03-18T20:25:22.245633+0000","last_active":"2024-03-18T20:25:23.673204+0000","last_peered":"2024-03-18T20:25:23.673204+0000","last_clean":"2024-03-18T20:25:23.673204+0000","last_became_active":"2024-03-18T20:25:22.245372+0000","last_became_peered":"2024-03-18T20:25:22.245372+0000","last_unstale":"2024-03-18T20:25:23.673204+0000","last_undegraded":"2024-03-18T20:25:23.673204+0000","last_fullsized":"2024-03-18T20:25:23.673204+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:28:25.048682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104385+0000","last_change":"2024-03-18T20:25:22.696025+0000","last_active":"2024-03-18T20:25:24.104385+0000","last_peered":"2024-03-18T20:25:24.104385+0000","last_clean":"2024-03-18T20:25:24.104385+0000","last_became_active":"2024-03-18T20:25:22.695759+0000","last_became_peered":"2024-03-18T20:25:22.695759+0000","last_unstale":"2024-03-18T20:25:24.104385+0000","last_undegraded":"2024-03-18T20:25:24.104385+0000","last_fullsized":"2024-03-18T20:25:24.104385+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:10.681789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:22.224682+0000","last_change":"2024-03-18T20:25:22.224682+0000","last_active":"2024-03-18T20:25:22.224682+0000","last_peered":"2024-03-18T20:25:22.224682+0000","last_clean":"2024-03-18T20:25:22.224682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:22.224682+0000","last_undegraded":"2024-03-18T20:25:22.224682+0000","last_fullsized":"2024-03-18T20:25:22.224682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:21.217378+0000","last_change":"2024-03-18T20:25:21.217378+0000","last_active":"2024-03-18T20:25:21.217378+0000","last_peered":"2024-03-18T20:25:21.217378+0000","last_clean":"2024-03-18T20:25:21.217378+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:21.217378+0000","last_undegraded":"2024-03-18T20:25:21.217378+0000","last_fullsized":"2024-03-18T20:25:21.217378+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"221'348","reported_seq":1196,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-03-18T20:25:18.234579+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:25:18.234579+0000","last_peered":"2024-03-18T20:25:18.234579+0000","last_clean":"2024-03-18T20:25:18.234579+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:25:18.234579+0000","last_undegraded":"2024-03-18T20:25:18.234579+0000","last_fullsized":"2024-03-18T20:25:18.234579+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":42,"acting":42,"num_store_stats":2},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":2},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575980,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52544,"kb_used_data":3376,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319296,"statfs":{"total":96636764160,"available":96582959104,"internally_reserved":0,"allocated":3457024,"data_stored":2056562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80700000000000005,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.434},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.48499999999999999,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.81899999999999995,"5min":1.093,"15min":1.093},"last":0.46800000000000003}]},{"osd":1,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61799999999999999,"15min":0.61699999999999999},"min":{"1min":0.47599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.82999999999999996,"5min":0.99399999999999999,"15min":1.4299999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.60999999999999999,"15min":0.61899999999999999},"min":{"1min":0.47099999999999997,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.77500000000000002,"5min":1.0449999999999999,"15min":1.2629999999999999},"last":0.71299999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.46400000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84299999999999997,"5min":1.216,"15min":1.216},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.62,"15min":0.625},"min":{"1min":0.48399999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.81799999999999995,"5min":1.177,"15min":1.2490000000000001},"last":0.73999999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.42599999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84699999999999998,"5min":0.93100000000000005,"15min":1.4450000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.5,"5min":0.44900000000000001,"15min":0.38600000000000001},"max":{"1min":0.84399999999999997,"5min":1.028,"15min":1.5029999999999999},"last":0.83399999999999996}]},{"osd":4,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.47499999999999998,"15min":0.47699999999999998},"min":{"1min":0.33700000000000002,"5min":0.314,"15min":0.29899999999999999},"max":{"1min":0.68400000000000005,"5min":0.79100000000000004,"15min":0.86099999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.35199999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71599999999999997,"5min":0.76600000000000001,"15min":0.80200000000000005},"last":0.96199999999999997}]},{"osd":5,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51500000000000001,"15min":0.51600000000000001},"min":{"1min":0.34899999999999998,"5min":0.34399999999999997,"15min":0.316},"max":{"1min":0.68999999999999995,"5min":0.86599999999999999,"15min":0.89400000000000002},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.40699999999999997,"5min":0.377,"15min":0.36799999999999999},"max":{"1min":0.75600000000000001,"5min":0.81599999999999995,"15min":1.042},"last":0.877}]},{"osd":6,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56399999999999995,"15min":0.57699999999999996},"min":{"1min":0.39900000000000002,"5min":0.36199999999999999,"15min":0.34999999999999998},"max":{"1min":0.71099999999999997,"5min":0.85599999999999998,"15min":1.129},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.371,"5min":0.36299999999999999,"15min":0.34000000000000002},"max":{"1min":0.82899999999999996,"5min":0.85299999999999998,"15min":1.0609999999999999},"last":0.91900000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575981,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42180,"kb_used_data":2860,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329660,"statfs":{"total":96636764160,"available":96593571840,"internally_reserved":0,"allocated":2928640,"data_stored":1495187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57599999999999996,"15min":0.60099999999999998},"min":{"1min":0.41399999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72199999999999998,"5min":0.80900000000000005,"15min":1.1519999999999999},"last":0.371},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59099999999999997,"15min":0.60799999999999998},"min":{"1min":0.33700000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.74299999999999999,"5min":0.81499999999999995,"15min":0.83299999999999996},"last":0.45900000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57899999999999996,"15min":0.60199999999999998},"min":{"1min":0.39400000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.79100000000000004,"5min":0.79800000000000004,"15min":1.244},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58499999999999996,"15min":0.60399999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33200000000000002},"max":{"1min":0.77000000000000002,"5min":0.89200000000000002,"15min":1.286},"last":0.47099999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60999999999999999,"15min":0.63},"min":{"1min":0.47899999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71199999999999997,"5min":0.92000000000000004,"15min":1.276},"last":0.308},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.59299999999999997,"15min":0.61099999999999999},"min":{"1min":0.434,"5min":0.40100000000000002,"15min":0.378},"max":{"1min":0.77500000000000002,"5min":0.95399999999999996,"15min":1.2669999999999999},"last":0.39500000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59699999999999998,"15min":0.622},"min":{"1min":0.38,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70699999999999996,"5min":0.83199999999999996,"15min":1.02},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60599999999999998,"15min":0.61899999999999999},"min":{"1min":0.45200000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82399999999999995,"5min":1.0049999999999999,"15min":1.337},"last":0.44400000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.48099999999999998,"15min":0.48299999999999998},"min":{"1min":0.308,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.61699999999999999,"5min":0.85799999999999998,"15min":0.88300000000000001},"last":0.435},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.50900000000000001,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72099999999999997,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.49099999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.5,"15min":0.504},"min":{"1min":0.375,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.84799999999999998},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39000000000000001,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.73399999999999999,"5min":0.92600000000000005,"15min":0.94199999999999995},"last":0.501}]},{"osd":7,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40600000000000003,"5min":0.35799999999999998,"15min":0.34499999999999997},"max":{"1min":0.74299999999999999,"5min":0.95599999999999996,"15min":1.0429999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.44800000000000001,"5min":0.36599999999999999,"15min":0.34599999999999997},"max":{"1min":0.83899999999999997,"5min":1.028,"15min":1.028},"last":0.51200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575981,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47912,"kb_used_data":2844,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323928,"statfs":{"total":96636764160,"available":96587702272,"internally_reserved":0,"allocated":2912256,"data_stored":1493041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.40400000000000003,"15min":0.39600000000000002},"min":{"1min":0.25600000000000001,"5min":0.246,"15min":0.23999999999999999},"max":{"1min":0.57999999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.41899999999999998,"15min":0.42999999999999999},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.58999999999999997,"5min":0.69699999999999995,"15min":0.72299999999999998},"last":0.56499999999999995}]},{"osd":2,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.44,"15min":0.439},"min":{"1min":0.309,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61199999999999999,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48099999999999998,"15min":0.47799999999999998},"min":{"1min":0.35099999999999998,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.70099999999999996,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.39300000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.495,"15min":0.495},"min":{"1min":0.36199999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.64300000000000002,"5min":0.75900000000000001,"15min":0.76100000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.44800000000000001,"15min":0.442},"min":{"1min":0.32800000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68200000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.41799999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.68300000000000005,"15min":0.67300000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86799999999999999,"5min":1.0680000000000001,"15min":1.2909999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.68100000000000005,"15min":0.68000000000000005},"min":{"1min":0.50900000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.84499999999999997,"5min":0.92500000000000004,"15min":1.746},"last":0.70299999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.68100000000000005,"15min":0.67000000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.6910000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.71999999999999997,"15min":0.71399999999999997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.46899999999999997},"max":{"1min":0.89400000000000002,"5min":1.109,"15min":1.2549999999999999},"last":0.91100000000000003}]},{"osd":6,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.69699999999999995,"15min":0.68500000000000005},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.437},"max":{"1min":0.97399999999999998,"5min":0.98899999999999999,"15min":1.637},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71899999999999997,"15min":0.71199999999999997},"min":{"1min":0.49099999999999999,"5min":0.46999999999999997,"15min":0.46100000000000002},"max":{"1min":0.89500000000000002,"5min":1.014,"15min":1.722},"last":0.68899999999999995}]},{"osd":7,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.70699999999999996,"15min":0.71299999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79800000000000004,"5min":1.02,"15min":1.2729999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.498,"5min":0.495,"15min":0.42799999999999999},"max":{"1min":0.90300000000000002,"5min":1.0329999999999999,"15min":1.3380000000000001},"last":0.84199999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575981,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46780,"kb_used_data":3376,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94325060,"statfs":{"total":96636764160,"available":96588861440,"internally_reserved":0,"allocated":3457024,"data_stored":2056562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44431814},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.42399999999999999,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.42099999999999999,"15min":0.42799999999999999},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.46600000000000003,"5min":0.64700000000000002,"15min":1.1850000000000001},"last":0.40100000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.42599999999999999,"15min":0.435},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.47099999999999997,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.5},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.438,"15min":0.45300000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29799999999999999},"max":{"1min":0.59199999999999997,"5min":0.64300000000000002,"15min":0.70999999999999996},"last":0.28100000000000003}]},{"osd":3,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.26100000000000001},"max":{"1min":0.45800000000000002,"5min":0.60099999999999998,"15min":0.628},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48599999999999999,"15min":0.496},"min":{"1min":0.35499999999999998,"5min":0.35199999999999998,"15min":0.32500000000000001},"max":{"1min":0.52300000000000002,"5min":0.73999999999999999,"15min":0.91500000000000004},"last":0.46100000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64300000000000002,"15min":0.65500000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81299999999999994,"5min":0.91500000000000004,"15min":1.0029999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.66200000000000003,"15min":0.66300000000000003},"min":{"1min":0.47499999999999998,"5min":0.47099999999999997,"15min":0.39700000000000002},"max":{"1min":0.78100000000000003,"5min":0.95299999999999996,"15min":1.0469999999999999},"last":0.624}]},{"osd":5,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.67800000000000005,"15min":0.67100000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89700000000000002,"5min":0.97099999999999997,"15min":1.0129999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91100000000000003,"5min":0.92900000000000005,"15min":1.518},"last":0.58099999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.47399999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.78600000000000003,"5min":0.95899999999999996,"15min":1.1279999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71199999999999997,"15min":0.70899999999999996},"min":{"1min":0.498,"5min":0.48999999999999999,"15min":0.47899999999999998},"max":{"1min":0.94899999999999995,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.751}]},{"osd":7,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85099999999999998,"5min":0.96399999999999997,"15min":1.464},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.69899999999999995,"15min":0.70399999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.47599999999999998},"max":{"1min":0.75800000000000001,"5min":1.042,"15min":1.2350000000000001},"last":0.70399999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575981,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":47872,"kb_used_data":2796,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323968,"statfs":{"total":96636764160,"available":96587743232,"internally_reserved":0,"allocated":2863104,"data_stored":1466194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.379,"15min":0.38},"min":{"1min":0.27700000000000002,"5min":0.251,"15min":0.23899999999999999},"max":{"1min":0.54800000000000004,"5min":0.55300000000000005,"15min":0.70799999999999996},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.48199999999999998,"15min":0.48899999999999999},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.24199999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.83399999999999996},"last":0.47899999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48099999999999998,"15min":0.48199999999999998},"min":{"1min":0.36299999999999999,"5min":0.34399999999999997,"15min":0.30299999999999999},"max":{"1min":0.59499999999999997,"5min":0.72599999999999998,"15min":0.77100000000000002},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.42099999999999999,"15min":0.41999999999999998},"min":{"1min":0.28399999999999997,"5min":0.27800000000000002,"15min":0.27600000000000002},"max":{"1min":0.52000000000000002,"5min":0.59299999999999997,"15min":0.71099999999999997},"last":0.45200000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.437,"15min":0.437},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.58999999999999997,"5min":0.64700000000000002,"15min":0.66300000000000003},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.51800000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.313},"max":{"1min":0.66200000000000003,"5min":0.752,"15min":0.752},"last":0.42799999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.66500000000000004,"15min":0.65800000000000003},"min":{"1min":0.44400000000000001,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":0.76500000000000001,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.66800000000000004},"min":{"1min":0.48899999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.95799999999999996,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.60799999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68200000000000005,"15min":0.68400000000000005},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.45300000000000001},"max":{"1min":0.97299999999999998,"5min":1.028,"15min":1.028},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.68100000000000005,"15min":0.67900000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.88200000000000001,"5min":0.95599999999999996,"15min":1.0629999999999999},"last":0.876}]},{"osd":6,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69499999999999995,"15min":0.69399999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.014,"5min":1.014,"15min":1.2050000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.73199999999999998,"15min":0.73099999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.90200000000000002,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.85899999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.501,"5min":0.442,"15min":0.42199999999999999},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":1.0469999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.70699999999999996,"15min":0.70899999999999996},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.92400000000000004,"5min":1.371,"15min":1.371},"last":0.67800000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575981,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51948,"kb_used_data":2784,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319892,"statfs":{"total":96636764160,"available":96583569408,"internally_reserved":0,"allocated":2850816,"data_stored":1460087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.38800000000000001,"15min":0.39900000000000002},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.53700000000000003,"5min":0.66500000000000004,"15min":0.98099999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.40999999999999998,"15min":0.41499999999999998},"min":{"1min":0.27700000000000002,"5min":0.25800000000000001,"15min":0.24199999999999999},"max":{"1min":0.434,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.29899999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39000000000000001,"15min":0.40999999999999998},"min":{"1min":0.30099999999999999,"5min":0.26100000000000001,"15min":0.254},"max":{"1min":0.45900000000000002,"5min":0.63200000000000001,"15min":0.64000000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.436,"5min":0.46100000000000002,"15min":0.46600000000000003},"min":{"1min":0.36699999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57499999999999996,"5min":0.65500000000000003,"15min":0.73099999999999998},"last":0.35199999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.45500000000000002,"15min":0.46600000000000003},"min":{"1min":0.34999999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53100000000000003,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.48199999999999998,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.51700000000000002,"5min":0.67000000000000004,"15min":0.73099999999999998},"last":0.40899999999999997}]},{"osd":4,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.63600000000000001,"15min":0.65200000000000002},"min":{"1min":0.48799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81899999999999995,"5min":0.91700000000000004,"15min":1.097},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65200000000000002,"15min":0.67500000000000004},"min":{"1min":0.435,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78800000000000003,"5min":0.99299999999999999,"15min":1.0569999999999999},"last":0.79800000000000004}]},{"osd":5,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68400000000000005,"15min":0.69699999999999995},"min":{"1min":0.48999999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92100000000000004,"5min":0.995,"15min":1.1599999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.72499999999999998,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.82099999999999995,"5min":1.171,"15min":1.171},"last":0.70699999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68500000000000005,"15min":0.70799999999999996},"min":{"1min":0.48099999999999998,"5min":0.436,"15min":0.42499999999999999},"max":{"1min":0.90500000000000003,"5min":1.012,"15min":1.194},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.71799999999999997,"15min":0.73299999999999998},"min":{"1min":0.53300000000000003,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":0.88200000000000001,"5min":1.0409999999999999,"15min":1.2410000000000001},"last":0.57299999999999995}]},{"osd":7,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.70099999999999996,"15min":0.71899999999999997},"min":{"1min":0.49099999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82799999999999996,"5min":1.079,"15min":1.079},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68100000000000005,"15min":0.69999999999999996},"min":{"1min":0.44500000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83799999999999997,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.74299999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575981,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64236,"kb_used_data":2784,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307604,"statfs":{"total":96636764160,"available":96570986496,"internally_reserved":0,"allocated":2850816,"data_stored":1460087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58999999999999997,"15min":0.60099999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.81799999999999995},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.745,"5min":0.79800000000000004,"15min":0.84199999999999997},"last":0.59799999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.60599999999999998,"15min":0.61899999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.79000000000000004,"5min":0.81899999999999995,"15min":0.88300000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.75700000000000001,"5min":0.79200000000000004,"15min":0.96499999999999997},"last":0.35399999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.60299999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39200000000000002},"max":{"1min":0.79100000000000004,"5min":0.80500000000000005,"15min":0.82599999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.60999999999999999,"15min":0.623},"min":{"1min":0.46600000000000003,"5min":0.41699999999999998,"15min":0.38500000000000001},"max":{"1min":0.70599999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.52100000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59799999999999998,"15min":0.60599999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":0.76900000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60399999999999998,"15min":0.625},"min":{"1min":0.432,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73199999999999998,"5min":0.79100000000000004,"15min":0.86199999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.46100000000000002,"15min":0.47799999999999998},"min":{"1min":0.35599999999999998,"5min":0.33600000000000002,"15min":0.315},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.76000000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48599999999999999,"15min":0.504},"min":{"1min":0.30299999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.432}]},{"osd":6,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48599999999999999,"15min":0.50600000000000001},"min":{"1min":0.34899999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.77100000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.54700000000000004},"min":{"1min":0.35399999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.74099999999999999,"5min":0.76200000000000001,"15min":0.79900000000000004},"last":0.61899999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52700000000000002,"15min":0.54500000000000004},"min":{"1min":0.39600000000000002,"5min":0.38,"15min":0.35699999999999998},"max":{"1min":0.78800000000000003,"5min":0.87,"15min":0.87},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55700000000000005,"15min":0.57899999999999996},"min":{"1min":0.438,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.71999999999999997,"5min":0.81200000000000006,"15min":0.85399999999999998},"last":0.58199999999999996}]}]},{"osd":5,"up_from":13,"seq":55834575980,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":47872,"kb_used_data":2796,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323968,"statfs":{"total":96636764160,"available":96587743232,"internally_reserved":0,"allocated":2863104,"data_stored":1466194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42499999999999999,"5min":0.33900000000000002,"15min":0.31900000000000001},"max":{"1min":0.72699999999999998,"5min":0.85999999999999999,"15min":0.96499999999999997},"last":0.443},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57799999999999996,"15min":0.60399999999999998},"min":{"1min":0.432,"5min":0.37,"15min":0.33100000000000002},"max":{"1min":0.69799999999999995,"5min":0.82599999999999996,"15min":1.3100000000000001},"last":0.57199999999999995}]},{"osd":1,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58499999999999996,"15min":0.60199999999999998},"min":{"1min":0.36899999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.74399999999999999,"5min":0.79600000000000004,"15min":1.056},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58399999999999996,"15min":0.60199999999999998},"min":{"1min":0.46600000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.63100000000000001,"5min":0.89800000000000002,"15min":1.018},"last":0.52400000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.38700000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.84299999999999997,"15min":0.99299999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59399999999999997,"15min":0.60899999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.67500000000000004,"5min":0.875,"15min":1.1120000000000001},"last":0.41599999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.42799999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.78100000000000003,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63,"15min":0.64600000000000002},"min":{"1min":0.443,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69899999999999995,"5min":0.82399999999999995,"15min":1.161},"last":0.53800000000000003}]},{"osd":4,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.47899999999999998,"15min":0.47499999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.77500000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51200000000000001,"15min":0.50600000000000001},"min":{"1min":0.36199999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.89700000000000002},"last":0.38700000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.503,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.34000000000000002,"15min":0.32800000000000001},"max":{"1min":0.69999999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.40300000000000002,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.86599999999999999},"last":0.45600000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.40799999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.94499999999999995},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.45300000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74099999999999999,"5min":0.85999999999999999,"15min":0.91200000000000003},"last":0.61799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:25:26.830 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:25:26.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.987+0000 7fb6a6e24700 1 Processor -- start 2024-03-18T20:25:26.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.987+0000 7fb6a6e24700 1 -- start start 2024-03-18T20:25:26.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.988+0000 7fb6a6e24700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6a0146190 0x7fb6a006a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:26.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.988+0000 7fb6a6e24700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb6a0146af0 0x7fb6a006ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:26.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.988+0000 7fb6a6e24700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a006ebd0 con 0x7fb6a01465a0 2024-03-18T20:25:26.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.988+0000 7fb6a6e24700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6a006f850 con 0x7fb6a0146af0 2024-03-18T20:25:26.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.988+0000 7fb6a6e24700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a006eeb0 con 0x7fb6a0146190 2024-03-18T20:25:26.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.988+0000 7fb69ffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb6a01465a0 0x7fb6a0071650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39518/0 (socket says 172.21.15.1:39518) 2024-03-18T20:25:26.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.988+0000 7fb69ffff700 1 -- 172.21.15.1:0/2432350519 learned_addr learned my addr 172.21.15.1:0/2432350519 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:26.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb6a4bc0700 1 --2- 172.21.15.1:0/2432350519 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6a0146190 0x7fb6a006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:26.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb6a53c1700 1 --2- 172.21.15.1:0/2432350519 >> v2:172.21.15.165:3300/0 conn(0x7fb6a0146af0 0x7fb6a006ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:26.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69f7fe700 1 -- 172.21.15.1:0/2432350519 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935109140 0 0) 0x7fb6a006ebd0 con 0x7fb6a01465a0 2024-03-18T20:25:26.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69f7fe700 1 -- 172.21.15.1:0/2432350519 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6900035b0 con 0x7fb6a01465a0 2024-03-18T20:25:26.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69f7fe700 1 -- 172.21.15.1:0/2432350519 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1556853253 0 0) 0x7fb6900035b0 con 0x7fb6a01465a0 2024-03-18T20:25:26.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69f7fe700 1 -- 172.21.15.1:0/2432350519 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6a0146190 msgr2=0x7fb6a006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:26.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69f7fe700 1 --2- 172.21.15.1:0/2432350519 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6a0146190 0x7fb6a006a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69f7fe700 1 -- 172.21.15.1:0/2432350519 >> v2:172.21.15.165:3300/0 conn(0x7fb6a0146af0 msgr2=0x7fb6a006ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:26.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69f7fe700 1 --2- 172.21.15.1:0/2432350519 >> v2:172.21.15.165:3300/0 conn(0x7fb6a0146af0 0x7fb6a006ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69f7fe700 1 -- 172.21.15.1:0/2432350519 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a006f190 con 0x7fb6a01465a0 2024-03-18T20:25:26.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69f7fe700 1 -- 172.21.15.1:0/2432350519 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fb68c001670 con 0x7fb6a01465a0 2024-03-18T20:25:26.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69f7fe700 1 -- 172.21.15.1:0/2432350519 >> v1:172.21.15.1:6789/0 conn(0x7fb6a01465a0 legacy=0x7fb6a0071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:26.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69f7fe700 1 --2- 172.21.15.1:0/2432350519 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb690003a00 0x7fb690003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:26.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69f7fe700 1 --2- 172.21.15.1:0/2432350519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb690004360 0x7fb690004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:26.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69f7fe700 1 --2- 172.21.15.1:0/2432350519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb690004ea0 0x7fb690005320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:26.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69f7fe700 1 -- 172.21.15.1:0/2432350519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a006f850 con 0x7fb690004360 2024-03-18T20:25:26.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69f7fe700 1 -- 172.21.15.1:0/2432350519 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a006eeb0 con 0x7fb690004ea0 2024-03-18T20:25:26.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69f7fe700 1 -- 172.21.15.1:0/2432350519 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb69000bd90 con 0x7fb690003a00 2024-03-18T20:25:26.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69ffff700 1 --2- 172.21.15.1:0/2432350519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb690004360 0x7fb690004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:26.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb6a4bc0700 1 --2- 172.21.15.1:0/2432350519 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb690003a00 0x7fb690003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:26.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb6a53c1700 1 --2- 172.21.15.1:0/2432350519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb690004ea0 0x7fb690005320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:26.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69ffff700 1 -- 172.21.15.1:0/2432350519 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb690003a00 msgr2=0x7fb690003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:26.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69ffff700 1 --2- 172.21.15.1:0/2432350519 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb690003a00 0x7fb690003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69ffff700 1 -- 172.21.15.1:0/2432350519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb690004ea0 msgr2=0x7fb690005320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:26.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69ffff700 1 --2- 172.21.15.1:0/2432350519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb690004ea0 0x7fb690005320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:26.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.989+0000 7fb69ffff700 1 -- 172.21.15.1:0/2432350519 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6900035b0 con 0x7fb690004360 2024-03-18T20:25:26.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.990+0000 7fb69ffff700 1 --2- 172.21.15.1:0/2432350519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb690004360 0x7fb690004850 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00c2f0 tx=0x7fb690005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea1234defa8e575 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:26.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.990+0000 7fb69f7fe700 1 -- 172.21.15.1:0/2432350519 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c00ce30 con 0x7fb690004360 2024-03-18T20:25:26.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.990+0000 7fb69f7fe700 1 -- 172.21.15.1:0/2432350519 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68c005960 con 0x7fb690004360 2024-03-18T20:25:26.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.990+0000 7fb69f7fe700 1 -- 172.21.15.1:0/2432350519 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c005d10 con 0x7fb690004360 2024-03-18T20:25:26.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.990+0000 7fb6a6e24700 1 -- 172.21.15.1:0/2432350519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb690004360 msgr2=0x7fb690004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:26.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.990+0000 7fb6a6e24700 1 --2- 172.21.15.1:0/2432350519 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb690004360 0x7fb690004850 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00c2f0 tx=0x7fb690005880 comp rx=0 tx=0).stop 2024-03-18T20:25:26.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.990+0000 7fb6a53c1700 1 -- 172.21.15.1:0/2432350519 reap_dead start 2024-03-18T20:25:26.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.990+0000 7fb6a6e24700 1 -- 172.21.15.1:0/2432350519 shutdown_connections 2024-03-18T20:25:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.990+0000 7fb6a6e24700 1 -- 172.21.15.1:0/2432350519 >> 172.21.15.1:0/2432350519 conn(0x7fb6a00b1d20 msgr2=0x7fb6a00b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.990+0000 7fb6a6e24700 1 -- 172.21.15.1:0/2432350519 shutdown_connections 2024-03-18T20:25:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.990+0000 7fb6a6e24700 1 -- 172.21.15.1:0/2432350519 wait complete. 2024-03-18T20:25:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.991+0000 7fb6a6e24700 1 Processor -- start 2024-03-18T20:25:26.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.992+0000 7fb6a6e24700 1 -- start start 2024-03-18T20:25:27.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.992+0000 7fb6a6e24700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb690004360 0x7fb6a0144a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:27.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.992+0000 7fb6a6e24700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6a01f2b20 0x7fb6a00ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:27.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.992+0000 7fb6a6e24700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6a00cea00 0x7fb6a01e7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:27.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.992+0000 7fb6a6e24700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a0144f50 con 0x7fb690004360 2024-03-18T20:25:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.992+0000 7fb6a6e24700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a01f7050 con 0x7fb6a01f2b20 2024-03-18T20:25:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.992+0000 7fb6a6e24700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a01f64d0 con 0x7fb6a00cea00 2024-03-18T20:25:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.992+0000 7fb6a4bc0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb690004360 0x7fb6a0144a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.993+0000 7fb6a4bc0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb690004360 0x7fb6a0144a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59864/0 (socket says 172.21.15.1:59864) 2024-03-18T20:25:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.993+0000 7fb6a4bc0700 1 -- 172.21.15.1:0/619748214 learned_addr learned my addr 172.21.15.1:0/619748214 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.993+0000 7fb69ffff700 1 --2- 172.21.15.1:0/619748214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6a01f2b20 0x7fb6a00ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.993+0000 7fb6a53c1700 1 --2- 172.21.15.1:0/619748214 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6a00cea00 0x7fb6a01e7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.993+0000 7fb6a4bc0700 1 -- 172.21.15.1:0/619748214 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6a00cea00 msgr2=0x7fb6a01e7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.993+0000 7fb6a4bc0700 1 --2- 172.21.15.1:0/619748214 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6a00cea00 0x7fb6a01e7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.993+0000 7fb6a4bc0700 1 -- 172.21.15.1:0/619748214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6a01f2b20 msgr2=0x7fb6a00ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.993+0000 7fb6a4bc0700 1 --2- 172.21.15.1:0/619748214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6a01f2b20 0x7fb6a00ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.993+0000 7fb6a4bc0700 1 -- 172.21.15.1:0/619748214 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a01f67b0 con 0x7fb690004360 2024-03-18T20:25:27.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.993+0000 7fb6a53c1700 1 --2- 172.21.15.1:0/619748214 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6a00cea00 0x7fb6a01e7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:27.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.993+0000 7fb69ffff700 1 --2- 172.21.15.1:0/619748214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6a01f2b20 0x7fb6a00ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:27.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.993+0000 7fb6a4bc0700 1 --2- 172.21.15.1:0/619748214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb690004360 0x7fb6a0144a10 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7fb6940119a0 tx=0x7fb694011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:27.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.993+0000 7fb69d7fa700 1 -- 172.21.15.1:0/619748214 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69400f940 con 0x7fb690004360 2024-03-18T20:25:27.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.993+0000 7fb69d7fa700 1 -- 172.21.15.1:0/619748214 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69400faa0 con 0x7fb690004360 2024-03-18T20:25:27.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.993+0000 7fb6a6e24700 1 -- 172.21.15.1:0/619748214 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a01f6a90 con 0x7fb690004360 2024-03-18T20:25:27.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.993+0000 7fb69d7fa700 1 -- 172.21.15.1:0/619748214 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69400fe30 con 0x7fb690004360 2024-03-18T20:25:27.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.993+0000 7fb6a6e24700 1 -- 172.21.15.1:0/619748214 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a01f60f0 con 0x7fb690004360 2024-03-18T20:25:27.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.994+0000 7fb6a6e24700 1 -- 172.21.15.1:0/619748214 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a00c5150 con 0x7fb690004360 2024-03-18T20:25:27.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.994+0000 7fb69d7fa700 1 -- 172.21.15.1:0/619748214 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fb694014440 con 0x7fb690004360 2024-03-18T20:25:27.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.995+0000 7fb69d7fa700 1 --2- 172.21.15.1:0/619748214 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb68406adb0 0x7fb68406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:27.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.995+0000 7fb69d7fa700 1 -- 172.21.15.1:0/619748214 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fb694089c30 con 0x7fb690004360 2024-03-18T20:25:27.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.995+0000 7fb69ffff700 1 --2- 172.21.15.1:0/619748214 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb68406adb0 0x7fb68406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:27.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.995+0000 7fb69ffff700 1 --2- 172.21.15.1:0/619748214 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb68406adb0 0x7fb68406d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb68c008fd0 tx=0x7fb68c00a000 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:27.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:26.999+0000 7fb69d7fa700 1 -- 172.21.15.1:0/619748214 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb694054c90 con 0x7fb690004360 2024-03-18T20:25:27.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.198+0000 7fb6a6e24700 1 -- 172.21.15.1:0/619748214 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb6a00b3dd0 con 0x7fb68406adb0 2024-03-18T20:25:27.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.206+0000 7fb69d7fa700 1 -- 172.21.15.1:0/619748214 <== mgr.11322 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290223 (secure 0 0 0) 0x7fb68c005cf0 con 0x7fb68406adb0 2024-03-18T20:25:27.214 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:27.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.210+0000 7fb6a6e24700 1 -- 172.21.15.1:0/619748214 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb68406adb0 msgr2=0x7fb68406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:27.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.210+0000 7fb6a6e24700 1 --2- 172.21.15.1:0/619748214 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb68406adb0 0x7fb68406d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb68c008fd0 tx=0x7fb68c00a000 comp rx=0 tx=0).stop 2024-03-18T20:25:27.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.210+0000 7fb6a6e24700 1 -- 172.21.15.1:0/619748214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb690004360 msgr2=0x7fb6a0144a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:27.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.210+0000 7fb6a6e24700 1 --2- 172.21.15.1:0/619748214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb690004360 0x7fb6a0144a10 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7fb6940119a0 tx=0x7fb694011d60 comp rx=0 tx=0).stop 2024-03-18T20:25:27.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.210+0000 7fb6a6e24700 1 -- 172.21.15.1:0/619748214 shutdown_connections 2024-03-18T20:25:27.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.210+0000 7fb6a6e24700 1 --2- 172.21.15.1:0/619748214 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb68406adb0 0x7fb68406d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.210+0000 7fb6a6e24700 1 --2- 172.21.15.1:0/619748214 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb6a00cea00 0x7fb6a01e7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.210+0000 7fb6a6e24700 1 --2- 172.21.15.1:0/619748214 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb690004360 0x7fb6a0144a10 unknown :-1 s=CLOSED pgs=2525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.210+0000 7fb6a6e24700 1 --2- 172.21.15.1:0/619748214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6a01f2b20 0x7fb6a00ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.210+0000 7fb6a6e24700 1 -- 172.21.15.1:0/619748214 >> 172.21.15.1:0/619748214 conn(0x7fb6a00b1d20 msgr2=0x7fb6a00724b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:27.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.210+0000 7fb6a6e24700 1 -- 172.21.15.1:0/619748214 shutdown_connections 2024-03-18T20:25:27.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.210+0000 7fb6a6e24700 1 -- 172.21.15.1:0/619748214 wait complete. 2024-03-18T20:25:27.217 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:25:27.226 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-18T20:25:27.100321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":623,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":386,"ondisk_log_size":386,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":385296,"kb_used_data":23940,"kb_used_omap":12,"kb_used_meta":361203,"kb_avail":754589424,"statfs":{"total":773094113280,"available":772699570176,"internally_reserved":0,"allocated":24514560,"data_stored":13109628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":369872441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.875860"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.607056+0000","last_change":"2024-03-18T20:25:22.696207+0000","last_active":"2024-03-18T20:25:24.607056+0000","last_peered":"2024-03-18T20:25:24.607056+0000","last_clean":"2024-03-18T20:25:24.607056+0000","last_became_active":"2024-03-18T20:25:22.695943+0000","last_became_peered":"2024-03-18T20:25:22.695943+0000","last_unstale":"2024-03-18T20:25:24.607056+0000","last_undegraded":"2024-03-18T20:25:24.607056+0000","last_fullsized":"2024-03-18T20:25:24.607056+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:54:33.571448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605436+0000","last_change":"2024-03-18T20:25:22.682750+0000","last_active":"2024-03-18T20:25:24.605436+0000","last_peered":"2024-03-18T20:25:24.605436+0000","last_clean":"2024-03-18T20:25:24.605436+0000","last_became_active":"2024-03-18T20:25:22.682497+0000","last_became_peered":"2024-03-18T20:25:22.682497+0000","last_unstale":"2024-03-18T20:25:24.605436+0000","last_undegraded":"2024-03-18T20:25:24.605436+0000","last_fullsized":"2024-03-18T20:25:24.605436+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:33.092503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104328+0000","last_change":"2024-03-18T20:25:22.679426+0000","last_active":"2024-03-18T20:25:24.104328+0000","last_peered":"2024-03-18T20:25:24.104328+0000","last_clean":"2024-03-18T20:25:24.104328+0000","last_became_active":"2024-03-18T20:25:22.679159+0000","last_became_peered":"2024-03-18T20:25:22.679159+0000","last_unstale":"2024-03-18T20:25:24.104328+0000","last_undegraded":"2024-03-18T20:25:24.104328+0000","last_fullsized":"2024-03-18T20:25:24.104328+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:17:59.556803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605887+0000","last_change":"2024-03-18T20:25:22.683021+0000","last_active":"2024-03-18T20:25:24.605887+0000","last_peered":"2024-03-18T20:25:24.605887+0000","last_clean":"2024-03-18T20:25:24.605887+0000","last_became_active":"2024-03-18T20:25:22.682722+0000","last_became_peered":"2024-03-18T20:25:22.682722+0000","last_unstale":"2024-03-18T20:25:24.605887+0000","last_undegraded":"2024-03-18T20:25:24.605887+0000","last_fullsized":"2024-03-18T20:25:24.605887+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:27.207710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673930+0000","last_change":"2024-03-18T20:25:22.695525+0000","last_active":"2024-03-18T20:25:23.673930+0000","last_peered":"2024-03-18T20:25:23.673930+0000","last_clean":"2024-03-18T20:25:23.673930+0000","last_became_active":"2024-03-18T20:25:22.694079+0000","last_became_peered":"2024-03-18T20:25:22.694079+0000","last_unstale":"2024-03-18T20:25:23.673930+0000","last_undegraded":"2024-03-18T20:25:23.673930+0000","last_fullsized":"2024-03-18T20:25:23.673930+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:45:14.955608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605655+0000","last_change":"2024-03-18T20:25:22.681736+0000","last_active":"2024-03-18T20:25:24.605655+0000","last_peered":"2024-03-18T20:25:24.605655+0000","last_clean":"2024-03-18T20:25:24.605655+0000","last_became_active":"2024-03-18T20:25:22.679120+0000","last_became_peered":"2024-03-18T20:25:22.679120+0000","last_unstale":"2024-03-18T20:25:24.605655+0000","last_undegraded":"2024-03-18T20:25:24.605655+0000","last_fullsized":"2024-03-18T20:25:24.605655+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:00.365377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674711+0000","last_change":"2024-03-18T20:25:22.694269+0000","last_active":"2024-03-18T20:25:23.674711+0000","last_peered":"2024-03-18T20:25:23.674711+0000","last_clean":"2024-03-18T20:25:23.674711+0000","last_became_active":"2024-03-18T20:25:22.694014+0000","last_became_peered":"2024-03-18T20:25:22.694014+0000","last_unstale":"2024-03-18T20:25:23.674711+0000","last_undegraded":"2024-03-18T20:25:23.674711+0000","last_fullsized":"2024-03-18T20:25:23.674711+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:43:52.409254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.672828+0000","last_change":"2024-03-18T20:25:22.685766+0000","last_active":"2024-03-18T20:25:23.672828+0000","last_peered":"2024-03-18T20:25:23.672828+0000","last_clean":"2024-03-18T20:25:23.672828+0000","last_became_active":"2024-03-18T20:25:22.683799+0000","last_became_peered":"2024-03-18T20:25:22.683799+0000","last_unstale":"2024-03-18T20:25:23.672828+0000","last_undegraded":"2024-03-18T20:25:23.672828+0000","last_fullsized":"2024-03-18T20:25:23.672828+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:42:25.581359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674361+0000","last_change":"2024-03-18T20:25:22.681784+0000","last_active":"2024-03-18T20:25:23.674361+0000","last_peered":"2024-03-18T20:25:23.674361+0000","last_clean":"2024-03-18T20:25:23.674361+0000","last_became_active":"2024-03-18T20:25:22.681198+0000","last_became_peered":"2024-03-18T20:25:22.681198+0000","last_unstale":"2024-03-18T20:25:23.674361+0000","last_undegraded":"2024-03-18T20:25:23.674361+0000","last_fullsized":"2024-03-18T20:25:23.674361+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:35.307035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673444+0000","last_change":"2024-03-18T20:25:22.693152+0000","last_active":"2024-03-18T20:25:23.673444+0000","last_peered":"2024-03-18T20:25:23.673444+0000","last_clean":"2024-03-18T20:25:23.673444+0000","last_became_active":"2024-03-18T20:25:22.686521+0000","last_became_peered":"2024-03-18T20:25:22.686521+0000","last_unstale":"2024-03-18T20:25:23.673444+0000","last_undegraded":"2024-03-18T20:25:23.673444+0000","last_fullsized":"2024-03-18T20:25:23.673444+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:40:18.302530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104395+0000","last_change":"2024-03-18T20:25:22.693546+0000","last_active":"2024-03-18T20:25:24.104395+0000","last_peered":"2024-03-18T20:25:24.104395+0000","last_clean":"2024-03-18T20:25:24.104395+0000","last_became_active":"2024-03-18T20:25:22.693267+0000","last_became_peered":"2024-03-18T20:25:22.693267+0000","last_unstale":"2024-03-18T20:25:24.104395+0000","last_undegraded":"2024-03-18T20:25:24.104395+0000","last_fullsized":"2024-03-18T20:25:24.104395+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:35.486064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674346+0000","last_change":"2024-03-18T20:25:22.693640+0000","last_active":"2024-03-18T20:25:23.674346+0000","last_peered":"2024-03-18T20:25:23.674346+0000","last_clean":"2024-03-18T20:25:23.674346+0000","last_became_active":"2024-03-18T20:25:22.693403+0000","last_became_peered":"2024-03-18T20:25:22.693403+0000","last_unstale":"2024-03-18T20:25:23.674346+0000","last_undegraded":"2024-03-18T20:25:23.674346+0000","last_fullsized":"2024-03-18T20:25:23.674346+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:58.151146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605662+0000","last_change":"2024-03-18T20:25:22.685172+0000","last_active":"2024-03-18T20:25:24.605662+0000","last_peered":"2024-03-18T20:25:24.605662+0000","last_clean":"2024-03-18T20:25:24.605662+0000","last_became_active":"2024-03-18T20:25:22.684855+0000","last_became_peered":"2024-03-18T20:25:22.684855+0000","last_unstale":"2024-03-18T20:25:24.605662+0000","last_undegraded":"2024-03-18T20:25:24.605662+0000","last_fullsized":"2024-03-18T20:25:24.605662+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:00.030804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.672936+0000","last_change":"2024-03-18T20:25:22.690898+0000","last_active":"2024-03-18T20:25:23.672936+0000","last_peered":"2024-03-18T20:25:23.672936+0000","last_clean":"2024-03-18T20:25:23.672936+0000","last_became_active":"2024-03-18T20:25:22.690461+0000","last_became_peered":"2024-03-18T20:25:22.690461+0000","last_unstale":"2024-03-18T20:25:23.672936+0000","last_undegraded":"2024-03-18T20:25:23.672936+0000","last_fullsized":"2024-03-18T20:25:23.672936+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:15.208122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605918+0000","last_change":"2024-03-18T20:25:22.692497+0000","last_active":"2024-03-18T20:25:24.605918+0000","last_peered":"2024-03-18T20:25:24.605918+0000","last_clean":"2024-03-18T20:25:24.605918+0000","last_became_active":"2024-03-18T20:25:22.692192+0000","last_became_peered":"2024-03-18T20:25:22.692192+0000","last_unstale":"2024-03-18T20:25:24.605918+0000","last_undegraded":"2024-03-18T20:25:24.605918+0000","last_fullsized":"2024-03-18T20:25:24.605918+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:03:47.775258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104291+0000","last_change":"2024-03-18T20:25:22.681389+0000","last_active":"2024-03-18T20:25:24.104291+0000","last_peered":"2024-03-18T20:25:24.104291+0000","last_clean":"2024-03-18T20:25:24.104291+0000","last_became_active":"2024-03-18T20:25:22.681146+0000","last_became_peered":"2024-03-18T20:25:22.681146+0000","last_unstale":"2024-03-18T20:25:24.104291+0000","last_undegraded":"2024-03-18T20:25:24.104291+0000","last_fullsized":"2024-03-18T20:25:24.104291+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:05:57.372944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674448+0000","last_change":"2024-03-18T20:25:22.696148+0000","last_active":"2024-03-18T20:25:23.674448+0000","last_peered":"2024-03-18T20:25:23.674448+0000","last_clean":"2024-03-18T20:25:23.674448+0000","last_became_active":"2024-03-18T20:25:22.695928+0000","last_became_peered":"2024-03-18T20:25:22.695928+0000","last_unstale":"2024-03-18T20:25:23.674448+0000","last_undegraded":"2024-03-18T20:25:23.674448+0000","last_fullsized":"2024-03-18T20:25:23.674448+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:57.565171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674177+0000","last_change":"2024-03-18T20:25:22.683388+0000","last_active":"2024-03-18T20:25:23.674177+0000","last_peered":"2024-03-18T20:25:23.674177+0000","last_clean":"2024-03-18T20:25:23.674177+0000","last_became_active":"2024-03-18T20:25:22.683137+0000","last_became_peered":"2024-03-18T20:25:22.683137+0000","last_unstale":"2024-03-18T20:25:23.674177+0000","last_undegraded":"2024-03-18T20:25:23.674177+0000","last_fullsized":"2024-03-18T20:25:23.674177+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:02.506299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673199+0000","last_change":"2024-03-18T20:25:22.692857+0000","last_active":"2024-03-18T20:25:23.673199+0000","last_peered":"2024-03-18T20:25:23.673199+0000","last_clean":"2024-03-18T20:25:23.673199+0000","last_became_active":"2024-03-18T20:25:22.692615+0000","last_became_peered":"2024-03-18T20:25:22.692615+0000","last_unstale":"2024-03-18T20:25:23.673199+0000","last_undegraded":"2024-03-18T20:25:23.673199+0000","last_fullsized":"2024-03-18T20:25:23.673199+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:21:45.938034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673244+0000","last_change":"2024-03-18T20:25:22.693769+0000","last_active":"2024-03-18T20:25:23.673244+0000","last_peered":"2024-03-18T20:25:23.673244+0000","last_clean":"2024-03-18T20:25:23.673244+0000","last_became_active":"2024-03-18T20:25:22.691936+0000","last_became_peered":"2024-03-18T20:25:22.691936+0000","last_unstale":"2024-03-18T20:25:23.673244+0000","last_undegraded":"2024-03-18T20:25:23.673244+0000","last_fullsized":"2024-03-18T20:25:23.673244+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:29:49.401587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673379+0000","last_change":"2024-03-18T20:25:22.693670+0000","last_active":"2024-03-18T20:25:23.673379+0000","last_peered":"2024-03-18T20:25:23.673379+0000","last_clean":"2024-03-18T20:25:23.673379+0000","last_became_active":"2024-03-18T20:25:22.693178+0000","last_became_peered":"2024-03-18T20:25:22.693178+0000","last_unstale":"2024-03-18T20:25:23.673379+0000","last_undegraded":"2024-03-18T20:25:23.673379+0000","last_fullsized":"2024-03-18T20:25:23.673379+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:13.992732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673104+0000","last_change":"2024-03-18T20:25:22.691577+0000","last_active":"2024-03-18T20:25:23.673104+0000","last_peered":"2024-03-18T20:25:23.673104+0000","last_clean":"2024-03-18T20:25:23.673104+0000","last_became_active":"2024-03-18T20:25:22.691250+0000","last_became_peered":"2024-03-18T20:25:22.691250+0000","last_unstale":"2024-03-18T20:25:23.673104+0000","last_undegraded":"2024-03-18T20:25:23.673104+0000","last_fullsized":"2024-03-18T20:25:23.673104+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:55:21.946848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605193+0000","last_change":"2024-03-18T20:25:22.680418+0000","last_active":"2024-03-18T20:25:24.605193+0000","last_peered":"2024-03-18T20:25:24.605193+0000","last_clean":"2024-03-18T20:25:24.605193+0000","last_became_active":"2024-03-18T20:25:22.680149+0000","last_became_peered":"2024-03-18T20:25:22.680149+0000","last_unstale":"2024-03-18T20:25:24.605193+0000","last_undegraded":"2024-03-18T20:25:24.605193+0000","last_fullsized":"2024-03-18T20:25:24.605193+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:51:14.804124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673769+0000","last_change":"2024-03-18T20:25:22.684030+0000","last_active":"2024-03-18T20:25:23.673769+0000","last_peered":"2024-03-18T20:25:23.673769+0000","last_clean":"2024-03-18T20:25:23.673769+0000","last_became_active":"2024-03-18T20:25:22.683767+0000","last_became_peered":"2024-03-18T20:25:22.683767+0000","last_unstale":"2024-03-18T20:25:23.673769+0000","last_undegraded":"2024-03-18T20:25:23.673769+0000","last_fullsized":"2024-03-18T20:25:23.673769+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:30.204046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674534+0000","last_change":"2024-03-18T20:25:22.683777+0000","last_active":"2024-03-18T20:25:23.674534+0000","last_peered":"2024-03-18T20:25:23.674534+0000","last_clean":"2024-03-18T20:25:23.674534+0000","last_became_active":"2024-03-18T20:25:22.683484+0000","last_became_peered":"2024-03-18T20:25:22.683484+0000","last_unstale":"2024-03-18T20:25:23.674534+0000","last_undegraded":"2024-03-18T20:25:23.674534+0000","last_fullsized":"2024-03-18T20:25:23.674534+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:03:45.332668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.606865+0000","last_change":"2024-03-18T20:25:22.687342+0000","last_active":"2024-03-18T20:25:24.606865+0000","last_peered":"2024-03-18T20:25:24.606865+0000","last_clean":"2024-03-18T20:25:24.606865+0000","last_became_active":"2024-03-18T20:25:22.687092+0000","last_became_peered":"2024-03-18T20:25:22.687092+0000","last_unstale":"2024-03-18T20:25:24.606865+0000","last_undegraded":"2024-03-18T20:25:24.606865+0000","last_fullsized":"2024-03-18T20:25:24.606865+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:00:33.383893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104453+0000","last_change":"2024-03-18T20:25:22.689262+0000","last_active":"2024-03-18T20:25:24.104453+0000","last_peered":"2024-03-18T20:25:24.104453+0000","last_clean":"2024-03-18T20:25:24.104453+0000","last_became_active":"2024-03-18T20:25:22.688913+0000","last_became_peered":"2024-03-18T20:25:22.688913+0000","last_unstale":"2024-03-18T20:25:24.104453+0000","last_undegraded":"2024-03-18T20:25:24.104453+0000","last_fullsized":"2024-03-18T20:25:24.104453+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:15:50.135885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674346+0000","last_change":"2024-03-18T20:25:22.688098+0000","last_active":"2024-03-18T20:25:23.674346+0000","last_peered":"2024-03-18T20:25:23.674346+0000","last_clean":"2024-03-18T20:25:23.674346+0000","last_became_active":"2024-03-18T20:25:22.687830+0000","last_became_peered":"2024-03-18T20:25:22.687830+0000","last_unstale":"2024-03-18T20:25:23.674346+0000","last_undegraded":"2024-03-18T20:25:23.674346+0000","last_fullsized":"2024-03-18T20:25:23.674346+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:08.456234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673944+0000","last_change":"2024-03-18T20:25:22.680503+0000","last_active":"2024-03-18T20:25:23.673944+0000","last_peered":"2024-03-18T20:25:23.673944+0000","last_clean":"2024-03-18T20:25:23.673944+0000","last_became_active":"2024-03-18T20:25:22.680258+0000","last_became_peered":"2024-03-18T20:25:22.680258+0000","last_unstale":"2024-03-18T20:25:23.673944+0000","last_undegraded":"2024-03-18T20:25:23.673944+0000","last_fullsized":"2024-03-18T20:25:23.673944+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:49.478525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605579+0000","last_change":"2024-03-18T20:25:22.685981+0000","last_active":"2024-03-18T20:25:24.605579+0000","last_peered":"2024-03-18T20:25:24.605579+0000","last_clean":"2024-03-18T20:25:24.605579+0000","last_became_active":"2024-03-18T20:25:22.685683+0000","last_became_peered":"2024-03-18T20:25:22.685683+0000","last_unstale":"2024-03-18T20:25:24.605579+0000","last_undegraded":"2024-03-18T20:25:24.605579+0000","last_fullsized":"2024-03-18T20:25:24.605579+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:36:57.458670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605466+0000","last_change":"2024-03-18T20:25:22.679640+0000","last_active":"2024-03-18T20:25:24.605466+0000","last_peered":"2024-03-18T20:25:24.605466+0000","last_clean":"2024-03-18T20:25:24.605466+0000","last_became_active":"2024-03-18T20:25:22.679408+0000","last_became_peered":"2024-03-18T20:25:22.679408+0000","last_unstale":"2024-03-18T20:25:24.605466+0000","last_undegraded":"2024-03-18T20:25:24.605466+0000","last_fullsized":"2024-03-18T20:25:24.605466+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:24:33.293594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605407+0000","last_change":"2024-03-18T20:25:22.679227+0000","last_active":"2024-03-18T20:25:24.605407+0000","last_peered":"2024-03-18T20:25:24.605407+0000","last_clean":"2024-03-18T20:25:24.605407+0000","last_became_active":"2024-03-18T20:25:22.678827+0000","last_became_peered":"2024-03-18T20:25:22.678827+0000","last_unstale":"2024-03-18T20:25:24.605407+0000","last_undegraded":"2024-03-18T20:25:24.605407+0000","last_fullsized":"2024-03-18T20:25:24.605407+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:05.394457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605152+0000","last_change":"2024-03-18T20:25:22.682272+0000","last_active":"2024-03-18T20:25:24.605152+0000","last_peered":"2024-03-18T20:25:24.605152+0000","last_clean":"2024-03-18T20:25:24.605152+0000","last_became_active":"2024-03-18T20:25:22.681785+0000","last_became_peered":"2024-03-18T20:25:22.681785+0000","last_unstale":"2024-03-18T20:25:24.605152+0000","last_undegraded":"2024-03-18T20:25:24.605152+0000","last_fullsized":"2024-03-18T20:25:24.605152+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:53:46.174877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"231'4","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.606489+0000","last_change":"2024-03-18T20:25:22.251710+0000","last_active":"2024-03-18T20:25:24.606489+0000","last_peered":"2024-03-18T20:25:24.606489+0000","last_clean":"2024-03-18T20:25:24.606489+0000","last_became_active":"2024-03-18T20:25:22.251420+0000","last_became_peered":"2024-03-18T20:25:22.251420+0000","last_unstale":"2024-03-18T20:25:24.606489+0000","last_undegraded":"2024-03-18T20:25:24.606489+0000","last_fullsized":"2024-03-18T20:25:24.606489+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:00.435897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.606655+0000","last_change":"2024-03-18T20:25:22.679158+0000","last_active":"2024-03-18T20:25:24.606655+0000","last_peered":"2024-03-18T20:25:24.606655+0000","last_clean":"2024-03-18T20:25:24.606655+0000","last_became_active":"2024-03-18T20:25:22.678889+0000","last_became_peered":"2024-03-18T20:25:22.678889+0000","last_unstale":"2024-03-18T20:25:24.606655+0000","last_undegraded":"2024-03-18T20:25:24.606655+0000","last_fullsized":"2024-03-18T20:25:24.606655+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:00.365377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"231'1","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.606943+0000","last_change":"2024-03-18T20:25:22.251725+0000","last_active":"2024-03-18T20:25:24.606943+0000","last_peered":"2024-03-18T20:25:24.606943+0000","last_clean":"2024-03-18T20:25:24.606943+0000","last_became_active":"2024-03-18T20:25:22.251416+0000","last_became_peered":"2024-03-18T20:25:22.251416+0000","last_unstale":"2024-03-18T20:25:24.606943+0000","last_undegraded":"2024-03-18T20:25:24.606943+0000","last_fullsized":"2024-03-18T20:25:24.606943+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:26.827223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673294+0000","last_change":"2024-03-18T20:25:22.691295+0000","last_active":"2024-03-18T20:25:23.673294+0000","last_peered":"2024-03-18T20:25:23.673294+0000","last_clean":"2024-03-18T20:25:23.673294+0000","last_became_active":"2024-03-18T20:25:22.690890+0000","last_became_peered":"2024-03-18T20:25:22.690890+0000","last_unstale":"2024-03-18T20:25:23.673294+0000","last_undegraded":"2024-03-18T20:25:23.673294+0000","last_fullsized":"2024-03-18T20:25:23.673294+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:23:16.320044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"231'2","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605362+0000","last_change":"2024-03-18T20:25:22.255583+0000","last_active":"2024-03-18T20:25:24.605362+0000","last_peered":"2024-03-18T20:25:24.605362+0000","last_clean":"2024-03-18T20:25:24.605362+0000","last_became_active":"2024-03-18T20:25:22.255243+0000","last_became_peered":"2024-03-18T20:25:22.255243+0000","last_unstale":"2024-03-18T20:25:24.605362+0000","last_undegraded":"2024-03-18T20:25:24.605362+0000","last_fullsized":"2024-03-18T20:25:24.605362+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:38.084389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674021+0000","last_change":"2024-03-18T20:25:22.690894+0000","last_active":"2024-03-18T20:25:23.674021+0000","last_peered":"2024-03-18T20:25:23.674021+0000","last_clean":"2024-03-18T20:25:23.674021+0000","last_became_active":"2024-03-18T20:25:22.688184+0000","last_became_peered":"2024-03-18T20:25:22.688184+0000","last_unstale":"2024-03-18T20:25:23.674021+0000","last_undegraded":"2024-03-18T20:25:23.674021+0000","last_fullsized":"2024-03-18T20:25:23.674021+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:59.967502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"231'2","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674153+0000","last_change":"2024-03-18T20:25:22.251018+0000","last_active":"2024-03-18T20:25:23.674153+0000","last_peered":"2024-03-18T20:25:23.674153+0000","last_clean":"2024-03-18T20:25:23.674153+0000","last_became_active":"2024-03-18T20:25:22.250730+0000","last_became_peered":"2024-03-18T20:25:22.250730+0000","last_unstale":"2024-03-18T20:25:23.674153+0000","last_undegraded":"2024-03-18T20:25:23.674153+0000","last_fullsized":"2024-03-18T20:25:23.674153+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:07.579514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605702+0000","last_change":"2024-03-18T20:25:22.680768+0000","last_active":"2024-03-18T20:25:24.605702+0000","last_peered":"2024-03-18T20:25:24.605702+0000","last_clean":"2024-03-18T20:25:24.605702+0000","last_became_active":"2024-03-18T20:25:22.680516+0000","last_became_peered":"2024-03-18T20:25:22.680516+0000","last_unstale":"2024-03-18T20:25:24.605702+0000","last_undegraded":"2024-03-18T20:25:24.605702+0000","last_fullsized":"2024-03-18T20:25:24.605702+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:19:19.377686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"231'1","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605776+0000","last_change":"2024-03-18T20:25:22.251076+0000","last_active":"2024-03-18T20:25:24.605776+0000","last_peered":"2024-03-18T20:25:24.605776+0000","last_clean":"2024-03-18T20:25:24.605776+0000","last_became_active":"2024-03-18T20:25:22.250741+0000","last_became_peered":"2024-03-18T20:25:22.250741+0000","last_unstale":"2024-03-18T20:25:24.605776+0000","last_undegraded":"2024-03-18T20:25:24.605776+0000","last_fullsized":"2024-03-18T20:25:24.605776+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:07.448931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.606082+0000","last_change":"2024-03-18T20:25:22.691744+0000","last_active":"2024-03-18T20:25:24.606082+0000","last_peered":"2024-03-18T20:25:24.606082+0000","last_clean":"2024-03-18T20:25:24.606082+0000","last_became_active":"2024-03-18T20:25:22.691491+0000","last_became_peered":"2024-03-18T20:25:22.691491+0000","last_unstale":"2024-03-18T20:25:24.606082+0000","last_undegraded":"2024-03-18T20:25:24.606082+0000","last_fullsized":"2024-03-18T20:25:24.606082+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:04:09.670541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605566+0000","last_change":"2024-03-18T20:25:22.251522+0000","last_active":"2024-03-18T20:25:24.605566+0000","last_peered":"2024-03-18T20:25:24.605566+0000","last_clean":"2024-03-18T20:25:24.605566+0000","last_became_active":"2024-03-18T20:25:22.251143+0000","last_became_peered":"2024-03-18T20:25:22.251143+0000","last_unstale":"2024-03-18T20:25:24.605566+0000","last_undegraded":"2024-03-18T20:25:24.605566+0000","last_fullsized":"2024-03-18T20:25:24.605566+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:58.960198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674625+0000","last_change":"2024-03-18T20:25:22.681952+0000","last_active":"2024-03-18T20:25:23.674625+0000","last_peered":"2024-03-18T20:25:23.674625+0000","last_clean":"2024-03-18T20:25:23.674625+0000","last_became_active":"2024-03-18T20:25:22.681537+0000","last_became_peered":"2024-03-18T20:25:22.681537+0000","last_unstale":"2024-03-18T20:25:23.674625+0000","last_undegraded":"2024-03-18T20:25:23.674625+0000","last_fullsized":"2024-03-18T20:25:23.674625+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:32:22.923763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104245+0000","last_change":"2024-03-18T20:25:22.244727+0000","last_active":"2024-03-18T20:25:24.104245+0000","last_peered":"2024-03-18T20:25:24.104245+0000","last_clean":"2024-03-18T20:25:24.104245+0000","last_became_active":"2024-03-18T20:25:22.244417+0000","last_became_peered":"2024-03-18T20:25:22.244417+0000","last_unstale":"2024-03-18T20:25:24.104245+0000","last_undegraded":"2024-03-18T20:25:24.104245+0000","last_fullsized":"2024-03-18T20:25:24.104245+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:10.187509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605544+0000","last_change":"2024-03-18T20:25:22.680660+0000","last_active":"2024-03-18T20:25:24.605544+0000","last_peered":"2024-03-18T20:25:24.605544+0000","last_clean":"2024-03-18T20:25:24.605544+0000","last_became_active":"2024-03-18T20:25:22.680484+0000","last_became_peered":"2024-03-18T20:25:22.680484+0000","last_unstale":"2024-03-18T20:25:24.605544+0000","last_undegraded":"2024-03-18T20:25:24.605544+0000","last_fullsized":"2024-03-18T20:25:24.605544+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:48:21.194536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674487+0000","last_change":"2024-03-18T20:25:22.251887+0000","last_active":"2024-03-18T20:25:23.674487+0000","last_peered":"2024-03-18T20:25:23.674487+0000","last_clean":"2024-03-18T20:25:23.674487+0000","last_became_active":"2024-03-18T20:25:22.251594+0000","last_became_peered":"2024-03-18T20:25:22.251594+0000","last_unstale":"2024-03-18T20:25:23.674487+0000","last_undegraded":"2024-03-18T20:25:23.674487+0000","last_fullsized":"2024-03-18T20:25:23.674487+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:48:47.954568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104215+0000","last_change":"2024-03-18T20:25:22.684633+0000","last_active":"2024-03-18T20:25:24.104215+0000","last_peered":"2024-03-18T20:25:24.104215+0000","last_clean":"2024-03-18T20:25:24.104215+0000","last_became_active":"2024-03-18T20:25:22.684366+0000","last_became_peered":"2024-03-18T20:25:22.684366+0000","last_unstale":"2024-03-18T20:25:24.104215+0000","last_undegraded":"2024-03-18T20:25:24.104215+0000","last_fullsized":"2024-03-18T20:25:24.104215+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:15.299142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.606128+0000","last_change":"2024-03-18T20:25:22.252980+0000","last_active":"2024-03-18T20:25:24.606128+0000","last_peered":"2024-03-18T20:25:24.606128+0000","last_clean":"2024-03-18T20:25:24.606128+0000","last_became_active":"2024-03-18T20:25:22.252706+0000","last_became_peered":"2024-03-18T20:25:22.252706+0000","last_unstale":"2024-03-18T20:25:24.606128+0000","last_undegraded":"2024-03-18T20:25:24.606128+0000","last_fullsized":"2024-03-18T20:25:24.606128+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:45:13.948304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673856+0000","last_change":"2024-03-18T20:25:22.682861+0000","last_active":"2024-03-18T20:25:23.673856+0000","last_peered":"2024-03-18T20:25:23.673856+0000","last_clean":"2024-03-18T20:25:23.673856+0000","last_became_active":"2024-03-18T20:25:22.682610+0000","last_became_peered":"2024-03-18T20:25:22.682610+0000","last_unstale":"2024-03-18T20:25:23.673856+0000","last_undegraded":"2024-03-18T20:25:23.673856+0000","last_fullsized":"2024-03-18T20:25:23.673856+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:51:43.095770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"231'3","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674862+0000","last_change":"2024-03-18T20:25:22.251534+0000","last_active":"2024-03-18T20:25:23.674862+0000","last_peered":"2024-03-18T20:25:23.674862+0000","last_clean":"2024-03-18T20:25:23.674862+0000","last_became_active":"2024-03-18T20:25:22.251275+0000","last_became_peered":"2024-03-18T20:25:22.251275+0000","last_unstale":"2024-03-18T20:25:23.674862+0000","last_undegraded":"2024-03-18T20:25:23.674862+0000","last_fullsized":"2024-03-18T20:25:23.674862+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:38:51.553676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674116+0000","last_change":"2024-03-18T20:25:22.696408+0000","last_active":"2024-03-18T20:25:23.674116+0000","last_peered":"2024-03-18T20:25:23.674116+0000","last_clean":"2024-03-18T20:25:23.674116+0000","last_became_active":"2024-03-18T20:25:22.696095+0000","last_became_peered":"2024-03-18T20:25:22.696095+0000","last_unstale":"2024-03-18T20:25:23.674116+0000","last_undegraded":"2024-03-18T20:25:23.674116+0000","last_fullsized":"2024-03-18T20:25:23.674116+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:19:19.377686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"231'6","reported_seq":17,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605999+0000","last_change":"2024-03-18T20:25:22.249429+0000","last_active":"2024-03-18T20:25:24.605999+0000","last_peered":"2024-03-18T20:25:24.605999+0000","last_clean":"2024-03-18T20:25:24.605999+0000","last_became_active":"2024-03-18T20:25:22.248842+0000","last_became_peered":"2024-03-18T20:25:22.248842+0000","last_unstale":"2024-03-18T20:25:24.605999+0000","last_undegraded":"2024-03-18T20:25:24.605999+0000","last_fullsized":"2024-03-18T20:25:24.605999+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:26.827223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605338+0000","last_change":"2024-03-18T20:25:22.691203+0000","last_active":"2024-03-18T20:25:24.605338+0000","last_peered":"2024-03-18T20:25:24.605338+0000","last_clean":"2024-03-18T20:25:24.605338+0000","last_became_active":"2024-03-18T20:25:22.690964+0000","last_became_peered":"2024-03-18T20:25:22.690964+0000","last_unstale":"2024-03-18T20:25:24.605338+0000","last_undegraded":"2024-03-18T20:25:24.605338+0000","last_fullsized":"2024-03-18T20:25:24.605338+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:19.068003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"231'2","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605301+0000","last_change":"2024-03-18T20:25:22.254668+0000","last_active":"2024-03-18T20:25:24.605301+0000","last_peered":"2024-03-18T20:25:24.605301+0000","last_clean":"2024-03-18T20:25:24.605301+0000","last_became_active":"2024-03-18T20:25:22.254437+0000","last_became_peered":"2024-03-18T20:25:22.254437+0000","last_unstale":"2024-03-18T20:25:24.605301+0000","last_undegraded":"2024-03-18T20:25:24.605301+0000","last_fullsized":"2024-03-18T20:25:24.605301+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:32:02.691847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104504+0000","last_change":"2024-03-18T20:25:22.685090+0000","last_active":"2024-03-18T20:25:24.104504+0000","last_peered":"2024-03-18T20:25:24.104504+0000","last_clean":"2024-03-18T20:25:24.104504+0000","last_became_active":"2024-03-18T20:25:22.684824+0000","last_became_peered":"2024-03-18T20:25:22.684824+0000","last_unstale":"2024-03-18T20:25:24.104504+0000","last_undegraded":"2024-03-18T20:25:24.104504+0000","last_fullsized":"2024-03-18T20:25:24.104504+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:21.750701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104292+0000","last_change":"2024-03-18T20:25:22.249078+0000","last_active":"2024-03-18T20:25:24.104292+0000","last_peered":"2024-03-18T20:25:24.104292+0000","last_clean":"2024-03-18T20:25:24.104292+0000","last_became_active":"2024-03-18T20:25:22.248588+0000","last_became_peered":"2024-03-18T20:25:22.248588+0000","last_unstale":"2024-03-18T20:25:24.104292+0000","last_undegraded":"2024-03-18T20:25:24.104292+0000","last_fullsized":"2024-03-18T20:25:24.104292+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:34:42.893253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674267+0000","last_change":"2024-03-18T20:25:22.694369+0000","last_active":"2024-03-18T20:25:23.674267+0000","last_peered":"2024-03-18T20:25:23.674267+0000","last_clean":"2024-03-18T20:25:23.674267+0000","last_became_active":"2024-03-18T20:25:22.694114+0000","last_became_peered":"2024-03-18T20:25:22.694114+0000","last_unstale":"2024-03-18T20:25:23.674267+0000","last_undegraded":"2024-03-18T20:25:23.674267+0000","last_fullsized":"2024-03-18T20:25:23.674267+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:35:13.631181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673880+0000","last_change":"2024-03-18T20:25:22.252927+0000","last_active":"2024-03-18T20:25:23.673880+0000","last_peered":"2024-03-18T20:25:23.673880+0000","last_clean":"2024-03-18T20:25:23.673880+0000","last_became_active":"2024-03-18T20:25:22.252633+0000","last_became_peered":"2024-03-18T20:25:22.252633+0000","last_unstale":"2024-03-18T20:25:23.673880+0000","last_undegraded":"2024-03-18T20:25:23.673880+0000","last_fullsized":"2024-03-18T20:25:23.673880+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:32.345526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605687+0000","last_change":"2024-03-18T20:25:22.673559+0000","last_active":"2024-03-18T20:25:24.605687+0000","last_peered":"2024-03-18T20:25:24.605687+0000","last_clean":"2024-03-18T20:25:24.605687+0000","last_became_active":"2024-03-18T20:25:22.673267+0000","last_became_peered":"2024-03-18T20:25:22.673267+0000","last_unstale":"2024-03-18T20:25:24.605687+0000","last_undegraded":"2024-03-18T20:25:24.605687+0000","last_fullsized":"2024-03-18T20:25:24.605687+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:44:15.325153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"231'8","reported_seq":19,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673866+0000","last_change":"2024-03-18T20:25:22.252976+0000","last_active":"2024-03-18T20:25:23.673866+0000","last_peered":"2024-03-18T20:25:23.673866+0000","last_clean":"2024-03-18T20:25:23.673866+0000","last_became_active":"2024-03-18T20:25:22.252528+0000","last_became_peered":"2024-03-18T20:25:22.252528+0000","last_unstale":"2024-03-18T20:25:23.673866+0000","last_undegraded":"2024-03-18T20:25:23.673866+0000","last_fullsized":"2024-03-18T20:25:23.673866+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:42:24.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.672855+0000","last_change":"2024-03-18T20:25:22.686185+0000","last_active":"2024-03-18T20:25:23.672855+0000","last_peered":"2024-03-18T20:25:23.672855+0000","last_clean":"2024-03-18T20:25:23.672855+0000","last_became_active":"2024-03-18T20:25:22.685939+0000","last_became_peered":"2024-03-18T20:25:22.685939+0000","last_unstale":"2024-03-18T20:25:23.672855+0000","last_undegraded":"2024-03-18T20:25:23.672855+0000","last_fullsized":"2024-03-18T20:25:23.672855+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:48:48.961872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"231'1","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.607061+0000","last_change":"2024-03-18T20:25:22.255372+0000","last_active":"2024-03-18T20:25:24.607061+0000","last_peered":"2024-03-18T20:25:24.607061+0000","last_clean":"2024-03-18T20:25:24.607061+0000","last_became_active":"2024-03-18T20:25:22.255063+0000","last_became_peered":"2024-03-18T20:25:22.255063+0000","last_unstale":"2024-03-18T20:25:24.607061+0000","last_undegraded":"2024-03-18T20:25:24.607061+0000","last_fullsized":"2024-03-18T20:25:24.607061+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:23.143345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.606505+0000","last_change":"2024-03-18T20:25:22.687892+0000","last_active":"2024-03-18T20:25:24.606505+0000","last_peered":"2024-03-18T20:25:24.606505+0000","last_clean":"2024-03-18T20:25:24.606505+0000","last_became_active":"2024-03-18T20:25:22.687086+0000","last_became_peered":"2024-03-18T20:25:22.687086+0000","last_unstale":"2024-03-18T20:25:24.606505+0000","last_undegraded":"2024-03-18T20:25:24.606505+0000","last_fullsized":"2024-03-18T20:25:24.606505+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:15:29.461741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"231'2","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.606992+0000","last_change":"2024-03-18T20:25:22.253997+0000","last_active":"2024-03-18T20:25:24.606992+0000","last_peered":"2024-03-18T20:25:24.606992+0000","last_clean":"2024-03-18T20:25:24.606992+0000","last_became_active":"2024-03-18T20:25:22.253649+0000","last_became_peered":"2024-03-18T20:25:22.253649+0000","last_unstale":"2024-03-18T20:25:24.606992+0000","last_undegraded":"2024-03-18T20:25:24.606992+0000","last_fullsized":"2024-03-18T20:25:24.606992+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:00.435897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.607019+0000","last_change":"2024-03-18T20:25:22.684910+0000","last_active":"2024-03-18T20:25:24.607019+0000","last_peered":"2024-03-18T20:25:24.607019+0000","last_clean":"2024-03-18T20:25:24.607019+0000","last_became_active":"2024-03-18T20:25:22.684472+0000","last_became_peered":"2024-03-18T20:25:22.684472+0000","last_unstale":"2024-03-18T20:25:24.607019+0000","last_undegraded":"2024-03-18T20:25:24.607019+0000","last_fullsized":"2024-03-18T20:25:24.607019+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:24:33.293594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"231'1","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605461+0000","last_change":"2024-03-18T20:25:22.248228+0000","last_active":"2024-03-18T20:25:24.605461+0000","last_peered":"2024-03-18T20:25:24.605461+0000","last_clean":"2024-03-18T20:25:24.605461+0000","last_became_active":"2024-03-18T20:25:22.247916+0000","last_became_peered":"2024-03-18T20:25:22.247916+0000","last_unstale":"2024-03-18T20:25:24.605461+0000","last_undegraded":"2024-03-18T20:25:24.605461+0000","last_fullsized":"2024-03-18T20:25:24.605461+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:56.236862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673112+0000","last_change":"2024-03-18T20:25:22.692895+0000","last_active":"2024-03-18T20:25:23.673112+0000","last_peered":"2024-03-18T20:25:23.673112+0000","last_clean":"2024-03-18T20:25:23.673112+0000","last_became_active":"2024-03-18T20:25:22.685766+0000","last_became_peered":"2024-03-18T20:25:22.685766+0000","last_unstale":"2024-03-18T20:25:23.673112+0000","last_undegraded":"2024-03-18T20:25:23.673112+0000","last_fullsized":"2024-03-18T20:25:23.673112+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:16:08.020891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674418+0000","last_change":"2024-03-18T20:25:22.254305+0000","last_active":"2024-03-18T20:25:23.674418+0000","last_peered":"2024-03-18T20:25:23.674418+0000","last_clean":"2024-03-18T20:25:23.674418+0000","last_became_active":"2024-03-18T20:25:22.253898+0000","last_became_peered":"2024-03-18T20:25:22.253898+0000","last_unstale":"2024-03-18T20:25:23.674418+0000","last_undegraded":"2024-03-18T20:25:23.674418+0000","last_fullsized":"2024-03-18T20:25:23.674418+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:29:48.394283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605376+0000","last_change":"2024-03-18T20:25:22.687129+0000","last_active":"2024-03-18T20:25:24.605376+0000","last_peered":"2024-03-18T20:25:24.605376+0000","last_clean":"2024-03-18T20:25:24.605376+0000","last_became_active":"2024-03-18T20:25:22.686871+0000","last_became_peered":"2024-03-18T20:25:22.686871+0000","last_unstale":"2024-03-18T20:25:24.605376+0000","last_undegraded":"2024-03-18T20:25:24.605376+0000","last_fullsized":"2024-03-18T20:25:24.605376+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:18:58.196895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"231'2","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674205+0000","last_change":"2024-03-18T20:25:22.251851+0000","last_active":"2024-03-18T20:25:23.674205+0000","last_peered":"2024-03-18T20:25:23.674205+0000","last_clean":"2024-03-18T20:25:23.674205+0000","last_became_active":"2024-03-18T20:25:22.251334+0000","last_became_peered":"2024-03-18T20:25:22.251334+0000","last_unstale":"2024-03-18T20:25:23.674205+0000","last_undegraded":"2024-03-18T20:25:23.674205+0000","last_fullsized":"2024-03-18T20:25:23.674205+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:22.889781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.606007+0000","last_change":"2024-03-18T20:25:22.681615+0000","last_active":"2024-03-18T20:25:24.606007+0000","last_peered":"2024-03-18T20:25:24.606007+0000","last_clean":"2024-03-18T20:25:24.606007+0000","last_became_active":"2024-03-18T20:25:22.681261+0000","last_became_peered":"2024-03-18T20:25:22.681261+0000","last_unstale":"2024-03-18T20:25:24.606007+0000","last_undegraded":"2024-03-18T20:25:24.606007+0000","last_fullsized":"2024-03-18T20:25:24.606007+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:15:29.461741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605241+0000","last_change":"2024-03-18T20:25:22.247489+0000","last_active":"2024-03-18T20:25:24.605241+0000","last_peered":"2024-03-18T20:25:24.605241+0000","last_clean":"2024-03-18T20:25:24.605241+0000","last_became_active":"2024-03-18T20:25:22.247203+0000","last_became_peered":"2024-03-18T20:25:22.247203+0000","last_unstale":"2024-03-18T20:25:24.605241+0000","last_undegraded":"2024-03-18T20:25:24.605241+0000","last_fullsized":"2024-03-18T20:25:24.605241+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:32.481118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674335+0000","last_change":"2024-03-18T20:25:22.684929+0000","last_active":"2024-03-18T20:25:23.674335+0000","last_peered":"2024-03-18T20:25:23.674335+0000","last_clean":"2024-03-18T20:25:23.674335+0000","last_became_active":"2024-03-18T20:25:22.684658+0000","last_became_peered":"2024-03-18T20:25:22.684658+0000","last_unstale":"2024-03-18T20:25:23.674335+0000","last_undegraded":"2024-03-18T20:25:23.674335+0000","last_fullsized":"2024-03-18T20:25:23.674335+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:39.733357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.606006+0000","last_change":"2024-03-18T20:25:22.251818+0000","last_active":"2024-03-18T20:25:24.606006+0000","last_peered":"2024-03-18T20:25:24.606006+0000","last_clean":"2024-03-18T20:25:24.606006+0000","last_became_active":"2024-03-18T20:25:22.251407+0000","last_became_peered":"2024-03-18T20:25:22.251407+0000","last_unstale":"2024-03-18T20:25:24.606006+0000","last_undegraded":"2024-03-18T20:25:24.606006+0000","last_fullsized":"2024-03-18T20:25:24.606006+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:29.196742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673490+0000","last_change":"2024-03-18T20:25:22.695653+0000","last_active":"2024-03-18T20:25:23.673490+0000","last_peered":"2024-03-18T20:25:23.673490+0000","last_clean":"2024-03-18T20:25:23.673490+0000","last_became_active":"2024-03-18T20:25:22.694092+0000","last_became_peered":"2024-03-18T20:25:22.694092+0000","last_unstale":"2024-03-18T20:25:23.673490+0000","last_undegraded":"2024-03-18T20:25:23.673490+0000","last_fullsized":"2024-03-18T20:25:23.673490+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:20.319345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.606523+0000","last_change":"2024-03-18T20:25:22.251509+0000","last_active":"2024-03-18T20:25:24.606523+0000","last_peered":"2024-03-18T20:25:24.606523+0000","last_clean":"2024-03-18T20:25:24.606523+0000","last_became_active":"2024-03-18T20:25:22.251240+0000","last_became_peered":"2024-03-18T20:25:22.251240+0000","last_unstale":"2024-03-18T20:25:24.606523+0000","last_undegraded":"2024-03-18T20:25:24.606523+0000","last_fullsized":"2024-03-18T20:25:24.606523+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:13:14.870270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.672984+0000","last_change":"2024-03-18T20:25:22.693170+0000","last_active":"2024-03-18T20:25:23.672984+0000","last_peered":"2024-03-18T20:25:23.672984+0000","last_clean":"2024-03-18T20:25:23.672984+0000","last_became_active":"2024-03-18T20:25:22.691592+0000","last_became_peered":"2024-03-18T20:25:22.691592+0000","last_unstale":"2024-03-18T20:25:23.672984+0000","last_undegraded":"2024-03-18T20:25:23.672984+0000","last_fullsized":"2024-03-18T20:25:23.672984+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:23.897085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"231'2","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674710+0000","last_change":"2024-03-18T20:25:22.254151+0000","last_active":"2024-03-18T20:25:23.674710+0000","last_peered":"2024-03-18T20:25:23.674710+0000","last_clean":"2024-03-18T20:25:23.674710+0000","last_became_active":"2024-03-18T20:25:22.253873+0000","last_became_peered":"2024-03-18T20:25:22.253873+0000","last_unstale":"2024-03-18T20:25:23.674710+0000","last_undegraded":"2024-03-18T20:25:23.674710+0000","last_fullsized":"2024-03-18T20:25:23.674710+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:26:58.327699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104222+0000","last_change":"2024-03-18T20:25:22.689011+0000","last_active":"2024-03-18T20:25:24.104222+0000","last_peered":"2024-03-18T20:25:24.104222+0000","last_clean":"2024-03-18T20:25:24.104222+0000","last_became_active":"2024-03-18T20:25:22.688750+0000","last_became_peered":"2024-03-18T20:25:22.688750+0000","last_unstale":"2024-03-18T20:25:24.104222+0000","last_undegraded":"2024-03-18T20:25:24.104222+0000","last_fullsized":"2024-03-18T20:25:24.104222+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:57.345369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673442+0000","last_change":"2024-03-18T20:25:22.247183+0000","last_active":"2024-03-18T20:25:23.673442+0000","last_peered":"2024-03-18T20:25:23.673442+0000","last_clean":"2024-03-18T20:25:23.673442+0000","last_became_active":"2024-03-18T20:25:22.246890+0000","last_became_peered":"2024-03-18T20:25:22.246890+0000","last_unstale":"2024-03-18T20:25:23.673442+0000","last_undegraded":"2024-03-18T20:25:23.673442+0000","last_fullsized":"2024-03-18T20:25:23.673442+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:40:21.886849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673951+0000","last_change":"2024-03-18T20:25:22.691245+0000","last_active":"2024-03-18T20:25:23.673951+0000","last_peered":"2024-03-18T20:25:23.673951+0000","last_clean":"2024-03-18T20:25:23.673951+0000","last_became_active":"2024-03-18T20:25:22.690988+0000","last_became_peered":"2024-03-18T20:25:22.690988+0000","last_unstale":"2024-03-18T20:25:23.673951+0000","last_undegraded":"2024-03-18T20:25:23.673951+0000","last_fullsized":"2024-03-18T20:25:23.673951+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:32:54.125659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.606085+0000","last_change":"2024-03-18T20:25:22.250457+0000","last_active":"2024-03-18T20:25:24.606085+0000","last_peered":"2024-03-18T20:25:24.606085+0000","last_clean":"2024-03-18T20:25:24.606085+0000","last_became_active":"2024-03-18T20:25:22.250157+0000","last_became_peered":"2024-03-18T20:25:22.250157+0000","last_unstale":"2024-03-18T20:25:24.606085+0000","last_undegraded":"2024-03-18T20:25:24.606085+0000","last_fullsized":"2024-03-18T20:25:24.606085+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:18.568149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.606876+0000","last_change":"2024-03-18T20:25:22.682910+0000","last_active":"2024-03-18T20:25:24.606876+0000","last_peered":"2024-03-18T20:25:24.606876+0000","last_clean":"2024-03-18T20:25:24.606876+0000","last_became_active":"2024-03-18T20:25:22.682667+0000","last_became_peered":"2024-03-18T20:25:22.682667+0000","last_unstale":"2024-03-18T20:25:24.606876+0000","last_undegraded":"2024-03-18T20:25:24.606876+0000","last_fullsized":"2024-03-18T20:25:24.606876+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:02.037780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.606952+0000","last_change":"2024-03-18T20:25:22.255435+0000","last_active":"2024-03-18T20:25:24.606952+0000","last_peered":"2024-03-18T20:25:24.606952+0000","last_clean":"2024-03-18T20:25:24.606952+0000","last_became_active":"2024-03-18T20:25:22.255149+0000","last_became_peered":"2024-03-18T20:25:22.255149+0000","last_unstale":"2024-03-18T20:25:24.606952+0000","last_undegraded":"2024-03-18T20:25:24.606952+0000","last_fullsized":"2024-03-18T20:25:24.606952+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:58.960198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673372+0000","last_change":"2024-03-18T20:25:22.683574+0000","last_active":"2024-03-18T20:25:23.673372+0000","last_peered":"2024-03-18T20:25:23.673372+0000","last_clean":"2024-03-18T20:25:23.673372+0000","last_became_active":"2024-03-18T20:25:22.683015+0000","last_became_peered":"2024-03-18T20:25:22.683015+0000","last_unstale":"2024-03-18T20:25:23.673372+0000","last_undegraded":"2024-03-18T20:25:23.673372+0000","last_fullsized":"2024-03-18T20:25:23.673372+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:11.606826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673204+0000","last_change":"2024-03-18T20:25:22.245633+0000","last_active":"2024-03-18T20:25:23.673204+0000","last_peered":"2024-03-18T20:25:23.673204+0000","last_clean":"2024-03-18T20:25:23.673204+0000","last_became_active":"2024-03-18T20:25:22.245372+0000","last_became_peered":"2024-03-18T20:25:22.245372+0000","last_unstale":"2024-03-18T20:25:23.673204+0000","last_undegraded":"2024-03-18T20:25:23.673204+0000","last_fullsized":"2024-03-18T20:25:23.673204+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:28:25.048682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.606123+0000","last_change":"2024-03-18T20:25:22.685300+0000","last_active":"2024-03-18T20:25:24.606123+0000","last_peered":"2024-03-18T20:25:24.606123+0000","last_clean":"2024-03-18T20:25:24.606123+0000","last_became_active":"2024-03-18T20:25:22.684869+0000","last_became_peered":"2024-03-18T20:25:22.684869+0000","last_unstale":"2024-03-18T20:25:24.606123+0000","last_undegraded":"2024-03-18T20:25:24.606123+0000","last_fullsized":"2024-03-18T20:25:24.606123+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:02.037780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674052+0000","last_change":"2024-03-18T20:25:22.254053+0000","last_active":"2024-03-18T20:25:23.674052+0000","last_peered":"2024-03-18T20:25:23.674052+0000","last_clean":"2024-03-18T20:25:23.674052+0000","last_became_active":"2024-03-18T20:25:22.253747+0000","last_became_peered":"2024-03-18T20:25:22.253747+0000","last_unstale":"2024-03-18T20:25:23.674052+0000","last_undegraded":"2024-03-18T20:25:23.674052+0000","last_fullsized":"2024-03-18T20:25:23.674052+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:32.777143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.606005+0000","last_change":"2024-03-18T20:25:22.689532+0000","last_active":"2024-03-18T20:25:24.606005+0000","last_peered":"2024-03-18T20:25:24.606005+0000","last_clean":"2024-03-18T20:25:24.606005+0000","last_became_active":"2024-03-18T20:25:22.689210+0000","last_became_peered":"2024-03-18T20:25:22.689210+0000","last_unstale":"2024-03-18T20:25:24.606005+0000","last_undegraded":"2024-03-18T20:25:24.606005+0000","last_fullsized":"2024-03-18T20:25:24.606005+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:21.229791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.606716+0000","last_change":"2024-03-18T20:25:22.252889+0000","last_active":"2024-03-18T20:25:24.606716+0000","last_peered":"2024-03-18T20:25:24.606716+0000","last_clean":"2024-03-18T20:25:24.606716+0000","last_became_active":"2024-03-18T20:25:22.252096+0000","last_became_peered":"2024-03-18T20:25:22.252096+0000","last_unstale":"2024-03-18T20:25:24.606716+0000","last_undegraded":"2024-03-18T20:25:24.606716+0000","last_fullsized":"2024-03-18T20:25:24.606716+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:18.568149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.104385+0000","last_change":"2024-03-18T20:25:22.696025+0000","last_active":"2024-03-18T20:25:24.104385+0000","last_peered":"2024-03-18T20:25:24.104385+0000","last_clean":"2024-03-18T20:25:24.104385+0000","last_became_active":"2024-03-18T20:25:22.695759+0000","last_became_peered":"2024-03-18T20:25:22.695759+0000","last_unstale":"2024-03-18T20:25:24.104385+0000","last_undegraded":"2024-03-18T20:25:24.104385+0000","last_fullsized":"2024-03-18T20:25:24.104385+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:10.681789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.673946+0000","last_change":"2024-03-18T20:25:22.252803+0000","last_active":"2024-03-18T20:25:23.673946+0000","last_peered":"2024-03-18T20:25:23.673946+0000","last_clean":"2024-03-18T20:25:23.673946+0000","last_became_active":"2024-03-18T20:25:22.252169+0000","last_became_peered":"2024-03-18T20:25:22.252169+0000","last_unstale":"2024-03-18T20:25:23.673946+0000","last_undegraded":"2024-03-18T20:25:23.673946+0000","last_fullsized":"2024-03-18T20:25:23.673946+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:12.985428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.606854+0000","last_change":"2024-03-18T20:25:22.687598+0000","last_active":"2024-03-18T20:25:24.606854+0000","last_peered":"2024-03-18T20:25:24.606854+0000","last_clean":"2024-03-18T20:25:24.606854+0000","last_became_active":"2024-03-18T20:25:22.686962+0000","last_became_peered":"2024-03-18T20:25:22.686962+0000","last_unstale":"2024-03-18T20:25:24.606854+0000","last_undegraded":"2024-03-18T20:25:24.606854+0000","last_fullsized":"2024-03-18T20:25:24.606854+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:23:10.983253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"231'1","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:23.674060+0000","last_change":"2024-03-18T20:25:22.248248+0000","last_active":"2024-03-18T20:25:23.674060+0000","last_peered":"2024-03-18T20:25:23.674060+0000","last_clean":"2024-03-18T20:25:23.674060+0000","last_became_active":"2024-03-18T20:25:22.247970+0000","last_became_peered":"2024-03-18T20:25:22.247970+0000","last_unstale":"2024-03-18T20:25:23.674060+0000","last_undegraded":"2024-03-18T20:25:23.674060+0000","last_fullsized":"2024-03-18T20:25:23.674060+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:01:50.523530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"221'348","reported_seq":1200,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-18T20:25:22.668677+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:25:22.668677+0000","last_peered":"2024-03-18T20:25:22.668677+0000","last_clean":"2024-03-18T20:25:22.668677+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:25:22.668677+0000","last_undegraded":"2024-03-18T20:25:22.668677+0000","last_fullsized":"2024-03-18T20:25:22.668677+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575981,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52600,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319240,"statfs":{"total":96636764160,"available":96582901760,"internally_reserved":0,"allocated":3510272,"data_stored":2078323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80700000000000005,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.48499999999999999,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.81899999999999995,"5min":1.093,"15min":1.093},"last":0.59699999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61799999999999999,"15min":0.61699999999999999},"min":{"1min":0.47599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.82999999999999996,"5min":0.99399999999999999,"15min":1.4299999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.60999999999999999,"15min":0.61899999999999999},"min":{"1min":0.47099999999999997,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.77500000000000002,"5min":1.0449999999999999,"15min":1.2629999999999999},"last":0.52000000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.46400000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84299999999999997,"5min":1.216,"15min":1.216},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.62,"15min":0.625},"min":{"1min":0.48399999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.81799999999999995,"5min":1.177,"15min":1.2490000000000001},"last":0.61499999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.42599999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84699999999999998,"5min":0.93100000000000005,"15min":1.4450000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.5,"5min":0.44900000000000001,"15min":0.38600000000000001},"max":{"1min":0.84399999999999997,"5min":1.028,"15min":1.5029999999999999},"last":0.502}]},{"osd":4,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.47499999999999998,"15min":0.47699999999999998},"min":{"1min":0.33700000000000002,"5min":0.314,"15min":0.29899999999999999},"max":{"1min":0.68400000000000005,"5min":0.79100000000000004,"15min":0.86099999999999999},"last":0.375},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.35199999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71599999999999997,"5min":0.76600000000000001,"15min":0.80200000000000005},"last":0.42799999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51500000000000001,"15min":0.51600000000000001},"min":{"1min":0.34899999999999998,"5min":0.34399999999999997,"15min":0.316},"max":{"1min":0.68999999999999995,"5min":0.86599999999999999,"15min":0.89400000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.40699999999999997,"5min":0.377,"15min":0.36799999999999999},"max":{"1min":0.75600000000000001,"5min":0.81599999999999995,"15min":1.042},"last":0.45100000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56399999999999995,"15min":0.57699999999999996},"min":{"1min":0.39900000000000002,"5min":0.36199999999999999,"15min":0.34999999999999998},"max":{"1min":0.71099999999999997,"5min":0.85599999999999998,"15min":1.129},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.371,"5min":0.36299999999999999,"15min":0.34000000000000002},"max":{"1min":0.82899999999999996,"5min":0.85299999999999998,"15min":1.0609999999999999},"last":0.48099999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575981,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42180,"kb_used_data":2860,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329660,"statfs":{"total":96636764160,"available":96593571840,"internally_reserved":0,"allocated":2928640,"data_stored":1495187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57599999999999996,"15min":0.60099999999999998},"min":{"1min":0.41399999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72199999999999998,"5min":0.80900000000000005,"15min":1.1519999999999999},"last":0.371},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59099999999999997,"15min":0.60799999999999998},"min":{"1min":0.33700000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.74299999999999999,"5min":0.81499999999999995,"15min":0.83299999999999996},"last":0.45900000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57899999999999996,"15min":0.60199999999999998},"min":{"1min":0.39400000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.79100000000000004,"5min":0.79800000000000004,"15min":1.244},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58499999999999996,"15min":0.60399999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33200000000000002},"max":{"1min":0.77000000000000002,"5min":0.89200000000000002,"15min":1.286},"last":0.47099999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60999999999999999,"15min":0.63},"min":{"1min":0.47899999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71199999999999997,"5min":0.92000000000000004,"15min":1.276},"last":0.308},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.59299999999999997,"15min":0.61099999999999999},"min":{"1min":0.434,"5min":0.40100000000000002,"15min":0.378},"max":{"1min":0.77500000000000002,"5min":0.95399999999999996,"15min":1.2669999999999999},"last":0.39500000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59699999999999998,"15min":0.622},"min":{"1min":0.38,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70699999999999996,"5min":0.83199999999999996,"15min":1.02},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60599999999999998,"15min":0.61899999999999999},"min":{"1min":0.45200000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82399999999999995,"5min":1.0049999999999999,"15min":1.337},"last":0.44400000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.48099999999999998,"15min":0.48299999999999998},"min":{"1min":0.308,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.61699999999999999,"5min":0.85799999999999998,"15min":0.88300000000000001},"last":0.435},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.50900000000000001,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72099999999999997,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.49099999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.5,"15min":0.504},"min":{"1min":0.375,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.84799999999999998},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39000000000000001,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.73399999999999999,"5min":0.92600000000000005,"15min":0.94199999999999995},"last":0.501}]},{"osd":7,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40600000000000003,"5min":0.35799999999999998,"15min":0.34499999999999997},"max":{"1min":0.74299999999999999,"5min":0.95599999999999996,"15min":1.0429999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.44800000000000001,"5min":0.36599999999999999,"15min":0.34599999999999997},"max":{"1min":0.83899999999999997,"5min":1.028,"15min":1.028},"last":0.51200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575981,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47912,"kb_used_data":2844,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323928,"statfs":{"total":96636764160,"available":96587702272,"internally_reserved":0,"allocated":2912256,"data_stored":1493041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.40400000000000003,"15min":0.39600000000000002},"min":{"1min":0.25600000000000001,"5min":0.246,"15min":0.23999999999999999},"max":{"1min":0.57999999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.41899999999999998,"15min":0.42999999999999999},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.58999999999999997,"5min":0.69699999999999995,"15min":0.72299999999999998},"last":0.56499999999999995}]},{"osd":2,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.44,"15min":0.439},"min":{"1min":0.309,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61199999999999999,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48099999999999998,"15min":0.47799999999999998},"min":{"1min":0.35099999999999998,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.70099999999999996,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.39300000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.495,"15min":0.495},"min":{"1min":0.36199999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.64300000000000002,"5min":0.75900000000000001,"15min":0.76100000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.44800000000000001,"15min":0.442},"min":{"1min":0.32800000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68200000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.41799999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.68300000000000005,"15min":0.67300000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86799999999999999,"5min":1.0680000000000001,"15min":1.2909999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.68100000000000005,"15min":0.68000000000000005},"min":{"1min":0.50900000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.84499999999999997,"5min":0.92500000000000004,"15min":1.746},"last":0.70299999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.68100000000000005,"15min":0.67000000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.6910000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.71999999999999997,"15min":0.71399999999999997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.46899999999999997},"max":{"1min":0.89400000000000002,"5min":1.109,"15min":1.2549999999999999},"last":0.91100000000000003}]},{"osd":6,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.69699999999999995,"15min":0.68500000000000005},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.437},"max":{"1min":0.97399999999999998,"5min":0.98899999999999999,"15min":1.637},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71899999999999997,"15min":0.71199999999999997},"min":{"1min":0.49099999999999999,"5min":0.46999999999999997,"15min":0.46100000000000002},"max":{"1min":0.89500000000000002,"5min":1.014,"15min":1.722},"last":0.68899999999999995}]},{"osd":7,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.70699999999999996,"15min":0.71299999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79800000000000004,"5min":1.02,"15min":1.2729999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.498,"5min":0.495,"15min":0.42799999999999999},"max":{"1min":0.90300000000000002,"5min":1.0329999999999999,"15min":1.3380000000000001},"last":0.84199999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575982,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30456,"kb_used_data":3428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341384,"statfs":{"total":96636764160,"available":96605577216,"internally_reserved":0,"allocated":3510272,"data_stored":2083499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.42399999999999999,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.42099999999999999,"15min":0.42799999999999999},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.46600000000000003,"5min":0.64700000000000002,"15min":1.1850000000000001},"last":0.39100000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.42599999999999999,"15min":0.435},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.47099999999999997,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.438,"15min":0.45300000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29799999999999999},"max":{"1min":0.59199999999999997,"5min":0.64300000000000002,"15min":0.70999999999999996},"last":0.27900000000000003}]},{"osd":3,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.26100000000000001},"max":{"1min":0.45800000000000002,"5min":0.60099999999999998,"15min":0.628},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48599999999999999,"15min":0.496},"min":{"1min":0.35499999999999998,"5min":0.35199999999999998,"15min":0.32500000000000001},"max":{"1min":0.52300000000000002,"5min":0.73999999999999999,"15min":0.91500000000000004},"last":0.33200000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64300000000000002,"15min":0.65500000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81299999999999994,"5min":0.91500000000000004,"15min":1.0029999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.66200000000000003,"15min":0.66300000000000003},"min":{"1min":0.47499999999999998,"5min":0.47099999999999997,"15min":0.39700000000000002},"max":{"1min":0.78100000000000003,"5min":0.95299999999999996,"15min":1.0469999999999999},"last":0.71299999999999997}]},{"osd":5,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.67800000000000005,"15min":0.67100000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89700000000000002,"5min":0.97099999999999997,"15min":1.0129999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91100000000000003,"5min":0.92900000000000005,"15min":1.518},"last":0.64100000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.47399999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.78600000000000003,"5min":0.95899999999999996,"15min":1.1279999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71199999999999997,"15min":0.70899999999999996},"min":{"1min":0.498,"5min":0.48999999999999999,"15min":0.47899999999999998},"max":{"1min":0.94899999999999995,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.55800000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85099999999999998,"5min":0.96399999999999997,"15min":1.464},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.69899999999999995,"15min":0.70399999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.47599999999999998},"max":{"1min":0.75800000000000001,"5min":1.042,"15min":1.2350000000000001},"last":0.56799999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575982,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47916,"kb_used_data":2848,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323924,"statfs":{"total":96636764160,"available":96587698176,"internally_reserved":0,"allocated":2916352,"data_stored":1493131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.379,"15min":0.38},"min":{"1min":0.27700000000000002,"5min":0.251,"15min":0.23899999999999999},"max":{"1min":0.54800000000000004,"5min":0.55300000000000005,"15min":0.70799999999999996},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.48199999999999998,"15min":0.48899999999999999},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.24199999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.83399999999999996},"last":0.36299999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48099999999999998,"15min":0.48199999999999998},"min":{"1min":0.36299999999999999,"5min":0.34399999999999997,"15min":0.30299999999999999},"max":{"1min":0.59499999999999997,"5min":0.72599999999999998,"15min":0.77100000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.42099999999999999,"15min":0.41999999999999998},"min":{"1min":0.28399999999999997,"5min":0.27800000000000002,"15min":0.27600000000000002},"max":{"1min":0.52000000000000002,"5min":0.59299999999999997,"15min":0.71099999999999997},"last":0.34399999999999997}]},{"osd":3,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.437,"15min":0.437},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.58999999999999997,"5min":0.64700000000000002,"15min":0.66300000000000003},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.51800000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.313},"max":{"1min":0.66200000000000003,"5min":0.752,"15min":0.752},"last":0.45400000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.66500000000000004,"15min":0.65800000000000003},"min":{"1min":0.44400000000000001,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":0.76500000000000001,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.66800000000000004},"min":{"1min":0.48899999999999999,"5min":0.443,"15min":0.443},"max":{"1min":0.95799999999999996,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.63600000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68200000000000005,"15min":0.68400000000000005},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.45300000000000001},"max":{"1min":0.97299999999999998,"5min":1.028,"15min":1.028},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.68100000000000005,"15min":0.67900000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.88200000000000001,"5min":0.95599999999999996,"15min":1.0629999999999999},"last":0.56399999999999995}]},{"osd":6,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69499999999999995,"15min":0.69399999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.014,"5min":1.014,"15min":1.2050000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.73199999999999998,"15min":0.73099999999999998},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.90200000000000002,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.70799999999999996}]},{"osd":7,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.501,"5min":0.442,"15min":0.42199999999999999},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":1.0469999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.70699999999999996,"15min":0.70899999999999996},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.92400000000000004,"5min":1.371,"15min":1.371},"last":0.71699999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575982,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52016,"kb_used_data":2844,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319824,"statfs":{"total":96636764160,"available":96583499776,"internally_reserved":0,"allocated":2912256,"data_stored":1486447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.38800000000000001,"15min":0.39900000000000002},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.53700000000000003,"5min":0.66500000000000004,"15min":0.98099999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.40999999999999998,"15min":0.41499999999999998},"min":{"1min":0.27700000000000002,"5min":0.25800000000000001,"15min":0.24199999999999999},"max":{"1min":0.434,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.317}]},{"osd":1,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39000000000000001,"15min":0.40999999999999998},"min":{"1min":0.30099999999999999,"5min":0.26100000000000001,"15min":0.254},"max":{"1min":0.45900000000000002,"5min":0.63200000000000001,"15min":0.64000000000000001},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.436,"5min":0.46100000000000002,"15min":0.46600000000000003},"min":{"1min":0.36699999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57499999999999996,"5min":0.65500000000000003,"15min":0.73099999999999998},"last":0.371}]},{"osd":2,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.45500000000000002,"15min":0.46600000000000003},"min":{"1min":0.34999999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53100000000000003,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.48199999999999998,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.51700000000000002,"5min":0.67000000000000004,"15min":0.73099999999999998},"last":0.33800000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.63600000000000001,"15min":0.65200000000000002},"min":{"1min":0.48799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81899999999999995,"5min":0.91700000000000004,"15min":1.097},"last":1.1950000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65200000000000002,"15min":0.67500000000000004},"min":{"1min":0.435,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78800000000000003,"5min":0.99299999999999999,"15min":1.0569999999999999},"last":1.0980000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68400000000000005,"15min":0.69699999999999995},"min":{"1min":0.48999999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92100000000000004,"5min":0.995,"15min":1.1599999999999999},"last":1.1819999999999999},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.72499999999999998,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.82099999999999995,"5min":1.171,"15min":1.171},"last":1.1120000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68500000000000005,"15min":0.70799999999999996},"min":{"1min":0.48099999999999998,"5min":0.436,"15min":0.42499999999999999},"max":{"1min":0.90500000000000003,"5min":1.012,"15min":1.194},"last":1.214},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.71799999999999997,"15min":0.73299999999999998},"min":{"1min":0.53300000000000003,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":0.88200000000000001,"5min":1.0409999999999999,"15min":1.2410000000000001},"last":1.22}]},{"osd":7,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.70099999999999996,"15min":0.71899999999999997},"min":{"1min":0.49099999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82799999999999996,"5min":1.079,"15min":1.079},"last":1.0820000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68100000000000005,"15min":0.69999999999999996},"min":{"1min":0.44500000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83799999999999997,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":1.1699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575982,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64296,"kb_used_data":2836,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307544,"statfs":{"total":96636764160,"available":96570925056,"internally_reserved":0,"allocated":2904064,"data_stored":1486903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58999999999999997,"15min":0.60099999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.81799999999999995},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.745,"5min":0.79800000000000004,"15min":0.84199999999999997},"last":0.56599999999999995}]},{"osd":1,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.60599999999999998,"15min":0.61899999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.79000000000000004,"5min":0.81899999999999995,"15min":0.88300000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.75700000000000001,"5min":0.79200000000000004,"15min":0.96499999999999997},"last":0.48099999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.60299999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39200000000000002},"max":{"1min":0.79100000000000004,"5min":0.80500000000000005,"15min":0.82599999999999996},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.60999999999999999,"15min":0.623},"min":{"1min":0.46600000000000003,"5min":0.41699999999999998,"15min":0.38500000000000001},"max":{"1min":0.70599999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.64600000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59799999999999998,"15min":0.60599999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":0.76900000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.622},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60399999999999998,"15min":0.625},"min":{"1min":0.432,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73199999999999998,"5min":0.79100000000000004,"15min":0.86199999999999999},"last":0.66500000000000004}]},{"osd":5,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.46100000000000002,"15min":0.47799999999999998},"min":{"1min":0.35599999999999998,"5min":0.33600000000000002,"15min":0.315},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.76000000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48599999999999999,"15min":0.504},"min":{"1min":0.30299999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.68100000000000005}]},{"osd":6,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48599999999999999,"15min":0.50600000000000001},"min":{"1min":0.34899999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.77100000000000002},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.54700000000000004},"min":{"1min":0.35399999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.74099999999999999,"5min":0.76200000000000001,"15min":0.79900000000000004},"last":0.72499999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52700000000000002,"15min":0.54500000000000004},"min":{"1min":0.39600000000000002,"5min":0.38,"15min":0.35699999999999998},"max":{"1min":0.78800000000000003,"5min":0.87,"15min":0.87},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55700000000000005,"15min":0.57899999999999996},"min":{"1min":0.438,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.71999999999999997,"5min":0.81200000000000006,"15min":0.85399999999999998},"last":0.73599999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575981,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47920,"kb_used_data":2852,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323920,"statfs":{"total":96636764160,"available":96587694080,"internally_reserved":0,"allocated":2920448,"data_stored":1493097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42499999999999999,"5min":0.33900000000000002,"15min":0.31900000000000001},"max":{"1min":0.72699999999999998,"5min":0.85999999999999999,"15min":0.96499999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57799999999999996,"15min":0.60399999999999998},"min":{"1min":0.432,"5min":0.37,"15min":0.33100000000000002},"max":{"1min":0.69799999999999995,"5min":0.82599999999999996,"15min":1.3100000000000001},"last":0.57099999999999995}]},{"osd":1,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58499999999999996,"15min":0.60199999999999998},"min":{"1min":0.36899999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.74399999999999999,"5min":0.79600000000000004,"15min":1.056},"last":0.443},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58399999999999996,"15min":0.60199999999999998},"min":{"1min":0.46600000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.63100000000000001,"5min":0.89800000000000002,"15min":1.018},"last":0.56000000000000005}]},{"osd":2,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.38700000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.84299999999999997,"15min":0.99299999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59399999999999997,"15min":0.60899999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.67500000000000004,"5min":0.875,"15min":1.1120000000000001},"last":0.54700000000000004}]},{"osd":3,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.42799999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.78100000000000003,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63,"15min":0.64600000000000002},"min":{"1min":0.443,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69899999999999995,"5min":0.82399999999999995,"15min":1.161},"last":0.52800000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.47899999999999998,"15min":0.47499999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.77500000000000002},"last":0.502},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51200000000000001,"15min":0.50600000000000001},"min":{"1min":0.36199999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.89700000000000002},"last":0.59899999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.503,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.34000000000000002,"15min":0.32800000000000001},"max":{"1min":0.69999999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.40300000000000002,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.86599999999999999},"last":0.58399999999999996}]},{"osd":7,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.40799999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.94499999999999995},"last":0.629},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.45300000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74099999999999999,"5min":0.85999999999999999,"15min":0.91200000000000003},"last":0.63900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:25:27.230 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T20:25:27.231 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-18T20:25:27.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.395+0000 7f806101b700 1 Processor -- start 2024-03-18T20:25:27.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.395+0000 7f806101b700 1 -- start start 2024-03-18T20:25:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.396+0000 7f806101b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f805c06ca80 0x7f805c15d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.396+0000 7f806101b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f805c09a150 0x7f805c0cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.396+0000 7f806101b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805c0b3b00 con 0x7f805c15d5b0 2024-03-18T20:25:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.396+0000 7f806101b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f805c0cbbb0 con 0x7f805c06ca80 2024-03-18T20:25:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.396+0000 7f806101b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f805c0cf7f0 con 0x7f805c09a150 2024-03-18T20:25:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.396+0000 7f805ad9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f805c15d5b0 0x7f805c099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39526/0 (socket says 172.21.15.1:39526) 2024-03-18T20:25:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.396+0000 7f805ad9d700 1 -- 172.21.15.1:0/2203562236 learned_addr learned my addr 172.21.15.1:0/2203562236 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.396+0000 7f805a59c700 1 --2- 172.21.15.1:0/2203562236 >> v2:172.21.15.165:3300/0 conn(0x7f805c06ca80 0x7f805c15d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.396+0000 7f805b59e700 1 --2- 172.21.15.1:0/2203562236 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f805c09a150 0x7f805c0cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.396+0000 7f8059d9b700 1 -- 172.21.15.1:0/2203562236 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829708736 0 0) 0x7f805c0b3b00 con 0x7f805c15d5b0 2024-03-18T20:25:27.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.396+0000 7f8059d9b700 1 -- 172.21.15.1:0/2203562236 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80440035b0 con 0x7f805c15d5b0 2024-03-18T20:25:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.396+0000 7f8059d9b700 1 -- 172.21.15.1:0/2203562236 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1955162468 0 0) 0x7f80440035b0 con 0x7f805c15d5b0 2024-03-18T20:25:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.396+0000 7f8059d9b700 1 -- 172.21.15.1:0/2203562236 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f805c09a150 msgr2=0x7f805c0cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.396+0000 7f8059d9b700 1 --2- 172.21.15.1:0/2203562236 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f805c09a150 0x7f805c0cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.396+0000 7f8059d9b700 1 -- 172.21.15.1:0/2203562236 >> v2:172.21.15.165:3300/0 conn(0x7f805c06ca80 msgr2=0x7f805c15d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.396+0000 7f8059d9b700 1 --2- 172.21.15.1:0/2203562236 >> v2:172.21.15.165:3300/0 conn(0x7f805c06ca80 0x7f805c15d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.396+0000 7f8059d9b700 1 -- 172.21.15.1:0/2203562236 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805c0d04d0 con 0x7f805c15d5b0 2024-03-18T20:25:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.396+0000 7f8059d9b700 1 -- 172.21.15.1:0/2203562236 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f8050001670 con 0x7f805c15d5b0 2024-03-18T20:25:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f8059d9b700 1 -- 172.21.15.1:0/2203562236 >> v1:172.21.15.1:6789/0 conn(0x7f805c15d5b0 legacy=0x7f805c099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f8059d9b700 1 --2- 172.21.15.1:0/2203562236 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8044003a00 0x7f8044003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f8059d9b700 1 --2- 172.21.15.1:0/2203562236 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8044004850 0x7f8044004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f8059d9b700 1 --2- 172.21.15.1:0/2203562236 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f805c09a150 0x7f8044005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f8059d9b700 1 -- 172.21.15.1:0/2203562236 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c0cbbb0 con 0x7f8044003a00 2024-03-18T20:25:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f8059d9b700 1 -- 172.21.15.1:0/2203562236 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c0cf7f0 con 0x7f8044004850 2024-03-18T20:25:27.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f8059d9b700 1 -- 172.21.15.1:0/2203562236 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f804400ba10 con 0x7f805c09a150 2024-03-18T20:25:27.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f805ad9d700 1 --2- 172.21.15.1:0/2203562236 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8044003a00 0x7f8044003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:27.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f805a59c700 1 --2- 172.21.15.1:0/2203562236 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8044004850 0x7f8044004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:27.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f805b59e700 1 --2- 172.21.15.1:0/2203562236 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f805c09a150 0x7f8044005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:27.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f805ad9d700 1 -- 172.21.15.1:0/2203562236 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f805c09a150 msgr2=0x7f8044005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:27.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f805ad9d700 1 --2- 172.21.15.1:0/2203562236 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f805c09a150 0x7f8044005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f805ad9d700 1 -- 172.21.15.1:0/2203562236 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8044004850 msgr2=0x7f8044004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:27.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f805ad9d700 1 --2- 172.21.15.1:0/2203562236 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8044004850 0x7f8044004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f805ad9d700 1 -- 172.21.15.1:0/2203562236 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80440035b0 con 0x7f8044003a00 2024-03-18T20:25:27.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f805b59e700 1 --2- 172.21.15.1:0/2203562236 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f805c09a150 0x7f8044005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:27.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f805a59c700 1 --2- 172.21.15.1:0/2203562236 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8044004850 0x7f8044004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:27.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f805ad9d700 1 --2- 172.21.15.1:0/2203562236 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8044003a00 0x7f8044003e20 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7f805000c350 tx=0x7f8044005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c6a4a685ae2c43 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:27.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f8059d9b700 1 -- 172.21.15.1:0/2203562236 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805000ce50 con 0x7f8044003a00 2024-03-18T20:25:27.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f8059d9b700 1 -- 172.21.15.1:0/2203562236 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8050005960 con 0x7f8044003a00 2024-03-18T20:25:27.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.397+0000 7f8059d9b700 1 -- 172.21.15.1:0/2203562236 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050005d10 con 0x7f8044003a00 2024-03-18T20:25:27.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.398+0000 7f806101b700 1 -- 172.21.15.1:0/2203562236 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8044003a00 msgr2=0x7f8044003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:27.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.398+0000 7f806101b700 1 --2- 172.21.15.1:0/2203562236 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8044003a00 0x7f8044003e20 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7f805000c350 tx=0x7f8044005880 comp rx=0 tx=0).stop 2024-03-18T20:25:27.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.398+0000 7f805b59e700 1 -- 172.21.15.1:0/2203562236 reap_dead start 2024-03-18T20:25:27.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.398+0000 7f806101b700 1 -- 172.21.15.1:0/2203562236 shutdown_connections 2024-03-18T20:25:27.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.398+0000 7f806101b700 1 -- 172.21.15.1:0/2203562236 >> 172.21.15.1:0/2203562236 conn(0x7f805c0b2bd0 msgr2=0x7f805c0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:27.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.398+0000 7f806101b700 1 -- 172.21.15.1:0/2203562236 shutdown_connections 2024-03-18T20:25:27.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.398+0000 7f806101b700 1 -- 172.21.15.1:0/2203562236 wait complete. 2024-03-18T20:25:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.399+0000 7f806101b700 1 Processor -- start 2024-03-18T20:25:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.399+0000 7f806101b700 1 -- start start 2024-03-18T20:25:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.400+0000 7f806101b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8044003a00 0x7f805c0ae910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.400+0000 7f806101b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f805c0b0530 0x7f805c0b09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.400+0000 7f806101b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f805c0b2fb0 0x7f805c0b3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.400+0000 7f806101b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c0b0ef0 con 0x7f805c0b0530 2024-03-18T20:25:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.400+0000 7f806101b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c0b1c70 con 0x7f805c0b2fb0 2024-03-18T20:25:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.400+0000 7f806101b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f805c0b10f0 con 0x7f8044003a00 2024-03-18T20:25:27.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.400+0000 7f805a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f805c0b0530 0x7f805c0b09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.400+0000 7f805a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f805c0b0530 0x7f805c0b09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59894/0 (socket says 172.21.15.1:59894) 2024-03-18T20:25:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.400+0000 7f805a59c700 1 -- 172.21.15.1:0/2239335566 learned_addr learned my addr 172.21.15.1:0/2239335566 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.400+0000 7f805b59e700 1 --2- 172.21.15.1:0/2239335566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f805c0b2fb0 0x7f805c0b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.400+0000 7f805ad9d700 1 --2- 172.21.15.1:0/2239335566 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8044003a00 0x7f805c0ae910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.400+0000 7f805a59c700 1 -- 172.21.15.1:0/2239335566 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8044003a00 msgr2=0x7f805c0ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.400+0000 7f805a59c700 1 --2- 172.21.15.1:0/2239335566 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8044003a00 0x7f805c0ae910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.400+0000 7f805a59c700 1 -- 172.21.15.1:0/2239335566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f805c0b2fb0 msgr2=0x7f805c0b3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.400+0000 7f805a59c700 1 --2- 172.21.15.1:0/2239335566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f805c0b2fb0 0x7f805c0b3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.400+0000 7f805a59c700 1 -- 172.21.15.1:0/2239335566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805c0b13d0 con 0x7f805c0b0530 2024-03-18T20:25:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.401+0000 7f805a59c700 1 --2- 172.21.15.1:0/2239335566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f805c0b0530 0x7f805c0b09b0 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f804c010920 tx=0x7f804c010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.401+0000 7f80437fe700 1 -- 172.21.15.1:0/2239335566 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c00e850 con 0x7f805c0b0530 2024-03-18T20:25:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.401+0000 7f80437fe700 1 -- 172.21.15.1:0/2239335566 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804c00e9b0 con 0x7f805c0b0530 2024-03-18T20:25:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.401+0000 7f806101b700 1 -- 172.21.15.1:0/2239335566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f805c0b16b0 con 0x7f805c0b0530 2024-03-18T20:25:27.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.401+0000 7f80437fe700 1 -- 172.21.15.1:0/2239335566 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c00ed60 con 0x7f805c0b0530 2024-03-18T20:25:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.401+0000 7f806101b700 1 -- 172.21.15.1:0/2239335566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f805c15bb50 con 0x7f805c0b0530 2024-03-18T20:25:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.402+0000 7f806101b700 1 -- 172.21.15.1:0/2239335566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f805c0b4e60 con 0x7f805c0b0530 2024-03-18T20:25:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.402+0000 7f80437fe700 1 -- 172.21.15.1:0/2239335566 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f804c016460 con 0x7f805c0b0530 2024-03-18T20:25:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.402+0000 7f80437fe700 1 --2- 172.21.15.1:0/2239335566 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f804806adb0 0x7f804806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.403+0000 7f80437fe700 1 -- 172.21.15.1:0/2239335566 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f804c089560 con 0x7f805c0b0530 2024-03-18T20:25:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.403+0000 7f805ad9d700 1 --2- 172.21.15.1:0/2239335566 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f804806adb0 0x7f804806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.403+0000 7f805ad9d700 1 --2- 172.21.15.1:0/2239335566 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f804806adb0 0x7f804806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8050008fd0 tx=0x7f805000a000 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:27.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.407+0000 7f80437fe700 1 -- 172.21.15.1:0/2239335566 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f804c0543f0 con 0x7f805c0b0530 2024-03-18T20:25:27.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.693+0000 7f806101b700 1 -- 172.21.15.1:0/2239335566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f805c06d510 con 0x7f805c0b0530 2024-03-18T20:25:27.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.696+0000 7f80437fe700 1 -- 172.21.15.1:0/2239335566 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v50) v1 ==== 190+0+0 (secure 0 0 0) 0x7f804c057a10 con 0x7f805c0b0530 2024-03-18T20:25:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.698+0000 7f806101b700 1 -- 172.21.15.1:0/2239335566 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f804806adb0 msgr2=0x7f804806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.698+0000 7f806101b700 1 --2- 172.21.15.1:0/2239335566 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f804806adb0 0x7f804806d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8050008fd0 tx=0x7f805000a000 comp rx=0 tx=0).stop 2024-03-18T20:25:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.698+0000 7f806101b700 1 -- 172.21.15.1:0/2239335566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f805c0b0530 msgr2=0x7f805c0b09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.698+0000 7f806101b700 1 --2- 172.21.15.1:0/2239335566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f805c0b0530 0x7f805c0b09b0 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f804c010920 tx=0x7f804c010ce0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.698+0000 7f806101b700 1 -- 172.21.15.1:0/2239335566 shutdown_connections 2024-03-18T20:25:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.698+0000 7f806101b700 1 --2- 172.21.15.1:0/2239335566 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f804806adb0 0x7f804806d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.698+0000 7f806101b700 1 --2- 172.21.15.1:0/2239335566 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8044003a00 0x7f805c0ae910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.698+0000 7f806101b700 1 --2- 172.21.15.1:0/2239335566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f805c0b2fb0 0x7f805c0b3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.698+0000 7f806101b700 1 --2- 172.21.15.1:0/2239335566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f805c0b0530 0x7f805c0b09b0 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.698+0000 7f806101b700 1 -- 172.21.15.1:0/2239335566 >> 172.21.15.1:0/2239335566 conn(0x7f805c0b2bd0 msgr2=0x7f805c0c6d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.699+0000 7f806101b700 1 -- 172.21.15.1:0/2239335566 shutdown_connections 2024-03-18T20:25:27.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.699+0000 7f806101b700 1 -- 172.21.15.1:0/2239335566 wait complete. 2024-03-18T20:25:27.704 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-18T20:25:27.715 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:25:27.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.835+0000 7f8276ce8700 1 Processor -- start 2024-03-18T20:25:27.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.835+0000 7f8276ce8700 1 -- start start 2024-03-18T20:25:27.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.836+0000 7f8276ce8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8270099ea0 0x7f82700cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:27.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.836+0000 7f8276ce8700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82700d0c80 con 0x7f8270099a40 2024-03-18T20:25:27.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.836+0000 7f8276ce8700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82700d09a0 con 0x7f827015d680 2024-03-18T20:25:27.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.836+0000 7f8276ce8700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82700cb830 con 0x7f8270099ea0 2024-03-18T20:25:27.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.836+0000 7f8275285700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8270099ea0 0x7f82700cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:27.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.836+0000 7f826ffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8270099a40 0x7f82700cc310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39534/0 (socket says 172.21.15.1:39534) 2024-03-18T20:25:27.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.836+0000 7f826ffff700 1 -- 172.21.15.1:0/3163764358 learned_addr learned my addr 172.21.15.1:0/3163764358 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.837+0000 7f8275285700 1 -- 172.21.15.1:0/3163764358 >> v1:172.21.15.165:6789/0 conn(0x7f827015d680 legacy=0x7f82700cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.837+0000 7f8275285700 1 -- 172.21.15.1:0/3163764358 >> v1:172.21.15.1:6789/0 conn(0x7f8270099a40 legacy=0x7f82700cc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.837+0000 7f826f7fe700 1 -- 172.21.15.1:0/3163764358 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 505360506 0 0) 0x7f82700d0c80 con 0x7f8270099a40 2024-03-18T20:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.837+0000 7f8275285700 1 -- 172.21.15.1:0/3163764358 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82700cbb10 con 0x7f8270099ea0 2024-03-18T20:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.837+0000 7f8275285700 1 --2- 172.21.15.1:0/3163764358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8270099ea0 0x7f82700cca20 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7f8268009d90 tx=0x7f826800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a24243e54fe882d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.837+0000 7f826f7fe700 1 -- 172.21.15.1:0/3163764358 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826800e070 con 0x7f8270099ea0 2024-03-18T20:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.837+0000 7f826f7fe700 1 -- 172.21.15.1:0/3163764358 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826800bb90 con 0x7f8270099ea0 2024-03-18T20:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.837+0000 7f826f7fe700 1 -- 172.21.15.1:0/3163764358 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8268003a20 con 0x7f8270099ea0 2024-03-18T20:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.838+0000 7f8276ce8700 1 -- 172.21.15.1:0/3163764358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8270099ea0 msgr2=0x7f82700cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.838+0000 7f8276ce8700 1 --2- 172.21.15.1:0/3163764358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8270099ea0 0x7f82700cca20 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7f8268009d90 tx=0x7f826800b330 comp rx=0 tx=0).stop 2024-03-18T20:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.838+0000 7f8276ce8700 1 -- 172.21.15.1:0/3163764358 shutdown_connections 2024-03-18T20:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.838+0000 7f8276ce8700 1 --2- 172.21.15.1:0/3163764358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8270099ea0 0x7f82700cca20 unknown :-1 s=CLOSED pgs=2530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.838+0000 7f8276ce8700 1 -- 172.21.15.1:0/3163764358 >> 172.21.15.1:0/3163764358 conn(0x7f82700b2ba0 msgr2=0x7f82700b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:27.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.838+0000 7f8276ce8700 1 -- 172.21.15.1:0/3163764358 shutdown_connections 2024-03-18T20:25:27.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.838+0000 7f8276ce8700 1 -- 172.21.15.1:0/3163764358 wait complete. 2024-03-18T20:25:27.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.839+0000 7f8276ce8700 1 Processor -- start 2024-03-18T20:25:27.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.839+0000 7f8276ce8700 1 -- start start 2024-03-18T20:25:27.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.840+0000 7f8276ce8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8270099a40 0x7f82701f2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:27.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.840+0000 7f8276ce8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8270099ea0 0x7f82700c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:27.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.840+0000 7f8276ce8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f827015d680 0x7f82700cac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:27.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.840+0000 7f8276ce8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82700c90d0 con 0x7f8270099a40 2024-03-18T20:25:27.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.840+0000 7f8276ce8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82700c9960 con 0x7f827015d680 2024-03-18T20:25:27.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.840+0000 7f8276ce8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82700c9bb0 con 0x7f8270099ea0 2024-03-18T20:25:27.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.840+0000 7f8274a84700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8270099a40 0x7f82701f2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:27.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.840+0000 7f8274a84700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8270099a40 0x7f82701f2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59906/0 (socket says 172.21.15.1:59906) 2024-03-18T20:25:27.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.840+0000 7f8274a84700 1 -- 172.21.15.1:0/1640879172 learned_addr learned my addr 172.21.15.1:0/1640879172 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.840+0000 7f8274a84700 1 -- 172.21.15.1:0/1640879172 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8270099ea0 msgr2=0x7f82700c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.840+0000 7f8274a84700 1 --2- 172.21.15.1:0/1640879172 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8270099ea0 0x7f82700c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.840+0000 7f8274a84700 1 -- 172.21.15.1:0/1640879172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f827015d680 msgr2=0x7f82700cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.840+0000 7f8274a84700 1 --2- 172.21.15.1:0/1640879172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f827015d680 0x7f82700cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.840+0000 7f8274a84700 1 -- 172.21.15.1:0/1640879172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8268009a70 con 0x7f8270099a40 2024-03-18T20:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.840+0000 7f8274a84700 1 --2- 172.21.15.1:0/1640879172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8270099a40 0x7f82701f2950 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f826400b4a0 tx=0x7f826400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.841+0000 7f826d7fa700 1 -- 172.21.15.1:0/1640879172 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264011830 con 0x7f8270099a40 2024-03-18T20:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.841+0000 7f826d7fa700 1 -- 172.21.15.1:0/1640879172 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8264011990 con 0x7f8270099a40 2024-03-18T20:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.841+0000 7f8276ce8700 1 -- 172.21.15.1:0/1640879172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82700c9e90 con 0x7f8270099a40 2024-03-18T20:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.841+0000 7f826d7fa700 1 -- 172.21.15.1:0/1640879172 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264011d40 con 0x7f8270099a40 2024-03-18T20:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.841+0000 7f8276ce8700 1 -- 172.21.15.1:0/1640879172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82700ca450 con 0x7f8270099a40 2024-03-18T20:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.842+0000 7f8276ce8700 1 -- 172.21.15.1:0/1640879172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82700676a0 con 0x7f8270099a40 2024-03-18T20:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.842+0000 7f826d7fa700 1 -- 172.21.15.1:0/1640879172 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f826401a400 con 0x7f8270099a40 2024-03-18T20:25:27.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.842+0000 7f826d7fa700 1 --2- 172.21.15.1:0/1640879172 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f825406adb0 0x7f825406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.843+0000 7f826d7fa700 1 -- 172.21.15.1:0/1640879172 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f8264089340 con 0x7f8270099a40 2024-03-18T20:25:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.843+0000 7f826ffff700 1 --2- 172.21.15.1:0/1640879172 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f825406adb0 0x7f825406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:27.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.843+0000 7f826ffff700 1 --2- 172.21.15.1:0/1640879172 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f825406adb0 0x7f825406d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f825c0011e0 tx=0x7f825c0079b0 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:27.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:27.847+0000 7f826d7fa700 1 -- 172.21.15.1:0/1640879172 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f82640542f0 con 0x7f8270099a40 2024-03-18T20:25:28.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.097+0000 7f8276ce8700 1 -- 172.21.15.1:0/1640879172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82700cf650 con 0x7f8270099a40 2024-03-18T20:25:28.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.098+0000 7f826d7fa700 1 -- 172.21.15.1:0/1640879172 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4747 (secure 0 0 0) 0x7f8264057910 con 0x7f8270099a40 2024-03-18T20:25:28.102 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:28.102 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":104,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10369,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/677258734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":677258734},{"type":"v1","addr":"172.21.15.165:6835","nonce":677258734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":11544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2018398457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2018398457},{"type":"v1","addr":"172.21.15.1:6836","nonce":2018398457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":11547,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1635684321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1635684321},{"type":"v1","addr":"172.21.15.1:6837","nonce":1635684321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:25:22.663717+0000","modified":"2024-03-18T20:25:25.668584+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11556":{"gid":11556,"name":"d","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6834/4024828630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4024828630},{"type":"v1","addr":"172.21.15.165:6834","nonce":4024828630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-03-18T20:25:28.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.100+0000 7f8276ce8700 1 -- 172.21.15.1:0/1640879172 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f825406adb0 msgr2=0x7f825406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.100+0000 7f8276ce8700 1 --2- 172.21.15.1:0/1640879172 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f825406adb0 0x7f825406d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f825c0011e0 tx=0x7f825c0079b0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.100+0000 7f8276ce8700 1 -- 172.21.15.1:0/1640879172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8270099a40 msgr2=0x7f82701f2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.100+0000 7f8276ce8700 1 --2- 172.21.15.1:0/1640879172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8270099a40 0x7f82701f2950 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f826400b4a0 tx=0x7f826400b860 comp rx=0 tx=0).stop 2024-03-18T20:25:28.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.100+0000 7f8276ce8700 1 -- 172.21.15.1:0/1640879172 shutdown_connections 2024-03-18T20:25:28.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.100+0000 7f8276ce8700 1 --2- 172.21.15.1:0/1640879172 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f825406adb0 0x7f825406d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.100+0000 7f8276ce8700 1 --2- 172.21.15.1:0/1640879172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f827015d680 0x7f82700cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.100+0000 7f8276ce8700 1 --2- 172.21.15.1:0/1640879172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8270099a40 0x7f82701f2950 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.100+0000 7f8276ce8700 1 --2- 172.21.15.1:0/1640879172 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8270099ea0 0x7f82700c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.100+0000 7f8276ce8700 1 -- 172.21.15.1:0/1640879172 >> 172.21.15.1:0/1640879172 conn(0x7f82700b2ba0 msgr2=0x7f82700b1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:28.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.100+0000 7f8276ce8700 1 -- 172.21.15.1:0/1640879172 shutdown_connections 2024-03-18T20:25:28.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.100+0000 7f8276ce8700 1 -- 172.21.15.1:0/1640879172 wait complete. 2024-03-18T20:25:28.106 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 104 2024-03-18T20:25:28.519 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T20:25:22.663717+0000', 'modified': '2024-03-18T20:25:25.668584+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11556}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11556': {'gid': 11556, 'name': 'd', 'rank': 0, 'incarnation': 101, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.165:6834/4024828630', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6832', 'nonce': 4024828630}, {'type': 'v1', 'addr': '172.21.15.165:6834', 'nonce': 4024828630}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T20:25:28.519 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T20:25:28.519 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-18T20:25:28.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.650+0000 7f4c3b643700 1 Processor -- start 2024-03-18T20:25:28.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.650+0000 7f4c3b643700 1 -- start start 2024-03-18T20:25:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.651+0000 7f4c3b643700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c34070e40 0x7f4c34071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.651+0000 7f4c3b643700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c3406e200 0x7f4c3406cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.651+0000 7f4c3b643700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4c3406e680 0x7f4c3406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.651+0000 7f4c3b643700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c34071760 con 0x7f4c3406e200 2024-03-18T20:25:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.651+0000 7f4c3b643700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c3415df40 con 0x7f4c3406e680 2024-03-18T20:25:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.651+0000 7f4c3b643700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3415e220 con 0x7f4c34070e40 2024-03-18T20:25:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.651+0000 7f4c38bde700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c3406e200 0x7f4c3406cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.651+0000 7f4c38bde700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c3406e200 0x7f4c3406cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59916/0 (socket says 172.21.15.1:59916) 2024-03-18T20:25:28.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.651+0000 7f4c38bde700 1 -- 172.21.15.1:0/97355136 learned_addr learned my addr 172.21.15.1:0/97355136 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:28.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.651+0000 7f4c39be0700 1 --2- 172.21.15.1:0/97355136 >> v2:172.21.15.165:3300/0 conn(0x7f4c3406e680 0x7f4c3406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.651+0000 7f4c393df700 1 --2- 172.21.15.1:0/97355136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c34070e40 0x7f4c34071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.652+0000 7f4c38bde700 1 -- 172.21.15.1:0/97355136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c34070e40 msgr2=0x7f4c34071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:28.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.652+0000 7f4c38bde700 1 --2- 172.21.15.1:0/97355136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c34070e40 0x7f4c34071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.652+0000 7f4c38bde700 1 -- 172.21.15.1:0/97355136 >> v2:172.21.15.165:3300/0 conn(0x7f4c3406e680 msgr2=0x7f4c3406d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:28.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.652+0000 7f4c38bde700 1 --2- 172.21.15.1:0/97355136 >> v2:172.21.15.165:3300/0 conn(0x7f4c3406e680 0x7f4c3406d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.652+0000 7f4c38bde700 1 -- 172.21.15.1:0/97355136 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3415e500 con 0x7f4c3406e200 2024-03-18T20:25:28.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.652+0000 7f4c38bde700 1 --2- 172.21.15.1:0/97355136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3406e200 0x7f4c3406cea0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f4c3406b2b0 tx=0x7f4c30009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff2bb2301ed42b66 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:28.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.652+0000 7f4c2bfff700 1 -- 172.21.15.1:0/97355136 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c30014e90 con 0x7f4c3406e200 2024-03-18T20:25:28.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.652+0000 7f4c2bfff700 1 -- 172.21.15.1:0/97355136 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3001c440 con 0x7f4c3406e200 2024-03-18T20:25:28.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.652+0000 7f4c2bfff700 1 -- 172.21.15.1:0/97355136 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3001c7d0 con 0x7f4c3406e200 2024-03-18T20:25:28.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.653+0000 7f4c3b643700 1 -- 172.21.15.1:0/97355136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3406e200 msgr2=0x7f4c3406cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.653+0000 7f4c3b643700 1 --2- 172.21.15.1:0/97355136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3406e200 0x7f4c3406cea0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f4c3406b2b0 tx=0x7f4c30009bd0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.653+0000 7f4c3b643700 1 -- 172.21.15.1:0/97355136 shutdown_connections 2024-03-18T20:25:28.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.653+0000 7f4c3b643700 1 --2- 172.21.15.1:0/97355136 >> v2:172.21.15.165:3300/0 conn(0x7f4c3406e680 0x7f4c3406d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.653+0000 7f4c3b643700 1 --2- 172.21.15.1:0/97355136 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c34070e40 0x7f4c34071220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.653+0000 7f4c3b643700 1 --2- 172.21.15.1:0/97355136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3406e200 0x7f4c3406cea0 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.653+0000 7f4c3b643700 1 -- 172.21.15.1:0/97355136 >> 172.21.15.1:0/97355136 conn(0x7f4c3406a750 msgr2=0x7f4c3406ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:28.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.653+0000 7f4c3b643700 1 -- 172.21.15.1:0/97355136 shutdown_connections 2024-03-18T20:25:28.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.653+0000 7f4c3b643700 1 -- 172.21.15.1:0/97355136 wait complete. 2024-03-18T20:25:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.654+0000 7f4c3b643700 1 Processor -- start 2024-03-18T20:25:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.654+0000 7f4c3b643700 1 -- start start 2024-03-18T20:25:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c3b643700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c3406e200 0x7f4c3414b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c3b643700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3406e680 0x7f4c34159690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c3b643700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c34070e40 0x7f4c34153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c3b643700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c34152f90 con 0x7f4c3406e680 2024-03-18T20:25:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c3b643700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c34153b10 con 0x7f4c3406e200 2024-03-18T20:25:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c3b643700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c341531e0 con 0x7f4c34070e40 2024-03-18T20:25:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c38bde700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3406e680 0x7f4c34159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c38bde700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3406e680 0x7f4c34159690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59918/0 (socket says 172.21.15.1:59918) 2024-03-18T20:25:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c38bde700 1 -- 172.21.15.1:0/175561285 learned_addr learned my addr 172.21.15.1:0/175561285 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c393df700 1 --2- 172.21.15.1:0/175561285 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c3406e200 0x7f4c3414b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c39be0700 1 --2- 172.21.15.1:0/175561285 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c34070e40 0x7f4c34153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c38bde700 1 -- 172.21.15.1:0/175561285 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c34070e40 msgr2=0x7f4c34153fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c38bde700 1 --2- 172.21.15.1:0/175561285 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c34070e40 0x7f4c34153fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c38bde700 1 -- 172.21.15.1:0/175561285 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c3406e200 msgr2=0x7f4c3414b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c38bde700 1 --2- 172.21.15.1:0/175561285 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c3406e200 0x7f4c3414b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c38bde700 1 -- 172.21.15.1:0/175561285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c30009550 con 0x7f4c3406e680 2024-03-18T20:25:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c39be0700 1 --2- 172.21.15.1:0/175561285 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c34070e40 0x7f4c34153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c38bde700 1 --2- 172.21.15.1:0/175561285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3406e680 0x7f4c34159690 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7f4c3001f040 tx=0x7f4c30004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c29ffb700 1 -- 172.21.15.1:0/175561285 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c30003b70 con 0x7f4c3406e680 2024-03-18T20:25:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.655+0000 7f4c29ffb700 1 -- 172.21.15.1:0/175561285 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c30003cf0 con 0x7f4c3406e680 2024-03-18T20:25:28.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.656+0000 7f4c3b643700 1 -- 172.21.15.1:0/175561285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c341534c0 con 0x7f4c3406e680 2024-03-18T20:25:28.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.656+0000 7f4c29ffb700 1 -- 172.21.15.1:0/175561285 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3001c7f0 con 0x7f4c3406e680 2024-03-18T20:25:28.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.656+0000 7f4c3b643700 1 -- 172.21.15.1:0/175561285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c34159bd0 con 0x7f4c3406e680 2024-03-18T20:25:28.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.657+0000 7f4c29ffb700 1 -- 172.21.15.1:0/175561285 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4c30023070 con 0x7f4c3406e680 2024-03-18T20:25:28.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.658+0000 7f4c29ffb700 1 --2- 172.21.15.1:0/175561285 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c1806adb0 0x7f4c1806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.658+0000 7f4c29ffb700 1 -- 172.21.15.1:0/175561285 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f4c300932d0 con 0x7f4c3406e680 2024-03-18T20:25:28.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.658+0000 7f4c393df700 1 --2- 172.21.15.1:0/175561285 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c1806adb0 0x7f4c1806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.658+0000 7f4c393df700 1 --2- 172.21.15.1:0/175561285 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c1806adb0 0x7f4c1806d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4c340ae620 tx=0x7f4c24008040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:28.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.660+0000 7f4c477ecb80 1 Processor -- start 2024-03-18T20:25:28.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.660+0000 7f4c477ecb80 1 -- start start 2024-03-18T20:25:28.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.660+0000 7f4c477ecb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 0x5570b289dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.660+0000 7f4c477ecb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b286f0a0 0x5570b28ba580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.660+0000 7f4c477ecb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5570b28af630 0x5570b2872160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.660+0000 7f4c477ecb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570b27e8f20 con 0x5570b289dc10 2024-03-18T20:25:28.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.660+0000 7f4c477ecb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570b27cac30 con 0x5570b286f0a0 2024-03-18T20:25:28.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.660+0000 7f4c477ecb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x5570b27c8430 con 0x5570b28af630 2024-03-18T20:25:28.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.661+0000 7f4c393df700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 0x5570b289dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.661+0000 7f4c393df700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 0x5570b289dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59930/0 (socket says 172.21.15.1:59930) 2024-03-18T20:25:28.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.661+0000 7f4c393df700 1 -- 172.21.15.1:0/963258415 learned_addr learned my addr 172.21.15.1:0/963258415 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:28.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.661+0000 7f4c39be0700 1 --2- 172.21.15.1:0/963258415 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5570b28af630 0x5570b2872160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.661+0000 7f4c393df700 1 -- 172.21.15.1:0/963258415 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5570b28af630 msgr2=0x5570b2872160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:28.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.661+0000 7f4c393df700 1 --2- 172.21.15.1:0/963258415 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5570b28af630 0x5570b2872160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.661+0000 7f4c393df700 1 -- 172.21.15.1:0/963258415 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b286f0a0 msgr2=0x5570b28ba580 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T20:25:28.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.661+0000 7f4c393df700 1 --2- 172.21.15.1:0/963258415 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b286f0a0 0x5570b28ba580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.661+0000 7f4c393df700 1 -- 172.21.15.1:0/963258415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570b25b7790 con 0x5570b289dc10 2024-03-18T20:25:28.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.661+0000 7f4c393df700 1 --2- 172.21.15.1:0/963258415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 0x5570b289dff0 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f4c240097d0 tx=0x7f4c24014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f79f292e062af9d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:28.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.661+0000 7f4c1effd700 1 -- 172.21.15.1:0/963258415 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24014070 con 0x5570b289dc10 2024-03-18T20:25:28.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.661+0000 7f4c1effd700 1 -- 172.21.15.1:0/963258415 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c24004020 con 0x5570b289dc10 2024-03-18T20:25:28.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.661+0000 7f4c1effd700 1 -- 172.21.15.1:0/963258415 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24004390 con 0x5570b289dc10 2024-03-18T20:25:28.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.662+0000 7f4c477ecb80 1 -- 172.21.15.1:0/963258415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 msgr2=0x5570b289dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.662+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/963258415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 0x5570b289dff0 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f4c240097d0 tx=0x7f4c24014000 comp rx=0 tx=0).stop 2024-03-18T20:25:28.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.662+0000 7f4c477ecb80 1 -- 172.21.15.1:0/963258415 shutdown_connections 2024-03-18T20:25:28.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.662+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/963258415 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5570b28af630 0x5570b2872160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.662+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/963258415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 0x5570b289dff0 unknown :-1 s=CLOSED pgs=2534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.662+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/963258415 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b286f0a0 0x5570b28ba580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.662+0000 7f4c477ecb80 1 -- 172.21.15.1:0/963258415 >> 172.21.15.1:0/963258415 conn(0x5570b27d1cf0 msgr2=0x5570b2871a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:28.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.662+0000 7f4c477ecb80 1 -- 172.21.15.1:0/963258415 shutdown_connections 2024-03-18T20:25:28.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.662+0000 7f4c477ecb80 1 -- 172.21.15.1:0/963258415 wait complete. 2024-03-18T20:25:28.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.662+0000 7f4c477ecb80 1 -- start start 2024-03-18T20:25:28.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.662+0000 7f4c477ecb80 1 Processor -- start 2024-03-18T20:25:28.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.663+0000 7f4c477ecb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5570b286f0a0 0x5570b28926c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.663+0000 7f4c477ecb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 0x5570b289f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.663+0000 7f4c477ecb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b28af630 0x5570b2872c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.663+0000 7f4c477ecb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570b27c3d20 con 0x5570b289dc10 2024-03-18T20:25:28.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.663+0000 7f4c477ecb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570b275f770 con 0x5570b28af630 2024-03-18T20:25:28.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.663+0000 7f4c477ecb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x5570b27826f0 con 0x5570b286f0a0 2024-03-18T20:25:28.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.663+0000 7f4c477ecb80 10 client.0 _ll_register_callbacks cb 0x5570b25cb5a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:25:28.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.663+0000 7f4c38bde700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 0x5570b289f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.663+0000 7f4c38bde700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 0x5570b289f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59946/0 (socket says 172.21.15.1:59946) 2024-03-18T20:25:28.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.663+0000 7f4c38bde700 1 -- 172.21.15.1:0/3102421356 learned_addr learned my addr 172.21.15.1:0/3102421356 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:28.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c39be0700 1 --2- 172.21.15.1:0/3102421356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b28af630 0x5570b2872c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c38bde700 1 -- 172.21.15.1:0/3102421356 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5570b286f0a0 msgr2=0x5570b28926c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T20:25:28.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c38bde700 1 --2- 172.21.15.1:0/3102421356 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5570b286f0a0 0x5570b28926c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c38bde700 1 -- 172.21.15.1:0/3102421356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b28af630 msgr2=0x5570b2872c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:28.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c38bde700 1 --2- 172.21.15.1:0/3102421356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b28af630 0x5570b2872c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c38bde700 1 -- 172.21.15.1:0/3102421356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2400e040 con 0x5570b289dc10 2024-03-18T20:25:28.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c39be0700 1 --2- 172.21.15.1:0/3102421356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b28af630 0x5570b2872c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:28.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c38bde700 1 --2- 172.21.15.1:0/3102421356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 0x5570b289f090 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f4c30061ff0 tx=0x7f4c30003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8bfb505210f40da server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:28.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c13fff700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T20:25:28.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c13fff700 1 -- 172.21.15.1:0/3102421356 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c30075ea0 con 0x5570b289dc10 2024-03-18T20:25:28.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c13fff700 1 -- 172.21.15.1:0/3102421356 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(232..232 src has 1..232) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f4c30062060 con 0x5570b289dc10 2024-03-18T20:25:28.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c477ecb80 1 -- 172.21.15.1:0/3102421356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5570b2865e90 con 0x5570b289dc10 2024-03-18T20:25:28.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c13fff700 1 -- 172.21.15.1:0/3102421356 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c30065e90 con 0x5570b289dc10 2024-03-18T20:25:28.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c477ecb80 10 client.11700 fetch_fsmap learned FSMap version 104 2024-03-18T20:25:28.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c477ecb80 1 -- 172.21.15.1:0/3102421356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=2+}) v3 -- 0x5570b26a40a0 con 0x5570b289dc10 2024-03-18T20:25:28.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c13fff700 1 -- 172.21.15.1:0/3102421356 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5570b2865e90 con 0x5570b289dc10 2024-03-18T20:25:28.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c13fff700 1 -- 172.21.15.1:0/3102421356 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 104) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4c300622c0 con 0x5570b289dc10 2024-03-18T20:25:28.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c477ecb80 10 client.11700 fetch_fsmap finished waiting for FSMap version 104 2024-03-18T20:25:28.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c477ecb80 10 client.11700 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-18T20:25:28.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.664+0000 7f4c477ecb80 20 client.11700 populate_metadata read hostname 'smithi001' 2024-03-18T20:25:28.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.665+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/3102421356 >> [v2:172.21.15.165:6832/4024828630,v1:172.21.15.165:6834/4024828630] conn(0x5570b28b0c00 0x5570b28b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.665+0000 7f4c477ecb80 4 client.11700 mds_command: new command op to 11556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T20:25:28.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.665+0000 7f4c477ecb80 1 -- 172.21.15.1:0/3102421356 --> [v2:172.21.15.165:6832/4024828630,v1:172.21.15.165:6834/4024828630] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5570b2843010 con 0x5570b28b0c00 2024-03-18T20:25:28.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.666+0000 7f4c393df700 1 --2- 172.21.15.1:0/3102421356 >> [v2:172.21.15.165:6832/4024828630,v1:172.21.15.165:6834/4024828630] conn(0x5570b28b0c00 0x5570b28b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.667+0000 7f4c393df700 1 --2- 172.21.15.1:0/3102421356 >> [v2:172.21.15.165:6832/4024828630,v1:172.21.15.165:6834/4024828630] conn(0x5570b28b0c00 0x5570b28b0fe0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51eb3de60eadf11c server_cookie=b764473f5374e029 in_seq=0 out_seq=0 2024-03-18T20:25:28.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.667+0000 7f4c13fff700 10 client.11700 ms_handle_connect on v2:172.21.15.165:6832/4024828630 2024-03-18T20:25:28.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.668+0000 7f4c13fff700 1 -- 172.21.15.1:0/3102421356 <== mds.0 v2:172.21.15.165:6832/4024828630 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5570b2843010 con 0x5570b28b0c00 2024-03-18T20:25:28.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.668+0000 7f4c13fff700 10 client.11700 handle_command_reply: tid=0 2024-03-18T20:25:28.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.668+0000 7f4c477ecb80 1 client.11700 shutdown 2024-03-18T20:25:28.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.668+0000 7f4c477ecb80 1 -- 172.21.15.1:0/3102421356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 msgr2=0x5570b289f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.668+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/3102421356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 0x5570b289f090 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f4c30061ff0 tx=0x7f4c30003f90 comp rx=0 tx=0).stop 2024-03-18T20:25:28.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.668+0000 7f4c477ecb80 1 -- 172.21.15.1:0/3102421356 shutdown_connections 2024-03-18T20:25:28.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.668+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/3102421356 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5570b286f0a0 0x5570b28926c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.668+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/3102421356 >> [v2:172.21.15.165:6832/4024828630,v1:172.21.15.165:6834/4024828630] conn(0x5570b28b0c00 0x5570b28b0fe0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.668+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/3102421356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b28af630 0x5570b2872c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.668+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/3102421356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 0x5570b289f090 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.668+0000 7f4c13fff700 0 client.11700 ms_handle_reset on v2:172.21.15.165:6832/4024828630 2024-03-18T20:25:28.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.668+0000 7f4c477ecb80 1 -- 172.21.15.1:0/3102421356 >> 172.21.15.1:0/3102421356 conn(0x5570b27d1cf0 msgr2=0x5570b289b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:28.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.668+0000 7f4c477ecb80 1 -- 172.21.15.1:0/3102421356 shutdown_connections 2024-03-18T20:25:28.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.668+0000 7f4c477ecb80 1 -- 172.21.15.1:0/3102421356 wait complete. 2024-03-18T20:25:28.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.669+0000 7f4c477ecb80 20 client.11700 trim_cache size 0 max 16384 2024-03-18T20:25:28.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.686+0000 7f4c477ecb80 1 Processor -- start 2024-03-18T20:25:28.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.686+0000 7f4c477ecb80 1 -- start start 2024-03-18T20:25:28.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.687+0000 7f4c477ecb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b286f0a0 0x5570b286db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.687+0000 7f4c477ecb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5570b289dc10 0x5570b289efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.687+0000 7f4c477ecb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b28af630 0x5570b289e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.687+0000 7f4c477ecb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570b25b8390 con 0x5570b286f0a0 2024-03-18T20:25:28.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.687+0000 7f4c477ecb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570b28cb3d0 con 0x5570b28af630 2024-03-18T20:25:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.687+0000 7f4c477ecb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x5570b2892920 con 0x5570b289dc10 2024-03-18T20:25:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.687+0000 7f4c393df700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b286f0a0 0x5570b286db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.687+0000 7f4c393df700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b286f0a0 0x5570b286db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59958/0 (socket says 172.21.15.1:59958) 2024-03-18T20:25:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.687+0000 7f4c393df700 1 -- 172.21.15.1:0/1517535157 learned_addr learned my addr 172.21.15.1:0/1517535157 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.687+0000 7f4c39be0700 1 --2- 172.21.15.1:0/1517535157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b28af630 0x5570b289e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.687+0000 7f4c38bde700 1 --2- 172.21.15.1:0/1517535157 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5570b289dc10 0x5570b289efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.687+0000 7f4c393df700 1 -- 172.21.15.1:0/1517535157 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5570b289dc10 msgr2=0x5570b289efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.687+0000 7f4c393df700 1 --2- 172.21.15.1:0/1517535157 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5570b289dc10 0x5570b289efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.687+0000 7f4c393df700 1 -- 172.21.15.1:0/1517535157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b28af630 msgr2=0x5570b289e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.687+0000 7f4c393df700 1 --2- 172.21.15.1:0/1517535157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b28af630 0x5570b289e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.687+0000 7f4c393df700 1 -- 172.21.15.1:0/1517535157 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570b278a010 con 0x5570b286f0a0 2024-03-18T20:25:28.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.687+0000 7f4c39be0700 1 --2- 172.21.15.1:0/1517535157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b28af630 0x5570b289e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:28.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.688+0000 7f4c393df700 1 --2- 172.21.15.1:0/1517535157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b286f0a0 0x5570b286db40 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f4c2400f4e0 tx=0x7f4c240047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b9370944b41c28 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:28.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.688+0000 7f4c1f7fe700 1 -- 172.21.15.1:0/1517535157 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24014040 con 0x5570b286f0a0 2024-03-18T20:25:28.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.688+0000 7f4c1f7fe700 1 -- 172.21.15.1:0/1517535157 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c24030950 con 0x5570b286f0a0 2024-03-18T20:25:28.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.688+0000 7f4c1f7fe700 1 -- 172.21.15.1:0/1517535157 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24030cc0 con 0x5570b286f0a0 2024-03-18T20:25:28.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.688+0000 7f4c477ecb80 1 -- 172.21.15.1:0/1517535157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b286f0a0 msgr2=0x5570b286db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.688+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/1517535157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b286f0a0 0x5570b286db40 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f4c2400f4e0 tx=0x7f4c240047c0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.688+0000 7f4c477ecb80 1 -- 172.21.15.1:0/1517535157 shutdown_connections 2024-03-18T20:25:28.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.688+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/1517535157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b28af630 0x5570b289e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.688+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/1517535157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b286f0a0 0x5570b286db40 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.688+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/1517535157 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5570b289dc10 0x5570b289efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.688+0000 7f4c477ecb80 1 -- 172.21.15.1:0/1517535157 >> 172.21.15.1:0/1517535157 conn(0x5570b27d1cf0 msgr2=0x5570b286fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:28.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.688+0000 7f4c477ecb80 1 -- 172.21.15.1:0/1517535157 shutdown_connections 2024-03-18T20:25:28.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.688+0000 7f4c477ecb80 1 -- 172.21.15.1:0/1517535157 wait complete. 2024-03-18T20:25:28.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.689+0000 7f4c477ecb80 1 -- start start 2024-03-18T20:25:28.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.689+0000 7f4c477ecb80 1 Processor -- start 2024-03-18T20:25:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.690+0000 7f4c477ecb80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5570b286f0a0 0x5570b2872bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.690+0000 7f4c477ecb80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 0x5570b28b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.690+0000 7f4c477ecb80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b28af630 0x5570b28b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.690+0000 7f4c477ecb80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570b28a1200 con 0x5570b289dc10 2024-03-18T20:25:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.690+0000 7f4c477ecb80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570b28b1a40 con 0x5570b28af630 2024-03-18T20:25:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.690+0000 7f4c477ecb80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x5570b28a0cd0 con 0x5570b286f0a0 2024-03-18T20:25:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.690+0000 7f4c477ecb80 10 client.0 _ll_register_callbacks cb 0x7f4c30061c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:25:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.690+0000 7f4c38bde700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 0x5570b28b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.690+0000 7f4c38bde700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 0x5570b28b02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59964/0 (socket says 172.21.15.1:59964) 2024-03-18T20:25:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.690+0000 7f4c38bde700 1 -- 172.21.15.1:0/3631871842 learned_addr learned my addr 172.21.15.1:0/3631871842 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.690+0000 7f4c39be0700 1 --2- 172.21.15.1:0/3631871842 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b28af630 0x5570b28b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.690+0000 7f4c393df700 1 --2- 172.21.15.1:0/3631871842 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5570b286f0a0 0x5570b2872bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.690+0000 7f4c38bde700 1 -- 172.21.15.1:0/3631871842 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5570b286f0a0 msgr2=0x5570b2872bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.690+0000 7f4c38bde700 1 --2- 172.21.15.1:0/3631871842 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5570b286f0a0 0x5570b2872bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.690+0000 7f4c38bde700 1 -- 172.21.15.1:0/3631871842 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b28af630 msgr2=0x5570b28b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.690+0000 7f4c38bde700 1 --2- 172.21.15.1:0/3631871842 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b28af630 0x5570b28b0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.690+0000 7f4c38bde700 1 -- 172.21.15.1:0/3631871842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5570b25b7790 con 0x5570b289dc10 2024-03-18T20:25:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.691+0000 7f4c38bde700 1 --2- 172.21.15.1:0/3631871842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 0x5570b28b02d0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f4c3006a580 tx=0x7f4c30093050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ccfe356b56ddd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.691+0000 7f4c13fff700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T20:25:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.691+0000 7f4c13fff700 1 -- 172.21.15.1:0/3631871842 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c30075ea0 con 0x5570b289dc10 2024-03-18T20:25:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.691+0000 7f4c13fff700 1 -- 172.21.15.1:0/3631871842 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(232..232 src has 1..232) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f4c30062060 con 0x5570b289dc10 2024-03-18T20:25:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.691+0000 7f4c477ecb80 1 -- 172.21.15.1:0/3631871842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c24004020 con 0x5570b289dc10 2024-03-18T20:25:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.691+0000 7f4c13fff700 1 -- 172.21.15.1:0/3631871842 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c30065e90 con 0x5570b289dc10 2024-03-18T20:25:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.691+0000 7f4c477ecb80 10 client.11706 fetch_fsmap learned FSMap version 104 2024-03-18T20:25:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.691+0000 7f4c477ecb80 1 -- 172.21.15.1:0/3631871842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=2+}) v3 -- 0x5570b26181d0 con 0x5570b289dc10 2024-03-18T20:25:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.691+0000 7f4c13fff700 1 -- 172.21.15.1:0/3631871842 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c24004020 con 0x5570b289dc10 2024-03-18T20:25:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.691+0000 7f4c13fff700 1 -- 172.21.15.1:0/3631871842 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 104) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4c300912a0 con 0x5570b289dc10 2024-03-18T20:25:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.691+0000 7f4c477ecb80 10 client.11706 fetch_fsmap finished waiting for FSMap version 104 2024-03-18T20:25:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.691+0000 7f4c477ecb80 10 client.11706 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-18T20:25:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.691+0000 7f4c477ecb80 20 client.11706 populate_metadata read hostname 'smithi001' 2024-03-18T20:25:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.691+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/3631871842 >> [v2:172.21.15.165:6832/4024828630,v1:172.21.15.165:6834/4024828630] conn(0x5570b2893500 0x5570b291f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.691+0000 7f4c477ecb80 4 client.11706 mds_command: new command op to 11556 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T20:25:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.691+0000 7f4c477ecb80 1 -- 172.21.15.1:0/3631871842 --> [v2:172.21.15.165:6832/4024828630,v1:172.21.15.165:6834/4024828630] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5570b25fd040 con 0x5570b2893500 2024-03-18T20:25:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.692+0000 7f4c393df700 1 --2- 172.21.15.1:0/3631871842 >> [v2:172.21.15.165:6832/4024828630,v1:172.21.15.165:6834/4024828630] conn(0x5570b2893500 0x5570b291f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.694+0000 7f4c13fff700 10 client.11706 ms_handle_connect on v2:172.21.15.165:6832/4024828630 2024-03-18T20:25:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.694+0000 7f4c393df700 1 --2- 172.21.15.1:0/3631871842 >> [v2:172.21.15.165:6832/4024828630,v1:172.21.15.165:6834/4024828630] conn(0x5570b2893500 0x5570b291f460 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d05723edd09692fe server_cookie=1fb4a0ad986fde79 in_seq=0 out_seq=0 2024-03-18T20:25:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.694+0000 7f4c13fff700 1 -- 172.21.15.1:0/3631871842 <== mds.0 v2:172.21.15.165:6832/4024828630 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x5570b25fd040 con 0x5570b2893500 2024-03-18T20:25:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.694+0000 7f4c13fff700 10 client.11706 handle_command_reply: tid=0 2024-03-18T20:25:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.694+0000 7f4c477ecb80 1 client.11706 shutdown 2024-03-18T20:25:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.694+0000 7f4c477ecb80 1 -- 172.21.15.1:0/3631871842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 msgr2=0x5570b28b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.694+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/3631871842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 0x5570b28b02d0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f4c3006a580 tx=0x7f4c30093050 comp rx=0 tx=0).stop 2024-03-18T20:25:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.694+0000 7f4c477ecb80 1 -- 172.21.15.1:0/3631871842 shutdown_connections 2024-03-18T20:25:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.694+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/3631871842 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x5570b286f0a0 0x5570b2872bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.694+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/3631871842 >> [v2:172.21.15.165:6832/4024828630,v1:172.21.15.165:6834/4024828630] conn(0x5570b2893500 0x5570b291f460 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.694+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/3631871842 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5570b28af630 0x5570b28b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.694+0000 7f4c477ecb80 1 --2- 172.21.15.1:0/3631871842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5570b289dc10 0x5570b28b02d0 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.694+0000 7f4c13fff700 0 client.11706 ms_handle_reset on v2:172.21.15.165:6832/4024828630 2024-03-18T20:25:28.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.694+0000 7f4c477ecb80 1 -- 172.21.15.1:0/3631871842 >> 172.21.15.1:0/3631871842 conn(0x5570b27d1cf0 msgr2=0x5570b286fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:28.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.694+0000 7f4c477ecb80 1 -- 172.21.15.1:0/3631871842 shutdown_connections 2024-03-18T20:25:28.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.695+0000 7f4c477ecb80 1 -- 172.21.15.1:0/3631871842 wait complete. 2024-03-18T20:25:28.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.695+0000 7f4c477ecb80 20 client.11706 trim_cache size 0 max 16384 2024-03-18T20:25:28.704 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:25:28.704 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "9c1c697f-8655-4ef3-8a13-da568d2cbe86", 2024-03-18T20:25:28.704 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-18T20:25:28.704 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 11556, 2024-03-18T20:25:28.704 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-18T20:25:28.704 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-18T20:25:28.704 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-18T20:25:28.705 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 6.0296553900000003, 2024-03-18T20:25:28.705 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 104, 2024-03-18T20:25:28.705 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 232, 2024-03-18T20:25:28.705 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 232, 2024-03-18T20:25:28.705 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 11.341132656999999 2024-03-18T20:25:28.705 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:25:28.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.695+0000 7f4c3b643700 1 -- 172.21.15.1:0/175561285 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c1806adb0 msgr2=0x7f4c1806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.695+0000 7f4c3b643700 1 --2- 172.21.15.1:0/175561285 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c1806adb0 0x7f4c1806d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4c340ae620 tx=0x7f4c24008040 comp rx=0 tx=0).stop 2024-03-18T20:25:28.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.695+0000 7f4c3b643700 1 -- 172.21.15.1:0/175561285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3406e680 msgr2=0x7f4c34159690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.695+0000 7f4c3b643700 1 --2- 172.21.15.1:0/175561285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3406e680 0x7f4c34159690 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7f4c3001f040 tx=0x7f4c30004750 comp rx=0 tx=0).stop 2024-03-18T20:25:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.695+0000 7f4c3b643700 1 -- 172.21.15.1:0/175561285 shutdown_connections 2024-03-18T20:25:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.695+0000 7f4c3b643700 1 --2- 172.21.15.1:0/175561285 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c1806adb0 0x7f4c1806d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.695+0000 7f4c3b643700 1 --2- 172.21.15.1:0/175561285 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c34070e40 0x7f4c34153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.695+0000 7f4c3b643700 1 --2- 172.21.15.1:0/175561285 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c3406e200 0x7f4c3414b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.695+0000 7f4c3b643700 1 --2- 172.21.15.1:0/175561285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c3406e680 0x7f4c34159690 unknown :-1 s=CLOSED pgs=2533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.695+0000 7f4c3b643700 1 -- 172.21.15.1:0/175561285 >> 172.21.15.1:0/175561285 conn(0x7f4c3406a750 msgr2=0x7f4c340b3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.695+0000 7f4c3b643700 1 -- 172.21.15.1:0/175561285 shutdown_connections 2024-03-18T20:25:28.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.695+0000 7f4c3b643700 1 -- 172.21.15.1:0/175561285 wait complete. 2024-03-18T20:25:28.711 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T20:25:28.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.832+0000 7f3fad268700 1 Processor -- start 2024-03-18T20:25:28.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.833+0000 7f3fad268700 1 -- start start 2024-03-18T20:25:28.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.833+0000 7f3fad268700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3fa80d2e80 0x7f3fa80d3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.833+0000 7f3fad268700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3fa815d810 0x7f3fa8161240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.833+0000 7f3fad268700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3fa8161780 0x7f3fa80b3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.833+0000 7f3fad268700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fa8161fa0 con 0x7f3fa8161780 2024-03-18T20:25:28.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.833+0000 7f3fad268700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fa815eaa0 con 0x7f3fa815d810 2024-03-18T20:25:28.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.833+0000 7f3fad268700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa815e610 con 0x7f3fa80d2e80 2024-03-18T20:25:28.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.834+0000 7f3fa77fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3fa8161780 0x7f3fa80b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.834+0000 7f3fa77fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3fa8161780 0x7f3fa80b3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59974/0 (socket says 172.21.15.1:59974) 2024-03-18T20:25:28.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.834+0000 7f3fa77fe700 1 -- 172.21.15.1:0/2929438924 learned_addr learned my addr 172.21.15.1:0/2929438924 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:28.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.834+0000 7f3fa67fc700 1 --2- 172.21.15.1:0/2929438924 >> v2:172.21.15.165:3300/0 conn(0x7f3fa815d810 0x7f3fa8161240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.834+0000 7f3fa77fe700 1 -- 172.21.15.1:0/2929438924 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3fa80d2e80 msgr2=0x7f3fa80d3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.834+0000 7f3fa77fe700 1 --2- 172.21.15.1:0/2929438924 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3fa80d2e80 0x7f3fa80d3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.834+0000 7f3fa77fe700 1 -- 172.21.15.1:0/2929438924 >> v2:172.21.15.165:3300/0 conn(0x7f3fa815d810 msgr2=0x7f3fa8161240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.834+0000 7f3fa77fe700 1 --2- 172.21.15.1:0/2929438924 >> v2:172.21.15.165:3300/0 conn(0x7f3fa815d810 0x7f3fa8161240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.834+0000 7f3fa77fe700 1 -- 172.21.15.1:0/2929438924 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa815dce0 con 0x7f3fa8161780 2024-03-18T20:25:28.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.834+0000 7f3fa77fe700 1 --2- 172.21.15.1:0/2929438924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa8161780 0x7f3fa80b3340 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f3fa80d0cb0 tx=0x7f3fa000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd6ecf7db66379b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:28.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.834+0000 7f3fa5ffb700 1 -- 172.21.15.1:0/2929438924 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa000e070 con 0x7f3fa8161780 2024-03-18T20:25:28.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.834+0000 7f3fa5ffb700 1 -- 172.21.15.1:0/2929438924 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa000b910 con 0x7f3fa8161780 2024-03-18T20:25:28.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.834+0000 7f3fa5ffb700 1 -- 172.21.15.1:0/2929438924 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa000bcc0 con 0x7f3fa8161780 2024-03-18T20:25:28.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.835+0000 7f3fad268700 1 -- 172.21.15.1:0/2929438924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa8161780 msgr2=0x7f3fa80b3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.835+0000 7f3fad268700 1 --2- 172.21.15.1:0/2929438924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa8161780 0x7f3fa80b3340 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f3fa80d0cb0 tx=0x7f3fa000b330 comp rx=0 tx=0).stop 2024-03-18T20:25:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.835+0000 7f3fad268700 1 -- 172.21.15.1:0/2929438924 shutdown_connections 2024-03-18T20:25:28.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.835+0000 7f3fad268700 1 --2- 172.21.15.1:0/2929438924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa8161780 0x7f3fa80b3340 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.835+0000 7f3fad268700 1 --2- 172.21.15.1:0/2929438924 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3fa80d2e80 0x7f3fa80d3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.835+0000 7f3fad268700 1 --2- 172.21.15.1:0/2929438924 >> v2:172.21.15.165:3300/0 conn(0x7f3fa815d810 0x7f3fa8161240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.835+0000 7f3fad268700 1 -- 172.21.15.1:0/2929438924 >> 172.21.15.1:0/2929438924 conn(0x7f3fa808cbf0 msgr2=0x7f3fa809c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:28.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.835+0000 7f3fad268700 1 -- 172.21.15.1:0/2929438924 shutdown_connections 2024-03-18T20:25:28.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.835+0000 7f3fad268700 1 -- 172.21.15.1:0/2929438924 wait complete. 2024-03-18T20:25:28.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.836+0000 7f3fad268700 1 Processor -- start 2024-03-18T20:25:28.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.836+0000 7f3fad268700 1 -- start start 2024-03-18T20:25:28.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.837+0000 7f3fad268700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3fa80d2e80 0x7f3fa815b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.837+0000 7f3fad268700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3fa815d810 0x7f3fa80c9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.837+0000 7f3fad268700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa8161780 0x7f3fa806c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.837+0000 7f3fad268700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa80c9c10 con 0x7f3fa8161780 2024-03-18T20:25:28.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.837+0000 7f3fad268700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa815bb80 con 0x7f3fa80d2e80 2024-03-18T20:25:28.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.837+0000 7f3fad268700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa80b1cb0 con 0x7f3fa815d810 2024-03-18T20:25:28.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.837+0000 7f3fa77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa8161780 0x7f3fa806c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.837+0000 7f3fa77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa8161780 0x7f3fa806c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59986/0 (socket says 172.21.15.1:59986) 2024-03-18T20:25:28.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.837+0000 7f3fa77fe700 1 -- 172.21.15.1:0/3764271450 learned_addr learned my addr 172.21.15.1:0/3764271450 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:28.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.837+0000 7f3fa67fc700 1 --2- 172.21.15.1:0/3764271450 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3fa815d810 0x7f3fa80c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.837+0000 7f3fa6ffd700 1 --2- 172.21.15.1:0/3764271450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3fa80d2e80 0x7f3fa815b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.837+0000 7f3fa77fe700 1 -- 172.21.15.1:0/3764271450 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3fa815d810 msgr2=0x7f3fa80c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.837+0000 7f3fa77fe700 1 --2- 172.21.15.1:0/3764271450 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3fa815d810 0x7f3fa80c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.837+0000 7f3fa77fe700 1 -- 172.21.15.1:0/3764271450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3fa80d2e80 msgr2=0x7f3fa815b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:28.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.837+0000 7f3fa77fe700 1 --2- 172.21.15.1:0/3764271450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3fa80d2e80 0x7f3fa815b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:28.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.837+0000 7f3fa77fe700 1 -- 172.21.15.1:0/3764271450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa0009b30 con 0x7f3fa8161780 2024-03-18T20:25:28.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.837+0000 7f3fa67fc700 1 --2- 172.21.15.1:0/3764271450 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3fa815d810 0x7f3fa80c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:28.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.837+0000 7f3fa6ffd700 1 --2- 172.21.15.1:0/3764271450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3fa80d2e80 0x7f3fa815b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:28.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.837+0000 7f3fa77fe700 1 --2- 172.21.15.1:0/3764271450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa8161780 0x7f3fa806c590 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f3fa808be20 tx=0x7f3fa0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:28.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.838+0000 7f3f9f7fe700 1 -- 172.21.15.1:0/3764271450 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa00099b0 con 0x7f3fa8161780 2024-03-18T20:25:28.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.838+0000 7f3f9f7fe700 1 -- 172.21.15.1:0/3764271450 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa0004250 con 0x7f3fa8161780 2024-03-18T20:25:28.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.838+0000 7f3fad268700 1 -- 172.21.15.1:0/3764271450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa80c9930 con 0x7f3fa8161780 2024-03-18T20:25:28.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.838+0000 7f3f9f7fe700 1 -- 172.21.15.1:0/3764271450 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa0004600 con 0x7f3fa8161780 2024-03-18T20:25:28.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.838+0000 7f3fad268700 1 -- 172.21.15.1:0/3764271450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa80b14b0 con 0x7f3fa8161780 2024-03-18T20:25:28.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.839+0000 7f3fad268700 1 -- 172.21.15.1:0/3764271450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa80b1030 con 0x7f3fa8161780 2024-03-18T20:25:28.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.839+0000 7f3f9f7fe700 1 -- 172.21.15.1:0/3764271450 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3fa0004760 con 0x7f3fa8161780 2024-03-18T20:25:28.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.840+0000 7f3f9f7fe700 1 --2- 172.21.15.1:0/3764271450 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3f8806adb0 0x7f3f8806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:28.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.840+0000 7f3f9f7fe700 1 -- 172.21.15.1:0/3764271450 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f3fa008abf0 con 0x7f3fa8161780 2024-03-18T20:25:28.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.840+0000 7f3fa6ffd700 1 --2- 172.21.15.1:0/3764271450 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3f8806adb0 0x7f3f8806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:28.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.840+0000 7f3fa6ffd700 1 --2- 172.21.15.1:0/3764271450 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3f8806adb0 0x7f3f8806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3fa80ca930 tx=0x7f3f90008040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:28.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:28.844+0000 7f3f9f7fe700 1 -- 172.21.15.1:0/3764271450 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3fa80b1030 con 0x7f3fa8161780 2024-03-18T20:25:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:29.089+0000 7f3fad268700 1 -- 172.21.15.1:0/3764271450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3fa80b1030 con 0x7f3fa8161780 2024-03-18T20:25:29.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:29.089+0000 7f3f9f7fe700 1 -- 172.21.15.1:0/3764271450 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v104) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3fa80b1030 con 0x7f3fa8161780 2024-03-18T20:25:29.093 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T20:25:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:29.091+0000 7f3fad268700 1 -- 172.21.15.1:0/3764271450 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3f8806adb0 msgr2=0x7f3f8806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:29.091+0000 7f3fad268700 1 --2- 172.21.15.1:0/3764271450 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3f8806adb0 0x7f3f8806d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3fa80ca930 tx=0x7f3f90008040 comp rx=0 tx=0).stop 2024-03-18T20:25:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:29.091+0000 7f3fad268700 1 -- 172.21.15.1:0/3764271450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa8161780 msgr2=0x7f3fa806c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:29.091+0000 7f3fad268700 1 --2- 172.21.15.1:0/3764271450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa8161780 0x7f3fa806c590 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f3fa808be20 tx=0x7f3fa0004ad0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:29.091+0000 7f3fad268700 1 -- 172.21.15.1:0/3764271450 shutdown_connections 2024-03-18T20:25:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:29.091+0000 7f3fad268700 1 --2- 172.21.15.1:0/3764271450 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3f8806adb0 0x7f3f8806d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:29.091+0000 7f3fad268700 1 --2- 172.21.15.1:0/3764271450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa8161780 0x7f3fa806c590 unknown :-1 s=CLOSED pgs=2539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:29.091+0000 7f3fad268700 1 --2- 172.21.15.1:0/3764271450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3fa80d2e80 0x7f3fa815b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:29.091+0000 7f3fad268700 1 --2- 172.21.15.1:0/3764271450 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3fa815d810 0x7f3fa80c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:29.091+0000 7f3fad268700 1 -- 172.21.15.1:0/3764271450 >> 172.21.15.1:0/3764271450 conn(0x7f3fa808cbf0 msgr2=0x7f3fa806b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:29.092+0000 7f3fad268700 1 -- 172.21.15.1:0/3764271450 shutdown_connections 2024-03-18T20:25:29.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:29.092+0000 7f3fad268700 1 -- 172.21.15.1:0/3764271450 wait complete. 2024-03-18T20:25:29.108 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T20:25:29.108 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T20:25:29.108 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-18T20:25:29.108 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T20:25:29.108 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T20:25:29.108 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T20:25:29.109 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T20:25:29.109 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T20:25:29.109 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:25:29.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:29.109 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-18T20:25:29.128 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T20:25:29.128 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T20:25:29.128 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T20:25:29.128 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:25:29.128 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T20:25:29.128 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:25:29.129 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:25:29.129 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-18T20:25:29.129 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-18T20:25:29.129 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:25:29.129 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-18T20:25:29.129 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-18T20:25:29.129 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T20:25:29.129 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:25:29.129 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-18T20:25:29.129 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T20:25:29.129 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 36840sec preferred_lft 36840sec 2024-03-18T20:25:29.129 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link 2024-03-18T20:25:29.130 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:25:29.130 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:25:29.130 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:25:29.130 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-18T20:25:29.130 INFO:teuthology.orchestra.run.smithi001.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T20:25:29.130 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:25:29.130 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T20:25:29.130 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:25:29.130 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::4ca9:f5ff:feb3:79a3/64 scope link 2024-03-18T20:25:29.130 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:25:29.130 INFO:teuthology.orchestra.run.smithi001.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T20:25:29.130 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T20:25:29.131 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::282f:a4ff:fe1a:9908/64 scope link 2024-03-18T20:25:29.131 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:25:29.131 INFO:teuthology.orchestra.run.smithi001.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T20:25:29.131 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2e:5c:60:78:b2:8c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T20:25:29.131 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::2c5c:60ff:fe78:b28c/64 scope link 2024-03-18T20:25:29.131 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:25:29.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:29.131 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T20:25:29.194 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-18T20:25:29.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:29.195 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-18T20:25:29.257 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-18T20:25:29.257 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T20:25:29.257 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T20:25:29.257 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T20:25:29.257 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T20:25:29.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:29.258 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T20:25:29.258 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T20:25:29.258 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T20:25:29.258 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T20:25:29.258 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T20:25:29.258 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T20:25:29.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:29.363 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T20:25:29.381 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T20:25:29.381 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T20:25:29.381 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:25:29.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:29.382 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-18T20:25:29.450 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T20:25:29.450 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:25:29.450 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T20:25:29.508 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:25:29.508 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:25:29.509 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T20:25:29.567 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T20:25:29.636 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:25:29.636 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:25:29.712 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:25:29.782 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:25:29.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:29.783 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T20:25:29.845 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T20:25:29.846 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T20:25:29.889 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T20:25:29.917 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:25:29.918 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:25:29.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-18T20:25:29.973+0000 7ff5f66845c0 -1 init, newargv = 0x5633a4d5fe60 newargc=15 2024-03-18T20:25:29.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[158300]: starting ceph client 2024-03-18T20:25:29.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[158300]: starting fuse 2024-03-18T20:25:29.988 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:25:29.988 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:25:29.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:29.989 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T20:25:30.048 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T20:25:30.048 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-03-18T20:25:30.049 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T20:25:30.049 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-18T20:25:30.049 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-18T20:25:30.049 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-18T20:25:30.049 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-18T20:25:30.049 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:25:30.049 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-18T20:25:30.049 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T20:25:30.049 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-18T20:25:30.049 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T20:25:30.049 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:25:30.049 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-18T20:25:30.049 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-18T20:25:30.049 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-18T20:25:30.050 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-18T20:25:30.050 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:25:30.050 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-18T20:25:30.050 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T20:25:30.050 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T20:25:30.050 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T20:25:30.050 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-18T20:25:30.050 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-18T20:25:30.050 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-18T20:25:30.050 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T20:25:30.050 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:25:30.050 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-18T20:25:30.050 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-18T20:25:30.175 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.158300.asok 2024-03-18T20:25:30.356 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.158300.asok 2024-03-18T20:25:30.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:30.357 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.158300.asok status 2024-03-18T20:25:30.518 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:25:30.518 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-18T20:25:30.518 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T20:25:30.518 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T20:25:30.518 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-18T20:25:30.518 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-18T20:25:30.518 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T20:25:30.518 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "158300", 2024-03-18T20:25:30.519 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-18T20:25:30.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:30.519 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-18T20:25:30.519 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-18T20:25:30.519 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 11715, 2024-03-18T20:25:30.519 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-18T20:25:30.519 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-18T20:25:30.519 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-18T20:25:30.519 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 11715 2024-03-18T20:25:30.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:30.519 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T20:25:30.519 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T20:25:30.519 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:25:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1334459522 2024-03-18T20:25:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T20:25:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T20:25:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:25:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1334459522 2024-03-18T20:25:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.11715 192.168.0.1:0/1334459522", 2024-03-18T20:25:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/1334459522", 2024-03-18T20:25:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-18T20:25:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 104, 2024-03-18T20:25:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 232, 2024-03-18T20:25:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-18T20:25:30.520 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-18T20:25:30.521 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-18T20:25:30.521 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:25:30.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:30.528 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-18T20:25:30.528 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T20:25:30.546 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-18T20:25:30.547 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T20:25:30.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:30.547 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T20:25:30.616 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-03-18T20:25:30.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.784+0000 7efd82c29700 1 Processor -- start 2024-03-18T20:25:30.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.784+0000 7efd82c29700 1 -- start start 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.785+0000 7efd82c29700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7efd7c15af20 0x7efd7c072480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.785+0000 7efd82c29700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efd7c15b300 0x7efd7c0aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.785+0000 7efd82c29700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd7c0af4e0 0x7efd7c15dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.785+0000 7efd82c29700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd7c0af970 con 0x7efd7c15b300 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.785+0000 7efd82c29700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd7c15e860 con 0x7efd7c15af20 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.785+0000 7efd82c29700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c0719a0 con 0x7efd7c0af4e0 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.785+0000 7efd7bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efd7c15b300 0x7efd7c0aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.785+0000 7efd7bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efd7c15b300 0x7efd7c0aefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59992/0 (socket says 172.21.15.1:59992) 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.785+0000 7efd7bfff700 1 -- 172.21.15.1:0/1625927896 learned_addr learned my addr 172.21.15.1:0/1625927896 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.786+0000 7efd809c5700 1 --2- 172.21.15.1:0/1625927896 >> v2:172.21.15.165:3300/0 conn(0x7efd7c15af20 0x7efd7c072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.786+0000 7efd811c6700 1 --2- 172.21.15.1:0/1625927896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd7c0af4e0 0x7efd7c15dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.786+0000 7efd7bfff700 1 -- 172.21.15.1:0/1625927896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd7c0af4e0 msgr2=0x7efd7c15dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.786+0000 7efd7bfff700 1 --2- 172.21.15.1:0/1625927896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd7c0af4e0 0x7efd7c15dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.786+0000 7efd7bfff700 1 -- 172.21.15.1:0/1625927896 >> v2:172.21.15.165:3300/0 conn(0x7efd7c15af20 msgr2=0x7efd7c072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.786+0000 7efd7bfff700 1 --2- 172.21.15.1:0/1625927896 >> v2:172.21.15.165:3300/0 conn(0x7efd7c15af20 0x7efd7c072480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.786+0000 7efd7bfff700 1 -- 172.21.15.1:0/1625927896 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7c071c80 con 0x7efd7c15b300 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.786+0000 7efd7bfff700 1 --2- 172.21.15.1:0/1625927896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c15b300 0x7efd7c0aefa0 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7efd680098f0 tx=0x7efd68009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57d2e1254c2ca606 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.786+0000 7efd7b7fe700 1 -- 172.21.15.1:0/1625927896 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6800b860 con 0x7efd7c15b300 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.786+0000 7efd7b7fe700 1 -- 172.21.15.1:0/1625927896 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6800b9c0 con 0x7efd7c15b300 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.786+0000 7efd7b7fe700 1 -- 172.21.15.1:0/1625927896 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6800bd50 con 0x7efd7c15b300 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.787+0000 7efd82c29700 1 -- 172.21.15.1:0/1625927896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c15b300 msgr2=0x7efd7c0aefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.787+0000 7efd82c29700 1 --2- 172.21.15.1:0/1625927896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c15b300 0x7efd7c0aefa0 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7efd680098f0 tx=0x7efd68009c00 comp rx=0 tx=0).stop 2024-03-18T20:25:30.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.787+0000 7efd82c29700 1 -- 172.21.15.1:0/1625927896 shutdown_connections 2024-03-18T20:25:30.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.787+0000 7efd82c29700 1 --2- 172.21.15.1:0/1625927896 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd7c0af4e0 0x7efd7c15dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.787+0000 7efd82c29700 1 --2- 172.21.15.1:0/1625927896 >> v2:172.21.15.165:3300/0 conn(0x7efd7c15af20 0x7efd7c072480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.787+0000 7efd82c29700 1 --2- 172.21.15.1:0/1625927896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c15b300 0x7efd7c0aefa0 unknown :-1 s=CLOSED pgs=2541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.787+0000 7efd82c29700 1 -- 172.21.15.1:0/1625927896 >> 172.21.15.1:0/1625927896 conn(0x7efd7c0d00a0 msgr2=0x7efd7c0d04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:30.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.787+0000 7efd82c29700 1 -- 172.21.15.1:0/1625927896 shutdown_connections 2024-03-18T20:25:30.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.787+0000 7efd82c29700 1 -- 172.21.15.1:0/1625927896 wait complete. 2024-03-18T20:25:30.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.788+0000 7efd82c29700 1 Processor -- start 2024-03-18T20:25:30.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.788+0000 7efd82c29700 1 -- start start 2024-03-18T20:25:30.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.789+0000 7efd82c29700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd7c0af4e0 0x7efd7c149e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.789+0000 7efd82c29700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd7c15af20 0x7efd7c0c5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.789+0000 7efd82c29700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c15b300 0x7efd7c0aeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.789+0000 7efd82c29700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c0ada00 con 0x7efd7c15b300 2024-03-18T20:25:30.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.789+0000 7efd82c29700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c0adce0 con 0x7efd7c15af20 2024-03-18T20:25:30.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.789+0000 7efd82c29700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c0adfc0 con 0x7efd7c0af4e0 2024-03-18T20:25:30.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.789+0000 7efd811c6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c15b300 0x7efd7c0aeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.789+0000 7efd811c6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c15b300 0x7efd7c0aeaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60004/0 (socket says 172.21.15.1:60004) 2024-03-18T20:25:30.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.789+0000 7efd811c6700 1 -- 172.21.15.1:0/2457790676 learned_addr learned my addr 172.21.15.1:0/2457790676 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:30.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.789+0000 7efd7bfff700 1 --2- 172.21.15.1:0/2457790676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd7c15af20 0x7efd7c0c5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.789+0000 7efd809c5700 1 --2- 172.21.15.1:0/2457790676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd7c0af4e0 0x7efd7c149e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.789+0000 7efd811c6700 1 -- 172.21.15.1:0/2457790676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd7c0af4e0 msgr2=0x7efd7c149e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.789+0000 7efd811c6700 1 --2- 172.21.15.1:0/2457790676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd7c0af4e0 0x7efd7c149e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.789+0000 7efd811c6700 1 -- 172.21.15.1:0/2457790676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd7c15af20 msgr2=0x7efd7c0c5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:30.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.789+0000 7efd811c6700 1 --2- 172.21.15.1:0/2457790676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd7c15af20 0x7efd7c0c5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:30.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.789+0000 7efd811c6700 1 -- 172.21.15.1:0/2457790676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd68009550 con 0x7efd7c15b300 2024-03-18T20:25:30.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.789+0000 7efd811c6700 1 --2- 172.21.15.1:0/2457790676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c15b300 0x7efd7c0aeaa0 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7efd74007e60 tx=0x7efd7400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:30.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.789+0000 7efd797fa700 1 -- 172.21.15.1:0/2457790676 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7400ed30 con 0x7efd7c15b300 2024-03-18T20:25:30.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.790+0000 7efd797fa700 1 -- 172.21.15.1:0/2457790676 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd74014420 con 0x7efd7c15b300 2024-03-18T20:25:30.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.790+0000 7efd82c29700 1 -- 172.21.15.1:0/2457790676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7c0ae2a0 con 0x7efd7c15b300 2024-03-18T20:25:30.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.790+0000 7efd797fa700 1 -- 172.21.15.1:0/2457790676 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd740147d0 con 0x7efd7c15b300 2024-03-18T20:25:30.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.790+0000 7efd82c29700 1 -- 172.21.15.1:0/2457790676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd7c14a350 con 0x7efd7c15b300 2024-03-18T20:25:30.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.790+0000 7efd82c29700 1 -- 172.21.15.1:0/2457790676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7c070270 con 0x7efd7c15b300 2024-03-18T20:25:30.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.791+0000 7efd797fa700 1 -- 172.21.15.1:0/2457790676 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7efd74014930 con 0x7efd7c15b300 2024-03-18T20:25:30.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.791+0000 7efd797fa700 1 --2- 172.21.15.1:0/2457790676 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efd6006adb0 0x7efd6006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:30.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.792+0000 7efd797fa700 1 -- 172.21.15.1:0/2457790676 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6129+0+0 (secure 0 0 0) 0x7efd74089d40 con 0x7efd7c15b300 2024-03-18T20:25:30.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.792+0000 7efd809c5700 1 --2- 172.21.15.1:0/2457790676 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efd6006adb0 0x7efd6006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:30.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.792+0000 7efd809c5700 1 --2- 172.21.15.1:0/2457790676 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efd6006adb0 0x7efd6006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efd70006b20 tx=0x7efd70008040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:30.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:30.796+0000 7efd797fa700 1 -- 172.21.15.1:0/2457790676 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efd74054c80 con 0x7efd7c15b300 2024-03-18T20:25:31.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.043+0000 7efd82c29700 1 -- 172.21.15.1:0/2457790676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efd7c0ae7c0 con 0x7efd7c15b300 2024-03-18T20:25:31.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.097+0000 7efd797fa700 1 -- 172.21.15.1:0/2457790676 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v105) v1 ==== 136+0+0 (secure 0 0 0) 0x7efd7c0ae7c0 con 0x7efd7c15b300 2024-03-18T20:25:31.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.099+0000 7efd82c29700 1 -- 172.21.15.1:0/2457790676 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efd6006adb0 msgr2=0x7efd6006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:31.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.099+0000 7efd82c29700 1 --2- 172.21.15.1:0/2457790676 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efd6006adb0 0x7efd6006d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efd70006b20 tx=0x7efd70008040 comp rx=0 tx=0).stop 2024-03-18T20:25:31.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.099+0000 7efd82c29700 1 -- 172.21.15.1:0/2457790676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c15b300 msgr2=0x7efd7c0aeaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:31.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.099+0000 7efd82c29700 1 --2- 172.21.15.1:0/2457790676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c15b300 0x7efd7c0aeaa0 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7efd74007e60 tx=0x7efd7400e3c0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.099+0000 7efd82c29700 1 -- 172.21.15.1:0/2457790676 shutdown_connections 2024-03-18T20:25:31.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.099+0000 7efd82c29700 1 --2- 172.21.15.1:0/2457790676 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efd6006adb0 0x7efd6006d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.099+0000 7efd82c29700 1 --2- 172.21.15.1:0/2457790676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efd7c0af4e0 0x7efd7c149e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.099+0000 7efd82c29700 1 --2- 172.21.15.1:0/2457790676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7c15b300 0x7efd7c0aeaa0 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.099+0000 7efd82c29700 1 --2- 172.21.15.1:0/2457790676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd7c15af20 0x7efd7c0c5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.099+0000 7efd82c29700 1 -- 172.21.15.1:0/2457790676 >> 172.21.15.1:0/2457790676 conn(0x7efd7c0d00a0 msgr2=0x7efd7c06d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:31.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.099+0000 7efd82c29700 1 -- 172.21.15.1:0/2457790676 shutdown_connections 2024-03-18T20:25:31.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.099+0000 7efd82c29700 1 -- 172.21.15.1:0/2457790676 wait complete. 2024-03-18T20:25:31.115 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:25:31.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.237+0000 7f3e76654700 1 Processor -- start 2024-03-18T20:25:31.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.237+0000 7f3e76654700 1 -- start start 2024-03-18T20:25:31.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.238+0000 7f3e76654700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3e7015d740 0x7f3e700d3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:31.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.238+0000 7f3e76654700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e700d1490 0x7f3e700d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:31.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.238+0000 7f3e76654700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3e700d1e50 0x7f3e700cf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:31.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.238+0000 7f3e76654700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e700d22e0 con 0x7f3e700d1e50 2024-03-18T20:25:31.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.238+0000 7f3e76654700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e700b2c60 con 0x7f3e7015d740 2024-03-18T20:25:31.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.238+0000 7f3e76654700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7015db20 con 0x7f3e700d1490 2024-03-18T20:25:31.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.238+0000 7f3e74bf1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3e700d1e50 0x7f3e700cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:31.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.238+0000 7f3e74bf1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3e700d1e50 0x7f3e700cf960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60020/0 (socket says 172.21.15.1:60020) 2024-03-18T20:25:31.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.238+0000 7f3e74bf1700 1 -- 172.21.15.1:0/1048253209 learned_addr learned my addr 172.21.15.1:0/1048253209 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:31.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.239+0000 7f3e6ffff700 1 --2- 172.21.15.1:0/1048253209 >> v2:172.21.15.165:3300/0 conn(0x7f3e7015d740 0x7f3e700d3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.239+0000 7f3e6f7fe700 1 --2- 172.21.15.1:0/1048253209 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e700d1490 0x7f3e700d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.239+0000 7f3e74bf1700 1 -- 172.21.15.1:0/1048253209 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e700d1490 msgr2=0x7f3e700d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.239+0000 7f3e74bf1700 1 --2- 172.21.15.1:0/1048253209 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e700d1490 0x7f3e700d1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.239+0000 7f3e74bf1700 1 -- 172.21.15.1:0/1048253209 >> v2:172.21.15.165:3300/0 conn(0x7f3e7015d740 msgr2=0x7f3e700d3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.239+0000 7f3e74bf1700 1 --2- 172.21.15.1:0/1048253209 >> v2:172.21.15.165:3300/0 conn(0x7f3e7015d740 0x7f3e700d3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.239+0000 7f3e74bf1700 1 -- 172.21.15.1:0/1048253209 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e700cea80 con 0x7f3e700d1e50 2024-03-18T20:25:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.239+0000 7f3e6ffff700 1 --2- 172.21.15.1:0/1048253209 >> v2:172.21.15.165:3300/0 conn(0x7f3e7015d740 0x7f3e700d3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:31.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.239+0000 7f3e6f7fe700 1 --2- 172.21.15.1:0/1048253209 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e700d1490 0x7f3e700d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:31.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.239+0000 7f3e74bf1700 1 --2- 172.21.15.1:0/1048253209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e700d1e50 0x7f3e700cf960 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f3e7008d9e0 tx=0x7f3e6800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719adf3296fd18df server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:31.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.239+0000 7f3e6effd700 1 -- 172.21.15.1:0/1048253209 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6800e070 con 0x7f3e700d1e50 2024-03-18T20:25:31.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.239+0000 7f3e6effd700 1 -- 172.21.15.1:0/1048253209 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e6800b9c0 con 0x7f3e700d1e50 2024-03-18T20:25:31.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.239+0000 7f3e6effd700 1 -- 172.21.15.1:0/1048253209 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6800bd70 con 0x7f3e700d1e50 2024-03-18T20:25:31.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.240+0000 7f3e76654700 1 -- 172.21.15.1:0/1048253209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e700d1e50 msgr2=0x7f3e700cf960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:31.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.240+0000 7f3e76654700 1 --2- 172.21.15.1:0/1048253209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e700d1e50 0x7f3e700cf960 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f3e7008d9e0 tx=0x7f3e6800b330 comp rx=0 tx=0).stop 2024-03-18T20:25:31.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.240+0000 7f3e76654700 1 -- 172.21.15.1:0/1048253209 shutdown_connections 2024-03-18T20:25:31.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.240+0000 7f3e76654700 1 --2- 172.21.15.1:0/1048253209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e700d1e50 0x7f3e700cf960 unknown :-1 s=CLOSED pgs=2543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.240+0000 7f3e76654700 1 --2- 172.21.15.1:0/1048253209 >> v2:172.21.15.165:3300/0 conn(0x7f3e7015d740 0x7f3e700d3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.240+0000 7f3e76654700 1 --2- 172.21.15.1:0/1048253209 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e700d1490 0x7f3e700d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.240+0000 7f3e76654700 1 -- 172.21.15.1:0/1048253209 >> 172.21.15.1:0/1048253209 conn(0x7f3e7009c600 msgr2=0x7f3e700d3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:31.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.240+0000 7f3e76654700 1 -- 172.21.15.1:0/1048253209 shutdown_connections 2024-03-18T20:25:31.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.240+0000 7f3e76654700 1 -- 172.21.15.1:0/1048253209 wait complete. 2024-03-18T20:25:31.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.241+0000 7f3e76654700 1 Processor -- start 2024-03-18T20:25:31.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.241+0000 7f3e76654700 1 -- start start 2024-03-18T20:25:31.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.242+0000 7f3e76654700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e700d1490 0x7f3e70152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:31.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.242+0000 7f3e76654700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e700d1e50 0x7f3e700c7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:31.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.242+0000 7f3e76654700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7015d740 0x7f3e700b85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:31.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.242+0000 7f3e76654700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7015f740 con 0x7f3e7015d740 2024-03-18T20:25:31.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.242+0000 7f3e76654700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e700c8320 con 0x7f3e700d1e50 2024-03-18T20:25:31.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.242+0000 7f3e76654700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e70160100 con 0x7f3e700d1490 2024-03-18T20:25:31.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.242+0000 7f3e74bf1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7015d740 0x7f3e700b85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:31.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.242+0000 7f3e74bf1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7015d740 0x7f3e700b85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60036/0 (socket says 172.21.15.1:60036) 2024-03-18T20:25:31.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.242+0000 7f3e74bf1700 1 -- 172.21.15.1:0/1253478047 learned_addr learned my addr 172.21.15.1:0/1253478047 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:31.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.242+0000 7f3e6ffff700 1 --2- 172.21.15.1:0/1253478047 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e700d1490 0x7f3e70152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:31.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.242+0000 7f3e6f7fe700 1 --2- 172.21.15.1:0/1253478047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e700d1e50 0x7f3e700c7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:31.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.242+0000 7f3e74bf1700 1 -- 172.21.15.1:0/1253478047 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e700d1490 msgr2=0x7f3e70152690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:31.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.242+0000 7f3e74bf1700 1 --2- 172.21.15.1:0/1253478047 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e700d1490 0x7f3e70152690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.242+0000 7f3e74bf1700 1 -- 172.21.15.1:0/1253478047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e700d1e50 msgr2=0x7f3e700c7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:31.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.242+0000 7f3e74bf1700 1 --2- 172.21.15.1:0/1253478047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e700d1e50 0x7f3e700c7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.242+0000 7f3e74bf1700 1 -- 172.21.15.1:0/1253478047 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e68009b30 con 0x7f3e7015d740 2024-03-18T20:25:31.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.242+0000 7f3e6ffff700 1 --2- 172.21.15.1:0/1253478047 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e700d1490 0x7f3e70152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:31.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.242+0000 7f3e6f7fe700 1 --2- 172.21.15.1:0/1253478047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e700d1e50 0x7f3e700c7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:31.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.243+0000 7f3e74bf1700 1 --2- 172.21.15.1:0/1253478047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7015d740 0x7f3e700b85a0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7f3e7008cbb0 tx=0x7f3e68004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:31.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.243+0000 7f3e6cff9700 1 -- 172.21.15.1:0/1253478047 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e680099b0 con 0x7f3e7015d740 2024-03-18T20:25:31.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.243+0000 7f3e6cff9700 1 -- 172.21.15.1:0/1253478047 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e68004250 con 0x7f3e7015d740 2024-03-18T20:25:31.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.243+0000 7f3e76654700 1 -- 172.21.15.1:0/1253478047 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e701603e0 con 0x7f3e7015d740 2024-03-18T20:25:31.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.243+0000 7f3e6cff9700 1 -- 172.21.15.1:0/1253478047 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e68004600 con 0x7f3e7015d740 2024-03-18T20:25:31.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.243+0000 7f3e76654700 1 -- 172.21.15.1:0/1253478047 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e700b62d0 con 0x7f3e7015d740 2024-03-18T20:25:31.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.244+0000 7f3e76654700 1 -- 172.21.15.1:0/1253478047 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e70073350 con 0x7f3e7015d740 2024-03-18T20:25:31.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.244+0000 7f3e6cff9700 1 -- 172.21.15.1:0/1253478047 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3e68004760 con 0x7f3e7015d740 2024-03-18T20:25:31.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.245+0000 7f3e6cff9700 1 --2- 172.21.15.1:0/1253478047 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3e5006adb0 0x7f3e5006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:31.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.245+0000 7f3e6cff9700 1 -- 172.21.15.1:0/1253478047 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f3e6808a7e0 con 0x7f3e7015d740 2024-03-18T20:25:31.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.245+0000 7f3e6ffff700 1 --2- 172.21.15.1:0/1253478047 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3e5006adb0 0x7f3e5006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:31.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.245+0000 7f3e6ffff700 1 --2- 172.21.15.1:0/1253478047 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3e5006adb0 0x7f3e5006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800c960 tx=0x7f3e58008040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:31.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.249+0000 7f3e6cff9700 1 -- 172.21.15.1:0/1253478047 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3e68055680 con 0x7f3e7015d740 2024-03-18T20:25:31.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.451+0000 7f3e76654700 1 -- 172.21.15.1:0/1253478047 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e700d1c70 con 0x7f3e7015d740 2024-03-18T20:25:31.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.452+0000 7f3e6cff9700 1 -- 172.21.15.1:0/1253478047 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) v1 ==== 74+0+16429 (secure 0 0 0) 0x7f3e6801f090 con 0x7f3e7015d740 2024-03-18T20:25:31.457 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:31.457 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":232,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:25:23.665001+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:25:20.511297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-18T20:25:21.637693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":230,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:31.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.454+0000 7f3e76654700 1 -- 172.21.15.1:0/1253478047 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3e5006adb0 msgr2=0x7f3e5006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:31.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.454+0000 7f3e76654700 1 --2- 172.21.15.1:0/1253478047 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3e5006adb0 0x7f3e5006d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800c960 tx=0x7f3e58008040 comp rx=0 tx=0).stop 2024-03-18T20:25:31.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.454+0000 7f3e76654700 1 -- 172.21.15.1:0/1253478047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7015d740 msgr2=0x7f3e700b85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:31.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.454+0000 7f3e76654700 1 --2- 172.21.15.1:0/1253478047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7015d740 0x7f3e700b85a0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7f3e7008cbb0 tx=0x7f3e68004ad0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.454+0000 7f3e76654700 1 -- 172.21.15.1:0/1253478047 shutdown_connections 2024-03-18T20:25:31.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.454+0000 7f3e76654700 1 --2- 172.21.15.1:0/1253478047 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3e5006adb0 0x7f3e5006d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.454+0000 7f3e76654700 1 --2- 172.21.15.1:0/1253478047 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3e700d1490 0x7f3e70152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.454+0000 7f3e76654700 1 --2- 172.21.15.1:0/1253478047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7015d740 0x7f3e700b85a0 unknown :-1 s=CLOSED pgs=2544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.454+0000 7f3e76654700 1 --2- 172.21.15.1:0/1253478047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e700d1e50 0x7f3e700c7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.454+0000 7f3e76654700 1 -- 172.21.15.1:0/1253478047 >> 172.21.15.1:0/1253478047 conn(0x7f3e7009c600 msgr2=0x7f3e7015b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:31.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.454+0000 7f3e76654700 1 -- 172.21.15.1:0/1253478047 shutdown_connections 2024-03-18T20:25:31.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.454+0000 7f3e76654700 1 -- 172.21.15.1:0/1253478047 wait complete. 2024-03-18T20:25:31.471 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:25:20.511297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '231', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:25:21.637693+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '232', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:25:31.471 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:25:31.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.594+0000 7f2ca057e700 1 Processor -- start 2024-03-18T20:25:31.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.594+0000 7f2ca057e700 1 -- start start 2024-03-18T20:25:31.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.595+0000 7f2ca057e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c980cc7b0 0x7f2c980cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:31.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.595+0000 7f2ca057e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2c980cfc00 0x7f2c980c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:31.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.595+0000 7f2ca057e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c980c4c50 con 0x7f2c9815b480 2024-03-18T20:25:31.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.595+0000 7f2ca057e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c980d0080 con 0x7f2c980cfc00 2024-03-18T20:25:31.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.595+0000 7f2ca057e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c980c5210 con 0x7f2c980cc7b0 2024-03-18T20:25:31.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.596+0000 7f2c9db19700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c980cc7b0 0x7f2c980cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:31.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.596+0000 7f2c9db19700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c980cc7b0 0x7f2c980cf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:59570/0 (socket says 172.21.15.1:59570) 2024-03-18T20:25:31.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.596+0000 7f2c9db19700 1 -- 172.21.15.1:0/2225319589 learned_addr learned my addr 172.21.15.1:0/2225319589 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:31.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.596+0000 7f2c9eb1b700 1 --2- 172.21.15.1:0/2225319589 >> v2:172.21.15.165:3300/0 conn(0x7f2c980cfc00 0x7f2c980c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:31.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.596+0000 7f2c9db19700 1 -- 172.21.15.1:0/2225319589 >> v2:172.21.15.165:3300/0 conn(0x7f2c980cfc00 msgr2=0x7f2c980c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:31.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.596+0000 7f2c9db19700 1 --2- 172.21.15.1:0/2225319589 >> v2:172.21.15.165:3300/0 conn(0x7f2c980cfc00 0x7f2c980c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.596+0000 7f2c9db19700 1 -- 172.21.15.1:0/2225319589 >> v1:172.21.15.165:6789/0 conn(0x7f2c9815b480 legacy=0x7f2c980cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:31.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.596+0000 7f2c9db19700 1 -- 172.21.15.1:0/2225319589 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c980cf4e0 con 0x7f2c980cc7b0 2024-03-18T20:25:31.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.596+0000 7f2c9db19700 1 --2- 172.21.15.1:0/2225319589 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c980cc7b0 0x7f2c980cf6c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2c940098f0 tx=0x7f2c94009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d0934a69a9c134e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:31.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.596+0000 7f2c9d318700 1 -- 172.21.15.1:0/2225319589 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c94011830 con 0x7f2c980cc7b0 2024-03-18T20:25:31.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.596+0000 7f2c9d318700 1 -- 172.21.15.1:0/2225319589 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c94011990 con 0x7f2c980cc7b0 2024-03-18T20:25:31.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.596+0000 7f2c9d318700 1 -- 172.21.15.1:0/2225319589 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c94011d20 con 0x7f2c980cc7b0 2024-03-18T20:25:31.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.597+0000 7f2ca057e700 1 -- 172.21.15.1:0/2225319589 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c980cc7b0 msgr2=0x7f2c980cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:31.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.597+0000 7f2ca057e700 1 --2- 172.21.15.1:0/2225319589 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c980cc7b0 0x7f2c980cf6c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2c940098f0 tx=0x7f2c94009c00 comp rx=0 tx=0).stop 2024-03-18T20:25:31.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.597+0000 7f2ca057e700 1 -- 172.21.15.1:0/2225319589 shutdown_connections 2024-03-18T20:25:31.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.597+0000 7f2ca057e700 1 --2- 172.21.15.1:0/2225319589 >> v2:172.21.15.165:3300/0 conn(0x7f2c980cfc00 0x7f2c980c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.597+0000 7f2ca057e700 1 --2- 172.21.15.1:0/2225319589 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c980cc7b0 0x7f2c980cf6c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.597+0000 7f2ca057e700 1 -- 172.21.15.1:0/2225319589 >> 172.21.15.1:0/2225319589 conn(0x7f2c980b2bf0 msgr2=0x7f2c980b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:31.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.597+0000 7f2ca057e700 1 -- 172.21.15.1:0/2225319589 shutdown_connections 2024-03-18T20:25:31.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.597+0000 7f2ca057e700 1 -- 172.21.15.1:0/2225319589 wait complete. 2024-03-18T20:25:31.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.598+0000 7f2ca057e700 1 Processor -- start 2024-03-18T20:25:31.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.598+0000 7f2ca057e700 1 -- start start 2024-03-18T20:25:31.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.599+0000 7f2ca057e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c980cc7b0 0x7f2c981498b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:31.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.599+0000 7f2ca057e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c980cfc00 0x7f2c980c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:31.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.599+0000 7f2ca057e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c9815b480 0x7f2c980c7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:31.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.599+0000 7f2ca057e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9814bab0 con 0x7f2c980cc7b0 2024-03-18T20:25:31.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.599+0000 7f2ca057e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c98159b60 con 0x7f2c980cfc00 2024-03-18T20:25:31.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.599+0000 7f2ca057e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9814ae30 con 0x7f2c9815b480 2024-03-18T20:25:31.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.599+0000 7f2c9e31a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c980cc7b0 0x7f2c981498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:31.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.599+0000 7f2c9e31a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c980cc7b0 0x7f2c981498b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60050/0 (socket says 172.21.15.1:60050) 2024-03-18T20:25:31.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.599+0000 7f2c9e31a700 1 -- 172.21.15.1:0/3704794476 learned_addr learned my addr 172.21.15.1:0/3704794476 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.599+0000 7f2c9db19700 1 --2- 172.21.15.1:0/3704794476 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c980cfc00 0x7f2c980c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.599+0000 7f2c9eb1b700 1 --2- 172.21.15.1:0/3704794476 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c9815b480 0x7f2c980c7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.599+0000 7f2c9e31a700 1 -- 172.21.15.1:0/3704794476 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c9815b480 msgr2=0x7f2c980c7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.599+0000 7f2c9e31a700 1 --2- 172.21.15.1:0/3704794476 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c9815b480 0x7f2c980c7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.599+0000 7f2c9e31a700 1 -- 172.21.15.1:0/3704794476 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c980cfc00 msgr2=0x7f2c980c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.599+0000 7f2c9e31a700 1 --2- 172.21.15.1:0/3704794476 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c980cfc00 0x7f2c980c6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.599+0000 7f2c9e31a700 1 -- 172.21.15.1:0/3704794476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c94009550 con 0x7f2c980cc7b0 2024-03-18T20:25:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.600+0000 7f2c9e31a700 1 --2- 172.21.15.1:0/3704794476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c980cc7b0 0x7f2c981498b0 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f2c8800c450 tx=0x7f2c8800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.600+0000 7f2c8effd700 1 -- 172.21.15.1:0/3704794476 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c880052d0 con 0x7f2c980cc7b0 2024-03-18T20:25:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.600+0000 7f2c8effd700 1 -- 172.21.15.1:0/3704794476 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c88005430 con 0x7f2c980cc7b0 2024-03-18T20:25:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.600+0000 7f2ca057e700 1 -- 172.21.15.1:0/3704794476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c9814b110 con 0x7f2c980cc7b0 2024-03-18T20:25:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.600+0000 7f2c8effd700 1 -- 172.21.15.1:0/3704794476 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c880038c0 con 0x7f2c980cc7b0 2024-03-18T20:25:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.600+0000 7f2ca057e700 1 -- 172.21.15.1:0/3704794476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c9814b6d0 con 0x7f2c980cc7b0 2024-03-18T20:25:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.601+0000 7f2ca057e700 1 -- 172.21.15.1:0/3704794476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c980676a0 con 0x7f2c980cc7b0 2024-03-18T20:25:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.601+0000 7f2c8effd700 1 -- 172.21.15.1:0/3704794476 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2c88003a20 con 0x7f2c980cc7b0 2024-03-18T20:25:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.602+0000 7f2c8effd700 1 --2- 172.21.15.1:0/3704794476 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2c7c06adb0 0x7f2c7c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.602+0000 7f2c8effd700 1 -- 172.21.15.1:0/3704794476 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f2c880888d0 con 0x7f2c980cc7b0 2024-03-18T20:25:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.602+0000 7f2c9db19700 1 --2- 172.21.15.1:0/3704794476 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2c7c06adb0 0x7f2c7c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.602+0000 7f2c9db19700 1 --2- 172.21.15.1:0/3704794476 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2c7c06adb0 0x7f2c7c06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009840 tx=0x7f2c94008030 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:31.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.606+0000 7f2c8effd700 1 -- 172.21.15.1:0/3704794476 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2c88053880 con 0x7f2c980cc7b0 2024-03-18T20:25:31.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.809+0000 7f2ca057e700 1 -- 172.21.15.1:0/3704794476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2c980c45a0 con 0x7f2c980cc7b0 2024-03-18T20:25:31.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.809+0000 7f2c8effd700 1 -- 172.21.15.1:0/3704794476 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v232) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2c880170a0 con 0x7f2c980cc7b0 2024-03-18T20:25:31.814 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:25:31.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.811+0000 7f2ca057e700 1 -- 172.21.15.1:0/3704794476 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2c7c06adb0 msgr2=0x7f2c7c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:31.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.811+0000 7f2ca057e700 1 --2- 172.21.15.1:0/3704794476 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2c7c06adb0 0x7f2c7c06d270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009840 tx=0x7f2c94008030 comp rx=0 tx=0).stop 2024-03-18T20:25:31.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.811+0000 7f2ca057e700 1 -- 172.21.15.1:0/3704794476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c980cc7b0 msgr2=0x7f2c981498b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:31.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.811+0000 7f2ca057e700 1 --2- 172.21.15.1:0/3704794476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c980cc7b0 0x7f2c981498b0 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f2c8800c450 tx=0x7f2c8800c760 comp rx=0 tx=0).stop 2024-03-18T20:25:31.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.811+0000 7f2ca057e700 1 -- 172.21.15.1:0/3704794476 shutdown_connections 2024-03-18T20:25:31.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.811+0000 7f2ca057e700 1 --2- 172.21.15.1:0/3704794476 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2c7c06adb0 0x7f2c7c06d270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.811+0000 7f2ca057e700 1 --2- 172.21.15.1:0/3704794476 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c9815b480 0x7f2c980c7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.811+0000 7f2ca057e700 1 --2- 172.21.15.1:0/3704794476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c980cc7b0 0x7f2c981498b0 unknown :-1 s=CLOSED pgs=2545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.811+0000 7f2ca057e700 1 --2- 172.21.15.1:0/3704794476 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c980cfc00 0x7f2c980c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.811+0000 7f2ca057e700 1 -- 172.21.15.1:0/3704794476 >> 172.21.15.1:0/3704794476 conn(0x7f2c980b2bf0 msgr2=0x7f2c980c5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:31.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.811+0000 7f2ca057e700 1 -- 172.21.15.1:0/3704794476 shutdown_connections 2024-03-18T20:25:31.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.812+0000 7f2ca057e700 1 -- 172.21.15.1:0/3704794476 wait complete. 2024-03-18T20:25:31.827 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:25:31.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.951+0000 7fd731bfa700 1 Processor -- start 2024-03-18T20:25:31.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.951+0000 7fd731bfa700 1 -- start start 2024-03-18T20:25:31.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.952+0000 7fd731bfa700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd72c0b1e30 0x7fd72c166210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:31.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.952+0000 7fd731bfa700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd72c15ef80 con 0x7fd72c15eba0 2024-03-18T20:25:31.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.952+0000 7fd731bfa700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd72c0cc9a0 con 0x7fd72c0b0500 2024-03-18T20:25:31.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.952+0000 7fd731bfa700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd72c0d0190 con 0x7fd72c0b1e30 2024-03-18T20:25:31.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.952+0000 7fd72b7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd72c15eba0 0x7fd72c0cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39536/0 (socket says 172.21.15.1:39536) 2024-03-18T20:25:31.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.952+0000 7fd72b7fe700 1 -- 172.21.15.1:0/1150715781 learned_addr learned my addr 172.21.15.1:0/1150715781 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:31.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.953+0000 7fd72bfff700 1 --2- 172.21.15.1:0/1150715781 >> v2:172.21.15.165:3300/0 conn(0x7fd72c0b1e30 0x7fd72c166210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:31.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.953+0000 7fd72a7fc700 1 -- 172.21.15.1:0/1150715781 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271052825 0 0) 0x7fd72c15ef80 con 0x7fd72c15eba0 2024-03-18T20:25:31.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.953+0000 7fd72a7fc700 1 -- 172.21.15.1:0/1150715781 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7180035b0 con 0x7fd72c15eba0 2024-03-18T20:25:31.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.953+0000 7fd72a7fc700 1 -- 172.21.15.1:0/1150715781 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 924716156 0 0) 0x7fd7180035b0 con 0x7fd72c15eba0 2024-03-18T20:25:31.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.953+0000 7fd72bfff700 1 -- 172.21.15.1:0/1150715781 >> v1:172.21.15.165:6789/0 conn(0x7fd72c0b0500 legacy=0x7fd72c0cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:31.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.953+0000 7fd72bfff700 1 -- 172.21.15.1:0/1150715781 >> v1:172.21.15.1:6789/0 conn(0x7fd72c15eba0 legacy=0x7fd72c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:31.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.953+0000 7fd72bfff700 1 -- 172.21.15.1:0/1150715781 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd72c0ccc80 con 0x7fd72c0b1e30 2024-03-18T20:25:31.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.953+0000 7fd72bfff700 1 --2- 172.21.15.1:0/1150715781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd72c0b1e30 0x7fd72c166210 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd724009d90 tx=0x7fd72400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c67b91199afb59f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:31.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.953+0000 7fd72a7fc700 1 -- 172.21.15.1:0/1150715781 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72400e070 con 0x7fd72c0b1e30 2024-03-18T20:25:31.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.953+0000 7fd72a7fc700 1 -- 172.21.15.1:0/1150715781 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd724003a20 con 0x7fd72c0b1e30 2024-03-18T20:25:31.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.953+0000 7fd72a7fc700 1 -- 172.21.15.1:0/1150715781 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd714001670 con 0x7fd72c0b1e30 2024-03-18T20:25:31.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.954+0000 7fd731bfa700 1 -- 172.21.15.1:0/1150715781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd72c0b1e30 msgr2=0x7fd72c166210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:31.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.954+0000 7fd731bfa700 1 --2- 172.21.15.1:0/1150715781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd72c0b1e30 0x7fd72c166210 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd724009d90 tx=0x7fd72400b330 comp rx=0 tx=0).stop 2024-03-18T20:25:31.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.954+0000 7fd731bfa700 1 -- 172.21.15.1:0/1150715781 shutdown_connections 2024-03-18T20:25:31.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.954+0000 7fd731bfa700 1 --2- 172.21.15.1:0/1150715781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd72c0b1e30 0x7fd72c166210 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.954+0000 7fd731bfa700 1 -- 172.21.15.1:0/1150715781 >> 172.21.15.1:0/1150715781 conn(0x7fd72c0b2bd0 msgr2=0x7fd72c0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:31.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.954+0000 7fd731bfa700 1 -- 172.21.15.1:0/1150715781 shutdown_connections 2024-03-18T20:25:31.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.954+0000 7fd731bfa700 1 -- 172.21.15.1:0/1150715781 wait complete. 2024-03-18T20:25:31.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.955+0000 7fd731bfa700 1 Processor -- start 2024-03-18T20:25:31.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.955+0000 7fd731bfa700 1 -- start start 2024-03-18T20:25:31.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.956+0000 7fd731bfa700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd72c0b0500 0x7fd72c14cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:31.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.956+0000 7fd731bfa700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd72c0b1e30 0x7fd72c14c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:31.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.956+0000 7fd731bfa700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd72c15eba0 0x7fd72c1515a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:31.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.956+0000 7fd731bfa700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c0add90 con 0x7fd72c0b0500 2024-03-18T20:25:31.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.956+0000 7fd731bfa700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c14d340 con 0x7fd72c15eba0 2024-03-18T20:25:31.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.956+0000 7fd731bfa700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c0adfe0 con 0x7fd72c0b1e30 2024-03-18T20:25:31.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.956+0000 7fd72b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd72c0b0500 0x7fd72c14cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:31.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.956+0000 7fd72b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd72c0b0500 0x7fd72c14cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60060/0 (socket says 172.21.15.1:60060) 2024-03-18T20:25:31.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.956+0000 7fd72b7fe700 1 -- 172.21.15.1:0/4083141821 learned_addr learned my addr 172.21.15.1:0/4083141821 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:31.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.956+0000 7fd72bfff700 1 --2- 172.21.15.1:0/4083141821 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd72c15eba0 0x7fd72c1515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:31.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.956+0000 7fd72affd700 1 --2- 172.21.15.1:0/4083141821 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd72c0b1e30 0x7fd72c14c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:31.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.956+0000 7fd72b7fe700 1 -- 172.21.15.1:0/4083141821 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd72c0b1e30 msgr2=0x7fd72c14c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:31.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.956+0000 7fd72b7fe700 1 --2- 172.21.15.1:0/4083141821 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd72c0b1e30 0x7fd72c14c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.957+0000 7fd72b7fe700 1 -- 172.21.15.1:0/4083141821 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd72c15eba0 msgr2=0x7fd72c1515a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:31.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.957+0000 7fd72b7fe700 1 --2- 172.21.15.1:0/4083141821 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd72c15eba0 0x7fd72c1515a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:31.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.957+0000 7fd72b7fe700 1 -- 172.21.15.1:0/4083141821 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd724009a70 con 0x7fd72c0b0500 2024-03-18T20:25:31.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.957+0000 7fd72b7fe700 1 --2- 172.21.15.1:0/4083141821 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd72c0b0500 0x7fd72c14cac0 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7fd71400c520 tx=0x7fd71400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:31.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.957+0000 7fd723fff700 1 -- 172.21.15.1:0/4083141821 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd714004c80 con 0x7fd72c0b0500 2024-03-18T20:25:31.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.957+0000 7fd723fff700 1 -- 172.21.15.1:0/4083141821 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd714004de0 con 0x7fd72c0b0500 2024-03-18T20:25:31.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.957+0000 7fd731bfa700 1 -- 172.21.15.1:0/4083141821 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd72c0ae2c0 con 0x7fd72c0b0500 2024-03-18T20:25:31.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.957+0000 7fd723fff700 1 -- 172.21.15.1:0/4083141821 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd714007310 con 0x7fd72c0b0500 2024-03-18T20:25:31.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.957+0000 7fd731bfa700 1 -- 172.21.15.1:0/4083141821 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd72c0ae880 con 0x7fd72c0b0500 2024-03-18T20:25:31.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.958+0000 7fd731bfa700 1 -- 172.21.15.1:0/4083141821 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd72c0aeaa0 con 0x7fd72c0b0500 2024-03-18T20:25:31.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.958+0000 7fd723fff700 1 -- 172.21.15.1:0/4083141821 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd714007470 con 0x7fd72c0b0500 2024-03-18T20:25:31.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.959+0000 7fd723fff700 1 --2- 172.21.15.1:0/4083141821 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd70c06adb0 0x7fd70c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:31.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.959+0000 7fd723fff700 1 -- 172.21.15.1:0/4083141821 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fd714089230 con 0x7fd72c0b0500 2024-03-18T20:25:31.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.959+0000 7fd72affd700 1 --2- 172.21.15.1:0/4083141821 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd70c06adb0 0x7fd70c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:31.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.959+0000 7fd72affd700 1 --2- 172.21.15.1:0/4083141821 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd70c06adb0 0x7fd70c06d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd72c151d50 tx=0x7fd71c009040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:31.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:31.963+0000 7fd723fff700 1 -- 172.21.15.1:0/4083141821 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd7140541e0 con 0x7fd72c0b0500 2024-03-18T20:25:32.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.166+0000 7fd731bfa700 1 -- 172.21.15.1:0/4083141821 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd72c0676a0 con 0x7fd72c0b0500 2024-03-18T20:25:32.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.166+0000 7fd723fff700 1 -- 172.21.15.1:0/4083141821 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v232) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd714057800 con 0x7fd72c0b0500 2024-03-18T20:25:32.170 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:25:32.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.168+0000 7fd731bfa700 1 -- 172.21.15.1:0/4083141821 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd70c06adb0 msgr2=0x7fd70c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:32.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.168+0000 7fd731bfa700 1 --2- 172.21.15.1:0/4083141821 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd70c06adb0 0x7fd70c06d270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd72c151d50 tx=0x7fd71c009040 comp rx=0 tx=0).stop 2024-03-18T20:25:32.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.168+0000 7fd731bfa700 1 -- 172.21.15.1:0/4083141821 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd72c0b0500 msgr2=0x7fd72c14cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:32.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.168+0000 7fd731bfa700 1 --2- 172.21.15.1:0/4083141821 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd72c0b0500 0x7fd72c14cac0 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7fd71400c520 tx=0x7fd71400c830 comp rx=0 tx=0).stop 2024-03-18T20:25:32.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.168+0000 7fd731bfa700 1 -- 172.21.15.1:0/4083141821 shutdown_connections 2024-03-18T20:25:32.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.168+0000 7fd731bfa700 1 --2- 172.21.15.1:0/4083141821 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd70c06adb0 0x7fd70c06d270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.168+0000 7fd731bfa700 1 --2- 172.21.15.1:0/4083141821 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd72c15eba0 0x7fd72c1515a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.168+0000 7fd731bfa700 1 --2- 172.21.15.1:0/4083141821 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd72c0b0500 0x7fd72c14cac0 unknown :-1 s=CLOSED pgs=2547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.168+0000 7fd731bfa700 1 --2- 172.21.15.1:0/4083141821 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd72c0b1e30 0x7fd72c14c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.168+0000 7fd731bfa700 1 -- 172.21.15.1:0/4083141821 >> 172.21.15.1:0/4083141821 conn(0x7fd72c0b2bd0 msgr2=0x7fd72c099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:32.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.168+0000 7fd731bfa700 1 -- 172.21.15.1:0/4083141821 shutdown_connections 2024-03-18T20:25:32.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.168+0000 7fd731bfa700 1 -- 172.21.15.1:0/4083141821 wait complete. 2024-03-18T20:25:32.184 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:25:32.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.308+0000 7f4749ab5700 1 Processor -- start 2024-03-18T20:25:32.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.308+0000 7f4749ab5700 1 -- start start 2024-03-18T20:25:32.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.309+0000 7f4749ab5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f474406ca80 0x7f474415d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:32.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.309+0000 7f4749ab5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f474409a150 0x7f47440cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:32.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.309+0000 7f4749ab5700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47440b3b00 con 0x7f474415d5b0 2024-03-18T20:25:32.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.309+0000 7f4749ab5700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47440cbbb0 con 0x7f474409a150 2024-03-18T20:25:32.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.309+0000 7f4749ab5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47440cf7f0 con 0x7f474406ca80 2024-03-18T20:25:32.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.309+0000 7f4743fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f474409a150 0x7f47440cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:32.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.309+0000 7f4743fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f474409a150 0x7f47440cbfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60074/0 (socket says 172.21.15.1:60074) 2024-03-18T20:25:32.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.309+0000 7f4743fff700 1 -- 172.21.15.1:0/1434341530 learned_addr learned my addr 172.21.15.1:0/1434341530 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:32.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.309+0000 7f47437fe700 1 --1- 172.21.15.1:0/1434341530 >> v1:172.21.15.1:6789/0 conn(0x7f474415d5b0 0x7f4744099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39538/0 (socket says 172.21.15.1:39538) 2024-03-18T20:25:32.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.309+0000 7f4742ffd700 1 --2- 172.21.15.1:0/1434341530 >> v2:172.21.15.165:3300/0 conn(0x7f474406ca80 0x7f474415d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:32.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.309+0000 7f4743fff700 1 -- 172.21.15.1:0/1434341530 >> v2:172.21.15.165:3300/0 conn(0x7f474406ca80 msgr2=0x7f474415d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:32.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.309+0000 7f4743fff700 1 --2- 172.21.15.1:0/1434341530 >> v2:172.21.15.165:3300/0 conn(0x7f474406ca80 0x7f474415d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.309+0000 7f4743fff700 1 -- 172.21.15.1:0/1434341530 >> v1:172.21.15.1:6789/0 conn(0x7f474415d5b0 legacy=0x7f4744099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:32.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.309+0000 7f4743fff700 1 -- 172.21.15.1:0/1434341530 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47440d04d0 con 0x7f474409a150 2024-03-18T20:25:32.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.309+0000 7f4742ffd700 1 --2- 172.21.15.1:0/1434341530 >> v2:172.21.15.165:3300/0 conn(0x7f474406ca80 0x7f474415d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:32.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.309+0000 7f47427fc700 1 -- 172.21.15.1:0/1434341530 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237519488 0 0) 0x7f47440b3b00 con 0x7f474415d5b0 2024-03-18T20:25:32.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.309+0000 7f4743fff700 1 --2- 172.21.15.1:0/1434341530 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474409a150 0x7f47440cbfd0 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f4744072c10 tx=0x7f473c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=446dcc857766bbe1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:32.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.310+0000 7f47427fc700 1 -- 172.21.15.1:0/1434341530 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c00e070 con 0x7f474409a150 2024-03-18T20:25:32.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.310+0000 7f47427fc700 1 -- 172.21.15.1:0/1434341530 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473c00b9c0 con 0x7f474409a150 2024-03-18T20:25:32.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.310+0000 7f47427fc700 1 -- 172.21.15.1:0/1434341530 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c00bd70 con 0x7f474409a150 2024-03-18T20:25:32.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.310+0000 7f4749ab5700 1 -- 172.21.15.1:0/1434341530 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474409a150 msgr2=0x7f47440cbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:32.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.310+0000 7f4749ab5700 1 --2- 172.21.15.1:0/1434341530 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474409a150 0x7f47440cbfd0 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f4744072c10 tx=0x7f473c00b330 comp rx=0 tx=0).stop 2024-03-18T20:25:32.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.310+0000 7f4749ab5700 1 -- 172.21.15.1:0/1434341530 shutdown_connections 2024-03-18T20:25:32.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.310+0000 7f4749ab5700 1 --2- 172.21.15.1:0/1434341530 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474409a150 0x7f47440cbfd0 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.310+0000 7f4749ab5700 1 --2- 172.21.15.1:0/1434341530 >> v2:172.21.15.165:3300/0 conn(0x7f474406ca80 0x7f474415d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.310+0000 7f4749ab5700 1 -- 172.21.15.1:0/1434341530 >> 172.21.15.1:0/1434341530 conn(0x7f47440b2bd0 msgr2=0x7f47440b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:32.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.310+0000 7f4749ab5700 1 -- 172.21.15.1:0/1434341530 shutdown_connections 2024-03-18T20:25:32.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.310+0000 7f4749ab5700 1 -- 172.21.15.1:0/1434341530 wait complete. 2024-03-18T20:25:32.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.311+0000 7f4749ab5700 1 Processor -- start 2024-03-18T20:25:32.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.312+0000 7f4749ab5700 1 -- start start 2024-03-18T20:25:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.312+0000 7f4749ab5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474406ca80 0x7f47440afa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.312+0000 7f4749ab5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f474409a150 0x7f47440ca920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.312+0000 7f4749ab5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f474415d5b0 0x7f47440cae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.312+0000 7f4749ab5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47440b6c60 con 0x7f474406ca80 2024-03-18T20:25:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.312+0000 7f4749ab5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47440b3d80 con 0x7f474415d5b0 2024-03-18T20:25:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.312+0000 7f4749ab5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47440b5ea0 con 0x7f474409a150 2024-03-18T20:25:32.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.312+0000 7f47437fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474406ca80 0x7f47440afa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:32.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.312+0000 7f47437fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474406ca80 0x7f47440afa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60080/0 (socket says 172.21.15.1:60080) 2024-03-18T20:25:32.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.312+0000 7f47437fe700 1 -- 172.21.15.1:0/302202956 learned_addr learned my addr 172.21.15.1:0/302202956 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:32.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.313+0000 7f4743fff700 1 --2- 172.21.15.1:0/302202956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f474415d5b0 0x7f47440cae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:32.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.313+0000 7f4742ffd700 1 --2- 172.21.15.1:0/302202956 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f474409a150 0x7f47440ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:32.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.313+0000 7f47437fe700 1 -- 172.21.15.1:0/302202956 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f474409a150 msgr2=0x7f47440ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:32.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.313+0000 7f47437fe700 1 --2- 172.21.15.1:0/302202956 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f474409a150 0x7f47440ca920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.313+0000 7f47437fe700 1 -- 172.21.15.1:0/302202956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f474415d5b0 msgr2=0x7f47440cae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:32.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.313+0000 7f47437fe700 1 --2- 172.21.15.1:0/302202956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f474415d5b0 0x7f47440cae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.313+0000 7f47437fe700 1 -- 172.21.15.1:0/302202956 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473c009b30 con 0x7f474406ca80 2024-03-18T20:25:32.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.313+0000 7f47437fe700 1 --2- 172.21.15.1:0/302202956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474406ca80 0x7f47440afa80 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f47340011e0 tx=0x7f473400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:32.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.313+0000 7f473bfff700 1 -- 172.21.15.1:0/302202956 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734004c80 con 0x7f474406ca80 2024-03-18T20:25:32.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.313+0000 7f473bfff700 1 -- 172.21.15.1:0/302202956 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4734004e00 con 0x7f474406ca80 2024-03-18T20:25:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.313+0000 7f4749ab5700 1 -- 172.21.15.1:0/302202956 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47440b6180 con 0x7f474406ca80 2024-03-18T20:25:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.313+0000 7f473bfff700 1 -- 172.21.15.1:0/302202956 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734007310 con 0x7f474406ca80 2024-03-18T20:25:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.313+0000 7f4749ab5700 1 -- 172.21.15.1:0/302202956 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47440b6740 con 0x7f474406ca80 2024-03-18T20:25:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.314+0000 7f4749ab5700 1 -- 172.21.15.1:0/302202956 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47440676a0 con 0x7f474406ca80 2024-03-18T20:25:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.314+0000 7f473bfff700 1 -- 172.21.15.1:0/302202956 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4734007470 con 0x7f474406ca80 2024-03-18T20:25:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.315+0000 7f473bfff700 1 --2- 172.21.15.1:0/302202956 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f472406adb0 0x7f472406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.315+0000 7f4742ffd700 1 --2- 172.21.15.1:0/302202956 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f472406adb0 0x7f472406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.315+0000 7f473bfff700 1 -- 172.21.15.1:0/302202956 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f47340893d0 con 0x7f474406ca80 2024-03-18T20:25:32.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.315+0000 7f4742ffd700 1 --2- 172.21.15.1:0/302202956 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f472406adb0 0x7f472406d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f472c006fd0 tx=0x7f472c008040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:32.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.320+0000 7f473bfff700 1 -- 172.21.15.1:0/302202956 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f47340542f0 con 0x7f474406ca80 2024-03-18T20:25:32.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.523+0000 7f4749ab5700 1 -- 172.21.15.1:0/302202956 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4744099dc0 con 0x7f474406ca80 2024-03-18T20:25:32.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.523+0000 7f473bfff700 1 -- 172.21.15.1:0/302202956 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v232) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4734057910 con 0x7f474406ca80 2024-03-18T20:25:32.527 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:25:32.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.525+0000 7f4749ab5700 1 -- 172.21.15.1:0/302202956 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f472406adb0 msgr2=0x7f472406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.525+0000 7f4749ab5700 1 --2- 172.21.15.1:0/302202956 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f472406adb0 0x7f472406d270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f472c006fd0 tx=0x7f472c008040 comp rx=0 tx=0).stop 2024-03-18T20:25:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.525+0000 7f4749ab5700 1 -- 172.21.15.1:0/302202956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474406ca80 msgr2=0x7f47440afa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.525+0000 7f4749ab5700 1 --2- 172.21.15.1:0/302202956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474406ca80 0x7f47440afa80 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f47340011e0 tx=0x7f473400cae0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.525+0000 7f4749ab5700 1 -- 172.21.15.1:0/302202956 shutdown_connections 2024-03-18T20:25:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.525+0000 7f4749ab5700 1 --2- 172.21.15.1:0/302202956 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f472406adb0 0x7f472406d270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.525+0000 7f4749ab5700 1 --2- 172.21.15.1:0/302202956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f474415d5b0 0x7f47440cae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.525+0000 7f4749ab5700 1 --2- 172.21.15.1:0/302202956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f474406ca80 0x7f47440afa80 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.525+0000 7f4749ab5700 1 --2- 172.21.15.1:0/302202956 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f474409a150 0x7f47440ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.525+0000 7f4749ab5700 1 -- 172.21.15.1:0/302202956 >> 172.21.15.1:0/302202956 conn(0x7f47440b2bd0 msgr2=0x7f474406e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.525+0000 7f4749ab5700 1 -- 172.21.15.1:0/302202956 shutdown_connections 2024-03-18T20:25:32.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.525+0000 7f4749ab5700 1 -- 172.21.15.1:0/302202956 wait complete. 2024-03-18T20:25:32.541 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T20:25:32.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.661+0000 7f3a59e6c700 1 Processor -- start 2024-03-18T20:25:32.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.661+0000 7f3a59e6c700 1 -- start start 2024-03-18T20:25:32.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.662+0000 7f3a59e6c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a5415d680 0x7f3a540cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:32.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.662+0000 7f3a59e6c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3a540d04b0 0x7f3a54099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:32.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.662+0000 7f3a59e6c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a540d0c80 con 0x7f3a54099f80 2024-03-18T20:25:32.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.662+0000 7f3a59e6c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a5409a430 con 0x7f3a540d04b0 2024-03-18T20:25:32.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.662+0000 7f3a59e6c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a540d09a0 con 0x7f3a5415d680 2024-03-18T20:25:32.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.662+0000 7f3a53fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3a54099f80 0x7f3a5406ba70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39554/0 (socket says 172.21.15.1:39554) 2024-03-18T20:25:32.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.662+0000 7f3a53fff700 1 -- 172.21.15.1:0/4050132372 learned_addr learned my addr 172.21.15.1:0/4050132372 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:32.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.662+0000 7f3a52ffd700 1 --2- 172.21.15.1:0/4050132372 >> v2:172.21.15.165:3300/0 conn(0x7f3a540d04b0 0x7f3a54099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:32.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.662+0000 7f3a537fe700 1 --2- 172.21.15.1:0/4050132372 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a5415d680 0x7f3a540cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:32.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.662+0000 7f3a527fc700 1 -- 172.21.15.1:0/4050132372 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958597103 0 0) 0x7f3a540d0c80 con 0x7f3a54099f80 2024-03-18T20:25:32.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.662+0000 7f3a527fc700 1 -- 172.21.15.1:0/4050132372 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a400035b0 con 0x7f3a54099f80 2024-03-18T20:25:32.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a527fc700 1 -- 172.21.15.1:0/4050132372 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 339696258 0 0) 0x7f3a400035b0 con 0x7f3a54099f80 2024-03-18T20:25:32.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a527fc700 1 -- 172.21.15.1:0/4050132372 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a5415d680 msgr2=0x7f3a540cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:32.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a527fc700 1 --2- 172.21.15.1:0/4050132372 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a5415d680 0x7f3a540cff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a527fc700 1 -- 172.21.15.1:0/4050132372 >> v2:172.21.15.165:3300/0 conn(0x7f3a540d04b0 msgr2=0x7f3a54099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:32.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a527fc700 1 --2- 172.21.15.1:0/4050132372 >> v2:172.21.15.165:3300/0 conn(0x7f3a540d04b0 0x7f3a54099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a527fc700 1 -- 172.21.15.1:0/4050132372 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a540cb830 con 0x7f3a54099f80 2024-03-18T20:25:32.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a527fc700 1 -- 172.21.15.1:0/4050132372 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f3a4c0019f0 con 0x7f3a54099f80 2024-03-18T20:25:32.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a527fc700 1 -- 172.21.15.1:0/4050132372 >> v1:172.21.15.1:6789/0 conn(0x7f3a54099f80 legacy=0x7f3a5406ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:32.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a527fc700 1 --2- 172.21.15.1:0/4050132372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a40003a00 0x7f3a40003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:32.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a527fc700 1 --2- 172.21.15.1:0/4050132372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a40004850 0x7f3a40004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:32.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a527fc700 1 --2- 172.21.15.1:0/4050132372 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a5415d680 0x7f3a40005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:32.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a527fc700 1 -- 172.21.15.1:0/4050132372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5409a430 con 0x7f3a40003a00 2024-03-18T20:25:32.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a527fc700 1 -- 172.21.15.1:0/4050132372 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a540d09a0 con 0x7f3a40004850 2024-03-18T20:25:32.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a527fc700 1 -- 172.21.15.1:0/4050132372 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4000ba10 con 0x7f3a5415d680 2024-03-18T20:25:32.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a537fe700 1 --2- 172.21.15.1:0/4050132372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a40003a00 0x7f3a40003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:32.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a537fe700 1 -- 172.21.15.1:0/4050132372 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a5415d680 msgr2=0x7f3a40005210 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T20:25:32.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a537fe700 1 --2- 172.21.15.1:0/4050132372 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a5415d680 0x7f3a40005210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a537fe700 1 -- 172.21.15.1:0/4050132372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a40004850 msgr2=0x7f3a40004cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T20:25:32.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a537fe700 1 --2- 172.21.15.1:0/4050132372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a40004850 0x7f3a40004cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a537fe700 1 -- 172.21.15.1:0/4050132372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a400035b0 con 0x7f3a40003a00 2024-03-18T20:25:32.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a537fe700 1 --2- 172.21.15.1:0/4050132372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a40003a00 0x7f3a40003e20 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009b60 tx=0x7f3a40005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df04aeabdcc59b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:32.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a527fc700 1 -- 172.21.15.1:0/4050132372 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4400e6c0 con 0x7f3a40003a00 2024-03-18T20:25:32.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.663+0000 7f3a527fc700 1 -- 172.21.15.1:0/4050132372 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a4400e820 con 0x7f3a40003a00 2024-03-18T20:25:32.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.664+0000 7f3a527fc700 1 -- 172.21.15.1:0/4050132372 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4400eb90 con 0x7f3a40003a00 2024-03-18T20:25:32.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.664+0000 7f3a59e6c700 1 -- 172.21.15.1:0/4050132372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a40003a00 msgr2=0x7f3a40003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:32.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.664+0000 7f3a59e6c700 1 --2- 172.21.15.1:0/4050132372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a40003a00 0x7f3a40003e20 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009b60 tx=0x7f3a40005880 comp rx=0 tx=0).stop 2024-03-18T20:25:32.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.664+0000 7f3a53fff700 1 -- 172.21.15.1:0/4050132372 reap_dead start 2024-03-18T20:25:32.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.664+0000 7f3a59e6c700 1 -- 172.21.15.1:0/4050132372 shutdown_connections 2024-03-18T20:25:32.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.664+0000 7f3a59e6c700 1 -- 172.21.15.1:0/4050132372 >> 172.21.15.1:0/4050132372 conn(0x7f3a540b2ba0 msgr2=0x7f3a540b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:32.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.664+0000 7f3a59e6c700 1 -- 172.21.15.1:0/4050132372 shutdown_connections 2024-03-18T20:25:32.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.664+0000 7f3a59e6c700 1 -- 172.21.15.1:0/4050132372 wait complete. 2024-03-18T20:25:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.666+0000 7f3a59e6c700 1 Processor -- start 2024-03-18T20:25:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.666+0000 7f3a59e6c700 1 -- start start 2024-03-18T20:25:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.667+0000 7f3a59e6c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a40003a00 0x7f3a54157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.667+0000 7f3a59e6c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a54158b70 0x7f3a54158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.667+0000 7f3a59e6c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a5415f920 0x7f3a5415fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.667+0000 7f3a59e6c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a541602e0 con 0x7f3a5415f920 2024-03-18T20:25:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.667+0000 7f3a59e6c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a540c8290 con 0x7f3a40003a00 2024-03-18T20:25:32.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.667+0000 7f3a59e6c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a540c8570 con 0x7f3a54158b70 2024-03-18T20:25:32.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.667+0000 7f3a53fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a5415f920 0x7f3a5415fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:32.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.667+0000 7f3a53fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a5415f920 0x7f3a5415fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60100/0 (socket says 172.21.15.1:60100) 2024-03-18T20:25:32.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.667+0000 7f3a53fff700 1 -- 172.21.15.1:0/1788320430 learned_addr learned my addr 172.21.15.1:0/1788320430 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:32.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.667+0000 7f3a52ffd700 1 --2- 172.21.15.1:0/1788320430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a54158b70 0x7f3a54158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:32.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.667+0000 7f3a537fe700 1 --2- 172.21.15.1:0/1788320430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a40003a00 0x7f3a54157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:32.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.667+0000 7f3a53fff700 1 -- 172.21.15.1:0/1788320430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a54158b70 msgr2=0x7f3a54158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:32.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.667+0000 7f3a53fff700 1 --2- 172.21.15.1:0/1788320430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a54158b70 0x7f3a54158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.667+0000 7f3a53fff700 1 -- 172.21.15.1:0/1788320430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a40003a00 msgr2=0x7f3a54157950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:32.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.667+0000 7f3a53fff700 1 --2- 172.21.15.1:0/1788320430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a40003a00 0x7f3a54157950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.667+0000 7f3a53fff700 1 -- 172.21.15.1:0/1788320430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5415dbe0 con 0x7f3a5415f920 2024-03-18T20:25:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.667+0000 7f3a537fe700 1 --2- 172.21.15.1:0/1788320430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a40003a00 0x7f3a54157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.667+0000 7f3a52ffd700 1 --2- 172.21.15.1:0/1788320430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a54158b70 0x7f3a54158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.667+0000 7f3a53fff700 1 --2- 172.21.15.1:0/1788320430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a5415f920 0x7f3a5415fda0 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c0013d0 tx=0x7f3a4c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.668+0000 7f3a4bfff700 1 -- 172.21.15.1:0/1788320430 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c00ad30 con 0x7f3a5415f920 2024-03-18T20:25:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.668+0000 7f3a4bfff700 1 -- 172.21.15.1:0/1788320430 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a4c00ae90 con 0x7f3a5415f920 2024-03-18T20:25:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.668+0000 7f3a59e6c700 1 -- 172.21.15.1:0/1788320430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a540c8850 con 0x7f3a5415f920 2024-03-18T20:25:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.668+0000 7f3a4bfff700 1 -- 172.21.15.1:0/1788320430 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c007270 con 0x7f3a5415f920 2024-03-18T20:25:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.668+0000 7f3a59e6c700 1 -- 172.21.15.1:0/1788320430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a540c8e10 con 0x7f3a5415f920 2024-03-18T20:25:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.669+0000 7f3a59e6c700 1 -- 172.21.15.1:0/1788320430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a540676a0 con 0x7f3a5415f920 2024-03-18T20:25:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.669+0000 7f3a4bfff700 1 -- 172.21.15.1:0/1788320430 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f3a4c0073d0 con 0x7f3a5415f920 2024-03-18T20:25:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.669+0000 7f3a4bfff700 1 --2- 172.21.15.1:0/1788320430 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3a3406adb0 0x7f3a3406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.670+0000 7f3a4bfff700 1 -- 172.21.15.1:0/1788320430 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f3a4c088fc0 con 0x7f3a5415f920 2024-03-18T20:25:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.670+0000 7f3a537fe700 1 --2- 172.21.15.1:0/1788320430 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3a3406adb0 0x7f3a3406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:32.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.670+0000 7f3a537fe700 1 --2- 172.21.15.1:0/1788320430 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3a3406adb0 0x7f3a3406d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3a440090b0 tx=0x7f3a44008030 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:32.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.674+0000 7f3a4bfff700 1 -- 172.21.15.1:0/1788320430 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3a4c053e60 con 0x7f3a5415f920 2024-03-18T20:25:32.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.878+0000 7f3a59e6c700 1 -- 172.21.15.1:0/1788320430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3a540b2450 con 0x7f3a5415f920 2024-03-18T20:25:32.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.879+0000 7f3a4bfff700 1 -- 172.21.15.1:0/1788320430 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v232) v1 ==== 84+0+209 (secure 0 0 0) 0x7f3a4c018090 con 0x7f3a5415f920 2024-03-18T20:25:32.883 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:32.883 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T20:25:32.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:32.883 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-18T20:25:32.883 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-18T20:25:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 27, 2024-03-18T20:25:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": "cephfs_metadata" 2024-03-18T20:25:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 28, 2024-03-18T20:25:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": "cephfs_data" 2024-03-18T20:25:32.884 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:32.884 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T20:25:32.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.881+0000 7f3a59e6c700 1 -- 172.21.15.1:0/1788320430 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3a3406adb0 msgr2=0x7f3a3406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:32.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.881+0000 7f3a59e6c700 1 --2- 172.21.15.1:0/1788320430 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3a3406adb0 0x7f3a3406d270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3a440090b0 tx=0x7f3a44008030 comp rx=0 tx=0).stop 2024-03-18T20:25:32.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.881+0000 7f3a59e6c700 1 -- 172.21.15.1:0/1788320430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a5415f920 msgr2=0x7f3a5415fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:32.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.881+0000 7f3a59e6c700 1 --2- 172.21.15.1:0/1788320430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a5415f920 0x7f3a5415fda0 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c0013d0 tx=0x7f3a4c00a340 comp rx=0 tx=0).stop 2024-03-18T20:25:32.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.881+0000 7f3a59e6c700 1 -- 172.21.15.1:0/1788320430 shutdown_connections 2024-03-18T20:25:32.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.881+0000 7f3a59e6c700 1 --2- 172.21.15.1:0/1788320430 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3a3406adb0 0x7f3a3406d270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.881+0000 7f3a59e6c700 1 --2- 172.21.15.1:0/1788320430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a5415f920 0x7f3a5415fda0 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.881+0000 7f3a59e6c700 1 --2- 172.21.15.1:0/1788320430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a40003a00 0x7f3a54157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.881+0000 7f3a59e6c700 1 --2- 172.21.15.1:0/1788320430 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3a54158b70 0x7f3a54158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:32.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.881+0000 7f3a59e6c700 1 -- 172.21.15.1:0/1788320430 >> 172.21.15.1:0/1788320430 conn(0x7f3a540b2ba0 msgr2=0x7f3a540cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:32.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.881+0000 7f3a59e6c700 1 -- 172.21.15.1:0/1788320430 shutdown_connections 2024-03-18T20:25:32.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:32.881+0000 7f3a59e6c700 1 -- 172.21.15.1:0/1788320430 wait complete. 2024-03-18T20:25:32.898 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2024-03-18T20:25:32.898 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2024-03-18T20:25:33.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.023+0000 7f21f6e22700 1 Processor -- start 2024-03-18T20:25:33.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.023+0000 7f21f6e22700 1 -- start start 2024-03-18T20:25:33.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.024+0000 7f21f6e22700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f21f015e8d0 0x7f21f015a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:33.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.024+0000 7f21f6e22700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f21f015dca0 0x7f21f00b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:33.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.024+0000 7f21f6e22700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21f015ac60 con 0x7f21f00b3d60 2024-03-18T20:25:33.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.024+0000 7f21f6e22700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21f00701d0 con 0x7f21f015e8d0 2024-03-18T20:25:33.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.024+0000 7f21f6e22700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21f00b30b0 con 0x7f21f015dca0 2024-03-18T20:25:33.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.024+0000 7f21f4bbe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f21f015e8d0 0x7f21f015a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:33.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.024+0000 7f21f4bbe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f21f015e8d0 0x7f21f015a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60110/0 (socket says 172.21.15.1:60110) 2024-03-18T20:25:33.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.024+0000 7f21f4bbe700 1 -- 172.21.15.1:0/2346838343 learned_addr learned my addr 172.21.15.1:0/2346838343 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:33.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.025+0000 7f21effff700 1 --2- 172.21.15.1:0/2346838343 >> v2:172.21.15.165:3300/0 conn(0x7f21f015dca0 0x7f21f00b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:33.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.025+0000 7f21f4bbe700 1 -- 172.21.15.1:0/2346838343 >> v2:172.21.15.165:3300/0 conn(0x7f21f015dca0 msgr2=0x7f21f00b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:33.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.025+0000 7f21f4bbe700 1 --2- 172.21.15.1:0/2346838343 >> v2:172.21.15.165:3300/0 conn(0x7f21f015dca0 0x7f21f00b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:33.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.025+0000 7f21f4bbe700 1 -- 172.21.15.1:0/2346838343 >> v1:172.21.15.165:6789/0 conn(0x7f21f00b3d60 legacy=0x7f21f00717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:33.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.025+0000 7f21f4bbe700 1 -- 172.21.15.1:0/2346838343 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f00b3390 con 0x7f21f015e8d0 2024-03-18T20:25:33.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.025+0000 7f21f4bbe700 1 --2- 172.21.15.1:0/2346838343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21f015e8d0 0x7f21f015a720 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f21e40098f0 tx=0x7f21e4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d982bc06c6bcfb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:33.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.025+0000 7f21ef7fe700 1 -- 172.21.15.1:0/2346838343 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e400b860 con 0x7f21f015e8d0 2024-03-18T20:25:33.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.025+0000 7f21ef7fe700 1 -- 172.21.15.1:0/2346838343 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400b9c0 con 0x7f21f015e8d0 2024-03-18T20:25:33.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.025+0000 7f21ef7fe700 1 -- 172.21.15.1:0/2346838343 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e400bd50 con 0x7f21f015e8d0 2024-03-18T20:25:33.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.026+0000 7f21f6e22700 1 -- 172.21.15.1:0/2346838343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21f015e8d0 msgr2=0x7f21f015a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:33.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.026+0000 7f21f6e22700 1 --2- 172.21.15.1:0/2346838343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21f015e8d0 0x7f21f015a720 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f21e40098f0 tx=0x7f21e4009c00 comp rx=0 tx=0).stop 2024-03-18T20:25:33.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.026+0000 7f21f6e22700 1 -- 172.21.15.1:0/2346838343 shutdown_connections 2024-03-18T20:25:33.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.026+0000 7f21f6e22700 1 --2- 172.21.15.1:0/2346838343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21f015e8d0 0x7f21f015a720 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:33.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.026+0000 7f21f6e22700 1 --2- 172.21.15.1:0/2346838343 >> v2:172.21.15.165:3300/0 conn(0x7f21f015dca0 0x7f21f00b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:33.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.026+0000 7f21f6e22700 1 -- 172.21.15.1:0/2346838343 >> 172.21.15.1:0/2346838343 conn(0x7f21f00d0080 msgr2=0x7f21f00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:33.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.026+0000 7f21f6e22700 1 -- 172.21.15.1:0/2346838343 shutdown_connections 2024-03-18T20:25:33.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.026+0000 7f21f6e22700 1 -- 172.21.15.1:0/2346838343 wait complete. 2024-03-18T20:25:33.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.027+0000 7f21f6e22700 1 Processor -- start 2024-03-18T20:25:33.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.027+0000 7f21f6e22700 1 -- start start 2024-03-18T20:25:33.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21f6e22700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21f00b3d60 0x7f21f0152630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:33.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21f6e22700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21f015dca0 0x7f21f01e4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:33.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21f6e22700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21f015e8d0 0x7f21f01590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:33.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21f6e22700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f0153890 con 0x7f21f015e8d0 2024-03-18T20:25:33.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21f6e22700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f0152c80 con 0x7f21f015dca0 2024-03-18T20:25:33.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21f6e22700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f0152f60 con 0x7f21f00b3d60 2024-03-18T20:25:33.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21f53bf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21f015e8d0 0x7f21f01590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:33.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21f53bf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21f015e8d0 0x7f21f01590f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60114/0 (socket says 172.21.15.1:60114) 2024-03-18T20:25:33.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21f53bf700 1 -- 172.21.15.1:0/27779099 learned_addr learned my addr 172.21.15.1:0/27779099 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:33.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21effff700 1 --2- 172.21.15.1:0/27779099 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21f015dca0 0x7f21f01e4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:33.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21f4bbe700 1 --2- 172.21.15.1:0/27779099 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21f00b3d60 0x7f21f0152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:33.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21f53bf700 1 -- 172.21.15.1:0/27779099 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21f00b3d60 msgr2=0x7f21f0152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:33.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21f53bf700 1 --2- 172.21.15.1:0/27779099 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21f00b3d60 0x7f21f0152630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:33.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21f53bf700 1 -- 172.21.15.1:0/27779099 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21f015dca0 msgr2=0x7f21f01e4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:33.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21f53bf700 1 --2- 172.21.15.1:0/27779099 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21f015dca0 0x7f21f01e4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:33.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21f53bf700 1 -- 172.21.15.1:0/27779099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e4009550 con 0x7f21f015e8d0 2024-03-18T20:25:33.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21effff700 1 --2- 172.21.15.1:0/27779099 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21f015dca0 0x7f21f01e4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:33.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21f4bbe700 1 --2- 172.21.15.1:0/27779099 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21f00b3d60 0x7f21f0152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:33.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21f53bf700 1 --2- 172.21.15.1:0/27779099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21f015e8d0 0x7f21f01590f0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f21e800cb90 tx=0x7f21e800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:33.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.028+0000 7f21ed7fa700 1 -- 172.21.15.1:0/27779099 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e8003c00 con 0x7f21f015e8d0 2024-03-18T20:25:33.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.029+0000 7f21ed7fa700 1 -- 172.21.15.1:0/27779099 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e8003d60 con 0x7f21f015e8d0 2024-03-18T20:25:33.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.029+0000 7f21f6e22700 1 -- 172.21.15.1:0/27779099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f0153240 con 0x7f21f015e8d0 2024-03-18T20:25:33.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.029+0000 7f21ed7fa700 1 -- 172.21.15.1:0/27779099 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e800abb0 con 0x7f21f015e8d0 2024-03-18T20:25:33.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.029+0000 7f21f6e22700 1 -- 172.21.15.1:0/27779099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f01e4c60 con 0x7f21f015e8d0 2024-03-18T20:25:33.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.029+0000 7f21f6e22700 1 -- 172.21.15.1:0/27779099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21f00cd070 con 0x7f21f015e8d0 2024-03-18T20:25:33.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.030+0000 7f21ed7fa700 1 -- 172.21.15.1:0/27779099 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f21e800ad10 con 0x7f21f015e8d0 2024-03-18T20:25:33.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.030+0000 7f21ed7fa700 1 --2- 172.21.15.1:0/27779099 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f21d406adb0 0x7f21d406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:33.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.030+0000 7f21ed7fa700 1 -- 172.21.15.1:0/27779099 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f21e80896e0 con 0x7f21f015e8d0 2024-03-18T20:25:33.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.031+0000 7f21f4bbe700 1 --2- 172.21.15.1:0/27779099 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f21d406adb0 0x7f21d406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:33.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.031+0000 7f21f4bbe700 1 --2- 172.21.15.1:0/27779099 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f21d406adb0 0x7f21d406d270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f21e4014040 tx=0x7f21e4008030 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:33.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.035+0000 7f21ed7fa700 1 -- 172.21.15.1:0/27779099 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f21e8054600 con 0x7f21f015e8d0 2024-03-18T20:25:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:33.241+0000 7f21f6e22700 1 -- 172.21.15.1:0/27779099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f21f015b150 con 0x7f21f015e8d0 2024-03-18T20:25:34.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.108+0000 7f21ed7fa700 1 -- 172.21.15.1:0/27779099 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v233) v1 ==== 144+0+0 (secure 0 0 0) 0x7f21f015b150 con 0x7f21f015e8d0 2024-03-18T20:25:34.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.111+0000 7f21f6e22700 1 -- 172.21.15.1:0/27779099 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f21d406adb0 msgr2=0x7f21d406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:34.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.111+0000 7f21f6e22700 1 --2- 172.21.15.1:0/27779099 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f21d406adb0 0x7f21d406d270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f21e4014040 tx=0x7f21e4008030 comp rx=0 tx=0).stop 2024-03-18T20:25:34.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.111+0000 7f21f6e22700 1 -- 172.21.15.1:0/27779099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21f015e8d0 msgr2=0x7f21f01590f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:34.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.111+0000 7f21f6e22700 1 --2- 172.21.15.1:0/27779099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21f015e8d0 0x7f21f01590f0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f21e800cb90 tx=0x7f21e800cf50 comp rx=0 tx=0).stop 2024-03-18T20:25:34.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.111+0000 7f21f6e22700 1 -- 172.21.15.1:0/27779099 shutdown_connections 2024-03-18T20:25:34.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.111+0000 7f21f6e22700 1 --2- 172.21.15.1:0/27779099 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f21d406adb0 0x7f21d406d270 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:34.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.111+0000 7f21f6e22700 1 --2- 172.21.15.1:0/27779099 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f21f00b3d60 0x7f21f0152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:34.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.111+0000 7f21f6e22700 1 --2- 172.21.15.1:0/27779099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f21f015e8d0 0x7f21f01590f0 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:34.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.111+0000 7f21f6e22700 1 --2- 172.21.15.1:0/27779099 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21f015dca0 0x7f21f01e4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:34.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.112+0000 7f21f6e22700 1 -- 172.21.15.1:0/27779099 >> 172.21.15.1:0/27779099 conn(0x7f21f00d0080 msgr2=0x7f21f00b0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:34.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.112+0000 7f21f6e22700 1 -- 172.21.15.1:0/27779099 shutdown_connections 2024-03-18T20:25:34.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.112+0000 7f21f6e22700 1 -- 172.21.15.1:0/27779099 wait complete. 2024-03-18T20:25:34.120 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'backup_fs_metadata' created 2024-03-18T20:25:34.135 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T20:25:34.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.270+0000 7f4c9be61700 1 Processor -- start 2024-03-18T20:25:34.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.270+0000 7f4c9be61700 1 -- start start 2024-03-18T20:25:34.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.271+0000 7f4c9be61700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c94071f20 0x7f4c94071420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:34.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.271+0000 7f4c9be61700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c94072300 0x7f4c940cf380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:34.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.271+0000 7f4c9be61700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c9406f3b0 con 0x7f4c940af030 2024-03-18T20:25:34.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.271+0000 7f4c9be61700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c940cdf40 con 0x7f4c94071f20 2024-03-18T20:25:34.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.271+0000 7f4c9be61700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9406e4c0 con 0x7f4c94072300 2024-03-18T20:25:34.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.271+0000 7f4c99bfd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c94071f20 0x7f4c94071420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:34.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.271+0000 7f4c99bfd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c94071f20 0x7f4c94071420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60128/0 (socket says 172.21.15.1:60128) 2024-03-18T20:25:34.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.271+0000 7f4c99bfd700 1 -- 172.21.15.1:0/701460512 learned_addr learned my addr 172.21.15.1:0/701460512 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:34.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.272+0000 7f4c993fc700 1 --2- 172.21.15.1:0/701460512 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c94072300 0x7f4c940cf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:34.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.272+0000 7f4c99bfd700 1 -- 172.21.15.1:0/701460512 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c94072300 msgr2=0x7f4c940cf380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:34.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.272+0000 7f4c99bfd700 1 --2- 172.21.15.1:0/701460512 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c94072300 0x7f4c940cf380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:34.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.272+0000 7f4c99bfd700 1 -- 172.21.15.1:0/701460512 >> v1:172.21.15.1:6789/0 conn(0x7f4c940af030 legacy=0x7f4c94161310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:34.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.272+0000 7f4c99bfd700 1 -- 172.21.15.1:0/701460512 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9406e7a0 con 0x7f4c94071f20 2024-03-18T20:25:34.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.272+0000 7f4c98bfb700 1 -- 172.21.15.1:0/701460512 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501873397 0 0) 0x7f4c9406f3b0 con 0x7f4c940af030 2024-03-18T20:25:34.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.272+0000 7f4c99bfd700 1 --2- 172.21.15.1:0/701460512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c94071f20 0x7f4c94071420 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f4c840098f0 tx=0x7f4c84009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d802600faa33f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:34.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.272+0000 7f4c98bfb700 1 -- 172.21.15.1:0/701460512 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8400b8a0 con 0x7f4c94071f20 2024-03-18T20:25:34.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.272+0000 7f4c98bfb700 1 -- 172.21.15.1:0/701460512 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8400ba00 con 0x7f4c94071f20 2024-03-18T20:25:34.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.272+0000 7f4c98bfb700 1 -- 172.21.15.1:0/701460512 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8400bd90 con 0x7f4c94071f20 2024-03-18T20:25:34.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.273+0000 7f4c9be61700 1 -- 172.21.15.1:0/701460512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c94071f20 msgr2=0x7f4c94071420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:34.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.273+0000 7f4c9be61700 1 --2- 172.21.15.1:0/701460512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c94071f20 0x7f4c94071420 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f4c840098f0 tx=0x7f4c84009c00 comp rx=0 tx=0).stop 2024-03-18T20:25:34.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.273+0000 7f4c9be61700 1 -- 172.21.15.1:0/701460512 shutdown_connections 2024-03-18T20:25:34.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.273+0000 7f4c9be61700 1 --2- 172.21.15.1:0/701460512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c94071f20 0x7f4c94071420 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:34.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.273+0000 7f4c9be61700 1 --2- 172.21.15.1:0/701460512 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c94072300 0x7f4c940cf380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:34.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.273+0000 7f4c9be61700 1 -- 172.21.15.1:0/701460512 >> 172.21.15.1:0/701460512 conn(0x7f4c940cca50 msgr2=0x7f4c940cce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:34.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.273+0000 7f4c9be61700 1 -- 172.21.15.1:0/701460512 shutdown_connections 2024-03-18T20:25:34.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.273+0000 7f4c9be61700 1 -- 172.21.15.1:0/701460512 wait complete. 2024-03-18T20:25:34.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.274+0000 7f4c9be61700 1 Processor -- start 2024-03-18T20:25:34.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.274+0000 7f4c9be61700 1 -- start start 2024-03-18T20:25:34.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.275+0000 7f4c9be61700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c94071f20 0x7f4c9415a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:34.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.275+0000 7f4c9be61700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c94072300 0x7f4c9415dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:34.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.275+0000 7f4c9be61700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c940af030 0x7f4c9414ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:34.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.275+0000 7f4c9be61700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c940c8d30 con 0x7f4c94072300 2024-03-18T20:25:34.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.275+0000 7f4c9be61700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c940c8190 con 0x7f4c940af030 2024-03-18T20:25:34.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.275+0000 7f4c9be61700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c940c8400 con 0x7f4c94071f20 2024-03-18T20:25:34.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.275+0000 7f4c993fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c94072300 0x7f4c9415dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:34.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.275+0000 7f4c993fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c94072300 0x7f4c9415dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60134/0 (socket says 172.21.15.1:60134) 2024-03-18T20:25:34.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.275+0000 7f4c993fc700 1 -- 172.21.15.1:0/3052106095 learned_addr learned my addr 172.21.15.1:0/3052106095 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:34.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.275+0000 7f4c993fc700 1 -- 172.21.15.1:0/3052106095 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c94071f20 msgr2=0x7f4c9415a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:34.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.275+0000 7f4c993fc700 1 --2- 172.21.15.1:0/3052106095 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c94071f20 0x7f4c9415a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:34.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.275+0000 7f4c993fc700 1 -- 172.21.15.1:0/3052106095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c940af030 msgr2=0x7f4c9414ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:34.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.275+0000 7f4c993fc700 1 --2- 172.21.15.1:0/3052106095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c940af030 0x7f4c9414ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:34.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.275+0000 7f4c993fc700 1 -- 172.21.15.1:0/3052106095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c84009550 con 0x7f4c94072300 2024-03-18T20:25:34.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.275+0000 7f4c993fc700 1 --2- 172.21.15.1:0/3052106095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c94072300 0x7f4c9415dbe0 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f4c9000e810 tx=0x7f4c9000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:34.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.275+0000 7f4c8a7fc700 1 -- 172.21.15.1:0/3052106095 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c90013610 con 0x7f4c94072300 2024-03-18T20:25:34.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.276+0000 7f4c8a7fc700 1 -- 172.21.15.1:0/3052106095 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c90013770 con 0x7f4c94072300 2024-03-18T20:25:34.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.276+0000 7f4c8a7fc700 1 -- 172.21.15.1:0/3052106095 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c90013b00 con 0x7f4c94072300 2024-03-18T20:25:34.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.276+0000 7f4c9be61700 1 -- 172.21.15.1:0/3052106095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c940c86e0 con 0x7f4c94072300 2024-03-18T20:25:34.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.276+0000 7f4c9be61700 1 -- 172.21.15.1:0/3052106095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c9415e120 con 0x7f4c94072300 2024-03-18T20:25:34.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.276+0000 7f4c9be61700 1 -- 172.21.15.1:0/3052106095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c94067bc0 con 0x7f4c94072300 2024-03-18T20:25:34.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.277+0000 7f4c8a7fc700 1 -- 172.21.15.1:0/3052106095 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4c90013c60 con 0x7f4c94072300 2024-03-18T20:25:34.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.278+0000 7f4c8a7fc700 1 --2- 172.21.15.1:0/3052106095 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c7806adb0 0x7f4c7806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:34.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.278+0000 7f4c8a7fc700 1 -- 172.21.15.1:0/3052106095 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 6502+0+0 (secure 0 0 0) 0x7f4c90089930 con 0x7f4c94072300 2024-03-18T20:25:34.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.278+0000 7f4c99bfd700 1 --2- 172.21.15.1:0/3052106095 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c7806adb0 0x7f4c7806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:34.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.278+0000 7f4c99bfd700 1 --2- 172.21.15.1:0/3052106095 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c7806adb0 0x7f4c7806d270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4c9415fd50 tx=0x7f4c84010000 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:34.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.282+0000 7f4c8a7fc700 1 -- 172.21.15.1:0/3052106095 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4c900546b0 con 0x7f4c94072300 2024-03-18T20:25:34.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:34.484+0000 7f4c9be61700 1 -- 172.21.15.1:0/3052106095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4c940c8e70 con 0x7f4c94072300 2024-03-18T20:25:35.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.120+0000 7f4c8a7fc700 1 -- 172.21.15.1:0/3052106095 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v234) v1 ==== 176+0+0 (secure 0 0 0) 0x7f4c940c8e70 con 0x7f4c94072300 2024-03-18T20:25:35.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.126+0000 7f4c9be61700 1 -- 172.21.15.1:0/3052106095 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c7806adb0 msgr2=0x7f4c7806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:35.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.126+0000 7f4c9be61700 1 --2- 172.21.15.1:0/3052106095 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c7806adb0 0x7f4c7806d270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4c9415fd50 tx=0x7f4c84010000 comp rx=0 tx=0).stop 2024-03-18T20:25:35.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.126+0000 7f4c9be61700 1 -- 172.21.15.1:0/3052106095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c94072300 msgr2=0x7f4c9415dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:35.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.126+0000 7f4c9be61700 1 --2- 172.21.15.1:0/3052106095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c94072300 0x7f4c9415dbe0 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f4c9000e810 tx=0x7f4c9000ebd0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.126+0000 7f4c9be61700 1 -- 172.21.15.1:0/3052106095 shutdown_connections 2024-03-18T20:25:35.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.126+0000 7f4c9be61700 1 --2- 172.21.15.1:0/3052106095 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4c7806adb0 0x7f4c7806d270 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.126+0000 7f4c9be61700 1 --2- 172.21.15.1:0/3052106095 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4c94071f20 0x7f4c9415a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.126+0000 7f4c9be61700 1 --2- 172.21.15.1:0/3052106095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c940af030 0x7f4c9414ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.126+0000 7f4c9be61700 1 --2- 172.21.15.1:0/3052106095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c94072300 0x7f4c9415dbe0 secure :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f4c9000e810 tx=0x7f4c9000ebd0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.126+0000 7f4c9be61700 1 -- 172.21.15.1:0/3052106095 >> 172.21.15.1:0/3052106095 conn(0x7f4c940cca50 msgr2=0x7f4c94071420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:35.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.130+0000 7f4c9be61700 1 -- 172.21.15.1:0/3052106095 shutdown_connections 2024-03-18T20:25:35.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.132+0000 7f4c9be61700 1 -- 172.21.15.1:0/3052106095 wait complete. 2024-03-18T20:25:35.140 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'backup_fs_data' created 2024-03-18T20:25:35.157 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2024-03-18T20:25:35.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.287+0000 7f75e3aff700 1 Processor -- start 2024-03-18T20:25:35.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.288+0000 7f75e3aff700 1 -- start start 2024-03-18T20:25:35.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.288+0000 7f75e3aff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f75dc15d5a0 0x7f75dc0ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:35.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.288+0000 7f75e3aff700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f75dc0ce550 0x7f75dc06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:35.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.288+0000 7f75e3aff700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc0cf600 con 0x7f75dc0cfa90 2024-03-18T20:25:35.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.288+0000 7f75e3aff700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75dc0ce9f0 con 0x7f75dc15d5a0 2024-03-18T20:25:35.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.288+0000 7f75e3aff700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75dc0cecd0 con 0x7f75dc0ce550 2024-03-18T20:25:35.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.289+0000 7f75e189b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f75dc15d5a0 0x7f75dc0ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:35.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.289+0000 7f75e189b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f75dc15d5a0 0x7f75dc0ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49728/0 (socket says 172.21.15.1:49728) 2024-03-18T20:25:35.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.289+0000 7f75e189b700 1 -- 172.21.15.1:0/4006004383 learned_addr learned my addr 172.21.15.1:0/4006004383 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.289+0000 7f75e109a700 1 --2- 172.21.15.1:0/4006004383 >> v2:172.21.15.165:3300/0 conn(0x7f75dc0ce550 0x7f75dc06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.289+0000 7f75e189b700 1 -- 172.21.15.1:0/4006004383 >> v2:172.21.15.165:3300/0 conn(0x7f75dc0ce550 msgr2=0x7f75dc06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.289+0000 7f75e189b700 1 --2- 172.21.15.1:0/4006004383 >> v2:172.21.15.165:3300/0 conn(0x7f75dc0ce550 0x7f75dc06a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.289+0000 7f75e189b700 1 -- 172.21.15.1:0/4006004383 >> v1:172.21.15.165:6789/0 conn(0x7f75dc0cfa90 legacy=0x7f75dc071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.289+0000 7f75e189b700 1 -- 172.21.15.1:0/4006004383 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75dc0cefb0 con 0x7f75dc15d5a0 2024-03-18T20:25:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.289+0000 7f75e109a700 1 --2- 172.21.15.1:0/4006004383 >> v2:172.21.15.165:3300/0 conn(0x7f75dc0ce550 0x7f75dc06a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.289+0000 7f75e189b700 1 --2- 172.21.15.1:0/4006004383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75dc15d5a0 0x7f75dc0ce010 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f75dc0af730 tx=0x7f75d0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cadcce2ea158c584 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.289+0000 7f75e0899700 1 -- 172.21.15.1:0/4006004383 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d0005e30 con 0x7f75dc15d5a0 2024-03-18T20:25:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.289+0000 7f75e0899700 1 -- 172.21.15.1:0/4006004383 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d0006030 con 0x7f75dc15d5a0 2024-03-18T20:25:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.289+0000 7f75e0899700 1 -- 172.21.15.1:0/4006004383 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d00063e0 con 0x7f75dc15d5a0 2024-03-18T20:25:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.290+0000 7f75e3aff700 1 -- 172.21.15.1:0/4006004383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75dc15d5a0 msgr2=0x7f75dc0ce010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.290+0000 7f75e3aff700 1 --2- 172.21.15.1:0/4006004383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75dc15d5a0 0x7f75dc0ce010 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f75dc0af730 tx=0x7f75d0009bd0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.290+0000 7f75e3aff700 1 -- 172.21.15.1:0/4006004383 shutdown_connections 2024-03-18T20:25:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.290+0000 7f75e3aff700 1 --2- 172.21.15.1:0/4006004383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75dc15d5a0 0x7f75dc0ce010 unknown :-1 s=CLOSED pgs=2559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.290+0000 7f75e3aff700 1 --2- 172.21.15.1:0/4006004383 >> v2:172.21.15.165:3300/0 conn(0x7f75dc0ce550 0x7f75dc06a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.290+0000 7f75e3aff700 1 -- 172.21.15.1:0/4006004383 >> 172.21.15.1:0/4006004383 conn(0x7f75dc0cc9b0 msgr2=0x7f75dc0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.290+0000 7f75e3aff700 1 -- 172.21.15.1:0/4006004383 shutdown_connections 2024-03-18T20:25:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.290+0000 7f75e3aff700 1 -- 172.21.15.1:0/4006004383 wait complete. 2024-03-18T20:25:35.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.291+0000 7f75e3aff700 1 Processor -- start 2024-03-18T20:25:35.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.291+0000 7f75e3aff700 1 -- start start 2024-03-18T20:25:35.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.292+0000 7f75e3aff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f75dc0ce550 0x7f75dc15b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:35.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.292+0000 7f75e3aff700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75dc0cfa90 0x7f75dc0adff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:35.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.292+0000 7f75e3aff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75dc15d5a0 0x7f75dc0ae530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:35.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.292+0000 7f75e3aff700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc0b3910 con 0x7f75dc15d5a0 2024-03-18T20:25:35.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.292+0000 7f75e3aff700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc0b2d00 con 0x7f75dc0cfa90 2024-03-18T20:25:35.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.292+0000 7f75e3aff700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc0b2fe0 con 0x7f75dc0ce550 2024-03-18T20:25:35.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.292+0000 7f75e209c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75dc15d5a0 0x7f75dc0ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:35.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.292+0000 7f75e209c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75dc15d5a0 0x7f75dc0ae530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49740/0 (socket says 172.21.15.1:49740) 2024-03-18T20:25:35.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.292+0000 7f75e209c700 1 -- 172.21.15.1:0/423806883 learned_addr learned my addr 172.21.15.1:0/423806883 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:35.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.292+0000 7f75e189b700 1 --2- 172.21.15.1:0/423806883 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f75dc0ce550 0x7f75dc15b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:35.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.292+0000 7f75e109a700 1 --2- 172.21.15.1:0/423806883 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75dc0cfa90 0x7f75dc0adff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:35.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.292+0000 7f75e209c700 1 -- 172.21.15.1:0/423806883 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f75dc0ce550 msgr2=0x7f75dc15b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:35.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.292+0000 7f75e209c700 1 --2- 172.21.15.1:0/423806883 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f75dc0ce550 0x7f75dc15b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.292+0000 7f75e209c700 1 -- 172.21.15.1:0/423806883 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75dc0cfa90 msgr2=0x7f75dc0adff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:35.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.292+0000 7f75e209c700 1 --2- 172.21.15.1:0/423806883 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75dc0cfa90 0x7f75dc0adff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.292+0000 7f75e209c700 1 -- 172.21.15.1:0/423806883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d0009500 con 0x7f75dc15d5a0 2024-03-18T20:25:35.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.292+0000 7f75e209c700 1 --2- 172.21.15.1:0/423806883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75dc15d5a0 0x7f75dc0ae530 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f75d400cba0 tx=0x7f75d400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:35.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.293+0000 7f75ce7fc700 1 -- 172.21.15.1:0/423806883 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d400ab80 con 0x7f75dc15d5a0 2024-03-18T20:25:35.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.293+0000 7f75ce7fc700 1 -- 172.21.15.1:0/423806883 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d400ace0 con 0x7f75dc15d5a0 2024-03-18T20:25:35.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.293+0000 7f75e3aff700 1 -- 172.21.15.1:0/423806883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75dc0b32c0 con 0x7f75dc15d5a0 2024-03-18T20:25:35.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.293+0000 7f75ce7fc700 1 -- 172.21.15.1:0/423806883 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d4012660 con 0x7f75dc15d5a0 2024-03-18T20:25:35.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.293+0000 7f75e3aff700 1 -- 172.21.15.1:0/423806883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75dc15ba30 con 0x7f75dc15d5a0 2024-03-18T20:25:35.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.293+0000 7f75e3aff700 1 -- 172.21.15.1:0/423806883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75dc0b3ac0 con 0x7f75dc15d5a0 2024-03-18T20:25:35.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.294+0000 7f75ce7fc700 1 -- 172.21.15.1:0/423806883 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f75d40127c0 con 0x7f75dc15d5a0 2024-03-18T20:25:35.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.295+0000 7f75ce7fc700 1 --2- 172.21.15.1:0/423806883 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f75c006adb0 0x7f75c006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:35.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.295+0000 7f75ce7fc700 1 -- 172.21.15.1:0/423806883 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 6887+0+0 (secure 0 0 0) 0x7f75d408a4b0 con 0x7f75dc15d5a0 2024-03-18T20:25:35.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.295+0000 7f75e189b700 1 --2- 172.21.15.1:0/423806883 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f75c006adb0 0x7f75c006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:35.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.295+0000 7f75e189b700 1 --2- 172.21.15.1:0/423806883 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f75c006adb0 0x7f75c006d270 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f75dc0af730 tx=0x7f75d0008070 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:35.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.299+0000 7f75ce7fc700 1 -- 172.21.15.1:0/423806883 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f75d4055140 con 0x7f75dc15d5a0 2024-03-18T20:25:35.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.547+0000 7f75e3aff700 1 -- 172.21.15.1:0/423806883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7f75dc0676a0 con 0x7f75dc15d5a0 2024-03-18T20:25:35.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.552+0000 7f75ce7fc700 1 -- 172.21.15.1:0/423806883 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 29 and data pool 30 v106) v1 ==== 183+0+0 (secure 0 0 0) 0x7f75d4016020 con 0x7f75dc15d5a0 2024-03-18T20:25:35.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.556+0000 7f75e3aff700 1 -- 172.21.15.1:0/423806883 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f75c006adb0 msgr2=0x7f75c006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:35.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.556+0000 7f75e3aff700 1 --2- 172.21.15.1:0/423806883 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f75c006adb0 0x7f75c006d270 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f75dc0af730 tx=0x7f75d0008070 comp rx=0 tx=0).stop 2024-03-18T20:25:35.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.556+0000 7f75e3aff700 1 -- 172.21.15.1:0/423806883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75dc15d5a0 msgr2=0x7f75dc0ae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:35.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.556+0000 7f75e3aff700 1 --2- 172.21.15.1:0/423806883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75dc15d5a0 0x7f75dc0ae530 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f75d400cba0 tx=0x7f75d400cf60 comp rx=0 tx=0).stop 2024-03-18T20:25:35.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.556+0000 7f75e3aff700 1 -- 172.21.15.1:0/423806883 shutdown_connections 2024-03-18T20:25:35.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.556+0000 7f75e3aff700 1 --2- 172.21.15.1:0/423806883 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f75c006adb0 0x7f75c006d270 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.556+0000 7f75e3aff700 1 --2- 172.21.15.1:0/423806883 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f75dc0ce550 0x7f75dc15b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.556+0000 7f75e3aff700 1 --2- 172.21.15.1:0/423806883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f75dc15d5a0 0x7f75dc0ae530 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.556+0000 7f75e3aff700 1 --2- 172.21.15.1:0/423806883 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75dc0cfa90 0x7f75dc0adff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.556+0000 7f75e3aff700 1 -- 172.21.15.1:0/423806883 >> 172.21.15.1:0/423806883 conn(0x7f75dc0cc9b0 msgr2=0x7f75dc0afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:35.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.557+0000 7f75e3aff700 1 -- 172.21.15.1:0/423806883 shutdown_connections 2024-03-18T20:25:35.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.557+0000 7f75e3aff700 1 -- 172.21.15.1:0/423806883 wait complete. 2024-03-18T20:25:35.562 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 29 and data pool 30 2024-03-18T20:25:35.581 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:25:35.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.703+0000 7fd57c03f700 1 Processor -- start 2024-03-18T20:25:35.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.703+0000 7fd57c03f700 1 -- start start 2024-03-18T20:25:35.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.704+0000 7fd57c03f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd57415b4d0 0x7fd57409a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.704+0000 7fd57c03f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57406f070 con 0x7fd57409a150 2024-03-18T20:25:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.704+0000 7fd57c03f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5740d0260 con 0x7fd5740cf750 2024-03-18T20:25:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.704+0000 7fd57c03f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5740cff80 con 0x7fd57415b4d0 2024-03-18T20:25:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.704+0000 7fd57a5dc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd57409a150 0x7fd57406f500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51476/0 (socket says 172.21.15.1:51476) 2024-03-18T20:25:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.704+0000 7fd57a5dc700 1 -- 172.21.15.1:0/264307447 learned_addr learned my addr 172.21.15.1:0/264307447 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd578dd9700 1 -- 172.21.15.1:0/264307447 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219874849 0 0) 0x7fd57406f070 con 0x7fd57409a150 2024-03-18T20:25:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd579ddb700 1 --2- 172.21.15.1:0/264307447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd57415b4d0 0x7fd57409a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd578dd9700 1 -- 172.21.15.1:0/264307447 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5600035b0 con 0x7fd57409a150 2024-03-18T20:25:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd578dd9700 1 -- 172.21.15.1:0/264307447 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3130294275 0 0) 0x7fd5600035b0 con 0x7fd57409a150 2024-03-18T20:25:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd578dd9700 1 -- 172.21.15.1:0/264307447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd57415b4d0 msgr2=0x7fd57409a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd578dd9700 1 --2- 172.21.15.1:0/264307447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd57415b4d0 0x7fd57409a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd578dd9700 1 -- 172.21.15.1:0/264307447 >> v1:172.21.15.165:6789/0 conn(0x7fd5740cf750 legacy=0x7fd574099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:35.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd578dd9700 1 -- 172.21.15.1:0/264307447 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57406ea20 con 0x7fd57409a150 2024-03-18T20:25:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd578dd9700 1 -- 172.21.15.1:0/264307447 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fd56c0019f0 con 0x7fd57409a150 2024-03-18T20:25:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd578dd9700 1 -- 172.21.15.1:0/264307447 >> v1:172.21.15.1:6789/0 conn(0x7fd57409a150 legacy=0x7fd57406f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd578dd9700 1 --2- 172.21.15.1:0/264307447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd560003a30 0x7fd560003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd578dd9700 1 --2- 172.21.15.1:0/264307447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd57415b4d0 0x7fd560004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd578dd9700 1 --2- 172.21.15.1:0/264307447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd560004d90 0x7fd560005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd578dd9700 1 -- 172.21.15.1:0/264307447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5740cff80 con 0x7fd560003a30 2024-03-18T20:25:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd578dd9700 1 -- 172.21.15.1:0/264307447 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56000fed0 con 0x7fd560004d90 2024-03-18T20:25:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd578dd9700 1 -- 172.21.15.1:0/264307447 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd560010090 con 0x7fd57415b4d0 2024-03-18T20:25:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd579ddb700 1 --2- 172.21.15.1:0/264307447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd560003a30 0x7fd560003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:35.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd57a5dc700 1 --2- 172.21.15.1:0/264307447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd560004d90 0x7fd560005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd5795da700 1 --2- 172.21.15.1:0/264307447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd57415b4d0 0x7fd560004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd579ddb700 1 -- 172.21.15.1:0/264307447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd57415b4d0 msgr2=0x7fd560004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd579ddb700 1 --2- 172.21.15.1:0/264307447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd57415b4d0 0x7fd560004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd579ddb700 1 -- 172.21.15.1:0/264307447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd560004d90 msgr2=0x7fd560005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.705+0000 7fd579ddb700 1 --2- 172.21.15.1:0/264307447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd560004d90 0x7fd560005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.706+0000 7fd579ddb700 1 -- 172.21.15.1:0/264307447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5600035b0 con 0x7fd560003a30 2024-03-18T20:25:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.706+0000 7fd579ddb700 1 --2- 172.21.15.1:0/264307447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd560003a30 0x7fd560003e10 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7fd564014f00 tx=0x7fd560005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e75021ce5e79eef3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.706+0000 7fd578dd9700 1 -- 172.21.15.1:0/264307447 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd564019070 con 0x7fd560003a30 2024-03-18T20:25:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.706+0000 7fd578dd9700 1 -- 172.21.15.1:0/264307447 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd564009dd0 con 0x7fd560003a30 2024-03-18T20:25:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.706+0000 7fd578dd9700 1 -- 172.21.15.1:0/264307447 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56401d610 con 0x7fd560003a30 2024-03-18T20:25:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.706+0000 7fd57c03f700 1 -- 172.21.15.1:0/264307447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd560003a30 msgr2=0x7fd560003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.706+0000 7fd57c03f700 1 --2- 172.21.15.1:0/264307447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd560003a30 0x7fd560003e10 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7fd564014f00 tx=0x7fd560005880 comp rx=0 tx=0).stop 2024-03-18T20:25:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.706+0000 7fd57a5dc700 1 -- 172.21.15.1:0/264307447 reap_dead start 2024-03-18T20:25:35.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.706+0000 7fd57c03f700 1 -- 172.21.15.1:0/264307447 shutdown_connections 2024-03-18T20:25:35.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.706+0000 7fd57c03f700 1 -- 172.21.15.1:0/264307447 >> 172.21.15.1:0/264307447 conn(0x7fd5740b2cb0 msgr2=0x7fd5740b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:35.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.707+0000 7fd57c03f700 1 -- 172.21.15.1:0/264307447 shutdown_connections 2024-03-18T20:25:35.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.707+0000 7fd57c03f700 1 -- 172.21.15.1:0/264307447 wait complete. 2024-03-18T20:25:35.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.708+0000 7fd57c03f700 1 Processor -- start 2024-03-18T20:25:35.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.708+0000 7fd57c03f700 1 -- start start 2024-03-18T20:25:35.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.708+0000 7fd57c03f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd560003a30 0x7fd574154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:35.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.708+0000 7fd57c03f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5740ca070 0x7fd5740ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:35.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.708+0000 7fd57c03f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd57415d1b0 0x7fd57415d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:35.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.708+0000 7fd57c03f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57415db70 con 0x7fd560003a30 2024-03-18T20:25:35.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.708+0000 7fd57c03f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5740b6c90 con 0x7fd57415d1b0 2024-03-18T20:25:35.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.708+0000 7fd57c03f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd574159080 con 0x7fd5740ca070 2024-03-18T20:25:35.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.709+0000 7fd579ddb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd560003a30 0x7fd574154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:35.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.709+0000 7fd579ddb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd560003a30 0x7fd574154240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49762/0 (socket says 172.21.15.1:49762) 2024-03-18T20:25:35.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.709+0000 7fd579ddb700 1 -- 172.21.15.1:0/4095062360 learned_addr learned my addr 172.21.15.1:0/4095062360 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:35.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.709+0000 7fd5795da700 1 --2- 172.21.15.1:0/4095062360 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5740ca070 0x7fd5740ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:35.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.709+0000 7fd57a5dc700 1 --2- 172.21.15.1:0/4095062360 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd57415d1b0 0x7fd57415d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:35.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.709+0000 7fd579ddb700 1 -- 172.21.15.1:0/4095062360 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5740ca070 msgr2=0x7fd5740ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:35.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.709+0000 7fd579ddb700 1 --2- 172.21.15.1:0/4095062360 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5740ca070 0x7fd5740ca4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.709+0000 7fd579ddb700 1 -- 172.21.15.1:0/4095062360 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd57415d1b0 msgr2=0x7fd57415d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:35.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.709+0000 7fd579ddb700 1 --2- 172.21.15.1:0/4095062360 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd57415d1b0 0x7fd57415d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.709+0000 7fd579ddb700 1 -- 172.21.15.1:0/4095062360 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5740cfd50 con 0x7fd560003a30 2024-03-18T20:25:35.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.709+0000 7fd579ddb700 1 --2- 172.21.15.1:0/4095062360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd560003a30 0x7fd574154240 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7fd5600047b0 tx=0x7fd564004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:35.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.709+0000 7fd56a7fc700 1 -- 172.21.15.1:0/4095062360 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd564004440 con 0x7fd560003a30 2024-03-18T20:25:35.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.709+0000 7fd56a7fc700 1 -- 172.21.15.1:0/4095062360 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5640045a0 con 0x7fd560003a30 2024-03-18T20:25:35.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.709+0000 7fd57c03f700 1 -- 172.21.15.1:0/4095062360 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5740c8330 con 0x7fd560003a30 2024-03-18T20:25:35.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.709+0000 7fd56a7fc700 1 -- 172.21.15.1:0/4095062360 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5640038e0 con 0x7fd560003a30 2024-03-18T20:25:35.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.709+0000 7fd57c03f700 1 -- 172.21.15.1:0/4095062360 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd574158ac0 con 0x7fd560003a30 2024-03-18T20:25:35.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.710+0000 7fd57c03f700 1 -- 172.21.15.1:0/4095062360 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd574068240 con 0x7fd560003a30 2024-03-18T20:25:35.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.711+0000 7fd56a7fc700 1 -- 172.21.15.1:0/4095062360 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd564024040 con 0x7fd560003a30 2024-03-18T20:25:35.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.711+0000 7fd56a7fc700 1 --2- 172.21.15.1:0/4095062360 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd55806ade0 0x7fd55806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:35.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.711+0000 7fd56a7fc700 1 -- 172.21.15.1:0/4095062360 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6993+0+0 (secure 0 0 0) 0x7fd564092420 con 0x7fd560003a30 2024-03-18T20:25:35.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.711+0000 7fd5795da700 1 --2- 172.21.15.1:0/4095062360 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd55806ade0 0x7fd55806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:35.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.711+0000 7fd5795da700 1 --2- 172.21.15.1:0/4095062360 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd55806ade0 0x7fd55806d2a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd5741f1b00 tx=0x7fd570009040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:35.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.716+0000 7fd56a7fc700 1 -- 172.21.15.1:0/4095062360 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd56405d070 con 0x7fd560003a30 2024-03-18T20:25:35.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.920+0000 7fd57c03f700 1 -- 172.21.15.1:0/4095062360 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5740c84f0 con 0x7fd560003a30 2024-03-18T20:25:35.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.921+0000 7fd56a7fc700 1 -- 172.21.15.1:0/4095062360 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v235) v1 ==== 74+0+20098 (secure 0 0 0) 0x7fd564060690 con 0x7fd560003a30 2024-03-18T20:25:35.926 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:35.926 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":235,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:25:35.548973+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:25:20.511297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-18T20:25:21.637693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:25:33.275375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:25:34.518347+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:35.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.923+0000 7fd57c03f700 1 -- 172.21.15.1:0/4095062360 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd55806ade0 msgr2=0x7fd55806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:35.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.923+0000 7fd57c03f700 1 --2- 172.21.15.1:0/4095062360 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd55806ade0 0x7fd55806d2a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd5741f1b00 tx=0x7fd570009040 comp rx=0 tx=0).stop 2024-03-18T20:25:35.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.923+0000 7fd57c03f700 1 -- 172.21.15.1:0/4095062360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd560003a30 msgr2=0x7fd574154240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:35.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.923+0000 7fd57c03f700 1 --2- 172.21.15.1:0/4095062360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd560003a30 0x7fd574154240 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7fd5600047b0 tx=0x7fd564004750 comp rx=0 tx=0).stop 2024-03-18T20:25:35.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.923+0000 7fd57c03f700 1 -- 172.21.15.1:0/4095062360 shutdown_connections 2024-03-18T20:25:35.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.923+0000 7fd57c03f700 1 --2- 172.21.15.1:0/4095062360 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd55806ade0 0x7fd55806d2a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.923+0000 7fd57c03f700 1 --2- 172.21.15.1:0/4095062360 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd57415d1b0 0x7fd57415d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.923+0000 7fd57c03f700 1 --2- 172.21.15.1:0/4095062360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd560003a30 0x7fd574154240 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.923+0000 7fd57c03f700 1 --2- 172.21.15.1:0/4095062360 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5740ca070 0x7fd5740ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:35.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.923+0000 7fd57c03f700 1 -- 172.21.15.1:0/4095062360 >> 172.21.15.1:0/4095062360 conn(0x7fd5740b2cb0 msgr2=0x7fd5740cbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:35.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.923+0000 7fd57c03f700 1 -- 172.21.15.1:0/4095062360 shutdown_connections 2024-03-18T20:25:35.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:35.923+0000 7fd57c03f700 1 -- 172.21.15.1:0/4095062360 wait complete. 2024-03-18T20:25:35.940 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:25:36.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.061+0000 7faa7dbb6700 1 Processor -- start 2024-03-18T20:25:36.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.061+0000 7faa7dbb6700 1 -- start start 2024-03-18T20:25:36.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.061+0000 7faa7dbb6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faa78071a80 0x7faa7806e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:36.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.061+0000 7faa7dbb6700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa780acd80 con 0x7faa7815a320 2024-03-18T20:25:36.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.061+0000 7faa7dbb6700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa780cf220 con 0x7faa78071600 2024-03-18T20:25:36.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.061+0000 7faa7dbb6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa78071f10 con 0x7faa78071a80 2024-03-18T20:25:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.062+0000 7faa77fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faa78071a80 0x7faa7806e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.062+0000 7faa77fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faa78071a80 0x7faa7806e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49774/0 (socket says 172.21.15.1:49774) 2024-03-18T20:25:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.062+0000 7faa77fff700 1 -- 172.21.15.1:0/4101104881 learned_addr learned my addr 172.21.15.1:0/4101104881 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.062+0000 7faa77fff700 1 -- 172.21.15.1:0/4101104881 >> v1:172.21.15.165:6789/0 conn(0x7faa78071600 legacy=0x7faa7806e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.062+0000 7faa77fff700 1 -- 172.21.15.1:0/4101104881 >> v1:172.21.15.1:6789/0 conn(0x7faa7815a320 legacy=0x7faa780ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.062+0000 7faa77fff700 1 -- 172.21.15.1:0/4101104881 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa780cef40 con 0x7faa78071a80 2024-03-18T20:25:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.062+0000 7faa77fff700 1 --2- 172.21.15.1:0/4101104881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa78071a80 0x7faa7806e9d0 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7faa70009e50 tx=0x7faa7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0fbf2c77613bb6f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.062+0000 7faa767fc700 1 -- 172.21.15.1:0/4101104881 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7000e070 con 0x7faa78071a80 2024-03-18T20:25:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.062+0000 7faa767fc700 1 -- 172.21.15.1:0/4101104881 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa70003a20 con 0x7faa78071a80 2024-03-18T20:25:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.062+0000 7faa767fc700 1 -- 172.21.15.1:0/4101104881 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa70003dd0 con 0x7faa78071a80 2024-03-18T20:25:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.063+0000 7faa7dbb6700 1 -- 172.21.15.1:0/4101104881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa78071a80 msgr2=0x7faa7806e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.063+0000 7faa7dbb6700 1 --2- 172.21.15.1:0/4101104881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa78071a80 0x7faa7806e9d0 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7faa70009e50 tx=0x7faa7000b330 comp rx=0 tx=0).stop 2024-03-18T20:25:36.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.063+0000 7faa7dbb6700 1 -- 172.21.15.1:0/4101104881 shutdown_connections 2024-03-18T20:25:36.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.063+0000 7faa7dbb6700 1 --2- 172.21.15.1:0/4101104881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa78071a80 0x7faa7806e9d0 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.063+0000 7faa7dbb6700 1 -- 172.21.15.1:0/4101104881 >> 172.21.15.1:0/4101104881 conn(0x7faa780cc990 msgr2=0x7faa780ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:36.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.063+0000 7faa7dbb6700 1 -- 172.21.15.1:0/4101104881 shutdown_connections 2024-03-18T20:25:36.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.063+0000 7faa7dbb6700 1 -- 172.21.15.1:0/4101104881 wait complete. 2024-03-18T20:25:36.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.064+0000 7faa7dbb6700 1 Processor -- start 2024-03-18T20:25:36.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.064+0000 7faa7dbb6700 1 -- start start 2024-03-18T20:25:36.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.065+0000 7faa7dbb6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa78071600 0x7faa7815c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:36.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.065+0000 7faa7dbb6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faa78071a80 0x7faa781efd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:36.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.065+0000 7faa7dbb6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa78148500 0x7faa78148990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:36.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.065+0000 7faa7dbb6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa78146230 con 0x7faa78148500 2024-03-18T20:25:36.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.065+0000 7faa7dbb6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa78148ed0 con 0x7faa78071600 2024-03-18T20:25:36.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.065+0000 7faa7dbb6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa780c4090 con 0x7faa78071a80 2024-03-18T20:25:36.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.065+0000 7faa77fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa78148500 0x7faa78148990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:36.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.065+0000 7faa77fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa78148500 0x7faa78148990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49778/0 (socket says 172.21.15.1:49778) 2024-03-18T20:25:36.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.065+0000 7faa77fff700 1 -- 172.21.15.1:0/1876778834 learned_addr learned my addr 172.21.15.1:0/1876778834 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:36.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.065+0000 7faa76ffd700 1 --2- 172.21.15.1:0/1876778834 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faa78071a80 0x7faa781efd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:36.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.065+0000 7faa777fe700 1 --2- 172.21.15.1:0/1876778834 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa78071600 0x7faa7815c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:36.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.065+0000 7faa77fff700 1 -- 172.21.15.1:0/1876778834 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faa78071a80 msgr2=0x7faa781efd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:36.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.065+0000 7faa77fff700 1 --2- 172.21.15.1:0/1876778834 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faa78071a80 0x7faa781efd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.065+0000 7faa77fff700 1 -- 172.21.15.1:0/1876778834 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa78071600 msgr2=0x7faa7815c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:36.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.065+0000 7faa77fff700 1 --2- 172.21.15.1:0/1876778834 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa78071600 0x7faa7815c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.065+0000 7faa77fff700 1 -- 172.21.15.1:0/1876778834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa70009b30 con 0x7faa78148500 2024-03-18T20:25:36.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.065+0000 7faa777fe700 1 --2- 172.21.15.1:0/1876778834 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa78071600 0x7faa7815c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:36.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.065+0000 7faa77fff700 1 --2- 172.21.15.1:0/1876778834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa78148500 0x7faa78148990 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7faa78067310 tx=0x7faa70004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:36.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.066+0000 7faa5ffff700 1 -- 172.21.15.1:0/1876778834 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa70004440 con 0x7faa78148500 2024-03-18T20:25:36.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.066+0000 7faa5ffff700 1 -- 172.21.15.1:0/1876778834 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa700045a0 con 0x7faa78148500 2024-03-18T20:25:36.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.066+0000 7faa7dbb6700 1 -- 172.21.15.1:0/1876778834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa780c3db0 con 0x7faa78148500 2024-03-18T20:25:36.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.066+0000 7faa5ffff700 1 -- 172.21.15.1:0/1876778834 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa70004950 con 0x7faa78148500 2024-03-18T20:25:36.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.066+0000 7faa7dbb6700 1 -- 172.21.15.1:0/1876778834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7815b9c0 con 0x7faa78148500 2024-03-18T20:25:36.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.067+0000 7faa7dbb6700 1 -- 172.21.15.1:0/1876778834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa780676a0 con 0x7faa78148500 2024-03-18T20:25:36.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.067+0000 7faa5ffff700 1 -- 172.21.15.1:0/1876778834 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7faa7002c430 con 0x7faa78148500 2024-03-18T20:25:36.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.068+0000 7faa5ffff700 1 --2- 172.21.15.1:0/1876778834 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faa6406adb0 0x7faa6406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:36.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.068+0000 7faa5ffff700 1 -- 172.21.15.1:0/1876778834 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6993+0+0 (secure 0 0 0) 0x7faa7008c950 con 0x7faa78148500 2024-03-18T20:25:36.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.068+0000 7faa777fe700 1 --2- 172.21.15.1:0/1876778834 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faa6406adb0 0x7faa6406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:36.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.068+0000 7faa777fe700 1 --2- 172.21.15.1:0/1876778834 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faa6406adb0 0x7faa6406d270 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faa68001710 tx=0x7faa68009040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:36.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.072+0000 7faa5ffff700 1 -- 172.21.15.1:0/1876778834 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faa70057410 con 0x7faa78148500 2024-03-18T20:25:36.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.276+0000 7faa7dbb6700 1 -- 172.21.15.1:0/1876778834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa780cf220 con 0x7faa78148500 2024-03-18T20:25:36.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.277+0000 7faa5ffff700 1 -- 172.21.15.1:0/1876778834 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v235) v1 ==== 74+0+20098 (secure 0 0 0) 0x7faa780cf220 con 0x7faa78148500 2024-03-18T20:25:36.282 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:36.282 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":235,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:25:35.548973+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:25:20.511297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-18T20:25:21.637693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:25:33.275375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:25:34.518347+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:36.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.279+0000 7faa7dbb6700 1 -- 172.21.15.1:0/1876778834 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faa6406adb0 msgr2=0x7faa6406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:36.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.279+0000 7faa7dbb6700 1 --2- 172.21.15.1:0/1876778834 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faa6406adb0 0x7faa6406d270 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faa68001710 tx=0x7faa68009040 comp rx=0 tx=0).stop 2024-03-18T20:25:36.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.279+0000 7faa7dbb6700 1 -- 172.21.15.1:0/1876778834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa78148500 msgr2=0x7faa78148990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:36.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.279+0000 7faa7dbb6700 1 --2- 172.21.15.1:0/1876778834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa78148500 0x7faa78148990 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7faa78067310 tx=0x7faa70004ad0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.280+0000 7faa7dbb6700 1 -- 172.21.15.1:0/1876778834 shutdown_connections 2024-03-18T20:25:36.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.280+0000 7faa7dbb6700 1 --2- 172.21.15.1:0/1876778834 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faa6406adb0 0x7faa6406d270 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.280+0000 7faa7dbb6700 1 --2- 172.21.15.1:0/1876778834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa78148500 0x7faa78148990 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.280+0000 7faa7dbb6700 1 --2- 172.21.15.1:0/1876778834 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa78071600 0x7faa7815c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.280+0000 7faa7dbb6700 1 --2- 172.21.15.1:0/1876778834 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faa78071a80 0x7faa781efd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.280+0000 7faa7dbb6700 1 -- 172.21.15.1:0/1876778834 >> 172.21.15.1:0/1876778834 conn(0x7faa780cc990 msgr2=0x7faa780b0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:36.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.280+0000 7faa7dbb6700 1 -- 172.21.15.1:0/1876778834 shutdown_connections 2024-03-18T20:25:36.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.280+0000 7faa7dbb6700 1 -- 172.21.15.1:0/1876778834 wait complete. 2024-03-18T20:25:36.296 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2024-03-18T20:25:36.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.420+0000 7f0752a84700 1 Processor -- start 2024-03-18T20:25:36.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.420+0000 7f0752a84700 1 -- start start 2024-03-18T20:25:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.421+0000 7f0752a84700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f074c06f180 0x7f074c161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.421+0000 7f0752a84700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f074c0cad60 con 0x7f074c06a750 2024-03-18T20:25:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.421+0000 7f0752a84700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f074c0cb040 con 0x7f074c06eda0 2024-03-18T20:25:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.421+0000 7f0752a84700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f074c0cb320 con 0x7f074c06f180 2024-03-18T20:25:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.421+0000 7f0751021700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f074c06a750 0x7f074c06ab60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51480/0 (socket says 172.21.15.1:51480) 2024-03-18T20:25:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.421+0000 7f0751021700 1 -- 172.21.15.1:0/2028611633 learned_addr learned my addr 172.21.15.1:0/2028611633 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.421+0000 7f074bfff700 1 --2- 172.21.15.1:0/2028611633 >> v2:172.21.15.165:3300/0 conn(0x7f074c06f180 0x7f074c161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.421+0000 7f074b7fe700 1 -- 172.21.15.1:0/2028611633 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828137981 0 0) 0x7f074c0cad60 con 0x7f074c06a750 2024-03-18T20:25:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.422+0000 7f074b7fe700 1 -- 172.21.15.1:0/2028611633 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f073c0035b0 con 0x7f074c06a750 2024-03-18T20:25:36.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.422+0000 7f074bfff700 1 -- 172.21.15.1:0/2028611633 >> v1:172.21.15.165:6789/0 conn(0x7f074c06eda0 legacy=0x7f074c0ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.422+0000 7f074bfff700 1 -- 172.21.15.1:0/2028611633 >> v1:172.21.15.1:6789/0 conn(0x7f074c06a750 legacy=0x7f074c06ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.422+0000 7f074b7fe700 1 -- 172.21.15.1:0/2028611633 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 309168967 0 0) 0x7f073c0035b0 con 0x7f074c06a750 2024-03-18T20:25:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.422+0000 7f074bfff700 1 -- 172.21.15.1:0/2028611633 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074c0ce220 con 0x7f074c06f180 2024-03-18T20:25:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.422+0000 7f074bfff700 1 --2- 172.21.15.1:0/2028611633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f074c06f180 0x7f074c161c60 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f0738009840 tx=0x7f0738009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a81b7e64a9a2b02d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.422+0000 7f074b7fe700 1 -- 172.21.15.1:0/2028611633 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07440019f0 con 0x7f074c06f180 2024-03-18T20:25:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.422+0000 7f074b7fe700 1 -- 172.21.15.1:0/2028611633 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0738004030 con 0x7f074c06f180 2024-03-18T20:25:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.422+0000 7f074b7fe700 1 -- 172.21.15.1:0/2028611633 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07380043c0 con 0x7f074c06f180 2024-03-18T20:25:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.423+0000 7f0752a84700 1 -- 172.21.15.1:0/2028611633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f074c06f180 msgr2=0x7f074c161c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.423+0000 7f0752a84700 1 --2- 172.21.15.1:0/2028611633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f074c06f180 0x7f074c161c60 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f0738009840 tx=0x7f0738009b50 comp rx=0 tx=0).stop 2024-03-18T20:25:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.423+0000 7f0752a84700 1 -- 172.21.15.1:0/2028611633 shutdown_connections 2024-03-18T20:25:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.423+0000 7f0752a84700 1 --2- 172.21.15.1:0/2028611633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f074c06f180 0x7f074c161c60 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.423+0000 7f0752a84700 1 -- 172.21.15.1:0/2028611633 >> 172.21.15.1:0/2028611633 conn(0x7f074c0cff30 msgr2=0x7f074c0d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.423+0000 7f0752a84700 1 -- 172.21.15.1:0/2028611633 shutdown_connections 2024-03-18T20:25:36.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.423+0000 7f0752a84700 1 -- 172.21.15.1:0/2028611633 wait complete. 2024-03-18T20:25:36.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.424+0000 7f0752a84700 1 Processor -- start 2024-03-18T20:25:36.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.424+0000 7f0752a84700 1 -- start start 2024-03-18T20:25:36.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.425+0000 7f0752a84700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f074c06a750 0x7f074c0b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:36.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.425+0000 7f0752a84700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f074c06eda0 0x7f074c1f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:36.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.425+0000 7f0752a84700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f074c06f180 0x7f074c0afe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:36.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.425+0000 7f0752a84700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074c0c5a70 con 0x7f074c06f180 2024-03-18T20:25:36.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.425+0000 7f0752a84700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074c1f3470 con 0x7f074c06a750 2024-03-18T20:25:36.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.425+0000 7f0752a84700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074c1f3710 con 0x7f074c06eda0 2024-03-18T20:25:36.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.426+0000 7f0751021700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f074c06f180 0x7f074c0afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:36.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.426+0000 7f0751021700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f074c06f180 0x7f074c0afe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49784/0 (socket says 172.21.15.1:49784) 2024-03-18T20:25:36.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.426+0000 7f0751021700 1 -- 172.21.15.1:0/642088748 learned_addr learned my addr 172.21.15.1:0/642088748 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:36.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.426+0000 7f0750820700 1 --2- 172.21.15.1:0/642088748 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f074c06a750 0x7f074c0b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:36.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.426+0000 7f074bfff700 1 --2- 172.21.15.1:0/642088748 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f074c06eda0 0x7f074c1f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:36.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.426+0000 7f0751021700 1 -- 172.21.15.1:0/642088748 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f074c06eda0 msgr2=0x7f074c1f2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:36.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.426+0000 7f0751021700 1 --2- 172.21.15.1:0/642088748 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f074c06eda0 0x7f074c1f2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.426+0000 7f0751021700 1 -- 172.21.15.1:0/642088748 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f074c06a750 msgr2=0x7f074c0b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:36.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.426+0000 7f0751021700 1 --2- 172.21.15.1:0/642088748 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f074c06a750 0x7f074c0b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:36.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.426+0000 7f0751021700 1 -- 172.21.15.1:0/642088748 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07380094a0 con 0x7f074c06f180 2024-03-18T20:25:36.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.426+0000 7f0751021700 1 --2- 172.21.15.1:0/642088748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f074c06f180 0x7f074c0afe80 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f074400cc90 tx=0x7f074400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:36.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.426+0000 7f07497fa700 1 -- 172.21.15.1:0/642088748 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074400ada0 con 0x7f074c06f180 2024-03-18T20:25:36.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.426+0000 7f07497fa700 1 -- 172.21.15.1:0/642088748 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074400ce90 con 0x7f074c06f180 2024-03-18T20:25:36.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.426+0000 7f0752a84700 1 -- 172.21.15.1:0/642088748 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f074c1f39f0 con 0x7f074c06f180 2024-03-18T20:25:36.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.426+0000 7f07497fa700 1 -- 172.21.15.1:0/642088748 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0744007310 con 0x7f074c06f180 2024-03-18T20:25:36.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.426+0000 7f0752a84700 1 -- 172.21.15.1:0/642088748 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f074c0c5cc0 con 0x7f074c06f180 2024-03-18T20:25:36.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.427+0000 7f0752a84700 1 -- 172.21.15.1:0/642088748 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f074c0676a0 con 0x7f074c06f180 2024-03-18T20:25:36.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.428+0000 7f07497fa700 1 -- 172.21.15.1:0/642088748 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0744007470 con 0x7f074c06f180 2024-03-18T20:25:36.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.428+0000 7f07497fa700 1 --2- 172.21.15.1:0/642088748 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f073006ade0 0x7f073006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:36.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.428+0000 7f07497fa700 1 -- 172.21.15.1:0/642088748 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f0744089870 con 0x7f074c06f180 2024-03-18T20:25:36.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.428+0000 7f0750820700 1 --2- 172.21.15.1:0/642088748 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f073006ade0 0x7f073006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:36.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.428+0000 7f0750820700 1 --2- 172.21.15.1:0/642088748 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f073006ade0 0x7f073006d2a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f074c0c8a90 tx=0x7f0740009040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:36.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.433+0000 7f07497fa700 1 -- 172.21.15.1:0/642088748 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f07440544c0 con 0x7f074c06f180 2024-03-18T20:25:36.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:36.687+0000 7f0752a84700 1 -- 172.21.15.1:0/642088748 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f074c0d0bc0 con 0x7f074c06f180 2024-03-18T20:25:37.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.566+0000 7f07497fa700 1 -- 172.21.15.1:0/642088748 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v109) v1 ==== 121+0+0 (secure 0 0 0) 0x7f0744057ae0 con 0x7f074c06f180 2024-03-18T20:25:37.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.569+0000 7f0752a84700 1 -- 172.21.15.1:0/642088748 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f073006ade0 msgr2=0x7f073006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:37.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.569+0000 7f0752a84700 1 --2- 172.21.15.1:0/642088748 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f073006ade0 0x7f073006d2a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f074c0c8a90 tx=0x7f0740009040 comp rx=0 tx=0).stop 2024-03-18T20:25:37.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.569+0000 7f0752a84700 1 -- 172.21.15.1:0/642088748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f074c06f180 msgr2=0x7f074c0afe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:37.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.569+0000 7f0752a84700 1 --2- 172.21.15.1:0/642088748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f074c06f180 0x7f074c0afe80 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f074400cc90 tx=0x7f074400a340 comp rx=0 tx=0).stop 2024-03-18T20:25:37.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.569+0000 7f0752a84700 1 -- 172.21.15.1:0/642088748 shutdown_connections 2024-03-18T20:25:37.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.569+0000 7f0752a84700 1 --2- 172.21.15.1:0/642088748 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f073006ade0 0x7f073006d2a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.569+0000 7f0752a84700 1 --2- 172.21.15.1:0/642088748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f074c06f180 0x7f074c0afe80 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.569+0000 7f0752a84700 1 --2- 172.21.15.1:0/642088748 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f074c06a750 0x7f074c0b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.569+0000 7f0752a84700 1 --2- 172.21.15.1:0/642088748 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f074c06eda0 0x7f074c1f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.569+0000 7f0752a84700 1 -- 172.21.15.1:0/642088748 >> 172.21.15.1:0/642088748 conn(0x7f074c0cff30 msgr2=0x7f074c06d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:37.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.569+0000 7f0752a84700 1 -- 172.21.15.1:0/642088748 shutdown_connections 2024-03-18T20:25:37.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.569+0000 7f0752a84700 1 -- 172.21.15.1:0/642088748 wait complete. 2024-03-18T20:25:37.588 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-18T20:25:37.588 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2024-03-18T20:25:37.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.719+0000 7f2e5bbaf700 1 Processor -- start 2024-03-18T20:25:37.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.719+0000 7f2e5bbaf700 1 -- start start 2024-03-18T20:25:37.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.720+0000 7f2e5bbaf700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2e54071d30 0x7f2e540add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:37.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.720+0000 7f2e5bbaf700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e540cb9e0 0x7f2e540cbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:37.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.720+0000 7f2e5bbaf700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e540ce8f0 con 0x7f2e5415a1d0 2024-03-18T20:25:37.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.720+0000 7f2e5bbaf700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e540cdb00 con 0x7f2e540cb9e0 2024-03-18T20:25:37.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.720+0000 7f2e5bbaf700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e5415a5b0 con 0x7f2e54071d30 2024-03-18T20:25:37.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.720+0000 7f2e5a14c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2e540cb9e0 0x7f2e540cbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:37.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.720+0000 7f2e5994b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2e5415a1d0 0x7f2e54071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51490/0 (socket says 172.21.15.1:51490) 2024-03-18T20:25:37.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.720+0000 7f2e5994b700 1 -- 172.21.15.1:0/2211826755 learned_addr learned my addr 172.21.15.1:0/2211826755 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:37.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.721+0000 7f2e5914a700 1 --2- 172.21.15.1:0/2211826755 >> v2:172.21.15.165:3300/0 conn(0x7f2e54071d30 0x7f2e540add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:37.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.721+0000 7f2e5a14c700 1 -- 172.21.15.1:0/2211826755 >> v2:172.21.15.165:3300/0 conn(0x7f2e54071d30 msgr2=0x7f2e540add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:37.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.721+0000 7f2e5a14c700 1 --2- 172.21.15.1:0/2211826755 >> v2:172.21.15.165:3300/0 conn(0x7f2e54071d30 0x7f2e540add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.721+0000 7f2e5a14c700 1 -- 172.21.15.1:0/2211826755 >> v1:172.21.15.1:6789/0 conn(0x7f2e5415a1d0 legacy=0x7f2e54071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:37.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.721+0000 7f2e5a14c700 1 -- 172.21.15.1:0/2211826755 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e540ae260 con 0x7f2e540cb9e0 2024-03-18T20:25:37.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.721+0000 7f2e58949700 1 -- 172.21.15.1:0/2211826755 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752893865 0 0) 0x7f2e540ce8f0 con 0x7f2e5415a1d0 2024-03-18T20:25:37.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.721+0000 7f2e5a14c700 1 --2- 172.21.15.1:0/2211826755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e540cb9e0 0x7f2e540cbe30 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f2e540cebb0 tx=0x7f2e4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94735f0cf2ff8493 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:37.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.721+0000 7f2e58949700 1 -- 172.21.15.1:0/2211826755 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c00e070 con 0x7f2e540cb9e0 2024-03-18T20:25:37.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.721+0000 7f2e58949700 1 -- 172.21.15.1:0/2211826755 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4c00bbd0 con 0x7f2e540cb9e0 2024-03-18T20:25:37.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.721+0000 7f2e58949700 1 -- 172.21.15.1:0/2211826755 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c012430 con 0x7f2e540cb9e0 2024-03-18T20:25:37.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.722+0000 7f2e5bbaf700 1 -- 172.21.15.1:0/2211826755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e540cb9e0 msgr2=0x7f2e540cbe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:37.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.722+0000 7f2e5bbaf700 1 --2- 172.21.15.1:0/2211826755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e540cb9e0 0x7f2e540cbe30 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f2e540cebb0 tx=0x7f2e4c00b330 comp rx=0 tx=0).stop 2024-03-18T20:25:37.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.722+0000 7f2e5bbaf700 1 -- 172.21.15.1:0/2211826755 shutdown_connections 2024-03-18T20:25:37.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.722+0000 7f2e5bbaf700 1 --2- 172.21.15.1:0/2211826755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e540cb9e0 0x7f2e540cbe30 unknown :-1 s=CLOSED pgs=2570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.722+0000 7f2e5bbaf700 1 --2- 172.21.15.1:0/2211826755 >> v2:172.21.15.165:3300/0 conn(0x7f2e54071d30 0x7f2e540add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.722+0000 7f2e5bbaf700 1 -- 172.21.15.1:0/2211826755 >> 172.21.15.1:0/2211826755 conn(0x7f2e540cc9b0 msgr2=0x7f2e540ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:37.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.722+0000 7f2e5bbaf700 1 -- 172.21.15.1:0/2211826755 shutdown_connections 2024-03-18T20:25:37.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.722+0000 7f2e5bbaf700 1 -- 172.21.15.1:0/2211826755 wait complete. 2024-03-18T20:25:37.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.723+0000 7f2e5bbaf700 1 Processor -- start 2024-03-18T20:25:37.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.723+0000 7f2e5bbaf700 1 -- start start 2024-03-18T20:25:37.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e5bbaf700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e54071d30 0x7f2e541578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:37.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e5bbaf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e540cb9e0 0x7f2e541f27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:37.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e5bbaf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e5415a1d0 0x7f2e540c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:37.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e5bbaf700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e540c8180 con 0x7f2e5415a1d0 2024-03-18T20:25:37.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e5bbaf700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e540c7f50 con 0x7f2e54071d30 2024-03-18T20:25:37.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e5bbaf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5415b8b0 con 0x7f2e540cb9e0 2024-03-18T20:25:37.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e5a14c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e5415a1d0 0x7f2e540c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:37.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e5a14c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e5415a1d0 0x7f2e540c7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49788/0 (socket says 172.21.15.1:49788) 2024-03-18T20:25:37.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e5a14c700 1 -- 172.21.15.1:0/2877969929 learned_addr learned my addr 172.21.15.1:0/2877969929 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:37.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e5914a700 1 --2- 172.21.15.1:0/2877969929 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e540cb9e0 0x7f2e541f27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:37.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e5a14c700 1 -- 172.21.15.1:0/2877969929 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e540cb9e0 msgr2=0x7f2e541f27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:37.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e5a14c700 1 --2- 172.21.15.1:0/2877969929 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e540cb9e0 0x7f2e541f27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e5a14c700 1 -- 172.21.15.1:0/2877969929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e54071d30 msgr2=0x7f2e541578a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:37.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e5a14c700 1 --2- 172.21.15.1:0/2877969929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e54071d30 0x7f2e541578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:37.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e5a14c700 1 -- 172.21.15.1:0/2877969929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4c009a70 con 0x7f2e5415a1d0 2024-03-18T20:25:37.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e5a14c700 1 --2- 172.21.15.1:0/2877969929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e5415a1d0 0x7f2e540c7a10 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f2e5406cfd0 tx=0x7f2e4c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:37.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e4a7fc700 1 -- 172.21.15.1:0/2877969929 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c012430 con 0x7f2e5415a1d0 2024-03-18T20:25:37.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e4a7fc700 1 -- 172.21.15.1:0/2877969929 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4c020070 con 0x7f2e5415a1d0 2024-03-18T20:25:37.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e4a7fc700 1 -- 172.21.15.1:0/2877969929 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c004280 con 0x7f2e5415a1d0 2024-03-18T20:25:37.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e5bbaf700 1 -- 172.21.15.1:0/2877969929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e5415bb90 con 0x7f2e5415a1d0 2024-03-18T20:25:37.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.724+0000 7f2e5bbaf700 1 -- 172.21.15.1:0/2877969929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e5415c150 con 0x7f2e5415a1d0 2024-03-18T20:25:37.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.725+0000 7f2e5bbaf700 1 -- 172.21.15.1:0/2877969929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e5415c360 con 0x7f2e5415a1d0 2024-03-18T20:25:37.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.726+0000 7f2e4a7fc700 1 -- 172.21.15.1:0/2877969929 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2e4c0043e0 con 0x7f2e5415a1d0 2024-03-18T20:25:37.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.726+0000 7f2e4a7fc700 1 --2- 172.21.15.1:0/2877969929 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2e3806ade0 0x7f2e3806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:37.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.726+0000 7f2e5994b700 1 --2- 172.21.15.1:0/2877969929 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2e3806ade0 0x7f2e3806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:37.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.726+0000 7f2e4a7fc700 1 -- 172.21.15.1:0/2877969929 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f2e4c022070 con 0x7f2e5415a1d0 2024-03-18T20:25:37.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.727+0000 7f2e5994b700 1 --2- 172.21.15.1:0/2877969929 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2e3806ade0 0x7f2e3806d2a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2e440010e0 tx=0x7f2e44007850 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:37.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.731+0000 7f2e4a7fc700 1 -- 172.21.15.1:0/2877969929 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2e4c056700 con 0x7f2e5415a1d0 2024-03-18T20:25:37.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:37.979+0000 7f2e5bbaf700 1 -- 172.21.15.1:0/2877969929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2e540676a0 con 0x7f2e5415a1d0 2024-03-18T20:25:38.220 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-18T20:25:38.216+0000 7fc643c72700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T20:25:38.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.568+0000 7f2e4a7fc700 1 -- 172.21.15.1:0/2877969929 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v110) v1 ==== 125+0+0 (secure 0 0 0) 0x7f2e4c059d20 con 0x7f2e5415a1d0 2024-03-18T20:25:38.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.571+0000 7f2e5bbaf700 1 -- 172.21.15.1:0/2877969929 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2e3806ade0 msgr2=0x7f2e3806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.571+0000 7f2e5bbaf700 1 --2- 172.21.15.1:0/2877969929 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2e3806ade0 0x7f2e3806d2a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2e440010e0 tx=0x7f2e44007850 comp rx=0 tx=0).stop 2024-03-18T20:25:38.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.571+0000 7f2e5bbaf700 1 -- 172.21.15.1:0/2877969929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e5415a1d0 msgr2=0x7f2e540c7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.571+0000 7f2e5bbaf700 1 --2- 172.21.15.1:0/2877969929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e5415a1d0 0x7f2e540c7a10 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f2e5406cfd0 tx=0x7f2e4c004ad0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.571+0000 7f2e5bbaf700 1 -- 172.21.15.1:0/2877969929 shutdown_connections 2024-03-18T20:25:38.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.571+0000 7f2e5bbaf700 1 --2- 172.21.15.1:0/2877969929 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2e3806ade0 0x7f2e3806d2a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.571+0000 7f2e5bbaf700 1 --2- 172.21.15.1:0/2877969929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e5415a1d0 0x7f2e540c7a10 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.571+0000 7f2e5bbaf700 1 --2- 172.21.15.1:0/2877969929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e54071d30 0x7f2e541578a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.571+0000 7f2e5bbaf700 1 --2- 172.21.15.1:0/2877969929 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2e540cb9e0 0x7f2e541f27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.571+0000 7f2e5bbaf700 1 -- 172.21.15.1:0/2877969929 >> 172.21.15.1:0/2877969929 conn(0x7f2e540cc9b0 msgr2=0x7f2e541594a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:38.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.571+0000 7f2e5bbaf700 1 -- 172.21.15.1:0/2877969929 shutdown_connections 2024-03-18T20:25:38.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.572+0000 7f2e5bbaf700 1 -- 172.21.15.1:0/2877969929 wait complete. 2024-03-18T20:25:38.591 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2024-03-18T20:25:38.591 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:25:38.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.720+0000 7fea53c86700 1 Processor -- start 2024-03-18T20:25:38.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.721+0000 7fea53c86700 1 -- start start 2024-03-18T20:25:38.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.721+0000 7fea53c86700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea4c146190 0x7fea4c06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:38.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.721+0000 7fea53c86700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fea4c1465a0 0x7fea4c06ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:38.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.721+0000 7fea53c86700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fea4c146af0 0x7fea4c071650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:38.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.721+0000 7fea53c86700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea4c06f850 con 0x7fea4c146af0 2024-03-18T20:25:38.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.721+0000 7fea53c86700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea4c06ebd0 con 0x7fea4c1465a0 2024-03-18T20:25:38.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.721+0000 7fea53c86700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c06eeb0 con 0x7fea4c146190 2024-03-18T20:25:38.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.722+0000 7fea52223700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fea4c146af0 0x7fea4c071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:38.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.722+0000 7fea52223700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fea4c146af0 0x7fea4c071650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49792/0 (socket says 172.21.15.1:49792) 2024-03-18T20:25:38.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.722+0000 7fea52223700 1 -- 172.21.15.1:0/1753317807 learned_addr learned my addr 172.21.15.1:0/1753317807 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:38.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.722+0000 7fea51221700 1 --2- 172.21.15.1:0/1753317807 >> v2:172.21.15.165:3300/0 conn(0x7fea4c1465a0 0x7fea4c06ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:38.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.722+0000 7fea51a22700 1 --2- 172.21.15.1:0/1753317807 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea4c146190 0x7fea4c06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:38.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.722+0000 7fea52223700 1 -- 172.21.15.1:0/1753317807 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea4c146190 msgr2=0x7fea4c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:38.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.722+0000 7fea52223700 1 --2- 172.21.15.1:0/1753317807 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea4c146190 0x7fea4c06a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.722+0000 7fea52223700 1 -- 172.21.15.1:0/1753317807 >> v2:172.21.15.165:3300/0 conn(0x7fea4c1465a0 msgr2=0x7fea4c06ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:38.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.722+0000 7fea52223700 1 --2- 172.21.15.1:0/1753317807 >> v2:172.21.15.165:3300/0 conn(0x7fea4c1465a0 0x7fea4c06ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.722+0000 7fea52223700 1 -- 172.21.15.1:0/1753317807 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4c06f190 con 0x7fea4c146af0 2024-03-18T20:25:38.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.722+0000 7fea52223700 1 --2- 172.21.15.1:0/1753317807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea4c146af0 0x7fea4c071650 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7fea4c0b62e0 tx=0x7fea4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf87e925e99aa8a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:38.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.722+0000 7fea50a20700 1 -- 172.21.15.1:0/1753317807 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4400e070 con 0x7fea4c146af0 2024-03-18T20:25:38.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.722+0000 7fea50a20700 1 -- 172.21.15.1:0/1753317807 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea4400b9c0 con 0x7fea4c146af0 2024-03-18T20:25:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.722+0000 7fea50a20700 1 -- 172.21.15.1:0/1753317807 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4400bd70 con 0x7fea4c146af0 2024-03-18T20:25:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.723+0000 7fea53c86700 1 -- 172.21.15.1:0/1753317807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea4c146af0 msgr2=0x7fea4c071650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.723+0000 7fea53c86700 1 --2- 172.21.15.1:0/1753317807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea4c146af0 0x7fea4c071650 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7fea4c0b62e0 tx=0x7fea4400b330 comp rx=0 tx=0).stop 2024-03-18T20:25:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.723+0000 7fea53c86700 1 -- 172.21.15.1:0/1753317807 shutdown_connections 2024-03-18T20:25:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.723+0000 7fea53c86700 1 --2- 172.21.15.1:0/1753317807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea4c146af0 0x7fea4c071650 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.723+0000 7fea53c86700 1 --2- 172.21.15.1:0/1753317807 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea4c146190 0x7fea4c06a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.723+0000 7fea53c86700 1 --2- 172.21.15.1:0/1753317807 >> v2:172.21.15.165:3300/0 conn(0x7fea4c1465a0 0x7fea4c06ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.723+0000 7fea53c86700 1 -- 172.21.15.1:0/1753317807 >> 172.21.15.1:0/1753317807 conn(0x7fea4c0b1d20 msgr2=0x7fea4c0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.723+0000 7fea53c86700 1 -- 172.21.15.1:0/1753317807 shutdown_connections 2024-03-18T20:25:38.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.723+0000 7fea53c86700 1 -- 172.21.15.1:0/1753317807 wait complete. 2024-03-18T20:25:38.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.724+0000 7fea53c86700 1 Processor -- start 2024-03-18T20:25:38.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.724+0000 7fea53c86700 1 -- start start 2024-03-18T20:25:38.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.725+0000 7fea53c86700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea4c1465a0 0x7fea4c1e3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:38.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.725+0000 7fea53c86700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea4c146af0 0x7fea4c0cf020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:38.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.725+0000 7fea53c86700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea4c1e2790 0x7fea4c12f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:38.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.725+0000 7fea53c86700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c1e38b0 con 0x7fea4c1465a0 2024-03-18T20:25:38.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.725+0000 7fea53c86700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c1e4210 con 0x7fea4c1e2790 2024-03-18T20:25:38.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.725+0000 7fea53c86700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c1e2c30 con 0x7fea4c146af0 2024-03-18T20:25:38.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.725+0000 7fea51a22700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea4c1465a0 0x7fea4c1e3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:38.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.725+0000 7fea51a22700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea4c1465a0 0x7fea4c1e3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49796/0 (socket says 172.21.15.1:49796) 2024-03-18T20:25:38.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.725+0000 7fea51a22700 1 -- 172.21.15.1:0/1911366907 learned_addr learned my addr 172.21.15.1:0/1911366907 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:38.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.725+0000 7fea51221700 1 --2- 172.21.15.1:0/1911366907 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea4c146af0 0x7fea4c0cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:38.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.725+0000 7fea52223700 1 --2- 172.21.15.1:0/1911366907 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea4c1e2790 0x7fea4c12f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:38.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.725+0000 7fea51a22700 1 -- 172.21.15.1:0/1911366907 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea4c146af0 msgr2=0x7fea4c0cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.725+0000 7fea51a22700 1 --2- 172.21.15.1:0/1911366907 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea4c146af0 0x7fea4c0cf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.725+0000 7fea51a22700 1 -- 172.21.15.1:0/1911366907 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea4c1e2790 msgr2=0x7fea4c12f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.725+0000 7fea51a22700 1 --2- 172.21.15.1:0/1911366907 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea4c1e2790 0x7fea4c12f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.725+0000 7fea51a22700 1 -- 172.21.15.1:0/1911366907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea44009b30 con 0x7fea4c1465a0 2024-03-18T20:25:38.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.726+0000 7fea51a22700 1 --2- 172.21.15.1:0/1911366907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea4c1465a0 0x7fea4c1e3cd0 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7fea4800e810 tx=0x7fea4800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:38.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.726+0000 7fea3e7fc700 1 -- 172.21.15.1:0/1911366907 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4800c070 con 0x7fea4c1465a0 2024-03-18T20:25:38.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.726+0000 7fea3e7fc700 1 -- 172.21.15.1:0/1911366907 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea48013610 con 0x7fea4c1465a0 2024-03-18T20:25:38.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.726+0000 7fea3e7fc700 1 -- 172.21.15.1:0/1911366907 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea480139a0 con 0x7fea4c1465a0 2024-03-18T20:25:38.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.726+0000 7fea53c86700 1 -- 172.21.15.1:0/1911366907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea4c1e2f10 con 0x7fea4c1465a0 2024-03-18T20:25:38.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.726+0000 7fea53c86700 1 -- 172.21.15.1:0/1911366907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea4c1e34d0 con 0x7fea4c1465a0 2024-03-18T20:25:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.727+0000 7fea53c86700 1 -- 172.21.15.1:0/1911366907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea4c0676a0 con 0x7fea4c1465a0 2024-03-18T20:25:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.727+0000 7fea3e7fc700 1 -- 172.21.15.1:0/1911366907 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fea48013b00 con 0x7fea4c1465a0 2024-03-18T20:25:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.728+0000 7fea3e7fc700 1 --2- 172.21.15.1:0/1911366907 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fea3006adb0 0x7fea3006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.728+0000 7fea3e7fc700 1 -- 172.21.15.1:0/1911366907 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6993+0+0 (secure 0 0 0) 0x7fea48089700 con 0x7fea4c1465a0 2024-03-18T20:25:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.728+0000 7fea51221700 1 --2- 172.21.15.1:0/1911366907 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fea3006adb0 0x7fea3006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.728+0000 7fea51221700 1 --2- 172.21.15.1:0/1911366907 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fea3006adb0 0x7fea3006d270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fea4c1e49b0 tx=0x7fea40008040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:38.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.732+0000 7fea3e7fc700 1 -- 172.21.15.1:0/1911366907 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fea48054240 con 0x7fea4c1465a0 2024-03-18T20:25:38.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.980+0000 7fea53c86700 1 -- 172.21.15.1:0/1911366907 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea4c072420 con 0x7fea4c1465a0 2024-03-18T20:25:38.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.981+0000 7fea3e7fc700 1 -- 172.21.15.1:0/1911366907 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5963 (secure 0 0 0) 0x7fea48057860 con 0x7fea4c1465a0 2024-03-18T20:25:38.985 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:38.985 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":110,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10369,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/677258734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":677258734},{"type":"v1","addr":"172.21.15.165:6835","nonce":677258734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":11544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2018398457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2018398457},{"type":"v1","addr":"172.21.15.1:6836","nonce":2018398457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:25:22.663717+0000","modified":"2024-03-18T20:25:25.668584+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11556":{"gid":11556,"name":"d","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6834/4024828630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4024828630},{"type":"v1","addr":"172.21.15.165:6834","nonce":4024828630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:25:35.549382+0000","modified":"2024-03-18T20:25:38.565325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11547":{"gid":11547,"name":"c","rank":0,"incarnation":106,"state":"up:active","state_seq":6,"addr":"172.21.15.1:6837/1635684321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1635684321},{"type":"v1","addr":"172.21.15.1:6837","nonce":1635684321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-03-18T20:25:38.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.983+0000 7fea53c86700 1 -- 172.21.15.1:0/1911366907 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fea3006adb0 msgr2=0x7fea3006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.983+0000 7fea53c86700 1 --2- 172.21.15.1:0/1911366907 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fea3006adb0 0x7fea3006d270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fea4c1e49b0 tx=0x7fea40008040 comp rx=0 tx=0).stop 2024-03-18T20:25:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.983+0000 7fea53c86700 1 -- 172.21.15.1:0/1911366907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea4c1465a0 msgr2=0x7fea4c1e3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.983+0000 7fea53c86700 1 --2- 172.21.15.1:0/1911366907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea4c1465a0 0x7fea4c1e3cd0 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7fea4800e810 tx=0x7fea4800eb20 comp rx=0 tx=0).stop 2024-03-18T20:25:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.983+0000 7fea53c86700 1 -- 172.21.15.1:0/1911366907 shutdown_connections 2024-03-18T20:25:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.983+0000 7fea53c86700 1 --2- 172.21.15.1:0/1911366907 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fea3006adb0 0x7fea3006d270 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.983+0000 7fea53c86700 1 --2- 172.21.15.1:0/1911366907 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea4c1e2790 0x7fea4c12f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.983+0000 7fea53c86700 1 --2- 172.21.15.1:0/1911366907 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea4c1465a0 0x7fea4c1e3cd0 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.983+0000 7fea53c86700 1 --2- 172.21.15.1:0/1911366907 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fea4c146af0 0x7fea4c0cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.983+0000 7fea53c86700 1 -- 172.21.15.1:0/1911366907 >> 172.21.15.1:0/1911366907 conn(0x7fea4c0b1d20 msgr2=0x7fea4c0b4a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:38.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.983+0000 7fea53c86700 1 -- 172.21.15.1:0/1911366907 shutdown_connections 2024-03-18T20:25:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:38.983+0000 7fea53c86700 1 -- 172.21.15.1:0/1911366907 wait complete. 2024-03-18T20:25:38.989 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 110 2024-03-18T20:25:38.999 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:25:39.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.121+0000 7f962d253700 1 Processor -- start 2024-03-18T20:25:39.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.121+0000 7f962d253700 1 -- start start 2024-03-18T20:25:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f962d253700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9628099a40 0x7f9628099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f962d253700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96280cc310 0x7f96280cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f962d253700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962809a3e0 con 0x7f962815d680 2024-03-18T20:25:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f962d253700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96280d0c80 con 0x7f9628099a40 2024-03-18T20:25:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f962d253700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96280d09a0 con 0x7f96280cc310 2024-03-18T20:25:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f9626d9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f962815d680 0x7f96280cff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51500/0 (socket says 172.21.15.1:51500) 2024-03-18T20:25:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f9626d9d700 1 -- 172.21.15.1:0/519562536 learned_addr learned my addr 172.21.15.1:0/519562536 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f9625d9b700 1 -- 172.21.15.1:0/519562536 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299386531 0 0) 0x7f962809a3e0 con 0x7f962815d680 2024-03-18T20:25:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f9625d9b700 1 -- 172.21.15.1:0/519562536 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96100035b0 con 0x7f962815d680 2024-03-18T20:25:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f9625d9b700 1 -- 172.21.15.1:0/519562536 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1301062567 0 0) 0x7f96100035b0 con 0x7f962815d680 2024-03-18T20:25:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f9625d9b700 1 -- 172.21.15.1:0/519562536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96280cc310 msgr2=0x7f96280cc7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T20:25:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f9625d9b700 1 --2- 172.21.15.1:0/519562536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96280cc310 0x7f96280cc7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f9625d9b700 1 -- 172.21.15.1:0/519562536 >> v2:172.21.15.165:3300/0 conn(0x7f9628099a40 msgr2=0x7f9628099ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T20:25:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f9625d9b700 1 --2- 172.21.15.1:0/519562536 >> v2:172.21.15.165:3300/0 conn(0x7f9628099a40 0x7f9628099ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f9625d9b700 1 -- 172.21.15.1:0/519562536 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96280cb830 con 0x7f962815d680 2024-03-18T20:25:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f9625d9b700 1 -- 172.21.15.1:0/519562536 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f9618001670 con 0x7f962815d680 2024-03-18T20:25:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f9625d9b700 1 -- 172.21.15.1:0/519562536 >> v1:172.21.15.1:6789/0 conn(0x7f962815d680 legacy=0x7f96280cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f9625d9b700 1 --2- 172.21.15.1:0/519562536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9610003a00 0x7f9610003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f9625d9b700 1 --2- 172.21.15.1:0/519562536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96280cc310 0x7f9610004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f9625d9b700 1 --2- 172.21.15.1:0/519562536 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9610004e00 0x7f9610005280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f9625d9b700 1 -- 172.21.15.1:0/519562536 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96280d0c80 con 0x7f9610003a00 2024-03-18T20:25:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f9625d9b700 1 -- 172.21.15.1:0/519562536 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96280d09a0 con 0x7f9610004e00 2024-03-18T20:25:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f9625d9b700 1 -- 172.21.15.1:0/519562536 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961000bcd0 con 0x7f96280cc310 2024-03-18T20:25:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.122+0000 7f9626d9d700 1 --2- 172.21.15.1:0/519562536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9610003a00 0x7f9610003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.123+0000 7f962659c700 1 --2- 172.21.15.1:0/519562536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96280cc310 0x7f9610004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.123+0000 7f9626d9d700 1 -- 172.21.15.1:0/519562536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96280cc310 msgr2=0x7f9610004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.123+0000 7f9626d9d700 1 --2- 172.21.15.1:0/519562536 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96280cc310 0x7f9610004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.123+0000 7f9626d9d700 1 -- 172.21.15.1:0/519562536 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9610004e00 msgr2=0x7f9610005280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.123+0000 7f9626d9d700 1 --2- 172.21.15.1:0/519562536 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9610004e00 0x7f9610005280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.123+0000 7f9626d9d700 1 -- 172.21.15.1:0/519562536 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96100035b0 con 0x7f9610003a00 2024-03-18T20:25:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.123+0000 7f9626d9d700 1 --2- 172.21.15.1:0/519562536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9610003a00 0x7f9610003e20 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f961800c270 tx=0x7f9610005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f1a14593628918 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.123+0000 7f9625d9b700 1 -- 172.21.15.1:0/519562536 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618007c80 con 0x7f9610003a00 2024-03-18T20:25:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.123+0000 7f9625d9b700 1 -- 172.21.15.1:0/519562536 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961800cdb0 con 0x7f9610003a00 2024-03-18T20:25:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.123+0000 7f9625d9b700 1 -- 172.21.15.1:0/519562536 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618005ba0 con 0x7f9610003a00 2024-03-18T20:25:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.123+0000 7f962d253700 1 -- 172.21.15.1:0/519562536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9610003a00 msgr2=0x7f9610003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.123+0000 7f962d253700 1 --2- 172.21.15.1:0/519562536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9610003a00 0x7f9610003e20 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f961800c270 tx=0x7f9610005880 comp rx=0 tx=0).stop 2024-03-18T20:25:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.123+0000 7f962759e700 1 -- 172.21.15.1:0/519562536 reap_dead start 2024-03-18T20:25:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.124+0000 7f962d253700 1 -- 172.21.15.1:0/519562536 shutdown_connections 2024-03-18T20:25:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.124+0000 7f962d253700 1 -- 172.21.15.1:0/519562536 >> 172.21.15.1:0/519562536 conn(0x7f96280b2ba0 msgr2=0x7f96280b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.124+0000 7f962d253700 1 -- 172.21.15.1:0/519562536 shutdown_connections 2024-03-18T20:25:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.124+0000 7f962d253700 1 -- 172.21.15.1:0/519562536 wait complete. 2024-03-18T20:25:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.125+0000 7f962d253700 1 Processor -- start 2024-03-18T20:25:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.125+0000 7f962d253700 1 -- start start 2024-03-18T20:25:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.126+0000 7f962d253700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9610003a00 0x7f962814f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.126+0000 7f962d253700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96280b4500 0x7f96280c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.126+0000 7f962d253700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96280b5a60 0x7f96280ca350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.126+0000 7f962d253700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96280b49c0 con 0x7f9610003a00 2024-03-18T20:25:39.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.126+0000 7f962d253700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96280b4ca0 con 0x7f96280b4500 2024-03-18T20:25:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.126+0000 7f962d253700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96280b4f80 con 0x7f96280b5a60 2024-03-18T20:25:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.126+0000 7f9626d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9610003a00 0x7f962814f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.126+0000 7f9626d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9610003a00 0x7f962814f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49818/0 (socket says 172.21.15.1:49818) 2024-03-18T20:25:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.126+0000 7f9626d9d700 1 -- 172.21.15.1:0/933754845 learned_addr learned my addr 172.21.15.1:0/933754845 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.126+0000 7f962659c700 1 --2- 172.21.15.1:0/933754845 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96280b4500 0x7f96280c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.126+0000 7f962759e700 1 --2- 172.21.15.1:0/933754845 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96280b5a60 0x7f96280ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.126+0000 7f9626d9d700 1 -- 172.21.15.1:0/933754845 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96280b5a60 msgr2=0x7f96280ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.126+0000 7f9626d9d700 1 --2- 172.21.15.1:0/933754845 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96280b5a60 0x7f96280ca350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.126+0000 7f9626d9d700 1 -- 172.21.15.1:0/933754845 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96280b4500 msgr2=0x7f96280c77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.126+0000 7f9626d9d700 1 --2- 172.21.15.1:0/933754845 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96280b4500 0x7f96280c77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.126+0000 7f9626d9d700 1 -- 172.21.15.1:0/933754845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96280b5260 con 0x7f9610003a00 2024-03-18T20:25:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.126+0000 7f9626d9d700 1 --2- 172.21.15.1:0/933754845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9610003a00 0x7f962814f530 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7f9618015040 tx=0x7f9618001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:39.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.126+0000 7f960f7fe700 1 -- 172.21.15.1:0/933754845 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96180072d0 con 0x7f9610003a00 2024-03-18T20:25:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.127+0000 7f960f7fe700 1 -- 172.21.15.1:0/933754845 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9618007450 con 0x7f9610003a00 2024-03-18T20:25:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.127+0000 7f962d253700 1 -- 172.21.15.1:0/933754845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96280b5540 con 0x7f9610003a00 2024-03-18T20:25:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.127+0000 7f960f7fe700 1 -- 172.21.15.1:0/933754845 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961800cc30 con 0x7f9610003a00 2024-03-18T20:25:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.127+0000 7f962d253700 1 -- 172.21.15.1:0/933754845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f962814fa70 con 0x7f9610003a00 2024-03-18T20:25:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.128+0000 7f962d253700 1 -- 172.21.15.1:0/933754845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96280b4de0 con 0x7f9610003a00 2024-03-18T20:25:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.128+0000 7f960f7fe700 1 -- 172.21.15.1:0/933754845 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9618019030 con 0x7f9610003a00 2024-03-18T20:25:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.128+0000 7f960f7fe700 1 --2- 172.21.15.1:0/933754845 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f961406adb0 0x7f961406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.128+0000 7f960f7fe700 1 -- 172.21.15.1:0/933754845 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f961808aa50 con 0x7f9610003a00 2024-03-18T20:25:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.128+0000 7f962659c700 1 --2- 172.21.15.1:0/933754845 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f961406adb0 0x7f961406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.129+0000 7f962659c700 1 --2- 172.21.15.1:0/933754845 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f961406adb0 0x7f961406d270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f961c009a20 tx=0x7f961c008040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:39.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.133+0000 7f960f7fe700 1 -- 172.21.15.1:0/933754845 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f96280b4de0 con 0x7f9610003a00 2024-03-18T20:25:39.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.337+0000 7f962d253700 1 -- 172.21.15.1:0/933754845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96280b4de0 con 0x7f9610003a00 2024-03-18T20:25:39.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.339+0000 7f960f7fe700 1 -- 172.21.15.1:0/933754845 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v236) v1 ==== 74+0+20085 (secure 0 0 0) 0x7f96280b4de0 con 0x7f9610003a00 2024-03-18T20:25:39.344 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:39.344 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":236,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:25:36.551258+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:25:20.511297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-18T20:25:21.637693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:25:33.275375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:25:34.518347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:39.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.341+0000 7f962d253700 1 -- 172.21.15.1:0/933754845 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f961406adb0 msgr2=0x7f961406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.341+0000 7f962d253700 1 --2- 172.21.15.1:0/933754845 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f961406adb0 0x7f961406d270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f961c009a20 tx=0x7f961c008040 comp rx=0 tx=0).stop 2024-03-18T20:25:39.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.341+0000 7f962d253700 1 -- 172.21.15.1:0/933754845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9610003a00 msgr2=0x7f962814f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.341+0000 7f962d253700 1 --2- 172.21.15.1:0/933754845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9610003a00 0x7f962814f530 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7f9618015040 tx=0x7f9618001a00 comp rx=0 tx=0).stop 2024-03-18T20:25:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.341+0000 7f962d253700 1 -- 172.21.15.1:0/933754845 shutdown_connections 2024-03-18T20:25:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.341+0000 7f962d253700 1 --2- 172.21.15.1:0/933754845 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f961406adb0 0x7f961406d270 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.341+0000 7f962d253700 1 --2- 172.21.15.1:0/933754845 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f96280b5a60 0x7f96280ca350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.341+0000 7f962d253700 1 --2- 172.21.15.1:0/933754845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9610003a00 0x7f962814f530 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.341+0000 7f962d253700 1 --2- 172.21.15.1:0/933754845 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96280b4500 0x7f96280c77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.341+0000 7f962d253700 1 -- 172.21.15.1:0/933754845 >> 172.21.15.1:0/933754845 conn(0x7f96280b2ba0 msgr2=0x7f96280b0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.341+0000 7f962d253700 1 -- 172.21.15.1:0/933754845 shutdown_connections 2024-03-18T20:25:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.342+0000 7f962d253700 1 -- 172.21.15.1:0/933754845 wait complete. 2024-03-18T20:25:39.358 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T20:25:39.358 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:25:39.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.479+0000 7f2c4e38a700 1 Processor -- start 2024-03-18T20:25:39.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.479+0000 7f2c4e38a700 1 -- start start 2024-03-18T20:25:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.480+0000 7f2c4e38a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2c4806ae60 0x7f2c48164f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.480+0000 7f2c4e38a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2c481465a0 0x7f2c48167480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.480+0000 7f2c4e38a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c48072660 con 0x7f2c48146190 2024-03-18T20:25:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.480+0000 7f2c4e38a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c4806f850 con 0x7f2c4806ae60 2024-03-18T20:25:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.480+0000 7f2c4e38a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c4806ebd0 con 0x7f2c481465a0 2024-03-18T20:25:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.480+0000 7f2c477fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2c4806ae60 0x7f2c48164f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.480+0000 7f2c477fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2c4806ae60 0x7f2c48164f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49828/0 (socket says 172.21.15.1:49828) 2024-03-18T20:25:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.480+0000 7f2c47fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2c48146190 0x7f2c4806a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51510/0 (socket says 172.21.15.1:51510) 2024-03-18T20:25:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.480+0000 7f2c477fe700 1 -- 172.21.15.1:0/1416970496 learned_addr learned my addr 172.21.15.1:0/1416970496 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.480+0000 7f2c4c927700 1 --2- 172.21.15.1:0/1416970496 >> v2:172.21.15.165:3300/0 conn(0x7f2c481465a0 0x7f2c48167480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.480+0000 7f2c477fe700 1 -- 172.21.15.1:0/1416970496 >> v2:172.21.15.165:3300/0 conn(0x7f2c481465a0 msgr2=0x7f2c48167480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.480+0000 7f2c477fe700 1 --2- 172.21.15.1:0/1416970496 >> v2:172.21.15.165:3300/0 conn(0x7f2c481465a0 0x7f2c48167480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.480+0000 7f2c477fe700 1 -- 172.21.15.1:0/1416970496 >> v1:172.21.15.1:6789/0 conn(0x7f2c48146190 legacy=0x7f2c4806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.480+0000 7f2c477fe700 1 -- 172.21.15.1:0/1416970496 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4806eeb0 con 0x7f2c4806ae60 2024-03-18T20:25:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.480+0000 7f2c477fe700 1 --2- 172.21.15.1:0/1416970496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4806ae60 0x7f2c48164f30 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f2c480715e0 tx=0x7f2c30009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9ecc2ac46110535 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.481+0000 7f2c46ffd700 1 -- 172.21.15.1:0/1416970496 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3000b870 con 0x7f2c4806ae60 2024-03-18T20:25:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.481+0000 7f2c46ffd700 1 -- 172.21.15.1:0/1416970496 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3000b9d0 con 0x7f2c4806ae60 2024-03-18T20:25:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.481+0000 7f2c46ffd700 1 -- 172.21.15.1:0/1416970496 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3000bd60 con 0x7f2c4806ae60 2024-03-18T20:25:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.481+0000 7f2c4e38a700 1 -- 172.21.15.1:0/1416970496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4806ae60 msgr2=0x7f2c48164f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.481+0000 7f2c4e38a700 1 --2- 172.21.15.1:0/1416970496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4806ae60 0x7f2c48164f30 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f2c480715e0 tx=0x7f2c30009bd0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.481+0000 7f2c4e38a700 1 -- 172.21.15.1:0/1416970496 shutdown_connections 2024-03-18T20:25:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.481+0000 7f2c4e38a700 1 --2- 172.21.15.1:0/1416970496 >> v2:172.21.15.165:3300/0 conn(0x7f2c481465a0 0x7f2c48167480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.481+0000 7f2c4e38a700 1 --2- 172.21.15.1:0/1416970496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4806ae60 0x7f2c48164f30 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.481+0000 7f2c4e38a700 1 -- 172.21.15.1:0/1416970496 >> 172.21.15.1:0/1416970496 conn(0x7f2c480b1d20 msgr2=0x7f2c480b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:39.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.481+0000 7f2c4e38a700 1 -- 172.21.15.1:0/1416970496 shutdown_connections 2024-03-18T20:25:39.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.481+0000 7f2c4e38a700 1 -- 172.21.15.1:0/1416970496 wait complete. 2024-03-18T20:25:39.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.482+0000 7f2c4e38a700 1 Processor -- start 2024-03-18T20:25:39.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.483+0000 7f2c4e38a700 1 -- start start 2024-03-18T20:25:39.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.483+0000 7f2c4e38a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4806ae60 0x7f2c480c4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.483+0000 7f2c4e38a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c481465a0 0x7f2c480cda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.483+0000 7f2c4e38a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c4813fb60 0x7f2c480cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.483+0000 7f2c4e38a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4813e810 con 0x7f2c4806ae60 2024-03-18T20:25:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.483+0000 7f2c4e38a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4815d3e0 con 0x7f2c4813fb60 2024-03-18T20:25:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.483+0000 7f2c4e38a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c480c54d0 con 0x7f2c481465a0 2024-03-18T20:25:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.483+0000 7f2c47fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4806ae60 0x7f2c480c4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.483+0000 7f2c47fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4806ae60 0x7f2c480c4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49832/0 (socket says 172.21.15.1:49832) 2024-03-18T20:25:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.484+0000 7f2c47fff700 1 -- 172.21.15.1:0/2255221758 learned_addr learned my addr 172.21.15.1:0/2255221758 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:39.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.484+0000 7f2c477fe700 1 --2- 172.21.15.1:0/2255221758 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c481465a0 0x7f2c480cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.484+0000 7f2c47fff700 1 -- 172.21.15.1:0/2255221758 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c481465a0 msgr2=0x7f2c480cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.484+0000 7f2c4c927700 1 --2- 172.21.15.1:0/2255221758 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c4813fb60 0x7f2c480cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.484+0000 7f2c47fff700 1 --2- 172.21.15.1:0/2255221758 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c481465a0 0x7f2c480cda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.484+0000 7f2c47fff700 1 -- 172.21.15.1:0/2255221758 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c4813fb60 msgr2=0x7f2c480cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.484+0000 7f2c47fff700 1 --2- 172.21.15.1:0/2255221758 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c4813fb60 0x7f2c480cdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.484+0000 7f2c47fff700 1 -- 172.21.15.1:0/2255221758 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c30009550 con 0x7f2c4806ae60 2024-03-18T20:25:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.484+0000 7f2c47fff700 1 --2- 172.21.15.1:0/2255221758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4806ae60 0x7f2c480c4f90 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f2c3800c550 tx=0x7f2c3800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.484+0000 7f2c44ff9700 1 -- 172.21.15.1:0/2255221758 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c38005cb0 con 0x7f2c4806ae60 2024-03-18T20:25:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.484+0000 7f2c44ff9700 1 -- 172.21.15.1:0/2255221758 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c38005e10 con 0x7f2c4806ae60 2024-03-18T20:25:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.484+0000 7f2c44ff9700 1 -- 172.21.15.1:0/2255221758 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c380072d0 con 0x7f2c4806ae60 2024-03-18T20:25:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.484+0000 7f2c4e38a700 1 -- 172.21.15.1:0/2255221758 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c48140100 con 0x7f2c4806ae60 2024-03-18T20:25:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.484+0000 7f2c4e38a700 1 -- 172.21.15.1:0/2255221758 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c481406c0 con 0x7f2c4806ae60 2024-03-18T20:25:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.485+0000 7f2c4e38a700 1 -- 172.21.15.1:0/2255221758 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c480676a0 con 0x7f2c4806ae60 2024-03-18T20:25:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.485+0000 7f2c44ff9700 1 -- 172.21.15.1:0/2255221758 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2c38007430 con 0x7f2c4806ae60 2024-03-18T20:25:39.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.486+0000 7f2c44ff9700 1 --2- 172.21.15.1:0/2255221758 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2c2806adb0 0x7f2c2806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.486+0000 7f2c44ff9700 1 -- 172.21.15.1:0/2255221758 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f2c38088d40 con 0x7f2c4806ae60 2024-03-18T20:25:39.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.486+0000 7f2c477fe700 1 --2- 172.21.15.1:0/2255221758 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2c2806adb0 0x7f2c2806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.486+0000 7f2c477fe700 1 --2- 172.21.15.1:0/2255221758 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2c2806adb0 0x7f2c2806d270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2c480ad3d0 tx=0x7f2c30008030 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:39.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.491+0000 7f2c44ff9700 1 -- 172.21.15.1:0/2255221758 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2c380539b0 con 0x7f2c4806ae60 2024-03-18T20:25:39.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.688+0000 7f2c4e38a700 1 -- 172.21.15.1:0/2255221758 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2c4806f470 con 0x7f2c2806adb0 2024-03-18T20:25:39.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.702+0000 7f2c44ff9700 1 -- 172.21.15.1:0/2255221758 <== mgr.11322 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+518107 (secure 0 0 0) 0x7f2c3001f070 con 0x7f2c2806adb0 2024-03-18T20:25:39.715 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:39.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.706+0000 7f2c4e38a700 1 -- 172.21.15.1:0/2255221758 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2c2806adb0 msgr2=0x7f2c2806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.706+0000 7f2c4e38a700 1 --2- 172.21.15.1:0/2255221758 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2c2806adb0 0x7f2c2806d270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2c480ad3d0 tx=0x7f2c30008030 comp rx=0 tx=0).stop 2024-03-18T20:25:39.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.706+0000 7f2c4e38a700 1 -- 172.21.15.1:0/2255221758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4806ae60 msgr2=0x7f2c480c4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.706+0000 7f2c4e38a700 1 --2- 172.21.15.1:0/2255221758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4806ae60 0x7f2c480c4f90 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f2c3800c550 tx=0x7f2c3800c910 comp rx=0 tx=0).stop 2024-03-18T20:25:39.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.706+0000 7f2c4e38a700 1 -- 172.21.15.1:0/2255221758 shutdown_connections 2024-03-18T20:25:39.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.706+0000 7f2c4e38a700 1 --2- 172.21.15.1:0/2255221758 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2c2806adb0 0x7f2c2806d270 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.706+0000 7f2c4e38a700 1 --2- 172.21.15.1:0/2255221758 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c4813fb60 0x7f2c480cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.706+0000 7f2c4e38a700 1 --2- 172.21.15.1:0/2255221758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4806ae60 0x7f2c480c4f90 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.706+0000 7f2c4e38a700 1 --2- 172.21.15.1:0/2255221758 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c481465a0 0x7f2c480cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.706+0000 7f2c4e38a700 1 -- 172.21.15.1:0/2255221758 >> 172.21.15.1:0/2255221758 conn(0x7f2c480b1d20 msgr2=0x7f2c480b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:39.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.706+0000 7f2c4e38a700 1 -- 172.21.15.1:0/2255221758 shutdown_connections 2024-03-18T20:25:39.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.706+0000 7f2c4e38a700 1 -- 172.21.15.1:0/2255221758 wait complete. 2024-03-18T20:25:39.722 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:25:39.723 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-18T20:25:39.102708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593988,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":626,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":389,"ondisk_log_size":389,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":212,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":385384,"kb_used_data":24072,"kb_used_omap":12,"kb_used_meta":361203,"kb_avail":754589336,"statfs":{"total":773094113280,"available":772699480064,"internally_reserved":0,"allocated":24649728,"data_stored":13189064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":369872441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":815,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.995476"},"pg_stats":[{"pgid":"1.0","version":"221'348","reported_seq":1201,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605524+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:25:24.605524+0000","last_peered":"2024-03-18T20:25:24.605524+0000","last_clean":"2024-03-18T20:25:24.605524+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:25:24.605524+0000","last_undegraded":"2024-03-18T20:25:24.605524+0000","last_fullsized":"2024-03-18T20:25:24.605524+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"231'1","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100643+0000","last_change":"2024-03-18T20:25:22.248248+0000","last_active":"2024-03-18T20:25:25.100643+0000","last_peered":"2024-03-18T20:25:25.100643+0000","last_clean":"2024-03-18T20:25:25.100643+0000","last_became_active":"2024-03-18T20:25:22.247970+0000","last_became_peered":"2024-03-18T20:25:22.247970+0000","last_unstale":"2024-03-18T20:25:25.100643+0000","last_undegraded":"2024-03-18T20:25:25.100643+0000","last_fullsized":"2024-03-18T20:25:25.100643+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:01:50.523530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101112+0000","last_change":"2024-03-18T20:25:22.687598+0000","last_active":"2024-03-18T20:25:25.101112+0000","last_peered":"2024-03-18T20:25:25.101112+0000","last_clean":"2024-03-18T20:25:25.101112+0000","last_became_active":"2024-03-18T20:25:22.686962+0000","last_became_peered":"2024-03-18T20:25:22.686962+0000","last_unstale":"2024-03-18T20:25:25.101112+0000","last_undegraded":"2024-03-18T20:25:25.101112+0000","last_fullsized":"2024-03-18T20:25:25.101112+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:23:10.983253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.114498+0000","last_change":"2024-03-18T20:25:34.114498+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.114498+0000","last_undegraded":"2024-03-18T20:25:34.114498+0000","last_fullsized":"2024-03-18T20:25:34.114498+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:47.366224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101303+0000","last_change":"2024-03-18T20:25:22.252803+0000","last_active":"2024-03-18T20:25:25.101303+0000","last_peered":"2024-03-18T20:25:25.101303+0000","last_clean":"2024-03-18T20:25:25.101303+0000","last_became_active":"2024-03-18T20:25:22.252169+0000","last_became_peered":"2024-03-18T20:25:22.252169+0000","last_unstale":"2024-03-18T20:25:25.101303+0000","last_undegraded":"2024-03-18T20:25:25.101303+0000","last_fullsized":"2024-03-18T20:25:25.101303+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:12.985428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097715+0000","last_change":"2024-03-18T20:25:22.696025+0000","last_active":"2024-03-18T20:25:25.097715+0000","last_peered":"2024-03-18T20:25:25.097715+0000","last_clean":"2024-03-18T20:25:25.097715+0000","last_became_active":"2024-03-18T20:25:22.695759+0000","last_became_peered":"2024-03-18T20:25:22.695759+0000","last_unstale":"2024-03-18T20:25:25.097715+0000","last_undegraded":"2024-03-18T20:25:25.097715+0000","last_fullsized":"2024-03-18T20:25:25.097715+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:10.681789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.117162+0000","last_change":"2024-03-18T20:25:34.117162+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.117162+0000","last_undegraded":"2024-03-18T20:25:34.117162+0000","last_fullsized":"2024-03-18T20:25:34.117162+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:13.759015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101033+0000","last_change":"2024-03-18T20:25:22.252889+0000","last_active":"2024-03-18T20:25:25.101033+0000","last_peered":"2024-03-18T20:25:25.101033+0000","last_clean":"2024-03-18T20:25:25.101033+0000","last_became_active":"2024-03-18T20:25:22.252096+0000","last_became_peered":"2024-03-18T20:25:22.252096+0000","last_unstale":"2024-03-18T20:25:25.101033+0000","last_undegraded":"2024-03-18T20:25:25.101033+0000","last_fullsized":"2024-03-18T20:25:25.101033+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:18.568149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101094+0000","last_change":"2024-03-18T20:25:22.689532+0000","last_active":"2024-03-18T20:25:25.101094+0000","last_peered":"2024-03-18T20:25:25.101094+0000","last_clean":"2024-03-18T20:25:25.101094+0000","last_became_active":"2024-03-18T20:25:22.689210+0000","last_became_peered":"2024-03-18T20:25:22.689210+0000","last_unstale":"2024-03-18T20:25:25.101094+0000","last_undegraded":"2024-03-18T20:25:25.101094+0000","last_fullsized":"2024-03-18T20:25:25.101094+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:21.229791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.111213+0000","last_change":"2024-03-18T20:25:34.111213+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.111213+0000","last_undegraded":"2024-03-18T20:25:34.111213+0000","last_fullsized":"2024-03-18T20:25:34.111213+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:17.295248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100663+0000","last_change":"2024-03-18T20:25:22.254053+0000","last_active":"2024-03-18T20:25:25.100663+0000","last_peered":"2024-03-18T20:25:25.100663+0000","last_clean":"2024-03-18T20:25:25.100663+0000","last_became_active":"2024-03-18T20:25:22.253747+0000","last_became_peered":"2024-03-18T20:25:22.253747+0000","last_unstale":"2024-03-18T20:25:25.100663+0000","last_undegraded":"2024-03-18T20:25:25.100663+0000","last_fullsized":"2024-03-18T20:25:25.100663+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:32.777143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101083+0000","last_change":"2024-03-18T20:25:22.685300+0000","last_active":"2024-03-18T20:25:25.101083+0000","last_peered":"2024-03-18T20:25:25.101083+0000","last_clean":"2024-03-18T20:25:25.101083+0000","last_became_active":"2024-03-18T20:25:22.684869+0000","last_became_peered":"2024-03-18T20:25:22.684869+0000","last_unstale":"2024-03-18T20:25:25.101083+0000","last_undegraded":"2024-03-18T20:25:25.101083+0000","last_fullsized":"2024-03-18T20:25:25.101083+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:02.037780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.110932+0000","last_change":"2024-03-18T20:25:34.110932+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.110932+0000","last_undegraded":"2024-03-18T20:25:34.110932+0000","last_fullsized":"2024-03-18T20:25:34.110932+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:44:27.205313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099377+0000","last_change":"2024-03-18T20:25:22.245633+0000","last_active":"2024-03-18T20:25:25.099377+0000","last_peered":"2024-03-18T20:25:25.099377+0000","last_clean":"2024-03-18T20:25:25.099377+0000","last_became_active":"2024-03-18T20:25:22.245372+0000","last_became_peered":"2024-03-18T20:25:22.245372+0000","last_unstale":"2024-03-18T20:25:25.099377+0000","last_undegraded":"2024-03-18T20:25:25.099377+0000","last_fullsized":"2024-03-18T20:25:25.099377+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:28:25.048682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099401+0000","last_change":"2024-03-18T20:25:22.683574+0000","last_active":"2024-03-18T20:25:25.099401+0000","last_peered":"2024-03-18T20:25:25.099401+0000","last_clean":"2024-03-18T20:25:25.099401+0000","last_became_active":"2024-03-18T20:25:22.683015+0000","last_became_peered":"2024-03-18T20:25:22.683015+0000","last_unstale":"2024-03-18T20:25:25.099401+0000","last_undegraded":"2024-03-18T20:25:25.099401+0000","last_fullsized":"2024-03-18T20:25:25.099401+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:11.606826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.116497+0000","last_change":"2024-03-18T20:25:34.116497+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.116497+0000","last_undegraded":"2024-03-18T20:25:34.116497+0000","last_fullsized":"2024-03-18T20:25:34.116497+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:41.670198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101098+0000","last_change":"2024-03-18T20:25:22.255435+0000","last_active":"2024-03-18T20:25:25.101098+0000","last_peered":"2024-03-18T20:25:25.101098+0000","last_clean":"2024-03-18T20:25:25.101098+0000","last_became_active":"2024-03-18T20:25:22.255149+0000","last_became_peered":"2024-03-18T20:25:22.255149+0000","last_unstale":"2024-03-18T20:25:25.101098+0000","last_undegraded":"2024-03-18T20:25:25.101098+0000","last_fullsized":"2024-03-18T20:25:25.101098+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:58.960198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101079+0000","last_change":"2024-03-18T20:25:22.682910+0000","last_active":"2024-03-18T20:25:25.101079+0000","last_peered":"2024-03-18T20:25:25.101079+0000","last_clean":"2024-03-18T20:25:25.101079+0000","last_became_active":"2024-03-18T20:25:22.682667+0000","last_became_peered":"2024-03-18T20:25:22.682667+0000","last_unstale":"2024-03-18T20:25:25.101079+0000","last_undegraded":"2024-03-18T20:25:25.101079+0000","last_fullsized":"2024-03-18T20:25:25.101079+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:02.037780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.115535+0000","last_change":"2024-03-18T20:25:34.115535+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.115535+0000","last_undegraded":"2024-03-18T20:25:34.115535+0000","last_fullsized":"2024-03-18T20:25:34.115535+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:56.513767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100915+0000","last_change":"2024-03-18T20:25:22.250457+0000","last_active":"2024-03-18T20:25:25.100915+0000","last_peered":"2024-03-18T20:25:25.100915+0000","last_clean":"2024-03-18T20:25:25.100915+0000","last_became_active":"2024-03-18T20:25:22.250157+0000","last_became_peered":"2024-03-18T20:25:22.250157+0000","last_unstale":"2024-03-18T20:25:25.100915+0000","last_undegraded":"2024-03-18T20:25:25.100915+0000","last_fullsized":"2024-03-18T20:25:25.100915+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:18.568149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100623+0000","last_change":"2024-03-18T20:25:22.691245+0000","last_active":"2024-03-18T20:25:25.100623+0000","last_peered":"2024-03-18T20:25:25.100623+0000","last_clean":"2024-03-18T20:25:25.100623+0000","last_became_active":"2024-03-18T20:25:22.690988+0000","last_became_peered":"2024-03-18T20:25:22.690988+0000","last_unstale":"2024-03-18T20:25:25.100623+0000","last_undegraded":"2024-03-18T20:25:25.100623+0000","last_fullsized":"2024-03-18T20:25:25.100623+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:32:54.125659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.112128+0000","last_change":"2024-03-18T20:25:34.112128+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.112128+0000","last_undegraded":"2024-03-18T20:25:34.112128+0000","last_fullsized":"2024-03-18T20:25:34.112128+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:25:32.071310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.6","version":"232'1","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:27.359403+0000","last_change":"2024-03-18T20:25:22.247183+0000","last_active":"2024-03-18T20:25:27.359403+0000","last_peered":"2024-03-18T20:25:27.359403+0000","last_clean":"2024-03-18T20:25:27.359403+0000","last_became_active":"2024-03-18T20:25:22.246890+0000","last_became_peered":"2024-03-18T20:25:22.246890+0000","last_unstale":"2024-03-18T20:25:27.359403+0000","last_undegraded":"2024-03-18T20:25:27.359403+0000","last_fullsized":"2024-03-18T20:25:27.359403+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:40:21.886849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097599+0000","last_change":"2024-03-18T20:25:22.689011+0000","last_active":"2024-03-18T20:25:25.097599+0000","last_peered":"2024-03-18T20:25:25.097599+0000","last_clean":"2024-03-18T20:25:25.097599+0000","last_became_active":"2024-03-18T20:25:22.688750+0000","last_became_peered":"2024-03-18T20:25:22.688750+0000","last_unstale":"2024-03-18T20:25:25.097599+0000","last_undegraded":"2024-03-18T20:25:25.097599+0000","last_fullsized":"2024-03-18T20:25:25.097599+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:57.345369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.111764+0000","last_change":"2024-03-18T20:25:34.111764+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.111764+0000","last_undegraded":"2024-03-18T20:25:34.111764+0000","last_fullsized":"2024-03-18T20:25:34.111764+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:25.724222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.7","version":"231'2","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100865+0000","last_change":"2024-03-18T20:25:22.254151+0000","last_active":"2024-03-18T20:25:25.100865+0000","last_peered":"2024-03-18T20:25:25.100865+0000","last_clean":"2024-03-18T20:25:25.100865+0000","last_became_active":"2024-03-18T20:25:22.253873+0000","last_became_peered":"2024-03-18T20:25:22.253873+0000","last_unstale":"2024-03-18T20:25:25.100865+0000","last_undegraded":"2024-03-18T20:25:25.100865+0000","last_fullsized":"2024-03-18T20:25:25.100865+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:26:58.327699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099308+0000","last_change":"2024-03-18T20:25:22.693170+0000","last_active":"2024-03-18T20:25:25.099308+0000","last_peered":"2024-03-18T20:25:25.099308+0000","last_clean":"2024-03-18T20:25:25.099308+0000","last_became_active":"2024-03-18T20:25:22.691592+0000","last_became_peered":"2024-03-18T20:25:22.691592+0000","last_unstale":"2024-03-18T20:25:25.099308+0000","last_undegraded":"2024-03-18T20:25:25.099308+0000","last_fullsized":"2024-03-18T20:25:25.099308+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:23.897085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.114953+0000","last_change":"2024-03-18T20:25:34.114953+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.114953+0000","last_undegraded":"2024-03-18T20:25:34.114953+0000","last_fullsized":"2024-03-18T20:25:34.114953+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:06:09.253104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100981+0000","last_change":"2024-03-18T20:25:22.251509+0000","last_active":"2024-03-18T20:25:25.100981+0000","last_peered":"2024-03-18T20:25:25.100981+0000","last_clean":"2024-03-18T20:25:25.100981+0000","last_became_active":"2024-03-18T20:25:22.251240+0000","last_became_peered":"2024-03-18T20:25:22.251240+0000","last_unstale":"2024-03-18T20:25:25.100981+0000","last_undegraded":"2024-03-18T20:25:25.100981+0000","last_fullsized":"2024-03-18T20:25:25.100981+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:13:14.870270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099451+0000","last_change":"2024-03-18T20:25:22.695653+0000","last_active":"2024-03-18T20:25:25.099451+0000","last_peered":"2024-03-18T20:25:25.099451+0000","last_clean":"2024-03-18T20:25:25.099451+0000","last_became_active":"2024-03-18T20:25:22.694092+0000","last_became_peered":"2024-03-18T20:25:22.694092+0000","last_unstale":"2024-03-18T20:25:25.099451+0000","last_undegraded":"2024-03-18T20:25:25.099451+0000","last_fullsized":"2024-03-18T20:25:25.099451+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:20.319345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.115435+0000","last_change":"2024-03-18T20:25:34.115435+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.115435+0000","last_undegraded":"2024-03-18T20:25:34.115435+0000","last_fullsized":"2024-03-18T20:25:34.115435+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:32:42.191508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100853+0000","last_change":"2024-03-18T20:25:22.251818+0000","last_active":"2024-03-18T20:25:25.100853+0000","last_peered":"2024-03-18T20:25:25.100853+0000","last_clean":"2024-03-18T20:25:25.100853+0000","last_became_active":"2024-03-18T20:25:22.251407+0000","last_became_peered":"2024-03-18T20:25:22.251407+0000","last_unstale":"2024-03-18T20:25:25.100853+0000","last_undegraded":"2024-03-18T20:25:25.100853+0000","last_fullsized":"2024-03-18T20:25:25.100853+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:29.196742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100780+0000","last_change":"2024-03-18T20:25:22.684929+0000","last_active":"2024-03-18T20:25:25.100780+0000","last_peered":"2024-03-18T20:25:25.100780+0000","last_clean":"2024-03-18T20:25:25.100780+0000","last_became_active":"2024-03-18T20:25:22.684658+0000","last_became_peered":"2024-03-18T20:25:22.684658+0000","last_unstale":"2024-03-18T20:25:25.100780+0000","last_undegraded":"2024-03-18T20:25:25.100780+0000","last_fullsized":"2024-03-18T20:25:25.100780+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:39.733357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.113347+0000","last_change":"2024-03-18T20:25:34.113347+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.113347+0000","last_undegraded":"2024-03-18T20:25:34.113347+0000","last_fullsized":"2024-03-18T20:25:34.113347+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:22:13.432979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100944+0000","last_change":"2024-03-18T20:25:22.247489+0000","last_active":"2024-03-18T20:25:25.100944+0000","last_peered":"2024-03-18T20:25:25.100944+0000","last_clean":"2024-03-18T20:25:25.100944+0000","last_became_active":"2024-03-18T20:25:22.247203+0000","last_became_peered":"2024-03-18T20:25:22.247203+0000","last_unstale":"2024-03-18T20:25:25.100944+0000","last_undegraded":"2024-03-18T20:25:25.100944+0000","last_fullsized":"2024-03-18T20:25:25.100944+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:32.481118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101073+0000","last_change":"2024-03-18T20:25:22.681615+0000","last_active":"2024-03-18T20:25:25.101073+0000","last_peered":"2024-03-18T20:25:25.101073+0000","last_clean":"2024-03-18T20:25:25.101073+0000","last_became_active":"2024-03-18T20:25:22.681261+0000","last_became_peered":"2024-03-18T20:25:22.681261+0000","last_unstale":"2024-03-18T20:25:25.101073+0000","last_undegraded":"2024-03-18T20:25:25.101073+0000","last_fullsized":"2024-03-18T20:25:25.101073+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:15:29.461741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.111539+0000","last_change":"2024-03-18T20:25:34.111539+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.111539+0000","last_undegraded":"2024-03-18T20:25:34.111539+0000","last_fullsized":"2024-03-18T20:25:34.111539+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:33.109951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.b","version":"231'2","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101371+0000","last_change":"2024-03-18T20:25:22.251851+0000","last_active":"2024-03-18T20:25:25.101371+0000","last_peered":"2024-03-18T20:25:25.101371+0000","last_clean":"2024-03-18T20:25:25.101371+0000","last_became_active":"2024-03-18T20:25:22.251334+0000","last_became_peered":"2024-03-18T20:25:22.251334+0000","last_unstale":"2024-03-18T20:25:25.101371+0000","last_undegraded":"2024-03-18T20:25:25.101371+0000","last_fullsized":"2024-03-18T20:25:25.101371+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:22.889781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100991+0000","last_change":"2024-03-18T20:25:22.687129+0000","last_active":"2024-03-18T20:25:25.100991+0000","last_peered":"2024-03-18T20:25:25.100991+0000","last_clean":"2024-03-18T20:25:25.100991+0000","last_became_active":"2024-03-18T20:25:22.686871+0000","last_became_peered":"2024-03-18T20:25:22.686871+0000","last_unstale":"2024-03-18T20:25:25.100991+0000","last_undegraded":"2024-03-18T20:25:25.100991+0000","last_fullsized":"2024-03-18T20:25:25.100991+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:18:58.196895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.111594+0000","last_change":"2024-03-18T20:25:34.111594+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.111594+0000","last_undegraded":"2024-03-18T20:25:34.111594+0000","last_fullsized":"2024-03-18T20:25:34.111594+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:13.917940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101413+0000","last_change":"2024-03-18T20:25:22.254305+0000","last_active":"2024-03-18T20:25:25.101413+0000","last_peered":"2024-03-18T20:25:25.101413+0000","last_clean":"2024-03-18T20:25:25.101413+0000","last_became_active":"2024-03-18T20:25:22.253898+0000","last_became_peered":"2024-03-18T20:25:22.253898+0000","last_unstale":"2024-03-18T20:25:25.101413+0000","last_undegraded":"2024-03-18T20:25:25.101413+0000","last_fullsized":"2024-03-18T20:25:25.101413+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:29:48.394283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099365+0000","last_change":"2024-03-18T20:25:22.692895+0000","last_active":"2024-03-18T20:25:25.099365+0000","last_peered":"2024-03-18T20:25:25.099365+0000","last_clean":"2024-03-18T20:25:25.099365+0000","last_became_active":"2024-03-18T20:25:22.685766+0000","last_became_peered":"2024-03-18T20:25:22.685766+0000","last_unstale":"2024-03-18T20:25:25.099365+0000","last_undegraded":"2024-03-18T20:25:25.099365+0000","last_fullsized":"2024-03-18T20:25:25.099365+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:16:08.020891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.117145+0000","last_change":"2024-03-18T20:25:34.117145+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.117145+0000","last_undegraded":"2024-03-18T20:25:34.117145+0000","last_fullsized":"2024-03-18T20:25:34.117145+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:38.721170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.d","version":"231'1","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101013+0000","last_change":"2024-03-18T20:25:22.248228+0000","last_active":"2024-03-18T20:25:25.101013+0000","last_peered":"2024-03-18T20:25:25.101013+0000","last_clean":"2024-03-18T20:25:25.101013+0000","last_became_active":"2024-03-18T20:25:22.247916+0000","last_became_peered":"2024-03-18T20:25:22.247916+0000","last_unstale":"2024-03-18T20:25:25.101013+0000","last_undegraded":"2024-03-18T20:25:25.101013+0000","last_fullsized":"2024-03-18T20:25:25.101013+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:56.236862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101055+0000","last_change":"2024-03-18T20:25:22.684910+0000","last_active":"2024-03-18T20:25:25.101055+0000","last_peered":"2024-03-18T20:25:25.101055+0000","last_clean":"2024-03-18T20:25:25.101055+0000","last_became_active":"2024-03-18T20:25:22.684472+0000","last_became_peered":"2024-03-18T20:25:22.684472+0000","last_unstale":"2024-03-18T20:25:25.101055+0000","last_undegraded":"2024-03-18T20:25:25.101055+0000","last_fullsized":"2024-03-18T20:25:25.101055+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:24:33.293594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.112636+0000","last_change":"2024-03-18T20:25:34.112636+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.112636+0000","last_undegraded":"2024-03-18T20:25:34.112636+0000","last_fullsized":"2024-03-18T20:25:34.112636+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:15.369281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.e","version":"231'2","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101154+0000","last_change":"2024-03-18T20:25:22.253997+0000","last_active":"2024-03-18T20:25:25.101154+0000","last_peered":"2024-03-18T20:25:25.101154+0000","last_clean":"2024-03-18T20:25:25.101154+0000","last_became_active":"2024-03-18T20:25:22.253649+0000","last_became_peered":"2024-03-18T20:25:22.253649+0000","last_unstale":"2024-03-18T20:25:25.101154+0000","last_undegraded":"2024-03-18T20:25:25.101154+0000","last_fullsized":"2024-03-18T20:25:25.101154+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:00.435897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100997+0000","last_change":"2024-03-18T20:25:22.687892+0000","last_active":"2024-03-18T20:25:25.100997+0000","last_peered":"2024-03-18T20:25:25.100997+0000","last_clean":"2024-03-18T20:25:25.100997+0000","last_became_active":"2024-03-18T20:25:22.687086+0000","last_became_peered":"2024-03-18T20:25:22.687086+0000","last_unstale":"2024-03-18T20:25:25.100997+0000","last_undegraded":"2024-03-18T20:25:25.100997+0000","last_fullsized":"2024-03-18T20:25:25.100997+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:15:29.461741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.115166+0000","last_change":"2024-03-18T20:25:34.115166+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.115166+0000","last_undegraded":"2024-03-18T20:25:34.115166+0000","last_fullsized":"2024-03-18T20:25:34.115166+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:17.295248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.f","version":"231'1","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101196+0000","last_change":"2024-03-18T20:25:22.255372+0000","last_active":"2024-03-18T20:25:25.101196+0000","last_peered":"2024-03-18T20:25:25.101196+0000","last_clean":"2024-03-18T20:25:25.101196+0000","last_became_active":"2024-03-18T20:25:22.255063+0000","last_became_peered":"2024-03-18T20:25:22.255063+0000","last_unstale":"2024-03-18T20:25:25.101196+0000","last_undegraded":"2024-03-18T20:25:25.101196+0000","last_fullsized":"2024-03-18T20:25:25.101196+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:23.143345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099295+0000","last_change":"2024-03-18T20:25:22.686185+0000","last_active":"2024-03-18T20:25:25.099295+0000","last_peered":"2024-03-18T20:25:25.099295+0000","last_clean":"2024-03-18T20:25:25.099295+0000","last_became_active":"2024-03-18T20:25:22.685939+0000","last_became_peered":"2024-03-18T20:25:22.685939+0000","last_unstale":"2024-03-18T20:25:25.099295+0000","last_undegraded":"2024-03-18T20:25:25.099295+0000","last_fullsized":"2024-03-18T20:25:25.099295+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:48:48.961872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.116524+0000","last_change":"2024-03-18T20:25:34.116524+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.116524+0000","last_undegraded":"2024-03-18T20:25:34.116524+0000","last_fullsized":"2024-03-18T20:25:34.116524+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:50.264082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.10","version":"231'8","reported_seq":20,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101286+0000","last_change":"2024-03-18T20:25:22.252976+0000","last_active":"2024-03-18T20:25:25.101286+0000","last_peered":"2024-03-18T20:25:25.101286+0000","last_clean":"2024-03-18T20:25:25.101286+0000","last_became_active":"2024-03-18T20:25:22.252528+0000","last_became_peered":"2024-03-18T20:25:22.252528+0000","last_unstale":"2024-03-18T20:25:25.101286+0000","last_undegraded":"2024-03-18T20:25:25.101286+0000","last_fullsized":"2024-03-18T20:25:25.101286+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:42:24.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101120+0000","last_change":"2024-03-18T20:25:22.673559+0000","last_active":"2024-03-18T20:25:25.101120+0000","last_peered":"2024-03-18T20:25:25.101120+0000","last_clean":"2024-03-18T20:25:25.101120+0000","last_became_active":"2024-03-18T20:25:22.673267+0000","last_became_peered":"2024-03-18T20:25:22.673267+0000","last_unstale":"2024-03-18T20:25:25.101120+0000","last_undegraded":"2024-03-18T20:25:25.101120+0000","last_fullsized":"2024-03-18T20:25:25.101120+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:44:15.325153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.116256+0000","last_change":"2024-03-18T20:25:34.116256+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.116256+0000","last_undegraded":"2024-03-18T20:25:34.116256+0000","last_fullsized":"2024-03-18T20:25:34.116256+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:50.170851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100588+0000","last_change":"2024-03-18T20:25:22.252927+0000","last_active":"2024-03-18T20:25:25.100588+0000","last_peered":"2024-03-18T20:25:25.100588+0000","last_clean":"2024-03-18T20:25:25.100588+0000","last_became_active":"2024-03-18T20:25:22.252633+0000","last_became_peered":"2024-03-18T20:25:22.252633+0000","last_unstale":"2024-03-18T20:25:25.100588+0000","last_undegraded":"2024-03-18T20:25:25.100588+0000","last_fullsized":"2024-03-18T20:25:25.100588+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:32.345526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100761+0000","last_change":"2024-03-18T20:25:22.694369+0000","last_active":"2024-03-18T20:25:25.100761+0000","last_peered":"2024-03-18T20:25:25.100761+0000","last_clean":"2024-03-18T20:25:25.100761+0000","last_became_active":"2024-03-18T20:25:22.694114+0000","last_became_peered":"2024-03-18T20:25:22.694114+0000","last_unstale":"2024-03-18T20:25:25.100761+0000","last_undegraded":"2024-03-18T20:25:25.100761+0000","last_fullsized":"2024-03-18T20:25:25.100761+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:35:13.631181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.114153+0000","last_change":"2024-03-18T20:25:34.114153+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.114153+0000","last_undegraded":"2024-03-18T20:25:34.114153+0000","last_fullsized":"2024-03-18T20:25:34.114153+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:27.179302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097618+0000","last_change":"2024-03-18T20:25:22.249078+0000","last_active":"2024-03-18T20:25:25.097618+0000","last_peered":"2024-03-18T20:25:25.097618+0000","last_clean":"2024-03-18T20:25:25.097618+0000","last_became_active":"2024-03-18T20:25:22.248588+0000","last_became_peered":"2024-03-18T20:25:22.248588+0000","last_unstale":"2024-03-18T20:25:25.097618+0000","last_undegraded":"2024-03-18T20:25:25.097618+0000","last_fullsized":"2024-03-18T20:25:25.097618+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:34:42.893253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097697+0000","last_change":"2024-03-18T20:25:22.685090+0000","last_active":"2024-03-18T20:25:25.097697+0000","last_peered":"2024-03-18T20:25:25.097697+0000","last_clean":"2024-03-18T20:25:25.097697+0000","last_became_active":"2024-03-18T20:25:22.684824+0000","last_became_peered":"2024-03-18T20:25:22.684824+0000","last_unstale":"2024-03-18T20:25:25.097697+0000","last_undegraded":"2024-03-18T20:25:25.097697+0000","last_fullsized":"2024-03-18T20:25:25.097697+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:21.750701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.112308+0000","last_change":"2024-03-18T20:25:34.112308+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.112308+0000","last_undegraded":"2024-03-18T20:25:34.112308+0000","last_fullsized":"2024-03-18T20:25:34.112308+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:03.410994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.13","version":"231'2","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100964+0000","last_change":"2024-03-18T20:25:22.254668+0000","last_active":"2024-03-18T20:25:25.100964+0000","last_peered":"2024-03-18T20:25:25.100964+0000","last_clean":"2024-03-18T20:25:25.100964+0000","last_became_active":"2024-03-18T20:25:22.254437+0000","last_became_peered":"2024-03-18T20:25:22.254437+0000","last_unstale":"2024-03-18T20:25:25.100964+0000","last_undegraded":"2024-03-18T20:25:25.100964+0000","last_fullsized":"2024-03-18T20:25:25.100964+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:32:02.691847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100981+0000","last_change":"2024-03-18T20:25:22.691203+0000","last_active":"2024-03-18T20:25:25.100981+0000","last_peered":"2024-03-18T20:25:25.100981+0000","last_clean":"2024-03-18T20:25:25.100981+0000","last_became_active":"2024-03-18T20:25:22.690964+0000","last_became_peered":"2024-03-18T20:25:22.690964+0000","last_unstale":"2024-03-18T20:25:25.100981+0000","last_undegraded":"2024-03-18T20:25:25.100981+0000","last_fullsized":"2024-03-18T20:25:25.100981+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:19.068003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.113878+0000","last_change":"2024-03-18T20:25:34.113878+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.113878+0000","last_undegraded":"2024-03-18T20:25:34.113878+0000","last_fullsized":"2024-03-18T20:25:34.113878+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:02.016045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.14","version":"232'8","reported_seq":20,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:32.359025+0000","last_change":"2024-03-18T20:25:22.249429+0000","last_active":"2024-03-18T20:25:32.359025+0000","last_peered":"2024-03-18T20:25:32.359025+0000","last_clean":"2024-03-18T20:25:32.359025+0000","last_became_active":"2024-03-18T20:25:22.248842+0000","last_became_peered":"2024-03-18T20:25:22.248842+0000","last_unstale":"2024-03-18T20:25:32.359025+0000","last_undegraded":"2024-03-18T20:25:32.359025+0000","last_fullsized":"2024-03-18T20:25:32.359025+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:26.827223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2822,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101359+0000","last_change":"2024-03-18T20:25:22.696408+0000","last_active":"2024-03-18T20:25:25.101359+0000","last_peered":"2024-03-18T20:25:25.101359+0000","last_clean":"2024-03-18T20:25:25.101359+0000","last_became_active":"2024-03-18T20:25:22.696095+0000","last_became_peered":"2024-03-18T20:25:22.696095+0000","last_unstale":"2024-03-18T20:25:25.101359+0000","last_undegraded":"2024-03-18T20:25:25.101359+0000","last_fullsized":"2024-03-18T20:25:25.101359+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:19:19.377686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.119645+0000","last_change":"2024-03-18T20:25:34.119645+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.119645+0000","last_undegraded":"2024-03-18T20:25:34.119645+0000","last_fullsized":"2024-03-18T20:25:34.119645+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:50.971852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.15","version":"231'3","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100719+0000","last_change":"2024-03-18T20:25:22.251534+0000","last_active":"2024-03-18T20:25:25.100719+0000","last_peered":"2024-03-18T20:25:25.100719+0000","last_clean":"2024-03-18T20:25:25.100719+0000","last_became_active":"2024-03-18T20:25:22.251275+0000","last_became_peered":"2024-03-18T20:25:22.251275+0000","last_unstale":"2024-03-18T20:25:25.100719+0000","last_undegraded":"2024-03-18T20:25:25.100719+0000","last_fullsized":"2024-03-18T20:25:25.100719+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:38:51.553676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100607+0000","last_change":"2024-03-18T20:25:22.682861+0000","last_active":"2024-03-18T20:25:25.100607+0000","last_peered":"2024-03-18T20:25:25.100607+0000","last_clean":"2024-03-18T20:25:25.100607+0000","last_became_active":"2024-03-18T20:25:22.682610+0000","last_became_peered":"2024-03-18T20:25:22.682610+0000","last_unstale":"2024-03-18T20:25:25.100607+0000","last_undegraded":"2024-03-18T20:25:25.100607+0000","last_fullsized":"2024-03-18T20:25:25.100607+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:51:43.095770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.114296+0000","last_change":"2024-03-18T20:25:34.114296+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.114296+0000","last_undegraded":"2024-03-18T20:25:34.114296+0000","last_fullsized":"2024-03-18T20:25:34.114296+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:32:42.191508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101107+0000","last_change":"2024-03-18T20:25:22.252980+0000","last_active":"2024-03-18T20:25:25.101107+0000","last_peered":"2024-03-18T20:25:25.101107+0000","last_clean":"2024-03-18T20:25:25.101107+0000","last_became_active":"2024-03-18T20:25:22.252706+0000","last_became_peered":"2024-03-18T20:25:22.252706+0000","last_unstale":"2024-03-18T20:25:25.101107+0000","last_undegraded":"2024-03-18T20:25:25.101107+0000","last_fullsized":"2024-03-18T20:25:25.101107+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:45:13.948304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097580+0000","last_change":"2024-03-18T20:25:22.684633+0000","last_active":"2024-03-18T20:25:25.097580+0000","last_peered":"2024-03-18T20:25:25.097580+0000","last_clean":"2024-03-18T20:25:25.097580+0000","last_became_active":"2024-03-18T20:25:22.684366+0000","last_became_peered":"2024-03-18T20:25:22.684366+0000","last_unstale":"2024-03-18T20:25:25.097580+0000","last_undegraded":"2024-03-18T20:25:25.097580+0000","last_fullsized":"2024-03-18T20:25:25.097580+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:15.299142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.112662+0000","last_change":"2024-03-18T20:25:34.112662+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.112662+0000","last_undegraded":"2024-03-18T20:25:34.112662+0000","last_fullsized":"2024-03-18T20:25:34.112662+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:50.170851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101424+0000","last_change":"2024-03-18T20:25:22.251887+0000","last_active":"2024-03-18T20:25:25.101424+0000","last_peered":"2024-03-18T20:25:25.101424+0000","last_clean":"2024-03-18T20:25:25.101424+0000","last_became_active":"2024-03-18T20:25:22.251594+0000","last_became_peered":"2024-03-18T20:25:22.251594+0000","last_unstale":"2024-03-18T20:25:25.101424+0000","last_undegraded":"2024-03-18T20:25:25.101424+0000","last_fullsized":"2024-03-18T20:25:25.101424+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:48:47.954568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100806+0000","last_change":"2024-03-18T20:25:22.680660+0000","last_active":"2024-03-18T20:25:25.100806+0000","last_peered":"2024-03-18T20:25:25.100806+0000","last_clean":"2024-03-18T20:25:25.100806+0000","last_became_active":"2024-03-18T20:25:22.680484+0000","last_became_peered":"2024-03-18T20:25:22.680484+0000","last_unstale":"2024-03-18T20:25:25.100806+0000","last_undegraded":"2024-03-18T20:25:25.100806+0000","last_fullsized":"2024-03-18T20:25:25.100806+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:48:21.194536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.110675+0000","last_change":"2024-03-18T20:25:34.110675+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.110675+0000","last_undegraded":"2024-03-18T20:25:34.110675+0000","last_fullsized":"2024-03-18T20:25:34.110675+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:20.466978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097560+0000","last_change":"2024-03-18T20:25:22.244727+0000","last_active":"2024-03-18T20:25:25.097560+0000","last_peered":"2024-03-18T20:25:25.097560+0000","last_clean":"2024-03-18T20:25:25.097560+0000","last_became_active":"2024-03-18T20:25:22.244417+0000","last_became_peered":"2024-03-18T20:25:22.244417+0000","last_unstale":"2024-03-18T20:25:25.097560+0000","last_undegraded":"2024-03-18T20:25:25.097560+0000","last_fullsized":"2024-03-18T20:25:25.097560+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:10.187509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100848+0000","last_change":"2024-03-18T20:25:22.681952+0000","last_active":"2024-03-18T20:25:25.100848+0000","last_peered":"2024-03-18T20:25:25.100848+0000","last_clean":"2024-03-18T20:25:25.100848+0000","last_became_active":"2024-03-18T20:25:22.681537+0000","last_became_peered":"2024-03-18T20:25:22.681537+0000","last_unstale":"2024-03-18T20:25:25.100848+0000","last_undegraded":"2024-03-18T20:25:25.100848+0000","last_fullsized":"2024-03-18T20:25:25.100848+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:32:22.923763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.113196+0000","last_change":"2024-03-18T20:25:34.113196+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.113196+0000","last_undegraded":"2024-03-18T20:25:34.113196+0000","last_fullsized":"2024-03-18T20:25:34.113196+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:39.087869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100983+0000","last_change":"2024-03-18T20:25:22.251522+0000","last_active":"2024-03-18T20:25:25.100983+0000","last_peered":"2024-03-18T20:25:25.100983+0000","last_clean":"2024-03-18T20:25:25.100983+0000","last_became_active":"2024-03-18T20:25:22.251143+0000","last_became_peered":"2024-03-18T20:25:22.251143+0000","last_unstale":"2024-03-18T20:25:25.100983+0000","last_undegraded":"2024-03-18T20:25:25.100983+0000","last_fullsized":"2024-03-18T20:25:25.100983+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:58.960198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100935+0000","last_change":"2024-03-18T20:25:22.691744+0000","last_active":"2024-03-18T20:25:25.100935+0000","last_peered":"2024-03-18T20:25:25.100935+0000","last_clean":"2024-03-18T20:25:25.100935+0000","last_became_active":"2024-03-18T20:25:22.691491+0000","last_became_peered":"2024-03-18T20:25:22.691491+0000","last_unstale":"2024-03-18T20:25:25.100935+0000","last_undegraded":"2024-03-18T20:25:25.100935+0000","last_fullsized":"2024-03-18T20:25:25.100935+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:04:09.670541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.115119+0000","last_change":"2024-03-18T20:25:34.115119+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.115119+0000","last_undegraded":"2024-03-18T20:25:34.115119+0000","last_fullsized":"2024-03-18T20:25:34.115119+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:25.724222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1b","version":"231'1","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101037+0000","last_change":"2024-03-18T20:25:22.251076+0000","last_active":"2024-03-18T20:25:25.101037+0000","last_peered":"2024-03-18T20:25:25.101037+0000","last_clean":"2024-03-18T20:25:25.101037+0000","last_became_active":"2024-03-18T20:25:22.250741+0000","last_became_peered":"2024-03-18T20:25:22.250741+0000","last_unstale":"2024-03-18T20:25:25.101037+0000","last_undegraded":"2024-03-18T20:25:25.101037+0000","last_fullsized":"2024-03-18T20:25:25.101037+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:07.448931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100894+0000","last_change":"2024-03-18T20:25:22.680768+0000","last_active":"2024-03-18T20:25:25.100894+0000","last_peered":"2024-03-18T20:25:25.100894+0000","last_clean":"2024-03-18T20:25:25.100894+0000","last_became_active":"2024-03-18T20:25:22.680516+0000","last_became_peered":"2024-03-18T20:25:22.680516+0000","last_unstale":"2024-03-18T20:25:25.100894+0000","last_undegraded":"2024-03-18T20:25:25.100894+0000","last_fullsized":"2024-03-18T20:25:25.100894+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:19:19.377686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.111795+0000","last_change":"2024-03-18T20:25:34.111795+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.111795+0000","last_undegraded":"2024-03-18T20:25:34.111795+0000","last_fullsized":"2024-03-18T20:25:34.111795+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:45.664606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1c","version":"231'2","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100697+0000","last_change":"2024-03-18T20:25:22.251018+0000","last_active":"2024-03-18T20:25:25.100697+0000","last_peered":"2024-03-18T20:25:25.100697+0000","last_clean":"2024-03-18T20:25:25.100697+0000","last_became_active":"2024-03-18T20:25:22.250730+0000","last_became_peered":"2024-03-18T20:25:22.250730+0000","last_unstale":"2024-03-18T20:25:25.100697+0000","last_undegraded":"2024-03-18T20:25:25.100697+0000","last_fullsized":"2024-03-18T20:25:25.100697+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:07.579514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101349+0000","last_change":"2024-03-18T20:25:22.690894+0000","last_active":"2024-03-18T20:25:25.101349+0000","last_peered":"2024-03-18T20:25:25.101349+0000","last_clean":"2024-03-18T20:25:25.101349+0000","last_became_active":"2024-03-18T20:25:22.688184+0000","last_became_peered":"2024-03-18T20:25:22.688184+0000","last_unstale":"2024-03-18T20:25:25.101349+0000","last_undegraded":"2024-03-18T20:25:25.101349+0000","last_fullsized":"2024-03-18T20:25:25.101349+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:59.967502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.114019+0000","last_change":"2024-03-18T20:25:34.114019+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.114019+0000","last_undegraded":"2024-03-18T20:25:34.114019+0000","last_fullsized":"2024-03-18T20:25:34.114019+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:33.630861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1d","version":"231'2","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101002+0000","last_change":"2024-03-18T20:25:22.255583+0000","last_active":"2024-03-18T20:25:25.101002+0000","last_peered":"2024-03-18T20:25:25.101002+0000","last_clean":"2024-03-18T20:25:25.101002+0000","last_became_active":"2024-03-18T20:25:22.255243+0000","last_became_peered":"2024-03-18T20:25:22.255243+0000","last_unstale":"2024-03-18T20:25:25.101002+0000","last_undegraded":"2024-03-18T20:25:25.101002+0000","last_fullsized":"2024-03-18T20:25:25.101002+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:38.084389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099342+0000","last_change":"2024-03-18T20:25:22.691295+0000","last_active":"2024-03-18T20:25:25.099342+0000","last_peered":"2024-03-18T20:25:25.099342+0000","last_clean":"2024-03-18T20:25:25.099342+0000","last_became_active":"2024-03-18T20:25:22.690890+0000","last_became_peered":"2024-03-18T20:25:22.690890+0000","last_unstale":"2024-03-18T20:25:25.099342+0000","last_undegraded":"2024-03-18T20:25:25.099342+0000","last_fullsized":"2024-03-18T20:25:25.099342+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:23:16.320044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.117626+0000","last_change":"2024-03-18T20:25:34.117626+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.117626+0000","last_undegraded":"2024-03-18T20:25:34.117626+0000","last_fullsized":"2024-03-18T20:25:34.117626+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:19:02.788524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1e","version":"231'1","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101139+0000","last_change":"2024-03-18T20:25:22.251725+0000","last_active":"2024-03-18T20:25:25.101139+0000","last_peered":"2024-03-18T20:25:25.101139+0000","last_clean":"2024-03-18T20:25:25.101139+0000","last_became_active":"2024-03-18T20:25:22.251416+0000","last_became_peered":"2024-03-18T20:25:22.251416+0000","last_unstale":"2024-03-18T20:25:25.101139+0000","last_undegraded":"2024-03-18T20:25:25.101139+0000","last_fullsized":"2024-03-18T20:25:25.101139+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:26.827223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101012+0000","last_change":"2024-03-18T20:25:22.679158+0000","last_active":"2024-03-18T20:25:25.101012+0000","last_peered":"2024-03-18T20:25:25.101012+0000","last_clean":"2024-03-18T20:25:25.101012+0000","last_became_active":"2024-03-18T20:25:22.678889+0000","last_became_peered":"2024-03-18T20:25:22.678889+0000","last_unstale":"2024-03-18T20:25:25.101012+0000","last_undegraded":"2024-03-18T20:25:25.101012+0000","last_fullsized":"2024-03-18T20:25:25.101012+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:00.365377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.113361+0000","last_change":"2024-03-18T20:25:34.113361+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.113361+0000","last_undegraded":"2024-03-18T20:25:34.113361+0000","last_fullsized":"2024-03-18T20:25:34.113361+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:10.985415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1f","version":"231'4","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100956+0000","last_change":"2024-03-18T20:25:22.251710+0000","last_active":"2024-03-18T20:25:25.100956+0000","last_peered":"2024-03-18T20:25:25.100956+0000","last_clean":"2024-03-18T20:25:25.100956+0000","last_became_active":"2024-03-18T20:25:22.251420+0000","last_became_peered":"2024-03-18T20:25:22.251420+0000","last_unstale":"2024-03-18T20:25:25.100956+0000","last_undegraded":"2024-03-18T20:25:25.100956+0000","last_fullsized":"2024-03-18T20:25:25.100956+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:00.435897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100904+0000","last_change":"2024-03-18T20:25:22.682272+0000","last_active":"2024-03-18T20:25:25.100904+0000","last_peered":"2024-03-18T20:25:25.100904+0000","last_clean":"2024-03-18T20:25:25.100904+0000","last_became_active":"2024-03-18T20:25:22.681785+0000","last_became_peered":"2024-03-18T20:25:22.681785+0000","last_unstale":"2024-03-18T20:25:25.100904+0000","last_undegraded":"2024-03-18T20:25:25.100904+0000","last_fullsized":"2024-03-18T20:25:25.100904+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:53:46.174877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.115690+0000","last_change":"2024-03-18T20:25:34.115690+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.115690+0000","last_undegraded":"2024-03-18T20:25:34.115690+0000","last_fullsized":"2024-03-18T20:25:34.115690+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:25.724222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100917+0000","last_change":"2024-03-18T20:25:22.679227+0000","last_active":"2024-03-18T20:25:25.100917+0000","last_peered":"2024-03-18T20:25:25.100917+0000","last_clean":"2024-03-18T20:25:25.100917+0000","last_became_active":"2024-03-18T20:25:22.678827+0000","last_became_peered":"2024-03-18T20:25:22.678827+0000","last_unstale":"2024-03-18T20:25:25.100917+0000","last_undegraded":"2024-03-18T20:25:25.100917+0000","last_fullsized":"2024-03-18T20:25:25.100917+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:05.394457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100961+0000","last_change":"2024-03-18T20:25:22.679640+0000","last_active":"2024-03-18T20:25:25.100961+0000","last_peered":"2024-03-18T20:25:25.100961+0000","last_clean":"2024-03-18T20:25:25.100961+0000","last_became_active":"2024-03-18T20:25:22.679408+0000","last_became_peered":"2024-03-18T20:25:22.679408+0000","last_unstale":"2024-03-18T20:25:25.100961+0000","last_undegraded":"2024-03-18T20:25:25.100961+0000","last_fullsized":"2024-03-18T20:25:25.100961+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:24:33.293594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100873+0000","last_change":"2024-03-18T20:25:22.685981+0000","last_active":"2024-03-18T20:25:25.100873+0000","last_peered":"2024-03-18T20:25:25.100873+0000","last_clean":"2024-03-18T20:25:25.100873+0000","last_became_active":"2024-03-18T20:25:22.685683+0000","last_became_peered":"2024-03-18T20:25:22.685683+0000","last_unstale":"2024-03-18T20:25:25.100873+0000","last_undegraded":"2024-03-18T20:25:25.100873+0000","last_fullsized":"2024-03-18T20:25:25.100873+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:36:57.458670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100679+0000","last_change":"2024-03-18T20:25:22.680503+0000","last_active":"2024-03-18T20:25:25.100679+0000","last_peered":"2024-03-18T20:25:25.100679+0000","last_clean":"2024-03-18T20:25:25.100679+0000","last_became_active":"2024-03-18T20:25:22.680258+0000","last_became_peered":"2024-03-18T20:25:22.680258+0000","last_unstale":"2024-03-18T20:25:25.100679+0000","last_undegraded":"2024-03-18T20:25:25.100679+0000","last_fullsized":"2024-03-18T20:25:25.100679+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:49.478525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101401+0000","last_change":"2024-03-18T20:25:22.688098+0000","last_active":"2024-03-18T20:25:25.101401+0000","last_peered":"2024-03-18T20:25:25.101401+0000","last_clean":"2024-03-18T20:25:25.101401+0000","last_became_active":"2024-03-18T20:25:22.687830+0000","last_became_peered":"2024-03-18T20:25:22.687830+0000","last_unstale":"2024-03-18T20:25:25.101401+0000","last_undegraded":"2024-03-18T20:25:25.101401+0000","last_fullsized":"2024-03-18T20:25:25.101401+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:08.456234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097677+0000","last_change":"2024-03-18T20:25:22.689262+0000","last_active":"2024-03-18T20:25:25.097677+0000","last_peered":"2024-03-18T20:25:25.097677+0000","last_clean":"2024-03-18T20:25:25.097677+0000","last_became_active":"2024-03-18T20:25:22.688913+0000","last_became_peered":"2024-03-18T20:25:22.688913+0000","last_unstale":"2024-03-18T20:25:25.097677+0000","last_undegraded":"2024-03-18T20:25:25.097677+0000","last_fullsized":"2024-03-18T20:25:25.097677+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:15:50.135885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101124+0000","last_change":"2024-03-18T20:25:22.687342+0000","last_active":"2024-03-18T20:25:25.101124+0000","last_peered":"2024-03-18T20:25:25.101124+0000","last_clean":"2024-03-18T20:25:25.101124+0000","last_became_active":"2024-03-18T20:25:22.687092+0000","last_became_peered":"2024-03-18T20:25:22.687092+0000","last_unstale":"2024-03-18T20:25:25.101124+0000","last_undegraded":"2024-03-18T20:25:25.101124+0000","last_fullsized":"2024-03-18T20:25:25.101124+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:00:33.383893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100833+0000","last_change":"2024-03-18T20:25:22.683777+0000","last_active":"2024-03-18T20:25:25.100833+0000","last_peered":"2024-03-18T20:25:25.100833+0000","last_clean":"2024-03-18T20:25:25.100833+0000","last_became_active":"2024-03-18T20:25:22.683484+0000","last_became_peered":"2024-03-18T20:25:22.683484+0000","last_unstale":"2024-03-18T20:25:25.100833+0000","last_undegraded":"2024-03-18T20:25:25.100833+0000","last_fullsized":"2024-03-18T20:25:25.100833+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:03:45.332668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101271+0000","last_change":"2024-03-18T20:25:22.684030+0000","last_active":"2024-03-18T20:25:25.101271+0000","last_peered":"2024-03-18T20:25:25.101271+0000","last_clean":"2024-03-18T20:25:25.101271+0000","last_became_active":"2024-03-18T20:25:22.683767+0000","last_became_peered":"2024-03-18T20:25:22.683767+0000","last_unstale":"2024-03-18T20:25:25.101271+0000","last_undegraded":"2024-03-18T20:25:25.101271+0000","last_fullsized":"2024-03-18T20:25:25.101271+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:30.204046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100925+0000","last_change":"2024-03-18T20:25:22.680418+0000","last_active":"2024-03-18T20:25:25.100925+0000","last_peered":"2024-03-18T20:25:25.100925+0000","last_clean":"2024-03-18T20:25:25.100925+0000","last_became_active":"2024-03-18T20:25:22.680149+0000","last_became_peered":"2024-03-18T20:25:22.680149+0000","last_unstale":"2024-03-18T20:25:25.100925+0000","last_undegraded":"2024-03-18T20:25:25.100925+0000","last_fullsized":"2024-03-18T20:25:25.100925+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:51:14.804124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099330+0000","last_change":"2024-03-18T20:25:22.691577+0000","last_active":"2024-03-18T20:25:25.099330+0000","last_peered":"2024-03-18T20:25:25.099330+0000","last_clean":"2024-03-18T20:25:25.099330+0000","last_became_active":"2024-03-18T20:25:22.691250+0000","last_became_peered":"2024-03-18T20:25:22.691250+0000","last_unstale":"2024-03-18T20:25:25.099330+0000","last_undegraded":"2024-03-18T20:25:25.099330+0000","last_fullsized":"2024-03-18T20:25:25.099330+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:55:21.946848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099355+0000","last_change":"2024-03-18T20:25:22.693670+0000","last_active":"2024-03-18T20:25:25.099355+0000","last_peered":"2024-03-18T20:25:25.099355+0000","last_clean":"2024-03-18T20:25:25.099355+0000","last_became_active":"2024-03-18T20:25:22.693178+0000","last_became_peered":"2024-03-18T20:25:22.693178+0000","last_unstale":"2024-03-18T20:25:25.099355+0000","last_undegraded":"2024-03-18T20:25:25.099355+0000","last_fullsized":"2024-03-18T20:25:25.099355+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:13.992732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099388+0000","last_change":"2024-03-18T20:25:22.693769+0000","last_active":"2024-03-18T20:25:25.099388+0000","last_peered":"2024-03-18T20:25:25.099388+0000","last_clean":"2024-03-18T20:25:25.099388+0000","last_became_active":"2024-03-18T20:25:22.691936+0000","last_became_peered":"2024-03-18T20:25:22.691936+0000","last_unstale":"2024-03-18T20:25:25.099388+0000","last_undegraded":"2024-03-18T20:25:25.099388+0000","last_fullsized":"2024-03-18T20:25:25.099388+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:29:49.401587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099414+0000","last_change":"2024-03-18T20:25:22.692857+0000","last_active":"2024-03-18T20:25:25.099414+0000","last_peered":"2024-03-18T20:25:25.099414+0000","last_clean":"2024-03-18T20:25:25.099414+0000","last_became_active":"2024-03-18T20:25:22.692615+0000","last_became_peered":"2024-03-18T20:25:22.692615+0000","last_unstale":"2024-03-18T20:25:25.099414+0000","last_undegraded":"2024-03-18T20:25:25.099414+0000","last_fullsized":"2024-03-18T20:25:25.099414+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:21:45.938034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100738+0000","last_change":"2024-03-18T20:25:22.683388+0000","last_active":"2024-03-18T20:25:25.100738+0000","last_peered":"2024-03-18T20:25:25.100738+0000","last_clean":"2024-03-18T20:25:25.100738+0000","last_became_active":"2024-03-18T20:25:22.683137+0000","last_became_peered":"2024-03-18T20:25:22.683137+0000","last_unstale":"2024-03-18T20:25:25.100738+0000","last_undegraded":"2024-03-18T20:25:25.100738+0000","last_fullsized":"2024-03-18T20:25:25.100738+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:02.506299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100813+0000","last_change":"2024-03-18T20:25:22.696148+0000","last_active":"2024-03-18T20:25:25.100813+0000","last_peered":"2024-03-18T20:25:25.100813+0000","last_clean":"2024-03-18T20:25:25.100813+0000","last_became_active":"2024-03-18T20:25:22.695928+0000","last_became_peered":"2024-03-18T20:25:22.695928+0000","last_unstale":"2024-03-18T20:25:25.100813+0000","last_undegraded":"2024-03-18T20:25:25.100813+0000","last_fullsized":"2024-03-18T20:25:25.100813+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:57.565171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097534+0000","last_change":"2024-03-18T20:25:22.681389+0000","last_active":"2024-03-18T20:25:25.097534+0000","last_peered":"2024-03-18T20:25:25.097534+0000","last_clean":"2024-03-18T20:25:25.097534+0000","last_became_active":"2024-03-18T20:25:22.681146+0000","last_became_peered":"2024-03-18T20:25:22.681146+0000","last_unstale":"2024-03-18T20:25:25.097534+0000","last_undegraded":"2024-03-18T20:25:25.097534+0000","last_fullsized":"2024-03-18T20:25:25.097534+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:05:57.372944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100832+0000","last_change":"2024-03-18T20:25:22.692497+0000","last_active":"2024-03-18T20:25:25.100832+0000","last_peered":"2024-03-18T20:25:25.100832+0000","last_clean":"2024-03-18T20:25:25.100832+0000","last_became_active":"2024-03-18T20:25:22.692192+0000","last_became_peered":"2024-03-18T20:25:22.692192+0000","last_unstale":"2024-03-18T20:25:25.100832+0000","last_undegraded":"2024-03-18T20:25:25.100832+0000","last_fullsized":"2024-03-18T20:25:25.100832+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:03:47.775258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099320+0000","last_change":"2024-03-18T20:25:22.690898+0000","last_active":"2024-03-18T20:25:25.099320+0000","last_peered":"2024-03-18T20:25:25.099320+0000","last_clean":"2024-03-18T20:25:25.099320+0000","last_became_active":"2024-03-18T20:25:22.690461+0000","last_became_peered":"2024-03-18T20:25:22.690461+0000","last_unstale":"2024-03-18T20:25:25.099320+0000","last_undegraded":"2024-03-18T20:25:25.099320+0000","last_fullsized":"2024-03-18T20:25:25.099320+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:15.208122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101023+0000","last_change":"2024-03-18T20:25:22.685172+0000","last_active":"2024-03-18T20:25:25.101023+0000","last_peered":"2024-03-18T20:25:25.101023+0000","last_clean":"2024-03-18T20:25:25.101023+0000","last_became_active":"2024-03-18T20:25:22.684855+0000","last_became_peered":"2024-03-18T20:25:22.684855+0000","last_unstale":"2024-03-18T20:25:25.101023+0000","last_undegraded":"2024-03-18T20:25:25.101023+0000","last_fullsized":"2024-03-18T20:25:25.101023+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:00.030804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101390+0000","last_change":"2024-03-18T20:25:22.693640+0000","last_active":"2024-03-18T20:25:25.101390+0000","last_peered":"2024-03-18T20:25:25.101390+0000","last_clean":"2024-03-18T20:25:25.101390+0000","last_became_active":"2024-03-18T20:25:22.693403+0000","last_became_peered":"2024-03-18T20:25:22.693403+0000","last_unstale":"2024-03-18T20:25:25.101390+0000","last_undegraded":"2024-03-18T20:25:25.101390+0000","last_fullsized":"2024-03-18T20:25:25.101390+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:58.151146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097657+0000","last_change":"2024-03-18T20:25:22.693546+0000","last_active":"2024-03-18T20:25:25.097657+0000","last_peered":"2024-03-18T20:25:25.097657+0000","last_clean":"2024-03-18T20:25:25.097657+0000","last_became_active":"2024-03-18T20:25:22.693267+0000","last_became_peered":"2024-03-18T20:25:22.693267+0000","last_unstale":"2024-03-18T20:25:25.097657+0000","last_undegraded":"2024-03-18T20:25:25.097657+0000","last_fullsized":"2024-03-18T20:25:25.097657+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:35.486064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099426+0000","last_change":"2024-03-18T20:25:22.693152+0000","last_active":"2024-03-18T20:25:25.099426+0000","last_peered":"2024-03-18T20:25:25.099426+0000","last_clean":"2024-03-18T20:25:25.099426+0000","last_became_active":"2024-03-18T20:25:22.686521+0000","last_became_peered":"2024-03-18T20:25:22.686521+0000","last_unstale":"2024-03-18T20:25:25.099426+0000","last_undegraded":"2024-03-18T20:25:25.099426+0000","last_fullsized":"2024-03-18T20:25:25.099426+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:40:18.302530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100797+0000","last_change":"2024-03-18T20:25:22.681784+0000","last_active":"2024-03-18T20:25:25.100797+0000","last_peered":"2024-03-18T20:25:25.100797+0000","last_clean":"2024-03-18T20:25:25.100797+0000","last_became_active":"2024-03-18T20:25:22.681198+0000","last_became_peered":"2024-03-18T20:25:22.681198+0000","last_unstale":"2024-03-18T20:25:25.100797+0000","last_undegraded":"2024-03-18T20:25:25.100797+0000","last_fullsized":"2024-03-18T20:25:25.100797+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:35.307035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099272+0000","last_change":"2024-03-18T20:25:22.685766+0000","last_active":"2024-03-18T20:25:25.099272+0000","last_peered":"2024-03-18T20:25:25.099272+0000","last_clean":"2024-03-18T20:25:25.099272+0000","last_became_active":"2024-03-18T20:25:22.683799+0000","last_became_peered":"2024-03-18T20:25:22.683799+0000","last_unstale":"2024-03-18T20:25:25.099272+0000","last_undegraded":"2024-03-18T20:25:25.099272+0000","last_fullsized":"2024-03-18T20:25:25.099272+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:42:25.581359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101317+0000","last_change":"2024-03-18T20:25:22.694269+0000","last_active":"2024-03-18T20:25:25.101317+0000","last_peered":"2024-03-18T20:25:25.101317+0000","last_clean":"2024-03-18T20:25:25.101317+0000","last_became_active":"2024-03-18T20:25:22.694014+0000","last_became_peered":"2024-03-18T20:25:22.694014+0000","last_unstale":"2024-03-18T20:25:25.101317+0000","last_undegraded":"2024-03-18T20:25:25.101317+0000","last_fullsized":"2024-03-18T20:25:25.101317+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:43:52.409254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101002+0000","last_change":"2024-03-18T20:25:22.681736+0000","last_active":"2024-03-18T20:25:25.101002+0000","last_peered":"2024-03-18T20:25:25.101002+0000","last_clean":"2024-03-18T20:25:25.101002+0000","last_became_active":"2024-03-18T20:25:22.679120+0000","last_became_peered":"2024-03-18T20:25:22.679120+0000","last_unstale":"2024-03-18T20:25:25.101002+0000","last_undegraded":"2024-03-18T20:25:25.101002+0000","last_fullsized":"2024-03-18T20:25:25.101002+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:00.365377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101337+0000","last_change":"2024-03-18T20:25:22.695525+0000","last_active":"2024-03-18T20:25:25.101337+0000","last_peered":"2024-03-18T20:25:25.101337+0000","last_clean":"2024-03-18T20:25:25.101337+0000","last_became_active":"2024-03-18T20:25:22.694079+0000","last_became_peered":"2024-03-18T20:25:22.694079+0000","last_unstale":"2024-03-18T20:25:25.101337+0000","last_undegraded":"2024-03-18T20:25:25.101337+0000","last_fullsized":"2024-03-18T20:25:25.101337+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:45:14.955608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101048+0000","last_change":"2024-03-18T20:25:22.683021+0000","last_active":"2024-03-18T20:25:25.101048+0000","last_peered":"2024-03-18T20:25:25.101048+0000","last_clean":"2024-03-18T20:25:25.101048+0000","last_became_active":"2024-03-18T20:25:22.682722+0000","last_became_peered":"2024-03-18T20:25:22.682722+0000","last_unstale":"2024-03-18T20:25:25.101048+0000","last_undegraded":"2024-03-18T20:25:25.101048+0000","last_fullsized":"2024-03-18T20:25:25.101048+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:27.207710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097638+0000","last_change":"2024-03-18T20:25:22.679426+0000","last_active":"2024-03-18T20:25:25.097638+0000","last_peered":"2024-03-18T20:25:25.097638+0000","last_clean":"2024-03-18T20:25:25.097638+0000","last_became_active":"2024-03-18T20:25:22.679159+0000","last_became_peered":"2024-03-18T20:25:22.679159+0000","last_unstale":"2024-03-18T20:25:25.097638+0000","last_undegraded":"2024-03-18T20:25:25.097638+0000","last_fullsized":"2024-03-18T20:25:25.097638+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:17:59.556803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101022+0000","last_change":"2024-03-18T20:25:22.682750+0000","last_active":"2024-03-18T20:25:25.101022+0000","last_peered":"2024-03-18T20:25:25.101022+0000","last_clean":"2024-03-18T20:25:25.101022+0000","last_became_active":"2024-03-18T20:25:22.682497+0000","last_became_peered":"2024-03-18T20:25:22.682497+0000","last_unstale":"2024-03-18T20:25:25.101022+0000","last_undegraded":"2024-03-18T20:25:25.101022+0000","last_fullsized":"2024-03-18T20:25:25.101022+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:33.092503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101174+0000","last_change":"2024-03-18T20:25:22.696207+0000","last_active":"2024-03-18T20:25:25.101174+0000","last_peered":"2024-03-18T20:25:25.101174+0000","last_clean":"2024-03-18T20:25:25.101174+0000","last_became_active":"2024-03-18T20:25:22.695943+0000","last_became_peered":"2024-03-18T20:25:22.695943+0000","last_unstale":"2024-03-18T20:25:25.101174+0000","last_undegraded":"2024-03-18T20:25:25.101174+0000","last_fullsized":"2024-03-18T20:25:25.101174+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:54:33.571448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3620,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575983,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52616,"kb_used_data":3452,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319224,"statfs":{"total":96636764160,"available":96582885376,"internally_reserved":0,"allocated":3534848,"data_stored":2093396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80700000000000005,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.48499999999999999,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.81899999999999995,"5min":1.093,"15min":1.093},"last":0.497}]},{"osd":1,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61799999999999999,"15min":0.61699999999999999},"min":{"1min":0.47599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.82999999999999996,"5min":0.99399999999999999,"15min":1.4299999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.60999999999999999,"15min":0.61899999999999999},"min":{"1min":0.47099999999999997,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.77500000000000002,"5min":1.0449999999999999,"15min":1.2629999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.46400000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84299999999999997,"5min":1.216,"15min":1.216},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.62,"15min":0.625},"min":{"1min":0.48399999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.81799999999999995,"5min":1.177,"15min":1.2490000000000001},"last":0.66000000000000003}]},{"osd":3,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.42599999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84699999999999998,"5min":0.93100000000000005,"15min":1.4450000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.5,"5min":0.44900000000000001,"15min":0.38600000000000001},"max":{"1min":0.84399999999999997,"5min":1.028,"15min":1.5029999999999999},"last":0.52200000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.47499999999999998,"15min":0.47699999999999998},"min":{"1min":0.33700000000000002,"5min":0.314,"15min":0.29899999999999999},"max":{"1min":0.68400000000000005,"5min":0.79100000000000004,"15min":0.86099999999999999},"last":0.439},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.35199999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71599999999999997,"5min":0.76600000000000001,"15min":0.80200000000000005},"last":0.53600000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51500000000000001,"15min":0.51600000000000001},"min":{"1min":0.34899999999999998,"5min":0.34399999999999997,"15min":0.316},"max":{"1min":0.68999999999999995,"5min":0.86599999999999999,"15min":0.89400000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.40699999999999997,"5min":0.377,"15min":0.36799999999999999},"max":{"1min":0.75600000000000001,"5min":0.81599999999999995,"15min":1.042},"last":0.64400000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56399999999999995,"15min":0.57699999999999996},"min":{"1min":0.39900000000000002,"5min":0.36199999999999999,"15min":0.34999999999999998},"max":{"1min":0.71099999999999997,"5min":0.85599999999999998,"15min":1.129},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.371,"5min":0.36299999999999999,"15min":0.34000000000000002},"max":{"1min":0.82899999999999996,"5min":0.85299999999999998,"15min":1.0609999999999999},"last":0.63600000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575983,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42180,"kb_used_data":2872,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329660,"statfs":{"total":96636764160,"available":96593571840,"internally_reserved":0,"allocated":2940928,"data_stored":1503540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57599999999999996,"15min":0.60099999999999998},"min":{"1min":0.41399999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72199999999999998,"5min":0.80900000000000005,"15min":1.1519999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59099999999999997,"15min":0.60799999999999998},"min":{"1min":0.33700000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.74299999999999999,"5min":0.81499999999999995,"15min":0.83299999999999996},"last":0.55000000000000004}]},{"osd":1,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57899999999999996,"15min":0.60199999999999998},"min":{"1min":0.39400000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.79100000000000004,"5min":0.79800000000000004,"15min":1.244},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58499999999999996,"15min":0.60399999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33200000000000002},"max":{"1min":0.77000000000000002,"5min":0.89200000000000002,"15min":1.286},"last":0.61299999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60999999999999999,"15min":0.63},"min":{"1min":0.47899999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71199999999999997,"5min":0.92000000000000004,"15min":1.276},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.59299999999999997,"15min":0.61099999999999999},"min":{"1min":0.434,"5min":0.40100000000000002,"15min":0.378},"max":{"1min":0.77500000000000002,"5min":0.95399999999999996,"15min":1.2669999999999999},"last":0.748}]},{"osd":3,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59699999999999998,"15min":0.622},"min":{"1min":0.38,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70699999999999996,"5min":0.83199999999999996,"15min":1.02},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60599999999999998,"15min":0.61899999999999999},"min":{"1min":0.45200000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82399999999999995,"5min":1.0049999999999999,"15min":1.337},"last":0.63800000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.48099999999999998,"15min":0.48299999999999998},"min":{"1min":0.308,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.61699999999999999,"5min":0.85799999999999998,"15min":0.88300000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.50900000000000001,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72099999999999997,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.65500000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.5,"15min":0.504},"min":{"1min":0.375,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.84799999999999998},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39000000000000001,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.73399999999999999,"5min":0.92600000000000005,"15min":0.94199999999999995},"last":0.72299999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40600000000000003,"5min":0.35799999999999998,"15min":0.34499999999999997},"max":{"1min":0.74299999999999999,"5min":0.95599999999999996,"15min":1.0429999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.44800000000000001,"5min":0.36599999999999999,"15min":0.34599999999999997},"max":{"1min":0.83899999999999997,"5min":1.028,"15min":1.028},"last":0.76300000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575983,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47924,"kb_used_data":2856,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323916,"statfs":{"total":96636764160,"available":96587689984,"internally_reserved":0,"allocated":2924544,"data_stored":1500136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.40400000000000003,"15min":0.39600000000000002},"min":{"1min":0.25600000000000001,"5min":0.246,"15min":0.23999999999999999},"max":{"1min":0.57999999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.41899999999999998,"15min":0.42999999999999999},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.58999999999999997,"5min":0.69699999999999995,"15min":0.72299999999999998},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.44,"15min":0.439},"min":{"1min":0.309,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61199999999999999,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48099999999999998,"15min":0.47799999999999998},"min":{"1min":0.35099999999999998,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.70099999999999996,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.377}]},{"osd":3,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.495,"15min":0.495},"min":{"1min":0.36199999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.64300000000000002,"5min":0.75900000000000001,"15min":0.76100000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.44800000000000001,"15min":0.442},"min":{"1min":0.32800000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68200000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.36199999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.68300000000000005,"15min":0.67300000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86799999999999999,"5min":1.0680000000000001,"15min":1.2909999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.68100000000000005,"15min":0.68000000000000005},"min":{"1min":0.50900000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.84499999999999997,"5min":0.92500000000000004,"15min":1.746},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.68100000000000005,"15min":0.67000000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.6910000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.71999999999999997,"15min":0.71399999999999997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.46899999999999997},"max":{"1min":0.89400000000000002,"5min":1.109,"15min":1.2549999999999999},"last":0.61699999999999999}]},{"osd":6,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.69699999999999995,"15min":0.68500000000000005},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.437},"max":{"1min":0.97399999999999998,"5min":0.98899999999999999,"15min":1.637},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71899999999999997,"15min":0.71199999999999997},"min":{"1min":0.49099999999999999,"5min":0.46999999999999997,"15min":0.46100000000000002},"max":{"1min":0.89500000000000002,"5min":1.014,"15min":1.722},"last":0.58799999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.70699999999999996,"15min":0.71299999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79800000000000004,"5min":1.02,"15min":1.2729999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.498,"5min":0.495,"15min":0.42799999999999999},"max":{"1min":0.90300000000000002,"5min":1.0329999999999999,"15min":1.3380000000000001},"last":0.63}]}]},{"osd":0,"up_from":13,"seq":55834575984,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30460,"kb_used_data":3440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341380,"statfs":{"total":96636764160,"available":96605573120,"internally_reserved":0,"allocated":3522560,"data_stored":2090594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.42399999999999999,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.374},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.42099999999999999,"15min":0.42799999999999999},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.46600000000000003,"5min":0.64700000000000002,"15min":1.1850000000000001},"last":0.28999999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.42599999999999999,"15min":0.435},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.47099999999999997,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.438,"15min":0.45300000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29799999999999999},"max":{"1min":0.59199999999999997,"5min":0.64300000000000002,"15min":0.70999999999999996},"last":0.33000000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.26100000000000001},"max":{"1min":0.45800000000000002,"5min":0.60099999999999998,"15min":0.628},"last":0.442},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48599999999999999,"15min":0.496},"min":{"1min":0.35499999999999998,"5min":0.35199999999999998,"15min":0.32500000000000001},"max":{"1min":0.52300000000000002,"5min":0.73999999999999999,"15min":0.91500000000000004},"last":0.34300000000000003}]},{"osd":4,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64300000000000002,"15min":0.65500000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81299999999999994,"5min":0.91500000000000004,"15min":1.0029999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.66200000000000003,"15min":0.66300000000000003},"min":{"1min":0.47499999999999998,"5min":0.47099999999999997,"15min":0.39700000000000002},"max":{"1min":0.78100000000000003,"5min":0.95299999999999996,"15min":1.0469999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.67800000000000005,"15min":0.67100000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89700000000000002,"5min":0.97099999999999997,"15min":1.0129999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91100000000000003,"5min":0.92900000000000005,"15min":1.518},"last":0.61899999999999999}]},{"osd":6,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.47399999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.78600000000000003,"5min":0.95899999999999996,"15min":1.1279999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71199999999999997,"15min":0.70899999999999996},"min":{"1min":0.498,"5min":0.48999999999999999,"15min":0.47899999999999998},"max":{"1min":0.94899999999999995,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.65300000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85099999999999998,"5min":0.96399999999999997,"15min":1.464},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.69899999999999995,"15min":0.70399999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.47599999999999998},"max":{"1min":0.75800000000000001,"5min":1.042,"15min":1.2350000000000001},"last":0.63300000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575984,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47928,"kb_used_data":2860,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323912,"statfs":{"total":96636764160,"available":96587685888,"internally_reserved":0,"allocated":2928640,"data_stored":1500226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36599999999999999,"15min":0.378},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.496,"5min":0.54800000000000004,"15min":0.59399999999999997},"last":0.318},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.47399999999999998,"15min":0.48699999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.63600000000000001,"5min":0.71499999999999997,"15min":0.753},"last":0.34599999999999997}]},{"osd":1,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46899999999999997,"15min":0.48199999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64300000000000002,"5min":0.72599999999999998,"15min":0.77100000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40899999999999997,"15min":0.41599999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.53800000000000003,"5min":0.59299999999999997,"15min":0.71099999999999997},"last":0.30399999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.433,"15min":0.436},"min":{"1min":0.28899999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.497,"15min":0.51700000000000002},"min":{"1min":0.33800000000000002,"5min":0.32200000000000001,"15min":0.313},"max":{"1min":0.68100000000000005,"5min":0.752,"15min":0.752},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.66300000000000003,"15min":0.65800000000000003},"min":{"1min":0.503,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":0.82499999999999996,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67500000000000004,"15min":0.66800000000000004},"min":{"1min":0.497,"5min":0.443,"15min":0.443},"max":{"1min":0.91600000000000004,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.67900000000000005,"15min":0.68500000000000005},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.93999999999999995,"5min":1.028,"15min":1.028},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67300000000000004,"15min":0.68000000000000005},"min":{"1min":0.47299999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.88300000000000001,"5min":0.95599999999999996,"15min":1.0629999999999999},"last":0.50600000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68000000000000005,"15min":0.69099999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86099999999999999,"5min":1.014,"15min":1.2050000000000001},"last":0.432},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.72499999999999998,"15min":0.73199999999999998},"min":{"1min":0.53800000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.97199999999999998,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.63200000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68899999999999995,"15min":0.70199999999999996},"min":{"1min":0.51500000000000001,"5min":0.442,"15min":0.42199999999999999},"max":{"1min":0.92900000000000005,"5min":0.99399999999999999,"15min":1.0469999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69899999999999995,"15min":0.70699999999999996},"min":{"1min":0.45500000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.93999999999999995,"5min":1.371,"15min":1.371},"last":0.59899999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575984,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52032,"kb_used_data":2868,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319808,"statfs":{"total":96636764160,"available":96583483392,"internally_reserved":0,"allocated":2936832,"data_stored":1500262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.38800000000000001,"15min":0.39900000000000002},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.53700000000000003,"5min":0.66500000000000004,"15min":0.98099999999999998},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.40999999999999998,"15min":0.41499999999999998},"min":{"1min":0.27700000000000002,"5min":0.25800000000000001,"15min":0.24199999999999999},"max":{"1min":0.434,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.29399999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39000000000000001,"15min":0.40999999999999998},"min":{"1min":0.30099999999999999,"5min":0.26100000000000001,"15min":0.254},"max":{"1min":0.45900000000000002,"5min":0.63200000000000001,"15min":0.64000000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.46100000000000002,"15min":0.46600000000000003},"min":{"1min":0.36699999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57499999999999996,"5min":0.65500000000000003,"15min":0.73099999999999998},"last":0.35399999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.45500000000000002,"15min":0.46600000000000003},"min":{"1min":0.34999999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53100000000000003,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.48199999999999998,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.51700000000000002,"5min":0.67000000000000004,"15min":0.73099999999999998},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.63600000000000001,"15min":0.65200000000000002},"min":{"1min":0.48799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81899999999999995,"5min":0.91700000000000004,"15min":1.097},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65200000000000002,"15min":0.67500000000000004},"min":{"1min":0.435,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78800000000000003,"5min":0.99299999999999999,"15min":1.0569999999999999},"last":0.56599999999999995}]},{"osd":5,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68400000000000005,"15min":0.69699999999999995},"min":{"1min":0.48999999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92100000000000004,"5min":0.995,"15min":1.1599999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.72499999999999998,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.82099999999999995,"5min":1.171,"15min":1.171},"last":0.54000000000000004}]},{"osd":6,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68500000000000005,"15min":0.70799999999999996},"min":{"1min":0.48099999999999998,"5min":0.436,"15min":0.42499999999999999},"max":{"1min":0.90500000000000003,"5min":1.012,"15min":1.194},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.71799999999999997,"15min":0.73299999999999998},"min":{"1min":0.53300000000000003,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":0.88200000000000001,"5min":1.0409999999999999,"15min":1.2410000000000001},"last":0.68000000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.70099999999999996,"15min":0.71899999999999997},"min":{"1min":0.49099999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82799999999999996,"5min":1.079,"15min":1.079},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68100000000000005,"15min":0.69999999999999996},"min":{"1min":0.44500000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83799999999999997,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.60899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575984,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64312,"kb_used_data":2860,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307528,"statfs":{"total":96636764160,"available":96570908672,"internally_reserved":0,"allocated":2928640,"data_stored":1500718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.57899999999999996,"15min":0.59299999999999997},"min":{"1min":0.34300000000000003,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65300000000000002,"5min":0.77500000000000002,"15min":0.81799999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57799999999999996,"15min":0.59799999999999998},"min":{"1min":0.40300000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69999999999999996,"5min":0.79600000000000004,"15min":0.84199999999999997},"last":0.58599999999999997}]},{"osd":1,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59999999999999998,"15min":0.61399999999999999},"min":{"1min":0.438,"5min":0.318,"15min":0.318},"max":{"1min":0.73199999999999998,"5min":0.81899999999999995,"15min":0.88300000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56999999999999995,"15min":0.58499999999999996},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.71199999999999997,"5min":0.79200000000000004,"15min":0.96499999999999997},"last":0.71199999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58299999999999996,"15min":0.59899999999999998},"min":{"1min":0.439,"5min":0.39500000000000002,"15min":0.39200000000000002},"max":{"1min":0.68600000000000005,"5min":0.80500000000000005,"15min":0.82599999999999996},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61099999999999999,"15min":0.61799999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.38500000000000001},"max":{"1min":0.81399999999999995,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.40799999999999997,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.86199999999999999},"last":0.52600000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.498,"15min":0.504},"min":{"1min":0.35199999999999998,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.85499999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.442}]},{"osd":6,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.499,"15min":0.505},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.46700000000000003}]},{"osd":7,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54100000000000004,"15min":0.54800000000000004},"min":{"1min":0.435,"5min":0.38,"15min":0.35699999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.57199999999999995,"15min":0.57999999999999996},"min":{"1min":0.41899999999999998,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.65100000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575983,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47932,"kb_used_data":2864,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323908,"statfs":{"total":96636764160,"available":96587681792,"internally_reserved":0,"allocated":2932736,"data_stored":1500192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42499999999999999,"5min":0.33900000000000002,"15min":0.31900000000000001},"max":{"1min":0.72699999999999998,"5min":0.85999999999999999,"15min":0.96499999999999997},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57799999999999996,"15min":0.60399999999999998},"min":{"1min":0.432,"5min":0.37,"15min":0.33100000000000002},"max":{"1min":0.69799999999999995,"5min":0.82599999999999996,"15min":1.3100000000000001},"last":0.55500000000000005}]},{"osd":1,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58499999999999996,"15min":0.60199999999999998},"min":{"1min":0.36899999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.74399999999999999,"5min":0.79600000000000004,"15min":1.056},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58399999999999996,"15min":0.60199999999999998},"min":{"1min":0.46600000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.63100000000000001,"5min":0.89800000000000002,"15min":1.018},"last":0.54000000000000004}]},{"osd":2,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.38700000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.84299999999999997,"15min":0.99299999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59399999999999997,"15min":0.60899999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.67500000000000004,"5min":0.875,"15min":1.1120000000000001},"last":0.52300000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.42799999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.78100000000000003,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63,"15min":0.64600000000000002},"min":{"1min":0.443,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69899999999999995,"5min":0.82399999999999995,"15min":1.161},"last":0.51000000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.47899999999999998,"15min":0.47499999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.77500000000000002},"last":0.432},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51200000000000001,"15min":0.50600000000000001},"min":{"1min":0.36199999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.89700000000000002},"last":0.39300000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.503,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.34000000000000002,"15min":0.32800000000000001},"max":{"1min":0.69999999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.40300000000000002,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.86599999999999999},"last":0.36699999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.40799999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.94499999999999995},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.45300000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74099999999999999,"5min":0.85999999999999999,"15min":0.91200000000000003},"last":0.41699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:25:39.736 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:25:39.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.884+0000 7f88a4e48700 1 Processor -- start 2024-03-18T20:25:39.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.884+0000 7f88a4e48700 1 -- start start 2024-03-18T20:25:39.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.885+0000 7f88a4e48700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88a006a750 0x7f88a006ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.885+0000 7f88a4e48700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a00acfb0 con 0x7f88a015d6b0 2024-03-18T20:25:39.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.885+0000 7f88a4e48700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a0161e10 con 0x7f88a00ad790 2024-03-18T20:25:39.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.885+0000 7f88a4e48700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88a00af7a0 con 0x7f88a006a750 2024-03-18T20:25:39.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.885+0000 7f889e59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88a006a750 0x7f88a006ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.885+0000 7f889dd9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f88a015d6b0 0x7f88a0160ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51512/0 (socket says 172.21.15.1:51512) 2024-03-18T20:25:39.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.885+0000 7f889dd9b700 1 -- 172.21.15.1:0/187470358 learned_addr learned my addr 172.21.15.1:0/187470358 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:39.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.886+0000 7f889e59c700 1 -- 172.21.15.1:0/187470358 >> v1:172.21.15.165:6789/0 conn(0x7f88a00ad790 legacy=0x7f88a0161700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.886+0000 7f889e59c700 1 -- 172.21.15.1:0/187470358 >> v1:172.21.15.1:6789/0 conn(0x7f88a015d6b0 legacy=0x7f88a0160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.886+0000 7f889e59c700 1 -- 172.21.15.1:0/187470358 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a006b070 con 0x7f88a006a750 2024-03-18T20:25:39.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.886+0000 7f889d59a700 1 -- 172.21.15.1:0/187470358 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600204616 0 0) 0x7f88a00acfb0 con 0x7f88a015d6b0 2024-03-18T20:25:39.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.886+0000 7f889e59c700 1 --2- 172.21.15.1:0/187470358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a006a750 0x7f88a006ab30 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f8890009840 tx=0x7f8890009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6451400e9370308 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:39.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.886+0000 7f889d59a700 1 -- 172.21.15.1:0/187470358 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890011830 con 0x7f88a006a750 2024-03-18T20:25:39.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.886+0000 7f889d59a700 1 -- 172.21.15.1:0/187470358 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8890011990 con 0x7f88a006a750 2024-03-18T20:25:39.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.886+0000 7f889d59a700 1 -- 172.21.15.1:0/187470358 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890011d20 con 0x7f88a006a750 2024-03-18T20:25:39.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.886+0000 7f88a4e48700 1 -- 172.21.15.1:0/187470358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a006a750 msgr2=0x7f88a006ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.886+0000 7f88a4e48700 1 --2- 172.21.15.1:0/187470358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a006a750 0x7f88a006ab30 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f8890009840 tx=0x7f8890009b50 comp rx=0 tx=0).stop 2024-03-18T20:25:39.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.887+0000 7f88a4e48700 1 -- 172.21.15.1:0/187470358 shutdown_connections 2024-03-18T20:25:39.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.887+0000 7f88a4e48700 1 --2- 172.21.15.1:0/187470358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a006a750 0x7f88a006ab30 unknown :-1 s=CLOSED pgs=2581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.887+0000 7f88a4e48700 1 -- 172.21.15.1:0/187470358 >> 172.21.15.1:0/187470358 conn(0x7f88a006b860 msgr2=0x7f88a006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:39.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.887+0000 7f88a4e48700 1 -- 172.21.15.1:0/187470358 shutdown_connections 2024-03-18T20:25:39.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.887+0000 7f88a4e48700 1 -- 172.21.15.1:0/187470358 wait complete. 2024-03-18T20:25:39.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.888+0000 7f88a4e48700 1 Processor -- start 2024-03-18T20:25:39.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.888+0000 7f88a4e48700 1 -- start start 2024-03-18T20:25:39.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f88a4e48700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88a006a750 0x7f88a0151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f88a4e48700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a00ad790 0x7f88a0153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f88a4e48700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a015d6b0 0x7f88a00cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f88a4e48700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a00cd480 con 0x7f88a015d6b0 2024-03-18T20:25:39.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f88a4e48700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a015dc90 con 0x7f88a006a750 2024-03-18T20:25:39.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f88a4e48700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a015df30 con 0x7f88a00ad790 2024-03-18T20:25:39.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f889ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a015d6b0 0x7f88a00cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f889ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a015d6b0 0x7f88a00cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49860/0 (socket says 172.21.15.1:49860) 2024-03-18T20:25:39.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f889ed9d700 1 -- 172.21.15.1:0/2129857663 learned_addr learned my addr 172.21.15.1:0/2129857663 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:39.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f889ed9d700 1 -- 172.21.15.1:0/2129857663 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a00ad790 msgr2=0x7f88a0153960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f889dd9b700 1 --2- 172.21.15.1:0/2129857663 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a00ad790 0x7f88a0153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f889e59c700 1 --2- 172.21.15.1:0/2129857663 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88a006a750 0x7f88a0151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f889ed9d700 1 --2- 172.21.15.1:0/2129857663 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a00ad790 0x7f88a0153960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f889ed9d700 1 -- 172.21.15.1:0/2129857663 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88a006a750 msgr2=0x7f88a0151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:39.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f889ed9d700 1 --2- 172.21.15.1:0/2129857663 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88a006a750 0x7f88a0151f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:39.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f889ed9d700 1 -- 172.21.15.1:0/2129857663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88900094a0 con 0x7f88a015d6b0 2024-03-18T20:25:39.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f889ed9d700 1 --2- 172.21.15.1:0/2129857663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a015d6b0 0x7f88a00cc9d0 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7f889800cc50 tx=0x7f889800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:39.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f8896ffd700 1 -- 172.21.15.1:0/2129857663 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889800ad80 con 0x7f88a015d6b0 2024-03-18T20:25:39.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f8896ffd700 1 -- 172.21.15.1:0/2129857663 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889800ce50 con 0x7f88a015d6b0 2024-03-18T20:25:39.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f88a4e48700 1 -- 172.21.15.1:0/2129857663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a015e210 con 0x7f88a015d6b0 2024-03-18T20:25:39.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f8896ffd700 1 -- 172.21.15.1:0/2129857663 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8898003c40 con 0x7f88a015d6b0 2024-03-18T20:25:39.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.889+0000 7f88a4e48700 1 -- 172.21.15.1:0/2129857663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a00cdef0 con 0x7f88a015d6b0 2024-03-18T20:25:39.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.890+0000 7f88a4e48700 1 -- 172.21.15.1:0/2129857663 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a00ad420 con 0x7f88a015d6b0 2024-03-18T20:25:39.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.891+0000 7f8896ffd700 1 -- 172.21.15.1:0/2129857663 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f88980050d0 con 0x7f88a015d6b0 2024-03-18T20:25:39.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.891+0000 7f8896ffd700 1 --2- 172.21.15.1:0/2129857663 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f888006adb0 0x7f888006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:39.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.891+0000 7f8896ffd700 1 -- 172.21.15.1:0/2129857663 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f88980899f0 con 0x7f88a015d6b0 2024-03-18T20:25:39.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.891+0000 7f889e59c700 1 --2- 172.21.15.1:0/2129857663 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f888006adb0 0x7f888006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:39.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.892+0000 7f889e59c700 1 --2- 172.21.15.1:0/2129857663 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f888006adb0 0x7f888006d270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f88a00ae9f0 tx=0x7f889001c040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:39.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:39.896+0000 7f8896ffd700 1 -- 172.21.15.1:0/2129857663 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f88a00ad420 con 0x7f88a015d6b0 2024-03-18T20:25:40.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.093+0000 7f88a4e48700 1 -- 172.21.15.1:0/2129857663 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f88a00af4c0 con 0x7f888006adb0 2024-03-18T20:25:40.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.107+0000 7f8896ffd700 1 -- 172.21.15.1:0/2129857663 <== mgr.11322 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+518107 (secure 0 0 0) 0x7f8890011b00 con 0x7f888006adb0 2024-03-18T20:25:40.118 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:40.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.111+0000 7f88a4e48700 1 -- 172.21.15.1:0/2129857663 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f888006adb0 msgr2=0x7f888006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.111+0000 7f88a4e48700 1 --2- 172.21.15.1:0/2129857663 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f888006adb0 0x7f888006d270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f88a00ae9f0 tx=0x7f889001c040 comp rx=0 tx=0).stop 2024-03-18T20:25:40.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.111+0000 7f88a4e48700 1 -- 172.21.15.1:0/2129857663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a015d6b0 msgr2=0x7f88a00cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.111+0000 7f88a4e48700 1 --2- 172.21.15.1:0/2129857663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a015d6b0 0x7f88a00cc9d0 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7f889800cc50 tx=0x7f889800a340 comp rx=0 tx=0).stop 2024-03-18T20:25:40.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.111+0000 7f88a4e48700 1 -- 172.21.15.1:0/2129857663 shutdown_connections 2024-03-18T20:25:40.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.111+0000 7f88a4e48700 1 --2- 172.21.15.1:0/2129857663 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f888006adb0 0x7f888006d270 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.111+0000 7f88a4e48700 1 --2- 172.21.15.1:0/2129857663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88a015d6b0 0x7f88a00cc9d0 unknown :-1 s=CLOSED pgs=2582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.111+0000 7f88a4e48700 1 --2- 172.21.15.1:0/2129857663 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88a006a750 0x7f88a0151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.111+0000 7f88a4e48700 1 --2- 172.21.15.1:0/2129857663 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88a00ad790 0x7f88a0153960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.111+0000 7f88a4e48700 1 -- 172.21.15.1:0/2129857663 >> 172.21.15.1:0/2129857663 conn(0x7f88a006b860 msgr2=0x7f88a00b11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:40.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.111+0000 7f88a4e48700 1 -- 172.21.15.1:0/2129857663 shutdown_connections 2024-03-18T20:25:40.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.111+0000 7f88a4e48700 1 -- 172.21.15.1:0/2129857663 wait complete. 2024-03-18T20:25:40.124 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:25:40.127 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-18T20:25:39.102708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593988,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":626,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":389,"ondisk_log_size":389,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":212,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":385384,"kb_used_data":24072,"kb_used_omap":12,"kb_used_meta":361203,"kb_avail":754589336,"statfs":{"total":773094113280,"available":772699480064,"internally_reserved":0,"allocated":24649728,"data_stored":13189064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":369872441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":815,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.995476"},"pg_stats":[{"pgid":"1.0","version":"221'348","reported_seq":1201,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605524+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:25:24.605524+0000","last_peered":"2024-03-18T20:25:24.605524+0000","last_clean":"2024-03-18T20:25:24.605524+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:25:24.605524+0000","last_undegraded":"2024-03-18T20:25:24.605524+0000","last_fullsized":"2024-03-18T20:25:24.605524+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"231'1","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100643+0000","last_change":"2024-03-18T20:25:22.248248+0000","last_active":"2024-03-18T20:25:25.100643+0000","last_peered":"2024-03-18T20:25:25.100643+0000","last_clean":"2024-03-18T20:25:25.100643+0000","last_became_active":"2024-03-18T20:25:22.247970+0000","last_became_peered":"2024-03-18T20:25:22.247970+0000","last_unstale":"2024-03-18T20:25:25.100643+0000","last_undegraded":"2024-03-18T20:25:25.100643+0000","last_fullsized":"2024-03-18T20:25:25.100643+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:01:50.523530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101112+0000","last_change":"2024-03-18T20:25:22.687598+0000","last_active":"2024-03-18T20:25:25.101112+0000","last_peered":"2024-03-18T20:25:25.101112+0000","last_clean":"2024-03-18T20:25:25.101112+0000","last_became_active":"2024-03-18T20:25:22.686962+0000","last_became_peered":"2024-03-18T20:25:22.686962+0000","last_unstale":"2024-03-18T20:25:25.101112+0000","last_undegraded":"2024-03-18T20:25:25.101112+0000","last_fullsized":"2024-03-18T20:25:25.101112+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:23:10.983253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.114498+0000","last_change":"2024-03-18T20:25:34.114498+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.114498+0000","last_undegraded":"2024-03-18T20:25:34.114498+0000","last_fullsized":"2024-03-18T20:25:34.114498+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:47.366224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101303+0000","last_change":"2024-03-18T20:25:22.252803+0000","last_active":"2024-03-18T20:25:25.101303+0000","last_peered":"2024-03-18T20:25:25.101303+0000","last_clean":"2024-03-18T20:25:25.101303+0000","last_became_active":"2024-03-18T20:25:22.252169+0000","last_became_peered":"2024-03-18T20:25:22.252169+0000","last_unstale":"2024-03-18T20:25:25.101303+0000","last_undegraded":"2024-03-18T20:25:25.101303+0000","last_fullsized":"2024-03-18T20:25:25.101303+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:12.985428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097715+0000","last_change":"2024-03-18T20:25:22.696025+0000","last_active":"2024-03-18T20:25:25.097715+0000","last_peered":"2024-03-18T20:25:25.097715+0000","last_clean":"2024-03-18T20:25:25.097715+0000","last_became_active":"2024-03-18T20:25:22.695759+0000","last_became_peered":"2024-03-18T20:25:22.695759+0000","last_unstale":"2024-03-18T20:25:25.097715+0000","last_undegraded":"2024-03-18T20:25:25.097715+0000","last_fullsized":"2024-03-18T20:25:25.097715+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:10.681789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.117162+0000","last_change":"2024-03-18T20:25:34.117162+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.117162+0000","last_undegraded":"2024-03-18T20:25:34.117162+0000","last_fullsized":"2024-03-18T20:25:34.117162+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:13.759015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101033+0000","last_change":"2024-03-18T20:25:22.252889+0000","last_active":"2024-03-18T20:25:25.101033+0000","last_peered":"2024-03-18T20:25:25.101033+0000","last_clean":"2024-03-18T20:25:25.101033+0000","last_became_active":"2024-03-18T20:25:22.252096+0000","last_became_peered":"2024-03-18T20:25:22.252096+0000","last_unstale":"2024-03-18T20:25:25.101033+0000","last_undegraded":"2024-03-18T20:25:25.101033+0000","last_fullsized":"2024-03-18T20:25:25.101033+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:18.568149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101094+0000","last_change":"2024-03-18T20:25:22.689532+0000","last_active":"2024-03-18T20:25:25.101094+0000","last_peered":"2024-03-18T20:25:25.101094+0000","last_clean":"2024-03-18T20:25:25.101094+0000","last_became_active":"2024-03-18T20:25:22.689210+0000","last_became_peered":"2024-03-18T20:25:22.689210+0000","last_unstale":"2024-03-18T20:25:25.101094+0000","last_undegraded":"2024-03-18T20:25:25.101094+0000","last_fullsized":"2024-03-18T20:25:25.101094+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:21.229791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.111213+0000","last_change":"2024-03-18T20:25:34.111213+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.111213+0000","last_undegraded":"2024-03-18T20:25:34.111213+0000","last_fullsized":"2024-03-18T20:25:34.111213+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:17.295248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100663+0000","last_change":"2024-03-18T20:25:22.254053+0000","last_active":"2024-03-18T20:25:25.100663+0000","last_peered":"2024-03-18T20:25:25.100663+0000","last_clean":"2024-03-18T20:25:25.100663+0000","last_became_active":"2024-03-18T20:25:22.253747+0000","last_became_peered":"2024-03-18T20:25:22.253747+0000","last_unstale":"2024-03-18T20:25:25.100663+0000","last_undegraded":"2024-03-18T20:25:25.100663+0000","last_fullsized":"2024-03-18T20:25:25.100663+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:32.777143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101083+0000","last_change":"2024-03-18T20:25:22.685300+0000","last_active":"2024-03-18T20:25:25.101083+0000","last_peered":"2024-03-18T20:25:25.101083+0000","last_clean":"2024-03-18T20:25:25.101083+0000","last_became_active":"2024-03-18T20:25:22.684869+0000","last_became_peered":"2024-03-18T20:25:22.684869+0000","last_unstale":"2024-03-18T20:25:25.101083+0000","last_undegraded":"2024-03-18T20:25:25.101083+0000","last_fullsized":"2024-03-18T20:25:25.101083+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:02.037780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.110932+0000","last_change":"2024-03-18T20:25:34.110932+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.110932+0000","last_undegraded":"2024-03-18T20:25:34.110932+0000","last_fullsized":"2024-03-18T20:25:34.110932+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:44:27.205313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099377+0000","last_change":"2024-03-18T20:25:22.245633+0000","last_active":"2024-03-18T20:25:25.099377+0000","last_peered":"2024-03-18T20:25:25.099377+0000","last_clean":"2024-03-18T20:25:25.099377+0000","last_became_active":"2024-03-18T20:25:22.245372+0000","last_became_peered":"2024-03-18T20:25:22.245372+0000","last_unstale":"2024-03-18T20:25:25.099377+0000","last_undegraded":"2024-03-18T20:25:25.099377+0000","last_fullsized":"2024-03-18T20:25:25.099377+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:28:25.048682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099401+0000","last_change":"2024-03-18T20:25:22.683574+0000","last_active":"2024-03-18T20:25:25.099401+0000","last_peered":"2024-03-18T20:25:25.099401+0000","last_clean":"2024-03-18T20:25:25.099401+0000","last_became_active":"2024-03-18T20:25:22.683015+0000","last_became_peered":"2024-03-18T20:25:22.683015+0000","last_unstale":"2024-03-18T20:25:25.099401+0000","last_undegraded":"2024-03-18T20:25:25.099401+0000","last_fullsized":"2024-03-18T20:25:25.099401+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:11.606826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.116497+0000","last_change":"2024-03-18T20:25:34.116497+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.116497+0000","last_undegraded":"2024-03-18T20:25:34.116497+0000","last_fullsized":"2024-03-18T20:25:34.116497+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:41.670198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101098+0000","last_change":"2024-03-18T20:25:22.255435+0000","last_active":"2024-03-18T20:25:25.101098+0000","last_peered":"2024-03-18T20:25:25.101098+0000","last_clean":"2024-03-18T20:25:25.101098+0000","last_became_active":"2024-03-18T20:25:22.255149+0000","last_became_peered":"2024-03-18T20:25:22.255149+0000","last_unstale":"2024-03-18T20:25:25.101098+0000","last_undegraded":"2024-03-18T20:25:25.101098+0000","last_fullsized":"2024-03-18T20:25:25.101098+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:58.960198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101079+0000","last_change":"2024-03-18T20:25:22.682910+0000","last_active":"2024-03-18T20:25:25.101079+0000","last_peered":"2024-03-18T20:25:25.101079+0000","last_clean":"2024-03-18T20:25:25.101079+0000","last_became_active":"2024-03-18T20:25:22.682667+0000","last_became_peered":"2024-03-18T20:25:22.682667+0000","last_unstale":"2024-03-18T20:25:25.101079+0000","last_undegraded":"2024-03-18T20:25:25.101079+0000","last_fullsized":"2024-03-18T20:25:25.101079+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:02.037780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.115535+0000","last_change":"2024-03-18T20:25:34.115535+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.115535+0000","last_undegraded":"2024-03-18T20:25:34.115535+0000","last_fullsized":"2024-03-18T20:25:34.115535+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:56.513767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100915+0000","last_change":"2024-03-18T20:25:22.250457+0000","last_active":"2024-03-18T20:25:25.100915+0000","last_peered":"2024-03-18T20:25:25.100915+0000","last_clean":"2024-03-18T20:25:25.100915+0000","last_became_active":"2024-03-18T20:25:22.250157+0000","last_became_peered":"2024-03-18T20:25:22.250157+0000","last_unstale":"2024-03-18T20:25:25.100915+0000","last_undegraded":"2024-03-18T20:25:25.100915+0000","last_fullsized":"2024-03-18T20:25:25.100915+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:18.568149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100623+0000","last_change":"2024-03-18T20:25:22.691245+0000","last_active":"2024-03-18T20:25:25.100623+0000","last_peered":"2024-03-18T20:25:25.100623+0000","last_clean":"2024-03-18T20:25:25.100623+0000","last_became_active":"2024-03-18T20:25:22.690988+0000","last_became_peered":"2024-03-18T20:25:22.690988+0000","last_unstale":"2024-03-18T20:25:25.100623+0000","last_undegraded":"2024-03-18T20:25:25.100623+0000","last_fullsized":"2024-03-18T20:25:25.100623+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:32:54.125659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.112128+0000","last_change":"2024-03-18T20:25:34.112128+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.112128+0000","last_undegraded":"2024-03-18T20:25:34.112128+0000","last_fullsized":"2024-03-18T20:25:34.112128+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:25:32.071310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.6","version":"232'1","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:27.359403+0000","last_change":"2024-03-18T20:25:22.247183+0000","last_active":"2024-03-18T20:25:27.359403+0000","last_peered":"2024-03-18T20:25:27.359403+0000","last_clean":"2024-03-18T20:25:27.359403+0000","last_became_active":"2024-03-18T20:25:22.246890+0000","last_became_peered":"2024-03-18T20:25:22.246890+0000","last_unstale":"2024-03-18T20:25:27.359403+0000","last_undegraded":"2024-03-18T20:25:27.359403+0000","last_fullsized":"2024-03-18T20:25:27.359403+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:40:21.886849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097599+0000","last_change":"2024-03-18T20:25:22.689011+0000","last_active":"2024-03-18T20:25:25.097599+0000","last_peered":"2024-03-18T20:25:25.097599+0000","last_clean":"2024-03-18T20:25:25.097599+0000","last_became_active":"2024-03-18T20:25:22.688750+0000","last_became_peered":"2024-03-18T20:25:22.688750+0000","last_unstale":"2024-03-18T20:25:25.097599+0000","last_undegraded":"2024-03-18T20:25:25.097599+0000","last_fullsized":"2024-03-18T20:25:25.097599+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:57.345369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.111764+0000","last_change":"2024-03-18T20:25:34.111764+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.111764+0000","last_undegraded":"2024-03-18T20:25:34.111764+0000","last_fullsized":"2024-03-18T20:25:34.111764+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:25.724222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.7","version":"231'2","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100865+0000","last_change":"2024-03-18T20:25:22.254151+0000","last_active":"2024-03-18T20:25:25.100865+0000","last_peered":"2024-03-18T20:25:25.100865+0000","last_clean":"2024-03-18T20:25:25.100865+0000","last_became_active":"2024-03-18T20:25:22.253873+0000","last_became_peered":"2024-03-18T20:25:22.253873+0000","last_unstale":"2024-03-18T20:25:25.100865+0000","last_undegraded":"2024-03-18T20:25:25.100865+0000","last_fullsized":"2024-03-18T20:25:25.100865+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:26:58.327699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099308+0000","last_change":"2024-03-18T20:25:22.693170+0000","last_active":"2024-03-18T20:25:25.099308+0000","last_peered":"2024-03-18T20:25:25.099308+0000","last_clean":"2024-03-18T20:25:25.099308+0000","last_became_active":"2024-03-18T20:25:22.691592+0000","last_became_peered":"2024-03-18T20:25:22.691592+0000","last_unstale":"2024-03-18T20:25:25.099308+0000","last_undegraded":"2024-03-18T20:25:25.099308+0000","last_fullsized":"2024-03-18T20:25:25.099308+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:23.897085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.114953+0000","last_change":"2024-03-18T20:25:34.114953+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.114953+0000","last_undegraded":"2024-03-18T20:25:34.114953+0000","last_fullsized":"2024-03-18T20:25:34.114953+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:06:09.253104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100981+0000","last_change":"2024-03-18T20:25:22.251509+0000","last_active":"2024-03-18T20:25:25.100981+0000","last_peered":"2024-03-18T20:25:25.100981+0000","last_clean":"2024-03-18T20:25:25.100981+0000","last_became_active":"2024-03-18T20:25:22.251240+0000","last_became_peered":"2024-03-18T20:25:22.251240+0000","last_unstale":"2024-03-18T20:25:25.100981+0000","last_undegraded":"2024-03-18T20:25:25.100981+0000","last_fullsized":"2024-03-18T20:25:25.100981+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:13:14.870270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099451+0000","last_change":"2024-03-18T20:25:22.695653+0000","last_active":"2024-03-18T20:25:25.099451+0000","last_peered":"2024-03-18T20:25:25.099451+0000","last_clean":"2024-03-18T20:25:25.099451+0000","last_became_active":"2024-03-18T20:25:22.694092+0000","last_became_peered":"2024-03-18T20:25:22.694092+0000","last_unstale":"2024-03-18T20:25:25.099451+0000","last_undegraded":"2024-03-18T20:25:25.099451+0000","last_fullsized":"2024-03-18T20:25:25.099451+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:20.319345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.115435+0000","last_change":"2024-03-18T20:25:34.115435+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.115435+0000","last_undegraded":"2024-03-18T20:25:34.115435+0000","last_fullsized":"2024-03-18T20:25:34.115435+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:32:42.191508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100853+0000","last_change":"2024-03-18T20:25:22.251818+0000","last_active":"2024-03-18T20:25:25.100853+0000","last_peered":"2024-03-18T20:25:25.100853+0000","last_clean":"2024-03-18T20:25:25.100853+0000","last_became_active":"2024-03-18T20:25:22.251407+0000","last_became_peered":"2024-03-18T20:25:22.251407+0000","last_unstale":"2024-03-18T20:25:25.100853+0000","last_undegraded":"2024-03-18T20:25:25.100853+0000","last_fullsized":"2024-03-18T20:25:25.100853+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:29.196742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100780+0000","last_change":"2024-03-18T20:25:22.684929+0000","last_active":"2024-03-18T20:25:25.100780+0000","last_peered":"2024-03-18T20:25:25.100780+0000","last_clean":"2024-03-18T20:25:25.100780+0000","last_became_active":"2024-03-18T20:25:22.684658+0000","last_became_peered":"2024-03-18T20:25:22.684658+0000","last_unstale":"2024-03-18T20:25:25.100780+0000","last_undegraded":"2024-03-18T20:25:25.100780+0000","last_fullsized":"2024-03-18T20:25:25.100780+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:39.733357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.113347+0000","last_change":"2024-03-18T20:25:34.113347+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.113347+0000","last_undegraded":"2024-03-18T20:25:34.113347+0000","last_fullsized":"2024-03-18T20:25:34.113347+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:22:13.432979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100944+0000","last_change":"2024-03-18T20:25:22.247489+0000","last_active":"2024-03-18T20:25:25.100944+0000","last_peered":"2024-03-18T20:25:25.100944+0000","last_clean":"2024-03-18T20:25:25.100944+0000","last_became_active":"2024-03-18T20:25:22.247203+0000","last_became_peered":"2024-03-18T20:25:22.247203+0000","last_unstale":"2024-03-18T20:25:25.100944+0000","last_undegraded":"2024-03-18T20:25:25.100944+0000","last_fullsized":"2024-03-18T20:25:25.100944+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:32.481118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101073+0000","last_change":"2024-03-18T20:25:22.681615+0000","last_active":"2024-03-18T20:25:25.101073+0000","last_peered":"2024-03-18T20:25:25.101073+0000","last_clean":"2024-03-18T20:25:25.101073+0000","last_became_active":"2024-03-18T20:25:22.681261+0000","last_became_peered":"2024-03-18T20:25:22.681261+0000","last_unstale":"2024-03-18T20:25:25.101073+0000","last_undegraded":"2024-03-18T20:25:25.101073+0000","last_fullsized":"2024-03-18T20:25:25.101073+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:15:29.461741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.111539+0000","last_change":"2024-03-18T20:25:34.111539+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.111539+0000","last_undegraded":"2024-03-18T20:25:34.111539+0000","last_fullsized":"2024-03-18T20:25:34.111539+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:33.109951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.b","version":"231'2","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101371+0000","last_change":"2024-03-18T20:25:22.251851+0000","last_active":"2024-03-18T20:25:25.101371+0000","last_peered":"2024-03-18T20:25:25.101371+0000","last_clean":"2024-03-18T20:25:25.101371+0000","last_became_active":"2024-03-18T20:25:22.251334+0000","last_became_peered":"2024-03-18T20:25:22.251334+0000","last_unstale":"2024-03-18T20:25:25.101371+0000","last_undegraded":"2024-03-18T20:25:25.101371+0000","last_fullsized":"2024-03-18T20:25:25.101371+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:22.889781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100991+0000","last_change":"2024-03-18T20:25:22.687129+0000","last_active":"2024-03-18T20:25:25.100991+0000","last_peered":"2024-03-18T20:25:25.100991+0000","last_clean":"2024-03-18T20:25:25.100991+0000","last_became_active":"2024-03-18T20:25:22.686871+0000","last_became_peered":"2024-03-18T20:25:22.686871+0000","last_unstale":"2024-03-18T20:25:25.100991+0000","last_undegraded":"2024-03-18T20:25:25.100991+0000","last_fullsized":"2024-03-18T20:25:25.100991+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:18:58.196895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.111594+0000","last_change":"2024-03-18T20:25:34.111594+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.111594+0000","last_undegraded":"2024-03-18T20:25:34.111594+0000","last_fullsized":"2024-03-18T20:25:34.111594+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:13.917940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101413+0000","last_change":"2024-03-18T20:25:22.254305+0000","last_active":"2024-03-18T20:25:25.101413+0000","last_peered":"2024-03-18T20:25:25.101413+0000","last_clean":"2024-03-18T20:25:25.101413+0000","last_became_active":"2024-03-18T20:25:22.253898+0000","last_became_peered":"2024-03-18T20:25:22.253898+0000","last_unstale":"2024-03-18T20:25:25.101413+0000","last_undegraded":"2024-03-18T20:25:25.101413+0000","last_fullsized":"2024-03-18T20:25:25.101413+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:29:48.394283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099365+0000","last_change":"2024-03-18T20:25:22.692895+0000","last_active":"2024-03-18T20:25:25.099365+0000","last_peered":"2024-03-18T20:25:25.099365+0000","last_clean":"2024-03-18T20:25:25.099365+0000","last_became_active":"2024-03-18T20:25:22.685766+0000","last_became_peered":"2024-03-18T20:25:22.685766+0000","last_unstale":"2024-03-18T20:25:25.099365+0000","last_undegraded":"2024-03-18T20:25:25.099365+0000","last_fullsized":"2024-03-18T20:25:25.099365+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:16:08.020891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.117145+0000","last_change":"2024-03-18T20:25:34.117145+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.117145+0000","last_undegraded":"2024-03-18T20:25:34.117145+0000","last_fullsized":"2024-03-18T20:25:34.117145+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:38.721170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.d","version":"231'1","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101013+0000","last_change":"2024-03-18T20:25:22.248228+0000","last_active":"2024-03-18T20:25:25.101013+0000","last_peered":"2024-03-18T20:25:25.101013+0000","last_clean":"2024-03-18T20:25:25.101013+0000","last_became_active":"2024-03-18T20:25:22.247916+0000","last_became_peered":"2024-03-18T20:25:22.247916+0000","last_unstale":"2024-03-18T20:25:25.101013+0000","last_undegraded":"2024-03-18T20:25:25.101013+0000","last_fullsized":"2024-03-18T20:25:25.101013+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:56.236862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101055+0000","last_change":"2024-03-18T20:25:22.684910+0000","last_active":"2024-03-18T20:25:25.101055+0000","last_peered":"2024-03-18T20:25:25.101055+0000","last_clean":"2024-03-18T20:25:25.101055+0000","last_became_active":"2024-03-18T20:25:22.684472+0000","last_became_peered":"2024-03-18T20:25:22.684472+0000","last_unstale":"2024-03-18T20:25:25.101055+0000","last_undegraded":"2024-03-18T20:25:25.101055+0000","last_fullsized":"2024-03-18T20:25:25.101055+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:24:33.293594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.112636+0000","last_change":"2024-03-18T20:25:34.112636+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.112636+0000","last_undegraded":"2024-03-18T20:25:34.112636+0000","last_fullsized":"2024-03-18T20:25:34.112636+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:15.369281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.e","version":"231'2","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101154+0000","last_change":"2024-03-18T20:25:22.253997+0000","last_active":"2024-03-18T20:25:25.101154+0000","last_peered":"2024-03-18T20:25:25.101154+0000","last_clean":"2024-03-18T20:25:25.101154+0000","last_became_active":"2024-03-18T20:25:22.253649+0000","last_became_peered":"2024-03-18T20:25:22.253649+0000","last_unstale":"2024-03-18T20:25:25.101154+0000","last_undegraded":"2024-03-18T20:25:25.101154+0000","last_fullsized":"2024-03-18T20:25:25.101154+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:00.435897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100997+0000","last_change":"2024-03-18T20:25:22.687892+0000","last_active":"2024-03-18T20:25:25.100997+0000","last_peered":"2024-03-18T20:25:25.100997+0000","last_clean":"2024-03-18T20:25:25.100997+0000","last_became_active":"2024-03-18T20:25:22.687086+0000","last_became_peered":"2024-03-18T20:25:22.687086+0000","last_unstale":"2024-03-18T20:25:25.100997+0000","last_undegraded":"2024-03-18T20:25:25.100997+0000","last_fullsized":"2024-03-18T20:25:25.100997+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:15:29.461741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.115166+0000","last_change":"2024-03-18T20:25:34.115166+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.115166+0000","last_undegraded":"2024-03-18T20:25:34.115166+0000","last_fullsized":"2024-03-18T20:25:34.115166+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:17.295248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.f","version":"231'1","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101196+0000","last_change":"2024-03-18T20:25:22.255372+0000","last_active":"2024-03-18T20:25:25.101196+0000","last_peered":"2024-03-18T20:25:25.101196+0000","last_clean":"2024-03-18T20:25:25.101196+0000","last_became_active":"2024-03-18T20:25:22.255063+0000","last_became_peered":"2024-03-18T20:25:22.255063+0000","last_unstale":"2024-03-18T20:25:25.101196+0000","last_undegraded":"2024-03-18T20:25:25.101196+0000","last_fullsized":"2024-03-18T20:25:25.101196+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:23.143345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099295+0000","last_change":"2024-03-18T20:25:22.686185+0000","last_active":"2024-03-18T20:25:25.099295+0000","last_peered":"2024-03-18T20:25:25.099295+0000","last_clean":"2024-03-18T20:25:25.099295+0000","last_became_active":"2024-03-18T20:25:22.685939+0000","last_became_peered":"2024-03-18T20:25:22.685939+0000","last_unstale":"2024-03-18T20:25:25.099295+0000","last_undegraded":"2024-03-18T20:25:25.099295+0000","last_fullsized":"2024-03-18T20:25:25.099295+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:48:48.961872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.116524+0000","last_change":"2024-03-18T20:25:34.116524+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.116524+0000","last_undegraded":"2024-03-18T20:25:34.116524+0000","last_fullsized":"2024-03-18T20:25:34.116524+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:50.264082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.10","version":"231'8","reported_seq":20,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101286+0000","last_change":"2024-03-18T20:25:22.252976+0000","last_active":"2024-03-18T20:25:25.101286+0000","last_peered":"2024-03-18T20:25:25.101286+0000","last_clean":"2024-03-18T20:25:25.101286+0000","last_became_active":"2024-03-18T20:25:22.252528+0000","last_became_peered":"2024-03-18T20:25:22.252528+0000","last_unstale":"2024-03-18T20:25:25.101286+0000","last_undegraded":"2024-03-18T20:25:25.101286+0000","last_fullsized":"2024-03-18T20:25:25.101286+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:42:24.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101120+0000","last_change":"2024-03-18T20:25:22.673559+0000","last_active":"2024-03-18T20:25:25.101120+0000","last_peered":"2024-03-18T20:25:25.101120+0000","last_clean":"2024-03-18T20:25:25.101120+0000","last_became_active":"2024-03-18T20:25:22.673267+0000","last_became_peered":"2024-03-18T20:25:22.673267+0000","last_unstale":"2024-03-18T20:25:25.101120+0000","last_undegraded":"2024-03-18T20:25:25.101120+0000","last_fullsized":"2024-03-18T20:25:25.101120+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:44:15.325153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.116256+0000","last_change":"2024-03-18T20:25:34.116256+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.116256+0000","last_undegraded":"2024-03-18T20:25:34.116256+0000","last_fullsized":"2024-03-18T20:25:34.116256+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:50.170851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100588+0000","last_change":"2024-03-18T20:25:22.252927+0000","last_active":"2024-03-18T20:25:25.100588+0000","last_peered":"2024-03-18T20:25:25.100588+0000","last_clean":"2024-03-18T20:25:25.100588+0000","last_became_active":"2024-03-18T20:25:22.252633+0000","last_became_peered":"2024-03-18T20:25:22.252633+0000","last_unstale":"2024-03-18T20:25:25.100588+0000","last_undegraded":"2024-03-18T20:25:25.100588+0000","last_fullsized":"2024-03-18T20:25:25.100588+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:32.345526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100761+0000","last_change":"2024-03-18T20:25:22.694369+0000","last_active":"2024-03-18T20:25:25.100761+0000","last_peered":"2024-03-18T20:25:25.100761+0000","last_clean":"2024-03-18T20:25:25.100761+0000","last_became_active":"2024-03-18T20:25:22.694114+0000","last_became_peered":"2024-03-18T20:25:22.694114+0000","last_unstale":"2024-03-18T20:25:25.100761+0000","last_undegraded":"2024-03-18T20:25:25.100761+0000","last_fullsized":"2024-03-18T20:25:25.100761+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:35:13.631181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.114153+0000","last_change":"2024-03-18T20:25:34.114153+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.114153+0000","last_undegraded":"2024-03-18T20:25:34.114153+0000","last_fullsized":"2024-03-18T20:25:34.114153+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:27.179302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097618+0000","last_change":"2024-03-18T20:25:22.249078+0000","last_active":"2024-03-18T20:25:25.097618+0000","last_peered":"2024-03-18T20:25:25.097618+0000","last_clean":"2024-03-18T20:25:25.097618+0000","last_became_active":"2024-03-18T20:25:22.248588+0000","last_became_peered":"2024-03-18T20:25:22.248588+0000","last_unstale":"2024-03-18T20:25:25.097618+0000","last_undegraded":"2024-03-18T20:25:25.097618+0000","last_fullsized":"2024-03-18T20:25:25.097618+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:34:42.893253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097697+0000","last_change":"2024-03-18T20:25:22.685090+0000","last_active":"2024-03-18T20:25:25.097697+0000","last_peered":"2024-03-18T20:25:25.097697+0000","last_clean":"2024-03-18T20:25:25.097697+0000","last_became_active":"2024-03-18T20:25:22.684824+0000","last_became_peered":"2024-03-18T20:25:22.684824+0000","last_unstale":"2024-03-18T20:25:25.097697+0000","last_undegraded":"2024-03-18T20:25:25.097697+0000","last_fullsized":"2024-03-18T20:25:25.097697+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:21.750701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.112308+0000","last_change":"2024-03-18T20:25:34.112308+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.112308+0000","last_undegraded":"2024-03-18T20:25:34.112308+0000","last_fullsized":"2024-03-18T20:25:34.112308+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:03.410994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.13","version":"231'2","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100964+0000","last_change":"2024-03-18T20:25:22.254668+0000","last_active":"2024-03-18T20:25:25.100964+0000","last_peered":"2024-03-18T20:25:25.100964+0000","last_clean":"2024-03-18T20:25:25.100964+0000","last_became_active":"2024-03-18T20:25:22.254437+0000","last_became_peered":"2024-03-18T20:25:22.254437+0000","last_unstale":"2024-03-18T20:25:25.100964+0000","last_undegraded":"2024-03-18T20:25:25.100964+0000","last_fullsized":"2024-03-18T20:25:25.100964+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:32:02.691847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100981+0000","last_change":"2024-03-18T20:25:22.691203+0000","last_active":"2024-03-18T20:25:25.100981+0000","last_peered":"2024-03-18T20:25:25.100981+0000","last_clean":"2024-03-18T20:25:25.100981+0000","last_became_active":"2024-03-18T20:25:22.690964+0000","last_became_peered":"2024-03-18T20:25:22.690964+0000","last_unstale":"2024-03-18T20:25:25.100981+0000","last_undegraded":"2024-03-18T20:25:25.100981+0000","last_fullsized":"2024-03-18T20:25:25.100981+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:19.068003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.113878+0000","last_change":"2024-03-18T20:25:34.113878+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.113878+0000","last_undegraded":"2024-03-18T20:25:34.113878+0000","last_fullsized":"2024-03-18T20:25:34.113878+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:02.016045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.14","version":"232'8","reported_seq":20,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:32.359025+0000","last_change":"2024-03-18T20:25:22.249429+0000","last_active":"2024-03-18T20:25:32.359025+0000","last_peered":"2024-03-18T20:25:32.359025+0000","last_clean":"2024-03-18T20:25:32.359025+0000","last_became_active":"2024-03-18T20:25:22.248842+0000","last_became_peered":"2024-03-18T20:25:22.248842+0000","last_unstale":"2024-03-18T20:25:32.359025+0000","last_undegraded":"2024-03-18T20:25:32.359025+0000","last_fullsized":"2024-03-18T20:25:32.359025+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:26.827223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2822,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101359+0000","last_change":"2024-03-18T20:25:22.696408+0000","last_active":"2024-03-18T20:25:25.101359+0000","last_peered":"2024-03-18T20:25:25.101359+0000","last_clean":"2024-03-18T20:25:25.101359+0000","last_became_active":"2024-03-18T20:25:22.696095+0000","last_became_peered":"2024-03-18T20:25:22.696095+0000","last_unstale":"2024-03-18T20:25:25.101359+0000","last_undegraded":"2024-03-18T20:25:25.101359+0000","last_fullsized":"2024-03-18T20:25:25.101359+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:19:19.377686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.119645+0000","last_change":"2024-03-18T20:25:34.119645+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.119645+0000","last_undegraded":"2024-03-18T20:25:34.119645+0000","last_fullsized":"2024-03-18T20:25:34.119645+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:50.971852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.15","version":"231'3","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100719+0000","last_change":"2024-03-18T20:25:22.251534+0000","last_active":"2024-03-18T20:25:25.100719+0000","last_peered":"2024-03-18T20:25:25.100719+0000","last_clean":"2024-03-18T20:25:25.100719+0000","last_became_active":"2024-03-18T20:25:22.251275+0000","last_became_peered":"2024-03-18T20:25:22.251275+0000","last_unstale":"2024-03-18T20:25:25.100719+0000","last_undegraded":"2024-03-18T20:25:25.100719+0000","last_fullsized":"2024-03-18T20:25:25.100719+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:38:51.553676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100607+0000","last_change":"2024-03-18T20:25:22.682861+0000","last_active":"2024-03-18T20:25:25.100607+0000","last_peered":"2024-03-18T20:25:25.100607+0000","last_clean":"2024-03-18T20:25:25.100607+0000","last_became_active":"2024-03-18T20:25:22.682610+0000","last_became_peered":"2024-03-18T20:25:22.682610+0000","last_unstale":"2024-03-18T20:25:25.100607+0000","last_undegraded":"2024-03-18T20:25:25.100607+0000","last_fullsized":"2024-03-18T20:25:25.100607+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:51:43.095770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.114296+0000","last_change":"2024-03-18T20:25:34.114296+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.114296+0000","last_undegraded":"2024-03-18T20:25:34.114296+0000","last_fullsized":"2024-03-18T20:25:34.114296+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:32:42.191508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101107+0000","last_change":"2024-03-18T20:25:22.252980+0000","last_active":"2024-03-18T20:25:25.101107+0000","last_peered":"2024-03-18T20:25:25.101107+0000","last_clean":"2024-03-18T20:25:25.101107+0000","last_became_active":"2024-03-18T20:25:22.252706+0000","last_became_peered":"2024-03-18T20:25:22.252706+0000","last_unstale":"2024-03-18T20:25:25.101107+0000","last_undegraded":"2024-03-18T20:25:25.101107+0000","last_fullsized":"2024-03-18T20:25:25.101107+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:45:13.948304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097580+0000","last_change":"2024-03-18T20:25:22.684633+0000","last_active":"2024-03-18T20:25:25.097580+0000","last_peered":"2024-03-18T20:25:25.097580+0000","last_clean":"2024-03-18T20:25:25.097580+0000","last_became_active":"2024-03-18T20:25:22.684366+0000","last_became_peered":"2024-03-18T20:25:22.684366+0000","last_unstale":"2024-03-18T20:25:25.097580+0000","last_undegraded":"2024-03-18T20:25:25.097580+0000","last_fullsized":"2024-03-18T20:25:25.097580+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:15.299142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.112662+0000","last_change":"2024-03-18T20:25:34.112662+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.112662+0000","last_undegraded":"2024-03-18T20:25:34.112662+0000","last_fullsized":"2024-03-18T20:25:34.112662+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:50.170851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101424+0000","last_change":"2024-03-18T20:25:22.251887+0000","last_active":"2024-03-18T20:25:25.101424+0000","last_peered":"2024-03-18T20:25:25.101424+0000","last_clean":"2024-03-18T20:25:25.101424+0000","last_became_active":"2024-03-18T20:25:22.251594+0000","last_became_peered":"2024-03-18T20:25:22.251594+0000","last_unstale":"2024-03-18T20:25:25.101424+0000","last_undegraded":"2024-03-18T20:25:25.101424+0000","last_fullsized":"2024-03-18T20:25:25.101424+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:48:47.954568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100806+0000","last_change":"2024-03-18T20:25:22.680660+0000","last_active":"2024-03-18T20:25:25.100806+0000","last_peered":"2024-03-18T20:25:25.100806+0000","last_clean":"2024-03-18T20:25:25.100806+0000","last_became_active":"2024-03-18T20:25:22.680484+0000","last_became_peered":"2024-03-18T20:25:22.680484+0000","last_unstale":"2024-03-18T20:25:25.100806+0000","last_undegraded":"2024-03-18T20:25:25.100806+0000","last_fullsized":"2024-03-18T20:25:25.100806+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:48:21.194536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.110675+0000","last_change":"2024-03-18T20:25:34.110675+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.110675+0000","last_undegraded":"2024-03-18T20:25:34.110675+0000","last_fullsized":"2024-03-18T20:25:34.110675+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:20.466978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097560+0000","last_change":"2024-03-18T20:25:22.244727+0000","last_active":"2024-03-18T20:25:25.097560+0000","last_peered":"2024-03-18T20:25:25.097560+0000","last_clean":"2024-03-18T20:25:25.097560+0000","last_became_active":"2024-03-18T20:25:22.244417+0000","last_became_peered":"2024-03-18T20:25:22.244417+0000","last_unstale":"2024-03-18T20:25:25.097560+0000","last_undegraded":"2024-03-18T20:25:25.097560+0000","last_fullsized":"2024-03-18T20:25:25.097560+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:10.187509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100848+0000","last_change":"2024-03-18T20:25:22.681952+0000","last_active":"2024-03-18T20:25:25.100848+0000","last_peered":"2024-03-18T20:25:25.100848+0000","last_clean":"2024-03-18T20:25:25.100848+0000","last_became_active":"2024-03-18T20:25:22.681537+0000","last_became_peered":"2024-03-18T20:25:22.681537+0000","last_unstale":"2024-03-18T20:25:25.100848+0000","last_undegraded":"2024-03-18T20:25:25.100848+0000","last_fullsized":"2024-03-18T20:25:25.100848+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:32:22.923763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.113196+0000","last_change":"2024-03-18T20:25:34.113196+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.113196+0000","last_undegraded":"2024-03-18T20:25:34.113196+0000","last_fullsized":"2024-03-18T20:25:34.113196+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:39.087869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100983+0000","last_change":"2024-03-18T20:25:22.251522+0000","last_active":"2024-03-18T20:25:25.100983+0000","last_peered":"2024-03-18T20:25:25.100983+0000","last_clean":"2024-03-18T20:25:25.100983+0000","last_became_active":"2024-03-18T20:25:22.251143+0000","last_became_peered":"2024-03-18T20:25:22.251143+0000","last_unstale":"2024-03-18T20:25:25.100983+0000","last_undegraded":"2024-03-18T20:25:25.100983+0000","last_fullsized":"2024-03-18T20:25:25.100983+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:58.960198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100935+0000","last_change":"2024-03-18T20:25:22.691744+0000","last_active":"2024-03-18T20:25:25.100935+0000","last_peered":"2024-03-18T20:25:25.100935+0000","last_clean":"2024-03-18T20:25:25.100935+0000","last_became_active":"2024-03-18T20:25:22.691491+0000","last_became_peered":"2024-03-18T20:25:22.691491+0000","last_unstale":"2024-03-18T20:25:25.100935+0000","last_undegraded":"2024-03-18T20:25:25.100935+0000","last_fullsized":"2024-03-18T20:25:25.100935+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:04:09.670541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.115119+0000","last_change":"2024-03-18T20:25:34.115119+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.115119+0000","last_undegraded":"2024-03-18T20:25:34.115119+0000","last_fullsized":"2024-03-18T20:25:34.115119+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:25.724222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1b","version":"231'1","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101037+0000","last_change":"2024-03-18T20:25:22.251076+0000","last_active":"2024-03-18T20:25:25.101037+0000","last_peered":"2024-03-18T20:25:25.101037+0000","last_clean":"2024-03-18T20:25:25.101037+0000","last_became_active":"2024-03-18T20:25:22.250741+0000","last_became_peered":"2024-03-18T20:25:22.250741+0000","last_unstale":"2024-03-18T20:25:25.101037+0000","last_undegraded":"2024-03-18T20:25:25.101037+0000","last_fullsized":"2024-03-18T20:25:25.101037+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:07.448931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100894+0000","last_change":"2024-03-18T20:25:22.680768+0000","last_active":"2024-03-18T20:25:25.100894+0000","last_peered":"2024-03-18T20:25:25.100894+0000","last_clean":"2024-03-18T20:25:25.100894+0000","last_became_active":"2024-03-18T20:25:22.680516+0000","last_became_peered":"2024-03-18T20:25:22.680516+0000","last_unstale":"2024-03-18T20:25:25.100894+0000","last_undegraded":"2024-03-18T20:25:25.100894+0000","last_fullsized":"2024-03-18T20:25:25.100894+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:19:19.377686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.111795+0000","last_change":"2024-03-18T20:25:34.111795+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.111795+0000","last_undegraded":"2024-03-18T20:25:34.111795+0000","last_fullsized":"2024-03-18T20:25:34.111795+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:45.664606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1c","version":"231'2","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100697+0000","last_change":"2024-03-18T20:25:22.251018+0000","last_active":"2024-03-18T20:25:25.100697+0000","last_peered":"2024-03-18T20:25:25.100697+0000","last_clean":"2024-03-18T20:25:25.100697+0000","last_became_active":"2024-03-18T20:25:22.250730+0000","last_became_peered":"2024-03-18T20:25:22.250730+0000","last_unstale":"2024-03-18T20:25:25.100697+0000","last_undegraded":"2024-03-18T20:25:25.100697+0000","last_fullsized":"2024-03-18T20:25:25.100697+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:07.579514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101349+0000","last_change":"2024-03-18T20:25:22.690894+0000","last_active":"2024-03-18T20:25:25.101349+0000","last_peered":"2024-03-18T20:25:25.101349+0000","last_clean":"2024-03-18T20:25:25.101349+0000","last_became_active":"2024-03-18T20:25:22.688184+0000","last_became_peered":"2024-03-18T20:25:22.688184+0000","last_unstale":"2024-03-18T20:25:25.101349+0000","last_undegraded":"2024-03-18T20:25:25.101349+0000","last_fullsized":"2024-03-18T20:25:25.101349+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:59.967502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.114019+0000","last_change":"2024-03-18T20:25:34.114019+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.114019+0000","last_undegraded":"2024-03-18T20:25:34.114019+0000","last_fullsized":"2024-03-18T20:25:34.114019+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:33.630861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1d","version":"231'2","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101002+0000","last_change":"2024-03-18T20:25:22.255583+0000","last_active":"2024-03-18T20:25:25.101002+0000","last_peered":"2024-03-18T20:25:25.101002+0000","last_clean":"2024-03-18T20:25:25.101002+0000","last_became_active":"2024-03-18T20:25:22.255243+0000","last_became_peered":"2024-03-18T20:25:22.255243+0000","last_unstale":"2024-03-18T20:25:25.101002+0000","last_undegraded":"2024-03-18T20:25:25.101002+0000","last_fullsized":"2024-03-18T20:25:25.101002+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:38.084389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099342+0000","last_change":"2024-03-18T20:25:22.691295+0000","last_active":"2024-03-18T20:25:25.099342+0000","last_peered":"2024-03-18T20:25:25.099342+0000","last_clean":"2024-03-18T20:25:25.099342+0000","last_became_active":"2024-03-18T20:25:22.690890+0000","last_became_peered":"2024-03-18T20:25:22.690890+0000","last_unstale":"2024-03-18T20:25:25.099342+0000","last_undegraded":"2024-03-18T20:25:25.099342+0000","last_fullsized":"2024-03-18T20:25:25.099342+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:23:16.320044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.117626+0000","last_change":"2024-03-18T20:25:34.117626+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.117626+0000","last_undegraded":"2024-03-18T20:25:34.117626+0000","last_fullsized":"2024-03-18T20:25:34.117626+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:19:02.788524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1e","version":"231'1","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101139+0000","last_change":"2024-03-18T20:25:22.251725+0000","last_active":"2024-03-18T20:25:25.101139+0000","last_peered":"2024-03-18T20:25:25.101139+0000","last_clean":"2024-03-18T20:25:25.101139+0000","last_became_active":"2024-03-18T20:25:22.251416+0000","last_became_peered":"2024-03-18T20:25:22.251416+0000","last_unstale":"2024-03-18T20:25:25.101139+0000","last_undegraded":"2024-03-18T20:25:25.101139+0000","last_fullsized":"2024-03-18T20:25:25.101139+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:26.827223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101012+0000","last_change":"2024-03-18T20:25:22.679158+0000","last_active":"2024-03-18T20:25:25.101012+0000","last_peered":"2024-03-18T20:25:25.101012+0000","last_clean":"2024-03-18T20:25:25.101012+0000","last_became_active":"2024-03-18T20:25:22.678889+0000","last_became_peered":"2024-03-18T20:25:22.678889+0000","last_unstale":"2024-03-18T20:25:25.101012+0000","last_undegraded":"2024-03-18T20:25:25.101012+0000","last_fullsized":"2024-03-18T20:25:25.101012+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:00.365377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.113361+0000","last_change":"2024-03-18T20:25:34.113361+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.113361+0000","last_undegraded":"2024-03-18T20:25:34.113361+0000","last_fullsized":"2024-03-18T20:25:34.113361+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:10.985415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1f","version":"231'4","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100956+0000","last_change":"2024-03-18T20:25:22.251710+0000","last_active":"2024-03-18T20:25:25.100956+0000","last_peered":"2024-03-18T20:25:25.100956+0000","last_clean":"2024-03-18T20:25:25.100956+0000","last_became_active":"2024-03-18T20:25:22.251420+0000","last_became_peered":"2024-03-18T20:25:22.251420+0000","last_unstale":"2024-03-18T20:25:25.100956+0000","last_undegraded":"2024-03-18T20:25:25.100956+0000","last_fullsized":"2024-03-18T20:25:25.100956+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:00.435897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100904+0000","last_change":"2024-03-18T20:25:22.682272+0000","last_active":"2024-03-18T20:25:25.100904+0000","last_peered":"2024-03-18T20:25:25.100904+0000","last_clean":"2024-03-18T20:25:25.100904+0000","last_became_active":"2024-03-18T20:25:22.681785+0000","last_became_peered":"2024-03-18T20:25:22.681785+0000","last_unstale":"2024-03-18T20:25:25.100904+0000","last_undegraded":"2024-03-18T20:25:25.100904+0000","last_fullsized":"2024-03-18T20:25:25.100904+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:53:46.174877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.115690+0000","last_change":"2024-03-18T20:25:34.115690+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.115690+0000","last_undegraded":"2024-03-18T20:25:34.115690+0000","last_fullsized":"2024-03-18T20:25:34.115690+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:25.724222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100917+0000","last_change":"2024-03-18T20:25:22.679227+0000","last_active":"2024-03-18T20:25:25.100917+0000","last_peered":"2024-03-18T20:25:25.100917+0000","last_clean":"2024-03-18T20:25:25.100917+0000","last_became_active":"2024-03-18T20:25:22.678827+0000","last_became_peered":"2024-03-18T20:25:22.678827+0000","last_unstale":"2024-03-18T20:25:25.100917+0000","last_undegraded":"2024-03-18T20:25:25.100917+0000","last_fullsized":"2024-03-18T20:25:25.100917+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:05.394457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100961+0000","last_change":"2024-03-18T20:25:22.679640+0000","last_active":"2024-03-18T20:25:25.100961+0000","last_peered":"2024-03-18T20:25:25.100961+0000","last_clean":"2024-03-18T20:25:25.100961+0000","last_became_active":"2024-03-18T20:25:22.679408+0000","last_became_peered":"2024-03-18T20:25:22.679408+0000","last_unstale":"2024-03-18T20:25:25.100961+0000","last_undegraded":"2024-03-18T20:25:25.100961+0000","last_fullsized":"2024-03-18T20:25:25.100961+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:24:33.293594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100873+0000","last_change":"2024-03-18T20:25:22.685981+0000","last_active":"2024-03-18T20:25:25.100873+0000","last_peered":"2024-03-18T20:25:25.100873+0000","last_clean":"2024-03-18T20:25:25.100873+0000","last_became_active":"2024-03-18T20:25:22.685683+0000","last_became_peered":"2024-03-18T20:25:22.685683+0000","last_unstale":"2024-03-18T20:25:25.100873+0000","last_undegraded":"2024-03-18T20:25:25.100873+0000","last_fullsized":"2024-03-18T20:25:25.100873+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:36:57.458670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100679+0000","last_change":"2024-03-18T20:25:22.680503+0000","last_active":"2024-03-18T20:25:25.100679+0000","last_peered":"2024-03-18T20:25:25.100679+0000","last_clean":"2024-03-18T20:25:25.100679+0000","last_became_active":"2024-03-18T20:25:22.680258+0000","last_became_peered":"2024-03-18T20:25:22.680258+0000","last_unstale":"2024-03-18T20:25:25.100679+0000","last_undegraded":"2024-03-18T20:25:25.100679+0000","last_fullsized":"2024-03-18T20:25:25.100679+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:49.478525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101401+0000","last_change":"2024-03-18T20:25:22.688098+0000","last_active":"2024-03-18T20:25:25.101401+0000","last_peered":"2024-03-18T20:25:25.101401+0000","last_clean":"2024-03-18T20:25:25.101401+0000","last_became_active":"2024-03-18T20:25:22.687830+0000","last_became_peered":"2024-03-18T20:25:22.687830+0000","last_unstale":"2024-03-18T20:25:25.101401+0000","last_undegraded":"2024-03-18T20:25:25.101401+0000","last_fullsized":"2024-03-18T20:25:25.101401+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:08.456234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097677+0000","last_change":"2024-03-18T20:25:22.689262+0000","last_active":"2024-03-18T20:25:25.097677+0000","last_peered":"2024-03-18T20:25:25.097677+0000","last_clean":"2024-03-18T20:25:25.097677+0000","last_became_active":"2024-03-18T20:25:22.688913+0000","last_became_peered":"2024-03-18T20:25:22.688913+0000","last_unstale":"2024-03-18T20:25:25.097677+0000","last_undegraded":"2024-03-18T20:25:25.097677+0000","last_fullsized":"2024-03-18T20:25:25.097677+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:15:50.135885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101124+0000","last_change":"2024-03-18T20:25:22.687342+0000","last_active":"2024-03-18T20:25:25.101124+0000","last_peered":"2024-03-18T20:25:25.101124+0000","last_clean":"2024-03-18T20:25:25.101124+0000","last_became_active":"2024-03-18T20:25:22.687092+0000","last_became_peered":"2024-03-18T20:25:22.687092+0000","last_unstale":"2024-03-18T20:25:25.101124+0000","last_undegraded":"2024-03-18T20:25:25.101124+0000","last_fullsized":"2024-03-18T20:25:25.101124+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:00:33.383893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100833+0000","last_change":"2024-03-18T20:25:22.683777+0000","last_active":"2024-03-18T20:25:25.100833+0000","last_peered":"2024-03-18T20:25:25.100833+0000","last_clean":"2024-03-18T20:25:25.100833+0000","last_became_active":"2024-03-18T20:25:22.683484+0000","last_became_peered":"2024-03-18T20:25:22.683484+0000","last_unstale":"2024-03-18T20:25:25.100833+0000","last_undegraded":"2024-03-18T20:25:25.100833+0000","last_fullsized":"2024-03-18T20:25:25.100833+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:03:45.332668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101271+0000","last_change":"2024-03-18T20:25:22.684030+0000","last_active":"2024-03-18T20:25:25.101271+0000","last_peered":"2024-03-18T20:25:25.101271+0000","last_clean":"2024-03-18T20:25:25.101271+0000","last_became_active":"2024-03-18T20:25:22.683767+0000","last_became_peered":"2024-03-18T20:25:22.683767+0000","last_unstale":"2024-03-18T20:25:25.101271+0000","last_undegraded":"2024-03-18T20:25:25.101271+0000","last_fullsized":"2024-03-18T20:25:25.101271+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:30.204046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100925+0000","last_change":"2024-03-18T20:25:22.680418+0000","last_active":"2024-03-18T20:25:25.100925+0000","last_peered":"2024-03-18T20:25:25.100925+0000","last_clean":"2024-03-18T20:25:25.100925+0000","last_became_active":"2024-03-18T20:25:22.680149+0000","last_became_peered":"2024-03-18T20:25:22.680149+0000","last_unstale":"2024-03-18T20:25:25.100925+0000","last_undegraded":"2024-03-18T20:25:25.100925+0000","last_fullsized":"2024-03-18T20:25:25.100925+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:51:14.804124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099330+0000","last_change":"2024-03-18T20:25:22.691577+0000","last_active":"2024-03-18T20:25:25.099330+0000","last_peered":"2024-03-18T20:25:25.099330+0000","last_clean":"2024-03-18T20:25:25.099330+0000","last_became_active":"2024-03-18T20:25:22.691250+0000","last_became_peered":"2024-03-18T20:25:22.691250+0000","last_unstale":"2024-03-18T20:25:25.099330+0000","last_undegraded":"2024-03-18T20:25:25.099330+0000","last_fullsized":"2024-03-18T20:25:25.099330+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:55:21.946848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099355+0000","last_change":"2024-03-18T20:25:22.693670+0000","last_active":"2024-03-18T20:25:25.099355+0000","last_peered":"2024-03-18T20:25:25.099355+0000","last_clean":"2024-03-18T20:25:25.099355+0000","last_became_active":"2024-03-18T20:25:22.693178+0000","last_became_peered":"2024-03-18T20:25:22.693178+0000","last_unstale":"2024-03-18T20:25:25.099355+0000","last_undegraded":"2024-03-18T20:25:25.099355+0000","last_fullsized":"2024-03-18T20:25:25.099355+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:13.992732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099388+0000","last_change":"2024-03-18T20:25:22.693769+0000","last_active":"2024-03-18T20:25:25.099388+0000","last_peered":"2024-03-18T20:25:25.099388+0000","last_clean":"2024-03-18T20:25:25.099388+0000","last_became_active":"2024-03-18T20:25:22.691936+0000","last_became_peered":"2024-03-18T20:25:22.691936+0000","last_unstale":"2024-03-18T20:25:25.099388+0000","last_undegraded":"2024-03-18T20:25:25.099388+0000","last_fullsized":"2024-03-18T20:25:25.099388+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:29:49.401587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099414+0000","last_change":"2024-03-18T20:25:22.692857+0000","last_active":"2024-03-18T20:25:25.099414+0000","last_peered":"2024-03-18T20:25:25.099414+0000","last_clean":"2024-03-18T20:25:25.099414+0000","last_became_active":"2024-03-18T20:25:22.692615+0000","last_became_peered":"2024-03-18T20:25:22.692615+0000","last_unstale":"2024-03-18T20:25:25.099414+0000","last_undegraded":"2024-03-18T20:25:25.099414+0000","last_fullsized":"2024-03-18T20:25:25.099414+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:21:45.938034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100738+0000","last_change":"2024-03-18T20:25:22.683388+0000","last_active":"2024-03-18T20:25:25.100738+0000","last_peered":"2024-03-18T20:25:25.100738+0000","last_clean":"2024-03-18T20:25:25.100738+0000","last_became_active":"2024-03-18T20:25:22.683137+0000","last_became_peered":"2024-03-18T20:25:22.683137+0000","last_unstale":"2024-03-18T20:25:25.100738+0000","last_undegraded":"2024-03-18T20:25:25.100738+0000","last_fullsized":"2024-03-18T20:25:25.100738+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:02.506299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100813+0000","last_change":"2024-03-18T20:25:22.696148+0000","last_active":"2024-03-18T20:25:25.100813+0000","last_peered":"2024-03-18T20:25:25.100813+0000","last_clean":"2024-03-18T20:25:25.100813+0000","last_became_active":"2024-03-18T20:25:22.695928+0000","last_became_peered":"2024-03-18T20:25:22.695928+0000","last_unstale":"2024-03-18T20:25:25.100813+0000","last_undegraded":"2024-03-18T20:25:25.100813+0000","last_fullsized":"2024-03-18T20:25:25.100813+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:57.565171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097534+0000","last_change":"2024-03-18T20:25:22.681389+0000","last_active":"2024-03-18T20:25:25.097534+0000","last_peered":"2024-03-18T20:25:25.097534+0000","last_clean":"2024-03-18T20:25:25.097534+0000","last_became_active":"2024-03-18T20:25:22.681146+0000","last_became_peered":"2024-03-18T20:25:22.681146+0000","last_unstale":"2024-03-18T20:25:25.097534+0000","last_undegraded":"2024-03-18T20:25:25.097534+0000","last_fullsized":"2024-03-18T20:25:25.097534+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:05:57.372944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100832+0000","last_change":"2024-03-18T20:25:22.692497+0000","last_active":"2024-03-18T20:25:25.100832+0000","last_peered":"2024-03-18T20:25:25.100832+0000","last_clean":"2024-03-18T20:25:25.100832+0000","last_became_active":"2024-03-18T20:25:22.692192+0000","last_became_peered":"2024-03-18T20:25:22.692192+0000","last_unstale":"2024-03-18T20:25:25.100832+0000","last_undegraded":"2024-03-18T20:25:25.100832+0000","last_fullsized":"2024-03-18T20:25:25.100832+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:03:47.775258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099320+0000","last_change":"2024-03-18T20:25:22.690898+0000","last_active":"2024-03-18T20:25:25.099320+0000","last_peered":"2024-03-18T20:25:25.099320+0000","last_clean":"2024-03-18T20:25:25.099320+0000","last_became_active":"2024-03-18T20:25:22.690461+0000","last_became_peered":"2024-03-18T20:25:22.690461+0000","last_unstale":"2024-03-18T20:25:25.099320+0000","last_undegraded":"2024-03-18T20:25:25.099320+0000","last_fullsized":"2024-03-18T20:25:25.099320+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:15.208122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101023+0000","last_change":"2024-03-18T20:25:22.685172+0000","last_active":"2024-03-18T20:25:25.101023+0000","last_peered":"2024-03-18T20:25:25.101023+0000","last_clean":"2024-03-18T20:25:25.101023+0000","last_became_active":"2024-03-18T20:25:22.684855+0000","last_became_peered":"2024-03-18T20:25:22.684855+0000","last_unstale":"2024-03-18T20:25:25.101023+0000","last_undegraded":"2024-03-18T20:25:25.101023+0000","last_fullsized":"2024-03-18T20:25:25.101023+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:00.030804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101390+0000","last_change":"2024-03-18T20:25:22.693640+0000","last_active":"2024-03-18T20:25:25.101390+0000","last_peered":"2024-03-18T20:25:25.101390+0000","last_clean":"2024-03-18T20:25:25.101390+0000","last_became_active":"2024-03-18T20:25:22.693403+0000","last_became_peered":"2024-03-18T20:25:22.693403+0000","last_unstale":"2024-03-18T20:25:25.101390+0000","last_undegraded":"2024-03-18T20:25:25.101390+0000","last_fullsized":"2024-03-18T20:25:25.101390+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:58.151146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097657+0000","last_change":"2024-03-18T20:25:22.693546+0000","last_active":"2024-03-18T20:25:25.097657+0000","last_peered":"2024-03-18T20:25:25.097657+0000","last_clean":"2024-03-18T20:25:25.097657+0000","last_became_active":"2024-03-18T20:25:22.693267+0000","last_became_peered":"2024-03-18T20:25:22.693267+0000","last_unstale":"2024-03-18T20:25:25.097657+0000","last_undegraded":"2024-03-18T20:25:25.097657+0000","last_fullsized":"2024-03-18T20:25:25.097657+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:35.486064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099426+0000","last_change":"2024-03-18T20:25:22.693152+0000","last_active":"2024-03-18T20:25:25.099426+0000","last_peered":"2024-03-18T20:25:25.099426+0000","last_clean":"2024-03-18T20:25:25.099426+0000","last_became_active":"2024-03-18T20:25:22.686521+0000","last_became_peered":"2024-03-18T20:25:22.686521+0000","last_unstale":"2024-03-18T20:25:25.099426+0000","last_undegraded":"2024-03-18T20:25:25.099426+0000","last_fullsized":"2024-03-18T20:25:25.099426+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:40:18.302530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100797+0000","last_change":"2024-03-18T20:25:22.681784+0000","last_active":"2024-03-18T20:25:25.100797+0000","last_peered":"2024-03-18T20:25:25.100797+0000","last_clean":"2024-03-18T20:25:25.100797+0000","last_became_active":"2024-03-18T20:25:22.681198+0000","last_became_peered":"2024-03-18T20:25:22.681198+0000","last_unstale":"2024-03-18T20:25:25.100797+0000","last_undegraded":"2024-03-18T20:25:25.100797+0000","last_fullsized":"2024-03-18T20:25:25.100797+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:35.307035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099272+0000","last_change":"2024-03-18T20:25:22.685766+0000","last_active":"2024-03-18T20:25:25.099272+0000","last_peered":"2024-03-18T20:25:25.099272+0000","last_clean":"2024-03-18T20:25:25.099272+0000","last_became_active":"2024-03-18T20:25:22.683799+0000","last_became_peered":"2024-03-18T20:25:22.683799+0000","last_unstale":"2024-03-18T20:25:25.099272+0000","last_undegraded":"2024-03-18T20:25:25.099272+0000","last_fullsized":"2024-03-18T20:25:25.099272+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:42:25.581359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101317+0000","last_change":"2024-03-18T20:25:22.694269+0000","last_active":"2024-03-18T20:25:25.101317+0000","last_peered":"2024-03-18T20:25:25.101317+0000","last_clean":"2024-03-18T20:25:25.101317+0000","last_became_active":"2024-03-18T20:25:22.694014+0000","last_became_peered":"2024-03-18T20:25:22.694014+0000","last_unstale":"2024-03-18T20:25:25.101317+0000","last_undegraded":"2024-03-18T20:25:25.101317+0000","last_fullsized":"2024-03-18T20:25:25.101317+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:43:52.409254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101002+0000","last_change":"2024-03-18T20:25:22.681736+0000","last_active":"2024-03-18T20:25:25.101002+0000","last_peered":"2024-03-18T20:25:25.101002+0000","last_clean":"2024-03-18T20:25:25.101002+0000","last_became_active":"2024-03-18T20:25:22.679120+0000","last_became_peered":"2024-03-18T20:25:22.679120+0000","last_unstale":"2024-03-18T20:25:25.101002+0000","last_undegraded":"2024-03-18T20:25:25.101002+0000","last_fullsized":"2024-03-18T20:25:25.101002+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:00.365377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101337+0000","last_change":"2024-03-18T20:25:22.695525+0000","last_active":"2024-03-18T20:25:25.101337+0000","last_peered":"2024-03-18T20:25:25.101337+0000","last_clean":"2024-03-18T20:25:25.101337+0000","last_became_active":"2024-03-18T20:25:22.694079+0000","last_became_peered":"2024-03-18T20:25:22.694079+0000","last_unstale":"2024-03-18T20:25:25.101337+0000","last_undegraded":"2024-03-18T20:25:25.101337+0000","last_fullsized":"2024-03-18T20:25:25.101337+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:45:14.955608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101048+0000","last_change":"2024-03-18T20:25:22.683021+0000","last_active":"2024-03-18T20:25:25.101048+0000","last_peered":"2024-03-18T20:25:25.101048+0000","last_clean":"2024-03-18T20:25:25.101048+0000","last_became_active":"2024-03-18T20:25:22.682722+0000","last_became_peered":"2024-03-18T20:25:22.682722+0000","last_unstale":"2024-03-18T20:25:25.101048+0000","last_undegraded":"2024-03-18T20:25:25.101048+0000","last_fullsized":"2024-03-18T20:25:25.101048+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:27.207710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097638+0000","last_change":"2024-03-18T20:25:22.679426+0000","last_active":"2024-03-18T20:25:25.097638+0000","last_peered":"2024-03-18T20:25:25.097638+0000","last_clean":"2024-03-18T20:25:25.097638+0000","last_became_active":"2024-03-18T20:25:22.679159+0000","last_became_peered":"2024-03-18T20:25:22.679159+0000","last_unstale":"2024-03-18T20:25:25.097638+0000","last_undegraded":"2024-03-18T20:25:25.097638+0000","last_fullsized":"2024-03-18T20:25:25.097638+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:17:59.556803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101022+0000","last_change":"2024-03-18T20:25:22.682750+0000","last_active":"2024-03-18T20:25:25.101022+0000","last_peered":"2024-03-18T20:25:25.101022+0000","last_clean":"2024-03-18T20:25:25.101022+0000","last_became_active":"2024-03-18T20:25:22.682497+0000","last_became_peered":"2024-03-18T20:25:22.682497+0000","last_unstale":"2024-03-18T20:25:25.101022+0000","last_undegraded":"2024-03-18T20:25:25.101022+0000","last_fullsized":"2024-03-18T20:25:25.101022+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:33.092503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101174+0000","last_change":"2024-03-18T20:25:22.696207+0000","last_active":"2024-03-18T20:25:25.101174+0000","last_peered":"2024-03-18T20:25:25.101174+0000","last_clean":"2024-03-18T20:25:25.101174+0000","last_became_active":"2024-03-18T20:25:22.695943+0000","last_became_peered":"2024-03-18T20:25:22.695943+0000","last_unstale":"2024-03-18T20:25:25.101174+0000","last_undegraded":"2024-03-18T20:25:25.101174+0000","last_fullsized":"2024-03-18T20:25:25.101174+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:54:33.571448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3620,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575983,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52616,"kb_used_data":3452,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319224,"statfs":{"total":96636764160,"available":96582885376,"internally_reserved":0,"allocated":3534848,"data_stored":2093396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80700000000000005,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.48499999999999999,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.81899999999999995,"5min":1.093,"15min":1.093},"last":0.497}]},{"osd":1,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61799999999999999,"15min":0.61699999999999999},"min":{"1min":0.47599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.82999999999999996,"5min":0.99399999999999999,"15min":1.4299999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.60999999999999999,"15min":0.61899999999999999},"min":{"1min":0.47099999999999997,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.77500000000000002,"5min":1.0449999999999999,"15min":1.2629999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.46400000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84299999999999997,"5min":1.216,"15min":1.216},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.62,"15min":0.625},"min":{"1min":0.48399999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.81799999999999995,"5min":1.177,"15min":1.2490000000000001},"last":0.66000000000000003}]},{"osd":3,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.42599999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84699999999999998,"5min":0.93100000000000005,"15min":1.4450000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.5,"5min":0.44900000000000001,"15min":0.38600000000000001},"max":{"1min":0.84399999999999997,"5min":1.028,"15min":1.5029999999999999},"last":0.52200000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.47499999999999998,"15min":0.47699999999999998},"min":{"1min":0.33700000000000002,"5min":0.314,"15min":0.29899999999999999},"max":{"1min":0.68400000000000005,"5min":0.79100000000000004,"15min":0.86099999999999999},"last":0.439},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.35199999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71599999999999997,"5min":0.76600000000000001,"15min":0.80200000000000005},"last":0.53600000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51500000000000001,"15min":0.51600000000000001},"min":{"1min":0.34899999999999998,"5min":0.34399999999999997,"15min":0.316},"max":{"1min":0.68999999999999995,"5min":0.86599999999999999,"15min":0.89400000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.40699999999999997,"5min":0.377,"15min":0.36799999999999999},"max":{"1min":0.75600000000000001,"5min":0.81599999999999995,"15min":1.042},"last":0.64400000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56399999999999995,"15min":0.57699999999999996},"min":{"1min":0.39900000000000002,"5min":0.36199999999999999,"15min":0.34999999999999998},"max":{"1min":0.71099999999999997,"5min":0.85599999999999998,"15min":1.129},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.371,"5min":0.36299999999999999,"15min":0.34000000000000002},"max":{"1min":0.82899999999999996,"5min":0.85299999999999998,"15min":1.0609999999999999},"last":0.63600000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575983,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42180,"kb_used_data":2872,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329660,"statfs":{"total":96636764160,"available":96593571840,"internally_reserved":0,"allocated":2940928,"data_stored":1503540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57599999999999996,"15min":0.60099999999999998},"min":{"1min":0.41399999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72199999999999998,"5min":0.80900000000000005,"15min":1.1519999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59099999999999997,"15min":0.60799999999999998},"min":{"1min":0.33700000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.74299999999999999,"5min":0.81499999999999995,"15min":0.83299999999999996},"last":0.55000000000000004}]},{"osd":1,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57899999999999996,"15min":0.60199999999999998},"min":{"1min":0.39400000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.79100000000000004,"5min":0.79800000000000004,"15min":1.244},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58499999999999996,"15min":0.60399999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33200000000000002},"max":{"1min":0.77000000000000002,"5min":0.89200000000000002,"15min":1.286},"last":0.61299999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60999999999999999,"15min":0.63},"min":{"1min":0.47899999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71199999999999997,"5min":0.92000000000000004,"15min":1.276},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.59299999999999997,"15min":0.61099999999999999},"min":{"1min":0.434,"5min":0.40100000000000002,"15min":0.378},"max":{"1min":0.77500000000000002,"5min":0.95399999999999996,"15min":1.2669999999999999},"last":0.748}]},{"osd":3,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59699999999999998,"15min":0.622},"min":{"1min":0.38,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70699999999999996,"5min":0.83199999999999996,"15min":1.02},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60599999999999998,"15min":0.61899999999999999},"min":{"1min":0.45200000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82399999999999995,"5min":1.0049999999999999,"15min":1.337},"last":0.63800000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.48099999999999998,"15min":0.48299999999999998},"min":{"1min":0.308,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.61699999999999999,"5min":0.85799999999999998,"15min":0.88300000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.50900000000000001,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72099999999999997,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.65500000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.5,"15min":0.504},"min":{"1min":0.375,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.84799999999999998},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39000000000000001,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.73399999999999999,"5min":0.92600000000000005,"15min":0.94199999999999995},"last":0.72299999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40600000000000003,"5min":0.35799999999999998,"15min":0.34499999999999997},"max":{"1min":0.74299999999999999,"5min":0.95599999999999996,"15min":1.0429999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.44800000000000001,"5min":0.36599999999999999,"15min":0.34599999999999997},"max":{"1min":0.83899999999999997,"5min":1.028,"15min":1.028},"last":0.76300000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575983,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47924,"kb_used_data":2856,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323916,"statfs":{"total":96636764160,"available":96587689984,"internally_reserved":0,"allocated":2924544,"data_stored":1500136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.40400000000000003,"15min":0.39600000000000002},"min":{"1min":0.25600000000000001,"5min":0.246,"15min":0.23999999999999999},"max":{"1min":0.57999999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.41899999999999998,"15min":0.42999999999999999},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.58999999999999997,"5min":0.69699999999999995,"15min":0.72299999999999998},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.44,"15min":0.439},"min":{"1min":0.309,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61199999999999999,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48099999999999998,"15min":0.47799999999999998},"min":{"1min":0.35099999999999998,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.70099999999999996,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.377}]},{"osd":3,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.495,"15min":0.495},"min":{"1min":0.36199999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.64300000000000002,"5min":0.75900000000000001,"15min":0.76100000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.44800000000000001,"15min":0.442},"min":{"1min":0.32800000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68200000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.36199999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.68300000000000005,"15min":0.67300000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86799999999999999,"5min":1.0680000000000001,"15min":1.2909999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.68100000000000005,"15min":0.68000000000000005},"min":{"1min":0.50900000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.84499999999999997,"5min":0.92500000000000004,"15min":1.746},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.68100000000000005,"15min":0.67000000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.6910000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.71999999999999997,"15min":0.71399999999999997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.46899999999999997},"max":{"1min":0.89400000000000002,"5min":1.109,"15min":1.2549999999999999},"last":0.61699999999999999}]},{"osd":6,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.69699999999999995,"15min":0.68500000000000005},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.437},"max":{"1min":0.97399999999999998,"5min":0.98899999999999999,"15min":1.637},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71899999999999997,"15min":0.71199999999999997},"min":{"1min":0.49099999999999999,"5min":0.46999999999999997,"15min":0.46100000000000002},"max":{"1min":0.89500000000000002,"5min":1.014,"15min":1.722},"last":0.58799999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.70699999999999996,"15min":0.71299999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79800000000000004,"5min":1.02,"15min":1.2729999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.498,"5min":0.495,"15min":0.42799999999999999},"max":{"1min":0.90300000000000002,"5min":1.0329999999999999,"15min":1.3380000000000001},"last":0.63}]}]},{"osd":0,"up_from":13,"seq":55834575984,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30460,"kb_used_data":3440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341380,"statfs":{"total":96636764160,"available":96605573120,"internally_reserved":0,"allocated":3522560,"data_stored":2090594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.42399999999999999,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.374},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.42099999999999999,"15min":0.42799999999999999},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.46600000000000003,"5min":0.64700000000000002,"15min":1.1850000000000001},"last":0.28999999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.42599999999999999,"15min":0.435},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.47099999999999997,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.438,"15min":0.45300000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29799999999999999},"max":{"1min":0.59199999999999997,"5min":0.64300000000000002,"15min":0.70999999999999996},"last":0.33000000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.26100000000000001},"max":{"1min":0.45800000000000002,"5min":0.60099999999999998,"15min":0.628},"last":0.442},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48599999999999999,"15min":0.496},"min":{"1min":0.35499999999999998,"5min":0.35199999999999998,"15min":0.32500000000000001},"max":{"1min":0.52300000000000002,"5min":0.73999999999999999,"15min":0.91500000000000004},"last":0.34300000000000003}]},{"osd":4,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64300000000000002,"15min":0.65500000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81299999999999994,"5min":0.91500000000000004,"15min":1.0029999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.66200000000000003,"15min":0.66300000000000003},"min":{"1min":0.47499999999999998,"5min":0.47099999999999997,"15min":0.39700000000000002},"max":{"1min":0.78100000000000003,"5min":0.95299999999999996,"15min":1.0469999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.67800000000000005,"15min":0.67100000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89700000000000002,"5min":0.97099999999999997,"15min":1.0129999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91100000000000003,"5min":0.92900000000000005,"15min":1.518},"last":0.61899999999999999}]},{"osd":6,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.47399999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.78600000000000003,"5min":0.95899999999999996,"15min":1.1279999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71199999999999997,"15min":0.70899999999999996},"min":{"1min":0.498,"5min":0.48999999999999999,"15min":0.47899999999999998},"max":{"1min":0.94899999999999995,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.65300000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85099999999999998,"5min":0.96399999999999997,"15min":1.464},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.69899999999999995,"15min":0.70399999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.47599999999999998},"max":{"1min":0.75800000000000001,"5min":1.042,"15min":1.2350000000000001},"last":0.63300000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575984,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47928,"kb_used_data":2860,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323912,"statfs":{"total":96636764160,"available":96587685888,"internally_reserved":0,"allocated":2928640,"data_stored":1500226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36599999999999999,"15min":0.378},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.496,"5min":0.54800000000000004,"15min":0.59399999999999997},"last":0.318},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.47399999999999998,"15min":0.48699999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.63600000000000001,"5min":0.71499999999999997,"15min":0.753},"last":0.34599999999999997}]},{"osd":1,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46899999999999997,"15min":0.48199999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64300000000000002,"5min":0.72599999999999998,"15min":0.77100000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40899999999999997,"15min":0.41599999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.53800000000000003,"5min":0.59299999999999997,"15min":0.71099999999999997},"last":0.30399999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.433,"15min":0.436},"min":{"1min":0.28899999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.497,"15min":0.51700000000000002},"min":{"1min":0.33800000000000002,"5min":0.32200000000000001,"15min":0.313},"max":{"1min":0.68100000000000005,"5min":0.752,"15min":0.752},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.66300000000000003,"15min":0.65800000000000003},"min":{"1min":0.503,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":0.82499999999999996,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67500000000000004,"15min":0.66800000000000004},"min":{"1min":0.497,"5min":0.443,"15min":0.443},"max":{"1min":0.91600000000000004,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.67900000000000005,"15min":0.68500000000000005},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.93999999999999995,"5min":1.028,"15min":1.028},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67300000000000004,"15min":0.68000000000000005},"min":{"1min":0.47299999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.88300000000000001,"5min":0.95599999999999996,"15min":1.0629999999999999},"last":0.50600000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68000000000000005,"15min":0.69099999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86099999999999999,"5min":1.014,"15min":1.2050000000000001},"last":0.432},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.72499999999999998,"15min":0.73199999999999998},"min":{"1min":0.53800000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.97199999999999998,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.63200000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68899999999999995,"15min":0.70199999999999996},"min":{"1min":0.51500000000000001,"5min":0.442,"15min":0.42199999999999999},"max":{"1min":0.92900000000000005,"5min":0.99399999999999999,"15min":1.0469999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69899999999999995,"15min":0.70699999999999996},"min":{"1min":0.45500000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.93999999999999995,"5min":1.371,"15min":1.371},"last":0.59899999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575984,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52032,"kb_used_data":2868,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319808,"statfs":{"total":96636764160,"available":96583483392,"internally_reserved":0,"allocated":2936832,"data_stored":1500262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.38800000000000001,"15min":0.39900000000000002},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.53700000000000003,"5min":0.66500000000000004,"15min":0.98099999999999998},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.40999999999999998,"15min":0.41499999999999998},"min":{"1min":0.27700000000000002,"5min":0.25800000000000001,"15min":0.24199999999999999},"max":{"1min":0.434,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.29399999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39000000000000001,"15min":0.40999999999999998},"min":{"1min":0.30099999999999999,"5min":0.26100000000000001,"15min":0.254},"max":{"1min":0.45900000000000002,"5min":0.63200000000000001,"15min":0.64000000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.46100000000000002,"15min":0.46600000000000003},"min":{"1min":0.36699999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57499999999999996,"5min":0.65500000000000003,"15min":0.73099999999999998},"last":0.35399999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.45500000000000002,"15min":0.46600000000000003},"min":{"1min":0.34999999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53100000000000003,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.48199999999999998,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.51700000000000002,"5min":0.67000000000000004,"15min":0.73099999999999998},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.63600000000000001,"15min":0.65200000000000002},"min":{"1min":0.48799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81899999999999995,"5min":0.91700000000000004,"15min":1.097},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65200000000000002,"15min":0.67500000000000004},"min":{"1min":0.435,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78800000000000003,"5min":0.99299999999999999,"15min":1.0569999999999999},"last":0.56599999999999995}]},{"osd":5,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68400000000000005,"15min":0.69699999999999995},"min":{"1min":0.48999999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92100000000000004,"5min":0.995,"15min":1.1599999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.72499999999999998,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.82099999999999995,"5min":1.171,"15min":1.171},"last":0.54000000000000004}]},{"osd":6,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68500000000000005,"15min":0.70799999999999996},"min":{"1min":0.48099999999999998,"5min":0.436,"15min":0.42499999999999999},"max":{"1min":0.90500000000000003,"5min":1.012,"15min":1.194},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.71799999999999997,"15min":0.73299999999999998},"min":{"1min":0.53300000000000003,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":0.88200000000000001,"5min":1.0409999999999999,"15min":1.2410000000000001},"last":0.68000000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.70099999999999996,"15min":0.71899999999999997},"min":{"1min":0.49099999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82799999999999996,"5min":1.079,"15min":1.079},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68100000000000005,"15min":0.69999999999999996},"min":{"1min":0.44500000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83799999999999997,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.60899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575984,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64312,"kb_used_data":2860,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307528,"statfs":{"total":96636764160,"available":96570908672,"internally_reserved":0,"allocated":2928640,"data_stored":1500718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.57899999999999996,"15min":0.59299999999999997},"min":{"1min":0.34300000000000003,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65300000000000002,"5min":0.77500000000000002,"15min":0.81799999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57799999999999996,"15min":0.59799999999999998},"min":{"1min":0.40300000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69999999999999996,"5min":0.79600000000000004,"15min":0.84199999999999997},"last":0.58599999999999997}]},{"osd":1,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59999999999999998,"15min":0.61399999999999999},"min":{"1min":0.438,"5min":0.318,"15min":0.318},"max":{"1min":0.73199999999999998,"5min":0.81899999999999995,"15min":0.88300000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56999999999999995,"15min":0.58499999999999996},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.71199999999999997,"5min":0.79200000000000004,"15min":0.96499999999999997},"last":0.71199999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58299999999999996,"15min":0.59899999999999998},"min":{"1min":0.439,"5min":0.39500000000000002,"15min":0.39200000000000002},"max":{"1min":0.68600000000000005,"5min":0.80500000000000005,"15min":0.82599999999999996},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61099999999999999,"15min":0.61799999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.38500000000000001},"max":{"1min":0.81399999999999995,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.40799999999999997,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.86199999999999999},"last":0.52600000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.498,"15min":0.504},"min":{"1min":0.35199999999999998,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.85499999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.442}]},{"osd":6,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.499,"15min":0.505},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.46700000000000003}]},{"osd":7,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54100000000000004,"15min":0.54800000000000004},"min":{"1min":0.435,"5min":0.38,"15min":0.35699999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.57199999999999995,"15min":0.57999999999999996},"min":{"1min":0.41899999999999998,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.65100000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575983,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47932,"kb_used_data":2864,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323908,"statfs":{"total":96636764160,"available":96587681792,"internally_reserved":0,"allocated":2932736,"data_stored":1500192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42499999999999999,"5min":0.33900000000000002,"15min":0.31900000000000001},"max":{"1min":0.72699999999999998,"5min":0.85999999999999999,"15min":0.96499999999999997},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57799999999999996,"15min":0.60399999999999998},"min":{"1min":0.432,"5min":0.37,"15min":0.33100000000000002},"max":{"1min":0.69799999999999995,"5min":0.82599999999999996,"15min":1.3100000000000001},"last":0.55500000000000005}]},{"osd":1,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58499999999999996,"15min":0.60199999999999998},"min":{"1min":0.36899999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.74399999999999999,"5min":0.79600000000000004,"15min":1.056},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58399999999999996,"15min":0.60199999999999998},"min":{"1min":0.46600000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.63100000000000001,"5min":0.89800000000000002,"15min":1.018},"last":0.54000000000000004}]},{"osd":2,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.38700000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.84299999999999997,"15min":0.99299999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59399999999999997,"15min":0.60899999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.67500000000000004,"5min":0.875,"15min":1.1120000000000001},"last":0.52300000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.42799999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.78100000000000003,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63,"15min":0.64600000000000002},"min":{"1min":0.443,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69899999999999995,"5min":0.82399999999999995,"15min":1.161},"last":0.51000000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.47899999999999998,"15min":0.47499999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.77500000000000002},"last":0.432},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51200000000000001,"15min":0.50600000000000001},"min":{"1min":0.36199999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.89700000000000002},"last":0.39300000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.503,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.34000000000000002,"15min":0.32800000000000001},"max":{"1min":0.69999999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.40300000000000002,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.86599999999999999},"last":0.36699999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.40799999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.94499999999999995},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.45300000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74099999999999999,"5min":0.85999999999999999,"15min":0.91200000000000003},"last":0.41699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:25:40.140 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:25:40.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.288+0000 7f576538f700 1 Processor -- start 2024-03-18T20:25:40.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.288+0000 7f576538f700 1 -- start start 2024-03-18T20:25:40.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.289+0000 7f576538f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f576006a750 0x7f57600cf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.289+0000 7f576538f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f57600cf960 0x7f576006ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.289+0000 7f576538f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576006d640 con 0x7f57600cc6d0 2024-03-18T20:25:40.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.289+0000 7f576538f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f576006de50 con 0x7f57600cf960 2024-03-18T20:25:40.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.289+0000 7f576538f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57600acd80 con 0x7f576006a750 2024-03-18T20:25:40.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.289+0000 7f575e7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f57600cf960 0x7f576006ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:40.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.289+0000 7f575f7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f57600cc6d0 0x7f57600ccb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51524/0 (socket says 172.21.15.1:51524) 2024-03-18T20:25:40.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.289+0000 7f575f7fe700 1 -- 172.21.15.1:0/675140489 learned_addr learned my addr 172.21.15.1:0/675140489 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:40.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.289+0000 7f575effd700 1 --2- 172.21.15.1:0/675140489 >> v2:172.21.15.165:3300/0 conn(0x7f576006a750 0x7f57600cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:40.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.290+0000 7f575dffb700 1 -- 172.21.15.1:0/675140489 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352409053 0 0) 0x7f576006d640 con 0x7f57600cc6d0 2024-03-18T20:25:40.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.290+0000 7f575dffb700 1 -- 172.21.15.1:0/675140489 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f574c0035b0 con 0x7f57600cc6d0 2024-03-18T20:25:40.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.290+0000 7f575e7fc700 1 -- 172.21.15.1:0/675140489 >> v2:172.21.15.165:3300/0 conn(0x7f576006a750 msgr2=0x7f57600cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:40.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.290+0000 7f575e7fc700 1 --2- 172.21.15.1:0/675140489 >> v2:172.21.15.165:3300/0 conn(0x7f576006a750 0x7f57600cf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.290+0000 7f575e7fc700 1 -- 172.21.15.1:0/675140489 >> v1:172.21.15.1:6789/0 conn(0x7f57600cc6d0 legacy=0x7f57600ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.290+0000 7f575e7fc700 1 -- 172.21.15.1:0/675140489 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576006b070 con 0x7f57600cf960 2024-03-18T20:25:40.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.290+0000 7f575e7fc700 1 --2- 172.21.15.1:0/675140489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57600cf960 0x7f576006ab30 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f576006c800 tx=0x7f5748009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e36f13701511a9cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:40.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.290+0000 7f575dffb700 1 -- 172.21.15.1:0/675140489 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748011830 con 0x7f57600cf960 2024-03-18T20:25:40.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.290+0000 7f575dffb700 1 -- 172.21.15.1:0/675140489 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5748011990 con 0x7f57600cf960 2024-03-18T20:25:40.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.290+0000 7f575dffb700 1 -- 172.21.15.1:0/675140489 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748011d20 con 0x7f57600cf960 2024-03-18T20:25:40.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.291+0000 7f576538f700 1 -- 172.21.15.1:0/675140489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57600cf960 msgr2=0x7f576006ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.291+0000 7f576538f700 1 --2- 172.21.15.1:0/675140489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57600cf960 0x7f576006ab30 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f576006c800 tx=0x7f5748009b20 comp rx=0 tx=0).stop 2024-03-18T20:25:40.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.291+0000 7f576538f700 1 -- 172.21.15.1:0/675140489 shutdown_connections 2024-03-18T20:25:40.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.291+0000 7f576538f700 1 --2- 172.21.15.1:0/675140489 >> v2:172.21.15.165:3300/0 conn(0x7f576006a750 0x7f57600cf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.291+0000 7f576538f700 1 --2- 172.21.15.1:0/675140489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57600cf960 0x7f576006ab30 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.291+0000 7f576538f700 1 -- 172.21.15.1:0/675140489 >> 172.21.15.1:0/675140489 conn(0x7f576006b860 msgr2=0x7f576006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:40.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.291+0000 7f576538f700 1 -- 172.21.15.1:0/675140489 shutdown_connections 2024-03-18T20:25:40.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.291+0000 7f576538f700 1 -- 172.21.15.1:0/675140489 wait complete. 2024-03-18T20:25:40.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.292+0000 7f576538f700 1 Processor -- start 2024-03-18T20:25:40.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.292+0000 7f576538f700 1 -- start start 2024-03-18T20:25:40.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.293+0000 7f576538f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f576006a750 0x7f57600b4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.293+0000 7f576538f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57600cc6d0 0x7f576014b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.293+0000 7f576538f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57600cf960 0x7f5760159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.293+0000 7f576538f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57601f70f0 con 0x7f576006a750 2024-03-18T20:25:40.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.293+0000 7f576538f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57601f64e0 con 0x7f57600cf960 2024-03-18T20:25:40.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.293+0000 7f576538f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57601f67c0 con 0x7f57600cc6d0 2024-03-18T20:25:40.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.293+0000 7f575effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f576006a750 0x7f57600b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:40.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.293+0000 7f575effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f576006a750 0x7f57600b4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49874/0 (socket says 172.21.15.1:49874) 2024-03-18T20:25:40.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.293+0000 7f575effd700 1 -- 172.21.15.1:0/3717057337 learned_addr learned my addr 172.21.15.1:0/3717057337 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:40.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.293+0000 7f575effd700 1 -- 172.21.15.1:0/3717057337 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57600cc6d0 msgr2=0x7f576014b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.293+0000 7f575effd700 1 --2- 172.21.15.1:0/3717057337 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57600cc6d0 0x7f576014b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.293+0000 7f575f7fe700 1 --2- 172.21.15.1:0/3717057337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57600cf960 0x7f5760159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:40.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.293+0000 7f575effd700 1 -- 172.21.15.1:0/3717057337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57600cf960 msgr2=0x7f5760159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.293+0000 7f575effd700 1 --2- 172.21.15.1:0/3717057337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57600cf960 0x7f5760159d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.293+0000 7f575effd700 1 -- 172.21.15.1:0/3717057337 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57480094a0 con 0x7f576006a750 2024-03-18T20:25:40.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.293+0000 7f575f7fe700 1 --2- 172.21.15.1:0/3717057337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57600cf960 0x7f5760159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:40.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.293+0000 7f575effd700 1 --2- 172.21.15.1:0/3717057337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f576006a750 0x7f57600b4880 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f575000e870 tx=0x7f575000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:40.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.293+0000 7f57577fe700 1 -- 172.21.15.1:0/3717057337 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5750009590 con 0x7f576006a750 2024-03-18T20:25:40.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.294+0000 7f57577fe700 1 -- 172.21.15.1:0/3717057337 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575000ee70 con 0x7f576006a750 2024-03-18T20:25:40.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.294+0000 7f576538f700 1 -- 172.21.15.1:0/3717057337 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57601f6aa0 con 0x7f576006a750 2024-03-18T20:25:40.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.294+0000 7f57577fe700 1 -- 172.21.15.1:0/3717057337 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575000ca50 con 0x7f576006a750 2024-03-18T20:25:40.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.294+0000 7f576538f700 1 -- 172.21.15.1:0/3717057337 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57600b4dc0 con 0x7f576006a750 2024-03-18T20:25:40.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.294+0000 7f576538f700 1 -- 172.21.15.1:0/3717057337 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57600c95f0 con 0x7f576006a750 2024-03-18T20:25:40.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.295+0000 7f57577fe700 1 -- 172.21.15.1:0/3717057337 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f575000cbb0 con 0x7f576006a750 2024-03-18T20:25:40.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.296+0000 7f57577fe700 1 --2- 172.21.15.1:0/3717057337 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f574006adb0 0x7f574006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.296+0000 7f575e7fc700 1 --2- 172.21.15.1:0/3717057337 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f574006adb0 0x7f574006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:40.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.296+0000 7f57577fe700 1 -- 172.21.15.1:0/3717057337 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f5750089920 con 0x7f576006a750 2024-03-18T20:25:40.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.296+0000 7f575e7fc700 1 --2- 172.21.15.1:0/3717057337 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f574006adb0 0x7f574006d270 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f576006c800 tx=0x7f5748010000 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:40.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.300+0000 7f57577fe700 1 -- 172.21.15.1:0/3717057337 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5750054570 con 0x7f576006a750 2024-03-18T20:25:40.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.497+0000 7f576538f700 1 -- 172.21.15.1:0/3717057337 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f57600cd880 con 0x7f574006adb0 2024-03-18T20:25:40.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.511+0000 7f57577fe700 1 -- 172.21.15.1:0/3717057337 <== mgr.11322 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+518107 (secure 0 0 0) 0x7f574801f070 con 0x7f574006adb0 2024-03-18T20:25:40.523 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.515+0000 7f576538f700 1 -- 172.21.15.1:0/3717057337 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f574006adb0 msgr2=0x7f574006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.515+0000 7f576538f700 1 --2- 172.21.15.1:0/3717057337 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f574006adb0 0x7f574006d270 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f576006c800 tx=0x7f5748010000 comp rx=0 tx=0).stop 2024-03-18T20:25:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.515+0000 7f576538f700 1 -- 172.21.15.1:0/3717057337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f576006a750 msgr2=0x7f57600b4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.515+0000 7f576538f700 1 --2- 172.21.15.1:0/3717057337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f576006a750 0x7f57600b4880 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f575000e870 tx=0x7f575000ec30 comp rx=0 tx=0).stop 2024-03-18T20:25:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.515+0000 7f576538f700 1 -- 172.21.15.1:0/3717057337 shutdown_connections 2024-03-18T20:25:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.515+0000 7f576538f700 1 --2- 172.21.15.1:0/3717057337 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f574006adb0 0x7f574006d270 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.515+0000 7f576538f700 1 --2- 172.21.15.1:0/3717057337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57600cf960 0x7f5760159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.515+0000 7f576538f700 1 --2- 172.21.15.1:0/3717057337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f576006a750 0x7f57600b4880 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.515+0000 7f576538f700 1 --2- 172.21.15.1:0/3717057337 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57600cc6d0 0x7f576014b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.515+0000 7f576538f700 1 -- 172.21.15.1:0/3717057337 >> 172.21.15.1:0/3717057337 conn(0x7f576006b860 msgr2=0x7f576006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.515+0000 7f576538f700 1 -- 172.21.15.1:0/3717057337 shutdown_connections 2024-03-18T20:25:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.516+0000 7f576538f700 1 -- 172.21.15.1:0/3717057337 wait complete. 2024-03-18T20:25:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:25:40.531 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-18T20:25:39.102708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593988,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":626,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":389,"ondisk_log_size":389,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":212,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":385384,"kb_used_data":24072,"kb_used_omap":12,"kb_used_meta":361203,"kb_avail":754589336,"statfs":{"total":773094113280,"available":772699480064,"internally_reserved":0,"allocated":24649728,"data_stored":13189064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":369872441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":815,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.995476"},"pg_stats":[{"pgid":"1.0","version":"221'348","reported_seq":1201,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:24.605524+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:25:24.605524+0000","last_peered":"2024-03-18T20:25:24.605524+0000","last_clean":"2024-03-18T20:25:24.605524+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:25:24.605524+0000","last_undegraded":"2024-03-18T20:25:24.605524+0000","last_fullsized":"2024-03-18T20:25:24.605524+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"231'1","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100643+0000","last_change":"2024-03-18T20:25:22.248248+0000","last_active":"2024-03-18T20:25:25.100643+0000","last_peered":"2024-03-18T20:25:25.100643+0000","last_clean":"2024-03-18T20:25:25.100643+0000","last_became_active":"2024-03-18T20:25:22.247970+0000","last_became_peered":"2024-03-18T20:25:22.247970+0000","last_unstale":"2024-03-18T20:25:25.100643+0000","last_undegraded":"2024-03-18T20:25:25.100643+0000","last_fullsized":"2024-03-18T20:25:25.100643+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:01:50.523530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101112+0000","last_change":"2024-03-18T20:25:22.687598+0000","last_active":"2024-03-18T20:25:25.101112+0000","last_peered":"2024-03-18T20:25:25.101112+0000","last_clean":"2024-03-18T20:25:25.101112+0000","last_became_active":"2024-03-18T20:25:22.686962+0000","last_became_peered":"2024-03-18T20:25:22.686962+0000","last_unstale":"2024-03-18T20:25:25.101112+0000","last_undegraded":"2024-03-18T20:25:25.101112+0000","last_fullsized":"2024-03-18T20:25:25.101112+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:23:10.983253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.114498+0000","last_change":"2024-03-18T20:25:34.114498+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.114498+0000","last_undegraded":"2024-03-18T20:25:34.114498+0000","last_fullsized":"2024-03-18T20:25:34.114498+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:47.366224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101303+0000","last_change":"2024-03-18T20:25:22.252803+0000","last_active":"2024-03-18T20:25:25.101303+0000","last_peered":"2024-03-18T20:25:25.101303+0000","last_clean":"2024-03-18T20:25:25.101303+0000","last_became_active":"2024-03-18T20:25:22.252169+0000","last_became_peered":"2024-03-18T20:25:22.252169+0000","last_unstale":"2024-03-18T20:25:25.101303+0000","last_undegraded":"2024-03-18T20:25:25.101303+0000","last_fullsized":"2024-03-18T20:25:25.101303+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:12.985428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097715+0000","last_change":"2024-03-18T20:25:22.696025+0000","last_active":"2024-03-18T20:25:25.097715+0000","last_peered":"2024-03-18T20:25:25.097715+0000","last_clean":"2024-03-18T20:25:25.097715+0000","last_became_active":"2024-03-18T20:25:22.695759+0000","last_became_peered":"2024-03-18T20:25:22.695759+0000","last_unstale":"2024-03-18T20:25:25.097715+0000","last_undegraded":"2024-03-18T20:25:25.097715+0000","last_fullsized":"2024-03-18T20:25:25.097715+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:10.681789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.117162+0000","last_change":"2024-03-18T20:25:34.117162+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.117162+0000","last_undegraded":"2024-03-18T20:25:34.117162+0000","last_fullsized":"2024-03-18T20:25:34.117162+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:13.759015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101033+0000","last_change":"2024-03-18T20:25:22.252889+0000","last_active":"2024-03-18T20:25:25.101033+0000","last_peered":"2024-03-18T20:25:25.101033+0000","last_clean":"2024-03-18T20:25:25.101033+0000","last_became_active":"2024-03-18T20:25:22.252096+0000","last_became_peered":"2024-03-18T20:25:22.252096+0000","last_unstale":"2024-03-18T20:25:25.101033+0000","last_undegraded":"2024-03-18T20:25:25.101033+0000","last_fullsized":"2024-03-18T20:25:25.101033+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:18.568149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101094+0000","last_change":"2024-03-18T20:25:22.689532+0000","last_active":"2024-03-18T20:25:25.101094+0000","last_peered":"2024-03-18T20:25:25.101094+0000","last_clean":"2024-03-18T20:25:25.101094+0000","last_became_active":"2024-03-18T20:25:22.689210+0000","last_became_peered":"2024-03-18T20:25:22.689210+0000","last_unstale":"2024-03-18T20:25:25.101094+0000","last_undegraded":"2024-03-18T20:25:25.101094+0000","last_fullsized":"2024-03-18T20:25:25.101094+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:21.229791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.111213+0000","last_change":"2024-03-18T20:25:34.111213+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.111213+0000","last_undegraded":"2024-03-18T20:25:34.111213+0000","last_fullsized":"2024-03-18T20:25:34.111213+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:17.295248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100663+0000","last_change":"2024-03-18T20:25:22.254053+0000","last_active":"2024-03-18T20:25:25.100663+0000","last_peered":"2024-03-18T20:25:25.100663+0000","last_clean":"2024-03-18T20:25:25.100663+0000","last_became_active":"2024-03-18T20:25:22.253747+0000","last_became_peered":"2024-03-18T20:25:22.253747+0000","last_unstale":"2024-03-18T20:25:25.100663+0000","last_undegraded":"2024-03-18T20:25:25.100663+0000","last_fullsized":"2024-03-18T20:25:25.100663+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:32.777143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101083+0000","last_change":"2024-03-18T20:25:22.685300+0000","last_active":"2024-03-18T20:25:25.101083+0000","last_peered":"2024-03-18T20:25:25.101083+0000","last_clean":"2024-03-18T20:25:25.101083+0000","last_became_active":"2024-03-18T20:25:22.684869+0000","last_became_peered":"2024-03-18T20:25:22.684869+0000","last_unstale":"2024-03-18T20:25:25.101083+0000","last_undegraded":"2024-03-18T20:25:25.101083+0000","last_fullsized":"2024-03-18T20:25:25.101083+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:02.037780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.110932+0000","last_change":"2024-03-18T20:25:34.110932+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.110932+0000","last_undegraded":"2024-03-18T20:25:34.110932+0000","last_fullsized":"2024-03-18T20:25:34.110932+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:44:27.205313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099377+0000","last_change":"2024-03-18T20:25:22.245633+0000","last_active":"2024-03-18T20:25:25.099377+0000","last_peered":"2024-03-18T20:25:25.099377+0000","last_clean":"2024-03-18T20:25:25.099377+0000","last_became_active":"2024-03-18T20:25:22.245372+0000","last_became_peered":"2024-03-18T20:25:22.245372+0000","last_unstale":"2024-03-18T20:25:25.099377+0000","last_undegraded":"2024-03-18T20:25:25.099377+0000","last_fullsized":"2024-03-18T20:25:25.099377+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:28:25.048682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099401+0000","last_change":"2024-03-18T20:25:22.683574+0000","last_active":"2024-03-18T20:25:25.099401+0000","last_peered":"2024-03-18T20:25:25.099401+0000","last_clean":"2024-03-18T20:25:25.099401+0000","last_became_active":"2024-03-18T20:25:22.683015+0000","last_became_peered":"2024-03-18T20:25:22.683015+0000","last_unstale":"2024-03-18T20:25:25.099401+0000","last_undegraded":"2024-03-18T20:25:25.099401+0000","last_fullsized":"2024-03-18T20:25:25.099401+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:11.606826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.116497+0000","last_change":"2024-03-18T20:25:34.116497+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.116497+0000","last_undegraded":"2024-03-18T20:25:34.116497+0000","last_fullsized":"2024-03-18T20:25:34.116497+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:41.670198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101098+0000","last_change":"2024-03-18T20:25:22.255435+0000","last_active":"2024-03-18T20:25:25.101098+0000","last_peered":"2024-03-18T20:25:25.101098+0000","last_clean":"2024-03-18T20:25:25.101098+0000","last_became_active":"2024-03-18T20:25:22.255149+0000","last_became_peered":"2024-03-18T20:25:22.255149+0000","last_unstale":"2024-03-18T20:25:25.101098+0000","last_undegraded":"2024-03-18T20:25:25.101098+0000","last_fullsized":"2024-03-18T20:25:25.101098+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:58.960198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101079+0000","last_change":"2024-03-18T20:25:22.682910+0000","last_active":"2024-03-18T20:25:25.101079+0000","last_peered":"2024-03-18T20:25:25.101079+0000","last_clean":"2024-03-18T20:25:25.101079+0000","last_became_active":"2024-03-18T20:25:22.682667+0000","last_became_peered":"2024-03-18T20:25:22.682667+0000","last_unstale":"2024-03-18T20:25:25.101079+0000","last_undegraded":"2024-03-18T20:25:25.101079+0000","last_fullsized":"2024-03-18T20:25:25.101079+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:02.037780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.115535+0000","last_change":"2024-03-18T20:25:34.115535+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.115535+0000","last_undegraded":"2024-03-18T20:25:34.115535+0000","last_fullsized":"2024-03-18T20:25:34.115535+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:56.513767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100915+0000","last_change":"2024-03-18T20:25:22.250457+0000","last_active":"2024-03-18T20:25:25.100915+0000","last_peered":"2024-03-18T20:25:25.100915+0000","last_clean":"2024-03-18T20:25:25.100915+0000","last_became_active":"2024-03-18T20:25:22.250157+0000","last_became_peered":"2024-03-18T20:25:22.250157+0000","last_unstale":"2024-03-18T20:25:25.100915+0000","last_undegraded":"2024-03-18T20:25:25.100915+0000","last_fullsized":"2024-03-18T20:25:25.100915+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:18.568149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100623+0000","last_change":"2024-03-18T20:25:22.691245+0000","last_active":"2024-03-18T20:25:25.100623+0000","last_peered":"2024-03-18T20:25:25.100623+0000","last_clean":"2024-03-18T20:25:25.100623+0000","last_became_active":"2024-03-18T20:25:22.690988+0000","last_became_peered":"2024-03-18T20:25:22.690988+0000","last_unstale":"2024-03-18T20:25:25.100623+0000","last_undegraded":"2024-03-18T20:25:25.100623+0000","last_fullsized":"2024-03-18T20:25:25.100623+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:32:54.125659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.112128+0000","last_change":"2024-03-18T20:25:34.112128+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.112128+0000","last_undegraded":"2024-03-18T20:25:34.112128+0000","last_fullsized":"2024-03-18T20:25:34.112128+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:25:32.071310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.6","version":"232'1","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:27.359403+0000","last_change":"2024-03-18T20:25:22.247183+0000","last_active":"2024-03-18T20:25:27.359403+0000","last_peered":"2024-03-18T20:25:27.359403+0000","last_clean":"2024-03-18T20:25:27.359403+0000","last_became_active":"2024-03-18T20:25:22.246890+0000","last_became_peered":"2024-03-18T20:25:22.246890+0000","last_unstale":"2024-03-18T20:25:27.359403+0000","last_undegraded":"2024-03-18T20:25:27.359403+0000","last_fullsized":"2024-03-18T20:25:27.359403+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:40:21.886849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097599+0000","last_change":"2024-03-18T20:25:22.689011+0000","last_active":"2024-03-18T20:25:25.097599+0000","last_peered":"2024-03-18T20:25:25.097599+0000","last_clean":"2024-03-18T20:25:25.097599+0000","last_became_active":"2024-03-18T20:25:22.688750+0000","last_became_peered":"2024-03-18T20:25:22.688750+0000","last_unstale":"2024-03-18T20:25:25.097599+0000","last_undegraded":"2024-03-18T20:25:25.097599+0000","last_fullsized":"2024-03-18T20:25:25.097599+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:57.345369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.111764+0000","last_change":"2024-03-18T20:25:34.111764+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.111764+0000","last_undegraded":"2024-03-18T20:25:34.111764+0000","last_fullsized":"2024-03-18T20:25:34.111764+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:25.724222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.7","version":"231'2","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100865+0000","last_change":"2024-03-18T20:25:22.254151+0000","last_active":"2024-03-18T20:25:25.100865+0000","last_peered":"2024-03-18T20:25:25.100865+0000","last_clean":"2024-03-18T20:25:25.100865+0000","last_became_active":"2024-03-18T20:25:22.253873+0000","last_became_peered":"2024-03-18T20:25:22.253873+0000","last_unstale":"2024-03-18T20:25:25.100865+0000","last_undegraded":"2024-03-18T20:25:25.100865+0000","last_fullsized":"2024-03-18T20:25:25.100865+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:26:58.327699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099308+0000","last_change":"2024-03-18T20:25:22.693170+0000","last_active":"2024-03-18T20:25:25.099308+0000","last_peered":"2024-03-18T20:25:25.099308+0000","last_clean":"2024-03-18T20:25:25.099308+0000","last_became_active":"2024-03-18T20:25:22.691592+0000","last_became_peered":"2024-03-18T20:25:22.691592+0000","last_unstale":"2024-03-18T20:25:25.099308+0000","last_undegraded":"2024-03-18T20:25:25.099308+0000","last_fullsized":"2024-03-18T20:25:25.099308+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:23.897085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.114953+0000","last_change":"2024-03-18T20:25:34.114953+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.114953+0000","last_undegraded":"2024-03-18T20:25:34.114953+0000","last_fullsized":"2024-03-18T20:25:34.114953+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:06:09.253104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100981+0000","last_change":"2024-03-18T20:25:22.251509+0000","last_active":"2024-03-18T20:25:25.100981+0000","last_peered":"2024-03-18T20:25:25.100981+0000","last_clean":"2024-03-18T20:25:25.100981+0000","last_became_active":"2024-03-18T20:25:22.251240+0000","last_became_peered":"2024-03-18T20:25:22.251240+0000","last_unstale":"2024-03-18T20:25:25.100981+0000","last_undegraded":"2024-03-18T20:25:25.100981+0000","last_fullsized":"2024-03-18T20:25:25.100981+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:13:14.870270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099451+0000","last_change":"2024-03-18T20:25:22.695653+0000","last_active":"2024-03-18T20:25:25.099451+0000","last_peered":"2024-03-18T20:25:25.099451+0000","last_clean":"2024-03-18T20:25:25.099451+0000","last_became_active":"2024-03-18T20:25:22.694092+0000","last_became_peered":"2024-03-18T20:25:22.694092+0000","last_unstale":"2024-03-18T20:25:25.099451+0000","last_undegraded":"2024-03-18T20:25:25.099451+0000","last_fullsized":"2024-03-18T20:25:25.099451+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:20.319345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.115435+0000","last_change":"2024-03-18T20:25:34.115435+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.115435+0000","last_undegraded":"2024-03-18T20:25:34.115435+0000","last_fullsized":"2024-03-18T20:25:34.115435+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:32:42.191508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100853+0000","last_change":"2024-03-18T20:25:22.251818+0000","last_active":"2024-03-18T20:25:25.100853+0000","last_peered":"2024-03-18T20:25:25.100853+0000","last_clean":"2024-03-18T20:25:25.100853+0000","last_became_active":"2024-03-18T20:25:22.251407+0000","last_became_peered":"2024-03-18T20:25:22.251407+0000","last_unstale":"2024-03-18T20:25:25.100853+0000","last_undegraded":"2024-03-18T20:25:25.100853+0000","last_fullsized":"2024-03-18T20:25:25.100853+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:29.196742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100780+0000","last_change":"2024-03-18T20:25:22.684929+0000","last_active":"2024-03-18T20:25:25.100780+0000","last_peered":"2024-03-18T20:25:25.100780+0000","last_clean":"2024-03-18T20:25:25.100780+0000","last_became_active":"2024-03-18T20:25:22.684658+0000","last_became_peered":"2024-03-18T20:25:22.684658+0000","last_unstale":"2024-03-18T20:25:25.100780+0000","last_undegraded":"2024-03-18T20:25:25.100780+0000","last_fullsized":"2024-03-18T20:25:25.100780+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:39.733357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.113347+0000","last_change":"2024-03-18T20:25:34.113347+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.113347+0000","last_undegraded":"2024-03-18T20:25:34.113347+0000","last_fullsized":"2024-03-18T20:25:34.113347+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:22:13.432979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100944+0000","last_change":"2024-03-18T20:25:22.247489+0000","last_active":"2024-03-18T20:25:25.100944+0000","last_peered":"2024-03-18T20:25:25.100944+0000","last_clean":"2024-03-18T20:25:25.100944+0000","last_became_active":"2024-03-18T20:25:22.247203+0000","last_became_peered":"2024-03-18T20:25:22.247203+0000","last_unstale":"2024-03-18T20:25:25.100944+0000","last_undegraded":"2024-03-18T20:25:25.100944+0000","last_fullsized":"2024-03-18T20:25:25.100944+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:32.481118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101073+0000","last_change":"2024-03-18T20:25:22.681615+0000","last_active":"2024-03-18T20:25:25.101073+0000","last_peered":"2024-03-18T20:25:25.101073+0000","last_clean":"2024-03-18T20:25:25.101073+0000","last_became_active":"2024-03-18T20:25:22.681261+0000","last_became_peered":"2024-03-18T20:25:22.681261+0000","last_unstale":"2024-03-18T20:25:25.101073+0000","last_undegraded":"2024-03-18T20:25:25.101073+0000","last_fullsized":"2024-03-18T20:25:25.101073+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:15:29.461741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.111539+0000","last_change":"2024-03-18T20:25:34.111539+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.111539+0000","last_undegraded":"2024-03-18T20:25:34.111539+0000","last_fullsized":"2024-03-18T20:25:34.111539+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:33.109951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.b","version":"231'2","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101371+0000","last_change":"2024-03-18T20:25:22.251851+0000","last_active":"2024-03-18T20:25:25.101371+0000","last_peered":"2024-03-18T20:25:25.101371+0000","last_clean":"2024-03-18T20:25:25.101371+0000","last_became_active":"2024-03-18T20:25:22.251334+0000","last_became_peered":"2024-03-18T20:25:22.251334+0000","last_unstale":"2024-03-18T20:25:25.101371+0000","last_undegraded":"2024-03-18T20:25:25.101371+0000","last_fullsized":"2024-03-18T20:25:25.101371+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:22.889781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100991+0000","last_change":"2024-03-18T20:25:22.687129+0000","last_active":"2024-03-18T20:25:25.100991+0000","last_peered":"2024-03-18T20:25:25.100991+0000","last_clean":"2024-03-18T20:25:25.100991+0000","last_became_active":"2024-03-18T20:25:22.686871+0000","last_became_peered":"2024-03-18T20:25:22.686871+0000","last_unstale":"2024-03-18T20:25:25.100991+0000","last_undegraded":"2024-03-18T20:25:25.100991+0000","last_fullsized":"2024-03-18T20:25:25.100991+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:18:58.196895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.111594+0000","last_change":"2024-03-18T20:25:34.111594+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.111594+0000","last_undegraded":"2024-03-18T20:25:34.111594+0000","last_fullsized":"2024-03-18T20:25:34.111594+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:13.917940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101413+0000","last_change":"2024-03-18T20:25:22.254305+0000","last_active":"2024-03-18T20:25:25.101413+0000","last_peered":"2024-03-18T20:25:25.101413+0000","last_clean":"2024-03-18T20:25:25.101413+0000","last_became_active":"2024-03-18T20:25:22.253898+0000","last_became_peered":"2024-03-18T20:25:22.253898+0000","last_unstale":"2024-03-18T20:25:25.101413+0000","last_undegraded":"2024-03-18T20:25:25.101413+0000","last_fullsized":"2024-03-18T20:25:25.101413+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:29:48.394283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099365+0000","last_change":"2024-03-18T20:25:22.692895+0000","last_active":"2024-03-18T20:25:25.099365+0000","last_peered":"2024-03-18T20:25:25.099365+0000","last_clean":"2024-03-18T20:25:25.099365+0000","last_became_active":"2024-03-18T20:25:22.685766+0000","last_became_peered":"2024-03-18T20:25:22.685766+0000","last_unstale":"2024-03-18T20:25:25.099365+0000","last_undegraded":"2024-03-18T20:25:25.099365+0000","last_fullsized":"2024-03-18T20:25:25.099365+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:16:08.020891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.117145+0000","last_change":"2024-03-18T20:25:34.117145+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.117145+0000","last_undegraded":"2024-03-18T20:25:34.117145+0000","last_fullsized":"2024-03-18T20:25:34.117145+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:38.721170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.d","version":"231'1","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101013+0000","last_change":"2024-03-18T20:25:22.248228+0000","last_active":"2024-03-18T20:25:25.101013+0000","last_peered":"2024-03-18T20:25:25.101013+0000","last_clean":"2024-03-18T20:25:25.101013+0000","last_became_active":"2024-03-18T20:25:22.247916+0000","last_became_peered":"2024-03-18T20:25:22.247916+0000","last_unstale":"2024-03-18T20:25:25.101013+0000","last_undegraded":"2024-03-18T20:25:25.101013+0000","last_fullsized":"2024-03-18T20:25:25.101013+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:56.236862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101055+0000","last_change":"2024-03-18T20:25:22.684910+0000","last_active":"2024-03-18T20:25:25.101055+0000","last_peered":"2024-03-18T20:25:25.101055+0000","last_clean":"2024-03-18T20:25:25.101055+0000","last_became_active":"2024-03-18T20:25:22.684472+0000","last_became_peered":"2024-03-18T20:25:22.684472+0000","last_unstale":"2024-03-18T20:25:25.101055+0000","last_undegraded":"2024-03-18T20:25:25.101055+0000","last_fullsized":"2024-03-18T20:25:25.101055+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:24:33.293594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.112636+0000","last_change":"2024-03-18T20:25:34.112636+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.112636+0000","last_undegraded":"2024-03-18T20:25:34.112636+0000","last_fullsized":"2024-03-18T20:25:34.112636+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:15.369281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.e","version":"231'2","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101154+0000","last_change":"2024-03-18T20:25:22.253997+0000","last_active":"2024-03-18T20:25:25.101154+0000","last_peered":"2024-03-18T20:25:25.101154+0000","last_clean":"2024-03-18T20:25:25.101154+0000","last_became_active":"2024-03-18T20:25:22.253649+0000","last_became_peered":"2024-03-18T20:25:22.253649+0000","last_unstale":"2024-03-18T20:25:25.101154+0000","last_undegraded":"2024-03-18T20:25:25.101154+0000","last_fullsized":"2024-03-18T20:25:25.101154+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:00.435897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100997+0000","last_change":"2024-03-18T20:25:22.687892+0000","last_active":"2024-03-18T20:25:25.100997+0000","last_peered":"2024-03-18T20:25:25.100997+0000","last_clean":"2024-03-18T20:25:25.100997+0000","last_became_active":"2024-03-18T20:25:22.687086+0000","last_became_peered":"2024-03-18T20:25:22.687086+0000","last_unstale":"2024-03-18T20:25:25.100997+0000","last_undegraded":"2024-03-18T20:25:25.100997+0000","last_fullsized":"2024-03-18T20:25:25.100997+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:15:29.461741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.115166+0000","last_change":"2024-03-18T20:25:34.115166+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.115166+0000","last_undegraded":"2024-03-18T20:25:34.115166+0000","last_fullsized":"2024-03-18T20:25:34.115166+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:17.295248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.f","version":"231'1","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101196+0000","last_change":"2024-03-18T20:25:22.255372+0000","last_active":"2024-03-18T20:25:25.101196+0000","last_peered":"2024-03-18T20:25:25.101196+0000","last_clean":"2024-03-18T20:25:25.101196+0000","last_became_active":"2024-03-18T20:25:22.255063+0000","last_became_peered":"2024-03-18T20:25:22.255063+0000","last_unstale":"2024-03-18T20:25:25.101196+0000","last_undegraded":"2024-03-18T20:25:25.101196+0000","last_fullsized":"2024-03-18T20:25:25.101196+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:23.143345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099295+0000","last_change":"2024-03-18T20:25:22.686185+0000","last_active":"2024-03-18T20:25:25.099295+0000","last_peered":"2024-03-18T20:25:25.099295+0000","last_clean":"2024-03-18T20:25:25.099295+0000","last_became_active":"2024-03-18T20:25:22.685939+0000","last_became_peered":"2024-03-18T20:25:22.685939+0000","last_unstale":"2024-03-18T20:25:25.099295+0000","last_undegraded":"2024-03-18T20:25:25.099295+0000","last_fullsized":"2024-03-18T20:25:25.099295+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:48:48.961872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.116524+0000","last_change":"2024-03-18T20:25:34.116524+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.116524+0000","last_undegraded":"2024-03-18T20:25:34.116524+0000","last_fullsized":"2024-03-18T20:25:34.116524+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:50.264082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.10","version":"231'8","reported_seq":20,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101286+0000","last_change":"2024-03-18T20:25:22.252976+0000","last_active":"2024-03-18T20:25:25.101286+0000","last_peered":"2024-03-18T20:25:25.101286+0000","last_clean":"2024-03-18T20:25:25.101286+0000","last_became_active":"2024-03-18T20:25:22.252528+0000","last_became_peered":"2024-03-18T20:25:22.252528+0000","last_unstale":"2024-03-18T20:25:25.101286+0000","last_undegraded":"2024-03-18T20:25:25.101286+0000","last_fullsized":"2024-03-18T20:25:25.101286+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:42:24.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101120+0000","last_change":"2024-03-18T20:25:22.673559+0000","last_active":"2024-03-18T20:25:25.101120+0000","last_peered":"2024-03-18T20:25:25.101120+0000","last_clean":"2024-03-18T20:25:25.101120+0000","last_became_active":"2024-03-18T20:25:22.673267+0000","last_became_peered":"2024-03-18T20:25:22.673267+0000","last_unstale":"2024-03-18T20:25:25.101120+0000","last_undegraded":"2024-03-18T20:25:25.101120+0000","last_fullsized":"2024-03-18T20:25:25.101120+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:44:15.325153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.116256+0000","last_change":"2024-03-18T20:25:34.116256+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.116256+0000","last_undegraded":"2024-03-18T20:25:34.116256+0000","last_fullsized":"2024-03-18T20:25:34.116256+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:50.170851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100588+0000","last_change":"2024-03-18T20:25:22.252927+0000","last_active":"2024-03-18T20:25:25.100588+0000","last_peered":"2024-03-18T20:25:25.100588+0000","last_clean":"2024-03-18T20:25:25.100588+0000","last_became_active":"2024-03-18T20:25:22.252633+0000","last_became_peered":"2024-03-18T20:25:22.252633+0000","last_unstale":"2024-03-18T20:25:25.100588+0000","last_undegraded":"2024-03-18T20:25:25.100588+0000","last_fullsized":"2024-03-18T20:25:25.100588+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:32.345526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100761+0000","last_change":"2024-03-18T20:25:22.694369+0000","last_active":"2024-03-18T20:25:25.100761+0000","last_peered":"2024-03-18T20:25:25.100761+0000","last_clean":"2024-03-18T20:25:25.100761+0000","last_became_active":"2024-03-18T20:25:22.694114+0000","last_became_peered":"2024-03-18T20:25:22.694114+0000","last_unstale":"2024-03-18T20:25:25.100761+0000","last_undegraded":"2024-03-18T20:25:25.100761+0000","last_fullsized":"2024-03-18T20:25:25.100761+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:35:13.631181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.114153+0000","last_change":"2024-03-18T20:25:34.114153+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.114153+0000","last_undegraded":"2024-03-18T20:25:34.114153+0000","last_fullsized":"2024-03-18T20:25:34.114153+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:27.179302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097618+0000","last_change":"2024-03-18T20:25:22.249078+0000","last_active":"2024-03-18T20:25:25.097618+0000","last_peered":"2024-03-18T20:25:25.097618+0000","last_clean":"2024-03-18T20:25:25.097618+0000","last_became_active":"2024-03-18T20:25:22.248588+0000","last_became_peered":"2024-03-18T20:25:22.248588+0000","last_unstale":"2024-03-18T20:25:25.097618+0000","last_undegraded":"2024-03-18T20:25:25.097618+0000","last_fullsized":"2024-03-18T20:25:25.097618+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:34:42.893253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097697+0000","last_change":"2024-03-18T20:25:22.685090+0000","last_active":"2024-03-18T20:25:25.097697+0000","last_peered":"2024-03-18T20:25:25.097697+0000","last_clean":"2024-03-18T20:25:25.097697+0000","last_became_active":"2024-03-18T20:25:22.684824+0000","last_became_peered":"2024-03-18T20:25:22.684824+0000","last_unstale":"2024-03-18T20:25:25.097697+0000","last_undegraded":"2024-03-18T20:25:25.097697+0000","last_fullsized":"2024-03-18T20:25:25.097697+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:21.750701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.112308+0000","last_change":"2024-03-18T20:25:34.112308+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.112308+0000","last_undegraded":"2024-03-18T20:25:34.112308+0000","last_fullsized":"2024-03-18T20:25:34.112308+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:03.410994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.13","version":"231'2","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100964+0000","last_change":"2024-03-18T20:25:22.254668+0000","last_active":"2024-03-18T20:25:25.100964+0000","last_peered":"2024-03-18T20:25:25.100964+0000","last_clean":"2024-03-18T20:25:25.100964+0000","last_became_active":"2024-03-18T20:25:22.254437+0000","last_became_peered":"2024-03-18T20:25:22.254437+0000","last_unstale":"2024-03-18T20:25:25.100964+0000","last_undegraded":"2024-03-18T20:25:25.100964+0000","last_fullsized":"2024-03-18T20:25:25.100964+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:32:02.691847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100981+0000","last_change":"2024-03-18T20:25:22.691203+0000","last_active":"2024-03-18T20:25:25.100981+0000","last_peered":"2024-03-18T20:25:25.100981+0000","last_clean":"2024-03-18T20:25:25.100981+0000","last_became_active":"2024-03-18T20:25:22.690964+0000","last_became_peered":"2024-03-18T20:25:22.690964+0000","last_unstale":"2024-03-18T20:25:25.100981+0000","last_undegraded":"2024-03-18T20:25:25.100981+0000","last_fullsized":"2024-03-18T20:25:25.100981+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:19.068003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.113878+0000","last_change":"2024-03-18T20:25:34.113878+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.113878+0000","last_undegraded":"2024-03-18T20:25:34.113878+0000","last_fullsized":"2024-03-18T20:25:34.113878+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:02.016045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.14","version":"232'8","reported_seq":20,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:32.359025+0000","last_change":"2024-03-18T20:25:22.249429+0000","last_active":"2024-03-18T20:25:32.359025+0000","last_peered":"2024-03-18T20:25:32.359025+0000","last_clean":"2024-03-18T20:25:32.359025+0000","last_became_active":"2024-03-18T20:25:22.248842+0000","last_became_peered":"2024-03-18T20:25:22.248842+0000","last_unstale":"2024-03-18T20:25:32.359025+0000","last_undegraded":"2024-03-18T20:25:32.359025+0000","last_fullsized":"2024-03-18T20:25:32.359025+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:26.827223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2822,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101359+0000","last_change":"2024-03-18T20:25:22.696408+0000","last_active":"2024-03-18T20:25:25.101359+0000","last_peered":"2024-03-18T20:25:25.101359+0000","last_clean":"2024-03-18T20:25:25.101359+0000","last_became_active":"2024-03-18T20:25:22.696095+0000","last_became_peered":"2024-03-18T20:25:22.696095+0000","last_unstale":"2024-03-18T20:25:25.101359+0000","last_undegraded":"2024-03-18T20:25:25.101359+0000","last_fullsized":"2024-03-18T20:25:25.101359+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:19:19.377686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.119645+0000","last_change":"2024-03-18T20:25:34.119645+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.119645+0000","last_undegraded":"2024-03-18T20:25:34.119645+0000","last_fullsized":"2024-03-18T20:25:34.119645+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:50.971852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.15","version":"231'3","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100719+0000","last_change":"2024-03-18T20:25:22.251534+0000","last_active":"2024-03-18T20:25:25.100719+0000","last_peered":"2024-03-18T20:25:25.100719+0000","last_clean":"2024-03-18T20:25:25.100719+0000","last_became_active":"2024-03-18T20:25:22.251275+0000","last_became_peered":"2024-03-18T20:25:22.251275+0000","last_unstale":"2024-03-18T20:25:25.100719+0000","last_undegraded":"2024-03-18T20:25:25.100719+0000","last_fullsized":"2024-03-18T20:25:25.100719+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:38:51.553676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100607+0000","last_change":"2024-03-18T20:25:22.682861+0000","last_active":"2024-03-18T20:25:25.100607+0000","last_peered":"2024-03-18T20:25:25.100607+0000","last_clean":"2024-03-18T20:25:25.100607+0000","last_became_active":"2024-03-18T20:25:22.682610+0000","last_became_peered":"2024-03-18T20:25:22.682610+0000","last_unstale":"2024-03-18T20:25:25.100607+0000","last_undegraded":"2024-03-18T20:25:25.100607+0000","last_fullsized":"2024-03-18T20:25:25.100607+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:51:43.095770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.114296+0000","last_change":"2024-03-18T20:25:34.114296+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.114296+0000","last_undegraded":"2024-03-18T20:25:34.114296+0000","last_fullsized":"2024-03-18T20:25:34.114296+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:32:42.191508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101107+0000","last_change":"2024-03-18T20:25:22.252980+0000","last_active":"2024-03-18T20:25:25.101107+0000","last_peered":"2024-03-18T20:25:25.101107+0000","last_clean":"2024-03-18T20:25:25.101107+0000","last_became_active":"2024-03-18T20:25:22.252706+0000","last_became_peered":"2024-03-18T20:25:22.252706+0000","last_unstale":"2024-03-18T20:25:25.101107+0000","last_undegraded":"2024-03-18T20:25:25.101107+0000","last_fullsized":"2024-03-18T20:25:25.101107+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:45:13.948304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097580+0000","last_change":"2024-03-18T20:25:22.684633+0000","last_active":"2024-03-18T20:25:25.097580+0000","last_peered":"2024-03-18T20:25:25.097580+0000","last_clean":"2024-03-18T20:25:25.097580+0000","last_became_active":"2024-03-18T20:25:22.684366+0000","last_became_peered":"2024-03-18T20:25:22.684366+0000","last_unstale":"2024-03-18T20:25:25.097580+0000","last_undegraded":"2024-03-18T20:25:25.097580+0000","last_fullsized":"2024-03-18T20:25:25.097580+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:15.299142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.112662+0000","last_change":"2024-03-18T20:25:34.112662+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.112662+0000","last_undegraded":"2024-03-18T20:25:34.112662+0000","last_fullsized":"2024-03-18T20:25:34.112662+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:50.170851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101424+0000","last_change":"2024-03-18T20:25:22.251887+0000","last_active":"2024-03-18T20:25:25.101424+0000","last_peered":"2024-03-18T20:25:25.101424+0000","last_clean":"2024-03-18T20:25:25.101424+0000","last_became_active":"2024-03-18T20:25:22.251594+0000","last_became_peered":"2024-03-18T20:25:22.251594+0000","last_unstale":"2024-03-18T20:25:25.101424+0000","last_undegraded":"2024-03-18T20:25:25.101424+0000","last_fullsized":"2024-03-18T20:25:25.101424+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:48:47.954568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100806+0000","last_change":"2024-03-18T20:25:22.680660+0000","last_active":"2024-03-18T20:25:25.100806+0000","last_peered":"2024-03-18T20:25:25.100806+0000","last_clean":"2024-03-18T20:25:25.100806+0000","last_became_active":"2024-03-18T20:25:22.680484+0000","last_became_peered":"2024-03-18T20:25:22.680484+0000","last_unstale":"2024-03-18T20:25:25.100806+0000","last_undegraded":"2024-03-18T20:25:25.100806+0000","last_fullsized":"2024-03-18T20:25:25.100806+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:48:21.194536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.110675+0000","last_change":"2024-03-18T20:25:34.110675+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.110675+0000","last_undegraded":"2024-03-18T20:25:34.110675+0000","last_fullsized":"2024-03-18T20:25:34.110675+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:20.466978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097560+0000","last_change":"2024-03-18T20:25:22.244727+0000","last_active":"2024-03-18T20:25:25.097560+0000","last_peered":"2024-03-18T20:25:25.097560+0000","last_clean":"2024-03-18T20:25:25.097560+0000","last_became_active":"2024-03-18T20:25:22.244417+0000","last_became_peered":"2024-03-18T20:25:22.244417+0000","last_unstale":"2024-03-18T20:25:25.097560+0000","last_undegraded":"2024-03-18T20:25:25.097560+0000","last_fullsized":"2024-03-18T20:25:25.097560+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:10.187509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100848+0000","last_change":"2024-03-18T20:25:22.681952+0000","last_active":"2024-03-18T20:25:25.100848+0000","last_peered":"2024-03-18T20:25:25.100848+0000","last_clean":"2024-03-18T20:25:25.100848+0000","last_became_active":"2024-03-18T20:25:22.681537+0000","last_became_peered":"2024-03-18T20:25:22.681537+0000","last_unstale":"2024-03-18T20:25:25.100848+0000","last_undegraded":"2024-03-18T20:25:25.100848+0000","last_fullsized":"2024-03-18T20:25:25.100848+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:32:22.923763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.113196+0000","last_change":"2024-03-18T20:25:34.113196+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.113196+0000","last_undegraded":"2024-03-18T20:25:34.113196+0000","last_fullsized":"2024-03-18T20:25:34.113196+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:39.087869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100983+0000","last_change":"2024-03-18T20:25:22.251522+0000","last_active":"2024-03-18T20:25:25.100983+0000","last_peered":"2024-03-18T20:25:25.100983+0000","last_clean":"2024-03-18T20:25:25.100983+0000","last_became_active":"2024-03-18T20:25:22.251143+0000","last_became_peered":"2024-03-18T20:25:22.251143+0000","last_unstale":"2024-03-18T20:25:25.100983+0000","last_undegraded":"2024-03-18T20:25:25.100983+0000","last_fullsized":"2024-03-18T20:25:25.100983+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:58.960198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100935+0000","last_change":"2024-03-18T20:25:22.691744+0000","last_active":"2024-03-18T20:25:25.100935+0000","last_peered":"2024-03-18T20:25:25.100935+0000","last_clean":"2024-03-18T20:25:25.100935+0000","last_became_active":"2024-03-18T20:25:22.691491+0000","last_became_peered":"2024-03-18T20:25:22.691491+0000","last_unstale":"2024-03-18T20:25:25.100935+0000","last_undegraded":"2024-03-18T20:25:25.100935+0000","last_fullsized":"2024-03-18T20:25:25.100935+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:04:09.670541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.115119+0000","last_change":"2024-03-18T20:25:34.115119+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.115119+0000","last_undegraded":"2024-03-18T20:25:34.115119+0000","last_fullsized":"2024-03-18T20:25:34.115119+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:25.724222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1b","version":"231'1","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101037+0000","last_change":"2024-03-18T20:25:22.251076+0000","last_active":"2024-03-18T20:25:25.101037+0000","last_peered":"2024-03-18T20:25:25.101037+0000","last_clean":"2024-03-18T20:25:25.101037+0000","last_became_active":"2024-03-18T20:25:22.250741+0000","last_became_peered":"2024-03-18T20:25:22.250741+0000","last_unstale":"2024-03-18T20:25:25.101037+0000","last_undegraded":"2024-03-18T20:25:25.101037+0000","last_fullsized":"2024-03-18T20:25:25.101037+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:07.448931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100894+0000","last_change":"2024-03-18T20:25:22.680768+0000","last_active":"2024-03-18T20:25:25.100894+0000","last_peered":"2024-03-18T20:25:25.100894+0000","last_clean":"2024-03-18T20:25:25.100894+0000","last_became_active":"2024-03-18T20:25:22.680516+0000","last_became_peered":"2024-03-18T20:25:22.680516+0000","last_unstale":"2024-03-18T20:25:25.100894+0000","last_undegraded":"2024-03-18T20:25:25.100894+0000","last_fullsized":"2024-03-18T20:25:25.100894+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:19:19.377686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.111795+0000","last_change":"2024-03-18T20:25:34.111795+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.111795+0000","last_undegraded":"2024-03-18T20:25:34.111795+0000","last_fullsized":"2024-03-18T20:25:34.111795+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:45.664606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1c","version":"231'2","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100697+0000","last_change":"2024-03-18T20:25:22.251018+0000","last_active":"2024-03-18T20:25:25.100697+0000","last_peered":"2024-03-18T20:25:25.100697+0000","last_clean":"2024-03-18T20:25:25.100697+0000","last_became_active":"2024-03-18T20:25:22.250730+0000","last_became_peered":"2024-03-18T20:25:22.250730+0000","last_unstale":"2024-03-18T20:25:25.100697+0000","last_undegraded":"2024-03-18T20:25:25.100697+0000","last_fullsized":"2024-03-18T20:25:25.100697+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:07.579514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101349+0000","last_change":"2024-03-18T20:25:22.690894+0000","last_active":"2024-03-18T20:25:25.101349+0000","last_peered":"2024-03-18T20:25:25.101349+0000","last_clean":"2024-03-18T20:25:25.101349+0000","last_became_active":"2024-03-18T20:25:22.688184+0000","last_became_peered":"2024-03-18T20:25:22.688184+0000","last_unstale":"2024-03-18T20:25:25.101349+0000","last_undegraded":"2024-03-18T20:25:25.101349+0000","last_fullsized":"2024-03-18T20:25:25.101349+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:59.967502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.114019+0000","last_change":"2024-03-18T20:25:34.114019+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.114019+0000","last_undegraded":"2024-03-18T20:25:34.114019+0000","last_fullsized":"2024-03-18T20:25:34.114019+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:33.630861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1d","version":"231'2","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101002+0000","last_change":"2024-03-18T20:25:22.255583+0000","last_active":"2024-03-18T20:25:25.101002+0000","last_peered":"2024-03-18T20:25:25.101002+0000","last_clean":"2024-03-18T20:25:25.101002+0000","last_became_active":"2024-03-18T20:25:22.255243+0000","last_became_peered":"2024-03-18T20:25:22.255243+0000","last_unstale":"2024-03-18T20:25:25.101002+0000","last_undegraded":"2024-03-18T20:25:25.101002+0000","last_fullsized":"2024-03-18T20:25:25.101002+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:38.084389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099342+0000","last_change":"2024-03-18T20:25:22.691295+0000","last_active":"2024-03-18T20:25:25.099342+0000","last_peered":"2024-03-18T20:25:25.099342+0000","last_clean":"2024-03-18T20:25:25.099342+0000","last_became_active":"2024-03-18T20:25:22.690890+0000","last_became_peered":"2024-03-18T20:25:22.690890+0000","last_unstale":"2024-03-18T20:25:25.099342+0000","last_undegraded":"2024-03-18T20:25:25.099342+0000","last_fullsized":"2024-03-18T20:25:25.099342+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:23:16.320044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.117626+0000","last_change":"2024-03-18T20:25:34.117626+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.117626+0000","last_undegraded":"2024-03-18T20:25:34.117626+0000","last_fullsized":"2024-03-18T20:25:34.117626+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:19:02.788524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1e","version":"231'1","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101139+0000","last_change":"2024-03-18T20:25:22.251725+0000","last_active":"2024-03-18T20:25:25.101139+0000","last_peered":"2024-03-18T20:25:25.101139+0000","last_clean":"2024-03-18T20:25:25.101139+0000","last_became_active":"2024-03-18T20:25:22.251416+0000","last_became_peered":"2024-03-18T20:25:22.251416+0000","last_unstale":"2024-03-18T20:25:25.101139+0000","last_undegraded":"2024-03-18T20:25:25.101139+0000","last_fullsized":"2024-03-18T20:25:25.101139+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:26.827223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101012+0000","last_change":"2024-03-18T20:25:22.679158+0000","last_active":"2024-03-18T20:25:25.101012+0000","last_peered":"2024-03-18T20:25:25.101012+0000","last_clean":"2024-03-18T20:25:25.101012+0000","last_became_active":"2024-03-18T20:25:22.678889+0000","last_became_peered":"2024-03-18T20:25:22.678889+0000","last_unstale":"2024-03-18T20:25:25.101012+0000","last_undegraded":"2024-03-18T20:25:25.101012+0000","last_fullsized":"2024-03-18T20:25:25.101012+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:00.365377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.113361+0000","last_change":"2024-03-18T20:25:34.113361+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.113361+0000","last_undegraded":"2024-03-18T20:25:34.113361+0000","last_fullsized":"2024-03-18T20:25:34.113361+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:10.985415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"27.1f","version":"231'4","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100956+0000","last_change":"2024-03-18T20:25:22.251710+0000","last_active":"2024-03-18T20:25:25.100956+0000","last_peered":"2024-03-18T20:25:25.100956+0000","last_clean":"2024-03-18T20:25:25.100956+0000","last_became_active":"2024-03-18T20:25:22.251420+0000","last_became_peered":"2024-03-18T20:25:22.251420+0000","last_unstale":"2024-03-18T20:25:25.100956+0000","last_undegraded":"2024-03-18T20:25:25.100956+0000","last_fullsized":"2024-03-18T20:25:25.100956+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:00.435897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100904+0000","last_change":"2024-03-18T20:25:22.682272+0000","last_active":"2024-03-18T20:25:25.100904+0000","last_peered":"2024-03-18T20:25:25.100904+0000","last_clean":"2024-03-18T20:25:25.100904+0000","last_became_active":"2024-03-18T20:25:22.681785+0000","last_became_peered":"2024-03-18T20:25:22.681785+0000","last_unstale":"2024-03-18T20:25:25.100904+0000","last_undegraded":"2024-03-18T20:25:25.100904+0000","last_fullsized":"2024-03-18T20:25:25.100904+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:53:46.174877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":1,"reported_epoch":233,"state":"creating+peering","last_fresh":"2024-03-18T20:25:34.115690+0000","last_change":"2024-03-18T20:25:34.115690+0000","last_active":"2024-03-18T20:25:34.104842+0000","last_peered":"2024-03-18T20:25:34.104842+0000","last_clean":"2024-03-18T20:25:34.104842+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:34.115690+0000","last_undegraded":"2024-03-18T20:25:34.115690+0000","last_fullsized":"2024-03-18T20:25:34.115690+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:25.724222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100917+0000","last_change":"2024-03-18T20:25:22.679227+0000","last_active":"2024-03-18T20:25:25.100917+0000","last_peered":"2024-03-18T20:25:25.100917+0000","last_clean":"2024-03-18T20:25:25.100917+0000","last_became_active":"2024-03-18T20:25:22.678827+0000","last_became_peered":"2024-03-18T20:25:22.678827+0000","last_unstale":"2024-03-18T20:25:25.100917+0000","last_undegraded":"2024-03-18T20:25:25.100917+0000","last_fullsized":"2024-03-18T20:25:25.100917+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:05.394457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100961+0000","last_change":"2024-03-18T20:25:22.679640+0000","last_active":"2024-03-18T20:25:25.100961+0000","last_peered":"2024-03-18T20:25:25.100961+0000","last_clean":"2024-03-18T20:25:25.100961+0000","last_became_active":"2024-03-18T20:25:22.679408+0000","last_became_peered":"2024-03-18T20:25:22.679408+0000","last_unstale":"2024-03-18T20:25:25.100961+0000","last_undegraded":"2024-03-18T20:25:25.100961+0000","last_fullsized":"2024-03-18T20:25:25.100961+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:24:33.293594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100873+0000","last_change":"2024-03-18T20:25:22.685981+0000","last_active":"2024-03-18T20:25:25.100873+0000","last_peered":"2024-03-18T20:25:25.100873+0000","last_clean":"2024-03-18T20:25:25.100873+0000","last_became_active":"2024-03-18T20:25:22.685683+0000","last_became_peered":"2024-03-18T20:25:22.685683+0000","last_unstale":"2024-03-18T20:25:25.100873+0000","last_undegraded":"2024-03-18T20:25:25.100873+0000","last_fullsized":"2024-03-18T20:25:25.100873+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:36:57.458670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100679+0000","last_change":"2024-03-18T20:25:22.680503+0000","last_active":"2024-03-18T20:25:25.100679+0000","last_peered":"2024-03-18T20:25:25.100679+0000","last_clean":"2024-03-18T20:25:25.100679+0000","last_became_active":"2024-03-18T20:25:22.680258+0000","last_became_peered":"2024-03-18T20:25:22.680258+0000","last_unstale":"2024-03-18T20:25:25.100679+0000","last_undegraded":"2024-03-18T20:25:25.100679+0000","last_fullsized":"2024-03-18T20:25:25.100679+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:49.478525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101401+0000","last_change":"2024-03-18T20:25:22.688098+0000","last_active":"2024-03-18T20:25:25.101401+0000","last_peered":"2024-03-18T20:25:25.101401+0000","last_clean":"2024-03-18T20:25:25.101401+0000","last_became_active":"2024-03-18T20:25:22.687830+0000","last_became_peered":"2024-03-18T20:25:22.687830+0000","last_unstale":"2024-03-18T20:25:25.101401+0000","last_undegraded":"2024-03-18T20:25:25.101401+0000","last_fullsized":"2024-03-18T20:25:25.101401+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:08.456234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097677+0000","last_change":"2024-03-18T20:25:22.689262+0000","last_active":"2024-03-18T20:25:25.097677+0000","last_peered":"2024-03-18T20:25:25.097677+0000","last_clean":"2024-03-18T20:25:25.097677+0000","last_became_active":"2024-03-18T20:25:22.688913+0000","last_became_peered":"2024-03-18T20:25:22.688913+0000","last_unstale":"2024-03-18T20:25:25.097677+0000","last_undegraded":"2024-03-18T20:25:25.097677+0000","last_fullsized":"2024-03-18T20:25:25.097677+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:15:50.135885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101124+0000","last_change":"2024-03-18T20:25:22.687342+0000","last_active":"2024-03-18T20:25:25.101124+0000","last_peered":"2024-03-18T20:25:25.101124+0000","last_clean":"2024-03-18T20:25:25.101124+0000","last_became_active":"2024-03-18T20:25:22.687092+0000","last_became_peered":"2024-03-18T20:25:22.687092+0000","last_unstale":"2024-03-18T20:25:25.101124+0000","last_undegraded":"2024-03-18T20:25:25.101124+0000","last_fullsized":"2024-03-18T20:25:25.101124+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:00:33.383893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100833+0000","last_change":"2024-03-18T20:25:22.683777+0000","last_active":"2024-03-18T20:25:25.100833+0000","last_peered":"2024-03-18T20:25:25.100833+0000","last_clean":"2024-03-18T20:25:25.100833+0000","last_became_active":"2024-03-18T20:25:22.683484+0000","last_became_peered":"2024-03-18T20:25:22.683484+0000","last_unstale":"2024-03-18T20:25:25.100833+0000","last_undegraded":"2024-03-18T20:25:25.100833+0000","last_fullsized":"2024-03-18T20:25:25.100833+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:03:45.332668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101271+0000","last_change":"2024-03-18T20:25:22.684030+0000","last_active":"2024-03-18T20:25:25.101271+0000","last_peered":"2024-03-18T20:25:25.101271+0000","last_clean":"2024-03-18T20:25:25.101271+0000","last_became_active":"2024-03-18T20:25:22.683767+0000","last_became_peered":"2024-03-18T20:25:22.683767+0000","last_unstale":"2024-03-18T20:25:25.101271+0000","last_undegraded":"2024-03-18T20:25:25.101271+0000","last_fullsized":"2024-03-18T20:25:25.101271+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:30.204046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100925+0000","last_change":"2024-03-18T20:25:22.680418+0000","last_active":"2024-03-18T20:25:25.100925+0000","last_peered":"2024-03-18T20:25:25.100925+0000","last_clean":"2024-03-18T20:25:25.100925+0000","last_became_active":"2024-03-18T20:25:22.680149+0000","last_became_peered":"2024-03-18T20:25:22.680149+0000","last_unstale":"2024-03-18T20:25:25.100925+0000","last_undegraded":"2024-03-18T20:25:25.100925+0000","last_fullsized":"2024-03-18T20:25:25.100925+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:51:14.804124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099330+0000","last_change":"2024-03-18T20:25:22.691577+0000","last_active":"2024-03-18T20:25:25.099330+0000","last_peered":"2024-03-18T20:25:25.099330+0000","last_clean":"2024-03-18T20:25:25.099330+0000","last_became_active":"2024-03-18T20:25:22.691250+0000","last_became_peered":"2024-03-18T20:25:22.691250+0000","last_unstale":"2024-03-18T20:25:25.099330+0000","last_undegraded":"2024-03-18T20:25:25.099330+0000","last_fullsized":"2024-03-18T20:25:25.099330+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:55:21.946848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099355+0000","last_change":"2024-03-18T20:25:22.693670+0000","last_active":"2024-03-18T20:25:25.099355+0000","last_peered":"2024-03-18T20:25:25.099355+0000","last_clean":"2024-03-18T20:25:25.099355+0000","last_became_active":"2024-03-18T20:25:22.693178+0000","last_became_peered":"2024-03-18T20:25:22.693178+0000","last_unstale":"2024-03-18T20:25:25.099355+0000","last_undegraded":"2024-03-18T20:25:25.099355+0000","last_fullsized":"2024-03-18T20:25:25.099355+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:13.992732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099388+0000","last_change":"2024-03-18T20:25:22.693769+0000","last_active":"2024-03-18T20:25:25.099388+0000","last_peered":"2024-03-18T20:25:25.099388+0000","last_clean":"2024-03-18T20:25:25.099388+0000","last_became_active":"2024-03-18T20:25:22.691936+0000","last_became_peered":"2024-03-18T20:25:22.691936+0000","last_unstale":"2024-03-18T20:25:25.099388+0000","last_undegraded":"2024-03-18T20:25:25.099388+0000","last_fullsized":"2024-03-18T20:25:25.099388+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:29:49.401587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099414+0000","last_change":"2024-03-18T20:25:22.692857+0000","last_active":"2024-03-18T20:25:25.099414+0000","last_peered":"2024-03-18T20:25:25.099414+0000","last_clean":"2024-03-18T20:25:25.099414+0000","last_became_active":"2024-03-18T20:25:22.692615+0000","last_became_peered":"2024-03-18T20:25:22.692615+0000","last_unstale":"2024-03-18T20:25:25.099414+0000","last_undegraded":"2024-03-18T20:25:25.099414+0000","last_fullsized":"2024-03-18T20:25:25.099414+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:21:45.938034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100738+0000","last_change":"2024-03-18T20:25:22.683388+0000","last_active":"2024-03-18T20:25:25.100738+0000","last_peered":"2024-03-18T20:25:25.100738+0000","last_clean":"2024-03-18T20:25:25.100738+0000","last_became_active":"2024-03-18T20:25:22.683137+0000","last_became_peered":"2024-03-18T20:25:22.683137+0000","last_unstale":"2024-03-18T20:25:25.100738+0000","last_undegraded":"2024-03-18T20:25:25.100738+0000","last_fullsized":"2024-03-18T20:25:25.100738+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:02.506299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100813+0000","last_change":"2024-03-18T20:25:22.696148+0000","last_active":"2024-03-18T20:25:25.100813+0000","last_peered":"2024-03-18T20:25:25.100813+0000","last_clean":"2024-03-18T20:25:25.100813+0000","last_became_active":"2024-03-18T20:25:22.695928+0000","last_became_peered":"2024-03-18T20:25:22.695928+0000","last_unstale":"2024-03-18T20:25:25.100813+0000","last_undegraded":"2024-03-18T20:25:25.100813+0000","last_fullsized":"2024-03-18T20:25:25.100813+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:57.565171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097534+0000","last_change":"2024-03-18T20:25:22.681389+0000","last_active":"2024-03-18T20:25:25.097534+0000","last_peered":"2024-03-18T20:25:25.097534+0000","last_clean":"2024-03-18T20:25:25.097534+0000","last_became_active":"2024-03-18T20:25:22.681146+0000","last_became_peered":"2024-03-18T20:25:22.681146+0000","last_unstale":"2024-03-18T20:25:25.097534+0000","last_undegraded":"2024-03-18T20:25:25.097534+0000","last_fullsized":"2024-03-18T20:25:25.097534+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:05:57.372944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100832+0000","last_change":"2024-03-18T20:25:22.692497+0000","last_active":"2024-03-18T20:25:25.100832+0000","last_peered":"2024-03-18T20:25:25.100832+0000","last_clean":"2024-03-18T20:25:25.100832+0000","last_became_active":"2024-03-18T20:25:22.692192+0000","last_became_peered":"2024-03-18T20:25:22.692192+0000","last_unstale":"2024-03-18T20:25:25.100832+0000","last_undegraded":"2024-03-18T20:25:25.100832+0000","last_fullsized":"2024-03-18T20:25:25.100832+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:03:47.775258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099320+0000","last_change":"2024-03-18T20:25:22.690898+0000","last_active":"2024-03-18T20:25:25.099320+0000","last_peered":"2024-03-18T20:25:25.099320+0000","last_clean":"2024-03-18T20:25:25.099320+0000","last_became_active":"2024-03-18T20:25:22.690461+0000","last_became_peered":"2024-03-18T20:25:22.690461+0000","last_unstale":"2024-03-18T20:25:25.099320+0000","last_undegraded":"2024-03-18T20:25:25.099320+0000","last_fullsized":"2024-03-18T20:25:25.099320+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:15.208122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101023+0000","last_change":"2024-03-18T20:25:22.685172+0000","last_active":"2024-03-18T20:25:25.101023+0000","last_peered":"2024-03-18T20:25:25.101023+0000","last_clean":"2024-03-18T20:25:25.101023+0000","last_became_active":"2024-03-18T20:25:22.684855+0000","last_became_peered":"2024-03-18T20:25:22.684855+0000","last_unstale":"2024-03-18T20:25:25.101023+0000","last_undegraded":"2024-03-18T20:25:25.101023+0000","last_fullsized":"2024-03-18T20:25:25.101023+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:00.030804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101390+0000","last_change":"2024-03-18T20:25:22.693640+0000","last_active":"2024-03-18T20:25:25.101390+0000","last_peered":"2024-03-18T20:25:25.101390+0000","last_clean":"2024-03-18T20:25:25.101390+0000","last_became_active":"2024-03-18T20:25:22.693403+0000","last_became_peered":"2024-03-18T20:25:22.693403+0000","last_unstale":"2024-03-18T20:25:25.101390+0000","last_undegraded":"2024-03-18T20:25:25.101390+0000","last_fullsized":"2024-03-18T20:25:25.101390+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:58.151146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097657+0000","last_change":"2024-03-18T20:25:22.693546+0000","last_active":"2024-03-18T20:25:25.097657+0000","last_peered":"2024-03-18T20:25:25.097657+0000","last_clean":"2024-03-18T20:25:25.097657+0000","last_became_active":"2024-03-18T20:25:22.693267+0000","last_became_peered":"2024-03-18T20:25:22.693267+0000","last_unstale":"2024-03-18T20:25:25.097657+0000","last_undegraded":"2024-03-18T20:25:25.097657+0000","last_fullsized":"2024-03-18T20:25:25.097657+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:35.486064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099426+0000","last_change":"2024-03-18T20:25:22.693152+0000","last_active":"2024-03-18T20:25:25.099426+0000","last_peered":"2024-03-18T20:25:25.099426+0000","last_clean":"2024-03-18T20:25:25.099426+0000","last_became_active":"2024-03-18T20:25:22.686521+0000","last_became_peered":"2024-03-18T20:25:22.686521+0000","last_unstale":"2024-03-18T20:25:25.099426+0000","last_undegraded":"2024-03-18T20:25:25.099426+0000","last_fullsized":"2024-03-18T20:25:25.099426+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:40:18.302530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.100797+0000","last_change":"2024-03-18T20:25:22.681784+0000","last_active":"2024-03-18T20:25:25.100797+0000","last_peered":"2024-03-18T20:25:25.100797+0000","last_clean":"2024-03-18T20:25:25.100797+0000","last_became_active":"2024-03-18T20:25:22.681198+0000","last_became_peered":"2024-03-18T20:25:22.681198+0000","last_unstale":"2024-03-18T20:25:25.100797+0000","last_undegraded":"2024-03-18T20:25:25.100797+0000","last_fullsized":"2024-03-18T20:25:25.100797+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:35.307035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.099272+0000","last_change":"2024-03-18T20:25:22.685766+0000","last_active":"2024-03-18T20:25:25.099272+0000","last_peered":"2024-03-18T20:25:25.099272+0000","last_clean":"2024-03-18T20:25:25.099272+0000","last_became_active":"2024-03-18T20:25:22.683799+0000","last_became_peered":"2024-03-18T20:25:22.683799+0000","last_unstale":"2024-03-18T20:25:25.099272+0000","last_undegraded":"2024-03-18T20:25:25.099272+0000","last_fullsized":"2024-03-18T20:25:25.099272+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:42:25.581359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101317+0000","last_change":"2024-03-18T20:25:22.694269+0000","last_active":"2024-03-18T20:25:25.101317+0000","last_peered":"2024-03-18T20:25:25.101317+0000","last_clean":"2024-03-18T20:25:25.101317+0000","last_became_active":"2024-03-18T20:25:22.694014+0000","last_became_peered":"2024-03-18T20:25:22.694014+0000","last_unstale":"2024-03-18T20:25:25.101317+0000","last_undegraded":"2024-03-18T20:25:25.101317+0000","last_fullsized":"2024-03-18T20:25:25.101317+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:43:52.409254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101002+0000","last_change":"2024-03-18T20:25:22.681736+0000","last_active":"2024-03-18T20:25:25.101002+0000","last_peered":"2024-03-18T20:25:25.101002+0000","last_clean":"2024-03-18T20:25:25.101002+0000","last_became_active":"2024-03-18T20:25:22.679120+0000","last_became_peered":"2024-03-18T20:25:22.679120+0000","last_unstale":"2024-03-18T20:25:25.101002+0000","last_undegraded":"2024-03-18T20:25:25.101002+0000","last_fullsized":"2024-03-18T20:25:25.101002+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:00.365377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101337+0000","last_change":"2024-03-18T20:25:22.695525+0000","last_active":"2024-03-18T20:25:25.101337+0000","last_peered":"2024-03-18T20:25:25.101337+0000","last_clean":"2024-03-18T20:25:25.101337+0000","last_became_active":"2024-03-18T20:25:22.694079+0000","last_became_peered":"2024-03-18T20:25:22.694079+0000","last_unstale":"2024-03-18T20:25:25.101337+0000","last_undegraded":"2024-03-18T20:25:25.101337+0000","last_fullsized":"2024-03-18T20:25:25.101337+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:45:14.955608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101048+0000","last_change":"2024-03-18T20:25:22.683021+0000","last_active":"2024-03-18T20:25:25.101048+0000","last_peered":"2024-03-18T20:25:25.101048+0000","last_clean":"2024-03-18T20:25:25.101048+0000","last_became_active":"2024-03-18T20:25:22.682722+0000","last_became_peered":"2024-03-18T20:25:22.682722+0000","last_unstale":"2024-03-18T20:25:25.101048+0000","last_undegraded":"2024-03-18T20:25:25.101048+0000","last_fullsized":"2024-03-18T20:25:25.101048+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:27.207710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.097638+0000","last_change":"2024-03-18T20:25:22.679426+0000","last_active":"2024-03-18T20:25:25.097638+0000","last_peered":"2024-03-18T20:25:25.097638+0000","last_clean":"2024-03-18T20:25:25.097638+0000","last_became_active":"2024-03-18T20:25:22.679159+0000","last_became_peered":"2024-03-18T20:25:22.679159+0000","last_unstale":"2024-03-18T20:25:25.097638+0000","last_undegraded":"2024-03-18T20:25:25.097638+0000","last_fullsized":"2024-03-18T20:25:25.097638+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:17:59.556803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101022+0000","last_change":"2024-03-18T20:25:22.682750+0000","last_active":"2024-03-18T20:25:25.101022+0000","last_peered":"2024-03-18T20:25:25.101022+0000","last_clean":"2024-03-18T20:25:25.101022+0000","last_became_active":"2024-03-18T20:25:22.682497+0000","last_became_peered":"2024-03-18T20:25:22.682497+0000","last_unstale":"2024-03-18T20:25:25.101022+0000","last_undegraded":"2024-03-18T20:25:25.101022+0000","last_fullsized":"2024-03-18T20:25:25.101022+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:33.092503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:25:35.113201+0000","last_change":"2024-03-18T20:25:35.113201+0000","last_active":"2024-03-18T20:25:35.113201+0000","last_peered":"2024-03-18T20:25:35.113201+0000","last_clean":"2024-03-18T20:25:35.113201+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:25:35.113201+0000","last_undegraded":"2024-03-18T20:25:35.113201+0000","last_fullsized":"2024-03-18T20:25:35.113201+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":11,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-03-18T20:25:25.101174+0000","last_change":"2024-03-18T20:25:22.696207+0000","last_active":"2024-03-18T20:25:25.101174+0000","last_peered":"2024-03-18T20:25:25.101174+0000","last_clean":"2024-03-18T20:25:25.101174+0000","last_became_active":"2024-03-18T20:25:22.695943+0000","last_became_peered":"2024-03-18T20:25:22.695943+0000","last_unstale":"2024-03-18T20:25:25.101174+0000","last_undegraded":"2024-03-18T20:25:25.101174+0000","last_fullsized":"2024-03-18T20:25:25.101174+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:54:33.571448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3620,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575983,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52616,"kb_used_data":3452,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319224,"statfs":{"total":96636764160,"available":96582885376,"internally_reserved":0,"allocated":3534848,"data_stored":2093396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80700000000000005,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.48499999999999999,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.81899999999999995,"5min":1.093,"15min":1.093},"last":0.497}]},{"osd":1,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61799999999999999,"15min":0.61699999999999999},"min":{"1min":0.47599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.82999999999999996,"5min":0.99399999999999999,"15min":1.4299999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.60999999999999999,"15min":0.61899999999999999},"min":{"1min":0.47099999999999997,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.77500000000000002,"5min":1.0449999999999999,"15min":1.2629999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.46400000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84299999999999997,"5min":1.216,"15min":1.216},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.62,"15min":0.625},"min":{"1min":0.48399999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.81799999999999995,"5min":1.177,"15min":1.2490000000000001},"last":0.66000000000000003}]},{"osd":3,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.42599999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84699999999999998,"5min":0.93100000000000005,"15min":1.4450000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.5,"5min":0.44900000000000001,"15min":0.38600000000000001},"max":{"1min":0.84399999999999997,"5min":1.028,"15min":1.5029999999999999},"last":0.52200000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.47499999999999998,"15min":0.47699999999999998},"min":{"1min":0.33700000000000002,"5min":0.314,"15min":0.29899999999999999},"max":{"1min":0.68400000000000005,"5min":0.79100000000000004,"15min":0.86099999999999999},"last":0.439},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.35199999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71599999999999997,"5min":0.76600000000000001,"15min":0.80200000000000005},"last":0.53600000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51500000000000001,"15min":0.51600000000000001},"min":{"1min":0.34899999999999998,"5min":0.34399999999999997,"15min":0.316},"max":{"1min":0.68999999999999995,"5min":0.86599999999999999,"15min":0.89400000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.40699999999999997,"5min":0.377,"15min":0.36799999999999999},"max":{"1min":0.75600000000000001,"5min":0.81599999999999995,"15min":1.042},"last":0.64400000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56399999999999995,"15min":0.57699999999999996},"min":{"1min":0.39900000000000002,"5min":0.36199999999999999,"15min":0.34999999999999998},"max":{"1min":0.71099999999999997,"5min":0.85599999999999998,"15min":1.129},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.371,"5min":0.36299999999999999,"15min":0.34000000000000002},"max":{"1min":0.82899999999999996,"5min":0.85299999999999998,"15min":1.0609999999999999},"last":0.63600000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575983,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42180,"kb_used_data":2872,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329660,"statfs":{"total":96636764160,"available":96593571840,"internally_reserved":0,"allocated":2940928,"data_stored":1503540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57599999999999996,"15min":0.60099999999999998},"min":{"1min":0.41399999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72199999999999998,"5min":0.80900000000000005,"15min":1.1519999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59099999999999997,"15min":0.60799999999999998},"min":{"1min":0.33700000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.74299999999999999,"5min":0.81499999999999995,"15min":0.83299999999999996},"last":0.55000000000000004}]},{"osd":1,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57899999999999996,"15min":0.60199999999999998},"min":{"1min":0.39400000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.79100000000000004,"5min":0.79800000000000004,"15min":1.244},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58499999999999996,"15min":0.60399999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33200000000000002},"max":{"1min":0.77000000000000002,"5min":0.89200000000000002,"15min":1.286},"last":0.61299999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60999999999999999,"15min":0.63},"min":{"1min":0.47899999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71199999999999997,"5min":0.92000000000000004,"15min":1.276},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.59299999999999997,"15min":0.61099999999999999},"min":{"1min":0.434,"5min":0.40100000000000002,"15min":0.378},"max":{"1min":0.77500000000000002,"5min":0.95399999999999996,"15min":1.2669999999999999},"last":0.748}]},{"osd":3,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59699999999999998,"15min":0.622},"min":{"1min":0.38,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70699999999999996,"5min":0.83199999999999996,"15min":1.02},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60599999999999998,"15min":0.61899999999999999},"min":{"1min":0.45200000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82399999999999995,"5min":1.0049999999999999,"15min":1.337},"last":0.63800000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.48099999999999998,"15min":0.48299999999999998},"min":{"1min":0.308,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.61699999999999999,"5min":0.85799999999999998,"15min":0.88300000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.50900000000000001,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72099999999999997,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.65500000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.5,"15min":0.504},"min":{"1min":0.375,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.84799999999999998},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39000000000000001,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.73399999999999999,"5min":0.92600000000000005,"15min":0.94199999999999995},"last":0.72299999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40600000000000003,"5min":0.35799999999999998,"15min":0.34499999999999997},"max":{"1min":0.74299999999999999,"5min":0.95599999999999996,"15min":1.0429999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.44800000000000001,"5min":0.36599999999999999,"15min":0.34599999999999997},"max":{"1min":0.83899999999999997,"5min":1.028,"15min":1.028},"last":0.76300000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575983,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47924,"kb_used_data":2856,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323916,"statfs":{"total":96636764160,"available":96587689984,"internally_reserved":0,"allocated":2924544,"data_stored":1500136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.40400000000000003,"15min":0.39600000000000002},"min":{"1min":0.25600000000000001,"5min":0.246,"15min":0.23999999999999999},"max":{"1min":0.57999999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.41899999999999998,"15min":0.42999999999999999},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.58999999999999997,"5min":0.69699999999999995,"15min":0.72299999999999998},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.44,"15min":0.439},"min":{"1min":0.309,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61199999999999999,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48099999999999998,"15min":0.47799999999999998},"min":{"1min":0.35099999999999998,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.70099999999999996,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.377}]},{"osd":3,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.495,"15min":0.495},"min":{"1min":0.36199999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.64300000000000002,"5min":0.75900000000000001,"15min":0.76100000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.44800000000000001,"15min":0.442},"min":{"1min":0.32800000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68200000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.36199999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.68300000000000005,"15min":0.67300000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86799999999999999,"5min":1.0680000000000001,"15min":1.2909999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.68100000000000005,"15min":0.68000000000000005},"min":{"1min":0.50900000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.84499999999999997,"5min":0.92500000000000004,"15min":1.746},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.68100000000000005,"15min":0.67000000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.6910000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.71999999999999997,"15min":0.71399999999999997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.46899999999999997},"max":{"1min":0.89400000000000002,"5min":1.109,"15min":1.2549999999999999},"last":0.61699999999999999}]},{"osd":6,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.69699999999999995,"15min":0.68500000000000005},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.437},"max":{"1min":0.97399999999999998,"5min":0.98899999999999999,"15min":1.637},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71899999999999997,"15min":0.71199999999999997},"min":{"1min":0.49099999999999999,"5min":0.46999999999999997,"15min":0.46100000000000002},"max":{"1min":0.89500000000000002,"5min":1.014,"15min":1.722},"last":0.58799999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.70699999999999996,"15min":0.71299999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79800000000000004,"5min":1.02,"15min":1.2729999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.498,"5min":0.495,"15min":0.42799999999999999},"max":{"1min":0.90300000000000002,"5min":1.0329999999999999,"15min":1.3380000000000001},"last":0.63}]}]},{"osd":0,"up_from":13,"seq":55834575984,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30460,"kb_used_data":3440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341380,"statfs":{"total":96636764160,"available":96605573120,"internally_reserved":0,"allocated":3522560,"data_stored":2090594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.42399999999999999,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.374},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.42099999999999999,"15min":0.42799999999999999},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.46600000000000003,"5min":0.64700000000000002,"15min":1.1850000000000001},"last":0.28999999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.42599999999999999,"15min":0.435},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.47099999999999997,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.438,"15min":0.45300000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29799999999999999},"max":{"1min":0.59199999999999997,"5min":0.64300000000000002,"15min":0.70999999999999996},"last":0.33000000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.26100000000000001},"max":{"1min":0.45800000000000002,"5min":0.60099999999999998,"15min":0.628},"last":0.442},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48599999999999999,"15min":0.496},"min":{"1min":0.35499999999999998,"5min":0.35199999999999998,"15min":0.32500000000000001},"max":{"1min":0.52300000000000002,"5min":0.73999999999999999,"15min":0.91500000000000004},"last":0.34300000000000003}]},{"osd":4,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64300000000000002,"15min":0.65500000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81299999999999994,"5min":0.91500000000000004,"15min":1.0029999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.66200000000000003,"15min":0.66300000000000003},"min":{"1min":0.47499999999999998,"5min":0.47099999999999997,"15min":0.39700000000000002},"max":{"1min":0.78100000000000003,"5min":0.95299999999999996,"15min":1.0469999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.67800000000000005,"15min":0.67100000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89700000000000002,"5min":0.97099999999999997,"15min":1.0129999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91100000000000003,"5min":0.92900000000000005,"15min":1.518},"last":0.61899999999999999}]},{"osd":6,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.47399999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.78600000000000003,"5min":0.95899999999999996,"15min":1.1279999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71199999999999997,"15min":0.70899999999999996},"min":{"1min":0.498,"5min":0.48999999999999999,"15min":0.47899999999999998},"max":{"1min":0.94899999999999995,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.65300000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85099999999999998,"5min":0.96399999999999997,"15min":1.464},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.69899999999999995,"15min":0.70399999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.47599999999999998},"max":{"1min":0.75800000000000001,"5min":1.042,"15min":1.2350000000000001},"last":0.63300000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575984,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47928,"kb_used_data":2860,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323912,"statfs":{"total":96636764160,"available":96587685888,"internally_reserved":0,"allocated":2928640,"data_stored":1500226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36599999999999999,"15min":0.378},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.496,"5min":0.54800000000000004,"15min":0.59399999999999997},"last":0.318},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.47399999999999998,"15min":0.48699999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.63600000000000001,"5min":0.71499999999999997,"15min":0.753},"last":0.34599999999999997}]},{"osd":1,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46899999999999997,"15min":0.48199999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64300000000000002,"5min":0.72599999999999998,"15min":0.77100000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40899999999999997,"15min":0.41599999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.53800000000000003,"5min":0.59299999999999997,"15min":0.71099999999999997},"last":0.30399999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.433,"15min":0.436},"min":{"1min":0.28899999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.497,"15min":0.51700000000000002},"min":{"1min":0.33800000000000002,"5min":0.32200000000000001,"15min":0.313},"max":{"1min":0.68100000000000005,"5min":0.752,"15min":0.752},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.66300000000000003,"15min":0.65800000000000003},"min":{"1min":0.503,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":0.82499999999999996,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67500000000000004,"15min":0.66800000000000004},"min":{"1min":0.497,"5min":0.443,"15min":0.443},"max":{"1min":0.91600000000000004,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.67900000000000005,"15min":0.68500000000000005},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.93999999999999995,"5min":1.028,"15min":1.028},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67300000000000004,"15min":0.68000000000000005},"min":{"1min":0.47299999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.88300000000000001,"5min":0.95599999999999996,"15min":1.0629999999999999},"last":0.50600000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68000000000000005,"15min":0.69099999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86099999999999999,"5min":1.014,"15min":1.2050000000000001},"last":0.432},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.72499999999999998,"15min":0.73199999999999998},"min":{"1min":0.53800000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.97199999999999998,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.63200000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68899999999999995,"15min":0.70199999999999996},"min":{"1min":0.51500000000000001,"5min":0.442,"15min":0.42199999999999999},"max":{"1min":0.92900000000000005,"5min":0.99399999999999999,"15min":1.0469999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69899999999999995,"15min":0.70699999999999996},"min":{"1min":0.45500000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.93999999999999995,"5min":1.371,"15min":1.371},"last":0.59899999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575984,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52032,"kb_used_data":2868,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319808,"statfs":{"total":96636764160,"available":96583483392,"internally_reserved":0,"allocated":2936832,"data_stored":1500262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.38800000000000001,"15min":0.39900000000000002},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.53700000000000003,"5min":0.66500000000000004,"15min":0.98099999999999998},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.40999999999999998,"15min":0.41499999999999998},"min":{"1min":0.27700000000000002,"5min":0.25800000000000001,"15min":0.24199999999999999},"max":{"1min":0.434,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.29399999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39000000000000001,"15min":0.40999999999999998},"min":{"1min":0.30099999999999999,"5min":0.26100000000000001,"15min":0.254},"max":{"1min":0.45900000000000002,"5min":0.63200000000000001,"15min":0.64000000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.46100000000000002,"15min":0.46600000000000003},"min":{"1min":0.36699999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57499999999999996,"5min":0.65500000000000003,"15min":0.73099999999999998},"last":0.35399999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.45500000000000002,"15min":0.46600000000000003},"min":{"1min":0.34999999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53100000000000003,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.48199999999999998,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.51700000000000002,"5min":0.67000000000000004,"15min":0.73099999999999998},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.63600000000000001,"15min":0.65200000000000002},"min":{"1min":0.48799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81899999999999995,"5min":0.91700000000000004,"15min":1.097},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65200000000000002,"15min":0.67500000000000004},"min":{"1min":0.435,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78800000000000003,"5min":0.99299999999999999,"15min":1.0569999999999999},"last":0.56599999999999995}]},{"osd":5,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68400000000000005,"15min":0.69699999999999995},"min":{"1min":0.48999999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92100000000000004,"5min":0.995,"15min":1.1599999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.72499999999999998,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.82099999999999995,"5min":1.171,"15min":1.171},"last":0.54000000000000004}]},{"osd":6,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68500000000000005,"15min":0.70799999999999996},"min":{"1min":0.48099999999999998,"5min":0.436,"15min":0.42499999999999999},"max":{"1min":0.90500000000000003,"5min":1.012,"15min":1.194},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.71799999999999997,"15min":0.73299999999999998},"min":{"1min":0.53300000000000003,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":0.88200000000000001,"5min":1.0409999999999999,"15min":1.2410000000000001},"last":0.68000000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.70099999999999996,"15min":0.71899999999999997},"min":{"1min":0.49099999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82799999999999996,"5min":1.079,"15min":1.079},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68100000000000005,"15min":0.69999999999999996},"min":{"1min":0.44500000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83799999999999997,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.60899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575984,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64312,"kb_used_data":2860,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307528,"statfs":{"total":96636764160,"available":96570908672,"internally_reserved":0,"allocated":2928640,"data_stored":1500718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.57899999999999996,"15min":0.59299999999999997},"min":{"1min":0.34300000000000003,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65300000000000002,"5min":0.77500000000000002,"15min":0.81799999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57799999999999996,"15min":0.59799999999999998},"min":{"1min":0.40300000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69999999999999996,"5min":0.79600000000000004,"15min":0.84199999999999997},"last":0.58599999999999997}]},{"osd":1,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59999999999999998,"15min":0.61399999999999999},"min":{"1min":0.438,"5min":0.318,"15min":0.318},"max":{"1min":0.73199999999999998,"5min":0.81899999999999995,"15min":0.88300000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56999999999999995,"15min":0.58499999999999996},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.71199999999999997,"5min":0.79200000000000004,"15min":0.96499999999999997},"last":0.71199999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58299999999999996,"15min":0.59899999999999998},"min":{"1min":0.439,"5min":0.39500000000000002,"15min":0.39200000000000002},"max":{"1min":0.68600000000000005,"5min":0.80500000000000005,"15min":0.82599999999999996},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61099999999999999,"15min":0.61799999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.38500000000000001},"max":{"1min":0.81399999999999995,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.40799999999999997,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.86199999999999999},"last":0.52600000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.498,"15min":0.504},"min":{"1min":0.35199999999999998,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.85499999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.442}]},{"osd":6,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.499,"15min":0.505},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.46700000000000003}]},{"osd":7,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54100000000000004,"15min":0.54800000000000004},"min":{"1min":0.435,"5min":0.38,"15min":0.35699999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.57199999999999995,"15min":0.57999999999999996},"min":{"1min":0.41899999999999998,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.65100000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575983,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47932,"kb_used_data":2864,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323908,"statfs":{"total":96636764160,"available":96587681792,"internally_reserved":0,"allocated":2932736,"data_stored":1500192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42499999999999999,"5min":0.33900000000000002,"15min":0.31900000000000001},"max":{"1min":0.72699999999999998,"5min":0.85999999999999999,"15min":0.96499999999999997},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57799999999999996,"15min":0.60399999999999998},"min":{"1min":0.432,"5min":0.37,"15min":0.33100000000000002},"max":{"1min":0.69799999999999995,"5min":0.82599999999999996,"15min":1.3100000000000001},"last":0.55500000000000005}]},{"osd":1,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58499999999999996,"15min":0.60199999999999998},"min":{"1min":0.36899999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.74399999999999999,"5min":0.79600000000000004,"15min":1.056},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58399999999999996,"15min":0.60199999999999998},"min":{"1min":0.46600000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.63100000000000001,"5min":0.89800000000000002,"15min":1.018},"last":0.54000000000000004}]},{"osd":2,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.38700000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.84299999999999997,"15min":0.99299999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59399999999999997,"15min":0.60899999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.67500000000000004,"5min":0.875,"15min":1.1120000000000001},"last":0.52300000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.42799999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.78100000000000003,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63,"15min":0.64600000000000002},"min":{"1min":0.443,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69899999999999995,"5min":0.82399999999999995,"15min":1.161},"last":0.51000000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.47899999999999998,"15min":0.47499999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.77500000000000002},"last":0.432},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51200000000000001,"15min":0.50600000000000001},"min":{"1min":0.36199999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.89700000000000002},"last":0.39300000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.503,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.34000000000000002,"15min":0.32800000000000001},"max":{"1min":0.69999999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.40300000000000002,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.86599999999999999},"last":0.36699999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.40799999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.94499999999999995},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.45300000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74099999999999999,"5min":0.85999999999999999,"15min":0.91200000000000003},"last":0.41699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:25:40.539 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1c is not active+clean 2024-03-18T20:25:40.539 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.114498+0000', 'last_change': '2024-03-18T20:25:34.114498+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.114498+0000', 'last_undegraded': '2024-03-18T20:25:34.114498+0000', 'last_fullsized': '2024-03-18T20:25:34.114498+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T07:46:47.366224+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-18T20:25:40.540 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1f is not active+clean 2024-03-18T20:25:40.540 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.540 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.6 is not active+clean 2024-03-18T20:25:40.540 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.117162+0000', 'last_change': '2024-03-18T20:25:34.117162+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.117162+0000', 'last_undegraded': '2024-03-18T20:25:34.117162+0000', 'last_fullsized': '2024-03-18T20:25:34.117162+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T05:04:13.759015+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-18T20:25:40.540 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.5 is not active+clean 2024-03-18T20:25:40.540 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.540 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.7 is not active+clean 2024-03-18T20:25:40.540 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.111213+0000', 'last_change': '2024-03-18T20:25:34.111213+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.111213+0000', 'last_undegraded': '2024-03-18T20:25:34.111213+0000', 'last_fullsized': '2024-03-18T20:25:34.111213+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T02:02:17.295248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-18T20:25:40.540 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.4 is not active+clean 2024-03-18T20:25:40.541 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.541 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.4 is not active+clean 2024-03-18T20:25:40.541 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.110932+0000', 'last_change': '2024-03-18T20:25:34.110932+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.110932+0000', 'last_undegraded': '2024-03-18T20:25:34.110932+0000', 'last_fullsized': '2024-03-18T20:25:34.110932+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T03:44:27.205313+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-18T20:25:40.541 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.7 is not active+clean 2024-03-18T20:25:40.541 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.541 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.5 is not active+clean 2024-03-18T20:25:40.541 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.116497+0000', 'last_change': '2024-03-18T20:25:34.116497+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.116497+0000', 'last_undegraded': '2024-03-18T20:25:34.116497+0000', 'last_fullsized': '2024-03-18T20:25:34.116497+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-19T20:54:41.670198+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-18T20:25:40.541 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.6 is not active+clean 2024-03-18T20:25:40.541 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.542 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.2 is not active+clean 2024-03-18T20:25:40.542 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.115535+0000', 'last_change': '2024-03-18T20:25:34.115535+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.115535+0000', 'last_undegraded': '2024-03-18T20:25:34.115535+0000', 'last_fullsized': '2024-03-18T20:25:34.115535+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T06:56:56.513767+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-18T20:25:40.542 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1 is not active+clean 2024-03-18T20:25:40.542 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.542 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.3 is not active+clean 2024-03-18T20:25:40.542 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.112128+0000', 'last_change': '2024-03-18T20:25:34.112128+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.112128+0000', 'last_undegraded': '2024-03-18T20:25:34.112128+0000', 'last_fullsized': '2024-03-18T20:25:34.112128+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T07:25:32.071310+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-18T20:25:40.542 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.0 is not active+clean 2024-03-18T20:25:40.542 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.542 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.0 is not active+clean 2024-03-18T20:25:40.542 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.111764+0000', 'last_change': '2024-03-18T20:25:34.111764+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.111764+0000', 'last_undegraded': '2024-03-18T20:25:34.111764+0000', 'last_fullsized': '2024-03-18T20:25:34.111764+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T08:13:25.724222+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-18T20:25:40.543 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3 is not active+clean 2024-03-18T20:25:40.543 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.543 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1 is not active+clean 2024-03-18T20:25:40.543 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.114953+0000', 'last_change': '2024-03-18T20:25:34.114953+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.114953+0000', 'last_undegraded': '2024-03-18T20:25:34.114953+0000', 'last_fullsized': '2024-03-18T20:25:34.114953+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T01:06:09.253104+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-18T20:25:40.543 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2 is not active+clean 2024-03-18T20:25:40.543 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.543 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.e is not active+clean 2024-03-18T20:25:40.543 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.115435+0000', 'last_change': '2024-03-18T20:25:34.115435+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.115435+0000', 'last_undegraded': '2024-03-18T20:25:34.115435+0000', 'last_fullsized': '2024-03-18T20:25:34.115435+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T07:32:42.191508+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-18T20:25:40.543 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.d is not active+clean 2024-03-18T20:25:40.543 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.544 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.f is not active+clean 2024-03-18T20:25:40.544 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.113347+0000', 'last_change': '2024-03-18T20:25:34.113347+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.113347+0000', 'last_undegraded': '2024-03-18T20:25:34.113347+0000', 'last_fullsized': '2024-03-18T20:25:34.113347+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T07:22:13.432979+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-18T20:25:40.544 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.c is not active+clean 2024-03-18T20:25:40.544 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.544 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.c is not active+clean 2024-03-18T20:25:40.544 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.111539+0000', 'last_change': '2024-03-18T20:25:34.111539+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.111539+0000', 'last_undegraded': '2024-03-18T20:25:34.111539+0000', 'last_fullsized': '2024-03-18T20:25:34.111539+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-19T21:46:33.109951+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-18T20:25:40.544 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.f is not active+clean 2024-03-18T20:25:40.544 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.544 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.d is not active+clean 2024-03-18T20:25:40.544 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.111594+0000', 'last_change': '2024-03-18T20:25:34.111594+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.111594+0000', 'last_undegraded': '2024-03-18T20:25:34.111594+0000', 'last_fullsized': '2024-03-18T20:25:34.111594+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T02:24:13.917940+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-18T20:25:40.545 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.e is not active+clean 2024-03-18T20:25:40.545 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.545 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.a is not active+clean 2024-03-18T20:25:40.545 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.117145+0000', 'last_change': '2024-03-18T20:25:34.117145+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.117145+0000', 'last_undegraded': '2024-03-18T20:25:34.117145+0000', 'last_fullsized': '2024-03-18T20:25:34.117145+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T01:09:38.721170+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-18T20:25:40.545 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.9 is not active+clean 2024-03-18T20:25:40.545 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.545 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.b is not active+clean 2024-03-18T20:25:40.545 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.112636+0000', 'last_change': '2024-03-18T20:25:34.112636+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.112636+0000', 'last_undegraded': '2024-03-18T20:25:34.112636+0000', 'last_fullsized': '2024-03-18T20:25:34.112636+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T04:20:15.369281+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-18T20:25:40.545 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.8 is not active+clean 2024-03-18T20:25:40.545 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.546 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.8 is not active+clean 2024-03-18T20:25:40.546 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.115166+0000', 'last_change': '2024-03-18T20:25:34.115166+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.115166+0000', 'last_undegraded': '2024-03-18T20:25:34.115166+0000', 'last_fullsized': '2024-03-18T20:25:34.115166+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T02:02:17.295248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-18T20:25:40.546 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.b is not active+clean 2024-03-18T20:25:40.546 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.546 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.9 is not active+clean 2024-03-18T20:25:40.546 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.116524+0000', 'last_change': '2024-03-18T20:25:34.116524+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.116524+0000', 'last_undegraded': '2024-03-18T20:25:34.116524+0000', 'last_fullsized': '2024-03-18T20:25:34.116524+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T02:38:50.264082+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-18T20:25:40.546 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.a is not active+clean 2024-03-18T20:25:40.546 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.547 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.16 is not active+clean 2024-03-18T20:25:40.547 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.116256+0000', 'last_change': '2024-03-18T20:25:34.116256+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.116256+0000', 'last_undegraded': '2024-03-18T20:25:34.116256+0000', 'last_fullsized': '2024-03-18T20:25:34.116256+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T02:07:50.170851+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-18T20:25:40.547 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.15 is not active+clean 2024-03-18T20:25:40.547 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.547 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.17 is not active+clean 2024-03-18T20:25:40.547 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.114153+0000', 'last_change': '2024-03-18T20:25:34.114153+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.114153+0000', 'last_undegraded': '2024-03-18T20:25:34.114153+0000', 'last_fullsized': '2024-03-18T20:25:34.114153+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T01:45:27.179302+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-18T20:25:40.547 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.14 is not active+clean 2024-03-18T20:25:40.547 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.547 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.14 is not active+clean 2024-03-18T20:25:40.547 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.112308+0000', 'last_change': '2024-03-18T20:25:34.112308+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.112308+0000', 'last_undegraded': '2024-03-18T20:25:34.112308+0000', 'last_fullsized': '2024-03-18T20:25:34.112308+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T02:02:03.410994+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-18T20:25:40.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.17 is not active+clean 2024-03-18T20:25:40.548 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.15 is not active+clean 2024-03-18T20:25:40.548 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.113878+0000', 'last_change': '2024-03-18T20:25:34.113878+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.113878+0000', 'last_undegraded': '2024-03-18T20:25:34.113878+0000', 'last_fullsized': '2024-03-18T20:25:34.113878+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T00:16:02.016045+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-18T20:25:40.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.16 is not active+clean 2024-03-18T20:25:40.548 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.12 is not active+clean 2024-03-18T20:25:40.548 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.119645+0000', 'last_change': '2024-03-18T20:25:34.119645+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.119645+0000', 'last_undegraded': '2024-03-18T20:25:34.119645+0000', 'last_fullsized': '2024-03-18T20:25:34.119645+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T07:36:50.971852+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-18T20:25:40.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.11 is not active+clean 2024-03-18T20:25:40.548 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.13 is not active+clean 2024-03-18T20:25:40.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.114296+0000', 'last_change': '2024-03-18T20:25:34.114296+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.114296+0000', 'last_undegraded': '2024-03-18T20:25:34.114296+0000', 'last_fullsized': '2024-03-18T20:25:34.114296+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T07:32:42.191508+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-18T20:25:40.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.10 is not active+clean 2024-03-18T20:25:40.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.10 is not active+clean 2024-03-18T20:25:40.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.112662+0000', 'last_change': '2024-03-18T20:25:34.112662+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.112662+0000', 'last_undegraded': '2024-03-18T20:25:34.112662+0000', 'last_fullsized': '2024-03-18T20:25:34.112662+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T02:07:50.170851+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-18T20:25:40.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.13 is not active+clean 2024-03-18T20:25:40.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.11 is not active+clean 2024-03-18T20:25:40.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.110675+0000', 'last_change': '2024-03-18T20:25:34.110675+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.110675+0000', 'last_undegraded': '2024-03-18T20:25:34.110675+0000', 'last_fullsized': '2024-03-18T20:25:34.110675+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T07:56:20.466978+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-18T20:25:40.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.12 is not active+clean 2024-03-18T20:25:40.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1e is not active+clean 2024-03-18T20:25:40.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.113196+0000', 'last_change': '2024-03-18T20:25:34.113196+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.113196+0000', 'last_undegraded': '2024-03-18T20:25:34.113196+0000', 'last_fullsized': '2024-03-18T20:25:34.113196+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-19T21:52:39.087869+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-18T20:25:40.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1d is not active+clean 2024-03-18T20:25:40.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1f is not active+clean 2024-03-18T20:25:40.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.115119+0000', 'last_change': '2024-03-18T20:25:34.115119+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.115119+0000', 'last_undegraded': '2024-03-18T20:25:34.115119+0000', 'last_fullsized': '2024-03-18T20:25:34.115119+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T08:13:25.724222+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-18T20:25:40.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1c is not active+clean 2024-03-18T20:25:40.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1d is not active+clean 2024-03-18T20:25:40.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.111795+0000', 'last_change': '2024-03-18T20:25:34.111795+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.111795+0000', 'last_undegraded': '2024-03-18T20:25:34.111795+0000', 'last_fullsized': '2024-03-18T20:25:34.111795+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T01:48:45.664606+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-18T20:25:40.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1e is not active+clean 2024-03-18T20:25:40.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1a is not active+clean 2024-03-18T20:25:40.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.114019+0000', 'last_change': '2024-03-18T20:25:34.114019+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.114019+0000', 'last_undegraded': '2024-03-18T20:25:34.114019+0000', 'last_fullsized': '2024-03-18T20:25:34.114019+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T07:55:33.630861+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-18T20:25:40.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.19 is not active+clean 2024-03-18T20:25:40.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1b is not active+clean 2024-03-18T20:25:40.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.117626+0000', 'last_change': '2024-03-18T20:25:34.117626+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.117626+0000', 'last_undegraded': '2024-03-18T20:25:34.117626+0000', 'last_fullsized': '2024-03-18T20:25:34.117626+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T04:19:02.788524+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-18T20:25:40.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.18 is not active+clean 2024-03-18T20:25:40.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.18 is not active+clean 2024-03-18T20:25:40.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.113361+0000', 'last_change': '2024-03-18T20:25:34.113361+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.113361+0000', 'last_undegraded': '2024-03-18T20:25:34.113361+0000', 'last_fullsized': '2024-03-18T20:25:34.113361+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T05:12:10.985415+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-18T20:25:40.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1b is not active+clean 2024-03-18T20:25:40.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.19 is not active+clean 2024-03-18T20:25:40.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 233, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:25:34.115690+0000', 'last_change': '2024-03-18T20:25:34.115690+0000', 'last_active': '2024-03-18T20:25:34.104842+0000', 'last_peered': '2024-03-18T20:25:34.104842+0000', 'last_clean': '2024-03-18T20:25:34.104842+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:34.115690+0000', 'last_undegraded': '2024-03-18T20:25:34.115690+0000', 'last_fullsized': '2024-03-18T20:25:34.115690+0000', 'mapping_epoch': 233, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 233, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:34.104842+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T08:13:25.724222+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-18T20:25:40.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1a is not active+clean 2024-03-18T20:25:40.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.22 is not active+clean 2024-03-18T20:25:40.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.23 is not active+clean 2024-03-18T20:25:40.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.20 is not active+clean 2024-03-18T20:25:40.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.21 is not active+clean 2024-03-18T20:25:40.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.26 is not active+clean 2024-03-18T20:25:40.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.27 is not active+clean 2024-03-18T20:25:40.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.24 is not active+clean 2024-03-18T20:25:40.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.25 is not active+clean 2024-03-18T20:25:40.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2a is not active+clean 2024-03-18T20:25:40.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2b is not active+clean 2024-03-18T20:25:40.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.28 is not active+clean 2024-03-18T20:25:40.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.29 is not active+clean 2024-03-18T20:25:40.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2e is not active+clean 2024-03-18T20:25:40.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2f is not active+clean 2024-03-18T20:25:40.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2c is not active+clean 2024-03-18T20:25:40.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2d is not active+clean 2024-03-18T20:25:40.556 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.556 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.32 is not active+clean 2024-03-18T20:25:40.556 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.556 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.33 is not active+clean 2024-03-18T20:25:40.556 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.556 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.30 is not active+clean 2024-03-18T20:25:40.556 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.556 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.31 is not active+clean 2024-03-18T20:25:40.556 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.556 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.36 is not active+clean 2024-03-18T20:25:40.556 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.557 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.37 is not active+clean 2024-03-18T20:25:40.557 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.557 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.34 is not active+clean 2024-03-18T20:25:40.557 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.557 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.35 is not active+clean 2024-03-18T20:25:40.557 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.557 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3a is not active+clean 2024-03-18T20:25:40.557 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.557 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3b is not active+clean 2024-03-18T20:25:40.557 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.558 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.38 is not active+clean 2024-03-18T20:25:40.558 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.558 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.39 is not active+clean 2024-03-18T20:25:40.558 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.558 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3e is not active+clean 2024-03-18T20:25:40.558 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.558 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3f is not active+clean 2024-03-18T20:25:40.558 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.558 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3c is not active+clean 2024-03-18T20:25:40.558 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.558 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3d is not active+clean 2024-03-18T20:25:40.559 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:25:35.113201+0000', 'last_change': '2024-03-18T20:25:35.113201+0000', 'last_active': '2024-03-18T20:25:35.113201+0000', 'last_peered': '2024-03-18T20:25:35.113201+0000', 'last_clean': '2024-03-18T20:25:35.113201+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:25:35.113201+0000', 'last_undegraded': '2024-03-18T20:25:35.113201+0000', 'last_fullsized': '2024-03-18T20:25:35.113201+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'last_clean_scrub_stamp': '2024-03-18T20:25:35.113201+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:25:40.559 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-18T20:25:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.691+0000 7f7803f51700 1 Processor -- start 2024-03-18T20:25:40.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.692+0000 7f7803f51700 1 -- start start 2024-03-18T20:25:40.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.692+0000 7f7803f51700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77fc15b4d0 0x7f77fc09a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.692+0000 7f7803f51700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f77fc0cf750 0x7f77fc099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.692+0000 7f7803f51700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77fc06f070 con 0x7f77fc099f80 2024-03-18T20:25:40.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.693+0000 7f7803f51700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77fc09a410 con 0x7f77fc0cf750 2024-03-18T20:25:40.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.693+0000 7f7803f51700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc0d0260 con 0x7f77fc15b4d0 2024-03-18T20:25:40.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.693+0000 7f78014ec700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f77fc0cf750 0x7f77fc099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:40.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.693+0000 7f78024ee700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f77fc099f80 0x7f77fc0cbd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51536/0 (socket says 172.21.15.1:51536) 2024-03-18T20:25:40.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.693+0000 7f78024ee700 1 -- 172.21.15.1:0/164550553 learned_addr learned my addr 172.21.15.1:0/164550553 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:40.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.693+0000 7f78014ec700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f77fc0cf750 0x7f77fc099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49878/0 (socket says 172.21.15.1:49878) 2024-03-18T20:25:40.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.693+0000 7f7801ced700 1 --2- 172.21.15.1:0/164550553 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77fc15b4d0 0x7f77fc09a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:40.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.693+0000 7f78014ec700 1 -- 172.21.15.1:0/164550553 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77fc15b4d0 msgr2=0x7f77fc09a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:40.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.693+0000 7f78014ec700 1 --2- 172.21.15.1:0/164550553 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77fc15b4d0 0x7f77fc09a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.693+0000 7f78014ec700 1 -- 172.21.15.1:0/164550553 >> v1:172.21.15.1:6789/0 conn(0x7f77fc099f80 legacy=0x7f77fc0cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.693+0000 7f78014ec700 1 -- 172.21.15.1:0/164550553 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77fc0cff80 con 0x7f77fc0cf750 2024-03-18T20:25:40.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.693+0000 7f7800ceb700 1 -- 172.21.15.1:0/164550553 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490848600 0 0) 0x7f77fc06f070 con 0x7f77fc099f80 2024-03-18T20:25:40.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.693+0000 7f78014ec700 1 --2- 172.21.15.1:0/164550553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77fc0cf750 0x7f77fc099a40 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f77f0009840 tx=0x7f77f0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f664ac3bbd404c38 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.693+0000 7f7800ceb700 1 -- 172.21.15.1:0/164550553 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f0011830 con 0x7f77fc0cf750 2024-03-18T20:25:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.693+0000 7f7800ceb700 1 -- 172.21.15.1:0/164550553 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f0011990 con 0x7f77fc0cf750 2024-03-18T20:25:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.693+0000 7f7800ceb700 1 -- 172.21.15.1:0/164550553 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f0011d20 con 0x7f77fc0cf750 2024-03-18T20:25:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.694+0000 7f7803f51700 1 -- 172.21.15.1:0/164550553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77fc0cf750 msgr2=0x7f77fc099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.694+0000 7f7803f51700 1 --2- 172.21.15.1:0/164550553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77fc0cf750 0x7f77fc099a40 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f77f0009840 tx=0x7f77f0009b50 comp rx=0 tx=0).stop 2024-03-18T20:25:40.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.694+0000 7f7803f51700 1 -- 172.21.15.1:0/164550553 shutdown_connections 2024-03-18T20:25:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.694+0000 7f7803f51700 1 --2- 172.21.15.1:0/164550553 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77fc15b4d0 0x7f77fc09a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.694+0000 7f7803f51700 1 --2- 172.21.15.1:0/164550553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77fc0cf750 0x7f77fc099a40 unknown :-1 s=CLOSED pgs=2587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.694+0000 7f7803f51700 1 -- 172.21.15.1:0/164550553 >> 172.21.15.1:0/164550553 conn(0x7f77fc0b2cb0 msgr2=0x7f77fc0b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.694+0000 7f7803f51700 1 -- 172.21.15.1:0/164550553 shutdown_connections 2024-03-18T20:25:40.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.694+0000 7f7803f51700 1 -- 172.21.15.1:0/164550553 wait complete. 2024-03-18T20:25:40.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.695+0000 7f7803f51700 1 Processor -- start 2024-03-18T20:25:40.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.695+0000 7f7803f51700 1 -- start start 2024-03-18T20:25:40.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.696+0000 7f7803f51700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77fc099f80 0x7f77fc0af220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.696+0000 7f7803f51700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77fc0cf750 0x7f77fc1f1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.696+0000 7f7803f51700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77fc15b4d0 0x7f77fc1f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.696+0000 7f7803f51700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc0af760 con 0x7f77fc0cf750 2024-03-18T20:25:40.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.696+0000 7f7803f51700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc1f18c0 con 0x7f77fc15b4d0 2024-03-18T20:25:40.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.696+0000 7f7803f51700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc0ae180 con 0x7f77fc099f80 2024-03-18T20:25:40.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.696+0000 7f78014ec700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77fc0cf750 0x7f77fc1f1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:40.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.696+0000 7f78014ec700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77fc0cf750 0x7f77fc1f1380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49888/0 (socket says 172.21.15.1:49888) 2024-03-18T20:25:40.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.696+0000 7f78014ec700 1 -- 172.21.15.1:0/1801543975 learned_addr learned my addr 172.21.15.1:0/1801543975 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:40.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.696+0000 7f7801ced700 1 --2- 172.21.15.1:0/1801543975 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77fc099f80 0x7f77fc0af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:40.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.696+0000 7f78024ee700 1 --2- 172.21.15.1:0/1801543975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77fc15b4d0 0x7f77fc1f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:40.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.696+0000 7f78014ec700 1 -- 172.21.15.1:0/1801543975 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77fc099f80 msgr2=0x7f77fc0af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.696+0000 7f78014ec700 1 --2- 172.21.15.1:0/1801543975 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77fc099f80 0x7f77fc0af220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.696+0000 7f78014ec700 1 -- 172.21.15.1:0/1801543975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77fc15b4d0 msgr2=0x7f77fc1f3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:40.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.696+0000 7f78014ec700 1 --2- 172.21.15.1:0/1801543975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77fc15b4d0 0x7f77fc1f3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:40.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.696+0000 7f78014ec700 1 -- 172.21.15.1:0/1801543975 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f00094a0 con 0x7f77fc0cf750 2024-03-18T20:25:40.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.697+0000 7f78014ec700 1 --2- 172.21.15.1:0/1801543975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77fc0cf750 0x7f77fc1f1380 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f77f00090b0 tx=0x7f77f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:40.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.697+0000 7f77ee7fc700 1 -- 172.21.15.1:0/1801543975 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f00036a0 con 0x7f77fc0cf750 2024-03-18T20:25:40.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.697+0000 7f77ee7fc700 1 -- 172.21.15.1:0/1801543975 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f001f070 con 0x7f77fc0cf750 2024-03-18T20:25:40.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.697+0000 7f7803f51700 1 -- 172.21.15.1:0/1801543975 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77fc0ae460 con 0x7f77fc0cf750 2024-03-18T20:25:40.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.697+0000 7f77ee7fc700 1 -- 172.21.15.1:0/1801543975 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f0003a60 con 0x7f77fc0cf750 2024-03-18T20:25:40.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.697+0000 7f7803f51700 1 -- 172.21.15.1:0/1801543975 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77fc0aea20 con 0x7f77fc0cf750 2024-03-18T20:25:40.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.698+0000 7f7803f51700 1 -- 172.21.15.1:0/1801543975 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77fc068240 con 0x7f77fc0cf750 2024-03-18T20:25:40.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.698+0000 7f77ee7fc700 1 -- 172.21.15.1:0/1801543975 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f77f0003bc0 con 0x7f77fc0cf750 2024-03-18T20:25:40.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.699+0000 7f77ee7fc700 1 --2- 172.21.15.1:0/1801543975 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f77e006adb0 0x7f77e006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:40.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.699+0000 7f77ee7fc700 1 -- 172.21.15.1:0/1801543975 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f77f0021070 con 0x7f77fc0cf750 2024-03-18T20:25:40.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.699+0000 7f7801ced700 1 --2- 172.21.15.1:0/1801543975 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f77e006adb0 0x7f77e006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:40.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.699+0000 7f7801ced700 1 --2- 172.21.15.1:0/1801543975 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f77e006adb0 0x7f77e006d270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f77f8006b20 tx=0x7f77f8008040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:40.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.703+0000 7f77ee7fc700 1 -- 172.21.15.1:0/1801543975 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f77f0055e00 con 0x7f77fc0cf750 2024-03-18T20:25:40.739 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:25:40.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:25:41.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.996+0000 7f7803f51700 1 -- 172.21.15.1:0/1801543975 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f77fc0676a0 con 0x7f77fc0cf750 2024-03-18T20:25:41.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.996+0000 7f77ee7fc700 1 -- 172.21.15.1:0/1801543975 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1285 (secure 0 0 0) 0x7f77f0015520 con 0x7f77fc0cf750 2024-03-18T20:25:41.001 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:41.001 INFO:teuthology.orchestra.run.smithi001.stdout:{"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":5773,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":236,"num_osds":8,"num_up_osds":8,"osd_up_since":1710787777,"num_in_osds":8,"osd_in_since":1710787769,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97},{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32}],"num_pgs":193,"num_pools":5,"num_objects":24,"data_bytes":593988,"bytes_used":394633216,"bytes_avail":772699480064,"bytes_total":773094113280,"unknown_pgs_ratio":0.33160620927810669,"inactive_pgs_ratio":0.16580310463905334,"write_bytes_sec":204,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":110,"id":13,"up":1,"in":1,"max":1,"id":14,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":14,"rank":0,"name":"c","status":"up:active","gid":11547},{"filesystem_id":13,"rank":0,"name":"d","status":"up:active","gid":11556}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-03-18T20:24:56.871609+0000","services":{}},"progress_events":{}} 2024-03-18T20:25:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.999+0000 7f7803f51700 1 -- 172.21.15.1:0/1801543975 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f77e006adb0 msgr2=0x7f77e006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.999+0000 7f7803f51700 1 --2- 172.21.15.1:0/1801543975 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f77e006adb0 0x7f77e006d270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f77f8006b20 tx=0x7f77f8008040 comp rx=0 tx=0).stop 2024-03-18T20:25:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.999+0000 7f7803f51700 1 -- 172.21.15.1:0/1801543975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77fc0cf750 msgr2=0x7f77fc1f1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.999+0000 7f7803f51700 1 --2- 172.21.15.1:0/1801543975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77fc0cf750 0x7f77fc1f1380 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f77f00090b0 tx=0x7f77f0004750 comp rx=0 tx=0).stop 2024-03-18T20:25:41.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.999+0000 7f7803f51700 1 -- 172.21.15.1:0/1801543975 shutdown_connections 2024-03-18T20:25:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.999+0000 7f7803f51700 1 --2- 172.21.15.1:0/1801543975 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f77e006adb0 0x7f77e006d270 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.999+0000 7f7803f51700 1 --2- 172.21.15.1:0/1801543975 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77fc099f80 0x7f77fc0af220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.999+0000 7f7803f51700 1 --2- 172.21.15.1:0/1801543975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77fc15b4d0 0x7f77fc1f3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.999+0000 7f7803f51700 1 --2- 172.21.15.1:0/1801543975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77fc0cf750 0x7f77fc1f1380 unknown :-1 s=CLOSED pgs=2588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.999+0000 7f7803f51700 1 -- 172.21.15.1:0/1801543975 >> 172.21.15.1:0/1801543975 conn(0x7f77fc0b2cb0 msgr2=0x7f77fc06f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.999+0000 7f7803f51700 1 -- 172.21.15.1:0/1801543975 shutdown_connections 2024-03-18T20:25:41.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:40.999+0000 7f7803f51700 1 -- 172.21.15.1:0/1801543975 wait complete. 2024-03-18T20:25:41.015 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-18T20:25:41.015 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:25:41.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.136+0000 7f7e787dd700 1 Processor -- start 2024-03-18T20:25:41.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.136+0000 7f7e787dd700 1 -- start start 2024-03-18T20:25:41.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e787dd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7e700cdc00 0x7f7e70161740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e787dd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e70161c80 0x7f7e70071c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e787dd700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e700cfd20 con 0x7f7e70161360 2024-03-18T20:25:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e787dd700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e700717c0 con 0x7f7e700cdc00 2024-03-18T20:25:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e787dd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7015da00 con 0x7f7e70161c80 2024-03-18T20:25:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e76579700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7e70161360 0x7f7e7006b130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51548/0 (socket says 172.21.15.1:51548) 2024-03-18T20:25:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e76579700 1 -- 172.21.15.1:0/3946950905 learned_addr learned my addr 172.21.15.1:0/3946950905 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e75d78700 1 --2- 172.21.15.1:0/3946950905 >> v2:172.21.15.165:3300/0 conn(0x7f7e700cdc00 0x7f7e70161740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e75577700 1 -- 172.21.15.1:0/3946950905 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50876904 0 0) 0x7f7e700cfd20 con 0x7f7e70161360 2024-03-18T20:25:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e75577700 1 -- 172.21.15.1:0/3946950905 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e5c0035b0 con 0x7f7e70161360 2024-03-18T20:25:41.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e76d7a700 1 --2- 172.21.15.1:0/3946950905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e70161c80 0x7f7e70071c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e75577700 1 -- 172.21.15.1:0/3946950905 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1883865481 0 0) 0x7f7e5c0035b0 con 0x7f7e70161360 2024-03-18T20:25:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e75577700 1 -- 172.21.15.1:0/3946950905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e70161c80 msgr2=0x7f7e70071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e75577700 1 --2- 172.21.15.1:0/3946950905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e70161c80 0x7f7e70071c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e75577700 1 -- 172.21.15.1:0/3946950905 >> v2:172.21.15.165:3300/0 conn(0x7f7e700cdc00 msgr2=0x7f7e70161740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e75577700 1 --2- 172.21.15.1:0/3946950905 >> v2:172.21.15.165:3300/0 conn(0x7f7e700cdc00 0x7f7e70161740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e75577700 1 -- 172.21.15.1:0/3946950905 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e700cc8b0 con 0x7f7e70161360 2024-03-18T20:25:41.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e75577700 1 -- 172.21.15.1:0/3946950905 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f7e6c001670 con 0x7f7e70161360 2024-03-18T20:25:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e75577700 1 -- 172.21.15.1:0/3946950905 >> v1:172.21.15.1:6789/0 conn(0x7f7e70161360 legacy=0x7f7e7006b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e75577700 1 --2- 172.21.15.1:0/3946950905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e70161c80 0x7f7e5c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.137+0000 7f7e75577700 1 --2- 172.21.15.1:0/3946950905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e5c003fd0 0x7f7e5c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.138+0000 7f7e75577700 1 --2- 172.21.15.1:0/3946950905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e5c004d90 0x7f7e5c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.138+0000 7f7e75577700 1 -- 172.21.15.1:0/3946950905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e700717c0 con 0x7f7e5c003fd0 2024-03-18T20:25:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.138+0000 7f7e75577700 1 -- 172.21.15.1:0/3946950905 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7015da00 con 0x7f7e5c004d90 2024-03-18T20:25:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.138+0000 7f7e75577700 1 -- 172.21.15.1:0/3946950905 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c00bbf0 con 0x7f7e70161c80 2024-03-18T20:25:41.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.138+0000 7f7e75d78700 1 --2- 172.21.15.1:0/3946950905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e5c003fd0 0x7f7e5c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.138+0000 7f7e76579700 1 --2- 172.21.15.1:0/3946950905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e70161c80 0x7f7e5c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.138+0000 7f7e76d7a700 1 --2- 172.21.15.1:0/3946950905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e5c004d90 0x7f7e5c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.138+0000 7f7e75d78700 1 -- 172.21.15.1:0/3946950905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e70161c80 msgr2=0x7f7e5c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.138+0000 7f7e75d78700 1 --2- 172.21.15.1:0/3946950905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e70161c80 0x7f7e5c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.138+0000 7f7e75d78700 1 -- 172.21.15.1:0/3946950905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e5c004d90 msgr2=0x7f7e5c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.138+0000 7f7e75d78700 1 --2- 172.21.15.1:0/3946950905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e5c004d90 0x7f7e5c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.138+0000 7f7e75d78700 1 -- 172.21.15.1:0/3946950905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5c0035b0 con 0x7f7e5c003fd0 2024-03-18T20:25:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.138+0000 7f7e75d78700 1 --2- 172.21.15.1:0/3946950905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e5c003fd0 0x7f7e5c004850 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 crypto rx=0x7f7e60009b60 tx=0x7f7e5c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30bc4cfcf1d46d6c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.138+0000 7f7e75577700 1 -- 172.21.15.1:0/3946950905 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6000e6c0 con 0x7f7e5c003fd0 2024-03-18T20:25:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.138+0000 7f7e75577700 1 -- 172.21.15.1:0/3946950905 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6000e820 con 0x7f7e5c003fd0 2024-03-18T20:25:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.138+0000 7f7e75577700 1 -- 172.21.15.1:0/3946950905 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6000eb90 con 0x7f7e5c003fd0 2024-03-18T20:25:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.139+0000 7f7e787dd700 1 -- 172.21.15.1:0/3946950905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e5c003fd0 msgr2=0x7f7e5c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.139+0000 7f7e787dd700 1 --2- 172.21.15.1:0/3946950905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e5c003fd0 0x7f7e5c004850 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 crypto rx=0x7f7e60009b60 tx=0x7f7e5c005880 comp rx=0 tx=0).stop 2024-03-18T20:25:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.139+0000 7f7e76d7a700 1 -- 172.21.15.1:0/3946950905 reap_dead start 2024-03-18T20:25:41.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.139+0000 7f7e787dd700 1 -- 172.21.15.1:0/3946950905 shutdown_connections 2024-03-18T20:25:41.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.139+0000 7f7e787dd700 1 -- 172.21.15.1:0/3946950905 >> 172.21.15.1:0/3946950905 conn(0x7f7e700ccab0 msgr2=0x7f7e700ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:41.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.139+0000 7f7e787dd700 1 -- 172.21.15.1:0/3946950905 shutdown_connections 2024-03-18T20:25:41.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.139+0000 7f7e787dd700 1 -- 172.21.15.1:0/3946950905 wait complete. 2024-03-18T20:25:41.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.140+0000 7f7e787dd700 1 Processor -- start 2024-03-18T20:25:41.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.140+0000 7f7e787dd700 1 -- start start 2024-03-18T20:25:41.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.141+0000 7f7e787dd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e5c003fd0 0x7f7e7015eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.141+0000 7f7e787dd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e700b56d0 0x7f7e7015bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.141+0000 7f7e787dd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e7009ed60 0x7f7e700a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.141+0000 7f7e787dd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7015c090 con 0x7f7e7009ed60 2024-03-18T20:25:41.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.141+0000 7f7e787dd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7009f8a0 con 0x7f7e5c003fd0 2024-03-18T20:25:41.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.141+0000 7f7e787dd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7009f5c0 con 0x7f7e700b56d0 2024-03-18T20:25:41.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.141+0000 7f7e76d7a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e7009ed60 0x7f7e700a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:41.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.141+0000 7f7e76d7a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e7009ed60 0x7f7e700a0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49910/0 (socket says 172.21.15.1:49910) 2024-03-18T20:25:41.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.141+0000 7f7e76d7a700 1 -- 172.21.15.1:0/2133926623 learned_addr learned my addr 172.21.15.1:0/2133926623 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:41.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.141+0000 7f7e75d78700 1 --2- 172.21.15.1:0/2133926623 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e700b56d0 0x7f7e7015bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.141+0000 7f7e76579700 1 --2- 172.21.15.1:0/2133926623 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e5c003fd0 0x7f7e7015eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.141+0000 7f7e76d7a700 1 -- 172.21.15.1:0/2133926623 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e700b56d0 msgr2=0x7f7e7015bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.141+0000 7f7e76d7a700 1 --2- 172.21.15.1:0/2133926623 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e700b56d0 0x7f7e7015bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.141+0000 7f7e76d7a700 1 -- 172.21.15.1:0/2133926623 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e5c003fd0 msgr2=0x7f7e7015eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.141+0000 7f7e76d7a700 1 --2- 172.21.15.1:0/2133926623 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e5c003fd0 0x7f7e7015eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.141+0000 7f7e76d7a700 1 -- 172.21.15.1:0/2133926623 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e700b2d10 con 0x7f7e7009ed60 2024-03-18T20:25:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.141+0000 7f7e76579700 1 --2- 172.21.15.1:0/2133926623 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e5c003fd0 0x7f7e7015eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.141+0000 7f7e76d7a700 1 --2- 172.21.15.1:0/2133926623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e7009ed60 0x7f7e700a0430 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f7e700715b0 tx=0x7f7e680048c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.142+0000 7f7e66ffd700 1 -- 172.21.15.1:0/2133926623 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e680079c0 con 0x7f7e7009ed60 2024-03-18T20:25:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.142+0000 7f7e66ffd700 1 -- 172.21.15.1:0/2133926623 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e68007b20 con 0x7f7e7009ed60 2024-03-18T20:25:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.142+0000 7f7e787dd700 1 -- 172.21.15.1:0/2133926623 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e700b2ff0 con 0x7f7e7009ed60 2024-03-18T20:25:41.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.142+0000 7f7e66ffd700 1 -- 172.21.15.1:0/2133926623 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68015420 con 0x7f7e7009ed60 2024-03-18T20:25:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.142+0000 7f7e787dd700 1 -- 172.21.15.1:0/2133926623 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e7009f1e0 con 0x7f7e7009ed60 2024-03-18T20:25:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.143+0000 7f7e787dd700 1 -- 172.21.15.1:0/2133926623 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e700676a0 con 0x7f7e7009ed60 2024-03-18T20:25:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.143+0000 7f7e66ffd700 1 -- 172.21.15.1:0/2133926623 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7e68015580 con 0x7f7e7009ed60 2024-03-18T20:25:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.144+0000 7f7e66ffd700 1 --2- 172.21.15.1:0/2133926623 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7e5406adb0 0x7f7e5406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.144+0000 7f7e66ffd700 1 -- 172.21.15.1:0/2133926623 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f7e68089ce0 con 0x7f7e7009ed60 2024-03-18T20:25:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.144+0000 7f7e76579700 1 --2- 172.21.15.1:0/2133926623 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7e5406adb0 0x7f7e5406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.144+0000 7f7e76579700 1 --2- 172.21.15.1:0/2133926623 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7e5406adb0 0x7f7e5406d270 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c007fd0 tx=0x7f7e6c009040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.148+0000 7f7e66ffd700 1 -- 172.21.15.1:0/2133926623 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7e68054930 con 0x7f7e7009ed60 2024-03-18T20:25:41.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.349+0000 7f7e787dd700 1 -- 172.21.15.1:0/2133926623 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7e7009f3a0 con 0x7f7e5406adb0 2024-03-18T20:25:41.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.363+0000 7f7e66ffd700 1 -- 172.21.15.1:0/2133926623 <== mgr.11322 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+529167 (secure 0 0 0) 0x7f7e6c007e30 con 0x7f7e5406adb0 2024-03-18T20:25:41.374 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.368+0000 7f7e787dd700 1 -- 172.21.15.1:0/2133926623 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7e5406adb0 msgr2=0x7f7e5406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.368+0000 7f7e787dd700 1 --2- 172.21.15.1:0/2133926623 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7e5406adb0 0x7f7e5406d270 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c007fd0 tx=0x7f7e6c009040 comp rx=0 tx=0).stop 2024-03-18T20:25:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.368+0000 7f7e787dd700 1 -- 172.21.15.1:0/2133926623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e7009ed60 msgr2=0x7f7e700a0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.368+0000 7f7e787dd700 1 --2- 172.21.15.1:0/2133926623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e7009ed60 0x7f7e700a0430 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f7e700715b0 tx=0x7f7e680048c0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.368+0000 7f7e787dd700 1 -- 172.21.15.1:0/2133926623 shutdown_connections 2024-03-18T20:25:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.368+0000 7f7e787dd700 1 --2- 172.21.15.1:0/2133926623 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7e5406adb0 0x7f7e5406d270 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.368+0000 7f7e787dd700 1 --2- 172.21.15.1:0/2133926623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e7009ed60 0x7f7e700a0430 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.368+0000 7f7e787dd700 1 --2- 172.21.15.1:0/2133926623 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e5c003fd0 0x7f7e7015eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.368+0000 7f7e787dd700 1 --2- 172.21.15.1:0/2133926623 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7e700b56d0 0x7f7e7015bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:41.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.368+0000 7f7e787dd700 1 -- 172.21.15.1:0/2133926623 >> 172.21.15.1:0/2133926623 conn(0x7f7e700ccab0 msgr2=0x7f7e70071c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:41.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.368+0000 7f7e787dd700 1 -- 172.21.15.1:0/2133926623 shutdown_connections 2024-03-18T20:25:41.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:41.368+0000 7f7e787dd700 1 -- 172.21.15.1:0/2133926623 wait complete. 2024-03-18T20:25:41.380 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:25:41.384 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-03-18T20:25:41.103109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596350,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":671,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":428,"ondisk_log_size":428,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":385756,"kb_used_data":24424,"kb_used_omap":12,"kb_used_meta":361203,"kb_avail":754588964,"statfs":{"total":773094113280,"available":772699099136,"internally_reserved":0,"allocated":25010176,"data_stored":13379452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":369872441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3177,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001341"},"pg_stats":[{"pgid":"1.0","version":"221'348","reported_seq":1204,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561810+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:25:35.561810+0000","last_peered":"2024-03-18T20:25:35.561810+0000","last_clean":"2024-03-18T20:25:35.561810+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:25:35.561810+0000","last_undegraded":"2024-03-18T20:25:35.561810+0000","last_fullsized":"2024-03-18T20:25:35.561810+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"231'1","reported_seq":16,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560978+0000","last_change":"2024-03-18T20:25:22.248248+0000","last_active":"2024-03-18T20:25:35.560978+0000","last_peered":"2024-03-18T20:25:35.560978+0000","last_clean":"2024-03-18T20:25:35.560978+0000","last_became_active":"2024-03-18T20:25:22.247970+0000","last_became_peered":"2024-03-18T20:25:22.247970+0000","last_unstale":"2024-03-18T20:25:35.560978+0000","last_undegraded":"2024-03-18T20:25:35.560978+0000","last_fullsized":"2024-03-18T20:25:35.560978+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:01:50.523530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.572089+0000","last_change":"2024-03-18T20:25:22.687598+0000","last_active":"2024-03-18T20:25:35.572089+0000","last_peered":"2024-03-18T20:25:35.572089+0000","last_clean":"2024-03-18T20:25:35.572089+0000","last_became_active":"2024-03-18T20:25:22.686962+0000","last_became_peered":"2024-03-18T20:25:22.686962+0000","last_unstale":"2024-03-18T20:25:35.572089+0000","last_undegraded":"2024-03-18T20:25:35.572089+0000","last_fullsized":"2024-03-18T20:25:35.572089+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:23:10.983253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"235'2","reported_seq":13,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.583521+0000","last_change":"2024-03-18T20:25:35.139062+0000","last_active":"2024-03-18T20:25:35.583521+0000","last_peered":"2024-03-18T20:25:35.583521+0000","last_clean":"2024-03-18T20:25:35.583521+0000","last_became_active":"2024-03-18T20:25:35.138799+0000","last_became_peered":"2024-03-18T20:25:35.138799+0000","last_unstale":"2024-03-18T20:25:35.583521+0000","last_undegraded":"2024-03-18T20:25:35.583521+0000","last_fullsized":"2024-03-18T20:25:35.583521+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:47.366224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188735+0000","last_change":"2024-03-18T20:25:35.575458+0000","last_active":"2024-03-18T20:25:38.188735+0000","last_peered":"2024-03-18T20:25:38.188735+0000","last_clean":"2024-03-18T20:25:38.188735+0000","last_became_active":"2024-03-18T20:25:35.574984+0000","last_became_peered":"2024-03-18T20:25:35.574984+0000","last_unstale":"2024-03-18T20:25:38.188735+0000","last_undegraded":"2024-03-18T20:25:38.188735+0000","last_fullsized":"2024-03-18T20:25:38.188735+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:26.055273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.564794+0000","last_change":"2024-03-18T20:25:22.252803+0000","last_active":"2024-03-18T20:25:35.564794+0000","last_peered":"2024-03-18T20:25:35.564794+0000","last_clean":"2024-03-18T20:25:35.564794+0000","last_became_active":"2024-03-18T20:25:22.252169+0000","last_became_peered":"2024-03-18T20:25:22.252169+0000","last_unstale":"2024-03-18T20:25:35.564794+0000","last_undegraded":"2024-03-18T20:25:35.564794+0000","last_fullsized":"2024-03-18T20:25:35.564794+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:12.985428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.555564+0000","last_change":"2024-03-18T20:25:22.696025+0000","last_active":"2024-03-18T20:25:35.555564+0000","last_peered":"2024-03-18T20:25:35.555564+0000","last_clean":"2024-03-18T20:25:35.555564+0000","last_became_active":"2024-03-18T20:25:22.695759+0000","last_became_peered":"2024-03-18T20:25:22.695759+0000","last_unstale":"2024-03-18T20:25:35.555564+0000","last_undegraded":"2024-03-18T20:25:35.555564+0000","last_fullsized":"2024-03-18T20:25:35.555564+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:10.681789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"236'1","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:37.348587+0000","last_change":"2024-03-18T20:25:35.132963+0000","last_active":"2024-03-18T20:25:37.348587+0000","last_peered":"2024-03-18T20:25:37.348587+0000","last_clean":"2024-03-18T20:25:37.348587+0000","last_became_active":"2024-03-18T20:25:35.132654+0000","last_became_peered":"2024-03-18T20:25:35.132654+0000","last_unstale":"2024-03-18T20:25:37.348587+0000","last_undegraded":"2024-03-18T20:25:37.348587+0000","last_fullsized":"2024-03-18T20:25:37.348587+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:13.759015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188001+0000","last_change":"2024-03-18T20:25:35.579270+0000","last_active":"2024-03-18T20:25:38.188001+0000","last_peered":"2024-03-18T20:25:38.188001+0000","last_clean":"2024-03-18T20:25:38.188001+0000","last_became_active":"2024-03-18T20:25:35.578800+0000","last_became_peered":"2024-03-18T20:25:35.578800+0000","last_unstale":"2024-03-18T20:25:38.188001+0000","last_undegraded":"2024-03-18T20:25:38.188001+0000","last_fullsized":"2024-03-18T20:25:38.188001+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:58:58.372932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560229+0000","last_change":"2024-03-18T20:25:22.252889+0000","last_active":"2024-03-18T20:25:35.560229+0000","last_peered":"2024-03-18T20:25:35.560229+0000","last_clean":"2024-03-18T20:25:35.560229+0000","last_became_active":"2024-03-18T20:25:22.252096+0000","last_became_peered":"2024-03-18T20:25:22.252096+0000","last_unstale":"2024-03-18T20:25:35.560229+0000","last_undegraded":"2024-03-18T20:25:35.560229+0000","last_fullsized":"2024-03-18T20:25:35.560229+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:18.568149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561455+0000","last_change":"2024-03-18T20:25:22.689532+0000","last_active":"2024-03-18T20:25:35.561455+0000","last_peered":"2024-03-18T20:25:35.561455+0000","last_clean":"2024-03-18T20:25:35.561455+0000","last_became_active":"2024-03-18T20:25:22.689210+0000","last_became_peered":"2024-03-18T20:25:22.689210+0000","last_unstale":"2024-03-18T20:25:35.561455+0000","last_undegraded":"2024-03-18T20:25:35.561455+0000","last_fullsized":"2024-03-18T20:25:35.561455+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:21.229791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"235'2","reported_seq":13,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.582084+0000","last_change":"2024-03-18T20:25:35.145037+0000","last_active":"2024-03-18T20:25:35.582084+0000","last_peered":"2024-03-18T20:25:35.582084+0000","last_clean":"2024-03-18T20:25:35.582084+0000","last_became_active":"2024-03-18T20:25:35.144629+0000","last_became_peered":"2024-03-18T20:25:35.144629+0000","last_unstale":"2024-03-18T20:25:35.582084+0000","last_undegraded":"2024-03-18T20:25:35.582084+0000","last_fullsized":"2024-03-18T20:25:35.582084+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:17.295248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187642+0000","last_change":"2024-03-18T20:25:35.576406+0000","last_active":"2024-03-18T20:25:38.187642+0000","last_peered":"2024-03-18T20:25:38.187642+0000","last_clean":"2024-03-18T20:25:38.187642+0000","last_became_active":"2024-03-18T20:25:35.575245+0000","last_became_peered":"2024-03-18T20:25:35.575245+0000","last_unstale":"2024-03-18T20:25:38.187642+0000","last_undegraded":"2024-03-18T20:25:38.187642+0000","last_fullsized":"2024-03-18T20:25:38.187642+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:31.956522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561056+0000","last_change":"2024-03-18T20:25:22.254053+0000","last_active":"2024-03-18T20:25:35.561056+0000","last_peered":"2024-03-18T20:25:35.561056+0000","last_clean":"2024-03-18T20:25:35.561056+0000","last_became_active":"2024-03-18T20:25:22.253747+0000","last_became_peered":"2024-03-18T20:25:22.253747+0000","last_unstale":"2024-03-18T20:25:35.561056+0000","last_undegraded":"2024-03-18T20:25:35.561056+0000","last_fullsized":"2024-03-18T20:25:35.561056+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:32.777143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561855+0000","last_change":"2024-03-18T20:25:22.685300+0000","last_active":"2024-03-18T20:25:35.561855+0000","last_peered":"2024-03-18T20:25:35.561855+0000","last_clean":"2024-03-18T20:25:35.561855+0000","last_became_active":"2024-03-18T20:25:22.684869+0000","last_became_peered":"2024-03-18T20:25:22.684869+0000","last_unstale":"2024-03-18T20:25:35.561855+0000","last_undegraded":"2024-03-18T20:25:35.561855+0000","last_fullsized":"2024-03-18T20:25:35.561855+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:02.037780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.559649+0000","last_change":"2024-03-18T20:25:35.137734+0000","last_active":"2024-03-18T20:25:35.559649+0000","last_peered":"2024-03-18T20:25:35.559649+0000","last_clean":"2024-03-18T20:25:35.559649+0000","last_became_active":"2024-03-18T20:25:35.137503+0000","last_became_peered":"2024-03-18T20:25:35.137503+0000","last_unstale":"2024-03-18T20:25:35.559649+0000","last_undegraded":"2024-03-18T20:25:35.559649+0000","last_fullsized":"2024-03-18T20:25:35.559649+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:44:27.205313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.567713+0000","last_change":"2024-03-18T20:25:35.570919+0000","last_active":"2024-03-18T20:25:36.567713+0000","last_peered":"2024-03-18T20:25:36.567713+0000","last_clean":"2024-03-18T20:25:36.567713+0000","last_became_active":"2024-03-18T20:25:35.570665+0000","last_became_peered":"2024-03-18T20:25:35.570665+0000","last_unstale":"2024-03-18T20:25:36.567713+0000","last_undegraded":"2024-03-18T20:25:36.567713+0000","last_fullsized":"2024-03-18T20:25:36.567713+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:46.672966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.564287+0000","last_change":"2024-03-18T20:25:22.245633+0000","last_active":"2024-03-18T20:25:35.564287+0000","last_peered":"2024-03-18T20:25:35.564287+0000","last_clean":"2024-03-18T20:25:35.564287+0000","last_became_active":"2024-03-18T20:25:22.245372+0000","last_became_peered":"2024-03-18T20:25:22.245372+0000","last_unstale":"2024-03-18T20:25:35.564287+0000","last_undegraded":"2024-03-18T20:25:35.564287+0000","last_fullsized":"2024-03-18T20:25:35.564287+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:28:25.048682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.559740+0000","last_change":"2024-03-18T20:25:22.683574+0000","last_active":"2024-03-18T20:25:35.559740+0000","last_peered":"2024-03-18T20:25:35.559740+0000","last_clean":"2024-03-18T20:25:35.559740+0000","last_became_active":"2024-03-18T20:25:22.683015+0000","last_became_peered":"2024-03-18T20:25:22.683015+0000","last_unstale":"2024-03-18T20:25:35.559740+0000","last_undegraded":"2024-03-18T20:25:35.559740+0000","last_fullsized":"2024-03-18T20:25:35.559740+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:11.606826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.555834+0000","last_change":"2024-03-18T20:25:35.133030+0000","last_active":"2024-03-18T20:25:35.555834+0000","last_peered":"2024-03-18T20:25:35.555834+0000","last_clean":"2024-03-18T20:25:35.555834+0000","last_became_active":"2024-03-18T20:25:35.132715+0000","last_became_peered":"2024-03-18T20:25:35.132715+0000","last_unstale":"2024-03-18T20:25:35.555834+0000","last_undegraded":"2024-03-18T20:25:35.555834+0000","last_fullsized":"2024-03-18T20:25:35.555834+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:41.670198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.923677+0000","last_change":"2024-03-18T20:25:35.574963+0000","last_active":"2024-03-18T20:25:36.923677+0000","last_peered":"2024-03-18T20:25:36.923677+0000","last_clean":"2024-03-18T20:25:36.923677+0000","last_became_active":"2024-03-18T20:25:35.574685+0000","last_became_peered":"2024-03-18T20:25:35.574685+0000","last_unstale":"2024-03-18T20:25:36.923677+0000","last_undegraded":"2024-03-18T20:25:36.923677+0000","last_fullsized":"2024-03-18T20:25:36.923677+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:15.394818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.571863+0000","last_change":"2024-03-18T20:25:22.255435+0000","last_active":"2024-03-18T20:25:35.571863+0000","last_peered":"2024-03-18T20:25:35.571863+0000","last_clean":"2024-03-18T20:25:35.571863+0000","last_became_active":"2024-03-18T20:25:22.255149+0000","last_became_peered":"2024-03-18T20:25:22.255149+0000","last_unstale":"2024-03-18T20:25:35.571863+0000","last_undegraded":"2024-03-18T20:25:35.571863+0000","last_fullsized":"2024-03-18T20:25:35.571863+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:58.960198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.564166+0000","last_change":"2024-03-18T20:25:22.682910+0000","last_active":"2024-03-18T20:25:35.564166+0000","last_peered":"2024-03-18T20:25:35.564166+0000","last_clean":"2024-03-18T20:25:35.564166+0000","last_became_active":"2024-03-18T20:25:22.682667+0000","last_became_peered":"2024-03-18T20:25:22.682667+0000","last_unstale":"2024-03-18T20:25:35.564166+0000","last_undegraded":"2024-03-18T20:25:35.564166+0000","last_fullsized":"2024-03-18T20:25:35.564166+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:02.037780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.572087+0000","last_change":"2024-03-18T20:25:35.137107+0000","last_active":"2024-03-18T20:25:35.572087+0000","last_peered":"2024-03-18T20:25:35.572087+0000","last_clean":"2024-03-18T20:25:35.572087+0000","last_became_active":"2024-03-18T20:25:35.132663+0000","last_became_peered":"2024-03-18T20:25:35.132663+0000","last_unstale":"2024-03-18T20:25:35.572087+0000","last_undegraded":"2024-03-18T20:25:35.572087+0000","last_fullsized":"2024-03-18T20:25:35.572087+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:56.513767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188079+0000","last_change":"2024-03-18T20:25:35.579484+0000","last_active":"2024-03-18T20:25:38.188079+0000","last_peered":"2024-03-18T20:25:38.188079+0000","last_clean":"2024-03-18T20:25:38.188079+0000","last_became_active":"2024-03-18T20:25:35.579220+0000","last_became_peered":"2024-03-18T20:25:35.579220+0000","last_unstale":"2024-03-18T20:25:38.188079+0000","last_undegraded":"2024-03-18T20:25:38.188079+0000","last_fullsized":"2024-03-18T20:25:38.188079+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:08:55.975267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.574988+0000","last_change":"2024-03-18T20:25:22.250457+0000","last_active":"2024-03-18T20:25:35.574988+0000","last_peered":"2024-03-18T20:25:35.574988+0000","last_clean":"2024-03-18T20:25:35.574988+0000","last_became_active":"2024-03-18T20:25:22.250157+0000","last_became_peered":"2024-03-18T20:25:22.250157+0000","last_unstale":"2024-03-18T20:25:35.574988+0000","last_undegraded":"2024-03-18T20:25:35.574988+0000","last_fullsized":"2024-03-18T20:25:35.574988+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:18.568149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560885+0000","last_change":"2024-03-18T20:25:22.691245+0000","last_active":"2024-03-18T20:25:35.560885+0000","last_peered":"2024-03-18T20:25:35.560885+0000","last_clean":"2024-03-18T20:25:35.560885+0000","last_became_active":"2024-03-18T20:25:22.690988+0000","last_became_peered":"2024-03-18T20:25:22.690988+0000","last_unstale":"2024-03-18T20:25:35.560885+0000","last_undegraded":"2024-03-18T20:25:35.560885+0000","last_fullsized":"2024-03-18T20:25:35.560885+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:32:54.125659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561627+0000","last_change":"2024-03-18T20:25:35.144694+0000","last_active":"2024-03-18T20:25:35.561627+0000","last_peered":"2024-03-18T20:25:35.561627+0000","last_clean":"2024-03-18T20:25:35.561627+0000","last_became_active":"2024-03-18T20:25:35.144468+0000","last_became_peered":"2024-03-18T20:25:35.144468+0000","last_unstale":"2024-03-18T20:25:35.561627+0000","last_undegraded":"2024-03-18T20:25:35.561627+0000","last_fullsized":"2024-03-18T20:25:35.561627+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:25:32.071310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187577+0000","last_change":"2024-03-18T20:25:35.581378+0000","last_active":"2024-03-18T20:25:38.187577+0000","last_peered":"2024-03-18T20:25:38.187577+0000","last_clean":"2024-03-18T20:25:38.187577+0000","last_became_active":"2024-03-18T20:25:35.575394+0000","last_became_peered":"2024-03-18T20:25:35.575394+0000","last_unstale":"2024-03-18T20:25:38.187577+0000","last_undegraded":"2024-03-18T20:25:38.187577+0000","last_fullsized":"2024-03-18T20:25:38.187577+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:43:21.746640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.6","version":"232'1","reported_seq":16,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560294+0000","last_change":"2024-03-18T20:25:22.247183+0000","last_active":"2024-03-18T20:25:35.560294+0000","last_peered":"2024-03-18T20:25:35.560294+0000","last_clean":"2024-03-18T20:25:35.560294+0000","last_became_active":"2024-03-18T20:25:22.246890+0000","last_became_peered":"2024-03-18T20:25:22.246890+0000","last_unstale":"2024-03-18T20:25:35.560294+0000","last_undegraded":"2024-03-18T20:25:35.560294+0000","last_fullsized":"2024-03-18T20:25:35.560294+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:40:21.886849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.557004+0000","last_change":"2024-03-18T20:25:22.689011+0000","last_active":"2024-03-18T20:25:35.557004+0000","last_peered":"2024-03-18T20:25:35.557004+0000","last_clean":"2024-03-18T20:25:35.557004+0000","last_became_active":"2024-03-18T20:25:22.688750+0000","last_became_peered":"2024-03-18T20:25:22.688750+0000","last_unstale":"2024-03-18T20:25:35.557004+0000","last_undegraded":"2024-03-18T20:25:35.557004+0000","last_fullsized":"2024-03-18T20:25:35.557004+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:57.345369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.556476+0000","last_change":"2024-03-18T20:25:35.145538+0000","last_active":"2024-03-18T20:25:35.556476+0000","last_peered":"2024-03-18T20:25:35.556476+0000","last_clean":"2024-03-18T20:25:35.556476+0000","last_became_active":"2024-03-18T20:25:35.145257+0000","last_became_peered":"2024-03-18T20:25:35.145257+0000","last_unstale":"2024-03-18T20:25:35.556476+0000","last_undegraded":"2024-03-18T20:25:35.556476+0000","last_fullsized":"2024-03-18T20:25:35.556476+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:25.724222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.563395+0000","last_change":"2024-03-18T20:25:35.582929+0000","last_active":"2024-03-18T20:25:36.563395+0000","last_peered":"2024-03-18T20:25:36.563395+0000","last_clean":"2024-03-18T20:25:36.563395+0000","last_became_active":"2024-03-18T20:25:35.582610+0000","last_became_peered":"2024-03-18T20:25:35.582610+0000","last_unstale":"2024-03-18T20:25:36.563395+0000","last_undegraded":"2024-03-18T20:25:36.563395+0000","last_fullsized":"2024-03-18T20:25:36.563395+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:23.570308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"231'2","reported_seq":17,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565668+0000","last_change":"2024-03-18T20:25:22.254151+0000","last_active":"2024-03-18T20:25:35.565668+0000","last_peered":"2024-03-18T20:25:35.565668+0000","last_clean":"2024-03-18T20:25:35.565668+0000","last_became_active":"2024-03-18T20:25:22.253873+0000","last_became_peered":"2024-03-18T20:25:22.253873+0000","last_unstale":"2024-03-18T20:25:35.565668+0000","last_undegraded":"2024-03-18T20:25:35.565668+0000","last_fullsized":"2024-03-18T20:25:35.565668+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:26:58.327699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.557400+0000","last_change":"2024-03-18T20:25:22.693170+0000","last_active":"2024-03-18T20:25:35.557400+0000","last_peered":"2024-03-18T20:25:35.557400+0000","last_clean":"2024-03-18T20:25:35.557400+0000","last_became_active":"2024-03-18T20:25:22.691592+0000","last_became_peered":"2024-03-18T20:25:22.691592+0000","last_unstale":"2024-03-18T20:25:35.557400+0000","last_undegraded":"2024-03-18T20:25:35.557400+0000","last_fullsized":"2024-03-18T20:25:35.557400+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:23.897085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560491+0000","last_change":"2024-03-18T20:25:35.132572+0000","last_active":"2024-03-18T20:25:35.560491+0000","last_peered":"2024-03-18T20:25:35.560491+0000","last_clean":"2024-03-18T20:25:35.560491+0000","last_became_active":"2024-03-18T20:25:35.132292+0000","last_became_peered":"2024-03-18T20:25:35.132292+0000","last_unstale":"2024-03-18T20:25:35.560491+0000","last_undegraded":"2024-03-18T20:25:35.560491+0000","last_fullsized":"2024-03-18T20:25:35.560491+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:06:09.253104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.566955+0000","last_change":"2024-03-18T20:25:35.578940+0000","last_active":"2024-03-18T20:25:36.566955+0000","last_peered":"2024-03-18T20:25:36.566955+0000","last_clean":"2024-03-18T20:25:36.566955+0000","last_became_active":"2024-03-18T20:25:35.578646+0000","last_became_peered":"2024-03-18T20:25:35.578646+0000","last_unstale":"2024-03-18T20:25:36.566955+0000","last_undegraded":"2024-03-18T20:25:36.566955+0000","last_fullsized":"2024-03-18T20:25:36.566955+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:21.475337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.563462+0000","last_change":"2024-03-18T20:25:22.251509+0000","last_active":"2024-03-18T20:25:35.563462+0000","last_peered":"2024-03-18T20:25:35.563462+0000","last_clean":"2024-03-18T20:25:35.563462+0000","last_became_active":"2024-03-18T20:25:22.251240+0000","last_became_peered":"2024-03-18T20:25:22.251240+0000","last_unstale":"2024-03-18T20:25:35.563462+0000","last_undegraded":"2024-03-18T20:25:35.563462+0000","last_fullsized":"2024-03-18T20:25:35.563462+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:13:14.870270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560861+0000","last_change":"2024-03-18T20:25:22.695653+0000","last_active":"2024-03-18T20:25:35.560861+0000","last_peered":"2024-03-18T20:25:35.560861+0000","last_clean":"2024-03-18T20:25:35.560861+0000","last_became_active":"2024-03-18T20:25:22.694092+0000","last_became_peered":"2024-03-18T20:25:22.694092+0000","last_unstale":"2024-03-18T20:25:35.560861+0000","last_undegraded":"2024-03-18T20:25:35.560861+0000","last_fullsized":"2024-03-18T20:25:35.560861+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:20.319345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"235'2","reported_seq":13,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.574711+0000","last_change":"2024-03-18T20:25:35.143861+0000","last_active":"2024-03-18T20:25:35.574711+0000","last_peered":"2024-03-18T20:25:35.574711+0000","last_clean":"2024-03-18T20:25:35.574711+0000","last_became_active":"2024-03-18T20:25:35.143574+0000","last_became_peered":"2024-03-18T20:25:35.143574+0000","last_unstale":"2024-03-18T20:25:35.574711+0000","last_undegraded":"2024-03-18T20:25:35.574711+0000","last_fullsized":"2024-03-18T20:25:35.574711+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:32:42.191508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187983+0000","last_change":"2024-03-18T20:25:35.575576+0000","last_active":"2024-03-18T20:25:38.187983+0000","last_peered":"2024-03-18T20:25:38.187983+0000","last_clean":"2024-03-18T20:25:38.187983+0000","last_became_active":"2024-03-18T20:25:35.575291+0000","last_became_peered":"2024-03-18T20:25:35.575291+0000","last_unstale":"2024-03-18T20:25:38.187983+0000","last_undegraded":"2024-03-18T20:25:38.187983+0000","last_fullsized":"2024-03-18T20:25:38.187983+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:00:31.185972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.576539+0000","last_change":"2024-03-18T20:25:22.251818+0000","last_active":"2024-03-18T20:25:35.576539+0000","last_peered":"2024-03-18T20:25:35.576539+0000","last_clean":"2024-03-18T20:25:35.576539+0000","last_became_active":"2024-03-18T20:25:22.251407+0000","last_became_peered":"2024-03-18T20:25:22.251407+0000","last_unstale":"2024-03-18T20:25:35.576539+0000","last_undegraded":"2024-03-18T20:25:35.576539+0000","last_fullsized":"2024-03-18T20:25:35.576539+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:29.196742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565352+0000","last_change":"2024-03-18T20:25:22.684929+0000","last_active":"2024-03-18T20:25:35.565352+0000","last_peered":"2024-03-18T20:25:35.565352+0000","last_clean":"2024-03-18T20:25:35.565352+0000","last_became_active":"2024-03-18T20:25:22.684658+0000","last_became_peered":"2024-03-18T20:25:22.684658+0000","last_unstale":"2024-03-18T20:25:35.565352+0000","last_undegraded":"2024-03-18T20:25:35.565352+0000","last_fullsized":"2024-03-18T20:25:35.565352+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:39.733357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"235'1","reported_seq":12,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.583701+0000","last_change":"2024-03-18T20:25:35.138420+0000","last_active":"2024-03-18T20:25:35.583701+0000","last_peered":"2024-03-18T20:25:35.583701+0000","last_clean":"2024-03-18T20:25:35.583701+0000","last_became_active":"2024-03-18T20:25:35.138082+0000","last_became_peered":"2024-03-18T20:25:35.138082+0000","last_unstale":"2024-03-18T20:25:35.583701+0000","last_undegraded":"2024-03-18T20:25:35.583701+0000","last_fullsized":"2024-03-18T20:25:35.583701+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:22:13.432979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187553+0000","last_change":"2024-03-18T20:25:35.577432+0000","last_active":"2024-03-18T20:25:38.187553+0000","last_peered":"2024-03-18T20:25:38.187553+0000","last_clean":"2024-03-18T20:25:38.187553+0000","last_became_active":"2024-03-18T20:25:35.576147+0000","last_became_peered":"2024-03-18T20:25:35.576147+0000","last_unstale":"2024-03-18T20:25:38.187553+0000","last_undegraded":"2024-03-18T20:25:38.187553+0000","last_fullsized":"2024-03-18T20:25:38.187553+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:24:50.952354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.569694+0000","last_change":"2024-03-18T20:25:22.247489+0000","last_active":"2024-03-18T20:25:35.569694+0000","last_peered":"2024-03-18T20:25:35.569694+0000","last_clean":"2024-03-18T20:25:35.569694+0000","last_became_active":"2024-03-18T20:25:22.247203+0000","last_became_peered":"2024-03-18T20:25:22.247203+0000","last_unstale":"2024-03-18T20:25:35.569694+0000","last_undegraded":"2024-03-18T20:25:35.569694+0000","last_fullsized":"2024-03-18T20:25:35.569694+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:32.481118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561585+0000","last_change":"2024-03-18T20:25:22.681615+0000","last_active":"2024-03-18T20:25:35.561585+0000","last_peered":"2024-03-18T20:25:35.561585+0000","last_clean":"2024-03-18T20:25:35.561585+0000","last_became_active":"2024-03-18T20:25:22.681261+0000","last_became_peered":"2024-03-18T20:25:22.681261+0000","last_unstale":"2024-03-18T20:25:35.561585+0000","last_undegraded":"2024-03-18T20:25:35.561585+0000","last_fullsized":"2024-03-18T20:25:35.561585+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:15:29.461741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.559342+0000","last_change":"2024-03-18T20:25:35.139343+0000","last_active":"2024-03-18T20:25:35.559342+0000","last_peered":"2024-03-18T20:25:35.559342+0000","last_clean":"2024-03-18T20:25:35.559342+0000","last_became_active":"2024-03-18T20:25:35.139075+0000","last_became_peered":"2024-03-18T20:25:35.139075+0000","last_unstale":"2024-03-18T20:25:35.559342+0000","last_undegraded":"2024-03-18T20:25:35.559342+0000","last_fullsized":"2024-03-18T20:25:35.559342+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:33.109951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.564199+0000","last_change":"2024-03-18T20:25:35.582741+0000","last_active":"2024-03-18T20:25:36.564199+0000","last_peered":"2024-03-18T20:25:36.564199+0000","last_clean":"2024-03-18T20:25:36.564199+0000","last_became_active":"2024-03-18T20:25:35.582115+0000","last_became_peered":"2024-03-18T20:25:35.582115+0000","last_unstale":"2024-03-18T20:25:36.564199+0000","last_undegraded":"2024-03-18T20:25:36.564199+0000","last_fullsized":"2024-03-18T20:25:36.564199+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:03.024404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.b","version":"231'2","reported_seq":17,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565542+0000","last_change":"2024-03-18T20:25:22.251851+0000","last_active":"2024-03-18T20:25:35.565542+0000","last_peered":"2024-03-18T20:25:35.565542+0000","last_clean":"2024-03-18T20:25:35.565542+0000","last_became_active":"2024-03-18T20:25:22.251334+0000","last_became_peered":"2024-03-18T20:25:22.251334+0000","last_unstale":"2024-03-18T20:25:35.565542+0000","last_undegraded":"2024-03-18T20:25:35.565542+0000","last_fullsized":"2024-03-18T20:25:35.565542+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:22.889781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.568499+0000","last_change":"2024-03-18T20:25:22.687129+0000","last_active":"2024-03-18T20:25:35.568499+0000","last_peered":"2024-03-18T20:25:35.568499+0000","last_clean":"2024-03-18T20:25:35.568499+0000","last_became_active":"2024-03-18T20:25:22.686871+0000","last_became_peered":"2024-03-18T20:25:22.686871+0000","last_unstale":"2024-03-18T20:25:35.568499+0000","last_undegraded":"2024-03-18T20:25:35.568499+0000","last_fullsized":"2024-03-18T20:25:35.568499+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:18:58.196895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"235'1","reported_seq":12,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.570610+0000","last_change":"2024-03-18T20:25:35.138760+0000","last_active":"2024-03-18T20:25:35.570610+0000","last_peered":"2024-03-18T20:25:35.570610+0000","last_clean":"2024-03-18T20:25:35.570610+0000","last_became_active":"2024-03-18T20:25:35.138499+0000","last_became_peered":"2024-03-18T20:25:35.138499+0000","last_unstale":"2024-03-18T20:25:35.570610+0000","last_undegraded":"2024-03-18T20:25:35.570610+0000","last_fullsized":"2024-03-18T20:25:35.570610+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:13.917940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.189057+0000","last_change":"2024-03-18T20:25:35.582286+0000","last_active":"2024-03-18T20:25:38.189057+0000","last_peered":"2024-03-18T20:25:38.189057+0000","last_clean":"2024-03-18T20:25:38.189057+0000","last_became_active":"2024-03-18T20:25:35.582065+0000","last_became_peered":"2024-03-18T20:25:35.582065+0000","last_unstale":"2024-03-18T20:25:38.189057+0000","last_undegraded":"2024-03-18T20:25:38.189057+0000","last_fullsized":"2024-03-18T20:25:38.189057+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:26:45.688482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.570098+0000","last_change":"2024-03-18T20:25:22.254305+0000","last_active":"2024-03-18T20:25:35.570098+0000","last_peered":"2024-03-18T20:25:35.570098+0000","last_clean":"2024-03-18T20:25:35.570098+0000","last_became_active":"2024-03-18T20:25:22.253898+0000","last_became_peered":"2024-03-18T20:25:22.253898+0000","last_unstale":"2024-03-18T20:25:35.570098+0000","last_undegraded":"2024-03-18T20:25:35.570098+0000","last_fullsized":"2024-03-18T20:25:35.570098+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:29:48.394283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560025+0000","last_change":"2024-03-18T20:25:22.692895+0000","last_active":"2024-03-18T20:25:35.560025+0000","last_peered":"2024-03-18T20:25:35.560025+0000","last_clean":"2024-03-18T20:25:35.560025+0000","last_became_active":"2024-03-18T20:25:22.685766+0000","last_became_peered":"2024-03-18T20:25:22.685766+0000","last_unstale":"2024-03-18T20:25:35.560025+0000","last_undegraded":"2024-03-18T20:25:35.560025+0000","last_fullsized":"2024-03-18T20:25:35.560025+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:16:08.020891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.556303+0000","last_change":"2024-03-18T20:25:35.135046+0000","last_active":"2024-03-18T20:25:35.556303+0000","last_peered":"2024-03-18T20:25:35.556303+0000","last_clean":"2024-03-18T20:25:35.556303+0000","last_became_active":"2024-03-18T20:25:35.134784+0000","last_became_peered":"2024-03-18T20:25:35.134784+0000","last_unstale":"2024-03-18T20:25:35.556303+0000","last_undegraded":"2024-03-18T20:25:35.556303+0000","last_fullsized":"2024-03-18T20:25:35.556303+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:38.721170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187693+0000","last_change":"2024-03-18T20:25:35.578197+0000","last_active":"2024-03-18T20:25:38.187693+0000","last_peered":"2024-03-18T20:25:38.187693+0000","last_clean":"2024-03-18T20:25:38.187693+0000","last_became_active":"2024-03-18T20:25:35.577916+0000","last_became_peered":"2024-03-18T20:25:35.577916+0000","last_unstale":"2024-03-18T20:25:38.187693+0000","last_undegraded":"2024-03-18T20:25:38.187693+0000","last_fullsized":"2024-03-18T20:25:38.187693+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:51:55.984289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.d","version":"231'1","reported_seq":16,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.569332+0000","last_change":"2024-03-18T20:25:22.248228+0000","last_active":"2024-03-18T20:25:35.569332+0000","last_peered":"2024-03-18T20:25:35.569332+0000","last_clean":"2024-03-18T20:25:35.569332+0000","last_became_active":"2024-03-18T20:25:22.247916+0000","last_became_peered":"2024-03-18T20:25:22.247916+0000","last_unstale":"2024-03-18T20:25:35.569332+0000","last_undegraded":"2024-03-18T20:25:35.569332+0000","last_fullsized":"2024-03-18T20:25:35.569332+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:56.236862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.564071+0000","last_change":"2024-03-18T20:25:22.684910+0000","last_active":"2024-03-18T20:25:35.564071+0000","last_peered":"2024-03-18T20:25:35.564071+0000","last_clean":"2024-03-18T20:25:35.564071+0000","last_became_active":"2024-03-18T20:25:22.684472+0000","last_became_peered":"2024-03-18T20:25:22.684472+0000","last_unstale":"2024-03-18T20:25:35.564071+0000","last_undegraded":"2024-03-18T20:25:35.564071+0000","last_fullsized":"2024-03-18T20:25:35.564071+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:24:33.293594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"235'2","reported_seq":13,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.583640+0000","last_change":"2024-03-18T20:25:35.138829+0000","last_active":"2024-03-18T20:25:35.583640+0000","last_peered":"2024-03-18T20:25:35.583640+0000","last_clean":"2024-03-18T20:25:35.583640+0000","last_became_active":"2024-03-18T20:25:35.138571+0000","last_became_peered":"2024-03-18T20:25:35.138571+0000","last_unstale":"2024-03-18T20:25:35.583640+0000","last_undegraded":"2024-03-18T20:25:35.583640+0000","last_fullsized":"2024-03-18T20:25:35.583640+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:15.369281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.563098+0000","last_change":"2024-03-18T20:25:35.570376+0000","last_active":"2024-03-18T20:25:36.563098+0000","last_peered":"2024-03-18T20:25:36.563098+0000","last_clean":"2024-03-18T20:25:36.563098+0000","last_became_active":"2024-03-18T20:25:35.570081+0000","last_became_peered":"2024-03-18T20:25:35.570081+0000","last_unstale":"2024-03-18T20:25:36.563098+0000","last_undegraded":"2024-03-18T20:25:36.563098+0000","last_fullsized":"2024-03-18T20:25:36.563098+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:39:05.449499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.e","version":"231'2","reported_seq":17,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.573028+0000","last_change":"2024-03-18T20:25:22.253997+0000","last_active":"2024-03-18T20:25:35.573028+0000","last_peered":"2024-03-18T20:25:35.573028+0000","last_clean":"2024-03-18T20:25:35.573028+0000","last_became_active":"2024-03-18T20:25:22.253649+0000","last_became_peered":"2024-03-18T20:25:22.253649+0000","last_unstale":"2024-03-18T20:25:35.573028+0000","last_undegraded":"2024-03-18T20:25:35.573028+0000","last_fullsized":"2024-03-18T20:25:35.573028+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:00.435897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.563682+0000","last_change":"2024-03-18T20:25:22.687892+0000","last_active":"2024-03-18T20:25:35.563682+0000","last_peered":"2024-03-18T20:25:35.563682+0000","last_clean":"2024-03-18T20:25:35.563682+0000","last_became_active":"2024-03-18T20:25:22.687086+0000","last_became_peered":"2024-03-18T20:25:22.687086+0000","last_unstale":"2024-03-18T20:25:35.563682+0000","last_undegraded":"2024-03-18T20:25:35.563682+0000","last_fullsized":"2024-03-18T20:25:35.563682+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:15:29.461741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.562971+0000","last_change":"2024-03-18T20:25:35.144597+0000","last_active":"2024-03-18T20:25:35.562971+0000","last_peered":"2024-03-18T20:25:35.562971+0000","last_clean":"2024-03-18T20:25:35.562971+0000","last_became_active":"2024-03-18T20:25:35.144316+0000","last_became_peered":"2024-03-18T20:25:35.144316+0000","last_unstale":"2024-03-18T20:25:35.562971+0000","last_undegraded":"2024-03-18T20:25:35.562971+0000","last_fullsized":"2024-03-18T20:25:35.562971+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:17.295248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188344+0000","last_change":"2024-03-18T20:25:35.584065+0000","last_active":"2024-03-18T20:25:38.188344+0000","last_peered":"2024-03-18T20:25:38.188344+0000","last_clean":"2024-03-18T20:25:38.188344+0000","last_became_active":"2024-03-18T20:25:35.583758+0000","last_became_peered":"2024-03-18T20:25:35.583758+0000","last_unstale":"2024-03-18T20:25:38.188344+0000","last_undegraded":"2024-03-18T20:25:38.188344+0000","last_fullsized":"2024-03-18T20:25:38.188344+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:45:27.895703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.f","version":"231'1","reported_seq":16,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.572056+0000","last_change":"2024-03-18T20:25:22.255372+0000","last_active":"2024-03-18T20:25:35.572056+0000","last_peered":"2024-03-18T20:25:35.572056+0000","last_clean":"2024-03-18T20:25:35.572056+0000","last_became_active":"2024-03-18T20:25:22.255063+0000","last_became_peered":"2024-03-18T20:25:22.255063+0000","last_unstale":"2024-03-18T20:25:35.572056+0000","last_undegraded":"2024-03-18T20:25:35.572056+0000","last_fullsized":"2024-03-18T20:25:35.572056+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:23.143345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.556731+0000","last_change":"2024-03-18T20:25:22.686185+0000","last_active":"2024-03-18T20:25:35.556731+0000","last_peered":"2024-03-18T20:25:35.556731+0000","last_clean":"2024-03-18T20:25:35.556731+0000","last_became_active":"2024-03-18T20:25:22.685939+0000","last_became_peered":"2024-03-18T20:25:22.685939+0000","last_unstale":"2024-03-18T20:25:35.556731+0000","last_undegraded":"2024-03-18T20:25:35.556731+0000","last_fullsized":"2024-03-18T20:25:35.556731+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:48:48.961872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.556939+0000","last_change":"2024-03-18T20:25:35.130591+0000","last_active":"2024-03-18T20:25:35.556939+0000","last_peered":"2024-03-18T20:25:35.556939+0000","last_clean":"2024-03-18T20:25:35.556939+0000","last_became_active":"2024-03-18T20:25:35.130354+0000","last_became_peered":"2024-03-18T20:25:35.130354+0000","last_unstale":"2024-03-18T20:25:35.556939+0000","last_undegraded":"2024-03-18T20:25:35.556939+0000","last_fullsized":"2024-03-18T20:25:35.556939+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:50.264082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187506+0000","last_change":"2024-03-18T20:25:35.576215+0000","last_active":"2024-03-18T20:25:38.187506+0000","last_peered":"2024-03-18T20:25:38.187506+0000","last_clean":"2024-03-18T20:25:38.187506+0000","last_became_active":"2024-03-18T20:25:35.575836+0000","last_became_peered":"2024-03-18T20:25:35.575836+0000","last_unstale":"2024-03-18T20:25:38.187506+0000","last_undegraded":"2024-03-18T20:25:38.187506+0000","last_fullsized":"2024-03-18T20:25:38.187506+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:53:59.063396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.10","version":"231'8","reported_seq":23,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.564701+0000","last_change":"2024-03-18T20:25:22.252976+0000","last_active":"2024-03-18T20:25:35.564701+0000","last_peered":"2024-03-18T20:25:35.564701+0000","last_clean":"2024-03-18T20:25:35.564701+0000","last_became_active":"2024-03-18T20:25:22.252528+0000","last_became_peered":"2024-03-18T20:25:22.252528+0000","last_unstale":"2024-03-18T20:25:35.564701+0000","last_undegraded":"2024-03-18T20:25:35.564701+0000","last_fullsized":"2024-03-18T20:25:35.564701+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:42:24.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.563715+0000","last_change":"2024-03-18T20:25:22.673559+0000","last_active":"2024-03-18T20:25:35.563715+0000","last_peered":"2024-03-18T20:25:35.563715+0000","last_clean":"2024-03-18T20:25:35.563715+0000","last_became_active":"2024-03-18T20:25:22.673267+0000","last_became_peered":"2024-03-18T20:25:22.673267+0000","last_unstale":"2024-03-18T20:25:35.563715+0000","last_undegraded":"2024-03-18T20:25:35.563715+0000","last_fullsized":"2024-03-18T20:25:35.563715+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:44:15.325153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.573396+0000","last_change":"2024-03-18T20:25:35.137760+0000","last_active":"2024-03-18T20:25:35.573396+0000","last_peered":"2024-03-18T20:25:35.573396+0000","last_clean":"2024-03-18T20:25:35.573396+0000","last_became_active":"2024-03-18T20:25:35.137438+0000","last_became_peered":"2024-03-18T20:25:35.137438+0000","last_unstale":"2024-03-18T20:25:35.573396+0000","last_undegraded":"2024-03-18T20:25:35.573396+0000","last_fullsized":"2024-03-18T20:25:35.573396+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:50.170851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:37.348365+0000","last_change":"2024-03-18T20:25:35.578102+0000","last_active":"2024-03-18T20:25:37.348365+0000","last_peered":"2024-03-18T20:25:37.348365+0000","last_clean":"2024-03-18T20:25:37.348365+0000","last_became_active":"2024-03-18T20:25:35.577822+0000","last_became_peered":"2024-03-18T20:25:35.577822+0000","last_unstale":"2024-03-18T20:25:37.348365+0000","last_undegraded":"2024-03-18T20:25:37.348365+0000","last_fullsized":"2024-03-18T20:25:37.348365+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:57:19.234034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560397+0000","last_change":"2024-03-18T20:25:22.252927+0000","last_active":"2024-03-18T20:25:35.560397+0000","last_peered":"2024-03-18T20:25:35.560397+0000","last_clean":"2024-03-18T20:25:35.560397+0000","last_became_active":"2024-03-18T20:25:22.252633+0000","last_became_peered":"2024-03-18T20:25:22.252633+0000","last_unstale":"2024-03-18T20:25:35.560397+0000","last_undegraded":"2024-03-18T20:25:35.560397+0000","last_fullsized":"2024-03-18T20:25:35.560397+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:32.345526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565256+0000","last_change":"2024-03-18T20:25:22.694369+0000","last_active":"2024-03-18T20:25:35.565256+0000","last_peered":"2024-03-18T20:25:35.565256+0000","last_clean":"2024-03-18T20:25:35.565256+0000","last_became_active":"2024-03-18T20:25:22.694114+0000","last_became_peered":"2024-03-18T20:25:22.694114+0000","last_unstale":"2024-03-18T20:25:35.565256+0000","last_undegraded":"2024-03-18T20:25:35.565256+0000","last_fullsized":"2024-03-18T20:25:35.565256+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:35:13.631181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.563412+0000","last_change":"2024-03-18T20:25:35.137498+0000","last_active":"2024-03-18T20:25:35.563412+0000","last_peered":"2024-03-18T20:25:35.563412+0000","last_clean":"2024-03-18T20:25:35.563412+0000","last_became_active":"2024-03-18T20:25:35.137248+0000","last_became_peered":"2024-03-18T20:25:35.137248+0000","last_unstale":"2024-03-18T20:25:35.563412+0000","last_undegraded":"2024-03-18T20:25:35.563412+0000","last_fullsized":"2024-03-18T20:25:35.563412+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:27.179302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:37.348277+0000","last_change":"2024-03-18T20:25:35.578336+0000","last_active":"2024-03-18T20:25:37.348277+0000","last_peered":"2024-03-18T20:25:37.348277+0000","last_clean":"2024-03-18T20:25:37.348277+0000","last_became_active":"2024-03-18T20:25:35.576678+0000","last_became_peered":"2024-03-18T20:25:35.576678+0000","last_unstale":"2024-03-18T20:25:37.348277+0000","last_undegraded":"2024-03-18T20:25:37.348277+0000","last_fullsized":"2024-03-18T20:25:37.348277+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:51:58.902819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.556510+0000","last_change":"2024-03-18T20:25:22.249078+0000","last_active":"2024-03-18T20:25:35.556510+0000","last_peered":"2024-03-18T20:25:35.556510+0000","last_clean":"2024-03-18T20:25:35.556510+0000","last_became_active":"2024-03-18T20:25:22.248588+0000","last_became_peered":"2024-03-18T20:25:22.248588+0000","last_unstale":"2024-03-18T20:25:35.556510+0000","last_undegraded":"2024-03-18T20:25:35.556510+0000","last_fullsized":"2024-03-18T20:25:35.556510+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:34:42.893253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.556073+0000","last_change":"2024-03-18T20:25:22.685090+0000","last_active":"2024-03-18T20:25:35.556073+0000","last_peered":"2024-03-18T20:25:35.556073+0000","last_clean":"2024-03-18T20:25:35.556073+0000","last_became_active":"2024-03-18T20:25:22.684824+0000","last_became_peered":"2024-03-18T20:25:22.684824+0000","last_unstale":"2024-03-18T20:25:35.556073+0000","last_undegraded":"2024-03-18T20:25:35.556073+0000","last_fullsized":"2024-03-18T20:25:35.556073+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:21.750701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"235'6","reported_seq":17,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.587494+0000","last_change":"2024-03-18T20:25:35.143291+0000","last_active":"2024-03-18T20:25:35.587494+0000","last_peered":"2024-03-18T20:25:35.587494+0000","last_clean":"2024-03-18T20:25:35.587494+0000","last_became_active":"2024-03-18T20:25:35.143003+0000","last_became_peered":"2024-03-18T20:25:35.143003+0000","last_unstale":"2024-03-18T20:25:35.587494+0000","last_undegraded":"2024-03-18T20:25:35.587494+0000","last_fullsized":"2024-03-18T20:25:35.587494+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:03.410994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.570211+0000","last_change":"2024-03-18T20:25:35.577440+0000","last_active":"2024-03-18T20:25:36.570211+0000","last_peered":"2024-03-18T20:25:36.570211+0000","last_clean":"2024-03-18T20:25:36.570211+0000","last_became_active":"2024-03-18T20:25:35.577186+0000","last_became_peered":"2024-03-18T20:25:35.577186+0000","last_unstale":"2024-03-18T20:25:36.570211+0000","last_undegraded":"2024-03-18T20:25:36.570211+0000","last_fullsized":"2024-03-18T20:25:36.570211+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:39:05.449499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"231'2","reported_seq":17,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.568807+0000","last_change":"2024-03-18T20:25:22.254668+0000","last_active":"2024-03-18T20:25:35.568807+0000","last_peered":"2024-03-18T20:25:35.568807+0000","last_clean":"2024-03-18T20:25:35.568807+0000","last_became_active":"2024-03-18T20:25:22.254437+0000","last_became_peered":"2024-03-18T20:25:22.254437+0000","last_unstale":"2024-03-18T20:25:35.568807+0000","last_undegraded":"2024-03-18T20:25:35.568807+0000","last_fullsized":"2024-03-18T20:25:35.568807+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:32:02.691847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.568423+0000","last_change":"2024-03-18T20:25:22.691203+0000","last_active":"2024-03-18T20:25:35.568423+0000","last_peered":"2024-03-18T20:25:35.568423+0000","last_clean":"2024-03-18T20:25:35.568423+0000","last_became_active":"2024-03-18T20:25:22.690964+0000","last_became_peered":"2024-03-18T20:25:22.690964+0000","last_unstale":"2024-03-18T20:25:35.568423+0000","last_undegraded":"2024-03-18T20:25:35.568423+0000","last_fullsized":"2024-03-18T20:25:35.568423+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:19.068003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"235'3","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.579067+0000","last_change":"2024-03-18T20:25:35.134538+0000","last_active":"2024-03-18T20:25:35.579067+0000","last_peered":"2024-03-18T20:25:35.579067+0000","last_clean":"2024-03-18T20:25:35.579067+0000","last_became_active":"2024-03-18T20:25:35.133526+0000","last_became_peered":"2024-03-18T20:25:35.133526+0000","last_unstale":"2024-03-18T20:25:35.579067+0000","last_undegraded":"2024-03-18T20:25:35.579067+0000","last_fullsized":"2024-03-18T20:25:35.579067+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:02.016045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.189078+0000","last_change":"2024-03-18T20:25:35.573614+0000","last_active":"2024-03-18T20:25:38.189078+0000","last_peered":"2024-03-18T20:25:38.189078+0000","last_clean":"2024-03-18T20:25:38.189078+0000","last_became_active":"2024-03-18T20:25:35.573325+0000","last_became_peered":"2024-03-18T20:25:35.573325+0000","last_unstale":"2024-03-18T20:25:38.189078+0000","last_undegraded":"2024-03-18T20:25:38.189078+0000","last_fullsized":"2024-03-18T20:25:38.189078+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:32:07.096495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"232'8","reported_seq":23,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561908+0000","last_change":"2024-03-18T20:25:22.249429+0000","last_active":"2024-03-18T20:25:35.561908+0000","last_peered":"2024-03-18T20:25:35.561908+0000","last_clean":"2024-03-18T20:25:35.561908+0000","last_became_active":"2024-03-18T20:25:22.248842+0000","last_became_peered":"2024-03-18T20:25:22.248842+0000","last_unstale":"2024-03-18T20:25:35.561908+0000","last_undegraded":"2024-03-18T20:25:35.561908+0000","last_fullsized":"2024-03-18T20:25:35.561908+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:26.827223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2822,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565637+0000","last_change":"2024-03-18T20:25:22.696408+0000","last_active":"2024-03-18T20:25:35.565637+0000","last_peered":"2024-03-18T20:25:35.565637+0000","last_clean":"2024-03-18T20:25:35.565637+0000","last_became_active":"2024-03-18T20:25:22.696095+0000","last_became_peered":"2024-03-18T20:25:22.696095+0000","last_unstale":"2024-03-18T20:25:35.565637+0000","last_undegraded":"2024-03-18T20:25:35.565637+0000","last_fullsized":"2024-03-18T20:25:35.565637+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:19:19.377686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560621+0000","last_change":"2024-03-18T20:25:35.139401+0000","last_active":"2024-03-18T20:25:35.560621+0000","last_peered":"2024-03-18T20:25:35.560621+0000","last_clean":"2024-03-18T20:25:35.560621+0000","last_became_active":"2024-03-18T20:25:35.138928+0000","last_became_peered":"2024-03-18T20:25:35.138928+0000","last_unstale":"2024-03-18T20:25:35.560621+0000","last_undegraded":"2024-03-18T20:25:35.560621+0000","last_fullsized":"2024-03-18T20:25:35.560621+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:50.971852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.563311+0000","last_change":"2024-03-18T20:25:35.581002+0000","last_active":"2024-03-18T20:25:36.563311+0000","last_peered":"2024-03-18T20:25:36.563311+0000","last_clean":"2024-03-18T20:25:36.563311+0000","last_became_active":"2024-03-18T20:25:35.571822+0000","last_became_peered":"2024-03-18T20:25:35.571822+0000","last_unstale":"2024-03-18T20:25:36.563311+0000","last_undegraded":"2024-03-18T20:25:36.563311+0000","last_fullsized":"2024-03-18T20:25:36.563311+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:06:10.261463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.15","version":"231'3","reported_seq":18,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565332+0000","last_change":"2024-03-18T20:25:22.251534+0000","last_active":"2024-03-18T20:25:35.565332+0000","last_peered":"2024-03-18T20:25:35.565332+0000","last_clean":"2024-03-18T20:25:35.565332+0000","last_became_active":"2024-03-18T20:25:22.251275+0000","last_became_peered":"2024-03-18T20:25:22.251275+0000","last_unstale":"2024-03-18T20:25:35.565332+0000","last_undegraded":"2024-03-18T20:25:35.565332+0000","last_fullsized":"2024-03-18T20:25:35.565332+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:38:51.553676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560794+0000","last_change":"2024-03-18T20:25:22.682861+0000","last_active":"2024-03-18T20:25:35.560794+0000","last_peered":"2024-03-18T20:25:35.560794+0000","last_clean":"2024-03-18T20:25:35.560794+0000","last_became_active":"2024-03-18T20:25:22.682610+0000","last_became_peered":"2024-03-18T20:25:22.682610+0000","last_unstale":"2024-03-18T20:25:35.560794+0000","last_undegraded":"2024-03-18T20:25:35.560794+0000","last_fullsized":"2024-03-18T20:25:35.560794+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:51:43.095770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"235'2","reported_seq":13,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.580704+0000","last_change":"2024-03-18T20:25:35.144704+0000","last_active":"2024-03-18T20:25:35.580704+0000","last_peered":"2024-03-18T20:25:35.580704+0000","last_clean":"2024-03-18T20:25:35.580704+0000","last_became_active":"2024-03-18T20:25:35.144416+0000","last_became_peered":"2024-03-18T20:25:35.144416+0000","last_unstale":"2024-03-18T20:25:35.580704+0000","last_undegraded":"2024-03-18T20:25:35.580704+0000","last_fullsized":"2024-03-18T20:25:35.580704+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:32:42.191508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187376+0000","last_change":"2024-03-18T20:25:35.579165+0000","last_active":"2024-03-18T20:25:38.187376+0000","last_peered":"2024-03-18T20:25:38.187376+0000","last_clean":"2024-03-18T20:25:38.187376+0000","last_became_active":"2024-03-18T20:25:35.578906+0000","last_became_peered":"2024-03-18T20:25:35.578906+0000","last_unstale":"2024-03-18T20:25:38.187376+0000","last_undegraded":"2024-03-18T20:25:38.187376+0000","last_fullsized":"2024-03-18T20:25:38.187376+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:11:39.184436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560710+0000","last_change":"2024-03-18T20:25:22.252980+0000","last_active":"2024-03-18T20:25:35.560710+0000","last_peered":"2024-03-18T20:25:35.560710+0000","last_clean":"2024-03-18T20:25:35.560710+0000","last_became_active":"2024-03-18T20:25:22.252706+0000","last_became_peered":"2024-03-18T20:25:22.252706+0000","last_unstale":"2024-03-18T20:25:35.560710+0000","last_undegraded":"2024-03-18T20:25:35.560710+0000","last_fullsized":"2024-03-18T20:25:35.560710+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:45:13.948304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.556921+0000","last_change":"2024-03-18T20:25:22.684633+0000","last_active":"2024-03-18T20:25:35.556921+0000","last_peered":"2024-03-18T20:25:35.556921+0000","last_clean":"2024-03-18T20:25:35.556921+0000","last_became_active":"2024-03-18T20:25:22.684366+0000","last_became_peered":"2024-03-18T20:25:22.684366+0000","last_unstale":"2024-03-18T20:25:35.556921+0000","last_undegraded":"2024-03-18T20:25:35.556921+0000","last_fullsized":"2024-03-18T20:25:35.556921+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:15.299142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"235'8","reported_seq":19,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.583207+0000","last_change":"2024-03-18T20:25:35.145365+0000","last_active":"2024-03-18T20:25:35.583207+0000","last_peered":"2024-03-18T20:25:35.583207+0000","last_clean":"2024-03-18T20:25:35.583207+0000","last_became_active":"2024-03-18T20:25:35.145044+0000","last_became_peered":"2024-03-18T20:25:35.145044+0000","last_unstale":"2024-03-18T20:25:35.583207+0000","last_undegraded":"2024-03-18T20:25:35.583207+0000","last_fullsized":"2024-03-18T20:25:35.583207+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:50.170851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:37.348421+0000","last_change":"2024-03-18T20:25:35.571459+0000","last_active":"2024-03-18T20:25:37.348421+0000","last_peered":"2024-03-18T20:25:37.348421+0000","last_clean":"2024-03-18T20:25:37.348421+0000","last_became_active":"2024-03-18T20:25:35.571203+0000","last_became_peered":"2024-03-18T20:25:35.571203+0000","last_unstale":"2024-03-18T20:25:37.348421+0000","last_undegraded":"2024-03-18T20:25:37.348421+0000","last_fullsized":"2024-03-18T20:25:37.348421+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:06:03.885616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.570436+0000","last_change":"2024-03-18T20:25:22.251887+0000","last_active":"2024-03-18T20:25:35.570436+0000","last_peered":"2024-03-18T20:25:35.570436+0000","last_clean":"2024-03-18T20:25:35.570436+0000","last_became_active":"2024-03-18T20:25:22.251594+0000","last_became_peered":"2024-03-18T20:25:22.251594+0000","last_unstale":"2024-03-18T20:25:35.570436+0000","last_undegraded":"2024-03-18T20:25:35.570436+0000","last_fullsized":"2024-03-18T20:25:35.570436+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:48:47.954568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.572421+0000","last_change":"2024-03-18T20:25:22.680660+0000","last_active":"2024-03-18T20:25:35.572421+0000","last_peered":"2024-03-18T20:25:35.572421+0000","last_clean":"2024-03-18T20:25:35.572421+0000","last_became_active":"2024-03-18T20:25:22.680484+0000","last_became_peered":"2024-03-18T20:25:22.680484+0000","last_unstale":"2024-03-18T20:25:35.572421+0000","last_undegraded":"2024-03-18T20:25:35.572421+0000","last_fullsized":"2024-03-18T20:25:35.572421+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:48:21.194536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.562227+0000","last_change":"2024-03-18T20:25:35.139935+0000","last_active":"2024-03-18T20:25:35.562227+0000","last_peered":"2024-03-18T20:25:35.562227+0000","last_clean":"2024-03-18T20:25:35.562227+0000","last_became_active":"2024-03-18T20:25:35.139680+0000","last_became_peered":"2024-03-18T20:25:35.139680+0000","last_unstale":"2024-03-18T20:25:35.562227+0000","last_undegraded":"2024-03-18T20:25:35.562227+0000","last_fullsized":"2024-03-18T20:25:35.562227+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:20.466978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188202+0000","last_change":"2024-03-18T20:25:35.579679+0000","last_active":"2024-03-18T20:25:38.188202+0000","last_peered":"2024-03-18T20:25:38.188202+0000","last_clean":"2024-03-18T20:25:38.188202+0000","last_became_active":"2024-03-18T20:25:35.579425+0000","last_became_peered":"2024-03-18T20:25:35.579425+0000","last_unstale":"2024-03-18T20:25:38.188202+0000","last_undegraded":"2024-03-18T20:25:38.188202+0000","last_fullsized":"2024-03-18T20:25:38.188202+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:22:05.703105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.555546+0000","last_change":"2024-03-18T20:25:22.244727+0000","last_active":"2024-03-18T20:25:35.555546+0000","last_peered":"2024-03-18T20:25:35.555546+0000","last_clean":"2024-03-18T20:25:35.555546+0000","last_became_active":"2024-03-18T20:25:22.244417+0000","last_became_peered":"2024-03-18T20:25:22.244417+0000","last_unstale":"2024-03-18T20:25:35.555546+0000","last_undegraded":"2024-03-18T20:25:35.555546+0000","last_fullsized":"2024-03-18T20:25:35.555546+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:10.187509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565445+0000","last_change":"2024-03-18T20:25:22.681952+0000","last_active":"2024-03-18T20:25:35.565445+0000","last_peered":"2024-03-18T20:25:35.565445+0000","last_clean":"2024-03-18T20:25:35.565445+0000","last_became_active":"2024-03-18T20:25:22.681537+0000","last_became_peered":"2024-03-18T20:25:22.681537+0000","last_unstale":"2024-03-18T20:25:35.565445+0000","last_undegraded":"2024-03-18T20:25:35.565445+0000","last_fullsized":"2024-03-18T20:25:35.565445+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:32:22.923763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"235'1","reported_seq":12,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.569910+0000","last_change":"2024-03-18T20:25:35.137498+0000","last_active":"2024-03-18T20:25:35.569910+0000","last_peered":"2024-03-18T20:25:35.569910+0000","last_clean":"2024-03-18T20:25:35.569910+0000","last_became_active":"2024-03-18T20:25:35.137199+0000","last_became_peered":"2024-03-18T20:25:35.137199+0000","last_unstale":"2024-03-18T20:25:35.569910+0000","last_undegraded":"2024-03-18T20:25:35.569910+0000","last_fullsized":"2024-03-18T20:25:35.569910+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:39.087869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188667+0000","last_change":"2024-03-18T20:25:35.569437+0000","last_active":"2024-03-18T20:25:38.188667+0000","last_peered":"2024-03-18T20:25:38.188667+0000","last_clean":"2024-03-18T20:25:38.188667+0000","last_became_active":"2024-03-18T20:25:35.568989+0000","last_became_peered":"2024-03-18T20:25:35.568989+0000","last_unstale":"2024-03-18T20:25:38.188667+0000","last_undegraded":"2024-03-18T20:25:38.188667+0000","last_fullsized":"2024-03-18T20:25:38.188667+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:32:04.776255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560802+0000","last_change":"2024-03-18T20:25:22.251522+0000","last_active":"2024-03-18T20:25:35.560802+0000","last_peered":"2024-03-18T20:25:35.560802+0000","last_clean":"2024-03-18T20:25:35.560802+0000","last_became_active":"2024-03-18T20:25:22.251143+0000","last_became_peered":"2024-03-18T20:25:22.251143+0000","last_unstale":"2024-03-18T20:25:35.560802+0000","last_undegraded":"2024-03-18T20:25:35.560802+0000","last_fullsized":"2024-03-18T20:25:35.560802+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:58.960198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.573418+0000","last_change":"2024-03-18T20:25:22.691744+0000","last_active":"2024-03-18T20:25:35.573418+0000","last_peered":"2024-03-18T20:25:35.573418+0000","last_clean":"2024-03-18T20:25:35.573418+0000","last_became_active":"2024-03-18T20:25:22.691491+0000","last_became_peered":"2024-03-18T20:25:22.691491+0000","last_unstale":"2024-03-18T20:25:35.573418+0000","last_undegraded":"2024-03-18T20:25:35.573418+0000","last_fullsized":"2024-03-18T20:25:35.573418+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:04:09.670541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"235'4","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.577517+0000","last_change":"2024-03-18T20:25:35.134695+0000","last_active":"2024-03-18T20:25:35.577517+0000","last_peered":"2024-03-18T20:25:35.577517+0000","last_clean":"2024-03-18T20:25:35.577517+0000","last_became_active":"2024-03-18T20:25:35.134333+0000","last_became_peered":"2024-03-18T20:25:35.134333+0000","last_unstale":"2024-03-18T20:25:35.577517+0000","last_undegraded":"2024-03-18T20:25:35.577517+0000","last_fullsized":"2024-03-18T20:25:35.577517+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:25.724222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188603+0000","last_change":"2024-03-18T20:25:35.581732+0000","last_active":"2024-03-18T20:25:38.188603+0000","last_peered":"2024-03-18T20:25:38.188603+0000","last_clean":"2024-03-18T20:25:38.188603+0000","last_became_active":"2024-03-18T20:25:35.581481+0000","last_became_peered":"2024-03-18T20:25:35.581481+0000","last_unstale":"2024-03-18T20:25:38.188603+0000","last_undegraded":"2024-03-18T20:25:38.188603+0000","last_fullsized":"2024-03-18T20:25:38.188603+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:09:26.131589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1b","version":"231'1","reported_seq":16,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561291+0000","last_change":"2024-03-18T20:25:22.251076+0000","last_active":"2024-03-18T20:25:35.561291+0000","last_peered":"2024-03-18T20:25:35.561291+0000","last_clean":"2024-03-18T20:25:35.561291+0000","last_became_active":"2024-03-18T20:25:22.250741+0000","last_became_peered":"2024-03-18T20:25:22.250741+0000","last_unstale":"2024-03-18T20:25:35.561291+0000","last_undegraded":"2024-03-18T20:25:35.561291+0000","last_fullsized":"2024-03-18T20:25:35.561291+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:07.448931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.573252+0000","last_change":"2024-03-18T20:25:22.680768+0000","last_active":"2024-03-18T20:25:35.573252+0000","last_peered":"2024-03-18T20:25:35.573252+0000","last_clean":"2024-03-18T20:25:35.573252+0000","last_became_active":"2024-03-18T20:25:22.680516+0000","last_became_peered":"2024-03-18T20:25:22.680516+0000","last_unstale":"2024-03-18T20:25:35.573252+0000","last_undegraded":"2024-03-18T20:25:35.573252+0000","last_fullsized":"2024-03-18T20:25:35.573252+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:19:19.377686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"235'2","reported_seq":13,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.584585+0000","last_change":"2024-03-18T20:25:35.143346+0000","last_active":"2024-03-18T20:25:35.584585+0000","last_peered":"2024-03-18T20:25:35.584585+0000","last_clean":"2024-03-18T20:25:35.584585+0000","last_became_active":"2024-03-18T20:25:35.143020+0000","last_became_peered":"2024-03-18T20:25:35.143020+0000","last_unstale":"2024-03-18T20:25:35.584585+0000","last_undegraded":"2024-03-18T20:25:35.584585+0000","last_fullsized":"2024-03-18T20:25:35.584585+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:45.664606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.564141+0000","last_change":"2024-03-18T20:25:35.561335+0000","last_active":"2024-03-18T20:25:36.564141+0000","last_peered":"2024-03-18T20:25:36.564141+0000","last_clean":"2024-03-18T20:25:36.564141+0000","last_became_active":"2024-03-18T20:25:35.561062+0000","last_became_peered":"2024-03-18T20:25:35.561062+0000","last_unstale":"2024-03-18T20:25:36.564141+0000","last_undegraded":"2024-03-18T20:25:36.564141+0000","last_fullsized":"2024-03-18T20:25:36.564141+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:48.374583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1c","version":"231'2","reported_seq":17,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.566246+0000","last_change":"2024-03-18T20:25:22.251018+0000","last_active":"2024-03-18T20:25:35.566246+0000","last_peered":"2024-03-18T20:25:35.566246+0000","last_clean":"2024-03-18T20:25:35.566246+0000","last_became_active":"2024-03-18T20:25:22.250730+0000","last_became_peered":"2024-03-18T20:25:22.250730+0000","last_unstale":"2024-03-18T20:25:35.566246+0000","last_undegraded":"2024-03-18T20:25:35.566246+0000","last_fullsized":"2024-03-18T20:25:35.566246+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:07.579514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565721+0000","last_change":"2024-03-18T20:25:22.690894+0000","last_active":"2024-03-18T20:25:35.565721+0000","last_peered":"2024-03-18T20:25:35.565721+0000","last_clean":"2024-03-18T20:25:35.565721+0000","last_became_active":"2024-03-18T20:25:22.688184+0000","last_became_peered":"2024-03-18T20:25:22.688184+0000","last_unstale":"2024-03-18T20:25:35.565721+0000","last_undegraded":"2024-03-18T20:25:35.565721+0000","last_fullsized":"2024-03-18T20:25:35.565721+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:59.967502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"235'1","reported_seq":12,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.576155+0000","last_change":"2024-03-18T20:25:35.137495+0000","last_active":"2024-03-18T20:25:35.576155+0000","last_peered":"2024-03-18T20:25:35.576155+0000","last_clean":"2024-03-18T20:25:35.576155+0000","last_became_active":"2024-03-18T20:25:35.137194+0000","last_became_peered":"2024-03-18T20:25:35.137194+0000","last_unstale":"2024-03-18T20:25:35.576155+0000","last_undegraded":"2024-03-18T20:25:35.576155+0000","last_fullsized":"2024-03-18T20:25:35.576155+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:33.630861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.563180+0000","last_change":"2024-03-18T20:25:35.577560+0000","last_active":"2024-03-18T20:25:36.563180+0000","last_peered":"2024-03-18T20:25:36.563180+0000","last_clean":"2024-03-18T20:25:36.563180+0000","last_became_active":"2024-03-18T20:25:35.573230+0000","last_became_peered":"2024-03-18T20:25:35.573230+0000","last_unstale":"2024-03-18T20:25:36.563180+0000","last_undegraded":"2024-03-18T20:25:36.563180+0000","last_fullsized":"2024-03-18T20:25:36.563180+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:27:09.266573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"231'2","reported_seq":17,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.569297+0000","last_change":"2024-03-18T20:25:22.255583+0000","last_active":"2024-03-18T20:25:35.569297+0000","last_peered":"2024-03-18T20:25:35.569297+0000","last_clean":"2024-03-18T20:25:35.569297+0000","last_became_active":"2024-03-18T20:25:22.255243+0000","last_became_peered":"2024-03-18T20:25:22.255243+0000","last_unstale":"2024-03-18T20:25:35.569297+0000","last_undegraded":"2024-03-18T20:25:35.569297+0000","last_fullsized":"2024-03-18T20:25:35.569297+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:38.084389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.558583+0000","last_change":"2024-03-18T20:25:22.691295+0000","last_active":"2024-03-18T20:25:35.558583+0000","last_peered":"2024-03-18T20:25:35.558583+0000","last_clean":"2024-03-18T20:25:35.558583+0000","last_became_active":"2024-03-18T20:25:22.690890+0000","last_became_peered":"2024-03-18T20:25:22.690890+0000","last_unstale":"2024-03-18T20:25:35.558583+0000","last_undegraded":"2024-03-18T20:25:35.558583+0000","last_fullsized":"2024-03-18T20:25:35.558583+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:23:16.320044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"235'1","reported_seq":12,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.576711+0000","last_change":"2024-03-18T20:25:35.140336+0000","last_active":"2024-03-18T20:25:35.576711+0000","last_peered":"2024-03-18T20:25:35.576711+0000","last_clean":"2024-03-18T20:25:35.576711+0000","last_became_active":"2024-03-18T20:25:35.140047+0000","last_became_peered":"2024-03-18T20:25:35.140047+0000","last_unstale":"2024-03-18T20:25:35.576711+0000","last_undegraded":"2024-03-18T20:25:35.576711+0000","last_fullsized":"2024-03-18T20:25:35.576711+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:19:02.788524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187397+0000","last_change":"2024-03-18T20:25:35.571761+0000","last_active":"2024-03-18T20:25:38.187397+0000","last_peered":"2024-03-18T20:25:38.187397+0000","last_clean":"2024-03-18T20:25:38.187397+0000","last_became_active":"2024-03-18T20:25:35.571491+0000","last_became_peered":"2024-03-18T20:25:35.571491+0000","last_unstale":"2024-03-18T20:25:38.187397+0000","last_undegraded":"2024-03-18T20:25:38.187397+0000","last_fullsized":"2024-03-18T20:25:38.187397+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:47.958993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1e","version":"231'1","reported_seq":16,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.572920+0000","last_change":"2024-03-18T20:25:22.251725+0000","last_active":"2024-03-18T20:25:35.572920+0000","last_peered":"2024-03-18T20:25:35.572920+0000","last_clean":"2024-03-18T20:25:35.572920+0000","last_became_active":"2024-03-18T20:25:22.251416+0000","last_became_peered":"2024-03-18T20:25:22.251416+0000","last_unstale":"2024-03-18T20:25:35.572920+0000","last_undegraded":"2024-03-18T20:25:35.572920+0000","last_fullsized":"2024-03-18T20:25:35.572920+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:26.827223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560339+0000","last_change":"2024-03-18T20:25:22.679158+0000","last_active":"2024-03-18T20:25:35.560339+0000","last_peered":"2024-03-18T20:25:35.560339+0000","last_clean":"2024-03-18T20:25:35.560339+0000","last_became_active":"2024-03-18T20:25:22.678889+0000","last_became_peered":"2024-03-18T20:25:22.678889+0000","last_unstale":"2024-03-18T20:25:35.560339+0000","last_undegraded":"2024-03-18T20:25:35.560339+0000","last_fullsized":"2024-03-18T20:25:35.560339+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:00.365377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.566521+0000","last_change":"2024-03-18T20:25:35.145539+0000","last_active":"2024-03-18T20:25:35.566521+0000","last_peered":"2024-03-18T20:25:35.566521+0000","last_clean":"2024-03-18T20:25:35.566521+0000","last_became_active":"2024-03-18T20:25:35.145266+0000","last_became_peered":"2024-03-18T20:25:35.145266+0000","last_unstale":"2024-03-18T20:25:35.566521+0000","last_undegraded":"2024-03-18T20:25:35.566521+0000","last_fullsized":"2024-03-18T20:25:35.566521+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:10.985415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.567301+0000","last_change":"2024-03-18T20:25:35.573235+0000","last_active":"2024-03-18T20:25:36.567301+0000","last_peered":"2024-03-18T20:25:36.567301+0000","last_clean":"2024-03-18T20:25:36.567301+0000","last_became_active":"2024-03-18T20:25:35.573002+0000","last_became_peered":"2024-03-18T20:25:35.573002+0000","last_unstale":"2024-03-18T20:25:36.567301+0000","last_undegraded":"2024-03-18T20:25:36.567301+0000","last_fullsized":"2024-03-18T20:25:36.567301+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:12.223522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1f","version":"231'4","reported_seq":19,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.576986+0000","last_change":"2024-03-18T20:25:22.251710+0000","last_active":"2024-03-18T20:25:35.576986+0000","last_peered":"2024-03-18T20:25:35.576986+0000","last_clean":"2024-03-18T20:25:35.576986+0000","last_became_active":"2024-03-18T20:25:22.251420+0000","last_became_peered":"2024-03-18T20:25:22.251420+0000","last_unstale":"2024-03-18T20:25:35.576986+0000","last_undegraded":"2024-03-18T20:25:35.576986+0000","last_fullsized":"2024-03-18T20:25:35.576986+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:00.435897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.566687+0000","last_change":"2024-03-18T20:25:22.682272+0000","last_active":"2024-03-18T20:25:35.566687+0000","last_peered":"2024-03-18T20:25:35.566687+0000","last_clean":"2024-03-18T20:25:35.566687+0000","last_became_active":"2024-03-18T20:25:22.681785+0000","last_became_peered":"2024-03-18T20:25:22.681785+0000","last_unstale":"2024-03-18T20:25:35.566687+0000","last_undegraded":"2024-03-18T20:25:35.566687+0000","last_fullsized":"2024-03-18T20:25:35.566687+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:53:46.174877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.576035+0000","last_change":"2024-03-18T20:25:35.140554+0000","last_active":"2024-03-18T20:25:35.576035+0000","last_peered":"2024-03-18T20:25:35.576035+0000","last_clean":"2024-03-18T20:25:35.576035+0000","last_became_active":"2024-03-18T20:25:35.140240+0000","last_became_peered":"2024-03-18T20:25:35.140240+0000","last_unstale":"2024-03-18T20:25:35.576035+0000","last_undegraded":"2024-03-18T20:25:35.576035+0000","last_fullsized":"2024-03-18T20:25:35.576035+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:25.724222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:37.348255+0000","last_change":"2024-03-18T20:25:35.572462+0000","last_active":"2024-03-18T20:25:37.348255+0000","last_peered":"2024-03-18T20:25:37.348255+0000","last_clean":"2024-03-18T20:25:37.348255+0000","last_became_active":"2024-03-18T20:25:35.572188+0000","last_became_peered":"2024-03-18T20:25:35.572188+0000","last_unstale":"2024-03-18T20:25:37.348255+0000","last_undegraded":"2024-03-18T20:25:37.348255+0000","last_fullsized":"2024-03-18T20:25:37.348255+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:11:39.184436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561718+0000","last_change":"2024-03-18T20:25:22.679227+0000","last_active":"2024-03-18T20:25:35.561718+0000","last_peered":"2024-03-18T20:25:35.561718+0000","last_clean":"2024-03-18T20:25:35.561718+0000","last_became_active":"2024-03-18T20:25:22.678827+0000","last_became_peered":"2024-03-18T20:25:22.678827+0000","last_unstale":"2024-03-18T20:25:35.561718+0000","last_undegraded":"2024-03-18T20:25:35.561718+0000","last_fullsized":"2024-03-18T20:25:35.561718+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:05.394457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.923226+0000","last_change":"2024-03-18T20:25:35.571705+0000","last_active":"2024-03-18T20:25:36.923226+0000","last_peered":"2024-03-18T20:25:36.923226+0000","last_clean":"2024-03-18T20:25:36.923226+0000","last_became_active":"2024-03-18T20:25:35.571345+0000","last_became_peered":"2024-03-18T20:25:35.571345+0000","last_unstale":"2024-03-18T20:25:36.923226+0000","last_undegraded":"2024-03-18T20:25:36.923226+0000","last_fullsized":"2024-03-18T20:25:36.923226+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:45:17.428350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560689+0000","last_change":"2024-03-18T20:25:22.679640+0000","last_active":"2024-03-18T20:25:35.560689+0000","last_peered":"2024-03-18T20:25:35.560689+0000","last_clean":"2024-03-18T20:25:35.560689+0000","last_became_active":"2024-03-18T20:25:22.679408+0000","last_became_peered":"2024-03-18T20:25:22.679408+0000","last_unstale":"2024-03-18T20:25:35.560689+0000","last_undegraded":"2024-03-18T20:25:35.560689+0000","last_fullsized":"2024-03-18T20:25:35.560689+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:24:33.293594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187693+0000","last_change":"2024-03-18T20:25:35.579678+0000","last_active":"2024-03-18T20:25:38.187693+0000","last_peered":"2024-03-18T20:25:38.187693+0000","last_clean":"2024-03-18T20:25:38.187693+0000","last_became_active":"2024-03-18T20:25:35.579427+0000","last_became_peered":"2024-03-18T20:25:35.579427+0000","last_unstale":"2024-03-18T20:25:38.187693+0000","last_undegraded":"2024-03-18T20:25:38.187693+0000","last_fullsized":"2024-03-18T20:25:38.187693+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:19:11.085414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.572182+0000","last_change":"2024-03-18T20:25:22.685981+0000","last_active":"2024-03-18T20:25:35.572182+0000","last_peered":"2024-03-18T20:25:35.572182+0000","last_clean":"2024-03-18T20:25:35.572182+0000","last_became_active":"2024-03-18T20:25:22.685683+0000","last_became_peered":"2024-03-18T20:25:22.685683+0000","last_unstale":"2024-03-18T20:25:35.572182+0000","last_undegraded":"2024-03-18T20:25:35.572182+0000","last_fullsized":"2024-03-18T20:25:35.572182+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:36:57.458670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.923216+0000","last_change":"2024-03-18T20:25:35.574184+0000","last_active":"2024-03-18T20:25:36.923216+0000","last_peered":"2024-03-18T20:25:36.923216+0000","last_clean":"2024-03-18T20:25:36.923216+0000","last_became_active":"2024-03-18T20:25:35.572736+0000","last_became_peered":"2024-03-18T20:25:35.572736+0000","last_unstale":"2024-03-18T20:25:36.923216+0000","last_undegraded":"2024-03-18T20:25:36.923216+0000","last_fullsized":"2024-03-18T20:25:36.923216+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:02.367044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.567118+0000","last_change":"2024-03-18T20:25:22.680503+0000","last_active":"2024-03-18T20:25:35.567118+0000","last_peered":"2024-03-18T20:25:35.567118+0000","last_clean":"2024-03-18T20:25:35.567118+0000","last_became_active":"2024-03-18T20:25:22.680258+0000","last_became_peered":"2024-03-18T20:25:22.680258+0000","last_unstale":"2024-03-18T20:25:35.567118+0000","last_undegraded":"2024-03-18T20:25:35.567118+0000","last_fullsized":"2024-03-18T20:25:35.567118+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:49.478525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.566906+0000","last_change":"2024-03-18T20:25:35.577992+0000","last_active":"2024-03-18T20:25:36.566906+0000","last_peered":"2024-03-18T20:25:36.566906+0000","last_clean":"2024-03-18T20:25:36.566906+0000","last_became_active":"2024-03-18T20:25:35.577723+0000","last_became_peered":"2024-03-18T20:25:35.577723+0000","last_unstale":"2024-03-18T20:25:36.566906+0000","last_undegraded":"2024-03-18T20:25:36.566906+0000","last_fullsized":"2024-03-18T20:25:36.566906+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:46.241349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188995+0000","last_change":"2024-03-18T20:25:35.575324+0000","last_active":"2024-03-18T20:25:38.188995+0000","last_peered":"2024-03-18T20:25:38.188995+0000","last_clean":"2024-03-18T20:25:38.188995+0000","last_became_active":"2024-03-18T20:25:35.575124+0000","last_became_peered":"2024-03-18T20:25:35.575124+0000","last_unstale":"2024-03-18T20:25:38.188995+0000","last_undegraded":"2024-03-18T20:25:38.188995+0000","last_fullsized":"2024-03-18T20:25:38.188995+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:05:32.489599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.566309+0000","last_change":"2024-03-18T20:25:22.688098+0000","last_active":"2024-03-18T20:25:35.566309+0000","last_peered":"2024-03-18T20:25:35.566309+0000","last_clean":"2024-03-18T20:25:35.566309+0000","last_became_active":"2024-03-18T20:25:22.687830+0000","last_became_peered":"2024-03-18T20:25:22.687830+0000","last_unstale":"2024-03-18T20:25:35.566309+0000","last_undegraded":"2024-03-18T20:25:35.566309+0000","last_fullsized":"2024-03-18T20:25:35.566309+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:08.456234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188474+0000","last_change":"2024-03-18T20:25:35.583496+0000","last_active":"2024-03-18T20:25:38.188474+0000","last_peered":"2024-03-18T20:25:38.188474+0000","last_clean":"2024-03-18T20:25:38.188474+0000","last_became_active":"2024-03-18T20:25:35.583147+0000","last_became_peered":"2024-03-18T20:25:35.583147+0000","last_unstale":"2024-03-18T20:25:38.188474+0000","last_undegraded":"2024-03-18T20:25:38.188474+0000","last_fullsized":"2024-03-18T20:25:38.188474+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:14:02.602355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.555994+0000","last_change":"2024-03-18T20:25:22.689262+0000","last_active":"2024-03-18T20:25:35.555994+0000","last_peered":"2024-03-18T20:25:35.555994+0000","last_clean":"2024-03-18T20:25:35.555994+0000","last_became_active":"2024-03-18T20:25:22.688913+0000","last_became_peered":"2024-03-18T20:25:22.688913+0000","last_unstale":"2024-03-18T20:25:35.555994+0000","last_undegraded":"2024-03-18T20:25:35.555994+0000","last_fullsized":"2024-03-18T20:25:35.555994+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:15:50.135885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.572556+0000","last_change":"2024-03-18T20:25:22.687342+0000","last_active":"2024-03-18T20:25:35.572556+0000","last_peered":"2024-03-18T20:25:35.572556+0000","last_clean":"2024-03-18T20:25:35.572556+0000","last_became_active":"2024-03-18T20:25:22.687092+0000","last_became_peered":"2024-03-18T20:25:22.687092+0000","last_unstale":"2024-03-18T20:25:35.572556+0000","last_undegraded":"2024-03-18T20:25:35.572556+0000","last_fullsized":"2024-03-18T20:25:35.572556+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:00:33.383893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.564160+0000","last_change":"2024-03-18T20:25:35.575591+0000","last_active":"2024-03-18T20:25:36.564160+0000","last_peered":"2024-03-18T20:25:36.564160+0000","last_clean":"2024-03-18T20:25:36.564160+0000","last_became_active":"2024-03-18T20:25:35.575099+0000","last_became_peered":"2024-03-18T20:25:35.575099+0000","last_unstale":"2024-03-18T20:25:36.564160+0000","last_undegraded":"2024-03-18T20:25:36.564160+0000","last_fullsized":"2024-03-18T20:25:36.564160+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:54:10.233888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.570166+0000","last_change":"2024-03-18T20:25:35.572663+0000","last_active":"2024-03-18T20:25:36.570166+0000","last_peered":"2024-03-18T20:25:36.570166+0000","last_clean":"2024-03-18T20:25:36.570166+0000","last_became_active":"2024-03-18T20:25:35.572150+0000","last_became_peered":"2024-03-18T20:25:35.572150+0000","last_unstale":"2024-03-18T20:25:36.570166+0000","last_undegraded":"2024-03-18T20:25:36.570166+0000","last_fullsized":"2024-03-18T20:25:36.570166+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:04.419353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565349+0000","last_change":"2024-03-18T20:25:22.683777+0000","last_active":"2024-03-18T20:25:35.565349+0000","last_peered":"2024-03-18T20:25:35.565349+0000","last_clean":"2024-03-18T20:25:35.565349+0000","last_became_active":"2024-03-18T20:25:22.683484+0000","last_became_peered":"2024-03-18T20:25:22.683484+0000","last_unstale":"2024-03-18T20:25:35.565349+0000","last_undegraded":"2024-03-18T20:25:35.565349+0000","last_fullsized":"2024-03-18T20:25:35.565349+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:03:45.332668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:37.348061+0000","last_change":"2024-03-18T20:25:35.572089+0000","last_active":"2024-03-18T20:25:37.348061+0000","last_peered":"2024-03-18T20:25:37.348061+0000","last_clean":"2024-03-18T20:25:37.348061+0000","last_became_active":"2024-03-18T20:25:35.571834+0000","last_became_peered":"2024-03-18T20:25:35.571834+0000","last_unstale":"2024-03-18T20:25:37.348061+0000","last_undegraded":"2024-03-18T20:25:37.348061+0000","last_fullsized":"2024-03-18T20:25:37.348061+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:24:54.574523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.564608+0000","last_change":"2024-03-18T20:25:22.684030+0000","last_active":"2024-03-18T20:25:35.564608+0000","last_peered":"2024-03-18T20:25:35.564608+0000","last_clean":"2024-03-18T20:25:35.564608+0000","last_became_active":"2024-03-18T20:25:22.683767+0000","last_became_peered":"2024-03-18T20:25:22.683767+0000","last_unstale":"2024-03-18T20:25:35.564608+0000","last_undegraded":"2024-03-18T20:25:35.564608+0000","last_fullsized":"2024-03-18T20:25:35.564608+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:30.204046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188263+0000","last_change":"2024-03-18T20:25:35.569309+0000","last_active":"2024-03-18T20:25:38.188263+0000","last_peered":"2024-03-18T20:25:38.188263+0000","last_clean":"2024-03-18T20:25:38.188263+0000","last_became_active":"2024-03-18T20:25:35.569035+0000","last_became_peered":"2024-03-18T20:25:35.569035+0000","last_unstale":"2024-03-18T20:25:38.188263+0000","last_undegraded":"2024-03-18T20:25:38.188263+0000","last_fullsized":"2024-03-18T20:25:38.188263+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:35:59.383051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.569947+0000","last_change":"2024-03-18T20:25:22.680418+0000","last_active":"2024-03-18T20:25:35.569947+0000","last_peered":"2024-03-18T20:25:35.569947+0000","last_clean":"2024-03-18T20:25:35.569947+0000","last_became_active":"2024-03-18T20:25:22.680149+0000","last_became_peered":"2024-03-18T20:25:22.680149+0000","last_unstale":"2024-03-18T20:25:35.569947+0000","last_undegraded":"2024-03-18T20:25:35.569947+0000","last_fullsized":"2024-03-18T20:25:35.569947+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:51:14.804124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187541+0000","last_change":"2024-03-18T20:25:35.568991+0000","last_active":"2024-03-18T20:25:38.187541+0000","last_peered":"2024-03-18T20:25:38.187541+0000","last_clean":"2024-03-18T20:25:38.187541+0000","last_became_active":"2024-03-18T20:25:35.568685+0000","last_became_peered":"2024-03-18T20:25:35.568685+0000","last_unstale":"2024-03-18T20:25:38.187541+0000","last_undegraded":"2024-03-18T20:25:38.187541+0000","last_fullsized":"2024-03-18T20:25:38.187541+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:07.014178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.558788+0000","last_change":"2024-03-18T20:25:22.691577+0000","last_active":"2024-03-18T20:25:35.558788+0000","last_peered":"2024-03-18T20:25:35.558788+0000","last_clean":"2024-03-18T20:25:35.558788+0000","last_became_active":"2024-03-18T20:25:22.691250+0000","last_became_peered":"2024-03-18T20:25:22.691250+0000","last_unstale":"2024-03-18T20:25:35.558788+0000","last_undegraded":"2024-03-18T20:25:35.558788+0000","last_fullsized":"2024-03-18T20:25:35.558788+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:55:21.946848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.566991+0000","last_change":"2024-03-18T20:25:35.574717+0000","last_active":"2024-03-18T20:25:36.566991+0000","last_peered":"2024-03-18T20:25:36.566991+0000","last_clean":"2024-03-18T20:25:36.566991+0000","last_became_active":"2024-03-18T20:25:35.574431+0000","last_became_peered":"2024-03-18T20:25:35.574431+0000","last_unstale":"2024-03-18T20:25:36.566991+0000","last_undegraded":"2024-03-18T20:25:36.566991+0000","last_fullsized":"2024-03-18T20:25:36.566991+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:54:49.519982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561716+0000","last_change":"2024-03-18T20:25:22.693670+0000","last_active":"2024-03-18T20:25:35.561716+0000","last_peered":"2024-03-18T20:25:35.561716+0000","last_clean":"2024-03-18T20:25:35.561716+0000","last_became_active":"2024-03-18T20:25:22.693178+0000","last_became_peered":"2024-03-18T20:25:22.693178+0000","last_unstale":"2024-03-18T20:25:35.561716+0000","last_undegraded":"2024-03-18T20:25:35.561716+0000","last_fullsized":"2024-03-18T20:25:35.561716+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:13.992732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188941+0000","last_change":"2024-03-18T20:25:35.575271+0000","last_active":"2024-03-18T20:25:38.188941+0000","last_peered":"2024-03-18T20:25:38.188941+0000","last_clean":"2024-03-18T20:25:38.188941+0000","last_became_active":"2024-03-18T20:25:35.573025+0000","last_became_peered":"2024-03-18T20:25:35.573025+0000","last_unstale":"2024-03-18T20:25:38.188941+0000","last_undegraded":"2024-03-18T20:25:38.188941+0000","last_fullsized":"2024-03-18T20:25:38.188941+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:05.446446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.559446+0000","last_change":"2024-03-18T20:25:22.693769+0000","last_active":"2024-03-18T20:25:35.559446+0000","last_peered":"2024-03-18T20:25:35.559446+0000","last_clean":"2024-03-18T20:25:35.559446+0000","last_became_active":"2024-03-18T20:25:22.691936+0000","last_became_peered":"2024-03-18T20:25:22.691936+0000","last_unstale":"2024-03-18T20:25:35.559446+0000","last_undegraded":"2024-03-18T20:25:35.559446+0000","last_fullsized":"2024-03-18T20:25:35.559446+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:29:49.401587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.564058+0000","last_change":"2024-03-18T20:25:35.570394+0000","last_active":"2024-03-18T20:25:36.564058+0000","last_peered":"2024-03-18T20:25:36.564058+0000","last_clean":"2024-03-18T20:25:36.564058+0000","last_became_active":"2024-03-18T20:25:35.570154+0000","last_became_peered":"2024-03-18T20:25:35.570154+0000","last_unstale":"2024-03-18T20:25:36.564058+0000","last_undegraded":"2024-03-18T20:25:36.564058+0000","last_fullsized":"2024-03-18T20:25:36.564058+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:34.639220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561470+0000","last_change":"2024-03-18T20:25:22.692857+0000","last_active":"2024-03-18T20:25:35.561470+0000","last_peered":"2024-03-18T20:25:35.561470+0000","last_clean":"2024-03-18T20:25:35.561470+0000","last_became_active":"2024-03-18T20:25:22.692615+0000","last_became_peered":"2024-03-18T20:25:22.692615+0000","last_unstale":"2024-03-18T20:25:35.561470+0000","last_undegraded":"2024-03-18T20:25:35.561470+0000","last_fullsized":"2024-03-18T20:25:35.561470+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:21:45.938034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.923657+0000","last_change":"2024-03-18T20:25:35.570607+0000","last_active":"2024-03-18T20:25:36.923657+0000","last_peered":"2024-03-18T20:25:36.923657+0000","last_clean":"2024-03-18T20:25:36.923657+0000","last_became_active":"2024-03-18T20:25:35.570102+0000","last_became_peered":"2024-03-18T20:25:35.570102+0000","last_unstale":"2024-03-18T20:25:36.923657+0000","last_undegraded":"2024-03-18T20:25:36.923657+0000","last_fullsized":"2024-03-18T20:25:36.923657+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:11.357437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565175+0000","last_change":"2024-03-18T20:25:22.683388+0000","last_active":"2024-03-18T20:25:35.565175+0000","last_peered":"2024-03-18T20:25:35.565175+0000","last_clean":"2024-03-18T20:25:35.565175+0000","last_became_active":"2024-03-18T20:25:22.683137+0000","last_became_peered":"2024-03-18T20:25:22.683137+0000","last_unstale":"2024-03-18T20:25:35.565175+0000","last_undegraded":"2024-03-18T20:25:35.565175+0000","last_fullsized":"2024-03-18T20:25:35.565175+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:02.506299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188256+0000","last_change":"2024-03-18T20:25:35.579506+0000","last_active":"2024-03-18T20:25:38.188256+0000","last_peered":"2024-03-18T20:25:38.188256+0000","last_clean":"2024-03-18T20:25:38.188256+0000","last_became_active":"2024-03-18T20:25:35.578971+0000","last_became_peered":"2024-03-18T20:25:35.578971+0000","last_unstale":"2024-03-18T20:25:38.188256+0000","last_undegraded":"2024-03-18T20:25:38.188256+0000","last_fullsized":"2024-03-18T20:25:38.188256+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:48.195554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565266+0000","last_change":"2024-03-18T20:25:22.696148+0000","last_active":"2024-03-18T20:25:35.565266+0000","last_peered":"2024-03-18T20:25:35.565266+0000","last_clean":"2024-03-18T20:25:35.565266+0000","last_became_active":"2024-03-18T20:25:22.695928+0000","last_became_peered":"2024-03-18T20:25:22.695928+0000","last_unstale":"2024-03-18T20:25:35.565266+0000","last_undegraded":"2024-03-18T20:25:35.565266+0000","last_fullsized":"2024-03-18T20:25:35.565266+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:57.565171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.563721+0000","last_change":"2024-03-18T20:25:35.570123+0000","last_active":"2024-03-18T20:25:36.563721+0000","last_peered":"2024-03-18T20:25:36.563721+0000","last_clean":"2024-03-18T20:25:36.563721+0000","last_became_active":"2024-03-18T20:25:35.569711+0000","last_became_peered":"2024-03-18T20:25:35.569711+0000","last_unstale":"2024-03-18T20:25:36.563721+0000","last_undegraded":"2024-03-18T20:25:36.563721+0000","last_fullsized":"2024-03-18T20:25:36.563721+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:28.187661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.555465+0000","last_change":"2024-03-18T20:25:22.681389+0000","last_active":"2024-03-18T20:25:35.555465+0000","last_peered":"2024-03-18T20:25:35.555465+0000","last_clean":"2024-03-18T20:25:35.555465+0000","last_became_active":"2024-03-18T20:25:22.681146+0000","last_became_peered":"2024-03-18T20:25:22.681146+0000","last_unstale":"2024-03-18T20:25:35.555465+0000","last_undegraded":"2024-03-18T20:25:35.555465+0000","last_fullsized":"2024-03-18T20:25:35.555465+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:05:57.372944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.923398+0000","last_change":"2024-03-18T20:25:35.580268+0000","last_active":"2024-03-18T20:25:36.923398+0000","last_peered":"2024-03-18T20:25:36.923398+0000","last_clean":"2024-03-18T20:25:36.923398+0000","last_became_active":"2024-03-18T20:25:35.580007+0000","last_became_peered":"2024-03-18T20:25:35.580007+0000","last_unstale":"2024-03-18T20:25:36.923398+0000","last_undegraded":"2024-03-18T20:25:36.923398+0000","last_fullsized":"2024-03-18T20:25:36.923398+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:32:35.812282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.576463+0000","last_change":"2024-03-18T20:25:22.692497+0000","last_active":"2024-03-18T20:25:35.576463+0000","last_peered":"2024-03-18T20:25:35.576463+0000","last_clean":"2024-03-18T20:25:35.576463+0000","last_became_active":"2024-03-18T20:25:22.692192+0000","last_became_peered":"2024-03-18T20:25:22.692192+0000","last_unstale":"2024-03-18T20:25:35.576463+0000","last_undegraded":"2024-03-18T20:25:35.576463+0000","last_fullsized":"2024-03-18T20:25:35.576463+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:03:47.775258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.923002+0000","last_change":"2024-03-18T20:25:35.575256+0000","last_active":"2024-03-18T20:25:36.923002+0000","last_peered":"2024-03-18T20:25:36.923002+0000","last_clean":"2024-03-18T20:25:36.923002+0000","last_became_active":"2024-03-18T20:25:35.572998+0000","last_became_peered":"2024-03-18T20:25:35.572998+0000","last_unstale":"2024-03-18T20:25:36.923002+0000","last_undegraded":"2024-03-18T20:25:36.923002+0000","last_fullsized":"2024-03-18T20:25:36.923002+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:27:09.266573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.559312+0000","last_change":"2024-03-18T20:25:22.690898+0000","last_active":"2024-03-18T20:25:35.559312+0000","last_peered":"2024-03-18T20:25:35.559312+0000","last_clean":"2024-03-18T20:25:35.559312+0000","last_became_active":"2024-03-18T20:25:22.690461+0000","last_became_peered":"2024-03-18T20:25:22.690461+0000","last_unstale":"2024-03-18T20:25:35.559312+0000","last_undegraded":"2024-03-18T20:25:35.559312+0000","last_fullsized":"2024-03-18T20:25:35.559312+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:15.208122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:37.348030+0000","last_change":"2024-03-18T20:25:35.562389+0000","last_active":"2024-03-18T20:25:37.348030+0000","last_peered":"2024-03-18T20:25:37.348030+0000","last_clean":"2024-03-18T20:25:37.348030+0000","last_became_active":"2024-03-18T20:25:35.562118+0000","last_became_peered":"2024-03-18T20:25:35.562118+0000","last_unstale":"2024-03-18T20:25:37.348030+0000","last_undegraded":"2024-03-18T20:25:37.348030+0000","last_fullsized":"2024-03-18T20:25:37.348030+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:04.584677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561196+0000","last_change":"2024-03-18T20:25:22.685172+0000","last_active":"2024-03-18T20:25:35.561196+0000","last_peered":"2024-03-18T20:25:35.561196+0000","last_clean":"2024-03-18T20:25:35.561196+0000","last_became_active":"2024-03-18T20:25:22.684855+0000","last_became_peered":"2024-03-18T20:25:22.684855+0000","last_unstale":"2024-03-18T20:25:35.561196+0000","last_undegraded":"2024-03-18T20:25:35.561196+0000","last_fullsized":"2024-03-18T20:25:35.561196+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:00.030804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188874+0000","last_change":"2024-03-18T20:25:35.582068+0000","last_active":"2024-03-18T20:25:38.188874+0000","last_peered":"2024-03-18T20:25:38.188874+0000","last_clean":"2024-03-18T20:25:38.188874+0000","last_became_active":"2024-03-18T20:25:35.581688+0000","last_became_peered":"2024-03-18T20:25:35.581688+0000","last_unstale":"2024-03-18T20:25:38.188874+0000","last_undegraded":"2024-03-18T20:25:38.188874+0000","last_fullsized":"2024-03-18T20:25:38.188874+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:02:05.383430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.566253+0000","last_change":"2024-03-18T20:25:22.693640+0000","last_active":"2024-03-18T20:25:35.566253+0000","last_peered":"2024-03-18T20:25:35.566253+0000","last_clean":"2024-03-18T20:25:35.566253+0000","last_became_active":"2024-03-18T20:25:22.693403+0000","last_became_peered":"2024-03-18T20:25:22.693403+0000","last_unstale":"2024-03-18T20:25:35.566253+0000","last_undegraded":"2024-03-18T20:25:35.566253+0000","last_fullsized":"2024-03-18T20:25:35.566253+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:58.151146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.567667+0000","last_change":"2024-03-18T20:25:35.572108+0000","last_active":"2024-03-18T20:25:36.567667+0000","last_peered":"2024-03-18T20:25:36.567667+0000","last_clean":"2024-03-18T20:25:36.567667+0000","last_became_active":"2024-03-18T20:25:35.571863+0000","last_became_peered":"2024-03-18T20:25:35.571863+0000","last_unstale":"2024-03-18T20:25:36.567667+0000","last_undegraded":"2024-03-18T20:25:36.567667+0000","last_fullsized":"2024-03-18T20:25:36.567667+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:04.419353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.555916+0000","last_change":"2024-03-18T20:25:22.693546+0000","last_active":"2024-03-18T20:25:35.555916+0000","last_peered":"2024-03-18T20:25:35.555916+0000","last_clean":"2024-03-18T20:25:35.555916+0000","last_became_active":"2024-03-18T20:25:22.693267+0000","last_became_peered":"2024-03-18T20:25:22.693267+0000","last_unstale":"2024-03-18T20:25:35.555916+0000","last_undegraded":"2024-03-18T20:25:35.555916+0000","last_fullsized":"2024-03-18T20:25:35.555916+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:35.486064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.563988+0000","last_change":"2024-03-18T20:25:35.575392+0000","last_active":"2024-03-18T20:25:36.563988+0000","last_peered":"2024-03-18T20:25:36.563988+0000","last_clean":"2024-03-18T20:25:36.563988+0000","last_became_active":"2024-03-18T20:25:35.574664+0000","last_became_peered":"2024-03-18T20:25:35.574664+0000","last_unstale":"2024-03-18T20:25:36.563988+0000","last_undegraded":"2024-03-18T20:25:36.563988+0000","last_fullsized":"2024-03-18T20:25:36.563988+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:12.445322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560194+0000","last_change":"2024-03-18T20:25:22.693152+0000","last_active":"2024-03-18T20:25:35.560194+0000","last_peered":"2024-03-18T20:25:35.560194+0000","last_clean":"2024-03-18T20:25:35.560194+0000","last_became_active":"2024-03-18T20:25:22.686521+0000","last_became_peered":"2024-03-18T20:25:22.686521+0000","last_unstale":"2024-03-18T20:25:35.560194+0000","last_undegraded":"2024-03-18T20:25:35.560194+0000","last_fullsized":"2024-03-18T20:25:35.560194+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:40:18.302530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.569992+0000","last_change":"2024-03-18T20:25:35.572687+0000","last_active":"2024-03-18T20:25:36.569992+0000","last_peered":"2024-03-18T20:25:36.569992+0000","last_clean":"2024-03-18T20:25:36.569992+0000","last_became_active":"2024-03-18T20:25:35.572459+0000","last_became_peered":"2024-03-18T20:25:35.572459+0000","last_unstale":"2024-03-18T20:25:36.569992+0000","last_undegraded":"2024-03-18T20:25:36.569992+0000","last_fullsized":"2024-03-18T20:25:36.569992+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:46.672966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565187+0000","last_change":"2024-03-18T20:25:22.681784+0000","last_active":"2024-03-18T20:25:35.565187+0000","last_peered":"2024-03-18T20:25:35.565187+0000","last_clean":"2024-03-18T20:25:35.565187+0000","last_became_active":"2024-03-18T20:25:22.681198+0000","last_became_peered":"2024-03-18T20:25:22.681198+0000","last_unstale":"2024-03-18T20:25:35.565187+0000","last_undegraded":"2024-03-18T20:25:35.565187+0000","last_fullsized":"2024-03-18T20:25:35.565187+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:35.307035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.923156+0000","last_change":"2024-03-18T20:25:35.580547+0000","last_active":"2024-03-18T20:25:36.923156+0000","last_peered":"2024-03-18T20:25:36.923156+0000","last_clean":"2024-03-18T20:25:36.923156+0000","last_became_active":"2024-03-18T20:25:35.579377+0000","last_became_peered":"2024-03-18T20:25:35.579377+0000","last_unstale":"2024-03-18T20:25:36.923156+0000","last_undegraded":"2024-03-18T20:25:36.923156+0000","last_fullsized":"2024-03-18T20:25:36.923156+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:53:10.453690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.556638+0000","last_change":"2024-03-18T20:25:22.685766+0000","last_active":"2024-03-18T20:25:35.556638+0000","last_peered":"2024-03-18T20:25:35.556638+0000","last_clean":"2024-03-18T20:25:35.556638+0000","last_became_active":"2024-03-18T20:25:22.683799+0000","last_became_peered":"2024-03-18T20:25:22.683799+0000","last_unstale":"2024-03-18T20:25:35.556638+0000","last_undegraded":"2024-03-18T20:25:35.556638+0000","last_fullsized":"2024-03-18T20:25:35.556638+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:42:25.581359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187895+0000","last_change":"2024-03-18T20:25:35.569195+0000","last_active":"2024-03-18T20:25:38.187895+0000","last_peered":"2024-03-18T20:25:38.187895+0000","last_clean":"2024-03-18T20:25:38.187895+0000","last_became_active":"2024-03-18T20:25:35.568915+0000","last_became_peered":"2024-03-18T20:25:35.568915+0000","last_unstale":"2024-03-18T20:25:38.187895+0000","last_undegraded":"2024-03-18T20:25:38.187895+0000","last_fullsized":"2024-03-18T20:25:38.187895+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:03:58.221187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.567994+0000","last_change":"2024-03-18T20:25:22.694269+0000","last_active":"2024-03-18T20:25:35.567994+0000","last_peered":"2024-03-18T20:25:35.567994+0000","last_clean":"2024-03-18T20:25:35.567994+0000","last_became_active":"2024-03-18T20:25:22.694014+0000","last_became_peered":"2024-03-18T20:25:22.694014+0000","last_unstale":"2024-03-18T20:25:35.567994+0000","last_undegraded":"2024-03-18T20:25:35.567994+0000","last_fullsized":"2024-03-18T20:25:35.567994+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:43:52.409254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.923125+0000","last_change":"2024-03-18T20:25:35.581607+0000","last_active":"2024-03-18T20:25:36.923125+0000","last_peered":"2024-03-18T20:25:36.923125+0000","last_clean":"2024-03-18T20:25:36.923125+0000","last_became_active":"2024-03-18T20:25:35.581338+0000","last_became_peered":"2024-03-18T20:25:35.581338+0000","last_unstale":"2024-03-18T20:25:36.923125+0000","last_undegraded":"2024-03-18T20:25:36.923125+0000","last_fullsized":"2024-03-18T20:25:36.923125+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:52.621876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560392+0000","last_change":"2024-03-18T20:25:22.681736+0000","last_active":"2024-03-18T20:25:35.560392+0000","last_peered":"2024-03-18T20:25:35.560392+0000","last_clean":"2024-03-18T20:25:35.560392+0000","last_became_active":"2024-03-18T20:25:22.679120+0000","last_became_peered":"2024-03-18T20:25:22.679120+0000","last_unstale":"2024-03-18T20:25:35.560392+0000","last_undegraded":"2024-03-18T20:25:35.560392+0000","last_fullsized":"2024-03-18T20:25:35.560392+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:00.365377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188013+0000","last_change":"2024-03-18T20:25:35.580208+0000","last_active":"2024-03-18T20:25:38.188013+0000","last_peered":"2024-03-18T20:25:38.188013+0000","last_clean":"2024-03-18T20:25:38.188013+0000","last_became_active":"2024-03-18T20:25:35.579486+0000","last_became_peered":"2024-03-18T20:25:35.579486+0000","last_unstale":"2024-03-18T20:25:38.188013+0000","last_undegraded":"2024-03-18T20:25:38.188013+0000","last_fullsized":"2024-03-18T20:25:38.188013+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:21:14.140052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.568868+0000","last_change":"2024-03-18T20:25:22.695525+0000","last_active":"2024-03-18T20:25:35.568868+0000","last_peered":"2024-03-18T20:25:35.568868+0000","last_clean":"2024-03-18T20:25:35.568868+0000","last_became_active":"2024-03-18T20:25:22.694079+0000","last_became_peered":"2024-03-18T20:25:22.694079+0000","last_unstale":"2024-03-18T20:25:35.568868+0000","last_undegraded":"2024-03-18T20:25:35.568868+0000","last_fullsized":"2024-03-18T20:25:35.568868+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:45:14.955608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188412+0000","last_change":"2024-03-18T20:25:35.569851+0000","last_active":"2024-03-18T20:25:38.188412+0000","last_peered":"2024-03-18T20:25:38.188412+0000","last_clean":"2024-03-18T20:25:38.188412+0000","last_became_active":"2024-03-18T20:25:35.569597+0000","last_became_peered":"2024-03-18T20:25:35.569597+0000","last_unstale":"2024-03-18T20:25:38.188412+0000","last_undegraded":"2024-03-18T20:25:38.188412+0000","last_fullsized":"2024-03-18T20:25:38.188412+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:51:27.692643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.562303+0000","last_change":"2024-03-18T20:25:22.683021+0000","last_active":"2024-03-18T20:25:35.562303+0000","last_peered":"2024-03-18T20:25:35.562303+0000","last_clean":"2024-03-18T20:25:35.562303+0000","last_became_active":"2024-03-18T20:25:22.682722+0000","last_became_peered":"2024-03-18T20:25:22.682722+0000","last_unstale":"2024-03-18T20:25:35.562303+0000","last_undegraded":"2024-03-18T20:25:35.562303+0000","last_fullsized":"2024-03-18T20:25:35.562303+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:27.207710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188384+0000","last_change":"2024-03-18T20:25:35.578783+0000","last_active":"2024-03-18T20:25:38.188384+0000","last_peered":"2024-03-18T20:25:38.188384+0000","last_clean":"2024-03-18T20:25:38.188384+0000","last_became_active":"2024-03-18T20:25:35.578460+0000","last_became_peered":"2024-03-18T20:25:35.578460+0000","last_unstale":"2024-03-18T20:25:38.188384+0000","last_undegraded":"2024-03-18T20:25:38.188384+0000","last_fullsized":"2024-03-18T20:25:38.188384+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:35:26.519700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.555724+0000","last_change":"2024-03-18T20:25:22.679426+0000","last_active":"2024-03-18T20:25:35.555724+0000","last_peered":"2024-03-18T20:25:35.555724+0000","last_clean":"2024-03-18T20:25:35.555724+0000","last_became_active":"2024-03-18T20:25:22.679159+0000","last_became_peered":"2024-03-18T20:25:22.679159+0000","last_unstale":"2024-03-18T20:25:35.555724+0000","last_undegraded":"2024-03-18T20:25:35.555724+0000","last_fullsized":"2024-03-18T20:25:35.555724+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:17:59.556803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.569822+0000","last_change":"2024-03-18T20:25:35.581809+0000","last_active":"2024-03-18T20:25:36.569822+0000","last_peered":"2024-03-18T20:25:36.569822+0000","last_clean":"2024-03-18T20:25:36.569822+0000","last_became_active":"2024-03-18T20:25:35.581546+0000","last_became_peered":"2024-03-18T20:25:35.581546+0000","last_unstale":"2024-03-18T20:25:36.569822+0000","last_undegraded":"2024-03-18T20:25:36.569822+0000","last_fullsized":"2024-03-18T20:25:36.569822+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:21.475337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.571372+0000","last_change":"2024-03-18T20:25:22.682750+0000","last_active":"2024-03-18T20:25:35.571372+0000","last_peered":"2024-03-18T20:25:35.571372+0000","last_clean":"2024-03-18T20:25:35.571372+0000","last_became_active":"2024-03-18T20:25:22.682497+0000","last_became_peered":"2024-03-18T20:25:22.682497+0000","last_unstale":"2024-03-18T20:25:35.571372+0000","last_undegraded":"2024-03-18T20:25:35.571372+0000","last_fullsized":"2024-03-18T20:25:35.571372+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:33.092503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188636+0000","last_change":"2024-03-18T20:25:35.571829+0000","last_active":"2024-03-18T20:25:38.188636+0000","last_peered":"2024-03-18T20:25:38.188636+0000","last_clean":"2024-03-18T20:25:38.188636+0000","last_became_active":"2024-03-18T20:25:35.571594+0000","last_became_peered":"2024-03-18T20:25:35.571594+0000","last_unstale":"2024-03-18T20:25:38.188636+0000","last_undegraded":"2024-03-18T20:25:38.188636+0000","last_fullsized":"2024-03-18T20:25:38.188636+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:41.225842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.571968+0000","last_change":"2024-03-18T20:25:22.696207+0000","last_active":"2024-03-18T20:25:35.571968+0000","last_peered":"2024-03-18T20:25:35.571968+0000","last_clean":"2024-03-18T20:25:35.571968+0000","last_became_active":"2024-03-18T20:25:22.695943+0000","last_became_peered":"2024-03-18T20:25:22.695943+0000","last_unstale":"2024-03-18T20:25:35.571968+0000","last_undegraded":"2024-03-18T20:25:35.571968+0000","last_fullsized":"2024-03-18T20:25:35.571968+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:54:33.571448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3620,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575984,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52664,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319176,"statfs":{"total":96636764160,"available":96582836224,"internally_reserved":0,"allocated":3579904,"data_stored":2117276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80700000000000005,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.48499999999999999,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.81899999999999995,"5min":1.093,"15min":1.093},"last":0.52500000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61799999999999999,"15min":0.61699999999999999},"min":{"1min":0.47599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.82999999999999996,"5min":0.99399999999999999,"15min":1.4299999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.60999999999999999,"15min":0.61899999999999999},"min":{"1min":0.47099999999999997,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.77500000000000002,"5min":1.0449999999999999,"15min":1.2629999999999999},"last":0.47199999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.46400000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84299999999999997,"5min":1.216,"15min":1.216},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.62,"15min":0.625},"min":{"1min":0.48399999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.81799999999999995,"5min":1.177,"15min":1.2490000000000001},"last":0.36599999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.42599999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84699999999999998,"5min":0.93100000000000005,"15min":1.4450000000000001},"last":0.377},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.5,"5min":0.44900000000000001,"15min":0.38600000000000001},"max":{"1min":0.84399999999999997,"5min":1.028,"15min":1.5029999999999999},"last":0.51500000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.47499999999999998,"15min":0.47699999999999998},"min":{"1min":0.33700000000000002,"5min":0.314,"15min":0.29899999999999999},"max":{"1min":0.68400000000000005,"5min":0.79100000000000004,"15min":0.86099999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.35199999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71599999999999997,"5min":0.76600000000000001,"15min":0.80200000000000005},"last":0.46200000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51500000000000001,"15min":0.51600000000000001},"min":{"1min":0.34899999999999998,"5min":0.34399999999999997,"15min":0.316},"max":{"1min":0.68999999999999995,"5min":0.86599999999999999,"15min":0.89400000000000002},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.40699999999999997,"5min":0.377,"15min":0.36799999999999999},"max":{"1min":0.75600000000000001,"5min":0.81599999999999995,"15min":1.042},"last":0.57599999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56399999999999995,"15min":0.57699999999999996},"min":{"1min":0.39900000000000002,"5min":0.36199999999999999,"15min":0.34999999999999998},"max":{"1min":0.71099999999999997,"5min":0.85599999999999998,"15min":1.129},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.371,"5min":0.36299999999999999,"15min":0.34000000000000002},"max":{"1min":0.82899999999999996,"5min":0.85299999999999998,"15min":1.0609999999999999},"last":0.501}]}]},{"osd":6,"up_from":13,"seq":55834575984,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42240,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329600,"statfs":{"total":96636764160,"available":96593510400,"internally_reserved":0,"allocated":3002368,"data_stored":1528438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57599999999999996,"15min":0.60099999999999998},"min":{"1min":0.41399999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72199999999999998,"5min":0.80900000000000005,"15min":1.1519999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59099999999999997,"15min":0.60799999999999998},"min":{"1min":0.33700000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.74299999999999999,"5min":0.81499999999999995,"15min":0.83299999999999996},"last":0.42299999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57899999999999996,"15min":0.60199999999999998},"min":{"1min":0.39400000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.79100000000000004,"5min":0.79800000000000004,"15min":1.244},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58499999999999996,"15min":0.60399999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33200000000000002},"max":{"1min":0.77000000000000002,"5min":0.89200000000000002,"15min":1.286},"last":0.64200000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60999999999999999,"15min":0.63},"min":{"1min":0.47899999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71199999999999997,"5min":0.92000000000000004,"15min":1.276},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.59299999999999997,"15min":0.61099999999999999},"min":{"1min":0.434,"5min":0.40100000000000002,"15min":0.378},"max":{"1min":0.77500000000000002,"5min":0.95399999999999996,"15min":1.2669999999999999},"last":0.51800000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59699999999999998,"15min":0.622},"min":{"1min":0.38,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70699999999999996,"5min":0.83199999999999996,"15min":1.02},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60599999999999998,"15min":0.61899999999999999},"min":{"1min":0.45200000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82399999999999995,"5min":1.0049999999999999,"15min":1.337},"last":0.59699999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.48099999999999998,"15min":0.48299999999999998},"min":{"1min":0.308,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.61699999999999999,"5min":0.85799999999999998,"15min":0.88300000000000001},"last":0.755},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.50900000000000001,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72099999999999997,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.623}]},{"osd":5,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.5,"15min":0.504},"min":{"1min":0.375,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.84799999999999998},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39000000000000001,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.73399999999999999,"5min":0.92600000000000005,"15min":0.94199999999999995},"last":0.72999999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40600000000000003,"5min":0.35799999999999998,"15min":0.34499999999999997},"max":{"1min":0.74299999999999999,"5min":0.95599999999999996,"15min":1.0429999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.44800000000000001,"5min":0.36599999999999999,"15min":0.34599999999999997},"max":{"1min":0.83899999999999997,"5min":1.028,"15min":1.028},"last":0.66600000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575984,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47960,"kb_used_data":2892,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323880,"statfs":{"total":96636764160,"available":96587653120,"internally_reserved":0,"allocated":2961408,"data_stored":1523344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.40400000000000003,"15min":0.39600000000000002},"min":{"1min":0.25600000000000001,"5min":0.246,"15min":0.23999999999999999},"max":{"1min":0.57999999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.41899999999999998,"15min":0.42999999999999999},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.58999999999999997,"5min":0.69699999999999995,"15min":0.72299999999999998},"last":0.32200000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.44,"15min":0.439},"min":{"1min":0.309,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61199999999999999,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48099999999999998,"15min":0.47799999999999998},"min":{"1min":0.35099999999999998,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.70099999999999996,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.36399999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.495,"15min":0.495},"min":{"1min":0.36199999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.64300000000000002,"5min":0.75900000000000001,"15min":0.76100000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.44800000000000001,"15min":0.442},"min":{"1min":0.32800000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68200000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.43099999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.68300000000000005,"15min":0.67300000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86799999999999999,"5min":1.0680000000000001,"15min":1.2909999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.68100000000000005,"15min":0.68000000000000005},"min":{"1min":0.50900000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.84499999999999997,"5min":0.92500000000000004,"15min":1.746},"last":0.52600000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.68100000000000005,"15min":0.67000000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.6910000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.71999999999999997,"15min":0.71399999999999997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.46899999999999997},"max":{"1min":0.89400000000000002,"5min":1.109,"15min":1.2549999999999999},"last":0.51100000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.69699999999999995,"15min":0.68500000000000005},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.437},"max":{"1min":0.97399999999999998,"5min":0.98899999999999999,"15min":1.637},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71899999999999997,"15min":0.71199999999999997},"min":{"1min":0.49099999999999999,"5min":0.46999999999999997,"15min":0.46100000000000002},"max":{"1min":0.89500000000000002,"5min":1.014,"15min":1.722},"last":0.44700000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.70699999999999996,"15min":0.71299999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79800000000000004,"5min":1.02,"15min":1.2729999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.498,"5min":0.495,"15min":0.42799999999999999},"max":{"1min":0.90300000000000002,"5min":1.0329999999999999,"15min":1.3380000000000001},"last":0.64700000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575985,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30496,"kb_used_data":3476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341344,"statfs":{"total":96636764160,"available":96605536256,"internally_reserved":0,"allocated":3559424,"data_stored":2113802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.42399999999999999,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.42099999999999999,"15min":0.42799999999999999},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.46600000000000003,"5min":0.64700000000000002,"15min":1.1850000000000001},"last":0.376}]},{"osd":2,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.42599999999999999,"15min":0.435},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.47099999999999997,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.438,"15min":0.45300000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29799999999999999},"max":{"1min":0.59199999999999997,"5min":0.64300000000000002,"15min":0.70999999999999996},"last":0.32400000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.26100000000000001},"max":{"1min":0.45800000000000002,"5min":0.60099999999999998,"15min":0.628},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48599999999999999,"15min":0.496},"min":{"1min":0.35499999999999998,"5min":0.35199999999999998,"15min":0.32500000000000001},"max":{"1min":0.52300000000000002,"5min":0.73999999999999999,"15min":0.91500000000000004},"last":0.34899999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64300000000000002,"15min":0.65500000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81299999999999994,"5min":0.91500000000000004,"15min":1.0029999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.66200000000000003,"15min":0.66300000000000003},"min":{"1min":0.47499999999999998,"5min":0.47099999999999997,"15min":0.39700000000000002},"max":{"1min":0.78100000000000003,"5min":0.95299999999999996,"15min":1.0469999999999999},"last":1.2989999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.67800000000000005,"15min":0.67100000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89700000000000002,"5min":0.97099999999999997,"15min":1.0129999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91100000000000003,"5min":0.92900000000000005,"15min":1.518},"last":0.53200000000000003}]},{"osd":6,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.47399999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.78600000000000003,"5min":0.95899999999999996,"15min":1.1279999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71199999999999997,"15min":0.70899999999999996},"min":{"1min":0.498,"5min":0.48999999999999999,"15min":0.47899999999999998},"max":{"1min":0.94899999999999995,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.54200000000000004}]},{"osd":7,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85099999999999998,"5min":0.96399999999999997,"15min":1.464},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.69899999999999995,"15min":0.70399999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.47599999999999998},"max":{"1min":0.75800000000000001,"5min":1.042,"15min":1.2350000000000001},"last":1.2769999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575985,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47980,"kb_used_data":2904,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323860,"statfs":{"total":96636764160,"available":96587632640,"internally_reserved":0,"allocated":2973696,"data_stored":1524050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36599999999999999,"15min":0.378},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.496,"5min":0.54800000000000004,"15min":0.59399999999999997},"last":0.315},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.47399999999999998,"15min":0.48699999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.63600000000000001,"5min":0.71499999999999997,"15min":0.753},"last":0.39000000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46899999999999997,"15min":0.48199999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64300000000000002,"5min":0.72599999999999998,"15min":0.77100000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40899999999999997,"15min":0.41599999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.53800000000000003,"5min":0.59299999999999997,"15min":0.71099999999999997},"last":0.33500000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.433,"15min":0.436},"min":{"1min":0.28899999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.497,"15min":0.51700000000000002},"min":{"1min":0.33800000000000002,"5min":0.32200000000000001,"15min":0.313},"max":{"1min":0.68100000000000005,"5min":0.752,"15min":0.752},"last":0.40999999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.66300000000000003,"15min":0.65800000000000003},"min":{"1min":0.503,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":0.82499999999999996,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67500000000000004,"15min":0.66800000000000004},"min":{"1min":0.497,"5min":0.443,"15min":0.443},"max":{"1min":0.91600000000000004,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.64600000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.67900000000000005,"15min":0.68500000000000005},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.93999999999999995,"5min":1.028,"15min":1.028},"last":0.434},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67300000000000004,"15min":0.68000000000000005},"min":{"1min":0.47299999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.88300000000000001,"5min":0.95599999999999996,"15min":1.0629999999999999},"last":0.39900000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68000000000000005,"15min":0.69099999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86099999999999999,"5min":1.014,"15min":1.2050000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.72499999999999998,"15min":0.73199999999999998},"min":{"1min":0.53800000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.97199999999999998,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.80400000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68899999999999995,"15min":0.70199999999999996},"min":{"1min":0.51500000000000001,"5min":0.442,"15min":0.42199999999999999},"max":{"1min":0.92900000000000005,"5min":0.99399999999999999,"15min":1.0469999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69899999999999995,"15min":0.70699999999999996},"min":{"1min":0.45500000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.93999999999999995,"5min":1.371,"15min":1.371},"last":0.92600000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575985,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52072,"kb_used_data":2908,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319768,"statfs":{"total":96636764160,"available":96583442432,"internally_reserved":0,"allocated":2977792,"data_stored":1523492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.38800000000000001,"15min":0.39900000000000002},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.53700000000000003,"5min":0.66500000000000004,"15min":0.98099999999999998},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.40999999999999998,"15min":0.41499999999999998},"min":{"1min":0.27700000000000002,"5min":0.25800000000000001,"15min":0.24199999999999999},"max":{"1min":0.434,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.30499999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39000000000000001,"15min":0.40999999999999998},"min":{"1min":0.30099999999999999,"5min":0.26100000000000001,"15min":0.254},"max":{"1min":0.45900000000000002,"5min":0.63200000000000001,"15min":0.64000000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.436,"5min":0.46100000000000002,"15min":0.46600000000000003},"min":{"1min":0.36699999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57499999999999996,"5min":0.65500000000000003,"15min":0.73099999999999998},"last":0.34999999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.45500000000000002,"15min":0.46600000000000003},"min":{"1min":0.34999999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53100000000000003,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.48199999999999998,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.51700000000000002,"5min":0.67000000000000004,"15min":0.73099999999999998},"last":0.374}]},{"osd":4,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.63600000000000001,"15min":0.65200000000000002},"min":{"1min":0.48799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81899999999999995,"5min":0.91700000000000004,"15min":1.097},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65200000000000002,"15min":0.67500000000000004},"min":{"1min":0.435,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78800000000000003,"5min":0.99299999999999999,"15min":1.0569999999999999},"last":0.46100000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68400000000000005,"15min":0.69699999999999995},"min":{"1min":0.48999999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92100000000000004,"5min":0.995,"15min":1.1599999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.72499999999999998,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.82099999999999995,"5min":1.171,"15min":1.171},"last":0.57099999999999995}]},{"osd":6,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68500000000000005,"15min":0.70799999999999996},"min":{"1min":0.48099999999999998,"5min":0.436,"15min":0.42499999999999999},"max":{"1min":0.90500000000000003,"5min":1.012,"15min":1.194},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.71799999999999997,"15min":0.73299999999999998},"min":{"1min":0.53300000000000003,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":0.88200000000000001,"5min":1.0409999999999999,"15min":1.2410000000000001},"last":0.51600000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.70099999999999996,"15min":0.71899999999999997},"min":{"1min":0.49099999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82799999999999996,"5min":1.079,"15min":1.079},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68100000000000005,"15min":0.69999999999999996},"min":{"1min":0.44500000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83799999999999997,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.48399999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575985,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64352,"kb_used_data":2900,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307488,"statfs":{"total":96636764160,"available":96570867712,"internally_reserved":0,"allocated":2969600,"data_stored":1524016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.57899999999999996,"15min":0.59299999999999997},"min":{"1min":0.34300000000000003,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65300000000000002,"5min":0.77500000000000002,"15min":0.81799999999999995},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57799999999999996,"15min":0.59799999999999998},"min":{"1min":0.40300000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69999999999999996,"5min":0.79600000000000004,"15min":0.84199999999999997},"last":0.80600000000000005}]},{"osd":1,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59999999999999998,"15min":0.61399999999999999},"min":{"1min":0.438,"5min":0.318,"15min":0.318},"max":{"1min":0.73199999999999998,"5min":0.81899999999999995,"15min":0.88300000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56999999999999995,"15min":0.58499999999999996},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.71199999999999997,"5min":0.79200000000000004,"15min":0.96499999999999997},"last":0.73099999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58299999999999996,"15min":0.59899999999999998},"min":{"1min":0.439,"5min":0.39500000000000002,"15min":0.39200000000000002},"max":{"1min":0.68600000000000005,"5min":0.80500000000000005,"15min":0.82599999999999996},"last":0.748},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61099999999999999,"15min":0.61799999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.38500000000000001},"max":{"1min":0.81399999999999995,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.67200000000000004}]},{"osd":3,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.40799999999999997,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.86199999999999999},"last":0.70499999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.498,"15min":0.504},"min":{"1min":0.35199999999999998,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.85499999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.53300000000000003}]},{"osd":6,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.499,"15min":0.505},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.621}]},{"osd":7,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54100000000000004,"15min":0.54800000000000004},"min":{"1min":0.435,"5min":0.38,"15min":0.35699999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.57199999999999995,"15min":0.57999999999999996},"min":{"1min":0.41899999999999998,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.56200000000000006}]}]},{"osd":5,"up_from":13,"seq":55834575984,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47992,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323848,"statfs":{"total":96636764160,"available":96587620352,"internally_reserved":0,"allocated":2985984,"data_stored":1525034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42499999999999999,"5min":0.33900000000000002,"15min":0.31900000000000001},"max":{"1min":0.72699999999999998,"5min":0.85999999999999999,"15min":0.96499999999999997},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57799999999999996,"15min":0.60399999999999998},"min":{"1min":0.432,"5min":0.37,"15min":0.33100000000000002},"max":{"1min":0.69799999999999995,"5min":0.82599999999999996,"15min":1.3100000000000001},"last":0.68000000000000005}]},{"osd":1,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58499999999999996,"15min":0.60199999999999998},"min":{"1min":0.36899999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.74399999999999999,"5min":0.79600000000000004,"15min":1.056},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58399999999999996,"15min":0.60199999999999998},"min":{"1min":0.46600000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.63100000000000001,"5min":0.89800000000000002,"15min":1.018},"last":0.84399999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.38700000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.84299999999999997,"15min":0.99299999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59399999999999997,"15min":0.60899999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.67500000000000004,"5min":0.875,"15min":1.1120000000000001},"last":0.73799999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.42799999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.78100000000000003,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63,"15min":0.64600000000000002},"min":{"1min":0.443,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69899999999999995,"5min":0.82399999999999995,"15min":1.161},"last":0.71999999999999997}]},{"osd":4,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.47899999999999998,"15min":0.47499999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.77500000000000002},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51200000000000001,"15min":0.50600000000000001},"min":{"1min":0.36199999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.89700000000000002},"last":0.75700000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.503,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.34000000000000002,"15min":0.32800000000000001},"max":{"1min":0.69999999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.40300000000000002,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.86599999999999999},"last":0.59999999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.40799999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.94499999999999995},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.45300000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74099999999999999,"5min":0.85999999999999999,"15min":0.91200000000000003},"last":0.63600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:25:44.394 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:25:44.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.529+0000 7f26bf112700 1 Processor -- start 2024-03-18T20:25:44.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.529+0000 7f26bf112700 1 -- start start 2024-03-18T20:25:44.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.530+0000 7f26bf112700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f26b806a750 0x7f26b806ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:44.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.530+0000 7f26bf112700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f26b815d6b0 0x7f26b8160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:44.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.530+0000 7f26bf112700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26b80ad790 0x7f26b8161530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:44.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.530+0000 7f26bf112700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b80af7a0 con 0x7f26b80ad790 2024-03-18T20:25:44.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.530+0000 7f26bf112700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b80acfb0 con 0x7f26b806a750 2024-03-18T20:25:44.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.530+0000 7f26bf112700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b806b070 con 0x7f26b815d6b0 2024-03-18T20:25:44.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.530+0000 7f26bd6af700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26b80ad790 0x7f26b8161530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.530+0000 7f26bd6af700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26b80ad790 0x7f26b8161530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54074/0 (socket says 172.21.15.1:54074) 2024-03-18T20:25:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.530+0000 7f26bd6af700 1 -- 172.21.15.1:0/3071401318 learned_addr learned my addr 172.21.15.1:0/3071401318 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.530+0000 7f26b7fff700 1 --2- 172.21.15.1:0/3071401318 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f26b815d6b0 0x7f26b8160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.530+0000 7f26bceae700 1 --2- 172.21.15.1:0/3071401318 >> v2:172.21.15.165:3300/0 conn(0x7f26b806a750 0x7f26b806ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.531+0000 7f26bd6af700 1 -- 172.21.15.1:0/3071401318 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f26b815d6b0 msgr2=0x7f26b8160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.531+0000 7f26bd6af700 1 --2- 172.21.15.1:0/3071401318 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f26b815d6b0 0x7f26b8160ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.531+0000 7f26bd6af700 1 -- 172.21.15.1:0/3071401318 >> v2:172.21.15.165:3300/0 conn(0x7f26b806a750 msgr2=0x7f26b806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.531+0000 7f26bd6af700 1 --2- 172.21.15.1:0/3071401318 >> v2:172.21.15.165:3300/0 conn(0x7f26b806a750 0x7f26b806ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.531+0000 7f26bd6af700 1 -- 172.21.15.1:0/3071401318 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b80b50a0 con 0x7f26b80ad790 2024-03-18T20:25:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.531+0000 7f26bceae700 1 --2- 172.21.15.1:0/3071401318 >> v2:172.21.15.165:3300/0 conn(0x7f26b806a750 0x7f26b806ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.531+0000 7f26bd6af700 1 --2- 172.21.15.1:0/3071401318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26b80ad790 0x7f26b8161530 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f26b8073a90 tx=0x7f26b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb2fb96d67b4f61 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.531+0000 7f26b77fe700 1 -- 172.21.15.1:0/3071401318 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b000e070 con 0x7f26b80ad790 2024-03-18T20:25:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.531+0000 7f26b77fe700 1 -- 172.21.15.1:0/3071401318 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b000b910 con 0x7f26b80ad790 2024-03-18T20:25:44.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.531+0000 7f26b77fe700 1 -- 172.21.15.1:0/3071401318 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b000bcc0 con 0x7f26b80ad790 2024-03-18T20:25:44.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.531+0000 7f26bf112700 1 -- 172.21.15.1:0/3071401318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26b80ad790 msgr2=0x7f26b8161530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:44.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.531+0000 7f26bf112700 1 --2- 172.21.15.1:0/3071401318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26b80ad790 0x7f26b8161530 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f26b8073a90 tx=0x7f26b000b330 comp rx=0 tx=0).stop 2024-03-18T20:25:44.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.532+0000 7f26bf112700 1 -- 172.21.15.1:0/3071401318 shutdown_connections 2024-03-18T20:25:44.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.532+0000 7f26bf112700 1 --2- 172.21.15.1:0/3071401318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26b80ad790 0x7f26b8161530 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.532+0000 7f26bf112700 1 --2- 172.21.15.1:0/3071401318 >> v2:172.21.15.165:3300/0 conn(0x7f26b806a750 0x7f26b806ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.532+0000 7f26bf112700 1 --2- 172.21.15.1:0/3071401318 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f26b815d6b0 0x7f26b8160ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.532+0000 7f26bf112700 1 -- 172.21.15.1:0/3071401318 >> 172.21.15.1:0/3071401318 conn(0x7f26b806b860 msgr2=0x7f26b806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:44.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.532+0000 7f26bf112700 1 -- 172.21.15.1:0/3071401318 shutdown_connections 2024-03-18T20:25:44.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.532+0000 7f26bf112700 1 -- 172.21.15.1:0/3071401318 wait complete. 2024-03-18T20:25:44.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.533+0000 7f26bf112700 1 Processor -- start 2024-03-18T20:25:44.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.533+0000 7f26bf112700 1 -- start start 2024-03-18T20:25:44.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.534+0000 7f26bf112700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26b806a750 0x7f26b8158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:44.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.534+0000 7f26bf112700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f26b80ad790 0x7f26b80cde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:44.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.534+0000 7f26bf112700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26b815d6b0 0x7f26b815b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:44.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.534+0000 7f26bf112700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b81590f0 con 0x7f26b815d6b0 2024-03-18T20:25:44.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.534+0000 7f26bf112700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b80ce370 con 0x7f26b806a750 2024-03-18T20:25:44.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.534+0000 7f26bf112700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b8159540 con 0x7f26b80ad790 2024-03-18T20:25:44.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.534+0000 7f26bd6af700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26b815d6b0 0x7f26b815b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:44.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.534+0000 7f26bd6af700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26b815d6b0 0x7f26b815b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54084/0 (socket says 172.21.15.1:54084) 2024-03-18T20:25:44.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.534+0000 7f26bd6af700 1 -- 172.21.15.1:0/2600224190 learned_addr learned my addr 172.21.15.1:0/2600224190 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:44.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.534+0000 7f26b7fff700 1 --2- 172.21.15.1:0/2600224190 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f26b80ad790 0x7f26b80cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:44.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.534+0000 7f26bceae700 1 --2- 172.21.15.1:0/2600224190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26b806a750 0x7f26b8158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:44.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.534+0000 7f26bd6af700 1 -- 172.21.15.1:0/2600224190 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f26b80ad790 msgr2=0x7f26b80cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:44.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.534+0000 7f26bd6af700 1 --2- 172.21.15.1:0/2600224190 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f26b80ad790 0x7f26b80cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.534+0000 7f26bd6af700 1 -- 172.21.15.1:0/2600224190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26b806a750 msgr2=0x7f26b8158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:44.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.534+0000 7f26bd6af700 1 --2- 172.21.15.1:0/2600224190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26b806a750 0x7f26b8158bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.534+0000 7f26bd6af700 1 -- 172.21.15.1:0/2600224190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b0009b30 con 0x7f26b815d6b0 2024-03-18T20:25:44.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.534+0000 7f26bceae700 1 --2- 172.21.15.1:0/2600224190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26b806a750 0x7f26b8158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:44.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.534+0000 7f26bd6af700 1 --2- 172.21.15.1:0/2600224190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26b815d6b0 0x7f26b815b2b0 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f26b80672c0 tx=0x7f26b0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:44.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.534+0000 7f26b57fa700 1 -- 172.21.15.1:0/2600224190 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b00099b0 con 0x7f26b815d6b0 2024-03-18T20:25:44.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.535+0000 7f26b57fa700 1 -- 172.21.15.1:0/2600224190 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b00092e0 con 0x7f26b815d6b0 2024-03-18T20:25:44.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.535+0000 7f26bf112700 1 -- 172.21.15.1:0/2600224190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b8159820 con 0x7f26b815d6b0 2024-03-18T20:25:44.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.535+0000 7f26b57fa700 1 -- 172.21.15.1:0/2600224190 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b0004470 con 0x7f26b815d6b0 2024-03-18T20:25:44.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.535+0000 7f26bf112700 1 -- 172.21.15.1:0/2600224190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b8159de0 con 0x7f26b815d6b0 2024-03-18T20:25:44.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.536+0000 7f26bf112700 1 -- 172.21.15.1:0/2600224190 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b80b4c60 con 0x7f26b815d6b0 2024-03-18T20:25:44.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.536+0000 7f26b57fa700 1 -- 172.21.15.1:0/2600224190 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f26b00045d0 con 0x7f26b815d6b0 2024-03-18T20:25:44.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.536+0000 7f26b57fa700 1 --2- 172.21.15.1:0/2600224190 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f269c06adb0 0x7f269c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:44.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.537+0000 7f26b57fa700 1 -- 172.21.15.1:0/2600224190 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f26b008aed0 con 0x7f26b815d6b0 2024-03-18T20:25:44.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.537+0000 7f26bceae700 1 --2- 172.21.15.1:0/2600224190 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f269c06adb0 0x7f269c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:44.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.537+0000 7f26bceae700 1 --2- 172.21.15.1:0/2600224190 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f269c06adb0 0x7f269c06d270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f26ac003930 tx=0x7f26ac008040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:44.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.541+0000 7f26b57fa700 1 -- 172.21.15.1:0/2600224190 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f26b0059420 con 0x7f26b815d6b0 2024-03-18T20:25:44.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.741+0000 7f26bf112700 1 -- 172.21.15.1:0/2600224190 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f26b80ad440 con 0x7f269c06adb0 2024-03-18T20:25:44.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.755+0000 7f26b57fa700 1 -- 172.21.15.1:0/2600224190 <== mgr.11322 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+529167 (secure 0 0 0) 0x7f26ac003e50 con 0x7f269c06adb0 2024-03-18T20:25:44.766 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:44.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.759+0000 7f26bf112700 1 -- 172.21.15.1:0/2600224190 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f269c06adb0 msgr2=0x7f269c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:44.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.759+0000 7f26bf112700 1 --2- 172.21.15.1:0/2600224190 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f269c06adb0 0x7f269c06d270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f26ac003930 tx=0x7f26ac008040 comp rx=0 tx=0).stop 2024-03-18T20:25:44.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.759+0000 7f26bf112700 1 -- 172.21.15.1:0/2600224190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26b815d6b0 msgr2=0x7f26b815b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:44.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.759+0000 7f26bf112700 1 --2- 172.21.15.1:0/2600224190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26b815d6b0 0x7f26b815b2b0 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f26b80672c0 tx=0x7f26b0004ad0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.760+0000 7f26bf112700 1 -- 172.21.15.1:0/2600224190 shutdown_connections 2024-03-18T20:25:44.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.760+0000 7f26bf112700 1 --2- 172.21.15.1:0/2600224190 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f269c06adb0 0x7f269c06d270 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.760+0000 7f26bf112700 1 --2- 172.21.15.1:0/2600224190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26b815d6b0 0x7f26b815b2b0 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.760+0000 7f26bf112700 1 --2- 172.21.15.1:0/2600224190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26b806a750 0x7f26b8158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.760+0000 7f26bf112700 1 --2- 172.21.15.1:0/2600224190 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f26b80ad790 0x7f26b80cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.760+0000 7f26bf112700 1 -- 172.21.15.1:0/2600224190 >> 172.21.15.1:0/2600224190 conn(0x7f26b806b860 msgr2=0x7f26b815c890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.760+0000 7f26bf112700 1 -- 172.21.15.1:0/2600224190 shutdown_connections 2024-03-18T20:25:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.760+0000 7f26bf112700 1 -- 172.21.15.1:0/2600224190 wait complete. 2024-03-18T20:25:44.772 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:25:44.776 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-03-18T20:25:43.103801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596350,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":671,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":428,"ondisk_log_size":428,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":385756,"kb_used_data":24424,"kb_used_omap":12,"kb_used_meta":361203,"kb_avail":754588964,"statfs":{"total":773094113280,"available":772699099136,"internally_reserved":0,"allocated":25010176,"data_stored":13379452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":369872441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.984880"},"pg_stats":[{"pgid":"1.0","version":"221'348","reported_seq":1204,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561810+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:25:35.561810+0000","last_peered":"2024-03-18T20:25:35.561810+0000","last_clean":"2024-03-18T20:25:35.561810+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:25:35.561810+0000","last_undegraded":"2024-03-18T20:25:35.561810+0000","last_fullsized":"2024-03-18T20:25:35.561810+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1a","version":"231'1","reported_seq":16,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560978+0000","last_change":"2024-03-18T20:25:22.248248+0000","last_active":"2024-03-18T20:25:35.560978+0000","last_peered":"2024-03-18T20:25:35.560978+0000","last_clean":"2024-03-18T20:25:35.560978+0000","last_became_active":"2024-03-18T20:25:22.247970+0000","last_became_peered":"2024-03-18T20:25:22.247970+0000","last_unstale":"2024-03-18T20:25:35.560978+0000","last_undegraded":"2024-03-18T20:25:35.560978+0000","last_fullsized":"2024-03-18T20:25:35.560978+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:01:50.523530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.572089+0000","last_change":"2024-03-18T20:25:22.687598+0000","last_active":"2024-03-18T20:25:35.572089+0000","last_peered":"2024-03-18T20:25:35.572089+0000","last_clean":"2024-03-18T20:25:35.572089+0000","last_became_active":"2024-03-18T20:25:22.686962+0000","last_became_peered":"2024-03-18T20:25:22.686962+0000","last_unstale":"2024-03-18T20:25:35.572089+0000","last_undegraded":"2024-03-18T20:25:35.572089+0000","last_fullsized":"2024-03-18T20:25:35.572089+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:23:10.983253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1c","version":"235'2","reported_seq":13,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.583521+0000","last_change":"2024-03-18T20:25:35.139062+0000","last_active":"2024-03-18T20:25:35.583521+0000","last_peered":"2024-03-18T20:25:35.583521+0000","last_clean":"2024-03-18T20:25:35.583521+0000","last_became_active":"2024-03-18T20:25:35.138799+0000","last_became_peered":"2024-03-18T20:25:35.138799+0000","last_unstale":"2024-03-18T20:25:35.583521+0000","last_undegraded":"2024-03-18T20:25:35.583521+0000","last_fullsized":"2024-03-18T20:25:35.583521+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:47.366224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188735+0000","last_change":"2024-03-18T20:25:35.575458+0000","last_active":"2024-03-18T20:25:38.188735+0000","last_peered":"2024-03-18T20:25:38.188735+0000","last_clean":"2024-03-18T20:25:38.188735+0000","last_became_active":"2024-03-18T20:25:35.574984+0000","last_became_peered":"2024-03-18T20:25:35.574984+0000","last_unstale":"2024-03-18T20:25:38.188735+0000","last_undegraded":"2024-03-18T20:25:38.188735+0000","last_fullsized":"2024-03-18T20:25:38.188735+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:26.055273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.564794+0000","last_change":"2024-03-18T20:25:22.252803+0000","last_active":"2024-03-18T20:25:35.564794+0000","last_peered":"2024-03-18T20:25:35.564794+0000","last_clean":"2024-03-18T20:25:35.564794+0000","last_became_active":"2024-03-18T20:25:22.252169+0000","last_became_peered":"2024-03-18T20:25:22.252169+0000","last_unstale":"2024-03-18T20:25:35.564794+0000","last_undegraded":"2024-03-18T20:25:35.564794+0000","last_fullsized":"2024-03-18T20:25:35.564794+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:12.985428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.555564+0000","last_change":"2024-03-18T20:25:22.696025+0000","last_active":"2024-03-18T20:25:35.555564+0000","last_peered":"2024-03-18T20:25:35.555564+0000","last_clean":"2024-03-18T20:25:35.555564+0000","last_became_active":"2024-03-18T20:25:22.695759+0000","last_became_peered":"2024-03-18T20:25:22.695759+0000","last_unstale":"2024-03-18T20:25:35.555564+0000","last_undegraded":"2024-03-18T20:25:35.555564+0000","last_fullsized":"2024-03-18T20:25:35.555564+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:10.681789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.6","version":"236'1","reported_seq":12,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:37.348587+0000","last_change":"2024-03-18T20:25:35.132963+0000","last_active":"2024-03-18T20:25:37.348587+0000","last_peered":"2024-03-18T20:25:37.348587+0000","last_clean":"2024-03-18T20:25:37.348587+0000","last_became_active":"2024-03-18T20:25:35.132654+0000","last_became_peered":"2024-03-18T20:25:35.132654+0000","last_unstale":"2024-03-18T20:25:37.348587+0000","last_undegraded":"2024-03-18T20:25:37.348587+0000","last_fullsized":"2024-03-18T20:25:37.348587+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:13.759015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188001+0000","last_change":"2024-03-18T20:25:35.579270+0000","last_active":"2024-03-18T20:25:38.188001+0000","last_peered":"2024-03-18T20:25:38.188001+0000","last_clean":"2024-03-18T20:25:38.188001+0000","last_became_active":"2024-03-18T20:25:35.578800+0000","last_became_peered":"2024-03-18T20:25:35.578800+0000","last_unstale":"2024-03-18T20:25:38.188001+0000","last_undegraded":"2024-03-18T20:25:38.188001+0000","last_fullsized":"2024-03-18T20:25:38.188001+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:58:58.372932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560229+0000","last_change":"2024-03-18T20:25:22.252889+0000","last_active":"2024-03-18T20:25:35.560229+0000","last_peered":"2024-03-18T20:25:35.560229+0000","last_clean":"2024-03-18T20:25:35.560229+0000","last_became_active":"2024-03-18T20:25:22.252096+0000","last_became_peered":"2024-03-18T20:25:22.252096+0000","last_unstale":"2024-03-18T20:25:35.560229+0000","last_undegraded":"2024-03-18T20:25:35.560229+0000","last_fullsized":"2024-03-18T20:25:35.560229+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:18.568149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561455+0000","last_change":"2024-03-18T20:25:22.689532+0000","last_active":"2024-03-18T20:25:35.561455+0000","last_peered":"2024-03-18T20:25:35.561455+0000","last_clean":"2024-03-18T20:25:35.561455+0000","last_became_active":"2024-03-18T20:25:22.689210+0000","last_became_peered":"2024-03-18T20:25:22.689210+0000","last_unstale":"2024-03-18T20:25:35.561455+0000","last_undegraded":"2024-03-18T20:25:35.561455+0000","last_fullsized":"2024-03-18T20:25:35.561455+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:21.229791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.7","version":"235'2","reported_seq":13,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.582084+0000","last_change":"2024-03-18T20:25:35.145037+0000","last_active":"2024-03-18T20:25:35.582084+0000","last_peered":"2024-03-18T20:25:35.582084+0000","last_clean":"2024-03-18T20:25:35.582084+0000","last_became_active":"2024-03-18T20:25:35.144629+0000","last_became_peered":"2024-03-18T20:25:35.144629+0000","last_unstale":"2024-03-18T20:25:35.582084+0000","last_undegraded":"2024-03-18T20:25:35.582084+0000","last_fullsized":"2024-03-18T20:25:35.582084+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:17.295248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187642+0000","last_change":"2024-03-18T20:25:35.576406+0000","last_active":"2024-03-18T20:25:38.187642+0000","last_peered":"2024-03-18T20:25:38.187642+0000","last_clean":"2024-03-18T20:25:38.187642+0000","last_became_active":"2024-03-18T20:25:35.575245+0000","last_became_peered":"2024-03-18T20:25:35.575245+0000","last_unstale":"2024-03-18T20:25:38.187642+0000","last_undegraded":"2024-03-18T20:25:38.187642+0000","last_fullsized":"2024-03-18T20:25:38.187642+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:31.956522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561056+0000","last_change":"2024-03-18T20:25:22.254053+0000","last_active":"2024-03-18T20:25:35.561056+0000","last_peered":"2024-03-18T20:25:35.561056+0000","last_clean":"2024-03-18T20:25:35.561056+0000","last_became_active":"2024-03-18T20:25:22.253747+0000","last_became_peered":"2024-03-18T20:25:22.253747+0000","last_unstale":"2024-03-18T20:25:35.561056+0000","last_undegraded":"2024-03-18T20:25:35.561056+0000","last_fullsized":"2024-03-18T20:25:35.561056+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:32.777143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561855+0000","last_change":"2024-03-18T20:25:22.685300+0000","last_active":"2024-03-18T20:25:35.561855+0000","last_peered":"2024-03-18T20:25:35.561855+0000","last_clean":"2024-03-18T20:25:35.561855+0000","last_became_active":"2024-03-18T20:25:22.684869+0000","last_became_peered":"2024-03-18T20:25:22.684869+0000","last_unstale":"2024-03-18T20:25:35.561855+0000","last_undegraded":"2024-03-18T20:25:35.561855+0000","last_fullsized":"2024-03-18T20:25:35.561855+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:02.037780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.559649+0000","last_change":"2024-03-18T20:25:35.137734+0000","last_active":"2024-03-18T20:25:35.559649+0000","last_peered":"2024-03-18T20:25:35.559649+0000","last_clean":"2024-03-18T20:25:35.559649+0000","last_became_active":"2024-03-18T20:25:35.137503+0000","last_became_peered":"2024-03-18T20:25:35.137503+0000","last_unstale":"2024-03-18T20:25:35.559649+0000","last_undegraded":"2024-03-18T20:25:35.559649+0000","last_fullsized":"2024-03-18T20:25:35.559649+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:44:27.205313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.567713+0000","last_change":"2024-03-18T20:25:35.570919+0000","last_active":"2024-03-18T20:25:36.567713+0000","last_peered":"2024-03-18T20:25:36.567713+0000","last_clean":"2024-03-18T20:25:36.567713+0000","last_became_active":"2024-03-18T20:25:35.570665+0000","last_became_peered":"2024-03-18T20:25:35.570665+0000","last_unstale":"2024-03-18T20:25:36.567713+0000","last_undegraded":"2024-03-18T20:25:36.567713+0000","last_fullsized":"2024-03-18T20:25:36.567713+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:46.672966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.564287+0000","last_change":"2024-03-18T20:25:22.245633+0000","last_active":"2024-03-18T20:25:35.564287+0000","last_peered":"2024-03-18T20:25:35.564287+0000","last_clean":"2024-03-18T20:25:35.564287+0000","last_became_active":"2024-03-18T20:25:22.245372+0000","last_became_peered":"2024-03-18T20:25:22.245372+0000","last_unstale":"2024-03-18T20:25:35.564287+0000","last_undegraded":"2024-03-18T20:25:35.564287+0000","last_fullsized":"2024-03-18T20:25:35.564287+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:28:25.048682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.559740+0000","last_change":"2024-03-18T20:25:22.683574+0000","last_active":"2024-03-18T20:25:35.559740+0000","last_peered":"2024-03-18T20:25:35.559740+0000","last_clean":"2024-03-18T20:25:35.559740+0000","last_became_active":"2024-03-18T20:25:22.683015+0000","last_became_peered":"2024-03-18T20:25:22.683015+0000","last_unstale":"2024-03-18T20:25:35.559740+0000","last_undegraded":"2024-03-18T20:25:35.559740+0000","last_fullsized":"2024-03-18T20:25:35.559740+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:11.606826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.555834+0000","last_change":"2024-03-18T20:25:35.133030+0000","last_active":"2024-03-18T20:25:35.555834+0000","last_peered":"2024-03-18T20:25:35.555834+0000","last_clean":"2024-03-18T20:25:35.555834+0000","last_became_active":"2024-03-18T20:25:35.132715+0000","last_became_peered":"2024-03-18T20:25:35.132715+0000","last_unstale":"2024-03-18T20:25:35.555834+0000","last_undegraded":"2024-03-18T20:25:35.555834+0000","last_fullsized":"2024-03-18T20:25:35.555834+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:41.670198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.923677+0000","last_change":"2024-03-18T20:25:35.574963+0000","last_active":"2024-03-18T20:25:36.923677+0000","last_peered":"2024-03-18T20:25:36.923677+0000","last_clean":"2024-03-18T20:25:36.923677+0000","last_became_active":"2024-03-18T20:25:35.574685+0000","last_became_peered":"2024-03-18T20:25:35.574685+0000","last_unstale":"2024-03-18T20:25:36.923677+0000","last_undegraded":"2024-03-18T20:25:36.923677+0000","last_fullsized":"2024-03-18T20:25:36.923677+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:15.394818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.571863+0000","last_change":"2024-03-18T20:25:22.255435+0000","last_active":"2024-03-18T20:25:35.571863+0000","last_peered":"2024-03-18T20:25:35.571863+0000","last_clean":"2024-03-18T20:25:35.571863+0000","last_became_active":"2024-03-18T20:25:22.255149+0000","last_became_peered":"2024-03-18T20:25:22.255149+0000","last_unstale":"2024-03-18T20:25:35.571863+0000","last_undegraded":"2024-03-18T20:25:35.571863+0000","last_fullsized":"2024-03-18T20:25:35.571863+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:58.960198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.564166+0000","last_change":"2024-03-18T20:25:22.682910+0000","last_active":"2024-03-18T20:25:35.564166+0000","last_peered":"2024-03-18T20:25:35.564166+0000","last_clean":"2024-03-18T20:25:35.564166+0000","last_became_active":"2024-03-18T20:25:22.682667+0000","last_became_peered":"2024-03-18T20:25:22.682667+0000","last_unstale":"2024-03-18T20:25:35.564166+0000","last_undegraded":"2024-03-18T20:25:35.564166+0000","last_fullsized":"2024-03-18T20:25:35.564166+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:02.037780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.572087+0000","last_change":"2024-03-18T20:25:35.137107+0000","last_active":"2024-03-18T20:25:35.572087+0000","last_peered":"2024-03-18T20:25:35.572087+0000","last_clean":"2024-03-18T20:25:35.572087+0000","last_became_active":"2024-03-18T20:25:35.132663+0000","last_became_peered":"2024-03-18T20:25:35.132663+0000","last_unstale":"2024-03-18T20:25:35.572087+0000","last_undegraded":"2024-03-18T20:25:35.572087+0000","last_fullsized":"2024-03-18T20:25:35.572087+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:56.513767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188079+0000","last_change":"2024-03-18T20:25:35.579484+0000","last_active":"2024-03-18T20:25:38.188079+0000","last_peered":"2024-03-18T20:25:38.188079+0000","last_clean":"2024-03-18T20:25:38.188079+0000","last_became_active":"2024-03-18T20:25:35.579220+0000","last_became_peered":"2024-03-18T20:25:35.579220+0000","last_unstale":"2024-03-18T20:25:38.188079+0000","last_undegraded":"2024-03-18T20:25:38.188079+0000","last_fullsized":"2024-03-18T20:25:38.188079+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:08:55.975267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.574988+0000","last_change":"2024-03-18T20:25:22.250457+0000","last_active":"2024-03-18T20:25:35.574988+0000","last_peered":"2024-03-18T20:25:35.574988+0000","last_clean":"2024-03-18T20:25:35.574988+0000","last_became_active":"2024-03-18T20:25:22.250157+0000","last_became_peered":"2024-03-18T20:25:22.250157+0000","last_unstale":"2024-03-18T20:25:35.574988+0000","last_undegraded":"2024-03-18T20:25:35.574988+0000","last_fullsized":"2024-03-18T20:25:35.574988+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:18.568149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560885+0000","last_change":"2024-03-18T20:25:22.691245+0000","last_active":"2024-03-18T20:25:35.560885+0000","last_peered":"2024-03-18T20:25:35.560885+0000","last_clean":"2024-03-18T20:25:35.560885+0000","last_became_active":"2024-03-18T20:25:22.690988+0000","last_became_peered":"2024-03-18T20:25:22.690988+0000","last_unstale":"2024-03-18T20:25:35.560885+0000","last_undegraded":"2024-03-18T20:25:35.560885+0000","last_fullsized":"2024-03-18T20:25:35.560885+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:32:54.125659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561627+0000","last_change":"2024-03-18T20:25:35.144694+0000","last_active":"2024-03-18T20:25:35.561627+0000","last_peered":"2024-03-18T20:25:35.561627+0000","last_clean":"2024-03-18T20:25:35.561627+0000","last_became_active":"2024-03-18T20:25:35.144468+0000","last_became_peered":"2024-03-18T20:25:35.144468+0000","last_unstale":"2024-03-18T20:25:35.561627+0000","last_undegraded":"2024-03-18T20:25:35.561627+0000","last_fullsized":"2024-03-18T20:25:35.561627+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:25:32.071310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187577+0000","last_change":"2024-03-18T20:25:35.581378+0000","last_active":"2024-03-18T20:25:38.187577+0000","last_peered":"2024-03-18T20:25:38.187577+0000","last_clean":"2024-03-18T20:25:38.187577+0000","last_became_active":"2024-03-18T20:25:35.575394+0000","last_became_peered":"2024-03-18T20:25:35.575394+0000","last_unstale":"2024-03-18T20:25:38.187577+0000","last_undegraded":"2024-03-18T20:25:38.187577+0000","last_fullsized":"2024-03-18T20:25:38.187577+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:43:21.746640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.6","version":"232'1","reported_seq":16,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560294+0000","last_change":"2024-03-18T20:25:22.247183+0000","last_active":"2024-03-18T20:25:35.560294+0000","last_peered":"2024-03-18T20:25:35.560294+0000","last_clean":"2024-03-18T20:25:35.560294+0000","last_became_active":"2024-03-18T20:25:22.246890+0000","last_became_peered":"2024-03-18T20:25:22.246890+0000","last_unstale":"2024-03-18T20:25:35.560294+0000","last_undegraded":"2024-03-18T20:25:35.560294+0000","last_fullsized":"2024-03-18T20:25:35.560294+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:40:21.886849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.557004+0000","last_change":"2024-03-18T20:25:22.689011+0000","last_active":"2024-03-18T20:25:35.557004+0000","last_peered":"2024-03-18T20:25:35.557004+0000","last_clean":"2024-03-18T20:25:35.557004+0000","last_became_active":"2024-03-18T20:25:22.688750+0000","last_became_peered":"2024-03-18T20:25:22.688750+0000","last_unstale":"2024-03-18T20:25:35.557004+0000","last_undegraded":"2024-03-18T20:25:35.557004+0000","last_fullsized":"2024-03-18T20:25:35.557004+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:57.345369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.556476+0000","last_change":"2024-03-18T20:25:35.145538+0000","last_active":"2024-03-18T20:25:35.556476+0000","last_peered":"2024-03-18T20:25:35.556476+0000","last_clean":"2024-03-18T20:25:35.556476+0000","last_became_active":"2024-03-18T20:25:35.145257+0000","last_became_peered":"2024-03-18T20:25:35.145257+0000","last_unstale":"2024-03-18T20:25:35.556476+0000","last_undegraded":"2024-03-18T20:25:35.556476+0000","last_fullsized":"2024-03-18T20:25:35.556476+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:25.724222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.563395+0000","last_change":"2024-03-18T20:25:35.582929+0000","last_active":"2024-03-18T20:25:36.563395+0000","last_peered":"2024-03-18T20:25:36.563395+0000","last_clean":"2024-03-18T20:25:36.563395+0000","last_became_active":"2024-03-18T20:25:35.582610+0000","last_became_peered":"2024-03-18T20:25:35.582610+0000","last_unstale":"2024-03-18T20:25:36.563395+0000","last_undegraded":"2024-03-18T20:25:36.563395+0000","last_fullsized":"2024-03-18T20:25:36.563395+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:56:23.570308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"231'2","reported_seq":17,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565668+0000","last_change":"2024-03-18T20:25:22.254151+0000","last_active":"2024-03-18T20:25:35.565668+0000","last_peered":"2024-03-18T20:25:35.565668+0000","last_clean":"2024-03-18T20:25:35.565668+0000","last_became_active":"2024-03-18T20:25:22.253873+0000","last_became_peered":"2024-03-18T20:25:22.253873+0000","last_unstale":"2024-03-18T20:25:35.565668+0000","last_undegraded":"2024-03-18T20:25:35.565668+0000","last_fullsized":"2024-03-18T20:25:35.565668+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:26:58.327699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.557400+0000","last_change":"2024-03-18T20:25:22.693170+0000","last_active":"2024-03-18T20:25:35.557400+0000","last_peered":"2024-03-18T20:25:35.557400+0000","last_clean":"2024-03-18T20:25:35.557400+0000","last_became_active":"2024-03-18T20:25:22.691592+0000","last_became_peered":"2024-03-18T20:25:22.691592+0000","last_unstale":"2024-03-18T20:25:35.557400+0000","last_undegraded":"2024-03-18T20:25:35.557400+0000","last_fullsized":"2024-03-18T20:25:35.557400+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:23.897085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560491+0000","last_change":"2024-03-18T20:25:35.132572+0000","last_active":"2024-03-18T20:25:35.560491+0000","last_peered":"2024-03-18T20:25:35.560491+0000","last_clean":"2024-03-18T20:25:35.560491+0000","last_became_active":"2024-03-18T20:25:35.132292+0000","last_became_peered":"2024-03-18T20:25:35.132292+0000","last_unstale":"2024-03-18T20:25:35.560491+0000","last_undegraded":"2024-03-18T20:25:35.560491+0000","last_fullsized":"2024-03-18T20:25:35.560491+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:06:09.253104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.566955+0000","last_change":"2024-03-18T20:25:35.578940+0000","last_active":"2024-03-18T20:25:36.566955+0000","last_peered":"2024-03-18T20:25:36.566955+0000","last_clean":"2024-03-18T20:25:36.566955+0000","last_became_active":"2024-03-18T20:25:35.578646+0000","last_became_peered":"2024-03-18T20:25:35.578646+0000","last_unstale":"2024-03-18T20:25:36.566955+0000","last_undegraded":"2024-03-18T20:25:36.566955+0000","last_fullsized":"2024-03-18T20:25:36.566955+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:21.475337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.563462+0000","last_change":"2024-03-18T20:25:22.251509+0000","last_active":"2024-03-18T20:25:35.563462+0000","last_peered":"2024-03-18T20:25:35.563462+0000","last_clean":"2024-03-18T20:25:35.563462+0000","last_became_active":"2024-03-18T20:25:22.251240+0000","last_became_peered":"2024-03-18T20:25:22.251240+0000","last_unstale":"2024-03-18T20:25:35.563462+0000","last_undegraded":"2024-03-18T20:25:35.563462+0000","last_fullsized":"2024-03-18T20:25:35.563462+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:13:14.870270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560861+0000","last_change":"2024-03-18T20:25:22.695653+0000","last_active":"2024-03-18T20:25:35.560861+0000","last_peered":"2024-03-18T20:25:35.560861+0000","last_clean":"2024-03-18T20:25:35.560861+0000","last_became_active":"2024-03-18T20:25:22.694092+0000","last_became_peered":"2024-03-18T20:25:22.694092+0000","last_unstale":"2024-03-18T20:25:35.560861+0000","last_undegraded":"2024-03-18T20:25:35.560861+0000","last_fullsized":"2024-03-18T20:25:35.560861+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:20.319345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.e","version":"235'2","reported_seq":13,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.574711+0000","last_change":"2024-03-18T20:25:35.143861+0000","last_active":"2024-03-18T20:25:35.574711+0000","last_peered":"2024-03-18T20:25:35.574711+0000","last_clean":"2024-03-18T20:25:35.574711+0000","last_became_active":"2024-03-18T20:25:35.143574+0000","last_became_peered":"2024-03-18T20:25:35.143574+0000","last_unstale":"2024-03-18T20:25:35.574711+0000","last_undegraded":"2024-03-18T20:25:35.574711+0000","last_fullsized":"2024-03-18T20:25:35.574711+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:32:42.191508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187983+0000","last_change":"2024-03-18T20:25:35.575576+0000","last_active":"2024-03-18T20:25:38.187983+0000","last_peered":"2024-03-18T20:25:38.187983+0000","last_clean":"2024-03-18T20:25:38.187983+0000","last_became_active":"2024-03-18T20:25:35.575291+0000","last_became_peered":"2024-03-18T20:25:35.575291+0000","last_unstale":"2024-03-18T20:25:38.187983+0000","last_undegraded":"2024-03-18T20:25:38.187983+0000","last_fullsized":"2024-03-18T20:25:38.187983+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:00:31.185972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.576539+0000","last_change":"2024-03-18T20:25:22.251818+0000","last_active":"2024-03-18T20:25:35.576539+0000","last_peered":"2024-03-18T20:25:35.576539+0000","last_clean":"2024-03-18T20:25:35.576539+0000","last_became_active":"2024-03-18T20:25:22.251407+0000","last_became_peered":"2024-03-18T20:25:22.251407+0000","last_unstale":"2024-03-18T20:25:35.576539+0000","last_undegraded":"2024-03-18T20:25:35.576539+0000","last_fullsized":"2024-03-18T20:25:35.576539+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:29.196742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565352+0000","last_change":"2024-03-18T20:25:22.684929+0000","last_active":"2024-03-18T20:25:35.565352+0000","last_peered":"2024-03-18T20:25:35.565352+0000","last_clean":"2024-03-18T20:25:35.565352+0000","last_became_active":"2024-03-18T20:25:22.684658+0000","last_became_peered":"2024-03-18T20:25:22.684658+0000","last_unstale":"2024-03-18T20:25:35.565352+0000","last_undegraded":"2024-03-18T20:25:35.565352+0000","last_fullsized":"2024-03-18T20:25:35.565352+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:39.733357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"235'1","reported_seq":12,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.583701+0000","last_change":"2024-03-18T20:25:35.138420+0000","last_active":"2024-03-18T20:25:35.583701+0000","last_peered":"2024-03-18T20:25:35.583701+0000","last_clean":"2024-03-18T20:25:35.583701+0000","last_became_active":"2024-03-18T20:25:35.138082+0000","last_became_peered":"2024-03-18T20:25:35.138082+0000","last_unstale":"2024-03-18T20:25:35.583701+0000","last_undegraded":"2024-03-18T20:25:35.583701+0000","last_fullsized":"2024-03-18T20:25:35.583701+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:22:13.432979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187553+0000","last_change":"2024-03-18T20:25:35.577432+0000","last_active":"2024-03-18T20:25:38.187553+0000","last_peered":"2024-03-18T20:25:38.187553+0000","last_clean":"2024-03-18T20:25:38.187553+0000","last_became_active":"2024-03-18T20:25:35.576147+0000","last_became_peered":"2024-03-18T20:25:35.576147+0000","last_unstale":"2024-03-18T20:25:38.187553+0000","last_undegraded":"2024-03-18T20:25:38.187553+0000","last_fullsized":"2024-03-18T20:25:38.187553+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:24:50.952354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.569694+0000","last_change":"2024-03-18T20:25:22.247489+0000","last_active":"2024-03-18T20:25:35.569694+0000","last_peered":"2024-03-18T20:25:35.569694+0000","last_clean":"2024-03-18T20:25:35.569694+0000","last_became_active":"2024-03-18T20:25:22.247203+0000","last_became_peered":"2024-03-18T20:25:22.247203+0000","last_unstale":"2024-03-18T20:25:35.569694+0000","last_undegraded":"2024-03-18T20:25:35.569694+0000","last_fullsized":"2024-03-18T20:25:35.569694+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:35:32.481118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561585+0000","last_change":"2024-03-18T20:25:22.681615+0000","last_active":"2024-03-18T20:25:35.561585+0000","last_peered":"2024-03-18T20:25:35.561585+0000","last_clean":"2024-03-18T20:25:35.561585+0000","last_became_active":"2024-03-18T20:25:22.681261+0000","last_became_peered":"2024-03-18T20:25:22.681261+0000","last_unstale":"2024-03-18T20:25:35.561585+0000","last_undegraded":"2024-03-18T20:25:35.561585+0000","last_fullsized":"2024-03-18T20:25:35.561585+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:15:29.461741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.559342+0000","last_change":"2024-03-18T20:25:35.139343+0000","last_active":"2024-03-18T20:25:35.559342+0000","last_peered":"2024-03-18T20:25:35.559342+0000","last_clean":"2024-03-18T20:25:35.559342+0000","last_became_active":"2024-03-18T20:25:35.139075+0000","last_became_peered":"2024-03-18T20:25:35.139075+0000","last_unstale":"2024-03-18T20:25:35.559342+0000","last_undegraded":"2024-03-18T20:25:35.559342+0000","last_fullsized":"2024-03-18T20:25:35.559342+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:33.109951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.564199+0000","last_change":"2024-03-18T20:25:35.582741+0000","last_active":"2024-03-18T20:25:36.564199+0000","last_peered":"2024-03-18T20:25:36.564199+0000","last_clean":"2024-03-18T20:25:36.564199+0000","last_became_active":"2024-03-18T20:25:35.582115+0000","last_became_peered":"2024-03-18T20:25:35.582115+0000","last_unstale":"2024-03-18T20:25:36.564199+0000","last_undegraded":"2024-03-18T20:25:36.564199+0000","last_fullsized":"2024-03-18T20:25:36.564199+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:03.024404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.b","version":"231'2","reported_seq":17,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565542+0000","last_change":"2024-03-18T20:25:22.251851+0000","last_active":"2024-03-18T20:25:35.565542+0000","last_peered":"2024-03-18T20:25:35.565542+0000","last_clean":"2024-03-18T20:25:35.565542+0000","last_became_active":"2024-03-18T20:25:22.251334+0000","last_became_peered":"2024-03-18T20:25:22.251334+0000","last_unstale":"2024-03-18T20:25:35.565542+0000","last_undegraded":"2024-03-18T20:25:35.565542+0000","last_fullsized":"2024-03-18T20:25:35.565542+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:31:22.889781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.568499+0000","last_change":"2024-03-18T20:25:22.687129+0000","last_active":"2024-03-18T20:25:35.568499+0000","last_peered":"2024-03-18T20:25:35.568499+0000","last_clean":"2024-03-18T20:25:35.568499+0000","last_became_active":"2024-03-18T20:25:22.686871+0000","last_became_peered":"2024-03-18T20:25:22.686871+0000","last_unstale":"2024-03-18T20:25:35.568499+0000","last_undegraded":"2024-03-18T20:25:35.568499+0000","last_fullsized":"2024-03-18T20:25:35.568499+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:18:58.196895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"235'1","reported_seq":12,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.570610+0000","last_change":"2024-03-18T20:25:35.138760+0000","last_active":"2024-03-18T20:25:35.570610+0000","last_peered":"2024-03-18T20:25:35.570610+0000","last_clean":"2024-03-18T20:25:35.570610+0000","last_became_active":"2024-03-18T20:25:35.138499+0000","last_became_peered":"2024-03-18T20:25:35.138499+0000","last_unstale":"2024-03-18T20:25:35.570610+0000","last_undegraded":"2024-03-18T20:25:35.570610+0000","last_fullsized":"2024-03-18T20:25:35.570610+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:13.917940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.189057+0000","last_change":"2024-03-18T20:25:35.582286+0000","last_active":"2024-03-18T20:25:38.189057+0000","last_peered":"2024-03-18T20:25:38.189057+0000","last_clean":"2024-03-18T20:25:38.189057+0000","last_became_active":"2024-03-18T20:25:35.582065+0000","last_became_peered":"2024-03-18T20:25:35.582065+0000","last_unstale":"2024-03-18T20:25:38.189057+0000","last_undegraded":"2024-03-18T20:25:38.189057+0000","last_fullsized":"2024-03-18T20:25:38.189057+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:26:45.688482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.570098+0000","last_change":"2024-03-18T20:25:22.254305+0000","last_active":"2024-03-18T20:25:35.570098+0000","last_peered":"2024-03-18T20:25:35.570098+0000","last_clean":"2024-03-18T20:25:35.570098+0000","last_became_active":"2024-03-18T20:25:22.253898+0000","last_became_peered":"2024-03-18T20:25:22.253898+0000","last_unstale":"2024-03-18T20:25:35.570098+0000","last_undegraded":"2024-03-18T20:25:35.570098+0000","last_fullsized":"2024-03-18T20:25:35.570098+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:29:48.394283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560025+0000","last_change":"2024-03-18T20:25:22.692895+0000","last_active":"2024-03-18T20:25:35.560025+0000","last_peered":"2024-03-18T20:25:35.560025+0000","last_clean":"2024-03-18T20:25:35.560025+0000","last_became_active":"2024-03-18T20:25:22.685766+0000","last_became_peered":"2024-03-18T20:25:22.685766+0000","last_unstale":"2024-03-18T20:25:35.560025+0000","last_undegraded":"2024-03-18T20:25:35.560025+0000","last_fullsized":"2024-03-18T20:25:35.560025+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:16:08.020891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.556303+0000","last_change":"2024-03-18T20:25:35.135046+0000","last_active":"2024-03-18T20:25:35.556303+0000","last_peered":"2024-03-18T20:25:35.556303+0000","last_clean":"2024-03-18T20:25:35.556303+0000","last_became_active":"2024-03-18T20:25:35.134784+0000","last_became_peered":"2024-03-18T20:25:35.134784+0000","last_unstale":"2024-03-18T20:25:35.556303+0000","last_undegraded":"2024-03-18T20:25:35.556303+0000","last_fullsized":"2024-03-18T20:25:35.556303+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:38.721170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187693+0000","last_change":"2024-03-18T20:25:35.578197+0000","last_active":"2024-03-18T20:25:38.187693+0000","last_peered":"2024-03-18T20:25:38.187693+0000","last_clean":"2024-03-18T20:25:38.187693+0000","last_became_active":"2024-03-18T20:25:35.577916+0000","last_became_peered":"2024-03-18T20:25:35.577916+0000","last_unstale":"2024-03-18T20:25:38.187693+0000","last_undegraded":"2024-03-18T20:25:38.187693+0000","last_fullsized":"2024-03-18T20:25:38.187693+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:51:55.984289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.d","version":"231'1","reported_seq":16,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.569332+0000","last_change":"2024-03-18T20:25:22.248228+0000","last_active":"2024-03-18T20:25:35.569332+0000","last_peered":"2024-03-18T20:25:35.569332+0000","last_clean":"2024-03-18T20:25:35.569332+0000","last_became_active":"2024-03-18T20:25:22.247916+0000","last_became_peered":"2024-03-18T20:25:22.247916+0000","last_unstale":"2024-03-18T20:25:35.569332+0000","last_undegraded":"2024-03-18T20:25:35.569332+0000","last_fullsized":"2024-03-18T20:25:35.569332+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:56.236862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.564071+0000","last_change":"2024-03-18T20:25:22.684910+0000","last_active":"2024-03-18T20:25:35.564071+0000","last_peered":"2024-03-18T20:25:35.564071+0000","last_clean":"2024-03-18T20:25:35.564071+0000","last_became_active":"2024-03-18T20:25:22.684472+0000","last_became_peered":"2024-03-18T20:25:22.684472+0000","last_unstale":"2024-03-18T20:25:35.564071+0000","last_undegraded":"2024-03-18T20:25:35.564071+0000","last_fullsized":"2024-03-18T20:25:35.564071+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:24:33.293594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.b","version":"235'2","reported_seq":13,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.583640+0000","last_change":"2024-03-18T20:25:35.138829+0000","last_active":"2024-03-18T20:25:35.583640+0000","last_peered":"2024-03-18T20:25:35.583640+0000","last_clean":"2024-03-18T20:25:35.583640+0000","last_became_active":"2024-03-18T20:25:35.138571+0000","last_became_peered":"2024-03-18T20:25:35.138571+0000","last_unstale":"2024-03-18T20:25:35.583640+0000","last_undegraded":"2024-03-18T20:25:35.583640+0000","last_fullsized":"2024-03-18T20:25:35.583640+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:15.369281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.563098+0000","last_change":"2024-03-18T20:25:35.570376+0000","last_active":"2024-03-18T20:25:36.563098+0000","last_peered":"2024-03-18T20:25:36.563098+0000","last_clean":"2024-03-18T20:25:36.563098+0000","last_became_active":"2024-03-18T20:25:35.570081+0000","last_became_peered":"2024-03-18T20:25:35.570081+0000","last_unstale":"2024-03-18T20:25:36.563098+0000","last_undegraded":"2024-03-18T20:25:36.563098+0000","last_fullsized":"2024-03-18T20:25:36.563098+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:39:05.449499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.e","version":"231'2","reported_seq":17,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.573028+0000","last_change":"2024-03-18T20:25:22.253997+0000","last_active":"2024-03-18T20:25:35.573028+0000","last_peered":"2024-03-18T20:25:35.573028+0000","last_clean":"2024-03-18T20:25:35.573028+0000","last_became_active":"2024-03-18T20:25:22.253649+0000","last_became_peered":"2024-03-18T20:25:22.253649+0000","last_unstale":"2024-03-18T20:25:35.573028+0000","last_undegraded":"2024-03-18T20:25:35.573028+0000","last_fullsized":"2024-03-18T20:25:35.573028+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:00.435897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.563682+0000","last_change":"2024-03-18T20:25:22.687892+0000","last_active":"2024-03-18T20:25:35.563682+0000","last_peered":"2024-03-18T20:25:35.563682+0000","last_clean":"2024-03-18T20:25:35.563682+0000","last_became_active":"2024-03-18T20:25:22.687086+0000","last_became_peered":"2024-03-18T20:25:22.687086+0000","last_unstale":"2024-03-18T20:25:35.563682+0000","last_undegraded":"2024-03-18T20:25:35.563682+0000","last_fullsized":"2024-03-18T20:25:35.563682+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:15:29.461741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.562971+0000","last_change":"2024-03-18T20:25:35.144597+0000","last_active":"2024-03-18T20:25:35.562971+0000","last_peered":"2024-03-18T20:25:35.562971+0000","last_clean":"2024-03-18T20:25:35.562971+0000","last_became_active":"2024-03-18T20:25:35.144316+0000","last_became_peered":"2024-03-18T20:25:35.144316+0000","last_unstale":"2024-03-18T20:25:35.562971+0000","last_undegraded":"2024-03-18T20:25:35.562971+0000","last_fullsized":"2024-03-18T20:25:35.562971+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:17.295248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188344+0000","last_change":"2024-03-18T20:25:35.584065+0000","last_active":"2024-03-18T20:25:38.188344+0000","last_peered":"2024-03-18T20:25:38.188344+0000","last_clean":"2024-03-18T20:25:38.188344+0000","last_became_active":"2024-03-18T20:25:35.583758+0000","last_became_peered":"2024-03-18T20:25:35.583758+0000","last_unstale":"2024-03-18T20:25:38.188344+0000","last_undegraded":"2024-03-18T20:25:38.188344+0000","last_fullsized":"2024-03-18T20:25:38.188344+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:45:27.895703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.f","version":"231'1","reported_seq":16,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.572056+0000","last_change":"2024-03-18T20:25:22.255372+0000","last_active":"2024-03-18T20:25:35.572056+0000","last_peered":"2024-03-18T20:25:35.572056+0000","last_clean":"2024-03-18T20:25:35.572056+0000","last_became_active":"2024-03-18T20:25:22.255063+0000","last_became_peered":"2024-03-18T20:25:22.255063+0000","last_unstale":"2024-03-18T20:25:35.572056+0000","last_undegraded":"2024-03-18T20:25:35.572056+0000","last_fullsized":"2024-03-18T20:25:35.572056+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:23.143345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.556731+0000","last_change":"2024-03-18T20:25:22.686185+0000","last_active":"2024-03-18T20:25:35.556731+0000","last_peered":"2024-03-18T20:25:35.556731+0000","last_clean":"2024-03-18T20:25:35.556731+0000","last_became_active":"2024-03-18T20:25:22.685939+0000","last_became_peered":"2024-03-18T20:25:22.685939+0000","last_unstale":"2024-03-18T20:25:35.556731+0000","last_undegraded":"2024-03-18T20:25:35.556731+0000","last_fullsized":"2024-03-18T20:25:35.556731+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:48:48.961872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.556939+0000","last_change":"2024-03-18T20:25:35.130591+0000","last_active":"2024-03-18T20:25:35.556939+0000","last_peered":"2024-03-18T20:25:35.556939+0000","last_clean":"2024-03-18T20:25:35.556939+0000","last_became_active":"2024-03-18T20:25:35.130354+0000","last_became_peered":"2024-03-18T20:25:35.130354+0000","last_unstale":"2024-03-18T20:25:35.556939+0000","last_undegraded":"2024-03-18T20:25:35.556939+0000","last_fullsized":"2024-03-18T20:25:35.556939+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:50.264082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187506+0000","last_change":"2024-03-18T20:25:35.576215+0000","last_active":"2024-03-18T20:25:38.187506+0000","last_peered":"2024-03-18T20:25:38.187506+0000","last_clean":"2024-03-18T20:25:38.187506+0000","last_became_active":"2024-03-18T20:25:35.575836+0000","last_became_peered":"2024-03-18T20:25:35.575836+0000","last_unstale":"2024-03-18T20:25:38.187506+0000","last_undegraded":"2024-03-18T20:25:38.187506+0000","last_fullsized":"2024-03-18T20:25:38.187506+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:53:59.063396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.10","version":"231'8","reported_seq":23,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.564701+0000","last_change":"2024-03-18T20:25:22.252976+0000","last_active":"2024-03-18T20:25:35.564701+0000","last_peered":"2024-03-18T20:25:35.564701+0000","last_clean":"2024-03-18T20:25:35.564701+0000","last_became_active":"2024-03-18T20:25:22.252528+0000","last_became_peered":"2024-03-18T20:25:22.252528+0000","last_unstale":"2024-03-18T20:25:35.564701+0000","last_undegraded":"2024-03-18T20:25:35.564701+0000","last_fullsized":"2024-03-18T20:25:35.564701+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:42:24.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.563715+0000","last_change":"2024-03-18T20:25:22.673559+0000","last_active":"2024-03-18T20:25:35.563715+0000","last_peered":"2024-03-18T20:25:35.563715+0000","last_clean":"2024-03-18T20:25:35.563715+0000","last_became_active":"2024-03-18T20:25:22.673267+0000","last_became_peered":"2024-03-18T20:25:22.673267+0000","last_unstale":"2024-03-18T20:25:35.563715+0000","last_undegraded":"2024-03-18T20:25:35.563715+0000","last_fullsized":"2024-03-18T20:25:35.563715+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:44:15.325153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.573396+0000","last_change":"2024-03-18T20:25:35.137760+0000","last_active":"2024-03-18T20:25:35.573396+0000","last_peered":"2024-03-18T20:25:35.573396+0000","last_clean":"2024-03-18T20:25:35.573396+0000","last_became_active":"2024-03-18T20:25:35.137438+0000","last_became_peered":"2024-03-18T20:25:35.137438+0000","last_unstale":"2024-03-18T20:25:35.573396+0000","last_undegraded":"2024-03-18T20:25:35.573396+0000","last_fullsized":"2024-03-18T20:25:35.573396+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:50.170851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:37.348365+0000","last_change":"2024-03-18T20:25:35.578102+0000","last_active":"2024-03-18T20:25:37.348365+0000","last_peered":"2024-03-18T20:25:37.348365+0000","last_clean":"2024-03-18T20:25:37.348365+0000","last_became_active":"2024-03-18T20:25:35.577822+0000","last_became_peered":"2024-03-18T20:25:35.577822+0000","last_unstale":"2024-03-18T20:25:37.348365+0000","last_undegraded":"2024-03-18T20:25:37.348365+0000","last_fullsized":"2024-03-18T20:25:37.348365+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:57:19.234034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560397+0000","last_change":"2024-03-18T20:25:22.252927+0000","last_active":"2024-03-18T20:25:35.560397+0000","last_peered":"2024-03-18T20:25:35.560397+0000","last_clean":"2024-03-18T20:25:35.560397+0000","last_became_active":"2024-03-18T20:25:22.252633+0000","last_became_peered":"2024-03-18T20:25:22.252633+0000","last_unstale":"2024-03-18T20:25:35.560397+0000","last_undegraded":"2024-03-18T20:25:35.560397+0000","last_fullsized":"2024-03-18T20:25:35.560397+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:32.345526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565256+0000","last_change":"2024-03-18T20:25:22.694369+0000","last_active":"2024-03-18T20:25:35.565256+0000","last_peered":"2024-03-18T20:25:35.565256+0000","last_clean":"2024-03-18T20:25:35.565256+0000","last_became_active":"2024-03-18T20:25:22.694114+0000","last_became_peered":"2024-03-18T20:25:22.694114+0000","last_unstale":"2024-03-18T20:25:35.565256+0000","last_undegraded":"2024-03-18T20:25:35.565256+0000","last_fullsized":"2024-03-18T20:25:35.565256+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:35:13.631181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.563412+0000","last_change":"2024-03-18T20:25:35.137498+0000","last_active":"2024-03-18T20:25:35.563412+0000","last_peered":"2024-03-18T20:25:35.563412+0000","last_clean":"2024-03-18T20:25:35.563412+0000","last_became_active":"2024-03-18T20:25:35.137248+0000","last_became_peered":"2024-03-18T20:25:35.137248+0000","last_unstale":"2024-03-18T20:25:35.563412+0000","last_undegraded":"2024-03-18T20:25:35.563412+0000","last_fullsized":"2024-03-18T20:25:35.563412+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:27.179302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:37.348277+0000","last_change":"2024-03-18T20:25:35.578336+0000","last_active":"2024-03-18T20:25:37.348277+0000","last_peered":"2024-03-18T20:25:37.348277+0000","last_clean":"2024-03-18T20:25:37.348277+0000","last_became_active":"2024-03-18T20:25:35.576678+0000","last_became_peered":"2024-03-18T20:25:35.576678+0000","last_unstale":"2024-03-18T20:25:37.348277+0000","last_undegraded":"2024-03-18T20:25:37.348277+0000","last_fullsized":"2024-03-18T20:25:37.348277+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:51:58.902819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.556510+0000","last_change":"2024-03-18T20:25:22.249078+0000","last_active":"2024-03-18T20:25:35.556510+0000","last_peered":"2024-03-18T20:25:35.556510+0000","last_clean":"2024-03-18T20:25:35.556510+0000","last_became_active":"2024-03-18T20:25:22.248588+0000","last_became_peered":"2024-03-18T20:25:22.248588+0000","last_unstale":"2024-03-18T20:25:35.556510+0000","last_undegraded":"2024-03-18T20:25:35.556510+0000","last_fullsized":"2024-03-18T20:25:35.556510+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:34:42.893253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.556073+0000","last_change":"2024-03-18T20:25:22.685090+0000","last_active":"2024-03-18T20:25:35.556073+0000","last_peered":"2024-03-18T20:25:35.556073+0000","last_clean":"2024-03-18T20:25:35.556073+0000","last_became_active":"2024-03-18T20:25:22.684824+0000","last_became_peered":"2024-03-18T20:25:22.684824+0000","last_unstale":"2024-03-18T20:25:35.556073+0000","last_undegraded":"2024-03-18T20:25:35.556073+0000","last_fullsized":"2024-03-18T20:25:35.556073+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:21.750701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.14","version":"235'6","reported_seq":17,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.587494+0000","last_change":"2024-03-18T20:25:35.143291+0000","last_active":"2024-03-18T20:25:35.587494+0000","last_peered":"2024-03-18T20:25:35.587494+0000","last_clean":"2024-03-18T20:25:35.587494+0000","last_became_active":"2024-03-18T20:25:35.143003+0000","last_became_peered":"2024-03-18T20:25:35.143003+0000","last_unstale":"2024-03-18T20:25:35.587494+0000","last_undegraded":"2024-03-18T20:25:35.587494+0000","last_fullsized":"2024-03-18T20:25:35.587494+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:03.410994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.570211+0000","last_change":"2024-03-18T20:25:35.577440+0000","last_active":"2024-03-18T20:25:36.570211+0000","last_peered":"2024-03-18T20:25:36.570211+0000","last_clean":"2024-03-18T20:25:36.570211+0000","last_became_active":"2024-03-18T20:25:35.577186+0000","last_became_peered":"2024-03-18T20:25:35.577186+0000","last_unstale":"2024-03-18T20:25:36.570211+0000","last_undegraded":"2024-03-18T20:25:36.570211+0000","last_fullsized":"2024-03-18T20:25:36.570211+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:39:05.449499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"231'2","reported_seq":17,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.568807+0000","last_change":"2024-03-18T20:25:22.254668+0000","last_active":"2024-03-18T20:25:35.568807+0000","last_peered":"2024-03-18T20:25:35.568807+0000","last_clean":"2024-03-18T20:25:35.568807+0000","last_became_active":"2024-03-18T20:25:22.254437+0000","last_became_peered":"2024-03-18T20:25:22.254437+0000","last_unstale":"2024-03-18T20:25:35.568807+0000","last_undegraded":"2024-03-18T20:25:35.568807+0000","last_fullsized":"2024-03-18T20:25:35.568807+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:32:02.691847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.568423+0000","last_change":"2024-03-18T20:25:22.691203+0000","last_active":"2024-03-18T20:25:35.568423+0000","last_peered":"2024-03-18T20:25:35.568423+0000","last_clean":"2024-03-18T20:25:35.568423+0000","last_became_active":"2024-03-18T20:25:22.690964+0000","last_became_peered":"2024-03-18T20:25:22.690964+0000","last_unstale":"2024-03-18T20:25:35.568423+0000","last_undegraded":"2024-03-18T20:25:35.568423+0000","last_fullsized":"2024-03-18T20:25:35.568423+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:19.068003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"235'3","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.579067+0000","last_change":"2024-03-18T20:25:35.134538+0000","last_active":"2024-03-18T20:25:35.579067+0000","last_peered":"2024-03-18T20:25:35.579067+0000","last_clean":"2024-03-18T20:25:35.579067+0000","last_became_active":"2024-03-18T20:25:35.133526+0000","last_became_peered":"2024-03-18T20:25:35.133526+0000","last_unstale":"2024-03-18T20:25:35.579067+0000","last_undegraded":"2024-03-18T20:25:35.579067+0000","last_fullsized":"2024-03-18T20:25:35.579067+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:02.016045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.189078+0000","last_change":"2024-03-18T20:25:35.573614+0000","last_active":"2024-03-18T20:25:38.189078+0000","last_peered":"2024-03-18T20:25:38.189078+0000","last_clean":"2024-03-18T20:25:38.189078+0000","last_became_active":"2024-03-18T20:25:35.573325+0000","last_became_peered":"2024-03-18T20:25:35.573325+0000","last_unstale":"2024-03-18T20:25:38.189078+0000","last_undegraded":"2024-03-18T20:25:38.189078+0000","last_fullsized":"2024-03-18T20:25:38.189078+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:32:07.096495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"232'8","reported_seq":23,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561908+0000","last_change":"2024-03-18T20:25:22.249429+0000","last_active":"2024-03-18T20:25:35.561908+0000","last_peered":"2024-03-18T20:25:35.561908+0000","last_clean":"2024-03-18T20:25:35.561908+0000","last_became_active":"2024-03-18T20:25:22.248842+0000","last_became_peered":"2024-03-18T20:25:22.248842+0000","last_unstale":"2024-03-18T20:25:35.561908+0000","last_undegraded":"2024-03-18T20:25:35.561908+0000","last_fullsized":"2024-03-18T20:25:35.561908+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:26.827223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2822,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565637+0000","last_change":"2024-03-18T20:25:22.696408+0000","last_active":"2024-03-18T20:25:35.565637+0000","last_peered":"2024-03-18T20:25:35.565637+0000","last_clean":"2024-03-18T20:25:35.565637+0000","last_became_active":"2024-03-18T20:25:22.696095+0000","last_became_peered":"2024-03-18T20:25:22.696095+0000","last_unstale":"2024-03-18T20:25:35.565637+0000","last_undegraded":"2024-03-18T20:25:35.565637+0000","last_fullsized":"2024-03-18T20:25:35.565637+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:19:19.377686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560621+0000","last_change":"2024-03-18T20:25:35.139401+0000","last_active":"2024-03-18T20:25:35.560621+0000","last_peered":"2024-03-18T20:25:35.560621+0000","last_clean":"2024-03-18T20:25:35.560621+0000","last_became_active":"2024-03-18T20:25:35.138928+0000","last_became_peered":"2024-03-18T20:25:35.138928+0000","last_unstale":"2024-03-18T20:25:35.560621+0000","last_undegraded":"2024-03-18T20:25:35.560621+0000","last_fullsized":"2024-03-18T20:25:35.560621+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:50.971852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.563311+0000","last_change":"2024-03-18T20:25:35.581002+0000","last_active":"2024-03-18T20:25:36.563311+0000","last_peered":"2024-03-18T20:25:36.563311+0000","last_clean":"2024-03-18T20:25:36.563311+0000","last_became_active":"2024-03-18T20:25:35.571822+0000","last_became_peered":"2024-03-18T20:25:35.571822+0000","last_unstale":"2024-03-18T20:25:36.563311+0000","last_undegraded":"2024-03-18T20:25:36.563311+0000","last_fullsized":"2024-03-18T20:25:36.563311+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:06:10.261463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.15","version":"231'3","reported_seq":18,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565332+0000","last_change":"2024-03-18T20:25:22.251534+0000","last_active":"2024-03-18T20:25:35.565332+0000","last_peered":"2024-03-18T20:25:35.565332+0000","last_clean":"2024-03-18T20:25:35.565332+0000","last_became_active":"2024-03-18T20:25:22.251275+0000","last_became_peered":"2024-03-18T20:25:22.251275+0000","last_unstale":"2024-03-18T20:25:35.565332+0000","last_undegraded":"2024-03-18T20:25:35.565332+0000","last_fullsized":"2024-03-18T20:25:35.565332+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:38:51.553676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560794+0000","last_change":"2024-03-18T20:25:22.682861+0000","last_active":"2024-03-18T20:25:35.560794+0000","last_peered":"2024-03-18T20:25:35.560794+0000","last_clean":"2024-03-18T20:25:35.560794+0000","last_became_active":"2024-03-18T20:25:22.682610+0000","last_became_peered":"2024-03-18T20:25:22.682610+0000","last_unstale":"2024-03-18T20:25:35.560794+0000","last_undegraded":"2024-03-18T20:25:35.560794+0000","last_fullsized":"2024-03-18T20:25:35.560794+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:51:43.095770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"235'2","reported_seq":13,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.580704+0000","last_change":"2024-03-18T20:25:35.144704+0000","last_active":"2024-03-18T20:25:35.580704+0000","last_peered":"2024-03-18T20:25:35.580704+0000","last_clean":"2024-03-18T20:25:35.580704+0000","last_became_active":"2024-03-18T20:25:35.144416+0000","last_became_peered":"2024-03-18T20:25:35.144416+0000","last_unstale":"2024-03-18T20:25:35.580704+0000","last_undegraded":"2024-03-18T20:25:35.580704+0000","last_fullsized":"2024-03-18T20:25:35.580704+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:32:42.191508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187376+0000","last_change":"2024-03-18T20:25:35.579165+0000","last_active":"2024-03-18T20:25:38.187376+0000","last_peered":"2024-03-18T20:25:38.187376+0000","last_clean":"2024-03-18T20:25:38.187376+0000","last_became_active":"2024-03-18T20:25:35.578906+0000","last_became_peered":"2024-03-18T20:25:35.578906+0000","last_unstale":"2024-03-18T20:25:38.187376+0000","last_undegraded":"2024-03-18T20:25:38.187376+0000","last_fullsized":"2024-03-18T20:25:38.187376+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:11:39.184436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560710+0000","last_change":"2024-03-18T20:25:22.252980+0000","last_active":"2024-03-18T20:25:35.560710+0000","last_peered":"2024-03-18T20:25:35.560710+0000","last_clean":"2024-03-18T20:25:35.560710+0000","last_became_active":"2024-03-18T20:25:22.252706+0000","last_became_peered":"2024-03-18T20:25:22.252706+0000","last_unstale":"2024-03-18T20:25:35.560710+0000","last_undegraded":"2024-03-18T20:25:35.560710+0000","last_fullsized":"2024-03-18T20:25:35.560710+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:45:13.948304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.556921+0000","last_change":"2024-03-18T20:25:22.684633+0000","last_active":"2024-03-18T20:25:35.556921+0000","last_peered":"2024-03-18T20:25:35.556921+0000","last_clean":"2024-03-18T20:25:35.556921+0000","last_became_active":"2024-03-18T20:25:22.684366+0000","last_became_peered":"2024-03-18T20:25:22.684366+0000","last_unstale":"2024-03-18T20:25:35.556921+0000","last_undegraded":"2024-03-18T20:25:35.556921+0000","last_fullsized":"2024-03-18T20:25:35.556921+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:15.299142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"235'8","reported_seq":19,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.583207+0000","last_change":"2024-03-18T20:25:35.145365+0000","last_active":"2024-03-18T20:25:35.583207+0000","last_peered":"2024-03-18T20:25:35.583207+0000","last_clean":"2024-03-18T20:25:35.583207+0000","last_became_active":"2024-03-18T20:25:35.145044+0000","last_became_peered":"2024-03-18T20:25:35.145044+0000","last_unstale":"2024-03-18T20:25:35.583207+0000","last_undegraded":"2024-03-18T20:25:35.583207+0000","last_fullsized":"2024-03-18T20:25:35.583207+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:07:50.170851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:37.348421+0000","last_change":"2024-03-18T20:25:35.571459+0000","last_active":"2024-03-18T20:25:37.348421+0000","last_peered":"2024-03-18T20:25:37.348421+0000","last_clean":"2024-03-18T20:25:37.348421+0000","last_became_active":"2024-03-18T20:25:35.571203+0000","last_became_peered":"2024-03-18T20:25:35.571203+0000","last_unstale":"2024-03-18T20:25:37.348421+0000","last_undegraded":"2024-03-18T20:25:37.348421+0000","last_fullsized":"2024-03-18T20:25:37.348421+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:06:03.885616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.570436+0000","last_change":"2024-03-18T20:25:22.251887+0000","last_active":"2024-03-18T20:25:35.570436+0000","last_peered":"2024-03-18T20:25:35.570436+0000","last_clean":"2024-03-18T20:25:35.570436+0000","last_became_active":"2024-03-18T20:25:22.251594+0000","last_became_peered":"2024-03-18T20:25:22.251594+0000","last_unstale":"2024-03-18T20:25:35.570436+0000","last_undegraded":"2024-03-18T20:25:35.570436+0000","last_fullsized":"2024-03-18T20:25:35.570436+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:48:47.954568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.572421+0000","last_change":"2024-03-18T20:25:22.680660+0000","last_active":"2024-03-18T20:25:35.572421+0000","last_peered":"2024-03-18T20:25:35.572421+0000","last_clean":"2024-03-18T20:25:35.572421+0000","last_became_active":"2024-03-18T20:25:22.680484+0000","last_became_peered":"2024-03-18T20:25:22.680484+0000","last_unstale":"2024-03-18T20:25:35.572421+0000","last_undegraded":"2024-03-18T20:25:35.572421+0000","last_fullsized":"2024-03-18T20:25:35.572421+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:48:21.194536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.562227+0000","last_change":"2024-03-18T20:25:35.139935+0000","last_active":"2024-03-18T20:25:35.562227+0000","last_peered":"2024-03-18T20:25:35.562227+0000","last_clean":"2024-03-18T20:25:35.562227+0000","last_became_active":"2024-03-18T20:25:35.139680+0000","last_became_peered":"2024-03-18T20:25:35.139680+0000","last_unstale":"2024-03-18T20:25:35.562227+0000","last_undegraded":"2024-03-18T20:25:35.562227+0000","last_fullsized":"2024-03-18T20:25:35.562227+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:20.466978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188202+0000","last_change":"2024-03-18T20:25:35.579679+0000","last_active":"2024-03-18T20:25:38.188202+0000","last_peered":"2024-03-18T20:25:38.188202+0000","last_clean":"2024-03-18T20:25:38.188202+0000","last_became_active":"2024-03-18T20:25:35.579425+0000","last_became_peered":"2024-03-18T20:25:35.579425+0000","last_unstale":"2024-03-18T20:25:38.188202+0000","last_undegraded":"2024-03-18T20:25:38.188202+0000","last_fullsized":"2024-03-18T20:25:38.188202+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:22:05.703105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.555546+0000","last_change":"2024-03-18T20:25:22.244727+0000","last_active":"2024-03-18T20:25:35.555546+0000","last_peered":"2024-03-18T20:25:35.555546+0000","last_clean":"2024-03-18T20:25:35.555546+0000","last_became_active":"2024-03-18T20:25:22.244417+0000","last_became_peered":"2024-03-18T20:25:22.244417+0000","last_unstale":"2024-03-18T20:25:35.555546+0000","last_undegraded":"2024-03-18T20:25:35.555546+0000","last_fullsized":"2024-03-18T20:25:35.555546+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:10.187509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565445+0000","last_change":"2024-03-18T20:25:22.681952+0000","last_active":"2024-03-18T20:25:35.565445+0000","last_peered":"2024-03-18T20:25:35.565445+0000","last_clean":"2024-03-18T20:25:35.565445+0000","last_became_active":"2024-03-18T20:25:22.681537+0000","last_became_peered":"2024-03-18T20:25:22.681537+0000","last_unstale":"2024-03-18T20:25:35.565445+0000","last_undegraded":"2024-03-18T20:25:35.565445+0000","last_fullsized":"2024-03-18T20:25:35.565445+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:32:22.923763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"235'1","reported_seq":12,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.569910+0000","last_change":"2024-03-18T20:25:35.137498+0000","last_active":"2024-03-18T20:25:35.569910+0000","last_peered":"2024-03-18T20:25:35.569910+0000","last_clean":"2024-03-18T20:25:35.569910+0000","last_became_active":"2024-03-18T20:25:35.137199+0000","last_became_peered":"2024-03-18T20:25:35.137199+0000","last_unstale":"2024-03-18T20:25:35.569910+0000","last_undegraded":"2024-03-18T20:25:35.569910+0000","last_fullsized":"2024-03-18T20:25:35.569910+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:39.087869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188667+0000","last_change":"2024-03-18T20:25:35.569437+0000","last_active":"2024-03-18T20:25:38.188667+0000","last_peered":"2024-03-18T20:25:38.188667+0000","last_clean":"2024-03-18T20:25:38.188667+0000","last_became_active":"2024-03-18T20:25:35.568989+0000","last_became_peered":"2024-03-18T20:25:35.568989+0000","last_unstale":"2024-03-18T20:25:38.188667+0000","last_undegraded":"2024-03-18T20:25:38.188667+0000","last_fullsized":"2024-03-18T20:25:38.188667+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:32:04.776255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560802+0000","last_change":"2024-03-18T20:25:22.251522+0000","last_active":"2024-03-18T20:25:35.560802+0000","last_peered":"2024-03-18T20:25:35.560802+0000","last_clean":"2024-03-18T20:25:35.560802+0000","last_became_active":"2024-03-18T20:25:22.251143+0000","last_became_peered":"2024-03-18T20:25:22.251143+0000","last_unstale":"2024-03-18T20:25:35.560802+0000","last_undegraded":"2024-03-18T20:25:35.560802+0000","last_fullsized":"2024-03-18T20:25:35.560802+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:58.960198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.573418+0000","last_change":"2024-03-18T20:25:22.691744+0000","last_active":"2024-03-18T20:25:35.573418+0000","last_peered":"2024-03-18T20:25:35.573418+0000","last_clean":"2024-03-18T20:25:35.573418+0000","last_became_active":"2024-03-18T20:25:22.691491+0000","last_became_peered":"2024-03-18T20:25:22.691491+0000","last_unstale":"2024-03-18T20:25:35.573418+0000","last_undegraded":"2024-03-18T20:25:35.573418+0000","last_fullsized":"2024-03-18T20:25:35.573418+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:04:09.670541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1f","version":"235'4","reported_seq":15,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.577517+0000","last_change":"2024-03-18T20:25:35.134695+0000","last_active":"2024-03-18T20:25:35.577517+0000","last_peered":"2024-03-18T20:25:35.577517+0000","last_clean":"2024-03-18T20:25:35.577517+0000","last_became_active":"2024-03-18T20:25:35.134333+0000","last_became_peered":"2024-03-18T20:25:35.134333+0000","last_unstale":"2024-03-18T20:25:35.577517+0000","last_undegraded":"2024-03-18T20:25:35.577517+0000","last_fullsized":"2024-03-18T20:25:35.577517+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:25.724222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188603+0000","last_change":"2024-03-18T20:25:35.581732+0000","last_active":"2024-03-18T20:25:38.188603+0000","last_peered":"2024-03-18T20:25:38.188603+0000","last_clean":"2024-03-18T20:25:38.188603+0000","last_became_active":"2024-03-18T20:25:35.581481+0000","last_became_peered":"2024-03-18T20:25:35.581481+0000","last_unstale":"2024-03-18T20:25:38.188603+0000","last_undegraded":"2024-03-18T20:25:38.188603+0000","last_fullsized":"2024-03-18T20:25:38.188603+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:09:26.131589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1b","version":"231'1","reported_seq":16,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561291+0000","last_change":"2024-03-18T20:25:22.251076+0000","last_active":"2024-03-18T20:25:35.561291+0000","last_peered":"2024-03-18T20:25:35.561291+0000","last_clean":"2024-03-18T20:25:35.561291+0000","last_became_active":"2024-03-18T20:25:22.250741+0000","last_became_peered":"2024-03-18T20:25:22.250741+0000","last_unstale":"2024-03-18T20:25:35.561291+0000","last_undegraded":"2024-03-18T20:25:35.561291+0000","last_fullsized":"2024-03-18T20:25:35.561291+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:07.448931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.573252+0000","last_change":"2024-03-18T20:25:22.680768+0000","last_active":"2024-03-18T20:25:35.573252+0000","last_peered":"2024-03-18T20:25:35.573252+0000","last_clean":"2024-03-18T20:25:35.573252+0000","last_became_active":"2024-03-18T20:25:22.680516+0000","last_became_peered":"2024-03-18T20:25:22.680516+0000","last_unstale":"2024-03-18T20:25:35.573252+0000","last_undegraded":"2024-03-18T20:25:35.573252+0000","last_fullsized":"2024-03-18T20:25:35.573252+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:19:19.377686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"235'2","reported_seq":13,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.584585+0000","last_change":"2024-03-18T20:25:35.143346+0000","last_active":"2024-03-18T20:25:35.584585+0000","last_peered":"2024-03-18T20:25:35.584585+0000","last_clean":"2024-03-18T20:25:35.584585+0000","last_became_active":"2024-03-18T20:25:35.143020+0000","last_became_peered":"2024-03-18T20:25:35.143020+0000","last_unstale":"2024-03-18T20:25:35.584585+0000","last_undegraded":"2024-03-18T20:25:35.584585+0000","last_fullsized":"2024-03-18T20:25:35.584585+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:45.664606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.564141+0000","last_change":"2024-03-18T20:25:35.561335+0000","last_active":"2024-03-18T20:25:36.564141+0000","last_peered":"2024-03-18T20:25:36.564141+0000","last_clean":"2024-03-18T20:25:36.564141+0000","last_became_active":"2024-03-18T20:25:35.561062+0000","last_became_peered":"2024-03-18T20:25:35.561062+0000","last_unstale":"2024-03-18T20:25:36.564141+0000","last_undegraded":"2024-03-18T20:25:36.564141+0000","last_fullsized":"2024-03-18T20:25:36.564141+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:48.374583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1c","version":"231'2","reported_seq":17,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.566246+0000","last_change":"2024-03-18T20:25:22.251018+0000","last_active":"2024-03-18T20:25:35.566246+0000","last_peered":"2024-03-18T20:25:35.566246+0000","last_clean":"2024-03-18T20:25:35.566246+0000","last_became_active":"2024-03-18T20:25:22.250730+0000","last_became_peered":"2024-03-18T20:25:22.250730+0000","last_unstale":"2024-03-18T20:25:35.566246+0000","last_undegraded":"2024-03-18T20:25:35.566246+0000","last_fullsized":"2024-03-18T20:25:35.566246+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:07.579514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565721+0000","last_change":"2024-03-18T20:25:22.690894+0000","last_active":"2024-03-18T20:25:35.565721+0000","last_peered":"2024-03-18T20:25:35.565721+0000","last_clean":"2024-03-18T20:25:35.565721+0000","last_became_active":"2024-03-18T20:25:22.688184+0000","last_became_peered":"2024-03-18T20:25:22.688184+0000","last_unstale":"2024-03-18T20:25:35.565721+0000","last_undegraded":"2024-03-18T20:25:35.565721+0000","last_fullsized":"2024-03-18T20:25:35.565721+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:48:59.967502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1a","version":"235'1","reported_seq":12,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.576155+0000","last_change":"2024-03-18T20:25:35.137495+0000","last_active":"2024-03-18T20:25:35.576155+0000","last_peered":"2024-03-18T20:25:35.576155+0000","last_clean":"2024-03-18T20:25:35.576155+0000","last_became_active":"2024-03-18T20:25:35.137194+0000","last_became_peered":"2024-03-18T20:25:35.137194+0000","last_unstale":"2024-03-18T20:25:35.576155+0000","last_undegraded":"2024-03-18T20:25:35.576155+0000","last_fullsized":"2024-03-18T20:25:35.576155+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:33.630861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.563180+0000","last_change":"2024-03-18T20:25:35.577560+0000","last_active":"2024-03-18T20:25:36.563180+0000","last_peered":"2024-03-18T20:25:36.563180+0000","last_clean":"2024-03-18T20:25:36.563180+0000","last_became_active":"2024-03-18T20:25:35.573230+0000","last_became_peered":"2024-03-18T20:25:35.573230+0000","last_unstale":"2024-03-18T20:25:36.563180+0000","last_undegraded":"2024-03-18T20:25:36.563180+0000","last_fullsized":"2024-03-18T20:25:36.563180+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:27:09.266573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"231'2","reported_seq":17,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.569297+0000","last_change":"2024-03-18T20:25:22.255583+0000","last_active":"2024-03-18T20:25:35.569297+0000","last_peered":"2024-03-18T20:25:35.569297+0000","last_clean":"2024-03-18T20:25:35.569297+0000","last_became_active":"2024-03-18T20:25:22.255243+0000","last_became_peered":"2024-03-18T20:25:22.255243+0000","last_unstale":"2024-03-18T20:25:35.569297+0000","last_undegraded":"2024-03-18T20:25:35.569297+0000","last_fullsized":"2024-03-18T20:25:35.569297+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:38.084389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.558583+0000","last_change":"2024-03-18T20:25:22.691295+0000","last_active":"2024-03-18T20:25:35.558583+0000","last_peered":"2024-03-18T20:25:35.558583+0000","last_clean":"2024-03-18T20:25:35.558583+0000","last_became_active":"2024-03-18T20:25:22.690890+0000","last_became_peered":"2024-03-18T20:25:22.690890+0000","last_unstale":"2024-03-18T20:25:35.558583+0000","last_undegraded":"2024-03-18T20:25:35.558583+0000","last_fullsized":"2024-03-18T20:25:35.558583+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:23:16.320044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1b","version":"235'1","reported_seq":12,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.576711+0000","last_change":"2024-03-18T20:25:35.140336+0000","last_active":"2024-03-18T20:25:35.576711+0000","last_peered":"2024-03-18T20:25:35.576711+0000","last_clean":"2024-03-18T20:25:35.576711+0000","last_became_active":"2024-03-18T20:25:35.140047+0000","last_became_peered":"2024-03-18T20:25:35.140047+0000","last_unstale":"2024-03-18T20:25:35.576711+0000","last_undegraded":"2024-03-18T20:25:35.576711+0000","last_fullsized":"2024-03-18T20:25:35.576711+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:19:02.788524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187397+0000","last_change":"2024-03-18T20:25:35.571761+0000","last_active":"2024-03-18T20:25:38.187397+0000","last_peered":"2024-03-18T20:25:38.187397+0000","last_clean":"2024-03-18T20:25:38.187397+0000","last_became_active":"2024-03-18T20:25:35.571491+0000","last_became_peered":"2024-03-18T20:25:35.571491+0000","last_unstale":"2024-03-18T20:25:38.187397+0000","last_undegraded":"2024-03-18T20:25:38.187397+0000","last_fullsized":"2024-03-18T20:25:38.187397+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:53:47.958993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1e","version":"231'1","reported_seq":16,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.572920+0000","last_change":"2024-03-18T20:25:22.251725+0000","last_active":"2024-03-18T20:25:35.572920+0000","last_peered":"2024-03-18T20:25:35.572920+0000","last_clean":"2024-03-18T20:25:35.572920+0000","last_became_active":"2024-03-18T20:25:22.251416+0000","last_became_peered":"2024-03-18T20:25:22.251416+0000","last_unstale":"2024-03-18T20:25:35.572920+0000","last_undegraded":"2024-03-18T20:25:35.572920+0000","last_fullsized":"2024-03-18T20:25:35.572920+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:26.827223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560339+0000","last_change":"2024-03-18T20:25:22.679158+0000","last_active":"2024-03-18T20:25:35.560339+0000","last_peered":"2024-03-18T20:25:35.560339+0000","last_clean":"2024-03-18T20:25:35.560339+0000","last_became_active":"2024-03-18T20:25:22.678889+0000","last_became_peered":"2024-03-18T20:25:22.678889+0000","last_unstale":"2024-03-18T20:25:35.560339+0000","last_undegraded":"2024-03-18T20:25:35.560339+0000","last_fullsized":"2024-03-18T20:25:35.560339+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:00.365377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.566521+0000","last_change":"2024-03-18T20:25:35.145539+0000","last_active":"2024-03-18T20:25:35.566521+0000","last_peered":"2024-03-18T20:25:35.566521+0000","last_clean":"2024-03-18T20:25:35.566521+0000","last_became_active":"2024-03-18T20:25:35.145266+0000","last_became_peered":"2024-03-18T20:25:35.145266+0000","last_unstale":"2024-03-18T20:25:35.566521+0000","last_undegraded":"2024-03-18T20:25:35.566521+0000","last_fullsized":"2024-03-18T20:25:35.566521+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:10.985415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.567301+0000","last_change":"2024-03-18T20:25:35.573235+0000","last_active":"2024-03-18T20:25:36.567301+0000","last_peered":"2024-03-18T20:25:36.567301+0000","last_clean":"2024-03-18T20:25:36.567301+0000","last_became_active":"2024-03-18T20:25:35.573002+0000","last_became_peered":"2024-03-18T20:25:35.573002+0000","last_unstale":"2024-03-18T20:25:36.567301+0000","last_undegraded":"2024-03-18T20:25:36.567301+0000","last_fullsized":"2024-03-18T20:25:36.567301+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:12.223522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1f","version":"231'4","reported_seq":19,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.576986+0000","last_change":"2024-03-18T20:25:22.251710+0000","last_active":"2024-03-18T20:25:35.576986+0000","last_peered":"2024-03-18T20:25:35.576986+0000","last_clean":"2024-03-18T20:25:35.576986+0000","last_became_active":"2024-03-18T20:25:22.251420+0000","last_became_peered":"2024-03-18T20:25:22.251420+0000","last_unstale":"2024-03-18T20:25:35.576986+0000","last_undegraded":"2024-03-18T20:25:35.576986+0000","last_fullsized":"2024-03-18T20:25:35.576986+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:21.217378+0000","last_clean_scrub_stamp":"2024-03-18T20:25:21.217378+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:00.435897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.566687+0000","last_change":"2024-03-18T20:25:22.682272+0000","last_active":"2024-03-18T20:25:35.566687+0000","last_peered":"2024-03-18T20:25:35.566687+0000","last_clean":"2024-03-18T20:25:35.566687+0000","last_became_active":"2024-03-18T20:25:22.681785+0000","last_became_peered":"2024-03-18T20:25:22.681785+0000","last_unstale":"2024-03-18T20:25:35.566687+0000","last_undegraded":"2024-03-18T20:25:35.566687+0000","last_fullsized":"2024-03-18T20:25:35.566687+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:53:46.174877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":11,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.576035+0000","last_change":"2024-03-18T20:25:35.140554+0000","last_active":"2024-03-18T20:25:35.576035+0000","last_peered":"2024-03-18T20:25:35.576035+0000","last_clean":"2024-03-18T20:25:35.576035+0000","last_became_active":"2024-03-18T20:25:35.140240+0000","last_became_peered":"2024-03-18T20:25:35.140240+0000","last_unstale":"2024-03-18T20:25:35.576035+0000","last_undegraded":"2024-03-18T20:25:35.576035+0000","last_fullsized":"2024-03-18T20:25:35.576035+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:34.104842+0000","last_clean_scrub_stamp":"2024-03-18T20:25:34.104842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:25.724222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:37.348255+0000","last_change":"2024-03-18T20:25:35.572462+0000","last_active":"2024-03-18T20:25:37.348255+0000","last_peered":"2024-03-18T20:25:37.348255+0000","last_clean":"2024-03-18T20:25:37.348255+0000","last_became_active":"2024-03-18T20:25:35.572188+0000","last_became_peered":"2024-03-18T20:25:35.572188+0000","last_unstale":"2024-03-18T20:25:37.348255+0000","last_undegraded":"2024-03-18T20:25:37.348255+0000","last_fullsized":"2024-03-18T20:25:37.348255+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:11:39.184436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561718+0000","last_change":"2024-03-18T20:25:22.679227+0000","last_active":"2024-03-18T20:25:35.561718+0000","last_peered":"2024-03-18T20:25:35.561718+0000","last_clean":"2024-03-18T20:25:35.561718+0000","last_became_active":"2024-03-18T20:25:22.678827+0000","last_became_peered":"2024-03-18T20:25:22.678827+0000","last_unstale":"2024-03-18T20:25:35.561718+0000","last_undegraded":"2024-03-18T20:25:35.561718+0000","last_fullsized":"2024-03-18T20:25:35.561718+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:05.394457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.923226+0000","last_change":"2024-03-18T20:25:35.571705+0000","last_active":"2024-03-18T20:25:36.923226+0000","last_peered":"2024-03-18T20:25:36.923226+0000","last_clean":"2024-03-18T20:25:36.923226+0000","last_became_active":"2024-03-18T20:25:35.571345+0000","last_became_peered":"2024-03-18T20:25:35.571345+0000","last_unstale":"2024-03-18T20:25:36.923226+0000","last_undegraded":"2024-03-18T20:25:36.923226+0000","last_fullsized":"2024-03-18T20:25:36.923226+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:45:17.428350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560689+0000","last_change":"2024-03-18T20:25:22.679640+0000","last_active":"2024-03-18T20:25:35.560689+0000","last_peered":"2024-03-18T20:25:35.560689+0000","last_clean":"2024-03-18T20:25:35.560689+0000","last_became_active":"2024-03-18T20:25:22.679408+0000","last_became_peered":"2024-03-18T20:25:22.679408+0000","last_unstale":"2024-03-18T20:25:35.560689+0000","last_undegraded":"2024-03-18T20:25:35.560689+0000","last_fullsized":"2024-03-18T20:25:35.560689+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:24:33.293594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187693+0000","last_change":"2024-03-18T20:25:35.579678+0000","last_active":"2024-03-18T20:25:38.187693+0000","last_peered":"2024-03-18T20:25:38.187693+0000","last_clean":"2024-03-18T20:25:38.187693+0000","last_became_active":"2024-03-18T20:25:35.579427+0000","last_became_peered":"2024-03-18T20:25:35.579427+0000","last_unstale":"2024-03-18T20:25:38.187693+0000","last_undegraded":"2024-03-18T20:25:38.187693+0000","last_fullsized":"2024-03-18T20:25:38.187693+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:19:11.085414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.572182+0000","last_change":"2024-03-18T20:25:22.685981+0000","last_active":"2024-03-18T20:25:35.572182+0000","last_peered":"2024-03-18T20:25:35.572182+0000","last_clean":"2024-03-18T20:25:35.572182+0000","last_became_active":"2024-03-18T20:25:22.685683+0000","last_became_peered":"2024-03-18T20:25:22.685683+0000","last_unstale":"2024-03-18T20:25:35.572182+0000","last_undegraded":"2024-03-18T20:25:35.572182+0000","last_fullsized":"2024-03-18T20:25:35.572182+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:36:57.458670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.923216+0000","last_change":"2024-03-18T20:25:35.574184+0000","last_active":"2024-03-18T20:25:36.923216+0000","last_peered":"2024-03-18T20:25:36.923216+0000","last_clean":"2024-03-18T20:25:36.923216+0000","last_became_active":"2024-03-18T20:25:35.572736+0000","last_became_peered":"2024-03-18T20:25:35.572736+0000","last_unstale":"2024-03-18T20:25:36.923216+0000","last_undegraded":"2024-03-18T20:25:36.923216+0000","last_fullsized":"2024-03-18T20:25:36.923216+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:46:02.367044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.567118+0000","last_change":"2024-03-18T20:25:22.680503+0000","last_active":"2024-03-18T20:25:35.567118+0000","last_peered":"2024-03-18T20:25:35.567118+0000","last_clean":"2024-03-18T20:25:35.567118+0000","last_became_active":"2024-03-18T20:25:22.680258+0000","last_became_peered":"2024-03-18T20:25:22.680258+0000","last_unstale":"2024-03-18T20:25:35.567118+0000","last_undegraded":"2024-03-18T20:25:35.567118+0000","last_fullsized":"2024-03-18T20:25:35.567118+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:49.478525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.566906+0000","last_change":"2024-03-18T20:25:35.577992+0000","last_active":"2024-03-18T20:25:36.566906+0000","last_peered":"2024-03-18T20:25:36.566906+0000","last_clean":"2024-03-18T20:25:36.566906+0000","last_became_active":"2024-03-18T20:25:35.577723+0000","last_became_peered":"2024-03-18T20:25:35.577723+0000","last_unstale":"2024-03-18T20:25:36.566906+0000","last_undegraded":"2024-03-18T20:25:36.566906+0000","last_fullsized":"2024-03-18T20:25:36.566906+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:46.241349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188995+0000","last_change":"2024-03-18T20:25:35.575324+0000","last_active":"2024-03-18T20:25:38.188995+0000","last_peered":"2024-03-18T20:25:38.188995+0000","last_clean":"2024-03-18T20:25:38.188995+0000","last_became_active":"2024-03-18T20:25:35.575124+0000","last_became_peered":"2024-03-18T20:25:35.575124+0000","last_unstale":"2024-03-18T20:25:38.188995+0000","last_undegraded":"2024-03-18T20:25:38.188995+0000","last_fullsized":"2024-03-18T20:25:38.188995+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:05:32.489599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.566309+0000","last_change":"2024-03-18T20:25:22.688098+0000","last_active":"2024-03-18T20:25:35.566309+0000","last_peered":"2024-03-18T20:25:35.566309+0000","last_clean":"2024-03-18T20:25:35.566309+0000","last_became_active":"2024-03-18T20:25:22.687830+0000","last_became_peered":"2024-03-18T20:25:22.687830+0000","last_unstale":"2024-03-18T20:25:35.566309+0000","last_undegraded":"2024-03-18T20:25:35.566309+0000","last_fullsized":"2024-03-18T20:25:35.566309+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:08.456234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188474+0000","last_change":"2024-03-18T20:25:35.583496+0000","last_active":"2024-03-18T20:25:38.188474+0000","last_peered":"2024-03-18T20:25:38.188474+0000","last_clean":"2024-03-18T20:25:38.188474+0000","last_became_active":"2024-03-18T20:25:35.583147+0000","last_became_peered":"2024-03-18T20:25:35.583147+0000","last_unstale":"2024-03-18T20:25:38.188474+0000","last_undegraded":"2024-03-18T20:25:38.188474+0000","last_fullsized":"2024-03-18T20:25:38.188474+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:14:02.602355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.555994+0000","last_change":"2024-03-18T20:25:22.689262+0000","last_active":"2024-03-18T20:25:35.555994+0000","last_peered":"2024-03-18T20:25:35.555994+0000","last_clean":"2024-03-18T20:25:35.555994+0000","last_became_active":"2024-03-18T20:25:22.688913+0000","last_became_peered":"2024-03-18T20:25:22.688913+0000","last_unstale":"2024-03-18T20:25:35.555994+0000","last_undegraded":"2024-03-18T20:25:35.555994+0000","last_fullsized":"2024-03-18T20:25:35.555994+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:15:50.135885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.572556+0000","last_change":"2024-03-18T20:25:22.687342+0000","last_active":"2024-03-18T20:25:35.572556+0000","last_peered":"2024-03-18T20:25:35.572556+0000","last_clean":"2024-03-18T20:25:35.572556+0000","last_became_active":"2024-03-18T20:25:22.687092+0000","last_became_peered":"2024-03-18T20:25:22.687092+0000","last_unstale":"2024-03-18T20:25:35.572556+0000","last_undegraded":"2024-03-18T20:25:35.572556+0000","last_fullsized":"2024-03-18T20:25:35.572556+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:00:33.383893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.564160+0000","last_change":"2024-03-18T20:25:35.575591+0000","last_active":"2024-03-18T20:25:36.564160+0000","last_peered":"2024-03-18T20:25:36.564160+0000","last_clean":"2024-03-18T20:25:36.564160+0000","last_became_active":"2024-03-18T20:25:35.575099+0000","last_became_peered":"2024-03-18T20:25:35.575099+0000","last_unstale":"2024-03-18T20:25:36.564160+0000","last_undegraded":"2024-03-18T20:25:36.564160+0000","last_fullsized":"2024-03-18T20:25:36.564160+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:54:10.233888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.570166+0000","last_change":"2024-03-18T20:25:35.572663+0000","last_active":"2024-03-18T20:25:36.570166+0000","last_peered":"2024-03-18T20:25:36.570166+0000","last_clean":"2024-03-18T20:25:36.570166+0000","last_became_active":"2024-03-18T20:25:35.572150+0000","last_became_peered":"2024-03-18T20:25:35.572150+0000","last_unstale":"2024-03-18T20:25:36.570166+0000","last_undegraded":"2024-03-18T20:25:36.570166+0000","last_fullsized":"2024-03-18T20:25:36.570166+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:04.419353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565349+0000","last_change":"2024-03-18T20:25:22.683777+0000","last_active":"2024-03-18T20:25:35.565349+0000","last_peered":"2024-03-18T20:25:35.565349+0000","last_clean":"2024-03-18T20:25:35.565349+0000","last_became_active":"2024-03-18T20:25:22.683484+0000","last_became_peered":"2024-03-18T20:25:22.683484+0000","last_unstale":"2024-03-18T20:25:35.565349+0000","last_undegraded":"2024-03-18T20:25:35.565349+0000","last_fullsized":"2024-03-18T20:25:35.565349+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:03:45.332668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:37.348061+0000","last_change":"2024-03-18T20:25:35.572089+0000","last_active":"2024-03-18T20:25:37.348061+0000","last_peered":"2024-03-18T20:25:37.348061+0000","last_clean":"2024-03-18T20:25:37.348061+0000","last_became_active":"2024-03-18T20:25:35.571834+0000","last_became_peered":"2024-03-18T20:25:35.571834+0000","last_unstale":"2024-03-18T20:25:37.348061+0000","last_undegraded":"2024-03-18T20:25:37.348061+0000","last_fullsized":"2024-03-18T20:25:37.348061+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:24:54.574523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.564608+0000","last_change":"2024-03-18T20:25:22.684030+0000","last_active":"2024-03-18T20:25:35.564608+0000","last_peered":"2024-03-18T20:25:35.564608+0000","last_clean":"2024-03-18T20:25:35.564608+0000","last_became_active":"2024-03-18T20:25:22.683767+0000","last_became_peered":"2024-03-18T20:25:22.683767+0000","last_unstale":"2024-03-18T20:25:35.564608+0000","last_undegraded":"2024-03-18T20:25:35.564608+0000","last_fullsized":"2024-03-18T20:25:35.564608+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:30.204046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188263+0000","last_change":"2024-03-18T20:25:35.569309+0000","last_active":"2024-03-18T20:25:38.188263+0000","last_peered":"2024-03-18T20:25:38.188263+0000","last_clean":"2024-03-18T20:25:38.188263+0000","last_became_active":"2024-03-18T20:25:35.569035+0000","last_became_peered":"2024-03-18T20:25:35.569035+0000","last_unstale":"2024-03-18T20:25:38.188263+0000","last_undegraded":"2024-03-18T20:25:38.188263+0000","last_fullsized":"2024-03-18T20:25:38.188263+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:35:59.383051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.569947+0000","last_change":"2024-03-18T20:25:22.680418+0000","last_active":"2024-03-18T20:25:35.569947+0000","last_peered":"2024-03-18T20:25:35.569947+0000","last_clean":"2024-03-18T20:25:35.569947+0000","last_became_active":"2024-03-18T20:25:22.680149+0000","last_became_peered":"2024-03-18T20:25:22.680149+0000","last_unstale":"2024-03-18T20:25:35.569947+0000","last_undegraded":"2024-03-18T20:25:35.569947+0000","last_fullsized":"2024-03-18T20:25:35.569947+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:51:14.804124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187541+0000","last_change":"2024-03-18T20:25:35.568991+0000","last_active":"2024-03-18T20:25:38.187541+0000","last_peered":"2024-03-18T20:25:38.187541+0000","last_clean":"2024-03-18T20:25:38.187541+0000","last_became_active":"2024-03-18T20:25:35.568685+0000","last_became_peered":"2024-03-18T20:25:35.568685+0000","last_unstale":"2024-03-18T20:25:38.187541+0000","last_undegraded":"2024-03-18T20:25:38.187541+0000","last_fullsized":"2024-03-18T20:25:38.187541+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:07.014178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.558788+0000","last_change":"2024-03-18T20:25:22.691577+0000","last_active":"2024-03-18T20:25:35.558788+0000","last_peered":"2024-03-18T20:25:35.558788+0000","last_clean":"2024-03-18T20:25:35.558788+0000","last_became_active":"2024-03-18T20:25:22.691250+0000","last_became_peered":"2024-03-18T20:25:22.691250+0000","last_unstale":"2024-03-18T20:25:35.558788+0000","last_undegraded":"2024-03-18T20:25:35.558788+0000","last_fullsized":"2024-03-18T20:25:35.558788+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:55:21.946848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.566991+0000","last_change":"2024-03-18T20:25:35.574717+0000","last_active":"2024-03-18T20:25:36.566991+0000","last_peered":"2024-03-18T20:25:36.566991+0000","last_clean":"2024-03-18T20:25:36.566991+0000","last_became_active":"2024-03-18T20:25:35.574431+0000","last_became_peered":"2024-03-18T20:25:35.574431+0000","last_unstale":"2024-03-18T20:25:36.566991+0000","last_undegraded":"2024-03-18T20:25:36.566991+0000","last_fullsized":"2024-03-18T20:25:36.566991+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:54:49.519982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561716+0000","last_change":"2024-03-18T20:25:22.693670+0000","last_active":"2024-03-18T20:25:35.561716+0000","last_peered":"2024-03-18T20:25:35.561716+0000","last_clean":"2024-03-18T20:25:35.561716+0000","last_became_active":"2024-03-18T20:25:22.693178+0000","last_became_peered":"2024-03-18T20:25:22.693178+0000","last_unstale":"2024-03-18T20:25:35.561716+0000","last_undegraded":"2024-03-18T20:25:35.561716+0000","last_fullsized":"2024-03-18T20:25:35.561716+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:13.992732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188941+0000","last_change":"2024-03-18T20:25:35.575271+0000","last_active":"2024-03-18T20:25:38.188941+0000","last_peered":"2024-03-18T20:25:38.188941+0000","last_clean":"2024-03-18T20:25:38.188941+0000","last_became_active":"2024-03-18T20:25:35.573025+0000","last_became_peered":"2024-03-18T20:25:35.573025+0000","last_unstale":"2024-03-18T20:25:38.188941+0000","last_undegraded":"2024-03-18T20:25:38.188941+0000","last_fullsized":"2024-03-18T20:25:38.188941+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:05.446446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.559446+0000","last_change":"2024-03-18T20:25:22.693769+0000","last_active":"2024-03-18T20:25:35.559446+0000","last_peered":"2024-03-18T20:25:35.559446+0000","last_clean":"2024-03-18T20:25:35.559446+0000","last_became_active":"2024-03-18T20:25:22.691936+0000","last_became_peered":"2024-03-18T20:25:22.691936+0000","last_unstale":"2024-03-18T20:25:35.559446+0000","last_undegraded":"2024-03-18T20:25:35.559446+0000","last_fullsized":"2024-03-18T20:25:35.559446+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:29:49.401587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.564058+0000","last_change":"2024-03-18T20:25:35.570394+0000","last_active":"2024-03-18T20:25:36.564058+0000","last_peered":"2024-03-18T20:25:36.564058+0000","last_clean":"2024-03-18T20:25:36.564058+0000","last_became_active":"2024-03-18T20:25:35.570154+0000","last_became_peered":"2024-03-18T20:25:35.570154+0000","last_unstale":"2024-03-18T20:25:36.564058+0000","last_undegraded":"2024-03-18T20:25:36.564058+0000","last_fullsized":"2024-03-18T20:25:36.564058+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:34.639220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561470+0000","last_change":"2024-03-18T20:25:22.692857+0000","last_active":"2024-03-18T20:25:35.561470+0000","last_peered":"2024-03-18T20:25:35.561470+0000","last_clean":"2024-03-18T20:25:35.561470+0000","last_became_active":"2024-03-18T20:25:22.692615+0000","last_became_peered":"2024-03-18T20:25:22.692615+0000","last_unstale":"2024-03-18T20:25:35.561470+0000","last_undegraded":"2024-03-18T20:25:35.561470+0000","last_fullsized":"2024-03-18T20:25:35.561470+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:21:45.938034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.923657+0000","last_change":"2024-03-18T20:25:35.570607+0000","last_active":"2024-03-18T20:25:36.923657+0000","last_peered":"2024-03-18T20:25:36.923657+0000","last_clean":"2024-03-18T20:25:36.923657+0000","last_became_active":"2024-03-18T20:25:35.570102+0000","last_became_peered":"2024-03-18T20:25:35.570102+0000","last_unstale":"2024-03-18T20:25:36.923657+0000","last_undegraded":"2024-03-18T20:25:36.923657+0000","last_fullsized":"2024-03-18T20:25:36.923657+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:11.357437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565175+0000","last_change":"2024-03-18T20:25:22.683388+0000","last_active":"2024-03-18T20:25:35.565175+0000","last_peered":"2024-03-18T20:25:35.565175+0000","last_clean":"2024-03-18T20:25:35.565175+0000","last_became_active":"2024-03-18T20:25:22.683137+0000","last_became_peered":"2024-03-18T20:25:22.683137+0000","last_unstale":"2024-03-18T20:25:35.565175+0000","last_undegraded":"2024-03-18T20:25:35.565175+0000","last_fullsized":"2024-03-18T20:25:35.565175+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:45:02.506299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188256+0000","last_change":"2024-03-18T20:25:35.579506+0000","last_active":"2024-03-18T20:25:38.188256+0000","last_peered":"2024-03-18T20:25:38.188256+0000","last_clean":"2024-03-18T20:25:38.188256+0000","last_became_active":"2024-03-18T20:25:35.578971+0000","last_became_peered":"2024-03-18T20:25:35.578971+0000","last_unstale":"2024-03-18T20:25:38.188256+0000","last_undegraded":"2024-03-18T20:25:38.188256+0000","last_fullsized":"2024-03-18T20:25:38.188256+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:48.195554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565266+0000","last_change":"2024-03-18T20:25:22.696148+0000","last_active":"2024-03-18T20:25:35.565266+0000","last_peered":"2024-03-18T20:25:35.565266+0000","last_clean":"2024-03-18T20:25:35.565266+0000","last_became_active":"2024-03-18T20:25:22.695928+0000","last_became_peered":"2024-03-18T20:25:22.695928+0000","last_unstale":"2024-03-18T20:25:35.565266+0000","last_undegraded":"2024-03-18T20:25:35.565266+0000","last_fullsized":"2024-03-18T20:25:35.565266+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:57.565171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.563721+0000","last_change":"2024-03-18T20:25:35.570123+0000","last_active":"2024-03-18T20:25:36.563721+0000","last_peered":"2024-03-18T20:25:36.563721+0000","last_clean":"2024-03-18T20:25:36.563721+0000","last_became_active":"2024-03-18T20:25:35.569711+0000","last_became_peered":"2024-03-18T20:25:35.569711+0000","last_unstale":"2024-03-18T20:25:36.563721+0000","last_undegraded":"2024-03-18T20:25:36.563721+0000","last_fullsized":"2024-03-18T20:25:36.563721+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:28.187661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.555465+0000","last_change":"2024-03-18T20:25:22.681389+0000","last_active":"2024-03-18T20:25:35.555465+0000","last_peered":"2024-03-18T20:25:35.555465+0000","last_clean":"2024-03-18T20:25:35.555465+0000","last_became_active":"2024-03-18T20:25:22.681146+0000","last_became_peered":"2024-03-18T20:25:22.681146+0000","last_unstale":"2024-03-18T20:25:35.555465+0000","last_undegraded":"2024-03-18T20:25:35.555465+0000","last_fullsized":"2024-03-18T20:25:35.555465+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:05:57.372944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.923398+0000","last_change":"2024-03-18T20:25:35.580268+0000","last_active":"2024-03-18T20:25:36.923398+0000","last_peered":"2024-03-18T20:25:36.923398+0000","last_clean":"2024-03-18T20:25:36.923398+0000","last_became_active":"2024-03-18T20:25:35.580007+0000","last_became_peered":"2024-03-18T20:25:35.580007+0000","last_unstale":"2024-03-18T20:25:36.923398+0000","last_undegraded":"2024-03-18T20:25:36.923398+0000","last_fullsized":"2024-03-18T20:25:36.923398+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:32:35.812282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.576463+0000","last_change":"2024-03-18T20:25:22.692497+0000","last_active":"2024-03-18T20:25:35.576463+0000","last_peered":"2024-03-18T20:25:35.576463+0000","last_clean":"2024-03-18T20:25:35.576463+0000","last_became_active":"2024-03-18T20:25:22.692192+0000","last_became_peered":"2024-03-18T20:25:22.692192+0000","last_unstale":"2024-03-18T20:25:35.576463+0000","last_undegraded":"2024-03-18T20:25:35.576463+0000","last_fullsized":"2024-03-18T20:25:35.576463+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:03:47.775258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.923002+0000","last_change":"2024-03-18T20:25:35.575256+0000","last_active":"2024-03-18T20:25:36.923002+0000","last_peered":"2024-03-18T20:25:36.923002+0000","last_clean":"2024-03-18T20:25:36.923002+0000","last_became_active":"2024-03-18T20:25:35.572998+0000","last_became_peered":"2024-03-18T20:25:35.572998+0000","last_unstale":"2024-03-18T20:25:36.923002+0000","last_undegraded":"2024-03-18T20:25:36.923002+0000","last_fullsized":"2024-03-18T20:25:36.923002+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:27:09.266573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.559312+0000","last_change":"2024-03-18T20:25:22.690898+0000","last_active":"2024-03-18T20:25:35.559312+0000","last_peered":"2024-03-18T20:25:35.559312+0000","last_clean":"2024-03-18T20:25:35.559312+0000","last_became_active":"2024-03-18T20:25:22.690461+0000","last_became_peered":"2024-03-18T20:25:22.690461+0000","last_unstale":"2024-03-18T20:25:35.559312+0000","last_undegraded":"2024-03-18T20:25:35.559312+0000","last_fullsized":"2024-03-18T20:25:35.559312+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:15.208122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:37.348030+0000","last_change":"2024-03-18T20:25:35.562389+0000","last_active":"2024-03-18T20:25:37.348030+0000","last_peered":"2024-03-18T20:25:37.348030+0000","last_clean":"2024-03-18T20:25:37.348030+0000","last_became_active":"2024-03-18T20:25:35.562118+0000","last_became_peered":"2024-03-18T20:25:35.562118+0000","last_unstale":"2024-03-18T20:25:37.348030+0000","last_undegraded":"2024-03-18T20:25:37.348030+0000","last_fullsized":"2024-03-18T20:25:37.348030+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:04.584677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.561196+0000","last_change":"2024-03-18T20:25:22.685172+0000","last_active":"2024-03-18T20:25:35.561196+0000","last_peered":"2024-03-18T20:25:35.561196+0000","last_clean":"2024-03-18T20:25:35.561196+0000","last_became_active":"2024-03-18T20:25:22.684855+0000","last_became_peered":"2024-03-18T20:25:22.684855+0000","last_unstale":"2024-03-18T20:25:35.561196+0000","last_undegraded":"2024-03-18T20:25:35.561196+0000","last_fullsized":"2024-03-18T20:25:35.561196+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:00.030804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188874+0000","last_change":"2024-03-18T20:25:35.582068+0000","last_active":"2024-03-18T20:25:38.188874+0000","last_peered":"2024-03-18T20:25:38.188874+0000","last_clean":"2024-03-18T20:25:38.188874+0000","last_became_active":"2024-03-18T20:25:35.581688+0000","last_became_peered":"2024-03-18T20:25:35.581688+0000","last_unstale":"2024-03-18T20:25:38.188874+0000","last_undegraded":"2024-03-18T20:25:38.188874+0000","last_fullsized":"2024-03-18T20:25:38.188874+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:02:05.383430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.566253+0000","last_change":"2024-03-18T20:25:22.693640+0000","last_active":"2024-03-18T20:25:35.566253+0000","last_peered":"2024-03-18T20:25:35.566253+0000","last_clean":"2024-03-18T20:25:35.566253+0000","last_became_active":"2024-03-18T20:25:22.693403+0000","last_became_peered":"2024-03-18T20:25:22.693403+0000","last_unstale":"2024-03-18T20:25:35.566253+0000","last_undegraded":"2024-03-18T20:25:35.566253+0000","last_fullsized":"2024-03-18T20:25:35.566253+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:35:58.151146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.567667+0000","last_change":"2024-03-18T20:25:35.572108+0000","last_active":"2024-03-18T20:25:36.567667+0000","last_peered":"2024-03-18T20:25:36.567667+0000","last_clean":"2024-03-18T20:25:36.567667+0000","last_became_active":"2024-03-18T20:25:35.571863+0000","last_became_peered":"2024-03-18T20:25:35.571863+0000","last_unstale":"2024-03-18T20:25:36.567667+0000","last_undegraded":"2024-03-18T20:25:36.567667+0000","last_fullsized":"2024-03-18T20:25:36.567667+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:04.419353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.555916+0000","last_change":"2024-03-18T20:25:22.693546+0000","last_active":"2024-03-18T20:25:35.555916+0000","last_peered":"2024-03-18T20:25:35.555916+0000","last_clean":"2024-03-18T20:25:35.555916+0000","last_became_active":"2024-03-18T20:25:22.693267+0000","last_became_peered":"2024-03-18T20:25:22.693267+0000","last_unstale":"2024-03-18T20:25:35.555916+0000","last_undegraded":"2024-03-18T20:25:35.555916+0000","last_fullsized":"2024-03-18T20:25:35.555916+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:35.486064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.563988+0000","last_change":"2024-03-18T20:25:35.575392+0000","last_active":"2024-03-18T20:25:36.563988+0000","last_peered":"2024-03-18T20:25:36.563988+0000","last_clean":"2024-03-18T20:25:36.563988+0000","last_became_active":"2024-03-18T20:25:35.574664+0000","last_became_peered":"2024-03-18T20:25:35.574664+0000","last_unstale":"2024-03-18T20:25:36.563988+0000","last_undegraded":"2024-03-18T20:25:36.563988+0000","last_fullsized":"2024-03-18T20:25:36.563988+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:18:12.445322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560194+0000","last_change":"2024-03-18T20:25:22.693152+0000","last_active":"2024-03-18T20:25:35.560194+0000","last_peered":"2024-03-18T20:25:35.560194+0000","last_clean":"2024-03-18T20:25:35.560194+0000","last_became_active":"2024-03-18T20:25:22.686521+0000","last_became_peered":"2024-03-18T20:25:22.686521+0000","last_unstale":"2024-03-18T20:25:35.560194+0000","last_undegraded":"2024-03-18T20:25:35.560194+0000","last_fullsized":"2024-03-18T20:25:35.560194+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:40:18.302530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.569992+0000","last_change":"2024-03-18T20:25:35.572687+0000","last_active":"2024-03-18T20:25:36.569992+0000","last_peered":"2024-03-18T20:25:36.569992+0000","last_clean":"2024-03-18T20:25:36.569992+0000","last_became_active":"2024-03-18T20:25:35.572459+0000","last_became_peered":"2024-03-18T20:25:35.572459+0000","last_unstale":"2024-03-18T20:25:36.569992+0000","last_undegraded":"2024-03-18T20:25:36.569992+0000","last_fullsized":"2024-03-18T20:25:36.569992+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:46.672966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.565187+0000","last_change":"2024-03-18T20:25:22.681784+0000","last_active":"2024-03-18T20:25:35.565187+0000","last_peered":"2024-03-18T20:25:35.565187+0000","last_clean":"2024-03-18T20:25:35.565187+0000","last_became_active":"2024-03-18T20:25:22.681198+0000","last_became_peered":"2024-03-18T20:25:22.681198+0000","last_unstale":"2024-03-18T20:25:35.565187+0000","last_undegraded":"2024-03-18T20:25:35.565187+0000","last_fullsized":"2024-03-18T20:25:35.565187+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:44:35.307035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.923156+0000","last_change":"2024-03-18T20:25:35.580547+0000","last_active":"2024-03-18T20:25:36.923156+0000","last_peered":"2024-03-18T20:25:36.923156+0000","last_clean":"2024-03-18T20:25:36.923156+0000","last_became_active":"2024-03-18T20:25:35.579377+0000","last_became_peered":"2024-03-18T20:25:35.579377+0000","last_unstale":"2024-03-18T20:25:36.923156+0000","last_undegraded":"2024-03-18T20:25:36.923156+0000","last_fullsized":"2024-03-18T20:25:36.923156+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:53:10.453690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.556638+0000","last_change":"2024-03-18T20:25:22.685766+0000","last_active":"2024-03-18T20:25:35.556638+0000","last_peered":"2024-03-18T20:25:35.556638+0000","last_clean":"2024-03-18T20:25:35.556638+0000","last_became_active":"2024-03-18T20:25:22.683799+0000","last_became_peered":"2024-03-18T20:25:22.683799+0000","last_unstale":"2024-03-18T20:25:35.556638+0000","last_undegraded":"2024-03-18T20:25:35.556638+0000","last_fullsized":"2024-03-18T20:25:35.556638+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:42:25.581359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.187895+0000","last_change":"2024-03-18T20:25:35.569195+0000","last_active":"2024-03-18T20:25:38.187895+0000","last_peered":"2024-03-18T20:25:38.187895+0000","last_clean":"2024-03-18T20:25:38.187895+0000","last_became_active":"2024-03-18T20:25:35.568915+0000","last_became_peered":"2024-03-18T20:25:35.568915+0000","last_unstale":"2024-03-18T20:25:38.187895+0000","last_undegraded":"2024-03-18T20:25:38.187895+0000","last_fullsized":"2024-03-18T20:25:38.187895+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:03:58.221187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.567994+0000","last_change":"2024-03-18T20:25:22.694269+0000","last_active":"2024-03-18T20:25:35.567994+0000","last_peered":"2024-03-18T20:25:35.567994+0000","last_clean":"2024-03-18T20:25:35.567994+0000","last_became_active":"2024-03-18T20:25:22.694014+0000","last_became_peered":"2024-03-18T20:25:22.694014+0000","last_unstale":"2024-03-18T20:25:35.567994+0000","last_undegraded":"2024-03-18T20:25:35.567994+0000","last_fullsized":"2024-03-18T20:25:35.567994+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:43:52.409254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.923125+0000","last_change":"2024-03-18T20:25:35.581607+0000","last_active":"2024-03-18T20:25:36.923125+0000","last_peered":"2024-03-18T20:25:36.923125+0000","last_clean":"2024-03-18T20:25:36.923125+0000","last_became_active":"2024-03-18T20:25:35.581338+0000","last_became_peered":"2024-03-18T20:25:35.581338+0000","last_unstale":"2024-03-18T20:25:36.923125+0000","last_undegraded":"2024-03-18T20:25:36.923125+0000","last_fullsized":"2024-03-18T20:25:36.923125+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:52.621876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.560392+0000","last_change":"2024-03-18T20:25:22.681736+0000","last_active":"2024-03-18T20:25:35.560392+0000","last_peered":"2024-03-18T20:25:35.560392+0000","last_clean":"2024-03-18T20:25:35.560392+0000","last_became_active":"2024-03-18T20:25:22.679120+0000","last_became_peered":"2024-03-18T20:25:22.679120+0000","last_unstale":"2024-03-18T20:25:35.560392+0000","last_undegraded":"2024-03-18T20:25:35.560392+0000","last_fullsized":"2024-03-18T20:25:35.560392+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:00.365377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188013+0000","last_change":"2024-03-18T20:25:35.580208+0000","last_active":"2024-03-18T20:25:38.188013+0000","last_peered":"2024-03-18T20:25:38.188013+0000","last_clean":"2024-03-18T20:25:38.188013+0000","last_became_active":"2024-03-18T20:25:35.579486+0000","last_became_peered":"2024-03-18T20:25:35.579486+0000","last_unstale":"2024-03-18T20:25:38.188013+0000","last_undegraded":"2024-03-18T20:25:38.188013+0000","last_fullsized":"2024-03-18T20:25:38.188013+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:21:14.140052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.568868+0000","last_change":"2024-03-18T20:25:22.695525+0000","last_active":"2024-03-18T20:25:35.568868+0000","last_peered":"2024-03-18T20:25:35.568868+0000","last_clean":"2024-03-18T20:25:35.568868+0000","last_became_active":"2024-03-18T20:25:22.694079+0000","last_became_peered":"2024-03-18T20:25:22.694079+0000","last_unstale":"2024-03-18T20:25:35.568868+0000","last_undegraded":"2024-03-18T20:25:35.568868+0000","last_fullsized":"2024-03-18T20:25:35.568868+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:45:14.955608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188412+0000","last_change":"2024-03-18T20:25:35.569851+0000","last_active":"2024-03-18T20:25:38.188412+0000","last_peered":"2024-03-18T20:25:38.188412+0000","last_clean":"2024-03-18T20:25:38.188412+0000","last_became_active":"2024-03-18T20:25:35.569597+0000","last_became_peered":"2024-03-18T20:25:35.569597+0000","last_unstale":"2024-03-18T20:25:38.188412+0000","last_undegraded":"2024-03-18T20:25:38.188412+0000","last_fullsized":"2024-03-18T20:25:38.188412+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:51:27.692643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.562303+0000","last_change":"2024-03-18T20:25:22.683021+0000","last_active":"2024-03-18T20:25:35.562303+0000","last_peered":"2024-03-18T20:25:35.562303+0000","last_clean":"2024-03-18T20:25:35.562303+0000","last_became_active":"2024-03-18T20:25:22.682722+0000","last_became_peered":"2024-03-18T20:25:22.682722+0000","last_unstale":"2024-03-18T20:25:35.562303+0000","last_undegraded":"2024-03-18T20:25:35.562303+0000","last_fullsized":"2024-03-18T20:25:35.562303+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:27.207710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3f","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188384+0000","last_change":"2024-03-18T20:25:35.578783+0000","last_active":"2024-03-18T20:25:38.188384+0000","last_peered":"2024-03-18T20:25:38.188384+0000","last_clean":"2024-03-18T20:25:38.188384+0000","last_became_active":"2024-03-18T20:25:35.578460+0000","last_became_peered":"2024-03-18T20:25:35.578460+0000","last_unstale":"2024-03-18T20:25:38.188384+0000","last_undegraded":"2024-03-18T20:25:38.188384+0000","last_fullsized":"2024-03-18T20:25:38.188384+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:35:26.519700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.555724+0000","last_change":"2024-03-18T20:25:22.679426+0000","last_active":"2024-03-18T20:25:35.555724+0000","last_peered":"2024-03-18T20:25:35.555724+0000","last_clean":"2024-03-18T20:25:35.555724+0000","last_became_active":"2024-03-18T20:25:22.679159+0000","last_became_peered":"2024-03-18T20:25:22.679159+0000","last_unstale":"2024-03-18T20:25:35.555724+0000","last_undegraded":"2024-03-18T20:25:35.555724+0000","last_fullsized":"2024-03-18T20:25:35.555724+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:17:59.556803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:36.569822+0000","last_change":"2024-03-18T20:25:35.581809+0000","last_active":"2024-03-18T20:25:36.569822+0000","last_peered":"2024-03-18T20:25:36.569822+0000","last_clean":"2024-03-18T20:25:36.569822+0000","last_became_active":"2024-03-18T20:25:35.581546+0000","last_became_peered":"2024-03-18T20:25:35.581546+0000","last_unstale":"2024-03-18T20:25:36.569822+0000","last_undegraded":"2024-03-18T20:25:36.569822+0000","last_fullsized":"2024-03-18T20:25:36.569822+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:21.475337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.571372+0000","last_change":"2024-03-18T20:25:22.682750+0000","last_active":"2024-03-18T20:25:35.571372+0000","last_peered":"2024-03-18T20:25:35.571372+0000","last_clean":"2024-03-18T20:25:35.571372+0000","last_became_active":"2024-03-18T20:25:22.682497+0000","last_became_peered":"2024-03-18T20:25:22.682497+0000","last_unstale":"2024-03-18T20:25:35.571372+0000","last_undegraded":"2024-03-18T20:25:35.571372+0000","last_fullsized":"2024-03-18T20:25:35.571372+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:33.092503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":10,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-18T20:25:38.188636+0000","last_change":"2024-03-18T20:25:35.571829+0000","last_active":"2024-03-18T20:25:38.188636+0000","last_peered":"2024-03-18T20:25:38.188636+0000","last_clean":"2024-03-18T20:25:38.188636+0000","last_became_active":"2024-03-18T20:25:35.571594+0000","last_became_peered":"2024-03-18T20:25:35.571594+0000","last_unstale":"2024-03-18T20:25:38.188636+0000","last_undegraded":"2024-03-18T20:25:38.188636+0000","last_fullsized":"2024-03-18T20:25:38.188636+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:35.113201+0000","last_clean_scrub_stamp":"2024-03-18T20:25:35.113201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:41.225842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3f","version":"0'0","reported_seq":14,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-18T20:25:35.571968+0000","last_change":"2024-03-18T20:25:22.696207+0000","last_active":"2024-03-18T20:25:35.571968+0000","last_peered":"2024-03-18T20:25:35.571968+0000","last_clean":"2024-03-18T20:25:35.571968+0000","last_became_active":"2024-03-18T20:25:22.695943+0000","last_became_peered":"2024-03-18T20:25:22.695943+0000","last_unstale":"2024-03-18T20:25:35.571968+0000","last_undegraded":"2024-03-18T20:25:35.571968+0000","last_fullsized":"2024-03-18T20:25:35.571968+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:25:22.224682+0000","last_clean_scrub_stamp":"2024-03-18T20:25:22.224682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:54:33.571448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":3620,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575984,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52664,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319176,"statfs":{"total":96636764160,"available":96582836224,"internally_reserved":0,"allocated":3579904,"data_stored":2117276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.60799999999999998,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80700000000000005,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.48499999999999999,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.81899999999999995,"5min":1.093,"15min":1.093},"last":0.52500000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61799999999999999,"15min":0.61699999999999999},"min":{"1min":0.47599999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.82999999999999996,"5min":0.99399999999999999,"15min":1.4299999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.60999999999999999,"15min":0.61899999999999999},"min":{"1min":0.47099999999999997,"5min":0.40300000000000002,"15min":0.39400000000000002},"max":{"1min":0.77500000000000002,"5min":1.0449999999999999,"15min":1.2629999999999999},"last":0.47199999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.46400000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84299999999999997,"5min":1.216,"15min":1.216},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.62,"15min":0.625},"min":{"1min":0.48399999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.81799999999999995,"5min":1.177,"15min":1.2490000000000001},"last":0.36599999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.42599999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84699999999999998,"5min":0.93100000000000005,"15min":1.4450000000000001},"last":0.377},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.5,"5min":0.44900000000000001,"15min":0.38600000000000001},"max":{"1min":0.84399999999999997,"5min":1.028,"15min":1.5029999999999999},"last":0.51500000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.47499999999999998,"15min":0.47699999999999998},"min":{"1min":0.33700000000000002,"5min":0.314,"15min":0.29899999999999999},"max":{"1min":0.68400000000000005,"5min":0.79100000000000004,"15min":0.86099999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.35199999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.71599999999999997,"5min":0.76600000000000001,"15min":0.80200000000000005},"last":0.46200000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51500000000000001,"15min":0.51600000000000001},"min":{"1min":0.34899999999999998,"5min":0.34399999999999997,"15min":0.316},"max":{"1min":0.68999999999999995,"5min":0.86599999999999999,"15min":0.89400000000000002},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.40699999999999997,"5min":0.377,"15min":0.36799999999999999},"max":{"1min":0.75600000000000001,"5min":0.81599999999999995,"15min":1.042},"last":0.57599999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:24:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56399999999999995,"15min":0.57699999999999996},"min":{"1min":0.39900000000000002,"5min":0.36199999999999999,"15min":0.34999999999999998},"max":{"1min":0.71099999999999997,"5min":0.85599999999999998,"15min":1.129},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.371,"5min":0.36299999999999999,"15min":0.34000000000000002},"max":{"1min":0.82899999999999996,"5min":0.85299999999999998,"15min":1.0609999999999999},"last":0.501}]}]},{"osd":6,"up_from":13,"seq":55834575984,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42240,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329600,"statfs":{"total":96636764160,"available":96593510400,"internally_reserved":0,"allocated":3002368,"data_stored":1528438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57599999999999996,"15min":0.60099999999999998},"min":{"1min":0.41399999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72199999999999998,"5min":0.80900000000000005,"15min":1.1519999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.59099999999999997,"15min":0.60799999999999998},"min":{"1min":0.33700000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.74299999999999999,"5min":0.81499999999999995,"15min":0.83299999999999996},"last":0.42299999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57899999999999996,"15min":0.60199999999999998},"min":{"1min":0.39400000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.79100000000000004,"5min":0.79800000000000004,"15min":1.244},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58499999999999996,"15min":0.60399999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33200000000000002},"max":{"1min":0.77000000000000002,"5min":0.89200000000000002,"15min":1.286},"last":0.64200000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.60999999999999999,"15min":0.63},"min":{"1min":0.47899999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71199999999999997,"5min":0.92000000000000004,"15min":1.276},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.59299999999999997,"15min":0.61099999999999999},"min":{"1min":0.434,"5min":0.40100000000000002,"15min":0.378},"max":{"1min":0.77500000000000002,"5min":0.95399999999999996,"15min":1.2669999999999999},"last":0.51800000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59699999999999998,"15min":0.622},"min":{"1min":0.38,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70699999999999996,"5min":0.83199999999999996,"15min":1.02},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60599999999999998,"15min":0.61899999999999999},"min":{"1min":0.45200000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82399999999999995,"5min":1.0049999999999999,"15min":1.337},"last":0.59699999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.48099999999999998,"15min":0.48299999999999998},"min":{"1min":0.308,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.61699999999999999,"5min":0.85799999999999998,"15min":0.88300000000000001},"last":0.755},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.50900000000000001,"15min":0.51300000000000001},"min":{"1min":0.34699999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72099999999999997,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.623}]},{"osd":5,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.5,"15min":0.504},"min":{"1min":0.375,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.84799999999999998},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39000000000000001,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.73399999999999999,"5min":0.92600000000000005,"15min":0.94199999999999995},"last":0.72999999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40600000000000003,"5min":0.35799999999999998,"15min":0.34499999999999997},"max":{"1min":0.74299999999999999,"5min":0.95599999999999996,"15min":1.0429999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.44800000000000001,"5min":0.36599999999999999,"15min":0.34599999999999997},"max":{"1min":0.83899999999999997,"5min":1.028,"15min":1.028},"last":0.66600000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575984,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47960,"kb_used_data":2892,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323880,"statfs":{"total":96636764160,"available":96587653120,"internally_reserved":0,"allocated":2961408,"data_stored":1523344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.40400000000000003,"15min":0.39600000000000002},"min":{"1min":0.25600000000000001,"5min":0.246,"15min":0.23999999999999999},"max":{"1min":0.57999999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.41899999999999998,"15min":0.42999999999999999},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.58999999999999997,"5min":0.69699999999999995,"15min":0.72299999999999998},"last":0.32200000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.44,"15min":0.439},"min":{"1min":0.309,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61199999999999999,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48099999999999998,"15min":0.47799999999999998},"min":{"1min":0.35099999999999998,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.70099999999999996,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.36399999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.495,"15min":0.495},"min":{"1min":0.36199999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.64300000000000002,"5min":0.75900000000000001,"15min":0.76100000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.44800000000000001,"15min":0.442},"min":{"1min":0.32800000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68200000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.43099999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.68300000000000005,"15min":0.67300000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86799999999999999,"5min":1.0680000000000001,"15min":1.2909999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.68100000000000005,"15min":0.68000000000000005},"min":{"1min":0.50900000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.84499999999999997,"5min":0.92500000000000004,"15min":1.746},"last":0.52600000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.68100000000000005,"15min":0.67000000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.6910000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.71999999999999997,"15min":0.71399999999999997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.46899999999999997},"max":{"1min":0.89400000000000002,"5min":1.109,"15min":1.2549999999999999},"last":0.51100000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.69699999999999995,"15min":0.68500000000000005},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.437},"max":{"1min":0.97399999999999998,"5min":0.98899999999999999,"15min":1.637},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71899999999999997,"15min":0.71199999999999997},"min":{"1min":0.49099999999999999,"5min":0.46999999999999997,"15min":0.46100000000000002},"max":{"1min":0.89500000000000002,"5min":1.014,"15min":1.722},"last":0.44700000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.70699999999999996,"15min":0.71299999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79800000000000004,"5min":1.02,"15min":1.2729999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.498,"5min":0.495,"15min":0.42799999999999999},"max":{"1min":0.90300000000000002,"5min":1.0329999999999999,"15min":1.3380000000000001},"last":0.64700000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575985,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30496,"kb_used_data":3476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341344,"statfs":{"total":96636764160,"available":96605536256,"internally_reserved":0,"allocated":3559424,"data_stored":2113802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.42399999999999999,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.42099999999999999,"15min":0.42799999999999999},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.46600000000000003,"5min":0.64700000000000002,"15min":1.1850000000000001},"last":0.376}]},{"osd":2,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.42599999999999999,"15min":0.435},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.47099999999999997,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.438,"15min":0.45300000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29799999999999999},"max":{"1min":0.59199999999999997,"5min":0.64300000000000002,"15min":0.70999999999999996},"last":0.32400000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.26100000000000001},"max":{"1min":0.45800000000000002,"5min":0.60099999999999998,"15min":0.628},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.48599999999999999,"15min":0.496},"min":{"1min":0.35499999999999998,"5min":0.35199999999999998,"15min":0.32500000000000001},"max":{"1min":0.52300000000000002,"5min":0.73999999999999999,"15min":0.91500000000000004},"last":0.34899999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64300000000000002,"15min":0.65500000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81299999999999994,"5min":0.91500000000000004,"15min":1.0029999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.66200000000000003,"15min":0.66300000000000003},"min":{"1min":0.47499999999999998,"5min":0.47099999999999997,"15min":0.39700000000000002},"max":{"1min":0.78100000000000003,"5min":0.95299999999999996,"15min":1.0469999999999999},"last":1.2989999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.67800000000000005,"15min":0.67100000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89700000000000002,"5min":0.97099999999999997,"15min":1.0129999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.68300000000000005,"15min":0.68500000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.91100000000000003,"5min":0.92900000000000005,"15min":1.518},"last":0.53200000000000003}]},{"osd":6,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.47399999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.78600000000000003,"5min":0.95899999999999996,"15min":1.1279999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71199999999999997,"15min":0.70899999999999996},"min":{"1min":0.498,"5min":0.48999999999999999,"15min":0.47899999999999998},"max":{"1min":0.94899999999999995,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.54200000000000004}]},{"osd":7,"last update":"Mon Mar 18 20:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85099999999999998,"5min":0.96399999999999997,"15min":1.464},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.69899999999999995,"15min":0.70399999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.47599999999999998},"max":{"1min":0.75800000000000001,"5min":1.042,"15min":1.2350000000000001},"last":1.2769999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575985,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47980,"kb_used_data":2904,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323860,"statfs":{"total":96636764160,"available":96587632640,"internally_reserved":0,"allocated":2973696,"data_stored":1524050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36599999999999999,"15min":0.378},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.496,"5min":0.54800000000000004,"15min":0.59399999999999997},"last":0.315},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.47399999999999998,"15min":0.48699999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.63600000000000001,"5min":0.71499999999999997,"15min":0.753},"last":0.39000000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46899999999999997,"15min":0.48199999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64300000000000002,"5min":0.72599999999999998,"15min":0.77100000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40899999999999997,"15min":0.41599999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.53800000000000003,"5min":0.59299999999999997,"15min":0.71099999999999997},"last":0.33500000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.433,"15min":0.436},"min":{"1min":0.28899999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.497,"15min":0.51700000000000002},"min":{"1min":0.33800000000000002,"5min":0.32200000000000001,"15min":0.313},"max":{"1min":0.68100000000000005,"5min":0.752,"15min":0.752},"last":0.40999999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.66300000000000003,"15min":0.65800000000000003},"min":{"1min":0.503,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":0.82499999999999996,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67500000000000004,"15min":0.66800000000000004},"min":{"1min":0.497,"5min":0.443,"15min":0.443},"max":{"1min":0.91600000000000004,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.64600000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.67900000000000005,"15min":0.68500000000000005},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.93999999999999995,"5min":1.028,"15min":1.028},"last":0.434},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67300000000000004,"15min":0.68000000000000005},"min":{"1min":0.47299999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.88300000000000001,"5min":0.95599999999999996,"15min":1.0629999999999999},"last":0.39900000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68000000000000005,"15min":0.69099999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86099999999999999,"5min":1.014,"15min":1.2050000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.72499999999999998,"15min":0.73199999999999998},"min":{"1min":0.53800000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.97199999999999998,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.80400000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68899999999999995,"15min":0.70199999999999996},"min":{"1min":0.51500000000000001,"5min":0.442,"15min":0.42199999999999999},"max":{"1min":0.92900000000000005,"5min":0.99399999999999999,"15min":1.0469999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69899999999999995,"15min":0.70699999999999996},"min":{"1min":0.45500000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.93999999999999995,"5min":1.371,"15min":1.371},"last":0.92600000000000005}]}]},{"osd":3,"up_from":13,"seq":55834575985,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52072,"kb_used_data":2908,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319768,"statfs":{"total":96636764160,"available":96583442432,"internally_reserved":0,"allocated":2977792,"data_stored":1523492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.38800000000000001,"15min":0.39900000000000002},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.53700000000000003,"5min":0.66500000000000004,"15min":0.98099999999999998},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.40999999999999998,"15min":0.41499999999999998},"min":{"1min":0.27700000000000002,"5min":0.25800000000000001,"15min":0.24199999999999999},"max":{"1min":0.434,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.30499999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39000000000000001,"15min":0.40999999999999998},"min":{"1min":0.30099999999999999,"5min":0.26100000000000001,"15min":0.254},"max":{"1min":0.45900000000000002,"5min":0.63200000000000001,"15min":0.64000000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.436,"5min":0.46100000000000002,"15min":0.46600000000000003},"min":{"1min":0.36699999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57499999999999996,"5min":0.65500000000000003,"15min":0.73099999999999998},"last":0.34999999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.45500000000000002,"15min":0.46600000000000003},"min":{"1min":0.34999999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53100000000000003,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.48199999999999998,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.51700000000000002,"5min":0.67000000000000004,"15min":0.73099999999999998},"last":0.374}]},{"osd":4,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.63600000000000001,"15min":0.65200000000000002},"min":{"1min":0.48799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81899999999999995,"5min":0.91700000000000004,"15min":1.097},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65200000000000002,"15min":0.67500000000000004},"min":{"1min":0.435,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78800000000000003,"5min":0.99299999999999999,"15min":1.0569999999999999},"last":0.46100000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68400000000000005,"15min":0.69699999999999995},"min":{"1min":0.48999999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92100000000000004,"5min":0.995,"15min":1.1599999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.72499999999999998,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.82099999999999995,"5min":1.171,"15min":1.171},"last":0.57099999999999995}]},{"osd":6,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.68500000000000005,"15min":0.70799999999999996},"min":{"1min":0.48099999999999998,"5min":0.436,"15min":0.42499999999999999},"max":{"1min":0.90500000000000003,"5min":1.012,"15min":1.194},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.71799999999999997,"15min":0.73299999999999998},"min":{"1min":0.53300000000000003,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":0.88200000000000001,"5min":1.0409999999999999,"15min":1.2410000000000001},"last":0.51600000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:25:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.70099999999999996,"15min":0.71899999999999997},"min":{"1min":0.49099999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82799999999999996,"5min":1.079,"15min":1.079},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68100000000000005,"15min":0.69999999999999996},"min":{"1min":0.44500000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83799999999999997,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.48399999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575985,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64352,"kb_used_data":2900,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307488,"statfs":{"total":96636764160,"available":96570867712,"internally_reserved":0,"allocated":2969600,"data_stored":1524016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.57899999999999996,"15min":0.59299999999999997},"min":{"1min":0.34300000000000003,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65300000000000002,"5min":0.77500000000000002,"15min":0.81799999999999995},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57799999999999996,"15min":0.59799999999999998},"min":{"1min":0.40300000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69999999999999996,"5min":0.79600000000000004,"15min":0.84199999999999997},"last":0.80600000000000005}]},{"osd":1,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59999999999999998,"15min":0.61399999999999999},"min":{"1min":0.438,"5min":0.318,"15min":0.318},"max":{"1min":0.73199999999999998,"5min":0.81899999999999995,"15min":0.88300000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56999999999999995,"15min":0.58499999999999996},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.71199999999999997,"5min":0.79200000000000004,"15min":0.96499999999999997},"last":0.73099999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58299999999999996,"15min":0.59899999999999998},"min":{"1min":0.439,"5min":0.39500000000000002,"15min":0.39200000000000002},"max":{"1min":0.68600000000000005,"5min":0.80500000000000005,"15min":0.82599999999999996},"last":0.748},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61099999999999999,"15min":0.61799999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.38500000000000001},"max":{"1min":0.81399999999999995,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.67200000000000004}]},{"osd":3,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.40799999999999997,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.86199999999999999},"last":0.70499999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.498,"15min":0.504},"min":{"1min":0.35199999999999998,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.85499999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.53300000000000003}]},{"osd":6,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.499,"15min":0.505},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.621}]},{"osd":7,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54100000000000004,"15min":0.54800000000000004},"min":{"1min":0.435,"5min":0.38,"15min":0.35699999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.57199999999999995,"15min":0.57999999999999996},"min":{"1min":0.41899999999999998,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.56200000000000006}]}]},{"osd":5,"up_from":13,"seq":55834575984,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47992,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323848,"statfs":{"total":96636764160,"available":96587620352,"internally_reserved":0,"allocated":2985984,"data_stored":1525034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42499999999999999,"5min":0.33900000000000002,"15min":0.31900000000000001},"max":{"1min":0.72699999999999998,"5min":0.85999999999999999,"15min":0.96499999999999997},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57799999999999996,"15min":0.60399999999999998},"min":{"1min":0.432,"5min":0.37,"15min":0.33100000000000002},"max":{"1min":0.69799999999999995,"5min":0.82599999999999996,"15min":1.3100000000000001},"last":0.68000000000000005}]},{"osd":1,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58499999999999996,"15min":0.60199999999999998},"min":{"1min":0.36899999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.74399999999999999,"5min":0.79600000000000004,"15min":1.056},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58399999999999996,"15min":0.60199999999999998},"min":{"1min":0.46600000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.63100000000000001,"5min":0.89800000000000002,"15min":1.018},"last":0.84399999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.38700000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.84299999999999997,"15min":0.99299999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.59399999999999997,"15min":0.60899999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.67500000000000004,"5min":0.875,"15min":1.1120000000000001},"last":0.73799999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.61199999999999999,"15min":0.61299999999999999},"min":{"1min":0.42799999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.78100000000000003,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63,"15min":0.64600000000000002},"min":{"1min":0.443,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69899999999999995,"5min":0.82399999999999995,"15min":1.161},"last":0.71999999999999997}]},{"osd":4,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.47899999999999998,"15min":0.47499999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.77500000000000002},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51200000000000001,"15min":0.50600000000000001},"min":{"1min":0.36199999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.89700000000000002},"last":0.75700000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.503,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.34000000000000002,"15min":0.32800000000000001},"max":{"1min":0.69999999999999996,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.40300000000000002,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.86599999999999999},"last":0.59999999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.40799999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.94499999999999995},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.45300000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74099999999999999,"5min":0.85999999999999999,"15min":0.91200000000000003},"last":0.63600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:25:44.784 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T20:25:44.784 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:25:44.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.937+0000 7f632eca0700 1 Processor -- start 2024-03-18T20:25:44.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.937+0000 7f632eca0700 1 -- start start 2024-03-18T20:25:44.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.938+0000 7f632eca0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f63280ae600 0x7f63280ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:44.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.938+0000 7f632eca0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63280cc170 0x7f632806ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:44.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.938+0000 7f632eca0700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632806e270 con 0x7f632815a2e0 2024-03-18T20:25:44.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.938+0000 7f632eca0700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63280ae050 con 0x7f63280cc170 2024-03-18T20:25:44.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.938+0000 7f632eca0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63280add70 con 0x7f63280ae600 2024-03-18T20:25:44.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.938+0000 7f632d23d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63280cc170 0x7f632806ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:44.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.938+0000 7f6327fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f632815a2e0 0x7f63280ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45228/0 (socket says 172.21.15.1:45228) 2024-03-18T20:25:44.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.938+0000 7f6327fff700 1 -- 172.21.15.1:0/2249193414 learned_addr learned my addr 172.21.15.1:0/2249193414 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:44.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.938+0000 7f632ca3c700 1 --2- 172.21.15.1:0/2249193414 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f63280ae600 0x7f63280ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:44.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.939+0000 7f632d23d700 1 -- 172.21.15.1:0/2249193414 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f63280ae600 msgr2=0x7f63280ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:44.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.939+0000 7f632d23d700 1 --2- 172.21.15.1:0/2249193414 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f63280ae600 0x7f63280ad790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.939+0000 7f632d23d700 1 -- 172.21.15.1:0/2249193414 >> v1:172.21.15.1:6789/0 conn(0x7f632815a2e0 legacy=0x7f63280ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:44.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.939+0000 7f632d23d700 1 -- 172.21.15.1:0/2249193414 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632806e4a0 con 0x7f63280cc170 2024-03-18T20:25:44.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.939+0000 7f63277fe700 1 -- 172.21.15.1:0/2249193414 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215626518 0 0) 0x7f632806e270 con 0x7f632815a2e0 2024-03-18T20:25:44.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.939+0000 7f632d23d700 1 --2- 172.21.15.1:0/2249193414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280cc170 0x7f632806ef80 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7f6320009d90 tx=0x7f632000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b7e1f669cb3d5e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:44.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.939+0000 7f63277fe700 1 -- 172.21.15.1:0/2249193414 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f632000e070 con 0x7f63280cc170 2024-03-18T20:25:44.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.939+0000 7f63277fe700 1 -- 172.21.15.1:0/2249193414 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f632000bbd0 con 0x7f63280cc170 2024-03-18T20:25:44.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.939+0000 7f63277fe700 1 -- 172.21.15.1:0/2249193414 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6320012430 con 0x7f63280cc170 2024-03-18T20:25:44.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.939+0000 7f632eca0700 1 -- 172.21.15.1:0/2249193414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280cc170 msgr2=0x7f632806ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:44.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.939+0000 7f632eca0700 1 --2- 172.21.15.1:0/2249193414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280cc170 0x7f632806ef80 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7f6320009d90 tx=0x7f632000b330 comp rx=0 tx=0).stop 2024-03-18T20:25:44.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.939+0000 7f632eca0700 1 -- 172.21.15.1:0/2249193414 shutdown_connections 2024-03-18T20:25:44.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.939+0000 7f632eca0700 1 --2- 172.21.15.1:0/2249193414 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280cc170 0x7f632806ef80 unknown :-1 s=CLOSED pgs=2595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.939+0000 7f632eca0700 1 --2- 172.21.15.1:0/2249193414 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f63280ae600 0x7f63280ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.939+0000 7f632eca0700 1 -- 172.21.15.1:0/2249193414 >> 172.21.15.1:0/2249193414 conn(0x7f632806b860 msgr2=0x7f632806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:44.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.939+0000 7f632eca0700 1 -- 172.21.15.1:0/2249193414 shutdown_connections 2024-03-18T20:25:44.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.939+0000 7f632eca0700 1 -- 172.21.15.1:0/2249193414 wait complete. 2024-03-18T20:25:44.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.941+0000 7f632eca0700 1 Processor -- start 2024-03-18T20:25:44.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.941+0000 7f632eca0700 1 -- start start 2024-03-18T20:25:44.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.942+0000 7f632eca0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63280ae600 0x7f632814f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:44.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.942+0000 7f632eca0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280cc170 0x7f6328155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:44.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.942+0000 7f632eca0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f632815a2e0 0x7f6328155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:44.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.942+0000 7f632eca0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63281517f0 con 0x7f63280cc170 2024-03-18T20:25:44.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.942+0000 7f632eca0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6328151ad0 con 0x7f63280ae600 2024-03-18T20:25:44.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.942+0000 7f632eca0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6328151db0 con 0x7f632815a2e0 2024-03-18T20:25:44.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.942+0000 7f6327fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280cc170 0x7f6328155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:44.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.942+0000 7f6327fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280cc170 0x7f6328155200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54100/0 (socket says 172.21.15.1:54100) 2024-03-18T20:25:44.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.942+0000 7f6327fff700 1 -- 172.21.15.1:0/4260873208 learned_addr learned my addr 172.21.15.1:0/4260873208 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:44.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.942+0000 7f632ca3c700 1 --2- 172.21.15.1:0/4260873208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63280ae600 0x7f632814f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:44.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.942+0000 7f632d23d700 1 --2- 172.21.15.1:0/4260873208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f632815a2e0 0x7f6328155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:44.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.942+0000 7f6327fff700 1 -- 172.21.15.1:0/4260873208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f632815a2e0 msgr2=0x7f6328155740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:44.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.943+0000 7f6327fff700 1 --2- 172.21.15.1:0/4260873208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f632815a2e0 0x7f6328155740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.943+0000 7f6327fff700 1 -- 172.21.15.1:0/4260873208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63280ae600 msgr2=0x7f632814f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:44.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.943+0000 7f6327fff700 1 --2- 172.21.15.1:0/4260873208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63280ae600 0x7f632814f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:44.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.943+0000 7f6327fff700 1 -- 172.21.15.1:0/4260873208 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6320009a70 con 0x7f63280cc170 2024-03-18T20:25:44.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.943+0000 7f632ca3c700 1 --2- 172.21.15.1:0/4260873208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63280ae600 0x7f632814f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:44.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.943+0000 7f632d23d700 1 --2- 172.21.15.1:0/4260873208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f632815a2e0 0x7f6328155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:44.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.943+0000 7f6327fff700 1 --2- 172.21.15.1:0/4260873208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280cc170 0x7f6328155200 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7f631400c540 tx=0x7f631400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:44.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.943+0000 7f63257fa700 1 -- 172.21.15.1:0/4260873208 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6314004c80 con 0x7f63280cc170 2024-03-18T20:25:44.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.943+0000 7f63257fa700 1 -- 172.21.15.1:0/4260873208 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6314004de0 con 0x7f63280cc170 2024-03-18T20:25:44.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.943+0000 7f632eca0700 1 -- 172.21.15.1:0/4260873208 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6328152090 con 0x7f63280cc170 2024-03-18T20:25:44.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.943+0000 7f63257fa700 1 -- 172.21.15.1:0/4260873208 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6314007310 con 0x7f63280cc170 2024-03-18T20:25:44.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.943+0000 7f632eca0700 1 -- 172.21.15.1:0/4260873208 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6328158aa0 con 0x7f63280cc170 2024-03-18T20:25:44.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.944+0000 7f632eca0700 1 -- 172.21.15.1:0/4260873208 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6328152280 con 0x7f63280cc170 2024-03-18T20:25:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.944+0000 7f63257fa700 1 -- 172.21.15.1:0/4260873208 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6314007470 con 0x7f63280cc170 2024-03-18T20:25:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.945+0000 7f63257fa700 1 --2- 172.21.15.1:0/4260873208 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f630c06adb0 0x7f630c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.945+0000 7f63257fa700 1 -- 172.21.15.1:0/4260873208 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f6314089770 con 0x7f63280cc170 2024-03-18T20:25:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.945+0000 7f632ca3c700 1 --2- 172.21.15.1:0/4260873208 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f630c06adb0 0x7f630c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.945+0000 7f632ca3c700 1 --2- 172.21.15.1:0/4260873208 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f630c06adb0 0x7f630c06d270 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6328154f90 tx=0x7f631c008040 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:44.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:44.950+0000 7f63257fa700 1 -- 172.21.15.1:0/4260873208 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6314054320 con 0x7f63280cc170 2024-03-18T20:25:45.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.197+0000 7f632eca0700 1 -- 172.21.15.1:0/4260873208 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6328151930 con 0x7f63280cc170 2024-03-18T20:25:45.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.197+0000 7f63257fa700 1 -- 172.21.15.1:0/4260873208 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5963 (secure 0 0 0) 0x7f6328151930 con 0x7f63280cc170 2024-03-18T20:25:45.202 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:45.202 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":110,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10369,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/677258734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":677258734},{"type":"v1","addr":"172.21.15.165:6835","nonce":677258734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":11544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2018398457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2018398457},{"type":"v1","addr":"172.21.15.1:6836","nonce":2018398457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:25:22.663717+0000","modified":"2024-03-18T20:25:25.668584+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11556":{"gid":11556,"name":"d","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6834/4024828630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4024828630},{"type":"v1","addr":"172.21.15.165:6834","nonce":4024828630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:25:35.549382+0000","modified":"2024-03-18T20:25:38.565325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11547":{"gid":11547,"name":"c","rank":0,"incarnation":106,"state":"up:active","state_seq":6,"addr":"172.21.15.1:6837/1635684321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1635684321},{"type":"v1","addr":"172.21.15.1:6837","nonce":1635684321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-03-18T20:25:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.199+0000 7f632eca0700 1 -- 172.21.15.1:0/4260873208 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f630c06adb0 msgr2=0x7f630c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.199+0000 7f632eca0700 1 --2- 172.21.15.1:0/4260873208 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f630c06adb0 0x7f630c06d270 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6328154f90 tx=0x7f631c008040 comp rx=0 tx=0).stop 2024-03-18T20:25:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.199+0000 7f632eca0700 1 -- 172.21.15.1:0/4260873208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280cc170 msgr2=0x7f6328155200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.199+0000 7f632eca0700 1 --2- 172.21.15.1:0/4260873208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280cc170 0x7f6328155200 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7f631400c540 tx=0x7f631400c900 comp rx=0 tx=0).stop 2024-03-18T20:25:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.199+0000 7f632eca0700 1 -- 172.21.15.1:0/4260873208 shutdown_connections 2024-03-18T20:25:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.200+0000 7f632eca0700 1 --2- 172.21.15.1:0/4260873208 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f630c06adb0 0x7f630c06d270 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.200+0000 7f632eca0700 1 --2- 172.21.15.1:0/4260873208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f632815a2e0 0x7f6328155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.200+0000 7f632eca0700 1 --2- 172.21.15.1:0/4260873208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63280ae600 0x7f632814f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.200+0000 7f632eca0700 1 --2- 172.21.15.1:0/4260873208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280cc170 0x7f6328155200 unknown :-1 s=CLOSED pgs=2596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.200+0000 7f632eca0700 1 -- 172.21.15.1:0/4260873208 >> 172.21.15.1:0/4260873208 conn(0x7f632806b860 msgr2=0x7f63280b03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.200+0000 7f632eca0700 1 -- 172.21.15.1:0/4260873208 shutdown_connections 2024-03-18T20:25:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.200+0000 7f632eca0700 1 -- 172.21.15.1:0/4260873208 wait complete. 2024-03-18T20:25:45.205 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 110 2024-03-18T20:25:45.618 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 110, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T20:25:35.549382+0000', 'modified': '2024-03-18T20:25:38.565325+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11547}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11547': {'gid': 11547, 'name': 'c', 'rank': 0, 'incarnation': 106, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.1:6837/1635684321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6835', 'nonce': 1635684321}, {'type': 'v1', 'addr': '172.21.15.1:6837', 'nonce': 1635684321}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T20:25:45.618 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T20:25:45.619 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T20:25:45.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.742+0000 7ff392609700 1 Processor -- start 2024-03-18T20:25:45.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.743+0000 7ff392609700 1 -- start start 2024-03-18T20:25:45.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.743+0000 7ff392609700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff38c15d500 0x7ff38c06c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.743+0000 7ff392609700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff38c06c7b0 0x7ff38c099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.743+0000 7ff392609700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38c06f490 con 0x7ff38c0cb7b0 2024-03-18T20:25:45.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.743+0000 7ff392609700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff38c06e880 con 0x7ff38c06c7b0 2024-03-18T20:25:45.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.743+0000 7ff392609700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c06eb60 con 0x7ff38c15d500 2024-03-18T20:25:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.743+0000 7ff38b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff38c06c7b0 0x7ff38c099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.744+0000 7ff38b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff38c06c7b0 0x7ff38c099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54110/0 (socket says 172.21.15.1:54110) 2024-03-18T20:25:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.744+0000 7ff38b7fe700 1 -- 172.21.15.1:0/1541146204 learned_addr learned my addr 172.21.15.1:0/1541146204 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.744+0000 7ff38bfff700 1 --2- 172.21.15.1:0/1541146204 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff38c15d500 0x7ff38c06c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.744+0000 7ff38b7fe700 1 -- 172.21.15.1:0/1541146204 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff38c15d500 msgr2=0x7ff38c06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.744+0000 7ff38b7fe700 1 --2- 172.21.15.1:0/1541146204 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff38c15d500 0x7ff38c06c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.744+0000 7ff38b7fe700 1 -- 172.21.15.1:0/1541146204 >> v1:172.21.15.165:6789/0 conn(0x7ff38c0cb7b0 legacy=0x7ff38c0cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.744+0000 7ff38b7fe700 1 -- 172.21.15.1:0/1541146204 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c06ee40 con 0x7ff38c06c7b0 2024-03-18T20:25:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.744+0000 7ff38b7fe700 1 --2- 172.21.15.1:0/1541146204 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff38c06c7b0 0x7ff38c099a40 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7ff3740098f0 tx=0x7ff374009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b3d6ae0ad731a2b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.744+0000 7ff38affd700 1 -- 172.21.15.1:0/1541146204 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37400b8a0 con 0x7ff38c06c7b0 2024-03-18T20:25:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.744+0000 7ff38affd700 1 -- 172.21.15.1:0/1541146204 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37400ba00 con 0x7ff38c06c7b0 2024-03-18T20:25:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.744+0000 7ff38affd700 1 -- 172.21.15.1:0/1541146204 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37400bd90 con 0x7ff38c06c7b0 2024-03-18T20:25:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.745+0000 7ff392609700 1 -- 172.21.15.1:0/1541146204 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff38c06c7b0 msgr2=0x7ff38c099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.745+0000 7ff392609700 1 --2- 172.21.15.1:0/1541146204 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff38c06c7b0 0x7ff38c099a40 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7ff3740098f0 tx=0x7ff374009c00 comp rx=0 tx=0).stop 2024-03-18T20:25:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.745+0000 7ff392609700 1 -- 172.21.15.1:0/1541146204 shutdown_connections 2024-03-18T20:25:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.745+0000 7ff392609700 1 --2- 172.21.15.1:0/1541146204 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff38c15d500 0x7ff38c06c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.745+0000 7ff392609700 1 --2- 172.21.15.1:0/1541146204 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff38c06c7b0 0x7ff38c099a40 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.745+0000 7ff392609700 1 -- 172.21.15.1:0/1541146204 >> 172.21.15.1:0/1541146204 conn(0x7ff38c0b2c50 msgr2=0x7ff38c0b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.745+0000 7ff392609700 1 -- 172.21.15.1:0/1541146204 shutdown_connections 2024-03-18T20:25:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.745+0000 7ff392609700 1 -- 172.21.15.1:0/1541146204 wait complete. 2024-03-18T20:25:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.746+0000 7ff392609700 1 Processor -- start 2024-03-18T20:25:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.746+0000 7ff392609700 1 -- start start 2024-03-18T20:25:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.747+0000 7ff392609700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff38c06c7b0 0x7ff38c0c9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.747+0000 7ff392609700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff38c0cb7b0 0x7ff38c152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.747+0000 7ff392609700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff38c15d500 0x7ff38c0cde90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.747+0000 7ff392609700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c0c98f0 con 0x7ff38c06c7b0 2024-03-18T20:25:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.747+0000 7ff392609700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c0c9bd0 con 0x7ff38c0cb7b0 2024-03-18T20:25:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.747+0000 7ff392609700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c0c9eb0 con 0x7ff38c15d500 2024-03-18T20:25:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.747+0000 7ff38bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff38c06c7b0 0x7ff38c0c9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.747+0000 7ff38bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff38c06c7b0 0x7ff38c0c9280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54124/0 (socket says 172.21.15.1:54124) 2024-03-18T20:25:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.747+0000 7ff38bfff700 1 -- 172.21.15.1:0/3771531999 learned_addr learned my addr 172.21.15.1:0/3771531999 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:45.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.747+0000 7ff38b7fe700 1 --2- 172.21.15.1:0/3771531999 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff38c0cb7b0 0x7ff38c152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.747+0000 7ff390ba6700 1 --2- 172.21.15.1:0/3771531999 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff38c15d500 0x7ff38c0cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.747+0000 7ff38bfff700 1 -- 172.21.15.1:0/3771531999 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff38c15d500 msgr2=0x7ff38c0cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:45.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.747+0000 7ff38bfff700 1 --2- 172.21.15.1:0/3771531999 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff38c15d500 0x7ff38c0cde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.747+0000 7ff38bfff700 1 -- 172.21.15.1:0/3771531999 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff38c0cb7b0 msgr2=0x7ff38c152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:45.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.747+0000 7ff38bfff700 1 --2- 172.21.15.1:0/3771531999 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff38c0cb7b0 0x7ff38c152580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.747+0000 7ff38bfff700 1 -- 172.21.15.1:0/3771531999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff374009550 con 0x7ff38c06c7b0 2024-03-18T20:25:45.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.747+0000 7ff390ba6700 1 --2- 172.21.15.1:0/3771531999 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff38c15d500 0x7ff38c0cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:45.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.747+0000 7ff38bfff700 1 --2- 172.21.15.1:0/3771531999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff38c06c7b0 0x7ff38c0c9280 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00d800 tx=0x7ff37c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:45.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.747+0000 7ff388ff9700 1 -- 172.21.15.1:0/3771531999 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c0125f0 con 0x7ff38c06c7b0 2024-03-18T20:25:45.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.748+0000 7ff388ff9700 1 -- 172.21.15.1:0/3771531999 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37c012750 con 0x7ff38c06c7b0 2024-03-18T20:25:45.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.748+0000 7ff392609700 1 -- 172.21.15.1:0/3771531999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38c0ca190 con 0x7ff38c06c7b0 2024-03-18T20:25:45.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.748+0000 7ff388ff9700 1 -- 172.21.15.1:0/3771531999 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c012ae0 con 0x7ff38c06c7b0 2024-03-18T20:25:45.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.748+0000 7ff392609700 1 -- 172.21.15.1:0/3771531999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38c0cac20 con 0x7ff38c06c7b0 2024-03-18T20:25:45.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.749+0000 7ff388ff9700 1 -- 172.21.15.1:0/3771531999 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ff37c012c40 con 0x7ff38c06c7b0 2024-03-18T20:25:45.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.749+0000 7ff388ff9700 1 --2- 172.21.15.1:0/3771531999 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff36c06adb0 0x7ff36c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.750+0000 7ff388ff9700 1 -- 172.21.15.1:0/3771531999 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6993+0+0 (secure 0 0 0) 0x7ff37c089c20 con 0x7ff38c06c7b0 2024-03-18T20:25:45.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.750+0000 7ff38b7fe700 1 --2- 172.21.15.1:0/3771531999 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff36c06adb0 0x7ff36c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.750+0000 7ff38b7fe700 1 --2- 172.21.15.1:0/3771531999 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff36c06adb0 0x7ff36c06d270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff374009840 tx=0x7ff374008030 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.751+0000 7ff39e7b2b80 1 Processor -- start 2024-03-18T20:25:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.751+0000 7ff39e7b2b80 1 -- start start 2024-03-18T20:25:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.752+0000 7ff39e7b2b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d626ff2c10 0x55d626ff2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.752+0000 7ff39e7b2b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 0x55d62700f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.752+0000 7ff39e7b2b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d627004630 0x55d626fc7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.752+0000 7ff39e7b2b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d626f3df20 con 0x55d627004630 2024-03-18T20:25:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.752+0000 7ff39e7b2b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d626f1fc30 con 0x55d626fc40a0 2024-03-18T20:25:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.752+0000 7ff39e7b2b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d626f1d430 con 0x55d626ff2c10 2024-03-18T20:25:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.752+0000 7ff390ba6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d627004630 0x55d626fc7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.752+0000 7ff390ba6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d627004630 0x55d626fc7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54134/0 (socket says 172.21.15.1:54134) 2024-03-18T20:25:45.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.752+0000 7ff390ba6700 1 -- 172.21.15.1:0/3468167521 learned_addr learned my addr 172.21.15.1:0/3468167521 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.752+0000 7ff38bfff700 1 --2- 172.21.15.1:0/3468167521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d626ff2c10 0x55d626ff2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.752+0000 7ff38b7fe700 1 --2- 172.21.15.1:0/3468167521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 0x55d62700f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.752+0000 7ff390ba6700 1 -- 172.21.15.1:0/3468167521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d626ff2c10 msgr2=0x55d626ff2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.753+0000 7ff390ba6700 1 --2- 172.21.15.1:0/3468167521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d626ff2c10 0x55d626ff2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.753+0000 7ff390ba6700 1 -- 172.21.15.1:0/3468167521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 msgr2=0x55d62700f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.753+0000 7ff390ba6700 1 --2- 172.21.15.1:0/3468167521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 0x55d62700f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.753+0000 7ff390ba6700 1 -- 172.21.15.1:0/3468167521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d626d0c790 con 0x55d627004630 2024-03-18T20:25:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.753+0000 7ff390ba6700 1 --2- 172.21.15.1:0/3468167521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d627004630 0x55d626fc7160 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7ff384002180 tx=0x7ff38400e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e7e5328f5136c60 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.753+0000 7ff381ffb700 1 -- 172.21.15.1:0/3468167521 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38400ec70 con 0x55d627004630 2024-03-18T20:25:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.753+0000 7ff381ffb700 1 -- 172.21.15.1:0/3468167521 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38400edd0 con 0x55d627004630 2024-03-18T20:25:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.753+0000 7ff381ffb700 1 -- 172.21.15.1:0/3468167521 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff384003c00 con 0x55d627004630 2024-03-18T20:25:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.754+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/3468167521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d627004630 msgr2=0x55d626fc7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.754+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/3468167521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d627004630 0x55d626fc7160 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7ff384002180 tx=0x7ff38400e440 comp rx=0 tx=0).stop 2024-03-18T20:25:45.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.754+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/3468167521 shutdown_connections 2024-03-18T20:25:45.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.754+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/3468167521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d626ff2c10 0x55d626ff2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.754+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/3468167521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d627004630 0x55d626fc7160 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.754+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/3468167521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 0x55d62700f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.754+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/3468167521 >> 172.21.15.1:0/3468167521 conn(0x55d626f26cf0 msgr2=0x55d626fc6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:45.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.754+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/3468167521 shutdown_connections 2024-03-18T20:25:45.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.754+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/3468167521 wait complete. 2024-03-18T20:25:45.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.754+0000 7ff39e7b2b80 1 -- start start 2024-03-18T20:25:45.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.755+0000 7ff39e7b2b80 1 Processor -- start 2024-03-18T20:25:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.755+0000 7ff39e7b2b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 0x55d626feabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.755+0000 7ff39e7b2b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d626ff2c10 0x55d62700f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.755+0000 7ff39e7b2b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d627004630 0x55d62700f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.755+0000 7ff39e7b2b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d626f18d20 con 0x55d626ff2c10 2024-03-18T20:25:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.755+0000 7ff39e7b2b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d626eb4770 con 0x55d626fc40a0 2024-03-18T20:25:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.755+0000 7ff39e7b2b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d626ed76f0 con 0x55d627004630 2024-03-18T20:25:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.755+0000 7ff39e7b2b80 10 client.0 _ll_register_callbacks cb 0x55d626d205a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:25:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.755+0000 7ff38b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d626ff2c10 0x55d62700f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.755+0000 7ff38b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d626ff2c10 0x55d62700f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54146/0 (socket says 172.21.15.1:54146) 2024-03-18T20:25:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.755+0000 7ff38b7fe700 1 -- 172.21.15.1:0/3479662295 learned_addr learned my addr 172.21.15.1:0/3479662295 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.756+0000 7ff38b7fe700 1 -- 172.21.15.1:0/3479662295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d627004630 msgr2=0x55d62700f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.756+0000 7ff38bfff700 1 --2- 172.21.15.1:0/3479662295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 0x55d626feabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.756+0000 7ff390ba6700 1 --2- 172.21.15.1:0/3479662295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d627004630 0x55d62700f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.756+0000 7ff38b7fe700 1 --2- 172.21.15.1:0/3479662295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d627004630 0x55d62700f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.756+0000 7ff38b7fe700 1 -- 172.21.15.1:0/3479662295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 msgr2=0x55d626feabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.756+0000 7ff38b7fe700 1 --2- 172.21.15.1:0/3479662295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 0x55d626feabb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.756+0000 7ff38b7fe700 1 -- 172.21.15.1:0/3479662295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38400f040 con 0x55d626ff2c10 2024-03-18T20:25:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.756+0000 7ff38b7fe700 1 --2- 172.21.15.1:0/3479662295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d626ff2c10 0x55d62700f420 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7ff374009c40 tx=0x7ff3740047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8abfbda3397993fb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.756+0000 7ff36affd700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T20:25:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.756+0000 7ff36affd700 1 -- 172.21.15.1:0/3479662295 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374020070 con 0x55d626ff2c10 2024-03-18T20:25:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.756+0000 7ff36affd700 1 -- 172.21.15.1:0/3479662295 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(236..236 src has 1..236) v4 ==== 6993+0+0 (secure 0 0 0) 0x7ff374003ab0 con 0x55d626ff2c10 2024-03-18T20:25:45.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.756+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/3479662295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d626fbae90 con 0x55d626ff2c10 2024-03-18T20:25:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.756+0000 7ff36affd700 1 -- 172.21.15.1:0/3479662295 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff374003c70 con 0x55d626ff2c10 2024-03-18T20:25:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.756+0000 7ff39e7b2b80 10 client.11883 fetch_fsmap learned FSMap version 110 2024-03-18T20:25:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.756+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/3479662295 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=2+}) v3 -- 0x55d626df90a0 con 0x55d626ff2c10 2024-03-18T20:25:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.756+0000 7ff36affd700 1 -- 172.21.15.1:0/3479662295 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d626fbae90 con 0x55d626ff2c10 2024-03-18T20:25:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.757+0000 7ff36affd700 1 -- 172.21.15.1:0/3479662295 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 110) v1 ==== 3681+0+0 (secure 0 0 0) 0x7ff374030070 con 0x55d626ff2c10 2024-03-18T20:25:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.757+0000 7ff39e7b2b80 10 client.11883 fetch_fsmap finished waiting for FSMap version 110 2024-03-18T20:25:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.757+0000 7ff39e7b2b80 10 client.11883 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T20:25:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.757+0000 7ff39e7b2b80 20 client.11883 populate_metadata read hostname 'smithi001' 2024-03-18T20:25:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.757+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/3479662295 >> [v2:172.21.15.1:6835/1635684321,v1:172.21.15.1:6837/1635684321] conn(0x55d626fc8010 0x55d62702a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.757+0000 7ff39e7b2b80 4 client.11883 mds_command: new command op to 11547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T20:25:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.757+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/3479662295 --> [v2:172.21.15.1:6835/1635684321,v1:172.21.15.1:6837/1635684321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d626f98010 con 0x55d626fc8010 2024-03-18T20:25:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.757+0000 7ff38bfff700 1 --2- 172.21.15.1:0/3479662295 >> [v2:172.21.15.1:6835/1635684321,v1:172.21.15.1:6837/1635684321] conn(0x55d626fc8010 0x55d62702a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.758+0000 7ff36affd700 10 client.11883 ms_handle_connect on v2:172.21.15.1:6835/1635684321 2024-03-18T20:25:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.758+0000 7ff38bfff700 1 --2- 172.21.15.1:0/3479662295 >> [v2:172.21.15.1:6835/1635684321,v1:172.21.15.1:6837/1635684321] conn(0x55d626fc8010 0x55d62702a6f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3134e550caa37230 server_cookie=b1cce1793c9d1d31 in_seq=0 out_seq=0 2024-03-18T20:25:45.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.758+0000 7ff36affd700 1 -- 172.21.15.1:0/3479662295 <== mds.0 v2:172.21.15.1:6835/1635684321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d626f98010 con 0x55d626fc8010 2024-03-18T20:25:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.758+0000 7ff36affd700 10 client.11883 handle_command_reply: tid=0 2024-03-18T20:25:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.758+0000 7ff39e7b2b80 1 client.11883 shutdown 2024-03-18T20:25:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.758+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/3479662295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d626ff2c10 msgr2=0x55d62700f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.758+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/3479662295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d626ff2c10 0x55d62700f420 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7ff374009c40 tx=0x7ff3740047c0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.758+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/3479662295 shutdown_connections 2024-03-18T20:25:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.758+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/3479662295 >> [v2:172.21.15.1:6835/1635684321,v1:172.21.15.1:6837/1635684321] conn(0x55d626fc8010 0x55d62702a6f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.758+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/3479662295 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d627004630 0x55d62700f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.758+0000 7ff36affd700 0 client.11883 ms_handle_reset on v2:172.21.15.1:6835/1635684321 2024-03-18T20:25:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.758+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/3479662295 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 0x55d626feabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.758+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/3479662295 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d626ff2c10 0x55d62700f420 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.758+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/3479662295 >> 172.21.15.1:0/3479662295 conn(0x55d626f26cf0 msgr2=0x55d626ff3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.758+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/3479662295 shutdown_connections 2024-03-18T20:25:45.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.758+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/3479662295 wait complete. 2024-03-18T20:25:45.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.759+0000 7ff39e7b2b80 20 client.11883 trim_cache size 0 max 16384 2024-03-18T20:25:45.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.776+0000 7ff39e7b2b80 1 Processor -- start 2024-03-18T20:25:45.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.776+0000 7ff39e7b2b80 1 -- start start 2024-03-18T20:25:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.777+0000 7ff39e7b2b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 0x55d626ff4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.777+0000 7ff39e7b2b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d626ff2c10 0x55d626fedd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.777+0000 7ff39e7b2b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d627004630 0x55d626fef830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.777+0000 7ff39e7b2b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d626d0d390 con 0x55d627004630 2024-03-18T20:25:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.777+0000 7ff39e7b2b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d626fc8c00 con 0x55d626fc40a0 2024-03-18T20:25:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.777+0000 7ff39e7b2b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d626ff6c60 con 0x55d626ff2c10 2024-03-18T20:25:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.777+0000 7ff390ba6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d627004630 0x55d626fef830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.777+0000 7ff390ba6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d627004630 0x55d626fef830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54152/0 (socket says 172.21.15.1:54152) 2024-03-18T20:25:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.777+0000 7ff390ba6700 1 -- 172.21.15.1:0/3946721024 learned_addr learned my addr 172.21.15.1:0/3946721024 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.777+0000 7ff38b7fe700 1 --2- 172.21.15.1:0/3946721024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d626ff2c10 0x55d626fedd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.777+0000 7ff38bfff700 1 --2- 172.21.15.1:0/3946721024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 0x55d626ff4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.777+0000 7ff390ba6700 1 -- 172.21.15.1:0/3946721024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d626ff2c10 msgr2=0x55d626fedd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:45.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.777+0000 7ff390ba6700 1 --2- 172.21.15.1:0/3946721024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d626ff2c10 0x55d626fedd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.777+0000 7ff390ba6700 1 -- 172.21.15.1:0/3946721024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 msgr2=0x55d626ff4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:45.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.777+0000 7ff390ba6700 1 --2- 172.21.15.1:0/3946721024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 0x55d626ff4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.777+0000 7ff390ba6700 1 -- 172.21.15.1:0/3946721024 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d626edf010 con 0x55d627004630 2024-03-18T20:25:45.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.777+0000 7ff38bfff700 1 --2- 172.21.15.1:0/3946721024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 0x55d626ff4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:25:45.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.777+0000 7ff390ba6700 1 --2- 172.21.15.1:0/3946721024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d627004630 0x55d626fef830 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7ff384013000 tx=0x7ff38400a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=613649356128f447 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:45.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.778+0000 7ff3827fc700 1 -- 172.21.15.1:0/3946721024 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff384001df0 con 0x55d627004630 2024-03-18T20:25:45.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.778+0000 7ff3827fc700 1 -- 172.21.15.1:0/3946721024 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff384003a00 con 0x55d627004630 2024-03-18T20:25:45.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.778+0000 7ff3827fc700 1 -- 172.21.15.1:0/3946721024 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff384003db0 con 0x55d627004630 2024-03-18T20:25:45.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.778+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/3946721024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d627004630 msgr2=0x55d626fef830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:45.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.778+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/3946721024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d627004630 0x55d626fef830 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7ff384013000 tx=0x7ff38400a000 comp rx=0 tx=0).stop 2024-03-18T20:25:45.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.778+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/3946721024 shutdown_connections 2024-03-18T20:25:45.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.778+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/3946721024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d627004630 0x55d626fef830 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.778+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/3946721024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 0x55d626ff4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.778+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/3946721024 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d626ff2c10 0x55d626fedd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.778+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/3946721024 >> 172.21.15.1:0/3946721024 conn(0x55d626f26cf0 msgr2=0x55d626fc8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.778+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/3946721024 shutdown_connections 2024-03-18T20:25:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.778+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/3946721024 wait complete. 2024-03-18T20:25:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.779+0000 7ff39e7b2b80 1 -- start start 2024-03-18T20:25:45.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.779+0000 7ff39e7b2b80 1 Processor -- start 2024-03-18T20:25:45.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff39e7b2b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 0x55d626ff1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff39e7b2b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d626ff2c10 0x55d626ff0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff39e7b2b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d627004630 0x55d626ff0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff39e7b2b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d626ff1760 con 0x55d626ff2c10 2024-03-18T20:25:45.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff39e7b2b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d626ff4650 con 0x55d626fc40a0 2024-03-18T20:25:45.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff39e7b2b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d626ff4120 con 0x55d627004630 2024-03-18T20:25:45.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff39e7b2b80 10 client.0 _ll_register_callbacks cb 0x7ff37400bbf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:25:45.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff38b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d626ff2c10 0x55d626ff0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff38b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d626ff2c10 0x55d626ff0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54156/0 (socket says 172.21.15.1:54156) 2024-03-18T20:25:45.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff38b7fe700 1 -- 172.21.15.1:0/1330047518 learned_addr learned my addr 172.21.15.1:0/1330047518 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:45.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff390ba6700 1 --2- 172.21.15.1:0/1330047518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d627004630 0x55d626ff0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff38bfff700 1 --2- 172.21.15.1:0/1330047518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 0x55d626ff1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff38b7fe700 1 -- 172.21.15.1:0/1330047518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d627004630 msgr2=0x55d626ff0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:45.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff38b7fe700 1 --2- 172.21.15.1:0/1330047518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d627004630 0x55d626ff0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff38b7fe700 1 -- 172.21.15.1:0/1330047518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 msgr2=0x55d626ff1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:45.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff38b7fe700 1 --2- 172.21.15.1:0/1330047518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 0x55d626ff1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff38b7fe700 1 -- 172.21.15.1:0/1330047518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d626d0c790 con 0x55d626ff2c10 2024-03-18T20:25:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff390ba6700 1 --2- 172.21.15.1:0/1330047518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d627004630 0x55d626ff0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff38b7fe700 1 --2- 172.21.15.1:0/1330047518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d626ff2c10 0x55d626ff0570 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7ff37400b0e0 tx=0x7ff3740092c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=739452b14f327a9c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.780+0000 7ff36affd700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T20:25:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.781+0000 7ff36affd700 1 -- 172.21.15.1:0/1330047518 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374039c40 con 0x55d626ff2c10 2024-03-18T20:25:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.781+0000 7ff36affd700 1 -- 172.21.15.1:0/1330047518 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(236..236 src has 1..236) v4 ==== 6993+0+0 (secure 0 0 0) 0x7ff374011420 con 0x55d626ff2c10 2024-03-18T20:25:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.781+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/1330047518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff38400edd0 con 0x55d626ff2c10 2024-03-18T20:25:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.781+0000 7ff36affd700 1 -- 172.21.15.1:0/1330047518 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3740115e0 con 0x55d626ff2c10 2024-03-18T20:25:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.781+0000 7ff39e7b2b80 10 client.11889 fetch_fsmap learned FSMap version 110 2024-03-18T20:25:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.781+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/1330047518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=2+}) v3 -- 0x55d626d6d1d0 con 0x55d626ff2c10 2024-03-18T20:25:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.781+0000 7ff36affd700 1 -- 172.21.15.1:0/1330047518 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38400edd0 con 0x55d626ff2c10 2024-03-18T20:25:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.781+0000 7ff36affd700 1 -- 172.21.15.1:0/1330047518 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 110) v1 ==== 3681+0+0 (secure 0 0 0) 0x7ff374004040 con 0x55d626ff2c10 2024-03-18T20:25:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.781+0000 7ff39e7b2b80 10 client.11889 fetch_fsmap finished waiting for FSMap version 110 2024-03-18T20:25:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.781+0000 7ff39e7b2b80 10 client.11889 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T20:25:45.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.781+0000 7ff39e7b2b80 20 client.11889 populate_metadata read hostname 'smithi001' 2024-03-18T20:25:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.781+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/1330047518 >> [v2:172.21.15.1:6835/1635684321,v1:172.21.15.1:6837/1635684321] conn(0x55d627077c40 0x55d627078020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.781+0000 7ff39e7b2b80 4 client.11889 mds_command: new command op to 11547 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T20:25:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.781+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/1330047518 --> [v2:172.21.15.1:6835/1635684321,v1:172.21.15.1:6837/1635684321] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d626d52040 con 0x55d627077c40 2024-03-18T20:25:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.781+0000 7ff38bfff700 1 --2- 172.21.15.1:0/1330047518 >> [v2:172.21.15.1:6835/1635684321,v1:172.21.15.1:6837/1635684321] conn(0x55d627077c40 0x55d627078020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.782+0000 7ff36affd700 10 client.11889 ms_handle_connect on v2:172.21.15.1:6835/1635684321 2024-03-18T20:25:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.782+0000 7ff38bfff700 1 --2- 172.21.15.1:0/1330047518 >> [v2:172.21.15.1:6835/1635684321,v1:172.21.15.1:6837/1635684321] conn(0x55d627077c40 0x55d627078020 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2b5c28b6b1b0773 server_cookie=ac2b1e1bd4e1c5f in_seq=0 out_seq=0 2024-03-18T20:25:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.782+0000 7ff36affd700 1 -- 172.21.15.1:0/1330047518 <== mds.0 v2:172.21.15.1:6835/1635684321 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x55d626d52040 con 0x55d627077c40 2024-03-18T20:25:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.782+0000 7ff36affd700 10 client.11889 handle_command_reply: tid=0 2024-03-18T20:25:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.782+0000 7ff39e7b2b80 1 client.11889 shutdown 2024-03-18T20:25:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.782+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/1330047518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d626ff2c10 msgr2=0x55d626ff0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.782+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/1330047518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d626ff2c10 0x55d626ff0570 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7ff37400b0e0 tx=0x7ff3740092c0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.782+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/1330047518 shutdown_connections 2024-03-18T20:25:45.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.782+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/1330047518 >> [v2:172.21.15.1:6835/1635684321,v1:172.21.15.1:6837/1635684321] conn(0x55d627077c40 0x55d627078020 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.782+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/1330047518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55d627004630 0x55d626ff0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.782+0000 7ff36affd700 0 client.11889 ms_handle_reset on v2:172.21.15.1:6835/1635684321 2024-03-18T20:25:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.782+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/1330047518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d626fc40a0 0x55d626ff1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.782+0000 7ff39e7b2b80 1 --2- 172.21.15.1:0/1330047518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d626ff2c10 0x55d626ff0570 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.782+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/1330047518 >> 172.21.15.1:0/1330047518 conn(0x55d626f26cf0 msgr2=0x55d626fef830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.782+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/1330047518 shutdown_connections 2024-03-18T20:25:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.782+0000 7ff39e7b2b80 1 -- 172.21.15.1:0/1330047518 wait complete. 2024-03-18T20:25:45.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.782+0000 7ff39e7b2b80 20 client.11889 trim_cache size 0 max 16384 2024-03-18T20:25:45.793 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:25:45.793 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "9c1c697f-8655-4ef3-8a13-da568d2cbe86", 2024-03-18T20:25:45.793 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-18T20:25:45.794 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 11547, 2024-03-18T20:25:45.794 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-18T20:25:45.794 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-18T20:25:45.794 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "backup_fs", 2024-03-18T20:25:45.794 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 10.231492867, 2024-03-18T20:25:45.794 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 110, 2024-03-18T20:25:45.794 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 236, 2024-03-18T20:25:45.794 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 236, 2024-03-18T20:25:45.794 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 28.439296183 2024-03-18T20:25:45.794 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:25:45.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.783+0000 7ff392609700 1 -- 172.21.15.1:0/3771531999 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff36c06adb0 msgr2=0x7ff36c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:45.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.783+0000 7ff392609700 1 --2- 172.21.15.1:0/3771531999 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff36c06adb0 0x7ff36c06d270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff374009840 tx=0x7ff374008030 comp rx=0 tx=0).stop 2024-03-18T20:25:45.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.783+0000 7ff392609700 1 -- 172.21.15.1:0/3771531999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff38c06c7b0 msgr2=0x7ff38c0c9280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:45.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.783+0000 7ff392609700 1 --2- 172.21.15.1:0/3771531999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff38c06c7b0 0x7ff38c0c9280 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00d800 tx=0x7ff37c00db10 comp rx=0 tx=0).stop 2024-03-18T20:25:45.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.783+0000 7ff392609700 1 -- 172.21.15.1:0/3771531999 shutdown_connections 2024-03-18T20:25:45.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.783+0000 7ff392609700 1 --2- 172.21.15.1:0/3771531999 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff36c06adb0 0x7ff36c06d270 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.783+0000 7ff392609700 1 --2- 172.21.15.1:0/3771531999 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff38c15d500 0x7ff38c0cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.783+0000 7ff392609700 1 --2- 172.21.15.1:0/3771531999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff38c06c7b0 0x7ff38c0c9280 unknown :-1 s=CLOSED pgs=2598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.783+0000 7ff392609700 1 --2- 172.21.15.1:0/3771531999 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff38c0cb7b0 0x7ff38c152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.783+0000 7ff392609700 1 -- 172.21.15.1:0/3771531999 >> 172.21.15.1:0/3771531999 conn(0x7ff38c0b2c50 msgr2=0x7ff38c0ae4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:45.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.783+0000 7ff392609700 1 -- 172.21.15.1:0/3771531999 shutdown_connections 2024-03-18T20:25:45.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.783+0000 7ff392609700 1 -- 172.21.15.1:0/3771531999 wait complete. 2024-03-18T20:25:45.800 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-18T20:25:45.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.921+0000 7f67a4f41700 1 Processor -- start 2024-03-18T20:25:45.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.921+0000 7f67a4f41700 1 -- start start 2024-03-18T20:25:45.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.922+0000 7f67a4f41700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67a0160a70 0x7f67a00d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.922+0000 7f67a4f41700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f67a01612d0 0x7f67a0161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.922+0000 7f67a4f41700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a0162170 con 0x7f67a0160e50 2024-03-18T20:25:45.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.922+0000 7f67a4f41700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67a0161c60 con 0x7f67a01612d0 2024-03-18T20:25:45.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.922+0000 7f67a4f41700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a00d06e0 con 0x7f67a0160a70 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.922+0000 7f679ed9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f67a01612d0 0x7f67a0161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.923+0000 7f679ed9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f67a01612d0 0x7f67a0161720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54160/0 (socket says 172.21.15.1:54160) 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.923+0000 7f679ed9d700 1 -- 172.21.15.1:0/135367492 learned_addr learned my addr 172.21.15.1:0/135367492 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.923+0000 7f679e59c700 1 --2- 172.21.15.1:0/135367492 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67a0160a70 0x7f67a00d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.923+0000 7f679ed9d700 1 -- 172.21.15.1:0/135367492 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67a0160a70 msgr2=0x7f67a00d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.923+0000 7f679ed9d700 1 --2- 172.21.15.1:0/135367492 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67a0160a70 0x7f67a00d3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.923+0000 7f679ed9d700 1 -- 172.21.15.1:0/135367492 >> v1:172.21.15.165:6789/0 conn(0x7f67a0160e50 legacy=0x7f67a00cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.923+0000 7f679ed9d700 1 -- 172.21.15.1:0/135367492 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a00cef30 con 0x7f67a01612d0 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.923+0000 7f679e59c700 1 --2- 172.21.15.1:0/135367492 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67a0160a70 0x7f67a00d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.923+0000 7f679ed9d700 1 --2- 172.21.15.1:0/135367492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a01612d0 0x7f67a0161720 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f67980098f0 tx=0x7f6798009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c855b063420b0399 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.923+0000 7f679d59a700 1 -- 172.21.15.1:0/135367492 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6798004230 con 0x7f67a01612d0 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.923+0000 7f679d59a700 1 -- 172.21.15.1:0/135367492 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6798004390 con 0x7f67a01612d0 2024-03-18T20:25:45.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.923+0000 7f679d59a700 1 -- 172.21.15.1:0/135367492 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67980036a0 con 0x7f67a01612d0 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.924+0000 7f67a4f41700 1 -- 172.21.15.1:0/135367492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a01612d0 msgr2=0x7f67a0161720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.924+0000 7f67a4f41700 1 --2- 172.21.15.1:0/135367492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a01612d0 0x7f67a0161720 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f67980098f0 tx=0x7f6798009c00 comp rx=0 tx=0).stop 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.924+0000 7f67a4f41700 1 -- 172.21.15.1:0/135367492 shutdown_connections 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.924+0000 7f67a4f41700 1 --2- 172.21.15.1:0/135367492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a01612d0 0x7f67a0161720 unknown :-1 s=CLOSED pgs=2603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.924+0000 7f67a4f41700 1 --2- 172.21.15.1:0/135367492 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67a0160a70 0x7f67a00d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.924+0000 7f67a4f41700 1 -- 172.21.15.1:0/135367492 >> 172.21.15.1:0/135367492 conn(0x7f67a009c560 msgr2=0x7f67a00d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.924+0000 7f67a4f41700 1 -- 172.21.15.1:0/135367492 shutdown_connections 2024-03-18T20:25:45.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.924+0000 7f67a4f41700 1 -- 172.21.15.1:0/135367492 wait complete. 2024-03-18T20:25:45.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.925+0000 7f67a4f41700 1 Processor -- start 2024-03-18T20:25:45.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.925+0000 7f67a4f41700 1 -- start start 2024-03-18T20:25:45.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.926+0000 7f67a4f41700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67a0160a70 0x7f67a01f6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.926+0000 7f67a4f41700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a0160e50 0x7f67a01f73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.926+0000 7f67a4f41700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67a01612d0 0x7f67a00b5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.926+0000 7f67a4f41700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a01f41f0 con 0x7f67a0160e50 2024-03-18T20:25:45.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.926+0000 7f67a4f41700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a00b8020 con 0x7f67a01612d0 2024-03-18T20:25:45.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.926+0000 7f67a4f41700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a00b8300 con 0x7f67a0160a70 2024-03-18T20:25:45.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.926+0000 7f679dd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a0160e50 0x7f67a01f73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.926+0000 7f679dd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a0160e50 0x7f67a01f73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54176/0 (socket says 172.21.15.1:54176) 2024-03-18T20:25:45.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.926+0000 7f679dd9b700 1 -- 172.21.15.1:0/4072021863 learned_addr learned my addr 172.21.15.1:0/4072021863 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:45.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.926+0000 7f679e59c700 1 --2- 172.21.15.1:0/4072021863 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67a0160a70 0x7f67a01f6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.926+0000 7f679ed9d700 1 --2- 172.21.15.1:0/4072021863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67a01612d0 0x7f67a00b5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.926+0000 7f679dd9b700 1 -- 172.21.15.1:0/4072021863 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67a0160a70 msgr2=0x7f67a01f6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:45.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.926+0000 7f679dd9b700 1 --2- 172.21.15.1:0/4072021863 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67a0160a70 0x7f67a01f6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.926+0000 7f679dd9b700 1 -- 172.21.15.1:0/4072021863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67a01612d0 msgr2=0x7f67a00b5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:45.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.926+0000 7f679dd9b700 1 --2- 172.21.15.1:0/4072021863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67a01612d0 0x7f67a00b5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:45.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.926+0000 7f679dd9b700 1 -- 172.21.15.1:0/4072021863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6798009550 con 0x7f67a0160e50 2024-03-18T20:25:45.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.927+0000 7f679dd9b700 1 --2- 172.21.15.1:0/4072021863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a0160e50 0x7f67a01f73d0 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f678800b450 tx=0x7f678800b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:45.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.927+0000 7f6796ffd700 1 -- 172.21.15.1:0/4072021863 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6788011830 con 0x7f67a0160e50 2024-03-18T20:25:45.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.927+0000 7f6796ffd700 1 -- 172.21.15.1:0/4072021863 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6788011990 con 0x7f67a0160e50 2024-03-18T20:25:45.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.927+0000 7f67a4f41700 1 -- 172.21.15.1:0/4072021863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a00b85e0 con 0x7f67a0160e50 2024-03-18T20:25:45.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.927+0000 7f6796ffd700 1 -- 172.21.15.1:0/4072021863 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6788011d20 con 0x7f67a0160e50 2024-03-18T20:25:45.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.927+0000 7f67a4f41700 1 -- 172.21.15.1:0/4072021863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a00b8de0 con 0x7f67a0160e50 2024-03-18T20:25:45.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.928+0000 7f67a4f41700 1 -- 172.21.15.1:0/4072021863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a00b8b00 con 0x7f67a0160e50 2024-03-18T20:25:45.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.928+0000 7f6796ffd700 1 -- 172.21.15.1:0/4072021863 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f678801a400 con 0x7f67a0160e50 2024-03-18T20:25:45.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.929+0000 7f6796ffd700 1 --2- 172.21.15.1:0/4072021863 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f678006adb0 0x7f678006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:45.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.929+0000 7f679e59c700 1 --2- 172.21.15.1:0/4072021863 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f678006adb0 0x7f678006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:45.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.929+0000 7f6796ffd700 1 -- 172.21.15.1:0/4072021863 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f6788089950 con 0x7f67a0160e50 2024-03-18T20:25:45.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.929+0000 7f679e59c700 1 --2- 172.21.15.1:0/4072021863 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f678006adb0 0x7f678006d270 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6790009fd0 tx=0x7f67900097f0 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:45.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:45.933+0000 7f6796ffd700 1 -- 172.21.15.1:0/4072021863 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f67a00b8b00 con 0x7f67a0160e50 2024-03-18T20:25:46.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.163+0000 7f67a4f41700 1 -- 172.21.15.1:0/4072021863 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f67a00cf7d0 con 0x7f678006adb0 2024-03-18T20:25:46.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.164+0000 7f6796ffd700 1 -- 172.21.15.1:0/4072021863 <== mgr.11322 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7f679000f040 con 0x7f678006adb0 2024-03-18T20:25:46.168 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T20:25:46.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:46.169 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephfs" 2024-03-18T20:25:46.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:25:46.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:25:46.169 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "backup_fs" 2024-03-18T20:25:46.169 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:25:46.169 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T20:25:46.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.166+0000 7f67a4f41700 1 -- 172.21.15.1:0/4072021863 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f678006adb0 msgr2=0x7f678006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:46.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.166+0000 7f67a4f41700 1 --2- 172.21.15.1:0/4072021863 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f678006adb0 0x7f678006d270 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6790009fd0 tx=0x7f67900097f0 comp rx=0 tx=0).stop 2024-03-18T20:25:46.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.166+0000 7f67a4f41700 1 -- 172.21.15.1:0/4072021863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a0160e50 msgr2=0x7f67a01f73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:46.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.166+0000 7f67a4f41700 1 --2- 172.21.15.1:0/4072021863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a0160e50 0x7f67a01f73d0 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f678800b450 tx=0x7f678800b760 comp rx=0 tx=0).stop 2024-03-18T20:25:46.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.166+0000 7f67a4f41700 1 -- 172.21.15.1:0/4072021863 shutdown_connections 2024-03-18T20:25:46.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.166+0000 7f67a4f41700 1 --2- 172.21.15.1:0/4072021863 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f678006adb0 0x7f678006d270 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:46.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.166+0000 7f67a4f41700 1 --2- 172.21.15.1:0/4072021863 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67a0160a70 0x7f67a01f6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:46.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.166+0000 7f67a4f41700 1 --2- 172.21.15.1:0/4072021863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67a01612d0 0x7f67a00b5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:46.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.166+0000 7f67a4f41700 1 --2- 172.21.15.1:0/4072021863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67a0160e50 0x7f67a01f73d0 unknown :-1 s=CLOSED pgs=2604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:46.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.166+0000 7f67a4f41700 1 -- 172.21.15.1:0/4072021863 >> 172.21.15.1:0/4072021863 conn(0x7f67a009c560 msgr2=0x7f67a00d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:46.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.166+0000 7f67a4f41700 1 -- 172.21.15.1:0/4072021863 shutdown_connections 2024-03-18T20:25:46.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.166+0000 7f67a4f41700 1 -- 172.21.15.1:0/4072021863 wait complete. 2024-03-18T20:25:46.182 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-18T20:25:46.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.307+0000 7f7ff0631700 1 Processor -- start 2024-03-18T20:25:46.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.307+0000 7f7ff0631700 1 -- start start 2024-03-18T20:25:46.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.308+0000 7f7ff0631700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7fe815d3e0 0x7f7fe80d0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:46.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.308+0000 7f7ff0631700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe815d7c0 0x7f7fe80d17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:46.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.308+0000 7f7ff0631700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe80ce1e0 con 0x7f7fe80d1cf0 2024-03-18T20:25:46.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.308+0000 7f7ff0631700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe80cee60 con 0x7f7fe815d3e0 2024-03-18T20:25:46.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.308+0000 7f7ff0631700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe80d2160 con 0x7f7fe815d7c0 2024-03-18T20:25:46.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.308+0000 7f7fedbcc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe815d7c0 0x7f7fe80d17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:46.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.308+0000 7f7fee3cd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7fe815d3e0 0x7f7fe80d0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:46.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.308+0000 7f7fedbcc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe815d7c0 0x7f7fe80d17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:39960/0 (socket says 172.21.15.1:39960) 2024-03-18T20:25:46.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.308+0000 7f7fedbcc700 1 -- 172.21.15.1:0/3897869428 learned_addr learned my addr 172.21.15.1:0/3897869428 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:46.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.308+0000 7f7fee3cd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7fe815d3e0 0x7f7fe80d0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:40268/0 (socket says 172.21.15.1:40268) 2024-03-18T20:25:46.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.309+0000 7f7fedbcc700 1 -- 172.21.15.1:0/3897869428 >> v2:172.21.15.165:3300/0 conn(0x7f7fe815d3e0 msgr2=0x7f7fe80d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:46.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.309+0000 7f7fedbcc700 1 --2- 172.21.15.1:0/3897869428 >> v2:172.21.15.165:3300/0 conn(0x7f7fe815d3e0 0x7f7fe80d0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:46.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.309+0000 7f7fedbcc700 1 -- 172.21.15.1:0/3897869428 >> v1:172.21.15.165:6789/0 conn(0x7f7fe80d1cf0 legacy=0x7f7fe80b82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:46.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.309+0000 7f7fedbcc700 1 -- 172.21.15.1:0/3897869428 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe80ce4c0 con 0x7f7fe815d7c0 2024-03-18T20:25:46.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.309+0000 7f7fedbcc700 1 --2- 172.21.15.1:0/3897869428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe815d7c0 0x7f7fe80d17b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009e50 tx=0x7f7fdc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63f22d49512f20dd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:46.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.309+0000 7f7fed3cb700 1 -- 172.21.15.1:0/3897869428 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc00e070 con 0x7f7fe815d7c0 2024-03-18T20:25:46.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.309+0000 7f7fed3cb700 1 -- 172.21.15.1:0/3897869428 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fdc00b910 con 0x7f7fe815d7c0 2024-03-18T20:25:46.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.309+0000 7f7fed3cb700 1 -- 172.21.15.1:0/3897869428 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc00bcc0 con 0x7f7fe815d7c0 2024-03-18T20:25:46.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.310+0000 7f7ff0631700 1 -- 172.21.15.1:0/3897869428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe815d7c0 msgr2=0x7f7fe80d17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:46.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.310+0000 7f7ff0631700 1 --2- 172.21.15.1:0/3897869428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe815d7c0 0x7f7fe80d17b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009e50 tx=0x7f7fdc00b330 comp rx=0 tx=0).stop 2024-03-18T20:25:46.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.310+0000 7f7ff0631700 1 -- 172.21.15.1:0/3897869428 shutdown_connections 2024-03-18T20:25:46.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.310+0000 7f7ff0631700 1 --2- 172.21.15.1:0/3897869428 >> v2:172.21.15.165:3300/0 conn(0x7f7fe815d3e0 0x7f7fe80d0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:46.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.310+0000 7f7ff0631700 1 --2- 172.21.15.1:0/3897869428 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe815d7c0 0x7f7fe80d17b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:46.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.310+0000 7f7ff0631700 1 -- 172.21.15.1:0/3897869428 >> 172.21.15.1:0/3897869428 conn(0x7f7fe809c540 msgr2=0x7f7fe80d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:46.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.310+0000 7f7ff0631700 1 -- 172.21.15.1:0/3897869428 shutdown_connections 2024-03-18T20:25:46.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.310+0000 7f7ff0631700 1 -- 172.21.15.1:0/3897869428 wait complete. 2024-03-18T20:25:46.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.311+0000 7f7ff0631700 1 Processor -- start 2024-03-18T20:25:46.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.311+0000 7f7ff0631700 1 -- start start 2024-03-18T20:25:46.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.312+0000 7f7ff0631700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe80d1cf0 0x7f7fe80c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:46.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.312+0000 7f7ff0631700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fe815d3e0 0x7f7fe81570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:46.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.312+0000 7f7ff0631700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe81482a0 0x7f7fe8148730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:46.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.312+0000 7f7ff0631700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe80c8810 con 0x7f7fe80d1cf0 2024-03-18T20:25:46.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.312+0000 7f7ff0631700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe80c9650 con 0x7f7fe815d3e0 2024-03-18T20:25:46.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.312+0000 7f7ff0631700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe80c8a40 con 0x7f7fe81482a0 2024-03-18T20:25:46.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.312+0000 7f7fee3cd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe80d1cf0 0x7f7fe80c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:46.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.312+0000 7f7fee3cd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe80d1cf0 0x7f7fe80c82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54190/0 (socket says 172.21.15.1:54190) 2024-03-18T20:25:46.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.312+0000 7f7fee3cd700 1 -- 172.21.15.1:0/1613391384 learned_addr learned my addr 172.21.15.1:0/1613391384 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:46.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.313+0000 7f7fedbcc700 1 --2- 172.21.15.1:0/1613391384 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fe815d3e0 0x7f7fe81570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:46.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.313+0000 7f7feebce700 1 --2- 172.21.15.1:0/1613391384 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe81482a0 0x7f7fe8148730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:46.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.313+0000 7f7fee3cd700 1 -- 172.21.15.1:0/1613391384 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe81482a0 msgr2=0x7f7fe8148730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:46.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.313+0000 7f7fee3cd700 1 --2- 172.21.15.1:0/1613391384 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe81482a0 0x7f7fe8148730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:46.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.313+0000 7f7fee3cd700 1 -- 172.21.15.1:0/1613391384 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fe815d3e0 msgr2=0x7f7fe81570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:46.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.313+0000 7f7fee3cd700 1 --2- 172.21.15.1:0/1613391384 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fe815d3e0 0x7f7fe81570c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:46.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.313+0000 7f7fee3cd700 1 -- 172.21.15.1:0/1613391384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fdc009b30 con 0x7f7fe80d1cf0 2024-03-18T20:25:46.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.313+0000 7f7feebce700 1 --2- 172.21.15.1:0/1613391384 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe81482a0 0x7f7fe8148730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:46.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.313+0000 7f7fee3cd700 1 --2- 172.21.15.1:0/1613391384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe80d1cf0 0x7f7fe80c82d0 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f7fe000e9b0 tx=0x7f7fe000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:46.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.313+0000 7f7fe6ffd700 1 -- 172.21.15.1:0/1613391384 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe0009710 con 0x7f7fe80d1cf0 2024-03-18T20:25:46.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.313+0000 7f7fe6ffd700 1 -- 172.21.15.1:0/1613391384 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe000c850 con 0x7f7fe80d1cf0 2024-03-18T20:25:46.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.313+0000 7f7ff0631700 1 -- 172.21.15.1:0/1613391384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe80c8d20 con 0x7f7fe80d1cf0 2024-03-18T20:25:46.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.313+0000 7f7fe6ffd700 1 -- 172.21.15.1:0/1613391384 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe000cbc0 con 0x7f7fe80d1cf0 2024-03-18T20:25:46.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.313+0000 7f7ff0631700 1 -- 172.21.15.1:0/1613391384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe80c92e0 con 0x7f7fe80d1cf0 2024-03-18T20:25:46.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.314+0000 7f7ff0631700 1 -- 172.21.15.1:0/1613391384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe8073350 con 0x7f7fe80d1cf0 2024-03-18T20:25:46.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.315+0000 7f7fe6ffd700 1 -- 172.21.15.1:0/1613391384 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 135) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7fe000cd20 con 0x7f7fe80d1cf0 2024-03-18T20:25:46.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.316+0000 7f7fe6ffd700 1 --2- 172.21.15.1:0/1613391384 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fd406adb0 0x7f7fd406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:46.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.316+0000 7f7fe6ffd700 1 -- 172.21.15.1:0/1613391384 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f7fe008a240 con 0x7f7fe80d1cf0 2024-03-18T20:25:46.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.316+0000 7f7fedbcc700 1 --2- 172.21.15.1:0/1613391384 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fd406adb0 0x7f7fd406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:46.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.316+0000 7f7fedbcc700 1 --2- 172.21.15.1:0/1613391384 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fd406adb0 0x7f7fd406d270 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc000f80 tx=0x7f7fdc009000 comp rx=0 tx=0).ready entity=mgr.11322 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:46.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.320+0000 7f7fe6ffd700 1 -- 172.21.15.1:0/1613391384 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7fe0054de0 con 0x7f7fe80d1cf0 2024-03-18T20:25:46.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:46.581+0000 7f7ff0631700 1 -- 172.21.15.1:0/1613391384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f7fe80d08c0 con 0x7f7fe80d1cf0 2024-03-18T20:25:47.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.172+0000 7f7fe6ffd700 1 -- 172.21.15.1:0/1613391384 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 136) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f7fe00551b0 con 0x7f7fe80d1cf0 2024-03-18T20:25:47.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.173+0000 7f7fe6ffd700 1 -- 172.21.15.1:0/1613391384 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v136) v1 ==== 92+0+0 (secure 0 0 0) 0x7f7fe0058870 con 0x7f7fe80d1cf0 2024-03-18T20:25:47.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.176+0000 7f7ff0631700 1 -- 172.21.15.1:0/1613391384 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fd406adb0 msgr2=0x7f7fd406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:47.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.176+0000 7f7ff0631700 1 --2- 172.21.15.1:0/1613391384 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fd406adb0 0x7f7fd406d270 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc000f80 tx=0x7f7fdc009000 comp rx=0 tx=0).stop 2024-03-18T20:25:47.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.176+0000 7f7ff0631700 1 -- 172.21.15.1:0/1613391384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe80d1cf0 msgr2=0x7f7fe80c82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:47.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.176+0000 7f7ff0631700 1 --2- 172.21.15.1:0/1613391384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe80d1cf0 0x7f7fe80c82d0 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f7fe000e9b0 tx=0x7f7fe000ecc0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.176+0000 7f7ff0631700 1 -- 172.21.15.1:0/1613391384 shutdown_connections 2024-03-18T20:25:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.176+0000 7f7ff0631700 1 --2- 172.21.15.1:0/1613391384 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fd406adb0 0x7f7fd406d270 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.176+0000 7f7ff0631700 1 --2- 172.21.15.1:0/1613391384 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fe81482a0 0x7f7fe8148730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.176+0000 7f7ff0631700 1 --2- 172.21.15.1:0/1613391384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe80d1cf0 0x7f7fe80c82d0 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.176+0000 7f7ff0631700 1 --2- 172.21.15.1:0/1613391384 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fe815d3e0 0x7f7fe81570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.176+0000 7f7ff0631700 1 -- 172.21.15.1:0/1613391384 >> 172.21.15.1:0/1613391384 conn(0x7f7fe809c540 msgr2=0x7f7fe80cf4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.176+0000 7f7ff0631700 1 -- 172.21.15.1:0/1613391384 shutdown_connections 2024-03-18T20:25:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.176+0000 7f7ff0631700 1 -- 172.21.15.1:0/1613391384 wait complete. 2024-03-18T20:25:47.194 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-18T20:25:47.320 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:47.316+0000 7f17bbc35240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:25:47.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.325+0000 7f93c32b6700 1 Processor -- start 2024-03-18T20:25:47.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.325+0000 7f93c32b6700 1 -- start start 2024-03-18T20:25:47.330 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:47.326+0000 7f41d28b5240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:25:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.326+0000 7f93c32b6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f93bc09dd40 0x7f93bc0cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.326+0000 7f93c32b6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f93bc0ce5a0 0x7f93bc0ce9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.326+0000 7f93c32b6700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93bc0ce090 con 0x7f93bc15d780 2024-03-18T20:25:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.326+0000 7f93c32b6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93bc0cb480 con 0x7f93bc09dd40 2024-03-18T20:25:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.326+0000 7f93c32b6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93bc06db30 con 0x7f93bc0ce5a0 2024-03-18T20:25:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.327+0000 7f93c0851700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f93bc09dd40 0x7f93bc0cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.327+0000 7f93c1052700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f93bc15d780 0x7f93bc0cad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45238/0 (socket says 172.21.15.1:45238) 2024-03-18T20:25:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.327+0000 7f93c1052700 1 -- 172.21.15.1:0/3648927934 learned_addr learned my addr 172.21.15.1:0/3648927934 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.327+0000 7f93c1853700 1 --2- 172.21.15.1:0/3648927934 >> v2:172.21.15.165:3300/0 conn(0x7f93bc0ce5a0 0x7f93bc0ce9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.327+0000 7f93c0851700 1 -- 172.21.15.1:0/3648927934 >> v2:172.21.15.165:3300/0 conn(0x7f93bc0ce5a0 msgr2=0x7f93bc0ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.327+0000 7f93c0851700 1 --2- 172.21.15.1:0/3648927934 >> v2:172.21.15.165:3300/0 conn(0x7f93bc0ce5a0 0x7f93bc0ce9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.327+0000 7f93c0851700 1 -- 172.21.15.1:0/3648927934 >> v1:172.21.15.1:6789/0 conn(0x7f93bc15d780 legacy=0x7f93bc0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.327+0000 7f93b3fff700 1 -- 172.21.15.1:0/3648927934 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162690224 0 0) 0x7f93bc0ce090 con 0x7f93bc15d780 2024-03-18T20:25:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.327+0000 7f93c0851700 1 -- 172.21.15.1:0/3648927934 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93bc06d520 con 0x7f93bc09dd40 2024-03-18T20:25:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.327+0000 7f93c0851700 1 --2- 172.21.15.1:0/3648927934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93bc09dd40 0x7f93bc0cdb50 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f93bc15ed20 tx=0x7f93ac009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d6271723e85c5d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.327+0000 7f93b3fff700 1 -- 172.21.15.1:0/3648927934 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac011830 con 0x7f93bc09dd40 2024-03-18T20:25:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.327+0000 7f93b3fff700 1 -- 172.21.15.1:0/3648927934 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ac011990 con 0x7f93bc09dd40 2024-03-18T20:25:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.327+0000 7f93b3fff700 1 -- 172.21.15.1:0/3648927934 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac011d20 con 0x7f93bc09dd40 2024-03-18T20:25:47.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.328+0000 7f93c32b6700 1 -- 172.21.15.1:0/3648927934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93bc09dd40 msgr2=0x7f93bc0cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:47.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.328+0000 7f93c32b6700 1 --2- 172.21.15.1:0/3648927934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93bc09dd40 0x7f93bc0cdb50 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f93bc15ed20 tx=0x7f93ac009b20 comp rx=0 tx=0).stop 2024-03-18T20:25:47.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.328+0000 7f93c32b6700 1 -- 172.21.15.1:0/3648927934 shutdown_connections 2024-03-18T20:25:47.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.328+0000 7f93c32b6700 1 --2- 172.21.15.1:0/3648927934 >> v2:172.21.15.165:3300/0 conn(0x7f93bc0ce5a0 0x7f93bc0ce9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.328+0000 7f93c32b6700 1 --2- 172.21.15.1:0/3648927934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93bc09dd40 0x7f93bc0cdb50 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.328+0000 7f93c32b6700 1 -- 172.21.15.1:0/3648927934 >> 172.21.15.1:0/3648927934 conn(0x7f93bc0cca00 msgr2=0x7f93bc0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:47.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.328+0000 7f93c32b6700 1 -- 172.21.15.1:0/3648927934 shutdown_connections 2024-03-18T20:25:47.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.328+0000 7f93c32b6700 1 -- 172.21.15.1:0/3648927934 wait complete. 2024-03-18T20:25:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.329+0000 7f93c32b6700 1 Processor -- start 2024-03-18T20:25:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.329+0000 7f93c32b6700 1 -- start start 2024-03-18T20:25:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.330+0000 7f93c32b6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93bc09dd40 0x7f93bc1f65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.330+0000 7f93c32b6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93bc0ce5a0 0x7f93bc155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:47.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.330+0000 7f93c32b6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93bc15d780 0x7f93bc1f3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.330+0000 7f93c32b6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc1f7540 con 0x7f93bc15d780 2024-03-18T20:25:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.330+0000 7f93c32b6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc1f3f80 con 0x7f93bc0ce5a0 2024-03-18T20:25:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.330+0000 7f93c32b6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc0c3ce0 con 0x7f93bc09dd40 2024-03-18T20:25:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.330+0000 7f93c1853700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93bc15d780 0x7f93bc1f3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.330+0000 7f93c1853700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93bc15d780 0x7f93bc1f3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54216/0 (socket says 172.21.15.1:54216) 2024-03-18T20:25:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.330+0000 7f93c1853700 1 -- 172.21.15.1:0/459788383 learned_addr learned my addr 172.21.15.1:0/459788383 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.330+0000 7f93c0851700 1 --2- 172.21.15.1:0/459788383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93bc0ce5a0 0x7f93bc155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.330+0000 7f93c1052700 1 --2- 172.21.15.1:0/459788383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93bc09dd40 0x7f93bc1f65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.330+0000 7f93c1853700 1 -- 172.21.15.1:0/459788383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93bc09dd40 msgr2=0x7f93bc1f65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.331+0000 7f93c1853700 1 --2- 172.21.15.1:0/459788383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93bc09dd40 0x7f93bc1f65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.331+0000 7f93c1853700 1 -- 172.21.15.1:0/459788383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93bc0ce5a0 msgr2=0x7f93bc155a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:47.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.331+0000 7f93c1853700 1 --2- 172.21.15.1:0/459788383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93bc0ce5a0 0x7f93bc155a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.331+0000 7f93c1853700 1 -- 172.21.15.1:0/459788383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ac0094a0 con 0x7f93bc15d780 2024-03-18T20:25:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.331+0000 7f93c1853700 1 --2- 172.21.15.1:0/459788383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93bc15d780 0x7f93bc1f3a40 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f93b400efc0 tx=0x7f93b400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.331+0000 7f93b1ffb700 1 -- 172.21.15.1:0/459788383 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b400cc30 con 0x7f93bc15d780 2024-03-18T20:25:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.331+0000 7f93b1ffb700 1 -- 172.21.15.1:0/459788383 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b400f040 con 0x7f93bc15d780 2024-03-18T20:25:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.331+0000 7f93b1ffb700 1 -- 172.21.15.1:0/459788383 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b4009e40 con 0x7f93bc15d780 2024-03-18T20:25:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.331+0000 7f93c32b6700 1 -- 172.21.15.1:0/459788383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93bc1f6ca0 con 0x7f93bc15d780 2024-03-18T20:25:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.331+0000 7f93c32b6700 1 -- 172.21.15.1:0/459788383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93bc1f7260 con 0x7f93bc15d780 2024-03-18T20:25:47.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.332+0000 7f93c32b6700 1 -- 172.21.15.1:0/459788383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93bc072250 con 0x7f93bc15d780 2024-03-18T20:25:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.333+0000 7f93b1ffb700 1 -- 172.21.15.1:0/459788383 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 136) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f93b4014480 con 0x7f93bc15d780 2024-03-18T20:25:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.333+0000 7f93b1ffb700 1 --2- 172.21.15.1:0/459788383 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93a006ae00 0x7f93a006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.333+0000 7f93c1052700 1 -- 172.21.15.1:0/459788383 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93a006ae00 msgr2=0x7f93a006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.333+0000 7f93b1ffb700 1 -- 172.21.15.1:0/459788383 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f93b408a300 con 0x7f93bc15d780 2024-03-18T20:25:47.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.333+0000 7f93c1052700 1 --2- 172.21.15.1:0/459788383 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93a006ae00 0x7f93a006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:47.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.338+0000 7f93b1ffb700 1 -- 172.21.15.1:0/459788383 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f93b4054db0 con 0x7f93bc15d780 2024-03-18T20:25:47.468 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:47.463+0000 7f17bbc35240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:25:47.499 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:47.495+0000 7f41d28b5240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:25:47.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.534+0000 7f93c1052700 1 -- 172.21.15.1:0/459788383 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93a006ae00 msgr2=0x7f93a006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:47.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.534+0000 7f93c1052700 1 --2- 172.21.15.1:0/459788383 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93a006ae00 0x7f93a006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:47.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.588+0000 7f93c32b6700 1 -- 172.21.15.1:0/459788383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f93bc070e00 con 0x7f93bc15d780 2024-03-18T20:25:47.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.592+0000 7f93b1ffb700 1 -- 172.21.15.1:0/459788383 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v77)=0 v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7f93b40583d0 con 0x7f93bc15d780 2024-03-18T20:25:47.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.594+0000 7f93c32b6700 1 -- 172.21.15.1:0/459788383 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93a006ae00 msgr2=0x7f93a006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.594+0000 7f93c32b6700 1 --2- 172.21.15.1:0/459788383 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93a006ae00 0x7f93a006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.594+0000 7f93c32b6700 1 -- 172.21.15.1:0/459788383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93bc15d780 msgr2=0x7f93bc1f3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.594+0000 7f93c32b6700 1 --2- 172.21.15.1:0/459788383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93bc15d780 0x7f93bc1f3a40 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f93b400efc0 tx=0x7f93b400c3a0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.594+0000 7f93c32b6700 1 -- 172.21.15.1:0/459788383 shutdown_connections 2024-03-18T20:25:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.594+0000 7f93c32b6700 1 --2- 172.21.15.1:0/459788383 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93a006ae00 0x7f93a006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.594+0000 7f93c32b6700 1 --2- 172.21.15.1:0/459788383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93bc09dd40 0x7f93bc1f65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.594+0000 7f93c32b6700 1 --2- 172.21.15.1:0/459788383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93bc15d780 0x7f93bc1f3a40 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.594+0000 7f93c32b6700 1 --2- 172.21.15.1:0/459788383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93bc0ce5a0 0x7f93bc155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.594+0000 7f93c32b6700 1 -- 172.21.15.1:0/459788383 >> 172.21.15.1:0/459788383 conn(0x7f93bc0cca00 msgr2=0x7f93bc071500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.594+0000 7f93c32b6700 1 -- 172.21.15.1:0/459788383 shutdown_connections 2024-03-18T20:25:47.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.594+0000 7f93c32b6700 1 -- 172.21.15.1:0/459788383 wait complete. 2024-03-18T20:25:47.611 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-18T20:25:47.734 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:47.730+0000 7f17bbc35240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:25:47.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.736+0000 7f6343704700 1 Processor -- start 2024-03-18T20:25:47.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.736+0000 7f6343704700 1 -- start start 2024-03-18T20:25:47.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.737+0000 7f6343704700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f633c15b6c0 0x7f633c0b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:47.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.737+0000 7f6343704700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f633c0c87d0 0x7f633c0cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:47.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.737+0000 7f6343704700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f633c0d01f0 con 0x7f633c0cd430 2024-03-18T20:25:47.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.737+0000 7f6343704700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f633c0ccfa0 con 0x7f633c0c87d0 2024-03-18T20:25:47.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.737+0000 7f6343704700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f633c0d1db0 con 0x7f633c15b6c0 2024-03-18T20:25:47.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.737+0000 7f6341ca1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f633c0c87d0 0x7f633c0cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:47.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.737+0000 7f6340c9f700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f633c0cd430 0x7f633c0c80c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45254/0 (socket says 172.21.15.1:45254) 2024-03-18T20:25:47.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.737+0000 7f6340c9f700 1 -- 172.21.15.1:0/1991556643 learned_addr learned my addr 172.21.15.1:0/1991556643 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:47.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.737+0000 7f63414a0700 1 --2- 172.21.15.1:0/1991556643 >> v2:172.21.15.165:3300/0 conn(0x7f633c15b6c0 0x7f633c0b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:47.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.737+0000 7f6341ca1700 1 -- 172.21.15.1:0/1991556643 >> v2:172.21.15.165:3300/0 conn(0x7f633c15b6c0 msgr2=0x7f633c0b2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:47.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.737+0000 7f6341ca1700 1 --2- 172.21.15.1:0/1991556643 >> v2:172.21.15.165:3300/0 conn(0x7f633c15b6c0 0x7f633c0b2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.737+0000 7f6341ca1700 1 -- 172.21.15.1:0/1991556643 >> v1:172.21.15.1:6789/0 conn(0x7f633c0cd430 legacy=0x7f633c0c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:47.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.737+0000 7f6341ca1700 1 -- 172.21.15.1:0/1991556643 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633c0cff10 con 0x7f633c0c87d0 2024-03-18T20:25:47.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.737+0000 7f6333fff700 1 -- 172.21.15.1:0/1991556643 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083529388 0 0) 0x7f633c0d01f0 con 0x7f633c0cd430 2024-03-18T20:25:47.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.738+0000 7f6341ca1700 1 --2- 172.21.15.1:0/1991556643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f633c0c87d0 0x7f633c0cd9e0 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f633c071890 tx=0x7f633400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0db3db6902c32f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:47.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.738+0000 7f6333fff700 1 -- 172.21.15.1:0/1991556643 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633400e070 con 0x7f633c0c87d0 2024-03-18T20:25:47.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.738+0000 7f6333fff700 1 -- 172.21.15.1:0/1991556643 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f633400bb20 con 0x7f633c0c87d0 2024-03-18T20:25:47.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.738+0000 7f6333fff700 1 -- 172.21.15.1:0/1991556643 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6334012430 con 0x7f633c0c87d0 2024-03-18T20:25:47.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.738+0000 7f6343704700 1 -- 172.21.15.1:0/1991556643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f633c0c87d0 msgr2=0x7f633c0cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:47.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.738+0000 7f6343704700 1 --2- 172.21.15.1:0/1991556643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f633c0c87d0 0x7f633c0cd9e0 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f633c071890 tx=0x7f633400b330 comp rx=0 tx=0).stop 2024-03-18T20:25:47.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.738+0000 7f6343704700 1 -- 172.21.15.1:0/1991556643 shutdown_connections 2024-03-18T20:25:47.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.738+0000 7f6343704700 1 --2- 172.21.15.1:0/1991556643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f633c0c87d0 0x7f633c0cd9e0 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.738+0000 7f6343704700 1 --2- 172.21.15.1:0/1991556643 >> v2:172.21.15.165:3300/0 conn(0x7f633c15b6c0 0x7f633c0b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.738+0000 7f6343704700 1 -- 172.21.15.1:0/1991556643 >> 172.21.15.1:0/1991556643 conn(0x7f633c0d0ff0 msgr2=0x7f633c0d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:47.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.738+0000 7f6343704700 1 -- 172.21.15.1:0/1991556643 shutdown_connections 2024-03-18T20:25:47.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.738+0000 7f6343704700 1 -- 172.21.15.1:0/1991556643 wait complete. 2024-03-18T20:25:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.740+0000 7f6343704700 1 Processor -- start 2024-03-18T20:25:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.740+0000 7f6343704700 1 -- start start 2024-03-18T20:25:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.740+0000 7f6343704700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f633c0c87d0 0x7f633c09e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.740+0000 7f6343704700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f633c0cd430 0x7f633c0c6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.740+0000 7f6343704700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f633c15b6c0 0x7f633c0b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.740+0000 7f6343704700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633c06ca00 con 0x7f633c15b6c0 2024-03-18T20:25:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.740+0000 7f6343704700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633c09ea20 con 0x7f633c0cd430 2024-03-18T20:25:47.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.740+0000 7f6343704700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f633c09ed00 con 0x7f633c0c87d0 2024-03-18T20:25:47.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.740+0000 7f6341ca1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f633c15b6c0 0x7f633c0b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:47.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.741+0000 7f6341ca1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f633c15b6c0 0x7f633c0b5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54234/0 (socket says 172.21.15.1:54234) 2024-03-18T20:25:47.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.741+0000 7f6341ca1700 1 -- 172.21.15.1:0/173526516 learned_addr learned my addr 172.21.15.1:0/173526516 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:47.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.741+0000 7f6340c9f700 1 --2- 172.21.15.1:0/173526516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f633c0cd430 0x7f633c0c6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:47.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.741+0000 7f63414a0700 1 --2- 172.21.15.1:0/173526516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f633c0c87d0 0x7f633c09e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:47.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.741+0000 7f6341ca1700 1 -- 172.21.15.1:0/173526516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f633c0c87d0 msgr2=0x7f633c09e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:47.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.741+0000 7f6341ca1700 1 --2- 172.21.15.1:0/173526516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f633c0c87d0 0x7f633c09e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.741+0000 7f6341ca1700 1 -- 172.21.15.1:0/173526516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f633c0cd430 msgr2=0x7f633c0c6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:47.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.741+0000 7f6341ca1700 1 --2- 172.21.15.1:0/173526516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f633c0cd430 0x7f633c0c6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.741+0000 7f6341ca1700 1 -- 172.21.15.1:0/173526516 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6334009a70 con 0x7f633c15b6c0 2024-03-18T20:25:47.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.741+0000 7f63414a0700 1 --2- 172.21.15.1:0/173526516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f633c0c87d0 0x7f633c09e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.741+0000 7f6340c9f700 1 --2- 172.21.15.1:0/173526516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f633c0cd430 0x7f633c0c6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.741+0000 7f6341ca1700 1 --2- 172.21.15.1:0/173526516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f633c15b6c0 0x7f633c0b5b20 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f633c072000 tx=0x7f6334004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.741+0000 7f6331ffb700 1 -- 172.21.15.1:0/173526516 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6334012430 con 0x7f633c15b6c0 2024-03-18T20:25:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.741+0000 7f6331ffb700 1 -- 172.21.15.1:0/173526516 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6334020070 con 0x7f633c15b6c0 2024-03-18T20:25:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.741+0000 7f6343704700 1 -- 172.21.15.1:0/173526516 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f633c06c3b0 con 0x7f633c15b6c0 2024-03-18T20:25:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.741+0000 7f6331ffb700 1 -- 172.21.15.1:0/173526516 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6334004280 con 0x7f633c15b6c0 2024-03-18T20:25:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.741+0000 7f6343704700 1 -- 172.21.15.1:0/173526516 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f633c06ce90 con 0x7f633c15b6c0 2024-03-18T20:25:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.743+0000 7f6343704700 1 -- 172.21.15.1:0/173526516 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f633c06cb70 con 0x7f633c15b6c0 2024-03-18T20:25:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.743+0000 7f6331ffb700 1 -- 172.21.15.1:0/173526516 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 136) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f63340043e0 con 0x7f633c15b6c0 2024-03-18T20:25:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.743+0000 7f6331ffb700 1 --2- 172.21.15.1:0/173526516 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f632006ae00 0x7f632006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.743+0000 7f63414a0700 1 -- 172.21.15.1:0/173526516 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f632006ae00 msgr2=0x7f632006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:47.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.744+0000 7f6331ffb700 1 -- 172.21.15.1:0/173526516 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f6334022070 con 0x7f633c15b6c0 2024-03-18T20:25:47.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.744+0000 7f63414a0700 1 --2- 172.21.15.1:0/173526516 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f632006ae00 0x7f632006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:47.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.748+0000 7f6331ffb700 1 -- 172.21.15.1:0/173526516 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f63340567b0 con 0x7f633c15b6c0 2024-03-18T20:25:47.783 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:47.779+0000 7f41d28b5240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:25:47.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.945+0000 7f63414a0700 1 -- 172.21.15.1:0/173526516 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f632006ae00 msgr2=0x7f632006d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:47.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.945+0000 7f63414a0700 1 --2- 172.21.15.1:0/173526516 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f632006ae00 0x7f632006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:47.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.951+0000 7f6343704700 1 -- 172.21.15.1:0/173526516 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f633c0676a0 con 0x7f633c15b6c0 2024-03-18T20:25:47.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.955+0000 7f6331ffb700 1 -- 172.21.15.1:0/173526516 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v78)=0 v78) v1 ==== 133+0+0 (secure 0 0 0) 0x7f6334059dd0 con 0x7f633c15b6c0 2024-03-18T20:25:47.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.959+0000 7f6343704700 1 -- 172.21.15.1:0/173526516 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f632006ae00 msgr2=0x7f632006d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:47.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.959+0000 7f6343704700 1 --2- 172.21.15.1:0/173526516 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f632006ae00 0x7f632006d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.959+0000 7f6343704700 1 -- 172.21.15.1:0/173526516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f633c15b6c0 msgr2=0x7f633c0b5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:47.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.959+0000 7f6343704700 1 --2- 172.21.15.1:0/173526516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f633c15b6c0 0x7f633c0b5b20 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f633c072000 tx=0x7f6334004ad0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.959+0000 7f6343704700 1 -- 172.21.15.1:0/173526516 shutdown_connections 2024-03-18T20:25:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.959+0000 7f6343704700 1 --2- 172.21.15.1:0/173526516 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f632006ae00 0x7f632006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.959+0000 7f6343704700 1 --2- 172.21.15.1:0/173526516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f633c0c87d0 0x7f633c09e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.959+0000 7f6343704700 1 --2- 172.21.15.1:0/173526516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f633c15b6c0 0x7f633c0b5b20 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.959+0000 7f6343704700 1 --2- 172.21.15.1:0/173526516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f633c0cd430 0x7f633c0c6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.959+0000 7f6343704700 1 -- 172.21.15.1:0/173526516 >> 172.21.15.1:0/173526516 conn(0x7f633c0d0ff0 msgr2=0x7f633c06fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.959+0000 7f6343704700 1 -- 172.21.15.1:0/173526516 shutdown_connections 2024-03-18T20:25:47.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:47.959+0000 7f6343704700 1 -- 172.21.15.1:0/173526516 wait complete. 2024-03-18T20:25:47.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:47.976 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2024-03-18T20:25:47.998 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-03-18T20:25:48.164 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:48.159+0000 7f17bbc35240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:25:48.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.168+0000 7fdc9dc76700 1 Processor -- start 2024-03-18T20:25:48.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.169+0000 7fdc9dc76700 1 -- start start 2024-03-18T20:25:48.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.169+0000 7fdc9dc76700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdc9815af20 0x7fdc98072480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:48.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.169+0000 7fdc9dc76700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdc980af6b0 0x7fdc9815dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:48.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.169+0000 7fdc9dc76700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9815e860 con 0x7fdc9815b300 2024-03-18T20:25:48.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.169+0000 7fdc9dc76700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc980719a0 con 0x7fdc9815af20 2024-03-18T20:25:48.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.169+0000 7fdc9dc76700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc98071c80 con 0x7fdc980af6b0 2024-03-18T20:25:48.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.170+0000 7fdc977fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdc9815af20 0x7fdc98072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:48.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.170+0000 7fdc977fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdc9815af20 0x7fdc98072480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54248/0 (socket says 172.21.15.1:54248) 2024-03-18T20:25:48.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.170+0000 7fdc977fe700 1 -- 172.21.15.1:0/2568109374 learned_addr learned my addr 172.21.15.1:0/2568109374 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:48.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.170+0000 7fdc97fff700 1 --2- 172.21.15.1:0/2568109374 >> v2:172.21.15.165:3300/0 conn(0x7fdc980af6b0 0x7fdc9815dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:48.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.170+0000 7fdc977fe700 1 -- 172.21.15.1:0/2568109374 >> v2:172.21.15.165:3300/0 conn(0x7fdc980af6b0 msgr2=0x7fdc9815dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:48.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.170+0000 7fdc977fe700 1 --2- 172.21.15.1:0/2568109374 >> v2:172.21.15.165:3300/0 conn(0x7fdc980af6b0 0x7fdc9815dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:48.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.170+0000 7fdc977fe700 1 -- 172.21.15.1:0/2568109374 >> v1:172.21.15.165:6789/0 conn(0x7fdc9815b300 legacy=0x7fdc980aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:48.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.170+0000 7fdc977fe700 1 -- 172.21.15.1:0/2568109374 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc98071f60 con 0x7fdc9815af20 2024-03-18T20:25:48.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.170+0000 7fdc977fe700 1 --2- 172.21.15.1:0/2568109374 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9815af20 0x7fdc98072480 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7fdc880098f0 tx=0x7fdc88009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e90493b532ec1279 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:48.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.170+0000 7fdc96ffd700 1 -- 172.21.15.1:0/2568109374 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88011830 con 0x7fdc9815af20 2024-03-18T20:25:48.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.170+0000 7fdc96ffd700 1 -- 172.21.15.1:0/2568109374 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc88011990 con 0x7fdc9815af20 2024-03-18T20:25:48.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.170+0000 7fdc96ffd700 1 -- 172.21.15.1:0/2568109374 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88011d20 con 0x7fdc9815af20 2024-03-18T20:25:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.171+0000 7fdc9dc76700 1 -- 172.21.15.1:0/2568109374 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9815af20 msgr2=0x7fdc98072480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.171+0000 7fdc9dc76700 1 --2- 172.21.15.1:0/2568109374 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9815af20 0x7fdc98072480 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7fdc880098f0 tx=0x7fdc88009c00 comp rx=0 tx=0).stop 2024-03-18T20:25:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.171+0000 7fdc9dc76700 1 -- 172.21.15.1:0/2568109374 shutdown_connections 2024-03-18T20:25:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.171+0000 7fdc9dc76700 1 --2- 172.21.15.1:0/2568109374 >> v2:172.21.15.165:3300/0 conn(0x7fdc980af6b0 0x7fdc9815dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.171+0000 7fdc9dc76700 1 --2- 172.21.15.1:0/2568109374 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9815af20 0x7fdc98072480 unknown :-1 s=CLOSED pgs=2613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.171+0000 7fdc9dc76700 1 -- 172.21.15.1:0/2568109374 >> 172.21.15.1:0/2568109374 conn(0x7fdc980d00a0 msgr2=0x7fdc980d04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.171+0000 7fdc9dc76700 1 -- 172.21.15.1:0/2568109374 shutdown_connections 2024-03-18T20:25:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.171+0000 7fdc9dc76700 1 -- 172.21.15.1:0/2568109374 wait complete. 2024-03-18T20:25:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.172+0000 7fdc9dc76700 1 Processor -- start 2024-03-18T20:25:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.172+0000 7fdc9dc76700 1 -- start start 2024-03-18T20:25:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.173+0000 7fdc9dc76700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc980af6b0 0x7fdc98147cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.173+0000 7fdc9dc76700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9815af20 0x7fdc980ada60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.173+0000 7fdc9dc76700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc9815b300 0x7fdc980adfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.173+0000 7fdc9dc76700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc98149630 con 0x7fdc9815af20 2024-03-18T20:25:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.173+0000 7fdc9dc76700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc98149910 con 0x7fdc9815b300 2024-03-18T20:25:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.173+0000 7fdc9dc76700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc98149bf0 con 0x7fdc980af6b0 2024-03-18T20:25:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.173+0000 7fdc8effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9815af20 0x7fdc980ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.173+0000 7fdc8effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9815af20 0x7fdc980ada60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54258/0 (socket says 172.21.15.1:54258) 2024-03-18T20:25:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.173+0000 7fdc8effd700 1 -- 172.21.15.1:0/1498590682 learned_addr learned my addr 172.21.15.1:0/1498590682 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.173+0000 7fdc97fff700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc9815b300 0x7fdc980adfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.173+0000 7fdc977fe700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc980af6b0 0x7fdc98147cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.174+0000 7fdc8effd700 1 -- 172.21.15.1:0/1498590682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc980af6b0 msgr2=0x7fdc98147cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.174+0000 7fdc8effd700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc980af6b0 0x7fdc98147cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.174+0000 7fdc8effd700 1 -- 172.21.15.1:0/1498590682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc9815b300 msgr2=0x7fdc980adfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.174+0000 7fdc8effd700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc9815b300 0x7fdc980adfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.174+0000 7fdc8effd700 1 -- 172.21.15.1:0/1498590682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc88009550 con 0x7fdc9815af20 2024-03-18T20:25:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.174+0000 7fdc8effd700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9815af20 0x7fdc980ada60 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7fdc8000c400 tx=0x7fdc8000c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.174+0000 7fdc94ff9700 1 -- 172.21.15.1:0/1498590682 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc800052d0 con 0x7fdc9815af20 2024-03-18T20:25:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.174+0000 7fdc94ff9700 1 -- 172.21.15.1:0/1498590682 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc80005430 con 0x7fdc9815af20 2024-03-18T20:25:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.174+0000 7fdc9dc76700 1 -- 172.21.15.1:0/1498590682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc98149ed0 con 0x7fdc9815af20 2024-03-18T20:25:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.174+0000 7fdc94ff9700 1 -- 172.21.15.1:0/1498590682 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc800038c0 con 0x7fdc9815af20 2024-03-18T20:25:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.174+0000 7fdc9dc76700 1 -- 172.21.15.1:0/1498590682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc98148210 con 0x7fdc9815af20 2024-03-18T20:25:48.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.175+0000 7fdc9dc76700 1 -- 172.21.15.1:0/1498590682 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc98070270 con 0x7fdc9815af20 2024-03-18T20:25:48.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.175+0000 7fdc94ff9700 1 -- 172.21.15.1:0/1498590682 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 136) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fdc80003a20 con 0x7fdc9815af20 2024-03-18T20:25:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.176+0000 7fdc94ff9700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdc7806ae00 0x7fdc7806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.176+0000 7fdc977fe700 1 -- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdc7806ae00 msgr2=0x7fdc7806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.176+0000 7fdc94ff9700 1 -- 172.21.15.1:0/1498590682 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 6993+0+0 (secure 0 0 0) 0x7fdc80088d00 con 0x7fdc9815af20 2024-03-18T20:25:48.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.176+0000 7fdc977fe700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdc7806ae00 0x7fdc7806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:48.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.180+0000 7fdc94ff9700 1 -- 172.21.15.1:0/1498590682 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdc80053840 con 0x7fdc9815af20 2024-03-18T20:25:48.247 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:48.243+0000 7f41d28b5240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:25:48.374 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:48.370+0000 7f17bbc35240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:25:48.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.377+0000 7fdc977fe700 1 -- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdc7806ae00 msgr2=0x7fdc7806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:48.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.377+0000 7fdc977fe700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdc7806ae00 0x7fdc7806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:48.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.412+0000 7fdc9dc76700 1 -- 172.21.15.1:0/1498590682 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc98149a80 con 0x7fdc7806ae00 2024-03-18T20:25:48.466 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:48.462+0000 7f41d28b5240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:25:48.745 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:48.741+0000 7f17bbc35240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:25:48.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.778+0000 7fdc977fe700 1 -- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdc7806ae00 msgr2=0x7fdc7806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:48.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:48.778+0000 7fdc977fe700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdc7806ae00 0x7fdc7806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T20:25:48.857 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:48.853+0000 7f41d28b5240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:25:49.328 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:49.323+0000 7f17bbc35240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:25:49.400 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:49.395+0000 7f17bbc35240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:25:49.476 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:49.472+0000 7f41d28b5240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:25:49.526 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:49.522+0000 7f17bbc35240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:25:49.552 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:49.548+0000 7f41d28b5240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:25:49.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:49.579+0000 7fdc977fe700 1 -- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdc7806ae00 msgr2=0x7fdc7806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:49.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:49.579+0000 7fdc977fe700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdc7806ae00 0x7fdc7806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T20:25:49.687 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:49.683+0000 7f41d28b5240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:25:49.706 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:49.701+0000 7f17bbc35240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:25:49.777 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:49.773+0000 7f17bbc35240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:25:49.856 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:49.851+0000 7f17bbc35240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:25:49.874 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:49.870+0000 7f41d28b5240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:25:49.934 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:49.929+0000 7f17bbc35240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:25:49.950 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:49.946+0000 7f41d28b5240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:25:50.011 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:50.007+0000 7f17bbc35240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:25:50.034 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:50.030+0000 7f41d28b5240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:25:50.116 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:50.112+0000 7f41d28b5240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:25:50.181 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:50.177+0000 7f17bbc35240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:25:50.199 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:50.195+0000 7f41d28b5240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:25:50.379 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:50.375+0000 7f41d28b5240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:25:50.396 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:50.391+0000 7f17bbc35240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:25:50.547 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:50.542+0000 7f17bbc35240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:25:50.603 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:50.599+0000 7f41d28b5240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:25:50.620 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:50.615+0000 7f17bbc35240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:25:50.764 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:50.760+0000 7f41d28b5240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:25:50.841 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:50.837+0000 7f41d28b5240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:25:51.066 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:51.062+0000 7f17bbc35240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:25:51.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:51.180+0000 7fdc977fe700 1 -- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdc7806ae00 msgr2=0x7fdc7806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:51.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:51.180+0000 7fdc977fe700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdc7806ae00 0x7fdc7806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T20:25:51.321 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:51.317+0000 7f41d28b5240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:25:51.405 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:51.401+0000 7f17bbc35240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:25:51.568 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:51.564+0000 7f17bbc35240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:25:51.678 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:51.674+0000 7f41d28b5240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:25:51.851 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:51.847+0000 7f41d28b5240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:25:51.958 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:51.953+0000 7f17bbc35240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:25:52.124 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:52.119+0000 7f17bbc35240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:25:52.269 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:52.265+0000 7f41d28b5240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:25:52.447 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:52.443+0000 7f41d28b5240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:25:52.486 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:52.482+0000 7f17bbc35240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:25:52.575 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:52.571+0000 7f17bbc35240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:25:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:52.641+0000 7fdc94ff9700 1 -- 172.21.15.1:0/1498590682 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 137) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fdc80086d10 con 0x7fdc9815af20 2024-03-18T20:25:52.828 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:52.824+0000 7f41d28b5240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:25:52.922 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:52.918+0000 7f41d28b5240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:25:52.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:52.941+0000 7fdc94ff9700 1 -- 172.21.15.1:0/1498590682 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 138) v1 ==== 88865+0+0 (secure 0 0 0) 0x7fdc80053c10 con 0x7fdc9815af20 2024-03-18T20:25:52.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:52.941+0000 7fdc94ff9700 1 -- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdc7806ae00 msgr2=0x7fdc7806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:52.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:52.941+0000 7fdc94ff9700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdc7806ae00 0x7fdc7806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:53.160 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.156+0000 7f4147cd7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.160 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.156+0000 7f4147cd7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.160 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.156+0000 7f4147cd7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.160 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.156+0000 7f4147cd7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.160 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.156+0000 7f4147cd7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.160 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.156+0000 7f4144cd1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.161 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.156+0000 7f4144cd1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.161 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.156+0000 7f4144cd1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.161 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.156+0000 7f4144cd1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.161 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.156+0000 7f4144cd1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.169 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.165+0000 7f4144cd1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.169 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.165+0000 7f4144cd1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.170 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.165+0000 7f4144cd1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.170 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.165+0000 7f4144cd1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.170 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.165+0000 7f4144cd1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.170 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.166+0000 7f4147cd7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.170 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.166+0000 7f4147cd7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.170 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.166+0000 7f4147cd7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.170 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.166+0000 7f4147cd7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.170 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:53.166+0000 7f4147cd7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:25:53.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:53.949+0000 7fdc94ff9700 1 -- 172.21.15.1:0/1498590682 <== mon.0 v2:172.21.15.1:3300/0 9 ==== mgrmap(e 139) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fdc800544e0 con 0x7fdc9815af20 2024-03-18T20:25:53.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:53.949+0000 7fdc94ff9700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdc7806ae00 0x7fdc7806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:53.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:53.949+0000 7fdc94ff9700 1 -- 172.21.15.1:0/1498590682 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc98149a80 con 0x7fdc7806ae00 2024-03-18T20:25:53.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:53.950+0000 7fdc977fe700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdc7806ae00 0x7fdc7806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:53.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:53.954+0000 7fdc977fe700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdc7806ae00 0x7fdc7806d2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009c40 tx=0x7fdc88008030 comp rx=0 tx=0).ready entity=mgr.9122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:54.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.011+0000 7fdc94ff9700 1 -- 172.21.15.1:0/1498590682 <== mgr.9122 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+53 (secure 0 0 0) 0x7fdc98149a80 con 0x7fdc7806ae00 2024-03-18T20:25:54.015 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path mandatory_fs_argument_test_dir 2024-03-18T20:25:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.013+0000 7fdc9dc76700 1 -- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdc7806ae00 msgr2=0x7fdc7806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.013+0000 7fdc9dc76700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdc7806ae00 0x7fdc7806d2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009c40 tx=0x7fdc88008030 comp rx=0 tx=0).stop 2024-03-18T20:25:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.013+0000 7fdc9dc76700 1 -- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9815af20 msgr2=0x7fdc980ada60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.013+0000 7fdc9dc76700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9815af20 0x7fdc980ada60 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7fdc8000c400 tx=0x7fdc8000c710 comp rx=0 tx=0).stop 2024-03-18T20:25:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.013+0000 7fdc9dc76700 1 -- 172.21.15.1:0/1498590682 shutdown_connections 2024-03-18T20:25:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.013+0000 7fdc9dc76700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdc7806ae00 0x7fdc7806d2c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.013+0000 7fdc9dc76700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdc980af6b0 0x7fdc98147cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.013+0000 7fdc9dc76700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdc9815b300 0x7fdc980adfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.013+0000 7fdc9dc76700 1 --2- 172.21.15.1:0/1498590682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc9815af20 0x7fdc980ada60 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.013+0000 7fdc9dc76700 1 -- 172.21.15.1:0/1498590682 >> 172.21.15.1:0/1498590682 conn(0x7fdc980d00a0 msgr2=0x7fdc980b1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.013+0000 7fdc9dc76700 1 -- 172.21.15.1:0/1498590682 shutdown_connections 2024-03-18T20:25:54.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.014+0000 7fdc9dc76700 1 -- 172.21.15.1:0/1498590682 wait complete. 2024-03-18T20:25:54.031 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path mandatory_fs_argument_test_dir 2024-03-18T20:25:54.031 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-03-18T20:25:54.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.161+0000 7fd2f43cc700 1 Processor -- start 2024-03-18T20:25:54.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.161+0000 7fd2f43cc700 1 -- start start 2024-03-18T20:25:54.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.162+0000 7fd2f43cc700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd2ec15a160 0x7fd2ec0b3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:54.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.162+0000 7fd2f43cc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd2ec06f7d0 0x7fd2ec15dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:54.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.162+0000 7fd2f43cc700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ec0b4d10 con 0x7fd2ec06fc50 2024-03-18T20:25:54.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.162+0000 7fd2f43cc700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ec15ec10 con 0x7fd2ec06f7d0 2024-03-18T20:25:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.162+0000 7fd2f43cc700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ec0b4a30 con 0x7fd2ec15a160 2024-03-18T20:25:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.162+0000 7fd2f1967700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd2ec06f7d0 0x7fd2ec15dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.162+0000 7fd2f1967700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd2ec06f7d0 0x7fd2ec15dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54408/0 (socket says 172.21.15.1:54408) 2024-03-18T20:25:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.162+0000 7fd2f1967700 1 -- 172.21.15.1:0/2458335820 learned_addr learned my addr 172.21.15.1:0/2458335820 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.162+0000 7fd2f2168700 1 --2- 172.21.15.1:0/2458335820 >> v2:172.21.15.165:3300/0 conn(0x7fd2ec15a160 0x7fd2ec0b3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.162+0000 7fd2f1967700 1 -- 172.21.15.1:0/2458335820 >> v2:172.21.15.165:3300/0 conn(0x7fd2ec15a160 msgr2=0x7fd2ec0b3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.162+0000 7fd2f1967700 1 --2- 172.21.15.1:0/2458335820 >> v2:172.21.15.165:3300/0 conn(0x7fd2ec15a160 0x7fd2ec0b3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.162+0000 7fd2f1967700 1 -- 172.21.15.1:0/2458335820 >> v1:172.21.15.165:6789/0 conn(0x7fd2ec06fc50 legacy=0x7fd2ec0717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.162+0000 7fd2f1967700 1 -- 172.21.15.1:0/2458335820 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec06ecf0 con 0x7fd2ec06f7d0 2024-03-18T20:25:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.163+0000 7fd2f1967700 1 --2- 172.21.15.1:0/2458335820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2ec06f7d0 0x7fd2ec15dd50 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0d0cd0 tx=0x7fd2e8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6273e21bc2351bae server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.163+0000 7fd2f1166700 1 -- 172.21.15.1:0/2458335820 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8011830 con 0x7fd2ec06f7d0 2024-03-18T20:25:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.163+0000 7fd2f1166700 1 -- 172.21.15.1:0/2458335820 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e8011990 con 0x7fd2ec06f7d0 2024-03-18T20:25:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.163+0000 7fd2f1166700 1 -- 172.21.15.1:0/2458335820 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8011d20 con 0x7fd2ec06f7d0 2024-03-18T20:25:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.163+0000 7fd2f43cc700 1 -- 172.21.15.1:0/2458335820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2ec06f7d0 msgr2=0x7fd2ec15dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:54.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.163+0000 7fd2f43cc700 1 --2- 172.21.15.1:0/2458335820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2ec06f7d0 0x7fd2ec15dd50 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0d0cd0 tx=0x7fd2e8009bd0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.163+0000 7fd2f43cc700 1 -- 172.21.15.1:0/2458335820 shutdown_connections 2024-03-18T20:25:54.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.163+0000 7fd2f43cc700 1 --2- 172.21.15.1:0/2458335820 >> v2:172.21.15.165:3300/0 conn(0x7fd2ec15a160 0x7fd2ec0b3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.163+0000 7fd2f43cc700 1 --2- 172.21.15.1:0/2458335820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2ec06f7d0 0x7fd2ec15dd50 unknown :-1 s=CLOSED pgs=2628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.163+0000 7fd2f43cc700 1 -- 172.21.15.1:0/2458335820 >> 172.21.15.1:0/2458335820 conn(0x7fd2ec0d00c0 msgr2=0x7fd2ec0d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:54.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.163+0000 7fd2f43cc700 1 -- 172.21.15.1:0/2458335820 shutdown_connections 2024-03-18T20:25:54.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.163+0000 7fd2f43cc700 1 -- 172.21.15.1:0/2458335820 wait complete. 2024-03-18T20:25:54.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.165+0000 7fd2f43cc700 1 Processor -- start 2024-03-18T20:25:54.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.165+0000 7fd2f43cc700 1 -- start start 2024-03-18T20:25:54.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.165+0000 7fd2f43cc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2ec06f7d0 0x7fd2ec1f3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:54.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.165+0000 7fd2f43cc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2ec06fc50 0x7fd2ec158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:54.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.165+0000 7fd2f43cc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2ec15a160 0x7fd2ec14e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:54.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.165+0000 7fd2f43cc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec14ea40 con 0x7fd2ec06fc50 2024-03-18T20:25:54.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.165+0000 7fd2f43cc700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec1f3840 con 0x7fd2ec06f7d0 2024-03-18T20:25:54.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.165+0000 7fd2f43cc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec1f3b20 con 0x7fd2ec15a160 2024-03-18T20:25:54.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.165+0000 7fd2f1967700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2ec06fc50 0x7fd2ec158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:54.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.165+0000 7fd2f1967700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2ec06fc50 0x7fd2ec158f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54414/0 (socket says 172.21.15.1:54414) 2024-03-18T20:25:54.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.166+0000 7fd2f1967700 1 -- 172.21.15.1:0/49462280 learned_addr learned my addr 172.21.15.1:0/49462280 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:54.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.166+0000 7fd2f2168700 1 --2- 172.21.15.1:0/49462280 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2ec06f7d0 0x7fd2ec1f3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:54.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.166+0000 7fd2f2969700 1 --2- 172.21.15.1:0/49462280 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2ec15a160 0x7fd2ec14e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:54.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.166+0000 7fd2f1967700 1 -- 172.21.15.1:0/49462280 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2ec15a160 msgr2=0x7fd2ec14e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:54.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.166+0000 7fd2f1967700 1 --2- 172.21.15.1:0/49462280 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2ec15a160 0x7fd2ec14e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.166+0000 7fd2f1967700 1 -- 172.21.15.1:0/49462280 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2ec06f7d0 msgr2=0x7fd2ec1f3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:54.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.166+0000 7fd2f1967700 1 --2- 172.21.15.1:0/49462280 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2ec06f7d0 0x7fd2ec1f3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.166+0000 7fd2f1967700 1 -- 172.21.15.1:0/49462280 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e8009550 con 0x7fd2ec06fc50 2024-03-18T20:25:54.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.166+0000 7fd2f1967700 1 --2- 172.21.15.1:0/49462280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2ec06fc50 0x7fd2ec158f90 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009c10 tx=0x7fd2e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:54.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.166+0000 7fd2deffd700 1 -- 172.21.15.1:0/49462280 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8004590 con 0x7fd2ec06fc50 2024-03-18T20:25:54.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.166+0000 7fd2deffd700 1 -- 172.21.15.1:0/49462280 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e801e070 con 0x7fd2ec06fc50 2024-03-18T20:25:54.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.166+0000 7fd2deffd700 1 -- 172.21.15.1:0/49462280 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e80038e0 con 0x7fd2ec06fc50 2024-03-18T20:25:54.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.166+0000 7fd2f43cc700 1 -- 172.21.15.1:0/49462280 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ec1f3e00 con 0x7fd2ec06fc50 2024-03-18T20:25:54.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.166+0000 7fd2f43cc700 1 -- 172.21.15.1:0/49462280 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ec1efd40 con 0x7fd2ec06fc50 2024-03-18T20:25:54.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.167+0000 7fd2f43cc700 1 -- 172.21.15.1:0/49462280 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ec1eff30 con 0x7fd2ec06fc50 2024-03-18T20:25:54.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.167+0000 7fd2deffd700 1 -- 172.21.15.1:0/49462280 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 139) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fd2e8003a40 con 0x7fd2ec06fc50 2024-03-18T20:25:54.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.168+0000 7fd2deffd700 1 --2- 172.21.15.1:0/49462280 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd2d006ab30 0x7fd2d006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:54.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.168+0000 7fd2deffd700 1 -- 172.21.15.1:0/49462280 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7fd2e808a4a0 con 0x7fd2ec06fc50 2024-03-18T20:25:54.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.168+0000 7fd2f2168700 1 --2- 172.21.15.1:0/49462280 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd2d006ab30 0x7fd2d006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:54.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.168+0000 7fd2f2168700 1 --2- 172.21.15.1:0/49462280 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd2d006ab30 0x7fd2d006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0cc2a0 tx=0x7fd2e0009210 comp rx=0 tx=0).ready entity=mgr.9122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:54.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.173+0000 7fd2deffd700 1 -- 172.21.15.1:0/49462280 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd2e8055360 con 0x7fd2ec06fc50 2024-03-18T20:25:54.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.405+0000 7fd2f43cc700 1 -- 172.21.15.1:0/49462280 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2ec15b550 con 0x7fd2d006ab30 2024-03-18T20:25:54.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.451+0000 7fd2deffd700 1 -- 172.21.15.1:0/49462280 <== mgr.9122 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fd2e0011860 con 0x7fd2d006ab30 2024-03-18T20:25:54.455 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule activated for path mandatory_fs_argument_test_dir 2024-03-18T20:25:54.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.453+0000 7fd2f43cc700 1 -- 172.21.15.1:0/49462280 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd2d006ab30 msgr2=0x7fd2d006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:54.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.453+0000 7fd2f43cc700 1 --2- 172.21.15.1:0/49462280 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd2d006ab30 0x7fd2d006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0cc2a0 tx=0x7fd2e0009210 comp rx=0 tx=0).stop 2024-03-18T20:25:54.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.453+0000 7fd2f43cc700 1 -- 172.21.15.1:0/49462280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2ec06fc50 msgr2=0x7fd2ec158f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:54.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.453+0000 7fd2f43cc700 1 --2- 172.21.15.1:0/49462280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2ec06fc50 0x7fd2ec158f90 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009c10 tx=0x7fd2e8004750 comp rx=0 tx=0).stop 2024-03-18T20:25:54.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.454+0000 7fd2f43cc700 1 -- 172.21.15.1:0/49462280 shutdown_connections 2024-03-18T20:25:54.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.454+0000 7fd2f43cc700 1 --2- 172.21.15.1:0/49462280 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd2d006ab30 0x7fd2d006cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.454+0000 7fd2f43cc700 1 --2- 172.21.15.1:0/49462280 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd2ec15a160 0x7fd2ec14e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.454+0000 7fd2f43cc700 1 --2- 172.21.15.1:0/49462280 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd2ec06f7d0 0x7fd2ec1f3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.454+0000 7fd2f43cc700 1 --2- 172.21.15.1:0/49462280 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2ec06fc50 0x7fd2ec158f90 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.454+0000 7fd2f43cc700 1 -- 172.21.15.1:0/49462280 >> 172.21.15.1:0/49462280 conn(0x7fd2ec0d00c0 msgr2=0x7fd2ec0685d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:54.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.454+0000 7fd2f43cc700 1 -- 172.21.15.1:0/49462280 shutdown_connections 2024-03-18T20:25:54.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.454+0000 7fd2f43cc700 1 -- 172.21.15.1:0/49462280 wait complete. 2024-03-18T20:25:54.470 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path mandatory_fs_argument_test_dir 2024-03-18T20:25:54.470 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-03-18T20:25:54.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.597+0000 7fbf08c47700 1 Processor -- start 2024-03-18T20:25:54.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.597+0000 7fbf08c47700 1 -- start start 2024-03-18T20:25:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.598+0000 7fbf08c47700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf040cd870 0x7fbf040cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.598+0000 7fbf08c47700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbf040b1e20 0x7fbf040cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.598+0000 7fbf08c47700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf040d0df0 con 0x7fbf0406bc20 2024-03-18T20:25:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.598+0000 7fbf08c47700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf040ce190 con 0x7fbf040cd870 2024-03-18T20:25:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.598+0000 7fbf08c47700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf040d0550 con 0x7fbf040b1e20 2024-03-18T20:25:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.598+0000 7fbf0259c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf040cd870 0x7fbf040cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.598+0000 7fbf02d9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbf0406bc20 0x7fbf0406c070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47204/0 (socket says 172.21.15.1:47204) 2024-03-18T20:25:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.598+0000 7fbf02d9d700 1 -- 172.21.15.1:0/1805284044 learned_addr learned my addr 172.21.15.1:0/1805284044 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.599+0000 7fbf01d9b700 1 --2- 172.21.15.1:0/1805284044 >> v2:172.21.15.165:3300/0 conn(0x7fbf040b1e20 0x7fbf040cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.599+0000 7fbf0159a700 1 -- 172.21.15.1:0/1805284044 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995652207 0 0) 0x7fbf040d0df0 con 0x7fbf0406bc20 2024-03-18T20:25:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.599+0000 7fbf0159a700 1 -- 172.21.15.1:0/1805284044 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeec0035b0 con 0x7fbf0406bc20 2024-03-18T20:25:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.599+0000 7fbf0259c700 1 -- 172.21.15.1:0/1805284044 >> v2:172.21.15.165:3300/0 conn(0x7fbf040b1e20 msgr2=0x7fbf040cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.599+0000 7fbf0259c700 1 --2- 172.21.15.1:0/1805284044 >> v2:172.21.15.165:3300/0 conn(0x7fbf040b1e20 0x7fbf040cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.599+0000 7fbf0259c700 1 -- 172.21.15.1:0/1805284044 >> v1:172.21.15.1:6789/0 conn(0x7fbf0406bc20 legacy=0x7fbf0406c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.599+0000 7fbf0259c700 1 -- 172.21.15.1:0/1805284044 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf040d0830 con 0x7fbf040cd870 2024-03-18T20:25:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.599+0000 7fbf0259c700 1 --2- 172.21.15.1:0/1805284044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf040cd870 0x7fbf040cff70 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7fbf040715c0 tx=0x7fbefc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=778143e4b11d9206 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.599+0000 7fbf0159a700 1 -- 172.21.15.1:0/1805284044 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc00e070 con 0x7fbf040cd870 2024-03-18T20:25:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.599+0000 7fbf0159a700 1 -- 172.21.15.1:0/1805284044 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbefc00bb20 con 0x7fbf040cd870 2024-03-18T20:25:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.599+0000 7fbf0159a700 1 -- 172.21.15.1:0/1805284044 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc012430 con 0x7fbf040cd870 2024-03-18T20:25:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.600+0000 7fbf08c47700 1 -- 172.21.15.1:0/1805284044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf040cd870 msgr2=0x7fbf040cff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.600+0000 7fbf08c47700 1 --2- 172.21.15.1:0/1805284044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf040cd870 0x7fbf040cff70 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7fbf040715c0 tx=0x7fbefc00b330 comp rx=0 tx=0).stop 2024-03-18T20:25:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.600+0000 7fbf08c47700 1 -- 172.21.15.1:0/1805284044 shutdown_connections 2024-03-18T20:25:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.600+0000 7fbf08c47700 1 --2- 172.21.15.1:0/1805284044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf040cd870 0x7fbf040cff70 unknown :-1 s=CLOSED pgs=2631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.600+0000 7fbf08c47700 1 --2- 172.21.15.1:0/1805284044 >> v2:172.21.15.165:3300/0 conn(0x7fbf040b1e20 0x7fbf040cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.600+0000 7fbf08c47700 1 -- 172.21.15.1:0/1805284044 >> 172.21.15.1:0/1805284044 conn(0x7fbf040d0ff0 msgr2=0x7fbf040d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:54.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.600+0000 7fbf08c47700 1 -- 172.21.15.1:0/1805284044 shutdown_connections 2024-03-18T20:25:54.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.600+0000 7fbf08c47700 1 -- 172.21.15.1:0/1805284044 wait complete. 2024-03-18T20:25:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.601+0000 7fbf08c47700 1 Processor -- start 2024-03-18T20:25:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.601+0000 7fbf08c47700 1 -- start start 2024-03-18T20:25:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.602+0000 7fbf08c47700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf0406bc20 0x7fbf0414d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.602+0000 7fbf08c47700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf040b1e20 0x7fbf041e51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.602+0000 7fbf08c47700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf040cd870 0x7fbf041f32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.602+0000 7fbf08c47700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf041e4540 con 0x7fbf040cd870 2024-03-18T20:25:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.602+0000 7fbf08c47700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf041e49d0 con 0x7fbf040b1e20 2024-03-18T20:25:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.602+0000 7fbf08c47700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf041e3930 con 0x7fbf0406bc20 2024-03-18T20:25:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.602+0000 7fbf02d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf040cd870 0x7fbf041f32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.602+0000 7fbf02d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf040cd870 0x7fbf041f32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33506/0 (socket says 172.21.15.1:33506) 2024-03-18T20:25:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.602+0000 7fbf02d9d700 1 -- 172.21.15.1:0/2451777858 learned_addr learned my addr 172.21.15.1:0/2451777858 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.602+0000 7fbf0259c700 1 --2- 172.21.15.1:0/2451777858 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf0406bc20 0x7fbf0414d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.602+0000 7fbf01d9b700 1 --2- 172.21.15.1:0/2451777858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf040b1e20 0x7fbf041e51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.602+0000 7fbf02d9d700 1 -- 172.21.15.1:0/2451777858 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf0406bc20 msgr2=0x7fbf0414d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.602+0000 7fbf02d9d700 1 --2- 172.21.15.1:0/2451777858 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf0406bc20 0x7fbf0414d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.602+0000 7fbf02d9d700 1 -- 172.21.15.1:0/2451777858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf040b1e20 msgr2=0x7fbf041e51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.602+0000 7fbf02d9d700 1 --2- 172.21.15.1:0/2451777858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf040b1e20 0x7fbf041e51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.602+0000 7fbf02d9d700 1 -- 172.21.15.1:0/2451777858 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbefc009a70 con 0x7fbf040cd870 2024-03-18T20:25:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.602+0000 7fbf01d9b700 1 --2- 172.21.15.1:0/2451777858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf040b1e20 0x7fbf041e51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.602+0000 7fbf02d9d700 1 --2- 172.21.15.1:0/2451777858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf040cd870 0x7fbf041f32a0 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7fbef400c420 tx=0x7fbef400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.603+0000 7fbef2ffd700 1 -- 172.21.15.1:0/2451777858 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef4005d70 con 0x7fbf040cd870 2024-03-18T20:25:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.603+0000 7fbef2ffd700 1 -- 172.21.15.1:0/2451777858 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef40070d0 con 0x7fbf040cd870 2024-03-18T20:25:54.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.603+0000 7fbf08c47700 1 -- 172.21.15.1:0/2451777858 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf041e3c10 con 0x7fbf040cd870 2024-03-18T20:25:54.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.603+0000 7fbef2ffd700 1 -- 172.21.15.1:0/2451777858 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef4007440 con 0x7fbf040cd870 2024-03-18T20:25:54.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.603+0000 7fbf08c47700 1 -- 172.21.15.1:0/2451777858 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf041e41d0 con 0x7fbf040cd870 2024-03-18T20:25:54.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.603+0000 7fbf08c47700 1 -- 172.21.15.1:0/2451777858 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf040c8880 con 0x7fbf040cd870 2024-03-18T20:25:54.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.604+0000 7fbef2ffd700 1 -- 172.21.15.1:0/2451777858 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 139) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fbef40075a0 con 0x7fbf040cd870 2024-03-18T20:25:54.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.605+0000 7fbef2ffd700 1 --2- 172.21.15.1:0/2451777858 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbee406ab30 0x7fbee406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:54.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.605+0000 7fbf0259c700 1 --2- 172.21.15.1:0/2451777858 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbee406ab30 0x7fbee406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:54.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.605+0000 7fbef2ffd700 1 -- 172.21.15.1:0/2451777858 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7fbef40889e0 con 0x7fbf040cd870 2024-03-18T20:25:54.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.605+0000 7fbf0259c700 1 --2- 172.21.15.1:0/2451777858 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbee406ab30 0x7fbee406cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbefc014040 tx=0x7fbefc011000 comp rx=0 tx=0).ready entity=mgr.9122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:54.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.609+0000 7fbef2ffd700 1 -- 172.21.15.1:0/2451777858 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbef4053880 con 0x7fbf040cd870 2024-03-18T20:25:54.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.842+0000 7fbf08c47700 1 -- 172.21.15.1:0/2451777858 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf0415dcc0 con 0x7fbee406ab30 2024-03-18T20:25:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.898+0000 7fbef2ffd700 1 -- 172.21.15.1:0/2451777858 <== mgr.9122 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7fbefc029050 con 0x7fbee406ab30 2024-03-18T20:25:54.902 INFO:teuthology.orchestra.run.smithi001.stdout:Retention added to path mandatory_fs_argument_test_dir 2024-03-18T20:25:54.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.900+0000 7fbf08c47700 1 -- 172.21.15.1:0/2451777858 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbee406ab30 msgr2=0x7fbee406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:54.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.900+0000 7fbf08c47700 1 --2- 172.21.15.1:0/2451777858 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbee406ab30 0x7fbee406cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbefc014040 tx=0x7fbefc011000 comp rx=0 tx=0).stop 2024-03-18T20:25:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.900+0000 7fbf08c47700 1 -- 172.21.15.1:0/2451777858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf040cd870 msgr2=0x7fbf041f32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.900+0000 7fbf08c47700 1 --2- 172.21.15.1:0/2451777858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf040cd870 0x7fbf041f32a0 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7fbef400c420 tx=0x7fbef400c7e0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.900+0000 7fbf08c47700 1 -- 172.21.15.1:0/2451777858 shutdown_connections 2024-03-18T20:25:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.900+0000 7fbf08c47700 1 --2- 172.21.15.1:0/2451777858 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbee406ab30 0x7fbee406cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.900+0000 7fbf08c47700 1 --2- 172.21.15.1:0/2451777858 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf0406bc20 0x7fbf0414d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.900+0000 7fbf08c47700 1 --2- 172.21.15.1:0/2451777858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf040cd870 0x7fbf041f32a0 unknown :-1 s=CLOSED pgs=2632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.900+0000 7fbf08c47700 1 --2- 172.21.15.1:0/2451777858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf040b1e20 0x7fbf041e51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.900+0000 7fbf08c47700 1 -- 172.21.15.1:0/2451777858 >> 172.21.15.1:0/2451777858 conn(0x7fbf040d0ff0 msgr2=0x7fbf040cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.900+0000 7fbf08c47700 1 -- 172.21.15.1:0/2451777858 shutdown_connections 2024-03-18T20:25:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:54.900+0000 7fbf08c47700 1 -- 172.21.15.1:0/2451777858 wait complete. 2024-03-18T20:25:54.916 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path mandatory_fs_argument_test_dir 2024-03-18T20:25:54.917 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list mandatory_fs_argument_test_dir --fs backup_fs --format json 2024-03-18T20:25:55.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.050+0000 7f6a44f9d700 1 Processor -- start 2024-03-18T20:25:55.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.050+0000 7f6a44f9d700 1 -- start start 2024-03-18T20:25:55.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.051+0000 7f6a44f9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6a4015d8b0 0x7f6a400d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.051+0000 7f6a44f9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a400d0040 0x7f6a400cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.051+0000 7f6a44f9d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4015dc90 con 0x7f6a400d1170 2024-03-18T20:25:55.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.051+0000 7f6a44f9d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a4015ec90 con 0x7f6a4015d8b0 2024-03-18T20:25:55.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.051+0000 7f6a44f9d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4015ded0 con 0x7f6a400d0040 2024-03-18T20:25:55.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.051+0000 7f6a3dd9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6a400d1170 0x7f6a400cf930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47208/0 (socket says 172.21.15.1:47208) 2024-03-18T20:25:55.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.051+0000 7f6a3e59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6a4015d8b0 0x7f6a400d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.051+0000 7f6a3dd9b700 1 -- 172.21.15.1:0/3183155606 learned_addr learned my addr 172.21.15.1:0/3183155606 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:55.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.051+0000 7f6a3ed9d700 1 --2- 172.21.15.1:0/3183155606 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a400d0040 0x7f6a400cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.051+0000 7f6a3e59c700 1 -- 172.21.15.1:0/3183155606 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a400d0040 msgr2=0x7f6a400cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:55.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.051+0000 7f6a3e59c700 1 --2- 172.21.15.1:0/3183155606 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a400d0040 0x7f6a400cde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.051+0000 7f6a3e59c700 1 -- 172.21.15.1:0/3183155606 >> v1:172.21.15.1:6789/0 conn(0x7f6a400d1170 legacy=0x7f6a400cf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.051+0000 7f6a3e59c700 1 -- 172.21.15.1:0/3183155606 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4015e1b0 con 0x7f6a4015d8b0 2024-03-18T20:25:55.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.051+0000 7f6a3d59a700 1 -- 172.21.15.1:0/3183155606 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498697322 0 0) 0x7f6a34001110 con 0x7f6a400d1170 2024-03-18T20:25:55.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.051+0000 7f6a3e59c700 1 --2- 172.21.15.1:0/3183155606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a4015d8b0 0x7f6a400d3c30 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009840 tx=0x7f6a30009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4565333dd87a960e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:55.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.051+0000 7f6a3d59a700 1 -- 172.21.15.1:0/3183155606 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30011830 con 0x7f6a4015d8b0 2024-03-18T20:25:55.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.052+0000 7f6a3d59a700 1 -- 172.21.15.1:0/3183155606 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a30011990 con 0x7f6a4015d8b0 2024-03-18T20:25:55.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.052+0000 7f6a3d59a700 1 -- 172.21.15.1:0/3183155606 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30011d20 con 0x7f6a4015d8b0 2024-03-18T20:25:55.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.052+0000 7f6a44f9d700 1 -- 172.21.15.1:0/3183155606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a4015d8b0 msgr2=0x7f6a400d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.052+0000 7f6a44f9d700 1 --2- 172.21.15.1:0/3183155606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a4015d8b0 0x7f6a400d3c30 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009840 tx=0x7f6a30009b50 comp rx=0 tx=0).stop 2024-03-18T20:25:55.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.052+0000 7f6a44f9d700 1 -- 172.21.15.1:0/3183155606 shutdown_connections 2024-03-18T20:25:55.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.052+0000 7f6a44f9d700 1 --2- 172.21.15.1:0/3183155606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a4015d8b0 0x7f6a400d3c30 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.052+0000 7f6a44f9d700 1 --2- 172.21.15.1:0/3183155606 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a400d0040 0x7f6a400cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.052+0000 7f6a44f9d700 1 -- 172.21.15.1:0/3183155606 >> 172.21.15.1:0/3183155606 conn(0x7f6a4009c640 msgr2=0x7f6a400d3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:55.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.052+0000 7f6a44f9d700 1 -- 172.21.15.1:0/3183155606 shutdown_connections 2024-03-18T20:25:55.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.052+0000 7f6a44f9d700 1 -- 172.21.15.1:0/3183155606 wait complete. 2024-03-18T20:25:55.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.054+0000 7f6a44f9d700 1 Processor -- start 2024-03-18T20:25:55.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.054+0000 7f6a44f9d700 1 -- start start 2024-03-18T20:25:55.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.055+0000 7f6a44f9d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a400d0040 0x7f6a400b8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.055+0000 7f6a44f9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a400d1170 0x7f6a400b7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.055+0000 7f6a44f9d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a4015d8b0 0x7f6a400b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.055+0000 7f6a44f9d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a400b84b0 con 0x7f6a400d1170 2024-03-18T20:25:55.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.055+0000 7f6a44f9d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a400b7ef0 con 0x7f6a4015d8b0 2024-03-18T20:25:55.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.055+0000 7f6a44f9d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a400b81d0 con 0x7f6a400d0040 2024-03-18T20:25:55.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.055+0000 7f6a3dd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a400d1170 0x7f6a400b7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.055+0000 7f6a3dd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a400d1170 0x7f6a400b7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33546/0 (socket says 172.21.15.1:33546) 2024-03-18T20:25:55.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.055+0000 7f6a3dd9b700 1 -- 172.21.15.1:0/2103071512 learned_addr learned my addr 172.21.15.1:0/2103071512 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:55.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.055+0000 7f6a3e59c700 1 --2- 172.21.15.1:0/2103071512 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a400d0040 0x7f6a400b8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.055+0000 7f6a3ed9d700 1 --2- 172.21.15.1:0/2103071512 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a4015d8b0 0x7f6a400b78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.055+0000 7f6a3dd9b700 1 -- 172.21.15.1:0/2103071512 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a400d0040 msgr2=0x7f6a400b8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.055+0000 7f6a3dd9b700 1 --2- 172.21.15.1:0/2103071512 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a400d0040 0x7f6a400b8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.055+0000 7f6a3dd9b700 1 -- 172.21.15.1:0/2103071512 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a4015d8b0 msgr2=0x7f6a400b78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.055+0000 7f6a3dd9b700 1 --2- 172.21.15.1:0/2103071512 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a4015d8b0 0x7f6a400b78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.055+0000 7f6a3dd9b700 1 -- 172.21.15.1:0/2103071512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a300094a0 con 0x7f6a400d1170 2024-03-18T20:25:55.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.055+0000 7f6a3dd9b700 1 --2- 172.21.15.1:0/2103071512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a400d1170 0x7f6a400b7390 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400c730 tx=0x7f6a3400ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:55.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.056+0000 7f6a2effd700 1 -- 172.21.15.1:0/2103071512 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a34004c80 con 0x7f6a400d1170 2024-03-18T20:25:55.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.056+0000 7f6a2effd700 1 -- 172.21.15.1:0/2103071512 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a34004e00 con 0x7f6a400d1170 2024-03-18T20:25:55.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.056+0000 7f6a44f9d700 1 -- 172.21.15.1:0/2103071512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a400ca570 con 0x7f6a400d1170 2024-03-18T20:25:55.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.056+0000 7f6a2effd700 1 -- 172.21.15.1:0/2103071512 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a34007310 con 0x7f6a400d1170 2024-03-18T20:25:55.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.056+0000 7f6a44f9d700 1 -- 172.21.15.1:0/2103071512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a401562c0 con 0x7f6a400d1170 2024-03-18T20:25:55.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.057+0000 7f6a44f9d700 1 -- 172.21.15.1:0/2103071512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a400ca3b0 con 0x7f6a400d1170 2024-03-18T20:25:55.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.057+0000 7f6a2effd700 1 -- 172.21.15.1:0/2103071512 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 140) v1 ==== 89359+0+0 (secure 0 0 0) 0x7f6a34007470 con 0x7f6a400d1170 2024-03-18T20:25:55.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.057+0000 7f6a2effd700 1 --2- 172.21.15.1:0/2103071512 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6a2006af20 0x7f6a2006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.058+0000 7f6a2effd700 1 -- 172.21.15.1:0/2103071512 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7f6a340898d0 con 0x7f6a400d1170 2024-03-18T20:25:55.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.058+0000 7f6a3e59c700 1 --2- 172.21.15.1:0/2103071512 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6a2006af20 0x7f6a2006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.058+0000 7f6a3e59c700 1 --2- 172.21.15.1:0/2103071512 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6a2006af20 0x7f6a2006d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6a300090b0 tx=0x7f6a3001c040 comp rx=0 tx=0).ready entity=mgr.9122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:55.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.062+0000 7f6a2effd700 1 -- 172.21.15.1:0/2103071512 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6a340540f0 con 0x7f6a400d1170 2024-03-18T20:25:55.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.290+0000 7f6a44f9d700 1 -- 172.21.15.1:0/2103071512 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6a400d15d0 con 0x7f6a2006af20 2024-03-18T20:25:55.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.296+0000 7f6a2effd700 1 -- 172.21.15.1:0/2103071512 <== mgr.9122 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+88 (secure 0 0 0) 0x7f6a30011b80 con 0x7f6a2006af20 2024-03-18T20:25:55.300 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:55.300 INFO:teuthology.orchestra.run.smithi001.stdout:{"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2024-03-18T20:25:55.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.298+0000 7f6a44f9d700 1 -- 172.21.15.1:0/2103071512 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6a2006af20 msgr2=0x7f6a2006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.298+0000 7f6a44f9d700 1 --2- 172.21.15.1:0/2103071512 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6a2006af20 0x7f6a2006d3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6a300090b0 tx=0x7f6a3001c040 comp rx=0 tx=0).stop 2024-03-18T20:25:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.298+0000 7f6a44f9d700 1 -- 172.21.15.1:0/2103071512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a400d1170 msgr2=0x7f6a400b7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.298+0000 7f6a44f9d700 1 --2- 172.21.15.1:0/2103071512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a400d1170 0x7f6a400b7390 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400c730 tx=0x7f6a3400ca40 comp rx=0 tx=0).stop 2024-03-18T20:25:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.298+0000 7f6a44f9d700 1 -- 172.21.15.1:0/2103071512 shutdown_connections 2024-03-18T20:25:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.298+0000 7f6a44f9d700 1 --2- 172.21.15.1:0/2103071512 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6a2006af20 0x7f6a2006d3e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.298+0000 7f6a44f9d700 1 --2- 172.21.15.1:0/2103071512 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6a400d0040 0x7f6a400b8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.298+0000 7f6a44f9d700 1 --2- 172.21.15.1:0/2103071512 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a4015d8b0 0x7f6a400b78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.298+0000 7f6a44f9d700 1 --2- 172.21.15.1:0/2103071512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a400d1170 0x7f6a400b7390 unknown :-1 s=CLOSED pgs=2637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.298+0000 7f6a44f9d700 1 -- 172.21.15.1:0/2103071512 >> 172.21.15.1:0/2103071512 conn(0x7f6a4009c640 msgr2=0x7f6a4006b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.298+0000 7f6a44f9d700 1 -- 172.21.15.1:0/2103071512 shutdown_connections 2024-03-18T20:25:55.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.298+0000 7f6a44f9d700 1 -- 172.21.15.1:0/2103071512 wait complete. 2024-03-18T20:25:55.314 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2024-03-18T20:25:55.314 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status mandatory_fs_argument_test_dir --fs backup_fs --format json 2024-03-18T20:25:55.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.440+0000 7f1b40736700 1 Processor -- start 2024-03-18T20:25:55.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.440+0000 7f1b40736700 1 -- start start 2024-03-18T20:25:55.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.441+0000 7f1b40736700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1b38144120 0x7f1b380b5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.441+0000 7f1b40736700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1b380b08f0 0x7f1b3815ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.441+0000 7f1b40736700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b3815f3a0 0x7f1b38070210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.441+0000 7f1b40736700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b3815b830 con 0x7f1b38144120 2024-03-18T20:25:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.441+0000 7f1b40736700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b3815f7f0 con 0x7f1b380b08f0 2024-03-18T20:25:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.441+0000 7f1b40736700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b380b58e0 con 0x7f1b3815f3a0 2024-03-18T20:25:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.441+0000 7f1b3e4d2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1b38144120 0x7f1b380b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.441+0000 7f1b3e4d2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1b38144120 0x7f1b380b5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33562/0 (socket says 172.21.15.1:33562) 2024-03-18T20:25:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.441+0000 7f1b3e4d2700 1 -- 172.21.15.1:0/1158338879 learned_addr learned my addr 172.21.15.1:0/1158338879 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.441+0000 7f1b3dcd1700 1 --2- 172.21.15.1:0/1158338879 >> v2:172.21.15.165:3300/0 conn(0x7f1b380b08f0 0x7f1b3815ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.441+0000 7f1b3ecd3700 1 --2- 172.21.15.1:0/1158338879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b3815f3a0 0x7f1b38070210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.441+0000 7f1b3e4d2700 1 -- 172.21.15.1:0/1158338879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b3815f3a0 msgr2=0x7f1b38070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.441+0000 7f1b3e4d2700 1 --2- 172.21.15.1:0/1158338879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b3815f3a0 0x7f1b38070210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.441+0000 7f1b3e4d2700 1 -- 172.21.15.1:0/1158338879 >> v2:172.21.15.165:3300/0 conn(0x7f1b380b08f0 msgr2=0x7f1b3815ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.441+0000 7f1b3e4d2700 1 --2- 172.21.15.1:0/1158338879 >> v2:172.21.15.165:3300/0 conn(0x7f1b380b08f0 0x7f1b3815ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.441+0000 7f1b3e4d2700 1 -- 172.21.15.1:0/1158338879 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b380d00b0 con 0x7f1b38144120 2024-03-18T20:25:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.441+0000 7f1b3e4d2700 1 --2- 172.21.15.1:0/1158338879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b38144120 0x7f1b380b5090 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f1b380b4230 tx=0x7f1b28009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81511d48e370e0a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.442+0000 7f1b3d4d0700 1 -- 172.21.15.1:0/1158338879 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2800b870 con 0x7f1b38144120 2024-03-18T20:25:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.442+0000 7f1b3d4d0700 1 -- 172.21.15.1:0/1158338879 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2800b9d0 con 0x7f1b38144120 2024-03-18T20:25:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.442+0000 7f1b3d4d0700 1 -- 172.21.15.1:0/1158338879 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2800bd60 con 0x7f1b38144120 2024-03-18T20:25:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.442+0000 7f1b40736700 1 -- 172.21.15.1:0/1158338879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b38144120 msgr2=0x7f1b380b5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.442+0000 7f1b40736700 1 --2- 172.21.15.1:0/1158338879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b38144120 0x7f1b380b5090 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f1b380b4230 tx=0x7f1b28009bd0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.442+0000 7f1b40736700 1 -- 172.21.15.1:0/1158338879 shutdown_connections 2024-03-18T20:25:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.442+0000 7f1b40736700 1 --2- 172.21.15.1:0/1158338879 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b3815f3a0 0x7f1b38070210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.442+0000 7f1b40736700 1 --2- 172.21.15.1:0/1158338879 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b38144120 0x7f1b380b5090 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.442+0000 7f1b40736700 1 --2- 172.21.15.1:0/1158338879 >> v2:172.21.15.165:3300/0 conn(0x7f1b380b08f0 0x7f1b3815ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.442+0000 7f1b40736700 1 -- 172.21.15.1:0/1158338879 >> 172.21.15.1:0/1158338879 conn(0x7f1b380b2e80 msgr2=0x7f1b380b3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.442+0000 7f1b40736700 1 -- 172.21.15.1:0/1158338879 shutdown_connections 2024-03-18T20:25:55.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.442+0000 7f1b40736700 1 -- 172.21.15.1:0/1158338879 wait complete. 2024-03-18T20:25:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.443+0000 7f1b40736700 1 Processor -- start 2024-03-18T20:25:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.443+0000 7f1b40736700 1 -- start start 2024-03-18T20:25:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.444+0000 7f1b40736700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b380b08f0 0x7f1b380ada00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.444+0000 7f1b40736700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b38144120 0x7f1b3815dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.444+0000 7f1b40736700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b3815f3a0 0x7f1b38157de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.444+0000 7f1b40736700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b38131690 con 0x7f1b38144120 2024-03-18T20:25:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.444+0000 7f1b40736700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b380adf40 con 0x7f1b380b08f0 2024-03-18T20:25:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.444+0000 7f1b40736700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b38130a80 con 0x7f1b3815f3a0 2024-03-18T20:25:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.444+0000 7f1b3dcd1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b38144120 0x7f1b3815dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.444+0000 7f1b3dcd1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b38144120 0x7f1b3815dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33570/0 (socket says 172.21.15.1:33570) 2024-03-18T20:25:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.444+0000 7f1b3dcd1700 1 -- 172.21.15.1:0/1312139658 learned_addr learned my addr 172.21.15.1:0/1312139658 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.444+0000 7f1b3ecd3700 1 --2- 172.21.15.1:0/1312139658 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b3815f3a0 0x7f1b38157de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.444+0000 7f1b3e4d2700 1 --2- 172.21.15.1:0/1312139658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b380b08f0 0x7f1b380ada00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.445+0000 7f1b3dcd1700 1 -- 172.21.15.1:0/1312139658 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b3815f3a0 msgr2=0x7f1b38157de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.445+0000 7f1b3dcd1700 1 --2- 172.21.15.1:0/1312139658 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b3815f3a0 0x7f1b38157de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.445+0000 7f1b3dcd1700 1 -- 172.21.15.1:0/1312139658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b380b08f0 msgr2=0x7f1b380ada00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.445+0000 7f1b3dcd1700 1 --2- 172.21.15.1:0/1312139658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b380b08f0 0x7f1b380ada00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.445+0000 7f1b3dcd1700 1 -- 172.21.15.1:0/1312139658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b28009550 con 0x7f1b38144120 2024-03-18T20:25:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.445+0000 7f1b3dcd1700 1 --2- 172.21.15.1:0/1312139658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b38144120 0x7f1b3815dde0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7f1b3400e810 tx=0x7f1b3400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.445+0000 7f1b2effd700 1 -- 172.21.15.1:0/1312139658 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3400c070 con 0x7f1b38144120 2024-03-18T20:25:55.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.445+0000 7f1b2effd700 1 -- 172.21.15.1:0/1312139658 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b34013610 con 0x7f1b38144120 2024-03-18T20:25:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.445+0000 7f1b40736700 1 -- 172.21.15.1:0/1312139658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b38130d60 con 0x7f1b38144120 2024-03-18T20:25:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.445+0000 7f1b2effd700 1 -- 172.21.15.1:0/1312139658 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b340139a0 con 0x7f1b38144120 2024-03-18T20:25:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.445+0000 7f1b40736700 1 -- 172.21.15.1:0/1312139658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b38131320 con 0x7f1b38144120 2024-03-18T20:25:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.446+0000 7f1b40736700 1 -- 172.21.15.1:0/1312139658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b380676a0 con 0x7f1b38144120 2024-03-18T20:25:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.446+0000 7f1b2effd700 1 -- 172.21.15.1:0/1312139658 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 140) v1 ==== 89359+0+0 (secure 0 0 0) 0x7f1b34013b00 con 0x7f1b38144120 2024-03-18T20:25:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.447+0000 7f1b2effd700 1 --2- 172.21.15.1:0/1312139658 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1b1c06af20 0x7f1b1c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.447+0000 7f1b2effd700 1 -- 172.21.15.1:0/1312139658 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7f1b34089b60 con 0x7f1b38144120 2024-03-18T20:25:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.447+0000 7f1b3e4d2700 1 --2- 172.21.15.1:0/1312139658 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1b1c06af20 0x7f1b1c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.447+0000 7f1b3e4d2700 1 --2- 172.21.15.1:0/1312139658 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1b1c06af20 0x7f1b1c06d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1b28000c00 tx=0x7f1b28008030 comp rx=0 tx=0).ready entity=mgr.9122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:55.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.451+0000 7f1b2effd700 1 -- 172.21.15.1:0/1312139658 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1b34054300 con 0x7f1b38144120 2024-03-18T20:25:55.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.676+0000 7f1b40736700 1 -- 172.21.15.1:0/1312139658 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1b380b55d0 con 0x7f1b1c06af20 2024-03-18T20:25:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.684+0000 7f1b2effd700 1 -- 172.21.15.1:0/1312139658 <== mgr.9122 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+353 (secure 0 0 0) 0x7f1b2800bb40 con 0x7f1b1c06af20 2024-03-18T20:25:55.688 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:55.689 INFO:teuthology.orchestra.run.smithi001.stdout:[{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T20:25:53", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-03-18T20:25:55.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.686+0000 7f1b40736700 1 -- 172.21.15.1:0/1312139658 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1b1c06af20 msgr2=0x7f1b1c06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.686+0000 7f1b40736700 1 --2- 172.21.15.1:0/1312139658 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1b1c06af20 0x7f1b1c06d3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1b28000c00 tx=0x7f1b28008030 comp rx=0 tx=0).stop 2024-03-18T20:25:55.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.686+0000 7f1b40736700 1 -- 172.21.15.1:0/1312139658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b38144120 msgr2=0x7f1b3815dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.686+0000 7f1b40736700 1 --2- 172.21.15.1:0/1312139658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b38144120 0x7f1b3815dde0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7f1b3400e810 tx=0x7f1b3400eb20 comp rx=0 tx=0).stop 2024-03-18T20:25:55.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.686+0000 7f1b40736700 1 -- 172.21.15.1:0/1312139658 shutdown_connections 2024-03-18T20:25:55.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.686+0000 7f1b40736700 1 --2- 172.21.15.1:0/1312139658 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1b1c06af20 0x7f1b1c06d3e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.686+0000 7f1b40736700 1 --2- 172.21.15.1:0/1312139658 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1b3815f3a0 0x7f1b38157de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.686+0000 7f1b40736700 1 --2- 172.21.15.1:0/1312139658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b380b08f0 0x7f1b380ada00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.686+0000 7f1b40736700 1 --2- 172.21.15.1:0/1312139658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b38144120 0x7f1b3815dde0 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.686+0000 7f1b40736700 1 -- 172.21.15.1:0/1312139658 >> 172.21.15.1:0/1312139658 conn(0x7f1b380b2e80 msgr2=0x7f1b380d0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.686+0000 7f1b40736700 1 -- 172.21.15.1:0/1312139658 shutdown_connections 2024-03-18T20:25:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.686+0000 7f1b40736700 1 -- 172.21.15.1:0/1312139658 wait complete. 2024-03-18T20:25:55.702 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2024-03-18T00:00:00", "created": "2024-03-18T20:25:53", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-03-18T20:25:55.703 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-03-18T20:25:55.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.830+0000 7f4883060700 1 Processor -- start 2024-03-18T20:25:55.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.830+0000 7f4883060700 1 -- start start 2024-03-18T20:25:55.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.831+0000 7f4883060700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f487c160a70 0x7f487c0d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.831+0000 7f4883060700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f487c160e50 0x7f487c0cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.831+0000 7f4883060700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f487c162170 con 0x7f487c1612d0 2024-03-18T20:25:55.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.831+0000 7f4883060700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f487c0cef30 con 0x7f487c160e50 2024-03-18T20:25:55.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.831+0000 7f4883060700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487c0cf210 con 0x7f487c160a70 2024-03-18T20:25:55.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.831+0000 7f48815fd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f487c1612d0 0x7f487c0c6da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47210/0 (socket says 172.21.15.1:47210) 2024-03-18T20:25:55.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.831+0000 7f48815fd700 1 -- 172.21.15.1:0/1395805972 learned_addr learned my addr 172.21.15.1:0/1395805972 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:55.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.831+0000 7f487b7fe700 1 -- 172.21.15.1:0/1395805972 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450196466 0 0) 0x7f4874001490 con 0x7f487c1612d0 2024-03-18T20:25:55.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.831+0000 7f487b7fe700 1 -- 172.21.15.1:0/1395805972 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f486c0035b0 con 0x7f487c1612d0 2024-03-18T20:25:55.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.831+0000 7f487bfff700 1 --2- 172.21.15.1:0/1395805972 >> v2:172.21.15.165:3300/0 conn(0x7f487c160e50 0x7f487c0cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.831+0000 7f4880dfc700 1 --2- 172.21.15.1:0/1395805972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f487c160a70 0x7f487c0d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f487b7fe700 1 -- 172.21.15.1:0/1395805972 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1906904577 0 0) 0x7f486c0035b0 con 0x7f487c1612d0 2024-03-18T20:25:55.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f487b7fe700 1 -- 172.21.15.1:0/1395805972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f487c160a70 msgr2=0x7f487c0d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:55.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f487b7fe700 1 --2- 172.21.15.1:0/1395805972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f487c160a70 0x7f487c0d3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f487b7fe700 1 -- 172.21.15.1:0/1395805972 >> v2:172.21.15.165:3300/0 conn(0x7f487c160e50 msgr2=0x7f487c0cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:55.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f487b7fe700 1 --2- 172.21.15.1:0/1395805972 >> v2:172.21.15.165:3300/0 conn(0x7f487c160e50 0x7f487c0cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f487b7fe700 1 -- 172.21.15.1:0/1395805972 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487c0cf4f0 con 0x7f487c1612d0 2024-03-18T20:25:55.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f487b7fe700 1 -- 172.21.15.1:0/1395805972 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f4874001b80 con 0x7f487c1612d0 2024-03-18T20:25:55.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f487b7fe700 1 -- 172.21.15.1:0/1395805972 >> v1:172.21.15.1:6789/0 conn(0x7f487c1612d0 legacy=0x7f487c0c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f487b7fe700 1 --2- 172.21.15.1:0/1395805972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f487c160a70 0x7f486c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f487b7fe700 1 --2- 172.21.15.1:0/1395805972 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f486c003f80 0x7f486c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f487b7fe700 1 --2- 172.21.15.1:0/1395805972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f486c004d90 0x7f486c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f487b7fe700 1 -- 172.21.15.1:0/1395805972 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487c0cef30 con 0x7f486c004d90 2024-03-18T20:25:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f487b7fe700 1 -- 172.21.15.1:0/1395805972 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487c0cf210 con 0x7f486c003f80 2024-03-18T20:25:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f487b7fe700 1 -- 172.21.15.1:0/1395805972 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f486c00ba10 con 0x7f487c160a70 2024-03-18T20:25:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f48815fd700 1 --2- 172.21.15.1:0/1395805972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f486c004d90 0x7f486c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f4880dfc700 1 --2- 172.21.15.1:0/1395805972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f487c160a70 0x7f486c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f487bfff700 1 --2- 172.21.15.1:0/1395805972 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f486c003f80 0x7f486c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f48815fd700 1 -- 172.21.15.1:0/1395805972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f487c160a70 msgr2=0x7f486c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f48815fd700 1 --2- 172.21.15.1:0/1395805972 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f487c160a70 0x7f486c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f48815fd700 1 -- 172.21.15.1:0/1395805972 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f486c003f80 msgr2=0x7f486c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f48815fd700 1 --2- 172.21.15.1:0/1395805972 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f486c003f80 0x7f486c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f48815fd700 1 -- 172.21.15.1:0/1395805972 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486c0035b0 con 0x7f486c004d90 2024-03-18T20:25:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.832+0000 7f48815fd700 1 --2- 172.21.15.1:0/1395805972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f486c004d90 0x7f486c005210 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f487400a830 tx=0x7f486c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1d0fc5dd790d75a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.833+0000 7f487b7fe700 1 -- 172.21.15.1:0/1395805972 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487400a4c0 con 0x7f486c004d90 2024-03-18T20:25:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.833+0000 7f487b7fe700 1 -- 172.21.15.1:0/1395805972 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4874012600 con 0x7f486c004d90 2024-03-18T20:25:55.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.833+0000 7f487b7fe700 1 -- 172.21.15.1:0/1395805972 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48740129b0 con 0x7f486c004d90 2024-03-18T20:25:55.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.833+0000 7f4883060700 1 -- 172.21.15.1:0/1395805972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f486c004d90 msgr2=0x7f486c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.833+0000 7f4883060700 1 --2- 172.21.15.1:0/1395805972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f486c004d90 0x7f486c005210 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f487400a830 tx=0x7f486c005880 comp rx=0 tx=0).stop 2024-03-18T20:25:55.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.833+0000 7f48815fd700 1 -- 172.21.15.1:0/1395805972 reap_dead start 2024-03-18T20:25:55.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.833+0000 7f4883060700 1 -- 172.21.15.1:0/1395805972 shutdown_connections 2024-03-18T20:25:55.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.833+0000 7f4883060700 1 -- 172.21.15.1:0/1395805972 >> 172.21.15.1:0/1395805972 conn(0x7f487c09c560 msgr2=0x7f487c0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:55.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.833+0000 7f4883060700 1 -- 172.21.15.1:0/1395805972 shutdown_connections 2024-03-18T20:25:55.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.833+0000 7f4883060700 1 -- 172.21.15.1:0/1395805972 wait complete. 2024-03-18T20:25:55.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.835+0000 7f4883060700 1 Processor -- start 2024-03-18T20:25:55.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.835+0000 7f4883060700 1 -- start start 2024-03-18T20:25:55.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.836+0000 7f4883060700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f487c0cc0e0 0x7f487c0cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.836+0000 7f4883060700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f487c0cb220 0x7f487c0cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.836+0000 7f4883060700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f487c0cca00 0x7f487c15d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.836+0000 7f4883060700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487c0cbd20 con 0x7f487c0cc0e0 2024-03-18T20:25:55.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.836+0000 7f4883060700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487c0cb760 con 0x7f487c0cca00 2024-03-18T20:25:55.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.836+0000 7f4883060700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487c0cba40 con 0x7f487c0cb220 2024-03-18T20:25:55.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.836+0000 7f4880dfc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f487c0cc0e0 0x7f487c0cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.836+0000 7f4880dfc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f487c0cc0e0 0x7f487c0cace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33590/0 (socket says 172.21.15.1:33590) 2024-03-18T20:25:55.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.836+0000 7f4880dfc700 1 -- 172.21.15.1:0/1419872372 learned_addr learned my addr 172.21.15.1:0/1419872372 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:55.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.836+0000 7f48815fd700 1 --2- 172.21.15.1:0/1419872372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f487c0cca00 0x7f487c15d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.836+0000 7f487bfff700 1 --2- 172.21.15.1:0/1419872372 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f487c0cb220 0x7f487c0cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.836+0000 7f4880dfc700 1 -- 172.21.15.1:0/1419872372 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f487c0cb220 msgr2=0x7f487c0cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.836+0000 7f4880dfc700 1 --2- 172.21.15.1:0/1419872372 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f487c0cb220 0x7f487c0cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.836+0000 7f4880dfc700 1 -- 172.21.15.1:0/1419872372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f487c0cca00 msgr2=0x7f487c15d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:55.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.836+0000 7f4880dfc700 1 --2- 172.21.15.1:0/1419872372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f487c0cca00 0x7f487c15d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:55.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.836+0000 7f4880dfc700 1 -- 172.21.15.1:0/1419872372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487c15c9f0 con 0x7f487c0cc0e0 2024-03-18T20:25:55.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.837+0000 7f4880dfc700 1 --2- 172.21.15.1:0/1419872372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f487c0cc0e0 0x7f487c0cace0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f4870011aa0 tx=0x7f4870011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:55.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.837+0000 7f48797fa700 1 -- 172.21.15.1:0/1419872372 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487000f980 con 0x7f487c0cc0e0 2024-03-18T20:25:55.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.837+0000 7f48797fa700 1 -- 172.21.15.1:0/1419872372 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f487000fae0 con 0x7f487c0cc0e0 2024-03-18T20:25:55.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.837+0000 7f4883060700 1 -- 172.21.15.1:0/1419872372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f487c15ccd0 con 0x7f487c0cc0e0 2024-03-18T20:25:55.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.837+0000 7f48797fa700 1 -- 172.21.15.1:0/1419872372 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487000fe90 con 0x7f487c0cc0e0 2024-03-18T20:25:55.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.837+0000 7f4883060700 1 -- 172.21.15.1:0/1419872372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f487c0cce90 con 0x7f487c0cc0e0 2024-03-18T20:25:55.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.838+0000 7f4883060700 1 -- 172.21.15.1:0/1419872372 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f487c15cef0 con 0x7f487c0cc0e0 2024-03-18T20:25:55.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.838+0000 7f48797fa700 1 -- 172.21.15.1:0/1419872372 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 140) v1 ==== 89359+0+0 (secure 0 0 0) 0x7f4870014440 con 0x7f487c0cc0e0 2024-03-18T20:25:55.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.839+0000 7f48797fa700 1 --2- 172.21.15.1:0/1419872372 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f486006af50 0x7f486006d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:55.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.839+0000 7f487bfff700 1 --2- 172.21.15.1:0/1419872372 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f486006af50 0x7f486006d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:55.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.839+0000 7f48797fa700 1 -- 172.21.15.1:0/1419872372 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7f487008a530 con 0x7f487c0cc0e0 2024-03-18T20:25:55.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.839+0000 7f487bfff700 1 --2- 172.21.15.1:0/1419872372 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f486006af50 0x7f486006d410 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f487c15ad90 tx=0x7f4868008040 comp rx=0 tx=0).ready entity=mgr.9122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:55.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:55.843+0000 7f48797fa700 1 -- 172.21.15.1:0/1419872372 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4870054ea0 con 0x7f487c0cc0e0 2024-03-18T20:25:56.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.078+0000 7f4883060700 1 -- 172.21.15.1:0/1419872372 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f487c067390 con 0x7f486006af50 2024-03-18T20:25:56.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.134+0000 7f48797fa700 1 -- 172.21.15.1:0/1419872372 <== mgr.9122 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f487c067390 con 0x7f486006af50 2024-03-18T20:25:56.139 INFO:teuthology.orchestra.run.smithi001.stdout:Retention removed from path mandatory_fs_argument_test_dir 2024-03-18T20:25:56.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.137+0000 7f4883060700 1 -- 172.21.15.1:0/1419872372 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f486006af50 msgr2=0x7f486006d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.137+0000 7f4883060700 1 --2- 172.21.15.1:0/1419872372 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f486006af50 0x7f486006d410 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f487c15ad90 tx=0x7f4868008040 comp rx=0 tx=0).stop 2024-03-18T20:25:56.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.137+0000 7f4883060700 1 -- 172.21.15.1:0/1419872372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f487c0cc0e0 msgr2=0x7f487c0cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.137+0000 7f4883060700 1 --2- 172.21.15.1:0/1419872372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f487c0cc0e0 0x7f487c0cace0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f4870011aa0 tx=0x7f4870011e60 comp rx=0 tx=0).stop 2024-03-18T20:25:56.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.137+0000 7f4883060700 1 -- 172.21.15.1:0/1419872372 shutdown_connections 2024-03-18T20:25:56.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.137+0000 7f4883060700 1 --2- 172.21.15.1:0/1419872372 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f486006af50 0x7f486006d410 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.137+0000 7f4883060700 1 --2- 172.21.15.1:0/1419872372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f487c0cca00 0x7f487c15d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.137+0000 7f4883060700 1 --2- 172.21.15.1:0/1419872372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f487c0cc0e0 0x7f487c0cace0 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.137+0000 7f4883060700 1 --2- 172.21.15.1:0/1419872372 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f487c0cb220 0x7f487c0cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.137+0000 7f4883060700 1 -- 172.21.15.1:0/1419872372 >> 172.21.15.1:0/1419872372 conn(0x7f487c09c560 msgr2=0x7f487c0b7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:56.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.137+0000 7f4883060700 1 -- 172.21.15.1:0/1419872372 shutdown_connections 2024-03-18T20:25:56.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.137+0000 7f4883060700 1 -- 172.21.15.1:0/1419872372 wait complete. 2024-03-18T20:25:56.153 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention removed from path mandatory_fs_argument_test_dir 2024-03-18T20:25:56.153 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-03-18T20:25:56.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.280+0000 7fd7c5e6c700 1 Processor -- start 2024-03-18T20:25:56.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.280+0000 7fd7c5e6c700 1 -- start start 2024-03-18T20:25:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.281+0000 7fd7c5e6c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7c0154850 0x7fd7c00723a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.281+0000 7fd7c5e6c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7c015e440 0x7fd7c006ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.281+0000 7fd7c5e6c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c0154c30 con 0x7fd7c00b3a80 2024-03-18T20:25:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.281+0000 7fd7c5e6c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c015eb40 con 0x7fd7c0154850 2024-03-18T20:25:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.281+0000 7fd7c5e6c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c00b49e0 con 0x7fd7c015e440 2024-03-18T20:25:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.281+0000 7fd7bf7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7c0154850 0x7fd7c00723a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.281+0000 7fd7beffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd7c00b3a80 0x7fd7c015dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47218/0 (socket says 172.21.15.1:47218) 2024-03-18T20:25:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.281+0000 7fd7beffd700 1 -- 172.21.15.1:0/197480417 learned_addr learned my addr 172.21.15.1:0/197480417 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.281+0000 7fd7bffff700 1 --2- 172.21.15.1:0/197480417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7c015e440 0x7fd7c006ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.281+0000 7fd7bf7fe700 1 -- 172.21.15.1:0/197480417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7c015e440 msgr2=0x7fd7c006ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.281+0000 7fd7bf7fe700 1 --2- 172.21.15.1:0/197480417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7c015e440 0x7fd7c006ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.281+0000 7fd7bf7fe700 1 -- 172.21.15.1:0/197480417 >> v1:172.21.15.1:6789/0 conn(0x7fd7c00b3a80 legacy=0x7fd7c015dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.281+0000 7fd7bf7fe700 1 -- 172.21.15.1:0/197480417 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c00718c0 con 0x7fd7c0154850 2024-03-18T20:25:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.281+0000 7fd7bf7fe700 1 --2- 172.21.15.1:0/197480417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c0154850 0x7fd7c00723a0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0009840 tx=0x7fd7b0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d6ddeff1ae14f53 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.281+0000 7fd7be7fc700 1 -- 172.21.15.1:0/197480417 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b0011830 con 0x7fd7c0154850 2024-03-18T20:25:56.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.281+0000 7fd7be7fc700 1 -- 172.21.15.1:0/197480417 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b0011990 con 0x7fd7c0154850 2024-03-18T20:25:56.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.281+0000 7fd7be7fc700 1 -- 172.21.15.1:0/197480417 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b0011d20 con 0x7fd7c0154850 2024-03-18T20:25:56.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.282+0000 7fd7c5e6c700 1 -- 172.21.15.1:0/197480417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c0154850 msgr2=0x7fd7c00723a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.282+0000 7fd7c5e6c700 1 --2- 172.21.15.1:0/197480417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c0154850 0x7fd7c00723a0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0009840 tx=0x7fd7b0009b50 comp rx=0 tx=0).stop 2024-03-18T20:25:56.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.282+0000 7fd7c5e6c700 1 -- 172.21.15.1:0/197480417 shutdown_connections 2024-03-18T20:25:56.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.282+0000 7fd7c5e6c700 1 --2- 172.21.15.1:0/197480417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c0154850 0x7fd7c00723a0 unknown :-1 s=CLOSED pgs=2644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.282+0000 7fd7c5e6c700 1 --2- 172.21.15.1:0/197480417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7c015e440 0x7fd7c006ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.282+0000 7fd7c5e6c700 1 -- 172.21.15.1:0/197480417 >> 172.21.15.1:0/197480417 conn(0x7fd7c00d00a0 msgr2=0x7fd7c00d04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:56.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.282+0000 7fd7c5e6c700 1 -- 172.21.15.1:0/197480417 shutdown_connections 2024-03-18T20:25:56.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.282+0000 7fd7c5e6c700 1 -- 172.21.15.1:0/197480417 wait complete. 2024-03-18T20:25:56.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.283+0000 7fd7c5e6c700 1 Processor -- start 2024-03-18T20:25:56.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.283+0000 7fd7c5e6c700 1 -- start start 2024-03-18T20:25:56.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.284+0000 7fd7c5e6c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7c00b3a80 0x7fd7c01f1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.284+0000 7fd7c5e6c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c0154850 0x7fd7c00ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.284+0000 7fd7c5e6c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7c015e440 0x7fd7c00aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.284+0000 7fd7c5e6c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c01f17c0 con 0x7fd7c0154850 2024-03-18T20:25:56.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.284+0000 7fd7c5e6c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c01572b0 con 0x7fd7c015e440 2024-03-18T20:25:56.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.284+0000 7fd7c5e6c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c006d6c0 con 0x7fd7c00b3a80 2024-03-18T20:25:56.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.284+0000 7fd7beffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c0154850 0x7fd7c00ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.284+0000 7fd7beffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c0154850 0x7fd7c00ce2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33604/0 (socket says 172.21.15.1:33604) 2024-03-18T20:25:56.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.284+0000 7fd7beffd700 1 -- 172.21.15.1:0/3756704504 learned_addr learned my addr 172.21.15.1:0/3756704504 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:56.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.284+0000 7fd7bf7fe700 1 --2- 172.21.15.1:0/3756704504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7c00b3a80 0x7fd7c01f1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.284+0000 7fd7bffff700 1 --2- 172.21.15.1:0/3756704504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7c015e440 0x7fd7c00aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.285+0000 7fd7beffd700 1 -- 172.21.15.1:0/3756704504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7c00b3a80 msgr2=0x7fd7c01f1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.285+0000 7fd7beffd700 1 --2- 172.21.15.1:0/3756704504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7c00b3a80 0x7fd7c01f1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.285+0000 7fd7beffd700 1 -- 172.21.15.1:0/3756704504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7c015e440 msgr2=0x7fd7c00aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.285+0000 7fd7beffd700 1 --2- 172.21.15.1:0/3756704504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7c015e440 0x7fd7c00aec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.285+0000 7fd7beffd700 1 -- 172.21.15.1:0/3756704504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b00094a0 con 0x7fd7c0154850 2024-03-18T20:25:56.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.285+0000 7fd7beffd700 1 --2- 172.21.15.1:0/3756704504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c0154850 0x7fd7c00ce2a0 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400c520 tx=0x7fd7b400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:56.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.285+0000 7fd7a7fff700 1 -- 172.21.15.1:0/3756704504 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4005d70 con 0x7fd7c0154850 2024-03-18T20:25:56.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.285+0000 7fd7a7fff700 1 -- 172.21.15.1:0/3756704504 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b40070d0 con 0x7fd7c0154850 2024-03-18T20:25:56.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.285+0000 7fd7c5e6c700 1 -- 172.21.15.1:0/3756704504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c006d9a0 con 0x7fd7c0154850 2024-03-18T20:25:56.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.285+0000 7fd7a7fff700 1 -- 172.21.15.1:0/3756704504 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4007440 con 0x7fd7c0154850 2024-03-18T20:25:56.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.285+0000 7fd7c5e6c700 1 -- 172.21.15.1:0/3756704504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c006df60 con 0x7fd7c0154850 2024-03-18T20:25:56.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.286+0000 7fd7c5e6c700 1 -- 172.21.15.1:0/3756704504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c00b4610 con 0x7fd7c0154850 2024-03-18T20:25:56.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.286+0000 7fd7a7fff700 1 -- 172.21.15.1:0/3756704504 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 140) v1 ==== 89359+0+0 (secure 0 0 0) 0x7fd7b40075a0 con 0x7fd7c0154850 2024-03-18T20:25:56.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.287+0000 7fd7a7fff700 1 --2- 172.21.15.1:0/3756704504 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd7ac06af20 0x7fd7ac06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.287+0000 7fd7a7fff700 1 -- 172.21.15.1:0/3756704504 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7fd7b40894e0 con 0x7fd7c0154850 2024-03-18T20:25:56.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.287+0000 7fd7bf7fe700 1 --2- 172.21.15.1:0/3756704504 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd7ac06af20 0x7fd7ac06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.287+0000 7fd7bf7fe700 1 --2- 172.21.15.1:0/3756704504 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd7ac06af20 0x7fd7ac06d3e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0000c00 tx=0x7fd7b0008030 comp rx=0 tx=0).ready entity=mgr.9122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:56.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.291+0000 7fd7a7fff700 1 -- 172.21.15.1:0/3756704504 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd7b4053e90 con 0x7fd7c0154850 2024-03-18T20:25:56.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.520+0000 7fd7c5e6c700 1 -- 172.21.15.1:0/3756704504 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7c0071e80 con 0x7fd7ac06af20 2024-03-18T20:25:56.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.574+0000 7fd7a7fff700 1 -- 172.21.15.1:0/3756704504 <== mgr.9122 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+61 (secure 0 0 0) 0x7fd7b0011630 con 0x7fd7ac06af20 2024-03-18T20:25:56.578 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule deactivated for path mandatory_fs_argument_test_dir 2024-03-18T20:25:56.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.576+0000 7fd7c5e6c700 1 -- 172.21.15.1:0/3756704504 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd7ac06af20 msgr2=0x7fd7ac06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.576+0000 7fd7c5e6c700 1 --2- 172.21.15.1:0/3756704504 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd7ac06af20 0x7fd7ac06d3e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0000c00 tx=0x7fd7b0008030 comp rx=0 tx=0).stop 2024-03-18T20:25:56.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.576+0000 7fd7c5e6c700 1 -- 172.21.15.1:0/3756704504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c0154850 msgr2=0x7fd7c00ce2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.576+0000 7fd7c5e6c700 1 --2- 172.21.15.1:0/3756704504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c0154850 0x7fd7c00ce2a0 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400c520 tx=0x7fd7b400c830 comp rx=0 tx=0).stop 2024-03-18T20:25:56.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.576+0000 7fd7c5e6c700 1 -- 172.21.15.1:0/3756704504 shutdown_connections 2024-03-18T20:25:56.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.576+0000 7fd7c5e6c700 1 --2- 172.21.15.1:0/3756704504 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd7ac06af20 0x7fd7ac06d3e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.576+0000 7fd7c5e6c700 1 --2- 172.21.15.1:0/3756704504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7c00b3a80 0x7fd7c01f1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.576+0000 7fd7c5e6c700 1 --2- 172.21.15.1:0/3756704504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7c015e440 0x7fd7c00aec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.576+0000 7fd7c5e6c700 1 --2- 172.21.15.1:0/3756704504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c0154850 0x7fd7c00ce2a0 unknown :-1 s=CLOSED pgs=2645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.576+0000 7fd7c5e6c700 1 -- 172.21.15.1:0/3756704504 >> 172.21.15.1:0/3756704504 conn(0x7fd7c00d00a0 msgr2=0x7fd7c015be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:56.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.576+0000 7fd7c5e6c700 1 -- 172.21.15.1:0/3756704504 shutdown_connections 2024-03-18T20:25:56.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.576+0000 7fd7c5e6c700 1 -- 172.21.15.1:0/3756704504 wait complete. 2024-03-18T20:25:56.593 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path mandatory_fs_argument_test_dir 2024-03-18T20:25:56.593 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-03-18T20:25:56.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.719+0000 7fe6b2232700 1 Processor -- start 2024-03-18T20:25:56.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.720+0000 7fe6b2232700 1 -- start start 2024-03-18T20:25:56.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.720+0000 7fe6b2232700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe6ac15d430 0x7fe6ac0cea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.720+0000 7fe6b2232700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe6ac15d810 0x7fe6ac0cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.720+0000 7fe6b2232700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ac0d1280 con 0x7fe6ac0cfd50 2024-03-18T20:25:56.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.720+0000 7fe6b2232700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6ac0d1f00 con 0x7fe6ac15d430 2024-03-18T20:25:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.720+0000 7fe6b2232700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6ac0d1560 con 0x7fe6ac15d810 2024-03-18T20:25:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.721+0000 7fe6ab7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe6ac15d430 0x7fe6ac0cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.721+0000 7fe6abfff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe6ac0cfd50 0x7fe6ac0b8440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47230/0 (socket says 172.21.15.1:47230) 2024-03-18T20:25:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.721+0000 7fe6abfff700 1 -- 172.21.15.1:0/1169765870 learned_addr learned my addr 172.21.15.1:0/1169765870 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.721+0000 7fe6aaffd700 1 --2- 172.21.15.1:0/1169765870 >> v2:172.21.15.165:3300/0 conn(0x7fe6ac15d810 0x7fe6ac0cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.721+0000 7fe6ab7fe700 1 -- 172.21.15.1:0/1169765870 >> v2:172.21.15.165:3300/0 conn(0x7fe6ac15d810 msgr2=0x7fe6ac0cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.721+0000 7fe6ab7fe700 1 --2- 172.21.15.1:0/1169765870 >> v2:172.21.15.165:3300/0 conn(0x7fe6ac15d810 0x7fe6ac0cefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.721+0000 7fe6ab7fe700 1 -- 172.21.15.1:0/1169765870 >> v1:172.21.15.1:6789/0 conn(0x7fe6ac0cfd50 legacy=0x7fe6ac0b8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.721+0000 7fe6ab7fe700 1 -- 172.21.15.1:0/1169765870 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ac0d1840 con 0x7fe6ac15d430 2024-03-18T20:25:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.721+0000 7fe6aa7fc700 1 -- 172.21.15.1:0/1169765870 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210527223 0 0) 0x7fe6ac0d1280 con 0x7fe6ac0cfd50 2024-03-18T20:25:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.721+0000 7fe6ab7fe700 1 --2- 172.21.15.1:0/1169765870 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6ac15d430 0x7fe6ac0cea80 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac08d5d0 tx=0x7fe69c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=624ada60b766bf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.721+0000 7fe6aa7fc700 1 -- 172.21.15.1:0/1169765870 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c00e070 con 0x7fe6ac15d430 2024-03-18T20:25:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.721+0000 7fe6aa7fc700 1 -- 172.21.15.1:0/1169765870 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69c00bbd0 con 0x7fe6ac15d430 2024-03-18T20:25:56.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.721+0000 7fe6aa7fc700 1 -- 172.21.15.1:0/1169765870 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c012430 con 0x7fe6ac15d430 2024-03-18T20:25:56.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.722+0000 7fe6b2232700 1 -- 172.21.15.1:0/1169765870 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6ac15d430 msgr2=0x7fe6ac0cea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.722+0000 7fe6b2232700 1 --2- 172.21.15.1:0/1169765870 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6ac15d430 0x7fe6ac0cea80 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac08d5d0 tx=0x7fe69c00b330 comp rx=0 tx=0).stop 2024-03-18T20:25:56.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.722+0000 7fe6b2232700 1 -- 172.21.15.1:0/1169765870 shutdown_connections 2024-03-18T20:25:56.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.722+0000 7fe6b2232700 1 --2- 172.21.15.1:0/1169765870 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6ac15d430 0x7fe6ac0cea80 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.722+0000 7fe6b2232700 1 --2- 172.21.15.1:0/1169765870 >> v2:172.21.15.165:3300/0 conn(0x7fe6ac15d810 0x7fe6ac0cefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.722+0000 7fe6b2232700 1 -- 172.21.15.1:0/1169765870 >> 172.21.15.1:0/1169765870 conn(0x7fe6ac09c630 msgr2=0x7fe6ac0d3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:56.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.722+0000 7fe6b2232700 1 -- 172.21.15.1:0/1169765870 shutdown_connections 2024-03-18T20:25:56.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.722+0000 7fe6b2232700 1 -- 172.21.15.1:0/1169765870 wait complete. 2024-03-18T20:25:56.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.723+0000 7fe6b2232700 1 Processor -- start 2024-03-18T20:25:56.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.723+0000 7fe6b2232700 1 -- start start 2024-03-18T20:25:56.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.724+0000 7fe6b2232700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6ac0cfd50 0x7fe6ac152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.724+0000 7fe6b2232700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6ac15d430 0x7fe6ac145f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.724+0000 7fe6b2232700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6ac15d810 0x7fe6ac0c9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.724+0000 7fe6b2232700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ac152980 con 0x7fe6ac0cfd50 2024-03-18T20:25:56.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.724+0000 7fe6b2232700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ac156140 con 0x7fe6ac15d810 2024-03-18T20:25:56.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.724+0000 7fe6b2232700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ac1464b0 con 0x7fe6ac15d430 2024-03-18T20:25:56.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.724+0000 7fe6ab7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6ac0cfd50 0x7fe6ac152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.724+0000 7fe6ab7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6ac0cfd50 0x7fe6ac152440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33622/0 (socket says 172.21.15.1:33622) 2024-03-18T20:25:56.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.724+0000 7fe6ab7fe700 1 -- 172.21.15.1:0/3302937506 learned_addr learned my addr 172.21.15.1:0/3302937506 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:56.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.724+0000 7fe6aaffd700 1 --2- 172.21.15.1:0/3302937506 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6ac15d430 0x7fe6ac145f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.724+0000 7fe6abfff700 1 --2- 172.21.15.1:0/3302937506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6ac15d810 0x7fe6ac0c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.724+0000 7fe6ab7fe700 1 -- 172.21.15.1:0/3302937506 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6ac15d430 msgr2=0x7fe6ac145f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.724+0000 7fe6ab7fe700 1 --2- 172.21.15.1:0/3302937506 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6ac15d430 0x7fe6ac145f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.724+0000 7fe6ab7fe700 1 -- 172.21.15.1:0/3302937506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6ac15d810 msgr2=0x7fe6ac0c9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:56.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.724+0000 7fe6ab7fe700 1 --2- 172.21.15.1:0/3302937506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6ac15d810 0x7fe6ac0c9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:56.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.724+0000 7fe6ab7fe700 1 -- 172.21.15.1:0/3302937506 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69c009a70 con 0x7fe6ac0cfd50 2024-03-18T20:25:56.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.724+0000 7fe6ab7fe700 1 --2- 172.21.15.1:0/3302937506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6ac0cfd50 0x7fe6ac152440 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7fe69c014040 tx=0x7fe69c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:56.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.725+0000 7fe6a3fff700 1 -- 172.21.15.1:0/3302937506 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c012430 con 0x7fe6ac0cfd50 2024-03-18T20:25:56.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.725+0000 7fe6a3fff700 1 -- 172.21.15.1:0/3302937506 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69c020070 con 0x7fe6ac0cfd50 2024-03-18T20:25:56.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.725+0000 7fe6b2232700 1 -- 172.21.15.1:0/3302937506 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6ac155530 con 0x7fe6ac0cfd50 2024-03-18T20:25:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.725+0000 7fe6a3fff700 1 -- 172.21.15.1:0/3302937506 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c004280 con 0x7fe6ac0cfd50 2024-03-18T20:25:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.725+0000 7fe6b2232700 1 -- 172.21.15.1:0/3302937506 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6ac155af0 con 0x7fe6ac0cfd50 2024-03-18T20:25:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.726+0000 7fe6b2232700 1 -- 172.21.15.1:0/3302937506 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6ac0d2300 con 0x7fe6ac0cfd50 2024-03-18T20:25:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.726+0000 7fe6a3fff700 1 -- 172.21.15.1:0/3302937506 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 140) v1 ==== 89359+0+0 (secure 0 0 0) 0x7fe69c0043e0 con 0x7fe6ac0cfd50 2024-03-18T20:25:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.727+0000 7fe6a3fff700 1 --2- 172.21.15.1:0/3302937506 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe68c06af20 0x7fe68c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.727+0000 7fe6a3fff700 1 -- 172.21.15.1:0/3302937506 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7fe69c022070 con 0x7fe6ac0cfd50 2024-03-18T20:25:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.727+0000 7fe6aaffd700 1 --2- 172.21.15.1:0/3302937506 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe68c06af20 0x7fe68c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:56.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.727+0000 7fe6aaffd700 1 --2- 172.21.15.1:0/3302937506 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe68c06af20 0x7fe68c06d3e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac0ced90 tx=0x7fe694008040 comp rx=0 tx=0).ready entity=mgr.9122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:56.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.731+0000 7fe6a3fff700 1 -- 172.21.15.1:0/3302937506 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe6ac0d2300 con 0x7fe6ac0cfd50 2024-03-18T20:25:56.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.961+0000 7fe6b2232700 1 -- 172.21.15.1:0/3302937506 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6ac0d0f10 con 0x7fe68c06af20 2024-03-18T20:25:56.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:56.969+0000 7fe6a3fff700 1 -- 172.21.15.1:0/3302937506 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 141) v1 ==== 89415+0+0 (secure 0 0 0) 0x7fe69c0566d0 con 0x7fe6ac0cfd50 2024-03-18T20:25:57.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.025+0000 7fe6a3fff700 1 -- 172.21.15.1:0/3302937506 <== mgr.9122 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7fe6940076b0 con 0x7fe68c06af20 2024-03-18T20:25:57.030 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule removed for path mandatory_fs_argument_test_dir 2024-03-18T20:25:57.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.027+0000 7fe6b2232700 1 -- 172.21.15.1:0/3302937506 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe68c06af20 msgr2=0x7fe68c06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.028+0000 7fe6b2232700 1 --2- 172.21.15.1:0/3302937506 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe68c06af20 0x7fe68c06d3e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac0ced90 tx=0x7fe694008040 comp rx=0 tx=0).stop 2024-03-18T20:25:57.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.028+0000 7fe6b2232700 1 -- 172.21.15.1:0/3302937506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6ac0cfd50 msgr2=0x7fe6ac152440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.028+0000 7fe6b2232700 1 --2- 172.21.15.1:0/3302937506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6ac0cfd50 0x7fe6ac152440 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7fe69c014040 tx=0x7fe69c004ad0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.028+0000 7fe6b2232700 1 -- 172.21.15.1:0/3302937506 shutdown_connections 2024-03-18T20:25:57.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.028+0000 7fe6b2232700 1 --2- 172.21.15.1:0/3302937506 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe68c06af20 0x7fe68c06d3e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.028+0000 7fe6b2232700 1 --2- 172.21.15.1:0/3302937506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6ac15d810 0x7fe6ac0c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.028+0000 7fe6b2232700 1 --2- 172.21.15.1:0/3302937506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6ac0cfd50 0x7fe6ac152440 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.028+0000 7fe6b2232700 1 --2- 172.21.15.1:0/3302937506 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6ac15d430 0x7fe6ac145f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.028+0000 7fe6b2232700 1 -- 172.21.15.1:0/3302937506 >> 172.21.15.1:0/3302937506 conn(0x7fe6ac09c630 msgr2=0x7fe6ac0b4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:57.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.028+0000 7fe6b2232700 1 -- 172.21.15.1:0/3302937506 shutdown_connections 2024-03-18T20:25:57.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.028+0000 7fe6b2232700 1 -- 172.21.15.1:0/3302937506 wait complete. 2024-03-18T20:25:57.044 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path mandatory_fs_argument_test_dir 2024-03-18T20:25:57.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:25:57.044 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2024-03-18T20:25:57.068 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-18T20:25:57.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.232+0000 7fc0ef72b700 1 Processor -- start 2024-03-18T20:25:57.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.232+0000 7fc0ef72b700 1 -- start start 2024-03-18T20:25:57.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.233+0000 7fc0ef72b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc0e80cfb50 0x7fc0e80d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:57.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.233+0000 7fc0ef72b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e80cff30 0x7fc0e80b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:57.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.233+0000 7fc0ef72b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e80b3370 con 0x7fc0e80b3d40 2024-03-18T20:25:57.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.233+0000 7fc0ef72b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0e80b2760 con 0x7fc0e80cfb50 2024-03-18T20:25:57.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.233+0000 7fc0ef72b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e80b2a40 con 0x7fc0e80cff30 2024-03-18T20:25:57.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.233+0000 7fc0ed4c7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc0e80cfb50 0x7fc0e80d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:57.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.233+0000 7fc0ed4c7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc0e80cfb50 0x7fc0e80d3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33636/0 (socket says 172.21.15.1:33636) 2024-03-18T20:25:57.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.233+0000 7fc0ed4c7700 1 -- 172.21.15.1:0/3276315081 learned_addr learned my addr 172.21.15.1:0/3276315081 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:57.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.233+0000 7fc0eccc6700 1 --2- 172.21.15.1:0/3276315081 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e80cff30 0x7fc0e80b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:57.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.233+0000 7fc0ed4c7700 1 -- 172.21.15.1:0/3276315081 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e80cff30 msgr2=0x7fc0e80b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:57.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.233+0000 7fc0ed4c7700 1 --2- 172.21.15.1:0/3276315081 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e80cff30 0x7fc0e80b3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.233+0000 7fc0ed4c7700 1 -- 172.21.15.1:0/3276315081 >> v1:172.21.15.165:6789/0 conn(0x7fc0e80b3d40 legacy=0x7fc0e80b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.233+0000 7fc0ed4c7700 1 -- 172.21.15.1:0/3276315081 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e80b2d20 con 0x7fc0e80cfb50 2024-03-18T20:25:57.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.234+0000 7fc0ed4c7700 1 --2- 172.21.15.1:0/3276315081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0e80cfb50 0x7fc0e80d3c60 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8006fd0 tx=0x7fc0d8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8111706d3bcbb8e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:57.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.234+0000 7fc0dffff700 1 -- 172.21.15.1:0/3276315081 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8005e30 con 0x7fc0e80cfb50 2024-03-18T20:25:57.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.234+0000 7fc0dffff700 1 -- 172.21.15.1:0/3276315081 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d8004030 con 0x7fc0e80cfb50 2024-03-18T20:25:57.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.234+0000 7fc0dffff700 1 -- 172.21.15.1:0/3276315081 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d80043e0 con 0x7fc0e80cfb50 2024-03-18T20:25:57.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.234+0000 7fc0ef72b700 1 -- 172.21.15.1:0/3276315081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0e80cfb50 msgr2=0x7fc0e80d3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.234+0000 7fc0ef72b700 1 --2- 172.21.15.1:0/3276315081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0e80cfb50 0x7fc0e80d3c60 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8006fd0 tx=0x7fc0d8009bd0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.234+0000 7fc0ef72b700 1 -- 172.21.15.1:0/3276315081 shutdown_connections 2024-03-18T20:25:57.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.234+0000 7fc0ef72b700 1 --2- 172.21.15.1:0/3276315081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0e80cfb50 0x7fc0e80d3c60 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.234+0000 7fc0ef72b700 1 --2- 172.21.15.1:0/3276315081 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e80cff30 0x7fc0e80b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.234+0000 7fc0ef72b700 1 -- 172.21.15.1:0/3276315081 >> 172.21.15.1:0/3276315081 conn(0x7fc0e80d3140 msgr2=0x7fc0e80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:57.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.234+0000 7fc0ef72b700 1 -- 172.21.15.1:0/3276315081 shutdown_connections 2024-03-18T20:25:57.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.234+0000 7fc0ef72b700 1 -- 172.21.15.1:0/3276315081 wait complete. 2024-03-18T20:25:57.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.235+0000 7fc0ef72b700 1 Processor -- start 2024-03-18T20:25:57.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.236+0000 7fc0ef72b700 1 -- start start 2024-03-18T20:25:57.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.236+0000 7fc0ef72b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0e80b3d40 0x7fc0e8159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:57.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.236+0000 7fc0ef72b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0e80cfb50 0x7fc0e815c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:57.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.236+0000 7fc0ef72b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e80cff30 0x7fc0e806b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:57.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.236+0000 7fc0ef72b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e80ccff0 con 0x7fc0e80cfb50 2024-03-18T20:25:57.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.236+0000 7fc0ef72b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e80cadf0 con 0x7fc0e80b3d40 2024-03-18T20:25:57.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.236+0000 7fc0ef72b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e80cb0d0 con 0x7fc0e80cff30 2024-03-18T20:25:57.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.236+0000 7fc0eccc6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0e80cfb50 0x7fc0e815c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:57.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.236+0000 7fc0eccc6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0e80cfb50 0x7fc0e815c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33648/0 (socket says 172.21.15.1:33648) 2024-03-18T20:25:57.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.236+0000 7fc0eccc6700 1 -- 172.21.15.1:0/792486108 learned_addr learned my addr 172.21.15.1:0/792486108 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:57.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.236+0000 7fc0ed4c7700 1 --2- 172.21.15.1:0/792486108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0e80b3d40 0x7fc0e8159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:57.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.236+0000 7fc0edcc8700 1 --2- 172.21.15.1:0/792486108 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e80cff30 0x7fc0e806b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:57.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.237+0000 7fc0eccc6700 1 -- 172.21.15.1:0/792486108 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e80cff30 msgr2=0x7fc0e806b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.237+0000 7fc0eccc6700 1 --2- 172.21.15.1:0/792486108 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e80cff30 0x7fc0e806b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.237+0000 7fc0eccc6700 1 -- 172.21.15.1:0/792486108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0e80b3d40 msgr2=0x7fc0e8159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.237+0000 7fc0eccc6700 1 --2- 172.21.15.1:0/792486108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0e80b3d40 0x7fc0e8159cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.237+0000 7fc0eccc6700 1 -- 172.21.15.1:0/792486108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d8009500 con 0x7fc0e80cfb50 2024-03-18T20:25:57.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.237+0000 7fc0ed4c7700 1 --2- 172.21.15.1:0/792486108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0e80b3d40 0x7fc0e8159cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:25:57.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.237+0000 7fc0eccc6700 1 --2- 172.21.15.1:0/792486108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0e80cfb50 0x7fc0e815c220 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400d800 tx=0x7fc0e400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:57.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.237+0000 7fc0ddffb700 1 -- 172.21.15.1:0/792486108 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e40125f0 con 0x7fc0e80cfb50 2024-03-18T20:25:57.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.237+0000 7fc0ddffb700 1 -- 172.21.15.1:0/792486108 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e4012750 con 0x7fc0e80cfb50 2024-03-18T20:25:57.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.237+0000 7fc0ddffb700 1 -- 172.21.15.1:0/792486108 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e4012b00 con 0x7fc0e80cfb50 2024-03-18T20:25:57.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.237+0000 7fc0ef72b700 1 -- 172.21.15.1:0/792486108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e80cb3b0 con 0x7fc0e80cfb50 2024-03-18T20:25:57.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.237+0000 7fc0ef72b700 1 -- 172.21.15.1:0/792486108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e806ba60 con 0x7fc0e80cfb50 2024-03-18T20:25:57.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.238+0000 7fc0ef72b700 1 -- 172.21.15.1:0/792486108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0e80b2360 con 0x7fc0e80cfb50 2024-03-18T20:25:57.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.239+0000 7fc0ddffb700 1 -- 172.21.15.1:0/792486108 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 141) v1 ==== 89415+0+0 (secure 0 0 0) 0x7fc0e4012c60 con 0x7fc0e80cfb50 2024-03-18T20:25:57.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.239+0000 7fc0ddffb700 1 --2- 172.21.15.1:0/792486108 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc0cc06afe0 0x7fc0cc06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:57.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.239+0000 7fc0ddffb700 1 -- 172.21.15.1:0/792486108 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7fc0e408a220 con 0x7fc0e80cfb50 2024-03-18T20:25:57.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.239+0000 7fc0ed4c7700 1 --2- 172.21.15.1:0/792486108 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc0cc06afe0 0x7fc0cc06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:57.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.239+0000 7fc0ed4c7700 1 --2- 172.21.15.1:0/792486108 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc0cc06afe0 0x7fc0cc06d4a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8009130 tx=0x7fc0d8008070 comp rx=0 tx=0).ready entity=mgr.9122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:57.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.244+0000 7fc0ddffb700 1 -- 172.21.15.1:0/792486108 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc0e40549d0 con 0x7fc0e80cfb50 2024-03-18T20:25:57.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.504+0000 7fc0ef72b700 1 -- 172.21.15.1:0/792486108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fc0e806bbf0 con 0x7fc0e80cfb50 2024-03-18T20:25:57.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.974+0000 7fc0ddffb700 1 -- 172.21.15.1:0/792486108 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 142) v1 ==== 89398+0+0 (secure 0 0 0) 0x7fc0e4054da0 con 0x7fc0e80cfb50 2024-03-18T20:25:57.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.975+0000 7fc0ddffb700 1 -- 172.21.15.1:0/792486108 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v142) v1 ==== 93+0+0 (secure 0 0 0) 0x7fc0e4073af0 con 0x7fc0e80cfb50 2024-03-18T20:25:57.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.978+0000 7fc0ef72b700 1 -- 172.21.15.1:0/792486108 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc0cc06afe0 msgr2=0x7fc0cc06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.978+0000 7fc0ef72b700 1 --2- 172.21.15.1:0/792486108 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc0cc06afe0 0x7fc0cc06d4a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8009130 tx=0x7fc0d8008070 comp rx=0 tx=0).stop 2024-03-18T20:25:57.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.978+0000 7fc0ef72b700 1 -- 172.21.15.1:0/792486108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0e80cfb50 msgr2=0x7fc0e815c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:57.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.978+0000 7fc0ef72b700 1 --2- 172.21.15.1:0/792486108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0e80cfb50 0x7fc0e815c220 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400d800 tx=0x7fc0e400dbc0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.978+0000 7fc0ef72b700 1 -- 172.21.15.1:0/792486108 shutdown_connections 2024-03-18T20:25:57.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.978+0000 7fc0ef72b700 1 --2- 172.21.15.1:0/792486108 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc0cc06afe0 0x7fc0cc06d4a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.978+0000 7fc0ef72b700 1 --2- 172.21.15.1:0/792486108 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc0e80cff30 0x7fc0e806b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.978+0000 7fc0ef72b700 1 --2- 172.21.15.1:0/792486108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc0e80b3d40 0x7fc0e8159cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.978+0000 7fc0ef72b700 1 --2- 172.21.15.1:0/792486108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0e80cfb50 0x7fc0e815c220 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:57.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.978+0000 7fc0ef72b700 1 -- 172.21.15.1:0/792486108 >> 172.21.15.1:0/792486108 conn(0x7fc0e80d3140 msgr2=0x7fc0e815e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:57.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.978+0000 7fc0ef72b700 1 -- 172.21.15.1:0/792486108 shutdown_connections 2024-03-18T20:25:57.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:57.978+0000 7fc0ef72b700 1 -- 172.21.15.1:0/792486108 wait complete. 2024-03-18T20:25:57.997 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T20:25:57.997 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:25:58.041 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:25:58.083 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T20:25:58.083 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:25:58.083 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:25:58.102 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:25:58.121 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:58.118+0000 7f583fa9c240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:25:58.131 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:58.127+0000 7f47e8e57240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:25:58.266 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:58.263+0000 7f583fa9c240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:25:58.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.292+0000 7faff7089700 1 Processor -- start 2024-03-18T20:25:58.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.292+0000 7faff7089700 1 -- start start 2024-03-18T20:25:58.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.294+0000 7faff7089700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faff008a7c0 0x7faff016b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.294+0000 7faff7089700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faff008a270 0x7faff016a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.294+0000 7faff7089700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff0070f30 con 0x7faff006a750 2024-03-18T20:25:58.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.294+0000 7faff7089700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff016dcc0 con 0x7faff008a270 2024-03-18T20:25:58.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.294+0000 7faff7089700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff016dfa0 con 0x7faff008a7c0 2024-03-18T20:25:58.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.294+0000 7fafeffff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faff008a7c0 0x7faff016b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:58.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.294+0000 7faff4e25700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7faff006a750 0x7faff006ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47256/0 (socket says 172.21.15.1:47256) 2024-03-18T20:25:58.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.294+0000 7faff4e25700 1 -- 172.21.15.1:0/2278456444 learned_addr learned my addr 172.21.15.1:0/2278456444 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:58.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.295+0000 7fafef7fe700 1 -- 172.21.15.1:0/2278456444 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 313882881 0 0) 0x7faff0070f30 con 0x7faff006a750 2024-03-18T20:25:58.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.295+0000 7fafef7fe700 1 -- 172.21.15.1:0/2278456444 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafd8003590 con 0x7faff006a750 2024-03-18T20:25:58.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.295+0000 7fafeffff700 1 -- 172.21.15.1:0/2278456444 >> v2:172.21.15.1:3300/0 conn(0x7faff008a270 msgr2=0x7faff016a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:58.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.295+0000 7fafeffff700 1 --2- 172.21.15.1:0/2278456444 >> v2:172.21.15.1:3300/0 conn(0x7faff008a270 0x7faff016a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.295+0000 7fafeffff700 1 -- 172.21.15.1:0/2278456444 >> v1:172.21.15.1:6789/0 conn(0x7faff006a750 legacy=0x7faff006ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.295+0000 7fafef7fe700 1 -- 172.21.15.1:0/2278456444 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4072051659 0 0) 0x7fafd8003590 con 0x7faff006a750 2024-03-18T20:25:58.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.295+0000 7fafeffff700 1 -- 172.21.15.1:0/2278456444 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff016e280 con 0x7faff008a7c0 2024-03-18T20:25:58.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.295+0000 7fafeffff700 1 --2- 172.21.15.1:0/2278456444 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faff008a7c0 0x7faff016b920 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fafe000a8e0 tx=0x7fafe000abf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=276abff78517cfc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:58.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.296+0000 7fafef7fe700 1 -- 172.21.15.1:0/2278456444 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe80019f0 con 0x7faff008a7c0 2024-03-18T20:25:58.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.296+0000 7fafef7fe700 1 -- 172.21.15.1:0/2278456444 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe0004030 con 0x7faff008a7c0 2024-03-18T20:25:58.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.296+0000 7fafef7fe700 1 -- 172.21.15.1:0/2278456444 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe00043c0 con 0x7faff008a7c0 2024-03-18T20:25:58.302 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:58.296+0000 7f47e8e57240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:25:58.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.297+0000 7faff7089700 1 -- 172.21.15.1:0/2278456444 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faff008a7c0 msgr2=0x7faff016b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.297+0000 7faff7089700 1 --2- 172.21.15.1:0/2278456444 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faff008a7c0 0x7faff016b920 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fafe000a8e0 tx=0x7fafe000abf0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.297+0000 7faff7089700 1 -- 172.21.15.1:0/2278456444 shutdown_connections 2024-03-18T20:25:58.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.297+0000 7faff7089700 1 --2- 172.21.15.1:0/2278456444 >> v2:172.21.15.1:3300/0 conn(0x7faff008a270 0x7faff016a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.297+0000 7faff7089700 1 --2- 172.21.15.1:0/2278456444 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faff008a7c0 0x7faff016b920 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.297+0000 7faff7089700 1 -- 172.21.15.1:0/2278456444 >> 172.21.15.1:0/2278456444 conn(0x7faff00b0a00 msgr2=0x7faff00b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:58.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.297+0000 7faff7089700 1 -- 172.21.15.1:0/2278456444 shutdown_connections 2024-03-18T20:25:58.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.297+0000 7faff7089700 1 -- 172.21.15.1:0/2278456444 wait complete. 2024-03-18T20:25:58.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.298+0000 7faff7089700 1 Processor -- start 2024-03-18T20:25:58.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.298+0000 7faff7089700 1 -- start start 2024-03-18T20:25:58.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.299+0000 7faff7089700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faff006a750 0x7faff0169960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.299+0000 7faff7089700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faff008a270 0x7faff0099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.299+0000 7faff7089700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faff008a7c0 0x7faff009b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.299+0000 7faff7089700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff009ad50 con 0x7faff006a750 2024-03-18T20:25:58.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.299+0000 7faff7089700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff015cc40 con 0x7faff008a7c0 2024-03-18T20:25:58.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.299+0000 7faff7089700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff009a140 con 0x7faff008a270 2024-03-18T20:25:58.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.299+0000 7faff4e25700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faff006a750 0x7faff0169960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:58.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.299+0000 7fafeffff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faff008a270 0x7faff0099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:58.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.299+0000 7faff4e25700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faff006a750 0x7faff0169960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33672/0 (socket says 172.21.15.1:33672) 2024-03-18T20:25:58.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.299+0000 7faff4e25700 1 -- 172.21.15.1:0/304706135 learned_addr learned my addr 172.21.15.1:0/304706135 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:58.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.299+0000 7faff5626700 1 --2- 172.21.15.1:0/304706135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faff008a7c0 0x7faff009b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:58.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.299+0000 7faff4e25700 1 -- 172.21.15.1:0/304706135 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faff008a270 msgr2=0x7faff0099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.299+0000 7faff4e25700 1 --2- 172.21.15.1:0/304706135 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faff008a270 0x7faff0099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.299+0000 7faff4e25700 1 -- 172.21.15.1:0/304706135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faff008a7c0 msgr2=0x7faff009b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.299+0000 7faff4e25700 1 --2- 172.21.15.1:0/304706135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faff008a7c0 0x7faff009b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.299+0000 7faff4e25700 1 -- 172.21.15.1:0/304706135 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe000a550 con 0x7faff006a750 2024-03-18T20:25:58.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.299+0000 7fafeffff700 1 --2- 172.21.15.1:0/304706135 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faff008a270 0x7faff0099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:58.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.300+0000 7faff4e25700 1 --2- 172.21.15.1:0/304706135 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faff006a750 0x7faff0169960 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7fafe8001980 tx=0x7fafe800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:58.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.300+0000 7fafed7fa700 1 -- 172.21.15.1:0/304706135 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe800ac90 con 0x7faff006a750 2024-03-18T20:25:58.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.300+0000 7fafed7fa700 1 -- 172.21.15.1:0/304706135 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe800adf0 con 0x7faff006a750 2024-03-18T20:25:58.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.300+0000 7faff7089700 1 -- 172.21.15.1:0/304706135 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff009a420 con 0x7faff006a750 2024-03-18T20:25:58.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.300+0000 7fafed7fa700 1 -- 172.21.15.1:0/304706135 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe8007310 con 0x7faff006a750 2024-03-18T20:25:58.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.300+0000 7faff7089700 1 -- 172.21.15.1:0/304706135 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff009a9e0 con 0x7faff006a750 2024-03-18T20:25:58.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.301+0000 7faff7089700 1 -- 172.21.15.1:0/304706135 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff00671c0 con 0x7faff006a750 2024-03-18T20:25:58.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.302+0000 7fafed7fa700 1 -- 172.21.15.1:0/304706135 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 142) v1 ==== 89398+0+0 (secure 0 0 0) 0x7fafe8007470 con 0x7faff006a750 2024-03-18T20:25:58.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.302+0000 7fafed7fa700 1 --2- 172.21.15.1:0/304706135 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fafdc06af60 0x7fafdc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.302+0000 7fafeffff700 1 -- 172.21.15.1:0/304706135 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fafdc06af60 msgr2=0x7fafdc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:58.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.302+0000 7fafeffff700 1 --2- 172.21.15.1:0/304706135 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fafdc06af60 0x7fafdc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:58.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.305+0000 7fafed7fa700 1 -- 172.21.15.1:0/304706135 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7fafe80586e0 con 0x7faff006a750 2024-03-18T20:25:58.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.307+0000 7fafed7fa700 1 -- 172.21.15.1:0/304706135 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fafe808e060 con 0x7faff006a750 2024-03-18T20:25:58.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.503+0000 7fafeffff700 1 -- 172.21.15.1:0/304706135 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fafdc06af60 msgr2=0x7fafdc06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:58.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.503+0000 7fafeffff700 1 --2- 172.21.15.1:0/304706135 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fafdc06af60 0x7fafdc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:58.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.513+0000 7faff7089700 1 -- 172.21.15.1:0/304706135 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faff0071f60 con 0x7faff006a750 2024-03-18T20:25:58.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.514+0000 7fafed7fa700 1 -- 172.21.15.1:0/304706135 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v237) v1 ==== 74+0+20570 (secure 0 0 0) 0x7fafe808e020 con 0x7faff006a750 2024-03-18T20:25:58.518 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:25:58.519 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":237,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:25:52.933780+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:25:20.511297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-18T20:25:21.637693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:25:33.275375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:25:34.518347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2535232994":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2652258802":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2404324807":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/3657737679":"2024-03-19T20:25:52.933722+0000","172.21.15.1:6833/148356":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2324250327":"2024-03-19T20:25:52.933722+0000","172.21.15.1:6832/148356":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2165965129":"2024-03-19T20:25:52.933722+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:25:58.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.516+0000 7faff7089700 1 -- 172.21.15.1:0/304706135 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fafdc06af60 msgr2=0x7fafdc06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:58.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.516+0000 7faff7089700 1 --2- 172.21.15.1:0/304706135 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fafdc06af60 0x7fafdc06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.516+0000 7faff7089700 1 -- 172.21.15.1:0/304706135 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faff006a750 msgr2=0x7faff0169960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.516+0000 7faff7089700 1 --2- 172.21.15.1:0/304706135 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faff006a750 0x7faff0169960 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7fafe8001980 tx=0x7fafe800a340 comp rx=0 tx=0).stop 2024-03-18T20:25:58.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.517+0000 7faff7089700 1 -- 172.21.15.1:0/304706135 shutdown_connections 2024-03-18T20:25:58.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.517+0000 7faff7089700 1 --2- 172.21.15.1:0/304706135 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fafdc06af60 0x7fafdc06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.517+0000 7faff7089700 1 --2- 172.21.15.1:0/304706135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faff008a7c0 0x7faff009b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.517+0000 7faff7089700 1 --2- 172.21.15.1:0/304706135 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faff006a750 0x7faff0169960 unknown :-1 s=CLOSED pgs=2655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.517+0000 7faff7089700 1 --2- 172.21.15.1:0/304706135 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faff008a270 0x7faff0099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.517+0000 7faff7089700 1 -- 172.21.15.1:0/304706135 >> 172.21.15.1:0/304706135 conn(0x7faff00b0a00 msgr2=0x7faff00ae970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:58.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.517+0000 7faff7089700 1 -- 172.21.15.1:0/304706135 shutdown_connections 2024-03-18T20:25:58.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.517+0000 7faff7089700 1 -- 172.21.15.1:0/304706135 wait complete. 2024-03-18T20:25:58.531 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:58.527+0000 7f583fa9c240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:25:58.534 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:25:20.511297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '231', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:25:21.637693+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '232', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-03-18T20:25:33.275375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '235', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2024-03-18T20:25:34.518347+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '236', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:25:58.534 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:25:58.579 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:58.575+0000 7f47e8e57240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:25:58.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.661+0000 7fef064d7700 1 Processor -- start 2024-03-18T20:25:58.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.661+0000 7fef064d7700 1 -- start start 2024-03-18T20:25:58.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.662+0000 7fef064d7700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fef00099a40 0x7fef000ade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.662+0000 7fef064d7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fef00099e20 0x7fef000cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.662+0000 7fef064d7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef000ccc20 0x7fef0006fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.662+0000 7fef064d7700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef000b3aa0 con 0x7fef00099e20 2024-03-18T20:25:58.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.662+0000 7fef064d7700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef0006e9b0 con 0x7fef00099a40 2024-03-18T20:25:58.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.662+0000 7fef064d7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef0006ec90 con 0x7fef000ccc20 2024-03-18T20:25:58.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.662+0000 7feeff7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fef00099e20 0x7fef000cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:58.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.662+0000 7feeff7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fef00099e20 0x7fef000cc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33684/0 (socket says 172.21.15.1:33684) 2024-03-18T20:25:58.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.662+0000 7feeff7fe700 1 -- 172.21.15.1:0/3135182228 learned_addr learned my addr 172.21.15.1:0/3135182228 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:58.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.662+0000 7feeff7fe700 1 -- 172.21.15.1:0/3135182228 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef000ccc20 msgr2=0x7fef0006fa50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T20:25:58.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.662+0000 7feeff7fe700 1 --2- 172.21.15.1:0/3135182228 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef000ccc20 0x7fef0006fa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.662+0000 7feeff7fe700 1 -- 172.21.15.1:0/3135182228 >> v2:172.21.15.165:3300/0 conn(0x7fef00099a40 msgr2=0x7fef000ade00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T20:25:58.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.662+0000 7feeff7fe700 1 --2- 172.21.15.1:0/3135182228 >> v2:172.21.15.165:3300/0 conn(0x7fef00099a40 0x7fef000ade00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.662+0000 7feeff7fe700 1 -- 172.21.15.1:0/3135182228 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0006ef70 con 0x7fef00099e20 2024-03-18T20:25:58.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.663+0000 7feeff7fe700 1 --2- 172.21.15.1:0/3135182228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef00099e20 0x7fef000cc6e0 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7fef00071d40 tx=0x7feef4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e6aaeea6259f6bb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:58.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.663+0000 7feefeffd700 1 -- 172.21.15.1:0/3135182228 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef400b870 con 0x7fef00099e20 2024-03-18T20:25:58.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.663+0000 7feefeffd700 1 -- 172.21.15.1:0/3135182228 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef400b9d0 con 0x7fef00099e20 2024-03-18T20:25:58.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.663+0000 7feefeffd700 1 -- 172.21.15.1:0/3135182228 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef400bd60 con 0x7fef00099e20 2024-03-18T20:25:58.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.663+0000 7fef064d7700 1 -- 172.21.15.1:0/3135182228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef00099e20 msgr2=0x7fef000cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.663+0000 7fef064d7700 1 --2- 172.21.15.1:0/3135182228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef00099e20 0x7fef000cc6e0 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7fef00071d40 tx=0x7feef4009bd0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.664+0000 7fef064d7700 1 -- 172.21.15.1:0/3135182228 shutdown_connections 2024-03-18T20:25:58.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.664+0000 7fef064d7700 1 --2- 172.21.15.1:0/3135182228 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef000ccc20 0x7fef0006fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.664+0000 7fef064d7700 1 --2- 172.21.15.1:0/3135182228 >> v2:172.21.15.165:3300/0 conn(0x7fef00099a40 0x7fef000ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.664+0000 7fef064d7700 1 --2- 172.21.15.1:0/3135182228 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef00099e20 0x7fef000cc6e0 unknown :-1 s=CLOSED pgs=2656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.664+0000 7fef064d7700 1 -- 172.21.15.1:0/3135182228 >> 172.21.15.1:0/3135182228 conn(0x7fef000b2c50 msgr2=0x7fef000b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:58.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.664+0000 7fef064d7700 1 -- 172.21.15.1:0/3135182228 shutdown_connections 2024-03-18T20:25:58.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.664+0000 7fef064d7700 1 -- 172.21.15.1:0/3135182228 wait complete. 2024-03-18T20:25:58.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.666+0000 7fef064d7700 1 Processor -- start 2024-03-18T20:25:58.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.666+0000 7fef064d7700 1 -- start start 2024-03-18T20:25:58.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.667+0000 7fef064d7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef00099a40 0x7fef0015d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.667+0000 7fef064d7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef00099e20 0x7fef000af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.667+0000 7fef064d7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef000ccc20 0x7fef00151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.667+0000 7fef064d7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef000c5c60 con 0x7fef00099e20 2024-03-18T20:25:58.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.667+0000 7fef064d7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef000c5050 con 0x7fef00099a40 2024-03-18T20:25:58.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.667+0000 7fef064d7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef000c5330 con 0x7fef000ccc20 2024-03-18T20:25:58.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.667+0000 7feeff7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef00099e20 0x7fef000af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:58.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.667+0000 7feeff7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef00099e20 0x7fef000af3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33696/0 (socket says 172.21.15.1:33696) 2024-03-18T20:25:58.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.667+0000 7feeff7fe700 1 -- 172.21.15.1:0/464766112 learned_addr learned my addr 172.21.15.1:0/464766112 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:58.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.668+0000 7feeff7fe700 1 -- 172.21.15.1:0/464766112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef000ccc20 msgr2=0x7fef00151740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.668+0000 7feeff7fe700 1 --2- 172.21.15.1:0/464766112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef000ccc20 0x7fef00151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.668+0000 7feeff7fe700 1 -- 172.21.15.1:0/464766112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef00099a40 msgr2=0x7fef0015d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.668+0000 7feeff7fe700 1 --2- 172.21.15.1:0/464766112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef00099a40 0x7fef0015d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.668+0000 7feeff7fe700 1 -- 172.21.15.1:0/464766112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef4009550 con 0x7fef00099e20 2024-03-18T20:25:58.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.668+0000 7feeff7fe700 1 --2- 172.21.15.1:0/464766112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef00099e20 0x7fef000af3b0 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7feef4009c10 tx=0x7feef4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:58.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.668+0000 7feefcff9700 1 -- 172.21.15.1:0/464766112 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef40036a0 con 0x7fef00099e20 2024-03-18T20:25:58.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.668+0000 7feefcff9700 1 -- 172.21.15.1:0/464766112 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef4003800 con 0x7fef00099e20 2024-03-18T20:25:58.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.668+0000 7feefcff9700 1 -- 172.21.15.1:0/464766112 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef4003b90 con 0x7fef00099e20 2024-03-18T20:25:58.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.668+0000 7fef064d7700 1 -- 172.21.15.1:0/464766112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef000c5610 con 0x7fef00099e20 2024-03-18T20:25:58.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.668+0000 7fef064d7700 1 -- 172.21.15.1:0/464766112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef00152a00 con 0x7fef00099e20 2024-03-18T20:25:58.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.669+0000 7fef064d7700 1 -- 172.21.15.1:0/464766112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef00152b90 con 0x7fef00099e20 2024-03-18T20:25:58.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.670+0000 7feefcff9700 1 -- 172.21.15.1:0/464766112 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 142) v1 ==== 89398+0+0 (secure 0 0 0) 0x7feef4021070 con 0x7fef00099e20 2024-03-18T20:25:58.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.670+0000 7feefcff9700 1 --2- 172.21.15.1:0/464766112 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7feeec06af60 0x7feeec06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:58.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.670+0000 7feefffff700 1 -- 172.21.15.1:0/464766112 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7feeec06af60 msgr2=0x7feeec06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:58.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.670+0000 7feefcff9700 1 -- 172.21.15.1:0/464766112 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7feef408b520 con 0x7fef00099e20 2024-03-18T20:25:58.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.670+0000 7feefffff700 1 --2- 172.21.15.1:0/464766112 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7feeec06af60 0x7feeec06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:58.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.674+0000 7feefcff9700 1 -- 172.21.15.1:0/464766112 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feef4056000 con 0x7fef00099e20 2024-03-18T20:25:58.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.871+0000 7feefffff700 1 -- 172.21.15.1:0/464766112 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7feeec06af60 msgr2=0x7feeec06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:58.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.871+0000 7feefffff700 1 --2- 172.21.15.1:0/464766112 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7feeec06af60 0x7feeec06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:58.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.874+0000 7fef064d7700 1 -- 172.21.15.1:0/464766112 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fef000676a0 con 0x7fef00099e20 2024-03-18T20:25:58.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.875+0000 7feefcff9700 1 -- 172.21.15.1:0/464766112 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v237) v1 ==== 93+0+10 (secure 0 0 0) 0x7feef4028020 con 0x7fef00099e20 2024-03-18T20:25:58.879 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:25:58.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.877+0000 7fef064d7700 1 -- 172.21.15.1:0/464766112 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7feeec06af60 msgr2=0x7feeec06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:58.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.877+0000 7fef064d7700 1 --2- 172.21.15.1:0/464766112 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7feeec06af60 0x7feeec06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.877+0000 7fef064d7700 1 -- 172.21.15.1:0/464766112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef00099e20 msgr2=0x7fef000af3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:58.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.877+0000 7fef064d7700 1 --2- 172.21.15.1:0/464766112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef00099e20 0x7fef000af3b0 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7feef4009c10 tx=0x7feef4004750 comp rx=0 tx=0).stop 2024-03-18T20:25:58.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.877+0000 7fef064d7700 1 -- 172.21.15.1:0/464766112 shutdown_connections 2024-03-18T20:25:58.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.877+0000 7fef064d7700 1 --2- 172.21.15.1:0/464766112 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7feeec06af60 0x7feeec06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.877+0000 7fef064d7700 1 --2- 172.21.15.1:0/464766112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef000ccc20 0x7fef00151740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.877+0000 7fef064d7700 1 --2- 172.21.15.1:0/464766112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef00099a40 0x7fef0015d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.877+0000 7fef064d7700 1 --2- 172.21.15.1:0/464766112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef00099e20 0x7fef000af3b0 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:58.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.877+0000 7fef064d7700 1 -- 172.21.15.1:0/464766112 >> 172.21.15.1:0/464766112 conn(0x7fef000b2c50 msgr2=0x7fef000b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:58.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.877+0000 7fef064d7700 1 -- 172.21.15.1:0/464766112 shutdown_connections 2024-03-18T20:25:58.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:58.877+0000 7fef064d7700 1 -- 172.21.15.1:0/464766112 wait complete. 2024-03-18T20:25:58.894 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:25:58.954 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:58.950+0000 7f583fa9c240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:25:59.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.019+0000 7f74a8b72700 1 Processor -- start 2024-03-18T20:25:59.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.019+0000 7f74a8b72700 1 -- start start 2024-03-18T20:25:59.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.020+0000 7f74a8b72700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74a41542e0 0x7f74a40cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.020+0000 7f74a8b72700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f74a40b1a70 0x7f74a40cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.020+0000 7f74a8b72700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a415dbf0 con 0x7f74a41546c0 2024-03-18T20:25:59.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.020+0000 7f74a8b72700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a415de90 con 0x7f74a40b1a70 2024-03-18T20:25:59.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.020+0000 7f74a8b72700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a415e170 con 0x7f74a41542e0 2024-03-18T20:25:59.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.021+0000 7f74a1d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f74a41546c0 0x7f74a40b1360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47260/0 (socket says 172.21.15.1:47260) 2024-03-18T20:25:59.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.021+0000 7f74a1d9b700 1 -- 172.21.15.1:0/2503770310 learned_addr learned my addr 172.21.15.1:0/2503770310 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:59.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.021+0000 7f74a259c700 1 --2- 172.21.15.1:0/2503770310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74a41542e0 0x7f74a40cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.021+0000 7f74a2d9d700 1 --2- 172.21.15.1:0/2503770310 >> v2:172.21.15.165:3300/0 conn(0x7f74a40b1a70 0x7f74a40cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.021+0000 7f74a159a700 1 -- 172.21.15.1:0/2503770310 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624148251 0 0) 0x7f74a415dbf0 con 0x7f74a41546c0 2024-03-18T20:25:59.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.021+0000 7f74a159a700 1 -- 172.21.15.1:0/2503770310 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f748c0035b0 con 0x7f74a41546c0 2024-03-18T20:25:59.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.021+0000 7f74a259c700 1 -- 172.21.15.1:0/2503770310 >> v2:172.21.15.165:3300/0 conn(0x7f74a40b1a70 msgr2=0x7f74a40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:59.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.021+0000 7f74a259c700 1 --2- 172.21.15.1:0/2503770310 >> v2:172.21.15.165:3300/0 conn(0x7f74a40b1a70 0x7f74a40cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.021+0000 7f74a259c700 1 -- 172.21.15.1:0/2503770310 >> v1:172.21.15.1:6789/0 conn(0x7f74a41546c0 legacy=0x7f74a40b1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.021+0000 7f74a259c700 1 -- 172.21.15.1:0/2503770310 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a415e450 con 0x7f74a41542e0 2024-03-18T20:25:59.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.021+0000 7f74a2d9d700 1 --2- 172.21.15.1:0/2503770310 >> v2:172.21.15.165:3300/0 conn(0x7f74a40b1a70 0x7f74a40cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:59.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.021+0000 7f74a259c700 1 --2- 172.21.15.1:0/2503770310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74a41542e0 0x7f74a40cc020 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7494009840 tx=0x7f7494009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=461eea4816d3df56 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:59.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.021+0000 7f74a159a700 1 -- 172.21.15.1:0/2503770310 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494011830 con 0x7f74a41542e0 2024-03-18T20:25:59.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.022+0000 7f74a159a700 1 -- 172.21.15.1:0/2503770310 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7494011990 con 0x7f74a41542e0 2024-03-18T20:25:59.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.022+0000 7f74a159a700 1 -- 172.21.15.1:0/2503770310 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494011d20 con 0x7f74a41542e0 2024-03-18T20:25:59.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.022+0000 7f74a8b72700 1 -- 172.21.15.1:0/2503770310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74a41542e0 msgr2=0x7f74a40cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.022+0000 7f74a8b72700 1 --2- 172.21.15.1:0/2503770310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74a41542e0 0x7f74a40cc020 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7494009840 tx=0x7f7494009b50 comp rx=0 tx=0).stop 2024-03-18T20:25:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.022+0000 7f74a8b72700 1 -- 172.21.15.1:0/2503770310 shutdown_connections 2024-03-18T20:25:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.022+0000 7f74a8b72700 1 --2- 172.21.15.1:0/2503770310 >> v2:172.21.15.165:3300/0 conn(0x7f74a40b1a70 0x7f74a40cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.022+0000 7f74a8b72700 1 --2- 172.21.15.1:0/2503770310 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74a41542e0 0x7f74a40cc020 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.022+0000 7f74a8b72700 1 -- 172.21.15.1:0/2503770310 >> 172.21.15.1:0/2503770310 conn(0x7f74a4066540 msgr2=0x7f74a4066950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.022+0000 7f74a8b72700 1 -- 172.21.15.1:0/2503770310 shutdown_connections 2024-03-18T20:25:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.022+0000 7f74a8b72700 1 -- 172.21.15.1:0/2503770310 wait complete. 2024-03-18T20:25:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.024+0000 7f74a8b72700 1 Processor -- start 2024-03-18T20:25:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.024+0000 7f74a8b72700 1 -- start start 2024-03-18T20:25:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.024+0000 7f74a8b72700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74a40b1a70 0x7f74a4153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.024+0000 7f74a8b72700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74a41542e0 0x7f74a41398e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.024+0000 7f74a8b72700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74a41546c0 0x7f74a41f2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.024+0000 7f74a8b72700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a4136e40 con 0x7f74a40b1a70 2024-03-18T20:25:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.024+0000 7f74a8b72700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a41376e0 con 0x7f74a41542e0 2024-03-18T20:25:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.024+0000 7f74a8b72700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a4150990 con 0x7f74a41546c0 2024-03-18T20:25:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.024+0000 7f74a259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74a40b1a70 0x7f74a4153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.024+0000 7f74a259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74a40b1a70 0x7f74a4153350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33700/0 (socket says 172.21.15.1:33700) 2024-03-18T20:25:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.025+0000 7f74a259c700 1 -- 172.21.15.1:0/3795649300 learned_addr learned my addr 172.21.15.1:0/3795649300 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.025+0000 7f74a1d9b700 1 --2- 172.21.15.1:0/3795649300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74a41542e0 0x7f74a41398e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.025+0000 7f74a2d9d700 1 --2- 172.21.15.1:0/3795649300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74a41546c0 0x7f74a41f2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.025+0000 7f74a259c700 1 -- 172.21.15.1:0/3795649300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74a41546c0 msgr2=0x7f74a41f2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.025+0000 7f74a259c700 1 --2- 172.21.15.1:0/3795649300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74a41546c0 0x7f74a41f2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.025+0000 7f74a259c700 1 -- 172.21.15.1:0/3795649300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74a41542e0 msgr2=0x7f74a41398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.025+0000 7f74a259c700 1 --2- 172.21.15.1:0/3795649300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74a41542e0 0x7f74a41398e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.025+0000 7f74a259c700 1 -- 172.21.15.1:0/3795649300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c009a70 con 0x7f74a40b1a70 2024-03-18T20:25:59.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.025+0000 7f74a1d9b700 1 --2- 172.21.15.1:0/3795649300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74a41542e0 0x7f74a41398e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:25:59.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.025+0000 7f74a259c700 1 --2- 172.21.15.1:0/3795649300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74a40b1a70 0x7f74a4153350 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f7494000c00 tx=0x7f7494004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:59.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.025+0000 7f7492ffd700 1 -- 172.21.15.1:0/3795649300 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494004550 con 0x7f74a40b1a70 2024-03-18T20:25:59.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.025+0000 7f7492ffd700 1 -- 172.21.15.1:0/3795649300 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749401f070 con 0x7f74a40b1a70 2024-03-18T20:25:59.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.025+0000 7f74a8b72700 1 -- 172.21.15.1:0/3795649300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74940094a0 con 0x7f74a40b1a70 2024-03-18T20:25:59.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.025+0000 7f7492ffd700 1 -- 172.21.15.1:0/3795649300 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74940038e0 con 0x7f74a40b1a70 2024-03-18T20:25:59.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.025+0000 7f74a8b72700 1 -- 172.21.15.1:0/3795649300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a4153950 con 0x7f74a40b1a70 2024-03-18T20:25:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.026+0000 7f74a8b72700 1 -- 172.21.15.1:0/3795649300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a40676a0 con 0x7f74a40b1a70 2024-03-18T20:25:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.027+0000 7f7492ffd700 1 -- 172.21.15.1:0/3795649300 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 142) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f7494003a40 con 0x7f74a40b1a70 2024-03-18T20:25:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.028+0000 7f7492ffd700 1 --2- 172.21.15.1:0/3795649300 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f748406af60 0x7f748406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.028+0000 7f74a1d9b700 1 -- 172.21.15.1:0/3795649300 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f748406af60 msgr2=0x7f748406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.028+0000 7f7492ffd700 1 -- 172.21.15.1:0/3795649300 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7f7494021070 con 0x7f74a40b1a70 2024-03-18T20:25:59.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.028+0000 7f74a1d9b700 1 --2- 172.21.15.1:0/3795649300 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f748406af60 0x7f748406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:59.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.032+0000 7f7492ffd700 1 -- 172.21.15.1:0/3795649300 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f74940561b0 con 0x7f74a40b1a70 2024-03-18T20:25:59.042 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:59.039+0000 7f47e8e57240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:25:59.167 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:59.163+0000 7f583fa9c240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:25:59.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.229+0000 7f74a1d9b700 1 -- 172.21.15.1:0/3795649300 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f748406af60 msgr2=0x7f748406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:59.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.229+0000 7f74a1d9b700 1 --2- 172.21.15.1:0/3795649300 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f748406af60 0x7f748406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:59.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.237+0000 7f74a8b72700 1 -- 172.21.15.1:0/3795649300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f74a4070f30 con 0x7f74a40b1a70 2024-03-18T20:25:59.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.238+0000 7f7492ffd700 1 -- 172.21.15.1:0/3795649300 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v237) v1 ==== 104+0+11 (secure 0 0 0) 0x7f74940597d0 con 0x7f74a40b1a70 2024-03-18T20:25:59.242 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:25:59.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.240+0000 7f74a8b72700 1 -- 172.21.15.1:0/3795649300 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f748406af60 msgr2=0x7f748406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:59.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.240+0000 7f74a8b72700 1 --2- 172.21.15.1:0/3795649300 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f748406af60 0x7f748406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.240+0000 7f74a8b72700 1 -- 172.21.15.1:0/3795649300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74a40b1a70 msgr2=0x7f74a4153350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.240+0000 7f74a8b72700 1 --2- 172.21.15.1:0/3795649300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74a40b1a70 0x7f74a4153350 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f7494000c00 tx=0x7f7494004750 comp rx=0 tx=0).stop 2024-03-18T20:25:59.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.240+0000 7f74a8b72700 1 -- 172.21.15.1:0/3795649300 shutdown_connections 2024-03-18T20:25:59.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.240+0000 7f74a8b72700 1 --2- 172.21.15.1:0/3795649300 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f748406af60 0x7f748406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.240+0000 7f74a8b72700 1 --2- 172.21.15.1:0/3795649300 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f74a41546c0 0x7f74a41f2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.240+0000 7f74a8b72700 1 --2- 172.21.15.1:0/3795649300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74a40b1a70 0x7f74a4153350 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.240+0000 7f74a8b72700 1 --2- 172.21.15.1:0/3795649300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74a41542e0 0x7f74a41398e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.240+0000 7f74a8b72700 1 -- 172.21.15.1:0/3795649300 >> 172.21.15.1:0/3795649300 conn(0x7f74a4066540 msgr2=0x7f74a40cc020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:59.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.240+0000 7f74a8b72700 1 -- 172.21.15.1:0/3795649300 shutdown_connections 2024-03-18T20:25:59.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.240+0000 7f74a8b72700 1 -- 172.21.15.1:0/3795649300 wait complete. 2024-03-18T20:25:59.257 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:25:59.265 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:59.261+0000 7f47e8e57240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:25:59.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.382+0000 7f4f05be4700 1 Processor -- start 2024-03-18T20:25:59.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.382+0000 7f4f05be4700 1 -- start start 2024-03-18T20:25:59.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.383+0000 7f4f05be4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f000b2650 0x7f4f0015da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.383+0000 7f4f05be4700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f000b3560 con 0x7f4f000b3c30 2024-03-18T20:25:59.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.383+0000 7f4f05be4700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f00071a80 con 0x7f4f000cef70 2024-03-18T20:25:59.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.383+0000 7f4f05be4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f000b2ae0 con 0x7f4f000b2650 2024-03-18T20:25:59.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.383+0000 7f4eff7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4f000b3c30 0x7f4f000b4010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47274/0 (socket says 172.21.15.1:47274) 2024-03-18T20:25:59.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.383+0000 7f4eff7fe700 1 -- 172.21.15.1:0/1630625013 learned_addr learned my addr 172.21.15.1:0/1630625013 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:25:59.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.383+0000 7f4efffff700 1 --2- 172.21.15.1:0/1630625013 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f000b2650 0x7f4f0015da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.383+0000 7f4efe7fc700 1 -- 172.21.15.1:0/1630625013 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912683891 0 0) 0x7f4f000b3560 con 0x7f4f000b3c30 2024-03-18T20:25:59.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.383+0000 7f4efe7fc700 1 -- 172.21.15.1:0/1630625013 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eec0035b0 con 0x7f4f000b3c30 2024-03-18T20:25:59.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.383+0000 7f4efe7fc700 1 -- 172.21.15.1:0/1630625013 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2683105975 0 0) 0x7f4f00071a80 con 0x7f4f000cef70 2024-03-18T20:25:59.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.383+0000 7f4efe7fc700 1 -- 172.21.15.1:0/1630625013 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f000b3560 con 0x7f4f000cef70 2024-03-18T20:25:59.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.383+0000 7f4efffff700 1 -- 172.21.15.1:0/1630625013 >> v1:172.21.15.165:6789/0 conn(0x7f4f000cef70 legacy=0x7f4f0006ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.383+0000 7f4efffff700 1 -- 172.21.15.1:0/1630625013 >> v1:172.21.15.1:6789/0 conn(0x7f4f000b3c30 legacy=0x7f4f000b4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.384+0000 7f4efffff700 1 -- 172.21.15.1:0/1630625013 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f000b2cc0 con 0x7f4f000b2650 2024-03-18T20:25:59.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.384+0000 7f4efffff700 1 --2- 172.21.15.1:0/1630625013 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f000b2650 0x7f4f0015da10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4ef000aed0 tx=0x7f4ef000c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97eaeb61205d560f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:59.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.384+0000 7f4efe7fc700 1 -- 172.21.15.1:0/1630625013 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef000f070 con 0x7f4f000b2650 2024-03-18T20:25:59.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.384+0000 7f4efe7fc700 1 -- 172.21.15.1:0/1630625013 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef000a2e0 con 0x7f4f000b2650 2024-03-18T20:25:59.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.384+0000 7f4efe7fc700 1 -- 172.21.15.1:0/1630625013 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef000cc10 con 0x7f4f000b2650 2024-03-18T20:25:59.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.385+0000 7f4f05be4700 1 -- 172.21.15.1:0/1630625013 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f000b2650 msgr2=0x7f4f0015da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.385+0000 7f4f05be4700 1 --2- 172.21.15.1:0/1630625013 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f000b2650 0x7f4f0015da10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4ef000aed0 tx=0x7f4ef000c330 comp rx=0 tx=0).stop 2024-03-18T20:25:59.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.385+0000 7f4f05be4700 1 -- 172.21.15.1:0/1630625013 shutdown_connections 2024-03-18T20:25:59.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.385+0000 7f4f05be4700 1 --2- 172.21.15.1:0/1630625013 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f000b2650 0x7f4f0015da10 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.385+0000 7f4f05be4700 1 -- 172.21.15.1:0/1630625013 >> 172.21.15.1:0/1630625013 conn(0x7f4f000b1350 msgr2=0x7f4f000b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:59.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.385+0000 7f4f05be4700 1 -- 172.21.15.1:0/1630625013 shutdown_connections 2024-03-18T20:25:59.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.385+0000 7f4f05be4700 1 -- 172.21.15.1:0/1630625013 wait complete. 2024-03-18T20:25:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.386+0000 7f4f05be4700 1 Processor -- start 2024-03-18T20:25:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.386+0000 7f4f05be4700 1 -- start start 2024-03-18T20:25:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.387+0000 7f4f05be4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f000b2650 0x7f4f001517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.387+0000 7f4f05be4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f000b3c30 0x7f4f0014ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.387+0000 7f4f05be4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f000cef70 0x7f4f00154870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.387+0000 7f4f05be4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0014f3c0 con 0x7f4f000b2650 2024-03-18T20:25:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.387+0000 7f4f05be4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0014f7f0 con 0x7f4f000cef70 2024-03-18T20:25:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.387+0000 7f4f05be4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f00150750 con 0x7f4f000b3c30 2024-03-18T20:25:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.387+0000 7f4eff7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f000b2650 0x7f4f001517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.387+0000 7f4eff7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f000b2650 0x7f4f001517f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33710/0 (socket says 172.21.15.1:33710) 2024-03-18T20:25:59.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.387+0000 7f4eff7fe700 1 -- 172.21.15.1:0/2180741085 learned_addr learned my addr 172.21.15.1:0/2180741085 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:59.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.387+0000 7f4efffff700 1 --2- 172.21.15.1:0/2180741085 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f000cef70 0x7f4f00154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.387+0000 7f4efeffd700 1 --2- 172.21.15.1:0/2180741085 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f000b3c30 0x7f4f0014ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.387+0000 7f4eff7fe700 1 -- 172.21.15.1:0/2180741085 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f000b3c30 msgr2=0x7f4f0014ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.387+0000 7f4eff7fe700 1 --2- 172.21.15.1:0/2180741085 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f000b3c30 0x7f4f0014ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.387+0000 7f4eff7fe700 1 -- 172.21.15.1:0/2180741085 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f000cef70 msgr2=0x7f4f00154870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.387+0000 7f4eff7fe700 1 --2- 172.21.15.1:0/2180741085 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f000cef70 0x7f4f00154870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.387+0000 7f4eff7fe700 1 -- 172.21.15.1:0/2180741085 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef000ab80 con 0x7f4f000b2650 2024-03-18T20:25:59.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.387+0000 7f4eff7fe700 1 --2- 172.21.15.1:0/2180741085 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f000b2650 0x7f4f001517f0 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f4ef800c700 tx=0x7f4ef800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:59.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.388+0000 7f4ef7fff700 1 -- 172.21.15.1:0/2180741085 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8005d70 con 0x7f4f000b2650 2024-03-18T20:25:59.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.388+0000 7f4ef7fff700 1 -- 172.21.15.1:0/2180741085 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef80070d0 con 0x7f4f000b2650 2024-03-18T20:25:59.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.388+0000 7f4f05be4700 1 -- 172.21.15.1:0/2180741085 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f00150a30 con 0x7f4f000b2650 2024-03-18T20:25:59.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.388+0000 7f4ef7fff700 1 -- 172.21.15.1:0/2180741085 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8007460 con 0x7f4f000b2650 2024-03-18T20:25:59.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.388+0000 7f4f05be4700 1 -- 172.21.15.1:0/2180741085 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f00150ff0 con 0x7f4f000b2650 2024-03-18T20:25:59.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.389+0000 7f4f05be4700 1 -- 172.21.15.1:0/2180741085 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f000676a0 con 0x7f4f000b2650 2024-03-18T20:25:59.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.389+0000 7f4ef7fff700 1 -- 172.21.15.1:0/2180741085 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 142) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f4ef80075c0 con 0x7f4f000b2650 2024-03-18T20:25:59.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.389+0000 7f4ef7fff700 1 --2- 172.21.15.1:0/2180741085 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4ee006af60 0x7f4ee006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.390+0000 7f4efeffd700 1 -- 172.21.15.1:0/2180741085 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4ee006af60 msgr2=0x7f4ee006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:59.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.390+0000 7f4ef7fff700 1 -- 172.21.15.1:0/2180741085 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7f4ef80892d0 con 0x7f4f000b2650 2024-03-18T20:25:59.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.390+0000 7f4efeffd700 1 --2- 172.21.15.1:0/2180741085 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4ee006af60 0x7f4ee006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:59.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.394+0000 7f4ef7fff700 1 -- 172.21.15.1:0/2180741085 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4ef8053a80 con 0x7f4f000b2650 2024-03-18T20:25:59.540 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:25:59.537+0000 7f583fa9c240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:25:59.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.591+0000 7f4efeffd700 1 -- 172.21.15.1:0/2180741085 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4ee006af60 msgr2=0x7f4ee006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:59.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.591+0000 7f4efeffd700 1 --2- 172.21.15.1:0/2180741085 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4ee006af60 0x7f4ee006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:25:59.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.613+0000 7f4f05be4700 1 -- 172.21.15.1:0/2180741085 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4f0006f7c0 con 0x7f4f000b2650 2024-03-18T20:25:59.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.613+0000 7f4ef7fff700 1 -- 172.21.15.1:0/2180741085 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v237) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4ef80570a0 con 0x7f4f000b2650 2024-03-18T20:25:59.617 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:25:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.616+0000 7f4f05be4700 1 -- 172.21.15.1:0/2180741085 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4ee006af60 msgr2=0x7f4ee006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:25:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.616+0000 7f4f05be4700 1 --2- 172.21.15.1:0/2180741085 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4ee006af60 0x7f4ee006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.616+0000 7f4f05be4700 1 -- 172.21.15.1:0/2180741085 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f000b2650 msgr2=0x7f4f001517f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.616+0000 7f4f05be4700 1 --2- 172.21.15.1:0/2180741085 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f000b2650 0x7f4f001517f0 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f4ef800c700 tx=0x7f4ef800ca10 comp rx=0 tx=0).stop 2024-03-18T20:25:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.616+0000 7f4f05be4700 1 -- 172.21.15.1:0/2180741085 shutdown_connections 2024-03-18T20:25:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.616+0000 7f4f05be4700 1 --2- 172.21.15.1:0/2180741085 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4ee006af60 0x7f4ee006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.616+0000 7f4f05be4700 1 --2- 172.21.15.1:0/2180741085 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f000cef70 0x7f4f00154870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.616+0000 7f4f05be4700 1 --2- 172.21.15.1:0/2180741085 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4f000b2650 0x7f4f001517f0 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.616+0000 7f4f05be4700 1 --2- 172.21.15.1:0/2180741085 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4f000b3c30 0x7f4f0014ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.616+0000 7f4f05be4700 1 -- 172.21.15.1:0/2180741085 >> 172.21.15.1:0/2180741085 conn(0x7f4f000b1350 msgr2=0x7f4f0006e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:59.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.616+0000 7f4f05be4700 1 -- 172.21.15.1:0/2180741085 shutdown_connections 2024-03-18T20:25:59.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.616+0000 7f4f05be4700 1 -- 172.21.15.1:0/2180741085 wait complete. 2024-03-18T20:25:59.633 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-03-18T20:25:59.673 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:25:59.669+0000 7f47e8e57240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:25:59.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.799+0000 7fa8f2168700 1 Processor -- start 2024-03-18T20:25:59.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.799+0000 7fa8f2168700 1 -- start start 2024-03-18T20:25:59.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.799+0000 7fa8f2168700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8ec15b6c0 0x7fa8ec0b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.800+0000 7fa8f2168700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa8ec0cd430 0x7fa8ec0c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.800+0000 7fa8f2168700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa8ec0c8600 0x7fa8ec0c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.800+0000 7fa8f2168700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8ec0c8fd0 con 0x7fa8ec0c8600 2024-03-18T20:25:59.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.800+0000 7fa8f2168700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8ec0d01f0 con 0x7fa8ec0cd430 2024-03-18T20:25:59.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.800+0000 7fa8f2168700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec0ccfa0 con 0x7fa8ec15b6c0 2024-03-18T20:25:59.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.800+0000 7fa8ebfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa8ec0c8600 0x7fa8ec0c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.800+0000 7fa8ebfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa8ec0c8600 0x7fa8ec0c8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33718/0 (socket says 172.21.15.1:33718) 2024-03-18T20:25:59.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.800+0000 7fa8ebfff700 1 -- 172.21.15.1:0/4120302045 learned_addr learned my addr 172.21.15.1:0/4120302045 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:59.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.800+0000 7fa8eaffd700 1 --2- 172.21.15.1:0/4120302045 >> v2:172.21.15.165:3300/0 conn(0x7fa8ec0cd430 0x7fa8ec0c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.800+0000 7fa8eb7fe700 1 --2- 172.21.15.1:0/4120302045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8ec15b6c0 0x7fa8ec0b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.800+0000 7fa8ebfff700 1 -- 172.21.15.1:0/4120302045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8ec15b6c0 msgr2=0x7fa8ec0b2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:59.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.800+0000 7fa8ebfff700 1 --2- 172.21.15.1:0/4120302045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8ec15b6c0 0x7fa8ec0b2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.800+0000 7fa8ebfff700 1 -- 172.21.15.1:0/4120302045 >> v2:172.21.15.165:3300/0 conn(0x7fa8ec0cd430 msgr2=0x7fa8ec0c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:25:59.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.800+0000 7fa8ebfff700 1 --2- 172.21.15.1:0/4120302045 >> v2:172.21.15.165:3300/0 conn(0x7fa8ec0cd430 0x7fa8ec0c80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.800+0000 7fa8ebfff700 1 -- 172.21.15.1:0/4120302045 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec0d1db0 con 0x7fa8ec0c8600 2024-03-18T20:25:59.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.800+0000 7fa8ebfff700 1 --2- 172.21.15.1:0/4120302045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8ec0c8600 0x7fa8ec0c8a90 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec072030 tx=0x7fa8e4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edefacc2ecb37f91 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:59.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.800+0000 7fa8ea7fc700 1 -- 172.21.15.1:0/4120302045 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4011830 con 0x7fa8ec0c8600 2024-03-18T20:25:59.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.800+0000 7fa8ea7fc700 1 -- 172.21.15.1:0/4120302045 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e4011990 con 0x7fa8ec0c8600 2024-03-18T20:25:59.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.801+0000 7fa8ea7fc700 1 -- 172.21.15.1:0/4120302045 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4011d20 con 0x7fa8ec0c8600 2024-03-18T20:25:59.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.801+0000 7fa8f2168700 1 -- 172.21.15.1:0/4120302045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8ec0c8600 msgr2=0x7fa8ec0c8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.801+0000 7fa8f2168700 1 --2- 172.21.15.1:0/4120302045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8ec0c8600 0x7fa8ec0c8a90 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec072030 tx=0x7fa8e4009bd0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.801+0000 7fa8f2168700 1 -- 172.21.15.1:0/4120302045 shutdown_connections 2024-03-18T20:25:59.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.801+0000 7fa8f2168700 1 --2- 172.21.15.1:0/4120302045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8ec0c8600 0x7fa8ec0c8a90 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.801+0000 7fa8f2168700 1 --2- 172.21.15.1:0/4120302045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8ec15b6c0 0x7fa8ec0b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.801+0000 7fa8f2168700 1 --2- 172.21.15.1:0/4120302045 >> v2:172.21.15.165:3300/0 conn(0x7fa8ec0cd430 0x7fa8ec0c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.801+0000 7fa8f2168700 1 -- 172.21.15.1:0/4120302045 >> 172.21.15.1:0/4120302045 conn(0x7fa8ec0d0ff0 msgr2=0x7fa8ec0d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:25:59.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.801+0000 7fa8f2168700 1 -- 172.21.15.1:0/4120302045 shutdown_connections 2024-03-18T20:25:59.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.801+0000 7fa8f2168700 1 -- 172.21.15.1:0/4120302045 wait complete. 2024-03-18T20:25:59.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.803+0000 7fa8f2168700 1 Processor -- start 2024-03-18T20:25:59.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.804+0000 7fa8f2168700 1 -- start start 2024-03-18T20:25:59.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.805+0000 7fa8f2168700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8ec0c8600 0x7fa8ec0c6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.805+0000 7fa8f2168700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8ec0cd430 0x7fa8ec157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.805+0000 7fa8f2168700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8ec15b6c0 0x7fa8ec09e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.805+0000 7fa8f2168700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec1f2cd0 con 0x7fa8ec15b6c0 2024-03-18T20:25:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.805+0000 7fa8f2168700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec157a80 con 0x7fa8ec0cd430 2024-03-18T20:25:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.805+0000 7fa8f2168700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec09ed90 con 0x7fa8ec0c8600 2024-03-18T20:25:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.805+0000 7fa8ebfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8ec15b6c0 0x7fa8ec09e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.805+0000 7fa8ebfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8ec15b6c0 0x7fa8ec09e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33724/0 (socket says 172.21.15.1:33724) 2024-03-18T20:25:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.805+0000 7fa8ebfff700 1 -- 172.21.15.1:0/2594668608 learned_addr learned my addr 172.21.15.1:0/2594668608 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:25:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.805+0000 7fa8eaffd700 1 --2- 172.21.15.1:0/2594668608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8ec0cd430 0x7fa8ec157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.805+0000 7fa8eb7fe700 1 --2- 172.21.15.1:0/2594668608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8ec0c8600 0x7fa8ec0c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:25:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.805+0000 7fa8ebfff700 1 -- 172.21.15.1:0/2594668608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8ec0c8600 msgr2=0x7fa8ec0c6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.805+0000 7fa8ebfff700 1 --2- 172.21.15.1:0/2594668608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8ec0c8600 0x7fa8ec0c6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.805+0000 7fa8ebfff700 1 -- 172.21.15.1:0/2594668608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8ec0cd430 msgr2=0x7fa8ec157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:25:59.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.805+0000 7fa8ebfff700 1 --2- 172.21.15.1:0/2594668608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8ec0cd430 0x7fa8ec157540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:25:59.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.805+0000 7fa8ebfff700 1 -- 172.21.15.1:0/2594668608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e4009550 con 0x7fa8ec15b6c0 2024-03-18T20:25:59.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.806+0000 7fa8ebfff700 1 --2- 172.21.15.1:0/2594668608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8ec15b6c0 0x7fa8ec09e540 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4009c10 tx=0x7fa8e4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:25:59.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.806+0000 7fa8d3fff700 1 -- 172.21.15.1:0/2594668608 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4004240 con 0x7fa8ec15b6c0 2024-03-18T20:25:59.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.806+0000 7fa8d3fff700 1 -- 172.21.15.1:0/2594668608 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e401f070 con 0x7fa8ec15b6c0 2024-03-18T20:25:59.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.806+0000 7fa8f2168700 1 -- 172.21.15.1:0/2594668608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8ec1f2330 con 0x7fa8ec15b6c0 2024-03-18T20:25:59.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.806+0000 7fa8d3fff700 1 -- 172.21.15.1:0/2594668608 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e40045c0 con 0x7fa8ec15b6c0 2024-03-18T20:25:59.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.806+0000 7fa8f2168700 1 -- 172.21.15.1:0/2594668608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8ec1f28f0 con 0x7fa8ec15b6c0 2024-03-18T20:25:59.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.807+0000 7fa8f2168700 1 -- 172.21.15.1:0/2594668608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8ec1f2e10 con 0x7fa8ec15b6c0 2024-03-18T20:25:59.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.807+0000 7fa8d3fff700 1 -- 172.21.15.1:0/2594668608 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 142) v1 ==== 89398+0+0 (secure 0 0 0) 0x7fa8e4021030 con 0x7fa8ec15b6c0 2024-03-18T20:25:59.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.808+0000 7fa8d3fff700 1 --2- 172.21.15.1:0/2594668608 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8d806af60 0x7fa8d806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:25:59.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.808+0000 7fa8eb7fe700 1 -- 172.21.15.1:0/2594668608 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8d806af60 msgr2=0x7fa8d806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:25:59.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.808+0000 7fa8d3fff700 1 -- 172.21.15.1:0/2594668608 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7fa8e408aeb0 con 0x7fa8ec15b6c0 2024-03-18T20:25:59.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.808+0000 7fa8eb7fe700 1 --2- 172.21.15.1:0/2594668608 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8d806af60 0x7fa8d806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:25:59.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:25:59.812+0000 7fa8d3fff700 1 -- 172.21.15.1:0/2594668608 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa8ec1f2e10 con 0x7fa8ec15b6c0 2024-03-18T20:26:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.009+0000 7fa8eb7fe700 1 -- 172.21.15.1:0/2594668608 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8d806af60 msgr2=0x7fa8d806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:00.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.009+0000 7fa8eb7fe700 1 --2- 172.21.15.1:0/2594668608 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8d806af60 0x7fa8d806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:00.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.017+0000 7fa8f2168700 1 -- 172.21.15.1:0/2594668608 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa8ec1f2e10 con 0x7fa8ec15b6c0 2024-03-18T20:26:00.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.017+0000 7fa8d3fff700 1 -- 172.21.15.1:0/2594668608 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v237) v1 ==== 107+0+11 (secure 0 0 0) 0x7fa8ec1f2e10 con 0x7fa8ec15b6c0 2024-03-18T20:26:00.022 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:26:00.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.020+0000 7fa8f2168700 1 -- 172.21.15.1:0/2594668608 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8d806af60 msgr2=0x7fa8d806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:00.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.020+0000 7fa8f2168700 1 --2- 172.21.15.1:0/2594668608 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8d806af60 0x7fa8d806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.020+0000 7fa8f2168700 1 -- 172.21.15.1:0/2594668608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8ec15b6c0 msgr2=0x7fa8ec09e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.020+0000 7fa8f2168700 1 --2- 172.21.15.1:0/2594668608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8ec15b6c0 0x7fa8ec09e540 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4009c10 tx=0x7fa8e4004750 comp rx=0 tx=0).stop 2024-03-18T20:26:00.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.020+0000 7fa8f2168700 1 -- 172.21.15.1:0/2594668608 shutdown_connections 2024-03-18T20:26:00.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.020+0000 7fa8f2168700 1 --2- 172.21.15.1:0/2594668608 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8d806af60 0x7fa8d806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.020+0000 7fa8f2168700 1 --2- 172.21.15.1:0/2594668608 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8ec0c8600 0x7fa8ec0c6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.020+0000 7fa8f2168700 1 --2- 172.21.15.1:0/2594668608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8ec15b6c0 0x7fa8ec09e540 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.020+0000 7fa8f2168700 1 --2- 172.21.15.1:0/2594668608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8ec0cd430 0x7fa8ec157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.020+0000 7fa8f2168700 1 -- 172.21.15.1:0/2594668608 >> 172.21.15.1:0/2594668608 conn(0x7fa8ec0d0ff0 msgr2=0x7fa8ec06fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.020+0000 7fa8f2168700 1 -- 172.21.15.1:0/2594668608 shutdown_connections 2024-03-18T20:26:00.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.020+0000 7fa8f2168700 1 -- 172.21.15.1:0/2594668608 wait complete. 2024-03-18T20:26:00.037 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-03-18T20:26:00.130 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:00.127+0000 7f583fa9c240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:26:00.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.169+0000 7fc197051700 1 Processor -- start 2024-03-18T20:26:00.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.169+0000 7fc197051700 1 -- start start 2024-03-18T20:26:00.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.170+0000 7fc197051700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc190160a70 0x7fc1900d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.170+0000 7fc197051700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc190160e50 0x7fc1900cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.170+0000 7fc197051700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc190162170 con 0x7fc1901612d0 2024-03-18T20:26:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.170+0000 7fc197051700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1900cef30 con 0x7fc190160e50 2024-03-18T20:26:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.170+0000 7fc197051700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1900cf210 con 0x7fc190160a70 2024-03-18T20:26:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.170+0000 7fc194ded700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc190160a70 0x7fc1900d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.170+0000 7fc18ffff700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc190160e50 0x7fc1900cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.170+0000 7fc194ded700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc190160a70 0x7fc1900d3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:53008/0 (socket says 172.21.15.1:53008) 2024-03-18T20:26:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.170+0000 7fc1955ee700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fc1901612d0 0x7fc1900c6da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:59184/0 (socket says 172.21.15.1:59184) 2024-03-18T20:26:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.170+0000 7fc194ded700 1 -- 172.21.15.1:0/1114979688 learned_addr learned my addr 172.21.15.1:0/1114979688 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.170+0000 7fc18ffff700 1 -- 172.21.15.1:0/1114979688 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc190160a70 msgr2=0x7fc1900d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.170+0000 7fc18ffff700 1 --2- 172.21.15.1:0/1114979688 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc190160a70 0x7fc1900d3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.170+0000 7fc18ffff700 1 -- 172.21.15.1:0/1114979688 >> v1:172.21.15.165:6789/0 conn(0x7fc1901612d0 legacy=0x7fc1900c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.170+0000 7fc18ffff700 1 -- 172.21.15.1:0/1114979688 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1900cf4f0 con 0x7fc190160e50 2024-03-18T20:26:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.170+0000 7fc18f7fe700 1 -- 172.21.15.1:0/1114979688 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540371994 0 0) 0x7fc188001490 con 0x7fc1901612d0 2024-03-18T20:26:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.170+0000 7fc194ded700 1 --2- 172.21.15.1:0/1114979688 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc190160a70 0x7fc1900d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:26:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.171+0000 7fc18ffff700 1 --2- 172.21.15.1:0/1114979688 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc190160e50 0x7fc1900cffd0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fc17c009840 tx=0x7fc17c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c378ff76ebb6d4bf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.171+0000 7fc18f7fe700 1 -- 172.21.15.1:0/1114979688 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c011830 con 0x7fc190160e50 2024-03-18T20:26:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.171+0000 7fc18f7fe700 1 -- 172.21.15.1:0/1114979688 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c011990 con 0x7fc190160e50 2024-03-18T20:26:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.171+0000 7fc18f7fe700 1 -- 172.21.15.1:0/1114979688 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c011d20 con 0x7fc190160e50 2024-03-18T20:26:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.171+0000 7fc197051700 1 -- 172.21.15.1:0/1114979688 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc190160e50 msgr2=0x7fc1900cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.171+0000 7fc197051700 1 --2- 172.21.15.1:0/1114979688 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc190160e50 0x7fc1900cffd0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fc17c009840 tx=0x7fc17c009b50 comp rx=0 tx=0).stop 2024-03-18T20:26:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.171+0000 7fc197051700 1 -- 172.21.15.1:0/1114979688 shutdown_connections 2024-03-18T20:26:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.172+0000 7fc197051700 1 --2- 172.21.15.1:0/1114979688 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc190160a70 0x7fc1900d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.172+0000 7fc197051700 1 --2- 172.21.15.1:0/1114979688 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc190160e50 0x7fc1900cffd0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.172+0000 7fc197051700 1 -- 172.21.15.1:0/1114979688 >> 172.21.15.1:0/1114979688 conn(0x7fc19009c560 msgr2=0x7fc1900d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.172+0000 7fc197051700 1 -- 172.21.15.1:0/1114979688 shutdown_connections 2024-03-18T20:26:00.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.172+0000 7fc197051700 1 -- 172.21.15.1:0/1114979688 wait complete. 2024-03-18T20:26:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.173+0000 7fc197051700 1 Processor -- start 2024-03-18T20:26:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.173+0000 7fc197051700 1 -- start start 2024-03-18T20:26:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.174+0000 7fc197051700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc190160a70 0x7fc1900cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.174+0000 7fc197051700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc190160e50 0x7fc190149220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.174+0000 7fc197051700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1901612d0 0x7fc1900cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.174+0000 7fc197051700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc190148d90 con 0x7fc190160e50 2024-03-18T20:26:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.174+0000 7fc197051700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1900cd400 con 0x7fc1901612d0 2024-03-18T20:26:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.174+0000 7fc197051700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1900cce40 con 0x7fc190160a70 2024-03-18T20:26:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.174+0000 7fc18ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc190160e50 0x7fc190149220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.174+0000 7fc18ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc190160e50 0x7fc190149220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33728/0 (socket says 172.21.15.1:33728) 2024-03-18T20:26:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.174+0000 7fc18ffff700 1 -- 172.21.15.1:0/124721105 learned_addr learned my addr 172.21.15.1:0/124721105 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:00.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.174+0000 7fc1955ee700 1 --2- 172.21.15.1:0/124721105 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1901612d0 0x7fc1900cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.174+0000 7fc194ded700 1 --2- 172.21.15.1:0/124721105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc190160a70 0x7fc1900cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.174+0000 7fc18ffff700 1 -- 172.21.15.1:0/124721105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc190160a70 msgr2=0x7fc1900cad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.174+0000 7fc18ffff700 1 --2- 172.21.15.1:0/124721105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc190160a70 0x7fc1900cad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.174+0000 7fc18ffff700 1 -- 172.21.15.1:0/124721105 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1901612d0 msgr2=0x7fc1900cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.174+0000 7fc18ffff700 1 --2- 172.21.15.1:0/124721105 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1901612d0 0x7fc1900cc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.174+0000 7fc18ffff700 1 -- 172.21.15.1:0/124721105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc184009550 con 0x7fc190160e50 2024-03-18T20:26:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.174+0000 7fc18ffff700 1 --2- 172.21.15.1:0/124721105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc190160e50 0x7fc190149220 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7fc17c0090b0 tx=0x7fc17c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.175+0000 7fc18d7fa700 1 -- 172.21.15.1:0/124721105 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c0036a0 con 0x7fc190160e50 2024-03-18T20:26:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.175+0000 7fc18d7fa700 1 -- 172.21.15.1:0/124721105 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c01f070 con 0x7fc190160e50 2024-03-18T20:26:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.175+0000 7fc197051700 1 -- 172.21.15.1:0/124721105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17c0094a0 con 0x7fc190160e50 2024-03-18T20:26:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.175+0000 7fc18d7fa700 1 -- 172.21.15.1:0/124721105 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c003a60 con 0x7fc190160e50 2024-03-18T20:26:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.175+0000 7fc197051700 1 -- 172.21.15.1:0/124721105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc190148a20 con 0x7fc190160e50 2024-03-18T20:26:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.176+0000 7fc197051700 1 -- 172.21.15.1:0/124721105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc190148be0 con 0x7fc190160e50 2024-03-18T20:26:00.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.176+0000 7fc18d7fa700 1 -- 172.21.15.1:0/124721105 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 142) v1 ==== 89398+0+0 (secure 0 0 0) 0x7fc17c003bc0 con 0x7fc190160e50 2024-03-18T20:26:00.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.176+0000 7fc18d7fa700 1 --2- 172.21.15.1:0/124721105 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc17406af60 0x7fc17406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.177+0000 7fc194ded700 1 -- 172.21.15.1:0/124721105 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc17406af60 msgr2=0x7fc17406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:00.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.177+0000 7fc18d7fa700 1 -- 172.21.15.1:0/124721105 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7fc17c021070 con 0x7fc190160e50 2024-03-18T20:26:00.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.177+0000 7fc194ded700 1 --2- 172.21.15.1:0/124721105 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc17406af60 0x7fc17406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:00.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.185+0000 7fc18d7fa700 1 -- 172.21.15.1:0/124721105 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc17c055e70 con 0x7fc190160e50 2024-03-18T20:26:00.202 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:00.199+0000 7f583fa9c240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:26:00.314 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:00.311+0000 7f47e8e57240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:26:00.329 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:00.326+0000 7f583fa9c240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:26:00.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.378+0000 7fc194ded700 1 -- 172.21.15.1:0/124721105 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc17406af60 msgr2=0x7fc17406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:00.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.378+0000 7fc194ded700 1 --2- 172.21.15.1:0/124721105 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc17406af60 0x7fc17406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:00.494 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:00.387+0000 7f47e8e57240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:26:00.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.391+0000 7fc197051700 1 -- 172.21.15.1:0/124721105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7fc190148f40 con 0x7fc190160e50 2024-03-18T20:26:00.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.391+0000 7fc18d7fa700 1 -- 172.21.15.1:0/124721105 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v237) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc190148f40 con 0x7fc190160e50 2024-03-18T20:26:00.495 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:26:00.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.394+0000 7fc197051700 1 -- 172.21.15.1:0/124721105 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc17406af60 msgr2=0x7fc17406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:00.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.394+0000 7fc197051700 1 --2- 172.21.15.1:0/124721105 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc17406af60 0x7fc17406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.394+0000 7fc197051700 1 -- 172.21.15.1:0/124721105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc190160e50 msgr2=0x7fc190149220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.394+0000 7fc197051700 1 --2- 172.21.15.1:0/124721105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc190160e50 0x7fc190149220 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7fc17c0090b0 tx=0x7fc17c004750 comp rx=0 tx=0).stop 2024-03-18T20:26:00.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.394+0000 7fc197051700 1 -- 172.21.15.1:0/124721105 shutdown_connections 2024-03-18T20:26:00.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.394+0000 7fc197051700 1 --2- 172.21.15.1:0/124721105 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc17406af60 0x7fc17406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.394+0000 7fc197051700 1 --2- 172.21.15.1:0/124721105 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc190160a70 0x7fc1900cad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.394+0000 7fc197051700 1 --2- 172.21.15.1:0/124721105 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1901612d0 0x7fc1900cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.394+0000 7fc197051700 1 --2- 172.21.15.1:0/124721105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc190160e50 0x7fc190149220 unknown :-1 s=CLOSED pgs=2664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.394+0000 7fc197051700 1 -- 172.21.15.1:0/124721105 >> 172.21.15.1:0/124721105 conn(0x7fc19009c560 msgr2=0x7fc1900d36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.394+0000 7fc197051700 1 -- 172.21.15.1:0/124721105 shutdown_connections 2024-03-18T20:26:00.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.394+0000 7fc197051700 1 -- 172.21.15.1:0/124721105 wait complete. 2024-03-18T20:26:00.497 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T20:26:00.508 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:00.505+0000 7f583fa9c240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:26:00.528 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:00.524+0000 7f47e8e57240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:26:00.580 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:00.577+0000 7f583fa9c240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:26:00.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.621+0000 7ff8d0ed9700 1 Processor -- start 2024-03-18T20:26:00.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.621+0000 7ff8d0ed9700 1 -- start start 2024-03-18T20:26:00.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.622+0000 7ff8d0ed9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8cc160a70 0x7ff8cc0d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.622+0000 7ff8d0ed9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff8cc160e50 0x7ff8cc0cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.622+0000 7ff8d0ed9700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8cc162170 con 0x7ff8cc1612d0 2024-03-18T20:26:00.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.622+0000 7ff8d0ed9700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8cc0cef30 con 0x7ff8cc160e50 2024-03-18T20:26:00.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.622+0000 7ff8d0ed9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc0cf210 con 0x7ff8cc160a70 2024-03-18T20:26:00.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.622+0000 7ff8ca59c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8cc160a70 0x7ff8cc0d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.622+0000 7ff8c9d9b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff8cc160e50 0x7ff8cc0cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.622+0000 7ff8ca59c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8cc160a70 0x7ff8cc0d3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:53022/0 (socket says 172.21.15.1:53022) 2024-03-18T20:26:00.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.622+0000 7ff8cad9d700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7ff8cc1612d0 0x7ff8cc0c6da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:59192/0 (socket says 172.21.15.1:59192) 2024-03-18T20:26:00.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.622+0000 7ff8ca59c700 1 -- 172.21.15.1:0/3582232742 learned_addr learned my addr 172.21.15.1:0/3582232742 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:00.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.622+0000 7ff8ca59c700 1 -- 172.21.15.1:0/3582232742 >> v2:172.21.15.165:3300/0 conn(0x7ff8cc160e50 msgr2=0x7ff8cc0cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:00.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.622+0000 7ff8ca59c700 1 --2- 172.21.15.1:0/3582232742 >> v2:172.21.15.165:3300/0 conn(0x7ff8cc160e50 0x7ff8cc0cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.622+0000 7ff8ca59c700 1 -- 172.21.15.1:0/3582232742 >> v1:172.21.15.165:6789/0 conn(0x7ff8cc1612d0 legacy=0x7ff8cc0c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.622+0000 7ff8ca59c700 1 -- 172.21.15.1:0/3582232742 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8cc0cf4f0 con 0x7ff8cc160a70 2024-03-18T20:26:00.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.623+0000 7ff8ca59c700 1 --2- 172.21.15.1:0/3582232742 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8cc160a70 0x7ff8cc0d3b70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc0098f0 tx=0x7ff8bc009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b68f0fc9fa8847f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.623+0000 7ff8c959a700 1 -- 172.21.15.1:0/3582232742 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc011830 con 0x7ff8cc160a70 2024-03-18T20:26:00.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.623+0000 7ff8c959a700 1 -- 172.21.15.1:0/3582232742 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8bc011990 con 0x7ff8cc160a70 2024-03-18T20:26:00.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.623+0000 7ff8c959a700 1 -- 172.21.15.1:0/3582232742 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc011d20 con 0x7ff8cc160a70 2024-03-18T20:26:00.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.623+0000 7ff8d0ed9700 1 -- 172.21.15.1:0/3582232742 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8cc160a70 msgr2=0x7ff8cc0d3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.623+0000 7ff8d0ed9700 1 --2- 172.21.15.1:0/3582232742 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8cc160a70 0x7ff8cc0d3b70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc0098f0 tx=0x7ff8bc009c00 comp rx=0 tx=0).stop 2024-03-18T20:26:00.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.623+0000 7ff8d0ed9700 1 -- 172.21.15.1:0/3582232742 shutdown_connections 2024-03-18T20:26:00.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.623+0000 7ff8d0ed9700 1 --2- 172.21.15.1:0/3582232742 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8cc160a70 0x7ff8cc0d3b70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.623+0000 7ff8d0ed9700 1 --2- 172.21.15.1:0/3582232742 >> v2:172.21.15.165:3300/0 conn(0x7ff8cc160e50 0x7ff8cc0cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.623+0000 7ff8d0ed9700 1 -- 172.21.15.1:0/3582232742 >> 172.21.15.1:0/3582232742 conn(0x7ff8cc09c560 msgr2=0x7ff8cc0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.623+0000 7ff8d0ed9700 1 -- 172.21.15.1:0/3582232742 shutdown_connections 2024-03-18T20:26:00.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.623+0000 7ff8d0ed9700 1 -- 172.21.15.1:0/3582232742 wait complete. 2024-03-18T20:26:00.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.625+0000 7ff8d0ed9700 1 Processor -- start 2024-03-18T20:26:00.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.625+0000 7ff8d0ed9700 1 -- start start 2024-03-18T20:26:00.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8d0ed9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8cc160a70 0x7ff8cc0b07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8d0ed9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8cc160e50 0x7ff8cc1f7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8d0ed9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8cc1612d0 0x7ff8cc15f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8d0ed9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc0b64d0 con 0x7ff8cc1612d0 2024-03-18T20:26:00.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8d0ed9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc0b1090 con 0x7ff8cc160e50 2024-03-18T20:26:00.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8d0ed9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc0b0db0 con 0x7ff8cc160a70 2024-03-18T20:26:00.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8cad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8cc1612d0 0x7ff8cc15f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8cad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8cc1612d0 0x7ff8cc15f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33740/0 (socket says 172.21.15.1:33740) 2024-03-18T20:26:00.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8cad9d700 1 -- 172.21.15.1:0/3522848392 learned_addr learned my addr 172.21.15.1:0/3522848392 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:00.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8c9d9b700 1 --2- 172.21.15.1:0/3522848392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8cc160e50 0x7ff8cc1f7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8ca59c700 1 --2- 172.21.15.1:0/3522848392 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8cc160a70 0x7ff8cc0b07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:00.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8cad9d700 1 -- 172.21.15.1:0/3522848392 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8cc160a70 msgr2=0x7ff8cc0b07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8cad9d700 1 --2- 172.21.15.1:0/3522848392 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8cc160a70 0x7ff8cc0b07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8cad9d700 1 -- 172.21.15.1:0/3522848392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8cc160e50 msgr2=0x7ff8cc1f7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8cad9d700 1 --2- 172.21.15.1:0/3522848392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8cc160e50 0x7ff8cc1f7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8cad9d700 1 -- 172.21.15.1:0/3522848392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8bc009550 con 0x7ff8cc1612d0 2024-03-18T20:26:00.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8c9d9b700 1 --2- 172.21.15.1:0/3522848392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8cc160e50 0x7ff8cc1f7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:26:00.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8ca59c700 1 --2- 172.21.15.1:0/3522848392 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8cc160a70 0x7ff8cc0b07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:26:00.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8cad9d700 1 --2- 172.21.15.1:0/3522848392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8cc1612d0 0x7ff8cc15f190 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400cbd0 tx=0x7ff8c400cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:00.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8baffd700 1 -- 172.21.15.1:0/3522848392 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c400aaa0 con 0x7ff8cc1612d0 2024-03-18T20:26:00.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.626+0000 7ff8baffd700 1 -- 172.21.15.1:0/3522848392 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c400ac00 con 0x7ff8cc1612d0 2024-03-18T20:26:00.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.627+0000 7ff8baffd700 1 -- 172.21.15.1:0/3522848392 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4007070 con 0x7ff8cc1612d0 2024-03-18T20:26:00.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.627+0000 7ff8d0ed9700 1 -- 172.21.15.1:0/3522848392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8cc0b5e80 con 0x7ff8cc1612d0 2024-03-18T20:26:00.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.627+0000 7ff8d0ed9700 1 -- 172.21.15.1:0/3522848392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8cc0b8790 con 0x7ff8cc1612d0 2024-03-18T20:26:00.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.627+0000 7ff8d0ed9700 1 -- 172.21.15.1:0/3522848392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8cc067390 con 0x7ff8cc1612d0 2024-03-18T20:26:00.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.628+0000 7ff8baffd700 1 -- 172.21.15.1:0/3522848392 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 142) v1 ==== 89398+0+0 (secure 0 0 0) 0x7ff8c40071d0 con 0x7ff8cc1612d0 2024-03-18T20:26:00.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.629+0000 7ff8baffd700 1 --2- 172.21.15.1:0/3522848392 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff8ac06af60 0x7ff8ac06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:00.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.629+0000 7ff8ca59c700 1 -- 172.21.15.1:0/3522848392 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff8ac06af60 msgr2=0x7ff8ac06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:00.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.629+0000 7ff8baffd700 1 -- 172.21.15.1:0/3522848392 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7ff8c4089c20 con 0x7ff8cc1612d0 2024-03-18T20:26:00.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.629+0000 7ff8ca59c700 1 --2- 172.21.15.1:0/3522848392 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff8ac06af60 0x7ff8ac06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:00.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.633+0000 7ff8baffd700 1 -- 172.21.15.1:0/3522848392 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff8c4054600 con 0x7ff8cc1612d0 2024-03-18T20:26:00.659 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:00.655+0000 7f583fa9c240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:26:00.720 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:00.716+0000 7f47e8e57240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:26:00.736 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:00.733+0000 7f583fa9c240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:26:00.797 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:00.793+0000 7f47e8e57240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:26:00.814 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:00.811+0000 7f583fa9c240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:26:00.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.830+0000 7ff8ca59c700 1 -- 172.21.15.1:0/3522848392 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff8ac06af60 msgr2=0x7ff8ac06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:00.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.830+0000 7ff8ca59c700 1 --2- 172.21.15.1:0/3522848392 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff8ac06af60 0x7ff8ac06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:00.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.855+0000 7ff8d0ed9700 1 -- 172.21.15.1:0/3522848392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff8cc162170 con 0x7ff8cc1612d0 2024-03-18T20:26:00.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.856+0000 7ff8baffd700 1 -- 172.21.15.1:0/3522848392 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v237) v1 ==== 80+0+454 (secure 0 0 0) 0x7ff8c40171e0 con 0x7ff8cc1612d0 2024-03-18T20:26:00.860 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2535232994 2024-03-19T20:25:52.933722+0000 2024-03-18T20:26:00.860 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2652258802 2024-03-19T20:25:52.933722+0000 2024-03-18T20:26:00.860 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2404324807 2024-03-19T20:25:52.933722+0000 2024-03-18T20:26:00.860 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3657737679 2024-03-19T20:25:52.933722+0000 2024-03-18T20:26:00.860 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6833/148356 2024-03-19T20:25:52.933722+0000 2024-03-18T20:26:00.860 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2324250327 2024-03-19T20:25:52.933722+0000 2024-03-18T20:26:00.860 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6832/148356 2024-03-19T20:25:52.933722+0000 2024-03-18T20:26:00.860 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2165965129 2024-03-19T20:25:52.933722+0000 2024-03-18T20:26:00.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.858+0000 7ff8d0ed9700 1 -- 172.21.15.1:0/3522848392 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff8ac06af60 msgr2=0x7ff8ac06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.858+0000 7ff8d0ed9700 1 --2- 172.21.15.1:0/3522848392 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff8ac06af60 0x7ff8ac06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.858+0000 7ff8d0ed9700 1 -- 172.21.15.1:0/3522848392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8cc1612d0 msgr2=0x7ff8cc15f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.858+0000 7ff8d0ed9700 1 --2- 172.21.15.1:0/3522848392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8cc1612d0 0x7ff8cc15f190 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400cbd0 tx=0x7ff8c400cee0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.858+0000 7ff8d0ed9700 1 -- 172.21.15.1:0/3522848392 shutdown_connections 2024-03-18T20:26:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.858+0000 7ff8d0ed9700 1 --2- 172.21.15.1:0/3522848392 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff8ac06af60 0x7ff8ac06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.858+0000 7ff8d0ed9700 1 --2- 172.21.15.1:0/3522848392 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff8cc160a70 0x7ff8cc0b07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.858+0000 7ff8d0ed9700 1 --2- 172.21.15.1:0/3522848392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8cc1612d0 0x7ff8cc15f190 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.858+0000 7ff8d0ed9700 1 --2- 172.21.15.1:0/3522848392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff8cc160e50 0x7ff8cc1f7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.858+0000 7ff8d0ed9700 1 -- 172.21.15.1:0/3522848392 >> 172.21.15.1:0/3522848392 conn(0x7ff8cc09c560 msgr2=0x7ff8cc0c9560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.858+0000 7ff8d0ed9700 1 -- 172.21.15.1:0/3522848392 shutdown_connections 2024-03-18T20:26:00.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:00.858+0000 7ff8d0ed9700 1 -- 172.21.15.1:0/3522848392 wait complete. 2024-03-18T20:26:00.864 INFO:teuthology.orchestra.run.smithi001.stderr:listed 8 entries 2024-03-18T20:26:00.875 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-18T20:26:00.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:00.875 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T20:26:00.881 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:00.877+0000 7f47e8e57240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:26:00.918 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[158300]: fuse finished with error 0 and tester_r 0 2024-03-18T20:26:00.919 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:26:00.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:00.920 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T20:26:00.965 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:00.961+0000 7f47e8e57240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:26:00.985 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:00.982+0000 7f583fa9c240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:26:01.002 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:26:01.002 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:26:01.021 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:26:01.021 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:26:01.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:01.021 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-18T20:26:01.049 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:01.046+0000 7f47e8e57240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:26:01.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:26:01.079 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-18T20:26:01.080 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:26:01.080 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:26:01.098 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:26:01.098 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:26:01.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:01.098 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-18T20:26:01.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:26:01.159 INFO:teuthology.orchestra.run.smithi001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-18T20:26:01.159 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:26:01.159 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:26:01.177 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:26:01.178 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:26:01.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:01.178 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-18T20:26:01.206 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:01.203+0000 7f583fa9c240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:26:01.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:26:01.223 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-18T20:26:01.223 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:26:01.234 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:01.230+0000 7f47e8e57240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:26:01.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.350+0000 7f3045761700 1 Processor -- start 2024-03-18T20:26:01.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.350+0000 7f3045761700 1 -- start start 2024-03-18T20:26:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.351+0000 7f3045761700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f304015d500 0x7f30400d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.351+0000 7f3045761700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f30400d1300 0x7f30400d17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.351+0000 7f3045761700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30400d1cf0 0x7f30400b83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.351+0000 7f3045761700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30400d2120 con 0x7f304015d500 2024-03-18T20:26:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.351+0000 7f3045761700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30400cddc0 con 0x7f30400d1300 2024-03-18T20:26:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.351+0000 7f3045761700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30400d3dc0 con 0x7f30400d1cf0 2024-03-18T20:26:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.351+0000 7f303effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f304015d500 0x7f30400d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.351+0000 7f303effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f304015d500 0x7f30400d0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33750/0 (socket says 172.21.15.1:33750) 2024-03-18T20:26:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.351+0000 7f303effd700 1 -- 172.21.15.1:0/402196984 learned_addr learned my addr 172.21.15.1:0/402196984 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.351+0000 7f303f7fe700 1 --2- 172.21.15.1:0/402196984 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30400d1cf0 0x7f30400b83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.351+0000 7f303e7fc700 1 --2- 172.21.15.1:0/402196984 >> v2:172.21.15.165:3300/0 conn(0x7f30400d1300 0x7f30400d17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:01.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.351+0000 7f303effd700 1 -- 172.21.15.1:0/402196984 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30400d1cf0 msgr2=0x7f30400b83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:01.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.351+0000 7f303effd700 1 --2- 172.21.15.1:0/402196984 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30400d1cf0 0x7f30400b83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:01.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.351+0000 7f303effd700 1 -- 172.21.15.1:0/402196984 >> v2:172.21.15.165:3300/0 conn(0x7f30400d1300 msgr2=0x7f30400d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:01.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.351+0000 7f303effd700 1 --2- 172.21.15.1:0/402196984 >> v2:172.21.15.165:3300/0 conn(0x7f30400d1300 0x7f30400d17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:01.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.351+0000 7f303effd700 1 -- 172.21.15.1:0/402196984 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30400ce060 con 0x7f304015d500 2024-03-18T20:26:01.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.352+0000 7f303effd700 1 --2- 172.21.15.1:0/402196984 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f304015d500 0x7f30400d0dc0 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f30400cd7d0 tx=0x7f3030009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a18a19145cc9564 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:01.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.352+0000 7f303dffb700 1 -- 172.21.15.1:0/402196984 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3030011830 con 0x7f304015d500 2024-03-18T20:26:01.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.352+0000 7f303dffb700 1 -- 172.21.15.1:0/402196984 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3030011990 con 0x7f304015d500 2024-03-18T20:26:01.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.352+0000 7f303dffb700 1 -- 172.21.15.1:0/402196984 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3030011d20 con 0x7f304015d500 2024-03-18T20:26:01.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.352+0000 7f3045761700 1 -- 172.21.15.1:0/402196984 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f304015d500 msgr2=0x7f30400d0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:01.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.352+0000 7f3045761700 1 --2- 172.21.15.1:0/402196984 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f304015d500 0x7f30400d0dc0 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f30400cd7d0 tx=0x7f3030009bd0 comp rx=0 tx=0).stop 2024-03-18T20:26:01.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.352+0000 7f3045761700 1 -- 172.21.15.1:0/402196984 shutdown_connections 2024-03-18T20:26:01.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.352+0000 7f3045761700 1 --2- 172.21.15.1:0/402196984 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30400d1cf0 0x7f30400b83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:01.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.352+0000 7f3045761700 1 --2- 172.21.15.1:0/402196984 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f304015d500 0x7f30400d0dc0 unknown :-1 s=CLOSED pgs=2666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:01.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.352+0000 7f3045761700 1 --2- 172.21.15.1:0/402196984 >> v2:172.21.15.165:3300/0 conn(0x7f30400d1300 0x7f30400d17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:01.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.352+0000 7f3045761700 1 -- 172.21.15.1:0/402196984 >> 172.21.15.1:0/402196984 conn(0x7f304009c530 msgr2=0x7f30400d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:01.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.353+0000 7f3045761700 1 -- 172.21.15.1:0/402196984 shutdown_connections 2024-03-18T20:26:01.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.353+0000 7f3045761700 1 -- 172.21.15.1:0/402196984 wait complete. 2024-03-18T20:26:01.360 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:01.356+0000 7f583fa9c240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:26:01.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.354+0000 7f3045761700 1 Processor -- start 2024-03-18T20:26:01.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.354+0000 7f3045761700 1 -- start start 2024-03-18T20:26:01.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f3045761700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30400d1300 0x7f30401e8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:01.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f3045761700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30400d1cf0 0x7f30400ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:01.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f3045761700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f304015d500 0x7f304015aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:01.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f3045761700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30400c7ad0 con 0x7f304015d500 2024-03-18T20:26:01.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f3045761700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30400c6e50 con 0x7f30400d1300 2024-03-18T20:26:01.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f3045761700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30400c7130 con 0x7f30400d1cf0 2024-03-18T20:26:01.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f303f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f304015d500 0x7f304015aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:01.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f303f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f304015d500 0x7f304015aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33764/0 (socket says 172.21.15.1:33764) 2024-03-18T20:26:01.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f303f7fe700 1 -- 172.21.15.1:0/268730533 learned_addr learned my addr 172.21.15.1:0/268730533 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:01.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f303effd700 1 --2- 172.21.15.1:0/268730533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30400d1300 0x7f30401e8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:01.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f303e7fc700 1 --2- 172.21.15.1:0/268730533 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30400d1cf0 0x7f30400ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:01.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f303f7fe700 1 -- 172.21.15.1:0/268730533 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30400d1cf0 msgr2=0x7f30400ca2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:01.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f303f7fe700 1 --2- 172.21.15.1:0/268730533 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30400d1cf0 0x7f30400ca2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:01.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f303f7fe700 1 -- 172.21.15.1:0/268730533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30400d1300 msgr2=0x7f30401e8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:01.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f303f7fe700 1 --2- 172.21.15.1:0/268730533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30400d1300 0x7f30401e8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:01.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f303f7fe700 1 -- 172.21.15.1:0/268730533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3030009550 con 0x7f304015d500 2024-03-18T20:26:01.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f303f7fe700 1 --2- 172.21.15.1:0/268730533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f304015d500 0x7f304015aba0 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f303800efc0 tx=0x7f303800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:01.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f30277fe700 1 -- 172.21.15.1:0/268730533 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3038009ea0 con 0x7f304015d500 2024-03-18T20:26:01.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f30277fe700 1 -- 172.21.15.1:0/268730533 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f303800f040 con 0x7f304015d500 2024-03-18T20:26:01.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.355+0000 7f3045761700 1 -- 172.21.15.1:0/268730533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30400c7410 con 0x7f304015d500 2024-03-18T20:26:01.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.356+0000 7f3045761700 1 -- 172.21.15.1:0/268730533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30401e94e0 con 0x7f304015d500 2024-03-18T20:26:01.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.356+0000 7f30277fe700 1 -- 172.21.15.1:0/268730533 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303800ced0 con 0x7f304015d500 2024-03-18T20:26:01.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.357+0000 7f30277fe700 1 -- 172.21.15.1:0/268730533 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 142) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f3038014480 con 0x7f304015d500 2024-03-18T20:26:01.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.357+0000 7f3045761700 1 -- 172.21.15.1:0/268730533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30400c7c10 con 0x7f304015d500 2024-03-18T20:26:01.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.357+0000 7f30277fe700 1 --2- 172.21.15.1:0/268730533 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302c06af60 0x7f302c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:01.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.357+0000 7f303effd700 1 -- 172.21.15.1:0/268730533 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302c06af60 msgr2=0x7f302c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:01.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.357+0000 7f30277fe700 1 -- 172.21.15.1:0/268730533 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7f303808aa40 con 0x7f304015d500 2024-03-18T20:26:01.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.358+0000 7f303effd700 1 --2- 172.21.15.1:0/268730533 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302c06af60 0x7f302c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:01.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.362+0000 7f30277fe700 1 -- 172.21.15.1:0/268730533 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f30400c7c10 con 0x7f304015d500 2024-03-18T20:26:01.433 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:01.430+0000 7f583fa9c240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:26:01.462 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:01.458+0000 7f47e8e57240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:26:01.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.559+0000 7f303effd700 1 -- 172.21.15.1:0/268730533 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302c06af60 msgr2=0x7f302c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:01.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.559+0000 7f303effd700 1 --2- 172.21.15.1:0/268730533 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302c06af60 0x7f302c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:01.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.614+0000 7f3045761700 1 -- 172.21.15.1:0/268730533 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30400c7c10 con 0x7f304015d500 2024-03-18T20:26:01.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.614+0000 7f30277fe700 1 -- 172.21.15.1:0/268730533 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5963 (secure 0 0 0) 0x7f30400c7c10 con 0x7f304015d500 2024-03-18T20:26:01.618 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:01.619 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":110,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10369,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/677258734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":677258734},{"type":"v1","addr":"172.21.15.165:6835","nonce":677258734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":11544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2018398457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2018398457},{"type":"v1","addr":"172.21.15.1:6836","nonce":2018398457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:25:22.663717+0000","modified":"2024-03-18T20:25:25.668584+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11556":{"gid":11556,"name":"d","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6834/4024828630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4024828630},{"type":"v1","addr":"172.21.15.165:6834","nonce":4024828630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:25:35.549382+0000","modified":"2024-03-18T20:25:38.565325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11547":{"gid":11547,"name":"c","rank":0,"incarnation":106,"state":"up:active","state_seq":6,"addr":"172.21.15.1:6837/1635684321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1635684321},{"type":"v1","addr":"172.21.15.1:6837","nonce":1635684321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-03-18T20:26:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.617+0000 7f3045761700 1 -- 172.21.15.1:0/268730533 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302c06af60 msgr2=0x7f302c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.617+0000 7f3045761700 1 --2- 172.21.15.1:0/268730533 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302c06af60 0x7f302c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.617+0000 7f3045761700 1 -- 172.21.15.1:0/268730533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f304015d500 msgr2=0x7f304015aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.617+0000 7f3045761700 1 --2- 172.21.15.1:0/268730533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f304015d500 0x7f304015aba0 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f303800efc0 tx=0x7f303800c3a0 comp rx=0 tx=0).stop 2024-03-18T20:26:01.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.617+0000 7f3045761700 1 -- 172.21.15.1:0/268730533 shutdown_connections 2024-03-18T20:26:01.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.617+0000 7f3045761700 1 --2- 172.21.15.1:0/268730533 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302c06af60 0x7f302c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:01.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.617+0000 7f3045761700 1 --2- 172.21.15.1:0/268730533 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f304015d500 0x7f304015aba0 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:01.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.617+0000 7f3045761700 1 --2- 172.21.15.1:0/268730533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30400d1300 0x7f30401e8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:01.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.617+0000 7f3045761700 1 --2- 172.21.15.1:0/268730533 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30400d1cf0 0x7f30400ca2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:01.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.617+0000 7f3045761700 1 -- 172.21.15.1:0/268730533 >> 172.21.15.1:0/268730533 conn(0x7f304009c530 msgr2=0x7f304006bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:01.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.617+0000 7f3045761700 1 -- 172.21.15.1:0/268730533 shutdown_connections 2024-03-18T20:26:01.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.617+0000 7f3045761700 1 -- 172.21.15.1:0/268730533 wait complete. 2024-03-18T20:26:01.622 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 110 2024-03-18T20:26:01.624 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:01.620+0000 7f47e8e57240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:26:01.634 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:26:01.704 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:01.700+0000 7f47e8e57240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:26:01.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.796+0000 7f971911a700 1 Processor -- start 2024-03-18T20:26:01.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.796+0000 7f971911a700 1 -- start start 2024-03-18T20:26:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.797+0000 7f971911a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f971415b4d0 0x7f971409a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.797+0000 7f971911a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f971406f070 con 0x7f97140cf750 2024-03-18T20:26:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.797+0000 7f971911a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97140d0260 con 0x7f971409a150 2024-03-18T20:26:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.797+0000 7f971911a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97140cff80 con 0x7f971415b4d0 2024-03-18T20:26:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.797+0000 7f971259c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f97140cf750 0x7f9714099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47278/0 (socket says 172.21.15.1:47278) 2024-03-18T20:26:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.797+0000 7f971259c700 1 -- 172.21.15.1:0/2754250659 learned_addr learned my addr 172.21.15.1:0/2754250659 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.797+0000 7f9712d9d700 1 --2- 172.21.15.1:0/2754250659 >> v2:172.21.15.165:3300/0 conn(0x7f971415b4d0 0x7f971409a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.798+0000 7f9711d9b700 1 -- 172.21.15.1:0/2754250659 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989132141 0 0) 0x7f971406f070 con 0x7f97140cf750 2024-03-18T20:26:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.798+0000 7f9711d9b700 1 -- 172.21.15.1:0/2754250659 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96fc0035b0 con 0x7f97140cf750 2024-03-18T20:26:01.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.798+0000 7f9711d9b700 1 -- 172.21.15.1:0/2754250659 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769470245 0 0) 0x7f97140d0260 con 0x7f971409a150 2024-03-18T20:26:01.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.798+0000 7f9711d9b700 1 -- 172.21.15.1:0/2754250659 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f971406f070 con 0x7f971409a150 2024-03-18T20:26:01.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.798+0000 7f9712d9d700 1 -- 172.21.15.1:0/2754250659 >> v1:172.21.15.165:6789/0 conn(0x7f971409a150 legacy=0x7f971406f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:01.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.798+0000 7f9712d9d700 1 -- 172.21.15.1:0/2754250659 >> v1:172.21.15.1:6789/0 conn(0x7f97140cf750 legacy=0x7f9714099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:01.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.798+0000 7f9712d9d700 1 -- 172.21.15.1:0/2754250659 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971406ea20 con 0x7f971415b4d0 2024-03-18T20:26:01.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.798+0000 7f9712d9d700 1 --2- 172.21.15.1:0/2754250659 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f971415b4d0 0x7f971409a650 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f9704009840 tx=0x7f9704009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d352dc587cca162 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:01.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.798+0000 7f9711d9b700 1 -- 172.21.15.1:0/2754250659 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9704011830 con 0x7f971415b4d0 2024-03-18T20:26:01.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.798+0000 7f9711d9b700 1 -- 172.21.15.1:0/2754250659 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9704011990 con 0x7f971415b4d0 2024-03-18T20:26:01.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.798+0000 7f9711d9b700 1 -- 172.21.15.1:0/2754250659 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9704011d20 con 0x7f971415b4d0 2024-03-18T20:26:01.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.799+0000 7f971911a700 1 -- 172.21.15.1:0/2754250659 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f971415b4d0 msgr2=0x7f971409a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:01.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.799+0000 7f971911a700 1 --2- 172.21.15.1:0/2754250659 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f971415b4d0 0x7f971409a650 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f9704009840 tx=0x7f9704009b50 comp rx=0 tx=0).stop 2024-03-18T20:26:01.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.799+0000 7f971911a700 1 -- 172.21.15.1:0/2754250659 shutdown_connections 2024-03-18T20:26:01.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.799+0000 7f971911a700 1 --2- 172.21.15.1:0/2754250659 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f971415b4d0 0x7f971409a650 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:01.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.799+0000 7f971911a700 1 -- 172.21.15.1:0/2754250659 >> 172.21.15.1:0/2754250659 conn(0x7f97140b2cb0 msgr2=0x7f97140b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:01.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.799+0000 7f971911a700 1 -- 172.21.15.1:0/2754250659 shutdown_connections 2024-03-18T20:26:01.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.799+0000 7f971911a700 1 -- 172.21.15.1:0/2754250659 wait complete. 2024-03-18T20:26:01.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.800+0000 7f971911a700 1 Processor -- start 2024-03-18T20:26:01.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.800+0000 7f971911a700 1 -- start start 2024-03-18T20:26:01.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.801+0000 7f971911a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f971409a150 0x7f971406bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:01.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.801+0000 7f971911a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97140cf750 0x7f971406d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:01.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.801+0000 7f971911a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f971415b4d0 0x7f9714150630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:01.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.801+0000 7f971911a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9714150b70 con 0x7f971415b4d0 2024-03-18T20:26:01.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.801+0000 7f971911a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971406c3c0 con 0x7f97140cf750 2024-03-18T20:26:01.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.801+0000 7f971911a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f971406c6a0 con 0x7f971409a150 2024-03-18T20:26:01.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.801+0000 7f971359e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f971415b4d0 0x7f9714150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:01.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.801+0000 7f971259c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97140cf750 0x7f971406d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:01.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.801+0000 7f971359e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f971415b4d0 0x7f9714150630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33774/0 (socket says 172.21.15.1:33774) 2024-03-18T20:26:01.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.801+0000 7f971359e700 1 -- 172.21.15.1:0/666450804 learned_addr learned my addr 172.21.15.1:0/666450804 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:01.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.801+0000 7f9712d9d700 1 --2- 172.21.15.1:0/666450804 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f971409a150 0x7f971406bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:01.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.802+0000 7f971359e700 1 -- 172.21.15.1:0/666450804 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f971409a150 msgr2=0x7f971406bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:01.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.802+0000 7f971359e700 1 --2- 172.21.15.1:0/666450804 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f971409a150 0x7f971406bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:01.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.802+0000 7f971359e700 1 -- 172.21.15.1:0/666450804 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97140cf750 msgr2=0x7f971406d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:01.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.802+0000 7f971359e700 1 --2- 172.21.15.1:0/666450804 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97140cf750 0x7f971406d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:01.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.802+0000 7f971359e700 1 -- 172.21.15.1:0/666450804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97040094a0 con 0x7f971415b4d0 2024-03-18T20:26:01.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.802+0000 7f971259c700 1 --2- 172.21.15.1:0/666450804 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97140cf750 0x7f971406d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:26:01.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.802+0000 7f971359e700 1 --2- 172.21.15.1:0/666450804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f971415b4d0 0x7f9714150630 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f970c00cbc0 tx=0x7f970c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:01.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.802+0000 7f96fb7fe700 1 -- 172.21.15.1:0/666450804 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970c003c00 con 0x7f971415b4d0 2024-03-18T20:26:01.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.802+0000 7f96fb7fe700 1 -- 172.21.15.1:0/666450804 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970c003d60 con 0x7f971415b4d0 2024-03-18T20:26:01.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.802+0000 7f971911a700 1 -- 172.21.15.1:0/666450804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971406c980 con 0x7f971415b4d0 2024-03-18T20:26:01.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.802+0000 7f96fb7fe700 1 -- 172.21.15.1:0/666450804 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970c00aae0 con 0x7f971415b4d0 2024-03-18T20:26:01.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.802+0000 7f971911a700 1 -- 172.21.15.1:0/666450804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f971406cf40 con 0x7f971415b4d0 2024-03-18T20:26:01.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.803+0000 7f971911a700 1 -- 172.21.15.1:0/666450804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9714068240 con 0x7f971415b4d0 2024-03-18T20:26:01.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.803+0000 7f96fb7fe700 1 -- 172.21.15.1:0/666450804 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 142) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f970c00ac40 con 0x7f971415b4d0 2024-03-18T20:26:01.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.804+0000 7f96fb7fe700 1 --2- 172.21.15.1:0/666450804 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f970006af60 0x7f970006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:01.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.804+0000 7f9712d9d700 1 -- 172.21.15.1:0/666450804 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f970006af60 msgr2=0x7f970006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:01.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.804+0000 7f96fb7fe700 1 -- 172.21.15.1:0/666450804 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7f970c089d80 con 0x7f971415b4d0 2024-03-18T20:26:01.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.804+0000 7f9712d9d700 1 --2- 172.21.15.1:0/666450804 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f970006af60 0x7f970006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:01.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:01.808+0000 7f96fb7fe700 1 -- 172.21.15.1:0/666450804 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f970c0545b0 con 0x7f971415b4d0 2024-03-18T20:26:01.876 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:01.873+0000 7f583fa9c240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:26:02.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.005+0000 7f9712d9d700 1 -- 172.21.15.1:0/666450804 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f970006af60 msgr2=0x7f970006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:02.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.005+0000 7f9712d9d700 1 --2- 172.21.15.1:0/666450804 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f970006af60 0x7f970006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:02.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.013+0000 7f971911a700 1 -- 172.21.15.1:0/666450804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97140676a0 con 0x7f971415b4d0 2024-03-18T20:26:02.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.015+0000 7f96fb7fe700 1 -- 172.21.15.1:0/666450804 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v237) v1 ==== 74+0+20570 (secure 0 0 0) 0x7f970c057bd0 con 0x7f971415b4d0 2024-03-18T20:26:02.019 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:02.019 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":237,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:25:52.933780+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:25:20.511297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-18T20:25:21.637693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:25:33.275375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:25:34.518347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2535232994":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2652258802":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2404324807":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/3657737679":"2024-03-19T20:25:52.933722+0000","172.21.15.1:6833/148356":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2324250327":"2024-03-19T20:25:52.933722+0000","172.21.15.1:6832/148356":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2165965129":"2024-03-19T20:25:52.933722+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:26:02.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.017+0000 7f971911a700 1 -- 172.21.15.1:0/666450804 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f970006af60 msgr2=0x7f970006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:02.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.017+0000 7f971911a700 1 --2- 172.21.15.1:0/666450804 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f970006af60 0x7f970006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.017+0000 7f971911a700 1 -- 172.21.15.1:0/666450804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f971415b4d0 msgr2=0x7f9714150630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.017+0000 7f971911a700 1 --2- 172.21.15.1:0/666450804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f971415b4d0 0x7f9714150630 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f970c00cbc0 tx=0x7f970c00ced0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.017+0000 7f971911a700 1 -- 172.21.15.1:0/666450804 shutdown_connections 2024-03-18T20:26:02.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.017+0000 7f971911a700 1 --2- 172.21.15.1:0/666450804 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f970006af60 0x7f970006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.017+0000 7f971911a700 1 --2- 172.21.15.1:0/666450804 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f971409a150 0x7f971406bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.017+0000 7f971911a700 1 --2- 172.21.15.1:0/666450804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f971415b4d0 0x7f9714150630 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.017+0000 7f971911a700 1 --2- 172.21.15.1:0/666450804 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97140cf750 0x7f971406d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.017+0000 7f971911a700 1 -- 172.21.15.1:0/666450804 >> 172.21.15.1:0/666450804 conn(0x7f97140b2cb0 msgr2=0x7f97140b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:02.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.018+0000 7f971911a700 1 -- 172.21.15.1:0/666450804 shutdown_connections 2024-03-18T20:26:02.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.018+0000 7f971911a700 1 -- 172.21.15.1:0/666450804 wait complete. 2024-03-18T20:26:02.035 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:25:20.511297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '231', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:25:21.637693+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '232', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-03-18T20:25:33.275375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '235', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2024-03-18T20:25:34.518347+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '236', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:26:02.035 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:26:02.195 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:02.192+0000 7f47e8e57240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:26:02.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.201+0000 7f0b5d9a2700 1 Processor -- start 2024-03-18T20:26:02.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.201+0000 7f0b5d9a2700 1 -- start start 2024-03-18T20:26:02.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.201+0000 7f0b5d9a2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b580b4810 0x7f0b580d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.201+0000 7f0b5d9a2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0b580cde70 0x7f0b580ce320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.202+0000 7f0b5d9a2700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b580b3370 con 0x7f0b580cfb50 2024-03-18T20:26:02.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.202+0000 7f0b5d9a2700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b580b2760 con 0x7f0b580cde70 2024-03-18T20:26:02.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.202+0000 7f0b5d9a2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b580b2a40 con 0x7f0b580b4810 2024-03-18T20:26:02.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.202+0000 7f0b577fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0b580cde70 0x7f0b580ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:02.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.202+0000 7f0b56ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0b580cfb50 0x7f0b580cff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47292/0 (socket says 172.21.15.1:47292) 2024-03-18T20:26:02.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.202+0000 7f0b56ffd700 1 -- 172.21.15.1:0/3791494491 learned_addr learned my addr 172.21.15.1:0/3791494491 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:02.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.202+0000 7f0b567fc700 1 --2- 172.21.15.1:0/3791494491 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b580b4810 0x7f0b580d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:02.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.202+0000 7f0b577fe700 1 -- 172.21.15.1:0/3791494491 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b580b4810 msgr2=0x7f0b580d3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:02.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.202+0000 7f0b577fe700 1 --2- 172.21.15.1:0/3791494491 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b580b4810 0x7f0b580d3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.202+0000 7f0b577fe700 1 -- 172.21.15.1:0/3791494491 >> v1:172.21.15.1:6789/0 conn(0x7f0b580cfb50 legacy=0x7f0b580cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.202+0000 7f0b577fe700 1 -- 172.21.15.1:0/3791494491 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b580b2d20 con 0x7f0b580cde70 2024-03-18T20:26:02.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.202+0000 7f0b577fe700 1 --2- 172.21.15.1:0/3791494491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b580cde70 0x7f0b580ce320 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009d90 tx=0x7f0b5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ca4642967018ab server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:02.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.202+0000 7f0b55ffb700 1 -- 172.21.15.1:0/3791494491 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5000e070 con 0x7f0b580cde70 2024-03-18T20:26:02.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.202+0000 7f0b55ffb700 1 -- 172.21.15.1:0/3791494491 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5000bb20 con 0x7f0b580cde70 2024-03-18T20:26:02.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.202+0000 7f0b55ffb700 1 -- 172.21.15.1:0/3791494491 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b50012430 con 0x7f0b580cde70 2024-03-18T20:26:02.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.203+0000 7f0b5d9a2700 1 -- 172.21.15.1:0/3791494491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b580cde70 msgr2=0x7f0b580ce320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.203+0000 7f0b5d9a2700 1 --2- 172.21.15.1:0/3791494491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b580cde70 0x7f0b580ce320 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009d90 tx=0x7f0b5000b330 comp rx=0 tx=0).stop 2024-03-18T20:26:02.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.203+0000 7f0b5d9a2700 1 -- 172.21.15.1:0/3791494491 shutdown_connections 2024-03-18T20:26:02.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.203+0000 7f0b5d9a2700 1 --2- 172.21.15.1:0/3791494491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b580cde70 0x7f0b580ce320 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.203+0000 7f0b5d9a2700 1 --2- 172.21.15.1:0/3791494491 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b580b4810 0x7f0b580d3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.203+0000 7f0b5d9a2700 1 -- 172.21.15.1:0/3791494491 >> 172.21.15.1:0/3791494491 conn(0x7f0b580d3140 msgr2=0x7f0b580d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:02.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.203+0000 7f0b5d9a2700 1 -- 172.21.15.1:0/3791494491 shutdown_connections 2024-03-18T20:26:02.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.203+0000 7f0b5d9a2700 1 -- 172.21.15.1:0/3791494491 wait complete. 2024-03-18T20:26:02.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.204+0000 7f0b5d9a2700 1 Processor -- start 2024-03-18T20:26:02.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.205+0000 7f0b5d9a2700 1 -- start start 2024-03-18T20:26:02.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.205+0000 7f0b5d9a2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b580b4810 0x7f0b5815cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.205+0000 7f0b5d9a2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b580cde70 0x7f0b580b7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.205+0000 7f0b5d9a2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b580cfb50 0x7f0b5815f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.205+0000 7f0b5d9a2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b581f31b0 con 0x7f0b580cde70 2024-03-18T20:26:02.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.205+0000 7f0b5d9a2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5815d070 con 0x7f0b580cfb50 2024-03-18T20:26:02.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.205+0000 7f0b5d9a2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b580c7430 con 0x7f0b580b4810 2024-03-18T20:26:02.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.205+0000 7f0b567fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b580cde70 0x7f0b580b7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:02.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.205+0000 7f0b567fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b580cde70 0x7f0b580b7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33792/0 (socket says 172.21.15.1:33792) 2024-03-18T20:26:02.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.205+0000 7f0b567fc700 1 -- 172.21.15.1:0/2541571064 learned_addr learned my addr 172.21.15.1:0/2541571064 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:02.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.205+0000 7f0b577fe700 1 --2- 172.21.15.1:0/2541571064 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b580cfb50 0x7f0b5815f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:02.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.205+0000 7f0b56ffd700 1 --2- 172.21.15.1:0/2541571064 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b580b4810 0x7f0b5815cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:02.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.206+0000 7f0b567fc700 1 -- 172.21.15.1:0/2541571064 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b580b4810 msgr2=0x7f0b5815cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.206+0000 7f0b567fc700 1 --2- 172.21.15.1:0/2541571064 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b580b4810 0x7f0b5815cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.206+0000 7f0b567fc700 1 -- 172.21.15.1:0/2541571064 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b580cfb50 msgr2=0x7f0b5815f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.206+0000 7f0b567fc700 1 --2- 172.21.15.1:0/2541571064 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b580cfb50 0x7f0b5815f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.206+0000 7f0b567fc700 1 -- 172.21.15.1:0/2541571064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b50009a70 con 0x7f0b580cde70 2024-03-18T20:26:02.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.206+0000 7f0b567fc700 1 --2- 172.21.15.1:0/2541571064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b580cde70 0x7f0b580b7160 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c00e9b0 tx=0x7f0b4c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:02.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.206+0000 7f0b3ffff700 1 -- 172.21.15.1:0/2541571064 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c009710 con 0x7f0b580cde70 2024-03-18T20:26:02.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.206+0000 7f0b3ffff700 1 -- 172.21.15.1:0/2541571064 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4c00ee50 con 0x7f0b580cde70 2024-03-18T20:26:02.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.206+0000 7f0b5d9a2700 1 -- 172.21.15.1:0/2541571064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b580c7710 con 0x7f0b580cde70 2024-03-18T20:26:02.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.206+0000 7f0b3ffff700 1 -- 172.21.15.1:0/2541571064 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c00ca50 con 0x7f0b580cde70 2024-03-18T20:26:02.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.206+0000 7f0b5d9a2700 1 -- 172.21.15.1:0/2541571064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b580c7cd0 con 0x7f0b580cde70 2024-03-18T20:26:02.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.207+0000 7f0b5d9a2700 1 -- 172.21.15.1:0/2541571064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b58073350 con 0x7f0b580cde70 2024-03-18T20:26:02.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.208+0000 7f0b3ffff700 1 -- 172.21.15.1:0/2541571064 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 142) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f0b4c00cbb0 con 0x7f0b580cde70 2024-03-18T20:26:02.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.208+0000 7f0b3ffff700 1 --2- 172.21.15.1:0/2541571064 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b4406af60 0x7f0b4406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.208+0000 7f0b56ffd700 1 -- 172.21.15.1:0/2541571064 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b4406af60 msgr2=0x7f0b4406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:02.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.208+0000 7f0b3ffff700 1 -- 172.21.15.1:0/2541571064 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7f0b4c089d80 con 0x7f0b580cde70 2024-03-18T20:26:02.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.208+0000 7f0b56ffd700 1 --2- 172.21.15.1:0/2541571064 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b4406af60 0x7f0b4406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:02.219 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:02.216+0000 7f583fa9c240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:26:02.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.218+0000 7f0b3ffff700 1 -- 172.21.15.1:0/2541571064 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0b4c054570 con 0x7f0b580cde70 2024-03-18T20:26:02.386 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:02.382+0000 7f583fa9c240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:26:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.409+0000 7f0b56ffd700 1 -- 172.21.15.1:0/2541571064 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b4406af60 msgr2=0x7f0b4406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:02.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.409+0000 7f0b56ffd700 1 --2- 172.21.15.1:0/2541571064 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b4406af60 0x7f0b4406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:02.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.422+0000 7f0b5d9a2700 1 -- 172.21.15.1:0/2541571064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0b58067b30 con 0x7f0b580cde70 2024-03-18T20:26:02.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.423+0000 7f0b3ffff700 1 -- 172.21.15.1:0/2541571064 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v237) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0b4c00ce60 con 0x7f0b580cde70 2024-03-18T20:26:02.427 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:26:02.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.425+0000 7f0b5d9a2700 1 -- 172.21.15.1:0/2541571064 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b4406af60 msgr2=0x7f0b4406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:02.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.425+0000 7f0b5d9a2700 1 --2- 172.21.15.1:0/2541571064 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b4406af60 0x7f0b4406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.425+0000 7f0b5d9a2700 1 -- 172.21.15.1:0/2541571064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b580cde70 msgr2=0x7f0b580b7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.425+0000 7f0b5d9a2700 1 --2- 172.21.15.1:0/2541571064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b580cde70 0x7f0b580b7160 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c00e9b0 tx=0x7f0b4c00ecc0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.425+0000 7f0b5d9a2700 1 -- 172.21.15.1:0/2541571064 shutdown_connections 2024-03-18T20:26:02.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.425+0000 7f0b5d9a2700 1 --2- 172.21.15.1:0/2541571064 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b4406af60 0x7f0b4406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.425+0000 7f0b5d9a2700 1 --2- 172.21.15.1:0/2541571064 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b580b4810 0x7f0b5815cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.425+0000 7f0b5d9a2700 1 --2- 172.21.15.1:0/2541571064 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b580cfb50 0x7f0b5815f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.425+0000 7f0b5d9a2700 1 --2- 172.21.15.1:0/2541571064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b580cde70 0x7f0b580b7160 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.425+0000 7f0b5d9a2700 1 -- 172.21.15.1:0/2541571064 >> 172.21.15.1:0/2541571064 conn(0x7f0b580d3140 msgr2=0x7f0b5806ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:02.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.425+0000 7f0b5d9a2700 1 -- 172.21.15.1:0/2541571064 shutdown_connections 2024-03-18T20:26:02.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.425+0000 7f0b5d9a2700 1 -- 172.21.15.1:0/2541571064 wait complete. 2024-03-18T20:26:02.442 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:26:02.558 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:02.554+0000 7f47e8e57240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:26:02.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.567+0000 7f317dda0700 1 Processor -- start 2024-03-18T20:26:02.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.567+0000 7f317dda0700 1 -- start start 2024-03-18T20:26:02.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.569+0000 7f317dda0700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f31780cec90 0x7f3178165040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.569+0000 7f317dda0700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f317815a620 con 0x7f317806dc20 2024-03-18T20:26:02.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.569+0000 7f317dda0700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31780cf0b0 con 0x7f317815a240 2024-03-18T20:26:02.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.569+0000 7f317dda0700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31780ce160 con 0x7f31780cec90 2024-03-18T20:26:02.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.569+0000 7f3177fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f317806dc20 0x7f31780ce580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47296/0 (socket says 172.21.15.1:47296) 2024-03-18T20:26:02.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.569+0000 7f3177fff700 1 -- 172.21.15.1:0/740452340 learned_addr learned my addr 172.21.15.1:0/740452340 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:02.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.569+0000 7f317d59f700 1 --2- 172.21.15.1:0/740452340 >> v2:172.21.15.165:3300/0 conn(0x7f31780cec90 0x7f3178165040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:02.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.569+0000 7f31777fe700 1 -- 172.21.15.1:0/740452340 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917524421 0 0) 0x7f317815a620 con 0x7f317806dc20 2024-03-18T20:26:02.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.569+0000 7f31777fe700 1 -- 172.21.15.1:0/740452340 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31680035b0 con 0x7f317806dc20 2024-03-18T20:26:02.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.569+0000 7f31777fe700 1 -- 172.21.15.1:0/740452340 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097296662 0 0) 0x7f31780cf0b0 con 0x7f317815a240 2024-03-18T20:26:02.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.569+0000 7f31777fe700 1 -- 172.21.15.1:0/740452340 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f317815a620 con 0x7f317815a240 2024-03-18T20:26:02.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.569+0000 7f317d59f700 1 -- 172.21.15.1:0/740452340 >> v1:172.21.15.165:6789/0 conn(0x7f317815a240 legacy=0x7f31780acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.569+0000 7f317d59f700 1 -- 172.21.15.1:0/740452340 >> v1:172.21.15.1:6789/0 conn(0x7f317806dc20 legacy=0x7f31780ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.570+0000 7f317d59f700 1 -- 172.21.15.1:0/740452340 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317806ee20 con 0x7f31780cec90 2024-03-18T20:26:02.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.570+0000 7f317d59f700 1 --2- 172.21.15.1:0/740452340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31780cec90 0x7f3178165040 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f3170009d90 tx=0x7f317000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c16e6cf36bbf97b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:02.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.570+0000 7f31777fe700 1 -- 172.21.15.1:0/740452340 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317000e070 con 0x7f31780cec90 2024-03-18T20:26:02.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.570+0000 7f31777fe700 1 -- 172.21.15.1:0/740452340 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317000bb20 con 0x7f31780cec90 2024-03-18T20:26:02.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.570+0000 7f31777fe700 1 -- 172.21.15.1:0/740452340 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3170012430 con 0x7f31780cec90 2024-03-18T20:26:02.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.570+0000 7f317dda0700 1 -- 172.21.15.1:0/740452340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31780cec90 msgr2=0x7f3178165040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.570+0000 7f317dda0700 1 --2- 172.21.15.1:0/740452340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31780cec90 0x7f3178165040 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f3170009d90 tx=0x7f317000b330 comp rx=0 tx=0).stop 2024-03-18T20:26:02.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.570+0000 7f317dda0700 1 -- 172.21.15.1:0/740452340 shutdown_connections 2024-03-18T20:26:02.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.571+0000 7f317dda0700 1 --2- 172.21.15.1:0/740452340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f31780cec90 0x7f3178165040 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.571+0000 7f317dda0700 1 -- 172.21.15.1:0/740452340 >> 172.21.15.1:0/740452340 conn(0x7f31780b19a0 msgr2=0x7f31780b1d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:02.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.571+0000 7f317dda0700 1 -- 172.21.15.1:0/740452340 shutdown_connections 2024-03-18T20:26:02.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.571+0000 7f317dda0700 1 -- 172.21.15.1:0/740452340 wait complete. 2024-03-18T20:26:02.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.572+0000 7f317dda0700 1 Processor -- start 2024-03-18T20:26:02.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.572+0000 7f317dda0700 1 -- start start 2024-03-18T20:26:02.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.573+0000 7f317dda0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f317806dc20 0x7f3178146c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.573+0000 7f317dda0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31780cec90 0x7f31780cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.573+0000 7f317dda0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f317815a240 0x7f31781482e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.573+0000 7f317dda0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31780c6260 con 0x7f31780cec90 2024-03-18T20:26:02.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.573+0000 7f317dda0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3178147f50 con 0x7f317815a240 2024-03-18T20:26:02.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.573+0000 7f317dda0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3178147340 con 0x7f317806dc20 2024-03-18T20:26:02.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.573+0000 7f3177fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31780cec90 0x7f31780cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:02.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.573+0000 7f3177fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31780cec90 0x7f31780cc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33794/0 (socket says 172.21.15.1:33794) 2024-03-18T20:26:02.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.573+0000 7f3177fff700 1 -- 172.21.15.1:0/911726316 learned_addr learned my addr 172.21.15.1:0/911726316 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:02.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.573+0000 7f317cd9e700 1 --2- 172.21.15.1:0/911726316 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f317806dc20 0x7f3178146c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:02.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.573+0000 7f317d59f700 1 --2- 172.21.15.1:0/911726316 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f317815a240 0x7f31781482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:02.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.573+0000 7f3177fff700 1 -- 172.21.15.1:0/911726316 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f317806dc20 msgr2=0x7f3178146c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.573+0000 7f3177fff700 1 --2- 172.21.15.1:0/911726316 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f317806dc20 0x7f3178146c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.573+0000 7f3177fff700 1 -- 172.21.15.1:0/911726316 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f317815a240 msgr2=0x7f31781482e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.573+0000 7f3177fff700 1 --2- 172.21.15.1:0/911726316 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f317815a240 0x7f31781482e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.573+0000 7f3177fff700 1 -- 172.21.15.1:0/911726316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3170009a70 con 0x7f31780cec90 2024-03-18T20:26:02.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.573+0000 7f317d59f700 1 --2- 172.21.15.1:0/911726316 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f317815a240 0x7f31781482e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:26:02.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.573+0000 7f3177fff700 1 --2- 172.21.15.1:0/911726316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31780cec90 0x7f31780cc1c0 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7f3164005910 tx=0x7f316400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:02.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.574+0000 7f31757fa700 1 -- 172.21.15.1:0/911726316 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3164005d70 con 0x7f31780cec90 2024-03-18T20:26:02.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.574+0000 7f31757fa700 1 -- 172.21.15.1:0/911726316 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f316400ce80 con 0x7f31780cec90 2024-03-18T20:26:02.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.574+0000 7f31757fa700 1 -- 172.21.15.1:0/911726316 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31640072d0 con 0x7f31780cec90 2024-03-18T20:26:02.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.574+0000 7f317dda0700 1 -- 172.21.15.1:0/911726316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3178147620 con 0x7f31780cec90 2024-03-18T20:26:02.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.574+0000 7f317dda0700 1 -- 172.21.15.1:0/911726316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3178147be0 con 0x7f31780cec90 2024-03-18T20:26:02.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.575+0000 7f3162ffd700 1 -- 172.21.15.1:0/911726316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3178148090 con 0x7f31780cec90 2024-03-18T20:26:02.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.575+0000 7f31757fa700 1 -- 172.21.15.1:0/911726316 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 142) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f3164007430 con 0x7f31780cec90 2024-03-18T20:26:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.576+0000 7f31757fa700 1 --2- 172.21.15.1:0/911726316 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f315c06af60 0x7f315c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.576+0000 7f317cd9e700 1 -- 172.21.15.1:0/911726316 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f315c06af60 msgr2=0x7f315c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.576+0000 7f31757fa700 1 -- 172.21.15.1:0/911726316 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7f3164089060 con 0x7f31780cec90 2024-03-18T20:26:02.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.576+0000 7f317cd9e700 1 --2- 172.21.15.1:0/911726316 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f315c06af60 0x7f315c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:02.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.580+0000 7f31757fa700 1 -- 172.21.15.1:0/911726316 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f31640bc580 con 0x7f31780cec90 2024-03-18T20:26:02.739 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:02.735+0000 7f47e8e57240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:26:02.777 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:02.773+0000 7f583fa9c240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:26:02.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.777+0000 7f317cd9e700 1 -- 172.21.15.1:0/911726316 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f315c06af60 msgr2=0x7f315c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:02.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.777+0000 7f317cd9e700 1 --2- 172.21.15.1:0/911726316 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f315c06af60 0x7f315c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.786+0000 7f3162ffd700 1 -- 172.21.15.1:0/911726316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f31780747a0 con 0x7f31780cec90 2024-03-18T20:26:02.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.787+0000 7f31757fa700 1 -- 172.21.15.1:0/911726316 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v237) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3164017020 con 0x7f31780cec90 2024-03-18T20:26:02.791 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:26:02.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.789+0000 7f3162ffd700 1 -- 172.21.15.1:0/911726316 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f315c06af60 msgr2=0x7f315c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:02.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.789+0000 7f3162ffd700 1 --2- 172.21.15.1:0/911726316 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f315c06af60 0x7f315c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.789+0000 7f3162ffd700 1 -- 172.21.15.1:0/911726316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31780cec90 msgr2=0x7f31780cc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.789+0000 7f3162ffd700 1 --2- 172.21.15.1:0/911726316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31780cec90 0x7f31780cc1c0 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7f3164005910 tx=0x7f316400c7b0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.789+0000 7f3162ffd700 1 -- 172.21.15.1:0/911726316 shutdown_connections 2024-03-18T20:26:02.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.789+0000 7f3162ffd700 1 --2- 172.21.15.1:0/911726316 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f315c06af60 0x7f315c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.789+0000 7f3162ffd700 1 --2- 172.21.15.1:0/911726316 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f317806dc20 0x7f3178146c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.789+0000 7f3162ffd700 1 --2- 172.21.15.1:0/911726316 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f317815a240 0x7f31781482e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.789+0000 7f3162ffd700 1 --2- 172.21.15.1:0/911726316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31780cec90 0x7f31780cc1c0 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.789+0000 7f3162ffd700 1 -- 172.21.15.1:0/911726316 >> 172.21.15.1:0/911726316 conn(0x7f31780b19a0 msgr2=0x7f31780b1d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:02.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.790+0000 7f3162ffd700 1 -- 172.21.15.1:0/911726316 shutdown_connections 2024-03-18T20:26:02.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.790+0000 7f3162ffd700 1 -- 172.21.15.1:0/911726316 wait complete. 2024-03-18T20:26:02.806 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:26:02.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.933+0000 7fd7aebef700 1 Processor -- start 2024-03-18T20:26:02.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.933+0000 7fd7aebef700 1 -- start start 2024-03-18T20:26:02.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.934+0000 7fd7aebef700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7a80b4810 0x7fd7a80d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.934+0000 7fd7aebef700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7a80cde70 0x7fd7a80ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.934+0000 7fd7aebef700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a80b3370 con 0x7fd7a80cfb50 2024-03-18T20:26:02.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.934+0000 7fd7aebef700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7a80b2760 con 0x7fd7a80b4810 2024-03-18T20:26:02.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.934+0000 7fd7aebef700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a80b2a40 con 0x7fd7a80cde70 2024-03-18T20:26:02.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.934+0000 7fd7a7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7a80b4810 0x7fd7a80d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.934+0000 7fd7ac98b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd7a80cfb50 0x7fd7a80cff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47300/0 (socket says 172.21.15.1:47300) 2024-03-18T20:26:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.934+0000 7fd7ac98b700 1 -- 172.21.15.1:0/1283788673 learned_addr learned my addr 172.21.15.1:0/1283788673 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.935+0000 7fd7ad18c700 1 --2- 172.21.15.1:0/1283788673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7a80cde70 0x7fd7a80ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.935+0000 7fd7a7fff700 1 -- 172.21.15.1:0/1283788673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7a80cde70 msgr2=0x7fd7a80ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.935+0000 7fd7a7fff700 1 --2- 172.21.15.1:0/1283788673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7a80cde70 0x7fd7a80ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.935+0000 7fd7a7fff700 1 -- 172.21.15.1:0/1283788673 >> v1:172.21.15.1:6789/0 conn(0x7fd7a80cfb50 legacy=0x7fd7a80cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.935+0000 7fd7a7fff700 1 -- 172.21.15.1:0/1283788673 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a80b2d20 con 0x7fd7a80b4810 2024-03-18T20:26:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.935+0000 7fd7ad18c700 1 --2- 172.21.15.1:0/1283788673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7a80cde70 0x7fd7a80ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.935+0000 7fd7a7fff700 1 --2- 172.21.15.1:0/1283788673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7a80b4810 0x7fd7a80d3c60 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7fd794009840 tx=0x7fd794009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae358c51c24b03f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.935+0000 7fd7a77fe700 1 -- 172.21.15.1:0/1283788673 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd794011830 con 0x7fd7a80b4810 2024-03-18T20:26:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.935+0000 7fd7a77fe700 1 -- 172.21.15.1:0/1283788673 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd794011990 con 0x7fd7a80b4810 2024-03-18T20:26:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.935+0000 7fd7a77fe700 1 -- 172.21.15.1:0/1283788673 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd794011d20 con 0x7fd7a80b4810 2024-03-18T20:26:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.936+0000 7fd7aebef700 1 -- 172.21.15.1:0/1283788673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7a80b4810 msgr2=0x7fd7a80d3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.936+0000 7fd7aebef700 1 --2- 172.21.15.1:0/1283788673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7a80b4810 0x7fd7a80d3c60 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7fd794009840 tx=0x7fd794009b50 comp rx=0 tx=0).stop 2024-03-18T20:26:02.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.936+0000 7fd7aebef700 1 -- 172.21.15.1:0/1283788673 shutdown_connections 2024-03-18T20:26:02.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.936+0000 7fd7aebef700 1 --2- 172.21.15.1:0/1283788673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7a80cde70 0x7fd7a80ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.936+0000 7fd7aebef700 1 --2- 172.21.15.1:0/1283788673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7a80b4810 0x7fd7a80d3c60 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.936+0000 7fd7aebef700 1 -- 172.21.15.1:0/1283788673 >> 172.21.15.1:0/1283788673 conn(0x7fd7a80d3140 msgr2=0x7fd7a80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:02.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.936+0000 7fd7aebef700 1 -- 172.21.15.1:0/1283788673 shutdown_connections 2024-03-18T20:26:02.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.936+0000 7fd7aebef700 1 -- 172.21.15.1:0/1283788673 wait complete. 2024-03-18T20:26:02.942 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:02.938+0000 7f583fa9c240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:26:02.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.937+0000 7fd7aebef700 1 Processor -- start 2024-03-18T20:26:02.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.937+0000 7fd7aebef700 1 -- start start 2024-03-18T20:26:02.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.938+0000 7fd7aebef700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7a80b4810 0x7fd7a80b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.938+0000 7fd7aebef700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7a80cde70 0x7fd7a80b5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.938+0000 7fd7aebef700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7a80cfb50 0x7fd7a80c8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.938+0000 7fd7aebef700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a8157c70 con 0x7fd7a80cfb50 2024-03-18T20:26:02.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.938+0000 7fd7aebef700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a80c7550 con 0x7fd7a80b4810 2024-03-18T20:26:02.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.938+0000 7fd7aebef700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a80c7830 con 0x7fd7a80cde70 2024-03-18T20:26:02.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.938+0000 7fd7ad18c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7a80cfb50 0x7fd7a80c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:02.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.938+0000 7fd7ad18c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7a80cfb50 0x7fd7a80c8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33814/0 (socket says 172.21.15.1:33814) 2024-03-18T20:26:02.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.938+0000 7fd7ad18c700 1 -- 172.21.15.1:0/2918145149 learned_addr learned my addr 172.21.15.1:0/2918145149 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:02.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.938+0000 7fd7a7fff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7a80cde70 0x7fd7a80b5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:02.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.938+0000 7fd7ac98b700 1 --2- 172.21.15.1:0/2918145149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7a80b4810 0x7fd7a80b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:02.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.938+0000 7fd7ad18c700 1 -- 172.21.15.1:0/2918145149 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7a80cde70 msgr2=0x7fd7a80b5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.938+0000 7fd7ad18c700 1 --2- 172.21.15.1:0/2918145149 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7a80cde70 0x7fd7a80b5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.938+0000 7fd7ad18c700 1 -- 172.21.15.1:0/2918145149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7a80b4810 msgr2=0x7fd7a80b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:02.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.938+0000 7fd7ad18c700 1 --2- 172.21.15.1:0/2918145149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7a80b4810 0x7fd7a80b1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:02.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.938+0000 7fd7ad18c700 1 -- 172.21.15.1:0/2918145149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7940094a0 con 0x7fd7a80cfb50 2024-03-18T20:26:02.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.938+0000 7fd7ad18c700 1 --2- 172.21.15.1:0/2918145149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7a80cfb50 0x7fd7a80c8310 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0007ef0 tx=0x7fd7a000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:02.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.939+0000 7fd7a57fa700 1 -- 172.21.15.1:0/2918145149 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a000dd80 con 0x7fd7a80cfb50 2024-03-18T20:26:02.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.939+0000 7fd7a57fa700 1 -- 172.21.15.1:0/2918145149 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a000f040 con 0x7fd7a80cfb50 2024-03-18T20:26:02.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.939+0000 7fd7a57fa700 1 -- 172.21.15.1:0/2918145149 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a0003c60 con 0x7fd7a80cfb50 2024-03-18T20:26:02.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.939+0000 7fd7aebef700 1 -- 172.21.15.1:0/2918145149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a80c7b10 con 0x7fd7a80cfb50 2024-03-18T20:26:02.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.939+0000 7fd7aebef700 1 -- 172.21.15.1:0/2918145149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a80b6230 con 0x7fd7a80cfb50 2024-03-18T20:26:02.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.940+0000 7fd7aebef700 1 -- 172.21.15.1:0/2918145149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a80c8030 con 0x7fd7a80cfb50 2024-03-18T20:26:02.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.940+0000 7fd7a57fa700 1 -- 172.21.15.1:0/2918145149 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 142) v1 ==== 89398+0+0 (secure 0 0 0) 0x7fd7a0004030 con 0x7fd7a80cfb50 2024-03-18T20:26:02.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.941+0000 7fd7a57fa700 1 --2- 172.21.15.1:0/2918145149 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd78c06af60 0x7fd78c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:02.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.941+0000 7fd7ac98b700 1 -- 172.21.15.1:0/2918145149 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd78c06af60 msgr2=0x7fd78c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:02.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.941+0000 7fd7a57fa700 1 -- 172.21.15.1:0/2918145149 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7fd7a0089ee0 con 0x7fd7a80cfb50 2024-03-18T20:26:02.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.941+0000 7fd7ac98b700 1 --2- 172.21.15.1:0/2918145149 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd78c06af60 0x7fd78c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:02.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:02.945+0000 7fd7a57fa700 1 -- 172.21.15.1:0/2918145149 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd7a80c8030 con 0x7fd7a80cfb50 2024-03-18T20:26:03.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.142+0000 7fd7ac98b700 1 -- 172.21.15.1:0/2918145149 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd78c06af60 msgr2=0x7fd78c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:03.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.142+0000 7fd7ac98b700 1 --2- 172.21.15.1:0/2918145149 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd78c06af60 0x7fd78c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:03.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.149+0000 7fd7aebef700 1 -- 172.21.15.1:0/2918145149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd7a80c8030 con 0x7fd7a80cfb50 2024-03-18T20:26:03.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.149+0000 7fd7a57fa700 1 -- 172.21.15.1:0/2918145149 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v237) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd7a80c8030 con 0x7fd7a80cfb50 2024-03-18T20:26:03.153 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:26:03.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.152+0000 7fd7aebef700 1 -- 172.21.15.1:0/2918145149 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd78c06af60 msgr2=0x7fd78c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:03.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.152+0000 7fd7aebef700 1 --2- 172.21.15.1:0/2918145149 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd78c06af60 0x7fd78c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.152+0000 7fd7aebef700 1 -- 172.21.15.1:0/2918145149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7a80cfb50 msgr2=0x7fd7a80c8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:03.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.152+0000 7fd7aebef700 1 --2- 172.21.15.1:0/2918145149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7a80cfb50 0x7fd7a80c8310 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0007ef0 tx=0x7fd7a000d3b0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.152+0000 7fd7aebef700 1 -- 172.21.15.1:0/2918145149 shutdown_connections 2024-03-18T20:26:03.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.152+0000 7fd7aebef700 1 --2- 172.21.15.1:0/2918145149 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd78c06af60 0x7fd78c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.152+0000 7fd7aebef700 1 --2- 172.21.15.1:0/2918145149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7a80cfb50 0x7fd7a80c8310 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.152+0000 7fd7aebef700 1 --2- 172.21.15.1:0/2918145149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7a80b4810 0x7fd7a80b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.152+0000 7fd7aebef700 1 --2- 172.21.15.1:0/2918145149 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd7a80cde70 0x7fd7a80b5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.152+0000 7fd7aebef700 1 -- 172.21.15.1:0/2918145149 >> 172.21.15.1:0/2918145149 conn(0x7fd7a80d3140 msgr2=0x7fd7a80d3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:03.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.152+0000 7fd7aebef700 1 -- 172.21.15.1:0/2918145149 shutdown_connections 2024-03-18T20:26:03.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.152+0000 7fd7aebef700 1 -- 172.21.15.1:0/2918145149 wait complete. 2024-03-18T20:26:03.169 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-03-18T20:26:03.172 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.168+0000 7f47e8e57240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:26:03.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.294+0000 7f4abd9c6700 1 Processor -- start 2024-03-18T20:26:03.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.294+0000 7f4abd9c6700 1 -- start start 2024-03-18T20:26:03.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.295+0000 7f4abd9c6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4ab80b4810 0x7f4ab80d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:03.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.295+0000 7f4abd9c6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ab80cde70 0x7f4ab80ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:03.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.295+0000 7f4abd9c6700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab80b3370 con 0x7f4ab80cfb50 2024-03-18T20:26:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.295+0000 7f4abd9c6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ab80b2760 con 0x7f4ab80b4810 2024-03-18T20:26:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.295+0000 7f4abd9c6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab80b2a40 con 0x7f4ab80cde70 2024-03-18T20:26:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.295+0000 7f4ab67fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4ab80b4810 0x7f4ab80d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.296+0000 7f4ab67fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4ab80b4810 0x7f4ab80d3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33826/0 (socket says 172.21.15.1:33826) 2024-03-18T20:26:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.296+0000 7f4ab67fc700 1 -- 172.21.15.1:0/1548166610 learned_addr learned my addr 172.21.15.1:0/1548166610 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.296+0000 7f4ab77fe700 1 --2- 172.21.15.1:0/1548166610 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ab80cde70 0x7f4ab80ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.296+0000 7f4ab67fc700 1 -- 172.21.15.1:0/1548166610 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ab80cde70 msgr2=0x7f4ab80ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.296+0000 7f4ab67fc700 1 --2- 172.21.15.1:0/1548166610 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ab80cde70 0x7f4ab80ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.296+0000 7f4ab67fc700 1 -- 172.21.15.1:0/1548166610 >> v1:172.21.15.165:6789/0 conn(0x7f4ab80cfb50 legacy=0x7f4ab80cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.296+0000 7f4ab67fc700 1 -- 172.21.15.1:0/1548166610 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab80b2d20 con 0x7f4ab80b4810 2024-03-18T20:26:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.296+0000 7f4ab5ffb700 1 -- 172.21.15.1:0/1548166610 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090157675 0 0) 0x7f4ab80b3370 con 0x7f4ab80cfb50 2024-03-18T20:26:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.296+0000 7f4ab67fc700 1 --2- 172.21.15.1:0/1548166610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab80b4810 0x7f4ab80d3c60 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7f4aac0098f0 tx=0x7f4aac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d46e4c6ad9004252 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.296+0000 7f4ab5ffb700 1 -- 172.21.15.1:0/1548166610 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac011830 con 0x7f4ab80b4810 2024-03-18T20:26:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.296+0000 7f4ab5ffb700 1 -- 172.21.15.1:0/1548166610 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aac011990 con 0x7f4ab80b4810 2024-03-18T20:26:03.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.296+0000 7f4ab5ffb700 1 -- 172.21.15.1:0/1548166610 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac011d20 con 0x7f4ab80b4810 2024-03-18T20:26:03.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.297+0000 7f4abd9c6700 1 -- 172.21.15.1:0/1548166610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab80b4810 msgr2=0x7f4ab80d3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:03.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.297+0000 7f4abd9c6700 1 --2- 172.21.15.1:0/1548166610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab80b4810 0x7f4ab80d3c60 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7f4aac0098f0 tx=0x7f4aac009c00 comp rx=0 tx=0).stop 2024-03-18T20:26:03.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.297+0000 7f4abd9c6700 1 -- 172.21.15.1:0/1548166610 shutdown_connections 2024-03-18T20:26:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.297+0000 7f4abd9c6700 1 --2- 172.21.15.1:0/1548166610 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ab80cde70 0x7f4ab80ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.297+0000 7f4abd9c6700 1 --2- 172.21.15.1:0/1548166610 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab80b4810 0x7f4ab80d3c60 unknown :-1 s=CLOSED pgs=2678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.297+0000 7f4abd9c6700 1 -- 172.21.15.1:0/1548166610 >> 172.21.15.1:0/1548166610 conn(0x7f4ab80d3140 msgr2=0x7f4ab80d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.297+0000 7f4abd9c6700 1 -- 172.21.15.1:0/1548166610 shutdown_connections 2024-03-18T20:26:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.297+0000 7f4abd9c6700 1 -- 172.21.15.1:0/1548166610 wait complete. 2024-03-18T20:26:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.298+0000 7f4abd9c6700 1 Processor -- start 2024-03-18T20:26:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.298+0000 7f4abd9c6700 1 -- start start 2024-03-18T20:26:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.299+0000 7f4abd9c6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ab80b4810 0x7f4ab815a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.299+0000 7f4abd9c6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ab80cde70 0x7f4ab80b85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.299+0000 7f4abd9c6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab80cfb50 0x7f4ab80b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:03.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.299+0000 7f4abd9c6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab80c6b40 con 0x7f4ab80cfb50 2024-03-18T20:26:03.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.299+0000 7f4abd9c6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab80c6e20 con 0x7f4ab80b4810 2024-03-18T20:26:03.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.299+0000 7f4abd9c6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab80c7100 con 0x7f4ab80cde70 2024-03-18T20:26:03.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.299+0000 7f4ab77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab80cfb50 0x7f4ab80b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:03.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.299+0000 7f4ab6ffd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ab80b4810 0x7f4ab815a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:03.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.299+0000 7f4ab77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab80cfb50 0x7f4ab80b8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33832/0 (socket says 172.21.15.1:33832) 2024-03-18T20:26:03.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.299+0000 7f4ab77fe700 1 -- 172.21.15.1:0/1250195918 learned_addr learned my addr 172.21.15.1:0/1250195918 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:03.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.299+0000 7f4ab67fc700 1 --2- 172.21.15.1:0/1250195918 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ab80cde70 0x7f4ab80b85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:03.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.300+0000 7f4ab77fe700 1 -- 172.21.15.1:0/1250195918 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ab80cde70 msgr2=0x7f4ab80b85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:03.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.300+0000 7f4ab77fe700 1 --2- 172.21.15.1:0/1250195918 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ab80cde70 0x7f4ab80b85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.300+0000 7f4ab77fe700 1 -- 172.21.15.1:0/1250195918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ab80b4810 msgr2=0x7f4ab815a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:03.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.300+0000 7f4ab77fe700 1 --2- 172.21.15.1:0/1250195918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ab80b4810 0x7f4ab815a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.300+0000 7f4ab77fe700 1 -- 172.21.15.1:0/1250195918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac009550 con 0x7f4ab80cfb50 2024-03-18T20:26:03.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.300+0000 7f4ab6ffd700 1 --2- 172.21.15.1:0/1250195918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ab80b4810 0x7f4ab815a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:26:03.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.300+0000 7f4ab77fe700 1 --2- 172.21.15.1:0/1250195918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab80cfb50 0x7f4ab80b8b20 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7f4ab000efc0 tx=0x7f4ab000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:03.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.300+0000 7f4a9ffff700 1 -- 172.21.15.1:0/1250195918 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0009ea0 con 0x7f4ab80cfb50 2024-03-18T20:26:03.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.300+0000 7f4a9ffff700 1 -- 172.21.15.1:0/1250195918 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab000f040 con 0x7f4ab80cfb50 2024-03-18T20:26:03.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.300+0000 7f4abd9c6700 1 -- 172.21.15.1:0/1250195918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab80c73e0 con 0x7f4ab80cfb50 2024-03-18T20:26:03.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.300+0000 7f4a9ffff700 1 -- 172.21.15.1:0/1250195918 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab000ced0 con 0x7f4ab80cfb50 2024-03-18T20:26:03.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.300+0000 7f4abd9c6700 1 -- 172.21.15.1:0/1250195918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab80b5cf0 con 0x7f4ab80cfb50 2024-03-18T20:26:03.307 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:03.303+0000 7f583fa9c240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:26:03.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.301+0000 7f4abd9c6700 1 -- 172.21.15.1:0/1250195918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab80c7900 con 0x7f4ab80cfb50 2024-03-18T20:26:03.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.302+0000 7f4a9ffff700 1 -- 172.21.15.1:0/1250195918 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 142) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f4ab0014480 con 0x7f4ab80cfb50 2024-03-18T20:26:03.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.302+0000 7f4a9ffff700 1 --2- 172.21.15.1:0/1250195918 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4aa406af60 0x7f4aa406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:03.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.302+0000 7f4ab6ffd700 1 -- 172.21.15.1:0/1250195918 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4aa406af60 msgr2=0x7f4aa406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:03.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.302+0000 7f4a9ffff700 1 -- 172.21.15.1:0/1250195918 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7f4ab008a6d0 con 0x7f4ab80cfb50 2024-03-18T20:26:03.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.302+0000 7f4ab6ffd700 1 --2- 172.21.15.1:0/1250195918 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4aa406af60 0x7f4aa406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:03.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.307+0000 7f4a9ffff700 1 -- 172.21.15.1:0/1250195918 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4ab80c7900 con 0x7f4ab80cfb50 2024-03-18T20:26:03.356 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.352+0000 7f47e8e57240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:26:03.395 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:03.391+0000 7f583fa9c240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:26:03.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.503+0000 7f4ab6ffd700 1 -- 172.21.15.1:0/1250195918 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4aa406af60 msgr2=0x7f4aa406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:03.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.503+0000 7f4ab6ffd700 1 --2- 172.21.15.1:0/1250195918 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4aa406af60 0x7f4aa406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.511+0000 7f4abd9c6700 1 -- 172.21.15.1:0/1250195918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4ab80c7900 con 0x7f4ab80cfb50 2024-03-18T20:26:03.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.511+0000 7f4a9ffff700 1 -- 172.21.15.1:0/1250195918 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v237) v1 ==== 107+0+11 (secure 0 0 0) 0x7f4ab80c7900 con 0x7f4ab80cfb50 2024-03-18T20:26:03.515 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:26:03.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.513+0000 7f4abd9c6700 1 -- 172.21.15.1:0/1250195918 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4aa406af60 msgr2=0x7f4aa406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:03.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.513+0000 7f4abd9c6700 1 --2- 172.21.15.1:0/1250195918 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4aa406af60 0x7f4aa406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.514+0000 7f4abd9c6700 1 -- 172.21.15.1:0/1250195918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab80cfb50 msgr2=0x7f4ab80b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:03.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.514+0000 7f4abd9c6700 1 --2- 172.21.15.1:0/1250195918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab80cfb50 0x7f4ab80b8b20 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7f4ab000efc0 tx=0x7f4ab000c3a0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.514+0000 7f4abd9c6700 1 -- 172.21.15.1:0/1250195918 shutdown_connections 2024-03-18T20:26:03.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.514+0000 7f4abd9c6700 1 --2- 172.21.15.1:0/1250195918 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4aa406af60 0x7f4aa406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.514+0000 7f4abd9c6700 1 --2- 172.21.15.1:0/1250195918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab80cfb50 0x7f4ab80b8b20 unknown :-1 s=CLOSED pgs=2679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.514+0000 7f4abd9c6700 1 --2- 172.21.15.1:0/1250195918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ab80b4810 0x7f4ab815a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.514+0000 7f4abd9c6700 1 --2- 172.21.15.1:0/1250195918 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4ab80cde70 0x7f4ab80b85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.514+0000 7f4abd9c6700 1 -- 172.21.15.1:0/1250195918 >> 172.21.15.1:0/1250195918 conn(0x7f4ab80d3140 msgr2=0x7f4ab806ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:03.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.514+0000 7f4abd9c6700 1 -- 172.21.15.1:0/1250195918 shutdown_connections 2024-03-18T20:26:03.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.514+0000 7f4abd9c6700 1 -- 172.21.15.1:0/1250195918 wait complete. 2024-03-18T20:26:03.532 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-03-18T20:26:03.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.701+0000 7f005d7ae700 1 Processor -- start 2024-03-18T20:26:03.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.701+0000 7f005d7ae700 1 -- start start 2024-03-18T20:26:03.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.702+0000 7f005d7ae700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f005815d580 0x7f005806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:03.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.702+0000 7f005d7ae700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00580724b0 0x7f00580cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:03.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.702+0000 7f005d7ae700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005806f4f0 con 0x7f005806d380 2024-03-18T20:26:03.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.702+0000 7f005d7ae700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f005815d960 con 0x7f005815d580 2024-03-18T20:26:03.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.702+0000 7f005d7ae700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005806ab30 con 0x7f00580724b0 2024-03-18T20:26:03.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.702+0000 7f00567fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f005806d380 0x7f0058071da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47304/0 (socket says 172.21.15.1:47304) 2024-03-18T20:26:03.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.702+0000 7f00567fc700 1 -- 172.21.15.1:0/1608948667 learned_addr learned my addr 172.21.15.1:0/1608948667 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:03.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f0055ffb700 1 -- 172.21.15.1:0/1608948667 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794091895 0 0) 0x7f004c001110 con 0x7f005806d380 2024-03-18T20:26:03.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f0055ffb700 1 -- 172.21.15.1:0/1608948667 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00400035b0 con 0x7f005806d380 2024-03-18T20:26:03.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f0055ffb700 1 -- 172.21.15.1:0/1608948667 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2336216266 0 0) 0x7f00400035b0 con 0x7f005806d380 2024-03-18T20:26:03.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f0055ffb700 1 -- 172.21.15.1:0/1608948667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00580724b0 msgr2=0x7f00580cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:03.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f0055ffb700 1 --2- 172.21.15.1:0/1608948667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00580724b0 0x7f00580cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f0055ffb700 1 -- 172.21.15.1:0/1608948667 >> v2:172.21.15.165:3300/0 conn(0x7f005815d580 msgr2=0x7f005806efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:03.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f0055ffb700 1 --2- 172.21.15.1:0/1608948667 >> v2:172.21.15.165:3300/0 conn(0x7f005815d580 0x7f005806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f0055ffb700 1 -- 172.21.15.1:0/1608948667 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005806ae10 con 0x7f005806d380 2024-03-18T20:26:03.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f0055ffb700 1 -- 172.21.15.1:0/1608948667 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f004c001800 con 0x7f005806d380 2024-03-18T20:26:03.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f0055ffb700 1 -- 172.21.15.1:0/1608948667 >> v1:172.21.15.1:6789/0 conn(0x7f005806d380 legacy=0x7f0058071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:03.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f0055ffb700 1 --2- 172.21.15.1:0/1608948667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00580724b0 0x7f0040003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:03.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f0055ffb700 1 --2- 172.21.15.1:0/1608948667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0040004110 0x7f0040004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:03.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f0055ffb700 1 --2- 172.21.15.1:0/1608948667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0040004de0 0x7f0040005260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:03.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f0055ffb700 1 -- 172.21.15.1:0/1608948667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005815d960 con 0x7f0040004110 2024-03-18T20:26:03.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f0055ffb700 1 -- 172.21.15.1:0/1608948667 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005806ab30 con 0x7f0040004de0 2024-03-18T20:26:03.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f0055ffb700 1 -- 172.21.15.1:0/1608948667 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f004000bcd0 con 0x7f00580724b0 2024-03-18T20:26:03.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f00567fc700 1 --2- 172.21.15.1:0/1608948667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0040004110 0x7f0040004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:03.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f00577fe700 1 --2- 172.21.15.1:0/1608948667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0040004de0 0x7f0040005260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:03.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f0056ffd700 1 --2- 172.21.15.1:0/1608948667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00580724b0 0x7f0040003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:03.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f00567fc700 1 -- 172.21.15.1:0/1608948667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00580724b0 msgr2=0x7f0040003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:03.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f00567fc700 1 --2- 172.21.15.1:0/1608948667 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00580724b0 0x7f0040003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f00567fc700 1 -- 172.21.15.1:0/1608948667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0040004de0 msgr2=0x7f0040005260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:03.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f00567fc700 1 --2- 172.21.15.1:0/1608948667 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0040004de0 0x7f0040005260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.703+0000 7f00567fc700 1 -- 172.21.15.1:0/1608948667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00400035b0 con 0x7f0040004110 2024-03-18T20:26:03.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.704+0000 7f00567fc700 1 --2- 172.21.15.1:0/1608948667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0040004110 0x7f0040004850 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7f004c00c2f0 tx=0x7f0040005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4da218f2656f00 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:03.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.704+0000 7f0055ffb700 1 -- 172.21.15.1:0/1608948667 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c00ce30 con 0x7f0040004110 2024-03-18T20:26:03.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.704+0000 7f0055ffb700 1 -- 172.21.15.1:0/1608948667 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f004c005960 con 0x7f0040004110 2024-03-18T20:26:03.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.704+0000 7f0055ffb700 1 -- 172.21.15.1:0/1608948667 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004c005d10 con 0x7f0040004110 2024-03-18T20:26:03.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.704+0000 7f005d7ae700 1 -- 172.21.15.1:0/1608948667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0040004110 msgr2=0x7f0040004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:03.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.704+0000 7f005d7ae700 1 --2- 172.21.15.1:0/1608948667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0040004110 0x7f0040004850 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7f004c00c2f0 tx=0x7f0040005880 comp rx=0 tx=0).stop 2024-03-18T20:26:03.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.704+0000 7f00577fe700 1 -- 172.21.15.1:0/1608948667 reap_dead start 2024-03-18T20:26:03.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.704+0000 7f005d7ae700 1 -- 172.21.15.1:0/1608948667 shutdown_connections 2024-03-18T20:26:03.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.704+0000 7f005d7ae700 1 -- 172.21.15.1:0/1608948667 >> 172.21.15.1:0/1608948667 conn(0x7f00580cca00 msgr2=0x7f00580cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:03.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.704+0000 7f005d7ae700 1 -- 172.21.15.1:0/1608948667 shutdown_connections 2024-03-18T20:26:03.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.704+0000 7f005d7ae700 1 -- 172.21.15.1:0/1608948667 wait complete. 2024-03-18T20:26:03.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.706+0000 7f005d7ae700 1 Processor -- start 2024-03-18T20:26:03.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.706+0000 7f005d7ae700 1 -- start start 2024-03-18T20:26:03.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f005d7ae700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0040004110 0x7f00581f2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:03.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f005d7ae700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f005815ec00 0x7f005815f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f005d7ae700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00580b4070 0x7f00580b4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f005d7ae700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00580c49d0 con 0x7f0040004110 2024-03-18T20:26:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f005d7ae700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00580b5a90 con 0x7f00580b4070 2024-03-18T20:26:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f005d7ae700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00580c3d50 con 0x7f005815ec00 2024-03-18T20:26:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f0056ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0040004110 0x7f00581f2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f0056ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0040004110 0x7f00581f2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33852/0 (socket says 172.21.15.1:33852) 2024-03-18T20:26:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f0056ffd700 1 -- 172.21.15.1:0/3413907145 learned_addr learned my addr 172.21.15.1:0/3413907145 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f00577fe700 1 --2- 172.21.15.1:0/3413907145 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00580b4070 0x7f00580b4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f00567fc700 1 --2- 172.21.15.1:0/3413907145 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f005815ec00 0x7f005815f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f0056ffd700 1 -- 172.21.15.1:0/3413907145 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f005815ec00 msgr2=0x7f005815f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:03.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f0056ffd700 1 --2- 172.21.15.1:0/3413907145 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f005815ec00 0x7f005815f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f0056ffd700 1 -- 172.21.15.1:0/3413907145 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00580b4070 msgr2=0x7f00580b4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:03.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f0056ffd700 1 --2- 172.21.15.1:0/3413907145 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00580b4070 0x7f00580b4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f0056ffd700 1 -- 172.21.15.1:0/3413907145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00580c4030 con 0x7f0040004110 2024-03-18T20:26:03.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f00577fe700 1 --2- 172.21.15.1:0/3413907145 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00580b4070 0x7f00580b4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:03.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f00567fc700 1 --2- 172.21.15.1:0/3413907145 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f005815ec00 0x7f005815f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:03.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f0056ffd700 1 --2- 172.21.15.1:0/3413907145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0040004110 0x7f00581f2fc0 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7f004800e850 tx=0x7f004800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:03.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f003f7fe700 1 -- 172.21.15.1:0/3413907145 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00480095b0 con 0x7f0040004110 2024-03-18T20:26:03.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f003f7fe700 1 -- 172.21.15.1:0/3413907145 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f004800ee50 con 0x7f0040004110 2024-03-18T20:26:03.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f005d7ae700 1 -- 172.21.15.1:0/3413907145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00580c4310 con 0x7f0040004110 2024-03-18T20:26:03.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f003f7fe700 1 -- 172.21.15.1:0/3413907145 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004800ca90 con 0x7f0040004110 2024-03-18T20:26:03.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.707+0000 7f005d7ae700 1 -- 172.21.15.1:0/3413907145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00580b4a60 con 0x7f0040004110 2024-03-18T20:26:03.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.709+0000 7f005d7ae700 1 -- 172.21.15.1:0/3413907145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00580676a0 con 0x7f0040004110 2024-03-18T20:26:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.709+0000 7f003f7fe700 1 -- 172.21.15.1:0/3413907145 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 142) v1 ==== 89398+0+0 (secure 0 0 0) 0x7f004800cbf0 con 0x7f0040004110 2024-03-18T20:26:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.710+0000 7f003f7fe700 1 --2- 172.21.15.1:0/3413907145 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f004406af60 0x7f004406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.710+0000 7f00567fc700 1 -- 172.21.15.1:0/3413907145 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f004406af60 msgr2=0x7f004406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.710+0000 7f003f7fe700 1 -- 172.21.15.1:0/3413907145 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7337+0+0 (secure 0 0 0) 0x7f0048089d70 con 0x7f0040004110 2024-03-18T20:26:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.710+0000 7f00567fc700 1 --2- 172.21.15.1:0/3413907145 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f004406af60 0x7f004406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.715+0000 7f003f7fe700 1 -- 172.21.15.1:0/3413907145 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f00480545e0 con 0x7f0040004110 2024-03-18T20:26:03.744 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.740+0000 7f47e8e57240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:26:03.840 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.836+0000 7f47e8e57240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:26:03.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.855+0000 7f003f7fe700 1 -- 172.21.15.1:0/3413907145 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 143) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f00480549b0 con 0x7f0040004110 2024-03-18T20:26:03.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.855+0000 7f003f7fe700 1 -- 172.21.15.1:0/3413907145 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f004406af60 msgr2=0x7f004406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:03.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.855+0000 7f003f7fe700 1 --2- 172.21.15.1:0/3413907145 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f004406af60 0x7f004406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.926+0000 7f005d7ae700 1 -- 172.21.15.1:0/3413907145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f00580b1880 con 0x7f0040004110 2024-03-18T20:26:03.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.926+0000 7f003f7fe700 1 -- 172.21.15.1:0/3413907145 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v238) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0048086ae0 con 0x7f0040004110 2024-03-18T20:26:03.931 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:26:03.931 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.927+0000 7f476127f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.931 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.927+0000 7f476127f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.931 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.927+0000 7f476127f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.931 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.927+0000 7f476127f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.932 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.927+0000 7f476127f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.929+0000 7f005d7ae700 1 -- 172.21.15.1:0/3413907145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0040004110 msgr2=0x7f00581f2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:03.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.929+0000 7f005d7ae700 1 --2- 172.21.15.1:0/3413907145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0040004110 0x7f00581f2fc0 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7f004800e850 tx=0x7f004800ec10 comp rx=0 tx=0).stop 2024-03-18T20:26:03.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.929+0000 7f005d7ae700 1 -- 172.21.15.1:0/3413907145 shutdown_connections 2024-03-18T20:26:03.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.929+0000 7f005d7ae700 1 --2- 172.21.15.1:0/3413907145 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f004406af60 0x7f004406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.929+0000 7f005d7ae700 1 --2- 172.21.15.1:0/3413907145 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00580b4070 0x7f00580b4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.929+0000 7f005d7ae700 1 --2- 172.21.15.1:0/3413907145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0040004110 0x7f00581f2fc0 unknown :-1 s=CLOSED pgs=2682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.929+0000 7f005d7ae700 1 --2- 172.21.15.1:0/3413907145 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f005815ec00 0x7f005815f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:03.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.929+0000 7f005d7ae700 1 -- 172.21.15.1:0/3413907145 >> 172.21.15.1:0/3413907145 conn(0x7f00580cca00 msgr2=0x7f00580ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:03.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.929+0000 7f005d7ae700 1 -- 172.21.15.1:0/3413907145 shutdown_connections 2024-03-18T20:26:03.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:03.929+0000 7f005d7ae700 1 -- 172.21.15.1:0/3413907145 wait complete. 2024-03-18T20:26:03.936 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.930+0000 7f4764285700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.936 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.930+0000 7f4764285700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.937 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.930+0000 7f4764285700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.937 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.930+0000 7f4764285700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.937 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.930+0000 7f4764285700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.942 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.938+0000 7f4764285700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.942 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.938+0000 7f4764285700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.942 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.938+0000 7f4764285700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.943 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.938+0000 7f4764285700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.943 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.938+0000 7f4764285700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.944 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.941+0000 7f476127f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.945 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.941+0000 7f476127f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.945 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.941+0000 7f476127f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.945 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.941+0000 7f476127f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.945 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:03.941+0000 7f476127f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:03.949 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:26:04.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.070+0000 7fd9c041f700 1 Processor -- start 2024-03-18T20:26:04.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.070+0000 7fd9c041f700 1 -- start start 2024-03-18T20:26:04.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.071+0000 7fd9c041f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd9b815d580 0x7fd9b8072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.071+0000 7fd9c041f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9b806dd50 0x7fd9b80cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.071+0000 7fd9c041f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b806f490 con 0x7fd9b806e290 2024-03-18T20:26:04.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.071+0000 7fd9c041f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9b8071cf0 con 0x7fd9b815d580 2024-03-18T20:26:04.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.071+0000 7fd9c041f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b806f1b0 con 0x7fd9b806dd50 2024-03-18T20:26:04.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.071+0000 7fd9bd9ba700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd9b806e290 0x7fd9b806d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47318/0 (socket says 172.21.15.1:47318) 2024-03-18T20:26:04.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.071+0000 7fd9bd9ba700 1 -- 172.21.15.1:0/4144948664 learned_addr learned my addr 172.21.15.1:0/4144948664 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:04.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.071+0000 7fd9be1bb700 1 --2- 172.21.15.1:0/4144948664 >> v2:172.21.15.165:3300/0 conn(0x7fd9b815d580 0x7fd9b8072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.071+0000 7fd9be9bc700 1 --2- 172.21.15.1:0/4144948664 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9b806dd50 0x7fd9b80cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.071+0000 7fd9bd1b9700 1 -- 172.21.15.1:0/4144948664 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482182951 0 0) 0x7fd9b806f490 con 0x7fd9b806e290 2024-03-18T20:26:04.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.071+0000 7fd9bd1b9700 1 -- 172.21.15.1:0/4144948664 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9a40035b0 con 0x7fd9b806e290 2024-03-18T20:26:04.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.071+0000 7fd9be1bb700 1 -- 172.21.15.1:0/4144948664 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9b806dd50 msgr2=0x7fd9b80cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:04.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.071+0000 7fd9be1bb700 1 --2- 172.21.15.1:0/4144948664 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9b806dd50 0x7fd9b80cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.071+0000 7fd9be1bb700 1 -- 172.21.15.1:0/4144948664 >> v1:172.21.15.1:6789/0 conn(0x7fd9b806e290 legacy=0x7fd9b806d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:04.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.071+0000 7fd9be1bb700 1 -- 172.21.15.1:0/4144948664 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b80716a0 con 0x7fd9b815d580 2024-03-18T20:26:04.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.072+0000 7fd9be1bb700 1 --2- 172.21.15.1:0/4144948664 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9b815d580 0x7fd9b8072180 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4009840 tx=0x7fd9b4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e61a998920453ac4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:04.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.072+0000 7fd9bd1b9700 1 -- 172.21.15.1:0/4144948664 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4011830 con 0x7fd9b815d580 2024-03-18T20:26:04.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.072+0000 7fd9bd1b9700 1 -- 172.21.15.1:0/4144948664 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b4011990 con 0x7fd9b815d580 2024-03-18T20:26:04.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.072+0000 7fd9bd1b9700 1 -- 172.21.15.1:0/4144948664 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4011d20 con 0x7fd9b815d580 2024-03-18T20:26:04.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.072+0000 7fd9c041f700 1 -- 172.21.15.1:0/4144948664 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9b815d580 msgr2=0x7fd9b8072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:04.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.072+0000 7fd9c041f700 1 --2- 172.21.15.1:0/4144948664 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9b815d580 0x7fd9b8072180 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4009840 tx=0x7fd9b4009b50 comp rx=0 tx=0).stop 2024-03-18T20:26:04.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.072+0000 7fd9c041f700 1 -- 172.21.15.1:0/4144948664 shutdown_connections 2024-03-18T20:26:04.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.072+0000 7fd9c041f700 1 --2- 172.21.15.1:0/4144948664 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9b815d580 0x7fd9b8072180 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.072+0000 7fd9c041f700 1 --2- 172.21.15.1:0/4144948664 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9b806dd50 0x7fd9b80cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.072+0000 7fd9c041f700 1 -- 172.21.15.1:0/4144948664 >> 172.21.15.1:0/4144948664 conn(0x7fd9b80cc990 msgr2=0x7fd9b80ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:04.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.072+0000 7fd9c041f700 1 -- 172.21.15.1:0/4144948664 shutdown_connections 2024-03-18T20:26:04.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.072+0000 7fd9c041f700 1 -- 172.21.15.1:0/4144948664 wait complete. 2024-03-18T20:26:04.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.074+0000 7fd9c041f700 1 Processor -- start 2024-03-18T20:26:04.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.074+0000 7fd9c041f700 1 -- start start 2024-03-18T20:26:04.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.074+0000 7fd9c041f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9b806dd50 0x7fd9b80c5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.074+0000 7fd9c041f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9b815d580 0x7fd9b8159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.074+0000 7fd9c041f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9b80b08a0 0x7fd9b80b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.074+0000 7fd9c041f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b80b1290 con 0x7fd9b815d580 2024-03-18T20:26:04.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.074+0000 7fd9c041f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b80c4b60 con 0x7fd9b80b08a0 2024-03-18T20:26:04.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.074+0000 7fd9c041f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b80c4e40 con 0x7fd9b806dd50 2024-03-18T20:26:04.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.074+0000 7fd9bd9ba700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9b815d580 0x7fd9b8159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.074+0000 7fd9bd9ba700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9b815d580 0x7fd9b8159310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33996/0 (socket says 172.21.15.1:33996) 2024-03-18T20:26:04.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.075+0000 7fd9bd9ba700 1 -- 172.21.15.1:0/927571169 learned_addr learned my addr 172.21.15.1:0/927571169 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:04.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.075+0000 7fd9be9bc700 1 --2- 172.21.15.1:0/927571169 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9b80b08a0 0x7fd9b80b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.075+0000 7fd9be1bb700 1 --2- 172.21.15.1:0/927571169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9b806dd50 0x7fd9b80c5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.075+0000 7fd9bd9ba700 1 -- 172.21.15.1:0/927571169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9b806dd50 msgr2=0x7fd9b80c5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:04.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.075+0000 7fd9bd9ba700 1 --2- 172.21.15.1:0/927571169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9b806dd50 0x7fd9b80c5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.075+0000 7fd9bd9ba700 1 -- 172.21.15.1:0/927571169 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9b80b08a0 msgr2=0x7fd9b80b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:04.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.075+0000 7fd9bd9ba700 1 --2- 172.21.15.1:0/927571169 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9b80b08a0 0x7fd9b80b0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.075+0000 7fd9bd9ba700 1 -- 172.21.15.1:0/927571169 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b40094a0 con 0x7fd9b815d580 2024-03-18T20:26:04.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.075+0000 7fd9bd9ba700 1 --2- 172.21.15.1:0/927571169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9b815d580 0x7fd9b8159310 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800c700 tx=0x7fd9a800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:04.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.075+0000 7fd9aeffd700 1 -- 172.21.15.1:0/927571169 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8005d70 con 0x7fd9b815d580 2024-03-18T20:26:04.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.075+0000 7fd9aeffd700 1 -- 172.21.15.1:0/927571169 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a80070d0 con 0x7fd9b815d580 2024-03-18T20:26:04.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.075+0000 7fd9aeffd700 1 -- 172.21.15.1:0/927571169 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8007460 con 0x7fd9b815d580 2024-03-18T20:26:04.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.075+0000 7fd9c041f700 1 -- 172.21.15.1:0/927571169 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b80c5120 con 0x7fd9b815d580 2024-03-18T20:26:04.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.075+0000 7fd9c041f700 1 -- 172.21.15.1:0/927571169 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b80c4780 con 0x7fd9b815d580 2024-03-18T20:26:04.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.076+0000 7fd9c041f700 1 -- 172.21.15.1:0/927571169 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b80c4ca0 con 0x7fd9b815d580 2024-03-18T20:26:04.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.077+0000 7fd9aeffd700 1 -- 172.21.15.1:0/927571169 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 143) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fd9a80075c0 con 0x7fd9b815d580 2024-03-18T20:26:04.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.077+0000 7fd9aeffd700 1 -- 172.21.15.1:0/927571169 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 7724+0+0 (secure 0 0 0) 0x7fd9a8088a60 con 0x7fd9b815d580 2024-03-18T20:26:04.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.082+0000 7fd9aeffd700 1 -- 172.21.15.1:0/927571169 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd9b80c4ca0 con 0x7fd9b815d580 2024-03-18T20:26:04.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.334+0000 7fd9c041f700 1 -- 172.21.15.1:0/927571169 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9b80c5640 con 0x7fd9b815d580 2024-03-18T20:26:04.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.334+0000 7fd9aeffd700 1 -- 172.21.15.1:0/927571169 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5963 (secure 0 0 0) 0x7fd9b80c5640 con 0x7fd9b815d580 2024-03-18T20:26:04.338 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:04.339 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":110,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10369,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/677258734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":677258734},{"type":"v1","addr":"172.21.15.165:6835","nonce":677258734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":11544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2018398457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2018398457},{"type":"v1","addr":"172.21.15.1:6836","nonce":2018398457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:25:22.663717+0000","modified":"2024-03-18T20:25:25.668584+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11556":{"gid":11556,"name":"d","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6834/4024828630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4024828630},{"type":"v1","addr":"172.21.15.165:6834","nonce":4024828630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:25:35.549382+0000","modified":"2024-03-18T20:25:38.565325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11547":{"gid":11547,"name":"c","rank":0,"incarnation":106,"state":"up:active","state_seq":6,"addr":"172.21.15.1:6837/1635684321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1635684321},{"type":"v1","addr":"172.21.15.1:6837","nonce":1635684321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-03-18T20:26:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.336+0000 7fd9c041f700 1 -- 172.21.15.1:0/927571169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9b815d580 msgr2=0x7fd9b8159310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.336+0000 7fd9c041f700 1 --2- 172.21.15.1:0/927571169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9b815d580 0x7fd9b8159310 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800c700 tx=0x7fd9a800ca10 comp rx=0 tx=0).stop 2024-03-18T20:26:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.336+0000 7fd9c041f700 1 -- 172.21.15.1:0/927571169 shutdown_connections 2024-03-18T20:26:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.336+0000 7fd9c041f700 1 --2- 172.21.15.1:0/927571169 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9b806dd50 0x7fd9b80c5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.336+0000 7fd9c041f700 1 --2- 172.21.15.1:0/927571169 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9b80b08a0 0x7fd9b80b0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.336+0000 7fd9c041f700 1 --2- 172.21.15.1:0/927571169 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9b815d580 0x7fd9b8159310 unknown :-1 s=CLOSED pgs=2691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.336+0000 7fd9c041f700 1 -- 172.21.15.1:0/927571169 >> 172.21.15.1:0/927571169 conn(0x7fd9b80cc990 msgr2=0x7fd9b80aed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.337+0000 7fd9c041f700 1 -- 172.21.15.1:0/927571169 shutdown_connections 2024-03-18T20:26:04.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.337+0000 7fd9c041f700 1 -- 172.21.15.1:0/927571169 wait complete. 2024-03-18T20:26:04.342 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 110 2024-03-18T20:26:04.353 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:26:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.474+0000 7f24211f8700 1 Processor -- start 2024-03-18T20:26:04.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.474+0000 7f24211f8700 1 -- start start 2024-03-18T20:26:04.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.475+0000 7f24211f8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f241c099ea0 0x7f241c0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.475+0000 7f24211f8700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f241c0d0c80 con 0x7f241c15d680 2024-03-18T20:26:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.475+0000 7f24211f8700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f241c0d09a0 con 0x7f241c099a40 2024-03-18T20:26:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.475+0000 7f24211f8700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f241c0cb830 con 0x7f241c099ea0 2024-03-18T20:26:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.475+0000 7f241ad9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f241c15d680 0x7f241c0cff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57698/0 (socket says 172.21.15.1:57698) 2024-03-18T20:26:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.475+0000 7f241ad9d700 1 -- 172.21.15.1:0/3484595471 learned_addr learned my addr 172.21.15.1:0/3484595471 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.475+0000 7f241b59e700 1 --2- 172.21.15.1:0/3484595471 >> v2:172.21.15.1:3300/0 conn(0x7f241c099ea0 0x7f241c0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.475+0000 7f2419d9b700 1 -- 172.21.15.1:0/3484595471 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979738494 0 0) 0x7f241c0d0c80 con 0x7f241c15d680 2024-03-18T20:26:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.475+0000 7f2419d9b700 1 -- 172.21.15.1:0/3484595471 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24040035b0 con 0x7f241c15d680 2024-03-18T20:26:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.475+0000 7f241b59e700 1 -- 172.21.15.1:0/3484595471 >> v1:172.21.15.165:6789/0 conn(0x7f241c099a40 legacy=0x7f241c0cc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.475+0000 7f241b59e700 1 -- 172.21.15.1:0/3484595471 >> v1:172.21.15.1:6789/0 conn(0x7f241c15d680 legacy=0x7f241c0cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.475+0000 7f241b59e700 1 -- 172.21.15.1:0/3484595471 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c0cbb10 con 0x7f241c099ea0 2024-03-18T20:26:04.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.476+0000 7f241b59e700 1 --2- 172.21.15.1:0/3484595471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c099ea0 0x7f241c0cca20 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 crypto rx=0x7f2414009d90 tx=0x7f241400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ec36348a04ad860 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.476+0000 7f2419d9b700 1 -- 172.21.15.1:0/3484595471 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241400e070 con 0x7f241c099ea0 2024-03-18T20:26:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.476+0000 7f2419d9b700 1 -- 172.21.15.1:0/3484595471 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f241400bb20 con 0x7f241c099ea0 2024-03-18T20:26:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.476+0000 7f2419d9b700 1 -- 172.21.15.1:0/3484595471 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2414012430 con 0x7f241c099ea0 2024-03-18T20:26:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.476+0000 7f24211f8700 1 -- 172.21.15.1:0/3484595471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c099ea0 msgr2=0x7f241c0cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.476+0000 7f24211f8700 1 --2- 172.21.15.1:0/3484595471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c099ea0 0x7f241c0cca20 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 crypto rx=0x7f2414009d90 tx=0x7f241400b330 comp rx=0 tx=0).stop 2024-03-18T20:26:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.476+0000 7f24211f8700 1 -- 172.21.15.1:0/3484595471 shutdown_connections 2024-03-18T20:26:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.476+0000 7f24211f8700 1 --2- 172.21.15.1:0/3484595471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c099ea0 0x7f241c0cca20 unknown :-1 s=CLOSED pgs=2693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.476+0000 7f24211f8700 1 -- 172.21.15.1:0/3484595471 >> 172.21.15.1:0/3484595471 conn(0x7f241c0b2ba0 msgr2=0x7f241c0b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.476+0000 7f24211f8700 1 -- 172.21.15.1:0/3484595471 shutdown_connections 2024-03-18T20:26:04.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.476+0000 7f24211f8700 1 -- 172.21.15.1:0/3484595471 wait complete. 2024-03-18T20:26:04.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.478+0000 7f24211f8700 1 Processor -- start 2024-03-18T20:26:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.478+0000 7f24211f8700 1 -- start start 2024-03-18T20:26:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f24211f8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f241c099a40 0x7f241c1f5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f24211f8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f241c099ea0 0x7f241c1f2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f24211f8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c15d680 0x7f241c0c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f24211f8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c0c6cc0 con 0x7f241c15d680 2024-03-18T20:26:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f24211f8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c1f2ff0 con 0x7f241c099ea0 2024-03-18T20:26:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f24211f8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241c0c60b0 con 0x7f241c099a40 2024-03-18T20:26:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f241b59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c15d680 0x7f241c0c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f241b59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c15d680 0x7f241c0c55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43168/0 (socket says 172.21.15.1:43168) 2024-03-18T20:26:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f241b59e700 1 -- 172.21.15.1:0/3419047521 learned_addr learned my addr 172.21.15.1:0/3419047521 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f241a59c700 1 --2- 172.21.15.1:0/3419047521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f241c099ea0 0x7f241c1f2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f241b59e700 1 -- 172.21.15.1:0/3419047521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f241c099a40 msgr2=0x7f241c1f5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f241b59e700 1 --2- 172.21.15.1:0/3419047521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f241c099a40 0x7f241c1f5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f241b59e700 1 -- 172.21.15.1:0/3419047521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f241c099ea0 msgr2=0x7f241c1f2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f241b59e700 1 --2- 172.21.15.1:0/3419047521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f241c099ea0 0x7f241c1f2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f241b59e700 1 -- 172.21.15.1:0/3419047521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2414009a70 con 0x7f241c15d680 2024-03-18T20:26:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f241b59e700 1 --2- 172.21.15.1:0/3419047521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c15d680 0x7f241c0c55a0 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f241c072140 tx=0x7f2414004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f24037fe700 1 -- 172.21.15.1:0/3419047521 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2414012430 con 0x7f241c15d680 2024-03-18T20:26:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f24037fe700 1 -- 172.21.15.1:0/3419047521 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2414020070 con 0x7f241c15d680 2024-03-18T20:26:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f24211f8700 1 -- 172.21.15.1:0/3419047521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241c0c6390 con 0x7f241c15d680 2024-03-18T20:26:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f24037fe700 1 -- 172.21.15.1:0/3419047521 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2414004280 con 0x7f241c15d680 2024-03-18T20:26:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.479+0000 7f24211f8700 1 -- 172.21.15.1:0/3419047521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241c0c6950 con 0x7f241c15d680 2024-03-18T20:26:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.480+0000 7f24211f8700 1 -- 172.21.15.1:0/3419047521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f241c06cd50 con 0x7f241c15d680 2024-03-18T20:26:04.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.481+0000 7f24037fe700 1 -- 172.21.15.1:0/3419047521 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 143) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f24140043e0 con 0x7f241c15d680 2024-03-18T20:26:04.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.481+0000 7f24037fe700 1 -- 172.21.15.1:0/3419047521 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 7724+0+0 (secure 0 0 0) 0x7f2414022070 con 0x7f241c15d680 2024-03-18T20:26:04.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.486+0000 7f24037fe700 1 -- 172.21.15.1:0/3419047521 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2414056130 con 0x7f241c15d680 2024-03-18T20:26:04.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.689+0000 7f24211f8700 1 -- 172.21.15.1:0/3419047521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f241c0676a0 con 0x7f241c15d680 2024-03-18T20:26:04.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.690+0000 7f24037fe700 1 -- 172.21.15.1:0/3419047521 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v238) v1 ==== 74+0+21117 (secure 0 0 0) 0x7f241402c020 con 0x7f241c15d680 2024-03-18T20:26:04.695 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:04.695 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":238,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:26:03.849604+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:25:20.511297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-18T20:25:21.637693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:25:33.275375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:25:34.518347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/4210355381":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/1879196326":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/101306769":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/997525540":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/1622795270":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/2535232994":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2652258802":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2404324807":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2059399259":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/3657737679":"2024-03-19T20:25:52.933722+0000","172.21.15.1:6833/148356":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2685347289":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/1965726293":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/2324250327":"2024-03-19T20:25:52.933722+0000","172.21.15.1:6832/148356":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/1022040508":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/2165965129":"2024-03-19T20:25:52.933722+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:26:04.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.692+0000 7f24211f8700 1 -- 172.21.15.1:0/3419047521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c15d680 msgr2=0x7f241c0c55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:04.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.692+0000 7f24211f8700 1 --2- 172.21.15.1:0/3419047521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c15d680 0x7f241c0c55a0 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f241c072140 tx=0x7f2414004ad0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.692+0000 7f24211f8700 1 -- 172.21.15.1:0/3419047521 shutdown_connections 2024-03-18T20:26:04.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.692+0000 7f24211f8700 1 --2- 172.21.15.1:0/3419047521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f241c099a40 0x7f241c1f5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.692+0000 7f24211f8700 1 --2- 172.21.15.1:0/3419047521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f241c15d680 0x7f241c0c55a0 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.692+0000 7f24211f8700 1 --2- 172.21.15.1:0/3419047521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f241c099ea0 0x7f241c1f2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.692+0000 7f24211f8700 1 -- 172.21.15.1:0/3419047521 >> 172.21.15.1:0/3419047521 conn(0x7f241c0b2ba0 msgr2=0x7f241c0ade00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:04.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.692+0000 7f24211f8700 1 -- 172.21.15.1:0/3419047521 shutdown_connections 2024-03-18T20:26:04.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.692+0000 7f24211f8700 1 -- 172.21.15.1:0/3419047521 wait complete. 2024-03-18T20:26:04.709 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T20:26:04.709 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:26:04.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.830+0000 7f0b74bf0700 1 Processor -- start 2024-03-18T20:26:04.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.831+0000 7f0b74bf0700 1 -- start start 2024-03-18T20:26:04.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.832+0000 7f0b74bf0700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0b70071a80 0x7f0b7006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.832+0000 7f0b74bf0700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b700acd80 con 0x7f0b70071600 2024-03-18T20:26:04.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.832+0000 7f0b74bf0700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b700cf220 con 0x7f0b7015a320 2024-03-18T20:26:04.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.832+0000 7f0b74bf0700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b70071f10 con 0x7f0b70071a80 2024-03-18T20:26:04.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.832+0000 7f0b6dd9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0b70071600 0x7f0b7006e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57708/0 (socket says 172.21.15.1:57708) 2024-03-18T20:26:04.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.832+0000 7f0b6dd9b700 1 -- 172.21.15.1:0/3319995921 learned_addr learned my addr 172.21.15.1:0/3319995921 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:04.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.832+0000 7f0b6ed9d700 1 --2- 172.21.15.1:0/3319995921 >> v2:172.21.15.165:3300/0 conn(0x7f0b70071a80 0x7f0b7006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.832+0000 7f0b6d59a700 1 -- 172.21.15.1:0/3319995921 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975389745 0 0) 0x7f0b700acd80 con 0x7f0b70071600 2024-03-18T20:26:04.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.832+0000 7f0b6d59a700 1 -- 172.21.15.1:0/3319995921 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b580035b0 con 0x7f0b70071600 2024-03-18T20:26:04.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.832+0000 7f0b6d59a700 1 -- 172.21.15.1:0/3319995921 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190384801 0 0) 0x7f0b60001110 con 0x7f0b7015a320 2024-03-18T20:26:04.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.832+0000 7f0b6d59a700 1 -- 172.21.15.1:0/3319995921 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b700acd80 con 0x7f0b7015a320 2024-03-18T20:26:04.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6d59a700 1 -- 172.21.15.1:0/3319995921 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 736472692 0 0) 0x7f0b580035b0 con 0x7f0b70071600 2024-03-18T20:26:04.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6d59a700 1 -- 172.21.15.1:0/3319995921 >> v2:172.21.15.165:3300/0 conn(0x7f0b70071a80 msgr2=0x7f0b7006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:04.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6d59a700 1 --2- 172.21.15.1:0/3319995921 >> v2:172.21.15.165:3300/0 conn(0x7f0b70071a80 0x7f0b7006e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6d59a700 1 -- 172.21.15.1:0/3319995921 >> v1:172.21.15.165:6789/0 conn(0x7f0b7015a320 legacy=0x7f0b700ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:04.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6d59a700 1 -- 172.21.15.1:0/3319995921 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b700cef40 con 0x7f0b70071600 2024-03-18T20:26:04.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6d59a700 1 -- 172.21.15.1:0/3319995921 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f0b64001670 con 0x7f0b70071600 2024-03-18T20:26:04.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6d59a700 1 -- 172.21.15.1:0/3319995921 >> v1:172.21.15.1:6789/0 conn(0x7f0b70071600 legacy=0x7f0b7006e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:04.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6d59a700 1 --2- 172.21.15.1:0/3319995921 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b58003b90 0x7f0b58003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6d59a700 1 --2- 172.21.15.1:0/3319995921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b58004850 0x7f0b58004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6d59a700 1 --2- 172.21.15.1:0/3319995921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b58005210 0x7f0b5800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6d59a700 1 -- 172.21.15.1:0/3319995921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b70071f10 con 0x7f0b58004850 2024-03-18T20:26:04.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6d59a700 1 -- 172.21.15.1:0/3319995921 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5800c090 con 0x7f0b58005210 2024-03-18T20:26:04.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6d59a700 1 -- 172.21.15.1:0/3319995921 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5800c250 con 0x7f0b58003b90 2024-03-18T20:26:04.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6dd9b700 1 --2- 172.21.15.1:0/3319995921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b58004850 0x7f0b58004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6ed9d700 1 --2- 172.21.15.1:0/3319995921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b58005210 0x7f0b5800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6e59c700 1 --2- 172.21.15.1:0/3319995921 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b58003b90 0x7f0b58003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6dd9b700 1 -- 172.21.15.1:0/3319995921 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b58003b90 msgr2=0x7f0b58003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:04.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6dd9b700 1 --2- 172.21.15.1:0/3319995921 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b58003b90 0x7f0b58003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6dd9b700 1 -- 172.21.15.1:0/3319995921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b58005210 msgr2=0x7f0b5800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:04.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6dd9b700 1 --2- 172.21.15.1:0/3319995921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b58005210 0x7f0b5800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6dd9b700 1 -- 172.21.15.1:0/3319995921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b580035b0 con 0x7f0b58004850 2024-03-18T20:26:04.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6ed9d700 1 --2- 172.21.15.1:0/3319995921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b58005210 0x7f0b5800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:04.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6ed9d700 1 -- 172.21.15.1:0/3319995921 reap_dead start 2024-03-18T20:26:04.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6dd9b700 1 --2- 172.21.15.1:0/3319995921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b58004850 0x7f0b58004cd0 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f0b6400c210 tx=0x7f0b58005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41010abc52f9b9c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:04.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6d59a700 1 -- 172.21.15.1:0/3319995921 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6400cd50 con 0x7f0b58004850 2024-03-18T20:26:04.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6d59a700 1 -- 172.21.15.1:0/3319995921 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b64001af0 con 0x7f0b58004850 2024-03-18T20:26:04.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.833+0000 7f0b6d59a700 1 -- 172.21.15.1:0/3319995921 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b64005ba0 con 0x7f0b58004850 2024-03-18T20:26:04.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.834+0000 7f0b74bf0700 1 -- 172.21.15.1:0/3319995921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b58004850 msgr2=0x7f0b58004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:04.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.834+0000 7f0b74bf0700 1 --2- 172.21.15.1:0/3319995921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b58004850 0x7f0b58004cd0 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f0b6400c210 tx=0x7f0b58005880 comp rx=0 tx=0).stop 2024-03-18T20:26:04.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.834+0000 7f0b74bf0700 1 -- 172.21.15.1:0/3319995921 shutdown_connections 2024-03-18T20:26:04.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.834+0000 7f0b74bf0700 1 --2- 172.21.15.1:0/3319995921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b58004850 0x7f0b58004cd0 unknown :-1 s=CLOSED pgs=2696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.834+0000 7f0b74bf0700 1 -- 172.21.15.1:0/3319995921 >> 172.21.15.1:0/3319995921 conn(0x7f0b700cc990 msgr2=0x7f0b700ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:04.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.834+0000 7f0b74bf0700 1 -- 172.21.15.1:0/3319995921 shutdown_connections 2024-03-18T20:26:04.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.834+0000 7f0b74bf0700 1 -- 172.21.15.1:0/3319995921 wait complete. 2024-03-18T20:26:04.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.835+0000 7f0b74bf0700 1 Processor -- start 2024-03-18T20:26:04.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.835+0000 7f0b74bf0700 1 -- start start 2024-03-18T20:26:04.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.836+0000 7f0b74bf0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b58004850 0x7f0b7014b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.836+0000 7f0b74bf0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b700b3510 0x7f0b700b3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.836+0000 7f0b74bf0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b7015b790 0x7f0b7015bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.836+0000 7f0b74bf0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b700b3ed0 con 0x7f0b58004850 2024-03-18T20:26:04.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.836+0000 7f0b74bf0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7015d560 con 0x7f0b700b3510 2024-03-18T20:26:04.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.836+0000 7f0b74bf0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7014b810 con 0x7f0b7015b790 2024-03-18T20:26:04.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.836+0000 7f0b6e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b58004850 0x7f0b7014b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.836+0000 7f0b6e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b58004850 0x7f0b7014b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43190/0 (socket says 172.21.15.1:43190) 2024-03-18T20:26:04.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.836+0000 7f0b6e59c700 1 -- 172.21.15.1:0/513938475 learned_addr learned my addr 172.21.15.1:0/513938475 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:04.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.836+0000 7f0b6dd9b700 1 --2- 172.21.15.1:0/513938475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b700b3510 0x7f0b700b3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.836+0000 7f0b6ed9d700 1 --2- 172.21.15.1:0/513938475 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b7015b790 0x7f0b7015bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.836+0000 7f0b6e59c700 1 -- 172.21.15.1:0/513938475 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b7015b790 msgr2=0x7f0b7015bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:04.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.836+0000 7f0b6e59c700 1 --2- 172.21.15.1:0/513938475 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b7015b790 0x7f0b7015bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.836+0000 7f0b6e59c700 1 -- 172.21.15.1:0/513938475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b700b3510 msgr2=0x7f0b700b3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:04.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.836+0000 7f0b6e59c700 1 --2- 172.21.15.1:0/513938475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b700b3510 0x7f0b700b3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:04.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.836+0000 7f0b6e59c700 1 -- 172.21.15.1:0/513938475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7015c4c0 con 0x7f0b58004850 2024-03-18T20:26:04.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.837+0000 7f0b6dd9b700 1 --2- 172.21.15.1:0/513938475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b700b3510 0x7f0b700b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:04.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.837+0000 7f0b6ed9d700 1 --2- 172.21.15.1:0/513938475 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b7015b790 0x7f0b7015bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:04.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.837+0000 7f0b6e59c700 1 --2- 172.21.15.1:0/513938475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b58004850 0x7f0b7014b2d0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f0b6000e6d0 tx=0x7f0b6000ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:04.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.837+0000 7f0b5effd700 1 -- 172.21.15.1:0/513938475 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b600052e0 con 0x7f0b58004850 2024-03-18T20:26:04.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.837+0000 7f0b5effd700 1 -- 172.21.15.1:0/513938475 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b60005440 con 0x7f0b58004850 2024-03-18T20:26:04.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.837+0000 7f0b5effd700 1 -- 172.21.15.1:0/513938475 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b600038c0 con 0x7f0b58004850 2024-03-18T20:26:04.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.837+0000 7f0b74bf0700 1 -- 172.21.15.1:0/513938475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7015c7a0 con 0x7f0b58004850 2024-03-18T20:26:04.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.837+0000 7f0b74bf0700 1 -- 172.21.15.1:0/513938475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7015cd60 con 0x7f0b58004850 2024-03-18T20:26:04.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.838+0000 7f0b74bf0700 1 -- 172.21.15.1:0/513938475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b700cdbf0 con 0x7f0b58004850 2024-03-18T20:26:04.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.838+0000 7f0b5effd700 1 -- 172.21.15.1:0/513938475 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 143) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f0b60003a20 con 0x7f0b58004850 2024-03-18T20:26:04.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.839+0000 7f0b5effd700 1 -- 172.21.15.1:0/513938475 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 7724+0+0 (secure 0 0 0) 0x7f0b60089260 con 0x7f0b58004850 2024-03-18T20:26:04.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.843+0000 7f0b5effd700 1 -- 172.21.15.1:0/513938475 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0b60053fc0 con 0x7f0b58004850 2024-03-18T20:26:04.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.859+0000 7f0b5effd700 1 -- 172.21.15.1:0/513938475 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 144) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f0b60053a70 con 0x7f0b58004850 2024-03-18T20:26:04.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.859+0000 7f0b5effd700 1 --2- 172.21.15.1:0/513938475 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b50036150 0x7f0b50075220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:04.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.861+0000 7f0b6dd9b700 1 --2- 172.21.15.1:0/513938475 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b50036150 0x7f0b50075220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:04.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:04.862+0000 7f0b6dd9b700 1 --2- 172.21.15.1:0/513938475 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b50036150 0x7f0b50075220 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0b7006c6f0 tx=0x7f0b64006040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:05.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.099+0000 7f0b74bf0700 1 -- 172.21.15.1:0/513938475 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b700cf220 con 0x7f0b58004850 2024-03-18T20:26:05.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.100+0000 7f0b5effd700 1 -- 172.21.15.1:0/513938475 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5963 (secure 0 0 0) 0x7f0b700cf220 con 0x7f0b58004850 2024-03-18T20:26:05.104 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:05.104 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":110,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10369,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/677258734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":677258734},{"type":"v1","addr":"172.21.15.165:6835","nonce":677258734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":11544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2018398457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2018398457},{"type":"v1","addr":"172.21.15.1:6836","nonce":2018398457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:25:22.663717+0000","modified":"2024-03-18T20:25:25.668584+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11556":{"gid":11556,"name":"d","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6834/4024828630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4024828630},{"type":"v1","addr":"172.21.15.165:6834","nonce":4024828630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:25:35.549382+0000","modified":"2024-03-18T20:25:38.565325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11547":{"gid":11547,"name":"c","rank":0,"incarnation":106,"state":"up:active","state_seq":6,"addr":"172.21.15.1:6837/1635684321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1635684321},{"type":"v1","addr":"172.21.15.1:6837","nonce":1635684321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-03-18T20:26:05.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.102+0000 7f0b74bf0700 1 -- 172.21.15.1:0/513938475 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b50036150 msgr2=0x7f0b50075220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.102+0000 7f0b74bf0700 1 --2- 172.21.15.1:0/513938475 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b50036150 0x7f0b50075220 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0b7006c6f0 tx=0x7f0b64006040 comp rx=0 tx=0).stop 2024-03-18T20:26:05.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.102+0000 7f0b74bf0700 1 -- 172.21.15.1:0/513938475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b58004850 msgr2=0x7f0b7014b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.102+0000 7f0b74bf0700 1 --2- 172.21.15.1:0/513938475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b58004850 0x7f0b7014b2d0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f0b6000e6d0 tx=0x7f0b6000ea90 comp rx=0 tx=0).stop 2024-03-18T20:26:05.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.102+0000 7f0b74bf0700 1 -- 172.21.15.1:0/513938475 shutdown_connections 2024-03-18T20:26:05.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.102+0000 7f0b74bf0700 1 --2- 172.21.15.1:0/513938475 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0b50036150 0x7f0b50075220 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.102+0000 7f0b74bf0700 1 --2- 172.21.15.1:0/513938475 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0b7015b790 0x7f0b7015bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.102+0000 7f0b74bf0700 1 --2- 172.21.15.1:0/513938475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b58004850 0x7f0b7014b2d0 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.102+0000 7f0b74bf0700 1 --2- 172.21.15.1:0/513938475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0b700b3510 0x7f0b700b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.102+0000 7f0b74bf0700 1 -- 172.21.15.1:0/513938475 >> 172.21.15.1:0/513938475 conn(0x7f0b700cc990 msgr2=0x7f0b700b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:05.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.102+0000 7f0b74bf0700 1 -- 172.21.15.1:0/513938475 shutdown_connections 2024-03-18T20:26:05.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.102+0000 7f0b74bf0700 1 -- 172.21.15.1:0/513938475 wait complete. 2024-03-18T20:26:05.107 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 110 2024-03-18T20:26:05.118 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:26:05.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.239+0000 7fc5c81c2700 1 Processor -- start 2024-03-18T20:26:05.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.240+0000 7fc5c81c2700 1 -- start start 2024-03-18T20:26:05.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.240+0000 7fc5c81c2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5c015a2e0 0x7fc5c00ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.240+0000 7fc5c81c2700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c006e270 con 0x7fc5c00cc170 2024-03-18T20:26:05.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.240+0000 7fc5c81c2700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c00ae050 con 0x7fc5c00ae600 2024-03-18T20:26:05.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.240+0000 7fc5c81c2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c006f690 con 0x7fc5c015a2e0 2024-03-18T20:26:05.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c675f700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc5c00cc170 0x7fc5c006ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57724/0 (socket says 172.21.15.1:57724) 2024-03-18T20:26:05.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c675f700 1 -- 172.21.15.1:0/2486550939 learned_addr learned my addr 172.21.15.1:0/2486550939 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:05.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c4f5c700 1 -- 172.21.15.1:0/2486550939 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287015640 0 0) 0x7fc5c006e270 con 0x7fc5c00cc170 2024-03-18T20:26:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c4f5c700 1 -- 172.21.15.1:0/2486550939 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5ac0035b0 con 0x7fc5c00cc170 2024-03-18T20:26:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c4f5c700 1 -- 172.21.15.1:0/2486550939 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3925048003 0 0) 0x7fc5ac0035b0 con 0x7fc5c00cc170 2024-03-18T20:26:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c4f5c700 1 -- 172.21.15.1:0/2486550939 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5c015a2e0 msgr2=0x7fc5c00ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c4f5c700 1 --2- 172.21.15.1:0/2486550939 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5c015a2e0 0x7fc5c00ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c4f5c700 1 -- 172.21.15.1:0/2486550939 >> v1:172.21.15.165:6789/0 conn(0x7fc5c00ae600 legacy=0x7fc5c00ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c4f5c700 1 -- 172.21.15.1:0/2486550939 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c00add70 con 0x7fc5c00cc170 2024-03-18T20:26:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c4f5c700 1 -- 172.21.15.1:0/2486550939 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fc5b80019f0 con 0x7fc5c00cc170 2024-03-18T20:26:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c4f5c700 1 -- 172.21.15.1:0/2486550939 >> v1:172.21.15.1:6789/0 conn(0x7fc5c00cc170 legacy=0x7fc5c006ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c4f5c700 1 --2- 172.21.15.1:0/2486550939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5ac003a30 0x7fc5ac003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c4f5c700 1 --2- 172.21.15.1:0/2486550939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5ac004850 0x7fc5ac004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c4f5c700 1 --2- 172.21.15.1:0/2486550939 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5c015a2e0 0x7fc5ac00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c4f5c700 1 -- 172.21.15.1:0/2486550939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c006f690 con 0x7fc5ac003a30 2024-03-18T20:26:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c4f5c700 1 -- 172.21.15.1:0/2486550939 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ac005540 con 0x7fc5ac004850 2024-03-18T20:26:05.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c4f5c700 1 -- 172.21.15.1:0/2486550939 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ac005700 con 0x7fc5c015a2e0 2024-03-18T20:26:05.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c5f5e700 1 --2- 172.21.15.1:0/2486550939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5ac003a30 0x7fc5ac003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:05.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c675f700 1 --2- 172.21.15.1:0/2486550939 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5c015a2e0 0x7fc5ac00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:05.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.241+0000 7fc5c575d700 1 --2- 172.21.15.1:0/2486550939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5ac004850 0x7fc5ac004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:05.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.242+0000 7fc5c5f5e700 1 -- 172.21.15.1:0/2486550939 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5c015a2e0 msgr2=0x7fc5ac00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:05.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.242+0000 7fc5c5f5e700 1 --2- 172.21.15.1:0/2486550939 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5c015a2e0 0x7fc5ac00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.242+0000 7fc5c5f5e700 1 -- 172.21.15.1:0/2486550939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5ac004850 msgr2=0x7fc5ac004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:05.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.242+0000 7fc5c5f5e700 1 --2- 172.21.15.1:0/2486550939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5ac004850 0x7fc5ac004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.242+0000 7fc5c5f5e700 1 -- 172.21.15.1:0/2486550939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ac0035b0 con 0x7fc5ac003a30 2024-03-18T20:26:05.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.242+0000 7fc5c5f5e700 1 --2- 172.21.15.1:0/2486550939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5ac003a30 0x7fc5ac003e10 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc0018d0 tx=0x7fc5ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84aed81753208f4a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:05.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.242+0000 7fc5c4f5c700 1 -- 172.21.15.1:0/2486550939 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc00c690 con 0x7fc5ac003a30 2024-03-18T20:26:05.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.242+0000 7fc5c4f5c700 1 -- 172.21.15.1:0/2486550939 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5bc00c810 con 0x7fc5ac003a30 2024-03-18T20:26:05.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.242+0000 7fc5c4f5c700 1 -- 172.21.15.1:0/2486550939 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5bc00cbc0 con 0x7fc5ac003a30 2024-03-18T20:26:05.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.242+0000 7fc5c81c2700 1 -- 172.21.15.1:0/2486550939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5ac003a30 msgr2=0x7fc5ac003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.242+0000 7fc5c81c2700 1 --2- 172.21.15.1:0/2486550939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5ac003a30 0x7fc5ac003e10 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc0018d0 tx=0x7fc5ac005880 comp rx=0 tx=0).stop 2024-03-18T20:26:05.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.242+0000 7fc5c675f700 1 -- 172.21.15.1:0/2486550939 reap_dead start 2024-03-18T20:26:05.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.243+0000 7fc5c81c2700 1 -- 172.21.15.1:0/2486550939 shutdown_connections 2024-03-18T20:26:05.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.243+0000 7fc5c81c2700 1 -- 172.21.15.1:0/2486550939 >> 172.21.15.1:0/2486550939 conn(0x7fc5c006b860 msgr2=0x7fc5c006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:05.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.243+0000 7fc5c81c2700 1 -- 172.21.15.1:0/2486550939 shutdown_connections 2024-03-18T20:26:05.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.243+0000 7fc5c81c2700 1 -- 172.21.15.1:0/2486550939 wait complete. 2024-03-18T20:26:05.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.244+0000 7fc5c81c2700 1 Processor -- start 2024-03-18T20:26:05.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.244+0000 7fc5c81c2700 1 -- start start 2024-03-18T20:26:05.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.245+0000 7fc5c81c2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ac003a30 0x7fc5c0149a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.245+0000 7fc5c81c2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c0152ca0 0x7fc5c0099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.245+0000 7fc5c81c2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5c009b310 0x7fc5c006a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.245+0000 7fc5c81c2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c0149f90 con 0x7fc5c0152ca0 2024-03-18T20:26:05.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.245+0000 7fc5c81c2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c009a270 con 0x7fc5c009b310 2024-03-18T20:26:05.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.245+0000 7fc5c81c2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c009a550 con 0x7fc5ac003a30 2024-03-18T20:26:05.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.245+0000 7fc5c575d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c0152ca0 0x7fc5c0099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:05.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.245+0000 7fc5c575d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c0152ca0 0x7fc5c0099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43200/0 (socket says 172.21.15.1:43200) 2024-03-18T20:26:05.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.245+0000 7fc5c575d700 1 -- 172.21.15.1:0/3997335070 learned_addr learned my addr 172.21.15.1:0/3997335070 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:05.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.245+0000 7fc5c5f5e700 1 --2- 172.21.15.1:0/3997335070 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ac003a30 0x7fc5c0149a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:05.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.245+0000 7fc5c675f700 1 --2- 172.21.15.1:0/3997335070 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5c009b310 0x7fc5c006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:05.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.246+0000 7fc5c575d700 1 -- 172.21.15.1:0/3997335070 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ac003a30 msgr2=0x7fc5c0149a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.246+0000 7fc5c575d700 1 --2- 172.21.15.1:0/3997335070 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ac003a30 0x7fc5c0149a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.246+0000 7fc5c575d700 1 -- 172.21.15.1:0/3997335070 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5c009b310 msgr2=0x7fc5c006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.246+0000 7fc5c575d700 1 --2- 172.21.15.1:0/3997335070 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5c009b310 0x7fc5c006a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.246+0000 7fc5c575d700 1 -- 172.21.15.1:0/3997335070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c009a830 con 0x7fc5c0152ca0 2024-03-18T20:26:05.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.246+0000 7fc5c575d700 1 --2- 172.21.15.1:0/3997335070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c0152ca0 0x7fc5c0099a40 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400e870 tx=0x7fc5b400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:05.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.246+0000 7fc5b27fc700 1 -- 172.21.15.1:0/3997335070 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4013680 con 0x7fc5c0152ca0 2024-03-18T20:26:05.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.246+0000 7fc5b27fc700 1 -- 172.21.15.1:0/3997335070 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b40137e0 con 0x7fc5c0152ca0 2024-03-18T20:26:05.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.246+0000 7fc5c81c2700 1 -- 172.21.15.1:0/3997335070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c009ab10 con 0x7fc5c0152ca0 2024-03-18T20:26:05.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.246+0000 7fc5b27fc700 1 -- 172.21.15.1:0/3997335070 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4013b90 con 0x7fc5c0152ca0 2024-03-18T20:26:05.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.246+0000 7fc5c81c2700 1 -- 172.21.15.1:0/3997335070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c0099f80 con 0x7fc5c0152ca0 2024-03-18T20:26:05.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.247+0000 7fc5c81c2700 1 -- 172.21.15.1:0/3997335070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c00676a0 con 0x7fc5c0152ca0 2024-03-18T20:26:05.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.248+0000 7fc5b27fc700 1 -- 172.21.15.1:0/3997335070 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 144) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fc5b4013cf0 con 0x7fc5c0152ca0 2024-03-18T20:26:05.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.248+0000 7fc5b27fc700 1 --2- 172.21.15.1:0/3997335070 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc5a406aae0 0x7fc5a406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.248+0000 7fc5b27fc700 1 -- 172.21.15.1:0/3997335070 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 7724+0+0 (secure 0 0 0) 0x7fc5b4089430 con 0x7fc5c0152ca0 2024-03-18T20:26:05.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.248+0000 7fc5c5f5e700 1 --2- 172.21.15.1:0/3997335070 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc5a406aae0 0x7fc5a406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:05.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.249+0000 7fc5c5f5e700 1 --2- 172.21.15.1:0/3997335070 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc5a406aae0 0x7fc5a406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc014040 tx=0x7fc5bc005040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:05.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.253+0000 7fc5b27fc700 1 -- 172.21.15.1:0/3997335070 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc5b4054170 con 0x7fc5c0152ca0 2024-03-18T20:26:05.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.497+0000 7fc5c81c2700 1 -- 172.21.15.1:0/3997335070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5c006ffc0 con 0x7fc5c0152ca0 2024-03-18T20:26:05.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.497+0000 7fc5b27fc700 1 -- 172.21.15.1:0/3997335070 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5963 (secure 0 0 0) 0x7fc5b4057790 con 0x7fc5c0152ca0 2024-03-18T20:26:05.501 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:05.502 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":110,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10369,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/677258734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":677258734},{"type":"v1","addr":"172.21.15.165:6835","nonce":677258734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":11544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2018398457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2018398457},{"type":"v1","addr":"172.21.15.1:6836","nonce":2018398457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:25:22.663717+0000","modified":"2024-03-18T20:25:25.668584+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11556":{"gid":11556,"name":"d","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6834/4024828630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4024828630},{"type":"v1","addr":"172.21.15.165:6834","nonce":4024828630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:25:35.549382+0000","modified":"2024-03-18T20:25:38.565325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11547":{"gid":11547,"name":"c","rank":0,"incarnation":106,"state":"up:active","state_seq":6,"addr":"172.21.15.1:6837/1635684321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1635684321},{"type":"v1","addr":"172.21.15.1:6837","nonce":1635684321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-03-18T20:26:05.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.499+0000 7fc5c81c2700 1 -- 172.21.15.1:0/3997335070 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc5a406aae0 msgr2=0x7fc5a406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.499+0000 7fc5c81c2700 1 --2- 172.21.15.1:0/3997335070 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc5a406aae0 0x7fc5a406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc014040 tx=0x7fc5bc005040 comp rx=0 tx=0).stop 2024-03-18T20:26:05.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.499+0000 7fc5c81c2700 1 -- 172.21.15.1:0/3997335070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c0152ca0 msgr2=0x7fc5c0099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.499+0000 7fc5c81c2700 1 --2- 172.21.15.1:0/3997335070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c0152ca0 0x7fc5c0099a40 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400e870 tx=0x7fc5b400eb80 comp rx=0 tx=0).stop 2024-03-18T20:26:05.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.499+0000 7fc5c81c2700 1 -- 172.21.15.1:0/3997335070 shutdown_connections 2024-03-18T20:26:05.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.499+0000 7fc5c81c2700 1 --2- 172.21.15.1:0/3997335070 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc5a406aae0 0x7fc5a406cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.499+0000 7fc5c81c2700 1 --2- 172.21.15.1:0/3997335070 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5ac003a30 0x7fc5c0149a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.499+0000 7fc5c81c2700 1 --2- 172.21.15.1:0/3997335070 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5c009b310 0x7fc5c006a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.499+0000 7fc5c81c2700 1 --2- 172.21.15.1:0/3997335070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c0152ca0 0x7fc5c0099a40 unknown :-1 s=CLOSED pgs=2700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.499+0000 7fc5c81c2700 1 -- 172.21.15.1:0/3997335070 >> 172.21.15.1:0/3997335070 conn(0x7fc5c006b860 msgr2=0x7fc5c0159530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:05.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.499+0000 7fc5c81c2700 1 -- 172.21.15.1:0/3997335070 shutdown_connections 2024-03-18T20:26:05.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.499+0000 7fc5c81c2700 1 -- 172.21.15.1:0/3997335070 wait complete. 2024-03-18T20:26:05.505 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 110 2024-03-18T20:26:05.516 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:26:05.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.634+0000 7f2604865700 1 Processor -- start 2024-03-18T20:26:05.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.634+0000 7f2604865700 1 -- start start 2024-03-18T20:26:05.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.635+0000 7f2604865700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2600099a40 0x7f2600099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.635+0000 7f2604865700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f260009a3e0 con 0x7f26000cc310 2024-03-18T20:26:05.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.635+0000 7f2604865700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26000d0c80 con 0x7f260015d680 2024-03-18T20:26:05.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.635+0000 7f2604865700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26000d09a0 con 0x7f2600099a40 2024-03-18T20:26:05.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fed9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f26000cc310 0x7f26000cc7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57738/0 (socket says 172.21.15.1:57738) 2024-03-18T20:26:05.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fed9d700 1 -- 172.21.15.1:0/937590518 learned_addr learned my addr 172.21.15.1:0/937590518 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:05.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fdd9b700 1 --2- 172.21.15.1:0/937590518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2600099a40 0x7f2600099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:05.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fd59a700 1 -- 172.21.15.1:0/937590518 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622336667 0 0) 0x7f260009a3e0 con 0x7f26000cc310 2024-03-18T20:26:05.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fd59a700 1 -- 172.21.15.1:0/937590518 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25ec0035b0 con 0x7f26000cc310 2024-03-18T20:26:05.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fd59a700 1 -- 172.21.15.1:0/937590518 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803473984 0 0) 0x7f26000d0c80 con 0x7f260015d680 2024-03-18T20:26:05.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fd59a700 1 -- 172.21.15.1:0/937590518 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f260009a3e0 con 0x7f260015d680 2024-03-18T20:26:05.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fd59a700 1 -- 172.21.15.1:0/937590518 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2126508303 0 0) 0x7f25ec0035b0 con 0x7f26000cc310 2024-03-18T20:26:05.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fd59a700 1 -- 172.21.15.1:0/937590518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2600099a40 msgr2=0x7f2600099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:05.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fd59a700 1 --2- 172.21.15.1:0/937590518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2600099a40 0x7f2600099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fd59a700 1 -- 172.21.15.1:0/937590518 >> v1:172.21.15.165:6789/0 conn(0x7f260015d680 legacy=0x7f26000cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fd59a700 1 -- 172.21.15.1:0/937590518 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26000cb830 con 0x7f26000cc310 2024-03-18T20:26:05.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fd59a700 1 -- 172.21.15.1:0/937590518 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f25f80019f0 con 0x7f26000cc310 2024-03-18T20:26:05.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fd59a700 1 -- 172.21.15.1:0/937590518 >> v1:172.21.15.1:6789/0 conn(0x7f26000cc310 legacy=0x7f26000cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fd59a700 1 --2- 172.21.15.1:0/937590518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25ec003bc0 0x7f25ec003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fd59a700 1 --2- 172.21.15.1:0/937590518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2600099a40 0x7f25ec004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fd59a700 1 --2- 172.21.15.1:0/937590518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f25ec004d90 0x7f25ec005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fd59a700 1 -- 172.21.15.1:0/937590518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26000d09a0 con 0x7f25ec003bc0 2024-03-18T20:26:05.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fd59a700 1 -- 172.21.15.1:0/937590518 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec00bac0 con 0x7f25ec004d90 2024-03-18T20:26:05.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fd59a700 1 -- 172.21.15.1:0/937590518 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec00bc80 con 0x7f2600099a40 2024-03-18T20:26:05.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fe59c700 1 --2- 172.21.15.1:0/937590518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25ec003bc0 0x7f25ec003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:05.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fdd9b700 1 --2- 172.21.15.1:0/937590518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2600099a40 0x7f25ec004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:05.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.636+0000 7f25fed9d700 1 --2- 172.21.15.1:0/937590518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f25ec004d90 0x7f25ec005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:05.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.637+0000 7f25fe59c700 1 -- 172.21.15.1:0/937590518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2600099a40 msgr2=0x7f25ec004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:05.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.637+0000 7f25fe59c700 1 --2- 172.21.15.1:0/937590518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2600099a40 0x7f25ec004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.637+0000 7f25fe59c700 1 -- 172.21.15.1:0/937590518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f25ec004d90 msgr2=0x7f25ec005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:05.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.637+0000 7f25fe59c700 1 --2- 172.21.15.1:0/937590518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f25ec004d90 0x7f25ec005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.637+0000 7f25fe59c700 1 -- 172.21.15.1:0/937590518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec0035b0 con 0x7f25ec003bc0 2024-03-18T20:26:05.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.637+0000 7f25fed9d700 1 --2- 172.21.15.1:0/937590518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f25ec004d90 0x7f25ec005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:05.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.637+0000 7f25fe59c700 1 --2- 172.21.15.1:0/937590518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25ec003bc0 0x7f25ec003fc0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f25f0001940 tx=0x7f25ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e6936fca18fca2b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:05.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.637+0000 7f25fd59a700 1 -- 172.21.15.1:0/937590518 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f0001430 con 0x7f25ec003bc0 2024-03-18T20:26:05.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.637+0000 7f25fd59a700 1 -- 172.21.15.1:0/937590518 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25f0001d70 con 0x7f25ec003bc0 2024-03-18T20:26:05.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.637+0000 7f25fd59a700 1 -- 172.21.15.1:0/937590518 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f0005510 con 0x7f25ec003bc0 2024-03-18T20:26:05.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.637+0000 7f2604865700 1 -- 172.21.15.1:0/937590518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25ec003bc0 msgr2=0x7f25ec003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.637+0000 7f2604865700 1 --2- 172.21.15.1:0/937590518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25ec003bc0 0x7f25ec003fc0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f25f0001940 tx=0x7f25ec005880 comp rx=0 tx=0).stop 2024-03-18T20:26:05.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.637+0000 7f25fed9d700 1 -- 172.21.15.1:0/937590518 reap_dead start 2024-03-18T20:26:05.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.637+0000 7f2604865700 1 -- 172.21.15.1:0/937590518 shutdown_connections 2024-03-18T20:26:05.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.638+0000 7f2604865700 1 -- 172.21.15.1:0/937590518 >> 172.21.15.1:0/937590518 conn(0x7f26000b2ba0 msgr2=0x7f26000b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:05.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.638+0000 7f2604865700 1 -- 172.21.15.1:0/937590518 shutdown_connections 2024-03-18T20:26:05.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.638+0000 7f2604865700 1 -- 172.21.15.1:0/937590518 wait complete. 2024-03-18T20:26:05.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.639+0000 7f2604865700 1 Processor -- start 2024-03-18T20:26:05.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.639+0000 7f2604865700 1 -- start start 2024-03-18T20:26:05.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f2604865700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25ec003bc0 0x7f2600148ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f2604865700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26000c9630 0x7f2600160400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f2604865700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f260015a7d0 0x7f260015ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f2604865700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26001597e0 con 0x7f25ec003bc0 2024-03-18T20:26:05.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f2604865700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26001583c0 con 0x7f26000c9630 2024-03-18T20:26:05.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f2604865700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2600160c10 con 0x7f260015a7d0 2024-03-18T20:26:05.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f25fe59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25ec003bc0 0x7f2600148ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:05.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f25fe59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25ec003bc0 0x7f2600148ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43228/0 (socket says 172.21.15.1:43228) 2024-03-18T20:26:05.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f25fe59c700 1 -- 172.21.15.1:0/1899116171 learned_addr learned my addr 172.21.15.1:0/1899116171 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:05.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f25fdd9b700 1 --2- 172.21.15.1:0/1899116171 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26000c9630 0x7f2600160400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:05.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f25fed9d700 1 --2- 172.21.15.1:0/1899116171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f260015a7d0 0x7f260015ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:05.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f25fe59c700 1 -- 172.21.15.1:0/1899116171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f260015a7d0 msgr2=0x7f260015ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f25fe59c700 1 --2- 172.21.15.1:0/1899116171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f260015a7d0 0x7f260015ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f25fe59c700 1 -- 172.21.15.1:0/1899116171 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26000c9630 msgr2=0x7f2600160400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f25fe59c700 1 --2- 172.21.15.1:0/1899116171 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26000c9630 0x7f2600160400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f25fe59c700 1 -- 172.21.15.1:0/1899116171 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2600160940 con 0x7f25ec003bc0 2024-03-18T20:26:05.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f25fed9d700 1 --2- 172.21.15.1:0/1899116171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f260015a7d0 0x7f260015ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:05.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f25fe59c700 1 --2- 172.21.15.1:0/1899116171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25ec003bc0 0x7f2600148ce0 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f25ec005830 tx=0x7f25f00012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:05.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f25f6ffd700 1 -- 172.21.15.1:0/1899116171 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f0006c70 con 0x7f25ec003bc0 2024-03-18T20:26:05.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f25f6ffd700 1 -- 172.21.15.1:0/1899116171 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25f0006dd0 con 0x7f25ec003bc0 2024-03-18T20:26:05.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f2604865700 1 -- 172.21.15.1:0/1899116171 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2600159a10 con 0x7f25ec003bc0 2024-03-18T20:26:05.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.640+0000 7f25f6ffd700 1 -- 172.21.15.1:0/1899116171 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f0007240 con 0x7f25ec003bc0 2024-03-18T20:26:05.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.641+0000 7f2604865700 1 -- 172.21.15.1:0/1899116171 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2600159fd0 con 0x7f25ec003bc0 2024-03-18T20:26:05.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.642+0000 7f2604865700 1 -- 172.21.15.1:0/1899116171 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26001597e0 con 0x7f25ec003bc0 2024-03-18T20:26:05.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.642+0000 7f25f6ffd700 1 -- 172.21.15.1:0/1899116171 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 144) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f25f00073a0 con 0x7f25ec003bc0 2024-03-18T20:26:05.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.643+0000 7f25f6ffd700 1 --2- 172.21.15.1:0/1899116171 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f25e006aae0 0x7f25e006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:05.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.643+0000 7f25fdd9b700 1 --2- 172.21.15.1:0/1899116171 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f25e006aae0 0x7f25e006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:05.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.643+0000 7f25f6ffd700 1 -- 172.21.15.1:0/1899116171 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 7724+0+0 (secure 0 0 0) 0x7f25f008a820 con 0x7f25ec003bc0 2024-03-18T20:26:05.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.643+0000 7f25fdd9b700 1 --2- 172.21.15.1:0/1899116171 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f25e006aae0 0x7f25e006cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f260014f3f0 tx=0x7f25e8009210 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:05.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.647+0000 7f25f6ffd700 1 -- 172.21.15.1:0/1899116171 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f25f00554e0 con 0x7f25ec003bc0 2024-03-18T20:26:05.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.849+0000 7f2604865700 1 -- 172.21.15.1:0/1899116171 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f260015a620 con 0x7f25ec003bc0 2024-03-18T20:26:05.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.851+0000 7f25f6ffd700 1 -- 172.21.15.1:0/1899116171 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v238) v1 ==== 74+0+21117 (secure 0 0 0) 0x7f25f0007650 con 0x7f25ec003bc0 2024-03-18T20:26:05.856 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:05.857 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":238,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:26:03.849604+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:25:20.511297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"231","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-18T20:25:21.637693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:25:33.275375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:25:34.518347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/4210355381":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/1879196326":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/101306769":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/997525540":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/1622795270":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/2535232994":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2652258802":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2404324807":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2059399259":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/3657737679":"2024-03-19T20:25:52.933722+0000","172.21.15.1:6833/148356":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2685347289":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/1965726293":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/2324250327":"2024-03-19T20:25:52.933722+0000","172.21.15.1:6832/148356":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/1022040508":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/2165965129":"2024-03-19T20:25:52.933722+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:26:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.853+0000 7f2604865700 1 -- 172.21.15.1:0/1899116171 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f25e006aae0 msgr2=0x7f25e006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.853+0000 7f2604865700 1 --2- 172.21.15.1:0/1899116171 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f25e006aae0 0x7f25e006cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f260014f3f0 tx=0x7f25e8009210 comp rx=0 tx=0).stop 2024-03-18T20:26:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.853+0000 7f2604865700 1 -- 172.21.15.1:0/1899116171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25ec003bc0 msgr2=0x7f2600148ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.853+0000 7f2604865700 1 --2- 172.21.15.1:0/1899116171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25ec003bc0 0x7f2600148ce0 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f25ec005830 tx=0x7f25f00012d0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.853+0000 7f2604865700 1 -- 172.21.15.1:0/1899116171 shutdown_connections 2024-03-18T20:26:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.853+0000 7f2604865700 1 --2- 172.21.15.1:0/1899116171 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f25e006aae0 0x7f25e006cfa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.853+0000 7f2604865700 1 --2- 172.21.15.1:0/1899116171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f260015a7d0 0x7f260015ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.853+0000 7f2604865700 1 --2- 172.21.15.1:0/1899116171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25ec003bc0 0x7f2600148ce0 unknown :-1 s=CLOSED pgs=2703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.853+0000 7f2604865700 1 --2- 172.21.15.1:0/1899116171 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26000c9630 0x7f2600160400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:05.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.853+0000 7f2604865700 1 -- 172.21.15.1:0/1899116171 >> 172.21.15.1:0/1899116171 conn(0x7f26000b2ba0 msgr2=0x7f26000cc760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:05.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.853+0000 7f2604865700 1 -- 172.21.15.1:0/1899116171 shutdown_connections 2024-03-18T20:26:05.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.853+0000 7f2604865700 1 -- 172.21.15.1:0/1899116171 wait complete. 2024-03-18T20:26:05.871 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T20:26:06.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.997+0000 7fbcc0229700 1 Processor -- start 2024-03-18T20:26:06.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.997+0000 7fbcc0229700 1 -- start start 2024-03-18T20:26:06.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.998+0000 7fbcc0229700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbcb815d5b0 0x7fbcb815d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:06.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.998+0000 7fbcc0229700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbcb8099a40 0x7fbcb8099ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:06.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.998+0000 7fbcc0229700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbcb809a410 0x7fbcb806ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:06.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.998+0000 7fbcc0229700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcb80cbbb0 con 0x7fbcb8099a40 2024-03-18T20:26:06.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.998+0000 7fbcc0229700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcb809a860 con 0x7fbcb815d5b0 2024-03-18T20:26:06.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.998+0000 7fbcc0229700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb80b3b00 con 0x7fbcb809a410 2024-03-18T20:26:06.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.998+0000 7fbcbd7c4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbcb8099a40 0x7fbcb8099ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:06.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.998+0000 7fbcbd7c4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbcb8099a40 0x7fbcb8099ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43246/0 (socket says 172.21.15.1:43246) 2024-03-18T20:26:06.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.998+0000 7fbcbd7c4700 1 -- 172.21.15.1:0/3667488747 learned_addr learned my addr 172.21.15.1:0/3667488747 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:06.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.998+0000 7fbcbe7c6700 1 --2- 172.21.15.1:0/3667488747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbcb809a410 0x7fbcb806ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:06.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.998+0000 7fbcbdfc5700 1 --2- 172.21.15.1:0/3667488747 >> v2:172.21.15.165:3300/0 conn(0x7fbcb815d5b0 0x7fbcb815d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:06.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.998+0000 7fbcbd7c4700 1 -- 172.21.15.1:0/3667488747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbcb809a410 msgr2=0x7fbcb806ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:06.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.998+0000 7fbcbd7c4700 1 --2- 172.21.15.1:0/3667488747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbcb809a410 0x7fbcb806ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:06.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.998+0000 7fbcbd7c4700 1 -- 172.21.15.1:0/3667488747 >> v2:172.21.15.165:3300/0 conn(0x7fbcb815d5b0 msgr2=0x7fbcb815d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:06.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.998+0000 7fbcbd7c4700 1 --2- 172.21.15.1:0/3667488747 >> v2:172.21.15.165:3300/0 conn(0x7fbcb815d5b0 0x7fbcb815d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:06.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.998+0000 7fbcbd7c4700 1 -- 172.21.15.1:0/3667488747 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb80cf7f0 con 0x7fbcb8099a40 2024-03-18T20:26:06.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.998+0000 7fbcbd7c4700 1 --2- 172.21.15.1:0/3667488747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcb8099a40 0x7fbcb8099ed0 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8072080 tx=0x7fbca8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27769b4119bbcf91 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:06.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.998+0000 7fbcbcfc3700 1 -- 172.21.15.1:0/3667488747 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca800b860 con 0x7fbcb8099a40 2024-03-18T20:26:06.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.999+0000 7fbcbcfc3700 1 -- 172.21.15.1:0/3667488747 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca800b9c0 con 0x7fbcb8099a40 2024-03-18T20:26:06.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.999+0000 7fbcbcfc3700 1 -- 172.21.15.1:0/3667488747 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca800bd50 con 0x7fbcb8099a40 2024-03-18T20:26:06.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.999+0000 7fbcc0229700 1 -- 172.21.15.1:0/3667488747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcb8099a40 msgr2=0x7fbcb8099ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:06.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.999+0000 7fbcc0229700 1 --2- 172.21.15.1:0/3667488747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcb8099a40 0x7fbcb8099ed0 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8072080 tx=0x7fbca8009c00 comp rx=0 tx=0).stop 2024-03-18T20:26:06.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.999+0000 7fbcc0229700 1 -- 172.21.15.1:0/3667488747 shutdown_connections 2024-03-18T20:26:06.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.999+0000 7fbcc0229700 1 --2- 172.21.15.1:0/3667488747 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbcb809a410 0x7fbcb806ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:06.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.999+0000 7fbcc0229700 1 --2- 172.21.15.1:0/3667488747 >> v2:172.21.15.165:3300/0 conn(0x7fbcb815d5b0 0x7fbcb815d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:06.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.999+0000 7fbcc0229700 1 --2- 172.21.15.1:0/3667488747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcb8099a40 0x7fbcb8099ed0 unknown :-1 s=CLOSED pgs=2706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:06.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.999+0000 7fbcc0229700 1 -- 172.21.15.1:0/3667488747 >> 172.21.15.1:0/3667488747 conn(0x7fbcb80b2bd0 msgr2=0x7fbcb80b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:06.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.999+0000 7fbcc0229700 1 -- 172.21.15.1:0/3667488747 shutdown_connections 2024-03-18T20:26:06.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:05.999+0000 7fbcc0229700 1 -- 172.21.15.1:0/3667488747 wait complete. 2024-03-18T20:26:06.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.000+0000 7fbcc0229700 1 Processor -- start 2024-03-18T20:26:06.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.000+0000 7fbcc0229700 1 -- start start 2024-03-18T20:26:06.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.001+0000 7fbcc0229700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcb8099a40 0x7fbcb8153630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:06.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.001+0000 7fbcc0229700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbcb809a410 0x7fbcb80ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:06.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.001+0000 7fbcc0229700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbcb815d5b0 0x7fbcb80caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:06.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.001+0000 7fbcc0229700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb80c8500 con 0x7fbcb8099a40 2024-03-18T20:26:06.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.001+0000 7fbcc0229700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb8153b70 con 0x7fbcb815d5b0 2024-03-18T20:26:06.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.001+0000 7fbcc0229700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb80c81b0 con 0x7fbcb809a410 2024-03-18T20:26:06.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.001+0000 7fbcbdfc5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcb8099a40 0x7fbcb8153630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:06.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.001+0000 7fbcbdfc5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcb8099a40 0x7fbcb8153630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43262/0 (socket says 172.21.15.1:43262) 2024-03-18T20:26:06.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.001+0000 7fbcbdfc5700 1 -- 172.21.15.1:0/2269412908 learned_addr learned my addr 172.21.15.1:0/2269412908 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:06.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.001+0000 7fbcbd7c4700 1 --2- 172.21.15.1:0/2269412908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbcb809a410 0x7fbcb80ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:06.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.001+0000 7fbcbe7c6700 1 --2- 172.21.15.1:0/2269412908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbcb815d5b0 0x7fbcb80caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:06.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.002+0000 7fbcbdfc5700 1 -- 172.21.15.1:0/2269412908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbcb809a410 msgr2=0x7fbcb80ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:06.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.002+0000 7fbcbdfc5700 1 --2- 172.21.15.1:0/2269412908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbcb809a410 0x7fbcb80ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:06.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.002+0000 7fbcbdfc5700 1 -- 172.21.15.1:0/2269412908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbcb815d5b0 msgr2=0x7fbcb80caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:06.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.002+0000 7fbcbdfc5700 1 --2- 172.21.15.1:0/2269412908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbcb815d5b0 0x7fbcb80caae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:06.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.002+0000 7fbcbdfc5700 1 -- 172.21.15.1:0/2269412908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca8009550 con 0x7fbcb8099a40 2024-03-18T20:26:06.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.002+0000 7fbcbdfc5700 1 --2- 172.21.15.1:0/2269412908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcb8099a40 0x7fbcb8153630 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7fbcb400e840 tx=0x7fbcb400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:06.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.002+0000 7fbcae7fc700 1 -- 172.21.15.1:0/2269412908 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb4013610 con 0x7fbcb8099a40 2024-03-18T20:26:06.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.002+0000 7fbcae7fc700 1 -- 172.21.15.1:0/2269412908 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb4013770 con 0x7fbcb8099a40 2024-03-18T20:26:06.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.002+0000 7fbcc0229700 1 -- 172.21.15.1:0/2269412908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb80b4a30 con 0x7fbcb8099a40 2024-03-18T20:26:06.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.002+0000 7fbcae7fc700 1 -- 172.21.15.1:0/2269412908 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb4013b20 con 0x7fbcb8099a40 2024-03-18T20:26:06.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.002+0000 7fbcc0229700 1 -- 172.21.15.1:0/2269412908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb80b4ff0 con 0x7fbcb8099a40 2024-03-18T20:26:06.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.003+0000 7fbcc0229700 1 -- 172.21.15.1:0/2269412908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb80c8770 con 0x7fbcb8099a40 2024-03-18T20:26:06.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.003+0000 7fbcae7fc700 1 -- 172.21.15.1:0/2269412908 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 144) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fbcb4013c80 con 0x7fbcb8099a40 2024-03-18T20:26:06.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.004+0000 7fbcae7fc700 1 --2- 172.21.15.1:0/2269412908 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbc9c06aae0 0x7fbc9c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:06.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.004+0000 7fbcae7fc700 1 -- 172.21.15.1:0/2269412908 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 7724+0+0 (secure 0 0 0) 0x7fbcb40892b0 con 0x7fbcb8099a40 2024-03-18T20:26:06.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.004+0000 7fbcbd7c4700 1 --2- 172.21.15.1:0/2269412908 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbc9c06aae0 0x7fbc9c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:06.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.004+0000 7fbcbd7c4700 1 --2- 172.21.15.1:0/2269412908 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbc9c06aae0 0x7fbc9c06cfa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbca8000c00 tx=0x7fbca8008030 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:06.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.008+0000 7fbcae7fc700 1 -- 172.21.15.1:0/2269412908 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbcb4053f70 con 0x7fbcb8099a40 2024-03-18T20:26:06.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.258+0000 7fbcc0229700 1 -- 172.21.15.1:0/2269412908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbcb80676a0 con 0x7fbcb8099a40 2024-03-18T20:26:06.263 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-18T20:26:06.259+0000 7fbdbeb46700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T20:26:06.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.269+0000 7fbcae7fc700 1 -- 172.21.15.1:0/2269412908 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v111) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbcb4057590 con 0x7fbcb8099a40 2024-03-18T20:26:06.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.271+0000 7fbcc0229700 1 -- 172.21.15.1:0/2269412908 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbc9c06aae0 msgr2=0x7fbc9c06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:06.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.271+0000 7fbcc0229700 1 --2- 172.21.15.1:0/2269412908 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbc9c06aae0 0x7fbc9c06cfa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbca8000c00 tx=0x7fbca8008030 comp rx=0 tx=0).stop 2024-03-18T20:26:06.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.271+0000 7fbcc0229700 1 -- 172.21.15.1:0/2269412908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcb8099a40 msgr2=0x7fbcb8153630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:06.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.271+0000 7fbcc0229700 1 --2- 172.21.15.1:0/2269412908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcb8099a40 0x7fbcb8153630 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7fbcb400e840 tx=0x7fbcb400eb50 comp rx=0 tx=0).stop 2024-03-18T20:26:06.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.271+0000 7fbcc0229700 1 -- 172.21.15.1:0/2269412908 shutdown_connections 2024-03-18T20:26:06.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.271+0000 7fbcc0229700 1 --2- 172.21.15.1:0/2269412908 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbc9c06aae0 0x7fbc9c06cfa0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:06.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.271+0000 7fbcc0229700 1 --2- 172.21.15.1:0/2269412908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbcb815d5b0 0x7fbcb80caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:06.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.271+0000 7fbcc0229700 1 --2- 172.21.15.1:0/2269412908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbcb8099a40 0x7fbcb8153630 unknown :-1 s=CLOSED pgs=2707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:06.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.271+0000 7fbcc0229700 1 --2- 172.21.15.1:0/2269412908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbcb809a410 0x7fbcb80ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:06.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.271+0000 7fbcc0229700 1 -- 172.21.15.1:0/2269412908 >> 172.21.15.1:0/2269412908 conn(0x7fbcb80b2bd0 msgr2=0x7fbcb806f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:06.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.271+0000 7fbcc0229700 1 -- 172.21.15.1:0/2269412908 shutdown_connections 2024-03-18T20:26:06.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.271+0000 7fbcc0229700 1 -- 172.21.15.1:0/2269412908 wait complete. 2024-03-18T20:26:06.277 INFO:tasks.ceph.mds.d.smithi165.stderr: -3869> 2024-03-18T20:25:23.216+0000 7f52f3512700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T20:26:06.278 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T20:26:06.288 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T20:26:06.307 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-03-18T20:26:06.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.410+0000 7f2c4e44d700 1 Processor -- start 2024-03-18T20:26:06.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.410+0000 7f2c4e44d700 1 -- start start 2024-03-18T20:26:06.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.411+0000 7f2c4e44d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2c4815eba0 0x7f2c480cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:06.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.411+0000 7f2c4e44d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c480b0e20 0x7f2c480b12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:06.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.411+0000 7f2c4e44d700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c4815ef80 con 0x7f2c480b0500 2024-03-18T20:26:06.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.411+0000 7f2c4e44d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c480cc9a0 con 0x7f2c4815eba0 2024-03-18T20:26:06.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.411+0000 7f2c4e44d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c480ccc80 con 0x7f2c480b0e20 2024-03-18T20:26:06.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.411+0000 7f2c47fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2c4815eba0 0x7f2c480cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:06.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.411+0000 7f2c47fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2c4815eba0 0x7f2c480cf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43278/0 (socket says 172.21.15.1:43278) 2024-03-18T20:26:06.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.411+0000 7f2c47fff700 1 -- 172.21.15.1:0/196074929 learned_addr learned my addr 172.21.15.1:0/196074929 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:06.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.411+0000 7f2c4c9ea700 1 --2- 172.21.15.1:0/196074929 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c480b0e20 0x7f2c480b12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:06.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.411+0000 7f2c47fff700 1 -- 172.21.15.1:0/196074929 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c480b0e20 msgr2=0x7f2c480b12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:06.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.411+0000 7f2c47fff700 1 --2- 172.21.15.1:0/196074929 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c480b0e20 0x7f2c480b12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:06.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.411+0000 7f2c47fff700 1 -- 172.21.15.1:0/196074929 >> v1:172.21.15.165:6789/0 conn(0x7f2c480b0500 legacy=0x7f2c480cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:06.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.411+0000 7f2c47fff700 1 -- 172.21.15.1:0/196074929 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c480ccf60 con 0x7f2c4815eba0 2024-03-18T20:26:06.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.412+0000 7f2c47fff700 1 --2- 172.21.15.1:0/196074929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4815eba0 0x7f2c480cf370 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f2c380098f0 tx=0x7f2c38009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=825f2308de6fcb3a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:06.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.412+0000 7f2c46ffd700 1 -- 172.21.15.1:0/196074929 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c38011830 con 0x7f2c4815eba0 2024-03-18T20:26:06.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.412+0000 7f2c46ffd700 1 -- 172.21.15.1:0/196074929 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c38011990 con 0x7f2c4815eba0 2024-03-18T20:26:06.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.412+0000 7f2c46ffd700 1 -- 172.21.15.1:0/196074929 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c38011d20 con 0x7f2c4815eba0 2024-03-18T20:26:06.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.412+0000 7f2c4e44d700 1 -- 172.21.15.1:0/196074929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4815eba0 msgr2=0x7f2c480cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:06.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.412+0000 7f2c4e44d700 1 --2- 172.21.15.1:0/196074929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4815eba0 0x7f2c480cf370 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f2c380098f0 tx=0x7f2c38009c00 comp rx=0 tx=0).stop 2024-03-18T20:26:06.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.412+0000 7f2c4e44d700 1 -- 172.21.15.1:0/196074929 shutdown_connections 2024-03-18T20:26:06.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.412+0000 7f2c4e44d700 1 --2- 172.21.15.1:0/196074929 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c480b0e20 0x7f2c480b12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:06.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.412+0000 7f2c4e44d700 1 --2- 172.21.15.1:0/196074929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c4815eba0 0x7f2c480cf370 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:06.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.412+0000 7f2c4e44d700 1 -- 172.21.15.1:0/196074929 >> 172.21.15.1:0/196074929 conn(0x7f2c480b2bd0 msgr2=0x7f2c480b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:06.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.412+0000 7f2c4e44d700 1 -- 172.21.15.1:0/196074929 shutdown_connections 2024-03-18T20:26:06.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.412+0000 7f2c4e44d700 1 -- 172.21.15.1:0/196074929 wait complete. 2024-03-18T20:26:06.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.413+0000 7f2c4e44d700 1 Processor -- start 2024-03-18T20:26:06.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.414+0000 7f2c4e44d700 1 -- start start 2024-03-18T20:26:06.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.414+0000 7f2c4e44d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c480b0500 0x7f2c481f5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:06.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.414+0000 7f2c4e44d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c480b0e20 0x7f2c481598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:06.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.414+0000 7f2c4e44d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c4815eba0 0x7f2c480c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.414+0000 7f2c4e44d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c481f55d0 con 0x7f2c480b0e20 2024-03-18T20:26:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.414+0000 7f2c4e44d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c480c81b0 con 0x7f2c4815eba0 2024-03-18T20:26:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.414+0000 7f2c4e44d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c481f4c00 con 0x7f2c480b0500 2024-03-18T20:26:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.415+0000 7f2c477fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c480b0e20 0x7f2c481598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:06.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.415+0000 7f2c477fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c480b0e20 0x7f2c481598b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43284/0 (socket says 172.21.15.1:43284) 2024-03-18T20:26:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.415+0000 7f2c477fe700 1 -- 172.21.15.1:0/741655049 learned_addr learned my addr 172.21.15.1:0/741655049 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.415+0000 7f2c4c9ea700 1 --2- 172.21.15.1:0/741655049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c4815eba0 0x7f2c480c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.415+0000 7f2c47fff700 1 --2- 172.21.15.1:0/741655049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c480b0500 0x7f2c481f5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.415+0000 7f2c477fe700 1 -- 172.21.15.1:0/741655049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c480b0500 msgr2=0x7f2c481f5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.415+0000 7f2c477fe700 1 --2- 172.21.15.1:0/741655049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c480b0500 0x7f2c481f5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:06.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.415+0000 7f2c477fe700 1 -- 172.21.15.1:0/741655049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c4815eba0 msgr2=0x7f2c480c8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.415+0000 7f2c477fe700 1 --2- 172.21.15.1:0/741655049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c4815eba0 0x7f2c480c8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.415+0000 7f2c477fe700 1 -- 172.21.15.1:0/741655049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c38009550 con 0x7f2c480b0e20 2024-03-18T20:26:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.415+0000 7f2c47fff700 1 --2- 172.21.15.1:0/741655049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c480b0500 0x7f2c481f5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.415+0000 7f2c477fe700 1 --2- 172.21.15.1:0/741655049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c480b0e20 0x7f2c481598b0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c00c4e0 tx=0x7f2c3c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.415+0000 7f2c44ff9700 1 -- 172.21.15.1:0/741655049 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c0052d0 con 0x7f2c480b0e20 2024-03-18T20:26:06.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.415+0000 7f2c44ff9700 1 -- 172.21.15.1:0/741655049 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3c005430 con 0x7f2c480b0e20 2024-03-18T20:26:06.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.415+0000 7f2c4e44d700 1 -- 172.21.15.1:0/741655049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c481f4610 con 0x7f2c480b0e20 2024-03-18T20:26:06.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.415+0000 7f2c44ff9700 1 -- 172.21.15.1:0/741655049 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c0038c0 con 0x7f2c480b0e20 2024-03-18T20:26:06.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.415+0000 7f2c4e44d700 1 -- 172.21.15.1:0/741655049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c480c86e0 con 0x7f2c480b0e20 2024-03-18T20:26:06.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.416+0000 7f2c4e44d700 1 -- 172.21.15.1:0/741655049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c480676a0 con 0x7f2c480b0e20 2024-03-18T20:26:06.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.417+0000 7f2c44ff9700 1 -- 172.21.15.1:0/741655049 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 144) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f2c3c003a20 con 0x7f2c480b0e20 2024-03-18T20:26:06.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.417+0000 7f2c44ff9700 1 --2- 172.21.15.1:0/741655049 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2c3406aae0 0x7f2c3406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:06.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.417+0000 7f2c44ff9700 1 -- 172.21.15.1:0/741655049 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 7810+0+0 (secure 0 0 0) 0x7f2c3c088c20 con 0x7f2c480b0e20 2024-03-18T20:26:06.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.417+0000 7f2c47fff700 1 --2- 172.21.15.1:0/741655049 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2c3406aae0 0x7f2c3406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:06.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.417+0000 7f2c47fff700 1 --2- 172.21.15.1:0/741655049 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2c3406aae0 0x7f2c3406cfa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2c38014040 tx=0x7f2c38008030 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:06.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.422+0000 7f2c44ff9700 1 -- 172.21.15.1:0/741655049 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2c3c053900 con 0x7f2c480b0e20 2024-03-18T20:26:06.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.674+0000 7f2c4e44d700 1 -- 172.21.15.1:0/741655049 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2c4815d6b0 con 0x7f2c480b0e20 2024-03-18T20:26:06.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:06.864+0000 7f2c44ff9700 1 -- 172.21.15.1:0/741655049 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 145) v1 ==== 89285+0+0 (secure 0 0 0) 0x7f2c3c053cd0 con 0x7f2c480b0e20 2024-03-18T20:26:07.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.271+0000 7f2c44ff9700 1 -- 172.21.15.1:0/741655049 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v112) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2c3c071fe0 con 0x7f2c480b0e20 2024-03-18T20:26:07.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.274+0000 7f2c4e44d700 1 -- 172.21.15.1:0/741655049 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2c3406aae0 msgr2=0x7f2c3406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:07.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.274+0000 7f2c4e44d700 1 --2- 172.21.15.1:0/741655049 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2c3406aae0 0x7f2c3406cfa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2c38014040 tx=0x7f2c38008030 comp rx=0 tx=0).stop 2024-03-18T20:26:07.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.274+0000 7f2c4e44d700 1 -- 172.21.15.1:0/741655049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c480b0e20 msgr2=0x7f2c481598b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:07.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.274+0000 7f2c4e44d700 1 --2- 172.21.15.1:0/741655049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c480b0e20 0x7f2c481598b0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c00c4e0 tx=0x7f2c3c00c8a0 comp rx=0 tx=0).stop 2024-03-18T20:26:07.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.274+0000 7f2c4e44d700 1 -- 172.21.15.1:0/741655049 shutdown_connections 2024-03-18T20:26:07.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.274+0000 7f2c4e44d700 1 --2- 172.21.15.1:0/741655049 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2c3406aae0 0x7f2c3406cfa0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:07.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.274+0000 7f2c4e44d700 1 --2- 172.21.15.1:0/741655049 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2c480b0500 0x7f2c481f5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:07.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.274+0000 7f2c4e44d700 1 --2- 172.21.15.1:0/741655049 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c4815eba0 0x7f2c480c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:07.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.274+0000 7f2c4e44d700 1 --2- 172.21.15.1:0/741655049 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c480b0e20 0x7f2c481598b0 unknown :-1 s=CLOSED pgs=2709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:07.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.274+0000 7f2c4e44d700 1 -- 172.21.15.1:0/741655049 >> 172.21.15.1:0/741655049 conn(0x7f2c480b2bd0 msgr2=0x7f2c4815b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:07.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.274+0000 7f2c4e44d700 1 -- 172.21.15.1:0/741655049 shutdown_connections 2024-03-18T20:26:07.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.274+0000 7f2c4e44d700 1 -- 172.21.15.1:0/741655049 wait complete. 2024-03-18T20:26:07.295 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T20:26:07.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.425+0000 7ff8a381f700 1 Processor -- start 2024-03-18T20:26:07.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.425+0000 7ff8a381f700 1 -- start start 2024-03-18T20:26:07.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.426+0000 7ff8a381f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff89c0d0040 0x7ff89c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:07.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.426+0000 7ff8a381f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89c15dc90 con 0x7ff89c0d1170 2024-03-18T20:26:07.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.426+0000 7ff8a381f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89c15ded0 con 0x7ff89c15d8b0 2024-03-18T20:26:07.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.426+0000 7ff8a381f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff89c15ec90 con 0x7ff89c0d0040 2024-03-18T20:26:07.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.426+0000 7ff8a0dba700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff89c0d0040 0x7ff89c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.426+0000 7ff8a15bb700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7ff89c15d8b0 0x7ff89c0cf930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:53480/0 (socket says 172.21.15.1:53480) 2024-03-18T20:26:07.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.426+0000 7ff8a15bb700 1 -- 172.21.15.1:0/3990997487 learned_addr learned my addr 172.21.15.1:0/3990997487 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.427+0000 7ff8a0dba700 1 -- 172.21.15.1:0/3990997487 >> v1:172.21.15.165:6789/0 conn(0x7ff89c15d8b0 legacy=0x7ff89c0cf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.427+0000 7ff8a0dba700 1 -- 172.21.15.1:0/3990997487 >> v1:172.21.15.1:6789/0 conn(0x7ff89c0d1170 legacy=0x7ff89c0cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.427+0000 7ff8a0dba700 1 -- 172.21.15.1:0/3990997487 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89c15e1b0 con 0x7ff89c0d0040 2024-03-18T20:26:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.427+0000 7ff8a0dba700 1 --2- 172.21.15.1:0/3990997487 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff89c0d0040 0x7ff89c0d3c30 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7ff898009840 tx=0x7ff898009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb2e09d14a582a50 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.427+0000 7ff893fff700 1 -- 172.21.15.1:0/3990997487 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff898011830 con 0x7ff89c0d0040 2024-03-18T20:26:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.427+0000 7ff893fff700 1 -- 172.21.15.1:0/3990997487 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff898011990 con 0x7ff89c0d0040 2024-03-18T20:26:07.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.427+0000 7ff893fff700 1 -- 172.21.15.1:0/3990997487 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff898011d20 con 0x7ff89c0d0040 2024-03-18T20:26:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.428+0000 7ff8a381f700 1 -- 172.21.15.1:0/3990997487 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff89c0d0040 msgr2=0x7ff89c0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.428+0000 7ff8a381f700 1 --2- 172.21.15.1:0/3990997487 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff89c0d0040 0x7ff89c0d3c30 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7ff898009840 tx=0x7ff898009b50 comp rx=0 tx=0).stop 2024-03-18T20:26:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.428+0000 7ff8a381f700 1 -- 172.21.15.1:0/3990997487 shutdown_connections 2024-03-18T20:26:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.428+0000 7ff8a381f700 1 --2- 172.21.15.1:0/3990997487 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff89c0d0040 0x7ff89c0d3c30 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.428+0000 7ff8a381f700 1 -- 172.21.15.1:0/3990997487 >> 172.21.15.1:0/3990997487 conn(0x7ff89c09c640 msgr2=0x7ff89c0d3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.428+0000 7ff8a381f700 1 -- 172.21.15.1:0/3990997487 shutdown_connections 2024-03-18T20:26:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.428+0000 7ff8a381f700 1 -- 172.21.15.1:0/3990997487 wait complete. 2024-03-18T20:26:07.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.429+0000 7ff8a381f700 1 Processor -- start 2024-03-18T20:26:07.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.429+0000 7ff8a381f700 1 -- start start 2024-03-18T20:26:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.430+0000 7ff8a381f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff89c0d0040 0x7ff89c161d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.430+0000 7ff8a381f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff89c0d1170 0x7ff89c15b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.430+0000 7ff8a381f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff89c15d8b0 0x7ff89c0cc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.430+0000 7ff8a381f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89c15edd0 con 0x7ff89c15d8b0 2024-03-18T20:26:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.430+0000 7ff8a381f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89c0b5ee0 con 0x7ff89c0d1170 2024-03-18T20:26:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.430+0000 7ff8a381f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff89c0c9000 con 0x7ff89c0d0040 2024-03-18T20:26:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.430+0000 7ff8a1dbc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff89c15d8b0 0x7ff89c0cc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.431+0000 7ff8a1dbc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff89c15d8b0 0x7ff89c0cc5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43300/0 (socket says 172.21.15.1:43300) 2024-03-18T20:26:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.431+0000 7ff8a1dbc700 1 -- 172.21.15.1:0/1524977752 learned_addr learned my addr 172.21.15.1:0/1524977752 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.431+0000 7ff8a0dba700 1 --2- 172.21.15.1:0/1524977752 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff89c0d1170 0x7ff89c15b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.431+0000 7ff8a1dbc700 1 -- 172.21.15.1:0/1524977752 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff89c0d0040 msgr2=0x7ff89c161d30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T20:26:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.431+0000 7ff8a1dbc700 1 --2- 172.21.15.1:0/1524977752 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff89c0d0040 0x7ff89c161d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.431+0000 7ff8a1dbc700 1 -- 172.21.15.1:0/1524977752 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff89c0d1170 msgr2=0x7ff89c15b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:07.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.431+0000 7ff8a1dbc700 1 --2- 172.21.15.1:0/1524977752 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff89c0d1170 0x7ff89c15b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.431+0000 7ff8a1dbc700 1 -- 172.21.15.1:0/1524977752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8980094a0 con 0x7ff89c15d8b0 2024-03-18T20:26:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.431+0000 7ff8a0dba700 1 --2- 172.21.15.1:0/1524977752 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff89c0d1170 0x7ff89c15b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.431+0000 7ff8a1dbc700 1 --2- 172.21.15.1:0/1524977752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff89c15d8b0 0x7ff89c0cc5a0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7ff89400cc50 tx=0x7ff89400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.431+0000 7ff891ffb700 1 -- 172.21.15.1:0/1524977752 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89400ad80 con 0x7ff89c15d8b0 2024-03-18T20:26:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.431+0000 7ff8a381f700 1 -- 172.21.15.1:0/1524977752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff89c0b6130 con 0x7ff89c15d8b0 2024-03-18T20:26:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.431+0000 7ff891ffb700 1 -- 172.21.15.1:0/1524977752 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff89400ce50 con 0x7ff89c15d8b0 2024-03-18T20:26:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.431+0000 7ff891ffb700 1 -- 172.21.15.1:0/1524977752 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff894003c40 con 0x7ff89c15d8b0 2024-03-18T20:26:07.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.431+0000 7ff8a381f700 1 -- 172.21.15.1:0/1524977752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff89c0b66f0 con 0x7ff89c15d8b0 2024-03-18T20:26:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.432+0000 7ff8a381f700 1 -- 172.21.15.1:0/1524977752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff89c073350 con 0x7ff89c15d8b0 2024-03-18T20:26:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.433+0000 7ff891ffb700 1 -- 172.21.15.1:0/1524977752 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 145) v1 ==== 89285+0+0 (secure 0 0 0) 0x7ff8940050d0 con 0x7ff89c15d8b0 2024-03-18T20:26:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.433+0000 7ff891ffb700 1 --2- 172.21.15.1:0/1524977752 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff88006ae40 0x7ff88006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.433+0000 7ff891ffb700 1 -- 172.21.15.1:0/1524977752 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 7810+0+0 (secure 0 0 0) 0x7ff894089e70 con 0x7ff89c15d8b0 2024-03-18T20:26:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.433+0000 7ff8a15bb700 1 --2- 172.21.15.1:0/1524977752 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff88006ae40 0x7ff88006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:07.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.433+0000 7ff8a15bb700 1 --2- 172.21.15.1:0/1524977752 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff88006ae40 0x7ff88006d300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff88c001560 tx=0x7ff88c007ad0 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:07.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.438+0000 7ff891ffb700 1 -- 172.21.15.1:0/1524977752 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff894054510 con 0x7ff89c15d8b0 2024-03-18T20:26:07.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:07.645+0000 7ff8a381f700 1 -- 172.21.15.1:0/1524977752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff89c15dc90 con 0x7ff89c15d8b0 2024-03-18T20:26:08.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.286+0000 7ff891ffb700 1 -- 172.21.15.1:0/1524977752 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7ff8940548e0 con 0x7ff89c15d8b0 2024-03-18T20:26:08.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.287+0000 7ff891ffb700 1 -- 172.21.15.1:0/1524977752 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v240) v1 ==== 181+0+0 (secure 0 0 0) 0x7ff8940505d0 con 0x7ff89c15d8b0 2024-03-18T20:26:08.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.291+0000 7ff8a381f700 1 -- 172.21.15.1:0/1524977752 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff88006ae40 msgr2=0x7ff88006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:08.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.291+0000 7ff8a381f700 1 --2- 172.21.15.1:0/1524977752 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff88006ae40 0x7ff88006d300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff88c001560 tx=0x7ff88c007ad0 comp rx=0 tx=0).stop 2024-03-18T20:26:08.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.291+0000 7ff8a381f700 1 -- 172.21.15.1:0/1524977752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff89c15d8b0 msgr2=0x7ff89c0cc5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:08.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.291+0000 7ff8a381f700 1 --2- 172.21.15.1:0/1524977752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff89c15d8b0 0x7ff89c0cc5a0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7ff89400cc50 tx=0x7ff89400a340 comp rx=0 tx=0).stop 2024-03-18T20:26:08.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.291+0000 7ff8a381f700 1 -- 172.21.15.1:0/1524977752 shutdown_connections 2024-03-18T20:26:08.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.291+0000 7ff8a381f700 1 --2- 172.21.15.1:0/1524977752 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff88006ae40 0x7ff88006d300 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:08.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.291+0000 7ff8a381f700 1 --2- 172.21.15.1:0/1524977752 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff89c0d0040 0x7ff89c161d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:08.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.291+0000 7ff8a381f700 1 --2- 172.21.15.1:0/1524977752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff89c15d8b0 0x7ff89c0cc5a0 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:08.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.291+0000 7ff8a381f700 1 --2- 172.21.15.1:0/1524977752 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff89c0d1170 0x7ff89c15b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:08.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.291+0000 7ff8a381f700 1 -- 172.21.15.1:0/1524977752 >> 172.21.15.1:0/1524977752 conn(0x7ff89c09c640 msgr2=0x7ff89c0d3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:08.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.291+0000 7ff8a381f700 1 -- 172.21.15.1:0/1524977752 shutdown_connections 2024-03-18T20:26:08.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.291+0000 7ff8a381f700 1 -- 172.21.15.1:0/1524977752 wait complete. 2024-03-18T20:26:08.297 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-18T20:26:08.311 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T20:26:08.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.441+0000 7fa8eb73f700 1 Processor -- start 2024-03-18T20:26:08.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.441+0000 7fa8eb73f700 1 -- start start 2024-03-18T20:26:08.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.442+0000 7fa8eb73f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa8e415d590 0x7fa8e40b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.442+0000 7fa8eb73f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8e406a750 0x7fa8e406abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.442+0000 7fa8eb73f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e415d970 con 0x7fa8e406b110 2024-03-18T20:26:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.442+0000 7fa8eb73f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8e4072320 con 0x7fa8e415d590 2024-03-18T20:26:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.442+0000 7fa8eb73f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e4072600 con 0x7fa8e406a750 2024-03-18T20:26:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.442+0000 7fa8e94db700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa8e415d590 0x7fa8e40b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.442+0000 7fa8e9cdc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa8e406b110 0x7fa8e406ee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57752/0 (socket says 172.21.15.1:57752) 2024-03-18T20:26:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.442+0000 7fa8e9cdc700 1 -- 172.21.15.1:0/4072596349 learned_addr learned my addr 172.21.15.1:0/4072596349 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.442+0000 7fa8e94db700 1 --2- 172.21.15.1:0/4072596349 >> v2:172.21.15.1:3300/0 conn(0x7fa8e415d590 0x7fa8e40b3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43304/0 (socket says 172.21.15.1:43304) 2024-03-18T20:26:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.442+0000 7fa8e8cda700 1 --2- 172.21.15.1:0/4072596349 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8e406a750 0x7fa8e406abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.442+0000 7fa8e94db700 1 -- 172.21.15.1:0/4072596349 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8e406a750 msgr2=0x7fa8e406abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.442+0000 7fa8e94db700 1 --2- 172.21.15.1:0/4072596349 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8e406a750 0x7fa8e406abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.442+0000 7fa8e94db700 1 -- 172.21.15.1:0/4072596349 >> v1:172.21.15.1:6789/0 conn(0x7fa8e406b110 legacy=0x7fa8e406ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.442+0000 7fa8e94db700 1 -- 172.21.15.1:0/4072596349 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e40b30b0 con 0x7fa8e415d590 2024-03-18T20:26:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.442+0000 7fa8dbfff700 1 -- 172.21.15.1:0/4072596349 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174265955 0 0) 0x7fa8e415d970 con 0x7fa8e406b110 2024-03-18T20:26:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.442+0000 7fa8e8cda700 1 --2- 172.21.15.1:0/4072596349 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8e406a750 0x7fa8e406abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.442+0000 7fa8e94db700 1 --2- 172.21.15.1:0/4072596349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8e415d590 0x7fa8e40b3bc0 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0009840 tx=0x7fa8e0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a46adadf34e6174d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.442+0000 7fa8dbfff700 1 -- 172.21.15.1:0/4072596349 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0011830 con 0x7fa8e415d590 2024-03-18T20:26:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.443+0000 7fa8dbfff700 1 -- 172.21.15.1:0/4072596349 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e0011990 con 0x7fa8e415d590 2024-03-18T20:26:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.443+0000 7fa8dbfff700 1 -- 172.21.15.1:0/4072596349 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0011d20 con 0x7fa8e415d590 2024-03-18T20:26:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.443+0000 7fa8eb73f700 1 -- 172.21.15.1:0/4072596349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8e415d590 msgr2=0x7fa8e40b3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.443+0000 7fa8eb73f700 1 --2- 172.21.15.1:0/4072596349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8e415d590 0x7fa8e40b3bc0 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0009840 tx=0x7fa8e0009b50 comp rx=0 tx=0).stop 2024-03-18T20:26:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.443+0000 7fa8eb73f700 1 -- 172.21.15.1:0/4072596349 shutdown_connections 2024-03-18T20:26:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.443+0000 7fa8eb73f700 1 --2- 172.21.15.1:0/4072596349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8e415d590 0x7fa8e40b3bc0 unknown :-1 s=CLOSED pgs=2713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.443+0000 7fa8eb73f700 1 --2- 172.21.15.1:0/4072596349 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8e406a750 0x7fa8e406abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:08.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.443+0000 7fa8eb73f700 1 -- 172.21.15.1:0/4072596349 >> 172.21.15.1:0/4072596349 conn(0x7fa8e40d00c0 msgr2=0x7fa8e40d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:08.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.443+0000 7fa8eb73f700 1 -- 172.21.15.1:0/4072596349 shutdown_connections 2024-03-18T20:26:08.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.443+0000 7fa8eb73f700 1 -- 172.21.15.1:0/4072596349 wait complete. 2024-03-18T20:26:08.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.444+0000 7fa8eb73f700 1 Processor -- start 2024-03-18T20:26:08.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.444+0000 7fa8eb73f700 1 -- start start 2024-03-18T20:26:08.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.445+0000 7fa8eb73f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8e406a750 0x7fa8e4154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:08.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.445+0000 7fa8eb73f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8e406b110 0x7fa8e41f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:08.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.445+0000 7fa8eb73f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8e415d590 0x7fa8e415bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:08.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.445+0000 7fa8eb73f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e4155450 con 0x7fa8e406b110 2024-03-18T20:26:08.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.445+0000 7fa8eb73f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e41589c0 con 0x7fa8e406a750 2024-03-18T20:26:08.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.445+0000 7fa8eb73f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e4158ca0 con 0x7fa8e415d590 2024-03-18T20:26:08.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.445+0000 7fa8e8cda700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8e406b110 0x7fa8e41f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:08.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.445+0000 7fa8e8cda700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8e406b110 0x7fa8e41f2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43314/0 (socket says 172.21.15.1:43314) 2024-03-18T20:26:08.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.445+0000 7fa8e8cda700 1 -- 172.21.15.1:0/2458444540 learned_addr learned my addr 172.21.15.1:0/2458444540 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:08.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.445+0000 7fa8e9cdc700 1 --2- 172.21.15.1:0/2458444540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8e415d590 0x7fa8e415bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:08.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.445+0000 7fa8e94db700 1 --2- 172.21.15.1:0/2458444540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8e406a750 0x7fa8e4154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:08.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.446+0000 7fa8e8cda700 1 -- 172.21.15.1:0/2458444540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8e415d590 msgr2=0x7fa8e415bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:08.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.446+0000 7fa8e8cda700 1 --2- 172.21.15.1:0/2458444540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8e415d590 0x7fa8e415bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:08.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.446+0000 7fa8e8cda700 1 -- 172.21.15.1:0/2458444540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8e406a750 msgr2=0x7fa8e4154f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:08.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.446+0000 7fa8e8cda700 1 --2- 172.21.15.1:0/2458444540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8e406a750 0x7fa8e4154f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:08.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.446+0000 7fa8e8cda700 1 -- 172.21.15.1:0/2458444540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e00094a0 con 0x7fa8e406b110 2024-03-18T20:26:08.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.446+0000 7fa8e8cda700 1 --2- 172.21.15.1:0/2458444540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8e406b110 0x7fa8e41f2da0 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7fa8d400e810 tx=0x7fa8d400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:08.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.446+0000 7fa8d9ffb700 1 -- 172.21.15.1:0/2458444540 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d400c070 con 0x7fa8e406b110 2024-03-18T20:26:08.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.446+0000 7fa8d9ffb700 1 -- 172.21.15.1:0/2458444540 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d4004230 con 0x7fa8e406b110 2024-03-18T20:26:08.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.446+0000 7fa8eb73f700 1 -- 172.21.15.1:0/2458444540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e4158f80 con 0x7fa8e406b110 2024-03-18T20:26:08.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.446+0000 7fa8d9ffb700 1 -- 172.21.15.1:0/2458444540 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d40036a0 con 0x7fa8e406b110 2024-03-18T20:26:08.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.446+0000 7fa8eb73f700 1 -- 172.21.15.1:0/2458444540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e4159540 con 0x7fa8e406b110 2024-03-18T20:26:08.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.447+0000 7fa8eb73f700 1 -- 172.21.15.1:0/2458444540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e4159700 con 0x7fa8e406b110 2024-03-18T20:26:08.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.448+0000 7fa8d9ffb700 1 -- 172.21.15.1:0/2458444540 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fa8d4003800 con 0x7fa8e406b110 2024-03-18T20:26:08.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.448+0000 7fa8d9ffb700 1 --2- 172.21.15.1:0/2458444540 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8c806aed0 0x7fa8c806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:08.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.448+0000 7fa8d9ffb700 1 -- 172.21.15.1:0/2458444540 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 7372+0+0 (secure 0 0 0) 0x7fa8d4089ab0 con 0x7fa8e406b110 2024-03-18T20:26:08.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.448+0000 7fa8e94db700 1 --2- 172.21.15.1:0/2458444540 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8c806aed0 0x7fa8c806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:08.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.448+0000 7fa8e94db700 1 --2- 172.21.15.1:0/2458444540 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8c806aed0 0x7fa8c806d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00090b0 tx=0x7fa8e001c040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:08.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.453+0000 7fa8d9ffb700 1 -- 172.21.15.1:0/2458444540 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa8d4054230 con 0x7fa8e406b110 2024-03-18T20:26:08.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:08.659+0000 7fa8eb73f700 1 -- 172.21.15.1:0/2458444540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa8e40676a0 con 0x7fa8e406b110 2024-03-18T20:26:09.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.283+0000 7fa8d9ffb700 1 -- 172.21.15.1:0/2458444540 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v241) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa8d4057850 con 0x7fa8e406b110 2024-03-18T20:26:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.286+0000 7fa8eb73f700 1 -- 172.21.15.1:0/2458444540 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8c806aed0 msgr2=0x7fa8c806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.286+0000 7fa8eb73f700 1 --2- 172.21.15.1:0/2458444540 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8c806aed0 0x7fa8c806d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00090b0 tx=0x7fa8e001c040 comp rx=0 tx=0).stop 2024-03-18T20:26:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.286+0000 7fa8eb73f700 1 -- 172.21.15.1:0/2458444540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8e406b110 msgr2=0x7fa8e41f2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.286+0000 7fa8eb73f700 1 --2- 172.21.15.1:0/2458444540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8e406b110 0x7fa8e41f2da0 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7fa8d400e810 tx=0x7fa8d400eb20 comp rx=0 tx=0).stop 2024-03-18T20:26:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.286+0000 7fa8eb73f700 1 -- 172.21.15.1:0/2458444540 shutdown_connections 2024-03-18T20:26:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.286+0000 7fa8eb73f700 1 --2- 172.21.15.1:0/2458444540 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8c806aed0 0x7fa8c806d390 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.286+0000 7fa8eb73f700 1 --2- 172.21.15.1:0/2458444540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8e415d590 0x7fa8e415bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.286+0000 7fa8eb73f700 1 --2- 172.21.15.1:0/2458444540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8e406a750 0x7fa8e4154f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.286+0000 7fa8eb73f700 1 --2- 172.21.15.1:0/2458444540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8e406b110 0x7fa8e41f2da0 unknown :-1 s=CLOSED pgs=2714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.286+0000 7fa8eb73f700 1 -- 172.21.15.1:0/2458444540 >> 172.21.15.1:0/2458444540 conn(0x7fa8e40d00c0 msgr2=0x7fa8e40aff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.286+0000 7fa8eb73f700 1 -- 172.21.15.1:0/2458444540 shutdown_connections 2024-03-18T20:26:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.286+0000 7fa8eb73f700 1 -- 172.21.15.1:0/2458444540 wait complete. 2024-03-18T20:26:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-18T20:26:09.306 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:26:09.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.432+0000 7fe4f7b98700 1 Processor -- start 2024-03-18T20:26:09.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.432+0000 7fe4f7b98700 1 -- start start 2024-03-18T20:26:09.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.433+0000 7fe4f7b98700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe4f0099a40 0x7fe4f0099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:09.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.433+0000 7fe4f7b98700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe4f00cc310 0x7fe4f00cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:09.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.433+0000 7fe4f7b98700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f009a3e0 con 0x7fe4f015d680 2024-03-18T20:26:09.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.433+0000 7fe4f7b98700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f00d0c80 con 0x7fe4f0099a40 2024-03-18T20:26:09.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.433+0000 7fe4f7b98700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f00d09a0 con 0x7fe4f00cc310 2024-03-18T20:26:09.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.433+0000 7fe4f5133700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe4f0099a40 0x7fe4f0099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:09.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.433+0000 7fe4f5133700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe4f0099a40 0x7fe4f0099ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43330/0 (socket says 172.21.15.1:43330) 2024-03-18T20:26:09.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.433+0000 7fe4f5133700 1 -- 172.21.15.1:0/1696294461 learned_addr learned my addr 172.21.15.1:0/1696294461 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:09.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.434+0000 7fe4f6135700 1 --2- 172.21.15.1:0/1696294461 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe4f00cc310 0x7fe4f00cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:09.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.434+0000 7fe4f5133700 1 -- 172.21.15.1:0/1696294461 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe4f00cc310 msgr2=0x7fe4f00cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:09.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.434+0000 7fe4f5133700 1 --2- 172.21.15.1:0/1696294461 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe4f00cc310 0x7fe4f00cc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:09.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.434+0000 7fe4f5133700 1 -- 172.21.15.1:0/1696294461 >> v1:172.21.15.165:6789/0 conn(0x7fe4f015d680 legacy=0x7fe4f00cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:09.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.434+0000 7fe4f5133700 1 -- 172.21.15.1:0/1696294461 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f00cb830 con 0x7fe4f0099a40 2024-03-18T20:26:09.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.434+0000 7fe4f5133700 1 --2- 172.21.15.1:0/1696294461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4f0099a40 0x7fe4f0099ea0 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7fe4e40098f0 tx=0x7fe4e4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7803d5d8a7b4e88 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:09.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.434+0000 7fe4f4932700 1 -- 172.21.15.1:0/1696294461 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e4004230 con 0x7fe4f0099a40 2024-03-18T20:26:09.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.434+0000 7fe4f4932700 1 -- 172.21.15.1:0/1696294461 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e4004390 con 0x7fe4f0099a40 2024-03-18T20:26:09.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.434+0000 7fe4f4932700 1 -- 172.21.15.1:0/1696294461 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e40036a0 con 0x7fe4f0099a40 2024-03-18T20:26:09.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.435+0000 7fe4f7b98700 1 -- 172.21.15.1:0/1696294461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4f0099a40 msgr2=0x7fe4f0099ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:09.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.435+0000 7fe4f7b98700 1 --2- 172.21.15.1:0/1696294461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4f0099a40 0x7fe4f0099ea0 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7fe4e40098f0 tx=0x7fe4e4009c00 comp rx=0 tx=0).stop 2024-03-18T20:26:09.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.435+0000 7fe4f7b98700 1 -- 172.21.15.1:0/1696294461 shutdown_connections 2024-03-18T20:26:09.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.435+0000 7fe4f7b98700 1 --2- 172.21.15.1:0/1696294461 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe4f00cc310 0x7fe4f00cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:09.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.435+0000 7fe4f7b98700 1 --2- 172.21.15.1:0/1696294461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4f0099a40 0x7fe4f0099ea0 unknown :-1 s=CLOSED pgs=2715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:09.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.435+0000 7fe4f7b98700 1 -- 172.21.15.1:0/1696294461 >> 172.21.15.1:0/1696294461 conn(0x7fe4f00b2ba0 msgr2=0x7fe4f00b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:09.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.435+0000 7fe4f7b98700 1 -- 172.21.15.1:0/1696294461 shutdown_connections 2024-03-18T20:26:09.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.435+0000 7fe4f7b98700 1 -- 172.21.15.1:0/1696294461 wait complete. 2024-03-18T20:26:09.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.436+0000 7fe4f7b98700 1 Processor -- start 2024-03-18T20:26:09.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.436+0000 7fe4f7b98700 1 -- start start 2024-03-18T20:26:09.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.437+0000 7fe4f7b98700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe4f0099a40 0x7fe4f0159620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:09.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.437+0000 7fe4f7b98700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4f00cc310 0x7fe4f015eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:09.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.437+0000 7fe4f7b98700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe4f015d680 0x7fe4f006f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:09.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.437+0000 7fe4f7b98700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f00c9ba0 con 0x7fe4f00cc310 2024-03-18T20:26:09.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.437+0000 7fe4f7b98700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f015f550 con 0x7fe4f0099a40 2024-03-18T20:26:09.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.437+0000 7fe4f7b98700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f00c9270 con 0x7fe4f015d680 2024-03-18T20:26:09.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.437+0000 7fe4f5133700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4f00cc310 0x7fe4f015eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:09.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.437+0000 7fe4f5133700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4f00cc310 0x7fe4f015eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43342/0 (socket says 172.21.15.1:43342) 2024-03-18T20:26:09.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.437+0000 7fe4f5133700 1 -- 172.21.15.1:0/3070679754 learned_addr learned my addr 172.21.15.1:0/3070679754 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:09.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.437+0000 7fe4f5934700 1 --2- 172.21.15.1:0/3070679754 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe4f0099a40 0x7fe4f0159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:09.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.438+0000 7fe4f6135700 1 --2- 172.21.15.1:0/3070679754 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe4f015d680 0x7fe4f006f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:09.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.438+0000 7fe4f5133700 1 -- 172.21.15.1:0/3070679754 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe4f015d680 msgr2=0x7fe4f006f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:09.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.438+0000 7fe4f5133700 1 --2- 172.21.15.1:0/3070679754 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe4f015d680 0x7fe4f006f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:09.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.438+0000 7fe4f5133700 1 -- 172.21.15.1:0/3070679754 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe4f0099a40 msgr2=0x7fe4f0159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:09.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.438+0000 7fe4f5133700 1 --2- 172.21.15.1:0/3070679754 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe4f0099a40 0x7fe4f0159620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:09.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.438+0000 7fe4f5133700 1 -- 172.21.15.1:0/3070679754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e4009550 con 0x7fe4f00cc310 2024-03-18T20:26:09.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.438+0000 7fe4f5934700 1 --2- 172.21.15.1:0/3070679754 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe4f0099a40 0x7fe4f0159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:09.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.438+0000 7fe4f5133700 1 --2- 172.21.15.1:0/3070679754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4f00cc310 0x7fe4f015eab0 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4009840 tx=0x7fe4e4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:09.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.438+0000 7fe4e27fc700 1 -- 172.21.15.1:0/3070679754 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e4003e30 con 0x7fe4f00cc310 2024-03-18T20:26:09.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.438+0000 7fe4e27fc700 1 -- 172.21.15.1:0/3070679754 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e401f070 con 0x7fe4f00cc310 2024-03-18T20:26:09.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.438+0000 7fe4e27fc700 1 -- 172.21.15.1:0/3070679754 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e400b8a0 con 0x7fe4f00cc310 2024-03-18T20:26:09.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.438+0000 7fe4f7b98700 1 -- 172.21.15.1:0/3070679754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f00c9550 con 0x7fe4f00cc310 2024-03-18T20:26:09.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.438+0000 7fe4f7b98700 1 -- 172.21.15.1:0/3070679754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f0159b60 con 0x7fe4f00cc310 2024-03-18T20:26:09.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.439+0000 7fe4f7b98700 1 -- 172.21.15.1:0/3070679754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f00676a0 con 0x7fe4f00cc310 2024-03-18T20:26:09.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.440+0000 7fe4e27fc700 1 -- 172.21.15.1:0/3070679754 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fe4e4021030 con 0x7fe4f00cc310 2024-03-18T20:26:09.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.440+0000 7fe4e27fc700 1 --2- 172.21.15.1:0/3070679754 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe4d406aed0 0x7fe4d406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:09.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.440+0000 7fe4e27fc700 1 -- 172.21.15.1:0/3070679754 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 6958+0+0 (secure 0 0 0) 0x7fe4e408b2d0 con 0x7fe4f00cc310 2024-03-18T20:26:09.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.440+0000 7fe4f5934700 1 --2- 172.21.15.1:0/3070679754 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe4d406aed0 0x7fe4d406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:09.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.440+0000 7fe4f5934700 1 --2- 172.21.15.1:0/3070679754 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe4d406aed0 0x7fe4d406d390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec009910 tx=0x7fe4ec008040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:09.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.445+0000 7fe4e27fc700 1 -- 172.21.15.1:0/3070679754 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe4e4055dd0 con 0x7fe4f00cc310 2024-03-18T20:26:09.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.649+0000 7fe4f7b98700 1 -- 172.21.15.1:0/3070679754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4f00676a0 con 0x7fe4f00cc310 2024-03-18T20:26:09.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.650+0000 7fe4e27fc700 1 -- 172.21.15.1:0/3070679754 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v241) v1 ==== 74+0+17537 (secure 0 0 0) 0x7fe4e40593f0 con 0x7fe4f00cc310 2024-03-18T20:26:09.655 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:09.655 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":241,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:26:09.280336+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:25:33.275375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:25:34.518347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.165:6834/4024828630":"2024-03-19T20:26:06.260180+0000","172.21.15.1:0/4210355381":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/1879196326":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/101306769":"2024-03-19T20:26:03.849558+0000","172.21.15.165:6832/4024828630":"2024-03-19T20:26:06.260180+0000","172.21.15.1:0/997525540":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/1622795270":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/2535232994":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2652258802":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2404324807":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2059399259":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/3657737679":"2024-03-19T20:25:52.933722+0000","172.21.15.1:6833/148356":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2685347289":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/1965726293":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/2324250327":"2024-03-19T20:25:52.933722+0000","172.21.15.1:6832/148356":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/1022040508":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/2165965129":"2024-03-19T20:25:52.933722+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:26:09.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.652+0000 7fe4f7b98700 1 -- 172.21.15.1:0/3070679754 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe4d406aed0 msgr2=0x7fe4d406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:09.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.652+0000 7fe4f7b98700 1 --2- 172.21.15.1:0/3070679754 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe4d406aed0 0x7fe4d406d390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec009910 tx=0x7fe4ec008040 comp rx=0 tx=0).stop 2024-03-18T20:26:09.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.652+0000 7fe4f7b98700 1 -- 172.21.15.1:0/3070679754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4f00cc310 msgr2=0x7fe4f015eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:09.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.652+0000 7fe4f7b98700 1 --2- 172.21.15.1:0/3070679754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4f00cc310 0x7fe4f015eab0 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4009840 tx=0x7fe4e4004750 comp rx=0 tx=0).stop 2024-03-18T20:26:09.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.652+0000 7fe4f7b98700 1 -- 172.21.15.1:0/3070679754 shutdown_connections 2024-03-18T20:26:09.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.652+0000 7fe4f7b98700 1 --2- 172.21.15.1:0/3070679754 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe4d406aed0 0x7fe4d406d390 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:09.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.652+0000 7fe4f7b98700 1 --2- 172.21.15.1:0/3070679754 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe4f015d680 0x7fe4f006f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:09.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.652+0000 7fe4f7b98700 1 --2- 172.21.15.1:0/3070679754 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe4f0099a40 0x7fe4f0159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:09.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.652+0000 7fe4f7b98700 1 --2- 172.21.15.1:0/3070679754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4f00cc310 0x7fe4f015eab0 unknown :-1 s=CLOSED pgs=2716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:09.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.652+0000 7fe4f7b98700 1 -- 172.21.15.1:0/3070679754 >> 172.21.15.1:0/3070679754 conn(0x7fe4f00b2ba0 msgr2=0x7fe4f00cd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:09.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.652+0000 7fe4f7b98700 1 -- 172.21.15.1:0/3070679754 shutdown_connections 2024-03-18T20:26:09.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.652+0000 7fe4f7b98700 1 -- 172.21.15.1:0/3070679754 wait complete. 2024-03-18T20:26:09.669 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-03-18T20:25:33.275375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '235', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'backup_fs_data', 'create_time': '2024-03-18T20:25:34.518347+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '236', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:26:09.670 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:26:09.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.804+0000 7f46a90a3700 1 Processor -- start 2024-03-18T20:26:09.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.804+0000 7f46a90a3700 1 -- start start 2024-03-18T20:26:09.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.804+0000 7f46a90a3700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f46a4158e30 0x7f46a4071500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:09.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.805+0000 7f46a90a3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f46a40cbf70 0x7f46a40cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:09.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.805+0000 7f46a90a3700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a415c500 con 0x7f46a406a750 2024-03-18T20:26:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.805+0000 7f46a90a3700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a40af8b0 con 0x7f46a40cbf70 2024-03-18T20:26:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.805+0000 7f46a90a3700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a415b8f0 con 0x7f46a4158e30 2024-03-18T20:26:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.805+0000 7f46a259c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f46a40cbf70 0x7f46a40cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.805+0000 7f46a359e700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f46a406a750 0x7f46a40cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57768/0 (socket says 172.21.15.1:57768) 2024-03-18T20:26:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.805+0000 7f46a359e700 1 -- 172.21.15.1:0/3109889000 learned_addr learned my addr 172.21.15.1:0/3109889000 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.805+0000 7f46a2d9d700 1 --2- 172.21.15.1:0/3109889000 >> v2:172.21.15.165:3300/0 conn(0x7f46a4158e30 0x7f46a4071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.805+0000 7f46a259c700 1 -- 172.21.15.1:0/3109889000 >> v2:172.21.15.165:3300/0 conn(0x7f46a4158e30 msgr2=0x7f46a4071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.805+0000 7f46a259c700 1 --2- 172.21.15.1:0/3109889000 >> v2:172.21.15.165:3300/0 conn(0x7f46a4158e30 0x7f46a4071500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.805+0000 7f46a259c700 1 -- 172.21.15.1:0/3109889000 >> v1:172.21.15.1:6789/0 conn(0x7f46a406a750 legacy=0x7f46a40cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.805+0000 7f46a259c700 1 -- 172.21.15.1:0/3109889000 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a415bbd0 con 0x7f46a40cbf70 2024-03-18T20:26:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.805+0000 7f46a1d9b700 1 -- 172.21.15.1:0/3109889000 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440894076 0 0) 0x7f46a415c500 con 0x7f46a406a750 2024-03-18T20:26:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.805+0000 7f46a2d9d700 1 --2- 172.21.15.1:0/3109889000 >> v2:172.21.15.165:3300/0 conn(0x7f46a4158e30 0x7f46a4071500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.805+0000 7f46a259c700 1 --2- 172.21.15.1:0/3109889000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46a40cbf70 0x7f46a40cc3f0 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7f46a404ebb0 tx=0x7f4698009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7adbc41df90a89e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:09.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.805+0000 7f46a1d9b700 1 -- 172.21.15.1:0/3109889000 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4698011830 con 0x7f46a40cbf70 2024-03-18T20:26:09.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.805+0000 7f46a1d9b700 1 -- 172.21.15.1:0/3109889000 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4698011990 con 0x7f46a40cbf70 2024-03-18T20:26:09.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.805+0000 7f46a1d9b700 1 -- 172.21.15.1:0/3109889000 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4698011d20 con 0x7f46a40cbf70 2024-03-18T20:26:09.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.806+0000 7f46a90a3700 1 -- 172.21.15.1:0/3109889000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46a40cbf70 msgr2=0x7f46a40cc3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:09.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.806+0000 7f46a90a3700 1 --2- 172.21.15.1:0/3109889000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46a40cbf70 0x7f46a40cc3f0 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7f46a404ebb0 tx=0x7f4698009b20 comp rx=0 tx=0).stop 2024-03-18T20:26:09.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.806+0000 7f46a90a3700 1 -- 172.21.15.1:0/3109889000 shutdown_connections 2024-03-18T20:26:09.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.806+0000 7f46a90a3700 1 --2- 172.21.15.1:0/3109889000 >> v2:172.21.15.165:3300/0 conn(0x7f46a4158e30 0x7f46a4071500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:09.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.806+0000 7f46a90a3700 1 --2- 172.21.15.1:0/3109889000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46a40cbf70 0x7f46a40cc3f0 unknown :-1 s=CLOSED pgs=2718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:09.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.806+0000 7f46a90a3700 1 -- 172.21.15.1:0/3109889000 >> 172.21.15.1:0/3109889000 conn(0x7f46a40cca00 msgr2=0x7f46a40cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:09.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.806+0000 7f46a90a3700 1 -- 172.21.15.1:0/3109889000 shutdown_connections 2024-03-18T20:26:09.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.806+0000 7f46a90a3700 1 -- 172.21.15.1:0/3109889000 wait complete. 2024-03-18T20:26:09.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.807+0000 7f46a90a3700 1 Processor -- start 2024-03-18T20:26:09.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.807+0000 7f46a90a3700 1 -- start start 2024-03-18T20:26:09.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.808+0000 7f46a90a3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46a406a750 0x7f46a4154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:09.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.808+0000 7f46a90a3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f46a40cbf70 0x7f46a40c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:09.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.808+0000 7f46a90a3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46a4158e30 0x7f46a40ae3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:09.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.808+0000 7f46a90a3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a41549a0 con 0x7f46a4158e30 2024-03-18T20:26:09.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.808+0000 7f46a90a3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a40c5170 con 0x7f46a40cbf70 2024-03-18T20:26:09.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.808+0000 7f46a90a3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a40c5450 con 0x7f46a406a750 2024-03-18T20:26:09.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.808+0000 7f46a359e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46a4158e30 0x7f46a40ae3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:09.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.808+0000 7f46a359e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46a4158e30 0x7f46a40ae3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43366/0 (socket says 172.21.15.1:43366) 2024-03-18T20:26:09.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.808+0000 7f46a359e700 1 -- 172.21.15.1:0/2844511850 learned_addr learned my addr 172.21.15.1:0/2844511850 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:09.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.809+0000 7f46a2d9d700 1 --2- 172.21.15.1:0/2844511850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46a406a750 0x7f46a4154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:09.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.809+0000 7f46a359e700 1 -- 172.21.15.1:0/2844511850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46a406a750 msgr2=0x7f46a4154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:09.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.809+0000 7f46a359e700 1 --2- 172.21.15.1:0/2844511850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46a406a750 0x7f46a4154460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:09.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.809+0000 7f46a359e700 1 -- 172.21.15.1:0/2844511850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f46a40cbf70 msgr2=0x7f46a40c5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:09.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.809+0000 7f46a259c700 1 --2- 172.21.15.1:0/2844511850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f46a40cbf70 0x7f46a40c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:09.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.809+0000 7f46a359e700 1 --2- 172.21.15.1:0/2844511850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f46a40cbf70 0x7f46a40c5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:09.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.809+0000 7f46a359e700 1 -- 172.21.15.1:0/2844511850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46980094a0 con 0x7f46a4158e30 2024-03-18T20:26:09.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.809+0000 7f46a359e700 1 --2- 172.21.15.1:0/2844511850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46a4158e30 0x7f46a40ae3e0 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7f469c002230 tx=0x7f469c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:09.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.809+0000 7f468b7fe700 1 -- 172.21.15.1:0/2844511850 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469c00ac20 con 0x7f46a4158e30 2024-03-18T20:26:09.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.809+0000 7f468b7fe700 1 -- 172.21.15.1:0/2844511850 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f469c00ada0 con 0x7f46a4158e30 2024-03-18T20:26:09.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.809+0000 7f46a90a3700 1 -- 172.21.15.1:0/2844511850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a40c5730 con 0x7f46a4158e30 2024-03-18T20:26:09.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.809+0000 7f468b7fe700 1 -- 172.21.15.1:0/2844511850 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469c007310 con 0x7f46a4158e30 2024-03-18T20:26:09.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.809+0000 7f46a90a3700 1 -- 172.21.15.1:0/2844511850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a40c4e70 con 0x7f46a4158e30 2024-03-18T20:26:09.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.810+0000 7f46a90a3700 1 -- 172.21.15.1:0/2844511850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a40c5c50 con 0x7f46a4158e30 2024-03-18T20:26:09.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.810+0000 7f468b7fe700 1 -- 172.21.15.1:0/2844511850 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f469c007470 con 0x7f46a4158e30 2024-03-18T20:26:09.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.811+0000 7f468b7fe700 1 --2- 172.21.15.1:0/2844511850 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f469006af00 0x7f469006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:09.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.811+0000 7f46a2d9d700 1 --2- 172.21.15.1:0/2844511850 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f469006af00 0x7f469006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:09.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.811+0000 7f468b7fe700 1 -- 172.21.15.1:0/2844511850 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 6958+0+0 (secure 0 0 0) 0x7f469c089be0 con 0x7f46a4158e30 2024-03-18T20:26:09.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.811+0000 7f46a2d9d700 1 --2- 172.21.15.1:0/2844511850 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f469006af00 0x7f469006d3c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4694006fd0 tx=0x7f4694008040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:09.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:09.816+0000 7f468b7fe700 1 -- 172.21.15.1:0/2844511850 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f46a40c5c50 con 0x7f46a4158e30 2024-03-18T20:26:10.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.018+0000 7f46a90a3700 1 -- 172.21.15.1:0/2844511850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f46a40c5c50 con 0x7f46a4158e30 2024-03-18T20:26:10.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.019+0000 7f468b7fe700 1 -- 172.21.15.1:0/2844511850 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v241) v1 ==== 93+0+10 (secure 0 0 0) 0x7f46a40c5c50 con 0x7f46a4158e30 2024-03-18T20:26:10.024 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:26:10.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.021+0000 7f46a90a3700 1 -- 172.21.15.1:0/2844511850 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f469006af00 msgr2=0x7f469006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:10.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.021+0000 7f46a90a3700 1 --2- 172.21.15.1:0/2844511850 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f469006af00 0x7f469006d3c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4694006fd0 tx=0x7f4694008040 comp rx=0 tx=0).stop 2024-03-18T20:26:10.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.021+0000 7f46a90a3700 1 -- 172.21.15.1:0/2844511850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46a4158e30 msgr2=0x7f46a40ae3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:10.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.021+0000 7f46a90a3700 1 --2- 172.21.15.1:0/2844511850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46a4158e30 0x7f46a40ae3e0 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7f469c002230 tx=0x7f469c00a060 comp rx=0 tx=0).stop 2024-03-18T20:26:10.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.021+0000 7f46a90a3700 1 -- 172.21.15.1:0/2844511850 shutdown_connections 2024-03-18T20:26:10.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.021+0000 7f46a90a3700 1 --2- 172.21.15.1:0/2844511850 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f469006af00 0x7f469006d3c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.021+0000 7f46a90a3700 1 --2- 172.21.15.1:0/2844511850 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f46a406a750 0x7f46a4154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.021+0000 7f46a90a3700 1 --2- 172.21.15.1:0/2844511850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46a4158e30 0x7f46a40ae3e0 unknown :-1 s=CLOSED pgs=2719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.021+0000 7f46a90a3700 1 --2- 172.21.15.1:0/2844511850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f46a40cbf70 0x7f46a40c5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.021+0000 7f46a90a3700 1 -- 172.21.15.1:0/2844511850 >> 172.21.15.1:0/2844511850 conn(0x7f46a40cca00 msgr2=0x7f46a415cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:10.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.021+0000 7f46a90a3700 1 -- 172.21.15.1:0/2844511850 shutdown_connections 2024-03-18T20:26:10.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.021+0000 7f46a90a3700 1 -- 172.21.15.1:0/2844511850 wait complete. 2024-03-18T20:26:10.039 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-03-18T20:26:10.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.161+0000 7f0ed89ba700 1 Processor -- start 2024-03-18T20:26:10.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.161+0000 7f0ed89ba700 1 -- start start 2024-03-18T20:26:10.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.162+0000 7f0ed89ba700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0ed415d3b0 0x7f0ed406cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:10.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.162+0000 7f0ed89ba700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ed4071550 0x7f0ed40b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:10.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.162+0000 7f0ed89ba700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed40b2b80 con 0x7f0ed406d430 2024-03-18T20:26:10.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.162+0000 7f0ed89ba700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed40b3800 con 0x7f0ed415d3b0 2024-03-18T20:26:10.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.162+0000 7f0ed89ba700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed40b2e60 con 0x7f0ed4071550 2024-03-18T20:26:10.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.162+0000 7f0ed259c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0ed415d3b0 0x7f0ed406cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:10.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.162+0000 7f0ed1d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0ed406d430 0x7f0ed4070e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57770/0 (socket says 172.21.15.1:57770) 2024-03-18T20:26:10.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.162+0000 7f0ed1d9b700 1 -- 172.21.15.1:0/3517918176 learned_addr learned my addr 172.21.15.1:0/3517918176 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:10.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.162+0000 7f0ed2d9d700 1 --2- 172.21.15.1:0/3517918176 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ed4071550 0x7f0ed40b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:10.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.162+0000 7f0ed259c700 1 -- 172.21.15.1:0/3517918176 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ed4071550 msgr2=0x7f0ed40b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:10.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.162+0000 7f0ed259c700 1 --2- 172.21.15.1:0/3517918176 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ed4071550 0x7f0ed40b3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.162+0000 7f0ed259c700 1 -- 172.21.15.1:0/3517918176 >> v1:172.21.15.1:6789/0 conn(0x7f0ed406d430 legacy=0x7f0ed4070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:10.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.162+0000 7f0ed259c700 1 -- 172.21.15.1:0/3517918176 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed40b3140 con 0x7f0ed415d3b0 2024-03-18T20:26:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.162+0000 7f0ed159a700 1 -- 172.21.15.1:0/3517918176 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952231209 0 0) 0x7f0ed40b2b80 con 0x7f0ed406d430 2024-03-18T20:26:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.162+0000 7f0ed2d9d700 1 --2- 172.21.15.1:0/3517918176 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ed4071550 0x7f0ed40b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.163+0000 7f0ed259c700 1 --2- 172.21.15.1:0/3517918176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ed415d3b0 0x7f0ed406cef0 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8009840 tx=0x7f0ec8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee881ac2190cc8e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.163+0000 7f0ed159a700 1 -- 172.21.15.1:0/3517918176 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8011830 con 0x7f0ed415d3b0 2024-03-18T20:26:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.163+0000 7f0ed159a700 1 -- 172.21.15.1:0/3517918176 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec8011990 con 0x7f0ed415d3b0 2024-03-18T20:26:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.163+0000 7f0ed159a700 1 -- 172.21.15.1:0/3517918176 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8011d20 con 0x7f0ed415d3b0 2024-03-18T20:26:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.163+0000 7f0ed89ba700 1 -- 172.21.15.1:0/3517918176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ed415d3b0 msgr2=0x7f0ed406cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.163+0000 7f0ed89ba700 1 --2- 172.21.15.1:0/3517918176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ed415d3b0 0x7f0ed406cef0 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8009840 tx=0x7f0ec8009b50 comp rx=0 tx=0).stop 2024-03-18T20:26:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.163+0000 7f0ed89ba700 1 -- 172.21.15.1:0/3517918176 shutdown_connections 2024-03-18T20:26:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.163+0000 7f0ed89ba700 1 --2- 172.21.15.1:0/3517918176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ed415d3b0 0x7f0ed406cef0 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.163+0000 7f0ed89ba700 1 --2- 172.21.15.1:0/3517918176 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ed4071550 0x7f0ed40b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.163+0000 7f0ed89ba700 1 -- 172.21.15.1:0/3517918176 >> 172.21.15.1:0/3517918176 conn(0x7f0ed40b1350 msgr2=0x7f0ed40b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.163+0000 7f0ed89ba700 1 -- 172.21.15.1:0/3517918176 shutdown_connections 2024-03-18T20:26:10.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.163+0000 7f0ed89ba700 1 -- 172.21.15.1:0/3517918176 wait complete. 2024-03-18T20:26:10.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.165+0000 7f0ed89ba700 1 Processor -- start 2024-03-18T20:26:10.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.165+0000 7f0ed89ba700 1 -- start start 2024-03-18T20:26:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.165+0000 7f0ed89ba700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ed406d430 0x7f0ed4142b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.165+0000 7f0ed89ba700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ed4071550 0x7f0ed40b0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.165+0000 7f0ed89ba700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ed415d3b0 0x7f0ed40b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.165+0000 7f0ed89ba700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed4143060 con 0x7f0ed406d430 2024-03-18T20:26:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.165+0000 7f0ed89ba700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed4143bd0 con 0x7f0ed4071550 2024-03-18T20:26:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.165+0000 7f0ed89ba700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed41432e0 con 0x7f0ed415d3b0 2024-03-18T20:26:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.166+0000 7f0ed259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ed406d430 0x7f0ed4142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.166+0000 7f0ed259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ed406d430 0x7f0ed4142b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43394/0 (socket says 172.21.15.1:43394) 2024-03-18T20:26:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.166+0000 7f0ed259c700 1 -- 172.21.15.1:0/3265431559 learned_addr learned my addr 172.21.15.1:0/3265431559 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.166+0000 7f0ed1d9b700 1 --2- 172.21.15.1:0/3265431559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ed4071550 0x7f0ed40b0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.166+0000 7f0ed2d9d700 1 --2- 172.21.15.1:0/3265431559 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ed415d3b0 0x7f0ed40b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.166+0000 7f0ed259c700 1 -- 172.21.15.1:0/3265431559 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ed415d3b0 msgr2=0x7f0ed40b0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:10.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.166+0000 7f0ed259c700 1 --2- 172.21.15.1:0/3265431559 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ed415d3b0 0x7f0ed40b0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.166+0000 7f0ed259c700 1 -- 172.21.15.1:0/3265431559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ed4071550 msgr2=0x7f0ed40b0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.166+0000 7f0ed259c700 1 --2- 172.21.15.1:0/3265431559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ed4071550 0x7f0ed40b0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.166+0000 7f0ed259c700 1 -- 172.21.15.1:0/3265431559 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec80094a0 con 0x7f0ed406d430 2024-03-18T20:26:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.166+0000 7f0ed259c700 1 --2- 172.21.15.1:0/3265431559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ed406d430 0x7f0ed4142b20 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8000c00 tx=0x7f0ec8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.166+0000 7f0ec2ffd700 1 -- 172.21.15.1:0/3265431559 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec80038a0 con 0x7f0ed406d430 2024-03-18T20:26:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.166+0000 7f0ec2ffd700 1 -- 172.21.15.1:0/3265431559 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec801e070 con 0x7f0ed406d430 2024-03-18T20:26:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.166+0000 7f0ed89ba700 1 -- 172.21.15.1:0/3265431559 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed4146180 con 0x7f0ed406d430 2024-03-18T20:26:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.166+0000 7f0ec2ffd700 1 -- 172.21.15.1:0/3265431559 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8003c40 con 0x7f0ed406d430 2024-03-18T20:26:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.166+0000 7f0ed89ba700 1 -- 172.21.15.1:0/3265431559 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed4143860 con 0x7f0ed406d430 2024-03-18T20:26:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.167+0000 7f0ed89ba700 1 -- 172.21.15.1:0/3265431559 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed40676a0 con 0x7f0ed406d430 2024-03-18T20:26:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.168+0000 7f0ec2ffd700 1 -- 172.21.15.1:0/3265431559 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f0ec801a420 con 0x7f0ed406d430 2024-03-18T20:26:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.168+0000 7f0ec2ffd700 1 --2- 172.21.15.1:0/3265431559 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0eb406aed0 0x7f0eb406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.168+0000 7f0ec2ffd700 1 -- 172.21.15.1:0/3265431559 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 6958+0+0 (secure 0 0 0) 0x7f0ec808ad70 con 0x7f0ed406d430 2024-03-18T20:26:10.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.168+0000 7f0ed1d9b700 1 --2- 172.21.15.1:0/3265431559 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0eb406aed0 0x7f0eb406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:10.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.168+0000 7f0ed1d9b700 1 --2- 172.21.15.1:0/3265431559 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0eb406aed0 0x7f0eb406d390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4155d20 tx=0x7f0ec4007be0 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:10.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.173+0000 7f0ec2ffd700 1 -- 172.21.15.1:0/3265431559 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0ec8055710 con 0x7f0ed406d430 2024-03-18T20:26:10.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.373+0000 7f0ed89ba700 1 -- 172.21.15.1:0/3265431559 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0ed415d9a0 con 0x7f0ed406d430 2024-03-18T20:26:10.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.374+0000 7f0ec2ffd700 1 -- 172.21.15.1:0/3265431559 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v241) v1 ==== 107+0+11 (secure 0 0 0) 0x7f0ec8015020 con 0x7f0ed406d430 2024-03-18T20:26:10.379 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:26:10.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.376+0000 7f0ed89ba700 1 -- 172.21.15.1:0/3265431559 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0eb406aed0 msgr2=0x7f0eb406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:10.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.376+0000 7f0ed89ba700 1 --2- 172.21.15.1:0/3265431559 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0eb406aed0 0x7f0eb406d390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4155d20 tx=0x7f0ec4007be0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.376+0000 7f0ed89ba700 1 -- 172.21.15.1:0/3265431559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ed406d430 msgr2=0x7f0ed4142b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:10.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.376+0000 7f0ed89ba700 1 --2- 172.21.15.1:0/3265431559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ed406d430 0x7f0ed4142b20 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8000c00 tx=0x7f0ec8004750 comp rx=0 tx=0).stop 2024-03-18T20:26:10.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.376+0000 7f0ed89ba700 1 -- 172.21.15.1:0/3265431559 shutdown_connections 2024-03-18T20:26:10.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.376+0000 7f0ed89ba700 1 --2- 172.21.15.1:0/3265431559 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0eb406aed0 0x7f0eb406d390 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.376+0000 7f0ed89ba700 1 --2- 172.21.15.1:0/3265431559 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0ed415d3b0 0x7f0ed40b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.376+0000 7f0ed89ba700 1 --2- 172.21.15.1:0/3265431559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ed406d430 0x7f0ed4142b20 unknown :-1 s=CLOSED pgs=2722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.376+0000 7f0ed89ba700 1 --2- 172.21.15.1:0/3265431559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ed4071550 0x7f0ed40b0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.376+0000 7f0ed89ba700 1 -- 172.21.15.1:0/3265431559 >> 172.21.15.1:0/3265431559 conn(0x7f0ed40b1350 msgr2=0x7f0ed40ad740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:10.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.376+0000 7f0ed89ba700 1 -- 172.21.15.1:0/3265431559 shutdown_connections 2024-03-18T20:26:10.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.376+0000 7f0ed89ba700 1 -- 172.21.15.1:0/3265431559 wait complete. 2024-03-18T20:26:10.393 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-03-18T20:26:10.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.516+0000 7fe0e54bf700 1 Processor -- start 2024-03-18T20:26:10.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.517+0000 7fe0e54bf700 1 -- start start 2024-03-18T20:26:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.517+0000 7fe0e54bf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0e00cfee0 0x7fe0e00d3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.517+0000 7fe0e54bf700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e00d1670 con 0x7fe0e00d1160 2024-03-18T20:26:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.517+0000 7fe0e54bf700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e00d22a0 con 0x7fe0e0160e90 2024-03-18T20:26:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.517+0000 7fe0e54bf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e00b25f0 con 0x7fe0e00cfee0 2024-03-18T20:26:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.517+0000 7fe0de7fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe0e00d1160 0x7fe0e0160780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57772/0 (socket says 172.21.15.1:57772) 2024-03-18T20:26:10.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.517+0000 7fe0de7fc700 1 -- 172.21.15.1:0/2326729321 learned_addr learned my addr 172.21.15.1:0/2326729321 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.518+0000 7fe0deffd700 1 --2- 172.21.15.1:0/2326729321 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0e00cfee0 0x7fe0e00d3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.518+0000 7fe0ddffb700 1 -- 172.21.15.1:0/2326729321 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 961815853 0 0) 0x7fe0e00d1670 con 0x7fe0e00d1160 2024-03-18T20:26:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.518+0000 7fe0ddffb700 1 -- 172.21.15.1:0/2326729321 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0c80035b0 con 0x7fe0e00d1160 2024-03-18T20:26:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.518+0000 7fe0ddffb700 1 -- 172.21.15.1:0/2326729321 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2478329287 0 0) 0x7fe0c80035b0 con 0x7fe0e00d1160 2024-03-18T20:26:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.518+0000 7fe0ddffb700 1 -- 172.21.15.1:0/2326729321 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0e00cfee0 msgr2=0x7fe0e00d3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.518+0000 7fe0ddffb700 1 --2- 172.21.15.1:0/2326729321 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0e00cfee0 0x7fe0e00d3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.518+0000 7fe0ddffb700 1 -- 172.21.15.1:0/2326729321 >> v1:172.21.15.165:6789/0 conn(0x7fe0e0160e90 legacy=0x7fe0e00b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.518+0000 7fe0ddffb700 1 -- 172.21.15.1:0/2326729321 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e00d1970 con 0x7fe0e00d1160 2024-03-18T20:26:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.518+0000 7fe0ddffb700 1 -- 172.21.15.1:0/2326729321 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fe0d0001670 con 0x7fe0e00d1160 2024-03-18T20:26:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.518+0000 7fe0ddffb700 1 -- 172.21.15.1:0/2326729321 >> v1:172.21.15.1:6789/0 conn(0x7fe0e00d1160 legacy=0x7fe0e0160780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.518+0000 7fe0ddffb700 1 --2- 172.21.15.1:0/2326729321 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0c8003a30 0x7fe0c8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.518+0000 7fe0ddffb700 1 --2- 172.21.15.1:0/2326729321 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0e00cfee0 0x7fe0c8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.518+0000 7fe0ddffb700 1 --2- 172.21.15.1:0/2326729321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0c8004d90 0x7fe0c8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.518+0000 7fe0ddffb700 1 -- 172.21.15.1:0/2326729321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e00b25f0 con 0x7fe0c8004d90 2024-03-18T20:26:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.518+0000 7fe0ddffb700 1 -- 172.21.15.1:0/2326729321 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c800bbd0 con 0x7fe0c8003a30 2024-03-18T20:26:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.518+0000 7fe0ddffb700 1 -- 172.21.15.1:0/2326729321 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c800bd90 con 0x7fe0e00cfee0 2024-03-18T20:26:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.518+0000 7fe0df7fe700 1 --2- 172.21.15.1:0/2326729321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0c8004d90 0x7fe0c8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.518+0000 7fe0de7fc700 1 --2- 172.21.15.1:0/2326729321 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0e00cfee0 0x7fe0c8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.518+0000 7fe0deffd700 1 --2- 172.21.15.1:0/2326729321 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0c8003a30 0x7fe0c8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.519+0000 7fe0df7fe700 1 -- 172.21.15.1:0/2326729321 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0e00cfee0 msgr2=0x7fe0c8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.519+0000 7fe0df7fe700 1 --2- 172.21.15.1:0/2326729321 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0e00cfee0 0x7fe0c8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.519+0000 7fe0df7fe700 1 -- 172.21.15.1:0/2326729321 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0c8003a30 msgr2=0x7fe0c8003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.519+0000 7fe0df7fe700 1 --2- 172.21.15.1:0/2326729321 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0c8003a30 0x7fe0c8003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.519+0000 7fe0df7fe700 1 -- 172.21.15.1:0/2326729321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c80035b0 con 0x7fe0c8004d90 2024-03-18T20:26:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.519+0000 7fe0df7fe700 1 --2- 172.21.15.1:0/2326729321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0c8004d90 0x7fe0c8005210 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7fe0d800c160 tx=0x7fe0c8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e17f2ba5455b69 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.519+0000 7fe0ddffb700 1 -- 172.21.15.1:0/2326729321 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d8001db0 con 0x7fe0c8004d90 2024-03-18T20:26:10.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.519+0000 7fe0ddffb700 1 -- 172.21.15.1:0/2326729321 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d800cca0 con 0x7fe0c8004d90 2024-03-18T20:26:10.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.519+0000 7fe0ddffb700 1 -- 172.21.15.1:0/2326729321 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d8003c40 con 0x7fe0c8004d90 2024-03-18T20:26:10.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.519+0000 7fe0e54bf700 1 -- 172.21.15.1:0/2326729321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0c8004d90 msgr2=0x7fe0c8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:10.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.519+0000 7fe0e54bf700 1 --2- 172.21.15.1:0/2326729321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0c8004d90 0x7fe0c8005210 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7fe0d800c160 tx=0x7fe0c8005880 comp rx=0 tx=0).stop 2024-03-18T20:26:10.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.519+0000 7fe0df7fe700 1 -- 172.21.15.1:0/2326729321 reap_dead start 2024-03-18T20:26:10.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.519+0000 7fe0e54bf700 1 -- 172.21.15.1:0/2326729321 shutdown_connections 2024-03-18T20:26:10.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.519+0000 7fe0e54bf700 1 -- 172.21.15.1:0/2326729321 >> 172.21.15.1:0/2326729321 conn(0x7fe0e00d3260 msgr2=0x7fe0e00d3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:10.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.519+0000 7fe0e54bf700 1 -- 172.21.15.1:0/2326729321 shutdown_connections 2024-03-18T20:26:10.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.519+0000 7fe0e54bf700 1 -- 172.21.15.1:0/2326729321 wait complete. 2024-03-18T20:26:10.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.521+0000 7fe0e54bf700 1 Processor -- start 2024-03-18T20:26:10.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.521+0000 7fe0e54bf700 1 -- start start 2024-03-18T20:26:10.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0e54bf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0c8004d90 0x7fe0e01f28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:10.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0e54bf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0e0152f70 0x7fe0e01f6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:10.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0e54bf700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0e01f6a80 0x7fe0e01f6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:10.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0e54bf700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e01f2e20 con 0x7fe0c8004d90 2024-03-18T20:26:10.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0e54bf700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e00c7af0 con 0x7fe0e01f6a80 2024-03-18T20:26:10.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0e54bf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e00c7dd0 con 0x7fe0e0152f70 2024-03-18T20:26:10.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0deffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0c8004d90 0x7fe0e01f28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:10.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0deffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0c8004d90 0x7fe0e01f28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43420/0 (socket says 172.21.15.1:43420) 2024-03-18T20:26:10.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0deffd700 1 -- 172.21.15.1:0/2354331348 learned_addr learned my addr 172.21.15.1:0/2354331348 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:10.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0de7fc700 1 --2- 172.21.15.1:0/2354331348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0e0152f70 0x7fe0e01f6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:10.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0df7fe700 1 --2- 172.21.15.1:0/2354331348 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0e01f6a80 0x7fe0e01f6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:10.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0deffd700 1 -- 172.21.15.1:0/2354331348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0e0152f70 msgr2=0x7fe0e01f6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:10.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0deffd700 1 --2- 172.21.15.1:0/2354331348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0e0152f70 0x7fe0e01f6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0deffd700 1 -- 172.21.15.1:0/2354331348 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0e01f6a80 msgr2=0x7fe0e01f6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:10.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0deffd700 1 --2- 172.21.15.1:0/2354331348 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0e01f6a80 0x7fe0e01f6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0deffd700 1 -- 172.21.15.1:0/2354331348 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e00c80b0 con 0x7fe0c8004d90 2024-03-18T20:26:10.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0deffd700 1 --2- 172.21.15.1:0/2354331348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0c8004d90 0x7fe0e01f28e0 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4011aa0 tx=0x7fe0d4011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:10.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0c77fe700 1 -- 172.21.15.1:0/2354331348 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d400f980 con 0x7fe0c8004d90 2024-03-18T20:26:10.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0c77fe700 1 -- 172.21.15.1:0/2354331348 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d400fae0 con 0x7fe0c8004d90 2024-03-18T20:26:10.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0c77fe700 1 -- 172.21.15.1:0/2354331348 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d400fe90 con 0x7fe0c8004d90 2024-03-18T20:26:10.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0e54bf700 1 -- 172.21.15.1:0/2354331348 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e00c8390 con 0x7fe0c8004d90 2024-03-18T20:26:10.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.522+0000 7fe0e54bf700 1 -- 172.21.15.1:0/2354331348 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e00c77a0 con 0x7fe0c8004d90 2024-03-18T20:26:10.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.523+0000 7fe0e54bf700 1 -- 172.21.15.1:0/2354331348 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e0067bc0 con 0x7fe0c8004d90 2024-03-18T20:26:10.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.524+0000 7fe0c77fe700 1 -- 172.21.15.1:0/2354331348 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fe0d4014440 con 0x7fe0c8004d90 2024-03-18T20:26:10.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.524+0000 7fe0c77fe700 1 --2- 172.21.15.1:0/2354331348 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe0cc06aed0 0x7fe0cc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:10.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.524+0000 7fe0c77fe700 1 -- 172.21.15.1:0/2354331348 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 6958+0+0 (secure 0 0 0) 0x7fe0d408a360 con 0x7fe0c8004d90 2024-03-18T20:26:10.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.524+0000 7fe0de7fc700 1 --2- 172.21.15.1:0/2354331348 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe0cc06aed0 0x7fe0cc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:10.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.525+0000 7fe0de7fc700 1 --2- 172.21.15.1:0/2354331348 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe0cc06aed0 0x7fe0cc06d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0153370 tx=0x7fe0d0009040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:10.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.529+0000 7fe0c77fe700 1 -- 172.21.15.1:0/2354331348 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe0d4054e50 con 0x7fe0c8004d90 2024-03-18T20:26:10.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.734+0000 7fe0e54bf700 1 -- 172.21.15.1:0/2354331348 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7fe0e0073350 con 0x7fe0c8004d90 2024-03-18T20:26:10.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.735+0000 7fe0c77fe700 1 -- 172.21.15.1:0/2354331348 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v241) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe0d4058470 con 0x7fe0c8004d90 2024-03-18T20:26:10.740 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:26:10.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.737+0000 7fe0e54bf700 1 -- 172.21.15.1:0/2354331348 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe0cc06aed0 msgr2=0x7fe0cc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:10.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.737+0000 7fe0e54bf700 1 --2- 172.21.15.1:0/2354331348 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe0cc06aed0 0x7fe0cc06d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0153370 tx=0x7fe0d0009040 comp rx=0 tx=0).stop 2024-03-18T20:26:10.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.737+0000 7fe0e54bf700 1 -- 172.21.15.1:0/2354331348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0c8004d90 msgr2=0x7fe0e01f28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:10.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.737+0000 7fe0e54bf700 1 --2- 172.21.15.1:0/2354331348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0c8004d90 0x7fe0e01f28e0 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4011aa0 tx=0x7fe0d4011e60 comp rx=0 tx=0).stop 2024-03-18T20:26:10.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.737+0000 7fe0e54bf700 1 -- 172.21.15.1:0/2354331348 shutdown_connections 2024-03-18T20:26:10.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.737+0000 7fe0e54bf700 1 --2- 172.21.15.1:0/2354331348 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe0cc06aed0 0x7fe0cc06d390 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.737+0000 7fe0e54bf700 1 --2- 172.21.15.1:0/2354331348 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0e01f6a80 0x7fe0e01f6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.737+0000 7fe0e54bf700 1 --2- 172.21.15.1:0/2354331348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0c8004d90 0x7fe0e01f28e0 unknown :-1 s=CLOSED pgs=2725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.737+0000 7fe0e54bf700 1 --2- 172.21.15.1:0/2354331348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe0e0152f70 0x7fe0e01f6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.737+0000 7fe0e54bf700 1 -- 172.21.15.1:0/2354331348 >> 172.21.15.1:0/2354331348 conn(0x7fe0e00d3260 msgr2=0x7fe0e00b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:10.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.737+0000 7fe0e54bf700 1 -- 172.21.15.1:0/2354331348 shutdown_connections 2024-03-18T20:26:10.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.737+0000 7fe0e54bf700 1 -- 172.21.15.1:0/2354331348 wait complete. 2024-03-18T20:26:10.754 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:26:10.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.875+0000 7f444515f700 1 Processor -- start 2024-03-18T20:26:10.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.875+0000 7f444515f700 1 -- start start 2024-03-18T20:26:10.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.876+0000 7f444515f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f444015d6b0 0x7f44400b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:10.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.876+0000 7f444515f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44400acfb0 con 0x7f44400ad790 2024-03-18T20:26:10.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.876+0000 7f444515f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44400b50a0 con 0x7f444006a750 2024-03-18T20:26:10.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.876+0000 7f444515f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44400af7a0 con 0x7f444015d6b0 2024-03-18T20:26:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.876+0000 7f443e59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f444015d6b0 0x7f44400b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.876+0000 7f443f59e700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f44400ad790 0x7f44400b32e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57788/0 (socket says 172.21.15.1:57788) 2024-03-18T20:26:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.876+0000 7f443f59e700 1 -- 172.21.15.1:0/2123723223 learned_addr learned my addr 172.21.15.1:0/2123723223 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.876+0000 7f443e59c700 1 -- 172.21.15.1:0/2123723223 >> v1:172.21.15.165:6789/0 conn(0x7f444006a750 legacy=0x7f444006ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.876+0000 7f443e59c700 1 -- 172.21.15.1:0/2123723223 >> v1:172.21.15.1:6789/0 conn(0x7f44400ad790 legacy=0x7f44400b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.876+0000 7f443e59c700 1 -- 172.21.15.1:0/2123723223 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44400af1e0 con 0x7f444015d6b0 2024-03-18T20:26:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.876+0000 7f443dd9b700 1 -- 172.21.15.1:0/2123723223 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630268310 0 0) 0x7f44400acfb0 con 0x7f44400ad790 2024-03-18T20:26:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.876+0000 7f443e59c700 1 --2- 172.21.15.1:0/2123723223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f444015d6b0 0x7f44400b2da0 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7f4434009840 tx=0x7f4434009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeb097e1dcc74db5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.876+0000 7f443dd9b700 1 -- 172.21.15.1:0/2123723223 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4434004230 con 0x7f444015d6b0 2024-03-18T20:26:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.876+0000 7f443dd9b700 1 -- 172.21.15.1:0/2123723223 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4434004390 con 0x7f444015d6b0 2024-03-18T20:26:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.876+0000 7f443dd9b700 1 -- 172.21.15.1:0/2123723223 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44340036a0 con 0x7f444015d6b0 2024-03-18T20:26:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.877+0000 7f444515f700 1 -- 172.21.15.1:0/2123723223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f444015d6b0 msgr2=0x7f44400b2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.877+0000 7f444515f700 1 --2- 172.21.15.1:0/2123723223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f444015d6b0 0x7f44400b2da0 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7f4434009840 tx=0x7f4434009b50 comp rx=0 tx=0).stop 2024-03-18T20:26:10.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.877+0000 7f444515f700 1 -- 172.21.15.1:0/2123723223 shutdown_connections 2024-03-18T20:26:10.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.877+0000 7f444515f700 1 --2- 172.21.15.1:0/2123723223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f444015d6b0 0x7f44400b2da0 unknown :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.877+0000 7f444515f700 1 -- 172.21.15.1:0/2123723223 >> 172.21.15.1:0/2123723223 conn(0x7f444006b860 msgr2=0x7f444006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:10.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.877+0000 7f444515f700 1 -- 172.21.15.1:0/2123723223 shutdown_connections 2024-03-18T20:26:10.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.877+0000 7f444515f700 1 -- 172.21.15.1:0/2123723223 wait complete. 2024-03-18T20:26:10.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.878+0000 7f444515f700 1 Processor -- start 2024-03-18T20:26:10.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.879+0000 7f444515f700 1 -- start start 2024-03-18T20:26:10.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.879+0000 7f444515f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f444006a750 0x7f444015c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.879+0000 7f444515f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44400ad790 0x7f4440157f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.879+0000 7f444515f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f444015d6b0 0x7f44401e5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.879+0000 7f444515f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444015bcc0 con 0x7f444006a750 2024-03-18T20:26:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.879+0000 7f444515f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44401e6930 con 0x7f444015d6b0 2024-03-18T20:26:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.879+0000 7f444515f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f444015e410 con 0x7f44400ad790 2024-03-18T20:26:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.879+0000 7f443ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f444006a750 0x7f444015c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.879+0000 7f443ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f444006a750 0x7f444015c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43436/0 (socket says 172.21.15.1:43436) 2024-03-18T20:26:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.879+0000 7f443ed9d700 1 -- 172.21.15.1:0/1935255222 learned_addr learned my addr 172.21.15.1:0/1935255222 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.880+0000 7f443e59c700 1 --2- 172.21.15.1:0/1935255222 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44400ad790 0x7f4440157f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.880+0000 7f443f59e700 1 --2- 172.21.15.1:0/1935255222 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f444015d6b0 0x7f44401e5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.880+0000 7f443ed9d700 1 -- 172.21.15.1:0/1935255222 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44400ad790 msgr2=0x7f4440157f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.880+0000 7f443ed9d700 1 --2- 172.21.15.1:0/1935255222 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44400ad790 0x7f4440157f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.880+0000 7f443ed9d700 1 -- 172.21.15.1:0/1935255222 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f444015d6b0 msgr2=0x7f44401e5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:10.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.880+0000 7f443ed9d700 1 --2- 172.21.15.1:0/1935255222 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f444015d6b0 0x7f44401e5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:10.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.880+0000 7f443ed9d700 1 -- 172.21.15.1:0/1935255222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44340094a0 con 0x7f444006a750 2024-03-18T20:26:10.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.880+0000 7f443e59c700 1 --2- 172.21.15.1:0/1935255222 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44400ad790 0x7f4440157f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:10.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.880+0000 7f443ed9d700 1 --2- 172.21.15.1:0/1935255222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f444006a750 0x7f444015c0e0 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f443000c4d0 tx=0x7f443000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:10.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.880+0000 7f44277fe700 1 -- 172.21.15.1:0/1935255222 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44300054e0 con 0x7f444006a750 2024-03-18T20:26:10.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.880+0000 7f44277fe700 1 -- 172.21.15.1:0/1935255222 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4430003680 con 0x7f444006a750 2024-03-18T20:26:10.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.880+0000 7f444515f700 1 -- 172.21.15.1:0/1935255222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f444015e6f0 con 0x7f444006a750 2024-03-18T20:26:10.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.880+0000 7f44277fe700 1 -- 172.21.15.1:0/1935255222 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4430003a30 con 0x7f444006a750 2024-03-18T20:26:10.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.880+0000 7f444515f700 1 -- 172.21.15.1:0/1935255222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f444015b8e0 con 0x7f444006a750 2024-03-18T20:26:10.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.881+0000 7f444515f700 1 -- 172.21.15.1:0/1935255222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44400ad420 con 0x7f444006a750 2024-03-18T20:26:10.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.881+0000 7f44277fe700 1 -- 172.21.15.1:0/1935255222 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f4430003b90 con 0x7f444006a750 2024-03-18T20:26:10.889 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:26:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.882+0000 7f44277fe700 1 --2- 172.21.15.1:0/1935255222 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f442c06aed0 0x7f442c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.882+0000 7f443e59c700 1 --2- 172.21.15.1:0/1935255222 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f442c06aed0 0x7f442c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.882+0000 7f44277fe700 1 -- 172.21.15.1:0/1935255222 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 6958+0+0 (secure 0 0 0) 0x7f4430089190 con 0x7f444006a750 2024-03-18T20:26:10.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.882+0000 7f443e59c700 1 --2- 172.21.15.1:0/1935255222 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f442c06aed0 0x7f442c06d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f44340090b0 tx=0x7f4434008030 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:10.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:26:10.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:10.886+0000 7f44277fe700 1 -- 172.21.15.1:0/1935255222 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f44400ad420 con 0x7f444006a750 2024-03-18T20:26:11.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.137+0000 7f444515f700 1 -- 172.21.15.1:0/1935255222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44400ad420 con 0x7f444006a750 2024-03-18T20:26:11.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.137+0000 7f44277fe700 1 -- 172.21.15.1:0/1935255222 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+4747 (secure 0 0 0) 0x7f44400ad420 con 0x7f444006a750 2024-03-18T20:26:11.142 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:11.143 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":112,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10369,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/677258734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":677258734},{"type":"v1","addr":"172.21.15.165:6835","nonce":677258734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/658650553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":658650553},{"type":"v1","addr":"172.21.15.165:6834","nonce":658650553}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":112},{"gid":11544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2018398457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2018398457},{"type":"v1","addr":"172.21.15.1:6836","nonce":2018398457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:25:35.549382+0000","modified":"2024-03-18T20:25:38.565325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11547":{"gid":11547,"name":"c","rank":0,"incarnation":106,"state":"up:active","state_seq":6,"addr":"172.21.15.1:6837/1635684321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1635684321},{"type":"v1","addr":"172.21.15.1:6837","nonce":1635684321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-03-18T20:26:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.139+0000 7f444515f700 1 -- 172.21.15.1:0/1935255222 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f442c06aed0 msgr2=0x7f442c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.139+0000 7f444515f700 1 --2- 172.21.15.1:0/1935255222 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f442c06aed0 0x7f442c06d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f44340090b0 tx=0x7f4434008030 comp rx=0 tx=0).stop 2024-03-18T20:26:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.139+0000 7f444515f700 1 -- 172.21.15.1:0/1935255222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f444006a750 msgr2=0x7f444015c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.139+0000 7f444515f700 1 --2- 172.21.15.1:0/1935255222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f444006a750 0x7f444015c0e0 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f443000c4d0 tx=0x7f443000c890 comp rx=0 tx=0).stop 2024-03-18T20:26:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.139+0000 7f444515f700 1 -- 172.21.15.1:0/1935255222 shutdown_connections 2024-03-18T20:26:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.139+0000 7f444515f700 1 --2- 172.21.15.1:0/1935255222 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f442c06aed0 0x7f442c06d390 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.139+0000 7f444515f700 1 --2- 172.21.15.1:0/1935255222 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f444015d6b0 0x7f44401e5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.139+0000 7f444515f700 1 --2- 172.21.15.1:0/1935255222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f444006a750 0x7f444015c0e0 unknown :-1 s=CLOSED pgs=2728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.139+0000 7f444515f700 1 --2- 172.21.15.1:0/1935255222 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f44400ad790 0x7f4440157f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.140+0000 7f444515f700 1 -- 172.21.15.1:0/1935255222 >> 172.21.15.1:0/1935255222 conn(0x7f444006b860 msgr2=0x7f444006e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.140+0000 7f444515f700 1 -- 172.21.15.1:0/1935255222 shutdown_connections 2024-03-18T20:26:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.140+0000 7f444515f700 1 -- 172.21.15.1:0/1935255222 wait complete. 2024-03-18T20:26:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 112 2024-03-18T20:26:11.157 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:26:11.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.276+0000 7f61b8559700 1 Processor -- start 2024-03-18T20:26:11.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.276+0000 7f61b8559700 1 -- start start 2024-03-18T20:26:11.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.277+0000 7f61b8559700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f61b015b300 0x7f61b00cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:11.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.277+0000 7f61b8559700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f61b006ef20 0x7f61b0099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:11.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.277+0000 7f61b8559700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b015b6e0 con 0x7f61b00cc5e0 2024-03-18T20:26:11.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.277+0000 7f61b8559700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b00cffd0 con 0x7f61b006ef20 2024-03-18T20:26:11.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.277+0000 7f61b8559700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b00cce60 con 0x7f61b015b300 2024-03-18T20:26:11.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.277+0000 7f61b6af6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f61b006ef20 0x7f61b0099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:11.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.278+0000 7f61b5af4700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f61b00cc5e0 0x7f61b006e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57792/0 (socket says 172.21.15.1:57792) 2024-03-18T20:26:11.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.278+0000 7f61b5af4700 1 -- 172.21.15.1:0/3143385857 learned_addr learned my addr 172.21.15.1:0/3143385857 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:11.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.278+0000 7f61b6af6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f61b006ef20 0x7f61b0099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43444/0 (socket says 172.21.15.1:43444) 2024-03-18T20:26:11.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.278+0000 7f61b62f5700 1 --2- 172.21.15.1:0/3143385857 >> v2:172.21.15.165:3300/0 conn(0x7f61b015b300 0x7f61b00cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:11.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.278+0000 7f61b6af6700 1 -- 172.21.15.1:0/3143385857 >> v2:172.21.15.165:3300/0 conn(0x7f61b015b300 msgr2=0x7f61b00cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:11.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.278+0000 7f61b6af6700 1 --2- 172.21.15.1:0/3143385857 >> v2:172.21.15.165:3300/0 conn(0x7f61b015b300 0x7f61b00cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.278+0000 7f61b6af6700 1 -- 172.21.15.1:0/3143385857 >> v1:172.21.15.1:6789/0 conn(0x7f61b00cc5e0 legacy=0x7f61b006e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:11.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.278+0000 7f61b6af6700 1 -- 172.21.15.1:0/3143385857 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b00cecd0 con 0x7f61b006ef20 2024-03-18T20:26:11.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.278+0000 7f61b6af6700 1 --2- 172.21.15.1:0/3143385857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b006ef20 0x7f61b0099a40 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7f61b0071a60 tx=0x7f61a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f12f8f460e1405c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:11.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.278+0000 7f61b52f3700 1 -- 172.21.15.1:0/3143385857 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a800e070 con 0x7f61b006ef20 2024-03-18T20:26:11.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.278+0000 7f61b52f3700 1 -- 172.21.15.1:0/3143385857 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a800bbd0 con 0x7f61b006ef20 2024-03-18T20:26:11.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.278+0000 7f61b52f3700 1 -- 172.21.15.1:0/3143385857 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a8012430 con 0x7f61b006ef20 2024-03-18T20:26:11.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.279+0000 7f61b8559700 1 -- 172.21.15.1:0/3143385857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b006ef20 msgr2=0x7f61b0099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:11.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.279+0000 7f61b8559700 1 --2- 172.21.15.1:0/3143385857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b006ef20 0x7f61b0099a40 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7f61b0071a60 tx=0x7f61a800b330 comp rx=0 tx=0).stop 2024-03-18T20:26:11.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.279+0000 7f61b8559700 1 -- 172.21.15.1:0/3143385857 shutdown_connections 2024-03-18T20:26:11.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.279+0000 7f61b8559700 1 --2- 172.21.15.1:0/3143385857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b006ef20 0x7f61b0099a40 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.279+0000 7f61b8559700 1 --2- 172.21.15.1:0/3143385857 >> v2:172.21.15.165:3300/0 conn(0x7f61b015b300 0x7f61b00cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.279+0000 7f61b8559700 1 -- 172.21.15.1:0/3143385857 >> 172.21.15.1:0/3143385857 conn(0x7f61b00b2b50 msgr2=0x7f61b00b2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:11.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.279+0000 7f61b8559700 1 -- 172.21.15.1:0/3143385857 shutdown_connections 2024-03-18T20:26:11.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.279+0000 7f61b8559700 1 -- 172.21.15.1:0/3143385857 wait complete. 2024-03-18T20:26:11.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.280+0000 7f61b8559700 1 Processor -- start 2024-03-18T20:26:11.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.280+0000 7f61b8559700 1 -- start start 2024-03-18T20:26:11.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61b8559700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61b006ef20 0x7f61b00c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:11.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61b8559700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b00cc5e0 0x7f61b00c4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:11.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61b8559700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61b015b300 0x7f61b015e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:11.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61b8559700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b0153660 con 0x7f61b00cc5e0 2024-03-18T20:26:11.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61b8559700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b0150680 con 0x7f61b006ef20 2024-03-18T20:26:11.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61b8559700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b014f7b0 con 0x7f61b015b300 2024-03-18T20:26:11.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61b5af4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b00cc5e0 0x7f61b00c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:11.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61b5af4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b00cc5e0 0x7f61b00c4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43454/0 (socket says 172.21.15.1:43454) 2024-03-18T20:26:11.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61b5af4700 1 -- 172.21.15.1:0/4139738179 learned_addr learned my addr 172.21.15.1:0/4139738179 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:11.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61b62f5700 1 --2- 172.21.15.1:0/4139738179 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61b006ef20 0x7f61b00c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:11.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61b6af6700 1 --2- 172.21.15.1:0/4139738179 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61b015b300 0x7f61b015e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:11.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61b5af4700 1 -- 172.21.15.1:0/4139738179 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61b015b300 msgr2=0x7f61b015e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:11.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61b5af4700 1 --2- 172.21.15.1:0/4139738179 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61b015b300 0x7f61b015e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61b5af4700 1 -- 172.21.15.1:0/4139738179 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61b006ef20 msgr2=0x7f61b00c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:11.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61b5af4700 1 --2- 172.21.15.1:0/4139738179 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61b006ef20 0x7f61b00c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61b5af4700 1 -- 172.21.15.1:0/4139738179 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a8009a70 con 0x7f61b00cc5e0 2024-03-18T20:26:11.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61b6af6700 1 --2- 172.21.15.1:0/4139738179 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61b015b300 0x7f61b015e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:11.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61b62f5700 1 --2- 172.21.15.1:0/4139738179 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61b006ef20 0x7f61b00c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:11.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61b5af4700 1 --2- 172.21.15.1:0/4139738179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b00cc5e0 0x7f61b00c4000 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00c550 tx=0x7f61ac00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:11.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.281+0000 7f61a6ffd700 1 -- 172.21.15.1:0/4139738179 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac005d70 con 0x7f61b00cc5e0 2024-03-18T20:26:11.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.282+0000 7f61a6ffd700 1 -- 172.21.15.1:0/4139738179 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ac0070d0 con 0x7f61b00cc5e0 2024-03-18T20:26:11.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.282+0000 7f61a6ffd700 1 -- 172.21.15.1:0/4139738179 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac007440 con 0x7f61b00cc5e0 2024-03-18T20:26:11.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.282+0000 7f61b8559700 1 -- 172.21.15.1:0/4139738179 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b014fa00 con 0x7f61b00cc5e0 2024-03-18T20:26:11.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.282+0000 7f61b8559700 1 -- 172.21.15.1:0/4139738179 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b014ffc0 con 0x7f61b00cc5e0 2024-03-18T20:26:11.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.282+0000 7f61b8559700 1 -- 172.21.15.1:0/4139738179 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b00ca470 con 0x7f61b00cc5e0 2024-03-18T20:26:11.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.283+0000 7f61a6ffd700 1 -- 172.21.15.1:0/4139738179 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f61ac0075a0 con 0x7f61b00cc5e0 2024-03-18T20:26:11.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.283+0000 7f61a6ffd700 1 --2- 172.21.15.1:0/4139738179 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f619406aed0 0x7f619406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:11.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.283+0000 7f61a6ffd700 1 -- 172.21.15.1:0/4139738179 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 6958+0+0 (secure 0 0 0) 0x7f61ac089010 con 0x7f61b00cc5e0 2024-03-18T20:26:11.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.284+0000 7f61b62f5700 1 --2- 172.21.15.1:0/4139738179 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f619406aed0 0x7f619406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:11.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.284+0000 7f61b62f5700 1 --2- 172.21.15.1:0/4139738179 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f619406aed0 0x7f619406d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f61a0007a50 tx=0x7f61a0008040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:11.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.288+0000 7f61a6ffd700 1 -- 172.21.15.1:0/4139738179 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f61ac053a60 con 0x7f61b00cc5e0 2024-03-18T20:26:11.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.493+0000 7f61b8559700 1 -- 172.21.15.1:0/4139738179 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61b00aff90 con 0x7f61b00cc5e0 2024-03-18T20:26:11.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.494+0000 7f61a6ffd700 1 -- 172.21.15.1:0/4139738179 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v241) v1 ==== 74+0+17537 (secure 0 0 0) 0x7f61ac057080 con 0x7f61b00cc5e0 2024-03-18T20:26:11.499 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:11.500 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":241,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:26:09.280336+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:25:33.275375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:25:34.518347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.165:6834/4024828630":"2024-03-19T20:26:06.260180+0000","172.21.15.1:0/4210355381":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/1879196326":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/101306769":"2024-03-19T20:26:03.849558+0000","172.21.15.165:6832/4024828630":"2024-03-19T20:26:06.260180+0000","172.21.15.1:0/997525540":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/1622795270":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/2535232994":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2652258802":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2404324807":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2059399259":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/3657737679":"2024-03-19T20:25:52.933722+0000","172.21.15.1:6833/148356":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2685347289":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/1965726293":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/2324250327":"2024-03-19T20:25:52.933722+0000","172.21.15.1:6832/148356":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/1022040508":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/2165965129":"2024-03-19T20:25:52.933722+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:26:11.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.496+0000 7f61b8559700 1 -- 172.21.15.1:0/4139738179 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f619406aed0 msgr2=0x7f619406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:11.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.496+0000 7f61b8559700 1 --2- 172.21.15.1:0/4139738179 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f619406aed0 0x7f619406d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f61a0007a50 tx=0x7f61a0008040 comp rx=0 tx=0).stop 2024-03-18T20:26:11.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.496+0000 7f61b8559700 1 -- 172.21.15.1:0/4139738179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b00cc5e0 msgr2=0x7f61b00c4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:11.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.496+0000 7f61b8559700 1 --2- 172.21.15.1:0/4139738179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b00cc5e0 0x7f61b00c4000 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00c550 tx=0x7f61ac00c910 comp rx=0 tx=0).stop 2024-03-18T20:26:11.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.496+0000 7f61b8559700 1 -- 172.21.15.1:0/4139738179 shutdown_connections 2024-03-18T20:26:11.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.496+0000 7f61b8559700 1 --2- 172.21.15.1:0/4139738179 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f619406aed0 0x7f619406d390 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.496+0000 7f61b8559700 1 --2- 172.21.15.1:0/4139738179 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61b015b300 0x7f61b015e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.496+0000 7f61b8559700 1 --2- 172.21.15.1:0/4139738179 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61b006ef20 0x7f61b00c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.496+0000 7f61b8559700 1 --2- 172.21.15.1:0/4139738179 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61b00cc5e0 0x7f61b00c4000 unknown :-1 s=CLOSED pgs=2731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.496+0000 7f61b8559700 1 -- 172.21.15.1:0/4139738179 >> 172.21.15.1:0/4139738179 conn(0x7f61b00b2b50 msgr2=0x7f61b00b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:11.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.496+0000 7f61b8559700 1 -- 172.21.15.1:0/4139738179 shutdown_connections 2024-03-18T20:26:11.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.496+0000 7f61b8559700 1 -- 172.21.15.1:0/4139738179 wait complete. 2024-03-18T20:26:11.513 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2024-03-18T20:26:11.514 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:26:11.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.633+0000 7f939e171700 1 Processor -- start 2024-03-18T20:26:11.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.634+0000 7f939e171700 1 -- start start 2024-03-18T20:26:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.634+0000 7f939e171700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f939815d6b0 0x7f93980b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.634+0000 7f939e171700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f93980ad790 0x7f93980b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.634+0000 7f939e171700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93980acfb0 con 0x7f939806a750 2024-03-18T20:26:11.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.634+0000 7f939e171700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93980af7a0 con 0x7f93980ad790 2024-03-18T20:26:11.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.634+0000 7f939e171700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93980b50a0 con 0x7f939815d6b0 2024-03-18T20:26:11.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.634+0000 7f93977fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f939806a750 0x7f939806ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57806/0 (socket says 172.21.15.1:57806) 2024-03-18T20:26:11.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.634+0000 7f93977fe700 1 -- 172.21.15.1:0/2064292301 learned_addr learned my addr 172.21.15.1:0/2064292301 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:11.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f9397fff700 1 --2- 172.21.15.1:0/2064292301 >> v2:172.21.15.165:3300/0 conn(0x7f93980ad790 0x7f93980b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:11.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f9396ffd700 1 --2- 172.21.15.1:0/2064292301 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f939815d6b0 0x7f93980b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:11.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f93967fc700 1 -- 172.21.15.1:0/2064292301 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805464867 0 0) 0x7f93980acfb0 con 0x7f939806a750 2024-03-18T20:26:11.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f93967fc700 1 -- 172.21.15.1:0/2064292301 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93840035b0 con 0x7f939806a750 2024-03-18T20:26:11.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f93967fc700 1 -- 172.21.15.1:0/2064292301 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1276999738 0 0) 0x7f93840035b0 con 0x7f939806a750 2024-03-18T20:26:11.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f93967fc700 1 -- 172.21.15.1:0/2064292301 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f939815d6b0 msgr2=0x7f93980b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:11.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f93967fc700 1 --2- 172.21.15.1:0/2064292301 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f939815d6b0 0x7f93980b2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f93967fc700 1 -- 172.21.15.1:0/2064292301 >> v2:172.21.15.165:3300/0 conn(0x7f93980ad790 msgr2=0x7f93980b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:11.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f93967fc700 1 --2- 172.21.15.1:0/2064292301 >> v2:172.21.15.165:3300/0 conn(0x7f93980ad790 0x7f93980b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f93967fc700 1 -- 172.21.15.1:0/2064292301 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93980af1e0 con 0x7f939806a750 2024-03-18T20:26:11.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f93967fc700 1 -- 172.21.15.1:0/2064292301 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f9388001670 con 0x7f939806a750 2024-03-18T20:26:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f93967fc700 1 -- 172.21.15.1:0/2064292301 >> v1:172.21.15.1:6789/0 conn(0x7f939806a750 legacy=0x7f939806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f93967fc700 1 --2- 172.21.15.1:0/2064292301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9384003a00 0x7f9384003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f93967fc700 1 --2- 172.21.15.1:0/2064292301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9384004850 0x7f9384004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f93967fc700 1 --2- 172.21.15.1:0/2064292301 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f939815d6b0 0x7f9384005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f93967fc700 1 -- 172.21.15.1:0/2064292301 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93980af7a0 con 0x7f9384004850 2024-03-18T20:26:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f93967fc700 1 -- 172.21.15.1:0/2064292301 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93980b50a0 con 0x7f9384003a00 2024-03-18T20:26:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f93967fc700 1 -- 172.21.15.1:0/2064292301 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f938400ba10 con 0x7f939815d6b0 2024-03-18T20:26:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f9396ffd700 1 --2- 172.21.15.1:0/2064292301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9384004850 0x7f9384004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f9397fff700 1 --2- 172.21.15.1:0/2064292301 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f939815d6b0 0x7f9384005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.635+0000 7f93977fe700 1 --2- 172.21.15.1:0/2064292301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9384003a00 0x7f9384003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.636+0000 7f9396ffd700 1 -- 172.21.15.1:0/2064292301 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f939815d6b0 msgr2=0x7f9384005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.636+0000 7f9396ffd700 1 --2- 172.21.15.1:0/2064292301 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f939815d6b0 0x7f9384005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.636+0000 7f9396ffd700 1 -- 172.21.15.1:0/2064292301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9384003a00 msgr2=0x7f9384003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.636+0000 7f9396ffd700 1 --2- 172.21.15.1:0/2064292301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9384003a00 0x7f9384003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.636+0000 7f9396ffd700 1 -- 172.21.15.1:0/2064292301 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93840035b0 con 0x7f9384004850 2024-03-18T20:26:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.636+0000 7f9396ffd700 1 --2- 172.21.15.1:0/2064292301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9384004850 0x7f9384004cd0 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7f9380009b60 tx=0x7f9384005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9735b5fa31d552b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.636+0000 7f93967fc700 1 -- 172.21.15.1:0/2064292301 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938000e6c0 con 0x7f9384004850 2024-03-18T20:26:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.636+0000 7f93967fc700 1 -- 172.21.15.1:0/2064292301 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938000e820 con 0x7f9384004850 2024-03-18T20:26:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.636+0000 7f93967fc700 1 -- 172.21.15.1:0/2064292301 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938000eb90 con 0x7f9384004850 2024-03-18T20:26:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.636+0000 7f939e171700 1 -- 172.21.15.1:0/2064292301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9384004850 msgr2=0x7f9384004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.636+0000 7f939e171700 1 --2- 172.21.15.1:0/2064292301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9384004850 0x7f9384004cd0 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7f9380009b60 tx=0x7f9384005880 comp rx=0 tx=0).stop 2024-03-18T20:26:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.636+0000 7f9397fff700 1 -- 172.21.15.1:0/2064292301 reap_dead start 2024-03-18T20:26:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.636+0000 7f939e171700 1 -- 172.21.15.1:0/2064292301 shutdown_connections 2024-03-18T20:26:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.636+0000 7f939e171700 1 -- 172.21.15.1:0/2064292301 >> 172.21.15.1:0/2064292301 conn(0x7f939806b860 msgr2=0x7f939806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.637+0000 7f939e171700 1 -- 172.21.15.1:0/2064292301 shutdown_connections 2024-03-18T20:26:11.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.637+0000 7f939e171700 1 -- 172.21.15.1:0/2064292301 wait complete. 2024-03-18T20:26:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.638+0000 7f939e171700 1 Processor -- start 2024-03-18T20:26:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.638+0000 7f939e171700 1 -- start start 2024-03-18T20:26:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.639+0000 7f939e171700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9384004850 0x7f939815f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.639+0000 7f939e171700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93981e98f0 0x7f9398160450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.639+0000 7f939e171700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9398154640 0x7f93981e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:11.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.639+0000 7f939e171700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9398160990 con 0x7f9398154640 2024-03-18T20:26:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.639+0000 7f939e171700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93981e7fc0 con 0x7f9384004850 2024-03-18T20:26:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.639+0000 7f939e171700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93981e82a0 con 0x7f93981e98f0 2024-03-18T20:26:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.639+0000 7f9397fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9398154640 0x7f93981e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.639+0000 7f9397fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9398154640 0x7f93981e7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43472/0 (socket says 172.21.15.1:43472) 2024-03-18T20:26:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.639+0000 7f9397fff700 1 -- 172.21.15.1:0/1959781256 learned_addr learned my addr 172.21.15.1:0/1959781256 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.640+0000 7f9396ffd700 1 --2- 172.21.15.1:0/1959781256 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93981e98f0 0x7f9398160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.640+0000 7f93977fe700 1 --2- 172.21.15.1:0/1959781256 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9384004850 0x7f939815f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.640+0000 7f9397fff700 1 -- 172.21.15.1:0/1959781256 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93981e98f0 msgr2=0x7f9398160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.640+0000 7f9397fff700 1 --2- 172.21.15.1:0/1959781256 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93981e98f0 0x7f9398160450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.640+0000 7f9397fff700 1 -- 172.21.15.1:0/1959781256 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9384004850 msgr2=0x7f939815f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.640+0000 7f9397fff700 1 --2- 172.21.15.1:0/1959781256 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9384004850 0x7f939815f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.640+0000 7f9397fff700 1 -- 172.21.15.1:0/1959781256 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939815e540 con 0x7f9398154640 2024-03-18T20:26:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.640+0000 7f9397fff700 1 --2- 172.21.15.1:0/1959781256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9398154640 0x7f93981e7280 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f93980726a0 tx=0x7f939000f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.640+0000 7f938ffff700 1 -- 172.21.15.1:0/1959781256 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939000e040 con 0x7f9398154640 2024-03-18T20:26:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.640+0000 7f938ffff700 1 -- 172.21.15.1:0/1959781256 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939000f040 con 0x7f9398154640 2024-03-18T20:26:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.640+0000 7f938ffff700 1 -- 172.21.15.1:0/1959781256 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9390015430 con 0x7f9398154640 2024-03-18T20:26:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.640+0000 7f939e171700 1 -- 172.21.15.1:0/1959781256 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f939815e820 con 0x7f9398154640 2024-03-18T20:26:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.640+0000 7f939e171700 1 -- 172.21.15.1:0/1959781256 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9398154aa0 con 0x7f9398154640 2024-03-18T20:26:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.641+0000 7f939e171700 1 -- 172.21.15.1:0/1959781256 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f939815ed40 con 0x7f9398154640 2024-03-18T20:26:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.641+0000 7f938ffff700 1 -- 172.21.15.1:0/1959781256 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f9390015590 con 0x7f9398154640 2024-03-18T20:26:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.642+0000 7f938ffff700 1 --2- 172.21.15.1:0/1959781256 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f937806aed0 0x7f937806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.642+0000 7f938ffff700 1 -- 172.21.15.1:0/1959781256 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 6958+0+0 (secure 0 0 0) 0x7f939008a350 con 0x7f9398154640 2024-03-18T20:26:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.642+0000 7f93977fe700 1 --2- 172.21.15.1:0/1959781256 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f937806aed0 0x7f937806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.642+0000 7f93977fe700 1 --2- 172.21.15.1:0/1959781256 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f937806aed0 0x7f937806d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9388000fb0 tx=0x7f9388009040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:11.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.646+0000 7f938ffff700 1 -- 172.21.15.1:0/1959781256 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f939815ed40 con 0x7f9398154640 2024-03-18T20:26:11.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.893+0000 7f939e171700 1 -- 172.21.15.1:0/1959781256 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f939815ed40 con 0x7f9398154640 2024-03-18T20:26:11.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.893+0000 7f938ffff700 1 -- 172.21.15.1:0/1959781256 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+4747 (secure 0 0 0) 0x7f939815ed40 con 0x7f9398154640 2024-03-18T20:26:11.899 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:11.899 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":112,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10369,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/677258734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":677258734},{"type":"v1","addr":"172.21.15.165:6835","nonce":677258734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/658650553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":658650553},{"type":"v1","addr":"172.21.15.165:6834","nonce":658650553}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":112},{"gid":11544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2018398457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2018398457},{"type":"v1","addr":"172.21.15.1:6836","nonce":2018398457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:25:35.549382+0000","modified":"2024-03-18T20:25:38.565325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11547":{"gid":11547,"name":"c","rank":0,"incarnation":106,"state":"up:active","state_seq":6,"addr":"172.21.15.1:6837/1635684321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1635684321},{"type":"v1","addr":"172.21.15.1:6837","nonce":1635684321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-03-18T20:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.895+0000 7f939e171700 1 -- 172.21.15.1:0/1959781256 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f937806aed0 msgr2=0x7f937806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.895+0000 7f939e171700 1 --2- 172.21.15.1:0/1959781256 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f937806aed0 0x7f937806d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9388000fb0 tx=0x7f9388009040 comp rx=0 tx=0).stop 2024-03-18T20:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.895+0000 7f939e171700 1 -- 172.21.15.1:0/1959781256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9398154640 msgr2=0x7f93981e7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.895+0000 7f939e171700 1 --2- 172.21.15.1:0/1959781256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9398154640 0x7f93981e7280 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f93980726a0 tx=0x7f939000f490 comp rx=0 tx=0).stop 2024-03-18T20:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.895+0000 7f939e171700 1 -- 172.21.15.1:0/1959781256 shutdown_connections 2024-03-18T20:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.895+0000 7f939e171700 1 --2- 172.21.15.1:0/1959781256 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f937806aed0 0x7f937806d390 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.895+0000 7f939e171700 1 --2- 172.21.15.1:0/1959781256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9398154640 0x7f93981e7280 unknown :-1 s=CLOSED pgs=2734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.895+0000 7f939e171700 1 --2- 172.21.15.1:0/1959781256 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9384004850 0x7f939815f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.895+0000 7f939e171700 1 --2- 172.21.15.1:0/1959781256 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93981e98f0 0x7f9398160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.895+0000 7f939e171700 1 -- 172.21.15.1:0/1959781256 >> 172.21.15.1:0/1959781256 conn(0x7f939806b860 msgr2=0x7f93980b0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.896+0000 7f939e171700 1 -- 172.21.15.1:0/1959781256 shutdown_connections 2024-03-18T20:26:11.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:11.896+0000 7f939e171700 1 -- 172.21.15.1:0/1959781256 wait complete. 2024-03-18T20:26:11.902 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 112 2024-03-18T20:26:11.913 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:26:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.033+0000 7ff6c354d700 1 Processor -- start 2024-03-18T20:26:12.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.033+0000 7ff6c354d700 1 -- start start 2024-03-18T20:26:12.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.034+0000 7ff6c354d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff6bc146190 0x7ff6bc06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.034+0000 7ff6c354d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff6bc1465a0 0x7ff6bc06ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.034+0000 7ff6c354d700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6bc06ebd0 con 0x7ff6bc146af0 2024-03-18T20:26:12.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.034+0000 7ff6c354d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6bc06f850 con 0x7ff6bc1465a0 2024-03-18T20:26:12.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.034+0000 7ff6c354d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6bc06eeb0 con 0x7ff6bc146190 2024-03-18T20:26:12.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.034+0000 7ff6c0ae8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff6bc1465a0 0x7ff6bc06ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.034+0000 7ff6c0ae8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff6bc1465a0 0x7ff6bc06ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43488/0 (socket says 172.21.15.1:43488) 2024-03-18T20:26:12.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.034+0000 7ff6c0ae8700 1 -- 172.21.15.1:0/1553110916 learned_addr learned my addr 172.21.15.1:0/1553110916 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:12.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.034+0000 7ff6c12e9700 1 --2- 172.21.15.1:0/1553110916 >> v2:172.21.15.165:3300/0 conn(0x7ff6bc146190 0x7ff6bc06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.034+0000 7ff6c0ae8700 1 -- 172.21.15.1:0/1553110916 >> v2:172.21.15.165:3300/0 conn(0x7ff6bc146190 msgr2=0x7ff6bc06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:12.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.034+0000 7ff6c0ae8700 1 --2- 172.21.15.1:0/1553110916 >> v2:172.21.15.165:3300/0 conn(0x7ff6bc146190 0x7ff6bc06a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.034+0000 7ff6c0ae8700 1 -- 172.21.15.1:0/1553110916 >> v1:172.21.15.165:6789/0 conn(0x7ff6bc146af0 legacy=0x7ff6bc071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:12.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.034+0000 7ff6c0ae8700 1 -- 172.21.15.1:0/1553110916 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6bc06f190 con 0x7ff6bc1465a0 2024-03-18T20:26:12.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.035+0000 7ff6c0ae8700 1 --2- 172.21.15.1:0/1553110916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6bc1465a0 0x7ff6bc06ac90 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc06e8f0 tx=0x7ff6b8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7ccd77453950ee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:12.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.035+0000 7ff6b3fff700 1 -- 172.21.15.1:0/1553110916 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b800b870 con 0x7ff6bc1465a0 2024-03-18T20:26:12.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.035+0000 7ff6b3fff700 1 -- 172.21.15.1:0/1553110916 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b800b9d0 con 0x7ff6bc1465a0 2024-03-18T20:26:12.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.035+0000 7ff6b3fff700 1 -- 172.21.15.1:0/1553110916 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b800bd60 con 0x7ff6bc1465a0 2024-03-18T20:26:12.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.035+0000 7ff6c354d700 1 -- 172.21.15.1:0/1553110916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6bc1465a0 msgr2=0x7ff6bc06ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:12.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.035+0000 7ff6c354d700 1 --2- 172.21.15.1:0/1553110916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6bc1465a0 0x7ff6bc06ac90 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc06e8f0 tx=0x7ff6b8009bd0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.035+0000 7ff6c354d700 1 -- 172.21.15.1:0/1553110916 shutdown_connections 2024-03-18T20:26:12.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.035+0000 7ff6c354d700 1 --2- 172.21.15.1:0/1553110916 >> v2:172.21.15.165:3300/0 conn(0x7ff6bc146190 0x7ff6bc06a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.035+0000 7ff6c354d700 1 --2- 172.21.15.1:0/1553110916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6bc1465a0 0x7ff6bc06ac90 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.035+0000 7ff6c354d700 1 -- 172.21.15.1:0/1553110916 >> 172.21.15.1:0/1553110916 conn(0x7ff6bc0b1d20 msgr2=0x7ff6bc0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:12.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.035+0000 7ff6c354d700 1 -- 172.21.15.1:0/1553110916 shutdown_connections 2024-03-18T20:26:12.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.035+0000 7ff6c354d700 1 -- 172.21.15.1:0/1553110916 wait complete. 2024-03-18T20:26:12.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.036+0000 7ff6c354d700 1 Processor -- start 2024-03-18T20:26:12.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.037+0000 7ff6c354d700 1 -- start start 2024-03-18T20:26:12.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.037+0000 7ff6c354d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6bc1465a0 0x7ff6bc15c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.037+0000 7ff6c354d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff6bc146af0 0x7ff6bc1417a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.037+0000 7ff6c354d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff6bc15c620 0x7ff6bc1e72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.037+0000 7ff6c354d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6bc1e6af0 con 0x7ff6bc1465a0 2024-03-18T20:26:12.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.037+0000 7ff6c354d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6bc1421f0 con 0x7ff6bc15c620 2024-03-18T20:26:12.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.037+0000 7ff6c354d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6bc1424d0 con 0x7ff6bc146af0 2024-03-18T20:26:12.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.037+0000 7ff6c12e9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6bc1465a0 0x7ff6bc15c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.037+0000 7ff6c12e9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6bc1465a0 0x7ff6bc15c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43498/0 (socket says 172.21.15.1:43498) 2024-03-18T20:26:12.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.037+0000 7ff6c12e9700 1 -- 172.21.15.1:0/3243406139 learned_addr learned my addr 172.21.15.1:0/3243406139 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:12.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.037+0000 7ff6c0ae8700 1 --2- 172.21.15.1:0/3243406139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff6bc146af0 0x7ff6bc1417a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.038+0000 7ff6c1aea700 1 --2- 172.21.15.1:0/3243406139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff6bc15c620 0x7ff6bc1e72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.038+0000 7ff6c12e9700 1 -- 172.21.15.1:0/3243406139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff6bc146af0 msgr2=0x7ff6bc1417a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:12.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.038+0000 7ff6c12e9700 1 --2- 172.21.15.1:0/3243406139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff6bc146af0 0x7ff6bc1417a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.038+0000 7ff6c12e9700 1 -- 172.21.15.1:0/3243406139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff6bc15c620 msgr2=0x7ff6bc1e72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:12.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.038+0000 7ff6c12e9700 1 --2- 172.21.15.1:0/3243406139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff6bc15c620 0x7ff6bc1e72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.038+0000 7ff6c12e9700 1 -- 172.21.15.1:0/3243406139 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b8009550 con 0x7ff6bc1465a0 2024-03-18T20:26:12.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.038+0000 7ff6c0ae8700 1 --2- 172.21.15.1:0/3243406139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff6bc146af0 0x7ff6bc1417a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:12.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.038+0000 7ff6c12e9700 1 --2- 172.21.15.1:0/3243406139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6bc1465a0 0x7ff6bc15c0e0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac00e810 tx=0x7ff6ac00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:12.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.038+0000 7ff6b1ffb700 1 -- 172.21.15.1:0/3243406139 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac013610 con 0x7ff6bc1465a0 2024-03-18T20:26:12.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.038+0000 7ff6b1ffb700 1 -- 172.21.15.1:0/3243406139 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6ac013770 con 0x7ff6bc1465a0 2024-03-18T20:26:12.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.038+0000 7ff6c354d700 1 -- 172.21.15.1:0/3243406139 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6bc1e6d90 con 0x7ff6bc1465a0 2024-03-18T20:26:12.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.038+0000 7ff6b1ffb700 1 -- 172.21.15.1:0/3243406139 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac013b20 con 0x7ff6bc1465a0 2024-03-18T20:26:12.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.038+0000 7ff6c354d700 1 -- 172.21.15.1:0/3243406139 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6bc13eb00 con 0x7ff6bc1465a0 2024-03-18T20:26:12.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.039+0000 7ff6c354d700 1 -- 172.21.15.1:0/3243406139 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6bc1e7100 con 0x7ff6bc1465a0 2024-03-18T20:26:12.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.040+0000 7ff6b1ffb700 1 -- 172.21.15.1:0/3243406139 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7ff6ac013c80 con 0x7ff6bc1465a0 2024-03-18T20:26:12.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.040+0000 7ff6b1ffb700 1 --2- 172.21.15.1:0/3243406139 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff6a006aed0 0x7ff6a006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.040+0000 7ff6b1ffb700 1 -- 172.21.15.1:0/3243406139 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 6958+0+0 (secure 0 0 0) 0x7ff6ac0899b0 con 0x7ff6bc1465a0 2024-03-18T20:26:12.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.040+0000 7ff6c0ae8700 1 --2- 172.21.15.1:0/3243406139 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff6a006aed0 0x7ff6a006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.040+0000 7ff6c0ae8700 1 --2- 172.21.15.1:0/3243406139 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff6a006aed0 0x7ff6a006d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8000c00 tx=0x7ff6b8008030 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:12.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.044+0000 7ff6b1ffb700 1 -- 172.21.15.1:0/3243406139 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff6ac054250 con 0x7ff6bc1465a0 2024-03-18T20:26:12.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.292+0000 7ff6c354d700 1 -- 172.21.15.1:0/3243406139 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6bc0b3da0 con 0x7ff6bc1465a0 2024-03-18T20:26:12.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.292+0000 7ff6b1ffb700 1 -- 172.21.15.1:0/3243406139 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+4747 (secure 0 0 0) 0x7ff6ac057870 con 0x7ff6bc1465a0 2024-03-18T20:26:12.297 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:12.297 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":112,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10369,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/677258734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":677258734},{"type":"v1","addr":"172.21.15.165:6835","nonce":677258734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/658650553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":658650553},{"type":"v1","addr":"172.21.15.165:6834","nonce":658650553}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":112},{"gid":11544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2018398457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2018398457},{"type":"v1","addr":"172.21.15.1:6836","nonce":2018398457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:25:35.549382+0000","modified":"2024-03-18T20:25:38.565325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11547":{"gid":11547,"name":"c","rank":0,"incarnation":106,"state":"up:active","state_seq":6,"addr":"172.21.15.1:6837/1635684321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1635684321},{"type":"v1","addr":"172.21.15.1:6837","nonce":1635684321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-03-18T20:26:12.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.294+0000 7ff6c354d700 1 -- 172.21.15.1:0/3243406139 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff6a006aed0 msgr2=0x7ff6a006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:12.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.294+0000 7ff6c354d700 1 --2- 172.21.15.1:0/3243406139 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff6a006aed0 0x7ff6a006d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8000c00 tx=0x7ff6b8008030 comp rx=0 tx=0).stop 2024-03-18T20:26:12.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.294+0000 7ff6c354d700 1 -- 172.21.15.1:0/3243406139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6bc1465a0 msgr2=0x7ff6bc15c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:12.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.294+0000 7ff6c354d700 1 --2- 172.21.15.1:0/3243406139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6bc1465a0 0x7ff6bc15c0e0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac00e810 tx=0x7ff6ac00ebd0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.294+0000 7ff6c354d700 1 -- 172.21.15.1:0/3243406139 shutdown_connections 2024-03-18T20:26:12.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.294+0000 7ff6c354d700 1 --2- 172.21.15.1:0/3243406139 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff6a006aed0 0x7ff6a006d390 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.294+0000 7ff6c354d700 1 --2- 172.21.15.1:0/3243406139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff6bc15c620 0x7ff6bc1e72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.294+0000 7ff6c354d700 1 --2- 172.21.15.1:0/3243406139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6bc1465a0 0x7ff6bc15c0e0 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.294+0000 7ff6c354d700 1 --2- 172.21.15.1:0/3243406139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff6bc146af0 0x7ff6bc1417a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.295+0000 7ff6c354d700 1 -- 172.21.15.1:0/3243406139 >> 172.21.15.1:0/3243406139 conn(0x7ff6bc0b1d20 msgr2=0x7ff6bc0b1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:12.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.295+0000 7ff6c354d700 1 -- 172.21.15.1:0/3243406139 shutdown_connections 2024-03-18T20:26:12.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.295+0000 7ff6c354d700 1 -- 172.21.15.1:0/3243406139 wait complete. 2024-03-18T20:26:12.301 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 112 2024-03-18T20:26:12.313 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:26:12.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.432+0000 7fe65932c700 1 Processor -- start 2024-03-18T20:26:12.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.432+0000 7fe65932c700 1 -- start start 2024-03-18T20:26:12.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.433+0000 7fe65932c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6540d1ea0 0x7fe654168560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.433+0000 7fe65932c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6540cddc0 con 0x7fe6540cf090 2024-03-18T20:26:12.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.433+0000 7fe65932c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6540cdff0 con 0x7fe654160940 2024-03-18T20:26:12.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.433+0000 7fe65932c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540ce2d0 con 0x7fe6540d1ea0 2024-03-18T20:26:12.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.433+0000 7fe6527fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe6540cf090 0x7fe6540d1790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57810/0 (socket says 172.21.15.1:57810) 2024-03-18T20:26:12.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.433+0000 7fe6527fc700 1 -- 172.21.15.1:0/4079714581 learned_addr learned my addr 172.21.15.1:0/4079714581 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:12.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.433+0000 7fe6537fe700 1 --2- 172.21.15.1:0/4079714581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6540d1ea0 0x7fe654168560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.433+0000 7fe651ffb700 1 -- 172.21.15.1:0/4079714581 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954256676 0 0) 0x7fe6540cddc0 con 0x7fe6540cf090 2024-03-18T20:26:12.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.433+0000 7fe651ffb700 1 -- 172.21.15.1:0/4079714581 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe63c0035b0 con 0x7fe6540cf090 2024-03-18T20:26:12.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe651ffb700 1 -- 172.21.15.1:0/4079714581 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 406610626 0 0) 0x7fe63c0035b0 con 0x7fe6540cf090 2024-03-18T20:26:12.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe651ffb700 1 -- 172.21.15.1:0/4079714581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6540d1ea0 msgr2=0x7fe654168560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:12.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe651ffb700 1 --2- 172.21.15.1:0/4079714581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6540d1ea0 0x7fe654168560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe651ffb700 1 -- 172.21.15.1:0/4079714581 >> v1:172.21.15.165:6789/0 conn(0x7fe654160940 legacy=0x7fe6540d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:12.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe651ffb700 1 -- 172.21.15.1:0/4079714581 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6540ce5b0 con 0x7fe6540cf090 2024-03-18T20:26:12.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe651ffb700 1 -- 172.21.15.1:0/4079714581 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fe644001670 con 0x7fe6540cf090 2024-03-18T20:26:12.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe651ffb700 1 -- 172.21.15.1:0/4079714581 >> v1:172.21.15.1:6789/0 conn(0x7fe6540cf090 legacy=0x7fe6540d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:12.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe651ffb700 1 --2- 172.21.15.1:0/4079714581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6540d1ea0 0x7fe63c003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe651ffb700 1 --2- 172.21.15.1:0/4079714581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe63c003f70 0x7fe63c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe651ffb700 1 --2- 172.21.15.1:0/4079714581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe63c004d90 0x7fe63c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe651ffb700 1 -- 172.21.15.1:0/4079714581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540ce2d0 con 0x7fe63c003f70 2024-03-18T20:26:12.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe651ffb700 1 -- 172.21.15.1:0/4079714581 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c00ba70 con 0x7fe63c004d90 2024-03-18T20:26:12.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe651ffb700 1 -- 172.21.15.1:0/4079714581 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c00bc30 con 0x7fe6540d1ea0 2024-03-18T20:26:12.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe6527fc700 1 --2- 172.21.15.1:0/4079714581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe63c003f70 0x7fe63c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe652ffd700 1 --2- 172.21.15.1:0/4079714581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6540d1ea0 0x7fe63c003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe6537fe700 1 --2- 172.21.15.1:0/4079714581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe63c004d90 0x7fe63c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe6527fc700 1 -- 172.21.15.1:0/4079714581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6540d1ea0 msgr2=0x7fe63c003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:12.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe6527fc700 1 --2- 172.21.15.1:0/4079714581 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6540d1ea0 0x7fe63c003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe6527fc700 1 -- 172.21.15.1:0/4079714581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe63c004d90 msgr2=0x7fe63c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:12.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe6527fc700 1 --2- 172.21.15.1:0/4079714581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe63c004d90 0x7fe63c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe6527fc700 1 -- 172.21.15.1:0/4079714581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63c0035b0 con 0x7fe63c003f70 2024-03-18T20:26:12.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe6527fc700 1 --2- 172.21.15.1:0/4079714581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe63c003f70 0x7fe63c004850 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7fe64400c210 tx=0x7fe63c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1cad7f81152524 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:12.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.434+0000 7fe651ffb700 1 -- 172.21.15.1:0/4079714581 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64400cc90 con 0x7fe63c003f70 2024-03-18T20:26:12.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.435+0000 7fe651ffb700 1 -- 172.21.15.1:0/4079714581 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64400cdf0 con 0x7fe63c003f70 2024-03-18T20:26:12.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.435+0000 7fe651ffb700 1 -- 172.21.15.1:0/4079714581 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644005ba0 con 0x7fe63c003f70 2024-03-18T20:26:12.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.435+0000 7fe65932c700 1 -- 172.21.15.1:0/4079714581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe63c003f70 msgr2=0x7fe63c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:12.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.435+0000 7fe65932c700 1 --2- 172.21.15.1:0/4079714581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe63c003f70 0x7fe63c004850 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7fe64400c210 tx=0x7fe63c005880 comp rx=0 tx=0).stop 2024-03-18T20:26:12.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.435+0000 7fe6537fe700 1 -- 172.21.15.1:0/4079714581 reap_dead start 2024-03-18T20:26:12.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.435+0000 7fe65932c700 1 -- 172.21.15.1:0/4079714581 shutdown_connections 2024-03-18T20:26:12.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.435+0000 7fe65932c700 1 -- 172.21.15.1:0/4079714581 >> 172.21.15.1:0/4079714581 conn(0x7fe65409c520 msgr2=0x7fe6540d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:12.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.435+0000 7fe65932c700 1 -- 172.21.15.1:0/4079714581 shutdown_connections 2024-03-18T20:26:12.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.435+0000 7fe65932c700 1 -- 172.21.15.1:0/4079714581 wait complete. 2024-03-18T20:26:12.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.436+0000 7fe65932c700 1 Processor -- start 2024-03-18T20:26:12.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.437+0000 7fe65932c700 1 -- start start 2024-03-18T20:26:12.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.437+0000 7fe65932c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe63c003f70 0x7fe6540b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.437+0000 7fe65932c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6540afd80 0x7fe65414e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.437+0000 7fe65932c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6540cac00 0x7fe6540cb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.437+0000 7fe65932c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540b11e0 con 0x7fe6540afd80 2024-03-18T20:26:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.437+0000 7fe65932c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540cb5c0 con 0x7fe63c003f70 2024-03-18T20:26:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.437+0000 7fe65932c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65415ec60 con 0x7fe6540cac00 2024-03-18T20:26:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.437+0000 7fe6527fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6540afd80 0x7fe65414e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.437+0000 7fe6527fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6540afd80 0x7fe65414e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43512/0 (socket says 172.21.15.1:43512) 2024-03-18T20:26:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.437+0000 7fe6527fc700 1 -- 172.21.15.1:0/2190744312 learned_addr learned my addr 172.21.15.1:0/2190744312 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.438+0000 7fe6537fe700 1 --2- 172.21.15.1:0/2190744312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6540cac00 0x7fe6540cb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.438+0000 7fe652ffd700 1 --2- 172.21.15.1:0/2190744312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe63c003f70 0x7fe6540b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.438+0000 7fe6527fc700 1 -- 172.21.15.1:0/2190744312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6540cac00 msgr2=0x7fe6540cb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.438+0000 7fe6527fc700 1 --2- 172.21.15.1:0/2190744312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6540cac00 0x7fe6540cb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.438+0000 7fe6527fc700 1 -- 172.21.15.1:0/2190744312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe63c003f70 msgr2=0x7fe6540b0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:12.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.438+0000 7fe6527fc700 1 --2- 172.21.15.1:0/2190744312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe63c003f70 0x7fe6540b0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.438+0000 7fe6527fc700 1 -- 172.21.15.1:0/2190744312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6540b16d0 con 0x7fe6540afd80 2024-03-18T20:26:12.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.438+0000 7fe652ffd700 1 --2- 172.21.15.1:0/2190744312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe63c003f70 0x7fe6540b0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:12.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.438+0000 7fe6537fe700 1 --2- 172.21.15.1:0/2190744312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6540cac00 0x7fe6540cb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:12.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.438+0000 7fe6527fc700 1 --2- 172.21.15.1:0/2190744312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6540afd80 0x7fe65414e920 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7fe6440018b0 tx=0x7fe644001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:12.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.438+0000 7fe63b7fe700 1 -- 172.21.15.1:0/2190744312 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6440074e0 con 0x7fe6540afd80 2024-03-18T20:26:12.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.438+0000 7fe63b7fe700 1 -- 172.21.15.1:0/2190744312 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe644007640 con 0x7fe6540afd80 2024-03-18T20:26:12.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.438+0000 7fe65932c700 1 -- 172.21.15.1:0/2190744312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe65415e650 con 0x7fe6540afd80 2024-03-18T20:26:12.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.438+0000 7fe63b7fe700 1 -- 172.21.15.1:0/2190744312 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64401e620 con 0x7fe6540afd80 2024-03-18T20:26:12.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.438+0000 7fe65932c700 1 -- 172.21.15.1:0/2190744312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6540b13f0 con 0x7fe6540afd80 2024-03-18T20:26:12.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.439+0000 7fe65932c700 1 -- 172.21.15.1:0/2190744312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe654073350 con 0x7fe6540afd80 2024-03-18T20:26:12.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.439+0000 7fe63b7fe700 1 -- 172.21.15.1:0/2190744312 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fe644019030 con 0x7fe6540afd80 2024-03-18T20:26:12.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.440+0000 7fe63b7fe700 1 --2- 172.21.15.1:0/2190744312 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe64006aed0 0x7fe64006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.440+0000 7fe63b7fe700 1 -- 172.21.15.1:0/2190744312 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 6958+0+0 (secure 0 0 0) 0x7fe64408ab80 con 0x7fe6540afd80 2024-03-18T20:26:12.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.440+0000 7fe652ffd700 1 --2- 172.21.15.1:0/2190744312 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe64006aed0 0x7fe64006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.440+0000 7fe652ffd700 1 --2- 172.21.15.1:0/2190744312 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe64006aed0 0x7fe64006d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe6540b5640 tx=0x7fe648009040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:12.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.444+0000 7fe63b7fe700 1 -- 172.21.15.1:0/2190744312 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe6440555d0 con 0x7fe6540afd80 2024-03-18T20:26:12.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.648+0000 7fe65932c700 1 -- 172.21.15.1:0/2190744312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6540d0980 con 0x7fe6540afd80 2024-03-18T20:26:12.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.650+0000 7fe63b7fe700 1 -- 172.21.15.1:0/2190744312 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v241) v1 ==== 74+0+17537 (secure 0 0 0) 0x7fe644058bf0 con 0x7fe6540afd80 2024-03-18T20:26:12.655 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:12.655 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":241,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:26:09.280336+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:25:33.275375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"235","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:25:34.518347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.165:6834/4024828630":"2024-03-19T20:26:06.260180+0000","172.21.15.1:0/4210355381":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/1879196326":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/101306769":"2024-03-19T20:26:03.849558+0000","172.21.15.165:6832/4024828630":"2024-03-19T20:26:06.260180+0000","172.21.15.1:0/997525540":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/1622795270":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/2535232994":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2652258802":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2404324807":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2059399259":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/3657737679":"2024-03-19T20:25:52.933722+0000","172.21.15.1:6833/148356":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/2685347289":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/1965726293":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/2324250327":"2024-03-19T20:25:52.933722+0000","172.21.15.1:6832/148356":"2024-03-19T20:25:52.933722+0000","172.21.15.1:0/1022040508":"2024-03-19T20:26:03.849558+0000","172.21.15.1:0/2165965129":"2024-03-19T20:25:52.933722+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:26:12.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.652+0000 7fe65932c700 1 -- 172.21.15.1:0/2190744312 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe64006aed0 msgr2=0x7fe64006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:12.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.652+0000 7fe65932c700 1 --2- 172.21.15.1:0/2190744312 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe64006aed0 0x7fe64006d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe6540b5640 tx=0x7fe648009040 comp rx=0 tx=0).stop 2024-03-18T20:26:12.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.652+0000 7fe65932c700 1 -- 172.21.15.1:0/2190744312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6540afd80 msgr2=0x7fe65414e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:12.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.652+0000 7fe65932c700 1 --2- 172.21.15.1:0/2190744312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6540afd80 0x7fe65414e920 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7fe6440018b0 tx=0x7fe644001980 comp rx=0 tx=0).stop 2024-03-18T20:26:12.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.652+0000 7fe65932c700 1 -- 172.21.15.1:0/2190744312 shutdown_connections 2024-03-18T20:26:12.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.652+0000 7fe65932c700 1 --2- 172.21.15.1:0/2190744312 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe64006aed0 0x7fe64006d390 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.652+0000 7fe65932c700 1 --2- 172.21.15.1:0/2190744312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6540cac00 0x7fe6540cb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.652+0000 7fe65932c700 1 --2- 172.21.15.1:0/2190744312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe63c003f70 0x7fe6540b0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.652+0000 7fe65932c700 1 --2- 172.21.15.1:0/2190744312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6540afd80 0x7fe65414e920 unknown :-1 s=CLOSED pgs=2739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.652+0000 7fe65932c700 1 -- 172.21.15.1:0/2190744312 >> 172.21.15.1:0/2190744312 conn(0x7fe65409c520 msgr2=0x7fe65406b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:12.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.652+0000 7fe65932c700 1 -- 172.21.15.1:0/2190744312 shutdown_connections 2024-03-18T20:26:12.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.652+0000 7fe65932c700 1 -- 172.21.15.1:0/2190744312 wait complete. 2024-03-18T20:26:12.670 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2024-03-18T20:26:12.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.791+0000 7f5d4c1b8700 1 Processor -- start 2024-03-18T20:26:12.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.791+0000 7f5d4c1b8700 1 -- start start 2024-03-18T20:26:12.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.792+0000 7f5d4c1b8700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5d4415d5a0 0x7f5d440ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.792+0000 7f5d4c1b8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5d440ce550 0x7f5d4406a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.792+0000 7f5d4c1b8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5d440cfa90 0x7f5d4406ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.792+0000 7f5d4c1b8700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d440cf600 con 0x7f5d440ce550 2024-03-18T20:26:12.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.792+0000 7f5d4c1b8700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d440ce9f0 con 0x7f5d4415d5a0 2024-03-18T20:26:12.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.792+0000 7f5d4c1b8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d440cecd0 con 0x7f5d440cfa90 2024-03-18T20:26:12.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.792+0000 7f5d49753700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5d440ce550 0x7f5d4406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.792+0000 7f5d49753700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5d440ce550 0x7f5d4406a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43522/0 (socket says 172.21.15.1:43522) 2024-03-18T20:26:12.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.792+0000 7f5d49753700 1 -- 172.21.15.1:0/2487437431 learned_addr learned my addr 172.21.15.1:0/2487437431 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:12.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.792+0000 7f5d4a755700 1 --2- 172.21.15.1:0/2487437431 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5d440cfa90 0x7f5d4406ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.792+0000 7f5d49f54700 1 --2- 172.21.15.1:0/2487437431 >> v2:172.21.15.165:3300/0 conn(0x7f5d4415d5a0 0x7f5d440ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.792+0000 7f5d49753700 1 -- 172.21.15.1:0/2487437431 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5d440cfa90 msgr2=0x7f5d4406ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:12.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.792+0000 7f5d49753700 1 --2- 172.21.15.1:0/2487437431 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5d440cfa90 0x7f5d4406ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.792+0000 7f5d49753700 1 -- 172.21.15.1:0/2487437431 >> v2:172.21.15.165:3300/0 conn(0x7f5d4415d5a0 msgr2=0x7f5d440ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:12.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.792+0000 7f5d49753700 1 --2- 172.21.15.1:0/2487437431 >> v2:172.21.15.165:3300/0 conn(0x7f5d4415d5a0 0x7f5d440ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.792+0000 7f5d49753700 1 -- 172.21.15.1:0/2487437431 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d440cefb0 con 0x7f5d440ce550 2024-03-18T20:26:12.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.792+0000 7f5d49753700 1 --2- 172.21.15.1:0/2487437431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d440ce550 0x7f5d4406a750 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7f5d44072f60 tx=0x7f5d34009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb57209b3ce1ccce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:12.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.792+0000 7f5d48f52700 1 -- 172.21.15.1:0/2487437431 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3400b870 con 0x7f5d440ce550 2024-03-18T20:26:12.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.793+0000 7f5d48f52700 1 -- 172.21.15.1:0/2487437431 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3400b9d0 con 0x7f5d440ce550 2024-03-18T20:26:12.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.793+0000 7f5d48f52700 1 -- 172.21.15.1:0/2487437431 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3400bd60 con 0x7f5d440ce550 2024-03-18T20:26:12.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.793+0000 7f5d4c1b8700 1 -- 172.21.15.1:0/2487437431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d440ce550 msgr2=0x7f5d4406a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:12.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.793+0000 7f5d4c1b8700 1 --2- 172.21.15.1:0/2487437431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d440ce550 0x7f5d4406a750 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7f5d44072f60 tx=0x7f5d34009bd0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.793+0000 7f5d4c1b8700 1 -- 172.21.15.1:0/2487437431 shutdown_connections 2024-03-18T20:26:12.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.793+0000 7f5d4c1b8700 1 --2- 172.21.15.1:0/2487437431 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5d440cfa90 0x7f5d4406ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.793+0000 7f5d4c1b8700 1 --2- 172.21.15.1:0/2487437431 >> v2:172.21.15.165:3300/0 conn(0x7f5d4415d5a0 0x7f5d440ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.793+0000 7f5d4c1b8700 1 --2- 172.21.15.1:0/2487437431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d440ce550 0x7f5d4406a750 unknown :-1 s=CLOSED pgs=2740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.793+0000 7f5d4c1b8700 1 -- 172.21.15.1:0/2487437431 >> 172.21.15.1:0/2487437431 conn(0x7f5d440cc9b0 msgr2=0x7f5d440ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:12.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.793+0000 7f5d4c1b8700 1 -- 172.21.15.1:0/2487437431 shutdown_connections 2024-03-18T20:26:12.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.793+0000 7f5d4c1b8700 1 -- 172.21.15.1:0/2487437431 wait complete. 2024-03-18T20:26:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.794+0000 7f5d4c1b8700 1 Processor -- start 2024-03-18T20:26:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.794+0000 7f5d4c1b8700 1 -- start start 2024-03-18T20:26:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.795+0000 7f5d4c1b8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5d440ce550 0x7f5d44154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.795+0000 7f5d4c1b8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d440cfa90 0x7f5d4415e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.795+0000 7f5d4c1b8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d4415d5a0 0x7f5d44153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.795+0000 7f5d4c1b8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d441600d0 con 0x7f5d440cfa90 2024-03-18T20:26:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.795+0000 7f5d4c1b8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4415fdf0 con 0x7f5d4415d5a0 2024-03-18T20:26:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.795+0000 7f5d4c1b8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d440c71a0 con 0x7f5d440ce550 2024-03-18T20:26:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.795+0000 7f5d49753700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d440cfa90 0x7f5d4415e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.795+0000 7f5d49753700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d440cfa90 0x7f5d4415e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43536/0 (socket says 172.21.15.1:43536) 2024-03-18T20:26:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.795+0000 7f5d49753700 1 -- 172.21.15.1:0/336426727 learned_addr learned my addr 172.21.15.1:0/336426727 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.795+0000 7f5d49f54700 1 --2- 172.21.15.1:0/336426727 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5d440ce550 0x7f5d44154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.795+0000 7f5d4a755700 1 --2- 172.21.15.1:0/336426727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d4415d5a0 0x7f5d44153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.795+0000 7f5d49753700 1 -- 172.21.15.1:0/336426727 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5d440ce550 msgr2=0x7f5d44154520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:12.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.795+0000 7f5d49753700 1 --2- 172.21.15.1:0/336426727 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5d440ce550 0x7f5d44154520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.795+0000 7f5d49753700 1 -- 172.21.15.1:0/336426727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d4415d5a0 msgr2=0x7f5d44153b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:12.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.795+0000 7f5d49753700 1 --2- 172.21.15.1:0/336426727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d4415d5a0 0x7f5d44153b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:12.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.795+0000 7f5d49753700 1 -- 172.21.15.1:0/336426727 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d34009550 con 0x7f5d440cfa90 2024-03-18T20:26:12.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.796+0000 7f5d49753700 1 --2- 172.21.15.1:0/336426727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d440cfa90 0x7f5d4415e6a0 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7f5d34000c00 tx=0x7f5d34004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:12.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.796+0000 7f5d3a7fc700 1 -- 172.21.15.1:0/336426727 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d34004480 con 0x7f5d440cfa90 2024-03-18T20:26:12.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.796+0000 7f5d3a7fc700 1 -- 172.21.15.1:0/336426727 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d340045e0 con 0x7f5d440cfa90 2024-03-18T20:26:12.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.796+0000 7f5d4c1b8700 1 -- 172.21.15.1:0/336426727 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d440c7480 con 0x7f5d440cfa90 2024-03-18T20:26:12.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.796+0000 7f5d3a7fc700 1 -- 172.21.15.1:0/336426727 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d340038e0 con 0x7f5d440cfa90 2024-03-18T20:26:12.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.796+0000 7f5d4c1b8700 1 -- 172.21.15.1:0/336426727 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d4415ebe0 con 0x7f5d440cfa90 2024-03-18T20:26:12.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.797+0000 7f5d4c1b8700 1 -- 172.21.15.1:0/336426727 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d44072960 con 0x7f5d440cfa90 2024-03-18T20:26:12.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.797+0000 7f5d3a7fc700 1 -- 172.21.15.1:0/336426727 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f5d34021070 con 0x7f5d440cfa90 2024-03-18T20:26:12.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.798+0000 7f5d3a7fc700 1 --2- 172.21.15.1:0/336426727 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5d2806aed0 0x7f5d2806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:12.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.798+0000 7f5d3a7fc700 1 -- 172.21.15.1:0/336426727 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 6958+0+0 (secure 0 0 0) 0x7f5d3408b4e0 con 0x7f5d440cfa90 2024-03-18T20:26:12.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.798+0000 7f5d49f54700 1 --2- 172.21.15.1:0/336426727 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5d2806aed0 0x7f5d2806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:12.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.798+0000 7f5d49f54700 1 --2- 172.21.15.1:0/336426727 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5d2806aed0 0x7f5d2806d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5d40006fd0 tx=0x7f5d40008040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:12.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:12.803+0000 7f5d3a7fc700 1 -- 172.21.15.1:0/336426727 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5d34056030 con 0x7f5d440cfa90 2024-03-18T20:26:13.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.051+0000 7f5d4c1b8700 1 -- 172.21.15.1:0/336426727 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7f5d440c6fe0 con 0x7f5d440cfa90 2024-03-18T20:26:13.058 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-18T20:26:13.052+0000 7fbdbeb46700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T20:26:13.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.064+0000 7f5d3a7fc700 1 -- 172.21.15.1:0/336426727 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v113) v1 ==== 167+0+0 (secure 0 0 0) 0x7f5d34019e60 con 0x7f5d440cfa90 2024-03-18T20:26:13.071 INFO:tasks.ceph.mds.c.smithi001.stderr: -1958> 2024-03-18T20:25:38.216+0000 7fc643c72700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T20:26:13.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.066+0000 7f5d4c1b8700 1 -- 172.21.15.1:0/336426727 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5d2806aed0 msgr2=0x7f5d2806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:13.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.066+0000 7f5d4c1b8700 1 --2- 172.21.15.1:0/336426727 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5d2806aed0 0x7f5d2806d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5d40006fd0 tx=0x7f5d40008040 comp rx=0 tx=0).stop 2024-03-18T20:26:13.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.066+0000 7f5d4c1b8700 1 -- 172.21.15.1:0/336426727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d440cfa90 msgr2=0x7f5d4415e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:13.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.066+0000 7f5d4c1b8700 1 --2- 172.21.15.1:0/336426727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d440cfa90 0x7f5d4415e6a0 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7f5d34000c00 tx=0x7f5d34004750 comp rx=0 tx=0).stop 2024-03-18T20:26:13.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.066+0000 7f5d4c1b8700 1 -- 172.21.15.1:0/336426727 shutdown_connections 2024-03-18T20:26:13.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.066+0000 7f5d4c1b8700 1 --2- 172.21.15.1:0/336426727 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5d2806aed0 0x7f5d2806d390 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:13.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.066+0000 7f5d4c1b8700 1 --2- 172.21.15.1:0/336426727 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5d440ce550 0x7f5d44154520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:13.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.066+0000 7f5d4c1b8700 1 --2- 172.21.15.1:0/336426727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d4415d5a0 0x7f5d44153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:13.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.066+0000 7f5d4c1b8700 1 --2- 172.21.15.1:0/336426727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d440cfa90 0x7f5d4415e6a0 unknown :-1 s=CLOSED pgs=2741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:13.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.066+0000 7f5d4c1b8700 1 -- 172.21.15.1:0/336426727 >> 172.21.15.1:0/336426727 conn(0x7f5d440cc9b0 msgr2=0x7f5d440ad340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:13.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.066+0000 7f5d4c1b8700 1 -- 172.21.15.1:0/336426727 shutdown_connections 2024-03-18T20:26:13.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.066+0000 7f5d4c1b8700 1 -- 172.21.15.1:0/336426727 wait complete. 2024-03-18T20:26:13.073 INFO:teuthology.orchestra.run.smithi001.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T20:26:13.085 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2024-03-18T20:26:13.098 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T20:26:13.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.208+0000 7fd8c3ccc700 1 Processor -- start 2024-03-18T20:26:13.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.208+0000 7fd8c3ccc700 1 -- start start 2024-03-18T20:26:13.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.209+0000 7fd8c3ccc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8bc071d30 0x7fd8bc0add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:13.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.209+0000 7fd8c3ccc700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd8bc0cb9e0 0x7fd8bc0cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:13.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.209+0000 7fd8c3ccc700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc0ce8f0 con 0x7fd8bc15a1d0 2024-03-18T20:26:13.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.209+0000 7fd8c3ccc700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8bc0cdb00 con 0x7fd8bc0cb9e0 2024-03-18T20:26:13.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.209+0000 7fd8c3ccc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc15a5b0 con 0x7fd8bc071d30 2024-03-18T20:26:13.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.209+0000 7fd8c1a68700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd8bc15a1d0 0x7fd8bc071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57812/0 (socket says 172.21.15.1:57812) 2024-03-18T20:26:13.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.209+0000 7fd8c1a68700 1 -- 172.21.15.1:0/909280241 learned_addr learned my addr 172.21.15.1:0/909280241 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:13.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.209+0000 7fd8c2269700 1 --2- 172.21.15.1:0/909280241 >> v2:172.21.15.165:3300/0 conn(0x7fd8bc0cb9e0 0x7fd8bc0cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:13.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.209+0000 7fd8c1267700 1 --2- 172.21.15.1:0/909280241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8bc071d30 0x7fd8bc0add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:13.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.209+0000 7fd8c0a66700 1 -- 172.21.15.1:0/909280241 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751124794 0 0) 0x7fd8bc0ce8f0 con 0x7fd8bc15a1d0 2024-03-18T20:26:13.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.209+0000 7fd8c0a66700 1 -- 172.21.15.1:0/909280241 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8a80035b0 con 0x7fd8bc15a1d0 2024-03-18T20:26:13.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c0a66700 1 -- 172.21.15.1:0/909280241 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2240515350 0 0) 0x7fd8a80035b0 con 0x7fd8bc15a1d0 2024-03-18T20:26:13.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c0a66700 1 -- 172.21.15.1:0/909280241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8bc071d30 msgr2=0x7fd8bc0add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:13.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c0a66700 1 --2- 172.21.15.1:0/909280241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8bc071d30 0x7fd8bc0add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:13.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c0a66700 1 -- 172.21.15.1:0/909280241 >> v2:172.21.15.165:3300/0 conn(0x7fd8bc0cb9e0 msgr2=0x7fd8bc0cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:13.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c0a66700 1 --2- 172.21.15.1:0/909280241 >> v2:172.21.15.165:3300/0 conn(0x7fd8bc0cb9e0 0x7fd8bc0cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:13.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c0a66700 1 -- 172.21.15.1:0/909280241 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc0ae260 con 0x7fd8bc15a1d0 2024-03-18T20:26:13.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c0a66700 1 -- 172.21.15.1:0/909280241 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fd8ac0017f0 con 0x7fd8bc15a1d0 2024-03-18T20:26:13.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c0a66700 1 -- 172.21.15.1:0/909280241 >> v1:172.21.15.1:6789/0 conn(0x7fd8bc15a1d0 legacy=0x7fd8bc071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:13.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c0a66700 1 --2- 172.21.15.1:0/909280241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8bc071d30 0x7fd8a8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:13.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c0a66700 1 --2- 172.21.15.1:0/909280241 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8a8003f80 0x7fd8a8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:13.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c0a66700 1 --2- 172.21.15.1:0/909280241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a8004d90 0x7fd8a8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:13.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c0a66700 1 -- 172.21.15.1:0/909280241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc0cdb00 con 0x7fd8a8004d90 2024-03-18T20:26:13.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c0a66700 1 -- 172.21.15.1:0/909280241 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a800ba10 con 0x7fd8a8003f80 2024-03-18T20:26:13.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c0a66700 1 -- 172.21.15.1:0/909280241 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a800bb50 con 0x7fd8bc071d30 2024-03-18T20:26:13.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c2269700 1 --2- 172.21.15.1:0/909280241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a8004d90 0x7fd8a8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:13.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c1a68700 1 --2- 172.21.15.1:0/909280241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8bc071d30 0x7fd8a8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:13.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c1267700 1 --2- 172.21.15.1:0/909280241 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8a8003f80 0x7fd8a8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:13.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c2269700 1 -- 172.21.15.1:0/909280241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8bc071d30 msgr2=0x7fd8a8003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:13.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c2269700 1 --2- 172.21.15.1:0/909280241 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8bc071d30 0x7fd8a8003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:13.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c2269700 1 -- 172.21.15.1:0/909280241 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8a8003f80 msgr2=0x7fd8a8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:13.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c2269700 1 --2- 172.21.15.1:0/909280241 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8a8003f80 0x7fd8a8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:13.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c2269700 1 -- 172.21.15.1:0/909280241 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a80035b0 con 0x7fd8a8004d90 2024-03-18T20:26:13.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.210+0000 7fd8c2269700 1 --2- 172.21.15.1:0/909280241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a8004d90 0x7fd8a8005210 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009fc0 tx=0x7fd8a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609c65f6f5644818 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:13.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.211+0000 7fd8c0a66700 1 -- 172.21.15.1:0/909280241 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b400ebd0 con 0x7fd8a8004d90 2024-03-18T20:26:13.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.211+0000 7fd8c0a66700 1 -- 172.21.15.1:0/909280241 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b400f040 con 0x7fd8a8004d90 2024-03-18T20:26:13.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.211+0000 7fd8c0a66700 1 -- 172.21.15.1:0/909280241 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4014480 con 0x7fd8a8004d90 2024-03-18T20:26:13.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.211+0000 7fd8c3ccc700 1 -- 172.21.15.1:0/909280241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a8004d90 msgr2=0x7fd8a8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:13.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.211+0000 7fd8c3ccc700 1 --2- 172.21.15.1:0/909280241 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8a8004d90 0x7fd8a8005210 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009fc0 tx=0x7fd8a8005880 comp rx=0 tx=0).stop 2024-03-18T20:26:13.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.211+0000 7fd8c2269700 1 -- 172.21.15.1:0/909280241 reap_dead start 2024-03-18T20:26:13.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.211+0000 7fd8c3ccc700 1 -- 172.21.15.1:0/909280241 shutdown_connections 2024-03-18T20:26:13.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.211+0000 7fd8c3ccc700 1 -- 172.21.15.1:0/909280241 >> 172.21.15.1:0/909280241 conn(0x7fd8bc0cc9b0 msgr2=0x7fd8bc0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:13.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.211+0000 7fd8c3ccc700 1 -- 172.21.15.1:0/909280241 shutdown_connections 2024-03-18T20:26:13.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.211+0000 7fd8c3ccc700 1 -- 172.21.15.1:0/909280241 wait complete. 2024-03-18T20:26:13.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.212+0000 7fd8c3ccc700 1 Processor -- start 2024-03-18T20:26:13.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.213+0000 7fd8c3ccc700 1 -- start start 2024-03-18T20:26:13.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.213+0000 7fd8c3ccc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8bc150f90 0x7fd8bc1f0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:13.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.213+0000 7fd8c3ccc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc09a8f0 0x7fd8bc09ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:13.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.213+0000 7fd8c3ccc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8bc09ad70 0x7fd8bc09bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:13.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.213+0000 7fd8c3ccc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc0b3260 con 0x7fd8bc09a8f0 2024-03-18T20:26:13.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.213+0000 7fd8c3ccc700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc151370 con 0x7fd8bc09ad70 2024-03-18T20:26:13.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.213+0000 7fd8c3ccc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc0b57f0 con 0x7fd8bc150f90 2024-03-18T20:26:13.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.213+0000 7fd8c1267700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc09a8f0 0x7fd8bc09ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:13.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.213+0000 7fd8c1267700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc09a8f0 0x7fd8bc09ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43550/0 (socket says 172.21.15.1:43550) 2024-03-18T20:26:13.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.213+0000 7fd8c1267700 1 -- 172.21.15.1:0/1510473584 learned_addr learned my addr 172.21.15.1:0/1510473584 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:13.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.214+0000 7fd8c2269700 1 --2- 172.21.15.1:0/1510473584 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8bc09ad70 0x7fd8bc09bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:13.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.214+0000 7fd8c1a68700 1 --2- 172.21.15.1:0/1510473584 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8bc150f90 0x7fd8bc1f0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:13.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.214+0000 7fd8c1267700 1 -- 172.21.15.1:0/1510473584 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8bc150f90 msgr2=0x7fd8bc1f0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:13.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.214+0000 7fd8c1267700 1 --2- 172.21.15.1:0/1510473584 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8bc150f90 0x7fd8bc1f0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:13.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.214+0000 7fd8c1267700 1 -- 172.21.15.1:0/1510473584 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8bc09ad70 msgr2=0x7fd8bc09bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:13.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.214+0000 7fd8c1267700 1 --2- 172.21.15.1:0/1510473584 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8bc09ad70 0x7fd8bc09bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:13.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.214+0000 7fd8c1267700 1 -- 172.21.15.1:0/1510473584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc0b2ba0 con 0x7fd8bc09a8f0 2024-03-18T20:26:13.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.214+0000 7fd8c1267700 1 --2- 172.21.15.1:0/1510473584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc09a8f0 0x7fd8bc09ba80 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7fd8b80119a0 tx=0x7fd8b8011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:13.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.214+0000 7fd8b27fc700 1 -- 172.21.15.1:0/1510473584 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b800f940 con 0x7fd8bc09a8f0 2024-03-18T20:26:13.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.214+0000 7fd8b27fc700 1 -- 172.21.15.1:0/1510473584 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b800faa0 con 0x7fd8bc09a8f0 2024-03-18T20:26:13.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.214+0000 7fd8c3ccc700 1 -- 172.21.15.1:0/1510473584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8bc0b2e80 con 0x7fd8bc09a8f0 2024-03-18T20:26:13.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.214+0000 7fd8b27fc700 1 -- 172.21.15.1:0/1510473584 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b800fe30 con 0x7fd8bc09a8f0 2024-03-18T20:26:13.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.214+0000 7fd8c3ccc700 1 -- 172.21.15.1:0/1510473584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8bc0b53c0 con 0x7fd8bc09a8f0 2024-03-18T20:26:13.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.215+0000 7fd8c3ccc700 1 -- 172.21.15.1:0/1510473584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8bc0b34d0 con 0x7fd8bc09a8f0 2024-03-18T20:26:13.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.216+0000 7fd8b27fc700 1 -- 172.21.15.1:0/1510473584 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fd8b8014440 con 0x7fd8bc09a8f0 2024-03-18T20:26:13.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.216+0000 7fd8b27fc700 1 --2- 172.21.15.1:0/1510473584 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd8a006aed0 0x7fd8a006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:13.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.216+0000 7fd8b27fc700 1 -- 172.21.15.1:0/1510473584 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 7044+0+0 (secure 0 0 0) 0x7fd8b808a320 con 0x7fd8bc09a8f0 2024-03-18T20:26:13.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.216+0000 7fd8c1a68700 1 --2- 172.21.15.1:0/1510473584 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd8a006aed0 0x7fd8a006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:13.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.216+0000 7fd8c1a68700 1 --2- 172.21.15.1:0/1510473584 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd8a006aed0 0x7fd8a006d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac000fd0 tx=0x7fd8ac009040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:13.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.221+0000 7fd8b27fc700 1 -- 172.21.15.1:0/1510473584 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd8b8054cf0 con 0x7fd8bc09a8f0 2024-03-18T20:26:13.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:13.469+0000 7fd8c3ccc700 1 -- 172.21.15.1:0/1510473584 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd8bc0acd80 con 0x7fd8bc09a8f0 2024-03-18T20:26:14.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.100+0000 7fd8b27fc700 1 -- 172.21.15.1:0/1510473584 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v114) v1 ==== 107+0+0 (secure 0 0 0) 0x7fd8b8058310 con 0x7fd8bc09a8f0 2024-03-18T20:26:14.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.103+0000 7fd8c3ccc700 1 -- 172.21.15.1:0/1510473584 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd8a006aed0 msgr2=0x7fd8a006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:14.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.103+0000 7fd8c3ccc700 1 --2- 172.21.15.1:0/1510473584 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd8a006aed0 0x7fd8a006d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac000fd0 tx=0x7fd8ac009040 comp rx=0 tx=0).stop 2024-03-18T20:26:14.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.103+0000 7fd8c3ccc700 1 -- 172.21.15.1:0/1510473584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc09a8f0 msgr2=0x7fd8bc09ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:14.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.103+0000 7fd8c3ccc700 1 --2- 172.21.15.1:0/1510473584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc09a8f0 0x7fd8bc09ba80 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7fd8b80119a0 tx=0x7fd8b8011d60 comp rx=0 tx=0).stop 2024-03-18T20:26:14.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.103+0000 7fd8c3ccc700 1 -- 172.21.15.1:0/1510473584 shutdown_connections 2024-03-18T20:26:14.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.103+0000 7fd8c3ccc700 1 --2- 172.21.15.1:0/1510473584 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd8a006aed0 0x7fd8a006d390 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:14.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.103+0000 7fd8c3ccc700 1 --2- 172.21.15.1:0/1510473584 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8bc150f90 0x7fd8bc1f0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:14.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.103+0000 7fd8c3ccc700 1 --2- 172.21.15.1:0/1510473584 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8bc09ad70 0x7fd8bc09bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:14.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.103+0000 7fd8c3ccc700 1 --2- 172.21.15.1:0/1510473584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc09a8f0 0x7fd8bc09ba80 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:14.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.103+0000 7fd8c3ccc700 1 -- 172.21.15.1:0/1510473584 >> 172.21.15.1:0/1510473584 conn(0x7fd8bc0cc9b0 msgr2=0x7fd8bc06f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:14.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.104+0000 7fd8c3ccc700 1 -- 172.21.15.1:0/1510473584 shutdown_connections 2024-03-18T20:26:14.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.104+0000 7fd8c3ccc700 1 -- 172.21.15.1:0/1510473584 wait complete. 2024-03-18T20:26:14.124 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2024-03-18T20:26:14.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.299+0000 7f7d21577700 1 Processor -- start 2024-03-18T20:26:14.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.299+0000 7f7d21577700 1 -- start start 2024-03-18T20:26:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.300+0000 7f7d21577700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d1c15d8b0 0x7f7d1c0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.300+0000 7f7d21577700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7d1c0d0040 0x7f7d1c0cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.300+0000 7f7d21577700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d1c15dc90 con 0x7f7d1c0d1170 2024-03-18T20:26:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.300+0000 7f7d21577700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d1c15ec90 con 0x7f7d1c0d0040 2024-03-18T20:26:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.300+0000 7f7d21577700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c15ded0 con 0x7f7d1c15d8b0 2024-03-18T20:26:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.300+0000 7f7d1a7fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7d1c0d1170 0x7f7d1c0cf930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57820/0 (socket says 172.21.15.1:57820) 2024-03-18T20:26:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.300+0000 7f7d1a7fc700 1 -- 172.21.15.1:0/2872969419 learned_addr learned my addr 172.21.15.1:0/2872969419 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.300+0000 7f7d1b7fe700 1 --2- 172.21.15.1:0/2872969419 >> v2:172.21.15.165:3300/0 conn(0x7f7d1c0d0040 0x7f7d1c0cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.300+0000 7f7d1affd700 1 --2- 172.21.15.1:0/2872969419 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d1c15d8b0 0x7f7d1c0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:14.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.300+0000 7f7d19ffb700 1 -- 172.21.15.1:0/2872969419 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45453864 0 0) 0x7f7d10001110 con 0x7f7d1c0d1170 2024-03-18T20:26:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.300+0000 7f7d19ffb700 1 -- 172.21.15.1:0/2872969419 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d040035b0 con 0x7f7d1c0d1170 2024-03-18T20:26:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.300+0000 7f7d1b7fe700 1 -- 172.21.15.1:0/2872969419 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d1c15d8b0 msgr2=0x7f7d1c0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.300+0000 7f7d1b7fe700 1 --2- 172.21.15.1:0/2872969419 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d1c15d8b0 0x7f7d1c0d3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.300+0000 7f7d1b7fe700 1 -- 172.21.15.1:0/2872969419 >> v1:172.21.15.1:6789/0 conn(0x7f7d1c0d1170 legacy=0x7f7d1c0cf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.300+0000 7f7d1b7fe700 1 -- 172.21.15.1:0/2872969419 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1c15e1b0 con 0x7f7d1c0d0040 2024-03-18T20:26:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.301+0000 7f7d1b7fe700 1 --2- 172.21.15.1:0/2872969419 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d1c0d0040 0x7f7d1c0cde10 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c009840 tx=0x7f7d0c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6ff698dd8db9a12 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.301+0000 7f7d19ffb700 1 -- 172.21.15.1:0/2872969419 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c011830 con 0x7f7d1c0d0040 2024-03-18T20:26:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.301+0000 7f7d19ffb700 1 -- 172.21.15.1:0/2872969419 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0c011990 con 0x7f7d1c0d0040 2024-03-18T20:26:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.301+0000 7f7d19ffb700 1 -- 172.21.15.1:0/2872969419 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c011d00 con 0x7f7d1c0d0040 2024-03-18T20:26:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.301+0000 7f7d21577700 1 -- 172.21.15.1:0/2872969419 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d1c0d0040 msgr2=0x7f7d1c0cde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.301+0000 7f7d21577700 1 --2- 172.21.15.1:0/2872969419 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d1c0d0040 0x7f7d1c0cde10 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c009840 tx=0x7f7d0c009b50 comp rx=0 tx=0).stop 2024-03-18T20:26:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.301+0000 7f7d21577700 1 -- 172.21.15.1:0/2872969419 shutdown_connections 2024-03-18T20:26:14.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.301+0000 7f7d21577700 1 --2- 172.21.15.1:0/2872969419 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d1c0d0040 0x7f7d1c0cde10 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:14.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.301+0000 7f7d21577700 1 --2- 172.21.15.1:0/2872969419 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d1c15d8b0 0x7f7d1c0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:14.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.301+0000 7f7d21577700 1 -- 172.21.15.1:0/2872969419 >> 172.21.15.1:0/2872969419 conn(0x7f7d1c09c640 msgr2=0x7f7d1c0d3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:14.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.301+0000 7f7d21577700 1 -- 172.21.15.1:0/2872969419 shutdown_connections 2024-03-18T20:26:14.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.302+0000 7f7d21577700 1 -- 172.21.15.1:0/2872969419 wait complete. 2024-03-18T20:26:14.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.303+0000 7f7d21577700 1 Processor -- start 2024-03-18T20:26:14.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.303+0000 7f7d21577700 1 -- start start 2024-03-18T20:26:14.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.304+0000 7f7d21577700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d1c0d0040 0x7f7d1c0c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:14.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.304+0000 7f7d21577700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d1c0d1170 0x7f7d1c0b8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:14.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.304+0000 7f7d21577700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d1c15d8b0 0x7f7d1c1f6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:14.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.304+0000 7f7d21577700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c0c8a30 con 0x7f7d1c0d0040 2024-03-18T20:26:14.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.304+0000 7f7d21577700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c0cba40 con 0x7f7d1c15d8b0 2024-03-18T20:26:14.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.304+0000 7f7d21577700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c0c7e20 con 0x7f7d1c0d1170 2024-03-18T20:26:14.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.304+0000 7f7d1affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d1c0d0040 0x7f7d1c0c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:14.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.304+0000 7f7d1affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d1c0d0040 0x7f7d1c0c7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43554/0 (socket says 172.21.15.1:43554) 2024-03-18T20:26:14.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.304+0000 7f7d1affd700 1 -- 172.21.15.1:0/3585943917 learned_addr learned my addr 172.21.15.1:0/3585943917 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:14.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.304+0000 7f7d1a7fc700 1 --2- 172.21.15.1:0/3585943917 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d1c0d1170 0x7f7d1c0b8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:14.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.304+0000 7f7d1b7fe700 1 --2- 172.21.15.1:0/3585943917 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d1c15d8b0 0x7f7d1c1f6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:14.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.304+0000 7f7d1affd700 1 -- 172.21.15.1:0/3585943917 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d1c0d1170 msgr2=0x7f7d1c0b8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:14.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.304+0000 7f7d1affd700 1 --2- 172.21.15.1:0/3585943917 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d1c0d1170 0x7f7d1c0b8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:14.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.304+0000 7f7d1affd700 1 -- 172.21.15.1:0/3585943917 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d1c15d8b0 msgr2=0x7f7d1c1f6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:14.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.304+0000 7f7d1affd700 1 --2- 172.21.15.1:0/3585943917 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d1c15d8b0 0x7f7d1c1f6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:14.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.304+0000 7f7d1affd700 1 -- 172.21.15.1:0/3585943917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0c0094a0 con 0x7f7d1c0d0040 2024-03-18T20:26:14.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.304+0000 7f7d1affd700 1 --2- 172.21.15.1:0/3585943917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d1c0d0040 0x7f7d1c0c7840 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7f7d1400efc0 tx=0x7f7d1400c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.306+0000 7f7d037fe700 1 -- 172.21.15.1:0/3585943917 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1400cb50 con 0x7f7d1c0d0040 2024-03-18T20:26:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.306+0000 7f7d037fe700 1 -- 172.21.15.1:0/3585943917 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1400f040 con 0x7f7d1c0d0040 2024-03-18T20:26:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.306+0000 7f7d21577700 1 -- 172.21.15.1:0/3585943917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d1c0c8100 con 0x7f7d1c0d0040 2024-03-18T20:26:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.306+0000 7f7d037fe700 1 -- 172.21.15.1:0/3585943917 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d14014480 con 0x7f7d1c0d0040 2024-03-18T20:26:14.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.306+0000 7f7d21577700 1 -- 172.21.15.1:0/3585943917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d1c0c86c0 con 0x7f7d1c0d0040 2024-03-18T20:26:14.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.307+0000 7f7d21577700 1 -- 172.21.15.1:0/3585943917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d1c0c8d10 con 0x7f7d1c0d0040 2024-03-18T20:26:14.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.308+0000 7f7d037fe700 1 -- 172.21.15.1:0/3585943917 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f7d140145e0 con 0x7f7d1c0d0040 2024-03-18T20:26:14.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.308+0000 7f7d037fe700 1 --2- 172.21.15.1:0/3585943917 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7d0806aed0 0x7f7d0806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:14.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.308+0000 7f7d037fe700 1 -- 172.21.15.1:0/3585943917 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(242..242 src has 1..242) v4 ==== 7044+0+0 (secure 0 0 0) 0x7f7d1408a510 con 0x7f7d1c0d0040 2024-03-18T20:26:14.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.308+0000 7f7d1a7fc700 1 --2- 172.21.15.1:0/3585943917 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7d0806aed0 0x7f7d0806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:14.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.308+0000 7f7d1a7fc700 1 --2- 172.21.15.1:0/3585943917 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7d0806aed0 0x7f7d0806d390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c0b7cd0 tx=0x7f7d10009040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:14.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.315+0000 7f7d037fe700 1 -- 172.21.15.1:0/3585943917 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7d14054dd0 con 0x7f7d1c0d0040 2024-03-18T20:26:14.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:14.514+0000 7f7d21577700 1 -- 172.21.15.1:0/3585943917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7d1c073350 con 0x7f7d1c0d0040 2024-03-18T20:26:15.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.315+0000 7f7d037fe700 1 -- 172.21.15.1:0/3585943917 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v243) v1 ==== 190+0+0 (secure 0 0 0) 0x7f7d140583f0 con 0x7f7d1c0d0040 2024-03-18T20:26:15.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.318+0000 7f7d21577700 1 -- 172.21.15.1:0/3585943917 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7d0806aed0 msgr2=0x7f7d0806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:15.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.318+0000 7f7d21577700 1 --2- 172.21.15.1:0/3585943917 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7d0806aed0 0x7f7d0806d390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c0b7cd0 tx=0x7f7d10009040 comp rx=0 tx=0).stop 2024-03-18T20:26:15.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.318+0000 7f7d21577700 1 -- 172.21.15.1:0/3585943917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d1c0d0040 msgr2=0x7f7d1c0c7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:15.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.318+0000 7f7d21577700 1 --2- 172.21.15.1:0/3585943917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d1c0d0040 0x7f7d1c0c7840 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7f7d1400efc0 tx=0x7f7d1400c0c0 comp rx=0 tx=0).stop 2024-03-18T20:26:15.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.318+0000 7f7d21577700 1 -- 172.21.15.1:0/3585943917 shutdown_connections 2024-03-18T20:26:15.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.318+0000 7f7d21577700 1 --2- 172.21.15.1:0/3585943917 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7d0806aed0 0x7f7d0806d390 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:15.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.318+0000 7f7d21577700 1 --2- 172.21.15.1:0/3585943917 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d1c15d8b0 0x7f7d1c1f6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:15.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.318+0000 7f7d21577700 1 --2- 172.21.15.1:0/3585943917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d1c0d0040 0x7f7d1c0c7840 unknown :-1 s=CLOSED pgs=2748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:15.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.318+0000 7f7d21577700 1 --2- 172.21.15.1:0/3585943917 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d1c0d1170 0x7f7d1c0b8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:15.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.318+0000 7f7d21577700 1 -- 172.21.15.1:0/3585943917 >> 172.21.15.1:0/3585943917 conn(0x7f7d1c09c640 msgr2=0x7f7d1c1588a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:15.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.318+0000 7f7d21577700 1 -- 172.21.15.1:0/3585943917 shutdown_connections 2024-03-18T20:26:15.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.318+0000 7f7d21577700 1 -- 172.21.15.1:0/3585943917 wait complete. 2024-03-18T20:26:15.325 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'backup_fs_metadata' removed 2024-03-18T20:26:15.340 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2024-03-18T20:26:15.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.471+0000 7f8ceffc2700 1 Processor -- start 2024-03-18T20:26:15.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.471+0000 7f8ceffc2700 1 -- start start 2024-03-18T20:26:15.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.472+0000 7f8ceffc2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8ce809a600 0x7f8ce80cc360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:15.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.472+0000 7f8ceffc2700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce80ae260 con 0x7f8ce809a150 2024-03-18T20:26:15.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.472+0000 7f8ceffc2700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce80d0150 con 0x7f8ce815b550 2024-03-18T20:26:15.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.472+0000 7f8ceffc2700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ce80cfb90 con 0x7f8ce809a600 2024-03-18T20:26:15.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.472+0000 7f8cee55f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8ce809a600 0x7f8ce80cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:15.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.472+0000 7f8cee55f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8ce809a600 0x7f8ce80cc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56364/0 (socket says 172.21.15.1:56364) 2024-03-18T20:26:15.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.472+0000 7f8ced55d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8ce809a150 0x7f8ce80cbc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42326/0 (socket says 172.21.15.1:42326) 2024-03-18T20:26:15.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.472+0000 7f8cee55f700 1 -- 172.21.15.1:0/992773264 learned_addr learned my addr 172.21.15.1:0/992773264 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:15.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.472+0000 7f8cee55f700 1 -- 172.21.15.1:0/992773264 >> v1:172.21.15.165:6789/0 conn(0x7f8ce815b550 legacy=0x7f8ce8099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:15.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.472+0000 7f8cee55f700 1 -- 172.21.15.1:0/992773264 >> v1:172.21.15.1:6789/0 conn(0x7f8ce809a150 legacy=0x7f8ce80cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:15.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.472+0000 7f8cee55f700 1 -- 172.21.15.1:0/992773264 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce80cfe70 con 0x7f8ce809a600 2024-03-18T20:26:15.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.472+0000 7f8cecd5c700 1 -- 172.21.15.1:0/992773264 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250857209 0 0) 0x7f8ce80ae260 con 0x7f8ce809a150 2024-03-18T20:26:15.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.473+0000 7f8cee55f700 1 --2- 172.21.15.1:0/992773264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce809a600 0x7f8ce80cc360 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0009e50 tx=0x7f8ce000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409f06780e488f92 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:15.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.473+0000 7f8cecd5c700 1 -- 172.21.15.1:0/992773264 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce000e070 con 0x7f8ce809a600 2024-03-18T20:26:15.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.473+0000 7f8cecd5c700 1 -- 172.21.15.1:0/992773264 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce0003a20 con 0x7f8ce809a600 2024-03-18T20:26:15.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.473+0000 7f8cecd5c700 1 -- 172.21.15.1:0/992773264 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0003dd0 con 0x7f8ce809a600 2024-03-18T20:26:15.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.473+0000 7f8ceffc2700 1 -- 172.21.15.1:0/992773264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce809a600 msgr2=0x7f8ce80cc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:15.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.473+0000 7f8ceffc2700 1 --2- 172.21.15.1:0/992773264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce809a600 0x7f8ce80cc360 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0009e50 tx=0x7f8ce000b330 comp rx=0 tx=0).stop 2024-03-18T20:26:15.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.473+0000 7f8ceffc2700 1 -- 172.21.15.1:0/992773264 shutdown_connections 2024-03-18T20:26:15.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.473+0000 7f8ceffc2700 1 --2- 172.21.15.1:0/992773264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce809a600 0x7f8ce80cc360 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:15.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.473+0000 7f8ceffc2700 1 -- 172.21.15.1:0/992773264 >> 172.21.15.1:0/992773264 conn(0x7f8ce80b2c30 msgr2=0x7f8ce80b3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:15.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.473+0000 7f8ceffc2700 1 -- 172.21.15.1:0/992773264 shutdown_connections 2024-03-18T20:26:15.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.473+0000 7f8ceffc2700 1 -- 172.21.15.1:0/992773264 wait complete. 2024-03-18T20:26:15.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.474+0000 7f8ceffc2700 1 Processor -- start 2024-03-18T20:26:15.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.475+0000 7f8ceffc2700 1 -- start start 2024-03-18T20:26:15.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.475+0000 7f8ceffc2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ce809a150 0x7f8ce81577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:15.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.475+0000 7f8ceffc2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ce809a600 0x7f8ce80c7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:15.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.475+0000 7f8ceffc2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce815b550 0x7f8ce81e49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:15.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.475+0000 7f8ceffc2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce8157d10 con 0x7f8ce815b550 2024-03-18T20:26:15.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.475+0000 7f8ceffc2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce806cc90 con 0x7f8ce809a150 2024-03-18T20:26:15.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.475+0000 7f8ceffc2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce806cf70 con 0x7f8ce809a600 2024-03-18T20:26:15.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.475+0000 7f8cee55f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce815b550 0x7f8ce81e49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:15.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.475+0000 7f8cee55f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce815b550 0x7f8ce81e49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56368/0 (socket says 172.21.15.1:56368) 2024-03-18T20:26:15.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.475+0000 7f8cee55f700 1 -- 172.21.15.1:0/4149398978 learned_addr learned my addr 172.21.15.1:0/4149398978 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:15.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.476+0000 7f8cedd5e700 1 --2- 172.21.15.1:0/4149398978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ce809a150 0x7f8ce81577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:15.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.476+0000 7f8ced55d700 1 --2- 172.21.15.1:0/4149398978 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ce809a600 0x7f8ce80c7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:15.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.476+0000 7f8cee55f700 1 -- 172.21.15.1:0/4149398978 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ce809a600 msgr2=0x7f8ce80c7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:15.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.476+0000 7f8cee55f700 1 --2- 172.21.15.1:0/4149398978 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ce809a600 0x7f8ce80c7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:15.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.476+0000 7f8cee55f700 1 -- 172.21.15.1:0/4149398978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ce809a150 msgr2=0x7f8ce81577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:15.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.476+0000 7f8cee55f700 1 --2- 172.21.15.1:0/4149398978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ce809a150 0x7f8ce81577d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:15.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.476+0000 7f8cee55f700 1 -- 172.21.15.1:0/4149398978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce0009b30 con 0x7f8ce815b550 2024-03-18T20:26:15.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.476+0000 7f8cedd5e700 1 --2- 172.21.15.1:0/4149398978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ce809a150 0x7f8ce81577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:15.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.476+0000 7f8ced55d700 1 --2- 172.21.15.1:0/4149398978 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ce809a600 0x7f8ce80c7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:15.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.476+0000 7f8cee55f700 1 --2- 172.21.15.1:0/4149398978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce815b550 0x7f8ce81e49c0 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0004000 tx=0x7f8ce0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:15.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.476+0000 7f8cde7fc700 1 -- 172.21.15.1:0/4149398978 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0004240 con 0x7f8ce815b550 2024-03-18T20:26:15.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.476+0000 7f8cde7fc700 1 -- 172.21.15.1:0/4149398978 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce00043a0 con 0x7f8ce815b550 2024-03-18T20:26:15.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.476+0000 7f8ceffc2700 1 -- 172.21.15.1:0/4149398978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce806d250 con 0x7f8ce815b550 2024-03-18T20:26:15.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.476+0000 7f8cde7fc700 1 -- 172.21.15.1:0/4149398978 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0004750 con 0x7f8ce815b550 2024-03-18T20:26:15.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.476+0000 7f8ceffc2700 1 -- 172.21.15.1:0/4149398978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce806d810 con 0x7f8ce815b550 2024-03-18T20:26:15.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.477+0000 7f8ceffc2700 1 -- 172.21.15.1:0/4149398978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce80676a0 con 0x7f8ce815b550 2024-03-18T20:26:15.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.478+0000 7f8cde7fc700 1 -- 172.21.15.1:0/4149398978 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f8ce0019070 con 0x7f8ce815b550 2024-03-18T20:26:15.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.478+0000 7f8cde7fc700 1 --2- 172.21.15.1:0/4149398978 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8ccc06aed0 0x7f8ccc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:15.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.478+0000 7f8cde7fc700 1 -- 172.21.15.1:0/4149398978 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 6600+0+0 (secure 0 0 0) 0x7f8ce008abe0 con 0x7f8ce815b550 2024-03-18T20:26:15.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.478+0000 7f8cedd5e700 1 --2- 172.21.15.1:0/4149398978 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8ccc06aed0 0x7f8ccc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:15.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.479+0000 7f8cedd5e700 1 --2- 172.21.15.1:0/4149398978 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8ccc06aed0 0x7f8ccc06d390 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8006fd0 tx=0x7f8cd8008040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:15.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.483+0000 7f8cde7fc700 1 -- 172.21.15.1:0/4149398978 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8ce00555e0 con 0x7f8ce815b550 2024-03-18T20:26:15.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:15.688+0000 7f8ceffc2700 1 -- 172.21.15.1:0/4149398978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8ce806b510 con 0x7f8ce815b550 2024-03-18T20:26:16.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.320+0000 7f8cde7fc700 1 -- 172.21.15.1:0/4149398978 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v244) v1 ==== 178+0+0 (secure 0 0 0) 0x7f8ce00200d0 con 0x7f8ce815b550 2024-03-18T20:26:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.324+0000 7f8ceffc2700 1 -- 172.21.15.1:0/4149398978 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8ccc06aed0 msgr2=0x7f8ccc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.324+0000 7f8ceffc2700 1 --2- 172.21.15.1:0/4149398978 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8ccc06aed0 0x7f8ccc06d390 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8006fd0 tx=0x7f8cd8008040 comp rx=0 tx=0).stop 2024-03-18T20:26:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.324+0000 7f8ceffc2700 1 -- 172.21.15.1:0/4149398978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce815b550 msgr2=0x7f8ce81e49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.324+0000 7f8ceffc2700 1 --2- 172.21.15.1:0/4149398978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce815b550 0x7f8ce81e49c0 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0004000 tx=0x7f8ce0004ad0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.324+0000 7f8ceffc2700 1 -- 172.21.15.1:0/4149398978 shutdown_connections 2024-03-18T20:26:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.324+0000 7f8ceffc2700 1 --2- 172.21.15.1:0/4149398978 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8ccc06aed0 0x7f8ccc06d390 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8006fd0 tx=0x7f8cd8008040 comp rx=0 tx=0).stop 2024-03-18T20:26:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.324+0000 7f8ceffc2700 1 --2- 172.21.15.1:0/4149398978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ce815b550 0x7f8ce81e49c0 unknown :-1 s=CLOSED pgs=2751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.324+0000 7f8ceffc2700 1 --2- 172.21.15.1:0/4149398978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8ce809a150 0x7f8ce81577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.324+0000 7f8ceffc2700 1 --2- 172.21.15.1:0/4149398978 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8ce809a600 0x7f8ce80c7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.325+0000 7f8ceffc2700 1 -- 172.21.15.1:0/4149398978 >> 172.21.15.1:0/4149398978 conn(0x7f8ce80b2c30 msgr2=0x7f8ce806f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.325+0000 7f8ceffc2700 1 -- 172.21.15.1:0/4149398978 shutdown_connections 2024-03-18T20:26:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.325+0000 7f8ceffc2700 1 -- 172.21.15.1:0/4149398978 wait complete. 2024-03-18T20:26:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'backup_fs_data' removed 2024-03-18T20:26:16.346 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-18T20:26:16.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.478+0000 7fc593115700 1 Processor -- start 2024-03-18T20:26:16.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.478+0000 7fc593115700 1 -- start start 2024-03-18T20:26:16.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.479+0000 7fc593115700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc58c15d3e0 0x7fc58c0d0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:16.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.479+0000 7fc593115700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc58c15d7c0 0x7fc58c0d17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:16.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.479+0000 7fc593115700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc58c0d1cf0 0x7fc58c0b82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:16.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.479+0000 7fc593115700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc58c0cee60 con 0x7fc58c0d1cf0 2024-03-18T20:26:16.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.479+0000 7fc593115700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc58c0d2160 con 0x7fc58c15d3e0 2024-03-18T20:26:16.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.479+0000 7fc593115700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c0ce1e0 con 0x7fc58c15d7c0 2024-03-18T20:26:16.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.479+0000 7fc5916b2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc58c0d1cf0 0x7fc58c0b82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:16.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.479+0000 7fc5916b2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc58c0d1cf0 0x7fc58c0b82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56384/0 (socket says 172.21.15.1:56384) 2024-03-18T20:26:16.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.479+0000 7fc5916b2700 1 -- 172.21.15.1:0/1306339854 learned_addr learned my addr 172.21.15.1:0/1306339854 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:16.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.480+0000 7fc58bfff700 1 --2- 172.21.15.1:0/1306339854 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc58c15d7c0 0x7fc58c0d17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:16.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.480+0000 7fc590eb1700 1 --2- 172.21.15.1:0/1306339854 >> v2:172.21.15.165:3300/0 conn(0x7fc58c15d3e0 0x7fc58c0d0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:16.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.480+0000 7fc5916b2700 1 -- 172.21.15.1:0/1306339854 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc58c15d7c0 msgr2=0x7fc58c0d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:16.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.480+0000 7fc5916b2700 1 --2- 172.21.15.1:0/1306339854 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc58c15d7c0 0x7fc58c0d17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.480+0000 7fc5916b2700 1 -- 172.21.15.1:0/1306339854 >> v2:172.21.15.165:3300/0 conn(0x7fc58c15d3e0 msgr2=0x7fc58c0d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:16.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.480+0000 7fc5916b2700 1 --2- 172.21.15.1:0/1306339854 >> v2:172.21.15.165:3300/0 conn(0x7fc58c15d3e0 0x7fc58c0d0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.480+0000 7fc5916b2700 1 -- 172.21.15.1:0/1306339854 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58c0ce4c0 con 0x7fc58c0d1cf0 2024-03-18T20:26:16.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.480+0000 7fc5916b2700 1 --2- 172.21.15.1:0/1306339854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc58c0d1cf0 0x7fc58c0b82d0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7fc58c0d3130 tx=0x7fc58400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d02fa7e490d3e9b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:16.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.480+0000 7fc58b7fe700 1 -- 172.21.15.1:0/1306339854 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58400e070 con 0x7fc58c0d1cf0 2024-03-18T20:26:16.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.480+0000 7fc58b7fe700 1 -- 172.21.15.1:0/1306339854 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58400b980 con 0x7fc58c0d1cf0 2024-03-18T20:26:16.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.480+0000 7fc58b7fe700 1 -- 172.21.15.1:0/1306339854 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58400bd30 con 0x7fc58c0d1cf0 2024-03-18T20:26:16.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.481+0000 7fc593115700 1 -- 172.21.15.1:0/1306339854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc58c0d1cf0 msgr2=0x7fc58c0b82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:16.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.481+0000 7fc593115700 1 --2- 172.21.15.1:0/1306339854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc58c0d1cf0 0x7fc58c0b82d0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7fc58c0d3130 tx=0x7fc58400b330 comp rx=0 tx=0).stop 2024-03-18T20:26:16.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.481+0000 7fc593115700 1 -- 172.21.15.1:0/1306339854 shutdown_connections 2024-03-18T20:26:16.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.481+0000 7fc593115700 1 --2- 172.21.15.1:0/1306339854 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc58c0d1cf0 0x7fc58c0b82d0 unknown :-1 s=CLOSED pgs=2752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.481+0000 7fc593115700 1 --2- 172.21.15.1:0/1306339854 >> v2:172.21.15.165:3300/0 conn(0x7fc58c15d3e0 0x7fc58c0d0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.481+0000 7fc593115700 1 --2- 172.21.15.1:0/1306339854 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc58c15d7c0 0x7fc58c0d17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.481+0000 7fc593115700 1 -- 172.21.15.1:0/1306339854 >> 172.21.15.1:0/1306339854 conn(0x7fc58c09c540 msgr2=0x7fc58c0d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:16.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.481+0000 7fc593115700 1 -- 172.21.15.1:0/1306339854 shutdown_connections 2024-03-18T20:26:16.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.481+0000 7fc593115700 1 -- 172.21.15.1:0/1306339854 wait complete. 2024-03-18T20:26:16.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.482+0000 7fc593115700 1 Processor -- start 2024-03-18T20:26:16.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.482+0000 7fc593115700 1 -- start start 2024-03-18T20:26:16.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.483+0000 7fc593115700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc58c0d1cf0 0x7fc58c0cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:16.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.483+0000 7fc593115700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc58c15d3e0 0x7fc58c0cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:16.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.483+0000 7fc593115700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc58c1f2a50 0x7fc58c0c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:16.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.483+0000 7fc593115700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c149130 con 0x7fc58c1f2a50 2024-03-18T20:26:16.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.483+0000 7fc593115700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c1493c0 con 0x7fc58c15d3e0 2024-03-18T20:26:16.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.483+0000 7fc593115700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c0c9640 con 0x7fc58c0d1cf0 2024-03-18T20:26:16.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.484+0000 7fc5916b2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc58c1f2a50 0x7fc58c0c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:16.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.484+0000 7fc5916b2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc58c1f2a50 0x7fc58c0c8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56390/0 (socket says 172.21.15.1:56390) 2024-03-18T20:26:16.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.484+0000 7fc5916b2700 1 -- 172.21.15.1:0/4174284811 learned_addr learned my addr 172.21.15.1:0/4174284811 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:16.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.484+0000 7fc590eb1700 1 --2- 172.21.15.1:0/4174284811 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc58c0d1cf0 0x7fc58c0cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:16.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.484+0000 7fc58bfff700 1 --2- 172.21.15.1:0/4174284811 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc58c15d3e0 0x7fc58c0cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:16.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.484+0000 7fc5916b2700 1 -- 172.21.15.1:0/4174284811 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc58c0d1cf0 msgr2=0x7fc58c0cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:16.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.484+0000 7fc5916b2700 1 --2- 172.21.15.1:0/4174284811 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc58c0d1cf0 0x7fc58c0cb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.484+0000 7fc5916b2700 1 -- 172.21.15.1:0/4174284811 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc58c15d3e0 msgr2=0x7fc58c0cbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:16.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.484+0000 7fc5916b2700 1 --2- 172.21.15.1:0/4174284811 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc58c15d3e0 0x7fc58c0cbdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.484+0000 7fc5916b2700 1 -- 172.21.15.1:0/4174284811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc584009b30 con 0x7fc58c1f2a50 2024-03-18T20:26:16.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.484+0000 7fc5916b2700 1 --2- 172.21.15.1:0/4174284811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc58c1f2a50 0x7fc58c0c8810 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7fc58c08bb50 tx=0x7fc584004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:16.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.484+0000 7fc5897fa700 1 -- 172.21.15.1:0/4174284811 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc584004250 con 0x7fc58c1f2a50 2024-03-18T20:26:16.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.484+0000 7fc5897fa700 1 -- 172.21.15.1:0/4174284811 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5840043b0 con 0x7fc58c1f2a50 2024-03-18T20:26:16.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.484+0000 7fc593115700 1 -- 172.21.15.1:0/4174284811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58c0c9920 con 0x7fc58c1f2a50 2024-03-18T20:26:16.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.484+0000 7fc5897fa700 1 -- 172.21.15.1:0/4174284811 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc584004760 con 0x7fc58c1f2a50 2024-03-18T20:26:16.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.484+0000 7fc593115700 1 -- 172.21.15.1:0/4174284811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc58c0c9ee0 con 0x7fc58c1f2a50 2024-03-18T20:26:16.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.485+0000 7fc576ffd700 1 -- 172.21.15.1:0/4174284811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5640052f0 con 0x7fc58c1f2a50 2024-03-18T20:26:16.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.486+0000 7fc5897fa700 1 -- 172.21.15.1:0/4174284811 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fc58401c440 con 0x7fc58c1f2a50 2024-03-18T20:26:16.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.486+0000 7fc5897fa700 1 --2- 172.21.15.1:0/4174284811 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc57006aed0 0x7fc57006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:16.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.486+0000 7fc5897fa700 1 -- 172.21.15.1:0/4174284811 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6180+0+0 (secure 0 0 0) 0x7fc58408b2b0 con 0x7fc58c1f2a50 2024-03-18T20:26:16.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.486+0000 7fc590eb1700 1 --2- 172.21.15.1:0/4174284811 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc57006aed0 0x7fc57006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:16.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.486+0000 7fc590eb1700 1 --2- 172.21.15.1:0/4174284811 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc57006aed0 0x7fc57006d390 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc58000d8d0 tx=0x7fc580008040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:16.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.491+0000 7fc5897fa700 1 -- 172.21.15.1:0/4174284811 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc584055ff0 con 0x7fc58c1f2a50 2024-03-18T20:26:16.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.696+0000 7fc576ffd700 1 -- 172.21.15.1:0/4174284811 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fc564005640 con 0x7fc58c1f2a50 2024-03-18T20:26:16.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.699+0000 7fc5897fa700 1 -- 172.21.15.1:0/4174284811 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v79) v1 ==== 115+0+0 (secure 0 0 0) 0x7fc584059610 con 0x7fc58c1f2a50 2024-03-18T20:26:16.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.702+0000 7fc576ffd700 1 -- 172.21.15.1:0/4174284811 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc57006aed0 msgr2=0x7fc57006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:16.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.702+0000 7fc576ffd700 1 --2- 172.21.15.1:0/4174284811 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc57006aed0 0x7fc57006d390 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc58000d8d0 tx=0x7fc580008040 comp rx=0 tx=0).stop 2024-03-18T20:26:16.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.702+0000 7fc576ffd700 1 -- 172.21.15.1:0/4174284811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc58c1f2a50 msgr2=0x7fc58c0c8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.702+0000 7fc576ffd700 1 --2- 172.21.15.1:0/4174284811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc58c1f2a50 0x7fc58c0c8810 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7fc58c08bb50 tx=0x7fc584004ad0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.702+0000 7fc576ffd700 1 -- 172.21.15.1:0/4174284811 shutdown_connections 2024-03-18T20:26:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.702+0000 7fc576ffd700 1 --2- 172.21.15.1:0/4174284811 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc57006aed0 0x7fc57006d390 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.702+0000 7fc576ffd700 1 --2- 172.21.15.1:0/4174284811 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc58c0d1cf0 0x7fc58c0cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.702+0000 7fc576ffd700 1 --2- 172.21.15.1:0/4174284811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc58c1f2a50 0x7fc58c0c8810 unknown :-1 s=CLOSED pgs=2753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.702+0000 7fc576ffd700 1 --2- 172.21.15.1:0/4174284811 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc58c15d3e0 0x7fc58c0cbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.702+0000 7fc576ffd700 1 -- 172.21.15.1:0/4174284811 >> 172.21.15.1:0/4174284811 conn(0x7fc58c09c540 msgr2=0x7fc58c15ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.702+0000 7fc576ffd700 1 -- 172.21.15.1:0/4174284811 shutdown_connections 2024-03-18T20:26:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.702+0000 7fc576ffd700 1 -- 172.21.15.1:0/4174284811 wait complete. 2024-03-18T20:26:16.720 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-18T20:26:16.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.842+0000 7ff329f02700 1 Processor -- start 2024-03-18T20:26:16.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.842+0000 7ff329f02700 1 -- start start 2024-03-18T20:26:16.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.843+0000 7ff329f02700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff324071cc0 0x7ff3240ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:16.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.843+0000 7ff329f02700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff32406d560 0x7ff32406fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:16.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.843+0000 7ff329f02700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff324070000 con 0x7ff32415a240 2024-03-18T20:26:16.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.843+0000 7ff329f02700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3240adff0 con 0x7ff324071cc0 2024-03-18T20:26:16.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.843+0000 7ff329f02700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3240cac80 con 0x7ff32406d560 2024-03-18T20:26:16.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.843+0000 7ff322ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff324071cc0 0x7ff3240ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:16.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.843+0000 7ff322ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff324071cc0 0x7ff3240ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56396/0 (socket says 172.21.15.1:56396) 2024-03-18T20:26:16.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.843+0000 7ff322ffd700 1 -- 172.21.15.1:0/3570643845 learned_addr learned my addr 172.21.15.1:0/3570643845 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:16.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.843+0000 7ff323fff700 1 --2- 172.21.15.1:0/3570643845 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff32406d560 0x7ff32406fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:16.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.843+0000 7ff322ffd700 1 -- 172.21.15.1:0/3570643845 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff32406d560 msgr2=0x7ff32406fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:16.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.843+0000 7ff322ffd700 1 --2- 172.21.15.1:0/3570643845 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff32406d560 0x7ff32406fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.843+0000 7ff322ffd700 1 -- 172.21.15.1:0/3570643845 >> v1:172.21.15.165:6789/0 conn(0x7ff32415a240 legacy=0x7ff3240715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:16.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.843+0000 7ff322ffd700 1 -- 172.21.15.1:0/3570643845 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3240caf60 con 0x7ff324071cc0 2024-03-18T20:26:16.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.843+0000 7ff322ffd700 1 --2- 172.21.15.1:0/3570643845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff324071cc0 0x7ff3240ca580 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7ff30c0098f0 tx=0x7ff30c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e81780be1db03120 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:16.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.844+0000 7ff3227fc700 1 -- 172.21.15.1:0/3570643845 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c004230 con 0x7ff324071cc0 2024-03-18T20:26:16.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.844+0000 7ff3227fc700 1 -- 172.21.15.1:0/3570643845 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30c004390 con 0x7ff324071cc0 2024-03-18T20:26:16.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.844+0000 7ff3227fc700 1 -- 172.21.15.1:0/3570643845 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c0036a0 con 0x7ff324071cc0 2024-03-18T20:26:16.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.844+0000 7ff329f02700 1 -- 172.21.15.1:0/3570643845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff324071cc0 msgr2=0x7ff3240ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:16.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.844+0000 7ff329f02700 1 --2- 172.21.15.1:0/3570643845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff324071cc0 0x7ff3240ca580 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7ff30c0098f0 tx=0x7ff30c009c00 comp rx=0 tx=0).stop 2024-03-18T20:26:16.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.844+0000 7ff329f02700 1 -- 172.21.15.1:0/3570643845 shutdown_connections 2024-03-18T20:26:16.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.844+0000 7ff329f02700 1 --2- 172.21.15.1:0/3570643845 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff32406d560 0x7ff32406fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.844+0000 7ff329f02700 1 --2- 172.21.15.1:0/3570643845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff324071cc0 0x7ff3240ca580 unknown :-1 s=CLOSED pgs=2754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.844+0000 7ff329f02700 1 -- 172.21.15.1:0/3570643845 >> 172.21.15.1:0/3570643845 conn(0x7ff3240cff30 msgr2=0x7ff3240d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:16.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.844+0000 7ff329f02700 1 -- 172.21.15.1:0/3570643845 shutdown_connections 2024-03-18T20:26:16.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.844+0000 7ff329f02700 1 -- 172.21.15.1:0/3570643845 wait complete. 2024-03-18T20:26:16.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.845+0000 7ff329f02700 1 Processor -- start 2024-03-18T20:26:16.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.846+0000 7ff329f02700 1 -- start start 2024-03-18T20:26:16.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.846+0000 7ff329f02700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff32406d560 0x7ff324158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:16.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.846+0000 7ff329f02700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff324071cc0 0x7ff3241570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:16.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.846+0000 7ff329f02700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff32415a240 0x7ff3240c6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:16.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.846+0000 7ff329f02700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff324157620 con 0x7ff324071cc0 2024-03-18T20:26:16.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.846+0000 7ff329f02700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3240c9f00 con 0x7ff32415a240 2024-03-18T20:26:16.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.846+0000 7ff329f02700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3240c9c20 con 0x7ff32406d560 2024-03-18T20:26:16.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.846+0000 7ff322ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff324071cc0 0x7ff3241570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:16.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.846+0000 7ff322ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff324071cc0 0x7ff3241570e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56412/0 (socket says 172.21.15.1:56412) 2024-03-18T20:26:16.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.846+0000 7ff322ffd700 1 -- 172.21.15.1:0/4032272958 learned_addr learned my addr 172.21.15.1:0/4032272958 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.847+0000 7ff322ffd700 1 -- 172.21.15.1:0/4032272958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff32406d560 msgr2=0x7ff324158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.847+0000 7ff322ffd700 1 --2- 172.21.15.1:0/4032272958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff32406d560 0x7ff324158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.847+0000 7ff322ffd700 1 -- 172.21.15.1:0/4032272958 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff32415a240 msgr2=0x7ff3240c6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.847+0000 7ff322ffd700 1 --2- 172.21.15.1:0/4032272958 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff32415a240 0x7ff3240c6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.847+0000 7ff322ffd700 1 -- 172.21.15.1:0/4032272958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c009550 con 0x7ff324071cc0 2024-03-18T20:26:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.847+0000 7ff322ffd700 1 --2- 172.21.15.1:0/4032272958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff324071cc0 0x7ff3241570e0 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7ff30c009130 tx=0x7ff30c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.847+0000 7ff31bfff700 1 -- 172.21.15.1:0/4032272958 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c003c00 con 0x7ff324071cc0 2024-03-18T20:26:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.847+0000 7ff31bfff700 1 -- 172.21.15.1:0/4032272958 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30c01f070 con 0x7ff324071cc0 2024-03-18T20:26:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.847+0000 7ff329f02700 1 -- 172.21.15.1:0/4032272958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff324157820 con 0x7ff324071cc0 2024-03-18T20:26:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.847+0000 7ff31bfff700 1 -- 172.21.15.1:0/4032272958 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c00b660 con 0x7ff324071cc0 2024-03-18T20:26:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.847+0000 7ff329f02700 1 -- 172.21.15.1:0/4032272958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff324157de0 con 0x7ff324071cc0 2024-03-18T20:26:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.848+0000 7ff329f02700 1 -- 172.21.15.1:0/4032272958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3240676a0 con 0x7ff324071cc0 2024-03-18T20:26:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.848+0000 7ff31bfff700 1 -- 172.21.15.1:0/4032272958 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7ff30c021030 con 0x7ff324071cc0 2024-03-18T20:26:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.849+0000 7ff31bfff700 1 --2- 172.21.15.1:0/4032272958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff30406aed0 0x7ff30406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.849+0000 7ff31bfff700 1 -- 172.21.15.1:0/4032272958 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6180+0+0 (secure 0 0 0) 0x7ff30c08ae80 con 0x7ff324071cc0 2024-03-18T20:26:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.849+0000 7ff3237fe700 1 --2- 172.21.15.1:0/4032272958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff30406aed0 0x7ff30406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.849+0000 7ff3237fe700 1 --2- 172.21.15.1:0/4032272958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff30406aed0 0x7ff30406d390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff3140096f0 tx=0x7ff314006c50 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:16.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:16.854+0000 7ff31bfff700 1 -- 172.21.15.1:0/4032272958 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff30c055c80 con 0x7ff324071cc0 2024-03-18T20:26:17.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.058+0000 7ff329f02700 1 -- 172.21.15.1:0/4032272958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7ff3240676a0 con 0x7ff324071cc0 2024-03-18T20:26:17.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.062+0000 7ff31bfff700 1 -- 172.21.15.1:0/4032272958 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v80) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff30c0592a0 con 0x7ff324071cc0 2024-03-18T20:26:17.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.064+0000 7ff329f02700 1 -- 172.21.15.1:0/4032272958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff30406aed0 msgr2=0x7ff30406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:17.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.064+0000 7ff329f02700 1 --2- 172.21.15.1:0/4032272958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff30406aed0 0x7ff30406d390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff3140096f0 tx=0x7ff314006c50 comp rx=0 tx=0).stop 2024-03-18T20:26:17.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.064+0000 7ff329f02700 1 -- 172.21.15.1:0/4032272958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff324071cc0 msgr2=0x7ff3241570e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:17.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.064+0000 7ff329f02700 1 --2- 172.21.15.1:0/4032272958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff324071cc0 0x7ff3241570e0 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7ff30c009130 tx=0x7ff30c004750 comp rx=0 tx=0).stop 2024-03-18T20:26:17.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.064+0000 7ff329f02700 1 -- 172.21.15.1:0/4032272958 shutdown_connections 2024-03-18T20:26:17.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.064+0000 7ff329f02700 1 --2- 172.21.15.1:0/4032272958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff30406aed0 0x7ff30406d390 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:17.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.064+0000 7ff329f02700 1 --2- 172.21.15.1:0/4032272958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff32406d560 0x7ff324158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:17.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.064+0000 7ff329f02700 1 --2- 172.21.15.1:0/4032272958 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff32415a240 0x7ff3240c6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:17.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.064+0000 7ff329f02700 1 --2- 172.21.15.1:0/4032272958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff324071cc0 0x7ff3241570e0 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:17.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.064+0000 7ff329f02700 1 -- 172.21.15.1:0/4032272958 >> 172.21.15.1:0/4032272958 conn(0x7ff3240cff30 msgr2=0x7ff3240ce8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:17.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.064+0000 7ff329f02700 1 -- 172.21.15.1:0/4032272958 shutdown_connections 2024-03-18T20:26:17.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.064+0000 7ff329f02700 1 -- 172.21.15.1:0/4032272958 wait complete. 2024-03-18T20:26:17.082 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-18T20:26:17.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.204+0000 7fec7033d700 1 Processor -- start 2024-03-18T20:26:17.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.204+0000 7fec7033d700 1 -- start start 2024-03-18T20:26:17.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec7033d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec6815d780 0x7fec680cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:17.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec7033d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fec6809dd40 0x7fec680cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:17.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec7033d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec680ce090 con 0x7fec680ce5a0 2024-03-18T20:26:17.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec7033d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec680cf120 con 0x7fec6809dd40 2024-03-18T20:26:17.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec7033d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec6806db30 con 0x7fec6815d780 2024-03-18T20:26:17.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6e8da700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fec680ce5a0 0x7fec680cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42336/0 (socket says 172.21.15.1:42336) 2024-03-18T20:26:17.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6e8da700 1 -- 172.21.15.1:0/780656540 learned_addr learned my addr 172.21.15.1:0/780656540 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:17.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6e0d9700 1 --2- 172.21.15.1:0/780656540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec6815d780 0x7fec680cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:17.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6d8d8700 1 --2- 172.21.15.1:0/780656540 >> v2:172.21.15.165:3300/0 conn(0x7fec6809dd40 0x7fec680cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:17.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6d0d7700 1 -- 172.21.15.1:0/780656540 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200313469 0 0) 0x7fec680ce090 con 0x7fec680ce5a0 2024-03-18T20:26:17.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6d0d7700 1 -- 172.21.15.1:0/780656540 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec540035b0 con 0x7fec680ce5a0 2024-03-18T20:26:17.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6d0d7700 1 -- 172.21.15.1:0/780656540 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2813730283 0 0) 0x7fec540035b0 con 0x7fec680ce5a0 2024-03-18T20:26:17.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6d0d7700 1 -- 172.21.15.1:0/780656540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec6815d780 msgr2=0x7fec680cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:17.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6d0d7700 1 --2- 172.21.15.1:0/780656540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec6815d780 0x7fec680cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:17.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6d0d7700 1 -- 172.21.15.1:0/780656540 >> v2:172.21.15.165:3300/0 conn(0x7fec6809dd40 msgr2=0x7fec680cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:17.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6d0d7700 1 --2- 172.21.15.1:0/780656540 >> v2:172.21.15.165:3300/0 conn(0x7fec6809dd40 0x7fec680cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:17.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6d0d7700 1 -- 172.21.15.1:0/780656540 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6806d520 con 0x7fec680ce5a0 2024-03-18T20:26:17.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6d0d7700 1 -- 172.21.15.1:0/780656540 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fec600019f0 con 0x7fec680ce5a0 2024-03-18T20:26:17.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6d0d7700 1 -- 172.21.15.1:0/780656540 >> v1:172.21.15.1:6789/0 conn(0x7fec680ce5a0 legacy=0x7fec680cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:17.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6d0d7700 1 --2- 172.21.15.1:0/780656540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec54003a00 0x7fec54003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:17.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6d0d7700 1 --2- 172.21.15.1:0/780656540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec6815d780 0x7fec54004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:17.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6d0d7700 1 --2- 172.21.15.1:0/780656540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec54004d90 0x7fec54005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:17.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6d0d7700 1 -- 172.21.15.1:0/780656540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6806db30 con 0x7fec54004d90 2024-03-18T20:26:17.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6d0d7700 1 -- 172.21.15.1:0/780656540 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5400ba10 con 0x7fec54003a00 2024-03-18T20:26:17.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.205+0000 7fec6d0d7700 1 -- 172.21.15.1:0/780656540 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5400bb50 con 0x7fec6815d780 2024-03-18T20:26:17.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.206+0000 7fec6e8da700 1 --2- 172.21.15.1:0/780656540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec54004d90 0x7fec54005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:17.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.206+0000 7fec6d8d8700 1 --2- 172.21.15.1:0/780656540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec6815d780 0x7fec54004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:17.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.206+0000 7fec6e0d9700 1 --2- 172.21.15.1:0/780656540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec54003a00 0x7fec54003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:17.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.206+0000 7fec6e8da700 1 -- 172.21.15.1:0/780656540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec6815d780 msgr2=0x7fec54004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:17.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.206+0000 7fec6e8da700 1 --2- 172.21.15.1:0/780656540 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec6815d780 0x7fec54004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:17.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.206+0000 7fec6e8da700 1 -- 172.21.15.1:0/780656540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec54003a00 msgr2=0x7fec54003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:17.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.206+0000 7fec6e8da700 1 --2- 172.21.15.1:0/780656540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec54003a00 0x7fec54003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:17.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.206+0000 7fec6e8da700 1 -- 172.21.15.1:0/780656540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec540035b0 con 0x7fec54004d90 2024-03-18T20:26:17.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.206+0000 7fec6e8da700 1 --2- 172.21.15.1:0/780656540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec54004d90 0x7fec54005210 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7fec6000a860 tx=0x7fec54005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8da5e83257c952f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:17.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.206+0000 7fec6d0d7700 1 -- 172.21.15.1:0/780656540 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec60012600 con 0x7fec54004d90 2024-03-18T20:26:17.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.206+0000 7fec6d0d7700 1 -- 172.21.15.1:0/780656540 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec60012780 con 0x7fec54004d90 2024-03-18T20:26:17.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.206+0000 7fec6d0d7700 1 -- 172.21.15.1:0/780656540 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec60012b30 con 0x7fec54004d90 2024-03-18T20:26:17.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.207+0000 7fec7033d700 1 -- 172.21.15.1:0/780656540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec54004d90 msgr2=0x7fec54005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:17.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.207+0000 7fec7033d700 1 --2- 172.21.15.1:0/780656540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec54004d90 0x7fec54005210 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7fec6000a860 tx=0x7fec54005880 comp rx=0 tx=0).stop 2024-03-18T20:26:17.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.207+0000 7fec6e8da700 1 -- 172.21.15.1:0/780656540 reap_dead start 2024-03-18T20:26:17.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.207+0000 7fec7033d700 1 -- 172.21.15.1:0/780656540 shutdown_connections 2024-03-18T20:26:17.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.207+0000 7fec7033d700 1 -- 172.21.15.1:0/780656540 >> 172.21.15.1:0/780656540 conn(0x7fec680cca00 msgr2=0x7fec680cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:17.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.207+0000 7fec7033d700 1 -- 172.21.15.1:0/780656540 shutdown_connections 2024-03-18T20:26:17.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.207+0000 7fec7033d700 1 -- 172.21.15.1:0/780656540 wait complete. 2024-03-18T20:26:17.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.208+0000 7fec7033d700 1 Processor -- start 2024-03-18T20:26:17.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.208+0000 7fec7033d700 1 -- start start 2024-03-18T20:26:17.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.209+0000 7fec7033d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec54004d90 0x7fec6815a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:17.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.209+0000 7fec7033d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec6815a940 0x7fec6806a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:17.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.209+0000 7fec7033d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec6806ac90 0x7fec680c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:17.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.209+0000 7fec7033d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6806b110 con 0x7fec54004d90 2024-03-18T20:26:17.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.209+0000 7fec7033d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec68156ff0 con 0x7fec6806ac90 2024-03-18T20:26:17.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.209+0000 7fec7033d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec68159f70 con 0x7fec6815a940 2024-03-18T20:26:17.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.209+0000 7fec6e0d9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec54004d90 0x7fec6815a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:17.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.210+0000 7fec6e0d9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec54004d90 0x7fec6815a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56440/0 (socket says 172.21.15.1:56440) 2024-03-18T20:26:17.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.210+0000 7fec6e0d9700 1 -- 172.21.15.1:0/3452059911 learned_addr learned my addr 172.21.15.1:0/3452059911 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:17.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.210+0000 7fec6e0d9700 1 -- 172.21.15.1:0/3452059911 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec6815a940 msgr2=0x7fec6806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:17.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.210+0000 7fec6d8d8700 1 --2- 172.21.15.1:0/3452059911 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec6815a940 0x7fec6806a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:17.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.210+0000 7fec6e0d9700 1 --2- 172.21.15.1:0/3452059911 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec6815a940 0x7fec6806a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:17.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.210+0000 7fec6e0d9700 1 -- 172.21.15.1:0/3452059911 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec6806ac90 msgr2=0x7fec680c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:17.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.210+0000 7fec6e0d9700 1 --2- 172.21.15.1:0/3452059911 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec6806ac90 0x7fec680c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:17.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.210+0000 7fec6e0d9700 1 -- 172.21.15.1:0/3452059911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec68159360 con 0x7fec54004d90 2024-03-18T20:26:17.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.210+0000 7fec6e0d9700 1 --2- 172.21.15.1:0/3452059911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec54004d90 0x7fec6815a400 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7fec64011aa0 tx=0x7fec64011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:17.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.210+0000 7fec5effd700 1 -- 172.21.15.1:0/3452059911 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6400f900 con 0x7fec54004d90 2024-03-18T20:26:17.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.210+0000 7fec5effd700 1 -- 172.21.15.1:0/3452059911 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6400fa60 con 0x7fec54004d90 2024-03-18T20:26:17.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.210+0000 7fec7033d700 1 -- 172.21.15.1:0/3452059911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec68159640 con 0x7fec54004d90 2024-03-18T20:26:17.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.210+0000 7fec5effd700 1 -- 172.21.15.1:0/3452059911 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6400fe10 con 0x7fec54004d90 2024-03-18T20:26:17.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.210+0000 7fec7033d700 1 -- 172.21.15.1:0/3452059911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec68159c00 con 0x7fec54004d90 2024-03-18T20:26:17.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.211+0000 7fec7033d700 1 -- 172.21.15.1:0/3452059911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec680676a0 con 0x7fec54004d90 2024-03-18T20:26:17.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.212+0000 7fec5effd700 1 -- 172.21.15.1:0/3452059911 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fec64014440 con 0x7fec54004d90 2024-03-18T20:26:17.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.212+0000 7fec5effd700 1 --2- 172.21.15.1:0/3452059911 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fec4c06aed0 0x7fec4c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:17.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.212+0000 7fec5effd700 1 -- 172.21.15.1:0/3452059911 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6180+0+0 (secure 0 0 0) 0x7fec6408a070 con 0x7fec54004d90 2024-03-18T20:26:17.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.212+0000 7fec6d8d8700 1 --2- 172.21.15.1:0/3452059911 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fec4c06aed0 0x7fec4c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:17.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.213+0000 7fec6d8d8700 1 --2- 172.21.15.1:0/3452059911 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fec4c06aed0 0x7fec4c06d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fec680c9440 tx=0x7fec58008040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:17.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.217+0000 7fec5effd700 1 -- 172.21.15.1:0/3452059911 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fec64054dd0 con 0x7fec54004d90 2024-03-18T20:26:17.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.427+0000 7fec7033d700 1 -- 172.21.15.1:0/3452059911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fec6809d8d0 con 0x7fec54004d90 2024-03-18T20:26:17.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.431+0000 7fec5effd700 1 -- 172.21.15.1:0/3452059911 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v81) v1 ==== 104+0+0 (secure 0 0 0) 0x7fec640583f0 con 0x7fec54004d90 2024-03-18T20:26:17.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.433+0000 7fec7033d700 1 -- 172.21.15.1:0/3452059911 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fec4c06aed0 msgr2=0x7fec4c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:17.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.433+0000 7fec7033d700 1 --2- 172.21.15.1:0/3452059911 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fec4c06aed0 0x7fec4c06d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fec680c9440 tx=0x7fec58008040 comp rx=0 tx=0).stop 2024-03-18T20:26:17.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.433+0000 7fec7033d700 1 -- 172.21.15.1:0/3452059911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec54004d90 msgr2=0x7fec6815a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:17.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.433+0000 7fec7033d700 1 --2- 172.21.15.1:0/3452059911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec54004d90 0x7fec6815a400 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7fec64011aa0 tx=0x7fec64011e60 comp rx=0 tx=0).stop 2024-03-18T20:26:17.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.433+0000 7fec7033d700 1 -- 172.21.15.1:0/3452059911 shutdown_connections 2024-03-18T20:26:17.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.433+0000 7fec7033d700 1 --2- 172.21.15.1:0/3452059911 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fec4c06aed0 0x7fec4c06d390 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:17.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.433+0000 7fec7033d700 1 --2- 172.21.15.1:0/3452059911 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec6806ac90 0x7fec680c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:17.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.433+0000 7fec7033d700 1 --2- 172.21.15.1:0/3452059911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec54004d90 0x7fec6815a400 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:17.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.433+0000 7fec7033d700 1 --2- 172.21.15.1:0/3452059911 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fec6815a940 0x7fec6806a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:17.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.433+0000 7fec7033d700 1 -- 172.21.15.1:0/3452059911 >> 172.21.15.1:0/3452059911 conn(0x7fec680cca00 msgr2=0x7fec680b02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:17.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.433+0000 7fec7033d700 1 -- 172.21.15.1:0/3452059911 shutdown_connections 2024-03-18T20:26:17.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.433+0000 7fec7033d700 1 -- 172.21.15.1:0/3452059911 wait complete. 2024-03-18T20:26:17.450 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2024-03-18T20:26:17.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.570+0000 7f6515967700 1 Processor -- start 2024-03-18T20:26:17.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.570+0000 7f6515967700 1 -- start start 2024-03-18T20:26:17.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.571+0000 7f6515967700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f65100cf090 0x7f65100d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:17.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.571+0000 7f6515967700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6510160d20 0x7f65100d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:17.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.571+0000 7f6515967700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65100cddc0 con 0x7f6510160940 2024-03-18T20:26:17.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.571+0000 7f6515967700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65100cdff0 con 0x7f6510160d20 2024-03-18T20:26:17.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.571+0000 7f6515967700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65100ce2d0 con 0x7f65100cf090 2024-03-18T20:26:17.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.571+0000 7f650f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6510160d20 0x7f65100d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:17.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.571+0000 7f650f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6510160d20 0x7f65100d1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56448/0 (socket says 172.21.15.1:56448) 2024-03-18T20:26:17.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.571+0000 7f650f7fe700 1 -- 172.21.15.1:0/524848779 learned_addr learned my addr 172.21.15.1:0/524848779 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:17.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.571+0000 7f650e7fc700 1 --2- 172.21.15.1:0/524848779 >> v2:172.21.15.165:3300/0 conn(0x7f65100cf090 0x7f65100d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:17.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.571+0000 7f650f7fe700 1 -- 172.21.15.1:0/524848779 >> v2:172.21.15.165:3300/0 conn(0x7f65100cf090 msgr2=0x7f65100d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:17.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.571+0000 7f650f7fe700 1 --2- 172.21.15.1:0/524848779 >> v2:172.21.15.165:3300/0 conn(0x7f65100cf090 0x7f65100d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:17.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.571+0000 7f650f7fe700 1 -- 172.21.15.1:0/524848779 >> v1:172.21.15.165:6789/0 conn(0x7f6510160940 legacy=0x7f65100d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:17.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.571+0000 7f650f7fe700 1 -- 172.21.15.1:0/524848779 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65100ce5b0 con 0x7f6510160d20 2024-03-18T20:26:17.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.571+0000 7f650e7fc700 1 --2- 172.21.15.1:0/524848779 >> v2:172.21.15.165:3300/0 conn(0x7f65100cf090 0x7f65100d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:17.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.571+0000 7f650f7fe700 1 --2- 172.21.15.1:0/524848779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6510160d20 0x7f65100d1cd0 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f65100b9230 tx=0x7f650800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb38407e4746e1d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:17.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.572+0000 7f650dffb700 1 -- 172.21.15.1:0/524848779 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650800e070 con 0x7f6510160d20 2024-03-18T20:26:17.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.572+0000 7f650dffb700 1 -- 172.21.15.1:0/524848779 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6508003a20 con 0x7f6510160d20 2024-03-18T20:26:17.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.572+0000 7f650dffb700 1 -- 172.21.15.1:0/524848779 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6508003dd0 con 0x7f6510160d20 2024-03-18T20:26:17.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.572+0000 7f6515967700 1 -- 172.21.15.1:0/524848779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6510160d20 msgr2=0x7f65100d1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:17.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.572+0000 7f6515967700 1 --2- 172.21.15.1:0/524848779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6510160d20 0x7f65100d1cd0 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f65100b9230 tx=0x7f650800b330 comp rx=0 tx=0).stop 2024-03-18T20:26:17.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.572+0000 7f6515967700 1 -- 172.21.15.1:0/524848779 shutdown_connections 2024-03-18T20:26:17.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.572+0000 7f6515967700 1 --2- 172.21.15.1:0/524848779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6510160d20 0x7f65100d1cd0 unknown :-1 s=CLOSED pgs=2759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:17.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.572+0000 7f6515967700 1 --2- 172.21.15.1:0/524848779 >> v2:172.21.15.165:3300/0 conn(0x7f65100cf090 0x7f65100d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:17.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.572+0000 7f6515967700 1 -- 172.21.15.1:0/524848779 >> 172.21.15.1:0/524848779 conn(0x7f651009c520 msgr2=0x7f65100d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:17.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.572+0000 7f6515967700 1 -- 172.21.15.1:0/524848779 shutdown_connections 2024-03-18T20:26:17.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.572+0000 7f6515967700 1 -- 172.21.15.1:0/524848779 wait complete. 2024-03-18T20:26:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.574+0000 7f6515967700 1 Processor -- start 2024-03-18T20:26:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.574+0000 7f6515967700 1 -- start start 2024-03-18T20:26:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.575+0000 7f6515967700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65100cf090 0x7f65100b8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.575+0000 7f6515967700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6510160940 0x7f651015b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:17.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.575+0000 7f6515967700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6510160d20 0x7f65100ca280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:17.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.575+0000 7f6515967700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65100b8ca0 con 0x7f6510160d20 2024-03-18T20:26:17.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.575+0000 7f6515967700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65100c7da0 con 0x7f65100cf090 2024-03-18T20:26:17.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.575+0000 7f6515967700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65100c8080 con 0x7f6510160940 2024-03-18T20:26:17.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.575+0000 7f650f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6510160d20 0x7f65100ca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:17.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.575+0000 7f650f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6510160d20 0x7f65100ca280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56460/0 (socket says 172.21.15.1:56460) 2024-03-18T20:26:17.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.575+0000 7f650f7fe700 1 -- 172.21.15.1:0/2681679107 learned_addr learned my addr 172.21.15.1:0/2681679107 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:17.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.575+0000 7f650effd700 1 --2- 172.21.15.1:0/2681679107 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65100cf090 0x7f65100b8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:17.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.575+0000 7f650e7fc700 1 --2- 172.21.15.1:0/2681679107 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6510160940 0x7f651015b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:17.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.575+0000 7f650f7fe700 1 -- 172.21.15.1:0/2681679107 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6510160940 msgr2=0x7f651015b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:17.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.575+0000 7f650f7fe700 1 --2- 172.21.15.1:0/2681679107 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6510160940 0x7f651015b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:17.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.575+0000 7f650f7fe700 1 -- 172.21.15.1:0/2681679107 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65100cf090 msgr2=0x7f65100b8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:17.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.575+0000 7f650f7fe700 1 --2- 172.21.15.1:0/2681679107 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65100cf090 0x7f65100b8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.575+0000 7f650f7fe700 1 -- 172.21.15.1:0/2681679107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6508009b30 con 0x7f6510160d20 2024-03-18T20:26:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.576+0000 7f650f7fe700 1 --2- 172.21.15.1:0/2681679107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6510160d20 0x7f65100ca280 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7f6508015040 tx=0x7f6508004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.576+0000 7f64f7fff700 1 -- 172.21.15.1:0/2681679107 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6508004440 con 0x7f6510160d20 2024-03-18T20:26:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.576+0000 7f64f7fff700 1 -- 172.21.15.1:0/2681679107 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65080045a0 con 0x7f6510160d20 2024-03-18T20:26:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.576+0000 7f6515967700 1 -- 172.21.15.1:0/2681679107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65100c8360 con 0x7f6510160d20 2024-03-18T20:26:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.576+0000 7f64f7fff700 1 -- 172.21.15.1:0/2681679107 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6508004950 con 0x7f6510160d20 2024-03-18T20:26:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.576+0000 7f6515967700 1 -- 172.21.15.1:0/2681679107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65100c8920 con 0x7f6510160d20 2024-03-18T20:26:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.577+0000 7f6515967700 1 -- 172.21.15.1:0/2681679107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6510073350 con 0x7f6510160d20 2024-03-18T20:26:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.577+0000 7f64f7fff700 1 -- 172.21.15.1:0/2681679107 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f6508012430 con 0x7f6510160d20 2024-03-18T20:26:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.578+0000 7f64f7fff700 1 --2- 172.21.15.1:0/2681679107 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f64fc06af20 0x7f64fc06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.578+0000 7f650effd700 1 --2- 172.21.15.1:0/2681679107 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f64fc06af20 0x7f64fc06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.578+0000 7f64f7fff700 1 -- 172.21.15.1:0/2681679107 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6180+0+0 (secure 0 0 0) 0x7f650808aef0 con 0x7f6510160d20 2024-03-18T20:26:17.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.578+0000 7f650effd700 1 --2- 172.21.15.1:0/2681679107 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f64fc06af20 0x7f64fc06d3e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6504004f30 tx=0x7f6504009040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:17.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.583+0000 7f64f7fff700 1 -- 172.21.15.1:0/2681679107 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6508059520 con 0x7f6510160d20 2024-03-18T20:26:17.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:17.873+0000 7f6515967700 1 -- 172.21.15.1:0/2681679107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7f651015dbb0 con 0x7f6510160d20 2024-03-18T20:26:18.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.322+0000 7f64f7fff700 1 -- 172.21.15.1:0/2681679107 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3159) v1 ==== 186+0+0 (secure 0 0 0) 0x7f65080590b0 con 0x7f6510160d20 2024-03-18T20:26:18.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.325+0000 7f6515967700 1 -- 172.21.15.1:0/2681679107 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f64fc06af20 msgr2=0x7f64fc06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:18.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.325+0000 7f6515967700 1 --2- 172.21.15.1:0/2681679107 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f64fc06af20 0x7f64fc06d3e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6504004f30 tx=0x7f6504009040 comp rx=0 tx=0).stop 2024-03-18T20:26:18.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.325+0000 7f6515967700 1 -- 172.21.15.1:0/2681679107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6510160d20 msgr2=0x7f65100ca280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:18.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.325+0000 7f6515967700 1 --2- 172.21.15.1:0/2681679107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6510160d20 0x7f65100ca280 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7f6508015040 tx=0x7f6508004ad0 comp rx=0 tx=0).stop 2024-03-18T20:26:18.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.325+0000 7f6515967700 1 -- 172.21.15.1:0/2681679107 shutdown_connections 2024-03-18T20:26:18.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.325+0000 7f6515967700 1 --2- 172.21.15.1:0/2681679107 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f64fc06af20 0x7f64fc06d3e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:18.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.325+0000 7f6515967700 1 --2- 172.21.15.1:0/2681679107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6510160d20 0x7f65100ca280 unknown :-1 s=CLOSED pgs=2760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:18.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.325+0000 7f6515967700 1 --2- 172.21.15.1:0/2681679107 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65100cf090 0x7f65100b8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:18.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.325+0000 7f6515967700 1 --2- 172.21.15.1:0/2681679107 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6510160940 0x7f651015b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:18.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.325+0000 7f6515967700 1 -- 172.21.15.1:0/2681679107 >> 172.21.15.1:0/2681679107 conn(0x7f651009c520 msgr2=0x7f65100cf4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:18.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.325+0000 7f6515967700 1 -- 172.21.15.1:0/2681679107 shutdown_connections 2024-03-18T20:26:18.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.325+0000 7f6515967700 1 -- 172.21.15.1:0/2681679107 wait complete. 2024-03-18T20:26:18.345 INFO:tasks.cephfs_test_runner:Test command succes with --fs argument for non-default fs ... ok 2024-03-18T20:26:18.345 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command fails without --fs argument in presence of multiple fs 2024-03-18T20:26:18.345 INFO:tasks.cephfs_test_runner:test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2024-03-18T20:26:18.345 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2024-03-18T20:26:18.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.470+0000 7f71d62d2700 1 Processor -- start 2024-03-18T20:26:18.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.470+0000 7f71d62d2700 1 -- start start 2024-03-18T20:26:18.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.471+0000 7f71d62d2700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f71d015b4d0 0x7f71d009a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:18.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.471+0000 7f71d62d2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f71d00cf750 0x7f71d0099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.471+0000 7f71d62d2700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d006f070 con 0x7f71d0099f80 2024-03-18T20:26:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.471+0000 7f71d62d2700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d009a3f0 con 0x7f71d00cf750 2024-03-18T20:26:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.471+0000 7f71d62d2700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d00d0260 con 0x7f71d015b4d0 2024-03-18T20:26:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.471+0000 7f71cf7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f71d00cf750 0x7f71d0099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.471+0000 7f71cf7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f71d00cf750 0x7f71d0099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56474/0 (socket says 172.21.15.1:56474) 2024-03-18T20:26:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.471+0000 7f71cf7fe700 1 -- 172.21.15.1:0/3401736721 learned_addr learned my addr 172.21.15.1:0/3401736721 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.471+0000 7f71cf7fe700 1 -- 172.21.15.1:0/3401736721 >> v2:172.21.15.165:3300/0 conn(0x7f71d015b4d0 msgr2=0x7f71d009a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.471+0000 7f71cf7fe700 1 --2- 172.21.15.1:0/3401736721 >> v2:172.21.15.165:3300/0 conn(0x7f71d015b4d0 0x7f71d009a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.471+0000 7f71cf7fe700 1 -- 172.21.15.1:0/3401736721 >> v1:172.21.15.165:6789/0 conn(0x7f71d0099f80 legacy=0x7f71d00cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.471+0000 7f71cf7fe700 1 -- 172.21.15.1:0/3401736721 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d00cff80 con 0x7f71d00cf750 2024-03-18T20:26:18.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.471+0000 7f71cf7fe700 1 --2- 172.21.15.1:0/3401736721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d00cf750 0x7f71d0099a40 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7f71d0071290 tx=0x7f71c0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda54dcf089cce44 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.471+0000 7f71ceffd700 1 -- 172.21.15.1:0/3401736721 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c0004230 con 0x7f71d00cf750 2024-03-18T20:26:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.472+0000 7f71ceffd700 1 -- 172.21.15.1:0/3401736721 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c0004390 con 0x7f71d00cf750 2024-03-18T20:26:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.472+0000 7f71ceffd700 1 -- 172.21.15.1:0/3401736721 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c00036a0 con 0x7f71d00cf750 2024-03-18T20:26:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.472+0000 7f71d62d2700 1 -- 172.21.15.1:0/3401736721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d00cf750 msgr2=0x7f71d0099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.472+0000 7f71d62d2700 1 --2- 172.21.15.1:0/3401736721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d00cf750 0x7f71d0099a40 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7f71d0071290 tx=0x7f71c0009bd0 comp rx=0 tx=0).stop 2024-03-18T20:26:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.472+0000 7f71d62d2700 1 -- 172.21.15.1:0/3401736721 shutdown_connections 2024-03-18T20:26:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.472+0000 7f71d62d2700 1 --2- 172.21.15.1:0/3401736721 >> v2:172.21.15.165:3300/0 conn(0x7f71d015b4d0 0x7f71d009a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.472+0000 7f71d62d2700 1 --2- 172.21.15.1:0/3401736721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d00cf750 0x7f71d0099a40 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.472+0000 7f71d62d2700 1 -- 172.21.15.1:0/3401736721 >> 172.21.15.1:0/3401736721 conn(0x7f71d00b2cb0 msgr2=0x7f71d00b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.472+0000 7f71d62d2700 1 -- 172.21.15.1:0/3401736721 shutdown_connections 2024-03-18T20:26:18.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.472+0000 7f71d62d2700 1 -- 172.21.15.1:0/3401736721 wait complete. 2024-03-18T20:26:18.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.473+0000 7f71d62d2700 1 Processor -- start 2024-03-18T20:26:18.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.473+0000 7f71d62d2700 1 -- start start 2024-03-18T20:26:18.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.474+0000 7f71d62d2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f71d0099f80 0x7f71d0154a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:18.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.474+0000 7f71d62d2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71d00cf750 0x7f71d015ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:18.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.474+0000 7f71d62d2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d015b4d0 0x7f71d015bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.474+0000 7f71d62d2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d015d2d0 con 0x7f71d015b4d0 2024-03-18T20:26:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.474+0000 7f71d62d2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d00b0050 con 0x7f71d00cf750 2024-03-18T20:26:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.474+0000 7f71d62d2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d00afbc0 con 0x7f71d0099f80 2024-03-18T20:26:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.474+0000 7f71d486f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d015b4d0 0x7f71d015bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.474+0000 7f71d486f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d015b4d0 0x7f71d015bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56490/0 (socket says 172.21.15.1:56490) 2024-03-18T20:26:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.474+0000 7f71d486f700 1 -- 172.21.15.1:0/9010193 learned_addr learned my addr 172.21.15.1:0/9010193 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.474+0000 7f71cf7fe700 1 --2- 172.21.15.1:0/9010193 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71d00cf750 0x7f71d015ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.474+0000 7f71cffff700 1 --2- 172.21.15.1:0/9010193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f71d0099f80 0x7f71d0154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.475+0000 7f71d486f700 1 -- 172.21.15.1:0/9010193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f71d0099f80 msgr2=0x7f71d0154a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.475+0000 7f71d486f700 1 --2- 172.21.15.1:0/9010193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f71d0099f80 0x7f71d0154a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.475+0000 7f71d486f700 1 -- 172.21.15.1:0/9010193 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71d00cf750 msgr2=0x7f71d015ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.475+0000 7f71d486f700 1 --2- 172.21.15.1:0/9010193 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71d00cf750 0x7f71d015ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:18.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.475+0000 7f71d486f700 1 -- 172.21.15.1:0/9010193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c0009550 con 0x7f71d015b4d0 2024-03-18T20:26:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.475+0000 7f71cf7fe700 1 --2- 172.21.15.1:0/9010193 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71d00cf750 0x7f71d015ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.475+0000 7f71cffff700 1 --2- 172.21.15.1:0/9010193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f71d0099f80 0x7f71d0154a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.475+0000 7f71d486f700 1 --2- 172.21.15.1:0/9010193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d015b4d0 0x7f71d015bf70 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f71c800cc40 tx=0x7f71c800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.475+0000 7f71ccff9700 1 -- 172.21.15.1:0/9010193 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c800aba0 con 0x7f71d015b4d0 2024-03-18T20:26:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.475+0000 7f71ccff9700 1 -- 172.21.15.1:0/9010193 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c800ad00 con 0x7f71d015b4d0 2024-03-18T20:26:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.475+0000 7f71d62d2700 1 -- 172.21.15.1:0/9010193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d00af030 con 0x7f71d015b4d0 2024-03-18T20:26:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.475+0000 7f71ccff9700 1 -- 172.21.15.1:0/9010193 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c8012660 con 0x7f71d015b4d0 2024-03-18T20:26:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.475+0000 7f71d62d2700 1 -- 172.21.15.1:0/9010193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d00af290 con 0x7f71d015b4d0 2024-03-18T20:26:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.476+0000 7f71d62d2700 1 -- 172.21.15.1:0/9010193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d0068240 con 0x7f71d015b4d0 2024-03-18T20:26:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.476+0000 7f71ccff9700 1 -- 172.21.15.1:0/9010193 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f71c80127c0 con 0x7f71d015b4d0 2024-03-18T20:26:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.477+0000 7f71ccff9700 1 --2- 172.21.15.1:0/9010193 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f71b006aed0 0x7f71b006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.477+0000 7f71ccff9700 1 -- 172.21.15.1:0/9010193 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6180+0+0 (secure 0 0 0) 0x7f71c808a0d0 con 0x7f71d015b4d0 2024-03-18T20:26:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.477+0000 7f71cffff700 1 --2- 172.21.15.1:0/9010193 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f71b006aed0 0x7f71b006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:18.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.477+0000 7f71cffff700 1 --2- 172.21.15.1:0/9010193 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f71b006aed0 0x7f71b006d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f71b8006fd0 tx=0x7f71b8008040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:18.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.482+0000 7f71ccff9700 1 -- 172.21.15.1:0/9010193 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f71c8054d70 con 0x7f71d015b4d0 2024-03-18T20:26:18.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:18.771+0000 7f71d62d2700 1 -- 172.21.15.1:0/9010193 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7f71d00afd30 con 0x7f71d015b4d0 2024-03-18T20:26:19.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.325+0000 7f71ccff9700 1 -- 172.21.15.1:0/9010193 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3160) v1 ==== 190+0+0 (secure 0 0 0) 0x7f71c8016100 con 0x7f71d015b4d0 2024-03-18T20:26:19.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.328+0000 7f71d62d2700 1 -- 172.21.15.1:0/9010193 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f71b006aed0 msgr2=0x7f71b006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:19.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.328+0000 7f71d62d2700 1 --2- 172.21.15.1:0/9010193 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f71b006aed0 0x7f71b006d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f71b8006fd0 tx=0x7f71b8008040 comp rx=0 tx=0).stop 2024-03-18T20:26:19.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.328+0000 7f71d62d2700 1 -- 172.21.15.1:0/9010193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d015b4d0 msgr2=0x7f71d015bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.328+0000 7f71d62d2700 1 --2- 172.21.15.1:0/9010193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d015b4d0 0x7f71d015bf70 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f71c800cc40 tx=0x7f71c800a060 comp rx=0 tx=0).stop 2024-03-18T20:26:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.328+0000 7f71d62d2700 1 -- 172.21.15.1:0/9010193 shutdown_connections 2024-03-18T20:26:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.328+0000 7f71d62d2700 1 --2- 172.21.15.1:0/9010193 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f71b006aed0 0x7f71b006d390 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.328+0000 7f71d62d2700 1 --2- 172.21.15.1:0/9010193 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f71d0099f80 0x7f71d0154a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.328+0000 7f71d62d2700 1 --2- 172.21.15.1:0/9010193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d015b4d0 0x7f71d015bf70 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.328+0000 7f71d62d2700 1 --2- 172.21.15.1:0/9010193 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71d00cf750 0x7f71d015ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.328+0000 7f71d62d2700 1 -- 172.21.15.1:0/9010193 >> 172.21.15.1:0/9010193 conn(0x7f71d00b2cb0 msgr2=0x7f71d00ade00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.328+0000 7f71d62d2700 1 -- 172.21.15.1:0/9010193 shutdown_connections 2024-03-18T20:26:19.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.328+0000 7f71d62d2700 1 -- 172.21.15.1:0/9010193 wait complete. 2024-03-18T20:26:19.348 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-18T20:26:19.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.478+0000 7fda11375700 1 Processor -- start 2024-03-18T20:26:19.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.478+0000 7fda11375700 1 -- start start 2024-03-18T20:26:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.479+0000 7fda11375700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fda0c15d500 0x7fda0c06c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.479+0000 7fda11375700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda0c06f490 con 0x7fda0c06c7b0 2024-03-18T20:26:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.479+0000 7fda11375700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda0c06e880 con 0x7fda0c09a150 2024-03-18T20:26:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.479+0000 7fda11375700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c06eb60 con 0x7fda0c15d500 2024-03-18T20:26:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.479+0000 7fda0a7fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fda0c06c7b0 0x7fda0c099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42350/0 (socket says 172.21.15.1:42350) 2024-03-18T20:26:19.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.479+0000 7fda0a7fc700 1 -- 172.21.15.1:0/2545868060 learned_addr learned my addr 172.21.15.1:0/2545868060 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda0affd700 1 --2- 172.21.15.1:0/2545868060 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fda0c15d500 0x7fda0c06c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda09ffb700 1 -- 172.21.15.1:0/2545868060 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188889419 0 0) 0x7fda0c06f490 con 0x7fda0c06c7b0 2024-03-18T20:26:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda09ffb700 1 -- 172.21.15.1:0/2545868060 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9f40035b0 con 0x7fda0c06c7b0 2024-03-18T20:26:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda09ffb700 1 -- 172.21.15.1:0/2545868060 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4241501334 0 0) 0x7fd9f40035b0 con 0x7fda0c06c7b0 2024-03-18T20:26:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda09ffb700 1 -- 172.21.15.1:0/2545868060 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fda0c15d500 msgr2=0x7fda0c06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda09ffb700 1 --2- 172.21.15.1:0/2545868060 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fda0c15d500 0x7fda0c06c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda09ffb700 1 -- 172.21.15.1:0/2545868060 >> v1:172.21.15.165:6789/0 conn(0x7fda0c09a150 legacy=0x7fda0c0cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda09ffb700 1 -- 172.21.15.1:0/2545868060 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0c06ee40 con 0x7fda0c06c7b0 2024-03-18T20:26:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda09ffb700 1 -- 172.21.15.1:0/2545868060 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fd9fc001670 con 0x7fda0c06c7b0 2024-03-18T20:26:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda09ffb700 1 -- 172.21.15.1:0/2545868060 >> v1:172.21.15.1:6789/0 conn(0x7fda0c06c7b0 legacy=0x7fda0c099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda09ffb700 1 --2- 172.21.15.1:0/2545868060 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9f4003a30 0x7fd9f4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda09ffb700 1 --2- 172.21.15.1:0/2545868060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f4004850 0x7fd9f4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda09ffb700 1 --2- 172.21.15.1:0/2545868060 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fda0c15d500 0x7fd9f4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:19.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda09ffb700 1 -- 172.21.15.1:0/2545868060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c06eb60 con 0x7fd9f4004850 2024-03-18T20:26:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda09ffb700 1 -- 172.21.15.1:0/2545868060 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f400bac0 con 0x7fd9f4003a30 2024-03-18T20:26:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda09ffb700 1 -- 172.21.15.1:0/2545868060 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f400bc80 con 0x7fda0c15d500 2024-03-18T20:26:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda0a7fc700 1 --2- 172.21.15.1:0/2545868060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f4004850 0x7fd9f4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda0affd700 1 --2- 172.21.15.1:0/2545868060 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9f4003a30 0x7fd9f4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda0b7fe700 1 --2- 172.21.15.1:0/2545868060 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fda0c15d500 0x7fd9f4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda0a7fc700 1 -- 172.21.15.1:0/2545868060 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fda0c15d500 msgr2=0x7fd9f4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda0a7fc700 1 --2- 172.21.15.1:0/2545868060 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fda0c15d500 0x7fd9f4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda0a7fc700 1 -- 172.21.15.1:0/2545868060 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9f4003a30 msgr2=0x7fd9f4003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda0a7fc700 1 --2- 172.21.15.1:0/2545868060 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9f4003a30 0x7fd9f4003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.480+0000 7fda0a7fc700 1 -- 172.21.15.1:0/2545868060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f40035b0 con 0x7fd9f4004850 2024-03-18T20:26:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.481+0000 7fda0a7fc700 1 --2- 172.21.15.1:0/2545868060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f4004850 0x7fd9f4004cd0 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc00c210 tx=0x7fd9f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c4c934f3da0857 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.481+0000 7fda09ffb700 1 -- 172.21.15.1:0/2545868060 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc007c10 con 0x7fd9f4004850 2024-03-18T20:26:19.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.481+0000 7fda09ffb700 1 -- 172.21.15.1:0/2545868060 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9fc00cdd0 con 0x7fd9f4004850 2024-03-18T20:26:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.481+0000 7fda09ffb700 1 -- 172.21.15.1:0/2545868060 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc005ba0 con 0x7fd9f4004850 2024-03-18T20:26:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.481+0000 7fda11375700 1 -- 172.21.15.1:0/2545868060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f4004850 msgr2=0x7fd9f4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.481+0000 7fda11375700 1 --2- 172.21.15.1:0/2545868060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f4004850 0x7fd9f4004cd0 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc00c210 tx=0x7fd9f4005880 comp rx=0 tx=0).stop 2024-03-18T20:26:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.481+0000 7fda0b7fe700 1 -- 172.21.15.1:0/2545868060 reap_dead start 2024-03-18T20:26:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.481+0000 7fda11375700 1 -- 172.21.15.1:0/2545868060 shutdown_connections 2024-03-18T20:26:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.481+0000 7fda11375700 1 -- 172.21.15.1:0/2545868060 >> 172.21.15.1:0/2545868060 conn(0x7fda0c0b2c50 msgr2=0x7fda0c0b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.481+0000 7fda11375700 1 -- 172.21.15.1:0/2545868060 shutdown_connections 2024-03-18T20:26:19.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.481+0000 7fda11375700 1 -- 172.21.15.1:0/2545868060 wait complete. 2024-03-18T20:26:19.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.483+0000 7fda11375700 1 Processor -- start 2024-03-18T20:26:19.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.483+0000 7fda11375700 1 -- start start 2024-03-18T20:26:19.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.483+0000 7fda11375700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f4004850 0x7fda0c0cd290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:19.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.483+0000 7fda11375700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fda0c1520e0 0x7fda0c151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:19.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.483+0000 7fda11375700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda0c15f320 0x7fda0c0ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:19.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.483+0000 7fda11375700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c0cda70 con 0x7fd9f4004850 2024-03-18T20:26:19.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.483+0000 7fda11375700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c15e5a0 con 0x7fda0c15f320 2024-03-18T20:26:19.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.483+0000 7fda11375700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c0cd7d0 con 0x7fda0c1520e0 2024-03-18T20:26:19.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.484+0000 7fda0affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f4004850 0x7fda0c0cd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:19.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.484+0000 7fda0affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f4004850 0x7fda0c0cd290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56508/0 (socket says 172.21.15.1:56508) 2024-03-18T20:26:19.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.484+0000 7fda0affd700 1 -- 172.21.15.1:0/448206150 learned_addr learned my addr 172.21.15.1:0/448206150 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:19.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.484+0000 7fda0affd700 1 -- 172.21.15.1:0/448206150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fda0c1520e0 msgr2=0x7fda0c151920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:19.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.484+0000 7fda0affd700 1 --2- 172.21.15.1:0/448206150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fda0c1520e0 0x7fda0c151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.484+0000 7fda0affd700 1 -- 172.21.15.1:0/448206150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda0c15f320 msgr2=0x7fda0c0ade60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.484+0000 7fda0affd700 1 --2- 172.21.15.1:0/448206150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda0c15f320 0x7fda0c0ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.484+0000 7fda0affd700 1 -- 172.21.15.1:0/448206150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0c15e840 con 0x7fd9f4004850 2024-03-18T20:26:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.484+0000 7fda0affd700 1 --2- 172.21.15.1:0/448206150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f4004850 0x7fda0c0cd290 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7fda00011aa0 tx=0x7fda00011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.484+0000 7fd9f37fe700 1 -- 172.21.15.1:0/448206150 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0000f900 con 0x7fd9f4004850 2024-03-18T20:26:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.484+0000 7fd9f37fe700 1 -- 172.21.15.1:0/448206150 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda0000fa60 con 0x7fd9f4004850 2024-03-18T20:26:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.484+0000 7fda11375700 1 -- 172.21.15.1:0/448206150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0c15eb20 con 0x7fd9f4004850 2024-03-18T20:26:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.484+0000 7fd9f37fe700 1 -- 172.21.15.1:0/448206150 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0000fe10 con 0x7fd9f4004850 2024-03-18T20:26:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.484+0000 7fda11375700 1 -- 172.21.15.1:0/448206150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda0c151e60 con 0x7fd9f4004850 2024-03-18T20:26:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.485+0000 7fda11375700 1 -- 172.21.15.1:0/448206150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda0c15f040 con 0x7fd9f4004850 2024-03-18T20:26:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.486+0000 7fd9f37fe700 1 -- 172.21.15.1:0/448206150 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fda00014440 con 0x7fd9f4004850 2024-03-18T20:26:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.486+0000 7fd9f37fe700 1 --2- 172.21.15.1:0/448206150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd9f806aed0 0x7fd9f806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:19.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.486+0000 7fd9f37fe700 1 -- 172.21.15.1:0/448206150 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6180+0+0 (secure 0 0 0) 0x7fda00089fa0 con 0x7fd9f4004850 2024-03-18T20:26:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.486+0000 7fda0a7fc700 1 --2- 172.21.15.1:0/448206150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd9f806aed0 0x7fd9f806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.487+0000 7fda0a7fc700 1 --2- 172.21.15.1:0/448206150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd9f806aed0 0x7fd9f806d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc001930 tx=0x7fd9fc00a000 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:19.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.491+0000 7fd9f37fe700 1 -- 172.21.15.1:0/448206150 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fda000bd580 con 0x7fd9f4004850 2024-03-18T20:26:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.691+0000 7fda11375700 1 -- 172.21.15.1:0/448206150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fda0c0676a0 con 0x7fd9f4004850 2024-03-18T20:26:19.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.695+0000 7fd9f37fe700 1 -- 172.21.15.1:0/448206150 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v82)=0 v82) v1 ==== 122+0+0 (secure 0 0 0) 0x7fda0c15f040 con 0x7fd9f4004850 2024-03-18T20:26:19.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.697+0000 7fda11375700 1 -- 172.21.15.1:0/448206150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd9f806aed0 msgr2=0x7fd9f806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.697+0000 7fda11375700 1 --2- 172.21.15.1:0/448206150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd9f806aed0 0x7fd9f806d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc001930 tx=0x7fd9fc00a000 comp rx=0 tx=0).stop 2024-03-18T20:26:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.697+0000 7fda11375700 1 -- 172.21.15.1:0/448206150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f4004850 msgr2=0x7fda0c0cd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.697+0000 7fda11375700 1 --2- 172.21.15.1:0/448206150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f4004850 0x7fda0c0cd290 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7fda00011aa0 tx=0x7fda00011e60 comp rx=0 tx=0).stop 2024-03-18T20:26:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.697+0000 7fda11375700 1 -- 172.21.15.1:0/448206150 shutdown_connections 2024-03-18T20:26:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.697+0000 7fda11375700 1 --2- 172.21.15.1:0/448206150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd9f806aed0 0x7fd9f806d390 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.697+0000 7fda11375700 1 --2- 172.21.15.1:0/448206150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda0c15f320 0x7fda0c0ade60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.697+0000 7fda11375700 1 --2- 172.21.15.1:0/448206150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f4004850 0x7fda0c0cd290 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.697+0000 7fda11375700 1 --2- 172.21.15.1:0/448206150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fda0c1520e0 0x7fda0c151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.697+0000 7fda11375700 1 -- 172.21.15.1:0/448206150 >> 172.21.15.1:0/448206150 conn(0x7fda0c0b2c50 msgr2=0x7fda0c0b1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.697+0000 7fda11375700 1 -- 172.21.15.1:0/448206150 shutdown_connections 2024-03-18T20:26:19.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:19.697+0000 7fda11375700 1 -- 172.21.15.1:0/448206150 wait complete. 2024-03-18T20:26:19.715 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T20:26:19.715 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:26:19.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:26:19.802 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T20:26:19.802 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:26:19.802 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:26:19.821 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:26:19.821 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:26:19.821 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:26:19.842 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:26:19.842 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:26:19.842 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:26:19.880 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:26:19.880 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:26:19.880 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:26:19.903 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:26:19.903 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-18T20:26:19.903 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:26:20.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.040+0000 7fa4d68ea700 1 Processor -- start 2024-03-18T20:26:20.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.040+0000 7fa4d68ea700 1 -- start start 2024-03-18T20:26:20.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.040+0000 7fa4d68ea700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa4d01465a0 0x7fa4d0168660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:20.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4d68ea700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d0072660 con 0x7fa4d0146190 2024-03-18T20:26:20.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4d68ea700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d006ebd0 con 0x7fa4d006ae60 2024-03-18T20:26:20.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4d68ea700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4d006f850 con 0x7fa4d01465a0 2024-03-18T20:26:20.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4cffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa4d0146190 0x7fa4d006a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42354/0 (socket says 172.21.15.1:42354) 2024-03-18T20:26:20.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4cffff700 1 -- 172.21.15.1:0/1471509195 learned_addr learned my addr 172.21.15.1:0/1471509195 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:20.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4d4e87700 1 --2- 172.21.15.1:0/1471509195 >> v2:172.21.15.165:3300/0 conn(0x7fa4d01465a0 0x7fa4d0168660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:20.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4ceffd700 1 -- 172.21.15.1:0/1471509195 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324272974 0 0) 0x7fa4d0072660 con 0x7fa4d0146190 2024-03-18T20:26:20.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4ceffd700 1 -- 172.21.15.1:0/1471509195 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4b80035b0 con 0x7fa4d0146190 2024-03-18T20:26:20.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4ceffd700 1 -- 172.21.15.1:0/1471509195 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1787150532 0 0) 0x7fa4b80035b0 con 0x7fa4d0146190 2024-03-18T20:26:20.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4ceffd700 1 -- 172.21.15.1:0/1471509195 >> v2:172.21.15.165:3300/0 conn(0x7fa4d01465a0 msgr2=0x7fa4d0168660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:20.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4ceffd700 1 --2- 172.21.15.1:0/1471509195 >> v2:172.21.15.165:3300/0 conn(0x7fa4d01465a0 0x7fa4d0168660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:20.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4ceffd700 1 -- 172.21.15.1:0/1471509195 >> v1:172.21.15.165:6789/0 conn(0x7fa4d006ae60 legacy=0x7fa4d0164f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:20.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4ceffd700 1 -- 172.21.15.1:0/1471509195 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d006eeb0 con 0x7fa4d0146190 2024-03-18T20:26:20.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4ceffd700 1 -- 172.21.15.1:0/1471509195 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fa4c0001670 con 0x7fa4d0146190 2024-03-18T20:26:20.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4ceffd700 1 -- 172.21.15.1:0/1471509195 >> v1:172.21.15.1:6789/0 conn(0x7fa4d0146190 legacy=0x7fa4d006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:20.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4ceffd700 1 --2- 172.21.15.1:0/1471509195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b8003a00 0x7fa4b8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:20.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4ceffd700 1 --2- 172.21.15.1:0/1471509195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4b8004850 0x7fa4b8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:20.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4ceffd700 1 --2- 172.21.15.1:0/1471509195 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4b8005210 0x7fa4b800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:20.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4ceffd700 1 -- 172.21.15.1:0/1471509195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d006f850 con 0x7fa4b8003a00 2024-03-18T20:26:20.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4ceffd700 1 -- 172.21.15.1:0/1471509195 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b800c0b0 con 0x7fa4b8004850 2024-03-18T20:26:20.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4ceffd700 1 -- 172.21.15.1:0/1471509195 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b800c270 con 0x7fa4b8005210 2024-03-18T20:26:20.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4cffff700 1 --2- 172.21.15.1:0/1471509195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b8003a00 0x7fa4b8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:20.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.041+0000 7fa4cf7fe700 1 --2- 172.21.15.1:0/1471509195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4b8004850 0x7fa4b8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:20.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.042+0000 7fa4cffff700 1 -- 172.21.15.1:0/1471509195 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4b8005210 msgr2=0x7fa4b800ba10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T20:26:20.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.042+0000 7fa4cffff700 1 --2- 172.21.15.1:0/1471509195 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4b8005210 0x7fa4b800ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:20.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.042+0000 7fa4cffff700 1 -- 172.21.15.1:0/1471509195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4b8004850 msgr2=0x7fa4b8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:20.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.042+0000 7fa4cffff700 1 --2- 172.21.15.1:0/1471509195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4b8004850 0x7fa4b8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:20.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.042+0000 7fa4cffff700 1 -- 172.21.15.1:0/1471509195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b80035b0 con 0x7fa4b8003a00 2024-03-18T20:26:20.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.042+0000 7fa4d4e87700 1 -- 172.21.15.1:0/1471509195 reap_dead start 2024-03-18T20:26:20.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.042+0000 7fa4cffff700 1 --2- 172.21.15.1:0/1471509195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b8003a00 0x7fa4b8003e20 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0001050 tx=0x7fa4b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d727198f3387ddc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:20.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.042+0000 7fa4ceffd700 1 -- 172.21.15.1:0/1471509195 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c000cd40 con 0x7fa4b8003a00 2024-03-18T20:26:20.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.042+0000 7fa4ceffd700 1 -- 172.21.15.1:0/1471509195 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c0005960 con 0x7fa4b8003a00 2024-03-18T20:26:20.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.042+0000 7fa4ceffd700 1 -- 172.21.15.1:0/1471509195 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c000cea0 con 0x7fa4b8003a00 2024-03-18T20:26:20.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.042+0000 7fa4d68ea700 1 -- 172.21.15.1:0/1471509195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b8003a00 msgr2=0x7fa4b8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:20.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.042+0000 7fa4d68ea700 1 --2- 172.21.15.1:0/1471509195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b8003a00 0x7fa4b8003e20 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0001050 tx=0x7fa4b8005880 comp rx=0 tx=0).stop 2024-03-18T20:26:20.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.042+0000 7fa4d68ea700 1 -- 172.21.15.1:0/1471509195 shutdown_connections 2024-03-18T20:26:20.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.042+0000 7fa4d68ea700 1 --2- 172.21.15.1:0/1471509195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b8003a00 0x7fa4b8003e20 unknown :-1 s=CLOSED pgs=2767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:20.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.042+0000 7fa4d68ea700 1 -- 172.21.15.1:0/1471509195 >> 172.21.15.1:0/1471509195 conn(0x7fa4d00b1d20 msgr2=0x7fa4d00b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:20.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.042+0000 7fa4d68ea700 1 -- 172.21.15.1:0/1471509195 shutdown_connections 2024-03-18T20:26:20.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.043+0000 7fa4d68ea700 1 -- 172.21.15.1:0/1471509195 wait complete. 2024-03-18T20:26:20.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.044+0000 7fa4d68ea700 1 Processor -- start 2024-03-18T20:26:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.044+0000 7fa4d68ea700 1 -- start start 2024-03-18T20:26:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4d68ea700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b8003a00 0x7fa4d013fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4d68ea700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4d015d020 0x7fa4d0141d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4d68ea700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d00cdcf0 0x7fa4d00ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4d68ea700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d013f550 con 0x7fa4b8003a00 2024-03-18T20:26:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4d68ea700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d01f3330 con 0x7fa4d015d020 2024-03-18T20:26:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4d68ea700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d013f7f0 con 0x7fa4d00cdcf0 2024-03-18T20:26:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4cffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b8003a00 0x7fa4d013fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4cffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b8003a00 0x7fa4d013fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56514/0 (socket says 172.21.15.1:56514) 2024-03-18T20:26:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4cffff700 1 -- 172.21.15.1:0/2291041102 learned_addr learned my addr 172.21.15.1:0/2291041102 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4cf7fe700 1 --2- 172.21.15.1:0/2291041102 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4d015d020 0x7fa4d0141d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4d4e87700 1 --2- 172.21.15.1:0/2291041102 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d00cdcf0 0x7fa4d00ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:20.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4cffff700 1 -- 172.21.15.1:0/2291041102 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d00cdcf0 msgr2=0x7fa4d00ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:20.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4cffff700 1 --2- 172.21.15.1:0/2291041102 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d00cdcf0 0x7fa4d00ce1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:20.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4cffff700 1 -- 172.21.15.1:0/2291041102 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4d015d020 msgr2=0x7fa4d0141d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:20.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4cffff700 1 --2- 172.21.15.1:0/2291041102 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4d015d020 0x7fa4d0141d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:20.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4cffff700 1 -- 172.21.15.1:0/2291041102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d01f35d0 con 0x7fa4b8003a00 2024-03-18T20:26:20.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4cf7fe700 1 --2- 172.21.15.1:0/2291041102 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4d015d020 0x7fa4d0141d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:20.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4cffff700 1 --2- 172.21.15.1:0/2291041102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b8003a00 0x7fa4d013fec0 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0001930 tx=0x7fa4c0000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:20.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4ccff9700 1 -- 172.21.15.1:0/2291041102 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0005ce0 con 0x7fa4b8003a00 2024-03-18T20:26:20.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4ccff9700 1 -- 172.21.15.1:0/2291041102 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c0005e80 con 0x7fa4b8003a00 2024-03-18T20:26:20.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4d68ea700 1 -- 172.21.15.1:0/2291041102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d013fad0 con 0x7fa4b8003a00 2024-03-18T20:26:20.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4ccff9700 1 -- 172.21.15.1:0/2291041102 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0007310 con 0x7fa4b8003a00 2024-03-18T20:26:20.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.045+0000 7fa4d68ea700 1 -- 172.21.15.1:0/2291041102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d0142260 con 0x7fa4b8003a00 2024-03-18T20:26:20.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.046+0000 7fa4d68ea700 1 -- 172.21.15.1:0/2291041102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d0142450 con 0x7fa4b8003a00 2024-03-18T20:26:20.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.047+0000 7fa4ccff9700 1 -- 172.21.15.1:0/2291041102 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fa4c0007470 con 0x7fa4b8003a00 2024-03-18T20:26:20.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.047+0000 7fa4ccff9700 1 --2- 172.21.15.1:0/2291041102 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa4bc06aed0 0x7fa4bc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:20.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.048+0000 7fa4cf7fe700 1 --2- 172.21.15.1:0/2291041102 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa4bc06aed0 0x7fa4bc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:20.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.048+0000 7fa4ccff9700 1 -- 172.21.15.1:0/2291041102 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 6180+0+0 (secure 0 0 0) 0x7fa4c008ae50 con 0x7fa4b8003a00 2024-03-18T20:26:20.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.048+0000 7fa4cf7fe700 1 --2- 172.21.15.1:0/2291041102 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa4bc06aed0 0x7fa4bc06d390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa4d00b4500 tx=0x7fa4c4009040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:20.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.052+0000 7fa4ccff9700 1 -- 172.21.15.1:0/2291041102 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa4c0055c00 con 0x7fa4b8003a00 2024-03-18T20:26:20.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.299+0000 7fa4d68ea700 1 -- 172.21.15.1:0/2291041102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4d01f3760 con 0x7fa4b8003a00 2024-03-18T20:26:20.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.300+0000 7fa4ccff9700 1 -- 172.21.15.1:0/2291041102 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+3532 (secure 0 0 0) 0x7fa4c0017090 con 0x7fa4b8003a00 2024-03-18T20:26:20.305 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:20.305 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":114,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10369,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/677258734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":677258734},{"type":"v1","addr":"172.21.15.165:6835","nonce":677258734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":10858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/658650553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":658650553},{"type":"v1","addr":"172.21.15.165:6834","nonce":658650553}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":112},{"gid":11544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2018398457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2018398457},{"type":"v1","addr":"172.21.15.1:6836","nonce":2018398457}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":100},{"gid":12327,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/1267741800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1267741800},{"type":"v1","addr":"172.21.15.1:6837","nonce":1267741800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114}],"filesystems":[]} 2024-03-18T20:26:20.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.302+0000 7fa4d68ea700 1 -- 172.21.15.1:0/2291041102 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa4bc06aed0 msgr2=0x7fa4bc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:20.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.302+0000 7fa4d68ea700 1 --2- 172.21.15.1:0/2291041102 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa4bc06aed0 0x7fa4bc06d390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa4d00b4500 tx=0x7fa4c4009040 comp rx=0 tx=0).stop 2024-03-18T20:26:20.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.302+0000 7fa4d68ea700 1 -- 172.21.15.1:0/2291041102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b8003a00 msgr2=0x7fa4d013fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:20.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.302+0000 7fa4d68ea700 1 --2- 172.21.15.1:0/2291041102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b8003a00 0x7fa4d013fec0 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0001930 tx=0x7fa4c0000fe0 comp rx=0 tx=0).stop 2024-03-18T20:26:20.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.302+0000 7fa4d68ea700 1 -- 172.21.15.1:0/2291041102 shutdown_connections 2024-03-18T20:26:20.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.302+0000 7fa4d68ea700 1 --2- 172.21.15.1:0/2291041102 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa4bc06aed0 0x7fa4bc06d390 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:20.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.302+0000 7fa4d68ea700 1 --2- 172.21.15.1:0/2291041102 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d00cdcf0 0x7fa4d00ce1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:20.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.302+0000 7fa4d68ea700 1 --2- 172.21.15.1:0/2291041102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4b8003a00 0x7fa4d013fec0 unknown :-1 s=CLOSED pgs=2768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:20.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.302+0000 7fa4d68ea700 1 --2- 172.21.15.1:0/2291041102 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4d015d020 0x7fa4d0141d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:20.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.302+0000 7fa4d68ea700 1 -- 172.21.15.1:0/2291041102 >> 172.21.15.1:0/2291041102 conn(0x7fa4d00b1d20 msgr2=0x7fa4d00ac990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:20.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.302+0000 7fa4d68ea700 1 -- 172.21.15.1:0/2291041102 shutdown_connections 2024-03-18T20:26:20.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.302+0000 7fa4d68ea700 1 -- 172.21.15.1:0/2291041102 wait complete. 2024-03-18T20:26:20.308 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 114 2024-03-18T20:26:20.319 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T20:26:20.319 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-18T20:26:20.320 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T20:26:20.320 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:26:20.320 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T20:26:20.320 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-18T20:26:20.320 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-18T20:26:20.320 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:26:20.320 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T20:26:20.320 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-18T20:26:20.321 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-18T20:26:20.321 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:26:20.321 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T20:26:20.321 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-18T20:26:20.321 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-18T20:26:20.321 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:26:20.329 INFO:tasks.ceph.mds.a:Stopped 2024-03-18T20:26:20.329 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-18T20:26:20.330 INFO:tasks.ceph.mds.c:Stopped 2024-03-18T20:26:20.330 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-18T20:26:20.332 INFO:tasks.ceph.mds.d:Stopped 2024-03-18T20:26:20.332 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-18T20:26:20.333 INFO:tasks.ceph.mds.b:Stopped 2024-03-18T20:26:20.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-18T20:26:20.371 INFO:tasks.ceph.mds.a:Started 2024-03-18T20:26:20.372 DEBUG:teuthology.parallel:result is None 2024-03-18T20:26:20.372 INFO:tasks.ceph.mds.c:Started 2024-03-18T20:26:20.372 DEBUG:teuthology.parallel:result is None 2024-03-18T20:26:20.375 INFO:tasks.ceph.mds.d:Started 2024-03-18T20:26:20.375 DEBUG:teuthology.parallel:result is None 2024-03-18T20:26:20.376 INFO:tasks.ceph.mds.b:Started 2024-03-18T20:26:20.376 DEBUG:teuthology.parallel:result is None 2024-03-18T20:26:20.376 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-18T20:26:20.453 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-18T20:26:20.453 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T20:26:20.463 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-03-18T20:26:20.463 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-03-18T20:26:20.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.505+0000 7efcf6833700 1 Processor -- start 2024-03-18T20:26:20.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.505+0000 7efcf6833700 1 -- start start 2024-03-18T20:26:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.506+0000 7efcf6833700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7efcf015ee70 0x7efcf015f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.506+0000 7efcf6833700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcf006f740 0x7efcf009a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:20.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.506+0000 7efcf6833700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf00cf5e0 con 0x7efcf0070410 2024-03-18T20:26:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.506+0000 7efcf6833700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcf015fe20 con 0x7efcf015ee70 2024-03-18T20:26:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.506+0000 7efcf6833700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf015ebc0 con 0x7efcf006f740 2024-03-18T20:26:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.506+0000 7efcef7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efcf0070410 0x7efcf0099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42356/0 (socket says 172.21.15.1:42356) 2024-03-18T20:26:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.506+0000 7efcef7fe700 1 -- 172.21.15.1:0/1428384559 learned_addr learned my addr 172.21.15.1:0/1428384559 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.507+0000 7efceffff700 1 --2- 172.21.15.1:0/1428384559 >> v2:172.21.15.165:3300/0 conn(0x7efcf015ee70 0x7efcf015f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.507+0000 7efcf4dd0700 1 --2- 172.21.15.1:0/1428384559 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcf006f740 0x7efcf009a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.507+0000 7efceeffd700 1 -- 172.21.15.1:0/1428384559 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107766563 0 0) 0x7efcf00cf5e0 con 0x7efcf0070410 2024-03-18T20:26:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.507+0000 7efceeffd700 1 -- 172.21.15.1:0/1428384559 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcd80035b0 con 0x7efcf0070410 2024-03-18T20:26:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.507+0000 7efceffff700 1 -- 172.21.15.1:0/1428384559 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcf006f740 msgr2=0x7efcf009a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:20.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.507+0000 7efceffff700 1 --2- 172.21.15.1:0/1428384559 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcf006f740 0x7efcf009a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.507+0000 7efceffff700 1 -- 172.21.15.1:0/1428384559 >> v1:172.21.15.1:6789/0 conn(0x7efcf0070410 legacy=0x7efcf0099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.507+0000 7efceeffd700 1 -- 172.21.15.1:0/1428384559 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3965264446 0 0) 0x7efcd80035b0 con 0x7efcf0070410 2024-03-18T20:26:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.507+0000 7efceffff700 1 -- 172.21.15.1:0/1428384559 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf015fb40 con 0x7efcf015ee70 2024-03-18T20:26:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.507+0000 7efcf4dd0700 1 --2- 172.21.15.1:0/1428384559 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcf006f740 0x7efcf009a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:26:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.507+0000 7efceffff700 1 --2- 172.21.15.1:0/1428384559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcf015ee70 0x7efcf015f250 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7efce0009840 tx=0x7efce0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=133ace7826a76c10 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.507+0000 7efceeffd700 1 -- 172.21.15.1:0/1428384559 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce4001670 con 0x7efcf015ee70 2024-03-18T20:26:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.507+0000 7efceeffd700 1 -- 172.21.15.1:0/1428384559 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce0004030 con 0x7efcf015ee70 2024-03-18T20:26:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.507+0000 7efceeffd700 1 -- 172.21.15.1:0/1428384559 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce00043c0 con 0x7efcf015ee70 2024-03-18T20:26:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.508+0000 7efcf6833700 1 -- 172.21.15.1:0/1428384559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcf015ee70 msgr2=0x7efcf015f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.508+0000 7efcf6833700 1 --2- 172.21.15.1:0/1428384559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcf015ee70 0x7efcf015f250 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7efce0009840 tx=0x7efce0009b50 comp rx=0 tx=0).stop 2024-03-18T20:26:20.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.508+0000 7efcf6833700 1 -- 172.21.15.1:0/1428384559 shutdown_connections 2024-03-18T20:26:20.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.508+0000 7efcf6833700 1 --2- 172.21.15.1:0/1428384559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcf015ee70 0x7efcf015f250 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:20.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.508+0000 7efcf6833700 1 --2- 172.21.15.1:0/1428384559 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcf006f740 0x7efcf009a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:20.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.508+0000 7efcf6833700 1 -- 172.21.15.1:0/1428384559 >> 172.21.15.1:0/1428384559 conn(0x7efcf00b2af0 msgr2=0x7efcf00b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:20.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.508+0000 7efcf6833700 1 -- 172.21.15.1:0/1428384559 shutdown_connections 2024-03-18T20:26:20.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.508+0000 7efcf6833700 1 -- 172.21.15.1:0/1428384559 wait complete. 2024-03-18T20:26:20.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.509+0000 7efcf6833700 1 Processor -- start 2024-03-18T20:26:20.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.509+0000 7efcf6833700 1 -- start start 2024-03-18T20:26:20.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.510+0000 7efcf6833700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcf006f740 0x7efcf00ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:20.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.510+0000 7efcf6833700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcf0070410 0x7efcf0153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:20.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.510+0000 7efcf6833700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcf015ee70 0x7efcf00c80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:20.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.510+0000 7efcf6833700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf00ca350 con 0x7efcf015ee70 2024-03-18T20:26:20.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.510+0000 7efcf6833700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf00cacb0 con 0x7efcf006f740 2024-03-18T20:26:20.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.510+0000 7efcf6833700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf01548f0 con 0x7efcf0070410 2024-03-18T20:26:20.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.510+0000 7efcf4dd0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcf015ee70 0x7efcf00c80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:20.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.510+0000 7efcf4dd0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcf015ee70 0x7efcf00c80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56558/0 (socket says 172.21.15.1:56558) 2024-03-18T20:26:20.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.510+0000 7efcf4dd0700 1 -- 172.21.15.1:0/260183863 learned_addr learned my addr 172.21.15.1:0/260183863 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:20.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.510+0000 7efcef7fe700 1 --2- 172.21.15.1:0/260183863 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcf0070410 0x7efcf0153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:20.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.510+0000 7efceffff700 1 --2- 172.21.15.1:0/260183863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcf006f740 0x7efcf00ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:20.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.510+0000 7efcf4dd0700 1 -- 172.21.15.1:0/260183863 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcf0070410 msgr2=0x7efcf0153e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:20.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.510+0000 7efcf4dd0700 1 --2- 172.21.15.1:0/260183863 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcf0070410 0x7efcf0153e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:20.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.510+0000 7efcf4dd0700 1 -- 172.21.15.1:0/260183863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcf006f740 msgr2=0x7efcf00ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:20.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.510+0000 7efcf4dd0700 1 --2- 172.21.15.1:0/260183863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcf006f740 0x7efcf00ca770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:20.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.510+0000 7efcf4dd0700 1 -- 172.21.15.1:0/260183863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce8009a70 con 0x7efcf015ee70 2024-03-18T20:26:20.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.511+0000 7efcf4dd0700 1 --2- 172.21.15.1:0/260183863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcf015ee70 0x7efcf00c80d0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7efce800efc0 tx=0x7efce800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:20.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.511+0000 7efcecff9700 1 -- 172.21.15.1:0/260183863 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce800ca90 con 0x7efcf015ee70 2024-03-18T20:26:20.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.511+0000 7efcecff9700 1 -- 172.21.15.1:0/260183863 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce800f040 con 0x7efcf015ee70 2024-03-18T20:26:20.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.511+0000 7efcf6833700 1 -- 172.21.15.1:0/260183863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce00094a0 con 0x7efcf015ee70 2024-03-18T20:26:20.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.511+0000 7efcecff9700 1 -- 172.21.15.1:0/260183863 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce800ce30 con 0x7efcf015ee70 2024-03-18T20:26:20.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.511+0000 7efcf6833700 1 -- 172.21.15.1:0/260183863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf015f910 con 0x7efcf015ee70 2024-03-18T20:26:20.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.512+0000 7efcf6833700 1 -- 172.21.15.1:0/260183863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf015f740 con 0x7efcf015ee70 2024-03-18T20:26:20.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.512+0000 7efcecff9700 1 -- 172.21.15.1:0/260183863 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7efce8014480 con 0x7efcf015ee70 2024-03-18T20:26:20.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.512+0000 7efcecff9700 1 --2- 172.21.15.1:0/260183863 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efcdc06aed0 0x7efcdc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:20.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.513+0000 7efcecff9700 1 -- 172.21.15.1:0/260183863 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 6180+0+0 (secure 0 0 0) 0x7efce8089f50 con 0x7efcf015ee70 2024-03-18T20:26:20.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.513+0000 7efceffff700 1 --2- 172.21.15.1:0/260183863 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efcdc06aed0 0x7efcdc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:20.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.513+0000 7efceffff700 1 --2- 172.21.15.1:0/260183863 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efcdc06aed0 0x7efcdc06d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efce00090b0 tx=0x7efce0008030 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:20.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.517+0000 7efcecff9700 1 -- 172.21.15.1:0/260183863 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efce8054c40 con 0x7efcf015ee70 2024-03-18T20:26:20.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:20.732+0000 7efcf6833700 1 -- 172.21.15.1:0/260183863 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7efcf00c9730 con 0x7efcf015ee70 2024-03-18T20:26:21.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.698+0000 7efcecff9700 1 -- 172.21.15.1:0/260183863 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v249) v1 ==== 97+0+0 (secure 0 0 0) 0x7efce8058260 con 0x7efcf015ee70 2024-03-18T20:26:21.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.701+0000 7efcf6833700 1 -- 172.21.15.1:0/260183863 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efcdc06aed0 msgr2=0x7efcdc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:21.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.701+0000 7efcf6833700 1 --2- 172.21.15.1:0/260183863 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efcdc06aed0 0x7efcdc06d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efce00090b0 tx=0x7efce0008030 comp rx=0 tx=0).stop 2024-03-18T20:26:21.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.701+0000 7efcf6833700 1 -- 172.21.15.1:0/260183863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcf015ee70 msgr2=0x7efcf00c80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:21.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.701+0000 7efcf6833700 1 --2- 172.21.15.1:0/260183863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcf015ee70 0x7efcf00c80d0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7efce800efc0 tx=0x7efce800c3a0 comp rx=0 tx=0).stop 2024-03-18T20:26:21.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.701+0000 7efcf6833700 1 -- 172.21.15.1:0/260183863 shutdown_connections 2024-03-18T20:26:21.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.701+0000 7efcf6833700 1 --2- 172.21.15.1:0/260183863 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efcdc06aed0 0x7efcdc06d390 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:21.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.701+0000 7efcf6833700 1 --2- 172.21.15.1:0/260183863 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efcf015ee70 0x7efcf00c80d0 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:21.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.701+0000 7efcf6833700 1 --2- 172.21.15.1:0/260183863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcf006f740 0x7efcf00ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:21.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.701+0000 7efcf6833700 1 --2- 172.21.15.1:0/260183863 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efcf0070410 0x7efcf0153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:21.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.701+0000 7efcf6833700 1 -- 172.21.15.1:0/260183863 >> 172.21.15.1:0/260183863 conn(0x7efcf00b2af0 msgr2=0x7efcf00af5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:21.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.701+0000 7efcf6833700 1 -- 172.21.15.1:0/260183863 shutdown_connections 2024-03-18T20:26:21.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.701+0000 7efcf6833700 1 -- 172.21.15.1:0/260183863 wait complete. 2024-03-18T20:26:21.708 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-18T20:26:21.722 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-18T20:26:21.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.848+0000 7f94faf75700 1 Processor -- start 2024-03-18T20:26:21.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.848+0000 7f94faf75700 1 -- start start 2024-03-18T20:26:21.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.849+0000 7f94faf75700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94f415d580 0x7f94f4072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:21.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.849+0000 7f94faf75700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94f406e290 0x7f94f40cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:21.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.849+0000 7f94faf75700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f406f490 con 0x7f94f406d640 2024-03-18T20:26:21.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.849+0000 7f94faf75700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f4071cf0 con 0x7f94f406e290 2024-03-18T20:26:21.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.849+0000 7f94faf75700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f406f1b0 con 0x7f94f415d580 2024-03-18T20:26:21.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.849+0000 7f94f3fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94f406e290 0x7f94f40cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:21.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.849+0000 7f94f3fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f94f406e290 0x7f94f40cd4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56574/0 (socket says 172.21.15.1:56574) 2024-03-18T20:26:21.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.849+0000 7f94f3fff700 1 -- 172.21.15.1:0/1393000096 learned_addr learned my addr 172.21.15.1:0/1393000096 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:21.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.849+0000 7f94f8d11700 1 --2- 172.21.15.1:0/1393000096 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94f415d580 0x7f94f4072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:21.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.849+0000 7f94f3fff700 1 -- 172.21.15.1:0/1393000096 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94f415d580 msgr2=0x7f94f4072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:21.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.849+0000 7f94f3fff700 1 --2- 172.21.15.1:0/1393000096 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94f415d580 0x7f94f4072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:21.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.849+0000 7f94f3fff700 1 -- 172.21.15.1:0/1393000096 >> v1:172.21.15.165:6789/0 conn(0x7f94f406d640 legacy=0x7f94f40ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:21.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.849+0000 7f94f3fff700 1 -- 172.21.15.1:0/1393000096 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f40716a0 con 0x7f94f406e290 2024-03-18T20:26:21.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.850+0000 7f94f3fff700 1 --2- 172.21.15.1:0/1393000096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94f406e290 0x7f94f40cd4b0 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f94e00098f0 tx=0x7f94e0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb9d8be25a22249c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:21.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.850+0000 7f94f37fe700 1 -- 172.21.15.1:0/1393000096 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e0004230 con 0x7f94f406e290 2024-03-18T20:26:21.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.850+0000 7f94f37fe700 1 -- 172.21.15.1:0/1393000096 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e0004390 con 0x7f94f406e290 2024-03-18T20:26:21.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.850+0000 7f94f37fe700 1 -- 172.21.15.1:0/1393000096 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e00036a0 con 0x7f94f406e290 2024-03-18T20:26:21.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.850+0000 7f94faf75700 1 -- 172.21.15.1:0/1393000096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94f406e290 msgr2=0x7f94f40cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:21.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.850+0000 7f94faf75700 1 --2- 172.21.15.1:0/1393000096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94f406e290 0x7f94f40cd4b0 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f94e00098f0 tx=0x7f94e0009c00 comp rx=0 tx=0).stop 2024-03-18T20:26:21.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.850+0000 7f94faf75700 1 -- 172.21.15.1:0/1393000096 shutdown_connections 2024-03-18T20:26:21.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.850+0000 7f94faf75700 1 --2- 172.21.15.1:0/1393000096 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94f415d580 0x7f94f4072180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:21.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.850+0000 7f94faf75700 1 --2- 172.21.15.1:0/1393000096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94f406e290 0x7f94f40cd4b0 unknown :-1 s=CLOSED pgs=2778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:21.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.850+0000 7f94faf75700 1 -- 172.21.15.1:0/1393000096 >> 172.21.15.1:0/1393000096 conn(0x7f94f40cc990 msgr2=0x7f94f40ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:21.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.850+0000 7f94faf75700 1 -- 172.21.15.1:0/1393000096 shutdown_connections 2024-03-18T20:26:21.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.850+0000 7f94faf75700 1 -- 172.21.15.1:0/1393000096 wait complete. 2024-03-18T20:26:21.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.852+0000 7f94faf75700 1 Processor -- start 2024-03-18T20:26:21.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.852+0000 7f94faf75700 1 -- start start 2024-03-18T20:26:21.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.852+0000 7f94faf75700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94f406d640 0x7f94f41f0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:21.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.852+0000 7f94faf75700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94f415d580 0x7f94f415e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:21.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.852+0000 7f94faf75700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94f415eab0 0x7f94f415f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:21.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.852+0000 7f94faf75700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f40b4970 con 0x7f94f415eab0 2024-03-18T20:26:21.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.853+0000 7f94faf75700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f40b4690 con 0x7f94f406d640 2024-03-18T20:26:21.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.853+0000 7f94faf75700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f415da90 con 0x7f94f415d580 2024-03-18T20:26:21.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.853+0000 7f94f9512700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94f415eab0 0x7f94f415f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:21.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.853+0000 7f94f9512700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94f415eab0 0x7f94f415f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56578/0 (socket says 172.21.15.1:56578) 2024-03-18T20:26:21.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.853+0000 7f94f9512700 1 -- 172.21.15.1:0/1717426427 learned_addr learned my addr 172.21.15.1:0/1717426427 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:21.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.853+0000 7f94f3fff700 1 --2- 172.21.15.1:0/1717426427 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94f415d580 0x7f94f415e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:21.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.853+0000 7f94f8d11700 1 --2- 172.21.15.1:0/1717426427 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94f406d640 0x7f94f41f0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:21.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.853+0000 7f94f9512700 1 -- 172.21.15.1:0/1717426427 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94f415d580 msgr2=0x7f94f415e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:21.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.853+0000 7f94f9512700 1 --2- 172.21.15.1:0/1717426427 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94f415d580 0x7f94f415e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:21.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.853+0000 7f94f9512700 1 -- 172.21.15.1:0/1717426427 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94f406d640 msgr2=0x7f94f41f0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:21.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.853+0000 7f94f9512700 1 --2- 172.21.15.1:0/1717426427 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94f406d640 0x7f94f41f0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:21.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.853+0000 7f94f9512700 1 -- 172.21.15.1:0/1717426427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e0009550 con 0x7f94f415eab0 2024-03-18T20:26:21.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.853+0000 7f94f9512700 1 --2- 172.21.15.1:0/1717426427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94f415eab0 0x7f94f415f1d0 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00cc10 tx=0x7f94ec00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:21.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.853+0000 7f94f17fa700 1 -- 172.21.15.1:0/1717426427 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ec00aa60 con 0x7f94f415eab0 2024-03-18T20:26:21.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.853+0000 7f94f17fa700 1 -- 172.21.15.1:0/1717426427 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94ec00abc0 con 0x7f94f415eab0 2024-03-18T20:26:21.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.853+0000 7f94faf75700 1 -- 172.21.15.1:0/1717426427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f415dd70 con 0x7f94f415eab0 2024-03-18T20:26:21.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.853+0000 7f94f17fa700 1 -- 172.21.15.1:0/1717426427 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ec012420 con 0x7f94f415eab0 2024-03-18T20:26:21.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.853+0000 7f94faf75700 1 -- 172.21.15.1:0/1717426427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f41f0660 con 0x7f94f415eab0 2024-03-18T20:26:21.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.854+0000 7f94faf75700 1 -- 172.21.15.1:0/1717426427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f40676a0 con 0x7f94f415eab0 2024-03-18T20:26:21.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.855+0000 7f94f17fa700 1 -- 172.21.15.1:0/1717426427 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f94ec012580 con 0x7f94f415eab0 2024-03-18T20:26:21.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.855+0000 7f94f17fa700 1 --2- 172.21.15.1:0/1717426427 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f94d806aed0 0x7f94d806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:21.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.855+0000 7f94f17fa700 1 -- 172.21.15.1:0/1717426427 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f94ec0897f0 con 0x7f94f415eab0 2024-03-18T20:26:21.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.855+0000 7f94f8d11700 1 --2- 172.21.15.1:0/1717426427 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f94d806aed0 0x7f94d806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:21.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.856+0000 7f94f8d11700 1 --2- 172.21.15.1:0/1717426427 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f94d806aed0 0x7f94d806d390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f94e8006b20 tx=0x7f94e8008040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:21.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:21.860+0000 7f94f17fa700 1 -- 172.21.15.1:0/1717426427 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f94ec0548b0 con 0x7f94f415eab0 2024-03-18T20:26:22.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.144+0000 7f94faf75700 1 -- 172.21.15.1:0/1717426427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f94f406b290 con 0x7f94f415eab0 2024-03-18T20:26:22.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.144+0000 7f94f17fa700 1 -- 172.21.15.1:0/1717426427 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v51) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f94ec057ed0 con 0x7f94f415eab0 2024-03-18T20:26:22.149 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:22.149 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:26:22.150 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-18T20:26:22.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:22.150 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-18T20:26:22.150 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlR93/CxAAbpe+5n3wHRur9xi8QzdCnQ==", 2024-03-18T20:26:22.150 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:26:22.150 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:26:22.150 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:26:22.150 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:26:22.150 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:26:22.150 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:22.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:22.151 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-18T20:26:22.151 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJ3OmGBAA37u227nxrMsh/JHt71HBxQ==", 2024-03-18T20:26:22.151 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:26:22.151 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:26:22.151 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:26:22.151 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:26:22.151 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:26:22.151 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:22.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:22.152 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-18T20:26:22.152 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJZFoExAAmI16oxwNKbf8dKetPfyk+g==", 2024-03-18T20:26:22.152 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:26:22.152 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:26:22.152 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:26:22.152 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:26:22.152 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:26:22.152 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:22.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:22.152 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-18T20:26:22.153 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlQrcvIBAA1LMv+EgC7S6wspX6QF49Pg==", 2024-03-18T20:26:22.153 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:26:22.153 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:26:22.153 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:26:22.153 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:26:22.153 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:26:22.153 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:22.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:22.153 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-18T20:26:22.153 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCUjPhlYOVHChAARFQrksZmMIdTlntd8TgvrQ==", 2024-03-18T20:26:22.153 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:26:22.154 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:26:22.154 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:26:22.154 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:26:22.154 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:22.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:22.154 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-18T20:26:22.154 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCXjPhl2zSDCBAA6Al1gkWVB79ZI9djGpVChA==", 2024-03-18T20:26:22.154 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:26:22.154 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:26:22.154 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:26:22.155 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:26:22.155 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:22.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:22.155 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-18T20:26:22.155 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCajPhlSoG/BhAAR93USDe6DCQW+hREOsnTlw==", 2024-03-18T20:26:22.155 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:26:22.155 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:26:22.156 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:26:22.156 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:26:22.156 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:22.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:22.156 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-18T20:26:22.156 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCdjPhlhJg5BRAAWuQhcnxyjR5UOiM13yKdAQ==", 2024-03-18T20:26:22.156 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:26:22.156 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:26:22.156 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:26:22.156 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:26:22.157 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:22.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:22.157 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-18T20:26:22.157 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCjjPhlCFnpCRAAKIyOQyVdjXZUTnAS6Nw/ng==", 2024-03-18T20:26:22.157 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:26:22.157 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:26:22.158 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:26:22.158 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:26:22.158 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:22.158 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:22.158 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-18T20:26:22.158 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCmjPhlAeFkCBAAJcEFpRk/03lQJxj+l2hEPA==", 2024-03-18T20:26:22.158 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:26:22.158 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:26:22.158 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:26:22.158 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:26:22.158 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:22.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:22.159 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-18T20:26:22.159 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCpjPhl/xuuBhAA0LgKfXEvN2O11ZlRQJVOkw==", 2024-03-18T20:26:22.159 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:26:22.159 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:26:22.159 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:26:22.159 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:26:22.159 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.159 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:22.159 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:22.159 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-18T20:26:22.160 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCsjPhldiR4BRAA571oh6U8PzOv/4LzjYLF/A==", 2024-03-18T20:26:22.160 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:26:22.160 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:26:22.160 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:26:22.160 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:26:22.160 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:22.160 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:22.160 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-18T20:26:22.160 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlu+2xJBAANnXGGjfeqP6S2FTecsfOsg==", 2024-03-18T20:26:22.160 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:26:22.160 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:26:22.160 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-18T20:26:22.161 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-18T20:26:22.161 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:22.161 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:22.161 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-18T20:26:22.161 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlhJ9+KxAAEnrQrCrgJR6xI+pucwzAXA==", 2024-03-18T20:26:22.161 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:26:22.161 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:26:22.161 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:26:22.161 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:26:22.161 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:26:22.161 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:22.162 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:22.162 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.2", 2024-03-18T20:26:22.162 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlxBUpKBAAIGTwmLWyj+P2Yvl55QVcWA==", 2024-03-18T20:26:22.162 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:26:22.162 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:26:22.162 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:26:22.162 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:26:22.163 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:26:22.163 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:22.163 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:22.163 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.3", 2024-03-18T20:26:22.163 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlmao3MRAAHeiQ6eyBh9QjjKPCtxkC8g==", 2024-03-18T20:26:22.163 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:26:22.163 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:26:22.163 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:26:22.163 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:26:22.163 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:26:22.163 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:22.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:22.164 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-18T20:26:22.164 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCRjPhlemOuKRAAVUggmtB7+EiVgyfJZ8TCCA==", 2024-03-18T20:26:22.164 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:26:22.164 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:26:22.164 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:26:22.164 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:26:22.164 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:26:22.164 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:22.164 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:22.164 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-18T20:26:22.164 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhleIuOCBAA9CH+AHewQbn+gpVr+6lJyw==", 2024-03-18T20:26:22.165 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:26:22.165 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:26:22.165 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T20:26:22.165 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:26:22.165 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:22.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:22.165 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-18T20:26:22.165 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlL0pTBBAAdFiPcdwwz514cuLAvioxYA==", 2024-03-18T20:26:22.165 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:26:22.165 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:26:22.165 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T20:26:22.165 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:26:22.166 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.166 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:22.166 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-18T20:26:22.166 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:26:22.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.146+0000 7f94faf75700 1 -- 172.21.15.1:0/1717426427 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f94d806aed0 msgr2=0x7f94d806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:22.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.146+0000 7f94faf75700 1 --2- 172.21.15.1:0/1717426427 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f94d806aed0 0x7f94d806d390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f94e8006b20 tx=0x7f94e8008040 comp rx=0 tx=0).stop 2024-03-18T20:26:22.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.146+0000 7f94faf75700 1 -- 172.21.15.1:0/1717426427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94f415eab0 msgr2=0x7f94f415f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:22.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.146+0000 7f94faf75700 1 --2- 172.21.15.1:0/1717426427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94f415eab0 0x7f94f415f1d0 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00cc10 tx=0x7f94ec00cf20 comp rx=0 tx=0).stop 2024-03-18T20:26:22.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.146+0000 7f94faf75700 1 -- 172.21.15.1:0/1717426427 shutdown_connections 2024-03-18T20:26:22.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.146+0000 7f94faf75700 1 --2- 172.21.15.1:0/1717426427 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f94d806aed0 0x7f94d806d390 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.146+0000 7f94faf75700 1 --2- 172.21.15.1:0/1717426427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94f415eab0 0x7f94f415f1d0 unknown :-1 s=CLOSED pgs=2779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.146+0000 7f94faf75700 1 --2- 172.21.15.1:0/1717426427 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94f406d640 0x7f94f41f0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.146+0000 7f94faf75700 1 --2- 172.21.15.1:0/1717426427 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f94f415d580 0x7f94f415e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.146+0000 7f94faf75700 1 -- 172.21.15.1:0/1717426427 >> 172.21.15.1:0/1717426427 conn(0x7f94f40cc990 msgr2=0x7f94f406c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:22.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.146+0000 7f94faf75700 1 -- 172.21.15.1:0/1717426427 shutdown_connections 2024-03-18T20:26:22.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.146+0000 7f94faf75700 1 -- 172.21.15.1:0/1717426427 wait complete. 2024-03-18T20:26:22.169 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:26:22.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.288+0000 7f276ee83700 1 Processor -- start 2024-03-18T20:26:22.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.289+0000 7f276ee83700 1 -- start start 2024-03-18T20:26:22.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.289+0000 7f276ee83700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f276815d580 0x7f2768072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:22.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.289+0000 7f276ee83700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f276806dd50 0x7f27680cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:22.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.289+0000 7f276ee83700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f276806f490 con 0x7f276806e290 2024-03-18T20:26:22.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.289+0000 7f276ee83700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2768071cf0 con 0x7f276815d580 2024-03-18T20:26:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.289+0000 7f276ee83700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f276806f1b0 con 0x7f276806dd50 2024-03-18T20:26:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.290+0000 7f276cc1f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f276815d580 0x7f2768072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.290+0000 7f276d420700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f276806dd50 0x7f27680cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.290+0000 7f2767fff700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f276806e290 0x7f276806d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:58584/0 (socket says 172.21.15.1:58584) 2024-03-18T20:26:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.290+0000 7f2767fff700 1 -- 172.21.15.1:0/44378200 learned_addr learned my addr 172.21.15.1:0/44378200 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.290+0000 7f276cc1f700 1 --2- 172.21.15.1:0/44378200 >> v2:172.21.15.165:3300/0 conn(0x7f276815d580 0x7f2768072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:37772/0 (socket says 172.21.15.1:37772) 2024-03-18T20:26:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.290+0000 7f27677fe700 1 -- 172.21.15.1:0/44378200 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686542176 0 0) 0x7f276806f490 con 0x7f276806e290 2024-03-18T20:26:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.290+0000 7f27677fe700 1 -- 172.21.15.1:0/44378200 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27580035b0 con 0x7f276806e290 2024-03-18T20:26:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.290+0000 7f276cc1f700 1 -- 172.21.15.1:0/44378200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f276806dd50 msgr2=0x7f27680cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.290+0000 7f276cc1f700 1 --2- 172.21.15.1:0/44378200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f276806dd50 0x7f27680cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.290+0000 7f276cc1f700 1 -- 172.21.15.1:0/44378200 >> v1:172.21.15.165:6789/0 conn(0x7f276806e290 legacy=0x7f276806d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.290+0000 7f276cc1f700 1 -- 172.21.15.1:0/44378200 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27680716a0 con 0x7f276815d580 2024-03-18T20:26:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.290+0000 7f276d420700 1 --2- 172.21.15.1:0/44378200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f276806dd50 0x7f27680cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:26:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.290+0000 7f276cc1f700 1 --2- 172.21.15.1:0/44378200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f276815d580 0x7f2768072180 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f275c009840 tx=0x7f275c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1652de97dbdc787d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.291+0000 7f27677fe700 1 -- 172.21.15.1:0/44378200 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275c011830 con 0x7f276815d580 2024-03-18T20:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.291+0000 7f27677fe700 1 -- 172.21.15.1:0/44378200 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f275c011990 con 0x7f276815d580 2024-03-18T20:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.291+0000 7f27677fe700 1 -- 172.21.15.1:0/44378200 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275c011d00 con 0x7f276815d580 2024-03-18T20:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.291+0000 7f276ee83700 1 -- 172.21.15.1:0/44378200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f276815d580 msgr2=0x7f2768072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.291+0000 7f276ee83700 1 --2- 172.21.15.1:0/44378200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f276815d580 0x7f2768072180 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f275c009840 tx=0x7f275c009b50 comp rx=0 tx=0).stop 2024-03-18T20:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.291+0000 7f276ee83700 1 -- 172.21.15.1:0/44378200 shutdown_connections 2024-03-18T20:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.291+0000 7f276ee83700 1 --2- 172.21.15.1:0/44378200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f276806dd50 0x7f27680cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.291+0000 7f276ee83700 1 --2- 172.21.15.1:0/44378200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f276815d580 0x7f2768072180 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.291+0000 7f276ee83700 1 -- 172.21.15.1:0/44378200 >> 172.21.15.1:0/44378200 conn(0x7f27680cc990 msgr2=0x7f27680ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.291+0000 7f276ee83700 1 -- 172.21.15.1:0/44378200 shutdown_connections 2024-03-18T20:26:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.291+0000 7f276ee83700 1 -- 172.21.15.1:0/44378200 wait complete. 2024-03-18T20:26:22.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.292+0000 7f276ee83700 1 Processor -- start 2024-03-18T20:26:22.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.292+0000 7f276ee83700 1 -- start start 2024-03-18T20:26:22.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.293+0000 7f276ee83700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f276806dd50 0x7f2768158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:22.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.293+0000 7f276ee83700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f276815d580 0x7f276815f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:22.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.293+0000 7f276ee83700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27681590f0 0x7f276815d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:22.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.293+0000 7f276ee83700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2768158e50 con 0x7f27681590f0 2024-03-18T20:26:22.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.293+0000 7f276ee83700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276815e240 con 0x7f276806dd50 2024-03-18T20:26:22.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.293+0000 7f276ee83700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f276815e520 con 0x7f276815d580 2024-03-18T20:26:22.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.293+0000 7f276d420700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27681590f0 0x7f276815d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:22.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.293+0000 7f276d420700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27681590f0 0x7f276815d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56594/0 (socket says 172.21.15.1:56594) 2024-03-18T20:26:22.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.293+0000 7f276d420700 1 -- 172.21.15.1:0/1188379659 learned_addr learned my addr 172.21.15.1:0/1188379659 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:22.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.294+0000 7f276cc1f700 1 --2- 172.21.15.1:0/1188379659 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f276806dd50 0x7f2768158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:22.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.294+0000 7f2767fff700 1 --2- 172.21.15.1:0/1188379659 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f276815d580 0x7f276815f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:22.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.294+0000 7f276d420700 1 -- 172.21.15.1:0/1188379659 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f276815d580 msgr2=0x7f276815f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:22.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.294+0000 7f276d420700 1 --2- 172.21.15.1:0/1188379659 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f276815d580 0x7f276815f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.294+0000 7f276d420700 1 -- 172.21.15.1:0/1188379659 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f276806dd50 msgr2=0x7f2768158910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.294+0000 7f276d420700 1 --2- 172.21.15.1:0/1188379659 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f276806dd50 0x7f2768158910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.294+0000 7f276d420700 1 -- 172.21.15.1:0/1188379659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2760009a70 con 0x7f27681590f0 2024-03-18T20:26:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.294+0000 7f276d420700 1 --2- 172.21.15.1:0/1188379659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27681590f0 0x7f276815d960 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f276000efc0 tx=0x7f276000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.294+0000 7f27657fa700 1 -- 172.21.15.1:0/1188379659 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276000cc00 con 0x7f27681590f0 2024-03-18T20:26:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.294+0000 7f27657fa700 1 -- 172.21.15.1:0/1188379659 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276000f040 con 0x7f27681590f0 2024-03-18T20:26:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.294+0000 7f276ee83700 1 -- 172.21.15.1:0/1188379659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f275c0094a0 con 0x7f27681590f0 2024-03-18T20:26:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.294+0000 7f27657fa700 1 -- 172.21.15.1:0/1188379659 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2760003a20 con 0x7f27681590f0 2024-03-18T20:26:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.294+0000 7f276ee83700 1 -- 172.21.15.1:0/1188379659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f276815eae0 con 0x7f27681590f0 2024-03-18T20:26:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.295+0000 7f276ee83700 1 -- 172.21.15.1:0/1188379659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27680676a0 con 0x7f27681590f0 2024-03-18T20:26:22.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.296+0000 7f27657fa700 1 -- 172.21.15.1:0/1188379659 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f2760014480 con 0x7f27681590f0 2024-03-18T20:26:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.296+0000 7f27657fa700 1 --2- 172.21.15.1:0/1188379659 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f274c06af00 0x7f274c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.296+0000 7f27657fa700 1 -- 172.21.15.1:0/1188379659 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f276008a100 con 0x7f27681590f0 2024-03-18T20:26:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.296+0000 7f276cc1f700 1 --2- 172.21.15.1:0/1188379659 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f274c06af00 0x7f274c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:22.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.296+0000 7f276cc1f700 1 --2- 172.21.15.1:0/1188379659 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f274c06af00 0x7f274c06d3c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f275c009fd0 tx=0x7f275c01c040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:22.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.301+0000 7f27657fa700 1 -- 172.21.15.1:0/1188379659 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2760054f90 con 0x7f27681590f0 2024-03-18T20:26:22.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.503+0000 7f276ee83700 1 -- 172.21.15.1:0/1188379659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f276806da50 con 0x7f27681590f0 2024-03-18T20:26:22.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.504+0000 7f27657fa700 1 -- 172.21.15.1:0/1188379659 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v249) v1 ==== 74+0+12717 (secure 0 0 0) 0x7f27600585b0 con 0x7f27681590f0 2024-03-18T20:26:22.509 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:22.509 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":249,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:26:21.695898+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:26:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.506+0000 7f276ee83700 1 -- 172.21.15.1:0/1188379659 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f274c06af00 msgr2=0x7f274c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:22.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.506+0000 7f276ee83700 1 --2- 172.21.15.1:0/1188379659 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f274c06af00 0x7f274c06d3c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f275c009fd0 tx=0x7f275c01c040 comp rx=0 tx=0).stop 2024-03-18T20:26:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.506+0000 7f276ee83700 1 -- 172.21.15.1:0/1188379659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27681590f0 msgr2=0x7f276815d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.506+0000 7f276ee83700 1 --2- 172.21.15.1:0/1188379659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27681590f0 0x7f276815d960 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f276000efc0 tx=0x7f276000c3a0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.506+0000 7f276ee83700 1 -- 172.21.15.1:0/1188379659 shutdown_connections 2024-03-18T20:26:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.506+0000 7f276ee83700 1 --2- 172.21.15.1:0/1188379659 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f274c06af00 0x7f274c06d3c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.506+0000 7f276ee83700 1 --2- 172.21.15.1:0/1188379659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27681590f0 0x7f276815d960 unknown :-1 s=CLOSED pgs=2780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.506+0000 7f276ee83700 1 --2- 172.21.15.1:0/1188379659 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f276806dd50 0x7f2768158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.506+0000 7f276ee83700 1 --2- 172.21.15.1:0/1188379659 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f276815d580 0x7f276815f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.506+0000 7f276ee83700 1 -- 172.21.15.1:0/1188379659 >> 172.21.15.1:0/1188379659 conn(0x7f27680cc990 msgr2=0x7f27680cba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.506+0000 7f276ee83700 1 -- 172.21.15.1:0/1188379659 shutdown_connections 2024-03-18T20:26:22.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.506+0000 7f276ee83700 1 -- 172.21.15.1:0/1188379659 wait complete. 2024-03-18T20:26:22.524 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:26:22.524 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:26:22.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.647+0000 7f9c34fa1700 1 Processor -- start 2024-03-18T20:26:22.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.648+0000 7f9c34fa1700 1 -- start start 2024-03-18T20:26:22.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.648+0000 7f9c34fa1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9c300b2010 0x7f9c300c5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:22.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.648+0000 7f9c34fa1700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c300cc620 con 0x7f9c3015b500 2024-03-18T20:26:22.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.648+0000 7f9c34fa1700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c300b2580 con 0x7f9c300cfc40 2024-03-18T20:26:22.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.648+0000 7f9c34fa1700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c300cc8c0 con 0x7f9c300b2010 2024-03-18T20:26:22.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.648+0000 7f9c2f7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9c3015b500 0x7f9c300cf530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42368/0 (socket says 172.21.15.1:42368) 2024-03-18T20:26:22.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.648+0000 7f9c2f7fe700 1 -- 172.21.15.1:0/1385830626 learned_addr learned my addr 172.21.15.1:0/1385830626 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:22.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2ffff700 1 --2- 172.21.15.1:0/1385830626 >> v2:172.21.15.165:3300/0 conn(0x7f9c300b2010 0x7f9c300c5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:22.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2e7fc700 1 -- 172.21.15.1:0/1385830626 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839515865 0 0) 0x7f9c300cc620 con 0x7f9c3015b500 2024-03-18T20:26:22.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2e7fc700 1 -- 172.21.15.1:0/1385830626 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c1c0035b0 con 0x7f9c3015b500 2024-03-18T20:26:22.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2e7fc700 1 -- 172.21.15.1:0/1385830626 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2264187588 0 0) 0x7f9c1c0035b0 con 0x7f9c3015b500 2024-03-18T20:26:22.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2e7fc700 1 -- 172.21.15.1:0/1385830626 >> v2:172.21.15.165:3300/0 conn(0x7f9c300b2010 msgr2=0x7f9c300c5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:22.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2e7fc700 1 --2- 172.21.15.1:0/1385830626 >> v2:172.21.15.165:3300/0 conn(0x7f9c300b2010 0x7f9c300c5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2e7fc700 1 -- 172.21.15.1:0/1385830626 >> v1:172.21.15.165:6789/0 conn(0x7f9c300cfc40 legacy=0x7f9c300c4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:22.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2e7fc700 1 -- 172.21.15.1:0/1385830626 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c300ccba0 con 0x7f9c3015b500 2024-03-18T20:26:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2e7fc700 1 -- 172.21.15.1:0/1385830626 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f9c18001670 con 0x7f9c3015b500 2024-03-18T20:26:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2e7fc700 1 -- 172.21.15.1:0/1385830626 >> v1:172.21.15.1:6789/0 conn(0x7f9c3015b500 legacy=0x7f9c300cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2e7fc700 1 --2- 172.21.15.1:0/1385830626 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c1c003a70 0x7f9c1c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2e7fc700 1 --2- 172.21.15.1:0/1385830626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c004850 0x7f9c1c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2e7fc700 1 --2- 172.21.15.1:0/1385830626 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c1c005210 0x7f9c1c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2e7fc700 1 -- 172.21.15.1:0/1385830626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c300cc8c0 con 0x7f9c1c004850 2024-03-18T20:26:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2e7fc700 1 -- 172.21.15.1:0/1385830626 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c00c090 con 0x7f9c1c003a70 2024-03-18T20:26:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2e7fc700 1 -- 172.21.15.1:0/1385830626 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c00c250 con 0x7f9c1c005210 2024-03-18T20:26:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2effd700 1 --2- 172.21.15.1:0/1385830626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c004850 0x7f9c1c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2f7fe700 1 --2- 172.21.15.1:0/1385830626 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c1c003a70 0x7f9c1c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2ffff700 1 --2- 172.21.15.1:0/1385830626 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c1c005210 0x7f9c1c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2effd700 1 -- 172.21.15.1:0/1385830626 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c1c005210 msgr2=0x7f9c1c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:22.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2effd700 1 --2- 172.21.15.1:0/1385830626 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c1c005210 0x7f9c1c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.649+0000 7f9c2effd700 1 -- 172.21.15.1:0/1385830626 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c1c003a70 msgr2=0x7f9c1c003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:22.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.650+0000 7f9c2effd700 1 --2- 172.21.15.1:0/1385830626 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c1c003a70 0x7f9c1c003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.650+0000 7f9c2effd700 1 -- 172.21.15.1:0/1385830626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1c0035b0 con 0x7f9c1c004850 2024-03-18T20:26:22.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.650+0000 7f9c2ffff700 1 --2- 172.21.15.1:0/1385830626 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c1c005210 0x7f9c1c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:22.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.650+0000 7f9c2ffff700 1 -- 172.21.15.1:0/1385830626 reap_dead start 2024-03-18T20:26:22.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.650+0000 7f9c2effd700 1 --2- 172.21.15.1:0/1385830626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c004850 0x7f9c1c004cd0 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7f9c20001980 tx=0x7f9c1c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c386082e1b14a3e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:22.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.650+0000 7f9c2e7fc700 1 -- 172.21.15.1:0/1385830626 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c20001420 con 0x7f9c1c004850 2024-03-18T20:26:22.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.650+0000 7f9c2e7fc700 1 -- 172.21.15.1:0/1385830626 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c200052f0 con 0x7f9c1c004850 2024-03-18T20:26:22.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.650+0000 7f9c2e7fc700 1 -- 172.21.15.1:0/1385830626 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c20003680 con 0x7f9c1c004850 2024-03-18T20:26:22.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.650+0000 7f9c34fa1700 1 -- 172.21.15.1:0/1385830626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c004850 msgr2=0x7f9c1c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:22.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.650+0000 7f9c34fa1700 1 --2- 172.21.15.1:0/1385830626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c004850 0x7f9c1c004cd0 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7f9c20001980 tx=0x7f9c1c005880 comp rx=0 tx=0).stop 2024-03-18T20:26:22.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.650+0000 7f9c34fa1700 1 -- 172.21.15.1:0/1385830626 shutdown_connections 2024-03-18T20:26:22.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.650+0000 7f9c34fa1700 1 --2- 172.21.15.1:0/1385830626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c004850 0x7f9c1c004cd0 unknown :-1 s=CLOSED pgs=2782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.650+0000 7f9c34fa1700 1 -- 172.21.15.1:0/1385830626 >> 172.21.15.1:0/1385830626 conn(0x7f9c300d0f50 msgr2=0x7f9c300d1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:22.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.650+0000 7f9c34fa1700 1 -- 172.21.15.1:0/1385830626 shutdown_connections 2024-03-18T20:26:22.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.650+0000 7f9c34fa1700 1 -- 172.21.15.1:0/1385830626 wait complete. 2024-03-18T20:26:22.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.652+0000 7f9c34fa1700 1 Processor -- start 2024-03-18T20:26:22.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.652+0000 7f9c34fa1700 1 -- start start 2024-03-18T20:26:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.652+0000 7f9c34fa1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c004850 0x7f9c301f43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.652+0000 7f9c34fa1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c3009e150 0x7f9c3009e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.652+0000 7f9c34fa1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c3015d6d0 0x7f9c3015db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:22.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.653+0000 7f9c34fa1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c301f27e0 con 0x7f9c1c004850 2024-03-18T20:26:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.653+0000 7f9c34fa1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3009eb10 con 0x7f9c3015d6d0 2024-03-18T20:26:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.653+0000 7f9c34fa1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3015e5a0 con 0x7f9c3009e150 2024-03-18T20:26:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.653+0000 7f9c2f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c004850 0x7f9c301f43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.653+0000 7f9c2f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c004850 0x7f9c301f43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56622/0 (socket says 172.21.15.1:56622) 2024-03-18T20:26:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.653+0000 7f9c2f7fe700 1 -- 172.21.15.1:0/3005744010 learned_addr learned my addr 172.21.15.1:0/3005744010 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.653+0000 7f9c2effd700 1 --2- 172.21.15.1:0/3005744010 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c3009e150 0x7f9c3009e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.653+0000 7f9c2ffff700 1 --2- 172.21.15.1:0/3005744010 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c3015d6d0 0x7f9c3015db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.653+0000 7f9c2f7fe700 1 -- 172.21.15.1:0/3005744010 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c3009e150 msgr2=0x7f9c3009e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.653+0000 7f9c2f7fe700 1 --2- 172.21.15.1:0/3005744010 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c3009e150 0x7f9c3009e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.653+0000 7f9c2f7fe700 1 -- 172.21.15.1:0/3005744010 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c3015d6d0 msgr2=0x7f9c3015db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.653+0000 7f9c2f7fe700 1 --2- 172.21.15.1:0/3005744010 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c3015d6d0 0x7f9c3015db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.653+0000 7f9c2f7fe700 1 -- 172.21.15.1:0/3005744010 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c301f1e40 con 0x7f9c1c004850 2024-03-18T20:26:22.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.653+0000 7f9c2f7fe700 1 --2- 172.21.15.1:0/3005744010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c004850 0x7f9c301f43d0 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7f9c180072c0 tx=0x7f9c180075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:22.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.653+0000 7f9c27fff700 1 -- 172.21.15.1:0/3005744010 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c180077b0 con 0x7f9c1c004850 2024-03-18T20:26:22.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.653+0000 7f9c27fff700 1 -- 172.21.15.1:0/3005744010 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1800f870 con 0x7f9c1c004850 2024-03-18T20:26:22.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.653+0000 7f9c34fa1700 1 -- 172.21.15.1:0/3005744010 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c301f2120 con 0x7f9c1c004850 2024-03-18T20:26:22.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.653+0000 7f9c27fff700 1 -- 172.21.15.1:0/3005744010 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1800fc20 con 0x7f9c1c004850 2024-03-18T20:26:22.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.653+0000 7f9c34fa1700 1 -- 172.21.15.1:0/3005744010 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c301f4910 con 0x7f9c1c004850 2024-03-18T20:26:22.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.654+0000 7f9c34fa1700 1 -- 172.21.15.1:0/3005744010 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c301f4ad0 con 0x7f9c1c004850 2024-03-18T20:26:22.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.655+0000 7f9c27fff700 1 -- 172.21.15.1:0/3005744010 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f9c1800fd80 con 0x7f9c1c004850 2024-03-18T20:26:22.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.655+0000 7f9c27fff700 1 --2- 172.21.15.1:0/3005744010 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9c1006aed0 0x7f9c1006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:22.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.655+0000 7f9c27fff700 1 -- 172.21.15.1:0/3005744010 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f9c18088fe0 con 0x7f9c1c004850 2024-03-18T20:26:22.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.655+0000 7f9c2effd700 1 --2- 172.21.15.1:0/3005744010 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9c1006aed0 0x7f9c1006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:22.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.656+0000 7f9c2effd700 1 --2- 172.21.15.1:0/3005744010 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9c1006aed0 0x7f9c1006d390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c300cac60 tx=0x7f9c2000a000 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:22.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.660+0000 7f9c27fff700 1 -- 172.21.15.1:0/3005744010 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9c18054010 con 0x7f9c1c004850 2024-03-18T20:26:22.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.860+0000 7f9c34fa1700 1 -- 172.21.15.1:0/3005744010 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c300c6fe0 con 0x7f9c1c004850 2024-03-18T20:26:22.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.860+0000 7f9c27fff700 1 -- 172.21.15.1:0/3005744010 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v249) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9c1801a0a0 con 0x7f9c1c004850 2024-03-18T20:26:22.865 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:26:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.862+0000 7f9c34fa1700 1 -- 172.21.15.1:0/3005744010 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9c1006aed0 msgr2=0x7f9c1006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.862+0000 7f9c34fa1700 1 --2- 172.21.15.1:0/3005744010 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9c1006aed0 0x7f9c1006d390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c300cac60 tx=0x7f9c2000a000 comp rx=0 tx=0).stop 2024-03-18T20:26:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.862+0000 7f9c34fa1700 1 -- 172.21.15.1:0/3005744010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c004850 msgr2=0x7f9c301f43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.862+0000 7f9c34fa1700 1 --2- 172.21.15.1:0/3005744010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c004850 0x7f9c301f43d0 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7f9c180072c0 tx=0x7f9c180075d0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.862+0000 7f9c34fa1700 1 -- 172.21.15.1:0/3005744010 shutdown_connections 2024-03-18T20:26:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.862+0000 7f9c34fa1700 1 --2- 172.21.15.1:0/3005744010 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9c1006aed0 0x7f9c1006d390 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.862+0000 7f9c34fa1700 1 --2- 172.21.15.1:0/3005744010 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c3015d6d0 0x7f9c3015db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.862+0000 7f9c34fa1700 1 --2- 172.21.15.1:0/3005744010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c1c004850 0x7f9c301f43d0 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.862+0000 7f9c34fa1700 1 --2- 172.21.15.1:0/3005744010 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9c3009e150 0x7f9c3009e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:22.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.862+0000 7f9c34fa1700 1 -- 172.21.15.1:0/3005744010 >> 172.21.15.1:0/3005744010 conn(0x7f9c300d0f50 msgr2=0x7f9c3006feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:22.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.863+0000 7f9c34fa1700 1 -- 172.21.15.1:0/3005744010 shutdown_connections 2024-03-18T20:26:22.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:22.863+0000 7f9c34fa1700 1 -- 172.21.15.1:0/3005744010 wait complete. 2024-03-18T20:26:22.880 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T20:26:23.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.000+0000 7f8d01f6d700 1 Processor -- start 2024-03-18T20:26:23.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.000+0000 7f8d01f6d700 1 -- start start 2024-03-18T20:26:23.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.001+0000 7f8d01f6d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8cfc15d620 0x7f8cfc070d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:23.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.001+0000 7f8d01f6d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cfc06e8f0 0x7f8cfc071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:23.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.001+0000 7f8d01f6d700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cfc0cd5a0 con 0x7f8cfc0717b0 2024-03-18T20:26:23.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.001+0000 7f8d01f6d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cfc15da00 con 0x7f8cfc15d620 2024-03-18T20:26:23.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.001+0000 7f8d01f6d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cfc071bd0 con 0x7f8cfc06e8f0 2024-03-18T20:26:23.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.001+0000 7f8cfb7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8cfc15d620 0x7f8cfc070d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:23.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.001+0000 7f8cfb7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8cfc15d620 0x7f8cfc070d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56632/0 (socket says 172.21.15.1:56632) 2024-03-18T20:26:23.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.001+0000 7f8cfb7fe700 1 -- 172.21.15.1:0/4006270502 learned_addr learned my addr 172.21.15.1:0/4006270502 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:23.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.001+0000 7f8cfaffd700 1 --2- 172.21.15.1:0/4006270502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cfc06e8f0 0x7f8cfc071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:23.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.001+0000 7f8cfb7fe700 1 -- 172.21.15.1:0/4006270502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cfc06e8f0 msgr2=0x7f8cfc071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:23.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.001+0000 7f8cfb7fe700 1 --2- 172.21.15.1:0/4006270502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cfc06e8f0 0x7f8cfc071270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:23.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.001+0000 7f8cfb7fe700 1 -- 172.21.15.1:0/4006270502 >> v1:172.21.15.165:6789/0 conn(0x7f8cfc0717b0 legacy=0x7f8cfc1610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:23.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.001+0000 7f8cfb7fe700 1 -- 172.21.15.1:0/4006270502 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cfc15e5f0 con 0x7f8cfc15d620 2024-03-18T20:26:23.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.001+0000 7f8cfaffd700 1 --2- 172.21.15.1:0/4006270502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cfc06e8f0 0x7f8cfc071270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:23.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.001+0000 7f8cfb7fe700 1 --2- 172.21.15.1:0/4006270502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cfc15d620 0x7f8cfc070d30 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7f8ce40098f0 tx=0x7f8ce4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443282088f253c82 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:23.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.002+0000 7f8cfa7fc700 1 -- 172.21.15.1:0/4006270502 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce4004230 con 0x7f8cfc15d620 2024-03-18T20:26:23.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.002+0000 7f8cfa7fc700 1 -- 172.21.15.1:0/4006270502 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce4004390 con 0x7f8cfc15d620 2024-03-18T20:26:23.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.002+0000 7f8cfa7fc700 1 -- 172.21.15.1:0/4006270502 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce40036a0 con 0x7f8cfc15d620 2024-03-18T20:26:23.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.002+0000 7f8d01f6d700 1 -- 172.21.15.1:0/4006270502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cfc15d620 msgr2=0x7f8cfc070d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:23.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.002+0000 7f8d01f6d700 1 --2- 172.21.15.1:0/4006270502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cfc15d620 0x7f8cfc070d30 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7f8ce40098f0 tx=0x7f8ce4009c00 comp rx=0 tx=0).stop 2024-03-18T20:26:23.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.002+0000 7f8d01f6d700 1 -- 172.21.15.1:0/4006270502 shutdown_connections 2024-03-18T20:26:23.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.002+0000 7f8d01f6d700 1 --2- 172.21.15.1:0/4006270502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cfc15d620 0x7f8cfc070d30 unknown :-1 s=CLOSED pgs=2784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:23.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.002+0000 7f8d01f6d700 1 --2- 172.21.15.1:0/4006270502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cfc06e8f0 0x7f8cfc071270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:23.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.002+0000 7f8d01f6d700 1 -- 172.21.15.1:0/4006270502 >> 172.21.15.1:0/4006270502 conn(0x7f8cfc0b1270 msgr2=0x7f8cfc0b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:23.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.002+0000 7f8d01f6d700 1 -- 172.21.15.1:0/4006270502 shutdown_connections 2024-03-18T20:26:23.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.002+0000 7f8d01f6d700 1 -- 172.21.15.1:0/4006270502 wait complete. 2024-03-18T20:26:23.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.003+0000 7f8d01f6d700 1 Processor -- start 2024-03-18T20:26:23.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.004+0000 7f8d01f6d700 1 -- start start 2024-03-18T20:26:23.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.004+0000 7f8d01f6d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cfc06e8f0 0x7f8cfc156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:23.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.004+0000 7f8d01f6d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cfc0717b0 0x7f8cfc0c3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:23.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.004+0000 7f8d01f6d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cfc15d620 0x7f8cfc15eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:23.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.004+0000 7f8d01f6d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cfc0c36d0 con 0x7f8cfc15d620 2024-03-18T20:26:23.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.004+0000 7f8d01f6d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cfc160f00 con 0x7f8cfc06e8f0 2024-03-18T20:26:23.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.004+0000 7f8d01f6d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cfc1e9cb0 con 0x7f8cfc0717b0 2024-03-18T20:26:23.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.004+0000 7f8cfbfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cfc15d620 0x7f8cfc15eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:23.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.005+0000 7f8cfbfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cfc15d620 0x7f8cfc15eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56642/0 (socket says 172.21.15.1:56642) 2024-03-18T20:26:23.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.005+0000 7f8cfbfff700 1 -- 172.21.15.1:0/900493446 learned_addr learned my addr 172.21.15.1:0/900493446 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:23.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.005+0000 7f8cfaffd700 1 --2- 172.21.15.1:0/900493446 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cfc0717b0 0x7f8cfc0c3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:23.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.005+0000 7f8cfb7fe700 1 --2- 172.21.15.1:0/900493446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cfc06e8f0 0x7f8cfc156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:23.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.005+0000 7f8cfbfff700 1 -- 172.21.15.1:0/900493446 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cfc0717b0 msgr2=0x7f8cfc0c3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:23.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.005+0000 7f8cfbfff700 1 --2- 172.21.15.1:0/900493446 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cfc0717b0 0x7f8cfc0c3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:23.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.005+0000 7f8cfbfff700 1 -- 172.21.15.1:0/900493446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cfc06e8f0 msgr2=0x7f8cfc156530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:23.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.005+0000 7f8cfbfff700 1 --2- 172.21.15.1:0/900493446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cfc06e8f0 0x7f8cfc156530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:23.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.005+0000 7f8cfbfff700 1 -- 172.21.15.1:0/900493446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce4009550 con 0x7f8cfc15d620 2024-03-18T20:26:23.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.005+0000 7f8cfb7fe700 1 --2- 172.21.15.1:0/900493446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cfc06e8f0 0x7f8cfc156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:23.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.005+0000 7f8cfbfff700 1 --2- 172.21.15.1:0/900493446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cfc15d620 0x7f8cfc15eb10 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400cc10 tx=0x7f8cf400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:23.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.005+0000 7f8cf3fff700 1 -- 172.21.15.1:0/900493446 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf400abb0 con 0x7f8cfc15d620 2024-03-18T20:26:23.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.005+0000 7f8cf3fff700 1 -- 172.21.15.1:0/900493446 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf400ad10 con 0x7f8cfc15d620 2024-03-18T20:26:23.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.005+0000 7f8d01f6d700 1 -- 172.21.15.1:0/900493446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cfc0c3920 con 0x7f8cfc15d620 2024-03-18T20:26:23.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.005+0000 7f8cf3fff700 1 -- 172.21.15.1:0/900493446 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4012660 con 0x7f8cfc15d620 2024-03-18T20:26:23.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.005+0000 7f8d01f6d700 1 -- 172.21.15.1:0/900493446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cfc0c3ee0 con 0x7f8cfc15d620 2024-03-18T20:26:23.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.006+0000 7f8d01f6d700 1 -- 172.21.15.1:0/900493446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cfc0b52f0 con 0x7f8cfc15d620 2024-03-18T20:26:23.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.007+0000 7f8cf3fff700 1 -- 172.21.15.1:0/900493446 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f8cf40127c0 con 0x7f8cfc15d620 2024-03-18T20:26:23.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.007+0000 7f8cf3fff700 1 --2- 172.21.15.1:0/900493446 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8cdc06aed0 0x7f8cdc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:23.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.007+0000 7f8cf3fff700 1 -- 172.21.15.1:0/900493446 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f8cf4089d40 con 0x7f8cfc15d620 2024-03-18T20:26:23.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.007+0000 7f8cfb7fe700 1 --2- 172.21.15.1:0/900493446 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8cdc06aed0 0x7f8cdc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:23.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.008+0000 7f8cfb7fe700 1 --2- 172.21.15.1:0/900493446 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8cdc06aed0 0x7f8cdc06d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4009130 tx=0x7f8ce4008030 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:23.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.012+0000 7f8cf3fff700 1 -- 172.21.15.1:0/900493446 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8cf4054d70 con 0x7f8cfc15d620 2024-03-18T20:26:23.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.220+0000 7f8d01f6d700 1 -- 172.21.15.1:0/900493446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8cfc070d30 con 0x7f8cfc15d620 2024-03-18T20:26:23.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.221+0000 7f8cf3fff700 1 -- 172.21.15.1:0/900493446 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v249) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8cf4016020 con 0x7f8cfc15d620 2024-03-18T20:26:23.226 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:23.226 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T20:26:23.226 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:23.226 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-18T20:26:23.226 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-18T20:26:23.226 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:23.226 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T20:26:23.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.223+0000 7f8d01f6d700 1 -- 172.21.15.1:0/900493446 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8cdc06aed0 msgr2=0x7f8cdc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:23.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.223+0000 7f8d01f6d700 1 --2- 172.21.15.1:0/900493446 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8cdc06aed0 0x7f8cdc06d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4009130 tx=0x7f8ce4008030 comp rx=0 tx=0).stop 2024-03-18T20:26:23.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.223+0000 7f8d01f6d700 1 -- 172.21.15.1:0/900493446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cfc15d620 msgr2=0x7f8cfc15eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:23.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.223+0000 7f8d01f6d700 1 --2- 172.21.15.1:0/900493446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cfc15d620 0x7f8cfc15eb10 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400cc10 tx=0x7f8cf400a060 comp rx=0 tx=0).stop 2024-03-18T20:26:23.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.223+0000 7f8d01f6d700 1 -- 172.21.15.1:0/900493446 shutdown_connections 2024-03-18T20:26:23.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.223+0000 7f8d01f6d700 1 --2- 172.21.15.1:0/900493446 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8cdc06aed0 0x7f8cdc06d390 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:23.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.223+0000 7f8d01f6d700 1 --2- 172.21.15.1:0/900493446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8cfc15d620 0x7f8cfc15eb10 unknown :-1 s=CLOSED pgs=2785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:23.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.223+0000 7f8d01f6d700 1 --2- 172.21.15.1:0/900493446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cfc06e8f0 0x7f8cfc156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:23.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.223+0000 7f8d01f6d700 1 --2- 172.21.15.1:0/900493446 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8cfc0717b0 0x7f8cfc0c3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:23.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.223+0000 7f8d01f6d700 1 -- 172.21.15.1:0/900493446 >> 172.21.15.1:0/900493446 conn(0x7f8cfc0b1270 msgr2=0x7f8cfc0adce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:23.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.223+0000 7f8d01f6d700 1 -- 172.21.15.1:0/900493446 shutdown_connections 2024-03-18T20:26:23.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.223+0000 7f8d01f6d700 1 -- 172.21.15.1:0/900493446 wait complete. 2024-03-18T20:26:23.240 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T20:26:23.240 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T20:26:23.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.365+0000 7f8e052d6700 1 Processor -- start 2024-03-18T20:26:23.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.365+0000 7f8e052d6700 1 -- start start 2024-03-18T20:26:23.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8e052d6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8e0015d8b0 0x7f8e000d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:23.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8e052d6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e000d1170 0x7f8e000cf930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:23.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8e052d6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8e000cfe70 0x7f8e00161250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:23.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8e052d6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e0015ec90 con 0x7f8e000cfe70 2024-03-18T20:26:23.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8e052d6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e000d02d0 con 0x7f8e0015d8b0 2024-03-18T20:26:23.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8e052d6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0015dc90 con 0x7f8e000d1170 2024-03-18T20:26:23.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8dff7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8e000cfe70 0x7f8e00161250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:23.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8dff7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8e000cfe70 0x7f8e00161250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56652/0 (socket says 172.21.15.1:56652) 2024-03-18T20:26:23.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8dff7fe700 1 -- 172.21.15.1:0/1789542452 learned_addr learned my addr 172.21.15.1:0/1789542452 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:23.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8dfe7fc700 1 --2- 172.21.15.1:0/1789542452 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e000d1170 0x7f8e000cf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:23.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8dfeffd700 1 --2- 172.21.15.1:0/1789542452 >> v2:172.21.15.165:3300/0 conn(0x7f8e0015d8b0 0x7f8e000d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:23.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8dff7fe700 1 -- 172.21.15.1:0/1789542452 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e000d1170 msgr2=0x7f8e000cf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:23.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8dff7fe700 1 --2- 172.21.15.1:0/1789542452 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e000d1170 0x7f8e000cf930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:23.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8dff7fe700 1 -- 172.21.15.1:0/1789542452 >> v2:172.21.15.165:3300/0 conn(0x7f8e0015d8b0 msgr2=0x7f8e000d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:23.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8dff7fe700 1 --2- 172.21.15.1:0/1789542452 >> v2:172.21.15.165:3300/0 conn(0x7f8e0015d8b0 0x7f8e000d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:23.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8dff7fe700 1 -- 172.21.15.1:0/1789542452 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0015ded0 con 0x7f8e000cfe70 2024-03-18T20:26:23.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8dfe7fc700 1 --2- 172.21.15.1:0/1789542452 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e000d1170 0x7f8e000cf930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:23.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8dff7fe700 1 --2- 172.21.15.1:0/1789542452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e000cfe70 0x7f8e00161250 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7f8e0008bb60 tx=0x7f8df800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c55164e5b2984a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:23.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8dfdffb700 1 -- 172.21.15.1:0/1789542452 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df800e070 con 0x7f8e000cfe70 2024-03-18T20:26:23.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8dfdffb700 1 -- 172.21.15.1:0/1789542452 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df800b980 con 0x7f8e000cfe70 2024-03-18T20:26:23.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8dfdffb700 1 -- 172.21.15.1:0/1789542452 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df800bd30 con 0x7f8e000cfe70 2024-03-18T20:26:23.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8e052d6700 1 -- 172.21.15.1:0/1789542452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e000cfe70 msgr2=0x7f8e00161250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:23.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8e052d6700 1 --2- 172.21.15.1:0/1789542452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e000cfe70 0x7f8e00161250 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7f8e0008bb60 tx=0x7f8df800b330 comp rx=0 tx=0).stop 2024-03-18T20:26:23.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8e052d6700 1 -- 172.21.15.1:0/1789542452 shutdown_connections 2024-03-18T20:26:23.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8e052d6700 1 --2- 172.21.15.1:0/1789542452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e000cfe70 0x7f8e00161250 unknown :-1 s=CLOSED pgs=2786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:23.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8e052d6700 1 --2- 172.21.15.1:0/1789542452 >> v2:172.21.15.165:3300/0 conn(0x7f8e0015d8b0 0x7f8e000d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:23.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8e052d6700 1 --2- 172.21.15.1:0/1789542452 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e000d1170 0x7f8e000cf930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:23.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8e052d6700 1 -- 172.21.15.1:0/1789542452 >> 172.21.15.1:0/1789542452 conn(0x7f8e0009c640 msgr2=0x7f8e000d3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:23.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8e052d6700 1 -- 172.21.15.1:0/1789542452 shutdown_connections 2024-03-18T20:26:23.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.367+0000 7f8e052d6700 1 -- 172.21.15.1:0/1789542452 wait complete. 2024-03-18T20:26:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.369+0000 7f8e052d6700 1 Processor -- start 2024-03-18T20:26:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.369+0000 7f8e052d6700 1 -- start start 2024-03-18T20:26:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.369+0000 7f8e052d6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e000cfe70 0x7f8e000b8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.369+0000 7f8e052d6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e000d1170 0x7f8e00157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.369+0000 7f8e052d6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e0015d8b0 0x7f8e000c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.369+0000 7f8e052d6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e000c8680 con 0x7f8e0015d8b0 2024-03-18T20:26:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.369+0000 7f8e052d6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e00158040 con 0x7f8e000d1170 2024-03-18T20:26:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.369+0000 7f8e052d6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e000c7680 con 0x7f8e000cfe70 2024-03-18T20:26:23.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.369+0000 7f8dff7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e0015d8b0 0x7f8e000c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.369+0000 7f8dff7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e0015d8b0 0x7f8e000c6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56662/0 (socket says 172.21.15.1:56662) 2024-03-18T20:26:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.369+0000 7f8dff7fe700 1 -- 172.21.15.1:0/3203010203 learned_addr learned my addr 172.21.15.1:0/3203010203 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.370+0000 7f8dfeffd700 1 --2- 172.21.15.1:0/3203010203 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e000cfe70 0x7f8e000b8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.370+0000 7f8dfe7fc700 1 --2- 172.21.15.1:0/3203010203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e000d1170 0x7f8e00157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.370+0000 7f8dff7fe700 1 -- 172.21.15.1:0/3203010203 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e000cfe70 msgr2=0x7f8e000b8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.370+0000 7f8dff7fe700 1 --2- 172.21.15.1:0/3203010203 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e000cfe70 0x7f8e000b8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.370+0000 7f8dff7fe700 1 -- 172.21.15.1:0/3203010203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e000d1170 msgr2=0x7f8e00157b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.370+0000 7f8dff7fe700 1 --2- 172.21.15.1:0/3203010203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e000d1170 0x7f8e00157b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.370+0000 7f8dff7fe700 1 -- 172.21.15.1:0/3203010203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df8009b30 con 0x7f8e0015d8b0 2024-03-18T20:26:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.370+0000 7f8dfeffd700 1 --2- 172.21.15.1:0/3203010203 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e000cfe70 0x7f8e000b8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.370+0000 7f8dfe7fc700 1 --2- 172.21.15.1:0/3203010203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e000d1170 0x7f8e00157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.370+0000 7f8dff7fe700 1 --2- 172.21.15.1:0/3203010203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e0015d8b0 0x7f8e000c6e30 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f8e0008bc30 tx=0x7f8df8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.370+0000 7f8de77fe700 1 -- 172.21.15.1:0/3203010203 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df8004250 con 0x7f8e0015d8b0 2024-03-18T20:26:23.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.370+0000 7f8de77fe700 1 -- 172.21.15.1:0/3203010203 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df80043b0 con 0x7f8e0015d8b0 2024-03-18T20:26:23.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.370+0000 7f8e052d6700 1 -- 172.21.15.1:0/3203010203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e000c7960 con 0x7f8e0015d8b0 2024-03-18T20:26:23.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.370+0000 7f8de77fe700 1 -- 172.21.15.1:0/3203010203 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df8004760 con 0x7f8e0015d8b0 2024-03-18T20:26:23.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.370+0000 7f8e052d6700 1 -- 172.21.15.1:0/3203010203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e000c88d0 con 0x7f8e0015d8b0 2024-03-18T20:26:23.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.371+0000 7f8e052d6700 1 -- 172.21.15.1:0/3203010203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e00073350 con 0x7f8e0015d8b0 2024-03-18T20:26:23.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.372+0000 7f8de77fe700 1 -- 172.21.15.1:0/3203010203 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f8df801b430 con 0x7f8e0015d8b0 2024-03-18T20:26:23.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.373+0000 7f8de77fe700 1 --2- 172.21.15.1:0/3203010203 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8dec06aed0 0x7f8dec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:23.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.373+0000 7f8dfeffd700 1 --2- 172.21.15.1:0/3203010203 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8dec06aed0 0x7f8dec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:23.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.373+0000 7f8de77fe700 1 -- 172.21.15.1:0/3203010203 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f8df808ad20 con 0x7f8e0015d8b0 2024-03-18T20:26:23.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.373+0000 7f8dfeffd700 1 --2- 172.21.15.1:0/3203010203 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8dec06aed0 0x7f8dec06d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8e000b46c0 tx=0x7f8df4008040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:23.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.378+0000 7f8de77fe700 1 -- 172.21.15.1:0/3203010203 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8df8055bb0 con 0x7f8e0015d8b0 2024-03-18T20:26:23.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:23.580+0000 7f8e052d6700 1 -- 172.21.15.1:0/3203010203 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8e000b2490 con 0x7f8e0015d8b0 2024-03-18T20:26:24.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.345+0000 7f8de77fe700 1 -- 172.21.15.1:0/3203010203 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v250) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8df80591d0 con 0x7f8e0015d8b0 2024-03-18T20:26:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.348+0000 7f8e052d6700 1 -- 172.21.15.1:0/3203010203 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8dec06aed0 msgr2=0x7f8dec06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.348+0000 7f8e052d6700 1 --2- 172.21.15.1:0/3203010203 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8dec06aed0 0x7f8dec06d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8e000b46c0 tx=0x7f8df4008040 comp rx=0 tx=0).stop 2024-03-18T20:26:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.348+0000 7f8e052d6700 1 -- 172.21.15.1:0/3203010203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e0015d8b0 msgr2=0x7f8e000c6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.348+0000 7f8e052d6700 1 --2- 172.21.15.1:0/3203010203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e0015d8b0 0x7f8e000c6e30 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f8e0008bc30 tx=0x7f8df8004ad0 comp rx=0 tx=0).stop 2024-03-18T20:26:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.349+0000 7f8e052d6700 1 -- 172.21.15.1:0/3203010203 shutdown_connections 2024-03-18T20:26:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.349+0000 7f8e052d6700 1 --2- 172.21.15.1:0/3203010203 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8dec06aed0 0x7f8dec06d390 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.349+0000 7f8e052d6700 1 --2- 172.21.15.1:0/3203010203 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8e000cfe70 0x7f8e000b8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.349+0000 7f8e052d6700 1 --2- 172.21.15.1:0/3203010203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e0015d8b0 0x7f8e000c6e30 unknown :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:24.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.349+0000 7f8e052d6700 1 --2- 172.21.15.1:0/3203010203 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e000d1170 0x7f8e00157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:24.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.349+0000 7f8e052d6700 1 -- 172.21.15.1:0/3203010203 >> 172.21.15.1:0/3203010203 conn(0x7f8e0009c640 msgr2=0x7f8e000d3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:24.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.349+0000 7f8e052d6700 1 -- 172.21.15.1:0/3203010203 shutdown_connections 2024-03-18T20:26:24.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.349+0000 7f8e052d6700 1 -- 172.21.15.1:0/3203010203 wait complete. 2024-03-18T20:26:24.354 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-18T20:26:24.371 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T20:26:24.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.507+0000 7f4d9ecb6700 1 Processor -- start 2024-03-18T20:26:24.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.507+0000 7f4d9ecb6700 1 -- start start 2024-03-18T20:26:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.507+0000 7f4d9ecb6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4d98158e30 0x7f4d98071500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:24.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.507+0000 7f4d9ecb6700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9815c500 con 0x7f4d9806ae60 2024-03-18T20:26:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.507+0000 7f4d9ecb6700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d980af8b0 con 0x7f4d980cbf70 2024-03-18T20:26:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.507+0000 7f4d9ecb6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d9815b8f0 con 0x7f4d98158e30 2024-03-18T20:26:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d9d253700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4d9806ae60 0x7f4d980cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53448/0 (socket says 172.21.15.1:53448) 2024-03-18T20:26:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d9d253700 1 -- 172.21.15.1:0/2422689108 learned_addr learned my addr 172.21.15.1:0/2422689108 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d9ca52700 1 --2- 172.21.15.1:0/2422689108 >> v2:172.21.15.165:3300/0 conn(0x7f4d98158e30 0x7f4d98071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d977fe700 1 -- 172.21.15.1:0/2422689108 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095638598 0 0) 0x7f4d9815c500 con 0x7f4d9806ae60 2024-03-18T20:26:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d977fe700 1 -- 172.21.15.1:0/2422689108 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d880035b0 con 0x7f4d9806ae60 2024-03-18T20:26:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d977fe700 1 -- 172.21.15.1:0/2422689108 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2429491509 0 0) 0x7f4d880035b0 con 0x7f4d9806ae60 2024-03-18T20:26:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d977fe700 1 -- 172.21.15.1:0/2422689108 >> v2:172.21.15.165:3300/0 conn(0x7f4d98158e30 msgr2=0x7f4d98071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d977fe700 1 --2- 172.21.15.1:0/2422689108 >> v2:172.21.15.165:3300/0 conn(0x7f4d98158e30 0x7f4d98071500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d977fe700 1 -- 172.21.15.1:0/2422689108 >> v1:172.21.15.165:6789/0 conn(0x7f4d980cbf70 legacy=0x7f4d9806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d977fe700 1 -- 172.21.15.1:0/2422689108 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9815bbd0 con 0x7f4d9806ae60 2024-03-18T20:26:24.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d977fe700 1 -- 172.21.15.1:0/2422689108 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f4d900019f0 con 0x7f4d9806ae60 2024-03-18T20:26:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d977fe700 1 -- 172.21.15.1:0/2422689108 >> v1:172.21.15.1:6789/0 conn(0x7f4d9806ae60 legacy=0x7f4d980cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d977fe700 1 --2- 172.21.15.1:0/2422689108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d88003a00 0x7f4d88003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d977fe700 1 --2- 172.21.15.1:0/2422689108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d88004400 0x7f4d880048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d977fe700 1 --2- 172.21.15.1:0/2422689108 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d880050d0 0x7f4d8800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d977fe700 1 -- 172.21.15.1:0/2422689108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9815b8f0 con 0x7f4d88003a00 2024-03-18T20:26:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d977fe700 1 -- 172.21.15.1:0/2422689108 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8800c030 con 0x7f4d88004400 2024-03-18T20:26:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d977fe700 1 -- 172.21.15.1:0/2422689108 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8800c1f0 con 0x7f4d880050d0 2024-03-18T20:26:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d9ca52700 1 --2- 172.21.15.1:0/2422689108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d88003a00 0x7f4d88003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d9d253700 1 --2- 172.21.15.1:0/2422689108 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d880050d0 0x7f4d8800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.508+0000 7f4d97fff700 1 --2- 172.21.15.1:0/2422689108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d88004400 0x7f4d880048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.509+0000 7f4d9ca52700 1 -- 172.21.15.1:0/2422689108 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d880050d0 msgr2=0x7f4d8800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.509+0000 7f4d9ca52700 1 --2- 172.21.15.1:0/2422689108 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d880050d0 0x7f4d8800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.509+0000 7f4d9ca52700 1 -- 172.21.15.1:0/2422689108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d88004400 msgr2=0x7f4d880048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:24.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.509+0000 7f4d9ca52700 1 --2- 172.21.15.1:0/2422689108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d88004400 0x7f4d880048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:24.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.509+0000 7f4d9ca52700 1 -- 172.21.15.1:0/2422689108 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d880035b0 con 0x7f4d88003a00 2024-03-18T20:26:24.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.509+0000 7f4d9d253700 1 -- 172.21.15.1:0/2422689108 reap_dead start 2024-03-18T20:26:24.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.509+0000 7f4d9ca52700 1 --2- 172.21.15.1:0/2422689108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d88003a00 0x7f4d88003de0 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c009b60 tx=0x7f4d88005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=668bd5fd32ee0390 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:24.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.509+0000 7f4d977fe700 1 -- 172.21.15.1:0/2422689108 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c00e6c0 con 0x7f4d88003a00 2024-03-18T20:26:24.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.509+0000 7f4d977fe700 1 -- 172.21.15.1:0/2422689108 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d8c00e820 con 0x7f4d88003a00 2024-03-18T20:26:24.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.509+0000 7f4d977fe700 1 -- 172.21.15.1:0/2422689108 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c00ebb0 con 0x7f4d88003a00 2024-03-18T20:26:24.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.509+0000 7f4d9ecb6700 1 -- 172.21.15.1:0/2422689108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d88003a00 msgr2=0x7f4d88003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:24.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.509+0000 7f4d9ecb6700 1 --2- 172.21.15.1:0/2422689108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d88003a00 0x7f4d88003de0 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c009b60 tx=0x7f4d88005880 comp rx=0 tx=0).stop 2024-03-18T20:26:24.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.509+0000 7f4d9ecb6700 1 -- 172.21.15.1:0/2422689108 shutdown_connections 2024-03-18T20:26:24.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.509+0000 7f4d9ecb6700 1 --2- 172.21.15.1:0/2422689108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d88003a00 0x7f4d88003de0 unknown :-1 s=CLOSED pgs=2789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:24.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.510+0000 7f4d9ecb6700 1 -- 172.21.15.1:0/2422689108 >> 172.21.15.1:0/2422689108 conn(0x7f4d980cca00 msgr2=0x7f4d980cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:24.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.510+0000 7f4d9ecb6700 1 -- 172.21.15.1:0/2422689108 shutdown_connections 2024-03-18T20:26:24.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.510+0000 7f4d9ecb6700 1 -- 172.21.15.1:0/2422689108 wait complete. 2024-03-18T20:26:24.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.511+0000 7f4d9ecb6700 1 Processor -- start 2024-03-18T20:26:24.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.511+0000 7f4d9ecb6700 1 -- start start 2024-03-18T20:26:24.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.512+0000 7f4d9ecb6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d88003a00 0x7f4d98150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:24.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.512+0000 7f4d9ecb6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d980c7050 0x7f4d980c3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:24.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.512+0000 7f4d9ecb6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d980c4200 0x7f4d9806e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:24.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.512+0000 7f4d9ecb6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d980c46b0 con 0x7f4d88003a00 2024-03-18T20:26:24.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.512+0000 7f4d9ecb6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d980c65a0 con 0x7f4d980c7050 2024-03-18T20:26:24.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.512+0000 7f4d9ecb6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d980ce1e0 con 0x7f4d980c4200 2024-03-18T20:26:24.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.512+0000 7f4d9ca52700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d88003a00 0x7f4d98150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:24.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.512+0000 7f4d9ca52700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d88003a00 0x7f4d98150400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46452/0 (socket says 172.21.15.1:46452) 2024-03-18T20:26:24.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.512+0000 7f4d9ca52700 1 -- 172.21.15.1:0/3114416385 learned_addr learned my addr 172.21.15.1:0/3114416385 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:24.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.513+0000 7f4d9d253700 1 --2- 172.21.15.1:0/3114416385 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d980c4200 0x7f4d9806e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:24.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.513+0000 7f4d97fff700 1 --2- 172.21.15.1:0/3114416385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d980c7050 0x7f4d980c3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:24.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.513+0000 7f4d9ca52700 1 -- 172.21.15.1:0/3114416385 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d980c4200 msgr2=0x7f4d9806e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:24.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.513+0000 7f4d9ca52700 1 --2- 172.21.15.1:0/3114416385 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d980c4200 0x7f4d9806e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:24.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.513+0000 7f4d9ca52700 1 -- 172.21.15.1:0/3114416385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d980c7050 msgr2=0x7f4d980c3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:24.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.513+0000 7f4d9ca52700 1 --2- 172.21.15.1:0/3114416385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d980c7050 0x7f4d980c3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:24.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.513+0000 7f4d9ca52700 1 -- 172.21.15.1:0/3114416385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d980ce4c0 con 0x7f4d88003a00 2024-03-18T20:26:24.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.513+0000 7f4d9ca52700 1 --2- 172.21.15.1:0/3114416385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d88003a00 0x7f4d98150400 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c0090b0 tx=0x7f4d8c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:24.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.513+0000 7f4d957fa700 1 -- 172.21.15.1:0/3114416385 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c00e6c0 con 0x7f4d88003a00 2024-03-18T20:26:24.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.513+0000 7f4d957fa700 1 -- 172.21.15.1:0/3114416385 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d8c003ab0 con 0x7f4d88003a00 2024-03-18T20:26:24.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.513+0000 7f4d9ecb6700 1 -- 172.21.15.1:0/3114416385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d980ce7a0 con 0x7f4d88003a00 2024-03-18T20:26:24.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.513+0000 7f4d957fa700 1 -- 172.21.15.1:0/3114416385 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c003e60 con 0x7f4d88003a00 2024-03-18T20:26:24.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.513+0000 7f4d9ecb6700 1 -- 172.21.15.1:0/3114416385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d980ced60 con 0x7f4d88003a00 2024-03-18T20:26:24.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.514+0000 7f4d9ecb6700 1 -- 172.21.15.1:0/3114416385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d98073940 con 0x7f4d88003a00 2024-03-18T20:26:24.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.515+0000 7f4d957fa700 1 -- 172.21.15.1:0/3114416385 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f4d8c013470 con 0x7f4d88003a00 2024-03-18T20:26:24.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.515+0000 7f4d957fa700 1 --2- 172.21.15.1:0/3114416385 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4d7c06aed0 0x7f4d7c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:24.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.515+0000 7f4d957fa700 1 -- 172.21.15.1:0/3114416385 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5647+0+0 (secure 0 0 0) 0x7f4d8c08afe0 con 0x7f4d88003a00 2024-03-18T20:26:24.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.515+0000 7f4d97fff700 1 --2- 172.21.15.1:0/3114416385 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4d7c06aed0 0x7f4d7c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:24.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.515+0000 7f4d97fff700 1 --2- 172.21.15.1:0/3114416385 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4d7c06aed0 0x7f4d7c06d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4d980ae4d0 tx=0x7f4d84009040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:24.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.520+0000 7f4d957fa700 1 -- 172.21.15.1:0/3114416385 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4d8c0be580 con 0x7f4d88003a00 2024-03-18T20:26:24.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:24.725+0000 7f4d9ecb6700 1 -- 172.21.15.1:0/3114416385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4d980c4610 con 0x7f4d88003a00 2024-03-18T20:26:25.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.349+0000 7f4d957fa700 1 -- 172.21.15.1:0/3114416385 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v251) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4d8c0170d0 con 0x7f4d88003a00 2024-03-18T20:26:25.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.358+0000 7f4d9ecb6700 1 -- 172.21.15.1:0/3114416385 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4d7c06aed0 msgr2=0x7f4d7c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:25.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.358+0000 7f4d9ecb6700 1 --2- 172.21.15.1:0/3114416385 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4d7c06aed0 0x7f4d7c06d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4d980ae4d0 tx=0x7f4d84009040 comp rx=0 tx=0).stop 2024-03-18T20:26:25.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.358+0000 7f4d9ecb6700 1 -- 172.21.15.1:0/3114416385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d88003a00 msgr2=0x7f4d98150400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:25.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.358+0000 7f4d9ecb6700 1 --2- 172.21.15.1:0/3114416385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d88003a00 0x7f4d98150400 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c0090b0 tx=0x7f4d8c0047c0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.359+0000 7f4d9ecb6700 1 -- 172.21.15.1:0/3114416385 shutdown_connections 2024-03-18T20:26:25.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.359+0000 7f4d9ecb6700 1 --2- 172.21.15.1:0/3114416385 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4d7c06aed0 0x7f4d7c06d390 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.359+0000 7f4d9ecb6700 1 --2- 172.21.15.1:0/3114416385 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d980c4200 0x7f4d9806e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.359+0000 7f4d9ecb6700 1 --2- 172.21.15.1:0/3114416385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d88003a00 0x7f4d98150400 unknown :-1 s=CLOSED pgs=2790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.359+0000 7f4d9ecb6700 1 --2- 172.21.15.1:0/3114416385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d980c7050 0x7f4d980c3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.359+0000 7f4d9ecb6700 1 -- 172.21.15.1:0/3114416385 >> 172.21.15.1:0/3114416385 conn(0x7f4d980cca00 msgr2=0x7f4d98158150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:25.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.361+0000 7f4d9ecb6700 1 -- 172.21.15.1:0/3114416385 shutdown_connections 2024-03-18T20:26:25.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.362+0000 7f4d9ecb6700 1 -- 172.21.15.1:0/3114416385 wait complete. 2024-03-18T20:26:25.376 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-18T20:26:25.390 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T20:26:25.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.520+0000 7f1506dc7700 1 Processor -- start 2024-03-18T20:26:25.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.520+0000 7f1506dc7700 1 -- start start 2024-03-18T20:26:25.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.521+0000 7f1506dc7700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f150015eba0 0x7f15000cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:25.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.521+0000 7f1506dc7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f150015ef80 con 0x7f15000b0e20 2024-03-18T20:26:25.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.521+0000 7f1506dc7700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15000cc9a0 con 0x7f15000b0500 2024-03-18T20:26:25.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.521+0000 7f1506dc7700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15000ccc80 con 0x7f150015eba0 2024-03-18T20:26:25.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.521+0000 7f1505364700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f15000b0e20 0x7f15000b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53454/0 (socket says 172.21.15.1:53454) 2024-03-18T20:26:25.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.521+0000 7f1505364700 1 -- 172.21.15.1:0/2529401777 learned_addr learned my addr 172.21.15.1:0/2529401777 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:25.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.521+0000 7f1504b63700 1 --2- 172.21.15.1:0/2529401777 >> v2:172.21.15.165:3300/0 conn(0x7f150015eba0 0x7f15000cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:25.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.521+0000 7f14ff7fe700 1 -- 172.21.15.1:0/2529401777 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623456525 0 0) 0x7f150015ef80 con 0x7f15000b0e20 2024-03-18T20:26:25.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f14ff7fe700 1 -- 172.21.15.1:0/2529401777 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14f00035b0 con 0x7f15000b0e20 2024-03-18T20:26:25.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f14ff7fe700 1 -- 172.21.15.1:0/2529401777 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714547773 0 0) 0x7f15000cc9a0 con 0x7f15000b0500 2024-03-18T20:26:25.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f14ff7fe700 1 -- 172.21.15.1:0/2529401777 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f150015ef80 con 0x7f15000b0500 2024-03-18T20:26:25.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f14ff7fe700 1 -- 172.21.15.1:0/2529401777 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2856031414 0 0) 0x7f14f00035b0 con 0x7f15000b0e20 2024-03-18T20:26:25.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f14ff7fe700 1 -- 172.21.15.1:0/2529401777 >> v2:172.21.15.165:3300/0 conn(0x7f150015eba0 msgr2=0x7f15000cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:25.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f14ff7fe700 1 --2- 172.21.15.1:0/2529401777 >> v2:172.21.15.165:3300/0 conn(0x7f150015eba0 0x7f15000cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f14ff7fe700 1 -- 172.21.15.1:0/2529401777 >> v1:172.21.15.165:6789/0 conn(0x7f15000b0500 legacy=0x7f15000cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:25.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f14ff7fe700 1 -- 172.21.15.1:0/2529401777 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15000ccf60 con 0x7f15000b0e20 2024-03-18T20:26:25.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f14ff7fe700 1 -- 172.21.15.1:0/2529401777 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f14f80019f0 con 0x7f15000b0e20 2024-03-18T20:26:25.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f14ff7fe700 1 -- 172.21.15.1:0/2529401777 >> v1:172.21.15.1:6789/0 conn(0x7f15000b0e20 legacy=0x7f15000b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f14ff7fe700 1 --2- 172.21.15.1:0/2529401777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f0003b90 0x7f14f0003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f14ff7fe700 1 --2- 172.21.15.1:0/2529401777 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14f0004850 0x7f14f0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f14ff7fe700 1 --2- 172.21.15.1:0/2529401777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14f0005210 0x7f14f000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f14ff7fe700 1 -- 172.21.15.1:0/2529401777 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15000ccc80 con 0x7f14f0003b90 2024-03-18T20:26:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f14ff7fe700 1 -- 172.21.15.1:0/2529401777 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f000c140 con 0x7f14f0005210 2024-03-18T20:26:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f14ff7fe700 1 -- 172.21.15.1:0/2529401777 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f000c300 con 0x7f14f0004850 2024-03-18T20:26:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f1504b63700 1 --2- 172.21.15.1:0/2529401777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f0003b90 0x7f14f0003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f1505364700 1 --2- 172.21.15.1:0/2529401777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14f0005210 0x7f14f000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f14fffff700 1 --2- 172.21.15.1:0/2529401777 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14f0004850 0x7f14f0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f1504b63700 1 -- 172.21.15.1:0/2529401777 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14f0004850 msgr2=0x7f14f0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f1504b63700 1 --2- 172.21.15.1:0/2529401777 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f14f0004850 0x7f14f0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f1504b63700 1 -- 172.21.15.1:0/2529401777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14f0005210 msgr2=0x7f14f000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:25.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f1504b63700 1 --2- 172.21.15.1:0/2529401777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14f0005210 0x7f14f000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f1504b63700 1 -- 172.21.15.1:0/2529401777 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f00035b0 con 0x7f14f0003b90 2024-03-18T20:26:25.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f1505364700 1 --2- 172.21.15.1:0/2529401777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14f0005210 0x7f14f000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:25.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f1505364700 1 -- 172.21.15.1:0/2529401777 reap_dead start 2024-03-18T20:26:25.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.522+0000 7f1504b63700 1 --2- 172.21.15.1:0/2529401777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f0003b90 0x7f14f0003f90 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f14f4009b60 tx=0x7f14f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2c3ea932dc5014 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:25.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.523+0000 7f14ff7fe700 1 -- 172.21.15.1:0/2529401777 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f400e6c0 con 0x7f14f0003b90 2024-03-18T20:26:25.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.523+0000 7f14ff7fe700 1 -- 172.21.15.1:0/2529401777 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f400e820 con 0x7f14f0003b90 2024-03-18T20:26:25.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.523+0000 7f14ff7fe700 1 -- 172.21.15.1:0/2529401777 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f400eb90 con 0x7f14f0003b90 2024-03-18T20:26:25.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.523+0000 7f1506dc7700 1 -- 172.21.15.1:0/2529401777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f0003b90 msgr2=0x7f14f0003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:25.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.523+0000 7f1506dc7700 1 --2- 172.21.15.1:0/2529401777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f0003b90 0x7f14f0003f90 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f14f4009b60 tx=0x7f14f0005880 comp rx=0 tx=0).stop 2024-03-18T20:26:25.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.523+0000 7f1506dc7700 1 -- 172.21.15.1:0/2529401777 shutdown_connections 2024-03-18T20:26:25.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.523+0000 7f1506dc7700 1 --2- 172.21.15.1:0/2529401777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f0003b90 0x7f14f0003f90 unknown :-1 s=CLOSED pgs=2792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.523+0000 7f1506dc7700 1 -- 172.21.15.1:0/2529401777 >> 172.21.15.1:0/2529401777 conn(0x7f15000b2bd0 msgr2=0x7f15000b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:25.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.523+0000 7f1506dc7700 1 -- 172.21.15.1:0/2529401777 shutdown_connections 2024-03-18T20:26:25.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.523+0000 7f1506dc7700 1 -- 172.21.15.1:0/2529401777 wait complete. 2024-03-18T20:26:25.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.525+0000 7f1506dc7700 1 Processor -- start 2024-03-18T20:26:25.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.525+0000 7f1506dc7700 1 -- start start 2024-03-18T20:26:25.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.526+0000 7f1506dc7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f0003b90 0x7f15001514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:25.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.526+0000 7f1506dc7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f150006c520 0x7f15000add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:25.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.526+0000 7f1506dc7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15000c81b0 0x7f15001e4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:25.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.526+0000 7f1506dc7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150006d150 con 0x7f14f0003b90 2024-03-18T20:26:25.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.526+0000 7f1506dc7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150006d430 con 0x7f150006c520 2024-03-18T20:26:25.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.526+0000 7f1506dc7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f150006d710 con 0x7f15000c81b0 2024-03-18T20:26:25.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.526+0000 7f1504b63700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f0003b90 0x7f15001514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:25.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.526+0000 7f1504b63700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f0003b90 0x7f15001514f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46474/0 (socket says 172.21.15.1:46474) 2024-03-18T20:26:25.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.526+0000 7f1504b63700 1 -- 172.21.15.1:0/2966828832 learned_addr learned my addr 172.21.15.1:0/2966828832 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:25.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.526+0000 7f14fffff700 1 --2- 172.21.15.1:0/2966828832 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f150006c520 0x7f15000add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:25.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.526+0000 7f1505364700 1 --2- 172.21.15.1:0/2966828832 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15000c81b0 0x7f15001e4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:25.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.526+0000 7f1504b63700 1 -- 172.21.15.1:0/2966828832 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15000c81b0 msgr2=0x7f15001e4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:25.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.526+0000 7f1504b63700 1 --2- 172.21.15.1:0/2966828832 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15000c81b0 0x7f15001e4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.526+0000 7f1504b63700 1 -- 172.21.15.1:0/2966828832 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f150006c520 msgr2=0x7f15000add90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:25.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.526+0000 7f1504b63700 1 --2- 172.21.15.1:0/2966828832 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f150006c520 0x7f15000add90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.526+0000 7f1504b63700 1 -- 172.21.15.1:0/2966828832 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15000ca0b0 con 0x7f14f0003b90 2024-03-18T20:26:25.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.527+0000 7f1504b63700 1 --2- 172.21.15.1:0/2966828832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f0003b90 0x7f15001514f0 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7f14f000c110 tx=0x7f14f40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:25.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.527+0000 7f14fd7fa700 1 -- 172.21.15.1:0/2966828832 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f400e6c0 con 0x7f14f0003b90 2024-03-18T20:26:25.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.527+0000 7f14fd7fa700 1 -- 172.21.15.1:0/2966828832 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f40038a0 con 0x7f14f0003b90 2024-03-18T20:26:25.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.527+0000 7f14fd7fa700 1 -- 172.21.15.1:0/2966828832 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f4003c50 con 0x7f14f0003b90 2024-03-18T20:26:25.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.527+0000 7f1506dc7700 1 -- 172.21.15.1:0/2966828832 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1500151a30 con 0x7f14f0003b90 2024-03-18T20:26:25.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.527+0000 7f1506dc7700 1 -- 172.21.15.1:0/2966828832 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15000ae2d0 con 0x7f14f0003b90 2024-03-18T20:26:25.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.528+0000 7f1506dc7700 1 -- 172.21.15.1:0/2966828832 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15000ca420 con 0x7f14f0003b90 2024-03-18T20:26:25.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.529+0000 7f14fd7fa700 1 -- 172.21.15.1:0/2966828832 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f14f4013470 con 0x7f14f0003b90 2024-03-18T20:26:25.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.529+0000 7f14fd7fa700 1 --2- 172.21.15.1:0/2966828832 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f14e406aed0 0x7f14e406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:25.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.529+0000 7f14fd7fa700 1 -- 172.21.15.1:0/2966828832 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f14f408b670 con 0x7f14f0003b90 2024-03-18T20:26:25.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.529+0000 7f14fffff700 1 --2- 172.21.15.1:0/2966828832 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f14e406aed0 0x7f14e406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:25.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.529+0000 7f14fffff700 1 --2- 172.21.15.1:0/2966828832 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f14e406aed0 0x7f14e406d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f15000c9b00 tx=0x7f14ec0012d0 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:25.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.533+0000 7f14fd7fa700 1 -- 172.21.15.1:0/2966828832 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f14f4056520 con 0x7f14f0003b90 2024-03-18T20:26:25.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.783+0000 7f1506dc7700 1 -- 172.21.15.1:0/2966828832 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f15000c9ef0 con 0x7f14f0003b90 2024-03-18T20:26:25.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.789+0000 7f14fd7fa700 1 -- 172.21.15.1:0/2966828832 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 31 and data pool 32 v116) v1 ==== 174+0+0 (secure 0 0 0) 0x7f14f4059b40 con 0x7f14f0003b90 2024-03-18T20:26:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.795+0000 7f1506dc7700 1 -- 172.21.15.1:0/2966828832 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f14e406aed0 msgr2=0x7f14e406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.795+0000 7f1506dc7700 1 --2- 172.21.15.1:0/2966828832 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f14e406aed0 0x7f14e406d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f15000c9b00 tx=0x7f14ec0012d0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.795+0000 7f1506dc7700 1 -- 172.21.15.1:0/2966828832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f0003b90 msgr2=0x7f15001514f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:25.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.795+0000 7f1506dc7700 1 --2- 172.21.15.1:0/2966828832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f0003b90 0x7f15001514f0 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7f14f000c110 tx=0x7f14f40047c0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.797+0000 7f1506dc7700 1 -- 172.21.15.1:0/2966828832 shutdown_connections 2024-03-18T20:26:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.797+0000 7f1506dc7700 1 --2- 172.21.15.1:0/2966828832 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f14e406aed0 0x7f14e406d390 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.797+0000 7f1506dc7700 1 --2- 172.21.15.1:0/2966828832 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15000c81b0 0x7f15001e4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.797+0000 7f1506dc7700 1 --2- 172.21.15.1:0/2966828832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14f0003b90 0x7f15001514f0 unknown :-1 s=CLOSED pgs=2793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.797+0000 7f1506dc7700 1 --2- 172.21.15.1:0/2966828832 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f150006c520 0x7f15000add90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.797+0000 7f1506dc7700 1 -- 172.21.15.1:0/2966828832 >> 172.21.15.1:0/2966828832 conn(0x7f15000b2bd0 msgr2=0x7f15000cfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:25.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.799+0000 7f1506dc7700 1 -- 172.21.15.1:0/2966828832 shutdown_connections 2024-03-18T20:26:25.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.800+0000 7f1506dc7700 1 -- 172.21.15.1:0/2966828832 wait complete. 2024-03-18T20:26:25.809 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 31 and data pool 32 2024-03-18T20:26:25.828 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:26:25.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.950+0000 7fe06e67e700 1 Processor -- start 2024-03-18T20:26:25.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.950+0000 7fe06e67e700 1 -- start start 2024-03-18T20:26:25.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.950+0000 7fe06e67e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe068099ea0 0x7fe0680cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:25.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.950+0000 7fe06e67e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0680d0c80 con 0x7fe06815d680 2024-03-18T20:26:25.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.950+0000 7fe06e67e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0680d09a0 con 0x7fe068099a40 2024-03-18T20:26:25.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.950+0000 7fe06e67e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0680cb830 con 0x7fe068099ea0 2024-03-18T20:26:25.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.950+0000 7fe067fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe06815d680 0x7fe0680cff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53462/0 (socket says 172.21.15.1:53462) 2024-03-18T20:26:25.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe067fff700 1 -- 172.21.15.1:0/345665195 learned_addr learned my addr 172.21.15.1:0/345665195 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:25.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe06cc1b700 1 --2- 172.21.15.1:0/345665195 >> v2:172.21.15.165:3300/0 conn(0x7fe068099ea0 0x7fe0680cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe066ffd700 1 -- 172.21.15.1:0/345665195 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210610768 0 0) 0x7fe0680d0c80 con 0x7fe06815d680 2024-03-18T20:26:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe066ffd700 1 -- 172.21.15.1:0/345665195 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0500035b0 con 0x7fe06815d680 2024-03-18T20:26:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe066ffd700 1 -- 172.21.15.1:0/345665195 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3223638210 0 0) 0x7fe0500035b0 con 0x7fe06815d680 2024-03-18T20:26:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe066ffd700 1 -- 172.21.15.1:0/345665195 >> v2:172.21.15.165:3300/0 conn(0x7fe068099ea0 msgr2=0x7fe0680cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe066ffd700 1 --2- 172.21.15.1:0/345665195 >> v2:172.21.15.165:3300/0 conn(0x7fe068099ea0 0x7fe0680cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe066ffd700 1 -- 172.21.15.1:0/345665195 >> v1:172.21.15.165:6789/0 conn(0x7fe068099a40 legacy=0x7fe0680cc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe066ffd700 1 -- 172.21.15.1:0/345665195 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0680cbb10 con 0x7fe06815d680 2024-03-18T20:26:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe066ffd700 1 -- 172.21.15.1:0/345665195 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fe058001670 con 0x7fe06815d680 2024-03-18T20:26:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe066ffd700 1 -- 172.21.15.1:0/345665195 >> v1:172.21.15.1:6789/0 conn(0x7fe06815d680 legacy=0x7fe0680cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe066ffd700 1 --2- 172.21.15.1:0/345665195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe050003a00 0x7fe050003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe066ffd700 1 --2- 172.21.15.1:0/345665195 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe050004850 0x7fe050004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe066ffd700 1 --2- 172.21.15.1:0/345665195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe050005210 0x7fe05000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe066ffd700 1 -- 172.21.15.1:0/345665195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0680cb830 con 0x7fe050003a00 2024-03-18T20:26:25.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe066ffd700 1 -- 172.21.15.1:0/345665195 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05000c0b0 con 0x7fe050005210 2024-03-18T20:26:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe066ffd700 1 -- 172.21.15.1:0/345665195 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe05000c270 con 0x7fe050004850 2024-03-18T20:26:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe067fff700 1 --2- 172.21.15.1:0/345665195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe050003a00 0x7fe050003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe06cc1b700 1 --2- 172.21.15.1:0/345665195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe050005210 0x7fe05000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.951+0000 7fe0677fe700 1 --2- 172.21.15.1:0/345665195 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe050004850 0x7fe050004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.952+0000 7fe067fff700 1 -- 172.21.15.1:0/345665195 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe050004850 msgr2=0x7fe050004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.952+0000 7fe067fff700 1 --2- 172.21.15.1:0/345665195 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe050004850 0x7fe050004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.952+0000 7fe067fff700 1 -- 172.21.15.1:0/345665195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe050005210 msgr2=0x7fe05000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.952+0000 7fe067fff700 1 --2- 172.21.15.1:0/345665195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe050005210 0x7fe05000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.952+0000 7fe067fff700 1 -- 172.21.15.1:0/345665195 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0500035b0 con 0x7fe050003a00 2024-03-18T20:26:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.952+0000 7fe06cc1b700 1 -- 172.21.15.1:0/345665195 reap_dead start 2024-03-18T20:26:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.952+0000 7fe067fff700 1 --2- 172.21.15.1:0/345665195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe050003a00 0x7fe050003e20 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7fe05800c240 tx=0x7fe050005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74ecf48f0da8f40 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.952+0000 7fe066ffd700 1 -- 172.21.15.1:0/345665195 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe058007c10 con 0x7fe050003a00 2024-03-18T20:26:25.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.952+0000 7fe066ffd700 1 -- 172.21.15.1:0/345665195 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05800cd80 con 0x7fe050003a00 2024-03-18T20:26:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.952+0000 7fe066ffd700 1 -- 172.21.15.1:0/345665195 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe058005ba0 con 0x7fe050003a00 2024-03-18T20:26:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.952+0000 7fe06e67e700 1 -- 172.21.15.1:0/345665195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe050003a00 msgr2=0x7fe050003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.952+0000 7fe06e67e700 1 --2- 172.21.15.1:0/345665195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe050003a00 0x7fe050003e20 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7fe05800c240 tx=0x7fe050005880 comp rx=0 tx=0).stop 2024-03-18T20:26:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.953+0000 7fe06e67e700 1 -- 172.21.15.1:0/345665195 shutdown_connections 2024-03-18T20:26:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.953+0000 7fe06e67e700 1 --2- 172.21.15.1:0/345665195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe050003a00 0x7fe050003e20 unknown :-1 s=CLOSED pgs=2795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.953+0000 7fe06e67e700 1 -- 172.21.15.1:0/345665195 >> 172.21.15.1:0/345665195 conn(0x7fe0680b2ba0 msgr2=0x7fe0680b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.953+0000 7fe06e67e700 1 -- 172.21.15.1:0/345665195 shutdown_connections 2024-03-18T20:26:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.953+0000 7fe06e67e700 1 -- 172.21.15.1:0/345665195 wait complete. 2024-03-18T20:26:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.954+0000 7fe06e67e700 1 Processor -- start 2024-03-18T20:26:25.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.954+0000 7fe06e67e700 1 -- start start 2024-03-18T20:26:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe06e67e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe050003a00 0x7fe0680c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe06e67e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0680cabf0 0x7fe068151ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe06e67e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe06814eb70 0x7fe06806f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe06e67e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0680cb070 con 0x7fe0680cabf0 2024-03-18T20:26:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe06e67e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe068152000 con 0x7fe06814eb70 2024-03-18T20:26:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe06e67e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0680c72f0 con 0x7fe050003a00 2024-03-18T20:26:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe0677fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0680cabf0 0x7fe068151ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe0677fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0680cabf0 0x7fe068151ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46494/0 (socket says 172.21.15.1:46494) 2024-03-18T20:26:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe0677fe700 1 -- 172.21.15.1:0/1134673122 learned_addr learned my addr 172.21.15.1:0/1134673122 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe067fff700 1 --2- 172.21.15.1:0/1134673122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe050003a00 0x7fe0680c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:25.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe06cc1b700 1 --2- 172.21.15.1:0/1134673122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe06814eb70 0x7fe06806f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe0677fe700 1 -- 172.21.15.1:0/1134673122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe050003a00 msgr2=0x7fe0680c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe0677fe700 1 --2- 172.21.15.1:0/1134673122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe050003a00 0x7fe0680c7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe0677fe700 1 -- 172.21.15.1:0/1134673122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe06814eb70 msgr2=0x7fe06806f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe0677fe700 1 --2- 172.21.15.1:0/1134673122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe06814eb70 0x7fe06806f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe0677fe700 1 -- 172.21.15.1:0/1134673122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06815dbe0 con 0x7fe0680cabf0 2024-03-18T20:26:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe067fff700 1 --2- 172.21.15.1:0/1134673122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe050003a00 0x7fe0680c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe0677fe700 1 --2- 172.21.15.1:0/1134673122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0680cabf0 0x7fe068151ac0 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00f850 tx=0x7fe05c00fc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe064ff9700 1 -- 172.21.15.1:0/1134673122 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c0050d0 con 0x7fe0680cabf0 2024-03-18T20:26:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe064ff9700 1 -- 172.21.15.1:0/1134673122 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05c005230 con 0x7fe0680cabf0 2024-03-18T20:26:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe06e67e700 1 -- 172.21.15.1:0/1134673122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0680c66e0 con 0x7fe0680cabf0 2024-03-18T20:26:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe064ff9700 1 -- 172.21.15.1:0/1134673122 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c003680 con 0x7fe0680cabf0 2024-03-18T20:26:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.955+0000 7fe06e67e700 1 -- 172.21.15.1:0/1134673122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0680c6ca0 con 0x7fe0680cabf0 2024-03-18T20:26:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.956+0000 7fe06e67e700 1 -- 172.21.15.1:0/1134673122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0680c7460 con 0x7fe0680cabf0 2024-03-18T20:26:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.957+0000 7fe064ff9700 1 -- 172.21.15.1:0/1134673122 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fe05c0037e0 con 0x7fe0680cabf0 2024-03-18T20:26:25.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.957+0000 7fe064ff9700 1 --2- 172.21.15.1:0/1134673122 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe05406aed0 0x7fe05406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:25.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.957+0000 7fe064ff9700 1 -- 172.21.15.1:0/1134673122 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fe05c089600 con 0x7fe0680cabf0 2024-03-18T20:26:25.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.957+0000 7fe067fff700 1 --2- 172.21.15.1:0/1134673122 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe05406aed0 0x7fe05406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:25.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.958+0000 7fe067fff700 1 --2- 172.21.15.1:0/1134673122 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe05406aed0 0x7fe05406d390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe058015040 tx=0x7fe058007000 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:25.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:25.962+0000 7fe064ff9700 1 -- 172.21.15.1:0/1134673122 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe05c0542e0 con 0x7fe0680cabf0 2024-03-18T20:26:26.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.168+0000 7fe06e67e700 1 -- 172.21.15.1:0/1134673122 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0680676a0 con 0x7fe0680cabf0 2024-03-18T20:26:26.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.169+0000 7fe064ff9700 1 -- 172.21.15.1:0/1134673122 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) v1 ==== 74+0+16442 (secure 0 0 0) 0x7fe05c057900 con 0x7fe0680cabf0 2024-03-18T20:26:26.173 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:26.174 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":252,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:26:25.784752+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:26:23.615691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-18T20:26:24.759650+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:26:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.171+0000 7fe06e67e700 1 -- 172.21.15.1:0/1134673122 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe05406aed0 msgr2=0x7fe05406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.171+0000 7fe06e67e700 1 --2- 172.21.15.1:0/1134673122 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe05406aed0 0x7fe05406d390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe058015040 tx=0x7fe058007000 comp rx=0 tx=0).stop 2024-03-18T20:26:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.171+0000 7fe06e67e700 1 -- 172.21.15.1:0/1134673122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0680cabf0 msgr2=0x7fe068151ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.171+0000 7fe06e67e700 1 --2- 172.21.15.1:0/1134673122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0680cabf0 0x7fe068151ac0 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00f850 tx=0x7fe05c00fc10 comp rx=0 tx=0).stop 2024-03-18T20:26:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.171+0000 7fe06e67e700 1 -- 172.21.15.1:0/1134673122 shutdown_connections 2024-03-18T20:26:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.171+0000 7fe06e67e700 1 --2- 172.21.15.1:0/1134673122 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe05406aed0 0x7fe05406d390 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.171+0000 7fe06e67e700 1 --2- 172.21.15.1:0/1134673122 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe050003a00 0x7fe0680c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.171+0000 7fe06e67e700 1 --2- 172.21.15.1:0/1134673122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe06814eb70 0x7fe06806f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.171+0000 7fe06e67e700 1 --2- 172.21.15.1:0/1134673122 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0680cabf0 0x7fe068151ac0 unknown :-1 s=CLOSED pgs=2796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:26.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.172+0000 7fe06e67e700 1 -- 172.21.15.1:0/1134673122 >> 172.21.15.1:0/1134673122 conn(0x7fe0680b2ba0 msgr2=0x7fe06815c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:26.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.172+0000 7fe06e67e700 1 -- 172.21.15.1:0/1134673122 shutdown_connections 2024-03-18T20:26:26.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.172+0000 7fe06e67e700 1 -- 172.21.15.1:0/1134673122 wait complete. 2024-03-18T20:26:26.188 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:26:26.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.310+0000 7f492213c700 1 Processor -- start 2024-03-18T20:26:26.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.310+0000 7f492213c700 1 -- start start 2024-03-18T20:26:26.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.311+0000 7f492213c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f491c0b4110 0x7f491c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:26.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.311+0000 7f492213c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f491c06f070 0x7f491c0cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:26.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.311+0000 7f492213c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491c144000 con 0x7f491c143c20 2024-03-18T20:26:26.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.311+0000 7f492213c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f491c06f520 con 0x7f491c0b4110 2024-03-18T20:26:26.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.311+0000 7f492213c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f491c0b51b0 con 0x7f491c06f070 2024-03-18T20:26:26.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.311+0000 7f491affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f491c0b4110 0x7f491c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:26.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.311+0000 7f491b7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f491c143c20 0x7f491c0b3a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53468/0 (socket says 172.21.15.1:53468) 2024-03-18T20:26:26.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.311+0000 7f491b7fe700 1 -- 172.21.15.1:0/63003455 learned_addr learned my addr 172.21.15.1:0/63003455 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:26.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.312+0000 7f491affd700 1 -- 172.21.15.1:0/63003455 >> v2:172.21.15.165:3300/0 conn(0x7f491c06f070 msgr2=0x7f491c0cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:26.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.312+0000 7f491affd700 1 --2- 172.21.15.1:0/63003455 >> v2:172.21.15.165:3300/0 conn(0x7f491c06f070 0x7f491c0cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:26.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.312+0000 7f491affd700 1 -- 172.21.15.1:0/63003455 >> v1:172.21.15.1:6789/0 conn(0x7f491c143c20 legacy=0x7f491c0b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:26.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.312+0000 7f491affd700 1 -- 172.21.15.1:0/63003455 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491c0cd8d0 con 0x7f491c0b4110 2024-03-18T20:26:26.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.312+0000 7f491affd700 1 --2- 172.21.15.1:0/63003455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f491c0b4110 0x7f491c06eb30 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f491c071410 tx=0x7f4910009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d0c3068224cddc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:26.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.312+0000 7f491a7fc700 1 -- 172.21.15.1:0/63003455 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491000f8a0 con 0x7f491c0b4110 2024-03-18T20:26:26.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.312+0000 7f491a7fc700 1 -- 172.21.15.1:0/63003455 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491000fa00 con 0x7f491c0b4110 2024-03-18T20:26:26.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.312+0000 7f491a7fc700 1 -- 172.21.15.1:0/63003455 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491000fd90 con 0x7f491c0b4110 2024-03-18T20:26:26.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.313+0000 7f492213c700 1 -- 172.21.15.1:0/63003455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f491c0b4110 msgr2=0x7f491c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:26.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.313+0000 7f492213c700 1 --2- 172.21.15.1:0/63003455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f491c0b4110 0x7f491c06eb30 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f491c071410 tx=0x7f4910009b20 comp rx=0 tx=0).stop 2024-03-18T20:26:26.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.313+0000 7f492213c700 1 -- 172.21.15.1:0/63003455 shutdown_connections 2024-03-18T20:26:26.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.313+0000 7f492213c700 1 --2- 172.21.15.1:0/63003455 >> v2:172.21.15.165:3300/0 conn(0x7f491c06f070 0x7f491c0cddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:26.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.313+0000 7f492213c700 1 --2- 172.21.15.1:0/63003455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f491c0b4110 0x7f491c06eb30 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:26.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.313+0000 7f492213c700 1 -- 172.21.15.1:0/63003455 >> 172.21.15.1:0/63003455 conn(0x7f491c0b1d20 msgr2=0x7f491c0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:26.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.313+0000 7f492213c700 1 -- 172.21.15.1:0/63003455 shutdown_connections 2024-03-18T20:26:26.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.313+0000 7f492213c700 1 -- 172.21.15.1:0/63003455 wait complete. 2024-03-18T20:26:26.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.314+0000 7f492213c700 1 Processor -- start 2024-03-18T20:26:26.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.314+0000 7f492213c700 1 -- start start 2024-03-18T20:26:26.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.315+0000 7f492213c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f491c06f070 0x7f491c0ce670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:26.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.315+0000 7f492213c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f491c143c20 0x7f491c13d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:26.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.315+0000 7f492213c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f491c13dd70 0x7f491c0c56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:26.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.315+0000 7f492213c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491c1e2700 con 0x7f491c13dd70 2024-03-18T20:26:26.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.315+0000 7f492213c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491c13e1f0 con 0x7f491c143c20 2024-03-18T20:26:26.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.315+0000 7f492213c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f491c1e29a0 con 0x7f491c06f070 2024-03-18T20:26:26.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.315+0000 7f491bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f491c13dd70 0x7f491c0c56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:26.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.315+0000 7f491bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f491c13dd70 0x7f491c0c56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46518/0 (socket says 172.21.15.1:46518) 2024-03-18T20:26:26.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.315+0000 7f491bfff700 1 -- 172.21.15.1:0/2392785074 learned_addr learned my addr 172.21.15.1:0/2392785074 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:26.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.315+0000 7f491affd700 1 --2- 172.21.15.1:0/2392785074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f491c143c20 0x7f491c13d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:26.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.315+0000 7f491b7fe700 1 --2- 172.21.15.1:0/2392785074 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f491c06f070 0x7f491c0ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:26.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.315+0000 7f491bfff700 1 -- 172.21.15.1:0/2392785074 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f491c06f070 msgr2=0x7f491c0ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.315+0000 7f491bfff700 1 --2- 172.21.15.1:0/2392785074 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f491c06f070 0x7f491c0ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.315+0000 7f491bfff700 1 -- 172.21.15.1:0/2392785074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f491c143c20 msgr2=0x7f491c13d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.315+0000 7f491bfff700 1 --2- 172.21.15.1:0/2392785074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f491c143c20 0x7f491c13d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.315+0000 7f491bfff700 1 -- 172.21.15.1:0/2392785074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49100094a0 con 0x7f491c13dd70 2024-03-18T20:26:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.315+0000 7f491affd700 1 --2- 172.21.15.1:0/2392785074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f491c143c20 0x7f491c13d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.315+0000 7f491bfff700 1 --2- 172.21.15.1:0/2392785074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f491c13dd70 0x7f491c0c56f0 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f491400b970 tx=0x7f491400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.316+0000 7f4903fff700 1 -- 172.21.15.1:0/2392785074 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491400c9b0 con 0x7f491c13dd70 2024-03-18T20:26:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.316+0000 7f4903fff700 1 -- 172.21.15.1:0/2392785074 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491400cb30 con 0x7f491c13dd70 2024-03-18T20:26:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.316+0000 7f492213c700 1 -- 172.21.15.1:0/2392785074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f491c1e2c80 con 0x7f491c13dd70 2024-03-18T20:26:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.316+0000 7f4903fff700 1 -- 172.21.15.1:0/2392785074 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4914014440 con 0x7f491c13dd70 2024-03-18T20:26:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.316+0000 7f492213c700 1 -- 172.21.15.1:0/2392785074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f491c1e3240 con 0x7f491c13dd70 2024-03-18T20:26:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.317+0000 7f492213c700 1 -- 172.21.15.1:0/2392785074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f491c0676a0 con 0x7f491c13dd70 2024-03-18T20:26:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.317+0000 7f4903fff700 1 -- 172.21.15.1:0/2392785074 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f491401d3b0 con 0x7f491c13dd70 2024-03-18T20:26:26.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.317+0000 7f4903fff700 1 --2- 172.21.15.1:0/2392785074 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f490806aed0 0x7f490806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:26.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.318+0000 7f4903fff700 1 -- 172.21.15.1:0/2392785074 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f4914006e40 con 0x7f491c13dd70 2024-03-18T20:26:26.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.318+0000 7f491b7fe700 1 --2- 172.21.15.1:0/2392785074 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f490806aed0 0x7f490806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:26.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.318+0000 7f491b7fe700 1 --2- 172.21.15.1:0/2392785074 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f490806aed0 0x7f490806d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f490c005910 tx=0x7f490c009040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:26.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.322+0000 7f4903fff700 1 -- 172.21.15.1:0/2392785074 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4914054370 con 0x7f491c13dd70 2024-03-18T20:26:26.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.528+0000 7f492213c700 1 -- 172.21.15.1:0/2392785074 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f491c071f90 con 0x7f491c13dd70 2024-03-18T20:26:26.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.530+0000 7f4903fff700 1 -- 172.21.15.1:0/2392785074 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) v1 ==== 74+0+16442 (secure 0 0 0) 0x7f4914019090 con 0x7f491c13dd70 2024-03-18T20:26:26.534 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:26.535 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":252,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:26:25.784752+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:26:23.615691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-18T20:26:24.759650+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:26:26.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.532+0000 7f492213c700 1 -- 172.21.15.1:0/2392785074 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f490806aed0 msgr2=0x7f490806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:26.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.532+0000 7f492213c700 1 --2- 172.21.15.1:0/2392785074 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f490806aed0 0x7f490806d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f490c005910 tx=0x7f490c009040 comp rx=0 tx=0).stop 2024-03-18T20:26:26.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.532+0000 7f492213c700 1 -- 172.21.15.1:0/2392785074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f491c13dd70 msgr2=0x7f491c0c56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.532+0000 7f492213c700 1 --2- 172.21.15.1:0/2392785074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f491c13dd70 0x7f491c0c56f0 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f491400b970 tx=0x7f491400bd30 comp rx=0 tx=0).stop 2024-03-18T20:26:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.532+0000 7f492213c700 1 -- 172.21.15.1:0/2392785074 shutdown_connections 2024-03-18T20:26:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.532+0000 7f492213c700 1 --2- 172.21.15.1:0/2392785074 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f490806aed0 0x7f490806d390 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.532+0000 7f492213c700 1 --2- 172.21.15.1:0/2392785074 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f491c06f070 0x7f491c0ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.532+0000 7f492213c700 1 --2- 172.21.15.1:0/2392785074 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f491c13dd70 0x7f491c0c56f0 unknown :-1 s=CLOSED pgs=2799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.532+0000 7f492213c700 1 --2- 172.21.15.1:0/2392785074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f491c143c20 0x7f491c13d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.532+0000 7f492213c700 1 -- 172.21.15.1:0/2392785074 >> 172.21.15.1:0/2392785074 conn(0x7f491c0b1d20 msgr2=0x7f491c142f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.532+0000 7f492213c700 1 -- 172.21.15.1:0/2392785074 shutdown_connections 2024-03-18T20:26:26.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.532+0000 7f492213c700 1 -- 172.21.15.1:0/2392785074 wait complete. 2024-03-18T20:26:26.549 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T20:26:26.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.674+0000 7f8c1038a700 1 Processor -- start 2024-03-18T20:26:26.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.674+0000 7f8c1038a700 1 -- start start 2024-03-18T20:26:26.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.675+0000 7f8c1038a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8c0806ca80 0x7f8c0815d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:26.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.675+0000 7f8c1038a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8c0809a150 0x7f8c080cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:26.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.675+0000 7f8c1038a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c080b3b00 con 0x7f8c0815d5b0 2024-03-18T20:26:26.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.675+0000 7f8c1038a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c080cbbb0 con 0x7f8c0806ca80 2024-03-18T20:26:26.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.675+0000 7f8c1038a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c080cf7f0 con 0x7f8c0809a150 2024-03-18T20:26:26.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.675+0000 7f8c0d925700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8c0806ca80 0x7f8c0815d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:26.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.675+0000 7f8c0e126700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8c0815d5b0 0x7f8c08099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53480/0 (socket says 172.21.15.1:53480) 2024-03-18T20:26:26.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.675+0000 7f8c0d925700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8c0806ca80 0x7f8c0815d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46530/0 (socket says 172.21.15.1:46530) 2024-03-18T20:26:26.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.675+0000 7f8c0e126700 1 -- 172.21.15.1:0/351178719 learned_addr learned my addr 172.21.15.1:0/351178719 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:26.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.675+0000 7f8c0e927700 1 --2- 172.21.15.1:0/351178719 >> v2:172.21.15.165:3300/0 conn(0x7f8c0809a150 0x7f8c080cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:26.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.675+0000 7f8c0d925700 1 -- 172.21.15.1:0/351178719 >> v2:172.21.15.165:3300/0 conn(0x7f8c0809a150 msgr2=0x7f8c080cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:26.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.675+0000 7f8c0d925700 1 --2- 172.21.15.1:0/351178719 >> v2:172.21.15.165:3300/0 conn(0x7f8c0809a150 0x7f8c080cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:26.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.675+0000 7f8c0d925700 1 -- 172.21.15.1:0/351178719 >> v1:172.21.15.1:6789/0 conn(0x7f8c0815d5b0 legacy=0x7f8c08099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:26.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.675+0000 7f8c0d925700 1 -- 172.21.15.1:0/351178719 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c080d04d0 con 0x7f8c0806ca80 2024-03-18T20:26:26.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.675+0000 7f8c0d124700 1 -- 172.21.15.1:0/351178719 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129405670 0 0) 0x7f8c080b3b00 con 0x7f8c0815d5b0 2024-03-18T20:26:26.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.675+0000 7f8c0e927700 1 --2- 172.21.15.1:0/351178719 >> v2:172.21.15.165:3300/0 conn(0x7f8c0809a150 0x7f8c080cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:26.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.675+0000 7f8c0d925700 1 --2- 172.21.15.1:0/351178719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c0806ca80 0x7f8c0815d990 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7f8c080670f0 tx=0x7f8c04009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78840a3a67912cc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:26.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.676+0000 7f8c0d124700 1 -- 172.21.15.1:0/351178719 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04011830 con 0x7f8c0806ca80 2024-03-18T20:26:26.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.676+0000 7f8c0d124700 1 -- 172.21.15.1:0/351178719 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c04011990 con 0x7f8c0806ca80 2024-03-18T20:26:26.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.676+0000 7f8c0d124700 1 -- 172.21.15.1:0/351178719 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04011d20 con 0x7f8c0806ca80 2024-03-18T20:26:26.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.676+0000 7f8c1038a700 1 -- 172.21.15.1:0/351178719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c0806ca80 msgr2=0x7f8c0815d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:26.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.676+0000 7f8c1038a700 1 --2- 172.21.15.1:0/351178719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c0806ca80 0x7f8c0815d990 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7f8c080670f0 tx=0x7f8c04009b20 comp rx=0 tx=0).stop 2024-03-18T20:26:26.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.676+0000 7f8c1038a700 1 -- 172.21.15.1:0/351178719 shutdown_connections 2024-03-18T20:26:26.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.676+0000 7f8c1038a700 1 --2- 172.21.15.1:0/351178719 >> v2:172.21.15.165:3300/0 conn(0x7f8c0809a150 0x7f8c080cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:26.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.676+0000 7f8c1038a700 1 --2- 172.21.15.1:0/351178719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c0806ca80 0x7f8c0815d990 unknown :-1 s=CLOSED pgs=2801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:26.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.676+0000 7f8c1038a700 1 -- 172.21.15.1:0/351178719 >> 172.21.15.1:0/351178719 conn(0x7f8c080b2bd0 msgr2=0x7f8c080b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:26.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.676+0000 7f8c1038a700 1 -- 172.21.15.1:0/351178719 shutdown_connections 2024-03-18T20:26:26.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.676+0000 7f8c1038a700 1 -- 172.21.15.1:0/351178719 wait complete. 2024-03-18T20:26:26.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.677+0000 7f8c1038a700 1 Processor -- start 2024-03-18T20:26:26.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.677+0000 7f8c1038a700 1 -- start start 2024-03-18T20:26:26.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.678+0000 7f8c1038a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c0806ca80 0x7f8c08148d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:26.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.678+0000 7f8c1038a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c0809a150 0x7f8c081f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:26.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.678+0000 7f8c1038a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c0815d5b0 0x7f8c081497d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:26.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.678+0000 7f8c1038a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c08159790 con 0x7f8c0815d5b0 2024-03-18T20:26:26.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.678+0000 7f8c1038a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c081473f0 con 0x7f8c0806ca80 2024-03-18T20:26:26.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.678+0000 7f8c1038a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c081492c0 con 0x7f8c0809a150 2024-03-18T20:26:26.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.678+0000 7f8c0e927700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c0815d5b0 0x7f8c081497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:26.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.678+0000 7f8c0e927700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c0815d5b0 0x7f8c081497d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46532/0 (socket says 172.21.15.1:46532) 2024-03-18T20:26:26.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.678+0000 7f8c0e927700 1 -- 172.21.15.1:0/1206844616 learned_addr learned my addr 172.21.15.1:0/1206844616 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:26.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.678+0000 7f8c0e126700 1 --2- 172.21.15.1:0/1206844616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c0806ca80 0x7f8c08148d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:26.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.679+0000 7f8c0d925700 1 --2- 172.21.15.1:0/1206844616 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c0809a150 0x7f8c081f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:26.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.679+0000 7f8c0e927700 1 -- 172.21.15.1:0/1206844616 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c0809a150 msgr2=0x7f8c081f27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:26.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.679+0000 7f8c0e927700 1 --2- 172.21.15.1:0/1206844616 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c0809a150 0x7f8c081f27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:26.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.679+0000 7f8c0e927700 1 -- 172.21.15.1:0/1206844616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c0806ca80 msgr2=0x7f8c08148d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:26.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.679+0000 7f8c0e927700 1 --2- 172.21.15.1:0/1206844616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c0806ca80 0x7f8c08148d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:26.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.679+0000 7f8c0e927700 1 -- 172.21.15.1:0/1206844616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c040094a0 con 0x7f8c0815d5b0 2024-03-18T20:26:26.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.679+0000 7f8c0d925700 1 --2- 172.21.15.1:0/1206844616 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c0809a150 0x7f8c081f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:26.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.679+0000 7f8c0e126700 1 --2- 172.21.15.1:0/1206844616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c0806ca80 0x7f8c08148d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:26.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.679+0000 7f8c0e927700 1 --2- 172.21.15.1:0/1206844616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c0815d5b0 0x7f8c081497d0 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000cd50 tx=0x7f8c0000a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:26.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.679+0000 7f8bfeffd700 1 -- 172.21.15.1:0/1206844616 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0000f480 con 0x7f8c0815d5b0 2024-03-18T20:26:26.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.679+0000 7f8bfeffd700 1 -- 172.21.15.1:0/1206844616 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0000f600 con 0x7f8c0815d5b0 2024-03-18T20:26:26.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.679+0000 7f8c1038a700 1 -- 172.21.15.1:0/1206844616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c081467e0 con 0x7f8c0815d5b0 2024-03-18T20:26:26.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.679+0000 7f8bfeffd700 1 -- 172.21.15.1:0/1206844616 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0000f9b0 con 0x7f8c0815d5b0 2024-03-18T20:26:26.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.679+0000 7f8c1038a700 1 -- 172.21.15.1:0/1206844616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c08146da0 con 0x7f8c0815d5b0 2024-03-18T20:26:26.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.680+0000 7f8c1038a700 1 -- 172.21.15.1:0/1206844616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c080676a0 con 0x7f8c0815d5b0 2024-03-18T20:26:26.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.680+0000 7f8bfeffd700 1 -- 172.21.15.1:0/1206844616 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f8c0000b040 con 0x7f8c0815d5b0 2024-03-18T20:26:26.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.681+0000 7f8bfeffd700 1 --2- 172.21.15.1:0/1206844616 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8bec06aed0 0x7f8bec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:26.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.681+0000 7f8bfeffd700 1 -- 172.21.15.1:0/1206844616 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f8c0008a190 con 0x7f8c0815d5b0 2024-03-18T20:26:26.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.681+0000 7f8c0e126700 1 --2- 172.21.15.1:0/1206844616 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8bec06aed0 0x7f8bec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:26.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.681+0000 7f8c0e126700 1 --2- 172.21.15.1:0/1206844616 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8bec06aed0 0x7f8bec06d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8000f60 tx=0x7f8bf8009040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:26.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.685+0000 7f8bfeffd700 1 -- 172.21.15.1:0/1206844616 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8c00054de0 con 0x7f8c0815d5b0 2024-03-18T20:26:26.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:26.936+0000 7f8c1038a700 1 -- 172.21.15.1:0/1206844616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8c08147530 con 0x7f8c0815d5b0 2024-03-18T20:26:27.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.807+0000 7f8bfeffd700 1 -- 172.21.15.1:0/1206844616 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v119) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8c00058400 con 0x7f8c0815d5b0 2024-03-18T20:26:27.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.810+0000 7f8c1038a700 1 -- 172.21.15.1:0/1206844616 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8bec06aed0 msgr2=0x7f8bec06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.810+0000 7f8c1038a700 1 --2- 172.21.15.1:0/1206844616 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8bec06aed0 0x7f8bec06d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8000f60 tx=0x7f8bf8009040 comp rx=0 tx=0).stop 2024-03-18T20:26:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.810+0000 7f8c1038a700 1 -- 172.21.15.1:0/1206844616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c0815d5b0 msgr2=0x7f8c081497d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.810+0000 7f8c1038a700 1 --2- 172.21.15.1:0/1206844616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c0815d5b0 0x7f8c081497d0 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000cd50 tx=0x7f8c0000a560 comp rx=0 tx=0).stop 2024-03-18T20:26:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.810+0000 7f8c1038a700 1 -- 172.21.15.1:0/1206844616 shutdown_connections 2024-03-18T20:26:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.810+0000 7f8c1038a700 1 --2- 172.21.15.1:0/1206844616 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8bec06aed0 0x7f8bec06d390 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.810+0000 7f8c1038a700 1 --2- 172.21.15.1:0/1206844616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c0815d5b0 0x7f8c081497d0 unknown :-1 s=CLOSED pgs=2802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.810+0000 7f8c1038a700 1 --2- 172.21.15.1:0/1206844616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c0806ca80 0x7f8c08148d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.810+0000 7f8c1038a700 1 --2- 172.21.15.1:0/1206844616 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c0809a150 0x7f8c081f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.811+0000 7f8c1038a700 1 -- 172.21.15.1:0/1206844616 >> 172.21.15.1:0/1206844616 conn(0x7f8c080b2bd0 msgr2=0x7f8c080b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.811+0000 7f8c1038a700 1 -- 172.21.15.1:0/1206844616 shutdown_connections 2024-03-18T20:26:27.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.811+0000 7f8c1038a700 1 -- 172.21.15.1:0/1206844616 wait complete. 2024-03-18T20:26:27.830 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-18T20:26:27.830 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T20:26:27.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.963+0000 7fd20147e700 1 Processor -- start 2024-03-18T20:26:27.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.963+0000 7fd20147e700 1 -- start start 2024-03-18T20:26:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd20147e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1fc0d1ec0 0x7fd1fc0b82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd20147e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1fc0ce1e0 con 0x7fd1fc15d3e0 2024-03-18T20:26:27.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd20147e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1fc0ce4c0 con 0x7fd1fc15d7c0 2024-03-18T20:26:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd20147e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1fc0cee60 con 0x7fd1fc0d1ec0 2024-03-18T20:26:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd1faffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd1fc15d3e0 0x7fd1fc0d0fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53492/0 (socket says 172.21.15.1:53492) 2024-03-18T20:26:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd1faffd700 1 -- 172.21.15.1:0/3999226000 learned_addr learned my addr 172.21.15.1:0/3999226000 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd1fb7fe700 1 --2- 172.21.15.1:0/3999226000 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1fc0d1ec0 0x7fd1fc0b82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd1f9ffb700 1 -- 172.21.15.1:0/3999226000 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1565393097 0 0) 0x7fd1fc0ce1e0 con 0x7fd1fc15d3e0 2024-03-18T20:26:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd1f9ffb700 1 -- 172.21.15.1:0/3999226000 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1e40035b0 con 0x7fd1fc15d3e0 2024-03-18T20:26:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd1f9ffb700 1 -- 172.21.15.1:0/3999226000 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2468926186 0 0) 0x7fd1e40035b0 con 0x7fd1fc15d3e0 2024-03-18T20:26:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd1f9ffb700 1 -- 172.21.15.1:0/3999226000 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1fc0d1ec0 msgr2=0x7fd1fc0b82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd1f9ffb700 1 --2- 172.21.15.1:0/3999226000 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1fc0d1ec0 0x7fd1fc0b82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd1f9ffb700 1 -- 172.21.15.1:0/3999226000 >> v1:172.21.15.165:6789/0 conn(0x7fd1fc15d7c0 legacy=0x7fd1fc0d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd1f9ffb700 1 -- 172.21.15.1:0/3999226000 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1fc0ce7a0 con 0x7fd1fc15d3e0 2024-03-18T20:26:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd1f9ffb700 1 -- 172.21.15.1:0/3999226000 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fd1ec001670 con 0x7fd1fc15d3e0 2024-03-18T20:26:27.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd1f9ffb700 1 -- 172.21.15.1:0/3999226000 >> v1:172.21.15.1:6789/0 conn(0x7fd1fc15d3e0 legacy=0x7fd1fc0d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:27.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd1f9ffb700 1 --2- 172.21.15.1:0/3999226000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1e4003a30 0x7fd1e4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:27.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd1f9ffb700 1 --2- 172.21.15.1:0/3999226000 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1fc0d1ec0 0x7fd1e4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:27.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd1f9ffb700 1 --2- 172.21.15.1:0/3999226000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1e4004d90 0x7fd1e4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:27.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd1f9ffb700 1 -- 172.21.15.1:0/3999226000 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1fc0cee60 con 0x7fd1e4003a30 2024-03-18T20:26:27.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd1f9ffb700 1 -- 172.21.15.1:0/3999226000 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e400baf0 con 0x7fd1e4004d90 2024-03-18T20:26:27.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.964+0000 7fd1f9ffb700 1 -- 172.21.15.1:0/3999226000 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e400bcb0 con 0x7fd1fc0d1ec0 2024-03-18T20:26:27.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.965+0000 7fd1faffd700 1 --2- 172.21.15.1:0/3999226000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1e4003a30 0x7fd1e4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:27.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.965+0000 7fd1fa7fc700 1 --2- 172.21.15.1:0/3999226000 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1fc0d1ec0 0x7fd1e4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:27.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.965+0000 7fd1fb7fe700 1 --2- 172.21.15.1:0/3999226000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1e4004d90 0x7fd1e4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:27.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.965+0000 7fd1faffd700 1 -- 172.21.15.1:0/3999226000 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1fc0d1ec0 msgr2=0x7fd1e4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:27.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.965+0000 7fd1faffd700 1 --2- 172.21.15.1:0/3999226000 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1fc0d1ec0 0x7fd1e4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:27.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.965+0000 7fd1faffd700 1 -- 172.21.15.1:0/3999226000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1e4004d90 msgr2=0x7fd1e4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:27.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.965+0000 7fd1faffd700 1 --2- 172.21.15.1:0/3999226000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1e4004d90 0x7fd1e4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:27.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.965+0000 7fd1faffd700 1 -- 172.21.15.1:0/3999226000 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e40035b0 con 0x7fd1e4003a30 2024-03-18T20:26:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.965+0000 7fd1faffd700 1 --2- 172.21.15.1:0/3999226000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1e4003a30 0x7fd1e4003e50 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00c2a0 tx=0x7fd1e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c1dedd1115961f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.965+0000 7fd1f9ffb700 1 -- 172.21.15.1:0/3999226000 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec007c10 con 0x7fd1e4003a30 2024-03-18T20:26:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.965+0000 7fd1f9ffb700 1 -- 172.21.15.1:0/3999226000 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1ec00cde0 con 0x7fd1e4003a30 2024-03-18T20:26:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.965+0000 7fd1f9ffb700 1 -- 172.21.15.1:0/3999226000 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec005ba0 con 0x7fd1e4003a30 2024-03-18T20:26:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.966+0000 7fd20147e700 1 -- 172.21.15.1:0/3999226000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1e4003a30 msgr2=0x7fd1e4003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.966+0000 7fd20147e700 1 --2- 172.21.15.1:0/3999226000 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1e4003a30 0x7fd1e4003e50 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00c2a0 tx=0x7fd1e4005880 comp rx=0 tx=0).stop 2024-03-18T20:26:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.966+0000 7fd1fb7fe700 1 -- 172.21.15.1:0/3999226000 reap_dead start 2024-03-18T20:26:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.966+0000 7fd20147e700 1 -- 172.21.15.1:0/3999226000 shutdown_connections 2024-03-18T20:26:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.966+0000 7fd20147e700 1 -- 172.21.15.1:0/3999226000 >> 172.21.15.1:0/3999226000 conn(0x7fd1fc09c540 msgr2=0x7fd1fc0d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.966+0000 7fd20147e700 1 -- 172.21.15.1:0/3999226000 shutdown_connections 2024-03-18T20:26:27.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.966+0000 7fd20147e700 1 -- 172.21.15.1:0/3999226000 wait complete. 2024-03-18T20:26:27.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.967+0000 7fd20147e700 1 Processor -- start 2024-03-18T20:26:27.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.967+0000 7fd20147e700 1 -- start start 2024-03-18T20:26:27.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.968+0000 7fd20147e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1e4003a30 0x7fd1fc0ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:27.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.968+0000 7fd20147e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1fc1f6330 0x7fd1fc15f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:27.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.968+0000 7fd20147e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1fc1f67b0 0x7fd1fc15fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:27.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.968+0000 7fd20147e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1fc1f2d30 con 0x7fd1fc1f67b0 2024-03-18T20:26:27.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.968+0000 7fd20147e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1fc0cd040 con 0x7fd1fc1f6330 2024-03-18T20:26:27.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.968+0000 7fd20147e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1fc0cb6b0 con 0x7fd1e4003a30 2024-03-18T20:26:27.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.968+0000 7fd1fb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1fc1f67b0 0x7fd1fc15fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:27.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.968+0000 7fd1fb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1fc1f67b0 0x7fd1fc15fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46552/0 (socket says 172.21.15.1:46552) 2024-03-18T20:26:27.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.968+0000 7fd1fb7fe700 1 -- 172.21.15.1:0/2659299158 learned_addr learned my addr 172.21.15.1:0/2659299158 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:27.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.968+0000 7fd1faffd700 1 --2- 172.21.15.1:0/2659299158 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1e4003a30 0x7fd1fc0ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:27.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.968+0000 7fd1fa7fc700 1 --2- 172.21.15.1:0/2659299158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1fc1f6330 0x7fd1fc15f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:27.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.968+0000 7fd1fb7fe700 1 -- 172.21.15.1:0/2659299158 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1e4003a30 msgr2=0x7fd1fc0ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:27.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.968+0000 7fd1fb7fe700 1 --2- 172.21.15.1:0/2659299158 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1e4003a30 0x7fd1fc0ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:27.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.968+0000 7fd1fb7fe700 1 -- 172.21.15.1:0/2659299158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1fc1f6330 msgr2=0x7fd1fc15f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:27.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.968+0000 7fd1fb7fe700 1 --2- 172.21.15.1:0/2659299158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1fc1f6330 0x7fd1fc15f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:27.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.968+0000 7fd1fb7fe700 1 -- 172.21.15.1:0/2659299158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1fc0cb990 con 0x7fd1fc1f67b0 2024-03-18T20:26:27.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.968+0000 7fd1fa7fc700 1 --2- 172.21.15.1:0/2659299158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1fc1f6330 0x7fd1fc15f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:27.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.968+0000 7fd1faffd700 1 --2- 172.21.15.1:0/2659299158 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1e4003a30 0x7fd1fc0ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:27.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.968+0000 7fd1fb7fe700 1 --2- 172.21.15.1:0/2659299158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1fc1f67b0 0x7fd1fc15fec0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc072c80 tx=0x7fd1f40114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:27.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.969+0000 7fd1e37fe700 1 -- 172.21.15.1:0/2659299158 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f400e070 con 0x7fd1fc1f67b0 2024-03-18T20:26:27.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.969+0000 7fd1e37fe700 1 -- 172.21.15.1:0/2659299158 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f4003c30 con 0x7fd1fc1f67b0 2024-03-18T20:26:27.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.969+0000 7fd20147e700 1 -- 172.21.15.1:0/2659299158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1fc0cbc70 con 0x7fd1fc1f67b0 2024-03-18T20:26:27.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.969+0000 7fd1e37fe700 1 -- 172.21.15.1:0/2659299158 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4003e20 con 0x7fd1fc1f67b0 2024-03-18T20:26:27.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.969+0000 7fd20147e700 1 -- 172.21.15.1:0/2659299158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1fc0cb2f0 con 0x7fd1fc1f67b0 2024-03-18T20:26:27.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.970+0000 7fd20147e700 1 -- 172.21.15.1:0/2659299158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1fc1f2ed0 con 0x7fd1fc1f67b0 2024-03-18T20:26:27.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.970+0000 7fd1e37fe700 1 -- 172.21.15.1:0/2659299158 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fd1f4011a10 con 0x7fd1fc1f67b0 2024-03-18T20:26:27.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.971+0000 7fd1e37fe700 1 --2- 172.21.15.1:0/2659299158 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd1e806aed0 0x7fd1e806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:27.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.971+0000 7fd1e37fe700 1 -- 172.21.15.1:0/2659299158 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fd1f4089cf0 con 0x7fd1fc1f67b0 2024-03-18T20:26:27.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.971+0000 7fd1faffd700 1 --2- 172.21.15.1:0/2659299158 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd1e806aed0 0x7fd1e806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:27.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.971+0000 7fd1faffd700 1 --2- 172.21.15.1:0/2659299158 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd1e806aed0 0x7fd1e806d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec015040 tx=0x7fd1ec007000 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:27.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:27.975+0000 7fd1e37fe700 1 -- 172.21.15.1:0/2659299158 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd1fc1f2ed0 con 0x7fd1fc1f67b0 2024-03-18T20:26:28.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:28.230+0000 7fd20147e700 1 -- 172.21.15.1:0/2659299158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd1fc1f2ed0 con 0x7fd1fc1f67b0 2024-03-18T20:26:29.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.104+0000 7fd1e37fe700 1 -- 172.21.15.1:0/2659299158 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v120) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd1fc1f2ed0 con 0x7fd1fc1f67b0 2024-03-18T20:26:29.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.107+0000 7fd20147e700 1 -- 172.21.15.1:0/2659299158 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd1e806aed0 msgr2=0x7fd1e806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:29.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.107+0000 7fd20147e700 1 --2- 172.21.15.1:0/2659299158 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd1e806aed0 0x7fd1e806d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec015040 tx=0x7fd1ec007000 comp rx=0 tx=0).stop 2024-03-18T20:26:29.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.107+0000 7fd20147e700 1 -- 172.21.15.1:0/2659299158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1fc1f67b0 msgr2=0x7fd1fc15fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:29.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.107+0000 7fd20147e700 1 --2- 172.21.15.1:0/2659299158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1fc1f67b0 0x7fd1fc15fec0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc072c80 tx=0x7fd1f40114b0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.107+0000 7fd20147e700 1 -- 172.21.15.1:0/2659299158 shutdown_connections 2024-03-18T20:26:29.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.107+0000 7fd20147e700 1 --2- 172.21.15.1:0/2659299158 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd1e806aed0 0x7fd1e806d390 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.107+0000 7fd20147e700 1 --2- 172.21.15.1:0/2659299158 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1e4003a30 0x7fd1fc0ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.107+0000 7fd20147e700 1 --2- 172.21.15.1:0/2659299158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1fc1f67b0 0x7fd1fc15fec0 unknown :-1 s=CLOSED pgs=2805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.107+0000 7fd20147e700 1 --2- 172.21.15.1:0/2659299158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1fc1f6330 0x7fd1fc15f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.107+0000 7fd20147e700 1 -- 172.21.15.1:0/2659299158 >> 172.21.15.1:0/2659299158 conn(0x7fd1fc09c540 msgr2=0x7fd1fc06b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:29.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.107+0000 7fd20147e700 1 -- 172.21.15.1:0/2659299158 shutdown_connections 2024-03-18T20:26:29.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.107+0000 7fd20147e700 1 -- 172.21.15.1:0/2659299158 wait complete. 2024-03-18T20:26:29.126 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T20:26:29.126 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:26:29.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.255+0000 7ff2b3144700 1 Processor -- start 2024-03-18T20:26:29.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.255+0000 7ff2b3144700 1 -- start start 2024-03-18T20:26:29.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.256+0000 7ff2b3144700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2ac15cfc0 0x7ff2ac06dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:29.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.256+0000 7ff2b3144700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ac071b50 con 0x7ff2ac0cbef0 2024-03-18T20:26:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.256+0000 7ff2b3144700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ac06e9c0 con 0x7ff2ac0cc370 2024-03-18T20:26:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.256+0000 7ff2b3144700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac06e1f0 con 0x7ff2ac15cfc0 2024-03-18T20:26:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.256+0000 7ff2abfff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff2ac0cbef0 0x7ff2ac070d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53500/0 (socket says 172.21.15.1:53500) 2024-03-18T20:26:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.256+0000 7ff2abfff700 1 -- 172.21.15.1:0/2105336303 learned_addr learned my addr 172.21.15.1:0/2105336303 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.256+0000 7ff2b0ee0700 1 --2- 172.21.15.1:0/2105336303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2ac15cfc0 0x7ff2ac06dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.256+0000 7ff2ab7fe700 1 -- 172.21.15.1:0/2105336303 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983713121 0 0) 0x7ff2ac071b50 con 0x7ff2ac0cbef0 2024-03-18T20:26:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.256+0000 7ff2ab7fe700 1 -- 172.21.15.1:0/2105336303 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff29c0035b0 con 0x7ff2ac0cbef0 2024-03-18T20:26:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.256+0000 7ff2ab7fe700 1 -- 172.21.15.1:0/2105336303 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597308487 0 0) 0x7ff2ac06e9c0 con 0x7ff2ac0cc370 2024-03-18T20:26:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.256+0000 7ff2ab7fe700 1 -- 172.21.15.1:0/2105336303 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2ac071b50 con 0x7ff2ac0cc370 2024-03-18T20:26:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.256+0000 7ff2ab7fe700 1 -- 172.21.15.1:0/2105336303 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1966795332 0 0) 0x7ff29c0035b0 con 0x7ff2ac0cbef0 2024-03-18T20:26:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.256+0000 7ff2ab7fe700 1 -- 172.21.15.1:0/2105336303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2ac15cfc0 msgr2=0x7ff2ac06dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.256+0000 7ff2ab7fe700 1 --2- 172.21.15.1:0/2105336303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2ac15cfc0 0x7ff2ac06dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.256+0000 7ff2ab7fe700 1 -- 172.21.15.1:0/2105336303 >> v1:172.21.15.165:6789/0 conn(0x7ff2ac0cc370 legacy=0x7ff2ac071440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.256+0000 7ff2ab7fe700 1 -- 172.21.15.1:0/2105336303 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ac0cc800 con 0x7ff2ac0cbef0 2024-03-18T20:26:29.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.256+0000 7ff2ab7fe700 1 -- 172.21.15.1:0/2105336303 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7ff298001670 con 0x7ff2ac0cbef0 2024-03-18T20:26:29.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.256+0000 7ff2b0ee0700 1 --2- 172.21.15.1:0/2105336303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2ac15cfc0 0x7ff2ac06dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:26:29.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2ab7fe700 1 -- 172.21.15.1:0/2105336303 >> v1:172.21.15.1:6789/0 conn(0x7ff2ac0cbef0 legacy=0x7ff2ac070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:29.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2ab7fe700 1 --2- 172.21.15.1:0/2105336303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff29c003bc0 0x7ff29c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:29.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2ab7fe700 1 --2- 172.21.15.1:0/2105336303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff29c004850 0x7ff29c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:29.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2ab7fe700 1 --2- 172.21.15.1:0/2105336303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2ac15cfc0 0x7ff29c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:29.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2ab7fe700 1 -- 172.21.15.1:0/2105336303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c00dcd0 con 0x7ff29c003bc0 2024-03-18T20:26:29.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2ab7fe700 1 -- 172.21.15.1:0/2105336303 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c00de10 con 0x7ff29c004850 2024-03-18T20:26:29.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2ab7fe700 1 -- 172.21.15.1:0/2105336303 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c00dfa0 con 0x7ff2ac15cfc0 2024-03-18T20:26:29.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2b0ee0700 1 --2- 172.21.15.1:0/2105336303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff29c003bc0 0x7ff29c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:29.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2abfff700 1 --2- 172.21.15.1:0/2105336303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff29c004850 0x7ff29c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:29.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2b16e1700 1 --2- 172.21.15.1:0/2105336303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2ac15cfc0 0x7ff29c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:29.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2b0ee0700 1 -- 172.21.15.1:0/2105336303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2ac15cfc0 msgr2=0x7ff29c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:29.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2b0ee0700 1 --2- 172.21.15.1:0/2105336303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2ac15cfc0 0x7ff29c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2b0ee0700 1 -- 172.21.15.1:0/2105336303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff29c004850 msgr2=0x7ff29c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:29.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2b0ee0700 1 --2- 172.21.15.1:0/2105336303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff29c004850 0x7ff29c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2b0ee0700 1 -- 172.21.15.1:0/2105336303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a00094a0 con 0x7ff29c003bc0 2024-03-18T20:26:29.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2b16e1700 1 --2- 172.21.15.1:0/2105336303 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff2ac15cfc0 0x7ff29c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:29.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2abfff700 1 --2- 172.21.15.1:0/2105336303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff29c004850 0x7ff29c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:29.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2b0ee0700 1 --2- 172.21.15.1:0/2105336303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff29c003bc0 0x7ff29c003fc0 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009d00 tx=0x7ff29c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6822a69fd7f5aa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:29.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2ab7fe700 1 -- 172.21.15.1:0/2105336303 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a0009e50 con 0x7ff29c003bc0 2024-03-18T20:26:29.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2ab7fe700 1 -- 172.21.15.1:0/2105336303 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2a000e9a0 con 0x7ff29c003bc0 2024-03-18T20:26:29.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.257+0000 7ff2ab7fe700 1 -- 172.21.15.1:0/2105336303 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a000ed30 con 0x7ff29c003bc0 2024-03-18T20:26:29.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.258+0000 7ff2b3144700 1 -- 172.21.15.1:0/2105336303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff29c003bc0 msgr2=0x7ff29c003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:29.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.258+0000 7ff2b3144700 1 --2- 172.21.15.1:0/2105336303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff29c003bc0 0x7ff29c003fc0 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009d00 tx=0x7ff29c005880 comp rx=0 tx=0).stop 2024-03-18T20:26:29.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.258+0000 7ff2b16e1700 1 -- 172.21.15.1:0/2105336303 reap_dead start 2024-03-18T20:26:29.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.258+0000 7ff2b3144700 1 -- 172.21.15.1:0/2105336303 shutdown_connections 2024-03-18T20:26:29.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.258+0000 7ff2b3144700 1 -- 172.21.15.1:0/2105336303 >> 172.21.15.1:0/2105336303 conn(0x7ff2ac0b1270 msgr2=0x7ff2ac0b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:29.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.258+0000 7ff2b3144700 1 -- 172.21.15.1:0/2105336303 shutdown_connections 2024-03-18T20:26:29.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.258+0000 7ff2b3144700 1 -- 172.21.15.1:0/2105336303 wait complete. 2024-03-18T20:26:29.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.259+0000 7ff2b3144700 1 Processor -- start 2024-03-18T20:26:29.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.259+0000 7ff2b3144700 1 -- start start 2024-03-18T20:26:29.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.260+0000 7ff2b3144700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff29c003bc0 0x7ff2ac151030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:29.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.260+0000 7ff2b3144700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2ac0c8bf0 0x7ff2ac0c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:29.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.260+0000 7ff2b3144700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2ac0c45d0 0x7ff2ac0c4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:29.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.260+0000 7ff2b3144700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac0c9640 con 0x7ff2ac0c45d0 2024-03-18T20:26:29.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.260+0000 7ff2b3144700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac151570 con 0x7ff2ac0c8bf0 2024-03-18T20:26:29.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.260+0000 7ff2b3144700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac0c6910 con 0x7ff29c003bc0 2024-03-18T20:26:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.260+0000 7ff2b16e1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2ac0c45d0 0x7ff2ac0c4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.260+0000 7ff2b16e1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2ac0c45d0 0x7ff2ac0c4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46560/0 (socket says 172.21.15.1:46560) 2024-03-18T20:26:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.260+0000 7ff2b16e1700 1 -- 172.21.15.1:0/2246277465 learned_addr learned my addr 172.21.15.1:0/2246277465 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.260+0000 7ff2b0ee0700 1 --2- 172.21.15.1:0/2246277465 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff29c003bc0 0x7ff2ac151030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.260+0000 7ff2abfff700 1 --2- 172.21.15.1:0/2246277465 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2ac0c8bf0 0x7ff2ac0c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.261+0000 7ff2b16e1700 1 -- 172.21.15.1:0/2246277465 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff29c003bc0 msgr2=0x7ff2ac151030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.261+0000 7ff2b16e1700 1 --2- 172.21.15.1:0/2246277465 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff29c003bc0 0x7ff2ac151030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.261+0000 7ff2b16e1700 1 -- 172.21.15.1:0/2246277465 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2ac0c8bf0 msgr2=0x7ff2ac0c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.261+0000 7ff2b16e1700 1 --2- 172.21.15.1:0/2246277465 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2ac0c8bf0 0x7ff2ac0c9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.261+0000 7ff2b16e1700 1 -- 172.21.15.1:0/2246277465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ac0c6bf0 con 0x7ff2ac0c45d0 2024-03-18T20:26:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.261+0000 7ff2b16e1700 1 --2- 172.21.15.1:0/2246277465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2ac0c45d0 0x7ff2ac0c4a60 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac0d0770 tx=0x7ff2a400ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:29.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.261+0000 7ff2a97fa700 1 -- 172.21.15.1:0/2246277465 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a4003c10 con 0x7ff2ac0c45d0 2024-03-18T20:26:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.261+0000 7ff2a97fa700 1 -- 172.21.15.1:0/2246277465 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2a4003d70 con 0x7ff2ac0c45d0 2024-03-18T20:26:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.261+0000 7ff2b3144700 1 -- 172.21.15.1:0/2246277465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2ac0c6ed0 con 0x7ff2ac0c45d0 2024-03-18T20:26:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.261+0000 7ff2a97fa700 1 -- 172.21.15.1:0/2246277465 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a40052f0 con 0x7ff2ac0c45d0 2024-03-18T20:26:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.261+0000 7ff2b3144700 1 -- 172.21.15.1:0/2246277465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2ac1f3390 con 0x7ff2ac0c45d0 2024-03-18T20:26:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.262+0000 7ff2b3144700 1 -- 172.21.15.1:0/2246277465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2ac0c73f0 con 0x7ff2ac0c45d0 2024-03-18T20:26:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.262+0000 7ff2a97fa700 1 -- 172.21.15.1:0/2246277465 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7ff2a4005450 con 0x7ff2ac0c45d0 2024-03-18T20:26:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.263+0000 7ff2a97fa700 1 --2- 172.21.15.1:0/2246277465 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff29006aed0 0x7ff29006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.263+0000 7ff2a97fa700 1 -- 172.21.15.1:0/2246277465 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 6129+0+0 (secure 0 0 0) 0x7ff2a40897f0 con 0x7ff2ac0c45d0 2024-03-18T20:26:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.263+0000 7ff2b0ee0700 1 --2- 172.21.15.1:0/2246277465 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff29006aed0 0x7ff29006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.263+0000 7ff2b0ee0700 1 --2- 172.21.15.1:0/2246277465 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff29006aed0 0x7ff29006d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff2a00090b0 tx=0x7ff2a001f040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:29.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.267+0000 7ff2a97fa700 1 -- 172.21.15.1:0/2246277465 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff2ac0c73f0 con 0x7ff2ac0c45d0 2024-03-18T20:26:29.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.516+0000 7ff2b3144700 1 -- 172.21.15.1:0/2246277465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2ac0c73f0 con 0x7ff2ac0c45d0 2024-03-18T20:26:29.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.516+0000 7ff2a97fa700 1 -- 172.21.15.1:0/2246277465 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+4747 (secure 0 0 0) 0x7ff2ac0c73f0 con 0x7ff2ac0c45d0 2024-03-18T20:26:29.520 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:29.520 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":120,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12396,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2565889517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2565889517},{"type":"v1","addr":"172.21.15.1:6836","nonce":2565889517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":12399,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/512665659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":512665659},{"type":"v1","addr":"172.21.15.1:6837","nonce":512665659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":12408,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1134181166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1134181166},{"type":"v1","addr":"172.21.15.165:6834","nonce":1134181166}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:26:25.785074+0000","modified":"2024-03-18T20:26:29.101123+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12411":{"gid":12411,"name":"d","rank":0,"incarnation":116,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/3886150679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":3886150679},{"type":"v1","addr":"172.21.15.165:6835","nonce":3886150679}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-03-18T20:26:29.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.518+0000 7ff2b3144700 1 -- 172.21.15.1:0/2246277465 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff29006aed0 msgr2=0x7ff29006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:29.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.518+0000 7ff2b3144700 1 --2- 172.21.15.1:0/2246277465 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff29006aed0 0x7ff29006d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff2a00090b0 tx=0x7ff2a001f040 comp rx=0 tx=0).stop 2024-03-18T20:26:29.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.518+0000 7ff2b3144700 1 -- 172.21.15.1:0/2246277465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2ac0c45d0 msgr2=0x7ff2ac0c4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:29.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.518+0000 7ff2b3144700 1 --2- 172.21.15.1:0/2246277465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2ac0c45d0 0x7ff2ac0c4a60 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac0d0770 tx=0x7ff2a400ef00 comp rx=0 tx=0).stop 2024-03-18T20:26:29.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.518+0000 7ff2b3144700 1 -- 172.21.15.1:0/2246277465 shutdown_connections 2024-03-18T20:26:29.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.518+0000 7ff2b3144700 1 --2- 172.21.15.1:0/2246277465 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff29006aed0 0x7ff29006d390 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.518+0000 7ff2b3144700 1 --2- 172.21.15.1:0/2246277465 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff29c003bc0 0x7ff2ac151030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.518+0000 7ff2b3144700 1 --2- 172.21.15.1:0/2246277465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2ac0c45d0 0x7ff2ac0c4a60 unknown :-1 s=CLOSED pgs=2808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.518+0000 7ff2b3144700 1 --2- 172.21.15.1:0/2246277465 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2ac0c8bf0 0x7ff2ac0c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.518+0000 7ff2b3144700 1 -- 172.21.15.1:0/2246277465 >> 172.21.15.1:0/2246277465 conn(0x7ff2ac0b1270 msgr2=0x7ff2ac158780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:29.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.519+0000 7ff2b3144700 1 -- 172.21.15.1:0/2246277465 shutdown_connections 2024-03-18T20:26:29.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.519+0000 7ff2b3144700 1 -- 172.21.15.1:0/2246277465 wait complete. 2024-03-18T20:26:29.524 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 120 2024-03-18T20:26:29.535 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:26:29.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.656+0000 7f52153d9700 1 Processor -- start 2024-03-18T20:26:29.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.656+0000 7f52153d9700 1 -- start start 2024-03-18T20:26:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.657+0000 7f52153d9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f521009a650 0x7f5210099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.657+0000 7f52153d9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5210099f80 0x7f521006f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.657+0000 7f52153d9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521006f070 con 0x7f521015b4d0 2024-03-18T20:26:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.657+0000 7f52153d9700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f521009a3f0 con 0x7f521009a650 2024-03-18T20:26:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.657+0000 7f52153d9700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52100d0260 con 0x7f5210099f80 2024-03-18T20:26:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.657+0000 7f520e7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f521009a650 0x7f5210099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.657+0000 7f520e7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f521009a650 0x7f5210099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46564/0 (socket says 172.21.15.1:46564) 2024-03-18T20:26:29.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.657+0000 7f520effd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f521015b4d0 0x7f52100cf750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53510/0 (socket says 172.21.15.1:53510) 2024-03-18T20:26:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.657+0000 7f520e7fc700 1 -- 172.21.15.1:0/729146446 learned_addr learned my addr 172.21.15.1:0/729146446 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.657+0000 7f520f7fe700 1 --2- 172.21.15.1:0/729146446 >> v2:172.21.15.165:3300/0 conn(0x7f5210099f80 0x7f521006f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.658+0000 7f520e7fc700 1 -- 172.21.15.1:0/729146446 >> v2:172.21.15.165:3300/0 conn(0x7f5210099f80 msgr2=0x7f521006f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.658+0000 7f520e7fc700 1 --2- 172.21.15.1:0/729146446 >> v2:172.21.15.165:3300/0 conn(0x7f5210099f80 0x7f521006f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.658+0000 7f520e7fc700 1 -- 172.21.15.1:0/729146446 >> v1:172.21.15.1:6789/0 conn(0x7f521015b4d0 legacy=0x7f52100cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.658+0000 7f520e7fc700 1 -- 172.21.15.1:0/729146446 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52100cff80 con 0x7f521009a650 2024-03-18T20:26:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.658+0000 7f520dffb700 1 -- 172.21.15.1:0/729146446 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712816979 0 0) 0x7f521006f070 con 0x7f521015b4d0 2024-03-18T20:26:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.658+0000 7f520f7fe700 1 --2- 172.21.15.1:0/729146446 >> v2:172.21.15.165:3300/0 conn(0x7f5210099f80 0x7f521006f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.658+0000 7f520e7fc700 1 --2- 172.21.15.1:0/729146446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f521009a650 0x7f5210099a40 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7f52100b2c40 tx=0x7f51f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68a927be503d6a85 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.658+0000 7f520dffb700 1 -- 172.21.15.1:0/729146446 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f800b870 con 0x7f521009a650 2024-03-18T20:26:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.658+0000 7f520dffb700 1 -- 172.21.15.1:0/729146446 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f800b9d0 con 0x7f521009a650 2024-03-18T20:26:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.658+0000 7f520dffb700 1 -- 172.21.15.1:0/729146446 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f800bd60 con 0x7f521009a650 2024-03-18T20:26:29.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.658+0000 7f52153d9700 1 -- 172.21.15.1:0/729146446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f521009a650 msgr2=0x7f5210099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:29.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.658+0000 7f52153d9700 1 --2- 172.21.15.1:0/729146446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f521009a650 0x7f5210099a40 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7f52100b2c40 tx=0x7f51f8009bd0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.658+0000 7f52153d9700 1 -- 172.21.15.1:0/729146446 shutdown_connections 2024-03-18T20:26:29.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.658+0000 7f52153d9700 1 --2- 172.21.15.1:0/729146446 >> v2:172.21.15.165:3300/0 conn(0x7f5210099f80 0x7f521006f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.658+0000 7f52153d9700 1 --2- 172.21.15.1:0/729146446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f521009a650 0x7f5210099a40 unknown :-1 s=CLOSED pgs=2810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.658+0000 7f52153d9700 1 -- 172.21.15.1:0/729146446 >> 172.21.15.1:0/729146446 conn(0x7f52100b2cb0 msgr2=0x7f52100b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:29.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.659+0000 7f52153d9700 1 -- 172.21.15.1:0/729146446 shutdown_connections 2024-03-18T20:26:29.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.659+0000 7f52153d9700 1 -- 172.21.15.1:0/729146446 wait complete. 2024-03-18T20:26:29.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.660+0000 7f52153d9700 1 Processor -- start 2024-03-18T20:26:29.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.660+0000 7f52153d9700 1 -- start start 2024-03-18T20:26:29.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f52153d9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5210099f80 0x7f521015e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:29.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f52153d9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f521009a650 0x7f5210158980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:29.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f52153d9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f521015b4d0 0x7f521015c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:29.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f52153d9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52100c6cb0 con 0x7f521009a650 2024-03-18T20:26:29.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f52153d9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52100c93a0 con 0x7f521015b4d0 2024-03-18T20:26:29.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f52153d9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52100c6eb0 con 0x7f5210099f80 2024-03-18T20:26:29.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f520e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f521009a650 0x7f5210158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:29.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f520e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f521009a650 0x7f5210158980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46572/0 (socket says 172.21.15.1:46572) 2024-03-18T20:26:29.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f520e7fc700 1 -- 172.21.15.1:0/426781791 learned_addr learned my addr 172.21.15.1:0/426781791 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:29.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f520e7fc700 1 -- 172.21.15.1:0/426781791 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5210099f80 msgr2=0x7f521015e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:29.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f520effd700 1 --2- 172.21.15.1:0/426781791 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5210099f80 0x7f521015e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:29.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f520f7fe700 1 --2- 172.21.15.1:0/426781791 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f521015b4d0 0x7f521015c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:29.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f520e7fc700 1 --2- 172.21.15.1:0/426781791 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5210099f80 0x7f521015e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f520e7fc700 1 -- 172.21.15.1:0/426781791 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f521015b4d0 msgr2=0x7f521015c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:29.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f520e7fc700 1 --2- 172.21.15.1:0/426781791 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f521015b4d0 0x7f521015c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f520e7fc700 1 -- 172.21.15.1:0/426781791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f8009550 con 0x7f521009a650 2024-03-18T20:26:29.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f520e7fc700 1 --2- 172.21.15.1:0/426781791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f521009a650 0x7f5210158980 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 crypto rx=0x7f51f8009c10 tx=0x7f51f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:29.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f52077fe700 1 -- 172.21.15.1:0/426781791 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8004250 con 0x7f521009a650 2024-03-18T20:26:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f52077fe700 1 -- 172.21.15.1:0/426781791 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f801f070 con 0x7f521009a650 2024-03-18T20:26:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f52077fe700 1 -- 172.21.15.1:0/426781791 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f80045b0 con 0x7f521009a650 2024-03-18T20:26:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f52153d9700 1 -- 172.21.15.1:0/426781791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52100c7190 con 0x7f521009a650 2024-03-18T20:26:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.661+0000 7f52153d9700 1 -- 172.21.15.1:0/426781791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52100c7750 con 0x7f521009a650 2024-03-18T20:26:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.662+0000 7f52153d9700 1 -- 172.21.15.1:0/426781791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5210068240 con 0x7f521009a650 2024-03-18T20:26:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.663+0000 7f52077fe700 1 -- 172.21.15.1:0/426781791 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f51f8021030 con 0x7f521009a650 2024-03-18T20:26:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.663+0000 7f52077fe700 1 --2- 172.21.15.1:0/426781791 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f51f006aed0 0x7f51f006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.664+0000 7f520effd700 1 --2- 172.21.15.1:0/426781791 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f51f006aed0 0x7f51f006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.664+0000 7f52077fe700 1 -- 172.21.15.1:0/426781791 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f51f808ac50 con 0x7f521009a650 2024-03-18T20:26:29.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.664+0000 7f520effd700 1 --2- 172.21.15.1:0/426781791 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f51f006aed0 0x7f51f006d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f52000012e0 tx=0x7f5200009040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:29.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.668+0000 7f52077fe700 1 -- 172.21.15.1:0/426781791 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f51f8090020 con 0x7f521009a650 2024-03-18T20:26:29.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.873+0000 7f52153d9700 1 -- 172.21.15.1:0/426781791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52100c95c0 con 0x7f521009a650 2024-03-18T20:26:29.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.874+0000 7f52077fe700 1 -- 172.21.15.1:0/426781791 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v253) v1 ==== 74+0+16429 (secure 0 0 0) 0x7f52100c95c0 con 0x7f521009a650 2024-03-18T20:26:29.878 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:29.878 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":253,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:26:26.786793+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:26:23.615691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-18T20:26:24.759650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:26:29.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.876+0000 7f52153d9700 1 -- 172.21.15.1:0/426781791 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f51f006aed0 msgr2=0x7f51f006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:29.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.876+0000 7f52153d9700 1 --2- 172.21.15.1:0/426781791 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f51f006aed0 0x7f51f006d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f52000012e0 tx=0x7f5200009040 comp rx=0 tx=0).stop 2024-03-18T20:26:29.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.876+0000 7f52153d9700 1 -- 172.21.15.1:0/426781791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f521009a650 msgr2=0x7f5210158980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:29.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.876+0000 7f52153d9700 1 --2- 172.21.15.1:0/426781791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f521009a650 0x7f5210158980 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 crypto rx=0x7f51f8009c10 tx=0x7f51f8004750 comp rx=0 tx=0).stop 2024-03-18T20:26:29.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.876+0000 7f52153d9700 1 -- 172.21.15.1:0/426781791 shutdown_connections 2024-03-18T20:26:29.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.876+0000 7f52153d9700 1 --2- 172.21.15.1:0/426781791 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f51f006aed0 0x7f51f006d390 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.876+0000 7f52153d9700 1 --2- 172.21.15.1:0/426781791 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5210099f80 0x7f521015e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.876+0000 7f52153d9700 1 --2- 172.21.15.1:0/426781791 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f521015b4d0 0x7f521015c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.876+0000 7f52153d9700 1 --2- 172.21.15.1:0/426781791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f521009a650 0x7f5210158980 unknown :-1 s=CLOSED pgs=2811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:29.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.876+0000 7f52153d9700 1 -- 172.21.15.1:0/426781791 >> 172.21.15.1:0/426781791 conn(0x7f52100b2cb0 msgr2=0x7f52100b15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:29.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.876+0000 7f52153d9700 1 -- 172.21.15.1:0/426781791 shutdown_connections 2024-03-18T20:26:29.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:29.876+0000 7f52153d9700 1 -- 172.21.15.1:0/426781791 wait complete. 2024-03-18T20:26:29.893 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T20:26:29.893 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:26:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.015+0000 7f7300215700 1 Processor -- start 2024-03-18T20:26:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.015+0000 7f7300215700 1 -- start start 2024-03-18T20:26:30.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.016+0000 7f7300215700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f72f81465a0 0x7f72f8168660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.016+0000 7f7300215700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f8072660 con 0x7f72f8146190 2024-03-18T20:26:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.016+0000 7f7300215700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f806ebd0 con 0x7f72f806ae60 2024-03-18T20:26:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.016+0000 7f7300215700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f806f850 con 0x7f72f81465a0 2024-03-18T20:26:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.016+0000 7f72fdfb1700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f72f8146190 0x7f72f806a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53524/0 (socket says 172.21.15.1:53524) 2024-03-18T20:26:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.016+0000 7f72fdfb1700 1 -- 172.21.15.1:0/1958958675 learned_addr learned my addr 172.21.15.1:0/1958958675 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.016+0000 7f72fcfaf700 1 -- 172.21.15.1:0/1958958675 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242908321 0 0) 0x7f72f8072660 con 0x7f72f8146190 2024-03-18T20:26:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.016+0000 7f72fe7b2700 1 --2- 172.21.15.1:0/1958958675 >> v2:172.21.15.165:3300/0 conn(0x7f72f81465a0 0x7f72f8168660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.016+0000 7f72fcfaf700 1 -- 172.21.15.1:0/1958958675 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e00035b0 con 0x7f72f8146190 2024-03-18T20:26:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fcfaf700 1 -- 172.21.15.1:0/1958958675 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4238611591 0 0) 0x7f72e00035b0 con 0x7f72f8146190 2024-03-18T20:26:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fcfaf700 1 -- 172.21.15.1:0/1958958675 >> v2:172.21.15.165:3300/0 conn(0x7f72f81465a0 msgr2=0x7f72f8168660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fcfaf700 1 --2- 172.21.15.1:0/1958958675 >> v2:172.21.15.165:3300/0 conn(0x7f72f81465a0 0x7f72f8168660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fcfaf700 1 -- 172.21.15.1:0/1958958675 >> v1:172.21.15.165:6789/0 conn(0x7f72f806ae60 legacy=0x7f72f8164f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fcfaf700 1 -- 172.21.15.1:0/1958958675 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f806eeb0 con 0x7f72f8146190 2024-03-18T20:26:30.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fcfaf700 1 -- 172.21.15.1:0/1958958675 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f72e80019f0 con 0x7f72f8146190 2024-03-18T20:26:30.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fcfaf700 1 -- 172.21.15.1:0/1958958675 >> v1:172.21.15.1:6789/0 conn(0x7f72f8146190 legacy=0x7f72f806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:30.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fcfaf700 1 --2- 172.21.15.1:0/1958958675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e0003a50 0x7f72e0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fcfaf700 1 --2- 172.21.15.1:0/1958958675 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72e0004850 0x7f72e0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fcfaf700 1 --2- 172.21.15.1:0/1958958675 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72e0005210 0x7f72e000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fcfaf700 1 -- 172.21.15.1:0/1958958675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f806f850 con 0x7f72e0003a50 2024-03-18T20:26:30.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fcfaf700 1 -- 172.21.15.1:0/1958958675 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e000c110 con 0x7f72e0004850 2024-03-18T20:26:30.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fcfaf700 1 -- 172.21.15.1:0/1958958675 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e000c2d0 con 0x7f72e0005210 2024-03-18T20:26:30.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fdfb1700 1 --2- 172.21.15.1:0/1958958675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e0003a50 0x7f72e0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fd7b0700 1 --2- 172.21.15.1:0/1958958675 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72e0004850 0x7f72e0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fe7b2700 1 --2- 172.21.15.1:0/1958958675 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72e0005210 0x7f72e000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fdfb1700 1 -- 172.21.15.1:0/1958958675 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72e0005210 msgr2=0x7f72e000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:30.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fdfb1700 1 --2- 172.21.15.1:0/1958958675 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72e0005210 0x7f72e000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fdfb1700 1 -- 172.21.15.1:0/1958958675 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72e0004850 msgr2=0x7f72e0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:30.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fdfb1700 1 --2- 172.21.15.1:0/1958958675 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72e0004850 0x7f72e0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fdfb1700 1 -- 172.21.15.1:0/1958958675 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e00035b0 con 0x7f72e0003a50 2024-03-18T20:26:30.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fe7b2700 1 -- 172.21.15.1:0/1958958675 reap_dead start 2024-03-18T20:26:30.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.017+0000 7f72fdfb1700 1 --2- 172.21.15.1:0/1958958675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e0003a50 0x7f72e0003e70 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 crypto rx=0x7f72e800a760 tx=0x7f72e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf9b65a107f028fb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:30.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.018+0000 7f72fcfaf700 1 -- 172.21.15.1:0/1958958675 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e800a250 con 0x7f72e0003a50 2024-03-18T20:26:30.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.018+0000 7f72fcfaf700 1 -- 172.21.15.1:0/1958958675 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e80072d0 con 0x7f72e0003a50 2024-03-18T20:26:30.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.018+0000 7f72fcfaf700 1 -- 172.21.15.1:0/1958958675 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8007640 con 0x7f72e0003a50 2024-03-18T20:26:30.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.018+0000 7f7300215700 1 -- 172.21.15.1:0/1958958675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e0003a50 msgr2=0x7f72e0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:30.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.018+0000 7f7300215700 1 --2- 172.21.15.1:0/1958958675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e0003a50 0x7f72e0003e70 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 crypto rx=0x7f72e800a760 tx=0x7f72e0005880 comp rx=0 tx=0).stop 2024-03-18T20:26:30.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.018+0000 7f7300215700 1 -- 172.21.15.1:0/1958958675 shutdown_connections 2024-03-18T20:26:30.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.018+0000 7f7300215700 1 --2- 172.21.15.1:0/1958958675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e0003a50 0x7f72e0003e70 unknown :-1 s=CLOSED pgs=2813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.018+0000 7f7300215700 1 -- 172.21.15.1:0/1958958675 >> 172.21.15.1:0/1958958675 conn(0x7f72f80b1d20 msgr2=0x7f72f80b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:30.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.018+0000 7f7300215700 1 -- 172.21.15.1:0/1958958675 shutdown_connections 2024-03-18T20:26:30.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.018+0000 7f7300215700 1 -- 172.21.15.1:0/1958958675 wait complete. 2024-03-18T20:26:30.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.019+0000 7f7300215700 1 Processor -- start 2024-03-18T20:26:30.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.019+0000 7f7300215700 1 -- start start 2024-03-18T20:26:30.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.020+0000 7f7300215700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e0003a50 0x7f72f80cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.020+0000 7f7300215700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72f8140ff0 0x7f72f8141470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.020+0000 7f7300215700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72f80ccbc0 0x7f72f80cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.020+0000 7f7300215700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f81419b0 con 0x7f72e0003a50 2024-03-18T20:26:30.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.020+0000 7f7300215700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80ce4b0 con 0x7f72f80ccbc0 2024-03-18T20:26:30.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.020+0000 7f7300215700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80cd930 con 0x7f72f8140ff0 2024-03-18T20:26:30.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.020+0000 7f72fdfb1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e0003a50 0x7f72f80cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.020+0000 7f72fdfb1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e0003a50 0x7f72f80cf1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46592/0 (socket says 172.21.15.1:46592) 2024-03-18T20:26:30.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.020+0000 7f72fdfb1700 1 -- 172.21.15.1:0/3174994098 learned_addr learned my addr 172.21.15.1:0/3174994098 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:30.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.020+0000 7f72fd7b0700 1 --2- 172.21.15.1:0/3174994098 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72f8140ff0 0x7f72f8141470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.020+0000 7f72fe7b2700 1 --2- 172.21.15.1:0/3174994098 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72f80ccbc0 0x7f72f80cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.021+0000 7f72fdfb1700 1 -- 172.21.15.1:0/3174994098 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72f8140ff0 msgr2=0x7f72f8141470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:30.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.021+0000 7f72fdfb1700 1 --2- 172.21.15.1:0/3174994098 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72f8140ff0 0x7f72f8141470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.021+0000 7f72fdfb1700 1 -- 172.21.15.1:0/3174994098 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72f80ccbc0 msgr2=0x7f72f80cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:30.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.021+0000 7f72fdfb1700 1 --2- 172.21.15.1:0/3174994098 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72f80ccbc0 0x7f72f80cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.021+0000 7f72fdfb1700 1 -- 172.21.15.1:0/3174994098 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f80cdc10 con 0x7f72e0003a50 2024-03-18T20:26:30.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.021+0000 7f72fe7b2700 1 --2- 172.21.15.1:0/3174994098 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72f80ccbc0 0x7f72f80cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:30.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.021+0000 7f72fdfb1700 1 --2- 172.21.15.1:0/3174994098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e0003a50 0x7f72f80cf1f0 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7f72e800ad50 tx=0x7f72e8001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:30.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.021+0000 7f72f67fc700 1 -- 172.21.15.1:0/3174994098 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8001250 con 0x7f72e0003a50 2024-03-18T20:26:30.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.021+0000 7f72f67fc700 1 -- 172.21.15.1:0/3174994098 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e801f430 con 0x7f72e0003a50 2024-03-18T20:26:30.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.021+0000 7f7300215700 1 -- 172.21.15.1:0/3174994098 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f80cdef0 con 0x7f72e0003a50 2024-03-18T20:26:30.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.021+0000 7f72f67fc700 1 -- 172.21.15.1:0/3174994098 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e801f7c0 con 0x7f72e0003a50 2024-03-18T20:26:30.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.021+0000 7f7300215700 1 -- 172.21.15.1:0/3174994098 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f80cf730 con 0x7f72e0003a50 2024-03-18T20:26:30.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.022+0000 7f7300215700 1 -- 172.21.15.1:0/3174994098 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f80676a0 con 0x7f72e0003a50 2024-03-18T20:26:30.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.022+0000 7f72f67fc700 1 -- 172.21.15.1:0/3174994098 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f72e801f920 con 0x7f72e0003a50 2024-03-18T20:26:30.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.023+0000 7f72f67fc700 1 --2- 172.21.15.1:0/3174994098 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f72e406aed0 0x7f72e406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.023+0000 7f72f67fc700 1 -- 172.21.15.1:0/3174994098 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f72e808b050 con 0x7f72e0003a50 2024-03-18T20:26:30.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.023+0000 7f72fd7b0700 1 --2- 172.21.15.1:0/3174994098 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f72e406aed0 0x7f72e406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.023+0000 7f72fd7b0700 1 --2- 172.21.15.1:0/3174994098 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f72e406aed0 0x7f72e406d390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f72f80cc6e0 tx=0x7f72ec009040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:30.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.027+0000 7f72f67fc700 1 -- 172.21.15.1:0/3174994098 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f72e8055db0 con 0x7f72e0003a50 2024-03-18T20:26:30.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.229+0000 7f7300215700 1 -- 172.21.15.1:0/3174994098 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f72f80cf8c0 con 0x7f72e406aed0 2024-03-18T20:26:30.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.237+0000 7f72f67fc700 1 -- 172.21.15.1:0/3174994098 <== mgr.12057 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290038 (secure 0 0 0) 0x7f72f80cf8c0 con 0x7f72e406aed0 2024-03-18T20:26:30.245 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:30.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.241+0000 7f7300215700 1 -- 172.21.15.1:0/3174994098 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f72e406aed0 msgr2=0x7f72e406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:30.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.241+0000 7f7300215700 1 --2- 172.21.15.1:0/3174994098 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f72e406aed0 0x7f72e406d390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f72f80cc6e0 tx=0x7f72ec009040 comp rx=0 tx=0).stop 2024-03-18T20:26:30.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.241+0000 7f7300215700 1 -- 172.21.15.1:0/3174994098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e0003a50 msgr2=0x7f72f80cf1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:30.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.241+0000 7f7300215700 1 --2- 172.21.15.1:0/3174994098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e0003a50 0x7f72f80cf1f0 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7f72e800ad50 tx=0x7f72e8001cb0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.241+0000 7f7300215700 1 -- 172.21.15.1:0/3174994098 shutdown_connections 2024-03-18T20:26:30.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.241+0000 7f7300215700 1 --2- 172.21.15.1:0/3174994098 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f72e406aed0 0x7f72e406d390 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.241+0000 7f7300215700 1 --2- 172.21.15.1:0/3174994098 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72f80ccbc0 0x7f72f80cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.241+0000 7f7300215700 1 --2- 172.21.15.1:0/3174994098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e0003a50 0x7f72f80cf1f0 unknown :-1 s=CLOSED pgs=2814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.241+0000 7f7300215700 1 --2- 172.21.15.1:0/3174994098 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72f8140ff0 0x7f72f8141470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.241+0000 7f7300215700 1 -- 172.21.15.1:0/3174994098 >> 172.21.15.1:0/3174994098 conn(0x7f72f80b1d20 msgr2=0x7f72f80afc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:30.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.241+0000 7f7300215700 1 -- 172.21.15.1:0/3174994098 shutdown_connections 2024-03-18T20:26:30.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.241+0000 7f7300215700 1 -- 172.21.15.1:0/3174994098 wait complete. 2024-03-18T20:26:30.248 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:26:30.258 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-03-18T20:26:29.876502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":663,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":408,"ondisk_log_size":408,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":181,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":397304,"kb_used_data":26120,"kb_used_omap":12,"kb_used_meta":371059,"kb_avail":754577416,"statfs":{"total":773094113280,"available":772687273984,"internally_reserved":0,"allocated":26746880,"data_stored":14331740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":379964985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.529114"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795771+0000","last_change":"2024-03-18T20:26:25.816062+0000","last_active":"2024-03-18T20:26:28.795771+0000","last_peered":"2024-03-18T20:26:28.795771+0000","last_clean":"2024-03-18T20:26:28.795771+0000","last_became_active":"2024-03-18T20:26:25.815866+0000","last_became_peered":"2024-03-18T20:26:25.815866+0000","last_unstale":"2024-03-18T20:26:28.795771+0000","last_undegraded":"2024-03-18T20:26:28.795771+0000","last_fullsized":"2024-03-18T20:26:28.795771+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:19:53.852641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795813+0000","last_change":"2024-03-18T20:26:25.806337+0000","last_active":"2024-03-18T20:26:28.795813+0000","last_peered":"2024-03-18T20:26:28.795813+0000","last_clean":"2024-03-18T20:26:28.795813+0000","last_became_active":"2024-03-18T20:26:25.806016+0000","last_became_peered":"2024-03-18T20:26:25.806016+0000","last_unstale":"2024-03-18T20:26:28.795813+0000","last_undegraded":"2024-03-18T20:26:28.795813+0000","last_fullsized":"2024-03-18T20:26:28.795813+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:33:55.816835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811592+0000","last_change":"2024-03-18T20:26:25.815050+0000","last_active":"2024-03-18T20:26:27.811592+0000","last_peered":"2024-03-18T20:26:27.811592+0000","last_clean":"2024-03-18T20:26:27.811592+0000","last_became_active":"2024-03-18T20:26:25.814617+0000","last_became_peered":"2024-03-18T20:26:25.814617+0000","last_unstale":"2024-03-18T20:26:27.811592+0000","last_undegraded":"2024-03-18T20:26:27.811592+0000","last_fullsized":"2024-03-18T20:26:27.811592+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:04:32.495090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811543+0000","last_change":"2024-03-18T20:26:25.812709+0000","last_active":"2024-03-18T20:26:27.811543+0000","last_peered":"2024-03-18T20:26:27.811543+0000","last_clean":"2024-03-18T20:26:27.811543+0000","last_became_active":"2024-03-18T20:26:25.812420+0000","last_became_peered":"2024-03-18T20:26:25.812420+0000","last_unstale":"2024-03-18T20:26:27.811543+0000","last_undegraded":"2024-03-18T20:26:27.811543+0000","last_fullsized":"2024-03-18T20:26:27.811543+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:52.955865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.801652+0000","last_change":"2024-03-18T20:26:25.811447+0000","last_active":"2024-03-18T20:26:26.801652+0000","last_peered":"2024-03-18T20:26:26.801652+0000","last_clean":"2024-03-18T20:26:26.801652+0000","last_became_active":"2024-03-18T20:26:25.811144+0000","last_became_peered":"2024-03-18T20:26:25.811144+0000","last_unstale":"2024-03-18T20:26:26.801652+0000","last_undegraded":"2024-03-18T20:26:26.801652+0000","last_fullsized":"2024-03-18T20:26:26.801652+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:54:38.189655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795458+0000","last_change":"2024-03-18T20:26:25.804382+0000","last_active":"2024-03-18T20:26:28.795458+0000","last_peered":"2024-03-18T20:26:28.795458+0000","last_clean":"2024-03-18T20:26:28.795458+0000","last_became_active":"2024-03-18T20:26:25.804096+0000","last_became_peered":"2024-03-18T20:26:25.804096+0000","last_unstale":"2024-03-18T20:26:28.795458+0000","last_undegraded":"2024-03-18T20:26:28.795458+0000","last_fullsized":"2024-03-18T20:26:28.795458+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:23.578829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795206+0000","last_change":"2024-03-18T20:26:25.806198+0000","last_active":"2024-03-18T20:26:28.795206+0000","last_peered":"2024-03-18T20:26:28.795206+0000","last_clean":"2024-03-18T20:26:28.795206+0000","last_became_active":"2024-03-18T20:26:25.805918+0000","last_became_peered":"2024-03-18T20:26:25.805918+0000","last_unstale":"2024-03-18T20:26:28.795206+0000","last_undegraded":"2024-03-18T20:26:28.795206+0000","last_fullsized":"2024-03-18T20:26:28.795206+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:11.977302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.801441+0000","last_change":"2024-03-18T20:26:25.805458+0000","last_active":"2024-03-18T20:26:26.801441+0000","last_peered":"2024-03-18T20:26:26.801441+0000","last_clean":"2024-03-18T20:26:26.801441+0000","last_became_active":"2024-03-18T20:26:25.805176+0000","last_became_peered":"2024-03-18T20:26:25.805176+0000","last_unstale":"2024-03-18T20:26:26.801441+0000","last_undegraded":"2024-03-18T20:26:26.801441+0000","last_fullsized":"2024-03-18T20:26:26.801441+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:06:22.720260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795838+0000","last_change":"2024-03-18T20:26:25.815665+0000","last_active":"2024-03-18T20:26:28.795838+0000","last_peered":"2024-03-18T20:26:28.795838+0000","last_clean":"2024-03-18T20:26:28.795838+0000","last_became_active":"2024-03-18T20:26:25.815259+0000","last_became_peered":"2024-03-18T20:26:25.815259+0000","last_unstale":"2024-03-18T20:26:28.795838+0000","last_undegraded":"2024-03-18T20:26:28.795838+0000","last_fullsized":"2024-03-18T20:26:28.795838+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:39:11.551926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811680+0000","last_change":"2024-03-18T20:26:25.805234+0000","last_active":"2024-03-18T20:26:27.811680+0000","last_peered":"2024-03-18T20:26:27.811680+0000","last_clean":"2024-03-18T20:26:27.811680+0000","last_became_active":"2024-03-18T20:26:25.805056+0000","last_became_peered":"2024-03-18T20:26:25.805056+0000","last_unstale":"2024-03-18T20:26:27.811680+0000","last_undegraded":"2024-03-18T20:26:27.811680+0000","last_fullsized":"2024-03-18T20:26:27.811680+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:50.003103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811626+0000","last_change":"2024-03-18T20:26:25.813598+0000","last_active":"2024-03-18T20:26:27.811626+0000","last_peered":"2024-03-18T20:26:27.811626+0000","last_clean":"2024-03-18T20:26:27.811626+0000","last_became_active":"2024-03-18T20:26:25.812093+0000","last_became_peered":"2024-03-18T20:26:25.812093+0000","last_unstale":"2024-03-18T20:26:27.811626+0000","last_undegraded":"2024-03-18T20:26:27.811626+0000","last_fullsized":"2024-03-18T20:26:27.811626+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:38.417379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795610+0000","last_change":"2024-03-18T20:26:25.811095+0000","last_active":"2024-03-18T20:26:28.795610+0000","last_peered":"2024-03-18T20:26:28.795610+0000","last_clean":"2024-03-18T20:26:28.795610+0000","last_became_active":"2024-03-18T20:26:25.810796+0000","last_became_peered":"2024-03-18T20:26:25.810796+0000","last_unstale":"2024-03-18T20:26:28.795610+0000","last_undegraded":"2024-03-18T20:26:28.795610+0000","last_fullsized":"2024-03-18T20:26:28.795610+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:30:08.833713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.294453+0000","last_change":"2024-03-18T20:26:25.815198+0000","last_active":"2024-03-18T20:26:28.294453+0000","last_peered":"2024-03-18T20:26:28.294453+0000","last_clean":"2024-03-18T20:26:28.294453+0000","last_became_active":"2024-03-18T20:26:25.814922+0000","last_became_peered":"2024-03-18T20:26:25.814922+0000","last_unstale":"2024-03-18T20:26:28.294453+0000","last_undegraded":"2024-03-18T20:26:28.294453+0000","last_fullsized":"2024-03-18T20:26:28.294453+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:30:14.320846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.802738+0000","last_change":"2024-03-18T20:26:25.809230+0000","last_active":"2024-03-18T20:26:26.802738+0000","last_peered":"2024-03-18T20:26:26.802738+0000","last_clean":"2024-03-18T20:26:26.802738+0000","last_became_active":"2024-03-18T20:26:25.808941+0000","last_became_peered":"2024-03-18T20:26:25.808941+0000","last_unstale":"2024-03-18T20:26:26.802738+0000","last_undegraded":"2024-03-18T20:26:26.802738+0000","last_fullsized":"2024-03-18T20:26:26.802738+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:34:05.214460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795265+0000","last_change":"2024-03-18T20:26:25.798972+0000","last_active":"2024-03-18T20:26:28.795265+0000","last_peered":"2024-03-18T20:26:28.795265+0000","last_clean":"2024-03-18T20:26:28.795265+0000","last_became_active":"2024-03-18T20:26:25.798736+0000","last_became_peered":"2024-03-18T20:26:25.798736+0000","last_unstale":"2024-03-18T20:26:28.795265+0000","last_undegraded":"2024-03-18T20:26:28.795265+0000","last_fullsized":"2024-03-18T20:26:28.795265+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:56.892630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811302+0000","last_change":"2024-03-18T20:26:25.799408+0000","last_active":"2024-03-18T20:26:27.811302+0000","last_peered":"2024-03-18T20:26:27.811302+0000","last_clean":"2024-03-18T20:26:27.811302+0000","last_became_active":"2024-03-18T20:26:25.798788+0000","last_became_peered":"2024-03-18T20:26:25.798788+0000","last_unstale":"2024-03-18T20:26:27.811302+0000","last_undegraded":"2024-03-18T20:26:27.811302+0000","last_fullsized":"2024-03-18T20:26:27.811302+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:34:05.214460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.294748+0000","last_change":"2024-03-18T20:26:25.811390+0000","last_active":"2024-03-18T20:26:28.294748+0000","last_peered":"2024-03-18T20:26:28.294748+0000","last_clean":"2024-03-18T20:26:28.294748+0000","last_became_active":"2024-03-18T20:26:25.811081+0000","last_became_peered":"2024-03-18T20:26:25.811081+0000","last_unstale":"2024-03-18T20:26:28.294748+0000","last_undegraded":"2024-03-18T20:26:28.294748+0000","last_fullsized":"2024-03-18T20:26:28.294748+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:33:55.816835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811789+0000","last_change":"2024-03-18T20:26:25.799778+0000","last_active":"2024-03-18T20:26:27.811789+0000","last_peered":"2024-03-18T20:26:27.811789+0000","last_clean":"2024-03-18T20:26:27.811789+0000","last_became_active":"2024-03-18T20:26:25.799517+0000","last_became_peered":"2024-03-18T20:26:25.799517+0000","last_unstale":"2024-03-18T20:26:27.811789+0000","last_undegraded":"2024-03-18T20:26:27.811789+0000","last_fullsized":"2024-03-18T20:26:27.811789+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:32:34.214872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795549+0000","last_change":"2024-03-18T20:26:25.812369+0000","last_active":"2024-03-18T20:26:28.795549+0000","last_peered":"2024-03-18T20:26:28.795549+0000","last_clean":"2024-03-18T20:26:28.795549+0000","last_became_active":"2024-03-18T20:26:25.812082+0000","last_became_peered":"2024-03-18T20:26:25.812082+0000","last_unstale":"2024-03-18T20:26:28.795549+0000","last_undegraded":"2024-03-18T20:26:28.795549+0000","last_fullsized":"2024-03-18T20:26:28.795549+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:43:00.585492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.796205+0000","last_change":"2024-03-18T20:26:25.807591+0000","last_active":"2024-03-18T20:26:28.796205+0000","last_peered":"2024-03-18T20:26:28.796205+0000","last_clean":"2024-03-18T20:26:28.796205+0000","last_became_active":"2024-03-18T20:26:25.807349+0000","last_became_peered":"2024-03-18T20:26:25.807349+0000","last_unstale":"2024-03-18T20:26:28.796205+0000","last_undegraded":"2024-03-18T20:26:28.796205+0000","last_fullsized":"2024-03-18T20:26:28.796205+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:58.931342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.794838+0000","last_change":"2024-03-18T20:26:25.799427+0000","last_active":"2024-03-18T20:26:28.794838+0000","last_peered":"2024-03-18T20:26:28.794838+0000","last_clean":"2024-03-18T20:26:28.794838+0000","last_became_active":"2024-03-18T20:26:25.799041+0000","last_became_peered":"2024-03-18T20:26:25.799041+0000","last_unstale":"2024-03-18T20:26:28.794838+0000","last_undegraded":"2024-03-18T20:26:28.794838+0000","last_fullsized":"2024-03-18T20:26:28.794838+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:35.919143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.294422+0000","last_change":"2024-03-18T20:26:25.818643+0000","last_active":"2024-03-18T20:26:28.294422+0000","last_peered":"2024-03-18T20:26:28.294422+0000","last_clean":"2024-03-18T20:26:28.294422+0000","last_became_active":"2024-03-18T20:26:25.818377+0000","last_became_peered":"2024-03-18T20:26:25.818377+0000","last_unstale":"2024-03-18T20:26:28.294422+0000","last_undegraded":"2024-03-18T20:26:28.294422+0000","last_fullsized":"2024-03-18T20:26:28.294422+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:44.809054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795391+0000","last_change":"2024-03-18T20:26:25.805834+0000","last_active":"2024-03-18T20:26:28.795391+0000","last_peered":"2024-03-18T20:26:28.795391+0000","last_clean":"2024-03-18T20:26:28.795391+0000","last_became_active":"2024-03-18T20:26:25.805491+0000","last_became_peered":"2024-03-18T20:26:25.805491+0000","last_unstale":"2024-03-18T20:26:28.795391+0000","last_undegraded":"2024-03-18T20:26:28.795391+0000","last_fullsized":"2024-03-18T20:26:28.795391+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:23.305150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.294230+0000","last_change":"2024-03-18T20:26:25.809091+0000","last_active":"2024-03-18T20:26:28.294230+0000","last_peered":"2024-03-18T20:26:28.294230+0000","last_clean":"2024-03-18T20:26:28.294230+0000","last_became_active":"2024-03-18T20:26:25.808800+0000","last_became_peered":"2024-03-18T20:26:25.808800+0000","last_unstale":"2024-03-18T20:26:28.294230+0000","last_undegraded":"2024-03-18T20:26:28.294230+0000","last_fullsized":"2024-03-18T20:26:28.294230+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:32.033370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795464+0000","last_change":"2024-03-18T20:26:25.798738+0000","last_active":"2024-03-18T20:26:28.795464+0000","last_peered":"2024-03-18T20:26:28.795464+0000","last_clean":"2024-03-18T20:26:28.795464+0000","last_became_active":"2024-03-18T20:26:25.798480+0000","last_became_peered":"2024-03-18T20:26:25.798480+0000","last_unstale":"2024-03-18T20:26:28.795464+0000","last_undegraded":"2024-03-18T20:26:28.795464+0000","last_fullsized":"2024-03-18T20:26:28.795464+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:02.900783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795007+0000","last_change":"2024-03-18T20:26:25.806356+0000","last_active":"2024-03-18T20:26:28.795007+0000","last_peered":"2024-03-18T20:26:28.795007+0000","last_clean":"2024-03-18T20:26:28.795007+0000","last_became_active":"2024-03-18T20:26:25.805895+0000","last_became_peered":"2024-03-18T20:26:25.805895+0000","last_unstale":"2024-03-18T20:26:28.795007+0000","last_undegraded":"2024-03-18T20:26:28.795007+0000","last_fullsized":"2024-03-18T20:26:28.795007+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:06:16.285934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.801835+0000","last_change":"2024-03-18T20:26:25.806860+0000","last_active":"2024-03-18T20:26:26.801835+0000","last_peered":"2024-03-18T20:26:26.801835+0000","last_clean":"2024-03-18T20:26:26.801835+0000","last_became_active":"2024-03-18T20:26:25.806650+0000","last_became_peered":"2024-03-18T20:26:25.806650+0000","last_unstale":"2024-03-18T20:26:26.801835+0000","last_undegraded":"2024-03-18T20:26:26.801835+0000","last_fullsized":"2024-03-18T20:26:26.801835+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:06:16.285934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795827+0000","last_change":"2024-03-18T20:26:25.815611+0000","last_active":"2024-03-18T20:26:28.795827+0000","last_peered":"2024-03-18T20:26:28.795827+0000","last_clean":"2024-03-18T20:26:28.795827+0000","last_became_active":"2024-03-18T20:26:25.815172+0000","last_became_peered":"2024-03-18T20:26:25.815172+0000","last_unstale":"2024-03-18T20:26:28.795827+0000","last_undegraded":"2024-03-18T20:26:28.795827+0000","last_fullsized":"2024-03-18T20:26:28.795827+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:48:15.078756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.802865+0000","last_change":"2024-03-18T20:26:25.810485+0000","last_active":"2024-03-18T20:26:26.802865+0000","last_peered":"2024-03-18T20:26:26.802865+0000","last_clean":"2024-03-18T20:26:26.802865+0000","last_became_active":"2024-03-18T20:26:25.810235+0000","last_became_peered":"2024-03-18T20:26:25.810235+0000","last_unstale":"2024-03-18T20:26:26.802865+0000","last_undegraded":"2024-03-18T20:26:26.802865+0000","last_fullsized":"2024-03-18T20:26:26.802865+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:59.140240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.802735+0000","last_change":"2024-03-18T20:26:25.807504+0000","last_active":"2024-03-18T20:26:26.802735+0000","last_peered":"2024-03-18T20:26:26.802735+0000","last_clean":"2024-03-18T20:26:26.802735+0000","last_became_active":"2024-03-18T20:26:25.807257+0000","last_became_peered":"2024-03-18T20:26:25.807257+0000","last_unstale":"2024-03-18T20:26:26.802735+0000","last_undegraded":"2024-03-18T20:26:26.802735+0000","last_fullsized":"2024-03-18T20:26:26.802735+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:40.456111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795466+0000","last_change":"2024-03-18T20:26:25.796362+0000","last_active":"2024-03-18T20:26:28.795466+0000","last_peered":"2024-03-18T20:26:28.795466+0000","last_clean":"2024-03-18T20:26:28.795466+0000","last_became_active":"2024-03-18T20:26:25.796061+0000","last_became_peered":"2024-03-18T20:26:25.796061+0000","last_unstale":"2024-03-18T20:26:28.795466+0000","last_undegraded":"2024-03-18T20:26:28.795466+0000","last_fullsized":"2024-03-18T20:26:28.795466+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:28.602198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.794936+0000","last_change":"2024-03-18T20:26:25.815799+0000","last_active":"2024-03-18T20:26:28.794936+0000","last_peered":"2024-03-18T20:26:28.794936+0000","last_clean":"2024-03-18T20:26:28.794936+0000","last_became_active":"2024-03-18T20:26:25.815392+0000","last_became_peered":"2024-03-18T20:26:25.815392+0000","last_unstale":"2024-03-18T20:26:28.794936+0000","last_undegraded":"2024-03-18T20:26:28.794936+0000","last_fullsized":"2024-03-18T20:26:28.794936+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:21.319444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795156+0000","last_change":"2024-03-18T20:26:25.808089+0000","last_active":"2024-03-18T20:26:28.795156+0000","last_peered":"2024-03-18T20:26:28.795156+0000","last_clean":"2024-03-18T20:26:28.795156+0000","last_became_active":"2024-03-18T20:26:25.806688+0000","last_became_peered":"2024-03-18T20:26:25.806688+0000","last_unstale":"2024-03-18T20:26:28.795156+0000","last_undegraded":"2024-03-18T20:26:28.795156+0000","last_fullsized":"2024-03-18T20:26:28.795156+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:22.598187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"252'4","reported_seq":15,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.818510+0000","last_change":"2024-03-18T20:26:25.369142+0000","last_active":"2024-03-18T20:26:25.818510+0000","last_peered":"2024-03-18T20:26:25.818510+0000","last_clean":"2024-03-18T20:26:25.818510+0000","last_became_active":"2024-03-18T20:26:25.368879+0000","last_became_peered":"2024-03-18T20:26:25.368879+0000","last_unstale":"2024-03-18T20:26:25.818510+0000","last_undegraded":"2024-03-18T20:26:25.818510+0000","last_fullsized":"2024-03-18T20:26:25.818510+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:42.709929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795253+0000","last_change":"2024-03-18T20:26:25.812175+0000","last_active":"2024-03-18T20:26:28.795253+0000","last_peered":"2024-03-18T20:26:28.795253+0000","last_clean":"2024-03-18T20:26:28.795253+0000","last_became_active":"2024-03-18T20:26:25.811898+0000","last_became_peered":"2024-03-18T20:26:25.811898+0000","last_unstale":"2024-03-18T20:26:28.795253+0000","last_undegraded":"2024-03-18T20:26:28.795253+0000","last_fullsized":"2024-03-18T20:26:28.795253+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:45:00.418514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"252'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.809720+0000","last_change":"2024-03-18T20:26:25.371848+0000","last_active":"2024-03-18T20:26:25.809720+0000","last_peered":"2024-03-18T20:26:25.809720+0000","last_clean":"2024-03-18T20:26:25.809720+0000","last_became_active":"2024-03-18T20:26:25.371656+0000","last_became_peered":"2024-03-18T20:26:25.371656+0000","last_unstale":"2024-03-18T20:26:25.809720+0000","last_undegraded":"2024-03-18T20:26:25.809720+0000","last_fullsized":"2024-03-18T20:26:25.809720+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:27.592158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.796244+0000","last_change":"2024-03-18T20:26:25.809815+0000","last_active":"2024-03-18T20:26:28.796244+0000","last_peered":"2024-03-18T20:26:28.796244+0000","last_clean":"2024-03-18T20:26:28.796244+0000","last_became_active":"2024-03-18T20:26:25.809504+0000","last_became_peered":"2024-03-18T20:26:25.809504+0000","last_unstale":"2024-03-18T20:26:28.796244+0000","last_undegraded":"2024-03-18T20:26:28.796244+0000","last_fullsized":"2024-03-18T20:26:28.796244+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:42:34.548735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"252'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.811148+0000","last_change":"2024-03-18T20:26:25.366016+0000","last_active":"2024-03-18T20:26:25.811148+0000","last_peered":"2024-03-18T20:26:25.811148+0000","last_clean":"2024-03-18T20:26:25.811148+0000","last_became_active":"2024-03-18T20:26:25.365694+0000","last_became_peered":"2024-03-18T20:26:25.365694+0000","last_unstale":"2024-03-18T20:26:25.811148+0000","last_undegraded":"2024-03-18T20:26:25.811148+0000","last_fullsized":"2024-03-18T20:26:25.811148+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:08.800093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.802792+0000","last_change":"2024-03-18T20:26:25.802194+0000","last_active":"2024-03-18T20:26:26.802792+0000","last_peered":"2024-03-18T20:26:26.802792+0000","last_clean":"2024-03-18T20:26:26.802792+0000","last_became_active":"2024-03-18T20:26:25.801915+0000","last_became_peered":"2024-03-18T20:26:25.801915+0000","last_unstale":"2024-03-18T20:26:26.802792+0000","last_undegraded":"2024-03-18T20:26:26.802792+0000","last_fullsized":"2024-03-18T20:26:26.802792+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:52.955865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"252'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.808116+0000","last_change":"2024-03-18T20:26:25.364855+0000","last_active":"2024-03-18T20:26:25.808116+0000","last_peered":"2024-03-18T20:26:25.808116+0000","last_clean":"2024-03-18T20:26:25.808116+0000","last_became_active":"2024-03-18T20:26:25.364563+0000","last_became_peered":"2024-03-18T20:26:25.364563+0000","last_unstale":"2024-03-18T20:26:25.808116+0000","last_undegraded":"2024-03-18T20:26:25.808116+0000","last_fullsized":"2024-03-18T20:26:25.808116+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:17.116325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.794972+0000","last_change":"2024-03-18T20:26:25.799795+0000","last_active":"2024-03-18T20:26:28.794972+0000","last_peered":"2024-03-18T20:26:28.794972+0000","last_clean":"2024-03-18T20:26:28.794972+0000","last_became_active":"2024-03-18T20:26:25.799539+0000","last_became_peered":"2024-03-18T20:26:25.799539+0000","last_unstale":"2024-03-18T20:26:28.794972+0000","last_undegraded":"2024-03-18T20:26:28.794972+0000","last_fullsized":"2024-03-18T20:26:28.794972+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:49.613713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"252'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.804832+0000","last_change":"2024-03-18T20:26:25.367422+0000","last_active":"2024-03-18T20:26:25.804832+0000","last_peered":"2024-03-18T20:26:25.804832+0000","last_clean":"2024-03-18T20:26:25.804832+0000","last_became_active":"2024-03-18T20:26:25.367153+0000","last_became_peered":"2024-03-18T20:26:25.367153+0000","last_unstale":"2024-03-18T20:26:25.804832+0000","last_undegraded":"2024-03-18T20:26:25.804832+0000","last_fullsized":"2024-03-18T20:26:25.804832+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:14:17.781705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811757+0000","last_change":"2024-03-18T20:26:25.812108+0000","last_active":"2024-03-18T20:26:27.811757+0000","last_peered":"2024-03-18T20:26:27.811757+0000","last_clean":"2024-03-18T20:26:27.811757+0000","last_became_active":"2024-03-18T20:26:25.811747+0000","last_became_peered":"2024-03-18T20:26:25.811747+0000","last_unstale":"2024-03-18T20:26:27.811757+0000","last_undegraded":"2024-03-18T20:26:27.811757+0000","last_fullsized":"2024-03-18T20:26:27.811757+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:40.456111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.790932+0000","last_change":"2024-03-18T20:26:25.360666+0000","last_active":"2024-03-18T20:26:25.790932+0000","last_peered":"2024-03-18T20:26:25.790932+0000","last_clean":"2024-03-18T20:26:25.790932+0000","last_became_active":"2024-03-18T20:26:25.360400+0000","last_became_peered":"2024-03-18T20:26:25.360400+0000","last_unstale":"2024-03-18T20:26:25.790932+0000","last_undegraded":"2024-03-18T20:26:25.790932+0000","last_fullsized":"2024-03-18T20:26:25.790932+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:18:38.754833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.802212+0000","last_change":"2024-03-18T20:26:25.810099+0000","last_active":"2024-03-18T20:26:26.802212+0000","last_peered":"2024-03-18T20:26:26.802212+0000","last_clean":"2024-03-18T20:26:26.802212+0000","last_became_active":"2024-03-18T20:26:25.806786+0000","last_became_peered":"2024-03-18T20:26:25.806786+0000","last_unstale":"2024-03-18T20:26:26.802212+0000","last_undegraded":"2024-03-18T20:26:26.802212+0000","last_fullsized":"2024-03-18T20:26:26.802212+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:35.919143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.804380+0000","last_change":"2024-03-18T20:26:25.365778+0000","last_active":"2024-03-18T20:26:25.804380+0000","last_peered":"2024-03-18T20:26:25.804380+0000","last_clean":"2024-03-18T20:26:25.804380+0000","last_became_active":"2024-03-18T20:26:25.365505+0000","last_became_peered":"2024-03-18T20:26:25.365505+0000","last_unstale":"2024-03-18T20:26:25.804380+0000","last_undegraded":"2024-03-18T20:26:25.804380+0000","last_fullsized":"2024-03-18T20:26:25.804380+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:46:21.036246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.794855+0000","last_change":"2024-03-18T20:26:25.798444+0000","last_active":"2024-03-18T20:26:28.794855+0000","last_peered":"2024-03-18T20:26:28.794855+0000","last_clean":"2024-03-18T20:26:28.794855+0000","last_became_active":"2024-03-18T20:26:25.798186+0000","last_became_peered":"2024-03-18T20:26:25.798186+0000","last_unstale":"2024-03-18T20:26:28.794855+0000","last_undegraded":"2024-03-18T20:26:28.794855+0000","last_fullsized":"2024-03-18T20:26:28.794855+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:06:22.720260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.794818+0000","last_change":"2024-03-18T20:26:25.370707+0000","last_active":"2024-03-18T20:26:25.794818+0000","last_peered":"2024-03-18T20:26:25.794818+0000","last_clean":"2024-03-18T20:26:25.794818+0000","last_became_active":"2024-03-18T20:26:25.370415+0000","last_became_peered":"2024-03-18T20:26:25.370415+0000","last_unstale":"2024-03-18T20:26:25.794818+0000","last_undegraded":"2024-03-18T20:26:25.794818+0000","last_fullsized":"2024-03-18T20:26:25.794818+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:05.351419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.294245+0000","last_change":"2024-03-18T20:26:25.816190+0000","last_active":"2024-03-18T20:26:28.294245+0000","last_peered":"2024-03-18T20:26:28.294245+0000","last_clean":"2024-03-18T20:26:28.294245+0000","last_became_active":"2024-03-18T20:26:25.815931+0000","last_became_peered":"2024-03-18T20:26:25.815931+0000","last_unstale":"2024-03-18T20:26:28.294245+0000","last_undegraded":"2024-03-18T20:26:28.294245+0000","last_fullsized":"2024-03-18T20:26:28.294245+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:49.613713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.791874+0000","last_change":"2024-03-18T20:26:25.363775+0000","last_active":"2024-03-18T20:26:25.791874+0000","last_peered":"2024-03-18T20:26:25.791874+0000","last_clean":"2024-03-18T20:26:25.791874+0000","last_became_active":"2024-03-18T20:26:25.363389+0000","last_became_peered":"2024-03-18T20:26:25.363389+0000","last_unstale":"2024-03-18T20:26:25.791874+0000","last_undegraded":"2024-03-18T20:26:25.791874+0000","last_fullsized":"2024-03-18T20:26:25.791874+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:42.397208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811342+0000","last_change":"2024-03-18T20:26:25.812196+0000","last_active":"2024-03-18T20:26:27.811342+0000","last_peered":"2024-03-18T20:26:27.811342+0000","last_clean":"2024-03-18T20:26:27.811342+0000","last_became_active":"2024-03-18T20:26:25.811934+0000","last_became_peered":"2024-03-18T20:26:25.811934+0000","last_unstale":"2024-03-18T20:26:27.811342+0000","last_undegraded":"2024-03-18T20:26:27.811342+0000","last_fullsized":"2024-03-18T20:26:27.811342+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:22.017892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"252'3","reported_seq":14,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.811258+0000","last_change":"2024-03-18T20:26:25.366952+0000","last_active":"2024-03-18T20:26:25.811258+0000","last_peered":"2024-03-18T20:26:25.811258+0000","last_clean":"2024-03-18T20:26:25.811258+0000","last_became_active":"2024-03-18T20:26:25.366671+0000","last_became_peered":"2024-03-18T20:26:25.366671+0000","last_unstale":"2024-03-18T20:26:25.811258+0000","last_undegraded":"2024-03-18T20:26:25.811258+0000","last_fullsized":"2024-03-18T20:26:25.811258+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:13:21.871477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.794932+0000","last_change":"2024-03-18T20:26:25.802694+0000","last_active":"2024-03-18T20:26:28.794932+0000","last_peered":"2024-03-18T20:26:28.794932+0000","last_clean":"2024-03-18T20:26:28.794932+0000","last_became_active":"2024-03-18T20:26:25.802443+0000","last_became_peered":"2024-03-18T20:26:25.802443+0000","last_unstale":"2024-03-18T20:26:28.794932+0000","last_undegraded":"2024-03-18T20:26:28.794932+0000","last_fullsized":"2024-03-18T20:26:28.794932+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:54:38.189655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"252'6","reported_seq":17,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.818245+0000","last_change":"2024-03-18T20:26:25.368198+0000","last_active":"2024-03-18T20:26:25.818245+0000","last_peered":"2024-03-18T20:26:25.818245+0000","last_clean":"2024-03-18T20:26:25.818245+0000","last_became_active":"2024-03-18T20:26:25.367762+0000","last_became_peered":"2024-03-18T20:26:25.367762+0000","last_unstale":"2024-03-18T20:26:25.818245+0000","last_undegraded":"2024-03-18T20:26:25.818245+0000","last_fullsized":"2024-03-18T20:26:25.818245+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:06.153535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.796159+0000","last_change":"2024-03-18T20:26:25.796336+0000","last_active":"2024-03-18T20:26:28.796159+0000","last_peered":"2024-03-18T20:26:28.796159+0000","last_clean":"2024-03-18T20:26:28.796159+0000","last_became_active":"2024-03-18T20:26:25.796002+0000","last_became_peered":"2024-03-18T20:26:25.796002+0000","last_unstale":"2024-03-18T20:26:28.796159+0000","last_undegraded":"2024-03-18T20:26:28.796159+0000","last_fullsized":"2024-03-18T20:26:28.796159+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:13:22.881517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"252'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.816996+0000","last_change":"2024-03-18T20:26:25.366019+0000","last_active":"2024-03-18T20:26:25.816996+0000","last_peered":"2024-03-18T20:26:25.816996+0000","last_clean":"2024-03-18T20:26:25.816996+0000","last_became_active":"2024-03-18T20:26:25.364848+0000","last_became_peered":"2024-03-18T20:26:25.364848+0000","last_unstale":"2024-03-18T20:26:25.816996+0000","last_undegraded":"2024-03-18T20:26:25.816996+0000","last_fullsized":"2024-03-18T20:26:25.816996+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:06:15.275895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.794903+0000","last_change":"2024-03-18T20:26:25.803741+0000","last_active":"2024-03-18T20:26:28.794903+0000","last_peered":"2024-03-18T20:26:28.794903+0000","last_clean":"2024-03-18T20:26:28.794903+0000","last_became_active":"2024-03-18T20:26:25.803470+0000","last_became_peered":"2024-03-18T20:26:25.803470+0000","last_unstale":"2024-03-18T20:26:28.794903+0000","last_undegraded":"2024-03-18T20:26:28.794903+0000","last_fullsized":"2024-03-18T20:26:28.794903+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:02:55.614091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.791265+0000","last_change":"2024-03-18T20:26:25.366963+0000","last_active":"2024-03-18T20:26:25.791265+0000","last_peered":"2024-03-18T20:26:25.791265+0000","last_clean":"2024-03-18T20:26:25.791265+0000","last_became_active":"2024-03-18T20:26:25.366416+0000","last_became_peered":"2024-03-18T20:26:25.366416+0000","last_unstale":"2024-03-18T20:26:25.791265+0000","last_undegraded":"2024-03-18T20:26:25.791265+0000","last_fullsized":"2024-03-18T20:26:25.791265+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:57.921302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795529+0000","last_change":"2024-03-18T20:26:25.808144+0000","last_active":"2024-03-18T20:26:28.795529+0000","last_peered":"2024-03-18T20:26:28.795529+0000","last_clean":"2024-03-18T20:26:28.795529+0000","last_became_active":"2024-03-18T20:26:25.807838+0000","last_became_peered":"2024-03-18T20:26:25.807838+0000","last_unstale":"2024-03-18T20:26:28.795529+0000","last_undegraded":"2024-03-18T20:26:28.795529+0000","last_fullsized":"2024-03-18T20:26:28.795529+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:37:41.132114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.791803+0000","last_change":"2024-03-18T20:26:25.366119+0000","last_active":"2024-03-18T20:26:25.791803+0000","last_peered":"2024-03-18T20:26:25.791803+0000","last_clean":"2024-03-18T20:26:25.791803+0000","last_became_active":"2024-03-18T20:26:25.365869+0000","last_became_peered":"2024-03-18T20:26:25.365869+0000","last_unstale":"2024-03-18T20:26:25.791803+0000","last_undegraded":"2024-03-18T20:26:25.791803+0000","last_fullsized":"2024-03-18T20:26:25.791803+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:01.890743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.802155+0000","last_change":"2024-03-18T20:26:25.802507+0000","last_active":"2024-03-18T20:26:26.802155+0000","last_peered":"2024-03-18T20:26:26.802155+0000","last_clean":"2024-03-18T20:26:26.802155+0000","last_became_active":"2024-03-18T20:26:25.801913+0000","last_became_peered":"2024-03-18T20:26:25.801913+0000","last_unstale":"2024-03-18T20:26:26.802155+0000","last_undegraded":"2024-03-18T20:26:26.802155+0000","last_fullsized":"2024-03-18T20:26:26.802155+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:11.977302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"252'8","reported_seq":19,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.817056+0000","last_change":"2024-03-18T20:26:25.365245+0000","last_active":"2024-03-18T20:26:25.817056+0000","last_peered":"2024-03-18T20:26:25.817056+0000","last_clean":"2024-03-18T20:26:25.817056+0000","last_became_active":"2024-03-18T20:26:25.364945+0000","last_became_peered":"2024-03-18T20:26:25.364945+0000","last_unstale":"2024-03-18T20:26:25.817056+0000","last_undegraded":"2024-03-18T20:26:25.817056+0000","last_fullsized":"2024-03-18T20:26:25.817056+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:19.739927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.801434+0000","last_change":"2024-03-18T20:26:25.802286+0000","last_active":"2024-03-18T20:26:26.801434+0000","last_peered":"2024-03-18T20:26:26.801434+0000","last_clean":"2024-03-18T20:26:26.801434+0000","last_became_active":"2024-03-18T20:26:25.802005+0000","last_became_peered":"2024-03-18T20:26:25.802005+0000","last_unstale":"2024-03-18T20:26:26.801434+0000","last_undegraded":"2024-03-18T20:26:26.801434+0000","last_fullsized":"2024-03-18T20:26:26.801434+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:10:16.362251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"252'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.815493+0000","last_change":"2024-03-18T20:26:25.368591+0000","last_active":"2024-03-18T20:26:25.815493+0000","last_peered":"2024-03-18T20:26:25.815493+0000","last_clean":"2024-03-18T20:26:25.815493+0000","last_became_active":"2024-03-18T20:26:25.367843+0000","last_became_peered":"2024-03-18T20:26:25.367843+0000","last_unstale":"2024-03-18T20:26:25.815493+0000","last_undegraded":"2024-03-18T20:26:25.815493+0000","last_fullsized":"2024-03-18T20:26:25.815493+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:59:40.867803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.801360+0000","last_change":"2024-03-18T20:26:25.810340+0000","last_active":"2024-03-18T20:26:26.801360+0000","last_peered":"2024-03-18T20:26:26.801360+0000","last_clean":"2024-03-18T20:26:26.801360+0000","last_became_active":"2024-03-18T20:26:25.809980+0000","last_became_peered":"2024-03-18T20:26:25.809980+0000","last_unstale":"2024-03-18T20:26:26.801360+0000","last_undegraded":"2024-03-18T20:26:26.801360+0000","last_fullsized":"2024-03-18T20:26:26.801360+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:02:55.614091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"252'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.809630+0000","last_change":"2024-03-18T20:26:25.366492+0000","last_active":"2024-03-18T20:26:25.809630+0000","last_peered":"2024-03-18T20:26:25.809630+0000","last_clean":"2024-03-18T20:26:25.809630+0000","last_became_active":"2024-03-18T20:26:25.366098+0000","last_became_peered":"2024-03-18T20:26:25.366098+0000","last_unstale":"2024-03-18T20:26:25.809630+0000","last_undegraded":"2024-03-18T20:26:25.809630+0000","last_fullsized":"2024-03-18T20:26:25.809630+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:53:26.604334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.796134+0000","last_change":"2024-03-18T20:26:25.807945+0000","last_active":"2024-03-18T20:26:28.796134+0000","last_peered":"2024-03-18T20:26:28.796134+0000","last_clean":"2024-03-18T20:26:28.796134+0000","last_became_active":"2024-03-18T20:26:25.807678+0000","last_became_peered":"2024-03-18T20:26:25.807678+0000","last_unstale":"2024-03-18T20:26:28.796134+0000","last_undegraded":"2024-03-18T20:26:28.796134+0000","last_fullsized":"2024-03-18T20:26:28.796134+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:06.759034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"252'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.809803+0000","last_change":"2024-03-18T20:26:25.371835+0000","last_active":"2024-03-18T20:26:25.809803+0000","last_peered":"2024-03-18T20:26:25.809803+0000","last_clean":"2024-03-18T20:26:25.809803+0000","last_became_active":"2024-03-18T20:26:25.371634+0000","last_became_peered":"2024-03-18T20:26:25.371634+0000","last_unstale":"2024-03-18T20:26:25.809803+0000","last_undegraded":"2024-03-18T20:26:25.809803+0000","last_fullsized":"2024-03-18T20:26:25.809803+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:43.471325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811447+0000","last_change":"2024-03-18T20:26:25.812207+0000","last_active":"2024-03-18T20:26:27.811447+0000","last_peered":"2024-03-18T20:26:27.811447+0000","last_clean":"2024-03-18T20:26:27.811447+0000","last_became_active":"2024-03-18T20:26:25.811924+0000","last_became_peered":"2024-03-18T20:26:25.811924+0000","last_unstale":"2024-03-18T20:26:27.811447+0000","last_undegraded":"2024-03-18T20:26:27.811447+0000","last_fullsized":"2024-03-18T20:26:27.811447+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:01:09.966146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.793608+0000","last_change":"2024-03-18T20:26:25.364241+0000","last_active":"2024-03-18T20:26:25.793608+0000","last_peered":"2024-03-18T20:26:25.793608+0000","last_clean":"2024-03-18T20:26:25.793608+0000","last_became_active":"2024-03-18T20:26:25.363945+0000","last_became_peered":"2024-03-18T20:26:25.363945+0000","last_unstale":"2024-03-18T20:26:25.793608+0000","last_undegraded":"2024-03-18T20:26:25.793608+0000","last_fullsized":"2024-03-18T20:26:25.793608+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:34.909104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811477+0000","last_change":"2024-03-18T20:26:25.799178+0000","last_active":"2024-03-18T20:26:27.811477+0000","last_peered":"2024-03-18T20:26:27.811477+0000","last_clean":"2024-03-18T20:26:27.811477+0000","last_became_active":"2024-03-18T20:26:25.798768+0000","last_became_peered":"2024-03-18T20:26:25.798768+0000","last_unstale":"2024-03-18T20:26:27.811477+0000","last_undegraded":"2024-03-18T20:26:27.811477+0000","last_fullsized":"2024-03-18T20:26:27.811477+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:22:12.729546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"252'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.814304+0000","last_change":"2024-03-18T20:26:25.370148+0000","last_active":"2024-03-18T20:26:25.814304+0000","last_peered":"2024-03-18T20:26:25.814304+0000","last_clean":"2024-03-18T20:26:25.814304+0000","last_became_active":"2024-03-18T20:26:25.369876+0000","last_became_peered":"2024-03-18T20:26:25.369876+0000","last_unstale":"2024-03-18T20:26:25.814304+0000","last_undegraded":"2024-03-18T20:26:25.814304+0000","last_fullsized":"2024-03-18T20:26:25.814304+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:57:48.151947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795608+0000","last_change":"2024-03-18T20:26:25.806909+0000","last_active":"2024-03-18T20:26:28.795608+0000","last_peered":"2024-03-18T20:26:28.795608+0000","last_clean":"2024-03-18T20:26:28.795608+0000","last_became_active":"2024-03-18T20:26:25.806643+0000","last_became_peered":"2024-03-18T20:26:25.806643+0000","last_unstale":"2024-03-18T20:26:28.795608+0000","last_undegraded":"2024-03-18T20:26:28.795608+0000","last_fullsized":"2024-03-18T20:26:28.795608+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:30:14.320846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.805502+0000","last_change":"2024-03-18T20:26:25.370582+0000","last_active":"2024-03-18T20:26:25.805502+0000","last_peered":"2024-03-18T20:26:25.805502+0000","last_clean":"2024-03-18T20:26:25.805502+0000","last_became_active":"2024-03-18T20:26:25.370336+0000","last_became_peered":"2024-03-18T20:26:25.370336+0000","last_unstale":"2024-03-18T20:26:25.805502+0000","last_undegraded":"2024-03-18T20:26:25.805502+0000","last_fullsized":"2024-03-18T20:26:25.805502+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:23:03.661960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.294584+0000","last_change":"2024-03-18T20:26:25.814360+0000","last_active":"2024-03-18T20:26:28.294584+0000","last_peered":"2024-03-18T20:26:28.294584+0000","last_clean":"2024-03-18T20:26:28.294584+0000","last_became_active":"2024-03-18T20:26:25.812682+0000","last_became_peered":"2024-03-18T20:26:25.812682+0000","last_unstale":"2024-03-18T20:26:28.294584+0000","last_undegraded":"2024-03-18T20:26:28.294584+0000","last_fullsized":"2024-03-18T20:26:28.294584+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:22.598187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.799106+0000","last_change":"2024-03-18T20:26:25.369740+0000","last_active":"2024-03-18T20:26:25.799106+0000","last_peered":"2024-03-18T20:26:25.799106+0000","last_clean":"2024-03-18T20:26:25.799106+0000","last_became_active":"2024-03-18T20:26:25.369448+0000","last_became_peered":"2024-03-18T20:26:25.369448+0000","last_unstale":"2024-03-18T20:26:25.799106+0000","last_undegraded":"2024-03-18T20:26:25.799106+0000","last_fullsized":"2024-03-18T20:26:25.799106+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:39:25.003994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.294662+0000","last_change":"2024-03-18T20:26:25.807777+0000","last_active":"2024-03-18T20:26:28.294662+0000","last_peered":"2024-03-18T20:26:28.294662+0000","last_clean":"2024-03-18T20:26:28.294662+0000","last_became_active":"2024-03-18T20:26:25.807447+0000","last_became_peered":"2024-03-18T20:26:25.807447+0000","last_unstale":"2024-03-18T20:26:28.294662+0000","last_undegraded":"2024-03-18T20:26:28.294662+0000","last_fullsized":"2024-03-18T20:26:28.294662+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:22:55.933766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.794516+0000","last_change":"2024-03-18T20:26:25.366647+0000","last_active":"2024-03-18T20:26:25.794516+0000","last_peered":"2024-03-18T20:26:25.794516+0000","last_clean":"2024-03-18T20:26:25.794516+0000","last_became_active":"2024-03-18T20:26:25.366363+0000","last_became_peered":"2024-03-18T20:26:25.366363+0000","last_unstale":"2024-03-18T20:26:25.794516+0000","last_undegraded":"2024-03-18T20:26:25.794516+0000","last_fullsized":"2024-03-18T20:26:25.794516+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:22:29.645161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795310+0000","last_change":"2024-03-18T20:26:25.807101+0000","last_active":"2024-03-18T20:26:28.795310+0000","last_peered":"2024-03-18T20:26:28.795310+0000","last_clean":"2024-03-18T20:26:28.795310+0000","last_became_active":"2024-03-18T20:26:25.806372+0000","last_became_peered":"2024-03-18T20:26:25.806372+0000","last_unstale":"2024-03-18T20:26:28.795310+0000","last_undegraded":"2024-03-18T20:26:28.795310+0000","last_fullsized":"2024-03-18T20:26:28.795310+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:18.126364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"252'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.814500+0000","last_change":"2024-03-18T20:26:25.365000+0000","last_active":"2024-03-18T20:26:25.814500+0000","last_peered":"2024-03-18T20:26:25.814500+0000","last_clean":"2024-03-18T20:26:25.814500+0000","last_became_active":"2024-03-18T20:26:25.364727+0000","last_became_peered":"2024-03-18T20:26:25.364727+0000","last_unstale":"2024-03-18T20:26:25.814500+0000","last_undegraded":"2024-03-18T20:26:25.814500+0000","last_fullsized":"2024-03-18T20:26:25.814500+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:18.741871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795257+0000","last_change":"2024-03-18T20:26:25.803062+0000","last_active":"2024-03-18T20:26:28.795257+0000","last_peered":"2024-03-18T20:26:28.795257+0000","last_clean":"2024-03-18T20:26:28.795257+0000","last_became_active":"2024-03-18T20:26:25.802825+0000","last_became_peered":"2024-03-18T20:26:25.802825+0000","last_unstale":"2024-03-18T20:26:28.795257+0000","last_undegraded":"2024-03-18T20:26:28.795257+0000","last_fullsized":"2024-03-18T20:26:28.795257+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:31:54.405889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.790280+0000","last_change":"2024-03-18T20:26:25.367649+0000","last_active":"2024-03-18T20:26:25.790280+0000","last_peered":"2024-03-18T20:26:25.790280+0000","last_clean":"2024-03-18T20:26:25.790280+0000","last_became_active":"2024-03-18T20:26:25.367351+0000","last_became_peered":"2024-03-18T20:26:25.367351+0000","last_unstale":"2024-03-18T20:26:25.790280+0000","last_undegraded":"2024-03-18T20:26:25.790280+0000","last_fullsized":"2024-03-18T20:26:25.790280+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:19:05.047442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811323+0000","last_change":"2024-03-18T20:26:25.812701+0000","last_active":"2024-03-18T20:26:27.811323+0000","last_peered":"2024-03-18T20:26:27.811323+0000","last_clean":"2024-03-18T20:26:27.811323+0000","last_became_active":"2024-03-18T20:26:25.812469+0000","last_became_peered":"2024-03-18T20:26:25.812469+0000","last_unstale":"2024-03-18T20:26:27.811323+0000","last_undegraded":"2024-03-18T20:26:27.811323+0000","last_fullsized":"2024-03-18T20:26:27.811323+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:59.140240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.794829+0000","last_change":"2024-03-18T20:26:25.370628+0000","last_active":"2024-03-18T20:26:25.794829+0000","last_peered":"2024-03-18T20:26:25.794829+0000","last_clean":"2024-03-18T20:26:25.794829+0000","last_became_active":"2024-03-18T20:26:25.370332+0000","last_became_peered":"2024-03-18T20:26:25.370332+0000","last_unstale":"2024-03-18T20:26:25.794829+0000","last_undegraded":"2024-03-18T20:26:25.794829+0000","last_fullsized":"2024-03-18T20:26:25.794829+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:43.011701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.801714+0000","last_change":"2024-03-18T20:26:25.802895+0000","last_active":"2024-03-18T20:26:26.801714+0000","last_peered":"2024-03-18T20:26:26.801714+0000","last_clean":"2024-03-18T20:26:26.801714+0000","last_became_active":"2024-03-18T20:26:25.802655+0000","last_became_peered":"2024-03-18T20:26:25.802655+0000","last_unstale":"2024-03-18T20:26:26.801714+0000","last_undegraded":"2024-03-18T20:26:26.801714+0000","last_fullsized":"2024-03-18T20:26:26.801714+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:18.126364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.794933+0000","last_change":"2024-03-18T20:26:25.368715+0000","last_active":"2024-03-18T20:26:25.794933+0000","last_peered":"2024-03-18T20:26:25.794933+0000","last_clean":"2024-03-18T20:26:25.794933+0000","last_became_active":"2024-03-18T20:26:25.368383+0000","last_became_peered":"2024-03-18T20:26:25.368383+0000","last_unstale":"2024-03-18T20:26:25.794933+0000","last_undegraded":"2024-03-18T20:26:25.794933+0000","last_fullsized":"2024-03-18T20:26:25.794933+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:07.975001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.802845+0000","last_change":"2024-03-18T20:26:25.805198+0000","last_active":"2024-03-18T20:26:26.802845+0000","last_peered":"2024-03-18T20:26:26.802845+0000","last_clean":"2024-03-18T20:26:26.802845+0000","last_became_active":"2024-03-18T20:26:25.804942+0000","last_became_peered":"2024-03-18T20:26:25.804942+0000","last_unstale":"2024-03-18T20:26:26.802845+0000","last_undegraded":"2024-03-18T20:26:26.802845+0000","last_fullsized":"2024-03-18T20:26:26.802845+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:22:12.729546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.808096+0000","last_change":"2024-03-18T20:26:25.370584+0000","last_active":"2024-03-18T20:26:25.808096+0000","last_peered":"2024-03-18T20:26:25.808096+0000","last_clean":"2024-03-18T20:26:25.808096+0000","last_became_active":"2024-03-18T20:26:25.370353+0000","last_became_peered":"2024-03-18T20:26:25.370353+0000","last_unstale":"2024-03-18T20:26:25.808096+0000","last_undegraded":"2024-03-18T20:26:25.808096+0000","last_fullsized":"2024-03-18T20:26:25.808096+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:33:14.925680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795655+0000","last_change":"2024-03-18T20:26:25.801336+0000","last_active":"2024-03-18T20:26:28.795655+0000","last_peered":"2024-03-18T20:26:28.795655+0000","last_clean":"2024-03-18T20:26:28.795655+0000","last_became_active":"2024-03-18T20:26:25.801053+0000","last_became_peered":"2024-03-18T20:26:25.801053+0000","last_unstale":"2024-03-18T20:26:28.795655+0000","last_undegraded":"2024-03-18T20:26:28.795655+0000","last_fullsized":"2024-03-18T20:26:28.795655+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:23.742824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.791211+0000","last_change":"2024-03-18T20:26:25.367183+0000","last_active":"2024-03-18T20:26:25.791211+0000","last_peered":"2024-03-18T20:26:25.791211+0000","last_clean":"2024-03-18T20:26:25.791211+0000","last_became_active":"2024-03-18T20:26:25.366701+0000","last_became_peered":"2024-03-18T20:26:25.366701+0000","last_unstale":"2024-03-18T20:26:25.791211+0000","last_undegraded":"2024-03-18T20:26:25.791211+0000","last_fullsized":"2024-03-18T20:26:25.791211+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:05.748994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795971+0000","last_change":"2024-03-18T20:26:25.806432+0000","last_active":"2024-03-18T20:26:28.795971+0000","last_peered":"2024-03-18T20:26:28.795971+0000","last_clean":"2024-03-18T20:26:28.795971+0000","last_became_active":"2024-03-18T20:26:25.806050+0000","last_became_peered":"2024-03-18T20:26:25.806050+0000","last_unstale":"2024-03-18T20:26:28.795971+0000","last_undegraded":"2024-03-18T20:26:28.795971+0000","last_fullsized":"2024-03-18T20:26:28.795971+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:32.033370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.790834+0000","last_change":"2024-03-18T20:26:25.364069+0000","last_active":"2024-03-18T20:26:25.790834+0000","last_peered":"2024-03-18T20:26:25.790834+0000","last_clean":"2024-03-18T20:26:25.790834+0000","last_became_active":"2024-03-18T20:26:25.363764+0000","last_became_peered":"2024-03-18T20:26:25.363764+0000","last_unstale":"2024-03-18T20:26:25.790834+0000","last_undegraded":"2024-03-18T20:26:25.790834+0000","last_fullsized":"2024-03-18T20:26:25.790834+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:01.862380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.294665+0000","last_change":"2024-03-18T20:26:25.817787+0000","last_active":"2024-03-18T20:26:28.294665+0000","last_peered":"2024-03-18T20:26:28.294665+0000","last_clean":"2024-03-18T20:26:28.294665+0000","last_became_active":"2024-03-18T20:26:25.817533+0000","last_became_peered":"2024-03-18T20:26:25.817533+0000","last_unstale":"2024-03-18T20:26:28.294665+0000","last_undegraded":"2024-03-18T20:26:28.294665+0000","last_fullsized":"2024-03-18T20:26:28.294665+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:22:04.370714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.799745+0000","last_change":"2024-03-18T20:26:25.368090+0000","last_active":"2024-03-18T20:26:25.799745+0000","last_peered":"2024-03-18T20:26:25.799745+0000","last_clean":"2024-03-18T20:26:25.799745+0000","last_became_active":"2024-03-18T20:26:25.367837+0000","last_became_peered":"2024-03-18T20:26:25.367837+0000","last_unstale":"2024-03-18T20:26:25.799745+0000","last_undegraded":"2024-03-18T20:26:25.799745+0000","last_fullsized":"2024-03-18T20:26:25.799745+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:19.739927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795099+0000","last_change":"2024-03-18T20:26:25.811045+0000","last_active":"2024-03-18T20:26:28.795099+0000","last_peered":"2024-03-18T20:26:28.795099+0000","last_clean":"2024-03-18T20:26:28.795099+0000","last_became_active":"2024-03-18T20:26:25.810754+0000","last_became_peered":"2024-03-18T20:26:25.810754+0000","last_unstale":"2024-03-18T20:26:28.795099+0000","last_undegraded":"2024-03-18T20:26:28.795099+0000","last_fullsized":"2024-03-18T20:26:28.795099+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:44.931187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"252'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.806470+0000","last_change":"2024-03-18T20:26:25.367689+0000","last_active":"2024-03-18T20:26:25.806470+0000","last_peered":"2024-03-18T20:26:25.806470+0000","last_clean":"2024-03-18T20:26:25.806470+0000","last_became_active":"2024-03-18T20:26:25.367431+0000","last_became_peered":"2024-03-18T20:26:25.367431+0000","last_unstale":"2024-03-18T20:26:25.806470+0000","last_undegraded":"2024-03-18T20:26:25.806470+0000","last_fullsized":"2024-03-18T20:26:25.806470+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:01.890743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"238'370","reported_seq":1297,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.792914+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:26:25.792914+0000","last_peered":"2024-03-18T20:26:25.792914+0000","last_clean":"2024-03-18T20:26:25.792914+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:26:25.792914+0000","last_undegraded":"2024-03-18T20:26:25.792914+0000","last_fullsized":"2024-03-18T20:26:25.792914+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575992,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52864,"kb_used_data":3700,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318976,"statfs":{"total":96636764160,"available":96582631424,"internally_reserved":0,"allocated":3788800,"data_stored":2233777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.57299999999999995,"15min":0.59499999999999997},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":1.1339999999999999},"last":0.627},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.58299999999999996,"15min":0.60899999999999999},"min":{"1min":0.35299999999999998,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.76800000000000002,"5min":0.84499999999999997,"15min":1.093},"last":0.52100000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.58199999999999996,"15min":0.60799999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72199999999999998,"5min":0.83099999999999996,"15min":1.4299999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.58499999999999996,"15min":0.61399999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":1.2629999999999999},"last":0.59899999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.58799999999999997,"15min":0.60899999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":1.216},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.58999999999999997,"15min":0.621},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.2490000000000001},"last":0.55400000000000005}]},{"osd":3,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.57199999999999995,"15min":0.60299999999999998},"min":{"1min":0.374,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83899999999999997,"5min":0.84699999999999998,"15min":1.4450000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.623,"15min":0.64600000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.38600000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":1.5029999999999999},"last":0.66700000000000004}]},{"osd":4,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46600000000000003,"15min":0.47699999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.76500000000000001,"5min":0.79100000000000004,"15min":0.86099999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.502,"5min":0.48899999999999999,"15min":0.5},"min":{"1min":0.34300000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.57599999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.502,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52500000000000002,"15min":0.54600000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.877,"5min":0.877,"15min":1.042},"last":0.73099999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55300000000000005,"15min":0.57699999999999996},"min":{"1min":0.39500000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":1.129},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54800000000000004,"15min":0.56799999999999995},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":1.0609999999999999},"last":0.71399999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575992,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42436,"kb_used_data":3120,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329404,"statfs":{"total":96636764160,"available":96593309696,"internally_reserved":0,"allocated":3194880,"data_stored":1643397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55400000000000005,"15min":0.59399999999999997},"min":{"1min":0.32500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.623,"5min":0.80900000000000005,"15min":1.1519999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57399999999999995,"15min":0.60399999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.53400000000000003}]},{"osd":1,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56499999999999995,"15min":0.59699999999999998},"min":{"1min":0.40400000000000003,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69899999999999995,"5min":0.79800000000000004,"15min":1.244},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57999999999999996,"15min":0.60299999999999998},"min":{"1min":0.42499999999999999,"5min":0.35399999999999998,"15min":0.33300000000000002},"max":{"1min":0.83699999999999997,"5min":0.89200000000000002,"15min":1.286},"last":0.34799999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.58699999999999997,"15min":0.623},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.69199999999999995,"5min":0.92000000000000004,"15min":1.276},"last":0.438},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57999999999999996,"15min":0.60899999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.378},"max":{"1min":1.089,"5min":1.089,"15min":1.2669999999999999},"last":0.46999999999999997}]},{"osd":3,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58099999999999996,"15min":0.621},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.59199999999999997,"15min":0.61599999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70599999999999996,"5min":1.0049999999999999,"15min":1.337},"last":0.51500000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.48899999999999999,"15min":0.48799999999999999},"min":{"1min":0.35799999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.75600000000000001,"5min":0.85799999999999998,"15min":0.88300000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.50800000000000001,"15min":0.51700000000000002},"min":{"1min":0.35899999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80300000000000005,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.41799999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.51100000000000001,"15min":0.51000000000000001},"min":{"1min":0.38300000000000001,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39600000000000002,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.874,"5min":0.92600000000000005,"15min":0.94199999999999995},"last":0.58499999999999996}]},{"osd":7,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.42399999999999999,"5min":0.38,"15min":0.34499999999999997},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.57799999999999996,"15min":0.57499999999999996},"min":{"1min":0.41399999999999998,"5min":0.36599999999999999,"15min":0.34599999999999997},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.59399999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575992,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52276,"kb_used_data":3112,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319564,"statfs":{"total":96636764160,"available":96583233536,"internally_reserved":0,"allocated":3186688,"data_stored":1643285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.56299999999999994,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.38,"5min":0.41199999999999998,"15min":0.42199999999999999},"min":{"1min":0.27200000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56499999999999995,"5min":0.67500000000000004,"15min":0.69699999999999995},"last":0.61299999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.43099999999999999,"15min":0.432},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.51500000000000001,"5min":0.64300000000000002,"15min":0.68100000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.34499999999999997,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.55500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.54800000000000004}]},{"osd":3,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.33300000000000002,"5min":0.33000000000000002,"15min":0.32300000000000001},"max":{"1min":0.58699999999999997,"5min":0.75900000000000001,"15min":0.76100000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.44,"15min":0.436},"min":{"1min":0.29199999999999998,"5min":0.28899999999999998,"15min":0.27000000000000002},"max":{"1min":0.55400000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.46400000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.66500000000000004,"15min":0.66800000000000004},"min":{"1min":0.42999999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82199999999999995,"5min":1.0680000000000001,"15min":1.2909999999999999},"last":0.746},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66900000000000004,"15min":0.67500000000000004},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.746},"last":0.73199999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.46300000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.88600000000000001,"5min":1.0389999999999999,"15min":1.6910000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.70199999999999996,"15min":0.70799999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.91100000000000003,"5min":1.109,"15min":1.2549999999999999},"last":0.626}]},{"osd":6,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.68200000000000005,"15min":0.68100000000000005},"min":{"1min":0.46800000000000003,"5min":0.45900000000000002,"15min":0.437},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.637},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.70299999999999996,"15min":0.70499999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92100000000000004,"5min":1.014,"15min":1.722},"last":0.69499999999999995}]},{"osd":7,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.68999999999999995,"15min":0.70399999999999996},"min":{"1min":0.51400000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.90300000000000002,"5min":1.02,"15min":1.2729999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.56599999999999995,"5min":0.495,"15min":0.42799999999999999},"max":{"1min":0.84199999999999997,"5min":1.0329999999999999,"15min":1.3380000000000001},"last":0.71299999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575993,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48780,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323060,"statfs":{"total":96636764160,"available":96586813440,"internally_reserved":0,"allocated":3792896,"data_stored":2235217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38300000000000001,"15min":0.39000000000000001},"min":{"1min":0.29299999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56399999999999995,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.41699999999999998,"15min":0.42599999999999999},"min":{"1min":0.28199999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.60099999999999998,"5min":0.64000000000000001,"15min":1.1850000000000001},"last":0.58799999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42899999999999999,"15min":0.437},"min":{"1min":0.28399999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67900000000000005,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.432,"15min":0.45000000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.70999999999999996},"last":0.441}]},{"osd":3,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.42099999999999999,"15min":0.42899999999999999},"min":{"1min":0.29499999999999998,"5min":0.27500000000000002,"15min":0.26100000000000001},"max":{"1min":0.58499999999999996,"5min":0.60099999999999998,"15min":0.628},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.48099999999999998,"15min":0.49299999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.32500000000000001},"max":{"1min":0.73399999999999999,"5min":0.73999999999999999,"15min":0.91500000000000004},"last":0.49099999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63200000000000001,"15min":0.65200000000000002},"min":{"1min":0.40000000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90200000000000002,"5min":0.91500000000000004,"15min":1.0029999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.58099999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.66700000000000004,"15min":0.67000000000000004},"min":{"1min":0.435,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.88500000000000001,"5min":0.97099999999999997,"15min":1.0129999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.626,"5min":0.67400000000000004,"15min":0.68000000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":1.518},"last":0.73199999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.66100000000000003,"15min":0.66300000000000003},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.1279999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.71799999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.69199999999999995,"15min":0.70499999999999996},"min":{"1min":0.47999999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.464},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69099999999999995,"15min":0.70299999999999996},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.47599999999999998},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.66100000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575993,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52284,"kb_used_data":3120,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319556,"statfs":{"total":96636764160,"available":96583225344,"internally_reserved":0,"allocated":3194880,"data_stored":1643937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36599999999999999,"15min":0.378},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.496,"5min":0.54800000000000004,"15min":0.59399999999999997},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.47399999999999998,"15min":0.48699999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.63600000000000001,"5min":0.71499999999999997,"15min":0.753},"last":0.379}]},{"osd":1,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46899999999999997,"15min":0.48199999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64300000000000002,"5min":0.72599999999999998,"15min":0.77100000000000002},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40899999999999997,"15min":0.41599999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.53800000000000003,"5min":0.59299999999999997,"15min":0.71099999999999997},"last":0.35399999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.433,"15min":0.436},"min":{"1min":0.28899999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.497,"15min":0.51700000000000002},"min":{"1min":0.33800000000000002,"5min":0.32200000000000001,"15min":0.313},"max":{"1min":0.68100000000000005,"5min":0.752,"15min":0.752},"last":0.435}]},{"osd":4,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.66300000000000003,"15min":0.65800000000000003},"min":{"1min":0.503,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":0.82499999999999996,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67500000000000004,"15min":0.66800000000000004},"min":{"1min":0.497,"5min":0.443,"15min":0.443},"max":{"1min":0.91600000000000004,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.81100000000000005}]},{"osd":5,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.67900000000000005,"15min":0.68500000000000005},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.93999999999999995,"5min":1.028,"15min":1.028},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67300000000000004,"15min":0.68000000000000005},"min":{"1min":0.47299999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.88300000000000001,"5min":0.95599999999999996,"15min":1.0629999999999999},"last":0.72199999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68000000000000005,"15min":0.69099999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86099999999999999,"5min":1.014,"15min":1.2050000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.72499999999999998,"15min":0.73199999999999998},"min":{"1min":0.53800000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.97199999999999998,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.82999999999999996}]},{"osd":7,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68899999999999995,"15min":0.70199999999999996},"min":{"1min":0.51500000000000001,"5min":0.442,"15min":0.42199999999999999},"max":{"1min":0.92900000000000005,"5min":0.99399999999999999,"15min":1.0469999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69899999999999995,"15min":0.70699999999999996},"min":{"1min":0.45500000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.93999999999999995,"5min":1.371,"15min":1.371},"last":0.84099999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575993,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52300,"kb_used_data":3136,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319540,"statfs":{"total":96636764160,"available":96583208960,"internally_reserved":0,"allocated":3211264,"data_stored":1645487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.38,"15min":0.39500000000000002},"min":{"1min":0.23100000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.48899999999999999,"5min":0.66500000000000004,"15min":0.98099999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.39100000000000001,"15min":0.40999999999999998},"min":{"1min":0.26100000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.42699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.35199999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.374,"15min":0.40300000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.47699999999999998,"5min":0.63200000000000001,"15min":0.64000000000000001},"last":0.308},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.44400000000000001,"15min":0.46200000000000002},"min":{"1min":0.315,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.50900000000000001,"5min":0.65500000000000003,"15min":0.73099999999999998},"last":0.40100000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.436,"15min":0.46000000000000002},"min":{"1min":0.29999999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53800000000000003,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.46000000000000002,"15min":0.48799999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.51700000000000002,"5min":0.66100000000000003,"15min":0.73099999999999998},"last":0.42799999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.61199999999999999,"15min":0.64500000000000002},"min":{"1min":0.41599999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.63300000000000001,"15min":0.66900000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.51800000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.67000000000000004,"15min":0.69299999999999995},"min":{"1min":0.42999999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.70299999999999996,"15min":0.72299999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.1120000000000001,"5min":1.171,"15min":1.171},"last":0.59799999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67000000000000004,"15min":0.70299999999999996},"min":{"1min":0.45400000000000001,"5min":0.436,"15min":0.42499999999999999},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.69999999999999996,"15min":0.72599999999999998},"min":{"1min":0.47399999999999998,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":1.22,"5min":1.22,"15min":1.2410000000000001},"last":0.63400000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.68100000000000005,"15min":0.71299999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.624,"5min":0.66000000000000003,"15min":0.69399999999999995},"min":{"1min":0.41099999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.64700000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575993,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48184,"kb_used_data":3116,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323656,"statfs":{"total":96636764160,"available":96587423744,"internally_reserved":0,"allocated":3190784,"data_stored":1643355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.57899999999999996,"15min":0.59299999999999997},"min":{"1min":0.34300000000000003,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65300000000000002,"5min":0.77500000000000002,"15min":0.81799999999999995},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57799999999999996,"15min":0.59799999999999998},"min":{"1min":0.40300000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69999999999999996,"5min":0.79600000000000004,"15min":0.84199999999999997},"last":0.44900000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59999999999999998,"15min":0.61399999999999999},"min":{"1min":0.438,"5min":0.318,"15min":0.318},"max":{"1min":0.73199999999999998,"5min":0.81899999999999995,"15min":0.88300000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56999999999999995,"15min":0.58499999999999996},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.71199999999999997,"5min":0.79200000000000004,"15min":0.96499999999999997},"last":0.48399999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58299999999999996,"15min":0.59899999999999998},"min":{"1min":0.439,"5min":0.39500000000000002,"15min":0.39200000000000002},"max":{"1min":0.68600000000000005,"5min":0.80500000000000005,"15min":0.82599999999999996},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61099999999999999,"15min":0.61799999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.38500000000000001},"max":{"1min":0.81399999999999995,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.63700000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.40799999999999997,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.623},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.86199999999999999},"last":0.59799999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.318},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.498,"15min":0.504},"min":{"1min":0.35199999999999998,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.85499999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.41099999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.499,"15min":0.505},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.53500000000000003}]},{"osd":7,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54100000000000004,"15min":0.54800000000000004},"min":{"1min":0.435,"5min":0.38,"15min":0.35699999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.57199999999999995,"15min":0.57999999999999996},"min":{"1min":0.41899999999999998,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.5}]}]},{"osd":5,"up_from":13,"seq":55834575992,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48180,"kb_used_data":3112,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323660,"statfs":{"total":96636764160,"available":96587427840,"internally_reserved":0,"allocated":3186688,"data_stored":1643285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.57699999999999996,"15min":0.59799999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.31900000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56999999999999995,"15min":0.60099999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82599999999999996,"15min":1.3100000000000001},"last":0.498}]},{"osd":1,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58199999999999996,"15min":0.60199999999999998},"min":{"1min":0.36599999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57499999999999996,"15min":0.59799999999999998},"min":{"1min":0.40500000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":1.018},"last":0.64200000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.57899999999999996,"15min":0.59899999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.99299999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58199999999999996,"15min":0.60899999999999999},"min":{"1min":0.41599999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.60599999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59299999999999997,"15min":0.60999999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.2430000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.61399999999999999,"15min":0.64200000000000002},"min":{"1min":0.40600000000000003,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":1.161},"last":0.69599999999999995}]},{"osd":4,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.49299999999999999,"15min":0.47999999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52200000000000002,"15min":0.51100000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.73399999999999999}]},{"osd":6,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.503,"15min":0.504},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.32800000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":0.57099999999999995}]},{"osd":7,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56000000000000005,"15min":0.55100000000000005},"min":{"1min":0.32800000000000001,"5min":0.317,"15min":0.317},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56999999999999995,"15min":0.57299999999999995},"min":{"1min":0.34300000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.72399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:26:30.263 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:26:30.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.419+0000 7f8a05a15700 1 Processor -- start 2024-03-18T20:26:30.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.419+0000 7f8a05a15700 1 -- start start 2024-03-18T20:26:30.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.420+0000 7f8a05a15700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8a00099a40 0x7f8a0006ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.420+0000 7f8a05a15700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a000d05a0 0x7f8a00099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.420+0000 7f8a05a15700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a000cf7e0 con 0x7f8a0009a360 2024-03-18T20:26:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.420+0000 7f8a05a15700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a000cfac0 con 0x7f8a000d05a0 2024-03-18T20:26:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.420+0000 7f8a05a15700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a000cfda0 con 0x7f8a00099a40 2024-03-18T20:26:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.420+0000 7f89fe7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a000d05a0 0x7f8a00099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.420+0000 7f89ff7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8a0009a360 0x7f8a0006e880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53534/0 (socket says 172.21.15.1:53534) 2024-03-18T20:26:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.420+0000 7f89fe7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a000d05a0 0x7f8a00099e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46598/0 (socket says 172.21.15.1:46598) 2024-03-18T20:26:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.420+0000 7f89ff7fe700 1 -- 172.21.15.1:0/732413469 learned_addr learned my addr 172.21.15.1:0/732413469 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.420+0000 7f89fe7fc700 1 -- 172.21.15.1:0/732413469 >> v2:172.21.15.165:3300/0 conn(0x7f8a00099a40 msgr2=0x7f8a0006ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.420+0000 7f89feffd700 1 --2- 172.21.15.1:0/732413469 >> v2:172.21.15.165:3300/0 conn(0x7f8a00099a40 0x7f8a0006ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.421+0000 7f89fe7fc700 1 --2- 172.21.15.1:0/732413469 >> v2:172.21.15.165:3300/0 conn(0x7f8a00099a40 0x7f8a0006ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.421+0000 7f89fe7fc700 1 -- 172.21.15.1:0/732413469 >> v1:172.21.15.1:6789/0 conn(0x7f8a0009a360 legacy=0x7f8a0006e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.421+0000 7f89fdffb700 1 -- 172.21.15.1:0/732413469 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918929007 0 0) 0x7f8a000cf7e0 con 0x7f8a0009a360 2024-03-18T20:26:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.421+0000 7f89fe7fc700 1 -- 172.21.15.1:0/732413469 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a000d0080 con 0x7f8a000d05a0 2024-03-18T20:26:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.421+0000 7f89fe7fc700 1 --2- 172.21.15.1:0/732413469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a000d05a0 0x7f8a00099e20 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7f8a00071dd0 tx=0x7f89f0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15bae881ed021385 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.421+0000 7f89fdffb700 1 -- 172.21.15.1:0/732413469 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f0011830 con 0x7f8a000d05a0 2024-03-18T20:26:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.421+0000 7f89fdffb700 1 -- 172.21.15.1:0/732413469 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f0011990 con 0x7f8a000d05a0 2024-03-18T20:26:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.421+0000 7f89fdffb700 1 -- 172.21.15.1:0/732413469 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f0011d20 con 0x7f8a000d05a0 2024-03-18T20:26:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.421+0000 7f8a05a15700 1 -- 172.21.15.1:0/732413469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a000d05a0 msgr2=0x7f8a00099e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.421+0000 7f8a05a15700 1 --2- 172.21.15.1:0/732413469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a000d05a0 0x7f8a00099e20 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7f8a00071dd0 tx=0x7f89f0009b20 comp rx=0 tx=0).stop 2024-03-18T20:26:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.421+0000 7f8a05a15700 1 -- 172.21.15.1:0/732413469 shutdown_connections 2024-03-18T20:26:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.421+0000 7f8a05a15700 1 --2- 172.21.15.1:0/732413469 >> v2:172.21.15.165:3300/0 conn(0x7f8a00099a40 0x7f8a0006ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.421+0000 7f8a05a15700 1 --2- 172.21.15.1:0/732413469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a000d05a0 0x7f8a00099e20 unknown :-1 s=CLOSED pgs=2816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.421+0000 7f8a05a15700 1 -- 172.21.15.1:0/732413469 >> 172.21.15.1:0/732413469 conn(0x7f8a000b2cb0 msgr2=0x7f8a000b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.421+0000 7f8a05a15700 1 -- 172.21.15.1:0/732413469 shutdown_connections 2024-03-18T20:26:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.421+0000 7f8a05a15700 1 -- 172.21.15.1:0/732413469 wait complete. 2024-03-18T20:26:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.423+0000 7f8a05a15700 1 Processor -- start 2024-03-18T20:26:30.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.423+0000 7f8a05a15700 1 -- start start 2024-03-18T20:26:30.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f8a05a15700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a00099a40 0x7f8a0015a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f8a05a15700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a0009a360 0x7f8a000c72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f8a05a15700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a000d05a0 0x7f8a000af2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f8a05a15700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a000c8500 con 0x7f8a0009a360 2024-03-18T20:26:30.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f8a05a15700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0015abe0 con 0x7f8a000d05a0 2024-03-18T20:26:30.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f8a05a15700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0015aec0 con 0x7f8a00099a40 2024-03-18T20:26:30.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f89fe7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a0009a360 0x7f8a000c72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f89fe7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a0009a360 0x7f8a000c72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46602/0 (socket says 172.21.15.1:46602) 2024-03-18T20:26:30.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f89fe7fc700 1 -- 172.21.15.1:0/3008319202 learned_addr learned my addr 172.21.15.1:0/3008319202 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:30.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f89feffd700 1 --2- 172.21.15.1:0/3008319202 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a00099a40 0x7f8a0015a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f89ff7fe700 1 --2- 172.21.15.1:0/3008319202 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a000d05a0 0x7f8a000af2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f89fe7fc700 1 -- 172.21.15.1:0/3008319202 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a00099a40 msgr2=0x7f8a0015a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f89fe7fc700 1 --2- 172.21.15.1:0/3008319202 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a00099a40 0x7f8a0015a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f89fe7fc700 1 -- 172.21.15.1:0/3008319202 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a000d05a0 msgr2=0x7f8a000af2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f89fe7fc700 1 --2- 172.21.15.1:0/3008319202 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a000d05a0 0x7f8a000af2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f89fe7fc700 1 -- 172.21.15.1:0/3008319202 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f00094a0 con 0x7f8a0009a360 2024-03-18T20:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f89ff7fe700 1 --2- 172.21.15.1:0/3008319202 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a000d05a0 0x7f8a000af2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f89fe7fc700 1 --2- 172.21.15.1:0/3008319202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a0009a360 0x7f8a000c72e0 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 crypto rx=0x7f89f0000c00 tx=0x7f89f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f89f7fff700 1 -- 172.21.15.1:0/3008319202 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f00038a0 con 0x7f8a0009a360 2024-03-18T20:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f89f7fff700 1 -- 172.21.15.1:0/3008319202 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f001e070 con 0x7f8a0009a360 2024-03-18T20:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f8a05a15700 1 -- 172.21.15.1:0/3008319202 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a000c8750 con 0x7f8a0009a360 2024-03-18T20:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f89f7fff700 1 -- 172.21.15.1:0/3008319202 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f0003c40 con 0x7f8a0009a360 2024-03-18T20:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.424+0000 7f8a05a15700 1 -- 172.21.15.1:0/3008319202 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a0015a590 con 0x7f8a0009a360 2024-03-18T20:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.425+0000 7f8a05a15700 1 -- 172.21.15.1:0/3008319202 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a00068240 con 0x7f8a0009a360 2024-03-18T20:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.426+0000 7f89f7fff700 1 -- 172.21.15.1:0/3008319202 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f89f002d450 con 0x7f8a0009a360 2024-03-18T20:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.426+0000 7f89f7fff700 1 --2- 172.21.15.1:0/3008319202 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f89e006aed0 0x7f89e006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.426+0000 7f89f7fff700 1 -- 172.21.15.1:0/3008319202 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f89f008a9f0 con 0x7f8a0009a360 2024-03-18T20:26:30.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.426+0000 7f89feffd700 1 --2- 172.21.15.1:0/3008319202 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f89e006aed0 0x7f89e006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.427+0000 7f89feffd700 1 --2- 172.21.15.1:0/3008319202 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f89e006aed0 0x7f89e006d390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f89e800a990 tx=0x7f89e8008040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:30.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.431+0000 7f89f7fff700 1 -- 172.21.15.1:0/3008319202 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f89f0055750 con 0x7f8a0009a360 2024-03-18T20:26:30.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.628+0000 7f8a05a15700 1 -- 172.21.15.1:0/3008319202 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8a000cf490 con 0x7f89e006aed0 2024-03-18T20:26:30.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.636+0000 7f89f7fff700 1 -- 172.21.15.1:0/3008319202 <== mgr.12057 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290038 (secure 0 0 0) 0x7f89e800a660 con 0x7f89e006aed0 2024-03-18T20:26:30.644 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.640+0000 7f8a05a15700 1 -- 172.21.15.1:0/3008319202 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f89e006aed0 msgr2=0x7f89e006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.640+0000 7f8a05a15700 1 --2- 172.21.15.1:0/3008319202 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f89e006aed0 0x7f89e006d390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f89e800a990 tx=0x7f89e8008040 comp rx=0 tx=0).stop 2024-03-18T20:26:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.640+0000 7f8a05a15700 1 -- 172.21.15.1:0/3008319202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a0009a360 msgr2=0x7f8a000c72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.640+0000 7f8a05a15700 1 --2- 172.21.15.1:0/3008319202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a0009a360 0x7f8a000c72e0 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 crypto rx=0x7f89f0000c00 tx=0x7f89f0004750 comp rx=0 tx=0).stop 2024-03-18T20:26:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.640+0000 7f8a05a15700 1 -- 172.21.15.1:0/3008319202 shutdown_connections 2024-03-18T20:26:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.640+0000 7f8a05a15700 1 --2- 172.21.15.1:0/3008319202 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f89e006aed0 0x7f89e006d390 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.640+0000 7f8a05a15700 1 --2- 172.21.15.1:0/3008319202 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a00099a40 0x7f8a0015a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.640+0000 7f8a05a15700 1 --2- 172.21.15.1:0/3008319202 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a000d05a0 0x7f8a000af2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.640+0000 7f8a05a15700 1 --2- 172.21.15.1:0/3008319202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a0009a360 0x7f8a000c72e0 unknown :-1 s=CLOSED pgs=2817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.640+0000 7f8a05a15700 1 -- 172.21.15.1:0/3008319202 >> 172.21.15.1:0/3008319202 conn(0x7f8a000b2cb0 msgr2=0x7f8a000b3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:30.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.640+0000 7f8a05a15700 1 -- 172.21.15.1:0/3008319202 shutdown_connections 2024-03-18T20:26:30.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.640+0000 7f8a05a15700 1 -- 172.21.15.1:0/3008319202 wait complete. 2024-03-18T20:26:30.647 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:26:30.656 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-03-18T20:26:29.876502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":663,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":408,"ondisk_log_size":408,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":181,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":397304,"kb_used_data":26120,"kb_used_omap":12,"kb_used_meta":371059,"kb_avail":754577416,"statfs":{"total":773094113280,"available":772687273984,"internally_reserved":0,"allocated":26746880,"data_stored":14331740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":379964985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.529114"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795771+0000","last_change":"2024-03-18T20:26:25.816062+0000","last_active":"2024-03-18T20:26:28.795771+0000","last_peered":"2024-03-18T20:26:28.795771+0000","last_clean":"2024-03-18T20:26:28.795771+0000","last_became_active":"2024-03-18T20:26:25.815866+0000","last_became_peered":"2024-03-18T20:26:25.815866+0000","last_unstale":"2024-03-18T20:26:28.795771+0000","last_undegraded":"2024-03-18T20:26:28.795771+0000","last_fullsized":"2024-03-18T20:26:28.795771+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:19:53.852641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795813+0000","last_change":"2024-03-18T20:26:25.806337+0000","last_active":"2024-03-18T20:26:28.795813+0000","last_peered":"2024-03-18T20:26:28.795813+0000","last_clean":"2024-03-18T20:26:28.795813+0000","last_became_active":"2024-03-18T20:26:25.806016+0000","last_became_peered":"2024-03-18T20:26:25.806016+0000","last_unstale":"2024-03-18T20:26:28.795813+0000","last_undegraded":"2024-03-18T20:26:28.795813+0000","last_fullsized":"2024-03-18T20:26:28.795813+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:33:55.816835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811592+0000","last_change":"2024-03-18T20:26:25.815050+0000","last_active":"2024-03-18T20:26:27.811592+0000","last_peered":"2024-03-18T20:26:27.811592+0000","last_clean":"2024-03-18T20:26:27.811592+0000","last_became_active":"2024-03-18T20:26:25.814617+0000","last_became_peered":"2024-03-18T20:26:25.814617+0000","last_unstale":"2024-03-18T20:26:27.811592+0000","last_undegraded":"2024-03-18T20:26:27.811592+0000","last_fullsized":"2024-03-18T20:26:27.811592+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:04:32.495090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811543+0000","last_change":"2024-03-18T20:26:25.812709+0000","last_active":"2024-03-18T20:26:27.811543+0000","last_peered":"2024-03-18T20:26:27.811543+0000","last_clean":"2024-03-18T20:26:27.811543+0000","last_became_active":"2024-03-18T20:26:25.812420+0000","last_became_peered":"2024-03-18T20:26:25.812420+0000","last_unstale":"2024-03-18T20:26:27.811543+0000","last_undegraded":"2024-03-18T20:26:27.811543+0000","last_fullsized":"2024-03-18T20:26:27.811543+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:52.955865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.801652+0000","last_change":"2024-03-18T20:26:25.811447+0000","last_active":"2024-03-18T20:26:26.801652+0000","last_peered":"2024-03-18T20:26:26.801652+0000","last_clean":"2024-03-18T20:26:26.801652+0000","last_became_active":"2024-03-18T20:26:25.811144+0000","last_became_peered":"2024-03-18T20:26:25.811144+0000","last_unstale":"2024-03-18T20:26:26.801652+0000","last_undegraded":"2024-03-18T20:26:26.801652+0000","last_fullsized":"2024-03-18T20:26:26.801652+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:54:38.189655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795458+0000","last_change":"2024-03-18T20:26:25.804382+0000","last_active":"2024-03-18T20:26:28.795458+0000","last_peered":"2024-03-18T20:26:28.795458+0000","last_clean":"2024-03-18T20:26:28.795458+0000","last_became_active":"2024-03-18T20:26:25.804096+0000","last_became_peered":"2024-03-18T20:26:25.804096+0000","last_unstale":"2024-03-18T20:26:28.795458+0000","last_undegraded":"2024-03-18T20:26:28.795458+0000","last_fullsized":"2024-03-18T20:26:28.795458+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:23.578829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795206+0000","last_change":"2024-03-18T20:26:25.806198+0000","last_active":"2024-03-18T20:26:28.795206+0000","last_peered":"2024-03-18T20:26:28.795206+0000","last_clean":"2024-03-18T20:26:28.795206+0000","last_became_active":"2024-03-18T20:26:25.805918+0000","last_became_peered":"2024-03-18T20:26:25.805918+0000","last_unstale":"2024-03-18T20:26:28.795206+0000","last_undegraded":"2024-03-18T20:26:28.795206+0000","last_fullsized":"2024-03-18T20:26:28.795206+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:11.977302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.801441+0000","last_change":"2024-03-18T20:26:25.805458+0000","last_active":"2024-03-18T20:26:26.801441+0000","last_peered":"2024-03-18T20:26:26.801441+0000","last_clean":"2024-03-18T20:26:26.801441+0000","last_became_active":"2024-03-18T20:26:25.805176+0000","last_became_peered":"2024-03-18T20:26:25.805176+0000","last_unstale":"2024-03-18T20:26:26.801441+0000","last_undegraded":"2024-03-18T20:26:26.801441+0000","last_fullsized":"2024-03-18T20:26:26.801441+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:06:22.720260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795838+0000","last_change":"2024-03-18T20:26:25.815665+0000","last_active":"2024-03-18T20:26:28.795838+0000","last_peered":"2024-03-18T20:26:28.795838+0000","last_clean":"2024-03-18T20:26:28.795838+0000","last_became_active":"2024-03-18T20:26:25.815259+0000","last_became_peered":"2024-03-18T20:26:25.815259+0000","last_unstale":"2024-03-18T20:26:28.795838+0000","last_undegraded":"2024-03-18T20:26:28.795838+0000","last_fullsized":"2024-03-18T20:26:28.795838+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:39:11.551926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811680+0000","last_change":"2024-03-18T20:26:25.805234+0000","last_active":"2024-03-18T20:26:27.811680+0000","last_peered":"2024-03-18T20:26:27.811680+0000","last_clean":"2024-03-18T20:26:27.811680+0000","last_became_active":"2024-03-18T20:26:25.805056+0000","last_became_peered":"2024-03-18T20:26:25.805056+0000","last_unstale":"2024-03-18T20:26:27.811680+0000","last_undegraded":"2024-03-18T20:26:27.811680+0000","last_fullsized":"2024-03-18T20:26:27.811680+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:50.003103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811626+0000","last_change":"2024-03-18T20:26:25.813598+0000","last_active":"2024-03-18T20:26:27.811626+0000","last_peered":"2024-03-18T20:26:27.811626+0000","last_clean":"2024-03-18T20:26:27.811626+0000","last_became_active":"2024-03-18T20:26:25.812093+0000","last_became_peered":"2024-03-18T20:26:25.812093+0000","last_unstale":"2024-03-18T20:26:27.811626+0000","last_undegraded":"2024-03-18T20:26:27.811626+0000","last_fullsized":"2024-03-18T20:26:27.811626+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:38.417379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795610+0000","last_change":"2024-03-18T20:26:25.811095+0000","last_active":"2024-03-18T20:26:28.795610+0000","last_peered":"2024-03-18T20:26:28.795610+0000","last_clean":"2024-03-18T20:26:28.795610+0000","last_became_active":"2024-03-18T20:26:25.810796+0000","last_became_peered":"2024-03-18T20:26:25.810796+0000","last_unstale":"2024-03-18T20:26:28.795610+0000","last_undegraded":"2024-03-18T20:26:28.795610+0000","last_fullsized":"2024-03-18T20:26:28.795610+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:30:08.833713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.294453+0000","last_change":"2024-03-18T20:26:25.815198+0000","last_active":"2024-03-18T20:26:28.294453+0000","last_peered":"2024-03-18T20:26:28.294453+0000","last_clean":"2024-03-18T20:26:28.294453+0000","last_became_active":"2024-03-18T20:26:25.814922+0000","last_became_peered":"2024-03-18T20:26:25.814922+0000","last_unstale":"2024-03-18T20:26:28.294453+0000","last_undegraded":"2024-03-18T20:26:28.294453+0000","last_fullsized":"2024-03-18T20:26:28.294453+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:30:14.320846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.802738+0000","last_change":"2024-03-18T20:26:25.809230+0000","last_active":"2024-03-18T20:26:26.802738+0000","last_peered":"2024-03-18T20:26:26.802738+0000","last_clean":"2024-03-18T20:26:26.802738+0000","last_became_active":"2024-03-18T20:26:25.808941+0000","last_became_peered":"2024-03-18T20:26:25.808941+0000","last_unstale":"2024-03-18T20:26:26.802738+0000","last_undegraded":"2024-03-18T20:26:26.802738+0000","last_fullsized":"2024-03-18T20:26:26.802738+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:34:05.214460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795265+0000","last_change":"2024-03-18T20:26:25.798972+0000","last_active":"2024-03-18T20:26:28.795265+0000","last_peered":"2024-03-18T20:26:28.795265+0000","last_clean":"2024-03-18T20:26:28.795265+0000","last_became_active":"2024-03-18T20:26:25.798736+0000","last_became_peered":"2024-03-18T20:26:25.798736+0000","last_unstale":"2024-03-18T20:26:28.795265+0000","last_undegraded":"2024-03-18T20:26:28.795265+0000","last_fullsized":"2024-03-18T20:26:28.795265+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:56.892630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811302+0000","last_change":"2024-03-18T20:26:25.799408+0000","last_active":"2024-03-18T20:26:27.811302+0000","last_peered":"2024-03-18T20:26:27.811302+0000","last_clean":"2024-03-18T20:26:27.811302+0000","last_became_active":"2024-03-18T20:26:25.798788+0000","last_became_peered":"2024-03-18T20:26:25.798788+0000","last_unstale":"2024-03-18T20:26:27.811302+0000","last_undegraded":"2024-03-18T20:26:27.811302+0000","last_fullsized":"2024-03-18T20:26:27.811302+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:34:05.214460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.294748+0000","last_change":"2024-03-18T20:26:25.811390+0000","last_active":"2024-03-18T20:26:28.294748+0000","last_peered":"2024-03-18T20:26:28.294748+0000","last_clean":"2024-03-18T20:26:28.294748+0000","last_became_active":"2024-03-18T20:26:25.811081+0000","last_became_peered":"2024-03-18T20:26:25.811081+0000","last_unstale":"2024-03-18T20:26:28.294748+0000","last_undegraded":"2024-03-18T20:26:28.294748+0000","last_fullsized":"2024-03-18T20:26:28.294748+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:33:55.816835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811789+0000","last_change":"2024-03-18T20:26:25.799778+0000","last_active":"2024-03-18T20:26:27.811789+0000","last_peered":"2024-03-18T20:26:27.811789+0000","last_clean":"2024-03-18T20:26:27.811789+0000","last_became_active":"2024-03-18T20:26:25.799517+0000","last_became_peered":"2024-03-18T20:26:25.799517+0000","last_unstale":"2024-03-18T20:26:27.811789+0000","last_undegraded":"2024-03-18T20:26:27.811789+0000","last_fullsized":"2024-03-18T20:26:27.811789+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:32:34.214872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795549+0000","last_change":"2024-03-18T20:26:25.812369+0000","last_active":"2024-03-18T20:26:28.795549+0000","last_peered":"2024-03-18T20:26:28.795549+0000","last_clean":"2024-03-18T20:26:28.795549+0000","last_became_active":"2024-03-18T20:26:25.812082+0000","last_became_peered":"2024-03-18T20:26:25.812082+0000","last_unstale":"2024-03-18T20:26:28.795549+0000","last_undegraded":"2024-03-18T20:26:28.795549+0000","last_fullsized":"2024-03-18T20:26:28.795549+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:43:00.585492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.796205+0000","last_change":"2024-03-18T20:26:25.807591+0000","last_active":"2024-03-18T20:26:28.796205+0000","last_peered":"2024-03-18T20:26:28.796205+0000","last_clean":"2024-03-18T20:26:28.796205+0000","last_became_active":"2024-03-18T20:26:25.807349+0000","last_became_peered":"2024-03-18T20:26:25.807349+0000","last_unstale":"2024-03-18T20:26:28.796205+0000","last_undegraded":"2024-03-18T20:26:28.796205+0000","last_fullsized":"2024-03-18T20:26:28.796205+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:58.931342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.794838+0000","last_change":"2024-03-18T20:26:25.799427+0000","last_active":"2024-03-18T20:26:28.794838+0000","last_peered":"2024-03-18T20:26:28.794838+0000","last_clean":"2024-03-18T20:26:28.794838+0000","last_became_active":"2024-03-18T20:26:25.799041+0000","last_became_peered":"2024-03-18T20:26:25.799041+0000","last_unstale":"2024-03-18T20:26:28.794838+0000","last_undegraded":"2024-03-18T20:26:28.794838+0000","last_fullsized":"2024-03-18T20:26:28.794838+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:35.919143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.294422+0000","last_change":"2024-03-18T20:26:25.818643+0000","last_active":"2024-03-18T20:26:28.294422+0000","last_peered":"2024-03-18T20:26:28.294422+0000","last_clean":"2024-03-18T20:26:28.294422+0000","last_became_active":"2024-03-18T20:26:25.818377+0000","last_became_peered":"2024-03-18T20:26:25.818377+0000","last_unstale":"2024-03-18T20:26:28.294422+0000","last_undegraded":"2024-03-18T20:26:28.294422+0000","last_fullsized":"2024-03-18T20:26:28.294422+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:44.809054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795391+0000","last_change":"2024-03-18T20:26:25.805834+0000","last_active":"2024-03-18T20:26:28.795391+0000","last_peered":"2024-03-18T20:26:28.795391+0000","last_clean":"2024-03-18T20:26:28.795391+0000","last_became_active":"2024-03-18T20:26:25.805491+0000","last_became_peered":"2024-03-18T20:26:25.805491+0000","last_unstale":"2024-03-18T20:26:28.795391+0000","last_undegraded":"2024-03-18T20:26:28.795391+0000","last_fullsized":"2024-03-18T20:26:28.795391+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:23.305150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.294230+0000","last_change":"2024-03-18T20:26:25.809091+0000","last_active":"2024-03-18T20:26:28.294230+0000","last_peered":"2024-03-18T20:26:28.294230+0000","last_clean":"2024-03-18T20:26:28.294230+0000","last_became_active":"2024-03-18T20:26:25.808800+0000","last_became_peered":"2024-03-18T20:26:25.808800+0000","last_unstale":"2024-03-18T20:26:28.294230+0000","last_undegraded":"2024-03-18T20:26:28.294230+0000","last_fullsized":"2024-03-18T20:26:28.294230+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:32.033370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795464+0000","last_change":"2024-03-18T20:26:25.798738+0000","last_active":"2024-03-18T20:26:28.795464+0000","last_peered":"2024-03-18T20:26:28.795464+0000","last_clean":"2024-03-18T20:26:28.795464+0000","last_became_active":"2024-03-18T20:26:25.798480+0000","last_became_peered":"2024-03-18T20:26:25.798480+0000","last_unstale":"2024-03-18T20:26:28.795464+0000","last_undegraded":"2024-03-18T20:26:28.795464+0000","last_fullsized":"2024-03-18T20:26:28.795464+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:02.900783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795007+0000","last_change":"2024-03-18T20:26:25.806356+0000","last_active":"2024-03-18T20:26:28.795007+0000","last_peered":"2024-03-18T20:26:28.795007+0000","last_clean":"2024-03-18T20:26:28.795007+0000","last_became_active":"2024-03-18T20:26:25.805895+0000","last_became_peered":"2024-03-18T20:26:25.805895+0000","last_unstale":"2024-03-18T20:26:28.795007+0000","last_undegraded":"2024-03-18T20:26:28.795007+0000","last_fullsized":"2024-03-18T20:26:28.795007+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:06:16.285934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.801835+0000","last_change":"2024-03-18T20:26:25.806860+0000","last_active":"2024-03-18T20:26:26.801835+0000","last_peered":"2024-03-18T20:26:26.801835+0000","last_clean":"2024-03-18T20:26:26.801835+0000","last_became_active":"2024-03-18T20:26:25.806650+0000","last_became_peered":"2024-03-18T20:26:25.806650+0000","last_unstale":"2024-03-18T20:26:26.801835+0000","last_undegraded":"2024-03-18T20:26:26.801835+0000","last_fullsized":"2024-03-18T20:26:26.801835+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:06:16.285934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795827+0000","last_change":"2024-03-18T20:26:25.815611+0000","last_active":"2024-03-18T20:26:28.795827+0000","last_peered":"2024-03-18T20:26:28.795827+0000","last_clean":"2024-03-18T20:26:28.795827+0000","last_became_active":"2024-03-18T20:26:25.815172+0000","last_became_peered":"2024-03-18T20:26:25.815172+0000","last_unstale":"2024-03-18T20:26:28.795827+0000","last_undegraded":"2024-03-18T20:26:28.795827+0000","last_fullsized":"2024-03-18T20:26:28.795827+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:48:15.078756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.802865+0000","last_change":"2024-03-18T20:26:25.810485+0000","last_active":"2024-03-18T20:26:26.802865+0000","last_peered":"2024-03-18T20:26:26.802865+0000","last_clean":"2024-03-18T20:26:26.802865+0000","last_became_active":"2024-03-18T20:26:25.810235+0000","last_became_peered":"2024-03-18T20:26:25.810235+0000","last_unstale":"2024-03-18T20:26:26.802865+0000","last_undegraded":"2024-03-18T20:26:26.802865+0000","last_fullsized":"2024-03-18T20:26:26.802865+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:59.140240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.802735+0000","last_change":"2024-03-18T20:26:25.807504+0000","last_active":"2024-03-18T20:26:26.802735+0000","last_peered":"2024-03-18T20:26:26.802735+0000","last_clean":"2024-03-18T20:26:26.802735+0000","last_became_active":"2024-03-18T20:26:25.807257+0000","last_became_peered":"2024-03-18T20:26:25.807257+0000","last_unstale":"2024-03-18T20:26:26.802735+0000","last_undegraded":"2024-03-18T20:26:26.802735+0000","last_fullsized":"2024-03-18T20:26:26.802735+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:40.456111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795466+0000","last_change":"2024-03-18T20:26:25.796362+0000","last_active":"2024-03-18T20:26:28.795466+0000","last_peered":"2024-03-18T20:26:28.795466+0000","last_clean":"2024-03-18T20:26:28.795466+0000","last_became_active":"2024-03-18T20:26:25.796061+0000","last_became_peered":"2024-03-18T20:26:25.796061+0000","last_unstale":"2024-03-18T20:26:28.795466+0000","last_undegraded":"2024-03-18T20:26:28.795466+0000","last_fullsized":"2024-03-18T20:26:28.795466+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:28.602198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.794936+0000","last_change":"2024-03-18T20:26:25.815799+0000","last_active":"2024-03-18T20:26:28.794936+0000","last_peered":"2024-03-18T20:26:28.794936+0000","last_clean":"2024-03-18T20:26:28.794936+0000","last_became_active":"2024-03-18T20:26:25.815392+0000","last_became_peered":"2024-03-18T20:26:25.815392+0000","last_unstale":"2024-03-18T20:26:28.794936+0000","last_undegraded":"2024-03-18T20:26:28.794936+0000","last_fullsized":"2024-03-18T20:26:28.794936+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:21.319444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795156+0000","last_change":"2024-03-18T20:26:25.808089+0000","last_active":"2024-03-18T20:26:28.795156+0000","last_peered":"2024-03-18T20:26:28.795156+0000","last_clean":"2024-03-18T20:26:28.795156+0000","last_became_active":"2024-03-18T20:26:25.806688+0000","last_became_peered":"2024-03-18T20:26:25.806688+0000","last_unstale":"2024-03-18T20:26:28.795156+0000","last_undegraded":"2024-03-18T20:26:28.795156+0000","last_fullsized":"2024-03-18T20:26:28.795156+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:22.598187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"252'4","reported_seq":15,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.818510+0000","last_change":"2024-03-18T20:26:25.369142+0000","last_active":"2024-03-18T20:26:25.818510+0000","last_peered":"2024-03-18T20:26:25.818510+0000","last_clean":"2024-03-18T20:26:25.818510+0000","last_became_active":"2024-03-18T20:26:25.368879+0000","last_became_peered":"2024-03-18T20:26:25.368879+0000","last_unstale":"2024-03-18T20:26:25.818510+0000","last_undegraded":"2024-03-18T20:26:25.818510+0000","last_fullsized":"2024-03-18T20:26:25.818510+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:42.709929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795253+0000","last_change":"2024-03-18T20:26:25.812175+0000","last_active":"2024-03-18T20:26:28.795253+0000","last_peered":"2024-03-18T20:26:28.795253+0000","last_clean":"2024-03-18T20:26:28.795253+0000","last_became_active":"2024-03-18T20:26:25.811898+0000","last_became_peered":"2024-03-18T20:26:25.811898+0000","last_unstale":"2024-03-18T20:26:28.795253+0000","last_undegraded":"2024-03-18T20:26:28.795253+0000","last_fullsized":"2024-03-18T20:26:28.795253+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:45:00.418514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"252'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.809720+0000","last_change":"2024-03-18T20:26:25.371848+0000","last_active":"2024-03-18T20:26:25.809720+0000","last_peered":"2024-03-18T20:26:25.809720+0000","last_clean":"2024-03-18T20:26:25.809720+0000","last_became_active":"2024-03-18T20:26:25.371656+0000","last_became_peered":"2024-03-18T20:26:25.371656+0000","last_unstale":"2024-03-18T20:26:25.809720+0000","last_undegraded":"2024-03-18T20:26:25.809720+0000","last_fullsized":"2024-03-18T20:26:25.809720+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:27.592158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.796244+0000","last_change":"2024-03-18T20:26:25.809815+0000","last_active":"2024-03-18T20:26:28.796244+0000","last_peered":"2024-03-18T20:26:28.796244+0000","last_clean":"2024-03-18T20:26:28.796244+0000","last_became_active":"2024-03-18T20:26:25.809504+0000","last_became_peered":"2024-03-18T20:26:25.809504+0000","last_unstale":"2024-03-18T20:26:28.796244+0000","last_undegraded":"2024-03-18T20:26:28.796244+0000","last_fullsized":"2024-03-18T20:26:28.796244+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:42:34.548735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"252'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.811148+0000","last_change":"2024-03-18T20:26:25.366016+0000","last_active":"2024-03-18T20:26:25.811148+0000","last_peered":"2024-03-18T20:26:25.811148+0000","last_clean":"2024-03-18T20:26:25.811148+0000","last_became_active":"2024-03-18T20:26:25.365694+0000","last_became_peered":"2024-03-18T20:26:25.365694+0000","last_unstale":"2024-03-18T20:26:25.811148+0000","last_undegraded":"2024-03-18T20:26:25.811148+0000","last_fullsized":"2024-03-18T20:26:25.811148+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:08.800093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.802792+0000","last_change":"2024-03-18T20:26:25.802194+0000","last_active":"2024-03-18T20:26:26.802792+0000","last_peered":"2024-03-18T20:26:26.802792+0000","last_clean":"2024-03-18T20:26:26.802792+0000","last_became_active":"2024-03-18T20:26:25.801915+0000","last_became_peered":"2024-03-18T20:26:25.801915+0000","last_unstale":"2024-03-18T20:26:26.802792+0000","last_undegraded":"2024-03-18T20:26:26.802792+0000","last_fullsized":"2024-03-18T20:26:26.802792+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:52.955865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"252'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.808116+0000","last_change":"2024-03-18T20:26:25.364855+0000","last_active":"2024-03-18T20:26:25.808116+0000","last_peered":"2024-03-18T20:26:25.808116+0000","last_clean":"2024-03-18T20:26:25.808116+0000","last_became_active":"2024-03-18T20:26:25.364563+0000","last_became_peered":"2024-03-18T20:26:25.364563+0000","last_unstale":"2024-03-18T20:26:25.808116+0000","last_undegraded":"2024-03-18T20:26:25.808116+0000","last_fullsized":"2024-03-18T20:26:25.808116+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:17.116325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.794972+0000","last_change":"2024-03-18T20:26:25.799795+0000","last_active":"2024-03-18T20:26:28.794972+0000","last_peered":"2024-03-18T20:26:28.794972+0000","last_clean":"2024-03-18T20:26:28.794972+0000","last_became_active":"2024-03-18T20:26:25.799539+0000","last_became_peered":"2024-03-18T20:26:25.799539+0000","last_unstale":"2024-03-18T20:26:28.794972+0000","last_undegraded":"2024-03-18T20:26:28.794972+0000","last_fullsized":"2024-03-18T20:26:28.794972+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:49.613713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"252'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.804832+0000","last_change":"2024-03-18T20:26:25.367422+0000","last_active":"2024-03-18T20:26:25.804832+0000","last_peered":"2024-03-18T20:26:25.804832+0000","last_clean":"2024-03-18T20:26:25.804832+0000","last_became_active":"2024-03-18T20:26:25.367153+0000","last_became_peered":"2024-03-18T20:26:25.367153+0000","last_unstale":"2024-03-18T20:26:25.804832+0000","last_undegraded":"2024-03-18T20:26:25.804832+0000","last_fullsized":"2024-03-18T20:26:25.804832+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:14:17.781705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811757+0000","last_change":"2024-03-18T20:26:25.812108+0000","last_active":"2024-03-18T20:26:27.811757+0000","last_peered":"2024-03-18T20:26:27.811757+0000","last_clean":"2024-03-18T20:26:27.811757+0000","last_became_active":"2024-03-18T20:26:25.811747+0000","last_became_peered":"2024-03-18T20:26:25.811747+0000","last_unstale":"2024-03-18T20:26:27.811757+0000","last_undegraded":"2024-03-18T20:26:27.811757+0000","last_fullsized":"2024-03-18T20:26:27.811757+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:40.456111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.790932+0000","last_change":"2024-03-18T20:26:25.360666+0000","last_active":"2024-03-18T20:26:25.790932+0000","last_peered":"2024-03-18T20:26:25.790932+0000","last_clean":"2024-03-18T20:26:25.790932+0000","last_became_active":"2024-03-18T20:26:25.360400+0000","last_became_peered":"2024-03-18T20:26:25.360400+0000","last_unstale":"2024-03-18T20:26:25.790932+0000","last_undegraded":"2024-03-18T20:26:25.790932+0000","last_fullsized":"2024-03-18T20:26:25.790932+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:18:38.754833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.802212+0000","last_change":"2024-03-18T20:26:25.810099+0000","last_active":"2024-03-18T20:26:26.802212+0000","last_peered":"2024-03-18T20:26:26.802212+0000","last_clean":"2024-03-18T20:26:26.802212+0000","last_became_active":"2024-03-18T20:26:25.806786+0000","last_became_peered":"2024-03-18T20:26:25.806786+0000","last_unstale":"2024-03-18T20:26:26.802212+0000","last_undegraded":"2024-03-18T20:26:26.802212+0000","last_fullsized":"2024-03-18T20:26:26.802212+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:35.919143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.804380+0000","last_change":"2024-03-18T20:26:25.365778+0000","last_active":"2024-03-18T20:26:25.804380+0000","last_peered":"2024-03-18T20:26:25.804380+0000","last_clean":"2024-03-18T20:26:25.804380+0000","last_became_active":"2024-03-18T20:26:25.365505+0000","last_became_peered":"2024-03-18T20:26:25.365505+0000","last_unstale":"2024-03-18T20:26:25.804380+0000","last_undegraded":"2024-03-18T20:26:25.804380+0000","last_fullsized":"2024-03-18T20:26:25.804380+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:46:21.036246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.794855+0000","last_change":"2024-03-18T20:26:25.798444+0000","last_active":"2024-03-18T20:26:28.794855+0000","last_peered":"2024-03-18T20:26:28.794855+0000","last_clean":"2024-03-18T20:26:28.794855+0000","last_became_active":"2024-03-18T20:26:25.798186+0000","last_became_peered":"2024-03-18T20:26:25.798186+0000","last_unstale":"2024-03-18T20:26:28.794855+0000","last_undegraded":"2024-03-18T20:26:28.794855+0000","last_fullsized":"2024-03-18T20:26:28.794855+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:06:22.720260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.794818+0000","last_change":"2024-03-18T20:26:25.370707+0000","last_active":"2024-03-18T20:26:25.794818+0000","last_peered":"2024-03-18T20:26:25.794818+0000","last_clean":"2024-03-18T20:26:25.794818+0000","last_became_active":"2024-03-18T20:26:25.370415+0000","last_became_peered":"2024-03-18T20:26:25.370415+0000","last_unstale":"2024-03-18T20:26:25.794818+0000","last_undegraded":"2024-03-18T20:26:25.794818+0000","last_fullsized":"2024-03-18T20:26:25.794818+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:05.351419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.294245+0000","last_change":"2024-03-18T20:26:25.816190+0000","last_active":"2024-03-18T20:26:28.294245+0000","last_peered":"2024-03-18T20:26:28.294245+0000","last_clean":"2024-03-18T20:26:28.294245+0000","last_became_active":"2024-03-18T20:26:25.815931+0000","last_became_peered":"2024-03-18T20:26:25.815931+0000","last_unstale":"2024-03-18T20:26:28.294245+0000","last_undegraded":"2024-03-18T20:26:28.294245+0000","last_fullsized":"2024-03-18T20:26:28.294245+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:49.613713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.791874+0000","last_change":"2024-03-18T20:26:25.363775+0000","last_active":"2024-03-18T20:26:25.791874+0000","last_peered":"2024-03-18T20:26:25.791874+0000","last_clean":"2024-03-18T20:26:25.791874+0000","last_became_active":"2024-03-18T20:26:25.363389+0000","last_became_peered":"2024-03-18T20:26:25.363389+0000","last_unstale":"2024-03-18T20:26:25.791874+0000","last_undegraded":"2024-03-18T20:26:25.791874+0000","last_fullsized":"2024-03-18T20:26:25.791874+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:42.397208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811342+0000","last_change":"2024-03-18T20:26:25.812196+0000","last_active":"2024-03-18T20:26:27.811342+0000","last_peered":"2024-03-18T20:26:27.811342+0000","last_clean":"2024-03-18T20:26:27.811342+0000","last_became_active":"2024-03-18T20:26:25.811934+0000","last_became_peered":"2024-03-18T20:26:25.811934+0000","last_unstale":"2024-03-18T20:26:27.811342+0000","last_undegraded":"2024-03-18T20:26:27.811342+0000","last_fullsized":"2024-03-18T20:26:27.811342+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:22.017892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"252'3","reported_seq":14,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.811258+0000","last_change":"2024-03-18T20:26:25.366952+0000","last_active":"2024-03-18T20:26:25.811258+0000","last_peered":"2024-03-18T20:26:25.811258+0000","last_clean":"2024-03-18T20:26:25.811258+0000","last_became_active":"2024-03-18T20:26:25.366671+0000","last_became_peered":"2024-03-18T20:26:25.366671+0000","last_unstale":"2024-03-18T20:26:25.811258+0000","last_undegraded":"2024-03-18T20:26:25.811258+0000","last_fullsized":"2024-03-18T20:26:25.811258+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:13:21.871477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.794932+0000","last_change":"2024-03-18T20:26:25.802694+0000","last_active":"2024-03-18T20:26:28.794932+0000","last_peered":"2024-03-18T20:26:28.794932+0000","last_clean":"2024-03-18T20:26:28.794932+0000","last_became_active":"2024-03-18T20:26:25.802443+0000","last_became_peered":"2024-03-18T20:26:25.802443+0000","last_unstale":"2024-03-18T20:26:28.794932+0000","last_undegraded":"2024-03-18T20:26:28.794932+0000","last_fullsized":"2024-03-18T20:26:28.794932+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:54:38.189655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"252'6","reported_seq":17,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.818245+0000","last_change":"2024-03-18T20:26:25.368198+0000","last_active":"2024-03-18T20:26:25.818245+0000","last_peered":"2024-03-18T20:26:25.818245+0000","last_clean":"2024-03-18T20:26:25.818245+0000","last_became_active":"2024-03-18T20:26:25.367762+0000","last_became_peered":"2024-03-18T20:26:25.367762+0000","last_unstale":"2024-03-18T20:26:25.818245+0000","last_undegraded":"2024-03-18T20:26:25.818245+0000","last_fullsized":"2024-03-18T20:26:25.818245+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:06.153535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.796159+0000","last_change":"2024-03-18T20:26:25.796336+0000","last_active":"2024-03-18T20:26:28.796159+0000","last_peered":"2024-03-18T20:26:28.796159+0000","last_clean":"2024-03-18T20:26:28.796159+0000","last_became_active":"2024-03-18T20:26:25.796002+0000","last_became_peered":"2024-03-18T20:26:25.796002+0000","last_unstale":"2024-03-18T20:26:28.796159+0000","last_undegraded":"2024-03-18T20:26:28.796159+0000","last_fullsized":"2024-03-18T20:26:28.796159+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:13:22.881517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"252'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.816996+0000","last_change":"2024-03-18T20:26:25.366019+0000","last_active":"2024-03-18T20:26:25.816996+0000","last_peered":"2024-03-18T20:26:25.816996+0000","last_clean":"2024-03-18T20:26:25.816996+0000","last_became_active":"2024-03-18T20:26:25.364848+0000","last_became_peered":"2024-03-18T20:26:25.364848+0000","last_unstale":"2024-03-18T20:26:25.816996+0000","last_undegraded":"2024-03-18T20:26:25.816996+0000","last_fullsized":"2024-03-18T20:26:25.816996+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:06:15.275895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.794903+0000","last_change":"2024-03-18T20:26:25.803741+0000","last_active":"2024-03-18T20:26:28.794903+0000","last_peered":"2024-03-18T20:26:28.794903+0000","last_clean":"2024-03-18T20:26:28.794903+0000","last_became_active":"2024-03-18T20:26:25.803470+0000","last_became_peered":"2024-03-18T20:26:25.803470+0000","last_unstale":"2024-03-18T20:26:28.794903+0000","last_undegraded":"2024-03-18T20:26:28.794903+0000","last_fullsized":"2024-03-18T20:26:28.794903+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:02:55.614091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.791265+0000","last_change":"2024-03-18T20:26:25.366963+0000","last_active":"2024-03-18T20:26:25.791265+0000","last_peered":"2024-03-18T20:26:25.791265+0000","last_clean":"2024-03-18T20:26:25.791265+0000","last_became_active":"2024-03-18T20:26:25.366416+0000","last_became_peered":"2024-03-18T20:26:25.366416+0000","last_unstale":"2024-03-18T20:26:25.791265+0000","last_undegraded":"2024-03-18T20:26:25.791265+0000","last_fullsized":"2024-03-18T20:26:25.791265+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:57.921302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795529+0000","last_change":"2024-03-18T20:26:25.808144+0000","last_active":"2024-03-18T20:26:28.795529+0000","last_peered":"2024-03-18T20:26:28.795529+0000","last_clean":"2024-03-18T20:26:28.795529+0000","last_became_active":"2024-03-18T20:26:25.807838+0000","last_became_peered":"2024-03-18T20:26:25.807838+0000","last_unstale":"2024-03-18T20:26:28.795529+0000","last_undegraded":"2024-03-18T20:26:28.795529+0000","last_fullsized":"2024-03-18T20:26:28.795529+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:37:41.132114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.791803+0000","last_change":"2024-03-18T20:26:25.366119+0000","last_active":"2024-03-18T20:26:25.791803+0000","last_peered":"2024-03-18T20:26:25.791803+0000","last_clean":"2024-03-18T20:26:25.791803+0000","last_became_active":"2024-03-18T20:26:25.365869+0000","last_became_peered":"2024-03-18T20:26:25.365869+0000","last_unstale":"2024-03-18T20:26:25.791803+0000","last_undegraded":"2024-03-18T20:26:25.791803+0000","last_fullsized":"2024-03-18T20:26:25.791803+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:01.890743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.802155+0000","last_change":"2024-03-18T20:26:25.802507+0000","last_active":"2024-03-18T20:26:26.802155+0000","last_peered":"2024-03-18T20:26:26.802155+0000","last_clean":"2024-03-18T20:26:26.802155+0000","last_became_active":"2024-03-18T20:26:25.801913+0000","last_became_peered":"2024-03-18T20:26:25.801913+0000","last_unstale":"2024-03-18T20:26:26.802155+0000","last_undegraded":"2024-03-18T20:26:26.802155+0000","last_fullsized":"2024-03-18T20:26:26.802155+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:11.977302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"252'8","reported_seq":19,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.817056+0000","last_change":"2024-03-18T20:26:25.365245+0000","last_active":"2024-03-18T20:26:25.817056+0000","last_peered":"2024-03-18T20:26:25.817056+0000","last_clean":"2024-03-18T20:26:25.817056+0000","last_became_active":"2024-03-18T20:26:25.364945+0000","last_became_peered":"2024-03-18T20:26:25.364945+0000","last_unstale":"2024-03-18T20:26:25.817056+0000","last_undegraded":"2024-03-18T20:26:25.817056+0000","last_fullsized":"2024-03-18T20:26:25.817056+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:19.739927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.801434+0000","last_change":"2024-03-18T20:26:25.802286+0000","last_active":"2024-03-18T20:26:26.801434+0000","last_peered":"2024-03-18T20:26:26.801434+0000","last_clean":"2024-03-18T20:26:26.801434+0000","last_became_active":"2024-03-18T20:26:25.802005+0000","last_became_peered":"2024-03-18T20:26:25.802005+0000","last_unstale":"2024-03-18T20:26:26.801434+0000","last_undegraded":"2024-03-18T20:26:26.801434+0000","last_fullsized":"2024-03-18T20:26:26.801434+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:10:16.362251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"252'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.815493+0000","last_change":"2024-03-18T20:26:25.368591+0000","last_active":"2024-03-18T20:26:25.815493+0000","last_peered":"2024-03-18T20:26:25.815493+0000","last_clean":"2024-03-18T20:26:25.815493+0000","last_became_active":"2024-03-18T20:26:25.367843+0000","last_became_peered":"2024-03-18T20:26:25.367843+0000","last_unstale":"2024-03-18T20:26:25.815493+0000","last_undegraded":"2024-03-18T20:26:25.815493+0000","last_fullsized":"2024-03-18T20:26:25.815493+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:59:40.867803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.801360+0000","last_change":"2024-03-18T20:26:25.810340+0000","last_active":"2024-03-18T20:26:26.801360+0000","last_peered":"2024-03-18T20:26:26.801360+0000","last_clean":"2024-03-18T20:26:26.801360+0000","last_became_active":"2024-03-18T20:26:25.809980+0000","last_became_peered":"2024-03-18T20:26:25.809980+0000","last_unstale":"2024-03-18T20:26:26.801360+0000","last_undegraded":"2024-03-18T20:26:26.801360+0000","last_fullsized":"2024-03-18T20:26:26.801360+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:02:55.614091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"252'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.809630+0000","last_change":"2024-03-18T20:26:25.366492+0000","last_active":"2024-03-18T20:26:25.809630+0000","last_peered":"2024-03-18T20:26:25.809630+0000","last_clean":"2024-03-18T20:26:25.809630+0000","last_became_active":"2024-03-18T20:26:25.366098+0000","last_became_peered":"2024-03-18T20:26:25.366098+0000","last_unstale":"2024-03-18T20:26:25.809630+0000","last_undegraded":"2024-03-18T20:26:25.809630+0000","last_fullsized":"2024-03-18T20:26:25.809630+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:53:26.604334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.796134+0000","last_change":"2024-03-18T20:26:25.807945+0000","last_active":"2024-03-18T20:26:28.796134+0000","last_peered":"2024-03-18T20:26:28.796134+0000","last_clean":"2024-03-18T20:26:28.796134+0000","last_became_active":"2024-03-18T20:26:25.807678+0000","last_became_peered":"2024-03-18T20:26:25.807678+0000","last_unstale":"2024-03-18T20:26:28.796134+0000","last_undegraded":"2024-03-18T20:26:28.796134+0000","last_fullsized":"2024-03-18T20:26:28.796134+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:06.759034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"252'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.809803+0000","last_change":"2024-03-18T20:26:25.371835+0000","last_active":"2024-03-18T20:26:25.809803+0000","last_peered":"2024-03-18T20:26:25.809803+0000","last_clean":"2024-03-18T20:26:25.809803+0000","last_became_active":"2024-03-18T20:26:25.371634+0000","last_became_peered":"2024-03-18T20:26:25.371634+0000","last_unstale":"2024-03-18T20:26:25.809803+0000","last_undegraded":"2024-03-18T20:26:25.809803+0000","last_fullsized":"2024-03-18T20:26:25.809803+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:43.471325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811447+0000","last_change":"2024-03-18T20:26:25.812207+0000","last_active":"2024-03-18T20:26:27.811447+0000","last_peered":"2024-03-18T20:26:27.811447+0000","last_clean":"2024-03-18T20:26:27.811447+0000","last_became_active":"2024-03-18T20:26:25.811924+0000","last_became_peered":"2024-03-18T20:26:25.811924+0000","last_unstale":"2024-03-18T20:26:27.811447+0000","last_undegraded":"2024-03-18T20:26:27.811447+0000","last_fullsized":"2024-03-18T20:26:27.811447+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:01:09.966146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.793608+0000","last_change":"2024-03-18T20:26:25.364241+0000","last_active":"2024-03-18T20:26:25.793608+0000","last_peered":"2024-03-18T20:26:25.793608+0000","last_clean":"2024-03-18T20:26:25.793608+0000","last_became_active":"2024-03-18T20:26:25.363945+0000","last_became_peered":"2024-03-18T20:26:25.363945+0000","last_unstale":"2024-03-18T20:26:25.793608+0000","last_undegraded":"2024-03-18T20:26:25.793608+0000","last_fullsized":"2024-03-18T20:26:25.793608+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:34.909104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811477+0000","last_change":"2024-03-18T20:26:25.799178+0000","last_active":"2024-03-18T20:26:27.811477+0000","last_peered":"2024-03-18T20:26:27.811477+0000","last_clean":"2024-03-18T20:26:27.811477+0000","last_became_active":"2024-03-18T20:26:25.798768+0000","last_became_peered":"2024-03-18T20:26:25.798768+0000","last_unstale":"2024-03-18T20:26:27.811477+0000","last_undegraded":"2024-03-18T20:26:27.811477+0000","last_fullsized":"2024-03-18T20:26:27.811477+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:22:12.729546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"252'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.814304+0000","last_change":"2024-03-18T20:26:25.370148+0000","last_active":"2024-03-18T20:26:25.814304+0000","last_peered":"2024-03-18T20:26:25.814304+0000","last_clean":"2024-03-18T20:26:25.814304+0000","last_became_active":"2024-03-18T20:26:25.369876+0000","last_became_peered":"2024-03-18T20:26:25.369876+0000","last_unstale":"2024-03-18T20:26:25.814304+0000","last_undegraded":"2024-03-18T20:26:25.814304+0000","last_fullsized":"2024-03-18T20:26:25.814304+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:57:48.151947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795608+0000","last_change":"2024-03-18T20:26:25.806909+0000","last_active":"2024-03-18T20:26:28.795608+0000","last_peered":"2024-03-18T20:26:28.795608+0000","last_clean":"2024-03-18T20:26:28.795608+0000","last_became_active":"2024-03-18T20:26:25.806643+0000","last_became_peered":"2024-03-18T20:26:25.806643+0000","last_unstale":"2024-03-18T20:26:28.795608+0000","last_undegraded":"2024-03-18T20:26:28.795608+0000","last_fullsized":"2024-03-18T20:26:28.795608+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:30:14.320846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.805502+0000","last_change":"2024-03-18T20:26:25.370582+0000","last_active":"2024-03-18T20:26:25.805502+0000","last_peered":"2024-03-18T20:26:25.805502+0000","last_clean":"2024-03-18T20:26:25.805502+0000","last_became_active":"2024-03-18T20:26:25.370336+0000","last_became_peered":"2024-03-18T20:26:25.370336+0000","last_unstale":"2024-03-18T20:26:25.805502+0000","last_undegraded":"2024-03-18T20:26:25.805502+0000","last_fullsized":"2024-03-18T20:26:25.805502+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:23:03.661960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.294584+0000","last_change":"2024-03-18T20:26:25.814360+0000","last_active":"2024-03-18T20:26:28.294584+0000","last_peered":"2024-03-18T20:26:28.294584+0000","last_clean":"2024-03-18T20:26:28.294584+0000","last_became_active":"2024-03-18T20:26:25.812682+0000","last_became_peered":"2024-03-18T20:26:25.812682+0000","last_unstale":"2024-03-18T20:26:28.294584+0000","last_undegraded":"2024-03-18T20:26:28.294584+0000","last_fullsized":"2024-03-18T20:26:28.294584+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:22.598187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.799106+0000","last_change":"2024-03-18T20:26:25.369740+0000","last_active":"2024-03-18T20:26:25.799106+0000","last_peered":"2024-03-18T20:26:25.799106+0000","last_clean":"2024-03-18T20:26:25.799106+0000","last_became_active":"2024-03-18T20:26:25.369448+0000","last_became_peered":"2024-03-18T20:26:25.369448+0000","last_unstale":"2024-03-18T20:26:25.799106+0000","last_undegraded":"2024-03-18T20:26:25.799106+0000","last_fullsized":"2024-03-18T20:26:25.799106+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:39:25.003994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.294662+0000","last_change":"2024-03-18T20:26:25.807777+0000","last_active":"2024-03-18T20:26:28.294662+0000","last_peered":"2024-03-18T20:26:28.294662+0000","last_clean":"2024-03-18T20:26:28.294662+0000","last_became_active":"2024-03-18T20:26:25.807447+0000","last_became_peered":"2024-03-18T20:26:25.807447+0000","last_unstale":"2024-03-18T20:26:28.294662+0000","last_undegraded":"2024-03-18T20:26:28.294662+0000","last_fullsized":"2024-03-18T20:26:28.294662+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:22:55.933766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.794516+0000","last_change":"2024-03-18T20:26:25.366647+0000","last_active":"2024-03-18T20:26:25.794516+0000","last_peered":"2024-03-18T20:26:25.794516+0000","last_clean":"2024-03-18T20:26:25.794516+0000","last_became_active":"2024-03-18T20:26:25.366363+0000","last_became_peered":"2024-03-18T20:26:25.366363+0000","last_unstale":"2024-03-18T20:26:25.794516+0000","last_undegraded":"2024-03-18T20:26:25.794516+0000","last_fullsized":"2024-03-18T20:26:25.794516+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:22:29.645161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795310+0000","last_change":"2024-03-18T20:26:25.807101+0000","last_active":"2024-03-18T20:26:28.795310+0000","last_peered":"2024-03-18T20:26:28.795310+0000","last_clean":"2024-03-18T20:26:28.795310+0000","last_became_active":"2024-03-18T20:26:25.806372+0000","last_became_peered":"2024-03-18T20:26:25.806372+0000","last_unstale":"2024-03-18T20:26:28.795310+0000","last_undegraded":"2024-03-18T20:26:28.795310+0000","last_fullsized":"2024-03-18T20:26:28.795310+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:18.126364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"252'2","reported_seq":13,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.814500+0000","last_change":"2024-03-18T20:26:25.365000+0000","last_active":"2024-03-18T20:26:25.814500+0000","last_peered":"2024-03-18T20:26:25.814500+0000","last_clean":"2024-03-18T20:26:25.814500+0000","last_became_active":"2024-03-18T20:26:25.364727+0000","last_became_peered":"2024-03-18T20:26:25.364727+0000","last_unstale":"2024-03-18T20:26:25.814500+0000","last_undegraded":"2024-03-18T20:26:25.814500+0000","last_fullsized":"2024-03-18T20:26:25.814500+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:18.741871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795257+0000","last_change":"2024-03-18T20:26:25.803062+0000","last_active":"2024-03-18T20:26:28.795257+0000","last_peered":"2024-03-18T20:26:28.795257+0000","last_clean":"2024-03-18T20:26:28.795257+0000","last_became_active":"2024-03-18T20:26:25.802825+0000","last_became_peered":"2024-03-18T20:26:25.802825+0000","last_unstale":"2024-03-18T20:26:28.795257+0000","last_undegraded":"2024-03-18T20:26:28.795257+0000","last_fullsized":"2024-03-18T20:26:28.795257+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:31:54.405889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.790280+0000","last_change":"2024-03-18T20:26:25.367649+0000","last_active":"2024-03-18T20:26:25.790280+0000","last_peered":"2024-03-18T20:26:25.790280+0000","last_clean":"2024-03-18T20:26:25.790280+0000","last_became_active":"2024-03-18T20:26:25.367351+0000","last_became_peered":"2024-03-18T20:26:25.367351+0000","last_unstale":"2024-03-18T20:26:25.790280+0000","last_undegraded":"2024-03-18T20:26:25.790280+0000","last_fullsized":"2024-03-18T20:26:25.790280+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:19:05.047442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:27.811323+0000","last_change":"2024-03-18T20:26:25.812701+0000","last_active":"2024-03-18T20:26:27.811323+0000","last_peered":"2024-03-18T20:26:27.811323+0000","last_clean":"2024-03-18T20:26:27.811323+0000","last_became_active":"2024-03-18T20:26:25.812469+0000","last_became_peered":"2024-03-18T20:26:25.812469+0000","last_unstale":"2024-03-18T20:26:27.811323+0000","last_undegraded":"2024-03-18T20:26:27.811323+0000","last_fullsized":"2024-03-18T20:26:27.811323+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:59.140240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.794829+0000","last_change":"2024-03-18T20:26:25.370628+0000","last_active":"2024-03-18T20:26:25.794829+0000","last_peered":"2024-03-18T20:26:25.794829+0000","last_clean":"2024-03-18T20:26:25.794829+0000","last_became_active":"2024-03-18T20:26:25.370332+0000","last_became_peered":"2024-03-18T20:26:25.370332+0000","last_unstale":"2024-03-18T20:26:25.794829+0000","last_undegraded":"2024-03-18T20:26:25.794829+0000","last_fullsized":"2024-03-18T20:26:25.794829+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:43.011701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.801714+0000","last_change":"2024-03-18T20:26:25.802895+0000","last_active":"2024-03-18T20:26:26.801714+0000","last_peered":"2024-03-18T20:26:26.801714+0000","last_clean":"2024-03-18T20:26:26.801714+0000","last_became_active":"2024-03-18T20:26:25.802655+0000","last_became_peered":"2024-03-18T20:26:25.802655+0000","last_unstale":"2024-03-18T20:26:26.801714+0000","last_undegraded":"2024-03-18T20:26:26.801714+0000","last_fullsized":"2024-03-18T20:26:26.801714+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:18.126364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.794933+0000","last_change":"2024-03-18T20:26:25.368715+0000","last_active":"2024-03-18T20:26:25.794933+0000","last_peered":"2024-03-18T20:26:25.794933+0000","last_clean":"2024-03-18T20:26:25.794933+0000","last_became_active":"2024-03-18T20:26:25.368383+0000","last_became_peered":"2024-03-18T20:26:25.368383+0000","last_unstale":"2024-03-18T20:26:25.794933+0000","last_undegraded":"2024-03-18T20:26:25.794933+0000","last_fullsized":"2024-03-18T20:26:25.794933+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:07.975001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:26.802845+0000","last_change":"2024-03-18T20:26:25.805198+0000","last_active":"2024-03-18T20:26:26.802845+0000","last_peered":"2024-03-18T20:26:26.802845+0000","last_clean":"2024-03-18T20:26:26.802845+0000","last_became_active":"2024-03-18T20:26:25.804942+0000","last_became_peered":"2024-03-18T20:26:25.804942+0000","last_unstale":"2024-03-18T20:26:26.802845+0000","last_undegraded":"2024-03-18T20:26:26.802845+0000","last_fullsized":"2024-03-18T20:26:26.802845+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:22:12.729546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.808096+0000","last_change":"2024-03-18T20:26:25.370584+0000","last_active":"2024-03-18T20:26:25.808096+0000","last_peered":"2024-03-18T20:26:25.808096+0000","last_clean":"2024-03-18T20:26:25.808096+0000","last_became_active":"2024-03-18T20:26:25.370353+0000","last_became_peered":"2024-03-18T20:26:25.370353+0000","last_unstale":"2024-03-18T20:26:25.808096+0000","last_undegraded":"2024-03-18T20:26:25.808096+0000","last_fullsized":"2024-03-18T20:26:25.808096+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:33:14.925680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795655+0000","last_change":"2024-03-18T20:26:25.801336+0000","last_active":"2024-03-18T20:26:28.795655+0000","last_peered":"2024-03-18T20:26:28.795655+0000","last_clean":"2024-03-18T20:26:28.795655+0000","last_became_active":"2024-03-18T20:26:25.801053+0000","last_became_peered":"2024-03-18T20:26:25.801053+0000","last_unstale":"2024-03-18T20:26:28.795655+0000","last_undegraded":"2024-03-18T20:26:28.795655+0000","last_fullsized":"2024-03-18T20:26:28.795655+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:23.742824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.791211+0000","last_change":"2024-03-18T20:26:25.367183+0000","last_active":"2024-03-18T20:26:25.791211+0000","last_peered":"2024-03-18T20:26:25.791211+0000","last_clean":"2024-03-18T20:26:25.791211+0000","last_became_active":"2024-03-18T20:26:25.366701+0000","last_became_peered":"2024-03-18T20:26:25.366701+0000","last_unstale":"2024-03-18T20:26:25.791211+0000","last_undegraded":"2024-03-18T20:26:25.791211+0000","last_fullsized":"2024-03-18T20:26:25.791211+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:05.748994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795971+0000","last_change":"2024-03-18T20:26:25.806432+0000","last_active":"2024-03-18T20:26:28.795971+0000","last_peered":"2024-03-18T20:26:28.795971+0000","last_clean":"2024-03-18T20:26:28.795971+0000","last_became_active":"2024-03-18T20:26:25.806050+0000","last_became_peered":"2024-03-18T20:26:25.806050+0000","last_unstale":"2024-03-18T20:26:28.795971+0000","last_undegraded":"2024-03-18T20:26:28.795971+0000","last_fullsized":"2024-03-18T20:26:28.795971+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:32.033370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.790834+0000","last_change":"2024-03-18T20:26:25.364069+0000","last_active":"2024-03-18T20:26:25.790834+0000","last_peered":"2024-03-18T20:26:25.790834+0000","last_clean":"2024-03-18T20:26:25.790834+0000","last_became_active":"2024-03-18T20:26:25.363764+0000","last_became_peered":"2024-03-18T20:26:25.363764+0000","last_unstale":"2024-03-18T20:26:25.790834+0000","last_undegraded":"2024-03-18T20:26:25.790834+0000","last_fullsized":"2024-03-18T20:26:25.790834+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:01.862380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.294665+0000","last_change":"2024-03-18T20:26:25.817787+0000","last_active":"2024-03-18T20:26:28.294665+0000","last_peered":"2024-03-18T20:26:28.294665+0000","last_clean":"2024-03-18T20:26:28.294665+0000","last_became_active":"2024-03-18T20:26:25.817533+0000","last_became_peered":"2024-03-18T20:26:25.817533+0000","last_unstale":"2024-03-18T20:26:28.294665+0000","last_undegraded":"2024-03-18T20:26:28.294665+0000","last_fullsized":"2024-03-18T20:26:28.294665+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:22:04.370714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":11,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.799745+0000","last_change":"2024-03-18T20:26:25.368090+0000","last_active":"2024-03-18T20:26:25.799745+0000","last_peered":"2024-03-18T20:26:25.799745+0000","last_clean":"2024-03-18T20:26:25.799745+0000","last_became_active":"2024-03-18T20:26:25.367837+0000","last_became_peered":"2024-03-18T20:26:25.367837+0000","last_unstale":"2024-03-18T20:26:25.799745+0000","last_undegraded":"2024-03-18T20:26:25.799745+0000","last_fullsized":"2024-03-18T20:26:25.799745+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:19.739927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":10,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-03-18T20:26:28.795099+0000","last_change":"2024-03-18T20:26:25.811045+0000","last_active":"2024-03-18T20:26:28.795099+0000","last_peered":"2024-03-18T20:26:28.795099+0000","last_clean":"2024-03-18T20:26:28.795099+0000","last_became_active":"2024-03-18T20:26:25.810754+0000","last_became_peered":"2024-03-18T20:26:25.810754+0000","last_unstale":"2024-03-18T20:26:28.795099+0000","last_undegraded":"2024-03-18T20:26:28.795099+0000","last_fullsized":"2024-03-18T20:26:28.795099+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:44.931187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"252'1","reported_seq":12,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.806470+0000","last_change":"2024-03-18T20:26:25.367689+0000","last_active":"2024-03-18T20:26:25.806470+0000","last_peered":"2024-03-18T20:26:25.806470+0000","last_clean":"2024-03-18T20:26:25.806470+0000","last_became_active":"2024-03-18T20:26:25.367431+0000","last_became_peered":"2024-03-18T20:26:25.367431+0000","last_unstale":"2024-03-18T20:26:25.806470+0000","last_undegraded":"2024-03-18T20:26:25.806470+0000","last_fullsized":"2024-03-18T20:26:25.806470+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:01.890743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"238'370","reported_seq":1297,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-03-18T20:26:25.792914+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:26:25.792914+0000","last_peered":"2024-03-18T20:26:25.792914+0000","last_clean":"2024-03-18T20:26:25.792914+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:26:25.792914+0000","last_undegraded":"2024-03-18T20:26:25.792914+0000","last_fullsized":"2024-03-18T20:26:25.792914+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575992,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52864,"kb_used_data":3700,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318976,"statfs":{"total":96636764160,"available":96582631424,"internally_reserved":0,"allocated":3788800,"data_stored":2233777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.57299999999999995,"15min":0.59499999999999997},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":1.1339999999999999},"last":0.627},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.58299999999999996,"15min":0.60899999999999999},"min":{"1min":0.35299999999999998,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.76800000000000002,"5min":0.84499999999999997,"15min":1.093},"last":0.52100000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.58199999999999996,"15min":0.60799999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72199999999999998,"5min":0.83099999999999996,"15min":1.4299999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.58499999999999996,"15min":0.61399999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":1.2629999999999999},"last":0.59899999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.58799999999999997,"15min":0.60899999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":1.216},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.58999999999999997,"15min":0.621},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.2490000000000001},"last":0.55400000000000005}]},{"osd":3,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.57199999999999995,"15min":0.60299999999999998},"min":{"1min":0.374,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83899999999999997,"5min":0.84699999999999998,"15min":1.4450000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.623,"15min":0.64600000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.38600000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":1.5029999999999999},"last":0.66700000000000004}]},{"osd":4,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46600000000000003,"15min":0.47699999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.76500000000000001,"5min":0.79100000000000004,"15min":0.86099999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.502,"5min":0.48899999999999999,"15min":0.5},"min":{"1min":0.34300000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.57599999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.502,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52500000000000002,"15min":0.54600000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.877,"5min":0.877,"15min":1.042},"last":0.73099999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55300000000000005,"15min":0.57699999999999996},"min":{"1min":0.39500000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":1.129},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54800000000000004,"15min":0.56799999999999995},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":1.0609999999999999},"last":0.71399999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575992,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42436,"kb_used_data":3120,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94329404,"statfs":{"total":96636764160,"available":96593309696,"internally_reserved":0,"allocated":3194880,"data_stored":1643397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55400000000000005,"15min":0.59399999999999997},"min":{"1min":0.32500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.623,"5min":0.80900000000000005,"15min":1.1519999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57399999999999995,"15min":0.60399999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.53400000000000003}]},{"osd":1,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56499999999999995,"15min":0.59699999999999998},"min":{"1min":0.40400000000000003,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69899999999999995,"5min":0.79800000000000004,"15min":1.244},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57999999999999996,"15min":0.60299999999999998},"min":{"1min":0.42499999999999999,"5min":0.35399999999999998,"15min":0.33300000000000002},"max":{"1min":0.83699999999999997,"5min":0.89200000000000002,"15min":1.286},"last":0.34799999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.58699999999999997,"15min":0.623},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.69199999999999995,"5min":0.92000000000000004,"15min":1.276},"last":0.438},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57999999999999996,"15min":0.60899999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.378},"max":{"1min":1.089,"5min":1.089,"15min":1.2669999999999999},"last":0.46999999999999997}]},{"osd":3,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58099999999999996,"15min":0.621},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.59199999999999997,"15min":0.61599999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70599999999999996,"5min":1.0049999999999999,"15min":1.337},"last":0.51500000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.48899999999999999,"15min":0.48799999999999999},"min":{"1min":0.35799999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.75600000000000001,"5min":0.85799999999999998,"15min":0.88300000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.50800000000000001,"15min":0.51700000000000002},"min":{"1min":0.35899999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80300000000000005,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.41799999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.51100000000000001,"15min":0.51000000000000001},"min":{"1min":0.38300000000000001,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39600000000000002,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.874,"5min":0.92600000000000005,"15min":0.94199999999999995},"last":0.58499999999999996}]},{"osd":7,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.42399999999999999,"5min":0.38,"15min":0.34499999999999997},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.57799999999999996,"15min":0.57499999999999996},"min":{"1min":0.41399999999999998,"5min":0.36599999999999999,"15min":0.34599999999999997},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.59399999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575992,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52276,"kb_used_data":3112,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319564,"statfs":{"total":96636764160,"available":96583233536,"internally_reserved":0,"allocated":3186688,"data_stored":1643285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.56299999999999994,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.38,"5min":0.41199999999999998,"15min":0.42199999999999999},"min":{"1min":0.27200000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56499999999999995,"5min":0.67500000000000004,"15min":0.69699999999999995},"last":0.61299999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.43099999999999999,"15min":0.432},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.51500000000000001,"5min":0.64300000000000002,"15min":0.68100000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.34499999999999997,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.55500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.54800000000000004}]},{"osd":3,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.33300000000000002,"5min":0.33000000000000002,"15min":0.32300000000000001},"max":{"1min":0.58699999999999997,"5min":0.75900000000000001,"15min":0.76100000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.44,"15min":0.436},"min":{"1min":0.29199999999999998,"5min":0.28899999999999998,"15min":0.27000000000000002},"max":{"1min":0.55400000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.46400000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.66500000000000004,"15min":0.66800000000000004},"min":{"1min":0.42999999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82199999999999995,"5min":1.0680000000000001,"15min":1.2909999999999999},"last":0.746},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66900000000000004,"15min":0.67500000000000004},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.746},"last":0.73199999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.46300000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.88600000000000001,"5min":1.0389999999999999,"15min":1.6910000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.70199999999999996,"15min":0.70799999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.91100000000000003,"5min":1.109,"15min":1.2549999999999999},"last":0.626}]},{"osd":6,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.68200000000000005,"15min":0.68100000000000005},"min":{"1min":0.46800000000000003,"5min":0.45900000000000002,"15min":0.437},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.637},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.70299999999999996,"15min":0.70499999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92100000000000004,"5min":1.014,"15min":1.722},"last":0.69499999999999995}]},{"osd":7,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.68999999999999995,"15min":0.70399999999999996},"min":{"1min":0.51400000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.90300000000000002,"5min":1.02,"15min":1.2729999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.56599999999999995,"5min":0.495,"15min":0.42799999999999999},"max":{"1min":0.84199999999999997,"5min":1.0329999999999999,"15min":1.3380000000000001},"last":0.71299999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575993,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48780,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323060,"statfs":{"total":96636764160,"available":96586813440,"internally_reserved":0,"allocated":3792896,"data_stored":2235217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38300000000000001,"15min":0.39000000000000001},"min":{"1min":0.29299999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56399999999999995,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.41699999999999998,"15min":0.42599999999999999},"min":{"1min":0.28199999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.60099999999999998,"5min":0.64000000000000001,"15min":1.1850000000000001},"last":0.58799999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42899999999999999,"15min":0.437},"min":{"1min":0.28399999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67900000000000005,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.432,"15min":0.45000000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.70999999999999996},"last":0.441}]},{"osd":3,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.42099999999999999,"15min":0.42899999999999999},"min":{"1min":0.29499999999999998,"5min":0.27500000000000002,"15min":0.26100000000000001},"max":{"1min":0.58499999999999996,"5min":0.60099999999999998,"15min":0.628},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.48099999999999998,"15min":0.49299999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.32500000000000001},"max":{"1min":0.73399999999999999,"5min":0.73999999999999999,"15min":0.91500000000000004},"last":0.49099999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63200000000000001,"15min":0.65200000000000002},"min":{"1min":0.40000000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90200000000000002,"5min":0.91500000000000004,"15min":1.0029999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.58099999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.66700000000000004,"15min":0.67000000000000004},"min":{"1min":0.435,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.88500000000000001,"5min":0.97099999999999997,"15min":1.0129999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.626,"5min":0.67400000000000004,"15min":0.68000000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":1.518},"last":0.73199999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.66100000000000003,"15min":0.66300000000000003},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.1279999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.71799999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.69199999999999995,"15min":0.70499999999999996},"min":{"1min":0.47999999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.464},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69099999999999995,"15min":0.70299999999999996},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.47599999999999998},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.66100000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575993,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52284,"kb_used_data":3120,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319556,"statfs":{"total":96636764160,"available":96583225344,"internally_reserved":0,"allocated":3194880,"data_stored":1643937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36599999999999999,"15min":0.378},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.496,"5min":0.54800000000000004,"15min":0.59399999999999997},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.47399999999999998,"15min":0.48699999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.63600000000000001,"5min":0.71499999999999997,"15min":0.753},"last":0.379}]},{"osd":1,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46899999999999997,"15min":0.48199999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64300000000000002,"5min":0.72599999999999998,"15min":0.77100000000000002},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40899999999999997,"15min":0.41599999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.53800000000000003,"5min":0.59299999999999997,"15min":0.71099999999999997},"last":0.35399999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.433,"15min":0.436},"min":{"1min":0.28899999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.497,"15min":0.51700000000000002},"min":{"1min":0.33800000000000002,"5min":0.32200000000000001,"15min":0.313},"max":{"1min":0.68100000000000005,"5min":0.752,"15min":0.752},"last":0.435}]},{"osd":4,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.66300000000000003,"15min":0.65800000000000003},"min":{"1min":0.503,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":0.82499999999999996,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67500000000000004,"15min":0.66800000000000004},"min":{"1min":0.497,"5min":0.443,"15min":0.443},"max":{"1min":0.91600000000000004,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.81100000000000005}]},{"osd":5,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.67900000000000005,"15min":0.68500000000000005},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.93999999999999995,"5min":1.028,"15min":1.028},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67300000000000004,"15min":0.68000000000000005},"min":{"1min":0.47299999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.88300000000000001,"5min":0.95599999999999996,"15min":1.0629999999999999},"last":0.72199999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68000000000000005,"15min":0.69099999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86099999999999999,"5min":1.014,"15min":1.2050000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.72499999999999998,"15min":0.73199999999999998},"min":{"1min":0.53800000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.97199999999999998,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.82999999999999996}]},{"osd":7,"last update":"Mon Mar 18 20:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68899999999999995,"15min":0.70199999999999996},"min":{"1min":0.51500000000000001,"5min":0.442,"15min":0.42199999999999999},"max":{"1min":0.92900000000000005,"5min":0.99399999999999999,"15min":1.0469999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69899999999999995,"15min":0.70699999999999996},"min":{"1min":0.45500000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.93999999999999995,"5min":1.371,"15min":1.371},"last":0.84099999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575993,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52300,"kb_used_data":3136,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319540,"statfs":{"total":96636764160,"available":96583208960,"internally_reserved":0,"allocated":3211264,"data_stored":1645487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.38,"15min":0.39500000000000002},"min":{"1min":0.23100000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.48899999999999999,"5min":0.66500000000000004,"15min":0.98099999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.39100000000000001,"15min":0.40999999999999998},"min":{"1min":0.26100000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.42699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.35199999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.374,"15min":0.40300000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.47699999999999998,"5min":0.63200000000000001,"15min":0.64000000000000001},"last":0.308},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.44400000000000001,"15min":0.46200000000000002},"min":{"1min":0.315,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.50900000000000001,"5min":0.65500000000000003,"15min":0.73099999999999998},"last":0.40100000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.436,"15min":0.46000000000000002},"min":{"1min":0.29999999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53800000000000003,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.46000000000000002,"15min":0.48799999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.51700000000000002,"5min":0.66100000000000003,"15min":0.73099999999999998},"last":0.42799999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.61199999999999999,"15min":0.64500000000000002},"min":{"1min":0.41599999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.63300000000000001,"15min":0.66900000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.51800000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.67000000000000004,"15min":0.69299999999999995},"min":{"1min":0.42999999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.70299999999999996,"15min":0.72299999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.1120000000000001,"5min":1.171,"15min":1.171},"last":0.59799999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67000000000000004,"15min":0.70299999999999996},"min":{"1min":0.45400000000000001,"5min":0.436,"15min":0.42499999999999999},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.69999999999999996,"15min":0.72599999999999998},"min":{"1min":0.47399999999999998,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":1.22,"5min":1.22,"15min":1.2410000000000001},"last":0.63400000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.68100000000000005,"15min":0.71299999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.624,"5min":0.66000000000000003,"15min":0.69399999999999995},"min":{"1min":0.41099999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.64700000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575993,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48184,"kb_used_data":3116,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323656,"statfs":{"total":96636764160,"available":96587423744,"internally_reserved":0,"allocated":3190784,"data_stored":1643355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.57899999999999996,"15min":0.59299999999999997},"min":{"1min":0.34300000000000003,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65300000000000002,"5min":0.77500000000000002,"15min":0.81799999999999995},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57799999999999996,"15min":0.59799999999999998},"min":{"1min":0.40300000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69999999999999996,"5min":0.79600000000000004,"15min":0.84199999999999997},"last":0.44900000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59999999999999998,"15min":0.61399999999999999},"min":{"1min":0.438,"5min":0.318,"15min":0.318},"max":{"1min":0.73199999999999998,"5min":0.81899999999999995,"15min":0.88300000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56999999999999995,"15min":0.58499999999999996},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.71199999999999997,"5min":0.79200000000000004,"15min":0.96499999999999997},"last":0.48399999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58299999999999996,"15min":0.59899999999999998},"min":{"1min":0.439,"5min":0.39500000000000002,"15min":0.39200000000000002},"max":{"1min":0.68600000000000005,"5min":0.80500000000000005,"15min":0.82599999999999996},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61099999999999999,"15min":0.61799999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.38500000000000001},"max":{"1min":0.81399999999999995,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.63700000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.40799999999999997,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.623},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.86199999999999999},"last":0.59799999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.318},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.498,"15min":0.504},"min":{"1min":0.35199999999999998,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.85499999999999998,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.41099999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.499,"15min":0.505},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.53500000000000003}]},{"osd":7,"last update":"Mon Mar 18 20:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54100000000000004,"15min":0.54800000000000004},"min":{"1min":0.435,"5min":0.38,"15min":0.35699999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.57199999999999995,"15min":0.57999999999999996},"min":{"1min":0.41899999999999998,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.5}]}]},{"osd":5,"up_from":13,"seq":55834575992,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48180,"kb_used_data":3112,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323660,"statfs":{"total":96636764160,"available":96587427840,"internally_reserved":0,"allocated":3186688,"data_stored":1643285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.57699999999999996,"15min":0.59799999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.31900000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56999999999999995,"15min":0.60099999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82599999999999996,"15min":1.3100000000000001},"last":0.498}]},{"osd":1,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58199999999999996,"15min":0.60199999999999998},"min":{"1min":0.36599999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57499999999999996,"15min":0.59799999999999998},"min":{"1min":0.40500000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":1.018},"last":0.64200000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.57899999999999996,"15min":0.59899999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.99299999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58199999999999996,"15min":0.60899999999999999},"min":{"1min":0.41599999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.60599999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59299999999999997,"15min":0.60999999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.2430000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.61399999999999999,"15min":0.64200000000000002},"min":{"1min":0.40600000000000003,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":1.161},"last":0.69599999999999995}]},{"osd":4,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.49299999999999999,"15min":0.47999999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52200000000000002,"15min":0.51100000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.73399999999999999}]},{"osd":6,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.503,"15min":0.504},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.32800000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":0.57099999999999995}]},{"osd":7,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56000000000000005,"15min":0.55100000000000005},"min":{"1min":0.32800000000000001,"5min":0.317,"15min":0.317},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56999999999999995,"15min":0.57299999999999995},"min":{"1min":0.34300000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.72399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:26:30.660 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T20:26:30.661 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-18T20:26:30.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.824+0000 7f292beb9700 1 Processor -- start 2024-03-18T20:26:30.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.825+0000 7f292beb9700 1 -- start start 2024-03-18T20:26:30.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.825+0000 7f292beb9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f292415d5a0 0x7f29240ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.825+0000 7f292beb9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f29240ce550 0x7f292406a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.825+0000 7f292beb9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f29240cfa90 0x7f292406ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.825+0000 7f292beb9700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29240cf600 con 0x7f29240cfa90 2024-03-18T20:26:30.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.825+0000 7f292beb9700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29240ce9f0 con 0x7f29240ce550 2024-03-18T20:26:30.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.825+0000 7f292beb9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29240cecd0 con 0x7f292415d5a0 2024-03-18T20:26:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.826+0000 7f292a456700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f29240cfa90 0x7f292406ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.826+0000 7f292a456700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f29240cfa90 0x7f292406ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46612/0 (socket says 172.21.15.1:46612) 2024-03-18T20:26:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.826+0000 7f292a456700 1 -- 172.21.15.1:0/284802242 learned_addr learned my addr 172.21.15.1:0/284802242 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.826+0000 7f2929c55700 1 --2- 172.21.15.1:0/284802242 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f292415d5a0 0x7f29240ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.826+0000 7f2929454700 1 --2- 172.21.15.1:0/284802242 >> v2:172.21.15.165:3300/0 conn(0x7f29240ce550 0x7f292406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.826+0000 7f292a456700 1 -- 172.21.15.1:0/284802242 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f292415d5a0 msgr2=0x7f29240ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.826+0000 7f292a456700 1 --2- 172.21.15.1:0/284802242 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f292415d5a0 0x7f29240ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.826+0000 7f292a456700 1 -- 172.21.15.1:0/284802242 >> v2:172.21.15.165:3300/0 conn(0x7f29240ce550 msgr2=0x7f292406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.826+0000 7f292a456700 1 --2- 172.21.15.1:0/284802242 >> v2:172.21.15.165:3300/0 conn(0x7f29240ce550 0x7f292406a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.826+0000 7f292a456700 1 -- 172.21.15.1:0/284802242 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29240cefb0 con 0x7f29240cfa90 2024-03-18T20:26:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.826+0000 7f292a456700 1 --2- 172.21.15.1:0/284802242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29240cfa90 0x7f292406ac90 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7f292406ce60 tx=0x7f2914009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45175b7eec97545 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.826+0000 7f2928c53700 1 -- 172.21.15.1:0/284802242 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2914011830 con 0x7f29240cfa90 2024-03-18T20:26:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.826+0000 7f2928c53700 1 -- 172.21.15.1:0/284802242 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2914011990 con 0x7f29240cfa90 2024-03-18T20:26:30.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.826+0000 7f2928c53700 1 -- 172.21.15.1:0/284802242 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2914011d20 con 0x7f29240cfa90 2024-03-18T20:26:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.827+0000 7f292beb9700 1 -- 172.21.15.1:0/284802242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29240cfa90 msgr2=0x7f292406ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.827+0000 7f292beb9700 1 --2- 172.21.15.1:0/284802242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29240cfa90 0x7f292406ac90 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7f292406ce60 tx=0x7f2914009bd0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.827+0000 7f292beb9700 1 -- 172.21.15.1:0/284802242 shutdown_connections 2024-03-18T20:26:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.827+0000 7f292beb9700 1 --2- 172.21.15.1:0/284802242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29240cfa90 0x7f292406ac90 unknown :-1 s=CLOSED pgs=2818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.827+0000 7f292beb9700 1 --2- 172.21.15.1:0/284802242 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f292415d5a0 0x7f29240ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.827+0000 7f292beb9700 1 --2- 172.21.15.1:0/284802242 >> v2:172.21.15.165:3300/0 conn(0x7f29240ce550 0x7f292406a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.827+0000 7f292beb9700 1 -- 172.21.15.1:0/284802242 >> 172.21.15.1:0/284802242 conn(0x7f29240cc9b0 msgr2=0x7f29240ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.827+0000 7f292beb9700 1 -- 172.21.15.1:0/284802242 shutdown_connections 2024-03-18T20:26:30.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.827+0000 7f292beb9700 1 -- 172.21.15.1:0/284802242 wait complete. 2024-03-18T20:26:30.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.828+0000 7f292beb9700 1 Processor -- start 2024-03-18T20:26:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.828+0000 7f292beb9700 1 -- start start 2024-03-18T20:26:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.829+0000 7f292beb9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29240ce550 0x7f292415f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.829+0000 7f292beb9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29240cfa90 0x7f2924157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.829+0000 7f292beb9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f292415d5a0 0x7f29240b5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.829+0000 7f292beb9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f292415fb10 con 0x7f29240ce550 2024-03-18T20:26:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.829+0000 7f292beb9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f292415fdf0 con 0x7f29240cfa90 2024-03-18T20:26:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.829+0000 7f292beb9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29241600d0 con 0x7f292415d5a0 2024-03-18T20:26:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.829+0000 7f2929c55700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29240ce550 0x7f292415f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.829+0000 7f2929c55700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29240ce550 0x7f292415f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46622/0 (socket says 172.21.15.1:46622) 2024-03-18T20:26:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.829+0000 7f2929c55700 1 -- 172.21.15.1:0/11614963 learned_addr learned my addr 172.21.15.1:0/11614963 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.829+0000 7f292a456700 1 --2- 172.21.15.1:0/11614963 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f292415d5a0 0x7f29240b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.829+0000 7f2929454700 1 --2- 172.21.15.1:0/11614963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29240cfa90 0x7f2924157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.829+0000 7f2929c55700 1 -- 172.21.15.1:0/11614963 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f292415d5a0 msgr2=0x7f29240b5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.829+0000 7f2929c55700 1 --2- 172.21.15.1:0/11614963 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f292415d5a0 0x7f29240b5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.829+0000 7f2929c55700 1 -- 172.21.15.1:0/11614963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29240cfa90 msgr2=0x7f2924157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.829+0000 7f2929c55700 1 --2- 172.21.15.1:0/11614963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29240cfa90 0x7f2924157a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.829+0000 7f2929c55700 1 -- 172.21.15.1:0/11614963 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2914009550 con 0x7f29240ce550 2024-03-18T20:26:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.829+0000 7f2929c55700 1 --2- 172.21.15.1:0/11614963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29240ce550 0x7f292415f270 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 crypto rx=0x7f291c00efc0 tx=0x7f291c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.830+0000 7f29227fc700 1 -- 172.21.15.1:0/11614963 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291c00cdc0 con 0x7f29240ce550 2024-03-18T20:26:30.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.830+0000 7f29227fc700 1 -- 172.21.15.1:0/11614963 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f291c00f040 con 0x7f29240ce550 2024-03-18T20:26:30.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.830+0000 7f292beb9700 1 -- 172.21.15.1:0/11614963 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29241603b0 con 0x7f29240ce550 2024-03-18T20:26:30.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.830+0000 7f29227fc700 1 -- 172.21.15.1:0/11614963 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291c0146a0 con 0x7f29240ce550 2024-03-18T20:26:30.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.830+0000 7f292beb9700 1 -- 172.21.15.1:0/11614963 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2924160bb0 con 0x7f29240ce550 2024-03-18T20:26:30.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.831+0000 7f292beb9700 1 -- 172.21.15.1:0/11614963 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29240676a0 con 0x7f29240ce550 2024-03-18T20:26:30.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.832+0000 7f29227fc700 1 -- 172.21.15.1:0/11614963 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f291c014800 con 0x7f29240ce550 2024-03-18T20:26:30.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.832+0000 7f29227fc700 1 --2- 172.21.15.1:0/11614963 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f291006aed0 0x7f291006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:30.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.832+0000 7f29227fc700 1 -- 172.21.15.1:0/11614963 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f291c08a7d0 con 0x7f29240ce550 2024-03-18T20:26:30.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.832+0000 7f2929454700 1 --2- 172.21.15.1:0/11614963 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f291006aed0 0x7f291006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:30.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.833+0000 7f2929454700 1 --2- 172.21.15.1:0/11614963 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f291006aed0 0x7f291006d390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2924072190 tx=0x7f2918008040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:30.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:30.836+0000 7f29227fc700 1 -- 172.21.15.1:0/11614963 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f291c055500 con 0x7f29240ce550 2024-03-18T20:26:31.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.122+0000 7f292beb9700 1 -- 172.21.15.1:0/11614963 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2924160a00 con 0x7f29240ce550 2024-03-18T20:26:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.124+0000 7f29227fc700 1 -- 172.21.15.1:0/11614963 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v52) v1 ==== 190+0+0 (secure 0 0 0) 0x7f291c058b20 con 0x7f29240ce550 2024-03-18T20:26:31.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.126+0000 7f292beb9700 1 -- 172.21.15.1:0/11614963 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f291006aed0 msgr2=0x7f291006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:31.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.126+0000 7f292beb9700 1 --2- 172.21.15.1:0/11614963 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f291006aed0 0x7f291006d390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2924072190 tx=0x7f2918008040 comp rx=0 tx=0).stop 2024-03-18T20:26:31.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.126+0000 7f292beb9700 1 -- 172.21.15.1:0/11614963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29240ce550 msgr2=0x7f292415f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:31.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.126+0000 7f292beb9700 1 --2- 172.21.15.1:0/11614963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29240ce550 0x7f292415f270 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 crypto rx=0x7f291c00efc0 tx=0x7f291c00c3a0 comp rx=0 tx=0).stop 2024-03-18T20:26:31.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.127+0000 7f292beb9700 1 -- 172.21.15.1:0/11614963 shutdown_connections 2024-03-18T20:26:31.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.127+0000 7f292beb9700 1 --2- 172.21.15.1:0/11614963 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f291006aed0 0x7f291006d390 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:31.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.127+0000 7f292beb9700 1 --2- 172.21.15.1:0/11614963 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f292415d5a0 0x7f29240b5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:31.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.127+0000 7f292beb9700 1 --2- 172.21.15.1:0/11614963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29240ce550 0x7f292415f270 unknown :-1 s=CLOSED pgs=2819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:31.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.127+0000 7f292beb9700 1 --2- 172.21.15.1:0/11614963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29240cfa90 0x7f2924157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:31.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.127+0000 7f292beb9700 1 -- 172.21.15.1:0/11614963 >> 172.21.15.1:0/11614963 conn(0x7f29240cc9b0 msgr2=0x7f292406e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:31.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.127+0000 7f292beb9700 1 -- 172.21.15.1:0/11614963 shutdown_connections 2024-03-18T20:26:31.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.127+0000 7f292beb9700 1 -- 172.21.15.1:0/11614963 wait complete. 2024-03-18T20:26:31.132 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-18T20:26:31.143 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:26:31.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.264+0000 7f15bfd28700 1 Processor -- start 2024-03-18T20:26:31.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.264+0000 7f15bfd28700 1 -- start start 2024-03-18T20:26:31.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.265+0000 7f15bfd28700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f15b8146190 0x7f15b806a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:31.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.265+0000 7f15bfd28700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f15b81465a0 0x7f15b806ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:31.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.265+0000 7f15bfd28700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b806ebd0 con 0x7f15b8146af0 2024-03-18T20:26:31.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.265+0000 7f15bfd28700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b806f850 con 0x7f15b81465a0 2024-03-18T20:26:31.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.265+0000 7f15bfd28700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b806eeb0 con 0x7f15b8146190 2024-03-18T20:26:31.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.265+0000 7f15bd2c3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f15b81465a0 0x7f15b806ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:31.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.265+0000 7f15bd2c3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f15b81465a0 0x7f15b806ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46632/0 (socket says 172.21.15.1:46632) 2024-03-18T20:26:31.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.265+0000 7f15bd2c3700 1 -- 172.21.15.1:0/3393062649 learned_addr learned my addr 172.21.15.1:0/3393062649 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:31.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.265+0000 7f15bdac4700 1 --2- 172.21.15.1:0/3393062649 >> v2:172.21.15.165:3300/0 conn(0x7f15b8146190 0x7f15b806a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:31.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.265+0000 7f15bd2c3700 1 -- 172.21.15.1:0/3393062649 >> v2:172.21.15.165:3300/0 conn(0x7f15b8146190 msgr2=0x7f15b806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:31.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.265+0000 7f15bd2c3700 1 --2- 172.21.15.1:0/3393062649 >> v2:172.21.15.165:3300/0 conn(0x7f15b8146190 0x7f15b806a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:31.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.265+0000 7f15bd2c3700 1 -- 172.21.15.1:0/3393062649 >> v1:172.21.15.165:6789/0 conn(0x7f15b8146af0 legacy=0x7f15b8071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:31.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.265+0000 7f15bd2c3700 1 -- 172.21.15.1:0/3393062649 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b806f190 con 0x7f15b81465a0 2024-03-18T20:26:31.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.266+0000 7f15bd2c3700 1 --2- 172.21.15.1:0/3393062649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b81465a0 0x7f15b806ac90 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 crypto rx=0x7f15b806e8f0 tx=0x7f15b4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb8c362b96430ac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:31.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.266+0000 7f15bcac2700 1 -- 172.21.15.1:0/3393062649 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b4011830 con 0x7f15b81465a0 2024-03-18T20:26:31.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.266+0000 7f15bcac2700 1 -- 172.21.15.1:0/3393062649 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b4011990 con 0x7f15b81465a0 2024-03-18T20:26:31.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.266+0000 7f15bcac2700 1 -- 172.21.15.1:0/3393062649 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b4011d20 con 0x7f15b81465a0 2024-03-18T20:26:31.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.266+0000 7f15bfd28700 1 -- 172.21.15.1:0/3393062649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b81465a0 msgr2=0x7f15b806ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:31.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.266+0000 7f15bfd28700 1 --2- 172.21.15.1:0/3393062649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b81465a0 0x7f15b806ac90 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 crypto rx=0x7f15b806e8f0 tx=0x7f15b4009bd0 comp rx=0 tx=0).stop 2024-03-18T20:26:31.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.266+0000 7f15bfd28700 1 -- 172.21.15.1:0/3393062649 shutdown_connections 2024-03-18T20:26:31.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.266+0000 7f15bfd28700 1 --2- 172.21.15.1:0/3393062649 >> v2:172.21.15.165:3300/0 conn(0x7f15b8146190 0x7f15b806a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:31.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.266+0000 7f15bfd28700 1 --2- 172.21.15.1:0/3393062649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b81465a0 0x7f15b806ac90 unknown :-1 s=CLOSED pgs=2820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:31.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.266+0000 7f15bfd28700 1 -- 172.21.15.1:0/3393062649 >> 172.21.15.1:0/3393062649 conn(0x7f15b80b1d20 msgr2=0x7f15b80b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:31.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.266+0000 7f15bfd28700 1 -- 172.21.15.1:0/3393062649 shutdown_connections 2024-03-18T20:26:31.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.267+0000 7f15bfd28700 1 -- 172.21.15.1:0/3393062649 wait complete. 2024-03-18T20:26:31.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.268+0000 7f15bfd28700 1 Processor -- start 2024-03-18T20:26:31.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.268+0000 7f15bfd28700 1 -- start start 2024-03-18T20:26:31.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.269+0000 7f15bfd28700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b81465a0 0x7f15b815c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:31.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.269+0000 7f15bfd28700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15b8146af0 0x7f15b81417a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:31.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.269+0000 7f15bfd28700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15b815c620 0x7f15b81e72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:31.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.269+0000 7f15bfd28700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b81e6af0 con 0x7f15b81465a0 2024-03-18T20:26:31.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.269+0000 7f15bfd28700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b81421f0 con 0x7f15b8146af0 2024-03-18T20:26:31.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.269+0000 7f15bfd28700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b81424d0 con 0x7f15b815c620 2024-03-18T20:26:31.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.269+0000 7f15bdac4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b81465a0 0x7f15b815c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:31.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.269+0000 7f15bdac4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b81465a0 0x7f15b815c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46636/0 (socket says 172.21.15.1:46636) 2024-03-18T20:26:31.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.269+0000 7f15bdac4700 1 -- 172.21.15.1:0/751342270 learned_addr learned my addr 172.21.15.1:0/751342270 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:31.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.269+0000 7f15bd2c3700 1 --2- 172.21.15.1:0/751342270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15b8146af0 0x7f15b81417a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:31.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.269+0000 7f15be2c5700 1 --2- 172.21.15.1:0/751342270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15b815c620 0x7f15b81e72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:31.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.269+0000 7f15bdac4700 1 -- 172.21.15.1:0/751342270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15b815c620 msgr2=0x7f15b81e72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:31.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.269+0000 7f15bdac4700 1 --2- 172.21.15.1:0/751342270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15b815c620 0x7f15b81e72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:31.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.269+0000 7f15bdac4700 1 -- 172.21.15.1:0/751342270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15b8146af0 msgr2=0x7f15b81417a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:31.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.269+0000 7f15bdac4700 1 --2- 172.21.15.1:0/751342270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15b8146af0 0x7f15b81417a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:31.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.269+0000 7f15bdac4700 1 -- 172.21.15.1:0/751342270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b4009550 con 0x7f15b81465a0 2024-03-18T20:26:31.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.270+0000 7f15bdac4700 1 --2- 172.21.15.1:0/751342270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b81465a0 0x7f15b815c0e0 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7f15a800e9b0 tx=0x7f15a800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:31.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.270+0000 7f15ae7fc700 1 -- 172.21.15.1:0/751342270 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a8009710 con 0x7f15b81465a0 2024-03-18T20:26:31.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.270+0000 7f15ae7fc700 1 -- 172.21.15.1:0/751342270 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a800c850 con 0x7f15b81465a0 2024-03-18T20:26:31.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.270+0000 7f15bfd28700 1 -- 172.21.15.1:0/751342270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b81e6d90 con 0x7f15b81465a0 2024-03-18T20:26:31.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.270+0000 7f15ae7fc700 1 -- 172.21.15.1:0/751342270 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a800cbc0 con 0x7f15b81465a0 2024-03-18T20:26:31.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.270+0000 7f15bfd28700 1 -- 172.21.15.1:0/751342270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b813eb00 con 0x7f15b81465a0 2024-03-18T20:26:31.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.271+0000 7f15bfd28700 1 -- 172.21.15.1:0/751342270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b81e7100 con 0x7f15b81465a0 2024-03-18T20:26:31.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.271+0000 7f15ae7fc700 1 -- 172.21.15.1:0/751342270 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f15a800cd20 con 0x7f15b81465a0 2024-03-18T20:26:31.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.272+0000 7f15ae7fc700 1 --2- 172.21.15.1:0/751342270 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f159c06aed0 0x7f159c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:31.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.272+0000 7f15ae7fc700 1 -- 172.21.15.1:0/751342270 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f15a8089a00 con 0x7f15b81465a0 2024-03-18T20:26:31.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.272+0000 7f15bd2c3700 1 --2- 172.21.15.1:0/751342270 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f159c06aed0 0x7f159c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:31.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.272+0000 7f15bd2c3700 1 --2- 172.21.15.1:0/751342270 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f159c06aed0 0x7f159c06d390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009c10 tx=0x7f15b4008030 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:31.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.276+0000 7f15ae7fc700 1 -- 172.21.15.1:0/751342270 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f15a80546d0 con 0x7f15b81465a0 2024-03-18T20:26:31.343 INFO:tasks.ceph.mds.d.smithi165.stderr:2024-03-18T20:26:31.338+0000 7f776e480700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T20:26:31.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.529+0000 7f15bfd28700 1 -- 172.21.15.1:0/751342270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15b80b3da0 con 0x7f15b81465a0 2024-03-18T20:26:31.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.529+0000 7f15ae7fc700 1 -- 172.21.15.1:0/751342270 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+4747 (secure 0 0 0) 0x7f15a8017090 con 0x7f15b81465a0 2024-03-18T20:26:31.534 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:31.534 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":120,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12396,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2565889517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2565889517},{"type":"v1","addr":"172.21.15.1:6836","nonce":2565889517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":12399,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/512665659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":512665659},{"type":"v1","addr":"172.21.15.1:6837","nonce":512665659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":12408,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/1134181166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1134181166},{"type":"v1","addr":"172.21.15.165:6834","nonce":1134181166}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:26:25.785074+0000","modified":"2024-03-18T20:26:29.101123+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12411":{"gid":12411,"name":"d","rank":0,"incarnation":116,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/3886150679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":3886150679},{"type":"v1","addr":"172.21.15.165:6835","nonce":3886150679}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-03-18T20:26:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.532+0000 7f15bfd28700 1 -- 172.21.15.1:0/751342270 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f159c06aed0 msgr2=0x7f159c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.532+0000 7f15bfd28700 1 --2- 172.21.15.1:0/751342270 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f159c06aed0 0x7f159c06d390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009c10 tx=0x7f15b4008030 comp rx=0 tx=0).stop 2024-03-18T20:26:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.532+0000 7f15bfd28700 1 -- 172.21.15.1:0/751342270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b81465a0 msgr2=0x7f15b815c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.532+0000 7f15bfd28700 1 --2- 172.21.15.1:0/751342270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b81465a0 0x7f15b815c0e0 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7f15a800e9b0 tx=0x7f15a800ed70 comp rx=0 tx=0).stop 2024-03-18T20:26:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.532+0000 7f15bfd28700 1 -- 172.21.15.1:0/751342270 shutdown_connections 2024-03-18T20:26:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.532+0000 7f15bfd28700 1 --2- 172.21.15.1:0/751342270 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f159c06aed0 0x7f159c06d390 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.532+0000 7f15bfd28700 1 --2- 172.21.15.1:0/751342270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f15b815c620 0x7f15b81e72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.532+0000 7f15bfd28700 1 --2- 172.21.15.1:0/751342270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b81465a0 0x7f15b815c0e0 unknown :-1 s=CLOSED pgs=2821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:31.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.532+0000 7f15bfd28700 1 --2- 172.21.15.1:0/751342270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15b8146af0 0x7f15b81417a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:31.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.532+0000 7f15bfd28700 1 -- 172.21.15.1:0/751342270 >> 172.21.15.1:0/751342270 conn(0x7f15b80b1d20 msgr2=0x7f15b80b1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:31.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.532+0000 7f15bfd28700 1 -- 172.21.15.1:0/751342270 shutdown_connections 2024-03-18T20:26:31.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:31.532+0000 7f15bfd28700 1 -- 172.21.15.1:0/751342270 wait complete. 2024-03-18T20:26:31.537 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 120 2024-03-18T20:26:31.953 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 120, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T20:26:25.785074+0000', 'modified': '2024-03-18T20:26:29.101123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12411}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12411': {'gid': 12411, 'name': 'd', 'rank': 0, 'incarnation': 116, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.165:6835/3886150679', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6833', 'nonce': 3886150679}, {'type': 'v1', 'addr': '172.21.15.165:6835', 'nonce': 3886150679}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [32], 'metadata_pool': 31, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T20:26:31.953 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T20:26:31.954 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-18T20:26:32.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.086+0000 7f43e9fcb700 1 Processor -- start 2024-03-18T20:26:32.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.086+0000 7f43e9fcb700 1 -- start start 2024-03-18T20:26:32.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.087+0000 7f43e9fcb700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f43e4071d30 0x7f43e40add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.087+0000 7f43e9fcb700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e40ce8f0 con 0x7f43e415a1d0 2024-03-18T20:26:32.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.087+0000 7f43e9fcb700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e40ae260 con 0x7f43e40cb9e0 2024-03-18T20:26:32.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.087+0000 7f43e9fcb700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e40cdb00 con 0x7f43e4071d30 2024-03-18T20:26:32.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.087+0000 7f43e37fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f43e415a1d0 0x7f43e4071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53536/0 (socket says 172.21.15.1:53536) 2024-03-18T20:26:32.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.087+0000 7f43e37fe700 1 -- 172.21.15.1:0/25281337 learned_addr learned my addr 172.21.15.1:0/25281337 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:32.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e2ffd700 1 --2- 172.21.15.1:0/25281337 >> v2:172.21.15.165:3300/0 conn(0x7f43e4071d30 0x7f43e40add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e27fc700 1 -- 172.21.15.1:0/25281337 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248669080 0 0) 0x7f43e40ce8f0 con 0x7f43e415a1d0 2024-03-18T20:26:32.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e27fc700 1 -- 172.21.15.1:0/25281337 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d00035b0 con 0x7f43e415a1d0 2024-03-18T20:26:32.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e27fc700 1 -- 172.21.15.1:0/25281337 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3368916638 0 0) 0x7f43d00035b0 con 0x7f43e415a1d0 2024-03-18T20:26:32.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e27fc700 1 -- 172.21.15.1:0/25281337 >> v2:172.21.15.165:3300/0 conn(0x7f43e4071d30 msgr2=0x7f43e40add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:32.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e27fc700 1 --2- 172.21.15.1:0/25281337 >> v2:172.21.15.165:3300/0 conn(0x7f43e4071d30 0x7f43e40add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e27fc700 1 -- 172.21.15.1:0/25281337 >> v1:172.21.15.165:6789/0 conn(0x7f43e40cb9e0 legacy=0x7f43e40cbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:32.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e27fc700 1 -- 172.21.15.1:0/25281337 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e40cdce0 con 0x7f43e415a1d0 2024-03-18T20:26:32.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e27fc700 1 -- 172.21.15.1:0/25281337 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f43cc0017f0 con 0x7f43e415a1d0 2024-03-18T20:26:32.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e27fc700 1 -- 172.21.15.1:0/25281337 >> v1:172.21.15.1:6789/0 conn(0x7f43e415a1d0 legacy=0x7f43e4071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:32.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e27fc700 1 --2- 172.21.15.1:0/25281337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d0003a00 0x7f43d0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e27fc700 1 --2- 172.21.15.1:0/25281337 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43d0004400 0x7f43d00048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e27fc700 1 --2- 172.21.15.1:0/25281337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43d00050d0 0x7f43d000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e27fc700 1 -- 172.21.15.1:0/25281337 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e40cdb00 con 0x7f43d0003a00 2024-03-18T20:26:32.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e27fc700 1 -- 172.21.15.1:0/25281337 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d000c030 con 0x7f43d00050d0 2024-03-18T20:26:32.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e27fc700 1 -- 172.21.15.1:0/25281337 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d000c1f0 con 0x7f43d0004400 2024-03-18T20:26:32.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e37fe700 1 --2- 172.21.15.1:0/25281337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d0003a00 0x7f43d0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e2ffd700 1 --2- 172.21.15.1:0/25281337 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43d0004400 0x7f43d00048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e3fff700 1 --2- 172.21.15.1:0/25281337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43d00050d0 0x7f43d000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e37fe700 1 -- 172.21.15.1:0/25281337 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43d0004400 msgr2=0x7f43d00048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:32.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e37fe700 1 --2- 172.21.15.1:0/25281337 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43d0004400 0x7f43d00048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e37fe700 1 -- 172.21.15.1:0/25281337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43d00050d0 msgr2=0x7f43d000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:32.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.088+0000 7f43e37fe700 1 --2- 172.21.15.1:0/25281337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43d00050d0 0x7f43d000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.089+0000 7f43e37fe700 1 -- 172.21.15.1:0/25281337 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d00035b0 con 0x7f43d0003a00 2024-03-18T20:26:32.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.089+0000 7f43e3fff700 1 --2- 172.21.15.1:0/25281337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43d00050d0 0x7f43d000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:32.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.089+0000 7f43e3fff700 1 -- 172.21.15.1:0/25281337 reap_dead start 2024-03-18T20:26:32.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.089+0000 7f43e37fe700 1 --2- 172.21.15.1:0/25281337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d0003a00 0x7f43d0003de0 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00c1c0 tx=0x7f43d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3ade4e82c6a4cae server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:32.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.089+0000 7f43e27fc700 1 -- 172.21.15.1:0/25281337 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc00cd60 con 0x7f43d0003a00 2024-03-18T20:26:32.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.089+0000 7f43e27fc700 1 -- 172.21.15.1:0/25281337 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43cc007b20 con 0x7f43d0003a00 2024-03-18T20:26:32.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.089+0000 7f43e27fc700 1 -- 172.21.15.1:0/25281337 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc007310 con 0x7f43d0003a00 2024-03-18T20:26:32.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.089+0000 7f43e9fcb700 1 -- 172.21.15.1:0/25281337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d0003a00 msgr2=0x7f43d0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:32.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.089+0000 7f43e9fcb700 1 --2- 172.21.15.1:0/25281337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d0003a00 0x7f43d0003de0 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00c1c0 tx=0x7f43d0005880 comp rx=0 tx=0).stop 2024-03-18T20:26:32.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.089+0000 7f43e9fcb700 1 -- 172.21.15.1:0/25281337 shutdown_connections 2024-03-18T20:26:32.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.089+0000 7f43e9fcb700 1 --2- 172.21.15.1:0/25281337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d0003a00 0x7f43d0003de0 unknown :-1 s=CLOSED pgs=2823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.089+0000 7f43e9fcb700 1 -- 172.21.15.1:0/25281337 >> 172.21.15.1:0/25281337 conn(0x7f43e40cc9b0 msgr2=0x7f43e40ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:32.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.089+0000 7f43e9fcb700 1 -- 172.21.15.1:0/25281337 shutdown_connections 2024-03-18T20:26:32.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.089+0000 7f43e9fcb700 1 -- 172.21.15.1:0/25281337 wait complete. 2024-03-18T20:26:32.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.091+0000 7f43e9fcb700 1 Processor -- start 2024-03-18T20:26:32.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.091+0000 7f43e9fcb700 1 -- start start 2024-03-18T20:26:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43e9fcb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d0003a00 0x7f43e41572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43e9fcb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43e4155d00 0x7f43e41546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43e9fcb700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43e415cb60 0x7f43e40c9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43e9fcb700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e4156200 con 0x7f43d0003a00 2024-03-18T20:26:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43e9fcb700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e41564e0 con 0x7f43e415cb60 2024-03-18T20:26:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43e9fcb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e41567c0 con 0x7f43e4155d00 2024-03-18T20:26:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43e37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d0003a00 0x7f43e41572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43e37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d0003a00 0x7f43e41572a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46656/0 (socket says 172.21.15.1:46656) 2024-03-18T20:26:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43e37fe700 1 -- 172.21.15.1:0/2696754737 learned_addr learned my addr 172.21.15.1:0/2696754737 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43e3fff700 1 --2- 172.21.15.1:0/2696754737 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43e415cb60 0x7f43e40c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43e2ffd700 1 --2- 172.21.15.1:0/2696754737 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43e4155d00 0x7f43e41546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43e37fe700 1 -- 172.21.15.1:0/2696754737 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43e4155d00 msgr2=0x7f43e41546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43e37fe700 1 --2- 172.21.15.1:0/2696754737 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43e4155d00 0x7f43e41546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43e37fe700 1 -- 172.21.15.1:0/2696754737 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43e415cb60 msgr2=0x7f43e40c9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43e37fe700 1 --2- 172.21.15.1:0/2696754737 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43e415cb60 0x7f43e40c9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43e37fe700 1 -- 172.21.15.1:0/2696754737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e4156aa0 con 0x7f43d0003a00 2024-03-18T20:26:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43e37fe700 1 --2- 172.21.15.1:0/2696754737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d0003a00 0x7f43e41572a0 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7f43cc001a30 tx=0x7f43cc001b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43dbfff700 1 -- 172.21.15.1:0/2696754737 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc005a30 con 0x7f43d0003a00 2024-03-18T20:26:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43dbfff700 1 -- 172.21.15.1:0/2696754737 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43cc005bb0 con 0x7f43d0003a00 2024-03-18T20:26:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43e9fcb700 1 -- 172.21.15.1:0/2696754737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e4156d80 con 0x7f43d0003a00 2024-03-18T20:26:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43dbfff700 1 -- 172.21.15.1:0/2696754737 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc000f70 con 0x7f43d0003a00 2024-03-18T20:26:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.092+0000 7f43e9fcb700 1 -- 172.21.15.1:0/2696754737 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e415d1f0 con 0x7f43d0003a00 2024-03-18T20:26:32.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.094+0000 7f43dbfff700 1 -- 172.21.15.1:0/2696754737 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f43cc01e430 con 0x7f43d0003a00 2024-03-18T20:26:32.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.095+0000 7f43dbfff700 1 --2- 172.21.15.1:0/2696754737 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f43c406aed0 0x7f43c406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.095+0000 7f43dbfff700 1 -- 172.21.15.1:0/2696754737 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f43cc08a7f0 con 0x7f43d0003a00 2024-03-18T20:26:32.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.095+0000 7f43e2ffd700 1 --2- 172.21.15.1:0/2696754737 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f43c406aed0 0x7f43c406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.095+0000 7f43e2ffd700 1 --2- 172.21.15.1:0/2696754737 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f43c406aed0 0x7f43c406d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f43e40c6c90 tx=0x7f43d4008040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:32.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.097+0000 7f43f6174b80 1 Processor -- start 2024-03-18T20:26:32.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.097+0000 7f43f6174b80 1 -- start start 2024-03-18T20:26:32.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.098+0000 7f43f6174b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6b9bc10 0x558da6b9bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.098+0000 7f43f6174b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b6d0a0 0x558da6bb8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.098+0000 7f43f6174b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6bad630 0x558da6b70160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.098+0000 7f43f6174b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x558da6ae6f20 con 0x558da6b9bc10 2024-03-18T20:26:32.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.098+0000 7f43f6174b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x558da6ac8c30 con 0x558da6bad630 2024-03-18T20:26:32.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.098+0000 7f43f6174b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x558da6ac6430 con 0x558da6b6d0a0 2024-03-18T20:26:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.098+0000 7f43e37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6b9bc10 0x558da6b9bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.098+0000 7f43e37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6b9bc10 0x558da6b9bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46672/0 (socket says 172.21.15.1:46672) 2024-03-18T20:26:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.098+0000 7f43e37fe700 1 -- 172.21.15.1:0/2831566506 learned_addr learned my addr 172.21.15.1:0/2831566506 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.098+0000 7f43e2ffd700 1 --2- 172.21.15.1:0/2831566506 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b6d0a0 0x558da6bb8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.098+0000 7f43e3fff700 1 --2- 172.21.15.1:0/2831566506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6bad630 0x558da6b70160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.098+0000 7f43e37fe700 1 -- 172.21.15.1:0/2831566506 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b6d0a0 msgr2=0x558da6bb8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.098+0000 7f43e37fe700 1 --2- 172.21.15.1:0/2831566506 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b6d0a0 0x558da6bb8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.098+0000 7f43e37fe700 1 -- 172.21.15.1:0/2831566506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6bad630 msgr2=0x558da6b70160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.098+0000 7f43e37fe700 1 --2- 172.21.15.1:0/2831566506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6bad630 0x558da6b70160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.098+0000 7f43e37fe700 1 -- 172.21.15.1:0/2831566506 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558da68b5790 con 0x558da6b9bc10 2024-03-18T20:26:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.098+0000 7f43e37fe700 1 --2- 172.21.15.1:0/2831566506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6b9bc10 0x558da6b9bff0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f43cc000c00 tx=0x7f43cc007880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0876a41b287f3e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.098+0000 7f43d97fa700 1 -- 172.21.15.1:0/2831566506 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc0592d0 con 0x558da6b9bc10 2024-03-18T20:26:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.099+0000 7f43d97fa700 1 -- 172.21.15.1:0/2831566506 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43cc05d100 con 0x558da6b9bc10 2024-03-18T20:26:32.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.099+0000 7f43d97fa700 1 -- 172.21.15.1:0/2831566506 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc0616c0 con 0x558da6b9bc10 2024-03-18T20:26:32.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.099+0000 7f43f6174b80 1 -- 172.21.15.1:0/2831566506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6b9bc10 msgr2=0x558da6b9bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:32.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.099+0000 7f43f6174b80 1 --2- 172.21.15.1:0/2831566506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6b9bc10 0x558da6b9bff0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f43cc000c00 tx=0x7f43cc007880 comp rx=0 tx=0).stop 2024-03-18T20:26:32.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.099+0000 7f43f6174b80 1 -- 172.21.15.1:0/2831566506 shutdown_connections 2024-03-18T20:26:32.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.099+0000 7f43f6174b80 1 --2- 172.21.15.1:0/2831566506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6bad630 0x558da6b70160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.099+0000 7f43f6174b80 1 --2- 172.21.15.1:0/2831566506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6b9bc10 0x558da6b9bff0 unknown :-1 s=CLOSED pgs=2825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.099+0000 7f43f6174b80 1 --2- 172.21.15.1:0/2831566506 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b6d0a0 0x558da6bb8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.099+0000 7f43f6174b80 1 -- 172.21.15.1:0/2831566506 >> 172.21.15.1:0/2831566506 conn(0x558da6acfcf0 msgr2=0x558da6b6fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:32.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.099+0000 7f43f6174b80 1 -- 172.21.15.1:0/2831566506 shutdown_connections 2024-03-18T20:26:32.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.099+0000 7f43f6174b80 1 -- 172.21.15.1:0/2831566506 wait complete. 2024-03-18T20:26:32.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.100+0000 7f43f6174b80 1 -- start start 2024-03-18T20:26:32.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.100+0000 7f43f6174b80 1 Processor -- start 2024-03-18T20:26:32.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.101+0000 7f43f6174b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6b6d0a0 0x558da6b9f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.101+0000 7f43f6174b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b9bc10 0x558da6bb03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.101+0000 7f43f6174b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6bad630 0x558da6b90a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.101+0000 7f43f6174b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x558da6ac1d20 con 0x558da6bad630 2024-03-18T20:26:32.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.101+0000 7f43f6174b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x558da6a5d770 con 0x558da6b6d0a0 2024-03-18T20:26:32.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.101+0000 7f43f6174b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x558da6a806f0 con 0x558da6b9bc10 2024-03-18T20:26:32.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.101+0000 7f43f6174b80 10 client.0 _ll_register_callbacks cb 0x558da68c95a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:26:32.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.101+0000 7f43e3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6bad630 0x558da6b90a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.101+0000 7f43e3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6bad630 0x558da6b90a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46676/0 (socket says 172.21.15.1:46676) 2024-03-18T20:26:32.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.101+0000 7f43e3fff700 1 -- 172.21.15.1:0/1083018909 learned_addr learned my addr 172.21.15.1:0/1083018909 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:32.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.101+0000 7f43e2ffd700 1 --2- 172.21.15.1:0/1083018909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b9bc10 0x558da6bb03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.101+0000 7f43e37fe700 1 --2- 172.21.15.1:0/1083018909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6b6d0a0 0x558da6b9f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.101+0000 7f43e3fff700 1 -- 172.21.15.1:0/1083018909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b9bc10 msgr2=0x558da6bb03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:32.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.101+0000 7f43e3fff700 1 --2- 172.21.15.1:0/1083018909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b9bc10 0x558da6bb03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.101+0000 7f43e3fff700 1 -- 172.21.15.1:0/1083018909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6b6d0a0 msgr2=0x558da6b9f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:32.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.101+0000 7f43e3fff700 1 --2- 172.21.15.1:0/1083018909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6b6d0a0 0x558da6b9f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.101+0000 7f43e3fff700 1 -- 172.21.15.1:0/1083018909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e415d1f0 con 0x558da6bad630 2024-03-18T20:26:32.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.101+0000 7f43e3fff700 1 --2- 172.21.15.1:0/1083018909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6bad630 0x558da6b90a70 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7f43dc001520 tx=0x7f43dc003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84bc6358454d1322 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:32.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.101+0000 7f43c27fc700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T20:26:32.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.102+0000 7f43c27fc700 1 -- 172.21.15.1:0/1083018909 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc016480 con 0x558da6bad630 2024-03-18T20:26:32.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.102+0000 7f43c27fc700 1 -- 172.21.15.1:0/1083018909 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(253..253 src has 1..253) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f43dc0165e0 con 0x558da6bad630 2024-03-18T20:26:32.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.102+0000 7f43f6174b80 1 -- 172.21.15.1:0/1083018909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558da6b63e90 con 0x558da6bad630 2024-03-18T20:26:32.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.102+0000 7f43c27fc700 1 -- 172.21.15.1:0/1083018909 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=120) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43dc003be0 con 0x558da6bad630 2024-03-18T20:26:32.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.102+0000 7f43f6174b80 10 client.12540 fetch_fsmap learned FSMap version 120 2024-03-18T20:26:32.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.102+0000 7f43f6174b80 1 -- 172.21.15.1:0/1083018909 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=120,monmap=2+}) v3 -- 0x558da69a20a0 con 0x558da6bad630 2024-03-18T20:26:32.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.102+0000 7f43c27fc700 1 -- 172.21.15.1:0/1083018909 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558da6b63e90 con 0x558da6bad630 2024-03-18T20:26:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.102+0000 7f43c27fc700 1 -- 172.21.15.1:0/1083018909 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 120) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f43dc0167a0 con 0x558da6bad630 2024-03-18T20:26:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.102+0000 7f43f6174b80 10 client.12540 fetch_fsmap finished waiting for FSMap version 120 2024-03-18T20:26:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.102+0000 7f43f6174b80 10 client.12540 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-18T20:26:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.102+0000 7f43f6174b80 20 client.12540 populate_metadata read hostname 'smithi001' 2024-03-18T20:26:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.102+0000 7f43f6174b80 1 --2- 172.21.15.1:0/1083018909 >> [v2:172.21.15.165:6833/3886150679,v1:172.21.15.165:6835/3886150679] conn(0x558da6b982c0 0x558da6b986a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.102+0000 7f43f6174b80 4 client.12540 mds_command: new command op to 12411 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T20:26:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.102+0000 7f43f6174b80 1 -- 172.21.15.1:0/1083018909 --> [v2:172.21.15.165:6833/3886150679,v1:172.21.15.165:6835/3886150679] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558da6b41010 con 0x558da6b982c0 2024-03-18T20:26:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.102+0000 7f43e37fe700 1 --2- 172.21.15.1:0/1083018909 >> [v2:172.21.15.165:6833/3886150679,v1:172.21.15.165:6835/3886150679] conn(0x558da6b982c0 0x558da6b986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.103+0000 7f43c27fc700 10 client.12540 ms_handle_connect on v2:172.21.15.165:6833/3886150679 2024-03-18T20:26:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.103+0000 7f43e37fe700 1 --2- 172.21.15.1:0/1083018909 >> [v2:172.21.15.165:6833/3886150679,v1:172.21.15.165:6835/3886150679] conn(0x558da6b982c0 0x558da6b986a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d07b0a3b39e0f5f server_cookie=edaa25ccc48a5888 in_seq=0 out_seq=0 2024-03-18T20:26:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.103+0000 7f43c27fc700 1 -- 172.21.15.1:0/1083018909 <== mds.0 v2:172.21.15.165:6833/3886150679 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558da6b41010 con 0x558da6b982c0 2024-03-18T20:26:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.103+0000 7f43c27fc700 10 client.12540 handle_command_reply: tid=0 2024-03-18T20:26:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.104+0000 7f43f6174b80 1 client.12540 shutdown 2024-03-18T20:26:32.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.104+0000 7f43f6174b80 1 -- 172.21.15.1:0/1083018909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6bad630 msgr2=0x558da6b90a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.104+0000 7f43f6174b80 1 --2- 172.21.15.1:0/1083018909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6bad630 0x558da6b90a70 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7f43dc001520 tx=0x7f43dc003f90 comp rx=0 tx=0).stop 2024-03-18T20:26:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.104+0000 7f43f6174b80 1 -- 172.21.15.1:0/1083018909 shutdown_connections 2024-03-18T20:26:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.104+0000 7f43f6174b80 1 --2- 172.21.15.1:0/1083018909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6bad630 0x558da6b90a70 unknown :-1 s=CLOSED pgs=2826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.104+0000 7f43f6174b80 1 --2- 172.21.15.1:0/1083018909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6b6d0a0 0x558da6b9f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.104+0000 7f43f6174b80 1 --2- 172.21.15.1:0/1083018909 >> [v2:172.21.15.165:6833/3886150679,v1:172.21.15.165:6835/3886150679] conn(0x558da6b982c0 0x558da6b986a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.104+0000 7f43f6174b80 1 --2- 172.21.15.1:0/1083018909 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b9bc10 0x558da6bb03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.104+0000 7f43c27fc700 0 client.12540 ms_handle_reset on v2:172.21.15.165:6833/3886150679 2024-03-18T20:26:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.104+0000 7f43f6174b80 1 -- 172.21.15.1:0/1083018909 >> 172.21.15.1:0/1083018909 conn(0x558da6acfcf0 msgr2=0x558da6b96140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.104+0000 7f43f6174b80 1 -- 172.21.15.1:0/1083018909 shutdown_connections 2024-03-18T20:26:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.104+0000 7f43f6174b80 1 -- 172.21.15.1:0/1083018909 wait complete. 2024-03-18T20:26:32.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.104+0000 7f43f6174b80 20 client.12540 trim_cache size 0 max 16384 2024-03-18T20:26:32.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.122+0000 7f43f6174b80 1 Processor -- start 2024-03-18T20:26:32.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.122+0000 7f43f6174b80 1 -- start start 2024-03-18T20:26:32.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.123+0000 7f43f6174b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6b6d0a0 0x558da6bafbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.123+0000 7f43f6174b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b9bc10 0x558da6b99890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.123+0000 7f43f6174b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6bad630 0x558da6b986a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.123+0000 7f43f6174b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x558da68b6390 con 0x558da6b6d0a0 2024-03-18T20:26:32.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.123+0000 7f43f6174b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x558da6b6c5d0 con 0x558da6bad630 2024-03-18T20:26:32.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.123+0000 7f43f6174b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x558da6b6e180 con 0x558da6b9bc10 2024-03-18T20:26:32.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.123+0000 7f43e37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6b6d0a0 0x558da6bafbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.123+0000 7f43e37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6b6d0a0 0x558da6bafbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46686/0 (socket says 172.21.15.1:46686) 2024-03-18T20:26:32.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.123+0000 7f43e37fe700 1 -- 172.21.15.1:0/262433569 learned_addr learned my addr 172.21.15.1:0/262433569 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:32.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.123+0000 7f43e2ffd700 1 --2- 172.21.15.1:0/262433569 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b9bc10 0x558da6b99890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.123+0000 7f43e3fff700 1 --2- 172.21.15.1:0/262433569 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6bad630 0x558da6b986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.123+0000 7f43e37fe700 1 -- 172.21.15.1:0/262433569 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b9bc10 msgr2=0x558da6b99890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:32.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.123+0000 7f43e37fe700 1 --2- 172.21.15.1:0/262433569 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b9bc10 0x558da6b99890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.123+0000 7f43e37fe700 1 -- 172.21.15.1:0/262433569 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6bad630 msgr2=0x558da6b986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:32.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.123+0000 7f43e37fe700 1 --2- 172.21.15.1:0/262433569 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6bad630 0x558da6b986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.123+0000 7f43e37fe700 1 -- 172.21.15.1:0/262433569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558da6a88010 con 0x558da6b6d0a0 2024-03-18T20:26:32.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.124+0000 7f43e37fe700 1 --2- 172.21.15.1:0/262433569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6b6d0a0 0x558da6bafbd0 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7f43cc059260 tx=0x7f43cc058f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92dbef58ff11e88 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:32.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.124+0000 7f43d9ffb700 1 -- 172.21.15.1:0/262433569 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc07b070 con 0x558da6b6d0a0 2024-03-18T20:26:32.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.124+0000 7f43d9ffb700 1 -- 172.21.15.1:0/262433569 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43cc0592d0 con 0x558da6b6d0a0 2024-03-18T20:26:32.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.124+0000 7f43d9ffb700 1 -- 172.21.15.1:0/262433569 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc076040 con 0x558da6b6d0a0 2024-03-18T20:26:32.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.125+0000 7f43f6174b80 1 -- 172.21.15.1:0/262433569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6b6d0a0 msgr2=0x558da6bafbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:32.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.125+0000 7f43f6174b80 1 --2- 172.21.15.1:0/262433569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6b6d0a0 0x558da6bafbd0 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7f43cc059260 tx=0x7f43cc058f40 comp rx=0 tx=0).stop 2024-03-18T20:26:32.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.125+0000 7f43f6174b80 1 -- 172.21.15.1:0/262433569 shutdown_connections 2024-03-18T20:26:32.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.125+0000 7f43f6174b80 1 --2- 172.21.15.1:0/262433569 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6bad630 0x558da6b986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.125+0000 7f43f6174b80 1 --2- 172.21.15.1:0/262433569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6b6d0a0 0x558da6bafbd0 unknown :-1 s=CLOSED pgs=2827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.125+0000 7f43f6174b80 1 --2- 172.21.15.1:0/262433569 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b9bc10 0x558da6b99890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.125+0000 7f43f6174b80 1 -- 172.21.15.1:0/262433569 >> 172.21.15.1:0/262433569 conn(0x558da6acfcf0 msgr2=0x558da6b9c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:32.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.125+0000 7f43f6174b80 1 -- 172.21.15.1:0/262433569 shutdown_connections 2024-03-18T20:26:32.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.125+0000 7f43f6174b80 1 -- 172.21.15.1:0/262433569 wait complete. 2024-03-18T20:26:32.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.125+0000 7f43f6174b80 1 -- start start 2024-03-18T20:26:32.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.126+0000 7f43f6174b80 1 Processor -- start 2024-03-18T20:26:32.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.127+0000 7f43f6174b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6b6d0a0 0x558da6c1d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.127+0000 7f43f6174b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b9bc10 0x558da6b6ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.127+0000 7f43f6174b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6bad630 0x558da6b6b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.127+0000 7f43f6174b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x558da6bac9a0 con 0x558da6bad630 2024-03-18T20:26:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.127+0000 7f43f6174b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x558da6b93350 con 0x558da6b6d0a0 2024-03-18T20:26:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.127+0000 7f43f6174b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x558da6b92c90 con 0x558da6b9bc10 2024-03-18T20:26:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.127+0000 7f43f6174b80 10 client.0 _ll_register_callbacks cb 0x7f43dc001e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:26:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.127+0000 7f43e3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6bad630 0x558da6b6b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.127+0000 7f43e3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6bad630 0x558da6b6b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46698/0 (socket says 172.21.15.1:46698) 2024-03-18T20:26:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.127+0000 7f43e3fff700 1 -- 172.21.15.1:0/499662578 learned_addr learned my addr 172.21.15.1:0/499662578 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.127+0000 7f43e37fe700 1 --2- 172.21.15.1:0/499662578 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6b6d0a0 0x558da6c1d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.127+0000 7f43e2ffd700 1 --2- 172.21.15.1:0/499662578 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b9bc10 0x558da6b6ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.127+0000 7f43e3fff700 1 -- 172.21.15.1:0/499662578 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b9bc10 msgr2=0x558da6b6ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:32.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.127+0000 7f43e3fff700 1 --2- 172.21.15.1:0/499662578 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b9bc10 0x558da6b6ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.127+0000 7f43e3fff700 1 -- 172.21.15.1:0/499662578 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6b6d0a0 msgr2=0x558da6c1d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:32.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.127+0000 7f43e3fff700 1 --2- 172.21.15.1:0/499662578 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6b6d0a0 0x558da6c1d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.127+0000 7f43e3fff700 1 -- 172.21.15.1:0/499662578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558da68b5790 con 0x558da6bad630 2024-03-18T20:26:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.127+0000 7f43e3fff700 1 --2- 172.21.15.1:0/499662578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6bad630 0x558da6b6b050 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0020b0 tx=0x7f43dc015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1ddefabfbd536f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.127+0000 7f43c27fc700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T20:26:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.128+0000 7f43c27fc700 1 -- 172.21.15.1:0/499662578 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc027610 con 0x558da6bad630 2024-03-18T20:26:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.128+0000 7f43c27fc700 1 -- 172.21.15.1:0/499662578 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(253..253 src has 1..253) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f43dc016d00 con 0x558da6bad630 2024-03-18T20:26:32.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.128+0000 7f43f6174b80 1 -- 172.21.15.1:0/499662578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f43cc05d100 con 0x558da6bad630 2024-03-18T20:26:32.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.128+0000 7f43c27fc700 1 -- 172.21.15.1:0/499662578 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=120) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43dc027770 con 0x558da6bad630 2024-03-18T20:26:32.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.128+0000 7f43f6174b80 10 client.12546 fetch_fsmap learned FSMap version 120 2024-03-18T20:26:32.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.128+0000 7f43f6174b80 1 -- 172.21.15.1:0/499662578 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=120,monmap=2+}) v3 -- 0x558da69161d0 con 0x558da6bad630 2024-03-18T20:26:32.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.128+0000 7f43c27fc700 1 -- 172.21.15.1:0/499662578 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43cc05d100 con 0x558da6bad630 2024-03-18T20:26:32.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.128+0000 7f43c27fc700 1 -- 172.21.15.1:0/499662578 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 120) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f43dc0278d0 con 0x558da6bad630 2024-03-18T20:26:32.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.128+0000 7f43f6174b80 10 client.12546 fetch_fsmap finished waiting for FSMap version 120 2024-03-18T20:26:32.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.128+0000 7f43f6174b80 10 client.12546 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-18T20:26:32.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.128+0000 7f43f6174b80 20 client.12546 populate_metadata read hostname 'smithi001' 2024-03-18T20:26:32.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.128+0000 7f43f6174b80 1 --2- 172.21.15.1:0/499662578 >> [v2:172.21.15.165:6833/3886150679,v1:172.21.15.165:6835/3886150679] conn(0x558da6bb66d0 0x558da6bb6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.128+0000 7f43f6174b80 4 client.12546 mds_command: new command op to 12411 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T20:26:32.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.128+0000 7f43f6174b80 1 -- 172.21.15.1:0/499662578 --> [v2:172.21.15.165:6833/3886150679,v1:172.21.15.165:6835/3886150679] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558da68fb040 con 0x558da6bb66d0 2024-03-18T20:26:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.128+0000 7f43e37fe700 1 --2- 172.21.15.1:0/499662578 >> [v2:172.21.15.165:6833/3886150679,v1:172.21.15.165:6835/3886150679] conn(0x558da6bb66d0 0x558da6bb6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.129+0000 7f43c27fc700 10 client.12546 ms_handle_connect on v2:172.21.15.165:6833/3886150679 2024-03-18T20:26:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.129+0000 7f43e37fe700 1 --2- 172.21.15.1:0/499662578 >> [v2:172.21.15.165:6833/3886150679,v1:172.21.15.165:6835/3886150679] conn(0x558da6bb66d0 0x558da6bb6ab0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4212bcaaaff8dec server_cookie=c355cd8c5eff6752 in_seq=0 out_seq=0 2024-03-18T20:26:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.129+0000 7f43c27fc700 1 -- 172.21.15.1:0/499662578 <== mds.0 v2:172.21.15.165:6833/3886150679 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x558da68fb040 con 0x558da6bb66d0 2024-03-18T20:26:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.129+0000 7f43c27fc700 10 client.12546 handle_command_reply: tid=0 2024-03-18T20:26:32.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.129+0000 7f43f6174b80 1 client.12546 shutdown 2024-03-18T20:26:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.129+0000 7f43f6174b80 1 -- 172.21.15.1:0/499662578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6bad630 msgr2=0x558da6b6b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.129+0000 7f43f6174b80 1 --2- 172.21.15.1:0/499662578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6bad630 0x558da6b6b050 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0020b0 tx=0x7f43dc015000 comp rx=0 tx=0).stop 2024-03-18T20:26:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.129+0000 7f43f6174b80 1 -- 172.21.15.1:0/499662578 shutdown_connections 2024-03-18T20:26:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.129+0000 7f43f6174b80 1 --2- 172.21.15.1:0/499662578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558da6bad630 0x558da6b6b050 unknown :-1 s=CLOSED pgs=2828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.129+0000 7f43f6174b80 1 --2- 172.21.15.1:0/499662578 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558da6b6d0a0 0x558da6c1d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.129+0000 7f43f6174b80 1 --2- 172.21.15.1:0/499662578 >> [v2:172.21.15.165:6833/3886150679,v1:172.21.15.165:6835/3886150679] conn(0x558da6bb66d0 0x558da6bb6ab0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.129+0000 7f43f6174b80 1 --2- 172.21.15.1:0/499662578 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x558da6b9bc10 0x558da6b6ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.129+0000 7f43c27fc700 0 client.12546 ms_handle_reset on v2:172.21.15.165:6833/3886150679 2024-03-18T20:26:32.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.129+0000 7f43f6174b80 1 -- 172.21.15.1:0/499662578 >> 172.21.15.1:0/499662578 conn(0x558da6acfcf0 msgr2=0x558da6b9f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:32.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.129+0000 7f43f6174b80 1 -- 172.21.15.1:0/499662578 shutdown_connections 2024-03-18T20:26:32.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.129+0000 7f43f6174b80 1 -- 172.21.15.1:0/499662578 wait complete. 2024-03-18T20:26:32.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.129+0000 7f43f6174b80 20 client.12546 trim_cache size 0 max 16384 2024-03-18T20:26:32.145 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:26:32.145 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "9c1c697f-8655-4ef3-8a13-da568d2cbe86", 2024-03-18T20:26:32.146 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-18T20:26:32.146 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 12411, 2024-03-18T20:26:32.146 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-18T20:26:32.146 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-18T20:26:32.146 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-18T20:26:32.146 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 6.3429494430000002, 2024-03-18T20:26:32.146 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 120, 2024-03-18T20:26:32.147 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 253, 2024-03-18T20:26:32.147 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 253, 2024-03-18T20:26:32.147 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 11.668145213000001 2024-03-18T20:26:32.147 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:26:32.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.130+0000 7f43e9fcb700 1 -- 172.21.15.1:0/2696754737 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f43c406aed0 msgr2=0x7f43c406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:32.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.130+0000 7f43e9fcb700 1 --2- 172.21.15.1:0/2696754737 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f43c406aed0 0x7f43c406d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f43e40c6c90 tx=0x7f43d4008040 comp rx=0 tx=0).stop 2024-03-18T20:26:32.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.130+0000 7f43e9fcb700 1 -- 172.21.15.1:0/2696754737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d0003a00 msgr2=0x7f43e41572a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:32.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.130+0000 7f43e9fcb700 1 --2- 172.21.15.1:0/2696754737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d0003a00 0x7f43e41572a0 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7f43cc001a30 tx=0x7f43cc001b00 comp rx=0 tx=0).stop 2024-03-18T20:26:32.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.130+0000 7f43e9fcb700 1 -- 172.21.15.1:0/2696754737 shutdown_connections 2024-03-18T20:26:32.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.130+0000 7f43e9fcb700 1 --2- 172.21.15.1:0/2696754737 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f43c406aed0 0x7f43c406d390 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.130+0000 7f43e9fcb700 1 --2- 172.21.15.1:0/2696754737 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43e415cb60 0x7f43e40c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.130+0000 7f43e9fcb700 1 --2- 172.21.15.1:0/2696754737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d0003a00 0x7f43e41572a0 unknown :-1 s=CLOSED pgs=2824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.130+0000 7f43e9fcb700 1 --2- 172.21.15.1:0/2696754737 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f43e4155d00 0x7f43e41546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.130+0000 7f43e9fcb700 1 -- 172.21.15.1:0/2696754737 >> 172.21.15.1:0/2696754737 conn(0x7f43e40cc9b0 msgr2=0x7f43e40cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:32.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.130+0000 7f43e9fcb700 1 -- 172.21.15.1:0/2696754737 shutdown_connections 2024-03-18T20:26:32.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.130+0000 7f43e9fcb700 1 -- 172.21.15.1:0/2696754737 wait complete. 2024-03-18T20:26:32.151 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T20:26:32.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.274+0000 7f69749a0700 1 Processor -- start 2024-03-18T20:26:32.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.275+0000 7f69749a0700 1 -- start start 2024-03-18T20:26:32.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.275+0000 7f69749a0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69700b6b60 0x7f69700cc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.275+0000 7f69749a0700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f69700cee90 0x7f69700cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.276+0000 7f69749a0700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69700cd0d0 con 0x7f69700cfbd0 2024-03-18T20:26:32.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.276+0000 7f69749a0700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69700d0060 con 0x7f69700cee90 2024-03-18T20:26:32.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.276+0000 7f69749a0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69700b3f00 con 0x7f69700b6b60 2024-03-18T20:26:32.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.276+0000 7f696e59c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69700b6b60 0x7f69700cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.276+0000 7f696dd9b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f69700cee90 0x7f69700cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.276+0000 7f696e59c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69700b6b60 0x7f69700cc830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:48016/0 (socket says 172.21.15.1:48016) 2024-03-18T20:26:32.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.276+0000 7f696e59c700 1 -- 172.21.15.1:0/3345504156 learned_addr learned my addr 172.21.15.1:0/3345504156 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:32.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.276+0000 7f696d59a700 1 -- 172.21.15.1:0/3345504156 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143671833 0 0) 0x7f69700cd0d0 con 0x7f69700cfbd0 2024-03-18T20:26:32.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.276+0000 7f696e59c700 1 -- 172.21.15.1:0/3345504156 >> v2:172.21.15.165:3300/0 conn(0x7f69700cee90 msgr2=0x7f69700cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:32.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.276+0000 7f696e59c700 1 --2- 172.21.15.1:0/3345504156 >> v2:172.21.15.165:3300/0 conn(0x7f69700cee90 0x7f69700cf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.276+0000 7f696e59c700 1 -- 172.21.15.1:0/3345504156 >> v1:172.21.15.165:6789/0 conn(0x7f69700cfbd0 legacy=0x7f6970099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:32.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.276+0000 7f696e59c700 1 -- 172.21.15.1:0/3345504156 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69700b41e0 con 0x7f69700b6b60 2024-03-18T20:26:32.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.277+0000 7f696e59c700 1 --2- 172.21.15.1:0/3345504156 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69700b6b60 0x7f69700cc830 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f69600098f0 tx=0x7f6960009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a1d71ddddc949ad server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:32.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.277+0000 7f696d59a700 1 -- 172.21.15.1:0/3345504156 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6960011830 con 0x7f69700b6b60 2024-03-18T20:26:32.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.277+0000 7f696d59a700 1 -- 172.21.15.1:0/3345504156 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6960011990 con 0x7f69700b6b60 2024-03-18T20:26:32.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.277+0000 7f696d59a700 1 -- 172.21.15.1:0/3345504156 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6960011d20 con 0x7f69700b6b60 2024-03-18T20:26:32.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.277+0000 7f69749a0700 1 -- 172.21.15.1:0/3345504156 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69700b6b60 msgr2=0x7f69700cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:32.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.277+0000 7f69749a0700 1 --2- 172.21.15.1:0/3345504156 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69700b6b60 0x7f69700cc830 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f69600098f0 tx=0x7f6960009c00 comp rx=0 tx=0).stop 2024-03-18T20:26:32.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.277+0000 7f69749a0700 1 -- 172.21.15.1:0/3345504156 shutdown_connections 2024-03-18T20:26:32.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.277+0000 7f69749a0700 1 --2- 172.21.15.1:0/3345504156 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69700b6b60 0x7f69700cc830 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.277+0000 7f69749a0700 1 --2- 172.21.15.1:0/3345504156 >> v2:172.21.15.165:3300/0 conn(0x7f69700cee90 0x7f69700cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.277+0000 7f69749a0700 1 -- 172.21.15.1:0/3345504156 >> 172.21.15.1:0/3345504156 conn(0x7f69700b2ba0 msgr2=0x7f69700b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:32.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.277+0000 7f69749a0700 1 -- 172.21.15.1:0/3345504156 shutdown_connections 2024-03-18T20:26:32.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.277+0000 7f69749a0700 1 -- 172.21.15.1:0/3345504156 wait complete. 2024-03-18T20:26:32.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.279+0000 7f69749a0700 1 Processor -- start 2024-03-18T20:26:32.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.279+0000 7f69749a0700 1 -- start start 2024-03-18T20:26:32.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.279+0000 7f69749a0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69700b6b60 0x7f6970154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.279+0000 7f69749a0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69700cee90 0x7f69701f44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.279+0000 7f69749a0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69700cfbd0 0x7f6970157d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.279+0000 7f69749a0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69701f5480 con 0x7f69700cfbd0 2024-03-18T20:26:32.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.279+0000 7f69749a0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69701f0ef0 con 0x7f69700cee90 2024-03-18T20:26:32.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.279+0000 7f69749a0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6970158290 con 0x7f69700b6b60 2024-03-18T20:26:32.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.280+0000 7f696ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69700cfbd0 0x7f6970157d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.280+0000 7f696ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69700cfbd0 0x7f6970157d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46700/0 (socket says 172.21.15.1:46700) 2024-03-18T20:26:32.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.280+0000 7f696ed9d700 1 -- 172.21.15.1:0/355560529 learned_addr learned my addr 172.21.15.1:0/355560529 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:32.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.280+0000 7f696e59c700 1 --2- 172.21.15.1:0/355560529 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69700b6b60 0x7f6970154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.280+0000 7f696dd9b700 1 --2- 172.21.15.1:0/355560529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69700cee90 0x7f69701f44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.280+0000 7f696ed9d700 1 -- 172.21.15.1:0/355560529 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69700b6b60 msgr2=0x7f6970154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:32.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.280+0000 7f696ed9d700 1 --2- 172.21.15.1:0/355560529 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69700b6b60 0x7f6970154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.280+0000 7f696ed9d700 1 -- 172.21.15.1:0/355560529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69700cee90 msgr2=0x7f69701f44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.280+0000 7f696ed9d700 1 --2- 172.21.15.1:0/355560529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69700cee90 0x7f69701f44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.280+0000 7f696ed9d700 1 -- 172.21.15.1:0/355560529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6960009550 con 0x7f69700cfbd0 2024-03-18T20:26:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.280+0000 7f696ed9d700 1 --2- 172.21.15.1:0/355560529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69700cfbd0 0x7f6970157d50 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 crypto rx=0x7f696800c7a0 tx=0x7f696800cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.280+0000 7f6966ffd700 1 -- 172.21.15.1:0/355560529 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6968003e30 con 0x7f69700cfbd0 2024-03-18T20:26:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.280+0000 7f6966ffd700 1 -- 172.21.15.1:0/355560529 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6968005070 con 0x7f69700cfbd0 2024-03-18T20:26:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.280+0000 7f6966ffd700 1 -- 172.21.15.1:0/355560529 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6968005420 con 0x7f69700cfbd0 2024-03-18T20:26:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.280+0000 7f69749a0700 1 -- 172.21.15.1:0/355560529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69701f4be0 con 0x7f69700cfbd0 2024-03-18T20:26:32.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.280+0000 7f69749a0700 1 -- 172.21.15.1:0/355560529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69701f51a0 con 0x7f69700cfbd0 2024-03-18T20:26:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.281+0000 7f69749a0700 1 -- 172.21.15.1:0/355560529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69700676a0 con 0x7f69700cfbd0 2024-03-18T20:26:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.282+0000 7f6966ffd700 1 -- 172.21.15.1:0/355560529 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f6968005580 con 0x7f69700cfbd0 2024-03-18T20:26:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.282+0000 7f6966ffd700 1 --2- 172.21.15.1:0/355560529 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f695006aed0 0x7f695006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.282+0000 7f6966ffd700 1 -- 172.21.15.1:0/355560529 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f6968088fc0 con 0x7f69700cfbd0 2024-03-18T20:26:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.282+0000 7f696e59c700 1 --2- 172.21.15.1:0/355560529 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f695006aed0 0x7f695006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.283+0000 7f696e59c700 1 --2- 172.21.15.1:0/355560529 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f695006aed0 0x7f695006d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6960000c00 tx=0x7f6960008030 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:32.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.287+0000 7f6966ffd700 1 -- 172.21.15.1:0/355560529 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f69680bc580 con 0x7f69700cfbd0 2024-03-18T20:26:32.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.541+0000 7f69749a0700 1 -- 172.21.15.1:0/355560529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f69700cb690 con 0x7f69700cfbd0 2024-03-18T20:26:32.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.542+0000 7f6966ffd700 1 -- 172.21.15.1:0/355560529 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v120) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6968017070 con 0x7f69700cfbd0 2024-03-18T20:26:32.546 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T20:26:32.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.544+0000 7f69749a0700 1 -- 172.21.15.1:0/355560529 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f695006aed0 msgr2=0x7f695006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:32.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.544+0000 7f69749a0700 1 --2- 172.21.15.1:0/355560529 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f695006aed0 0x7f695006d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6960000c00 tx=0x7f6960008030 comp rx=0 tx=0).stop 2024-03-18T20:26:32.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.544+0000 7f69749a0700 1 -- 172.21.15.1:0/355560529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69700cfbd0 msgr2=0x7f6970157d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:32.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.544+0000 7f69749a0700 1 --2- 172.21.15.1:0/355560529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69700cfbd0 0x7f6970157d50 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 crypto rx=0x7f696800c7a0 tx=0x7f696800cb60 comp rx=0 tx=0).stop 2024-03-18T20:26:32.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.544+0000 7f69749a0700 1 -- 172.21.15.1:0/355560529 shutdown_connections 2024-03-18T20:26:32.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.544+0000 7f69749a0700 1 --2- 172.21.15.1:0/355560529 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f695006aed0 0x7f695006d390 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.544+0000 7f69749a0700 1 --2- 172.21.15.1:0/355560529 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f69700b6b60 0x7f6970154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.544+0000 7f69749a0700 1 --2- 172.21.15.1:0/355560529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69700cfbd0 0x7f6970157d50 unknown :-1 s=CLOSED pgs=2829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.544+0000 7f69749a0700 1 --2- 172.21.15.1:0/355560529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69700cee90 0x7f69701f44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:32.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.544+0000 7f69749a0700 1 -- 172.21.15.1:0/355560529 >> 172.21.15.1:0/355560529 conn(0x7f69700b2ba0 msgr2=0x7f69700b5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:32.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.544+0000 7f69749a0700 1 -- 172.21.15.1:0/355560529 shutdown_connections 2024-03-18T20:26:32.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:32.544+0000 7f69749a0700 1 -- 172.21.15.1:0/355560529 wait complete. 2024-03-18T20:26:32.560 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T20:26:32.560 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T20:26:32.560 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-18T20:26:32.560 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T20:26:32.560 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T20:26:32.560 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T20:26:32.560 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T20:26:32.560 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T20:26:32.561 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:26:32.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:32.561 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-18T20:26:32.582 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T20:26:32.583 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T20:26:32.583 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T20:26:32.583 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:26:32.583 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T20:26:32.583 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:26:32.583 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:26:32.583 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-18T20:26:32.583 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-18T20:26:32.583 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:26:32.583 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-18T20:26:32.583 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-18T20:26:32.583 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T20:26:32.583 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:26:32.584 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-18T20:26:32.584 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T20:26:32.584 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 36777sec preferred_lft 36777sec 2024-03-18T20:26:32.584 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link 2024-03-18T20:26:32.584 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:26:32.584 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:26:32.584 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:26:32.584 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-18T20:26:32.584 INFO:teuthology.orchestra.run.smithi001.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T20:26:32.584 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:26:32.584 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T20:26:32.584 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:26:32.584 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::4ca9:f5ff:feb3:79a3/64 scope link 2024-03-18T20:26:32.584 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:26:32.584 INFO:teuthology.orchestra.run.smithi001.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T20:26:32.585 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T20:26:32.585 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::282f:a4ff:fe1a:9908/64 scope link 2024-03-18T20:26:32.585 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:26:32.585 INFO:teuthology.orchestra.run.smithi001.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T20:26:32.585 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2e:5c:60:78:b2:8c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T20:26:32.585 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::2c5c:60ff:fe78:b28c/64 scope link 2024-03-18T20:26:32.585 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:26:32.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:32.585 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T20:26:32.649 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-18T20:26:32.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:32.651 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-18T20:26:32.710 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-18T20:26:32.710 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T20:26:32.710 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T20:26:32.710 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T20:26:32.711 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T20:26:32.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:32.712 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T20:26:32.712 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T20:26:32.712 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T20:26:32.712 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T20:26:32.712 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T20:26:32.712 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T20:26:32.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:32.818 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T20:26:32.836 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T20:26:32.836 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T20:26:32.837 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:26:32.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:32.837 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-18T20:26:32.905 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T20:26:32.906 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:26:32.906 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T20:26:32.964 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:26:32.964 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:26:32.964 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T20:26:33.022 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T20:26:33.090 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:26:33.091 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:26:33.158 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:26:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:26:33.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:33.159 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T20:26:33.218 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T20:26:33.219 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T20:26:33.261 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T20:26:33.289 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:26:33.289 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:26:33.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-18T20:26:33.344+0000 7f9e0e17e5c0 -1 init, newargv = 0x55aaf55a9c30 newargc=15 2024-03-18T20:26:33.348 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[164619]: starting ceph client 2024-03-18T20:26:33.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[164619]: starting fuse 2024-03-18T20:26:33.358 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:26:33.359 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:26:33.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:33.359 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T20:26:33.418 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T20:26:33.418 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-03-18T20:26:33.418 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T20:26:33.418 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-18T20:26:33.418 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-18T20:26:33.419 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-18T20:26:33.419 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-18T20:26:33.419 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:26:33.419 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-18T20:26:33.419 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T20:26:33.419 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-18T20:26:33.419 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T20:26:33.419 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:26:33.419 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-18T20:26:33.419 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-18T20:26:33.419 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-18T20:26:33.419 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-18T20:26:33.419 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:26:33.419 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-18T20:26:33.419 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T20:26:33.420 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T20:26:33.420 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T20:26:33.420 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-18T20:26:33.420 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-18T20:26:33.420 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-18T20:26:33.420 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T20:26:33.420 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:26:33.420 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-18T20:26:33.420 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-18T20:26:33.542 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.164619.asok 2024-03-18T20:26:33.724 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.164619.asok 2024-03-18T20:26:33.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:33.725 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.164619.asok status 2024-03-18T20:26:33.889 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:26:33.889 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-18T20:26:33.889 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T20:26:33.889 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T20:26:33.889 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-18T20:26:33.889 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-18T20:26:33.889 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T20:26:33.889 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "164619", 2024-03-18T20:26:33.890 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-18T20:26:33.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:33.890 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-18T20:26:33.890 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-18T20:26:33.890 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 12558, 2024-03-18T20:26:33.890 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-18T20:26:33.890 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-18T20:26:33.890 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-18T20:26:33.890 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 12558 2024-03-18T20:26:33.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:33.891 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T20:26:33.891 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T20:26:33.891 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:26:33.891 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2866828946 2024-03-18T20:26:33.891 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:33.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:33.891 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T20:26:33.891 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T20:26:33.891 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:26:33.891 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2866828946 2024-03-18T20:26:33.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:33.891 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.12558 192.168.0.1:0/2866828946", 2024-03-18T20:26:33.891 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/2866828946", 2024-03-18T20:26:33.891 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-18T20:26:33.891 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 120, 2024-03-18T20:26:33.892 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 253, 2024-03-18T20:26:33.892 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-18T20:26:33.892 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-18T20:26:33.892 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-18T20:26:33.892 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:26:33.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:33.899 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-18T20:26:33.899 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T20:26:33.919 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-18T20:26:33.919 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T20:26:33.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:33.919 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T20:26:33.988 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-03-18T20:26:34.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.154+0000 7f7fdfd2b700 1 Processor -- start 2024-03-18T20:26:34.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.154+0000 7f7fdfd2b700 1 -- start start 2024-03-18T20:26:34.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.155+0000 7f7fdfd2b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd8144b30 0x7f7fd80723f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:34.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.155+0000 7f7fdfd2b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7fd815c050 0x7f7fd80cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:34.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.155+0000 7f7fdfd2b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd80d0ee0 con 0x7f7fd80ccd50 2024-03-18T20:26:34.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.155+0000 7f7fdfd2b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fd80cd710 con 0x7f7fd815c050 2024-03-18T20:26:34.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.155+0000 7f7fdfd2b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd815c8b0 con 0x7f7fd8144b30 2024-03-18T20:26:34.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.155+0000 7f7fde2c8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7fd815c050 0x7f7fd80cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:34.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.155+0000 7f7fdd2c6700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7fd80ccd50 0x7f7fd815b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53538/0 (socket says 172.21.15.1:53538) 2024-03-18T20:26:34.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.155+0000 7f7fdd2c6700 1 -- 172.21.15.1:0/1305714954 learned_addr learned my addr 172.21.15.1:0/1305714954 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:34.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.155+0000 7f7fddac7700 1 --2- 172.21.15.1:0/1305714954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd8144b30 0x7f7fd80723f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:34.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.156+0000 7f7fde2c8700 1 -- 172.21.15.1:0/1305714954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd8144b30 msgr2=0x7f7fd80723f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:34.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.156+0000 7f7fde2c8700 1 --2- 172.21.15.1:0/1305714954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd8144b30 0x7f7fd80723f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:34.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.156+0000 7f7fde2c8700 1 -- 172.21.15.1:0/1305714954 >> v1:172.21.15.1:6789/0 conn(0x7f7fd80ccd50 legacy=0x7f7fd815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.156+0000 7f7fde2c8700 1 -- 172.21.15.1:0/1305714954 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd815c5d0 con 0x7f7fd815c050 2024-03-18T20:26:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.156+0000 7f7fdcac5700 1 -- 172.21.15.1:0/1305714954 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176440386 0 0) 0x7f7fd80d0ee0 con 0x7f7fd80ccd50 2024-03-18T20:26:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.156+0000 7f7fde2c8700 1 --2- 172.21.15.1:0/1305714954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd815c050 0x7f7fd80cd1d0 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0009d90 tx=0x7f7fd000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad380823be9c0a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.156+0000 7f7fdcac5700 1 -- 172.21.15.1:0/1305714954 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd000e070 con 0x7f7fd815c050 2024-03-18T20:26:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.156+0000 7f7fdcac5700 1 -- 172.21.15.1:0/1305714954 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd000bbd0 con 0x7f7fd815c050 2024-03-18T20:26:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.156+0000 7f7fdcac5700 1 -- 172.21.15.1:0/1305714954 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0012430 con 0x7f7fd815c050 2024-03-18T20:26:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.156+0000 7f7fdfd2b700 1 -- 172.21.15.1:0/1305714954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd815c050 msgr2=0x7f7fd80cd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.156+0000 7f7fdfd2b700 1 --2- 172.21.15.1:0/1305714954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd815c050 0x7f7fd80cd1d0 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0009d90 tx=0x7f7fd000b330 comp rx=0 tx=0).stop 2024-03-18T20:26:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.156+0000 7f7fdfd2b700 1 -- 172.21.15.1:0/1305714954 shutdown_connections 2024-03-18T20:26:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.156+0000 7f7fdfd2b700 1 --2- 172.21.15.1:0/1305714954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd815c050 0x7f7fd80cd1d0 unknown :-1 s=CLOSED pgs=2834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.156+0000 7f7fdfd2b700 1 --2- 172.21.15.1:0/1305714954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd8144b30 0x7f7fd80723f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:34.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.156+0000 7f7fdfd2b700 1 -- 172.21.15.1:0/1305714954 >> 172.21.15.1:0/1305714954 conn(0x7f7fd80d00c0 msgr2=0x7f7fd80d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:34.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.157+0000 7f7fdfd2b700 1 -- 172.21.15.1:0/1305714954 shutdown_connections 2024-03-18T20:26:34.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.157+0000 7f7fdfd2b700 1 -- 172.21.15.1:0/1305714954 wait complete. 2024-03-18T20:26:34.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.158+0000 7f7fdfd2b700 1 Processor -- start 2024-03-18T20:26:34.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.158+0000 7f7fdfd2b700 1 -- start start 2024-03-18T20:26:34.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.158+0000 7f7fdfd2b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd80ccd50 0x7f7fd80af5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:34.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fdfd2b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fd8144b30 0x7f7fd81359b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:34.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fdfd2b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd815c050 0x7f7fd8138250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:34.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fdfd2b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd80afb10 con 0x7f7fd815c050 2024-03-18T20:26:34.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fdfd2b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd8136b40 con 0x7f7fd8144b30 2024-03-18T20:26:34.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fdfd2b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd8135ef0 con 0x7f7fd80ccd50 2024-03-18T20:26:34.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fde2c8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd815c050 0x7f7fd8138250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:34.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fde2c8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd815c050 0x7f7fd8138250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46716/0 (socket says 172.21.15.1:46716) 2024-03-18T20:26:34.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fde2c8700 1 -- 172.21.15.1:0/430322321 learned_addr learned my addr 172.21.15.1:0/430322321 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:34.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fddac7700 1 --2- 172.21.15.1:0/430322321 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd80ccd50 0x7f7fd80af5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:34.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fdd2c6700 1 --2- 172.21.15.1:0/430322321 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fd8144b30 0x7f7fd81359b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:34.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fde2c8700 1 -- 172.21.15.1:0/430322321 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd80ccd50 msgr2=0x7f7fd80af5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:34.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fde2c8700 1 --2- 172.21.15.1:0/430322321 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd80ccd50 0x7f7fd80af5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:34.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fde2c8700 1 -- 172.21.15.1:0/430322321 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fd8144b30 msgr2=0x7f7fd81359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:34.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fde2c8700 1 --2- 172.21.15.1:0/430322321 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fd8144b30 0x7f7fd81359b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:34.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fde2c8700 1 -- 172.21.15.1:0/430322321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd0009a70 con 0x7f7fd815c050 2024-03-18T20:26:34.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fde2c8700 1 --2- 172.21.15.1:0/430322321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd815c050 0x7f7fd8138250 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0014040 tx=0x7f7fd0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:34.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fce7fc700 1 -- 172.21.15.1:0/430322321 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0012430 con 0x7f7fd815c050 2024-03-18T20:26:34.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fce7fc700 1 -- 172.21.15.1:0/430322321 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd0020070 con 0x7f7fd815c050 2024-03-18T20:26:34.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fdfd2b700 1 -- 172.21.15.1:0/430322321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd80c8400 con 0x7f7fd815c050 2024-03-18T20:26:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fce7fc700 1 -- 172.21.15.1:0/430322321 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0004280 con 0x7f7fd815c050 2024-03-18T20:26:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.159+0000 7f7fdfd2b700 1 -- 172.21.15.1:0/430322321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd81364f0 con 0x7f7fd815c050 2024-03-18T20:26:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.160+0000 7f7fdfd2b700 1 -- 172.21.15.1:0/430322321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd8136cb0 con 0x7f7fd815c050 2024-03-18T20:26:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.161+0000 7f7fce7fc700 1 -- 172.21.15.1:0/430322321 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f7fd00043e0 con 0x7f7fd815c050 2024-03-18T20:26:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.161+0000 7f7fce7fc700 1 --2- 172.21.15.1:0/430322321 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fbc06aed0 0x7f7fbc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.161+0000 7f7fce7fc700 1 -- 172.21.15.1:0/430322321 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f7fd0022070 con 0x7f7fd815c050 2024-03-18T20:26:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.161+0000 7f7fddac7700 1 --2- 172.21.15.1:0/430322321 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fbc06aed0 0x7f7fbc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:34.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.162+0000 7f7fddac7700 1 --2- 172.21.15.1:0/430322321 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fbc06aed0 0x7f7fbc06d390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8006b20 tx=0x7f7fc8008040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:34.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.166+0000 7f7fce7fc700 1 -- 172.21.15.1:0/430322321 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7fd0056670 con 0x7f7fd815c050 2024-03-18T20:26:34.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:34.416+0000 7f7fdfd2b700 1 -- 172.21.15.1:0/430322321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7fd80676a0 con 0x7f7fd815c050 2024-03-18T20:26:35.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.137+0000 7f7fce7fc700 1 -- 172.21.15.1:0/430322321 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v121) v1 ==== 136+0+0 (secure 0 0 0) 0x7f7fd0059c90 con 0x7f7fd815c050 2024-03-18T20:26:35.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.140+0000 7f7fdfd2b700 1 -- 172.21.15.1:0/430322321 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fbc06aed0 msgr2=0x7f7fbc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:35.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.140+0000 7f7fdfd2b700 1 --2- 172.21.15.1:0/430322321 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fbc06aed0 0x7f7fbc06d390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8006b20 tx=0x7f7fc8008040 comp rx=0 tx=0).stop 2024-03-18T20:26:35.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.140+0000 7f7fdfd2b700 1 -- 172.21.15.1:0/430322321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd815c050 msgr2=0x7f7fd8138250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:35.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.140+0000 7f7fdfd2b700 1 --2- 172.21.15.1:0/430322321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd815c050 0x7f7fd8138250 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0014040 tx=0x7f7fd0004ad0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.140+0000 7f7fdfd2b700 1 -- 172.21.15.1:0/430322321 shutdown_connections 2024-03-18T20:26:35.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.140+0000 7f7fdfd2b700 1 --2- 172.21.15.1:0/430322321 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fbc06aed0 0x7f7fbc06d390 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.140+0000 7f7fdfd2b700 1 --2- 172.21.15.1:0/430322321 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fd80ccd50 0x7f7fd80af5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.140+0000 7f7fdfd2b700 1 --2- 172.21.15.1:0/430322321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fd815c050 0x7f7fd8138250 unknown :-1 s=CLOSED pgs=2835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.141+0000 7f7fdfd2b700 1 --2- 172.21.15.1:0/430322321 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fd8144b30 0x7f7fd81359b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.141+0000 7f7fdfd2b700 1 -- 172.21.15.1:0/430322321 >> 172.21.15.1:0/430322321 conn(0x7f7fd80d00c0 msgr2=0x7f7fd806f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:35.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.141+0000 7f7fdfd2b700 1 -- 172.21.15.1:0/430322321 shutdown_connections 2024-03-18T20:26:35.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.141+0000 7f7fdfd2b700 1 -- 172.21.15.1:0/430322321 wait complete. 2024-03-18T20:26:35.159 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:26:35.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.287+0000 7f54e24e6700 1 Processor -- start 2024-03-18T20:26:35.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.288+0000 7f54e24e6700 1 -- start start 2024-03-18T20:26:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.288+0000 7f54e24e6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f54dc06dea0 0x7f54dc168450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.288+0000 7f54e24e6700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0cd5a0 con 0x7f54dc06e8f0 2024-03-18T20:26:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.288+0000 7f54e24e6700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc071b50 con 0x7f54dc15d620 2024-03-18T20:26:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.288+0000 7f54e24e6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc15da00 con 0x7f54dc06dea0 2024-03-18T20:26:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.288+0000 7f54db7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f54dc06e8f0 0x7f54dc071440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56294/0 (socket says 172.21.15.1:56294) 2024-03-18T20:26:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.288+0000 7f54db7fe700 1 -- 172.21.15.1:0/3732743459 learned_addr learned my addr 172.21.15.1:0/3732743459 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.289+0000 7f54e0a83700 1 --2- 172.21.15.1:0/3732743459 >> v2:172.21.15.165:3300/0 conn(0x7f54dc06dea0 0x7f54dc168450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.289+0000 7f54daffd700 1 -- 172.21.15.1:0/3732743459 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265409448 0 0) 0x7f54dc0cd5a0 con 0x7f54dc06e8f0 2024-03-18T20:26:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.289+0000 7f54daffd700 1 -- 172.21.15.1:0/3732743459 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c80035b0 con 0x7f54dc06e8f0 2024-03-18T20:26:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.289+0000 7f54daffd700 1 -- 172.21.15.1:0/3732743459 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2300370020 0 0) 0x7f54c80035b0 con 0x7f54dc06e8f0 2024-03-18T20:26:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.289+0000 7f54daffd700 1 -- 172.21.15.1:0/3732743459 >> v2:172.21.15.165:3300/0 conn(0x7f54dc06dea0 msgr2=0x7f54dc168450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.289+0000 7f54daffd700 1 --2- 172.21.15.1:0/3732743459 >> v2:172.21.15.165:3300/0 conn(0x7f54dc06dea0 0x7f54dc168450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.289+0000 7f54daffd700 1 -- 172.21.15.1:0/3732743459 >> v1:172.21.15.165:6789/0 conn(0x7f54dc15d620 legacy=0x7f54dc070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.289+0000 7f54daffd700 1 -- 172.21.15.1:0/3732743459 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc15e5f0 con 0x7f54dc06e8f0 2024-03-18T20:26:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.289+0000 7f54daffd700 1 -- 172.21.15.1:0/3732743459 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f54cc001670 con 0x7f54dc06e8f0 2024-03-18T20:26:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.289+0000 7f54daffd700 1 -- 172.21.15.1:0/3732743459 >> v1:172.21.15.1:6789/0 conn(0x7f54dc06e8f0 legacy=0x7f54dc071440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.289+0000 7f54daffd700 1 --2- 172.21.15.1:0/3732743459 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54c8003a50 0x7f54c8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.289+0000 7f54daffd700 1 --2- 172.21.15.1:0/3732743459 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54c8004850 0x7f54c8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.289+0000 7f54daffd700 1 --2- 172.21.15.1:0/3732743459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54c8005260 0x7f54c800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.289+0000 7f54daffd700 1 -- 172.21.15.1:0/3732743459 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc15da00 con 0x7f54c8005260 2024-03-18T20:26:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.289+0000 7f54daffd700 1 -- 172.21.15.1:0/3732743459 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c800c1b0 con 0x7f54c8004850 2024-03-18T20:26:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.289+0000 7f54daffd700 1 -- 172.21.15.1:0/3732743459 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c800c370 con 0x7f54c8003a50 2024-03-18T20:26:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.289+0000 7f54e0a83700 1 --2- 172.21.15.1:0/3732743459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54c8005260 0x7f54c800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.289+0000 7f54db7fe700 1 --2- 172.21.15.1:0/3732743459 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54c8004850 0x7f54c8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.289+0000 7f54dbfff700 1 --2- 172.21.15.1:0/3732743459 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54c8003a50 0x7f54c8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.290+0000 7f54e0a83700 1 -- 172.21.15.1:0/3732743459 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54c8003a50 msgr2=0x7f54c8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.290+0000 7f54e0a83700 1 --2- 172.21.15.1:0/3732743459 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54c8003a50 0x7f54c8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.290+0000 7f54e0a83700 1 -- 172.21.15.1:0/3732743459 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54c8004850 msgr2=0x7f54c8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:35.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.290+0000 7f54e0a83700 1 --2- 172.21.15.1:0/3732743459 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54c8004850 0x7f54c8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.290+0000 7f54e0a83700 1 -- 172.21.15.1:0/3732743459 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c80035b0 con 0x7f54c8005260 2024-03-18T20:26:35.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.290+0000 7f54e0a83700 1 -- 172.21.15.1:0/3732743459 reap_dead start 2024-03-18T20:26:35.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.290+0000 7f54e0a83700 1 --2- 172.21.15.1:0/3732743459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54c8005260 0x7f54c800ba10 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f54d4009f90 tx=0x7f54c8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fe7cba5b638ed7e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:35.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.290+0000 7f54daffd700 1 -- 172.21.15.1:0/3732743459 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4003a20 con 0x7f54c8005260 2024-03-18T20:26:35.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.290+0000 7f54daffd700 1 -- 172.21.15.1:0/3732743459 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d400f040 con 0x7f54c8005260 2024-03-18T20:26:35.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.290+0000 7f54daffd700 1 -- 172.21.15.1:0/3732743459 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4003de0 con 0x7f54c8005260 2024-03-18T20:26:35.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.290+0000 7f54e24e6700 1 -- 172.21.15.1:0/3732743459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54c8005260 msgr2=0x7f54c800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:35.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.290+0000 7f54e24e6700 1 --2- 172.21.15.1:0/3732743459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54c8005260 0x7f54c800ba10 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f54d4009f90 tx=0x7f54c8005880 comp rx=0 tx=0).stop 2024-03-18T20:26:35.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.290+0000 7f54e24e6700 1 -- 172.21.15.1:0/3732743459 shutdown_connections 2024-03-18T20:26:35.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.290+0000 7f54e24e6700 1 --2- 172.21.15.1:0/3732743459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54c8005260 0x7f54c800ba10 unknown :-1 s=CLOSED pgs=2837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.290+0000 7f54e24e6700 1 -- 172.21.15.1:0/3732743459 >> 172.21.15.1:0/3732743459 conn(0x7f54dc0b1270 msgr2=0x7f54dc0b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:35.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.290+0000 7f54e24e6700 1 -- 172.21.15.1:0/3732743459 shutdown_connections 2024-03-18T20:26:35.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.291+0000 7f54e24e6700 1 -- 172.21.15.1:0/3732743459 wait complete. 2024-03-18T20:26:35.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.292+0000 7f54e24e6700 1 Processor -- start 2024-03-18T20:26:35.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.292+0000 7f54e24e6700 1 -- start start 2024-03-18T20:26:35.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54e24e6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54c8005260 0x7f54dc158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:35.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54e24e6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54dc1e8380 0x7f54dc15a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:35.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54e24e6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54dc15a940 0x7f54dc15aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:35.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54e24e6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc159210 con 0x7f54dc15a940 2024-03-18T20:26:35.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54e24e6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc1e72e0 con 0x7f54dc1e8380 2024-03-18T20:26:35.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54e24e6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc1e75c0 con 0x7f54c8005260 2024-03-18T20:26:35.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54e0a83700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54dc15a940 0x7f54dc15aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:35.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54e0a83700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54dc15a940 0x7f54dc15aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35690/0 (socket says 172.21.15.1:35690) 2024-03-18T20:26:35.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54e0a83700 1 -- 172.21.15.1:0/3741521763 learned_addr learned my addr 172.21.15.1:0/3741521763 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:35.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54e0a83700 1 -- 172.21.15.1:0/3741521763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54c8005260 msgr2=0x7f54dc158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:35.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54e0a83700 1 --2- 172.21.15.1:0/3741521763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54c8005260 0x7f54dc158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54e0a83700 1 -- 172.21.15.1:0/3741521763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54dc1e8380 msgr2=0x7f54dc15a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:35.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54db7fe700 1 --2- 172.21.15.1:0/3741521763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54dc1e8380 0x7f54dc15a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:35.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54e0a83700 1 --2- 172.21.15.1:0/3741521763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54dc1e8380 0x7f54dc15a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54e0a83700 1 -- 172.21.15.1:0/3741521763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc1e78a0 con 0x7f54dc15a940 2024-03-18T20:26:35.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54db7fe700 1 --2- 172.21.15.1:0/3741521763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54dc1e8380 0x7f54dc15a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:35.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54e0a83700 1 --2- 172.21.15.1:0/3741521763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54dc15a940 0x7f54dc15aea0 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 crypto rx=0x7f54dc0ce750 tx=0x7f54d4004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:35.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54d8ff9700 1 -- 172.21.15.1:0/3741521763 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4004230 con 0x7f54dc15a940 2024-03-18T20:26:35.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54d8ff9700 1 -- 172.21.15.1:0/3741521763 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d4004390 con 0x7f54dc15a940 2024-03-18T20:26:35.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54d8ff9700 1 -- 172.21.15.1:0/3741521763 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4004740 con 0x7f54dc15a940 2024-03-18T20:26:35.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54e24e6700 1 -- 172.21.15.1:0/3741521763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54dc1e7b80 con 0x7f54dc15a940 2024-03-18T20:26:35.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.293+0000 7f54e24e6700 1 -- 172.21.15.1:0/3741521763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54dc1e8800 con 0x7f54dc15a940 2024-03-18T20:26:35.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.294+0000 7f54e24e6700 1 -- 172.21.15.1:0/3741521763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54dc1e80a0 con 0x7f54dc15a940 2024-03-18T20:26:35.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.295+0000 7f54d8ff9700 1 -- 172.21.15.1:0/3741521763 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f54d4014480 con 0x7f54dc15a940 2024-03-18T20:26:35.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.295+0000 7f54d8ff9700 1 --2- 172.21.15.1:0/3741521763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f54bc06aed0 0x7f54bc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:35.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.296+0000 7f54dbfff700 1 --2- 172.21.15.1:0/3741521763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f54bc06aed0 0x7f54bc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:35.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.296+0000 7f54d8ff9700 1 -- 172.21.15.1:0/3741521763 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f54d408acf0 con 0x7f54dc15a940 2024-03-18T20:26:35.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.296+0000 7f54dbfff700 1 --2- 172.21.15.1:0/3741521763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f54bc06aed0 0x7f54bc06d390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f54c4000c00 tx=0x7f54c4009040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:35.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.300+0000 7f54d8ff9700 1 -- 172.21.15.1:0/3741521763 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f54dc1e80a0 con 0x7f54dc15a940 2024-03-18T20:26:35.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.503+0000 7f54e24e6700 1 -- 172.21.15.1:0/3741521763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54dc1e80a0 con 0x7f54dc15a940 2024-03-18T20:26:35.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.504+0000 7f54d8ff9700 1 -- 172.21.15.1:0/3741521763 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v253) v1 ==== 74+0+16429 (secure 0 0 0) 0x7f54d4018090 con 0x7f54dc15a940 2024-03-18T20:26:35.508 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:35.508 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":253,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:26:26.786793+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:26:23.615691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-18T20:26:24.759650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:26:35.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.506+0000 7f54e24e6700 1 -- 172.21.15.1:0/3741521763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f54bc06aed0 msgr2=0x7f54bc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.506+0000 7f54e24e6700 1 --2- 172.21.15.1:0/3741521763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f54bc06aed0 0x7f54bc06d390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f54c4000c00 tx=0x7f54c4009040 comp rx=0 tx=0).stop 2024-03-18T20:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.506+0000 7f54e24e6700 1 -- 172.21.15.1:0/3741521763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54dc15a940 msgr2=0x7f54dc15aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.506+0000 7f54e24e6700 1 --2- 172.21.15.1:0/3741521763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54dc15a940 0x7f54dc15aea0 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 crypto rx=0x7f54dc0ce750 tx=0x7f54d4004b40 comp rx=0 tx=0).stop 2024-03-18T20:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.506+0000 7f54e24e6700 1 -- 172.21.15.1:0/3741521763 shutdown_connections 2024-03-18T20:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.506+0000 7f54e24e6700 1 --2- 172.21.15.1:0/3741521763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f54bc06aed0 0x7f54bc06d390 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.506+0000 7f54e24e6700 1 --2- 172.21.15.1:0/3741521763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54c8005260 0x7f54dc158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.506+0000 7f54e24e6700 1 --2- 172.21.15.1:0/3741521763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54dc15a940 0x7f54dc15aea0 unknown :-1 s=CLOSED pgs=2838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.506+0000 7f54e24e6700 1 --2- 172.21.15.1:0/3741521763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54dc1e8380 0x7f54dc15a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.506+0000 7f54e24e6700 1 -- 172.21.15.1:0/3741521763 >> 172.21.15.1:0/3741521763 conn(0x7f54dc0b1270 msgr2=0x7f54dc0af0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.506+0000 7f54e24e6700 1 -- 172.21.15.1:0/3741521763 shutdown_connections 2024-03-18T20:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.506+0000 7f54e24e6700 1 -- 172.21.15.1:0/3741521763 wait complete. 2024-03-18T20:26:35.522 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:26:23.615691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '252', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:26:24.759650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '253', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:26:35.523 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:26:35.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.646+0000 7f26ae139700 1 Processor -- start 2024-03-18T20:26:35.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.646+0000 7f26ae139700 1 -- start start 2024-03-18T20:26:35.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.647+0000 7f26ae139700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26a806efb0 0x7f26a8071da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:35.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.647+0000 7f26ae139700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f26a80722e0 0x7f26a80cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:35.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.647+0000 7f26ae139700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a806da90 con 0x7f26a815d580 2024-03-18T20:26:35.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.647+0000 7f26ae139700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26a8072770 con 0x7f26a806efb0 2024-03-18T20:26:35.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.647+0000 7f26ae139700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26a815d960 con 0x7f26a80722e0 2024-03-18T20:26:35.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.647+0000 7f26a6ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26a806efb0 0x7f26a8071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:35.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.647+0000 7f26a6ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26a806efb0 0x7f26a8071da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35704/0 (socket says 172.21.15.1:35704) 2024-03-18T20:26:35.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.647+0000 7f26a6ffd700 1 -- 172.21.15.1:0/4022750516 learned_addr learned my addr 172.21.15.1:0/4022750516 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:35.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.647+0000 7f26a7fff700 1 --2- 172.21.15.1:0/4022750516 >> v2:172.21.15.165:3300/0 conn(0x7f26a80722e0 0x7f26a80cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:35.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.647+0000 7f26a6ffd700 1 -- 172.21.15.1:0/4022750516 >> v2:172.21.15.165:3300/0 conn(0x7f26a80722e0 msgr2=0x7f26a80cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:35.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.647+0000 7f26a6ffd700 1 --2- 172.21.15.1:0/4022750516 >> v2:172.21.15.165:3300/0 conn(0x7f26a80722e0 0x7f26a80cf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.647+0000 7f26a6ffd700 1 -- 172.21.15.1:0/4022750516 >> v1:172.21.15.165:6789/0 conn(0x7f26a815d580 legacy=0x7f26a806d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:35.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.647+0000 7f26a6ffd700 1 -- 172.21.15.1:0/4022750516 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a806ab30 con 0x7f26a806efb0 2024-03-18T20:26:35.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.647+0000 7f26a7fff700 1 --2- 172.21.15.1:0/4022750516 >> v2:172.21.15.165:3300/0 conn(0x7f26a80722e0 0x7f26a80cf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:35.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.647+0000 7f26a6ffd700 1 --2- 172.21.15.1:0/4022750516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a806efb0 0x7f26a8071da0 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f2690009920 tx=0x7f2690009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3493cb68d405978 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:35.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.648+0000 7f26a67fc700 1 -- 172.21.15.1:0/4022750516 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690004230 con 0x7f26a806efb0 2024-03-18T20:26:35.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.648+0000 7f26a67fc700 1 -- 172.21.15.1:0/4022750516 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2690004390 con 0x7f26a806efb0 2024-03-18T20:26:35.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.648+0000 7f26a67fc700 1 -- 172.21.15.1:0/4022750516 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690009e70 con 0x7f26a806efb0 2024-03-18T20:26:35.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.648+0000 7f26ae139700 1 -- 172.21.15.1:0/4022750516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a806efb0 msgr2=0x7f26a8071da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:35.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.648+0000 7f26ae139700 1 --2- 172.21.15.1:0/4022750516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a806efb0 0x7f26a8071da0 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f2690009920 tx=0x7f2690009c30 comp rx=0 tx=0).stop 2024-03-18T20:26:35.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.648+0000 7f26ae139700 1 -- 172.21.15.1:0/4022750516 shutdown_connections 2024-03-18T20:26:35.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.648+0000 7f26ae139700 1 --2- 172.21.15.1:0/4022750516 >> v2:172.21.15.165:3300/0 conn(0x7f26a80722e0 0x7f26a80cf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.648+0000 7f26ae139700 1 --2- 172.21.15.1:0/4022750516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a806efb0 0x7f26a8071da0 unknown :-1 s=CLOSED pgs=2839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.648+0000 7f26ae139700 1 -- 172.21.15.1:0/4022750516 >> 172.21.15.1:0/4022750516 conn(0x7f26a80cca00 msgr2=0x7f26a80cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:35.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.648+0000 7f26ae139700 1 -- 172.21.15.1:0/4022750516 shutdown_connections 2024-03-18T20:26:35.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.648+0000 7f26ae139700 1 -- 172.21.15.1:0/4022750516 wait complete. 2024-03-18T20:26:35.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.649+0000 7f26ae139700 1 Processor -- start 2024-03-18T20:26:35.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.650+0000 7f26ae139700 1 -- start start 2024-03-18T20:26:35.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.650+0000 7f26ae139700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26a806efb0 0x7f26a81f79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:35.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.650+0000 7f26ae139700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f26a80722e0 0x7f26a81f3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:35.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.650+0000 7f26ae139700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a815d580 0x7f26a815a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:35.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.650+0000 7f26ae139700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a81f4920 con 0x7f26a815d580 2024-03-18T20:26:35.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.650+0000 7f26ae139700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a81f7f20 con 0x7f26a806efb0 2024-03-18T20:26:35.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.650+0000 7f26ae139700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a81f3ca0 con 0x7f26a80722e0 2024-03-18T20:26:35.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.650+0000 7f26a7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a815d580 0x7f26a815a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:35.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.650+0000 7f26a7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a815d580 0x7f26a815a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35712/0 (socket says 172.21.15.1:35712) 2024-03-18T20:26:35.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.650+0000 7f26a7fff700 1 -- 172.21.15.1:0/946519793 learned_addr learned my addr 172.21.15.1:0/946519793 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:35.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.651+0000 7f26a77fe700 1 --2- 172.21.15.1:0/946519793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26a806efb0 0x7f26a81f79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:35.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.651+0000 7f26a6ffd700 1 --2- 172.21.15.1:0/946519793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f26a80722e0 0x7f26a81f3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:35.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.651+0000 7f26a7fff700 1 -- 172.21.15.1:0/946519793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f26a80722e0 msgr2=0x7f26a81f3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:35.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.651+0000 7f26a7fff700 1 --2- 172.21.15.1:0/946519793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f26a80722e0 0x7f26a81f3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.651+0000 7f26a7fff700 1 -- 172.21.15.1:0/946519793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26a806efb0 msgr2=0x7f26a81f79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:35.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.651+0000 7f26a7fff700 1 --2- 172.21.15.1:0/946519793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26a806efb0 0x7f26a81f79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.651+0000 7f26a7fff700 1 -- 172.21.15.1:0/946519793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2690009550 con 0x7f26a815d580 2024-03-18T20:26:35.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.651+0000 7f26a7fff700 1 --2- 172.21.15.1:0/946519793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a815d580 0x7f26a815a190 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f26a000cd50 tx=0x7f26a000a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:35.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.651+0000 7f269ffff700 1 -- 172.21.15.1:0/946519793 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a000f480 con 0x7f26a815d580 2024-03-18T20:26:35.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.651+0000 7f269ffff700 1 -- 172.21.15.1:0/946519793 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a000f600 con 0x7f26a815d580 2024-03-18T20:26:35.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.651+0000 7f269ffff700 1 -- 172.21.15.1:0/946519793 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a000f9b0 con 0x7f26a815d580 2024-03-18T20:26:35.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.651+0000 7f26ae139700 1 -- 172.21.15.1:0/946519793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a81f3f80 con 0x7f26a815d580 2024-03-18T20:26:35.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.651+0000 7f26ae139700 1 -- 172.21.15.1:0/946519793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a81f4540 con 0x7f26a815d580 2024-03-18T20:26:35.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.652+0000 7f26ae139700 1 -- 172.21.15.1:0/946519793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a80ce320 con 0x7f26a815d580 2024-03-18T20:26:35.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.653+0000 7f269ffff700 1 -- 172.21.15.1:0/946519793 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f26a000b040 con 0x7f26a815d580 2024-03-18T20:26:35.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.653+0000 7f269ffff700 1 --2- 172.21.15.1:0/946519793 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f268806aed0 0x7f268806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:35.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.653+0000 7f269ffff700 1 -- 172.21.15.1:0/946519793 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f26a008a220 con 0x7f26a815d580 2024-03-18T20:26:35.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.653+0000 7f26a77fe700 1 --2- 172.21.15.1:0/946519793 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f268806aed0 0x7f268806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:35.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.653+0000 7f26a77fe700 1 --2- 172.21.15.1:0/946519793 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f268806aed0 0x7f268806d390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2698001670 tx=0x7f2698009040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:35.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.658+0000 7f269ffff700 1 -- 172.21.15.1:0/946519793 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f26a0054de0 con 0x7f26a815d580 2024-03-18T20:26:35.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.858+0000 7f26ae139700 1 -- 172.21.15.1:0/946519793 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f26a80b16c0 con 0x7f26a815d580 2024-03-18T20:26:35.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.858+0000 7f269ffff700 1 -- 172.21.15.1:0/946519793 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v253) v1 ==== 93+0+10 (secure 0 0 0) 0x7f26a001d020 con 0x7f26a815d580 2024-03-18T20:26:35.863 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:26:35.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.861+0000 7f26ae139700 1 -- 172.21.15.1:0/946519793 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f268806aed0 msgr2=0x7f268806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:35.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.861+0000 7f26ae139700 1 --2- 172.21.15.1:0/946519793 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f268806aed0 0x7f268806d390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2698001670 tx=0x7f2698009040 comp rx=0 tx=0).stop 2024-03-18T20:26:35.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.861+0000 7f26ae139700 1 -- 172.21.15.1:0/946519793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a815d580 msgr2=0x7f26a815a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:35.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.861+0000 7f26ae139700 1 --2- 172.21.15.1:0/946519793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a815d580 0x7f26a815a190 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f26a000cd50 tx=0x7f26a000a560 comp rx=0 tx=0).stop 2024-03-18T20:26:35.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.861+0000 7f26ae139700 1 -- 172.21.15.1:0/946519793 shutdown_connections 2024-03-18T20:26:35.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.861+0000 7f26ae139700 1 --2- 172.21.15.1:0/946519793 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f268806aed0 0x7f268806d390 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.861+0000 7f26ae139700 1 --2- 172.21.15.1:0/946519793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a815d580 0x7f26a815a190 unknown :-1 s=CLOSED pgs=2840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.861+0000 7f26ae139700 1 --2- 172.21.15.1:0/946519793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f26a806efb0 0x7f26a81f79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.861+0000 7f26ae139700 1 --2- 172.21.15.1:0/946519793 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f26a80722e0 0x7f26a81f3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:35.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.861+0000 7f26ae139700 1 -- 172.21.15.1:0/946519793 >> 172.21.15.1:0/946519793 conn(0x7f26a80cca00 msgr2=0x7f26a815c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:35.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.861+0000 7f26ae139700 1 -- 172.21.15.1:0/946519793 shutdown_connections 2024-03-18T20:26:35.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:35.861+0000 7f26ae139700 1 -- 172.21.15.1:0/946519793 wait complete. 2024-03-18T20:26:35.878 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:26:36.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.001+0000 7fd5acf91700 1 Processor -- start 2024-03-18T20:26:36.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.001+0000 7fd5acf91700 1 -- start start 2024-03-18T20:26:36.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.002+0000 7fd5acf91700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd5a80c3670 0x7fd5a815e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.002+0000 7fd5acf91700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd5a815e660 0x7fd5a80cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.002+0000 7fd5acf91700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a80cf730 con 0x7fd5a80cbd40 2024-03-18T20:26:36.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.002+0000 7fd5acf91700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a80cf980 con 0x7fd5a80c3670 2024-03-18T20:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.002+0000 7fd5acf91700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a80cfc60 con 0x7fd5a815e660 2024-03-18T20:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.003+0000 7fd5a5d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd5a80c3670 0x7fd5a815e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.003+0000 7fd5a5d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd5a80c3670 0x7fd5a815e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35722/0 (socket says 172.21.15.1:35722) 2024-03-18T20:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.003+0000 7fd5a5d9b700 1 -- 172.21.15.1:0/293771608 learned_addr learned my addr 172.21.15.1:0/293771608 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.003+0000 7fd5a6d9d700 1 --2- 172.21.15.1:0/293771608 >> v2:172.21.15.165:3300/0 conn(0x7fd5a815e660 0x7fd5a80cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.003+0000 7fd5a5d9b700 1 -- 172.21.15.1:0/293771608 >> v2:172.21.15.165:3300/0 conn(0x7fd5a815e660 msgr2=0x7fd5a80cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.003+0000 7fd5a5d9b700 1 --2- 172.21.15.1:0/293771608 >> v2:172.21.15.165:3300/0 conn(0x7fd5a815e660 0x7fd5a80cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.003+0000 7fd5a5d9b700 1 -- 172.21.15.1:0/293771608 >> v1:172.21.15.165:6789/0 conn(0x7fd5a80cbd40 legacy=0x7fd5a80cc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.003+0000 7fd5a5d9b700 1 -- 172.21.15.1:0/293771608 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a815dc00 con 0x7fd5a80c3670 2024-03-18T20:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.003+0000 7fd5a5d9b700 1 --2- 172.21.15.1:0/293771608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5a80c3670 0x7fd5a815e120 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7fd5a80b5ea0 tx=0x7fd598009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4523a3173ac081 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.003+0000 7fd5a559a700 1 -- 172.21.15.1:0/293771608 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd598004230 con 0x7fd5a80c3670 2024-03-18T20:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.003+0000 7fd5a559a700 1 -- 172.21.15.1:0/293771608 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd598004390 con 0x7fd5a80c3670 2024-03-18T20:26:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.003+0000 7fd5a559a700 1 -- 172.21.15.1:0/293771608 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5980036a0 con 0x7fd5a80c3670 2024-03-18T20:26:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.004+0000 7fd5acf91700 1 -- 172.21.15.1:0/293771608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5a80c3670 msgr2=0x7fd5a815e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.004+0000 7fd5acf91700 1 --2- 172.21.15.1:0/293771608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5a80c3670 0x7fd5a815e120 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7fd5a80b5ea0 tx=0x7fd598009bd0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.004+0000 7fd5acf91700 1 -- 172.21.15.1:0/293771608 shutdown_connections 2024-03-18T20:26:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.004+0000 7fd5acf91700 1 --2- 172.21.15.1:0/293771608 >> v2:172.21.15.165:3300/0 conn(0x7fd5a815e660 0x7fd5a80cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.004+0000 7fd5acf91700 1 --2- 172.21.15.1:0/293771608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5a80c3670 0x7fd5a815e120 unknown :-1 s=CLOSED pgs=2841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.004+0000 7fd5acf91700 1 -- 172.21.15.1:0/293771608 >> 172.21.15.1:0/293771608 conn(0x7fd5a80b1aa0 msgr2=0x7fd5a80b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.004+0000 7fd5acf91700 1 -- 172.21.15.1:0/293771608 shutdown_connections 2024-03-18T20:26:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.004+0000 7fd5acf91700 1 -- 172.21.15.1:0/293771608 wait complete. 2024-03-18T20:26:36.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.005+0000 7fd5acf91700 1 Processor -- start 2024-03-18T20:26:36.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.005+0000 7fd5acf91700 1 -- start start 2024-03-18T20:26:36.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.006+0000 7fd5acf91700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5a80c3670 0x7fd5a8156630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.006+0000 7fd5acf91700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5a80cbd40 0x7fd5a81f6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.006+0000 7fd5acf91700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5a815e660 0x7fd5a80c42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.006+0000 7fd5acf91700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a80c4cb0 con 0x7fd5a80c3670 2024-03-18T20:26:36.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.006+0000 7fd5acf91700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a80c4f90 con 0x7fd5a80cbd40 2024-03-18T20:26:36.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.006+0000 7fd5acf91700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a80b4340 con 0x7fd5a815e660 2024-03-18T20:26:36.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.006+0000 7fd5a659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5a80c3670 0x7fd5a8156630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:36.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.006+0000 7fd5a659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5a80c3670 0x7fd5a8156630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35736/0 (socket says 172.21.15.1:35736) 2024-03-18T20:26:36.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.006+0000 7fd5a659c700 1 -- 172.21.15.1:0/1629913630 learned_addr learned my addr 172.21.15.1:0/1629913630 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:36.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.006+0000 7fd5a6d9d700 1 --2- 172.21.15.1:0/1629913630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5a815e660 0x7fd5a80c42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:36.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.006+0000 7fd5a659c700 1 -- 172.21.15.1:0/1629913630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5a815e660 msgr2=0x7fd5a80c42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:36.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.006+0000 7fd5a659c700 1 --2- 172.21.15.1:0/1629913630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5a815e660 0x7fd5a80c42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.006+0000 7fd5a659c700 1 -- 172.21.15.1:0/1629913630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5a80cbd40 msgr2=0x7fd5a81f6a50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T20:26:36.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.006+0000 7fd5a659c700 1 --2- 172.21.15.1:0/1629913630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5a80cbd40 0x7fd5a81f6a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.006+0000 7fd5a659c700 1 -- 172.21.15.1:0/1629913630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd598009550 con 0x7fd5a80c3670 2024-03-18T20:26:36.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.006+0000 7fd5a659c700 1 --2- 172.21.15.1:0/1629913630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5a80c3670 0x7fd5a8156630 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 crypto rx=0x7fd59c00c4d0 tx=0x7fd59c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:36.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.007+0000 7fd596ffd700 1 -- 172.21.15.1:0/1629913630 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c011790 con 0x7fd5a80c3670 2024-03-18T20:26:36.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.007+0000 7fd596ffd700 1 -- 172.21.15.1:0/1629913630 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59c0118f0 con 0x7fd5a80c3670 2024-03-18T20:26:36.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.007+0000 7fd5acf91700 1 -- 172.21.15.1:0/1629913630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a806b3c0 con 0x7fd5a80c3670 2024-03-18T20:26:36.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.007+0000 7fd596ffd700 1 -- 172.21.15.1:0/1629913630 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c011ca0 con 0x7fd5a80c3670 2024-03-18T20:26:36.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.007+0000 7fd5acf91700 1 -- 172.21.15.1:0/1629913630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a80c5270 con 0x7fd5a80c3670 2024-03-18T20:26:36.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.008+0000 7fd5acf91700 1 -- 172.21.15.1:0/1629913630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a80cf580 con 0x7fd5a80c3670 2024-03-18T20:26:36.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.008+0000 7fd596ffd700 1 -- 172.21.15.1:0/1629913630 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fd59c01c460 con 0x7fd5a80c3670 2024-03-18T20:26:36.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.009+0000 7fd596ffd700 1 --2- 172.21.15.1:0/1629913630 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd58806aed0 0x7fd58806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.009+0000 7fd596ffd700 1 -- 172.21.15.1:0/1629913630 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fd59c0895d0 con 0x7fd5a80c3670 2024-03-18T20:26:36.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.009+0000 7fd5a5d9b700 1 --2- 172.21.15.1:0/1629913630 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd58806aed0 0x7fd58806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:36.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.009+0000 7fd5a5d9b700 1 --2- 172.21.15.1:0/1629913630 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd58806aed0 0x7fd58806d390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd5a80b3e50 tx=0x7fd598008030 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:36.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.013+0000 7fd596ffd700 1 -- 172.21.15.1:0/1629913630 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd59c054360 con 0x7fd5a80c3670 2024-03-18T20:26:36.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.218+0000 7fd5acf91700 1 -- 172.21.15.1:0/1629913630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd5a80b4860 con 0x7fd5a80c3670 2024-03-18T20:26:36.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.218+0000 7fd596ffd700 1 -- 172.21.15.1:0/1629913630 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v253) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd5a80b4860 con 0x7fd5a80c3670 2024-03-18T20:26:36.222 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:26:36.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.220+0000 7fd5acf91700 1 -- 172.21.15.1:0/1629913630 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd58806aed0 msgr2=0x7fd58806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:36.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.220+0000 7fd5acf91700 1 --2- 172.21.15.1:0/1629913630 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd58806aed0 0x7fd58806d390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd5a80b3e50 tx=0x7fd598008030 comp rx=0 tx=0).stop 2024-03-18T20:26:36.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.220+0000 7fd5acf91700 1 -- 172.21.15.1:0/1629913630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5a80c3670 msgr2=0x7fd5a8156630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:36.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.220+0000 7fd5acf91700 1 --2- 172.21.15.1:0/1629913630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5a80c3670 0x7fd5a8156630 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 crypto rx=0x7fd59c00c4d0 tx=0x7fd59c00c7e0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.220+0000 7fd5acf91700 1 -- 172.21.15.1:0/1629913630 shutdown_connections 2024-03-18T20:26:36.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.220+0000 7fd5acf91700 1 --2- 172.21.15.1:0/1629913630 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd58806aed0 0x7fd58806d390 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.220+0000 7fd5acf91700 1 --2- 172.21.15.1:0/1629913630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5a815e660 0x7fd5a80c42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.220+0000 7fd5acf91700 1 --2- 172.21.15.1:0/1629913630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5a80c3670 0x7fd5a8156630 unknown :-1 s=CLOSED pgs=2842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.220+0000 7fd5acf91700 1 --2- 172.21.15.1:0/1629913630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5a80cbd40 0x7fd5a81f6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.220+0000 7fd5acf91700 1 -- 172.21.15.1:0/1629913630 >> 172.21.15.1:0/1629913630 conn(0x7fd5a80b1aa0 msgr2=0x7fd5a80b1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:36.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.220+0000 7fd5acf91700 1 -- 172.21.15.1:0/1629913630 shutdown_connections 2024-03-18T20:26:36.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.220+0000 7fd5acf91700 1 -- 172.21.15.1:0/1629913630 wait complete. 2024-03-18T20:26:36.236 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:26:36.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.359+0000 7ff36baaf700 1 Processor -- start 2024-03-18T20:26:36.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.359+0000 7ff36baaf700 1 -- start start 2024-03-18T20:26:36.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.360+0000 7ff36baaf700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff3640727b0 0x7ff364161610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.360+0000 7ff36baaf700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff364161b50 0x7ff36406a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.360+0000 7ff36baaf700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff364160ea0 con 0x7ff36415ad70 2024-03-18T20:26:36.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.360+0000 7ff36baaf700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3640725b0 con 0x7ff3640727b0 2024-03-18T20:26:36.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.360+0000 7ff36baaf700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff36415e090 con 0x7ff364161b50 2024-03-18T20:26:36.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.360+0000 7ff36904a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff3640727b0 0x7ff364161610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:36.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.360+0000 7ff36984b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff36415ad70 0x7ff36415d8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56310/0 (socket says 172.21.15.1:56310) 2024-03-18T20:26:36.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.360+0000 7ff36984b700 1 -- 172.21.15.1:0/772250025 learned_addr learned my addr 172.21.15.1:0/772250025 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:36.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.361+0000 7ff36904a700 1 -- 172.21.15.1:0/772250025 >> v2:172.21.15.165:3300/0 conn(0x7ff364161b50 msgr2=0x7ff36406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:36.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.361+0000 7ff36a04c700 1 --2- 172.21.15.1:0/772250025 >> v2:172.21.15.165:3300/0 conn(0x7ff364161b50 0x7ff36406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:36.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.361+0000 7ff36904a700 1 --2- 172.21.15.1:0/772250025 >> v2:172.21.15.165:3300/0 conn(0x7ff364161b50 0x7ff36406a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.361+0000 7ff36904a700 1 -- 172.21.15.1:0/772250025 >> v1:172.21.15.1:6789/0 conn(0x7ff36415ad70 legacy=0x7ff36415d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:36.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.361+0000 7ff368849700 1 -- 172.21.15.1:0/772250025 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531252201 0 0) 0x7ff364160ea0 con 0x7ff36415ad70 2024-03-18T20:26:36.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.361+0000 7ff36904a700 1 -- 172.21.15.1:0/772250025 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36415e370 con 0x7ff3640727b0 2024-03-18T20:26:36.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.361+0000 7ff36904a700 1 --2- 172.21.15.1:0/772250025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3640727b0 0x7ff364161610 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7ff364071600 tx=0x7ff360009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b4c313fa13abca0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:36.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.361+0000 7ff368849700 1 -- 172.21.15.1:0/772250025 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff360011830 con 0x7ff3640727b0 2024-03-18T20:26:36.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.361+0000 7ff368849700 1 -- 172.21.15.1:0/772250025 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff360011990 con 0x7ff3640727b0 2024-03-18T20:26:36.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.361+0000 7ff368849700 1 -- 172.21.15.1:0/772250025 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff360011d20 con 0x7ff3640727b0 2024-03-18T20:26:36.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.362+0000 7ff36baaf700 1 -- 172.21.15.1:0/772250025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3640727b0 msgr2=0x7ff364161610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:36.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.362+0000 7ff36baaf700 1 --2- 172.21.15.1:0/772250025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3640727b0 0x7ff364161610 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7ff364071600 tx=0x7ff360009b20 comp rx=0 tx=0).stop 2024-03-18T20:26:36.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.362+0000 7ff36baaf700 1 -- 172.21.15.1:0/772250025 shutdown_connections 2024-03-18T20:26:36.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.362+0000 7ff36baaf700 1 --2- 172.21.15.1:0/772250025 >> v2:172.21.15.165:3300/0 conn(0x7ff364161b50 0x7ff36406a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.362+0000 7ff36baaf700 1 --2- 172.21.15.1:0/772250025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3640727b0 0x7ff364161610 unknown :-1 s=CLOSED pgs=2844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.362+0000 7ff36baaf700 1 -- 172.21.15.1:0/772250025 >> 172.21.15.1:0/772250025 conn(0x7ff3640b1cb0 msgr2=0x7ff3640b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:36.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.362+0000 7ff36baaf700 1 -- 172.21.15.1:0/772250025 shutdown_connections 2024-03-18T20:26:36.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.362+0000 7ff36baaf700 1 -- 172.21.15.1:0/772250025 wait complete. 2024-03-18T20:26:36.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.363+0000 7ff36baaf700 1 Processor -- start 2024-03-18T20:26:36.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.363+0000 7ff36baaf700 1 -- start start 2024-03-18T20:26:36.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff36baaf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3640727b0 0x7ff3640ae310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff36baaf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff36415ad70 0x7ff3641485a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff36baaf700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff364161b50 0x7ff3640cf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff36baaf700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff364148e80 con 0x7ff3640727b0 2024-03-18T20:26:36.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff36baaf700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff364148ae0 con 0x7ff364161b50 2024-03-18T20:26:36.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff36baaf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3640ae8f0 con 0x7ff36415ad70 2024-03-18T20:26:36.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff36984b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3640727b0 0x7ff3640ae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:36.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff36984b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3640727b0 0x7ff3640ae310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35752/0 (socket says 172.21.15.1:35752) 2024-03-18T20:26:36.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff36984b700 1 -- 172.21.15.1:0/3751422936 learned_addr learned my addr 172.21.15.1:0/3751422936 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:36.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff36904a700 1 --2- 172.21.15.1:0/3751422936 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff36415ad70 0x7ff3641485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:36.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff36a04c700 1 --2- 172.21.15.1:0/3751422936 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff364161b50 0x7ff3640cf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:36.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff36984b700 1 -- 172.21.15.1:0/3751422936 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff36415ad70 msgr2=0x7ff3641485a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:36.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff36984b700 1 --2- 172.21.15.1:0/3751422936 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff36415ad70 0x7ff3641485a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff36984b700 1 -- 172.21.15.1:0/3751422936 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff364161b50 msgr2=0x7ff3640cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:36.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff36984b700 1 --2- 172.21.15.1:0/3751422936 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff364161b50 0x7ff3640cf6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff36984b700 1 -- 172.21.15.1:0/3751422936 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3600094a0 con 0x7ff3640727b0 2024-03-18T20:26:36.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff36a04c700 1 --2- 172.21.15.1:0/3751422936 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff364161b50 0x7ff3640cf6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:36.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff36984b700 1 --2- 172.21.15.1:0/3751422936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3640727b0 0x7ff3640ae310 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7ff35400c750 tx=0x7ff35400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:36.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff35a7fc700 1 -- 172.21.15.1:0/3751422936 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354004c80 con 0x7ff3640727b0 2024-03-18T20:26:36.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff35a7fc700 1 -- 172.21.15.1:0/3751422936 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff354004e00 con 0x7ff3640727b0 2024-03-18T20:26:36.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff36baaf700 1 -- 172.21.15.1:0/3751422936 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3640aebd0 con 0x7ff3640727b0 2024-03-18T20:26:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.364+0000 7ff35a7fc700 1 -- 172.21.15.1:0/3751422936 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354007310 con 0x7ff3640727b0 2024-03-18T20:26:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.365+0000 7ff36baaf700 1 -- 172.21.15.1:0/3751422936 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3640cfc30 con 0x7ff3640727b0 2024-03-18T20:26:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.365+0000 7ff36baaf700 1 -- 172.21.15.1:0/3751422936 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3641610b0 con 0x7ff3640727b0 2024-03-18T20:26:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.366+0000 7ff35a7fc700 1 -- 172.21.15.1:0/3751422936 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7ff354007470 con 0x7ff3640727b0 2024-03-18T20:26:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.366+0000 7ff35a7fc700 1 --2- 172.21.15.1:0/3751422936 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff34806aed0 0x7ff34806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.366+0000 7ff35a7fc700 1 -- 172.21.15.1:0/3751422936 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 6129+0+0 (secure 0 0 0) 0x7ff3540896c0 con 0x7ff3640727b0 2024-03-18T20:26:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.366+0000 7ff36904a700 1 --2- 172.21.15.1:0/3751422936 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff34806aed0 0x7ff34806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:36.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.367+0000 7ff36904a700 1 --2- 172.21.15.1:0/3751422936 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff34806aed0 0x7ff34806d390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff3600090b0 tx=0x7ff360008030 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:36.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.371+0000 7ff35a7fc700 1 -- 172.21.15.1:0/3751422936 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff354054390 con 0x7ff3640727b0 2024-03-18T20:26:36.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.575+0000 7ff36baaf700 1 -- 172.21.15.1:0/3751422936 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff36406ac30 con 0x7ff3640727b0 2024-03-18T20:26:36.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.576+0000 7ff35a7fc700 1 -- 172.21.15.1:0/3751422936 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v253) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff354087e60 con 0x7ff3640727b0 2024-03-18T20:26:36.580 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:26:36.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.578+0000 7ff36baaf700 1 -- 172.21.15.1:0/3751422936 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff34806aed0 msgr2=0x7ff34806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:36.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.578+0000 7ff36baaf700 1 --2- 172.21.15.1:0/3751422936 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff34806aed0 0x7ff34806d390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff3600090b0 tx=0x7ff360008030 comp rx=0 tx=0).stop 2024-03-18T20:26:36.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.578+0000 7ff36baaf700 1 -- 172.21.15.1:0/3751422936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3640727b0 msgr2=0x7ff3640ae310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.578+0000 7ff36baaf700 1 --2- 172.21.15.1:0/3751422936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3640727b0 0x7ff3640ae310 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7ff35400c750 tx=0x7ff35400cb10 comp rx=0 tx=0).stop 2024-03-18T20:26:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.578+0000 7ff36baaf700 1 -- 172.21.15.1:0/3751422936 shutdown_connections 2024-03-18T20:26:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.578+0000 7ff36baaf700 1 --2- 172.21.15.1:0/3751422936 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff34806aed0 0x7ff34806d390 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.578+0000 7ff36baaf700 1 --2- 172.21.15.1:0/3751422936 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff364161b50 0x7ff3640cf6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.578+0000 7ff36baaf700 1 --2- 172.21.15.1:0/3751422936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3640727b0 0x7ff3640ae310 unknown :-1 s=CLOSED pgs=2845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.578+0000 7ff36baaf700 1 --2- 172.21.15.1:0/3751422936 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff36415ad70 0x7ff3641485a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.578+0000 7ff36baaf700 1 -- 172.21.15.1:0/3751422936 >> 172.21.15.1:0/3751422936 conn(0x7ff3640b1cb0 msgr2=0x7ff3640b0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.578+0000 7ff36baaf700 1 -- 172.21.15.1:0/3751422936 shutdown_connections 2024-03-18T20:26:36.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.578+0000 7ff36baaf700 1 -- 172.21.15.1:0/3751422936 wait complete. 2024-03-18T20:26:36.594 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T20:26:36.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.715+0000 7f751898d700 1 Processor -- start 2024-03-18T20:26:36.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.715+0000 7f751898d700 1 -- start start 2024-03-18T20:26:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751898d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7514143c20 0x7f75140b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751898d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f751406f240 0x7f75140cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751898d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7514144000 con 0x7f75140b3f40 2024-03-18T20:26:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751898d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75140b43c0 con 0x7f7514143c20 2024-03-18T20:26:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751898d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75140b51b0 con 0x7f751406f240 2024-03-18T20:26:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f7511d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f75140b3f40 0x7f751406eb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56312/0 (socket says 172.21.15.1:56312) 2024-03-18T20:26:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f7511d9b700 1 -- 172.21.15.1:0/1243149592 learned_addr learned my addr 172.21.15.1:0/1243149592 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751159a700 1 -- 172.21.15.1:0/1243149592 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937182017 0 0) 0x7f7514144000 con 0x7f75140b3f40 2024-03-18T20:26:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751159a700 1 -- 172.21.15.1:0/1243149592 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74fc0035b0 con 0x7f75140b3f40 2024-03-18T20:26:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f7512d9d700 1 --2- 172.21.15.1:0/1243149592 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f751406f240 0x7f75140cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751159a700 1 -- 172.21.15.1:0/1243149592 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3294009255 0 0) 0x7f74fc0035b0 con 0x7f75140b3f40 2024-03-18T20:26:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751159a700 1 -- 172.21.15.1:0/1243149592 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f751406f240 msgr2=0x7f75140cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751159a700 1 --2- 172.21.15.1:0/1243149592 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f751406f240 0x7f75140cddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751159a700 1 -- 172.21.15.1:0/1243149592 >> v2:172.21.15.165:3300/0 conn(0x7f7514143c20 msgr2=0x7f75140b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751159a700 1 --2- 172.21.15.1:0/1243149592 >> v2:172.21.15.165:3300/0 conn(0x7f7514143c20 0x7f75140b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751159a700 1 -- 172.21.15.1:0/1243149592 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75140cd8d0 con 0x7f75140b3f40 2024-03-18T20:26:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751159a700 1 -- 172.21.15.1:0/1243149592 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f7508001670 con 0x7f75140b3f40 2024-03-18T20:26:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751159a700 1 -- 172.21.15.1:0/1243149592 >> v1:172.21.15.1:6789/0 conn(0x7f75140b3f40 legacy=0x7f751406eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751159a700 1 --2- 172.21.15.1:0/1243149592 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74fc003a30 0x7f74fc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751159a700 1 --2- 172.21.15.1:0/1243149592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc004850 0x7f74fc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751159a700 1 --2- 172.21.15.1:0/1243149592 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f751406f240 0x7f74fc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751159a700 1 -- 172.21.15.1:0/1243149592 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75140b43c0 con 0x7f74fc004850 2024-03-18T20:26:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751159a700 1 -- 172.21.15.1:0/1243149592 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc00bac0 con 0x7f74fc003a30 2024-03-18T20:26:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f751159a700 1 -- 172.21.15.1:0/1243149592 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc00bc80 con 0x7f751406f240 2024-03-18T20:26:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.716+0000 7f7511d9b700 1 --2- 172.21.15.1:0/1243149592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc004850 0x7f74fc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.717+0000 7f7512d9d700 1 --2- 172.21.15.1:0/1243149592 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f751406f240 0x7f74fc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.717+0000 7f7511d9b700 1 -- 172.21.15.1:0/1243149592 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f751406f240 msgr2=0x7f74fc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.717+0000 7f7511d9b700 1 --2- 172.21.15.1:0/1243149592 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f751406f240 0x7f74fc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.717+0000 7f7511d9b700 1 -- 172.21.15.1:0/1243149592 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74fc003a30 msgr2=0x7f74fc003e50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T20:26:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.717+0000 7f7511d9b700 1 --2- 172.21.15.1:0/1243149592 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74fc003a30 0x7f74fc003e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.717+0000 7f7511d9b700 1 -- 172.21.15.1:0/1243149592 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74fc0035b0 con 0x7f74fc004850 2024-03-18T20:26:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.717+0000 7f7511d9b700 1 --2- 172.21.15.1:0/1243149592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc004850 0x7f74fc004cd0 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 crypto rx=0x7f7508001eb0 tx=0x7f74fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce720980080969b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.717+0000 7f751159a700 1 -- 172.21.15.1:0/1243149592 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750800cd10 con 0x7f74fc004850 2024-03-18T20:26:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.717+0000 7f751159a700 1 -- 172.21.15.1:0/1243149592 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750800ce70 con 0x7f74fc004850 2024-03-18T20:26:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.717+0000 7f751159a700 1 -- 172.21.15.1:0/1243149592 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7508005b80 con 0x7f74fc004850 2024-03-18T20:26:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.717+0000 7f751898d700 1 -- 172.21.15.1:0/1243149592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc004850 msgr2=0x7f74fc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.717+0000 7f751898d700 1 --2- 172.21.15.1:0/1243149592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc004850 0x7f74fc004cd0 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 crypto rx=0x7f7508001eb0 tx=0x7f74fc005880 comp rx=0 tx=0).stop 2024-03-18T20:26:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.718+0000 7f7512d9d700 1 -- 172.21.15.1:0/1243149592 reap_dead start 2024-03-18T20:26:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.718+0000 7f751898d700 1 -- 172.21.15.1:0/1243149592 shutdown_connections 2024-03-18T20:26:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.718+0000 7f751898d700 1 -- 172.21.15.1:0/1243149592 >> 172.21.15.1:0/1243149592 conn(0x7f75140b1d20 msgr2=0x7f75140b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.718+0000 7f751898d700 1 -- 172.21.15.1:0/1243149592 shutdown_connections 2024-03-18T20:26:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.718+0000 7f751898d700 1 -- 172.21.15.1:0/1243149592 wait complete. 2024-03-18T20:26:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.719+0000 7f751898d700 1 Processor -- start 2024-03-18T20:26:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.719+0000 7f751898d700 1 -- start start 2024-03-18T20:26:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.720+0000 7f751898d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc004850 0x7f75140a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.720+0000 7f751898d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7514159b00 0x7f751414f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.720+0000 7f751898d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f75140a84a0 0x7f75140cec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.720+0000 7f751898d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751414fe50 con 0x7f74fc004850 2024-03-18T20:26:36.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.720+0000 7f751898d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75140a8df0 con 0x7f7514159b00 2024-03-18T20:26:36.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.720+0000 7f751898d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75140a90d0 con 0x7f75140a84a0 2024-03-18T20:26:36.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.720+0000 7f751259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc004850 0x7f75140a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:36.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.720+0000 7f751259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc004850 0x7f75140a9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35780/0 (socket says 172.21.15.1:35780) 2024-03-18T20:26:36.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.720+0000 7f751259c700 1 -- 172.21.15.1:0/904755458 learned_addr learned my addr 172.21.15.1:0/904755458 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:36.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.720+0000 7f7512d9d700 1 --2- 172.21.15.1:0/904755458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f75140a84a0 0x7f75140cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:36.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.720+0000 7f7511d9b700 1 --2- 172.21.15.1:0/904755458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7514159b00 0x7f751414f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:36.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.720+0000 7f751259c700 1 -- 172.21.15.1:0/904755458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f75140a84a0 msgr2=0x7f75140cec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:36.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.720+0000 7f751259c700 1 --2- 172.21.15.1:0/904755458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f75140a84a0 0x7f75140cec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.720+0000 7f751259c700 1 -- 172.21.15.1:0/904755458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7514159b00 msgr2=0x7f751414f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:36.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.720+0000 7f751259c700 1 --2- 172.21.15.1:0/904755458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7514159b00 0x7f751414f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.720+0000 7f751259c700 1 -- 172.21.15.1:0/904755458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75140a93b0 con 0x7f74fc004850 2024-03-18T20:26:36.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.720+0000 7f7512d9d700 1 --2- 172.21.15.1:0/904755458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f75140a84a0 0x7f75140cec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:36.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.720+0000 7f7511d9b700 1 --2- 172.21.15.1:0/904755458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7514159b00 0x7f751414f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:36.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.720+0000 7f751259c700 1 --2- 172.21.15.1:0/904755458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc004850 0x7f75140a9e90 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 crypto rx=0x7f750400b520 tx=0x7f750400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:36.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.721+0000 7f7502ffd700 1 -- 172.21.15.1:0/904755458 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7504011830 con 0x7f74fc004850 2024-03-18T20:26:36.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.721+0000 7f7502ffd700 1 -- 172.21.15.1:0/904755458 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7504011990 con 0x7f74fc004850 2024-03-18T20:26:36.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.721+0000 7f751898d700 1 -- 172.21.15.1:0/904755458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75140a9690 con 0x7f74fc004850 2024-03-18T20:26:36.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.721+0000 7f7502ffd700 1 -- 172.21.15.1:0/904755458 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7504011d40 con 0x7f74fc004850 2024-03-18T20:26:36.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.721+0000 7f751898d700 1 -- 172.21.15.1:0/904755458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75140a8950 con 0x7f74fc004850 2024-03-18T20:26:36.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.722+0000 7f751898d700 1 -- 172.21.15.1:0/904755458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75140a9bb0 con 0x7f74fc004850 2024-03-18T20:26:36.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.722+0000 7f7502ffd700 1 -- 172.21.15.1:0/904755458 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f750401a400 con 0x7f74fc004850 2024-03-18T20:26:36.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.723+0000 7f7502ffd700 1 --2- 172.21.15.1:0/904755458 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f74f406aed0 0x7f74f406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:36.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.723+0000 7f7502ffd700 1 -- 172.21.15.1:0/904755458 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f75040898b0 con 0x7f74fc004850 2024-03-18T20:26:36.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.723+0000 7f7511d9b700 1 --2- 172.21.15.1:0/904755458 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f74f406aed0 0x7f74f406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:36.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.723+0000 7f7511d9b700 1 --2- 172.21.15.1:0/904755458 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f74f406aed0 0x7f74f406d390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7514159f20 tx=0x7f750800a000 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:36.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.727+0000 7f7502ffd700 1 -- 172.21.15.1:0/904755458 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f75140a9bb0 con 0x7f74fc004850 2024-03-18T20:26:36.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.931+0000 7f751898d700 1 -- 172.21.15.1:0/904755458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f75140a9bb0 con 0x7f74fc004850 2024-03-18T20:26:36.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.931+0000 7f7502ffd700 1 -- 172.21.15.1:0/904755458 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v253) v1 ==== 84+0+209 (secure 0 0 0) 0x7f75140a9bb0 con 0x7f74fc004850 2024-03-18T20:26:36.936 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:36.936 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T20:26:36.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:36.936 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-18T20:26:36.936 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-18T20:26:36.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:36.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:36.936 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 31, 2024-03-18T20:26:36.937 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": "cephfs_metadata" 2024-03-18T20:26:36.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:36.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:36.937 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 32, 2024-03-18T20:26:36.937 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": "cephfs_data" 2024-03-18T20:26:36.937 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:36.937 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T20:26:36.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.933+0000 7f751898d700 1 -- 172.21.15.1:0/904755458 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f74f406aed0 msgr2=0x7f74f406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:36.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.933+0000 7f751898d700 1 --2- 172.21.15.1:0/904755458 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f74f406aed0 0x7f74f406d390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7514159f20 tx=0x7f750800a000 comp rx=0 tx=0).stop 2024-03-18T20:26:36.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.933+0000 7f751898d700 1 -- 172.21.15.1:0/904755458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc004850 msgr2=0x7f75140a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:36.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.933+0000 7f751898d700 1 --2- 172.21.15.1:0/904755458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc004850 0x7f75140a9e90 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 crypto rx=0x7f750400b520 tx=0x7f750400b8e0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.933+0000 7f751898d700 1 -- 172.21.15.1:0/904755458 shutdown_connections 2024-03-18T20:26:36.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.933+0000 7f751898d700 1 --2- 172.21.15.1:0/904755458 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f74f406aed0 0x7f74f406d390 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.933+0000 7f751898d700 1 --2- 172.21.15.1:0/904755458 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f75140a84a0 0x7f75140cec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.933+0000 7f751898d700 1 --2- 172.21.15.1:0/904755458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f74fc004850 0x7f75140a9e90 unknown :-1 s=CLOSED pgs=2848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.933+0000 7f751898d700 1 --2- 172.21.15.1:0/904755458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7514159b00 0x7f751414f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:36.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.933+0000 7f751898d700 1 -- 172.21.15.1:0/904755458 >> 172.21.15.1:0/904755458 conn(0x7f75140b1d20 msgr2=0x7f751406eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:36.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.933+0000 7f751898d700 1 -- 172.21.15.1:0/904755458 shutdown_connections 2024-03-18T20:26:36.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:36.933+0000 7f751898d700 1 -- 172.21.15.1:0/904755458 wait complete. 2024-03-18T20:26:36.950 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2024-03-18T20:26:36.950 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2024-03-18T20:26:37.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.075+0000 7f5843fb8700 1 Processor -- start 2024-03-18T20:26:37.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.076+0000 7f5843fb8700 1 -- start start 2024-03-18T20:26:37.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.076+0000 7f5843fb8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f583c15b300 0x7f583c0aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:37.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.076+0000 7f5843fb8700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583c15e860 con 0x7f583c15af20 2024-03-18T20:26:37.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.076+0000 7f5843fb8700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583c0719a0 con 0x7f583c0af4e0 2024-03-18T20:26:37.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.076+0000 7f5843fb8700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f583c0af970 con 0x7f583c15b300 2024-03-18T20:26:37.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.076+0000 7f5841553700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f583c15b300 0x7f583c0aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:37.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.076+0000 7f5841d54700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f583c15af20 0x7f583c072480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56314/0 (socket says 172.21.15.1:56314) 2024-03-18T20:26:37.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.076+0000 7f5841d54700 1 -- 172.21.15.1:0/3584186589 learned_addr learned my addr 172.21.15.1:0/3584186589 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:37.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.077+0000 7f5841553700 1 -- 172.21.15.1:0/3584186589 >> v1:172.21.15.165:6789/0 conn(0x7f583c0af4e0 legacy=0x7f583c15dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:37.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.077+0000 7f5841553700 1 -- 172.21.15.1:0/3584186589 >> v1:172.21.15.1:6789/0 conn(0x7f583c15af20 legacy=0x7f583c072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:37.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.077+0000 7f5841553700 1 -- 172.21.15.1:0/3584186589 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c071c80 con 0x7f583c15b300 2024-03-18T20:26:37.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.077+0000 7f5840d52700 1 -- 172.21.15.1:0/3584186589 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996071451 0 0) 0x7f583c15e860 con 0x7f583c15af20 2024-03-18T20:26:37.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.077+0000 7f5841553700 1 --2- 172.21.15.1:0/3584186589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f583c15b300 0x7f583c0aefa0 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 crypto rx=0x7f582c009870 tx=0x7f582c009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d89b88b1b3ec0821 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:37.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.077+0000 7f5840d52700 1 -- 172.21.15.1:0/3584186589 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c004230 con 0x7f583c15b300 2024-03-18T20:26:37.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.077+0000 7f5840d52700 1 -- 172.21.15.1:0/3584186589 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582c004390 con 0x7f583c15b300 2024-03-18T20:26:37.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.077+0000 7f5840d52700 1 -- 172.21.15.1:0/3584186589 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c0036a0 con 0x7f583c15b300 2024-03-18T20:26:37.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.078+0000 7f5843fb8700 1 -- 172.21.15.1:0/3584186589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f583c15b300 msgr2=0x7f583c0aefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:37.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.078+0000 7f5843fb8700 1 --2- 172.21.15.1:0/3584186589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f583c15b300 0x7f583c0aefa0 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 crypto rx=0x7f582c009870 tx=0x7f582c009b80 comp rx=0 tx=0).stop 2024-03-18T20:26:37.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.078+0000 7f5843fb8700 1 -- 172.21.15.1:0/3584186589 shutdown_connections 2024-03-18T20:26:37.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.078+0000 7f5843fb8700 1 --2- 172.21.15.1:0/3584186589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f583c15b300 0x7f583c0aefa0 unknown :-1 s=CLOSED pgs=2850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:37.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.078+0000 7f5843fb8700 1 -- 172.21.15.1:0/3584186589 >> 172.21.15.1:0/3584186589 conn(0x7f583c0d00a0 msgr2=0x7f583c0d04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:37.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.078+0000 7f5843fb8700 1 -- 172.21.15.1:0/3584186589 shutdown_connections 2024-03-18T20:26:37.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.078+0000 7f5843fb8700 1 -- 172.21.15.1:0/3584186589 wait complete. 2024-03-18T20:26:37.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.079+0000 7f5843fb8700 1 Processor -- start 2024-03-18T20:26:37.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.079+0000 7f5843fb8700 1 -- start start 2024-03-18T20:26:37.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.080+0000 7f5843fb8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f583c0af4e0 0x7f583c15b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:37.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.080+0000 7f5843fb8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f583c15af20 0x7f583c0adbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:37.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.080+0000 7f5843fb8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f583c15b300 0x7f583c144670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:37.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.080+0000 7f5843fb8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c14e550 con 0x7f583c0af4e0 2024-03-18T20:26:37.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.080+0000 7f5843fb8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c15bde0 con 0x7f583c15af20 2024-03-18T20:26:37.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.080+0000 7f5843fb8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f583c15c0c0 con 0x7f583c15b300 2024-03-18T20:26:37.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.080+0000 7f5841d54700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f583c0af4e0 0x7f583c15b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:37.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.080+0000 7f5841d54700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f583c0af4e0 0x7f583c15b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35798/0 (socket says 172.21.15.1:35798) 2024-03-18T20:26:37.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.080+0000 7f5841d54700 1 -- 172.21.15.1:0/2885401036 learned_addr learned my addr 172.21.15.1:0/2885401036 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:37.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.080+0000 7f5841553700 1 --2- 172.21.15.1:0/2885401036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f583c15af20 0x7f583c0adbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:37.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.080+0000 7f5842555700 1 --2- 172.21.15.1:0/2885401036 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f583c15b300 0x7f583c144670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:37.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.080+0000 7f5841d54700 1 -- 172.21.15.1:0/2885401036 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f583c15b300 msgr2=0x7f583c144670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:37.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.080+0000 7f5841d54700 1 --2- 172.21.15.1:0/2885401036 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f583c15b300 0x7f583c144670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:37.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.080+0000 7f5841d54700 1 -- 172.21.15.1:0/2885401036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f583c15af20 msgr2=0x7f583c0adbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:37.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.080+0000 7f5841d54700 1 --2- 172.21.15.1:0/2885401036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f583c15af20 0x7f583c0adbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:37.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.080+0000 7f5841d54700 1 -- 172.21.15.1:0/2885401036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582c0094a0 con 0x7f583c0af4e0 2024-03-18T20:26:37.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.080+0000 7f5842555700 1 --2- 172.21.15.1:0/2885401036 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f583c15b300 0x7f583c144670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:37.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.080+0000 7f5841553700 1 --2- 172.21.15.1:0/2885401036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f583c15af20 0x7f583c0adbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:37.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.080+0000 7f5841d54700 1 --2- 172.21.15.1:0/2885401036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f583c0af4e0 0x7f583c15b6e0 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f58380011e0 tx=0x7f583800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:37.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.081+0000 7f58327fc700 1 -- 172.21.15.1:0/2885401036 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5838004c80 con 0x7f583c0af4e0 2024-03-18T20:26:37.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.081+0000 7f58327fc700 1 -- 172.21.15.1:0/2885401036 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5838004e00 con 0x7f583c0af4e0 2024-03-18T20:26:37.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.081+0000 7f5843fb8700 1 -- 172.21.15.1:0/2885401036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583c15c3a0 con 0x7f583c0af4e0 2024-03-18T20:26:37.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.081+0000 7f58327fc700 1 -- 172.21.15.1:0/2885401036 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5838007310 con 0x7f583c0af4e0 2024-03-18T20:26:37.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.081+0000 7f5843fb8700 1 -- 172.21.15.1:0/2885401036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583c15c960 con 0x7f583c0af4e0 2024-03-18T20:26:37.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.081+0000 7f5843fb8700 1 -- 172.21.15.1:0/2885401036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583c0676a0 con 0x7f583c0af4e0 2024-03-18T20:26:37.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.082+0000 7f58327fc700 1 -- 172.21.15.1:0/2885401036 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f5838007470 con 0x7f583c0af4e0 2024-03-18T20:26:37.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.082+0000 7f58327fc700 1 --2- 172.21.15.1:0/2885401036 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f582006aed0 0x7f582006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:37.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.083+0000 7f58327fc700 1 -- 172.21.15.1:0/2885401036 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f58380895f0 con 0x7f583c0af4e0 2024-03-18T20:26:37.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.083+0000 7f5841553700 1 --2- 172.21.15.1:0/2885401036 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f582006aed0 0x7f582006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:37.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.083+0000 7f5841553700 1 --2- 172.21.15.1:0/2885401036 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f582006aed0 0x7f582006d390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f582c009870 tx=0x7f582c008030 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:37.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.087+0000 7f58327fc700 1 -- 172.21.15.1:0/2885401036 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5838054350 con 0x7f583c0af4e0 2024-03-18T20:26:37.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:37.291+0000 7f5843fb8700 1 -- 172.21.15.1:0/2885401036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f583c0b37d0 con 0x7f583c0af4e0 2024-03-18T20:26:38.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.153+0000 7f58327fc700 1 -- 172.21.15.1:0/2885401036 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v254) v1 ==== 144+0+0 (secure 0 0 0) 0x7f5838057970 con 0x7f583c0af4e0 2024-03-18T20:26:38.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.156+0000 7f5843fb8700 1 -- 172.21.15.1:0/2885401036 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f582006aed0 msgr2=0x7f582006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:38.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.156+0000 7f5843fb8700 1 --2- 172.21.15.1:0/2885401036 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f582006aed0 0x7f582006d390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f582c009870 tx=0x7f582c008030 comp rx=0 tx=0).stop 2024-03-18T20:26:38.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.156+0000 7f5843fb8700 1 -- 172.21.15.1:0/2885401036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f583c0af4e0 msgr2=0x7f583c15b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:38.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.156+0000 7f5843fb8700 1 --2- 172.21.15.1:0/2885401036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f583c0af4e0 0x7f583c15b6e0 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f58380011e0 tx=0x7f583800cae0 comp rx=0 tx=0).stop 2024-03-18T20:26:38.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.156+0000 7f5843fb8700 1 -- 172.21.15.1:0/2885401036 shutdown_connections 2024-03-18T20:26:38.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.156+0000 7f5843fb8700 1 --2- 172.21.15.1:0/2885401036 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f582006aed0 0x7f582006d390 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:38.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.156+0000 7f5843fb8700 1 --2- 172.21.15.1:0/2885401036 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f583c15b300 0x7f583c144670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:38.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.156+0000 7f5843fb8700 1 --2- 172.21.15.1:0/2885401036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f583c0af4e0 0x7f583c15b6e0 unknown :-1 s=CLOSED pgs=2851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:38.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.156+0000 7f5843fb8700 1 --2- 172.21.15.1:0/2885401036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f583c15af20 0x7f583c0adbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:38.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.156+0000 7f5843fb8700 1 -- 172.21.15.1:0/2885401036 >> 172.21.15.1:0/2885401036 conn(0x7f583c0d00a0 msgr2=0x7f583c159520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:38.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.156+0000 7f5843fb8700 1 -- 172.21.15.1:0/2885401036 shutdown_connections 2024-03-18T20:26:38.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.156+0000 7f5843fb8700 1 -- 172.21.15.1:0/2885401036 wait complete. 2024-03-18T20:26:38.162 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'backup_fs_metadata' created 2024-03-18T20:26:38.181 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T20:26:38.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.315+0000 7f619dd53700 1 Processor -- start 2024-03-18T20:26:38.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.316+0000 7f619dd53700 1 -- start start 2024-03-18T20:26:38.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.316+0000 7f619dd53700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f619806a750 0x7f619806ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:38.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.316+0000 7f619dd53700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f619815ec80 0x7f619815f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:38.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.316+0000 7f619dd53700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619815c340 con 0x7f61980ace80 2024-03-18T20:26:38.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.316+0000 7f619dd53700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6198070090 con 0x7f619806a750 2024-03-18T20:26:38.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.316+0000 7f619dd53700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619815b7a0 con 0x7f619815ec80 2024-03-18T20:26:38.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.317+0000 7f6196ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f61980ace80 0x7f61980ad310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56322/0 (socket says 172.21.15.1:56322) 2024-03-18T20:26:38.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.317+0000 7f6196ffd700 1 -- 172.21.15.1:0/3834023781 learned_addr learned my addr 172.21.15.1:0/3834023781 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:38.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.317+0000 7f6197fff700 1 --2- 172.21.15.1:0/3834023781 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f619815ec80 0x7f619815f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:38.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.317+0000 7f61977fe700 1 --2- 172.21.15.1:0/3834023781 >> v2:172.21.15.165:3300/0 conn(0x7f619806a750 0x7f619806ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:38.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.317+0000 7f61967fc700 1 -- 172.21.15.1:0/3834023781 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432796866 0 0) 0x7f619815c340 con 0x7f61980ace80 2024-03-18T20:26:38.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.317+0000 7f61967fc700 1 -- 172.21.15.1:0/3834023781 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61840035b0 con 0x7f61980ace80 2024-03-18T20:26:38.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.317+0000 7f61977fe700 1 -- 172.21.15.1:0/3834023781 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f619815ec80 msgr2=0x7f619815f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:38.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.317+0000 7f61977fe700 1 --2- 172.21.15.1:0/3834023781 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f619815ec80 0x7f619815f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:38.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.317+0000 7f61977fe700 1 -- 172.21.15.1:0/3834023781 >> v1:172.21.15.1:6789/0 conn(0x7f61980ace80 legacy=0x7f61980ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:38.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.317+0000 7f61977fe700 1 -- 172.21.15.1:0/3834023781 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619806b070 con 0x7f619806a750 2024-03-18T20:26:38.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.317+0000 7f6197fff700 1 --2- 172.21.15.1:0/3834023781 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f619815ec80 0x7f619815f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:26:38.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.317+0000 7f61977fe700 1 --2- 172.21.15.1:0/3834023781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f619806a750 0x7f619806ab30 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f6180009820 tx=0x7f6180009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab7153b80c9c183c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:38.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.317+0000 7f61967fc700 1 -- 172.21.15.1:0/3834023781 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6180005e30 con 0x7f619806a750 2024-03-18T20:26:38.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.317+0000 7f61967fc700 1 -- 172.21.15.1:0/3834023781 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6180006030 con 0x7f619806a750 2024-03-18T20:26:38.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.317+0000 7f61967fc700 1 -- 172.21.15.1:0/3834023781 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61800063a0 con 0x7f619806a750 2024-03-18T20:26:38.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.318+0000 7f619dd53700 1 -- 172.21.15.1:0/3834023781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f619806a750 msgr2=0x7f619806ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:38.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.318+0000 7f619dd53700 1 --2- 172.21.15.1:0/3834023781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f619806a750 0x7f619806ab30 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f6180009820 tx=0x7f6180009b30 comp rx=0 tx=0).stop 2024-03-18T20:26:38.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.318+0000 7f619dd53700 1 -- 172.21.15.1:0/3834023781 shutdown_connections 2024-03-18T20:26:38.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.318+0000 7f619dd53700 1 --2- 172.21.15.1:0/3834023781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f619806a750 0x7f619806ab30 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:38.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.318+0000 7f619dd53700 1 --2- 172.21.15.1:0/3834023781 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f619815ec80 0x7f619815f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:38.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.318+0000 7f619dd53700 1 -- 172.21.15.1:0/3834023781 >> 172.21.15.1:0/3834023781 conn(0x7f61980cc9b0 msgr2=0x7f61980ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:38.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.318+0000 7f619dd53700 1 -- 172.21.15.1:0/3834023781 shutdown_connections 2024-03-18T20:26:38.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.318+0000 7f619dd53700 1 -- 172.21.15.1:0/3834023781 wait complete. 2024-03-18T20:26:38.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.319+0000 7f619dd53700 1 Processor -- start 2024-03-18T20:26:38.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.319+0000 7f619dd53700 1 -- start start 2024-03-18T20:26:38.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.320+0000 7f619dd53700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f619806a750 0x7f61980b4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:38.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.320+0000 7f619dd53700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61980ace80 0x7f6198158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:38.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.320+0000 7f619dd53700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f619815ec80 0x7f61980cf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:38.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.320+0000 7f619dd53700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61980c8e50 con 0x7f619806a750 2024-03-18T20:26:38.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.320+0000 7f619dd53700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6198158970 con 0x7f619815ec80 2024-03-18T20:26:38.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.320+0000 7f619dd53700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61980c81d0 con 0x7f61980ace80 2024-03-18T20:26:38.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.320+0000 7f61977fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f619806a750 0x7f61980b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:38.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.320+0000 7f61977fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f619806a750 0x7f61980b4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35806/0 (socket says 172.21.15.1:35806) 2024-03-18T20:26:38.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.320+0000 7f61977fe700 1 -- 172.21.15.1:0/598419183 learned_addr learned my addr 172.21.15.1:0/598419183 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:38.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.320+0000 7f6197fff700 1 --2- 172.21.15.1:0/598419183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f619815ec80 0x7f61980cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:38.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.320+0000 7f6196ffd700 1 --2- 172.21.15.1:0/598419183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61980ace80 0x7f6198158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:38.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.321+0000 7f61977fe700 1 -- 172.21.15.1:0/598419183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61980ace80 msgr2=0x7f6198158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:38.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.321+0000 7f61977fe700 1 --2- 172.21.15.1:0/598419183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61980ace80 0x7f6198158430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:38.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.321+0000 7f61977fe700 1 -- 172.21.15.1:0/598419183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f619815ec80 msgr2=0x7f61980cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:38.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.321+0000 7f61977fe700 1 --2- 172.21.15.1:0/598419183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f619815ec80 0x7f61980cf400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:38.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.321+0000 7f61977fe700 1 -- 172.21.15.1:0/598419183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6190009a70 con 0x7f619806a750 2024-03-18T20:26:38.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.321+0000 7f6197fff700 1 --2- 172.21.15.1:0/598419183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f619815ec80 0x7f61980cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:38.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.321+0000 7f61977fe700 1 --2- 172.21.15.1:0/598419183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f619806a750 0x7f61980b4530 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7f6180009b70 tx=0x7f6180004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:38.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.321+0000 7f618ffff700 1 -- 172.21.15.1:0/598419183 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61800065b0 con 0x7f619806a750 2024-03-18T20:26:38.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.321+0000 7f618ffff700 1 -- 172.21.15.1:0/598419183 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6180006730 con 0x7f619806a750 2024-03-18T20:26:38.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.321+0000 7f619dd53700 1 -- 172.21.15.1:0/598419183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6180009430 con 0x7f619806a750 2024-03-18T20:26:38.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.321+0000 7f618ffff700 1 -- 172.21.15.1:0/598419183 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61800036a0 con 0x7f619806a750 2024-03-18T20:26:38.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.321+0000 7f619dd53700 1 -- 172.21.15.1:0/598419183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61980c8790 con 0x7f619806a750 2024-03-18T20:26:38.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.322+0000 7f619dd53700 1 -- 172.21.15.1:0/598419183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61981f2f70 con 0x7f619806a750 2024-03-18T20:26:38.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.323+0000 7f618ffff700 1 -- 172.21.15.1:0/598419183 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f6180022070 con 0x7f619806a750 2024-03-18T20:26:38.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.323+0000 7f618ffff700 1 --2- 172.21.15.1:0/598419183 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f617806aed0 0x7f617806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:38.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.323+0000 7f618ffff700 1 -- 172.21.15.1:0/598419183 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 6502+0+0 (secure 0 0 0) 0x7f618008b1a0 con 0x7f619806a750 2024-03-18T20:26:38.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.323+0000 7f6196ffd700 1 --2- 172.21.15.1:0/598419183 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f617806aed0 0x7f617806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:38.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.323+0000 7f6196ffd700 1 --2- 172.21.15.1:0/598419183 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f617806aed0 0x7f617806d390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6188005910 tx=0x7f6188009040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:38.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.328+0000 7f618ffff700 1 -- 172.21.15.1:0/598419183 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6180055d90 con 0x7f619806a750 2024-03-18T20:26:38.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:38.530+0000 7f619dd53700 1 -- 172.21.15.1:0/598419183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f61980c8f90 con 0x7f619806a750 2024-03-18T20:26:39.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.163+0000 7f618ffff700 1 -- 172.21.15.1:0/598419183 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v255) v1 ==== 176+0+0 (secure 0 0 0) 0x7f61800593b0 con 0x7f619806a750 2024-03-18T20:26:39.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.168+0000 7f619dd53700 1 -- 172.21.15.1:0/598419183 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f617806aed0 msgr2=0x7f617806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:39.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.168+0000 7f619dd53700 1 --2- 172.21.15.1:0/598419183 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f617806aed0 0x7f617806d390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6188005910 tx=0x7f6188009040 comp rx=0 tx=0).stop 2024-03-18T20:26:39.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.168+0000 7f619dd53700 1 -- 172.21.15.1:0/598419183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f619806a750 msgr2=0x7f61980b4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:39.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.168+0000 7f619dd53700 1 --2- 172.21.15.1:0/598419183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f619806a750 0x7f61980b4530 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7f6180009b70 tx=0x7f6180004750 comp rx=0 tx=0).stop 2024-03-18T20:26:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.170+0000 7f619dd53700 1 -- 172.21.15.1:0/598419183 shutdown_connections 2024-03-18T20:26:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.170+0000 7f619dd53700 1 --2- 172.21.15.1:0/598419183 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f617806aed0 0x7f617806d390 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.170+0000 7f619dd53700 1 --2- 172.21.15.1:0/598419183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f619815ec80 0x7f61980cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.170+0000 7f619dd53700 1 --2- 172.21.15.1:0/598419183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f619806a750 0x7f61980b4530 unknown :-1 s=CLOSED pgs=2853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.170+0000 7f619dd53700 1 --2- 172.21.15.1:0/598419183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f61980ace80 0x7f6198158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.170+0000 7f619dd53700 1 -- 172.21.15.1:0/598419183 >> 172.21.15.1:0/598419183 conn(0x7f61980cc9b0 msgr2=0x7f619806f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:39.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.177+0000 7f619dd53700 1 -- 172.21.15.1:0/598419183 shutdown_connections 2024-03-18T20:26:39.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.181+0000 7f619dd53700 1 -- 172.21.15.1:0/598419183 wait complete. 2024-03-18T20:26:39.191 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'backup_fs_data' created 2024-03-18T20:26:39.203 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2024-03-18T20:26:39.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.333+0000 7f30fd86e700 1 Processor -- start 2024-03-18T20:26:39.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.333+0000 7f30fd86e700 1 -- start start 2024-03-18T20:26:39.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.333+0000 7f30fd86e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f30f815d6f0 0x7f30f8099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:39.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.333+0000 7f30fd86e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f806ea40 con 0x7f30f806e530 2024-03-18T20:26:39.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.333+0000 7f30fd86e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f809a360 con 0x7f30f8099a40 2024-03-18T20:26:39.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.333+0000 7f30fd86e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30f806ece0 con 0x7f30f815d6f0 2024-03-18T20:26:39.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.334+0000 7f30f67fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f30f815d6f0 0x7f30f8099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:39.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.334+0000 7f30f67fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f30f815d6f0 0x7f30f8099e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35812/0 (socket says 172.21.15.1:35812) 2024-03-18T20:26:39.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.334+0000 7f30f77fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f30f806e530 0x7f30f80cf2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56324/0 (socket says 172.21.15.1:56324) 2024-03-18T20:26:39.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.334+0000 7f30f67fc700 1 -- 172.21.15.1:0/2838386758 learned_addr learned my addr 172.21.15.1:0/2838386758 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.334+0000 7f30f67fc700 1 -- 172.21.15.1:0/2838386758 >> v1:172.21.15.165:6789/0 conn(0x7f30f8099a40 legacy=0x7f30f815e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.334+0000 7f30f67fc700 1 -- 172.21.15.1:0/2838386758 >> v1:172.21.15.1:6789/0 conn(0x7f30f806e530 legacy=0x7f30f80cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.334+0000 7f30f67fc700 1 -- 172.21.15.1:0/2838386758 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f806efc0 con 0x7f30f815d6f0 2024-03-18T20:26:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.334+0000 7f30f67fc700 1 --2- 172.21.15.1:0/2838386758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f815d6f0 0x7f30f8099e20 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7f30ec0098f0 tx=0x7f30ec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb569e32ba3dca4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.334+0000 7f30f5ffb700 1 -- 172.21.15.1:0/2838386758 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec004230 con 0x7f30f815d6f0 2024-03-18T20:26:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.334+0000 7f30f5ffb700 1 -- 172.21.15.1:0/2838386758 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ec004390 con 0x7f30f815d6f0 2024-03-18T20:26:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.334+0000 7f30f5ffb700 1 -- 172.21.15.1:0/2838386758 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec0036a0 con 0x7f30f815d6f0 2024-03-18T20:26:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.335+0000 7f30fd86e700 1 -- 172.21.15.1:0/2838386758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f815d6f0 msgr2=0x7f30f8099e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.335+0000 7f30fd86e700 1 --2- 172.21.15.1:0/2838386758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f815d6f0 0x7f30f8099e20 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7f30ec0098f0 tx=0x7f30ec009c00 comp rx=0 tx=0).stop 2024-03-18T20:26:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.335+0000 7f30fd86e700 1 -- 172.21.15.1:0/2838386758 shutdown_connections 2024-03-18T20:26:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.335+0000 7f30fd86e700 1 --2- 172.21.15.1:0/2838386758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f815d6f0 0x7f30f8099e20 unknown :-1 s=CLOSED pgs=2855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.335+0000 7f30fd86e700 1 -- 172.21.15.1:0/2838386758 >> 172.21.15.1:0/2838386758 conn(0x7f30f80b2af0 msgr2=0x7f30f80b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.335+0000 7f30fd86e700 1 -- 172.21.15.1:0/2838386758 shutdown_connections 2024-03-18T20:26:39.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.335+0000 7f30fd86e700 1 -- 172.21.15.1:0/2838386758 wait complete. 2024-03-18T20:26:39.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.336+0000 7f30fd86e700 1 Processor -- start 2024-03-18T20:26:39.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.336+0000 7f30fd86e700 1 -- start start 2024-03-18T20:26:39.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.337+0000 7f30fd86e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f806e530 0x7f30f81f3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:39.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.337+0000 7f30fd86e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30f8099a40 0x7f30f8160e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:39.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.337+0000 7f30fd86e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30f80b0d40 0x7f30f815ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:39.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.337+0000 7f30fd86e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f81613c0 con 0x7f30f806e530 2024-03-18T20:26:39.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.337+0000 7f30fd86e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f81f3770 con 0x7f30f80b0d40 2024-03-18T20:26:39.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.337+0000 7f30fd86e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f81f3a50 con 0x7f30f8099a40 2024-03-18T20:26:39.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.337+0000 7f30f6ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f806e530 0x7f30f81f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:39.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.337+0000 7f30f6ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f806e530 0x7f30f81f3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35824/0 (socket says 172.21.15.1:35824) 2024-03-18T20:26:39.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.337+0000 7f30f6ffd700 1 -- 172.21.15.1:0/460058208 learned_addr learned my addr 172.21.15.1:0/460058208 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:39.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.337+0000 7f30f67fc700 1 --2- 172.21.15.1:0/460058208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30f8099a40 0x7f30f8160e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:39.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.337+0000 7f30f77fe700 1 --2- 172.21.15.1:0/460058208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30f80b0d40 0x7f30f815ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:39.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.337+0000 7f30f6ffd700 1 -- 172.21.15.1:0/460058208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30f8099a40 msgr2=0x7f30f8160e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:39.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.337+0000 7f30f6ffd700 1 --2- 172.21.15.1:0/460058208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30f8099a40 0x7f30f8160e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.337+0000 7f30f6ffd700 1 -- 172.21.15.1:0/460058208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30f80b0d40 msgr2=0x7f30f815ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:39.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.337+0000 7f30f6ffd700 1 --2- 172.21.15.1:0/460058208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30f80b0d40 0x7f30f815ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.337+0000 7f30f6ffd700 1 -- 172.21.15.1:0/460058208 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ec009550 con 0x7f30f806e530 2024-03-18T20:26:39.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.337+0000 7f30f6ffd700 1 --2- 172.21.15.1:0/460058208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f806e530 0x7f30f81f3120 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7f30e800c4d0 tx=0x7f30e800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:39.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.338+0000 7f30dffff700 1 -- 172.21.15.1:0/460058208 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e8011790 con 0x7f30f806e530 2024-03-18T20:26:39.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.338+0000 7f30dffff700 1 -- 172.21.15.1:0/460058208 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e80118f0 con 0x7f30f806e530 2024-03-18T20:26:39.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.338+0000 7f30fd86e700 1 -- 172.21.15.1:0/460058208 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f81f3d30 con 0x7f30f806e530 2024-03-18T20:26:39.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.338+0000 7f30dffff700 1 -- 172.21.15.1:0/460058208 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e8011ca0 con 0x7f30f806e530 2024-03-18T20:26:39.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.338+0000 7f30fd86e700 1 -- 172.21.15.1:0/460058208 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f81f67b0 con 0x7f30f806e530 2024-03-18T20:26:39.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.339+0000 7f30fd86e700 1 -- 172.21.15.1:0/460058208 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f8160b00 con 0x7f30f806e530 2024-03-18T20:26:39.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.339+0000 7f30dffff700 1 -- 172.21.15.1:0/460058208 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f30e801c460 con 0x7f30f806e530 2024-03-18T20:26:39.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.340+0000 7f30dffff700 1 --2- 172.21.15.1:0/460058208 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30e406aed0 0x7f30e406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:39.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.340+0000 7f30dffff700 1 -- 172.21.15.1:0/460058208 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 6887+0+0 (secure 0 0 0) 0x7f30e8089810 con 0x7f30f806e530 2024-03-18T20:26:39.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.340+0000 7f30f67fc700 1 --2- 172.21.15.1:0/460058208 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30e406aed0 0x7f30e406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:39.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.340+0000 7f30f67fc700 1 --2- 172.21.15.1:0/460058208 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30e406aed0 0x7f30e406d390 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f30ec009130 tx=0x7f30ec008030 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:39.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.344+0000 7f30dffff700 1 -- 172.21.15.1:0/460058208 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f30e80542c0 con 0x7f30f806e530 2024-03-18T20:26:39.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.594+0000 7f30fd86e700 1 -- 172.21.15.1:0/460058208 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7f30f8073350 con 0x7f30f806e530 2024-03-18T20:26:39.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.599+0000 7f30dffff700 1 -- 172.21.15.1:0/460058208 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 33 and data pool 34 v122) v1 ==== 183+0+0 (secure 0 0 0) 0x7f30e80578e0 con 0x7f30f806e530 2024-03-18T20:26:39.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.602+0000 7f30fd86e700 1 -- 172.21.15.1:0/460058208 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30e406aed0 msgr2=0x7f30e406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:39.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.602+0000 7f30fd86e700 1 --2- 172.21.15.1:0/460058208 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30e406aed0 0x7f30e406d390 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f30ec009130 tx=0x7f30ec008030 comp rx=0 tx=0).stop 2024-03-18T20:26:39.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.603+0000 7f30fd86e700 1 -- 172.21.15.1:0/460058208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f806e530 msgr2=0x7f30f81f3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:39.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.603+0000 7f30fd86e700 1 --2- 172.21.15.1:0/460058208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f806e530 0x7f30f81f3120 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7f30e800c4d0 tx=0x7f30e800c7e0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.603+0000 7f30fd86e700 1 -- 172.21.15.1:0/460058208 shutdown_connections 2024-03-18T20:26:39.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.603+0000 7f30fd86e700 1 --2- 172.21.15.1:0/460058208 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30e406aed0 0x7f30e406d390 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.603+0000 7f30fd86e700 1 --2- 172.21.15.1:0/460058208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30f80b0d40 0x7f30f815ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.603+0000 7f30fd86e700 1 --2- 172.21.15.1:0/460058208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30f806e530 0x7f30f81f3120 unknown :-1 s=CLOSED pgs=2856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.603+0000 7f30fd86e700 1 --2- 172.21.15.1:0/460058208 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30f8099a40 0x7f30f8160e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.603+0000 7f30fd86e700 1 -- 172.21.15.1:0/460058208 >> 172.21.15.1:0/460058208 conn(0x7f30f80b2af0 msgr2=0x7f30f815a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:39.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.603+0000 7f30fd86e700 1 -- 172.21.15.1:0/460058208 shutdown_connections 2024-03-18T20:26:39.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.603+0000 7f30fd86e700 1 -- 172.21.15.1:0/460058208 wait complete. 2024-03-18T20:26:39.609 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 33 and data pool 34 2024-03-18T20:26:39.630 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:26:39.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.754+0000 7fadc8162700 1 Processor -- start 2024-03-18T20:26:39.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.754+0000 7fadc8162700 1 -- start start 2024-03-18T20:26:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.755+0000 7fadc8162700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fadc006a750 0x7fadc006ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.755+0000 7fadc8162700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fadc015d6b0 0x7fadc0160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.755+0000 7fadc8162700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fadc00ad790 0x7fadc0161530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.755+0000 7fadc8162700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadc00af7a0 con 0x7fadc00ad790 2024-03-18T20:26:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.755+0000 7fadc8162700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadc00acfb0 con 0x7fadc015d6b0 2024-03-18T20:26:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.755+0000 7fadc8162700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc006b070 con 0x7fadc006a750 2024-03-18T20:26:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.755+0000 7fadc66ff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fadc00ad790 0x7fadc0161530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.755+0000 7fadc66ff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fadc00ad790 0x7fadc0161530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35838/0 (socket says 172.21.15.1:35838) 2024-03-18T20:26:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.755+0000 7fadc66ff700 1 -- 172.21.15.1:0/1952248348 learned_addr learned my addr 172.21.15.1:0/1952248348 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.755+0000 7fadc56fd700 1 --2- 172.21.15.1:0/1952248348 >> v2:172.21.15.165:3300/0 conn(0x7fadc015d6b0 0x7fadc0160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:39.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.755+0000 7fadc5efe700 1 --2- 172.21.15.1:0/1952248348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fadc006a750 0x7fadc006ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:39.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.755+0000 7fadc66ff700 1 -- 172.21.15.1:0/1952248348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fadc006a750 msgr2=0x7fadc006ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:39.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.755+0000 7fadc66ff700 1 --2- 172.21.15.1:0/1952248348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fadc006a750 0x7fadc006ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.755+0000 7fadc66ff700 1 -- 172.21.15.1:0/1952248348 >> v2:172.21.15.165:3300/0 conn(0x7fadc015d6b0 msgr2=0x7fadc0160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:39.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.755+0000 7fadc66ff700 1 --2- 172.21.15.1:0/1952248348 >> v2:172.21.15.165:3300/0 conn(0x7fadc015d6b0 0x7fadc0160ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.755+0000 7fadc66ff700 1 -- 172.21.15.1:0/1952248348 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc00b50a0 con 0x7fadc00ad790 2024-03-18T20:26:39.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.755+0000 7fadc5efe700 1 --2- 172.21.15.1:0/1952248348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fadc006a750 0x7fadc006ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:39.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.755+0000 7fadc66ff700 1 --2- 172.21.15.1:0/1952248348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc00ad790 0x7fadc0161530 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7fadc0067040 tx=0x7fadb800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cad37062d3c0383 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:39.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.756+0000 7fadc4efc700 1 -- 172.21.15.1:0/1952248348 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb800e070 con 0x7fadc00ad790 2024-03-18T20:26:39.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.756+0000 7fadc4efc700 1 -- 172.21.15.1:0/1952248348 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb800b910 con 0x7fadc00ad790 2024-03-18T20:26:39.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.756+0000 7fadc4efc700 1 -- 172.21.15.1:0/1952248348 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb800bcc0 con 0x7fadc00ad790 2024-03-18T20:26:39.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.756+0000 7fadc8162700 1 -- 172.21.15.1:0/1952248348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc00ad790 msgr2=0x7fadc0161530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:39.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.756+0000 7fadc8162700 1 --2- 172.21.15.1:0/1952248348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc00ad790 0x7fadc0161530 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7fadc0067040 tx=0x7fadb800b330 comp rx=0 tx=0).stop 2024-03-18T20:26:39.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.756+0000 7fadc8162700 1 -- 172.21.15.1:0/1952248348 shutdown_connections 2024-03-18T20:26:39.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.756+0000 7fadc8162700 1 --2- 172.21.15.1:0/1952248348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc00ad790 0x7fadc0161530 unknown :-1 s=CLOSED pgs=2857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.756+0000 7fadc8162700 1 --2- 172.21.15.1:0/1952248348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fadc006a750 0x7fadc006ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.756+0000 7fadc8162700 1 --2- 172.21.15.1:0/1952248348 >> v2:172.21.15.165:3300/0 conn(0x7fadc015d6b0 0x7fadc0160ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.756+0000 7fadc8162700 1 -- 172.21.15.1:0/1952248348 >> 172.21.15.1:0/1952248348 conn(0x7fadc006b860 msgr2=0x7fadc006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:39.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.756+0000 7fadc8162700 1 -- 172.21.15.1:0/1952248348 shutdown_connections 2024-03-18T20:26:39.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.756+0000 7fadc8162700 1 -- 172.21.15.1:0/1952248348 wait complete. 2024-03-18T20:26:39.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.758+0000 7fadc8162700 1 Processor -- start 2024-03-18T20:26:39.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.758+0000 7fadc8162700 1 -- start start 2024-03-18T20:26:39.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.759+0000 7fadc8162700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fadc006a750 0x7fadc01605a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:39.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.759+0000 7fadc8162700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc00ad790 0x7fadc0153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:39.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.759+0000 7fadc8162700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fadc015d6b0 0x7fadc015f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:39.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.759+0000 7fadc8162700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc0153320 con 0x7fadc00ad790 2024-03-18T20:26:39.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.759+0000 7fadc8162700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc0154880 con 0x7fadc006a750 2024-03-18T20:26:39.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.759+0000 7fadc8162700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc0153c70 con 0x7fadc015d6b0 2024-03-18T20:26:39.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.759+0000 7fadc56fd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc00ad790 0x7fadc0153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:39.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.759+0000 7fadc56fd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc00ad790 0x7fadc0153570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35854/0 (socket says 172.21.15.1:35854) 2024-03-18T20:26:39.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.759+0000 7fadc56fd700 1 -- 172.21.15.1:0/3615509500 learned_addr learned my addr 172.21.15.1:0/3615509500 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:39.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.759+0000 7fadc66ff700 1 --2- 172.21.15.1:0/3615509500 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fadc015d6b0 0x7fadc015f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:39.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.759+0000 7fadc5efe700 1 --2- 172.21.15.1:0/3615509500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fadc006a750 0x7fadc01605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:39.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.759+0000 7fadc56fd700 1 -- 172.21.15.1:0/3615509500 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fadc015d6b0 msgr2=0x7fadc015f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:39.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.759+0000 7fadc56fd700 1 --2- 172.21.15.1:0/3615509500 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fadc015d6b0 0x7fadc015f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.759+0000 7fadc56fd700 1 -- 172.21.15.1:0/3615509500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fadc006a750 msgr2=0x7fadc01605a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:39.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.759+0000 7fadc56fd700 1 --2- 172.21.15.1:0/3615509500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fadc006a750 0x7fadc01605a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.759+0000 7fadc56fd700 1 -- 172.21.15.1:0/3615509500 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb8009b30 con 0x7fadc00ad790 2024-03-18T20:26:39.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.759+0000 7fadc56fd700 1 --2- 172.21.15.1:0/3615509500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc00ad790 0x7fadc0153570 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7fadb000e870 tx=0x7fadb000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:39.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.759+0000 7fadb67fc700 1 -- 172.21.15.1:0/3615509500 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb0009590 con 0x7fadc00ad790 2024-03-18T20:26:39.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.760+0000 7fadb67fc700 1 -- 172.21.15.1:0/3615509500 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb000ee70 con 0x7fadc00ad790 2024-03-18T20:26:39.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.760+0000 7fadc8162700 1 -- 172.21.15.1:0/3615509500 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadc0153f50 con 0x7fadc00ad790 2024-03-18T20:26:39.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.760+0000 7fadb67fc700 1 -- 172.21.15.1:0/3615509500 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb000ca50 con 0x7fadc00ad790 2024-03-18T20:26:39.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.760+0000 7fadc8162700 1 -- 172.21.15.1:0/3615509500 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadc0154510 con 0x7fadc00ad790 2024-03-18T20:26:39.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.761+0000 7fadc8162700 1 -- 172.21.15.1:0/3615509500 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadc01549f0 con 0x7fadc00ad790 2024-03-18T20:26:39.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.761+0000 7fadb67fc700 1 -- 172.21.15.1:0/3615509500 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fadb000cbb0 con 0x7fadc00ad790 2024-03-18T20:26:39.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.761+0000 7fadb67fc700 1 --2- 172.21.15.1:0/3615509500 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fada406af00 0x7fada406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:39.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.761+0000 7fadb67fc700 1 -- 172.21.15.1:0/3615509500 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6993+0+0 (secure 0 0 0) 0x7fadb0089c60 con 0x7fadc00ad790 2024-03-18T20:26:39.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.762+0000 7fadc5efe700 1 --2- 172.21.15.1:0/3615509500 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fada406af00 0x7fada406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:39.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.762+0000 7fadc5efe700 1 --2- 172.21.15.1:0/3615509500 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fada406af00 0x7fada406d3c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0038f0 tx=0x7fadbc008040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:39.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.766+0000 7fadb67fc700 1 -- 172.21.15.1:0/3615509500 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fadb0054530 con 0x7fadc00ad790 2024-03-18T20:26:39.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.974+0000 7fadc8162700 1 -- 172.21.15.1:0/3615509500 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fadc00b4c60 con 0x7fadc00ad790 2024-03-18T20:26:39.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.975+0000 7fadb67fc700 1 -- 172.21.15.1:0/3615509500 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v256) v1 ==== 74+0+20102 (secure 0 0 0) 0x7fadb000ce60 con 0x7fadc00ad790 2024-03-18T20:26:39.979 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:39.980 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":256,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:26:39.595868+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:26:23.615691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-18T20:26:24.759650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:26:37.326244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:26:38.564628+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:26:39.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.977+0000 7fadc8162700 1 -- 172.21.15.1:0/3615509500 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fada406af00 msgr2=0x7fada406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:39.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.977+0000 7fadc8162700 1 --2- 172.21.15.1:0/3615509500 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fada406af00 0x7fada406d3c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0038f0 tx=0x7fadbc008040 comp rx=0 tx=0).stop 2024-03-18T20:26:39.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.977+0000 7fadc8162700 1 -- 172.21.15.1:0/3615509500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc00ad790 msgr2=0x7fadc0153570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:39.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.977+0000 7fadc8162700 1 --2- 172.21.15.1:0/3615509500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc00ad790 0x7fadc0153570 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7fadb000e870 tx=0x7fadb000ec30 comp rx=0 tx=0).stop 2024-03-18T20:26:39.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.978+0000 7fadc8162700 1 -- 172.21.15.1:0/3615509500 shutdown_connections 2024-03-18T20:26:39.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.978+0000 7fadc8162700 1 --2- 172.21.15.1:0/3615509500 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fada406af00 0x7fada406d3c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.978+0000 7fadc8162700 1 --2- 172.21.15.1:0/3615509500 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fadc015d6b0 0x7fadc015f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.978+0000 7fadc8162700 1 --2- 172.21.15.1:0/3615509500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fadc006a750 0x7fadc01605a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.978+0000 7fadc8162700 1 --2- 172.21.15.1:0/3615509500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fadc00ad790 0x7fadc0153570 unknown :-1 s=CLOSED pgs=2858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:39.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.978+0000 7fadc8162700 1 -- 172.21.15.1:0/3615509500 >> 172.21.15.1:0/3615509500 conn(0x7fadc006b860 msgr2=0x7fadc00b0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:39.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.978+0000 7fadc8162700 1 -- 172.21.15.1:0/3615509500 shutdown_connections 2024-03-18T20:26:39.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:39.978+0000 7fadc8162700 1 -- 172.21.15.1:0/3615509500 wait complete. 2024-03-18T20:26:39.994 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:26:40.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.115+0000 7fe629926700 1 Processor -- start 2024-03-18T20:26:40.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.115+0000 7fe629926700 1 -- start start 2024-03-18T20:26:40.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.116+0000 7fe629926700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe624160940 0x7fe6240d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:40.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.116+0000 7fe629926700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6240cf090 0x7fe6240d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:40.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.116+0000 7fe629926700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6240cddc0 con 0x7fe624160d20 2024-03-18T20:26:40.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.116+0000 7fe629926700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6240cdff0 con 0x7fe624160940 2024-03-18T20:26:40.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.116+0000 7fe629926700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6240ce2d0 con 0x7fe6240cf090 2024-03-18T20:26:40.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.116+0000 7fe622ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe624160940 0x7fe6240d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:40.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.116+0000 7fe6237fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe624160d20 0x7fe624161230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56338/0 (socket says 172.21.15.1:56338) 2024-03-18T20:26:40.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.116+0000 7fe6237fe700 1 -- 172.21.15.1:0/4186477389 learned_addr learned my addr 172.21.15.1:0/4186477389 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:40.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.116+0000 7fe6227fc700 1 --2- 172.21.15.1:0/4186477389 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6240cf090 0x7fe6240d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:40.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.116+0000 7fe622ffd700 1 -- 172.21.15.1:0/4186477389 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6240cf090 msgr2=0x7fe6240d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:40.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.116+0000 7fe622ffd700 1 --2- 172.21.15.1:0/4186477389 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6240cf090 0x7fe6240d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:40.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.116+0000 7fe622ffd700 1 -- 172.21.15.1:0/4186477389 >> v1:172.21.15.1:6789/0 conn(0x7fe624160d20 legacy=0x7fe624161230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:40.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.116+0000 7fe622ffd700 1 -- 172.21.15.1:0/4186477389 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6240ce5b0 con 0x7fe624160940 2024-03-18T20:26:40.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.116+0000 7fe621ffb700 1 -- 172.21.15.1:0/4186477389 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541263178 0 0) 0x7fe6240cddc0 con 0x7fe624160d20 2024-03-18T20:26:40.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.116+0000 7fe622ffd700 1 --2- 172.21.15.1:0/4186477389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe624160940 0x7fe6240d0190 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7fe618009840 tx=0x7fe618009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca0a439da7c479 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:40.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.117+0000 7fe621ffb700 1 -- 172.21.15.1:0/4186477389 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe618011830 con 0x7fe624160940 2024-03-18T20:26:40.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.117+0000 7fe621ffb700 1 -- 172.21.15.1:0/4186477389 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe618011990 con 0x7fe624160940 2024-03-18T20:26:40.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.117+0000 7fe621ffb700 1 -- 172.21.15.1:0/4186477389 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe618011d20 con 0x7fe624160940 2024-03-18T20:26:40.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.117+0000 7fe629926700 1 -- 172.21.15.1:0/4186477389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe624160940 msgr2=0x7fe6240d0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:40.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.117+0000 7fe629926700 1 --2- 172.21.15.1:0/4186477389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe624160940 0x7fe6240d0190 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7fe618009840 tx=0x7fe618009b50 comp rx=0 tx=0).stop 2024-03-18T20:26:40.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.117+0000 7fe629926700 1 -- 172.21.15.1:0/4186477389 shutdown_connections 2024-03-18T20:26:40.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.117+0000 7fe629926700 1 --2- 172.21.15.1:0/4186477389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe624160940 0x7fe6240d0190 unknown :-1 s=CLOSED pgs=2860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:40.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.117+0000 7fe629926700 1 --2- 172.21.15.1:0/4186477389 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6240cf090 0x7fe6240d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:40.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.117+0000 7fe629926700 1 -- 172.21.15.1:0/4186477389 >> 172.21.15.1:0/4186477389 conn(0x7fe62409c520 msgr2=0x7fe6240d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:40.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.117+0000 7fe629926700 1 -- 172.21.15.1:0/4186477389 shutdown_connections 2024-03-18T20:26:40.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.117+0000 7fe629926700 1 -- 172.21.15.1:0/4186477389 wait complete. 2024-03-18T20:26:40.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.118+0000 7fe629926700 1 Processor -- start 2024-03-18T20:26:40.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.119+0000 7fe629926700 1 -- start start 2024-03-18T20:26:40.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.119+0000 7fe629926700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6240cf090 0x7fe6240ca280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:40.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.119+0000 7fe629926700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe624160940 0x7fe6241f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:40.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.119+0000 7fe629926700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe624160d20 0x7fe6241f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:40.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.119+0000 7fe629926700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe624152110 con 0x7fe624160d20 2024-03-18T20:26:40.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.119+0000 7fe629926700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6240b5f00 con 0x7fe624160940 2024-03-18T20:26:40.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.119+0000 7fe629926700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6240ca980 con 0x7fe6240cf090 2024-03-18T20:26:40.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.119+0000 7fe6237fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe624160d20 0x7fe6241f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:40.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.119+0000 7fe6237fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe624160d20 0x7fe6241f27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35874/0 (socket says 172.21.15.1:35874) 2024-03-18T20:26:40.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.119+0000 7fe6237fe700 1 -- 172.21.15.1:0/2337169163 learned_addr learned my addr 172.21.15.1:0/2337169163 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:40.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.119+0000 7fe622ffd700 1 --2- 172.21.15.1:0/2337169163 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6240cf090 0x7fe6240ca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:40.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.119+0000 7fe6227fc700 1 --2- 172.21.15.1:0/2337169163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe624160940 0x7fe6241f6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:40.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.120+0000 7fe6237fe700 1 -- 172.21.15.1:0/2337169163 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6240cf090 msgr2=0x7fe6240ca280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:40.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.120+0000 7fe6237fe700 1 --2- 172.21.15.1:0/2337169163 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6240cf090 0x7fe6240ca280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:40.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.120+0000 7fe6237fe700 1 -- 172.21.15.1:0/2337169163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe624160940 msgr2=0x7fe6241f6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:40.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.120+0000 7fe6237fe700 1 --2- 172.21.15.1:0/2337169163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe624160940 0x7fe6241f6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:40.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.120+0000 7fe6237fe700 1 -- 172.21.15.1:0/2337169163 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6180094a0 con 0x7fe624160d20 2024-03-18T20:26:40.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.120+0000 7fe6237fe700 1 --2- 172.21.15.1:0/2337169163 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe624160d20 0x7fe6241f27a0 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7fe61c00cba0 tx=0x7fe61c00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:40.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.120+0000 7fe60bfff700 1 -- 172.21.15.1:0/2337169163 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c00aaa0 con 0x7fe624160d20 2024-03-18T20:26:40.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.120+0000 7fe60bfff700 1 -- 172.21.15.1:0/2337169163 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61c00ac20 con 0x7fe624160d20 2024-03-18T20:26:40.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.120+0000 7fe629926700 1 -- 172.21.15.1:0/2337169163 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6240cac60 con 0x7fe624160d20 2024-03-18T20:26:40.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.120+0000 7fe60bfff700 1 -- 172.21.15.1:0/2337169163 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c0070d0 con 0x7fe624160d20 2024-03-18T20:26:40.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.120+0000 7fe629926700 1 -- 172.21.15.1:0/2337169163 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6240cb220 con 0x7fe624160d20 2024-03-18T20:26:40.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.121+0000 7fe629926700 1 -- 172.21.15.1:0/2337169163 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6240b60b0 con 0x7fe624160d20 2024-03-18T20:26:40.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.121+0000 7fe60bfff700 1 -- 172.21.15.1:0/2337169163 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fe61c007230 con 0x7fe624160d20 2024-03-18T20:26:40.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.122+0000 7fe60bfff700 1 --2- 172.21.15.1:0/2337169163 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe61006aed0 0x7fe61006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:40.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.122+0000 7fe60bfff700 1 -- 172.21.15.1:0/2337169163 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6993+0+0 (secure 0 0 0) 0x7fe61c089660 con 0x7fe624160d20 2024-03-18T20:26:40.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.122+0000 7fe622ffd700 1 --2- 172.21.15.1:0/2337169163 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe61006aed0 0x7fe61006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:40.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.122+0000 7fe622ffd700 1 --2- 172.21.15.1:0/2337169163 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe61006aed0 0x7fe61006d390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe6180090b0 tx=0x7fe618008030 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:40.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.126+0000 7fe60bfff700 1 -- 172.21.15.1:0/2337169163 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe6240b60b0 con 0x7fe624160d20 2024-03-18T20:26:40.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.333+0000 7fe629926700 1 -- 172.21.15.1:0/2337169163 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6240b6070 con 0x7fe624160d20 2024-03-18T20:26:40.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.335+0000 7fe60bfff700 1 -- 172.21.15.1:0/2337169163 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v256) v1 ==== 74+0+20102 (secure 0 0 0) 0x7fe61c018090 con 0x7fe624160d20 2024-03-18T20:26:40.339 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:40.339 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":256,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:26:39.595868+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:26:23.615691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-18T20:26:24.759650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:26:37.326244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:26:38.564628+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:26:40.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.337+0000 7fe629926700 1 -- 172.21.15.1:0/2337169163 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe61006aed0 msgr2=0x7fe61006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:40.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.337+0000 7fe629926700 1 --2- 172.21.15.1:0/2337169163 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe61006aed0 0x7fe61006d390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe6180090b0 tx=0x7fe618008030 comp rx=0 tx=0).stop 2024-03-18T20:26:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.337+0000 7fe629926700 1 -- 172.21.15.1:0/2337169163 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe624160d20 msgr2=0x7fe6241f27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.337+0000 7fe629926700 1 --2- 172.21.15.1:0/2337169163 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe624160d20 0x7fe6241f27a0 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7fe61c00cba0 tx=0x7fe61c00ceb0 comp rx=0 tx=0).stop 2024-03-18T20:26:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.337+0000 7fe629926700 1 -- 172.21.15.1:0/2337169163 shutdown_connections 2024-03-18T20:26:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.337+0000 7fe629926700 1 --2- 172.21.15.1:0/2337169163 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe61006aed0 0x7fe61006d390 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.337+0000 7fe629926700 1 --2- 172.21.15.1:0/2337169163 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe6240cf090 0x7fe6240ca280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.337+0000 7fe629926700 1 --2- 172.21.15.1:0/2337169163 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe624160d20 0x7fe6241f27a0 unknown :-1 s=CLOSED pgs=2861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.337+0000 7fe629926700 1 --2- 172.21.15.1:0/2337169163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe624160940 0x7fe6241f6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.337+0000 7fe629926700 1 -- 172.21.15.1:0/2337169163 >> 172.21.15.1:0/2337169163 conn(0x7fe62409c520 msgr2=0x7fe6240b1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.337+0000 7fe629926700 1 -- 172.21.15.1:0/2337169163 shutdown_connections 2024-03-18T20:26:40.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.337+0000 7fe629926700 1 -- 172.21.15.1:0/2337169163 wait complete. 2024-03-18T20:26:40.354 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2024-03-18T20:26:40.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.478+0000 7fbbc8c50700 1 Processor -- start 2024-03-18T20:26:40.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.478+0000 7fbbc8c50700 1 -- start start 2024-03-18T20:26:40.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.479+0000 7fbbc8c50700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbbc406f180 0x7fbbc4161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:40.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.479+0000 7fbbc8c50700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbbc406a750 0x7fbbc406aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:40.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.479+0000 7fbbc8c50700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc406b0e0 con 0x7fbbc406eda0 2024-03-18T20:26:40.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.479+0000 7fbbc8c50700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc40cb320 con 0x7fbbc406f180 2024-03-18T20:26:40.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.479+0000 7fbbc8c50700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc40ce010 con 0x7fbbc406a750 2024-03-18T20:26:40.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.479+0000 7fbbc1d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbbc406f180 0x7fbbc4161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:40.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.479+0000 7fbbc259c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbbc406eda0 0x7fbbc40ca580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56354/0 (socket says 172.21.15.1:56354) 2024-03-18T20:26:40.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.479+0000 7fbbc259c700 1 -- 172.21.15.1:0/3673753068 learned_addr learned my addr 172.21.15.1:0/3673753068 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:40.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.479+0000 7fbbc2d9d700 1 --2- 172.21.15.1:0/3673753068 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbbc406a750 0x7fbbc406aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:40.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.479+0000 7fbbc1d9b700 1 -- 172.21.15.1:0/3673753068 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbbc406a750 msgr2=0x7fbbc406aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:40.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.479+0000 7fbbc1d9b700 1 --2- 172.21.15.1:0/3673753068 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbbc406a750 0x7fbbc406aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:40.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.479+0000 7fbbc1d9b700 1 -- 172.21.15.1:0/3673753068 >> v1:172.21.15.1:6789/0 conn(0x7fbbc406eda0 legacy=0x7fbbc40ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:40.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.479+0000 7fbbc1d9b700 1 -- 172.21.15.1:0/3673753068 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc40cad60 con 0x7fbbc406f180 2024-03-18T20:26:40.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.479+0000 7fbbc159a700 1 -- 172.21.15.1:0/3673753068 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182053862 0 0) 0x7fbbb4001110 con 0x7fbbc406eda0 2024-03-18T20:26:40.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.479+0000 7fbbc1d9b700 1 --2- 172.21.15.1:0/3673753068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbc406f180 0x7fbbc4161c60 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8009840 tx=0x7fbbb8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6411192aacd3b4ac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:40.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.479+0000 7fbbc159a700 1 -- 172.21.15.1:0/3673753068 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8011830 con 0x7fbbc406f180 2024-03-18T20:26:40.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.479+0000 7fbbc159a700 1 -- 172.21.15.1:0/3673753068 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb8011990 con 0x7fbbc406f180 2024-03-18T20:26:40.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.479+0000 7fbbc159a700 1 -- 172.21.15.1:0/3673753068 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8011d20 con 0x7fbbc406f180 2024-03-18T20:26:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.480+0000 7fbbc8c50700 1 -- 172.21.15.1:0/3673753068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbc406f180 msgr2=0x7fbbc4161c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.480+0000 7fbbc8c50700 1 --2- 172.21.15.1:0/3673753068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbc406f180 0x7fbbc4161c60 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8009840 tx=0x7fbbb8009b50 comp rx=0 tx=0).stop 2024-03-18T20:26:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.480+0000 7fbbc8c50700 1 -- 172.21.15.1:0/3673753068 shutdown_connections 2024-03-18T20:26:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.480+0000 7fbbc8c50700 1 --2- 172.21.15.1:0/3673753068 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbbc406a750 0x7fbbc406aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.480+0000 7fbbc8c50700 1 --2- 172.21.15.1:0/3673753068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbc406f180 0x7fbbc4161c60 unknown :-1 s=CLOSED pgs=2863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.480+0000 7fbbc8c50700 1 -- 172.21.15.1:0/3673753068 >> 172.21.15.1:0/3673753068 conn(0x7fbbc40cff30 msgr2=0x7fbbc40d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.480+0000 7fbbc8c50700 1 -- 172.21.15.1:0/3673753068 shutdown_connections 2024-03-18T20:26:40.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.480+0000 7fbbc8c50700 1 -- 172.21.15.1:0/3673753068 wait complete. 2024-03-18T20:26:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.481+0000 7fbbc8c50700 1 Processor -- start 2024-03-18T20:26:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.481+0000 7fbbc8c50700 1 -- start start 2024-03-18T20:26:40.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.482+0000 7fbbc8c50700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbc406a750 0x7fbbc415cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.482+0000 7fbbc8c50700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbbc406eda0 0x7fbbc40acff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.482+0000 7fbbc8c50700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbc406f180 0x7fbbc4156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.482+0000 7fbbc8c50700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc415d4e0 con 0x7fbbc406a750 2024-03-18T20:26:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.482+0000 7fbbc8c50700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc415dfc0 con 0x7fbbc406f180 2024-03-18T20:26:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.482+0000 7fbbc8c50700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc415d720 con 0x7fbbc406eda0 2024-03-18T20:26:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.482+0000 7fbbc259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbc406a750 0x7fbbc415cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.482+0000 7fbbc259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbc406a750 0x7fbbc415cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35884/0 (socket says 172.21.15.1:35884) 2024-03-18T20:26:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.482+0000 7fbbc259c700 1 -- 172.21.15.1:0/3267281824 learned_addr learned my addr 172.21.15.1:0/3267281824 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.482+0000 7fbbc1d9b700 1 --2- 172.21.15.1:0/3267281824 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbbc406eda0 0x7fbbc40acff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.482+0000 7fbbc2d9d700 1 --2- 172.21.15.1:0/3267281824 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbc406f180 0x7fbbc4156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.483+0000 7fbbc259c700 1 -- 172.21.15.1:0/3267281824 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbbc406eda0 msgr2=0x7fbbc40acff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.483+0000 7fbbc259c700 1 --2- 172.21.15.1:0/3267281824 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbbc406eda0 0x7fbbc40acff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.483+0000 7fbbc259c700 1 -- 172.21.15.1:0/3267281824 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbc406f180 msgr2=0x7fbbc4156250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:40.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.483+0000 7fbbc259c700 1 --2- 172.21.15.1:0/3267281824 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbc406f180 0x7fbbc4156250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:40.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.483+0000 7fbbc259c700 1 -- 172.21.15.1:0/3267281824 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb80094a0 con 0x7fbbc406a750 2024-03-18T20:26:40.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.483+0000 7fbbc259c700 1 --2- 172.21.15.1:0/3267281824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbc406a750 0x7fbbc415cfa0 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400c730 tx=0x7fbbb400ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:40.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.483+0000 7fbbb2ffd700 1 -- 172.21.15.1:0/3267281824 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4004c80 con 0x7fbbc406a750 2024-03-18T20:26:40.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.483+0000 7fbbb2ffd700 1 -- 172.21.15.1:0/3267281824 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb4004e00 con 0x7fbbc406a750 2024-03-18T20:26:40.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.483+0000 7fbbc8c50700 1 -- 172.21.15.1:0/3267281824 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc415da00 con 0x7fbbc406a750 2024-03-18T20:26:40.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.483+0000 7fbbb2ffd700 1 -- 172.21.15.1:0/3267281824 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4007310 con 0x7fbbc406a750 2024-03-18T20:26:40.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.483+0000 7fbbc8c50700 1 -- 172.21.15.1:0/3267281824 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc4157c90 con 0x7fbbc406a750 2024-03-18T20:26:40.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.484+0000 7fbbc8c50700 1 -- 172.21.15.1:0/3267281824 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc4157e80 con 0x7fbbc406a750 2024-03-18T20:26:40.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.485+0000 7fbbb2ffd700 1 -- 172.21.15.1:0/3267281824 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fbbb4007470 con 0x7fbbc406a750 2024-03-18T20:26:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.485+0000 7fbbb2ffd700 1 --2- 172.21.15.1:0/3267281824 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbba406af00 0x7fbba406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.485+0000 7fbbc1d9b700 1 --2- 172.21.15.1:0/3267281824 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbba406af00 0x7fbba406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.485+0000 7fbbb2ffd700 1 -- 172.21.15.1:0/3267281824 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 6993+0+0 (secure 0 0 0) 0x7fbbb4089740 con 0x7fbbc406a750 2024-03-18T20:26:40.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.486+0000 7fbbc1d9b700 1 --2- 172.21.15.1:0/3267281824 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbba406af00 0x7fbba406d3c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8009840 tx=0x7fbbb801c040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:40.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.490+0000 7fbbb2ffd700 1 -- 172.21.15.1:0/3267281824 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbbb4054010 con 0x7fbbc406a750 2024-03-18T20:26:40.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:40.741+0000 7fbbc8c50700 1 -- 172.21.15.1:0/3267281824 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fbbc406b0e0 con 0x7fbbc406a750 2024-03-18T20:26:41.165 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:26:41.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:26:41.342 INFO:tasks.ceph.mds.b.smithi165.stderr:2024-03-18T20:26:41.338+0000 7f6a4d30e700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T20:26:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.616+0000 7fbbb2ffd700 1 -- 172.21.15.1:0/3267281824 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v125) v1 ==== 121+0+0 (secure 0 0 0) 0x7fbbc406b0e0 con 0x7fbbc406a750 2024-03-18T20:26:41.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.619+0000 7fbbc8c50700 1 -- 172.21.15.1:0/3267281824 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbba406af00 msgr2=0x7fbba406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:41.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.619+0000 7fbbc8c50700 1 --2- 172.21.15.1:0/3267281824 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbba406af00 0x7fbba406d3c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8009840 tx=0x7fbbb801c040 comp rx=0 tx=0).stop 2024-03-18T20:26:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.619+0000 7fbbc8c50700 1 -- 172.21.15.1:0/3267281824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbc406a750 msgr2=0x7fbbc415cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.619+0000 7fbbc8c50700 1 --2- 172.21.15.1:0/3267281824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbc406a750 0x7fbbc415cfa0 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400c730 tx=0x7fbbb400ca40 comp rx=0 tx=0).stop 2024-03-18T20:26:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.619+0000 7fbbc8c50700 1 -- 172.21.15.1:0/3267281824 shutdown_connections 2024-03-18T20:26:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.619+0000 7fbbc8c50700 1 --2- 172.21.15.1:0/3267281824 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbba406af00 0x7fbba406d3c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.619+0000 7fbbc8c50700 1 --2- 172.21.15.1:0/3267281824 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbc406f180 0x7fbbc4156250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.619+0000 7fbbc8c50700 1 --2- 172.21.15.1:0/3267281824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbbc406a750 0x7fbbc415cfa0 unknown :-1 s=CLOSED pgs=2864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.619+0000 7fbbc8c50700 1 --2- 172.21.15.1:0/3267281824 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbbc406eda0 0x7fbbc40acff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.619+0000 7fbbc8c50700 1 -- 172.21.15.1:0/3267281824 >> 172.21.15.1:0/3267281824 conn(0x7fbbc40cff30 msgr2=0x7fbbc40714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.619+0000 7fbbc8c50700 1 -- 172.21.15.1:0/3267281824 shutdown_connections 2024-03-18T20:26:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.619+0000 7fbbc8c50700 1 -- 172.21.15.1:0/3267281824 wait complete. 2024-03-18T20:26:41.638 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-18T20:26:41.639 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2024-03-18T20:26:41.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.772+0000 7f71de566700 1 Processor -- start 2024-03-18T20:26:41.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.772+0000 7f71de566700 1 -- start start 2024-03-18T20:26:41.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.773+0000 7f71de566700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f71d815d5a0 0x7f71d80ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:41.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.773+0000 7f71de566700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f71d806ae60 0x7f71d8167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:41.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.773+0000 7f71de566700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d80cf600 con 0x7f71d80ce550 2024-03-18T20:26:41.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.773+0000 7f71de566700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d80ce9f0 con 0x7f71d815d5a0 2024-03-18T20:26:41.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.773+0000 7f71de566700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d80cecd0 con 0x7f71d806ae60 2024-03-18T20:26:41.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.773+0000 7f71d7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f71d815d5a0 0x7f71d80ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:41.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.773+0000 7f71d7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f71d815d5a0 0x7f71d80ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35896/0 (socket says 172.21.15.1:35896) 2024-03-18T20:26:41.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.773+0000 7f71d7fff700 1 -- 172.21.15.1:0/2236440268 learned_addr learned my addr 172.21.15.1:0/2236440268 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:41.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.773+0000 7f71dcb03700 1 --2- 172.21.15.1:0/2236440268 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f71d806ae60 0x7f71d8167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:41.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.773+0000 7f71d7fff700 1 -- 172.21.15.1:0/2236440268 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f71d806ae60 msgr2=0x7f71d8167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:41.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.773+0000 7f71d7fff700 1 --2- 172.21.15.1:0/2236440268 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f71d806ae60 0x7f71d8167140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:41.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.773+0000 7f71d7fff700 1 -- 172.21.15.1:0/2236440268 >> v1:172.21.15.165:6789/0 conn(0x7f71d80ce550 legacy=0x7f71d806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:41.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.773+0000 7f71d7fff700 1 -- 172.21.15.1:0/2236440268 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d80cefb0 con 0x7f71d815d5a0 2024-03-18T20:26:41.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.773+0000 7f71d7fff700 1 --2- 172.21.15.1:0/2236440268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d815d5a0 0x7f71d80ce010 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7f71c8006fd0 tx=0x7f71c8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191b4908b8431302 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:41.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.773+0000 7f71d6ffd700 1 -- 172.21.15.1:0/2236440268 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c8005e30 con 0x7f71d815d5a0 2024-03-18T20:26:41.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.774+0000 7f71d6ffd700 1 -- 172.21.15.1:0/2236440268 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c8004030 con 0x7f71d815d5a0 2024-03-18T20:26:41.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.774+0000 7f71d6ffd700 1 -- 172.21.15.1:0/2236440268 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c80043e0 con 0x7f71d815d5a0 2024-03-18T20:26:41.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.774+0000 7f71de566700 1 -- 172.21.15.1:0/2236440268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d815d5a0 msgr2=0x7f71d80ce010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:41.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.774+0000 7f71de566700 1 --2- 172.21.15.1:0/2236440268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d815d5a0 0x7f71d80ce010 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7f71c8006fd0 tx=0x7f71c8009bd0 comp rx=0 tx=0).stop 2024-03-18T20:26:41.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.774+0000 7f71de566700 1 -- 172.21.15.1:0/2236440268 shutdown_connections 2024-03-18T20:26:41.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.774+0000 7f71de566700 1 --2- 172.21.15.1:0/2236440268 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f71d806ae60 0x7f71d8167140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:41.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.774+0000 7f71de566700 1 --2- 172.21.15.1:0/2236440268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d815d5a0 0x7f71d80ce010 unknown :-1 s=CLOSED pgs=2865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:41.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.774+0000 7f71de566700 1 -- 172.21.15.1:0/2236440268 >> 172.21.15.1:0/2236440268 conn(0x7f71d80cc9b0 msgr2=0x7f71d80ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:41.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.774+0000 7f71de566700 1 -- 172.21.15.1:0/2236440268 shutdown_connections 2024-03-18T20:26:41.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.774+0000 7f71de566700 1 -- 172.21.15.1:0/2236440268 wait complete. 2024-03-18T20:26:41.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.775+0000 7f71de566700 1 Processor -- start 2024-03-18T20:26:41.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.775+0000 7f71de566700 1 -- start start 2024-03-18T20:26:41.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.776+0000 7f71de566700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f71d806ae60 0x7f71d80c8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:41.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.776+0000 7f71de566700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71d80ce550 0x7f71d815fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:41.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.776+0000 7f71de566700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d815d5a0 0x7f71d81600a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:41.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.776+0000 7f71de566700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d814ac60 con 0x7f71d815d5a0 2024-03-18T20:26:41.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.776+0000 7f71de566700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d8149db0 con 0x7f71d80ce550 2024-03-18T20:26:41.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.776+0000 7f71de566700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d814a050 con 0x7f71d806ae60 2024-03-18T20:26:41.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.776+0000 7f71dcb03700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d815d5a0 0x7f71d81600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:41.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.776+0000 7f71dcb03700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d815d5a0 0x7f71d81600a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35902/0 (socket says 172.21.15.1:35902) 2024-03-18T20:26:41.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.776+0000 7f71dcb03700 1 -- 172.21.15.1:0/2918970905 learned_addr learned my addr 172.21.15.1:0/2918970905 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:41.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.777+0000 7f71d7fff700 1 --2- 172.21.15.1:0/2918970905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f71d806ae60 0x7f71d80c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:41.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.777+0000 7f71d77fe700 1 --2- 172.21.15.1:0/2918970905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71d80ce550 0x7f71d815fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:41.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.777+0000 7f71dcb03700 1 -- 172.21.15.1:0/2918970905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f71d806ae60 msgr2=0x7f71d80c8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:41.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.777+0000 7f71dcb03700 1 --2- 172.21.15.1:0/2918970905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f71d806ae60 0x7f71d80c8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:41.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.777+0000 7f71dcb03700 1 -- 172.21.15.1:0/2918970905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71d80ce550 msgr2=0x7f71d815fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:41.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.777+0000 7f71dcb03700 1 --2- 172.21.15.1:0/2918970905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71d80ce550 0x7f71d815fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:41.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.777+0000 7f71dcb03700 1 -- 172.21.15.1:0/2918970905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c8009500 con 0x7f71d815d5a0 2024-03-18T20:26:41.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.777+0000 7f71d77fe700 1 --2- 172.21.15.1:0/2918970905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71d80ce550 0x7f71d815fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:41.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.777+0000 7f71dcb03700 1 --2- 172.21.15.1:0/2918970905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d815d5a0 0x7f71d81600a0 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7f71d0007e40 tx=0x7f71d000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:41.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.777+0000 7f71d4ff9700 1 -- 172.21.15.1:0/2918970905 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d000eea0 con 0x7f71d815d5a0 2024-03-18T20:26:41.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.777+0000 7f71d4ff9700 1 -- 172.21.15.1:0/2918970905 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d0014420 con 0x7f71d815d5a0 2024-03-18T20:26:41.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.777+0000 7f71de566700 1 -- 172.21.15.1:0/2918970905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d814a330 con 0x7f71d815d5a0 2024-03-18T20:26:41.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.777+0000 7f71d4ff9700 1 -- 172.21.15.1:0/2918970905 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d00147b0 con 0x7f71d815d5a0 2024-03-18T20:26:41.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.777+0000 7f71de566700 1 -- 172.21.15.1:0/2918970905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d814a8f0 con 0x7f71d815d5a0 2024-03-18T20:26:41.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.778+0000 7f71de566700 1 -- 172.21.15.1:0/2918970905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d814add0 con 0x7f71d815d5a0 2024-03-18T20:26:41.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.778+0000 7f71d4ff9700 1 -- 172.21.15.1:0/2918970905 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f71d0014910 con 0x7f71d815d5a0 2024-03-18T20:26:41.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.779+0000 7f71d4ff9700 1 --2- 172.21.15.1:0/2918970905 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f71b806aed0 0x7f71b806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:41.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.779+0000 7f71d4ff9700 1 -- 172.21.15.1:0/2918970905 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f71d008a3b0 con 0x7f71d815d5a0 2024-03-18T20:26:41.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.779+0000 7f71d7fff700 1 --2- 172.21.15.1:0/2918970905 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f71b806aed0 0x7f71b806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:41.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.779+0000 7f71d7fff700 1 --2- 172.21.15.1:0/2918970905 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f71b806aed0 0x7f71b806d390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f71c8014040 tx=0x7f71c8008070 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:41.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:41.783+0000 7f71d4ff9700 1 -- 172.21.15.1:0/2918970905 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f71d0054df0 con 0x7f71d815d5a0 2024-03-18T20:26:42.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.037+0000 7f71de566700 1 -- 172.21.15.1:0/2918970905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f71d80676a0 con 0x7f71d815d5a0 2024-03-18T20:26:42.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.621+0000 7f71d4ff9700 1 -- 172.21.15.1:0/2918970905 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v126) v1 ==== 125+0+0 (secure 0 0 0) 0x7f71d0058410 con 0x7f71d815d5a0 2024-03-18T20:26:42.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.624+0000 7f71de566700 1 -- 172.21.15.1:0/2918970905 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f71b806aed0 msgr2=0x7f71b806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:42.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.624+0000 7f71de566700 1 --2- 172.21.15.1:0/2918970905 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f71b806aed0 0x7f71b806d390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f71c8014040 tx=0x7f71c8008070 comp rx=0 tx=0).stop 2024-03-18T20:26:42.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.624+0000 7f71de566700 1 -- 172.21.15.1:0/2918970905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d815d5a0 msgr2=0x7f71d81600a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:42.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.624+0000 7f71de566700 1 --2- 172.21.15.1:0/2918970905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d815d5a0 0x7f71d81600a0 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7f71d0007e40 tx=0x7f71d000e3c0 comp rx=0 tx=0).stop 2024-03-18T20:26:42.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.624+0000 7f71de566700 1 -- 172.21.15.1:0/2918970905 shutdown_connections 2024-03-18T20:26:42.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.624+0000 7f71de566700 1 --2- 172.21.15.1:0/2918970905 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f71b806aed0 0x7f71b806d390 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:42.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.624+0000 7f71de566700 1 --2- 172.21.15.1:0/2918970905 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f71d806ae60 0x7f71d80c8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:42.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.624+0000 7f71de566700 1 --2- 172.21.15.1:0/2918970905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71d815d5a0 0x7f71d81600a0 unknown :-1 s=CLOSED pgs=2866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:42.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.624+0000 7f71de566700 1 --2- 172.21.15.1:0/2918970905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71d80ce550 0x7f71d815fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:42.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.624+0000 7f71de566700 1 -- 172.21.15.1:0/2918970905 >> 172.21.15.1:0/2918970905 conn(0x7f71d80cc9b0 msgr2=0x7f71d806df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:42.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.625+0000 7f71de566700 1 -- 172.21.15.1:0/2918970905 shutdown_connections 2024-03-18T20:26:42.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.625+0000 7f71de566700 1 -- 172.21.15.1:0/2918970905 wait complete. 2024-03-18T20:26:42.643 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2024-03-18T20:26:42.643 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:26:42.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.771+0000 7fb350540700 1 Processor -- start 2024-03-18T20:26:42.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.771+0000 7fb350540700 1 -- start start 2024-03-18T20:26:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.772+0000 7fb350540700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb34815bb90 0x7fb348099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.772+0000 7fb350540700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb34815b610 0x7fb348099f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.772+0000 7fb350540700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3480cc6e0 con 0x7fb34815b230 2024-03-18T20:26:42.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.772+0000 7fb350540700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3480ced80 con 0x7fb34815b610 2024-03-18T20:26:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.772+0000 7fb350540700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480d01b0 con 0x7fb34815bb90 2024-03-18T20:26:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.772+0000 7fb34eadd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb34815b610 0x7fb348099f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.772+0000 7fb34eadd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb34815b610 0x7fb348099f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35918/0 (socket says 172.21.15.1:35918) 2024-03-18T20:26:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.772+0000 7fb34eadd700 1 -- 172.21.15.1:0/326640538 learned_addr learned my addr 172.21.15.1:0/326640538 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.772+0000 7fb34dadb700 1 --2- 172.21.15.1:0/326640538 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb34815bb90 0x7fb348099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.772+0000 7fb34eadd700 1 -- 172.21.15.1:0/326640538 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb34815bb90 msgr2=0x7fb348099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.772+0000 7fb34eadd700 1 --2- 172.21.15.1:0/326640538 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb34815bb90 0x7fb348099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.772+0000 7fb34eadd700 1 -- 172.21.15.1:0/326640538 >> v1:172.21.15.165:6789/0 conn(0x7fb34815b230 legacy=0x7fb3480cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.772+0000 7fb34eadd700 1 -- 172.21.15.1:0/326640538 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3480cc9c0 con 0x7fb34815b610 2024-03-18T20:26:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.772+0000 7fb34eadd700 1 --2- 172.21.15.1:0/326640538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb34815b610 0x7fb348099f80 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7fb340009e50 tx=0x7fb34000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=412b7de998022b71 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:42.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.773+0000 7fb34d2da700 1 -- 172.21.15.1:0/326640538 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34000e070 con 0x7fb34815b610 2024-03-18T20:26:42.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.773+0000 7fb34d2da700 1 -- 172.21.15.1:0/326640538 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34000b9c0 con 0x7fb34815b610 2024-03-18T20:26:42.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.773+0000 7fb34d2da700 1 -- 172.21.15.1:0/326640538 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34000bd70 con 0x7fb34815b610 2024-03-18T20:26:42.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.773+0000 7fb350540700 1 -- 172.21.15.1:0/326640538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb34815b610 msgr2=0x7fb348099f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:42.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.773+0000 7fb350540700 1 --2- 172.21.15.1:0/326640538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb34815b610 0x7fb348099f80 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7fb340009e50 tx=0x7fb34000b330 comp rx=0 tx=0).stop 2024-03-18T20:26:42.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.773+0000 7fb350540700 1 -- 172.21.15.1:0/326640538 shutdown_connections 2024-03-18T20:26:42.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.773+0000 7fb350540700 1 --2- 172.21.15.1:0/326640538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb34815b610 0x7fb348099f80 unknown :-1 s=CLOSED pgs=2867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:42.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.773+0000 7fb350540700 1 --2- 172.21.15.1:0/326640538 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb34815bb90 0x7fb348099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:42.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.773+0000 7fb350540700 1 -- 172.21.15.1:0/326640538 >> 172.21.15.1:0/326640538 conn(0x7fb3480b2c30 msgr2=0x7fb3480b3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:42.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.773+0000 7fb350540700 1 -- 172.21.15.1:0/326640538 shutdown_connections 2024-03-18T20:26:42.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.773+0000 7fb350540700 1 -- 172.21.15.1:0/326640538 wait complete. 2024-03-18T20:26:42.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.774+0000 7fb350540700 1 Processor -- start 2024-03-18T20:26:42.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.774+0000 7fb350540700 1 -- start start 2024-03-18T20:26:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.775+0000 7fb350540700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb34815b230 0x7fb3480b6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.775+0000 7fb350540700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb34815b610 0x7fb34809b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.775+0000 7fb350540700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb34815bb90 0x7fb34806cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.775+0000 7fb350540700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480b5240 con 0x7fb34815b610 2024-03-18T20:26:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.775+0000 7fb350540700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480b5520 con 0x7fb34815b230 2024-03-18T20:26:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.775+0000 7fb350540700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480b5800 con 0x7fb34815bb90 2024-03-18T20:26:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.775+0000 7fb34dadb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb34815b610 0x7fb34809b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.775+0000 7fb34dadb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb34815b610 0x7fb34809b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35934/0 (socket says 172.21.15.1:35934) 2024-03-18T20:26:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.775+0000 7fb34dadb700 1 -- 172.21.15.1:0/289007933 learned_addr learned my addr 172.21.15.1:0/289007933 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:42.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.775+0000 7fb34dadb700 1 -- 172.21.15.1:0/289007933 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb34815bb90 msgr2=0x7fb34806cef0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T20:26:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.776+0000 7fb34dadb700 1 --2- 172.21.15.1:0/289007933 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb34815bb90 0x7fb34806cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.776+0000 7fb34dadb700 1 -- 172.21.15.1:0/289007933 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb34815b230 msgr2=0x7fb3480b6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.776+0000 7fb34dadb700 1 --2- 172.21.15.1:0/289007933 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb34815b230 0x7fb3480b6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.776+0000 7fb34dadb700 1 -- 172.21.15.1:0/289007933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb340009b30 con 0x7fb34815b610 2024-03-18T20:26:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.776+0000 7fb34dadb700 1 --2- 172.21.15.1:0/289007933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb34815b610 0x7fb34809b580 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7fb33800d800 tx=0x7fb33800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.776+0000 7fb33effd700 1 -- 172.21.15.1:0/289007933 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3380125f0 con 0x7fb34815b610 2024-03-18T20:26:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.776+0000 7fb33effd700 1 -- 172.21.15.1:0/289007933 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb338012750 con 0x7fb34815b610 2024-03-18T20:26:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.776+0000 7fb350540700 1 -- 172.21.15.1:0/289007933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3480b5ae0 con 0x7fb34815b610 2024-03-18T20:26:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.776+0000 7fb33effd700 1 -- 172.21.15.1:0/289007933 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338012ae0 con 0x7fb34815b610 2024-03-18T20:26:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.776+0000 7fb350540700 1 -- 172.21.15.1:0/289007933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34815a600 con 0x7fb34815b610 2024-03-18T20:26:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.777+0000 7fb350540700 1 -- 172.21.15.1:0/289007933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34806e800 con 0x7fb34815b610 2024-03-18T20:26:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.777+0000 7fb33effd700 1 -- 172.21.15.1:0/289007933 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fb338012c40 con 0x7fb34815b610 2024-03-18T20:26:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.778+0000 7fb33effd700 1 --2- 172.21.15.1:0/289007933 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb32c06aed0 0x7fb32c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.778+0000 7fb33effd700 1 -- 172.21.15.1:0/289007933 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6993+0+0 (secure 0 0 0) 0x7fb338089fe0 con 0x7fb34815b610 2024-03-18T20:26:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.778+0000 7fb34e2dc700 1 --2- 172.21.15.1:0/289007933 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb32c06aed0 0x7fb32c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.778+0000 7fb34e2dc700 1 --2- 172.21.15.1:0/289007933 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb32c06aed0 0x7fb32c06d390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb344000f60 tx=0x7fb344007a40 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:42.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:42.782+0000 7fb33effd700 1 -- 172.21.15.1:0/289007933 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb338054950 con 0x7fb34815b610 2024-03-18T20:26:43.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.030+0000 7fb350540700 1 -- 172.21.15.1:0/289007933 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb34815a790 con 0x7fb34815b610 2024-03-18T20:26:43.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.030+0000 7fb33effd700 1 -- 172.21.15.1:0/289007933 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+5963 (secure 0 0 0) 0x7fb338057f70 con 0x7fb34815b610 2024-03-18T20:26:43.034 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:43.034 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":126,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12396,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2565889517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2565889517},{"type":"v1","addr":"172.21.15.1:6836","nonce":2565889517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":12399,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/512665659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":512665659},{"type":"v1","addr":"172.21.15.1:6837","nonce":512665659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:26:25.785074+0000","modified":"2024-03-18T20:26:29.101123+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12411":{"gid":12411,"name":"d","rank":0,"incarnation":116,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/3886150679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":3886150679},{"type":"v1","addr":"172.21.15.165:6835","nonce":3886150679}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:26:39.596187+0000","modified":"2024-03-18T20:26:42.614846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12408":{"gid":12408,"name":"b","rank":0,"incarnation":122,"state":"up:active","state_seq":6,"addr":"172.21.15.165:6834/1134181166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1134181166},{"type":"v1","addr":"172.21.15.165:6834","nonce":1134181166}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-03-18T20:26:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.032+0000 7fb350540700 1 -- 172.21.15.1:0/289007933 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb32c06aed0 msgr2=0x7fb32c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.032+0000 7fb350540700 1 --2- 172.21.15.1:0/289007933 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb32c06aed0 0x7fb32c06d390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb344000f60 tx=0x7fb344007a40 comp rx=0 tx=0).stop 2024-03-18T20:26:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.032+0000 7fb350540700 1 -- 172.21.15.1:0/289007933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb34815b610 msgr2=0x7fb34809b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.032+0000 7fb350540700 1 --2- 172.21.15.1:0/289007933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb34815b610 0x7fb34809b580 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7fb33800d800 tx=0x7fb33800dbc0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.032+0000 7fb350540700 1 -- 172.21.15.1:0/289007933 shutdown_connections 2024-03-18T20:26:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.032+0000 7fb350540700 1 --2- 172.21.15.1:0/289007933 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb32c06aed0 0x7fb32c06d390 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.032+0000 7fb350540700 1 --2- 172.21.15.1:0/289007933 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb34815bb90 0x7fb34806cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.032+0000 7fb350540700 1 --2- 172.21.15.1:0/289007933 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb34815b230 0x7fb3480b6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.032+0000 7fb350540700 1 --2- 172.21.15.1:0/289007933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb34815b610 0x7fb34809b580 unknown :-1 s=CLOSED pgs=2868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.032+0000 7fb350540700 1 -- 172.21.15.1:0/289007933 >> 172.21.15.1:0/289007933 conn(0x7fb3480b2c30 msgr2=0x7fb3480b3010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.032+0000 7fb350540700 1 -- 172.21.15.1:0/289007933 shutdown_connections 2024-03-18T20:26:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.032+0000 7fb350540700 1 -- 172.21.15.1:0/289007933 wait complete. 2024-03-18T20:26:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 126 2024-03-18T20:26:43.048 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:26:43.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.171+0000 7effc27ec700 1 Processor -- start 2024-03-18T20:26:43.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.171+0000 7effc27ec700 1 -- start start 2024-03-18T20:26:43.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.171+0000 7effc27ec700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effbc143c20 0x7effbc0b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.171+0000 7effc27ec700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effbc144000 con 0x7effbc0b3f40 2024-03-18T20:26:43.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.171+0000 7effc27ec700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effbc0cd8d0 con 0x7effbc06f240 2024-03-18T20:26:43.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.171+0000 7effc27ec700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effbc0b43c0 con 0x7effbc143c20 2024-03-18T20:26:43.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbb7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7effbc0b3f40 0x7effbc06eb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56358/0 (socket says 172.21.15.1:56358) 2024-03-18T20:26:43.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbb7fe700 1 -- 172.21.15.1:0/2329613135 learned_addr learned my addr 172.21.15.1:0/2329613135 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:43.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbbfff700 1 --2- 172.21.15.1:0/2329613135 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effbc143c20 0x7effbc0b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbaffd700 1 -- 172.21.15.1:0/2329613135 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827346811 0 0) 0x7effbc144000 con 0x7effbc0b3f40 2024-03-18T20:26:43.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbaffd700 1 -- 172.21.15.1:0/2329613135 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effa40035b0 con 0x7effbc0b3f40 2024-03-18T20:26:43.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbaffd700 1 -- 172.21.15.1:0/2329613135 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624288823 0 0) 0x7effbc0cd8d0 con 0x7effbc06f240 2024-03-18T20:26:43.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbaffd700 1 -- 172.21.15.1:0/2329613135 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effbc144000 con 0x7effbc06f240 2024-03-18T20:26:43.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbaffd700 1 -- 172.21.15.1:0/2329613135 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3566530641 0 0) 0x7effa40035b0 con 0x7effbc0b3f40 2024-03-18T20:26:43.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbaffd700 1 -- 172.21.15.1:0/2329613135 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effbc143c20 msgr2=0x7effbc0b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:43.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbaffd700 1 --2- 172.21.15.1:0/2329613135 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effbc143c20 0x7effbc0b3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbaffd700 1 -- 172.21.15.1:0/2329613135 >> v1:172.21.15.165:6789/0 conn(0x7effbc06f240 legacy=0x7effbc0cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:43.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbaffd700 1 -- 172.21.15.1:0/2329613135 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effbc06fdc0 con 0x7effbc0b3f40 2024-03-18T20:26:43.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbaffd700 1 -- 172.21.15.1:0/2329613135 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7effb0001670 con 0x7effbc0b3f40 2024-03-18T20:26:43.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbaffd700 1 -- 172.21.15.1:0/2329613135 >> v1:172.21.15.1:6789/0 conn(0x7effbc0b3f40 legacy=0x7effbc06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:43.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbaffd700 1 --2- 172.21.15.1:0/2329613135 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effa4003bc0 0x7effa4003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbaffd700 1 --2- 172.21.15.1:0/2329613135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effa4004850 0x7effa4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbaffd700 1 --2- 172.21.15.1:0/2329613135 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effa4005210 0x7effa400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbaffd700 1 -- 172.21.15.1:0/2329613135 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effbc0b43c0 con 0x7effa4005210 2024-03-18T20:26:43.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbaffd700 1 -- 172.21.15.1:0/2329613135 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa400c000 con 0x7effa4004850 2024-03-18T20:26:43.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbaffd700 1 -- 172.21.15.1:0/2329613135 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa400c1c0 con 0x7effa4003bc0 2024-03-18T20:26:43.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbbfff700 1 --2- 172.21.15.1:0/2329613135 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effbc143c20 0x7effbc0b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:26:43.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effc0d89700 1 --2- 172.21.15.1:0/2329613135 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effa4005210 0x7effa400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbb7fe700 1 --2- 172.21.15.1:0/2329613135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effa4004850 0x7effa4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.172+0000 7effbbfff700 1 --2- 172.21.15.1:0/2329613135 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effa4003bc0 0x7effa4003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.173+0000 7effc0d89700 1 -- 172.21.15.1:0/2329613135 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effa4003bc0 msgr2=0x7effa4003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:43.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.173+0000 7effc0d89700 1 --2- 172.21.15.1:0/2329613135 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effa4003bc0 0x7effa4003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.173+0000 7effc0d89700 1 -- 172.21.15.1:0/2329613135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effa4004850 msgr2=0x7effa4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:43.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.173+0000 7effc0d89700 1 --2- 172.21.15.1:0/2329613135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effa4004850 0x7effa4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.173+0000 7effc0d89700 1 -- 172.21.15.1:0/2329613135 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa40035b0 con 0x7effa4005210 2024-03-18T20:26:43.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.173+0000 7effbbfff700 1 --2- 172.21.15.1:0/2329613135 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effa4003bc0 0x7effa4003fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:43.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.173+0000 7effbb7fe700 1 --2- 172.21.15.1:0/2329613135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effa4004850 0x7effa4004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:43.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.173+0000 7effc0d89700 1 --2- 172.21.15.1:0/2329613135 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effa4005210 0x7effa400ba10 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7effb400c2f0 tx=0x7effa4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93335f165684628a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:43.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.173+0000 7effbaffd700 1 -- 172.21.15.1:0/2329613135 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb4001f20 con 0x7effa4005210 2024-03-18T20:26:43.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.173+0000 7effbaffd700 1 -- 172.21.15.1:0/2329613135 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb400ce70 con 0x7effa4005210 2024-03-18T20:26:43.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.173+0000 7effbaffd700 1 -- 172.21.15.1:0/2329613135 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb4003c20 con 0x7effa4005210 2024-03-18T20:26:43.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.173+0000 7effc27ec700 1 -- 172.21.15.1:0/2329613135 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effa4005210 msgr2=0x7effa400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:43.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.173+0000 7effc27ec700 1 --2- 172.21.15.1:0/2329613135 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effa4005210 0x7effa400ba10 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7effb400c2f0 tx=0x7effa4005880 comp rx=0 tx=0).stop 2024-03-18T20:26:43.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.173+0000 7effc0d89700 1 -- 172.21.15.1:0/2329613135 reap_dead start 2024-03-18T20:26:43.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.173+0000 7effc27ec700 1 -- 172.21.15.1:0/2329613135 shutdown_connections 2024-03-18T20:26:43.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.173+0000 7effc27ec700 1 -- 172.21.15.1:0/2329613135 >> 172.21.15.1:0/2329613135 conn(0x7effbc0b1d20 msgr2=0x7effbc0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:43.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.174+0000 7effc27ec700 1 -- 172.21.15.1:0/2329613135 shutdown_connections 2024-03-18T20:26:43.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.174+0000 7effc27ec700 1 -- 172.21.15.1:0/2329613135 wait complete. 2024-03-18T20:26:43.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.175+0000 7effc27ec700 1 Processor -- start 2024-03-18T20:26:43.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.175+0000 7effc27ec700 1 -- start start 2024-03-18T20:26:43.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effc27ec700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effbc0cf730 0x7effbc0ae460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effc27ec700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effbc0ae9a0 0x7effbc1417b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effc27ec700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effbc0cfb10 0x7effbc141cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effc27ec700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effbc1f3070 con 0x7effbc0cf730 2024-03-18T20:26:43.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effc27ec700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effbc1f38d0 con 0x7effbc0cfb10 2024-03-18T20:26:43.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effc27ec700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effbc0af120 con 0x7effbc0ae9a0 2024-03-18T20:26:43.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effbbfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effbc0cf730 0x7effbc0ae460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effbbfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effbc0cf730 0x7effbc0ae460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35950/0 (socket says 172.21.15.1:35950) 2024-03-18T20:26:43.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effbbfff700 1 -- 172.21.15.1:0/2421022833 learned_addr learned my addr 172.21.15.1:0/2421022833 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:43.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effbb7fe700 1 --2- 172.21.15.1:0/2421022833 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effbc0ae9a0 0x7effbc1417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effc0d89700 1 --2- 172.21.15.1:0/2421022833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effbc0cfb10 0x7effbc141cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effbbfff700 1 -- 172.21.15.1:0/2421022833 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effbc0ae9a0 msgr2=0x7effbc1417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:43.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effbbfff700 1 --2- 172.21.15.1:0/2421022833 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effbc0ae9a0 0x7effbc1417b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effbbfff700 1 -- 172.21.15.1:0/2421022833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effbc0cfb10 msgr2=0x7effbc141cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:43.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effbbfff700 1 --2- 172.21.15.1:0/2421022833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effbc0cfb10 0x7effbc141cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effbbfff700 1 -- 172.21.15.1:0/2421022833 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effbc0aee40 con 0x7effbc0cf730 2024-03-18T20:26:43.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effc0d89700 1 --2- 172.21.15.1:0/2421022833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effbc0cfb10 0x7effbc141cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:43.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effbbfff700 1 --2- 172.21.15.1:0/2421022833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effbc0cf730 0x7effbc0ae460 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 crypto rx=0x7effac0109e0 tx=0x7effac010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:43.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effb8ff9700 1 -- 172.21.15.1:0/2421022833 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac00e870 con 0x7effbc0cf730 2024-03-18T20:26:43.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effb8ff9700 1 -- 172.21.15.1:0/2421022833 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effac00e9d0 con 0x7effbc0cf730 2024-03-18T20:26:43.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.176+0000 7effc27ec700 1 -- 172.21.15.1:0/2421022833 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effbc1f3280 con 0x7effbc0cf730 2024-03-18T20:26:43.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.177+0000 7effb8ff9700 1 -- 172.21.15.1:0/2421022833 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac00ed40 con 0x7effbc0cf730 2024-03-18T20:26:43.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.177+0000 7effc27ec700 1 -- 172.21.15.1:0/2421022833 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effbc15b200 con 0x7effbc0cf730 2024-03-18T20:26:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.177+0000 7effc27ec700 1 -- 172.21.15.1:0/2421022833 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effbc0cd1d0 con 0x7effbc0cf730 2024-03-18T20:26:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.178+0000 7effb8ff9700 1 -- 172.21.15.1:0/2421022833 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7effac016460 con 0x7effbc0cf730 2024-03-18T20:26:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.178+0000 7effb8ff9700 1 --2- 172.21.15.1:0/2421022833 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7effa806af00 0x7effa806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.178+0000 7effb8ff9700 1 -- 172.21.15.1:0/2421022833 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6993+0+0 (secure 0 0 0) 0x7effac08a140 con 0x7effbc0cf730 2024-03-18T20:26:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.178+0000 7effbb7fe700 1 --2- 172.21.15.1:0/2421022833 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7effa806af00 0x7effa806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.179+0000 7effbb7fe700 1 --2- 172.21.15.1:0/2421022833 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7effa806af00 0x7effa806d3c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7effbc062510 tx=0x7effb0009040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:43.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.183+0000 7effb8ff9700 1 -- 172.21.15.1:0/2421022833 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7effac054ab0 con 0x7effbc0cf730 2024-03-18T20:26:43.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.385+0000 7effc27ec700 1 -- 172.21.15.1:0/2421022833 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effbc0676a0 con 0x7effbc0cf730 2024-03-18T20:26:43.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.387+0000 7effb8ff9700 1 -- 172.21.15.1:0/2421022833 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v257) v1 ==== 74+0+20089 (secure 0 0 0) 0x7effac00eea0 con 0x7effbc0cf730 2024-03-18T20:26:43.391 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:43.391 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":257,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:26:40.598066+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:26:23.615691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-18T20:26:24.759650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:26:37.326244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:26:38.564628+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"257","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:26:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.389+0000 7effc27ec700 1 -- 172.21.15.1:0/2421022833 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7effa806af00 msgr2=0x7effa806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.389+0000 7effc27ec700 1 --2- 172.21.15.1:0/2421022833 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7effa806af00 0x7effa806d3c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7effbc062510 tx=0x7effb0009040 comp rx=0 tx=0).stop 2024-03-18T20:26:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.389+0000 7effc27ec700 1 -- 172.21.15.1:0/2421022833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effbc0cf730 msgr2=0x7effbc0ae460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.389+0000 7effc27ec700 1 --2- 172.21.15.1:0/2421022833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effbc0cf730 0x7effbc0ae460 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 crypto rx=0x7effac0109e0 tx=0x7effac010da0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.389+0000 7effc27ec700 1 -- 172.21.15.1:0/2421022833 shutdown_connections 2024-03-18T20:26:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.389+0000 7effc27ec700 1 --2- 172.21.15.1:0/2421022833 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7effa806af00 0x7effa806d3c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.389+0000 7effc27ec700 1 --2- 172.21.15.1:0/2421022833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effbc0cfb10 0x7effbc141cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.389+0000 7effc27ec700 1 --2- 172.21.15.1:0/2421022833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effbc0cf730 0x7effbc0ae460 unknown :-1 s=CLOSED pgs=2871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.389+0000 7effc27ec700 1 --2- 172.21.15.1:0/2421022833 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effbc0ae9a0 0x7effbc1417b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.389+0000 7effc27ec700 1 -- 172.21.15.1:0/2421022833 >> 172.21.15.1:0/2421022833 conn(0x7effbc0b1d20 msgr2=0x7effbc0afda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.389+0000 7effc27ec700 1 -- 172.21.15.1:0/2421022833 shutdown_connections 2024-03-18T20:26:43.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.389+0000 7effc27ec700 1 -- 172.21.15.1:0/2421022833 wait complete. 2024-03-18T20:26:43.405 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T20:26:43.406 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:26:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.526+0000 7f95ebe7b700 1 Processor -- start 2024-03-18T20:26:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.526+0000 7f95ebe7b700 1 -- start start 2024-03-18T20:26:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.527+0000 7f95ebe7b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f95e4144b70 0x7f95e40ce980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.527+0000 7f95ebe7b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95e40cd110 0x7f95e4071a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.527+0000 7f95ebe7b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95e40cf2d0 con 0x7f95e40af610 2024-03-18T20:26:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.527+0000 7f95ebe7b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95e4144f50 con 0x7f95e4144b70 2024-03-18T20:26:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.527+0000 7f95ebe7b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e40cf5b0 con 0x7f95e40cd110 2024-03-18T20:26:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.528+0000 7f95ea418700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95e40cd110 0x7f95e4071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.528+0000 7f95e9c17700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f95e4144b70 0x7f95e40ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.528+0000 7f95ea418700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95e40cd110 0x7f95e4071a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:35366/0 (socket says 172.21.15.1:35366) 2024-03-18T20:26:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.528+0000 7f95e9416700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f95e40af610 0x7f95e40cca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:60212/0 (socket says 172.21.15.1:60212) 2024-03-18T20:26:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.528+0000 7f95ea418700 1 -- 172.21.15.1:0/1596045146 learned_addr learned my addr 172.21.15.1:0/1596045146 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.528+0000 7f95ea418700 1 -- 172.21.15.1:0/1596045146 >> v2:172.21.15.165:3300/0 conn(0x7f95e4144b70 msgr2=0x7f95e40ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.528+0000 7f95ea418700 1 --2- 172.21.15.1:0/1596045146 >> v2:172.21.15.165:3300/0 conn(0x7f95e4144b70 0x7f95e40ce980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.528+0000 7f95ea418700 1 -- 172.21.15.1:0/1596045146 >> v1:172.21.15.165:6789/0 conn(0x7f95e40af610 legacy=0x7f95e40cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.528+0000 7f95ea418700 1 -- 172.21.15.1:0/1596045146 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e40cf890 con 0x7f95e40cd110 2024-03-18T20:26:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.528+0000 7f95ea418700 1 --2- 172.21.15.1:0/1596045146 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95e40cd110 0x7f95e4071a10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f95dc009e50 tx=0x7f95dc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5b7fa1f0c2fe7bb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.528+0000 7f95e8c15700 1 -- 172.21.15.1:0/1596045146 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95dc00e070 con 0x7f95e40cd110 2024-03-18T20:26:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.528+0000 7f95e8c15700 1 -- 172.21.15.1:0/1596045146 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95dc00b910 con 0x7f95e40cd110 2024-03-18T20:26:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.528+0000 7f95e8c15700 1 -- 172.21.15.1:0/1596045146 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95dc00bcc0 con 0x7f95e40cd110 2024-03-18T20:26:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.529+0000 7f95ebe7b700 1 -- 172.21.15.1:0/1596045146 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95e40cd110 msgr2=0x7f95e4071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.529+0000 7f95ebe7b700 1 --2- 172.21.15.1:0/1596045146 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95e40cd110 0x7f95e4071a10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f95dc009e50 tx=0x7f95dc00b330 comp rx=0 tx=0).stop 2024-03-18T20:26:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.529+0000 7f95ebe7b700 1 -- 172.21.15.1:0/1596045146 shutdown_connections 2024-03-18T20:26:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.529+0000 7f95ebe7b700 1 --2- 172.21.15.1:0/1596045146 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95e40cd110 0x7f95e4071a10 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.529+0000 7f95ebe7b700 1 --2- 172.21.15.1:0/1596045146 >> v2:172.21.15.165:3300/0 conn(0x7f95e4144b70 0x7f95e40ce980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.529+0000 7f95ebe7b700 1 -- 172.21.15.1:0/1596045146 >> 172.21.15.1:0/1596045146 conn(0x7f95e40b1d20 msgr2=0x7f95e40b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.529+0000 7f95ebe7b700 1 -- 172.21.15.1:0/1596045146 shutdown_connections 2024-03-18T20:26:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.529+0000 7f95ebe7b700 1 -- 172.21.15.1:0/1596045146 wait complete. 2024-03-18T20:26:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.530+0000 7f95ebe7b700 1 Processor -- start 2024-03-18T20:26:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.530+0000 7f95ebe7b700 1 -- start start 2024-03-18T20:26:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.531+0000 7f95ebe7b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95e40af610 0x7f95e41e6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.531+0000 7f95ebe7b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95e40cd110 0x7f95e41e71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.531+0000 7f95ebe7b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95e4144b70 0x7f95e4138080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.531+0000 7f95ebe7b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e4139330 con 0x7f95e40cd110 2024-03-18T20:26:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.531+0000 7f95ebe7b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e4139610 con 0x7f95e4144b70 2024-03-18T20:26:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.531+0000 7f95ebe7b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e41398f0 con 0x7f95e40af610 2024-03-18T20:26:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.531+0000 7f95e9416700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95e40cd110 0x7f95e41e71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.531+0000 7f95e9416700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95e40cd110 0x7f95e41e71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35960/0 (socket says 172.21.15.1:35960) 2024-03-18T20:26:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.531+0000 7f95e9416700 1 -- 172.21.15.1:0/2727171539 learned_addr learned my addr 172.21.15.1:0/2727171539 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.531+0000 7f95ea418700 1 --2- 172.21.15.1:0/2727171539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95e4144b70 0x7f95e4138080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.531+0000 7f95e9c17700 1 --2- 172.21.15.1:0/2727171539 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95e40af610 0x7f95e41e6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.531+0000 7f95e9416700 1 -- 172.21.15.1:0/2727171539 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95e40af610 msgr2=0x7f95e41e6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.532+0000 7f95e9416700 1 --2- 172.21.15.1:0/2727171539 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95e40af610 0x7f95e41e6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.532+0000 7f95e9416700 1 -- 172.21.15.1:0/2727171539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95e4144b70 msgr2=0x7f95e4138080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.532+0000 7f95e9416700 1 --2- 172.21.15.1:0/2727171539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95e4144b70 0x7f95e4138080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.532+0000 7f95e9416700 1 -- 172.21.15.1:0/2727171539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95dc009b30 con 0x7f95e40cd110 2024-03-18T20:26:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.532+0000 7f95ea418700 1 --2- 172.21.15.1:0/2727171539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95e4144b70 0x7f95e4138080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.532+0000 7f95e9416700 1 --2- 172.21.15.1:0/2727171539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95e40cd110 0x7f95e41e71d0 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f95d800c520 tx=0x7f95d800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.532+0000 7f95d67fc700 1 -- 172.21.15.1:0/2727171539 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d8005cb0 con 0x7f95e40cd110 2024-03-18T20:26:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.532+0000 7f95d67fc700 1 -- 172.21.15.1:0/2727171539 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d8005e10 con 0x7f95e40cd110 2024-03-18T20:26:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.532+0000 7f95ebe7b700 1 -- 172.21.15.1:0/2727171539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e406ecb0 con 0x7f95e40cd110 2024-03-18T20:26:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.532+0000 7f95d67fc700 1 -- 172.21.15.1:0/2727171539 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d80072d0 con 0x7f95e40cd110 2024-03-18T20:26:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.532+0000 7f95ebe7b700 1 -- 172.21.15.1:0/2727171539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e4137a40 con 0x7f95e40cd110 2024-03-18T20:26:43.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.533+0000 7f95ebe7b700 1 -- 172.21.15.1:0/2727171539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95e40cda90 con 0x7f95e40cd110 2024-03-18T20:26:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.533+0000 7f95d67fc700 1 -- 172.21.15.1:0/2727171539 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f95d8007430 con 0x7f95e40cd110 2024-03-18T20:26:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.534+0000 7f95d67fc700 1 --2- 172.21.15.1:0/2727171539 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f95c806aed0 0x7f95c806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.534+0000 7f95d67fc700 1 -- 172.21.15.1:0/2727171539 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f95d8089040 con 0x7f95e40cd110 2024-03-18T20:26:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.534+0000 7f95e9c17700 1 --2- 172.21.15.1:0/2727171539 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f95c806aed0 0x7f95c806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.534+0000 7f95e9c17700 1 --2- 172.21.15.1:0/2727171539 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f95c806aed0 0x7f95c806d390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f95e415a560 tx=0x7f95e0008040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:43.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.538+0000 7f95d67fc700 1 -- 172.21.15.1:0/2727171539 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f95d8053a60 con 0x7f95e40cd110 2024-03-18T20:26:43.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.737+0000 7f95ebe7b700 1 -- 172.21.15.1:0/2727171539 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f95e413fea0 con 0x7f95c806aed0 2024-03-18T20:26:43.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.752+0000 7f95d67fc700 1 -- 172.21.15.1:0/2727171539 <== mgr.12057 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528961 (secure 0 0 0) 0x7f95e413fea0 con 0x7f95c806aed0 2024-03-18T20:26:43.762 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.756+0000 7f95ebe7b700 1 -- 172.21.15.1:0/2727171539 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f95c806aed0 msgr2=0x7f95c806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.756+0000 7f95ebe7b700 1 --2- 172.21.15.1:0/2727171539 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f95c806aed0 0x7f95c806d390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f95e415a560 tx=0x7f95e0008040 comp rx=0 tx=0).stop 2024-03-18T20:26:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.756+0000 7f95ebe7b700 1 -- 172.21.15.1:0/2727171539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95e40cd110 msgr2=0x7f95e41e71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.756+0000 7f95ebe7b700 1 --2- 172.21.15.1:0/2727171539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95e40cd110 0x7f95e41e71d0 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f95d800c520 tx=0x7f95d800c8e0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.756+0000 7f95ebe7b700 1 -- 172.21.15.1:0/2727171539 shutdown_connections 2024-03-18T20:26:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.756+0000 7f95ebe7b700 1 --2- 172.21.15.1:0/2727171539 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f95c806aed0 0x7f95c806d390 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.756+0000 7f95ebe7b700 1 --2- 172.21.15.1:0/2727171539 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f95e40af610 0x7f95e41e6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.756+0000 7f95ebe7b700 1 --2- 172.21.15.1:0/2727171539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95e4144b70 0x7f95e4138080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.756+0000 7f95ebe7b700 1 --2- 172.21.15.1:0/2727171539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95e40cd110 0x7f95e41e71d0 unknown :-1 s=CLOSED pgs=2872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.756+0000 7f95ebe7b700 1 -- 172.21.15.1:0/2727171539 >> 172.21.15.1:0/2727171539 conn(0x7f95e40b1d20 msgr2=0x7f95e406a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.756+0000 7f95ebe7b700 1 -- 172.21.15.1:0/2727171539 shutdown_connections 2024-03-18T20:26:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.757+0000 7f95ebe7b700 1 -- 172.21.15.1:0/2727171539 wait complete. 2024-03-18T20:26:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:26:43.772 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-03-18T20:26:41.879236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596350,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":710,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":449,"ondisk_log_size":449,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":226,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":415544,"kb_used_data":26308,"kb_used_omap":12,"kb_used_meta":389107,"kb_avail":754559176,"statfs":{"total":773094113280,"available":772668596224,"internally_reserved":0,"allocated":26939392,"data_stored":14445086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":398446137},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3177,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000917"},"pg_stats":[{"pgid":"34.1d","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.622190+0000","last_change":"2024-03-18T20:26:39.614975+0000","last_active":"2024-03-18T20:26:39.622190+0000","last_peered":"2024-03-18T20:26:39.622190+0000","last_clean":"2024-03-18T20:26:39.622190+0000","last_became_active":"2024-03-18T20:26:39.614542+0000","last_became_peered":"2024-03-18T20:26:39.614542+0000","last_unstale":"2024-03-18T20:26:39.622190+0000","last_undegraded":"2024-03-18T20:26:39.622190+0000","last_fullsized":"2024-03-18T20:26:39.622190+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:20.170314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"256'1","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.615752+0000","last_change":"2024-03-18T20:26:39.176702+0000","last_active":"2024-03-18T20:26:39.615752+0000","last_peered":"2024-03-18T20:26:39.615752+0000","last_clean":"2024-03-18T20:26:39.615752+0000","last_became_active":"2024-03-18T20:26:39.176448+0000","last_became_peered":"2024-03-18T20:26:39.176448+0000","last_unstale":"2024-03-18T20:26:39.615752+0000","last_undegraded":"2024-03-18T20:26:39.615752+0000","last_fullsized":"2024-03-18T20:26:39.615752+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:56.821401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174088+0000","last_change":"2024-03-18T20:26:25.816062+0000","last_active":"2024-03-18T20:26:39.174088+0000","last_peered":"2024-03-18T20:26:39.174088+0000","last_clean":"2024-03-18T20:26:39.174088+0000","last_became_active":"2024-03-18T20:26:25.815866+0000","last_became_peered":"2024-03-18T20:26:25.815866+0000","last_unstale":"2024-03-18T20:26:39.174088+0000","last_undegraded":"2024-03-18T20:26:39.174088+0000","last_fullsized":"2024-03-18T20:26:39.174088+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:19:53.852641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.625092+0000","last_change":"2024-03-18T20:26:39.623961+0000","last_active":"2024-03-18T20:26:39.625092+0000","last_peered":"2024-03-18T20:26:39.625092+0000","last_clean":"2024-03-18T20:26:39.625092+0000","last_became_active":"2024-03-18T20:26:39.623698+0000","last_became_peered":"2024-03-18T20:26:39.623698+0000","last_unstale":"2024-03-18T20:26:39.625092+0000","last_undegraded":"2024-03-18T20:26:39.625092+0000","last_fullsized":"2024-03-18T20:26:39.625092+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:54:42.615870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"256'4","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.624260+0000","last_change":"2024-03-18T20:26:39.182576+0000","last_active":"2024-03-18T20:26:39.624260+0000","last_peered":"2024-03-18T20:26:39.624260+0000","last_clean":"2024-03-18T20:26:39.624260+0000","last_became_active":"2024-03-18T20:26:39.182244+0000","last_became_peered":"2024-03-18T20:26:39.182244+0000","last_unstale":"2024-03-18T20:26:39.624260+0000","last_undegraded":"2024-03-18T20:26:39.624260+0000","last_fullsized":"2024-03-18T20:26:39.624260+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:01.961647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.173247+0000","last_change":"2024-03-18T20:26:25.806337+0000","last_active":"2024-03-18T20:26:39.173247+0000","last_peered":"2024-03-18T20:26:39.173247+0000","last_clean":"2024-03-18T20:26:39.173247+0000","last_became_active":"2024-03-18T20:26:25.806016+0000","last_became_peered":"2024-03-18T20:26:25.806016+0000","last_unstale":"2024-03-18T20:26:39.173247+0000","last_undegraded":"2024-03-18T20:26:39.173247+0000","last_fullsized":"2024-03-18T20:26:39.173247+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:33:55.816835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.626048+0000","last_change":"2024-03-18T20:26:39.617752+0000","last_active":"2024-03-18T20:26:39.626048+0000","last_peered":"2024-03-18T20:26:39.626048+0000","last_clean":"2024-03-18T20:26:39.626048+0000","last_became_active":"2024-03-18T20:26:39.617409+0000","last_became_peered":"2024-03-18T20:26:39.617409+0000","last_unstale":"2024-03-18T20:26:39.626048+0000","last_undegraded":"2024-03-18T20:26:39.626048+0000","last_fullsized":"2024-03-18T20:26:39.626048+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:59:55.686698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"256'2","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.616698+0000","last_change":"2024-03-18T20:26:39.173108+0000","last_active":"2024-03-18T20:26:39.616698+0000","last_peered":"2024-03-18T20:26:39.616698+0000","last_clean":"2024-03-18T20:26:39.616698+0000","last_became_active":"2024-03-18T20:26:39.172828+0000","last_became_peered":"2024-03-18T20:26:39.172828+0000","last_unstale":"2024-03-18T20:26:39.616698+0000","last_undegraded":"2024-03-18T20:26:39.616698+0000","last_fullsized":"2024-03-18T20:26:39.616698+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:14:31.591405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.165765+0000","last_change":"2024-03-18T20:26:25.815050+0000","last_active":"2024-03-18T20:26:39.165765+0000","last_peered":"2024-03-18T20:26:39.165765+0000","last_clean":"2024-03-18T20:26:39.165765+0000","last_became_active":"2024-03-18T20:26:25.814617+0000","last_became_peered":"2024-03-18T20:26:25.814617+0000","last_unstale":"2024-03-18T20:26:39.165765+0000","last_undegraded":"2024-03-18T20:26:39.165765+0000","last_fullsized":"2024-03-18T20:26:39.165765+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:04:32.495090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.632378+0000","last_change":"2024-03-18T20:26:39.631768+0000","last_active":"2024-03-18T20:26:39.632378+0000","last_peered":"2024-03-18T20:26:39.632378+0000","last_clean":"2024-03-18T20:26:39.632378+0000","last_became_active":"2024-03-18T20:26:39.631349+0000","last_became_peered":"2024-03-18T20:26:39.631349+0000","last_unstale":"2024-03-18T20:26:39.632378+0000","last_undegraded":"2024-03-18T20:26:39.632378+0000","last_fullsized":"2024-03-18T20:26:39.632378+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:35.406128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"256'2","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.619823+0000","last_change":"2024-03-18T20:26:39.186202+0000","last_active":"2024-03-18T20:26:39.619823+0000","last_peered":"2024-03-18T20:26:39.619823+0000","last_clean":"2024-03-18T20:26:39.619823+0000","last_became_active":"2024-03-18T20:26:39.185958+0000","last_became_peered":"2024-03-18T20:26:39.185958+0000","last_unstale":"2024-03-18T20:26:39.619823+0000","last_undegraded":"2024-03-18T20:26:39.619823+0000","last_fullsized":"2024-03-18T20:26:39.619823+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:37:53.931774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166170+0000","last_change":"2024-03-18T20:26:25.812709+0000","last_active":"2024-03-18T20:26:39.166170+0000","last_peered":"2024-03-18T20:26:39.166170+0000","last_clean":"2024-03-18T20:26:39.166170+0000","last_became_active":"2024-03-18T20:26:25.812420+0000","last_became_peered":"2024-03-18T20:26:25.812420+0000","last_unstale":"2024-03-18T20:26:39.166170+0000","last_undegraded":"2024-03-18T20:26:39.166170+0000","last_fullsized":"2024-03-18T20:26:39.166170+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:52.955865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.617657+0000","last_change":"2024-03-18T20:26:39.614084+0000","last_active":"2024-03-18T20:26:39.617657+0000","last_peered":"2024-03-18T20:26:39.617657+0000","last_clean":"2024-03-18T20:26:39.617657+0000","last_became_active":"2024-03-18T20:26:39.613535+0000","last_became_peered":"2024-03-18T20:26:39.613535+0000","last_unstale":"2024-03-18T20:26:39.617657+0000","last_undegraded":"2024-03-18T20:26:39.617657+0000","last_fullsized":"2024-03-18T20:26:39.617657+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:14:33.759409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"256'1","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.625221+0000","last_change":"2024-03-18T20:26:39.186135+0000","last_active":"2024-03-18T20:26:39.625221+0000","last_peered":"2024-03-18T20:26:39.625221+0000","last_clean":"2024-03-18T20:26:39.625221+0000","last_became_active":"2024-03-18T20:26:39.185890+0000","last_became_peered":"2024-03-18T20:26:39.185890+0000","last_unstale":"2024-03-18T20:26:39.625221+0000","last_undegraded":"2024-03-18T20:26:39.625221+0000","last_fullsized":"2024-03-18T20:26:39.625221+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:41:38.363283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166761+0000","last_change":"2024-03-18T20:26:25.811447+0000","last_active":"2024-03-18T20:26:39.166761+0000","last_peered":"2024-03-18T20:26:39.166761+0000","last_clean":"2024-03-18T20:26:39.166761+0000","last_became_active":"2024-03-18T20:26:25.811144+0000","last_became_peered":"2024-03-18T20:26:25.811144+0000","last_unstale":"2024-03-18T20:26:39.166761+0000","last_undegraded":"2024-03-18T20:26:39.166761+0000","last_fullsized":"2024-03-18T20:26:39.166761+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:54:38.189655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629539+0000","last_change":"2024-03-18T20:26:39.624778+0000","last_active":"2024-03-18T20:26:39.629539+0000","last_peered":"2024-03-18T20:26:39.629539+0000","last_clean":"2024-03-18T20:26:39.629539+0000","last_became_active":"2024-03-18T20:26:39.621807+0000","last_became_peered":"2024-03-18T20:26:39.621807+0000","last_unstale":"2024-03-18T20:26:39.629539+0000","last_undegraded":"2024-03-18T20:26:39.629539+0000","last_fullsized":"2024-03-18T20:26:39.629539+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:29.701161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"256'1","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.621775+0000","last_change":"2024-03-18T20:26:39.179129+0000","last_active":"2024-03-18T20:26:39.621775+0000","last_peered":"2024-03-18T20:26:39.621775+0000","last_clean":"2024-03-18T20:26:39.621775+0000","last_became_active":"2024-03-18T20:26:39.178863+0000","last_became_peered":"2024-03-18T20:26:39.178863+0000","last_unstale":"2024-03-18T20:26:39.621775+0000","last_undegraded":"2024-03-18T20:26:39.621775+0000","last_fullsized":"2024-03-18T20:26:39.621775+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:16:23.162470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.173083+0000","last_change":"2024-03-18T20:26:25.804382+0000","last_active":"2024-03-18T20:26:39.173083+0000","last_peered":"2024-03-18T20:26:39.173083+0000","last_clean":"2024-03-18T20:26:39.173083+0000","last_became_active":"2024-03-18T20:26:25.804096+0000","last_became_peered":"2024-03-18T20:26:25.804096+0000","last_unstale":"2024-03-18T20:26:39.173083+0000","last_undegraded":"2024-03-18T20:26:39.173083+0000","last_fullsized":"2024-03-18T20:26:39.173083+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:23.578829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629593+0000","last_change":"2024-03-18T20:26:39.627440+0000","last_active":"2024-03-18T20:26:39.629593+0000","last_peered":"2024-03-18T20:26:39.629593+0000","last_clean":"2024-03-18T20:26:39.629593+0000","last_became_active":"2024-03-18T20:26:39.626732+0000","last_became_peered":"2024-03-18T20:26:39.626732+0000","last_unstale":"2024-03-18T20:26:39.629593+0000","last_undegraded":"2024-03-18T20:26:39.629593+0000","last_fullsized":"2024-03-18T20:26:39.629593+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:23:05.984738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.601980+0000","last_change":"2024-03-18T20:26:39.172303+0000","last_active":"2024-03-18T20:26:39.601980+0000","last_peered":"2024-03-18T20:26:39.601980+0000","last_clean":"2024-03-18T20:26:39.601980+0000","last_became_active":"2024-03-18T20:26:39.172023+0000","last_became_peered":"2024-03-18T20:26:39.172023+0000","last_unstale":"2024-03-18T20:26:39.601980+0000","last_undegraded":"2024-03-18T20:26:39.601980+0000","last_fullsized":"2024-03-18T20:26:39.601980+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:22:32.033460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.172295+0000","last_change":"2024-03-18T20:26:25.806198+0000","last_active":"2024-03-18T20:26:39.172295+0000","last_peered":"2024-03-18T20:26:39.172295+0000","last_clean":"2024-03-18T20:26:39.172295+0000","last_became_active":"2024-03-18T20:26:25.805918+0000","last_became_peered":"2024-03-18T20:26:25.805918+0000","last_unstale":"2024-03-18T20:26:39.172295+0000","last_undegraded":"2024-03-18T20:26:39.172295+0000","last_fullsized":"2024-03-18T20:26:39.172295+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:11.977302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629624+0000","last_change":"2024-03-18T20:26:39.629257+0000","last_active":"2024-03-18T20:26:39.629624+0000","last_peered":"2024-03-18T20:26:39.629624+0000","last_clean":"2024-03-18T20:26:39.629624+0000","last_became_active":"2024-03-18T20:26:39.628990+0000","last_became_peered":"2024-03-18T20:26:39.628990+0000","last_unstale":"2024-03-18T20:26:39.629624+0000","last_undegraded":"2024-03-18T20:26:39.629624+0000","last_fullsized":"2024-03-18T20:26:39.629624+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:14:12.949096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.601363+0000","last_change":"2024-03-18T20:26:39.172866+0000","last_active":"2024-03-18T20:26:39.601363+0000","last_peered":"2024-03-18T20:26:39.601363+0000","last_clean":"2024-03-18T20:26:39.601363+0000","last_became_active":"2024-03-18T20:26:39.172609+0000","last_became_peered":"2024-03-18T20:26:39.172609+0000","last_unstale":"2024-03-18T20:26:39.601363+0000","last_undegraded":"2024-03-18T20:26:39.601363+0000","last_fullsized":"2024-03-18T20:26:39.601363+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:21.784700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.165779+0000","last_change":"2024-03-18T20:26:25.805458+0000","last_active":"2024-03-18T20:26:39.165779+0000","last_peered":"2024-03-18T20:26:39.165779+0000","last_clean":"2024-03-18T20:26:39.165779+0000","last_became_active":"2024-03-18T20:26:25.805176+0000","last_became_peered":"2024-03-18T20:26:25.805176+0000","last_unstale":"2024-03-18T20:26:39.165779+0000","last_undegraded":"2024-03-18T20:26:39.165779+0000","last_fullsized":"2024-03-18T20:26:39.165779+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:06:22.720260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.621416+0000","last_change":"2024-03-18T20:26:39.614803+0000","last_active":"2024-03-18T20:26:39.621416+0000","last_peered":"2024-03-18T20:26:39.621416+0000","last_clean":"2024-03-18T20:26:39.621416+0000","last_became_active":"2024-03-18T20:26:39.614214+0000","last_became_peered":"2024-03-18T20:26:39.614214+0000","last_unstale":"2024-03-18T20:26:39.621416+0000","last_undegraded":"2024-03-18T20:26:39.621416+0000","last_fullsized":"2024-03-18T20:26:39.621416+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:48:12.952796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.606260+0000","last_change":"2024-03-18T20:26:39.172945+0000","last_active":"2024-03-18T20:26:39.606260+0000","last_peered":"2024-03-18T20:26:39.606260+0000","last_clean":"2024-03-18T20:26:39.606260+0000","last_became_active":"2024-03-18T20:26:39.172504+0000","last_became_peered":"2024-03-18T20:26:39.172504+0000","last_unstale":"2024-03-18T20:26:39.606260+0000","last_undegraded":"2024-03-18T20:26:39.606260+0000","last_fullsized":"2024-03-18T20:26:39.606260+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:22.609793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174207+0000","last_change":"2024-03-18T20:26:25.815665+0000","last_active":"2024-03-18T20:26:39.174207+0000","last_peered":"2024-03-18T20:26:39.174207+0000","last_clean":"2024-03-18T20:26:39.174207+0000","last_became_active":"2024-03-18T20:26:25.815259+0000","last_became_peered":"2024-03-18T20:26:25.815259+0000","last_unstale":"2024-03-18T20:26:39.174207+0000","last_undegraded":"2024-03-18T20:26:39.174207+0000","last_fullsized":"2024-03-18T20:26:39.174207+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:39:11.551926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.621981+0000","last_change":"2024-03-18T20:26:39.619565+0000","last_active":"2024-03-18T20:26:39.621981+0000","last_peered":"2024-03-18T20:26:39.621981+0000","last_clean":"2024-03-18T20:26:39.621981+0000","last_became_active":"2024-03-18T20:26:39.619317+0000","last_became_peered":"2024-03-18T20:26:39.619317+0000","last_unstale":"2024-03-18T20:26:39.621981+0000","last_undegraded":"2024-03-18T20:26:39.621981+0000","last_fullsized":"2024-03-18T20:26:39.621981+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:39:58.561762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.611666+0000","last_change":"2024-03-18T20:26:39.180298+0000","last_active":"2024-03-18T20:26:39.611666+0000","last_peered":"2024-03-18T20:26:39.611666+0000","last_clean":"2024-03-18T20:26:39.611666+0000","last_became_active":"2024-03-18T20:26:39.179626+0000","last_became_peered":"2024-03-18T20:26:39.179626+0000","last_unstale":"2024-03-18T20:26:39.611666+0000","last_undegraded":"2024-03-18T20:26:39.611666+0000","last_fullsized":"2024-03-18T20:26:39.611666+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:32:02.878109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.165829+0000","last_change":"2024-03-18T20:26:25.805234+0000","last_active":"2024-03-18T20:26:39.165829+0000","last_peered":"2024-03-18T20:26:39.165829+0000","last_clean":"2024-03-18T20:26:39.165829+0000","last_became_active":"2024-03-18T20:26:25.805056+0000","last_became_peered":"2024-03-18T20:26:25.805056+0000","last_unstale":"2024-03-18T20:26:39.165829+0000","last_undegraded":"2024-03-18T20:26:39.165829+0000","last_fullsized":"2024-03-18T20:26:39.165829+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:50.003103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.626369+0000","last_change":"2024-03-18T20:26:39.620111+0000","last_active":"2024-03-18T20:26:39.626369+0000","last_peered":"2024-03-18T20:26:39.626369+0000","last_clean":"2024-03-18T20:26:39.626369+0000","last_became_active":"2024-03-18T20:26:39.619855+0000","last_became_peered":"2024-03-18T20:26:39.619855+0000","last_unstale":"2024-03-18T20:26:39.626369+0000","last_undegraded":"2024-03-18T20:26:39.626369+0000","last_fullsized":"2024-03-18T20:26:39.626369+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:22:44.464056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"256'6","reported_seq":16,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.632214+0000","last_change":"2024-03-18T20:26:39.180393+0000","last_active":"2024-03-18T20:26:39.632214+0000","last_peered":"2024-03-18T20:26:39.632214+0000","last_clean":"2024-03-18T20:26:39.632214+0000","last_became_active":"2024-03-18T20:26:39.180140+0000","last_became_peered":"2024-03-18T20:26:39.180140+0000","last_unstale":"2024-03-18T20:26:39.632214+0000","last_undegraded":"2024-03-18T20:26:39.632214+0000","last_fullsized":"2024-03-18T20:26:39.632214+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:32.551571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.165802+0000","last_change":"2024-03-18T20:26:25.813598+0000","last_active":"2024-03-18T20:26:39.165802+0000","last_peered":"2024-03-18T20:26:39.165802+0000","last_clean":"2024-03-18T20:26:39.165802+0000","last_became_active":"2024-03-18T20:26:25.812093+0000","last_became_peered":"2024-03-18T20:26:25.812093+0000","last_unstale":"2024-03-18T20:26:39.165802+0000","last_undegraded":"2024-03-18T20:26:39.165802+0000","last_fullsized":"2024-03-18T20:26:39.165802+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:38.417379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.628746+0000","last_change":"2024-03-18T20:26:39.628200+0000","last_active":"2024-03-18T20:26:39.628746+0000","last_peered":"2024-03-18T20:26:39.628746+0000","last_clean":"2024-03-18T20:26:39.628746+0000","last_became_active":"2024-03-18T20:26:39.627978+0000","last_became_peered":"2024-03-18T20:26:39.627978+0000","last_unstale":"2024-03-18T20:26:39.628746+0000","last_undegraded":"2024-03-18T20:26:39.628746+0000","last_fullsized":"2024-03-18T20:26:39.628746+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:39:20.975255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"256'3","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.628662+0000","last_change":"2024-03-18T20:26:39.177990+0000","last_active":"2024-03-18T20:26:39.628662+0000","last_peered":"2024-03-18T20:26:39.628662+0000","last_clean":"2024-03-18T20:26:39.628662+0000","last_became_active":"2024-03-18T20:26:39.177721+0000","last_became_peered":"2024-03-18T20:26:39.177721+0000","last_unstale":"2024-03-18T20:26:39.628662+0000","last_undegraded":"2024-03-18T20:26:39.628662+0000","last_fullsized":"2024-03-18T20:26:39.628662+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:21:35.960350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174103+0000","last_change":"2024-03-18T20:26:25.811095+0000","last_active":"2024-03-18T20:26:39.174103+0000","last_peered":"2024-03-18T20:26:39.174103+0000","last_clean":"2024-03-18T20:26:39.174103+0000","last_became_active":"2024-03-18T20:26:25.810796+0000","last_became_peered":"2024-03-18T20:26:25.810796+0000","last_unstale":"2024-03-18T20:26:39.174103+0000","last_undegraded":"2024-03-18T20:26:39.174103+0000","last_fullsized":"2024-03-18T20:26:39.174103+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:30:08.833713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.624868+0000","last_change":"2024-03-18T20:26:39.619308+0000","last_active":"2024-03-18T20:26:39.624868+0000","last_peered":"2024-03-18T20:26:39.624868+0000","last_clean":"2024-03-18T20:26:39.624868+0000","last_became_active":"2024-03-18T20:26:39.619018+0000","last_became_peered":"2024-03-18T20:26:39.619018+0000","last_unstale":"2024-03-18T20:26:39.624868+0000","last_undegraded":"2024-03-18T20:26:39.624868+0000","last_fullsized":"2024-03-18T20:26:39.624868+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:14:12.949096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.619228+0000","last_change":"2024-03-18T20:26:39.177301+0000","last_active":"2024-03-18T20:26:39.619228+0000","last_peered":"2024-03-18T20:26:39.619228+0000","last_clean":"2024-03-18T20:26:39.619228+0000","last_became_active":"2024-03-18T20:26:39.176934+0000","last_became_peered":"2024-03-18T20:26:39.176934+0000","last_unstale":"2024-03-18T20:26:39.619228+0000","last_undegraded":"2024-03-18T20:26:39.619228+0000","last_fullsized":"2024-03-18T20:26:39.619228+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:37:06.271998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174153+0000","last_change":"2024-03-18T20:26:25.815198+0000","last_active":"2024-03-18T20:26:39.174153+0000","last_peered":"2024-03-18T20:26:39.174153+0000","last_clean":"2024-03-18T20:26:39.174153+0000","last_became_active":"2024-03-18T20:26:25.814922+0000","last_became_peered":"2024-03-18T20:26:25.814922+0000","last_unstale":"2024-03-18T20:26:39.174153+0000","last_undegraded":"2024-03-18T20:26:39.174153+0000","last_fullsized":"2024-03-18T20:26:39.174153+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:30:14.320846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.614689+0000","last_change":"2024-03-18T20:26:39.609822+0000","last_active":"2024-03-18T20:26:39.614689+0000","last_peered":"2024-03-18T20:26:39.614689+0000","last_clean":"2024-03-18T20:26:39.614689+0000","last_became_active":"2024-03-18T20:26:39.609495+0000","last_became_peered":"2024-03-18T20:26:39.609495+0000","last_unstale":"2024-03-18T20:26:39.614689+0000","last_undegraded":"2024-03-18T20:26:39.614689+0000","last_fullsized":"2024-03-18T20:26:39.614689+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:57.528824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"256'2","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.622034+0000","last_change":"2024-03-18T20:26:39.174564+0000","last_active":"2024-03-18T20:26:39.622034+0000","last_peered":"2024-03-18T20:26:39.622034+0000","last_clean":"2024-03-18T20:26:39.622034+0000","last_became_active":"2024-03-18T20:26:39.172872+0000","last_became_peered":"2024-03-18T20:26:39.172872+0000","last_unstale":"2024-03-18T20:26:39.622034+0000","last_undegraded":"2024-03-18T20:26:39.622034+0000","last_fullsized":"2024-03-18T20:26:39.622034+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:16.137977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.184511+0000","last_change":"2024-03-18T20:26:25.809230+0000","last_active":"2024-03-18T20:26:39.184511+0000","last_peered":"2024-03-18T20:26:39.184511+0000","last_clean":"2024-03-18T20:26:39.184511+0000","last_became_active":"2024-03-18T20:26:25.808941+0000","last_became_peered":"2024-03-18T20:26:25.808941+0000","last_unstale":"2024-03-18T20:26:39.184511+0000","last_undegraded":"2024-03-18T20:26:39.184511+0000","last_fullsized":"2024-03-18T20:26:39.184511+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:34:05.214460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.619296+0000","last_change":"2024-03-18T20:26:39.615032+0000","last_active":"2024-03-18T20:26:39.619296+0000","last_peered":"2024-03-18T20:26:39.619296+0000","last_clean":"2024-03-18T20:26:39.619296+0000","last_became_active":"2024-03-18T20:26:39.614785+0000","last_became_peered":"2024-03-18T20:26:39.614785+0000","last_unstale":"2024-03-18T20:26:39.619296+0000","last_undegraded":"2024-03-18T20:26:39.619296+0000","last_fullsized":"2024-03-18T20:26:39.619296+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:30:36.759356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"256'8","reported_seq":18,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.632913+0000","last_change":"2024-03-18T20:26:39.184566+0000","last_active":"2024-03-18T20:26:39.632913+0000","last_peered":"2024-03-18T20:26:39.632913+0000","last_clean":"2024-03-18T20:26:39.632913+0000","last_became_active":"2024-03-18T20:26:39.184288+0000","last_became_peered":"2024-03-18T20:26:39.184288+0000","last_unstale":"2024-03-18T20:26:39.632913+0000","last_undegraded":"2024-03-18T20:26:39.632913+0000","last_fullsized":"2024-03-18T20:26:39.632913+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:27:08.199457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164301+0000","last_change":"2024-03-18T20:26:25.798972+0000","last_active":"2024-03-18T20:26:39.164301+0000","last_peered":"2024-03-18T20:26:39.164301+0000","last_clean":"2024-03-18T20:26:39.164301+0000","last_became_active":"2024-03-18T20:26:25.798736+0000","last_became_peered":"2024-03-18T20:26:25.798736+0000","last_unstale":"2024-03-18T20:26:39.164301+0000","last_undegraded":"2024-03-18T20:26:39.164301+0000","last_fullsized":"2024-03-18T20:26:39.164301+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:56.892630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.614724+0000","last_change":"2024-03-18T20:26:39.612586+0000","last_active":"2024-03-18T20:26:39.614724+0000","last_peered":"2024-03-18T20:26:39.614724+0000","last_clean":"2024-03-18T20:26:39.614724+0000","last_became_active":"2024-03-18T20:26:39.612337+0000","last_became_peered":"2024-03-18T20:26:39.612337+0000","last_unstale":"2024-03-18T20:26:39.614724+0000","last_undegraded":"2024-03-18T20:26:39.614724+0000","last_fullsized":"2024-03-18T20:26:39.614724+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:14:33.759409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.619961+0000","last_change":"2024-03-18T20:26:39.178899+0000","last_active":"2024-03-18T20:26:39.619961+0000","last_peered":"2024-03-18T20:26:39.619961+0000","last_clean":"2024-03-18T20:26:39.619961+0000","last_became_active":"2024-03-18T20:26:39.178634+0000","last_became_peered":"2024-03-18T20:26:39.178634+0000","last_unstale":"2024-03-18T20:26:39.619961+0000","last_undegraded":"2024-03-18T20:26:39.619961+0000","last_fullsized":"2024-03-18T20:26:39.619961+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:16:43.290784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.165956+0000","last_change":"2024-03-18T20:26:25.799408+0000","last_active":"2024-03-18T20:26:39.165956+0000","last_peered":"2024-03-18T20:26:39.165956+0000","last_clean":"2024-03-18T20:26:39.165956+0000","last_became_active":"2024-03-18T20:26:25.798788+0000","last_became_peered":"2024-03-18T20:26:25.798788+0000","last_unstale":"2024-03-18T20:26:39.165956+0000","last_undegraded":"2024-03-18T20:26:39.165956+0000","last_fullsized":"2024-03-18T20:26:39.165956+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:34:05.214460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.621773+0000","last_change":"2024-03-18T20:26:39.618070+0000","last_active":"2024-03-18T20:26:39.621773+0000","last_peered":"2024-03-18T20:26:39.621773+0000","last_clean":"2024-03-18T20:26:39.621773+0000","last_became_active":"2024-03-18T20:26:39.617706+0000","last_became_peered":"2024-03-18T20:26:39.617706+0000","last_unstale":"2024-03-18T20:26:39.621773+0000","last_undegraded":"2024-03-18T20:26:39.621773+0000","last_fullsized":"2024-03-18T20:26:39.621773+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:20.170314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"256'2","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.625601+0000","last_change":"2024-03-18T20:26:39.182659+0000","last_active":"2024-03-18T20:26:39.625601+0000","last_peered":"2024-03-18T20:26:39.625601+0000","last_clean":"2024-03-18T20:26:39.625601+0000","last_became_active":"2024-03-18T20:26:39.180908+0000","last_became_peered":"2024-03-18T20:26:39.180908+0000","last_unstale":"2024-03-18T20:26:39.625601+0000","last_undegraded":"2024-03-18T20:26:39.625601+0000","last_fullsized":"2024-03-18T20:26:39.625601+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:37:53.931774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.175207+0000","last_change":"2024-03-18T20:26:25.811390+0000","last_active":"2024-03-18T20:26:39.175207+0000","last_peered":"2024-03-18T20:26:39.175207+0000","last_clean":"2024-03-18T20:26:39.175207+0000","last_became_active":"2024-03-18T20:26:25.811081+0000","last_became_peered":"2024-03-18T20:26:25.811081+0000","last_unstale":"2024-03-18T20:26:39.175207+0000","last_undegraded":"2024-03-18T20:26:39.175207+0000","last_fullsized":"2024-03-18T20:26:39.175207+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:33:55.816835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.627069+0000","last_change":"2024-03-18T20:26:39.626443+0000","last_active":"2024-03-18T20:26:39.627069+0000","last_peered":"2024-03-18T20:26:39.627069+0000","last_clean":"2024-03-18T20:26:39.627069+0000","last_became_active":"2024-03-18T20:26:39.626065+0000","last_became_peered":"2024-03-18T20:26:39.626065+0000","last_unstale":"2024-03-18T20:26:39.627069+0000","last_undegraded":"2024-03-18T20:26:39.627069+0000","last_fullsized":"2024-03-18T20:26:39.627069+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:48:12.952796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"256'1","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.624610+0000","last_change":"2024-03-18T20:26:39.179734+0000","last_active":"2024-03-18T20:26:39.624610+0000","last_peered":"2024-03-18T20:26:39.624610+0000","last_clean":"2024-03-18T20:26:39.624610+0000","last_became_active":"2024-03-18T20:26:39.178954+0000","last_became_peered":"2024-03-18T20:26:39.178954+0000","last_unstale":"2024-03-18T20:26:39.624610+0000","last_undegraded":"2024-03-18T20:26:39.624610+0000","last_fullsized":"2024-03-18T20:26:39.624610+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:48:11.943600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.168586+0000","last_change":"2024-03-18T20:26:25.799778+0000","last_active":"2024-03-18T20:26:39.168586+0000","last_peered":"2024-03-18T20:26:39.168586+0000","last_clean":"2024-03-18T20:26:39.168586+0000","last_became_active":"2024-03-18T20:26:25.799517+0000","last_became_peered":"2024-03-18T20:26:25.799517+0000","last_unstale":"2024-03-18T20:26:39.168586+0000","last_undegraded":"2024-03-18T20:26:39.168586+0000","last_fullsized":"2024-03-18T20:26:39.168586+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:32:34.214872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.626393+0000","last_change":"2024-03-18T20:26:39.621407+0000","last_active":"2024-03-18T20:26:39.626393+0000","last_peered":"2024-03-18T20:26:39.626393+0000","last_clean":"2024-03-18T20:26:39.626393+0000","last_became_active":"2024-03-18T20:26:39.620628+0000","last_became_peered":"2024-03-18T20:26:39.620628+0000","last_unstale":"2024-03-18T20:26:39.626393+0000","last_undegraded":"2024-03-18T20:26:39.626393+0000","last_fullsized":"2024-03-18T20:26:39.626393+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:27:47.040042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.607231+0000","last_change":"2024-03-18T20:26:39.179668+0000","last_active":"2024-03-18T20:26:39.607231+0000","last_peered":"2024-03-18T20:26:39.607231+0000","last_clean":"2024-03-18T20:26:39.607231+0000","last_became_active":"2024-03-18T20:26:39.179415+0000","last_became_peered":"2024-03-18T20:26:39.179415+0000","last_unstale":"2024-03-18T20:26:39.607231+0000","last_undegraded":"2024-03-18T20:26:39.607231+0000","last_fullsized":"2024-03-18T20:26:39.607231+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:15.672080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164327+0000","last_change":"2024-03-18T20:26:25.812369+0000","last_active":"2024-03-18T20:26:39.164327+0000","last_peered":"2024-03-18T20:26:39.164327+0000","last_clean":"2024-03-18T20:26:39.164327+0000","last_became_active":"2024-03-18T20:26:25.812082+0000","last_became_peered":"2024-03-18T20:26:25.812082+0000","last_unstale":"2024-03-18T20:26:39.164327+0000","last_undegraded":"2024-03-18T20:26:39.164327+0000","last_fullsized":"2024-03-18T20:26:39.164327+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:43:00.585492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.632057+0000","last_change":"2024-03-18T20:26:39.630173+0000","last_active":"2024-03-18T20:26:39.632057+0000","last_peered":"2024-03-18T20:26:39.632057+0000","last_clean":"2024-03-18T20:26:39.632057+0000","last_became_active":"2024-03-18T20:26:39.629870+0000","last_became_peered":"2024-03-18T20:26:39.629870+0000","last_unstale":"2024-03-18T20:26:39.632057+0000","last_undegraded":"2024-03-18T20:26:39.632057+0000","last_fullsized":"2024-03-18T20:26:39.632057+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:02:00.876556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"256'1","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629431+0000","last_change":"2024-03-18T20:26:39.186556+0000","last_active":"2024-03-18T20:26:39.629431+0000","last_peered":"2024-03-18T20:26:39.629431+0000","last_clean":"2024-03-18T20:26:39.629431+0000","last_became_active":"2024-03-18T20:26:39.186231+0000","last_became_peered":"2024-03-18T20:26:39.186231+0000","last_unstale":"2024-03-18T20:26:39.629431+0000","last_undegraded":"2024-03-18T20:26:39.629431+0000","last_fullsized":"2024-03-18T20:26:39.629431+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:21.784700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.171204+0000","last_change":"2024-03-18T20:26:25.807591+0000","last_active":"2024-03-18T20:26:39.171204+0000","last_peered":"2024-03-18T20:26:39.171204+0000","last_clean":"2024-03-18T20:26:39.171204+0000","last_became_active":"2024-03-18T20:26:25.807349+0000","last_became_peered":"2024-03-18T20:26:25.807349+0000","last_unstale":"2024-03-18T20:26:39.171204+0000","last_undegraded":"2024-03-18T20:26:39.171204+0000","last_fullsized":"2024-03-18T20:26:39.171204+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:58.931342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629648+0000","last_change":"2024-03-18T20:26:39.629099+0000","last_active":"2024-03-18T20:26:39.629648+0000","last_peered":"2024-03-18T20:26:39.629648+0000","last_clean":"2024-03-18T20:26:39.629648+0000","last_became_active":"2024-03-18T20:26:39.628813+0000","last_became_peered":"2024-03-18T20:26:39.628813+0000","last_unstale":"2024-03-18T20:26:39.629648+0000","last_undegraded":"2024-03-18T20:26:39.629648+0000","last_fullsized":"2024-03-18T20:26:39.629648+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:22.793896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.602364+0000","last_change":"2024-03-18T20:26:39.187040+0000","last_active":"2024-03-18T20:26:39.602364+0000","last_peered":"2024-03-18T20:26:39.602364+0000","last_clean":"2024-03-18T20:26:39.602364+0000","last_became_active":"2024-03-18T20:26:39.186729+0000","last_became_peered":"2024-03-18T20:26:39.186729+0000","last_unstale":"2024-03-18T20:26:39.602364+0000","last_undegraded":"2024-03-18T20:26:39.602364+0000","last_fullsized":"2024-03-18T20:26:39.602364+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:43.273058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.173622+0000","last_change":"2024-03-18T20:26:25.799427+0000","last_active":"2024-03-18T20:26:39.173622+0000","last_peered":"2024-03-18T20:26:39.173622+0000","last_clean":"2024-03-18T20:26:39.173622+0000","last_became_active":"2024-03-18T20:26:25.799041+0000","last_became_peered":"2024-03-18T20:26:25.799041+0000","last_unstale":"2024-03-18T20:26:39.173622+0000","last_undegraded":"2024-03-18T20:26:39.173622+0000","last_fullsized":"2024-03-18T20:26:39.173622+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:35.919143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.624605+0000","last_change":"2024-03-18T20:26:39.617792+0000","last_active":"2024-03-18T20:26:39.624605+0000","last_peered":"2024-03-18T20:26:39.624605+0000","last_clean":"2024-03-18T20:26:39.624605+0000","last_became_active":"2024-03-18T20:26:39.617372+0000","last_became_peered":"2024-03-18T20:26:39.617372+0000","last_unstale":"2024-03-18T20:26:39.624605+0000","last_undegraded":"2024-03-18T20:26:39.624605+0000","last_fullsized":"2024-03-18T20:26:39.624605+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:21:15.706308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"256'2","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.630616+0000","last_change":"2024-03-18T20:26:39.180590+0000","last_active":"2024-03-18T20:26:39.630616+0000","last_peered":"2024-03-18T20:26:39.630616+0000","last_clean":"2024-03-18T20:26:39.630616+0000","last_became_active":"2024-03-18T20:26:39.180324+0000","last_became_peered":"2024-03-18T20:26:39.180324+0000","last_unstale":"2024-03-18T20:26:39.630616+0000","last_undegraded":"2024-03-18T20:26:39.630616+0000","last_fullsized":"2024-03-18T20:26:39.630616+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:18:56.346574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.172919+0000","last_change":"2024-03-18T20:26:25.818643+0000","last_active":"2024-03-18T20:26:39.172919+0000","last_peered":"2024-03-18T20:26:39.172919+0000","last_clean":"2024-03-18T20:26:39.172919+0000","last_became_active":"2024-03-18T20:26:25.818377+0000","last_became_peered":"2024-03-18T20:26:25.818377+0000","last_unstale":"2024-03-18T20:26:39.172919+0000","last_undegraded":"2024-03-18T20:26:39.172919+0000","last_fullsized":"2024-03-18T20:26:39.172919+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:44.809054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629294+0000","last_change":"2024-03-18T20:26:39.623520+0000","last_active":"2024-03-18T20:26:39.629294+0000","last_peered":"2024-03-18T20:26:39.629294+0000","last_clean":"2024-03-18T20:26:39.629294+0000","last_became_active":"2024-03-18T20:26:39.623271+0000","last_became_peered":"2024-03-18T20:26:39.623271+0000","last_unstale":"2024-03-18T20:26:39.629294+0000","last_undegraded":"2024-03-18T20:26:39.629294+0000","last_fullsized":"2024-03-18T20:26:39.629294+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:33:28.667851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.621155+0000","last_change":"2024-03-18T20:26:39.181245+0000","last_active":"2024-03-18T20:26:39.621155+0000","last_peered":"2024-03-18T20:26:39.621155+0000","last_clean":"2024-03-18T20:26:39.621155+0000","last_became_active":"2024-03-18T20:26:39.181004+0000","last_became_peered":"2024-03-18T20:26:39.181004+0000","last_unstale":"2024-03-18T20:26:39.621155+0000","last_undegraded":"2024-03-18T20:26:39.621155+0000","last_fullsized":"2024-03-18T20:26:39.621155+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:04.469128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164452+0000","last_change":"2024-03-18T20:26:25.805834+0000","last_active":"2024-03-18T20:26:39.164452+0000","last_peered":"2024-03-18T20:26:39.164452+0000","last_clean":"2024-03-18T20:26:39.164452+0000","last_became_active":"2024-03-18T20:26:25.805491+0000","last_became_peered":"2024-03-18T20:26:25.805491+0000","last_unstale":"2024-03-18T20:26:39.164452+0000","last_undegraded":"2024-03-18T20:26:39.164452+0000","last_fullsized":"2024-03-18T20:26:39.164452+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:23.305150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.623087+0000","last_change":"2024-03-18T20:26:39.621764+0000","last_active":"2024-03-18T20:26:39.623087+0000","last_peered":"2024-03-18T20:26:39.623087+0000","last_clean":"2024-03-18T20:26:39.623087+0000","last_became_active":"2024-03-18T20:26:39.619894+0000","last_became_peered":"2024-03-18T20:26:39.619894+0000","last_unstale":"2024-03-18T20:26:39.623087+0000","last_undegraded":"2024-03-18T20:26:39.623087+0000","last_fullsized":"2024-03-18T20:26:39.623087+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:10:35.340285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.609745+0000","last_change":"2024-03-18T20:26:39.190146+0000","last_active":"2024-03-18T20:26:39.609745+0000","last_peered":"2024-03-18T20:26:39.609745+0000","last_clean":"2024-03-18T20:26:39.609745+0000","last_became_active":"2024-03-18T20:26:39.189849+0000","last_became_peered":"2024-03-18T20:26:39.189849+0000","last_unstale":"2024-03-18T20:26:39.609745+0000","last_undegraded":"2024-03-18T20:26:39.609745+0000","last_fullsized":"2024-03-18T20:26:39.609745+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:39:12.102903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.171482+0000","last_change":"2024-03-18T20:26:25.809091+0000","last_active":"2024-03-18T20:26:39.171482+0000","last_peered":"2024-03-18T20:26:39.171482+0000","last_clean":"2024-03-18T20:26:39.171482+0000","last_became_active":"2024-03-18T20:26:25.808800+0000","last_became_peered":"2024-03-18T20:26:25.808800+0000","last_unstale":"2024-03-18T20:26:39.171482+0000","last_undegraded":"2024-03-18T20:26:39.171482+0000","last_fullsized":"2024-03-18T20:26:39.171482+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:32.033370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.632278+0000","last_change":"2024-03-18T20:26:39.631978+0000","last_active":"2024-03-18T20:26:39.632278+0000","last_peered":"2024-03-18T20:26:39.632278+0000","last_clean":"2024-03-18T20:26:39.632278+0000","last_became_active":"2024-03-18T20:26:39.631690+0000","last_became_peered":"2024-03-18T20:26:39.631690+0000","last_unstale":"2024-03-18T20:26:39.632278+0000","last_undegraded":"2024-03-18T20:26:39.632278+0000","last_fullsized":"2024-03-18T20:26:39.632278+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:07:57.157110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.607518+0000","last_change":"2024-03-18T20:26:39.180973+0000","last_active":"2024-03-18T20:26:39.607518+0000","last_peered":"2024-03-18T20:26:39.607518+0000","last_clean":"2024-03-18T20:26:39.607518+0000","last_became_active":"2024-03-18T20:26:39.180658+0000","last_became_peered":"2024-03-18T20:26:39.180658+0000","last_unstale":"2024-03-18T20:26:39.607518+0000","last_undegraded":"2024-03-18T20:26:39.607518+0000","last_fullsized":"2024-03-18T20:26:39.607518+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:36.104811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.172675+0000","last_change":"2024-03-18T20:26:25.798738+0000","last_active":"2024-03-18T20:26:39.172675+0000","last_peered":"2024-03-18T20:26:39.172675+0000","last_clean":"2024-03-18T20:26:39.172675+0000","last_became_active":"2024-03-18T20:26:25.798480+0000","last_became_peered":"2024-03-18T20:26:25.798480+0000","last_unstale":"2024-03-18T20:26:39.172675+0000","last_undegraded":"2024-03-18T20:26:39.172675+0000","last_fullsized":"2024-03-18T20:26:39.172675+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:02.900783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.627064+0000","last_change":"2024-03-18T20:26:39.624864+0000","last_active":"2024-03-18T20:26:39.627064+0000","last_peered":"2024-03-18T20:26:39.627064+0000","last_clean":"2024-03-18T20:26:39.627064+0000","last_became_active":"2024-03-18T20:26:39.622509+0000","last_became_peered":"2024-03-18T20:26:39.622509+0000","last_unstale":"2024-03-18T20:26:39.627064+0000","last_undegraded":"2024-03-18T20:26:39.627064+0000","last_fullsized":"2024-03-18T20:26:39.627064+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:32:52.861404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"256'2","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629923+0000","last_change":"2024-03-18T20:26:39.181574+0000","last_active":"2024-03-18T20:26:39.629923+0000","last_peered":"2024-03-18T20:26:39.629923+0000","last_clean":"2024-03-18T20:26:39.629923+0000","last_became_active":"2024-03-18T20:26:39.180033+0000","last_became_peered":"2024-03-18T20:26:39.180033+0000","last_unstale":"2024-03-18T20:26:39.629923+0000","last_undegraded":"2024-03-18T20:26:39.629923+0000","last_fullsized":"2024-03-18T20:26:39.629923+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:31:04.670960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.175259+0000","last_change":"2024-03-18T20:26:25.806356+0000","last_active":"2024-03-18T20:26:39.175259+0000","last_peered":"2024-03-18T20:26:39.175259+0000","last_clean":"2024-03-18T20:26:39.175259+0000","last_became_active":"2024-03-18T20:26:25.805895+0000","last_became_peered":"2024-03-18T20:26:25.805895+0000","last_unstale":"2024-03-18T20:26:39.175259+0000","last_undegraded":"2024-03-18T20:26:39.175259+0000","last_fullsized":"2024-03-18T20:26:39.175259+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:06:16.285934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.626223+0000","last_change":"2024-03-18T20:26:39.624150+0000","last_active":"2024-03-18T20:26:39.626223+0000","last_peered":"2024-03-18T20:26:39.626223+0000","last_clean":"2024-03-18T20:26:39.626223+0000","last_became_active":"2024-03-18T20:26:39.622660+0000","last_became_peered":"2024-03-18T20:26:39.622660+0000","last_unstale":"2024-03-18T20:26:39.626223+0000","last_undegraded":"2024-03-18T20:26:39.626223+0000","last_fullsized":"2024-03-18T20:26:39.626223+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:08:10.924769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.609219+0000","last_change":"2024-03-18T20:26:39.187982+0000","last_active":"2024-03-18T20:26:39.609219+0000","last_peered":"2024-03-18T20:26:39.609219+0000","last_clean":"2024-03-18T20:26:39.609219+0000","last_became_active":"2024-03-18T20:26:39.187760+0000","last_became_peered":"2024-03-18T20:26:39.187760+0000","last_unstale":"2024-03-18T20:26:39.609219+0000","last_undegraded":"2024-03-18T20:26:39.609219+0000","last_fullsized":"2024-03-18T20:26:39.609219+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:56.821401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.167507+0000","last_change":"2024-03-18T20:26:25.806860+0000","last_active":"2024-03-18T20:26:39.167507+0000","last_peered":"2024-03-18T20:26:39.167507+0000","last_clean":"2024-03-18T20:26:39.167507+0000","last_became_active":"2024-03-18T20:26:25.806650+0000","last_became_peered":"2024-03-18T20:26:25.806650+0000","last_unstale":"2024-03-18T20:26:39.167507+0000","last_undegraded":"2024-03-18T20:26:39.167507+0000","last_fullsized":"2024-03-18T20:26:39.167507+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:06:16.285934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.619586+0000","last_change":"2024-03-18T20:26:39.618254+0000","last_active":"2024-03-18T20:26:39.619586+0000","last_peered":"2024-03-18T20:26:39.619586+0000","last_clean":"2024-03-18T20:26:39.619586+0000","last_became_active":"2024-03-18T20:26:39.617947+0000","last_became_peered":"2024-03-18T20:26:39.617947+0000","last_unstale":"2024-03-18T20:26:39.619586+0000","last_undegraded":"2024-03-18T20:26:39.619586+0000","last_fullsized":"2024-03-18T20:26:39.619586+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:34:19.023316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.612360+0000","last_change":"2024-03-18T20:26:39.177881+0000","last_active":"2024-03-18T20:26:39.612360+0000","last_peered":"2024-03-18T20:26:39.612360+0000","last_clean":"2024-03-18T20:26:39.612360+0000","last_became_active":"2024-03-18T20:26:39.177666+0000","last_became_peered":"2024-03-18T20:26:39.177666+0000","last_unstale":"2024-03-18T20:26:39.612360+0000","last_undegraded":"2024-03-18T20:26:39.612360+0000","last_fullsized":"2024-03-18T20:26:39.612360+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:22.609793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174282+0000","last_change":"2024-03-18T20:26:25.815611+0000","last_active":"2024-03-18T20:26:39.174282+0000","last_peered":"2024-03-18T20:26:39.174282+0000","last_clean":"2024-03-18T20:26:39.174282+0000","last_became_active":"2024-03-18T20:26:25.815172+0000","last_became_peered":"2024-03-18T20:26:25.815172+0000","last_unstale":"2024-03-18T20:26:39.174282+0000","last_undegraded":"2024-03-18T20:26:39.174282+0000","last_fullsized":"2024-03-18T20:26:39.174282+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:48:15.078756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.633907+0000","last_change":"2024-03-18T20:26:39.631495+0000","last_active":"2024-03-18T20:26:39.633907+0000","last_peered":"2024-03-18T20:26:39.633907+0000","last_clean":"2024-03-18T20:26:39.633907+0000","last_became_active":"2024-03-18T20:26:39.629341+0000","last_became_peered":"2024-03-18T20:26:39.629341+0000","last_unstale":"2024-03-18T20:26:39.633907+0000","last_undegraded":"2024-03-18T20:26:39.633907+0000","last_fullsized":"2024-03-18T20:26:39.633907+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:47:27.910466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.606729+0000","last_change":"2024-03-18T20:26:39.183122+0000","last_active":"2024-03-18T20:26:39.606729+0000","last_peered":"2024-03-18T20:26:39.606729+0000","last_clean":"2024-03-18T20:26:39.606729+0000","last_became_active":"2024-03-18T20:26:39.182830+0000","last_became_peered":"2024-03-18T20:26:39.182830+0000","last_unstale":"2024-03-18T20:26:39.606729+0000","last_undegraded":"2024-03-18T20:26:39.606729+0000","last_fullsized":"2024-03-18T20:26:39.606729+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:32:45.875046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.185621+0000","last_change":"2024-03-18T20:26:25.810485+0000","last_active":"2024-03-18T20:26:39.185621+0000","last_peered":"2024-03-18T20:26:39.185621+0000","last_clean":"2024-03-18T20:26:39.185621+0000","last_became_active":"2024-03-18T20:26:25.810235+0000","last_became_peered":"2024-03-18T20:26:25.810235+0000","last_unstale":"2024-03-18T20:26:39.185621+0000","last_undegraded":"2024-03-18T20:26:39.185621+0000","last_fullsized":"2024-03-18T20:26:39.185621+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:59.140240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.634232+0000","last_change":"2024-03-18T20:26:39.633292+0000","last_active":"2024-03-18T20:26:39.634232+0000","last_peered":"2024-03-18T20:26:39.634232+0000","last_clean":"2024-03-18T20:26:39.634232+0000","last_became_active":"2024-03-18T20:26:39.633138+0000","last_became_peered":"2024-03-18T20:26:39.633138+0000","last_unstale":"2024-03-18T20:26:39.634232+0000","last_undegraded":"2024-03-18T20:26:39.634232+0000","last_fullsized":"2024-03-18T20:26:39.634232+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:30:32.988249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.607849+0000","last_change":"2024-03-18T20:26:39.182439+0000","last_active":"2024-03-18T20:26:39.607849+0000","last_peered":"2024-03-18T20:26:39.607849+0000","last_clean":"2024-03-18T20:26:39.607849+0000","last_became_active":"2024-03-18T20:26:39.181085+0000","last_became_peered":"2024-03-18T20:26:39.181085+0000","last_unstale":"2024-03-18T20:26:39.607849+0000","last_undegraded":"2024-03-18T20:26:39.607849+0000","last_fullsized":"2024-03-18T20:26:39.607849+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:18:52.564533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.185216+0000","last_change":"2024-03-18T20:26:25.807504+0000","last_active":"2024-03-18T20:26:39.185216+0000","last_peered":"2024-03-18T20:26:39.185216+0000","last_clean":"2024-03-18T20:26:39.185216+0000","last_became_active":"2024-03-18T20:26:25.807257+0000","last_became_peered":"2024-03-18T20:26:25.807257+0000","last_unstale":"2024-03-18T20:26:39.185216+0000","last_undegraded":"2024-03-18T20:26:39.185216+0000","last_fullsized":"2024-03-18T20:26:39.185216+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:40.456111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.618474+0000","last_change":"2024-03-18T20:26:39.617258+0000","last_active":"2024-03-18T20:26:39.618474+0000","last_peered":"2024-03-18T20:26:39.618474+0000","last_clean":"2024-03-18T20:26:39.618474+0000","last_became_active":"2024-03-18T20:26:39.616305+0000","last_became_peered":"2024-03-18T20:26:39.616305+0000","last_unstale":"2024-03-18T20:26:39.618474+0000","last_undegraded":"2024-03-18T20:26:39.618474+0000","last_fullsized":"2024-03-18T20:26:39.618474+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:22:26.538402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.606616+0000","last_change":"2024-03-18T20:26:39.178912+0000","last_active":"2024-03-18T20:26:39.606616+0000","last_peered":"2024-03-18T20:26:39.606616+0000","last_clean":"2024-03-18T20:26:39.606616+0000","last_became_active":"2024-03-18T20:26:39.178637+0000","last_became_peered":"2024-03-18T20:26:39.178637+0000","last_unstale":"2024-03-18T20:26:39.606616+0000","last_undegraded":"2024-03-18T20:26:39.606616+0000","last_fullsized":"2024-03-18T20:26:39.606616+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:56.206908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.169677+0000","last_change":"2024-03-18T20:26:25.796362+0000","last_active":"2024-03-18T20:26:39.169677+0000","last_peered":"2024-03-18T20:26:39.169677+0000","last_clean":"2024-03-18T20:26:39.169677+0000","last_became_active":"2024-03-18T20:26:25.796061+0000","last_became_peered":"2024-03-18T20:26:25.796061+0000","last_unstale":"2024-03-18T20:26:39.169677+0000","last_undegraded":"2024-03-18T20:26:39.169677+0000","last_fullsized":"2024-03-18T20:26:39.169677+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:28.602198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629507+0000","last_change":"2024-03-18T20:26:39.625934+0000","last_active":"2024-03-18T20:26:39.629507+0000","last_peered":"2024-03-18T20:26:39.629507+0000","last_clean":"2024-03-18T20:26:39.629507+0000","last_became_active":"2024-03-18T20:26:39.625480+0000","last_became_peered":"2024-03-18T20:26:39.625480+0000","last_unstale":"2024-03-18T20:26:39.629507+0000","last_undegraded":"2024-03-18T20:26:39.629507+0000","last_fullsized":"2024-03-18T20:26:39.629507+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:23:05.158863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.610367+0000","last_change":"2024-03-18T20:26:39.180090+0000","last_active":"2024-03-18T20:26:39.610367+0000","last_peered":"2024-03-18T20:26:39.610367+0000","last_clean":"2024-03-18T20:26:39.610367+0000","last_became_active":"2024-03-18T20:26:39.179545+0000","last_became_peered":"2024-03-18T20:26:39.179545+0000","last_unstale":"2024-03-18T20:26:39.610367+0000","last_undegraded":"2024-03-18T20:26:39.610367+0000","last_fullsized":"2024-03-18T20:26:39.610367+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:22:43.454861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.175067+0000","last_change":"2024-03-18T20:26:25.815799+0000","last_active":"2024-03-18T20:26:39.175067+0000","last_peered":"2024-03-18T20:26:39.175067+0000","last_clean":"2024-03-18T20:26:39.175067+0000","last_became_active":"2024-03-18T20:26:25.815392+0000","last_became_peered":"2024-03-18T20:26:25.815392+0000","last_unstale":"2024-03-18T20:26:39.175067+0000","last_undegraded":"2024-03-18T20:26:39.175067+0000","last_fullsized":"2024-03-18T20:26:39.175067+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:21.319444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.626467+0000","last_change":"2024-03-18T20:26:39.623791+0000","last_active":"2024-03-18T20:26:39.626467+0000","last_peered":"2024-03-18T20:26:39.626467+0000","last_clean":"2024-03-18T20:26:39.626467+0000","last_became_active":"2024-03-18T20:26:39.623468+0000","last_became_peered":"2024-03-18T20:26:39.623468+0000","last_unstale":"2024-03-18T20:26:39.626467+0000","last_undegraded":"2024-03-18T20:26:39.626467+0000","last_fullsized":"2024-03-18T20:26:39.626467+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:44:03.811959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.172738+0000","last_change":"2024-03-18T20:26:25.808089+0000","last_active":"2024-03-18T20:26:39.172738+0000","last_peered":"2024-03-18T20:26:39.172738+0000","last_clean":"2024-03-18T20:26:39.172738+0000","last_became_active":"2024-03-18T20:26:25.806688+0000","last_became_peered":"2024-03-18T20:26:25.806688+0000","last_unstale":"2024-03-18T20:26:39.172738+0000","last_undegraded":"2024-03-18T20:26:39.172738+0000","last_fullsized":"2024-03-18T20:26:39.172738+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:22.598187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"252'4","reported_seq":18,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174329+0000","last_change":"2024-03-18T20:26:25.369142+0000","last_active":"2024-03-18T20:26:39.174329+0000","last_peered":"2024-03-18T20:26:39.174329+0000","last_clean":"2024-03-18T20:26:39.174329+0000","last_became_active":"2024-03-18T20:26:25.368879+0000","last_became_peered":"2024-03-18T20:26:25.368879+0000","last_unstale":"2024-03-18T20:26:39.174329+0000","last_undegraded":"2024-03-18T20:26:39.174329+0000","last_fullsized":"2024-03-18T20:26:39.174329+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:42.709929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.624550+0000","last_change":"2024-03-18T20:26:39.617628+0000","last_active":"2024-03-18T20:26:39.624550+0000","last_peered":"2024-03-18T20:26:39.624550+0000","last_clean":"2024-03-18T20:26:39.624550+0000","last_became_active":"2024-03-18T20:26:39.617022+0000","last_became_peered":"2024-03-18T20:26:39.617022+0000","last_unstale":"2024-03-18T20:26:39.624550+0000","last_undegraded":"2024-03-18T20:26:39.624550+0000","last_fullsized":"2024-03-18T20:26:39.624550+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:35.128300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164012+0000","last_change":"2024-03-18T20:26:25.812175+0000","last_active":"2024-03-18T20:26:39.164012+0000","last_peered":"2024-03-18T20:26:39.164012+0000","last_clean":"2024-03-18T20:26:39.164012+0000","last_became_active":"2024-03-18T20:26:25.811898+0000","last_became_peered":"2024-03-18T20:26:25.811898+0000","last_unstale":"2024-03-18T20:26:39.164012+0000","last_undegraded":"2024-03-18T20:26:39.164012+0000","last_fullsized":"2024-03-18T20:26:39.164012+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:45:00.418514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"252'2","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.173318+0000","last_change":"2024-03-18T20:26:25.371848+0000","last_active":"2024-03-18T20:26:39.173318+0000","last_peered":"2024-03-18T20:26:39.173318+0000","last_clean":"2024-03-18T20:26:39.173318+0000","last_became_active":"2024-03-18T20:26:25.371656+0000","last_became_peered":"2024-03-18T20:26:25.371656+0000","last_unstale":"2024-03-18T20:26:39.173318+0000","last_undegraded":"2024-03-18T20:26:39.173318+0000","last_fullsized":"2024-03-18T20:26:39.173318+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:27.592158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.633884+0000","last_change":"2024-03-18T20:26:39.630884+0000","last_active":"2024-03-18T20:26:39.633884+0000","last_peered":"2024-03-18T20:26:39.633884+0000","last_clean":"2024-03-18T20:26:39.633884+0000","last_became_active":"2024-03-18T20:26:39.630590+0000","last_became_peered":"2024-03-18T20:26:39.630590+0000","last_unstale":"2024-03-18T20:26:39.633884+0000","last_undegraded":"2024-03-18T20:26:39.633884+0000","last_fullsized":"2024-03-18T20:26:39.633884+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:30:36.759356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174628+0000","last_change":"2024-03-18T20:26:25.809815+0000","last_active":"2024-03-18T20:26:39.174628+0000","last_peered":"2024-03-18T20:26:39.174628+0000","last_clean":"2024-03-18T20:26:39.174628+0000","last_became_active":"2024-03-18T20:26:25.809504+0000","last_became_peered":"2024-03-18T20:26:25.809504+0000","last_unstale":"2024-03-18T20:26:39.174628+0000","last_undegraded":"2024-03-18T20:26:39.174628+0000","last_fullsized":"2024-03-18T20:26:39.174628+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:42:34.548735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"252'2","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.185729+0000","last_change":"2024-03-18T20:26:25.366016+0000","last_active":"2024-03-18T20:26:39.185729+0000","last_peered":"2024-03-18T20:26:39.185729+0000","last_clean":"2024-03-18T20:26:39.185729+0000","last_became_active":"2024-03-18T20:26:25.365694+0000","last_became_peered":"2024-03-18T20:26:25.365694+0000","last_unstale":"2024-03-18T20:26:39.185729+0000","last_undegraded":"2024-03-18T20:26:39.185729+0000","last_fullsized":"2024-03-18T20:26:39.185729+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:08.800093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629197+0000","last_change":"2024-03-18T20:26:39.621210+0000","last_active":"2024-03-18T20:26:39.629197+0000","last_peered":"2024-03-18T20:26:39.629197+0000","last_clean":"2024-03-18T20:26:39.629197+0000","last_became_active":"2024-03-18T20:26:39.620953+0000","last_became_peered":"2024-03-18T20:26:39.620953+0000","last_unstale":"2024-03-18T20:26:39.629197+0000","last_undegraded":"2024-03-18T20:26:39.629197+0000","last_fullsized":"2024-03-18T20:26:39.629197+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:57.528824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.185439+0000","last_change":"2024-03-18T20:26:25.802194+0000","last_active":"2024-03-18T20:26:39.185439+0000","last_peered":"2024-03-18T20:26:39.185439+0000","last_clean":"2024-03-18T20:26:39.185439+0000","last_became_active":"2024-03-18T20:26:25.801915+0000","last_became_peered":"2024-03-18T20:26:25.801915+0000","last_unstale":"2024-03-18T20:26:39.185439+0000","last_undegraded":"2024-03-18T20:26:39.185439+0000","last_fullsized":"2024-03-18T20:26:39.185439+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:52.955865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"252'1","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164167+0000","last_change":"2024-03-18T20:26:25.364855+0000","last_active":"2024-03-18T20:26:39.164167+0000","last_peered":"2024-03-18T20:26:39.164167+0000","last_clean":"2024-03-18T20:26:39.164167+0000","last_became_active":"2024-03-18T20:26:25.364563+0000","last_became_peered":"2024-03-18T20:26:25.364563+0000","last_unstale":"2024-03-18T20:26:39.164167+0000","last_undegraded":"2024-03-18T20:26:39.164167+0000","last_fullsized":"2024-03-18T20:26:39.164167+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:17.116325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.627216+0000","last_change":"2024-03-18T20:26:39.626171+0000","last_active":"2024-03-18T20:26:39.627216+0000","last_peered":"2024-03-18T20:26:39.627216+0000","last_clean":"2024-03-18T20:26:39.627216+0000","last_became_active":"2024-03-18T20:26:39.625918+0000","last_became_peered":"2024-03-18T20:26:39.625918+0000","last_unstale":"2024-03-18T20:26:39.627216+0000","last_undegraded":"2024-03-18T20:26:39.627216+0000","last_fullsized":"2024-03-18T20:26:39.627216+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:54:42.615870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174290+0000","last_change":"2024-03-18T20:26:25.799795+0000","last_active":"2024-03-18T20:26:39.174290+0000","last_peered":"2024-03-18T20:26:39.174290+0000","last_clean":"2024-03-18T20:26:39.174290+0000","last_became_active":"2024-03-18T20:26:25.799539+0000","last_became_peered":"2024-03-18T20:26:25.799539+0000","last_unstale":"2024-03-18T20:26:39.174290+0000","last_undegraded":"2024-03-18T20:26:39.174290+0000","last_fullsized":"2024-03-18T20:26:39.174290+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:49.613713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"252'1","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.185332+0000","last_change":"2024-03-18T20:26:25.367422+0000","last_active":"2024-03-18T20:26:39.185332+0000","last_peered":"2024-03-18T20:26:39.185332+0000","last_clean":"2024-03-18T20:26:39.185332+0000","last_became_active":"2024-03-18T20:26:25.367153+0000","last_became_peered":"2024-03-18T20:26:25.367153+0000","last_unstale":"2024-03-18T20:26:39.185332+0000","last_undegraded":"2024-03-18T20:26:39.185332+0000","last_fullsized":"2024-03-18T20:26:39.185332+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:14:17.781705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.624858+0000","last_change":"2024-03-18T20:26:39.619567+0000","last_active":"2024-03-18T20:26:39.624858+0000","last_peered":"2024-03-18T20:26:39.624858+0000","last_clean":"2024-03-18T20:26:39.624858+0000","last_became_active":"2024-03-18T20:26:39.618859+0000","last_became_peered":"2024-03-18T20:26:39.618859+0000","last_unstale":"2024-03-18T20:26:39.624858+0000","last_undegraded":"2024-03-18T20:26:39.624858+0000","last_fullsized":"2024-03-18T20:26:39.624858+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:22:26.538402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.165702+0000","last_change":"2024-03-18T20:26:25.812108+0000","last_active":"2024-03-18T20:26:39.165702+0000","last_peered":"2024-03-18T20:26:39.165702+0000","last_clean":"2024-03-18T20:26:39.165702+0000","last_became_active":"2024-03-18T20:26:25.811747+0000","last_became_peered":"2024-03-18T20:26:25.811747+0000","last_unstale":"2024-03-18T20:26:39.165702+0000","last_undegraded":"2024-03-18T20:26:39.165702+0000","last_fullsized":"2024-03-18T20:26:39.165702+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:40.456111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166399+0000","last_change":"2024-03-18T20:26:25.360666+0000","last_active":"2024-03-18T20:26:39.166399+0000","last_peered":"2024-03-18T20:26:39.166399+0000","last_clean":"2024-03-18T20:26:39.166399+0000","last_became_active":"2024-03-18T20:26:25.360400+0000","last_became_peered":"2024-03-18T20:26:25.360400+0000","last_unstale":"2024-03-18T20:26:39.166399+0000","last_undegraded":"2024-03-18T20:26:39.166399+0000","last_fullsized":"2024-03-18T20:26:39.166399+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:18:38.754833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.624983+0000","last_change":"2024-03-18T20:26:39.619868+0000","last_active":"2024-03-18T20:26:39.624983+0000","last_peered":"2024-03-18T20:26:39.624983+0000","last_clean":"2024-03-18T20:26:39.624983+0000","last_became_active":"2024-03-18T20:26:39.619113+0000","last_became_peered":"2024-03-18T20:26:39.619113+0000","last_unstale":"2024-03-18T20:26:39.624983+0000","last_undegraded":"2024-03-18T20:26:39.624983+0000","last_fullsized":"2024-03-18T20:26:39.624983+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:39:05.676337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.167958+0000","last_change":"2024-03-18T20:26:25.810099+0000","last_active":"2024-03-18T20:26:39.167958+0000","last_peered":"2024-03-18T20:26:39.167958+0000","last_clean":"2024-03-18T20:26:39.167958+0000","last_became_active":"2024-03-18T20:26:25.806786+0000","last_became_peered":"2024-03-18T20:26:25.806786+0000","last_unstale":"2024-03-18T20:26:39.167958+0000","last_undegraded":"2024-03-18T20:26:39.167958+0000","last_fullsized":"2024-03-18T20:26:39.167958+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:35.919143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.175261+0000","last_change":"2024-03-18T20:26:25.365778+0000","last_active":"2024-03-18T20:26:39.175261+0000","last_peered":"2024-03-18T20:26:39.175261+0000","last_clean":"2024-03-18T20:26:39.175261+0000","last_became_active":"2024-03-18T20:26:25.365505+0000","last_became_peered":"2024-03-18T20:26:25.365505+0000","last_unstale":"2024-03-18T20:26:39.175261+0000","last_undegraded":"2024-03-18T20:26:39.175261+0000","last_fullsized":"2024-03-18T20:26:39.175261+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:46:21.036246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.624849+0000","last_change":"2024-03-18T20:26:39.619374+0000","last_active":"2024-03-18T20:26:39.624849+0000","last_peered":"2024-03-18T20:26:39.624849+0000","last_clean":"2024-03-18T20:26:39.624849+0000","last_became_active":"2024-03-18T20:26:39.619098+0000","last_became_peered":"2024-03-18T20:26:39.619098+0000","last_unstale":"2024-03-18T20:26:39.624849+0000","last_undegraded":"2024-03-18T20:26:39.624849+0000","last_fullsized":"2024-03-18T20:26:39.624849+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:34:19.023316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.171441+0000","last_change":"2024-03-18T20:26:25.798444+0000","last_active":"2024-03-18T20:26:39.171441+0000","last_peered":"2024-03-18T20:26:39.171441+0000","last_clean":"2024-03-18T20:26:39.171441+0000","last_became_active":"2024-03-18T20:26:25.798186+0000","last_became_peered":"2024-03-18T20:26:25.798186+0000","last_unstale":"2024-03-18T20:26:39.171441+0000","last_undegraded":"2024-03-18T20:26:39.171441+0000","last_fullsized":"2024-03-18T20:26:39.171441+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:06:22.720260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.185459+0000","last_change":"2024-03-18T20:26:25.370707+0000","last_active":"2024-03-18T20:26:39.185459+0000","last_peered":"2024-03-18T20:26:39.185459+0000","last_clean":"2024-03-18T20:26:39.185459+0000","last_became_active":"2024-03-18T20:26:25.370415+0000","last_became_peered":"2024-03-18T20:26:25.370415+0000","last_unstale":"2024-03-18T20:26:39.185459+0000","last_undegraded":"2024-03-18T20:26:39.185459+0000","last_fullsized":"2024-03-18T20:26:39.185459+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:05.351419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.631982+0000","last_change":"2024-03-18T20:26:39.631096+0000","last_active":"2024-03-18T20:26:39.631982+0000","last_peered":"2024-03-18T20:26:39.631982+0000","last_clean":"2024-03-18T20:26:39.631982+0000","last_became_active":"2024-03-18T20:26:39.630814+0000","last_became_peered":"2024-03-18T20:26:39.630814+0000","last_unstale":"2024-03-18T20:26:39.631982+0000","last_undegraded":"2024-03-18T20:26:39.631982+0000","last_fullsized":"2024-03-18T20:26:39.631982+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:41:27.458748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.175384+0000","last_change":"2024-03-18T20:26:25.816190+0000","last_active":"2024-03-18T20:26:39.175384+0000","last_peered":"2024-03-18T20:26:39.175384+0000","last_clean":"2024-03-18T20:26:39.175384+0000","last_became_active":"2024-03-18T20:26:25.815931+0000","last_became_peered":"2024-03-18T20:26:25.815931+0000","last_unstale":"2024-03-18T20:26:39.175384+0000","last_undegraded":"2024-03-18T20:26:39.175384+0000","last_fullsized":"2024-03-18T20:26:39.175384+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:49.613713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.168724+0000","last_change":"2024-03-18T20:26:25.363775+0000","last_active":"2024-03-18T20:26:39.168724+0000","last_peered":"2024-03-18T20:26:39.168724+0000","last_clean":"2024-03-18T20:26:39.168724+0000","last_became_active":"2024-03-18T20:26:25.363389+0000","last_became_peered":"2024-03-18T20:26:25.363389+0000","last_unstale":"2024-03-18T20:26:39.168724+0000","last_undegraded":"2024-03-18T20:26:39.168724+0000","last_fullsized":"2024-03-18T20:26:39.168724+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:42.397208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.618838+0000","last_change":"2024-03-18T20:26:39.616272+0000","last_active":"2024-03-18T20:26:39.618838+0000","last_peered":"2024-03-18T20:26:39.618838+0000","last_clean":"2024-03-18T20:26:39.618838+0000","last_became_active":"2024-03-18T20:26:39.615990+0000","last_became_peered":"2024-03-18T20:26:39.615990+0000","last_unstale":"2024-03-18T20:26:39.618838+0000","last_undegraded":"2024-03-18T20:26:39.618838+0000","last_fullsized":"2024-03-18T20:26:39.618838+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:34.194250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166405+0000","last_change":"2024-03-18T20:26:25.812196+0000","last_active":"2024-03-18T20:26:39.166405+0000","last_peered":"2024-03-18T20:26:39.166405+0000","last_clean":"2024-03-18T20:26:39.166405+0000","last_became_active":"2024-03-18T20:26:25.811934+0000","last_became_peered":"2024-03-18T20:26:25.811934+0000","last_unstale":"2024-03-18T20:26:39.166405+0000","last_undegraded":"2024-03-18T20:26:39.166405+0000","last_fullsized":"2024-03-18T20:26:39.166405+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:22.017892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"252'3","reported_seq":17,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.167596+0000","last_change":"2024-03-18T20:26:25.366952+0000","last_active":"2024-03-18T20:26:39.167596+0000","last_peered":"2024-03-18T20:26:39.167596+0000","last_clean":"2024-03-18T20:26:39.167596+0000","last_became_active":"2024-03-18T20:26:25.366671+0000","last_became_peered":"2024-03-18T20:26:25.366671+0000","last_unstale":"2024-03-18T20:26:39.167596+0000","last_undegraded":"2024-03-18T20:26:39.167596+0000","last_fullsized":"2024-03-18T20:26:39.167596+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:13:21.871477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.630770+0000","last_change":"2024-03-18T20:26:39.621481+0000","last_active":"2024-03-18T20:26:39.630770+0000","last_peered":"2024-03-18T20:26:39.630770+0000","last_clean":"2024-03-18T20:26:39.630770+0000","last_became_active":"2024-03-18T20:26:39.621234+0000","last_became_peered":"2024-03-18T20:26:39.621234+0000","last_unstale":"2024-03-18T20:26:39.630770+0000","last_undegraded":"2024-03-18T20:26:39.630770+0000","last_fullsized":"2024-03-18T20:26:39.630770+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:46:35.855141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.173529+0000","last_change":"2024-03-18T20:26:25.802694+0000","last_active":"2024-03-18T20:26:39.173529+0000","last_peered":"2024-03-18T20:26:39.173529+0000","last_clean":"2024-03-18T20:26:39.173529+0000","last_became_active":"2024-03-18T20:26:25.802443+0000","last_became_peered":"2024-03-18T20:26:25.802443+0000","last_unstale":"2024-03-18T20:26:39.173529+0000","last_undegraded":"2024-03-18T20:26:39.173529+0000","last_fullsized":"2024-03-18T20:26:39.173529+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:54:38.189655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"253'8","reported_seq":22,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174500+0000","last_change":"2024-03-18T20:26:25.368198+0000","last_active":"2024-03-18T20:26:39.174500+0000","last_peered":"2024-03-18T20:26:39.174500+0000","last_clean":"2024-03-18T20:26:39.174500+0000","last_became_active":"2024-03-18T20:26:25.367762+0000","last_became_peered":"2024-03-18T20:26:25.367762+0000","last_unstale":"2024-03-18T20:26:39.174500+0000","last_undegraded":"2024-03-18T20:26:39.174500+0000","last_fullsized":"2024-03-18T20:26:39.174500+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:06.153535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2822,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.625655+0000","last_change":"2024-03-18T20:26:39.621524+0000","last_active":"2024-03-18T20:26:39.625655+0000","last_peered":"2024-03-18T20:26:39.625655+0000","last_clean":"2024-03-18T20:26:39.625655+0000","last_became_active":"2024-03-18T20:26:39.621268+0000","last_became_peered":"2024-03-18T20:26:39.621268+0000","last_unstale":"2024-03-18T20:26:39.625655+0000","last_undegraded":"2024-03-18T20:26:39.625655+0000","last_fullsized":"2024-03-18T20:26:39.625655+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:34.194250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.171112+0000","last_change":"2024-03-18T20:26:25.796336+0000","last_active":"2024-03-18T20:26:39.171112+0000","last_peered":"2024-03-18T20:26:39.171112+0000","last_clean":"2024-03-18T20:26:39.171112+0000","last_became_active":"2024-03-18T20:26:25.796002+0000","last_became_peered":"2024-03-18T20:26:25.796002+0000","last_unstale":"2024-03-18T20:26:39.171112+0000","last_undegraded":"2024-03-18T20:26:39.171112+0000","last_fullsized":"2024-03-18T20:26:39.171112+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:13:22.881517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"252'2","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164880+0000","last_change":"2024-03-18T20:26:25.366019+0000","last_active":"2024-03-18T20:26:39.164880+0000","last_peered":"2024-03-18T20:26:39.164880+0000","last_clean":"2024-03-18T20:26:39.164880+0000","last_became_active":"2024-03-18T20:26:25.364848+0000","last_became_peered":"2024-03-18T20:26:25.364848+0000","last_unstale":"2024-03-18T20:26:39.164880+0000","last_undegraded":"2024-03-18T20:26:39.164880+0000","last_fullsized":"2024-03-18T20:26:39.164880+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:06:15.275895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.623961+0000","last_change":"2024-03-18T20:26:39.623251+0000","last_active":"2024-03-18T20:26:39.623961+0000","last_peered":"2024-03-18T20:26:39.623961+0000","last_clean":"2024-03-18T20:26:39.623961+0000","last_became_active":"2024-03-18T20:26:39.623046+0000","last_became_peered":"2024-03-18T20:26:39.623046+0000","last_unstale":"2024-03-18T20:26:39.623961+0000","last_undegraded":"2024-03-18T20:26:39.623961+0000","last_fullsized":"2024-03-18T20:26:39.623961+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:14:12.949096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174382+0000","last_change":"2024-03-18T20:26:25.803741+0000","last_active":"2024-03-18T20:26:39.174382+0000","last_peered":"2024-03-18T20:26:39.174382+0000","last_clean":"2024-03-18T20:26:39.174382+0000","last_became_active":"2024-03-18T20:26:25.803470+0000","last_became_peered":"2024-03-18T20:26:25.803470+0000","last_unstale":"2024-03-18T20:26:39.174382+0000","last_undegraded":"2024-03-18T20:26:39.174382+0000","last_fullsized":"2024-03-18T20:26:39.174382+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:02:55.614091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166718+0000","last_change":"2024-03-18T20:26:25.366963+0000","last_active":"2024-03-18T20:26:39.166718+0000","last_peered":"2024-03-18T20:26:39.166718+0000","last_clean":"2024-03-18T20:26:39.166718+0000","last_became_active":"2024-03-18T20:26:25.366416+0000","last_became_peered":"2024-03-18T20:26:25.366416+0000","last_unstale":"2024-03-18T20:26:39.166718+0000","last_undegraded":"2024-03-18T20:26:39.166718+0000","last_fullsized":"2024-03-18T20:26:39.166718+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:57.921302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.627098+0000","last_change":"2024-03-18T20:26:39.625979+0000","last_active":"2024-03-18T20:26:39.627098+0000","last_peered":"2024-03-18T20:26:39.627098+0000","last_clean":"2024-03-18T20:26:39.627098+0000","last_became_active":"2024-03-18T20:26:39.623145+0000","last_became_peered":"2024-03-18T20:26:39.623145+0000","last_unstale":"2024-03-18T20:26:39.627098+0000","last_undegraded":"2024-03-18T20:26:39.627098+0000","last_fullsized":"2024-03-18T20:26:39.627098+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:54.035449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164420+0000","last_change":"2024-03-18T20:26:25.808144+0000","last_active":"2024-03-18T20:26:39.164420+0000","last_peered":"2024-03-18T20:26:39.164420+0000","last_clean":"2024-03-18T20:26:39.164420+0000","last_became_active":"2024-03-18T20:26:25.807838+0000","last_became_peered":"2024-03-18T20:26:25.807838+0000","last_unstale":"2024-03-18T20:26:39.164420+0000","last_undegraded":"2024-03-18T20:26:39.164420+0000","last_fullsized":"2024-03-18T20:26:39.164420+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:37:41.132114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166336+0000","last_change":"2024-03-18T20:26:25.366119+0000","last_active":"2024-03-18T20:26:39.166336+0000","last_peered":"2024-03-18T20:26:39.166336+0000","last_clean":"2024-03-18T20:26:39.166336+0000","last_became_active":"2024-03-18T20:26:25.365869+0000","last_became_peered":"2024-03-18T20:26:25.365869+0000","last_unstale":"2024-03-18T20:26:39.166336+0000","last_undegraded":"2024-03-18T20:26:39.166336+0000","last_fullsized":"2024-03-18T20:26:39.166336+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:01.890743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.617667+0000","last_change":"2024-03-18T20:26:39.613269+0000","last_active":"2024-03-18T20:26:39.617667+0000","last_peered":"2024-03-18T20:26:39.617667+0000","last_clean":"2024-03-18T20:26:39.617667+0000","last_became_active":"2024-03-18T20:26:39.612996+0000","last_became_peered":"2024-03-18T20:26:39.612996+0000","last_unstale":"2024-03-18T20:26:39.617667+0000","last_undegraded":"2024-03-18T20:26:39.617667+0000","last_fullsized":"2024-03-18T20:26:39.617667+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:54.264967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.167880+0000","last_change":"2024-03-18T20:26:25.802507+0000","last_active":"2024-03-18T20:26:39.167880+0000","last_peered":"2024-03-18T20:26:39.167880+0000","last_clean":"2024-03-18T20:26:39.167880+0000","last_became_active":"2024-03-18T20:26:25.801913+0000","last_became_peered":"2024-03-18T20:26:25.801913+0000","last_unstale":"2024-03-18T20:26:39.167880+0000","last_undegraded":"2024-03-18T20:26:39.167880+0000","last_fullsized":"2024-03-18T20:26:39.167880+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:11.977302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"252'8","reported_seq":22,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.172641+0000","last_change":"2024-03-18T20:26:25.365245+0000","last_active":"2024-03-18T20:26:39.172641+0000","last_peered":"2024-03-18T20:26:39.172641+0000","last_clean":"2024-03-18T20:26:39.172641+0000","last_became_active":"2024-03-18T20:26:25.364945+0000","last_became_peered":"2024-03-18T20:26:25.364945+0000","last_unstale":"2024-03-18T20:26:39.172641+0000","last_undegraded":"2024-03-18T20:26:39.172641+0000","last_fullsized":"2024-03-18T20:26:39.172641+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:19.739927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629488+0000","last_change":"2024-03-18T20:26:39.625776+0000","last_active":"2024-03-18T20:26:39.629488+0000","last_peered":"2024-03-18T20:26:39.629488+0000","last_clean":"2024-03-18T20:26:39.629488+0000","last_became_active":"2024-03-18T20:26:39.625004+0000","last_became_peered":"2024-03-18T20:26:39.625004+0000","last_unstale":"2024-03-18T20:26:39.629488+0000","last_undegraded":"2024-03-18T20:26:39.629488+0000","last_fullsized":"2024-03-18T20:26:39.629488+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:41:27.458748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.165685+0000","last_change":"2024-03-18T20:26:25.802286+0000","last_active":"2024-03-18T20:26:39.165685+0000","last_peered":"2024-03-18T20:26:39.165685+0000","last_clean":"2024-03-18T20:26:39.165685+0000","last_became_active":"2024-03-18T20:26:25.802005+0000","last_became_peered":"2024-03-18T20:26:25.802005+0000","last_unstale":"2024-03-18T20:26:39.165685+0000","last_undegraded":"2024-03-18T20:26:39.165685+0000","last_fullsized":"2024-03-18T20:26:39.165685+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:10:16.362251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"252'1","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174449+0000","last_change":"2024-03-18T20:26:25.368591+0000","last_active":"2024-03-18T20:26:39.174449+0000","last_peered":"2024-03-18T20:26:39.174449+0000","last_clean":"2024-03-18T20:26:39.174449+0000","last_became_active":"2024-03-18T20:26:25.367843+0000","last_became_peered":"2024-03-18T20:26:25.367843+0000","last_unstale":"2024-03-18T20:26:39.174449+0000","last_undegraded":"2024-03-18T20:26:39.174449+0000","last_fullsized":"2024-03-18T20:26:39.174449+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:59:40.867803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629601+0000","last_change":"2024-03-18T20:26:39.627613+0000","last_active":"2024-03-18T20:26:39.629601+0000","last_peered":"2024-03-18T20:26:39.629601+0000","last_clean":"2024-03-18T20:26:39.629601+0000","last_became_active":"2024-03-18T20:26:39.627030+0000","last_became_peered":"2024-03-18T20:26:39.627030+0000","last_unstale":"2024-03-18T20:26:39.629601+0000","last_undegraded":"2024-03-18T20:26:39.629601+0000","last_fullsized":"2024-03-18T20:26:39.629601+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:50.540181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166229+0000","last_change":"2024-03-18T20:26:25.810340+0000","last_active":"2024-03-18T20:26:39.166229+0000","last_peered":"2024-03-18T20:26:39.166229+0000","last_clean":"2024-03-18T20:26:39.166229+0000","last_became_active":"2024-03-18T20:26:25.809980+0000","last_became_peered":"2024-03-18T20:26:25.809980+0000","last_unstale":"2024-03-18T20:26:39.166229+0000","last_undegraded":"2024-03-18T20:26:39.166229+0000","last_fullsized":"2024-03-18T20:26:39.166229+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:02:55.614091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"252'2","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.173095+0000","last_change":"2024-03-18T20:26:25.366492+0000","last_active":"2024-03-18T20:26:39.173095+0000","last_peered":"2024-03-18T20:26:39.173095+0000","last_clean":"2024-03-18T20:26:39.173095+0000","last_became_active":"2024-03-18T20:26:25.366098+0000","last_became_peered":"2024-03-18T20:26:25.366098+0000","last_unstale":"2024-03-18T20:26:39.173095+0000","last_undegraded":"2024-03-18T20:26:39.173095+0000","last_fullsized":"2024-03-18T20:26:39.173095+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:53:26.604334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.617656+0000","last_change":"2024-03-18T20:26:39.614008+0000","last_active":"2024-03-18T20:26:39.617656+0000","last_peered":"2024-03-18T20:26:39.617656+0000","last_clean":"2024-03-18T20:26:39.617656+0000","last_became_active":"2024-03-18T20:26:39.613741+0000","last_became_peered":"2024-03-18T20:26:39.613741+0000","last_unstale":"2024-03-18T20:26:39.617656+0000","last_undegraded":"2024-03-18T20:26:39.617656+0000","last_fullsized":"2024-03-18T20:26:39.617656+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:06.764721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.169272+0000","last_change":"2024-03-18T20:26:25.807945+0000","last_active":"2024-03-18T20:26:39.169272+0000","last_peered":"2024-03-18T20:26:39.169272+0000","last_clean":"2024-03-18T20:26:39.169272+0000","last_became_active":"2024-03-18T20:26:25.807678+0000","last_became_peered":"2024-03-18T20:26:25.807678+0000","last_unstale":"2024-03-18T20:26:39.169272+0000","last_undegraded":"2024-03-18T20:26:39.169272+0000","last_fullsized":"2024-03-18T20:26:39.169272+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:06.759034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"252'1","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.173411+0000","last_change":"2024-03-18T20:26:25.371835+0000","last_active":"2024-03-18T20:26:39.173411+0000","last_peered":"2024-03-18T20:26:39.173411+0000","last_clean":"2024-03-18T20:26:39.173411+0000","last_became_active":"2024-03-18T20:26:25.371634+0000","last_became_peered":"2024-03-18T20:26:25.371634+0000","last_unstale":"2024-03-18T20:26:39.173411+0000","last_undegraded":"2024-03-18T20:26:39.173411+0000","last_fullsized":"2024-03-18T20:26:39.173411+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:43.471325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.620856+0000","last_change":"2024-03-18T20:26:39.617270+0000","last_active":"2024-03-18T20:26:39.620856+0000","last_peered":"2024-03-18T20:26:39.620856+0000","last_clean":"2024-03-18T20:26:39.620856+0000","last_became_active":"2024-03-18T20:26:39.616997+0000","last_became_peered":"2024-03-18T20:26:39.616997+0000","last_unstale":"2024-03-18T20:26:39.620856+0000","last_undegraded":"2024-03-18T20:26:39.620856+0000","last_fullsized":"2024-03-18T20:26:39.620856+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:20.972430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.167913+0000","last_change":"2024-03-18T20:26:25.812207+0000","last_active":"2024-03-18T20:26:39.167913+0000","last_peered":"2024-03-18T20:26:39.167913+0000","last_clean":"2024-03-18T20:26:39.167913+0000","last_became_active":"2024-03-18T20:26:25.811924+0000","last_became_peered":"2024-03-18T20:26:25.811924+0000","last_unstale":"2024-03-18T20:26:39.167913+0000","last_undegraded":"2024-03-18T20:26:39.167913+0000","last_fullsized":"2024-03-18T20:26:39.167913+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:01:09.966146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164342+0000","last_change":"2024-03-18T20:26:25.364241+0000","last_active":"2024-03-18T20:26:39.164342+0000","last_peered":"2024-03-18T20:26:39.164342+0000","last_clean":"2024-03-18T20:26:39.164342+0000","last_became_active":"2024-03-18T20:26:25.363945+0000","last_became_peered":"2024-03-18T20:26:25.363945+0000","last_unstale":"2024-03-18T20:26:39.164342+0000","last_undegraded":"2024-03-18T20:26:39.164342+0000","last_fullsized":"2024-03-18T20:26:39.164342+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:34.909104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.617802+0000","last_change":"2024-03-18T20:26:39.611228+0000","last_active":"2024-03-18T20:26:39.617802+0000","last_peered":"2024-03-18T20:26:39.617802+0000","last_clean":"2024-03-18T20:26:39.617802+0000","last_became_active":"2024-03-18T20:26:39.607052+0000","last_became_peered":"2024-03-18T20:26:39.607052+0000","last_unstale":"2024-03-18T20:26:39.617802+0000","last_undegraded":"2024-03-18T20:26:39.617802+0000","last_fullsized":"2024-03-18T20:26:39.617802+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:34:19.023316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166075+0000","last_change":"2024-03-18T20:26:25.799178+0000","last_active":"2024-03-18T20:26:39.166075+0000","last_peered":"2024-03-18T20:26:39.166075+0000","last_clean":"2024-03-18T20:26:39.166075+0000","last_became_active":"2024-03-18T20:26:25.798768+0000","last_became_peered":"2024-03-18T20:26:25.798768+0000","last_unstale":"2024-03-18T20:26:39.166075+0000","last_undegraded":"2024-03-18T20:26:39.166075+0000","last_fullsized":"2024-03-18T20:26:39.166075+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:22:12.729546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"252'2","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.168016+0000","last_change":"2024-03-18T20:26:25.370148+0000","last_active":"2024-03-18T20:26:39.168016+0000","last_peered":"2024-03-18T20:26:39.168016+0000","last_clean":"2024-03-18T20:26:39.168016+0000","last_became_active":"2024-03-18T20:26:25.369876+0000","last_became_peered":"2024-03-18T20:26:25.369876+0000","last_unstale":"2024-03-18T20:26:39.168016+0000","last_undegraded":"2024-03-18T20:26:39.168016+0000","last_fullsized":"2024-03-18T20:26:39.168016+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:57:48.151947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.624251+0000","last_change":"2024-03-18T20:26:39.622007+0000","last_active":"2024-03-18T20:26:39.624251+0000","last_peered":"2024-03-18T20:26:39.624251+0000","last_clean":"2024-03-18T20:26:39.624251+0000","last_became_active":"2024-03-18T20:26:39.621726+0000","last_became_peered":"2024-03-18T20:26:39.621726+0000","last_unstale":"2024-03-18T20:26:39.624251+0000","last_undegraded":"2024-03-18T20:26:39.624251+0000","last_fullsized":"2024-03-18T20:26:39.624251+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:52.226234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.177905+0000","last_change":"2024-03-18T20:26:25.806909+0000","last_active":"2024-03-18T20:26:39.177905+0000","last_peered":"2024-03-18T20:26:39.177905+0000","last_clean":"2024-03-18T20:26:39.177905+0000","last_became_active":"2024-03-18T20:26:25.806643+0000","last_became_peered":"2024-03-18T20:26:25.806643+0000","last_unstale":"2024-03-18T20:26:39.177905+0000","last_undegraded":"2024-03-18T20:26:39.177905+0000","last_fullsized":"2024-03-18T20:26:39.177905+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:30:14.320846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.172662+0000","last_change":"2024-03-18T20:26:25.370582+0000","last_active":"2024-03-18T20:26:39.172662+0000","last_peered":"2024-03-18T20:26:39.172662+0000","last_clean":"2024-03-18T20:26:39.172662+0000","last_became_active":"2024-03-18T20:26:25.370336+0000","last_became_peered":"2024-03-18T20:26:25.370336+0000","last_unstale":"2024-03-18T20:26:39.172662+0000","last_undegraded":"2024-03-18T20:26:39.172662+0000","last_fullsized":"2024-03-18T20:26:39.172662+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:23:03.661960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.627082+0000","last_change":"2024-03-18T20:26:39.625364+0000","last_active":"2024-03-18T20:26:39.627082+0000","last_peered":"2024-03-18T20:26:39.627082+0000","last_clean":"2024-03-18T20:26:39.627082+0000","last_became_active":"2024-03-18T20:26:39.624602+0000","last_became_peered":"2024-03-18T20:26:39.624602+0000","last_unstale":"2024-03-18T20:26:39.627082+0000","last_undegraded":"2024-03-18T20:26:39.627082+0000","last_fullsized":"2024-03-18T20:26:39.627082+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:01:23.775002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.171889+0000","last_change":"2024-03-18T20:26:25.814360+0000","last_active":"2024-03-18T20:26:39.171889+0000","last_peered":"2024-03-18T20:26:39.171889+0000","last_clean":"2024-03-18T20:26:39.171889+0000","last_became_active":"2024-03-18T20:26:25.812682+0000","last_became_peered":"2024-03-18T20:26:25.812682+0000","last_unstale":"2024-03-18T20:26:39.171889+0000","last_undegraded":"2024-03-18T20:26:39.171889+0000","last_fullsized":"2024-03-18T20:26:39.171889+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:22.598187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.175559+0000","last_change":"2024-03-18T20:26:25.369740+0000","last_active":"2024-03-18T20:26:39.175559+0000","last_peered":"2024-03-18T20:26:39.175559+0000","last_clean":"2024-03-18T20:26:39.175559+0000","last_became_active":"2024-03-18T20:26:25.369448+0000","last_became_peered":"2024-03-18T20:26:25.369448+0000","last_unstale":"2024-03-18T20:26:39.175559+0000","last_undegraded":"2024-03-18T20:26:39.175559+0000","last_fullsized":"2024-03-18T20:26:39.175559+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:39:25.003994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.626965+0000","last_change":"2024-03-18T20:26:39.621894+0000","last_active":"2024-03-18T20:26:39.626965+0000","last_peered":"2024-03-18T20:26:39.626965+0000","last_clean":"2024-03-18T20:26:39.626965+0000","last_became_active":"2024-03-18T20:26:39.621591+0000","last_became_peered":"2024-03-18T20:26:39.621591+0000","last_unstale":"2024-03-18T20:26:39.626965+0000","last_undegraded":"2024-03-18T20:26:39.626965+0000","last_fullsized":"2024-03-18T20:26:39.626965+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:35.406128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.175312+0000","last_change":"2024-03-18T20:26:25.807777+0000","last_active":"2024-03-18T20:26:39.175312+0000","last_peered":"2024-03-18T20:26:39.175312+0000","last_clean":"2024-03-18T20:26:39.175312+0000","last_became_active":"2024-03-18T20:26:25.807447+0000","last_became_peered":"2024-03-18T20:26:25.807447+0000","last_unstale":"2024-03-18T20:26:39.175312+0000","last_undegraded":"2024-03-18T20:26:39.175312+0000","last_fullsized":"2024-03-18T20:26:39.175312+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:22:55.933766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.184321+0000","last_change":"2024-03-18T20:26:25.366647+0000","last_active":"2024-03-18T20:26:39.184321+0000","last_peered":"2024-03-18T20:26:39.184321+0000","last_clean":"2024-03-18T20:26:39.184321+0000","last_became_active":"2024-03-18T20:26:25.366363+0000","last_became_peered":"2024-03-18T20:26:25.366363+0000","last_unstale":"2024-03-18T20:26:39.184321+0000","last_undegraded":"2024-03-18T20:26:39.184321+0000","last_fullsized":"2024-03-18T20:26:39.184321+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:22:29.645161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.623661+0000","last_change":"2024-03-18T20:26:39.620184+0000","last_active":"2024-03-18T20:26:39.623661+0000","last_peered":"2024-03-18T20:26:39.623661+0000","last_clean":"2024-03-18T20:26:39.623661+0000","last_became_active":"2024-03-18T20:26:39.615832+0000","last_became_peered":"2024-03-18T20:26:39.615832+0000","last_unstale":"2024-03-18T20:26:39.623661+0000","last_undegraded":"2024-03-18T20:26:39.623661+0000","last_fullsized":"2024-03-18T20:26:39.623661+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:04:46.303946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.171363+0000","last_change":"2024-03-18T20:26:25.807101+0000","last_active":"2024-03-18T20:26:39.171363+0000","last_peered":"2024-03-18T20:26:39.171363+0000","last_clean":"2024-03-18T20:26:39.171363+0000","last_became_active":"2024-03-18T20:26:25.806372+0000","last_became_peered":"2024-03-18T20:26:25.806372+0000","last_unstale":"2024-03-18T20:26:39.171363+0000","last_undegraded":"2024-03-18T20:26:39.171363+0000","last_fullsized":"2024-03-18T20:26:39.171363+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:18.126364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"252'2","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.168718+0000","last_change":"2024-03-18T20:26:25.365000+0000","last_active":"2024-03-18T20:26:39.168718+0000","last_peered":"2024-03-18T20:26:39.168718+0000","last_clean":"2024-03-18T20:26:39.168718+0000","last_became_active":"2024-03-18T20:26:25.364727+0000","last_became_peered":"2024-03-18T20:26:25.364727+0000","last_unstale":"2024-03-18T20:26:39.168718+0000","last_undegraded":"2024-03-18T20:26:39.168718+0000","last_fullsized":"2024-03-18T20:26:39.168718+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:18.741871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.631964+0000","last_change":"2024-03-18T20:26:39.629096+0000","last_active":"2024-03-18T20:26:39.631964+0000","last_peered":"2024-03-18T20:26:39.631964+0000","last_clean":"2024-03-18T20:26:39.631964+0000","last_became_active":"2024-03-18T20:26:39.628843+0000","last_became_peered":"2024-03-18T20:26:39.628843+0000","last_unstale":"2024-03-18T20:26:39.631964+0000","last_undegraded":"2024-03-18T20:26:39.631964+0000","last_fullsized":"2024-03-18T20:26:39.631964+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:23:05.158863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164206+0000","last_change":"2024-03-18T20:26:25.803062+0000","last_active":"2024-03-18T20:26:39.164206+0000","last_peered":"2024-03-18T20:26:39.164206+0000","last_clean":"2024-03-18T20:26:39.164206+0000","last_became_active":"2024-03-18T20:26:25.802825+0000","last_became_peered":"2024-03-18T20:26:25.802825+0000","last_unstale":"2024-03-18T20:26:39.164206+0000","last_undegraded":"2024-03-18T20:26:39.164206+0000","last_fullsized":"2024-03-18T20:26:39.164206+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:31:54.405889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"253'1","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166577+0000","last_change":"2024-03-18T20:26:25.367649+0000","last_active":"2024-03-18T20:26:39.166577+0000","last_peered":"2024-03-18T20:26:39.166577+0000","last_clean":"2024-03-18T20:26:39.166577+0000","last_became_active":"2024-03-18T20:26:25.367351+0000","last_became_peered":"2024-03-18T20:26:25.367351+0000","last_unstale":"2024-03-18T20:26:39.166577+0000","last_undegraded":"2024-03-18T20:26:39.166577+0000","last_fullsized":"2024-03-18T20:26:39.166577+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:19:05.047442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629826+0000","last_change":"2024-03-18T20:26:39.627878+0000","last_active":"2024-03-18T20:26:39.629826+0000","last_peered":"2024-03-18T20:26:39.629826+0000","last_clean":"2024-03-18T20:26:39.629826+0000","last_became_active":"2024-03-18T20:26:39.627630+0000","last_became_peered":"2024-03-18T20:26:39.627630+0000","last_unstale":"2024-03-18T20:26:39.629826+0000","last_undegraded":"2024-03-18T20:26:39.629826+0000","last_fullsized":"2024-03-18T20:26:39.629826+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:01:32.688385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166320+0000","last_change":"2024-03-18T20:26:25.812701+0000","last_active":"2024-03-18T20:26:39.166320+0000","last_peered":"2024-03-18T20:26:39.166320+0000","last_clean":"2024-03-18T20:26:39.166320+0000","last_became_active":"2024-03-18T20:26:25.812469+0000","last_became_peered":"2024-03-18T20:26:25.812469+0000","last_unstale":"2024-03-18T20:26:39.166320+0000","last_undegraded":"2024-03-18T20:26:39.166320+0000","last_fullsized":"2024-03-18T20:26:39.166320+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:59.140240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.185442+0000","last_change":"2024-03-18T20:26:25.370628+0000","last_active":"2024-03-18T20:26:39.185442+0000","last_peered":"2024-03-18T20:26:39.185442+0000","last_clean":"2024-03-18T20:26:39.185442+0000","last_became_active":"2024-03-18T20:26:25.370332+0000","last_became_peered":"2024-03-18T20:26:25.370332+0000","last_unstale":"2024-03-18T20:26:39.185442+0000","last_undegraded":"2024-03-18T20:26:39.185442+0000","last_fullsized":"2024-03-18T20:26:39.185442+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:43.011701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.619682+0000","last_change":"2024-03-18T20:26:39.618919+0000","last_active":"2024-03-18T20:26:39.619682+0000","last_peered":"2024-03-18T20:26:39.619682+0000","last_clean":"2024-03-18T20:26:39.619682+0000","last_became_active":"2024-03-18T20:26:39.618656+0000","last_became_peered":"2024-03-18T20:26:39.618656+0000","last_unstale":"2024-03-18T20:26:39.619682+0000","last_undegraded":"2024-03-18T20:26:39.619682+0000","last_fullsized":"2024-03-18T20:26:39.619682+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:22:26.538402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166675+0000","last_change":"2024-03-18T20:26:25.802895+0000","last_active":"2024-03-18T20:26:39.166675+0000","last_peered":"2024-03-18T20:26:39.166675+0000","last_clean":"2024-03-18T20:26:39.166675+0000","last_became_active":"2024-03-18T20:26:25.802655+0000","last_became_peered":"2024-03-18T20:26:25.802655+0000","last_unstale":"2024-03-18T20:26:39.166675+0000","last_undegraded":"2024-03-18T20:26:39.166675+0000","last_fullsized":"2024-03-18T20:26:39.166675+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:18.126364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.185827+0000","last_change":"2024-03-18T20:26:25.368715+0000","last_active":"2024-03-18T20:26:39.185827+0000","last_peered":"2024-03-18T20:26:39.185827+0000","last_clean":"2024-03-18T20:26:39.185827+0000","last_became_active":"2024-03-18T20:26:25.368383+0000","last_became_peered":"2024-03-18T20:26:25.368383+0000","last_unstale":"2024-03-18T20:26:39.185827+0000","last_undegraded":"2024-03-18T20:26:39.185827+0000","last_fullsized":"2024-03-18T20:26:39.185827+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:07.975001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629394+0000","last_change":"2024-03-18T20:26:39.628048+0000","last_active":"2024-03-18T20:26:39.629394+0000","last_peered":"2024-03-18T20:26:39.629394+0000","last_clean":"2024-03-18T20:26:39.629394+0000","last_became_active":"2024-03-18T20:26:39.627774+0000","last_became_peered":"2024-03-18T20:26:39.627774+0000","last_unstale":"2024-03-18T20:26:39.629394+0000","last_undegraded":"2024-03-18T20:26:39.629394+0000","last_fullsized":"2024-03-18T20:26:39.629394+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:48:28.887612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.185622+0000","last_change":"2024-03-18T20:26:25.805198+0000","last_active":"2024-03-18T20:26:39.185622+0000","last_peered":"2024-03-18T20:26:39.185622+0000","last_clean":"2024-03-18T20:26:39.185622+0000","last_became_active":"2024-03-18T20:26:25.804942+0000","last_became_peered":"2024-03-18T20:26:25.804942+0000","last_unstale":"2024-03-18T20:26:39.185622+0000","last_undegraded":"2024-03-18T20:26:39.185622+0000","last_fullsized":"2024-03-18T20:26:39.185622+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:22:12.729546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174310+0000","last_change":"2024-03-18T20:26:25.370584+0000","last_active":"2024-03-18T20:26:39.174310+0000","last_peered":"2024-03-18T20:26:39.174310+0000","last_clean":"2024-03-18T20:26:39.174310+0000","last_became_active":"2024-03-18T20:26:25.370353+0000","last_became_peered":"2024-03-18T20:26:25.370353+0000","last_unstale":"2024-03-18T20:26:39.174310+0000","last_undegraded":"2024-03-18T20:26:39.174310+0000","last_fullsized":"2024-03-18T20:26:39.174310+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:33:14.925680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3f","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.626454+0000","last_change":"2024-03-18T20:26:39.621607+0000","last_active":"2024-03-18T20:26:39.626454+0000","last_peered":"2024-03-18T20:26:39.626454+0000","last_clean":"2024-03-18T20:26:39.626454+0000","last_became_active":"2024-03-18T20:26:39.620969+0000","last_became_peered":"2024-03-18T20:26:39.620969+0000","last_unstale":"2024-03-18T20:26:39.626454+0000","last_undegraded":"2024-03-18T20:26:39.626454+0000","last_fullsized":"2024-03-18T20:26:39.626454+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:54:42.615870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174008+0000","last_change":"2024-03-18T20:26:25.801336+0000","last_active":"2024-03-18T20:26:39.174008+0000","last_peered":"2024-03-18T20:26:39.174008+0000","last_clean":"2024-03-18T20:26:39.174008+0000","last_became_active":"2024-03-18T20:26:25.801053+0000","last_became_peered":"2024-03-18T20:26:25.801053+0000","last_unstale":"2024-03-18T20:26:39.174008+0000","last_undegraded":"2024-03-18T20:26:39.174008+0000","last_fullsized":"2024-03-18T20:26:39.174008+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:23.742824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166803+0000","last_change":"2024-03-18T20:26:25.367183+0000","last_active":"2024-03-18T20:26:39.166803+0000","last_peered":"2024-03-18T20:26:39.166803+0000","last_clean":"2024-03-18T20:26:39.166803+0000","last_became_active":"2024-03-18T20:26:25.366701+0000","last_became_peered":"2024-03-18T20:26:25.366701+0000","last_unstale":"2024-03-18T20:26:39.166803+0000","last_undegraded":"2024-03-18T20:26:39.166803+0000","last_fullsized":"2024-03-18T20:26:39.166803+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:05.748994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.630082+0000","last_change":"2024-03-18T20:26:39.628068+0000","last_active":"2024-03-18T20:26:39.630082+0000","last_peered":"2024-03-18T20:26:39.630082+0000","last_clean":"2024-03-18T20:26:39.630082+0000","last_became_active":"2024-03-18T20:26:39.627774+0000","last_became_peered":"2024-03-18T20:26:39.627774+0000","last_unstale":"2024-03-18T20:26:39.630082+0000","last_undegraded":"2024-03-18T20:26:39.630082+0000","last_fullsized":"2024-03-18T20:26:39.630082+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:10:22.388409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.173328+0000","last_change":"2024-03-18T20:26:25.806432+0000","last_active":"2024-03-18T20:26:39.173328+0000","last_peered":"2024-03-18T20:26:39.173328+0000","last_clean":"2024-03-18T20:26:39.173328+0000","last_became_active":"2024-03-18T20:26:25.806050+0000","last_became_peered":"2024-03-18T20:26:25.806050+0000","last_unstale":"2024-03-18T20:26:39.173328+0000","last_undegraded":"2024-03-18T20:26:39.173328+0000","last_fullsized":"2024-03-18T20:26:39.173328+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:32.033370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166209+0000","last_change":"2024-03-18T20:26:25.364069+0000","last_active":"2024-03-18T20:26:39.166209+0000","last_peered":"2024-03-18T20:26:39.166209+0000","last_clean":"2024-03-18T20:26:39.166209+0000","last_became_active":"2024-03-18T20:26:25.363764+0000","last_became_peered":"2024-03-18T20:26:25.363764+0000","last_unstale":"2024-03-18T20:26:39.166209+0000","last_undegraded":"2024-03-18T20:26:39.166209+0000","last_fullsized":"2024-03-18T20:26:39.166209+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:01.862380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.630151+0000","last_change":"2024-03-18T20:26:39.629816+0000","last_active":"2024-03-18T20:26:39.630151+0000","last_peered":"2024-03-18T20:26:39.630151+0000","last_clean":"2024-03-18T20:26:39.630151+0000","last_became_active":"2024-03-18T20:26:39.629663+0000","last_became_peered":"2024-03-18T20:26:39.629663+0000","last_unstale":"2024-03-18T20:26:39.630151+0000","last_undegraded":"2024-03-18T20:26:39.630151+0000","last_fullsized":"2024-03-18T20:26:39.630151+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:54.264967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.175402+0000","last_change":"2024-03-18T20:26:25.817787+0000","last_active":"2024-03-18T20:26:39.175402+0000","last_peered":"2024-03-18T20:26:39.175402+0000","last_clean":"2024-03-18T20:26:39.175402+0000","last_became_active":"2024-03-18T20:26:25.817533+0000","last_became_peered":"2024-03-18T20:26:25.817533+0000","last_unstale":"2024-03-18T20:26:39.175402+0000","last_undegraded":"2024-03-18T20:26:39.175402+0000","last_fullsized":"2024-03-18T20:26:39.175402+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:22:04.370714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.171538+0000","last_change":"2024-03-18T20:26:25.368090+0000","last_active":"2024-03-18T20:26:39.171538+0000","last_peered":"2024-03-18T20:26:39.171538+0000","last_clean":"2024-03-18T20:26:39.171538+0000","last_became_active":"2024-03-18T20:26:25.367837+0000","last_became_peered":"2024-03-18T20:26:25.367837+0000","last_unstale":"2024-03-18T20:26:39.171538+0000","last_undegraded":"2024-03-18T20:26:39.171538+0000","last_fullsized":"2024-03-18T20:26:39.171538+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:19.739927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629583+0000","last_change":"2024-03-18T20:26:39.627267+0000","last_active":"2024-03-18T20:26:39.629583+0000","last_peered":"2024-03-18T20:26:39.629583+0000","last_clean":"2024-03-18T20:26:39.629583+0000","last_became_active":"2024-03-18T20:26:39.626469+0000","last_became_peered":"2024-03-18T20:26:39.626469+0000","last_unstale":"2024-03-18T20:26:39.629583+0000","last_undegraded":"2024-03-18T20:26:39.629583+0000","last_fullsized":"2024-03-18T20:26:39.629583+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:00:52.687797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.169093+0000","last_change":"2024-03-18T20:26:25.811045+0000","last_active":"2024-03-18T20:26:39.169093+0000","last_peered":"2024-03-18T20:26:39.169093+0000","last_clean":"2024-03-18T20:26:39.169093+0000","last_became_active":"2024-03-18T20:26:25.810754+0000","last_became_peered":"2024-03-18T20:26:25.810754+0000","last_unstale":"2024-03-18T20:26:39.169093+0000","last_undegraded":"2024-03-18T20:26:39.169093+0000","last_fullsized":"2024-03-18T20:26:39.169093+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:44.931187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"252'1","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.175487+0000","last_change":"2024-03-18T20:26:25.367689+0000","last_active":"2024-03-18T20:26:39.175487+0000","last_peered":"2024-03-18T20:26:39.175487+0000","last_clean":"2024-03-18T20:26:39.175487+0000","last_became_active":"2024-03-18T20:26:25.367431+0000","last_became_peered":"2024-03-18T20:26:25.367431+0000","last_unstale":"2024-03-18T20:26:39.175487+0000","last_undegraded":"2024-03-18T20:26:39.175487+0000","last_fullsized":"2024-03-18T20:26:39.175487+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:01.890743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"238'370","reported_seq":1300,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164037+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:26:39.164037+0000","last_peered":"2024-03-18T20:26:39.164037+0000","last_clean":"2024-03-18T20:26:39.164037+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:26:39.164037+0000","last_undegraded":"2024-03-18T20:26:39.164037+0000","last_fullsized":"2024-03-18T20:26:39.164037+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":3620,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575994,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52888,"kb_used_data":3724,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318952,"statfs":{"total":96636764160,"available":96582606848,"internally_reserved":0,"allocated":3813376,"data_stored":2248428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.57299999999999995,"15min":0.59499999999999997},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":1.1339999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.58299999999999996,"15min":0.60899999999999999},"min":{"1min":0.35299999999999998,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.76800000000000002,"5min":0.84499999999999997,"15min":1.093},"last":0.53000000000000003}]},{"osd":1,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.58199999999999996,"15min":0.60799999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72199999999999998,"5min":0.83099999999999996,"15min":1.4299999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.58499999999999996,"15min":0.61399999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":1.2629999999999999},"last":0.496}]},{"osd":2,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.58799999999999997,"15min":0.60899999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":1.216},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.58999999999999997,"15min":0.621},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.2490000000000001},"last":0.48099999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.57199999999999995,"15min":0.60299999999999998},"min":{"1min":0.374,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83899999999999997,"5min":0.84699999999999998,"15min":1.4450000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.623,"15min":0.64600000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.38600000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":1.5029999999999999},"last":0.55100000000000005}]},{"osd":4,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46600000000000003,"15min":0.47699999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.76500000000000001,"5min":0.79100000000000004,"15min":0.86099999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.48899999999999999,"15min":0.5},"min":{"1min":0.34300000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.50700000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.502,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52500000000000002,"15min":0.54600000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.877,"5min":0.877,"15min":1.042},"last":0.58499999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55300000000000005,"15min":0.57699999999999996},"min":{"1min":0.39500000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":1.129},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54800000000000004,"15min":0.56799999999999995},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":1.0609999999999999},"last":0.54300000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575994,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60504,"kb_used_data":3144,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311336,"statfs":{"total":96636764160,"available":96574808064,"internally_reserved":0,"allocated":3219456,"data_stored":1658048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55400000000000005,"15min":0.59399999999999997},"min":{"1min":0.32500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.623,"5min":0.80900000000000005,"15min":1.1519999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57399999999999995,"15min":0.60399999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.52600000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56499999999999995,"15min":0.59699999999999998},"min":{"1min":0.40400000000000003,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69899999999999995,"5min":0.79800000000000004,"15min":1.244},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57999999999999996,"15min":0.60299999999999998},"min":{"1min":0.42499999999999999,"5min":0.35399999999999998,"15min":0.33300000000000002},"max":{"1min":0.83699999999999997,"5min":0.89200000000000002,"15min":1.286},"last":0.54000000000000004}]},{"osd":2,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.58699999999999997,"15min":0.623},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.69199999999999995,"5min":0.92000000000000004,"15min":1.276},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57999999999999996,"15min":0.60899999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.378},"max":{"1min":1.089,"5min":1.089,"15min":1.2669999999999999},"last":0.439}]},{"osd":3,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58099999999999996,"15min":0.621},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.59199999999999997,"15min":0.61599999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70599999999999996,"5min":1.0049999999999999,"15min":1.337},"last":0.498}]},{"osd":4,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.48899999999999999,"15min":0.48799999999999999},"min":{"1min":0.35799999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.75600000000000001,"5min":0.85799999999999998,"15min":0.88300000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.50800000000000001,"15min":0.51700000000000002},"min":{"1min":0.35899999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80300000000000005,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.48699999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.51100000000000001,"15min":0.51000000000000001},"min":{"1min":0.38300000000000001,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39600000000000002,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.874,"5min":0.92600000000000005,"15min":0.94199999999999995},"last":0.625}]},{"osd":7,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.42399999999999999,"5min":0.38,"15min":0.34499999999999997},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.57799999999999996,"15min":0.57499999999999996},"min":{"1min":0.41399999999999998,"5min":0.36599999999999999,"15min":0.34599999999999997},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.58899999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575994,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52288,"kb_used_data":3124,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319552,"statfs":{"total":96636764160,"available":96583221248,"internally_reserved":0,"allocated":3198976,"data_stored":1650380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.56299999999999994,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.38,"5min":0.41199999999999998,"15min":0.42199999999999999},"min":{"1min":0.27200000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56499999999999995,"5min":0.67500000000000004,"15min":0.69699999999999995},"last":0.34899999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.43099999999999999,"15min":0.432},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.51500000000000001,"5min":0.64300000000000002,"15min":0.68100000000000005},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.34499999999999997,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.55500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.41399999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.33300000000000002,"5min":0.33000000000000002,"15min":0.32300000000000001},"max":{"1min":0.58699999999999997,"5min":0.75900000000000001,"15min":0.76100000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.44,"15min":0.436},"min":{"1min":0.29199999999999998,"5min":0.28899999999999998,"15min":0.27000000000000002},"max":{"1min":0.55400000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.39000000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.66500000000000004,"15min":0.66800000000000004},"min":{"1min":0.42999999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82199999999999995,"5min":1.0680000000000001,"15min":1.2909999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66900000000000004,"15min":0.67500000000000004},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.746},"last":0.57999999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.46300000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.88600000000000001,"5min":1.0389999999999999,"15min":1.6910000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.70199999999999996,"15min":0.70799999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.91100000000000003,"5min":1.109,"15min":1.2549999999999999},"last":0.69499999999999995}]},{"osd":6,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.68200000000000005,"15min":0.68100000000000005},"min":{"1min":0.46800000000000003,"5min":0.45900000000000002,"15min":0.437},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.637},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.70299999999999996,"15min":0.70499999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92100000000000004,"5min":1.014,"15min":1.722},"last":0.65300000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.68999999999999995,"15min":0.70399999999999996},"min":{"1min":0.51400000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.90300000000000002,"5min":1.02,"15min":1.2729999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.56599999999999995,"5min":0.495,"15min":0.42799999999999999},"max":{"1min":0.84199999999999997,"5min":1.0329999999999999,"15min":1.3380000000000001},"last":0.63900000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575995,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48820,"kb_used_data":3744,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323020,"statfs":{"total":96636764160,"available":96586772480,"internally_reserved":0,"allocated":3833856,"data_stored":2259261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38300000000000001,"15min":0.39000000000000001},"min":{"1min":0.29299999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56399999999999995,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.41699999999999998,"15min":0.42599999999999999},"min":{"1min":0.28199999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.60099999999999998,"5min":0.64000000000000001,"15min":1.1850000000000001},"last":0.25}]},{"osd":2,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42899999999999999,"15min":0.437},"min":{"1min":0.28399999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67900000000000005,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.432,"15min":0.45000000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.70999999999999996},"last":0.33600000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.42099999999999999,"15min":0.42899999999999999},"min":{"1min":0.29499999999999998,"5min":0.27500000000000002,"15min":0.26100000000000001},"max":{"1min":0.58499999999999996,"5min":0.60099999999999998,"15min":0.628},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.48099999999999998,"15min":0.49299999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.32500000000000001},"max":{"1min":0.73399999999999999,"5min":0.73999999999999999,"15min":0.91500000000000004},"last":0.30299999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63200000000000001,"15min":0.65200000000000002},"min":{"1min":0.40000000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90200000000000002,"5min":0.91500000000000004,"15min":1.0029999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.69899999999999995}]},{"osd":5,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.66700000000000004,"15min":0.67000000000000004},"min":{"1min":0.435,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.88500000000000001,"5min":0.97099999999999997,"15min":1.0129999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.626,"5min":0.67400000000000004,"15min":0.68000000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":1.518},"last":0.68799999999999994}]},{"osd":6,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.66100000000000003,"15min":0.66300000000000003},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.1279999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.79800000000000004}]},{"osd":7,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.69199999999999995,"15min":0.70499999999999996},"min":{"1min":0.47999999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.464},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69099999999999995,"15min":0.70299999999999996},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.47599999999999998},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.60299999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575995,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52296,"kb_used_data":3132,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319544,"statfs":{"total":96636764160,"available":96583213056,"internally_reserved":0,"allocated":3207168,"data_stored":1651032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.34699999999999998,"15min":0.371},"min":{"1min":0.20799999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.53800000000000003,"5min":0.54800000000000004,"15min":0.57499999999999996},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.44700000000000001,"15min":0.47799999999999998},"min":{"1min":0.26100000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60499999999999998,"5min":0.71499999999999997,"15min":0.753},"last":0.35599999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.44400000000000001,"15min":0.47399999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57399999999999995,"5min":0.64300000000000002,"15min":0.749},"last":0.377},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.38900000000000001,"15min":0.40899999999999997},"min":{"1min":0.246,"5min":0.224,"15min":0.224},"max":{"1min":0.501,"5min":0.55900000000000005,"15min":0.59999999999999998},"last":0.311}]},{"osd":3,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.41199999999999998,"15min":0.42899999999999999},"min":{"1min":0.27800000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.56100000000000005,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.47399999999999998,"15min":0.50700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.63200000000000001,"5min":0.68100000000000005,"15min":0.752},"last":0.32900000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63900000000000001,"15min":0.65300000000000002},"min":{"1min":0.41799999999999998,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":1.2090000000000001,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.65700000000000003,"15min":0.66200000000000003},"min":{"1min":0.45100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.268,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.64800000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.64900000000000002,"15min":0.67500000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79900000000000004,"5min":0.97299999999999998,"15min":1.028},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.63900000000000001,"15min":0.67000000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.77100000000000002,"5min":0.88300000000000001,"15min":1.0629999999999999},"last":0.47899999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.65500000000000003,"15min":0.68100000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.81899999999999995,"5min":1.014,"15min":1.2050000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.72699999999999998},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.3220000000000001,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.73899999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.65500000000000003,"15min":0.68899999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.42199999999999999},"max":{"1min":0.73699999999999999,"5min":0.99399999999999999,"15min":1.0269999999999999},"last":0.504},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68600000000000005,"15min":0.70099999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.2330000000000001,"5min":1.371,"15min":1.371},"last":0.69199999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575995,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52344,"kb_used_data":3176,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319496,"statfs":{"total":96636764160,"available":96583163904,"internally_reserved":0,"allocated":3252224,"data_stored":1669551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.38,"15min":0.39500000000000002},"min":{"1min":0.23100000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.48899999999999999,"5min":0.66500000000000004,"15min":0.98099999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.39100000000000001,"15min":0.40999999999999998},"min":{"1min":0.26100000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.42699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.39200000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.374,"15min":0.40300000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.47699999999999998,"5min":0.63200000000000001,"15min":0.64000000000000001},"last":0.375},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.44400000000000001,"15min":0.46200000000000002},"min":{"1min":0.315,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.50900000000000001,"5min":0.65500000000000003,"15min":0.73099999999999998},"last":0.36299999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.436,"15min":0.46000000000000002},"min":{"1min":0.29999999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53800000000000003,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.46000000000000002,"15min":0.48799999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.51700000000000002,"5min":0.66100000000000003,"15min":0.73099999999999998},"last":0.42399999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.61199999999999999,"15min":0.64500000000000002},"min":{"1min":0.41599999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":1.526},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.63300000000000001,"15min":0.66900000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.63}]},{"osd":5,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.67000000000000004,"15min":0.69299999999999995},"min":{"1min":0.42999999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":1.5149999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.70299999999999996,"15min":0.72299999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.1120000000000001,"5min":1.171,"15min":1.171},"last":0.63900000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67000000000000004,"15min":0.70299999999999996},"min":{"1min":0.45400000000000001,"5min":0.436,"15min":0.42499999999999999},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":1.573},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.69999999999999996,"15min":0.72599999999999998},"min":{"1min":0.47399999999999998,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":1.22,"5min":1.22,"15min":1.2410000000000001},"last":1.5600000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.68100000000000005,"15min":0.71299999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.624,"5min":0.66000000000000003,"15min":0.69399999999999995},"min":{"1min":0.41099999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":1.5509999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575995,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48212,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323628,"statfs":{"total":96636764160,"available":96587395072,"internally_reserved":0,"allocated":3215360,"data_stored":1658006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57799999999999996,"15min":0.58899999999999997},"min":{"1min":0.39600000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56799999999999995,"15min":0.59499999999999997},"min":{"1min":0.40100000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.84199999999999997},"last":0.40699999999999997}]},{"osd":1,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.79300000000000004,"5min":0.81899999999999995,"15min":0.88300000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56399999999999995,"15min":0.57999999999999996},"min":{"1min":0.34899999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.96499999999999997},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59799999999999998,"15min":0.61399999999999999},"min":{"1min":0.42899999999999999,"5min":0.41299999999999998,"15min":0.38500000000000001},"max":{"1min":0.70799999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.45200000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.59299999999999997,"15min":0.60199999999999998},"min":{"1min":0.433,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78100000000000003,"5min":1.026,"15min":1.026},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.41399999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.58299999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.48399999999999999,"15min":0.48199999999999998},"min":{"1min":0.318,"5min":0.309,"15min":0.309},"max":{"1min":0.76700000000000002,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.496,"15min":0.505},"min":{"1min":0.38300000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.81499999999999995,"5min":0.85499999999999998,"15min":0.90600000000000003},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.502,"15min":0.50600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.754,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.82799999999999996,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.59499999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.40600000000000003,"5min":0.38800000000000001,"15min":0.35699999999999998},"max":{"1min":0.80300000000000005,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.38500000000000001,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.84099999999999997,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.61499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575994,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48192,"kb_used_data":3124,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323648,"statfs":{"total":96636764160,"available":96587415552,"internally_reserved":0,"allocated":3198976,"data_stored":1650380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.57699999999999996,"15min":0.59799999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.31900000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56999999999999995,"15min":0.60099999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82599999999999996,"15min":1.3100000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58199999999999996,"15min":0.60199999999999998},"min":{"1min":0.36599999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.437},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57499999999999996,"15min":0.59799999999999998},"min":{"1min":0.40500000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":1.018},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.57899999999999996,"15min":0.59899999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.99299999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58199999999999996,"15min":0.60899999999999999},"min":{"1min":0.41599999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.52800000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59299999999999997,"15min":0.60999999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.2430000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.61399999999999999,"15min":0.64200000000000002},"min":{"1min":0.40600000000000003,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":1.161},"last":0.59999999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.49299999999999999,"15min":0.47999999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.501},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52200000000000002,"15min":0.51100000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.621}]},{"osd":6,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.503,"15min":0.504},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.32800000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":0.57599999999999996}]},{"osd":7,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56000000000000005,"15min":0.55100000000000005},"min":{"1min":0.32800000000000001,"5min":0.317,"15min":0.317},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56999999999999995,"15min":0.57299999999999995},"min":{"1min":0.34300000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.63600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:26:43.783 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:26:43.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.934+0000 7f3d3c0b4700 1 Processor -- start 2024-03-18T20:26:43.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.935+0000 7f3d3c0b4700 1 -- start start 2024-03-18T20:26:43.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.935+0000 7f3d3c0b4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3d34144b70 0x7f3d340ce980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.935+0000 7f3d3c0b4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d340af610 0x7f3d340cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.935+0000 7f3d3c0b4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3d340ccf40 0x7f3d3415ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.935+0000 7f3d3c0b4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d340cd3d0 con 0x7f3d340ccf40 2024-03-18T20:26:43.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.935+0000 7f3d3c0b4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d34144f50 con 0x7f3d34144b70 2024-03-18T20:26:43.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.935+0000 7f3d3c0b4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d340cf2d0 con 0x7f3d340af610 2024-03-18T20:26:43.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.936+0000 7f3d3a651700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3d340ccf40 0x7f3d3415ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.936+0000 7f3d3a651700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3d340ccf40 0x7f3d3415ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35966/0 (socket says 172.21.15.1:35966) 2024-03-18T20:26:43.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.936+0000 7f3d3a651700 1 -- 172.21.15.1:0/4270768285 learned_addr learned my addr 172.21.15.1:0/4270768285 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:43.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.936+0000 7f3d3964f700 1 --2- 172.21.15.1:0/4270768285 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d340af610 0x7f3d340cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.936+0000 7f3d39e50700 1 --2- 172.21.15.1:0/4270768285 >> v2:172.21.15.165:3300/0 conn(0x7f3d34144b70 0x7f3d340ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.936+0000 7f3d3a651700 1 -- 172.21.15.1:0/4270768285 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d340af610 msgr2=0x7f3d340cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:43.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.936+0000 7f3d3a651700 1 --2- 172.21.15.1:0/4270768285 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d340af610 0x7f3d340cca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.936+0000 7f3d3a651700 1 -- 172.21.15.1:0/4270768285 >> v2:172.21.15.165:3300/0 conn(0x7f3d34144b70 msgr2=0x7f3d340ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:43.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.936+0000 7f3d3a651700 1 --2- 172.21.15.1:0/4270768285 >> v2:172.21.15.165:3300/0 conn(0x7f3d34144b70 0x7f3d340ce980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.936+0000 7f3d3a651700 1 -- 172.21.15.1:0/4270768285 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d340cf5b0 con 0x7f3d340ccf40 2024-03-18T20:26:43.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.936+0000 7f3d3a651700 1 --2- 172.21.15.1:0/4270768285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d340ccf40 0x7f3d3415ecf0 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7f3d3406d080 tx=0x7f3d2c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29eac89b4285fc19 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:43.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.936+0000 7f3d38e4e700 1 -- 172.21.15.1:0/4270768285 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c00e070 con 0x7f3d340ccf40 2024-03-18T20:26:43.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.936+0000 7f3d38e4e700 1 -- 172.21.15.1:0/4270768285 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2c00b9c0 con 0x7f3d340ccf40 2024-03-18T20:26:43.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.936+0000 7f3d38e4e700 1 -- 172.21.15.1:0/4270768285 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c00bd70 con 0x7f3d340ccf40 2024-03-18T20:26:43.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.937+0000 7f3d3c0b4700 1 -- 172.21.15.1:0/4270768285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d340ccf40 msgr2=0x7f3d3415ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:43.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.937+0000 7f3d3c0b4700 1 --2- 172.21.15.1:0/4270768285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d340ccf40 0x7f3d3415ecf0 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7f3d3406d080 tx=0x7f3d2c00b330 comp rx=0 tx=0).stop 2024-03-18T20:26:43.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.937+0000 7f3d3c0b4700 1 -- 172.21.15.1:0/4270768285 shutdown_connections 2024-03-18T20:26:43.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.937+0000 7f3d3c0b4700 1 --2- 172.21.15.1:0/4270768285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d340ccf40 0x7f3d3415ecf0 unknown :-1 s=CLOSED pgs=2873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.937+0000 7f3d3c0b4700 1 --2- 172.21.15.1:0/4270768285 >> v2:172.21.15.165:3300/0 conn(0x7f3d34144b70 0x7f3d340ce980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.937+0000 7f3d3c0b4700 1 --2- 172.21.15.1:0/4270768285 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d340af610 0x7f3d340cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.937+0000 7f3d3c0b4700 1 -- 172.21.15.1:0/4270768285 >> 172.21.15.1:0/4270768285 conn(0x7f3d340b1d20 msgr2=0x7f3d340b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:43.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.937+0000 7f3d3c0b4700 1 -- 172.21.15.1:0/4270768285 shutdown_connections 2024-03-18T20:26:43.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.937+0000 7f3d3c0b4700 1 -- 172.21.15.1:0/4270768285 wait complete. 2024-03-18T20:26:43.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.938+0000 7f3d3c0b4700 1 Processor -- start 2024-03-18T20:26:43.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.938+0000 7f3d3c0b4700 1 -- start start 2024-03-18T20:26:43.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.939+0000 7f3d3c0b4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d340af610 0x7f3d3413a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.939+0000 7f3d3c0b4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d340ccf40 0x7f3d34156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.939+0000 7f3d3c0b4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d34144b70 0x7f3d34158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.939+0000 7f3d3c0b4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d34139e30 con 0x7f3d340af610 2024-03-18T20:26:43.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.939+0000 7f3d3c0b4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d34139260 con 0x7f3d34144b70 2024-03-18T20:26:43.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.939+0000 7f3d3c0b4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d34139500 con 0x7f3d340ccf40 2024-03-18T20:26:43.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.939+0000 7f3d39e50700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d340af610 0x7f3d3413a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.939+0000 7f3d39e50700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d340af610 0x7f3d3413a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35980/0 (socket says 172.21.15.1:35980) 2024-03-18T20:26:43.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.939+0000 7f3d39e50700 1 -- 172.21.15.1:0/1591153375 learned_addr learned my addr 172.21.15.1:0/1591153375 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:43.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.939+0000 7f3d3964f700 1 --2- 172.21.15.1:0/1591153375 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d340ccf40 0x7f3d34156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.939+0000 7f3d3a651700 1 --2- 172.21.15.1:0/1591153375 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d34144b70 0x7f3d34158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.939+0000 7f3d39e50700 1 -- 172.21.15.1:0/1591153375 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d340ccf40 msgr2=0x7f3d34156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:43.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.939+0000 7f3d39e50700 1 --2- 172.21.15.1:0/1591153375 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d340ccf40 0x7f3d34156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.939+0000 7f3d39e50700 1 -- 172.21.15.1:0/1591153375 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d34144b70 msgr2=0x7f3d34158880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.939+0000 7f3d39e50700 1 --2- 172.21.15.1:0/1591153375 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d34144b70 0x7f3d34158880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.939+0000 7f3d39e50700 1 -- 172.21.15.1:0/1591153375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2c009b30 con 0x7f3d340af610 2024-03-18T20:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.939+0000 7f3d39e50700 1 --2- 172.21.15.1:0/1591153375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d340af610 0x7f3d3413a2c0 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 crypto rx=0x7f3d3000d800 tx=0x7f3d3000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.940+0000 7f3d2a7fc700 1 -- 172.21.15.1:0/1591153375 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d300125f0 con 0x7f3d340af610 2024-03-18T20:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.940+0000 7f3d2a7fc700 1 -- 172.21.15.1:0/1591153375 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d30012750 con 0x7f3d340af610 2024-03-18T20:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.940+0000 7f3d3c0b4700 1 -- 172.21.15.1:0/1591153375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3406ecb0 con 0x7f3d340af610 2024-03-18T20:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.940+0000 7f3d2a7fc700 1 -- 172.21.15.1:0/1591153375 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d30012b00 con 0x7f3d340af610 2024-03-18T20:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.940+0000 7f3d3c0b4700 1 -- 172.21.15.1:0/1591153375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d34139ac0 con 0x7f3d340af610 2024-03-18T20:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.940+0000 7f3d3c0b4700 1 -- 172.21.15.1:0/1591153375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d340cda90 con 0x7f3d340af610 2024-03-18T20:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.941+0000 7f3d2a7fc700 1 -- 172.21.15.1:0/1591153375 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f3d30012c60 con 0x7f3d340af610 2024-03-18T20:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.941+0000 7f3d2a7fc700 1 --2- 172.21.15.1:0/1591153375 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3d1806aed0 0x7f3d1806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.942+0000 7f3d2a7fc700 1 -- 172.21.15.1:0/1591153375 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f3d30089c80 con 0x7f3d340af610 2024-03-18T20:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.942+0000 7f3d3964f700 1 --2- 172.21.15.1:0/1591153375 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3d1806aed0 0x7f3d1806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:43.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.942+0000 7f3d3964f700 1 --2- 172.21.15.1:0/1591153375 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3d1806aed0 0x7f3d1806d390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3d34142060 tx=0x7f3d24008040 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:43.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:43.946+0000 7f3d2a7fc700 1 -- 172.21.15.1:0/1591153375 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3d300545f0 con 0x7f3d340af610 2024-03-18T20:26:44.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.146+0000 7f3d3c0b4700 1 -- 172.21.15.1:0/1591153375 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3d3406eb30 con 0x7f3d1806aed0 2024-03-18T20:26:44.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.160+0000 7f3d2a7fc700 1 -- 172.21.15.1:0/1591153375 <== mgr.12057 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+528961 (secure 0 0 0) 0x7f3d24009e40 con 0x7f3d1806aed0 2024-03-18T20:26:44.171 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:44.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.164+0000 7f3d3c0b4700 1 -- 172.21.15.1:0/1591153375 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3d1806aed0 msgr2=0x7f3d1806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:44.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.164+0000 7f3d3c0b4700 1 --2- 172.21.15.1:0/1591153375 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3d1806aed0 0x7f3d1806d390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3d34142060 tx=0x7f3d24008040 comp rx=0 tx=0).stop 2024-03-18T20:26:44.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.164+0000 7f3d3c0b4700 1 -- 172.21.15.1:0/1591153375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d340af610 msgr2=0x7f3d3413a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:44.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.164+0000 7f3d3c0b4700 1 --2- 172.21.15.1:0/1591153375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d340af610 0x7f3d3413a2c0 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 crypto rx=0x7f3d3000d800 tx=0x7f3d3000db10 comp rx=0 tx=0).stop 2024-03-18T20:26:44.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.164+0000 7f3d3c0b4700 1 -- 172.21.15.1:0/1591153375 shutdown_connections 2024-03-18T20:26:44.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.164+0000 7f3d3c0b4700 1 --2- 172.21.15.1:0/1591153375 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3d1806aed0 0x7f3d1806d390 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:44.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.164+0000 7f3d3c0b4700 1 --2- 172.21.15.1:0/1591153375 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d34144b70 0x7f3d34158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:44.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.164+0000 7f3d3c0b4700 1 --2- 172.21.15.1:0/1591153375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d340af610 0x7f3d3413a2c0 unknown :-1 s=CLOSED pgs=2874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:44.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.164+0000 7f3d3c0b4700 1 --2- 172.21.15.1:0/1591153375 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3d340ccf40 0x7f3d34156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:44.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.164+0000 7f3d3c0b4700 1 -- 172.21.15.1:0/1591153375 >> 172.21.15.1:0/1591153375 conn(0x7f3d340b1d20 msgr2=0x7f3d340b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:44.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.164+0000 7f3d3c0b4700 1 -- 172.21.15.1:0/1591153375 shutdown_connections 2024-03-18T20:26:44.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.164+0000 7f3d3c0b4700 1 -- 172.21.15.1:0/1591153375 wait complete. 2024-03-18T20:26:44.177 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:26:44.180 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-03-18T20:26:43.879559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596350,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":710,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":449,"ondisk_log_size":449,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":226,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":415544,"kb_used_data":26308,"kb_used_omap":12,"kb_used_meta":389107,"kb_avail":754559176,"statfs":{"total":773094113280,"available":772668596224,"internally_reserved":0,"allocated":26939392,"data_stored":14445086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":398446137},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3177,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.721597"},"pg_stats":[{"pgid":"34.1d","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.622190+0000","last_change":"2024-03-18T20:26:39.614975+0000","last_active":"2024-03-18T20:26:39.622190+0000","last_peered":"2024-03-18T20:26:39.622190+0000","last_clean":"2024-03-18T20:26:39.622190+0000","last_became_active":"2024-03-18T20:26:39.614542+0000","last_became_peered":"2024-03-18T20:26:39.614542+0000","last_unstale":"2024-03-18T20:26:39.622190+0000","last_undegraded":"2024-03-18T20:26:39.622190+0000","last_fullsized":"2024-03-18T20:26:39.622190+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:20.170314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"256'1","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.615752+0000","last_change":"2024-03-18T20:26:39.176702+0000","last_active":"2024-03-18T20:26:39.615752+0000","last_peered":"2024-03-18T20:26:39.615752+0000","last_clean":"2024-03-18T20:26:39.615752+0000","last_became_active":"2024-03-18T20:26:39.176448+0000","last_became_peered":"2024-03-18T20:26:39.176448+0000","last_unstale":"2024-03-18T20:26:39.615752+0000","last_undegraded":"2024-03-18T20:26:39.615752+0000","last_fullsized":"2024-03-18T20:26:39.615752+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:56.821401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174088+0000","last_change":"2024-03-18T20:26:25.816062+0000","last_active":"2024-03-18T20:26:39.174088+0000","last_peered":"2024-03-18T20:26:39.174088+0000","last_clean":"2024-03-18T20:26:39.174088+0000","last_became_active":"2024-03-18T20:26:25.815866+0000","last_became_peered":"2024-03-18T20:26:25.815866+0000","last_unstale":"2024-03-18T20:26:39.174088+0000","last_undegraded":"2024-03-18T20:26:39.174088+0000","last_fullsized":"2024-03-18T20:26:39.174088+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:19:53.852641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.625092+0000","last_change":"2024-03-18T20:26:39.623961+0000","last_active":"2024-03-18T20:26:39.625092+0000","last_peered":"2024-03-18T20:26:39.625092+0000","last_clean":"2024-03-18T20:26:39.625092+0000","last_became_active":"2024-03-18T20:26:39.623698+0000","last_became_peered":"2024-03-18T20:26:39.623698+0000","last_unstale":"2024-03-18T20:26:39.625092+0000","last_undegraded":"2024-03-18T20:26:39.625092+0000","last_fullsized":"2024-03-18T20:26:39.625092+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:54:42.615870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"256'4","reported_seq":14,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.624260+0000","last_change":"2024-03-18T20:26:39.182576+0000","last_active":"2024-03-18T20:26:39.624260+0000","last_peered":"2024-03-18T20:26:39.624260+0000","last_clean":"2024-03-18T20:26:39.624260+0000","last_became_active":"2024-03-18T20:26:39.182244+0000","last_became_peered":"2024-03-18T20:26:39.182244+0000","last_unstale":"2024-03-18T20:26:39.624260+0000","last_undegraded":"2024-03-18T20:26:39.624260+0000","last_fullsized":"2024-03-18T20:26:39.624260+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:01.961647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.173247+0000","last_change":"2024-03-18T20:26:25.806337+0000","last_active":"2024-03-18T20:26:39.173247+0000","last_peered":"2024-03-18T20:26:39.173247+0000","last_clean":"2024-03-18T20:26:39.173247+0000","last_became_active":"2024-03-18T20:26:25.806016+0000","last_became_peered":"2024-03-18T20:26:25.806016+0000","last_unstale":"2024-03-18T20:26:39.173247+0000","last_undegraded":"2024-03-18T20:26:39.173247+0000","last_fullsized":"2024-03-18T20:26:39.173247+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:33:55.816835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.626048+0000","last_change":"2024-03-18T20:26:39.617752+0000","last_active":"2024-03-18T20:26:39.626048+0000","last_peered":"2024-03-18T20:26:39.626048+0000","last_clean":"2024-03-18T20:26:39.626048+0000","last_became_active":"2024-03-18T20:26:39.617409+0000","last_became_peered":"2024-03-18T20:26:39.617409+0000","last_unstale":"2024-03-18T20:26:39.626048+0000","last_undegraded":"2024-03-18T20:26:39.626048+0000","last_fullsized":"2024-03-18T20:26:39.626048+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:59:55.686698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"256'2","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.616698+0000","last_change":"2024-03-18T20:26:39.173108+0000","last_active":"2024-03-18T20:26:39.616698+0000","last_peered":"2024-03-18T20:26:39.616698+0000","last_clean":"2024-03-18T20:26:39.616698+0000","last_became_active":"2024-03-18T20:26:39.172828+0000","last_became_peered":"2024-03-18T20:26:39.172828+0000","last_unstale":"2024-03-18T20:26:39.616698+0000","last_undegraded":"2024-03-18T20:26:39.616698+0000","last_fullsized":"2024-03-18T20:26:39.616698+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:14:31.591405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.165765+0000","last_change":"2024-03-18T20:26:25.815050+0000","last_active":"2024-03-18T20:26:39.165765+0000","last_peered":"2024-03-18T20:26:39.165765+0000","last_clean":"2024-03-18T20:26:39.165765+0000","last_became_active":"2024-03-18T20:26:25.814617+0000","last_became_peered":"2024-03-18T20:26:25.814617+0000","last_unstale":"2024-03-18T20:26:39.165765+0000","last_undegraded":"2024-03-18T20:26:39.165765+0000","last_fullsized":"2024-03-18T20:26:39.165765+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:04:32.495090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.632378+0000","last_change":"2024-03-18T20:26:39.631768+0000","last_active":"2024-03-18T20:26:39.632378+0000","last_peered":"2024-03-18T20:26:39.632378+0000","last_clean":"2024-03-18T20:26:39.632378+0000","last_became_active":"2024-03-18T20:26:39.631349+0000","last_became_peered":"2024-03-18T20:26:39.631349+0000","last_unstale":"2024-03-18T20:26:39.632378+0000","last_undegraded":"2024-03-18T20:26:39.632378+0000","last_fullsized":"2024-03-18T20:26:39.632378+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:35.406128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"256'2","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.619823+0000","last_change":"2024-03-18T20:26:39.186202+0000","last_active":"2024-03-18T20:26:39.619823+0000","last_peered":"2024-03-18T20:26:39.619823+0000","last_clean":"2024-03-18T20:26:39.619823+0000","last_became_active":"2024-03-18T20:26:39.185958+0000","last_became_peered":"2024-03-18T20:26:39.185958+0000","last_unstale":"2024-03-18T20:26:39.619823+0000","last_undegraded":"2024-03-18T20:26:39.619823+0000","last_fullsized":"2024-03-18T20:26:39.619823+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:37:53.931774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166170+0000","last_change":"2024-03-18T20:26:25.812709+0000","last_active":"2024-03-18T20:26:39.166170+0000","last_peered":"2024-03-18T20:26:39.166170+0000","last_clean":"2024-03-18T20:26:39.166170+0000","last_became_active":"2024-03-18T20:26:25.812420+0000","last_became_peered":"2024-03-18T20:26:25.812420+0000","last_unstale":"2024-03-18T20:26:39.166170+0000","last_undegraded":"2024-03-18T20:26:39.166170+0000","last_fullsized":"2024-03-18T20:26:39.166170+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:52.955865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.617657+0000","last_change":"2024-03-18T20:26:39.614084+0000","last_active":"2024-03-18T20:26:39.617657+0000","last_peered":"2024-03-18T20:26:39.617657+0000","last_clean":"2024-03-18T20:26:39.617657+0000","last_became_active":"2024-03-18T20:26:39.613535+0000","last_became_peered":"2024-03-18T20:26:39.613535+0000","last_unstale":"2024-03-18T20:26:39.617657+0000","last_undegraded":"2024-03-18T20:26:39.617657+0000","last_fullsized":"2024-03-18T20:26:39.617657+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:14:33.759409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"256'1","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.625221+0000","last_change":"2024-03-18T20:26:39.186135+0000","last_active":"2024-03-18T20:26:39.625221+0000","last_peered":"2024-03-18T20:26:39.625221+0000","last_clean":"2024-03-18T20:26:39.625221+0000","last_became_active":"2024-03-18T20:26:39.185890+0000","last_became_peered":"2024-03-18T20:26:39.185890+0000","last_unstale":"2024-03-18T20:26:39.625221+0000","last_undegraded":"2024-03-18T20:26:39.625221+0000","last_fullsized":"2024-03-18T20:26:39.625221+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:41:38.363283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166761+0000","last_change":"2024-03-18T20:26:25.811447+0000","last_active":"2024-03-18T20:26:39.166761+0000","last_peered":"2024-03-18T20:26:39.166761+0000","last_clean":"2024-03-18T20:26:39.166761+0000","last_became_active":"2024-03-18T20:26:25.811144+0000","last_became_peered":"2024-03-18T20:26:25.811144+0000","last_unstale":"2024-03-18T20:26:39.166761+0000","last_undegraded":"2024-03-18T20:26:39.166761+0000","last_fullsized":"2024-03-18T20:26:39.166761+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:54:38.189655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629539+0000","last_change":"2024-03-18T20:26:39.624778+0000","last_active":"2024-03-18T20:26:39.629539+0000","last_peered":"2024-03-18T20:26:39.629539+0000","last_clean":"2024-03-18T20:26:39.629539+0000","last_became_active":"2024-03-18T20:26:39.621807+0000","last_became_peered":"2024-03-18T20:26:39.621807+0000","last_unstale":"2024-03-18T20:26:39.629539+0000","last_undegraded":"2024-03-18T20:26:39.629539+0000","last_fullsized":"2024-03-18T20:26:39.629539+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:29.701161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"256'1","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.621775+0000","last_change":"2024-03-18T20:26:39.179129+0000","last_active":"2024-03-18T20:26:39.621775+0000","last_peered":"2024-03-18T20:26:39.621775+0000","last_clean":"2024-03-18T20:26:39.621775+0000","last_became_active":"2024-03-18T20:26:39.178863+0000","last_became_peered":"2024-03-18T20:26:39.178863+0000","last_unstale":"2024-03-18T20:26:39.621775+0000","last_undegraded":"2024-03-18T20:26:39.621775+0000","last_fullsized":"2024-03-18T20:26:39.621775+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:16:23.162470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.173083+0000","last_change":"2024-03-18T20:26:25.804382+0000","last_active":"2024-03-18T20:26:39.173083+0000","last_peered":"2024-03-18T20:26:39.173083+0000","last_clean":"2024-03-18T20:26:39.173083+0000","last_became_active":"2024-03-18T20:26:25.804096+0000","last_became_peered":"2024-03-18T20:26:25.804096+0000","last_unstale":"2024-03-18T20:26:39.173083+0000","last_undegraded":"2024-03-18T20:26:39.173083+0000","last_fullsized":"2024-03-18T20:26:39.173083+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:23.578829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629593+0000","last_change":"2024-03-18T20:26:39.627440+0000","last_active":"2024-03-18T20:26:39.629593+0000","last_peered":"2024-03-18T20:26:39.629593+0000","last_clean":"2024-03-18T20:26:39.629593+0000","last_became_active":"2024-03-18T20:26:39.626732+0000","last_became_peered":"2024-03-18T20:26:39.626732+0000","last_unstale":"2024-03-18T20:26:39.629593+0000","last_undegraded":"2024-03-18T20:26:39.629593+0000","last_fullsized":"2024-03-18T20:26:39.629593+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:23:05.984738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.601980+0000","last_change":"2024-03-18T20:26:39.172303+0000","last_active":"2024-03-18T20:26:39.601980+0000","last_peered":"2024-03-18T20:26:39.601980+0000","last_clean":"2024-03-18T20:26:39.601980+0000","last_became_active":"2024-03-18T20:26:39.172023+0000","last_became_peered":"2024-03-18T20:26:39.172023+0000","last_unstale":"2024-03-18T20:26:39.601980+0000","last_undegraded":"2024-03-18T20:26:39.601980+0000","last_fullsized":"2024-03-18T20:26:39.601980+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:22:32.033460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.172295+0000","last_change":"2024-03-18T20:26:25.806198+0000","last_active":"2024-03-18T20:26:39.172295+0000","last_peered":"2024-03-18T20:26:39.172295+0000","last_clean":"2024-03-18T20:26:39.172295+0000","last_became_active":"2024-03-18T20:26:25.805918+0000","last_became_peered":"2024-03-18T20:26:25.805918+0000","last_unstale":"2024-03-18T20:26:39.172295+0000","last_undegraded":"2024-03-18T20:26:39.172295+0000","last_fullsized":"2024-03-18T20:26:39.172295+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:11.977302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629624+0000","last_change":"2024-03-18T20:26:39.629257+0000","last_active":"2024-03-18T20:26:39.629624+0000","last_peered":"2024-03-18T20:26:39.629624+0000","last_clean":"2024-03-18T20:26:39.629624+0000","last_became_active":"2024-03-18T20:26:39.628990+0000","last_became_peered":"2024-03-18T20:26:39.628990+0000","last_unstale":"2024-03-18T20:26:39.629624+0000","last_undegraded":"2024-03-18T20:26:39.629624+0000","last_fullsized":"2024-03-18T20:26:39.629624+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:14:12.949096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.601363+0000","last_change":"2024-03-18T20:26:39.172866+0000","last_active":"2024-03-18T20:26:39.601363+0000","last_peered":"2024-03-18T20:26:39.601363+0000","last_clean":"2024-03-18T20:26:39.601363+0000","last_became_active":"2024-03-18T20:26:39.172609+0000","last_became_peered":"2024-03-18T20:26:39.172609+0000","last_unstale":"2024-03-18T20:26:39.601363+0000","last_undegraded":"2024-03-18T20:26:39.601363+0000","last_fullsized":"2024-03-18T20:26:39.601363+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:21.784700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.165779+0000","last_change":"2024-03-18T20:26:25.805458+0000","last_active":"2024-03-18T20:26:39.165779+0000","last_peered":"2024-03-18T20:26:39.165779+0000","last_clean":"2024-03-18T20:26:39.165779+0000","last_became_active":"2024-03-18T20:26:25.805176+0000","last_became_peered":"2024-03-18T20:26:25.805176+0000","last_unstale":"2024-03-18T20:26:39.165779+0000","last_undegraded":"2024-03-18T20:26:39.165779+0000","last_fullsized":"2024-03-18T20:26:39.165779+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:06:22.720260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.621416+0000","last_change":"2024-03-18T20:26:39.614803+0000","last_active":"2024-03-18T20:26:39.621416+0000","last_peered":"2024-03-18T20:26:39.621416+0000","last_clean":"2024-03-18T20:26:39.621416+0000","last_became_active":"2024-03-18T20:26:39.614214+0000","last_became_peered":"2024-03-18T20:26:39.614214+0000","last_unstale":"2024-03-18T20:26:39.621416+0000","last_undegraded":"2024-03-18T20:26:39.621416+0000","last_fullsized":"2024-03-18T20:26:39.621416+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:48:12.952796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.606260+0000","last_change":"2024-03-18T20:26:39.172945+0000","last_active":"2024-03-18T20:26:39.606260+0000","last_peered":"2024-03-18T20:26:39.606260+0000","last_clean":"2024-03-18T20:26:39.606260+0000","last_became_active":"2024-03-18T20:26:39.172504+0000","last_became_peered":"2024-03-18T20:26:39.172504+0000","last_unstale":"2024-03-18T20:26:39.606260+0000","last_undegraded":"2024-03-18T20:26:39.606260+0000","last_fullsized":"2024-03-18T20:26:39.606260+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:22.609793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174207+0000","last_change":"2024-03-18T20:26:25.815665+0000","last_active":"2024-03-18T20:26:39.174207+0000","last_peered":"2024-03-18T20:26:39.174207+0000","last_clean":"2024-03-18T20:26:39.174207+0000","last_became_active":"2024-03-18T20:26:25.815259+0000","last_became_peered":"2024-03-18T20:26:25.815259+0000","last_unstale":"2024-03-18T20:26:39.174207+0000","last_undegraded":"2024-03-18T20:26:39.174207+0000","last_fullsized":"2024-03-18T20:26:39.174207+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:39:11.551926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.621981+0000","last_change":"2024-03-18T20:26:39.619565+0000","last_active":"2024-03-18T20:26:39.621981+0000","last_peered":"2024-03-18T20:26:39.621981+0000","last_clean":"2024-03-18T20:26:39.621981+0000","last_became_active":"2024-03-18T20:26:39.619317+0000","last_became_peered":"2024-03-18T20:26:39.619317+0000","last_unstale":"2024-03-18T20:26:39.621981+0000","last_undegraded":"2024-03-18T20:26:39.621981+0000","last_fullsized":"2024-03-18T20:26:39.621981+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:39:58.561762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.611666+0000","last_change":"2024-03-18T20:26:39.180298+0000","last_active":"2024-03-18T20:26:39.611666+0000","last_peered":"2024-03-18T20:26:39.611666+0000","last_clean":"2024-03-18T20:26:39.611666+0000","last_became_active":"2024-03-18T20:26:39.179626+0000","last_became_peered":"2024-03-18T20:26:39.179626+0000","last_unstale":"2024-03-18T20:26:39.611666+0000","last_undegraded":"2024-03-18T20:26:39.611666+0000","last_fullsized":"2024-03-18T20:26:39.611666+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:32:02.878109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.165829+0000","last_change":"2024-03-18T20:26:25.805234+0000","last_active":"2024-03-18T20:26:39.165829+0000","last_peered":"2024-03-18T20:26:39.165829+0000","last_clean":"2024-03-18T20:26:39.165829+0000","last_became_active":"2024-03-18T20:26:25.805056+0000","last_became_peered":"2024-03-18T20:26:25.805056+0000","last_unstale":"2024-03-18T20:26:39.165829+0000","last_undegraded":"2024-03-18T20:26:39.165829+0000","last_fullsized":"2024-03-18T20:26:39.165829+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:50.003103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.626369+0000","last_change":"2024-03-18T20:26:39.620111+0000","last_active":"2024-03-18T20:26:39.626369+0000","last_peered":"2024-03-18T20:26:39.626369+0000","last_clean":"2024-03-18T20:26:39.626369+0000","last_became_active":"2024-03-18T20:26:39.619855+0000","last_became_peered":"2024-03-18T20:26:39.619855+0000","last_unstale":"2024-03-18T20:26:39.626369+0000","last_undegraded":"2024-03-18T20:26:39.626369+0000","last_fullsized":"2024-03-18T20:26:39.626369+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:22:44.464056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"256'6","reported_seq":16,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.632214+0000","last_change":"2024-03-18T20:26:39.180393+0000","last_active":"2024-03-18T20:26:39.632214+0000","last_peered":"2024-03-18T20:26:39.632214+0000","last_clean":"2024-03-18T20:26:39.632214+0000","last_became_active":"2024-03-18T20:26:39.180140+0000","last_became_peered":"2024-03-18T20:26:39.180140+0000","last_unstale":"2024-03-18T20:26:39.632214+0000","last_undegraded":"2024-03-18T20:26:39.632214+0000","last_fullsized":"2024-03-18T20:26:39.632214+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:32.551571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.165802+0000","last_change":"2024-03-18T20:26:25.813598+0000","last_active":"2024-03-18T20:26:39.165802+0000","last_peered":"2024-03-18T20:26:39.165802+0000","last_clean":"2024-03-18T20:26:39.165802+0000","last_became_active":"2024-03-18T20:26:25.812093+0000","last_became_peered":"2024-03-18T20:26:25.812093+0000","last_unstale":"2024-03-18T20:26:39.165802+0000","last_undegraded":"2024-03-18T20:26:39.165802+0000","last_fullsized":"2024-03-18T20:26:39.165802+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:38.417379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.628746+0000","last_change":"2024-03-18T20:26:39.628200+0000","last_active":"2024-03-18T20:26:39.628746+0000","last_peered":"2024-03-18T20:26:39.628746+0000","last_clean":"2024-03-18T20:26:39.628746+0000","last_became_active":"2024-03-18T20:26:39.627978+0000","last_became_peered":"2024-03-18T20:26:39.627978+0000","last_unstale":"2024-03-18T20:26:39.628746+0000","last_undegraded":"2024-03-18T20:26:39.628746+0000","last_fullsized":"2024-03-18T20:26:39.628746+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:39:20.975255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"256'3","reported_seq":13,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.628662+0000","last_change":"2024-03-18T20:26:39.177990+0000","last_active":"2024-03-18T20:26:39.628662+0000","last_peered":"2024-03-18T20:26:39.628662+0000","last_clean":"2024-03-18T20:26:39.628662+0000","last_became_active":"2024-03-18T20:26:39.177721+0000","last_became_peered":"2024-03-18T20:26:39.177721+0000","last_unstale":"2024-03-18T20:26:39.628662+0000","last_undegraded":"2024-03-18T20:26:39.628662+0000","last_fullsized":"2024-03-18T20:26:39.628662+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:21:35.960350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174103+0000","last_change":"2024-03-18T20:26:25.811095+0000","last_active":"2024-03-18T20:26:39.174103+0000","last_peered":"2024-03-18T20:26:39.174103+0000","last_clean":"2024-03-18T20:26:39.174103+0000","last_became_active":"2024-03-18T20:26:25.810796+0000","last_became_peered":"2024-03-18T20:26:25.810796+0000","last_unstale":"2024-03-18T20:26:39.174103+0000","last_undegraded":"2024-03-18T20:26:39.174103+0000","last_fullsized":"2024-03-18T20:26:39.174103+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:30:08.833713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.624868+0000","last_change":"2024-03-18T20:26:39.619308+0000","last_active":"2024-03-18T20:26:39.624868+0000","last_peered":"2024-03-18T20:26:39.624868+0000","last_clean":"2024-03-18T20:26:39.624868+0000","last_became_active":"2024-03-18T20:26:39.619018+0000","last_became_peered":"2024-03-18T20:26:39.619018+0000","last_unstale":"2024-03-18T20:26:39.624868+0000","last_undegraded":"2024-03-18T20:26:39.624868+0000","last_fullsized":"2024-03-18T20:26:39.624868+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:14:12.949096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.619228+0000","last_change":"2024-03-18T20:26:39.177301+0000","last_active":"2024-03-18T20:26:39.619228+0000","last_peered":"2024-03-18T20:26:39.619228+0000","last_clean":"2024-03-18T20:26:39.619228+0000","last_became_active":"2024-03-18T20:26:39.176934+0000","last_became_peered":"2024-03-18T20:26:39.176934+0000","last_unstale":"2024-03-18T20:26:39.619228+0000","last_undegraded":"2024-03-18T20:26:39.619228+0000","last_fullsized":"2024-03-18T20:26:39.619228+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:37:06.271998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174153+0000","last_change":"2024-03-18T20:26:25.815198+0000","last_active":"2024-03-18T20:26:39.174153+0000","last_peered":"2024-03-18T20:26:39.174153+0000","last_clean":"2024-03-18T20:26:39.174153+0000","last_became_active":"2024-03-18T20:26:25.814922+0000","last_became_peered":"2024-03-18T20:26:25.814922+0000","last_unstale":"2024-03-18T20:26:39.174153+0000","last_undegraded":"2024-03-18T20:26:39.174153+0000","last_fullsized":"2024-03-18T20:26:39.174153+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:30:14.320846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.614689+0000","last_change":"2024-03-18T20:26:39.609822+0000","last_active":"2024-03-18T20:26:39.614689+0000","last_peered":"2024-03-18T20:26:39.614689+0000","last_clean":"2024-03-18T20:26:39.614689+0000","last_became_active":"2024-03-18T20:26:39.609495+0000","last_became_peered":"2024-03-18T20:26:39.609495+0000","last_unstale":"2024-03-18T20:26:39.614689+0000","last_undegraded":"2024-03-18T20:26:39.614689+0000","last_fullsized":"2024-03-18T20:26:39.614689+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:57.528824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"256'2","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.622034+0000","last_change":"2024-03-18T20:26:39.174564+0000","last_active":"2024-03-18T20:26:39.622034+0000","last_peered":"2024-03-18T20:26:39.622034+0000","last_clean":"2024-03-18T20:26:39.622034+0000","last_became_active":"2024-03-18T20:26:39.172872+0000","last_became_peered":"2024-03-18T20:26:39.172872+0000","last_unstale":"2024-03-18T20:26:39.622034+0000","last_undegraded":"2024-03-18T20:26:39.622034+0000","last_fullsized":"2024-03-18T20:26:39.622034+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:16.137977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.184511+0000","last_change":"2024-03-18T20:26:25.809230+0000","last_active":"2024-03-18T20:26:39.184511+0000","last_peered":"2024-03-18T20:26:39.184511+0000","last_clean":"2024-03-18T20:26:39.184511+0000","last_became_active":"2024-03-18T20:26:25.808941+0000","last_became_peered":"2024-03-18T20:26:25.808941+0000","last_unstale":"2024-03-18T20:26:39.184511+0000","last_undegraded":"2024-03-18T20:26:39.184511+0000","last_fullsized":"2024-03-18T20:26:39.184511+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:34:05.214460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.619296+0000","last_change":"2024-03-18T20:26:39.615032+0000","last_active":"2024-03-18T20:26:39.619296+0000","last_peered":"2024-03-18T20:26:39.619296+0000","last_clean":"2024-03-18T20:26:39.619296+0000","last_became_active":"2024-03-18T20:26:39.614785+0000","last_became_peered":"2024-03-18T20:26:39.614785+0000","last_unstale":"2024-03-18T20:26:39.619296+0000","last_undegraded":"2024-03-18T20:26:39.619296+0000","last_fullsized":"2024-03-18T20:26:39.619296+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:30:36.759356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"256'8","reported_seq":18,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.632913+0000","last_change":"2024-03-18T20:26:39.184566+0000","last_active":"2024-03-18T20:26:39.632913+0000","last_peered":"2024-03-18T20:26:39.632913+0000","last_clean":"2024-03-18T20:26:39.632913+0000","last_became_active":"2024-03-18T20:26:39.184288+0000","last_became_peered":"2024-03-18T20:26:39.184288+0000","last_unstale":"2024-03-18T20:26:39.632913+0000","last_undegraded":"2024-03-18T20:26:39.632913+0000","last_fullsized":"2024-03-18T20:26:39.632913+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:27:08.199457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164301+0000","last_change":"2024-03-18T20:26:25.798972+0000","last_active":"2024-03-18T20:26:39.164301+0000","last_peered":"2024-03-18T20:26:39.164301+0000","last_clean":"2024-03-18T20:26:39.164301+0000","last_became_active":"2024-03-18T20:26:25.798736+0000","last_became_peered":"2024-03-18T20:26:25.798736+0000","last_unstale":"2024-03-18T20:26:39.164301+0000","last_undegraded":"2024-03-18T20:26:39.164301+0000","last_fullsized":"2024-03-18T20:26:39.164301+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:56.892630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.614724+0000","last_change":"2024-03-18T20:26:39.612586+0000","last_active":"2024-03-18T20:26:39.614724+0000","last_peered":"2024-03-18T20:26:39.614724+0000","last_clean":"2024-03-18T20:26:39.614724+0000","last_became_active":"2024-03-18T20:26:39.612337+0000","last_became_peered":"2024-03-18T20:26:39.612337+0000","last_unstale":"2024-03-18T20:26:39.614724+0000","last_undegraded":"2024-03-18T20:26:39.614724+0000","last_fullsized":"2024-03-18T20:26:39.614724+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:14:33.759409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.619961+0000","last_change":"2024-03-18T20:26:39.178899+0000","last_active":"2024-03-18T20:26:39.619961+0000","last_peered":"2024-03-18T20:26:39.619961+0000","last_clean":"2024-03-18T20:26:39.619961+0000","last_became_active":"2024-03-18T20:26:39.178634+0000","last_became_peered":"2024-03-18T20:26:39.178634+0000","last_unstale":"2024-03-18T20:26:39.619961+0000","last_undegraded":"2024-03-18T20:26:39.619961+0000","last_fullsized":"2024-03-18T20:26:39.619961+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:16:43.290784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.165956+0000","last_change":"2024-03-18T20:26:25.799408+0000","last_active":"2024-03-18T20:26:39.165956+0000","last_peered":"2024-03-18T20:26:39.165956+0000","last_clean":"2024-03-18T20:26:39.165956+0000","last_became_active":"2024-03-18T20:26:25.798788+0000","last_became_peered":"2024-03-18T20:26:25.798788+0000","last_unstale":"2024-03-18T20:26:39.165956+0000","last_undegraded":"2024-03-18T20:26:39.165956+0000","last_fullsized":"2024-03-18T20:26:39.165956+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:34:05.214460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.621773+0000","last_change":"2024-03-18T20:26:39.618070+0000","last_active":"2024-03-18T20:26:39.621773+0000","last_peered":"2024-03-18T20:26:39.621773+0000","last_clean":"2024-03-18T20:26:39.621773+0000","last_became_active":"2024-03-18T20:26:39.617706+0000","last_became_peered":"2024-03-18T20:26:39.617706+0000","last_unstale":"2024-03-18T20:26:39.621773+0000","last_undegraded":"2024-03-18T20:26:39.621773+0000","last_fullsized":"2024-03-18T20:26:39.621773+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:20.170314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"256'2","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.625601+0000","last_change":"2024-03-18T20:26:39.182659+0000","last_active":"2024-03-18T20:26:39.625601+0000","last_peered":"2024-03-18T20:26:39.625601+0000","last_clean":"2024-03-18T20:26:39.625601+0000","last_became_active":"2024-03-18T20:26:39.180908+0000","last_became_peered":"2024-03-18T20:26:39.180908+0000","last_unstale":"2024-03-18T20:26:39.625601+0000","last_undegraded":"2024-03-18T20:26:39.625601+0000","last_fullsized":"2024-03-18T20:26:39.625601+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:37:53.931774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.175207+0000","last_change":"2024-03-18T20:26:25.811390+0000","last_active":"2024-03-18T20:26:39.175207+0000","last_peered":"2024-03-18T20:26:39.175207+0000","last_clean":"2024-03-18T20:26:39.175207+0000","last_became_active":"2024-03-18T20:26:25.811081+0000","last_became_peered":"2024-03-18T20:26:25.811081+0000","last_unstale":"2024-03-18T20:26:39.175207+0000","last_undegraded":"2024-03-18T20:26:39.175207+0000","last_fullsized":"2024-03-18T20:26:39.175207+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:33:55.816835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.627069+0000","last_change":"2024-03-18T20:26:39.626443+0000","last_active":"2024-03-18T20:26:39.627069+0000","last_peered":"2024-03-18T20:26:39.627069+0000","last_clean":"2024-03-18T20:26:39.627069+0000","last_became_active":"2024-03-18T20:26:39.626065+0000","last_became_peered":"2024-03-18T20:26:39.626065+0000","last_unstale":"2024-03-18T20:26:39.627069+0000","last_undegraded":"2024-03-18T20:26:39.627069+0000","last_fullsized":"2024-03-18T20:26:39.627069+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:48:12.952796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"256'1","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.624610+0000","last_change":"2024-03-18T20:26:39.179734+0000","last_active":"2024-03-18T20:26:39.624610+0000","last_peered":"2024-03-18T20:26:39.624610+0000","last_clean":"2024-03-18T20:26:39.624610+0000","last_became_active":"2024-03-18T20:26:39.178954+0000","last_became_peered":"2024-03-18T20:26:39.178954+0000","last_unstale":"2024-03-18T20:26:39.624610+0000","last_undegraded":"2024-03-18T20:26:39.624610+0000","last_fullsized":"2024-03-18T20:26:39.624610+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:48:11.943600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.168586+0000","last_change":"2024-03-18T20:26:25.799778+0000","last_active":"2024-03-18T20:26:39.168586+0000","last_peered":"2024-03-18T20:26:39.168586+0000","last_clean":"2024-03-18T20:26:39.168586+0000","last_became_active":"2024-03-18T20:26:25.799517+0000","last_became_peered":"2024-03-18T20:26:25.799517+0000","last_unstale":"2024-03-18T20:26:39.168586+0000","last_undegraded":"2024-03-18T20:26:39.168586+0000","last_fullsized":"2024-03-18T20:26:39.168586+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:32:34.214872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.626393+0000","last_change":"2024-03-18T20:26:39.621407+0000","last_active":"2024-03-18T20:26:39.626393+0000","last_peered":"2024-03-18T20:26:39.626393+0000","last_clean":"2024-03-18T20:26:39.626393+0000","last_became_active":"2024-03-18T20:26:39.620628+0000","last_became_peered":"2024-03-18T20:26:39.620628+0000","last_unstale":"2024-03-18T20:26:39.626393+0000","last_undegraded":"2024-03-18T20:26:39.626393+0000","last_fullsized":"2024-03-18T20:26:39.626393+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:27:47.040042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.607231+0000","last_change":"2024-03-18T20:26:39.179668+0000","last_active":"2024-03-18T20:26:39.607231+0000","last_peered":"2024-03-18T20:26:39.607231+0000","last_clean":"2024-03-18T20:26:39.607231+0000","last_became_active":"2024-03-18T20:26:39.179415+0000","last_became_peered":"2024-03-18T20:26:39.179415+0000","last_unstale":"2024-03-18T20:26:39.607231+0000","last_undegraded":"2024-03-18T20:26:39.607231+0000","last_fullsized":"2024-03-18T20:26:39.607231+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:15.672080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164327+0000","last_change":"2024-03-18T20:26:25.812369+0000","last_active":"2024-03-18T20:26:39.164327+0000","last_peered":"2024-03-18T20:26:39.164327+0000","last_clean":"2024-03-18T20:26:39.164327+0000","last_became_active":"2024-03-18T20:26:25.812082+0000","last_became_peered":"2024-03-18T20:26:25.812082+0000","last_unstale":"2024-03-18T20:26:39.164327+0000","last_undegraded":"2024-03-18T20:26:39.164327+0000","last_fullsized":"2024-03-18T20:26:39.164327+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:43:00.585492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.632057+0000","last_change":"2024-03-18T20:26:39.630173+0000","last_active":"2024-03-18T20:26:39.632057+0000","last_peered":"2024-03-18T20:26:39.632057+0000","last_clean":"2024-03-18T20:26:39.632057+0000","last_became_active":"2024-03-18T20:26:39.629870+0000","last_became_peered":"2024-03-18T20:26:39.629870+0000","last_unstale":"2024-03-18T20:26:39.632057+0000","last_undegraded":"2024-03-18T20:26:39.632057+0000","last_fullsized":"2024-03-18T20:26:39.632057+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:02:00.876556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"256'1","reported_seq":11,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629431+0000","last_change":"2024-03-18T20:26:39.186556+0000","last_active":"2024-03-18T20:26:39.629431+0000","last_peered":"2024-03-18T20:26:39.629431+0000","last_clean":"2024-03-18T20:26:39.629431+0000","last_became_active":"2024-03-18T20:26:39.186231+0000","last_became_peered":"2024-03-18T20:26:39.186231+0000","last_unstale":"2024-03-18T20:26:39.629431+0000","last_undegraded":"2024-03-18T20:26:39.629431+0000","last_fullsized":"2024-03-18T20:26:39.629431+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:21.784700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.171204+0000","last_change":"2024-03-18T20:26:25.807591+0000","last_active":"2024-03-18T20:26:39.171204+0000","last_peered":"2024-03-18T20:26:39.171204+0000","last_clean":"2024-03-18T20:26:39.171204+0000","last_became_active":"2024-03-18T20:26:25.807349+0000","last_became_peered":"2024-03-18T20:26:25.807349+0000","last_unstale":"2024-03-18T20:26:39.171204+0000","last_undegraded":"2024-03-18T20:26:39.171204+0000","last_fullsized":"2024-03-18T20:26:39.171204+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:58.931342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629648+0000","last_change":"2024-03-18T20:26:39.629099+0000","last_active":"2024-03-18T20:26:39.629648+0000","last_peered":"2024-03-18T20:26:39.629648+0000","last_clean":"2024-03-18T20:26:39.629648+0000","last_became_active":"2024-03-18T20:26:39.628813+0000","last_became_peered":"2024-03-18T20:26:39.628813+0000","last_unstale":"2024-03-18T20:26:39.629648+0000","last_undegraded":"2024-03-18T20:26:39.629648+0000","last_fullsized":"2024-03-18T20:26:39.629648+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:22.793896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.602364+0000","last_change":"2024-03-18T20:26:39.187040+0000","last_active":"2024-03-18T20:26:39.602364+0000","last_peered":"2024-03-18T20:26:39.602364+0000","last_clean":"2024-03-18T20:26:39.602364+0000","last_became_active":"2024-03-18T20:26:39.186729+0000","last_became_peered":"2024-03-18T20:26:39.186729+0000","last_unstale":"2024-03-18T20:26:39.602364+0000","last_undegraded":"2024-03-18T20:26:39.602364+0000","last_fullsized":"2024-03-18T20:26:39.602364+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:43.273058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.173622+0000","last_change":"2024-03-18T20:26:25.799427+0000","last_active":"2024-03-18T20:26:39.173622+0000","last_peered":"2024-03-18T20:26:39.173622+0000","last_clean":"2024-03-18T20:26:39.173622+0000","last_became_active":"2024-03-18T20:26:25.799041+0000","last_became_peered":"2024-03-18T20:26:25.799041+0000","last_unstale":"2024-03-18T20:26:39.173622+0000","last_undegraded":"2024-03-18T20:26:39.173622+0000","last_fullsized":"2024-03-18T20:26:39.173622+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:35.919143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.624605+0000","last_change":"2024-03-18T20:26:39.617792+0000","last_active":"2024-03-18T20:26:39.624605+0000","last_peered":"2024-03-18T20:26:39.624605+0000","last_clean":"2024-03-18T20:26:39.624605+0000","last_became_active":"2024-03-18T20:26:39.617372+0000","last_became_peered":"2024-03-18T20:26:39.617372+0000","last_unstale":"2024-03-18T20:26:39.624605+0000","last_undegraded":"2024-03-18T20:26:39.624605+0000","last_fullsized":"2024-03-18T20:26:39.624605+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:21:15.706308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"256'2","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.630616+0000","last_change":"2024-03-18T20:26:39.180590+0000","last_active":"2024-03-18T20:26:39.630616+0000","last_peered":"2024-03-18T20:26:39.630616+0000","last_clean":"2024-03-18T20:26:39.630616+0000","last_became_active":"2024-03-18T20:26:39.180324+0000","last_became_peered":"2024-03-18T20:26:39.180324+0000","last_unstale":"2024-03-18T20:26:39.630616+0000","last_undegraded":"2024-03-18T20:26:39.630616+0000","last_fullsized":"2024-03-18T20:26:39.630616+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:18:56.346574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.172919+0000","last_change":"2024-03-18T20:26:25.818643+0000","last_active":"2024-03-18T20:26:39.172919+0000","last_peered":"2024-03-18T20:26:39.172919+0000","last_clean":"2024-03-18T20:26:39.172919+0000","last_became_active":"2024-03-18T20:26:25.818377+0000","last_became_peered":"2024-03-18T20:26:25.818377+0000","last_unstale":"2024-03-18T20:26:39.172919+0000","last_undegraded":"2024-03-18T20:26:39.172919+0000","last_fullsized":"2024-03-18T20:26:39.172919+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:44.809054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629294+0000","last_change":"2024-03-18T20:26:39.623520+0000","last_active":"2024-03-18T20:26:39.629294+0000","last_peered":"2024-03-18T20:26:39.629294+0000","last_clean":"2024-03-18T20:26:39.629294+0000","last_became_active":"2024-03-18T20:26:39.623271+0000","last_became_peered":"2024-03-18T20:26:39.623271+0000","last_unstale":"2024-03-18T20:26:39.629294+0000","last_undegraded":"2024-03-18T20:26:39.629294+0000","last_fullsized":"2024-03-18T20:26:39.629294+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:33:28.667851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.621155+0000","last_change":"2024-03-18T20:26:39.181245+0000","last_active":"2024-03-18T20:26:39.621155+0000","last_peered":"2024-03-18T20:26:39.621155+0000","last_clean":"2024-03-18T20:26:39.621155+0000","last_became_active":"2024-03-18T20:26:39.181004+0000","last_became_peered":"2024-03-18T20:26:39.181004+0000","last_unstale":"2024-03-18T20:26:39.621155+0000","last_undegraded":"2024-03-18T20:26:39.621155+0000","last_fullsized":"2024-03-18T20:26:39.621155+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:04.469128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164452+0000","last_change":"2024-03-18T20:26:25.805834+0000","last_active":"2024-03-18T20:26:39.164452+0000","last_peered":"2024-03-18T20:26:39.164452+0000","last_clean":"2024-03-18T20:26:39.164452+0000","last_became_active":"2024-03-18T20:26:25.805491+0000","last_became_peered":"2024-03-18T20:26:25.805491+0000","last_unstale":"2024-03-18T20:26:39.164452+0000","last_undegraded":"2024-03-18T20:26:39.164452+0000","last_fullsized":"2024-03-18T20:26:39.164452+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:23.305150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.623087+0000","last_change":"2024-03-18T20:26:39.621764+0000","last_active":"2024-03-18T20:26:39.623087+0000","last_peered":"2024-03-18T20:26:39.623087+0000","last_clean":"2024-03-18T20:26:39.623087+0000","last_became_active":"2024-03-18T20:26:39.619894+0000","last_became_peered":"2024-03-18T20:26:39.619894+0000","last_unstale":"2024-03-18T20:26:39.623087+0000","last_undegraded":"2024-03-18T20:26:39.623087+0000","last_fullsized":"2024-03-18T20:26:39.623087+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:10:35.340285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.609745+0000","last_change":"2024-03-18T20:26:39.190146+0000","last_active":"2024-03-18T20:26:39.609745+0000","last_peered":"2024-03-18T20:26:39.609745+0000","last_clean":"2024-03-18T20:26:39.609745+0000","last_became_active":"2024-03-18T20:26:39.189849+0000","last_became_peered":"2024-03-18T20:26:39.189849+0000","last_unstale":"2024-03-18T20:26:39.609745+0000","last_undegraded":"2024-03-18T20:26:39.609745+0000","last_fullsized":"2024-03-18T20:26:39.609745+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:39:12.102903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.171482+0000","last_change":"2024-03-18T20:26:25.809091+0000","last_active":"2024-03-18T20:26:39.171482+0000","last_peered":"2024-03-18T20:26:39.171482+0000","last_clean":"2024-03-18T20:26:39.171482+0000","last_became_active":"2024-03-18T20:26:25.808800+0000","last_became_peered":"2024-03-18T20:26:25.808800+0000","last_unstale":"2024-03-18T20:26:39.171482+0000","last_undegraded":"2024-03-18T20:26:39.171482+0000","last_fullsized":"2024-03-18T20:26:39.171482+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:32.033370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.632278+0000","last_change":"2024-03-18T20:26:39.631978+0000","last_active":"2024-03-18T20:26:39.632278+0000","last_peered":"2024-03-18T20:26:39.632278+0000","last_clean":"2024-03-18T20:26:39.632278+0000","last_became_active":"2024-03-18T20:26:39.631690+0000","last_became_peered":"2024-03-18T20:26:39.631690+0000","last_unstale":"2024-03-18T20:26:39.632278+0000","last_undegraded":"2024-03-18T20:26:39.632278+0000","last_fullsized":"2024-03-18T20:26:39.632278+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:07:57.157110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.607518+0000","last_change":"2024-03-18T20:26:39.180973+0000","last_active":"2024-03-18T20:26:39.607518+0000","last_peered":"2024-03-18T20:26:39.607518+0000","last_clean":"2024-03-18T20:26:39.607518+0000","last_became_active":"2024-03-18T20:26:39.180658+0000","last_became_peered":"2024-03-18T20:26:39.180658+0000","last_unstale":"2024-03-18T20:26:39.607518+0000","last_undegraded":"2024-03-18T20:26:39.607518+0000","last_fullsized":"2024-03-18T20:26:39.607518+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:36.104811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.172675+0000","last_change":"2024-03-18T20:26:25.798738+0000","last_active":"2024-03-18T20:26:39.172675+0000","last_peered":"2024-03-18T20:26:39.172675+0000","last_clean":"2024-03-18T20:26:39.172675+0000","last_became_active":"2024-03-18T20:26:25.798480+0000","last_became_peered":"2024-03-18T20:26:25.798480+0000","last_unstale":"2024-03-18T20:26:39.172675+0000","last_undegraded":"2024-03-18T20:26:39.172675+0000","last_fullsized":"2024-03-18T20:26:39.172675+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:02.900783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.627064+0000","last_change":"2024-03-18T20:26:39.624864+0000","last_active":"2024-03-18T20:26:39.627064+0000","last_peered":"2024-03-18T20:26:39.627064+0000","last_clean":"2024-03-18T20:26:39.627064+0000","last_became_active":"2024-03-18T20:26:39.622509+0000","last_became_peered":"2024-03-18T20:26:39.622509+0000","last_unstale":"2024-03-18T20:26:39.627064+0000","last_undegraded":"2024-03-18T20:26:39.627064+0000","last_fullsized":"2024-03-18T20:26:39.627064+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:32:52.861404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"256'2","reported_seq":12,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629923+0000","last_change":"2024-03-18T20:26:39.181574+0000","last_active":"2024-03-18T20:26:39.629923+0000","last_peered":"2024-03-18T20:26:39.629923+0000","last_clean":"2024-03-18T20:26:39.629923+0000","last_became_active":"2024-03-18T20:26:39.180033+0000","last_became_peered":"2024-03-18T20:26:39.180033+0000","last_unstale":"2024-03-18T20:26:39.629923+0000","last_undegraded":"2024-03-18T20:26:39.629923+0000","last_fullsized":"2024-03-18T20:26:39.629923+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:31:04.670960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.175259+0000","last_change":"2024-03-18T20:26:25.806356+0000","last_active":"2024-03-18T20:26:39.175259+0000","last_peered":"2024-03-18T20:26:39.175259+0000","last_clean":"2024-03-18T20:26:39.175259+0000","last_became_active":"2024-03-18T20:26:25.805895+0000","last_became_peered":"2024-03-18T20:26:25.805895+0000","last_unstale":"2024-03-18T20:26:39.175259+0000","last_undegraded":"2024-03-18T20:26:39.175259+0000","last_fullsized":"2024-03-18T20:26:39.175259+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:06:16.285934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.626223+0000","last_change":"2024-03-18T20:26:39.624150+0000","last_active":"2024-03-18T20:26:39.626223+0000","last_peered":"2024-03-18T20:26:39.626223+0000","last_clean":"2024-03-18T20:26:39.626223+0000","last_became_active":"2024-03-18T20:26:39.622660+0000","last_became_peered":"2024-03-18T20:26:39.622660+0000","last_unstale":"2024-03-18T20:26:39.626223+0000","last_undegraded":"2024-03-18T20:26:39.626223+0000","last_fullsized":"2024-03-18T20:26:39.626223+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:08:10.924769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.609219+0000","last_change":"2024-03-18T20:26:39.187982+0000","last_active":"2024-03-18T20:26:39.609219+0000","last_peered":"2024-03-18T20:26:39.609219+0000","last_clean":"2024-03-18T20:26:39.609219+0000","last_became_active":"2024-03-18T20:26:39.187760+0000","last_became_peered":"2024-03-18T20:26:39.187760+0000","last_unstale":"2024-03-18T20:26:39.609219+0000","last_undegraded":"2024-03-18T20:26:39.609219+0000","last_fullsized":"2024-03-18T20:26:39.609219+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:56.821401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.167507+0000","last_change":"2024-03-18T20:26:25.806860+0000","last_active":"2024-03-18T20:26:39.167507+0000","last_peered":"2024-03-18T20:26:39.167507+0000","last_clean":"2024-03-18T20:26:39.167507+0000","last_became_active":"2024-03-18T20:26:25.806650+0000","last_became_peered":"2024-03-18T20:26:25.806650+0000","last_unstale":"2024-03-18T20:26:39.167507+0000","last_undegraded":"2024-03-18T20:26:39.167507+0000","last_fullsized":"2024-03-18T20:26:39.167507+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:06:16.285934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.619586+0000","last_change":"2024-03-18T20:26:39.618254+0000","last_active":"2024-03-18T20:26:39.619586+0000","last_peered":"2024-03-18T20:26:39.619586+0000","last_clean":"2024-03-18T20:26:39.619586+0000","last_became_active":"2024-03-18T20:26:39.617947+0000","last_became_peered":"2024-03-18T20:26:39.617947+0000","last_unstale":"2024-03-18T20:26:39.619586+0000","last_undegraded":"2024-03-18T20:26:39.619586+0000","last_fullsized":"2024-03-18T20:26:39.619586+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:34:19.023316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.612360+0000","last_change":"2024-03-18T20:26:39.177881+0000","last_active":"2024-03-18T20:26:39.612360+0000","last_peered":"2024-03-18T20:26:39.612360+0000","last_clean":"2024-03-18T20:26:39.612360+0000","last_became_active":"2024-03-18T20:26:39.177666+0000","last_became_peered":"2024-03-18T20:26:39.177666+0000","last_unstale":"2024-03-18T20:26:39.612360+0000","last_undegraded":"2024-03-18T20:26:39.612360+0000","last_fullsized":"2024-03-18T20:26:39.612360+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:22.609793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174282+0000","last_change":"2024-03-18T20:26:25.815611+0000","last_active":"2024-03-18T20:26:39.174282+0000","last_peered":"2024-03-18T20:26:39.174282+0000","last_clean":"2024-03-18T20:26:39.174282+0000","last_became_active":"2024-03-18T20:26:25.815172+0000","last_became_peered":"2024-03-18T20:26:25.815172+0000","last_unstale":"2024-03-18T20:26:39.174282+0000","last_undegraded":"2024-03-18T20:26:39.174282+0000","last_fullsized":"2024-03-18T20:26:39.174282+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:48:15.078756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.633907+0000","last_change":"2024-03-18T20:26:39.631495+0000","last_active":"2024-03-18T20:26:39.633907+0000","last_peered":"2024-03-18T20:26:39.633907+0000","last_clean":"2024-03-18T20:26:39.633907+0000","last_became_active":"2024-03-18T20:26:39.629341+0000","last_became_peered":"2024-03-18T20:26:39.629341+0000","last_unstale":"2024-03-18T20:26:39.633907+0000","last_undegraded":"2024-03-18T20:26:39.633907+0000","last_fullsized":"2024-03-18T20:26:39.633907+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:47:27.910466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.606729+0000","last_change":"2024-03-18T20:26:39.183122+0000","last_active":"2024-03-18T20:26:39.606729+0000","last_peered":"2024-03-18T20:26:39.606729+0000","last_clean":"2024-03-18T20:26:39.606729+0000","last_became_active":"2024-03-18T20:26:39.182830+0000","last_became_peered":"2024-03-18T20:26:39.182830+0000","last_unstale":"2024-03-18T20:26:39.606729+0000","last_undegraded":"2024-03-18T20:26:39.606729+0000","last_fullsized":"2024-03-18T20:26:39.606729+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:32:45.875046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.185621+0000","last_change":"2024-03-18T20:26:25.810485+0000","last_active":"2024-03-18T20:26:39.185621+0000","last_peered":"2024-03-18T20:26:39.185621+0000","last_clean":"2024-03-18T20:26:39.185621+0000","last_became_active":"2024-03-18T20:26:25.810235+0000","last_became_peered":"2024-03-18T20:26:25.810235+0000","last_unstale":"2024-03-18T20:26:39.185621+0000","last_undegraded":"2024-03-18T20:26:39.185621+0000","last_fullsized":"2024-03-18T20:26:39.185621+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:59.140240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.634232+0000","last_change":"2024-03-18T20:26:39.633292+0000","last_active":"2024-03-18T20:26:39.634232+0000","last_peered":"2024-03-18T20:26:39.634232+0000","last_clean":"2024-03-18T20:26:39.634232+0000","last_became_active":"2024-03-18T20:26:39.633138+0000","last_became_peered":"2024-03-18T20:26:39.633138+0000","last_unstale":"2024-03-18T20:26:39.634232+0000","last_undegraded":"2024-03-18T20:26:39.634232+0000","last_fullsized":"2024-03-18T20:26:39.634232+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:30:32.988249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.607849+0000","last_change":"2024-03-18T20:26:39.182439+0000","last_active":"2024-03-18T20:26:39.607849+0000","last_peered":"2024-03-18T20:26:39.607849+0000","last_clean":"2024-03-18T20:26:39.607849+0000","last_became_active":"2024-03-18T20:26:39.181085+0000","last_became_peered":"2024-03-18T20:26:39.181085+0000","last_unstale":"2024-03-18T20:26:39.607849+0000","last_undegraded":"2024-03-18T20:26:39.607849+0000","last_fullsized":"2024-03-18T20:26:39.607849+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:18:52.564533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.185216+0000","last_change":"2024-03-18T20:26:25.807504+0000","last_active":"2024-03-18T20:26:39.185216+0000","last_peered":"2024-03-18T20:26:39.185216+0000","last_clean":"2024-03-18T20:26:39.185216+0000","last_became_active":"2024-03-18T20:26:25.807257+0000","last_became_peered":"2024-03-18T20:26:25.807257+0000","last_unstale":"2024-03-18T20:26:39.185216+0000","last_undegraded":"2024-03-18T20:26:39.185216+0000","last_fullsized":"2024-03-18T20:26:39.185216+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:40.456111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.618474+0000","last_change":"2024-03-18T20:26:39.617258+0000","last_active":"2024-03-18T20:26:39.618474+0000","last_peered":"2024-03-18T20:26:39.618474+0000","last_clean":"2024-03-18T20:26:39.618474+0000","last_became_active":"2024-03-18T20:26:39.616305+0000","last_became_peered":"2024-03-18T20:26:39.616305+0000","last_unstale":"2024-03-18T20:26:39.618474+0000","last_undegraded":"2024-03-18T20:26:39.618474+0000","last_fullsized":"2024-03-18T20:26:39.618474+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:22:26.538402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.606616+0000","last_change":"2024-03-18T20:26:39.178912+0000","last_active":"2024-03-18T20:26:39.606616+0000","last_peered":"2024-03-18T20:26:39.606616+0000","last_clean":"2024-03-18T20:26:39.606616+0000","last_became_active":"2024-03-18T20:26:39.178637+0000","last_became_peered":"2024-03-18T20:26:39.178637+0000","last_unstale":"2024-03-18T20:26:39.606616+0000","last_undegraded":"2024-03-18T20:26:39.606616+0000","last_fullsized":"2024-03-18T20:26:39.606616+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:56.206908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.169677+0000","last_change":"2024-03-18T20:26:25.796362+0000","last_active":"2024-03-18T20:26:39.169677+0000","last_peered":"2024-03-18T20:26:39.169677+0000","last_clean":"2024-03-18T20:26:39.169677+0000","last_became_active":"2024-03-18T20:26:25.796061+0000","last_became_peered":"2024-03-18T20:26:25.796061+0000","last_unstale":"2024-03-18T20:26:39.169677+0000","last_undegraded":"2024-03-18T20:26:39.169677+0000","last_fullsized":"2024-03-18T20:26:39.169677+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:28.602198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629507+0000","last_change":"2024-03-18T20:26:39.625934+0000","last_active":"2024-03-18T20:26:39.629507+0000","last_peered":"2024-03-18T20:26:39.629507+0000","last_clean":"2024-03-18T20:26:39.629507+0000","last_became_active":"2024-03-18T20:26:39.625480+0000","last_became_peered":"2024-03-18T20:26:39.625480+0000","last_unstale":"2024-03-18T20:26:39.629507+0000","last_undegraded":"2024-03-18T20:26:39.629507+0000","last_fullsized":"2024-03-18T20:26:39.629507+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:23:05.158863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":10,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.610367+0000","last_change":"2024-03-18T20:26:39.180090+0000","last_active":"2024-03-18T20:26:39.610367+0000","last_peered":"2024-03-18T20:26:39.610367+0000","last_clean":"2024-03-18T20:26:39.610367+0000","last_became_active":"2024-03-18T20:26:39.179545+0000","last_became_peered":"2024-03-18T20:26:39.179545+0000","last_unstale":"2024-03-18T20:26:39.610367+0000","last_undegraded":"2024-03-18T20:26:39.610367+0000","last_fullsized":"2024-03-18T20:26:39.610367+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:38.143523+0000","last_clean_scrub_stamp":"2024-03-18T20:26:38.143523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:22:43.454861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.175067+0000","last_change":"2024-03-18T20:26:25.815799+0000","last_active":"2024-03-18T20:26:39.175067+0000","last_peered":"2024-03-18T20:26:39.175067+0000","last_clean":"2024-03-18T20:26:39.175067+0000","last_became_active":"2024-03-18T20:26:25.815392+0000","last_became_peered":"2024-03-18T20:26:25.815392+0000","last_unstale":"2024-03-18T20:26:39.175067+0000","last_undegraded":"2024-03-18T20:26:39.175067+0000","last_fullsized":"2024-03-18T20:26:39.175067+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:21.319444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.626467+0000","last_change":"2024-03-18T20:26:39.623791+0000","last_active":"2024-03-18T20:26:39.626467+0000","last_peered":"2024-03-18T20:26:39.626467+0000","last_clean":"2024-03-18T20:26:39.626467+0000","last_became_active":"2024-03-18T20:26:39.623468+0000","last_became_peered":"2024-03-18T20:26:39.623468+0000","last_unstale":"2024-03-18T20:26:39.626467+0000","last_undegraded":"2024-03-18T20:26:39.626467+0000","last_fullsized":"2024-03-18T20:26:39.626467+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:44:03.811959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.172738+0000","last_change":"2024-03-18T20:26:25.808089+0000","last_active":"2024-03-18T20:26:39.172738+0000","last_peered":"2024-03-18T20:26:39.172738+0000","last_clean":"2024-03-18T20:26:39.172738+0000","last_became_active":"2024-03-18T20:26:25.806688+0000","last_became_peered":"2024-03-18T20:26:25.806688+0000","last_unstale":"2024-03-18T20:26:39.172738+0000","last_undegraded":"2024-03-18T20:26:39.172738+0000","last_fullsized":"2024-03-18T20:26:39.172738+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:22.598187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"252'4","reported_seq":18,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174329+0000","last_change":"2024-03-18T20:26:25.369142+0000","last_active":"2024-03-18T20:26:39.174329+0000","last_peered":"2024-03-18T20:26:39.174329+0000","last_clean":"2024-03-18T20:26:39.174329+0000","last_became_active":"2024-03-18T20:26:25.368879+0000","last_became_peered":"2024-03-18T20:26:25.368879+0000","last_unstale":"2024-03-18T20:26:39.174329+0000","last_undegraded":"2024-03-18T20:26:39.174329+0000","last_fullsized":"2024-03-18T20:26:39.174329+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:42.709929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.624550+0000","last_change":"2024-03-18T20:26:39.617628+0000","last_active":"2024-03-18T20:26:39.624550+0000","last_peered":"2024-03-18T20:26:39.624550+0000","last_clean":"2024-03-18T20:26:39.624550+0000","last_became_active":"2024-03-18T20:26:39.617022+0000","last_became_peered":"2024-03-18T20:26:39.617022+0000","last_unstale":"2024-03-18T20:26:39.624550+0000","last_undegraded":"2024-03-18T20:26:39.624550+0000","last_fullsized":"2024-03-18T20:26:39.624550+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:35.128300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164012+0000","last_change":"2024-03-18T20:26:25.812175+0000","last_active":"2024-03-18T20:26:39.164012+0000","last_peered":"2024-03-18T20:26:39.164012+0000","last_clean":"2024-03-18T20:26:39.164012+0000","last_became_active":"2024-03-18T20:26:25.811898+0000","last_became_peered":"2024-03-18T20:26:25.811898+0000","last_unstale":"2024-03-18T20:26:39.164012+0000","last_undegraded":"2024-03-18T20:26:39.164012+0000","last_fullsized":"2024-03-18T20:26:39.164012+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:45:00.418514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"252'2","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.173318+0000","last_change":"2024-03-18T20:26:25.371848+0000","last_active":"2024-03-18T20:26:39.173318+0000","last_peered":"2024-03-18T20:26:39.173318+0000","last_clean":"2024-03-18T20:26:39.173318+0000","last_became_active":"2024-03-18T20:26:25.371656+0000","last_became_peered":"2024-03-18T20:26:25.371656+0000","last_unstale":"2024-03-18T20:26:39.173318+0000","last_undegraded":"2024-03-18T20:26:39.173318+0000","last_fullsized":"2024-03-18T20:26:39.173318+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:27.592158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.633884+0000","last_change":"2024-03-18T20:26:39.630884+0000","last_active":"2024-03-18T20:26:39.633884+0000","last_peered":"2024-03-18T20:26:39.633884+0000","last_clean":"2024-03-18T20:26:39.633884+0000","last_became_active":"2024-03-18T20:26:39.630590+0000","last_became_peered":"2024-03-18T20:26:39.630590+0000","last_unstale":"2024-03-18T20:26:39.633884+0000","last_undegraded":"2024-03-18T20:26:39.633884+0000","last_fullsized":"2024-03-18T20:26:39.633884+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:30:36.759356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174628+0000","last_change":"2024-03-18T20:26:25.809815+0000","last_active":"2024-03-18T20:26:39.174628+0000","last_peered":"2024-03-18T20:26:39.174628+0000","last_clean":"2024-03-18T20:26:39.174628+0000","last_became_active":"2024-03-18T20:26:25.809504+0000","last_became_peered":"2024-03-18T20:26:25.809504+0000","last_unstale":"2024-03-18T20:26:39.174628+0000","last_undegraded":"2024-03-18T20:26:39.174628+0000","last_fullsized":"2024-03-18T20:26:39.174628+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:42:34.548735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"252'2","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.185729+0000","last_change":"2024-03-18T20:26:25.366016+0000","last_active":"2024-03-18T20:26:39.185729+0000","last_peered":"2024-03-18T20:26:39.185729+0000","last_clean":"2024-03-18T20:26:39.185729+0000","last_became_active":"2024-03-18T20:26:25.365694+0000","last_became_peered":"2024-03-18T20:26:25.365694+0000","last_unstale":"2024-03-18T20:26:39.185729+0000","last_undegraded":"2024-03-18T20:26:39.185729+0000","last_fullsized":"2024-03-18T20:26:39.185729+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:08.800093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629197+0000","last_change":"2024-03-18T20:26:39.621210+0000","last_active":"2024-03-18T20:26:39.629197+0000","last_peered":"2024-03-18T20:26:39.629197+0000","last_clean":"2024-03-18T20:26:39.629197+0000","last_became_active":"2024-03-18T20:26:39.620953+0000","last_became_peered":"2024-03-18T20:26:39.620953+0000","last_unstale":"2024-03-18T20:26:39.629197+0000","last_undegraded":"2024-03-18T20:26:39.629197+0000","last_fullsized":"2024-03-18T20:26:39.629197+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:33:57.528824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.185439+0000","last_change":"2024-03-18T20:26:25.802194+0000","last_active":"2024-03-18T20:26:39.185439+0000","last_peered":"2024-03-18T20:26:39.185439+0000","last_clean":"2024-03-18T20:26:39.185439+0000","last_became_active":"2024-03-18T20:26:25.801915+0000","last_became_peered":"2024-03-18T20:26:25.801915+0000","last_unstale":"2024-03-18T20:26:39.185439+0000","last_undegraded":"2024-03-18T20:26:39.185439+0000","last_fullsized":"2024-03-18T20:26:39.185439+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:52.955865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"252'1","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164167+0000","last_change":"2024-03-18T20:26:25.364855+0000","last_active":"2024-03-18T20:26:39.164167+0000","last_peered":"2024-03-18T20:26:39.164167+0000","last_clean":"2024-03-18T20:26:39.164167+0000","last_became_active":"2024-03-18T20:26:25.364563+0000","last_became_peered":"2024-03-18T20:26:25.364563+0000","last_unstale":"2024-03-18T20:26:39.164167+0000","last_undegraded":"2024-03-18T20:26:39.164167+0000","last_fullsized":"2024-03-18T20:26:39.164167+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:17.116325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.627216+0000","last_change":"2024-03-18T20:26:39.626171+0000","last_active":"2024-03-18T20:26:39.627216+0000","last_peered":"2024-03-18T20:26:39.627216+0000","last_clean":"2024-03-18T20:26:39.627216+0000","last_became_active":"2024-03-18T20:26:39.625918+0000","last_became_peered":"2024-03-18T20:26:39.625918+0000","last_unstale":"2024-03-18T20:26:39.627216+0000","last_undegraded":"2024-03-18T20:26:39.627216+0000","last_fullsized":"2024-03-18T20:26:39.627216+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:54:42.615870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174290+0000","last_change":"2024-03-18T20:26:25.799795+0000","last_active":"2024-03-18T20:26:39.174290+0000","last_peered":"2024-03-18T20:26:39.174290+0000","last_clean":"2024-03-18T20:26:39.174290+0000","last_became_active":"2024-03-18T20:26:25.799539+0000","last_became_peered":"2024-03-18T20:26:25.799539+0000","last_unstale":"2024-03-18T20:26:39.174290+0000","last_undegraded":"2024-03-18T20:26:39.174290+0000","last_fullsized":"2024-03-18T20:26:39.174290+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:49.613713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"252'1","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.185332+0000","last_change":"2024-03-18T20:26:25.367422+0000","last_active":"2024-03-18T20:26:39.185332+0000","last_peered":"2024-03-18T20:26:39.185332+0000","last_clean":"2024-03-18T20:26:39.185332+0000","last_became_active":"2024-03-18T20:26:25.367153+0000","last_became_peered":"2024-03-18T20:26:25.367153+0000","last_unstale":"2024-03-18T20:26:39.185332+0000","last_undegraded":"2024-03-18T20:26:39.185332+0000","last_fullsized":"2024-03-18T20:26:39.185332+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:14:17.781705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.624858+0000","last_change":"2024-03-18T20:26:39.619567+0000","last_active":"2024-03-18T20:26:39.624858+0000","last_peered":"2024-03-18T20:26:39.624858+0000","last_clean":"2024-03-18T20:26:39.624858+0000","last_became_active":"2024-03-18T20:26:39.618859+0000","last_became_peered":"2024-03-18T20:26:39.618859+0000","last_unstale":"2024-03-18T20:26:39.624858+0000","last_undegraded":"2024-03-18T20:26:39.624858+0000","last_fullsized":"2024-03-18T20:26:39.624858+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:22:26.538402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.165702+0000","last_change":"2024-03-18T20:26:25.812108+0000","last_active":"2024-03-18T20:26:39.165702+0000","last_peered":"2024-03-18T20:26:39.165702+0000","last_clean":"2024-03-18T20:26:39.165702+0000","last_became_active":"2024-03-18T20:26:25.811747+0000","last_became_peered":"2024-03-18T20:26:25.811747+0000","last_unstale":"2024-03-18T20:26:39.165702+0000","last_undegraded":"2024-03-18T20:26:39.165702+0000","last_fullsized":"2024-03-18T20:26:39.165702+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:40.456111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166399+0000","last_change":"2024-03-18T20:26:25.360666+0000","last_active":"2024-03-18T20:26:39.166399+0000","last_peered":"2024-03-18T20:26:39.166399+0000","last_clean":"2024-03-18T20:26:39.166399+0000","last_became_active":"2024-03-18T20:26:25.360400+0000","last_became_peered":"2024-03-18T20:26:25.360400+0000","last_unstale":"2024-03-18T20:26:39.166399+0000","last_undegraded":"2024-03-18T20:26:39.166399+0000","last_fullsized":"2024-03-18T20:26:39.166399+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:18:38.754833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.624983+0000","last_change":"2024-03-18T20:26:39.619868+0000","last_active":"2024-03-18T20:26:39.624983+0000","last_peered":"2024-03-18T20:26:39.624983+0000","last_clean":"2024-03-18T20:26:39.624983+0000","last_became_active":"2024-03-18T20:26:39.619113+0000","last_became_peered":"2024-03-18T20:26:39.619113+0000","last_unstale":"2024-03-18T20:26:39.624983+0000","last_undegraded":"2024-03-18T20:26:39.624983+0000","last_fullsized":"2024-03-18T20:26:39.624983+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:39:05.676337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.167958+0000","last_change":"2024-03-18T20:26:25.810099+0000","last_active":"2024-03-18T20:26:39.167958+0000","last_peered":"2024-03-18T20:26:39.167958+0000","last_clean":"2024-03-18T20:26:39.167958+0000","last_became_active":"2024-03-18T20:26:25.806786+0000","last_became_peered":"2024-03-18T20:26:25.806786+0000","last_unstale":"2024-03-18T20:26:39.167958+0000","last_undegraded":"2024-03-18T20:26:39.167958+0000","last_fullsized":"2024-03-18T20:26:39.167958+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:35.919143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.175261+0000","last_change":"2024-03-18T20:26:25.365778+0000","last_active":"2024-03-18T20:26:39.175261+0000","last_peered":"2024-03-18T20:26:39.175261+0000","last_clean":"2024-03-18T20:26:39.175261+0000","last_became_active":"2024-03-18T20:26:25.365505+0000","last_became_peered":"2024-03-18T20:26:25.365505+0000","last_unstale":"2024-03-18T20:26:39.175261+0000","last_undegraded":"2024-03-18T20:26:39.175261+0000","last_fullsized":"2024-03-18T20:26:39.175261+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:46:21.036246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.624849+0000","last_change":"2024-03-18T20:26:39.619374+0000","last_active":"2024-03-18T20:26:39.624849+0000","last_peered":"2024-03-18T20:26:39.624849+0000","last_clean":"2024-03-18T20:26:39.624849+0000","last_became_active":"2024-03-18T20:26:39.619098+0000","last_became_peered":"2024-03-18T20:26:39.619098+0000","last_unstale":"2024-03-18T20:26:39.624849+0000","last_undegraded":"2024-03-18T20:26:39.624849+0000","last_fullsized":"2024-03-18T20:26:39.624849+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:34:19.023316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.171441+0000","last_change":"2024-03-18T20:26:25.798444+0000","last_active":"2024-03-18T20:26:39.171441+0000","last_peered":"2024-03-18T20:26:39.171441+0000","last_clean":"2024-03-18T20:26:39.171441+0000","last_became_active":"2024-03-18T20:26:25.798186+0000","last_became_peered":"2024-03-18T20:26:25.798186+0000","last_unstale":"2024-03-18T20:26:39.171441+0000","last_undegraded":"2024-03-18T20:26:39.171441+0000","last_fullsized":"2024-03-18T20:26:39.171441+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:06:22.720260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.185459+0000","last_change":"2024-03-18T20:26:25.370707+0000","last_active":"2024-03-18T20:26:39.185459+0000","last_peered":"2024-03-18T20:26:39.185459+0000","last_clean":"2024-03-18T20:26:39.185459+0000","last_became_active":"2024-03-18T20:26:25.370415+0000","last_became_peered":"2024-03-18T20:26:25.370415+0000","last_unstale":"2024-03-18T20:26:39.185459+0000","last_undegraded":"2024-03-18T20:26:39.185459+0000","last_fullsized":"2024-03-18T20:26:39.185459+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:05.351419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.631982+0000","last_change":"2024-03-18T20:26:39.631096+0000","last_active":"2024-03-18T20:26:39.631982+0000","last_peered":"2024-03-18T20:26:39.631982+0000","last_clean":"2024-03-18T20:26:39.631982+0000","last_became_active":"2024-03-18T20:26:39.630814+0000","last_became_peered":"2024-03-18T20:26:39.630814+0000","last_unstale":"2024-03-18T20:26:39.631982+0000","last_undegraded":"2024-03-18T20:26:39.631982+0000","last_fullsized":"2024-03-18T20:26:39.631982+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:41:27.458748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.175384+0000","last_change":"2024-03-18T20:26:25.816190+0000","last_active":"2024-03-18T20:26:39.175384+0000","last_peered":"2024-03-18T20:26:39.175384+0000","last_clean":"2024-03-18T20:26:39.175384+0000","last_became_active":"2024-03-18T20:26:25.815931+0000","last_became_peered":"2024-03-18T20:26:25.815931+0000","last_unstale":"2024-03-18T20:26:39.175384+0000","last_undegraded":"2024-03-18T20:26:39.175384+0000","last_fullsized":"2024-03-18T20:26:39.175384+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:49.613713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.168724+0000","last_change":"2024-03-18T20:26:25.363775+0000","last_active":"2024-03-18T20:26:39.168724+0000","last_peered":"2024-03-18T20:26:39.168724+0000","last_clean":"2024-03-18T20:26:39.168724+0000","last_became_active":"2024-03-18T20:26:25.363389+0000","last_became_peered":"2024-03-18T20:26:25.363389+0000","last_unstale":"2024-03-18T20:26:39.168724+0000","last_undegraded":"2024-03-18T20:26:39.168724+0000","last_fullsized":"2024-03-18T20:26:39.168724+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:42.397208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.618838+0000","last_change":"2024-03-18T20:26:39.616272+0000","last_active":"2024-03-18T20:26:39.618838+0000","last_peered":"2024-03-18T20:26:39.618838+0000","last_clean":"2024-03-18T20:26:39.618838+0000","last_became_active":"2024-03-18T20:26:39.615990+0000","last_became_peered":"2024-03-18T20:26:39.615990+0000","last_unstale":"2024-03-18T20:26:39.618838+0000","last_undegraded":"2024-03-18T20:26:39.618838+0000","last_fullsized":"2024-03-18T20:26:39.618838+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:34.194250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166405+0000","last_change":"2024-03-18T20:26:25.812196+0000","last_active":"2024-03-18T20:26:39.166405+0000","last_peered":"2024-03-18T20:26:39.166405+0000","last_clean":"2024-03-18T20:26:39.166405+0000","last_became_active":"2024-03-18T20:26:25.811934+0000","last_became_peered":"2024-03-18T20:26:25.811934+0000","last_unstale":"2024-03-18T20:26:39.166405+0000","last_undegraded":"2024-03-18T20:26:39.166405+0000","last_fullsized":"2024-03-18T20:26:39.166405+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:22.017892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"252'3","reported_seq":17,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.167596+0000","last_change":"2024-03-18T20:26:25.366952+0000","last_active":"2024-03-18T20:26:39.167596+0000","last_peered":"2024-03-18T20:26:39.167596+0000","last_clean":"2024-03-18T20:26:39.167596+0000","last_became_active":"2024-03-18T20:26:25.366671+0000","last_became_peered":"2024-03-18T20:26:25.366671+0000","last_unstale":"2024-03-18T20:26:39.167596+0000","last_undegraded":"2024-03-18T20:26:39.167596+0000","last_fullsized":"2024-03-18T20:26:39.167596+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:13:21.871477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.630770+0000","last_change":"2024-03-18T20:26:39.621481+0000","last_active":"2024-03-18T20:26:39.630770+0000","last_peered":"2024-03-18T20:26:39.630770+0000","last_clean":"2024-03-18T20:26:39.630770+0000","last_became_active":"2024-03-18T20:26:39.621234+0000","last_became_peered":"2024-03-18T20:26:39.621234+0000","last_unstale":"2024-03-18T20:26:39.630770+0000","last_undegraded":"2024-03-18T20:26:39.630770+0000","last_fullsized":"2024-03-18T20:26:39.630770+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:46:35.855141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.173529+0000","last_change":"2024-03-18T20:26:25.802694+0000","last_active":"2024-03-18T20:26:39.173529+0000","last_peered":"2024-03-18T20:26:39.173529+0000","last_clean":"2024-03-18T20:26:39.173529+0000","last_became_active":"2024-03-18T20:26:25.802443+0000","last_became_peered":"2024-03-18T20:26:25.802443+0000","last_unstale":"2024-03-18T20:26:39.173529+0000","last_undegraded":"2024-03-18T20:26:39.173529+0000","last_fullsized":"2024-03-18T20:26:39.173529+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:54:38.189655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"253'8","reported_seq":22,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174500+0000","last_change":"2024-03-18T20:26:25.368198+0000","last_active":"2024-03-18T20:26:39.174500+0000","last_peered":"2024-03-18T20:26:39.174500+0000","last_clean":"2024-03-18T20:26:39.174500+0000","last_became_active":"2024-03-18T20:26:25.367762+0000","last_became_peered":"2024-03-18T20:26:25.367762+0000","last_unstale":"2024-03-18T20:26:39.174500+0000","last_undegraded":"2024-03-18T20:26:39.174500+0000","last_fullsized":"2024-03-18T20:26:39.174500+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:06.153535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2822,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.625655+0000","last_change":"2024-03-18T20:26:39.621524+0000","last_active":"2024-03-18T20:26:39.625655+0000","last_peered":"2024-03-18T20:26:39.625655+0000","last_clean":"2024-03-18T20:26:39.625655+0000","last_became_active":"2024-03-18T20:26:39.621268+0000","last_became_peered":"2024-03-18T20:26:39.621268+0000","last_unstale":"2024-03-18T20:26:39.625655+0000","last_undegraded":"2024-03-18T20:26:39.625655+0000","last_fullsized":"2024-03-18T20:26:39.625655+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:59:34.194250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.171112+0000","last_change":"2024-03-18T20:26:25.796336+0000","last_active":"2024-03-18T20:26:39.171112+0000","last_peered":"2024-03-18T20:26:39.171112+0000","last_clean":"2024-03-18T20:26:39.171112+0000","last_became_active":"2024-03-18T20:26:25.796002+0000","last_became_peered":"2024-03-18T20:26:25.796002+0000","last_unstale":"2024-03-18T20:26:39.171112+0000","last_undegraded":"2024-03-18T20:26:39.171112+0000","last_fullsized":"2024-03-18T20:26:39.171112+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:13:22.881517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"252'2","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164880+0000","last_change":"2024-03-18T20:26:25.366019+0000","last_active":"2024-03-18T20:26:39.164880+0000","last_peered":"2024-03-18T20:26:39.164880+0000","last_clean":"2024-03-18T20:26:39.164880+0000","last_became_active":"2024-03-18T20:26:25.364848+0000","last_became_peered":"2024-03-18T20:26:25.364848+0000","last_unstale":"2024-03-18T20:26:39.164880+0000","last_undegraded":"2024-03-18T20:26:39.164880+0000","last_fullsized":"2024-03-18T20:26:39.164880+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:06:15.275895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.623961+0000","last_change":"2024-03-18T20:26:39.623251+0000","last_active":"2024-03-18T20:26:39.623961+0000","last_peered":"2024-03-18T20:26:39.623961+0000","last_clean":"2024-03-18T20:26:39.623961+0000","last_became_active":"2024-03-18T20:26:39.623046+0000","last_became_peered":"2024-03-18T20:26:39.623046+0000","last_unstale":"2024-03-18T20:26:39.623961+0000","last_undegraded":"2024-03-18T20:26:39.623961+0000","last_fullsized":"2024-03-18T20:26:39.623961+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:14:12.949096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174382+0000","last_change":"2024-03-18T20:26:25.803741+0000","last_active":"2024-03-18T20:26:39.174382+0000","last_peered":"2024-03-18T20:26:39.174382+0000","last_clean":"2024-03-18T20:26:39.174382+0000","last_became_active":"2024-03-18T20:26:25.803470+0000","last_became_peered":"2024-03-18T20:26:25.803470+0000","last_unstale":"2024-03-18T20:26:39.174382+0000","last_undegraded":"2024-03-18T20:26:39.174382+0000","last_fullsized":"2024-03-18T20:26:39.174382+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:02:55.614091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166718+0000","last_change":"2024-03-18T20:26:25.366963+0000","last_active":"2024-03-18T20:26:39.166718+0000","last_peered":"2024-03-18T20:26:39.166718+0000","last_clean":"2024-03-18T20:26:39.166718+0000","last_became_active":"2024-03-18T20:26:25.366416+0000","last_became_peered":"2024-03-18T20:26:25.366416+0000","last_unstale":"2024-03-18T20:26:39.166718+0000","last_undegraded":"2024-03-18T20:26:39.166718+0000","last_fullsized":"2024-03-18T20:26:39.166718+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:57.921302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.627098+0000","last_change":"2024-03-18T20:26:39.625979+0000","last_active":"2024-03-18T20:26:39.627098+0000","last_peered":"2024-03-18T20:26:39.627098+0000","last_clean":"2024-03-18T20:26:39.627098+0000","last_became_active":"2024-03-18T20:26:39.623145+0000","last_became_peered":"2024-03-18T20:26:39.623145+0000","last_unstale":"2024-03-18T20:26:39.627098+0000","last_undegraded":"2024-03-18T20:26:39.627098+0000","last_fullsized":"2024-03-18T20:26:39.627098+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:54.035449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164420+0000","last_change":"2024-03-18T20:26:25.808144+0000","last_active":"2024-03-18T20:26:39.164420+0000","last_peered":"2024-03-18T20:26:39.164420+0000","last_clean":"2024-03-18T20:26:39.164420+0000","last_became_active":"2024-03-18T20:26:25.807838+0000","last_became_peered":"2024-03-18T20:26:25.807838+0000","last_unstale":"2024-03-18T20:26:39.164420+0000","last_undegraded":"2024-03-18T20:26:39.164420+0000","last_fullsized":"2024-03-18T20:26:39.164420+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:37:41.132114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166336+0000","last_change":"2024-03-18T20:26:25.366119+0000","last_active":"2024-03-18T20:26:39.166336+0000","last_peered":"2024-03-18T20:26:39.166336+0000","last_clean":"2024-03-18T20:26:39.166336+0000","last_became_active":"2024-03-18T20:26:25.365869+0000","last_became_peered":"2024-03-18T20:26:25.365869+0000","last_unstale":"2024-03-18T20:26:39.166336+0000","last_undegraded":"2024-03-18T20:26:39.166336+0000","last_fullsized":"2024-03-18T20:26:39.166336+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:01.890743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.617667+0000","last_change":"2024-03-18T20:26:39.613269+0000","last_active":"2024-03-18T20:26:39.617667+0000","last_peered":"2024-03-18T20:26:39.617667+0000","last_clean":"2024-03-18T20:26:39.617667+0000","last_became_active":"2024-03-18T20:26:39.612996+0000","last_became_peered":"2024-03-18T20:26:39.612996+0000","last_unstale":"2024-03-18T20:26:39.617667+0000","last_undegraded":"2024-03-18T20:26:39.617667+0000","last_fullsized":"2024-03-18T20:26:39.617667+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:54.264967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.167880+0000","last_change":"2024-03-18T20:26:25.802507+0000","last_active":"2024-03-18T20:26:39.167880+0000","last_peered":"2024-03-18T20:26:39.167880+0000","last_clean":"2024-03-18T20:26:39.167880+0000","last_became_active":"2024-03-18T20:26:25.801913+0000","last_became_peered":"2024-03-18T20:26:25.801913+0000","last_unstale":"2024-03-18T20:26:39.167880+0000","last_undegraded":"2024-03-18T20:26:39.167880+0000","last_fullsized":"2024-03-18T20:26:39.167880+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:44:11.977302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"252'8","reported_seq":22,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.172641+0000","last_change":"2024-03-18T20:26:25.365245+0000","last_active":"2024-03-18T20:26:39.172641+0000","last_peered":"2024-03-18T20:26:39.172641+0000","last_clean":"2024-03-18T20:26:39.172641+0000","last_became_active":"2024-03-18T20:26:25.364945+0000","last_became_peered":"2024-03-18T20:26:25.364945+0000","last_unstale":"2024-03-18T20:26:39.172641+0000","last_undegraded":"2024-03-18T20:26:39.172641+0000","last_fullsized":"2024-03-18T20:26:39.172641+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:19.739927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629488+0000","last_change":"2024-03-18T20:26:39.625776+0000","last_active":"2024-03-18T20:26:39.629488+0000","last_peered":"2024-03-18T20:26:39.629488+0000","last_clean":"2024-03-18T20:26:39.629488+0000","last_became_active":"2024-03-18T20:26:39.625004+0000","last_became_peered":"2024-03-18T20:26:39.625004+0000","last_unstale":"2024-03-18T20:26:39.629488+0000","last_undegraded":"2024-03-18T20:26:39.629488+0000","last_fullsized":"2024-03-18T20:26:39.629488+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:41:27.458748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.165685+0000","last_change":"2024-03-18T20:26:25.802286+0000","last_active":"2024-03-18T20:26:39.165685+0000","last_peered":"2024-03-18T20:26:39.165685+0000","last_clean":"2024-03-18T20:26:39.165685+0000","last_became_active":"2024-03-18T20:26:25.802005+0000","last_became_peered":"2024-03-18T20:26:25.802005+0000","last_unstale":"2024-03-18T20:26:39.165685+0000","last_undegraded":"2024-03-18T20:26:39.165685+0000","last_fullsized":"2024-03-18T20:26:39.165685+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:10:16.362251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"252'1","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174449+0000","last_change":"2024-03-18T20:26:25.368591+0000","last_active":"2024-03-18T20:26:39.174449+0000","last_peered":"2024-03-18T20:26:39.174449+0000","last_clean":"2024-03-18T20:26:39.174449+0000","last_became_active":"2024-03-18T20:26:25.367843+0000","last_became_peered":"2024-03-18T20:26:25.367843+0000","last_unstale":"2024-03-18T20:26:39.174449+0000","last_undegraded":"2024-03-18T20:26:39.174449+0000","last_fullsized":"2024-03-18T20:26:39.174449+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:59:40.867803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629601+0000","last_change":"2024-03-18T20:26:39.627613+0000","last_active":"2024-03-18T20:26:39.629601+0000","last_peered":"2024-03-18T20:26:39.629601+0000","last_clean":"2024-03-18T20:26:39.629601+0000","last_became_active":"2024-03-18T20:26:39.627030+0000","last_became_peered":"2024-03-18T20:26:39.627030+0000","last_unstale":"2024-03-18T20:26:39.629601+0000","last_undegraded":"2024-03-18T20:26:39.629601+0000","last_fullsized":"2024-03-18T20:26:39.629601+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:50.540181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166229+0000","last_change":"2024-03-18T20:26:25.810340+0000","last_active":"2024-03-18T20:26:39.166229+0000","last_peered":"2024-03-18T20:26:39.166229+0000","last_clean":"2024-03-18T20:26:39.166229+0000","last_became_active":"2024-03-18T20:26:25.809980+0000","last_became_peered":"2024-03-18T20:26:25.809980+0000","last_unstale":"2024-03-18T20:26:39.166229+0000","last_undegraded":"2024-03-18T20:26:39.166229+0000","last_fullsized":"2024-03-18T20:26:39.166229+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:02:55.614091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"252'2","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.173095+0000","last_change":"2024-03-18T20:26:25.366492+0000","last_active":"2024-03-18T20:26:39.173095+0000","last_peered":"2024-03-18T20:26:39.173095+0000","last_clean":"2024-03-18T20:26:39.173095+0000","last_became_active":"2024-03-18T20:26:25.366098+0000","last_became_peered":"2024-03-18T20:26:25.366098+0000","last_unstale":"2024-03-18T20:26:39.173095+0000","last_undegraded":"2024-03-18T20:26:39.173095+0000","last_fullsized":"2024-03-18T20:26:39.173095+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:53:26.604334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.617656+0000","last_change":"2024-03-18T20:26:39.614008+0000","last_active":"2024-03-18T20:26:39.617656+0000","last_peered":"2024-03-18T20:26:39.617656+0000","last_clean":"2024-03-18T20:26:39.617656+0000","last_became_active":"2024-03-18T20:26:39.613741+0000","last_became_peered":"2024-03-18T20:26:39.613741+0000","last_unstale":"2024-03-18T20:26:39.617656+0000","last_undegraded":"2024-03-18T20:26:39.617656+0000","last_fullsized":"2024-03-18T20:26:39.617656+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:53:06.764721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.169272+0000","last_change":"2024-03-18T20:26:25.807945+0000","last_active":"2024-03-18T20:26:39.169272+0000","last_peered":"2024-03-18T20:26:39.169272+0000","last_clean":"2024-03-18T20:26:39.169272+0000","last_became_active":"2024-03-18T20:26:25.807678+0000","last_became_peered":"2024-03-18T20:26:25.807678+0000","last_unstale":"2024-03-18T20:26:39.169272+0000","last_undegraded":"2024-03-18T20:26:39.169272+0000","last_fullsized":"2024-03-18T20:26:39.169272+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:06.759034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"252'1","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.173411+0000","last_change":"2024-03-18T20:26:25.371835+0000","last_active":"2024-03-18T20:26:39.173411+0000","last_peered":"2024-03-18T20:26:39.173411+0000","last_clean":"2024-03-18T20:26:39.173411+0000","last_became_active":"2024-03-18T20:26:25.371634+0000","last_became_peered":"2024-03-18T20:26:25.371634+0000","last_unstale":"2024-03-18T20:26:39.173411+0000","last_undegraded":"2024-03-18T20:26:39.173411+0000","last_fullsized":"2024-03-18T20:26:39.173411+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:43.471325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.620856+0000","last_change":"2024-03-18T20:26:39.617270+0000","last_active":"2024-03-18T20:26:39.620856+0000","last_peered":"2024-03-18T20:26:39.620856+0000","last_clean":"2024-03-18T20:26:39.620856+0000","last_became_active":"2024-03-18T20:26:39.616997+0000","last_became_peered":"2024-03-18T20:26:39.616997+0000","last_unstale":"2024-03-18T20:26:39.620856+0000","last_undegraded":"2024-03-18T20:26:39.620856+0000","last_fullsized":"2024-03-18T20:26:39.620856+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:20.972430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.167913+0000","last_change":"2024-03-18T20:26:25.812207+0000","last_active":"2024-03-18T20:26:39.167913+0000","last_peered":"2024-03-18T20:26:39.167913+0000","last_clean":"2024-03-18T20:26:39.167913+0000","last_became_active":"2024-03-18T20:26:25.811924+0000","last_became_peered":"2024-03-18T20:26:25.811924+0000","last_unstale":"2024-03-18T20:26:39.167913+0000","last_undegraded":"2024-03-18T20:26:39.167913+0000","last_fullsized":"2024-03-18T20:26:39.167913+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:01:09.966146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164342+0000","last_change":"2024-03-18T20:26:25.364241+0000","last_active":"2024-03-18T20:26:39.164342+0000","last_peered":"2024-03-18T20:26:39.164342+0000","last_clean":"2024-03-18T20:26:39.164342+0000","last_became_active":"2024-03-18T20:26:25.363945+0000","last_became_peered":"2024-03-18T20:26:25.363945+0000","last_unstale":"2024-03-18T20:26:39.164342+0000","last_undegraded":"2024-03-18T20:26:39.164342+0000","last_fullsized":"2024-03-18T20:26:39.164342+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:34.909104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.617802+0000","last_change":"2024-03-18T20:26:39.611228+0000","last_active":"2024-03-18T20:26:39.617802+0000","last_peered":"2024-03-18T20:26:39.617802+0000","last_clean":"2024-03-18T20:26:39.617802+0000","last_became_active":"2024-03-18T20:26:39.607052+0000","last_became_peered":"2024-03-18T20:26:39.607052+0000","last_unstale":"2024-03-18T20:26:39.617802+0000","last_undegraded":"2024-03-18T20:26:39.617802+0000","last_fullsized":"2024-03-18T20:26:39.617802+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:34:19.023316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166075+0000","last_change":"2024-03-18T20:26:25.799178+0000","last_active":"2024-03-18T20:26:39.166075+0000","last_peered":"2024-03-18T20:26:39.166075+0000","last_clean":"2024-03-18T20:26:39.166075+0000","last_became_active":"2024-03-18T20:26:25.798768+0000","last_became_peered":"2024-03-18T20:26:25.798768+0000","last_unstale":"2024-03-18T20:26:39.166075+0000","last_undegraded":"2024-03-18T20:26:39.166075+0000","last_fullsized":"2024-03-18T20:26:39.166075+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:22:12.729546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"252'2","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.168016+0000","last_change":"2024-03-18T20:26:25.370148+0000","last_active":"2024-03-18T20:26:39.168016+0000","last_peered":"2024-03-18T20:26:39.168016+0000","last_clean":"2024-03-18T20:26:39.168016+0000","last_became_active":"2024-03-18T20:26:25.369876+0000","last_became_peered":"2024-03-18T20:26:25.369876+0000","last_unstale":"2024-03-18T20:26:39.168016+0000","last_undegraded":"2024-03-18T20:26:39.168016+0000","last_fullsized":"2024-03-18T20:26:39.168016+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:57:48.151947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.624251+0000","last_change":"2024-03-18T20:26:39.622007+0000","last_active":"2024-03-18T20:26:39.624251+0000","last_peered":"2024-03-18T20:26:39.624251+0000","last_clean":"2024-03-18T20:26:39.624251+0000","last_became_active":"2024-03-18T20:26:39.621726+0000","last_became_peered":"2024-03-18T20:26:39.621726+0000","last_unstale":"2024-03-18T20:26:39.624251+0000","last_undegraded":"2024-03-18T20:26:39.624251+0000","last_fullsized":"2024-03-18T20:26:39.624251+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:52.226234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.177905+0000","last_change":"2024-03-18T20:26:25.806909+0000","last_active":"2024-03-18T20:26:39.177905+0000","last_peered":"2024-03-18T20:26:39.177905+0000","last_clean":"2024-03-18T20:26:39.177905+0000","last_became_active":"2024-03-18T20:26:25.806643+0000","last_became_peered":"2024-03-18T20:26:25.806643+0000","last_unstale":"2024-03-18T20:26:39.177905+0000","last_undegraded":"2024-03-18T20:26:39.177905+0000","last_fullsized":"2024-03-18T20:26:39.177905+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:30:14.320846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.172662+0000","last_change":"2024-03-18T20:26:25.370582+0000","last_active":"2024-03-18T20:26:39.172662+0000","last_peered":"2024-03-18T20:26:39.172662+0000","last_clean":"2024-03-18T20:26:39.172662+0000","last_became_active":"2024-03-18T20:26:25.370336+0000","last_became_peered":"2024-03-18T20:26:25.370336+0000","last_unstale":"2024-03-18T20:26:39.172662+0000","last_undegraded":"2024-03-18T20:26:39.172662+0000","last_fullsized":"2024-03-18T20:26:39.172662+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:23:03.661960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.627082+0000","last_change":"2024-03-18T20:26:39.625364+0000","last_active":"2024-03-18T20:26:39.627082+0000","last_peered":"2024-03-18T20:26:39.627082+0000","last_clean":"2024-03-18T20:26:39.627082+0000","last_became_active":"2024-03-18T20:26:39.624602+0000","last_became_peered":"2024-03-18T20:26:39.624602+0000","last_unstale":"2024-03-18T20:26:39.627082+0000","last_undegraded":"2024-03-18T20:26:39.627082+0000","last_fullsized":"2024-03-18T20:26:39.627082+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:01:23.775002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.171889+0000","last_change":"2024-03-18T20:26:25.814360+0000","last_active":"2024-03-18T20:26:39.171889+0000","last_peered":"2024-03-18T20:26:39.171889+0000","last_clean":"2024-03-18T20:26:39.171889+0000","last_became_active":"2024-03-18T20:26:25.812682+0000","last_became_peered":"2024-03-18T20:26:25.812682+0000","last_unstale":"2024-03-18T20:26:39.171889+0000","last_undegraded":"2024-03-18T20:26:39.171889+0000","last_fullsized":"2024-03-18T20:26:39.171889+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:22.598187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.175559+0000","last_change":"2024-03-18T20:26:25.369740+0000","last_active":"2024-03-18T20:26:39.175559+0000","last_peered":"2024-03-18T20:26:39.175559+0000","last_clean":"2024-03-18T20:26:39.175559+0000","last_became_active":"2024-03-18T20:26:25.369448+0000","last_became_peered":"2024-03-18T20:26:25.369448+0000","last_unstale":"2024-03-18T20:26:39.175559+0000","last_undegraded":"2024-03-18T20:26:39.175559+0000","last_fullsized":"2024-03-18T20:26:39.175559+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:39:25.003994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.626965+0000","last_change":"2024-03-18T20:26:39.621894+0000","last_active":"2024-03-18T20:26:39.626965+0000","last_peered":"2024-03-18T20:26:39.626965+0000","last_clean":"2024-03-18T20:26:39.626965+0000","last_became_active":"2024-03-18T20:26:39.621591+0000","last_became_peered":"2024-03-18T20:26:39.621591+0000","last_unstale":"2024-03-18T20:26:39.626965+0000","last_undegraded":"2024-03-18T20:26:39.626965+0000","last_fullsized":"2024-03-18T20:26:39.626965+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:35.406128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.175312+0000","last_change":"2024-03-18T20:26:25.807777+0000","last_active":"2024-03-18T20:26:39.175312+0000","last_peered":"2024-03-18T20:26:39.175312+0000","last_clean":"2024-03-18T20:26:39.175312+0000","last_became_active":"2024-03-18T20:26:25.807447+0000","last_became_peered":"2024-03-18T20:26:25.807447+0000","last_unstale":"2024-03-18T20:26:39.175312+0000","last_undegraded":"2024-03-18T20:26:39.175312+0000","last_fullsized":"2024-03-18T20:26:39.175312+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:22:55.933766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.184321+0000","last_change":"2024-03-18T20:26:25.366647+0000","last_active":"2024-03-18T20:26:39.184321+0000","last_peered":"2024-03-18T20:26:39.184321+0000","last_clean":"2024-03-18T20:26:39.184321+0000","last_became_active":"2024-03-18T20:26:25.366363+0000","last_became_peered":"2024-03-18T20:26:25.366363+0000","last_unstale":"2024-03-18T20:26:39.184321+0000","last_undegraded":"2024-03-18T20:26:39.184321+0000","last_fullsized":"2024-03-18T20:26:39.184321+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:22:29.645161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.623661+0000","last_change":"2024-03-18T20:26:39.620184+0000","last_active":"2024-03-18T20:26:39.623661+0000","last_peered":"2024-03-18T20:26:39.623661+0000","last_clean":"2024-03-18T20:26:39.623661+0000","last_became_active":"2024-03-18T20:26:39.615832+0000","last_became_peered":"2024-03-18T20:26:39.615832+0000","last_unstale":"2024-03-18T20:26:39.623661+0000","last_undegraded":"2024-03-18T20:26:39.623661+0000","last_fullsized":"2024-03-18T20:26:39.623661+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:04:46.303946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.171363+0000","last_change":"2024-03-18T20:26:25.807101+0000","last_active":"2024-03-18T20:26:39.171363+0000","last_peered":"2024-03-18T20:26:39.171363+0000","last_clean":"2024-03-18T20:26:39.171363+0000","last_became_active":"2024-03-18T20:26:25.806372+0000","last_became_peered":"2024-03-18T20:26:25.806372+0000","last_unstale":"2024-03-18T20:26:39.171363+0000","last_undegraded":"2024-03-18T20:26:39.171363+0000","last_fullsized":"2024-03-18T20:26:39.171363+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:18.126364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"252'2","reported_seq":16,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.168718+0000","last_change":"2024-03-18T20:26:25.365000+0000","last_active":"2024-03-18T20:26:39.168718+0000","last_peered":"2024-03-18T20:26:39.168718+0000","last_clean":"2024-03-18T20:26:39.168718+0000","last_became_active":"2024-03-18T20:26:25.364727+0000","last_became_peered":"2024-03-18T20:26:25.364727+0000","last_unstale":"2024-03-18T20:26:39.168718+0000","last_undegraded":"2024-03-18T20:26:39.168718+0000","last_fullsized":"2024-03-18T20:26:39.168718+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:18.741871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.631964+0000","last_change":"2024-03-18T20:26:39.629096+0000","last_active":"2024-03-18T20:26:39.631964+0000","last_peered":"2024-03-18T20:26:39.631964+0000","last_clean":"2024-03-18T20:26:39.631964+0000","last_became_active":"2024-03-18T20:26:39.628843+0000","last_became_peered":"2024-03-18T20:26:39.628843+0000","last_unstale":"2024-03-18T20:26:39.631964+0000","last_undegraded":"2024-03-18T20:26:39.631964+0000","last_fullsized":"2024-03-18T20:26:39.631964+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:23:05.158863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164206+0000","last_change":"2024-03-18T20:26:25.803062+0000","last_active":"2024-03-18T20:26:39.164206+0000","last_peered":"2024-03-18T20:26:39.164206+0000","last_clean":"2024-03-18T20:26:39.164206+0000","last_became_active":"2024-03-18T20:26:25.802825+0000","last_became_peered":"2024-03-18T20:26:25.802825+0000","last_unstale":"2024-03-18T20:26:39.164206+0000","last_undegraded":"2024-03-18T20:26:39.164206+0000","last_fullsized":"2024-03-18T20:26:39.164206+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:31:54.405889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"253'1","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166577+0000","last_change":"2024-03-18T20:26:25.367649+0000","last_active":"2024-03-18T20:26:39.166577+0000","last_peered":"2024-03-18T20:26:39.166577+0000","last_clean":"2024-03-18T20:26:39.166577+0000","last_became_active":"2024-03-18T20:26:25.367351+0000","last_became_peered":"2024-03-18T20:26:25.367351+0000","last_unstale":"2024-03-18T20:26:39.166577+0000","last_undegraded":"2024-03-18T20:26:39.166577+0000","last_fullsized":"2024-03-18T20:26:39.166577+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:19:05.047442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629826+0000","last_change":"2024-03-18T20:26:39.627878+0000","last_active":"2024-03-18T20:26:39.629826+0000","last_peered":"2024-03-18T20:26:39.629826+0000","last_clean":"2024-03-18T20:26:39.629826+0000","last_became_active":"2024-03-18T20:26:39.627630+0000","last_became_peered":"2024-03-18T20:26:39.627630+0000","last_unstale":"2024-03-18T20:26:39.629826+0000","last_undegraded":"2024-03-18T20:26:39.629826+0000","last_fullsized":"2024-03-18T20:26:39.629826+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:01:32.688385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166320+0000","last_change":"2024-03-18T20:26:25.812701+0000","last_active":"2024-03-18T20:26:39.166320+0000","last_peered":"2024-03-18T20:26:39.166320+0000","last_clean":"2024-03-18T20:26:39.166320+0000","last_became_active":"2024-03-18T20:26:25.812469+0000","last_became_peered":"2024-03-18T20:26:25.812469+0000","last_unstale":"2024-03-18T20:26:39.166320+0000","last_undegraded":"2024-03-18T20:26:39.166320+0000","last_fullsized":"2024-03-18T20:26:39.166320+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:59.140240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.185442+0000","last_change":"2024-03-18T20:26:25.370628+0000","last_active":"2024-03-18T20:26:39.185442+0000","last_peered":"2024-03-18T20:26:39.185442+0000","last_clean":"2024-03-18T20:26:39.185442+0000","last_became_active":"2024-03-18T20:26:25.370332+0000","last_became_peered":"2024-03-18T20:26:25.370332+0000","last_unstale":"2024-03-18T20:26:39.185442+0000","last_undegraded":"2024-03-18T20:26:39.185442+0000","last_fullsized":"2024-03-18T20:26:39.185442+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:43.011701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.619682+0000","last_change":"2024-03-18T20:26:39.618919+0000","last_active":"2024-03-18T20:26:39.619682+0000","last_peered":"2024-03-18T20:26:39.619682+0000","last_clean":"2024-03-18T20:26:39.619682+0000","last_became_active":"2024-03-18T20:26:39.618656+0000","last_became_peered":"2024-03-18T20:26:39.618656+0000","last_unstale":"2024-03-18T20:26:39.619682+0000","last_undegraded":"2024-03-18T20:26:39.619682+0000","last_fullsized":"2024-03-18T20:26:39.619682+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:22:26.538402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166675+0000","last_change":"2024-03-18T20:26:25.802895+0000","last_active":"2024-03-18T20:26:39.166675+0000","last_peered":"2024-03-18T20:26:39.166675+0000","last_clean":"2024-03-18T20:26:39.166675+0000","last_became_active":"2024-03-18T20:26:25.802655+0000","last_became_peered":"2024-03-18T20:26:25.802655+0000","last_unstale":"2024-03-18T20:26:39.166675+0000","last_undegraded":"2024-03-18T20:26:39.166675+0000","last_fullsized":"2024-03-18T20:26:39.166675+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:18.126364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.185827+0000","last_change":"2024-03-18T20:26:25.368715+0000","last_active":"2024-03-18T20:26:39.185827+0000","last_peered":"2024-03-18T20:26:39.185827+0000","last_clean":"2024-03-18T20:26:39.185827+0000","last_became_active":"2024-03-18T20:26:25.368383+0000","last_became_peered":"2024-03-18T20:26:25.368383+0000","last_unstale":"2024-03-18T20:26:39.185827+0000","last_undegraded":"2024-03-18T20:26:39.185827+0000","last_fullsized":"2024-03-18T20:26:39.185827+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:07.975001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629394+0000","last_change":"2024-03-18T20:26:39.628048+0000","last_active":"2024-03-18T20:26:39.629394+0000","last_peered":"2024-03-18T20:26:39.629394+0000","last_clean":"2024-03-18T20:26:39.629394+0000","last_became_active":"2024-03-18T20:26:39.627774+0000","last_became_peered":"2024-03-18T20:26:39.627774+0000","last_unstale":"2024-03-18T20:26:39.629394+0000","last_undegraded":"2024-03-18T20:26:39.629394+0000","last_fullsized":"2024-03-18T20:26:39.629394+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:48:28.887612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.185622+0000","last_change":"2024-03-18T20:26:25.805198+0000","last_active":"2024-03-18T20:26:39.185622+0000","last_peered":"2024-03-18T20:26:39.185622+0000","last_clean":"2024-03-18T20:26:39.185622+0000","last_became_active":"2024-03-18T20:26:25.804942+0000","last_became_peered":"2024-03-18T20:26:25.804942+0000","last_unstale":"2024-03-18T20:26:39.185622+0000","last_undegraded":"2024-03-18T20:26:39.185622+0000","last_fullsized":"2024-03-18T20:26:39.185622+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:22:12.729546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174310+0000","last_change":"2024-03-18T20:26:25.370584+0000","last_active":"2024-03-18T20:26:39.174310+0000","last_peered":"2024-03-18T20:26:39.174310+0000","last_clean":"2024-03-18T20:26:39.174310+0000","last_became_active":"2024-03-18T20:26:25.370353+0000","last_became_peered":"2024-03-18T20:26:25.370353+0000","last_unstale":"2024-03-18T20:26:39.174310+0000","last_undegraded":"2024-03-18T20:26:39.174310+0000","last_fullsized":"2024-03-18T20:26:39.174310+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:33:14.925680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3f","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.626454+0000","last_change":"2024-03-18T20:26:39.621607+0000","last_active":"2024-03-18T20:26:39.626454+0000","last_peered":"2024-03-18T20:26:39.626454+0000","last_clean":"2024-03-18T20:26:39.626454+0000","last_became_active":"2024-03-18T20:26:39.620969+0000","last_became_peered":"2024-03-18T20:26:39.620969+0000","last_unstale":"2024-03-18T20:26:39.626454+0000","last_undegraded":"2024-03-18T20:26:39.626454+0000","last_fullsized":"2024-03-18T20:26:39.626454+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:54:42.615870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.174008+0000","last_change":"2024-03-18T20:26:25.801336+0000","last_active":"2024-03-18T20:26:39.174008+0000","last_peered":"2024-03-18T20:26:39.174008+0000","last_clean":"2024-03-18T20:26:39.174008+0000","last_became_active":"2024-03-18T20:26:25.801053+0000","last_became_peered":"2024-03-18T20:26:25.801053+0000","last_unstale":"2024-03-18T20:26:39.174008+0000","last_undegraded":"2024-03-18T20:26:39.174008+0000","last_fullsized":"2024-03-18T20:26:39.174008+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:23.742824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166803+0000","last_change":"2024-03-18T20:26:25.367183+0000","last_active":"2024-03-18T20:26:39.166803+0000","last_peered":"2024-03-18T20:26:39.166803+0000","last_clean":"2024-03-18T20:26:39.166803+0000","last_became_active":"2024-03-18T20:26:25.366701+0000","last_became_peered":"2024-03-18T20:26:25.366701+0000","last_unstale":"2024-03-18T20:26:39.166803+0000","last_undegraded":"2024-03-18T20:26:39.166803+0000","last_fullsized":"2024-03-18T20:26:39.166803+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:05.748994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.630082+0000","last_change":"2024-03-18T20:26:39.628068+0000","last_active":"2024-03-18T20:26:39.630082+0000","last_peered":"2024-03-18T20:26:39.630082+0000","last_clean":"2024-03-18T20:26:39.630082+0000","last_became_active":"2024-03-18T20:26:39.627774+0000","last_became_peered":"2024-03-18T20:26:39.627774+0000","last_unstale":"2024-03-18T20:26:39.630082+0000","last_undegraded":"2024-03-18T20:26:39.630082+0000","last_fullsized":"2024-03-18T20:26:39.630082+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:10:22.388409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.173328+0000","last_change":"2024-03-18T20:26:25.806432+0000","last_active":"2024-03-18T20:26:39.173328+0000","last_peered":"2024-03-18T20:26:39.173328+0000","last_clean":"2024-03-18T20:26:39.173328+0000","last_became_active":"2024-03-18T20:26:25.806050+0000","last_became_peered":"2024-03-18T20:26:25.806050+0000","last_unstale":"2024-03-18T20:26:39.173328+0000","last_undegraded":"2024-03-18T20:26:39.173328+0000","last_fullsized":"2024-03-18T20:26:39.173328+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:32.033370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.166209+0000","last_change":"2024-03-18T20:26:25.364069+0000","last_active":"2024-03-18T20:26:39.166209+0000","last_peered":"2024-03-18T20:26:39.166209+0000","last_clean":"2024-03-18T20:26:39.166209+0000","last_became_active":"2024-03-18T20:26:25.363764+0000","last_became_peered":"2024-03-18T20:26:25.363764+0000","last_unstale":"2024-03-18T20:26:39.166209+0000","last_undegraded":"2024-03-18T20:26:39.166209+0000","last_fullsized":"2024-03-18T20:26:39.166209+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:01.862380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.630151+0000","last_change":"2024-03-18T20:26:39.629816+0000","last_active":"2024-03-18T20:26:39.630151+0000","last_peered":"2024-03-18T20:26:39.630151+0000","last_clean":"2024-03-18T20:26:39.630151+0000","last_became_active":"2024-03-18T20:26:39.629663+0000","last_became_peered":"2024-03-18T20:26:39.629663+0000","last_unstale":"2024-03-18T20:26:39.630151+0000","last_undegraded":"2024-03-18T20:26:39.630151+0000","last_fullsized":"2024-03-18T20:26:39.630151+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:54.264967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.175402+0000","last_change":"2024-03-18T20:26:25.817787+0000","last_active":"2024-03-18T20:26:39.175402+0000","last_peered":"2024-03-18T20:26:39.175402+0000","last_clean":"2024-03-18T20:26:39.175402+0000","last_became_active":"2024-03-18T20:26:25.817533+0000","last_became_peered":"2024-03-18T20:26:25.817533+0000","last_unstale":"2024-03-18T20:26:39.175402+0000","last_undegraded":"2024-03-18T20:26:39.175402+0000","last_fullsized":"2024-03-18T20:26:39.175402+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:22:04.370714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":14,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.171538+0000","last_change":"2024-03-18T20:26:25.368090+0000","last_active":"2024-03-18T20:26:39.171538+0000","last_peered":"2024-03-18T20:26:39.171538+0000","last_clean":"2024-03-18T20:26:39.171538+0000","last_became_active":"2024-03-18T20:26:25.367837+0000","last_became_peered":"2024-03-18T20:26:25.367837+0000","last_unstale":"2024-03-18T20:26:39.171538+0000","last_undegraded":"2024-03-18T20:26:39.171538+0000","last_fullsized":"2024-03-18T20:26:39.171538+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:19.739927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":9,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.629583+0000","last_change":"2024-03-18T20:26:39.627267+0000","last_active":"2024-03-18T20:26:39.629583+0000","last_peered":"2024-03-18T20:26:39.629583+0000","last_clean":"2024-03-18T20:26:39.629583+0000","last_became_active":"2024-03-18T20:26:39.626469+0000","last_became_peered":"2024-03-18T20:26:39.626469+0000","last_unstale":"2024-03-18T20:26:39.629583+0000","last_undegraded":"2024-03-18T20:26:39.629583+0000","last_fullsized":"2024-03-18T20:26:39.629583+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":255,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:39.152718+0000","last_clean_scrub_stamp":"2024-03-18T20:26:39.152718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:00:52.687797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":13,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.169093+0000","last_change":"2024-03-18T20:26:25.811045+0000","last_active":"2024-03-18T20:26:39.169093+0000","last_peered":"2024-03-18T20:26:39.169093+0000","last_clean":"2024-03-18T20:26:39.169093+0000","last_became_active":"2024-03-18T20:26:25.810754+0000","last_became_peered":"2024-03-18T20:26:25.810754+0000","last_unstale":"2024-03-18T20:26:39.169093+0000","last_undegraded":"2024-03-18T20:26:39.169093+0000","last_fullsized":"2024-03-18T20:26:39.169093+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:25.343862+0000","last_clean_scrub_stamp":"2024-03-18T20:26:25.343862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:44.931187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"252'1","reported_seq":15,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.175487+0000","last_change":"2024-03-18T20:26:25.367689+0000","last_active":"2024-03-18T20:26:39.175487+0000","last_peered":"2024-03-18T20:26:39.175487+0000","last_clean":"2024-03-18T20:26:39.175487+0000","last_became_active":"2024-03-18T20:26:25.367431+0000","last_became_peered":"2024-03-18T20:26:25.367431+0000","last_unstale":"2024-03-18T20:26:39.175487+0000","last_undegraded":"2024-03-18T20:26:39.175487+0000","last_fullsized":"2024-03-18T20:26:39.175487+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:26:24.333823+0000","last_clean_scrub_stamp":"2024-03-18T20:26:24.333823+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:39:01.890743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"238'370","reported_seq":1300,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-03-18T20:26:39.164037+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:26:39.164037+0000","last_peered":"2024-03-18T20:26:39.164037+0000","last_clean":"2024-03-18T20:26:39.164037+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:26:39.164037+0000","last_undegraded":"2024-03-18T20:26:39.164037+0000","last_fullsized":"2024-03-18T20:26:39.164037+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":3620,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575994,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52888,"kb_used_data":3724,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318952,"statfs":{"total":96636764160,"available":96582606848,"internally_reserved":0,"allocated":3813376,"data_stored":2248428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.57299999999999995,"15min":0.59499999999999997},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":1.1339999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.58299999999999996,"15min":0.60899999999999999},"min":{"1min":0.35299999999999998,"5min":0.33500000000000002,"15min":0.33300000000000002},"max":{"1min":0.76800000000000002,"5min":0.84499999999999997,"15min":1.093},"last":0.53000000000000003}]},{"osd":1,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.58199999999999996,"15min":0.60799999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72199999999999998,"5min":0.83099999999999996,"15min":1.4299999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.58499999999999996,"15min":0.61399999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":1.2629999999999999},"last":0.496}]},{"osd":2,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.58799999999999997,"15min":0.60899999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":1.216},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.58999999999999997,"15min":0.621},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.2490000000000001},"last":0.48099999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.57199999999999995,"15min":0.60299999999999998},"min":{"1min":0.374,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83899999999999997,"5min":0.84699999999999998,"15min":1.4450000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.623,"15min":0.64600000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.38600000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":1.5029999999999999},"last":0.55100000000000005}]},{"osd":4,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46600000000000003,"15min":0.47699999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.76500000000000001,"5min":0.79100000000000004,"15min":0.86099999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.48899999999999999,"15min":0.5},"min":{"1min":0.34300000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.50700000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.502,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52500000000000002,"15min":0.54600000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.877,"5min":0.877,"15min":1.042},"last":0.58499999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55300000000000005,"15min":0.57699999999999996},"min":{"1min":0.39500000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":1.129},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54800000000000004,"15min":0.56799999999999995},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":1.0609999999999999},"last":0.54300000000000004}]}]},{"osd":6,"up_from":13,"seq":55834575994,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60504,"kb_used_data":3144,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311336,"statfs":{"total":96636764160,"available":96574808064,"internally_reserved":0,"allocated":3219456,"data_stored":1658048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55400000000000005,"15min":0.59399999999999997},"min":{"1min":0.32500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.623,"5min":0.80900000000000005,"15min":1.1519999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57399999999999995,"15min":0.60399999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.52600000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56499999999999995,"15min":0.59699999999999998},"min":{"1min":0.40400000000000003,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69899999999999995,"5min":0.79800000000000004,"15min":1.244},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57999999999999996,"15min":0.60299999999999998},"min":{"1min":0.42499999999999999,"5min":0.35399999999999998,"15min":0.33300000000000002},"max":{"1min":0.83699999999999997,"5min":0.89200000000000002,"15min":1.286},"last":0.54000000000000004}]},{"osd":2,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.58699999999999997,"15min":0.623},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.69199999999999995,"5min":0.92000000000000004,"15min":1.276},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57999999999999996,"15min":0.60899999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.378},"max":{"1min":1.089,"5min":1.089,"15min":1.2669999999999999},"last":0.439}]},{"osd":3,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58099999999999996,"15min":0.621},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.59199999999999997,"15min":0.61599999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70599999999999996,"5min":1.0049999999999999,"15min":1.337},"last":0.498}]},{"osd":4,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.48899999999999999,"15min":0.48799999999999999},"min":{"1min":0.35799999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.75600000000000001,"5min":0.85799999999999998,"15min":0.88300000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.50800000000000001,"15min":0.51700000000000002},"min":{"1min":0.35899999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80300000000000005,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.48699999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.51100000000000001,"15min":0.51000000000000001},"min":{"1min":0.38300000000000001,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39600000000000002,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.874,"5min":0.92600000000000005,"15min":0.94199999999999995},"last":0.625}]},{"osd":7,"last update":"Mon Mar 18 20:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.42399999999999999,"5min":0.38,"15min":0.34499999999999997},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.57799999999999996,"15min":0.57499999999999996},"min":{"1min":0.41399999999999998,"5min":0.36599999999999999,"15min":0.34599999999999997},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.58899999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575994,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52288,"kb_used_data":3124,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319552,"statfs":{"total":96636764160,"available":96583221248,"internally_reserved":0,"allocated":3198976,"data_stored":1650380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.56299999999999994,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.38,"5min":0.41199999999999998,"15min":0.42199999999999999},"min":{"1min":0.27200000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56499999999999995,"5min":0.67500000000000004,"15min":0.69699999999999995},"last":0.34899999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.43099999999999999,"15min":0.432},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.51500000000000001,"5min":0.64300000000000002,"15min":0.68100000000000005},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.34499999999999997,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.55500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.41399999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.33300000000000002,"5min":0.33000000000000002,"15min":0.32300000000000001},"max":{"1min":0.58699999999999997,"5min":0.75900000000000001,"15min":0.76100000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.44,"15min":0.436},"min":{"1min":0.29199999999999998,"5min":0.28899999999999998,"15min":0.27000000000000002},"max":{"1min":0.55400000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.39000000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.66500000000000004,"15min":0.66800000000000004},"min":{"1min":0.42999999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82199999999999995,"5min":1.0680000000000001,"15min":1.2909999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.66900000000000004,"15min":0.67500000000000004},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.746},"last":0.57999999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.46300000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.88600000000000001,"5min":1.0389999999999999,"15min":1.6910000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.70199999999999996,"15min":0.70799999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.91100000000000003,"5min":1.109,"15min":1.2549999999999999},"last":0.69499999999999995}]},{"osd":6,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.68200000000000005,"15min":0.68100000000000005},"min":{"1min":0.46800000000000003,"5min":0.45900000000000002,"15min":0.437},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.637},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.70299999999999996,"15min":0.70499999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92100000000000004,"5min":1.014,"15min":1.722},"last":0.65300000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.68999999999999995,"15min":0.70399999999999996},"min":{"1min":0.51400000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.90300000000000002,"5min":1.02,"15min":1.2729999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.56599999999999995,"5min":0.495,"15min":0.42799999999999999},"max":{"1min":0.84199999999999997,"5min":1.0329999999999999,"15min":1.3380000000000001},"last":0.63900000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575995,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48820,"kb_used_data":3744,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323020,"statfs":{"total":96636764160,"available":96586772480,"internally_reserved":0,"allocated":3833856,"data_stored":2259261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38300000000000001,"15min":0.39000000000000001},"min":{"1min":0.29299999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56399999999999995,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.41699999999999998,"15min":0.42599999999999999},"min":{"1min":0.28199999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.60099999999999998,"5min":0.64000000000000001,"15min":1.1850000000000001},"last":0.25}]},{"osd":2,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42899999999999999,"15min":0.437},"min":{"1min":0.28399999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67900000000000005,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.432,"15min":0.45000000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.70999999999999996},"last":0.33600000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.42099999999999999,"15min":0.42899999999999999},"min":{"1min":0.29499999999999998,"5min":0.27500000000000002,"15min":0.26100000000000001},"max":{"1min":0.58499999999999996,"5min":0.60099999999999998,"15min":0.628},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.48099999999999998,"15min":0.49299999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.32500000000000001},"max":{"1min":0.73399999999999999,"5min":0.73999999999999999,"15min":0.91500000000000004},"last":0.30299999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63200000000000001,"15min":0.65200000000000002},"min":{"1min":0.40000000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90200000000000002,"5min":0.91500000000000004,"15min":1.0029999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.69899999999999995}]},{"osd":5,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.66700000000000004,"15min":0.67000000000000004},"min":{"1min":0.435,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.88500000000000001,"5min":0.97099999999999997,"15min":1.0129999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.626,"5min":0.67400000000000004,"15min":0.68000000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":1.518},"last":0.68799999999999994}]},{"osd":6,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.66100000000000003,"15min":0.66300000000000003},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.1279999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.79800000000000004}]},{"osd":7,"last update":"Mon Mar 18 20:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.69199999999999995,"15min":0.70499999999999996},"min":{"1min":0.47999999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.464},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.69099999999999995,"15min":0.70299999999999996},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.47599999999999998},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.60299999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575995,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52296,"kb_used_data":3132,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319544,"statfs":{"total":96636764160,"available":96583213056,"internally_reserved":0,"allocated":3207168,"data_stored":1651032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.34699999999999998,"15min":0.371},"min":{"1min":0.20799999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.53800000000000003,"5min":0.54800000000000004,"15min":0.57499999999999996},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.44700000000000001,"15min":0.47799999999999998},"min":{"1min":0.26100000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60499999999999998,"5min":0.71499999999999997,"15min":0.753},"last":0.35599999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.44400000000000001,"15min":0.47399999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57399999999999995,"5min":0.64300000000000002,"15min":0.749},"last":0.377},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.38900000000000001,"15min":0.40899999999999997},"min":{"1min":0.246,"5min":0.224,"15min":0.224},"max":{"1min":0.501,"5min":0.55900000000000005,"15min":0.59999999999999998},"last":0.311}]},{"osd":3,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.41199999999999998,"15min":0.42899999999999999},"min":{"1min":0.27800000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.56100000000000005,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.47399999999999998,"15min":0.50700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.63200000000000001,"5min":0.68100000000000005,"15min":0.752},"last":0.32900000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63900000000000001,"15min":0.65300000000000002},"min":{"1min":0.41799999999999998,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":1.2090000000000001,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.65700000000000003,"15min":0.66200000000000003},"min":{"1min":0.45100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.268,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.64800000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.64900000000000002,"15min":0.67500000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79900000000000004,"5min":0.97299999999999998,"15min":1.028},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.63900000000000001,"15min":0.67000000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.77100000000000002,"5min":0.88300000000000001,"15min":1.0629999999999999},"last":0.47899999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.65500000000000003,"15min":0.68100000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.81899999999999995,"5min":1.014,"15min":1.2050000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.72699999999999998},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.3220000000000001,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.73899999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.65500000000000003,"15min":0.68899999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.42199999999999999},"max":{"1min":0.73699999999999999,"5min":0.99399999999999999,"15min":1.0269999999999999},"last":0.504},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68600000000000005,"15min":0.70099999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.2330000000000001,"5min":1.371,"15min":1.371},"last":0.69199999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575995,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52344,"kb_used_data":3176,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319496,"statfs":{"total":96636764160,"available":96583163904,"internally_reserved":0,"allocated":3252224,"data_stored":1669551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.38,"15min":0.39500000000000002},"min":{"1min":0.23100000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.48899999999999999,"5min":0.66500000000000004,"15min":0.98099999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.39100000000000001,"15min":0.40999999999999998},"min":{"1min":0.26100000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.42699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.39200000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.374,"15min":0.40300000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.47699999999999998,"5min":0.63200000000000001,"15min":0.64000000000000001},"last":0.375},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.44400000000000001,"15min":0.46200000000000002},"min":{"1min":0.315,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.50900000000000001,"5min":0.65500000000000003,"15min":0.73099999999999998},"last":0.36299999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.436,"15min":0.46000000000000002},"min":{"1min":0.29999999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53800000000000003,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.46000000000000002,"15min":0.48799999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.51700000000000002,"5min":0.66100000000000003,"15min":0.73099999999999998},"last":0.42399999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.61199999999999999,"15min":0.64500000000000002},"min":{"1min":0.41599999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":1.526},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.63300000000000001,"15min":0.66900000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.63}]},{"osd":5,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.67000000000000004,"15min":0.69299999999999995},"min":{"1min":0.42999999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":1.5149999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.70299999999999996,"15min":0.72299999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.1120000000000001,"5min":1.171,"15min":1.171},"last":0.63900000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67000000000000004,"15min":0.70299999999999996},"min":{"1min":0.45400000000000001,"5min":0.436,"15min":0.42499999999999999},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":1.573},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.69999999999999996,"15min":0.72599999999999998},"min":{"1min":0.47399999999999998,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":1.22,"5min":1.22,"15min":1.2410000000000001},"last":1.5600000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.68100000000000005,"15min":0.71299999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.624,"5min":0.66000000000000003,"15min":0.69399999999999995},"min":{"1min":0.41099999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":1.5509999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575995,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48212,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323628,"statfs":{"total":96636764160,"available":96587395072,"internally_reserved":0,"allocated":3215360,"data_stored":1658006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57799999999999996,"15min":0.58899999999999997},"min":{"1min":0.39600000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56799999999999995,"15min":0.59499999999999997},"min":{"1min":0.40100000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.84199999999999997},"last":0.40699999999999997}]},{"osd":1,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.79300000000000004,"5min":0.81899999999999995,"15min":0.88300000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56399999999999995,"15min":0.57999999999999996},"min":{"1min":0.34899999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.96499999999999997},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59799999999999998,"15min":0.61399999999999999},"min":{"1min":0.42899999999999999,"5min":0.41299999999999998,"15min":0.38500000000000001},"max":{"1min":0.70799999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.45200000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.59299999999999997,"15min":0.60199999999999998},"min":{"1min":0.433,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78100000000000003,"5min":1.026,"15min":1.026},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.41399999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.58299999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.48399999999999999,"15min":0.48199999999999998},"min":{"1min":0.318,"5min":0.309,"15min":0.309},"max":{"1min":0.76700000000000002,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.496,"15min":0.505},"min":{"1min":0.38300000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.81499999999999995,"5min":0.85499999999999998,"15min":0.90600000000000003},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.502,"15min":0.50600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.754,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.82799999999999996,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.59499999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.40600000000000003,"5min":0.38800000000000001,"15min":0.35699999999999998},"max":{"1min":0.80300000000000005,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.38500000000000001,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.84099999999999997,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.61499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575994,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48192,"kb_used_data":3124,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323648,"statfs":{"total":96636764160,"available":96587415552,"internally_reserved":0,"allocated":3198976,"data_stored":1650380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.57699999999999996,"15min":0.59799999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.31900000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56999999999999995,"15min":0.60099999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33100000000000002},"max":{"1min":0.82399999999999995,"5min":0.82599999999999996,"15min":1.3100000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58199999999999996,"15min":0.60199999999999998},"min":{"1min":0.36599999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.437},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57499999999999996,"15min":0.59799999999999998},"min":{"1min":0.40500000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":1.018},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.57899999999999996,"15min":0.59899999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.99299999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58199999999999996,"15min":0.60899999999999999},"min":{"1min":0.41599999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.52800000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59299999999999997,"15min":0.60999999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":1.2430000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.61399999999999999,"15min":0.64200000000000002},"min":{"1min":0.40600000000000003,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":1.161},"last":0.59999999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.49299999999999999,"15min":0.47999999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.501},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52200000000000002,"15min":0.51100000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.621}]},{"osd":6,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.503,"15min":0.504},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.32800000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":0.57599999999999996}]},{"osd":7,"last update":"Mon Mar 18 20:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56000000000000005,"15min":0.55100000000000005},"min":{"1min":0.32800000000000001,"5min":0.317,"15min":0.317},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56999999999999995,"15min":0.57299999999999995},"min":{"1min":0.34300000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.63600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:26:44.189 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T20:26:44.190 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:26:44.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.341+0000 7fb432389700 1 Processor -- start 2024-03-18T20:26:44.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.341+0000 7fb432389700 1 -- start start 2024-03-18T20:26:44.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.342+0000 7fb432389700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb42c0cf9b0 0x7fb42c099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:44.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.342+0000 7fb432389700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb42c06b860 0x7fb42c099f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:44.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.342+0000 7fb432389700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb42c0d06e0 con 0x7fb42c0b02e0 2024-03-18T20:26:44.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.342+0000 7fb432389700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb42c06ea70 con 0x7fb42c0cf9b0 2024-03-18T20:26:44.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.342+0000 7fb432389700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c0cfe40 con 0x7fb42c06b860 2024-03-18T20:26:44.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.342+0000 7fb42b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb42c0cf9b0 0x7fb42c099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:44.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.342+0000 7fb42b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb42c0cf9b0 0x7fb42c099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35992/0 (socket says 172.21.15.1:35992) 2024-03-18T20:26:44.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.342+0000 7fb42bfff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb42c0b02e0 0x7fb42c0cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56364/0 (socket says 172.21.15.1:56364) 2024-03-18T20:26:44.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.342+0000 7fb42b7fe700 1 -- 172.21.15.1:0/2773655969 learned_addr learned my addr 172.21.15.1:0/2773655969 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:44.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.342+0000 7fb430926700 1 --2- 172.21.15.1:0/2773655969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb42c06b860 0x7fb42c099f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:44.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.342+0000 7fb42b7fe700 1 -- 172.21.15.1:0/2773655969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb42c06b860 msgr2=0x7fb42c099f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:44.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.342+0000 7fb42b7fe700 1 --2- 172.21.15.1:0/2773655969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb42c06b860 0x7fb42c099f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:44.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.342+0000 7fb42b7fe700 1 -- 172.21.15.1:0/2773655969 >> v1:172.21.15.1:6789/0 conn(0x7fb42c0b02e0 legacy=0x7fb42c0cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:44.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.342+0000 7fb42b7fe700 1 -- 172.21.15.1:0/2773655969 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c0d0120 con 0x7fb42c0cf9b0 2024-03-18T20:26:44.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.342+0000 7fb42affd700 1 -- 172.21.15.1:0/2773655969 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554857959 0 0) 0x7fb42c0d06e0 con 0x7fb42c0b02e0 2024-03-18T20:26:44.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.342+0000 7fb42b7fe700 1 --2- 172.21.15.1:0/2773655969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb42c0cf9b0 0x7fb42c099a40 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 crypto rx=0x7fb4200098f0 tx=0x7fb420009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b1f79f55283fa11 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:44.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.342+0000 7fb42affd700 1 -- 172.21.15.1:0/2773655969 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42000b8a0 con 0x7fb42c0cf9b0 2024-03-18T20:26:44.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.343+0000 7fb42affd700 1 -- 172.21.15.1:0/2773655969 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42000ba00 con 0x7fb42c0cf9b0 2024-03-18T20:26:44.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.343+0000 7fb42affd700 1 -- 172.21.15.1:0/2773655969 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42000bd90 con 0x7fb42c0cf9b0 2024-03-18T20:26:44.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.343+0000 7fb432389700 1 -- 172.21.15.1:0/2773655969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb42c0cf9b0 msgr2=0x7fb42c099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:44.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.343+0000 7fb432389700 1 --2- 172.21.15.1:0/2773655969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb42c0cf9b0 0x7fb42c099a40 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 crypto rx=0x7fb4200098f0 tx=0x7fb420009c00 comp rx=0 tx=0).stop 2024-03-18T20:26:44.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.343+0000 7fb432389700 1 -- 172.21.15.1:0/2773655969 shutdown_connections 2024-03-18T20:26:44.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.343+0000 7fb432389700 1 --2- 172.21.15.1:0/2773655969 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb42c06b860 0x7fb42c099f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:44.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.343+0000 7fb432389700 1 --2- 172.21.15.1:0/2773655969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb42c0cf9b0 0x7fb42c099a40 unknown :-1 s=CLOSED pgs=2876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:44.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.343+0000 7fb432389700 1 -- 172.21.15.1:0/2773655969 >> 172.21.15.1:0/2773655969 conn(0x7fb42c0b2ac0 msgr2=0x7fb42c0b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:44.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.343+0000 7fb432389700 1 -- 172.21.15.1:0/2773655969 shutdown_connections 2024-03-18T20:26:44.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.343+0000 7fb432389700 1 -- 172.21.15.1:0/2773655969 wait complete. 2024-03-18T20:26:44.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.344+0000 7fb432389700 1 Processor -- start 2024-03-18T20:26:44.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.344+0000 7fb432389700 1 -- start start 2024-03-18T20:26:44.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.345+0000 7fb432389700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb42c06b860 0x7fb42c1f6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:44.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.345+0000 7fb432389700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb42c0b02e0 0x7fb42c154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:44.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.345+0000 7fb432389700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb42c0c9eb0 0x7fb42c0b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:44.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.345+0000 7fb432389700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c155030 con 0x7fb42c0c9eb0 2024-03-18T20:26:44.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.345+0000 7fb432389700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c1f6a20 con 0x7fb42c0b02e0 2024-03-18T20:26:44.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.345+0000 7fb432389700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c1f6d00 con 0x7fb42c06b860 2024-03-18T20:26:44.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.345+0000 7fb430926700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb42c0c9eb0 0x7fb42c0b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:44.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.345+0000 7fb430926700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb42c0c9eb0 0x7fb42c0b43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36006/0 (socket says 172.21.15.1:36006) 2024-03-18T20:26:44.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.345+0000 7fb430926700 1 -- 172.21.15.1:0/1247237380 learned_addr learned my addr 172.21.15.1:0/1247237380 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:44.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.345+0000 7fb42b7fe700 1 --2- 172.21.15.1:0/1247237380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb42c0b02e0 0x7fb42c154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:44.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.345+0000 7fb42bfff700 1 --2- 172.21.15.1:0/1247237380 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb42c06b860 0x7fb42c1f6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:44.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.346+0000 7fb430926700 1 -- 172.21.15.1:0/1247237380 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb42c06b860 msgr2=0x7fb42c1f6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:44.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.346+0000 7fb430926700 1 --2- 172.21.15.1:0/1247237380 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb42c06b860 0x7fb42c1f6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:44.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.346+0000 7fb430926700 1 -- 172.21.15.1:0/1247237380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb42c0b02e0 msgr2=0x7fb42c154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:44.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.346+0000 7fb430926700 1 --2- 172.21.15.1:0/1247237380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb42c0b02e0 0x7fb42c154af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:44.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.346+0000 7fb430926700 1 -- 172.21.15.1:0/1247237380 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb420009550 con 0x7fb42c0c9eb0 2024-03-18T20:26:44.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.346+0000 7fb430926700 1 --2- 172.21.15.1:0/1247237380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb42c0c9eb0 0x7fb42c0b43e0 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7fb424007e60 tx=0x7fb42400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:44.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.346+0000 7fb428ff9700 1 -- 172.21.15.1:0/1247237380 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42400ed30 con 0x7fb42c0c9eb0 2024-03-18T20:26:44.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.346+0000 7fb428ff9700 1 -- 172.21.15.1:0/1247237380 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb424014420 con 0x7fb42c0c9eb0 2024-03-18T20:26:44.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.346+0000 7fb432389700 1 -- 172.21.15.1:0/1247237380 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42c1f6fe0 con 0x7fb42c0c9eb0 2024-03-18T20:26:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.346+0000 7fb428ff9700 1 -- 172.21.15.1:0/1247237380 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4240147d0 con 0x7fb42c0c9eb0 2024-03-18T20:26:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.346+0000 7fb432389700 1 -- 172.21.15.1:0/1247237380 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb42c15ebf0 con 0x7fb42c0c9eb0 2024-03-18T20:26:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.347+0000 7fb432389700 1 -- 172.21.15.1:0/1247237380 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42c15ee10 con 0x7fb42c0c9eb0 2024-03-18T20:26:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.347+0000 7fb428ff9700 1 -- 172.21.15.1:0/1247237380 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fb424014930 con 0x7fb42c0c9eb0 2024-03-18T20:26:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.348+0000 7fb428ff9700 1 --2- 172.21.15.1:0/1247237380 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb41806aed0 0x7fb41806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.348+0000 7fb428ff9700 1 -- 172.21.15.1:0/1247237380 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6993+0+0 (secure 0 0 0) 0x7fb42408a110 con 0x7fb42c0c9eb0 2024-03-18T20:26:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.348+0000 7fb42bfff700 1 --2- 172.21.15.1:0/1247237380 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb41806aed0 0x7fb41806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.348+0000 7fb42bfff700 1 --2- 172.21.15.1:0/1247237380 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb41806aed0 0x7fb41806d390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb41c000f60 tx=0x7fb41c007e70 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:44.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.352+0000 7fb428ff9700 1 -- 172.21.15.1:0/1247237380 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb424054c90 con 0x7fb42c0c9eb0 2024-03-18T20:26:44.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.605+0000 7fb432389700 1 -- 172.21.15.1:0/1247237380 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb42c073350 con 0x7fb42c0c9eb0 2024-03-18T20:26:44.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.606+0000 7fb428ff9700 1 -- 172.21.15.1:0/1247237380 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+5963 (secure 0 0 0) 0x7fb4240582b0 con 0x7fb42c0c9eb0 2024-03-18T20:26:44.610 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:44.610 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":126,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12396,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2565889517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2565889517},{"type":"v1","addr":"172.21.15.1:6836","nonce":2565889517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":12399,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/512665659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":512665659},{"type":"v1","addr":"172.21.15.1:6837","nonce":512665659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:26:25.785074+0000","modified":"2024-03-18T20:26:29.101123+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12411":{"gid":12411,"name":"d","rank":0,"incarnation":116,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/3886150679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":3886150679},{"type":"v1","addr":"172.21.15.165:6835","nonce":3886150679}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:26:39.596187+0000","modified":"2024-03-18T20:26:42.614846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12408":{"gid":12408,"name":"b","rank":0,"incarnation":122,"state":"up:active","state_seq":6,"addr":"172.21.15.165:6834/1134181166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1134181166},{"type":"v1","addr":"172.21.15.165:6834","nonce":1134181166}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-03-18T20:26:44.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.608+0000 7fb432389700 1 -- 172.21.15.1:0/1247237380 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb41806aed0 msgr2=0x7fb41806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:44.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.608+0000 7fb432389700 1 --2- 172.21.15.1:0/1247237380 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb41806aed0 0x7fb41806d390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb41c000f60 tx=0x7fb41c007e70 comp rx=0 tx=0).stop 2024-03-18T20:26:44.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.608+0000 7fb432389700 1 -- 172.21.15.1:0/1247237380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb42c0c9eb0 msgr2=0x7fb42c0b43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:44.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.608+0000 7fb432389700 1 --2- 172.21.15.1:0/1247237380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb42c0c9eb0 0x7fb42c0b43e0 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7fb424007e60 tx=0x7fb42400e3c0 comp rx=0 tx=0).stop 2024-03-18T20:26:44.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.608+0000 7fb432389700 1 -- 172.21.15.1:0/1247237380 shutdown_connections 2024-03-18T20:26:44.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.608+0000 7fb432389700 1 --2- 172.21.15.1:0/1247237380 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb41806aed0 0x7fb41806d390 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:44.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.608+0000 7fb432389700 1 --2- 172.21.15.1:0/1247237380 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb42c06b860 0x7fb42c1f6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:44.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.608+0000 7fb432389700 1 --2- 172.21.15.1:0/1247237380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb42c0c9eb0 0x7fb42c0b43e0 unknown :-1 s=CLOSED pgs=2877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:44.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.608+0000 7fb432389700 1 --2- 172.21.15.1:0/1247237380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb42c0b02e0 0x7fb42c154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:44.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.608+0000 7fb432389700 1 -- 172.21.15.1:0/1247237380 >> 172.21.15.1:0/1247237380 conn(0x7fb42c0b2ac0 msgr2=0x7fb42c0cce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:44.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.608+0000 7fb432389700 1 -- 172.21.15.1:0/1247237380 shutdown_connections 2024-03-18T20:26:44.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:44.608+0000 7fb432389700 1 -- 172.21.15.1:0/1247237380 wait complete. 2024-03-18T20:26:44.613 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 126 2024-03-18T20:26:45.028 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 126, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T20:26:39.596187+0000', 'modified': '2024-03-18T20:26:42.614846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12408}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12408': {'gid': 12408, 'name': 'b', 'rank': 0, 'incarnation': 122, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.165:6834/1134181166', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6832', 'nonce': 1134181166}, {'type': 'v1', 'addr': '172.21.15.165:6834', 'nonce': 1134181166}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T20:26:45.028 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T20:26:45.029 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T20:26:45.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.155+0000 7ff312977700 1 Processor -- start 2024-03-18T20:26:45.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.156+0000 7ff312977700 1 -- start start 2024-03-18T20:26:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.156+0000 7ff312977700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff30c0727b0 0x7ff30c161610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.156+0000 7ff312977700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff30c160ea0 con 0x7ff30c15ad70 2024-03-18T20:26:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.156+0000 7ff312977700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff30c15e090 con 0x7ff30c161b50 2024-03-18T20:26:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.156+0000 7ff312977700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c0725b0 con 0x7ff30c0727b0 2024-03-18T20:26:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30bfff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff30c15ad70 0x7ff30c15d8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58394/0 (socket says 172.21.15.1:58394) 2024-03-18T20:26:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30bfff700 1 -- 172.21.15.1:0/682799968 learned_addr learned my addr 172.21.15.1:0/682799968 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:45.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30b7fe700 1 --2- 172.21.15.1:0/682799968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff30c0727b0 0x7ff30c161610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30affd700 1 -- 172.21.15.1:0/682799968 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852138494 0 0) 0x7ff30c160ea0 con 0x7ff30c15ad70 2024-03-18T20:26:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30affd700 1 -- 172.21.15.1:0/682799968 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f40035b0 con 0x7ff30c15ad70 2024-03-18T20:26:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30affd700 1 -- 172.21.15.1:0/682799968 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2623081471 0 0) 0x7ff2f40035b0 con 0x7ff30c15ad70 2024-03-18T20:26:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30affd700 1 -- 172.21.15.1:0/682799968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff30c0727b0 msgr2=0x7ff30c161610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30affd700 1 --2- 172.21.15.1:0/682799968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff30c0727b0 0x7ff30c161610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30affd700 1 -- 172.21.15.1:0/682799968 >> v1:172.21.15.165:6789/0 conn(0x7ff30c161b50 legacy=0x7ff30c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30affd700 1 -- 172.21.15.1:0/682799968 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c15e370 con 0x7ff30c15ad70 2024-03-18T20:26:45.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30affd700 1 -- 172.21.15.1:0/682799968 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7ff2fc001670 con 0x7ff30c15ad70 2024-03-18T20:26:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30affd700 1 -- 172.21.15.1:0/682799968 >> v1:172.21.15.1:6789/0 conn(0x7ff30c15ad70 legacy=0x7ff30c15d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30affd700 1 --2- 172.21.15.1:0/682799968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2f4003a30 0x7ff2f4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30affd700 1 --2- 172.21.15.1:0/682799968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2f4004850 0x7ff2f4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30affd700 1 --2- 172.21.15.1:0/682799968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff30c0727b0 0x7ff2f4005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30affd700 1 -- 172.21.15.1:0/682799968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f400bb10 con 0x7ff2f4003a30 2024-03-18T20:26:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30affd700 1 -- 172.21.15.1:0/682799968 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f400bc80 con 0x7ff2f4004850 2024-03-18T20:26:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30affd700 1 -- 172.21.15.1:0/682799968 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f400be40 con 0x7ff30c0727b0 2024-03-18T20:26:45.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30bfff700 1 --2- 172.21.15.1:0/682799968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2f4003a30 0x7ff2f4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff30b7fe700 1 --2- 172.21.15.1:0/682799968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2f4004850 0x7ff2f4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.157+0000 7ff310f14700 1 --2- 172.21.15.1:0/682799968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff30c0727b0 0x7ff2f4005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.158+0000 7ff30bfff700 1 -- 172.21.15.1:0/682799968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff30c0727b0 msgr2=0x7ff2f4005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.158+0000 7ff30bfff700 1 --2- 172.21.15.1:0/682799968 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff30c0727b0 0x7ff2f4005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.158+0000 7ff30bfff700 1 -- 172.21.15.1:0/682799968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2f4004850 msgr2=0x7ff2f4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.158+0000 7ff30bfff700 1 --2- 172.21.15.1:0/682799968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2f4004850 0x7ff2f4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.158+0000 7ff30bfff700 1 -- 172.21.15.1:0/682799968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f40035b0 con 0x7ff2f4003a30 2024-03-18T20:26:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.158+0000 7ff30b7fe700 1 --2- 172.21.15.1:0/682799968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2f4004850 0x7ff2f4004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.158+0000 7ff30bfff700 1 --2- 172.21.15.1:0/682799968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2f4003a30 0x7ff2f4003e50 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc00c240 tx=0x7ff2f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8706ac05f1f721c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.158+0000 7ff30affd700 1 -- 172.21.15.1:0/682799968 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc007c10 con 0x7ff2f4003a30 2024-03-18T20:26:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.158+0000 7ff30affd700 1 -- 172.21.15.1:0/682799968 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2fc00ce00 con 0x7ff2f4003a30 2024-03-18T20:26:45.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.158+0000 7ff30affd700 1 -- 172.21.15.1:0/682799968 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc005ba0 con 0x7ff2f4003a30 2024-03-18T20:26:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.158+0000 7ff312977700 1 -- 172.21.15.1:0/682799968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2f4003a30 msgr2=0x7ff2f4003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.158+0000 7ff312977700 1 --2- 172.21.15.1:0/682799968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2f4003a30 0x7ff2f4003e50 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc00c240 tx=0x7ff2f4005880 comp rx=0 tx=0).stop 2024-03-18T20:26:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.158+0000 7ff310f14700 1 -- 172.21.15.1:0/682799968 reap_dead start 2024-03-18T20:26:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.158+0000 7ff312977700 1 -- 172.21.15.1:0/682799968 shutdown_connections 2024-03-18T20:26:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.158+0000 7ff312977700 1 -- 172.21.15.1:0/682799968 >> 172.21.15.1:0/682799968 conn(0x7ff30c0b1cb0 msgr2=0x7ff30c0b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.158+0000 7ff312977700 1 -- 172.21.15.1:0/682799968 shutdown_connections 2024-03-18T20:26:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.158+0000 7ff312977700 1 -- 172.21.15.1:0/682799968 wait complete. 2024-03-18T20:26:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.160+0000 7ff312977700 1 Processor -- start 2024-03-18T20:26:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.160+0000 7ff312977700 1 -- start start 2024-03-18T20:26:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.160+0000 7ff312977700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2f4003a30 0x7ff30c1435c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.160+0000 7ff312977700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff30c1f76e0 0x7ff30c145f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.160+0000 7ff312977700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff30c0b0e50 0x7ff30c0b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff312977700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c0b1820 con 0x7ff30c0b0e50 2024-03-18T20:26:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff312977700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c146850 con 0x7ff2f4003a30 2024-03-18T20:26:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff312977700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c146b30 con 0x7ff30c1f76e0 2024-03-18T20:26:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff310f14700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff30c0b0e50 0x7ff30c0b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff310f14700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff30c0b0e50 0x7ff30c0b12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46336/0 (socket says 172.21.15.1:46336) 2024-03-18T20:26:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff310f14700 1 -- 172.21.15.1:0/1690496524 learned_addr learned my addr 172.21.15.1:0/1690496524 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff30b7fe700 1 --2- 172.21.15.1:0/1690496524 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff30c1f76e0 0x7ff30c145f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff30bfff700 1 --2- 172.21.15.1:0/1690496524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2f4003a30 0x7ff30c1435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff310f14700 1 -- 172.21.15.1:0/1690496524 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff30c1f76e0 msgr2=0x7ff30c145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff310f14700 1 --2- 172.21.15.1:0/1690496524 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff30c1f76e0 0x7ff30c145f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff310f14700 1 -- 172.21.15.1:0/1690496524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2f4003a30 msgr2=0x7ff30c1435c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff310f14700 1 --2- 172.21.15.1:0/1690496524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2f4003a30 0x7ff30c1435c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff310f14700 1 -- 172.21.15.1:0/1690496524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c0ae070 con 0x7ff30c0b0e50 2024-03-18T20:26:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff30bfff700 1 --2- 172.21.15.1:0/1690496524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2f4003a30 0x7ff30c1435c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff310f14700 1 --2- 172.21.15.1:0/1690496524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff30c0b0e50 0x7ff30c0b12e0 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7ff304000f80 tx=0x7ff30400ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff308ff9700 1 -- 172.21.15.1:0/1690496524 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff304003c10 con 0x7ff30c0b0e50 2024-03-18T20:26:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff308ff9700 1 -- 172.21.15.1:0/1690496524 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff304003d90 con 0x7ff30c0b0e50 2024-03-18T20:26:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff308ff9700 1 -- 172.21.15.1:0/1690496524 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff304005310 con 0x7ff30c0b0e50 2024-03-18T20:26:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff312977700 1 -- 172.21.15.1:0/1690496524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30c0ae350 con 0x7ff30c0b0e50 2024-03-18T20:26:45.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.161+0000 7ff312977700 1 -- 172.21.15.1:0/1690496524 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff30c143b00 con 0x7ff30c0b0e50 2024-03-18T20:26:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.163+0000 7ff308ff9700 1 -- 172.21.15.1:0/1690496524 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7ff304005470 con 0x7ff30c0b0e50 2024-03-18T20:26:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.163+0000 7ff308ff9700 1 --2- 172.21.15.1:0/1690496524 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff2f806aed0 0x7ff2f806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.164+0000 7ff308ff9700 1 -- 172.21.15.1:0/1690496524 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6993+0+0 (secure 0 0 0) 0x7ff304089be0 con 0x7ff30c0b0e50 2024-03-18T20:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.164+0000 7ff30bfff700 1 --2- 172.21.15.1:0/1690496524 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff2f806aed0 0x7ff2f806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.164+0000 7ff30bfff700 1 --2- 172.21.15.1:0/1690496524 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff2f806aed0 0x7ff2f806d390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc001930 tx=0x7ff2fc007000 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.165+0000 7ff31eb20b80 1 Processor -- start 2024-03-18T20:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.165+0000 7ff31eb20b80 1 -- start start 2024-03-18T20:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.166+0000 7ff31eb20b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55de2b4fac10 0x55de2b4faff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.166+0000 7ff31eb20b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4cc0a0 0x55de2b517580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.166+0000 7ff31eb20b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55de2b50c630 0x55de2b4cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.166+0000 7ff31eb20b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2b445f20 con 0x55de2b4cc0a0 2024-03-18T20:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.166+0000 7ff31eb20b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2b427c30 con 0x55de2b4fac10 2024-03-18T20:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.166+0000 7ff31eb20b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de2b425430 con 0x55de2b50c630 2024-03-18T20:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.166+0000 7ff30b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4cc0a0 0x55de2b517580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.166+0000 7ff30b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4cc0a0 0x55de2b517580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46338/0 (socket says 172.21.15.1:46338) 2024-03-18T20:26:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.166+0000 7ff30b7fe700 1 -- 172.21.15.1:0/1512440227 learned_addr learned my addr 172.21.15.1:0/1512440227 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.166+0000 7ff30bfff700 1 --2- 172.21.15.1:0/1512440227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55de2b4fac10 0x55de2b4faff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.166+0000 7ff30b7fe700 1 -- 172.21.15.1:0/1512440227 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55de2b50c630 msgr2=0x55de2b4cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.166+0000 7ff30b7fe700 1 --2- 172.21.15.1:0/1512440227 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55de2b50c630 0x55de2b4cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.166+0000 7ff30b7fe700 1 -- 172.21.15.1:0/1512440227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55de2b4fac10 msgr2=0x55de2b4faff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.166+0000 7ff30b7fe700 1 --2- 172.21.15.1:0/1512440227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55de2b4fac10 0x55de2b4faff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.167+0000 7ff30b7fe700 1 -- 172.21.15.1:0/1512440227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de2b214790 con 0x55de2b4cc0a0 2024-03-18T20:26:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.167+0000 7ff30b7fe700 1 --2- 172.21.15.1:0/1512440227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4cc0a0 0x55de2b517580 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7ff300009bd0 tx=0x7ff300009210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf952b6a1685ae1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.167+0000 7ff2f1ffb700 1 -- 172.21.15.1:0/1512440227 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff300014aa0 con 0x55de2b4cc0a0 2024-03-18T20:26:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.167+0000 7ff2f1ffb700 1 -- 172.21.15.1:0/1512440227 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff300014c00 con 0x55de2b4cc0a0 2024-03-18T20:26:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.167+0000 7ff2f1ffb700 1 -- 172.21.15.1:0/1512440227 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30001f480 con 0x55de2b4cc0a0 2024-03-18T20:26:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.167+0000 7ff31eb20b80 1 -- 172.21.15.1:0/1512440227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4cc0a0 msgr2=0x55de2b517580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.167+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/1512440227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4cc0a0 0x55de2b517580 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7ff300009bd0 tx=0x7ff300009210 comp rx=0 tx=0).stop 2024-03-18T20:26:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.167+0000 7ff31eb20b80 1 -- 172.21.15.1:0/1512440227 shutdown_connections 2024-03-18T20:26:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.167+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/1512440227 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55de2b50c630 0x55de2b4cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.167+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/1512440227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55de2b4fac10 0x55de2b4faff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.167+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/1512440227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4cc0a0 0x55de2b517580 unknown :-1 s=CLOSED pgs=2881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.167+0000 7ff31eb20b80 1 -- 172.21.15.1:0/1512440227 >> 172.21.15.1:0/1512440227 conn(0x55de2b42ecf0 msgr2=0x55de2b4cea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.168+0000 7ff31eb20b80 1 -- 172.21.15.1:0/1512440227 shutdown_connections 2024-03-18T20:26:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.168+0000 7ff31eb20b80 1 -- 172.21.15.1:0/1512440227 wait complete. 2024-03-18T20:26:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.168+0000 7ff31eb20b80 1 -- start start 2024-03-18T20:26:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.168+0000 7ff31eb20b80 1 Processor -- start 2024-03-18T20:26:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.169+0000 7ff31eb20b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55de2b4cc0a0 0x55de2b4cfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.169+0000 7ff31eb20b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55de2b4fac10 0x55de2b4fbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.169+0000 7ff31eb20b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b50c630 0x55de2b4ed7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.169+0000 7ff31eb20b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2b420d20 con 0x55de2b50c630 2024-03-18T20:26:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.169+0000 7ff31eb20b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2b3bc770 con 0x55de2b4cc0a0 2024-03-18T20:26:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.169+0000 7ff31eb20b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de2b3df6f0 con 0x55de2b4fac10 2024-03-18T20:26:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.169+0000 7ff31eb20b80 10 client.0 _ll_register_callbacks cb 0x55de2b2285a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:26:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.169+0000 7ff310f14700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b50c630 0x55de2b4ed7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.169+0000 7ff310f14700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b50c630 0x55de2b4ed7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46350/0 (socket says 172.21.15.1:46350) 2024-03-18T20:26:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.169+0000 7ff310f14700 1 -- 172.21.15.1:0/3723365004 learned_addr learned my addr 172.21.15.1:0/3723365004 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.169+0000 7ff30b7fe700 1 --2- 172.21.15.1:0/3723365004 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55de2b4fac10 0x55de2b4fbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.169+0000 7ff30bfff700 1 --2- 172.21.15.1:0/3723365004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55de2b4cc0a0 0x55de2b4cfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff310f14700 1 -- 172.21.15.1:0/3723365004 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55de2b4fac10 msgr2=0x55de2b4fbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff310f14700 1 --2- 172.21.15.1:0/3723365004 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55de2b4fac10 0x55de2b4fbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff310f14700 1 -- 172.21.15.1:0/3723365004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55de2b4cc0a0 msgr2=0x55de2b4cfed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff310f14700 1 --2- 172.21.15.1:0/3723365004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55de2b4cc0a0 0x55de2b4cfed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff310f14700 1 -- 172.21.15.1:0/3723365004 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff300009800 con 0x55de2b50c630 2024-03-18T20:26:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff310f14700 1 --2- 172.21.15.1:0/3723365004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b50c630 0x55de2b4ed7a0 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7ff3040013d0 tx=0x7ff3040583b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc5f307c29578e7f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff2eaffd700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T20:26:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff2eaffd700 1 -- 172.21.15.1:0/3723365004 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff304058720 con 0x55de2b50c630 2024-03-18T20:26:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff2eaffd700 1 -- 172.21.15.1:0/3723365004 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(257..257 src has 1..257) v4 ==== 6993+0+0 (secure 0 0 0) 0x7ff30405e300 con 0x55de2b50c630 2024-03-18T20:26:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff31eb20b80 1 -- 172.21.15.1:0/3723365004 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55de2b4c2e90 con 0x55de2b50c630 2024-03-18T20:26:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff2eaffd700 1 -- 172.21.15.1:0/3723365004 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=126) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff304060600 con 0x55de2b50c630 2024-03-18T20:26:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff31eb20b80 10 client.12696 fetch_fsmap learned FSMap version 126 2024-03-18T20:26:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff31eb20b80 1 -- 172.21.15.1:0/3723365004 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=126,monmap=2+}) v3 -- 0x55de2b3010a0 con 0x55de2b50c630 2024-03-18T20:26:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff2eaffd700 1 -- 172.21.15.1:0/3723365004 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55de2b4c2e90 con 0x55de2b50c630 2024-03-18T20:26:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff2eaffd700 1 -- 172.21.15.1:0/3723365004 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 126) v1 ==== 3681+0+0 (secure 0 0 0) 0x7ff3040577b0 con 0x55de2b50c630 2024-03-18T20:26:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff31eb20b80 10 client.12696 fetch_fsmap finished waiting for FSMap version 126 2024-03-18T20:26:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff31eb20b80 10 client.12696 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T20:26:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff31eb20b80 20 client.12696 populate_metadata read hostname 'smithi001' 2024-03-18T20:26:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/3723365004 >> [v2:172.21.15.165:6832/1134181166,v1:172.21.15.165:6834/1134181166] conn(0x55de2b4f1c30 0x55de2b4ee1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff31eb20b80 4 client.12696 mds_command: new command op to 12408 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T20:26:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.170+0000 7ff31eb20b80 1 -- 172.21.15.1:0/3723365004 --> [v2:172.21.15.165:6832/1134181166,v1:172.21.15.165:6834/1134181166] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55de2b4a0010 con 0x55de2b4f1c30 2024-03-18T20:26:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.171+0000 7ff30bfff700 1 --2- 172.21.15.1:0/3723365004 >> [v2:172.21.15.165:6832/1134181166,v1:172.21.15.165:6834/1134181166] conn(0x55de2b4f1c30 0x55de2b4ee1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.171+0000 7ff2eaffd700 10 client.12696 ms_handle_connect on v2:172.21.15.165:6832/1134181166 2024-03-18T20:26:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.171+0000 7ff30bfff700 1 --2- 172.21.15.1:0/3723365004 >> [v2:172.21.15.165:6832/1134181166,v1:172.21.15.165:6834/1134181166] conn(0x55de2b4f1c30 0x55de2b4ee1d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b802f260b2f7770 server_cookie=a1833da7587efa22 in_seq=0 out_seq=0 2024-03-18T20:26:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.172+0000 7ff2eaffd700 1 -- 172.21.15.1:0/3723365004 <== mds.0 v2:172.21.15.165:6832/1134181166 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55de2b4a0010 con 0x55de2b4f1c30 2024-03-18T20:26:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.172+0000 7ff2eaffd700 10 client.12696 handle_command_reply: tid=0 2024-03-18T20:26:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.172+0000 7ff31eb20b80 1 client.12696 shutdown 2024-03-18T20:26:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.172+0000 7ff31eb20b80 1 -- 172.21.15.1:0/3723365004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b50c630 msgr2=0x55de2b4ed7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.172+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/3723365004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b50c630 0x55de2b4ed7a0 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7ff3040013d0 tx=0x7ff3040583b0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.172+0000 7ff31eb20b80 1 -- 172.21.15.1:0/3723365004 shutdown_connections 2024-03-18T20:26:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.172+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/3723365004 >> [v2:172.21.15.165:6832/1134181166,v1:172.21.15.165:6834/1134181166] conn(0x55de2b4f1c30 0x55de2b4ee1d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.172+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/3723365004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b50c630 0x55de2b4ed7a0 unknown :-1 s=CLOSED pgs=2882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.172+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/3723365004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55de2b4cc0a0 0x55de2b4cfed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.172+0000 7ff2eaffd700 0 client.12696 ms_handle_reset on v2:172.21.15.165:6832/1134181166 2024-03-18T20:26:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.172+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/3723365004 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55de2b4fac10 0x55de2b4fbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.172+0000 7ff31eb20b80 1 -- 172.21.15.1:0/3723365004 >> 172.21.15.1:0/3723365004 conn(0x55de2b42ecf0 msgr2=0x55de2b4f5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:45.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.172+0000 7ff31eb20b80 1 -- 172.21.15.1:0/3723365004 shutdown_connections 2024-03-18T20:26:45.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.172+0000 7ff31eb20b80 1 -- 172.21.15.1:0/3723365004 wait complete. 2024-03-18T20:26:45.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.172+0000 7ff31eb20b80 20 client.12696 trim_cache size 0 max 16384 2024-03-18T20:26:45.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.190+0000 7ff31eb20b80 1 Processor -- start 2024-03-18T20:26:45.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.190+0000 7ff31eb20b80 1 -- start start 2024-03-18T20:26:45.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.190+0000 7ff31eb20b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4cc0a0 0x55de2b4cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.190+0000 7ff31eb20b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55de2b4fac10 0x55de2b4efe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.190+0000 7ff31eb20b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55de2b50c630 0x55de2b4f0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.190+0000 7ff31eb20b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2b215390 con 0x55de2b4cc0a0 2024-03-18T20:26:45.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.190+0000 7ff31eb20b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2b4f8fa0 con 0x55de2b50c630 2024-03-18T20:26:45.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.190+0000 7ff31eb20b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de2b4c29d0 con 0x55de2b4fac10 2024-03-18T20:26:45.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.191+0000 7ff30bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4cc0a0 0x55de2b4cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.191+0000 7ff30bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4cc0a0 0x55de2b4cdc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46362/0 (socket says 172.21.15.1:46362) 2024-03-18T20:26:45.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.191+0000 7ff30bfff700 1 -- 172.21.15.1:0/2020949529 learned_addr learned my addr 172.21.15.1:0/2020949529 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.191+0000 7ff30bfff700 1 -- 172.21.15.1:0/2020949529 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55de2b4fac10 msgr2=0x55de2b4efe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.191+0000 7ff30bfff700 1 --2- 172.21.15.1:0/2020949529 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55de2b4fac10 0x55de2b4efe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.191+0000 7ff30bfff700 1 -- 172.21.15.1:0/2020949529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55de2b50c630 msgr2=0x55de2b4f0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.191+0000 7ff30bfff700 1 --2- 172.21.15.1:0/2020949529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55de2b50c630 0x55de2b4f0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.191+0000 7ff30bfff700 1 -- 172.21.15.1:0/2020949529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de2b3e7010 con 0x55de2b4cc0a0 2024-03-18T20:26:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.191+0000 7ff30bfff700 1 --2- 172.21.15.1:0/2020949529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4cc0a0 0x55de2b4cdc00 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc011000 tx=0x7ff2fc00a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f345d7f10fb48a56 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.191+0000 7ff2f27fc700 1 -- 172.21.15.1:0/2020949529 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc032450 con 0x55de2b4cc0a0 2024-03-18T20:26:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.191+0000 7ff2f27fc700 1 -- 172.21.15.1:0/2020949529 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2fc031040 con 0x55de2b4cc0a0 2024-03-18T20:26:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.191+0000 7ff2f27fc700 1 -- 172.21.15.1:0/2020949529 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc004a90 con 0x55de2b4cc0a0 2024-03-18T20:26:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.192+0000 7ff31eb20b80 1 -- 172.21.15.1:0/2020949529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4cc0a0 msgr2=0x55de2b4cdc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.192+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/2020949529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4cc0a0 0x55de2b4cdc00 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc011000 tx=0x7ff2fc00a000 comp rx=0 tx=0).stop 2024-03-18T20:26:45.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.192+0000 7ff31eb20b80 1 -- 172.21.15.1:0/2020949529 shutdown_connections 2024-03-18T20:26:45.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.192+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/2020949529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55de2b50c630 0x55de2b4f0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.192+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/2020949529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4cc0a0 0x55de2b4cdc00 unknown :-1 s=CLOSED pgs=2883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.192+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/2020949529 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55de2b4fac10 0x55de2b4efe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.192+0000 7ff31eb20b80 1 -- 172.21.15.1:0/2020949529 >> 172.21.15.1:0/2020949529 conn(0x55de2b42ecf0 msgr2=0x55de2b4ed5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:45.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.192+0000 7ff31eb20b80 1 -- 172.21.15.1:0/2020949529 shutdown_connections 2024-03-18T20:26:45.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.192+0000 7ff31eb20b80 1 -- 172.21.15.1:0/2020949529 wait complete. 2024-03-18T20:26:45.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.192+0000 7ff31eb20b80 1 -- start start 2024-03-18T20:26:45.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.193+0000 7ff31eb20b80 1 Processor -- start 2024-03-18T20:26:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff31eb20b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55de2b4cc0a0 0x55de2b57c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff31eb20b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4fac10 0x55de2b4c9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff31eb20b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55de2b50c630 0x55de2b4ca050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff31eb20b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2b512300 con 0x55de2b4fac10 2024-03-18T20:26:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff31eb20b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2b4fca50 con 0x55de2b4cc0a0 2024-03-18T20:26:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff31eb20b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de2b4f8720 con 0x55de2b50c630 2024-03-18T20:26:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff31eb20b80 10 client.0 _ll_register_callbacks cb 0x55de2b2285a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:26:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff30b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4fac10 0x55de2b4c9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff30b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4fac10 0x55de2b4c9b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46374/0 (socket says 172.21.15.1:46374) 2024-03-18T20:26:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff30b7fe700 1 -- 172.21.15.1:0/3112176452 learned_addr learned my addr 172.21.15.1:0/3112176452 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff310f14700 1 --2- 172.21.15.1:0/3112176452 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55de2b50c630 0x55de2b4ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff30bfff700 1 --2- 172.21.15.1:0/3112176452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55de2b4cc0a0 0x55de2b57c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff30b7fe700 1 -- 172.21.15.1:0/3112176452 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55de2b50c630 msgr2=0x55de2b4ca050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:45.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff30b7fe700 1 --2- 172.21.15.1:0/3112176452 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55de2b50c630 0x55de2b4ca050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff30b7fe700 1 -- 172.21.15.1:0/3112176452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55de2b4cc0a0 msgr2=0x55de2b57c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:45.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff30b7fe700 1 --2- 172.21.15.1:0/3112176452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55de2b4cc0a0 0x55de2b57c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff30b7fe700 1 -- 172.21.15.1:0/3112176452 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f40035b0 con 0x55de2b4fac10 2024-03-18T20:26:45.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff30b7fe700 1 --2- 172.21.15.1:0/3112176452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4fac10 0x55de2b4c9b10 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7ff300008010 tx=0x7ff3000047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df4b9e1c3dd763e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:45.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff2eaffd700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T20:26:45.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff2eaffd700 1 -- 172.21.15.1:0/3112176452 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff300025040 con 0x55de2b4fac10 2024-03-18T20:26:45.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff2eaffd700 1 -- 172.21.15.1:0/3112176452 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(257..257 src has 1..257) v4 ==== 6993+0+0 (secure 0 0 0) 0x7ff30001a070 con 0x55de2b4fac10 2024-03-18T20:26:45.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.194+0000 7ff31eb20b80 1 -- 172.21.15.1:0/3112176452 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff300014c00 con 0x55de2b4fac10 2024-03-18T20:26:45.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.195+0000 7ff2eaffd700 1 -- 172.21.15.1:0/3112176452 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=126) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3000038a0 con 0x55de2b4fac10 2024-03-18T20:26:45.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.195+0000 7ff31eb20b80 10 client.12702 fetch_fsmap learned FSMap version 126 2024-03-18T20:26:45.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.195+0000 7ff31eb20b80 1 -- 172.21.15.1:0/3112176452 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=126,monmap=2+}) v3 -- 0x55de2b2751d0 con 0x55de2b4fac10 2024-03-18T20:26:45.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.195+0000 7ff2eaffd700 1 -- 172.21.15.1:0/3112176452 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff300014c00 con 0x55de2b4fac10 2024-03-18T20:26:45.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.195+0000 7ff2eaffd700 1 -- 172.21.15.1:0/3112176452 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 126) v1 ==== 3681+0+0 (secure 0 0 0) 0x7ff300031020 con 0x55de2b4fac10 2024-03-18T20:26:45.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.195+0000 7ff31eb20b80 10 client.12702 fetch_fsmap finished waiting for FSMap version 126 2024-03-18T20:26:45.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.195+0000 7ff31eb20b80 10 client.12702 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T20:26:45.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.195+0000 7ff31eb20b80 20 client.12702 populate_metadata read hostname 'smithi001' 2024-03-18T20:26:45.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.195+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/3112176452 >> [v2:172.21.15.165:6832/1134181166,v1:172.21.15.165:6834/1134181166] conn(0x55de2b515970 0x55de2b515d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.195+0000 7ff31eb20b80 4 client.12702 mds_command: new command op to 12408 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T20:26:45.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.195+0000 7ff31eb20b80 1 -- 172.21.15.1:0/3112176452 --> [v2:172.21.15.165:6832/1134181166,v1:172.21.15.165:6834/1134181166] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55de2b25a040 con 0x55de2b515970 2024-03-18T20:26:45.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.195+0000 7ff30bfff700 1 --2- 172.21.15.1:0/3112176452 >> [v2:172.21.15.165:6832/1134181166,v1:172.21.15.165:6834/1134181166] conn(0x55de2b515970 0x55de2b515d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.196+0000 7ff30bfff700 1 --2- 172.21.15.1:0/3112176452 >> [v2:172.21.15.165:6832/1134181166,v1:172.21.15.165:6834/1134181166] conn(0x55de2b515970 0x55de2b515d50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec54a642a18fdb46 server_cookie=84f55bebfcf8aa25 in_seq=0 out_seq=0 2024-03-18T20:26:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.196+0000 7ff2eaffd700 10 client.12702 ms_handle_connect on v2:172.21.15.165:6832/1134181166 2024-03-18T20:26:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.196+0000 7ff2eaffd700 1 -- 172.21.15.1:0/3112176452 <== mds.0 v2:172.21.15.165:6832/1134181166 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+338 (crc 0 0 0) 0x55de2b25a040 con 0x55de2b515970 2024-03-18T20:26:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.196+0000 7ff2eaffd700 10 client.12702 handle_command_reply: tid=0 2024-03-18T20:26:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.196+0000 7ff31eb20b80 1 client.12702 shutdown 2024-03-18T20:26:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.196+0000 7ff31eb20b80 1 -- 172.21.15.1:0/3112176452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4fac10 msgr2=0x55de2b4c9b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.196+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/3112176452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4fac10 0x55de2b4c9b10 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7ff300008010 tx=0x7ff3000047c0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.196+0000 7ff31eb20b80 1 -- 172.21.15.1:0/3112176452 shutdown_connections 2024-03-18T20:26:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.196+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/3112176452 >> [v2:172.21.15.165:6832/1134181166,v1:172.21.15.165:6834/1134181166] conn(0x55de2b515970 0x55de2b515d50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.196+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/3112176452 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55de2b50c630 0x55de2b4ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.196+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/3112176452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55de2b4cc0a0 0x55de2b57c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.196+0000 7ff31eb20b80 1 --2- 172.21.15.1:0/3112176452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de2b4fac10 0x55de2b4c9b10 unknown :-1 s=CLOSED pgs=2884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.196+0000 7ff2eaffd700 0 client.12702 ms_handle_reset on v2:172.21.15.165:6832/1134181166 2024-03-18T20:26:45.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.196+0000 7ff31eb20b80 1 -- 172.21.15.1:0/3112176452 >> 172.21.15.1:0/3112176452 conn(0x55de2b42ecf0 msgr2=0x55de2b4ec8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:45.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.196+0000 7ff31eb20b80 1 -- 172.21.15.1:0/3112176452 shutdown_connections 2024-03-18T20:26:45.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.196+0000 7ff31eb20b80 1 -- 172.21.15.1:0/3112176452 wait complete. 2024-03-18T20:26:45.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.196+0000 7ff31eb20b80 20 client.12702 trim_cache size 0 max 16384 2024-03-18T20:26:45.208 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:26:45.208 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "9c1c697f-8655-4ef3-8a13-da568d2cbe86", 2024-03-18T20:26:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-18T20:26:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 12408, 2024-03-18T20:26:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-18T20:26:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-18T20:26:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "backup_fs", 2024-03-18T20:26:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 5.5984914999999997, 2024-03-18T20:26:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 126, 2024-03-18T20:26:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 257, 2024-03-18T20:26:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 257, 2024-03-18T20:26:45.209 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 24.735357945000001 2024-03-18T20:26:45.209 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:26:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.197+0000 7ff312977700 1 -- 172.21.15.1:0/1690496524 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff2f806aed0 msgr2=0x7ff2f806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.197+0000 7ff312977700 1 --2- 172.21.15.1:0/1690496524 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff2f806aed0 0x7ff2f806d390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc001930 tx=0x7ff2fc007000 comp rx=0 tx=0).stop 2024-03-18T20:26:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.197+0000 7ff312977700 1 -- 172.21.15.1:0/1690496524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff30c0b0e50 msgr2=0x7ff30c0b12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.197+0000 7ff312977700 1 --2- 172.21.15.1:0/1690496524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff30c0b0e50 0x7ff30c0b12e0 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7ff304000f80 tx=0x7ff30400ed50 comp rx=0 tx=0).stop 2024-03-18T20:26:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.197+0000 7ff312977700 1 -- 172.21.15.1:0/1690496524 shutdown_connections 2024-03-18T20:26:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.197+0000 7ff312977700 1 --2- 172.21.15.1:0/1690496524 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff2f806aed0 0x7ff2f806d390 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.197+0000 7ff312977700 1 --2- 172.21.15.1:0/1690496524 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff30c0b0e50 0x7ff30c0b12e0 unknown :-1 s=CLOSED pgs=2880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.197+0000 7ff312977700 1 --2- 172.21.15.1:0/1690496524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff2f4003a30 0x7ff30c1435c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.197+0000 7ff312977700 1 --2- 172.21.15.1:0/1690496524 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff30c1f76e0 0x7ff30c145f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.197+0000 7ff312977700 1 -- 172.21.15.1:0/1690496524 >> 172.21.15.1:0/1690496524 conn(0x7ff30c0b1cb0 msgr2=0x7ff30c15ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.197+0000 7ff312977700 1 -- 172.21.15.1:0/1690496524 shutdown_connections 2024-03-18T20:26:45.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.197+0000 7ff312977700 1 -- 172.21.15.1:0/1690496524 wait complete. 2024-03-18T20:26:45.213 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-18T20:26:45.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.335+0000 7f00f382f700 1 Processor -- start 2024-03-18T20:26:45.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.335+0000 7f00f382f700 1 -- start start 2024-03-18T20:26:45.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00f382f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f00ec15a240 0x7f00ec0ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00f382f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ec0cac80 con 0x7f00ec071cc0 2024-03-18T20:26:45.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00f382f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ec0caf60 con 0x7f00ec06d560 2024-03-18T20:26:45.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00f382f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00ec0adff0 con 0x7f00ec15a240 2024-03-18T20:26:45.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00f1dcc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f00ec071cc0 0x7f00ec0ae410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58404/0 (socket says 172.21.15.1:58404) 2024-03-18T20:26:45.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00f1dcc700 1 -- 172.21.15.1:0/991937485 learned_addr learned my addr 172.21.15.1:0/991937485 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:45.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00f15cb700 1 --2- 172.21.15.1:0/991937485 >> v2:172.21.15.165:3300/0 conn(0x7f00ec15a240 0x7f00ec0ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00e3fff700 1 -- 172.21.15.1:0/991937485 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574787245 0 0) 0x7f00ec0cac80 con 0x7f00ec071cc0 2024-03-18T20:26:45.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00e3fff700 1 -- 172.21.15.1:0/991937485 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00d80035b0 con 0x7f00ec071cc0 2024-03-18T20:26:45.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00e3fff700 1 -- 172.21.15.1:0/991937485 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 909746850 0 0) 0x7f00d80035b0 con 0x7f00ec071cc0 2024-03-18T20:26:45.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00e3fff700 1 -- 172.21.15.1:0/991937485 >> v2:172.21.15.165:3300/0 conn(0x7f00ec15a240 msgr2=0x7f00ec0ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:45.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00e3fff700 1 --2- 172.21.15.1:0/991937485 >> v2:172.21.15.165:3300/0 conn(0x7f00ec15a240 0x7f00ec0ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00e3fff700 1 -- 172.21.15.1:0/991937485 >> v1:172.21.15.165:6789/0 conn(0x7f00ec06d560 legacy=0x7f00ec0715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00e3fff700 1 -- 172.21.15.1:0/991937485 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec0cb240 con 0x7f00ec071cc0 2024-03-18T20:26:45.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00e3fff700 1 -- 172.21.15.1:0/991937485 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f00e40019f0 con 0x7f00ec071cc0 2024-03-18T20:26:45.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00e3fff700 1 -- 172.21.15.1:0/991937485 >> v1:172.21.15.1:6789/0 conn(0x7f00ec071cc0 legacy=0x7f00ec0ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00e3fff700 1 --2- 172.21.15.1:0/991937485 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00d8003a00 0x7f00d8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00e3fff700 1 --2- 172.21.15.1:0/991937485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d8004850 0x7f00d8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00e3fff700 1 --2- 172.21.15.1:0/991937485 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00d8005210 0x7f00d800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.336+0000 7f00e3fff700 1 -- 172.21.15.1:0/991937485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec0adff0 con 0x7f00d8004850 2024-03-18T20:26:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.337+0000 7f00e3fff700 1 -- 172.21.15.1:0/991937485 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d800c0b0 con 0x7f00d8005210 2024-03-18T20:26:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.337+0000 7f00e3fff700 1 -- 172.21.15.1:0/991937485 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d800c270 con 0x7f00d8003a00 2024-03-18T20:26:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.337+0000 7f00f0dca700 1 --2- 172.21.15.1:0/991937485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d8004850 0x7f00d8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.337+0000 7f00f15cb700 1 --2- 172.21.15.1:0/991937485 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00d8003a00 0x7f00d8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.337+0000 7f00f1dcc700 1 --2- 172.21.15.1:0/991937485 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00d8005210 0x7f00d800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.337+0000 7f00f0dca700 1 -- 172.21.15.1:0/991937485 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00d8003a00 msgr2=0x7f00d8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.337+0000 7f00f0dca700 1 --2- 172.21.15.1:0/991937485 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00d8003a00 0x7f00d8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.337+0000 7f00f0dca700 1 -- 172.21.15.1:0/991937485 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00d8005210 msgr2=0x7f00d800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.337+0000 7f00f0dca700 1 --2- 172.21.15.1:0/991937485 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00d8005210 0x7f00d800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.337+0000 7f00f0dca700 1 -- 172.21.15.1:0/991937485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d80035b0 con 0x7f00d8004850 2024-03-18T20:26:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.337+0000 7f00f1dcc700 1 -- 172.21.15.1:0/991937485 reap_dead start 2024-03-18T20:26:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.337+0000 7f00f0dca700 1 --2- 172.21.15.1:0/991937485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d8004850 0x7f00d8004cd0 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7f00ec06da10 tx=0x7f00d8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e91be120a64da45 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:45.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.337+0000 7f00e3fff700 1 -- 172.21.15.1:0/991937485 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00dc0052d0 con 0x7f00d8004850 2024-03-18T20:26:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.337+0000 7f00e3fff700 1 -- 172.21.15.1:0/991937485 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00dc005450 con 0x7f00d8004850 2024-03-18T20:26:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.337+0000 7f00e3fff700 1 -- 172.21.15.1:0/991937485 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00dc0038c0 con 0x7f00d8004850 2024-03-18T20:26:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.338+0000 7f00f382f700 1 -- 172.21.15.1:0/991937485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d8004850 msgr2=0x7f00d8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.338+0000 7f00f382f700 1 --2- 172.21.15.1:0/991937485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d8004850 0x7f00d8004cd0 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7f00ec06da10 tx=0x7f00d8005880 comp rx=0 tx=0).stop 2024-03-18T20:26:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.338+0000 7f00f382f700 1 -- 172.21.15.1:0/991937485 shutdown_connections 2024-03-18T20:26:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.338+0000 7f00f382f700 1 --2- 172.21.15.1:0/991937485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d8004850 0x7f00d8004cd0 unknown :-1 s=CLOSED pgs=2886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.338+0000 7f00f382f700 1 -- 172.21.15.1:0/991937485 >> 172.21.15.1:0/991937485 conn(0x7f00ec0cff30 msgr2=0x7f00ec0d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.338+0000 7f00f382f700 1 -- 172.21.15.1:0/991937485 shutdown_connections 2024-03-18T20:26:45.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.338+0000 7f00f382f700 1 -- 172.21.15.1:0/991937485 wait complete. 2024-03-18T20:26:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.339+0000 7f00f382f700 1 Processor -- start 2024-03-18T20:26:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.339+0000 7f00f382f700 1 -- start start 2024-03-18T20:26:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.340+0000 7f00f382f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d8004850 0x7f00ec155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.340+0000 7f00f382f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00ec151450 0x7f00ec0c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.340+0000 7f00f382f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00ec0c4900 0x7f00ec0b1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.340+0000 7f00f382f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec0c4eb0 con 0x7f00d8004850 2024-03-18T20:26:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.340+0000 7f00f382f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec0b3b00 con 0x7f00ec0c4900 2024-03-18T20:26:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.340+0000 7f00f382f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec0b4640 con 0x7f00ec151450 2024-03-18T20:26:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.340+0000 7f00f15cb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d8004850 0x7f00ec155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.340+0000 7f00f15cb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d8004850 0x7f00ec155ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46398/0 (socket says 172.21.15.1:46398) 2024-03-18T20:26:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.340+0000 7f00f15cb700 1 -- 172.21.15.1:0/322784070 learned_addr learned my addr 172.21.15.1:0/322784070 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:45.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.340+0000 7f00f0dca700 1 --2- 172.21.15.1:0/322784070 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00ec151450 0x7f00ec0c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.340+0000 7f00f1dcc700 1 --2- 172.21.15.1:0/322784070 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00ec0c4900 0x7f00ec0b1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.340+0000 7f00f15cb700 1 -- 172.21.15.1:0/322784070 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00ec151450 msgr2=0x7f00ec0c43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.340+0000 7f00f15cb700 1 --2- 172.21.15.1:0/322784070 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00ec151450 0x7f00ec0c43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.340+0000 7f00f15cb700 1 -- 172.21.15.1:0/322784070 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00ec0c4900 msgr2=0x7f00ec0b1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.340+0000 7f00f15cb700 1 --2- 172.21.15.1:0/322784070 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00ec0c4900 0x7f00ec0b1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.340+0000 7f00f15cb700 1 -- 172.21.15.1:0/322784070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec0b3d10 con 0x7f00d8004850 2024-03-18T20:26:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.340+0000 7f00f15cb700 1 --2- 172.21.15.1:0/322784070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d8004850 0x7f00ec155ab0 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7f00e8011aa0 tx=0x7f00e8011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.341+0000 7f00e1ffb700 1 -- 172.21.15.1:0/322784070 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e8011ee0 con 0x7f00d8004850 2024-03-18T20:26:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.341+0000 7f00e1ffb700 1 -- 172.21.15.1:0/322784070 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e800f860 con 0x7f00d8004850 2024-03-18T20:26:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.341+0000 7f00f382f700 1 -- 172.21.15.1:0/322784070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ec0b3ff0 con 0x7f00d8004850 2024-03-18T20:26:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.341+0000 7f00e1ffb700 1 -- 172.21.15.1:0/322784070 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e800fc30 con 0x7f00d8004850 2024-03-18T20:26:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.341+0000 7f00f382f700 1 -- 172.21.15.1:0/322784070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ec0c6940 con 0x7f00d8004850 2024-03-18T20:26:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.342+0000 7f00f382f700 1 -- 172.21.15.1:0/322784070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00ec0b4920 con 0x7f00d8004850 2024-03-18T20:26:45.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.342+0000 7f00e1ffb700 1 -- 172.21.15.1:0/322784070 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f00e800fd90 con 0x7f00d8004850 2024-03-18T20:26:45.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.343+0000 7f00e1ffb700 1 --2- 172.21.15.1:0/322784070 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f00d006aed0 0x7f00d006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.343+0000 7f00e1ffb700 1 -- 172.21.15.1:0/322784070 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f00e808a150 con 0x7f00d8004850 2024-03-18T20:26:45.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.343+0000 7f00f0dca700 1 --2- 172.21.15.1:0/322784070 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f00d006aed0 0x7f00d006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.343+0000 7f00f0dca700 1 --2- 172.21.15.1:0/322784070 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f00d006aed0 0x7f00d006d390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f00dc005040 tx=0x7f00dc009030 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:45.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.348+0000 7f00e1ffb700 1 -- 172.21.15.1:0/322784070 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f00e8054c10 con 0x7f00d8004850 2024-03-18T20:26:45.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.578+0000 7f00f382f700 1 -- 172.21.15.1:0/322784070 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f00ec15a620 con 0x7f00d006aed0 2024-03-18T20:26:45.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.580+0000 7f00e1ffb700 1 -- 172.21.15.1:0/322784070 <== mgr.12057 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7f00dc022070 con 0x7f00d006aed0 2024-03-18T20:26:45.584 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T20:26:45.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:45.584 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephfs" 2024-03-18T20:26:45.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:26:45.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:26:45.584 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "backup_fs" 2024-03-18T20:26:45.584 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:26:45.585 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T20:26:45.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.582+0000 7f00f382f700 1 -- 172.21.15.1:0/322784070 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f00d006aed0 msgr2=0x7f00d006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.582+0000 7f00f382f700 1 --2- 172.21.15.1:0/322784070 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f00d006aed0 0x7f00d006d390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f00dc005040 tx=0x7f00dc009030 comp rx=0 tx=0).stop 2024-03-18T20:26:45.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.582+0000 7f00f382f700 1 -- 172.21.15.1:0/322784070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d8004850 msgr2=0x7f00ec155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.582+0000 7f00f382f700 1 --2- 172.21.15.1:0/322784070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d8004850 0x7f00ec155ab0 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7f00e8011aa0 tx=0x7f00e8011db0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.582+0000 7f00f382f700 1 -- 172.21.15.1:0/322784070 shutdown_connections 2024-03-18T20:26:45.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.582+0000 7f00f382f700 1 --2- 172.21.15.1:0/322784070 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f00d006aed0 0x7f00d006d390 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.582+0000 7f00f382f700 1 --2- 172.21.15.1:0/322784070 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00ec0c4900 0x7f00ec0b1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.582+0000 7f00f382f700 1 --2- 172.21.15.1:0/322784070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d8004850 0x7f00ec155ab0 unknown :-1 s=CLOSED pgs=2887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.582+0000 7f00f382f700 1 --2- 172.21.15.1:0/322784070 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f00ec151450 0x7f00ec0c43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.582+0000 7f00f382f700 1 -- 172.21.15.1:0/322784070 >> 172.21.15.1:0/322784070 conn(0x7f00ec0cff30 msgr2=0x7f00ec0ce8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:45.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.582+0000 7f00f382f700 1 -- 172.21.15.1:0/322784070 shutdown_connections 2024-03-18T20:26:45.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.582+0000 7f00f382f700 1 -- 172.21.15.1:0/322784070 wait complete. 2024-03-18T20:26:45.598 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-18T20:26:45.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.721+0000 7fd45f59e700 1 Processor -- start 2024-03-18T20:26:45.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.721+0000 7fd45f59e700 1 -- start start 2024-03-18T20:26:45.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.722+0000 7fd45f59e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd46015b0a0 0x7fd4600acd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.722+0000 7fd45f59e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd46006a750 0x7fd46006ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.722+0000 7fd45f59e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4600cfd70 con 0x7fd46015eca0 2024-03-18T20:26:45.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.722+0000 7fd45f59e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd46006b140 con 0x7fd46015b0a0 2024-03-18T20:26:45.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.722+0000 7fd45f59e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4600cdae0 con 0x7fd46006a750 2024-03-18T20:26:45.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.722+0000 7fd45e59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd46015b0a0 0x7fd4600acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.722+0000 7fd45e59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd46015b0a0 0x7fd4600acd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46408/0 (socket says 172.21.15.1:46408) 2024-03-18T20:26:45.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.722+0000 7fd45e59c700 1 -- 172.21.15.1:0/4157064575 learned_addr learned my addr 172.21.15.1:0/4157064575 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:45.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.722+0000 7fd45ed9d700 1 --2- 172.21.15.1:0/4157064575 >> v2:172.21.15.165:3300/0 conn(0x7fd46006a750 0x7fd46006ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.722+0000 7fd45e59c700 1 -- 172.21.15.1:0/4157064575 >> v2:172.21.15.165:3300/0 conn(0x7fd46006a750 msgr2=0x7fd46006ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:45.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.722+0000 7fd45e59c700 1 --2- 172.21.15.1:0/4157064575 >> v2:172.21.15.165:3300/0 conn(0x7fd46006a750 0x7fd46006ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.722+0000 7fd45e59c700 1 -- 172.21.15.1:0/4157064575 >> v1:172.21.15.165:6789/0 conn(0x7fd46015eca0 legacy=0x7fd4600cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:45.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.722+0000 7fd45e59c700 1 -- 172.21.15.1:0/4157064575 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46015f210 con 0x7fd46015b0a0 2024-03-18T20:26:45.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.722+0000 7fd45ed9d700 1 --2- 172.21.15.1:0/4157064575 >> v2:172.21.15.165:3300/0 conn(0x7fd46006a750 0x7fd46006ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:45.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.722+0000 7fd45e59c700 1 --2- 172.21.15.1:0/4157064575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd46015b0a0 0x7fd4600acd80 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7fd460072930 tx=0x7fd450009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e5b98e825c38dd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:45.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.723+0000 7fd45d59a700 1 -- 172.21.15.1:0/4157064575 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45000b870 con 0x7fd46015b0a0 2024-03-18T20:26:45.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.723+0000 7fd45d59a700 1 -- 172.21.15.1:0/4157064575 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45000b9d0 con 0x7fd46015b0a0 2024-03-18T20:26:45.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.723+0000 7fd45d59a700 1 -- 172.21.15.1:0/4157064575 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45000bd60 con 0x7fd46015b0a0 2024-03-18T20:26:45.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.723+0000 7fd45f59e700 1 -- 172.21.15.1:0/4157064575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd46015b0a0 msgr2=0x7fd4600acd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.723+0000 7fd45f59e700 1 --2- 172.21.15.1:0/4157064575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd46015b0a0 0x7fd4600acd80 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7fd460072930 tx=0x7fd450009bd0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.723+0000 7fd45f59e700 1 -- 172.21.15.1:0/4157064575 shutdown_connections 2024-03-18T20:26:45.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.723+0000 7fd45f59e700 1 --2- 172.21.15.1:0/4157064575 >> v2:172.21.15.165:3300/0 conn(0x7fd46006a750 0x7fd46006ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.723+0000 7fd45f59e700 1 --2- 172.21.15.1:0/4157064575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd46015b0a0 0x7fd4600acd80 unknown :-1 s=CLOSED pgs=2888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.723+0000 7fd45f59e700 1 -- 172.21.15.1:0/4157064575 >> 172.21.15.1:0/4157064575 conn(0x7fd4600b1aa0 msgr2=0x7fd4600b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:45.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.723+0000 7fd45f59e700 1 -- 172.21.15.1:0/4157064575 shutdown_connections 2024-03-18T20:26:45.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.723+0000 7fd45f59e700 1 -- 172.21.15.1:0/4157064575 wait complete. 2024-03-18T20:26:45.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.725+0000 7fd45f59e700 1 Processor -- start 2024-03-18T20:26:45.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.725+0000 7fd45f59e700 1 -- start start 2024-03-18T20:26:45.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.725+0000 7fd45f59e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd46006a750 0x7fd46009c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.725+0000 7fd45f59e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd46015b0a0 0x7fd460099d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.725+0000 7fd45f59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd46015eca0 0x7fd46015cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.725+0000 7fd45f59e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46009d910 con 0x7fd46015eca0 2024-03-18T20:26:45.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.725+0000 7fd45f59e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46009a2d0 con 0x7fd46015b0a0 2024-03-18T20:26:45.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.725+0000 7fd45f59e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46009cc90 con 0x7fd46006a750 2024-03-18T20:26:45.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.725+0000 7fd45ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd46015eca0 0x7fd46015cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.725+0000 7fd45ed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd46015eca0 0x7fd46015cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46410/0 (socket says 172.21.15.1:46410) 2024-03-18T20:26:45.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.725+0000 7fd45ed9d700 1 -- 172.21.15.1:0/224638484 learned_addr learned my addr 172.21.15.1:0/224638484 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:45.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.726+0000 7fd45dd9b700 1 --2- 172.21.15.1:0/224638484 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd46015b0a0 0x7fd460099d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.726+0000 7fd45e59c700 1 --2- 172.21.15.1:0/224638484 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd46006a750 0x7fd46009c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.726+0000 7fd45ed9d700 1 -- 172.21.15.1:0/224638484 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd46006a750 msgr2=0x7fd46009c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.726+0000 7fd45ed9d700 1 --2- 172.21.15.1:0/224638484 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd46006a750 0x7fd46009c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.726+0000 7fd45ed9d700 1 -- 172.21.15.1:0/224638484 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd46015b0a0 msgr2=0x7fd460099d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:45.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.726+0000 7fd45ed9d700 1 --2- 172.21.15.1:0/224638484 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd46015b0a0 0x7fd460099d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:45.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.726+0000 7fd45ed9d700 1 -- 172.21.15.1:0/224638484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd450009550 con 0x7fd46015eca0 2024-03-18T20:26:45.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.726+0000 7fd45dd9b700 1 --2- 172.21.15.1:0/224638484 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd46015b0a0 0x7fd460099d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:45.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.726+0000 7fd45ed9d700 1 --2- 172.21.15.1:0/224638484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd46015eca0 0x7fd46015cec0 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7fd458009fa0 tx=0x7fd45800d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:45.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.726+0000 7fd44effd700 1 -- 172.21.15.1:0/224638484 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd458003a20 con 0x7fd46015eca0 2024-03-18T20:26:45.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.726+0000 7fd44effd700 1 -- 172.21.15.1:0/224638484 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45800f040 con 0x7fd46015eca0 2024-03-18T20:26:45.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.726+0000 7fd45f59e700 1 -- 172.21.15.1:0/224638484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd46009cf70 con 0x7fd46015eca0 2024-03-18T20:26:45.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.726+0000 7fd44effd700 1 -- 172.21.15.1:0/224638484 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd458003de0 con 0x7fd46015eca0 2024-03-18T20:26:45.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.726+0000 7fd45f59e700 1 -- 172.21.15.1:0/224638484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd46009d530 con 0x7fd46015eca0 2024-03-18T20:26:45.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.727+0000 7fd45f59e700 1 -- 172.21.15.1:0/224638484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4600747a0 con 0x7fd46015eca0 2024-03-18T20:26:45.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.729+0000 7fd44effd700 1 -- 172.21.15.1:0/224638484 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 146) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fd458004030 con 0x7fd46015eca0 2024-03-18T20:26:45.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.729+0000 7fd44effd700 1 --2- 172.21.15.1:0/224638484 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd44006aed0 0x7fd44006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:45.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.729+0000 7fd44effd700 1 -- 172.21.15.1:0/224638484 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6993+0+0 (secure 0 0 0) 0x7fd45808a3c0 con 0x7fd46015eca0 2024-03-18T20:26:45.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.729+0000 7fd45e59c700 1 --2- 172.21.15.1:0/224638484 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd44006aed0 0x7fd44006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:45.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.729+0000 7fd45e59c700 1 --2- 172.21.15.1:0/224638484 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd44006aed0 0x7fd44006d390 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd450009c10 tx=0x7fd450008030 comp rx=0 tx=0).ready entity=mgr.12057 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:45.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.733+0000 7fd44effd700 1 -- 172.21.15.1:0/224638484 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd458054c90 con 0x7fd46015eca0 2024-03-18T20:26:45.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:45.993+0000 7fd45f59e700 1 -- 172.21.15.1:0/224638484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fd460071ce0 con 0x7fd46015eca0 2024-03-18T20:26:46.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.639+0000 7fd44effd700 1 -- 172.21.15.1:0/224638484 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 147) v1 ==== 89358+0+0 (secure 0 0 0) 0x7fd458055060 con 0x7fd46015eca0 2024-03-18T20:26:46.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.640+0000 7fd44effd700 1 -- 172.21.15.1:0/224638484 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v147) v1 ==== 92+0+0 (secure 0 0 0) 0x7fd458075970 con 0x7fd46015eca0 2024-03-18T20:26:46.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.643+0000 7fd45f59e700 1 -- 172.21.15.1:0/224638484 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd44006aed0 msgr2=0x7fd44006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:46.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.643+0000 7fd45f59e700 1 --2- 172.21.15.1:0/224638484 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd44006aed0 0x7fd44006d390 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd450009c10 tx=0x7fd450008030 comp rx=0 tx=0).stop 2024-03-18T20:26:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.643+0000 7fd45f59e700 1 -- 172.21.15.1:0/224638484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd46015eca0 msgr2=0x7fd46015cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.643+0000 7fd45f59e700 1 --2- 172.21.15.1:0/224638484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd46015eca0 0x7fd46015cec0 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7fd458009fa0 tx=0x7fd45800d460 comp rx=0 tx=0).stop 2024-03-18T20:26:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.643+0000 7fd45f59e700 1 -- 172.21.15.1:0/224638484 shutdown_connections 2024-03-18T20:26:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.643+0000 7fd45f59e700 1 --2- 172.21.15.1:0/224638484 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd44006aed0 0x7fd44006d390 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.643+0000 7fd45f59e700 1 --2- 172.21.15.1:0/224638484 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd46006a750 0x7fd46009c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.643+0000 7fd45f59e700 1 --2- 172.21.15.1:0/224638484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd46015eca0 0x7fd46015cec0 unknown :-1 s=CLOSED pgs=2889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.643+0000 7fd45f59e700 1 --2- 172.21.15.1:0/224638484 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd46015b0a0 0x7fd460099d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.643+0000 7fd45f59e700 1 -- 172.21.15.1:0/224638484 >> 172.21.15.1:0/224638484 conn(0x7fd4600b1aa0 msgr2=0x7fd4600b0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.643+0000 7fd45f59e700 1 -- 172.21.15.1:0/224638484 shutdown_connections 2024-03-18T20:26:46.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.643+0000 7fd45f59e700 1 -- 172.21.15.1:0/224638484 wait complete. 2024-03-18T20:26:46.663 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-18T20:26:46.786 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:46.781+0000 7ff5851bc240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:26:46.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.796+0000 7f59ee76b700 1 Processor -- start 2024-03-18T20:26:46.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.796+0000 7f59ee76b700 1 -- start start 2024-03-18T20:26:46.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.797+0000 7f59ee76b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f59e806ae60 0x7f59e80cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:46.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.797+0000 7f59ee76b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e815c500 con 0x7f59e8071500 2024-03-18T20:26:46.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.797+0000 7f59ee76b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e80af8b0 con 0x7f59e8158e30 2024-03-18T20:26:46.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.797+0000 7f59ee76b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e8071980 con 0x7f59e806ae60 2024-03-18T20:26:46.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.797+0000 7f59e77fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f59e8071500 0x7f59e806a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58416/0 (socket says 172.21.15.1:58416) 2024-03-18T20:26:46.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.797+0000 7f59e77fe700 1 -- 172.21.15.1:0/2813786042 learned_addr learned my addr 172.21.15.1:0/2813786042 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:46.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.797+0000 7f59ecd08700 1 --2- 172.21.15.1:0/2813786042 >> v2:172.21.15.165:3300/0 conn(0x7f59e806ae60 0x7f59e80cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:46.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.797+0000 7f59e6ffd700 1 -- 172.21.15.1:0/2813786042 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077726533 0 0) 0x7f59e815c500 con 0x7f59e8071500 2024-03-18T20:26:46.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.797+0000 7f59e6ffd700 1 -- 172.21.15.1:0/2813786042 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d40035b0 con 0x7f59e8071500 2024-03-18T20:26:46.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.797+0000 7f59ecd08700 1 -- 172.21.15.1:0/2813786042 >> v1:172.21.15.165:6789/0 conn(0x7f59e8158e30 legacy=0x7f59e80cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:46.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.797+0000 7f59ecd08700 1 -- 172.21.15.1:0/2813786042 >> v1:172.21.15.1:6789/0 conn(0x7f59e8071500 legacy=0x7f59e806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:46.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.797+0000 7f59e6ffd700 1 -- 172.21.15.1:0/2813786042 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493069181 0 0) 0x7f59e80af8b0 con 0x7f59e8158e30 2024-03-18T20:26:46.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.797+0000 7f59ecd08700 1 -- 172.21.15.1:0/2813786042 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e815b8f0 con 0x7f59e806ae60 2024-03-18T20:26:46.803 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:46.798+0000 7fbeecf4f240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:26:46.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.798+0000 7f59ecd08700 1 --2- 172.21.15.1:0/2813786042 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59e806ae60 0x7f59e80cf150 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f59e0009d90 tx=0x7f59e000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fe9399291e24679 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:46.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.798+0000 7f59e6ffd700 1 -- 172.21.15.1:0/2813786042 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e000e070 con 0x7f59e806ae60 2024-03-18T20:26:46.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.798+0000 7f59e6ffd700 1 -- 172.21.15.1:0/2813786042 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e000bb20 con 0x7f59e806ae60 2024-03-18T20:26:46.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.798+0000 7f59e6ffd700 1 -- 172.21.15.1:0/2813786042 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e000bed0 con 0x7f59e806ae60 2024-03-18T20:26:46.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.798+0000 7f59ee76b700 1 -- 172.21.15.1:0/2813786042 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59e806ae60 msgr2=0x7f59e80cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:46.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.798+0000 7f59ee76b700 1 --2- 172.21.15.1:0/2813786042 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59e806ae60 0x7f59e80cf150 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f59e0009d90 tx=0x7f59e000b330 comp rx=0 tx=0).stop 2024-03-18T20:26:46.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.798+0000 7f59ee76b700 1 -- 172.21.15.1:0/2813786042 shutdown_connections 2024-03-18T20:26:46.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.798+0000 7f59ee76b700 1 --2- 172.21.15.1:0/2813786042 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59e806ae60 0x7f59e80cf150 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:46.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.798+0000 7f59ee76b700 1 -- 172.21.15.1:0/2813786042 >> 172.21.15.1:0/2813786042 conn(0x7f59e80cca00 msgr2=0x7f59e80cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:46.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.799+0000 7f59ee76b700 1 -- 172.21.15.1:0/2813786042 shutdown_connections 2024-03-18T20:26:46.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.799+0000 7f59ee76b700 1 -- 172.21.15.1:0/2813786042 wait complete. 2024-03-18T20:26:46.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.800+0000 7f59ee76b700 1 Processor -- start 2024-03-18T20:26:46.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.800+0000 7f59ee76b700 1 -- start start 2024-03-18T20:26:46.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.801+0000 7f59ee76b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59e806ae60 0x7f59e81515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:46.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.801+0000 7f59ee76b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59e8071500 0x7f59e806bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:46.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.801+0000 7f59ee76b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59e8158e30 0x7f59e806c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:46.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.801+0000 7f59ee76b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e806f410 con 0x7f59e8071500 2024-03-18T20:26:46.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.801+0000 7f59ee76b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e80acf90 con 0x7f59e8158e30 2024-03-18T20:26:46.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.801+0000 7f59ee76b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e80ad230 con 0x7f59e806ae60 2024-03-18T20:26:46.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.801+0000 7f59e77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59e8071500 0x7f59e806bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:46.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.801+0000 7f59e77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59e8071500 0x7f59e806bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46424/0 (socket says 172.21.15.1:46424) 2024-03-18T20:26:46.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.801+0000 7f59e77fe700 1 -- 172.21.15.1:0/3262494665 learned_addr learned my addr 172.21.15.1:0/3262494665 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:46.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.801+0000 7f59e7fff700 1 --2- 172.21.15.1:0/3262494665 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59e806ae60 0x7f59e81515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:46.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.801+0000 7f59ecd08700 1 --2- 172.21.15.1:0/3262494665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59e8158e30 0x7f59e806c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:46.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.801+0000 7f59e77fe700 1 -- 172.21.15.1:0/3262494665 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59e806ae60 msgr2=0x7f59e81515c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:46.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.801+0000 7f59e77fe700 1 --2- 172.21.15.1:0/3262494665 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59e806ae60 0x7f59e81515c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:46.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.801+0000 7f59e77fe700 1 -- 172.21.15.1:0/3262494665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59e8158e30 msgr2=0x7f59e806c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:46.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.801+0000 7f59e77fe700 1 --2- 172.21.15.1:0/3262494665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59e8158e30 0x7f59e806c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:46.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.801+0000 7f59e77fe700 1 -- 172.21.15.1:0/3262494665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e0009a70 con 0x7f59e8071500 2024-03-18T20:26:46.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.802+0000 7f59e77fe700 1 --2- 172.21.15.1:0/3262494665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59e8071500 0x7f59e806bb50 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7f59d000c4d0 tx=0x7f59d000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:46.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.802+0000 7f59e4ff9700 1 -- 172.21.15.1:0/3262494665 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d0005d70 con 0x7f59e8071500 2024-03-18T20:26:46.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.802+0000 7f59e4ff9700 1 -- 172.21.15.1:0/3262494665 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d00070d0 con 0x7f59e8071500 2024-03-18T20:26:46.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.802+0000 7f59e4ff9700 1 -- 172.21.15.1:0/3262494665 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d0007440 con 0x7f59e8071500 2024-03-18T20:26:46.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.802+0000 7f59ee76b700 1 -- 172.21.15.1:0/3262494665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e80ad510 con 0x7f59e8071500 2024-03-18T20:26:46.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.802+0000 7f59ee76b700 1 -- 172.21.15.1:0/3262494665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e8151b00 con 0x7f59e8071500 2024-03-18T20:26:46.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.803+0000 7f59ee76b700 1 -- 172.21.15.1:0/3262494665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e80ada30 con 0x7f59e8071500 2024-03-18T20:26:46.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.804+0000 7f59e4ff9700 1 -- 172.21.15.1:0/3262494665 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 147) v1 ==== 89358+0+0 (secure 0 0 0) 0x7f59d00075a0 con 0x7f59e8071500 2024-03-18T20:26:46.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.804+0000 7f59e4ff9700 1 --2- 172.21.15.1:0/3262494665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f59c806af20 0x7f59c806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:46.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.804+0000 7f59e7fff700 1 -- 172.21.15.1:0/3262494665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f59c806af20 msgr2=0x7f59c806d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:46.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.804+0000 7f59e4ff9700 1 -- 172.21.15.1:0/3262494665 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f59d0089000 con 0x7f59e8071500 2024-03-18T20:26:46.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.804+0000 7f59e7fff700 1 --2- 172.21.15.1:0/3262494665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f59c806af20 0x7f59c806d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:46.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:46.809+0000 7f59e4ff9700 1 -- 172.21.15.1:0/3262494665 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f59e80ada30 con 0x7f59e8071500 2024-03-18T20:26:46.933 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:46.928+0000 7ff5851bc240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:26:46.973 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:46.969+0000 7fbeecf4f240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:26:47.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.005+0000 7f59e7fff700 1 -- 172.21.15.1:0/3262494665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f59c806af20 msgr2=0x7f59c806d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:47.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.005+0000 7f59e7fff700 1 --2- 172.21.15.1:0/3262494665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f59c806af20 0x7f59c806d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:47.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.060+0000 7f59ee76b700 1 -- 172.21.15.1:0/3262494665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f59e80ada30 con 0x7f59e8071500 2024-03-18T20:26:47.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.064+0000 7f59e4ff9700 1 -- 172.21.15.1:0/3262494665 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v83)=0 v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7f59e80ada30 con 0x7f59e8071500 2024-03-18T20:26:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.067+0000 7f59ee76b700 1 -- 172.21.15.1:0/3262494665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f59c806af20 msgr2=0x7f59c806d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.067+0000 7f59ee76b700 1 --2- 172.21.15.1:0/3262494665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f59c806af20 0x7f59c806d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.067+0000 7f59ee76b700 1 -- 172.21.15.1:0/3262494665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59e8071500 msgr2=0x7f59e806bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.067+0000 7f59ee76b700 1 --2- 172.21.15.1:0/3262494665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59e8071500 0x7f59e806bb50 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7f59d000c4d0 tx=0x7f59d000c7e0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.067+0000 7f59ee76b700 1 -- 172.21.15.1:0/3262494665 shutdown_connections 2024-03-18T20:26:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.067+0000 7f59ee76b700 1 --2- 172.21.15.1:0/3262494665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f59c806af20 0x7f59c806d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.067+0000 7f59ee76b700 1 --2- 172.21.15.1:0/3262494665 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59e806ae60 0x7f59e81515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.067+0000 7f59ee76b700 1 --2- 172.21.15.1:0/3262494665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59e8158e30 0x7f59e806c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.067+0000 7f59ee76b700 1 --2- 172.21.15.1:0/3262494665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59e8071500 0x7f59e806bb50 unknown :-1 s=CLOSED pgs=2893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.067+0000 7f59ee76b700 1 -- 172.21.15.1:0/3262494665 >> 172.21.15.1:0/3262494665 conn(0x7f59e80cca00 msgr2=0x7f59e80ce1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:47.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.067+0000 7f59ee76b700 1 -- 172.21.15.1:0/3262494665 shutdown_connections 2024-03-18T20:26:47.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.067+0000 7f59ee76b700 1 -- 172.21.15.1:0/3262494665 wait complete. 2024-03-18T20:26:47.083 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-18T20:26:47.200 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:47.195+0000 7ff5851bc240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:26:47.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.210+0000 7fcae1aa8700 1 Processor -- start 2024-03-18T20:26:47.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.210+0000 7fcae1aa8700 1 -- start start 2024-03-18T20:26:47.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.211+0000 7fcae1aa8700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcadc160a70 0x7fcadc0d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:47.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.211+0000 7fcae1aa8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcadc160e50 0x7fcadc0cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:47.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.211+0000 7fcae1aa8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcadc1612d0 0x7fcadc161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:47.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.211+0000 7fcae1aa8700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcadc162170 con 0x7fcadc1612d0 2024-03-18T20:26:47.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.211+0000 7fcae1aa8700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcadc161c60 con 0x7fcadc160a70 2024-03-18T20:26:47.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.211+0000 7fcae1aa8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcadc0cef30 con 0x7fcadc160e50 2024-03-18T20:26:47.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.211+0000 7fcadbfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcadc1612d0 0x7fcadc161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:47.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.211+0000 7fcadbfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcadc1612d0 0x7fcadc161720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46436/0 (socket says 172.21.15.1:46436) 2024-03-18T20:26:47.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.211+0000 7fcadbfff700 1 -- 172.21.15.1:0/3308867627 learned_addr learned my addr 172.21.15.1:0/3308867627 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:47.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.211+0000 7fcad2ffd700 1 --2- 172.21.15.1:0/3308867627 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcadc160e50 0x7fcadc0cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:47.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.211+0000 7fcadb7fe700 1 --2- 172.21.15.1:0/3308867627 >> v2:172.21.15.165:3300/0 conn(0x7fcadc160a70 0x7fcadc0d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:47.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.211+0000 7fcadbfff700 1 -- 172.21.15.1:0/3308867627 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcadc160e50 msgr2=0x7fcadc0cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:47.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.211+0000 7fcadbfff700 1 --2- 172.21.15.1:0/3308867627 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcadc160e50 0x7fcadc0cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.211+0000 7fcadbfff700 1 -- 172.21.15.1:0/3308867627 >> v2:172.21.15.165:3300/0 conn(0x7fcadc160a70 msgr2=0x7fcadc0d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:47.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.211+0000 7fcadbfff700 1 --2- 172.21.15.1:0/3308867627 >> v2:172.21.15.165:3300/0 conn(0x7fcadc160a70 0x7fcadc0d3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.211+0000 7fcadbfff700 1 -- 172.21.15.1:0/3308867627 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcadc0cf210 con 0x7fcadc1612d0 2024-03-18T20:26:47.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.211+0000 7fcadbfff700 1 --2- 172.21.15.1:0/3308867627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcadc1612d0 0x7fcadc161720 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7fcadc08d8f0 tx=0x7fcad400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3c435053ddb979e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:47.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.212+0000 7fcadaffd700 1 -- 172.21.15.1:0/3308867627 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad400e070 con 0x7fcadc1612d0 2024-03-18T20:26:47.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.212+0000 7fcadaffd700 1 -- 172.21.15.1:0/3308867627 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad400b910 con 0x7fcadc1612d0 2024-03-18T20:26:47.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.212+0000 7fcadaffd700 1 -- 172.21.15.1:0/3308867627 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad400bcc0 con 0x7fcadc1612d0 2024-03-18T20:26:47.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.212+0000 7fcae1aa8700 1 -- 172.21.15.1:0/3308867627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcadc1612d0 msgr2=0x7fcadc161720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:47.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.212+0000 7fcae1aa8700 1 --2- 172.21.15.1:0/3308867627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcadc1612d0 0x7fcadc161720 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7fcadc08d8f0 tx=0x7fcad400b330 comp rx=0 tx=0).stop 2024-03-18T20:26:47.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.212+0000 7fcae1aa8700 1 -- 172.21.15.1:0/3308867627 shutdown_connections 2024-03-18T20:26:47.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.212+0000 7fcae1aa8700 1 --2- 172.21.15.1:0/3308867627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcadc1612d0 0x7fcadc161720 unknown :-1 s=CLOSED pgs=2894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.212+0000 7fcae1aa8700 1 --2- 172.21.15.1:0/3308867627 >> v2:172.21.15.165:3300/0 conn(0x7fcadc160a70 0x7fcadc0d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.212+0000 7fcae1aa8700 1 --2- 172.21.15.1:0/3308867627 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcadc160e50 0x7fcadc0cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.212+0000 7fcae1aa8700 1 -- 172.21.15.1:0/3308867627 >> 172.21.15.1:0/3308867627 conn(0x7fcadc09c560 msgr2=0x7fcadc0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:47.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.212+0000 7fcae1aa8700 1 -- 172.21.15.1:0/3308867627 shutdown_connections 2024-03-18T20:26:47.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.212+0000 7fcae1aa8700 1 -- 172.21.15.1:0/3308867627 wait complete. 2024-03-18T20:26:47.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.213+0000 7fcae1aa8700 1 Processor -- start 2024-03-18T20:26:47.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.214+0000 7fcae1aa8700 1 -- start start 2024-03-18T20:26:47.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.214+0000 7fcae1aa8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcadc160a70 0x7fcadc06bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:47.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.214+0000 7fcae1aa8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcadc160e50 0x7fcadc0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.214+0000 7fcae1aa8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcadc1612d0 0x7fcadc0a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.214+0000 7fcae1aa8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcadc0ca320 con 0x7fcadc160a70 2024-03-18T20:26:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.214+0000 7fcae1aa8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcadc0ca570 con 0x7fcadc160e50 2024-03-18T20:26:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.214+0000 7fcae1aa8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcadc0ca850 con 0x7fcadc1612d0 2024-03-18T20:26:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.214+0000 7fcadb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcadc160a70 0x7fcadc06bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.214+0000 7fcadb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcadc160a70 0x7fcadc06bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46448/0 (socket says 172.21.15.1:46448) 2024-03-18T20:26:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.214+0000 7fcadb7fe700 1 -- 172.21.15.1:0/2385452261 learned_addr learned my addr 172.21.15.1:0/2385452261 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.215+0000 7fcad2ffd700 1 --2- 172.21.15.1:0/2385452261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcadc160e50 0x7fcadc0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.215+0000 7fcadbfff700 1 --2- 172.21.15.1:0/2385452261 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcadc1612d0 0x7fcadc0a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.215+0000 7fcadb7fe700 1 -- 172.21.15.1:0/2385452261 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcadc1612d0 msgr2=0x7fcadc0a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.215+0000 7fcadb7fe700 1 --2- 172.21.15.1:0/2385452261 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcadc1612d0 0x7fcadc0a0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.215+0000 7fcadb7fe700 1 -- 172.21.15.1:0/2385452261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcadc160e50 msgr2=0x7fcadc0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:47.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.215+0000 7fcadb7fe700 1 --2- 172.21.15.1:0/2385452261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcadc160e50 0x7fcadc0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.215+0000 7fcadb7fe700 1 -- 172.21.15.1:0/2385452261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad4009b30 con 0x7fcadc160a70 2024-03-18T20:26:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.215+0000 7fcadb7fe700 1 --2- 172.21.15.1:0/2385452261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcadc160a70 0x7fcadc06bfe0 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7fcacc00e870 tx=0x7fcacc00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.215+0000 7fcad8ff9700 1 -- 172.21.15.1:0/2385452261 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc013680 con 0x7fcadc160a70 2024-03-18T20:26:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.215+0000 7fcad8ff9700 1 -- 172.21.15.1:0/2385452261 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcacc0137e0 con 0x7fcadc160a70 2024-03-18T20:26:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.215+0000 7fcae1aa8700 1 -- 172.21.15.1:0/2385452261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcadc0cab30 con 0x7fcadc160a70 2024-03-18T20:26:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.215+0000 7fcad8ff9700 1 -- 172.21.15.1:0/2385452261 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc013b90 con 0x7fcadc160a70 2024-03-18T20:26:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.215+0000 7fcae1aa8700 1 -- 172.21.15.1:0/2385452261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcadc0cb0f0 con 0x7fcadc160a70 2024-03-18T20:26:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.216+0000 7fcae1aa8700 1 -- 172.21.15.1:0/2385452261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcadc067390 con 0x7fcadc160a70 2024-03-18T20:26:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.217+0000 7fcad8ff9700 1 -- 172.21.15.1:0/2385452261 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 147) v1 ==== 89358+0+0 (secure 0 0 0) 0x7fcacc013cf0 con 0x7fcadc160a70 2024-03-18T20:26:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.217+0000 7fcad8ff9700 1 --2- 172.21.15.1:0/2385452261 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcabc06af20 0x7fcabc06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.217+0000 7fcad2ffd700 1 -- 172.21.15.1:0/2385452261 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcabc06af20 msgr2=0x7fcabc06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.217+0000 7fcad8ff9700 1 -- 172.21.15.1:0/2385452261 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6993+0+0 (secure 0 0 0) 0x7fcacc089b80 con 0x7fcadc160a70 2024-03-18T20:26:47.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.217+0000 7fcad2ffd700 1 --2- 172.21.15.1:0/2385452261 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcabc06af20 0x7fcabc06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:47.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.222+0000 7fcad8ff9700 1 -- 172.21.15.1:0/2385452261 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcacc0544f0 con 0x7fcadc160a70 2024-03-18T20:26:47.259 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:47.255+0000 7fbeecf4f240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:26:47.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.418+0000 7fcad2ffd700 1 -- 172.21.15.1:0/2385452261 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcabc06af20 msgr2=0x7fcabc06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:47.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.418+0000 7fcad2ffd700 1 --2- 172.21.15.1:0/2385452261 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcabc06af20 0x7fcabc06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:47.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.425+0000 7fcae1aa8700 1 -- 172.21.15.1:0/2385452261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fcadc0676a0 con 0x7fcadc160a70 2024-03-18T20:26:47.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.429+0000 7fcad8ff9700 1 -- 172.21.15.1:0/2385452261 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v84)=0 v84) v1 ==== 133+0+0 (secure 0 0 0) 0x7fcacc057b10 con 0x7fcadc160a70 2024-03-18T20:26:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.431+0000 7fcae1aa8700 1 -- 172.21.15.1:0/2385452261 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcabc06af20 msgr2=0x7fcabc06d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.431+0000 7fcae1aa8700 1 --2- 172.21.15.1:0/2385452261 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcabc06af20 0x7fcabc06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.431+0000 7fcae1aa8700 1 -- 172.21.15.1:0/2385452261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcadc160a70 msgr2=0x7fcadc06bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.431+0000 7fcae1aa8700 1 --2- 172.21.15.1:0/2385452261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcadc160a70 0x7fcadc06bfe0 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7fcacc00e870 tx=0x7fcacc00eb80 comp rx=0 tx=0).stop 2024-03-18T20:26:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.431+0000 7fcae1aa8700 1 -- 172.21.15.1:0/2385452261 shutdown_connections 2024-03-18T20:26:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.431+0000 7fcae1aa8700 1 --2- 172.21.15.1:0/2385452261 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcabc06af20 0x7fcabc06d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.431+0000 7fcae1aa8700 1 --2- 172.21.15.1:0/2385452261 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcadc1612d0 0x7fcadc0a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.431+0000 7fcae1aa8700 1 --2- 172.21.15.1:0/2385452261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcadc160a70 0x7fcadc06bfe0 unknown :-1 s=CLOSED pgs=2895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.431+0000 7fcae1aa8700 1 --2- 172.21.15.1:0/2385452261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcadc160e50 0x7fcadc0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.431+0000 7fcae1aa8700 1 -- 172.21.15.1:0/2385452261 >> 172.21.15.1:0/2385452261 conn(0x7fcadc09c560 msgr2=0x7fcadc15f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:47.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.432+0000 7fcae1aa8700 1 -- 172.21.15.1:0/2385452261 shutdown_connections 2024-03-18T20:26:47.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.432+0000 7fcae1aa8700 1 -- 172.21.15.1:0/2385452261 wait complete. 2024-03-18T20:26:47.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:47.452 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2024-03-18T20:26:47.474 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir 1M 2024-03-18T20:26:47.631 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:47.627+0000 7ff5851bc240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:26:47.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.645+0000 7f55f521f700 1 Processor -- start 2024-03-18T20:26:47.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.645+0000 7f55f521f700 1 -- start start 2024-03-18T20:26:47.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.646+0000 7f55f521f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f55f006ef00 0x7f55f015b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:47.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.646+0000 7f55f521f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55f006d220 con 0x7f55f0151150 2024-03-18T20:26:47.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.646+0000 7f55f521f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55f006d500 con 0x7f55f006e2c0 2024-03-18T20:26:47.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.646+0000 7f55f521f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55f015b670 con 0x7f55f006ef00 2024-03-18T20:26:47.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.646+0000 7f55eed9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f55f0151150 0x7f55f006e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58432/0 (socket says 172.21.15.1:58432) 2024-03-18T20:26:47.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.646+0000 7f55eed9d700 1 -- 172.21.15.1:0/798161745 learned_addr learned my addr 172.21.15.1:0/798161745 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:47.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.646+0000 7f55ee59c700 1 --2- 172.21.15.1:0/798161745 >> v2:172.21.15.165:3300/0 conn(0x7f55f006ef00 0x7f55f015b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:47.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.646+0000 7f55edd9b700 1 -- 172.21.15.1:0/798161745 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564814286 0 0) 0x7f55f006d220 con 0x7f55f0151150 2024-03-18T20:26:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.646+0000 7f55edd9b700 1 -- 172.21.15.1:0/798161745 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55d80057c0 con 0x7f55f0151150 2024-03-18T20:26:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.647+0000 7f55ee59c700 1 -- 172.21.15.1:0/798161745 >> v1:172.21.15.165:6789/0 conn(0x7f55f006e2c0 legacy=0x7f55f00b2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.647+0000 7f55ee59c700 1 -- 172.21.15.1:0/798161745 >> v1:172.21.15.1:6789/0 conn(0x7f55f0151150 legacy=0x7f55f006e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.647+0000 7f55ee59c700 1 -- 172.21.15.1:0/798161745 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f006d7e0 con 0x7f55f006ef00 2024-03-18T20:26:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.647+0000 7f55ee59c700 1 --2- 172.21.15.1:0/798161745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f55f006ef00 0x7f55f015b130 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f55e0009840 tx=0x7f55e0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5417dda405525a43 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.647+0000 7f55edd9b700 1 -- 172.21.15.1:0/798161745 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e0011830 con 0x7f55f006ef00 2024-03-18T20:26:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.647+0000 7f55edd9b700 1 -- 172.21.15.1:0/798161745 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e0011990 con 0x7f55f006ef00 2024-03-18T20:26:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.647+0000 7f55edd9b700 1 -- 172.21.15.1:0/798161745 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e0011d20 con 0x7f55f006ef00 2024-03-18T20:26:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.647+0000 7f55f521f700 1 -- 172.21.15.1:0/798161745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f55f006ef00 msgr2=0x7f55f015b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.647+0000 7f55f521f700 1 --2- 172.21.15.1:0/798161745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f55f006ef00 0x7f55f015b130 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f55e0009840 tx=0x7f55e0009b50 comp rx=0 tx=0).stop 2024-03-18T20:26:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.648+0000 7f55f521f700 1 -- 172.21.15.1:0/798161745 shutdown_connections 2024-03-18T20:26:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.648+0000 7f55f521f700 1 --2- 172.21.15.1:0/798161745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f55f006ef00 0x7f55f015b130 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.648+0000 7f55f521f700 1 -- 172.21.15.1:0/798161745 >> 172.21.15.1:0/798161745 conn(0x7f55f00b1350 msgr2=0x7f55f00b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:47.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.648+0000 7f55f521f700 1 -- 172.21.15.1:0/798161745 shutdown_connections 2024-03-18T20:26:47.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.648+0000 7f55f521f700 1 -- 172.21.15.1:0/798161745 wait complete. 2024-03-18T20:26:47.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.649+0000 7f55f521f700 1 Processor -- start 2024-03-18T20:26:47.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.649+0000 7f55f521f700 1 -- start start 2024-03-18T20:26:47.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.650+0000 7f55f521f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f55f006e2c0 0x7f55f01520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:47.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.650+0000 7f55f521f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f55f006ef00 0x7f55f015baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:47.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.650+0000 7f55f521f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55f0151150 0x7f55f01f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:47.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.650+0000 7f55f521f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f0153620 con 0x7f55f0151150 2024-03-18T20:26:47.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.650+0000 7f55f521f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f0152a10 con 0x7f55f006ef00 2024-03-18T20:26:47.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.650+0000 7f55f521f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f0152cf0 con 0x7f55f006e2c0 2024-03-18T20:26:47.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.650+0000 7f55ef59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55f0151150 0x7f55f01f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:47.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.650+0000 7f55ef59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55f0151150 0x7f55f01f59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46458/0 (socket says 172.21.15.1:46458) 2024-03-18T20:26:47.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.650+0000 7f55ef59e700 1 -- 172.21.15.1:0/2355405449 learned_addr learned my addr 172.21.15.1:0/2355405449 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:47.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.650+0000 7f55ee59c700 1 --2- 172.21.15.1:0/2355405449 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f55f006ef00 0x7f55f015baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:47.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.650+0000 7f55eed9d700 1 --2- 172.21.15.1:0/2355405449 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f55f006e2c0 0x7f55f01520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:47.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.650+0000 7f55ef59e700 1 -- 172.21.15.1:0/2355405449 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f55f006e2c0 msgr2=0x7f55f01520c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:47.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.650+0000 7f55ef59e700 1 --2- 172.21.15.1:0/2355405449 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f55f006e2c0 0x7f55f01520c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.650+0000 7f55ef59e700 1 -- 172.21.15.1:0/2355405449 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f55f006ef00 msgr2=0x7f55f015baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:47.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.650+0000 7f55ef59e700 1 --2- 172.21.15.1:0/2355405449 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f55f006ef00 0x7f55f015baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:47.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.650+0000 7f55ef59e700 1 -- 172.21.15.1:0/2355405449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e00094a0 con 0x7f55f0151150 2024-03-18T20:26:47.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.651+0000 7f55ef59e700 1 --2- 172.21.15.1:0/2355405449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55f0151150 0x7f55f01f59b0 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f55e800cb50 tx=0x7f55e800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:47.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.651+0000 7f55d77fe700 1 -- 172.21.15.1:0/2355405449 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e8003c00 con 0x7f55f0151150 2024-03-18T20:26:47.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.651+0000 7f55d77fe700 1 -- 172.21.15.1:0/2355405449 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e8003d60 con 0x7f55f0151150 2024-03-18T20:26:47.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.651+0000 7f55f521f700 1 -- 172.21.15.1:0/2355405449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f0152fd0 con 0x7f55f0151150 2024-03-18T20:26:47.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.651+0000 7f55d77fe700 1 -- 172.21.15.1:0/2355405449 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e800aa20 con 0x7f55f0151150 2024-03-18T20:26:47.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.651+0000 7f55f521f700 1 -- 172.21.15.1:0/2355405449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f0152600 con 0x7f55f0151150 2024-03-18T20:26:47.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.652+0000 7f55f521f700 1 -- 172.21.15.1:0/2355405449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f0152790 con 0x7f55f0151150 2024-03-18T20:26:47.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.652+0000 7f55d77fe700 1 -- 172.21.15.1:0/2355405449 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 147) v1 ==== 89358+0+0 (secure 0 0 0) 0x7f55e800ab80 con 0x7f55f0151150 2024-03-18T20:26:47.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.653+0000 7f55d77fe700 1 --2- 172.21.15.1:0/2355405449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f55dc06af20 0x7f55dc06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:47.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.653+0000 7f55eed9d700 1 -- 172.21.15.1:0/2355405449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f55dc06af20 msgr2=0x7f55dc06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:47.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.653+0000 7f55d77fe700 1 -- 172.21.15.1:0/2355405449 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 6993+0+0 (secure 0 0 0) 0x7f55e8089c00 con 0x7f55f0151150 2024-03-18T20:26:47.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.653+0000 7f55eed9d700 1 --2- 172.21.15.1:0/2355405449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f55dc06af20 0x7f55dc06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:47.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.657+0000 7f55d77fe700 1 -- 172.21.15.1:0/2355405449 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f55e8090080 con 0x7f55f0151150 2024-03-18T20:26:47.725 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:47.721+0000 7fbeecf4f240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:26:47.843 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:47.839+0000 7ff5851bc240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:26:47.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.854+0000 7f55eed9d700 1 -- 172.21.15.1:0/2355405449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f55dc06af20 msgr2=0x7f55dc06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:47.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.854+0000 7f55eed9d700 1 --2- 172.21.15.1:0/2355405449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f55dc06af20 0x7f55dc06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:47.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:47.886+0000 7f55f521f700 1 -- 172.21.15.1:0/2355405449 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f55f0151530 con 0x7f55dc06af20 2024-03-18T20:26:47.943 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:47.939+0000 7fbeecf4f240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:26:48.216 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:48.212+0000 7ff5851bc240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:26:48.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:48.255+0000 7f55eed9d700 1 -- 172.21.15.1:0/2355405449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f55dc06af20 msgr2=0x7f55dc06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:48.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:48.255+0000 7f55eed9d700 1 --2- 172.21.15.1:0/2355405449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f55dc06af20 0x7f55dc06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T20:26:48.331 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:48.327+0000 7fbeecf4f240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:26:48.800 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:48.795+0000 7ff5851bc240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:26:48.871 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:48.867+0000 7ff5851bc240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:26:48.947 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:48.943+0000 7fbeecf4f240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:26:48.997 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:48.993+0000 7ff5851bc240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:26:49.024 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:49.020+0000 7fbeecf4f240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:26:49.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:49.056+0000 7f55eed9d700 1 -- 172.21.15.1:0/2355405449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f55dc06af20 msgr2=0x7f55dc06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:49.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:49.056+0000 7f55eed9d700 1 --2- 172.21.15.1:0/2355405449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f55dc06af20 0x7f55dc06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T20:26:49.158 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:49.154+0000 7fbeecf4f240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:26:49.175 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:49.171+0000 7ff5851bc240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:26:49.247 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:49.243+0000 7ff5851bc240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:26:49.325 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:49.321+0000 7ff5851bc240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:26:49.344 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:49.340+0000 7fbeecf4f240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:26:49.405 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:49.400+0000 7ff5851bc240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:26:49.421 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:49.417+0000 7fbeecf4f240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:26:49.483 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:49.478+0000 7ff5851bc240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:26:49.504 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:49.500+0000 7fbeecf4f240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:26:49.585 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:49.581+0000 7fbeecf4f240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:26:49.650 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:49.646+0000 7ff5851bc240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:26:49.666 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:49.662+0000 7fbeecf4f240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:26:49.847 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:49.843+0000 7fbeecf4f240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:26:49.866 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:49.862+0000 7ff5851bc240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:26:50.017 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:50.012+0000 7ff5851bc240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:26:50.070 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:50.066+0000 7fbeecf4f240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:26:50.089 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:50.085+0000 7ff5851bc240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:26:50.229 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:50.225+0000 7fbeecf4f240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:26:50.305 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:50.302+0000 7fbeecf4f240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:26:50.538 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:50.534+0000 7ff5851bc240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:26:50.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:50.657+0000 7f55eed9d700 1 -- 172.21.15.1:0/2355405449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f55dc06af20 msgr2=0x7f55dc06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:50.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:50.657+0000 7f55eed9d700 1 --2- 172.21.15.1:0/2355405449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f55dc06af20 0x7f55dc06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T20:26:50.778 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:50.774+0000 7fbeecf4f240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:26:50.875 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:50.870+0000 7ff5851bc240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:26:51.039 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:51.034+0000 7ff5851bc240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:26:51.132 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:51.128+0000 7fbeecf4f240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:26:51.305 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:51.301+0000 7fbeecf4f240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:26:51.426 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:51.422+0000 7ff5851bc240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:26:51.592 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:51.588+0000 7ff5851bc240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:26:51.719 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:51.715+0000 7fbeecf4f240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:26:51.895 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:51.891+0000 7fbeecf4f240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:26:51.954 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:51.949+0000 7ff5851bc240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:26:52.042 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:52.038+0000 7ff5851bc240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:26:52.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:52.109+0000 7f55d77fe700 1 -- 172.21.15.1:0/2355405449 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 148) v1 ==== 89358+0+0 (secure 0 0 0) 0x7f55e8054a10 con 0x7f55f0151150 2024-03-18T20:26:52.276 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.272+0000 7fbeecf4f240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:26:52.370 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.366+0000 7fbeecf4f240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:26:52.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:52.387+0000 7f55d77fe700 1 -- 172.21.15.1:0/2355405449 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 149) v1 ==== 88865+0+0 (secure 0 0 0) 0x7f55e805ee60 con 0x7f55f0151150 2024-03-18T20:26:52.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:52.387+0000 7f55d77fe700 1 -- 172.21.15.1:0/2355405449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f55dc06af20 msgr2=0x7f55dc06d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:52.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:52.387+0000 7f55d77fe700 1 --2- 172.21.15.1:0/2355405449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f55dc06af20 0x7f55dc06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:52.605 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.601+0000 7fbe5eb6a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:52.605 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.601+0000 7fbe5eb6a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:52.606 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.601+0000 7fbe5eb6a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:52.606 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.601+0000 7fbe5eb6a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:52.606 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.601+0000 7fbe5eb6a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:52.606 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.602+0000 7fbe62371700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:52.606 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.602+0000 7fbe62371700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:52.606 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.602+0000 7fbe62371700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:52.607 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.602+0000 7fbe62371700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:52.607 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.602+0000 7fbe62371700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:52.615 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.611+0000 7fbe5eb6a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:52.616 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.611+0000 7fbe5eb6a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:52.616 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.611+0000 7fbe5eb6a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:52.616 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.611+0000 7fbe5eb6a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:52.616 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.611+0000 7fbe5eb6a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:52.616 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.611+0000 7fbe62371700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:52.616 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.611+0000 7fbe62371700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:52.616 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.611+0000 7fbe62371700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:52.616 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.611+0000 7fbe62371700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:52.616 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:52.611+0000 7fbe62371700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:26:53.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.393+0000 7f55d77fe700 1 -- 172.21.15.1:0/2355405449 <== mon.0 v2:172.21.15.1:3300/0 9 ==== mgrmap(e 150) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f55e805bdd0 con 0x7f55f0151150 2024-03-18T20:26:53.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.393+0000 7f55d77fe700 1 --2- 172.21.15.1:0/2355405449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f55dc06af20 0x7f55dc06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:53.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.393+0000 7f55d77fe700 1 -- 172.21.15.1:0/2355405449 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f55dc01cd00 con 0x7f55dc06af20 2024-03-18T20:26:53.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.397+0000 7f55eed9d700 1 --2- 172.21.15.1:0/2355405449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f55dc06af20 0x7f55dc06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:53.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.399+0000 7f55eed9d700 1 --2- 172.21.15.1:0/2355405449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f55dc06af20 0x7f55dc06d3e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f55e4007fd0 tx=0x7f55e4007dc0 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:53.405 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:53.401+0000 7fbe6937f700 -1 mgr.server reply reply (22) Invalid argument filesystem argument is required when there is more than one file system 2024-03-18T20:26:53.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.401+0000 7f55d77fe700 1 -- 172.21.15.1:0/2355405449 <== mgr.9641 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: -22 filesystem argument is required when there is more than one file system) v1 ==== 79+0+0 (secure 0 0 0) 0x7f55dc01cd00 con 0x7f55dc06af20 2024-03-18T20:26:53.407 INFO:teuthology.orchestra.run.smithi001.stderr:Error EINVAL: filesystem argument is required when there is more than one file system 2024-03-18T20:26:53.411 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-18T20:26:53.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:53.412 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2024-03-18T20:26:53.437 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-18T20:26:53.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.608+0000 7f8ba479a700 1 Processor -- start 2024-03-18T20:26:53.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.608+0000 7f8ba479a700 1 -- start start 2024-03-18T20:26:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.609+0000 7f8ba479a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8b9c0714e0 0x7f8b9c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.609+0000 7f8ba479a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8b9c06e310 0x7f8b9c15bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.609+0000 7f8ba479a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b9c0aedf0 con 0x7f8b9c15a270 2024-03-18T20:26:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.609+0000 7f8ba479a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b9c15a940 con 0x7f8b9c0714e0 2024-03-18T20:26:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.609+0000 7f8ba479a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b9c15ac20 con 0x7f8b9c06e310 2024-03-18T20:26:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.610+0000 7f8ba1d35700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8b9c0714e0 0x7f8b9c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.610+0000 7f8ba1d35700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8b9c0714e0 0x7f8b9c15e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46604/0 (socket says 172.21.15.1:46604) 2024-03-18T20:26:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.610+0000 7f8ba1d35700 1 -- 172.21.15.1:0/2046170828 learned_addr learned my addr 172.21.15.1:0/2046170828 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.610+0000 7f8ba2d37700 1 --2- 172.21.15.1:0/2046170828 >> v2:172.21.15.165:3300/0 conn(0x7f8b9c06e310 0x7f8b9c15bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:53.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.610+0000 7f8ba1d35700 1 -- 172.21.15.1:0/2046170828 >> v2:172.21.15.165:3300/0 conn(0x7f8b9c06e310 msgr2=0x7f8b9c15bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:53.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.610+0000 7f8ba1d35700 1 --2- 172.21.15.1:0/2046170828 >> v2:172.21.15.165:3300/0 conn(0x7f8b9c06e310 0x7f8b9c15bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:53.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.610+0000 7f8ba1d35700 1 -- 172.21.15.1:0/2046170828 >> v1:172.21.15.165:6789/0 conn(0x7f8b9c15a270 legacy=0x7f8b9c070dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:53.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.610+0000 7f8ba1d35700 1 -- 172.21.15.1:0/2046170828 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9c15af00 con 0x7f8b9c0714e0 2024-03-18T20:26:53.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.610+0000 7f8ba1d35700 1 --2- 172.21.15.1:0/2046170828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b9c0714e0 0x7f8b9c15e680 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c15dbb0 tx=0x7f8b9800be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d11cefd4411a77ea server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:53.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.610+0000 7f8ba1534700 1 -- 172.21.15.1:0/2046170828 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9800dcf0 con 0x7f8b9c0714e0 2024-03-18T20:26:53.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.610+0000 7f8ba1534700 1 -- 172.21.15.1:0/2046170828 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9800de50 con 0x7f8b9c0714e0 2024-03-18T20:26:53.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.610+0000 7f8ba1534700 1 -- 172.21.15.1:0/2046170828 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b98004250 con 0x7f8b9c0714e0 2024-03-18T20:26:53.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.611+0000 7f8ba479a700 1 -- 172.21.15.1:0/2046170828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b9c0714e0 msgr2=0x7f8b9c15e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:53.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.611+0000 7f8ba479a700 1 --2- 172.21.15.1:0/2046170828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b9c0714e0 0x7f8b9c15e680 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c15dbb0 tx=0x7f8b9800be70 comp rx=0 tx=0).stop 2024-03-18T20:26:53.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.611+0000 7f8ba479a700 1 -- 172.21.15.1:0/2046170828 shutdown_connections 2024-03-18T20:26:53.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.611+0000 7f8ba479a700 1 --2- 172.21.15.1:0/2046170828 >> v2:172.21.15.165:3300/0 conn(0x7f8b9c06e310 0x7f8b9c15bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:53.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.611+0000 7f8ba479a700 1 --2- 172.21.15.1:0/2046170828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b9c0714e0 0x7f8b9c15e680 unknown :-1 s=CLOSED pgs=2910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:53.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.611+0000 7f8ba479a700 1 -- 172.21.15.1:0/2046170828 >> 172.21.15.1:0/2046170828 conn(0x7f8b9c066b70 msgr2=0x7f8b9c06a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:53.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.611+0000 7f8ba479a700 1 -- 172.21.15.1:0/2046170828 shutdown_connections 2024-03-18T20:26:53.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.611+0000 7f8ba479a700 1 -- 172.21.15.1:0/2046170828 wait complete. 2024-03-18T20:26:53.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.612+0000 7f8ba479a700 1 Processor -- start 2024-03-18T20:26:53.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.612+0000 7f8ba479a700 1 -- start start 2024-03-18T20:26:53.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.613+0000 7f8ba479a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b9c06e310 0x7f8b9c0a87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:53.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.613+0000 7f8ba479a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8b9c0714e0 0x7f8b9c155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:53.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.613+0000 7f8ba479a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8b9c15a270 0x7f8b9c1e3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:53.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.613+0000 7f8ba479a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c1e3fd0 con 0x7f8b9c06e310 2024-03-18T20:26:53.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.613+0000 7f8ba479a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c1ef4d0 con 0x7f8b9c0714e0 2024-03-18T20:26:53.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.613+0000 7f8ba479a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c1e4aa0 con 0x7f8b9c15a270 2024-03-18T20:26:53.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.613+0000 7f8ba2536700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b9c06e310 0x7f8b9c0a87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:53.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.613+0000 7f8ba2536700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b9c06e310 0x7f8b9c0a87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46606/0 (socket says 172.21.15.1:46606) 2024-03-18T20:26:53.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.613+0000 7f8ba2536700 1 -- 172.21.15.1:0/2579052047 learned_addr learned my addr 172.21.15.1:0/2579052047 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:53.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.613+0000 7f8ba1d35700 1 --2- 172.21.15.1:0/2579052047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8b9c0714e0 0x7f8b9c155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:53.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.613+0000 7f8ba2536700 1 -- 172.21.15.1:0/2579052047 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8b9c15a270 msgr2=0x7f8b9c1e3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:53.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.613+0000 7f8ba2d37700 1 --2- 172.21.15.1:0/2579052047 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8b9c15a270 0x7f8b9c1e3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:53.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.613+0000 7f8ba2536700 1 --2- 172.21.15.1:0/2579052047 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8b9c15a270 0x7f8b9c1e3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:53.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.613+0000 7f8ba2536700 1 -- 172.21.15.1:0/2579052047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8b9c0714e0 msgr2=0x7f8b9c155020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:53.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.613+0000 7f8ba2536700 1 --2- 172.21.15.1:0/2579052047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8b9c0714e0 0x7f8b9c155020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:53.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.613+0000 7f8ba2536700 1 -- 172.21.15.1:0/2579052047 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9800b7f0 con 0x7f8b9c06e310 2024-03-18T20:26:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.613+0000 7f8ba1d35700 1 --2- 172.21.15.1:0/2579052047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8b9c0714e0 0x7f8b9c155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:26:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.613+0000 7f8ba2536700 1 --2- 172.21.15.1:0/2579052047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b9c06e310 0x7f8b9c0a87f0 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000b470 tx=0x7f8b9000b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.613+0000 7f8b8effd700 1 -- 172.21.15.1:0/2579052047 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b90011830 con 0x7f8b9c06e310 2024-03-18T20:26:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.614+0000 7f8b8effd700 1 -- 172.21.15.1:0/2579052047 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b90011990 con 0x7f8b9c06e310 2024-03-18T20:26:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.614+0000 7f8ba479a700 1 -- 172.21.15.1:0/2579052047 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b9c1e4200 con 0x7f8b9c06e310 2024-03-18T20:26:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.614+0000 7f8b8effd700 1 -- 172.21.15.1:0/2579052047 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b90011d40 con 0x7f8b9c06e310 2024-03-18T20:26:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.614+0000 7f8ba479a700 1 -- 172.21.15.1:0/2579052047 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b9c1e47c0 con 0x7f8b9c06e310 2024-03-18T20:26:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.615+0000 7f8ba479a700 1 -- 172.21.15.1:0/2579052047 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b9c0b4a30 con 0x7f8b9c06e310 2024-03-18T20:26:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.615+0000 7f8b8effd700 1 -- 172.21.15.1:0/2579052047 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 150) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f8b9001a400 con 0x7f8b9c06e310 2024-03-18T20:26:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.615+0000 7f8b8effd700 1 --2- 172.21.15.1:0/2579052047 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8b8006ab30 0x7f8b8006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.616+0000 7f8b8effd700 1 -- 172.21.15.1:0/2579052047 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7423+0+0 (secure 0 0 0) 0x7f8b900896c0 con 0x7f8b9c06e310 2024-03-18T20:26:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.616+0000 7f8ba1d35700 1 --2- 172.21.15.1:0/2579052047 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8b8006ab30 0x7f8b8006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:53.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.616+0000 7f8ba1d35700 1 --2- 172.21.15.1:0/2579052047 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8b8006ab30 0x7f8b8006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c0b4e50 tx=0x7f8b98012000 comp rx=0 tx=0).ready entity=mgr.9641 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:53.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.620+0000 7f8b8effd700 1 -- 172.21.15.1:0/2579052047 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8b90054530 con 0x7f8b9c06e310 2024-03-18T20:26:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:53.884+0000 7f8ba479a700 1 -- 172.21.15.1:0/2579052047 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f8b9c073350 con 0x7f8b9c06e310 2024-03-18T20:26:54.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.400+0000 7f8b8effd700 1 -- 172.21.15.1:0/2579052047 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 151) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f8b90054900 con 0x7f8b9c06e310 2024-03-18T20:26:54.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.402+0000 7f8b8effd700 1 -- 172.21.15.1:0/2579052047 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v151) v1 ==== 93+0+0 (secure 0 0 0) 0x7f8b90087540 con 0x7f8b9c06e310 2024-03-18T20:26:54.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.406+0000 7f8ba479a700 1 -- 172.21.15.1:0/2579052047 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8b8006ab30 msgr2=0x7f8b8006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:54.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.406+0000 7f8ba479a700 1 --2- 172.21.15.1:0/2579052047 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8b8006ab30 0x7f8b8006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c0b4e50 tx=0x7f8b98012000 comp rx=0 tx=0).stop 2024-03-18T20:26:54.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.406+0000 7f8ba479a700 1 -- 172.21.15.1:0/2579052047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b9c06e310 msgr2=0x7f8b9c0a87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:54.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.406+0000 7f8ba479a700 1 --2- 172.21.15.1:0/2579052047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b9c06e310 0x7f8b9c0a87f0 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000b470 tx=0x7f8b9000b830 comp rx=0 tx=0).stop 2024-03-18T20:26:54.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.406+0000 7f8ba479a700 1 -- 172.21.15.1:0/2579052047 shutdown_connections 2024-03-18T20:26:54.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.406+0000 7f8ba479a700 1 --2- 172.21.15.1:0/2579052047 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8b8006ab30 0x7f8b8006cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.406+0000 7f8ba479a700 1 --2- 172.21.15.1:0/2579052047 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8b9c15a270 0x7f8b9c1e3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.406+0000 7f8ba479a700 1 --2- 172.21.15.1:0/2579052047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8b9c06e310 0x7f8b9c0a87f0 unknown :-1 s=CLOSED pgs=2911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.406+0000 7f8ba479a700 1 --2- 172.21.15.1:0/2579052047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8b9c0714e0 0x7f8b9c155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.406+0000 7f8ba479a700 1 -- 172.21.15.1:0/2579052047 >> 172.21.15.1:0/2579052047 conn(0x7f8b9c066b70 msgr2=0x7f8b9c06a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.406+0000 7f8ba479a700 1 -- 172.21.15.1:0/2579052047 shutdown_connections 2024-03-18T20:26:54.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.406+0000 7f8ba479a700 1 -- 172.21.15.1:0/2579052047 wait complete. 2024-03-18T20:26:54.430 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T20:26:54.431 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:26:54.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:26:54.516 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T20:26:54.516 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:26:54.516 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:26:54.535 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:26:54.554 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:54.549+0000 7f76988f6240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:26:54.566 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:54.562+0000 7f5294a3a240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:26:54.698 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:54.694+0000 7f76988f6240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:26:54.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.731+0000 7f82f88dc700 1 Processor -- start 2024-03-18T20:26:54.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.732+0000 7f82f88dc700 1 -- start start 2024-03-18T20:26:54.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.732+0000 7f82f88dc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82f415a320 0x7f82f40ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:54.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.732+0000 7f82f88dc700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f40acd80 con 0x7f82f4071d10 2024-03-18T20:26:54.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.732+0000 7f82f88dc700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f40cf220 con 0x7f82f40cea30 2024-03-18T20:26:54.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.732+0000 7f82f88dc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40cef40 con 0x7f82f415a320 2024-03-18T20:26:54.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.733+0000 7f82f2d9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f82f4071d10 0x7f82f406e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44982/0 (socket says 172.21.15.1:44982) 2024-03-18T20:26:54.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.733+0000 7f82f2d9d700 1 -- 172.21.15.1:0/3376538922 learned_addr learned my addr 172.21.15.1:0/3376538922 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:54.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.733+0000 7f82f259c700 1 --2- 172.21.15.1:0/3376538922 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82f415a320 0x7f82f40ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:54.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.733+0000 7f82f159a700 1 -- 172.21.15.1:0/3376538922 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493637586 0 0) 0x7f82f40acd80 con 0x7f82f4071d10 2024-03-18T20:26:54.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.733+0000 7f82f159a700 1 -- 172.21.15.1:0/3376538922 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82dc0035b0 con 0x7f82f4071d10 2024-03-18T20:26:54.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.733+0000 7f82f159a700 1 -- 172.21.15.1:0/3376538922 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44417535 0 0) 0x7f82e8001110 con 0x7f82f40cea30 2024-03-18T20:26:54.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.733+0000 7f82f159a700 1 -- 172.21.15.1:0/3376538922 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82f40acd80 con 0x7f82f40cea30 2024-03-18T20:26:54.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.733+0000 7f82f259c700 1 -- 172.21.15.1:0/3376538922 >> v1:172.21.15.165:6789/0 conn(0x7f82f40cea30 legacy=0x7f82f4071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:54.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.733+0000 7f82f259c700 1 -- 172.21.15.1:0/3376538922 >> v1:172.21.15.1:6789/0 conn(0x7f82f4071d10 legacy=0x7f82f406e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:54.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.733+0000 7f82f259c700 1 -- 172.21.15.1:0/3376538922 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f406d7e0 con 0x7f82f415a320 2024-03-18T20:26:54.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.733+0000 7f82f259c700 1 --2- 172.21.15.1:0/3376538922 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82f415a320 0x7f82f40ce4f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f82e4009870 tx=0x7f82e4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6383d13eb82f25e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:54.739 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:54.733+0000 7f5294a3a240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:26:54.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.734+0000 7f82f159a700 1 -- 172.21.15.1:0/3376538922 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e4004230 con 0x7f82f415a320 2024-03-18T20:26:54.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.734+0000 7f82f159a700 1 -- 172.21.15.1:0/3376538922 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e4004390 con 0x7f82f415a320 2024-03-18T20:26:54.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.734+0000 7f82f159a700 1 -- 172.21.15.1:0/3376538922 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e40036a0 con 0x7f82f415a320 2024-03-18T20:26:54.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.734+0000 7f82f88dc700 1 -- 172.21.15.1:0/3376538922 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82f415a320 msgr2=0x7f82f40ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:54.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.734+0000 7f82f88dc700 1 --2- 172.21.15.1:0/3376538922 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82f415a320 0x7f82f40ce4f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f82e4009870 tx=0x7f82e4009b80 comp rx=0 tx=0).stop 2024-03-18T20:26:54.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.734+0000 7f82f88dc700 1 -- 172.21.15.1:0/3376538922 shutdown_connections 2024-03-18T20:26:54.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.734+0000 7f82f88dc700 1 --2- 172.21.15.1:0/3376538922 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82f415a320 0x7f82f40ce4f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:54.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.734+0000 7f82f88dc700 1 -- 172.21.15.1:0/3376538922 >> 172.21.15.1:0/3376538922 conn(0x7f82f40cc990 msgr2=0x7f82f40ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:54.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.734+0000 7f82f88dc700 1 -- 172.21.15.1:0/3376538922 shutdown_connections 2024-03-18T20:26:54.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.735+0000 7f82f88dc700 1 -- 172.21.15.1:0/3376538922 wait complete. 2024-03-18T20:26:54.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.736+0000 7f82f88dc700 1 Processor -- start 2024-03-18T20:26:54.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.736+0000 7f82f88dc700 1 -- start start 2024-03-18T20:26:54.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82f88dc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82f4071d10 0x7f82f4154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:54.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82f88dc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82f40cea30 0x7f82f406e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:54.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82f88dc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82f406eca0 0x7f82f40b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:54.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82f88dc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f406f150 con 0x7f82f40cea30 2024-03-18T20:26:54.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82f88dc700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40c3b60 con 0x7f82f406eca0 2024-03-18T20:26:54.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82f88dc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f4154ac0 con 0x7f82f4071d10 2024-03-18T20:26:54.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82f1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82f40cea30 0x7f82f406e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:54.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82f259c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82f4071d10 0x7f82f4154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:54.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82f1d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82f40cea30 0x7f82f406e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34670/0 (socket says 172.21.15.1:34670) 2024-03-18T20:26:54.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82f1d9b700 1 -- 172.21.15.1:0/1611088680 learned_addr learned my addr 172.21.15.1:0/1611088680 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:54.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82f2d9d700 1 --2- 172.21.15.1:0/1611088680 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82f406eca0 0x7f82f40b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:54.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82f1d9b700 1 -- 172.21.15.1:0/1611088680 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82f4071d10 msgr2=0x7f82f4154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:54.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82f1d9b700 1 --2- 172.21.15.1:0/1611088680 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82f4071d10 0x7f82f4154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:54.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82f1d9b700 1 -- 172.21.15.1:0/1611088680 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82f406eca0 msgr2=0x7f82f40b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82f1d9b700 1 --2- 172.21.15.1:0/1611088680 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82f406eca0 0x7f82f40b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82f1d9b700 1 -- 172.21.15.1:0/1611088680 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e40094a0 con 0x7f82f40cea30 2024-03-18T20:26:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82f1d9b700 1 --2- 172.21.15.1:0/1611088680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82f40cea30 0x7f82f406e760 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7f82e800c7b0 tx=0x7f82e800cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82e2ffd700 1 -- 172.21.15.1:0/1611088680 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e80054e0 con 0x7f82f40cea30 2024-03-18T20:26:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82e2ffd700 1 -- 172.21.15.1:0/1611088680 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e8003680 con 0x7f82f40cea30 2024-03-18T20:26:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82f88dc700 1 -- 172.21.15.1:0/1611088680 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f40b5130 con 0x7f82f40cea30 2024-03-18T20:26:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.737+0000 7f82e2ffd700 1 -- 172.21.15.1:0/1611088680 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e8003a30 con 0x7f82f40cea30 2024-03-18T20:26:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.738+0000 7f82f88dc700 1 -- 172.21.15.1:0/1611088680 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f40b56f0 con 0x7f82f40cea30 2024-03-18T20:26:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.739+0000 7f82f88dc700 1 -- 172.21.15.1:0/1611088680 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f40c3d10 con 0x7f82f40cea30 2024-03-18T20:26:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.739+0000 7f82e2ffd700 1 -- 172.21.15.1:0/1611088680 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 151) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f82e8003b90 con 0x7f82f40cea30 2024-03-18T20:26:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.739+0000 7f82e2ffd700 1 --2- 172.21.15.1:0/1611088680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f82d406aae0 0x7f82d406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.739+0000 7f82f259c700 1 -- 172.21.15.1:0/1611088680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f82d406aae0 msgr2=0x7f82d406cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.739+0000 7f82e2ffd700 1 -- 172.21.15.1:0/1611088680 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7423+0+0 (secure 0 0 0) 0x7f82e8088760 con 0x7f82f40cea30 2024-03-18T20:26:54.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.739+0000 7f82f259c700 1 --2- 172.21.15.1:0/1611088680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f82d406aae0 0x7f82d406cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:54.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.744+0000 7f82e2ffd700 1 -- 172.21.15.1:0/1611088680 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f82f40c3d10 con 0x7f82f40cea30 2024-03-18T20:26:54.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.941+0000 7f82f259c700 1 -- 172.21.15.1:0/1611088680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f82d406aae0 msgr2=0x7f82d406cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:54.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.941+0000 7f82f259c700 1 --2- 172.21.15.1:0/1611088680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f82d406aae0 0x7f82d406cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:54.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.949+0000 7f82f88dc700 1 -- 172.21.15.1:0/1611088680 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82f40cf220 con 0x7f82f40cea30 2024-03-18T20:26:54.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.950+0000 7f82e2ffd700 1 -- 172.21.15.1:0/1611088680 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v258) v1 ==== 74+0+20693 (secure 0 0 0) 0x7f82f40cf220 con 0x7f82f40cea30 2024-03-18T20:26:54.955 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:54.955 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":258,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:26:52.380634+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:26:23.615691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-18T20:26:24.759650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:26:37.326244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:26:38.564628+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"257","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2340206734":"2024-03-19T20:26:52.380559+0000","172.21.15.1:6833/148356":"2024-03-19T20:26:52.380559+0000","172.21.15.1:6832/148356":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/376843885":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/1340983360":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/369631788":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/2840085801":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/322251557":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/3049322266":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/1886004750":"2024-03-19T20:26:52.380559+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:26:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.953+0000 7f82f88dc700 1 -- 172.21.15.1:0/1611088680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f82d406aae0 msgr2=0x7f82d406cfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.953+0000 7f82f88dc700 1 --2- 172.21.15.1:0/1611088680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f82d406aae0 0x7f82d406cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.953+0000 7f82f88dc700 1 -- 172.21.15.1:0/1611088680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82f40cea30 msgr2=0x7f82f406e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.953+0000 7f82f88dc700 1 --2- 172.21.15.1:0/1611088680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82f40cea30 0x7f82f406e760 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7f82e800c7b0 tx=0x7f82e800cac0 comp rx=0 tx=0).stop 2024-03-18T20:26:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.953+0000 7f82f88dc700 1 -- 172.21.15.1:0/1611088680 shutdown_connections 2024-03-18T20:26:54.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.953+0000 7f82f88dc700 1 --2- 172.21.15.1:0/1611088680 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f82d406aae0 0x7f82d406cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:54.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.953+0000 7f82f88dc700 1 --2- 172.21.15.1:0/1611088680 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f82f4071d10 0x7f82f4154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:54.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.953+0000 7f82f88dc700 1 --2- 172.21.15.1:0/1611088680 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f82f406eca0 0x7f82f40b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:54.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.953+0000 7f82f88dc700 1 --2- 172.21.15.1:0/1611088680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82f40cea30 0x7f82f406e760 unknown :-1 s=CLOSED pgs=2916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:54.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.953+0000 7f82f88dc700 1 -- 172.21.15.1:0/1611088680 >> 172.21.15.1:0/1611088680 conn(0x7f82f40cc990 msgr2=0x7f82f40b0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:54.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.953+0000 7f82f88dc700 1 -- 172.21.15.1:0/1611088680 shutdown_connections 2024-03-18T20:26:54.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:54.953+0000 7f82f88dc700 1 -- 172.21.15.1:0/1611088680 wait complete. 2024-03-18T20:26:54.964 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:54.960+0000 7f76988f6240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:26:54.970 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:26:23.615691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '252', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:26:24.759650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '253', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-03-18T20:26:37.326244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '256', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2024-03-18T20:26:38.564628+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '257', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:26:54.970 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:26:55.019 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:55.015+0000 7f5294a3a240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:26:55.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.097+0000 7f18388fa700 1 Processor -- start 2024-03-18T20:26:55.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.097+0000 7f18388fa700 1 -- start start 2024-03-18T20:26:55.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.098+0000 7f18388fa700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f183415d580 0x7f183406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:55.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.098+0000 7f18388fa700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f183406f4f0 con 0x7f183406d380 2024-03-18T20:26:55.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.098+0000 7f18388fa700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f183406ab30 con 0x7f18340724b0 2024-03-18T20:26:55.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.098+0000 7f18388fa700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183415d960 con 0x7f183415d580 2024-03-18T20:26:55.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.099+0000 7f1831d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f183406d380 0x7f1834071da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44996/0 (socket says 172.21.15.1:44996) 2024-03-18T20:26:55.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.099+0000 7f1831d9b700 1 -- 172.21.15.1:0/2413740074 learned_addr learned my addr 172.21.15.1:0/2413740074 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:55.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.099+0000 7f183259c700 1 --2- 172.21.15.1:0/2413740074 >> v2:172.21.15.165:3300/0 conn(0x7f183415d580 0x7f183406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:55.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.099+0000 7f183159a700 1 -- 172.21.15.1:0/2413740074 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530993623 0 0) 0x7f1828001110 con 0x7f183406d380 2024-03-18T20:26:55.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.099+0000 7f183159a700 1 -- 172.21.15.1:0/2413740074 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f181c0035b0 con 0x7f183406d380 2024-03-18T20:26:55.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.099+0000 7f183159a700 1 -- 172.21.15.1:0/2413740074 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693010437 0 0) 0x7f183406ab30 con 0x7f18340724b0 2024-03-18T20:26:55.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.099+0000 7f183159a700 1 -- 172.21.15.1:0/2413740074 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1828001110 con 0x7f18340724b0 2024-03-18T20:26:55.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.099+0000 7f183159a700 1 -- 172.21.15.1:0/2413740074 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1844367413 0 0) 0x7f181c0035b0 con 0x7f183406d380 2024-03-18T20:26:55.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.099+0000 7f183259c700 1 -- 172.21.15.1:0/2413740074 >> v1:172.21.15.165:6789/0 conn(0x7f18340724b0 legacy=0x7f18340cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.099+0000 7f183259c700 1 -- 172.21.15.1:0/2413740074 >> v1:172.21.15.1:6789/0 conn(0x7f183406d380 legacy=0x7f1834071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.099+0000 7f183259c700 1 -- 172.21.15.1:0/2413740074 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183406ae10 con 0x7f183415d580 2024-03-18T20:26:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.099+0000 7f183259c700 1 --2- 172.21.15.1:0/2413740074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f183415d580 0x7f183406efb0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f1824009840 tx=0x7f1824009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ccbde8b65d17f14 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.100+0000 7f183159a700 1 -- 172.21.15.1:0/2413740074 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1828001800 con 0x7f183415d580 2024-03-18T20:26:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.100+0000 7f183159a700 1 -- 172.21.15.1:0/2413740074 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1824004030 con 0x7f183415d580 2024-03-18T20:26:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.100+0000 7f183159a700 1 -- 172.21.15.1:0/2413740074 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18240043c0 con 0x7f183415d580 2024-03-18T20:26:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.100+0000 7f18388fa700 1 -- 172.21.15.1:0/2413740074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f183415d580 msgr2=0x7f183406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.100+0000 7f18388fa700 1 --2- 172.21.15.1:0/2413740074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f183415d580 0x7f183406efb0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f1824009840 tx=0x7f1824009b50 comp rx=0 tx=0).stop 2024-03-18T20:26:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.100+0000 7f18388fa700 1 -- 172.21.15.1:0/2413740074 shutdown_connections 2024-03-18T20:26:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.100+0000 7f18388fa700 1 --2- 172.21.15.1:0/2413740074 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f183415d580 0x7f183406efb0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.100+0000 7f18388fa700 1 -- 172.21.15.1:0/2413740074 >> 172.21.15.1:0/2413740074 conn(0x7f18340cca00 msgr2=0x7f18340cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:55.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.100+0000 7f18388fa700 1 -- 172.21.15.1:0/2413740074 shutdown_connections 2024-03-18T20:26:55.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.100+0000 7f18388fa700 1 -- 172.21.15.1:0/2413740074 wait complete. 2024-03-18T20:26:55.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.102+0000 7f18388fa700 1 Processor -- start 2024-03-18T20:26:55.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.102+0000 7f18388fa700 1 -- start start 2024-03-18T20:26:55.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.102+0000 7f18388fa700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f183406d380 0x7f183415b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:55.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.102+0000 7f18388fa700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18340724b0 0x7f18340b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:55.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.102+0000 7f18388fa700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f183415d580 0x7f18340b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:55.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.102+0000 7f18388fa700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18340b3460 con 0x7f18340724b0 2024-03-18T20:26:55.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.102+0000 7f18388fa700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18340b4260 con 0x7f183406d380 2024-03-18T20:26:55.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.102+0000 7f18388fa700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f183415c340 con 0x7f183415d580 2024-03-18T20:26:55.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.103+0000 7f1831d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18340724b0 0x7f18340b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:55.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.103+0000 7f1831d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18340724b0 0x7f18340b4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34680/0 (socket says 172.21.15.1:34680) 2024-03-18T20:26:55.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.103+0000 7f1831d9b700 1 -- 172.21.15.1:0/1570581150 learned_addr learned my addr 172.21.15.1:0/1570581150 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:55.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.103+0000 7f183259c700 1 --2- 172.21.15.1:0/1570581150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f183406d380 0x7f183415b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:55.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.103+0000 7f1832d9d700 1 --2- 172.21.15.1:0/1570581150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f183415d580 0x7f18340b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.103+0000 7f1831d9b700 1 -- 172.21.15.1:0/1570581150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f183415d580 msgr2=0x7f18340b5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.103+0000 7f1831d9b700 1 --2- 172.21.15.1:0/1570581150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f183415d580 0x7f18340b5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.103+0000 7f1831d9b700 1 -- 172.21.15.1:0/1570581150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f183406d380 msgr2=0x7f183415b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.103+0000 7f1831d9b700 1 --2- 172.21.15.1:0/1570581150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f183406d380 0x7f183415b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.103+0000 7f1831d9b700 1 -- 172.21.15.1:0/1570581150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18240094a0 con 0x7f18340724b0 2024-03-18T20:26:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.103+0000 7f1831d9b700 1 --2- 172.21.15.1:0/1570581150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18340724b0 0x7f18340b4b50 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f182800c780 tx=0x7f182800ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.103+0000 7f1822ffd700 1 -- 172.21.15.1:0/1570581150 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1828004c80 con 0x7f18340724b0 2024-03-18T20:26:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.103+0000 7f1822ffd700 1 -- 172.21.15.1:0/1570581150 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1828004e00 con 0x7f18340724b0 2024-03-18T20:26:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.103+0000 7f18388fa700 1 -- 172.21.15.1:0/1570581150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18340b3650 con 0x7f18340724b0 2024-03-18T20:26:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.103+0000 7f1822ffd700 1 -- 172.21.15.1:0/1570581150 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1828007310 con 0x7f18340724b0 2024-03-18T20:26:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.103+0000 7f18388fa700 1 -- 172.21.15.1:0/1570581150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18340b3c10 con 0x7f18340724b0 2024-03-18T20:26:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.105+0000 7f18388fa700 1 -- 172.21.15.1:0/1570581150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18340c3c10 con 0x7f18340724b0 2024-03-18T20:26:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.105+0000 7f1822ffd700 1 -- 172.21.15.1:0/1570581150 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 151) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f1828007470 con 0x7f18340724b0 2024-03-18T20:26:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.105+0000 7f1822ffd700 1 --2- 172.21.15.1:0/1570581150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f181406aae0 0x7f181406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:55.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.105+0000 7f183259c700 1 -- 172.21.15.1:0/1570581150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f181406aae0 msgr2=0x7f181406cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:55.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.105+0000 7f1822ffd700 1 -- 172.21.15.1:0/1570581150 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7423+0+0 (secure 0 0 0) 0x7f18280892b0 con 0x7f18340724b0 2024-03-18T20:26:55.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.105+0000 7f183259c700 1 --2- 172.21.15.1:0/1570581150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f181406aae0 0x7f181406cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:55.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.110+0000 7f1822ffd700 1 -- 172.21.15.1:0/1570581150 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f18340c3c10 con 0x7f18340724b0 2024-03-18T20:26:55.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.306+0000 7f183259c700 1 -- 172.21.15.1:0/1570581150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f181406aae0 msgr2=0x7f181406cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:55.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.306+0000 7f183259c700 1 --2- 172.21.15.1:0/1570581150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f181406aae0 0x7f181406cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:55.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.314+0000 7f18388fa700 1 -- 172.21.15.1:0/1570581150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f183406f4f0 con 0x7f18340724b0 2024-03-18T20:26:55.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.315+0000 7f1822ffd700 1 -- 172.21.15.1:0/1570581150 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v258) v1 ==== 93+0+10 (secure 0 0 0) 0x7f183406f4f0 con 0x7f18340724b0 2024-03-18T20:26:55.319 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:26:55.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.317+0000 7f18388fa700 1 -- 172.21.15.1:0/1570581150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f181406aae0 msgr2=0x7f181406cfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:55.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.317+0000 7f18388fa700 1 --2- 172.21.15.1:0/1570581150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f181406aae0 0x7f181406cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.317+0000 7f18388fa700 1 -- 172.21.15.1:0/1570581150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18340724b0 msgr2=0x7f18340b4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:55.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.317+0000 7f18388fa700 1 --2- 172.21.15.1:0/1570581150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18340724b0 0x7f18340b4b50 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f182800c780 tx=0x7f182800ca90 comp rx=0 tx=0).stop 2024-03-18T20:26:55.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.317+0000 7f18388fa700 1 -- 172.21.15.1:0/1570581150 shutdown_connections 2024-03-18T20:26:55.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.317+0000 7f18388fa700 1 --2- 172.21.15.1:0/1570581150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f181406aae0 0x7f181406cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.317+0000 7f18388fa700 1 --2- 172.21.15.1:0/1570581150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f183415d580 0x7f18340b5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.317+0000 7f18388fa700 1 --2- 172.21.15.1:0/1570581150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f183406d380 0x7f183415b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.317+0000 7f18388fa700 1 --2- 172.21.15.1:0/1570581150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18340724b0 0x7f18340b4b50 unknown :-1 s=CLOSED pgs=2918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.317+0000 7f18388fa700 1 -- 172.21.15.1:0/1570581150 >> 172.21.15.1:0/1570581150 conn(0x7f18340cca00 msgr2=0x7f18340cf280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:55.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.317+0000 7f18388fa700 1 -- 172.21.15.1:0/1570581150 shutdown_connections 2024-03-18T20:26:55.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.317+0000 7f18388fa700 1 -- 172.21.15.1:0/1570581150 wait complete. 2024-03-18T20:26:55.334 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:26:55.392 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:55.388+0000 7f76988f6240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:26:55.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.462+0000 7fad0f675700 1 Processor -- start 2024-03-18T20:26:55.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.462+0000 7fad0f675700 1 -- start start 2024-03-18T20:26:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.463+0000 7fad0f675700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fad0815b3a0 0x7fad080b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.463+0000 7fad0f675700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fad080cf570 0x7fad080cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.463+0000 7fad0f675700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad080b0f60 con 0x7fad0815eee0 2024-03-18T20:26:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.463+0000 7fad0f675700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad080cff60 con 0x7fad0815b3a0 2024-03-18T20:26:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.463+0000 7fad0f675700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0815f360 con 0x7fad080cf570 2024-03-18T20:26:55.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.463+0000 7fad0d411700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fad0815b3a0 0x7fad080b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.463+0000 7fad0d411700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fad0815b3a0 0x7fad080b3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34690/0 (socket says 172.21.15.1:34690) 2024-03-18T20:26:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.463+0000 7fad0d411700 1 -- 172.21.15.1:0/907333100 learned_addr learned my addr 172.21.15.1:0/907333100 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.464+0000 7fad0dc12700 1 --2- 172.21.15.1:0/907333100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fad080cf570 0x7fad080cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.464+0000 7fad0dc12700 1 -- 172.21.15.1:0/907333100 >> v2:172.21.15.1:3300/0 conn(0x7fad0815b3a0 msgr2=0x7fad080b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.464+0000 7fad0dc12700 1 --2- 172.21.15.1:0/907333100 >> v2:172.21.15.1:3300/0 conn(0x7fad0815b3a0 0x7fad080b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.464+0000 7facfffff700 1 -- 172.21.15.1:0/907333100 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545057957 0 0) 0x7fad080b0f60 con 0x7fad0815eee0 2024-03-18T20:26:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.464+0000 7fad0dc12700 1 -- 172.21.15.1:0/907333100 >> v1:172.21.15.165:6789/0 conn(0x7fad0815eee0 legacy=0x7fad080cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.464+0000 7fad0dc12700 1 -- 172.21.15.1:0/907333100 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0815f630 con 0x7fad080cf570 2024-03-18T20:26:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.464+0000 7fad0dc12700 1 --2- 172.21.15.1:0/907333100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fad080cf570 0x7fad080cfa20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fad00009d90 tx=0x7fad0000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9a4e9152cf47e65 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.464+0000 7facfffff700 1 -- 172.21.15.1:0/907333100 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0000e070 con 0x7fad080cf570 2024-03-18T20:26:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.464+0000 7facfffff700 1 -- 172.21.15.1:0/907333100 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad0000bb20 con 0x7fad080cf570 2024-03-18T20:26:55.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.464+0000 7facfffff700 1 -- 172.21.15.1:0/907333100 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad00012430 con 0x7fad080cf570 2024-03-18T20:26:55.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.465+0000 7fad0f675700 1 -- 172.21.15.1:0/907333100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fad080cf570 msgr2=0x7fad080cfa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.465+0000 7fad0f675700 1 --2- 172.21.15.1:0/907333100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fad080cf570 0x7fad080cfa20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fad00009d90 tx=0x7fad0000b330 comp rx=0 tx=0).stop 2024-03-18T20:26:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.465+0000 7fad0f675700 1 -- 172.21.15.1:0/907333100 shutdown_connections 2024-03-18T20:26:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.465+0000 7fad0f675700 1 --2- 172.21.15.1:0/907333100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fad080cf570 0x7fad080cfa20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.465+0000 7fad0f675700 1 --2- 172.21.15.1:0/907333100 >> v2:172.21.15.1:3300/0 conn(0x7fad0815b3a0 0x7fad080b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.465+0000 7fad0f675700 1 -- 172.21.15.1:0/907333100 >> 172.21.15.1:0/907333100 conn(0x7fad08099a40 msgr2=0x7fad08099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.465+0000 7fad0f675700 1 -- 172.21.15.1:0/907333100 shutdown_connections 2024-03-18T20:26:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.465+0000 7fad0f675700 1 -- 172.21.15.1:0/907333100 wait complete. 2024-03-18T20:26:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.467+0000 7fad0f675700 1 Processor -- start 2024-03-18T20:26:55.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.467+0000 7fad0f675700 1 -- start start 2024-03-18T20:26:55.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.468+0000 7fad0f675700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fad080cf570 0x7fad080caa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:55.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.468+0000 7fad0f675700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad0815b3a0 0x7fad0814f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:55.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.468+0000 7fad0f675700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad0815eee0 0x7fad080aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:55.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.468+0000 7fad0f675700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad080c8060 con 0x7fad0815eee0 2024-03-18T20:26:55.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.468+0000 7fad0f675700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad080c8300 con 0x7fad0815b3a0 2024-03-18T20:26:55.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.468+0000 7fad0f675700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad080c85e0 con 0x7fad080cf570 2024-03-18T20:26:55.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.468+0000 7fad0dc12700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad0815eee0 0x7fad080aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:55.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.468+0000 7fad0dc12700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad0815eee0 0x7fad080aeb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34694/0 (socket says 172.21.15.1:34694) 2024-03-18T20:26:55.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.468+0000 7fad0dc12700 1 -- 172.21.15.1:0/500314050 learned_addr learned my addr 172.21.15.1:0/500314050 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:55.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.468+0000 7fad0d411700 1 --2- 172.21.15.1:0/500314050 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fad080cf570 0x7fad080caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:55.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.468+0000 7fad0cc10700 1 --2- 172.21.15.1:0/500314050 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad0815b3a0 0x7fad0814f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:55.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.468+0000 7fad0dc12700 1 -- 172.21.15.1:0/500314050 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fad080cf570 msgr2=0x7fad080caa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:55.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.468+0000 7fad0dc12700 1 --2- 172.21.15.1:0/500314050 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fad080cf570 0x7fad080caa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.468+0000 7fad0dc12700 1 -- 172.21.15.1:0/500314050 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad0815b3a0 msgr2=0x7fad0814f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:55.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.468+0000 7fad0dc12700 1 --2- 172.21.15.1:0/500314050 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad0815b3a0 0x7fad0814f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.468+0000 7fad0dc12700 1 -- 172.21.15.1:0/500314050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad00009a70 con 0x7fad0815eee0 2024-03-18T20:26:55.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.468+0000 7fad0dc12700 1 --2- 172.21.15.1:0/500314050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad0815eee0 0x7fad080aeb90 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7fad0815f4a0 tx=0x7fad00004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:55.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.469+0000 7facfdffb700 1 -- 172.21.15.1:0/500314050 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad00012430 con 0x7fad0815eee0 2024-03-18T20:26:55.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.469+0000 7facfdffb700 1 -- 172.21.15.1:0/500314050 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad00020070 con 0x7fad0815eee0 2024-03-18T20:26:55.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.469+0000 7fad0f675700 1 -- 172.21.15.1:0/500314050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad080c88c0 con 0x7fad0815eee0 2024-03-18T20:26:55.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.469+0000 7facfdffb700 1 -- 172.21.15.1:0/500314050 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad00004280 con 0x7fad0815eee0 2024-03-18T20:26:55.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.469+0000 7fad0f675700 1 -- 172.21.15.1:0/500314050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad0815b900 con 0x7fad0815eee0 2024-03-18T20:26:55.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.470+0000 7fad0f675700 1 -- 172.21.15.1:0/500314050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad08067b90 con 0x7fad0815eee0 2024-03-18T20:26:55.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.470+0000 7facfdffb700 1 -- 172.21.15.1:0/500314050 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 152) v1 ==== 89342+0+0 (secure 0 0 0) 0x7fad000043e0 con 0x7fad0815eee0 2024-03-18T20:26:55.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.471+0000 7facfdffb700 1 --2- 172.21.15.1:0/500314050 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facec06aed0 0x7facec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:55.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.471+0000 7fad0d411700 1 -- 172.21.15.1:0/500314050 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facec06aed0 msgr2=0x7facec06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:55.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.471+0000 7facfdffb700 1 -- 172.21.15.1:0/500314050 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7423+0+0 (secure 0 0 0) 0x7fad00022070 con 0x7fad0815eee0 2024-03-18T20:26:55.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.471+0000 7fad0d411700 1 --2- 172.21.15.1:0/500314050 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facec06aed0 0x7facec06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:55.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.479+0000 7facfdffb700 1 -- 172.21.15.1:0/500314050 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fad00056610 con 0x7fad0815eee0 2024-03-18T20:26:55.485 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:55.481+0000 7f5294a3a240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:26:55.606 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:55.602+0000 7f76988f6240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:26:55.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.672+0000 7fad0d411700 1 -- 172.21.15.1:0/500314050 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facec06aed0 msgr2=0x7facec06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:55.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.672+0000 7fad0d411700 1 --2- 172.21.15.1:0/500314050 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facec06aed0 0x7facec06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.684+0000 7fad0f675700 1 -- 172.21.15.1:0/500314050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fad08073350 con 0x7fad0815eee0 2024-03-18T20:26:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.685+0000 7facfdffb700 1 -- 172.21.15.1:0/500314050 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v258) v1 ==== 104+0+11 (secure 0 0 0) 0x7fad00059c30 con 0x7fad0815eee0 2024-03-18T20:26:55.689 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:26:55.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.687+0000 7fad0f675700 1 -- 172.21.15.1:0/500314050 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facec06aed0 msgr2=0x7facec06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.687+0000 7fad0f675700 1 --2- 172.21.15.1:0/500314050 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facec06aed0 0x7facec06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.687+0000 7fad0f675700 1 -- 172.21.15.1:0/500314050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad0815eee0 msgr2=0x7fad080aeb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.687+0000 7fad0f675700 1 --2- 172.21.15.1:0/500314050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad0815eee0 0x7fad080aeb90 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7fad0815f4a0 tx=0x7fad00004ad0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.687+0000 7fad0f675700 1 -- 172.21.15.1:0/500314050 shutdown_connections 2024-03-18T20:26:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.687+0000 7fad0f675700 1 --2- 172.21.15.1:0/500314050 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facec06aed0 0x7facec06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.687+0000 7fad0f675700 1 --2- 172.21.15.1:0/500314050 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fad080cf570 0x7fad080caa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.687+0000 7fad0f675700 1 --2- 172.21.15.1:0/500314050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad0815eee0 0x7fad080aeb90 unknown :-1 s=CLOSED pgs=2919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.687+0000 7fad0f675700 1 --2- 172.21.15.1:0/500314050 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad0815b3a0 0x7fad0814f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.687+0000 7fad0f675700 1 -- 172.21.15.1:0/500314050 >> 172.21.15.1:0/500314050 conn(0x7fad08099a40 msgr2=0x7fad080ccc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.687+0000 7fad0f675700 1 -- 172.21.15.1:0/500314050 shutdown_connections 2024-03-18T20:26:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.687+0000 7fad0f675700 1 -- 172.21.15.1:0/500314050 wait complete. 2024-03-18T20:26:55.704 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:26:55.708 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:55.704+0000 7f5294a3a240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:26:55.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.829+0000 7f18badf6700 1 Processor -- start 2024-03-18T20:26:55.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.829+0000 7f18badf6700 1 -- start start 2024-03-18T20:26:55.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.830+0000 7f18badf6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f18b415b370 0x7f18b40cec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:55.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.830+0000 7f18badf6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f18b4071550 0x7f18b406d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:55.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.830+0000 7f18badf6700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b415b750 con 0x7f18b40aecb0 2024-03-18T20:26:55.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.830+0000 7f18badf6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b40cf510 con 0x7f18b415b370 2024-03-18T20:26:55.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.830+0000 7f18badf6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b40ccc20 con 0x7f18b4071550 2024-03-18T20:26:55.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.830+0000 7f18b8b92700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f18b415b370 0x7f18b40cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:55.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.830+0000 7f18b3fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f18b40aecb0 0x7f18b4070e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44998/0 (socket says 172.21.15.1:44998) 2024-03-18T20:26:55.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.830+0000 7f18b3fff700 1 -- 172.21.15.1:0/2087180707 learned_addr learned my addr 172.21.15.1:0/2087180707 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:55.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.830+0000 7f18b9393700 1 --2- 172.21.15.1:0/2087180707 >> v2:172.21.15.165:3300/0 conn(0x7f18b4071550 0x7f18b406d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:55.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.830+0000 7f18b8b92700 1 -- 172.21.15.1:0/2087180707 >> v2:172.21.15.165:3300/0 conn(0x7f18b4071550 msgr2=0x7f18b406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:55.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.830+0000 7f18b8b92700 1 --2- 172.21.15.1:0/2087180707 >> v2:172.21.15.165:3300/0 conn(0x7f18b4071550 0x7f18b406d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.830+0000 7f18b8b92700 1 -- 172.21.15.1:0/2087180707 >> v1:172.21.15.1:6789/0 conn(0x7f18b40aecb0 legacy=0x7f18b4070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:55.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.830+0000 7f18b8b92700 1 -- 172.21.15.1:0/2087180707 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b40cc2f0 con 0x7f18b415b370 2024-03-18T20:26:55.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.830+0000 7f18b8b92700 1 --2- 172.21.15.1:0/2087180707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b415b370 0x7f18b40cec70 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f18b40af270 tx=0x7f18a8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cfcc76adb1e1760 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:55.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.831+0000 7f18b37fe700 1 -- 172.21.15.1:0/2087180707 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a8011830 con 0x7f18b415b370 2024-03-18T20:26:55.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.831+0000 7f18b37fe700 1 -- 172.21.15.1:0/2087180707 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a8011990 con 0x7f18b415b370 2024-03-18T20:26:55.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.831+0000 7f18b37fe700 1 -- 172.21.15.1:0/2087180707 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a8011d20 con 0x7f18b415b370 2024-03-18T20:26:55.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.831+0000 7f18badf6700 1 -- 172.21.15.1:0/2087180707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b415b370 msgr2=0x7f18b40cec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:55.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.831+0000 7f18badf6700 1 --2- 172.21.15.1:0/2087180707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b415b370 0x7f18b40cec70 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f18b40af270 tx=0x7f18a8009b20 comp rx=0 tx=0).stop 2024-03-18T20:26:55.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.831+0000 7f18badf6700 1 -- 172.21.15.1:0/2087180707 shutdown_connections 2024-03-18T20:26:55.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.831+0000 7f18badf6700 1 --2- 172.21.15.1:0/2087180707 >> v2:172.21.15.165:3300/0 conn(0x7f18b4071550 0x7f18b406d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.831+0000 7f18badf6700 1 --2- 172.21.15.1:0/2087180707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b415b370 0x7f18b40cec70 unknown :-1 s=CLOSED pgs=2921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.831+0000 7f18badf6700 1 -- 172.21.15.1:0/2087180707 >> 172.21.15.1:0/2087180707 conn(0x7f18b40b1350 msgr2=0x7f18b40b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:55.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.832+0000 7f18badf6700 1 -- 172.21.15.1:0/2087180707 shutdown_connections 2024-03-18T20:26:55.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.832+0000 7f18badf6700 1 -- 172.21.15.1:0/2087180707 wait complete. 2024-03-18T20:26:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.833+0000 7f18badf6700 1 Processor -- start 2024-03-18T20:26:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.833+0000 7f18badf6700 1 -- start start 2024-03-18T20:26:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.834+0000 7f18badf6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b4071550 0x7f18b41457a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.834+0000 7f18badf6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f18b40aecb0 0x7f18b4157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.834+0000 7f18badf6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18b415b370 0x7f18b40ab670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.834+0000 7f18badf6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b4159910 con 0x7f18b4071550 2024-03-18T20:26:55.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.834+0000 7f18badf6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b4158870 con 0x7f18b415b370 2024-03-18T20:26:55.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.834+0000 7f18badf6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b4158b50 con 0x7f18b40aecb0 2024-03-18T20:26:55.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.834+0000 7f18b8b92700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b4071550 0x7f18b41457a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:55.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.834+0000 7f18b8b92700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b4071550 0x7f18b41457a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34710/0 (socket says 172.21.15.1:34710) 2024-03-18T20:26:55.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.834+0000 7f18b8b92700 1 -- 172.21.15.1:0/2224662110 learned_addr learned my addr 172.21.15.1:0/2224662110 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:55.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.834+0000 7f18b9393700 1 --2- 172.21.15.1:0/2224662110 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18b415b370 0x7f18b40ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:55.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.834+0000 7f18b3fff700 1 --2- 172.21.15.1:0/2224662110 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f18b40aecb0 0x7f18b4157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:55.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.834+0000 7f18b8b92700 1 -- 172.21.15.1:0/2224662110 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f18b40aecb0 msgr2=0x7f18b4157f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:55.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.834+0000 7f18b8b92700 1 --2- 172.21.15.1:0/2224662110 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f18b40aecb0 0x7f18b4157f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.834+0000 7f18b8b92700 1 -- 172.21.15.1:0/2224662110 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18b415b370 msgr2=0x7f18b40ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:55.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.834+0000 7f18b8b92700 1 --2- 172.21.15.1:0/2224662110 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18b415b370 0x7f18b40ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:55.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.834+0000 7f18b8b92700 1 -- 172.21.15.1:0/2224662110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a80094a0 con 0x7f18b4071550 2024-03-18T20:26:55.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.834+0000 7f18b8b92700 1 --2- 172.21.15.1:0/2224662110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b4071550 0x7f18b41457a0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7f18a80090b0 tx=0x7f18a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:55.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.834+0000 7f18b17fa700 1 -- 172.21.15.1:0/2224662110 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a80036a0 con 0x7f18b4071550 2024-03-18T20:26:55.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.835+0000 7f18b17fa700 1 -- 172.21.15.1:0/2224662110 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a801e070 con 0x7f18b4071550 2024-03-18T20:26:55.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.835+0000 7f18badf6700 1 -- 172.21.15.1:0/2224662110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b4158e30 con 0x7f18b4071550 2024-03-18T20:26:55.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.835+0000 7f18b17fa700 1 -- 172.21.15.1:0/2224662110 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a8003a60 con 0x7f18b4071550 2024-03-18T20:26:55.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.835+0000 7f18badf6700 1 -- 172.21.15.1:0/2224662110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b41593f0 con 0x7f18b4071550 2024-03-18T20:26:55.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.836+0000 7f18badf6700 1 -- 172.21.15.1:0/2224662110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b40676a0 con 0x7f18b4071550 2024-03-18T20:26:55.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.837+0000 7f18b17fa700 1 -- 172.21.15.1:0/2224662110 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 152) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f18a8003bc0 con 0x7f18b4071550 2024-03-18T20:26:55.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.837+0000 7f18b17fa700 1 --2- 172.21.15.1:0/2224662110 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f189806af00 0x7f189806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:55.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.837+0000 7f18b3fff700 1 -- 172.21.15.1:0/2224662110 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f189806af00 msgr2=0x7f189806d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:55.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.837+0000 7f18b17fa700 1 -- 172.21.15.1:0/2224662110 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7423+0+0 (secure 0 0 0) 0x7f18a808af30 con 0x7f18b4071550 2024-03-18T20:26:55.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.837+0000 7f18b3fff700 1 --2- 172.21.15.1:0/2224662110 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f189806af00 0x7f189806d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:55.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:55.841+0000 7f18b17fa700 1 -- 172.21.15.1:0/2224662110 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f18a8055870 con 0x7f18b4071550 2024-03-18T20:26:55.983 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:55.979+0000 7f76988f6240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:26:56.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.038+0000 7f18b3fff700 1 -- 172.21.15.1:0/2224662110 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f189806af00 msgr2=0x7f189806d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:56.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.038+0000 7f18b3fff700 1 --2- 172.21.15.1:0/2224662110 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f189806af00 0x7f189806d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:56.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.046+0000 7f18badf6700 1 -- 172.21.15.1:0/2224662110 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f18b4073db0 con 0x7f18b4071550 2024-03-18T20:26:56.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.046+0000 7f18b17fa700 1 -- 172.21.15.1:0/2224662110 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v258) v1 ==== 100+0+11 (secure 0 0 0) 0x7f18a8020070 con 0x7f18b4071550 2024-03-18T20:26:56.050 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:26:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.048+0000 7f18badf6700 1 -- 172.21.15.1:0/2224662110 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f189806af00 msgr2=0x7f189806d3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.048+0000 7f18badf6700 1 --2- 172.21.15.1:0/2224662110 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f189806af00 0x7f189806d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.048+0000 7f18badf6700 1 -- 172.21.15.1:0/2224662110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b4071550 msgr2=0x7f18b41457a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.048+0000 7f18badf6700 1 --2- 172.21.15.1:0/2224662110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b4071550 0x7f18b41457a0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7f18a80090b0 tx=0x7f18a8004750 comp rx=0 tx=0).stop 2024-03-18T20:26:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.048+0000 7f18badf6700 1 -- 172.21.15.1:0/2224662110 shutdown_connections 2024-03-18T20:26:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.048+0000 7f18badf6700 1 --2- 172.21.15.1:0/2224662110 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f189806af00 0x7f189806d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.048+0000 7f18badf6700 1 --2- 172.21.15.1:0/2224662110 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18b415b370 0x7f18b40ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.048+0000 7f18badf6700 1 --2- 172.21.15.1:0/2224662110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18b4071550 0x7f18b41457a0 unknown :-1 s=CLOSED pgs=2922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.048+0000 7f18badf6700 1 --2- 172.21.15.1:0/2224662110 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f18b40aecb0 0x7f18b4157f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.048+0000 7f18badf6700 1 -- 172.21.15.1:0/2224662110 >> 172.21.15.1:0/2224662110 conn(0x7f18b40b1350 msgr2=0x7f18b40cced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:56.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.048+0000 7f18badf6700 1 -- 172.21.15.1:0/2224662110 shutdown_connections 2024-03-18T20:26:56.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.048+0000 7f18badf6700 1 -- 172.21.15.1:0/2224662110 wait complete. 2024-03-18T20:26:56.066 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-03-18T20:26:56.105 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:56.101+0000 7f5294a3a240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:26:56.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.233+0000 7fbaf569b700 1 Processor -- start 2024-03-18T20:26:56.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.233+0000 7fbaf569b700 1 -- start start 2024-03-18T20:26:56.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.234+0000 7fbaf569b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbaf006ca80 0x7fbaf015d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.234+0000 7fbaf569b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbaf009a150 0x7fbaf00cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.234+0000 7fbaf569b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf00b3b00 con 0x7fbaf015d5b0 2024-03-18T20:26:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.234+0000 7fbaf569b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf00cbbb0 con 0x7fbaf009a150 2024-03-18T20:26:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.234+0000 7fbaf569b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf00cf7f0 con 0x7fbaf006ca80 2024-03-18T20:26:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.234+0000 7fbaef7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbaf009a150 0x7fbaf00cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.234+0000 7fbaeeffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbaf015d5b0 0x7fbaf0099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45006/0 (socket says 172.21.15.1:45006) 2024-03-18T20:26:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.234+0000 7fbaeeffd700 1 -- 172.21.15.1:0/1816521038 learned_addr learned my addr 172.21.15.1:0/1816521038 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.234+0000 7fbaee7fc700 1 --2- 172.21.15.1:0/1816521038 >> v2:172.21.15.165:3300/0 conn(0x7fbaf006ca80 0x7fbaf015d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.234+0000 7fbaef7fe700 1 -- 172.21.15.1:0/1816521038 >> v2:172.21.15.165:3300/0 conn(0x7fbaf006ca80 msgr2=0x7fbaf015d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.234+0000 7fbaef7fe700 1 --2- 172.21.15.1:0/1816521038 >> v2:172.21.15.165:3300/0 conn(0x7fbaf006ca80 0x7fbaf015d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.235+0000 7fbaef7fe700 1 -- 172.21.15.1:0/1816521038 >> v1:172.21.15.1:6789/0 conn(0x7fbaf015d5b0 legacy=0x7fbaf0099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:56.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.235+0000 7fbaef7fe700 1 -- 172.21.15.1:0/1816521038 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf00d04d0 con 0x7fbaf009a150 2024-03-18T20:26:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.235+0000 7fbaedffb700 1 -- 172.21.15.1:0/1816521038 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626007265 0 0) 0x7fbaf00b3b00 con 0x7fbaf015d5b0 2024-03-18T20:26:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.235+0000 7fbaef7fe700 1 --2- 172.21.15.1:0/1816521038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf009a150 0x7fbaf00cbfd0 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0072c10 tx=0x7fbae800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93e3ab004055202 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.235+0000 7fbaedffb700 1 -- 172.21.15.1:0/1816521038 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae800e070 con 0x7fbaf009a150 2024-03-18T20:26:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.235+0000 7fbaedffb700 1 -- 172.21.15.1:0/1816521038 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae800bb20 con 0x7fbaf009a150 2024-03-18T20:26:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.235+0000 7fbaedffb700 1 -- 172.21.15.1:0/1816521038 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae8012430 con 0x7fbaf009a150 2024-03-18T20:26:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.235+0000 7fbaf569b700 1 -- 172.21.15.1:0/1816521038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf009a150 msgr2=0x7fbaf00cbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.235+0000 7fbaf569b700 1 --2- 172.21.15.1:0/1816521038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf009a150 0x7fbaf00cbfd0 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0072c10 tx=0x7fbae800b330 comp rx=0 tx=0).stop 2024-03-18T20:26:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.235+0000 7fbaf569b700 1 -- 172.21.15.1:0/1816521038 shutdown_connections 2024-03-18T20:26:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.235+0000 7fbaf569b700 1 --2- 172.21.15.1:0/1816521038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf009a150 0x7fbaf00cbfd0 unknown :-1 s=CLOSED pgs=2924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.235+0000 7fbaf569b700 1 --2- 172.21.15.1:0/1816521038 >> v2:172.21.15.165:3300/0 conn(0x7fbaf006ca80 0x7fbaf015d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.236+0000 7fbaf569b700 1 -- 172.21.15.1:0/1816521038 >> 172.21.15.1:0/1816521038 conn(0x7fbaf00b2bd0 msgr2=0x7fbaf00b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.236+0000 7fbaf569b700 1 -- 172.21.15.1:0/1816521038 shutdown_connections 2024-03-18T20:26:56.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.236+0000 7fbaf569b700 1 -- 172.21.15.1:0/1816521038 wait complete. 2024-03-18T20:26:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.237+0000 7fbaf569b700 1 Processor -- start 2024-03-18T20:26:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.237+0000 7fbaf569b700 1 -- start start 2024-03-18T20:26:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.238+0000 7fbaf569b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf006ca80 0x7fbaf015aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.238+0000 7fbaf569b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbaf009a150 0x7fbaf00ae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.238+0000 7fbaf569b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbaf015d5b0 0x7fbaf009e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.238+0000 7fbaf569b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf009d280 con 0x7fbaf006ca80 2024-03-18T20:26:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.238+0000 7fbaf569b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf009d520 con 0x7fbaf015d5b0 2024-03-18T20:26:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.238+0000 7fbaf569b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf009d800 con 0x7fbaf009a150 2024-03-18T20:26:56.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.238+0000 7fbaeeffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf006ca80 0x7fbaf015aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.238+0000 7fbaeeffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf006ca80 0x7fbaf015aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34728/0 (socket says 172.21.15.1:34728) 2024-03-18T20:26:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.238+0000 7fbaeeffd700 1 -- 172.21.15.1:0/4134400443 learned_addr learned my addr 172.21.15.1:0/4134400443 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.238+0000 7fbaee7fc700 1 --2- 172.21.15.1:0/4134400443 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbaf009a150 0x7fbaf00ae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.238+0000 7fbaef7fe700 1 --2- 172.21.15.1:0/4134400443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbaf015d5b0 0x7fbaf009e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.239+0000 7fbaeeffd700 1 -- 172.21.15.1:0/4134400443 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbaf009a150 msgr2=0x7fbaf00ae7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.239+0000 7fbaeeffd700 1 --2- 172.21.15.1:0/4134400443 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbaf009a150 0x7fbaf00ae7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.239+0000 7fbaeeffd700 1 -- 172.21.15.1:0/4134400443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbaf015d5b0 msgr2=0x7fbaf009e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.239+0000 7fbaeeffd700 1 --2- 172.21.15.1:0/4134400443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbaf015d5b0 0x7fbaf009e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.239+0000 7fbaeeffd700 1 -- 172.21.15.1:0/4134400443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae8009a70 con 0x7fbaf006ca80 2024-03-18T20:26:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.239+0000 7fbaeeffd700 1 --2- 172.21.15.1:0/4134400443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf006ca80 0x7fbaf015aa10 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7fbae00011e0 tx=0x7fbae000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.239+0000 7fbad77fe700 1 -- 172.21.15.1:0/4134400443 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae0004c80 con 0x7fbaf006ca80 2024-03-18T20:26:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.239+0000 7fbad77fe700 1 -- 172.21.15.1:0/4134400443 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae0004e00 con 0x7fbaf006ca80 2024-03-18T20:26:56.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.239+0000 7fbaf569b700 1 -- 172.21.15.1:0/4134400443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf009dae0 con 0x7fbaf006ca80 2024-03-18T20:26:56.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.239+0000 7fbad77fe700 1 -- 172.21.15.1:0/4134400443 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae0007310 con 0x7fbaf006ca80 2024-03-18T20:26:56.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.239+0000 7fbaf569b700 1 -- 172.21.15.1:0/4134400443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf009e0a0 con 0x7fbaf006ca80 2024-03-18T20:26:56.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.240+0000 7fbaf569b700 1 -- 172.21.15.1:0/4134400443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf00676a0 con 0x7fbaf006ca80 2024-03-18T20:26:56.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.241+0000 7fbad77fe700 1 -- 172.21.15.1:0/4134400443 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 152) v1 ==== 89342+0+0 (secure 0 0 0) 0x7fbae0007470 con 0x7fbaf006ca80 2024-03-18T20:26:56.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.241+0000 7fbad77fe700 1 --2- 172.21.15.1:0/4134400443 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbadc06aed0 0x7fbadc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:56.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.241+0000 7fbaee7fc700 1 -- 172.21.15.1:0/4134400443 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbadc06aed0 msgr2=0x7fbadc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:56.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.241+0000 7fbad77fe700 1 -- 172.21.15.1:0/4134400443 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7423+0+0 (secure 0 0 0) 0x7fbae00899a0 con 0x7fbaf006ca80 2024-03-18T20:26:56.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.241+0000 7fbaee7fc700 1 --2- 172.21.15.1:0/4134400443 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbadc06aed0 0x7fbadc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:56.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.246+0000 7fbad77fe700 1 -- 172.21.15.1:0/4134400443 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbae0054200 con 0x7fbaf006ca80 2024-03-18T20:26:56.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.442+0000 7fbaee7fc700 1 -- 172.21.15.1:0/4134400443 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbadc06aed0 msgr2=0x7fbadc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:56.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.442+0000 7fbaee7fc700 1 --2- 172.21.15.1:0/4134400443 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbadc06aed0 0x7fbadc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:56.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.448+0000 7fbaf569b700 1 -- 172.21.15.1:0/4134400443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbaf006c5a0 con 0x7fbaf006ca80 2024-03-18T20:26:56.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.448+0000 7fbad77fe700 1 -- 172.21.15.1:0/4134400443 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v258) v1 ==== 107+0+11 (secure 0 0 0) 0x7fbae0057820 con 0x7fbaf006ca80 2024-03-18T20:26:56.452 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:26:56.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.450+0000 7fbaf569b700 1 -- 172.21.15.1:0/4134400443 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbadc06aed0 msgr2=0x7fbadc06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:56.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.450+0000 7fbaf569b700 1 --2- 172.21.15.1:0/4134400443 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbadc06aed0 0x7fbadc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.451+0000 7fbaf569b700 1 -- 172.21.15.1:0/4134400443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf006ca80 msgr2=0x7fbaf015aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:56.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.451+0000 7fbaf569b700 1 --2- 172.21.15.1:0/4134400443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf006ca80 0x7fbaf015aa10 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7fbae00011e0 tx=0x7fbae000ca30 comp rx=0 tx=0).stop 2024-03-18T20:26:56.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.451+0000 7fbaf569b700 1 -- 172.21.15.1:0/4134400443 shutdown_connections 2024-03-18T20:26:56.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.451+0000 7fbaf569b700 1 --2- 172.21.15.1:0/4134400443 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbadc06aed0 0x7fbadc06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.451+0000 7fbaf569b700 1 --2- 172.21.15.1:0/4134400443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbaf015d5b0 0x7fbaf009e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.451+0000 7fbaf569b700 1 --2- 172.21.15.1:0/4134400443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf006ca80 0x7fbaf015aa10 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.451+0000 7fbaf569b700 1 --2- 172.21.15.1:0/4134400443 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbaf009a150 0x7fbaf00ae7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.451+0000 7fbaf569b700 1 -- 172.21.15.1:0/4134400443 >> 172.21.15.1:0/4134400443 conn(0x7fbaf00b2bd0 msgr2=0x7fbaf00c6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:56.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.451+0000 7fbaf569b700 1 -- 172.21.15.1:0/4134400443 shutdown_connections 2024-03-18T20:26:56.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.451+0000 7fbaf569b700 1 -- 172.21.15.1:0/4134400443 wait complete. 2024-03-18T20:26:56.467 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-03-18T20:26:56.568 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:56.564+0000 7f76988f6240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:26:56.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.598+0000 7f3425b54700 1 Processor -- start 2024-03-18T20:26:56.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.599+0000 7f3425b54700 1 -- start start 2024-03-18T20:26:56.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.599+0000 7f3425b54700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f342015b6c0 0x7f34200b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:56.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.599+0000 7f3425b54700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f34200cd430 0x7f34200c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:56.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.599+0000 7f3425b54700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34200d01f0 con 0x7f34200c8600 2024-03-18T20:26:56.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.599+0000 7f3425b54700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34200ccfa0 con 0x7f34200cd430 2024-03-18T20:26:56.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.599+0000 7f3425b54700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34200d1db0 con 0x7f342015b6c0 2024-03-18T20:26:56.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.600+0000 7f341f7fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f342015b6c0 0x7f34200b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:56.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.600+0000 7f341effd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f34200cd430 0x7f34200c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:56.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.600+0000 7f341f7fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f342015b6c0 0x7f34200b2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:34728/0 (socket says 172.21.15.1:34728) 2024-03-18T20:26:56.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.600+0000 7f341ffff700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f34200c8600 0x7f34200cedd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:33974/0 (socket says 172.21.15.1:33974) 2024-03-18T20:26:56.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.600+0000 7f341f7fe700 1 -- 172.21.15.1:0/272146794 learned_addr learned my addr 172.21.15.1:0/272146794 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:56.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.600+0000 7f341f7fe700 1 -- 172.21.15.1:0/272146794 >> v2:172.21.15.165:3300/0 conn(0x7f34200cd430 msgr2=0x7f34200c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:56.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.600+0000 7f341f7fe700 1 --2- 172.21.15.1:0/272146794 >> v2:172.21.15.165:3300/0 conn(0x7f34200cd430 0x7f34200c80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.600+0000 7f341f7fe700 1 -- 172.21.15.1:0/272146794 >> v1:172.21.15.165:6789/0 conn(0x7f34200c8600 legacy=0x7f34200cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:56.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.600+0000 7f341f7fe700 1 -- 172.21.15.1:0/272146794 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34200cff10 con 0x7f342015b6c0 2024-03-18T20:26:56.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.600+0000 7f341effd700 1 --2- 172.21.15.1:0/272146794 >> v2:172.21.15.165:3300/0 conn(0x7f34200cd430 0x7f34200c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:26:56.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.600+0000 7f341e7fc700 1 -- 172.21.15.1:0/272146794 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831433091 0 0) 0x7f34200d01f0 con 0x7f34200c8600 2024-03-18T20:26:56.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.600+0000 7f341f7fe700 1 --2- 172.21.15.1:0/272146794 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f342015b6c0 0x7f34200b2160 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3410009e50 tx=0x7f341000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51a9194522d0f74e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:56.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.600+0000 7f341e7fc700 1 -- 172.21.15.1:0/272146794 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341000e070 con 0x7f342015b6c0 2024-03-18T20:26:56.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.600+0000 7f341e7fc700 1 -- 172.21.15.1:0/272146794 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341000b910 con 0x7f342015b6c0 2024-03-18T20:26:56.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.600+0000 7f341e7fc700 1 -- 172.21.15.1:0/272146794 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341000bcc0 con 0x7f342015b6c0 2024-03-18T20:26:56.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.601+0000 7f3425b54700 1 -- 172.21.15.1:0/272146794 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f342015b6c0 msgr2=0x7f34200b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:56.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.601+0000 7f3425b54700 1 --2- 172.21.15.1:0/272146794 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f342015b6c0 0x7f34200b2160 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3410009e50 tx=0x7f341000b330 comp rx=0 tx=0).stop 2024-03-18T20:26:56.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.601+0000 7f3425b54700 1 -- 172.21.15.1:0/272146794 shutdown_connections 2024-03-18T20:26:56.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.601+0000 7f3425b54700 1 --2- 172.21.15.1:0/272146794 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f342015b6c0 0x7f34200b2160 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.601+0000 7f3425b54700 1 --2- 172.21.15.1:0/272146794 >> v2:172.21.15.165:3300/0 conn(0x7f34200cd430 0x7f34200c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.601+0000 7f3425b54700 1 -- 172.21.15.1:0/272146794 >> 172.21.15.1:0/272146794 conn(0x7f34200d0ff0 msgr2=0x7f34200d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:56.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.601+0000 7f3425b54700 1 -- 172.21.15.1:0/272146794 shutdown_connections 2024-03-18T20:26:56.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.601+0000 7f3425b54700 1 -- 172.21.15.1:0/272146794 wait complete. 2024-03-18T20:26:56.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.602+0000 7f3425b54700 1 Processor -- start 2024-03-18T20:26:56.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.602+0000 7f3425b54700 1 -- start start 2024-03-18T20:26:56.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.603+0000 7f3425b54700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34200c8600 0x7f34201f3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:56.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.603+0000 7f3425b54700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34200cd430 0x7f34201f0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:56.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.603+0000 7f3425b54700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f342015b6c0 0x7f34200b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:56.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.603+0000 7f3425b54700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34201f3a50 con 0x7f34200c8600 2024-03-18T20:26:56.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.603+0000 7f3425b54700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34201f3d30 con 0x7f342015b6c0 2024-03-18T20:26:56.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.603+0000 7f3425b54700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34201f4010 con 0x7f34200cd430 2024-03-18T20:26:56.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.603+0000 7f341f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34200c8600 0x7f34201f3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:56.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.603+0000 7f341f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34200c8600 0x7f34201f3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34736/0 (socket says 172.21.15.1:34736) 2024-03-18T20:26:56.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.603+0000 7f341f7fe700 1 -- 172.21.15.1:0/2548394636 learned_addr learned my addr 172.21.15.1:0/2548394636 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:56.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.603+0000 7f341effd700 1 --2- 172.21.15.1:0/2548394636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34200cd430 0x7f34201f0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:56.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.603+0000 7f341ffff700 1 --2- 172.21.15.1:0/2548394636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f342015b6c0 0x7f34200b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:56.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.604+0000 7f341f7fe700 1 -- 172.21.15.1:0/2548394636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34200cd430 msgr2=0x7f34201f0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:56.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.604+0000 7f341f7fe700 1 --2- 172.21.15.1:0/2548394636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34200cd430 0x7f34201f0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.604+0000 7f341f7fe700 1 -- 172.21.15.1:0/2548394636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f342015b6c0 msgr2=0x7f34200b40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:56.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.604+0000 7f341f7fe700 1 --2- 172.21.15.1:0/2548394636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f342015b6c0 0x7f34200b40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.604+0000 7f341f7fe700 1 -- 172.21.15.1:0/2548394636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34140094a0 con 0x7f34200c8600 2024-03-18T20:26:56.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.604+0000 7f341f7fe700 1 --2- 172.21.15.1:0/2548394636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34200c8600 0x7f34201f3470 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7f34200d1ef0 tx=0x7f3410004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:56.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.604+0000 7f3407fff700 1 -- 172.21.15.1:0/2548394636 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3410003ea0 con 0x7f34200c8600 2024-03-18T20:26:56.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.604+0000 7f3407fff700 1 -- 172.21.15.1:0/2548394636 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3410004900 con 0x7f34200c8600 2024-03-18T20:26:56.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.604+0000 7f3425b54700 1 -- 172.21.15.1:0/2548394636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3410009b30 con 0x7f34200c8600 2024-03-18T20:26:56.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.604+0000 7f3407fff700 1 -- 172.21.15.1:0/2548394636 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341001c660 con 0x7f34200c8600 2024-03-18T20:26:56.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.604+0000 7f3425b54700 1 -- 172.21.15.1:0/2548394636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34201f6b00 con 0x7f34200c8600 2024-03-18T20:26:56.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.605+0000 7f3425b54700 1 -- 172.21.15.1:0/2548394636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34200676a0 con 0x7f34200c8600 2024-03-18T20:26:56.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.606+0000 7f3407fff700 1 -- 172.21.15.1:0/2548394636 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 152) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f341001c7c0 con 0x7f34200c8600 2024-03-18T20:26:56.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.606+0000 7f3407fff700 1 --2- 172.21.15.1:0/2548394636 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f340c06aed0 0x7f340c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:56.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.606+0000 7f341effd700 1 -- 172.21.15.1:0/2548394636 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f340c06aed0 msgr2=0x7f340c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:56.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.606+0000 7f3407fff700 1 -- 172.21.15.1:0/2548394636 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7423+0+0 (secure 0 0 0) 0x7f341008b870 con 0x7f34200c8600 2024-03-18T20:26:56.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.606+0000 7f341effd700 1 --2- 172.21.15.1:0/2548394636 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f340c06aed0 0x7f340c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:56.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.615+0000 7f3407fff700 1 -- 172.21.15.1:0/2548394636 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3410055ea0 con 0x7f34200c8600 2024-03-18T20:26:56.641 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:56.637+0000 7f76988f6240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:26:56.746 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:56.741+0000 7f5294a3a240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:26:56.768 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:56.764+0000 7f76988f6240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:26:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.807+0000 7f341effd700 1 -- 172.21.15.1:0/2548394636 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f340c06aed0 msgr2=0x7f340c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:56.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.807+0000 7f341effd700 1 --2- 172.21.15.1:0/2548394636 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f340c06aed0 0x7f340c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:56.821 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:56.817+0000 7f5294a3a240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:26:56.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.820+0000 7f3425b54700 1 -- 172.21.15.1:0/2548394636 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f342006f8a0 con 0x7f34200c8600 2024-03-18T20:26:56.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.820+0000 7f3407fff700 1 -- 172.21.15.1:0/2548394636 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v258) v1 ==== 103+0+11 (secure 0 0 0) 0x7f34100594c0 con 0x7f34200c8600 2024-03-18T20:26:56.824 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:26:56.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.822+0000 7f3425b54700 1 -- 172.21.15.1:0/2548394636 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f340c06aed0 msgr2=0x7f340c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:56.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.822+0000 7f3425b54700 1 --2- 172.21.15.1:0/2548394636 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f340c06aed0 0x7f340c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.822+0000 7f3425b54700 1 -- 172.21.15.1:0/2548394636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34200c8600 msgr2=0x7f34201f3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:56.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.822+0000 7f3425b54700 1 --2- 172.21.15.1:0/2548394636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34200c8600 0x7f34201f3470 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7f34200d1ef0 tx=0x7f3410004ad0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.823+0000 7f3425b54700 1 -- 172.21.15.1:0/2548394636 shutdown_connections 2024-03-18T20:26:56.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.823+0000 7f3425b54700 1 --2- 172.21.15.1:0/2548394636 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f340c06aed0 0x7f340c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.823+0000 7f3425b54700 1 --2- 172.21.15.1:0/2548394636 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f342015b6c0 0x7f34200b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.823+0000 7f3425b54700 1 --2- 172.21.15.1:0/2548394636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34200c8600 0x7f34201f3470 unknown :-1 s=CLOSED pgs=2926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.823+0000 7f3425b54700 1 --2- 172.21.15.1:0/2548394636 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34200cd430 0x7f34201f0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.823+0000 7f3425b54700 1 -- 172.21.15.1:0/2548394636 >> 172.21.15.1:0/2548394636 conn(0x7f34200d0ff0 msgr2=0x7f34200c9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:56.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.823+0000 7f3425b54700 1 -- 172.21.15.1:0/2548394636 shutdown_connections 2024-03-18T20:26:56.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.823+0000 7f3425b54700 1 -- 172.21.15.1:0/2548394636 wait complete. 2024-03-18T20:26:56.840 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T20:26:56.952 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:56.948+0000 7f76988f6240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:26:56.961 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:56.957+0000 7f5294a3a240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:26:56.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.963+0000 7f3bbed3f700 1 Processor -- start 2024-03-18T20:26:56.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.963+0000 7f3bbed3f700 1 -- start start 2024-03-18T20:26:56.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.964+0000 7f3bbed3f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3bb80cff70 0x7f3bb80cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:56.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.964+0000 7f3bbed3f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb80d0df0 con 0x7f3bb80cd870 2024-03-18T20:26:56.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.964+0000 7f3bbed3f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb80d0550 con 0x7f3bb80b1e20 2024-03-18T20:26:56.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.964+0000 7f3bbed3f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb80d0830 con 0x7f3bb80cff70 2024-03-18T20:26:56.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.964+0000 7f3bbd2dc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3bb80cff70 0x7f3bb80cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:56.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.964+0000 7f3bbcadb700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3bb80cd870 0x7f3bb80cdc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45022/0 (socket says 172.21.15.1:45022) 2024-03-18T20:26:56.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.964+0000 7f3bbcadb700 1 -- 172.21.15.1:0/2140943923 learned_addr learned my addr 172.21.15.1:0/2140943923 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:56.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.964+0000 7f3bbd2dc700 1 -- 172.21.15.1:0/2140943923 >> v1:172.21.15.165:6789/0 conn(0x7f3bb80b1e20 legacy=0x7f3bb80cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:56.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.964+0000 7f3bbd2dc700 1 -- 172.21.15.1:0/2140943923 >> v1:172.21.15.1:6789/0 conn(0x7f3bb80cd870 legacy=0x7f3bb80cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:56.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.965+0000 7f3bbd2dc700 1 -- 172.21.15.1:0/2140943923 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb80d0b10 con 0x7f3bb80cff70 2024-03-18T20:26:56.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.965+0000 7f3bb77fe700 1 -- 172.21.15.1:0/2140943923 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683839168 0 0) 0x7f3bb80d0df0 con 0x7f3bb80cd870 2024-03-18T20:26:56.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.965+0000 7f3bbd2dc700 1 --2- 172.21.15.1:0/2140943923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3bb80cff70 0x7f3bb80cc3f0 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0009d90 tx=0x7f3bb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddedf78b6c7964f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:56.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.965+0000 7f3bb77fe700 1 -- 172.21.15.1:0/2140943923 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb000e070 con 0x7f3bb80cff70 2024-03-18T20:26:56.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.965+0000 7f3bb77fe700 1 -- 172.21.15.1:0/2140943923 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb000bbd0 con 0x7f3bb80cff70 2024-03-18T20:26:56.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.965+0000 7f3bb77fe700 1 -- 172.21.15.1:0/2140943923 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb0012430 con 0x7f3bb80cff70 2024-03-18T20:26:56.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.965+0000 7f3bbed3f700 1 -- 172.21.15.1:0/2140943923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3bb80cff70 msgr2=0x7f3bb80cc3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:56.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.965+0000 7f3bbed3f700 1 --2- 172.21.15.1:0/2140943923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3bb80cff70 0x7f3bb80cc3f0 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0009d90 tx=0x7f3bb000b330 comp rx=0 tx=0).stop 2024-03-18T20:26:56.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.965+0000 7f3bbed3f700 1 -- 172.21.15.1:0/2140943923 shutdown_connections 2024-03-18T20:26:56.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.965+0000 7f3bbed3f700 1 --2- 172.21.15.1:0/2140943923 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3bb80cff70 0x7f3bb80cc3f0 unknown :-1 s=CLOSED pgs=2928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.965+0000 7f3bbed3f700 1 -- 172.21.15.1:0/2140943923 >> 172.21.15.1:0/2140943923 conn(0x7f3bb80d0ff0 msgr2=0x7f3bb80d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:56.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.966+0000 7f3bbed3f700 1 -- 172.21.15.1:0/2140943923 shutdown_connections 2024-03-18T20:26:56.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.966+0000 7f3bbed3f700 1 -- 172.21.15.1:0/2140943923 wait complete. 2024-03-18T20:26:56.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.967+0000 7f3bbed3f700 1 Processor -- start 2024-03-18T20:26:56.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.967+0000 7f3bbed3f700 1 -- start start 2024-03-18T20:26:56.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.968+0000 7f3bbed3f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3bb80b1e20 0x7f3bb80b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:56.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.968+0000 7f3bbed3f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3bb80cd870 0x7f3bb81f30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.968+0000 7f3bbed3f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3bb80cff70 0x7f3bb80ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.968+0000 7f3bbed3f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb80b0d60 con 0x7f3bb80cd870 2024-03-18T20:26:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.968+0000 7f3bbed3f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb80b1040 con 0x7f3bb80cff70 2024-03-18T20:26:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.968+0000 7f3bbed3f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb80b1320 con 0x7f3bb80b1e20 2024-03-18T20:26:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.968+0000 7f3bb7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3bb80cd870 0x7f3bb81f30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.968+0000 7f3bb7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3bb80cd870 0x7f3bb81f30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34750/0 (socket says 172.21.15.1:34750) 2024-03-18T20:26:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.968+0000 7f3bb7fff700 1 -- 172.21.15.1:0/871520973 learned_addr learned my addr 172.21.15.1:0/871520973 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.968+0000 7f3bbd2dc700 1 --2- 172.21.15.1:0/871520973 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3bb80cff70 0x7f3bb80ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.968+0000 7f3bbcadb700 1 --2- 172.21.15.1:0/871520973 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3bb80b1e20 0x7f3bb80b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.968+0000 7f3bb7fff700 1 -- 172.21.15.1:0/871520973 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3bb80b1e20 msgr2=0x7f3bb80b06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.968+0000 7f3bb7fff700 1 --2- 172.21.15.1:0/871520973 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3bb80b1e20 0x7f3bb80b06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.968+0000 7f3bb7fff700 1 -- 172.21.15.1:0/871520973 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3bb80cff70 msgr2=0x7f3bb80ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.968+0000 7f3bb7fff700 1 --2- 172.21.15.1:0/871520973 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3bb80cff70 0x7f3bb80ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.968+0000 7f3bb7fff700 1 -- 172.21.15.1:0/871520973 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb0009a70 con 0x7f3bb80cd870 2024-03-18T20:26:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.968+0000 7f3bb7fff700 1 --2- 172.21.15.1:0/871520973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3bb80cd870 0x7f3bb81f30f0 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7f3ba400c4e0 tx=0x7f3ba400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.969+0000 7f3bb57fa700 1 -- 172.21.15.1:0/871520973 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba40052d0 con 0x7f3bb80cd870 2024-03-18T20:26:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.969+0000 7f3bb57fa700 1 -- 172.21.15.1:0/871520973 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba4005430 con 0x7f3bb80cd870 2024-03-18T20:26:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.969+0000 7f3bb57fa700 1 -- 172.21.15.1:0/871520973 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba40038c0 con 0x7f3bb80cd870 2024-03-18T20:26:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.969+0000 7f3bbed3f700 1 -- 172.21.15.1:0/871520973 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb80b1600 con 0x7f3bb80cd870 2024-03-18T20:26:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.969+0000 7f3bbed3f700 1 -- 172.21.15.1:0/871520973 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb81f3630 con 0x7f3bb80cd870 2024-03-18T20:26:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.970+0000 7f3bbed3f700 1 -- 172.21.15.1:0/871520973 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb80b1b20 con 0x7f3bb80cd870 2024-03-18T20:26:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.970+0000 7f3bb57fa700 1 -- 172.21.15.1:0/871520973 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 152) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f3ba4003a20 con 0x7f3bb80cd870 2024-03-18T20:26:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.970+0000 7f3bb57fa700 1 --2- 172.21.15.1:0/871520973 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3b9c06aed0 0x7f3b9c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.971+0000 7f3bbcadb700 1 -- 172.21.15.1:0/871520973 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3b9c06aed0 msgr2=0x7f3b9c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.971+0000 7f3bb57fa700 1 -- 172.21.15.1:0/871520973 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7423+0+0 (secure 0 0 0) 0x7f3ba4089030 con 0x7f3bb80cd870 2024-03-18T20:26:56.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.971+0000 7f3bbcadb700 1 --2- 172.21.15.1:0/871520973 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3b9c06aed0 0x7f3b9c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:56.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:56.975+0000 7f3bb57fa700 1 -- 172.21.15.1:0/871520973 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3bb80b1b20 con 0x7f3bb80cd870 2024-03-18T20:26:57.026 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:57.021+0000 7f76988f6240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:26:57.104 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:57.099+0000 7f76988f6240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:26:57.150 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:57.147+0000 7f5294a3a240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:26:57.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.172+0000 7f3bbcadb700 1 -- 172.21.15.1:0/871520973 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3b9c06aed0 msgr2=0x7f3b9c06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:57.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.172+0000 7f3bbcadb700 1 --2- 172.21.15.1:0/871520973 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3b9c06aed0 0x7f3b9c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:57.181 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:57.177+0000 7f76988f6240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:26:57.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.195+0000 7f3bbed3f700 1 -- 172.21.15.1:0/871520973 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3bb80b1b20 con 0x7f3bb80cd870 2024-03-18T20:26:57.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.195+0000 7f3bb57fa700 1 -- 172.21.15.1:0/871520973 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v258) v1 ==== 81+0+565 (secure 0 0 0) 0x7f3ba4017250 con 0x7f3bb80cd870 2024-03-18T20:26:57.199 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2340206734 2024-03-19T20:26:52.380559+0000 2024-03-18T20:26:57.199 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6833/148356 2024-03-19T20:26:52.380559+0000 2024-03-18T20:26:57.199 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6832/148356 2024-03-19T20:26:52.380559+0000 2024-03-18T20:26:57.199 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/376843885 2024-03-19T20:26:52.380559+0000 2024-03-18T20:26:57.199 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1340983360 2024-03-19T20:26:52.380559+0000 2024-03-18T20:26:57.200 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/369631788 2024-03-19T20:26:52.380559+0000 2024-03-18T20:26:57.200 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2840085801 2024-03-19T20:26:52.380559+0000 2024-03-18T20:26:57.200 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/322251557 2024-03-19T20:26:52.380559+0000 2024-03-18T20:26:57.200 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3049322266 2024-03-19T20:26:52.380559+0000 2024-03-18T20:26:57.200 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1886004750 2024-03-19T20:26:52.380559+0000 2024-03-18T20:26:57.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.197+0000 7f3bbed3f700 1 -- 172.21.15.1:0/871520973 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3b9c06aed0 msgr2=0x7f3b9c06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:57.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.197+0000 7f3bbed3f700 1 --2- 172.21.15.1:0/871520973 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3b9c06aed0 0x7f3b9c06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:57.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.197+0000 7f3bbed3f700 1 -- 172.21.15.1:0/871520973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3bb80cd870 msgr2=0x7f3bb81f30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:57.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.197+0000 7f3bbed3f700 1 --2- 172.21.15.1:0/871520973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3bb80cd870 0x7f3bb81f30f0 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7f3ba400c4e0 tx=0x7f3ba400c7f0 comp rx=0 tx=0).stop 2024-03-18T20:26:57.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.197+0000 7f3bbed3f700 1 -- 172.21.15.1:0/871520973 shutdown_connections 2024-03-18T20:26:57.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.197+0000 7f3bbed3f700 1 --2- 172.21.15.1:0/871520973 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3b9c06aed0 0x7f3b9c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:57.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.197+0000 7f3bbed3f700 1 --2- 172.21.15.1:0/871520973 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3bb80b1e20 0x7f3bb80b06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:57.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.197+0000 7f3bbed3f700 1 --2- 172.21.15.1:0/871520973 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3bb80cff70 0x7f3bb80ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:57.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.197+0000 7f3bbed3f700 1 --2- 172.21.15.1:0/871520973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3bb80cd870 0x7f3bb81f30f0 unknown :-1 s=CLOSED pgs=2929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:57.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.197+0000 7f3bbed3f700 1 -- 172.21.15.1:0/871520973 >> 172.21.15.1:0/871520973 conn(0x7f3bb80d0ff0 msgr2=0x7f3bb806ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:57.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.198+0000 7f3bbed3f700 1 -- 172.21.15.1:0/871520973 shutdown_connections 2024-03-18T20:26:57.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.198+0000 7f3bbed3f700 1 -- 172.21.15.1:0/871520973 wait complete. 2024-03-18T20:26:57.203 INFO:teuthology.orchestra.run.smithi001.stderr:listed 10 entries 2024-03-18T20:26:57.214 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-18T20:26:57.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:57.214 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T20:26:57.230 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:57.226+0000 7f5294a3a240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:26:57.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[164619]: fuse finished with error 0 and tester_r 0 2024-03-18T20:26:57.257 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:26:57.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:57.257 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T20:26:57.260 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:57.256+0000 7f76988f6240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:26:57.318 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:57.314+0000 7f5294a3a240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:26:57.367 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:26:57.367 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:26:57.385 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:26:57.386 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:26:57.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:57.386 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-18T20:26:57.403 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:57.399+0000 7f5294a3a240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:26:57.435 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:57.431+0000 7f76988f6240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:26:57.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:26:57.444 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-18T20:26:57.444 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:26:57.444 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:26:57.463 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:26:57.463 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:26:57.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:57.463 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-18T20:26:57.487 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:57.483+0000 7f5294a3a240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:26:57.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:26:57.505 INFO:teuthology.orchestra.run.smithi001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-18T20:26:57.505 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:26:57.505 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:26:57.525 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:26:57.525 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:26:57.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:26:57.525 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-18T20:26:57.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:26:57.583 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-18T20:26:57.584 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:26:57.659 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:57.655+0000 7f76988f6240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:26:57.669 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:57.666+0000 7f5294a3a240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:26:57.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.707+0000 7fd1ff19f700 1 Processor -- start 2024-03-18T20:26:57.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.707+0000 7fd1ff19f700 1 -- start start 2024-03-18T20:26:57.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.708+0000 7fd1ff19f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1f80ae600 0x7fd1f80ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:57.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.708+0000 7fd1ff19f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1f815a2e0 0x7fd1f80ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:57.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.708+0000 7fd1ff19f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd1f80cc170 0x7fd1f806ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:57.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.708+0000 7fd1ff19f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1f806e270 con 0x7fd1f815a2e0 2024-03-18T20:26:57.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.708+0000 7fd1ff19f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1f80ae050 con 0x7fd1f80cc170 2024-03-18T20:26:57.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.708+0000 7fd1ff19f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f80aef20 con 0x7fd1f80ae600 2024-03-18T20:26:57.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.708+0000 7fd1f7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1f815a2e0 0x7fd1f80ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:57.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.708+0000 7fd1f7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd1f815a2e0 0x7fd1f80ae9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34754/0 (socket says 172.21.15.1:34754) 2024-03-18T20:26:57.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.708+0000 7fd1f7fff700 1 -- 172.21.15.1:0/63562647 learned_addr learned my addr 172.21.15.1:0/63562647 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:57.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.709+0000 7fd1fcf3b700 1 --2- 172.21.15.1:0/63562647 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1f80ae600 0x7fd1f80ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:57.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.709+0000 7fd1fd73c700 1 --2- 172.21.15.1:0/63562647 >> v2:172.21.15.165:3300/0 conn(0x7fd1f80cc170 0x7fd1f806ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:57.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.709+0000 7fd1f7fff700 1 -- 172.21.15.1:0/63562647 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1f80ae600 msgr2=0x7fd1f80ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:57.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.709+0000 7fd1f7fff700 1 --2- 172.21.15.1:0/63562647 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1f80ae600 0x7fd1f80ad790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:57.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.709+0000 7fd1f7fff700 1 -- 172.21.15.1:0/63562647 >> v2:172.21.15.165:3300/0 conn(0x7fd1f80cc170 msgr2=0x7fd1f806ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:57.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.709+0000 7fd1f7fff700 1 --2- 172.21.15.1:0/63562647 >> v2:172.21.15.165:3300/0 conn(0x7fd1f80cc170 0x7fd1f806ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:57.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.709+0000 7fd1f7fff700 1 -- 172.21.15.1:0/63562647 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f80add70 con 0x7fd1f815a2e0 2024-03-18T20:26:57.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.709+0000 7fd1f7fff700 1 --2- 172.21.15.1:0/63562647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f815a2e0 0x7fd1f80ae9e0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8071590 tx=0x7fd1e4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9cb477efa5c179a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:57.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.709+0000 7fd1f77fe700 1 -- 172.21.15.1:0/63562647 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e4011830 con 0x7fd1f815a2e0 2024-03-18T20:26:57.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.709+0000 7fd1f77fe700 1 -- 172.21.15.1:0/63562647 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e4011990 con 0x7fd1f815a2e0 2024-03-18T20:26:57.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.709+0000 7fd1f77fe700 1 -- 172.21.15.1:0/63562647 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e4011d20 con 0x7fd1f815a2e0 2024-03-18T20:26:57.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.710+0000 7fd1ff19f700 1 -- 172.21.15.1:0/63562647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f815a2e0 msgr2=0x7fd1f80ae9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:57.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.710+0000 7fd1ff19f700 1 --2- 172.21.15.1:0/63562647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f815a2e0 0x7fd1f80ae9e0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8071590 tx=0x7fd1e4009bd0 comp rx=0 tx=0).stop 2024-03-18T20:26:57.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.710+0000 7fd1ff19f700 1 -- 172.21.15.1:0/63562647 shutdown_connections 2024-03-18T20:26:57.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.710+0000 7fd1ff19f700 1 --2- 172.21.15.1:0/63562647 >> v2:172.21.15.165:3300/0 conn(0x7fd1f80cc170 0x7fd1f806ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:57.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.710+0000 7fd1ff19f700 1 --2- 172.21.15.1:0/63562647 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1f80ae600 0x7fd1f80ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:57.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.710+0000 7fd1ff19f700 1 --2- 172.21.15.1:0/63562647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f815a2e0 0x7fd1f80ae9e0 unknown :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:57.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.710+0000 7fd1ff19f700 1 -- 172.21.15.1:0/63562647 >> 172.21.15.1:0/63562647 conn(0x7fd1f806b860 msgr2=0x7fd1f806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:57.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.710+0000 7fd1ff19f700 1 -- 172.21.15.1:0/63562647 shutdown_connections 2024-03-18T20:26:57.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.710+0000 7fd1ff19f700 1 -- 172.21.15.1:0/63562647 wait complete. 2024-03-18T20:26:57.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.711+0000 7fd1ff19f700 1 Processor -- start 2024-03-18T20:26:57.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.711+0000 7fd1ff19f700 1 -- start start 2024-03-18T20:26:57.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.712+0000 7fd1ff19f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1f80ae600 0x7fd1f80cede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:57.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.712+0000 7fd1ff19f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f80cc170 0x7fd1f809a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:57.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.712+0000 7fd1ff19f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1f815a2e0 0x7fd1f81434e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:57.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.712+0000 7fd1ff19f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f80cc650 con 0x7fd1f80cc170 2024-03-18T20:26:57.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.712+0000 7fd1ff19f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f81540b0 con 0x7fd1f80ae600 2024-03-18T20:26:57.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.712+0000 7fd1ff19f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f80cf470 con 0x7fd1f815a2e0 2024-03-18T20:26:57.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.712+0000 7fd1f7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f80cc170 0x7fd1f809a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:57.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.712+0000 7fd1f7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f80cc170 0x7fd1f809a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34770/0 (socket says 172.21.15.1:34770) 2024-03-18T20:26:57.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.712+0000 7fd1f7fff700 1 -- 172.21.15.1:0/1691207062 learned_addr learned my addr 172.21.15.1:0/1691207062 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:57.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.712+0000 7fd1fd73c700 1 --2- 172.21.15.1:0/1691207062 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1f815a2e0 0x7fd1f81434e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:57.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.712+0000 7fd1fcf3b700 1 --2- 172.21.15.1:0/1691207062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1f80ae600 0x7fd1f80cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:57.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.713+0000 7fd1f7fff700 1 -- 172.21.15.1:0/1691207062 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1f815a2e0 msgr2=0x7fd1f81434e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:57.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.713+0000 7fd1f7fff700 1 --2- 172.21.15.1:0/1691207062 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1f815a2e0 0x7fd1f81434e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:57.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.713+0000 7fd1f7fff700 1 -- 172.21.15.1:0/1691207062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1f80ae600 msgr2=0x7fd1f80cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:57.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.713+0000 7fd1f7fff700 1 --2- 172.21.15.1:0/1691207062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1f80ae600 0x7fd1f80cede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:57.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.713+0000 7fd1f7fff700 1 -- 172.21.15.1:0/1691207062 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e4009550 con 0x7fd1f80cc170 2024-03-18T20:26:57.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.713+0000 7fd1f7fff700 1 --2- 172.21.15.1:0/1691207062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f80cc170 0x7fd1f809a430 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4000c00 tx=0x7fd1e4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:57.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.713+0000 7fd1f57fa700 1 -- 172.21.15.1:0/1691207062 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e4004240 con 0x7fd1f80cc170 2024-03-18T20:26:57.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.713+0000 7fd1f57fa700 1 -- 172.21.15.1:0/1691207062 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e401f070 con 0x7fd1f80cc170 2024-03-18T20:26:57.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.713+0000 7fd1ff19f700 1 -- 172.21.15.1:0/1691207062 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1f80cf750 con 0x7fd1f80cc170 2024-03-18T20:26:57.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.713+0000 7fd1f57fa700 1 -- 172.21.15.1:0/1691207062 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e40036a0 con 0x7fd1f80cc170 2024-03-18T20:26:57.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.713+0000 7fd1ff19f700 1 -- 172.21.15.1:0/1691207062 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1f80cc8a0 con 0x7fd1f80cc170 2024-03-18T20:26:57.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.714+0000 7fd1ff19f700 1 -- 172.21.15.1:0/1691207062 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1f80676a0 con 0x7fd1f80cc170 2024-03-18T20:26:57.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.715+0000 7fd1f57fa700 1 -- 172.21.15.1:0/1691207062 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 152) v1 ==== 89342+0+0 (secure 0 0 0) 0x7fd1e4021030 con 0x7fd1f80cc170 2024-03-18T20:26:57.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.715+0000 7fd1f57fa700 1 --2- 172.21.15.1:0/1691207062 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd1dc06aed0 0x7fd1dc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:57.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.715+0000 7fd1fcf3b700 1 -- 172.21.15.1:0/1691207062 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd1dc06aed0 msgr2=0x7fd1dc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:57.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.715+0000 7fd1f57fa700 1 -- 172.21.15.1:0/1691207062 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7423+0+0 (secure 0 0 0) 0x7fd1e408ae70 con 0x7fd1f80cc170 2024-03-18T20:26:57.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.715+0000 7fd1fcf3b700 1 --2- 172.21.15.1:0/1691207062 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd1dc06aed0 0x7fd1dc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:57.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.720+0000 7fd1f57fa700 1 -- 172.21.15.1:0/1691207062 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd1e40556d0 con 0x7fd1f80cc170 2024-03-18T20:26:57.813 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:57.808+0000 7f76988f6240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:26:57.887 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:57.883+0000 7f76988f6240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:26:57.898 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:57.894+0000 7f5294a3a240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:26:57.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.916+0000 7fd1fcf3b700 1 -- 172.21.15.1:0/1691207062 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd1dc06aed0 msgr2=0x7fd1dc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:57.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.916+0000 7fd1fcf3b700 1 --2- 172.21.15.1:0/1691207062 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd1dc06aed0 0x7fd1dc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:57.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.974+0000 7fd1ff19f700 1 -- 172.21.15.1:0/1691207062 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1f806c420 con 0x7fd1f80cc170 2024-03-18T20:26:57.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.975+0000 7fd1f57fa700 1 -- 172.21.15.1:0/1691207062 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+5963 (secure 0 0 0) 0x7fd1e4015090 con 0x7fd1f80cc170 2024-03-18T20:26:57.979 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:57.979 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":126,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12396,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2565889517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2565889517},{"type":"v1","addr":"172.21.15.1:6836","nonce":2565889517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":12399,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/512665659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":512665659},{"type":"v1","addr":"172.21.15.1:6837","nonce":512665659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:26:25.785074+0000","modified":"2024-03-18T20:26:29.101123+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12411":{"gid":12411,"name":"d","rank":0,"incarnation":116,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/3886150679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":3886150679},{"type":"v1","addr":"172.21.15.165:6835","nonce":3886150679}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:26:39.596187+0000","modified":"2024-03-18T20:26:42.614846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12408":{"gid":12408,"name":"b","rank":0,"incarnation":122,"state":"up:active","state_seq":6,"addr":"172.21.15.165:6834/1134181166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1134181166},{"type":"v1","addr":"172.21.15.165:6834","nonce":1134181166}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-03-18T20:26:57.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.977+0000 7fd1ff19f700 1 -- 172.21.15.1:0/1691207062 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd1dc06aed0 msgr2=0x7fd1dc06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:57.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.977+0000 7fd1ff19f700 1 --2- 172.21.15.1:0/1691207062 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd1dc06aed0 0x7fd1dc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:57.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.977+0000 7fd1ff19f700 1 -- 172.21.15.1:0/1691207062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f80cc170 msgr2=0x7fd1f809a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:57.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.977+0000 7fd1ff19f700 1 --2- 172.21.15.1:0/1691207062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f80cc170 0x7fd1f809a430 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4000c00 tx=0x7fd1e4004750 comp rx=0 tx=0).stop 2024-03-18T20:26:57.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.977+0000 7fd1ff19f700 1 -- 172.21.15.1:0/1691207062 shutdown_connections 2024-03-18T20:26:57.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.977+0000 7fd1ff19f700 1 --2- 172.21.15.1:0/1691207062 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd1dc06aed0 0x7fd1dc06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:57.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.977+0000 7fd1ff19f700 1 --2- 172.21.15.1:0/1691207062 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd1f815a2e0 0x7fd1f81434e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:57.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.977+0000 7fd1ff19f700 1 --2- 172.21.15.1:0/1691207062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd1f80ae600 0x7fd1f80cede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:57.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.977+0000 7fd1ff19f700 1 --2- 172.21.15.1:0/1691207062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1f80cc170 0x7fd1f809a430 unknown :-1 s=CLOSED pgs=2931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:57.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.977+0000 7fd1ff19f700 1 -- 172.21.15.1:0/1691207062 >> 172.21.15.1:0/1691207062 conn(0x7fd1f806b860 msgr2=0x7fd1f80b3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:57.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.977+0000 7fd1ff19f700 1 -- 172.21.15.1:0/1691207062 shutdown_connections 2024-03-18T20:26:57.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:57.977+0000 7fd1ff19f700 1 -- 172.21.15.1:0/1691207062 wait complete. 2024-03-18T20:26:57.983 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 126 2024-03-18T20:26:57.994 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:26:58.065 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:58.061+0000 7f5294a3a240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:26:58.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.117+0000 7f304afcf700 1 Processor -- start 2024-03-18T20:26:58.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.117+0000 7f304afcf700 1 -- start start 2024-03-18T20:26:58.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.118+0000 7f304afcf700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f304409a650 0x7f3044099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:58.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.118+0000 7f304afcf700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3044099f80 0x7f304406f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:58.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.118+0000 7f304afcf700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304406f070 con 0x7f304415b4d0 2024-03-18T20:26:58.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.118+0000 7f304afcf700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f304409a3f0 con 0x7f304409a650 2024-03-18T20:26:58.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.118+0000 7f304afcf700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30440d0260 con 0x7f3044099f80 2024-03-18T20:26:58.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.118+0000 7f3043fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f304409a650 0x7f3044099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:58.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.118+0000 7f3043fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f304409a650 0x7f3044099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34778/0 (socket says 172.21.15.1:34778) 2024-03-18T20:26:58.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.118+0000 7f3043fff700 1 -- 172.21.15.1:0/1028383994 learned_addr learned my addr 172.21.15.1:0/1028383994 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:58.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.118+0000 7f304956c700 1 --2- 172.21.15.1:0/1028383994 >> v2:172.21.15.165:3300/0 conn(0x7f3044099f80 0x7f304406f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:58.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.119+0000 7f3043fff700 1 -- 172.21.15.1:0/1028383994 >> v2:172.21.15.165:3300/0 conn(0x7f3044099f80 msgr2=0x7f304406f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:58.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.119+0000 7f3043fff700 1 --2- 172.21.15.1:0/1028383994 >> v2:172.21.15.165:3300/0 conn(0x7f3044099f80 0x7f304406f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.119+0000 7f3043fff700 1 -- 172.21.15.1:0/1028383994 >> v1:172.21.15.165:6789/0 conn(0x7f304415b4d0 legacy=0x7f30440cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:58.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.119+0000 7f3043fff700 1 -- 172.21.15.1:0/1028383994 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30440cff80 con 0x7f304409a650 2024-03-18T20:26:58.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.119+0000 7f3043fff700 1 --2- 172.21.15.1:0/1028383994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f304409a650 0x7f3044099a40 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 crypto rx=0x7f30440b2c40 tx=0x7f3030009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9db85618eb9451c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:58.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.119+0000 7f30437fe700 1 -- 172.21.15.1:0/1028383994 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3030011830 con 0x7f304409a650 2024-03-18T20:26:58.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.119+0000 7f30437fe700 1 -- 172.21.15.1:0/1028383994 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3030011990 con 0x7f304409a650 2024-03-18T20:26:58.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.119+0000 7f30437fe700 1 -- 172.21.15.1:0/1028383994 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3030011d20 con 0x7f304409a650 2024-03-18T20:26:58.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.120+0000 7f304afcf700 1 -- 172.21.15.1:0/1028383994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f304409a650 msgr2=0x7f3044099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:58.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.120+0000 7f304afcf700 1 --2- 172.21.15.1:0/1028383994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f304409a650 0x7f3044099a40 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 crypto rx=0x7f30440b2c40 tx=0x7f3030009bd0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.120+0000 7f304afcf700 1 -- 172.21.15.1:0/1028383994 shutdown_connections 2024-03-18T20:26:58.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.120+0000 7f304afcf700 1 --2- 172.21.15.1:0/1028383994 >> v2:172.21.15.165:3300/0 conn(0x7f3044099f80 0x7f304406f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.120+0000 7f304afcf700 1 --2- 172.21.15.1:0/1028383994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f304409a650 0x7f3044099a40 unknown :-1 s=CLOSED pgs=2932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.120+0000 7f304afcf700 1 -- 172.21.15.1:0/1028383994 >> 172.21.15.1:0/1028383994 conn(0x7f30440b2cb0 msgr2=0x7f30440b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:58.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.120+0000 7f304afcf700 1 -- 172.21.15.1:0/1028383994 shutdown_connections 2024-03-18T20:26:58.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.120+0000 7f304afcf700 1 -- 172.21.15.1:0/1028383994 wait complete. 2024-03-18T20:26:58.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.121+0000 7f304afcf700 1 Processor -- start 2024-03-18T20:26:58.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.121+0000 7f304afcf700 1 -- start start 2024-03-18T20:26:58.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.122+0000 7f304afcf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3044099f80 0x7f30440c4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:58.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.122+0000 7f304afcf700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f304409a650 0x7f30441f4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:58.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.122+0000 7f304afcf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f304415b4d0 0x7f30441f47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:58.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.122+0000 7f304afcf700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30440c97d0 con 0x7f3044099f80 2024-03-18T20:26:58.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.122+0000 7f304afcf700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304415bd50 con 0x7f304409a650 2024-03-18T20:26:58.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.122+0000 7f304afcf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304415ea00 con 0x7f304415b4d0 2024-03-18T20:26:58.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.122+0000 7f3048d6b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3044099f80 0x7f30440c4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:58.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.122+0000 7f3043fff700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f304409a650 0x7f30441f4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:58.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.122+0000 7f3048d6b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3044099f80 0x7f30440c4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34788/0 (socket says 172.21.15.1:34788) 2024-03-18T20:26:58.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.122+0000 7f3048d6b700 1 -- 172.21.15.1:0/3993823842 learned_addr learned my addr 172.21.15.1:0/3993823842 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:58.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.122+0000 7f304956c700 1 --2- 172.21.15.1:0/3993823842 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f304415b4d0 0x7f30441f47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:58.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.122+0000 7f3048d6b700 1 -- 172.21.15.1:0/3993823842 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f304415b4d0 msgr2=0x7f30441f47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:58.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.122+0000 7f3048d6b700 1 --2- 172.21.15.1:0/3993823842 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f304415b4d0 0x7f30441f47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.122+0000 7f3048d6b700 1 -- 172.21.15.1:0/3993823842 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f304409a650 msgr2=0x7f30441f4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:58.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.122+0000 7f3048d6b700 1 --2- 172.21.15.1:0/3993823842 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f304409a650 0x7f30441f4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.122+0000 7f3048d6b700 1 -- 172.21.15.1:0/3993823842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3030009550 con 0x7f3044099f80 2024-03-18T20:26:58.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.122+0000 7f3043fff700 1 --2- 172.21.15.1:0/3993823842 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f304409a650 0x7f30441f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:26:58.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.122+0000 7f304956c700 1 --2- 172.21.15.1:0/3993823842 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f304415b4d0 0x7f30441f47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:26:58.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.123+0000 7f3048d6b700 1 --2- 172.21.15.1:0/3993823842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3044099f80 0x7f30440c4f80 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7f303800c450 tx=0x7f303800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:58.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.123+0000 7f30417fa700 1 -- 172.21.15.1:0/3993823842 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30380052d0 con 0x7f3044099f80 2024-03-18T20:26:58.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.123+0000 7f30417fa700 1 -- 172.21.15.1:0/3993823842 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3038005430 con 0x7f3044099f80 2024-03-18T20:26:58.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.123+0000 7f30417fa700 1 -- 172.21.15.1:0/3993823842 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30380038c0 con 0x7f3044099f80 2024-03-18T20:26:58.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.123+0000 7f304afcf700 1 -- 172.21.15.1:0/3993823842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3044148210 con 0x7f3044099f80 2024-03-18T20:26:58.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.123+0000 7f304afcf700 1 -- 172.21.15.1:0/3993823842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f304415e720 con 0x7f3044099f80 2024-03-18T20:26:58.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.124+0000 7f304afcf700 1 -- 172.21.15.1:0/3993823842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3044068240 con 0x7f3044099f80 2024-03-18T20:26:58.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.125+0000 7f30417fa700 1 -- 172.21.15.1:0/3993823842 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 152) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f3038003a20 con 0x7f3044099f80 2024-03-18T20:26:58.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.126+0000 7f30417fa700 1 --2- 172.21.15.1:0/3993823842 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302806aed0 0x7f302806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:58.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.126+0000 7f3043fff700 1 -- 172.21.15.1:0/3993823842 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302806aed0 msgr2=0x7f302806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:58.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.126+0000 7f30417fa700 1 -- 172.21.15.1:0/3993823842 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7423+0+0 (secure 0 0 0) 0x7f3038089340 con 0x7f3044099f80 2024-03-18T20:26:58.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.126+0000 7f3043fff700 1 --2- 172.21.15.1:0/3993823842 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302806aed0 0x7f302806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:58.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.130+0000 7f30417fa700 1 -- 172.21.15.1:0/3993823842 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3038053d50 con 0x7f3044099f80 2024-03-18T20:26:58.146 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:58.142+0000 7f5294a3a240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:26:58.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.327+0000 7f3043fff700 1 -- 172.21.15.1:0/3993823842 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302806aed0 msgr2=0x7f302806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:58.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.327+0000 7f3043fff700 1 --2- 172.21.15.1:0/3993823842 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302806aed0 0x7f302806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:58.333 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:58.329+0000 7f76988f6240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:26:58.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.335+0000 7f304afcf700 1 -- 172.21.15.1:0/3993823842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30440676a0 con 0x7f3044099f80 2024-03-18T20:26:58.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.337+0000 7f30417fa700 1 -- 172.21.15.1:0/3993823842 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v258) v1 ==== 74+0+20693 (secure 0 0 0) 0x7f3038017020 con 0x7f3044099f80 2024-03-18T20:26:58.341 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:26:58.341 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":258,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:26:52.380634+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:26:23.615691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-18T20:26:24.759650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:26:37.326244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:26:38.564628+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"257","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2340206734":"2024-03-19T20:26:52.380559+0000","172.21.15.1:6833/148356":"2024-03-19T20:26:52.380559+0000","172.21.15.1:6832/148356":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/376843885":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/1340983360":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/369631788":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/2840085801":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/322251557":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/3049322266":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/1886004750":"2024-03-19T20:26:52.380559+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:26:58.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.339+0000 7f304afcf700 1 -- 172.21.15.1:0/3993823842 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302806aed0 msgr2=0x7f302806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:58.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.339+0000 7f304afcf700 1 --2- 172.21.15.1:0/3993823842 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302806aed0 0x7f302806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.339+0000 7f304afcf700 1 -- 172.21.15.1:0/3993823842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3044099f80 msgr2=0x7f30440c4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:58.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.339+0000 7f304afcf700 1 --2- 172.21.15.1:0/3993823842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3044099f80 0x7f30440c4f80 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7f303800c450 tx=0x7f303800c760 comp rx=0 tx=0).stop 2024-03-18T20:26:58.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.339+0000 7f304afcf700 1 -- 172.21.15.1:0/3993823842 shutdown_connections 2024-03-18T20:26:58.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.339+0000 7f304afcf700 1 --2- 172.21.15.1:0/3993823842 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302806aed0 0x7f302806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.339+0000 7f304afcf700 1 --2- 172.21.15.1:0/3993823842 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f304415b4d0 0x7f30441f47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.339+0000 7f304afcf700 1 --2- 172.21.15.1:0/3993823842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3044099f80 0x7f30440c4f80 unknown :-1 s=CLOSED pgs=2933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.339+0000 7f304afcf700 1 --2- 172.21.15.1:0/3993823842 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f304409a650 0x7f30441f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.339+0000 7f304afcf700 1 -- 172.21.15.1:0/3993823842 >> 172.21.15.1:0/3993823842 conn(0x7f30440b2cb0 msgr2=0x7f30440cc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:58.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.339+0000 7f304afcf700 1 -- 172.21.15.1:0/3993823842 shutdown_connections 2024-03-18T20:26:58.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.339+0000 7f304afcf700 1 -- 172.21.15.1:0/3993823842 wait complete. 2024-03-18T20:26:58.356 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:26:23.615691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '252', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:26:24.759650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '253', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-03-18T20:26:37.326244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '256', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2024-03-18T20:26:38.564628+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '257', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:26:58.357 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:26:58.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.482+0000 7ffbcf6f5700 1 Processor -- start 2024-03-18T20:26:58.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.483+0000 7ffbcf6f5700 1 -- start start 2024-03-18T20:26:58.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.483+0000 7ffbcf6f5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbc815b6c0 0x7ffbc80b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:58.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.483+0000 7ffbcf6f5700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc80d01f0 con 0x7ffbc80cd430 2024-03-18T20:26:58.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.483+0000 7ffbcf6f5700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc80ccfa0 con 0x7ffbc80c87d0 2024-03-18T20:26:58.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.483+0000 7ffbcf6f5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc80d1db0 con 0x7ffbc815b6c0 2024-03-18T20:26:58.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.484+0000 7ffbccc90700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ffbc80cd430 0x7ffbc80c80c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45030/0 (socket says 172.21.15.1:45030) 2024-03-18T20:26:58.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.484+0000 7ffbccc90700 1 -- 172.21.15.1:0/3270989317 learned_addr learned my addr 172.21.15.1:0/3270989317 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:58.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.484+0000 7ffbcd491700 1 --2- 172.21.15.1:0/3270989317 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbc815b6c0 0x7ffbc80b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:58.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.484+0000 7ffbbffff700 1 -- 172.21.15.1:0/3270989317 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672359086 0 0) 0x7ffbc80d01f0 con 0x7ffbc80cd430 2024-03-18T20:26:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.484+0000 7ffbbffff700 1 -- 172.21.15.1:0/3270989317 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbb40035b0 con 0x7ffbc80cd430 2024-03-18T20:26:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.484+0000 7ffbbffff700 1 -- 172.21.15.1:0/3270989317 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336246435 0 0) 0x7ffbc80ccfa0 con 0x7ffbc80c87d0 2024-03-18T20:26:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.484+0000 7ffbbffff700 1 -- 172.21.15.1:0/3270989317 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbc80d01f0 con 0x7ffbc80c87d0 2024-03-18T20:26:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.484+0000 7ffbbffff700 1 -- 172.21.15.1:0/3270989317 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 456183483 0 0) 0x7ffbb40035b0 con 0x7ffbc80cd430 2024-03-18T20:26:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.484+0000 7ffbcd491700 1 -- 172.21.15.1:0/3270989317 >> v1:172.21.15.165:6789/0 conn(0x7ffbc80c87d0 legacy=0x7ffbc80cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.484+0000 7ffbcd491700 1 -- 172.21.15.1:0/3270989317 >> v1:172.21.15.1:6789/0 conn(0x7ffbc80cd430 legacy=0x7ffbc80c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.484+0000 7ffbcd491700 1 -- 172.21.15.1:0/3270989317 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc80cff10 con 0x7ffbc815b6c0 2024-03-18T20:26:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.484+0000 7ffbcd491700 1 --2- 172.21.15.1:0/3270989317 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbc815b6c0 0x7ffbc80b2160 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4009870 tx=0x7ffbc4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a21566c14dc72a3c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.484+0000 7ffbbffff700 1 -- 172.21.15.1:0/3270989317 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb8001670 con 0x7ffbc815b6c0 2024-03-18T20:26:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.484+0000 7ffbbffff700 1 -- 172.21.15.1:0/3270989317 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc4004030 con 0x7ffbc815b6c0 2024-03-18T20:26:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.484+0000 7ffbbffff700 1 -- 172.21.15.1:0/3270989317 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc40043c0 con 0x7ffbc815b6c0 2024-03-18T20:26:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.485+0000 7ffbcf6f5700 1 -- 172.21.15.1:0/3270989317 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbc815b6c0 msgr2=0x7ffbc80b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.485+0000 7ffbcf6f5700 1 --2- 172.21.15.1:0/3270989317 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbc815b6c0 0x7ffbc80b2160 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4009870 tx=0x7ffbc4009b80 comp rx=0 tx=0).stop 2024-03-18T20:26:58.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.485+0000 7ffbcf6f5700 1 -- 172.21.15.1:0/3270989317 shutdown_connections 2024-03-18T20:26:58.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.485+0000 7ffbcf6f5700 1 --2- 172.21.15.1:0/3270989317 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbc815b6c0 0x7ffbc80b2160 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.485+0000 7ffbcf6f5700 1 -- 172.21.15.1:0/3270989317 >> 172.21.15.1:0/3270989317 conn(0x7ffbc80d0ff0 msgr2=0x7ffbc80d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:58.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.485+0000 7ffbcf6f5700 1 -- 172.21.15.1:0/3270989317 shutdown_connections 2024-03-18T20:26:58.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.485+0000 7ffbcf6f5700 1 -- 172.21.15.1:0/3270989317 wait complete. 2024-03-18T20:26:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.486+0000 7ffbcf6f5700 1 Processor -- start 2024-03-18T20:26:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.486+0000 7ffbcf6f5700 1 -- start start 2024-03-18T20:26:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.487+0000 7ffbcf6f5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbc80c87d0 0x7ffbc809b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.487+0000 7ffbcf6f5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffbc80cd430 0x7ffbc80b5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.487+0000 7ffbcf6f5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbc815b6c0 0x7ffbc81f0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.487+0000 7ffbcf6f5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc80b5e90 con 0x7ffbc80c87d0 2024-03-18T20:26:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.487+0000 7ffbcf6f5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc80b2c40 con 0x7ffbc80cd430 2024-03-18T20:26:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.487+0000 7ffbcf6f5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc80b6110 con 0x7ffbc815b6c0 2024-03-18T20:26:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.487+0000 7ffbcd491700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbc80c87d0 0x7ffbc809b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:58.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.487+0000 7ffbcd491700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbc80c87d0 0x7ffbc809b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34790/0 (socket says 172.21.15.1:34790) 2024-03-18T20:26:58.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.487+0000 7ffbcd491700 1 -- 172.21.15.1:0/2055729784 learned_addr learned my addr 172.21.15.1:0/2055729784 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:58.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.487+0000 7ffbccc90700 1 --2- 172.21.15.1:0/2055729784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffbc80cd430 0x7ffbc80b5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:58.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.487+0000 7ffbcdc92700 1 --2- 172.21.15.1:0/2055729784 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbc815b6c0 0x7ffbc81f0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:58.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.487+0000 7ffbcd491700 1 -- 172.21.15.1:0/2055729784 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbc815b6c0 msgr2=0x7ffbc81f0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:58.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.487+0000 7ffbcd491700 1 --2- 172.21.15.1:0/2055729784 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbc815b6c0 0x7ffbc81f0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.488+0000 7ffbcd491700 1 -- 172.21.15.1:0/2055729784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffbc80cd430 msgr2=0x7ffbc80b5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:58.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.488+0000 7ffbcd491700 1 --2- 172.21.15.1:0/2055729784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffbc80cd430 0x7ffbc80b5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.488+0000 7ffbcd491700 1 -- 172.21.15.1:0/2055729784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc40094a0 con 0x7ffbc80c87d0 2024-03-18T20:26:58.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.488+0000 7ffbcdc92700 1 --2- 172.21.15.1:0/2055729784 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbc815b6c0 0x7ffbc81f0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:26:58.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.488+0000 7ffbcd491700 1 --2- 172.21.15.1:0/2055729784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbc80c87d0 0x7ffbc809b140 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4000c00 tx=0x7ffbc4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:58.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.488+0000 7ffbbdffb700 1 -- 172.21.15.1:0/2055729784 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4003d80 con 0x7ffbc80c87d0 2024-03-18T20:26:58.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.488+0000 7ffbbdffb700 1 -- 172.21.15.1:0/2055729784 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc401f070 con 0x7ffbc80c87d0 2024-03-18T20:26:58.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.488+0000 7ffbcf6f5700 1 -- 172.21.15.1:0/2055729784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc80b63f0 con 0x7ffbc80c87d0 2024-03-18T20:26:58.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.488+0000 7ffbbdffb700 1 -- 172.21.15.1:0/2055729784 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4011870 con 0x7ffbc80c87d0 2024-03-18T20:26:58.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.488+0000 7ffbcf6f5700 1 -- 172.21.15.1:0/2055729784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc81f1140 con 0x7ffbc80c87d0 2024-03-18T20:26:58.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.489+0000 7ffbcf6f5700 1 -- 172.21.15.1:0/2055729784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc80b6910 con 0x7ffbc80c87d0 2024-03-18T20:26:58.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.490+0000 7ffbbdffb700 1 -- 172.21.15.1:0/2055729784 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 152) v1 ==== 89342+0+0 (secure 0 0 0) 0x7ffbc4021030 con 0x7ffbc80c87d0 2024-03-18T20:26:58.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.490+0000 7ffbbdffb700 1 --2- 172.21.15.1:0/2055729784 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffbac06af00 0x7ffbac06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:58.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.490+0000 7ffbccc90700 1 -- 172.21.15.1:0/2055729784 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffbac06af00 msgr2=0x7ffbac06d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:58.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.490+0000 7ffbbdffb700 1 -- 172.21.15.1:0/2055729784 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7423+0+0 (secure 0 0 0) 0x7ffbc408b000 con 0x7ffbc80c87d0 2024-03-18T20:26:58.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.490+0000 7ffbccc90700 1 --2- 172.21.15.1:0/2055729784 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffbac06af00 0x7ffbac06d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:58.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.495+0000 7ffbbdffb700 1 -- 172.21.15.1:0/2055729784 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffbc4055980 con 0x7ffbc80c87d0 2024-03-18T20:26:58.633 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:58.629+0000 7f5294a3a240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:26:58.670 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:58.666+0000 7f76988f6240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:26:58.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.691+0000 7ffbccc90700 1 -- 172.21.15.1:0/2055729784 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffbac06af00 msgr2=0x7ffbac06d3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:58.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.691+0000 7ffbccc90700 1 --2- 172.21.15.1:0/2055729784 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffbac06af00 0x7ffbac06d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:58.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.697+0000 7ffbcf6f5700 1 -- 172.21.15.1:0/2055729784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffbc80b6d10 con 0x7ffbc80c87d0 2024-03-18T20:26:58.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.697+0000 7ffbbdffb700 1 -- 172.21.15.1:0/2055729784 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v258) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffbc4058fa0 con 0x7ffbc80c87d0 2024-03-18T20:26:58.701 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:26:58.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.699+0000 7ffbcf6f5700 1 -- 172.21.15.1:0/2055729784 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffbac06af00 msgr2=0x7ffbac06d3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:58.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.699+0000 7ffbcf6f5700 1 --2- 172.21.15.1:0/2055729784 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffbac06af00 0x7ffbac06d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.699+0000 7ffbcf6f5700 1 -- 172.21.15.1:0/2055729784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbc80c87d0 msgr2=0x7ffbc809b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:58.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.700+0000 7ffbcf6f5700 1 --2- 172.21.15.1:0/2055729784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbc80c87d0 0x7ffbc809b140 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4000c00 tx=0x7ffbc4004750 comp rx=0 tx=0).stop 2024-03-18T20:26:58.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.700+0000 7ffbcf6f5700 1 -- 172.21.15.1:0/2055729784 shutdown_connections 2024-03-18T20:26:58.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.700+0000 7ffbcf6f5700 1 --2- 172.21.15.1:0/2055729784 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffbac06af00 0x7ffbac06d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.700+0000 7ffbcf6f5700 1 --2- 172.21.15.1:0/2055729784 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffbc815b6c0 0x7ffbc81f0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.700+0000 7ffbcf6f5700 1 --2- 172.21.15.1:0/2055729784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbc80c87d0 0x7ffbc809b140 unknown :-1 s=CLOSED pgs=2935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.700+0000 7ffbcf6f5700 1 --2- 172.21.15.1:0/2055729784 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffbc80cd430 0x7ffbc80b5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.700+0000 7ffbcf6f5700 1 -- 172.21.15.1:0/2055729784 >> 172.21.15.1:0/2055729784 conn(0x7ffbc80d0ff0 msgr2=0x7ffbc80c9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:58.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.700+0000 7ffbcf6f5700 1 -- 172.21.15.1:0/2055729784 shutdown_connections 2024-03-18T20:26:58.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.700+0000 7ffbcf6f5700 1 -- 172.21.15.1:0/2055729784 wait complete. 2024-03-18T20:26:58.718 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:26:58.833 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:58.829+0000 7f76988f6240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:26:58.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.846+0000 7fbc5bfb9700 1 Processor -- start 2024-03-18T20:26:58.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.846+0000 7fbc5bfb9700 1 -- start start 2024-03-18T20:26:58.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.847+0000 7fbc5bfb9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbc5415b480 0x7fbc540cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:58.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.847+0000 7fbc5bfb9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbc540cc7b0 0x7fbc540cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:58.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.847+0000 7fbc5bfb9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc540c4c50 con 0x7fbc540cfc00 2024-03-18T20:26:58.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.847+0000 7fbc5bfb9700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc540d0080 con 0x7fbc5415b480 2024-03-18T20:26:58.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.847+0000 7fbc5bfb9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc540c5210 con 0x7fbc540cc7b0 2024-03-18T20:26:58.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.847+0000 7fbc5a556700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbc540cfc00 0x7fbc54099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45034/0 (socket says 172.21.15.1:45034) 2024-03-18T20:26:58.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.847+0000 7fbc5a556700 1 -- 172.21.15.1:0/2330506397 learned_addr learned my addr 172.21.15.1:0/2330506397 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:58.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.847+0000 7fbc59554700 1 --2- 172.21.15.1:0/2330506397 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbc540cc7b0 0x7fbc540cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:58.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.847+0000 7fbc59d55700 1 --2- 172.21.15.1:0/2330506397 >> v2:172.21.15.165:3300/0 conn(0x7fbc5415b480 0x7fbc540cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:58.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.847+0000 7fbc58d53700 1 -- 172.21.15.1:0/2330506397 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127595246 0 0) 0x7fbc540c4c50 con 0x7fbc540cfc00 2024-03-18T20:26:58.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.847+0000 7fbc58d53700 1 -- 172.21.15.1:0/2330506397 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc400035b0 con 0x7fbc540cfc00 2024-03-18T20:26:58.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.847+0000 7fbc59554700 1 -- 172.21.15.1:0/2330506397 >> v2:172.21.15.165:3300/0 conn(0x7fbc5415b480 msgr2=0x7fbc540cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:58.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.847+0000 7fbc59554700 1 --2- 172.21.15.1:0/2330506397 >> v2:172.21.15.165:3300/0 conn(0x7fbc5415b480 0x7fbc540cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.847+0000 7fbc59554700 1 -- 172.21.15.1:0/2330506397 >> v1:172.21.15.1:6789/0 conn(0x7fbc540cfc00 legacy=0x7fbc54099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:58.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.847+0000 7fbc59554700 1 -- 172.21.15.1:0/2330506397 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc540cccb0 con 0x7fbc540cc7b0 2024-03-18T20:26:58.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.847+0000 7fbc59d55700 1 --2- 172.21.15.1:0/2330506397 >> v2:172.21.15.165:3300/0 conn(0x7fbc5415b480 0x7fbc540cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:26:58.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.848+0000 7fbc59554700 1 --2- 172.21.15.1:0/2330506397 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbc540cc7b0 0x7fbc540cf6c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbc44009840 tx=0x7fbc44009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=134caadbcf984d0b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:58.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.848+0000 7fbc58d53700 1 -- 172.21.15.1:0/2330506397 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc44011830 con 0x7fbc540cc7b0 2024-03-18T20:26:58.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.848+0000 7fbc58d53700 1 -- 172.21.15.1:0/2330506397 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc44011990 con 0x7fbc540cc7b0 2024-03-18T20:26:58.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.848+0000 7fbc58d53700 1 -- 172.21.15.1:0/2330506397 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc44011d20 con 0x7fbc540cc7b0 2024-03-18T20:26:58.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.848+0000 7fbc5bfb9700 1 -- 172.21.15.1:0/2330506397 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbc540cc7b0 msgr2=0x7fbc540cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:58.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.848+0000 7fbc5bfb9700 1 --2- 172.21.15.1:0/2330506397 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbc540cc7b0 0x7fbc540cf6c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbc44009840 tx=0x7fbc44009b50 comp rx=0 tx=0).stop 2024-03-18T20:26:58.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.848+0000 7fbc5bfb9700 1 -- 172.21.15.1:0/2330506397 shutdown_connections 2024-03-18T20:26:58.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.848+0000 7fbc5bfb9700 1 --2- 172.21.15.1:0/2330506397 >> v2:172.21.15.165:3300/0 conn(0x7fbc5415b480 0x7fbc540cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.848+0000 7fbc5bfb9700 1 --2- 172.21.15.1:0/2330506397 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbc540cc7b0 0x7fbc540cf6c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.848+0000 7fbc5bfb9700 1 -- 172.21.15.1:0/2330506397 >> 172.21.15.1:0/2330506397 conn(0x7fbc540b2bf0 msgr2=0x7fbc540b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:58.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.849+0000 7fbc5bfb9700 1 -- 172.21.15.1:0/2330506397 shutdown_connections 2024-03-18T20:26:58.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.849+0000 7fbc5bfb9700 1 -- 172.21.15.1:0/2330506397 wait complete. 2024-03-18T20:26:58.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.850+0000 7fbc5bfb9700 1 Processor -- start 2024-03-18T20:26:58.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.850+0000 7fbc5bfb9700 1 -- start start 2024-03-18T20:26:58.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.850+0000 7fbc5bfb9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc540cc7b0 0x7fbc54153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:58.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.850+0000 7fbc5bfb9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbc540cfc00 0x7fbc540b6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:58.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc5bfb9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc5415b480 0x7fbc540b4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:58.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc5bfb9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc540b5390 con 0x7fbc5415b480 2024-03-18T20:26:58.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc5bfb9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5406c6c0 con 0x7fbc540cc7b0 2024-03-18T20:26:58.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc5bfb9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5406c9a0 con 0x7fbc540cfc00 2024-03-18T20:26:58.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc5a556700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc5415b480 0x7fbc540b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:58.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc5a556700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc5415b480 0x7fbc540b4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34802/0 (socket says 172.21.15.1:34802) 2024-03-18T20:26:58.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc5a556700 1 -- 172.21.15.1:0/3378863056 learned_addr learned my addr 172.21.15.1:0/3378863056 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:58.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc59554700 1 --2- 172.21.15.1:0/3378863056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbc540cfc00 0x7fbc540b6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:58.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc59d55700 1 --2- 172.21.15.1:0/3378863056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc540cc7b0 0x7fbc54153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc5a556700 1 -- 172.21.15.1:0/3378863056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbc540cfc00 msgr2=0x7fbc540b6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc5a556700 1 --2- 172.21.15.1:0/3378863056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbc540cfc00 0x7fbc540b6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc5a556700 1 -- 172.21.15.1:0/3378863056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc540cc7b0 msgr2=0x7fbc54153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc5a556700 1 --2- 172.21.15.1:0/3378863056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc540cc7b0 0x7fbc54153550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc5a556700 1 -- 172.21.15.1:0/3378863056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc500094a0 con 0x7fbc5415b480 2024-03-18T20:26:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc5a556700 1 --2- 172.21.15.1:0/3378863056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc5415b480 0x7fbc540b4510 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00cbb0 tx=0x7fbc4c00cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc4a7fc700 1 -- 172.21.15.1:0/3378863056 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c00a9e0 con 0x7fbc5415b480 2024-03-18T20:26:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc4a7fc700 1 -- 172.21.15.1:0/3378863056 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc4c00ab40 con 0x7fbc5415b480 2024-03-18T20:26:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc4a7fc700 1 -- 172.21.15.1:0/3378863056 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c012400 con 0x7fbc5415b480 2024-03-18T20:26:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc5bfb9700 1 -- 172.21.15.1:0/3378863056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc440094a0 con 0x7fbc5415b480 2024-03-18T20:26:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.851+0000 7fbc5bfb9700 1 -- 172.21.15.1:0/3378863056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc5406cf60 con 0x7fbc5415b480 2024-03-18T20:26:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.853+0000 7fbc5bfb9700 1 -- 172.21.15.1:0/3378863056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc540676a0 con 0x7fbc5415b480 2024-03-18T20:26:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.853+0000 7fbc4a7fc700 1 -- 172.21.15.1:0/3378863056 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 152) v1 ==== 89342+0+0 (secure 0 0 0) 0x7fbc4c01b3b0 con 0x7fbc5415b480 2024-03-18T20:26:58.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.853+0000 7fbc4a7fc700 1 --2- 172.21.15.1:0/3378863056 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbc3806aed0 0x7fbc3806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:58.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.853+0000 7fbc59d55700 1 -- 172.21.15.1:0/3378863056 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbc3806aed0 msgr2=0x7fbc3806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:58.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.853+0000 7fbc4a7fc700 1 -- 172.21.15.1:0/3378863056 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7423+0+0 (secure 0 0 0) 0x7fbc4c089a00 con 0x7fbc5415b480 2024-03-18T20:26:58.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.853+0000 7fbc59d55700 1 --2- 172.21.15.1:0/3378863056 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbc3806aed0 0x7fbc3806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:58.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:58.858+0000 7fbc4a7fc700 1 -- 172.21.15.1:0/3378863056 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbc4c0542d0 con 0x7fbc5415b480 2024-03-18T20:26:58.998 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:58.993+0000 7f5294a3a240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:26:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.054+0000 7fbc59d55700 1 -- 172.21.15.1:0/3378863056 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbc3806aed0 msgr2=0x7fbc3806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.054+0000 7fbc59d55700 1 --2- 172.21.15.1:0/3378863056 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbc3806aed0 0x7fbc3806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:59.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.060+0000 7fbc5bfb9700 1 -- 172.21.15.1:0/3378863056 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbc540ceb70 con 0x7fbc5415b480 2024-03-18T20:26:59.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.060+0000 7fbc4a7fc700 1 -- 172.21.15.1:0/3378863056 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v258) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbc4c0578f0 con 0x7fbc5415b480 2024-03-18T20:26:59.064 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:26:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.063+0000 7fbc5bfb9700 1 -- 172.21.15.1:0/3378863056 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbc3806aed0 msgr2=0x7fbc3806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.063+0000 7fbc5bfb9700 1 --2- 172.21.15.1:0/3378863056 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbc3806aed0 0x7fbc3806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.063+0000 7fbc5bfb9700 1 -- 172.21.15.1:0/3378863056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc5415b480 msgr2=0x7fbc540b4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.063+0000 7fbc5bfb9700 1 --2- 172.21.15.1:0/3378863056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc5415b480 0x7fbc540b4510 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00cbb0 tx=0x7fbc4c00cec0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.063+0000 7fbc5bfb9700 1 -- 172.21.15.1:0/3378863056 shutdown_connections 2024-03-18T20:26:59.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.063+0000 7fbc5bfb9700 1 --2- 172.21.15.1:0/3378863056 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbc3806aed0 0x7fbc3806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.063+0000 7fbc5bfb9700 1 --2- 172.21.15.1:0/3378863056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc5415b480 0x7fbc540b4510 unknown :-1 s=CLOSED pgs=2937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.063+0000 7fbc5bfb9700 1 --2- 172.21.15.1:0/3378863056 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc540cc7b0 0x7fbc54153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.063+0000 7fbc5bfb9700 1 --2- 172.21.15.1:0/3378863056 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbc540cfc00 0x7fbc540b6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.063+0000 7fbc5bfb9700 1 -- 172.21.15.1:0/3378863056 >> 172.21.15.1:0/3378863056 conn(0x7fbc540b2bf0 msgr2=0x7fbc54159b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:59.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.063+0000 7fbc5bfb9700 1 -- 172.21.15.1:0/3378863056 shutdown_connections 2024-03-18T20:26:59.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.063+0000 7fbc5bfb9700 1 -- 172.21.15.1:0/3378863056 wait complete. 2024-03-18T20:26:59.079 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:26:59.178 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:59.174+0000 7f5294a3a240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:26:59.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.205+0000 7f25ffec5700 1 Processor -- start 2024-03-18T20:26:59.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.205+0000 7f25ffec5700 1 -- start start 2024-03-18T20:26:59.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.206+0000 7f25ffec5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f25f813e130 0x7f25f813e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:59.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.206+0000 7f25ffec5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25f815b750 0x7f25f815bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:59.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.206+0000 7f25ffec5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f25f8070db0 0x7f25f8071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:59.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.206+0000 7f25ffec5700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f8071760 con 0x7f25f8070db0 2024-03-18T20:26:59.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.206+0000 7f25ffec5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f80cbcc0 con 0x7f25f813e130 2024-03-18T20:26:59.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.206+0000 7f25ffec5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f815c210 con 0x7f25f815b750 2024-03-18T20:26:59.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.206+0000 7f25fe462700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f25f8070db0 0x7f25f8071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:59.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.206+0000 7f25fe462700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f25f8070db0 0x7f25f8071220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34818/0 (socket says 172.21.15.1:34818) 2024-03-18T20:26:59.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.206+0000 7f25fe462700 1 -- 172.21.15.1:0/588232006 learned_addr learned my addr 172.21.15.1:0/588232006 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:59.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.206+0000 7f25fdc61700 1 --2- 172.21.15.1:0/588232006 >> v2:172.21.15.165:3300/0 conn(0x7f25f813e130 0x7f25f813e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:59.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.206+0000 7f25fd460700 1 --2- 172.21.15.1:0/588232006 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25f815b750 0x7f25f815bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:59.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.206+0000 7f25fe462700 1 -- 172.21.15.1:0/588232006 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25f815b750 msgr2=0x7f25f815bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:59.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.206+0000 7f25fe462700 1 --2- 172.21.15.1:0/588232006 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25f815b750 0x7f25f815bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.206+0000 7f25fe462700 1 -- 172.21.15.1:0/588232006 >> v2:172.21.15.165:3300/0 conn(0x7f25f813e130 msgr2=0x7f25f813e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:59.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.206+0000 7f25fe462700 1 --2- 172.21.15.1:0/588232006 >> v2:172.21.15.165:3300/0 conn(0x7f25f813e130 0x7f25f813e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.206+0000 7f25fe462700 1 -- 172.21.15.1:0/588232006 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f815c4f0 con 0x7f25f8070db0 2024-03-18T20:26:59.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.206+0000 7f25fe462700 1 --2- 172.21.15.1:0/588232006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f8070db0 0x7f25f8071220 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 crypto rx=0x7f25f806b730 tx=0x7f25f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc7e797fb4f2ff8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:59.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.207+0000 7f25fcc5f700 1 -- 172.21.15.1:0/588232006 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f000e070 con 0x7f25f8070db0 2024-03-18T20:26:59.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.207+0000 7f25fcc5f700 1 -- 172.21.15.1:0/588232006 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25f000b910 con 0x7f25f8070db0 2024-03-18T20:26:59.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.207+0000 7f25fcc5f700 1 -- 172.21.15.1:0/588232006 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f000bcc0 con 0x7f25f8070db0 2024-03-18T20:26:59.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.207+0000 7f25ffec5700 1 -- 172.21.15.1:0/588232006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f8070db0 msgr2=0x7f25f8071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:59.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.207+0000 7f25ffec5700 1 --2- 172.21.15.1:0/588232006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f8070db0 0x7f25f8071220 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 crypto rx=0x7f25f806b730 tx=0x7f25f000b330 comp rx=0 tx=0).stop 2024-03-18T20:26:59.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.207+0000 7f25ffec5700 1 -- 172.21.15.1:0/588232006 shutdown_connections 2024-03-18T20:26:59.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.207+0000 7f25ffec5700 1 --2- 172.21.15.1:0/588232006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f8070db0 0x7f25f8071220 unknown :-1 s=CLOSED pgs=2938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.207+0000 7f25ffec5700 1 --2- 172.21.15.1:0/588232006 >> v2:172.21.15.165:3300/0 conn(0x7f25f813e130 0x7f25f813e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.207+0000 7f25ffec5700 1 --2- 172.21.15.1:0/588232006 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25f815b750 0x7f25f815bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.207+0000 7f25ffec5700 1 -- 172.21.15.1:0/588232006 >> 172.21.15.1:0/588232006 conn(0x7f25f806a750 msgr2=0x7f25f806ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:59.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.207+0000 7f25ffec5700 1 -- 172.21.15.1:0/588232006 shutdown_connections 2024-03-18T20:26:59.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.207+0000 7f25ffec5700 1 -- 172.21.15.1:0/588232006 wait complete. 2024-03-18T20:26:59.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.209+0000 7f25ffec5700 1 Processor -- start 2024-03-18T20:26:59.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.209+0000 7f25ffec5700 1 -- start start 2024-03-18T20:26:59.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.209+0000 7f25ffec5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f25f8070db0 0x7f25f80b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:59.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.209+0000 7f25ffec5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f813e130 0x7f25f813e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:59.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.209+0000 7f25ffec5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25f815b750 0x7f25f813ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:59.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.209+0000 7f25ffec5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f80b38d0 con 0x7f25f813e130 2024-03-18T20:26:59.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.209+0000 7f25ffec5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f813d230 con 0x7f25f8070db0 2024-03-18T20:26:59.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.209+0000 7f25ffec5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f81f3020 con 0x7f25f815b750 2024-03-18T20:26:59.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.210+0000 7f25fd460700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f813e130 0x7f25f813e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:59.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.210+0000 7f25fd460700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f813e130 0x7f25f813e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34820/0 (socket says 172.21.15.1:34820) 2024-03-18T20:26:59.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.210+0000 7f25fd460700 1 -- 172.21.15.1:0/3952628785 learned_addr learned my addr 172.21.15.1:0/3952628785 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:59.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.210+0000 7f25fdc61700 1 --2- 172.21.15.1:0/3952628785 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f25f8070db0 0x7f25f80b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:59.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.210+0000 7f25fe462700 1 --2- 172.21.15.1:0/3952628785 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25f815b750 0x7f25f813ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:59.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.210+0000 7f25fd460700 1 -- 172.21.15.1:0/3952628785 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25f815b750 msgr2=0x7f25f813ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:59.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.210+0000 7f25fd460700 1 --2- 172.21.15.1:0/3952628785 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25f815b750 0x7f25f813ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.210+0000 7f25fd460700 1 -- 172.21.15.1:0/3952628785 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f25f8070db0 msgr2=0x7f25f80b3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:59.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.210+0000 7f25fd460700 1 --2- 172.21.15.1:0/3952628785 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f25f8070db0 0x7f25f80b3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.210+0000 7f25fd460700 1 -- 172.21.15.1:0/3952628785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f0009b30 con 0x7f25f813e130 2024-03-18T20:26:59.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.210+0000 7f25fd460700 1 --2- 172.21.15.1:0/3952628785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f813e130 0x7f25f813e690 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 crypto rx=0x7f25e800e8a0 tx=0x7f25e800ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:59.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.210+0000 7f25ee7fc700 1 -- 172.21.15.1:0/3952628785 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8009590 con 0x7f25f813e130 2024-03-18T20:26:59.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.210+0000 7f25ee7fc700 1 -- 172.21.15.1:0/3952628785 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e8013680 con 0x7f25f813e130 2024-03-18T20:26:59.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.210+0000 7f25ee7fc700 1 -- 172.21.15.1:0/3952628785 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e80139f0 con 0x7f25f813e130 2024-03-18T20:26:59.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.210+0000 7f25ffec5700 1 -- 172.21.15.1:0/3952628785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f80b2c50 con 0x7f25f813e130 2024-03-18T20:26:59.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.210+0000 7f25ffec5700 1 -- 172.21.15.1:0/3952628785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f80b3210 con 0x7f25f813e130 2024-03-18T20:26:59.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.211+0000 7f25ffec5700 1 -- 172.21.15.1:0/3952628785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f80b3a10 con 0x7f25f813e130 2024-03-18T20:26:59.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.212+0000 7f25ee7fc700 1 -- 172.21.15.1:0/3952628785 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 152) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f25e8013b50 con 0x7f25f813e130 2024-03-18T20:26:59.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.212+0000 7f25ee7fc700 1 --2- 172.21.15.1:0/3952628785 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f25dc06aed0 0x7f25dc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:59.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.212+0000 7f25fdc61700 1 -- 172.21.15.1:0/3952628785 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f25dc06aed0 msgr2=0x7f25dc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:59.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.212+0000 7f25ee7fc700 1 -- 172.21.15.1:0/3952628785 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7423+0+0 (secure 0 0 0) 0x7f25e8089bb0 con 0x7f25f813e130 2024-03-18T20:26:59.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.212+0000 7f25fdc61700 1 --2- 172.21.15.1:0/3952628785 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f25dc06aed0 0x7f25dc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:59.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.217+0000 7f25ee7fc700 1 -- 172.21.15.1:0/3952628785 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f25f80b3a10 con 0x7f25f813e130 2024-03-18T20:26:59.223 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:59.219+0000 7f76988f6240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:26:59.390 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:59.385+0000 7f76988f6240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:26:59.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.413+0000 7f25fdc61700 1 -- 172.21.15.1:0/3952628785 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f25dc06aed0 msgr2=0x7f25dc06d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:59.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.413+0000 7f25fdc61700 1 --2- 172.21.15.1:0/3952628785 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f25dc06aed0 0x7f25dc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:59.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.418+0000 7f25ffec5700 1 -- 172.21.15.1:0/3952628785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f25f80b3a10 con 0x7f25f813e130 2024-03-18T20:26:59.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.419+0000 7f25ee7fc700 1 -- 172.21.15.1:0/3952628785 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v258) v1 ==== 100+0+11 (secure 0 0 0) 0x7f25f80b3a10 con 0x7f25f813e130 2024-03-18T20:26:59.422 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:26:59.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.421+0000 7f25ffec5700 1 -- 172.21.15.1:0/3952628785 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f25dc06aed0 msgr2=0x7f25dc06d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:59.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.421+0000 7f25ffec5700 1 --2- 172.21.15.1:0/3952628785 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f25dc06aed0 0x7f25dc06d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.421+0000 7f25ffec5700 1 -- 172.21.15.1:0/3952628785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f813e130 msgr2=0x7f25f813e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:59.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.421+0000 7f25ffec5700 1 --2- 172.21.15.1:0/3952628785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f813e130 0x7f25f813e690 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 crypto rx=0x7f25e800e8a0 tx=0x7f25e800ebb0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.421+0000 7f25ffec5700 1 -- 172.21.15.1:0/3952628785 shutdown_connections 2024-03-18T20:26:59.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.421+0000 7f25ffec5700 1 --2- 172.21.15.1:0/3952628785 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f25dc06aed0 0x7f25dc06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.421+0000 7f25ffec5700 1 --2- 172.21.15.1:0/3952628785 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f25f815b750 0x7f25f813ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.421+0000 7f25ffec5700 1 --2- 172.21.15.1:0/3952628785 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f25f8070db0 0x7f25f80b3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.421+0000 7f25ffec5700 1 --2- 172.21.15.1:0/3952628785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25f813e130 0x7f25f813e690 unknown :-1 s=CLOSED pgs=2939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.421+0000 7f25ffec5700 1 -- 172.21.15.1:0/3952628785 >> 172.21.15.1:0/3952628785 conn(0x7f25f806a750 msgr2=0x7f25f806e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:59.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.421+0000 7f25ffec5700 1 -- 172.21.15.1:0/3952628785 shutdown_connections 2024-03-18T20:26:59.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.421+0000 7f25ffec5700 1 -- 172.21.15.1:0/3952628785 wait complete. 2024-03-18T20:26:59.437 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-03-18T20:26:59.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.562+0000 7f541c693700 1 Processor -- start 2024-03-18T20:26:59.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.562+0000 7f541c693700 1 -- start start 2024-03-18T20:26:59.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.563+0000 7f541c693700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f541409dd40 0x7f54140cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:59.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.563+0000 7f541c693700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54140ce5a0 0x7f54140cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:59.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.563+0000 7f541c693700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54140ce090 con 0x7f541415d780 2024-03-18T20:26:59.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.563+0000 7f541c693700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54140cb480 con 0x7f541409dd40 2024-03-18T20:26:59.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.563+0000 7f541c693700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f541406db30 con 0x7f54140ce5a0 2024-03-18T20:26:59.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.563+0000 7f5419c2e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f541409dd40 0x7f54140cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:59.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.563+0000 7f5419c2e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f541409dd40 0x7f54140cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34832/0 (socket says 172.21.15.1:34832) 2024-03-18T20:26:59.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.563+0000 7f5419c2e700 1 -- 172.21.15.1:0/2192201080 learned_addr learned my addr 172.21.15.1:0/2192201080 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:59.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.563+0000 7f541ac30700 1 --2- 172.21.15.1:0/2192201080 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54140ce5a0 0x7f54140cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:59.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.564+0000 7f5419c2e700 1 -- 172.21.15.1:0/2192201080 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54140ce5a0 msgr2=0x7f54140cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:26:59.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.564+0000 7f5419c2e700 1 --2- 172.21.15.1:0/2192201080 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54140ce5a0 0x7f54140cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.564+0000 7f5419c2e700 1 -- 172.21.15.1:0/2192201080 >> v1:172.21.15.165:6789/0 conn(0x7f541415d780 legacy=0x7f54140cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:59.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.564+0000 7f5419c2e700 1 -- 172.21.15.1:0/2192201080 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541406d520 con 0x7f541409dd40 2024-03-18T20:26:59.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.564+0000 7f5419c2e700 1 --2- 172.21.15.1:0/2192201080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541409dd40 0x7f54140cdb50 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f54100098f0 tx=0x7f5410009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5895db124a95f27 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:59.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.564+0000 7f541942d700 1 -- 172.21.15.1:0/2192201080 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5410011830 con 0x7f541409dd40 2024-03-18T20:26:59.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.564+0000 7f541942d700 1 -- 172.21.15.1:0/2192201080 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5410011990 con 0x7f541409dd40 2024-03-18T20:26:59.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.564+0000 7f541942d700 1 -- 172.21.15.1:0/2192201080 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5410011d20 con 0x7f541409dd40 2024-03-18T20:26:59.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.565+0000 7f541c693700 1 -- 172.21.15.1:0/2192201080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541409dd40 msgr2=0x7f54140cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:59.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.565+0000 7f541c693700 1 --2- 172.21.15.1:0/2192201080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541409dd40 0x7f54140cdb50 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f54100098f0 tx=0x7f5410009c00 comp rx=0 tx=0).stop 2024-03-18T20:26:59.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.565+0000 7f541c693700 1 -- 172.21.15.1:0/2192201080 shutdown_connections 2024-03-18T20:26:59.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.565+0000 7f541c693700 1 --2- 172.21.15.1:0/2192201080 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f54140ce5a0 0x7f54140cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.565+0000 7f541c693700 1 --2- 172.21.15.1:0/2192201080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541409dd40 0x7f54140cdb50 unknown :-1 s=CLOSED pgs=2940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.565+0000 7f541c693700 1 -- 172.21.15.1:0/2192201080 >> 172.21.15.1:0/2192201080 conn(0x7f54140cca00 msgr2=0x7f54140cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:59.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.565+0000 7f541c693700 1 -- 172.21.15.1:0/2192201080 shutdown_connections 2024-03-18T20:26:59.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.565+0000 7f541c693700 1 -- 172.21.15.1:0/2192201080 wait complete. 2024-03-18T20:26:59.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.566+0000 7f541c693700 1 Processor -- start 2024-03-18T20:26:59.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.566+0000 7f541c693700 1 -- start start 2024-03-18T20:26:59.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.567+0000 7f541c693700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541409dd40 0x7f5414158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:59.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.567+0000 7f541c693700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54140ce5a0 0x7f54140c8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:59.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.567+0000 7f541c693700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f541415d780 0x7f54140c5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:59.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.567+0000 7f541c693700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54141f1a90 con 0x7f541409dd40 2024-03-18T20:26:59.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.567+0000 7f541c693700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54140c8b40 con 0x7f54140ce5a0 2024-03-18T20:26:59.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.567+0000 7f541c693700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54141f2060 con 0x7f541415d780 2024-03-18T20:26:59.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.567+0000 7f541a42f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541409dd40 0x7f5414158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:59.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.567+0000 7f541a42f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541409dd40 0x7f5414158750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34842/0 (socket says 172.21.15.1:34842) 2024-03-18T20:26:59.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.567+0000 7f541a42f700 1 -- 172.21.15.1:0/3894776407 learned_addr learned my addr 172.21.15.1:0/3894776407 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:59.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.568+0000 7f5419c2e700 1 --2- 172.21.15.1:0/3894776407 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54140ce5a0 0x7f54140c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:59.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.568+0000 7f541ac30700 1 --2- 172.21.15.1:0/3894776407 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f541415d780 0x7f54140c5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:59.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.568+0000 7f541a42f700 1 -- 172.21.15.1:0/3894776407 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f541415d780 msgr2=0x7f54140c5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:59.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.568+0000 7f541a42f700 1 --2- 172.21.15.1:0/3894776407 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f541415d780 0x7f54140c5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.568+0000 7f541a42f700 1 -- 172.21.15.1:0/3894776407 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54140ce5a0 msgr2=0x7f54140c8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:59.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.568+0000 7f541a42f700 1 --2- 172.21.15.1:0/3894776407 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54140ce5a0 0x7f54140c8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.568+0000 7f541a42f700 1 -- 172.21.15.1:0/3894776407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5410009550 con 0x7f541409dd40 2024-03-18T20:26:59.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.568+0000 7f541a42f700 1 --2- 172.21.15.1:0/3894776407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541409dd40 0x7f5414158750 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 crypto rx=0x7f540400c4e0 tx=0x7f540400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:59.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.568+0000 7f540affd700 1 -- 172.21.15.1:0/3894776407 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54040052d0 con 0x7f541409dd40 2024-03-18T20:26:59.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.568+0000 7f540affd700 1 -- 172.21.15.1:0/3894776407 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5404005430 con 0x7f541409dd40 2024-03-18T20:26:59.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.568+0000 7f541c693700 1 -- 172.21.15.1:0/3894776407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54141f0ea0 con 0x7f541409dd40 2024-03-18T20:26:59.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.568+0000 7f540affd700 1 -- 172.21.15.1:0/3894776407 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54040038c0 con 0x7f541409dd40 2024-03-18T20:26:59.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.568+0000 7f541c693700 1 -- 172.21.15.1:0/3894776407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54141f1440 con 0x7f541409dd40 2024-03-18T20:26:59.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.569+0000 7f541c693700 1 -- 172.21.15.1:0/3894776407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54140676a0 con 0x7f541409dd40 2024-03-18T20:26:59.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.570+0000 7f540affd700 1 -- 172.21.15.1:0/3894776407 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 152) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f5404003a20 con 0x7f541409dd40 2024-03-18T20:26:59.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.570+0000 7f540affd700 1 --2- 172.21.15.1:0/3894776407 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f53f806aed0 0x7f53f806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:59.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.570+0000 7f5419c2e700 1 -- 172.21.15.1:0/3894776407 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f53f806aed0 msgr2=0x7f53f806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:59.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.570+0000 7f540affd700 1 -- 172.21.15.1:0/3894776407 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7423+0+0 (secure 0 0 0) 0x7f54040890c0 con 0x7f541409dd40 2024-03-18T20:26:59.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.570+0000 7f5419c2e700 1 --2- 172.21.15.1:0/3894776407 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f53f806aed0 0x7f53f806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:59.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.575+0000 7f540affd700 1 -- 172.21.15.1:0/3894776407 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5404053960 con 0x7f541409dd40 2024-03-18T20:26:59.611 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:59.607+0000 7f5294a3a240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:26:59.753 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:59.749+0000 7f76988f6240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:26:59.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.771+0000 7f5419c2e700 1 -- 172.21.15.1:0/3894776407 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f53f806aed0 msgr2=0x7f53f806d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:59.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.771+0000 7f5419c2e700 1 --2- 172.21.15.1:0/3894776407 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f53f806aed0 0x7f53f806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:26:59.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.776+0000 7f541c693700 1 -- 172.21.15.1:0/3894776407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5414072490 con 0x7f541409dd40 2024-03-18T20:26:59.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.776+0000 7f540affd700 1 -- 172.21.15.1:0/3894776407 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v258) v1 ==== 107+0+11 (secure 0 0 0) 0x7f5404017020 con 0x7f541409dd40 2024-03-18T20:26:59.780 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:26:59.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.778+0000 7f541c693700 1 -- 172.21.15.1:0/3894776407 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f53f806aed0 msgr2=0x7f53f806d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:26:59.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.778+0000 7f541c693700 1 --2- 172.21.15.1:0/3894776407 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f53f806aed0 0x7f53f806d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.778+0000 7f541c693700 1 -- 172.21.15.1:0/3894776407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541409dd40 msgr2=0x7f5414158750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:59.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.778+0000 7f541c693700 1 --2- 172.21.15.1:0/3894776407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541409dd40 0x7f5414158750 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 crypto rx=0x7f540400c4e0 tx=0x7f540400c7f0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.778+0000 7f541c693700 1 -- 172.21.15.1:0/3894776407 shutdown_connections 2024-03-18T20:26:59.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.778+0000 7f541c693700 1 --2- 172.21.15.1:0/3894776407 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f53f806aed0 0x7f53f806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.778+0000 7f541c693700 1 --2- 172.21.15.1:0/3894776407 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f541415d780 0x7f54140c5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.779+0000 7f541c693700 1 --2- 172.21.15.1:0/3894776407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541409dd40 0x7f5414158750 unknown :-1 s=CLOSED pgs=2941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.779+0000 7f541c693700 1 --2- 172.21.15.1:0/3894776407 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f54140ce5a0 0x7f54140c8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.779+0000 7f541c693700 1 -- 172.21.15.1:0/3894776407 >> 172.21.15.1:0/3894776407 conn(0x7f54140cca00 msgr2=0x7f541406dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:59.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.779+0000 7f541c693700 1 -- 172.21.15.1:0/3894776407 shutdown_connections 2024-03-18T20:26:59.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.779+0000 7f541c693700 1 -- 172.21.15.1:0/3894776407 wait complete. 2024-03-18T20:26:59.790 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:26:59.786+0000 7f5294a3a240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:26:59.795 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-03-18T20:26:59.842 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:26:59.837+0000 7f76988f6240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:26:59.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.922+0000 7f2be972c700 1 Processor -- start 2024-03-18T20:26:59.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.922+0000 7f2be972c700 1 -- start start 2024-03-18T20:26:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.923+0000 7f2be972c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2be40d3c60 0x7f2be40ce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.923+0000 7f2be972c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be40b3370 con 0x7f2be40cfb50 2024-03-18T20:26:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.923+0000 7f2be972c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be40b2760 con 0x7f2be40b4810 2024-03-18T20:26:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.923+0000 7f2be972c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be40b2a40 con 0x7f2be40d3c60 2024-03-18T20:26:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.923+0000 7f2be2ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2be40cfb50 0x7f2be40cff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45038/0 (socket says 172.21.15.1:45038) 2024-03-18T20:26:59.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.923+0000 7f2be2ffd700 1 -- 172.21.15.1:0/3518814020 learned_addr learned my addr 172.21.15.1:0/3518814020 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:26:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.923+0000 7f2be37fe700 1 --2- 172.21.15.1:0/3518814020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2be40d3c60 0x7f2be40ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.923+0000 7f2be1ffb700 1 -- 172.21.15.1:0/3518814020 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806283933 0 0) 0x7f2be40b3370 con 0x7f2be40cfb50 2024-03-18T20:26:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.923+0000 7f2be1ffb700 1 -- 172.21.15.1:0/3518814020 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bcc0035b0 con 0x7f2be40cfb50 2024-03-18T20:26:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.923+0000 7f2be1ffb700 1 -- 172.21.15.1:0/3518814020 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283993202 0 0) 0x7f2be40b2760 con 0x7f2be40b4810 2024-03-18T20:26:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.923+0000 7f2be1ffb700 1 -- 172.21.15.1:0/3518814020 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be40b3370 con 0x7f2be40b4810 2024-03-18T20:26:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.923+0000 7f2be37fe700 1 -- 172.21.15.1:0/3518814020 >> v1:172.21.15.165:6789/0 conn(0x7f2be40b4810 legacy=0x7f2be40cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.923+0000 7f2be37fe700 1 -- 172.21.15.1:0/3518814020 >> v1:172.21.15.1:6789/0 conn(0x7f2be40cfb50 legacy=0x7f2be40cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.923+0000 7f2be37fe700 1 -- 172.21.15.1:0/3518814020 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be40b2d20 con 0x7f2be40d3c60 2024-03-18T20:26:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.923+0000 7f2be37fe700 1 --2- 172.21.15.1:0/3518814020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2be40d3c60 0x7f2be40ce580 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc009dc0 tx=0x7f2bdc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8daa0a28f690566 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.924+0000 7f2be1ffb700 1 -- 172.21.15.1:0/3518814020 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc00e070 con 0x7f2be40d3c60 2024-03-18T20:26:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.924+0000 7f2be1ffb700 1 -- 172.21.15.1:0/3518814020 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bdc00bb20 con 0x7f2be40d3c60 2024-03-18T20:26:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.924+0000 7f2be1ffb700 1 -- 172.21.15.1:0/3518814020 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc012430 con 0x7f2be40d3c60 2024-03-18T20:26:59.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.924+0000 7f2be972c700 1 -- 172.21.15.1:0/3518814020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2be40d3c60 msgr2=0x7f2be40ce580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.924+0000 7f2be972c700 1 --2- 172.21.15.1:0/3518814020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2be40d3c60 0x7f2be40ce580 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc009dc0 tx=0x7f2bdc00b330 comp rx=0 tx=0).stop 2024-03-18T20:26:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.924+0000 7f2be972c700 1 -- 172.21.15.1:0/3518814020 shutdown_connections 2024-03-18T20:26:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.924+0000 7f2be972c700 1 --2- 172.21.15.1:0/3518814020 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2be40d3c60 0x7f2be40ce580 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.924+0000 7f2be972c700 1 -- 172.21.15.1:0/3518814020 >> 172.21.15.1:0/3518814020 conn(0x7f2be40d3140 msgr2=0x7f2be40d3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:26:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.924+0000 7f2be972c700 1 -- 172.21.15.1:0/3518814020 shutdown_connections 2024-03-18T20:26:59.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.924+0000 7f2be972c700 1 -- 172.21.15.1:0/3518814020 wait complete. 2024-03-18T20:26:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.926+0000 7f2be972c700 1 Processor -- start 2024-03-18T20:26:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.926+0000 7f2be972c700 1 -- start start 2024-03-18T20:26:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.926+0000 7f2be972c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2be40b4810 0x7f2be4158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.926+0000 7f2be972c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2be40cfb50 0x7f2be40cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.926+0000 7f2be972c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2be40d3c60 0x7f2be40c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.926+0000 7f2be972c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be40cbba0 con 0x7f2be40d3c60 2024-03-18T20:26:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.926+0000 7f2be972c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be40cbe80 con 0x7f2be40b4810 2024-03-18T20:26:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.926+0000 7f2be972c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be40cc160 con 0x7f2be40cfb50 2024-03-18T20:26:59.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.927+0000 7f2be37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2be40d3c60 0x7f2be40c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.927+0000 7f2be37fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2be40d3c60 0x7f2be40c7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34858/0 (socket says 172.21.15.1:34858) 2024-03-18T20:26:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.927+0000 7f2be37fe700 1 -- 172.21.15.1:0/556705573 learned_addr learned my addr 172.21.15.1:0/556705573 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:26:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.927+0000 7f2be27fc700 1 --2- 172.21.15.1:0/556705573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2be40cfb50 0x7f2be40cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.927+0000 7f2be2ffd700 1 --2- 172.21.15.1:0/556705573 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2be40b4810 0x7f2be4158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:26:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.927+0000 7f2be37fe700 1 -- 172.21.15.1:0/556705573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2be40cfb50 msgr2=0x7f2be40cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.927+0000 7f2be37fe700 1 --2- 172.21.15.1:0/556705573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2be40cfb50 0x7f2be40cb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.927+0000 7f2be37fe700 1 -- 172.21.15.1:0/556705573 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2be40b4810 msgr2=0x7f2be4158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:26:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.927+0000 7f2be37fe700 1 --2- 172.21.15.1:0/556705573 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2be40b4810 0x7f2be4158b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:26:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.927+0000 7f2be37fe700 1 -- 172.21.15.1:0/556705573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bdc009a70 con 0x7f2be40d3c60 2024-03-18T20:26:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.927+0000 7f2be37fe700 1 --2- 172.21.15.1:0/556705573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2be40d3c60 0x7f2be40c7cc0 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00b370 tx=0x7f2bdc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:26:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.927+0000 7f2bcb7fe700 1 -- 172.21.15.1:0/556705573 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc01b810 con 0x7f2be40d3c60 2024-03-18T20:26:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.927+0000 7f2bcb7fe700 1 -- 172.21.15.1:0/556705573 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bdc020070 con 0x7f2be40d3c60 2024-03-18T20:26:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.927+0000 7f2bcb7fe700 1 -- 172.21.15.1:0/556705573 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc01bbb0 con 0x7f2be40d3c60 2024-03-18T20:26:59.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.927+0000 7f2be972c700 1 -- 172.21.15.1:0/556705573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be40cc440 con 0x7f2be40d3c60 2024-03-18T20:26:59.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.927+0000 7f2be972c700 1 -- 172.21.15.1:0/556705573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be40b8460 con 0x7f2be40d3c60 2024-03-18T20:26:59.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.928+0000 7f2be972c700 1 -- 172.21.15.1:0/556705573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be40b85f0 con 0x7f2be40d3c60 2024-03-18T20:26:59.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.929+0000 7f2bcb7fe700 1 -- 172.21.15.1:0/556705573 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 152) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f2bdc01bd10 con 0x7f2be40d3c60 2024-03-18T20:26:59.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.929+0000 7f2bcb7fe700 1 --2- 172.21.15.1:0/556705573 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2bd006aed0 0x7f2bd006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:26:59.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.929+0000 7f2be2ffd700 1 -- 172.21.15.1:0/556705573 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2bd006aed0 msgr2=0x7f2bd006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:26:59.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.929+0000 7f2bcb7fe700 1 -- 172.21.15.1:0/556705573 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7423+0+0 (secure 0 0 0) 0x7f2bdc022070 con 0x7f2be40d3c60 2024-03-18T20:26:59.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.929+0000 7f2be2ffd700 1 --2- 172.21.15.1:0/556705573 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2bd006aed0 0x7f2bd006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:26:59.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:26:59.934+0000 7f2bcb7fe700 1 -- 172.21.15.1:0/556705573 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2bdc056510 con 0x7f2be40d3c60 2024-03-18T20:27:00.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.130+0000 7f2be2ffd700 1 -- 172.21.15.1:0/556705573 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2bd006aed0 msgr2=0x7f2bd006d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:00.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.130+0000 7f2be2ffd700 1 --2- 172.21.15.1:0/556705573 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2bd006aed0 0x7f2bd006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:27:00.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.136+0000 7f2be972c700 1 -- 172.21.15.1:0/556705573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f2be40cc960 con 0x7f2be40d3c60 2024-03-18T20:27:00.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.136+0000 7f2bcb7fe700 1 -- 172.21.15.1:0/556705573 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v258) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2be40cc960 con 0x7f2be40d3c60 2024-03-18T20:27:00.140 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:27:00.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.139+0000 7f2be972c700 1 -- 172.21.15.1:0/556705573 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2bd006aed0 msgr2=0x7f2bd006d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:27:00.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.139+0000 7f2be972c700 1 --2- 172.21.15.1:0/556705573 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2bd006aed0 0x7f2bd006d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.139+0000 7f2be972c700 1 -- 172.21.15.1:0/556705573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2be40d3c60 msgr2=0x7f2be40c7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:00.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.139+0000 7f2be972c700 1 --2- 172.21.15.1:0/556705573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2be40d3c60 0x7f2be40c7cc0 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00b370 tx=0x7f2bdc004ad0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.139+0000 7f2be972c700 1 -- 172.21.15.1:0/556705573 shutdown_connections 2024-03-18T20:27:00.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.139+0000 7f2be972c700 1 --2- 172.21.15.1:0/556705573 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2bd006aed0 0x7f2bd006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.139+0000 7f2be972c700 1 --2- 172.21.15.1:0/556705573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2be40d3c60 0x7f2be40c7cc0 unknown :-1 s=CLOSED pgs=2943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.139+0000 7f2be972c700 1 --2- 172.21.15.1:0/556705573 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2be40b4810 0x7f2be4158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.139+0000 7f2be972c700 1 --2- 172.21.15.1:0/556705573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2be40cfb50 0x7f2be40cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.139+0000 7f2be972c700 1 -- 172.21.15.1:0/556705573 >> 172.21.15.1:0/556705573 conn(0x7f2be40d3140 msgr2=0x7f2be40b1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:00.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.139+0000 7f2be972c700 1 -- 172.21.15.1:0/556705573 shutdown_connections 2024-03-18T20:27:00.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.139+0000 7f2be972c700 1 -- 172.21.15.1:0/556705573 wait complete. 2024-03-18T20:27:00.156 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:27:00.182 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.179+0000 7f5294a3a240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:27:00.279 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.276+0000 7f5294a3a240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:27:00.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.280+0000 7f88ed120700 1 Processor -- start 2024-03-18T20:27:00.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.280+0000 7f88ed120700 1 -- start start 2024-03-18T20:27:00.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.281+0000 7f88ed120700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88e815d500 0x7f88e80d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:00.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.281+0000 7f88ed120700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88e80d1300 0x7f88e80d17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:00.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.281+0000 7f88ed120700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f88e80d1cf0 0x7f88e80b83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:00.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.281+0000 7f88ed120700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88e80d2140 con 0x7f88e80d1300 2024-03-18T20:27:00.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.281+0000 7f88ed120700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88e80cddc0 con 0x7f88e80d1cf0 2024-03-18T20:27:00.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.281+0000 7f88ed120700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e80d3dc0 con 0x7f88e815d500 2024-03-18T20:27:00.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.281+0000 7f88e659c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88e80d1300 0x7f88e80d17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:00.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.281+0000 7f88e659c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88e80d1300 0x7f88e80d17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34864/0 (socket says 172.21.15.1:34864) 2024-03-18T20:27:00.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.281+0000 7f88e659c700 1 -- 172.21.15.1:0/890315092 learned_addr learned my addr 172.21.15.1:0/890315092 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:00.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.281+0000 7f88e759e700 1 --2- 172.21.15.1:0/890315092 >> v2:172.21.15.165:3300/0 conn(0x7f88e80d1cf0 0x7f88e80b83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.281+0000 7f88e6d9d700 1 --2- 172.21.15.1:0/890315092 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88e815d500 0x7f88e80d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88e659c700 1 -- 172.21.15.1:0/890315092 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88e815d500 msgr2=0x7f88e80d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88e659c700 1 --2- 172.21.15.1:0/890315092 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88e815d500 0x7f88e80d0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88e659c700 1 -- 172.21.15.1:0/890315092 >> v2:172.21.15.165:3300/0 conn(0x7f88e80d1cf0 msgr2=0x7f88e80b83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88e659c700 1 --2- 172.21.15.1:0/890315092 >> v2:172.21.15.165:3300/0 conn(0x7f88e80d1cf0 0x7f88e80b83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88e659c700 1 -- 172.21.15.1:0/890315092 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e80ce060 con 0x7f88e80d1300 2024-03-18T20:27:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88e759e700 1 --2- 172.21.15.1:0/890315092 >> v2:172.21.15.165:3300/0 conn(0x7f88e80d1cf0 0x7f88e80b83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88e6d9d700 1 --2- 172.21.15.1:0/890315092 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88e815d500 0x7f88e80d0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88e659c700 1 --2- 172.21.15.1:0/890315092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88e80d1300 0x7f88e80d17b0 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 crypto rx=0x7f88e80c6a00 tx=0x7f88dc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31d5d7935521818e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88e5d9b700 1 -- 172.21.15.1:0/890315092 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc00b870 con 0x7f88e80d1300 2024-03-18T20:27:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88e5d9b700 1 -- 172.21.15.1:0/890315092 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88dc00b9d0 con 0x7f88e80d1300 2024-03-18T20:27:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88e5d9b700 1 -- 172.21.15.1:0/890315092 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc00bd60 con 0x7f88e80d1300 2024-03-18T20:27:00.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88ed120700 1 -- 172.21.15.1:0/890315092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88e80d1300 msgr2=0x7f88e80d17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:00.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88ed120700 1 --2- 172.21.15.1:0/890315092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88e80d1300 0x7f88e80d17b0 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 crypto rx=0x7f88e80c6a00 tx=0x7f88dc009bd0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88ed120700 1 -- 172.21.15.1:0/890315092 shutdown_connections 2024-03-18T20:27:00.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88ed120700 1 --2- 172.21.15.1:0/890315092 >> v2:172.21.15.165:3300/0 conn(0x7f88e80d1cf0 0x7f88e80b83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88ed120700 1 --2- 172.21.15.1:0/890315092 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88e815d500 0x7f88e80d0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88ed120700 1 --2- 172.21.15.1:0/890315092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88e80d1300 0x7f88e80d17b0 unknown :-1 s=CLOSED pgs=2944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88ed120700 1 -- 172.21.15.1:0/890315092 >> 172.21.15.1:0/890315092 conn(0x7f88e809c530 msgr2=0x7f88e80d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:00.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88ed120700 1 -- 172.21.15.1:0/890315092 shutdown_connections 2024-03-18T20:27:00.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.282+0000 7f88ed120700 1 -- 172.21.15.1:0/890315092 wait complete. 2024-03-18T20:27:00.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.284+0000 7f88ed120700 1 Processor -- start 2024-03-18T20:27:00.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.284+0000 7f88ed120700 1 -- start start 2024-03-18T20:27:00.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.285+0000 7f88ed120700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88e80d1300 0x7f88e81f4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:00.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.285+0000 7f88ed120700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88e80d1cf0 0x7f88e80b7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:00.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.285+0000 7f88ed120700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88e815d500 0x7f88e81f7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:00.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.285+0000 7f88ed120700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e81f3dc0 con 0x7f88e80d1cf0 2024-03-18T20:27:00.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.285+0000 7f88ed120700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e80b8730 con 0x7f88e815d500 2024-03-18T20:27:00.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.285+0000 7f88ed120700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e80b8a10 con 0x7f88e80d1300 2024-03-18T20:27:00.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.285+0000 7f88e659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88e80d1cf0 0x7f88e80b7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:00.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.285+0000 7f88e659c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88e80d1cf0 0x7f88e80b7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34866/0 (socket says 172.21.15.1:34866) 2024-03-18T20:27:00.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.285+0000 7f88e659c700 1 -- 172.21.15.1:0/2984373979 learned_addr learned my addr 172.21.15.1:0/2984373979 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:00.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.285+0000 7f88e6d9d700 1 --2- 172.21.15.1:0/2984373979 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88e80d1300 0x7f88e81f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:00.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.285+0000 7f88e759e700 1 --2- 172.21.15.1:0/2984373979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88e815d500 0x7f88e81f7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:00.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.285+0000 7f88e659c700 1 -- 172.21.15.1:0/2984373979 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88e80d1300 msgr2=0x7f88e81f4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:00.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.285+0000 7f88e659c700 1 --2- 172.21.15.1:0/2984373979 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88e80d1300 0x7f88e81f4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.285+0000 7f88e659c700 1 -- 172.21.15.1:0/2984373979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88e815d500 msgr2=0x7f88e81f7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:00.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.285+0000 7f88e659c700 1 --2- 172.21.15.1:0/2984373979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88e815d500 0x7f88e81f7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.285+0000 7f88e659c700 1 -- 172.21.15.1:0/2984373979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88dc009550 con 0x7f88e80d1cf0 2024-03-18T20:27:00.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.285+0000 7f88e6d9d700 1 --2- 172.21.15.1:0/2984373979 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88e80d1300 0x7f88e81f4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:00.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.285+0000 7f88e759e700 1 --2- 172.21.15.1:0/2984373979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88e815d500 0x7f88e81f7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:00.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.285+0000 7f88e659c700 1 --2- 172.21.15.1:0/2984373979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88e80d1cf0 0x7f88e80b7dc0 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 crypto rx=0x7f88dc000c00 tx=0x7f88dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:00.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.286+0000 7f88cf7fe700 1 -- 172.21.15.1:0/2984373979 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc004540 con 0x7f88e80d1cf0 2024-03-18T20:27:00.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.286+0000 7f88cf7fe700 1 -- 172.21.15.1:0/2984373979 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88dc0036a0 con 0x7f88e80d1cf0 2024-03-18T20:27:00.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.286+0000 7f88ed120700 1 -- 172.21.15.1:0/2984373979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e81f3a50 con 0x7f88e80d1cf0 2024-03-18T20:27:00.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.286+0000 7f88cf7fe700 1 -- 172.21.15.1:0/2984373979 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc003a50 con 0x7f88e80d1cf0 2024-03-18T20:27:00.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.286+0000 7f88ed120700 1 -- 172.21.15.1:0/2984373979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e81f4790 con 0x7f88e80d1cf0 2024-03-18T20:27:00.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.287+0000 7f88ed120700 1 -- 172.21.15.1:0/2984373979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e81f38b0 con 0x7f88e80d1cf0 2024-03-18T20:27:00.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.287+0000 7f88cf7fe700 1 -- 172.21.15.1:0/2984373979 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 152) v1 ==== 89342+0+0 (secure 0 0 0) 0x7f88dc021070 con 0x7f88e80d1cf0 2024-03-18T20:27:00.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.288+0000 7f88cf7fe700 1 --2- 172.21.15.1:0/2984373979 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f88d406aed0 0x7f88d406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:00.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.288+0000 7f88e6d9d700 1 -- 172.21.15.1:0/2984373979 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f88d406aed0 msgr2=0x7f88d406d390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:00.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.288+0000 7f88cf7fe700 1 -- 172.21.15.1:0/2984373979 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7423+0+0 (secure 0 0 0) 0x7f88dc08b480 con 0x7f88e80d1cf0 2024-03-18T20:27:00.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.288+0000 7f88e6d9d700 1 --2- 172.21.15.1:0/2984373979 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f88d406aed0 0x7f88d406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:27:00.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.292+0000 7f88cf7fe700 1 -- 172.21.15.1:0/2984373979 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f88e81f38b0 con 0x7f88e80d1cf0 2024-03-18T20:27:00.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.299+0000 7f88cf7fe700 1 -- 172.21.15.1:0/2984373979 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 153) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f88dc0561d0 con 0x7f88e80d1cf0 2024-03-18T20:27:00.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.299+0000 7f88cf7fe700 1 -- 172.21.15.1:0/2984373979 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f88d406aed0 msgr2=0x7f88d406d390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:27:00.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.299+0000 7f88cf7fe700 1 --2- 172.21.15.1:0/2984373979 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f88d406aed0 0x7f88d406d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.373 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.369+0000 7f520fe68700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.374 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.369+0000 7f520fe68700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.374 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.369+0000 7f520fe68700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.374 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.369+0000 7f520fe68700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.374 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.369+0000 7f520fe68700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.374 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.370+0000 7f520ce62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.374 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.370+0000 7f520ce62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.375 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.370+0000 7f520ce62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.375 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.370+0000 7f520ce62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.375 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.370+0000 7f520ce62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.384 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.380+0000 7f520fe68700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.384 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.380+0000 7f520fe68700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.384 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.380+0000 7f520fe68700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.384 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.380+0000 7f520fe68700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.384 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.380+0000 7f520fe68700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.385 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.381+0000 7f520ce62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.386 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.382+0000 7f520ce62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.386 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.382+0000 7f520ce62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.386 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.382+0000 7f520ce62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.386 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:00.382+0000 7f520ce62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:00.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.563+0000 7f88ed120700 1 -- 172.21.15.1:0/2984373979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88e81f38b0 con 0x7f88e80d1cf0 2024-03-18T20:27:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.564+0000 7f88cf7fe700 1 -- 172.21.15.1:0/2984373979 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+5963 (secure 0 0 0) 0x7f88e81f38b0 con 0x7f88e80d1cf0 2024-03-18T20:27:00.568 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:00.568 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":126,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12396,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2565889517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2565889517},{"type":"v1","addr":"172.21.15.1:6836","nonce":2565889517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":12399,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/512665659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":512665659},{"type":"v1","addr":"172.21.15.1:6837","nonce":512665659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:26:25.785074+0000","modified":"2024-03-18T20:26:29.101123+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12411":{"gid":12411,"name":"d","rank":0,"incarnation":116,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/3886150679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":3886150679},{"type":"v1","addr":"172.21.15.165:6835","nonce":3886150679}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:26:39.596187+0000","modified":"2024-03-18T20:26:42.614846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12408":{"gid":12408,"name":"b","rank":0,"incarnation":122,"state":"up:active","state_seq":6,"addr":"172.21.15.165:6834/1134181166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1134181166},{"type":"v1","addr":"172.21.15.165:6834","nonce":1134181166}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-03-18T20:27:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.566+0000 7f88ed120700 1 -- 172.21.15.1:0/2984373979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88e80d1cf0 msgr2=0x7f88e80b7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.566+0000 7f88ed120700 1 --2- 172.21.15.1:0/2984373979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88e80d1cf0 0x7f88e80b7dc0 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 crypto rx=0x7f88dc000c00 tx=0x7f88dc004750 comp rx=0 tx=0).stop 2024-03-18T20:27:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.566+0000 7f88ed120700 1 -- 172.21.15.1:0/2984373979 shutdown_connections 2024-03-18T20:27:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.566+0000 7f88ed120700 1 --2- 172.21.15.1:0/2984373979 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f88d406aed0 0x7f88d406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.566+0000 7f88ed120700 1 --2- 172.21.15.1:0/2984373979 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f88e80d1300 0x7f88e81f4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.566+0000 7f88ed120700 1 --2- 172.21.15.1:0/2984373979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88e815d500 0x7f88e81f7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.566+0000 7f88ed120700 1 --2- 172.21.15.1:0/2984373979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88e80d1cf0 0x7f88e80b7dc0 unknown :-1 s=CLOSED pgs=2945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.566+0000 7f88ed120700 1 -- 172.21.15.1:0/2984373979 >> 172.21.15.1:0/2984373979 conn(0x7f88e809c530 msgr2=0x7f88e815bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:00.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.566+0000 7f88ed120700 1 -- 172.21.15.1:0/2984373979 shutdown_connections 2024-03-18T20:27:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.566+0000 7f88ed120700 1 -- 172.21.15.1:0/2984373979 wait complete. 2024-03-18T20:27:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 126 2024-03-18T20:27:00.583 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:27:00.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.743+0000 7fa4dc0ac700 1 Processor -- start 2024-03-18T20:27:00.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.744+0000 7fa4dc0ac700 1 -- start start 2024-03-18T20:27:00.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.744+0000 7fa4dc0ac700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa4d40cc5e0 0x7fa4d406e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:00.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.744+0000 7fa4dc0ac700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d406ed50 0x7fa4d4099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:00.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.744+0000 7fa4dc0ac700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d415b6e0 con 0x7fa4d415b300 2024-03-18T20:27:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.744+0000 7fa4dc0ac700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4d40cce60 con 0x7fa4d40cc5e0 2024-03-18T20:27:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.744+0000 7fa4dc0ac700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d40cecd0 con 0x7fa4d406ed50 2024-03-18T20:27:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.745+0000 7fa4d9647700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa4d40cc5e0 0x7fa4d406e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.745+0000 7fa4d9647700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa4d40cc5e0 0x7fa4d406e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34998/0 (socket says 172.21.15.1:34998) 2024-03-18T20:27:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.745+0000 7fa4d9647700 1 -- 172.21.15.1:0/1098613959 learned_addr learned my addr 172.21.15.1:0/1098613959 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.745+0000 7fa4da649700 1 --2- 172.21.15.1:0/1098613959 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d406ed50 0x7fa4d4099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.745+0000 7fa4d9647700 1 -- 172.21.15.1:0/1098613959 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d406ed50 msgr2=0x7fa4d4099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.745+0000 7fa4d9647700 1 --2- 172.21.15.1:0/1098613959 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d406ed50 0x7fa4d4099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.745+0000 7fa4d9647700 1 -- 172.21.15.1:0/1098613959 >> v1:172.21.15.1:6789/0 conn(0x7fa4d415b300 legacy=0x7fa4d40cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.745+0000 7fa4d9647700 1 -- 172.21.15.1:0/1098613959 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d40cefb0 con 0x7fa4d40cc5e0 2024-03-18T20:27:00.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.745+0000 7fa4d9647700 1 --2- 172.21.15.1:0/1098613959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d40cc5e0 0x7fa4d406e810 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7fa4c80098f0 tx=0x7fa4c8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f641d3a2de57234 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.745+0000 7fa4d8e46700 1 -- 172.21.15.1:0/1098613959 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c800b8a0 con 0x7fa4d40cc5e0 2024-03-18T20:27:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.745+0000 7fa4d8e46700 1 -- 172.21.15.1:0/1098613959 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c800ba00 con 0x7fa4d40cc5e0 2024-03-18T20:27:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.745+0000 7fa4d8e46700 1 -- 172.21.15.1:0/1098613959 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c800bd90 con 0x7fa4d40cc5e0 2024-03-18T20:27:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.746+0000 7fa4dc0ac700 1 -- 172.21.15.1:0/1098613959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d40cc5e0 msgr2=0x7fa4d406e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.746+0000 7fa4dc0ac700 1 --2- 172.21.15.1:0/1098613959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d40cc5e0 0x7fa4d406e810 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7fa4c80098f0 tx=0x7fa4c8009c00 comp rx=0 tx=0).stop 2024-03-18T20:27:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.746+0000 7fa4dc0ac700 1 -- 172.21.15.1:0/1098613959 shutdown_connections 2024-03-18T20:27:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.746+0000 7fa4dc0ac700 1 --2- 172.21.15.1:0/1098613959 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d406ed50 0x7fa4d4099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.746+0000 7fa4dc0ac700 1 --2- 172.21.15.1:0/1098613959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d40cc5e0 0x7fa4d406e810 unknown :-1 s=CLOSED pgs=2955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.746+0000 7fa4dc0ac700 1 -- 172.21.15.1:0/1098613959 >> 172.21.15.1:0/1098613959 conn(0x7fa4d40b2b50 msgr2=0x7fa4d40b2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.746+0000 7fa4dc0ac700 1 -- 172.21.15.1:0/1098613959 shutdown_connections 2024-03-18T20:27:00.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.746+0000 7fa4dc0ac700 1 -- 172.21.15.1:0/1098613959 wait complete. 2024-03-18T20:27:00.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.747+0000 7fa4dc0ac700 1 Processor -- start 2024-03-18T20:27:00.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.747+0000 7fa4dc0ac700 1 -- start start 2024-03-18T20:27:00.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.748+0000 7fa4dc0ac700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4d406ed50 0x7fa4d4155980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:00.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.748+0000 7fa4dc0ac700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d40cc5e0 0x7fa4d4154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.748+0000 7fa4dc0ac700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d415b300 0x7fa4d41589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.748+0000 7fa4dc0ac700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d4156870 con 0x7fa4d415b300 2024-03-18T20:27:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.748+0000 7fa4dc0ac700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d40c6780 con 0x7fa4d406ed50 2024-03-18T20:27:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.748+0000 7fa4dc0ac700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d40c6a60 con 0x7fa4d40cc5e0 2024-03-18T20:27:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.748+0000 7fa4da649700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d415b300 0x7fa4d41589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.748+0000 7fa4da649700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d415b300 0x7fa4d41589a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35008/0 (socket says 172.21.15.1:35008) 2024-03-18T20:27:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.748+0000 7fa4da649700 1 -- 172.21.15.1:0/688428349 learned_addr learned my addr 172.21.15.1:0/688428349 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.748+0000 7fa4d9647700 1 --2- 172.21.15.1:0/688428349 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d40cc5e0 0x7fa4d4154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.748+0000 7fa4d9e48700 1 --2- 172.21.15.1:0/688428349 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4d406ed50 0x7fa4d4155980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.748+0000 7fa4da649700 1 -- 172.21.15.1:0/688428349 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d40cc5e0 msgr2=0x7fa4d4154ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.748+0000 7fa4da649700 1 --2- 172.21.15.1:0/688428349 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d40cc5e0 0x7fa4d4154ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.748+0000 7fa4da649700 1 -- 172.21.15.1:0/688428349 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4d406ed50 msgr2=0x7fa4d4155980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:00.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.748+0000 7fa4da649700 1 --2- 172.21.15.1:0/688428349 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4d406ed50 0x7fa4d4155980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.748+0000 7fa4da649700 1 -- 172.21.15.1:0/688428349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c8009550 con 0x7fa4d415b300 2024-03-18T20:27:00.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.748+0000 7fa4d9647700 1 --2- 172.21.15.1:0/688428349 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d40cc5e0 0x7fa4d4154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:00.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.749+0000 7fa4da649700 1 --2- 172.21.15.1:0/688428349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d415b300 0x7fa4d41589a0 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc007e40 tx=0x7fa4cc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:00.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.749+0000 7fa4c67fc700 1 -- 172.21.15.1:0/688428349 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc00eea0 con 0x7fa4d415b300 2024-03-18T20:27:00.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.749+0000 7fa4c67fc700 1 -- 172.21.15.1:0/688428349 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4cc014420 con 0x7fa4d415b300 2024-03-18T20:27:00.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.749+0000 7fa4dc0ac700 1 -- 172.21.15.1:0/688428349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d4156500 con 0x7fa4d415b300 2024-03-18T20:27:00.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.749+0000 7fa4c67fc700 1 -- 172.21.15.1:0/688428349 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc0147b0 con 0x7fa4d415b300 2024-03-18T20:27:00.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.749+0000 7fa4dc0ac700 1 -- 172.21.15.1:0/688428349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d41561b0 con 0x7fa4d415b300 2024-03-18T20:27:00.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.750+0000 7fa4dc0ac700 1 -- 172.21.15.1:0/688428349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d41569e0 con 0x7fa4d415b300 2024-03-18T20:27:00.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.751+0000 7fa4c67fc700 1 -- 172.21.15.1:0/688428349 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 153) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fa4cc014910 con 0x7fa4d415b300 2024-03-18T20:27:00.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.751+0000 7fa4c67fc700 1 -- 172.21.15.1:0/688428349 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 7767+0+0 (secure 0 0 0) 0x7fa4cc08acb0 con 0x7fa4d415b300 2024-03-18T20:27:00.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.755+0000 7fa4c67fc700 1 -- 172.21.15.1:0/688428349 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa4cc055a60 con 0x7fa4d415b300 2024-03-18T20:27:00.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.956+0000 7fa4dc0ac700 1 -- 172.21.15.1:0/688428349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4d40676a0 con 0x7fa4d415b300 2024-03-18T20:27:00.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.958+0000 7fa4c67fc700 1 -- 172.21.15.1:0/688428349 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v259) v1 ==== 74+0+21180 (secure 0 0 0) 0x7fa4cc059080 con 0x7fa4d415b300 2024-03-18T20:27:00.962 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:00.962 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":259,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:27:00.293297+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:26:23.615691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-18T20:26:24.759650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:26:37.326244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:26:38.564628+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"257","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/1135489018":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/1721669330":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/245086145":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3664103058":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3761024882":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3251013155":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/1287719189":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/2340206734":"2024-03-19T20:26:52.380559+0000","172.21.15.1:6833/148356":"2024-03-19T20:26:52.380559+0000","172.21.15.1:6832/148356":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/376843885":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/1340983360":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/369631788":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/2840085801":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/322251557":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/1615265957":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3049322266":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/1886004750":"2024-03-19T20:26:52.380559+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:27:00.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.960+0000 7fa4dc0ac700 1 -- 172.21.15.1:0/688428349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d415b300 msgr2=0x7fa4d41589a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:00.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.960+0000 7fa4dc0ac700 1 --2- 172.21.15.1:0/688428349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d415b300 0x7fa4d41589a0 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc007e40 tx=0x7fa4cc00e3c0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.960+0000 7fa4dc0ac700 1 -- 172.21.15.1:0/688428349 shutdown_connections 2024-03-18T20:27:00.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.960+0000 7fa4dc0ac700 1 --2- 172.21.15.1:0/688428349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4d415b300 0x7fa4d41589a0 unknown :-1 s=CLOSED pgs=2956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.960+0000 7fa4dc0ac700 1 --2- 172.21.15.1:0/688428349 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4d406ed50 0x7fa4d4155980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.960+0000 7fa4dc0ac700 1 --2- 172.21.15.1:0/688428349 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d40cc5e0 0x7fa4d4154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:00.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.960+0000 7fa4dc0ac700 1 -- 172.21.15.1:0/688428349 >> 172.21.15.1:0/688428349 conn(0x7fa4d40b2b50 msgr2=0x7fa4d40b10a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:00.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.960+0000 7fa4dc0ac700 1 -- 172.21.15.1:0/688428349 shutdown_connections 2024-03-18T20:27:00.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:00.960+0000 7fa4dc0ac700 1 -- 172.21.15.1:0/688428349 wait complete. 2024-03-18T20:27:00.977 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T20:27:00.977 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:27:01.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.098+0000 7fb15c37c700 1 Processor -- start 2024-03-18T20:27:01.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.098+0000 7fb15c37c700 1 -- start start 2024-03-18T20:27:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.099+0000 7fb15c37c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb15406a750 0x7fb1540cf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.099+0000 7fb15c37c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb1540cf960 0x7fb15406ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.099+0000 7fb15c37c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb1540cc6d0 0x7fb1540ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.099+0000 7fb15c37c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb15406d640 con 0x7fb15406a750 2024-03-18T20:27:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.099+0000 7fb15c37c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb15406de50 con 0x7fb1540cf960 2024-03-18T20:27:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.099+0000 7fb15c37c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1540acd80 con 0x7fb1540cc6d0 2024-03-18T20:27:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.099+0000 7fb15a118700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb15406a750 0x7fb1540cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.100+0000 7fb15a118700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb15406a750 0x7fb1540cf280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35014/0 (socket says 172.21.15.1:35014) 2024-03-18T20:27:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.100+0000 7fb15a118700 1 -- 172.21.15.1:0/4149245496 learned_addr learned my addr 172.21.15.1:0/4149245496 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.100+0000 7fb15a118700 1 -- 172.21.15.1:0/4149245496 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb1540cc6d0 msgr2=0x7fb1540ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.100+0000 7fb15a118700 1 --2- 172.21.15.1:0/4149245496 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb1540cc6d0 0x7fb1540ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.100+0000 7fb15a118700 1 -- 172.21.15.1:0/4149245496 >> v2:172.21.15.165:3300/0 conn(0x7fb1540cf960 msgr2=0x7fb15406ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.100+0000 7fb15a118700 1 --2- 172.21.15.1:0/4149245496 >> v2:172.21.15.165:3300/0 conn(0x7fb1540cf960 0x7fb15406ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.100+0000 7fb15a118700 1 -- 172.21.15.1:0/4149245496 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15406b070 con 0x7fb15406a750 2024-03-18T20:27:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.100+0000 7fb15a118700 1 --2- 172.21.15.1:0/4149245496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15406a750 0x7fb1540cf280 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 crypto rx=0x7fb1540715b0 tx=0x7fb150009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0bdbd1d8edd36 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.100+0000 7fb159116700 1 -- 172.21.15.1:0/4149245496 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15000b870 con 0x7fb15406a750 2024-03-18T20:27:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.100+0000 7fb159116700 1 -- 172.21.15.1:0/4149245496 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15000b9d0 con 0x7fb15406a750 2024-03-18T20:27:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.100+0000 7fb159116700 1 -- 172.21.15.1:0/4149245496 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15000bd60 con 0x7fb15406a750 2024-03-18T20:27:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.101+0000 7fb15c37c700 1 -- 172.21.15.1:0/4149245496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15406a750 msgr2=0x7fb1540cf280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.101+0000 7fb15c37c700 1 --2- 172.21.15.1:0/4149245496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15406a750 0x7fb1540cf280 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 crypto rx=0x7fb1540715b0 tx=0x7fb150009bd0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.101+0000 7fb15c37c700 1 -- 172.21.15.1:0/4149245496 shutdown_connections 2024-03-18T20:27:01.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.101+0000 7fb15c37c700 1 --2- 172.21.15.1:0/4149245496 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb1540cc6d0 0x7fb1540ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.101+0000 7fb15c37c700 1 --2- 172.21.15.1:0/4149245496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15406a750 0x7fb1540cf280 unknown :-1 s=CLOSED pgs=2957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.101+0000 7fb15c37c700 1 --2- 172.21.15.1:0/4149245496 >> v2:172.21.15.165:3300/0 conn(0x7fb1540cf960 0x7fb15406ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.101+0000 7fb15c37c700 1 -- 172.21.15.1:0/4149245496 >> 172.21.15.1:0/4149245496 conn(0x7fb15406b860 msgr2=0x7fb15406bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:01.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.101+0000 7fb15c37c700 1 -- 172.21.15.1:0/4149245496 shutdown_connections 2024-03-18T20:27:01.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.101+0000 7fb15c37c700 1 -- 172.21.15.1:0/4149245496 wait complete. 2024-03-18T20:27:01.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.102+0000 7fb15c37c700 1 Processor -- start 2024-03-18T20:27:01.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.102+0000 7fb15c37c700 1 -- start start 2024-03-18T20:27:01.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.103+0000 7fb15c37c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15406a750 0x7fb1541e5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.103+0000 7fb15c37c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb1540cc6d0 0x7fb15409cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.103+0000 7fb15c37c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1540cf960 0x7fb15414c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.103+0000 7fb15c37c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb154157f90 con 0x7fb15406a750 2024-03-18T20:27:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.103+0000 7fb15c37c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1541e5ab0 con 0x7fb1540cf960 2024-03-18T20:27:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.103+0000 7fb15c37c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1541e44d0 con 0x7fb1540cc6d0 2024-03-18T20:27:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.103+0000 7fb15a118700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15406a750 0x7fb1541e5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.103+0000 7fb15a118700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15406a750 0x7fb1541e5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35028/0 (socket says 172.21.15.1:35028) 2024-03-18T20:27:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.103+0000 7fb15a118700 1 -- 172.21.15.1:0/1201972183 learned_addr learned my addr 172.21.15.1:0/1201972183 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.103+0000 7fb15a919700 1 --2- 172.21.15.1:0/1201972183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1540cf960 0x7fb15414c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.103+0000 7fb159917700 1 --2- 172.21.15.1:0/1201972183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb1540cc6d0 0x7fb15409cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.103+0000 7fb15a118700 1 -- 172.21.15.1:0/1201972183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb1540cc6d0 msgr2=0x7fb15409cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.103+0000 7fb15a118700 1 --2- 172.21.15.1:0/1201972183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb1540cc6d0 0x7fb15409cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.103+0000 7fb15a118700 1 -- 172.21.15.1:0/1201972183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1540cf960 msgr2=0x7fb15414c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.103+0000 7fb15a118700 1 --2- 172.21.15.1:0/1201972183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1540cf960 0x7fb15414c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.103+0000 7fb15a118700 1 -- 172.21.15.1:0/1201972183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb150009550 con 0x7fb15406a750 2024-03-18T20:27:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.103+0000 7fb159917700 1 --2- 172.21.15.1:0/1201972183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb1540cc6d0 0x7fb15409cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.103+0000 7fb15a118700 1 --2- 172.21.15.1:0/1201972183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15406a750 0x7fb1541e5570 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7fb150014040 tx=0x7fb150004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.104+0000 7fb146ffd700 1 -- 172.21.15.1:0/1201972183 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1500128c0 con 0x7fb15406a750 2024-03-18T20:27:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.104+0000 7fb146ffd700 1 -- 172.21.15.1:0/1201972183 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15001e070 con 0x7fb15406a750 2024-03-18T20:27:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.104+0000 7fb15c37c700 1 -- 172.21.15.1:0/1201972183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1541e47b0 con 0x7fb15406a750 2024-03-18T20:27:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.104+0000 7fb146ffd700 1 -- 172.21.15.1:0/1201972183 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb150012c60 con 0x7fb15406a750 2024-03-18T20:27:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.104+0000 7fb15c37c700 1 -- 172.21.15.1:0/1201972183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1541e4d70 con 0x7fb15406a750 2024-03-18T20:27:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.105+0000 7fb15c37c700 1 -- 172.21.15.1:0/1201972183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1540676a0 con 0x7fb15406a750 2024-03-18T20:27:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.105+0000 7fb146ffd700 1 -- 172.21.15.1:0/1201972183 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 153) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fb15001a420 con 0x7fb15406a750 2024-03-18T20:27:01.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.106+0000 7fb146ffd700 1 -- 172.21.15.1:0/1201972183 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 7767+0+0 (secure 0 0 0) 0x7fb15008a7e0 con 0x7fb15406a750 2024-03-18T20:27:01.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.110+0000 7fb146ffd700 1 -- 172.21.15.1:0/1201972183 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb150055580 con 0x7fb15406a750 2024-03-18T20:27:01.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.310+0000 7fb146ffd700 1 -- 172.21.15.1:0/1201972183 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 154) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fb150055030 con 0x7fb15406a750 2024-03-18T20:27:01.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.310+0000 7fb146ffd700 1 --2- 172.21.15.1:0/1201972183 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb138073260 0x7fb138075650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.311+0000 7fb159917700 1 --2- 172.21.15.1:0/1201972183 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb138073260 0x7fb138075650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.312+0000 7fb159917700 1 --2- 172.21.15.1:0/1201972183 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb138073260 0x7fb138075650 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb148006fd0 tx=0x7fb148008040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.368+0000 7fb15c37c700 1 -- 172.21.15.1:0/1201972183 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb154067db0 con 0x7fb15406a750 2024-03-18T20:27:01.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.369+0000 7fb146ffd700 1 -- 172.21.15.1:0/1201972183 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+5963 (secure 0 0 0) 0x7fb150015090 con 0x7fb15406a750 2024-03-18T20:27:01.373 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:01.373 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":126,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12396,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2565889517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2565889517},{"type":"v1","addr":"172.21.15.1:6836","nonce":2565889517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":12399,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/512665659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":512665659},{"type":"v1","addr":"172.21.15.1:6837","nonce":512665659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:26:25.785074+0000","modified":"2024-03-18T20:26:29.101123+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12411":{"gid":12411,"name":"d","rank":0,"incarnation":116,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/3886150679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":3886150679},{"type":"v1","addr":"172.21.15.165:6835","nonce":3886150679}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:26:39.596187+0000","modified":"2024-03-18T20:26:42.614846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12408":{"gid":12408,"name":"b","rank":0,"incarnation":122,"state":"up:active","state_seq":6,"addr":"172.21.15.165:6834/1134181166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1134181166},{"type":"v1","addr":"172.21.15.165:6834","nonce":1134181166}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-03-18T20:27:01.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.371+0000 7fb15c37c700 1 -- 172.21.15.1:0/1201972183 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb138073260 msgr2=0x7fb138075650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:01.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.371+0000 7fb15c37c700 1 --2- 172.21.15.1:0/1201972183 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb138073260 0x7fb138075650 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb148006fd0 tx=0x7fb148008040 comp rx=0 tx=0).stop 2024-03-18T20:27:01.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.371+0000 7fb15c37c700 1 -- 172.21.15.1:0/1201972183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15406a750 msgr2=0x7fb1541e5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:01.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.371+0000 7fb15c37c700 1 --2- 172.21.15.1:0/1201972183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15406a750 0x7fb1541e5570 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7fb150014040 tx=0x7fb150004750 comp rx=0 tx=0).stop 2024-03-18T20:27:01.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.371+0000 7fb15c37c700 1 -- 172.21.15.1:0/1201972183 shutdown_connections 2024-03-18T20:27:01.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.371+0000 7fb15c37c700 1 --2- 172.21.15.1:0/1201972183 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb138073260 0x7fb138075650 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.371+0000 7fb15c37c700 1 --2- 172.21.15.1:0/1201972183 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb1540cf960 0x7fb15414c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.371+0000 7fb15c37c700 1 --2- 172.21.15.1:0/1201972183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb15406a750 0x7fb1541e5570 unknown :-1 s=CLOSED pgs=2958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.371+0000 7fb15c37c700 1 --2- 172.21.15.1:0/1201972183 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb1540cc6d0 0x7fb15409cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.371+0000 7fb15c37c700 1 -- 172.21.15.1:0/1201972183 >> 172.21.15.1:0/1201972183 conn(0x7fb15406b860 msgr2=0x7fb15406f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:01.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.371+0000 7fb15c37c700 1 -- 172.21.15.1:0/1201972183 shutdown_connections 2024-03-18T20:27:01.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.371+0000 7fb15c37c700 1 -- 172.21.15.1:0/1201972183 wait complete. 2024-03-18T20:27:01.377 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 126 2024-03-18T20:27:01.388 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:27:01.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.509+0000 7feba2099700 1 Processor -- start 2024-03-18T20:27:01.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.509+0000 7feba2099700 1 -- start start 2024-03-18T20:27:01.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.510+0000 7feba2099700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb9c06d4a0 0x7feb9c15c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.510+0000 7feba2099700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb9c15c8b0 con 0x7feb9c0cf0c0 2024-03-18T20:27:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.510+0000 7feba2099700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb9c0aefc0 con 0x7feb9c0aebe0 2024-03-18T20:27:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.510+0000 7feba2099700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c0cd740 con 0x7feb9c06d4a0 2024-03-18T20:27:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.510+0000 7feb9affd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7feb9c0cf0c0 0x7feb9c06cd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45060/0 (socket says 172.21.15.1:45060) 2024-03-18T20:27:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.510+0000 7feb9affd700 1 -- 172.21.15.1:0/3847735897 learned_addr learned my addr 172.21.15.1:0/3847735897 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.510+0000 7feb9a7fc700 1 -- 172.21.15.1:0/3847735897 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712970686 0 0) 0x7feb84001110 con 0x7feb9c0cf0c0 2024-03-18T20:27:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.510+0000 7feb9a7fc700 1 -- 172.21.15.1:0/3847735897 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb880035b0 con 0x7feb9c0cf0c0 2024-03-18T20:27:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.510+0000 7feb9bfff700 1 --2- 172.21.15.1:0/3847735897 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb9c06d4a0 0x7feb9c15c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9a7fc700 1 -- 172.21.15.1:0/3847735897 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4075147125 0 0) 0x7feb880035b0 con 0x7feb9c0cf0c0 2024-03-18T20:27:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9a7fc700 1 -- 172.21.15.1:0/3847735897 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb9c06d4a0 msgr2=0x7feb9c15c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9a7fc700 1 --2- 172.21.15.1:0/3847735897 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb9c06d4a0 0x7feb9c15c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9a7fc700 1 -- 172.21.15.1:0/3847735897 >> v1:172.21.15.165:6789/0 conn(0x7feb9c0aebe0 legacy=0x7feb9c0cc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9a7fc700 1 -- 172.21.15.1:0/3847735897 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb9c06e700 con 0x7feb9c0cf0c0 2024-03-18T20:27:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9a7fc700 1 -- 172.21.15.1:0/3847735897 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7feb84001800 con 0x7feb9c0cf0c0 2024-03-18T20:27:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9a7fc700 1 -- 172.21.15.1:0/3847735897 >> v1:172.21.15.1:6789/0 conn(0x7feb9c0cf0c0 legacy=0x7feb9c06cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9a7fc700 1 --2- 172.21.15.1:0/3847735897 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb88003a30 0x7feb88003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9a7fc700 1 --2- 172.21.15.1:0/3847735897 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb9c06d4a0 0x7feb88004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9a7fc700 1 --2- 172.21.15.1:0/3847735897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb88004d90 0x7feb88005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9a7fc700 1 -- 172.21.15.1:0/3847735897 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c0cd740 con 0x7feb88004d90 2024-03-18T20:27:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9a7fc700 1 -- 172.21.15.1:0/3847735897 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8800bba0 con 0x7feb88003a30 2024-03-18T20:27:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9a7fc700 1 -- 172.21.15.1:0/3847735897 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8800bd60 con 0x7feb9c06d4a0 2024-03-18T20:27:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9bfff700 1 --2- 172.21.15.1:0/3847735897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb88004d90 0x7feb88005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9affd700 1 --2- 172.21.15.1:0/3847735897 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb9c06d4a0 0x7feb88004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9b7fe700 1 --2- 172.21.15.1:0/3847735897 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb88003a30 0x7feb88003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9bfff700 1 -- 172.21.15.1:0/3847735897 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb9c06d4a0 msgr2=0x7feb88004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9bfff700 1 --2- 172.21.15.1:0/3847735897 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb9c06d4a0 0x7feb88004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9bfff700 1 -- 172.21.15.1:0/3847735897 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb88003a30 msgr2=0x7feb88003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:01.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9bfff700 1 --2- 172.21.15.1:0/3847735897 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb88003a30 0x7feb88003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9bfff700 1 -- 172.21.15.1:0/3847735897 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb880035b0 con 0x7feb88004d90 2024-03-18T20:27:01.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9bfff700 1 --2- 172.21.15.1:0/3847735897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb88004d90 0x7feb88005210 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0cf650 tx=0x7feb88005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f83d40ed19fbdc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:01.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9a7fc700 1 -- 172.21.15.1:0/3847735897 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9400ebd0 con 0x7feb88004d90 2024-03-18T20:27:01.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9a7fc700 1 -- 172.21.15.1:0/3847735897 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9400f040 con 0x7feb88004d90 2024-03-18T20:27:01.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.511+0000 7feb9a7fc700 1 -- 172.21.15.1:0/3847735897 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb94014480 con 0x7feb88004d90 2024-03-18T20:27:01.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.512+0000 7feba2099700 1 -- 172.21.15.1:0/3847735897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb88004d90 msgr2=0x7feb88005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:01.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.512+0000 7feba2099700 1 --2- 172.21.15.1:0/3847735897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb88004d90 0x7feb88005210 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0cf650 tx=0x7feb88005880 comp rx=0 tx=0).stop 2024-03-18T20:27:01.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.512+0000 7feb9bfff700 1 -- 172.21.15.1:0/3847735897 reap_dead start 2024-03-18T20:27:01.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.512+0000 7feba2099700 1 -- 172.21.15.1:0/3847735897 shutdown_connections 2024-03-18T20:27:01.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.512+0000 7feba2099700 1 -- 172.21.15.1:0/3847735897 >> 172.21.15.1:0/3847735897 conn(0x7feb9c0b1270 msgr2=0x7feb9c0b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:01.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.512+0000 7feba2099700 1 -- 172.21.15.1:0/3847735897 shutdown_connections 2024-03-18T20:27:01.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.512+0000 7feba2099700 1 -- 172.21.15.1:0/3847735897 wait complete. 2024-03-18T20:27:01.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.513+0000 7feba2099700 1 Processor -- start 2024-03-18T20:27:01.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.513+0000 7feba2099700 1 -- start start 2024-03-18T20:27:01.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.514+0000 7feba2099700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb88004d90 0x7feb9c0c95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.514+0000 7feba2099700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb9c14a1d0 0x7feb9c147000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.514+0000 7feba2099700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb9c1f4d90 0x7feb9c15a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.514+0000 7feba2099700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c1f4380 con 0x7feb88004d90 2024-03-18T20:27:01.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.514+0000 7feba2099700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c0c9ae0 con 0x7feb9c14a1d0 2024-03-18T20:27:01.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.514+0000 7feba2099700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c0c56b0 con 0x7feb9c1f4d90 2024-03-18T20:27:01.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.514+0000 7feb9b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb88004d90 0x7feb9c0c95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.514+0000 7feb9b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb88004d90 0x7feb9c0c95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35054/0 (socket says 172.21.15.1:35054) 2024-03-18T20:27:01.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.514+0000 7feb9b7fe700 1 -- 172.21.15.1:0/4047669373 learned_addr learned my addr 172.21.15.1:0/4047669373 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:01.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.514+0000 7feb9affd700 1 --2- 172.21.15.1:0/4047669373 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb9c14a1d0 0x7feb9c147000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.514+0000 7feb9bfff700 1 --2- 172.21.15.1:0/4047669373 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb9c1f4d90 0x7feb9c15a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.514+0000 7feb9b7fe700 1 -- 172.21.15.1:0/4047669373 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb9c1f4d90 msgr2=0x7feb9c15a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:01.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.514+0000 7feb9b7fe700 1 --2- 172.21.15.1:0/4047669373 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb9c1f4d90 0x7feb9c15a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.515+0000 7feb9b7fe700 1 -- 172.21.15.1:0/4047669373 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb9c14a1d0 msgr2=0x7feb9c147000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:01.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.515+0000 7feb9b7fe700 1 --2- 172.21.15.1:0/4047669373 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb9c14a1d0 0x7feb9c147000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.515+0000 7feb9b7fe700 1 -- 172.21.15.1:0/4047669373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb9c147540 con 0x7feb88004d90 2024-03-18T20:27:01.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.515+0000 7feb9b7fe700 1 --2- 172.21.15.1:0/4047669373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb88004d90 0x7feb9c0c95a0 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 crypto rx=0x7feb8c00f770 tx=0x7feb8c00fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:01.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.515+0000 7feb93fff700 1 -- 172.21.15.1:0/4047669373 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8c0052e0 con 0x7feb88004d90 2024-03-18T20:27:01.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.515+0000 7feb93fff700 1 -- 172.21.15.1:0/4047669373 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8c005460 con 0x7feb88004d90 2024-03-18T20:27:01.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.515+0000 7feba2099700 1 -- 172.21.15.1:0/4047669373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb9c0c5930 con 0x7feb88004d90 2024-03-18T20:27:01.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.515+0000 7feb93fff700 1 -- 172.21.15.1:0/4047669373 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8c0038c0 con 0x7feb88004d90 2024-03-18T20:27:01.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.515+0000 7feba2099700 1 -- 172.21.15.1:0/4047669373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb9c1f4590 con 0x7feb88004d90 2024-03-18T20:27:01.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.516+0000 7feba2099700 1 -- 172.21.15.1:0/4047669373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb9c0676a0 con 0x7feb88004d90 2024-03-18T20:27:01.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.516+0000 7feb93fff700 1 -- 172.21.15.1:0/4047669373 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 154) v1 ==== 88975+0+0 (secure 0 0 0) 0x7feb8c003a20 con 0x7feb88004d90 2024-03-18T20:27:01.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.517+0000 7feb93fff700 1 --2- 172.21.15.1:0/4047669373 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7feb7c06aae0 0x7feb7c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.517+0000 7feb93fff700 1 -- 172.21.15.1:0/4047669373 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 7767+0+0 (secure 0 0 0) 0x7feb8c089660 con 0x7feb88004d90 2024-03-18T20:27:01.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.517+0000 7feb9affd700 1 --2- 172.21.15.1:0/4047669373 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7feb7c06aae0 0x7feb7c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.517+0000 7feb9affd700 1 --2- 172.21.15.1:0/4047669373 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7feb7c06aae0 0x7feb7c06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7feb9c14a5b0 tx=0x7feb84009040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:01.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.521+0000 7feb93fff700 1 -- 172.21.15.1:0/4047669373 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feb8c054370 con 0x7feb88004d90 2024-03-18T20:27:01.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.770+0000 7feba2099700 1 -- 172.21.15.1:0/4047669373 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb9c15ec80 con 0x7feb88004d90 2024-03-18T20:27:01.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.771+0000 7feb93fff700 1 -- 172.21.15.1:0/4047669373 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+5963 (secure 0 0 0) 0x7feb8c057990 con 0x7feb88004d90 2024-03-18T20:27:01.775 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:01.775 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":126,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12396,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2565889517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2565889517},{"type":"v1","addr":"172.21.15.1:6836","nonce":2565889517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":12399,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/512665659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":512665659},{"type":"v1","addr":"172.21.15.1:6837","nonce":512665659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:26:25.785074+0000","modified":"2024-03-18T20:26:29.101123+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12411":{"gid":12411,"name":"d","rank":0,"incarnation":116,"state":"up:active","state_seq":3,"addr":"172.21.15.165:6835/3886150679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":3886150679},{"type":"v1","addr":"172.21.15.165:6835","nonce":3886150679}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:26:39.596187+0000","modified":"2024-03-18T20:26:42.614846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12408":{"gid":12408,"name":"b","rank":0,"incarnation":122,"state":"up:active","state_seq":6,"addr":"172.21.15.165:6834/1134181166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1134181166},{"type":"v1","addr":"172.21.15.165:6834","nonce":1134181166}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-03-18T20:27:01.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.773+0000 7feba2099700 1 -- 172.21.15.1:0/4047669373 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7feb7c06aae0 msgr2=0x7feb7c06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:01.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.773+0000 7feba2099700 1 --2- 172.21.15.1:0/4047669373 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7feb7c06aae0 0x7feb7c06cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7feb9c14a5b0 tx=0x7feb84009040 comp rx=0 tx=0).stop 2024-03-18T20:27:01.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.773+0000 7feba2099700 1 -- 172.21.15.1:0/4047669373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb88004d90 msgr2=0x7feb9c0c95a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:01.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.773+0000 7feba2099700 1 --2- 172.21.15.1:0/4047669373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb88004d90 0x7feb9c0c95a0 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 crypto rx=0x7feb8c00f770 tx=0x7feb8c00fb30 comp rx=0 tx=0).stop 2024-03-18T20:27:01.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.773+0000 7feba2099700 1 -- 172.21.15.1:0/4047669373 shutdown_connections 2024-03-18T20:27:01.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.773+0000 7feba2099700 1 --2- 172.21.15.1:0/4047669373 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7feb7c06aae0 0x7feb7c06cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.773+0000 7feba2099700 1 --2- 172.21.15.1:0/4047669373 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7feb9c1f4d90 0x7feb9c15a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.773+0000 7feba2099700 1 --2- 172.21.15.1:0/4047669373 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb88004d90 0x7feb9c0c95a0 unknown :-1 s=CLOSED pgs=2961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.773+0000 7feba2099700 1 --2- 172.21.15.1:0/4047669373 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb9c14a1d0 0x7feb9c147000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.773+0000 7feba2099700 1 -- 172.21.15.1:0/4047669373 >> 172.21.15.1:0/4047669373 conn(0x7feb9c0b1270 msgr2=0x7feb9c0cddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:01.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.773+0000 7feba2099700 1 -- 172.21.15.1:0/4047669373 shutdown_connections 2024-03-18T20:27:01.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.773+0000 7feba2099700 1 -- 172.21.15.1:0/4047669373 wait complete. 2024-03-18T20:27:01.779 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 126 2024-03-18T20:27:01.790 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:27:01.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.911+0000 7fa2ff5e8700 1 Processor -- start 2024-03-18T20:27:01.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.911+0000 7fa2ff5e8700 1 -- start start 2024-03-18T20:27:01.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.912+0000 7fa2ff5e8700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa2f809a150 0x7fa2f8166070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.912+0000 7fa2ff5e8700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f806f070 con 0x7fa2f809a650 2024-03-18T20:27:01.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.912+0000 7fa2ff5e8700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f80d0260 con 0x7fa2f815b4d0 2024-03-18T20:27:01.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.912+0000 7fa2ff5e8700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f80cff80 con 0x7fa2f809a150 2024-03-18T20:27:01.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.912+0000 7fa2fcb83700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa2f809a650 0x7fa2f8099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45074/0 (socket says 172.21.15.1:45074) 2024-03-18T20:27:01.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.912+0000 7fa2fcb83700 1 -- 172.21.15.1:0/1013149016 learned_addr learned my addr 172.21.15.1:0/1013149016 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:01.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.912+0000 7fa2fdb85700 1 --2- 172.21.15.1:0/1013149016 >> v2:172.21.15.165:3300/0 conn(0x7fa2f809a150 0x7fa2f8166070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.912+0000 7fa2effff700 1 -- 172.21.15.1:0/1013149016 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814428719 0 0) 0x7fa2f806f070 con 0x7fa2f809a650 2024-03-18T20:27:01.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.912+0000 7fa2effff700 1 -- 172.21.15.1:0/1013149016 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e40035b0 con 0x7fa2f809a650 2024-03-18T20:27:01.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.912+0000 7fa2effff700 1 -- 172.21.15.1:0/1013149016 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2325395196 0 0) 0x7fa2e40035b0 con 0x7fa2f809a650 2024-03-18T20:27:01.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.912+0000 7fa2effff700 1 -- 172.21.15.1:0/1013149016 >> v2:172.21.15.165:3300/0 conn(0x7fa2f809a150 msgr2=0x7fa2f8166070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:01.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2effff700 1 --2- 172.21.15.1:0/1013149016 >> v2:172.21.15.165:3300/0 conn(0x7fa2f809a150 0x7fa2f8166070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2effff700 1 -- 172.21.15.1:0/1013149016 >> v1:172.21.15.165:6789/0 conn(0x7fa2f815b4d0 legacy=0x7fa2f80cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:01.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2effff700 1 -- 172.21.15.1:0/1013149016 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f806ea20 con 0x7fa2f809a650 2024-03-18T20:27:01.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2effff700 1 -- 172.21.15.1:0/1013149016 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fa2f4001670 con 0x7fa2f809a650 2024-03-18T20:27:01.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2fdb85700 1 --2- 172.21.15.1:0/1013149016 >> v2:172.21.15.165:3300/0 conn(0x7fa2f809a150 0x7fa2f8166070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:27:01.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2effff700 1 -- 172.21.15.1:0/1013149016 >> v1:172.21.15.1:6789/0 conn(0x7fa2f809a650 legacy=0x7fa2f8099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:01.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2effff700 1 --2- 172.21.15.1:0/1013149016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2e4003a00 0x7fa2e4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2effff700 1 --2- 172.21.15.1:0/1013149016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2e4004400 0x7fa2e40048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2effff700 1 --2- 172.21.15.1:0/1013149016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2e4005170 0x7fa2e400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2effff700 1 -- 172.21.15.1:0/1013149016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f80cff80 con 0x7fa2e4004400 2024-03-18T20:27:01.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2effff700 1 -- 172.21.15.1:0/1013149016 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e400c0b0 con 0x7fa2e4005170 2024-03-18T20:27:01.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2effff700 1 -- 172.21.15.1:0/1013149016 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e400c270 con 0x7fa2e4003a00 2024-03-18T20:27:01.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2fcb83700 1 --2- 172.21.15.1:0/1013149016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2e4004400 0x7fa2e40048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2fdb85700 1 --2- 172.21.15.1:0/1013149016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2e4005170 0x7fa2e400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2fd384700 1 --2- 172.21.15.1:0/1013149016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2e4003a00 0x7fa2e4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2fcb83700 1 -- 172.21.15.1:0/1013149016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2e4003a00 msgr2=0x7fa2e4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:01.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2fcb83700 1 --2- 172.21.15.1:0/1013149016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2e4003a00 0x7fa2e4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2fcb83700 1 -- 172.21.15.1:0/1013149016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2e4005170 msgr2=0x7fa2e400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:01.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2fcb83700 1 --2- 172.21.15.1:0/1013149016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2e4005170 0x7fa2e400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2fcb83700 1 -- 172.21.15.1:0/1013149016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e40035b0 con 0x7fa2e4004400 2024-03-18T20:27:01.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2fdb85700 1 -- 172.21.15.1:0/1013149016 reap_dead start 2024-03-18T20:27:01.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2fd384700 1 --2- 172.21.15.1:0/1013149016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2e4003a00 0x7fa2e4003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:01.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2fcb83700 1 --2- 172.21.15.1:0/1013149016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2e4004400 0x7fa2e40048d0 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400c2f0 tx=0x7fa2e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ba24191c4aa7ff server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:01.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2effff700 1 -- 172.21.15.1:0/1013149016 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f400cdb0 con 0x7fa2e4004400 2024-03-18T20:27:01.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2effff700 1 -- 172.21.15.1:0/1013149016 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f4005960 con 0x7fa2e4004400 2024-03-18T20:27:01.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.913+0000 7fa2effff700 1 -- 172.21.15.1:0/1013149016 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4005d10 con 0x7fa2e4004400 2024-03-18T20:27:01.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.914+0000 7fa2ff5e8700 1 -- 172.21.15.1:0/1013149016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2e4004400 msgr2=0x7fa2e40048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:01.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.914+0000 7fa2ff5e8700 1 --2- 172.21.15.1:0/1013149016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2e4004400 0x7fa2e40048d0 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400c2f0 tx=0x7fa2e4005880 comp rx=0 tx=0).stop 2024-03-18T20:27:01.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.914+0000 7fa2ff5e8700 1 -- 172.21.15.1:0/1013149016 shutdown_connections 2024-03-18T20:27:01.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.914+0000 7fa2ff5e8700 1 --2- 172.21.15.1:0/1013149016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2e4004400 0x7fa2e40048d0 unknown :-1 s=CLOSED pgs=2963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.914+0000 7fa2ff5e8700 1 -- 172.21.15.1:0/1013149016 >> 172.21.15.1:0/1013149016 conn(0x7fa2f80b2cb0 msgr2=0x7fa2f80b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:01.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.914+0000 7fa2ff5e8700 1 -- 172.21.15.1:0/1013149016 shutdown_connections 2024-03-18T20:27:01.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.914+0000 7fa2ff5e8700 1 -- 172.21.15.1:0/1013149016 wait complete. 2024-03-18T20:27:01.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.915+0000 7fa2ff5e8700 1 Processor -- start 2024-03-18T20:27:01.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.915+0000 7fa2ff5e8700 1 -- start start 2024-03-18T20:27:01.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.916+0000 7fa2ff5e8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2e4004400 0x7fa2f80c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.916+0000 7fa2ff5e8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2f80c69d0 0x7fa2f814c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.916+0000 7fa2ff5e8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2f80c6e50 0x7fa2f814c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.916+0000 7fa2ff5e8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f80b3e40 con 0x7fa2e4004400 2024-03-18T20:27:01.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.916+0000 7fa2ff5e8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f80c51d0 con 0x7fa2f80c6e50 2024-03-18T20:27:01.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.916+0000 7fa2ff5e8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f806ce10 con 0x7fa2f80c69d0 2024-03-18T20:27:01.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.916+0000 7fa2fd384700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2e4004400 0x7fa2f80c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.916+0000 7fa2fd384700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2e4004400 0x7fa2f80c4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35066/0 (socket says 172.21.15.1:35066) 2024-03-18T20:27:01.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.916+0000 7fa2fd384700 1 -- 172.21.15.1:0/332967838 learned_addr learned my addr 172.21.15.1:0/332967838 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:01.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.916+0000 7fa2fcb83700 1 --2- 172.21.15.1:0/332967838 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2f80c69d0 0x7fa2f814c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.916+0000 7fa2fdb85700 1 --2- 172.21.15.1:0/332967838 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2f80c6e50 0x7fa2f814c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.916+0000 7fa2fd384700 1 -- 172.21.15.1:0/332967838 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2f80c69d0 msgr2=0x7fa2f814c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:01.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.916+0000 7fa2fd384700 1 --2- 172.21.15.1:0/332967838 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2f80c69d0 0x7fa2f814c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.916+0000 7fa2fd384700 1 -- 172.21.15.1:0/332967838 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2f80c6e50 msgr2=0x7fa2f814c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:01.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.916+0000 7fa2fd384700 1 --2- 172.21.15.1:0/332967838 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2f80c6e50 0x7fa2f814c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:01.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.916+0000 7fa2fd384700 1 -- 172.21.15.1:0/332967838 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f80b40b0 con 0x7fa2e4004400 2024-03-18T20:27:01.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.916+0000 7fa2fcb83700 1 --2- 172.21.15.1:0/332967838 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2f80c69d0 0x7fa2f814c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:01.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.916+0000 7fa2fdb85700 1 --2- 172.21.15.1:0/332967838 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2f80c6e50 0x7fa2f814c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:01.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.917+0000 7fa2fd384700 1 --2- 172.21.15.1:0/332967838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2e4004400 0x7fa2f80c4c90 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7fa2e800e580 tx=0x7fa2e800e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:01.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.917+0000 7fa2edffb700 1 -- 172.21.15.1:0/332967838 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e80053a0 con 0x7fa2e4004400 2024-03-18T20:27:01.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.917+0000 7fa2edffb700 1 -- 172.21.15.1:0/332967838 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e8005500 con 0x7fa2e4004400 2024-03-18T20:27:01.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.917+0000 7fa2ff5e8700 1 -- 172.21.15.1:0/332967838 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f80b4390 con 0x7fa2e4004400 2024-03-18T20:27:01.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.917+0000 7fa2edffb700 1 -- 172.21.15.1:0/332967838 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e80038a0 con 0x7fa2e4004400 2024-03-18T20:27:01.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.917+0000 7fa2ff5e8700 1 -- 172.21.15.1:0/332967838 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f80b48b0 con 0x7fa2e4004400 2024-03-18T20:27:01.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.918+0000 7fa2ff5e8700 1 -- 172.21.15.1:0/332967838 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f8068240 con 0x7fa2e4004400 2024-03-18T20:27:01.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.918+0000 7fa2edffb700 1 -- 172.21.15.1:0/332967838 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 154) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fa2e8003a00 con 0x7fa2e4004400 2024-03-18T20:27:01.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.919+0000 7fa2edffb700 1 --2- 172.21.15.1:0/332967838 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa2dc06aae0 0x7fa2dc06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:01.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.919+0000 7fa2edffb700 1 -- 172.21.15.1:0/332967838 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 7767+0+0 (secure 0 0 0) 0x7fa2e8088ef0 con 0x7fa2e4004400 2024-03-18T20:27:01.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.919+0000 7fa2fcb83700 1 --2- 172.21.15.1:0/332967838 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa2dc06aae0 0x7fa2dc06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:01.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.919+0000 7fa2fcb83700 1 --2- 172.21.15.1:0/332967838 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa2dc06aae0 0x7fa2dc06cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa2f80afa00 tx=0x7fa2f4007000 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:01.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:01.923+0000 7fa2edffb700 1 -- 172.21.15.1:0/332967838 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa2e8053c20 con 0x7fa2e4004400 2024-03-18T20:27:02.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.129+0000 7fa2ff5e8700 1 -- 172.21.15.1:0/332967838 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2f80b45d0 con 0x7fa2e4004400 2024-03-18T20:27:02.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.130+0000 7fa2edffb700 1 -- 172.21.15.1:0/332967838 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v259) v1 ==== 74+0+21180 (secure 0 0 0) 0x7fa2f80b45d0 con 0x7fa2e4004400 2024-03-18T20:27:02.134 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:02.135 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":259,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:27:00.293297+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:26:23.615691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-18T20:26:24.759650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:26:37.326244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:26:38.564628+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"257","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/1135489018":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/1721669330":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/245086145":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3664103058":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3761024882":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3251013155":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/1287719189":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/2340206734":"2024-03-19T20:26:52.380559+0000","172.21.15.1:6833/148356":"2024-03-19T20:26:52.380559+0000","172.21.15.1:6832/148356":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/376843885":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/1340983360":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/369631788":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/2840085801":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/322251557":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/1615265957":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3049322266":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/1886004750":"2024-03-19T20:26:52.380559+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:27:02.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.132+0000 7fa2ff5e8700 1 -- 172.21.15.1:0/332967838 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa2dc06aae0 msgr2=0x7fa2dc06cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:02.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.132+0000 7fa2ff5e8700 1 --2- 172.21.15.1:0/332967838 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa2dc06aae0 0x7fa2dc06cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa2f80afa00 tx=0x7fa2f4007000 comp rx=0 tx=0).stop 2024-03-18T20:27:02.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.132+0000 7fa2ff5e8700 1 -- 172.21.15.1:0/332967838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2e4004400 msgr2=0x7fa2f80c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:02.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.132+0000 7fa2ff5e8700 1 --2- 172.21.15.1:0/332967838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2e4004400 0x7fa2f80c4c90 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7fa2e800e580 tx=0x7fa2e800e940 comp rx=0 tx=0).stop 2024-03-18T20:27:02.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.133+0000 7fa2ff5e8700 1 -- 172.21.15.1:0/332967838 shutdown_connections 2024-03-18T20:27:02.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.133+0000 7fa2ff5e8700 1 --2- 172.21.15.1:0/332967838 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa2dc06aae0 0x7fa2dc06cfa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:02.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.133+0000 7fa2ff5e8700 1 --2- 172.21.15.1:0/332967838 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2f80c6e50 0x7fa2f814c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:02.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.133+0000 7fa2ff5e8700 1 --2- 172.21.15.1:0/332967838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2e4004400 0x7fa2f80c4c90 unknown :-1 s=CLOSED pgs=2964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:02.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.133+0000 7fa2ff5e8700 1 --2- 172.21.15.1:0/332967838 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2f80c69d0 0x7fa2f814c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:02.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.133+0000 7fa2ff5e8700 1 -- 172.21.15.1:0/332967838 >> 172.21.15.1:0/332967838 conn(0x7fa2f80b2cb0 msgr2=0x7fa2f80ce480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:02.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.133+0000 7fa2ff5e8700 1 -- 172.21.15.1:0/332967838 shutdown_connections 2024-03-18T20:27:02.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.133+0000 7fa2ff5e8700 1 -- 172.21.15.1:0/332967838 wait complete. 2024-03-18T20:27:02.149 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T20:27:02.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.271+0000 7fa4d66df700 1 Processor -- start 2024-03-18T20:27:02.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.271+0000 7fa4d66df700 1 -- start start 2024-03-18T20:27:02.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.272+0000 7fa4d66df700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d006d560 0x7fa4d00aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:02.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.272+0000 7fa4d66df700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d015ef00 con 0x7fa4d006a750 2024-03-18T20:27:02.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.272+0000 7fa4d66df700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d015f1e0 con 0x7fa4d00af510 2024-03-18T20:27:02.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.272+0000 7fa4d66df700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d015f4c0 con 0x7fa4d006d560 2024-03-18T20:27:02.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.272+0000 7fa4cffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa4d006a750 0x7fa4d006ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45082/0 (socket says 172.21.15.1:45082) 2024-03-18T20:27:02.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.272+0000 7fa4cffff700 1 -- 172.21.15.1:0/3593362348 learned_addr learned my addr 172.21.15.1:0/3593362348 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:02.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.272+0000 7fa4ceffd700 1 -- 172.21.15.1:0/3593362348 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2389623500 0 0) 0x7fa4d015ef00 con 0x7fa4d006a750 2024-03-18T20:27:02.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.272+0000 7fa4cf7fe700 1 --2- 172.21.15.1:0/3593362348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d006d560 0x7fa4d00aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:02.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.272+0000 7fa4ceffd700 1 -- 172.21.15.1:0/3593362348 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4bc0035b0 con 0x7fa4d006a750 2024-03-18T20:27:02.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4ceffd700 1 -- 172.21.15.1:0/3593362348 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2117921426 0 0) 0x7fa4bc0035b0 con 0x7fa4d006a750 2024-03-18T20:27:02.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4ceffd700 1 -- 172.21.15.1:0/3593362348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d006d560 msgr2=0x7fa4d00aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:02.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4ceffd700 1 --2- 172.21.15.1:0/3593362348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d006d560 0x7fa4d00aefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:02.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4ceffd700 1 -- 172.21.15.1:0/3593362348 >> v1:172.21.15.165:6789/0 conn(0x7fa4d00af510 legacy=0x7fa4d015b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:02.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4ceffd700 1 -- 172.21.15.1:0/3593362348 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d015f7a0 con 0x7fa4d006a750 2024-03-18T20:27:02.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4ceffd700 1 -- 172.21.15.1:0/3593362348 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fa4c0001670 con 0x7fa4d006a750 2024-03-18T20:27:02.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4ceffd700 1 -- 172.21.15.1:0/3593362348 >> v1:172.21.15.1:6789/0 conn(0x7fa4d006a750 legacy=0x7fa4d006ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:02.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4ceffd700 1 --2- 172.21.15.1:0/3593362348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4bc003a30 0x7fa4bc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:02.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4ceffd700 1 --2- 172.21.15.1:0/3593362348 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4bc0043e0 0x7fa4bc0048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:02.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4ceffd700 1 --2- 172.21.15.1:0/3593362348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d006d560 0x7fa4bc005110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:02.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4ceffd700 1 -- 172.21.15.1:0/3593362348 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d015f4c0 con 0x7fa4bc003a30 2024-03-18T20:27:02.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4ceffd700 1 -- 172.21.15.1:0/3593362348 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc00ba10 con 0x7fa4bc0043e0 2024-03-18T20:27:02.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4ceffd700 1 -- 172.21.15.1:0/3593362348 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc00bbd0 con 0x7fa4d006d560 2024-03-18T20:27:02.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4cffff700 1 --2- 172.21.15.1:0/3593362348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4bc003a30 0x7fa4bc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:02.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4cf7fe700 1 --2- 172.21.15.1:0/3593362348 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4bc0043e0 0x7fa4bc0048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:02.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4d4c7c700 1 --2- 172.21.15.1:0/3593362348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d006d560 0x7fa4bc005110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:02.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4cffff700 1 -- 172.21.15.1:0/3593362348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d006d560 msgr2=0x7fa4bc005110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:02.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4cffff700 1 --2- 172.21.15.1:0/3593362348 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d006d560 0x7fa4bc005110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:02.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4cffff700 1 -- 172.21.15.1:0/3593362348 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4bc0043e0 msgr2=0x7fa4bc0048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:02.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4cffff700 1 --2- 172.21.15.1:0/3593362348 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4bc0043e0 0x7fa4bc0048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:02.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4cffff700 1 -- 172.21.15.1:0/3593362348 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4bc0035b0 con 0x7fa4bc003a30 2024-03-18T20:27:02.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.273+0000 7fa4cffff700 1 --2- 172.21.15.1:0/3593362348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4bc003a30 0x7fa4bc003e10 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 crypto rx=0x7fa4c000c2f0 tx=0x7fa4bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3045b0c5a1abd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:02.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.274+0000 7fa4ceffd700 1 -- 172.21.15.1:0/3593362348 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c000ce30 con 0x7fa4bc003a30 2024-03-18T20:27:02.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.274+0000 7fa4ceffd700 1 -- 172.21.15.1:0/3593362348 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c0005960 con 0x7fa4bc003a30 2024-03-18T20:27:02.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.274+0000 7fa4ceffd700 1 -- 172.21.15.1:0/3593362348 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0005cf0 con 0x7fa4bc003a30 2024-03-18T20:27:02.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.274+0000 7fa4d66df700 1 -- 172.21.15.1:0/3593362348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4bc003a30 msgr2=0x7fa4bc003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:02.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.274+0000 7fa4d66df700 1 --2- 172.21.15.1:0/3593362348 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4bc003a30 0x7fa4bc003e10 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 crypto rx=0x7fa4c000c2f0 tx=0x7fa4bc005880 comp rx=0 tx=0).stop 2024-03-18T20:27:02.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.274+0000 7fa4d4c7c700 1 -- 172.21.15.1:0/3593362348 reap_dead start 2024-03-18T20:27:02.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.274+0000 7fa4d66df700 1 -- 172.21.15.1:0/3593362348 shutdown_connections 2024-03-18T20:27:02.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.274+0000 7fa4d66df700 1 -- 172.21.15.1:0/3593362348 >> 172.21.15.1:0/3593362348 conn(0x7fa4d00cff30 msgr2=0x7fa4d00d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:02.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.274+0000 7fa4d66df700 1 -- 172.21.15.1:0/3593362348 shutdown_connections 2024-03-18T20:27:02.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.274+0000 7fa4d66df700 1 -- 172.21.15.1:0/3593362348 wait complete. 2024-03-18T20:27:02.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.275+0000 7fa4d66df700 1 Processor -- start 2024-03-18T20:27:02.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.276+0000 7fa4d66df700 1 -- start start 2024-03-18T20:27:02.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.276+0000 7fa4d66df700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4bc003a30 0x7fa4d00c2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:02.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.276+0000 7fa4d66df700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4d00b5720 0x7fa4d01f6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:02.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.276+0000 7fa4d66df700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d01f6a40 0x7fa4d00ce830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:02.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.276+0000 7fa4d66df700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d0154ce0 con 0x7fa4bc003a30 2024-03-18T20:27:02.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.276+0000 7fa4d66df700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d01f6ea0 con 0x7fa4d00b5720 2024-03-18T20:27:02.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.276+0000 7fa4d66df700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d00cdb00 con 0x7fa4d01f6a40 2024-03-18T20:27:02.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.276+0000 7fa4cffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4bc003a30 0x7fa4d00c2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:02.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.276+0000 7fa4cffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4bc003a30 0x7fa4d00c2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35080/0 (socket says 172.21.15.1:35080) 2024-03-18T20:27:02.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.276+0000 7fa4cffff700 1 -- 172.21.15.1:0/2984919123 learned_addr learned my addr 172.21.15.1:0/2984919123 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:02.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.277+0000 7fa4cf7fe700 1 --2- 172.21.15.1:0/2984919123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4d00b5720 0x7fa4d01f6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:02.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.277+0000 7fa4d4c7c700 1 --2- 172.21.15.1:0/2984919123 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d01f6a40 0x7fa4d00ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:02.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.277+0000 7fa4cffff700 1 -- 172.21.15.1:0/2984919123 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d01f6a40 msgr2=0x7fa4d00ce830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:02.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.277+0000 7fa4cffff700 1 --2- 172.21.15.1:0/2984919123 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d01f6a40 0x7fa4d00ce830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:02.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.277+0000 7fa4cffff700 1 -- 172.21.15.1:0/2984919123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4d00b5720 msgr2=0x7fa4d01f6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:02.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.277+0000 7fa4cffff700 1 --2- 172.21.15.1:0/2984919123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4d00b5720 0x7fa4d01f6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:02.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.277+0000 7fa4cffff700 1 -- 172.21.15.1:0/2984919123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d00c6360 con 0x7fa4bc003a30 2024-03-18T20:27:02.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.277+0000 7fa4cffff700 1 --2- 172.21.15.1:0/2984919123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4bc003a30 0x7fa4d00c2b00 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00011c0 tx=0x7fa4c0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:02.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.277+0000 7fa4ccff9700 1 -- 172.21.15.1:0/2984919123 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0004870 con 0x7fa4bc003a30 2024-03-18T20:27:02.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.277+0000 7fa4ccff9700 1 -- 172.21.15.1:0/2984919123 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c00049d0 con 0x7fa4bc003a30 2024-03-18T20:27:02.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.277+0000 7fa4d66df700 1 -- 172.21.15.1:0/2984919123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d00cdd50 con 0x7fa4bc003a30 2024-03-18T20:27:02.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.277+0000 7fa4ccff9700 1 -- 172.21.15.1:0/2984919123 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0004d60 con 0x7fa4bc003a30 2024-03-18T20:27:02.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.277+0000 7fa4d66df700 1 -- 172.21.15.1:0/2984919123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d00ce310 con 0x7fa4bc003a30 2024-03-18T20:27:02.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.278+0000 7fa4d66df700 1 -- 172.21.15.1:0/2984919123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d00676a0 con 0x7fa4bc003a30 2024-03-18T20:27:02.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.279+0000 7fa4ccff9700 1 -- 172.21.15.1:0/2984919123 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 154) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fa4c00070d0 con 0x7fa4bc003a30 2024-03-18T20:27:02.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.279+0000 7fa4ccff9700 1 --2- 172.21.15.1:0/2984919123 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa4b006ab10 0x7fa4b006cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:02.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.279+0000 7fa4ccff9700 1 -- 172.21.15.1:0/2984919123 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 7767+0+0 (secure 0 0 0) 0x7fa4c008aa90 con 0x7fa4bc003a30 2024-03-18T20:27:02.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.279+0000 7fa4cf7fe700 1 --2- 172.21.15.1:0/2984919123 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa4b006ab10 0x7fa4b006cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:02.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.279+0000 7fa4cf7fe700 1 --2- 172.21.15.1:0/2984919123 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa4b006ab10 0x7fa4b006cfd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa4d00ca440 tx=0x7fa4b8008040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:02.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.284+0000 7fa4ccff9700 1 -- 172.21.15.1:0/2984919123 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa4c0055860 con 0x7fa4bc003a30 2024-03-18T20:27:02.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.412+0000 7fa4ccff9700 1 -- 172.21.15.1:0/2984919123 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 155) v1 ==== 89285+0+0 (secure 0 0 0) 0x7fa4c0055c30 con 0x7fa4bc003a30 2024-03-18T20:27:02.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.540+0000 7fa4d66df700 1 -- 172.21.15.1:0/2984919123 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa4d0068900 con 0x7fa4bc003a30 2024-03-18T20:27:02.545 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-18T20:27:02.541+0000 7fbdbeb46700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T20:27:02.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.552+0000 7fa4ccff9700 1 -- 172.21.15.1:0/2984919123 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v127) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa4c00592f0 con 0x7fa4bc003a30 2024-03-18T20:27:02.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.554+0000 7fa4d66df700 1 -- 172.21.15.1:0/2984919123 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa4b006ab10 msgr2=0x7fa4b006cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:02.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.554+0000 7fa4d66df700 1 --2- 172.21.15.1:0/2984919123 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa4b006ab10 0x7fa4b006cfd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa4d00ca440 tx=0x7fa4b8008040 comp rx=0 tx=0).stop 2024-03-18T20:27:02.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.554+0000 7fa4d66df700 1 -- 172.21.15.1:0/2984919123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4bc003a30 msgr2=0x7fa4d00c2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:02.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.554+0000 7fa4d66df700 1 --2- 172.21.15.1:0/2984919123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4bc003a30 0x7fa4d00c2b00 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00011c0 tx=0x7fa4c0001a00 comp rx=0 tx=0).stop 2024-03-18T20:27:02.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.554+0000 7fa4d66df700 1 -- 172.21.15.1:0/2984919123 shutdown_connections 2024-03-18T20:27:02.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.554+0000 7fa4d66df700 1 --2- 172.21.15.1:0/2984919123 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa4b006ab10 0x7fa4b006cfd0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:02.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.554+0000 7fa4d66df700 1 --2- 172.21.15.1:0/2984919123 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa4d01f6a40 0x7fa4d00ce830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:02.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.554+0000 7fa4d66df700 1 --2- 172.21.15.1:0/2984919123 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4bc003a30 0x7fa4d00c2b00 unknown :-1 s=CLOSED pgs=2967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:02.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.554+0000 7fa4d66df700 1 --2- 172.21.15.1:0/2984919123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4d00b5720 0x7fa4d01f6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:02.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.554+0000 7fa4d66df700 1 -- 172.21.15.1:0/2984919123 >> 172.21.15.1:0/2984919123 conn(0x7fa4d00cff30 msgr2=0x7fa4d006ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:02.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.555+0000 7fa4d66df700 1 -- 172.21.15.1:0/2984919123 shutdown_connections 2024-03-18T20:27:02.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.555+0000 7fa4d66df700 1 -- 172.21.15.1:0/2984919123 wait complete. 2024-03-18T20:27:02.560 INFO:tasks.ceph.mds.d.smithi165.stderr: -3366> 2024-03-18T20:26:31.338+0000 7f776e480700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T20:27:02.560 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T20:27:02.571 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T20:27:02.588 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-03-18T20:27:02.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.694+0000 7f941e9ba700 1 Processor -- start 2024-03-18T20:27:02.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.694+0000 7f941e9ba700 1 -- start start 2024-03-18T20:27:02.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.695+0000 7f941e9ba700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f941815a1d0 0x7f94180add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:02.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.695+0000 7f941e9ba700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94180ce8f0 con 0x7f9418071620 2024-03-18T20:27:02.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.695+0000 7f941e9ba700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94180ae260 con 0x7f94180cc0f0 2024-03-18T20:27:02.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.695+0000 7f941e9ba700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94180cdb00 con 0x7f941815a1d0 2024-03-18T20:27:02.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.695+0000 7f94177fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9418071620 0x7f94180cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45096/0 (socket says 172.21.15.1:45096) 2024-03-18T20:27:02.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.695+0000 7f94177fe700 1 -- 172.21.15.1:0/3498292570 learned_addr learned my addr 172.21.15.1:0/3498292570 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:02.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.696+0000 7f9416ffd700 1 -- 172.21.15.1:0/3498292570 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327652473 0 0) 0x7f94180ce8f0 con 0x7f9418071620 2024-03-18T20:27:02.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.696+0000 7f9416ffd700 1 -- 172.21.15.1:0/3498292570 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94040035b0 con 0x7f9418071620 2024-03-18T20:27:02.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.696+0000 7f9417fff700 1 --2- 172.21.15.1:0/3498292570 >> v2:172.21.15.165:3300/0 conn(0x7f941815a1d0 0x7f94180add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:02.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.696+0000 7f9416ffd700 1 -- 172.21.15.1:0/3498292570 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4155714333 0 0) 0x7f94040035b0 con 0x7f9418071620 2024-03-18T20:27:02.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.696+0000 7f9416ffd700 1 -- 172.21.15.1:0/3498292570 >> v2:172.21.15.165:3300/0 conn(0x7f941815a1d0 msgr2=0x7f94180add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:02.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.696+0000 7f9416ffd700 1 --2- 172.21.15.1:0/3498292570 >> v2:172.21.15.165:3300/0 conn(0x7f941815a1d0 0x7f94180add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:02.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.696+0000 7f9416ffd700 1 -- 172.21.15.1:0/3498292570 >> v1:172.21.15.165:6789/0 conn(0x7f94180cc0f0 legacy=0x7f941815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:02.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.696+0000 7f9416ffd700 1 -- 172.21.15.1:0/3498292570 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94180cdce0 con 0x7f9418071620 2024-03-18T20:27:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.696+0000 7f9416ffd700 1 -- 172.21.15.1:0/3498292570 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f9400001670 con 0x7f9418071620 2024-03-18T20:27:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.696+0000 7f9416ffd700 1 -- 172.21.15.1:0/3498292570 >> v1:172.21.15.1:6789/0 conn(0x7f9418071620 legacy=0x7f94180cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.696+0000 7f9416ffd700 1 --2- 172.21.15.1:0/3498292570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9404003a00 0x7f9404003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.696+0000 7f9416ffd700 1 --2- 172.21.15.1:0/3498292570 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9404004850 0x7f9404004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.696+0000 7f9416ffd700 1 --2- 172.21.15.1:0/3498292570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9404005210 0x7f940400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.696+0000 7f9416ffd700 1 -- 172.21.15.1:0/3498292570 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94180cdb00 con 0x7f9404005210 2024-03-18T20:27:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.696+0000 7f9416ffd700 1 -- 172.21.15.1:0/3498292570 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940400c0b0 con 0x7f9404003a00 2024-03-18T20:27:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.696+0000 7f9416ffd700 1 -- 172.21.15.1:0/3498292570 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f940400c270 con 0x7f9404004850 2024-03-18T20:27:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.696+0000 7f941cf57700 1 --2- 172.21.15.1:0/3498292570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9404005210 0x7f940400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.696+0000 7f9417fff700 1 --2- 172.21.15.1:0/3498292570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9404003a00 0x7f9404003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.696+0000 7f94177fe700 1 --2- 172.21.15.1:0/3498292570 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9404004850 0x7f9404004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.697+0000 7f941cf57700 1 -- 172.21.15.1:0/3498292570 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9404004850 msgr2=0x7f9404004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:02.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.697+0000 7f941cf57700 1 --2- 172.21.15.1:0/3498292570 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9404004850 0x7f9404004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.697+0000 7f941cf57700 1 -- 172.21.15.1:0/3498292570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9404003a00 msgr2=0x7f9404003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.697+0000 7f941cf57700 1 --2- 172.21.15.1:0/3498292570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9404003a00 0x7f9404003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.697+0000 7f941cf57700 1 -- 172.21.15.1:0/3498292570 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94040035b0 con 0x7f9404005210 2024-03-18T20:27:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.697+0000 7f941cf57700 1 -- 172.21.15.1:0/3498292570 reap_dead start 2024-03-18T20:27:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.697+0000 7f941cf57700 1 --2- 172.21.15.1:0/3498292570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9404005210 0x7f940400ba10 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 crypto rx=0x7f941000c160 tx=0x7f9404005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84c1718d5d39486 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.697+0000 7f9416ffd700 1 -- 172.21.15.1:0/3498292570 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9410001db0 con 0x7f9404005210 2024-03-18T20:27:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.697+0000 7f9416ffd700 1 -- 172.21.15.1:0/3498292570 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f941000ccc0 con 0x7f9404005210 2024-03-18T20:27:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.697+0000 7f9416ffd700 1 -- 172.21.15.1:0/3498292570 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9410003c40 con 0x7f9404005210 2024-03-18T20:27:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.697+0000 7f941e9ba700 1 -- 172.21.15.1:0/3498292570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9404005210 msgr2=0x7f940400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.697+0000 7f941e9ba700 1 --2- 172.21.15.1:0/3498292570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9404005210 0x7f940400ba10 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 crypto rx=0x7f941000c160 tx=0x7f9404005880 comp rx=0 tx=0).stop 2024-03-18T20:27:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.697+0000 7f941e9ba700 1 -- 172.21.15.1:0/3498292570 shutdown_connections 2024-03-18T20:27:02.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.697+0000 7f941e9ba700 1 --2- 172.21.15.1:0/3498292570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9404005210 0x7f940400ba10 unknown :-1 s=CLOSED pgs=2972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:02.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.698+0000 7f941e9ba700 1 -- 172.21.15.1:0/3498292570 >> 172.21.15.1:0/3498292570 conn(0x7f94180cc9b0 msgr2=0x7f94180ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:02.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.698+0000 7f941e9ba700 1 -- 172.21.15.1:0/3498292570 shutdown_connections 2024-03-18T20:27:02.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.698+0000 7f941e9ba700 1 -- 172.21.15.1:0/3498292570 wait complete. 2024-03-18T20:27:02.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.699+0000 7f941e9ba700 1 Processor -- start 2024-03-18T20:27:02.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.699+0000 7f941e9ba700 1 -- start start 2024-03-18T20:27:02.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f941e9ba700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9404005210 0x7f94181ee780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f941e9ba700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94181571e0 0x7f94180c8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f941e9ba700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9418143f90 0x7f9418144440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f941e9ba700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94181eecc0 con 0x7f9418143f90 2024-03-18T20:27:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f941e9ba700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94180c8aa0 con 0x7f94181571e0 2024-03-18T20:27:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f941e9ba700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f941815d140 con 0x7f9404005210 2024-03-18T20:27:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f941cf57700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9418143f90 0x7f9418144440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f941cf57700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9418143f90 0x7f9418144440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35120/0 (socket says 172.21.15.1:35120) 2024-03-18T20:27:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f941cf57700 1 -- 172.21.15.1:0/3145890769 learned_addr learned my addr 172.21.15.1:0/3145890769 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f94177fe700 1 --2- 172.21.15.1:0/3145890769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94181571e0 0x7f94180c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f9417fff700 1 --2- 172.21.15.1:0/3145890769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9404005210 0x7f94181ee780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f941cf57700 1 -- 172.21.15.1:0/3145890769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9404005210 msgr2=0x7f94181ee780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f941cf57700 1 --2- 172.21.15.1:0/3145890769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9404005210 0x7f94181ee780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:02.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f941cf57700 1 -- 172.21.15.1:0/3145890769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94181571e0 msgr2=0x7f94180c8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:02.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f941cf57700 1 --2- 172.21.15.1:0/3145890769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94181571e0 0x7f94180c8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:02.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f941cf57700 1 -- 172.21.15.1:0/3145890769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941815d420 con 0x7f9418143f90 2024-03-18T20:27:02.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f941cf57700 1 --2- 172.21.15.1:0/3145890769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9418143f90 0x7f9418144440 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 crypto rx=0x7f9418072a20 tx=0x7f94100014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:02.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f9414ff9700 1 -- 172.21.15.1:0/3145890769 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9410003e10 con 0x7f9418143f90 2024-03-18T20:27:02.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f9414ff9700 1 -- 172.21.15.1:0/3145890769 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9410005070 con 0x7f9418143f90 2024-03-18T20:27:02.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f9414ff9700 1 -- 172.21.15.1:0/3145890769 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9410005400 con 0x7f9418143f90 2024-03-18T20:27:02.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f941e9ba700 1 -- 172.21.15.1:0/3145890769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94180c7a80 con 0x7f9418143f90 2024-03-18T20:27:02.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.700+0000 7f941e9ba700 1 -- 172.21.15.1:0/3145890769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94180c8040 con 0x7f9418143f90 2024-03-18T20:27:02.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.701+0000 7f941e9ba700 1 -- 172.21.15.1:0/3145890769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94180676a0 con 0x7f9418143f90 2024-03-18T20:27:02.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.702+0000 7f9414ff9700 1 -- 172.21.15.1:0/3145890769 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 155) v1 ==== 89285+0+0 (secure 0 0 0) 0x7f9410005560 con 0x7f9418143f90 2024-03-18T20:27:02.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.702+0000 7f9414ff9700 1 --2- 172.21.15.1:0/3145890769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93f806ae40 0x7f93f806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:02.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.702+0000 7f9414ff9700 1 -- 172.21.15.1:0/3145890769 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 7853+0+0 (secure 0 0 0) 0x7f941008b100 con 0x7f9418143f90 2024-03-18T20:27:02.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.703+0000 7f9417fff700 1 --2- 172.21.15.1:0/3145890769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93f806ae40 0x7f93f806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:02.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.703+0000 7f9417fff700 1 --2- 172.21.15.1:0/3145890769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93f806ae40 0x7f93f806d300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9408009bc0 tx=0x7f94080091c0 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:02.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.707+0000 7f9414ff9700 1 -- 172.21.15.1:0/3145890769 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9410055890 con 0x7f9418143f90 2024-03-18T20:27:02.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:02.959+0000 7f941e9ba700 1 -- 172.21.15.1:0/3145890769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f94180cf940 con 0x7f9418143f90 2024-03-18T20:27:03.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.554+0000 7f9414ff9700 1 -- 172.21.15.1:0/3145890769 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v128) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9410016020 con 0x7f9418143f90 2024-03-18T20:27:03.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.557+0000 7f941e9ba700 1 -- 172.21.15.1:0/3145890769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93f806ae40 msgr2=0x7f93f806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:03.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.557+0000 7f941e9ba700 1 --2- 172.21.15.1:0/3145890769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93f806ae40 0x7f93f806d300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9408009bc0 tx=0x7f94080091c0 comp rx=0 tx=0).stop 2024-03-18T20:27:03.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.557+0000 7f941e9ba700 1 -- 172.21.15.1:0/3145890769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9418143f90 msgr2=0x7f9418144440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:03.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.557+0000 7f941e9ba700 1 --2- 172.21.15.1:0/3145890769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9418143f90 0x7f9418144440 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 crypto rx=0x7f9418072a20 tx=0x7f94100014a0 comp rx=0 tx=0).stop 2024-03-18T20:27:03.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.557+0000 7f941e9ba700 1 -- 172.21.15.1:0/3145890769 shutdown_connections 2024-03-18T20:27:03.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.557+0000 7f941e9ba700 1 --2- 172.21.15.1:0/3145890769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93f806ae40 0x7f93f806d300 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:03.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.557+0000 7f941e9ba700 1 --2- 172.21.15.1:0/3145890769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9404005210 0x7f94181ee780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:03.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.557+0000 7f941e9ba700 1 --2- 172.21.15.1:0/3145890769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9418143f90 0x7f9418144440 unknown :-1 s=CLOSED pgs=2973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:03.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.557+0000 7f941e9ba700 1 --2- 172.21.15.1:0/3145890769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94181571e0 0x7f94180c8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:03.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.557+0000 7f941e9ba700 1 -- 172.21.15.1:0/3145890769 >> 172.21.15.1:0/3145890769 conn(0x7f94180cc9b0 msgr2=0x7f9418157e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:03.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.557+0000 7f941e9ba700 1 -- 172.21.15.1:0/3145890769 shutdown_connections 2024-03-18T20:27:03.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.558+0000 7f941e9ba700 1 -- 172.21.15.1:0/3145890769 wait complete. 2024-03-18T20:27:03.577 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T20:27:03.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.711+0000 7fcea5f7f700 1 Processor -- start 2024-03-18T20:27:03.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.711+0000 7fcea5f7f700 1 -- start start 2024-03-18T20:27:03.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.712+0000 7fcea5f7f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcea0071e50 0x7fcea015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:03.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.712+0000 7fcea5f7f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea015b230 con 0x7fcea0157f10 2024-03-18T20:27:03.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.712+0000 7fcea5f7f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea015b510 con 0x7fcea015a780 2024-03-18T20:27:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.712+0000 7fcea5f7f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea006ecf0 con 0x7fcea0071e50 2024-03-18T20:27:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.712+0000 7fce9f7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcea0157f10 0x7fcea006f4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45112/0 (socket says 172.21.15.1:45112) 2024-03-18T20:27:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.712+0000 7fce9f7fe700 1 -- 172.21.15.1:0/1774618364 learned_addr learned my addr 172.21.15.1:0/1774618364 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.712+0000 7fce9ffff700 1 --2- 172.21.15.1:0/1774618364 >> v2:172.21.15.1:3300/0 conn(0x7fcea0071e50 0x7fcea015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.712+0000 7fce9e7fc700 1 -- 172.21.15.1:0/1774618364 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796963998 0 0) 0x7fcea015b230 con 0x7fcea0157f10 2024-03-18T20:27:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.712+0000 7fce9e7fc700 1 -- 172.21.15.1:0/1774618364 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce8c0035b0 con 0x7fcea0157f10 2024-03-18T20:27:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.712+0000 7fce9ffff700 1 -- 172.21.15.1:0/1774618364 >> v1:172.21.15.165:6789/0 conn(0x7fcea015a780 legacy=0x7fcea0071740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.712+0000 7fce9ffff700 1 -- 172.21.15.1:0/1774618364 >> v1:172.21.15.1:6789/0 conn(0x7fcea0157f10 legacy=0x7fcea006f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.712+0000 7fce9ffff700 1 -- 172.21.15.1:0/1774618364 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea006efd0 con 0x7fcea0071e50 2024-03-18T20:27:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.713+0000 7fce9ffff700 1 --2- 172.21.15.1:0/1774618364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea0071e50 0x7fcea015abe0 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 crypto rx=0x7fce98009d90 tx=0x7fce9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5a77531d36a7c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.713+0000 7fce9e7fc700 1 -- 172.21.15.1:0/1774618364 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9800e070 con 0x7fcea0071e50 2024-03-18T20:27:03.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.713+0000 7fce9e7fc700 1 -- 172.21.15.1:0/1774618364 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9800bb20 con 0x7fcea0071e50 2024-03-18T20:27:03.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.713+0000 7fce9e7fc700 1 -- 172.21.15.1:0/1774618364 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce98012430 con 0x7fcea0071e50 2024-03-18T20:27:03.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.713+0000 7fcea5f7f700 1 -- 172.21.15.1:0/1774618364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea0071e50 msgr2=0x7fcea015abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:03.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.713+0000 7fcea5f7f700 1 --2- 172.21.15.1:0/1774618364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea0071e50 0x7fcea015abe0 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 crypto rx=0x7fce98009d90 tx=0x7fce9800b330 comp rx=0 tx=0).stop 2024-03-18T20:27:03.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.713+0000 7fcea5f7f700 1 -- 172.21.15.1:0/1774618364 shutdown_connections 2024-03-18T20:27:03.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.713+0000 7fcea5f7f700 1 --2- 172.21.15.1:0/1774618364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea0071e50 0x7fcea015abe0 unknown :-1 s=CLOSED pgs=2975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:03.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.713+0000 7fcea5f7f700 1 -- 172.21.15.1:0/1774618364 >> 172.21.15.1:0/1774618364 conn(0x7fcea00b1cb0 msgr2=0x7fcea00b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:03.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.713+0000 7fcea5f7f700 1 -- 172.21.15.1:0/1774618364 shutdown_connections 2024-03-18T20:27:03.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.713+0000 7fcea5f7f700 1 -- 172.21.15.1:0/1774618364 wait complete. 2024-03-18T20:27:03.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.714+0000 7fcea5f7f700 1 Processor -- start 2024-03-18T20:27:03.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.715+0000 7fcea5f7f700 1 -- start start 2024-03-18T20:27:03.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.715+0000 7fcea5f7f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea0071e50 0x7fcea014be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:03.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.715+0000 7fcea5f7f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcea0157f10 0x7fcea00cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:03.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.715+0000 7fcea5f7f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcea015a780 0x7fcea0159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.715+0000 7fcea5f7f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea006b420 con 0x7fcea0071e50 2024-03-18T20:27:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.715+0000 7fcea5f7f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea0158530 con 0x7fcea0157f10 2024-03-18T20:27:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.715+0000 7fcea5f7f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea0158810 con 0x7fcea015a780 2024-03-18T20:27:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.716+0000 7fce9f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea0071e50 0x7fcea014be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.716+0000 7fce9f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea0071e50 0x7fcea014be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35144/0 (socket says 172.21.15.1:35144) 2024-03-18T20:27:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.716+0000 7fce9f7fe700 1 -- 172.21.15.1:0/2512149615 learned_addr learned my addr 172.21.15.1:0/2512149615 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.716+0000 7fce9f7fe700 1 -- 172.21.15.1:0/2512149615 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcea015a780 msgr2=0x7fcea0159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.716+0000 7fce9f7fe700 1 --2- 172.21.15.1:0/2512149615 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcea015a780 0x7fcea0159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.716+0000 7fce9f7fe700 1 -- 172.21.15.1:0/2512149615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcea0157f10 msgr2=0x7fcea00cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.716+0000 7fce9effd700 1 --2- 172.21.15.1:0/2512149615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcea0157f10 0x7fcea00cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.716+0000 7fce9f7fe700 1 --2- 172.21.15.1:0/2512149615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcea0157f10 0x7fcea00cb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.716+0000 7fce9f7fe700 1 -- 172.21.15.1:0/2512149615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce98009a70 con 0x7fcea0071e50 2024-03-18T20:27:03.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.716+0000 7fce9f7fe700 1 --2- 172.21.15.1:0/2512149615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea0071e50 0x7fcea014be50 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7fce9000c700 tx=0x7fce9000cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.716+0000 7fce97fff700 1 -- 172.21.15.1:0/2512149615 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce90005d70 con 0x7fcea0071e50 2024-03-18T20:27:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.716+0000 7fce97fff700 1 -- 172.21.15.1:0/2512149615 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce900070d0 con 0x7fcea0071e50 2024-03-18T20:27:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.716+0000 7fcea5f7f700 1 -- 172.21.15.1:0/2512149615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea0158af0 con 0x7fcea0071e50 2024-03-18T20:27:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.716+0000 7fce97fff700 1 -- 172.21.15.1:0/2512149615 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce90007460 con 0x7fcea0071e50 2024-03-18T20:27:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.716+0000 7fcea5f7f700 1 -- 172.21.15.1:0/2512149615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea0159550 con 0x7fcea0071e50 2024-03-18T20:27:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.717+0000 7fcea5f7f700 1 -- 172.21.15.1:0/2512149615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea0158390 con 0x7fcea0071e50 2024-03-18T20:27:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.718+0000 7fce97fff700 1 -- 172.21.15.1:0/2512149615 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 155) v1 ==== 89285+0+0 (secure 0 0 0) 0x7fce900075c0 con 0x7fcea0071e50 2024-03-18T20:27:03.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.718+0000 7fce97fff700 1 --2- 172.21.15.1:0/2512149615 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fce8006ae40 0x7fce8006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:03.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.718+0000 7fce97fff700 1 -- 172.21.15.1:0/2512149615 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 7853+0+0 (secure 0 0 0) 0x7fce900894c0 con 0x7fcea0071e50 2024-03-18T20:27:03.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.718+0000 7fce9effd700 1 --2- 172.21.15.1:0/2512149615 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fce8006ae40 0x7fce8006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:03.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.719+0000 7fce9effd700 1 --2- 172.21.15.1:0/2512149615 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fce8006ae40 0x7fce8006d300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcea014c570 tx=0x7fce88009040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:03.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.723+0000 7fce97fff700 1 -- 172.21.15.1:0/2512149615 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcea0158390 con 0x7fcea0071e50 2024-03-18T20:27:03.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:03.926+0000 7fcea5f7f700 1 -- 172.21.15.1:0/2512149615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcea0158390 con 0x7fcea0071e50 2024-03-18T20:27:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.569+0000 7fce97fff700 1 -- 172.21.15.1:0/2512149615 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fce90053fa0 con 0x7fcea0071e50 2024-03-18T20:27:04.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.570+0000 7fce97fff700 1 -- 172.21.15.1:0/2512149615 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v261) v1 ==== 181+0+0 (secure 0 0 0) 0x7fce90082510 con 0x7fcea0071e50 2024-03-18T20:27:04.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.573+0000 7fcea5f7f700 1 -- 172.21.15.1:0/2512149615 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fce8006ae40 msgr2=0x7fce8006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:04.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.573+0000 7fcea5f7f700 1 --2- 172.21.15.1:0/2512149615 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fce8006ae40 0x7fce8006d300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcea014c570 tx=0x7fce88009040 comp rx=0 tx=0).stop 2024-03-18T20:27:04.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.573+0000 7fcea5f7f700 1 -- 172.21.15.1:0/2512149615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea0071e50 msgr2=0x7fcea014be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.573+0000 7fcea5f7f700 1 --2- 172.21.15.1:0/2512149615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea0071e50 0x7fcea014be50 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7fce9000c700 tx=0x7fce9000cac0 comp rx=0 tx=0).stop 2024-03-18T20:27:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.573+0000 7fcea5f7f700 1 -- 172.21.15.1:0/2512149615 shutdown_connections 2024-03-18T20:27:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.573+0000 7fcea5f7f700 1 --2- 172.21.15.1:0/2512149615 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fce8006ae40 0x7fce8006d300 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.573+0000 7fcea5f7f700 1 --2- 172.21.15.1:0/2512149615 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcea015a780 0x7fcea0159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.573+0000 7fcea5f7f700 1 --2- 172.21.15.1:0/2512149615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcea0071e50 0x7fcea014be50 unknown :-1 s=CLOSED pgs=2976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.573+0000 7fcea5f7f700 1 --2- 172.21.15.1:0/2512149615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcea0157f10 0x7fcea00cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.573+0000 7fcea5f7f700 1 -- 172.21.15.1:0/2512149615 >> 172.21.15.1:0/2512149615 conn(0x7fcea00b1cb0 msgr2=0x7fcea015e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.574+0000 7fcea5f7f700 1 -- 172.21.15.1:0/2512149615 shutdown_connections 2024-03-18T20:27:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.574+0000 7fcea5f7f700 1 -- 172.21.15.1:0/2512149615 wait complete. 2024-03-18T20:27:04.579 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-18T20:27:04.593 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T20:27:04.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.726+0000 7f65c126a700 1 Processor -- start 2024-03-18T20:27:04.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.726+0000 7f65c126a700 1 -- start start 2024-03-18T20:27:04.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.727+0000 7f65c126a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f65bc0d0040 0x7f65bc0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:04.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.727+0000 7f65c126a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f65bc0d1170 0x7f65bc0cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:04.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.727+0000 7f65c126a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65bc15dc90 con 0x7f65bc15d8b0 2024-03-18T20:27:04.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.727+0000 7f65c126a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65bc15ec90 con 0x7f65bc0d1170 2024-03-18T20:27:04.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.727+0000 7f65c126a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65bc15ded0 con 0x7f65bc0d0040 2024-03-18T20:27:04.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.727+0000 7f65bb7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f65bc0d1170 0x7f65bc0cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:04.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.727+0000 7f65bb7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f65bc0d1170 0x7f65bc0cde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53928/0 (socket says 172.21.15.1:53928) 2024-03-18T20:27:04.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.727+0000 7f65bb7fe700 1 -- 172.21.15.1:0/2740750756 learned_addr learned my addr 172.21.15.1:0/2740750756 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.727+0000 7f65bb7fe700 1 -- 172.21.15.1:0/2740750756 >> v2:172.21.15.165:3300/0 conn(0x7f65bc0d0040 msgr2=0x7f65bc0d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.727+0000 7f65bb7fe700 1 --2- 172.21.15.1:0/2740750756 >> v2:172.21.15.165:3300/0 conn(0x7f65bc0d0040 0x7f65bc0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.727+0000 7f65bb7fe700 1 -- 172.21.15.1:0/2740750756 >> v1:172.21.15.165:6789/0 conn(0x7f65bc15d8b0 legacy=0x7f65bc0cf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.727+0000 7f65bb7fe700 1 -- 172.21.15.1:0/2740750756 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65bc15e1b0 con 0x7f65bc0d1170 2024-03-18T20:27:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.728+0000 7f65bb7fe700 1 --2- 172.21.15.1:0/2740750756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65bc0d1170 0x7f65bc0cde10 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7f65bc08d870 tx=0x7f65b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b813dd966024db03 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.728+0000 7f65b9ffb700 1 -- 172.21.15.1:0/2740750756 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b400e070 con 0x7f65bc0d1170 2024-03-18T20:27:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.728+0000 7f65b9ffb700 1 -- 172.21.15.1:0/2740750756 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b4003a20 con 0x7f65bc0d1170 2024-03-18T20:27:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.728+0000 7f65b9ffb700 1 -- 172.21.15.1:0/2740750756 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4003dd0 con 0x7f65bc0d1170 2024-03-18T20:27:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.728+0000 7f65c126a700 1 -- 172.21.15.1:0/2740750756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65bc0d1170 msgr2=0x7f65bc0cde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.728+0000 7f65c126a700 1 --2- 172.21.15.1:0/2740750756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65bc0d1170 0x7f65bc0cde10 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7f65bc08d870 tx=0x7f65b400b330 comp rx=0 tx=0).stop 2024-03-18T20:27:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.728+0000 7f65c126a700 1 -- 172.21.15.1:0/2740750756 shutdown_connections 2024-03-18T20:27:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.728+0000 7f65c126a700 1 --2- 172.21.15.1:0/2740750756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65bc0d1170 0x7f65bc0cde10 unknown :-1 s=CLOSED pgs=2977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:04.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.728+0000 7f65c126a700 1 --2- 172.21.15.1:0/2740750756 >> v2:172.21.15.165:3300/0 conn(0x7f65bc0d0040 0x7f65bc0d3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.728+0000 7f65c126a700 1 -- 172.21.15.1:0/2740750756 >> 172.21.15.1:0/2740750756 conn(0x7f65bc09c640 msgr2=0x7f65bc0d3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.728+0000 7f65c126a700 1 -- 172.21.15.1:0/2740750756 shutdown_connections 2024-03-18T20:27:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.728+0000 7f65c126a700 1 -- 172.21.15.1:0/2740750756 wait complete. 2024-03-18T20:27:04.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.730+0000 7f65c126a700 1 Processor -- start 2024-03-18T20:27:04.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.730+0000 7f65c126a700 1 -- start start 2024-03-18T20:27:04.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.730+0000 7f65c126a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65bc0d0040 0x7f65bc1f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:04.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.730+0000 7f65c126a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65bc0d1170 0x7f65bc0c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:04.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.730+0000 7f65c126a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65bc15d8b0 0x7f65bc0c83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:04.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.730+0000 7f65c126a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc0b6160 con 0x7f65bc0d0040 2024-03-18T20:27:04.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.730+0000 7f65c126a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc0cd570 con 0x7f65bc0d1170 2024-03-18T20:27:04.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.730+0000 7f65c126a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc0b6370 con 0x7f65bc15d8b0 2024-03-18T20:27:04.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.730+0000 7f65baffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65bc0d0040 0x7f65bc1f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:04.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.730+0000 7f65baffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65bc0d0040 0x7f65bc1f3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53938/0 (socket says 172.21.15.1:53938) 2024-03-18T20:27:04.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.730+0000 7f65baffd700 1 -- 172.21.15.1:0/1571963088 learned_addr learned my addr 172.21.15.1:0/1571963088 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:04.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.731+0000 7f65ba7fc700 1 --2- 172.21.15.1:0/1571963088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65bc0d1170 0x7f65bc0c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:04.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.731+0000 7f65bb7fe700 1 --2- 172.21.15.1:0/1571963088 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65bc15d8b0 0x7f65bc0c83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:04.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.731+0000 7f65baffd700 1 -- 172.21.15.1:0/1571963088 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65bc15d8b0 msgr2=0x7f65bc0c83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:04.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.731+0000 7f65baffd700 1 --2- 172.21.15.1:0/1571963088 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65bc15d8b0 0x7f65bc0c83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:04.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.731+0000 7f65baffd700 1 -- 172.21.15.1:0/1571963088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65bc0d1170 msgr2=0x7f65bc0c7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:04.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.731+0000 7f65baffd700 1 --2- 172.21.15.1:0/1571963088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65bc0d1170 0x7f65bc0c7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:04.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.731+0000 7f65baffd700 1 -- 172.21.15.1:0/1571963088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b4009b30 con 0x7f65bc0d0040 2024-03-18T20:27:04.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.731+0000 7f65baffd700 1 --2- 172.21.15.1:0/1571963088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65bc0d0040 0x7f65bc1f3140 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 crypto rx=0x7f65b000b470 tx=0x7f65b000b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:04.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.731+0000 7f65a37fe700 1 -- 172.21.15.1:0/1571963088 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b0011830 con 0x7f65bc0d0040 2024-03-18T20:27:04.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.731+0000 7f65a37fe700 1 -- 172.21.15.1:0/1571963088 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b0011990 con 0x7f65bc0d0040 2024-03-18T20:27:04.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.731+0000 7f65c126a700 1 -- 172.21.15.1:0/1571963088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65bc0b6650 con 0x7f65bc0d0040 2024-03-18T20:27:04.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.731+0000 7f65a37fe700 1 -- 172.21.15.1:0/1571963088 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b0011d40 con 0x7f65bc0d0040 2024-03-18T20:27:04.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.731+0000 7f65c126a700 1 -- 172.21.15.1:0/1571963088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65bc1f3680 con 0x7f65bc0d0040 2024-03-18T20:27:04.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.732+0000 7f65c126a700 1 -- 172.21.15.1:0/1571963088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65bc073350 con 0x7f65bc0d0040 2024-03-18T20:27:04.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.733+0000 7f65a37fe700 1 -- 172.21.15.1:0/1571963088 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f65b001a400 con 0x7f65bc0d0040 2024-03-18T20:27:04.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.733+0000 7f65a37fe700 1 --2- 172.21.15.1:0/1571963088 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f65a806aed0 0x7f65a806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:04.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.733+0000 7f65a37fe700 1 -- 172.21.15.1:0/1571963088 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 7415+0+0 (secure 0 0 0) 0x7f65b0089ec0 con 0x7f65bc0d0040 2024-03-18T20:27:04.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.733+0000 7f65ba7fc700 1 --2- 172.21.15.1:0/1571963088 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f65a806aed0 0x7f65a806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:04.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.734+0000 7f65ba7fc700 1 --2- 172.21.15.1:0/1571963088 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f65a806aed0 0x7f65a806d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f65ac006fd0 tx=0x7f65ac008040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:04.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.738+0000 7f65a37fe700 1 -- 172.21.15.1:0/1571963088 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f65b0054600 con 0x7f65bc0d0040 2024-03-18T20:27:04.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:04.941+0000 7f65c126a700 1 -- 172.21.15.1:0/1571963088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f65bc0d3a10 con 0x7f65bc0d0040 2024-03-18T20:27:05.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.566+0000 7f65a37fe700 1 -- 172.21.15.1:0/1571963088 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v262) v1 ==== 169+0+0 (secure 0 0 0) 0x7f65b0015070 con 0x7f65bc0d0040 2024-03-18T20:27:05.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.570+0000 7f65c126a700 1 -- 172.21.15.1:0/1571963088 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f65a806aed0 msgr2=0x7f65a806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:05.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.570+0000 7f65c126a700 1 --2- 172.21.15.1:0/1571963088 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f65a806aed0 0x7f65a806d390 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f65ac006fd0 tx=0x7f65ac008040 comp rx=0 tx=0).stop 2024-03-18T20:27:05.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.570+0000 7f65c126a700 1 -- 172.21.15.1:0/1571963088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65bc0d0040 msgr2=0x7f65bc1f3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:05.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.570+0000 7f65c126a700 1 --2- 172.21.15.1:0/1571963088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65bc0d0040 0x7f65bc1f3140 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 crypto rx=0x7f65b000b470 tx=0x7f65b000b830 comp rx=0 tx=0).stop 2024-03-18T20:27:05.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.571+0000 7f65c126a700 1 -- 172.21.15.1:0/1571963088 shutdown_connections 2024-03-18T20:27:05.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.571+0000 7f65c126a700 1 --2- 172.21.15.1:0/1571963088 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f65a806aed0 0x7f65a806d390 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:05.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.571+0000 7f65c126a700 1 --2- 172.21.15.1:0/1571963088 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f65bc15d8b0 0x7f65bc0c83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:05.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.571+0000 7f65c126a700 1 --2- 172.21.15.1:0/1571963088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65bc0d0040 0x7f65bc1f3140 unknown :-1 s=CLOSED pgs=2978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:05.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.571+0000 7f65c126a700 1 --2- 172.21.15.1:0/1571963088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65bc0d1170 0x7f65bc0c7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:05.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.571+0000 7f65c126a700 1 -- 172.21.15.1:0/1571963088 >> 172.21.15.1:0/1571963088 conn(0x7f65bc09c640 msgr2=0x7f65bc06c890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:05.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.571+0000 7f65c126a700 1 -- 172.21.15.1:0/1571963088 shutdown_connections 2024-03-18T20:27:05.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.571+0000 7f65c126a700 1 -- 172.21.15.1:0/1571963088 wait complete. 2024-03-18T20:27:05.578 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-18T20:27:05.593 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:27:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.720+0000 7f1df1b5d700 1 Processor -- start 2024-03-18T20:27:05.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.720+0000 7f1df1b5d700 1 -- start start 2024-03-18T20:27:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.721+0000 7f1df1b5d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1dec144b70 0x7f1dec0ce980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.721+0000 7f1df1b5d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec0af610 0x7f1dec0cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.721+0000 7f1df1b5d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1dec0ccf40 0x7f1dec15ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.721+0000 7f1df1b5d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dec0cd3d0 con 0x7f1dec0ccf40 2024-03-18T20:27:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.721+0000 7f1df1b5d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dec144f50 con 0x7f1dec144b70 2024-03-18T20:27:05.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.721+0000 7f1df1b5d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec0cf2d0 con 0x7f1dec0af610 2024-03-18T20:27:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.721+0000 7f1debfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1dec0ccf40 0x7f1dec15ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.721+0000 7f1debfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1dec0ccf40 0x7f1dec15ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53952/0 (socket says 172.21.15.1:53952) 2024-03-18T20:27:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.721+0000 7f1debfff700 1 -- 172.21.15.1:0/3268282031 learned_addr learned my addr 172.21.15.1:0/3268282031 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.721+0000 7f1debfff700 1 -- 172.21.15.1:0/3268282031 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec0af610 msgr2=0x7f1dec0cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.721+0000 7f1deb7fe700 1 --2- 172.21.15.1:0/3268282031 >> v2:172.21.15.165:3300/0 conn(0x7f1dec144b70 0x7f1dec0ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.721+0000 7f1deaffd700 1 --2- 172.21.15.1:0/3268282031 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec0af610 0x7f1dec0cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.721+0000 7f1debfff700 1 --2- 172.21.15.1:0/3268282031 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec0af610 0x7f1dec0cca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.721+0000 7f1debfff700 1 -- 172.21.15.1:0/3268282031 >> v2:172.21.15.165:3300/0 conn(0x7f1dec144b70 msgr2=0x7f1dec0ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.721+0000 7f1debfff700 1 --2- 172.21.15.1:0/3268282031 >> v2:172.21.15.165:3300/0 conn(0x7f1dec144b70 0x7f1dec0ce980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.721+0000 7f1debfff700 1 -- 172.21.15.1:0/3268282031 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dec0cf5b0 con 0x7f1dec0ccf40 2024-03-18T20:27:05.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.721+0000 7f1deaffd700 1 --2- 172.21.15.1:0/3268282031 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec0af610 0x7f1dec0cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.721+0000 7f1deb7fe700 1 --2- 172.21.15.1:0/3268282031 >> v2:172.21.15.165:3300/0 conn(0x7f1dec144b70 0x7f1dec0ce980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.722+0000 7f1debfff700 1 --2- 172.21.15.1:0/3268282031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec0ccf40 0x7f1dec15ecf0 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7f1dec06d080 tx=0x7f1de400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d57e2b62c6af9f19 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.722+0000 7f1dea7fc700 1 -- 172.21.15.1:0/3268282031 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de400e070 con 0x7f1dec0ccf40 2024-03-18T20:27:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.722+0000 7f1dea7fc700 1 -- 172.21.15.1:0/3268282031 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de400b9c0 con 0x7f1dec0ccf40 2024-03-18T20:27:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.722+0000 7f1dea7fc700 1 -- 172.21.15.1:0/3268282031 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de400bd70 con 0x7f1dec0ccf40 2024-03-18T20:27:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.722+0000 7f1df1b5d700 1 -- 172.21.15.1:0/3268282031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec0ccf40 msgr2=0x7f1dec15ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.722+0000 7f1df1b5d700 1 --2- 172.21.15.1:0/3268282031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec0ccf40 0x7f1dec15ecf0 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7f1dec06d080 tx=0x7f1de400b330 comp rx=0 tx=0).stop 2024-03-18T20:27:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.722+0000 7f1df1b5d700 1 -- 172.21.15.1:0/3268282031 shutdown_connections 2024-03-18T20:27:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.722+0000 7f1df1b5d700 1 --2- 172.21.15.1:0/3268282031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec0ccf40 0x7f1dec15ecf0 unknown :-1 s=CLOSED pgs=2979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.722+0000 7f1df1b5d700 1 --2- 172.21.15.1:0/3268282031 >> v2:172.21.15.165:3300/0 conn(0x7f1dec144b70 0x7f1dec0ce980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:05.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.722+0000 7f1df1b5d700 1 --2- 172.21.15.1:0/3268282031 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec0af610 0x7f1dec0cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.722+0000 7f1df1b5d700 1 -- 172.21.15.1:0/3268282031 >> 172.21.15.1:0/3268282031 conn(0x7f1dec0b1d20 msgr2=0x7f1dec0b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.722+0000 7f1df1b5d700 1 -- 172.21.15.1:0/3268282031 shutdown_connections 2024-03-18T20:27:05.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.722+0000 7f1df1b5d700 1 -- 172.21.15.1:0/3268282031 wait complete. 2024-03-18T20:27:05.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.724+0000 7f1df1b5d700 1 Processor -- start 2024-03-18T20:27:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.724+0000 7f1df1b5d700 1 -- start start 2024-03-18T20:27:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.725+0000 7f1df1b5d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dec0af610 0x7f1dec146950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.725+0000 7f1df1b5d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec0ccf40 0x7f1dec137860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.725+0000 7f1df1b5d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec144b70 0x7f1dec1e6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.725+0000 7f1df1b5d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec0b0c00 con 0x7f1dec144b70 2024-03-18T20:27:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.725+0000 7f1df1b5d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec146e90 con 0x7f1dec0af610 2024-03-18T20:27:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.725+0000 7f1df1b5d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec145bd0 con 0x7f1dec0ccf40 2024-03-18T20:27:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.725+0000 7f1debfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec144b70 0x7f1dec1e6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.725+0000 7f1debfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec144b70 0x7f1dec1e6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53966/0 (socket says 172.21.15.1:53966) 2024-03-18T20:27:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.725+0000 7f1debfff700 1 -- 172.21.15.1:0/3798373133 learned_addr learned my addr 172.21.15.1:0/3798373133 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.725+0000 7f1deaffd700 1 --2- 172.21.15.1:0/3798373133 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec0ccf40 0x7f1dec137860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.725+0000 7f1deb7fe700 1 --2- 172.21.15.1:0/3798373133 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dec0af610 0x7f1dec146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.725+0000 7f1debfff700 1 -- 172.21.15.1:0/3798373133 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec0ccf40 msgr2=0x7f1dec137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.725+0000 7f1debfff700 1 --2- 172.21.15.1:0/3798373133 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec0ccf40 0x7f1dec137860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:05.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.725+0000 7f1debfff700 1 -- 172.21.15.1:0/3798373133 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dec0af610 msgr2=0x7f1dec146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.725+0000 7f1debfff700 1 --2- 172.21.15.1:0/3798373133 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dec0af610 0x7f1dec146950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.725+0000 7f1debfff700 1 -- 172.21.15.1:0/3798373133 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de4009b30 con 0x7f1dec144b70 2024-03-18T20:27:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.725+0000 7f1deaffd700 1 --2- 172.21.15.1:0/3798373133 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec0ccf40 0x7f1dec137860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.725+0000 7f1deb7fe700 1 --2- 172.21.15.1:0/3798373133 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dec0af610 0x7f1dec146950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.726+0000 7f1debfff700 1 --2- 172.21.15.1:0/3798373133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec144b70 0x7f1dec1e6ba0 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7f1de400b370 tx=0x7f1de4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.726+0000 7f1de3fff700 1 -- 172.21.15.1:0/3798373133 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de40099b0 con 0x7f1dec144b70 2024-03-18T20:27:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.726+0000 7f1de3fff700 1 -- 172.21.15.1:0/3798373133 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de40092e0 con 0x7f1dec144b70 2024-03-18T20:27:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.726+0000 7f1df1b5d700 1 -- 172.21.15.1:0/3798373133 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dec06ecb0 con 0x7f1dec144b70 2024-03-18T20:27:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.726+0000 7f1de3fff700 1 -- 172.21.15.1:0/3798373133 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de401c640 con 0x7f1dec144b70 2024-03-18T20:27:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.726+0000 7f1df1b5d700 1 -- 172.21.15.1:0/3798373133 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dec146150 con 0x7f1dec144b70 2024-03-18T20:27:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.727+0000 7f1df1b5d700 1 -- 172.21.15.1:0/3798373133 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dec0cda90 con 0x7f1dec144b70 2024-03-18T20:27:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.727+0000 7f1de3fff700 1 -- 172.21.15.1:0/3798373133 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f1de401c7a0 con 0x7f1dec144b70 2024-03-18T20:27:05.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.727+0000 7f1de3fff700 1 --2- 172.21.15.1:0/3798373133 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1dcc06aed0 0x7f1dcc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.728+0000 7f1de3fff700 1 -- 172.21.15.1:0/3798373133 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 7001+0+0 (secure 0 0 0) 0x7f1de408b7c0 con 0x7f1dec144b70 2024-03-18T20:27:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.728+0000 7f1deb7fe700 1 --2- 172.21.15.1:0/3798373133 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1dcc06aed0 0x7f1dcc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:05.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.728+0000 7f1deb7fe700 1 --2- 172.21.15.1:0/3798373133 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1dcc06aed0 0x7f1dcc06d390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc006fd0 tx=0x7f1ddc008040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:05.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.732+0000 7f1de3fff700 1 -- 172.21.15.1:0/3798373133 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1de4056170 con 0x7f1dec144b70 2024-03-18T20:27:05.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.937+0000 7f1df1b5d700 1 -- 172.21.15.1:0/3798373133 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1dec0ce280 con 0x7f1dec144b70 2024-03-18T20:27:05.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.939+0000 7f1de3fff700 1 -- 172.21.15.1:0/3798373133 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v262) v1 ==== 74+0+17600 (secure 0 0 0) 0x7f1de4059790 con 0x7f1dec144b70 2024-03-18T20:27:05.943 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:05.943 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":262,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:27:05.562956+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:26:37.326244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:26:38.564628+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"257","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.165:6835/3886150679":"2024-03-19T20:27:02.541703+0000","172.21.15.165:6833/3886150679":"2024-03-19T20:27:02.541703+0000","172.21.15.1:0/1135489018":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/1721669330":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/245086145":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3664103058":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3761024882":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3251013155":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/1287719189":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/2340206734":"2024-03-19T20:26:52.380559+0000","172.21.15.1:6833/148356":"2024-03-19T20:26:52.380559+0000","172.21.15.1:6832/148356":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/376843885":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/1340983360":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/369631788":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/2840085801":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/322251557":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/1615265957":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3049322266":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/1886004750":"2024-03-19T20:26:52.380559+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:27:05.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.941+0000 7f1df1b5d700 1 -- 172.21.15.1:0/3798373133 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1dcc06aed0 msgr2=0x7f1dcc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:05.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.941+0000 7f1df1b5d700 1 --2- 172.21.15.1:0/3798373133 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1dcc06aed0 0x7f1dcc06d390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc006fd0 tx=0x7f1ddc008040 comp rx=0 tx=0).stop 2024-03-18T20:27:05.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.941+0000 7f1df1b5d700 1 -- 172.21.15.1:0/3798373133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec144b70 msgr2=0x7f1dec1e6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:05.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.941+0000 7f1df1b5d700 1 --2- 172.21.15.1:0/3798373133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec144b70 0x7f1dec1e6ba0 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7f1de400b370 tx=0x7f1de4004ad0 comp rx=0 tx=0).stop 2024-03-18T20:27:05.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.941+0000 7f1df1b5d700 1 -- 172.21.15.1:0/3798373133 shutdown_connections 2024-03-18T20:27:05.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.941+0000 7f1df1b5d700 1 --2- 172.21.15.1:0/3798373133 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1dcc06aed0 0x7f1dcc06d390 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:05.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.941+0000 7f1df1b5d700 1 --2- 172.21.15.1:0/3798373133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dec144b70 0x7f1dec1e6ba0 unknown :-1 s=CLOSED pgs=2980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:05.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.941+0000 7f1df1b5d700 1 --2- 172.21.15.1:0/3798373133 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1dec0af610 0x7f1dec146950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:05.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.941+0000 7f1df1b5d700 1 --2- 172.21.15.1:0/3798373133 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1dec0ccf40 0x7f1dec137860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:05.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.941+0000 7f1df1b5d700 1 -- 172.21.15.1:0/3798373133 >> 172.21.15.1:0/3798373133 conn(0x7f1dec0b1d20 msgr2=0x7f1dec06d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:05.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.941+0000 7f1df1b5d700 1 -- 172.21.15.1:0/3798373133 shutdown_connections 2024-03-18T20:27:05.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:05.941+0000 7f1df1b5d700 1 -- 172.21.15.1:0/3798373133 wait complete. 2024-03-18T20:27:05.958 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-03-18T20:26:37.326244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '256', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'backup_fs_data', 'create_time': '2024-03-18T20:26:38.564628+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '257', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:27:05.958 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:27:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.083+0000 7f5a29ed9700 1 Processor -- start 2024-03-18T20:27:06.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.083+0000 7f5a29ed9700 1 -- start start 2024-03-18T20:27:06.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.084+0000 7f5a29ed9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5a240cfa90 0x7f5a2406a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.084+0000 7f5a29ed9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a240cf600 con 0x7f5a2415d5a0 2024-03-18T20:27:06.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.084+0000 7f5a29ed9700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a240ce720 con 0x7f5a2406ac90 2024-03-18T20:27:06.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.084+0000 7f5a29ed9700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a2406b140 con 0x7f5a240cfa90 2024-03-18T20:27:06.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.084+0000 7f5a237fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5a2415d5a0 0x7f5a240ce010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49300/0 (socket says 172.21.15.1:49300) 2024-03-18T20:27:06.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.084+0000 7f5a237fe700 1 -- 172.21.15.1:0/1130632467 learned_addr learned my addr 172.21.15.1:0/1130632467 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:06.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.084+0000 7f5a22ffd700 1 --2- 172.21.15.1:0/1130632467 >> v2:172.21.15.165:3300/0 conn(0x7f5a240cfa90 0x7f5a2406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.084+0000 7f5a227fc700 1 -- 172.21.15.1:0/1130632467 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985350660 0 0) 0x7f5a240cf600 con 0x7f5a2415d5a0 2024-03-18T20:27:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.084+0000 7f5a227fc700 1 -- 172.21.15.1:0/1130632467 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a100035b0 con 0x7f5a2415d5a0 2024-03-18T20:27:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a227fc700 1 -- 172.21.15.1:0/1130632467 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1303204610 0 0) 0x7f5a100035b0 con 0x7f5a2415d5a0 2024-03-18T20:27:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a227fc700 1 -- 172.21.15.1:0/1130632467 >> v2:172.21.15.165:3300/0 conn(0x7f5a240cfa90 msgr2=0x7f5a2406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a227fc700 1 --2- 172.21.15.1:0/1130632467 >> v2:172.21.15.165:3300/0 conn(0x7f5a240cfa90 0x7f5a2406a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a227fc700 1 -- 172.21.15.1:0/1130632467 >> v1:172.21.15.165:6789/0 conn(0x7f5a2406ac90 legacy=0x7f5a24167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a227fc700 1 -- 172.21.15.1:0/1130632467 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a240ce9f0 con 0x7f5a2415d5a0 2024-03-18T20:27:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a227fc700 1 -- 172.21.15.1:0/1130632467 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f5a0c0017f0 con 0x7f5a2415d5a0 2024-03-18T20:27:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a227fc700 1 -- 172.21.15.1:0/1130632467 >> v1:172.21.15.1:6789/0 conn(0x7f5a2415d5a0 legacy=0x7f5a240ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a227fc700 1 --2- 172.21.15.1:0/1130632467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a10003a00 0x7f5a10003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a227fc700 1 --2- 172.21.15.1:0/1130632467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a10004400 0x7f5a100048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a227fc700 1 --2- 172.21.15.1:0/1130632467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a100050d0 0x7f5a1000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a227fc700 1 -- 172.21.15.1:0/1130632467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2406b140 con 0x7f5a100050d0 2024-03-18T20:27:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a227fc700 1 -- 172.21.15.1:0/1130632467 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1000c030 con 0x7f5a10004400 2024-03-18T20:27:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a227fc700 1 -- 172.21.15.1:0/1130632467 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1000c1f0 con 0x7f5a10003a00 2024-03-18T20:27:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a22ffd700 1 --2- 172.21.15.1:0/1130632467 >> v2:172.21.15.165:3300/0 conn(0x7f5a240cfa90 0x7f5a2406a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:27:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a23fff700 1 --2- 172.21.15.1:0/1130632467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a100050d0 0x7f5a1000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a237fe700 1 --2- 172.21.15.1:0/1130632467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a10003a00 0x7f5a10003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a22ffd700 1 --2- 172.21.15.1:0/1130632467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a10004400 0x7f5a100048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a23fff700 1 -- 172.21.15.1:0/1130632467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a10003a00 msgr2=0x7f5a10003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a23fff700 1 --2- 172.21.15.1:0/1130632467 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a10003a00 0x7f5a10003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a23fff700 1 -- 172.21.15.1:0/1130632467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a10004400 msgr2=0x7f5a100048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a23fff700 1 --2- 172.21.15.1:0/1130632467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a10004400 0x7f5a100048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a23fff700 1 -- 172.21.15.1:0/1130632467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a100035b0 con 0x7f5a100050d0 2024-03-18T20:27:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a22ffd700 1 --2- 172.21.15.1:0/1130632467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a10004400 0x7f5a100048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a23fff700 1 -- 172.21.15.1:0/1130632467 reap_dead start 2024-03-18T20:27:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a23fff700 1 --2- 172.21.15.1:0/1130632467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a100050d0 0x7f5a1000ba10 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c00c290 tx=0x7f5a10005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=759e7f1c1addd2fc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a227fc700 1 -- 172.21.15.1:0/1130632467 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1c001e70 con 0x7f5a100050d0 2024-03-18T20:27:06.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a227fc700 1 -- 172.21.15.1:0/1130632467 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1c00ce70 con 0x7f5a100050d0 2024-03-18T20:27:06.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.085+0000 7f5a227fc700 1 -- 172.21.15.1:0/1130632467 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1c003c20 con 0x7f5a100050d0 2024-03-18T20:27:06.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.086+0000 7f5a29ed9700 1 -- 172.21.15.1:0/1130632467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a100050d0 msgr2=0x7f5a1000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:06.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.086+0000 7f5a29ed9700 1 --2- 172.21.15.1:0/1130632467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a100050d0 0x7f5a1000ba10 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c00c290 tx=0x7f5a10005880 comp rx=0 tx=0).stop 2024-03-18T20:27:06.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.086+0000 7f5a29ed9700 1 -- 172.21.15.1:0/1130632467 shutdown_connections 2024-03-18T20:27:06.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.086+0000 7f5a29ed9700 1 --2- 172.21.15.1:0/1130632467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a100050d0 0x7f5a1000ba10 unknown :-1 s=CLOSED pgs=2982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.086+0000 7f5a29ed9700 1 -- 172.21.15.1:0/1130632467 >> 172.21.15.1:0/1130632467 conn(0x7f5a240cc9b0 msgr2=0x7f5a240ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:06.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.086+0000 7f5a29ed9700 1 -- 172.21.15.1:0/1130632467 shutdown_connections 2024-03-18T20:27:06.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.086+0000 7f5a29ed9700 1 -- 172.21.15.1:0/1130632467 wait complete. 2024-03-18T20:27:06.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.087+0000 7f5a29ed9700 1 Processor -- start 2024-03-18T20:27:06.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.087+0000 7f5a29ed9700 1 -- start start 2024-03-18T20:27:06.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.088+0000 7f5a29ed9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a100050d0 0x7f5a2415acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.088+0000 7f5a29ed9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a2415a600 0x7f5a2415e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.088+0000 7f5a29ed9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a241f6590 0x7f5a241f6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.088+0000 7f5a29ed9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a240b5890 con 0x7f5a100050d0 2024-03-18T20:27:06.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.088+0000 7f5a29ed9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a240b52a0 con 0x7f5a2415a600 2024-03-18T20:27:06.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.088+0000 7f5a29ed9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a240b2ba0 con 0x7f5a241f6590 2024-03-18T20:27:06.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.088+0000 7f5a237fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a100050d0 0x7f5a2415acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.088+0000 7f5a237fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a100050d0 0x7f5a2415acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53994/0 (socket says 172.21.15.1:53994) 2024-03-18T20:27:06.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.088+0000 7f5a237fe700 1 -- 172.21.15.1:0/2625219349 learned_addr learned my addr 172.21.15.1:0/2625219349 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:06.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.088+0000 7f5a22ffd700 1 --2- 172.21.15.1:0/2625219349 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a2415a600 0x7f5a2415e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.088+0000 7f5a23fff700 1 --2- 172.21.15.1:0/2625219349 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a241f6590 0x7f5a241f6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.089+0000 7f5a237fe700 1 -- 172.21.15.1:0/2625219349 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a241f6590 msgr2=0x7f5a241f6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:06.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.089+0000 7f5a237fe700 1 --2- 172.21.15.1:0/2625219349 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a241f6590 0x7f5a241f6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.089+0000 7f5a237fe700 1 -- 172.21.15.1:0/2625219349 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a2415a600 msgr2=0x7f5a2415e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:06.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.089+0000 7f5a237fe700 1 --2- 172.21.15.1:0/2625219349 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a2415a600 0x7f5a2415e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.089+0000 7f5a237fe700 1 -- 172.21.15.1:0/2625219349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a240b30f0 con 0x7f5a100050d0 2024-03-18T20:27:06.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.089+0000 7f5a237fe700 1 --2- 172.21.15.1:0/2625219349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a100050d0 0x7f5a2415acc0 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c007920 tx=0x7f5a0c00fc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:06.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.089+0000 7f5a1bfff700 1 -- 172.21.15.1:0/2625219349 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c00fe80 con 0x7f5a100050d0 2024-03-18T20:27:06.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.089+0000 7f5a1bfff700 1 -- 172.21.15.1:0/2625219349 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0c0072e0 con 0x7f5a100050d0 2024-03-18T20:27:06.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.089+0000 7f5a29ed9700 1 -- 172.21.15.1:0/2625219349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a2415aa80 con 0x7f5a100050d0 2024-03-18T20:27:06.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.089+0000 7f5a1bfff700 1 -- 172.21.15.1:0/2625219349 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c007690 con 0x7f5a100050d0 2024-03-18T20:27:06.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.089+0000 7f5a29ed9700 1 -- 172.21.15.1:0/2625219349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a240b54b0 con 0x7f5a100050d0 2024-03-18T20:27:06.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.090+0000 7f5a29ed9700 1 -- 172.21.15.1:0/2625219349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a240b59d0 con 0x7f5a100050d0 2024-03-18T20:27:06.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.090+0000 7f5a1bfff700 1 -- 172.21.15.1:0/2625219349 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f5a0c005830 con 0x7f5a100050d0 2024-03-18T20:27:06.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.091+0000 7f5a1bfff700 1 --2- 172.21.15.1:0/2625219349 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5a0406aed0 0x7f5a0406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.091+0000 7f5a1bfff700 1 -- 172.21.15.1:0/2625219349 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 7001+0+0 (secure 0 0 0) 0x7f5a0c089bd0 con 0x7f5a100050d0 2024-03-18T20:27:06.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.091+0000 7f5a22ffd700 1 --2- 172.21.15.1:0/2625219349 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5a0406aed0 0x7f5a0406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.091+0000 7f5a22ffd700 1 --2- 172.21.15.1:0/2625219349 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5a0406aed0 0x7f5a0406d390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5a240cb290 tx=0x7f5a14008040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:06.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.095+0000 7f5a1bfff700 1 -- 172.21.15.1:0/2625219349 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5a240b59d0 con 0x7f5a100050d0 2024-03-18T20:27:06.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.296+0000 7f5a29ed9700 1 -- 172.21.15.1:0/2625219349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5a240b59d0 con 0x7f5a100050d0 2024-03-18T20:27:06.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.297+0000 7f5a1bfff700 1 -- 172.21.15.1:0/2625219349 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v262) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5a240b59d0 con 0x7f5a100050d0 2024-03-18T20:27:06.301 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:27:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.299+0000 7f5a29ed9700 1 -- 172.21.15.1:0/2625219349 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5a0406aed0 msgr2=0x7f5a0406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.299+0000 7f5a29ed9700 1 --2- 172.21.15.1:0/2625219349 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5a0406aed0 0x7f5a0406d390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5a240cb290 tx=0x7f5a14008040 comp rx=0 tx=0).stop 2024-03-18T20:27:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.299+0000 7f5a29ed9700 1 -- 172.21.15.1:0/2625219349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a100050d0 msgr2=0x7f5a2415acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.299+0000 7f5a29ed9700 1 --2- 172.21.15.1:0/2625219349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a100050d0 0x7f5a2415acc0 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c007920 tx=0x7f5a0c00fc40 comp rx=0 tx=0).stop 2024-03-18T20:27:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.299+0000 7f5a29ed9700 1 -- 172.21.15.1:0/2625219349 shutdown_connections 2024-03-18T20:27:06.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.299+0000 7f5a29ed9700 1 --2- 172.21.15.1:0/2625219349 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5a0406aed0 0x7f5a0406d390 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.299+0000 7f5a29ed9700 1 --2- 172.21.15.1:0/2625219349 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a241f6590 0x7f5a241f6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.299+0000 7f5a29ed9700 1 --2- 172.21.15.1:0/2625219349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a100050d0 0x7f5a2415acc0 unknown :-1 s=CLOSED pgs=2983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.299+0000 7f5a29ed9700 1 --2- 172.21.15.1:0/2625219349 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a2415a600 0x7f5a2415e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.299+0000 7f5a29ed9700 1 -- 172.21.15.1:0/2625219349 >> 172.21.15.1:0/2625219349 conn(0x7f5a240cc9b0 msgr2=0x7f5a2406e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.299+0000 7f5a29ed9700 1 -- 172.21.15.1:0/2625219349 shutdown_connections 2024-03-18T20:27:06.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.299+0000 7f5a29ed9700 1 -- 172.21.15.1:0/2625219349 wait complete. 2024-03-18T20:27:06.315 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-03-18T20:27:06.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.439+0000 7f7a90c95700 1 Processor -- start 2024-03-18T20:27:06.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.439+0000 7f7a90c95700 1 -- start start 2024-03-18T20:27:06.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.440+0000 7f7a90c95700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7a8c15d780 0x7f7a8c0cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.440+0000 7f7a90c95700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7a8c0ce5a0 0x7f7a8c0ce9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.440+0000 7f7a90c95700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8c0ce090 con 0x7f7a8c09dd40 2024-03-18T20:27:06.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.440+0000 7f7a90c95700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a8c0cb480 con 0x7f7a8c15d780 2024-03-18T20:27:06.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.440+0000 7f7a90c95700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a8c06db30 con 0x7f7a8c0ce5a0 2024-03-18T20:27:06.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.440+0000 7f7a8a59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7a8c15d780 0x7f7a8c0cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.440+0000 7f7a8a59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7a8c15d780 0x7f7a8c0cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54006/0 (socket says 172.21.15.1:54006) 2024-03-18T20:27:06.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.440+0000 7f7a8a59c700 1 -- 172.21.15.1:0/2184809728 learned_addr learned my addr 172.21.15.1:0/2184809728 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:06.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.440+0000 7f7a8ad9d700 1 --2- 172.21.15.1:0/2184809728 >> v2:172.21.15.165:3300/0 conn(0x7f7a8c0ce5a0 0x7f7a8c0ce9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.441+0000 7f7a8a59c700 1 -- 172.21.15.1:0/2184809728 >> v2:172.21.15.165:3300/0 conn(0x7f7a8c0ce5a0 msgr2=0x7f7a8c0ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.441+0000 7f7a8a59c700 1 --2- 172.21.15.1:0/2184809728 >> v2:172.21.15.165:3300/0 conn(0x7f7a8c0ce5a0 0x7f7a8c0ce9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.441+0000 7f7a8a59c700 1 -- 172.21.15.1:0/2184809728 >> v1:172.21.15.165:6789/0 conn(0x7f7a8c09dd40 legacy=0x7f7a8c0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.441+0000 7f7a8a59c700 1 -- 172.21.15.1:0/2184809728 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8c06d520 con 0x7f7a8c15d780 2024-03-18T20:27:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.441+0000 7f7a8a59c700 1 --2- 172.21.15.1:0/2184809728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a8c15d780 0x7f7a8c0cdb50 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c15ed20 tx=0x7f7a80009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e17501300f164d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.441+0000 7f7a8959a700 1 -- 172.21.15.1:0/2184809728 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80004230 con 0x7f7a8c15d780 2024-03-18T20:27:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.441+0000 7f7a8959a700 1 -- 172.21.15.1:0/2184809728 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a80004390 con 0x7f7a8c15d780 2024-03-18T20:27:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.441+0000 7f7a8959a700 1 -- 172.21.15.1:0/2184809728 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a800036a0 con 0x7f7a8c15d780 2024-03-18T20:27:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.441+0000 7f7a90c95700 1 -- 172.21.15.1:0/2184809728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a8c15d780 msgr2=0x7f7a8c0cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.441+0000 7f7a90c95700 1 --2- 172.21.15.1:0/2184809728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a8c15d780 0x7f7a8c0cdb50 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c15ed20 tx=0x7f7a80009bd0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.441+0000 7f7a90c95700 1 -- 172.21.15.1:0/2184809728 shutdown_connections 2024-03-18T20:27:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.441+0000 7f7a90c95700 1 --2- 172.21.15.1:0/2184809728 >> v2:172.21.15.165:3300/0 conn(0x7f7a8c0ce5a0 0x7f7a8c0ce9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.441+0000 7f7a90c95700 1 --2- 172.21.15.1:0/2184809728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a8c15d780 0x7f7a8c0cdb50 unknown :-1 s=CLOSED pgs=2984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.441+0000 7f7a90c95700 1 -- 172.21.15.1:0/2184809728 >> 172.21.15.1:0/2184809728 conn(0x7f7a8c0cca00 msgr2=0x7f7a8c0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.442+0000 7f7a90c95700 1 -- 172.21.15.1:0/2184809728 shutdown_connections 2024-03-18T20:27:06.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.442+0000 7f7a90c95700 1 -- 172.21.15.1:0/2184809728 wait complete. 2024-03-18T20:27:06.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.443+0000 7f7a90c95700 1 Processor -- start 2024-03-18T20:27:06.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.443+0000 7f7a90c95700 1 -- start start 2024-03-18T20:27:06.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a90c95700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a8c09dd40 0x7f7a8c1517e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a90c95700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a8c0ce5a0 0x7f7a8c1555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a90c95700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a8c15d780 0x7f7a8c155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a90c95700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c0c5940 con 0x7f7a8c0ce5a0 2024-03-18T20:27:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a90c95700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c0c54b0 con 0x7f7a8c09dd40 2024-03-18T20:27:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a90c95700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c0c48a0 con 0x7f7a8c15d780 2024-03-18T20:27:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a89d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a8c0ce5a0 0x7f7a8c1555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a89d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a8c0ce5a0 0x7f7a8c1555c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54014/0 (socket says 172.21.15.1:54014) 2024-03-18T20:27:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a89d9b700 1 -- 172.21.15.1:0/367317743 learned_addr learned my addr 172.21.15.1:0/367317743 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a8a59c700 1 --2- 172.21.15.1:0/367317743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a8c09dd40 0x7f7a8c1517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a8ad9d700 1 --2- 172.21.15.1:0/367317743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a8c15d780 0x7f7a8c155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a89d9b700 1 -- 172.21.15.1:0/367317743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a8c15d780 msgr2=0x7f7a8c155b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a89d9b700 1 --2- 172.21.15.1:0/367317743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a8c15d780 0x7f7a8c155b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a89d9b700 1 -- 172.21.15.1:0/367317743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a8c09dd40 msgr2=0x7f7a8c1517e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a89d9b700 1 --2- 172.21.15.1:0/367317743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a8c09dd40 0x7f7a8c1517e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a89d9b700 1 -- 172.21.15.1:0/367317743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a80009550 con 0x7f7a8c0ce5a0 2024-03-18T20:27:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a8ad9d700 1 --2- 172.21.15.1:0/367317743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a8c15d780 0x7f7a8c155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a89d9b700 1 --2- 172.21.15.1:0/367317743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a8c0ce5a0 0x7f7a8c1555c0 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00c4d0 tx=0x7f7a7c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a7affd700 1 -- 172.21.15.1:0/367317743 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c011790 con 0x7f7a8c0ce5a0 2024-03-18T20:27:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a7affd700 1 -- 172.21.15.1:0/367317743 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7c0118f0 con 0x7f7a8c0ce5a0 2024-03-18T20:27:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a90c95700 1 -- 172.21.15.1:0/367317743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8c0c4b80 con 0x7f7a8c0ce5a0 2024-03-18T20:27:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a7affd700 1 -- 172.21.15.1:0/367317743 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c011ca0 con 0x7f7a8c0ce5a0 2024-03-18T20:27:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.444+0000 7f7a90c95700 1 -- 172.21.15.1:0/367317743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8c0c5140 con 0x7f7a8c0ce5a0 2024-03-18T20:27:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.446+0000 7f7a90c95700 1 -- 172.21.15.1:0/367317743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a8c0c5620 con 0x7f7a8c0ce5a0 2024-03-18T20:27:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.446+0000 7f7a7affd700 1 -- 172.21.15.1:0/367317743 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f7a7c01c460 con 0x7f7a8c0ce5a0 2024-03-18T20:27:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.446+0000 7f7a7affd700 1 --2- 172.21.15.1:0/367317743 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7a6c06aed0 0x7f7a6c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.446+0000 7f7a7affd700 1 -- 172.21.15.1:0/367317743 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 7001+0+0 (secure 0 0 0) 0x7f7a7c0899c0 con 0x7f7a8c0ce5a0 2024-03-18T20:27:06.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.447+0000 7f7a8a59c700 1 --2- 172.21.15.1:0/367317743 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7a6c06aed0 0x7f7a6c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.447+0000 7f7a8a59c700 1 --2- 172.21.15.1:0/367317743 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7a6c06aed0 0x7f7a6c06d390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7a80009130 tx=0x7f7a80008030 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:06.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.451+0000 7f7a7affd700 1 -- 172.21.15.1:0/367317743 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7a7c054400 con 0x7f7a8c0ce5a0 2024-03-18T20:27:06.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.664+0000 7f7a90c95700 1 -- 172.21.15.1:0/367317743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7a8c0676a0 con 0x7f7a8c0ce5a0 2024-03-18T20:27:06.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.665+0000 7f7a7affd700 1 -- 172.21.15.1:0/367317743 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v262) v1 ==== 107+0+11 (secure 0 0 0) 0x7f7a7c057a20 con 0x7f7a8c0ce5a0 2024-03-18T20:27:06.669 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:27:06.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.667+0000 7f7a90c95700 1 -- 172.21.15.1:0/367317743 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7a6c06aed0 msgr2=0x7f7a6c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:06.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.667+0000 7f7a90c95700 1 --2- 172.21.15.1:0/367317743 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7a6c06aed0 0x7f7a6c06d390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7a80009130 tx=0x7f7a80008030 comp rx=0 tx=0).stop 2024-03-18T20:27:06.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.667+0000 7f7a90c95700 1 -- 172.21.15.1:0/367317743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a8c0ce5a0 msgr2=0x7f7a8c1555c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:06.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.667+0000 7f7a90c95700 1 --2- 172.21.15.1:0/367317743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a8c0ce5a0 0x7f7a8c1555c0 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00c4d0 tx=0x7f7a7c00c890 comp rx=0 tx=0).stop 2024-03-18T20:27:06.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.667+0000 7f7a90c95700 1 -- 172.21.15.1:0/367317743 shutdown_connections 2024-03-18T20:27:06.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.667+0000 7f7a90c95700 1 --2- 172.21.15.1:0/367317743 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7a6c06aed0 0x7f7a6c06d390 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.667+0000 7f7a90c95700 1 --2- 172.21.15.1:0/367317743 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7a8c15d780 0x7f7a8c155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.667+0000 7f7a90c95700 1 --2- 172.21.15.1:0/367317743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a8c09dd40 0x7f7a8c1517e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.667+0000 7f7a90c95700 1 --2- 172.21.15.1:0/367317743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a8c0ce5a0 0x7f7a8c1555c0 unknown :-1 s=CLOSED pgs=2985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.667+0000 7f7a90c95700 1 -- 172.21.15.1:0/367317743 >> 172.21.15.1:0/367317743 conn(0x7f7a8c0cca00 msgr2=0x7f7a8c06a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:06.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.667+0000 7f7a90c95700 1 -- 172.21.15.1:0/367317743 shutdown_connections 2024-03-18T20:27:06.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.667+0000 7f7a90c95700 1 -- 172.21.15.1:0/367317743 wait complete. 2024-03-18T20:27:06.683 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-03-18T20:27:06.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.807+0000 7f1caba2c700 1 Processor -- start 2024-03-18T20:27:06.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.808+0000 7f1caba2c700 1 -- start start 2024-03-18T20:27:06.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.808+0000 7f1caba2c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1ca415eba0 0x7f1ca40cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.808+0000 7f1caba2c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ca40b0e20 0x7f1ca40b12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.808+0000 7f1caba2c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca415ef80 con 0x7f1ca40b0500 2024-03-18T20:27:06.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.808+0000 7f1caba2c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ca40cc9a0 con 0x7f1ca415eba0 2024-03-18T20:27:06.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.808+0000 7f1caba2c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca40ccc80 con 0x7f1ca40b0e20 2024-03-18T20:27:06.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.809+0000 7f1ca97c8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1ca415eba0 0x7f1ca40cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.809+0000 7f1ca8fc7700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1ca40b0500 0x7f1ca40cf8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49306/0 (socket says 172.21.15.1:49306) 2024-03-18T20:27:06.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.809+0000 7f1ca8fc7700 1 -- 172.21.15.1:0/147229509 learned_addr learned my addr 172.21.15.1:0/147229509 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:06.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.809+0000 7f1ca9fc9700 1 --2- 172.21.15.1:0/147229509 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ca40b0e20 0x7f1ca40b12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.809+0000 7f1ca97c8700 1 -- 172.21.15.1:0/147229509 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ca40b0e20 msgr2=0x7f1ca40b12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:06.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.809+0000 7f1ca97c8700 1 --2- 172.21.15.1:0/147229509 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ca40b0e20 0x7f1ca40b12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.809+0000 7f1ca97c8700 1 -- 172.21.15.1:0/147229509 >> v1:172.21.15.1:6789/0 conn(0x7f1ca40b0500 legacy=0x7f1ca40cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:06.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.809+0000 7f1ca97c8700 1 -- 172.21.15.1:0/147229509 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca40ccf60 con 0x7f1ca415eba0 2024-03-18T20:27:06.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.809+0000 7f1c97fff700 1 -- 172.21.15.1:0/147229509 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098584187 0 0) 0x7f1ca415ef80 con 0x7f1ca40b0500 2024-03-18T20:27:06.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.809+0000 7f1ca9fc9700 1 --2- 172.21.15.1:0/147229509 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ca40b0e20 0x7f1ca40b12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:06.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.809+0000 7f1ca97c8700 1 --2- 172.21.15.1:0/147229509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ca415eba0 0x7f1ca40cf370 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7f1ca0009840 tx=0x7f1ca0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16e8012d57d81af server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:06.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.809+0000 7f1c97fff700 1 -- 172.21.15.1:0/147229509 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca0011830 con 0x7f1ca415eba0 2024-03-18T20:27:06.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.809+0000 7f1c97fff700 1 -- 172.21.15.1:0/147229509 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca0011990 con 0x7f1ca415eba0 2024-03-18T20:27:06.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.809+0000 7f1c97fff700 1 -- 172.21.15.1:0/147229509 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca0011d20 con 0x7f1ca415eba0 2024-03-18T20:27:06.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.810+0000 7f1caba2c700 1 -- 172.21.15.1:0/147229509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ca415eba0 msgr2=0x7f1ca40cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:06.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.810+0000 7f1caba2c700 1 --2- 172.21.15.1:0/147229509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ca415eba0 0x7f1ca40cf370 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7f1ca0009840 tx=0x7f1ca0009b50 comp rx=0 tx=0).stop 2024-03-18T20:27:06.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.810+0000 7f1caba2c700 1 -- 172.21.15.1:0/147229509 shutdown_connections 2024-03-18T20:27:06.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.810+0000 7f1caba2c700 1 --2- 172.21.15.1:0/147229509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ca415eba0 0x7f1ca40cf370 unknown :-1 s=CLOSED pgs=2987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.810+0000 7f1caba2c700 1 --2- 172.21.15.1:0/147229509 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ca40b0e20 0x7f1ca40b12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.810+0000 7f1caba2c700 1 -- 172.21.15.1:0/147229509 >> 172.21.15.1:0/147229509 conn(0x7f1ca40b2bd0 msgr2=0x7f1ca40b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:06.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.810+0000 7f1caba2c700 1 -- 172.21.15.1:0/147229509 shutdown_connections 2024-03-18T20:27:06.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.810+0000 7f1caba2c700 1 -- 172.21.15.1:0/147229509 wait complete. 2024-03-18T20:27:06.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.811+0000 7f1caba2c700 1 Processor -- start 2024-03-18T20:27:06.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.811+0000 7f1caba2c700 1 -- start start 2024-03-18T20:27:06.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.812+0000 7f1caba2c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ca40b0500 0x7f1ca41f4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.812+0000 7f1caba2c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ca40b0e20 0x7f1ca40c53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.812+0000 7f1caba2c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ca415eba0 0x7f1ca4151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.812+0000 7f1caba2c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca40c5900 con 0x7f1ca40b0e20 2024-03-18T20:27:06.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.812+0000 7f1caba2c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca40c5b10 con 0x7f1ca415eba0 2024-03-18T20:27:06.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.812+0000 7f1caba2c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca40c5df0 con 0x7f1ca40b0500 2024-03-18T20:27:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.812+0000 7f1ca8fc7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ca40b0e20 0x7f1ca40c53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.812+0000 7f1ca8fc7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ca40b0e20 0x7f1ca40c53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54018/0 (socket says 172.21.15.1:54018) 2024-03-18T20:27:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.812+0000 7f1ca8fc7700 1 -- 172.21.15.1:0/1732020594 learned_addr learned my addr 172.21.15.1:0/1732020594 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.812+0000 7f1ca97c8700 1 --2- 172.21.15.1:0/1732020594 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ca40b0500 0x7f1ca41f4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.812+0000 7f1ca9fc9700 1 --2- 172.21.15.1:0/1732020594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ca415eba0 0x7f1ca4151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.812+0000 7f1ca8fc7700 1 -- 172.21.15.1:0/1732020594 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ca40b0500 msgr2=0x7f1ca41f4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.812+0000 7f1ca8fc7700 1 --2- 172.21.15.1:0/1732020594 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ca40b0500 0x7f1ca41f4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.812+0000 7f1ca8fc7700 1 -- 172.21.15.1:0/1732020594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ca415eba0 msgr2=0x7f1ca4151130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.812+0000 7f1ca8fc7700 1 --2- 172.21.15.1:0/1732020594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ca415eba0 0x7f1ca4151130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.812+0000 7f1ca8fc7700 1 -- 172.21.15.1:0/1732020594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca00094a0 con 0x7f1ca40b0e20 2024-03-18T20:27:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.813+0000 7f1ca8fc7700 1 --2- 172.21.15.1:0/1732020594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ca40b0e20 0x7f1ca40c53c0 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7f1c980011e0 tx=0x7f1c9800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.813+0000 7f1c95ffb700 1 -- 172.21.15.1:0/1732020594 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c98004c80 con 0x7f1ca40b0e20 2024-03-18T20:27:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.813+0000 7f1c95ffb700 1 -- 172.21.15.1:0/1732020594 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c98004de0 con 0x7f1ca40b0e20 2024-03-18T20:27:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.813+0000 7f1caba2c700 1 -- 172.21.15.1:0/1732020594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca40c60d0 con 0x7f1ca40b0e20 2024-03-18T20:27:06.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.813+0000 7f1c95ffb700 1 -- 172.21.15.1:0/1732020594 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c98007310 con 0x7f1ca40b0e20 2024-03-18T20:27:06.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.813+0000 7f1caba2c700 1 -- 172.21.15.1:0/1732020594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca4151670 con 0x7f1ca40b0e20 2024-03-18T20:27:06.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.814+0000 7f1caba2c700 1 -- 172.21.15.1:0/1732020594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca4156f90 con 0x7f1ca40b0e20 2024-03-18T20:27:06.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.814+0000 7f1c95ffb700 1 -- 172.21.15.1:0/1732020594 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f1c98007470 con 0x7f1ca40b0e20 2024-03-18T20:27:06.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.815+0000 7f1c95ffb700 1 --2- 172.21.15.1:0/1732020594 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1c8806aed0 0x7f1c8806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:06.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.815+0000 7f1c95ffb700 1 -- 172.21.15.1:0/1732020594 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 7001+0+0 (secure 0 0 0) 0x7f1c98089960 con 0x7f1ca40b0e20 2024-03-18T20:27:06.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.815+0000 7f1ca97c8700 1 --2- 172.21.15.1:0/1732020594 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1c8806aed0 0x7f1c8806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:06.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.815+0000 7f1ca97c8700 1 --2- 172.21.15.1:0/1732020594 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1c8806aed0 0x7f1c8806d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1ca00090b0 tx=0x7f1ca001c040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:06.819+0000 7f1c95ffb700 1 -- 172.21.15.1:0/1732020594 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1ca4156f90 con 0x7f1ca40b0e20 2024-03-18T20:27:07.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.019+0000 7f1caba2c700 1 -- 172.21.15.1:0/1732020594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f1ca4156f90 con 0x7f1ca40b0e20 2024-03-18T20:27:07.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.020+0000 7f1c95ffb700 1 -- 172.21.15.1:0/1732020594 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v262) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1ca4156f90 con 0x7f1ca40b0e20 2024-03-18T20:27:07.024 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:27:07.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.022+0000 7f1caba2c700 1 -- 172.21.15.1:0/1732020594 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1c8806aed0 msgr2=0x7f1c8806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:07.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.022+0000 7f1caba2c700 1 --2- 172.21.15.1:0/1732020594 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1c8806aed0 0x7f1c8806d390 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1ca00090b0 tx=0x7f1ca001c040 comp rx=0 tx=0).stop 2024-03-18T20:27:07.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.022+0000 7f1caba2c700 1 -- 172.21.15.1:0/1732020594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ca40b0e20 msgr2=0x7f1ca40c53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:07.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.022+0000 7f1caba2c700 1 --2- 172.21.15.1:0/1732020594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ca40b0e20 0x7f1ca40c53c0 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7f1c980011e0 tx=0x7f1c9800ca30 comp rx=0 tx=0).stop 2024-03-18T20:27:07.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.022+0000 7f1caba2c700 1 -- 172.21.15.1:0/1732020594 shutdown_connections 2024-03-18T20:27:07.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.022+0000 7f1caba2c700 1 --2- 172.21.15.1:0/1732020594 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1c8806aed0 0x7f1c8806d390 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.022+0000 7f1caba2c700 1 --2- 172.21.15.1:0/1732020594 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1ca40b0500 0x7f1ca41f4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.022+0000 7f1caba2c700 1 --2- 172.21.15.1:0/1732020594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ca415eba0 0x7f1ca4151130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.022+0000 7f1caba2c700 1 --2- 172.21.15.1:0/1732020594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ca40b0e20 0x7f1ca40c53c0 unknown :-1 s=CLOSED pgs=2988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.022+0000 7f1caba2c700 1 -- 172.21.15.1:0/1732020594 >> 172.21.15.1:0/1732020594 conn(0x7f1ca40b2bd0 msgr2=0x7f1ca415ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:07.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.022+0000 7f1caba2c700 1 -- 172.21.15.1:0/1732020594 shutdown_connections 2024-03-18T20:27:07.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.022+0000 7f1caba2c700 1 -- 172.21.15.1:0/1732020594 wait complete. 2024-03-18T20:27:07.039 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:27:07.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.160+0000 7fd555804700 1 Processor -- start 2024-03-18T20:27:07.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.160+0000 7fd555804700 1 -- start start 2024-03-18T20:27:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.161+0000 7fd555804700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd5501545a0 0x7fd5500ae580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.161+0000 7fd555804700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5500acd80 0x7fd5500aeac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.161+0000 7fd555804700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5500ade70 con 0x7fd55015a5e0 2024-03-18T20:27:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.161+0000 7fd555804700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5500b2da0 con 0x7fd5501545a0 2024-03-18T20:27:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.161+0000 7fd555804700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd550154980 con 0x7fd5500acd80 2024-03-18T20:27:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.161+0000 7fd54effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd5501545a0 0x7fd5500ae580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.161+0000 7fd54effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd5501545a0 0x7fd5500ae580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54032/0 (socket says 172.21.15.1:54032) 2024-03-18T20:27:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.161+0000 7fd54effd700 1 -- 172.21.15.1:0/247475293 learned_addr learned my addr 172.21.15.1:0/247475293 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.162+0000 7fd54e7fc700 1 --2- 172.21.15.1:0/247475293 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5500acd80 0x7fd5500aeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.162+0000 7fd54effd700 1 -- 172.21.15.1:0/247475293 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5500acd80 msgr2=0x7fd5500aeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:07.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.162+0000 7fd54effd700 1 --2- 172.21.15.1:0/247475293 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5500acd80 0x7fd5500aeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.162+0000 7fd54effd700 1 -- 172.21.15.1:0/247475293 >> v1:172.21.15.165:6789/0 conn(0x7fd55015a5e0 legacy=0x7fd55015db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.162+0000 7fd54effd700 1 -- 172.21.15.1:0/247475293 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5500ad260 con 0x7fd5501545a0 2024-03-18T20:27:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.162+0000 7fd54e7fc700 1 --2- 172.21.15.1:0/247475293 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5500acd80 0x7fd5500aeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.162+0000 7fd54effd700 1 --2- 172.21.15.1:0/247475293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5501545a0 0x7fd5500ae580 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 crypto rx=0x7fd54000bb90 tx=0x7fd54000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe50961ac05b4f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.162+0000 7fd54dffb700 1 -- 172.21.15.1:0/247475293 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54000dd10 con 0x7fd5501545a0 2024-03-18T20:27:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.162+0000 7fd54dffb700 1 -- 172.21.15.1:0/247475293 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54000de70 con 0x7fd5501545a0 2024-03-18T20:27:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.162+0000 7fd54dffb700 1 -- 172.21.15.1:0/247475293 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd540004230 con 0x7fd5501545a0 2024-03-18T20:27:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.163+0000 7fd555804700 1 -- 172.21.15.1:0/247475293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5501545a0 msgr2=0x7fd5500ae580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.163+0000 7fd555804700 1 --2- 172.21.15.1:0/247475293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5501545a0 0x7fd5500ae580 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 crypto rx=0x7fd54000bb90 tx=0x7fd54000bea0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.163+0000 7fd555804700 1 -- 172.21.15.1:0/247475293 shutdown_connections 2024-03-18T20:27:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.163+0000 7fd555804700 1 --2- 172.21.15.1:0/247475293 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5501545a0 0x7fd5500ae580 unknown :-1 s=CLOSED pgs=2989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.163+0000 7fd555804700 1 --2- 172.21.15.1:0/247475293 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5500acd80 0x7fd5500aeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.163+0000 7fd555804700 1 -- 172.21.15.1:0/247475293 >> 172.21.15.1:0/247475293 conn(0x7fd55006b860 msgr2=0x7fd55006bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.163+0000 7fd555804700 1 -- 172.21.15.1:0/247475293 shutdown_connections 2024-03-18T20:27:07.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.163+0000 7fd555804700 1 -- 172.21.15.1:0/247475293 wait complete. 2024-03-18T20:27:07.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.164+0000 7fd555804700 1 Processor -- start 2024-03-18T20:27:07.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.164+0000 7fd555804700 1 -- start start 2024-03-18T20:27:07.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd555804700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5500acd80 0x7fd550151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:07.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd555804700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5501545a0 0x7fd55013dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:07.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd555804700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd55015a5e0 0x7fd5501ed720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:07.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd555804700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55013ea80 con 0x7fd5500acd80 2024-03-18T20:27:07.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd555804700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55013ed20 con 0x7fd55015a5e0 2024-03-18T20:27:07.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd555804700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd550152180 con 0x7fd5501545a0 2024-03-18T20:27:07.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd54effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5500acd80 0x7fd550151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:07.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd54effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5500acd80 0x7fd550151c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54044/0 (socket says 172.21.15.1:54044) 2024-03-18T20:27:07.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd54effd700 1 -- 172.21.15.1:0/2080789941 learned_addr learned my addr 172.21.15.1:0/2080789941 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:07.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd54e7fc700 1 --2- 172.21.15.1:0/2080789941 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5501545a0 0x7fd55013dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:07.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd54f7fe700 1 --2- 172.21.15.1:0/2080789941 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd55015a5e0 0x7fd5501ed720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:07.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd54effd700 1 -- 172.21.15.1:0/2080789941 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5501545a0 msgr2=0x7fd55013dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:07.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd54effd700 1 --2- 172.21.15.1:0/2080789941 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5501545a0 0x7fd55013dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd54effd700 1 -- 172.21.15.1:0/2080789941 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd55015a5e0 msgr2=0x7fd5501ed720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:07.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd54effd700 1 --2- 172.21.15.1:0/2080789941 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd55015a5e0 0x7fd5501ed720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd54effd700 1 -- 172.21.15.1:0/2080789941 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54000b7f0 con 0x7fd5500acd80 2024-03-18T20:27:07.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd54f7fe700 1 --2- 172.21.15.1:0/2080789941 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd55015a5e0 0x7fd5501ed720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:07.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd54e7fc700 1 --2- 172.21.15.1:0/2080789941 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5501545a0 0x7fd55013dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:07.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd54effd700 1 --2- 172.21.15.1:0/2080789941 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5500acd80 0x7fd550151c40 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7fd54000b340 tx=0x7fd540004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:07.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd547fff700 1 -- 172.21.15.1:0/2080789941 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd540020070 con 0x7fd5500acd80 2024-03-18T20:27:07.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd547fff700 1 -- 172.21.15.1:0/2080789941 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd540003c10 con 0x7fd5500acd80 2024-03-18T20:27:07.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd555804700 1 -- 172.21.15.1:0/2080789941 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd550152400 con 0x7fd5500acd80 2024-03-18T20:27:07.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.165+0000 7fd547fff700 1 -- 172.21.15.1:0/2080789941 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd540010470 con 0x7fd5500acd80 2024-03-18T20:27:07.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.166+0000 7fd555804700 1 -- 172.21.15.1:0/2080789941 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5501529c0 con 0x7fd5500acd80 2024-03-18T20:27:07.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.167+0000 7fd555804700 1 -- 172.21.15.1:0/2080789941 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd550066ff0 con 0x7fd5500acd80 2024-03-18T20:27:07.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.167+0000 7fd547fff700 1 -- 172.21.15.1:0/2080789941 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fd5400193f0 con 0x7fd5500acd80 2024-03-18T20:27:07.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.167+0000 7fd547fff700 1 --2- 172.21.15.1:0/2080789941 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd53006aed0 0x7fd53006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:07.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.167+0000 7fd547fff700 1 -- 172.21.15.1:0/2080789941 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 7001+0+0 (secure 0 0 0) 0x7fd54008d3d0 con 0x7fd5500acd80 2024-03-18T20:27:07.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.167+0000 7fd54e7fc700 1 --2- 172.21.15.1:0/2080789941 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd53006aed0 0x7fd53006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:07.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.168+0000 7fd54e7fc700 1 --2- 172.21.15.1:0/2080789941 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd53006aed0 0x7fd53006d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd550150070 tx=0x7fd538008040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:07.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.172+0000 7fd547fff700 1 -- 172.21.15.1:0/2080789941 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd540057c40 con 0x7fd5500acd80 2024-03-18T20:27:07.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.419+0000 7fd555804700 1 -- 172.21.15.1:0/2080789941 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd550074810 con 0x7fd5500acd80 2024-03-18T20:27:07.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.420+0000 7fd547fff700 1 -- 172.21.15.1:0/2080789941 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+4749 (secure 0 0 0) 0x7fd54005b260 con 0x7fd5500acd80 2024-03-18T20:27:07.424 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:07.424 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":128,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9851,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3262033681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":3262033681},{"type":"v1","addr":"172.21.15.165:6835","nonce":3262033681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":128},{"gid":12396,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2565889517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2565889517},{"type":"v1","addr":"172.21.15.1:6836","nonce":2565889517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":12399,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/512665659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":512665659},{"type":"v1","addr":"172.21.15.1:6837","nonce":512665659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:26:39.596187+0000","modified":"2024-03-18T20:26:42.614846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12408":{"gid":12408,"name":"b","rank":0,"incarnation":122,"state":"up:active","state_seq":6,"addr":"172.21.15.165:6834/1134181166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1134181166},{"type":"v1","addr":"172.21.15.165:6834","nonce":1134181166}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-03-18T20:27:07.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.422+0000 7fd555804700 1 -- 172.21.15.1:0/2080789941 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd53006aed0 msgr2=0x7fd53006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:07.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.422+0000 7fd555804700 1 --2- 172.21.15.1:0/2080789941 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd53006aed0 0x7fd53006d390 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd550150070 tx=0x7fd538008040 comp rx=0 tx=0).stop 2024-03-18T20:27:07.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.422+0000 7fd555804700 1 -- 172.21.15.1:0/2080789941 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5500acd80 msgr2=0x7fd550151c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:07.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.422+0000 7fd555804700 1 --2- 172.21.15.1:0/2080789941 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5500acd80 0x7fd550151c40 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7fd54000b340 tx=0x7fd540004750 comp rx=0 tx=0).stop 2024-03-18T20:27:07.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.422+0000 7fd555804700 1 -- 172.21.15.1:0/2080789941 shutdown_connections 2024-03-18T20:27:07.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.422+0000 7fd555804700 1 --2- 172.21.15.1:0/2080789941 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd53006aed0 0x7fd53006d390 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.422+0000 7fd555804700 1 --2- 172.21.15.1:0/2080789941 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd55015a5e0 0x7fd5501ed720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.422+0000 7fd555804700 1 --2- 172.21.15.1:0/2080789941 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5500acd80 0x7fd550151c40 unknown :-1 s=CLOSED pgs=2990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.422+0000 7fd555804700 1 --2- 172.21.15.1:0/2080789941 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd5501545a0 0x7fd55013dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.422+0000 7fd555804700 1 -- 172.21.15.1:0/2080789941 >> 172.21.15.1:0/2080789941 conn(0x7fd55006b860 msgr2=0x7fd5500affe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:07.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.422+0000 7fd555804700 1 -- 172.21.15.1:0/2080789941 shutdown_connections 2024-03-18T20:27:07.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.422+0000 7fd555804700 1 -- 172.21.15.1:0/2080789941 wait complete. 2024-03-18T20:27:07.427 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 128 2024-03-18T20:27:07.438 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:27:07.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.558+0000 7f287a82c700 1 Processor -- start 2024-03-18T20:27:07.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.558+0000 7f287a82c700 1 -- start start 2024-03-18T20:27:07.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.559+0000 7f287a82c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2874099a40 0x7f2874099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:07.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.559+0000 7f287a82c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f287409a3e0 con 0x7f28740cc310 2024-03-18T20:27:07.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.559+0000 7f287a82c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28740d0c80 con 0x7f287415d680 2024-03-18T20:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.559+0000 7f287a82c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28740d09a0 con 0x7f2874099a40 2024-03-18T20:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.559+0000 7f2878dc9700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f28740cc310 0x7f28740cc7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49310/0 (socket says 172.21.15.1:49310) 2024-03-18T20:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.559+0000 7f2878dc9700 1 -- 172.21.15.1:0/876683180 learned_addr learned my addr 172.21.15.1:0/876683180 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.559+0000 7f28737fe700 1 --2- 172.21.15.1:0/876683180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2874099a40 0x7f2874099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.559+0000 7f2872ffd700 1 -- 172.21.15.1:0/876683180 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992553121 0 0) 0x7f287409a3e0 con 0x7f28740cc310 2024-03-18T20:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.559+0000 7f2872ffd700 1 -- 172.21.15.1:0/876683180 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28600035b0 con 0x7f28740cc310 2024-03-18T20:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f2872ffd700 1 -- 172.21.15.1:0/876683180 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3021038175 0 0) 0x7f28600035b0 con 0x7f28740cc310 2024-03-18T20:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f2872ffd700 1 -- 172.21.15.1:0/876683180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2874099a40 msgr2=0x7f2874099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f2872ffd700 1 --2- 172.21.15.1:0/876683180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2874099a40 0x7f2874099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f2872ffd700 1 -- 172.21.15.1:0/876683180 >> v1:172.21.15.165:6789/0 conn(0x7f287415d680 legacy=0x7f28740cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f2872ffd700 1 -- 172.21.15.1:0/876683180 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28740cb830 con 0x7f28740cc310 2024-03-18T20:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f2872ffd700 1 -- 172.21.15.1:0/876683180 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f286c0019f0 con 0x7f28740cc310 2024-03-18T20:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f2872ffd700 1 -- 172.21.15.1:0/876683180 >> v1:172.21.15.1:6789/0 conn(0x7f28740cc310 legacy=0x7f28740cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f2872ffd700 1 --2- 172.21.15.1:0/876683180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2860003a30 0x7f2860003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f2872ffd700 1 --2- 172.21.15.1:0/876683180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28600043e0 0x7f28600048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f2872ffd700 1 --2- 172.21.15.1:0/876683180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2874099a40 0x7f2860005110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f2872ffd700 1 -- 172.21.15.1:0/876683180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28740d09a0 con 0x7f28600043e0 2024-03-18T20:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f2872ffd700 1 -- 172.21.15.1:0/876683180 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286000ba10 con 0x7f2860003a30 2024-03-18T20:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f2872ffd700 1 -- 172.21.15.1:0/876683180 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286000bbd0 con 0x7f2874099a40 2024-03-18T20:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f28737fe700 1 --2- 172.21.15.1:0/876683180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28600043e0 0x7f28600048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f2873fff700 1 --2- 172.21.15.1:0/876683180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2860003a30 0x7f2860003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f2878dc9700 1 --2- 172.21.15.1:0/876683180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2874099a40 0x7f2860005110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f28737fe700 1 -- 172.21.15.1:0/876683180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2874099a40 msgr2=0x7f2860005110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f28737fe700 1 --2- 172.21.15.1:0/876683180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2874099a40 0x7f2860005110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f28737fe700 1 -- 172.21.15.1:0/876683180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2860003a30 msgr2=0x7f2860003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f28737fe700 1 --2- 172.21.15.1:0/876683180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2860003a30 0x7f2860003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f28737fe700 1 -- 172.21.15.1:0/876683180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28600035b0 con 0x7f28600043e0 2024-03-18T20:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.560+0000 7f28737fe700 1 --2- 172.21.15.1:0/876683180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28600043e0 0x7f28600048b0 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 crypto rx=0x7f285c009b60 tx=0x7f2860005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d2f5a270b09fa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:07.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.561+0000 7f2872ffd700 1 -- 172.21.15.1:0/876683180 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c00e6c0 con 0x7f28600043e0 2024-03-18T20:27:07.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.561+0000 7f2872ffd700 1 -- 172.21.15.1:0/876683180 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285c00e820 con 0x7f28600043e0 2024-03-18T20:27:07.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.561+0000 7f2872ffd700 1 -- 172.21.15.1:0/876683180 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c00eb90 con 0x7f28600043e0 2024-03-18T20:27:07.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.561+0000 7f287a82c700 1 -- 172.21.15.1:0/876683180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28600043e0 msgr2=0x7f28600048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:07.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.561+0000 7f287a82c700 1 --2- 172.21.15.1:0/876683180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28600043e0 0x7f28600048b0 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 crypto rx=0x7f285c009b60 tx=0x7f2860005880 comp rx=0 tx=0).stop 2024-03-18T20:27:07.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.561+0000 7f2878dc9700 1 -- 172.21.15.1:0/876683180 reap_dead start 2024-03-18T20:27:07.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.561+0000 7f287a82c700 1 -- 172.21.15.1:0/876683180 shutdown_connections 2024-03-18T20:27:07.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.561+0000 7f287a82c700 1 -- 172.21.15.1:0/876683180 >> 172.21.15.1:0/876683180 conn(0x7f28740b2ba0 msgr2=0x7f28740b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:07.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.561+0000 7f287a82c700 1 -- 172.21.15.1:0/876683180 shutdown_connections 2024-03-18T20:27:07.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.561+0000 7f287a82c700 1 -- 172.21.15.1:0/876683180 wait complete. 2024-03-18T20:27:07.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.563+0000 7f287a82c700 1 Processor -- start 2024-03-18T20:27:07.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.563+0000 7f287a82c700 1 -- start start 2024-03-18T20:27:07.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.563+0000 7f287a82c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28600043e0 0x7f28741f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:07.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.563+0000 7f287a82c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28741608a0 0x7f28740b55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:07.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.563+0000 7f287a82c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28740b5ae0 0x7f28740b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:07.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.563+0000 7f287a82c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28741f2e50 con 0x7f28600043e0 2024-03-18T20:27:07.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.563+0000 7f287a82c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287415af20 con 0x7f28741608a0 2024-03-18T20:27:07.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.563+0000 7f287a82c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28740b4740 con 0x7f28740b5ae0 2024-03-18T20:27:07.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.563+0000 7f2873fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28600043e0 0x7f28741f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:07.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.564+0000 7f2873fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28600043e0 0x7f28741f4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54070/0 (socket says 172.21.15.1:54070) 2024-03-18T20:27:07.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.564+0000 7f2873fff700 1 -- 172.21.15.1:0/2905086853 learned_addr learned my addr 172.21.15.1:0/2905086853 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:07.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.564+0000 7f28737fe700 1 --2- 172.21.15.1:0/2905086853 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28741608a0 0x7f28740b55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.564+0000 7f2878dc9700 1 --2- 172.21.15.1:0/2905086853 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28740b5ae0 0x7f28740b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.564+0000 7f2873fff700 1 -- 172.21.15.1:0/2905086853 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28740b5ae0 msgr2=0x7f28740b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.564+0000 7f2873fff700 1 --2- 172.21.15.1:0/2905086853 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28740b5ae0 0x7f28740b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.564+0000 7f2873fff700 1 -- 172.21.15.1:0/2905086853 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28741608a0 msgr2=0x7f28740b55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.564+0000 7f2873fff700 1 --2- 172.21.15.1:0/2905086853 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28741608a0 0x7f28740b55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.564+0000 7f2873fff700 1 -- 172.21.15.1:0/2905086853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28740b5380 con 0x7f28600043e0 2024-03-18T20:27:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.564+0000 7f2873fff700 1 --2- 172.21.15.1:0/2905086853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28600043e0 0x7f28741f4030 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 crypto rx=0x7f286400f780 tx=0x7f286400fa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.564+0000 7f2870ff9700 1 -- 172.21.15.1:0/2905086853 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2864017070 con 0x7f28600043e0 2024-03-18T20:27:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.564+0000 7f2870ff9700 1 -- 172.21.15.1:0/2905086853 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28640052e0 con 0x7f28600043e0 2024-03-18T20:27:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.564+0000 7f287a82c700 1 -- 172.21.15.1:0/2905086853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28740b3bc0 con 0x7f28600043e0 2024-03-18T20:27:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.564+0000 7f2870ff9700 1 -- 172.21.15.1:0/2905086853 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2864005470 con 0x7f28600043e0 2024-03-18T20:27:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.564+0000 7f287a82c700 1 -- 172.21.15.1:0/2905086853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28740b4180 con 0x7f28600043e0 2024-03-18T20:27:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.565+0000 7f287a82c700 1 -- 172.21.15.1:0/2905086853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f287406cd30 con 0x7f28600043e0 2024-03-18T20:27:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.566+0000 7f2870ff9700 1 -- 172.21.15.1:0/2905086853 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f2864006860 con 0x7f28600043e0 2024-03-18T20:27:07.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.566+0000 7f2870ff9700 1 --2- 172.21.15.1:0/2905086853 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f285406aed0 0x7f285406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:07.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.566+0000 7f28737fe700 1 --2- 172.21.15.1:0/2905086853 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f285406aed0 0x7f285406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:07.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.566+0000 7f2870ff9700 1 -- 172.21.15.1:0/2905086853 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 7001+0+0 (secure 0 0 0) 0x7f2864089340 con 0x7f28600043e0 2024-03-18T20:27:07.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.567+0000 7f28737fe700 1 --2- 172.21.15.1:0/2905086853 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f285406aed0 0x7f285406d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2874160d00 tx=0x7f285c01f040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:07.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.571+0000 7f2870ff9700 1 -- 172.21.15.1:0/2905086853 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2864053cb0 con 0x7f28600043e0 2024-03-18T20:27:07.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.772+0000 7f287a82c700 1 -- 172.21.15.1:0/2905086853 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28740676a0 con 0x7f28600043e0 2024-03-18T20:27:07.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.773+0000 7f2870ff9700 1 -- 172.21.15.1:0/2905086853 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v262) v1 ==== 74+0+17600 (secure 0 0 0) 0x7f28640572d0 con 0x7f28600043e0 2024-03-18T20:27:07.778 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:07.778 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":262,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:27:05.562956+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:26:37.326244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:26:38.564628+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"257","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.165:6835/3886150679":"2024-03-19T20:27:02.541703+0000","172.21.15.165:6833/3886150679":"2024-03-19T20:27:02.541703+0000","172.21.15.1:0/1135489018":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/1721669330":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/245086145":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3664103058":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3761024882":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3251013155":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/1287719189":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/2340206734":"2024-03-19T20:26:52.380559+0000","172.21.15.1:6833/148356":"2024-03-19T20:26:52.380559+0000","172.21.15.1:6832/148356":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/376843885":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/1340983360":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/369631788":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/2840085801":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/322251557":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/1615265957":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3049322266":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/1886004750":"2024-03-19T20:26:52.380559+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:27:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.776+0000 7f287a82c700 1 -- 172.21.15.1:0/2905086853 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f285406aed0 msgr2=0x7f285406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.776+0000 7f287a82c700 1 --2- 172.21.15.1:0/2905086853 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f285406aed0 0x7f285406d390 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2874160d00 tx=0x7f285c01f040 comp rx=0 tx=0).stop 2024-03-18T20:27:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.776+0000 7f287a82c700 1 -- 172.21.15.1:0/2905086853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28600043e0 msgr2=0x7f28741f4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.776+0000 7f287a82c700 1 --2- 172.21.15.1:0/2905086853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28600043e0 0x7f28741f4030 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 crypto rx=0x7f286400f780 tx=0x7f286400fa90 comp rx=0 tx=0).stop 2024-03-18T20:27:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.776+0000 7f287a82c700 1 -- 172.21.15.1:0/2905086853 shutdown_connections 2024-03-18T20:27:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.776+0000 7f287a82c700 1 --2- 172.21.15.1:0/2905086853 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f285406aed0 0x7f285406d390 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.776+0000 7f287a82c700 1 --2- 172.21.15.1:0/2905086853 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28740b5ae0 0x7f28740b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.776+0000 7f287a82c700 1 --2- 172.21.15.1:0/2905086853 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28600043e0 0x7f28741f4030 unknown :-1 s=CLOSED pgs=2993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.776+0000 7f287a82c700 1 --2- 172.21.15.1:0/2905086853 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28741608a0 0x7f28740b55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.776+0000 7f287a82c700 1 -- 172.21.15.1:0/2905086853 >> 172.21.15.1:0/2905086853 conn(0x7f28740b2ba0 msgr2=0x7f28740afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.776+0000 7f287a82c700 1 -- 172.21.15.1:0/2905086853 shutdown_connections 2024-03-18T20:27:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.776+0000 7f287a82c700 1 -- 172.21.15.1:0/2905086853 wait complete. 2024-03-18T20:27:07.793 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2024-03-18T20:27:07.793 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:27:07.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.914+0000 7f77fd60c700 1 Processor -- start 2024-03-18T20:27:07.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.914+0000 7f77fd60c700 1 -- start start 2024-03-18T20:27:07.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.914+0000 7f77fd60c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f77f80b02e0 0x7f77f80cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:07.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.914+0000 7f77fd60c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f80d06e0 con 0x7f77f809a150 2024-03-18T20:27:07.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.914+0000 7f77fd60c700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f80cfe40 con 0x7f77f806b860 2024-03-18T20:27:07.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.914+0000 7f77fd60c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77f806ea70 con 0x7f77f80b02e0 2024-03-18T20:27:07.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.915+0000 7f77f6ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f77f80b02e0 0x7f77f80cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:07.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.915+0000 7f77f77fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f77f809a150 0x7f77f815cc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49312/0 (socket says 172.21.15.1:49312) 2024-03-18T20:27:07.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.915+0000 7f77f77fe700 1 -- 172.21.15.1:0/3252341486 learned_addr learned my addr 172.21.15.1:0/3252341486 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:07.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.915+0000 7f77f6ffd700 1 -- 172.21.15.1:0/3252341486 >> v1:172.21.15.165:6789/0 conn(0x7f77f806b860 legacy=0x7f77f8099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:07.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.915+0000 7f77f6ffd700 1 -- 172.21.15.1:0/3252341486 >> v1:172.21.15.1:6789/0 conn(0x7f77f809a150 legacy=0x7f77f815cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:07.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.915+0000 7f77f6ffd700 1 -- 172.21.15.1:0/3252341486 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f80d0120 con 0x7f77f80b02e0 2024-03-18T20:27:07.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.915+0000 7f77f6ffd700 1 --2- 172.21.15.1:0/3252341486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77f80b02e0 0x7f77f80cf2a0 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7f77e8009840 tx=0x7f77e8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0716663082f03cb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:07.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.915+0000 7f77f5ffb700 1 -- 172.21.15.1:0/3252341486 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e8004230 con 0x7f77f80b02e0 2024-03-18T20:27:07.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.915+0000 7f77f5ffb700 1 -- 172.21.15.1:0/3252341486 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e8004390 con 0x7f77f80b02e0 2024-03-18T20:27:07.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.915+0000 7f77f5ffb700 1 -- 172.21.15.1:0/3252341486 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e80036a0 con 0x7f77f80b02e0 2024-03-18T20:27:07.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.916+0000 7f77fd60c700 1 -- 172.21.15.1:0/3252341486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77f80b02e0 msgr2=0x7f77f80cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:07.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.916+0000 7f77fd60c700 1 --2- 172.21.15.1:0/3252341486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77f80b02e0 0x7f77f80cf2a0 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7f77e8009840 tx=0x7f77e8009b50 comp rx=0 tx=0).stop 2024-03-18T20:27:07.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.916+0000 7f77fd60c700 1 -- 172.21.15.1:0/3252341486 shutdown_connections 2024-03-18T20:27:07.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.916+0000 7f77fd60c700 1 --2- 172.21.15.1:0/3252341486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77f80b02e0 0x7f77f80cf2a0 unknown :-1 s=CLOSED pgs=2995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.916+0000 7f77fd60c700 1 -- 172.21.15.1:0/3252341486 >> 172.21.15.1:0/3252341486 conn(0x7f77f80b2ac0 msgr2=0x7f77f80b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:07.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.916+0000 7f77fd60c700 1 -- 172.21.15.1:0/3252341486 shutdown_connections 2024-03-18T20:27:07.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.916+0000 7f77fd60c700 1 -- 172.21.15.1:0/3252341486 wait complete. 2024-03-18T20:27:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.917+0000 7f77fd60c700 1 Processor -- start 2024-03-18T20:27:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.917+0000 7f77fd60c700 1 -- start start 2024-03-18T20:27:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.918+0000 7f77fd60c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77f806b860 0x7f77f80b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.918+0000 7f77fd60c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77f809a150 0x7f77f81f7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.918+0000 7f77fd60c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77f80b02e0 0x7f77f8151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.918+0000 7f77fd60c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f8152960 con 0x7f77f806b860 2024-03-18T20:27:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.918+0000 7f77fd60c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f81f7a90 con 0x7f77f80b02e0 2024-03-18T20:27:07.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.918+0000 7f77fd60c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77f80b5770 con 0x7f77f809a150 2024-03-18T20:27:07.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.918+0000 7f77f6ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77f806b860 0x7f77f80b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:07.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.918+0000 7f77f6ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77f806b860 0x7f77f80b5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54086/0 (socket says 172.21.15.1:54086) 2024-03-18T20:27:07.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.918+0000 7f77f6ffd700 1 -- 172.21.15.1:0/275019417 learned_addr learned my addr 172.21.15.1:0/275019417 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:07.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.918+0000 7f77f77fe700 1 --2- 172.21.15.1:0/275019417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77f80b02e0 0x7f77f8151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:07.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.918+0000 7f77f6ffd700 1 -- 172.21.15.1:0/275019417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77f809a150 msgr2=0x7f77f81f7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:07.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.918+0000 7f77f67fc700 1 --2- 172.21.15.1:0/275019417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77f809a150 0x7f77f81f7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:07.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.918+0000 7f77f6ffd700 1 --2- 172.21.15.1:0/275019417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77f809a150 0x7f77f81f7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.918+0000 7f77f6ffd700 1 -- 172.21.15.1:0/275019417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77f80b02e0 msgr2=0x7f77f8151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:07.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.918+0000 7f77f6ffd700 1 --2- 172.21.15.1:0/275019417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77f80b02e0 0x7f77f8151dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:07.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.918+0000 7f77f6ffd700 1 -- 172.21.15.1:0/275019417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e80094a0 con 0x7f77f806b860 2024-03-18T20:27:07.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.918+0000 7f77f77fe700 1 --2- 172.21.15.1:0/275019417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77f80b02e0 0x7f77f8151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:07.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.918+0000 7f77f67fc700 1 --2- 172.21.15.1:0/275019417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77f809a150 0x7f77f81f7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:07.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.918+0000 7f77f6ffd700 1 --2- 172.21.15.1:0/275019417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77f806b860 0x7f77f80b5100 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 crypto rx=0x7f77e80090b0 tx=0x7f77e80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:07.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.919+0000 7f77df7fe700 1 -- 172.21.15.1:0/275019417 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e8011830 con 0x7f77f806b860 2024-03-18T20:27:07.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.919+0000 7f77df7fe700 1 -- 172.21.15.1:0/275019417 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e8011990 con 0x7f77f806b860 2024-03-18T20:27:07.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.919+0000 7f77df7fe700 1 -- 172.21.15.1:0/275019417 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e8011d40 con 0x7f77f806b860 2024-03-18T20:27:07.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.919+0000 7f77fd60c700 1 -- 172.21.15.1:0/275019417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f80b5a50 con 0x7f77f806b860 2024-03-18T20:27:07.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.919+0000 7f77fd60c700 1 -- 172.21.15.1:0/275019417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f80b6010 con 0x7f77f806b860 2024-03-18T20:27:07.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.920+0000 7f77fd60c700 1 -- 172.21.15.1:0/275019417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f8073350 con 0x7f77f806b860 2024-03-18T20:27:07.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.920+0000 7f77df7fe700 1 -- 172.21.15.1:0/275019417 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f77e801b440 con 0x7f77f806b860 2024-03-18T20:27:07.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.920+0000 7f77df7fe700 1 --2- 172.21.15.1:0/275019417 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f77e406aed0 0x7f77e406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:07.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.921+0000 7f77df7fe700 1 -- 172.21.15.1:0/275019417 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 7001+0+0 (secure 0 0 0) 0x7f77e808b530 con 0x7f77f806b860 2024-03-18T20:27:07.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.921+0000 7f77f67fc700 1 --2- 172.21.15.1:0/275019417 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f77e406aed0 0x7f77e406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:07.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.921+0000 7f77f67fc700 1 --2- 172.21.15.1:0/275019417 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f77e406aed0 0x7f77e406d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f77ec006fd0 tx=0x7f77ec008040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:07.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:07.925+0000 7f77df7fe700 1 -- 172.21.15.1:0/275019417 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f77e8055f30 con 0x7f77f806b860 2024-03-18T20:27:08.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.172+0000 7f77fd60c700 1 -- 172.21.15.1:0/275019417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77f80cb2e0 con 0x7f77f806b860 2024-03-18T20:27:08.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.172+0000 7f77df7fe700 1 -- 172.21.15.1:0/275019417 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+4749 (secure 0 0 0) 0x7f77e8059550 con 0x7f77f806b860 2024-03-18T20:27:08.177 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:08.177 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":128,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9851,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3262033681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":3262033681},{"type":"v1","addr":"172.21.15.165:6835","nonce":3262033681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":128},{"gid":12396,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2565889517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2565889517},{"type":"v1","addr":"172.21.15.1:6836","nonce":2565889517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":12399,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/512665659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":512665659},{"type":"v1","addr":"172.21.15.1:6837","nonce":512665659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:26:39.596187+0000","modified":"2024-03-18T20:26:42.614846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12408":{"gid":12408,"name":"b","rank":0,"incarnation":122,"state":"up:active","state_seq":6,"addr":"172.21.15.165:6834/1134181166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1134181166},{"type":"v1","addr":"172.21.15.165:6834","nonce":1134181166}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-03-18T20:27:08.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.175+0000 7f77fd60c700 1 -- 172.21.15.1:0/275019417 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f77e406aed0 msgr2=0x7f77e406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:08.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.175+0000 7f77fd60c700 1 --2- 172.21.15.1:0/275019417 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f77e406aed0 0x7f77e406d390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f77ec006fd0 tx=0x7f77ec008040 comp rx=0 tx=0).stop 2024-03-18T20:27:08.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.175+0000 7f77fd60c700 1 -- 172.21.15.1:0/275019417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77f806b860 msgr2=0x7f77f80b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:08.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.175+0000 7f77fd60c700 1 --2- 172.21.15.1:0/275019417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77f806b860 0x7f77f80b5100 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 crypto rx=0x7f77e80090b0 tx=0x7f77e80047c0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.175+0000 7f77fd60c700 1 -- 172.21.15.1:0/275019417 shutdown_connections 2024-03-18T20:27:08.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.175+0000 7f77fd60c700 1 --2- 172.21.15.1:0/275019417 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f77e406aed0 0x7f77e406d390 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.175+0000 7f77fd60c700 1 --2- 172.21.15.1:0/275019417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f77f80b02e0 0x7f77f8151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.175+0000 7f77fd60c700 1 --2- 172.21.15.1:0/275019417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77f806b860 0x7f77f80b5100 unknown :-1 s=CLOSED pgs=2996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.175+0000 7f77fd60c700 1 --2- 172.21.15.1:0/275019417 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f77f809a150 0x7f77f81f7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.175+0000 7f77fd60c700 1 -- 172.21.15.1:0/275019417 >> 172.21.15.1:0/275019417 conn(0x7f77f80b2ac0 msgr2=0x7f77f815c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:08.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.175+0000 7f77fd60c700 1 -- 172.21.15.1:0/275019417 shutdown_connections 2024-03-18T20:27:08.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.175+0000 7f77fd60c700 1 -- 172.21.15.1:0/275019417 wait complete. 2024-03-18T20:27:08.180 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 128 2024-03-18T20:27:08.191 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:27:08.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.312+0000 7fa105221700 1 Processor -- start 2024-03-18T20:27:08.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.312+0000 7fa105221700 1 -- start start 2024-03-18T20:27:08.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.313+0000 7fa105221700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1001465a0 0x7fa100168660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:08.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.313+0000 7fa105221700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa100072660 con 0x7fa100146190 2024-03-18T20:27:08.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.313+0000 7fa105221700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa10006ebd0 con 0x7fa10006ae60 2024-03-18T20:27:08.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.313+0000 7fa105221700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa10006f850 con 0x7fa1001465a0 2024-03-18T20:27:08.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.313+0000 7fa0fed9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa100146190 0x7fa10006a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49314/0 (socket says 172.21.15.1:49314) 2024-03-18T20:27:08.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.313+0000 7fa0fed9d700 1 -- 172.21.15.1:0/372557995 learned_addr learned my addr 172.21.15.1:0/372557995 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:08.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0ff59e700 1 --2- 172.21.15.1:0/372557995 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1001465a0 0x7fa100168660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0fdd9b700 1 -- 172.21.15.1:0/372557995 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747603713 0 0) 0x7fa100072660 con 0x7fa100146190 2024-03-18T20:27:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0fdd9b700 1 -- 172.21.15.1:0/372557995 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0ec0035b0 con 0x7fa100146190 2024-03-18T20:27:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0fdd9b700 1 -- 172.21.15.1:0/372557995 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 359250014 0 0) 0x7fa0ec0035b0 con 0x7fa100146190 2024-03-18T20:27:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0fdd9b700 1 -- 172.21.15.1:0/372557995 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1001465a0 msgr2=0x7fa100168660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0fdd9b700 1 --2- 172.21.15.1:0/372557995 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1001465a0 0x7fa100168660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0fdd9b700 1 -- 172.21.15.1:0/372557995 >> v1:172.21.15.165:6789/0 conn(0x7fa10006ae60 legacy=0x7fa100164f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0fdd9b700 1 -- 172.21.15.1:0/372557995 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa10006eeb0 con 0x7fa100146190 2024-03-18T20:27:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0fdd9b700 1 -- 172.21.15.1:0/372557995 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fa0f0001670 con 0x7fa100146190 2024-03-18T20:27:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0fdd9b700 1 -- 172.21.15.1:0/372557995 >> v1:172.21.15.1:6789/0 conn(0x7fa100146190 legacy=0x7fa10006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0fdd9b700 1 --2- 172.21.15.1:0/372557995 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1001465a0 0x7fa0ec003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0fdd9b700 1 --2- 172.21.15.1:0/372557995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa0ec003f70 0x7fa0ec004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0fdd9b700 1 --2- 172.21.15.1:0/372557995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0ec004d90 0x7fa0ec005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:08.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0fdd9b700 1 -- 172.21.15.1:0/372557995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10006f850 con 0x7fa0ec004d90 2024-03-18T20:27:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0fdd9b700 1 -- 172.21.15.1:0/372557995 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec00ba70 con 0x7fa0ec003f70 2024-03-18T20:27:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0fdd9b700 1 -- 172.21.15.1:0/372557995 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec00bc30 con 0x7fa1001465a0 2024-03-18T20:27:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0ff59e700 1 --2- 172.21.15.1:0/372557995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0ec004d90 0x7fa0ec005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0fed9d700 1 --2- 172.21.15.1:0/372557995 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1001465a0 0x7fa0ec003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0fe59c700 1 --2- 172.21.15.1:0/372557995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa0ec003f70 0x7fa0ec004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0ff59e700 1 -- 172.21.15.1:0/372557995 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1001465a0 msgr2=0x7fa0ec003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0ff59e700 1 --2- 172.21.15.1:0/372557995 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa1001465a0 0x7fa0ec003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0ff59e700 1 -- 172.21.15.1:0/372557995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa0ec003f70 msgr2=0x7fa0ec004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0ff59e700 1 --2- 172.21.15.1:0/372557995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa0ec003f70 0x7fa0ec004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.314+0000 7fa0ff59e700 1 -- 172.21.15.1:0/372557995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ec0035b0 con 0x7fa0ec004d90 2024-03-18T20:27:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.315+0000 7fa0ff59e700 1 --2- 172.21.15.1:0/372557995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0ec004d90 0x7fa0ec005210 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009fc0 tx=0x7fa0ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff7f4287f9fac6d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.315+0000 7fa0fdd9b700 1 -- 172.21.15.1:0/372557995 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f800ec30 con 0x7fa0ec004d90 2024-03-18T20:27:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.315+0000 7fa0fdd9b700 1 -- 172.21.15.1:0/372557995 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f800f040 con 0x7fa0ec004d90 2024-03-18T20:27:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.315+0000 7fa0fdd9b700 1 -- 172.21.15.1:0/372557995 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8014480 con 0x7fa0ec004d90 2024-03-18T20:27:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.315+0000 7fa105221700 1 -- 172.21.15.1:0/372557995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0ec004d90 msgr2=0x7fa0ec005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.315+0000 7fa105221700 1 --2- 172.21.15.1:0/372557995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa0ec004d90 0x7fa0ec005210 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009fc0 tx=0x7fa0ec005880 comp rx=0 tx=0).stop 2024-03-18T20:27:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.315+0000 7fa0ff59e700 1 -- 172.21.15.1:0/372557995 reap_dead start 2024-03-18T20:27:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.315+0000 7fa105221700 1 -- 172.21.15.1:0/372557995 shutdown_connections 2024-03-18T20:27:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.315+0000 7fa105221700 1 -- 172.21.15.1:0/372557995 >> 172.21.15.1:0/372557995 conn(0x7fa1000b1d20 msgr2=0x7fa1000b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.315+0000 7fa105221700 1 -- 172.21.15.1:0/372557995 shutdown_connections 2024-03-18T20:27:08.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.315+0000 7fa105221700 1 -- 172.21.15.1:0/372557995 wait complete. 2024-03-18T20:27:08.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.317+0000 7fa105221700 1 Processor -- start 2024-03-18T20:27:08.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.317+0000 7fa105221700 1 -- start start 2024-03-18T20:27:08.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.318+0000 7fa105221700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1001e8b80 0x7fa1001422e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:08.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.318+0000 7fa105221700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa100140680 0x7fa100140b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:08.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.318+0000 7fa105221700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1001f2fe0 0x7fa1001f3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:08.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.318+0000 7fa105221700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa100141ec0 con 0x7fa1001e8b80 2024-03-18T20:27:08.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.318+0000 7fa105221700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa100141040 con 0x7fa1001f2fe0 2024-03-18T20:27:08.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.318+0000 7fa105221700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa100141240 con 0x7fa100140680 2024-03-18T20:27:08.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.318+0000 7fa0fed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1001e8b80 0x7fa1001422e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:08.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.318+0000 7fa0fed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1001e8b80 0x7fa1001422e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54096/0 (socket says 172.21.15.1:54096) 2024-03-18T20:27:08.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.318+0000 7fa0fed9d700 1 -- 172.21.15.1:0/579468548 learned_addr learned my addr 172.21.15.1:0/579468548 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:08.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.318+0000 7fa0fe59c700 1 --2- 172.21.15.1:0/579468548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa100140680 0x7fa100140b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:08.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.318+0000 7fa0ff59e700 1 --2- 172.21.15.1:0/579468548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1001f2fe0 0x7fa1001f3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:08.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.318+0000 7fa0fed9d700 1 -- 172.21.15.1:0/579468548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa100140680 msgr2=0x7fa100140b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:08.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.318+0000 7fa0fed9d700 1 --2- 172.21.15.1:0/579468548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa100140680 0x7fa100140b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.318+0000 7fa0fed9d700 1 -- 172.21.15.1:0/579468548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1001f2fe0 msgr2=0x7fa1001f3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:08.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.318+0000 7fa0fed9d700 1 --2- 172.21.15.1:0/579468548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1001f2fe0 0x7fa1001f3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.318+0000 7fa0fed9d700 1 -- 172.21.15.1:0/579468548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa100141520 con 0x7fa1001e8b80 2024-03-18T20:27:08.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.318+0000 7fa0fed9d700 1 --2- 172.21.15.1:0/579468548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1001e8b80 0x7fa1001422e0 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0007290 tx=0x7fa0f00075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:08.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.318+0000 7fa0f77fe700 1 -- 172.21.15.1:0/579468548 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f000f600 con 0x7fa1001e8b80 2024-03-18T20:27:08.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.319+0000 7fa0f77fe700 1 -- 172.21.15.1:0/579468548 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f000f760 con 0x7fa1001e8b80 2024-03-18T20:27:08.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.319+0000 7fa105221700 1 -- 172.21.15.1:0/579468548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa100141800 con 0x7fa1001e8b80 2024-03-18T20:27:08.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.319+0000 7fa0f77fe700 1 -- 172.21.15.1:0/579468548 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f000faf0 con 0x7fa1001e8b80 2024-03-18T20:27:08.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.319+0000 7fa105221700 1 -- 172.21.15.1:0/579468548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1001e2940 con 0x7fa1001e8b80 2024-03-18T20:27:08.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.319+0000 7fa105221700 1 -- 172.21.15.1:0/579468548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa100142000 con 0x7fa1001e8b80 2024-03-18T20:27:08.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.320+0000 7fa0f77fe700 1 -- 172.21.15.1:0/579468548 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fa0f000fc50 con 0x7fa1001e8b80 2024-03-18T20:27:08.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.320+0000 7fa0f77fe700 1 --2- 172.21.15.1:0/579468548 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa0e006aed0 0x7fa0e006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:08.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.321+0000 7fa0f77fe700 1 -- 172.21.15.1:0/579468548 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 7001+0+0 (secure 0 0 0) 0x7fa0f0089bc0 con 0x7fa1001e8b80 2024-03-18T20:27:08.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.321+0000 7fa0fe59c700 1 --2- 172.21.15.1:0/579468548 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa0e006aed0 0x7fa0e006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:08.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.321+0000 7fa0fe59c700 1 --2- 172.21.15.1:0/579468548 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa0e006aed0 0x7fa0e006d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa0e80011a0 tx=0x7fa0e8009040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:08.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.325+0000 7fa0f77fe700 1 -- 172.21.15.1:0/579468548 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa100142000 con 0x7fa1001e8b80 2024-03-18T20:27:08.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.576+0000 7fa105221700 1 -- 172.21.15.1:0/579468548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa100142000 con 0x7fa1001e8b80 2024-03-18T20:27:08.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.577+0000 7fa0f77fe700 1 -- 172.21.15.1:0/579468548 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+4749 (secure 0 0 0) 0x7fa100142000 con 0x7fa1001e8b80 2024-03-18T20:27:08.581 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:08.581 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":128,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9851,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3262033681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":3262033681},{"type":"v1","addr":"172.21.15.165:6835","nonce":3262033681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":128},{"gid":12396,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2565889517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2565889517},{"type":"v1","addr":"172.21.15.1:6836","nonce":2565889517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":12399,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/512665659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":512665659},{"type":"v1","addr":"172.21.15.1:6837","nonce":512665659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:26:39.596187+0000","modified":"2024-03-18T20:26:42.614846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12408},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12408":{"gid":12408,"name":"b","rank":0,"incarnation":122,"state":"up:active","state_seq":6,"addr":"172.21.15.165:6834/1134181166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1134181166},{"type":"v1","addr":"172.21.15.165:6834","nonce":1134181166}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-03-18T20:27:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.579+0000 7fa105221700 1 -- 172.21.15.1:0/579468548 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa0e006aed0 msgr2=0x7fa0e006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.579+0000 7fa105221700 1 --2- 172.21.15.1:0/579468548 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa0e006aed0 0x7fa0e006d390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa0e80011a0 tx=0x7fa0e8009040 comp rx=0 tx=0).stop 2024-03-18T20:27:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.579+0000 7fa105221700 1 -- 172.21.15.1:0/579468548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1001e8b80 msgr2=0x7fa1001422e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.579+0000 7fa105221700 1 --2- 172.21.15.1:0/579468548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1001e8b80 0x7fa1001422e0 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0007290 tx=0x7fa0f00075a0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.579+0000 7fa105221700 1 -- 172.21.15.1:0/579468548 shutdown_connections 2024-03-18T20:27:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.579+0000 7fa105221700 1 --2- 172.21.15.1:0/579468548 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa0e006aed0 0x7fa0e006d390 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.579+0000 7fa105221700 1 --2- 172.21.15.1:0/579468548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1001f2fe0 0x7fa1001f3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.579+0000 7fa105221700 1 --2- 172.21.15.1:0/579468548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1001e8b80 0x7fa1001422e0 unknown :-1 s=CLOSED pgs=2999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.579+0000 7fa105221700 1 --2- 172.21.15.1:0/579468548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa100140680 0x7fa100140b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.579+0000 7fa105221700 1 -- 172.21.15.1:0/579468548 >> 172.21.15.1:0/579468548 conn(0x7fa1000b1d20 msgr2=0x7fa1000b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.579+0000 7fa105221700 1 -- 172.21.15.1:0/579468548 shutdown_connections 2024-03-18T20:27:08.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.579+0000 7fa105221700 1 -- 172.21.15.1:0/579468548 wait complete. 2024-03-18T20:27:08.585 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 128 2024-03-18T20:27:08.596 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:27:08.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.717+0000 7fcf3bab6700 1 Processor -- start 2024-03-18T20:27:08.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.717+0000 7fcf3bab6700 1 -- start start 2024-03-18T20:27:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.718+0000 7fcf3bab6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcf3415a2e0 0x7fcf340ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.718+0000 7fcf3bab6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcf340cc170 0x7fcf3406ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:08.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.718+0000 7fcf3bab6700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3406e270 con 0x7fcf340ae600 2024-03-18T20:27:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.718+0000 7fcf3bab6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf340ae050 con 0x7fcf3415a2e0 2024-03-18T20:27:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.718+0000 7fcf3bab6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf340add70 con 0x7fcf340cc170 2024-03-18T20:27:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.718+0000 7fcf39852700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcf340ae600 0x7fcf340ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49316/0 (socket says 172.21.15.1:49316) 2024-03-18T20:27:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.718+0000 7fcf39852700 1 -- 172.21.15.1:0/192913110 learned_addr learned my addr 172.21.15.1:0/192913110 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.718+0000 7fcf39051700 1 --2- 172.21.15.1:0/192913110 >> v2:172.21.15.1:3300/0 conn(0x7fcf3415a2e0 0x7fcf340ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.718+0000 7fcf3a053700 1 --2- 172.21.15.1:0/192913110 >> v2:172.21.15.165:3300/0 conn(0x7fcf340cc170 0x7fcf3406ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.719+0000 7fcf38850700 1 -- 172.21.15.1:0/192913110 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339062955 0 0) 0x7fcf3406e270 con 0x7fcf340ae600 2024-03-18T20:27:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.719+0000 7fcf38850700 1 -- 172.21.15.1:0/192913110 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf200035b0 con 0x7fcf340ae600 2024-03-18T20:27:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.719+0000 7fcf39051700 1 -- 172.21.15.1:0/192913110 >> v2:172.21.15.165:3300/0 conn(0x7fcf340cc170 msgr2=0x7fcf3406ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.719+0000 7fcf39051700 1 --2- 172.21.15.1:0/192913110 >> v2:172.21.15.165:3300/0 conn(0x7fcf340cc170 0x7fcf3406ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.719+0000 7fcf39051700 1 -- 172.21.15.1:0/192913110 >> v1:172.21.15.1:6789/0 conn(0x7fcf340ae600 legacy=0x7fcf340ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.719+0000 7fcf39051700 1 -- 172.21.15.1:0/192913110 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3406e4a0 con 0x7fcf3415a2e0 2024-03-18T20:27:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.719+0000 7fcf39051700 1 --2- 172.21.15.1:0/192913110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf3415a2e0 0x7fcf340ad790 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 crypto rx=0x7fcf340cc100 tx=0x7fcf30009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48c6b959ba4bb19e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.719+0000 7fcf38850700 1 -- 172.21.15.1:0/192913110 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf30011830 con 0x7fcf3415a2e0 2024-03-18T20:27:08.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.719+0000 7fcf38850700 1 -- 172.21.15.1:0/192913110 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf30011990 con 0x7fcf3415a2e0 2024-03-18T20:27:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.719+0000 7fcf38850700 1 -- 172.21.15.1:0/192913110 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf30011d20 con 0x7fcf3415a2e0 2024-03-18T20:27:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.719+0000 7fcf3bab6700 1 -- 172.21.15.1:0/192913110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf3415a2e0 msgr2=0x7fcf340ad790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.719+0000 7fcf3bab6700 1 --2- 172.21.15.1:0/192913110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf3415a2e0 0x7fcf340ad790 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 crypto rx=0x7fcf340cc100 tx=0x7fcf30009b20 comp rx=0 tx=0).stop 2024-03-18T20:27:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.720+0000 7fcf3bab6700 1 -- 172.21.15.1:0/192913110 shutdown_connections 2024-03-18T20:27:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.720+0000 7fcf3bab6700 1 --2- 172.21.15.1:0/192913110 >> v2:172.21.15.165:3300/0 conn(0x7fcf340cc170 0x7fcf3406ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.720+0000 7fcf3bab6700 1 --2- 172.21.15.1:0/192913110 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf3415a2e0 0x7fcf340ad790 unknown :-1 s=CLOSED pgs=3001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.720+0000 7fcf3bab6700 1 -- 172.21.15.1:0/192913110 >> 172.21.15.1:0/192913110 conn(0x7fcf3406b860 msgr2=0x7fcf3406bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.720+0000 7fcf3bab6700 1 -- 172.21.15.1:0/192913110 shutdown_connections 2024-03-18T20:27:08.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.720+0000 7fcf3bab6700 1 -- 172.21.15.1:0/192913110 wait complete. 2024-03-18T20:27:08.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.721+0000 7fcf3bab6700 1 Processor -- start 2024-03-18T20:27:08.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.721+0000 7fcf3bab6700 1 -- start start 2024-03-18T20:27:08.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf3bab6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf340ae600 0x7fcf341419c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:08.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf3bab6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf340cc170 0x7fcf340cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:08.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf3bab6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf3415a2e0 0x7fcf340ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:08.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf3bab6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf340cd160 con 0x7fcf3415a2e0 2024-03-18T20:27:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf3bab6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf340cd440 con 0x7fcf340cc170 2024-03-18T20:27:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf3bab6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf340cd720 con 0x7fcf340ae600 2024-03-18T20:27:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf3a053700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf3415a2e0 0x7fcf340ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf3a053700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf3415a2e0 0x7fcf340ccb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54122/0 (socket says 172.21.15.1:54122) 2024-03-18T20:27:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf3a053700 1 -- 172.21.15.1:0/2423658021 learned_addr learned my addr 172.21.15.1:0/2423658021 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf39051700 1 --2- 172.21.15.1:0/2423658021 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf340cc170 0x7fcf340cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf39852700 1 --2- 172.21.15.1:0/2423658021 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf340ae600 0x7fcf341419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf3a053700 1 -- 172.21.15.1:0/2423658021 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf340ae600 msgr2=0x7fcf341419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf3a053700 1 --2- 172.21.15.1:0/2423658021 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf340ae600 0x7fcf341419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf3a053700 1 -- 172.21.15.1:0/2423658021 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf340cc170 msgr2=0x7fcf340cc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf3a053700 1 --2- 172.21.15.1:0/2423658021 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf340cc170 0x7fcf340cc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf3a053700 1 -- 172.21.15.1:0/2423658021 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf300094a0 con 0x7fcf3415a2e0 2024-03-18T20:27:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf39051700 1 --2- 172.21.15.1:0/2423658021 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf340cc170 0x7fcf340cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:08.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf3a053700 1 --2- 172.21.15.1:0/2423658021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf3415a2e0 0x7fcf340ccb70 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c007e40 tx=0x7fcf2c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:08.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf2a7fc700 1 -- 172.21.15.1:0/2423658021 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c00eea0 con 0x7fcf3415a2e0 2024-03-18T20:27:08.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf2a7fc700 1 -- 172.21.15.1:0/2423658021 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2c014420 con 0x7fcf3415a2e0 2024-03-18T20:27:08.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf3bab6700 1 -- 172.21.15.1:0/2423658021 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf340cda00 con 0x7fcf3415a2e0 2024-03-18T20:27:08.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf2a7fc700 1 -- 172.21.15.1:0/2423658021 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c0147b0 con 0x7fcf3415a2e0 2024-03-18T20:27:08.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.722+0000 7fcf3bab6700 1 -- 172.21.15.1:0/2423658021 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf34152890 con 0x7fcf3415a2e0 2024-03-18T20:27:08.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.723+0000 7fcf3bab6700 1 -- 172.21.15.1:0/2423658021 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf340cdf20 con 0x7fcf3415a2e0 2024-03-18T20:27:08.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.724+0000 7fcf2a7fc700 1 -- 172.21.15.1:0/2423658021 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fcf2c014910 con 0x7fcf3415a2e0 2024-03-18T20:27:08.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.725+0000 7fcf2a7fc700 1 --2- 172.21.15.1:0/2423658021 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcf1806aed0 0x7fcf1806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:08.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.725+0000 7fcf2a7fc700 1 -- 172.21.15.1:0/2423658021 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 7001+0+0 (secure 0 0 0) 0x7fcf2c08b030 con 0x7fcf3415a2e0 2024-03-18T20:27:08.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.725+0000 7fcf39852700 1 --2- 172.21.15.1:0/2423658021 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcf1806aed0 0x7fcf1806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:08.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.725+0000 7fcf39852700 1 --2- 172.21.15.1:0/2423658021 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcf1806aed0 0x7fcf1806d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcf240011e0 tx=0x7fcf24009040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:08.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.729+0000 7fcf2a7fc700 1 -- 172.21.15.1:0/2423658021 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcf2c055b50 con 0x7fcf3415a2e0 2024-03-18T20:27:08.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.934+0000 7fcf3bab6700 1 -- 172.21.15.1:0/2423658021 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf340cdf20 con 0x7fcf3415a2e0 2024-03-18T20:27:08.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.935+0000 7fcf2a7fc700 1 -- 172.21.15.1:0/2423658021 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v262) v1 ==== 74+0+17600 (secure 0 0 0) 0x7fcf340cdf20 con 0x7fcf3415a2e0 2024-03-18T20:27:08.939 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:08.939 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":262,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:27:05.562956+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_metadata","create_time":"2024-03-18T20:26:37.326244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"256","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"backup_fs_data","create_time":"2024-03-18T20:26:38.564628+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"257","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.165:6835/3886150679":"2024-03-19T20:27:02.541703+0000","172.21.15.165:6833/3886150679":"2024-03-19T20:27:02.541703+0000","172.21.15.1:0/1135489018":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/1721669330":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/245086145":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3664103058":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3761024882":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3251013155":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/1287719189":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/2340206734":"2024-03-19T20:26:52.380559+0000","172.21.15.1:6833/148356":"2024-03-19T20:26:52.380559+0000","172.21.15.1:6832/148356":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/376843885":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/1340983360":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/369631788":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/2840085801":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/322251557":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/1615265957":"2024-03-19T20:27:00.293257+0000","172.21.15.1:0/3049322266":"2024-03-19T20:26:52.380559+0000","172.21.15.1:0/1886004750":"2024-03-19T20:26:52.380559+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:27:08.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.937+0000 7fcf3bab6700 1 -- 172.21.15.1:0/2423658021 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcf1806aed0 msgr2=0x7fcf1806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:08.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.937+0000 7fcf3bab6700 1 --2- 172.21.15.1:0/2423658021 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcf1806aed0 0x7fcf1806d390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcf240011e0 tx=0x7fcf24009040 comp rx=0 tx=0).stop 2024-03-18T20:27:08.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.937+0000 7fcf3bab6700 1 -- 172.21.15.1:0/2423658021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf3415a2e0 msgr2=0x7fcf340ccb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:08.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.937+0000 7fcf3bab6700 1 --2- 172.21.15.1:0/2423658021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf3415a2e0 0x7fcf340ccb70 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c007e40 tx=0x7fcf2c00e3c0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.937+0000 7fcf3bab6700 1 -- 172.21.15.1:0/2423658021 shutdown_connections 2024-03-18T20:27:08.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.937+0000 7fcf3bab6700 1 --2- 172.21.15.1:0/2423658021 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcf1806aed0 0x7fcf1806d390 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.937+0000 7fcf3bab6700 1 --2- 172.21.15.1:0/2423658021 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcf340ae600 0x7fcf341419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.937+0000 7fcf3bab6700 1 --2- 172.21.15.1:0/2423658021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf3415a2e0 0x7fcf340ccb70 unknown :-1 s=CLOSED pgs=3002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.937+0000 7fcf3bab6700 1 --2- 172.21.15.1:0/2423658021 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcf340cc170 0x7fcf340cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:08.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.937+0000 7fcf3bab6700 1 -- 172.21.15.1:0/2423658021 >> 172.21.15.1:0/2423658021 conn(0x7fcf3406b860 msgr2=0x7fcf340b2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:08.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.937+0000 7fcf3bab6700 1 -- 172.21.15.1:0/2423658021 shutdown_connections 2024-03-18T20:27:08.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:08.937+0000 7fcf3bab6700 1 -- 172.21.15.1:0/2423658021 wait complete. 2024-03-18T20:27:08.954 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2024-03-18T20:27:09.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.076+0000 7f59cf214700 1 Processor -- start 2024-03-18T20:27:09.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.076+0000 7f59cf214700 1 -- start start 2024-03-18T20:27:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.076+0000 7f59cf214700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f59c8071d30 0x7f59c80add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.076+0000 7f59cf214700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c80ce8f0 con 0x7f59c815a1d0 2024-03-18T20:27:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.076+0000 7f59cf214700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c80ae260 con 0x7f59c80cb9e0 2024-03-18T20:27:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.076+0000 7f59cf214700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59c80cdb00 con 0x7f59c8071d30 2024-03-18T20:27:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.077+0000 7f59c7fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f59c8071d30 0x7f59c80add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.077+0000 7f59ccfb0700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f59c815a1d0 0x7f59c8071620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49326/0 (socket says 172.21.15.1:49326) 2024-03-18T20:27:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.077+0000 7f59ccfb0700 1 -- 172.21.15.1:0/632044775 learned_addr learned my addr 172.21.15.1:0/632044775 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.077+0000 7f59c7fff700 1 -- 172.21.15.1:0/632044775 >> v1:172.21.15.165:6789/0 conn(0x7f59c80cb9e0 legacy=0x7f59c80cbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.077+0000 7f59c7fff700 1 -- 172.21.15.1:0/632044775 >> v1:172.21.15.1:6789/0 conn(0x7f59c815a1d0 legacy=0x7f59c8071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.077+0000 7f59c77fe700 1 -- 172.21.15.1:0/632044775 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751671485 0 0) 0x7f59c80ce8f0 con 0x7f59c815a1d0 2024-03-18T20:27:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.077+0000 7f59c7fff700 1 -- 172.21.15.1:0/632044775 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c80cdce0 con 0x7f59c8071d30 2024-03-18T20:27:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.077+0000 7f59c7fff700 1 --2- 172.21.15.1:0/632044775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c8071d30 0x7f59c80add20 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009840 tx=0x7f59b4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d71b1aef0f1a68 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.077+0000 7f59c77fe700 1 -- 172.21.15.1:0/632044775 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b4011830 con 0x7f59c8071d30 2024-03-18T20:27:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.077+0000 7f59c77fe700 1 -- 172.21.15.1:0/632044775 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b4011990 con 0x7f59c8071d30 2024-03-18T20:27:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.077+0000 7f59c77fe700 1 -- 172.21.15.1:0/632044775 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b4011d20 con 0x7f59c8071d30 2024-03-18T20:27:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.078+0000 7f59cf214700 1 -- 172.21.15.1:0/632044775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c8071d30 msgr2=0x7f59c80add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.078+0000 7f59cf214700 1 --2- 172.21.15.1:0/632044775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c8071d30 0x7f59c80add20 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009840 tx=0x7f59b4009b50 comp rx=0 tx=0).stop 2024-03-18T20:27:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.078+0000 7f59cf214700 1 -- 172.21.15.1:0/632044775 shutdown_connections 2024-03-18T20:27:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.078+0000 7f59cf214700 1 --2- 172.21.15.1:0/632044775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c8071d30 0x7f59c80add20 unknown :-1 s=CLOSED pgs=3004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.078+0000 7f59cf214700 1 -- 172.21.15.1:0/632044775 >> 172.21.15.1:0/632044775 conn(0x7f59c80cc9b0 msgr2=0x7f59c80ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.078+0000 7f59cf214700 1 -- 172.21.15.1:0/632044775 shutdown_connections 2024-03-18T20:27:09.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.078+0000 7f59cf214700 1 -- 172.21.15.1:0/632044775 wait complete. 2024-03-18T20:27:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.079+0000 7f59cf214700 1 Processor -- start 2024-03-18T20:27:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.079+0000 7f59cf214700 1 -- start start 2024-03-18T20:27:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.080+0000 7f59cf214700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59c8071d30 0x7f59c80c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.080+0000 7f59cf214700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59c80cb9e0 0x7f59c814a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.080+0000 7f59cf214700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c815a1d0 0x7f59c806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.080+0000 7f59cf214700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c80c8940 con 0x7f59c815a1d0 2024-03-18T20:27:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.080+0000 7f59cf214700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c814a9d0 con 0x7f59c80cb9e0 2024-03-18T20:27:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.080+0000 7f59cf214700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c80c7d30 con 0x7f59c8071d30 2024-03-18T20:27:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.080+0000 7f59cd7b1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c815a1d0 0x7f59c806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.080+0000 7f59cd7b1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c815a1d0 0x7f59c806ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54148/0 (socket says 172.21.15.1:54148) 2024-03-18T20:27:09.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.080+0000 7f59cd7b1700 1 -- 172.21.15.1:0/2790674466 learned_addr learned my addr 172.21.15.1:0/2790674466 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.080+0000 7f59c7fff700 1 --2- 172.21.15.1:0/2790674466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59c80cb9e0 0x7f59c814a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.080+0000 7f59ccfb0700 1 --2- 172.21.15.1:0/2790674466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59c8071d30 0x7f59c80c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.080+0000 7f59cd7b1700 1 -- 172.21.15.1:0/2790674466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59c8071d30 msgr2=0x7f59c80c8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.080+0000 7f59cd7b1700 1 --2- 172.21.15.1:0/2790674466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59c8071d30 0x7f59c80c8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.080+0000 7f59cd7b1700 1 -- 172.21.15.1:0/2790674466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59c80cb9e0 msgr2=0x7f59c814a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.080+0000 7f59cd7b1700 1 --2- 172.21.15.1:0/2790674466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59c80cb9e0 0x7f59c814a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.080+0000 7f59cd7b1700 1 -- 172.21.15.1:0/2790674466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b40094a0 con 0x7f59c815a1d0 2024-03-18T20:27:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.080+0000 7f59ccfb0700 1 --2- 172.21.15.1:0/2790674466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59c8071d30 0x7f59c80c8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.081+0000 7f59cd7b1700 1 --2- 172.21.15.1:0/2790674466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c815a1d0 0x7f59c806ec00 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7f59c000cc50 tx=0x7f59c000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.081+0000 7f59c57fa700 1 -- 172.21.15.1:0/2790674466 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c0003c00 con 0x7f59c815a1d0 2024-03-18T20:27:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.081+0000 7f59c57fa700 1 -- 172.21.15.1:0/2790674466 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c0003d60 con 0x7f59c815a1d0 2024-03-18T20:27:09.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.081+0000 7f59c57fa700 1 -- 172.21.15.1:0/2790674466 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c000aba0 con 0x7f59c815a1d0 2024-03-18T20:27:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.081+0000 7f59cf214700 1 -- 172.21.15.1:0/2790674466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c80c8010 con 0x7f59c815a1d0 2024-03-18T20:27:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.081+0000 7f59cf214700 1 -- 172.21.15.1:0/2790674466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c80c85d0 con 0x7f59c815a1d0 2024-03-18T20:27:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.082+0000 7f59cf214700 1 -- 172.21.15.1:0/2790674466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c80676a0 con 0x7f59c815a1d0 2024-03-18T20:27:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.082+0000 7f59c57fa700 1 -- 172.21.15.1:0/2790674466 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f59c000ad00 con 0x7f59c815a1d0 2024-03-18T20:27:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.083+0000 7f59c57fa700 1 --2- 172.21.15.1:0/2790674466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f59ac06aed0 0x7f59ac06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.083+0000 7f59c57fa700 1 -- 172.21.15.1:0/2790674466 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 7001+0+0 (secure 0 0 0) 0x7f59c0089ce0 con 0x7f59c815a1d0 2024-03-18T20:27:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.083+0000 7f59ccfb0700 1 --2- 172.21.15.1:0/2790674466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f59ac06aed0 0x7f59ac06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:09.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.083+0000 7f59ccfb0700 1 --2- 172.21.15.1:0/2790674466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f59ac06aed0 0x7f59ac06d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f59c80ad1f0 tx=0x7f59bc009040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:09.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.087+0000 7f59c57fa700 1 -- 172.21.15.1:0/2790674466 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f59c00546e0 con 0x7f59c815a1d0 2024-03-18T20:27:09.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.340+0000 7f59cf214700 1 -- 172.21.15.1:0/2790674466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7f59c815dd90 con 0x7f59c815a1d0 2024-03-18T20:27:09.345 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-18T20:27:09.341+0000 7fbdbeb46700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T20:27:09.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.350+0000 7f59c57fa700 1 -- 172.21.15.1:0/2790674466 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v129) v1 ==== 167+0+0 (secure 0 0 0) 0x7f59c815dd90 con 0x7f59c815a1d0 2024-03-18T20:27:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.352+0000 7f59cf214700 1 -- 172.21.15.1:0/2790674466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f59ac06aed0 msgr2=0x7f59ac06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.352+0000 7f59cf214700 1 --2- 172.21.15.1:0/2790674466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f59ac06aed0 0x7f59ac06d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f59c80ad1f0 tx=0x7f59bc009040 comp rx=0 tx=0).stop 2024-03-18T20:27:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.352+0000 7f59cf214700 1 -- 172.21.15.1:0/2790674466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c815a1d0 msgr2=0x7f59c806ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.352+0000 7f59cf214700 1 --2- 172.21.15.1:0/2790674466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c815a1d0 0x7f59c806ec00 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7f59c000cc50 tx=0x7f59c000cf60 comp rx=0 tx=0).stop 2024-03-18T20:27:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.353+0000 7f59cf214700 1 -- 172.21.15.1:0/2790674466 shutdown_connections 2024-03-18T20:27:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.353+0000 7f59cf214700 1 --2- 172.21.15.1:0/2790674466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f59ac06aed0 0x7f59ac06d390 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.353+0000 7f59cf214700 1 --2- 172.21.15.1:0/2790674466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f59c8071d30 0x7f59c80c8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:09.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.353+0000 7f59cf214700 1 --2- 172.21.15.1:0/2790674466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59c815a1d0 0x7f59c806ec00 unknown :-1 s=CLOSED pgs=3005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.353+0000 7f59cf214700 1 --2- 172.21.15.1:0/2790674466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59c80cb9e0 0x7f59c814a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.353+0000 7f59cf214700 1 -- 172.21.15.1:0/2790674466 >> 172.21.15.1:0/2790674466 conn(0x7f59c80cc9b0 msgr2=0x7f59c806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.353+0000 7f59cf214700 1 -- 172.21.15.1:0/2790674466 shutdown_connections 2024-03-18T20:27:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.353+0000 7f59cf214700 1 -- 172.21.15.1:0/2790674466 wait complete. 2024-03-18T20:27:09.358 INFO:tasks.ceph.mds.b.smithi165.stderr: -1673> 2024-03-18T20:26:41.338+0000 7f6a4d30e700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T20:27:09.358 INFO:teuthology.orchestra.run.smithi001.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T20:27:09.369 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2024-03-18T20:27:09.385 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-03-18T20:27:09.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.493+0000 7fac55b40700 1 Processor -- start 2024-03-18T20:27:09.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.493+0000 7fac55b40700 1 -- start start 2024-03-18T20:27:09.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.494+0000 7fac55b40700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fac5015d500 0x7fac5006c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:09.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.494+0000 7fac55b40700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fac5006c7b0 0x7fac50099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:09.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.494+0000 7fac55b40700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5006f490 con 0x7fac500cb7b0 2024-03-18T20:27:09.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.494+0000 7fac55b40700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5006e880 con 0x7fac5006c7b0 2024-03-18T20:27:09.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.494+0000 7fac55b40700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5006eb60 con 0x7fac5015d500 2024-03-18T20:27:09.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.494+0000 7fac4effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fac5006c7b0 0x7fac50099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:09.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.494+0000 7fac4effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fac5006c7b0 0x7fac50099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54160/0 (socket says 172.21.15.1:54160) 2024-03-18T20:27:09.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.494+0000 7fac4effd700 1 -- 172.21.15.1:0/4272962209 learned_addr learned my addr 172.21.15.1:0/4272962209 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:09.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.494+0000 7fac4f7fe700 1 --2- 172.21.15.1:0/4272962209 >> v2:172.21.15.165:3300/0 conn(0x7fac5015d500 0x7fac5006c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:09.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.494+0000 7fac4effd700 1 -- 172.21.15.1:0/4272962209 >> v2:172.21.15.165:3300/0 conn(0x7fac5015d500 msgr2=0x7fac5006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:09.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.494+0000 7fac4effd700 1 --2- 172.21.15.1:0/4272962209 >> v2:172.21.15.165:3300/0 conn(0x7fac5015d500 0x7fac5006c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:09.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.494+0000 7fac4effd700 1 -- 172.21.15.1:0/4272962209 >> v1:172.21.15.165:6789/0 conn(0x7fac500cb7b0 legacy=0x7fac500cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:09.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.494+0000 7fac4effd700 1 -- 172.21.15.1:0/4272962209 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5006ee40 con 0x7fac5006c7b0 2024-03-18T20:27:09.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.494+0000 7fac4effd700 1 --2- 172.21.15.1:0/4272962209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac5006c7b0 0x7fac50099a40 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 crypto rx=0x7fac50072160 tx=0x7fac38009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d10998dc56026c26 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:09.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.495+0000 7fac4e7fc700 1 -- 172.21.15.1:0/4272962209 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac38011830 con 0x7fac5006c7b0 2024-03-18T20:27:09.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.495+0000 7fac4e7fc700 1 -- 172.21.15.1:0/4272962209 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac38011990 con 0x7fac5006c7b0 2024-03-18T20:27:09.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.495+0000 7fac4e7fc700 1 -- 172.21.15.1:0/4272962209 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac38011d20 con 0x7fac5006c7b0 2024-03-18T20:27:09.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.495+0000 7fac55b40700 1 -- 172.21.15.1:0/4272962209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac5006c7b0 msgr2=0x7fac50099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:09.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.495+0000 7fac55b40700 1 --2- 172.21.15.1:0/4272962209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac5006c7b0 0x7fac50099a40 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 crypto rx=0x7fac50072160 tx=0x7fac38009bd0 comp rx=0 tx=0).stop 2024-03-18T20:27:09.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.495+0000 7fac55b40700 1 -- 172.21.15.1:0/4272962209 shutdown_connections 2024-03-18T20:27:09.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.495+0000 7fac55b40700 1 --2- 172.21.15.1:0/4272962209 >> v2:172.21.15.165:3300/0 conn(0x7fac5015d500 0x7fac5006c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:09.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.495+0000 7fac55b40700 1 --2- 172.21.15.1:0/4272962209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac5006c7b0 0x7fac50099a40 unknown :-1 s=CLOSED pgs=3007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:09.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.495+0000 7fac55b40700 1 -- 172.21.15.1:0/4272962209 >> 172.21.15.1:0/4272962209 conn(0x7fac500b2c50 msgr2=0x7fac500b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:09.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.495+0000 7fac55b40700 1 -- 172.21.15.1:0/4272962209 shutdown_connections 2024-03-18T20:27:09.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.495+0000 7fac55b40700 1 -- 172.21.15.1:0/4272962209 wait complete. 2024-03-18T20:27:09.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.496+0000 7fac55b40700 1 Processor -- start 2024-03-18T20:27:09.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.497+0000 7fac55b40700 1 -- start start 2024-03-18T20:27:09.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.497+0000 7fac55b40700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac5006c7b0 0x7fac501524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:09.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.497+0000 7fac55b40700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac500cb7b0 0x7fac5015ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:09.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.497+0000 7fac55b40700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac5015d500 0x7fac500cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:09.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.497+0000 7fac55b40700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5015e430 con 0x7fac500cb7b0 2024-03-18T20:27:09.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.497+0000 7fac55b40700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5015db00 con 0x7fac5015d500 2024-03-18T20:27:09.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.497+0000 7fac55b40700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5015dde0 con 0x7fac5006c7b0 2024-03-18T20:27:09.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.498+0000 7fac4effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac500cb7b0 0x7fac5015ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:09.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.498+0000 7fac4effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac500cb7b0 0x7fac5015ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54168/0 (socket says 172.21.15.1:54168) 2024-03-18T20:27:09.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.498+0000 7fac4effd700 1 -- 172.21.15.1:0/563334769 learned_addr learned my addr 172.21.15.1:0/563334769 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:09.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.498+0000 7fac4ffff700 1 --2- 172.21.15.1:0/563334769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac5015d500 0x7fac500cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:09.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.498+0000 7fac4f7fe700 1 --2- 172.21.15.1:0/563334769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac5006c7b0 0x7fac501524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:09.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.498+0000 7fac4effd700 1 -- 172.21.15.1:0/563334769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac5006c7b0 msgr2=0x7fac501524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:09.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.498+0000 7fac4effd700 1 --2- 172.21.15.1:0/563334769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac5006c7b0 0x7fac501524e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:09.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.498+0000 7fac4effd700 1 -- 172.21.15.1:0/563334769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac5015d500 msgr2=0x7fac500cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:09.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.498+0000 7fac4effd700 1 --2- 172.21.15.1:0/563334769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac5015d500 0x7fac500cccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:09.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.498+0000 7fac4effd700 1 -- 172.21.15.1:0/563334769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac38009550 con 0x7fac500cb7b0 2024-03-18T20:27:09.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.498+0000 7fac4ffff700 1 --2- 172.21.15.1:0/563334769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac5015d500 0x7fac500cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:09.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.498+0000 7fac4effd700 1 --2- 172.21.15.1:0/563334769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac500cb7b0 0x7fac5015ec40 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7fac38000c00 tx=0x7fac38004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:09.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.498+0000 7fac47fff700 1 -- 172.21.15.1:0/563334769 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac38004480 con 0x7fac500cb7b0 2024-03-18T20:27:09.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.498+0000 7fac47fff700 1 -- 172.21.15.1:0/563334769 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3801f070 con 0x7fac500cb7b0 2024-03-18T20:27:09.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.498+0000 7fac55b40700 1 -- 172.21.15.1:0/563334769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5015e0c0 con 0x7fac500cb7b0 2024-03-18T20:27:09.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.498+0000 7fac47fff700 1 -- 172.21.15.1:0/563334769 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac380038e0 con 0x7fac500cb7b0 2024-03-18T20:27:09.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.498+0000 7fac55b40700 1 -- 172.21.15.1:0/563334769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5015f700 con 0x7fac500cb7b0 2024-03-18T20:27:09.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.499+0000 7fac55b40700 1 -- 172.21.15.1:0/563334769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5006d690 con 0x7fac500cb7b0 2024-03-18T20:27:09.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.500+0000 7fac47fff700 1 -- 172.21.15.1:0/563334769 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fac38021030 con 0x7fac500cb7b0 2024-03-18T20:27:09.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.500+0000 7fac47fff700 1 --2- 172.21.15.1:0/563334769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fac3006aed0 0x7fac3006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:09.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.500+0000 7fac47fff700 1 -- 172.21.15.1:0/563334769 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 7087+0+0 (secure 0 0 0) 0x7fac3808b120 con 0x7fac500cb7b0 2024-03-18T20:27:09.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.500+0000 7fac4f7fe700 1 --2- 172.21.15.1:0/563334769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fac3006aed0 0x7fac3006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:09.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.501+0000 7fac4f7fe700 1 --2- 172.21.15.1:0/563334769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fac3006aed0 0x7fac3006d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fac40009ad0 tx=0x7fac40008040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:09.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.505+0000 7fac47fff700 1 -- 172.21.15.1:0/563334769 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fac38055bf0 con 0x7fac500cb7b0 2024-03-18T20:27:09.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:09.752+0000 7fac55b40700 1 -- 172.21.15.1:0/563334769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fac500676a0 con 0x7fac500cb7b0 2024-03-18T20:27:10.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.353+0000 7fac47fff700 1 -- 172.21.15.1:0/563334769 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v130) v1 ==== 107+0+0 (secure 0 0 0) 0x7fac38059210 con 0x7fac500cb7b0 2024-03-18T20:27:10.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.356+0000 7fac55b40700 1 -- 172.21.15.1:0/563334769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fac3006aed0 msgr2=0x7fac3006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:10.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.356+0000 7fac55b40700 1 --2- 172.21.15.1:0/563334769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fac3006aed0 0x7fac3006d390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fac40009ad0 tx=0x7fac40008040 comp rx=0 tx=0).stop 2024-03-18T20:27:10.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.356+0000 7fac55b40700 1 -- 172.21.15.1:0/563334769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac500cb7b0 msgr2=0x7fac5015ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:10.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.356+0000 7fac55b40700 1 --2- 172.21.15.1:0/563334769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac500cb7b0 0x7fac5015ec40 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7fac38000c00 tx=0x7fac38004750 comp rx=0 tx=0).stop 2024-03-18T20:27:10.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.356+0000 7fac55b40700 1 -- 172.21.15.1:0/563334769 shutdown_connections 2024-03-18T20:27:10.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.356+0000 7fac55b40700 1 --2- 172.21.15.1:0/563334769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fac3006aed0 0x7fac3006d390 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:10.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.356+0000 7fac55b40700 1 --2- 172.21.15.1:0/563334769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fac5006c7b0 0x7fac501524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:10.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.356+0000 7fac55b40700 1 --2- 172.21.15.1:0/563334769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac5015d500 0x7fac500cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:10.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.356+0000 7fac55b40700 1 --2- 172.21.15.1:0/563334769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac500cb7b0 0x7fac5015ec40 unknown :-1 s=CLOSED pgs=3008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:10.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.356+0000 7fac55b40700 1 -- 172.21.15.1:0/563334769 >> 172.21.15.1:0/563334769 conn(0x7fac500b2c50 msgr2=0x7fac500b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:10.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.356+0000 7fac55b40700 1 -- 172.21.15.1:0/563334769 shutdown_connections 2024-03-18T20:27:10.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.356+0000 7fac55b40700 1 -- 172.21.15.1:0/563334769 wait complete. 2024-03-18T20:27:10.375 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2024-03-18T20:27:10.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.507+0000 7f4b53a7c700 1 Processor -- start 2024-03-18T20:27:10.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.508+0000 7f4b53a7c700 1 -- start start 2024-03-18T20:27:10.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.508+0000 7f4b53a7c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b4c15e8d0 0x7f4b4c15a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:10.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.508+0000 7f4b53a7c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4b4c15dca0 0x7f4b4c0b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:10.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.508+0000 7f4b53a7c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b4c15ac60 con 0x7f4b4c0b3d60 2024-03-18T20:27:10.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.508+0000 7f4b53a7c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b4c0701d0 con 0x7f4b4c15dca0 2024-03-18T20:27:10.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.508+0000 7f4b53a7c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c0b30b0 con 0x7f4b4c15e8d0 2024-03-18T20:27:10.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.509+0000 7f4b52019700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4b4c0b3d60 0x7f4b4c0717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49334/0 (socket says 172.21.15.1:49334) 2024-03-18T20:27:10.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.509+0000 7f4b52019700 1 -- 172.21.15.1:0/128700262 learned_addr learned my addr 172.21.15.1:0/128700262 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:10.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.509+0000 7f4b51017700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4b4c15dca0 0x7f4b4c0b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:10.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.509+0000 7f4b51818700 1 --2- 172.21.15.1:0/128700262 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b4c15e8d0 0x7f4b4c15a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:10.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.509+0000 7f4b51017700 1 -- 172.21.15.1:0/128700262 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b4c15e8d0 msgr2=0x7f4b4c15a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:10.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.509+0000 7f4b51017700 1 --2- 172.21.15.1:0/128700262 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b4c15e8d0 0x7f4b4c15a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:10.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.509+0000 7f4b51017700 1 -- 172.21.15.1:0/128700262 >> v1:172.21.15.1:6789/0 conn(0x7f4b4c0b3d60 legacy=0x7f4b4c0717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:10.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.509+0000 7f4b51017700 1 -- 172.21.15.1:0/128700262 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b4c0b3390 con 0x7f4b4c15dca0 2024-03-18T20:27:10.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.509+0000 7f4b51017700 1 --2- 172.21.15.1:0/128700262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b4c15dca0 0x7f4b4c0b3820 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7f4b48009840 tx=0x7f4b48009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13862fd14f425cbd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:10.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.509+0000 7f4b50816700 1 -- 172.21.15.1:0/128700262 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b48011830 con 0x7f4b4c15dca0 2024-03-18T20:27:10.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.509+0000 7f4b50816700 1 -- 172.21.15.1:0/128700262 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b48011990 con 0x7f4b4c15dca0 2024-03-18T20:27:10.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.509+0000 7f4b50816700 1 -- 172.21.15.1:0/128700262 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b48011d20 con 0x7f4b4c15dca0 2024-03-18T20:27:10.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.510+0000 7f4b53a7c700 1 -- 172.21.15.1:0/128700262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b4c15dca0 msgr2=0x7f4b4c0b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:10.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.510+0000 7f4b53a7c700 1 --2- 172.21.15.1:0/128700262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b4c15dca0 0x7f4b4c0b3820 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7f4b48009840 tx=0x7f4b48009b50 comp rx=0 tx=0).stop 2024-03-18T20:27:10.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.510+0000 7f4b53a7c700 1 -- 172.21.15.1:0/128700262 shutdown_connections 2024-03-18T20:27:10.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.510+0000 7f4b53a7c700 1 --2- 172.21.15.1:0/128700262 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b4c15e8d0 0x7f4b4c15a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:10.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.510+0000 7f4b53a7c700 1 --2- 172.21.15.1:0/128700262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b4c15dca0 0x7f4b4c0b3820 unknown :-1 s=CLOSED pgs=3010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:10.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.510+0000 7f4b53a7c700 1 -- 172.21.15.1:0/128700262 >> 172.21.15.1:0/128700262 conn(0x7f4b4c0d0080 msgr2=0x7f4b4c0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:10.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.510+0000 7f4b53a7c700 1 -- 172.21.15.1:0/128700262 shutdown_connections 2024-03-18T20:27:10.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.510+0000 7f4b53a7c700 1 -- 172.21.15.1:0/128700262 wait complete. 2024-03-18T20:27:10.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.511+0000 7f4b53a7c700 1 Processor -- start 2024-03-18T20:27:10.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.511+0000 7f4b53a7c700 1 -- start start 2024-03-18T20:27:10.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.512+0000 7f4b53a7c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b4c0b3d60 0x7f4b4c154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:10.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.512+0000 7f4b53a7c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b4c15dca0 0x7f4b4c1504e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:10.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.512+0000 7f4b53a7c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b4c15e8d0 0x7f4b4c154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:10.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.512+0000 7f4b53a7c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c153610 con 0x7f4b4c0b3d60 2024-03-18T20:27:10.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.512+0000 7f4b53a7c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c153bb0 con 0x7f4b4c15e8d0 2024-03-18T20:27:10.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.512+0000 7f4b53a7c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c150a20 con 0x7f4b4c15dca0 2024-03-18T20:27:10.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.512+0000 7f4b51818700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b4c0b3d60 0x7f4b4c154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:10.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.512+0000 7f4b51818700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b4c0b3d60 0x7f4b4c154ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54182/0 (socket says 172.21.15.1:54182) 2024-03-18T20:27:10.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.512+0000 7f4b51818700 1 -- 172.21.15.1:0/3057768786 learned_addr learned my addr 172.21.15.1:0/3057768786 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:10.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.512+0000 7f4b51017700 1 --2- 172.21.15.1:0/3057768786 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b4c15dca0 0x7f4b4c1504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:10.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.512+0000 7f4b52019700 1 --2- 172.21.15.1:0/3057768786 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b4c15e8d0 0x7f4b4c154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:10.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.512+0000 7f4b51818700 1 -- 172.21.15.1:0/3057768786 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b4c15dca0 msgr2=0x7f4b4c1504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:10.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.512+0000 7f4b51818700 1 --2- 172.21.15.1:0/3057768786 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b4c15dca0 0x7f4b4c1504e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:10.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.512+0000 7f4b51818700 1 -- 172.21.15.1:0/3057768786 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b4c15e8d0 msgr2=0x7f4b4c154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:10.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.512+0000 7f4b51818700 1 --2- 172.21.15.1:0/3057768786 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b4c15e8d0 0x7f4b4c154040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:10.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.512+0000 7f4b51818700 1 -- 172.21.15.1:0/3057768786 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b480094a0 con 0x7f4b4c0b3d60 2024-03-18T20:27:10.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.512+0000 7f4b51818700 1 --2- 172.21.15.1:0/3057768786 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b4c0b3d60 0x7f4b4c154ea0 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00e810 tx=0x7f4b3c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:10.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.512+0000 7f4b427fc700 1 -- 172.21.15.1:0/3057768786 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c013610 con 0x7f4b4c0b3d60 2024-03-18T20:27:10.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.513+0000 7f4b427fc700 1 -- 172.21.15.1:0/3057768786 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3c013770 con 0x7f4b4c0b3d60 2024-03-18T20:27:10.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.513+0000 7f4b53a7c700 1 -- 172.21.15.1:0/3057768786 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b4c154580 con 0x7f4b4c0b3d60 2024-03-18T20:27:10.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.513+0000 7f4b427fc700 1 -- 172.21.15.1:0/3057768786 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c013b20 con 0x7f4b4c0b3d60 2024-03-18T20:27:10.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.513+0000 7f4b53a7c700 1 -- 172.21.15.1:0/3057768786 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b4c15a170 con 0x7f4b4c0b3d60 2024-03-18T20:27:10.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.514+0000 7f4b53a7c700 1 -- 172.21.15.1:0/3057768786 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b4c0cd070 con 0x7f4b4c0b3d60 2024-03-18T20:27:10.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.514+0000 7f4b427fc700 1 -- 172.21.15.1:0/3057768786 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f4b3c013c80 con 0x7f4b4c0b3d60 2024-03-18T20:27:10.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.514+0000 7f4b427fc700 1 --2- 172.21.15.1:0/3057768786 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4b3006aed0 0x7f4b3006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:10.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.514+0000 7f4b427fc700 1 -- 172.21.15.1:0/3057768786 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(263..263 src has 1..263) v4 ==== 7087+0+0 (secure 0 0 0) 0x7f4b3c089a30 con 0x7f4b4c0b3d60 2024-03-18T20:27:10.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.514+0000 7f4b51017700 1 --2- 172.21.15.1:0/3057768786 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4b3006aed0 0x7f4b3006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:10.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.515+0000 7f4b51017700 1 --2- 172.21.15.1:0/3057768786 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4b3006aed0 0x7f4b3006d390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4b480090b0 tx=0x7f4b4801c040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:10.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.519+0000 7f4b427fc700 1 -- 172.21.15.1:0/3057768786 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4b3c054250 con 0x7f4b4c0b3d60 2024-03-18T20:27:10.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:10.723+0000 7f4b53a7c700 1 -- 172.21.15.1:0/3057768786 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4b4c153e90 con 0x7f4b4c0b3d60 2024-03-18T20:27:11.344 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:27:11.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:27:11.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.588+0000 7f4b427fc700 1 -- 172.21.15.1:0/3057768786 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v264) v1 ==== 190+0+0 (secure 0 0 0) 0x7f4b3c057870 con 0x7f4b4c0b3d60 2024-03-18T20:27:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.591+0000 7f4b53a7c700 1 -- 172.21.15.1:0/3057768786 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4b3006aed0 msgr2=0x7f4b3006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.591+0000 7f4b53a7c700 1 --2- 172.21.15.1:0/3057768786 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4b3006aed0 0x7f4b3006d390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4b480090b0 tx=0x7f4b4801c040 comp rx=0 tx=0).stop 2024-03-18T20:27:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.591+0000 7f4b53a7c700 1 -- 172.21.15.1:0/3057768786 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b4c0b3d60 msgr2=0x7f4b4c154ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.591+0000 7f4b53a7c700 1 --2- 172.21.15.1:0/3057768786 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b4c0b3d60 0x7f4b4c154ea0 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00e810 tx=0x7f4b3c00ebd0 comp rx=0 tx=0).stop 2024-03-18T20:27:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.591+0000 7f4b53a7c700 1 -- 172.21.15.1:0/3057768786 shutdown_connections 2024-03-18T20:27:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.591+0000 7f4b53a7c700 1 --2- 172.21.15.1:0/3057768786 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4b3006aed0 0x7f4b3006d390 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.591+0000 7f4b53a7c700 1 --2- 172.21.15.1:0/3057768786 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b4c15e8d0 0x7f4b4c154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.591+0000 7f4b53a7c700 1 --2- 172.21.15.1:0/3057768786 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b4c0b3d60 0x7f4b4c154ea0 unknown :-1 s=CLOSED pgs=3011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.591+0000 7f4b53a7c700 1 --2- 172.21.15.1:0/3057768786 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4b4c15dca0 0x7f4b4c1504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:11.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.591+0000 7f4b53a7c700 1 -- 172.21.15.1:0/3057768786 >> 172.21.15.1:0/3057768786 conn(0x7f4b4c0d0080 msgr2=0x7f4b4c0717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:11.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.591+0000 7f4b53a7c700 1 -- 172.21.15.1:0/3057768786 shutdown_connections 2024-03-18T20:27:11.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.591+0000 7f4b53a7c700 1 -- 172.21.15.1:0/3057768786 wait complete. 2024-03-18T20:27:11.596 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'backup_fs_metadata' removed 2024-03-18T20:27:11.613 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2024-03-18T20:27:11.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.747+0000 7f6c5a120700 1 Processor -- start 2024-03-18T20:27:11.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.748+0000 7f6c5a120700 1 -- start start 2024-03-18T20:27:11.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.748+0000 7f6c5a120700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6c5415b550 0x7f6c54099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:11.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.748+0000 7f6c5a120700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6c5409a600 0x7f6c54099f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:11.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.748+0000 7f6c5a120700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c540ae6f0 0x7f6c540aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:11.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.748+0000 7f6c5a120700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c540ae260 con 0x7f6c5415b550 2024-03-18T20:27:11.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.748+0000 7f6c5a120700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c540d0150 con 0x7f6c5409a600 2024-03-18T20:27:11.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.748+0000 7f6c5a120700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c540cfb90 con 0x7f6c540ae6f0 2024-03-18T20:27:11.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.748+0000 7f6c537fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6c5415b550 0x7f6c54099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:11.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.749+0000 7f6c537fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6c5415b550 0x7f6c54099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54196/0 (socket says 172.21.15.1:54196) 2024-03-18T20:27:11.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.749+0000 7f6c537fe700 1 -- 172.21.15.1:0/4271775494 learned_addr learned my addr 172.21.15.1:0/4271775494 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:11.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.749+0000 7f6c52ffd700 1 --2- 172.21.15.1:0/4271775494 >> v2:172.21.15.165:3300/0 conn(0x7f6c5409a600 0x7f6c54099f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:11.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.749+0000 7f6c53fff700 1 --2- 172.21.15.1:0/4271775494 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c540ae6f0 0x7f6c540aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:11.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.749+0000 7f6c537fe700 1 -- 172.21.15.1:0/4271775494 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c540ae6f0 msgr2=0x7f6c540aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:11.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.749+0000 7f6c537fe700 1 --2- 172.21.15.1:0/4271775494 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c540ae6f0 0x7f6c540aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:11.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.749+0000 7f6c537fe700 1 -- 172.21.15.1:0/4271775494 >> v2:172.21.15.165:3300/0 conn(0x7f6c5409a600 msgr2=0x7f6c54099f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:11.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.749+0000 7f6c537fe700 1 --2- 172.21.15.1:0/4271775494 >> v2:172.21.15.165:3300/0 conn(0x7f6c5409a600 0x7f6c54099f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:11.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.749+0000 7f6c537fe700 1 -- 172.21.15.1:0/4271775494 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c540cfe70 con 0x7f6c5415b550 2024-03-18T20:27:11.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.749+0000 7f6c53fff700 1 --2- 172.21.15.1:0/4271775494 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c540ae6f0 0x7f6c540aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:11.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.749+0000 7f6c537fe700 1 --2- 172.21.15.1:0/4271775494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c5415b550 0x7f6c54099a40 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7f6c540c4c40 tx=0x7f6c44009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d0c93ead557349 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:11.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.749+0000 7f6c527fc700 1 -- 172.21.15.1:0/4271775494 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4400b870 con 0x7f6c5415b550 2024-03-18T20:27:11.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.749+0000 7f6c527fc700 1 -- 172.21.15.1:0/4271775494 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4400b9d0 con 0x7f6c5415b550 2024-03-18T20:27:11.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.749+0000 7f6c527fc700 1 -- 172.21.15.1:0/4271775494 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4400bd60 con 0x7f6c5415b550 2024-03-18T20:27:11.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.750+0000 7f6c5a120700 1 -- 172.21.15.1:0/4271775494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c5415b550 msgr2=0x7f6c54099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:11.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.750+0000 7f6c5a120700 1 --2- 172.21.15.1:0/4271775494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c5415b550 0x7f6c54099a40 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7f6c540c4c40 tx=0x7f6c44009bd0 comp rx=0 tx=0).stop 2024-03-18T20:27:11.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.750+0000 7f6c5a120700 1 -- 172.21.15.1:0/4271775494 shutdown_connections 2024-03-18T20:27:11.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.750+0000 7f6c5a120700 1 --2- 172.21.15.1:0/4271775494 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c540ae6f0 0x7f6c540aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:11.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.750+0000 7f6c5a120700 1 --2- 172.21.15.1:0/4271775494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c5415b550 0x7f6c54099a40 unknown :-1 s=CLOSED pgs=3012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:11.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.750+0000 7f6c5a120700 1 --2- 172.21.15.1:0/4271775494 >> v2:172.21.15.165:3300/0 conn(0x7f6c5409a600 0x7f6c54099f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:11.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.750+0000 7f6c5a120700 1 -- 172.21.15.1:0/4271775494 >> 172.21.15.1:0/4271775494 conn(0x7f6c540b2c30 msgr2=0x7f6c540b3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:11.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.750+0000 7f6c5a120700 1 -- 172.21.15.1:0/4271775494 shutdown_connections 2024-03-18T20:27:11.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.750+0000 7f6c5a120700 1 -- 172.21.15.1:0/4271775494 wait complete. 2024-03-18T20:27:11.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.751+0000 7f6c5a120700 1 Processor -- start 2024-03-18T20:27:11.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.751+0000 7f6c5a120700 1 -- start start 2024-03-18T20:27:11.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.752+0000 7f6c5a120700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c5409a600 0x7f6c54159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:11.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.752+0000 7f6c5a120700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c540ae6f0 0x7f6c540afdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:11.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.752+0000 7f6c5a120700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c5415b550 0x7f6c540b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:11.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.752+0000 7f6c5a120700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c540b1330 con 0x7f6c5409a600 2024-03-18T20:27:11.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.752+0000 7f6c5a120700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5415aac0 con 0x7f6c540ae6f0 2024-03-18T20:27:11.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.752+0000 7f6c5a120700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c540af950 con 0x7f6c5415b550 2024-03-18T20:27:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.752+0000 7f6c537fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c5409a600 0x7f6c54159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.752+0000 7f6c537fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c5409a600 0x7f6c54159fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54210/0 (socket says 172.21.15.1:54210) 2024-03-18T20:27:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.752+0000 7f6c537fe700 1 -- 172.21.15.1:0/476529336 learned_addr learned my addr 172.21.15.1:0/476529336 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.752+0000 7f6c53fff700 1 --2- 172.21.15.1:0/476529336 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c5415b550 0x7f6c540b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.752+0000 7f6c52ffd700 1 --2- 172.21.15.1:0/476529336 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c540ae6f0 0x7f6c540afdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.752+0000 7f6c537fe700 1 -- 172.21.15.1:0/476529336 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c5415b550 msgr2=0x7f6c540b3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.752+0000 7f6c537fe700 1 --2- 172.21.15.1:0/476529336 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c5415b550 0x7f6c540b3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.752+0000 7f6c537fe700 1 -- 172.21.15.1:0/476529336 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c540ae6f0 msgr2=0x7f6c540afdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.752+0000 7f6c537fe700 1 --2- 172.21.15.1:0/476529336 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c540ae6f0 0x7f6c540afdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.752+0000 7f6c537fe700 1 -- 172.21.15.1:0/476529336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c44009550 con 0x7f6c5409a600 2024-03-18T20:27:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.752+0000 7f6c52ffd700 1 --2- 172.21.15.1:0/476529336 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c540ae6f0 0x7f6c540afdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.752+0000 7f6c53fff700 1 --2- 172.21.15.1:0/476529336 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c5415b550 0x7f6c540b3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.752+0000 7f6c537fe700 1 --2- 172.21.15.1:0/476529336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c5409a600 0x7f6c54159fd0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7f6c44014040 tx=0x7f6c44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:11.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.753+0000 7f6c4bfff700 1 -- 172.21.15.1:0/476529336 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c44004520 con 0x7f6c5409a600 2024-03-18T20:27:11.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.753+0000 7f6c4bfff700 1 -- 172.21.15.1:0/476529336 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4401e070 con 0x7f6c5409a600 2024-03-18T20:27:11.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.753+0000 7f6c5a120700 1 -- 172.21.15.1:0/476529336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c540c9800 con 0x7f6c5409a600 2024-03-18T20:27:11.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.753+0000 7f6c4bfff700 1 -- 172.21.15.1:0/476529336 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c440038c0 con 0x7f6c5409a600 2024-03-18T20:27:11.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.753+0000 7f6c5a120700 1 -- 172.21.15.1:0/476529336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c540c9dc0 con 0x7f6c5409a600 2024-03-18T20:27:11.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.754+0000 7f6c5a120700 1 -- 172.21.15.1:0/476529336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c540676a0 con 0x7f6c5409a600 2024-03-18T20:27:11.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.755+0000 7f6c4bfff700 1 -- 172.21.15.1:0/476529336 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f6c44003a20 con 0x7f6c5409a600 2024-03-18T20:27:11.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.755+0000 7f6c4bfff700 1 --2- 172.21.15.1:0/476529336 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6c3406aed0 0x7f6c3406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:11.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.755+0000 7f6c4bfff700 1 -- 172.21.15.1:0/476529336 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(264..264 src has 1..264) v4 ==== 6643+0+0 (secure 0 0 0) 0x7f6c44020070 con 0x7f6c5409a600 2024-03-18T20:27:11.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.755+0000 7f6c52ffd700 1 --2- 172.21.15.1:0/476529336 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6c3406aed0 0x7f6c3406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:11.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.757+0000 7f6c52ffd700 1 --2- 172.21.15.1:0/476529336 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6c3406aed0 0x7f6c3406d390 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6c5406da40 tx=0x7f6c3c008040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:11.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.762+0000 7f6c4bfff700 1 -- 172.21.15.1:0/476529336 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6c44056030 con 0x7f6c5409a600 2024-03-18T20:27:11.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:11.969+0000 7f6c5a120700 1 -- 172.21.15.1:0/476529336 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6c540c4a20 con 0x7f6c5409a600 2024-03-18T20:27:12.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.596+0000 7f6c4bfff700 1 -- 172.21.15.1:0/476529336 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v265) v1 ==== 178+0+0 (secure 0 0 0) 0x7f6c44059650 con 0x7f6c5409a600 2024-03-18T20:27:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.600+0000 7f6c5a120700 1 -- 172.21.15.1:0/476529336 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6c3406aed0 msgr2=0x7f6c3406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.600+0000 7f6c5a120700 1 --2- 172.21.15.1:0/476529336 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6c3406aed0 0x7f6c3406d390 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6c5406da40 tx=0x7f6c3c008040 comp rx=0 tx=0).stop 2024-03-18T20:27:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.600+0000 7f6c5a120700 1 -- 172.21.15.1:0/476529336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c5409a600 msgr2=0x7f6c54159fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.600+0000 7f6c5a120700 1 --2- 172.21.15.1:0/476529336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c5409a600 0x7f6c54159fd0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7f6c44014040 tx=0x7f6c44004750 comp rx=0 tx=0).stop 2024-03-18T20:27:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.600+0000 7f6c5a120700 1 -- 172.21.15.1:0/476529336 shutdown_connections 2024-03-18T20:27:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.600+0000 7f6c5a120700 1 --2- 172.21.15.1:0/476529336 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6c3406aed0 0x7f6c3406d390 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.600+0000 7f6c5a120700 1 --2- 172.21.15.1:0/476529336 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6c5415b550 0x7f6c540b3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.600+0000 7f6c5a120700 1 --2- 172.21.15.1:0/476529336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c5409a600 0x7f6c54159fd0 unknown :-1 s=CLOSED pgs=3013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.600+0000 7f6c5a120700 1 --2- 172.21.15.1:0/476529336 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c540ae6f0 0x7f6c540afdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.600+0000 7f6c5a120700 1 -- 172.21.15.1:0/476529336 >> 172.21.15.1:0/476529336 conn(0x7f6c540b2c30 msgr2=0x7f6c540c7100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.601+0000 7f6c5a120700 1 -- 172.21.15.1:0/476529336 shutdown_connections 2024-03-18T20:27:12.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.601+0000 7f6c5a120700 1 -- 172.21.15.1:0/476529336 wait complete. 2024-03-18T20:27:12.606 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'backup_fs_data' removed 2024-03-18T20:27:12.621 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-18T20:27:12.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.751+0000 7fb269afe700 1 Processor -- start 2024-03-18T20:27:12.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.751+0000 7fb269afe700 1 -- start start 2024-03-18T20:27:12.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.751+0000 7fb269afe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb26406ca80 0x7fb26415d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:12.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.751+0000 7fb269afe700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb26409a150 0x7fb2640cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:12.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.751+0000 7fb269afe700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2640b3b00 con 0x7fb26415d5b0 2024-03-18T20:27:12.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.751+0000 7fb269afe700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2640cbbb0 con 0x7fb26409a150 2024-03-18T20:27:12.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.751+0000 7fb269afe700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2640cf7f0 con 0x7fb26406ca80 2024-03-18T20:27:12.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.752+0000 7fb263fff700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb26409a150 0x7fb2640cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:12.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.752+0000 7fb262ffd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb26406ca80 0x7fb26415d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:12.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.752+0000 7fb2637fe700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fb26415d5b0 0x7fb264099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:58114/0 (socket says 172.21.15.1:58114) 2024-03-18T20:27:12.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.752+0000 7fb2637fe700 1 -- 172.21.15.1:0/3984822658 learned_addr learned my addr 172.21.15.1:0/3984822658 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:12.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.752+0000 7fb262ffd700 1 -- 172.21.15.1:0/3984822658 >> v2:172.21.15.165:3300/0 conn(0x7fb26409a150 msgr2=0x7fb2640cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:12.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.752+0000 7fb262ffd700 1 --2- 172.21.15.1:0/3984822658 >> v2:172.21.15.165:3300/0 conn(0x7fb26409a150 0x7fb2640cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:12.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.752+0000 7fb262ffd700 1 -- 172.21.15.1:0/3984822658 >> v1:172.21.15.165:6789/0 conn(0x7fb26415d5b0 legacy=0x7fb264099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:12.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.752+0000 7fb262ffd700 1 -- 172.21.15.1:0/3984822658 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2640d04d0 con 0x7fb26406ca80 2024-03-18T20:27:12.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.752+0000 7fb263fff700 1 --2- 172.21.15.1:0/3984822658 >> v2:172.21.15.165:3300/0 conn(0x7fb26409a150 0x7fb2640cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:27:12.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.752+0000 7fb262ffd700 1 --2- 172.21.15.1:0/3984822658 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb26406ca80 0x7fb26415d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb258009840 tx=0x7fb258009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40f5014ba67d4b8d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:12.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.753+0000 7fb2627fc700 1 -- 172.21.15.1:0/3984822658 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb258011830 con 0x7fb26406ca80 2024-03-18T20:27:12.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.753+0000 7fb2627fc700 1 -- 172.21.15.1:0/3984822658 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb258011990 con 0x7fb26406ca80 2024-03-18T20:27:12.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.753+0000 7fb2627fc700 1 -- 172.21.15.1:0/3984822658 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb258011d20 con 0x7fb26406ca80 2024-03-18T20:27:12.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.753+0000 7fb269afe700 1 -- 172.21.15.1:0/3984822658 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb26406ca80 msgr2=0x7fb26415d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:12.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.753+0000 7fb269afe700 1 --2- 172.21.15.1:0/3984822658 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb26406ca80 0x7fb26415d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb258009840 tx=0x7fb258009b50 comp rx=0 tx=0).stop 2024-03-18T20:27:12.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.753+0000 7fb269afe700 1 -- 172.21.15.1:0/3984822658 shutdown_connections 2024-03-18T20:27:12.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.753+0000 7fb269afe700 1 --2- 172.21.15.1:0/3984822658 >> v2:172.21.15.165:3300/0 conn(0x7fb26409a150 0x7fb2640cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:12.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.753+0000 7fb269afe700 1 --2- 172.21.15.1:0/3984822658 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb26406ca80 0x7fb26415d990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:12.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.753+0000 7fb269afe700 1 -- 172.21.15.1:0/3984822658 >> 172.21.15.1:0/3984822658 conn(0x7fb2640b2bd0 msgr2=0x7fb2640b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:12.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.753+0000 7fb269afe700 1 -- 172.21.15.1:0/3984822658 shutdown_connections 2024-03-18T20:27:12.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.753+0000 7fb269afe700 1 -- 172.21.15.1:0/3984822658 wait complete. 2024-03-18T20:27:12.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.754+0000 7fb269afe700 1 Processor -- start 2024-03-18T20:27:12.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.755+0000 7fb269afe700 1 -- start start 2024-03-18T20:27:12.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.755+0000 7fb269afe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb26406ca80 0x7fb2640b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:12.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.755+0000 7fb269afe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb26409a150 0x7fb264158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:12.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.755+0000 7fb269afe700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb26415d5b0 0x7fb2640c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:12.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.755+0000 7fb269afe700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2640b44e0 con 0x7fb26406ca80 2024-03-18T20:27:12.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.755+0000 7fb269afe700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2640b47c0 con 0x7fb26415d5b0 2024-03-18T20:27:12.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.755+0000 7fb269afe700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2640b4aa0 con 0x7fb26409a150 2024-03-18T20:27:12.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.755+0000 7fb2637fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb26406ca80 0x7fb2640b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:12.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.755+0000 7fb2637fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb26406ca80 0x7fb2640b3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54214/0 (socket says 172.21.15.1:54214) 2024-03-18T20:27:12.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.755+0000 7fb2637fe700 1 -- 172.21.15.1:0/260581547 learned_addr learned my addr 172.21.15.1:0/260581547 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:12.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.756+0000 7fb263fff700 1 --2- 172.21.15.1:0/260581547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb26415d5b0 0x7fb2640c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:12.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.756+0000 7fb262ffd700 1 --2- 172.21.15.1:0/260581547 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb26409a150 0x7fb264158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:12.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.756+0000 7fb2637fe700 1 -- 172.21.15.1:0/260581547 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb26409a150 msgr2=0x7fb264158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:12.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.756+0000 7fb2637fe700 1 --2- 172.21.15.1:0/260581547 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb26409a150 0x7fb264158bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:12.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.756+0000 7fb2637fe700 1 -- 172.21.15.1:0/260581547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb26415d5b0 msgr2=0x7fb2640c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:12.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.756+0000 7fb2637fe700 1 --2- 172.21.15.1:0/260581547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb26415d5b0 0x7fb2640c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:12.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.756+0000 7fb2637fe700 1 -- 172.21.15.1:0/260581547 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25c0094a0 con 0x7fb26406ca80 2024-03-18T20:27:12.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.756+0000 7fb262ffd700 1 --2- 172.21.15.1:0/260581547 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb26409a150 0x7fb264158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:12.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.756+0000 7fb263fff700 1 --2- 172.21.15.1:0/260581547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb26415d5b0 0x7fb2640c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:12.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.756+0000 7fb2637fe700 1 --2- 172.21.15.1:0/260581547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb26406ca80 0x7fb2640b3c90 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7fb25400cbc0 tx=0x7fb25400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:12.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.756+0000 7fb24bfff700 1 -- 172.21.15.1:0/260581547 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25400aaf0 con 0x7fb26406ca80 2024-03-18T20:27:12.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.756+0000 7fb24bfff700 1 -- 172.21.15.1:0/260581547 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25400ac50 con 0x7fb26406ca80 2024-03-18T20:27:12.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.756+0000 7fb269afe700 1 -- 172.21.15.1:0/260581547 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2580094a0 con 0x7fb26406ca80 2024-03-18T20:27:12.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.756+0000 7fb24bfff700 1 -- 172.21.15.1:0/260581547 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb254012400 con 0x7fb26406ca80 2024-03-18T20:27:12.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.756+0000 7fb269afe700 1 -- 172.21.15.1:0/260581547 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2640b5060 con 0x7fb26406ca80 2024-03-18T20:27:12.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.757+0000 7fb269afe700 1 -- 172.21.15.1:0/260581547 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb26406cf50 con 0x7fb26406ca80 2024-03-18T20:27:12.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.758+0000 7fb24bfff700 1 -- 172.21.15.1:0/260581547 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fb25401b3b0 con 0x7fb26406ca80 2024-03-18T20:27:12.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.758+0000 7fb24bfff700 1 --2- 172.21.15.1:0/260581547 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb25006aed0 0x7fb25006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:12.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.758+0000 7fb24bfff700 1 -- 172.21.15.1:0/260581547 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6223+0+0 (secure 0 0 0) 0x7fb254089570 con 0x7fb26406ca80 2024-03-18T20:27:12.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.758+0000 7fb262ffd700 1 --2- 172.21.15.1:0/260581547 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb25006aed0 0x7fb25006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:12.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.758+0000 7fb262ffd700 1 --2- 172.21.15.1:0/260581547 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb25006aed0 0x7fb25006d390 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb2640b5fc0 tx=0x7fb258008030 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:12.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.763+0000 7fb24bfff700 1 -- 172.21.15.1:0/260581547 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb2540542a0 con 0x7fb26406ca80 2024-03-18T20:27:12.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.973+0000 7fb269afe700 1 -- 172.21.15.1:0/260581547 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fb2640676a0 con 0x7fb26406ca80 2024-03-18T20:27:12.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.977+0000 7fb24bfff700 1 -- 172.21.15.1:0/260581547 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v85) v1 ==== 115+0+0 (secure 0 0 0) 0x7fb2540578c0 con 0x7fb26406ca80 2024-03-18T20:27:12.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.979+0000 7fb269afe700 1 -- 172.21.15.1:0/260581547 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb25006aed0 msgr2=0x7fb25006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:12.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.979+0000 7fb269afe700 1 --2- 172.21.15.1:0/260581547 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb25006aed0 0x7fb25006d390 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb2640b5fc0 tx=0x7fb258008030 comp rx=0 tx=0).stop 2024-03-18T20:27:12.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.979+0000 7fb269afe700 1 -- 172.21.15.1:0/260581547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb26406ca80 msgr2=0x7fb2640b3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:12.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.979+0000 7fb269afe700 1 --2- 172.21.15.1:0/260581547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb26406ca80 0x7fb2640b3c90 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7fb25400cbc0 tx=0x7fb25400cf80 comp rx=0 tx=0).stop 2024-03-18T20:27:12.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.979+0000 7fb269afe700 1 -- 172.21.15.1:0/260581547 shutdown_connections 2024-03-18T20:27:12.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.979+0000 7fb269afe700 1 --2- 172.21.15.1:0/260581547 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb25006aed0 0x7fb25006d390 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:12.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.979+0000 7fb269afe700 1 --2- 172.21.15.1:0/260581547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb26415d5b0 0x7fb2640c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:12.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.979+0000 7fb269afe700 1 --2- 172.21.15.1:0/260581547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb26406ca80 0x7fb2640b3c90 unknown :-1 s=CLOSED pgs=3014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:12.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.979+0000 7fb269afe700 1 --2- 172.21.15.1:0/260581547 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb26409a150 0x7fb264158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:12.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.979+0000 7fb269afe700 1 -- 172.21.15.1:0/260581547 >> 172.21.15.1:0/260581547 conn(0x7fb2640b2bd0 msgr2=0x7fb2640b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:12.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.979+0000 7fb269afe700 1 -- 172.21.15.1:0/260581547 shutdown_connections 2024-03-18T20:27:12.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:12.979+0000 7fb269afe700 1 -- 172.21.15.1:0/260581547 wait complete. 2024-03-18T20:27:12.996 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-18T20:27:13.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.120+0000 7f4e07adc700 1 Processor -- start 2024-03-18T20:27:13.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.120+0000 7f4e07adc700 1 -- start start 2024-03-18T20:27:13.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.121+0000 7f4e07adc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e0015a1d0 0x7f4e000add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.121+0000 7f4e07adc700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e000ce8f0 con 0x7f4e000cc0f0 2024-03-18T20:27:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.121+0000 7f4e07adc700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e000ae260 con 0x7f4e00071620 2024-03-18T20:27:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.121+0000 7f4e07adc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e000cdb00 con 0x7f4e0015a1d0 2024-03-18T20:27:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.121+0000 7f4e06079700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4e000cc0f0 0x7f4e0015ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49342/0 (socket says 172.21.15.1:49342) 2024-03-18T20:27:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.121+0000 7f4e06079700 1 -- 172.21.15.1:0/237773472 learned_addr learned my addr 172.21.15.1:0/237773472 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.121+0000 7f4e05878700 1 --2- 172.21.15.1:0/237773472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e0015a1d0 0x7f4e000add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.121+0000 7f4e04876700 1 -- 172.21.15.1:0/237773472 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295200155 0 0) 0x7f4e000ce8f0 con 0x7f4e000cc0f0 2024-03-18T20:27:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.121+0000 7f4e04876700 1 -- 172.21.15.1:0/237773472 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dec0035b0 con 0x7f4e000cc0f0 2024-03-18T20:27:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.121+0000 7f4e04876700 1 -- 172.21.15.1:0/237773472 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078563811 0 0) 0x7f4e000ae260 con 0x7f4e00071620 2024-03-18T20:27:13.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.121+0000 7f4e04876700 1 -- 172.21.15.1:0/237773472 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e000ce8f0 con 0x7f4e00071620 2024-03-18T20:27:13.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.121+0000 7f4e04876700 1 -- 172.21.15.1:0/237773472 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1999696657 0 0) 0x7f4dec0035b0 con 0x7f4e000cc0f0 2024-03-18T20:27:13.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.121+0000 7f4e04876700 1 -- 172.21.15.1:0/237773472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e0015a1d0 msgr2=0x7f4e000add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:13.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.121+0000 7f4e04876700 1 --2- 172.21.15.1:0/237773472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e0015a1d0 0x7f4e000add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.121+0000 7f4e04876700 1 -- 172.21.15.1:0/237773472 >> v1:172.21.15.165:6789/0 conn(0x7f4e00071620 legacy=0x7f4e000cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:13.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.121+0000 7f4e04876700 1 -- 172.21.15.1:0/237773472 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e000cdce0 con 0x7f4e000cc0f0 2024-03-18T20:27:13.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.121+0000 7f4e04876700 1 -- 172.21.15.1:0/237773472 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f4df80019f0 con 0x7f4e000cc0f0 2024-03-18T20:27:13.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.122+0000 7f4e04876700 1 -- 172.21.15.1:0/237773472 >> v1:172.21.15.1:6789/0 conn(0x7f4e000cc0f0 legacy=0x7f4e0015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:13.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.122+0000 7f4e04876700 1 --2- 172.21.15.1:0/237773472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4dec003bc0 0x7f4dec003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.122+0000 7f4e04876700 1 --2- 172.21.15.1:0/237773472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4dec004850 0x7f4dec004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.122+0000 7f4e04876700 1 --2- 172.21.15.1:0/237773472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e0015a1d0 0x7f4dec005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.122+0000 7f4e04876700 1 -- 172.21.15.1:0/237773472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e000cdb00 con 0x7f4dec004850 2024-03-18T20:27:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.122+0000 7f4e04876700 1 -- 172.21.15.1:0/237773472 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec00bac0 con 0x7f4dec003bc0 2024-03-18T20:27:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.122+0000 7f4e04876700 1 -- 172.21.15.1:0/237773472 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec00de60 con 0x7f4e0015a1d0 2024-03-18T20:27:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.122+0000 7f4e05077700 1 --2- 172.21.15.1:0/237773472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4dec004850 0x7f4dec004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.122+0000 7f4e05878700 1 --2- 172.21.15.1:0/237773472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4dec003bc0 0x7f4dec003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.122+0000 7f4e06079700 1 --2- 172.21.15.1:0/237773472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e0015a1d0 0x7f4dec005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.122+0000 7f4e05077700 1 -- 172.21.15.1:0/237773472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e0015a1d0 msgr2=0x7f4dec005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.122+0000 7f4e05077700 1 --2- 172.21.15.1:0/237773472 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e0015a1d0 0x7f4dec005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.122+0000 7f4e05077700 1 -- 172.21.15.1:0/237773472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4dec003bc0 msgr2=0x7f4dec003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.122+0000 7f4e05077700 1 --2- 172.21.15.1:0/237773472 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4dec003bc0 0x7f4dec003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.122+0000 7f4e05077700 1 -- 172.21.15.1:0/237773472 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dec0035b0 con 0x7f4dec004850 2024-03-18T20:27:13.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.122+0000 7f4e05077700 1 --2- 172.21.15.1:0/237773472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4dec004850 0x7f4dec004cd0 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7f4df0001940 tx=0x7f4dec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3dafd70ed018fe0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.122+0000 7f4e04876700 1 -- 172.21.15.1:0/237773472 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df0001430 con 0x7f4dec004850 2024-03-18T20:27:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.122+0000 7f4e04876700 1 -- 172.21.15.1:0/237773472 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df0001d70 con 0x7f4dec004850 2024-03-18T20:27:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.122+0000 7f4e04876700 1 -- 172.21.15.1:0/237773472 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df0005510 con 0x7f4dec004850 2024-03-18T20:27:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.123+0000 7f4e07adc700 1 -- 172.21.15.1:0/237773472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4dec004850 msgr2=0x7f4dec004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.123+0000 7f4e07adc700 1 --2- 172.21.15.1:0/237773472 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4dec004850 0x7f4dec004cd0 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7f4df0001940 tx=0x7f4dec005880 comp rx=0 tx=0).stop 2024-03-18T20:27:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.123+0000 7f4e06079700 1 -- 172.21.15.1:0/237773472 reap_dead start 2024-03-18T20:27:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.123+0000 7f4e07adc700 1 -- 172.21.15.1:0/237773472 shutdown_connections 2024-03-18T20:27:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.123+0000 7f4e07adc700 1 -- 172.21.15.1:0/237773472 >> 172.21.15.1:0/237773472 conn(0x7f4e000cc9b0 msgr2=0x7f4e000ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:13.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.123+0000 7f4e07adc700 1 -- 172.21.15.1:0/237773472 shutdown_connections 2024-03-18T20:27:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.123+0000 7f4e07adc700 1 -- 172.21.15.1:0/237773472 wait complete. 2024-03-18T20:27:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.124+0000 7f4e07adc700 1 Processor -- start 2024-03-18T20:27:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.124+0000 7f4e07adc700 1 -- start start 2024-03-18T20:27:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.125+0000 7f4e07adc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4dec004850 0x7f4e000c3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.125+0000 7f4e07adc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e000cab00 0x7f4e001557b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.125+0000 7f4e07adc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e0015cba0 0x7f4e0015d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.125+0000 7f4e07adc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0015d590 con 0x7f4e000cab00 2024-03-18T20:27:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.125+0000 7f4e07adc700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e000c42b0 con 0x7f4dec004850 2024-03-18T20:27:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.125+0000 7f4e07adc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e000c4590 con 0x7f4e0015cba0 2024-03-18T20:27:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.125+0000 7f4e05077700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e000cab00 0x7f4e001557b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.125+0000 7f4e05077700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e000cab00 0x7f4e001557b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54226/0 (socket says 172.21.15.1:54226) 2024-03-18T20:27:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.125+0000 7f4e05077700 1 -- 172.21.15.1:0/3025334312 learned_addr learned my addr 172.21.15.1:0/3025334312 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.125+0000 7f4e06079700 1 --2- 172.21.15.1:0/3025334312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e0015cba0 0x7f4e0015d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.125+0000 7f4e05878700 1 --2- 172.21.15.1:0/3025334312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4dec004850 0x7f4e000c3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.125+0000 7f4e05077700 1 -- 172.21.15.1:0/3025334312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e0015cba0 msgr2=0x7f4e0015d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.125+0000 7f4e05077700 1 --2- 172.21.15.1:0/3025334312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e0015cba0 0x7f4e0015d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.125+0000 7f4e05077700 1 -- 172.21.15.1:0/3025334312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4dec004850 msgr2=0x7f4e000c3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.125+0000 7f4e05077700 1 --2- 172.21.15.1:0/3025334312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4dec004850 0x7f4e000c3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.125+0000 7f4e05077700 1 -- 172.21.15.1:0/3025334312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e000c4870 con 0x7f4e000cab00 2024-03-18T20:27:13.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.126+0000 7f4e05077700 1 --2- 172.21.15.1:0/3025334312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e000cab00 0x7f4e001557b0 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 crypto rx=0x7f4df000c3c0 tx=0x7f4df00012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:13.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.126+0000 7f4df67fc700 1 -- 172.21.15.1:0/3025334312 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df0006c70 con 0x7f4e000cab00 2024-03-18T20:27:13.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.126+0000 7f4df67fc700 1 -- 172.21.15.1:0/3025334312 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df0006dd0 con 0x7f4e000cab00 2024-03-18T20:27:13.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.126+0000 7f4e07adc700 1 -- 172.21.15.1:0/3025334312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e000c4b50 con 0x7f4e000cab00 2024-03-18T20:27:13.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.126+0000 7f4df67fc700 1 -- 172.21.15.1:0/3025334312 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df0007260 con 0x7f4e000cab00 2024-03-18T20:27:13.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.126+0000 7f4e07adc700 1 -- 172.21.15.1:0/3025334312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e00155cf0 con 0x7f4e000cab00 2024-03-18T20:27:13.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.127+0000 7f4e07adc700 1 -- 172.21.15.1:0/3025334312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e000c4ec0 con 0x7f4e000cab00 2024-03-18T20:27:13.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.127+0000 7f4df67fc700 1 -- 172.21.15.1:0/3025334312 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f4df00073c0 con 0x7f4e000cab00 2024-03-18T20:27:13.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.128+0000 7f4df67fc700 1 --2- 172.21.15.1:0/3025334312 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4de406aed0 0x7f4de406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.128+0000 7f4df67fc700 1 -- 172.21.15.1:0/3025334312 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6223+0+0 (secure 0 0 0) 0x7f4df008a5b0 con 0x7f4e000cab00 2024-03-18T20:27:13.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.128+0000 7f4e05878700 1 --2- 172.21.15.1:0/3025334312 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4de406aed0 0x7f4de406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.128+0000 7f4e05878700 1 --2- 172.21.15.1:0/3025334312 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4de406aed0 0x7f4de406d390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc009b60 tx=0x7f4dfc0091c0 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:13.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.132+0000 7f4df67fc700 1 -- 172.21.15.1:0/3025334312 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4df00554b0 con 0x7f4e000cab00 2024-03-18T20:27:13.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.333+0000 7f4e07adc700 1 -- 172.21.15.1:0/3025334312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f4e000676a0 con 0x7f4e000cab00 2024-03-18T20:27:13.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.337+0000 7f4df67fc700 1 -- 172.21.15.1:0/3025334312 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v86) v1 ==== 120+0+0 (secure 0 0 0) 0x7f4df0058ad0 con 0x7f4e000cab00 2024-03-18T20:27:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.339+0000 7f4e07adc700 1 -- 172.21.15.1:0/3025334312 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4de406aed0 msgr2=0x7f4de406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.339+0000 7f4e07adc700 1 --2- 172.21.15.1:0/3025334312 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4de406aed0 0x7f4de406d390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc009b60 tx=0x7f4dfc0091c0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.339+0000 7f4e07adc700 1 -- 172.21.15.1:0/3025334312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e000cab00 msgr2=0x7f4e001557b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.339+0000 7f4e07adc700 1 --2- 172.21.15.1:0/3025334312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e000cab00 0x7f4e001557b0 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 crypto rx=0x7f4df000c3c0 tx=0x7f4df00012d0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.339+0000 7f4e07adc700 1 -- 172.21.15.1:0/3025334312 shutdown_connections 2024-03-18T20:27:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.339+0000 7f4e07adc700 1 --2- 172.21.15.1:0/3025334312 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4de406aed0 0x7f4de406d390 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.339+0000 7f4e07adc700 1 --2- 172.21.15.1:0/3025334312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4e0015cba0 0x7f4e0015d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.339+0000 7f4e07adc700 1 --2- 172.21.15.1:0/3025334312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4dec004850 0x7f4e000c3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.339+0000 7f4e07adc700 1 --2- 172.21.15.1:0/3025334312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e000cab00 0x7f4e001557b0 unknown :-1 s=CLOSED pgs=3017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.339+0000 7f4e07adc700 1 -- 172.21.15.1:0/3025334312 >> 172.21.15.1:0/3025334312 conn(0x7f4e000cc9b0 msgr2=0x7f4e000afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:13.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.340+0000 7f4e07adc700 1 -- 172.21.15.1:0/3025334312 shutdown_connections 2024-03-18T20:27:13.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.340+0000 7f4e07adc700 1 -- 172.21.15.1:0/3025334312 wait complete. 2024-03-18T20:27:13.356 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-18T20:27:13.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.479+0000 7fb4f7675700 1 Processor -- start 2024-03-18T20:27:13.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.479+0000 7fb4f7675700 1 -- start start 2024-03-18T20:27:13.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.479+0000 7fb4f7675700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f015eba0 0x7fb4f00cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.479+0000 7fb4f7675700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb4f00b0500 0x7fb4f00cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.479+0000 7fb4f7675700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb4f00cfdf0 0x7fb4f0099a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.479+0000 7fb4f7675700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f015ef80 con 0x7fb4f00b0500 2024-03-18T20:27:13.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.480+0000 7fb4f7675700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f00cc9a0 con 0x7fb4f00cfdf0 2024-03-18T20:27:13.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.480+0000 7fb4f7675700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f00ccc80 con 0x7fb4f015eba0 2024-03-18T20:27:13.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.480+0000 7fb4f4c10700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb4f00b0500 0x7fb4f00cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.480+0000 7fb4f4c10700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb4f00b0500 0x7fb4f00cf8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54238/0 (socket says 172.21.15.1:54238) 2024-03-18T20:27:13.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.480+0000 7fb4f4c10700 1 -- 172.21.15.1:0/2312153858 learned_addr learned my addr 172.21.15.1:0/2312153858 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:13.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.480+0000 7fb4f4c10700 1 -- 172.21.15.1:0/2312153858 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f015eba0 msgr2=0x7fb4f00cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:13.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.480+0000 7fb4f5c12700 1 --2- 172.21.15.1:0/2312153858 >> v2:172.21.15.165:3300/0 conn(0x7fb4f00cfdf0 0x7fb4f0099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.480+0000 7fb4f5411700 1 --2- 172.21.15.1:0/2312153858 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f015eba0 0x7fb4f00cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.480+0000 7fb4f4c10700 1 --2- 172.21.15.1:0/2312153858 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f015eba0 0x7fb4f00cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.480+0000 7fb4f4c10700 1 -- 172.21.15.1:0/2312153858 >> v2:172.21.15.165:3300/0 conn(0x7fb4f00cfdf0 msgr2=0x7fb4f0099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:13.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.480+0000 7fb4f4c10700 1 --2- 172.21.15.1:0/2312153858 >> v2:172.21.15.165:3300/0 conn(0x7fb4f00cfdf0 0x7fb4f0099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.480+0000 7fb4f4c10700 1 -- 172.21.15.1:0/2312153858 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f00ccf60 con 0x7fb4f00b0500 2024-03-18T20:27:13.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.480+0000 7fb4f5c12700 1 --2- 172.21.15.1:0/2312153858 >> v2:172.21.15.165:3300/0 conn(0x7fb4f00cfdf0 0x7fb4f0099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:13.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.480+0000 7fb4f5411700 1 --2- 172.21.15.1:0/2312153858 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f015eba0 0x7fb4f00cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:13.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.480+0000 7fb4f4c10700 1 --2- 172.21.15.1:0/2312153858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4f00b0500 0x7fb4f00cf8b0 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7fb4f00d0af0 tx=0x7fb4ec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d9abd9d1dc37c95 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:13.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.480+0000 7fb4e3fff700 1 -- 172.21.15.1:0/2312153858 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec00b870 con 0x7fb4f00b0500 2024-03-18T20:27:13.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.480+0000 7fb4e3fff700 1 -- 172.21.15.1:0/2312153858 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ec00b9d0 con 0x7fb4f00b0500 2024-03-18T20:27:13.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.480+0000 7fb4e3fff700 1 -- 172.21.15.1:0/2312153858 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec00bd60 con 0x7fb4f00b0500 2024-03-18T20:27:13.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.481+0000 7fb4f7675700 1 -- 172.21.15.1:0/2312153858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4f00b0500 msgr2=0x7fb4f00cf8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:13.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.481+0000 7fb4f7675700 1 --2- 172.21.15.1:0/2312153858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4f00b0500 0x7fb4f00cf8b0 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7fb4f00d0af0 tx=0x7fb4ec009bd0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.481+0000 7fb4f7675700 1 -- 172.21.15.1:0/2312153858 shutdown_connections 2024-03-18T20:27:13.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.481+0000 7fb4f7675700 1 --2- 172.21.15.1:0/2312153858 >> v2:172.21.15.165:3300/0 conn(0x7fb4f00cfdf0 0x7fb4f0099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.481+0000 7fb4f7675700 1 --2- 172.21.15.1:0/2312153858 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f015eba0 0x7fb4f00cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.481+0000 7fb4f7675700 1 --2- 172.21.15.1:0/2312153858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4f00b0500 0x7fb4f00cf8b0 unknown :-1 s=CLOSED pgs=3018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.481+0000 7fb4f7675700 1 -- 172.21.15.1:0/2312153858 >> 172.21.15.1:0/2312153858 conn(0x7fb4f00b2bd0 msgr2=0x7fb4f00b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:13.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.481+0000 7fb4f7675700 1 -- 172.21.15.1:0/2312153858 shutdown_connections 2024-03-18T20:27:13.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.481+0000 7fb4f7675700 1 -- 172.21.15.1:0/2312153858 wait complete. 2024-03-18T20:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.482+0000 7fb4f7675700 1 Processor -- start 2024-03-18T20:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.482+0000 7fb4f7675700 1 -- start start 2024-03-18T20:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.483+0000 7fb4f7675700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4f00b0500 0x7fb4f0155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.483+0000 7fb4f7675700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4f00cfdf0 0x7fb4f00c6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.483+0000 7fb4f7675700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f015eba0 0x7fb4f006f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.483+0000 7fb4f7675700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f00c5950 con 0x7fb4f00b0500 2024-03-18T20:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.483+0000 7fb4f7675700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f00c4d40 con 0x7fb4f00cfdf0 2024-03-18T20:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.483+0000 7fb4f7675700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f00c5020 con 0x7fb4f015eba0 2024-03-18T20:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.483+0000 7fb4f5411700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4f00b0500 0x7fb4f0155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.483+0000 7fb4f5411700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4f00b0500 0x7fb4f0155f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54240/0 (socket says 172.21.15.1:54240) 2024-03-18T20:27:13.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.483+0000 7fb4f5411700 1 -- 172.21.15.1:0/1083403946 learned_addr learned my addr 172.21.15.1:0/1083403946 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:13.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.483+0000 7fb4f5c12700 1 --2- 172.21.15.1:0/1083403946 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f015eba0 0x7fb4f006f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.483+0000 7fb4f4c10700 1 --2- 172.21.15.1:0/1083403946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4f00cfdf0 0x7fb4f00c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.483+0000 7fb4f5411700 1 -- 172.21.15.1:0/1083403946 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f015eba0 msgr2=0x7fb4f006f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:13.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.483+0000 7fb4f5411700 1 --2- 172.21.15.1:0/1083403946 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f015eba0 0x7fb4f006f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.483+0000 7fb4f5411700 1 -- 172.21.15.1:0/1083403946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4f00cfdf0 msgr2=0x7fb4f00c6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:13.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.483+0000 7fb4f5411700 1 --2- 172.21.15.1:0/1083403946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4f00cfdf0 0x7fb4f00c6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.483+0000 7fb4f5411700 1 -- 172.21.15.1:0/1083403946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec009550 con 0x7fb4f00b0500 2024-03-18T20:27:13.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.484+0000 7fb4f5411700 1 --2- 172.21.15.1:0/1083403946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4f00b0500 0x7fb4f0155f10 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7fb4e400c7f0 tx=0x7fb4e400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:13.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.484+0000 7fb4e1ffb700 1 -- 172.21.15.1:0/1083403946 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4011800 con 0x7fb4f00b0500 2024-03-18T20:27:13.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.484+0000 7fb4e1ffb700 1 -- 172.21.15.1:0/1083403946 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e4011960 con 0x7fb4f00b0500 2024-03-18T20:27:13.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.484+0000 7fb4f7675700 1 -- 172.21.15.1:0/1083403946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f00c5300 con 0x7fb4f00b0500 2024-03-18T20:27:13.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.484+0000 7fb4e1ffb700 1 -- 172.21.15.1:0/1083403946 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4011cf0 con 0x7fb4f00b0500 2024-03-18T20:27:13.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.484+0000 7fb4f7675700 1 -- 172.21.15.1:0/1083403946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f00c49d0 con 0x7fb4f00b0500 2024-03-18T20:27:13.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.485+0000 7fb4f7675700 1 -- 172.21.15.1:0/1083403946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f00676a0 con 0x7fb4f00b0500 2024-03-18T20:27:13.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.485+0000 7fb4e1ffb700 1 -- 172.21.15.1:0/1083403946 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fb4e4018360 con 0x7fb4f00b0500 2024-03-18T20:27:13.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.486+0000 7fb4e1ffb700 1 --2- 172.21.15.1:0/1083403946 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb4d406aed0 0x7fb4d406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.486+0000 7fb4e1ffb700 1 -- 172.21.15.1:0/1083403946 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6223+0+0 (secure 0 0 0) 0x7fb4e4089920 con 0x7fb4f00b0500 2024-03-18T20:27:13.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.486+0000 7fb4f4c10700 1 --2- 172.21.15.1:0/1083403946 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb4d406aed0 0x7fb4d406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.486+0000 7fb4f4c10700 1 --2- 172.21.15.1:0/1083403946 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb4d406aed0 0x7fb4d406d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009840 tx=0x7fb4ec008030 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:13.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.490+0000 7fb4e1ffb700 1 -- 172.21.15.1:0/1083403946 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb4e4054490 con 0x7fb4f00b0500 2024-03-18T20:27:13.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.692+0000 7fb4f7675700 1 -- 172.21.15.1:0/1083403946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb4f00b0270 con 0x7fb4f00b0500 2024-03-18T20:27:13.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.696+0000 7fb4e1ffb700 1 -- 172.21.15.1:0/1083403946 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v87) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb4e4057ab0 con 0x7fb4f00b0500 2024-03-18T20:27:13.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.698+0000 7fb4f7675700 1 -- 172.21.15.1:0/1083403946 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb4d406aed0 msgr2=0x7fb4d406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.698+0000 7fb4f7675700 1 --2- 172.21.15.1:0/1083403946 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb4d406aed0 0x7fb4d406d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009840 tx=0x7fb4ec008030 comp rx=0 tx=0).stop 2024-03-18T20:27:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.698+0000 7fb4f7675700 1 -- 172.21.15.1:0/1083403946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4f00b0500 msgr2=0x7fb4f0155f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.698+0000 7fb4f7675700 1 --2- 172.21.15.1:0/1083403946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4f00b0500 0x7fb4f0155f10 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7fb4e400c7f0 tx=0x7fb4e400cb00 comp rx=0 tx=0).stop 2024-03-18T20:27:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.698+0000 7fb4f7675700 1 -- 172.21.15.1:0/1083403946 shutdown_connections 2024-03-18T20:27:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.698+0000 7fb4f7675700 1 --2- 172.21.15.1:0/1083403946 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb4d406aed0 0x7fb4d406d390 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.698+0000 7fb4f7675700 1 --2- 172.21.15.1:0/1083403946 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f015eba0 0x7fb4f006f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.698+0000 7fb4f7675700 1 --2- 172.21.15.1:0/1083403946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4f00b0500 0x7fb4f0155f10 unknown :-1 s=CLOSED pgs=3019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.698+0000 7fb4f7675700 1 --2- 172.21.15.1:0/1083403946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4f00cfdf0 0x7fb4f00c6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.698+0000 7fb4f7675700 1 -- 172.21.15.1:0/1083403946 >> 172.21.15.1:0/1083403946 conn(0x7fb4f00b2bd0 msgr2=0x7fb4f015a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.698+0000 7fb4f7675700 1 -- 172.21.15.1:0/1083403946 shutdown_connections 2024-03-18T20:27:13.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.698+0000 7fb4f7675700 1 -- 172.21.15.1:0/1083403946 wait complete. 2024-03-18T20:27:13.715 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2024-03-18T20:27:13.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.836+0000 7f4d36624700 1 Processor -- start 2024-03-18T20:27:13.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.836+0000 7f4d36624700 1 -- start start 2024-03-18T20:27:13.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.837+0000 7f4d36624700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4d3015b3d0 0x7f4d300cf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.837+0000 7f4d36624700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d300ce200 con 0x7f4d3015aff0 2024-03-18T20:27:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.837+0000 7f4d36624700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d300ce4a0 con 0x7f4d30073b00 2024-03-18T20:27:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.837+0000 7f4d36624700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d300ce780 con 0x7f4d3015b3d0 2024-03-18T20:27:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.837+0000 7f4d2ffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4d3015aff0 0x7f4d3015f780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:49346/0 (socket says 172.21.15.1:49346) 2024-03-18T20:27:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.837+0000 7f4d2ffff700 1 -- 172.21.15.1:0/2210284685 learned_addr learned my addr 172.21.15.1:0/2210284685 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2f7fe700 1 --2- 172.21.15.1:0/2210284685 >> v2:172.21.15.165:3300/0 conn(0x7f4d3015b3d0 0x7f4d300cf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2effd700 1 -- 172.21.15.1:0/2210284685 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405725635 0 0) 0x7f4d300ce200 con 0x7f4d3015aff0 2024-03-18T20:27:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2effd700 1 -- 172.21.15.1:0/2210284685 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d180035b0 con 0x7f4d3015aff0 2024-03-18T20:27:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2effd700 1 -- 172.21.15.1:0/2210284685 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3741600652 0 0) 0x7f4d180035b0 con 0x7f4d3015aff0 2024-03-18T20:27:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2effd700 1 -- 172.21.15.1:0/2210284685 >> v2:172.21.15.165:3300/0 conn(0x7f4d3015b3d0 msgr2=0x7f4d300cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2effd700 1 --2- 172.21.15.1:0/2210284685 >> v2:172.21.15.165:3300/0 conn(0x7f4d3015b3d0 0x7f4d300cf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2effd700 1 -- 172.21.15.1:0/2210284685 >> v1:172.21.15.165:6789/0 conn(0x7f4d30073b00 legacy=0x7f4d300cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2effd700 1 -- 172.21.15.1:0/2210284685 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d300cea60 con 0x7f4d3015aff0 2024-03-18T20:27:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2effd700 1 -- 172.21.15.1:0/2210284685 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f4d24001670 con 0x7f4d3015aff0 2024-03-18T20:27:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2effd700 1 -- 172.21.15.1:0/2210284685 >> v1:172.21.15.1:6789/0 conn(0x7f4d3015aff0 legacy=0x7f4d3015f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:13.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2effd700 1 --2- 172.21.15.1:0/2210284685 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d18003a00 0x7f4d18003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2effd700 1 --2- 172.21.15.1:0/2210284685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d18004850 0x7f4d18004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2effd700 1 --2- 172.21.15.1:0/2210284685 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d18005210 0x7f4d1800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2effd700 1 -- 172.21.15.1:0/2210284685 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d300ce780 con 0x7f4d18004850 2024-03-18T20:27:13.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2effd700 1 -- 172.21.15.1:0/2210284685 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1800c0b0 con 0x7f4d18003a00 2024-03-18T20:27:13.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2effd700 1 -- 172.21.15.1:0/2210284685 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1800c270 con 0x7f4d18005210 2024-03-18T20:27:13.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2f7fe700 1 --2- 172.21.15.1:0/2210284685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d18004850 0x7f4d18004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2ffff700 1 --2- 172.21.15.1:0/2210284685 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d18003a00 0x7f4d18003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d34bc1700 1 --2- 172.21.15.1:0/2210284685 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d18005210 0x7f4d1800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2f7fe700 1 -- 172.21.15.1:0/2210284685 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d18005210 msgr2=0x7f4d1800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:13.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2f7fe700 1 --2- 172.21.15.1:0/2210284685 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d18005210 0x7f4d1800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2f7fe700 1 -- 172.21.15.1:0/2210284685 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d18003a00 msgr2=0x7f4d18003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:13.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2f7fe700 1 --2- 172.21.15.1:0/2210284685 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d18003a00 0x7f4d18003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d34bc1700 1 -- 172.21.15.1:0/2210284685 reap_dead start 2024-03-18T20:27:13.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.838+0000 7f4d2f7fe700 1 -- 172.21.15.1:0/2210284685 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d180035b0 con 0x7f4d18004850 2024-03-18T20:27:13.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.839+0000 7f4d2f7fe700 1 --2- 172.21.15.1:0/2210284685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d18004850 0x7f4d18004cd0 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 crypto rx=0x7f4d20009b40 tx=0x7f4d18005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df80ea051307ebd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:13.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.839+0000 7f4d2effd700 1 -- 172.21.15.1:0/2210284685 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d20005a20 con 0x7f4d18004850 2024-03-18T20:27:13.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.839+0000 7f4d2effd700 1 -- 172.21.15.1:0/2210284685 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d20005b80 con 0x7f4d18004850 2024-03-18T20:27:13.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.839+0000 7f4d2effd700 1 -- 172.21.15.1:0/2210284685 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2000e6c0 con 0x7f4d18004850 2024-03-18T20:27:13.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.839+0000 7f4d36624700 1 -- 172.21.15.1:0/2210284685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d18004850 msgr2=0x7f4d18004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:13.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.839+0000 7f4d36624700 1 --2- 172.21.15.1:0/2210284685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d18004850 0x7f4d18004cd0 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 crypto rx=0x7f4d20009b40 tx=0x7f4d18005880 comp rx=0 tx=0).stop 2024-03-18T20:27:13.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.839+0000 7f4d36624700 1 -- 172.21.15.1:0/2210284685 shutdown_connections 2024-03-18T20:27:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.839+0000 7f4d36624700 1 --2- 172.21.15.1:0/2210284685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d18004850 0x7f4d18004cd0 unknown :-1 s=CLOSED pgs=3021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.839+0000 7f4d36624700 1 -- 172.21.15.1:0/2210284685 >> 172.21.15.1:0/2210284685 conn(0x7f4d3006a750 msgr2=0x7f4d3006ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.839+0000 7f4d36624700 1 -- 172.21.15.1:0/2210284685 shutdown_connections 2024-03-18T20:27:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.839+0000 7f4d36624700 1 -- 172.21.15.1:0/2210284685 wait complete. 2024-03-18T20:27:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.841+0000 7f4d36624700 1 Processor -- start 2024-03-18T20:27:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.841+0000 7f4d36624700 1 -- start start 2024-03-18T20:27:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.841+0000 7f4d36624700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d18004850 0x7f4d30156430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.841+0000 7f4d36624700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d300b2ec0 0x7f4d301415f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.841+0000 7f4d36624700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d30141b30 0x7f4d300ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.841+0000 7f4d36624700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d30142000 con 0x7f4d30141b30 2024-03-18T20:27:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.841+0000 7f4d36624700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3006ce80 con 0x7f4d18004850 2024-03-18T20:27:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.841+0000 7f4d36624700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3006d160 con 0x7f4d300b2ec0 2024-03-18T20:27:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.842+0000 7f4d34bc1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d30141b30 0x7f4d300ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.842+0000 7f4d34bc1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d30141b30 0x7f4d300ac8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54262/0 (socket says 172.21.15.1:54262) 2024-03-18T20:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.842+0000 7f4d34bc1700 1 -- 172.21.15.1:0/2187299279 learned_addr learned my addr 172.21.15.1:0/2187299279 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.842+0000 7f4d2ffff700 1 --2- 172.21.15.1:0/2187299279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d18004850 0x7f4d30156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.842+0000 7f4d34bc1700 1 -- 172.21.15.1:0/2187299279 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d300b2ec0 msgr2=0x7f4d301415f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.842+0000 7f4d34bc1700 1 --2- 172.21.15.1:0/2187299279 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d300b2ec0 0x7f4d301415f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.842+0000 7f4d34bc1700 1 -- 172.21.15.1:0/2187299279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d18004850 msgr2=0x7f4d30156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.842+0000 7f4d34bc1700 1 --2- 172.21.15.1:0/2187299279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d18004850 0x7f4d30156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.842+0000 7f4d34bc1700 1 -- 172.21.15.1:0/2187299279 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d3006d440 con 0x7f4d30141b30 2024-03-18T20:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.842+0000 7f4d34bc1700 1 --2- 172.21.15.1:0/2187299279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d30141b30 0x7f4d300ac8b0 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 crypto rx=0x7f4d30070db0 tx=0x7f4d2800ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.842+0000 7f4d2cff9700 1 -- 172.21.15.1:0/2187299279 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d28003c10 con 0x7f4d30141b30 2024-03-18T20:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.842+0000 7f4d2cff9700 1 -- 172.21.15.1:0/2187299279 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d28003d70 con 0x7f4d30141b30 2024-03-18T20:27:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.842+0000 7f4d36624700 1 -- 172.21.15.1:0/2187299279 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d3006d720 con 0x7f4d30141b30 2024-03-18T20:27:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.842+0000 7f4d2cff9700 1 -- 172.21.15.1:0/2187299279 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d28005310 con 0x7f4d30141b30 2024-03-18T20:27:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.842+0000 7f4d36624700 1 -- 172.21.15.1:0/2187299279 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d300ad800 con 0x7f4d30141b30 2024-03-18T20:27:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.843+0000 7f4d36624700 1 -- 172.21.15.1:0/2187299279 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d3006d940 con 0x7f4d30141b30 2024-03-18T20:27:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.844+0000 7f4d2cff9700 1 -- 172.21.15.1:0/2187299279 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f4d28005470 con 0x7f4d30141b30 2024-03-18T20:27:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.844+0000 7f4d2cff9700 1 --2- 172.21.15.1:0/2187299279 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4d1c06aed0 0x7f4d1c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.844+0000 7f4d2cff9700 1 -- 172.21.15.1:0/2187299279 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6223+0+0 (secure 0 0 0) 0x7f4d280898b0 con 0x7f4d30141b30 2024-03-18T20:27:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.844+0000 7f4d2ffff700 1 --2- 172.21.15.1:0/2187299279 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4d1c06aed0 0x7f4d1c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.844+0000 7f4d2ffff700 1 --2- 172.21.15.1:0/2187299279 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4d1c06aed0 0x7f4d1c06d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4d24001050 tx=0x7f4d24009040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:13.849+0000 7f4d2cff9700 1 -- 172.21.15.1:0/2187299279 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4d28054530 con 0x7f4d30141b30 2024-03-18T20:27:14.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.136+0000 7f4d36624700 1 -- 172.21.15.1:0/2187299279 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7f4d300676a0 con 0x7f4d30141b30 2024-03-18T20:27:14.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.597+0000 7f4d2cff9700 1 -- 172.21.15.1:0/2187299279 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3210) v1 ==== 187+0+0 (secure 0 0 0) 0x7f4d28057b50 con 0x7f4d30141b30 2024-03-18T20:27:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.600+0000 7f4d36624700 1 -- 172.21.15.1:0/2187299279 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4d1c06aed0 msgr2=0x7f4d1c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.600+0000 7f4d36624700 1 --2- 172.21.15.1:0/2187299279 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4d1c06aed0 0x7f4d1c06d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4d24001050 tx=0x7f4d24009040 comp rx=0 tx=0).stop 2024-03-18T20:27:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.600+0000 7f4d36624700 1 -- 172.21.15.1:0/2187299279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d30141b30 msgr2=0x7f4d300ac8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.600+0000 7f4d36624700 1 --2- 172.21.15.1:0/2187299279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d30141b30 0x7f4d300ac8b0 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 crypto rx=0x7f4d30070db0 tx=0x7f4d2800ee00 comp rx=0 tx=0).stop 2024-03-18T20:27:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.600+0000 7f4d36624700 1 -- 172.21.15.1:0/2187299279 shutdown_connections 2024-03-18T20:27:14.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.600+0000 7f4d36624700 1 --2- 172.21.15.1:0/2187299279 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4d1c06aed0 0x7f4d1c06d390 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.600+0000 7f4d36624700 1 --2- 172.21.15.1:0/2187299279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d30141b30 0x7f4d300ac8b0 unknown :-1 s=CLOSED pgs=3022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.600+0000 7f4d36624700 1 --2- 172.21.15.1:0/2187299279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d18004850 0x7f4d30156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.600+0000 7f4d36624700 1 --2- 172.21.15.1:0/2187299279 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4d300b2ec0 0x7f4d301415f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.600+0000 7f4d36624700 1 -- 172.21.15.1:0/2187299279 >> 172.21.15.1:0/2187299279 conn(0x7f4d3006a750 msgr2=0x7f4d300cc240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.600+0000 7f4d36624700 1 -- 172.21.15.1:0/2187299279 shutdown_connections 2024-03-18T20:27:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.600+0000 7f4d36624700 1 -- 172.21.15.1:0/2187299279 wait complete. 2024-03-18T20:27:14.619 INFO:tasks.cephfs_test_runner:Test command fails without --fs argument in presence of multiple fs ... ok 2024-03-18T20:27:14.619 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2024-03-18T20:27:14.619 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2024-03-18T20:27:14.619 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2024-03-18T20:27:14.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.748+0000 7f5947f10700 1 Processor -- start 2024-03-18T20:27:14.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.748+0000 7f5947f10700 1 -- start start 2024-03-18T20:27:14.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.749+0000 7f5947f10700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f594015a2e0 0x7f59400ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:14.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.749+0000 7f5947f10700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f59400cc170 0x7f594006ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:14.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.749+0000 7f5947f10700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594006e270 con 0x7f59400ae600 2024-03-18T20:27:14.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.749+0000 7f5947f10700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59400ae050 con 0x7f594015a2e0 2024-03-18T20:27:14.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.749+0000 7f5947f10700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59400add70 con 0x7f59400cc170 2024-03-18T20:27:14.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.749+0000 7f59454ab700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f594015a2e0 0x7f59400ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:14.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.749+0000 7f5945cac700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f59400ae600 0x7f59400ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33708/0 (socket says 172.21.15.1:33708) 2024-03-18T20:27:14.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.749+0000 7f5945cac700 1 -- 172.21.15.1:0/2037419051 learned_addr learned my addr 172.21.15.1:0/2037419051 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:14.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.749+0000 7f59464ad700 1 --2- 172.21.15.1:0/2037419051 >> v2:172.21.15.165:3300/0 conn(0x7f59400cc170 0x7f594006ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:14.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.750+0000 7f59454ab700 1 -- 172.21.15.1:0/2037419051 >> v2:172.21.15.165:3300/0 conn(0x7f59400cc170 msgr2=0x7f594006ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:14.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.750+0000 7f59454ab700 1 --2- 172.21.15.1:0/2037419051 >> v2:172.21.15.165:3300/0 conn(0x7f59400cc170 0x7f594006ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:14.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.750+0000 7f59454ab700 1 -- 172.21.15.1:0/2037419051 >> v1:172.21.15.1:6789/0 conn(0x7f59400ae600 legacy=0x7f59400ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:14.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.750+0000 7f59454ab700 1 -- 172.21.15.1:0/2037419051 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594006e4a0 con 0x7f594015a2e0 2024-03-18T20:27:14.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.750+0000 7f5944caa700 1 -- 172.21.15.1:0/2037419051 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3388776798 0 0) 0x7f594006e270 con 0x7f59400ae600 2024-03-18T20:27:14.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.750+0000 7f59454ab700 1 --2- 172.21.15.1:0/2037419051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f594015a2e0 0x7f59400ad790 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7f59400cc100 tx=0x7f5930009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2596b9d27ea91834 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:14.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.750+0000 7f5944caa700 1 -- 172.21.15.1:0/2037419051 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5930011830 con 0x7f594015a2e0 2024-03-18T20:27:14.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.750+0000 7f5944caa700 1 -- 172.21.15.1:0/2037419051 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5930011990 con 0x7f594015a2e0 2024-03-18T20:27:14.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.750+0000 7f5944caa700 1 -- 172.21.15.1:0/2037419051 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5930011d20 con 0x7f594015a2e0 2024-03-18T20:27:14.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.750+0000 7f5947f10700 1 -- 172.21.15.1:0/2037419051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f594015a2e0 msgr2=0x7f59400ad790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:14.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.750+0000 7f5947f10700 1 --2- 172.21.15.1:0/2037419051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f594015a2e0 0x7f59400ad790 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7f59400cc100 tx=0x7f5930009b20 comp rx=0 tx=0).stop 2024-03-18T20:27:14.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.750+0000 7f5947f10700 1 -- 172.21.15.1:0/2037419051 shutdown_connections 2024-03-18T20:27:14.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.750+0000 7f5947f10700 1 --2- 172.21.15.1:0/2037419051 >> v2:172.21.15.165:3300/0 conn(0x7f59400cc170 0x7f594006ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:14.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.750+0000 7f5947f10700 1 --2- 172.21.15.1:0/2037419051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f594015a2e0 0x7f59400ad790 unknown :-1 s=CLOSED pgs=3024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:14.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.750+0000 7f5947f10700 1 -- 172.21.15.1:0/2037419051 >> 172.21.15.1:0/2037419051 conn(0x7f594006b860 msgr2=0x7f594006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:14.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.751+0000 7f5947f10700 1 -- 172.21.15.1:0/2037419051 shutdown_connections 2024-03-18T20:27:14.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.751+0000 7f5947f10700 1 -- 172.21.15.1:0/2037419051 wait complete. 2024-03-18T20:27:14.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.752+0000 7f5947f10700 1 Processor -- start 2024-03-18T20:27:14.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.752+0000 7f5947f10700 1 -- start start 2024-03-18T20:27:14.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.753+0000 7f5947f10700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59400ae600 0x7f59400cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:14.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.753+0000 7f5947f10700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59400cc170 0x7f5940156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:14.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.753+0000 7f5947f10700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f594015a2e0 0x7f594014c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:14.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.753+0000 7f5947f10700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59401528e0 con 0x7f59400cc170 2024-03-18T20:27:14.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.753+0000 7f5947f10700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59400cd690 con 0x7f59400ae600 2024-03-18T20:27:14.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.753+0000 7f5947f10700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5940151fd0 con 0x7f594015a2e0 2024-03-18T20:27:14.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.753+0000 7f59454ab700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59400cc170 0x7f5940156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:14.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.753+0000 7f59454ab700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59400cc170 0x7f5940156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49934/0 (socket says 172.21.15.1:49934) 2024-03-18T20:27:14.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.753+0000 7f59454ab700 1 -- 172.21.15.1:0/1978048513 learned_addr learned my addr 172.21.15.1:0/1978048513 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:14.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.753+0000 7f5945cac700 1 --2- 172.21.15.1:0/1978048513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59400ae600 0x7f59400cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:14.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.753+0000 7f59464ad700 1 --2- 172.21.15.1:0/1978048513 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f594015a2e0 0x7f594014c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:14.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.754+0000 7f59454ab700 1 -- 172.21.15.1:0/1978048513 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f594015a2e0 msgr2=0x7f594014c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:14.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.754+0000 7f59454ab700 1 --2- 172.21.15.1:0/1978048513 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f594015a2e0 0x7f594014c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:14.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.754+0000 7f59454ab700 1 -- 172.21.15.1:0/1978048513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59400ae600 msgr2=0x7f59400cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:14.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.754+0000 7f59454ab700 1 --2- 172.21.15.1:0/1978048513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59400ae600 0x7f59400cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:14.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.754+0000 7f59454ab700 1 -- 172.21.15.1:0/1978048513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59300094a0 con 0x7f59400cc170 2024-03-18T20:27:14.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.754+0000 7f59454ab700 1 --2- 172.21.15.1:0/1978048513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59400cc170 0x7f5940156d00 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7f5930014040 tx=0x7f5930004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:14.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.754+0000 7f59367fc700 1 -- 172.21.15.1:0/1978048513 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59300036a0 con 0x7f59400cc170 2024-03-18T20:27:14.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.754+0000 7f59367fc700 1 -- 172.21.15.1:0/1978048513 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593001e070 con 0x7f59400cc170 2024-03-18T20:27:14.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.754+0000 7f5947f10700 1 -- 172.21.15.1:0/1978048513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5940152290 con 0x7f59400cc170 2024-03-18T20:27:14.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.754+0000 7f59367fc700 1 -- 172.21.15.1:0/1978048513 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5930003a00 con 0x7f59400cc170 2024-03-18T20:27:14.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.754+0000 7f5947f10700 1 -- 172.21.15.1:0/1978048513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594014c740 con 0x7f59400cc170 2024-03-18T20:27:14.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.755+0000 7f5947f10700 1 -- 172.21.15.1:0/1978048513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59400676a0 con 0x7f59400cc170 2024-03-18T20:27:14.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.755+0000 7f59367fc700 1 -- 172.21.15.1:0/1978048513 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f5930003b60 con 0x7f59400cc170 2024-03-18T20:27:14.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.756+0000 7f59367fc700 1 --2- 172.21.15.1:0/1978048513 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f592406aed0 0x7f592406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:14.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.756+0000 7f59367fc700 1 -- 172.21.15.1:0/1978048513 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6223+0+0 (secure 0 0 0) 0x7f593008a930 con 0x7f59400cc170 2024-03-18T20:27:14.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.756+0000 7f5945cac700 1 --2- 172.21.15.1:0/1978048513 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f592406aed0 0x7f592406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:14.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.756+0000 7f5945cac700 1 --2- 172.21.15.1:0/1978048513 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f592406aed0 0x7f592406d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f593c007fd0 tx=0x7f593c007de0 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:14.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:14.760+0000 7f59367fc700 1 -- 172.21.15.1:0/1978048513 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f59300555a0 con 0x7f59400cc170 2024-03-18T20:27:15.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.047+0000 7f5947f10700 1 -- 172.21.15.1:0/1978048513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f594006c420 con 0x7f59400cc170 2024-03-18T20:27:15.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.599+0000 7f59367fc700 1 -- 172.21.15.1:0/1978048513 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3211) v1 ==== 158+0+0 (secure 0 0 0) 0x7f5930058bc0 con 0x7f59400cc170 2024-03-18T20:27:15.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.602+0000 7f5947f10700 1 -- 172.21.15.1:0/1978048513 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f592406aed0 msgr2=0x7f592406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:15.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.602+0000 7f5947f10700 1 --2- 172.21.15.1:0/1978048513 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f592406aed0 0x7f592406d390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f593c007fd0 tx=0x7f593c007de0 comp rx=0 tx=0).stop 2024-03-18T20:27:15.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.602+0000 7f5947f10700 1 -- 172.21.15.1:0/1978048513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59400cc170 msgr2=0x7f5940156d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:15.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.602+0000 7f5947f10700 1 --2- 172.21.15.1:0/1978048513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59400cc170 0x7f5940156d00 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7f5930014040 tx=0x7f5930004750 comp rx=0 tx=0).stop 2024-03-18T20:27:15.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.602+0000 7f5947f10700 1 -- 172.21.15.1:0/1978048513 shutdown_connections 2024-03-18T20:27:15.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.602+0000 7f5947f10700 1 --2- 172.21.15.1:0/1978048513 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f592406aed0 0x7f592406d390 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:15.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.602+0000 7f5947f10700 1 --2- 172.21.15.1:0/1978048513 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f594015a2e0 0x7f594014c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:15.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.602+0000 7f5947f10700 1 --2- 172.21.15.1:0/1978048513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59400ae600 0x7f59400cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:15.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.602+0000 7f5947f10700 1 --2- 172.21.15.1:0/1978048513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59400cc170 0x7f5940156d00 unknown :-1 s=CLOSED pgs=3025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:15.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.602+0000 7f5947f10700 1 -- 172.21.15.1:0/1978048513 >> 172.21.15.1:0/1978048513 conn(0x7f594006b860 msgr2=0x7f59400b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:15.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.602+0000 7f5947f10700 1 -- 172.21.15.1:0/1978048513 shutdown_connections 2024-03-18T20:27:15.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.602+0000 7f5947f10700 1 -- 172.21.15.1:0/1978048513 wait complete. 2024-03-18T20:27:15.621 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-18T20:27:15.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.752+0000 7fde18be6700 1 Processor -- start 2024-03-18T20:27:15.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.752+0000 7fde18be6700 1 -- start start 2024-03-18T20:27:15.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.752+0000 7fde18be6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fde1415a1d0 0x7fde140add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:15.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.753+0000 7fde18be6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fde140cc0f0 0x7fde14071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:15.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.753+0000 7fde18be6700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde140ce8f0 con 0x7fde14071620 2024-03-18T20:27:15.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.753+0000 7fde18be6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde140cdb00 con 0x7fde140cc0f0 2024-03-18T20:27:15.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.753+0000 7fde18be6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde1415a5b0 con 0x7fde1415a1d0 2024-03-18T20:27:15.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.753+0000 7fde12d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fde140cc0f0 0x7fde14071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:15.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.753+0000 7fde11d9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fde14071620 0x7fde140cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33712/0 (socket says 172.21.15.1:33712) 2024-03-18T20:27:15.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.753+0000 7fde11d9b700 1 -- 172.21.15.1:0/4131445060 learned_addr learned my addr 172.21.15.1:0/4131445060 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:15.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.753+0000 7fde1259c700 1 --2- 172.21.15.1:0/4131445060 >> v2:172.21.15.165:3300/0 conn(0x7fde1415a1d0 0x7fde140add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:15.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.753+0000 7fde12d9d700 1 -- 172.21.15.1:0/4131445060 >> v2:172.21.15.165:3300/0 conn(0x7fde1415a1d0 msgr2=0x7fde140add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:15.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.753+0000 7fde12d9d700 1 --2- 172.21.15.1:0/4131445060 >> v2:172.21.15.165:3300/0 conn(0x7fde1415a1d0 0x7fde140add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:15.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.753+0000 7fde12d9d700 1 -- 172.21.15.1:0/4131445060 >> v1:172.21.15.1:6789/0 conn(0x7fde14071620 legacy=0x7fde140cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:15.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.753+0000 7fde12d9d700 1 -- 172.21.15.1:0/4131445060 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde14071fc0 con 0x7fde140cc0f0 2024-03-18T20:27:15.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.753+0000 7fde1159a700 1 -- 172.21.15.1:0/4131445060 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451716333 0 0) 0x7fde140ce8f0 con 0x7fde14071620 2024-03-18T20:27:15.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.753+0000 7fde12d9d700 1 --2- 172.21.15.1:0/4131445060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde140cc0f0 0x7fde14071a80 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 crypto rx=0x7fde140726b0 tx=0x7fde0c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a002051d493602b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:15.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.753+0000 7fde1159a700 1 -- 172.21.15.1:0/4131445060 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c00e070 con 0x7fde140cc0f0 2024-03-18T20:27:15.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.753+0000 7fde1159a700 1 -- 172.21.15.1:0/4131445060 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0c00bbd0 con 0x7fde140cc0f0 2024-03-18T20:27:15.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.753+0000 7fde1159a700 1 -- 172.21.15.1:0/4131445060 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c012430 con 0x7fde140cc0f0 2024-03-18T20:27:15.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.754+0000 7fde18be6700 1 -- 172.21.15.1:0/4131445060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde140cc0f0 msgr2=0x7fde14071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:15.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.754+0000 7fde18be6700 1 --2- 172.21.15.1:0/4131445060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde140cc0f0 0x7fde14071a80 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 crypto rx=0x7fde140726b0 tx=0x7fde0c00b330 comp rx=0 tx=0).stop 2024-03-18T20:27:15.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.754+0000 7fde18be6700 1 -- 172.21.15.1:0/4131445060 shutdown_connections 2024-03-18T20:27:15.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.754+0000 7fde18be6700 1 --2- 172.21.15.1:0/4131445060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde140cc0f0 0x7fde14071a80 unknown :-1 s=CLOSED pgs=3027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:15.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.754+0000 7fde18be6700 1 --2- 172.21.15.1:0/4131445060 >> v2:172.21.15.165:3300/0 conn(0x7fde1415a1d0 0x7fde140add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:15.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.754+0000 7fde18be6700 1 -- 172.21.15.1:0/4131445060 >> 172.21.15.1:0/4131445060 conn(0x7fde140cc9b0 msgr2=0x7fde140ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:15.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.754+0000 7fde18be6700 1 -- 172.21.15.1:0/4131445060 shutdown_connections 2024-03-18T20:27:15.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.754+0000 7fde18be6700 1 -- 172.21.15.1:0/4131445060 wait complete. 2024-03-18T20:27:15.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.755+0000 7fde18be6700 1 Processor -- start 2024-03-18T20:27:15.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.755+0000 7fde18be6700 1 -- start start 2024-03-18T20:27:15.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.756+0000 7fde18be6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde14071620 0x7fde140c46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:15.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.756+0000 7fde18be6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde140cc0f0 0x7fde140c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:15.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.756+0000 7fde18be6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde1415a1d0 0x7fde140b0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:15.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.756+0000 7fde18be6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde140c88d0 con 0x7fde1415a1d0 2024-03-18T20:27:15.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.756+0000 7fde18be6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde140c85f0 con 0x7fde14071620 2024-03-18T20:27:15.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.756+0000 7fde18be6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde140c3be0 con 0x7fde140cc0f0 2024-03-18T20:27:15.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.756+0000 7fde12d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde1415a1d0 0x7fde140b0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:15.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.756+0000 7fde12d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde1415a1d0 0x7fde140b0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49940/0 (socket says 172.21.15.1:49940) 2024-03-18T20:27:15.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.756+0000 7fde12d9d700 1 -- 172.21.15.1:0/3489906852 learned_addr learned my addr 172.21.15.1:0/3489906852 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:15.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.757+0000 7fde1259c700 1 --2- 172.21.15.1:0/3489906852 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde14071620 0x7fde140c46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:15.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.757+0000 7fde11d9b700 1 --2- 172.21.15.1:0/3489906852 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde140cc0f0 0x7fde140c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:15.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.757+0000 7fde12d9d700 1 -- 172.21.15.1:0/3489906852 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde140cc0f0 msgr2=0x7fde140c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:15.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.757+0000 7fde12d9d700 1 --2- 172.21.15.1:0/3489906852 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde140cc0f0 0x7fde140c7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:15.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.757+0000 7fde12d9d700 1 -- 172.21.15.1:0/3489906852 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde14071620 msgr2=0x7fde140c46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:15.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.757+0000 7fde12d9d700 1 --2- 172.21.15.1:0/3489906852 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde14071620 0x7fde140c46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:15.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.757+0000 7fde12d9d700 1 -- 172.21.15.1:0/3489906852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0c009a70 con 0x7fde1415a1d0 2024-03-18T20:27:15.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.757+0000 7fde1259c700 1 --2- 172.21.15.1:0/3489906852 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde14071620 0x7fde140c46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.757+0000 7fde12d9d700 1 --2- 172.21.15.1:0/3489906852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde1415a1d0 0x7fde140b0620 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7fde0c013000 tx=0x7fde0c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.757+0000 7fde02ffd700 1 -- 172.21.15.1:0/3489906852 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c012430 con 0x7fde1415a1d0 2024-03-18T20:27:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.757+0000 7fde02ffd700 1 -- 172.21.15.1:0/3489906852 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0c020070 con 0x7fde1415a1d0 2024-03-18T20:27:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.757+0000 7fde18be6700 1 -- 172.21.15.1:0/3489906852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde140c3ec0 con 0x7fde1415a1d0 2024-03-18T20:27:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.757+0000 7fde02ffd700 1 -- 172.21.15.1:0/3489906852 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c004280 con 0x7fde1415a1d0 2024-03-18T20:27:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.757+0000 7fde18be6700 1 -- 172.21.15.1:0/3489906852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde140c8200 con 0x7fde1415a1d0 2024-03-18T20:27:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.758+0000 7fde18be6700 1 -- 172.21.15.1:0/3489906852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde140c43e0 con 0x7fde1415a1d0 2024-03-18T20:27:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.758+0000 7fde02ffd700 1 -- 172.21.15.1:0/3489906852 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fde0c0043e0 con 0x7fde1415a1d0 2024-03-18T20:27:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.759+0000 7fde02ffd700 1 --2- 172.21.15.1:0/3489906852 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fddf406aed0 0x7fddf406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.759+0000 7fde1259c700 1 --2- 172.21.15.1:0/3489906852 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fddf406aed0 0x7fddf406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.759+0000 7fde02ffd700 1 -- 172.21.15.1:0/3489906852 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6223+0+0 (secure 0 0 0) 0x7fde0c022070 con 0x7fde1415a1d0 2024-03-18T20:27:15.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.759+0000 7fde1259c700 1 --2- 172.21.15.1:0/3489906852 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fddf406aed0 0x7fddf406d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fde140c6a60 tx=0x7fde04008040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:15.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.764+0000 7fde02ffd700 1 -- 172.21.15.1:0/3489906852 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fde140c43e0 con 0x7fde1415a1d0 2024-03-18T20:27:15.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.967+0000 7fde18be6700 1 -- 172.21.15.1:0/3489906852 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fde140c43e0 con 0x7fde1415a1d0 2024-03-18T20:27:15.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.971+0000 7fde02ffd700 1 -- 172.21.15.1:0/3489906852 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v88)=0 v88) v1 ==== 122+0+0 (secure 0 0 0) 0x7fde140c43e0 con 0x7fde1415a1d0 2024-03-18T20:27:15.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.973+0000 7fde18be6700 1 -- 172.21.15.1:0/3489906852 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fddf406aed0 msgr2=0x7fddf406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:15.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.973+0000 7fde18be6700 1 --2- 172.21.15.1:0/3489906852 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fddf406aed0 0x7fddf406d390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fde140c6a60 tx=0x7fde04008040 comp rx=0 tx=0).stop 2024-03-18T20:27:15.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.973+0000 7fde18be6700 1 -- 172.21.15.1:0/3489906852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde1415a1d0 msgr2=0x7fde140b0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:15.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.973+0000 7fde18be6700 1 --2- 172.21.15.1:0/3489906852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde1415a1d0 0x7fde140b0620 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7fde0c013000 tx=0x7fde0c004ad0 comp rx=0 tx=0).stop 2024-03-18T20:27:15.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.974+0000 7fde18be6700 1 -- 172.21.15.1:0/3489906852 shutdown_connections 2024-03-18T20:27:15.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.974+0000 7fde18be6700 1 --2- 172.21.15.1:0/3489906852 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fddf406aed0 0x7fddf406d390 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:15.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.974+0000 7fde18be6700 1 --2- 172.21.15.1:0/3489906852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde1415a1d0 0x7fde140b0620 unknown :-1 s=CLOSED pgs=3028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:15.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.974+0000 7fde18be6700 1 --2- 172.21.15.1:0/3489906852 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde14071620 0x7fde140c46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:15.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.974+0000 7fde18be6700 1 --2- 172.21.15.1:0/3489906852 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde140cc0f0 0x7fde140c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:15.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.974+0000 7fde18be6700 1 -- 172.21.15.1:0/3489906852 >> 172.21.15.1:0/3489906852 conn(0x7fde140cc9b0 msgr2=0x7fde140ca560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:15.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.974+0000 7fde18be6700 1 -- 172.21.15.1:0/3489906852 shutdown_connections 2024-03-18T20:27:15.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:15.974+0000 7fde18be6700 1 -- 172.21.15.1:0/3489906852 wait complete. 2024-03-18T20:27:15.990 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T20:27:15.990 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:27:16.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:27:16.075 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T20:27:16.076 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:27:16.076 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:27:16.095 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:27:16.095 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:27:16.095 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:27:16.136 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:27:16.137 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:27:16.137 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:27:16.155 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:27:16.155 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:27:16.155 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:27:16.197 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:27:16.197 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-18T20:27:16.197 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:27:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.317+0000 7fd4b4525700 1 Processor -- start 2024-03-18T20:27:16.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.317+0000 7fd4b4525700 1 -- start start 2024-03-18T20:27:16.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.318+0000 7fd4b4525700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd4ac0b6b60 0x7fd4ac0cc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:16.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.318+0000 7fd4b4525700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4ac0cee90 0x7fd4ac0cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:16.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.318+0000 7fd4b4525700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ac0cd0d0 con 0x7fd4ac0cfbd0 2024-03-18T20:27:16.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.318+0000 7fd4b4525700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ac0d0060 con 0x7fd4ac0b6b60 2024-03-18T20:27:16.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.318+0000 7fd4b4525700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0b3f00 con 0x7fd4ac0cee90 2024-03-18T20:27:16.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.318+0000 7fd4b22c1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd4ac0b6b60 0x7fd4ac0cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:16.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.318+0000 7fd4b2ac2700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd4ac0cfbd0 0x7fd4ac099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33714/0 (socket says 172.21.15.1:33714) 2024-03-18T20:27:16.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.318+0000 7fd4b2ac2700 1 -- 172.21.15.1:0/2483123867 learned_addr learned my addr 172.21.15.1:0/2483123867 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:16.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.318+0000 7fd4b1ac0700 1 --2- 172.21.15.1:0/2483123867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4ac0cee90 0x7fd4ac0cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:16.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.318+0000 7fd4b12bf700 1 -- 172.21.15.1:0/2483123867 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538696487 0 0) 0x7fd4ac0cd0d0 con 0x7fd4ac0cfbd0 2024-03-18T20:27:16.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.318+0000 7fd4b22c1700 1 -- 172.21.15.1:0/2483123867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4ac0cee90 msgr2=0x7fd4ac0cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:16.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.318+0000 7fd4b22c1700 1 --2- 172.21.15.1:0/2483123867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4ac0cee90 0x7fd4ac0cf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:16.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.318+0000 7fd4b22c1700 1 -- 172.21.15.1:0/2483123867 >> v1:172.21.15.1:6789/0 conn(0x7fd4ac0cfbd0 legacy=0x7fd4ac099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:16.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.318+0000 7fd4b22c1700 1 -- 172.21.15.1:0/2483123867 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac0b41e0 con 0x7fd4ac0b6b60 2024-03-18T20:27:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.318+0000 7fd4b22c1700 1 --2- 172.21.15.1:0/2483123867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4ac0b6b60 0x7fd4ac0cc830 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 crypto rx=0x7fd49c009840 tx=0x7fd49c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=717bc668a450905b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.318+0000 7fd4b12bf700 1 -- 172.21.15.1:0/2483123867 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c011830 con 0x7fd4ac0b6b60 2024-03-18T20:27:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.318+0000 7fd4b12bf700 1 -- 172.21.15.1:0/2483123867 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49c011990 con 0x7fd4ac0b6b60 2024-03-18T20:27:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.318+0000 7fd4b12bf700 1 -- 172.21.15.1:0/2483123867 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c011d20 con 0x7fd4ac0b6b60 2024-03-18T20:27:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.319+0000 7fd4b4525700 1 -- 172.21.15.1:0/2483123867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4ac0b6b60 msgr2=0x7fd4ac0cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.319+0000 7fd4b4525700 1 --2- 172.21.15.1:0/2483123867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4ac0b6b60 0x7fd4ac0cc830 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 crypto rx=0x7fd49c009840 tx=0x7fd49c009b50 comp rx=0 tx=0).stop 2024-03-18T20:27:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.319+0000 7fd4b4525700 1 -- 172.21.15.1:0/2483123867 shutdown_connections 2024-03-18T20:27:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.319+0000 7fd4b4525700 1 --2- 172.21.15.1:0/2483123867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4ac0b6b60 0x7fd4ac0cc830 unknown :-1 s=CLOSED pgs=3030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.319+0000 7fd4b4525700 1 --2- 172.21.15.1:0/2483123867 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4ac0cee90 0x7fd4ac0cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.319+0000 7fd4b4525700 1 -- 172.21.15.1:0/2483123867 >> 172.21.15.1:0/2483123867 conn(0x7fd4ac0b2ba0 msgr2=0x7fd4ac0b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.319+0000 7fd4b4525700 1 -- 172.21.15.1:0/2483123867 shutdown_connections 2024-03-18T20:27:16.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.319+0000 7fd4b4525700 1 -- 172.21.15.1:0/2483123867 wait complete. 2024-03-18T20:27:16.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.320+0000 7fd4b4525700 1 Processor -- start 2024-03-18T20:27:16.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.320+0000 7fd4b4525700 1 -- start start 2024-03-18T20:27:16.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.321+0000 7fd4b4525700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4ac0b6b60 0x7fd4ac15a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:16.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.321+0000 7fd4b4525700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4ac0cee90 0x7fd4ac06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:16.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.321+0000 7fd4b4525700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4ac0cfbd0 0x7fd4ac1f6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:16.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.321+0000 7fd4b4525700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac06d690 con 0x7fd4ac0cee90 2024-03-18T20:27:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.321+0000 7fd4b4525700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac15bf10 con 0x7fd4ac0cfbd0 2024-03-18T20:27:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.321+0000 7fd4b4525700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac15acc0 con 0x7fd4ac0b6b60 2024-03-18T20:27:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.321+0000 7fd4b1ac0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4ac0cee90 0x7fd4ac06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.321+0000 7fd4b1ac0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4ac0cee90 0x7fd4ac06d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49952/0 (socket says 172.21.15.1:49952) 2024-03-18T20:27:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.321+0000 7fd4b1ac0700 1 -- 172.21.15.1:0/564902647 learned_addr learned my addr 172.21.15.1:0/564902647 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.321+0000 7fd4b22c1700 1 --2- 172.21.15.1:0/564902647 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4ac0b6b60 0x7fd4ac15a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.321+0000 7fd4b2ac2700 1 --2- 172.21.15.1:0/564902647 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4ac0cfbd0 0x7fd4ac1f6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.321+0000 7fd4b1ac0700 1 -- 172.21.15.1:0/564902647 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4ac0b6b60 msgr2=0x7fd4ac15a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.321+0000 7fd4b1ac0700 1 --2- 172.21.15.1:0/564902647 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4ac0b6b60 0x7fd4ac15a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.321+0000 7fd4b1ac0700 1 -- 172.21.15.1:0/564902647 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4ac0cfbd0 msgr2=0x7fd4ac1f6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.321+0000 7fd4b1ac0700 1 --2- 172.21.15.1:0/564902647 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4ac0cfbd0 0x7fd4ac1f6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.321+0000 7fd4b1ac0700 1 -- 172.21.15.1:0/564902647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49c0094a0 con 0x7fd4ac0cee90 2024-03-18T20:27:16.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.322+0000 7fd4b1ac0700 1 --2- 172.21.15.1:0/564902647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4ac0cee90 0x7fd4ac06d150 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7fd4a800e810 tx=0x7fd4a800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.322+0000 7fd4a2ffd700 1 -- 172.21.15.1:0/564902647 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8013610 con 0x7fd4ac0cee90 2024-03-18T20:27:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.322+0000 7fd4a2ffd700 1 -- 172.21.15.1:0/564902647 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a8013770 con 0x7fd4ac0cee90 2024-03-18T20:27:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.322+0000 7fd4a2ffd700 1 -- 172.21.15.1:0/564902647 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8013b20 con 0x7fd4ac0cee90 2024-03-18T20:27:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.322+0000 7fd4b4525700 1 -- 172.21.15.1:0/564902647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ac15afa0 con 0x7fd4ac0cee90 2024-03-18T20:27:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.322+0000 7fd4b4525700 1 -- 172.21.15.1:0/564902647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ac15b560 con 0x7fd4ac0cee90 2024-03-18T20:27:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.323+0000 7fd4b4525700 1 -- 172.21.15.1:0/564902647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4ac15bce0 con 0x7fd4ac0cee90 2024-03-18T20:27:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.323+0000 7fd4a2ffd700 1 -- 172.21.15.1:0/564902647 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fd4a8013c80 con 0x7fd4ac0cee90 2024-03-18T20:27:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.324+0000 7fd4a2ffd700 1 --2- 172.21.15.1:0/564902647 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd49006aed0 0x7fd49006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.324+0000 7fd4b22c1700 1 --2- 172.21.15.1:0/564902647 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd49006aed0 0x7fd49006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.324+0000 7fd4a2ffd700 1 -- 172.21.15.1:0/564902647 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 6223+0+0 (secure 0 0 0) 0x7fd4a8089760 con 0x7fd4ac0cee90 2024-03-18T20:27:16.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.324+0000 7fd4b22c1700 1 --2- 172.21.15.1:0/564902647 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd49006aed0 0x7fd49006d390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd49c0090b0 tx=0x7fd49c008030 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:16.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.328+0000 7fd4a2ffd700 1 -- 172.21.15.1:0/564902647 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd4ac15bce0 con 0x7fd4ac0cee90 2024-03-18T20:27:16.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.577+0000 7fd4b4525700 1 -- 172.21.15.1:0/564902647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4ac15c080 con 0x7fd4ac0cee90 2024-03-18T20:27:16.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.577+0000 7fd4a2ffd700 1 -- 172.21.15.1:0/564902647 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+3533 (secure 0 0 0) 0x7fd4a8021de0 con 0x7fd4ac0cee90 2024-03-18T20:27:16.582 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:16.583 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":130,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9851,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/3262033681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":3262033681},{"type":"v1","addr":"172.21.15.165:6835","nonce":3262033681}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":128},{"gid":9890,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/2114242732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":2114242732},{"type":"v1","addr":"172.21.15.165:6834","nonce":2114242732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130},{"gid":12396,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/2565889517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2565889517},{"type":"v1","addr":"172.21.15.1:6836","nonce":2565889517}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115},{"gid":12399,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/512665659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":512665659},{"type":"v1","addr":"172.21.15.1:6837","nonce":512665659}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[]} 2024-03-18T20:27:16.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.579+0000 7fd4b4525700 1 -- 172.21.15.1:0/564902647 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd49006aed0 msgr2=0x7fd49006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:16.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.579+0000 7fd4b4525700 1 --2- 172.21.15.1:0/564902647 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd49006aed0 0x7fd49006d390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd49c0090b0 tx=0x7fd49c008030 comp rx=0 tx=0).stop 2024-03-18T20:27:16.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.579+0000 7fd4b4525700 1 -- 172.21.15.1:0/564902647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4ac0cee90 msgr2=0x7fd4ac06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:16.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.579+0000 7fd4b4525700 1 --2- 172.21.15.1:0/564902647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4ac0cee90 0x7fd4ac06d150 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7fd4a800e810 tx=0x7fd4a800ebd0 comp rx=0 tx=0).stop 2024-03-18T20:27:16.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.579+0000 7fd4b4525700 1 -- 172.21.15.1:0/564902647 shutdown_connections 2024-03-18T20:27:16.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.579+0000 7fd4b4525700 1 --2- 172.21.15.1:0/564902647 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd49006aed0 0x7fd49006d390 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:16.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.579+0000 7fd4b4525700 1 --2- 172.21.15.1:0/564902647 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd4ac0b6b60 0x7fd4ac15a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:16.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.579+0000 7fd4b4525700 1 --2- 172.21.15.1:0/564902647 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd4ac0cfbd0 0x7fd4ac1f6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:16.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.579+0000 7fd4b4525700 1 --2- 172.21.15.1:0/564902647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd4ac0cee90 0x7fd4ac06d150 unknown :-1 s=CLOSED pgs=3031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:16.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.579+0000 7fd4b4525700 1 -- 172.21.15.1:0/564902647 >> 172.21.15.1:0/564902647 conn(0x7fd4ac0b2ba0 msgr2=0x7fd4ac06ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:16.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.580+0000 7fd4b4525700 1 -- 172.21.15.1:0/564902647 shutdown_connections 2024-03-18T20:27:16.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.580+0000 7fd4b4525700 1 -- 172.21.15.1:0/564902647 wait complete. 2024-03-18T20:27:16.586 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 130 2024-03-18T20:27:16.597 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T20:27:16.597 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-18T20:27:16.597 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T20:27:16.597 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:27:16.598 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T20:27:16.598 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-18T20:27:16.598 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-18T20:27:16.598 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:27:16.598 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T20:27:16.598 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-18T20:27:16.599 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-18T20:27:16.599 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:27:16.599 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T20:27:16.599 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-18T20:27:16.599 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-18T20:27:16.599 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:27:16.606 INFO:tasks.ceph.mds.a:Stopped 2024-03-18T20:27:16.607 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-18T20:27:16.608 INFO:tasks.ceph.mds.c:Stopped 2024-03-18T20:27:16.608 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-18T20:27:16.610 INFO:tasks.ceph.mds.b:Stopped 2024-03-18T20:27:16.610 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-18T20:27:16.611 INFO:tasks.ceph.mds.d:Stopped 2024-03-18T20:27:16.611 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-18T20:27:16.649 INFO:tasks.ceph.mds.a:Started 2024-03-18T20:27:16.649 DEBUG:teuthology.parallel:result is None 2024-03-18T20:27:16.650 INFO:tasks.ceph.mds.c:Started 2024-03-18T20:27:16.650 DEBUG:teuthology.parallel:result is None 2024-03-18T20:27:16.653 INFO:tasks.ceph.mds.b:Started 2024-03-18T20:27:16.653 DEBUG:teuthology.parallel:result is None 2024-03-18T20:27:16.654 INFO:tasks.ceph.mds.d:Started 2024-03-18T20:27:16.654 DEBUG:teuthology.parallel:result is None 2024-03-18T20:27:16.655 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-18T20:27:16.730 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-18T20:27:16.731 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T20:27:16.742 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-03-18T20:27:16.743 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-03-18T20:27:16.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.785+0000 7f7b2e718700 1 Processor -- start 2024-03-18T20:27:16.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.785+0000 7f7b2e718700 1 -- start start 2024-03-18T20:27:16.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.786+0000 7f7b2e718700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7b2813f740 0x7f7b2813fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:16.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.786+0000 7f7b2e718700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b28070610 con 0x7f7b28070aa0 2024-03-18T20:27:16.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.786+0000 7f7b2e718700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b280ce300 con 0x7f7b2813d090 2024-03-18T20:27:16.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.786+0000 7f7b2e718700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b2813d470 con 0x7f7b2813f740 2024-03-18T20:27:16.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.786+0000 7f7b277fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7b2813f740 0x7f7b2813fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:16.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.786+0000 7f7b277fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7b2813f740 0x7f7b2813fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50010/0 (socket says 172.21.15.1:50010) 2024-03-18T20:27:16.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.786+0000 7f7b2ccb5700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7b28070aa0 0x7f7b28070ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33724/0 (socket says 172.21.15.1:33724) 2024-03-18T20:27:16.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.786+0000 7f7b277fe700 1 -- 172.21.15.1:0/3451424625 learned_addr learned my addr 172.21.15.1:0/3451424625 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:16.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.786+0000 7f7b277fe700 1 -- 172.21.15.1:0/3451424625 >> v1:172.21.15.165:6789/0 conn(0x7f7b2813d090 legacy=0x7f7b280cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:16.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.786+0000 7f7b277fe700 1 -- 172.21.15.1:0/3451424625 >> v1:172.21.15.1:6789/0 conn(0x7f7b28070aa0 legacy=0x7f7b28070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:16.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.786+0000 7f7b277fe700 1 -- 172.21.15.1:0/3451424625 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b280ce5e0 con 0x7f7b2813f740 2024-03-18T20:27:16.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.786+0000 7f7b26ffd700 1 -- 172.21.15.1:0/3451424625 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914737368 0 0) 0x7f7b28070610 con 0x7f7b28070aa0 2024-03-18T20:27:16.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.786+0000 7f7b277fe700 1 --2- 172.21.15.1:0/3451424625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b2813f740 0x7f7b2813fbf0 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 crypto rx=0x7f7b1800bb90 tx=0x7f7b1800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed074539bc8a3463 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:16.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.786+0000 7f7b26ffd700 1 -- 172.21.15.1:0/3451424625 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1800dd10 con 0x7f7b2813f740 2024-03-18T20:27:16.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.787+0000 7f7b26ffd700 1 -- 172.21.15.1:0/3451424625 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1800de70 con 0x7f7b2813f740 2024-03-18T20:27:16.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.787+0000 7f7b26ffd700 1 -- 172.21.15.1:0/3451424625 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b18004230 con 0x7f7b2813f740 2024-03-18T20:27:16.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.787+0000 7f7b2e718700 1 -- 172.21.15.1:0/3451424625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b2813f740 msgr2=0x7f7b2813fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:16.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.787+0000 7f7b2e718700 1 --2- 172.21.15.1:0/3451424625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b2813f740 0x7f7b2813fbf0 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 crypto rx=0x7f7b1800bb90 tx=0x7f7b1800bea0 comp rx=0 tx=0).stop 2024-03-18T20:27:16.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.787+0000 7f7b2e718700 1 -- 172.21.15.1:0/3451424625 shutdown_connections 2024-03-18T20:27:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.787+0000 7f7b2e718700 1 --2- 172.21.15.1:0/3451424625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b2813f740 0x7f7b2813fbf0 unknown :-1 s=CLOSED pgs=3036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.787+0000 7f7b2e718700 1 -- 172.21.15.1:0/3451424625 >> 172.21.15.1:0/3451424625 conn(0x7f7b2806ab70 msgr2=0x7f7b2806af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.787+0000 7f7b2e718700 1 -- 172.21.15.1:0/3451424625 shutdown_connections 2024-03-18T20:27:16.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.787+0000 7f7b2e718700 1 -- 172.21.15.1:0/3451424625 wait complete. 2024-03-18T20:27:16.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.788+0000 7f7b2e718700 1 Processor -- start 2024-03-18T20:27:16.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.788+0000 7f7b2e718700 1 -- start start 2024-03-18T20:27:16.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.789+0000 7f7b2e718700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7b28070aa0 0x7f7b2813c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.789+0000 7f7b2e718700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b2813d090 0x7f7b281425f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.789+0000 7f7b2e718700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b2813f740 0x7f7b281f3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.789+0000 7f7b2e718700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b28140130 con 0x7f7b2813f740 2024-03-18T20:27:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.789+0000 7f7b2e718700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2815c370 con 0x7f7b2813d090 2024-03-18T20:27:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.789+0000 7f7b2e718700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b28143070 con 0x7f7b28070aa0 2024-03-18T20:27:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.789+0000 7f7b2ccb5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b2813f740 0x7f7b281f3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.789+0000 7f7b2ccb5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b2813f740 0x7f7b281f3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50024/0 (socket says 172.21.15.1:50024) 2024-03-18T20:27:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.789+0000 7f7b2ccb5700 1 -- 172.21.15.1:0/1632418625 learned_addr learned my addr 172.21.15.1:0/1632418625 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.789+0000 7f7b27fff700 1 --2- 172.21.15.1:0/1632418625 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7b28070aa0 0x7f7b2813c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.789+0000 7f7b277fe700 1 --2- 172.21.15.1:0/1632418625 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b2813d090 0x7f7b281425f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.790+0000 7f7b2ccb5700 1 -- 172.21.15.1:0/1632418625 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7b28070aa0 msgr2=0x7f7b2813c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.790+0000 7f7b2ccb5700 1 --2- 172.21.15.1:0/1632418625 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7b28070aa0 0x7f7b2813c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.790+0000 7f7b2ccb5700 1 -- 172.21.15.1:0/1632418625 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b2813d090 msgr2=0x7f7b281425f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:16.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.790+0000 7f7b2ccb5700 1 --2- 172.21.15.1:0/1632418625 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b2813d090 0x7f7b281425f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.790+0000 7f7b2ccb5700 1 -- 172.21.15.1:0/1632418625 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b1800b7f0 con 0x7f7b2813f740 2024-03-18T20:27:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.790+0000 7f7b277fe700 1 --2- 172.21.15.1:0/1632418625 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b2813d090 0x7f7b281425f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.790+0000 7f7b2ccb5700 1 --2- 172.21.15.1:0/1632418625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b2813f740 0x7f7b281f3370 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 crypto rx=0x7f7b2000cb60 tx=0x7f7b2000ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.790+0000 7f7b24ff9700 1 -- 172.21.15.1:0/1632418625 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2000aaa0 con 0x7f7b2813f740 2024-03-18T20:27:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.790+0000 7f7b24ff9700 1 -- 172.21.15.1:0/1632418625 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2000ac00 con 0x7f7b2813f740 2024-03-18T20:27:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.790+0000 7f7b2e718700 1 -- 172.21.15.1:0/1632418625 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b2815ba40 con 0x7f7b2813f740 2024-03-18T20:27:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.790+0000 7f7b24ff9700 1 -- 172.21.15.1:0/1632418625 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b200070d0 con 0x7f7b2813f740 2024-03-18T20:27:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.790+0000 7f7b2e718700 1 -- 172.21.15.1:0/1632418625 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b2815c000 con 0x7f7b2813f740 2024-03-18T20:27:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.791+0000 7f7b2e718700 1 -- 172.21.15.1:0/1632418625 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b28073350 con 0x7f7b2813f740 2024-03-18T20:27:16.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.791+0000 7f7b24ff9700 1 -- 172.21.15.1:0/1632418625 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f7b20007230 con 0x7f7b2813f740 2024-03-18T20:27:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.792+0000 7f7b24ff9700 1 --2- 172.21.15.1:0/1632418625 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7b0806aed0 0x7f7b0806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.792+0000 7f7b27fff700 1 --2- 172.21.15.1:0/1632418625 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7b0806aed0 0x7f7b0806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.792+0000 7f7b24ff9700 1 -- 172.21.15.1:0/1632418625 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 6223+0+0 (secure 0 0 0) 0x7f7b20089600 con 0x7f7b2813f740 2024-03-18T20:27:16.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.793+0000 7f7b27fff700 1 --2- 172.21.15.1:0/1632418625 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7b0806aed0 0x7f7b0806d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7b10007fd0 tx=0x7f7b10007ec0 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:16.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:16.797+0000 7f7b24ff9700 1 -- 172.21.15.1:0/1632418625 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7b20054310 con 0x7f7b2813f740 2024-03-18T20:27:17.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:17.014+0000 7f7b2e718700 1 -- 172.21.15.1:0/1632418625 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f7b2815c4b0 con 0x7f7b2813f740 2024-03-18T20:27:17.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:17.973+0000 7f7b24ff9700 1 -- 172.21.15.1:0/1632418625 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v270) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7b20057930 con 0x7f7b2813f740 2024-03-18T20:27:17.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:17.976+0000 7f7b2e718700 1 -- 172.21.15.1:0/1632418625 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7b0806aed0 msgr2=0x7f7b0806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:17.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:17.976+0000 7f7b2e718700 1 --2- 172.21.15.1:0/1632418625 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7b0806aed0 0x7f7b0806d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7b10007fd0 tx=0x7f7b10007ec0 comp rx=0 tx=0).stop 2024-03-18T20:27:17.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:17.976+0000 7f7b2e718700 1 -- 172.21.15.1:0/1632418625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b2813f740 msgr2=0x7f7b281f3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:17.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:17.976+0000 7f7b2e718700 1 --2- 172.21.15.1:0/1632418625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b2813f740 0x7f7b281f3370 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 crypto rx=0x7f7b2000cb60 tx=0x7f7b2000ce70 comp rx=0 tx=0).stop 2024-03-18T20:27:17.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:17.976+0000 7f7b2e718700 1 -- 172.21.15.1:0/1632418625 shutdown_connections 2024-03-18T20:27:17.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:17.976+0000 7f7b2e718700 1 --2- 172.21.15.1:0/1632418625 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7b0806aed0 0x7f7b0806d390 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:17.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:17.976+0000 7f7b2e718700 1 --2- 172.21.15.1:0/1632418625 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7b28070aa0 0x7f7b2813c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:17.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:17.976+0000 7f7b2e718700 1 --2- 172.21.15.1:0/1632418625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b2813f740 0x7f7b281f3370 unknown :-1 s=CLOSED pgs=3037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:17.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:17.976+0000 7f7b2e718700 1 --2- 172.21.15.1:0/1632418625 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b2813d090 0x7f7b281425f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:17.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:17.976+0000 7f7b2e718700 1 -- 172.21.15.1:0/1632418625 >> 172.21.15.1:0/1632418625 conn(0x7f7b2806ab70 msgr2=0x7f7b2806d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:17.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:17.977+0000 7f7b2e718700 1 -- 172.21.15.1:0/1632418625 shutdown_connections 2024-03-18T20:27:17.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:17.977+0000 7f7b2e718700 1 -- 172.21.15.1:0/1632418625 wait complete. 2024-03-18T20:27:17.983 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-18T20:27:17.996 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-18T20:27:18.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.123+0000 7efea79d9700 1 Processor -- start 2024-03-18T20:27:18.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.123+0000 7efea79d9700 1 -- start start 2024-03-18T20:27:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efea79d9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7efea0099ea0 0x7efea00cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efea79d9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efea00d0c80 con 0x7efea0099a40 2024-03-18T20:27:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efea79d9700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efea00d09a0 con 0x7efea015d680 2024-03-18T20:27:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efea79d9700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efea00cb830 con 0x7efea0099ea0 2024-03-18T20:27:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efea4f74700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efea0099a40 0x7efea00cc310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33740/0 (socket says 172.21.15.1:33740) 2024-03-18T20:27:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efea4f74700 1 -- 172.21.15.1:0/3858425738 learned_addr learned my addr 172.21.15.1:0/3858425738 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:18.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efe97fff700 1 -- 172.21.15.1:0/3858425738 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012658020 0 0) 0x7efea00d0c80 con 0x7efea0099a40 2024-03-18T20:27:18.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efe97fff700 1 -- 172.21.15.1:0/3858425738 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe8c0035b0 con 0x7efea0099a40 2024-03-18T20:27:18.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efea5f76700 1 --2- 172.21.15.1:0/3858425738 >> v2:172.21.15.165:3300/0 conn(0x7efea0099ea0 0x7efea00cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efe97fff700 1 -- 172.21.15.1:0/3858425738 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4274288139 0 0) 0x7efe8c0035b0 con 0x7efea0099a40 2024-03-18T20:27:18.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efe97fff700 1 -- 172.21.15.1:0/3858425738 >> v2:172.21.15.165:3300/0 conn(0x7efea0099ea0 msgr2=0x7efea00cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:18.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efe97fff700 1 --2- 172.21.15.1:0/3858425738 >> v2:172.21.15.165:3300/0 conn(0x7efea0099ea0 0x7efea00cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efe97fff700 1 -- 172.21.15.1:0/3858425738 >> v1:172.21.15.165:6789/0 conn(0x7efea015d680 legacy=0x7efea00cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:18.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efe97fff700 1 -- 172.21.15.1:0/3858425738 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea00cbb10 con 0x7efea0099a40 2024-03-18T20:27:18.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efe97fff700 1 -- 172.21.15.1:0/3858425738 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7efe90001670 con 0x7efea0099a40 2024-03-18T20:27:18.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efe97fff700 1 -- 172.21.15.1:0/3858425738 >> v1:172.21.15.1:6789/0 conn(0x7efea0099a40 legacy=0x7efea00cc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:18.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efe97fff700 1 --2- 172.21.15.1:0/3858425738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe8c003a00 0x7efe8c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efe97fff700 1 --2- 172.21.15.1:0/3858425738 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe8c004850 0x7efe8c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efe97fff700 1 --2- 172.21.15.1:0/3858425738 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe8c005210 0x7efe8c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efe97fff700 1 -- 172.21.15.1:0/3858425738 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea00cb830 con 0x7efe8c003a00 2024-03-18T20:27:18.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efe97fff700 1 -- 172.21.15.1:0/3858425738 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8c00c0b0 con 0x7efe8c004850 2024-03-18T20:27:18.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.124+0000 7efe97fff700 1 -- 172.21.15.1:0/3858425738 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe8c00c270 con 0x7efe8c005210 2024-03-18T20:27:18.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.125+0000 7efea5775700 1 --2- 172.21.15.1:0/3858425738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe8c003a00 0x7efe8c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.125+0000 7efea4f74700 1 --2- 172.21.15.1:0/3858425738 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe8c004850 0x7efe8c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.125+0000 7efea5775700 1 -- 172.21.15.1:0/3858425738 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe8c005210 msgr2=0x7efe8c00ba10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T20:27:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.125+0000 7efea5775700 1 --2- 172.21.15.1:0/3858425738 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe8c005210 0x7efe8c00ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.125+0000 7efea5775700 1 -- 172.21.15.1:0/3858425738 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe8c004850 msgr2=0x7efe8c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.125+0000 7efea5775700 1 --2- 172.21.15.1:0/3858425738 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe8c004850 0x7efe8c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.125+0000 7efea5775700 1 -- 172.21.15.1:0/3858425738 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8c0035b0 con 0x7efe8c003a00 2024-03-18T20:27:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.125+0000 7efea4f74700 1 --2- 172.21.15.1:0/3858425738 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe8c004850 0x7efe8c004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.125+0000 7efea5f76700 1 -- 172.21.15.1:0/3858425738 reap_dead start 2024-03-18T20:27:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.125+0000 7efea5775700 1 --2- 172.21.15.1:0/3858425738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe8c003a00 0x7efe8c003e20 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7efe9c0017c0 tx=0x7efe8c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d176b48a4c5c4f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.125+0000 7efe97fff700 1 -- 172.21.15.1:0/3858425738 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c0052d0 con 0x7efe8c003a00 2024-03-18T20:27:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.125+0000 7efe97fff700 1 -- 172.21.15.1:0/3858425738 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9c005430 con 0x7efe8c003a00 2024-03-18T20:27:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.125+0000 7efe97fff700 1 -- 172.21.15.1:0/3858425738 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c0038c0 con 0x7efe8c003a00 2024-03-18T20:27:18.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.126+0000 7efea79d9700 1 -- 172.21.15.1:0/3858425738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe8c003a00 msgr2=0x7efe8c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.126+0000 7efea79d9700 1 --2- 172.21.15.1:0/3858425738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe8c003a00 0x7efe8c003e20 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7efe9c0017c0 tx=0x7efe8c005880 comp rx=0 tx=0).stop 2024-03-18T20:27:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.126+0000 7efea79d9700 1 -- 172.21.15.1:0/3858425738 shutdown_connections 2024-03-18T20:27:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.126+0000 7efea79d9700 1 --2- 172.21.15.1:0/3858425738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe8c003a00 0x7efe8c003e20 unknown :-1 s=CLOSED pgs=3039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.126+0000 7efea79d9700 1 -- 172.21.15.1:0/3858425738 >> 172.21.15.1:0/3858425738 conn(0x7efea00b2ba0 msgr2=0x7efea00b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.126+0000 7efea79d9700 1 -- 172.21.15.1:0/3858425738 shutdown_connections 2024-03-18T20:27:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.126+0000 7efea79d9700 1 -- 172.21.15.1:0/3858425738 wait complete. 2024-03-18T20:27:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.127+0000 7efea79d9700 1 Processor -- start 2024-03-18T20:27:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.127+0000 7efea79d9700 1 -- start start 2024-03-18T20:27:18.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.128+0000 7efea79d9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe8c003a00 0x7efea00c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.128+0000 7efea79d9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea014e010 0x7efea014e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.128+0000 7efea79d9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efea00c8180 0x7efea00c8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.128+0000 7efea79d9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea00ae1e0 con 0x7efea014e010 2024-03-18T20:27:18.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.128+0000 7efea79d9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea014e9d0 con 0x7efe8c003a00 2024-03-18T20:27:18.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.128+0000 7efea79d9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea01605d0 con 0x7efea00c8180 2024-03-18T20:27:18.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.128+0000 7efea4f74700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea014e010 0x7efea014e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.128+0000 7efea4f74700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea014e010 0x7efea014e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50040/0 (socket says 172.21.15.1:50040) 2024-03-18T20:27:18.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.128+0000 7efea4f74700 1 -- 172.21.15.1:0/3565430377 learned_addr learned my addr 172.21.15.1:0/3565430377 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:18.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.129+0000 7efea4f74700 1 -- 172.21.15.1:0/3565430377 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efea00c8180 msgr2=0x7efea00c8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:18.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.129+0000 7efea5f76700 1 --2- 172.21.15.1:0/3565430377 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efea00c8180 0x7efea00c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.129+0000 7efea5775700 1 --2- 172.21.15.1:0/3565430377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe8c003a00 0x7efea00c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.129+0000 7efea4f74700 1 --2- 172.21.15.1:0/3565430377 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efea00c8180 0x7efea00c8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.129+0000 7efea4f74700 1 -- 172.21.15.1:0/3565430377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe8c003a00 msgr2=0x7efea00c9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:18.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.129+0000 7efea4f74700 1 --2- 172.21.15.1:0/3565430377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe8c003a00 0x7efea00c9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.129+0000 7efea4f74700 1 -- 172.21.15.1:0/3565430377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea015dbe0 con 0x7efea014e010 2024-03-18T20:27:18.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.129+0000 7efea4f74700 1 --2- 172.21.15.1:0/3565430377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea014e010 0x7efea014e490 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 crypto rx=0x7efe9000e780 tx=0x7efe9000eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:18.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.129+0000 7efe95ffb700 1 -- 172.21.15.1:0/3565430377 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe90012590 con 0x7efea014e010 2024-03-18T20:27:18.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.129+0000 7efe95ffb700 1 -- 172.21.15.1:0/3565430377 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe900126f0 con 0x7efea014e010 2024-03-18T20:27:18.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.129+0000 7efea79d9700 1 -- 172.21.15.1:0/3565430377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea01608b0 con 0x7efea014e010 2024-03-18T20:27:18.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.129+0000 7efe95ffb700 1 -- 172.21.15.1:0/3565430377 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe90012a80 con 0x7efea014e010 2024-03-18T20:27:18.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.129+0000 7efea79d9700 1 -- 172.21.15.1:0/3565430377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea00ade00 con 0x7efea014e010 2024-03-18T20:27:18.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.130+0000 7efea79d9700 1 -- 172.21.15.1:0/3565430377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea00adfc0 con 0x7efea014e010 2024-03-18T20:27:18.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.131+0000 7efe95ffb700 1 -- 172.21.15.1:0/3565430377 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7efe90012be0 con 0x7efea014e010 2024-03-18T20:27:18.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.131+0000 7efe95ffb700 1 --2- 172.21.15.1:0/3565430377 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efe8406af00 0x7efe8406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.131+0000 7efe95ffb700 1 -- 172.21.15.1:0/3565430377 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 5277+0+0 (secure 0 0 0) 0x7efe90089230 con 0x7efea014e010 2024-03-18T20:27:18.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.131+0000 7efea5775700 1 --2- 172.21.15.1:0/3565430377 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efe8406af00 0x7efe8406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.131+0000 7efea5775700 1 --2- 172.21.15.1:0/3565430377 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efe8406af00 0x7efe8406d3c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efe9c015040 tx=0x7efe9c009030 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:18.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.136+0000 7efe95ffb700 1 -- 172.21.15.1:0/3565430377 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efe90054340 con 0x7efea014e010 2024-03-18T20:27:18.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.420+0000 7efea79d9700 1 -- 172.21.15.1:0/3565430377 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7efea00676a0 con 0x7efea014e010 2024-03-18T20:27:18.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.420+0000 7efe95ffb700 1 -- 172.21.15.1:0/3565430377 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v52) v1 ==== 80+0+5304 (secure 0 0 0) 0x7efe90057960 con 0x7efea014e010 2024-03-18T20:27:18.425 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:18.426 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:27:18.426 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-18T20:27:18.426 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:18.426 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-18T20:27:18.426 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlR93/CxAAbpe+5n3wHRur9xi8QzdCnQ==", 2024-03-18T20:27:18.426 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:27:18.426 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:27:18.426 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:27:18.426 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:27:18.426 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:27:18.426 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:18.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:18.427 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-18T20:27:18.427 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJ3OmGBAA37u227nxrMsh/JHt71HBxQ==", 2024-03-18T20:27:18.427 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:27:18.427 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:27:18.427 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:27:18.427 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:27:18.427 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:27:18.427 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.427 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:18.427 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:18.427 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-18T20:27:18.427 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJZFoExAAmI16oxwNKbf8dKetPfyk+g==", 2024-03-18T20:27:18.427 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:27:18.428 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:27:18.428 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:27:18.428 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:27:18.428 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:27:18.428 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:18.428 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:18.428 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-18T20:27:18.428 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlQrcvIBAA1LMv+EgC7S6wspX6QF49Pg==", 2024-03-18T20:27:18.428 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:27:18.428 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:27:18.428 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:27:18.428 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:27:18.428 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:27:18.428 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:18.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:18.429 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-18T20:27:18.429 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCUjPhlYOVHChAARFQrksZmMIdTlntd8TgvrQ==", 2024-03-18T20:27:18.429 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:27:18.429 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:27:18.429 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:27:18.429 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:27:18.429 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:18.429 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:18.429 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-18T20:27:18.429 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCXjPhl2zSDCBAA6Al1gkWVB79ZI9djGpVChA==", 2024-03-18T20:27:18.429 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:27:18.430 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:27:18.430 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:27:18.430 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:27:18.430 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:18.430 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:18.430 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-18T20:27:18.430 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCajPhlSoG/BhAAR93USDe6DCQW+hREOsnTlw==", 2024-03-18T20:27:18.430 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:27:18.430 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:27:18.430 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:27:18.430 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:27:18.430 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.430 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:18.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:18.431 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-18T20:27:18.431 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCdjPhlhJg5BRAAWuQhcnxyjR5UOiM13yKdAQ==", 2024-03-18T20:27:18.431 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:27:18.431 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:27:18.431 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:27:18.431 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:27:18.431 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:18.431 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:18.431 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-18T20:27:18.431 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCjjPhlCFnpCRAAKIyOQyVdjXZUTnAS6Nw/ng==", 2024-03-18T20:27:18.431 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:27:18.431 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:27:18.432 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:27:18.432 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:27:18.432 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:18.432 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:18.432 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-18T20:27:18.432 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCmjPhlAeFkCBAAJcEFpRk/03lQJxj+l2hEPA==", 2024-03-18T20:27:18.432 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:27:18.432 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:27:18.432 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:27:18.432 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:27:18.432 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:18.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:18.433 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-18T20:27:18.433 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCpjPhl/xuuBhAA0LgKfXEvN2O11ZlRQJVOkw==", 2024-03-18T20:27:18.433 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:27:18.433 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:27:18.433 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:27:18.433 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:27:18.433 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:18.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:18.433 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-18T20:27:18.433 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCsjPhldiR4BRAA571oh6U8PzOv/4LzjYLF/A==", 2024-03-18T20:27:18.433 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:27:18.433 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:27:18.434 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:27:18.434 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:27:18.434 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:18.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:18.434 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-18T20:27:18.434 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlu+2xJBAANnXGGjfeqP6S2FTecsfOsg==", 2024-03-18T20:27:18.434 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:27:18.434 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:27:18.434 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-18T20:27:18.434 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-18T20:27:18.434 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:18.434 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:18.435 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-18T20:27:18.435 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlhJ9+KxAAEnrQrCrgJR6xI+pucwzAXA==", 2024-03-18T20:27:18.435 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:27:18.435 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:27:18.435 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:27:18.435 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:27:18.435 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:27:18.435 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:18.435 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:18.435 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.2", 2024-03-18T20:27:18.435 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlxBUpKBAAIGTwmLWyj+P2Yvl55QVcWA==", 2024-03-18T20:27:18.435 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:27:18.435 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:27:18.436 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:27:18.436 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:27:18.436 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:27:18.436 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:18.436 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:18.436 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.3", 2024-03-18T20:27:18.436 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlmao3MRAAHeiQ6eyBh9QjjKPCtxkC8g==", 2024-03-18T20:27:18.436 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:27:18.436 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:27:18.436 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:27:18.436 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:27:18.436 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:27:18.436 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:18.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:18.437 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-18T20:27:18.437 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCRjPhlemOuKRAAVUggmtB7+EiVgyfJZ8TCCA==", 2024-03-18T20:27:18.437 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:27:18.437 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:27:18.437 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:27:18.437 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:27:18.437 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:27:18.437 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:18.437 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:18.437 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-18T20:27:18.437 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhleIuOCBAA9CH+AHewQbn+gpVr+6lJyw==", 2024-03-18T20:27:18.438 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:27:18.438 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:27:18.438 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T20:27:18.438 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:27:18.438 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:18.438 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:18.438 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-18T20:27:18.438 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlL0pTBBAAdFiPcdwwz514cuLAvioxYA==", 2024-03-18T20:27:18.438 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:27:18.438 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:27:18.438 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T20:27:18.438 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:27:18.439 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.439 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:18.439 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-18T20:27:18.439 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:27:18.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.422+0000 7efea79d9700 1 -- 172.21.15.1:0/3565430377 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efe8406af00 msgr2=0x7efe8406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:18.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.422+0000 7efea79d9700 1 --2- 172.21.15.1:0/3565430377 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efe8406af00 0x7efe8406d3c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efe9c015040 tx=0x7efe9c009030 comp rx=0 tx=0).stop 2024-03-18T20:27:18.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.422+0000 7efea79d9700 1 -- 172.21.15.1:0/3565430377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea014e010 msgr2=0x7efea014e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:18.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.422+0000 7efea79d9700 1 --2- 172.21.15.1:0/3565430377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea014e010 0x7efea014e490 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 crypto rx=0x7efe9000e780 tx=0x7efe9000eb40 comp rx=0 tx=0).stop 2024-03-18T20:27:18.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.422+0000 7efea79d9700 1 -- 172.21.15.1:0/3565430377 shutdown_connections 2024-03-18T20:27:18.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.422+0000 7efea79d9700 1 --2- 172.21.15.1:0/3565430377 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efe8406af00 0x7efe8406d3c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.422+0000 7efea79d9700 1 --2- 172.21.15.1:0/3565430377 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efea00c8180 0x7efea00c8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.422+0000 7efea79d9700 1 --2- 172.21.15.1:0/3565430377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe8c003a00 0x7efea00c9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.422+0000 7efea79d9700 1 --2- 172.21.15.1:0/3565430377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea014e010 0x7efea014e490 unknown :-1 s=CLOSED pgs=3040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.422+0000 7efea79d9700 1 -- 172.21.15.1:0/3565430377 >> 172.21.15.1:0/3565430377 conn(0x7efea00b2ba0 msgr2=0x7efea006fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:18.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.422+0000 7efea79d9700 1 -- 172.21.15.1:0/3565430377 shutdown_connections 2024-03-18T20:27:18.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.422+0000 7efea79d9700 1 -- 172.21.15.1:0/3565430377 wait complete. 2024-03-18T20:27:18.441 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:27:18.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.561+0000 7f0f6b7f7700 1 Processor -- start 2024-03-18T20:27:18.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.562+0000 7f0f6b7f7700 1 -- start start 2024-03-18T20:27:18.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.562+0000 7f0f6b7f7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f6415cfc0 0x7f0f6406dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.562+0000 7f0f6b7f7700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0f640cc370 0x7f0f64071440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.562+0000 7f0f6b7f7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6406e9c0 con 0x7f0f640cbef0 2024-03-18T20:27:18.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.562+0000 7f0f6b7f7700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f6406e1f0 con 0x7f0f640cc370 2024-03-18T20:27:18.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.562+0000 7f0f6b7f7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f640cc800 con 0x7f0f6415cfc0 2024-03-18T20:27:18.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.562+0000 7f0f68d92700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0f640cbef0 0x7f0f64070d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33746/0 (socket says 172.21.15.1:33746) 2024-03-18T20:27:18.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f68d92700 1 -- 172.21.15.1:0/1066454100 learned_addr learned my addr 172.21.15.1:0/1066454100 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:18.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f69593700 1 --2- 172.21.15.1:0/1066454100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f6415cfc0 0x7f0f6406dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f5bfff700 1 -- 172.21.15.1:0/1066454100 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813893463 0 0) 0x7f0f6406e9c0 con 0x7f0f640cbef0 2024-03-18T20:27:18.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f5bfff700 1 -- 172.21.15.1:0/1066454100 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f500035b0 con 0x7f0f640cbef0 2024-03-18T20:27:18.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f69d94700 1 --2- 172.21.15.1:0/1066454100 >> v2:172.21.15.165:3300/0 conn(0x7f0f640cc370 0x7f0f64071440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f5bfff700 1 -- 172.21.15.1:0/1066454100 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2391977901 0 0) 0x7f0f500035b0 con 0x7f0f640cbef0 2024-03-18T20:27:18.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f5bfff700 1 -- 172.21.15.1:0/1066454100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f6415cfc0 msgr2=0x7f0f6406dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:18.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f5bfff700 1 --2- 172.21.15.1:0/1066454100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f6415cfc0 0x7f0f6406dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f5bfff700 1 -- 172.21.15.1:0/1066454100 >> v2:172.21.15.165:3300/0 conn(0x7f0f640cc370 msgr2=0x7f0f64071440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:18.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f5bfff700 1 --2- 172.21.15.1:0/1066454100 >> v2:172.21.15.165:3300/0 conn(0x7f0f640cc370 0x7f0f64071440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f5bfff700 1 -- 172.21.15.1:0/1066454100 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f640cd640 con 0x7f0f640cbef0 2024-03-18T20:27:18.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f5bfff700 1 -- 172.21.15.1:0/1066454100 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f0f60001670 con 0x7f0f640cbef0 2024-03-18T20:27:18.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f69d94700 1 --2- 172.21.15.1:0/1066454100 >> v2:172.21.15.165:3300/0 conn(0x7f0f640cc370 0x7f0f64071440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:18.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f5bfff700 1 -- 172.21.15.1:0/1066454100 >> v1:172.21.15.1:6789/0 conn(0x7f0f640cbef0 legacy=0x7f0f64070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:18.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f5bfff700 1 --2- 172.21.15.1:0/1066454100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f50003a00 0x7f0f50003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f5bfff700 1 --2- 172.21.15.1:0/1066454100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f50004850 0x7f0f50004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f5bfff700 1 --2- 172.21.15.1:0/1066454100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f6415cfc0 0x7f0f50005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f5bfff700 1 -- 172.21.15.1:0/1066454100 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f640cc800 con 0x7f0f50004850 2024-03-18T20:27:18.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f5bfff700 1 -- 172.21.15.1:0/1066454100 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5000dcd0 con 0x7f0f50003a00 2024-03-18T20:27:18.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f5bfff700 1 -- 172.21.15.1:0/1066454100 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5000de40 con 0x7f0f6415cfc0 2024-03-18T20:27:18.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f68d92700 1 --2- 172.21.15.1:0/1066454100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f50004850 0x7f0f50004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f69593700 1 --2- 172.21.15.1:0/1066454100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f50003a00 0x7f0f50003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.563+0000 7f0f69d94700 1 --2- 172.21.15.1:0/1066454100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f6415cfc0 0x7f0f50005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.564+0000 7f0f68d92700 1 -- 172.21.15.1:0/1066454100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f6415cfc0 msgr2=0x7f0f50005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:18.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.564+0000 7f0f68d92700 1 --2- 172.21.15.1:0/1066454100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f6415cfc0 0x7f0f50005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.564+0000 7f0f68d92700 1 -- 172.21.15.1:0/1066454100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f50003a00 msgr2=0x7f0f50003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:18.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.564+0000 7f0f68d92700 1 --2- 172.21.15.1:0/1066454100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f50003a00 0x7f0f50003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.564+0000 7f0f68d92700 1 -- 172.21.15.1:0/1066454100 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f500035b0 con 0x7f0f50004850 2024-03-18T20:27:18.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.564+0000 7f0f69d94700 1 --2- 172.21.15.1:0/1066454100 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f6415cfc0 0x7f0f50005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:18.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.564+0000 7f0f68d92700 1 --2- 172.21.15.1:0/1066454100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f50004850 0x7f0f50004cd0 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 crypto rx=0x7f0f60001050 tx=0x7f0f50005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b68acb1dca35bff server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:18.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.564+0000 7f0f5bfff700 1 -- 172.21.15.1:0/1066454100 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60007c60 con 0x7f0f50004850 2024-03-18T20:27:18.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.564+0000 7f0f5bfff700 1 -- 172.21.15.1:0/1066454100 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6000ce80 con 0x7f0f50004850 2024-03-18T20:27:18.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.564+0000 7f0f5bfff700 1 -- 172.21.15.1:0/1066454100 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60005ba0 con 0x7f0f50004850 2024-03-18T20:27:18.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.564+0000 7f0f6b7f7700 1 -- 172.21.15.1:0/1066454100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f50004850 msgr2=0x7f0f50004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.564+0000 7f0f6b7f7700 1 --2- 172.21.15.1:0/1066454100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f50004850 0x7f0f50004cd0 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 crypto rx=0x7f0f60001050 tx=0x7f0f50005880 comp rx=0 tx=0).stop 2024-03-18T20:27:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.564+0000 7f0f69d94700 1 -- 172.21.15.1:0/1066454100 reap_dead start 2024-03-18T20:27:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.564+0000 7f0f6b7f7700 1 -- 172.21.15.1:0/1066454100 shutdown_connections 2024-03-18T20:27:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.564+0000 7f0f6b7f7700 1 -- 172.21.15.1:0/1066454100 >> 172.21.15.1:0/1066454100 conn(0x7f0f640b1270 msgr2=0x7f0f640b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.565+0000 7f0f6b7f7700 1 -- 172.21.15.1:0/1066454100 shutdown_connections 2024-03-18T20:27:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.565+0000 7f0f6b7f7700 1 -- 172.21.15.1:0/1066454100 wait complete. 2024-03-18T20:27:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.566+0000 7f0f6b7f7700 1 Processor -- start 2024-03-18T20:27:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.566+0000 7f0f6b7f7700 1 -- start start 2024-03-18T20:27:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f6b7f7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f50004850 0x7f0f640b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f6b7f7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f64157fe0 0x7f0f640b4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f6b7f7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f640c8e20 0x7f0f640b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f6b7f7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f640c5d40 con 0x7f0f64157fe0 2024-03-18T20:27:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f6b7f7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f640b3810 con 0x7f0f50004850 2024-03-18T20:27:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f6b7f7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f640b3af0 con 0x7f0f640c8e20 2024-03-18T20:27:18.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f68d92700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f64157fe0 0x7f0f640b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f68d92700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f64157fe0 0x7f0f640b4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50052/0 (socket says 172.21.15.1:50052) 2024-03-18T20:27:18.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f68d92700 1 -- 172.21.15.1:0/29731908 learned_addr learned my addr 172.21.15.1:0/29731908 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:18.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f69593700 1 --2- 172.21.15.1:0/29731908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f50004850 0x7f0f640b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f69d94700 1 --2- 172.21.15.1:0/29731908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f640c8e20 0x7f0f640b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f68d92700 1 -- 172.21.15.1:0/29731908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f640c8e20 msgr2=0x7f0f640b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:18.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f68d92700 1 --2- 172.21.15.1:0/29731908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f640c8e20 0x7f0f640b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f68d92700 1 -- 172.21.15.1:0/29731908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f50004850 msgr2=0x7f0f640b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:18.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f68d92700 1 --2- 172.21.15.1:0/29731908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f50004850 0x7f0f640b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f68d92700 1 -- 172.21.15.1:0/29731908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f640b3dd0 con 0x7f0f64157fe0 2024-03-18T20:27:18.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f68d92700 1 --2- 172.21.15.1:0/29731908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f64157fe0 0x7f0f640b4810 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 crypto rx=0x7f0f6000c930 tx=0x7f0f60000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:18.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f59ffb700 1 -- 172.21.15.1:0/29731908 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f600072d0 con 0x7f0f64157fe0 2024-03-18T20:27:18.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f59ffb700 1 -- 172.21.15.1:0/29731908 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f60007430 con 0x7f0f64157fe0 2024-03-18T20:27:18.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f59ffb700 1 -- 172.21.15.1:0/29731908 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f600077e0 con 0x7f0f64157fe0 2024-03-18T20:27:18.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.567+0000 7f0f6b7f7700 1 -- 172.21.15.1:0/29731908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f640b40b0 con 0x7f0f64157fe0 2024-03-18T20:27:18.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.568+0000 7f0f6b7f7700 1 -- 172.21.15.1:0/29731908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f640b34e0 con 0x7f0f64157fe0 2024-03-18T20:27:18.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.568+0000 7f0f6b7f7700 1 -- 172.21.15.1:0/29731908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f640c68a0 con 0x7f0f64157fe0 2024-03-18T20:27:18.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.569+0000 7f0f59ffb700 1 -- 172.21.15.1:0/29731908 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f0f6000cd40 con 0x7f0f64157fe0 2024-03-18T20:27:18.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.569+0000 7f0f59ffb700 1 --2- 172.21.15.1:0/29731908 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0f4806aed0 0x7f0f4806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.569+0000 7f0f59ffb700 1 -- 172.21.15.1:0/29731908 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f0f6008a9e0 con 0x7f0f64157fe0 2024-03-18T20:27:18.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.569+0000 7f0f69593700 1 --2- 172.21.15.1:0/29731908 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0f4806aed0 0x7f0f4806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.570+0000 7f0f69593700 1 --2- 172.21.15.1:0/29731908 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0f4806aed0 0x7f0f4806d390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0f640c9770 tx=0x7f0f54008040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:18.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.574+0000 7f0f59ffb700 1 -- 172.21.15.1:0/29731908 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0f60055a00 con 0x7f0f64157fe0 2024-03-18T20:27:18.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.778+0000 7f0f6b7f7700 1 -- 172.21.15.1:0/29731908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f640c5d40 con 0x7f0f64157fe0 2024-03-18T20:27:18.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.778+0000 7f0f59ffb700 1 -- 172.21.15.1:0/29731908 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v270) v1 ==== 74+0+12717 (secure 0 0 0) 0x7f0f60007dc0 con 0x7f0f64157fe0 2024-03-18T20:27:18.783 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:18.784 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":270,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:27:17.971207+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":255,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:27:18.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.781+0000 7f0f6b7f7700 1 -- 172.21.15.1:0/29731908 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0f4806aed0 msgr2=0x7f0f4806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:18.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.781+0000 7f0f6b7f7700 1 --2- 172.21.15.1:0/29731908 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0f4806aed0 0x7f0f4806d390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0f640c9770 tx=0x7f0f54008040 comp rx=0 tx=0).stop 2024-03-18T20:27:18.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.781+0000 7f0f6b7f7700 1 -- 172.21.15.1:0/29731908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f64157fe0 msgr2=0x7f0f640b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:18.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.781+0000 7f0f6b7f7700 1 --2- 172.21.15.1:0/29731908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f64157fe0 0x7f0f640b4810 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 crypto rx=0x7f0f6000c930 tx=0x7f0f60000fe0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.781+0000 7f0f6b7f7700 1 -- 172.21.15.1:0/29731908 shutdown_connections 2024-03-18T20:27:18.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.781+0000 7f0f6b7f7700 1 --2- 172.21.15.1:0/29731908 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0f4806aed0 0x7f0f4806d390 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.781+0000 7f0f6b7f7700 1 --2- 172.21.15.1:0/29731908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f640c8e20 0x7f0f640b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.781+0000 7f0f6b7f7700 1 --2- 172.21.15.1:0/29731908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f50004850 0x7f0f640b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.781+0000 7f0f6b7f7700 1 --2- 172.21.15.1:0/29731908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f64157fe0 0x7f0f640b4810 unknown :-1 s=CLOSED pgs=3043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.781+0000 7f0f6b7f7700 1 -- 172.21.15.1:0/29731908 >> 172.21.15.1:0/29731908 conn(0x7f0f640b1270 msgr2=0x7f0f64159b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:18.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.781+0000 7f0f6b7f7700 1 -- 172.21.15.1:0/29731908 shutdown_connections 2024-03-18T20:27:18.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.781+0000 7f0f6b7f7700 1 -- 172.21.15.1:0/29731908 wait complete. 2024-03-18T20:27:18.798 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:27:18.798 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:27:18.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.921+0000 7f0bc4b2e700 1 Processor -- start 2024-03-18T20:27:18.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.922+0000 7f0bc4b2e700 1 -- start start 2024-03-18T20:27:18.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.922+0000 7f0bc4b2e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0bc015d9e0 0x7f0bc009a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.922+0000 7f0bc4b2e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0bc00d0240 0x7f0bc00cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.922+0000 7f0bc4b2e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc00d06d0 con 0x7f0bc015d600 2024-03-18T20:27:18.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.922+0000 7f0bc4b2e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc00cfdb0 con 0x7f0bc015d9e0 2024-03-18T20:27:18.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.922+0000 7f0bc4b2e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc00b38d0 con 0x7f0bc00d0240 2024-03-18T20:27:18.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.923+0000 7f0bbdd9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0bc015d9e0 0x7f0bc009a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.923+0000 7f0bbe59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0bc015d600 0x7f0bc00cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33754/0 (socket says 172.21.15.1:33754) 2024-03-18T20:27:18.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.923+0000 7f0bbe59c700 1 -- 172.21.15.1:0/4113920964 learned_addr learned my addr 172.21.15.1:0/4113920964 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:18.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.923+0000 7f0bbed9d700 1 --2- 172.21.15.1:0/4113920964 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0bc00d0240 0x7f0bc00cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.923+0000 7f0bbdd9b700 1 -- 172.21.15.1:0/4113920964 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0bc00d0240 msgr2=0x7f0bc00cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:18.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.923+0000 7f0bbdd9b700 1 --2- 172.21.15.1:0/4113920964 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0bc00d0240 0x7f0bc00cbe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.923+0000 7f0bbdd9b700 1 -- 172.21.15.1:0/4113920964 >> v1:172.21.15.1:6789/0 conn(0x7f0bc015d600 legacy=0x7f0bc00cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:18.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.923+0000 7f0bbdd9b700 1 -- 172.21.15.1:0/4113920964 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc009aa30 con 0x7f0bc015d9e0 2024-03-18T20:27:18.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.923+0000 7f0bbd59a700 1 -- 172.21.15.1:0/4113920964 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138461697 0 0) 0x7f0bc00d06d0 con 0x7f0bc015d600 2024-03-18T20:27:18.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.923+0000 7f0bbed9d700 1 --2- 172.21.15.1:0/4113920964 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0bc00d0240 0x7f0bc00cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:18.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.923+0000 7f0bbdd9b700 1 --2- 172.21.15.1:0/4113920964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bc015d9e0 0x7f0bc009a420 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0009840 tx=0x7f0bb0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72b6a0001cabfde9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:18.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.923+0000 7f0bbd59a700 1 -- 172.21.15.1:0/4113920964 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb0011830 con 0x7f0bc015d9e0 2024-03-18T20:27:18.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.923+0000 7f0bbd59a700 1 -- 172.21.15.1:0/4113920964 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb0011990 con 0x7f0bc015d9e0 2024-03-18T20:27:18.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.923+0000 7f0bbd59a700 1 -- 172.21.15.1:0/4113920964 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb0011d20 con 0x7f0bc015d9e0 2024-03-18T20:27:18.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.924+0000 7f0bc4b2e700 1 -- 172.21.15.1:0/4113920964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bc015d9e0 msgr2=0x7f0bc009a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:18.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.924+0000 7f0bc4b2e700 1 --2- 172.21.15.1:0/4113920964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bc015d9e0 0x7f0bc009a420 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0009840 tx=0x7f0bb0009b50 comp rx=0 tx=0).stop 2024-03-18T20:27:18.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.924+0000 7f0bc4b2e700 1 -- 172.21.15.1:0/4113920964 shutdown_connections 2024-03-18T20:27:18.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.924+0000 7f0bc4b2e700 1 --2- 172.21.15.1:0/4113920964 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0bc00d0240 0x7f0bc00cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.924+0000 7f0bc4b2e700 1 --2- 172.21.15.1:0/4113920964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bc015d9e0 0x7f0bc009a420 unknown :-1 s=CLOSED pgs=3045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.924+0000 7f0bc4b2e700 1 -- 172.21.15.1:0/4113920964 >> 172.21.15.1:0/4113920964 conn(0x7f0bc00b2b90 msgr2=0x7f0bc00b2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:18.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.924+0000 7f0bc4b2e700 1 -- 172.21.15.1:0/4113920964 shutdown_connections 2024-03-18T20:27:18.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.924+0000 7f0bc4b2e700 1 -- 172.21.15.1:0/4113920964 wait complete. 2024-03-18T20:27:18.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.925+0000 7f0bc4b2e700 1 Processor -- start 2024-03-18T20:27:18.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.925+0000 7f0bc4b2e700 1 -- start start 2024-03-18T20:27:18.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.926+0000 7f0bc4b2e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0bc00d0240 0x7f0bc00c7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.926+0000 7f0bc4b2e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bc015d600 0x7f0bc014c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.926+0000 7f0bc4b2e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bc015d9e0 0x7f0bc014c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.926+0000 7f0bc4b2e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc014c5e0 con 0x7f0bc015d600 2024-03-18T20:27:18.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.926+0000 7f0bc4b2e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc00c60c0 con 0x7f0bc015d9e0 2024-03-18T20:27:18.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.926+0000 7f0bc4b2e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc00c63a0 con 0x7f0bc00d0240 2024-03-18T20:27:18.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.926+0000 7f0bbdd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bc015d600 0x7f0bc014c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.926+0000 7f0bbdd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bc015d600 0x7f0bc014c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50066/0 (socket says 172.21.15.1:50066) 2024-03-18T20:27:18.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.926+0000 7f0bbdd9b700 1 -- 172.21.15.1:0/475250423 learned_addr learned my addr 172.21.15.1:0/475250423 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:18.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.926+0000 7f0bbe59c700 1 --2- 172.21.15.1:0/475250423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0bc00d0240 0x7f0bc00c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.926+0000 7f0bbed9d700 1 --2- 172.21.15.1:0/475250423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bc015d9e0 0x7f0bc014c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.926+0000 7f0bbdd9b700 1 -- 172.21.15.1:0/475250423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0bc00d0240 msgr2=0x7f0bc00c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:18.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.926+0000 7f0bbdd9b700 1 --2- 172.21.15.1:0/475250423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0bc00d0240 0x7f0bc00c7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.926+0000 7f0bbdd9b700 1 -- 172.21.15.1:0/475250423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bc015d9e0 msgr2=0x7f0bc014c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:18.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.926+0000 7f0bbdd9b700 1 --2- 172.21.15.1:0/475250423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bc015d9e0 0x7f0bc014c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:18.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.926+0000 7f0bbdd9b700 1 -- 172.21.15.1:0/475250423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb00094a0 con 0x7f0bc015d600 2024-03-18T20:27:18.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.927+0000 7f0bbdd9b700 1 --2- 172.21.15.1:0/475250423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bc015d600 0x7f0bc014c0a0 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0014040 tx=0x7f0bb0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:18.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.927+0000 7f0bb6ffd700 1 -- 172.21.15.1:0/475250423 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb00038a0 con 0x7f0bc015d600 2024-03-18T20:27:18.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.927+0000 7f0bb6ffd700 1 -- 172.21.15.1:0/475250423 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb001f070 con 0x7f0bc015d600 2024-03-18T20:27:18.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.927+0000 7f0bc4b2e700 1 -- 172.21.15.1:0/475250423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc00c6680 con 0x7f0bc015d600 2024-03-18T20:27:18.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.927+0000 7f0bb6ffd700 1 -- 172.21.15.1:0/475250423 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb0003c60 con 0x7f0bc015d600 2024-03-18T20:27:18.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.927+0000 7f0bc4b2e700 1 -- 172.21.15.1:0/475250423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc00c6c40 con 0x7f0bc015d600 2024-03-18T20:27:18.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.928+0000 7f0bc4b2e700 1 -- 172.21.15.1:0/475250423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc00676a0 con 0x7f0bc015d600 2024-03-18T20:27:18.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.928+0000 7f0bb6ffd700 1 -- 172.21.15.1:0/475250423 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f0bb001a430 con 0x7f0bc015d600 2024-03-18T20:27:18.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.929+0000 7f0bb6ffd700 1 --2- 172.21.15.1:0/475250423 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0ba006aed0 0x7f0ba006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:18.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.929+0000 7f0bb6ffd700 1 -- 172.21.15.1:0/475250423 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f0bb0021070 con 0x7f0bc015d600 2024-03-18T20:27:18.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.929+0000 7f0bbe59c700 1 --2- 172.21.15.1:0/475250423 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0ba006aed0 0x7f0ba006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:18.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.929+0000 7f0bbe59c700 1 --2- 172.21.15.1:0/475250423 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0ba006aed0 0x7f0ba006d390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8007fd0 tx=0x7f0ba8007de0 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:18.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:18.933+0000 7f0bb6ffd700 1 -- 172.21.15.1:0/475250423 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0bb00560e0 con 0x7f0bc015d600 2024-03-18T20:27:19.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.137+0000 7f0bc4b2e700 1 -- 172.21.15.1:0/475250423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0bc006e500 con 0x7f0bc015d600 2024-03-18T20:27:19.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.138+0000 7f0bb6ffd700 1 -- 172.21.15.1:0/475250423 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v270) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0bb0015020 con 0x7f0bc015d600 2024-03-18T20:27:19.143 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:27:19.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.140+0000 7f0bc4b2e700 1 -- 172.21.15.1:0/475250423 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0ba006aed0 msgr2=0x7f0ba006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:19.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.140+0000 7f0bc4b2e700 1 --2- 172.21.15.1:0/475250423 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0ba006aed0 0x7f0ba006d390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8007fd0 tx=0x7f0ba8007de0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.140+0000 7f0bc4b2e700 1 -- 172.21.15.1:0/475250423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bc015d600 msgr2=0x7f0bc014c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:19.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.140+0000 7f0bc4b2e700 1 --2- 172.21.15.1:0/475250423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bc015d600 0x7f0bc014c0a0 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0014040 tx=0x7f0bb0004750 comp rx=0 tx=0).stop 2024-03-18T20:27:19.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.140+0000 7f0bc4b2e700 1 -- 172.21.15.1:0/475250423 shutdown_connections 2024-03-18T20:27:19.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.140+0000 7f0bc4b2e700 1 --2- 172.21.15.1:0/475250423 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0ba006aed0 0x7f0ba006d390 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.140+0000 7f0bc4b2e700 1 --2- 172.21.15.1:0/475250423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0bc00d0240 0x7f0bc00c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.140+0000 7f0bc4b2e700 1 --2- 172.21.15.1:0/475250423 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bc015d9e0 0x7f0bc014c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.140+0000 7f0bc4b2e700 1 --2- 172.21.15.1:0/475250423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0bc015d600 0x7f0bc014c0a0 unknown :-1 s=CLOSED pgs=3046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.140+0000 7f0bc4b2e700 1 -- 172.21.15.1:0/475250423 >> 172.21.15.1:0/475250423 conn(0x7f0bc00b2b90 msgr2=0x7f0bc006fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:19.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.140+0000 7f0bc4b2e700 1 -- 172.21.15.1:0/475250423 shutdown_connections 2024-03-18T20:27:19.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.140+0000 7f0bc4b2e700 1 -- 172.21.15.1:0/475250423 wait complete. 2024-03-18T20:27:19.157 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T20:27:19.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.277+0000 7f1e36d0e700 1 Processor -- start 2024-03-18T20:27:19.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.277+0000 7f1e36d0e700 1 -- start start 2024-03-18T20:27:19.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.278+0000 7f1e36d0e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1e300d1c60 0x7f1e300b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:19.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.278+0000 7f1e36d0e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e300cddc0 con 0x7f1e3015d500 2024-03-18T20:27:19.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.278+0000 7f1e36d0e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e300d3dc0 con 0x7f1e300d17b0 2024-03-18T20:27:19.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.278+0000 7f1e36d0e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e300ce060 con 0x7f1e300d1c60 2024-03-18T20:27:19.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.278+0000 7f1e34aaa700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1e3015d500 0x7f1e300d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33764/0 (socket says 172.21.15.1:33764) 2024-03-18T20:27:19.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.278+0000 7f1e34aaa700 1 -- 172.21.15.1:0/3015849486 learned_addr learned my addr 172.21.15.1:0/3015849486 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:19.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e352ab700 1 --2- 172.21.15.1:0/3015849486 >> v2:172.21.15.165:3300/0 conn(0x7f1e300d1c60 0x7f1e300b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:19.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e2f7fe700 1 -- 172.21.15.1:0/3015849486 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447683105 0 0) 0x7f1e300cddc0 con 0x7f1e3015d500 2024-03-18T20:27:19.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e2f7fe700 1 -- 172.21.15.1:0/3015849486 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e200035b0 con 0x7f1e3015d500 2024-03-18T20:27:19.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e2f7fe700 1 -- 172.21.15.1:0/3015849486 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3061894954 0 0) 0x7f1e200035b0 con 0x7f1e3015d500 2024-03-18T20:27:19.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e2f7fe700 1 -- 172.21.15.1:0/3015849486 >> v2:172.21.15.165:3300/0 conn(0x7f1e300d1c60 msgr2=0x7f1e300b8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:19.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e2f7fe700 1 --2- 172.21.15.1:0/3015849486 >> v2:172.21.15.165:3300/0 conn(0x7f1e300d1c60 0x7f1e300b8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e2f7fe700 1 -- 172.21.15.1:0/3015849486 >> v1:172.21.15.165:6789/0 conn(0x7f1e300d17b0 legacy=0x7f1e300b83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:19.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e2f7fe700 1 -- 172.21.15.1:0/3015849486 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e300ce340 con 0x7f1e3015d500 2024-03-18T20:27:19.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e2f7fe700 1 -- 172.21.15.1:0/3015849486 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f1e24001670 con 0x7f1e3015d500 2024-03-18T20:27:19.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e2f7fe700 1 -- 172.21.15.1:0/3015849486 >> v1:172.21.15.1:6789/0 conn(0x7f1e3015d500 legacy=0x7f1e300d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:19.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e2f7fe700 1 --2- 172.21.15.1:0/3015849486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e20003a00 0x7f1e20003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:19.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e2f7fe700 1 --2- 172.21.15.1:0/3015849486 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e20004850 0x7f1e20004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e2f7fe700 1 --2- 172.21.15.1:0/3015849486 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e20005210 0x7f1e2000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e2f7fe700 1 -- 172.21.15.1:0/3015849486 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e300ce060 con 0x7f1e20003a00 2024-03-18T20:27:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e2f7fe700 1 -- 172.21.15.1:0/3015849486 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2000c1b0 con 0x7f1e20005210 2024-03-18T20:27:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e2f7fe700 1 -- 172.21.15.1:0/3015849486 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2000c370 con 0x7f1e20004850 2024-03-18T20:27:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e34aaa700 1 --2- 172.21.15.1:0/3015849486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e20003a00 0x7f1e20003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e352ab700 1 --2- 172.21.15.1:0/3015849486 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e20005210 0x7f1e2000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e2ffff700 1 --2- 172.21.15.1:0/3015849486 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e20004850 0x7f1e20004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e34aaa700 1 -- 172.21.15.1:0/3015849486 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e20004850 msgr2=0x7f1e20004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.279+0000 7f1e34aaa700 1 --2- 172.21.15.1:0/3015849486 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e20004850 0x7f1e20004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.280+0000 7f1e34aaa700 1 -- 172.21.15.1:0/3015849486 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e20005210 msgr2=0x7f1e2000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.280+0000 7f1e34aaa700 1 --2- 172.21.15.1:0/3015849486 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e20005210 0x7f1e2000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.280+0000 7f1e34aaa700 1 -- 172.21.15.1:0/3015849486 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e200035b0 con 0x7f1e20003a00 2024-03-18T20:27:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.280+0000 7f1e352ab700 1 -- 172.21.15.1:0/3015849486 reap_dead start 2024-03-18T20:27:19.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.280+0000 7f1e34aaa700 1 --2- 172.21.15.1:0/3015849486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e20003a00 0x7f1e20003e20 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7f1e2400c2a0 tx=0x7f1e20005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48ad887ccf027aa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.280+0000 7f1e2f7fe700 1 -- 172.21.15.1:0/3015849486 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e24007c10 con 0x7f1e20003a00 2024-03-18T20:27:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.280+0000 7f1e2f7fe700 1 -- 172.21.15.1:0/3015849486 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2400cd80 con 0x7f1e20003a00 2024-03-18T20:27:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.280+0000 7f1e2f7fe700 1 -- 172.21.15.1:0/3015849486 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e24005ba0 con 0x7f1e20003a00 2024-03-18T20:27:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.280+0000 7f1e36d0e700 1 -- 172.21.15.1:0/3015849486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e20003a00 msgr2=0x7f1e20003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.280+0000 7f1e36d0e700 1 --2- 172.21.15.1:0/3015849486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e20003a00 0x7f1e20003e20 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7f1e2400c2a0 tx=0x7f1e20005880 comp rx=0 tx=0).stop 2024-03-18T20:27:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.280+0000 7f1e36d0e700 1 -- 172.21.15.1:0/3015849486 shutdown_connections 2024-03-18T20:27:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.280+0000 7f1e36d0e700 1 --2- 172.21.15.1:0/3015849486 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e20003a00 0x7f1e20003e20 unknown :-1 s=CLOSED pgs=3048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.280+0000 7f1e36d0e700 1 -- 172.21.15.1:0/3015849486 >> 172.21.15.1:0/3015849486 conn(0x7f1e3009c530 msgr2=0x7f1e300d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.280+0000 7f1e36d0e700 1 -- 172.21.15.1:0/3015849486 shutdown_connections 2024-03-18T20:27:19.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.280+0000 7f1e36d0e700 1 -- 172.21.15.1:0/3015849486 wait complete. 2024-03-18T20:27:19.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.282+0000 7f1e36d0e700 1 Processor -- start 2024-03-18T20:27:19.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.282+0000 7f1e36d0e700 1 -- start start 2024-03-18T20:27:19.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.282+0000 7f1e36d0e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e20003a00 0x7f1e30150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:19.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.282+0000 7f1e36d0e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e300b5d70 0x7f1e300b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:19.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e36d0e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e301e6e40 0x7f1e301e72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:19.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e36d0e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e30151b40 con 0x7f1e300b5d70 2024-03-18T20:27:19.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e36d0e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e300b6730 con 0x7f1e301e6e40 2024-03-18T20:27:19.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e36d0e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e301e7830 con 0x7f1e20003a00 2024-03-18T20:27:19.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e2ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e300b5d70 0x7f1e300b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:19.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e2ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e300b5d70 0x7f1e300b61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50072/0 (socket says 172.21.15.1:50072) 2024-03-18T20:27:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e2ffff700 1 -- 172.21.15.1:0/583080669 learned_addr learned my addr 172.21.15.1:0/583080669 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e352ab700 1 --2- 172.21.15.1:0/583080669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e301e6e40 0x7f1e301e72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e34aaa700 1 --2- 172.21.15.1:0/583080669 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e20003a00 0x7f1e30150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e2ffff700 1 -- 172.21.15.1:0/583080669 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e20003a00 msgr2=0x7f1e30150ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e2ffff700 1 --2- 172.21.15.1:0/583080669 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e20003a00 0x7f1e30150ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e2ffff700 1 -- 172.21.15.1:0/583080669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e301e6e40 msgr2=0x7f1e301e72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e2ffff700 1 --2- 172.21.15.1:0/583080669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e301e6e40 0x7f1e301e72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e2ffff700 1 -- 172.21.15.1:0/583080669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e301e8660 con 0x7f1e300b5d70 2024-03-18T20:27:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e352ab700 1 --2- 172.21.15.1:0/583080669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e301e6e40 0x7f1e301e72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e2ffff700 1 --2- 172.21.15.1:0/583080669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e300b5d70 0x7f1e300b61f0 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c00e490 tx=0x7f1e1c00e850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e2d7fa700 1 -- 172.21.15.1:0/583080669 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c0052e0 con 0x7f1e300b5d70 2024-03-18T20:27:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e2d7fa700 1 -- 172.21.15.1:0/583080669 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1c005440 con 0x7f1e300b5d70 2024-03-18T20:27:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e36d0e700 1 -- 172.21.15.1:0/583080669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e301e7a50 con 0x7f1e300b5d70 2024-03-18T20:27:19.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e2d7fa700 1 -- 172.21.15.1:0/583080669 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c0038a0 con 0x7f1e300b5d70 2024-03-18T20:27:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.283+0000 7f1e36d0e700 1 -- 172.21.15.1:0/583080669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e301e8010 con 0x7f1e300b5d70 2024-03-18T20:27:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.284+0000 7f1e36d0e700 1 -- 172.21.15.1:0/583080669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e300676a0 con 0x7f1e300b5d70 2024-03-18T20:27:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.285+0000 7f1e2d7fa700 1 -- 172.21.15.1:0/583080669 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f1e1c003a00 con 0x7f1e300b5d70 2024-03-18T20:27:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.285+0000 7f1e2d7fa700 1 --2- 172.21.15.1:0/583080669 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e1406aed0 0x7f1e1406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.285+0000 7f1e2d7fa700 1 -- 172.21.15.1:0/583080669 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f1e1c088ba0 con 0x7f1e300b5d70 2024-03-18T20:27:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.285+0000 7f1e34aaa700 1 --2- 172.21.15.1:0/583080669 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e1406aed0 0x7f1e1406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.286+0000 7f1e34aaa700 1 --2- 172.21.15.1:0/583080669 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e1406aed0 0x7f1e1406d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1e24000c00 tx=0x7f1e24007000 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:19.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.290+0000 7f1e2d7fa700 1 -- 172.21.15.1:0/583080669 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1e1c053dd0 con 0x7f1e300b5d70 2024-03-18T20:27:19.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.495+0000 7f1e36d0e700 1 -- 172.21.15.1:0/583080669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1e300d0950 con 0x7f1e300b5d70 2024-03-18T20:27:19.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.495+0000 7f1e2d7fa700 1 -- 172.21.15.1:0/583080669 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v270) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1e1c019020 con 0x7f1e300b5d70 2024-03-18T20:27:19.500 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:19.500 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T20:27:19.501 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:19.501 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-18T20:27:19.501 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-18T20:27:19.501 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:19.501 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T20:27:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.497+0000 7f1e36d0e700 1 -- 172.21.15.1:0/583080669 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e1406aed0 msgr2=0x7f1e1406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.497+0000 7f1e36d0e700 1 --2- 172.21.15.1:0/583080669 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e1406aed0 0x7f1e1406d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1e24000c00 tx=0x7f1e24007000 comp rx=0 tx=0).stop 2024-03-18T20:27:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.497+0000 7f1e36d0e700 1 -- 172.21.15.1:0/583080669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e300b5d70 msgr2=0x7f1e300b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.497+0000 7f1e36d0e700 1 --2- 172.21.15.1:0/583080669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e300b5d70 0x7f1e300b61f0 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c00e490 tx=0x7f1e1c00e850 comp rx=0 tx=0).stop 2024-03-18T20:27:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.497+0000 7f1e36d0e700 1 -- 172.21.15.1:0/583080669 shutdown_connections 2024-03-18T20:27:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.497+0000 7f1e36d0e700 1 --2- 172.21.15.1:0/583080669 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e1406aed0 0x7f1e1406d390 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.497+0000 7f1e36d0e700 1 --2- 172.21.15.1:0/583080669 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e20003a00 0x7f1e30150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.497+0000 7f1e36d0e700 1 --2- 172.21.15.1:0/583080669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e301e6e40 0x7f1e301e72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.497+0000 7f1e36d0e700 1 --2- 172.21.15.1:0/583080669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e300b5d70 0x7f1e300b61f0 unknown :-1 s=CLOSED pgs=3049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.497+0000 7f1e36d0e700 1 -- 172.21.15.1:0/583080669 >> 172.21.15.1:0/583080669 conn(0x7f1e3009c530 msgr2=0x7f1e300b3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:19.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.498+0000 7f1e36d0e700 1 -- 172.21.15.1:0/583080669 shutdown_connections 2024-03-18T20:27:19.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.498+0000 7f1e36d0e700 1 -- 172.21.15.1:0/583080669 wait complete. 2024-03-18T20:27:19.514 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T20:27:19.515 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T20:27:19.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.640+0000 7eff2f10d700 1 Processor -- start 2024-03-18T20:27:19.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.640+0000 7eff2f10d700 1 -- start start 2024-03-18T20:27:19.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.641+0000 7eff2f10d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7eff280b2fa0 0x7eff2815c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:19.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.641+0000 7eff2f10d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff2815fc10 0x7eff280d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:19.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.641+0000 7eff2f10d700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff2815c8a0 con 0x7eff280b3380 2024-03-18T20:27:19.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.641+0000 7eff2f10d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff280cc0a0 con 0x7eff280b2fa0 2024-03-18T20:27:19.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.641+0000 7eff2f10d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff2815ef20 con 0x7eff2815fc10 2024-03-18T20:27:19.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.641+0000 7eff2cea9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7eff280b2fa0 0x7eff2815c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:19.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.641+0000 7eff2cea9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7eff280b2fa0 0x7eff2815c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50088/0 (socket says 172.21.15.1:50088) 2024-03-18T20:27:19.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.641+0000 7eff2cea9700 1 -- 172.21.15.1:0/3574875877 learned_addr learned my addr 172.21.15.1:0/3574875877 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:19.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.641+0000 7eff27fff700 1 --2- 172.21.15.1:0/3574875877 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff2815fc10 0x7eff280d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:19.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.641+0000 7eff2cea9700 1 -- 172.21.15.1:0/3574875877 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff2815fc10 msgr2=0x7eff280d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:19.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.641+0000 7eff2cea9700 1 --2- 172.21.15.1:0/3574875877 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff2815fc10 0x7eff280d16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.641+0000 7eff2cea9700 1 -- 172.21.15.1:0/3574875877 >> v1:172.21.15.165:6789/0 conn(0x7eff280b3380 legacy=0x7eff280b60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:19.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.641+0000 7eff2cea9700 1 -- 172.21.15.1:0/3574875877 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff280cbdc0 con 0x7eff280b2fa0 2024-03-18T20:27:19.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.641+0000 7eff2cea9700 1 --2- 172.21.15.1:0/3574875877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff280b2fa0 0x7eff2815c360 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7eff1c0098f0 tx=0x7eff1c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de31323466ecd514 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:19.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.642+0000 7eff277fe700 1 -- 172.21.15.1:0/3574875877 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c004230 con 0x7eff280b2fa0 2024-03-18T20:27:19.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.642+0000 7eff277fe700 1 -- 172.21.15.1:0/3574875877 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1c004390 con 0x7eff280b2fa0 2024-03-18T20:27:19.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.642+0000 7eff277fe700 1 -- 172.21.15.1:0/3574875877 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c0036a0 con 0x7eff280b2fa0 2024-03-18T20:27:19.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.642+0000 7eff2f10d700 1 -- 172.21.15.1:0/3574875877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff280b2fa0 msgr2=0x7eff2815c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:19.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.642+0000 7eff2f10d700 1 --2- 172.21.15.1:0/3574875877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff280b2fa0 0x7eff2815c360 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7eff1c0098f0 tx=0x7eff1c009c00 comp rx=0 tx=0).stop 2024-03-18T20:27:19.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.642+0000 7eff2f10d700 1 -- 172.21.15.1:0/3574875877 shutdown_connections 2024-03-18T20:27:19.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.642+0000 7eff2f10d700 1 --2- 172.21.15.1:0/3574875877 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff280b2fa0 0x7eff2815c360 unknown :-1 s=CLOSED pgs=3050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.642+0000 7eff2f10d700 1 --2- 172.21.15.1:0/3574875877 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff2815fc10 0x7eff280d16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.642+0000 7eff2f10d700 1 -- 172.21.15.1:0/3574875877 >> 172.21.15.1:0/3574875877 conn(0x7eff2806a750 msgr2=0x7eff2806ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:19.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.642+0000 7eff2f10d700 1 -- 172.21.15.1:0/3574875877 shutdown_connections 2024-03-18T20:27:19.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.642+0000 7eff2f10d700 1 -- 172.21.15.1:0/3574875877 wait complete. 2024-03-18T20:27:19.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.643+0000 7eff2f10d700 1 Processor -- start 2024-03-18T20:27:19.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.643+0000 7eff2f10d700 1 -- start start 2024-03-18T20:27:19.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.644+0000 7eff2f10d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff280b2fa0 0x7eff2814d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:19.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.644+0000 7eff2f10d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff280b3380 0x7eff28152aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:19.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.644+0000 7eff2f10d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff2815fc10 0x7eff28152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:19.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.644+0000 7eff2f10d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff280b3e90 con 0x7eff2815fc10 2024-03-18T20:27:19.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.644+0000 7eff2f10d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff280b40a0 con 0x7eff280b3380 2024-03-18T20:27:19.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.644+0000 7eff2f10d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff280b4380 con 0x7eff280b2fa0 2024-03-18T20:27:19.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.644+0000 7eff2d6aa700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff2815fc10 0x7eff28152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:19.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.644+0000 7eff2d6aa700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff2815fc10 0x7eff28152fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50092/0 (socket says 172.21.15.1:50092) 2024-03-18T20:27:19.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.644+0000 7eff2d6aa700 1 -- 172.21.15.1:0/1199583773 learned_addr learned my addr 172.21.15.1:0/1199583773 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:19.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.645+0000 7eff27fff700 1 --2- 172.21.15.1:0/1199583773 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff280b3380 0x7eff28152aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:19.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.645+0000 7eff2cea9700 1 --2- 172.21.15.1:0/1199583773 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff280b2fa0 0x7eff2814d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:19.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.645+0000 7eff2d6aa700 1 -- 172.21.15.1:0/1199583773 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff280b2fa0 msgr2=0x7eff2814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:19.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.645+0000 7eff2d6aa700 1 --2- 172.21.15.1:0/1199583773 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff280b2fa0 0x7eff2814d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.645+0000 7eff2d6aa700 1 -- 172.21.15.1:0/1199583773 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff280b3380 msgr2=0x7eff28152aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:19.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.645+0000 7eff2d6aa700 1 --2- 172.21.15.1:0/1199583773 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff280b3380 0x7eff28152aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:19.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.645+0000 7eff2d6aa700 1 -- 172.21.15.1:0/1199583773 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1c009550 con 0x7eff2815fc10 2024-03-18T20:27:19.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.645+0000 7eff2d6aa700 1 --2- 172.21.15.1:0/1199583773 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff2815fc10 0x7eff28152fe0 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 crypto rx=0x7eff2000cc10 tx=0x7eff2000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:19.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.645+0000 7eff257fa700 1 -- 172.21.15.1:0/1199583773 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2000abb0 con 0x7eff2815fc10 2024-03-18T20:27:19.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.645+0000 7eff257fa700 1 -- 172.21.15.1:0/1199583773 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff2000ad10 con 0x7eff2815fc10 2024-03-18T20:27:19.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.645+0000 7eff257fa700 1 -- 172.21.15.1:0/1199583773 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff20012660 con 0x7eff2815fc10 2024-03-18T20:27:19.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.645+0000 7eff2f10d700 1 -- 172.21.15.1:0/1199583773 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff280b4660 con 0x7eff2815fc10 2024-03-18T20:27:19.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.645+0000 7eff2f10d700 1 -- 172.21.15.1:0/1199583773 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff2814de00 con 0x7eff2815fc10 2024-03-18T20:27:19.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.646+0000 7eff2f10d700 1 -- 172.21.15.1:0/1199583773 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff280676a0 con 0x7eff2815fc10 2024-03-18T20:27:19.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.647+0000 7eff257fa700 1 -- 172.21.15.1:0/1199583773 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7eff200127c0 con 0x7eff2815fc10 2024-03-18T20:27:19.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.647+0000 7eff257fa700 1 --2- 172.21.15.1:0/1199583773 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7eff0c06af00 0x7eff0c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:19.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.647+0000 7eff257fa700 1 -- 172.21.15.1:0/1199583773 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 5277+0+0 (secure 0 0 0) 0x7eff20089db0 con 0x7eff2815fc10 2024-03-18T20:27:19.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.647+0000 7eff2cea9700 1 --2- 172.21.15.1:0/1199583773 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7eff0c06af00 0x7eff0c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:19.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.647+0000 7eff2cea9700 1 --2- 172.21.15.1:0/1199583773 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7eff0c06af00 0x7eff0c06d3c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7eff1c009130 tx=0x7eff1c008030 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:19.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.652+0000 7eff257fa700 1 -- 172.21.15.1:0/1199583773 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7eff20054d50 con 0x7eff2815fc10 2024-03-18T20:27:19.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:19.855+0000 7eff2f10d700 1 -- 172.21.15.1:0/1199583773 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7eff280ce510 con 0x7eff2815fc10 2024-03-18T20:27:20.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.615+0000 7eff257fa700 1 -- 172.21.15.1:0/1199583773 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v271) v1 ==== 138+0+0 (secure 0 0 0) 0x7eff20016030 con 0x7eff2815fc10 2024-03-18T20:27:20.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.618+0000 7eff2f10d700 1 -- 172.21.15.1:0/1199583773 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7eff0c06af00 msgr2=0x7eff0c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:20.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.618+0000 7eff2f10d700 1 --2- 172.21.15.1:0/1199583773 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7eff0c06af00 0x7eff0c06d3c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7eff1c009130 tx=0x7eff1c008030 comp rx=0 tx=0).stop 2024-03-18T20:27:20.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.618+0000 7eff2f10d700 1 -- 172.21.15.1:0/1199583773 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff2815fc10 msgr2=0x7eff28152fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:20.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.618+0000 7eff2f10d700 1 --2- 172.21.15.1:0/1199583773 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff2815fc10 0x7eff28152fe0 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 crypto rx=0x7eff2000cc10 tx=0x7eff2000a060 comp rx=0 tx=0).stop 2024-03-18T20:27:20.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.620+0000 7eff2f10d700 1 -- 172.21.15.1:0/1199583773 shutdown_connections 2024-03-18T20:27:20.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.620+0000 7eff2f10d700 1 --2- 172.21.15.1:0/1199583773 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7eff0c06af00 0x7eff0c06d3c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:20.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.620+0000 7eff2f10d700 1 --2- 172.21.15.1:0/1199583773 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7eff280b2fa0 0x7eff2814d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:20.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.620+0000 7eff2f10d700 1 --2- 172.21.15.1:0/1199583773 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff2815fc10 0x7eff28152fe0 unknown :-1 s=CLOSED pgs=3051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:20.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.620+0000 7eff2f10d700 1 --2- 172.21.15.1:0/1199583773 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff280b3380 0x7eff28152aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:20.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.620+0000 7eff2f10d700 1 -- 172.21.15.1:0/1199583773 >> 172.21.15.1:0/1199583773 conn(0x7eff2806a750 msgr2=0x7eff28154050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:20.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.620+0000 7eff2f10d700 1 -- 172.21.15.1:0/1199583773 shutdown_connections 2024-03-18T20:27:20.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.620+0000 7eff2f10d700 1 -- 172.21.15.1:0/1199583773 wait complete. 2024-03-18T20:27:20.626 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-18T20:27:20.642 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T20:27:20.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.776+0000 7f592b3c0700 1 Processor -- start 2024-03-18T20:27:20.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.776+0000 7f592b3c0700 1 -- start start 2024-03-18T20:27:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.777+0000 7f592b3c0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f592415ad70 0x7f592415d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.777+0000 7f592b3c0700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f592406a750 0x7f592406ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.777+0000 7f592b3c0700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5924160ea0 con 0x7f59240727b0 2024-03-18T20:27:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.777+0000 7f592b3c0700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59240725b0 con 0x7f592415ad70 2024-03-18T20:27:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.777+0000 7f592b3c0700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f592406b0b0 con 0x7f592406a750 2024-03-18T20:27:20.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.777+0000 7f592915c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f592415ad70 0x7f592415d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.777+0000 7f592915c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f592415ad70 0x7f592415d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50106/0 (socket says 172.21.15.1:50106) 2024-03-18T20:27:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.777+0000 7f592895b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f59240727b0 0x7f5924161610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33778/0 (socket says 172.21.15.1:33778) 2024-03-18T20:27:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.777+0000 7f592915c700 1 -- 172.21.15.1:0/1815598731 learned_addr learned my addr 172.21.15.1:0/1815598731 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.777+0000 7f592995d700 1 --2- 172.21.15.1:0/1815598731 >> v2:172.21.15.165:3300/0 conn(0x7f592406a750 0x7f592406ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.777+0000 7f592915c700 1 -- 172.21.15.1:0/1815598731 >> v2:172.21.15.165:3300/0 conn(0x7f592406a750 msgr2=0x7f592406ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.777+0000 7f592915c700 1 --2- 172.21.15.1:0/1815598731 >> v2:172.21.15.165:3300/0 conn(0x7f592406a750 0x7f592406ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.777+0000 7f592915c700 1 -- 172.21.15.1:0/1815598731 >> v1:172.21.15.1:6789/0 conn(0x7f59240727b0 legacy=0x7f5924161610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.777+0000 7f591bfff700 1 -- 172.21.15.1:0/1815598731 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949794522 0 0) 0x7f5924160ea0 con 0x7f59240727b0 2024-03-18T20:27:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.777+0000 7f592915c700 1 -- 172.21.15.1:0/1815598731 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592415ddf0 con 0x7f592415ad70 2024-03-18T20:27:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.777+0000 7f592915c700 1 --2- 172.21.15.1:0/1815598731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f592415ad70 0x7f592415d8b0 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7f59240b2c00 tx=0x7f5914009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f7e415df5f2c399 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.778+0000 7f591bfff700 1 -- 172.21.15.1:0/1815598731 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591400b870 con 0x7f592415ad70 2024-03-18T20:27:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.778+0000 7f591bfff700 1 -- 172.21.15.1:0/1815598731 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591400b9d0 con 0x7f592415ad70 2024-03-18T20:27:20.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.778+0000 7f591bfff700 1 -- 172.21.15.1:0/1815598731 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591400bd60 con 0x7f592415ad70 2024-03-18T20:27:20.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.778+0000 7f592b3c0700 1 -- 172.21.15.1:0/1815598731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f592415ad70 msgr2=0x7f592415d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:20.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.778+0000 7f592b3c0700 1 --2- 172.21.15.1:0/1815598731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f592415ad70 0x7f592415d8b0 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7f59240b2c00 tx=0x7f5914009bd0 comp rx=0 tx=0).stop 2024-03-18T20:27:20.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.778+0000 7f592b3c0700 1 -- 172.21.15.1:0/1815598731 shutdown_connections 2024-03-18T20:27:20.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.778+0000 7f592b3c0700 1 --2- 172.21.15.1:0/1815598731 >> v2:172.21.15.165:3300/0 conn(0x7f592406a750 0x7f592406ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:20.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.778+0000 7f592b3c0700 1 --2- 172.21.15.1:0/1815598731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f592415ad70 0x7f592415d8b0 unknown :-1 s=CLOSED pgs=3053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:20.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.778+0000 7f592b3c0700 1 -- 172.21.15.1:0/1815598731 >> 172.21.15.1:0/1815598731 conn(0x7f59240b1cb0 msgr2=0x7f59240b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:20.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.778+0000 7f592b3c0700 1 -- 172.21.15.1:0/1815598731 shutdown_connections 2024-03-18T20:27:20.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.778+0000 7f592b3c0700 1 -- 172.21.15.1:0/1815598731 wait complete. 2024-03-18T20:27:20.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.779+0000 7f592b3c0700 1 Processor -- start 2024-03-18T20:27:20.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.779+0000 7f592b3c0700 1 -- start start 2024-03-18T20:27:20.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.780+0000 7f592b3c0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f592406a750 0x7f5924144990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:20.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.780+0000 7f592b3c0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59240727b0 0x7f59240ad9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:20.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.780+0000 7f592b3c0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f592415ad70 0x7f59241f2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:20.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.780+0000 7f592b3c0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59240ac950 con 0x7f592406a750 2024-03-18T20:27:20.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.780+0000 7f592b3c0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59240acc30 con 0x7f59240727b0 2024-03-18T20:27:20.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.780+0000 7f592b3c0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59240acf10 con 0x7f592415ad70 2024-03-18T20:27:20.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.780+0000 7f592915c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f592406a750 0x7f5924144990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:20.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.780+0000 7f592915c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f592406a750 0x7f5924144990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50118/0 (socket says 172.21.15.1:50118) 2024-03-18T20:27:20.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.780+0000 7f592915c700 1 -- 172.21.15.1:0/1150351055 learned_addr learned my addr 172.21.15.1:0/1150351055 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:20.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.780+0000 7f592995d700 1 --2- 172.21.15.1:0/1150351055 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f592415ad70 0x7f59241f2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:20.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.781+0000 7f592915c700 1 -- 172.21.15.1:0/1150351055 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f592415ad70 msgr2=0x7f59241f2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:20.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.781+0000 7f592915c700 1 --2- 172.21.15.1:0/1150351055 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f592415ad70 0x7f59241f2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:20.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.781+0000 7f592915c700 1 -- 172.21.15.1:0/1150351055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59240727b0 msgr2=0x7f59240ad9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:20.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.781+0000 7f592915c700 1 --2- 172.21.15.1:0/1150351055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59240727b0 0x7f59240ad9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:20.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.781+0000 7f592915c700 1 -- 172.21.15.1:0/1150351055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5914009550 con 0x7f592406a750 2024-03-18T20:27:20.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.781+0000 7f592995d700 1 --2- 172.21.15.1:0/1150351055 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f592415ad70 0x7f59241f2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:20.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.781+0000 7f592915c700 1 --2- 172.21.15.1:0/1150351055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f592406a750 0x7f5924144990 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 crypto rx=0x7f5914000c00 tx=0x7f5914004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:20.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.781+0000 7f5919ffb700 1 -- 172.21.15.1:0/1150351055 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5914004520 con 0x7f592406a750 2024-03-18T20:27:20.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.781+0000 7f5919ffb700 1 -- 172.21.15.1:0/1150351055 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591401e070 con 0x7f592406a750 2024-03-18T20:27:20.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.781+0000 7f592b3c0700 1 -- 172.21.15.1:0/1150351055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59240ad1f0 con 0x7f592406a750 2024-03-18T20:27:20.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.781+0000 7f5919ffb700 1 -- 172.21.15.1:0/1150351055 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59140038a0 con 0x7f592406a750 2024-03-18T20:27:20.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.781+0000 7f592b3c0700 1 -- 172.21.15.1:0/1150351055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5924144ed0 con 0x7f592406a750 2024-03-18T20:27:20.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.782+0000 7f592b3c0700 1 -- 172.21.15.1:0/1150351055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59240ad710 con 0x7f592406a750 2024-03-18T20:27:20.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.783+0000 7f5919ffb700 1 -- 172.21.15.1:0/1150351055 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f5914003a00 con 0x7f592406a750 2024-03-18T20:27:20.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.783+0000 7f5919ffb700 1 --2- 172.21.15.1:0/1150351055 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f590806aed0 0x7f590806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:20.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.783+0000 7f5919ffb700 1 -- 172.21.15.1:0/1150351055 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5647+0+0 (secure 0 0 0) 0x7f5914020070 con 0x7f592406a750 2024-03-18T20:27:20.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.783+0000 7f592895b700 1 --2- 172.21.15.1:0/1150351055 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f590806aed0 0x7f590806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:20.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.783+0000 7f592895b700 1 --2- 172.21.15.1:0/1150351055 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f590806aed0 0x7f590806d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f59200011e0 tx=0x7f59200079b0 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:20.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.788+0000 7f5919ffb700 1 -- 172.21.15.1:0/1150351055 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f59240ad710 con 0x7f592406a750 2024-03-18T20:27:20.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:20.992+0000 7f592b3c0700 1 -- 172.21.15.1:0/1150351055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f59240ad710 con 0x7f592406a750 2024-03-18T20:27:21.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.620+0000 7f5919ffb700 1 -- 172.21.15.1:0/1150351055 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v272) v1 ==== 170+0+0 (secure 0 0 0) 0x7f59240ad710 con 0x7f592406a750 2024-03-18T20:27:21.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.627+0000 7f592b3c0700 1 -- 172.21.15.1:0/1150351055 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f590806aed0 msgr2=0x7f590806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:21.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.627+0000 7f592b3c0700 1 --2- 172.21.15.1:0/1150351055 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f590806aed0 0x7f590806d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f59200011e0 tx=0x7f59200079b0 comp rx=0 tx=0).stop 2024-03-18T20:27:21.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.627+0000 7f592b3c0700 1 -- 172.21.15.1:0/1150351055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f592406a750 msgr2=0x7f5924144990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:21.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.627+0000 7f592b3c0700 1 --2- 172.21.15.1:0/1150351055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f592406a750 0x7f5924144990 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 crypto rx=0x7f5914000c00 tx=0x7f5914004750 comp rx=0 tx=0).stop 2024-03-18T20:27:21.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.630+0000 7f592b3c0700 1 -- 172.21.15.1:0/1150351055 shutdown_connections 2024-03-18T20:27:21.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.630+0000 7f592b3c0700 1 --2- 172.21.15.1:0/1150351055 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f590806aed0 0x7f590806d390 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:21.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.630+0000 7f592b3c0700 1 --2- 172.21.15.1:0/1150351055 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f592415ad70 0x7f59241f2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:21.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.630+0000 7f592b3c0700 1 --2- 172.21.15.1:0/1150351055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f592406a750 0x7f5924144990 unknown :-1 s=CLOSED pgs=3054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:21.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.630+0000 7f592b3c0700 1 --2- 172.21.15.1:0/1150351055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59240727b0 0x7f59240ad9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:21.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.630+0000 7f592b3c0700 1 -- 172.21.15.1:0/1150351055 >> 172.21.15.1:0/1150351055 conn(0x7f59240b1cb0 msgr2=0x7f5924161510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:21.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.630+0000 7f592b3c0700 1 -- 172.21.15.1:0/1150351055 shutdown_connections 2024-03-18T20:27:21.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.633+0000 7f592b3c0700 1 -- 172.21.15.1:0/1150351055 wait complete. 2024-03-18T20:27:21.641 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-18T20:27:21.658 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T20:27:21.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.786+0000 7fded5f17700 1 Processor -- start 2024-03-18T20:27:21.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.786+0000 7fded5f17700 1 -- start start 2024-03-18T20:27:21.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.787+0000 7fded5f17700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fded00cfa90 0x7fded006a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:21.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.787+0000 7fded5f17700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded00cf600 con 0x7fded015d5a0 2024-03-18T20:27:21.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.787+0000 7fded5f17700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded00ce720 con 0x7fded006ac90 2024-03-18T20:27:21.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.787+0000 7fded5f17700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fded006b140 con 0x7fded00cfa90 2024-03-18T20:27:21.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.787+0000 7fdeceffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fded00cfa90 0x7fded006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:21.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.787+0000 7fdecf7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fded015d5a0 0x7fded00ce010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33792/0 (socket says 172.21.15.1:33792) 2024-03-18T20:27:21.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.787+0000 7fdecf7fe700 1 -- 172.21.15.1:0/1957381276 learned_addr learned my addr 172.21.15.1:0/1957381276 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:21.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.788+0000 7fdeceffd700 1 -- 172.21.15.1:0/1957381276 >> v1:172.21.15.165:6789/0 conn(0x7fded006ac90 legacy=0x7fded0167140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:21.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.788+0000 7fdeceffd700 1 -- 172.21.15.1:0/1957381276 >> v1:172.21.15.1:6789/0 conn(0x7fded015d5a0 legacy=0x7fded00ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:21.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.788+0000 7fdeceffd700 1 -- 172.21.15.1:0/1957381276 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded00ce9f0 con 0x7fded00cfa90 2024-03-18T20:27:21.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.788+0000 7fdece7fc700 1 -- 172.21.15.1:0/1957381276 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718028656 0 0) 0x7fded00cf600 con 0x7fded015d5a0 2024-03-18T20:27:21.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.788+0000 7fdeceffd700 1 --2- 172.21.15.1:0/1957381276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded00cfa90 0x7fded006a750 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009840 tx=0x7fdeb8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c777dbf4008d56 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:21.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.788+0000 7fdece7fc700 1 -- 172.21.15.1:0/1957381276 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8004230 con 0x7fded00cfa90 2024-03-18T20:27:21.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.788+0000 7fdece7fc700 1 -- 172.21.15.1:0/1957381276 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb8004390 con 0x7fded00cfa90 2024-03-18T20:27:21.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.788+0000 7fdece7fc700 1 -- 172.21.15.1:0/1957381276 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb80036a0 con 0x7fded00cfa90 2024-03-18T20:27:21.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.788+0000 7fded5f17700 1 -- 172.21.15.1:0/1957381276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded00cfa90 msgr2=0x7fded006a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:21.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.788+0000 7fded5f17700 1 --2- 172.21.15.1:0/1957381276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded00cfa90 0x7fded006a750 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009840 tx=0x7fdeb8009b50 comp rx=0 tx=0).stop 2024-03-18T20:27:21.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.788+0000 7fded5f17700 1 -- 172.21.15.1:0/1957381276 shutdown_connections 2024-03-18T20:27:21.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.788+0000 7fded5f17700 1 --2- 172.21.15.1:0/1957381276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded00cfa90 0x7fded006a750 unknown :-1 s=CLOSED pgs=3056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:21.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.788+0000 7fded5f17700 1 -- 172.21.15.1:0/1957381276 >> 172.21.15.1:0/1957381276 conn(0x7fded00cc9b0 msgr2=0x7fded00ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:21.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.788+0000 7fded5f17700 1 -- 172.21.15.1:0/1957381276 shutdown_connections 2024-03-18T20:27:21.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.788+0000 7fded5f17700 1 -- 172.21.15.1:0/1957381276 wait complete. 2024-03-18T20:27:21.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.790+0000 7fded5f17700 1 Processor -- start 2024-03-18T20:27:21.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.790+0000 7fded5f17700 1 -- start start 2024-03-18T20:27:21.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.791+0000 7fded5f17700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fded006ac90 0x7fded00c3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:21.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.791+0000 7fded5f17700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded00cfa90 0x7fded00b0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:21.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.791+0000 7fded5f17700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fded015d5a0 0x7fded00c5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:21.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.791+0000 7fded5f17700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded00c3bd0 con 0x7fded00cfa90 2024-03-18T20:27:21.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.791+0000 7fded5f17700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded01f3260 con 0x7fded006ac90 2024-03-18T20:27:21.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.791+0000 7fded5f17700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded01f3540 con 0x7fded015d5a0 2024-03-18T20:27:21.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.791+0000 7fdeceffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded00cfa90 0x7fded00b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:21.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.791+0000 7fdeceffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded00cfa90 0x7fded00b0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50134/0 (socket says 172.21.15.1:50134) 2024-03-18T20:27:21.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.791+0000 7fdeceffd700 1 -- 172.21.15.1:0/2479166886 learned_addr learned my addr 172.21.15.1:0/2479166886 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:21.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.791+0000 7fdecf7fe700 1 --2- 172.21.15.1:0/2479166886 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fded006ac90 0x7fded00c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:21.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.791+0000 7fdecffff700 1 --2- 172.21.15.1:0/2479166886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fded015d5a0 0x7fded00c5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:21.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.792+0000 7fdeceffd700 1 -- 172.21.15.1:0/2479166886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fded015d5a0 msgr2=0x7fded00c5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:21.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.792+0000 7fdeceffd700 1 --2- 172.21.15.1:0/2479166886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fded015d5a0 0x7fded00c5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:21.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.792+0000 7fdeceffd700 1 -- 172.21.15.1:0/2479166886 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fded006ac90 msgr2=0x7fded00c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:21.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.792+0000 7fdeceffd700 1 --2- 172.21.15.1:0/2479166886 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fded006ac90 0x7fded00c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:21.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.792+0000 7fdeceffd700 1 -- 172.21.15.1:0/2479166886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb80094a0 con 0x7fded00cfa90 2024-03-18T20:27:21.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.792+0000 7fdeceffd700 1 --2- 172.21.15.1:0/2479166886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded00cfa90 0x7fded00b0770 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7fded006cfd0 tx=0x7fdeb8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:21.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.792+0000 7fdec7fff700 1 -- 172.21.15.1:0/2479166886 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8011630 con 0x7fded00cfa90 2024-03-18T20:27:21.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.792+0000 7fdec7fff700 1 -- 172.21.15.1:0/2479166886 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb801e070 con 0x7fded00cfa90 2024-03-18T20:27:21.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.792+0000 7fded5f17700 1 -- 172.21.15.1:0/2479166886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded01f3820 con 0x7fded00cfa90 2024-03-18T20:27:21.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.792+0000 7fdec7fff700 1 -- 172.21.15.1:0/2479166886 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb80119f0 con 0x7fded00cfa90 2024-03-18T20:27:21.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.792+0000 7fded5f17700 1 -- 172.21.15.1:0/2479166886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded00b0cb0 con 0x7fded00cfa90 2024-03-18T20:27:21.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.793+0000 7fded5f17700 1 -- 172.21.15.1:0/2479166886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded00b0e40 con 0x7fded00cfa90 2024-03-18T20:27:21.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.794+0000 7fdec7fff700 1 -- 172.21.15.1:0/2479166886 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fdeb8011b50 con 0x7fded00cfa90 2024-03-18T20:27:21.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.794+0000 7fdec7fff700 1 --2- 172.21.15.1:0/2479166886 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdeb006aed0 0x7fdeb006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:21.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.794+0000 7fdec7fff700 1 -- 172.21.15.1:0/2479166886 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fdeb808b0a0 con 0x7fded00cfa90 2024-03-18T20:27:21.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.794+0000 7fdecf7fe700 1 --2- 172.21.15.1:0/2479166886 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdeb006aed0 0x7fdeb006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:21.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.795+0000 7fdecf7fe700 1 --2- 172.21.15.1:0/2479166886 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdeb006aed0 0x7fdeb006d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdec0007ae0 tx=0x7fdec0009040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:21.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:21.798+0000 7fdec7fff700 1 -- 172.21.15.1:0/2479166886 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdeb8055f50 con 0x7fded00cfa90 2024-03-18T20:27:22.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.045+0000 7fded5f17700 1 -- 172.21.15.1:0/2479166886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fded00676a0 con 0x7fded00cfa90 2024-03-18T20:27:22.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.050+0000 7fdec7fff700 1 -- 172.21.15.1:0/2479166886 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 35 and data pool 36 v132) v1 ==== 174+0+0 (secure 0 0 0) 0x7fdeb80200c0 con 0x7fded00cfa90 2024-03-18T20:27:22.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.055+0000 7fded5f17700 1 -- 172.21.15.1:0/2479166886 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdeb006aed0 msgr2=0x7fdeb006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:22.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.055+0000 7fded5f17700 1 --2- 172.21.15.1:0/2479166886 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdeb006aed0 0x7fdeb006d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdec0007ae0 tx=0x7fdec0009040 comp rx=0 tx=0).stop 2024-03-18T20:27:22.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.055+0000 7fded5f17700 1 -- 172.21.15.1:0/2479166886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded00cfa90 msgr2=0x7fded00b0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:22.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.055+0000 7fded5f17700 1 --2- 172.21.15.1:0/2479166886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded00cfa90 0x7fded00b0770 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7fded006cfd0 tx=0x7fdeb8004750 comp rx=0 tx=0).stop 2024-03-18T20:27:22.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.056+0000 7fded5f17700 1 -- 172.21.15.1:0/2479166886 shutdown_connections 2024-03-18T20:27:22.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.056+0000 7fded5f17700 1 --2- 172.21.15.1:0/2479166886 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdeb006aed0 0x7fdeb006d390 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.056+0000 7fded5f17700 1 --2- 172.21.15.1:0/2479166886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fded015d5a0 0x7fded00c5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.056+0000 7fded5f17700 1 --2- 172.21.15.1:0/2479166886 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fded006ac90 0x7fded00c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.056+0000 7fded5f17700 1 --2- 172.21.15.1:0/2479166886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fded00cfa90 0x7fded00b0770 unknown :-1 s=CLOSED pgs=3057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.056+0000 7fded5f17700 1 -- 172.21.15.1:0/2479166886 >> 172.21.15.1:0/2479166886 conn(0x7fded00cc9b0 msgr2=0x7fded00ce010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:22.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.056+0000 7fded5f17700 1 -- 172.21.15.1:0/2479166886 shutdown_connections 2024-03-18T20:27:22.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.056+0000 7fded5f17700 1 -- 172.21.15.1:0/2479166886 wait complete. 2024-03-18T20:27:22.066 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 35 and data pool 36 2024-03-18T20:27:22.083 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:27:22.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.203+0000 7fc729ba4700 1 Processor -- start 2024-03-18T20:27:22.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.204+0000 7fc729ba4700 1 -- start start 2024-03-18T20:27:22.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.204+0000 7fc729ba4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc724146190 0x7fc72406a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.204+0000 7fc729ba4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc724146af0 0x7fc72406ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.204+0000 7fc729ba4700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72406ebd0 con 0x7fc7241465a0 2024-03-18T20:27:22.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.204+0000 7fc729ba4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72406f850 con 0x7fc724146190 2024-03-18T20:27:22.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.204+0000 7fc729ba4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72406eeb0 con 0x7fc724146af0 2024-03-18T20:27:22.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.205+0000 7fc7237fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc724146190 0x7fc72406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.205+0000 7fc722ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc7241465a0 0x7fc724071650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33800/0 (socket says 172.21.15.1:33800) 2024-03-18T20:27:22.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.205+0000 7fc722ffd700 1 -- 172.21.15.1:0/3887709522 learned_addr learned my addr 172.21.15.1:0/3887709522 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:22.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.205+0000 7fc723fff700 1 --2- 172.21.15.1:0/3887709522 >> v2:172.21.15.165:3300/0 conn(0x7fc724146af0 0x7fc72406ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.205+0000 7fc7237fe700 1 -- 172.21.15.1:0/3887709522 >> v2:172.21.15.165:3300/0 conn(0x7fc724146af0 msgr2=0x7fc72406ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:22.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.205+0000 7fc7237fe700 1 --2- 172.21.15.1:0/3887709522 >> v2:172.21.15.165:3300/0 conn(0x7fc724146af0 0x7fc72406ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.205+0000 7fc7237fe700 1 -- 172.21.15.1:0/3887709522 >> v1:172.21.15.1:6789/0 conn(0x7fc7241465a0 legacy=0x7fc724071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:22.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.205+0000 7fc7237fe700 1 -- 172.21.15.1:0/3887709522 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72406f190 con 0x7fc724146190 2024-03-18T20:27:22.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.205+0000 7fc7227fc700 1 -- 172.21.15.1:0/3887709522 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249864751 0 0) 0x7fc72406ebd0 con 0x7fc7241465a0 2024-03-18T20:27:22.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.205+0000 7fc7237fe700 1 --2- 172.21.15.1:0/3887709522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc724146190 0x7fc72406a750 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7fc72406c6b0 tx=0x7fc70c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ada4a1617046235 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:22.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.205+0000 7fc7227fc700 1 -- 172.21.15.1:0/3887709522 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c011830 con 0x7fc724146190 2024-03-18T20:27:22.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.205+0000 7fc7227fc700 1 -- 172.21.15.1:0/3887709522 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70c011990 con 0x7fc724146190 2024-03-18T20:27:22.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.205+0000 7fc7227fc700 1 -- 172.21.15.1:0/3887709522 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c011d20 con 0x7fc724146190 2024-03-18T20:27:22.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.206+0000 7fc729ba4700 1 -- 172.21.15.1:0/3887709522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc724146190 msgr2=0x7fc72406a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:22.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.206+0000 7fc729ba4700 1 --2- 172.21.15.1:0/3887709522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc724146190 0x7fc72406a750 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7fc72406c6b0 tx=0x7fc70c009b20 comp rx=0 tx=0).stop 2024-03-18T20:27:22.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.206+0000 7fc729ba4700 1 -- 172.21.15.1:0/3887709522 shutdown_connections 2024-03-18T20:27:22.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.206+0000 7fc729ba4700 1 --2- 172.21.15.1:0/3887709522 >> v2:172.21.15.165:3300/0 conn(0x7fc724146af0 0x7fc72406ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.206+0000 7fc729ba4700 1 --2- 172.21.15.1:0/3887709522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc724146190 0x7fc72406a750 unknown :-1 s=CLOSED pgs=3059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.206+0000 7fc729ba4700 1 -- 172.21.15.1:0/3887709522 >> 172.21.15.1:0/3887709522 conn(0x7fc7240b1d20 msgr2=0x7fc7240b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:22.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.206+0000 7fc729ba4700 1 -- 172.21.15.1:0/3887709522 shutdown_connections 2024-03-18T20:27:22.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.206+0000 7fc729ba4700 1 -- 172.21.15.1:0/3887709522 wait complete. 2024-03-18T20:27:22.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.207+0000 7fc729ba4700 1 Processor -- start 2024-03-18T20:27:22.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.207+0000 7fc729ba4700 1 -- start start 2024-03-18T20:27:22.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.208+0000 7fc729ba4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7241465a0 0x7fc7240ce430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.208+0000 7fc729ba4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc724146af0 0x7fc72414c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.208+0000 7fc729ba4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7240c56a0 0x7fc72414c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.208+0000 7fc729ba4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7240cf390 con 0x7fc7241465a0 2024-03-18T20:27:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.208+0000 7fc729ba4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7240cedd0 con 0x7fc7240c56a0 2024-03-18T20:27:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.208+0000 7fc729ba4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7240cf0b0 con 0x7fc724146af0 2024-03-18T20:27:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.208+0000 7fc7237fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7241465a0 0x7fc7240ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.208+0000 7fc7237fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7241465a0 0x7fc7240ce430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50146/0 (socket says 172.21.15.1:50146) 2024-03-18T20:27:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.208+0000 7fc7237fe700 1 -- 172.21.15.1:0/3564500898 learned_addr learned my addr 172.21.15.1:0/3564500898 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.208+0000 7fc723fff700 1 --2- 172.21.15.1:0/3564500898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7240c56a0 0x7fc72414c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.208+0000 7fc722ffd700 1 --2- 172.21.15.1:0/3564500898 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc724146af0 0x7fc72414c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.208+0000 7fc7237fe700 1 -- 172.21.15.1:0/3564500898 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc724146af0 msgr2=0x7fc72414c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.208+0000 7fc7237fe700 1 --2- 172.21.15.1:0/3564500898 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc724146af0 0x7fc72414c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.208+0000 7fc7237fe700 1 -- 172.21.15.1:0/3564500898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7240c56a0 msgr2=0x7fc72414c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.208+0000 7fc7237fe700 1 --2- 172.21.15.1:0/3564500898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7240c56a0 0x7fc72414c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.208+0000 7fc7237fe700 1 -- 172.21.15.1:0/3564500898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c0094a0 con 0x7fc7241465a0 2024-03-18T20:27:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.208+0000 7fc723fff700 1 --2- 172.21.15.1:0/3564500898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7240c56a0 0x7fc72414c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.209+0000 7fc7237fe700 1 --2- 172.21.15.1:0/3564500898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7241465a0 0x7fc7240ce430 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0090b0 tx=0x7fc70c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.209+0000 7fc71bfff700 1 -- 172.21.15.1:0/3564500898 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c0038a0 con 0x7fc7241465a0 2024-03-18T20:27:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.209+0000 7fc71bfff700 1 -- 172.21.15.1:0/3564500898 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70c01f070 con 0x7fc7241465a0 2024-03-18T20:27:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.209+0000 7fc729ba4700 1 -- 172.21.15.1:0/3564500898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72414a350 con 0x7fc7241465a0 2024-03-18T20:27:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.209+0000 7fc71bfff700 1 -- 172.21.15.1:0/3564500898 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c003c20 con 0x7fc7241465a0 2024-03-18T20:27:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.209+0000 7fc729ba4700 1 -- 172.21.15.1:0/3564500898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7240ce970 con 0x7fc7241465a0 2024-03-18T20:27:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.210+0000 7fc729ba4700 1 -- 172.21.15.1:0/3564500898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc72406b620 con 0x7fc7241465a0 2024-03-18T20:27:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.210+0000 7fc71bfff700 1 -- 172.21.15.1:0/3564500898 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fc70c01a430 con 0x7fc7241465a0 2024-03-18T20:27:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.211+0000 7fc71bfff700 1 --2- 172.21.15.1:0/3564500898 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc70406af00 0x7fc70406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.211+0000 7fc71bfff700 1 -- 172.21.15.1:0/3564500898 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fc70c021070 con 0x7fc7241465a0 2024-03-18T20:27:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.211+0000 7fc722ffd700 1 --2- 172.21.15.1:0/3564500898 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc70406af00 0x7fc70406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.211+0000 7fc722ffd700 1 --2- 172.21.15.1:0/3564500898 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc70406af00 0x7fc70406d3c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc724142950 tx=0x7fc714009040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:22.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.215+0000 7fc71bfff700 1 -- 172.21.15.1:0/3564500898 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc70c055e60 con 0x7fc7241465a0 2024-03-18T20:27:22.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.416+0000 7fc729ba4700 1 -- 172.21.15.1:0/3564500898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7241459f0 con 0x7fc7241465a0 2024-03-18T20:27:22.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.418+0000 7fc71bfff700 1 -- 172.21.15.1:0/3564500898 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v273) v1 ==== 74+0+16378 (secure 0 0 0) 0x7fc70c015090 con 0x7fc7241465a0 2024-03-18T20:27:22.423 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:22.423 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":273,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:27:22.047216+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:27:19.891286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"273","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-03-18T20:27:21.027315+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"273","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:27:22.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.420+0000 7fc729ba4700 1 -- 172.21.15.1:0/3564500898 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc70406af00 msgr2=0x7fc70406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:22.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.420+0000 7fc729ba4700 1 --2- 172.21.15.1:0/3564500898 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc70406af00 0x7fc70406d3c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc724142950 tx=0x7fc714009040 comp rx=0 tx=0).stop 2024-03-18T20:27:22.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.420+0000 7fc729ba4700 1 -- 172.21.15.1:0/3564500898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7241465a0 msgr2=0x7fc7240ce430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:22.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.420+0000 7fc729ba4700 1 --2- 172.21.15.1:0/3564500898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7241465a0 0x7fc7240ce430 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0090b0 tx=0x7fc70c004750 comp rx=0 tx=0).stop 2024-03-18T20:27:22.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.420+0000 7fc729ba4700 1 -- 172.21.15.1:0/3564500898 shutdown_connections 2024-03-18T20:27:22.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.420+0000 7fc729ba4700 1 --2- 172.21.15.1:0/3564500898 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc70406af00 0x7fc70406d3c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.420+0000 7fc729ba4700 1 --2- 172.21.15.1:0/3564500898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7240c56a0 0x7fc72414c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.420+0000 7fc729ba4700 1 --2- 172.21.15.1:0/3564500898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7241465a0 0x7fc7240ce430 unknown :-1 s=CLOSED pgs=3060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.420+0000 7fc729ba4700 1 --2- 172.21.15.1:0/3564500898 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc724146af0 0x7fc72414c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.420+0000 7fc729ba4700 1 -- 172.21.15.1:0/3564500898 >> 172.21.15.1:0/3564500898 conn(0x7fc7240b1d20 msgr2=0x7fc7240aca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:22.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.420+0000 7fc729ba4700 1 -- 172.21.15.1:0/3564500898 shutdown_connections 2024-03-18T20:27:22.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.420+0000 7fc729ba4700 1 -- 172.21.15.1:0/3564500898 wait complete. 2024-03-18T20:27:22.438 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:27:22.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.558+0000 7f6804c4d700 1 Processor -- start 2024-03-18T20:27:22.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.558+0000 7f6804c4d700 1 -- start start 2024-03-18T20:27:22.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.559+0000 7f6804c4d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6800099a40 0x7f6800099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.559+0000 7f6804c4d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f68000cc310 0x7f68000cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.559+0000 7f6804c4d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f680009a3e0 con 0x7f680015d680 2024-03-18T20:27:22.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.559+0000 7f6804c4d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68000d0c80 con 0x7f68000cc310 2024-03-18T20:27:22.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.559+0000 7f6804c4d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68000d09a0 con 0x7f6800099a40 2024-03-18T20:27:22.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.559+0000 7f67fe59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f680015d680 0x7f68000cff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33806/0 (socket says 172.21.15.1:33806) 2024-03-18T20:27:22.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.559+0000 7f67fe59c700 1 -- 172.21.15.1:0/2788481727 learned_addr learned my addr 172.21.15.1:0/2788481727 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:22.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fdd9b700 1 --2- 172.21.15.1:0/2788481727 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6800099a40 0x7f6800099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fed9d700 1 --2- 172.21.15.1:0/2788481727 >> v2:172.21.15.165:3300/0 conn(0x7f68000cc310 0x7f68000cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fd59a700 1 -- 172.21.15.1:0/2788481727 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169701902 0 0) 0x7f680009a3e0 con 0x7f680015d680 2024-03-18T20:27:22.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fd59a700 1 -- 172.21.15.1:0/2788481727 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67ec0035b0 con 0x7f680015d680 2024-03-18T20:27:22.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fd59a700 1 -- 172.21.15.1:0/2788481727 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 460867982 0 0) 0x7f67ec0035b0 con 0x7f680015d680 2024-03-18T20:27:22.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fd59a700 1 -- 172.21.15.1:0/2788481727 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6800099a40 msgr2=0x7f6800099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:22.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fd59a700 1 --2- 172.21.15.1:0/2788481727 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6800099a40 0x7f6800099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fd59a700 1 -- 172.21.15.1:0/2788481727 >> v2:172.21.15.165:3300/0 conn(0x7f68000cc310 msgr2=0x7f68000cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:22.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fd59a700 1 --2- 172.21.15.1:0/2788481727 >> v2:172.21.15.165:3300/0 conn(0x7f68000cc310 0x7f68000cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fd59a700 1 -- 172.21.15.1:0/2788481727 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68000cb830 con 0x7f680015d680 2024-03-18T20:27:22.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fd59a700 1 -- 172.21.15.1:0/2788481727 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f67f0001670 con 0x7f680015d680 2024-03-18T20:27:22.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fd59a700 1 -- 172.21.15.1:0/2788481727 >> v1:172.21.15.1:6789/0 conn(0x7f680015d680 legacy=0x7f68000cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:22.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fd59a700 1 --2- 172.21.15.1:0/2788481727 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67ec003a00 0x7f67ec003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fd59a700 1 --2- 172.21.15.1:0/2788481727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67ec0043f0 0x7f67ec0048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fd59a700 1 --2- 172.21.15.1:0/2788481727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67ec004f40 0x7f67ec00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fd59a700 1 -- 172.21.15.1:0/2788481727 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68000d0c80 con 0x7f67ec0043f0 2024-03-18T20:27:22.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fd59a700 1 -- 172.21.15.1:0/2788481727 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68000d09a0 con 0x7f67ec004f40 2024-03-18T20:27:22.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fd59a700 1 -- 172.21.15.1:0/2788481727 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec00bf50 con 0x7f67ec003a00 2024-03-18T20:27:22.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fdd9b700 1 --2- 172.21.15.1:0/2788481727 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6800099a40 0x7f6800099ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:27:22.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fdd9b700 1 --2- 172.21.15.1:0/2788481727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67ec0043f0 0x7f67ec0048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fe59c700 1 --2- 172.21.15.1:0/2788481727 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67ec003a00 0x7f67ec003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fed9d700 1 --2- 172.21.15.1:0/2788481727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67ec004f40 0x7f67ec00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fdd9b700 1 -- 172.21.15.1:0/2788481727 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67ec003a00 msgr2=0x7f67ec003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:22.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fdd9b700 1 --2- 172.21.15.1:0/2788481727 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67ec003a00 0x7f67ec003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fdd9b700 1 -- 172.21.15.1:0/2788481727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67ec004f40 msgr2=0x7f67ec00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:22.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fdd9b700 1 --2- 172.21.15.1:0/2788481727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f67ec004f40 0x7f67ec00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.560+0000 7f67fdd9b700 1 -- 172.21.15.1:0/2788481727 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ec0035b0 con 0x7f67ec0043f0 2024-03-18T20:27:22.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.561+0000 7f67fdd9b700 1 --2- 172.21.15.1:0/2788481727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67ec0043f0 0x7f67ec0048d0 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7f68000c4bb0 tx=0x7f67ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b67f38e05ae41ac8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:22.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.561+0000 7f67fd59a700 1 -- 172.21.15.1:0/2788481727 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e8004030 con 0x7f67ec0043f0 2024-03-18T20:27:22.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.561+0000 7f67fd59a700 1 -- 172.21.15.1:0/2788481727 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e8004190 con 0x7f67ec0043f0 2024-03-18T20:27:22.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.561+0000 7f67fd59a700 1 -- 172.21.15.1:0/2788481727 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e8004540 con 0x7f67ec0043f0 2024-03-18T20:27:22.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.561+0000 7f6804c4d700 1 -- 172.21.15.1:0/2788481727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67ec0043f0 msgr2=0x7f67ec0048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:22.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.561+0000 7f6804c4d700 1 --2- 172.21.15.1:0/2788481727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f67ec0043f0 0x7f67ec0048d0 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7f68000c4bb0 tx=0x7f67ec005880 comp rx=0 tx=0).stop 2024-03-18T20:27:22.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.561+0000 7f67fed9d700 1 -- 172.21.15.1:0/2788481727 reap_dead start 2024-03-18T20:27:22.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.561+0000 7f6804c4d700 1 -- 172.21.15.1:0/2788481727 shutdown_connections 2024-03-18T20:27:22.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.561+0000 7f6804c4d700 1 -- 172.21.15.1:0/2788481727 >> 172.21.15.1:0/2788481727 conn(0x7f68000b2ba0 msgr2=0x7f68000b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:22.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.561+0000 7f6804c4d700 1 -- 172.21.15.1:0/2788481727 shutdown_connections 2024-03-18T20:27:22.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.561+0000 7f6804c4d700 1 -- 172.21.15.1:0/2788481727 wait complete. 2024-03-18T20:27:22.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.563+0000 7f6804c4d700 1 Processor -- start 2024-03-18T20:27:22.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.563+0000 7f6804c4d700 1 -- start start 2024-03-18T20:27:22.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.563+0000 7f6804c4d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67ec0043f0 0x7f68000c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.563+0000 7f6804c4d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68000c8180 0x7f680014d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.563+0000 7f6804c4d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68000ca590 0x7f68000caa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.563+0000 7f6804c4d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68000c8600 con 0x7f68000ca590 2024-03-18T20:27:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.564+0000 7f6804c4d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68000caf50 con 0x7f68000c8180 2024-03-18T20:27:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.564+0000 7f6804c4d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68000c8810 con 0x7f67ec0043f0 2024-03-18T20:27:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.564+0000 7f67fed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68000ca590 0x7f68000caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.564+0000 7f67fed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68000ca590 0x7f68000caa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50164/0 (socket says 172.21.15.1:50164) 2024-03-18T20:27:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.564+0000 7f67fed9d700 1 -- 172.21.15.1:0/400125981 learned_addr learned my addr 172.21.15.1:0/400125981 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.564+0000 7f67fe59c700 1 --2- 172.21.15.1:0/400125981 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67ec0043f0 0x7f68000c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.564+0000 7f67fdd9b700 1 --2- 172.21.15.1:0/400125981 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68000c8180 0x7f680014d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.564+0000 7f67fed9d700 1 -- 172.21.15.1:0/400125981 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67ec0043f0 msgr2=0x7f68000c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.564+0000 7f67fed9d700 1 --2- 172.21.15.1:0/400125981 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67ec0043f0 0x7f68000c98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.564+0000 7f67fed9d700 1 -- 172.21.15.1:0/400125981 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68000c8180 msgr2=0x7f680014d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.564+0000 7f67fed9d700 1 --2- 172.21.15.1:0/400125981 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68000c8180 0x7f680014d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.564+0000 7f67fed9d700 1 -- 172.21.15.1:0/400125981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68000c8af0 con 0x7f68000ca590 2024-03-18T20:27:22.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.564+0000 7f67fe59c700 1 --2- 172.21.15.1:0/400125981 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67ec0043f0 0x7f68000c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:22.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.564+0000 7f67fed9d700 1 --2- 172.21.15.1:0/400125981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68000ca590 0x7f68000caa10 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7f680006d710 tx=0x7f67f8011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:22.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.564+0000 7f67f6ffd700 1 -- 172.21.15.1:0/400125981 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f800e040 con 0x7f68000ca590 2024-03-18T20:27:22.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.564+0000 7f67f6ffd700 1 -- 172.21.15.1:0/400125981 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f800f040 con 0x7f68000ca590 2024-03-18T20:27:22.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.564+0000 7f6804c4d700 1 -- 172.21.15.1:0/400125981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68000c8dd0 con 0x7f68000ca590 2024-03-18T20:27:22.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.564+0000 7f67f6ffd700 1 -- 172.21.15.1:0/400125981 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f800fdb0 con 0x7f68000ca590 2024-03-18T20:27:22.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.564+0000 7f6804c4d700 1 -- 172.21.15.1:0/400125981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68000c9390 con 0x7f68000ca590 2024-03-18T20:27:22.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.565+0000 7f6804c4d700 1 -- 172.21.15.1:0/400125981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f680006d230 con 0x7f68000ca590 2024-03-18T20:27:22.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.566+0000 7f67f6ffd700 1 -- 172.21.15.1:0/400125981 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f67f8015430 con 0x7f68000ca590 2024-03-18T20:27:22.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.566+0000 7f67f6ffd700 1 --2- 172.21.15.1:0/400125981 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f67e006aed0 0x7f67e006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.566+0000 7f67f6ffd700 1 -- 172.21.15.1:0/400125981 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f67f808a3f0 con 0x7f68000ca590 2024-03-18T20:27:22.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.566+0000 7f67fe59c700 1 --2- 172.21.15.1:0/400125981 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f67e006aed0 0x7f67e006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.567+0000 7f67fe59c700 1 --2- 172.21.15.1:0/400125981 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f67e006aed0 0x7f67e006d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f67f0001470 tx=0x7f67f0009040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:22.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.571+0000 7f67f6ffd700 1 -- 172.21.15.1:0/400125981 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f67f8054fa0 con 0x7f68000ca590 2024-03-18T20:27:22.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.776+0000 7f6804c4d700 1 -- 172.21.15.1:0/400125981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68000676a0 con 0x7f68000ca590 2024-03-18T20:27:22.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.777+0000 7f67f6ffd700 1 -- 172.21.15.1:0/400125981 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v273) v1 ==== 74+0+16378 (secure 0 0 0) 0x7f67f80585c0 con 0x7f68000ca590 2024-03-18T20:27:22.782 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:22.782 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":273,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:27:22.047216+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:27:19.891286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"273","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-03-18T20:27:21.027315+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"273","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:27:22.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.779+0000 7f6804c4d700 1 -- 172.21.15.1:0/400125981 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f67e006aed0 msgr2=0x7f67e006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:22.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.779+0000 7f6804c4d700 1 --2- 172.21.15.1:0/400125981 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f67e006aed0 0x7f67e006d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f67f0001470 tx=0x7f67f0009040 comp rx=0 tx=0).stop 2024-03-18T20:27:22.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.779+0000 7f6804c4d700 1 -- 172.21.15.1:0/400125981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68000ca590 msgr2=0x7f68000caa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:22.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.779+0000 7f6804c4d700 1 --2- 172.21.15.1:0/400125981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68000ca590 0x7f68000caa10 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7f680006d710 tx=0x7f67f8011f80 comp rx=0 tx=0).stop 2024-03-18T20:27:22.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.779+0000 7f6804c4d700 1 -- 172.21.15.1:0/400125981 shutdown_connections 2024-03-18T20:27:22.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.779+0000 7f6804c4d700 1 --2- 172.21.15.1:0/400125981 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f67e006aed0 0x7f67e006d390 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.779+0000 7f6804c4d700 1 --2- 172.21.15.1:0/400125981 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f67ec0043f0 0x7f68000c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.779+0000 7f6804c4d700 1 --2- 172.21.15.1:0/400125981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f68000ca590 0x7f68000caa10 unknown :-1 s=CLOSED pgs=3063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.779+0000 7f6804c4d700 1 --2- 172.21.15.1:0/400125981 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68000c8180 0x7f680014d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.779+0000 7f6804c4d700 1 -- 172.21.15.1:0/400125981 >> 172.21.15.1:0/400125981 conn(0x7f68000b2ba0 msgr2=0x7f68000b07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:22.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.779+0000 7f6804c4d700 1 -- 172.21.15.1:0/400125981 shutdown_connections 2024-03-18T20:27:22.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.779+0000 7f6804c4d700 1 -- 172.21.15.1:0/400125981 wait complete. 2024-03-18T20:27:22.797 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T20:27:22.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.920+0000 7f1e3b795700 1 Processor -- start 2024-03-18T20:27:22.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.920+0000 7f1e3b795700 1 -- start start 2024-03-18T20:27:22.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.921+0000 7f1e3b795700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1e34071490 0x7f1e3415b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.921+0000 7f1e3b795700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e3415a380 0x7f1e34071870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.921+0000 7f1e3b795700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1e340724e0 0x7f1e3415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.921+0000 7f1e3b795700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e3415b6f0 con 0x7f1e340724e0 2024-03-18T20:27:22.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.921+0000 7f1e3b795700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e340cb510 con 0x7f1e34071490 2024-03-18T20:27:22.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.921+0000 7f1e3b795700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e34071db0 con 0x7f1e3415a380 2024-03-18T20:27:22.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.921+0000 7f1e39d32700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1e340724e0 0x7f1e3415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.921+0000 7f1e39d32700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1e340724e0 0x7f1e3415dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50168/0 (socket says 172.21.15.1:50168) 2024-03-18T20:27:22.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.921+0000 7f1e39d32700 1 -- 172.21.15.1:0/1055575426 learned_addr learned my addr 172.21.15.1:0/1055575426 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:22.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.921+0000 7f1e38d30700 1 --2- 172.21.15.1:0/1055575426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e3415a380 0x7f1e34071870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.921+0000 7f1e39d32700 1 -- 172.21.15.1:0/1055575426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e3415a380 msgr2=0x7f1e34071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:22.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.921+0000 7f1e39d32700 1 --2- 172.21.15.1:0/1055575426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e3415a380 0x7f1e34071870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.921+0000 7f1e39d32700 1 -- 172.21.15.1:0/1055575426 >> v2:172.21.15.165:3300/0 conn(0x7f1e34071490 msgr2=0x7f1e3415b1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T20:27:22.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.921+0000 7f1e39d32700 1 --2- 172.21.15.1:0/1055575426 >> v2:172.21.15.165:3300/0 conn(0x7f1e34071490 0x7f1e3415b1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.921+0000 7f1e39d32700 1 -- 172.21.15.1:0/1055575426 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e340ca900 con 0x7f1e340724e0 2024-03-18T20:27:22.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.921+0000 7f1e39d32700 1 --2- 172.21.15.1:0/1055575426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e340724e0 0x7f1e3415dca0 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7f1e3406c7d0 tx=0x7f1e2c00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f733cd08f70f684 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:22.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.922+0000 7f1e2bfff700 1 -- 172.21.15.1:0/1055575426 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2c010070 con 0x7f1e340724e0 2024-03-18T20:27:22.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.922+0000 7f1e2bfff700 1 -- 172.21.15.1:0/1055575426 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2c00dce0 con 0x7f1e340724e0 2024-03-18T20:27:22.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.922+0000 7f1e2bfff700 1 -- 172.21.15.1:0/1055575426 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2c00b310 con 0x7f1e340724e0 2024-03-18T20:27:22.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.922+0000 7f1e3b795700 1 -- 172.21.15.1:0/1055575426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e340724e0 msgr2=0x7f1e3415dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:22.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.922+0000 7f1e3b795700 1 --2- 172.21.15.1:0/1055575426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e340724e0 0x7f1e3415dca0 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7f1e3406c7d0 tx=0x7f1e2c00d430 comp rx=0 tx=0).stop 2024-03-18T20:27:22.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.922+0000 7f1e3b795700 1 -- 172.21.15.1:0/1055575426 shutdown_connections 2024-03-18T20:27:22.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.922+0000 7f1e3b795700 1 --2- 172.21.15.1:0/1055575426 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e340724e0 0x7f1e3415dca0 unknown :-1 s=CLOSED pgs=3064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.922+0000 7f1e3b795700 1 --2- 172.21.15.1:0/1055575426 >> v2:172.21.15.165:3300/0 conn(0x7f1e34071490 0x7f1e3415b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.922+0000 7f1e3b795700 1 --2- 172.21.15.1:0/1055575426 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e3415a380 0x7f1e34071870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.922+0000 7f1e3b795700 1 -- 172.21.15.1:0/1055575426 >> 172.21.15.1:0/1055575426 conn(0x7f1e340b1aa0 msgr2=0x7f1e340b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:22.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.922+0000 7f1e3b795700 1 -- 172.21.15.1:0/1055575426 shutdown_connections 2024-03-18T20:27:22.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.922+0000 7f1e3b795700 1 -- 172.21.15.1:0/1055575426 wait complete. 2024-03-18T20:27:22.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.923+0000 7f1e3b795700 1 Processor -- start 2024-03-18T20:27:22.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.923+0000 7f1e3b795700 1 -- start start 2024-03-18T20:27:22.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.924+0000 7f1e3b795700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e34071490 0x7f1e341f2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.924+0000 7f1e3b795700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e340724e0 0x7f1e340c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.924+0000 7f1e3b795700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e3415a380 0x7f1e340c5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.924+0000 7f1e3b795700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e340c7430 con 0x7f1e34071490 2024-03-18T20:27:22.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.924+0000 7f1e3b795700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e340c8290 con 0x7f1e340724e0 2024-03-18T20:27:22.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.924+0000 7f1e3b795700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e340c6690 con 0x7f1e3415a380 2024-03-18T20:27:22.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.924+0000 7f1e39531700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e34071490 0x7f1e341f2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.924+0000 7f1e39531700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e34071490 0x7f1e341f2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50176/0 (socket says 172.21.15.1:50176) 2024-03-18T20:27:22.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.924+0000 7f1e39531700 1 -- 172.21.15.1:0/711174733 learned_addr learned my addr 172.21.15.1:0/711174733 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:22.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.925+0000 7f1e38d30700 1 --2- 172.21.15.1:0/711174733 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e340724e0 0x7f1e340c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.925+0000 7f1e39531700 1 -- 172.21.15.1:0/711174733 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e3415a380 msgr2=0x7f1e340c5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:22.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.925+0000 7f1e39531700 1 --2- 172.21.15.1:0/711174733 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e3415a380 0x7f1e340c5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.925+0000 7f1e39531700 1 -- 172.21.15.1:0/711174733 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e340724e0 msgr2=0x7f1e340c3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:22.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.925+0000 7f1e39531700 1 --2- 172.21.15.1:0/711174733 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e340724e0 0x7f1e340c3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:22.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.925+0000 7f1e39531700 1 -- 172.21.15.1:0/711174733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2c00bd40 con 0x7f1e34071490 2024-03-18T20:27:22.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.925+0000 7f1e39531700 1 --2- 172.21.15.1:0/711174733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e34071490 0x7f1e341f2cb0 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7f1e3000b470 tx=0x7f1e3000b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:22.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.925+0000 7f1e29ffb700 1 -- 172.21.15.1:0/711174733 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e30011830 con 0x7f1e34071490 2024-03-18T20:27:22.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.925+0000 7f1e29ffb700 1 -- 172.21.15.1:0/711174733 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e30011990 con 0x7f1e34071490 2024-03-18T20:27:22.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.925+0000 7f1e3b795700 1 -- 172.21.15.1:0/711174733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e340c7680 con 0x7f1e34071490 2024-03-18T20:27:22.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.925+0000 7f1e29ffb700 1 -- 172.21.15.1:0/711174733 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e30011d40 con 0x7f1e34071490 2024-03-18T20:27:22.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.925+0000 7f1e3b795700 1 -- 172.21.15.1:0/711174733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e340c7c40 con 0x7f1e34071490 2024-03-18T20:27:22.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.926+0000 7f1e3b795700 1 -- 172.21.15.1:0/711174733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e340cb980 con 0x7f1e34071490 2024-03-18T20:27:22.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.926+0000 7f1e29ffb700 1 -- 172.21.15.1:0/711174733 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f1e3001a400 con 0x7f1e34071490 2024-03-18T20:27:22.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.927+0000 7f1e29ffb700 1 --2- 172.21.15.1:0/711174733 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e1806aed0 0x7f1e1806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:22.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.927+0000 7f1e29ffb700 1 -- 172.21.15.1:0/711174733 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f1e300898a0 con 0x7f1e34071490 2024-03-18T20:27:22.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.927+0000 7f1e38d30700 1 --2- 172.21.15.1:0/711174733 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e1806aed0 0x7f1e1806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:22.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.927+0000 7f1e38d30700 1 --2- 172.21.15.1:0/711174733 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e1806aed0 0x7f1e1806d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1e341564e0 tx=0x7f1e24008040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:22.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:22.932+0000 7f1e29ffb700 1 -- 172.21.15.1:0/711174733 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1e340cb980 con 0x7f1e34071490 2024-03-18T20:27:23.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:23.186+0000 7f1e3b795700 1 -- 172.21.15.1:0/711174733 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1e340cb980 con 0x7f1e34071490 2024-03-18T20:27:24.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.113+0000 7f1e29ffb700 1 -- 172.21.15.1:0/711174733 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v135) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1e340cb980 con 0x7f1e34071490 2024-03-18T20:27:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.116+0000 7f1e3b795700 1 -- 172.21.15.1:0/711174733 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e1806aed0 msgr2=0x7f1e1806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.116+0000 7f1e3b795700 1 --2- 172.21.15.1:0/711174733 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e1806aed0 0x7f1e1806d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1e341564e0 tx=0x7f1e24008040 comp rx=0 tx=0).stop 2024-03-18T20:27:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.116+0000 7f1e3b795700 1 -- 172.21.15.1:0/711174733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e34071490 msgr2=0x7f1e341f2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:24.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.116+0000 7f1e3b795700 1 --2- 172.21.15.1:0/711174733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e34071490 0x7f1e341f2cb0 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7f1e3000b470 tx=0x7f1e3000b830 comp rx=0 tx=0).stop 2024-03-18T20:27:24.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.116+0000 7f1e3b795700 1 -- 172.21.15.1:0/711174733 shutdown_connections 2024-03-18T20:27:24.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.116+0000 7f1e3b795700 1 --2- 172.21.15.1:0/711174733 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1e1806aed0 0x7f1e1806d390 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:24.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.116+0000 7f1e3b795700 1 --2- 172.21.15.1:0/711174733 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1e3415a380 0x7f1e340c5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:24.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.116+0000 7f1e3b795700 1 --2- 172.21.15.1:0/711174733 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1e34071490 0x7f1e341f2cb0 unknown :-1 s=CLOSED pgs=3065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:24.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.116+0000 7f1e3b795700 1 --2- 172.21.15.1:0/711174733 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e340724e0 0x7f1e340c3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:24.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.116+0000 7f1e3b795700 1 -- 172.21.15.1:0/711174733 >> 172.21.15.1:0/711174733 conn(0x7f1e340b1aa0 msgr2=0x7f1e340cf030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:24.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.116+0000 7f1e3b795700 1 -- 172.21.15.1:0/711174733 shutdown_connections 2024-03-18T20:27:24.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.116+0000 7f1e3b795700 1 -- 172.21.15.1:0/711174733 wait complete. 2024-03-18T20:27:24.136 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-18T20:27:24.137 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T20:27:24.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.267+0000 7f12c0a58700 1 Processor -- start 2024-03-18T20:27:24.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.267+0000 7f12c0a58700 1 -- start start 2024-03-18T20:27:24.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.268+0000 7f12c0a58700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12bc071cc0 0x7f12bc0ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:24.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.268+0000 7f12c0a58700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f12bc06d560 0x7f12bc06fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:24.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.268+0000 7f12c0a58700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12bc070000 con 0x7f12bc15a240 2024-03-18T20:27:24.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.268+0000 7f12c0a58700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12bc0adff0 con 0x7f12bc06d560 2024-03-18T20:27:24.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.268+0000 7f12c0a58700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc0cac80 con 0x7f12bc071cc0 2024-03-18T20:27:24.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.268+0000 7f12bad9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f12bc06d560 0x7f12bc06fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:24.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.268+0000 7f12bad9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f12bc06d560 0x7f12bc06fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50188/0 (socket says 172.21.15.1:50188) 2024-03-18T20:27:24.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.268+0000 7f12bad9d700 1 -- 172.21.15.1:0/2117205350 learned_addr learned my addr 172.21.15.1:0/2117205350 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:24.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.268+0000 7f12bad9d700 1 -- 172.21.15.1:0/2117205350 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12bc071cc0 msgr2=0x7f12bc0ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:24.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.268+0000 7f12b9d9b700 1 --2- 172.21.15.1:0/2117205350 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12bc071cc0 0x7f12bc0ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:24.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.268+0000 7f12bad9d700 1 --2- 172.21.15.1:0/2117205350 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12bc071cc0 0x7f12bc0ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:24.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.268+0000 7f12bad9d700 1 -- 172.21.15.1:0/2117205350 >> v1:172.21.15.165:6789/0 conn(0x7f12bc15a240 legacy=0x7f12bc0715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:24.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.268+0000 7f12bad9d700 1 -- 172.21.15.1:0/2117205350 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12bc0caf60 con 0x7f12bc06d560 2024-03-18T20:27:24.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.268+0000 7f12bad9d700 1 --2- 172.21.15.1:0/2117205350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12bc06d560 0x7f12bc06fac0 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009e50 tx=0x7f12b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ddc1595c0aa5fa4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:24.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.269+0000 7f12b959a700 1 -- 172.21.15.1:0/2117205350 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b400e070 con 0x7f12bc06d560 2024-03-18T20:27:24.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.269+0000 7f12b959a700 1 -- 172.21.15.1:0/2117205350 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b4003a20 con 0x7f12bc06d560 2024-03-18T20:27:24.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.269+0000 7f12b959a700 1 -- 172.21.15.1:0/2117205350 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b4003dd0 con 0x7f12bc06d560 2024-03-18T20:27:24.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.269+0000 7f12c0a58700 1 -- 172.21.15.1:0/2117205350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12bc06d560 msgr2=0x7f12bc06fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:24.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.269+0000 7f12c0a58700 1 --2- 172.21.15.1:0/2117205350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12bc06d560 0x7f12bc06fac0 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009e50 tx=0x7f12b400b330 comp rx=0 tx=0).stop 2024-03-18T20:27:24.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.269+0000 7f12c0a58700 1 -- 172.21.15.1:0/2117205350 shutdown_connections 2024-03-18T20:27:24.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.269+0000 7f12c0a58700 1 --2- 172.21.15.1:0/2117205350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12bc06d560 0x7f12bc06fac0 unknown :-1 s=CLOSED pgs=3066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:24.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.269+0000 7f12c0a58700 1 --2- 172.21.15.1:0/2117205350 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12bc071cc0 0x7f12bc0ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:24.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.269+0000 7f12c0a58700 1 -- 172.21.15.1:0/2117205350 >> 172.21.15.1:0/2117205350 conn(0x7f12bc0cff30 msgr2=0x7f12bc0d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:24.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.269+0000 7f12c0a58700 1 -- 172.21.15.1:0/2117205350 shutdown_connections 2024-03-18T20:27:24.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.269+0000 7f12c0a58700 1 -- 172.21.15.1:0/2117205350 wait complete. 2024-03-18T20:27:24.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.271+0000 7f12c0a58700 1 Processor -- start 2024-03-18T20:27:24.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.271+0000 7f12c0a58700 1 -- start start 2024-03-18T20:27:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.272+0000 7f12c0a58700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12bc06d560 0x7f12bc1592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.272+0000 7f12c0a58700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12bc071cc0 0x7f12bc0b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.272+0000 7f12c0a58700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12bc15a240 0x7f12bc0c85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.272+0000 7f12c0a58700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc0c3e50 con 0x7f12bc071cc0 2024-03-18T20:27:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.272+0000 7f12c0a58700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc09d230 con 0x7f12bc15a240 2024-03-18T20:27:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.272+0000 7f12c0a58700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc09d510 con 0x7f12bc06d560 2024-03-18T20:27:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.272+0000 7f12b9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12bc071cc0 0x7f12bc0b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.272+0000 7f12b9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12bc071cc0 0x7f12bc0b2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50204/0 (socket says 172.21.15.1:50204) 2024-03-18T20:27:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.272+0000 7f12b9d9b700 1 -- 172.21.15.1:0/2948456665 learned_addr learned my addr 172.21.15.1:0/2948456665 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.272+0000 7f12ba59c700 1 --2- 172.21.15.1:0/2948456665 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12bc06d560 0x7f12bc1592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.272+0000 7f12bad9d700 1 --2- 172.21.15.1:0/2948456665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12bc15a240 0x7f12bc0c85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.272+0000 7f12b9d9b700 1 -- 172.21.15.1:0/2948456665 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12bc06d560 msgr2=0x7f12bc1592a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.272+0000 7f12b9d9b700 1 --2- 172.21.15.1:0/2948456665 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12bc06d560 0x7f12bc1592a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:24.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.272+0000 7f12b9d9b700 1 -- 172.21.15.1:0/2948456665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12bc15a240 msgr2=0x7f12bc0c85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.272+0000 7f12b9d9b700 1 --2- 172.21.15.1:0/2948456665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12bc15a240 0x7f12bc0c85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.272+0000 7f12b9d9b700 1 -- 172.21.15.1:0/2948456665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b4009b30 con 0x7f12bc071cc0 2024-03-18T20:27:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.272+0000 7f12bad9d700 1 --2- 172.21.15.1:0/2948456665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12bc15a240 0x7f12bc0c85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.273+0000 7f12b9d9b700 1 --2- 172.21.15.1:0/2948456665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12bc071cc0 0x7f12bc0b2b20 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f12ac00c7f0 tx=0x7f12ac00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.273+0000 7f12aaffd700 1 -- 172.21.15.1:0/2948456665 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac011800 con 0x7f12bc071cc0 2024-03-18T20:27:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.273+0000 7f12aaffd700 1 -- 172.21.15.1:0/2948456665 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ac011960 con 0x7f12bc071cc0 2024-03-18T20:27:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.273+0000 7f12c0a58700 1 -- 172.21.15.1:0/2948456665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12bc09d7f0 con 0x7f12bc071cc0 2024-03-18T20:27:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.273+0000 7f12aaffd700 1 -- 172.21.15.1:0/2948456665 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac011cf0 con 0x7f12bc071cc0 2024-03-18T20:27:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.273+0000 7f12c0a58700 1 -- 172.21.15.1:0/2948456665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12bc0b3060 con 0x7f12bc071cc0 2024-03-18T20:27:24.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.274+0000 7f12c0a58700 1 -- 172.21.15.1:0/2948456665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12bc0b32b0 con 0x7f12bc071cc0 2024-03-18T20:27:24.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.274+0000 7f12aaffd700 1 -- 172.21.15.1:0/2948456665 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f12ac018360 con 0x7f12bc071cc0 2024-03-18T20:27:24.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.275+0000 7f12aaffd700 1 --2- 172.21.15.1:0/2948456665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f129c06aed0 0x7f129c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:24.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.275+0000 7f12aaffd700 1 -- 172.21.15.1:0/2948456665 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f12ac089b20 con 0x7f12bc071cc0 2024-03-18T20:27:24.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.275+0000 7f12ba59c700 1 --2- 172.21.15.1:0/2948456665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f129c06aed0 0x7f129c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:24.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.275+0000 7f12ba59c700 1 --2- 172.21.15.1:0/2948456665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f129c06aed0 0x7f129c06d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009870 tx=0x7f12b0008040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:24.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.279+0000 7f12aaffd700 1 -- 172.21.15.1:0/2948456665 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f12ac054650 con 0x7f12bc071cc0 2024-03-18T20:27:24.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:24.527+0000 7f12c0a58700 1 -- 172.21.15.1:0/2948456665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f12bc0676a0 con 0x7f12bc071cc0 2024-03-18T20:27:25.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.122+0000 7f12aaffd700 1 -- 172.21.15.1:0/2948456665 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v136) v1 ==== 122+0+0 (secure 0 0 0) 0x7f12ac057c70 con 0x7f12bc071cc0 2024-03-18T20:27:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.125+0000 7f12c0a58700 1 -- 172.21.15.1:0/2948456665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f129c06aed0 msgr2=0x7f129c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.125+0000 7f12c0a58700 1 --2- 172.21.15.1:0/2948456665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f129c06aed0 0x7f129c06d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009870 tx=0x7f12b0008040 comp rx=0 tx=0).stop 2024-03-18T20:27:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.125+0000 7f12c0a58700 1 -- 172.21.15.1:0/2948456665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12bc071cc0 msgr2=0x7f12bc0b2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.125+0000 7f12c0a58700 1 --2- 172.21.15.1:0/2948456665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12bc071cc0 0x7f12bc0b2b20 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f12ac00c7f0 tx=0x7f12ac00cbb0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.125+0000 7f12c0a58700 1 -- 172.21.15.1:0/2948456665 shutdown_connections 2024-03-18T20:27:25.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.125+0000 7f12c0a58700 1 --2- 172.21.15.1:0/2948456665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f129c06aed0 0x7f129c06d390 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.125+0000 7f12c0a58700 1 --2- 172.21.15.1:0/2948456665 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12bc06d560 0x7f12bc1592a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.125+0000 7f12c0a58700 1 --2- 172.21.15.1:0/2948456665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f12bc15a240 0x7f12bc0c85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.125+0000 7f12c0a58700 1 --2- 172.21.15.1:0/2948456665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12bc071cc0 0x7f12bc0b2b20 unknown :-1 s=CLOSED pgs=3067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.125+0000 7f12c0a58700 1 -- 172.21.15.1:0/2948456665 >> 172.21.15.1:0/2948456665 conn(0x7f12bc0cff30 msgr2=0x7f12bc06f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:25.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.125+0000 7f12c0a58700 1 -- 172.21.15.1:0/2948456665 shutdown_connections 2024-03-18T20:27:25.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.125+0000 7f12c0a58700 1 -- 172.21.15.1:0/2948456665 wait complete. 2024-03-18T20:27:25.145 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T20:27:25.145 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:27:25.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.273+0000 7f92a88fc700 1 Processor -- start 2024-03-18T20:27:25.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.273+0000 7f92a88fc700 1 -- start start 2024-03-18T20:27:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.274+0000 7f92a88fc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92a40cc600 0x7f92a406dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.274+0000 7f92a88fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f92a40ca110 0x7f92a40ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.274+0000 7f92a88fc700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a406e9c0 con 0x7f92a415cfc0 2024-03-18T20:27:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.274+0000 7f92a88fc700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92a406e1f0 con 0x7f92a40ca110 2024-03-18T20:27:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.274+0000 7f92a88fc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92a40cd640 con 0x7f92a40cc600 2024-03-18T20:27:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.274+0000 7f92a2d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f92a40ca110 0x7f92a40ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.275+0000 7f92a2d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f92a40ca110 0x7f92a40ca5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36460/0 (socket says 172.21.15.1:36460) 2024-03-18T20:27:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.275+0000 7f92a2d9d700 1 -- 172.21.15.1:0/2302330347 learned_addr learned my addr 172.21.15.1:0/2302330347 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.275+0000 7f92a1d9b700 1 --2- 172.21.15.1:0/2302330347 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92a40cc600 0x7f92a406dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.275+0000 7f92a2d9d700 1 -- 172.21.15.1:0/2302330347 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92a40cc600 msgr2=0x7f92a406dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:25.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.275+0000 7f92a2d9d700 1 --2- 172.21.15.1:0/2302330347 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92a40cc600 0x7f92a406dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.275+0000 7f92a2d9d700 1 -- 172.21.15.1:0/2302330347 >> v1:172.21.15.165:6789/0 conn(0x7f92a415cfc0 legacy=0x7f92a40cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.275+0000 7f92a2d9d700 1 -- 172.21.15.1:0/2302330347 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a40cd920 con 0x7f92a40ca110 2024-03-18T20:27:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.275+0000 7f92a2d9d700 1 --2- 172.21.15.1:0/2302330347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92a40ca110 0x7f92a40ca5c0 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f929c009e50 tx=0x7f929c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd3cc4660c14ac8e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.275+0000 7f92a159a700 1 -- 172.21.15.1:0/2302330347 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929c00e070 con 0x7f92a40ca110 2024-03-18T20:27:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.275+0000 7f92a159a700 1 -- 172.21.15.1:0/2302330347 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f929c00b9c0 con 0x7f92a40ca110 2024-03-18T20:27:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.275+0000 7f92a159a700 1 -- 172.21.15.1:0/2302330347 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929c00bd70 con 0x7f92a40ca110 2024-03-18T20:27:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.276+0000 7f92a88fc700 1 -- 172.21.15.1:0/2302330347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92a40ca110 msgr2=0x7f92a40ca5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.276+0000 7f92a88fc700 1 --2- 172.21.15.1:0/2302330347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92a40ca110 0x7f92a40ca5c0 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f929c009e50 tx=0x7f929c00b330 comp rx=0 tx=0).stop 2024-03-18T20:27:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.276+0000 7f92a88fc700 1 -- 172.21.15.1:0/2302330347 shutdown_connections 2024-03-18T20:27:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.276+0000 7f92a88fc700 1 --2- 172.21.15.1:0/2302330347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92a40ca110 0x7f92a40ca5c0 unknown :-1 s=CLOSED pgs=3068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.276+0000 7f92a88fc700 1 --2- 172.21.15.1:0/2302330347 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92a40cc600 0x7f92a406dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.276+0000 7f92a88fc700 1 -- 172.21.15.1:0/2302330347 >> 172.21.15.1:0/2302330347 conn(0x7f92a40b1270 msgr2=0x7f92a40b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.276+0000 7f92a88fc700 1 -- 172.21.15.1:0/2302330347 shutdown_connections 2024-03-18T20:27:25.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.276+0000 7f92a88fc700 1 -- 172.21.15.1:0/2302330347 wait complete. 2024-03-18T20:27:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.277+0000 7f92a88fc700 1 Processor -- start 2024-03-18T20:27:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.277+0000 7f92a88fc700 1 -- start start 2024-03-18T20:27:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.278+0000 7f92a88fc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92a40ca110 0x7f92a41548f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.278+0000 7f92a88fc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92a40cc600 0x7f92a415ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.278+0000 7f92a88fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92a415cfc0 0x7f92a415b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.278+0000 7f92a88fc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a40c7080 con 0x7f92a415cfc0 2024-03-18T20:27:25.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.278+0000 7f92a88fc700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a4154e30 con 0x7f92a40ca110 2024-03-18T20:27:25.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.278+0000 7f92a88fc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92a40c6770 con 0x7f92a40cc600 2024-03-18T20:27:25.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.278+0000 7f92a2d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92a415cfc0 0x7f92a415b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:25.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.278+0000 7f92a2d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92a415cfc0 0x7f92a415b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36462/0 (socket says 172.21.15.1:36462) 2024-03-18T20:27:25.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.278+0000 7f92a2d9d700 1 -- 172.21.15.1:0/189589383 learned_addr learned my addr 172.21.15.1:0/189589383 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:25.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.278+0000 7f92a1d9b700 1 --2- 172.21.15.1:0/189589383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92a40cc600 0x7f92a415ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:25.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.278+0000 7f92a259c700 1 --2- 172.21.15.1:0/189589383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92a40ca110 0x7f92a41548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:25.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.278+0000 7f92a2d9d700 1 -- 172.21.15.1:0/189589383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92a40cc600 msgr2=0x7f92a415ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:25.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.278+0000 7f92a2d9d700 1 --2- 172.21.15.1:0/189589383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92a40cc600 0x7f92a415ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.278+0000 7f92a2d9d700 1 -- 172.21.15.1:0/189589383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92a40ca110 msgr2=0x7f92a41548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:25.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.278+0000 7f92a2d9d700 1 --2- 172.21.15.1:0/189589383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92a40ca110 0x7f92a41548f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.278+0000 7f92a2d9d700 1 -- 172.21.15.1:0/189589383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929c009b30 con 0x7f92a415cfc0 2024-03-18T20:27:25.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.279+0000 7f92a2d9d700 1 --2- 172.21.15.1:0/189589383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92a415cfc0 0x7f92a415b1b0 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7f929c00b730 tx=0x7f929c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:25.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.279+0000 7f9292ffd700 1 -- 172.21.15.1:0/189589383 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929c004830 con 0x7f92a415cfc0 2024-03-18T20:27:25.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.279+0000 7f9292ffd700 1 -- 172.21.15.1:0/189589383 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f929c0099b0 con 0x7f92a415cfc0 2024-03-18T20:27:25.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.279+0000 7f92a88fc700 1 -- 172.21.15.1:0/189589383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92a40c6070 con 0x7f92a415cfc0 2024-03-18T20:27:25.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.279+0000 7f9292ffd700 1 -- 172.21.15.1:0/189589383 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929c01c680 con 0x7f92a415cfc0 2024-03-18T20:27:25.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.279+0000 7f92a88fc700 1 -- 172.21.15.1:0/189589383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92a40c6ca0 con 0x7f92a415cfc0 2024-03-18T20:27:25.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.280+0000 7f92a88fc700 1 -- 172.21.15.1:0/189589383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a40c71c0 con 0x7f92a415cfc0 2024-03-18T20:27:25.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.280+0000 7f9292ffd700 1 -- 172.21.15.1:0/189589383 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f929c01c7e0 con 0x7f92a415cfc0 2024-03-18T20:27:25.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.281+0000 7f9292ffd700 1 --2- 172.21.15.1:0/189589383 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f928406aed0 0x7f928406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:25.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.281+0000 7f92a259c700 1 --2- 172.21.15.1:0/189589383 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f928406aed0 0x7f928406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:25.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.281+0000 7f9292ffd700 1 -- 172.21.15.1:0/189589383 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f929c08b530 con 0x7f92a415cfc0 2024-03-18T20:27:25.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.281+0000 7f92a259c700 1 --2- 172.21.15.1:0/189589383 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f928406aed0 0x7f928406d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f92a40c66d0 tx=0x7f9298009040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:25.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.286+0000 7f9292ffd700 1 -- 172.21.15.1:0/189589383 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f92a40c71c0 con 0x7f92a415cfc0 2024-03-18T20:27:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.535+0000 7f92a88fc700 1 -- 172.21.15.1:0/189589383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92a40c71c0 con 0x7f92a415cfc0 2024-03-18T20:27:25.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.536+0000 7f9292ffd700 1 -- 172.21.15.1:0/189589383 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+4749 (secure 0 0 0) 0x7f92a40c71c0 con 0x7f92a415cfc0 2024-03-18T20:27:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:25.541 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":136,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9938,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/1320583552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1320583552},{"type":"v1","addr":"172.21.15.165:6833","nonce":1320583552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131},{"gid":11566,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2669900315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2669900315},{"type":"v1","addr":"172.21.15.165:6835","nonce":2669900315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131},{"gid":13170,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2726723271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2726723271},{"type":"v1","addr":"172.21.15.1:6835","nonce":2726723271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:27:22.047616+0000","modified":"2024-03-18T20:27:25.113850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13173},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13173":{"gid":13173,"name":"c","rank":0,"incarnation":132,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/3909787366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":3909787366},{"type":"v1","addr":"172.21.15.1:6837","nonce":3909787366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-03-18T20:27:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.538+0000 7f92a88fc700 1 -- 172.21.15.1:0/189589383 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f928406aed0 msgr2=0x7f928406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.538+0000 7f92a88fc700 1 --2- 172.21.15.1:0/189589383 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f928406aed0 0x7f928406d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f92a40c66d0 tx=0x7f9298009040 comp rx=0 tx=0).stop 2024-03-18T20:27:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.538+0000 7f92a88fc700 1 -- 172.21.15.1:0/189589383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92a415cfc0 msgr2=0x7f92a415b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.538+0000 7f92a88fc700 1 --2- 172.21.15.1:0/189589383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92a415cfc0 0x7f92a415b1b0 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7f929c00b730 tx=0x7f929c004ad0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.538+0000 7f92a88fc700 1 -- 172.21.15.1:0/189589383 shutdown_connections 2024-03-18T20:27:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.538+0000 7f92a88fc700 1 --2- 172.21.15.1:0/189589383 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f928406aed0 0x7f928406d390 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.538+0000 7f92a88fc700 1 --2- 172.21.15.1:0/189589383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92a415cfc0 0x7f92a415b1b0 unknown :-1 s=CLOSED pgs=3069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.538+0000 7f92a88fc700 1 --2- 172.21.15.1:0/189589383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f92a40ca110 0x7f92a41548f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.538+0000 7f92a88fc700 1 --2- 172.21.15.1:0/189589383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f92a40cc600 0x7f92a415ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.538+0000 7f92a88fc700 1 -- 172.21.15.1:0/189589383 >> 172.21.15.1:0/189589383 conn(0x7f92a40b1270 msgr2=0x7f92a40cba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.538+0000 7f92a88fc700 1 -- 172.21.15.1:0/189589383 shutdown_connections 2024-03-18T20:27:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.538+0000 7f92a88fc700 1 -- 172.21.15.1:0/189589383 wait complete. 2024-03-18T20:27:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 136 2024-03-18T20:27:25.556 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:27:25.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.675+0000 7faa0e4e2700 1 Processor -- start 2024-03-18T20:27:25.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.675+0000 7faa0e4e2700 1 -- start start 2024-03-18T20:27:25.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.676+0000 7faa0e4e2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faa0806a750 0x7faa080cf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:25.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.676+0000 7faa0e4e2700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0806d640 con 0x7faa080cdda0 2024-03-18T20:27:25.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.676+0000 7faa0e4e2700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0806de50 con 0x7faa080cf960 2024-03-18T20:27:25.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.676+0000 7faa0e4e2700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa080acd80 con 0x7faa0806a750 2024-03-18T20:27:25.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.676+0000 7faa07fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faa0806a750 0x7faa080cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:25.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.676+0000 7faa0ca7f700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7faa080cdda0 0x7faa080ce250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37426/0 (socket says 172.21.15.1:37426) 2024-03-18T20:27:25.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.676+0000 7faa0ca7f700 1 -- 172.21.15.1:0/137298719 learned_addr learned my addr 172.21.15.1:0/137298719 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:25.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.677+0000 7faa07fff700 1 -- 172.21.15.1:0/137298719 >> v1:172.21.15.165:6789/0 conn(0x7faa080cf960 legacy=0x7faa0806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:25.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.677+0000 7faa07fff700 1 -- 172.21.15.1:0/137298719 >> v1:172.21.15.1:6789/0 conn(0x7faa080cdda0 legacy=0x7faa080ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:25.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.677+0000 7faa07fff700 1 -- 172.21.15.1:0/137298719 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0806d890 con 0x7faa0806a750 2024-03-18T20:27:25.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.677+0000 7faa06ffd700 1 -- 172.21.15.1:0/137298719 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282167594 0 0) 0x7faa0806d640 con 0x7faa080cdda0 2024-03-18T20:27:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.677+0000 7faa07fff700 1 --2- 172.21.15.1:0/137298719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0806a750 0x7faa080cf280 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8009840 tx=0x7fa9f8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb93db48e230804 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.677+0000 7faa06ffd700 1 -- 172.21.15.1:0/137298719 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8004230 con 0x7faa0806a750 2024-03-18T20:27:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.677+0000 7faa06ffd700 1 -- 172.21.15.1:0/137298719 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f8004390 con 0x7faa0806a750 2024-03-18T20:27:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.677+0000 7faa06ffd700 1 -- 172.21.15.1:0/137298719 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f80036a0 con 0x7faa0806a750 2024-03-18T20:27:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.678+0000 7faa0e4e2700 1 -- 172.21.15.1:0/137298719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0806a750 msgr2=0x7faa080cf280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.678+0000 7faa0e4e2700 1 --2- 172.21.15.1:0/137298719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0806a750 0x7faa080cf280 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8009840 tx=0x7fa9f8009b50 comp rx=0 tx=0).stop 2024-03-18T20:27:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.678+0000 7faa0e4e2700 1 -- 172.21.15.1:0/137298719 shutdown_connections 2024-03-18T20:27:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.678+0000 7faa0e4e2700 1 --2- 172.21.15.1:0/137298719 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0806a750 0x7faa080cf280 unknown :-1 s=CLOSED pgs=3071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.678+0000 7faa0e4e2700 1 -- 172.21.15.1:0/137298719 >> 172.21.15.1:0/137298719 conn(0x7faa0806b860 msgr2=0x7faa0806bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.678+0000 7faa0e4e2700 1 -- 172.21.15.1:0/137298719 shutdown_connections 2024-03-18T20:27:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.678+0000 7faa0e4e2700 1 -- 172.21.15.1:0/137298719 wait complete. 2024-03-18T20:27:25.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.679+0000 7faa0e4e2700 1 Processor -- start 2024-03-18T20:27:25.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.679+0000 7faa0e4e2700 1 -- start start 2024-03-18T20:27:25.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.680+0000 7faa0e4e2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0806a750 0x7faa08154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:25.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.680+0000 7faa0e4e2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faa080cdda0 0x7faa0815dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:25.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.680+0000 7faa0e4e2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa080cf960 0x7faa0815e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:25.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.680+0000 7faa0e4e2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa08152de0 con 0x7faa0806a750 2024-03-18T20:27:25.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.680+0000 7faa0e4e2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0815e760 con 0x7faa080cf960 2024-03-18T20:27:25.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.680+0000 7faa0e4e2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa08152440 con 0x7faa080cdda0 2024-03-18T20:27:25.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.680+0000 7faa07fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0806a750 0x7faa08154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:25.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.680+0000 7faa07fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0806a750 0x7faa08154e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36480/0 (socket says 172.21.15.1:36480) 2024-03-18T20:27:25.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.680+0000 7faa07fff700 1 -- 172.21.15.1:0/490669562 learned_addr learned my addr 172.21.15.1:0/490669562 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.680+0000 7faa0ca7f700 1 --2- 172.21.15.1:0/490669562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa080cf960 0x7faa0815e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.680+0000 7faa077fe700 1 --2- 172.21.15.1:0/490669562 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faa080cdda0 0x7faa0815dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.680+0000 7faa07fff700 1 -- 172.21.15.1:0/490669562 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faa080cdda0 msgr2=0x7faa0815dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.680+0000 7faa07fff700 1 --2- 172.21.15.1:0/490669562 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faa080cdda0 0x7faa0815dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.680+0000 7faa07fff700 1 -- 172.21.15.1:0/490669562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa080cf960 msgr2=0x7faa0815e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.680+0000 7faa07fff700 1 --2- 172.21.15.1:0/490669562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa080cf960 0x7faa0815e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.680+0000 7faa07fff700 1 -- 172.21.15.1:0/490669562 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f80094a0 con 0x7faa0806a750 2024-03-18T20:27:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.680+0000 7faa07fff700 1 --2- 172.21.15.1:0/490669562 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0806a750 0x7faa08154e90 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 crypto rx=0x7faa080672c0 tx=0x7fa9f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.680+0000 7faa04ff9700 1 -- 172.21.15.1:0/490669562 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8011630 con 0x7faa0806a750 2024-03-18T20:27:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.681+0000 7faa04ff9700 1 -- 172.21.15.1:0/490669562 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f801e070 con 0x7faa0806a750 2024-03-18T20:27:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.681+0000 7faa0e4e2700 1 -- 172.21.15.1:0/490669562 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa08152720 con 0x7faa0806a750 2024-03-18T20:27:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.681+0000 7faa04ff9700 1 -- 172.21.15.1:0/490669562 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f80119f0 con 0x7faa0806a750 2024-03-18T20:27:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.681+0000 7faa0e4e2700 1 -- 172.21.15.1:0/490669562 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa08151b20 con 0x7faa0806a750 2024-03-18T20:27:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.681+0000 7faa0e4e2700 1 -- 172.21.15.1:0/490669562 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa08158880 con 0x7faa0806a750 2024-03-18T20:27:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.682+0000 7faa04ff9700 1 -- 172.21.15.1:0/490669562 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fa9f8011b50 con 0x7faa0806a750 2024-03-18T20:27:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.682+0000 7faa04ff9700 1 --2- 172.21.15.1:0/490669562 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa9f406aed0 0x7fa9f406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.683+0000 7faa077fe700 1 --2- 172.21.15.1:0/490669562 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa9f406aed0 0x7fa9f406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.683+0000 7faa04ff9700 1 -- 172.21.15.1:0/490669562 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fa9f808ac50 con 0x7faa0806a750 2024-03-18T20:27:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.683+0000 7faa077fe700 1 --2- 172.21.15.1:0/490669562 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa9f406aed0 0x7fa9f406d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faa0815a6d0 tx=0x7fa9fc008040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:25.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.687+0000 7faa04ff9700 1 -- 172.21.15.1:0/490669562 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa9f8090020 con 0x7faa0806a750 2024-03-18T20:27:25.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.889+0000 7faa0e4e2700 1 -- 172.21.15.1:0/490669562 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa080676a0 con 0x7faa0806a750 2024-03-18T20:27:25.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.890+0000 7faa04ff9700 1 -- 172.21.15.1:0/490669562 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v274) v1 ==== 74+0+16365 (secure 0 0 0) 0x7fa9f8020020 con 0x7faa0806a750 2024-03-18T20:27:25.895 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:25.895 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":274,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:27:23.049336+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:27:19.891286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"273","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-03-18T20:27:21.027315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"274","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:27:25.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.892+0000 7faa0e4e2700 1 -- 172.21.15.1:0/490669562 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa9f406aed0 msgr2=0x7fa9f406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:25.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.892+0000 7faa0e4e2700 1 --2- 172.21.15.1:0/490669562 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa9f406aed0 0x7fa9f406d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faa0815a6d0 tx=0x7fa9fc008040 comp rx=0 tx=0).stop 2024-03-18T20:27:25.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.892+0000 7faa0e4e2700 1 -- 172.21.15.1:0/490669562 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0806a750 msgr2=0x7faa08154e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:25.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.892+0000 7faa0e4e2700 1 --2- 172.21.15.1:0/490669562 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0806a750 0x7faa08154e90 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 crypto rx=0x7faa080672c0 tx=0x7fa9f8004750 comp rx=0 tx=0).stop 2024-03-18T20:27:25.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.892+0000 7faa0e4e2700 1 -- 172.21.15.1:0/490669562 shutdown_connections 2024-03-18T20:27:25.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.892+0000 7faa0e4e2700 1 --2- 172.21.15.1:0/490669562 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa9f406aed0 0x7fa9f406d390 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.892+0000 7faa0e4e2700 1 --2- 172.21.15.1:0/490669562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa080cf960 0x7faa0815e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.892+0000 7faa0e4e2700 1 --2- 172.21.15.1:0/490669562 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa0806a750 0x7faa08154e90 unknown :-1 s=CLOSED pgs=3072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.892+0000 7faa0e4e2700 1 --2- 172.21.15.1:0/490669562 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faa080cdda0 0x7faa0815dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:25.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.892+0000 7faa0e4e2700 1 -- 172.21.15.1:0/490669562 >> 172.21.15.1:0/490669562 conn(0x7faa0806b860 msgr2=0x7faa080cc2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:25.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.892+0000 7faa0e4e2700 1 -- 172.21.15.1:0/490669562 shutdown_connections 2024-03-18T20:27:25.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:25.893+0000 7faa0e4e2700 1 -- 172.21.15.1:0/490669562 wait complete. 2024-03-18T20:27:25.910 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T20:27:25.910 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:27:26.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.031+0000 7fe09e2ff700 1 Processor -- start 2024-03-18T20:27:26.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.031+0000 7fe09e2ff700 1 -- start start 2024-03-18T20:27:26.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.032+0000 7fe09e2ff700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe098071600 0x7fe098071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.032+0000 7fe09e2ff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe09806e2c0 0x7fe09806e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.032+0000 7fe09e2ff700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0980acd80 con 0x7fe09815a320 2024-03-18T20:27:26.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.032+0000 7fe09e2ff700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe098071fc0 con 0x7fe09806e2c0 2024-03-18T20:27:26.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.032+0000 7fe09e2ff700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0980cf220 con 0x7fe098071600 2024-03-18T20:27:26.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.032+0000 7fe09c89c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe09806e2c0 0x7fe09806e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.032+0000 7fe097fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe09815a320 0x7fe0980ce4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37438/0 (socket says 172.21.15.1:37438) 2024-03-18T20:27:26.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.032+0000 7fe09c89c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe09806e2c0 0x7fe09806e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36492/0 (socket says 172.21.15.1:36492) 2024-03-18T20:27:26.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.032+0000 7fe097fff700 1 -- 172.21.15.1:0/1068310631 learned_addr learned my addr 172.21.15.1:0/1068310631 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:26.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.032+0000 7fe0977fe700 1 --2- 172.21.15.1:0/1068310631 >> v2:172.21.15.165:3300/0 conn(0x7fe098071600 0x7fe098071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.032+0000 7fe09c89c700 1 -- 172.21.15.1:0/1068310631 >> v2:172.21.15.165:3300/0 conn(0x7fe098071600 msgr2=0x7fe098071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:26.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.032+0000 7fe09c89c700 1 --2- 172.21.15.1:0/1068310631 >> v2:172.21.15.165:3300/0 conn(0x7fe098071600 0x7fe098071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.032+0000 7fe09c89c700 1 -- 172.21.15.1:0/1068310631 >> v1:172.21.15.1:6789/0 conn(0x7fe09815a320 legacy=0x7fe0980ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:26.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.032+0000 7fe09c89c700 1 -- 172.21.15.1:0/1068310631 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0980cef40 con 0x7fe09806e2c0 2024-03-18T20:27:26.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.032+0000 7fe096ffd700 1 -- 172.21.15.1:0/1068310631 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138443943 0 0) 0x7fe0980acd80 con 0x7fe09815a320 2024-03-18T20:27:26.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.033+0000 7fe09c89c700 1 --2- 172.21.15.1:0/1068310631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09806e2c0 0x7fe09806e750 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7fe0980672b0 tx=0x7fe09000d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be20ebdbce4e1c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:26.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.033+0000 7fe096ffd700 1 -- 172.21.15.1:0/1068310631 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe090010070 con 0x7fe09806e2c0 2024-03-18T20:27:26.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.033+0000 7fe096ffd700 1 -- 172.21.15.1:0/1068310631 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09000b2e0 con 0x7fe09806e2c0 2024-03-18T20:27:26.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.033+0000 7fe096ffd700 1 -- 172.21.15.1:0/1068310631 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09000dcc0 con 0x7fe09806e2c0 2024-03-18T20:27:26.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.033+0000 7fe09e2ff700 1 -- 172.21.15.1:0/1068310631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09806e2c0 msgr2=0x7fe09806e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:26.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.033+0000 7fe09e2ff700 1 --2- 172.21.15.1:0/1068310631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09806e2c0 0x7fe09806e750 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7fe0980672b0 tx=0x7fe09000d330 comp rx=0 tx=0).stop 2024-03-18T20:27:26.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.033+0000 7fe09e2ff700 1 -- 172.21.15.1:0/1068310631 shutdown_connections 2024-03-18T20:27:26.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.033+0000 7fe09e2ff700 1 --2- 172.21.15.1:0/1068310631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09806e2c0 0x7fe09806e750 unknown :-1 s=CLOSED pgs=3074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.033+0000 7fe09e2ff700 1 --2- 172.21.15.1:0/1068310631 >> v2:172.21.15.165:3300/0 conn(0x7fe098071600 0x7fe098071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.033+0000 7fe09e2ff700 1 -- 172.21.15.1:0/1068310631 >> 172.21.15.1:0/1068310631 conn(0x7fe0980cc990 msgr2=0x7fe0980ccda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:26.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.033+0000 7fe09e2ff700 1 -- 172.21.15.1:0/1068310631 shutdown_connections 2024-03-18T20:27:26.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.033+0000 7fe09e2ff700 1 -- 172.21.15.1:0/1068310631 wait complete. 2024-03-18T20:27:26.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.034+0000 7fe09e2ff700 1 Processor -- start 2024-03-18T20:27:26.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.035+0000 7fe09e2ff700 1 -- start start 2024-03-18T20:27:26.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.035+0000 7fe09e2ff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe09806e2c0 0x7fe0981f1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.035+0000 7fe09e2ff700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe098071600 0x7fe09806bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.035+0000 7fe09e2ff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09806c0b0 0x7fe09814f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.035+0000 7fe09e2ff700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe098150dd0 con 0x7fe09806c0b0 2024-03-18T20:27:26.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.035+0000 7fe09e2ff700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09806c560 con 0x7fe098071600 2024-03-18T20:27:26.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.035+0000 7fe09e2ff700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0981f30f0 con 0x7fe09806e2c0 2024-03-18T20:27:26.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.036+0000 7fe09c89c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09806c0b0 0x7fe09814f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.036+0000 7fe09c89c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09806c0b0 0x7fe09814f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36504/0 (socket says 172.21.15.1:36504) 2024-03-18T20:27:26.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.036+0000 7fe09c89c700 1 -- 172.21.15.1:0/589094043 learned_addr learned my addr 172.21.15.1:0/589094043 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:26.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.036+0000 7fe097fff700 1 --2- 172.21.15.1:0/589094043 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe09806e2c0 0x7fe0981f1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.036+0000 7fe0977fe700 1 --2- 172.21.15.1:0/589094043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe098071600 0x7fe09806bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.036+0000 7fe09c89c700 1 -- 172.21.15.1:0/589094043 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe09806e2c0 msgr2=0x7fe0981f1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:26.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.036+0000 7fe09c89c700 1 --2- 172.21.15.1:0/589094043 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe09806e2c0 0x7fe0981f1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.036+0000 7fe09c89c700 1 -- 172.21.15.1:0/589094043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe098071600 msgr2=0x7fe09806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:26.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.036+0000 7fe09c89c700 1 --2- 172.21.15.1:0/589094043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe098071600 0x7fe09806bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.036+0000 7fe09c89c700 1 -- 172.21.15.1:0/589094043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09000bc80 con 0x7fe09806c0b0 2024-03-18T20:27:26.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.036+0000 7fe09c89c700 1 --2- 172.21.15.1:0/589094043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09806c0b0 0x7fe09814f390 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7fe09000d730 tx=0x7fe090004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:26.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.036+0000 7fe094ff9700 1 -- 172.21.15.1:0/589094043 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09000b450 con 0x7fe09806c0b0 2024-03-18T20:27:26.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.036+0000 7fe094ff9700 1 -- 172.21.15.1:0/589094043 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09000bb00 con 0x7fe09806c0b0 2024-03-18T20:27:26.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.036+0000 7fe09e2ff700 1 -- 172.21.15.1:0/589094043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0980b5680 con 0x7fe09806c0b0 2024-03-18T20:27:26.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.036+0000 7fe094ff9700 1 -- 172.21.15.1:0/589094043 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe090009780 con 0x7fe09806c0b0 2024-03-18T20:27:26.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.036+0000 7fe09e2ff700 1 -- 172.21.15.1:0/589094043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0981f27c0 con 0x7fe09806c0b0 2024-03-18T20:27:26.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.037+0000 7fe09e2ff700 1 -- 172.21.15.1:0/589094043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0981f3260 con 0x7fe09806c0b0 2024-03-18T20:27:26.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.038+0000 7fe094ff9700 1 -- 172.21.15.1:0/589094043 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fe0900098e0 con 0x7fe09806c0b0 2024-03-18T20:27:26.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.038+0000 7fe094ff9700 1 --2- 172.21.15.1:0/589094043 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe07806aed0 0x7fe07806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.038+0000 7fe094ff9700 1 -- 172.21.15.1:0/589094043 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fe09008d5d0 con 0x7fe09806c0b0 2024-03-18T20:27:26.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.038+0000 7fe097fff700 1 --2- 172.21.15.1:0/589094043 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe07806aed0 0x7fe07806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.038+0000 7fe097fff700 1 --2- 172.21.15.1:0/589094043 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe07806aed0 0x7fe07806d390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe088000f60 tx=0x7fe088007be0 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:26.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.043+0000 7fe094ff9700 1 -- 172.21.15.1:0/589094043 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe090058220 con 0x7fe09806c0b0 2024-03-18T20:27:26.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.243+0000 7fe09e2ff700 1 -- 172.21.15.1:0/589094043 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe09806dda0 con 0x7fe07806aed0 2024-03-18T20:27:26.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.252+0000 7fe094ff9700 1 -- 172.21.15.1:0/589094043 <== mgr.12813 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278956 (secure 0 0 0) 0x7fe0880012b0 con 0x7fe07806aed0 2024-03-18T20:27:26.260 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:26.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.255+0000 7fe09e2ff700 1 -- 172.21.15.1:0/589094043 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe07806aed0 msgr2=0x7fe07806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:26.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.255+0000 7fe09e2ff700 1 --2- 172.21.15.1:0/589094043 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe07806aed0 0x7fe07806d390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe088000f60 tx=0x7fe088007be0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.255+0000 7fe09e2ff700 1 -- 172.21.15.1:0/589094043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09806c0b0 msgr2=0x7fe09814f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:26.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.255+0000 7fe09e2ff700 1 --2- 172.21.15.1:0/589094043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09806c0b0 0x7fe09814f390 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7fe09000d730 tx=0x7fe090004ad0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.255+0000 7fe09e2ff700 1 -- 172.21.15.1:0/589094043 shutdown_connections 2024-03-18T20:27:26.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.255+0000 7fe09e2ff700 1 --2- 172.21.15.1:0/589094043 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe07806aed0 0x7fe07806d390 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.255+0000 7fe09e2ff700 1 --2- 172.21.15.1:0/589094043 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe09806e2c0 0x7fe0981f1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.255+0000 7fe09e2ff700 1 --2- 172.21.15.1:0/589094043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09806c0b0 0x7fe09814f390 unknown :-1 s=CLOSED pgs=3075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.255+0000 7fe09e2ff700 1 --2- 172.21.15.1:0/589094043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe098071600 0x7fe09806bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.255+0000 7fe09e2ff700 1 -- 172.21.15.1:0/589094043 >> 172.21.15.1:0/589094043 conn(0x7fe0980cc990 msgr2=0x7fe0980cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:26.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.255+0000 7fe09e2ff700 1 -- 172.21.15.1:0/589094043 shutdown_connections 2024-03-18T20:27:26.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.255+0000 7fe09e2ff700 1 -- 172.21.15.1:0/589094043 wait complete. 2024-03-18T20:27:26.263 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:27:26.272 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-03-18T20:27:24.313447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":417120,"kb_used_data":27900,"kb_used_omap":12,"kb_used_meta":389107,"kb_avail":754557600,"statfs":{"total":773094113280,"available":772666982400,"internally_reserved":0,"allocated":28569600,"data_stored":15362947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":398446137},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000652"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.622037+0000","last_change":"2024-03-18T20:27:20.622037+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.622037+0000","last_undegraded":"2024-03-18T20:27:20.622037+0000","last_fullsized":"2024-03-18T20:27:20.622037+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:01.698353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621734+0000","last_change":"2024-03-18T20:27:20.621734+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621734+0000","last_undegraded":"2024-03-18T20:27:20.621734+0000","last_fullsized":"2024-03-18T20:27:20.621734+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:02:14.143177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.622565+0000","last_change":"2024-03-18T20:27:20.622565+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.622565+0000","last_undegraded":"2024-03-18T20:27:20.622565+0000","last_fullsized":"2024-03-18T20:27:20.622565+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:42:23.110707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.622373+0000","last_change":"2024-03-18T20:27:20.622373+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.622373+0000","last_undegraded":"2024-03-18T20:27:20.622373+0000","last_fullsized":"2024-03-18T20:27:20.622373+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:45:52.316434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617958+0000","last_change":"2024-03-18T20:27:20.617958+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617958+0000","last_undegraded":"2024-03-18T20:27:20.617958+0000","last_fullsized":"2024-03-18T20:27:20.617958+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:08:18.189309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.626102+0000","last_change":"2024-03-18T20:27:20.626102+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.626102+0000","last_undegraded":"2024-03-18T20:27:20.626102+0000","last_fullsized":"2024-03-18T20:27:20.626102+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:40:40.016554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623987+0000","last_change":"2024-03-18T20:27:20.623987+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623987+0000","last_undegraded":"2024-03-18T20:27:20.623987+0000","last_fullsized":"2024-03-18T20:27:20.623987+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:08:18.189309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621015+0000","last_change":"2024-03-18T20:27:20.621015+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621015+0000","last_undegraded":"2024-03-18T20:27:20.621015+0000","last_fullsized":"2024-03-18T20:27:20.621015+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:10.342403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623746+0000","last_change":"2024-03-18T20:27:20.623746+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623746+0000","last_undegraded":"2024-03-18T20:27:20.623746+0000","last_fullsized":"2024-03-18T20:27:20.623746+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:28:52.874751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623648+0000","last_change":"2024-03-18T20:27:20.623648+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623648+0000","last_undegraded":"2024-03-18T20:27:20.623648+0000","last_fullsized":"2024-03-18T20:27:20.623648+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:21:19.006472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.624317+0000","last_change":"2024-03-18T20:27:20.624317+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.624317+0000","last_undegraded":"2024-03-18T20:27:20.624317+0000","last_fullsized":"2024-03-18T20:27:20.624317+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:29.812382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621040+0000","last_change":"2024-03-18T20:27:20.621040+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621040+0000","last_undegraded":"2024-03-18T20:27:20.621040+0000","last_fullsized":"2024-03-18T20:27:20.621040+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:01:31.379909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.625530+0000","last_change":"2024-03-18T20:27:20.625530+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.625530+0000","last_undegraded":"2024-03-18T20:27:20.625530+0000","last_fullsized":"2024-03-18T20:27:20.625530+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:42.339058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617987+0000","last_change":"2024-03-18T20:27:20.617987+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617987+0000","last_undegraded":"2024-03-18T20:27:20.617987+0000","last_fullsized":"2024-03-18T20:27:20.617987+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:42.339058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617911+0000","last_change":"2024-03-18T20:27:20.617911+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617911+0000","last_undegraded":"2024-03-18T20:27:20.617911+0000","last_fullsized":"2024-03-18T20:27:20.617911+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:43.006246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621574+0000","last_change":"2024-03-18T20:27:20.621574+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621574+0000","last_undegraded":"2024-03-18T20:27:20.621574+0000","last_fullsized":"2024-03-18T20:27:20.621574+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:31:56.485139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621698+0000","last_change":"2024-03-18T20:27:20.621698+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621698+0000","last_undegraded":"2024-03-18T20:27:20.621698+0000","last_fullsized":"2024-03-18T20:27:20.621698+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:17:27.485874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.626140+0000","last_change":"2024-03-18T20:27:20.626140+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.626140+0000","last_undegraded":"2024-03-18T20:27:20.626140+0000","last_fullsized":"2024-03-18T20:27:20.626140+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:20:18.042657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621611+0000","last_change":"2024-03-18T20:27:20.621611+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621611+0000","last_undegraded":"2024-03-18T20:27:20.621611+0000","last_fullsized":"2024-03-18T20:27:20.621611+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:58:00.382684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623890+0000","last_change":"2024-03-18T20:27:20.623890+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623890+0000","last_undegraded":"2024-03-18T20:27:20.623890+0000","last_fullsized":"2024-03-18T20:27:20.623890+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:01.555443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.624331+0000","last_change":"2024-03-18T20:27:20.624331+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.624331+0000","last_undegraded":"2024-03-18T20:27:20.624331+0000","last_fullsized":"2024-03-18T20:27:20.624331+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:52.857794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.622224+0000","last_change":"2024-03-18T20:27:20.622224+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.622224+0000","last_undegraded":"2024-03-18T20:27:20.622224+0000","last_fullsized":"2024-03-18T20:27:20.622224+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:13:18.842477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621046+0000","last_change":"2024-03-18T20:27:20.621046+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621046+0000","last_undegraded":"2024-03-18T20:27:20.621046+0000","last_fullsized":"2024-03-18T20:27:20.621046+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:11:16.795077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.625820+0000","last_change":"2024-03-18T20:27:20.625820+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.625820+0000","last_undegraded":"2024-03-18T20:27:20.625820+0000","last_fullsized":"2024-03-18T20:27:20.625820+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:41:34.677506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.624685+0000","last_change":"2024-03-18T20:27:20.624685+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.624685+0000","last_undegraded":"2024-03-18T20:27:20.624685+0000","last_fullsized":"2024-03-18T20:27:20.624685+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:45.164373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617958+0000","last_change":"2024-03-18T20:27:20.617958+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617958+0000","last_undegraded":"2024-03-18T20:27:20.617958+0000","last_fullsized":"2024-03-18T20:27:20.617958+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:52.857794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621100+0000","last_change":"2024-03-18T20:27:20.621100+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621100+0000","last_undegraded":"2024-03-18T20:27:20.621100+0000","last_fullsized":"2024-03-18T20:27:20.621100+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:23:55.315193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617962+0000","last_change":"2024-03-18T20:27:20.617962+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617962+0000","last_undegraded":"2024-03-18T20:27:20.617962+0000","last_fullsized":"2024-03-18T20:27:20.617962+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:01.555443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623634+0000","last_change":"2024-03-18T20:27:20.623634+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623634+0000","last_undegraded":"2024-03-18T20:27:20.623634+0000","last_fullsized":"2024-03-18T20:27:20.623634+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:23:24.136007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621775+0000","last_change":"2024-03-18T20:27:20.621775+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621775+0000","last_undegraded":"2024-03-18T20:27:20.621775+0000","last_fullsized":"2024-03-18T20:27:20.621775+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:54:45.115313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623503+0000","last_change":"2024-03-18T20:27:20.623503+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623503+0000","last_undegraded":"2024-03-18T20:27:20.623503+0000","last_fullsized":"2024-03-18T20:27:20.623503+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:28.989185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.625365+0000","last_change":"2024-03-18T20:27:20.625365+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.625365+0000","last_undegraded":"2024-03-18T20:27:20.625365+0000","last_fullsized":"2024-03-18T20:27:20.625365+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:43.006246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"259'381","reported_seq":1352,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-18T20:27:18.438231+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:27:18.438231+0000","last_peered":"2024-03-18T20:27:18.438231+0000","last_clean":"2024-03-18T20:27:18.438231+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:27:18.438231+0000","last_undegraded":"2024-03-18T20:27:18.438231+0000","last_fullsized":"2024-03-18T20:27:18.438231+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576001,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53088,"kb_used_data":3924,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318752,"statfs":{"total":96636764160,"available":96582402048,"internally_reserved":0,"allocated":4018176,"data_stored":2363924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54900000000000004,"15min":0.58999999999999997},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.72599999999999998,"5min":0.80900000000000005,"15min":1.1339999999999999},"last":0.439},{"interface":"front","average":{"1min":0.498,"5min":0.56599999999999995,"15min":0.60199999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":0.73599999999999999,"5min":0.84499999999999997,"15min":1.093},"last":0.56699999999999995}]},{"osd":1,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.55500000000000005,"15min":0.60199999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.748,"5min":0.83099999999999996,"15min":1.4299999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.57099999999999995,"15min":0.60799999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73599999999999999,"5min":0.82399999999999995,"15min":1.2629999999999999},"last":0.61599999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.56100000000000005,"15min":0.60099999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67600000000000005,"5min":0.85499999999999998,"15min":1.216},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57799999999999996,"15min":0.61599999999999999},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75,"5min":1.1279999999999999,"15min":1.2490000000000001},"last":0.59899999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":0.59799999999999998},"min":{"1min":0.38700000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71999999999999997,"5min":0.84699999999999998,"15min":1.4450000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.60699999999999998,"15min":0.64100000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73099999999999998,"5min":0.94599999999999995,"15min":1.5029999999999999},"last":0.63300000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47099999999999997,"15min":0.47799999999999998},"min":{"1min":0.35999999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.68899999999999995,"5min":0.79100000000000004,"15min":0.86099999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48899999999999999,"15min":0.499},"min":{"1min":0.34100000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.66200000000000003,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.54500000000000004}]},{"osd":5,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.51300000000000001,"15min":0.51900000000000002},"min":{"1min":0.373,"5min":0.314,"15min":0.314},"max":{"1min":0.70399999999999996,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.54600000000000004},"min":{"1min":0.38300000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73099999999999998,"5min":0.877,"15min":1.042},"last":0.68799999999999994}]},{"osd":6,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55800000000000005,"15min":0.57699999999999996},"min":{"1min":0.435,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77300000000000002,"5min":0.93500000000000005,"15min":1.129},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.54700000000000004,"15min":0.56899999999999995},"min":{"1min":0.47399999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76200000000000001,"5min":0.91900000000000004,"15min":1.0609999999999999},"last":0.46899999999999997}]}]},{"osd":6,"up_from":13,"seq":55834576001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60700,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311140,"statfs":{"total":96636764160,"available":96574607360,"internally_reserved":0,"allocated":3424256,"data_stored":1773556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53300000000000003,"15min":0.58499999999999996},"min":{"1min":0.35399999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68799999999999994,"5min":0.79800000000000004,"15min":1.1519999999999999},"last":0.505},{"interface":"front","average":{"1min":0.503,"5min":0.54600000000000004,"15min":0.59499999999999997},"min":{"1min":0.33200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.73499999999999999,"5min":1.119,"15min":1.119},"last":0.67100000000000004}]},{"osd":1,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53800000000000003,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73199999999999998,"5min":0.79100000000000004,"15min":1.244},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":0.59499999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":0.70099999999999996,"5min":0.89200000000000002,"15min":1.286},"last":0.54600000000000004}]},{"osd":2,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56399999999999995,"15min":0.61699999999999999},"min":{"1min":0.38700000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.69599999999999995,"5min":0.79600000000000004,"15min":1.276},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.55400000000000005,"15min":0.59899999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.65200000000000002,"5min":1.089,"15min":1.2669999999999999},"last":0.53700000000000003}]},{"osd":3,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55500000000000005,"15min":0.61099999999999999},"min":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71599999999999997,"5min":1.163,"15min":1.163},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57099999999999995,"15min":0.61099999999999999},"min":{"1min":0.47499999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78100000000000003,"5min":0.82399999999999995,"15min":1.337},"last":0.55500000000000005}]},{"osd":4,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.35299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80200000000000005,"5min":0.85799999999999998,"15min":0.88300000000000001},"last":0.378},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.50600000000000001,"15min":0.51800000000000002},"min":{"1min":0.32500000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83399999999999996,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.40200000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.50900000000000001,"15min":0.51200000000000001},"min":{"1min":0.42599999999999999,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":0.65000000000000002,"5min":1.008,"15min":1.008},"last":0.441},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55900000000000005,"15min":0.55300000000000005},"min":{"1min":0.41799999999999998,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.755,"5min":0.92600000000000005,"15min":0.94199999999999995},"last":0.43099999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56699999999999995,"15min":0.56299999999999994},"min":{"1min":0.40999999999999998,"5min":0.38,"15min":0.34499999999999997},"max":{"1min":0.91200000000000003,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.34599999999999997},"max":{"1min":0.86699999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.52000000000000002}]}]},{"osd":1,"up_from":13,"seq":55834576001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52512,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319328,"statfs":{"total":96636764160,"available":96582991872,"internally_reserved":0,"allocated":3424256,"data_stored":1773556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.38700000000000001,"15min":0.39300000000000002},"min":{"1min":0.255,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.46400000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.317},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39500000000000002,"15min":0.41999999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.623,"5min":0.623,"15min":0.69699999999999995},"last":0.377}]},{"osd":2,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.40899999999999997,"15min":0.42899999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53100000000000003,"5min":0.61199999999999999,"15min":0.68100000000000005},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.45200000000000001,"15min":0.47099999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.69799999999999995,"5min":0.70099999999999996,"15min":0.80500000000000005},"last":0.33700000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.46500000000000002,"15min":0.48399999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.76100000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.41899999999999998,"15min":0.433},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.27000000000000002},"max":{"1min":0.54700000000000004,"5min":0.68200000000000005,"15min":0.71699999999999997},"last":0.34999999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.64000000000000001,"15min":0.66500000000000004},"min":{"1min":0.43099999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76800000000000002,"5min":1.0680000000000001,"15min":1.2909999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.64600000000000002,"15min":0.67100000000000004},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.88500000000000001,"5min":1.1160000000000001,"15min":1.746},"last":0.61299999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.63400000000000001,"15min":0.66100000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79300000000000004,"5min":1.0389999999999999,"15min":1.6910000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.67800000000000005,"15min":0.70499999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.85199999999999998,"5min":1.109,"15min":1.2549999999999999},"last":0.58399999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.65300000000000002,"15min":0.67700000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.70799999999999996,"5min":1.0289999999999999,"15min":1.637},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.622,"5min":0.67800000000000005,"15min":0.70199999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82099999999999995,"5min":1.014,"15min":1.722},"last":0.63100000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.66000000000000003,"15min":0.69899999999999995},"min":{"1min":0.46100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91100000000000003,"5min":0.99199999999999999,"15min":1.2729999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.67000000000000004,"15min":0.69699999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.80700000000000005,"5min":1.0329999999999999,"15min":1.3380000000000001},"last":0.66600000000000004}]}]},{"osd":0,"up_from":13,"seq":55834576002,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48992,"kb_used_data":3924,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322848,"statfs":{"total":96636764160,"available":96586596352,"internally_reserved":0,"allocated":4018176,"data_stored":2363924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.36399999999999999,"15min":0.38600000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.46100000000000002,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.39600000000000002,"15min":0.41999999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.58799999999999997,"5min":0.60099999999999998,"15min":1.1850000000000001},"last":0.255}]},{"osd":2,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.41299999999999998,"15min":0.433},"min":{"1min":0.28699999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.47199999999999998,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.41799999999999998,"15min":0.44600000000000001},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.54500000000000004,"5min":0.64800000000000002,"15min":0.70999999999999996},"last":0.39400000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.39600000000000002,"15min":0.42399999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.433,"5min":0.58499999999999996,"15min":0.628},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.45400000000000001,"15min":0.48699999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.52600000000000002,"5min":0.73399999999999999,"15min":0.91500000000000004},"last":0.379}]},{"osd":4,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.61299999999999999,"15min":0.64400000000000002},"min":{"1min":0.41299999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.752,"5min":0.91500000000000004,"15min":1.0029999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.64800000000000002,"15min":0.65900000000000003},"min":{"1min":0.51600000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.92300000000000004,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.72099999999999997}]},{"osd":5,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.64400000000000002,"15min":0.66600000000000004},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83299999999999996,"5min":0.92300000000000004,"15min":1.0129999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65300000000000002,"15min":0.67500000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.73199999999999998,"5min":0.96199999999999997,"15min":1.518},"last":0.70499999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.64600000000000002,"15min":0.65700000000000003},"min":{"1min":0.45800000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.73599999999999999,"5min":1.0669999999999999,"15min":1.1279999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.68200000000000005,"15min":0.70299999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.871,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.746}]},{"osd":7,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.67200000000000004,"15min":0.69899999999999995},"min":{"1min":0.45300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77500000000000002,"5min":1.008,"15min":1.464},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.67300000000000004,"15min":0.69799999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88900000000000001,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.76300000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52508,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319332,"statfs":{"total":96636764160,"available":96582995968,"internally_reserved":0,"allocated":3424256,"data_stored":1773556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.34699999999999998,"15min":0.371},"min":{"1min":0.20799999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.53800000000000003,"5min":0.54800000000000004,"15min":0.57499999999999996},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.44700000000000001,"15min":0.47799999999999998},"min":{"1min":0.26100000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60499999999999998,"5min":0.71499999999999997,"15min":0.753},"last":0.48999999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.44400000000000001,"15min":0.47399999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57399999999999995,"5min":0.64300000000000002,"15min":0.749},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.38900000000000001,"15min":0.40899999999999997},"min":{"1min":0.246,"5min":0.224,"15min":0.224},"max":{"1min":0.501,"5min":0.55900000000000005,"15min":0.59999999999999998},"last":0.32700000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.41199999999999998,"15min":0.42899999999999999},"min":{"1min":0.27800000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.56100000000000005,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.47399999999999998,"15min":0.50700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.63200000000000001,"5min":0.68100000000000005,"15min":0.752},"last":0.45400000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63900000000000001,"15min":0.65300000000000002},"min":{"1min":0.41799999999999998,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":1.2090000000000001,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.65700000000000003,"15min":0.66200000000000003},"min":{"1min":0.45100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.268,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.61899999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.64900000000000002,"15min":0.67500000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79900000000000004,"5min":0.97299999999999998,"15min":1.028},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.63900000000000001,"15min":0.67000000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.77100000000000002,"5min":0.88300000000000001,"15min":1.0629999999999999},"last":0.80800000000000005}]},{"osd":6,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.65500000000000003,"15min":0.68100000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.81899999999999995,"5min":1.014,"15min":1.2050000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.72699999999999998},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.3220000000000001,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.77000000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.65500000000000003,"15min":0.68899999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.42199999999999999},"max":{"1min":0.73699999999999999,"5min":0.99399999999999999,"15min":1.0269999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68600000000000005,"15min":0.70099999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.2330000000000001,"5min":1.371,"15min":1.371},"last":0.79100000000000004}]}]},{"osd":3,"up_from":13,"seq":55834576002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52508,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319332,"statfs":{"total":96636764160,"available":96582995968,"internally_reserved":0,"allocated":3424256,"data_stored":1773556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.36099999999999999,"15min":0.39000000000000001},"min":{"1min":0.247,"5min":0.223,"15min":0.223},"max":{"1min":0.46400000000000002,"5min":0.66500000000000004,"15min":0.98099999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.373,"15min":0.40400000000000003},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.45600000000000002,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.42399999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.35999999999999999,"15min":0.39700000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.44500000000000001,"5min":0.63200000000000001,"15min":0.64000000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.42199999999999999,"15min":0.45600000000000002},"min":{"1min":0.33400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.501,"5min":0.65500000000000003,"15min":0.73099999999999998},"last":0.44500000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.41499999999999998,"15min":0.45200000000000001},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.54900000000000004,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.437,"15min":0.48099999999999998},"min":{"1min":0.32600000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.52300000000000002,"5min":0.64400000000000002,"15min":0.71199999999999997},"last":0.32600000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59499999999999997,"15min":0.64000000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.526,"5min":1.526,"15min":1.526},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61399999999999999,"15min":0.66100000000000003},"min":{"1min":0.39200000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.65800000000000003,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.65800000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.65100000000000002,"15min":0.68700000000000006},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.66500000000000004,"15min":0.71499999999999997},"min":{"1min":0.45300000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78000000000000003,"5min":1.1120000000000001,"15min":1.171},"last":0.65000000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65800000000000003,"15min":0.69699999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.62},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.71999999999999997},"min":{"1min":0.48699999999999999,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.58999999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.64900000000000002,"15min":0.70299999999999996},"min":{"1min":0.43099999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79900000000000004,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64300000000000002,"15min":0.68899999999999995},"min":{"1min":0.51500000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.57299999999999995}]}]},{"osd":4,"up_from":13,"seq":55834576002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48400,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323440,"statfs":{"total":96636764160,"available":96587202560,"internally_reserved":0,"allocated":3411968,"data_stored":1767319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57799999999999996,"15min":0.58899999999999997},"min":{"1min":0.39600000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56799999999999995,"15min":0.59499999999999997},"min":{"1min":0.40100000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.84199999999999997},"last":0.621}]},{"osd":1,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.79300000000000004,"5min":0.81899999999999995,"15min":0.88300000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56399999999999995,"15min":0.57999999999999996},"min":{"1min":0.34899999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.96499999999999997},"last":0.58999999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59799999999999998,"15min":0.61399999999999999},"min":{"1min":0.42899999999999999,"5min":0.41299999999999998,"15min":0.38500000000000001},"max":{"1min":0.70799999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.54500000000000004}]},{"osd":3,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.59299999999999997,"15min":0.60199999999999998},"min":{"1min":0.433,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78100000000000003,"5min":1.026,"15min":1.026},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.41399999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.72999999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.48399999999999999,"15min":0.48199999999999998},"min":{"1min":0.318,"5min":0.309,"15min":0.309},"max":{"1min":0.76700000000000002,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.747},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.496,"15min":0.505},"min":{"1min":0.38300000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.81499999999999995,"5min":0.85499999999999998,"15min":0.90600000000000003},"last":0.63800000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.502,"15min":0.50600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.754,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.82799999999999996,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.71399999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.40600000000000003,"5min":0.38800000000000001,"15min":0.35699999999999998},"max":{"1min":0.80300000000000005,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.38500000000000001,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.84099999999999997,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.65200000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48412,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323428,"statfs":{"total":96636764160,"available":96587190272,"internally_reserved":0,"allocated":3424256,"data_stored":1773556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.55400000000000005,"15min":0.59099999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.31900000000000001},"max":{"1min":0.76300000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.621},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55600000000000005,"15min":0.59499999999999997},"min":{"1min":0.42099999999999999,"5min":0.35599999999999998,"15min":0.33100000000000002},"max":{"1min":0.81100000000000005,"5min":0.82599999999999996,"15min":1.3100000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.56100000000000005,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71499999999999997,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.55400000000000005,"15min":0.59199999999999997},"min":{"1min":0.372,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.69399999999999995,"5min":0.94299999999999995,"15min":1.018},"last":0.58499999999999996}]},{"osd":2,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.59399999999999997},"min":{"1min":0.40300000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.75600000000000001,"5min":0.85299999999999998,"15min":0.99299999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.55400000000000005,"15min":0.60099999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77000000000000002,"5min":1.599,"15min":1.599},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57499999999999996,"15min":0.60399999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72899999999999998,"5min":0.98699999999999999,"15min":1.2430000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59299999999999997,"15min":0.63700000000000001},"min":{"1min":0.42099999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":0.84399999999999997,"15min":1.161},"last":0.63300000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48999999999999999,"15min":0.48299999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.625,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.35699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.40699999999999997}]},{"osd":6,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.503,"15min":0.50900000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.32800000000000001},"max":{"1min":0.64800000000000002,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54300000000000004,"15min":0.54200000000000004},"min":{"1min":0.35199999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.64600000000000002,"5min":1.643,"15min":1.643},"last":0.501}]},{"osd":7,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.40200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.71299999999999997,"5min":1.028,"15min":1.028},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57099999999999995,"15min":0.57799999999999996},"min":{"1min":0.42599999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74299999999999999,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:27:26.277 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:27:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.433+0000 7fc8e82bd700 1 Processor -- start 2024-03-18T20:27:26.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.433+0000 7fc8e82bd700 1 -- start start 2024-03-18T20:27:26.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.434+0000 7fc8e82bd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc8e015d6b0 0x7fc8e00b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.434+0000 7fc8e82bd700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8e00acfb0 con 0x7fc8e006a750 2024-03-18T20:27:26.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.434+0000 7fc8e82bd700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8e00b50a0 con 0x7fc8e00ad790 2024-03-18T20:27:26.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.434+0000 7fc8e82bd700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8e00af7a0 con 0x7fc8e015d6b0 2024-03-18T20:27:26.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.434+0000 7fc8e6059700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc8e006a750 0x7fc8e006ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37452/0 (socket says 172.21.15.1:37452) 2024-03-18T20:27:26.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.434+0000 7fc8e6059700 1 -- 172.21.15.1:0/3089408231 learned_addr learned my addr 172.21.15.1:0/3089408231 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:26.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e5858700 1 --2- 172.21.15.1:0/3089408231 >> v2:172.21.15.165:3300/0 conn(0x7fc8e015d6b0 0x7fc8e00b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e5057700 1 -- 172.21.15.1:0/3089408231 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956121331 0 0) 0x7fc8e00acfb0 con 0x7fc8e006a750 2024-03-18T20:27:26.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e5057700 1 -- 172.21.15.1:0/3089408231 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8cc0035b0 con 0x7fc8e006a750 2024-03-18T20:27:26.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e5057700 1 -- 172.21.15.1:0/3089408231 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 734909186 0 0) 0x7fc8cc0035b0 con 0x7fc8e006a750 2024-03-18T20:27:26.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e5057700 1 -- 172.21.15.1:0/3089408231 >> v2:172.21.15.165:3300/0 conn(0x7fc8e015d6b0 msgr2=0x7fc8e00b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:26.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e5057700 1 --2- 172.21.15.1:0/3089408231 >> v2:172.21.15.165:3300/0 conn(0x7fc8e015d6b0 0x7fc8e00b2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e5057700 1 -- 172.21.15.1:0/3089408231 >> v1:172.21.15.165:6789/0 conn(0x7fc8e00ad790 legacy=0x7fc8e00b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:26.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e5057700 1 -- 172.21.15.1:0/3089408231 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e00af1e0 con 0x7fc8e006a750 2024-03-18T20:27:26.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e5057700 1 -- 172.21.15.1:0/3089408231 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fc8d4001670 con 0x7fc8e006a750 2024-03-18T20:27:26.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e5057700 1 -- 172.21.15.1:0/3089408231 >> v1:172.21.15.1:6789/0 conn(0x7fc8e006a750 legacy=0x7fc8e006ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:26.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e5057700 1 --2- 172.21.15.1:0/3089408231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8cc003a00 0x7fc8cc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e5057700 1 --2- 172.21.15.1:0/3089408231 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc8cc004850 0x7fc8cc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e5057700 1 --2- 172.21.15.1:0/3089408231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8cc005210 0x7fc8cc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e5057700 1 -- 172.21.15.1:0/3089408231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e00af7a0 con 0x7fc8cc003a00 2024-03-18T20:27:26.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e5057700 1 -- 172.21.15.1:0/3089408231 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc00c0c0 con 0x7fc8cc005210 2024-03-18T20:27:26.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e5057700 1 -- 172.21.15.1:0/3089408231 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc00c280 con 0x7fc8cc004850 2024-03-18T20:27:26.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e6059700 1 --2- 172.21.15.1:0/3089408231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8cc003a00 0x7fc8cc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e5858700 1 --2- 172.21.15.1:0/3089408231 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc8cc004850 0x7fc8cc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e685a700 1 --2- 172.21.15.1:0/3089408231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8cc005210 0x7fc8cc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e6059700 1 -- 172.21.15.1:0/3089408231 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc8cc004850 msgr2=0x7fc8cc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:26.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e6059700 1 --2- 172.21.15.1:0/3089408231 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc8cc004850 0x7fc8cc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e6059700 1 -- 172.21.15.1:0/3089408231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8cc005210 msgr2=0x7fc8cc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:26.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e6059700 1 --2- 172.21.15.1:0/3089408231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8cc005210 0x7fc8cc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e6059700 1 -- 172.21.15.1:0/3089408231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc0035b0 con 0x7fc8cc003a00 2024-03-18T20:27:26.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.435+0000 7fc8e685a700 1 -- 172.21.15.1:0/3089408231 reap_dead start 2024-03-18T20:27:26.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.436+0000 7fc8e6059700 1 --2- 172.21.15.1:0/3089408231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8cc003a00 0x7fc8cc003e20 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400c240 tx=0x7fc8cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=463dced36eb55249 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:26.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.436+0000 7fc8e5057700 1 -- 172.21.15.1:0/3089408231 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d4007c10 con 0x7fc8cc003a00 2024-03-18T20:27:26.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.436+0000 7fc8e5057700 1 -- 172.21.15.1:0/3089408231 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d400ce00 con 0x7fc8cc003a00 2024-03-18T20:27:26.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.436+0000 7fc8e5057700 1 -- 172.21.15.1:0/3089408231 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d4005ba0 con 0x7fc8cc003a00 2024-03-18T20:27:26.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.436+0000 7fc8e82bd700 1 -- 172.21.15.1:0/3089408231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8cc003a00 msgr2=0x7fc8cc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:26.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.436+0000 7fc8e82bd700 1 --2- 172.21.15.1:0/3089408231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8cc003a00 0x7fc8cc003e20 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400c240 tx=0x7fc8cc005880 comp rx=0 tx=0).stop 2024-03-18T20:27:26.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.436+0000 7fc8e82bd700 1 -- 172.21.15.1:0/3089408231 shutdown_connections 2024-03-18T20:27:26.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.436+0000 7fc8e82bd700 1 --2- 172.21.15.1:0/3089408231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8cc003a00 0x7fc8cc003e20 unknown :-1 s=CLOSED pgs=3077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.436+0000 7fc8e82bd700 1 -- 172.21.15.1:0/3089408231 >> 172.21.15.1:0/3089408231 conn(0x7fc8e006b860 msgr2=0x7fc8e006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:26.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.436+0000 7fc8e82bd700 1 -- 172.21.15.1:0/3089408231 shutdown_connections 2024-03-18T20:27:26.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.436+0000 7fc8e82bd700 1 -- 172.21.15.1:0/3089408231 wait complete. 2024-03-18T20:27:26.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.438+0000 7fc8e82bd700 1 Processor -- start 2024-03-18T20:27:26.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.438+0000 7fc8e82bd700 1 -- start start 2024-03-18T20:27:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.438+0000 7fc8e82bd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8cc003a00 0x7fc8e01e9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.438+0000 7fc8e82bd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc8e015e4a0 0x7fc8e01581c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.438+0000 7fc8e82bd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8e0158700 0x7fc8e015ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.438+0000 7fc8e82bd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e0158b70 con 0x7fc8e0158700 2024-03-18T20:27:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.438+0000 7fc8e82bd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e015e9a0 con 0x7fc8cc003a00 2024-03-18T20:27:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.438+0000 7fc8e82bd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e015ec80 con 0x7fc8e015e4a0 2024-03-18T20:27:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.438+0000 7fc8e685a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8e0158700 0x7fc8e015ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.438+0000 7fc8e685a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8e0158700 0x7fc8e015ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36512/0 (socket says 172.21.15.1:36512) 2024-03-18T20:27:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.439+0000 7fc8e685a700 1 -- 172.21.15.1:0/893644510 learned_addr learned my addr 172.21.15.1:0/893644510 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.439+0000 7fc8e5858700 1 --2- 172.21.15.1:0/893644510 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc8e015e4a0 0x7fc8e01581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.439+0000 7fc8e685a700 1 -- 172.21.15.1:0/893644510 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc8e015e4a0 msgr2=0x7fc8e01581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.439+0000 7fc8e6059700 1 --2- 172.21.15.1:0/893644510 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8cc003a00 0x7fc8e01e9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.439+0000 7fc8e685a700 1 --2- 172.21.15.1:0/893644510 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc8e015e4a0 0x7fc8e01581c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.439+0000 7fc8e685a700 1 -- 172.21.15.1:0/893644510 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8cc003a00 msgr2=0x7fc8e01e9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.439+0000 7fc8e685a700 1 --2- 172.21.15.1:0/893644510 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8cc003a00 0x7fc8e01e9210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.439+0000 7fc8e685a700 1 -- 172.21.15.1:0/893644510 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e015ef60 con 0x7fc8e0158700 2024-03-18T20:27:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.439+0000 7fc8e685a700 1 --2- 172.21.15.1:0/893644510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8e0158700 0x7fc8e015ae80 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0070190 tx=0x7fc8d800ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.439+0000 7fc8d2ffd700 1 -- 172.21.15.1:0/893644510 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8003c10 con 0x7fc8e0158700 2024-03-18T20:27:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.439+0000 7fc8d2ffd700 1 -- 172.21.15.1:0/893644510 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d8003db0 con 0x7fc8e0158700 2024-03-18T20:27:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.439+0000 7fc8e82bd700 1 -- 172.21.15.1:0/893644510 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8e015f240 con 0x7fc8e0158700 2024-03-18T20:27:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.439+0000 7fc8d2ffd700 1 -- 172.21.15.1:0/893644510 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8005310 con 0x7fc8e0158700 2024-03-18T20:27:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.439+0000 7fc8e82bd700 1 -- 172.21.15.1:0/893644510 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8e01e9750 con 0x7fc8e0158700 2024-03-18T20:27:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.440+0000 7fc8e82bd700 1 -- 172.21.15.1:0/893644510 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8e01e9940 con 0x7fc8e0158700 2024-03-18T20:27:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.441+0000 7fc8d2ffd700 1 -- 172.21.15.1:0/893644510 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fc8d8005470 con 0x7fc8e0158700 2024-03-18T20:27:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.442+0000 7fc8d2ffd700 1 --2- 172.21.15.1:0/893644510 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc8c406aed0 0x7fc8c406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.442+0000 7fc8d2ffd700 1 -- 172.21.15.1:0/893644510 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fc8d808a360 con 0x7fc8e0158700 2024-03-18T20:27:26.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.442+0000 7fc8e6059700 1 --2- 172.21.15.1:0/893644510 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc8c406aed0 0x7fc8c406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.442+0000 7fc8e6059700 1 --2- 172.21.15.1:0/893644510 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc8c406aed0 0x7fc8c406d390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4015040 tx=0x7fc8d4007000 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:26.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.445+0000 7fc8d2ffd700 1 -- 172.21.15.1:0/893644510 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc8d80551d0 con 0x7fc8e0158700 2024-03-18T20:27:26.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.645+0000 7fc8e82bd700 1 -- 172.21.15.1:0/893644510 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc8e00ad440 con 0x7fc8c406aed0 2024-03-18T20:27:26.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.652+0000 7fc8d2ffd700 1 -- 172.21.15.1:0/893644510 <== mgr.12813 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+280373 (secure 0 0 0) 0x7fc8d4007b30 con 0x7fc8c406aed0 2024-03-18T20:27:26.661 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:26.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.655+0000 7fc8e82bd700 1 -- 172.21.15.1:0/893644510 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc8c406aed0 msgr2=0x7fc8c406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:26.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.655+0000 7fc8e82bd700 1 --2- 172.21.15.1:0/893644510 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc8c406aed0 0x7fc8c406d390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4015040 tx=0x7fc8d4007000 comp rx=0 tx=0).stop 2024-03-18T20:27:26.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.656+0000 7fc8e82bd700 1 -- 172.21.15.1:0/893644510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8e0158700 msgr2=0x7fc8e015ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:26.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.656+0000 7fc8e82bd700 1 --2- 172.21.15.1:0/893644510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8e0158700 0x7fc8e015ae80 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0070190 tx=0x7fc8d800ee00 comp rx=0 tx=0).stop 2024-03-18T20:27:26.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.656+0000 7fc8e82bd700 1 -- 172.21.15.1:0/893644510 shutdown_connections 2024-03-18T20:27:26.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.656+0000 7fc8e82bd700 1 --2- 172.21.15.1:0/893644510 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc8c406aed0 0x7fc8c406d390 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.656+0000 7fc8e82bd700 1 --2- 172.21.15.1:0/893644510 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8e0158700 0x7fc8e015ae80 unknown :-1 s=CLOSED pgs=3078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.656+0000 7fc8e82bd700 1 --2- 172.21.15.1:0/893644510 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8cc003a00 0x7fc8e01e9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.656+0000 7fc8e82bd700 1 --2- 172.21.15.1:0/893644510 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc8e015e4a0 0x7fc8e01581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.656+0000 7fc8e82bd700 1 -- 172.21.15.1:0/893644510 >> 172.21.15.1:0/893644510 conn(0x7fc8e006b860 msgr2=0x7fc8e006dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:26.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.656+0000 7fc8e82bd700 1 -- 172.21.15.1:0/893644510 shutdown_connections 2024-03-18T20:27:26.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.656+0000 7fc8e82bd700 1 -- 172.21.15.1:0/893644510 wait complete. 2024-03-18T20:27:26.664 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:27:26.673 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-03-18T20:27:26.313769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":642,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":84,"acting":84,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":24,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":417156,"kb_used_data":27940,"kb_used_omap":12,"kb_used_meta":389107,"kb_avail":754557564,"statfs":{"total":773094113280,"available":772666945536,"internally_reserved":0,"allocated":28610560,"data_stored":15383647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":398446137},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.698740"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218004+0000","last_change":"2024-03-18T20:27:22.055309+0000","last_active":"2024-03-18T20:27:24.218004+0000","last_peered":"2024-03-18T20:27:24.218004+0000","last_clean":"2024-03-18T20:27:24.218004+0000","last_became_active":"2024-03-18T20:27:22.055064+0000","last_became_peered":"2024-03-18T20:27:22.055064+0000","last_unstale":"2024-03-18T20:27:24.218004+0000","last_undegraded":"2024-03-18T20:27:24.218004+0000","last_fullsized":"2024-03-18T20:27:24.218004+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:19.253084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.217979+0000","last_change":"2024-03-18T20:27:22.062816+0000","last_active":"2024-03-18T20:27:24.217979+0000","last_peered":"2024-03-18T20:27:24.217979+0000","last_clean":"2024-03-18T20:27:24.217979+0000","last_became_active":"2024-03-18T20:27:22.062600+0000","last_became_peered":"2024-03-18T20:27:22.062600+0000","last_unstale":"2024-03-18T20:27:24.217979+0000","last_undegraded":"2024-03-18T20:27:24.217979+0000","last_fullsized":"2024-03-18T20:27:24.217979+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:15:52.725667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218003+0000","last_change":"2024-03-18T20:27:22.062897+0000","last_active":"2024-03-18T20:27:24.218003+0000","last_peered":"2024-03-18T20:27:24.218003+0000","last_clean":"2024-03-18T20:27:24.218003+0000","last_became_active":"2024-03-18T20:27:22.062639+0000","last_became_peered":"2024-03-18T20:27:22.062639+0000","last_unstale":"2024-03-18T20:27:24.218003+0000","last_undegraded":"2024-03-18T20:27:24.218003+0000","last_fullsized":"2024-03-18T20:27:24.218003+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:45.973738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.217918+0000","last_change":"2024-03-18T20:27:22.073284+0000","last_active":"2024-03-18T20:27:24.217918+0000","last_peered":"2024-03-18T20:27:24.217918+0000","last_clean":"2024-03-18T20:27:24.217918+0000","last_became_active":"2024-03-18T20:27:22.072955+0000","last_became_peered":"2024-03-18T20:27:22.072955+0000","last_unstale":"2024-03-18T20:27:24.217918+0000","last_undegraded":"2024-03-18T20:27:24.217918+0000","last_fullsized":"2024-03-18T20:27:24.217918+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:10.708304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.622037+0000","last_change":"2024-03-18T20:27:20.622037+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.622037+0000","last_undegraded":"2024-03-18T20:27:20.622037+0000","last_fullsized":"2024-03-18T20:27:20.622037+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:01.698353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621734+0000","last_change":"2024-03-18T20:27:20.621734+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621734+0000","last_undegraded":"2024-03-18T20:27:20.621734+0000","last_fullsized":"2024-03-18T20:27:20.621734+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:02:14.143177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.622565+0000","last_change":"2024-03-18T20:27:20.622565+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.622565+0000","last_undegraded":"2024-03-18T20:27:20.622565+0000","last_fullsized":"2024-03-18T20:27:20.622565+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:42:23.110707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.622373+0000","last_change":"2024-03-18T20:27:20.622373+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.622373+0000","last_undegraded":"2024-03-18T20:27:20.622373+0000","last_fullsized":"2024-03-18T20:27:20.622373+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:45:52.316434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617958+0000","last_change":"2024-03-18T20:27:20.617958+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617958+0000","last_undegraded":"2024-03-18T20:27:20.617958+0000","last_fullsized":"2024-03-18T20:27:20.617958+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:08:18.189309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1a","version":"273'1","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.069050+0000","last_change":"2024-03-18T20:27:21.635634+0000","last_active":"2024-03-18T20:27:22.069050+0000","last_peered":"2024-03-18T20:27:22.069050+0000","last_clean":"2024-03-18T20:27:22.069050+0000","last_became_active":"2024-03-18T20:27:21.635366+0000","last_became_peered":"2024-03-18T20:27:21.635366+0000","last_unstale":"2024-03-18T20:27:22.069050+0000","last_undegraded":"2024-03-18T20:27:22.069050+0000","last_fullsized":"2024-03-18T20:27:22.069050+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:40:40.016554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623987+0000","last_change":"2024-03-18T20:27:20.623987+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623987+0000","last_undegraded":"2024-03-18T20:27:20.623987+0000","last_fullsized":"2024-03-18T20:27:20.623987+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:08:18.189309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621015+0000","last_change":"2024-03-18T20:27:20.621015+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621015+0000","last_undegraded":"2024-03-18T20:27:20.621015+0000","last_fullsized":"2024-03-18T20:27:20.621015+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:10.342403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623746+0000","last_change":"2024-03-18T20:27:20.623746+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623746+0000","last_undegraded":"2024-03-18T20:27:20.623746+0000","last_fullsized":"2024-03-18T20:27:20.623746+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:28:52.874751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623648+0000","last_change":"2024-03-18T20:27:20.623648+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623648+0000","last_undegraded":"2024-03-18T20:27:20.623648+0000","last_fullsized":"2024-03-18T20:27:20.623648+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:21:19.006472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.624317+0000","last_change":"2024-03-18T20:27:20.624317+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.624317+0000","last_undegraded":"2024-03-18T20:27:20.624317+0000","last_fullsized":"2024-03-18T20:27:20.624317+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:29.812382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218074+0000","last_change":"2024-03-18T20:27:22.081779+0000","last_active":"2024-03-18T20:27:24.218074+0000","last_peered":"2024-03-18T20:27:24.218074+0000","last_clean":"2024-03-18T20:27:24.218074+0000","last_became_active":"2024-03-18T20:27:22.081506+0000","last_became_peered":"2024-03-18T20:27:22.081506+0000","last_unstale":"2024-03-18T20:27:24.218074+0000","last_undegraded":"2024-03-18T20:27:24.218074+0000","last_fullsized":"2024-03-18T20:27:24.218074+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:29:27.205683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621040+0000","last_change":"2024-03-18T20:27:20.621040+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621040+0000","last_undegraded":"2024-03-18T20:27:20.621040+0000","last_fullsized":"2024-03-18T20:27:20.621040+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:01:31.379909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.625530+0000","last_change":"2024-03-18T20:27:20.625530+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.625530+0000","last_undegraded":"2024-03-18T20:27:20.625530+0000","last_fullsized":"2024-03-18T20:27:20.625530+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:42.339058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617987+0000","last_change":"2024-03-18T20:27:20.617987+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617987+0000","last_undegraded":"2024-03-18T20:27:20.617987+0000","last_fullsized":"2024-03-18T20:27:20.617987+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:42.339058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617911+0000","last_change":"2024-03-18T20:27:20.617911+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617911+0000","last_undegraded":"2024-03-18T20:27:20.617911+0000","last_fullsized":"2024-03-18T20:27:20.617911+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:43.006246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621574+0000","last_change":"2024-03-18T20:27:20.621574+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621574+0000","last_undegraded":"2024-03-18T20:27:20.621574+0000","last_fullsized":"2024-03-18T20:27:20.621574+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:31:56.485139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.217869+0000","last_change":"2024-03-18T20:27:22.054635+0000","last_active":"2024-03-18T20:27:24.217869+0000","last_peered":"2024-03-18T20:27:24.217869+0000","last_clean":"2024-03-18T20:27:24.217869+0000","last_became_active":"2024-03-18T20:27:22.054372+0000","last_became_peered":"2024-03-18T20:27:22.054372+0000","last_unstale":"2024-03-18T20:27:24.217869+0000","last_undegraded":"2024-03-18T20:27:24.217869+0000","last_fullsized":"2024-03-18T20:27:24.217869+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:40.826497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621698+0000","last_change":"2024-03-18T20:27:20.621698+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621698+0000","last_undegraded":"2024-03-18T20:27:20.621698+0000","last_fullsized":"2024-03-18T20:27:20.621698+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:17:27.485874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.e","version":"273'2","reported_seq":13,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.083274+0000","last_change":"2024-03-18T20:27:21.640607+0000","last_active":"2024-03-18T20:27:22.083274+0000","last_peered":"2024-03-18T20:27:22.083274+0000","last_clean":"2024-03-18T20:27:22.083274+0000","last_became_active":"2024-03-18T20:27:21.640341+0000","last_became_peered":"2024-03-18T20:27:21.640341+0000","last_unstale":"2024-03-18T20:27:22.083274+0000","last_undegraded":"2024-03-18T20:27:22.083274+0000","last_fullsized":"2024-03-18T20:27:22.083274+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:20:18.042657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218120+0000","last_change":"2024-03-18T20:27:22.055634+0000","last_active":"2024-03-18T20:27:24.218120+0000","last_peered":"2024-03-18T20:27:24.218120+0000","last_clean":"2024-03-18T20:27:24.218120+0000","last_became_active":"2024-03-18T20:27:22.055397+0000","last_became_peered":"2024-03-18T20:27:22.055397+0000","last_unstale":"2024-03-18T20:27:24.218120+0000","last_undegraded":"2024-03-18T20:27:24.218120+0000","last_fullsized":"2024-03-18T20:27:24.218120+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:29.388876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621611+0000","last_change":"2024-03-18T20:27:20.621611+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621611+0000","last_undegraded":"2024-03-18T20:27:20.621611+0000","last_fullsized":"2024-03-18T20:27:20.621611+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:58:00.382684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623890+0000","last_change":"2024-03-18T20:27:20.623890+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623890+0000","last_undegraded":"2024-03-18T20:27:20.623890+0000","last_fullsized":"2024-03-18T20:27:20.623890+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:01.555443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.624331+0000","last_change":"2024-03-18T20:27:20.624331+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.624331+0000","last_undegraded":"2024-03-18T20:27:20.624331+0000","last_fullsized":"2024-03-18T20:27:20.624331+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:52.857794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.622224+0000","last_change":"2024-03-18T20:27:20.622224+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.622224+0000","last_undegraded":"2024-03-18T20:27:20.622224+0000","last_fullsized":"2024-03-18T20:27:20.622224+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:13:18.842477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621046+0000","last_change":"2024-03-18T20:27:20.621046+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621046+0000","last_undegraded":"2024-03-18T20:27:20.621046+0000","last_fullsized":"2024-03-18T20:27:20.621046+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:11:16.795077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218061+0000","last_change":"2024-03-18T20:27:22.062557+0000","last_active":"2024-03-18T20:27:24.218061+0000","last_peered":"2024-03-18T20:27:24.218061+0000","last_clean":"2024-03-18T20:27:24.218061+0000","last_became_active":"2024-03-18T20:27:22.062283+0000","last_became_peered":"2024-03-18T20:27:22.062283+0000","last_unstale":"2024-03-18T20:27:24.218061+0000","last_undegraded":"2024-03-18T20:27:24.218061+0000","last_fullsized":"2024-03-18T20:27:24.218061+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:05:24.597193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.625820+0000","last_change":"2024-03-18T20:27:20.625820+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.625820+0000","last_undegraded":"2024-03-18T20:27:20.625820+0000","last_fullsized":"2024-03-18T20:27:20.625820+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:41:34.677506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.624685+0000","last_change":"2024-03-18T20:27:20.624685+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.624685+0000","last_undegraded":"2024-03-18T20:27:20.624685+0000","last_fullsized":"2024-03-18T20:27:20.624685+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:45.164373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617958+0000","last_change":"2024-03-18T20:27:20.617958+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617958+0000","last_undegraded":"2024-03-18T20:27:20.617958+0000","last_fullsized":"2024-03-18T20:27:20.617958+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:52.857794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621100+0000","last_change":"2024-03-18T20:27:20.621100+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621100+0000","last_undegraded":"2024-03-18T20:27:20.621100+0000","last_fullsized":"2024-03-18T20:27:20.621100+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:23:55.315193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617962+0000","last_change":"2024-03-18T20:27:20.617962+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617962+0000","last_undegraded":"2024-03-18T20:27:20.617962+0000","last_fullsized":"2024-03-18T20:27:20.617962+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:01.555443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623634+0000","last_change":"2024-03-18T20:27:20.623634+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623634+0000","last_undegraded":"2024-03-18T20:27:20.623634+0000","last_fullsized":"2024-03-18T20:27:20.623634+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:23:24.136007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621775+0000","last_change":"2024-03-18T20:27:20.621775+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621775+0000","last_undegraded":"2024-03-18T20:27:20.621775+0000","last_fullsized":"2024-03-18T20:27:20.621775+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:54:45.115313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218125+0000","last_change":"2024-03-18T20:27:22.069073+0000","last_active":"2024-03-18T20:27:24.218125+0000","last_peered":"2024-03-18T20:27:24.218125+0000","last_clean":"2024-03-18T20:27:24.218125+0000","last_became_active":"2024-03-18T20:27:22.068744+0000","last_became_peered":"2024-03-18T20:27:22.068744+0000","last_unstale":"2024-03-18T20:27:24.218125+0000","last_undegraded":"2024-03-18T20:27:24.218125+0000","last_fullsized":"2024-03-18T20:27:24.218125+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:44.884087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623503+0000","last_change":"2024-03-18T20:27:20.623503+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623503+0000","last_undegraded":"2024-03-18T20:27:20.623503+0000","last_fullsized":"2024-03-18T20:27:20.623503+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:28.989185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.625365+0000","last_change":"2024-03-18T20:27:20.625365+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.625365+0000","last_undegraded":"2024-03-18T20:27:20.625365+0000","last_fullsized":"2024-03-18T20:27:20.625365+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:43.006246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"259'381","reported_seq":1352,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-18T20:27:18.438231+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:27:18.438231+0000","last_peered":"2024-03-18T20:27:18.438231+0000","last_clean":"2024-03-18T20:27:18.438231+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:27:18.438231+0000","last_undegraded":"2024-03-18T20:27:18.438231+0000","last_fullsized":"2024-03-18T20:27:18.438231+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":64,"acting":64,"num_store_stats":8},{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":1},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576001,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53088,"kb_used_data":3924,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318752,"statfs":{"total":96636764160,"available":96582402048,"internally_reserved":0,"allocated":4018176,"data_stored":2363924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54900000000000004,"15min":0.58999999999999997},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.72599999999999998,"5min":0.80900000000000005,"15min":1.1339999999999999},"last":0.439},{"interface":"front","average":{"1min":0.498,"5min":0.56599999999999995,"15min":0.60199999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":0.73599999999999999,"5min":0.84499999999999997,"15min":1.093},"last":0.56699999999999995}]},{"osd":1,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.55500000000000005,"15min":0.60199999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.748,"5min":0.83099999999999996,"15min":1.4299999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.57099999999999995,"15min":0.60799999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73599999999999999,"5min":0.82399999999999995,"15min":1.2629999999999999},"last":0.61599999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.56100000000000005,"15min":0.60099999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67600000000000005,"5min":0.85499999999999998,"15min":1.216},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57799999999999996,"15min":0.61599999999999999},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75,"5min":1.1279999999999999,"15min":1.2490000000000001},"last":0.59899999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":0.59799999999999998},"min":{"1min":0.38700000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71999999999999997,"5min":0.84699999999999998,"15min":1.4450000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.60699999999999998,"15min":0.64100000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73099999999999998,"5min":0.94599999999999995,"15min":1.5029999999999999},"last":0.63300000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47099999999999997,"15min":0.47799999999999998},"min":{"1min":0.35999999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.68899999999999995,"5min":0.79100000000000004,"15min":0.86099999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48899999999999999,"15min":0.499},"min":{"1min":0.34100000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.66200000000000003,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.54500000000000004}]},{"osd":5,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.51300000000000001,"15min":0.51900000000000002},"min":{"1min":0.373,"5min":0.314,"15min":0.314},"max":{"1min":0.70399999999999996,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.54600000000000004},"min":{"1min":0.38300000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73099999999999998,"5min":0.877,"15min":1.042},"last":0.68799999999999994}]},{"osd":6,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55800000000000005,"15min":0.57699999999999996},"min":{"1min":0.435,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77300000000000002,"5min":0.93500000000000005,"15min":1.129},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.54700000000000004,"15min":0.56899999999999995},"min":{"1min":0.47399999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76200000000000001,"5min":0.91900000000000004,"15min":1.0609999999999999},"last":0.46899999999999997}]}]},{"osd":6,"up_from":13,"seq":55834576001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60700,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311140,"statfs":{"total":96636764160,"available":96574607360,"internally_reserved":0,"allocated":3424256,"data_stored":1773556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53300000000000003,"15min":0.58499999999999996},"min":{"1min":0.35399999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68799999999999994,"5min":0.79800000000000004,"15min":1.1519999999999999},"last":0.505},{"interface":"front","average":{"1min":0.503,"5min":0.54600000000000004,"15min":0.59499999999999997},"min":{"1min":0.33200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.73499999999999999,"5min":1.119,"15min":1.119},"last":0.67100000000000004}]},{"osd":1,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53800000000000003,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73199999999999998,"5min":0.79100000000000004,"15min":1.244},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":0.59499999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":0.70099999999999996,"5min":0.89200000000000002,"15min":1.286},"last":0.54600000000000004}]},{"osd":2,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56399999999999995,"15min":0.61699999999999999},"min":{"1min":0.38700000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.69599999999999995,"5min":0.79600000000000004,"15min":1.276},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.55400000000000005,"15min":0.59899999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.65200000000000002,"5min":1.089,"15min":1.2669999999999999},"last":0.53700000000000003}]},{"osd":3,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55500000000000005,"15min":0.61099999999999999},"min":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71599999999999997,"5min":1.163,"15min":1.163},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57099999999999995,"15min":0.61099999999999999},"min":{"1min":0.47499999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78100000000000003,"5min":0.82399999999999995,"15min":1.337},"last":0.55500000000000005}]},{"osd":4,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.35299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80200000000000005,"5min":0.85799999999999998,"15min":0.88300000000000001},"last":0.378},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.50600000000000001,"15min":0.51800000000000002},"min":{"1min":0.32500000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83399999999999996,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.40200000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.50900000000000001,"15min":0.51200000000000001},"min":{"1min":0.42599999999999999,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":0.65000000000000002,"5min":1.008,"15min":1.008},"last":0.441},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55900000000000005,"15min":0.55300000000000005},"min":{"1min":0.41799999999999998,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.755,"5min":0.92600000000000005,"15min":0.94199999999999995},"last":0.43099999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56699999999999995,"15min":0.56299999999999994},"min":{"1min":0.40999999999999998,"5min":0.38,"15min":0.34499999999999997},"max":{"1min":0.91200000000000003,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.34599999999999997},"max":{"1min":0.86699999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.52000000000000002}]}]},{"osd":1,"up_from":13,"seq":55834576002,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52548,"kb_used_data":3384,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319292,"statfs":{"total":96636764160,"available":96582955008,"internally_reserved":0,"allocated":3465216,"data_stored":1794256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.38700000000000001,"15min":0.39300000000000002},"min":{"1min":0.255,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.46400000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39500000000000002,"15min":0.41999999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.623,"5min":0.623,"15min":0.69699999999999995},"last":0.39900000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.40899999999999997,"15min":0.42899999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53100000000000003,"5min":0.61199999999999999,"15min":0.68100000000000005},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.45200000000000001,"15min":0.47099999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.69799999999999995,"5min":0.70099999999999996,"15min":0.80500000000000005},"last":0.34799999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.46500000000000002,"15min":0.48399999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.76100000000000001},"last":0.44},{"interface":"front","average":{"1min":0.372,"5min":0.41899999999999998,"15min":0.433},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.27000000000000002},"max":{"1min":0.54700000000000004,"5min":0.68200000000000005,"15min":0.71699999999999997},"last":0.32400000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.64000000000000001,"15min":0.66500000000000004},"min":{"1min":0.43099999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76800000000000002,"5min":1.0680000000000001,"15min":1.2909999999999999},"last":0.629},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.64600000000000002,"15min":0.67100000000000004},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.88500000000000001,"5min":1.1160000000000001,"15min":1.746},"last":0.55300000000000005}]},{"osd":5,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.63400000000000001,"15min":0.66100000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79300000000000004,"5min":1.0389999999999999,"15min":1.6910000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.67800000000000005,"15min":0.70499999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.85199999999999998,"5min":1.109,"15min":1.2549999999999999},"last":0.59199999999999997}]},{"osd":6,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.65300000000000002,"15min":0.67700000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.70799999999999996,"5min":1.0289999999999999,"15min":1.637},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.67800000000000005,"15min":0.70199999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82099999999999995,"5min":1.014,"15min":1.722},"last":0.52400000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.66000000000000003,"15min":0.69899999999999995},"min":{"1min":0.46100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91100000000000003,"5min":0.99199999999999999,"15min":1.2729999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.67000000000000004,"15min":0.69699999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.80700000000000005,"5min":1.0329999999999999,"15min":1.3380000000000001},"last":0.66900000000000004}]}]},{"osd":0,"up_from":13,"seq":55834576002,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48992,"kb_used_data":3924,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322848,"statfs":{"total":96636764160,"available":96586596352,"internally_reserved":0,"allocated":4018176,"data_stored":2363924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.36399999999999999,"15min":0.38600000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.46100000000000002,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.39600000000000002,"15min":0.41999999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.58799999999999997,"5min":0.60099999999999998,"15min":1.1850000000000001},"last":0.255}]},{"osd":2,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.41299999999999998,"15min":0.433},"min":{"1min":0.28699999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.47199999999999998,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.41799999999999998,"15min":0.44600000000000001},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.54500000000000004,"5min":0.64800000000000002,"15min":0.70999999999999996},"last":0.39400000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.39600000000000002,"15min":0.42399999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.433,"5min":0.58499999999999996,"15min":0.628},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.45400000000000001,"15min":0.48699999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.52600000000000002,"5min":0.73399999999999999,"15min":0.91500000000000004},"last":0.379}]},{"osd":4,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.61299999999999999,"15min":0.64400000000000002},"min":{"1min":0.41299999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.752,"5min":0.91500000000000004,"15min":1.0029999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.64800000000000002,"15min":0.65900000000000003},"min":{"1min":0.51600000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.92300000000000004,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.72099999999999997}]},{"osd":5,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.64400000000000002,"15min":0.66600000000000004},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83299999999999996,"5min":0.92300000000000004,"15min":1.0129999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65300000000000002,"15min":0.67500000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.73199999999999998,"5min":0.96199999999999997,"15min":1.518},"last":0.70499999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.64600000000000002,"15min":0.65700000000000003},"min":{"1min":0.45800000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.73599999999999999,"5min":1.0669999999999999,"15min":1.1279999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.68200000000000005,"15min":0.70299999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.871,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.746}]},{"osd":7,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.67200000000000004,"15min":0.69899999999999995},"min":{"1min":0.45300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77500000000000002,"5min":1.008,"15min":1.464},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.67300000000000004,"15min":0.69799999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88900000000000001,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.76300000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52508,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319332,"statfs":{"total":96636764160,"available":96582995968,"internally_reserved":0,"allocated":3424256,"data_stored":1773556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.34699999999999998,"15min":0.371},"min":{"1min":0.20799999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.53800000000000003,"5min":0.54800000000000004,"15min":0.57499999999999996},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.44700000000000001,"15min":0.47799999999999998},"min":{"1min":0.26100000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60499999999999998,"5min":0.71499999999999997,"15min":0.753},"last":0.48999999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.44400000000000001,"15min":0.47399999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57399999999999995,"5min":0.64300000000000002,"15min":0.749},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.38900000000000001,"15min":0.40899999999999997},"min":{"1min":0.246,"5min":0.224,"15min":0.224},"max":{"1min":0.501,"5min":0.55900000000000005,"15min":0.59999999999999998},"last":0.32700000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.41199999999999998,"15min":0.42899999999999999},"min":{"1min":0.27800000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.56100000000000005,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.47399999999999998,"15min":0.50700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.63200000000000001,"5min":0.68100000000000005,"15min":0.752},"last":0.45400000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63900000000000001,"15min":0.65300000000000002},"min":{"1min":0.41799999999999998,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":1.2090000000000001,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.65700000000000003,"15min":0.66200000000000003},"min":{"1min":0.45100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.268,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.61899999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.64900000000000002,"15min":0.67500000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79900000000000004,"5min":0.97299999999999998,"15min":1.028},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.63900000000000001,"15min":0.67000000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.77100000000000002,"5min":0.88300000000000001,"15min":1.0629999999999999},"last":0.80800000000000005}]},{"osd":6,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.65500000000000003,"15min":0.68100000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.81899999999999995,"5min":1.014,"15min":1.2050000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.72699999999999998},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.3220000000000001,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.77000000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.65500000000000003,"15min":0.68899999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.42199999999999999},"max":{"1min":0.73699999999999999,"5min":0.99399999999999999,"15min":1.0269999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68600000000000005,"15min":0.70099999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.2330000000000001,"5min":1.371,"15min":1.371},"last":0.79100000000000004}]}]},{"osd":3,"up_from":13,"seq":55834576002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52508,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319332,"statfs":{"total":96636764160,"available":96582995968,"internally_reserved":0,"allocated":3424256,"data_stored":1773556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.36099999999999999,"15min":0.39000000000000001},"min":{"1min":0.247,"5min":0.223,"15min":0.223},"max":{"1min":0.46400000000000002,"5min":0.66500000000000004,"15min":0.98099999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.373,"15min":0.40400000000000003},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.45600000000000002,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.42399999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.35999999999999999,"15min":0.39700000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.44500000000000001,"5min":0.63200000000000001,"15min":0.64000000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.42199999999999999,"15min":0.45600000000000002},"min":{"1min":0.33400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.501,"5min":0.65500000000000003,"15min":0.73099999999999998},"last":0.44500000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.41499999999999998,"15min":0.45200000000000001},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.54900000000000004,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.437,"15min":0.48099999999999998},"min":{"1min":0.32600000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.52300000000000002,"5min":0.64400000000000002,"15min":0.71199999999999997},"last":0.32600000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59499999999999997,"15min":0.64000000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.526,"5min":1.526,"15min":1.526},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61399999999999999,"15min":0.66100000000000003},"min":{"1min":0.39200000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.65800000000000003,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.65800000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.65100000000000002,"15min":0.68700000000000006},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.66500000000000004,"15min":0.71499999999999997},"min":{"1min":0.45300000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78000000000000003,"5min":1.1120000000000001,"15min":1.171},"last":0.65000000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65800000000000003,"15min":0.69699999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.62},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.71999999999999997},"min":{"1min":0.48699999999999999,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.58999999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.64900000000000002,"15min":0.70299999999999996},"min":{"1min":0.43099999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79900000000000004,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64300000000000002,"15min":0.68899999999999995},"min":{"1min":0.51500000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.57299999999999995}]}]},{"osd":4,"up_from":13,"seq":55834576002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48400,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323440,"statfs":{"total":96636764160,"available":96587202560,"internally_reserved":0,"allocated":3411968,"data_stored":1767319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57799999999999996,"15min":0.58899999999999997},"min":{"1min":0.39600000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56799999999999995,"15min":0.59499999999999997},"min":{"1min":0.40100000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.84199999999999997},"last":0.621}]},{"osd":1,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.79300000000000004,"5min":0.81899999999999995,"15min":0.88300000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56399999999999995,"15min":0.57999999999999996},"min":{"1min":0.34899999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.96499999999999997},"last":0.58999999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59799999999999998,"15min":0.61399999999999999},"min":{"1min":0.42899999999999999,"5min":0.41299999999999998,"15min":0.38500000000000001},"max":{"1min":0.70799999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.54500000000000004}]},{"osd":3,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.59299999999999997,"15min":0.60199999999999998},"min":{"1min":0.433,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78100000000000003,"5min":1.026,"15min":1.026},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.41399999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.72999999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.48399999999999999,"15min":0.48199999999999998},"min":{"1min":0.318,"5min":0.309,"15min":0.309},"max":{"1min":0.76700000000000002,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.747},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.496,"15min":0.505},"min":{"1min":0.38300000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.81499999999999995,"5min":0.85499999999999998,"15min":0.90600000000000003},"last":0.63800000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.502,"15min":0.50600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.754,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.82799999999999996,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.71399999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.40600000000000003,"5min":0.38800000000000001,"15min":0.35699999999999998},"max":{"1min":0.80300000000000005,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.38500000000000001,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.84099999999999997,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.65200000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48412,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323428,"statfs":{"total":96636764160,"available":96587190272,"internally_reserved":0,"allocated":3424256,"data_stored":1773556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.55400000000000005,"15min":0.59099999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.31900000000000001},"max":{"1min":0.76300000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.621},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55600000000000005,"15min":0.59499999999999997},"min":{"1min":0.42099999999999999,"5min":0.35599999999999998,"15min":0.33100000000000002},"max":{"1min":0.81100000000000005,"5min":0.82599999999999996,"15min":1.3100000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.56100000000000005,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71499999999999997,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.55400000000000005,"15min":0.59199999999999997},"min":{"1min":0.372,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.69399999999999995,"5min":0.94299999999999995,"15min":1.018},"last":0.58499999999999996}]},{"osd":2,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.59399999999999997},"min":{"1min":0.40300000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.75600000000000001,"5min":0.85299999999999998,"15min":0.99299999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.55400000000000005,"15min":0.60099999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77000000000000002,"5min":1.599,"15min":1.599},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57499999999999996,"15min":0.60399999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72899999999999998,"5min":0.98699999999999999,"15min":1.2430000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59299999999999997,"15min":0.63700000000000001},"min":{"1min":0.42099999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":0.84399999999999997,"15min":1.161},"last":0.63300000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48999999999999999,"15min":0.48299999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.625,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.35699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.40699999999999997}]},{"osd":6,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.503,"15min":0.50900000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.32800000000000001},"max":{"1min":0.64800000000000002,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54300000000000004,"15min":0.54200000000000004},"min":{"1min":0.35199999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.64600000000000002,"5min":1.643,"15min":1.643},"last":0.501}]},{"osd":7,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.40200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.71299999999999997,"5min":1.028,"15min":1.028},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57099999999999995,"15min":0.57799999999999996},"min":{"1min":0.42599999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74299999999999999,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:27:26.677 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:27:26.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.833+0000 7f068e52d700 1 Processor -- start 2024-03-18T20:27:26.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.834+0000 7f068e52d700 1 -- start start 2024-03-18T20:27:26.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.834+0000 7f068e52d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f068815d320 0x7f06880cfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.834+0000 7f068e52d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f068806cb10 con 0x7f06880d0210 2024-03-18T20:27:26.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.834+0000 7f068e52d700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f068806ba70 con 0x7f068809a150 2024-03-18T20:27:26.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.834+0000 7f068e52d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f068806b860 con 0x7f068815d320 2024-03-18T20:27:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.834+0000 7f06877fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f06880d0210 0x7f0688099a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37454/0 (socket says 172.21.15.1:37454) 2024-03-18T20:27:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f06877fe700 1 -- 172.21.15.1:0/2392416264 learned_addr learned my addr 172.21.15.1:0/2392416264 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f0686ffd700 1 -- 172.21.15.1:0/2392416264 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253262 0 0) 0x7f068806cb10 con 0x7f06880d0210 2024-03-18T20:27:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f0687fff700 1 --2- 172.21.15.1:0/2392416264 >> v2:172.21.15.165:3300/0 conn(0x7f068815d320 0x7f06880cfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f0686ffd700 1 -- 172.21.15.1:0/2392416264 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06740035b0 con 0x7f06880d0210 2024-03-18T20:27:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f0686ffd700 1 -- 172.21.15.1:0/2392416264 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2133088017 0 0) 0x7f06740035b0 con 0x7f06880d0210 2024-03-18T20:27:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f0686ffd700 1 -- 172.21.15.1:0/2392416264 >> v2:172.21.15.165:3300/0 conn(0x7f068815d320 msgr2=0x7f06880cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f0686ffd700 1 --2- 172.21.15.1:0/2392416264 >> v2:172.21.15.165:3300/0 conn(0x7f068815d320 0x7f06880cfcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f0686ffd700 1 -- 172.21.15.1:0/2392416264 >> v1:172.21.15.165:6789/0 conn(0x7f068809a150 legacy=0x7f0688167340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f0686ffd700 1 -- 172.21.15.1:0/2392416264 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f068806bd50 con 0x7f06880d0210 2024-03-18T20:27:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f0686ffd700 1 -- 172.21.15.1:0/2392416264 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f0670001670 con 0x7f06880d0210 2024-03-18T20:27:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f0686ffd700 1 -- 172.21.15.1:0/2392416264 >> v1:172.21.15.1:6789/0 conn(0x7f06880d0210 legacy=0x7f0688099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:26.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f0686ffd700 1 --2- 172.21.15.1:0/2392416264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0674003a00 0x7f0674003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f0686ffd700 1 --2- 172.21.15.1:0/2392416264 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0674004850 0x7f0674004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f0686ffd700 1 --2- 172.21.15.1:0/2392416264 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0674005210 0x7f067400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f0686ffd700 1 -- 172.21.15.1:0/2392416264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068806b860 con 0x7f0674003a00 2024-03-18T20:27:26.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f0686ffd700 1 -- 172.21.15.1:0/2392416264 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067400c0b0 con 0x7f0674004850 2024-03-18T20:27:26.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f0686ffd700 1 -- 172.21.15.1:0/2392416264 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067400c270 con 0x7f0674005210 2024-03-18T20:27:26.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f0687fff700 1 --2- 172.21.15.1:0/2392416264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0674003a00 0x7f0674003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f06877fe700 1 --2- 172.21.15.1:0/2392416264 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0674004850 0x7f0674004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.835+0000 7f068caca700 1 --2- 172.21.15.1:0/2392416264 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0674005210 0x7f067400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.836+0000 7f0687fff700 1 -- 172.21.15.1:0/2392416264 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0674005210 msgr2=0x7f067400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:26.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.836+0000 7f0687fff700 1 --2- 172.21.15.1:0/2392416264 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0674005210 0x7f067400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.836+0000 7f0687fff700 1 -- 172.21.15.1:0/2392416264 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0674004850 msgr2=0x7f0674004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:26.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.836+0000 7f0687fff700 1 --2- 172.21.15.1:0/2392416264 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0674004850 0x7f0674004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.836+0000 7f0687fff700 1 -- 172.21.15.1:0/2392416264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06740035b0 con 0x7f0674003a00 2024-03-18T20:27:26.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.836+0000 7f068caca700 1 --2- 172.21.15.1:0/2392416264 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0674005210 0x7f067400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:26.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.836+0000 7f068caca700 1 -- 172.21.15.1:0/2392416264 reap_dead start 2024-03-18T20:27:26.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.836+0000 7f0687fff700 1 --2- 172.21.15.1:0/2392416264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0674003a00 0x7f0674003e20 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 crypto rx=0x7f0678009b60 tx=0x7f0674005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25826e53162906c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:26.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.836+0000 7f0686ffd700 1 -- 172.21.15.1:0/2392416264 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067800e6c0 con 0x7f0674003a00 2024-03-18T20:27:26.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.836+0000 7f0686ffd700 1 -- 172.21.15.1:0/2392416264 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067800e820 con 0x7f0674003a00 2024-03-18T20:27:26.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.836+0000 7f0686ffd700 1 -- 172.21.15.1:0/2392416264 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067800eb90 con 0x7f0674003a00 2024-03-18T20:27:26.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.836+0000 7f068e52d700 1 -- 172.21.15.1:0/2392416264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0674003a00 msgr2=0x7f0674003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:26.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.836+0000 7f068e52d700 1 --2- 172.21.15.1:0/2392416264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0674003a00 0x7f0674003e20 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 crypto rx=0x7f0678009b60 tx=0x7f0674005880 comp rx=0 tx=0).stop 2024-03-18T20:27:26.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.836+0000 7f068e52d700 1 -- 172.21.15.1:0/2392416264 shutdown_connections 2024-03-18T20:27:26.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.836+0000 7f068e52d700 1 --2- 172.21.15.1:0/2392416264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0674003a00 0x7f0674003e20 unknown :-1 s=CLOSED pgs=3080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.836+0000 7f068e52d700 1 -- 172.21.15.1:0/2392416264 >> 172.21.15.1:0/2392416264 conn(0x7f06880b2b50 msgr2=0x7f06880b2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:26.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.836+0000 7f068e52d700 1 -- 172.21.15.1:0/2392416264 shutdown_connections 2024-03-18T20:27:26.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.836+0000 7f068e52d700 1 -- 172.21.15.1:0/2392416264 wait complete. 2024-03-18T20:27:26.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.838+0000 7f068e52d700 1 Processor -- start 2024-03-18T20:27:26.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.838+0000 7f068e52d700 1 -- start start 2024-03-18T20:27:26.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f068e52d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0674003a00 0x7f068814c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f068e52d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0688159590 0x7f06880b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f068e52d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06880b6af0 0x7f06880aef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f068e52d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06880b6660 con 0x7f0674003a00 2024-03-18T20:27:26.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f068e52d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06880b5d30 con 0x7f0688159590 2024-03-18T20:27:26.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f068e52d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06880b6010 con 0x7f06880b6af0 2024-03-18T20:27:26.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f0687fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0674003a00 0x7f068814c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f0687fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0674003a00 0x7f068814c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36530/0 (socket says 172.21.15.1:36530) 2024-03-18T20:27:26.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f0687fff700 1 -- 172.21.15.1:0/2689683746 learned_addr learned my addr 172.21.15.1:0/2689683746 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f0687fff700 1 -- 172.21.15.1:0/2689683746 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06880b6af0 msgr2=0x7f06880aef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f0687fff700 1 --2- 172.21.15.1:0/2689683746 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06880b6af0 0x7f06880aef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f0687fff700 1 -- 172.21.15.1:0/2689683746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0688159590 msgr2=0x7f06880b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f0687fff700 1 --2- 172.21.15.1:0/2689683746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0688159590 0x7f06880b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f0687fff700 1 -- 172.21.15.1:0/2689683746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06880b62f0 con 0x7f0674003a00 2024-03-18T20:27:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f0687fff700 1 --2- 172.21.15.1:0/2689683746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0674003a00 0x7f068814c210 secure :-1 s=READY pgs=3081 cs=0 l=1 rev1=1 crypto rx=0x7f067400c080 tx=0x7f06780047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f0684ff9700 1 -- 172.21.15.1:0/2689683746 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067800e6c0 con 0x7f0674003a00 2024-03-18T20:27:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f0684ff9700 1 -- 172.21.15.1:0/2689683746 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0678003ad0 con 0x7f0674003a00 2024-03-18T20:27:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f068e52d700 1 -- 172.21.15.1:0/2689683746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06880b59e0 con 0x7f0674003a00 2024-03-18T20:27:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f0684ff9700 1 -- 172.21.15.1:0/2689683746 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06780045d0 con 0x7f0674003a00 2024-03-18T20:27:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.839+0000 7f068e52d700 1 -- 172.21.15.1:0/2689683746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06880b3470 con 0x7f0674003a00 2024-03-18T20:27:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.840+0000 7f068e52d700 1 -- 172.21.15.1:0/2689683746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06880b6940 con 0x7f0674003a00 2024-03-18T20:27:26.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.841+0000 7f0684ff9700 1 -- 172.21.15.1:0/2689683746 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f0678013470 con 0x7f0674003a00 2024-03-18T20:27:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.842+0000 7f0684ff9700 1 --2- 172.21.15.1:0/2689683746 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f066806af00 0x7f066806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.842+0000 7f0684ff9700 1 -- 172.21.15.1:0/2689683746 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f067808b7e0 con 0x7f0674003a00 2024-03-18T20:27:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.842+0000 7f06877fe700 1 --2- 172.21.15.1:0/2689683746 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f066806af00 0x7f066806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.842+0000 7f06877fe700 1 --2- 172.21.15.1:0/2689683746 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f066806af00 0x7f066806d3c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f06700011e0 tx=0x7f0670001150 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:26.846+0000 7f0684ff9700 1 -- 172.21.15.1:0/2689683746 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0678056640 con 0x7f0674003a00 2024-03-18T20:27:27.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.047+0000 7f068e52d700 1 -- 172.21.15.1:0/2689683746 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f068806c5f0 con 0x7f066806af00 2024-03-18T20:27:27.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.055+0000 7f0684ff9700 1 -- 172.21.15.1:0/2689683746 <== mgr.12813 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+280373 (secure 0 0 0) 0x7f0670007e00 con 0x7f066806af00 2024-03-18T20:27:27.063 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:27.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.058+0000 7f068e52d700 1 -- 172.21.15.1:0/2689683746 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f066806af00 msgr2=0x7f066806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:27.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.058+0000 7f068e52d700 1 --2- 172.21.15.1:0/2689683746 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f066806af00 0x7f066806d3c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f06700011e0 tx=0x7f0670001150 comp rx=0 tx=0).stop 2024-03-18T20:27:27.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.058+0000 7f068e52d700 1 -- 172.21.15.1:0/2689683746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0674003a00 msgr2=0x7f068814c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:27.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.058+0000 7f068e52d700 1 --2- 172.21.15.1:0/2689683746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0674003a00 0x7f068814c210 secure :-1 s=READY pgs=3081 cs=0 l=1 rev1=1 crypto rx=0x7f067400c080 tx=0x7f06780047c0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.058+0000 7f068e52d700 1 -- 172.21.15.1:0/2689683746 shutdown_connections 2024-03-18T20:27:27.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.058+0000 7f068e52d700 1 --2- 172.21.15.1:0/2689683746 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f066806af00 0x7f066806d3c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.058+0000 7f068e52d700 1 --2- 172.21.15.1:0/2689683746 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f06880b6af0 0x7f06880aef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.058+0000 7f068e52d700 1 --2- 172.21.15.1:0/2689683746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0674003a00 0x7f068814c210 unknown :-1 s=CLOSED pgs=3081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.058+0000 7f068e52d700 1 --2- 172.21.15.1:0/2689683746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0688159590 0x7f06880b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.058+0000 7f068e52d700 1 -- 172.21.15.1:0/2689683746 >> 172.21.15.1:0/2689683746 conn(0x7f06880b2b50 msgr2=0x7f06880adf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.058+0000 7f068e52d700 1 -- 172.21.15.1:0/2689683746 shutdown_connections 2024-03-18T20:27:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.058+0000 7f068e52d700 1 -- 172.21.15.1:0/2689683746 wait complete. 2024-03-18T20:27:27.066 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:27:27.075 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-03-18T20:27:26.313769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":642,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":84,"acting":84,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":24,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":417156,"kb_used_data":27940,"kb_used_omap":12,"kb_used_meta":389107,"kb_avail":754557564,"statfs":{"total":773094113280,"available":772666945536,"internally_reserved":0,"allocated":28610560,"data_stored":15383647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":398446137},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.698740"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218004+0000","last_change":"2024-03-18T20:27:22.055309+0000","last_active":"2024-03-18T20:27:24.218004+0000","last_peered":"2024-03-18T20:27:24.218004+0000","last_clean":"2024-03-18T20:27:24.218004+0000","last_became_active":"2024-03-18T20:27:22.055064+0000","last_became_peered":"2024-03-18T20:27:22.055064+0000","last_unstale":"2024-03-18T20:27:24.218004+0000","last_undegraded":"2024-03-18T20:27:24.218004+0000","last_fullsized":"2024-03-18T20:27:24.218004+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:19.253084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.217979+0000","last_change":"2024-03-18T20:27:22.062816+0000","last_active":"2024-03-18T20:27:24.217979+0000","last_peered":"2024-03-18T20:27:24.217979+0000","last_clean":"2024-03-18T20:27:24.217979+0000","last_became_active":"2024-03-18T20:27:22.062600+0000","last_became_peered":"2024-03-18T20:27:22.062600+0000","last_unstale":"2024-03-18T20:27:24.217979+0000","last_undegraded":"2024-03-18T20:27:24.217979+0000","last_fullsized":"2024-03-18T20:27:24.217979+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:15:52.725667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218003+0000","last_change":"2024-03-18T20:27:22.062897+0000","last_active":"2024-03-18T20:27:24.218003+0000","last_peered":"2024-03-18T20:27:24.218003+0000","last_clean":"2024-03-18T20:27:24.218003+0000","last_became_active":"2024-03-18T20:27:22.062639+0000","last_became_peered":"2024-03-18T20:27:22.062639+0000","last_unstale":"2024-03-18T20:27:24.218003+0000","last_undegraded":"2024-03-18T20:27:24.218003+0000","last_fullsized":"2024-03-18T20:27:24.218003+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:45.973738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.217918+0000","last_change":"2024-03-18T20:27:22.073284+0000","last_active":"2024-03-18T20:27:24.217918+0000","last_peered":"2024-03-18T20:27:24.217918+0000","last_clean":"2024-03-18T20:27:24.217918+0000","last_became_active":"2024-03-18T20:27:22.072955+0000","last_became_peered":"2024-03-18T20:27:22.072955+0000","last_unstale":"2024-03-18T20:27:24.217918+0000","last_undegraded":"2024-03-18T20:27:24.217918+0000","last_fullsized":"2024-03-18T20:27:24.217918+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:10.708304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.622037+0000","last_change":"2024-03-18T20:27:20.622037+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.622037+0000","last_undegraded":"2024-03-18T20:27:20.622037+0000","last_fullsized":"2024-03-18T20:27:20.622037+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:01.698353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621734+0000","last_change":"2024-03-18T20:27:20.621734+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621734+0000","last_undegraded":"2024-03-18T20:27:20.621734+0000","last_fullsized":"2024-03-18T20:27:20.621734+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:02:14.143177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.622565+0000","last_change":"2024-03-18T20:27:20.622565+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.622565+0000","last_undegraded":"2024-03-18T20:27:20.622565+0000","last_fullsized":"2024-03-18T20:27:20.622565+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:42:23.110707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.622373+0000","last_change":"2024-03-18T20:27:20.622373+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.622373+0000","last_undegraded":"2024-03-18T20:27:20.622373+0000","last_fullsized":"2024-03-18T20:27:20.622373+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:45:52.316434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617958+0000","last_change":"2024-03-18T20:27:20.617958+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617958+0000","last_undegraded":"2024-03-18T20:27:20.617958+0000","last_fullsized":"2024-03-18T20:27:20.617958+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:08:18.189309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1a","version":"273'1","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.069050+0000","last_change":"2024-03-18T20:27:21.635634+0000","last_active":"2024-03-18T20:27:22.069050+0000","last_peered":"2024-03-18T20:27:22.069050+0000","last_clean":"2024-03-18T20:27:22.069050+0000","last_became_active":"2024-03-18T20:27:21.635366+0000","last_became_peered":"2024-03-18T20:27:21.635366+0000","last_unstale":"2024-03-18T20:27:22.069050+0000","last_undegraded":"2024-03-18T20:27:22.069050+0000","last_fullsized":"2024-03-18T20:27:22.069050+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:40:40.016554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623987+0000","last_change":"2024-03-18T20:27:20.623987+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623987+0000","last_undegraded":"2024-03-18T20:27:20.623987+0000","last_fullsized":"2024-03-18T20:27:20.623987+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:08:18.189309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621015+0000","last_change":"2024-03-18T20:27:20.621015+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621015+0000","last_undegraded":"2024-03-18T20:27:20.621015+0000","last_fullsized":"2024-03-18T20:27:20.621015+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:10.342403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623746+0000","last_change":"2024-03-18T20:27:20.623746+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623746+0000","last_undegraded":"2024-03-18T20:27:20.623746+0000","last_fullsized":"2024-03-18T20:27:20.623746+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:28:52.874751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623648+0000","last_change":"2024-03-18T20:27:20.623648+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623648+0000","last_undegraded":"2024-03-18T20:27:20.623648+0000","last_fullsized":"2024-03-18T20:27:20.623648+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:21:19.006472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.624317+0000","last_change":"2024-03-18T20:27:20.624317+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.624317+0000","last_undegraded":"2024-03-18T20:27:20.624317+0000","last_fullsized":"2024-03-18T20:27:20.624317+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:29.812382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218074+0000","last_change":"2024-03-18T20:27:22.081779+0000","last_active":"2024-03-18T20:27:24.218074+0000","last_peered":"2024-03-18T20:27:24.218074+0000","last_clean":"2024-03-18T20:27:24.218074+0000","last_became_active":"2024-03-18T20:27:22.081506+0000","last_became_peered":"2024-03-18T20:27:22.081506+0000","last_unstale":"2024-03-18T20:27:24.218074+0000","last_undegraded":"2024-03-18T20:27:24.218074+0000","last_fullsized":"2024-03-18T20:27:24.218074+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:29:27.205683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621040+0000","last_change":"2024-03-18T20:27:20.621040+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621040+0000","last_undegraded":"2024-03-18T20:27:20.621040+0000","last_fullsized":"2024-03-18T20:27:20.621040+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:01:31.379909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.625530+0000","last_change":"2024-03-18T20:27:20.625530+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.625530+0000","last_undegraded":"2024-03-18T20:27:20.625530+0000","last_fullsized":"2024-03-18T20:27:20.625530+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:42.339058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617987+0000","last_change":"2024-03-18T20:27:20.617987+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617987+0000","last_undegraded":"2024-03-18T20:27:20.617987+0000","last_fullsized":"2024-03-18T20:27:20.617987+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:42.339058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617911+0000","last_change":"2024-03-18T20:27:20.617911+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617911+0000","last_undegraded":"2024-03-18T20:27:20.617911+0000","last_fullsized":"2024-03-18T20:27:20.617911+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:43.006246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621574+0000","last_change":"2024-03-18T20:27:20.621574+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621574+0000","last_undegraded":"2024-03-18T20:27:20.621574+0000","last_fullsized":"2024-03-18T20:27:20.621574+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:31:56.485139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.217869+0000","last_change":"2024-03-18T20:27:22.054635+0000","last_active":"2024-03-18T20:27:24.217869+0000","last_peered":"2024-03-18T20:27:24.217869+0000","last_clean":"2024-03-18T20:27:24.217869+0000","last_became_active":"2024-03-18T20:27:22.054372+0000","last_became_peered":"2024-03-18T20:27:22.054372+0000","last_unstale":"2024-03-18T20:27:24.217869+0000","last_undegraded":"2024-03-18T20:27:24.217869+0000","last_fullsized":"2024-03-18T20:27:24.217869+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:40.826497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621698+0000","last_change":"2024-03-18T20:27:20.621698+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621698+0000","last_undegraded":"2024-03-18T20:27:20.621698+0000","last_fullsized":"2024-03-18T20:27:20.621698+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:17:27.485874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.e","version":"273'2","reported_seq":13,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.083274+0000","last_change":"2024-03-18T20:27:21.640607+0000","last_active":"2024-03-18T20:27:22.083274+0000","last_peered":"2024-03-18T20:27:22.083274+0000","last_clean":"2024-03-18T20:27:22.083274+0000","last_became_active":"2024-03-18T20:27:21.640341+0000","last_became_peered":"2024-03-18T20:27:21.640341+0000","last_unstale":"2024-03-18T20:27:22.083274+0000","last_undegraded":"2024-03-18T20:27:22.083274+0000","last_fullsized":"2024-03-18T20:27:22.083274+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:20:18.042657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218120+0000","last_change":"2024-03-18T20:27:22.055634+0000","last_active":"2024-03-18T20:27:24.218120+0000","last_peered":"2024-03-18T20:27:24.218120+0000","last_clean":"2024-03-18T20:27:24.218120+0000","last_became_active":"2024-03-18T20:27:22.055397+0000","last_became_peered":"2024-03-18T20:27:22.055397+0000","last_unstale":"2024-03-18T20:27:24.218120+0000","last_undegraded":"2024-03-18T20:27:24.218120+0000","last_fullsized":"2024-03-18T20:27:24.218120+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:29.388876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621611+0000","last_change":"2024-03-18T20:27:20.621611+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621611+0000","last_undegraded":"2024-03-18T20:27:20.621611+0000","last_fullsized":"2024-03-18T20:27:20.621611+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:58:00.382684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623890+0000","last_change":"2024-03-18T20:27:20.623890+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623890+0000","last_undegraded":"2024-03-18T20:27:20.623890+0000","last_fullsized":"2024-03-18T20:27:20.623890+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:01.555443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.624331+0000","last_change":"2024-03-18T20:27:20.624331+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.624331+0000","last_undegraded":"2024-03-18T20:27:20.624331+0000","last_fullsized":"2024-03-18T20:27:20.624331+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:52.857794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.622224+0000","last_change":"2024-03-18T20:27:20.622224+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.622224+0000","last_undegraded":"2024-03-18T20:27:20.622224+0000","last_fullsized":"2024-03-18T20:27:20.622224+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:13:18.842477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621046+0000","last_change":"2024-03-18T20:27:20.621046+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621046+0000","last_undegraded":"2024-03-18T20:27:20.621046+0000","last_fullsized":"2024-03-18T20:27:20.621046+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:11:16.795077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218061+0000","last_change":"2024-03-18T20:27:22.062557+0000","last_active":"2024-03-18T20:27:24.218061+0000","last_peered":"2024-03-18T20:27:24.218061+0000","last_clean":"2024-03-18T20:27:24.218061+0000","last_became_active":"2024-03-18T20:27:22.062283+0000","last_became_peered":"2024-03-18T20:27:22.062283+0000","last_unstale":"2024-03-18T20:27:24.218061+0000","last_undegraded":"2024-03-18T20:27:24.218061+0000","last_fullsized":"2024-03-18T20:27:24.218061+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:05:24.597193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.625820+0000","last_change":"2024-03-18T20:27:20.625820+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.625820+0000","last_undegraded":"2024-03-18T20:27:20.625820+0000","last_fullsized":"2024-03-18T20:27:20.625820+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:41:34.677506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.624685+0000","last_change":"2024-03-18T20:27:20.624685+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.624685+0000","last_undegraded":"2024-03-18T20:27:20.624685+0000","last_fullsized":"2024-03-18T20:27:20.624685+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:45.164373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617958+0000","last_change":"2024-03-18T20:27:20.617958+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617958+0000","last_undegraded":"2024-03-18T20:27:20.617958+0000","last_fullsized":"2024-03-18T20:27:20.617958+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:52.857794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621100+0000","last_change":"2024-03-18T20:27:20.621100+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621100+0000","last_undegraded":"2024-03-18T20:27:20.621100+0000","last_fullsized":"2024-03-18T20:27:20.621100+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:23:55.315193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617962+0000","last_change":"2024-03-18T20:27:20.617962+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617962+0000","last_undegraded":"2024-03-18T20:27:20.617962+0000","last_fullsized":"2024-03-18T20:27:20.617962+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:01.555443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623634+0000","last_change":"2024-03-18T20:27:20.623634+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623634+0000","last_undegraded":"2024-03-18T20:27:20.623634+0000","last_fullsized":"2024-03-18T20:27:20.623634+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:23:24.136007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621775+0000","last_change":"2024-03-18T20:27:20.621775+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621775+0000","last_undegraded":"2024-03-18T20:27:20.621775+0000","last_fullsized":"2024-03-18T20:27:20.621775+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:54:45.115313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218125+0000","last_change":"2024-03-18T20:27:22.069073+0000","last_active":"2024-03-18T20:27:24.218125+0000","last_peered":"2024-03-18T20:27:24.218125+0000","last_clean":"2024-03-18T20:27:24.218125+0000","last_became_active":"2024-03-18T20:27:22.068744+0000","last_became_peered":"2024-03-18T20:27:22.068744+0000","last_unstale":"2024-03-18T20:27:24.218125+0000","last_undegraded":"2024-03-18T20:27:24.218125+0000","last_fullsized":"2024-03-18T20:27:24.218125+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:44.884087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623503+0000","last_change":"2024-03-18T20:27:20.623503+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623503+0000","last_undegraded":"2024-03-18T20:27:20.623503+0000","last_fullsized":"2024-03-18T20:27:20.623503+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:28.989185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.625365+0000","last_change":"2024-03-18T20:27:20.625365+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.625365+0000","last_undegraded":"2024-03-18T20:27:20.625365+0000","last_fullsized":"2024-03-18T20:27:20.625365+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:43.006246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"259'381","reported_seq":1352,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-18T20:27:18.438231+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:27:18.438231+0000","last_peered":"2024-03-18T20:27:18.438231+0000","last_clean":"2024-03-18T20:27:18.438231+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:27:18.438231+0000","last_undegraded":"2024-03-18T20:27:18.438231+0000","last_fullsized":"2024-03-18T20:27:18.438231+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":64,"acting":64,"num_store_stats":8},{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":1},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576001,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53088,"kb_used_data":3924,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318752,"statfs":{"total":96636764160,"available":96582402048,"internally_reserved":0,"allocated":4018176,"data_stored":2363924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54900000000000004,"15min":0.58999999999999997},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.72599999999999998,"5min":0.80900000000000005,"15min":1.1339999999999999},"last":0.439},{"interface":"front","average":{"1min":0.498,"5min":0.56599999999999995,"15min":0.60199999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":0.73599999999999999,"5min":0.84499999999999997,"15min":1.093},"last":0.56699999999999995}]},{"osd":1,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.55500000000000005,"15min":0.60199999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.748,"5min":0.83099999999999996,"15min":1.4299999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.57099999999999995,"15min":0.60799999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73599999999999999,"5min":0.82399999999999995,"15min":1.2629999999999999},"last":0.61599999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.56100000000000005,"15min":0.60099999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67600000000000005,"5min":0.85499999999999998,"15min":1.216},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57799999999999996,"15min":0.61599999999999999},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75,"5min":1.1279999999999999,"15min":1.2490000000000001},"last":0.59899999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":0.59799999999999998},"min":{"1min":0.38700000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71999999999999997,"5min":0.84699999999999998,"15min":1.4450000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.60699999999999998,"15min":0.64100000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73099999999999998,"5min":0.94599999999999995,"15min":1.5029999999999999},"last":0.63300000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47099999999999997,"15min":0.47799999999999998},"min":{"1min":0.35999999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.68899999999999995,"5min":0.79100000000000004,"15min":0.86099999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48899999999999999,"15min":0.499},"min":{"1min":0.34100000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.66200000000000003,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.54500000000000004}]},{"osd":5,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.51300000000000001,"15min":0.51900000000000002},"min":{"1min":0.373,"5min":0.314,"15min":0.314},"max":{"1min":0.70399999999999996,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.54600000000000004},"min":{"1min":0.38300000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73099999999999998,"5min":0.877,"15min":1.042},"last":0.68799999999999994}]},{"osd":6,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55800000000000005,"15min":0.57699999999999996},"min":{"1min":0.435,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77300000000000002,"5min":0.93500000000000005,"15min":1.129},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.54700000000000004,"15min":0.56899999999999995},"min":{"1min":0.47399999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76200000000000001,"5min":0.91900000000000004,"15min":1.0609999999999999},"last":0.46899999999999997}]}]},{"osd":6,"up_from":13,"seq":55834576001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60700,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311140,"statfs":{"total":96636764160,"available":96574607360,"internally_reserved":0,"allocated":3424256,"data_stored":1773556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53300000000000003,"15min":0.58499999999999996},"min":{"1min":0.35399999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68799999999999994,"5min":0.79800000000000004,"15min":1.1519999999999999},"last":0.505},{"interface":"front","average":{"1min":0.503,"5min":0.54600000000000004,"15min":0.59499999999999997},"min":{"1min":0.33200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.73499999999999999,"5min":1.119,"15min":1.119},"last":0.67100000000000004}]},{"osd":1,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53800000000000003,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73199999999999998,"5min":0.79100000000000004,"15min":1.244},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":0.59499999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":0.70099999999999996,"5min":0.89200000000000002,"15min":1.286},"last":0.54600000000000004}]},{"osd":2,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56399999999999995,"15min":0.61699999999999999},"min":{"1min":0.38700000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.69599999999999995,"5min":0.79600000000000004,"15min":1.276},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.55400000000000005,"15min":0.59899999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.65200000000000002,"5min":1.089,"15min":1.2669999999999999},"last":0.53700000000000003}]},{"osd":3,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55500000000000005,"15min":0.61099999999999999},"min":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71599999999999997,"5min":1.163,"15min":1.163},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57099999999999995,"15min":0.61099999999999999},"min":{"1min":0.47499999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78100000000000003,"5min":0.82399999999999995,"15min":1.337},"last":0.55500000000000005}]},{"osd":4,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.35299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80200000000000005,"5min":0.85799999999999998,"15min":0.88300000000000001},"last":0.378},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.50600000000000001,"15min":0.51800000000000002},"min":{"1min":0.32500000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83399999999999996,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.40200000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.50900000000000001,"15min":0.51200000000000001},"min":{"1min":0.42599999999999999,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":0.65000000000000002,"5min":1.008,"15min":1.008},"last":0.441},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55900000000000005,"15min":0.55300000000000005},"min":{"1min":0.41799999999999998,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.755,"5min":0.92600000000000005,"15min":0.94199999999999995},"last":0.43099999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56699999999999995,"15min":0.56299999999999994},"min":{"1min":0.40999999999999998,"5min":0.38,"15min":0.34499999999999997},"max":{"1min":0.91200000000000003,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.34599999999999997},"max":{"1min":0.86699999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.52000000000000002}]}]},{"osd":1,"up_from":13,"seq":55834576002,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52548,"kb_used_data":3384,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319292,"statfs":{"total":96636764160,"available":96582955008,"internally_reserved":0,"allocated":3465216,"data_stored":1794256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.38700000000000001,"15min":0.39300000000000002},"min":{"1min":0.255,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.46400000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39500000000000002,"15min":0.41999999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.623,"5min":0.623,"15min":0.69699999999999995},"last":0.39900000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.40899999999999997,"15min":0.42899999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53100000000000003,"5min":0.61199999999999999,"15min":0.68100000000000005},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.45200000000000001,"15min":0.47099999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.69799999999999995,"5min":0.70099999999999996,"15min":0.80500000000000005},"last":0.34799999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.46500000000000002,"15min":0.48399999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.76100000000000001},"last":0.44},{"interface":"front","average":{"1min":0.372,"5min":0.41899999999999998,"15min":0.433},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.27000000000000002},"max":{"1min":0.54700000000000004,"5min":0.68200000000000005,"15min":0.71699999999999997},"last":0.32400000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.64000000000000001,"15min":0.66500000000000004},"min":{"1min":0.43099999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76800000000000002,"5min":1.0680000000000001,"15min":1.2909999999999999},"last":0.629},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.64600000000000002,"15min":0.67100000000000004},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.88500000000000001,"5min":1.1160000000000001,"15min":1.746},"last":0.55300000000000005}]},{"osd":5,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.63400000000000001,"15min":0.66100000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79300000000000004,"5min":1.0389999999999999,"15min":1.6910000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.67800000000000005,"15min":0.70499999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.85199999999999998,"5min":1.109,"15min":1.2549999999999999},"last":0.59199999999999997}]},{"osd":6,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.65300000000000002,"15min":0.67700000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.70799999999999996,"5min":1.0289999999999999,"15min":1.637},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.67800000000000005,"15min":0.70199999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82099999999999995,"5min":1.014,"15min":1.722},"last":0.52400000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.66000000000000003,"15min":0.69899999999999995},"min":{"1min":0.46100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91100000000000003,"5min":0.99199999999999999,"15min":1.2729999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.67000000000000004,"15min":0.69699999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.80700000000000005,"5min":1.0329999999999999,"15min":1.3380000000000001},"last":0.66900000000000004}]}]},{"osd":0,"up_from":13,"seq":55834576002,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48992,"kb_used_data":3924,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322848,"statfs":{"total":96636764160,"available":96586596352,"internally_reserved":0,"allocated":4018176,"data_stored":2363924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.36399999999999999,"15min":0.38600000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.46100000000000002,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.39600000000000002,"15min":0.41999999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.58799999999999997,"5min":0.60099999999999998,"15min":1.1850000000000001},"last":0.255}]},{"osd":2,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.41299999999999998,"15min":0.433},"min":{"1min":0.28699999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.47199999999999998,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.41799999999999998,"15min":0.44600000000000001},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.54500000000000004,"5min":0.64800000000000002,"15min":0.70999999999999996},"last":0.39400000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.39600000000000002,"15min":0.42399999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.433,"5min":0.58499999999999996,"15min":0.628},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.45400000000000001,"15min":0.48699999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.52600000000000002,"5min":0.73399999999999999,"15min":0.91500000000000004},"last":0.379}]},{"osd":4,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.61299999999999999,"15min":0.64400000000000002},"min":{"1min":0.41299999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.752,"5min":0.91500000000000004,"15min":1.0029999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.64800000000000002,"15min":0.65900000000000003},"min":{"1min":0.51600000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.92300000000000004,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.72099999999999997}]},{"osd":5,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.64400000000000002,"15min":0.66600000000000004},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83299999999999996,"5min":0.92300000000000004,"15min":1.0129999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65300000000000002,"15min":0.67500000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.73199999999999998,"5min":0.96199999999999997,"15min":1.518},"last":0.70499999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.64600000000000002,"15min":0.65700000000000003},"min":{"1min":0.45800000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.73599999999999999,"5min":1.0669999999999999,"15min":1.1279999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.68200000000000005,"15min":0.70299999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.871,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.746}]},{"osd":7,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.67200000000000004,"15min":0.69899999999999995},"min":{"1min":0.45300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77500000000000002,"5min":1.008,"15min":1.464},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.67300000000000004,"15min":0.69799999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88900000000000001,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.76300000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52508,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319332,"statfs":{"total":96636764160,"available":96582995968,"internally_reserved":0,"allocated":3424256,"data_stored":1773556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.34699999999999998,"15min":0.371},"min":{"1min":0.20799999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.53800000000000003,"5min":0.54800000000000004,"15min":0.57499999999999996},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.44700000000000001,"15min":0.47799999999999998},"min":{"1min":0.26100000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60499999999999998,"5min":0.71499999999999997,"15min":0.753},"last":0.48999999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.44400000000000001,"15min":0.47399999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57399999999999995,"5min":0.64300000000000002,"15min":0.749},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.38900000000000001,"15min":0.40899999999999997},"min":{"1min":0.246,"5min":0.224,"15min":0.224},"max":{"1min":0.501,"5min":0.55900000000000005,"15min":0.59999999999999998},"last":0.32700000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.41199999999999998,"15min":0.42899999999999999},"min":{"1min":0.27800000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.56100000000000005,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.47399999999999998,"15min":0.50700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.63200000000000001,"5min":0.68100000000000005,"15min":0.752},"last":0.45400000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63900000000000001,"15min":0.65300000000000002},"min":{"1min":0.41799999999999998,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":1.2090000000000001,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.65700000000000003,"15min":0.66200000000000003},"min":{"1min":0.45100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.268,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.61899999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.64900000000000002,"15min":0.67500000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79900000000000004,"5min":0.97299999999999998,"15min":1.028},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.63900000000000001,"15min":0.67000000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.77100000000000002,"5min":0.88300000000000001,"15min":1.0629999999999999},"last":0.80800000000000005}]},{"osd":6,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.65500000000000003,"15min":0.68100000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.81899999999999995,"5min":1.014,"15min":1.2050000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.72699999999999998},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.3220000000000001,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.77000000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.65500000000000003,"15min":0.68899999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.42199999999999999},"max":{"1min":0.73699999999999999,"5min":0.99399999999999999,"15min":1.0269999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68600000000000005,"15min":0.70099999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.2330000000000001,"5min":1.371,"15min":1.371},"last":0.79100000000000004}]}]},{"osd":3,"up_from":13,"seq":55834576002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52508,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319332,"statfs":{"total":96636764160,"available":96582995968,"internally_reserved":0,"allocated":3424256,"data_stored":1773556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.36099999999999999,"15min":0.39000000000000001},"min":{"1min":0.247,"5min":0.223,"15min":0.223},"max":{"1min":0.46400000000000002,"5min":0.66500000000000004,"15min":0.98099999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.373,"15min":0.40400000000000003},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.45600000000000002,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.42399999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.35999999999999999,"15min":0.39700000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.44500000000000001,"5min":0.63200000000000001,"15min":0.64000000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.42199999999999999,"15min":0.45600000000000002},"min":{"1min":0.33400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.501,"5min":0.65500000000000003,"15min":0.73099999999999998},"last":0.44500000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.41499999999999998,"15min":0.45200000000000001},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.54900000000000004,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.437,"15min":0.48099999999999998},"min":{"1min":0.32600000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.52300000000000002,"5min":0.64400000000000002,"15min":0.71199999999999997},"last":0.32600000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59499999999999997,"15min":0.64000000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.526,"5min":1.526,"15min":1.526},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61399999999999999,"15min":0.66100000000000003},"min":{"1min":0.39200000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.65800000000000003,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.65800000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.65100000000000002,"15min":0.68700000000000006},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.66500000000000004,"15min":0.71499999999999997},"min":{"1min":0.45300000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78000000000000003,"5min":1.1120000000000001,"15min":1.171},"last":0.65000000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65800000000000003,"15min":0.69699999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.62},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.71999999999999997},"min":{"1min":0.48699999999999999,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.58999999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.64900000000000002,"15min":0.70299999999999996},"min":{"1min":0.43099999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79900000000000004,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64300000000000002,"15min":0.68899999999999995},"min":{"1min":0.51500000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.57299999999999995}]}]},{"osd":4,"up_from":13,"seq":55834576002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48400,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323440,"statfs":{"total":96636764160,"available":96587202560,"internally_reserved":0,"allocated":3411968,"data_stored":1767319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57799999999999996,"15min":0.58899999999999997},"min":{"1min":0.39600000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56799999999999995,"15min":0.59499999999999997},"min":{"1min":0.40100000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.84199999999999997},"last":0.621}]},{"osd":1,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.79300000000000004,"5min":0.81899999999999995,"15min":0.88300000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56399999999999995,"15min":0.57999999999999996},"min":{"1min":0.34899999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.96499999999999997},"last":0.58999999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59799999999999998,"15min":0.61399999999999999},"min":{"1min":0.42899999999999999,"5min":0.41299999999999998,"15min":0.38500000000000001},"max":{"1min":0.70799999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.54500000000000004}]},{"osd":3,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.59299999999999997,"15min":0.60199999999999998},"min":{"1min":0.433,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78100000000000003,"5min":1.026,"15min":1.026},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.41399999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.72999999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.48399999999999999,"15min":0.48199999999999998},"min":{"1min":0.318,"5min":0.309,"15min":0.309},"max":{"1min":0.76700000000000002,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.747},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.496,"15min":0.505},"min":{"1min":0.38300000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.81499999999999995,"5min":0.85499999999999998,"15min":0.90600000000000003},"last":0.63800000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.502,"15min":0.50600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.754,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.82799999999999996,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.71399999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.40600000000000003,"5min":0.38800000000000001,"15min":0.35699999999999998},"max":{"1min":0.80300000000000005,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.38500000000000001,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.84099999999999997,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.65200000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48412,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323428,"statfs":{"total":96636764160,"available":96587190272,"internally_reserved":0,"allocated":3424256,"data_stored":1773556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.55400000000000005,"15min":0.59099999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.31900000000000001},"max":{"1min":0.76300000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.621},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55600000000000005,"15min":0.59499999999999997},"min":{"1min":0.42099999999999999,"5min":0.35599999999999998,"15min":0.33100000000000002},"max":{"1min":0.81100000000000005,"5min":0.82599999999999996,"15min":1.3100000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.56100000000000005,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71499999999999997,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.55400000000000005,"15min":0.59199999999999997},"min":{"1min":0.372,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.69399999999999995,"5min":0.94299999999999995,"15min":1.018},"last":0.58499999999999996}]},{"osd":2,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.59399999999999997},"min":{"1min":0.40300000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.75600000000000001,"5min":0.85299999999999998,"15min":0.99299999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.55400000000000005,"15min":0.60099999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77000000000000002,"5min":1.599,"15min":1.599},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57499999999999996,"15min":0.60399999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72899999999999998,"5min":0.98699999999999999,"15min":1.2430000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59299999999999997,"15min":0.63700000000000001},"min":{"1min":0.42099999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":0.84399999999999997,"15min":1.161},"last":0.63300000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48999999999999999,"15min":0.48299999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.625,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.35699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.40699999999999997}]},{"osd":6,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.503,"15min":0.50900000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.32800000000000001},"max":{"1min":0.64800000000000002,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54300000000000004,"15min":0.54200000000000004},"min":{"1min":0.35199999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.64600000000000002,"5min":1.643,"15min":1.643},"last":0.501}]},{"osd":7,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.40200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.71299999999999997,"5min":1.028,"15min":1.028},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57099999999999995,"15min":0.57799999999999996},"min":{"1min":0.42599999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74299999999999999,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:27:27.079 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.3f is not active+clean 2024-03-18T20:27:27.079 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.080 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.3e is not active+clean 2024-03-18T20:27:27.080 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.080 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.3d is not active+clean 2024-03-18T20:27:27.080 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.080 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.3c is not active+clean 2024-03-18T20:27:27.080 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.080 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.39 is not active+clean 2024-03-18T20:27:27.080 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.080 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.38 is not active+clean 2024-03-18T20:27:27.080 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.080 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.37 is not active+clean 2024-03-18T20:27:27.080 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.081 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.36 is not active+clean 2024-03-18T20:27:27.081 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.081 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.35 is not active+clean 2024-03-18T20:27:27.081 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.081 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.34 is not active+clean 2024-03-18T20:27:27.081 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.081 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.33 is not active+clean 2024-03-18T20:27:27.081 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.081 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.32 is not active+clean 2024-03-18T20:27:27.081 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.081 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.31 is not active+clean 2024-03-18T20:27:27.081 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.082 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.30 is not active+clean 2024-03-18T20:27:27.082 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.082 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.2f is not active+clean 2024-03-18T20:27:27.082 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.082 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.2e is not active+clean 2024-03-18T20:27:27.082 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.082 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.2d is not active+clean 2024-03-18T20:27:27.082 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.082 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.2c is not active+clean 2024-03-18T20:27:27.082 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.082 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.2b is not active+clean 2024-03-18T20:27:27.083 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.083 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.2a is not active+clean 2024-03-18T20:27:27.083 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.083 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.29 is not active+clean 2024-03-18T20:27:27.083 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.083 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.28 is not active+clean 2024-03-18T20:27:27.083 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.083 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.27 is not active+clean 2024-03-18T20:27:27.083 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.083 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.23 is not active+clean 2024-03-18T20:27:27.083 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.083 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.22 is not active+clean 2024-03-18T20:27:27.084 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.084 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.21 is not active+clean 2024-03-18T20:27:27.084 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.084 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.20 is not active+clean 2024-03-18T20:27:27.084 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.084 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.18 is not active+clean 2024-03-18T20:27:27.084 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.084 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.1f is not active+clean 2024-03-18T20:27:27.084 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.622037+0000', 'last_change': '2024-03-18T20:27:20.622037+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.622037+0000', 'last_undegraded': '2024-03-18T20:27:20.622037+0000', 'last_fullsized': '2024-03-18T20:27:20.622037+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T05:16:01.698353+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-18T20:27:27.084 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.19 is not active+clean 2024-03-18T20:27:27.084 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.085 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.1e is not active+clean 2024-03-18T20:27:27.085 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.621734+0000', 'last_change': '2024-03-18T20:27:20.621734+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.621734+0000', 'last_undegraded': '2024-03-18T20:27:20.621734+0000', 'last_fullsized': '2024-03-18T20:27:20.621734+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-19T22:02:14.143177+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-18T20:27:27.085 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.1a is not active+clean 2024-03-18T20:27:27.085 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.085 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.1d is not active+clean 2024-03-18T20:27:27.085 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.622565+0000', 'last_change': '2024-03-18T20:27:20.622565+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.622565+0000', 'last_undegraded': '2024-03-18T20:27:20.622565+0000', 'last_fullsized': '2024-03-18T20:27:20.622565+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-19T23:42:23.110707+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-18T20:27:27.085 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.1b is not active+clean 2024-03-18T20:27:27.085 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.085 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.1c is not active+clean 2024-03-18T20:27:27.085 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.622373+0000', 'last_change': '2024-03-18T20:27:20.622373+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.622373+0000', 'last_undegraded': '2024-03-18T20:27:20.622373+0000', 'last_fullsized': '2024-03-18T20:27:20.622373+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T04:45:52.316434+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-18T20:27:27.085 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.1c is not active+clean 2024-03-18T20:27:27.086 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.086 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.1b is not active+clean 2024-03-18T20:27:27.086 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.617958+0000', 'last_change': '2024-03-18T20:27:20.617958+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.617958+0000', 'last_undegraded': '2024-03-18T20:27:20.617958+0000', 'last_fullsized': '2024-03-18T20:27:20.617958+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-19T23:08:18.189309+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-18T20:27:27.086 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.1d is not active+clean 2024-03-18T20:27:27.086 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.086 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.1e is not active+clean 2024-03-18T20:27:27.086 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.086 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.19 is not active+clean 2024-03-18T20:27:27.086 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.623987+0000', 'last_change': '2024-03-18T20:27:20.623987+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.623987+0000', 'last_undegraded': '2024-03-18T20:27:20.623987+0000', 'last_fullsized': '2024-03-18T20:27:20.623987+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-19T23:08:18.189309+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-18T20:27:27.086 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.1f is not active+clean 2024-03-18T20:27:27.086 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.18 is not active+clean 2024-03-18T20:27:27.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.621015+0000', 'last_change': '2024-03-18T20:27:20.621015+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.621015+0000', 'last_undegraded': '2024-03-18T20:27:20.621015+0000', 'last_fullsized': '2024-03-18T20:27:20.621015+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T00:49:10.342403+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-18T20:27:27.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.10 is not active+clean 2024-03-18T20:27:27.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.17 is not active+clean 2024-03-18T20:27:27.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.623746+0000', 'last_change': '2024-03-18T20:27:20.623746+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.623746+0000', 'last_undegraded': '2024-03-18T20:27:20.623746+0000', 'last_fullsized': '2024-03-18T20:27:20.623746+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-19T21:28:52.874751+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-18T20:27:27.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.11 is not active+clean 2024-03-18T20:27:27.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.16 is not active+clean 2024-03-18T20:27:27.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.623648+0000', 'last_change': '2024-03-18T20:27:20.623648+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.623648+0000', 'last_undegraded': '2024-03-18T20:27:20.623648+0000', 'last_fullsized': '2024-03-18T20:27:20.623648+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T03:21:19.006472+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-18T20:27:27.087 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.12 is not active+clean 2024-03-18T20:27:27.087 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.15 is not active+clean 2024-03-18T20:27:27.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.624317+0000', 'last_change': '2024-03-18T20:27:20.624317+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.624317+0000', 'last_undegraded': '2024-03-18T20:27:20.624317+0000', 'last_fullsized': '2024-03-18T20:27:20.624317+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T02:43:29.812382+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-18T20:27:27.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.14 is not active+clean 2024-03-18T20:27:27.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.621040+0000', 'last_change': '2024-03-18T20:27:20.621040+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.621040+0000', 'last_undegraded': '2024-03-18T20:27:20.621040+0000', 'last_fullsized': '2024-03-18T20:27:20.621040+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-19T23:01:31.379909+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-18T20:27:27.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.14 is not active+clean 2024-03-18T20:27:27.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.13 is not active+clean 2024-03-18T20:27:27.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.625530+0000', 'last_change': '2024-03-18T20:27:20.625530+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.625530+0000', 'last_undegraded': '2024-03-18T20:27:20.625530+0000', 'last_fullsized': '2024-03-18T20:27:20.625530+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T08:15:42.339058+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-18T20:27:27.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.15 is not active+clean 2024-03-18T20:27:27.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.12 is not active+clean 2024-03-18T20:27:27.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.617987+0000', 'last_change': '2024-03-18T20:27:20.617987+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.617987+0000', 'last_undegraded': '2024-03-18T20:27:20.617987+0000', 'last_fullsized': '2024-03-18T20:27:20.617987+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T08:15:42.339058+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-18T20:27:27.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.16 is not active+clean 2024-03-18T20:27:27.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.11 is not active+clean 2024-03-18T20:27:27.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.617911+0000', 'last_change': '2024-03-18T20:27:20.617911+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.617911+0000', 'last_undegraded': '2024-03-18T20:27:20.617911+0000', 'last_fullsized': '2024-03-18T20:27:20.617911+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-19T23:16:43.006246+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-18T20:27:27.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.17 is not active+clean 2024-03-18T20:27:27.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.10 is not active+clean 2024-03-18T20:27:27.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.621574+0000', 'last_change': '2024-03-18T20:27:20.621574+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.621574+0000', 'last_undegraded': '2024-03-18T20:27:20.621574+0000', 'last_fullsized': '2024-03-18T20:27:20.621574+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T04:31:56.485139+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-18T20:27:27.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.f is not active+clean 2024-03-18T20:27:27.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.621698+0000', 'last_change': '2024-03-18T20:27:20.621698+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.621698+0000', 'last_undegraded': '2024-03-18T20:27:20.621698+0000', 'last_fullsized': '2024-03-18T20:27:20.621698+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T07:17:27.485874+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-18T20:27:27.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.9 is not active+clean 2024-03-18T20:27:27.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.d is not active+clean 2024-03-18T20:27:27.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.621611+0000', 'last_change': '2024-03-18T20:27:20.621611+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.621611+0000', 'last_undegraded': '2024-03-18T20:27:20.621611+0000', 'last_fullsized': '2024-03-18T20:27:20.621611+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T03:58:00.382684+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-18T20:27:27.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.b is not active+clean 2024-03-18T20:27:27.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.c is not active+clean 2024-03-18T20:27:27.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.623890+0000', 'last_change': '2024-03-18T20:27:20.623890+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.623890+0000', 'last_undegraded': '2024-03-18T20:27:20.623890+0000', 'last_fullsized': '2024-03-18T20:27:20.623890+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T06:30:01.555443+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-18T20:27:27.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.c is not active+clean 2024-03-18T20:27:27.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.b is not active+clean 2024-03-18T20:27:27.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.624331+0000', 'last_change': '2024-03-18T20:27:20.624331+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.624331+0000', 'last_undegraded': '2024-03-18T20:27:20.624331+0000', 'last_fullsized': '2024-03-18T20:27:20.624331+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-19T21:05:52.857794+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-18T20:27:27.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.d is not active+clean 2024-03-18T20:27:27.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.a is not active+clean 2024-03-18T20:27:27.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.622224+0000', 'last_change': '2024-03-18T20:27:20.622224+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.622224+0000', 'last_undegraded': '2024-03-18T20:27:20.622224+0000', 'last_fullsized': '2024-03-18T20:27:20.622224+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-19T23:13:18.842477+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-18T20:27:27.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.e is not active+clean 2024-03-18T20:27:27.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.9 is not active+clean 2024-03-18T20:27:27.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.621046+0000', 'last_change': '2024-03-18T20:27:20.621046+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.621046+0000', 'last_undegraded': '2024-03-18T20:27:20.621046+0000', 'last_fullsized': '2024-03-18T20:27:20.621046+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T01:11:16.795077+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-18T20:27:27.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.8 is not active+clean 2024-03-18T20:27:27.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.625820+0000', 'last_change': '2024-03-18T20:27:20.625820+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.625820+0000', 'last_undegraded': '2024-03-18T20:27:20.625820+0000', 'last_fullsized': '2024-03-18T20:27:20.625820+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T05:41:34.677506+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-18T20:27:27.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.0 is not active+clean 2024-03-18T20:27:27.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.7 is not active+clean 2024-03-18T20:27:27.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.624685+0000', 'last_change': '2024-03-18T20:27:20.624685+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.624685+0000', 'last_undegraded': '2024-03-18T20:27:20.624685+0000', 'last_fullsized': '2024-03-18T20:27:20.624685+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T03:42:45.164373+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-18T20:27:27.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.1 is not active+clean 2024-03-18T20:27:27.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.6 is not active+clean 2024-03-18T20:27:27.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.617958+0000', 'last_change': '2024-03-18T20:27:20.617958+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.617958+0000', 'last_undegraded': '2024-03-18T20:27:20.617958+0000', 'last_fullsized': '2024-03-18T20:27:20.617958+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-19T21:05:52.857794+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-18T20:27:27.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.2 is not active+clean 2024-03-18T20:27:27.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.5 is not active+clean 2024-03-18T20:27:27.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.621100+0000', 'last_change': '2024-03-18T20:27:20.621100+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.621100+0000', 'last_undegraded': '2024-03-18T20:27:20.621100+0000', 'last_fullsized': '2024-03-18T20:27:20.621100+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T07:23:55.315193+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-18T20:27:27.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.3 is not active+clean 2024-03-18T20:27:27.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.4 is not active+clean 2024-03-18T20:27:27.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.617962+0000', 'last_change': '2024-03-18T20:27:20.617962+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.617962+0000', 'last_undegraded': '2024-03-18T20:27:20.617962+0000', 'last_fullsized': '2024-03-18T20:27:20.617962+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T06:30:01.555443+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-18T20:27:27.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.4 is not active+clean 2024-03-18T20:27:27.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.3 is not active+clean 2024-03-18T20:27:27.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.623634+0000', 'last_change': '2024-03-18T20:27:20.623634+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.623634+0000', 'last_undegraded': '2024-03-18T20:27:20.623634+0000', 'last_fullsized': '2024-03-18T20:27:20.623634+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-19T22:23:24.136007+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-18T20:27:27.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.5 is not active+clean 2024-03-18T20:27:27.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.2 is not active+clean 2024-03-18T20:27:27.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.621775+0000', 'last_change': '2024-03-18T20:27:20.621775+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.621775+0000', 'last_undegraded': '2024-03-18T20:27:20.621775+0000', 'last_fullsized': '2024-03-18T20:27:20.621775+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-19T23:54:45.115313+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-18T20:27:27.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.1 is not active+clean 2024-03-18T20:27:27.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.623503+0000', 'last_change': '2024-03-18T20:27:20.623503+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.623503+0000', 'last_undegraded': '2024-03-18T20:27:20.623503+0000', 'last_fullsized': '2024-03-18T20:27:20.623503+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-20T06:43:28.989185+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-18T20:27:27.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.7 is not active+clean 2024-03-18T20:27:27.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 35.0 is not active+clean 2024-03-18T20:27:27.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '35.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 271, 'state': 'creating+peering', 'last_fresh': '2024-03-18T20:27:20.625365+0000', 'last_change': '2024-03-18T20:27:20.625365+0000', 'last_active': '2024-03-18T20:27:20.607510+0000', 'last_peered': '2024-03-18T20:27:20.607510+0000', 'last_clean': '2024-03-18T20:27:20.607510+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:20.625365+0000', 'last_undegraded': '2024-03-18T20:27:20.625365+0000', 'last_fullsized': '2024-03-18T20:27:20.625365+0000', 'mapping_epoch': 271, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 271, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:20.607510+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-19T23:16:43.006246+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-18T20:27:27.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 36.25 is not active+clean 2024-03-18T20:27:27.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '36.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-18T20:27:21.614682+0000', 'last_change': '2024-03-18T20:27:21.614682+0000', 'last_active': '2024-03-18T20:27:21.614682+0000', 'last_peered': '2024-03-18T20:27:21.614682+0000', 'last_clean': '2024-03-18T20:27:21.614682+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-18T20:27:21.614682+0000', 'last_undegraded': '2024-03-18T20:27:21.614682+0000', 'last_fullsized': '2024-03-18T20:27:21.614682+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'last_clean_scrub_stamp': '2024-03-18T20:27:21.614682+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-18T20:27:27.095 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-18T20:27:27.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.234+0000 7f304cb35700 1 Processor -- start 2024-03-18T20:27:27.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.234+0000 7f304cb35700 1 -- start start 2024-03-18T20:27:27.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.235+0000 7f304cb35700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f30481465a0 0x7f3048168660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:27.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.235+0000 7f304cb35700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3048072660 con 0x7f3048146190 2024-03-18T20:27:27.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.235+0000 7f304cb35700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304806ebd0 con 0x7f304806ae60 2024-03-18T20:27:27.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.235+0000 7f304cb35700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f304806f850 con 0x7f30481465a0 2024-03-18T20:27:27.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.235+0000 7f304659c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3048146190 0x7f304806a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37458/0 (socket says 172.21.15.1:37458) 2024-03-18T20:27:27.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.235+0000 7f304659c700 1 -- 172.21.15.1:0/1656043422 learned_addr learned my addr 172.21.15.1:0/1656043422 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:27.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f3046d9d700 1 --2- 172.21.15.1:0/1656043422 >> v2:172.21.15.165:3300/0 conn(0x7f30481465a0 0x7f3048168660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:27.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f304559a700 1 -- 172.21.15.1:0/1656043422 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618134991 0 0) 0x7f3048072660 con 0x7f3048146190 2024-03-18T20:27:27.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f304559a700 1 -- 172.21.15.1:0/1656043422 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30340035b0 con 0x7f3048146190 2024-03-18T20:27:27.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f304559a700 1 -- 172.21.15.1:0/1656043422 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1890839349 0 0) 0x7f30340035b0 con 0x7f3048146190 2024-03-18T20:27:27.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f304559a700 1 -- 172.21.15.1:0/1656043422 >> v2:172.21.15.165:3300/0 conn(0x7f30481465a0 msgr2=0x7f3048168660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:27.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f304559a700 1 --2- 172.21.15.1:0/1656043422 >> v2:172.21.15.165:3300/0 conn(0x7f30481465a0 0x7f3048168660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f304559a700 1 -- 172.21.15.1:0/1656043422 >> v1:172.21.15.165:6789/0 conn(0x7f304806ae60 legacy=0x7f3048164f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:27.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f304559a700 1 -- 172.21.15.1:0/1656043422 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304806eeb0 con 0x7f3048146190 2024-03-18T20:27:27.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f304559a700 1 -- 172.21.15.1:0/1656043422 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f3038001670 con 0x7f3048146190 2024-03-18T20:27:27.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f304559a700 1 -- 172.21.15.1:0/1656043422 >> v1:172.21.15.1:6789/0 conn(0x7f3048146190 legacy=0x7f304806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:27.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f304559a700 1 --2- 172.21.15.1:0/1656043422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3034003a00 0x7f3034003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:27.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f304559a700 1 --2- 172.21.15.1:0/1656043422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3034004850 0x7f3034004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:27.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f304559a700 1 --2- 172.21.15.1:0/1656043422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3034005210 0x7f303400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:27.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f304559a700 1 -- 172.21.15.1:0/1656043422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304806f850 con 0x7f3034004850 2024-03-18T20:27:27.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f304559a700 1 -- 172.21.15.1:0/1656043422 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303400c0b0 con 0x7f3034003a00 2024-03-18T20:27:27.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f304559a700 1 -- 172.21.15.1:0/1656043422 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f303400c270 con 0x7f3034005210 2024-03-18T20:27:27.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f3045d9b700 1 --2- 172.21.15.1:0/1656043422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3034004850 0x7f3034004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:27.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f304659c700 1 --2- 172.21.15.1:0/1656043422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3034003a00 0x7f3034003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:27.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f3046d9d700 1 --2- 172.21.15.1:0/1656043422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3034005210 0x7f303400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:27.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f3045d9b700 1 -- 172.21.15.1:0/1656043422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3034005210 msgr2=0x7f303400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:27.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f3045d9b700 1 --2- 172.21.15.1:0/1656043422 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3034005210 0x7f303400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f3045d9b700 1 -- 172.21.15.1:0/1656043422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3034003a00 msgr2=0x7f3034003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:27.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f3045d9b700 1 --2- 172.21.15.1:0/1656043422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3034003a00 0x7f3034003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f3045d9b700 1 -- 172.21.15.1:0/1656043422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30340035b0 con 0x7f3034004850 2024-03-18T20:27:27.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f304659c700 1 --2- 172.21.15.1:0/1656043422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3034003a00 0x7f3034003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:27.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.236+0000 7f3046d9d700 1 -- 172.21.15.1:0/1656043422 reap_dead start 2024-03-18T20:27:27.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.237+0000 7f3045d9b700 1 --2- 172.21.15.1:0/1656043422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3034004850 0x7f3034004cd0 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 crypto rx=0x7f30300017c0 tx=0x7f3034005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df96baad431c347d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:27.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.237+0000 7f304559a700 1 -- 172.21.15.1:0/1656043422 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30300052d0 con 0x7f3034004850 2024-03-18T20:27:27.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.237+0000 7f304559a700 1 -- 172.21.15.1:0/1656043422 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3030005430 con 0x7f3034004850 2024-03-18T20:27:27.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.237+0000 7f304559a700 1 -- 172.21.15.1:0/1656043422 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30300038c0 con 0x7f3034004850 2024-03-18T20:27:27.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.237+0000 7f304cb35700 1 -- 172.21.15.1:0/1656043422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3034004850 msgr2=0x7f3034004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:27.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.237+0000 7f304cb35700 1 --2- 172.21.15.1:0/1656043422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3034004850 0x7f3034004cd0 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 crypto rx=0x7f30300017c0 tx=0x7f3034005880 comp rx=0 tx=0).stop 2024-03-18T20:27:27.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.237+0000 7f304cb35700 1 -- 172.21.15.1:0/1656043422 shutdown_connections 2024-03-18T20:27:27.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.237+0000 7f304cb35700 1 --2- 172.21.15.1:0/1656043422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3034004850 0x7f3034004cd0 unknown :-1 s=CLOSED pgs=3083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.237+0000 7f304cb35700 1 -- 172.21.15.1:0/1656043422 >> 172.21.15.1:0/1656043422 conn(0x7f30480b1d20 msgr2=0x7f30480b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:27.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.237+0000 7f304cb35700 1 -- 172.21.15.1:0/1656043422 shutdown_connections 2024-03-18T20:27:27.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.237+0000 7f304cb35700 1 -- 172.21.15.1:0/1656043422 wait complete. 2024-03-18T20:27:27.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.239+0000 7f304cb35700 1 Processor -- start 2024-03-18T20:27:27.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.239+0000 7f304cb35700 1 -- start start 2024-03-18T20:27:27.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.240+0000 7f304cb35700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3034004850 0x7f30481f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:27.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.240+0000 7f304cb35700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30481e89a0 0x7f304813ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:27.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.240+0000 7f304cb35700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30481e8e20 0x7f30481404c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:27.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.240+0000 7f304cb35700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3048140b00 con 0x7f30481e8e20 2024-03-18T20:27:27.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.240+0000 7f304cb35700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3048140de0 con 0x7f3034004850 2024-03-18T20:27:27.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.240+0000 7f304cb35700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30481410c0 con 0x7f30481e89a0 2024-03-18T20:27:27.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.240+0000 7f3046d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30481e8e20 0x7f30481404c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:27.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.240+0000 7f3046d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30481e8e20 0x7f30481404c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36542/0 (socket says 172.21.15.1:36542) 2024-03-18T20:27:27.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.240+0000 7f3046d9d700 1 -- 172.21.15.1:0/1870030634 learned_addr learned my addr 172.21.15.1:0/1870030634 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:27.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.240+0000 7f304659c700 1 --2- 172.21.15.1:0/1870030634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3034004850 0x7f30481f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:27.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.240+0000 7f3045d9b700 1 --2- 172.21.15.1:0/1870030634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30481e89a0 0x7f304813ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:27.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.240+0000 7f3046d9d700 1 -- 172.21.15.1:0/1870030634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30481e89a0 msgr2=0x7f304813ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:27.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.240+0000 7f3046d9d700 1 --2- 172.21.15.1:0/1870030634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30481e89a0 0x7f304813ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.240+0000 7f3046d9d700 1 -- 172.21.15.1:0/1870030634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3034004850 msgr2=0x7f30481f31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:27.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.240+0000 7f3046d9d700 1 --2- 172.21.15.1:0/1870030634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3034004850 0x7f30481f31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.240+0000 7f3046d9d700 1 -- 172.21.15.1:0/1870030634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30481413a0 con 0x7f30481e8e20 2024-03-18T20:27:27.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.241+0000 7f3046d9d700 1 --2- 172.21.15.1:0/1870030634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30481e8e20 0x7f30481404c0 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 crypto rx=0x7f30480b62b0 tx=0x7f30400114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:27.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.241+0000 7f303effd700 1 -- 172.21.15.1:0/1870030634 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304000e070 con 0x7f30481e8e20 2024-03-18T20:27:27.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.241+0000 7f303effd700 1 -- 172.21.15.1:0/1870030634 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3040003c30 con 0x7f30481e8e20 2024-03-18T20:27:27.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.241+0000 7f304cb35700 1 -- 172.21.15.1:0/1870030634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3048141680 con 0x7f30481e8e20 2024-03-18T20:27:27.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.241+0000 7f303effd700 1 -- 172.21.15.1:0/1870030634 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040003e20 con 0x7f30481e8e20 2024-03-18T20:27:27.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.241+0000 7f304cb35700 1 -- 172.21.15.1:0/1870030634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30481f3720 con 0x7f30481e8e20 2024-03-18T20:27:27.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.242+0000 7f304cb35700 1 -- 172.21.15.1:0/1870030634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3048141870 con 0x7f30481e8e20 2024-03-18T20:27:27.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.242+0000 7f303effd700 1 -- 172.21.15.1:0/1870030634 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f3040011a10 con 0x7f30481e8e20 2024-03-18T20:27:27.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.243+0000 7f303effd700 1 --2- 172.21.15.1:0/1870030634 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302806aed0 0x7f302806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:27.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.243+0000 7f303effd700 1 -- 172.21.15.1:0/1870030634 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f304008a0d0 con 0x7f30481e8e20 2024-03-18T20:27:27.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.243+0000 7f304659c700 1 --2- 172.21.15.1:0/1870030634 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302806aed0 0x7f302806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:27.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.243+0000 7f304659c700 1 --2- 172.21.15.1:0/1870030634 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302806aed0 0x7f302806d390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3038001c90 tx=0x7f3038009040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:27.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.247+0000 7f303effd700 1 -- 172.21.15.1:0/1870030634 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3040054c00 con 0x7f30481e8e20 2024-03-18T20:27:27.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.537+0000 7f304cb35700 1 -- 172.21.15.1:0/1870030634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f30480676a0 con 0x7f30481e8e20 2024-03-18T20:27:27.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.538+0000 7f303effd700 1 -- 172.21.15.1:0/1870030634 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+3372 (secure 0 0 0) 0x7f3040058220 con 0x7f30481e8e20 2024-03-18T20:27:27.543 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:27.543 INFO:teuthology.orchestra.run.smithi001.stdout:{"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":5880,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":274,"num_osds":8,"num_up_osds":8,"osd_up_since":1710787777,"num_in_osds":8,"osd_in_since":1710787769,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":55},{"state_name":"creating+peering","count":30},{"state_name":"active+clean","count":12}],"num_pgs":97,"num_pools":3,"num_objects":3,"data_bytes":590368,"bytes_used":427167744,"bytes_avail":772666945536,"bytes_total":773094113280,"unknown_pgs_ratio":0.56701028347015381,"inactive_pgs_ratio":0.30927833914756775,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":136,"id":17,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":17,"rank":0,"name":"c","status":"up:active","gid":13173}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-03-18T20:27:10.310881+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"y":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"4":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"5":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"6":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"7":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-18T20:27:27.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.540+0000 7f304cb35700 1 -- 172.21.15.1:0/1870030634 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302806aed0 msgr2=0x7f302806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:27.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.540+0000 7f304cb35700 1 --2- 172.21.15.1:0/1870030634 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302806aed0 0x7f302806d390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3038001c90 tx=0x7f3038009040 comp rx=0 tx=0).stop 2024-03-18T20:27:27.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.540+0000 7f304cb35700 1 -- 172.21.15.1:0/1870030634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30481e8e20 msgr2=0x7f30481404c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:27.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.540+0000 7f304cb35700 1 --2- 172.21.15.1:0/1870030634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30481e8e20 0x7f30481404c0 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 crypto rx=0x7f30480b62b0 tx=0x7f30400114b0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.540+0000 7f304cb35700 1 -- 172.21.15.1:0/1870030634 shutdown_connections 2024-03-18T20:27:27.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.540+0000 7f304cb35700 1 --2- 172.21.15.1:0/1870030634 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f302806aed0 0x7f302806d390 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.540+0000 7f304cb35700 1 --2- 172.21.15.1:0/1870030634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30481e8e20 0x7f30481404c0 unknown :-1 s=CLOSED pgs=3084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.540+0000 7f304cb35700 1 --2- 172.21.15.1:0/1870030634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3034004850 0x7f30481f31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.540+0000 7f304cb35700 1 --2- 172.21.15.1:0/1870030634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30481e89a0 0x7f304813ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.540+0000 7f304cb35700 1 -- 172.21.15.1:0/1870030634 >> 172.21.15.1:0/1870030634 conn(0x7f30480b1d20 msgr2=0x7f3048071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:27.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.540+0000 7f304cb35700 1 -- 172.21.15.1:0/1870030634 shutdown_connections 2024-03-18T20:27:27.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.540+0000 7f304cb35700 1 -- 172.21.15.1:0/1870030634 wait complete. 2024-03-18T20:27:27.557 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-18T20:27:27.557 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:27:27.611 INFO:tasks.ceph.mds.c.smithi001.stderr:2024-03-18T20:27:27.606+0000 7f5a17ec2700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T20:27:27.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.677+0000 7f4085efb700 1 Processor -- start 2024-03-18T20:27:27.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.677+0000 7f4085efb700 1 -- start start 2024-03-18T20:27:27.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.678+0000 7f4085efb700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f408006a750 0x7f408006ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:27.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.678+0000 7f4085efb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f408015d6b0 0x7f4080160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.678+0000 7f4085efb700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40800acfb0 con 0x7f40800ad790 2024-03-18T20:27:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.678+0000 7f4085efb700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40800af7a0 con 0x7f408006a750 2024-03-18T20:27:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.678+0000 7f4085efb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f408006b070 con 0x7f408015d6b0 2024-03-18T20:27:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.678+0000 7f407f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f408006a750 0x7f408006ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.678+0000 7f407f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f408006a750 0x7f408006ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36558/0 (socket says 172.21.15.1:36558) 2024-03-18T20:27:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.678+0000 7f407f7fe700 1 -- 172.21.15.1:0/4083902671 learned_addr learned my addr 172.21.15.1:0/4083902671 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.678+0000 7f407effd700 1 --2- 172.21.15.1:0/4083902671 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f408015d6b0 0x7f4080160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.678+0000 7f407f7fe700 1 -- 172.21.15.1:0/4083902671 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f408015d6b0 msgr2=0x7f4080160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.678+0000 7f407f7fe700 1 --2- 172.21.15.1:0/4083902671 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f408015d6b0 0x7f4080160ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.678+0000 7f407f7fe700 1 -- 172.21.15.1:0/4083902671 >> v1:172.21.15.165:6789/0 conn(0x7f40800ad790 legacy=0x7f4080161530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.678+0000 7f407f7fe700 1 -- 172.21.15.1:0/4083902671 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40800b50a0 con 0x7f408006a750 2024-03-18T20:27:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.678+0000 7f407f7fe700 1 --2- 172.21.15.1:0/4083902671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f408006a750 0x7f408006ab30 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7f40680098f0 tx=0x7f4068009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad348e2bf5f45ba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.679+0000 7f407e7fc700 1 -- 172.21.15.1:0/4083902671 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4068011830 con 0x7f408006a750 2024-03-18T20:27:27.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.679+0000 7f407e7fc700 1 -- 172.21.15.1:0/4083902671 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4068011990 con 0x7f408006a750 2024-03-18T20:27:27.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.679+0000 7f407e7fc700 1 -- 172.21.15.1:0/4083902671 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4068011d20 con 0x7f408006a750 2024-03-18T20:27:27.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.679+0000 7f4085efb700 1 -- 172.21.15.1:0/4083902671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f408006a750 msgr2=0x7f408006ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:27.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.679+0000 7f4085efb700 1 --2- 172.21.15.1:0/4083902671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f408006a750 0x7f408006ab30 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7f40680098f0 tx=0x7f4068009c00 comp rx=0 tx=0).stop 2024-03-18T20:27:27.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.679+0000 7f4085efb700 1 -- 172.21.15.1:0/4083902671 shutdown_connections 2024-03-18T20:27:27.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.679+0000 7f4085efb700 1 --2- 172.21.15.1:0/4083902671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f408006a750 0x7f408006ab30 unknown :-1 s=CLOSED pgs=3085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.679+0000 7f4085efb700 1 --2- 172.21.15.1:0/4083902671 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f408015d6b0 0x7f4080160ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.679+0000 7f4085efb700 1 -- 172.21.15.1:0/4083902671 >> 172.21.15.1:0/4083902671 conn(0x7f408006b860 msgr2=0x7f408006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:27.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.679+0000 7f4085efb700 1 -- 172.21.15.1:0/4083902671 shutdown_connections 2024-03-18T20:27:27.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.679+0000 7f4085efb700 1 -- 172.21.15.1:0/4083902671 wait complete. 2024-03-18T20:27:27.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.680+0000 7f4085efb700 1 Processor -- start 2024-03-18T20:27:27.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.681+0000 7f4085efb700 1 -- start start 2024-03-18T20:27:27.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.681+0000 7f4085efb700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f408006a750 0x7f408014ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:27.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.681+0000 7f4085efb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40800ad790 0x7f408006fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:27.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.681+0000 7f4085efb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f408015d6b0 0x7f408015fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:27.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.681+0000 7f4085efb700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408015dd20 con 0x7f40800ad790 2024-03-18T20:27:27.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.681+0000 7f4085efb700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40801605a0 con 0x7f408006a750 2024-03-18T20:27:27.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.681+0000 7f4085efb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4080151280 con 0x7f408015d6b0 2024-03-18T20:27:27.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.681+0000 7f407effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40800ad790 0x7f408006fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:27.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.682+0000 7f407effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40800ad790 0x7f408006fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36566/0 (socket says 172.21.15.1:36566) 2024-03-18T20:27:27.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.682+0000 7f407effd700 1 -- 172.21.15.1:0/525864046 learned_addr learned my addr 172.21.15.1:0/525864046 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:27.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.682+0000 7f407f7fe700 1 --2- 172.21.15.1:0/525864046 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f408006a750 0x7f408014ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:27.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.682+0000 7f407ffff700 1 --2- 172.21.15.1:0/525864046 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f408015d6b0 0x7f408015fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:27.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.682+0000 7f407effd700 1 -- 172.21.15.1:0/525864046 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f408015d6b0 msgr2=0x7f408015fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:27.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.682+0000 7f407effd700 1 --2- 172.21.15.1:0/525864046 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f408015d6b0 0x7f408015fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.682+0000 7f407effd700 1 -- 172.21.15.1:0/525864046 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f408006a750 msgr2=0x7f408014ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:27.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.682+0000 7f407effd700 1 --2- 172.21.15.1:0/525864046 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f408006a750 0x7f408014ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.682+0000 7f407effd700 1 -- 172.21.15.1:0/525864046 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4068009550 con 0x7f40800ad790 2024-03-18T20:27:27.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.682+0000 7f407ffff700 1 --2- 172.21.15.1:0/525864046 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f408015d6b0 0x7f408015fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:27.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.682+0000 7f407effd700 1 --2- 172.21.15.1:0/525864046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40800ad790 0x7f408006fc20 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7f407000e8a0 tx=0x7f407000ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:27.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.682+0000 7f4077fff700 1 -- 172.21.15.1:0/525864046 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4070009590 con 0x7f40800ad790 2024-03-18T20:27:27.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.682+0000 7f4077fff700 1 -- 172.21.15.1:0/525864046 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4070013680 con 0x7f40800ad790 2024-03-18T20:27:27.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.682+0000 7f4077fff700 1 -- 172.21.15.1:0/525864046 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40700139f0 con 0x7f40800ad790 2024-03-18T20:27:27.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.682+0000 7f4085efb700 1 -- 172.21.15.1:0/525864046 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40801607f0 con 0x7f40800ad790 2024-03-18T20:27:27.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.682+0000 7f4085efb700 1 -- 172.21.15.1:0/525864046 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4080150fa0 con 0x7f40800ad790 2024-03-18T20:27:27.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.683+0000 7f4085efb700 1 -- 172.21.15.1:0/525864046 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40800ad420 con 0x7f40800ad790 2024-03-18T20:27:27.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.684+0000 7f4077fff700 1 -- 172.21.15.1:0/525864046 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f4070013b50 con 0x7f40800ad790 2024-03-18T20:27:27.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.684+0000 7f4077fff700 1 --2- 172.21.15.1:0/525864046 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f406006aed0 0x7f406006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:27.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.684+0000 7f4077fff700 1 -- 172.21.15.1:0/525864046 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f4070089770 con 0x7f40800ad790 2024-03-18T20:27:27.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.684+0000 7f407f7fe700 1 --2- 172.21.15.1:0/525864046 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f406006aed0 0x7f406006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:27.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.685+0000 7f407f7fe700 1 --2- 172.21.15.1:0/525864046 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f406006aed0 0x7f406006d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4068009c40 tx=0x7f4068008030 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:27.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.689+0000 7f4077fff700 1 -- 172.21.15.1:0/525864046 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f40800ad420 con 0x7f40800ad790 2024-03-18T20:27:27.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.888+0000 7f4085efb700 1 -- 172.21.15.1:0/525864046 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f40800af4c0 con 0x7f406006aed0 2024-03-18T20:27:27.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.896+0000 7f4077fff700 1 -- 172.21.15.1:0/525864046 <== mgr.12813 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+280373 (secure 0 0 0) 0x7f4068011c00 con 0x7f406006aed0 2024-03-18T20:27:27.904 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:27.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.899+0000 7f4085efb700 1 -- 172.21.15.1:0/525864046 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f406006aed0 msgr2=0x7f406006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:27.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.899+0000 7f4085efb700 1 --2- 172.21.15.1:0/525864046 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f406006aed0 0x7f406006d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4068009c40 tx=0x7f4068008030 comp rx=0 tx=0).stop 2024-03-18T20:27:27.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.899+0000 7f4085efb700 1 -- 172.21.15.1:0/525864046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40800ad790 msgr2=0x7f408006fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:27.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.899+0000 7f4085efb700 1 --2- 172.21.15.1:0/525864046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40800ad790 0x7f408006fc20 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7f407000e8a0 tx=0x7f407000ebb0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.899+0000 7f4085efb700 1 -- 172.21.15.1:0/525864046 shutdown_connections 2024-03-18T20:27:27.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.899+0000 7f4085efb700 1 --2- 172.21.15.1:0/525864046 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f406006aed0 0x7f406006d390 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.899+0000 7f4085efb700 1 --2- 172.21.15.1:0/525864046 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f408015d6b0 0x7f408015fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.899+0000 7f4085efb700 1 --2- 172.21.15.1:0/525864046 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f408006a750 0x7f408014ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.899+0000 7f4085efb700 1 --2- 172.21.15.1:0/525864046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40800ad790 0x7f408006fc20 unknown :-1 s=CLOSED pgs=3086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:27.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.899+0000 7f4085efb700 1 -- 172.21.15.1:0/525864046 >> 172.21.15.1:0/525864046 conn(0x7f408006b860 msgr2=0x7f408014fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:27.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.899+0000 7f4085efb700 1 -- 172.21.15.1:0/525864046 shutdown_connections 2024-03-18T20:27:27.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:27.899+0000 7f4085efb700 1 -- 172.21.15.1:0/525864046 wait complete. 2024-03-18T20:27:27.907 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:27:27.915 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-03-18T20:27:26.313769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":642,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":84,"acting":84,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":24,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":417156,"kb_used_data":27940,"kb_used_omap":12,"kb_used_meta":389107,"kb_avail":754557564,"statfs":{"total":773094113280,"available":772666945536,"internally_reserved":0,"allocated":28610560,"data_stored":15383647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":398446137},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.698740"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218004+0000","last_change":"2024-03-18T20:27:22.055309+0000","last_active":"2024-03-18T20:27:24.218004+0000","last_peered":"2024-03-18T20:27:24.218004+0000","last_clean":"2024-03-18T20:27:24.218004+0000","last_became_active":"2024-03-18T20:27:22.055064+0000","last_became_peered":"2024-03-18T20:27:22.055064+0000","last_unstale":"2024-03-18T20:27:24.218004+0000","last_undegraded":"2024-03-18T20:27:24.218004+0000","last_fullsized":"2024-03-18T20:27:24.218004+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:19.253084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.217979+0000","last_change":"2024-03-18T20:27:22.062816+0000","last_active":"2024-03-18T20:27:24.217979+0000","last_peered":"2024-03-18T20:27:24.217979+0000","last_clean":"2024-03-18T20:27:24.217979+0000","last_became_active":"2024-03-18T20:27:22.062600+0000","last_became_peered":"2024-03-18T20:27:22.062600+0000","last_unstale":"2024-03-18T20:27:24.217979+0000","last_undegraded":"2024-03-18T20:27:24.217979+0000","last_fullsized":"2024-03-18T20:27:24.217979+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:15:52.725667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218003+0000","last_change":"2024-03-18T20:27:22.062897+0000","last_active":"2024-03-18T20:27:24.218003+0000","last_peered":"2024-03-18T20:27:24.218003+0000","last_clean":"2024-03-18T20:27:24.218003+0000","last_became_active":"2024-03-18T20:27:22.062639+0000","last_became_peered":"2024-03-18T20:27:22.062639+0000","last_unstale":"2024-03-18T20:27:24.218003+0000","last_undegraded":"2024-03-18T20:27:24.218003+0000","last_fullsized":"2024-03-18T20:27:24.218003+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:45.973738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.217918+0000","last_change":"2024-03-18T20:27:22.073284+0000","last_active":"2024-03-18T20:27:24.217918+0000","last_peered":"2024-03-18T20:27:24.217918+0000","last_clean":"2024-03-18T20:27:24.217918+0000","last_became_active":"2024-03-18T20:27:22.072955+0000","last_became_peered":"2024-03-18T20:27:22.072955+0000","last_unstale":"2024-03-18T20:27:24.217918+0000","last_undegraded":"2024-03-18T20:27:24.217918+0000","last_fullsized":"2024-03-18T20:27:24.217918+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:10.708304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.622037+0000","last_change":"2024-03-18T20:27:20.622037+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.622037+0000","last_undegraded":"2024-03-18T20:27:20.622037+0000","last_fullsized":"2024-03-18T20:27:20.622037+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:01.698353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621734+0000","last_change":"2024-03-18T20:27:20.621734+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621734+0000","last_undegraded":"2024-03-18T20:27:20.621734+0000","last_fullsized":"2024-03-18T20:27:20.621734+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:02:14.143177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.622565+0000","last_change":"2024-03-18T20:27:20.622565+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.622565+0000","last_undegraded":"2024-03-18T20:27:20.622565+0000","last_fullsized":"2024-03-18T20:27:20.622565+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:42:23.110707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.622373+0000","last_change":"2024-03-18T20:27:20.622373+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.622373+0000","last_undegraded":"2024-03-18T20:27:20.622373+0000","last_fullsized":"2024-03-18T20:27:20.622373+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:45:52.316434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617958+0000","last_change":"2024-03-18T20:27:20.617958+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617958+0000","last_undegraded":"2024-03-18T20:27:20.617958+0000","last_fullsized":"2024-03-18T20:27:20.617958+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:08:18.189309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.1a","version":"273'1","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.069050+0000","last_change":"2024-03-18T20:27:21.635634+0000","last_active":"2024-03-18T20:27:22.069050+0000","last_peered":"2024-03-18T20:27:22.069050+0000","last_clean":"2024-03-18T20:27:22.069050+0000","last_became_active":"2024-03-18T20:27:21.635366+0000","last_became_peered":"2024-03-18T20:27:21.635366+0000","last_unstale":"2024-03-18T20:27:22.069050+0000","last_undegraded":"2024-03-18T20:27:22.069050+0000","last_fullsized":"2024-03-18T20:27:22.069050+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:40:40.016554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623987+0000","last_change":"2024-03-18T20:27:20.623987+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623987+0000","last_undegraded":"2024-03-18T20:27:20.623987+0000","last_fullsized":"2024-03-18T20:27:20.623987+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:08:18.189309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621015+0000","last_change":"2024-03-18T20:27:20.621015+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621015+0000","last_undegraded":"2024-03-18T20:27:20.621015+0000","last_fullsized":"2024-03-18T20:27:20.621015+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:10.342403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623746+0000","last_change":"2024-03-18T20:27:20.623746+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623746+0000","last_undegraded":"2024-03-18T20:27:20.623746+0000","last_fullsized":"2024-03-18T20:27:20.623746+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:28:52.874751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623648+0000","last_change":"2024-03-18T20:27:20.623648+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623648+0000","last_undegraded":"2024-03-18T20:27:20.623648+0000","last_fullsized":"2024-03-18T20:27:20.623648+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:21:19.006472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.624317+0000","last_change":"2024-03-18T20:27:20.624317+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.624317+0000","last_undegraded":"2024-03-18T20:27:20.624317+0000","last_fullsized":"2024-03-18T20:27:20.624317+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:29.812382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218074+0000","last_change":"2024-03-18T20:27:22.081779+0000","last_active":"2024-03-18T20:27:24.218074+0000","last_peered":"2024-03-18T20:27:24.218074+0000","last_clean":"2024-03-18T20:27:24.218074+0000","last_became_active":"2024-03-18T20:27:22.081506+0000","last_became_peered":"2024-03-18T20:27:22.081506+0000","last_unstale":"2024-03-18T20:27:24.218074+0000","last_undegraded":"2024-03-18T20:27:24.218074+0000","last_fullsized":"2024-03-18T20:27:24.218074+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:29:27.205683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621040+0000","last_change":"2024-03-18T20:27:20.621040+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621040+0000","last_undegraded":"2024-03-18T20:27:20.621040+0000","last_fullsized":"2024-03-18T20:27:20.621040+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:01:31.379909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.625530+0000","last_change":"2024-03-18T20:27:20.625530+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.625530+0000","last_undegraded":"2024-03-18T20:27:20.625530+0000","last_fullsized":"2024-03-18T20:27:20.625530+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:42.339058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617987+0000","last_change":"2024-03-18T20:27:20.617987+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617987+0000","last_undegraded":"2024-03-18T20:27:20.617987+0000","last_fullsized":"2024-03-18T20:27:20.617987+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:42.339058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617911+0000","last_change":"2024-03-18T20:27:20.617911+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617911+0000","last_undegraded":"2024-03-18T20:27:20.617911+0000","last_fullsized":"2024-03-18T20:27:20.617911+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:43.006246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621574+0000","last_change":"2024-03-18T20:27:20.621574+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621574+0000","last_undegraded":"2024-03-18T20:27:20.621574+0000","last_fullsized":"2024-03-18T20:27:20.621574+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:31:56.485139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.217869+0000","last_change":"2024-03-18T20:27:22.054635+0000","last_active":"2024-03-18T20:27:24.217869+0000","last_peered":"2024-03-18T20:27:24.217869+0000","last_clean":"2024-03-18T20:27:24.217869+0000","last_became_active":"2024-03-18T20:27:22.054372+0000","last_became_peered":"2024-03-18T20:27:22.054372+0000","last_unstale":"2024-03-18T20:27:24.217869+0000","last_undegraded":"2024-03-18T20:27:24.217869+0000","last_fullsized":"2024-03-18T20:27:24.217869+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:40.826497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621698+0000","last_change":"2024-03-18T20:27:20.621698+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621698+0000","last_undegraded":"2024-03-18T20:27:20.621698+0000","last_fullsized":"2024-03-18T20:27:20.621698+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:17:27.485874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.e","version":"273'2","reported_seq":13,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.083274+0000","last_change":"2024-03-18T20:27:21.640607+0000","last_active":"2024-03-18T20:27:22.083274+0000","last_peered":"2024-03-18T20:27:22.083274+0000","last_clean":"2024-03-18T20:27:22.083274+0000","last_became_active":"2024-03-18T20:27:21.640341+0000","last_became_peered":"2024-03-18T20:27:21.640341+0000","last_unstale":"2024-03-18T20:27:22.083274+0000","last_undegraded":"2024-03-18T20:27:22.083274+0000","last_fullsized":"2024-03-18T20:27:22.083274+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:20:18.042657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218120+0000","last_change":"2024-03-18T20:27:22.055634+0000","last_active":"2024-03-18T20:27:24.218120+0000","last_peered":"2024-03-18T20:27:24.218120+0000","last_clean":"2024-03-18T20:27:24.218120+0000","last_became_active":"2024-03-18T20:27:22.055397+0000","last_became_peered":"2024-03-18T20:27:22.055397+0000","last_unstale":"2024-03-18T20:27:24.218120+0000","last_undegraded":"2024-03-18T20:27:24.218120+0000","last_fullsized":"2024-03-18T20:27:24.218120+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:29.388876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621611+0000","last_change":"2024-03-18T20:27:20.621611+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621611+0000","last_undegraded":"2024-03-18T20:27:20.621611+0000","last_fullsized":"2024-03-18T20:27:20.621611+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:58:00.382684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623890+0000","last_change":"2024-03-18T20:27:20.623890+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623890+0000","last_undegraded":"2024-03-18T20:27:20.623890+0000","last_fullsized":"2024-03-18T20:27:20.623890+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:01.555443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.624331+0000","last_change":"2024-03-18T20:27:20.624331+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.624331+0000","last_undegraded":"2024-03-18T20:27:20.624331+0000","last_fullsized":"2024-03-18T20:27:20.624331+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:52.857794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.622224+0000","last_change":"2024-03-18T20:27:20.622224+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.622224+0000","last_undegraded":"2024-03-18T20:27:20.622224+0000","last_fullsized":"2024-03-18T20:27:20.622224+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:13:18.842477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621046+0000","last_change":"2024-03-18T20:27:20.621046+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621046+0000","last_undegraded":"2024-03-18T20:27:20.621046+0000","last_fullsized":"2024-03-18T20:27:20.621046+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:11:16.795077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218061+0000","last_change":"2024-03-18T20:27:22.062557+0000","last_active":"2024-03-18T20:27:24.218061+0000","last_peered":"2024-03-18T20:27:24.218061+0000","last_clean":"2024-03-18T20:27:24.218061+0000","last_became_active":"2024-03-18T20:27:22.062283+0000","last_became_peered":"2024-03-18T20:27:22.062283+0000","last_unstale":"2024-03-18T20:27:24.218061+0000","last_undegraded":"2024-03-18T20:27:24.218061+0000","last_fullsized":"2024-03-18T20:27:24.218061+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:05:24.597193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.625820+0000","last_change":"2024-03-18T20:27:20.625820+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.625820+0000","last_undegraded":"2024-03-18T20:27:20.625820+0000","last_fullsized":"2024-03-18T20:27:20.625820+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:41:34.677506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.624685+0000","last_change":"2024-03-18T20:27:20.624685+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.624685+0000","last_undegraded":"2024-03-18T20:27:20.624685+0000","last_fullsized":"2024-03-18T20:27:20.624685+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:45.164373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617958+0000","last_change":"2024-03-18T20:27:20.617958+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617958+0000","last_undegraded":"2024-03-18T20:27:20.617958+0000","last_fullsized":"2024-03-18T20:27:20.617958+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:52.857794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621100+0000","last_change":"2024-03-18T20:27:20.621100+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621100+0000","last_undegraded":"2024-03-18T20:27:20.621100+0000","last_fullsized":"2024-03-18T20:27:20.621100+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:23:55.315193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.617962+0000","last_change":"2024-03-18T20:27:20.617962+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.617962+0000","last_undegraded":"2024-03-18T20:27:20.617962+0000","last_fullsized":"2024-03-18T20:27:20.617962+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:01.555443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623634+0000","last_change":"2024-03-18T20:27:20.623634+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623634+0000","last_undegraded":"2024-03-18T20:27:20.623634+0000","last_fullsized":"2024-03-18T20:27:20.623634+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:23:24.136007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.621775+0000","last_change":"2024-03-18T20:27:20.621775+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.621775+0000","last_undegraded":"2024-03-18T20:27:20.621775+0000","last_fullsized":"2024-03-18T20:27:20.621775+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:54:45.115313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218125+0000","last_change":"2024-03-18T20:27:22.069073+0000","last_active":"2024-03-18T20:27:24.218125+0000","last_peered":"2024-03-18T20:27:24.218125+0000","last_clean":"2024-03-18T20:27:24.218125+0000","last_became_active":"2024-03-18T20:27:22.068744+0000","last_became_peered":"2024-03-18T20:27:22.068744+0000","last_unstale":"2024-03-18T20:27:24.218125+0000","last_undegraded":"2024-03-18T20:27:24.218125+0000","last_fullsized":"2024-03-18T20:27:24.218125+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:44.884087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.623503+0000","last_change":"2024-03-18T20:27:20.623503+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.623503+0000","last_undegraded":"2024-03-18T20:27:20.623503+0000","last_fullsized":"2024-03-18T20:27:20.623503+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:28.989185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":1,"reported_epoch":271,"state":"creating+peering","last_fresh":"2024-03-18T20:27:20.625365+0000","last_change":"2024-03-18T20:27:20.625365+0000","last_active":"2024-03-18T20:27:20.607510+0000","last_peered":"2024-03-18T20:27:20.607510+0000","last_clean":"2024-03-18T20:27:20.607510+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:20.625365+0000","last_undegraded":"2024-03-18T20:27:20.625365+0000","last_fullsized":"2024-03-18T20:27:20.625365+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:43.006246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:27:21.614682+0000","last_change":"2024-03-18T20:27:21.614682+0000","last_active":"2024-03-18T20:27:21.614682+0000","last_peered":"2024-03-18T20:27:21.614682+0000","last_clean":"2024-03-18T20:27:21.614682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:27:21.614682+0000","last_undegraded":"2024-03-18T20:27:21.614682+0000","last_fullsized":"2024-03-18T20:27:21.614682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"259'381","reported_seq":1352,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-18T20:27:18.438231+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:27:18.438231+0000","last_peered":"2024-03-18T20:27:18.438231+0000","last_clean":"2024-03-18T20:27:18.438231+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:27:18.438231+0000","last_undegraded":"2024-03-18T20:27:18.438231+0000","last_fullsized":"2024-03-18T20:27:18.438231+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":64,"acting":64,"num_store_stats":8},{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":1},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576001,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53088,"kb_used_data":3924,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318752,"statfs":{"total":96636764160,"available":96582402048,"internally_reserved":0,"allocated":4018176,"data_stored":2363924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54900000000000004,"15min":0.58999999999999997},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.72599999999999998,"5min":0.80900000000000005,"15min":1.1339999999999999},"last":0.439},{"interface":"front","average":{"1min":0.498,"5min":0.56599999999999995,"15min":0.60199999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":0.73599999999999999,"5min":0.84499999999999997,"15min":1.093},"last":0.56699999999999995}]},{"osd":1,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.55500000000000005,"15min":0.60199999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.748,"5min":0.83099999999999996,"15min":1.4299999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.57099999999999995,"15min":0.60799999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73599999999999999,"5min":0.82399999999999995,"15min":1.2629999999999999},"last":0.61599999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.56100000000000005,"15min":0.60099999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67600000000000005,"5min":0.85499999999999998,"15min":1.216},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57799999999999996,"15min":0.61599999999999999},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75,"5min":1.1279999999999999,"15min":1.2490000000000001},"last":0.59899999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":0.59799999999999998},"min":{"1min":0.38700000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71999999999999997,"5min":0.84699999999999998,"15min":1.4450000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.60699999999999998,"15min":0.64100000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73099999999999998,"5min":0.94599999999999995,"15min":1.5029999999999999},"last":0.63300000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47099999999999997,"15min":0.47799999999999998},"min":{"1min":0.35999999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.68899999999999995,"5min":0.79100000000000004,"15min":0.86099999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48899999999999999,"15min":0.499},"min":{"1min":0.34100000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.66200000000000003,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.54500000000000004}]},{"osd":5,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.51300000000000001,"15min":0.51900000000000002},"min":{"1min":0.373,"5min":0.314,"15min":0.314},"max":{"1min":0.70399999999999996,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.54600000000000004},"min":{"1min":0.38300000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73099999999999998,"5min":0.877,"15min":1.042},"last":0.68799999999999994}]},{"osd":6,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55800000000000005,"15min":0.57699999999999996},"min":{"1min":0.435,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77300000000000002,"5min":0.93500000000000005,"15min":1.129},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.54700000000000004,"15min":0.56899999999999995},"min":{"1min":0.47399999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76200000000000001,"5min":0.91900000000000004,"15min":1.0609999999999999},"last":0.46899999999999997}]}]},{"osd":6,"up_from":13,"seq":55834576001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60700,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311140,"statfs":{"total":96636764160,"available":96574607360,"internally_reserved":0,"allocated":3424256,"data_stored":1773556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53300000000000003,"15min":0.58499999999999996},"min":{"1min":0.35399999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68799999999999994,"5min":0.79800000000000004,"15min":1.1519999999999999},"last":0.505},{"interface":"front","average":{"1min":0.503,"5min":0.54600000000000004,"15min":0.59499999999999997},"min":{"1min":0.33200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.73499999999999999,"5min":1.119,"15min":1.119},"last":0.67100000000000004}]},{"osd":1,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53800000000000003,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73199999999999998,"5min":0.79100000000000004,"15min":1.244},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":0.59499999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":0.70099999999999996,"5min":0.89200000000000002,"15min":1.286},"last":0.54600000000000004}]},{"osd":2,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56399999999999995,"15min":0.61699999999999999},"min":{"1min":0.38700000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.69599999999999995,"5min":0.79600000000000004,"15min":1.276},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.55400000000000005,"15min":0.59899999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.65200000000000002,"5min":1.089,"15min":1.2669999999999999},"last":0.53700000000000003}]},{"osd":3,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55500000000000005,"15min":0.61099999999999999},"min":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71599999999999997,"5min":1.163,"15min":1.163},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57099999999999995,"15min":0.61099999999999999},"min":{"1min":0.47499999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78100000000000003,"5min":0.82399999999999995,"15min":1.337},"last":0.55500000000000005}]},{"osd":4,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.35299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80200000000000005,"5min":0.85799999999999998,"15min":0.88300000000000001},"last":0.378},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.50600000000000001,"15min":0.51800000000000002},"min":{"1min":0.32500000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83399999999999996,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.40200000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.50900000000000001,"15min":0.51200000000000001},"min":{"1min":0.42599999999999999,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":0.65000000000000002,"5min":1.008,"15min":1.008},"last":0.441},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55900000000000005,"15min":0.55300000000000005},"min":{"1min":0.41799999999999998,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.755,"5min":0.92600000000000005,"15min":0.94199999999999995},"last":0.43099999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56699999999999995,"15min":0.56299999999999994},"min":{"1min":0.40999999999999998,"5min":0.38,"15min":0.34499999999999997},"max":{"1min":0.91200000000000003,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.34599999999999997},"max":{"1min":0.86699999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.52000000000000002}]}]},{"osd":1,"up_from":13,"seq":55834576002,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52548,"kb_used_data":3384,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319292,"statfs":{"total":96636764160,"available":96582955008,"internally_reserved":0,"allocated":3465216,"data_stored":1794256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.38700000000000001,"15min":0.39300000000000002},"min":{"1min":0.255,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.46400000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39500000000000002,"15min":0.41999999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.623,"5min":0.623,"15min":0.69699999999999995},"last":0.39900000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.40899999999999997,"15min":0.42899999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53100000000000003,"5min":0.61199999999999999,"15min":0.68100000000000005},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.45200000000000001,"15min":0.47099999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.69799999999999995,"5min":0.70099999999999996,"15min":0.80500000000000005},"last":0.34799999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.46500000000000002,"15min":0.48399999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.76100000000000001},"last":0.44},{"interface":"front","average":{"1min":0.372,"5min":0.41899999999999998,"15min":0.433},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.27000000000000002},"max":{"1min":0.54700000000000004,"5min":0.68200000000000005,"15min":0.71699999999999997},"last":0.32400000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.64000000000000001,"15min":0.66500000000000004},"min":{"1min":0.43099999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76800000000000002,"5min":1.0680000000000001,"15min":1.2909999999999999},"last":0.629},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.64600000000000002,"15min":0.67100000000000004},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.88500000000000001,"5min":1.1160000000000001,"15min":1.746},"last":0.55300000000000005}]},{"osd":5,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.63400000000000001,"15min":0.66100000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79300000000000004,"5min":1.0389999999999999,"15min":1.6910000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.67800000000000005,"15min":0.70499999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.85199999999999998,"5min":1.109,"15min":1.2549999999999999},"last":0.59199999999999997}]},{"osd":6,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.65300000000000002,"15min":0.67700000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.70799999999999996,"5min":1.0289999999999999,"15min":1.637},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.67800000000000005,"15min":0.70199999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82099999999999995,"5min":1.014,"15min":1.722},"last":0.52400000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.66000000000000003,"15min":0.69899999999999995},"min":{"1min":0.46100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91100000000000003,"5min":0.99199999999999999,"15min":1.2729999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.67000000000000004,"15min":0.69699999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.80700000000000005,"5min":1.0329999999999999,"15min":1.3380000000000001},"last":0.66900000000000004}]}]},{"osd":0,"up_from":13,"seq":55834576002,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48992,"kb_used_data":3924,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322848,"statfs":{"total":96636764160,"available":96586596352,"internally_reserved":0,"allocated":4018176,"data_stored":2363924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.36399999999999999,"15min":0.38600000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.46100000000000002,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.39600000000000002,"15min":0.41999999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.58799999999999997,"5min":0.60099999999999998,"15min":1.1850000000000001},"last":0.255}]},{"osd":2,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.41299999999999998,"15min":0.433},"min":{"1min":0.28699999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.47199999999999998,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.41799999999999998,"15min":0.44600000000000001},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.54500000000000004,"5min":0.64800000000000002,"15min":0.70999999999999996},"last":0.39400000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.39600000000000002,"15min":0.42399999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.433,"5min":0.58499999999999996,"15min":0.628},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.45400000000000001,"15min":0.48699999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.52600000000000002,"5min":0.73399999999999999,"15min":0.91500000000000004},"last":0.379}]},{"osd":4,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.61299999999999999,"15min":0.64400000000000002},"min":{"1min":0.41299999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.752,"5min":0.91500000000000004,"15min":1.0029999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.64800000000000002,"15min":0.65900000000000003},"min":{"1min":0.51600000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.92300000000000004,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.72099999999999997}]},{"osd":5,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.64400000000000002,"15min":0.66600000000000004},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83299999999999996,"5min":0.92300000000000004,"15min":1.0129999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65300000000000002,"15min":0.67500000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.73199999999999998,"5min":0.96199999999999997,"15min":1.518},"last":0.70499999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.64600000000000002,"15min":0.65700000000000003},"min":{"1min":0.45800000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.73599999999999999,"5min":1.0669999999999999,"15min":1.1279999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.68200000000000005,"15min":0.70299999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.871,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.746}]},{"osd":7,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.67200000000000004,"15min":0.69899999999999995},"min":{"1min":0.45300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77500000000000002,"5min":1.008,"15min":1.464},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.67300000000000004,"15min":0.69799999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88900000000000001,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.76300000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52508,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319332,"statfs":{"total":96636764160,"available":96582995968,"internally_reserved":0,"allocated":3424256,"data_stored":1773556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.34699999999999998,"15min":0.371},"min":{"1min":0.20799999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.53800000000000003,"5min":0.54800000000000004,"15min":0.57499999999999996},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.44700000000000001,"15min":0.47799999999999998},"min":{"1min":0.26100000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60499999999999998,"5min":0.71499999999999997,"15min":0.753},"last":0.48999999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.44400000000000001,"15min":0.47399999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57399999999999995,"5min":0.64300000000000002,"15min":0.749},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.38900000000000001,"15min":0.40899999999999997},"min":{"1min":0.246,"5min":0.224,"15min":0.224},"max":{"1min":0.501,"5min":0.55900000000000005,"15min":0.59999999999999998},"last":0.32700000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.41199999999999998,"15min":0.42899999999999999},"min":{"1min":0.27800000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.56100000000000005,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.47399999999999998,"15min":0.50700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.63200000000000001,"5min":0.68100000000000005,"15min":0.752},"last":0.45400000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63900000000000001,"15min":0.65300000000000002},"min":{"1min":0.41799999999999998,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":1.2090000000000001,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.65700000000000003,"15min":0.66200000000000003},"min":{"1min":0.45100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.268,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.61899999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.64900000000000002,"15min":0.67500000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79900000000000004,"5min":0.97299999999999998,"15min":1.028},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.63900000000000001,"15min":0.67000000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.77100000000000002,"5min":0.88300000000000001,"15min":1.0629999999999999},"last":0.80800000000000005}]},{"osd":6,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.65500000000000003,"15min":0.68100000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.81899999999999995,"5min":1.014,"15min":1.2050000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.72699999999999998},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.3220000000000001,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.77000000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.65500000000000003,"15min":0.68899999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.42199999999999999},"max":{"1min":0.73699999999999999,"5min":0.99399999999999999,"15min":1.0269999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68600000000000005,"15min":0.70099999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.2330000000000001,"5min":1.371,"15min":1.371},"last":0.79100000000000004}]}]},{"osd":3,"up_from":13,"seq":55834576002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52508,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319332,"statfs":{"total":96636764160,"available":96582995968,"internally_reserved":0,"allocated":3424256,"data_stored":1773556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.36099999999999999,"15min":0.39000000000000001},"min":{"1min":0.247,"5min":0.223,"15min":0.223},"max":{"1min":0.46400000000000002,"5min":0.66500000000000004,"15min":0.98099999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.373,"15min":0.40400000000000003},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.45600000000000002,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.42399999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.35999999999999999,"15min":0.39700000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.44500000000000001,"5min":0.63200000000000001,"15min":0.64000000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.42199999999999999,"15min":0.45600000000000002},"min":{"1min":0.33400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.501,"5min":0.65500000000000003,"15min":0.73099999999999998},"last":0.44500000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.41499999999999998,"15min":0.45200000000000001},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.54900000000000004,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.437,"15min":0.48099999999999998},"min":{"1min":0.32600000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.52300000000000002,"5min":0.64400000000000002,"15min":0.71199999999999997},"last":0.32600000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59499999999999997,"15min":0.64000000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.526,"5min":1.526,"15min":1.526},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61399999999999999,"15min":0.66100000000000003},"min":{"1min":0.39200000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.65800000000000003,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.65800000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.65100000000000002,"15min":0.68700000000000006},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.66500000000000004,"15min":0.71499999999999997},"min":{"1min":0.45300000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78000000000000003,"5min":1.1120000000000001,"15min":1.171},"last":0.65000000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65800000000000003,"15min":0.69699999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.62},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.71999999999999997},"min":{"1min":0.48699999999999999,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.58999999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.64900000000000002,"15min":0.70299999999999996},"min":{"1min":0.43099999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79900000000000004,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64300000000000002,"15min":0.68899999999999995},"min":{"1min":0.51500000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.57299999999999995}]}]},{"osd":4,"up_from":13,"seq":55834576002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48400,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323440,"statfs":{"total":96636764160,"available":96587202560,"internally_reserved":0,"allocated":3411968,"data_stored":1767319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57799999999999996,"15min":0.58899999999999997},"min":{"1min":0.39600000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56799999999999995,"15min":0.59499999999999997},"min":{"1min":0.40100000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.84199999999999997},"last":0.621}]},{"osd":1,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.79300000000000004,"5min":0.81899999999999995,"15min":0.88300000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56399999999999995,"15min":0.57999999999999996},"min":{"1min":0.34899999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.96499999999999997},"last":0.58999999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59799999999999998,"15min":0.61399999999999999},"min":{"1min":0.42899999999999999,"5min":0.41299999999999998,"15min":0.38500000000000001},"max":{"1min":0.70799999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.54500000000000004}]},{"osd":3,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.59299999999999997,"15min":0.60199999999999998},"min":{"1min":0.433,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78100000000000003,"5min":1.026,"15min":1.026},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.41399999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.72999999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.48399999999999999,"15min":0.48199999999999998},"min":{"1min":0.318,"5min":0.309,"15min":0.309},"max":{"1min":0.76700000000000002,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.747},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.496,"15min":0.505},"min":{"1min":0.38300000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.81499999999999995,"5min":0.85499999999999998,"15min":0.90600000000000003},"last":0.63800000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.502,"15min":0.50600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.754,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.82799999999999996,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.71399999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.40600000000000003,"5min":0.38800000000000001,"15min":0.35699999999999998},"max":{"1min":0.80300000000000005,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.38500000000000001,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.84099999999999997,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.65200000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48412,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323428,"statfs":{"total":96636764160,"available":96587190272,"internally_reserved":0,"allocated":3424256,"data_stored":1773556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.55400000000000005,"15min":0.59099999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.31900000000000001},"max":{"1min":0.76300000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.621},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55600000000000005,"15min":0.59499999999999997},"min":{"1min":0.42099999999999999,"5min":0.35599999999999998,"15min":0.33100000000000002},"max":{"1min":0.81100000000000005,"5min":0.82599999999999996,"15min":1.3100000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.56100000000000005,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71499999999999997,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.55400000000000005,"15min":0.59199999999999997},"min":{"1min":0.372,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.69399999999999995,"5min":0.94299999999999995,"15min":1.018},"last":0.58499999999999996}]},{"osd":2,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.59399999999999997},"min":{"1min":0.40300000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.75600000000000001,"5min":0.85299999999999998,"15min":0.99299999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.55400000000000005,"15min":0.60099999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77000000000000002,"5min":1.599,"15min":1.599},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57499999999999996,"15min":0.60399999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72899999999999998,"5min":0.98699999999999999,"15min":1.2430000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59299999999999997,"15min":0.63700000000000001},"min":{"1min":0.42099999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":0.84399999999999997,"15min":1.161},"last":0.63300000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48999999999999999,"15min":0.48299999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.625,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.35699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.40699999999999997}]},{"osd":6,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.503,"15min":0.50900000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.32800000000000001},"max":{"1min":0.64800000000000002,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54300000000000004,"15min":0.54200000000000004},"min":{"1min":0.35199999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.64600000000000002,"5min":1.643,"15min":1.643},"last":0.501}]},{"osd":7,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.40200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.71299999999999997,"5min":1.028,"15min":1.028},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57099999999999995,"15min":0.57799999999999996},"min":{"1min":0.42599999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74299999999999999,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:27:30.921 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:27:31.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.056+0000 7f1a01a38700 1 Processor -- start 2024-03-18T20:27:31.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.056+0000 7f1a01a38700 1 -- start start 2024-03-18T20:27:31.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.057+0000 7f1a01a38700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19fc160940 0x7f19fc0d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:31.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.057+0000 7f1a01a38700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f19fc0cf090 0x7f19fc0d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:31.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.057+0000 7f1a01a38700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f19fc160d20 0x7f19fc0d1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:31.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.057+0000 7f1a01a38700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19fc0cddc0 con 0x7f19fc0cf090 2024-03-18T20:27:31.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.057+0000 7f1a01a38700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19fc0cdff0 con 0x7f19fc160d20 2024-03-18T20:27:31.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.057+0000 7f1a01a38700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc0ce2d0 con 0x7f19fc160940 2024-03-18T20:27:31.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.057+0000 7f19fa7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f19fc0cf090 0x7f19fc0d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:31.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.057+0000 7f19fa7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f19fc0cf090 0x7f19fc0d1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36582/0 (socket says 172.21.15.1:36582) 2024-03-18T20:27:31.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.057+0000 7f19fa7fc700 1 -- 172.21.15.1:0/550472377 learned_addr learned my addr 172.21.15.1:0/550472377 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:31.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.057+0000 7f19fb7fe700 1 --2- 172.21.15.1:0/550472377 >> v2:172.21.15.165:3300/0 conn(0x7f19fc160d20 0x7f19fc0d1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:31.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.057+0000 7f19faffd700 1 --2- 172.21.15.1:0/550472377 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19fc160940 0x7f19fc0d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:31.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.057+0000 7f19fa7fc700 1 -- 172.21.15.1:0/550472377 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19fc160940 msgr2=0x7f19fc0d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:31.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.057+0000 7f19fa7fc700 1 --2- 172.21.15.1:0/550472377 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19fc160940 0x7f19fc0d0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.057+0000 7f19fa7fc700 1 -- 172.21.15.1:0/550472377 >> v2:172.21.15.165:3300/0 conn(0x7f19fc160d20 msgr2=0x7f19fc0d1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:31.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.057+0000 7f19fa7fc700 1 --2- 172.21.15.1:0/550472377 >> v2:172.21.15.165:3300/0 conn(0x7f19fc160d20 0x7f19fc0d1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.058+0000 7f19fa7fc700 1 -- 172.21.15.1:0/550472377 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19fc0ce5b0 con 0x7f19fc0cf090 2024-03-18T20:27:31.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.058+0000 7f19fb7fe700 1 --2- 172.21.15.1:0/550472377 >> v2:172.21.15.165:3300/0 conn(0x7f19fc160d20 0x7f19fc0d1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:31.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.058+0000 7f19faffd700 1 --2- 172.21.15.1:0/550472377 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19fc160940 0x7f19fc0d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:31.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.058+0000 7f19fa7fc700 1 --2- 172.21.15.1:0/550472377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19fc0cf090 0x7f19fc0d1790 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 crypto rx=0x7f19fc0d2bb0 tx=0x7f19e4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=280c29764cbc0a7c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:31.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.058+0000 7f19f9ffb700 1 -- 172.21.15.1:0/550472377 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e400b870 con 0x7f19fc0cf090 2024-03-18T20:27:31.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.058+0000 7f19f9ffb700 1 -- 172.21.15.1:0/550472377 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e400b9d0 con 0x7f19fc0cf090 2024-03-18T20:27:31.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.058+0000 7f19f9ffb700 1 -- 172.21.15.1:0/550472377 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e400bd60 con 0x7f19fc0cf090 2024-03-18T20:27:31.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.058+0000 7f1a01a38700 1 -- 172.21.15.1:0/550472377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19fc0cf090 msgr2=0x7f19fc0d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:31.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.058+0000 7f1a01a38700 1 --2- 172.21.15.1:0/550472377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19fc0cf090 0x7f19fc0d1790 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 crypto rx=0x7f19fc0d2bb0 tx=0x7f19e4009bd0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.058+0000 7f1a01a38700 1 -- 172.21.15.1:0/550472377 shutdown_connections 2024-03-18T20:27:31.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.058+0000 7f1a01a38700 1 --2- 172.21.15.1:0/550472377 >> v2:172.21.15.165:3300/0 conn(0x7f19fc160d20 0x7f19fc0d1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.058+0000 7f1a01a38700 1 --2- 172.21.15.1:0/550472377 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19fc160940 0x7f19fc0d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.058+0000 7f1a01a38700 1 --2- 172.21.15.1:0/550472377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19fc0cf090 0x7f19fc0d1790 unknown :-1 s=CLOSED pgs=3087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.058+0000 7f1a01a38700 1 -- 172.21.15.1:0/550472377 >> 172.21.15.1:0/550472377 conn(0x7f19fc09c520 msgr2=0x7f19fc0d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:31.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.059+0000 7f1a01a38700 1 -- 172.21.15.1:0/550472377 shutdown_connections 2024-03-18T20:27:31.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.059+0000 7f1a01a38700 1 -- 172.21.15.1:0/550472377 wait complete. 2024-03-18T20:27:31.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.060+0000 7f1a01a38700 1 Processor -- start 2024-03-18T20:27:31.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.060+0000 7f1a01a38700 1 -- start start 2024-03-18T20:27:31.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f1a01a38700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19fc0cf090 0x7f19fc0b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:31.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f1a01a38700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19fc160940 0x7f19fc14f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:31.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f1a01a38700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19fc160d20 0x7f19fc15ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:31.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f1a01a38700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc0b8a00 con 0x7f19fc0cf090 2024-03-18T20:27:31.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f1a01a38700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc0b16e0 con 0x7f19fc160d20 2024-03-18T20:27:31.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f1a01a38700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc14f990 con 0x7f19fc160940 2024-03-18T20:27:31.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f19faffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19fc0cf090 0x7f19fc0b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:31.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f19faffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19fc0cf090 0x7f19fc0b11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36598/0 (socket says 172.21.15.1:36598) 2024-03-18T20:27:31.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f19faffd700 1 -- 172.21.15.1:0/1429357180 learned_addr learned my addr 172.21.15.1:0/1429357180 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:31.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f19fb7fe700 1 --2- 172.21.15.1:0/1429357180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19fc160d20 0x7f19fc15ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:31.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f19fa7fc700 1 --2- 172.21.15.1:0/1429357180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19fc160940 0x7f19fc14f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:31.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f19faffd700 1 -- 172.21.15.1:0/1429357180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19fc160940 msgr2=0x7f19fc14f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:31.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f19faffd700 1 --2- 172.21.15.1:0/1429357180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19fc160940 0x7f19fc14f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f19faffd700 1 -- 172.21.15.1:0/1429357180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19fc160d20 msgr2=0x7f19fc15ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:31.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f19faffd700 1 --2- 172.21.15.1:0/1429357180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19fc160d20 0x7f19fc15ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f19faffd700 1 -- 172.21.15.1:0/1429357180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e4009550 con 0x7f19fc0cf090 2024-03-18T20:27:31.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f19fa7fc700 1 --2- 172.21.15.1:0/1429357180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19fc160940 0x7f19fc14f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:31.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f19fb7fe700 1 --2- 172.21.15.1:0/1429357180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19fc160d20 0x7f19fc15ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:31.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f19faffd700 1 --2- 172.21.15.1:0/1429357180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19fc0cf090 0x7f19fc0b11a0 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00c750 tx=0x7f19ec00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:31.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.061+0000 7f19f3fff700 1 -- 172.21.15.1:0/1429357180 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec011800 con 0x7f19fc0cf090 2024-03-18T20:27:31.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.062+0000 7f19f3fff700 1 -- 172.21.15.1:0/1429357180 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ec011960 con 0x7f19fc0cf090 2024-03-18T20:27:31.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.062+0000 7f1a01a38700 1 -- 172.21.15.1:0/1429357180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19fc0b19b0 con 0x7f19fc0cf090 2024-03-18T20:27:31.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.062+0000 7f19f3fff700 1 -- 172.21.15.1:0/1429357180 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec011cd0 con 0x7f19fc0cf090 2024-03-18T20:27:31.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.062+0000 7f1a01a38700 1 -- 172.21.15.1:0/1429357180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19fc15af40 con 0x7f19fc0cf090 2024-03-18T20:27:31.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.063+0000 7f1a01a38700 1 -- 172.21.15.1:0/1429357180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19fc073350 con 0x7f19fc0cf090 2024-03-18T20:27:31.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.063+0000 7f19f3fff700 1 -- 172.21.15.1:0/1429357180 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f19ec021350 con 0x7f19fc0cf090 2024-03-18T20:27:31.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.063+0000 7f19f3fff700 1 --2- 172.21.15.1:0/1429357180 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f19dc06aed0 0x7f19dc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:31.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.064+0000 7f19f3fff700 1 -- 172.21.15.1:0/1429357180 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f19ec089a20 con 0x7f19fc0cf090 2024-03-18T20:27:31.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.064+0000 7f19fa7fc700 1 --2- 172.21.15.1:0/1429357180 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f19dc06aed0 0x7f19dc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:31.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.064+0000 7f19fa7fc700 1 --2- 172.21.15.1:0/1429357180 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f19dc06aed0 0x7f19dc06d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f19e4000c00 tx=0x7f19e4008030 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:31.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.069+0000 7f19f3fff700 1 -- 172.21.15.1:0/1429357180 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f19ec0545e0 con 0x7f19fc0cf090 2024-03-18T20:27:31.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.266+0000 7f1a01a38700 1 -- 172.21.15.1:0/1429357180 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f19fc0d06d0 con 0x7f19dc06aed0 2024-03-18T20:27:31.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.274+0000 7f19f3fff700 1 -- 172.21.15.1:0/1429357180 <== mgr.12813 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290070 (secure 0 0 0) 0x7f19e400bda0 con 0x7f19dc06aed0 2024-03-18T20:27:31.283 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:31.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.278+0000 7f1a01a38700 1 -- 172.21.15.1:0/1429357180 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f19dc06aed0 msgr2=0x7f19dc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:31.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.278+0000 7f1a01a38700 1 --2- 172.21.15.1:0/1429357180 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f19dc06aed0 0x7f19dc06d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f19e4000c00 tx=0x7f19e4008030 comp rx=0 tx=0).stop 2024-03-18T20:27:31.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.278+0000 7f1a01a38700 1 -- 172.21.15.1:0/1429357180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19fc0cf090 msgr2=0x7f19fc0b11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:31.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.278+0000 7f1a01a38700 1 --2- 172.21.15.1:0/1429357180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19fc0cf090 0x7f19fc0b11a0 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00c750 tx=0x7f19ec00cb10 comp rx=0 tx=0).stop 2024-03-18T20:27:31.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.278+0000 7f1a01a38700 1 -- 172.21.15.1:0/1429357180 shutdown_connections 2024-03-18T20:27:31.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.278+0000 7f1a01a38700 1 --2- 172.21.15.1:0/1429357180 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f19dc06aed0 0x7f19dc06d390 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.278+0000 7f1a01a38700 1 --2- 172.21.15.1:0/1429357180 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f19fc160d20 0x7f19fc15ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.278+0000 7f1a01a38700 1 --2- 172.21.15.1:0/1429357180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19fc0cf090 0x7f19fc0b11a0 unknown :-1 s=CLOSED pgs=3088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.278+0000 7f1a01a38700 1 --2- 172.21.15.1:0/1429357180 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f19fc160940 0x7f19fc14f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.278+0000 7f1a01a38700 1 -- 172.21.15.1:0/1429357180 >> 172.21.15.1:0/1429357180 conn(0x7f19fc09c520 msgr2=0x7f19fc0cf490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:31.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.278+0000 7f1a01a38700 1 -- 172.21.15.1:0/1429357180 shutdown_connections 2024-03-18T20:27:31.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.278+0000 7f1a01a38700 1 -- 172.21.15.1:0/1429357180 wait complete. 2024-03-18T20:27:31.286 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:27:31.295 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-03-18T20:27:30.314776+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":683,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":419,"ondisk_log_size":419,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":435556,"kb_used_data":28264,"kb_used_omap":12,"kb_used_meta":407155,"kb_avail":754539164,"statfs":{"total":773094113280,"available":772648103936,"internally_reserved":0,"allocated":28942336,"data_stored":15538788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":416927289},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.265237"},"pg_stats":[{"pgid":"36.3f","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.059704+0000","last_change":"2024-03-18T20:27:22.069088+0000","last_active":"2024-03-18T20:27:23.059704+0000","last_peered":"2024-03-18T20:27:23.059704+0000","last_clean":"2024-03-18T20:27:23.059704+0000","last_became_active":"2024-03-18T20:27:22.068757+0000","last_became_peered":"2024-03-18T20:27:22.068757+0000","last_unstale":"2024-03-18T20:27:23.059704+0000","last_undegraded":"2024-03-18T20:27:23.059704+0000","last_fullsized":"2024-03-18T20:27:23.059704+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:35:43.126535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3e","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.059815+0000","last_change":"2024-03-18T20:27:22.081802+0000","last_active":"2024-03-18T20:27:23.059815+0000","last_peered":"2024-03-18T20:27:23.059815+0000","last_clean":"2024-03-18T20:27:23.059815+0000","last_became_active":"2024-03-18T20:27:22.081508+0000","last_became_peered":"2024-03-18T20:27:22.081508+0000","last_unstale":"2024-03-18T20:27:23.059815+0000","last_undegraded":"2024-03-18T20:27:23.059815+0000","last_fullsized":"2024-03-18T20:27:23.059815+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:19:52.098982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.3d","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.060362+0000","last_change":"2024-03-18T20:27:22.079317+0000","last_active":"2024-03-18T20:27:23.060362+0000","last_peered":"2024-03-18T20:27:23.060362+0000","last_clean":"2024-03-18T20:27:23.060362+0000","last_became_active":"2024-03-18T20:27:22.074210+0000","last_became_peered":"2024-03-18T20:27:22.074210+0000","last_unstale":"2024-03-18T20:27:23.060362+0000","last_undegraded":"2024-03-18T20:27:23.060362+0000","last_fullsized":"2024-03-18T20:27:23.060362+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:34:48.618192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3c","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.060291+0000","last_change":"2024-03-18T20:27:22.082761+0000","last_active":"2024-03-18T20:27:23.060291+0000","last_peered":"2024-03-18T20:27:23.060291+0000","last_clean":"2024-03-18T20:27:23.060291+0000","last_became_active":"2024-03-18T20:27:22.082497+0000","last_became_peered":"2024-03-18T20:27:22.082497+0000","last_unstale":"2024-03-18T20:27:23.060291+0000","last_undegraded":"2024-03-18T20:27:23.060291+0000","last_fullsized":"2024-03-18T20:27:23.060291+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:18:47.150954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.3b","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218004+0000","last_change":"2024-03-18T20:27:22.055309+0000","last_active":"2024-03-18T20:27:24.218004+0000","last_peered":"2024-03-18T20:27:24.218004+0000","last_clean":"2024-03-18T20:27:24.218004+0000","last_became_active":"2024-03-18T20:27:22.055064+0000","last_became_peered":"2024-03-18T20:27:22.055064+0000","last_unstale":"2024-03-18T20:27:24.218004+0000","last_undegraded":"2024-03-18T20:27:24.218004+0000","last_fullsized":"2024-03-18T20:27:24.218004+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:19.253084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.3a","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.217979+0000","last_change":"2024-03-18T20:27:22.062816+0000","last_active":"2024-03-18T20:27:24.217979+0000","last_peered":"2024-03-18T20:27:24.217979+0000","last_clean":"2024-03-18T20:27:24.217979+0000","last_became_active":"2024-03-18T20:27:22.062600+0000","last_became_peered":"2024-03-18T20:27:22.062600+0000","last_unstale":"2024-03-18T20:27:24.217979+0000","last_undegraded":"2024-03-18T20:27:24.217979+0000","last_fullsized":"2024-03-18T20:27:24.217979+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:15:52.725667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.39","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.539805+0000","last_change":"2024-03-18T20:27:22.072417+0000","last_active":"2024-03-18T20:27:23.539805+0000","last_peered":"2024-03-18T20:27:23.539805+0000","last_clean":"2024-03-18T20:27:23.539805+0000","last_became_active":"2024-03-18T20:27:22.072073+0000","last_became_peered":"2024-03-18T20:27:22.072073+0000","last_unstale":"2024-03-18T20:27:23.539805+0000","last_undegraded":"2024-03-18T20:27:23.539805+0000","last_fullsized":"2024-03-18T20:27:23.539805+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:31:48.142119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.38","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.059976+0000","last_change":"2024-03-18T20:27:22.078524+0000","last_active":"2024-03-18T20:27:23.059976+0000","last_peered":"2024-03-18T20:27:23.059976+0000","last_clean":"2024-03-18T20:27:23.059976+0000","last_became_active":"2024-03-18T20:27:22.078339+0000","last_became_peered":"2024-03-18T20:27:22.078339+0000","last_unstale":"2024-03-18T20:27:23.059976+0000","last_undegraded":"2024-03-18T20:27:23.059976+0000","last_fullsized":"2024-03-18T20:27:23.059976+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:18:35.941658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.37","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.540118+0000","last_change":"2024-03-18T20:27:22.072171+0000","last_active":"2024-03-18T20:27:23.540118+0000","last_peered":"2024-03-18T20:27:23.540118+0000","last_clean":"2024-03-18T20:27:23.540118+0000","last_became_active":"2024-03-18T20:27:22.071860+0000","last_became_peered":"2024-03-18T20:27:22.071860+0000","last_unstale":"2024-03-18T20:27:23.540118+0000","last_undegraded":"2024-03-18T20:27:23.540118+0000","last_fullsized":"2024-03-18T20:27:23.540118+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:01:17.557072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.36","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.540140+0000","last_change":"2024-03-18T20:27:22.070958+0000","last_active":"2024-03-18T20:27:23.540140+0000","last_peered":"2024-03-18T20:27:23.540140+0000","last_clean":"2024-03-18T20:27:23.540140+0000","last_became_active":"2024-03-18T20:27:22.070673+0000","last_became_peered":"2024-03-18T20:27:22.070673+0000","last_unstale":"2024-03-18T20:27:23.540140+0000","last_undegraded":"2024-03-18T20:27:23.540140+0000","last_fullsized":"2024-03-18T20:27:23.540140+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:18:35.941658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.35","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.789497+0000","last_change":"2024-03-18T20:27:22.065852+0000","last_active":"2024-03-18T20:27:24.789497+0000","last_peered":"2024-03-18T20:27:24.789497+0000","last_clean":"2024-03-18T20:27:24.789497+0000","last_became_active":"2024-03-18T20:27:22.065591+0000","last_became_peered":"2024-03-18T20:27:22.065591+0000","last_unstale":"2024-03-18T20:27:24.789497+0000","last_undegraded":"2024-03-18T20:27:24.789497+0000","last_fullsized":"2024-03-18T20:27:24.789497+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:00:17.291074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.34","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.059567+0000","last_change":"2024-03-18T20:27:22.068122+0000","last_active":"2024-03-18T20:27:23.059567+0000","last_peered":"2024-03-18T20:27:23.059567+0000","last_clean":"2024-03-18T20:27:23.059567+0000","last_became_active":"2024-03-18T20:27:22.067807+0000","last_became_peered":"2024-03-18T20:27:22.067807+0000","last_unstale":"2024-03-18T20:27:23.059567+0000","last_undegraded":"2024-03-18T20:27:23.059567+0000","last_fullsized":"2024-03-18T20:27:23.059567+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:26:41.614115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.33","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.060290+0000","last_change":"2024-03-18T20:27:22.082276+0000","last_active":"2024-03-18T20:27:23.060290+0000","last_peered":"2024-03-18T20:27:23.060290+0000","last_clean":"2024-03-18T20:27:23.060290+0000","last_became_active":"2024-03-18T20:27:22.082106+0000","last_became_peered":"2024-03-18T20:27:22.082106+0000","last_unstale":"2024-03-18T20:27:23.060290+0000","last_undegraded":"2024-03-18T20:27:23.060290+0000","last_fullsized":"2024-03-18T20:27:23.060290+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:17:03.674418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.32","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.539938+0000","last_change":"2024-03-18T20:27:22.075857+0000","last_active":"2024-03-18T20:27:23.539938+0000","last_peered":"2024-03-18T20:27:23.539938+0000","last_clean":"2024-03-18T20:27:23.539938+0000","last_became_active":"2024-03-18T20:27:22.075579+0000","last_became_peered":"2024-03-18T20:27:22.075579+0000","last_unstale":"2024-03-18T20:27:23.539938+0000","last_undegraded":"2024-03-18T20:27:23.539938+0000","last_fullsized":"2024-03-18T20:27:23.539938+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:47.940288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.31","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.060061+0000","last_change":"2024-03-18T20:27:22.079873+0000","last_active":"2024-03-18T20:27:23.060061+0000","last_peered":"2024-03-18T20:27:23.060061+0000","last_clean":"2024-03-18T20:27:23.060061+0000","last_became_active":"2024-03-18T20:27:22.075416+0000","last_became_peered":"2024-03-18T20:27:22.075416+0000","last_unstale":"2024-03-18T20:27:23.060061+0000","last_undegraded":"2024-03-18T20:27:23.060061+0000","last_fullsized":"2024-03-18T20:27:23.060061+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:01:17.557072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.30","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:25.159271+0000","last_change":"2024-03-18T20:27:22.073247+0000","last_active":"2024-03-18T20:27:25.159271+0000","last_peered":"2024-03-18T20:27:25.159271+0000","last_clean":"2024-03-18T20:27:25.159271+0000","last_became_active":"2024-03-18T20:27:22.072948+0000","last_became_peered":"2024-03-18T20:27:22.072948+0000","last_unstale":"2024-03-18T20:27:25.159271+0000","last_undegraded":"2024-03-18T20:27:25.159271+0000","last_fullsized":"2024-03-18T20:27:25.159271+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:01:32.387081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2f","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:25.159500+0000","last_change":"2024-03-18T20:27:22.071359+0000","last_active":"2024-03-18T20:27:25.159500+0000","last_peered":"2024-03-18T20:27:25.159500+0000","last_clean":"2024-03-18T20:27:25.159500+0000","last_became_active":"2024-03-18T20:27:22.071086+0000","last_became_peered":"2024-03-18T20:27:22.071086+0000","last_unstale":"2024-03-18T20:27:25.159500+0000","last_undegraded":"2024-03-18T20:27:25.159500+0000","last_fullsized":"2024-03-18T20:27:25.159500+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:30:51.011555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2e","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.060515+0000","last_change":"2024-03-18T20:27:22.078433+0000","last_active":"2024-03-18T20:27:23.060515+0000","last_peered":"2024-03-18T20:27:23.060515+0000","last_clean":"2024-03-18T20:27:23.060515+0000","last_became_active":"2024-03-18T20:27:22.078200+0000","last_became_peered":"2024-03-18T20:27:22.078200+0000","last_unstale":"2024-03-18T20:27:23.060515+0000","last_undegraded":"2024-03-18T20:27:23.060515+0000","last_fullsized":"2024-03-18T20:27:23.060515+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:57:58.393225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2d","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:25.159507+0000","last_change":"2024-03-18T20:27:22.080594+0000","last_active":"2024-03-18T20:27:25.159507+0000","last_peered":"2024-03-18T20:27:25.159507+0000","last_clean":"2024-03-18T20:27:25.159507+0000","last_became_active":"2024-03-18T20:27:22.080285+0000","last_became_peered":"2024-03-18T20:27:22.080285+0000","last_unstale":"2024-03-18T20:27:25.159507+0000","last_undegraded":"2024-03-18T20:27:25.159507+0000","last_fullsized":"2024-03-18T20:27:25.159507+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:57:58.393225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2c","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:25.159613+0000","last_change":"2024-03-18T20:27:22.075878+0000","last_active":"2024-03-18T20:27:25.159613+0000","last_peered":"2024-03-18T20:27:25.159613+0000","last_clean":"2024-03-18T20:27:25.159613+0000","last_became_active":"2024-03-18T20:27:22.073489+0000","last_became_peered":"2024-03-18T20:27:22.073489+0000","last_unstale":"2024-03-18T20:27:25.159613+0000","last_undegraded":"2024-03-18T20:27:25.159613+0000","last_fullsized":"2024-03-18T20:27:25.159613+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:31:57.492312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.2b","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.789487+0000","last_change":"2024-03-18T20:27:22.070710+0000","last_active":"2024-03-18T20:27:24.789487+0000","last_peered":"2024-03-18T20:27:24.789487+0000","last_clean":"2024-03-18T20:27:24.789487+0000","last_became_active":"2024-03-18T20:27:22.070237+0000","last_became_peered":"2024-03-18T20:27:22.070237+0000","last_unstale":"2024-03-18T20:27:24.789487+0000","last_undegraded":"2024-03-18T20:27:24.789487+0000","last_fullsized":"2024-03-18T20:27:24.789487+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:33:29.419055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.2a","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.657955+0000","last_change":"2024-03-18T20:27:22.071230+0000","last_active":"2024-03-18T20:27:24.657955+0000","last_peered":"2024-03-18T20:27:24.657955+0000","last_clean":"2024-03-18T20:27:24.657955+0000","last_became_active":"2024-03-18T20:27:22.070949+0000","last_became_peered":"2024-03-18T20:27:22.070949+0000","last_unstale":"2024-03-18T20:27:24.657955+0000","last_undegraded":"2024-03-18T20:27:24.657955+0000","last_fullsized":"2024-03-18T20:27:24.657955+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:03:27.591360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.29","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.539909+0000","last_change":"2024-03-18T20:27:22.070551+0000","last_active":"2024-03-18T20:27:23.539909+0000","last_peered":"2024-03-18T20:27:23.539909+0000","last_clean":"2024-03-18T20:27:23.539909+0000","last_became_active":"2024-03-18T20:27:22.070271+0000","last_became_peered":"2024-03-18T20:27:22.070271+0000","last_unstale":"2024-03-18T20:27:23.539909+0000","last_undegraded":"2024-03-18T20:27:23.539909+0000","last_fullsized":"2024-03-18T20:27:23.539909+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:22:19.431509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.28","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.657865+0000","last_change":"2024-03-18T20:27:22.065580+0000","last_active":"2024-03-18T20:27:24.657865+0000","last_peered":"2024-03-18T20:27:24.657865+0000","last_clean":"2024-03-18T20:27:24.657865+0000","last_became_active":"2024-03-18T20:27:22.065288+0000","last_became_peered":"2024-03-18T20:27:22.065288+0000","last_unstale":"2024-03-18T20:27:24.657865+0000","last_undegraded":"2024-03-18T20:27:24.657865+0000","last_fullsized":"2024-03-18T20:27:24.657865+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:17:53.531289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.27","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.059784+0000","last_change":"2024-03-18T20:27:22.064810+0000","last_active":"2024-03-18T20:27:23.059784+0000","last_peered":"2024-03-18T20:27:23.059784+0000","last_clean":"2024-03-18T20:27:23.059784+0000","last_became_active":"2024-03-18T20:27:22.064514+0000","last_became_peered":"2024-03-18T20:27:22.064514+0000","last_unstale":"2024-03-18T20:27:23.059784+0000","last_undegraded":"2024-03-18T20:27:23.059784+0000","last_fullsized":"2024-03-18T20:27:23.059784+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:49:39.166635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.26","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218003+0000","last_change":"2024-03-18T20:27:22.062897+0000","last_active":"2024-03-18T20:27:24.218003+0000","last_peered":"2024-03-18T20:27:24.218003+0000","last_clean":"2024-03-18T20:27:24.218003+0000","last_became_active":"2024-03-18T20:27:22.062639+0000","last_became_peered":"2024-03-18T20:27:22.062639+0000","last_unstale":"2024-03-18T20:27:24.218003+0000","last_undegraded":"2024-03-18T20:27:24.218003+0000","last_fullsized":"2024-03-18T20:27:24.218003+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:45.973738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.24","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.217918+0000","last_change":"2024-03-18T20:27:22.073284+0000","last_active":"2024-03-18T20:27:24.217918+0000","last_peered":"2024-03-18T20:27:24.217918+0000","last_clean":"2024-03-18T20:27:24.217918+0000","last_became_active":"2024-03-18T20:27:22.072955+0000","last_became_peered":"2024-03-18T20:27:22.072955+0000","last_unstale":"2024-03-18T20:27:24.217918+0000","last_undegraded":"2024-03-18T20:27:24.217918+0000","last_fullsized":"2024-03-18T20:27:24.217918+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:10.708304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.23","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.658050+0000","last_change":"2024-03-18T20:27:22.072312+0000","last_active":"2024-03-18T20:27:24.658050+0000","last_peered":"2024-03-18T20:27:24.658050+0000","last_clean":"2024-03-18T20:27:24.658050+0000","last_became_active":"2024-03-18T20:27:22.072058+0000","last_became_peered":"2024-03-18T20:27:22.072058+0000","last_unstale":"2024-03-18T20:27:24.658050+0000","last_undegraded":"2024-03-18T20:27:24.658050+0000","last_fullsized":"2024-03-18T20:27:24.658050+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:38:33.024690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.22","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.337111+0000","last_change":"2024-03-18T20:27:22.067113+0000","last_active":"2024-03-18T20:27:24.337111+0000","last_peered":"2024-03-18T20:27:24.337111+0000","last_clean":"2024-03-18T20:27:24.337111+0000","last_became_active":"2024-03-18T20:27:22.066844+0000","last_became_peered":"2024-03-18T20:27:22.066844+0000","last_unstale":"2024-03-18T20:27:24.337111+0000","last_undegraded":"2024-03-18T20:27:24.337111+0000","last_fullsized":"2024-03-18T20:27:24.337111+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:31:57.492312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.21","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.657914+0000","last_change":"2024-03-18T20:27:22.073297+0000","last_active":"2024-03-18T20:27:24.657914+0000","last_peered":"2024-03-18T20:27:24.657914+0000","last_clean":"2024-03-18T20:27:24.657914+0000","last_became_active":"2024-03-18T20:27:22.073062+0000","last_became_peered":"2024-03-18T20:27:22.073062+0000","last_unstale":"2024-03-18T20:27:24.657914+0000","last_undegraded":"2024-03-18T20:27:24.657914+0000","last_fullsized":"2024-03-18T20:27:24.657914+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:45.973738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.20","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.060017+0000","last_change":"2024-03-18T20:27:22.078608+0000","last_active":"2024-03-18T20:27:23.060017+0000","last_peered":"2024-03-18T20:27:23.060017+0000","last_clean":"2024-03-18T20:27:23.060017+0000","last_became_active":"2024-03-18T20:27:22.078351+0000","last_became_peered":"2024-03-18T20:27:22.078351+0000","last_unstale":"2024-03-18T20:27:23.060017+0000","last_undegraded":"2024-03-18T20:27:23.060017+0000","last_fullsized":"2024-03-18T20:27:23.060017+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:48:31.349008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.657985+0000","last_change":"2024-03-18T20:27:22.061326+0000","last_active":"2024-03-18T20:27:24.657985+0000","last_peered":"2024-03-18T20:27:24.657985+0000","last_clean":"2024-03-18T20:27:24.657985+0000","last_became_active":"2024-03-18T20:27:22.061073+0000","last_became_peered":"2024-03-18T20:27:22.061073+0000","last_unstale":"2024-03-18T20:27:24.657985+0000","last_undegraded":"2024-03-18T20:27:24.657985+0000","last_fullsized":"2024-03-18T20:27:24.657985+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:49.309018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"273'4","reported_seq":15,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.077431+0000","last_change":"2024-03-18T20:27:21.631963+0000","last_active":"2024-03-18T20:27:22.077431+0000","last_peered":"2024-03-18T20:27:22.077431+0000","last_clean":"2024-03-18T20:27:22.077431+0000","last_became_active":"2024-03-18T20:27:21.631625+0000","last_became_peered":"2024-03-18T20:27:21.631625+0000","last_unstale":"2024-03-18T20:27:22.077431+0000","last_undegraded":"2024-03-18T20:27:22.077431+0000","last_fullsized":"2024-03-18T20:27:22.077431+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:01.698353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.539964+0000","last_change":"2024-03-18T20:27:22.061339+0000","last_active":"2024-03-18T20:27:23.539964+0000","last_peered":"2024-03-18T20:27:23.539964+0000","last_clean":"2024-03-18T20:27:23.539964+0000","last_became_active":"2024-03-18T20:27:22.061043+0000","last_became_peered":"2024-03-18T20:27:22.061043+0000","last_unstale":"2024-03-18T20:27:23.539964+0000","last_undegraded":"2024-03-18T20:27:23.539964+0000","last_fullsized":"2024-03-18T20:27:23.539964+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:37:49.743157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1e","version":"273'1","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.069949+0000","last_change":"2024-03-18T20:27:21.633880+0000","last_active":"2024-03-18T20:27:22.069949+0000","last_peered":"2024-03-18T20:27:22.069949+0000","last_clean":"2024-03-18T20:27:22.069949+0000","last_became_active":"2024-03-18T20:27:21.633600+0000","last_became_peered":"2024-03-18T20:27:21.633600+0000","last_unstale":"2024-03-18T20:27:22.069949+0000","last_undegraded":"2024-03-18T20:27:22.069949+0000","last_fullsized":"2024-03-18T20:27:22.069949+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:02:14.143177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.059675+0000","last_change":"2024-03-18T20:27:22.070382+0000","last_active":"2024-03-18T20:27:23.059675+0000","last_peered":"2024-03-18T20:27:23.059675+0000","last_clean":"2024-03-18T20:27:23.059675+0000","last_became_active":"2024-03-18T20:27:22.070071+0000","last_became_peered":"2024-03-18T20:27:22.070071+0000","last_unstale":"2024-03-18T20:27:23.059675+0000","last_undegraded":"2024-03-18T20:27:23.059675+0000","last_fullsized":"2024-03-18T20:27:23.059675+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:48:31.349008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1d","version":"273'2","reported_seq":13,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.072917+0000","last_change":"2024-03-18T20:27:21.642443+0000","last_active":"2024-03-18T20:27:22.072917+0000","last_peered":"2024-03-18T20:27:22.072917+0000","last_clean":"2024-03-18T20:27:22.072917+0000","last_became_active":"2024-03-18T20:27:21.642135+0000","last_became_peered":"2024-03-18T20:27:21.642135+0000","last_unstale":"2024-03-18T20:27:22.072917+0000","last_undegraded":"2024-03-18T20:27:22.072917+0000","last_fullsized":"2024-03-18T20:27:22.072917+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:42:23.110707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1b","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.540039+0000","last_change":"2024-03-18T20:27:22.068249+0000","last_active":"2024-03-18T20:27:23.540039+0000","last_peered":"2024-03-18T20:27:23.540039+0000","last_clean":"2024-03-18T20:27:23.540039+0000","last_became_active":"2024-03-18T20:27:22.067275+0000","last_became_peered":"2024-03-18T20:27:22.067275+0000","last_unstale":"2024-03-18T20:27:23.540039+0000","last_undegraded":"2024-03-18T20:27:23.540039+0000","last_fullsized":"2024-03-18T20:27:23.540039+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:21:41.361502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.1c","version":"273'2","reported_seq":13,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.072514+0000","last_change":"2024-03-18T20:27:21.643771+0000","last_active":"2024-03-18T20:27:22.072514+0000","last_peered":"2024-03-18T20:27:22.072514+0000","last_clean":"2024-03-18T20:27:22.072514+0000","last_became_active":"2024-03-18T20:27:21.643477+0000","last_became_peered":"2024-03-18T20:27:21.643477+0000","last_unstale":"2024-03-18T20:27:22.072514+0000","last_undegraded":"2024-03-18T20:27:22.072514+0000","last_fullsized":"2024-03-18T20:27:22.072514+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:45:52.316434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1c","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.060194+0000","last_change":"2024-03-18T20:27:22.077781+0000","last_active":"2024-03-18T20:27:23.060194+0000","last_peered":"2024-03-18T20:27:23.060194+0000","last_clean":"2024-03-18T20:27:23.060194+0000","last_became_active":"2024-03-18T20:27:22.077525+0000","last_became_peered":"2024-03-18T20:27:22.077525+0000","last_unstale":"2024-03-18T20:27:23.060194+0000","last_undegraded":"2024-03-18T20:27:23.060194+0000","last_fullsized":"2024-03-18T20:27:23.060194+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.116186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1b","version":"273'1","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.083104+0000","last_change":"2024-03-18T20:27:21.638424+0000","last_active":"2024-03-18T20:27:22.083104+0000","last_peered":"2024-03-18T20:27:22.083104+0000","last_clean":"2024-03-18T20:27:22.083104+0000","last_became_active":"2024-03-18T20:27:21.638091+0000","last_became_peered":"2024-03-18T20:27:21.638091+0000","last_unstale":"2024-03-18T20:27:22.083104+0000","last_undegraded":"2024-03-18T20:27:22.083104+0000","last_fullsized":"2024-03-18T20:27:22.083104+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:08:18.189309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.658101+0000","last_change":"2024-03-18T20:27:22.063893+0000","last_active":"2024-03-18T20:27:24.658101+0000","last_peered":"2024-03-18T20:27:24.658101+0000","last_clean":"2024-03-18T20:27:24.658101+0000","last_became_active":"2024-03-18T20:27:22.063639+0000","last_became_peered":"2024-03-18T20:27:22.063639+0000","last_unstale":"2024-03-18T20:27:24.658101+0000","last_undegraded":"2024-03-18T20:27:24.658101+0000","last_fullsized":"2024-03-18T20:27:24.658101+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:19.253084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1a","version":"273'1","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.069050+0000","last_change":"2024-03-18T20:27:21.635634+0000","last_active":"2024-03-18T20:27:22.069050+0000","last_peered":"2024-03-18T20:27:22.069050+0000","last_clean":"2024-03-18T20:27:22.069050+0000","last_became_active":"2024-03-18T20:27:21.635366+0000","last_became_peered":"2024-03-18T20:27:21.635366+0000","last_unstale":"2024-03-18T20:27:22.069050+0000","last_undegraded":"2024-03-18T20:27:22.069050+0000","last_fullsized":"2024-03-18T20:27:22.069050+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:40:40.016554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.658257+0000","last_change":"2024-03-18T20:27:22.080388+0000","last_active":"2024-03-18T20:27:24.658257+0000","last_peered":"2024-03-18T20:27:24.658257+0000","last_clean":"2024-03-18T20:27:24.658257+0000","last_became_active":"2024-03-18T20:27:22.080127+0000","last_became_peered":"2024-03-18T20:27:22.080127+0000","last_unstale":"2024-03-18T20:27:24.658257+0000","last_undegraded":"2024-03-18T20:27:24.658257+0000","last_fullsized":"2024-03-18T20:27:24.658257+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:55:42.255610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.053244+0000","last_change":"2024-03-18T20:27:21.639473+0000","last_active":"2024-03-18T20:27:22.053244+0000","last_peered":"2024-03-18T20:27:22.053244+0000","last_clean":"2024-03-18T20:27:22.053244+0000","last_became_active":"2024-03-18T20:27:21.639123+0000","last_became_peered":"2024-03-18T20:27:21.639123+0000","last_unstale":"2024-03-18T20:27:22.053244+0000","last_undegraded":"2024-03-18T20:27:22.053244+0000","last_fullsized":"2024-03-18T20:27:22.053244+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:08:18.189309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1f","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:25.159569+0000","last_change":"2024-03-18T20:27:22.071473+0000","last_active":"2024-03-18T20:27:25.159569+0000","last_peered":"2024-03-18T20:27:25.159569+0000","last_clean":"2024-03-18T20:27:25.159569+0000","last_became_active":"2024-03-18T20:27:22.070719+0000","last_became_peered":"2024-03-18T20:27:22.070719+0000","last_unstale":"2024-03-18T20:27:25.159569+0000","last_undegraded":"2024-03-18T20:27:25.159569+0000","last_fullsized":"2024-03-18T20:27:25.159569+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:02.705525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.067156+0000","last_change":"2024-03-18T20:27:21.641722+0000","last_active":"2024-03-18T20:27:22.067156+0000","last_peered":"2024-03-18T20:27:22.067156+0000","last_clean":"2024-03-18T20:27:22.067156+0000","last_became_active":"2024-03-18T20:27:21.641405+0000","last_became_peered":"2024-03-18T20:27:21.641405+0000","last_unstale":"2024-03-18T20:27:22.067156+0000","last_undegraded":"2024-03-18T20:27:22.067156+0000","last_fullsized":"2024-03-18T20:27:22.067156+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:49:10.342403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.10","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.060136+0000","last_change":"2024-03-18T20:27:22.077978+0000","last_active":"2024-03-18T20:27:23.060136+0000","last_peered":"2024-03-18T20:27:23.060136+0000","last_clean":"2024-03-18T20:27:23.060136+0000","last_became_active":"2024-03-18T20:27:22.067414+0000","last_became_peered":"2024-03-18T20:27:22.067414+0000","last_unstale":"2024-03-18T20:27:23.060136+0000","last_undegraded":"2024-03-18T20:27:23.060136+0000","last_fullsized":"2024-03-18T20:27:23.060136+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:41:35.684678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.061917+0000","last_change":"2024-03-18T20:27:21.642198+0000","last_active":"2024-03-18T20:27:22.061917+0000","last_peered":"2024-03-18T20:27:22.061917+0000","last_clean":"2024-03-18T20:27:22.061917+0000","last_became_active":"2024-03-18T20:27:21.641867+0000","last_became_peered":"2024-03-18T20:27:21.641867+0000","last_unstale":"2024-03-18T20:27:22.061917+0000","last_undegraded":"2024-03-18T20:27:22.061917+0000","last_fullsized":"2024-03-18T20:27:22.061917+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:28:52.874751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.059512+0000","last_change":"2024-03-18T20:27:22.076792+0000","last_active":"2024-03-18T20:27:23.059512+0000","last_peered":"2024-03-18T20:27:23.059512+0000","last_clean":"2024-03-18T20:27:23.059512+0000","last_became_active":"2024-03-18T20:27:22.076508+0000","last_became_peered":"2024-03-18T20:27:22.076508+0000","last_unstale":"2024-03-18T20:27:23.059512+0000","last_undegraded":"2024-03-18T20:27:23.059512+0000","last_fullsized":"2024-03-18T20:27:23.059512+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:34:48.618192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.053047+0000","last_change":"2024-03-18T20:27:21.642736+0000","last_active":"2024-03-18T20:27:22.053047+0000","last_peered":"2024-03-18T20:27:22.053047+0000","last_clean":"2024-03-18T20:27:22.053047+0000","last_became_active":"2024-03-18T20:27:21.637124+0000","last_became_peered":"2024-03-18T20:27:21.637124+0000","last_unstale":"2024-03-18T20:27:22.053047+0000","last_undegraded":"2024-03-18T20:27:22.053047+0000","last_fullsized":"2024-03-18T20:27:22.053047+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:21:19.006472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.060206+0000","last_change":"2024-03-18T20:27:22.068020+0000","last_active":"2024-03-18T20:27:23.060206+0000","last_peered":"2024-03-18T20:27:23.060206+0000","last_clean":"2024-03-18T20:27:23.060206+0000","last_became_active":"2024-03-18T20:27:22.067760+0000","last_became_peered":"2024-03-18T20:27:22.067760+0000","last_unstale":"2024-03-18T20:27:23.060206+0000","last_undegraded":"2024-03-18T20:27:23.060206+0000","last_fullsized":"2024-03-18T20:27:23.060206+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:35:43.126535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"273'3","reported_seq":14,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.075904+0000","last_change":"2024-03-18T20:27:21.642710+0000","last_active":"2024-03-18T20:27:22.075904+0000","last_peered":"2024-03-18T20:27:22.075904+0000","last_clean":"2024-03-18T20:27:22.075904+0000","last_became_active":"2024-03-18T20:27:21.642424+0000","last_became_peered":"2024-03-18T20:27:21.642424+0000","last_unstale":"2024-03-18T20:27:22.075904+0000","last_undegraded":"2024-03-18T20:27:22.075904+0000","last_fullsized":"2024-03-18T20:27:22.075904+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:29.812382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218074+0000","last_change":"2024-03-18T20:27:22.081779+0000","last_active":"2024-03-18T20:27:24.218074+0000","last_peered":"2024-03-18T20:27:24.218074+0000","last_clean":"2024-03-18T20:27:24.218074+0000","last_became_active":"2024-03-18T20:27:22.081506+0000","last_became_peered":"2024-03-18T20:27:22.081506+0000","last_unstale":"2024-03-18T20:27:24.218074+0000","last_undegraded":"2024-03-18T20:27:24.218074+0000","last_fullsized":"2024-03-18T20:27:24.218074+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:29:27.205683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.14","version":"273'6","reported_seq":17,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.084664+0000","last_change":"2024-03-18T20:27:21.642995+0000","last_active":"2024-03-18T20:27:22.084664+0000","last_peered":"2024-03-18T20:27:22.084664+0000","last_clean":"2024-03-18T20:27:22.084664+0000","last_became_active":"2024-03-18T20:27:21.642713+0000","last_became_peered":"2024-03-18T20:27:21.642713+0000","last_unstale":"2024-03-18T20:27:22.084664+0000","last_undegraded":"2024-03-18T20:27:22.084664+0000","last_fullsized":"2024-03-18T20:27:22.084664+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:01:31.379909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.059568+0000","last_change":"2024-03-18T20:27:22.068182+0000","last_active":"2024-03-18T20:27:23.059568+0000","last_peered":"2024-03-18T20:27:23.059568+0000","last_clean":"2024-03-18T20:27:23.059568+0000","last_became_active":"2024-03-18T20:27:22.067886+0000","last_became_peered":"2024-03-18T20:27:22.067886+0000","last_unstale":"2024-03-18T20:27:23.059568+0000","last_undegraded":"2024-03-18T20:27:23.059568+0000","last_fullsized":"2024-03-18T20:27:23.059568+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:18:47.150954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.13","version":"273'2","reported_seq":13,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.077949+0000","last_change":"2024-03-18T20:27:21.639798+0000","last_active":"2024-03-18T20:27:22.077949+0000","last_peered":"2024-03-18T20:27:22.077949+0000","last_clean":"2024-03-18T20:27:22.077949+0000","last_became_active":"2024-03-18T20:27:21.639569+0000","last_became_peered":"2024-03-18T20:27:21.639569+0000","last_unstale":"2024-03-18T20:27:22.077949+0000","last_undegraded":"2024-03-18T20:27:22.077949+0000","last_fullsized":"2024-03-18T20:27:22.077949+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:42.339058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.15","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.060406+0000","last_change":"2024-03-18T20:27:22.079520+0000","last_active":"2024-03-18T20:27:23.060406+0000","last_peered":"2024-03-18T20:27:23.060406+0000","last_clean":"2024-03-18T20:27:23.060406+0000","last_became_active":"2024-03-18T20:27:22.078866+0000","last_became_peered":"2024-03-18T20:27:22.078866+0000","last_unstale":"2024-03-18T20:27:23.060406+0000","last_undegraded":"2024-03-18T20:27:23.060406+0000","last_fullsized":"2024-03-18T20:27:23.060406+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:31:57.492312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.057508+0000","last_change":"2024-03-18T20:27:21.629305+0000","last_active":"2024-03-18T20:27:22.057508+0000","last_peered":"2024-03-18T20:27:22.057508+0000","last_clean":"2024-03-18T20:27:22.057508+0000","last_became_active":"2024-03-18T20:27:21.629026+0000","last_became_peered":"2024-03-18T20:27:21.629026+0000","last_unstale":"2024-03-18T20:27:22.057508+0000","last_undegraded":"2024-03-18T20:27:22.057508+0000","last_fullsized":"2024-03-18T20:27:22.057508+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:42.339058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:25.159432+0000","last_change":"2024-03-18T20:27:22.070621+0000","last_active":"2024-03-18T20:27:25.159432+0000","last_peered":"2024-03-18T20:27:25.159432+0000","last_clean":"2024-03-18T20:27:25.159432+0000","last_became_active":"2024-03-18T20:27:22.070364+0000","last_became_peered":"2024-03-18T20:27:22.070364+0000","last_unstale":"2024-03-18T20:27:25.159432+0000","last_undegraded":"2024-03-18T20:27:25.159432+0000","last_fullsized":"2024-03-18T20:27:25.159432+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.116186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.055651+0000","last_change":"2024-03-18T20:27:21.634956+0000","last_active":"2024-03-18T20:27:22.055651+0000","last_peered":"2024-03-18T20:27:22.055651+0000","last_clean":"2024-03-18T20:27:22.055651+0000","last_became_active":"2024-03-18T20:27:21.634661+0000","last_became_peered":"2024-03-18T20:27:21.634661+0000","last_unstale":"2024-03-18T20:27:22.055651+0000","last_undegraded":"2024-03-18T20:27:22.055651+0000","last_fullsized":"2024-03-18T20:27:22.055651+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:43.006246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:25.159611+0000","last_change":"2024-03-18T20:27:22.082807+0000","last_active":"2024-03-18T20:27:25.159611+0000","last_peered":"2024-03-18T20:27:25.159611+0000","last_clean":"2024-03-18T20:27:25.159611+0000","last_became_active":"2024-03-18T20:27:22.082581+0000","last_became_peered":"2024-03-18T20:27:22.082581+0000","last_unstale":"2024-03-18T20:27:25.159611+0000","last_undegraded":"2024-03-18T20:27:25.159611+0000","last_fullsized":"2024-03-18T20:27:25.159611+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:19:52.098982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"273'8","reported_seq":19,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.078076+0000","last_change":"2024-03-18T20:27:21.633877+0000","last_active":"2024-03-18T20:27:22.078076+0000","last_peered":"2024-03-18T20:27:22.078076+0000","last_clean":"2024-03-18T20:27:22.078076+0000","last_became_active":"2024-03-18T20:27:21.633593+0000","last_became_peered":"2024-03-18T20:27:21.633593+0000","last_unstale":"2024-03-18T20:27:22.078076+0000","last_undegraded":"2024-03-18T20:27:22.078076+0000","last_fullsized":"2024-03-18T20:27:22.078076+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:31:56.485139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.217869+0000","last_change":"2024-03-18T20:27:22.054635+0000","last_active":"2024-03-18T20:27:24.217869+0000","last_peered":"2024-03-18T20:27:24.217869+0000","last_clean":"2024-03-18T20:27:24.217869+0000","last_became_active":"2024-03-18T20:27:22.054372+0000","last_became_peered":"2024-03-18T20:27:22.054372+0000","last_unstale":"2024-03-18T20:27:24.217869+0000","last_undegraded":"2024-03-18T20:27:24.217869+0000","last_fullsized":"2024-03-18T20:27:24.217869+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:56:40.826497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.f","version":"273'1","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.064735+0000","last_change":"2024-03-18T20:27:21.637341+0000","last_active":"2024-03-18T20:27:22.064735+0000","last_peered":"2024-03-18T20:27:22.064735+0000","last_clean":"2024-03-18T20:27:22.064735+0000","last_became_active":"2024-03-18T20:27:21.636972+0000","last_became_peered":"2024-03-18T20:27:21.636972+0000","last_unstale":"2024-03-18T20:27:22.064735+0000","last_undegraded":"2024-03-18T20:27:22.064735+0000","last_fullsized":"2024-03-18T20:27:22.064735+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:17:27.485874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.657938+0000","last_change":"2024-03-18T20:27:22.066291+0000","last_active":"2024-03-18T20:27:24.657938+0000","last_peered":"2024-03-18T20:27:24.657938+0000","last_clean":"2024-03-18T20:27:24.657938+0000","last_became_active":"2024-03-18T20:27:22.066043+0000","last_became_peered":"2024-03-18T20:27:22.066043+0000","last_unstale":"2024-03-18T20:27:24.657938+0000","last_undegraded":"2024-03-18T20:27:24.657938+0000","last_fullsized":"2024-03-18T20:27:24.657938+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:52:10.708304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.e","version":"273'2","reported_seq":13,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.083274+0000","last_change":"2024-03-18T20:27:21.640607+0000","last_active":"2024-03-18T20:27:22.083274+0000","last_peered":"2024-03-18T20:27:22.083274+0000","last_clean":"2024-03-18T20:27:22.083274+0000","last_became_active":"2024-03-18T20:27:21.640341+0000","last_became_peered":"2024-03-18T20:27:21.640341+0000","last_unstale":"2024-03-18T20:27:22.083274+0000","last_undegraded":"2024-03-18T20:27:22.083274+0000","last_fullsized":"2024-03-18T20:27:22.083274+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:20:18.042657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218120+0000","last_change":"2024-03-18T20:27:22.055634+0000","last_active":"2024-03-18T20:27:24.218120+0000","last_peered":"2024-03-18T20:27:24.218120+0000","last_clean":"2024-03-18T20:27:24.218120+0000","last_became_active":"2024-03-18T20:27:22.055397+0000","last_became_peered":"2024-03-18T20:27:22.055397+0000","last_unstale":"2024-03-18T20:27:24.218120+0000","last_undegraded":"2024-03-18T20:27:24.218120+0000","last_fullsized":"2024-03-18T20:27:24.218120+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:29.388876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.d","version":"273'1","reported_seq":12,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.070106+0000","last_change":"2024-03-18T20:27:21.639938+0000","last_active":"2024-03-18T20:27:22.070106+0000","last_peered":"2024-03-18T20:27:22.070106+0000","last_clean":"2024-03-18T20:27:22.070106+0000","last_became_active":"2024-03-18T20:27:21.639661+0000","last_became_peered":"2024-03-18T20:27:21.639661+0000","last_unstale":"2024-03-18T20:27:22.070106+0000","last_undegraded":"2024-03-18T20:27:22.070106+0000","last_fullsized":"2024-03-18T20:27:22.070106+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:58:00.382684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.b","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.060417+0000","last_change":"2024-03-18T20:27:22.080939+0000","last_active":"2024-03-18T20:27:23.060417+0000","last_peered":"2024-03-18T20:27:23.060417+0000","last_clean":"2024-03-18T20:27:23.060417+0000","last_became_active":"2024-03-18T20:27:22.076643+0000","last_became_peered":"2024-03-18T20:27:22.076643+0000","last_unstale":"2024-03-18T20:27:23.060417+0000","last_undegraded":"2024-03-18T20:27:23.060417+0000","last_fullsized":"2024-03-18T20:27:23.060417+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:49:39.166635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.053127+0000","last_change":"2024-03-18T20:27:21.637564+0000","last_active":"2024-03-18T20:27:22.053127+0000","last_peered":"2024-03-18T20:27:22.053127+0000","last_clean":"2024-03-18T20:27:22.053127+0000","last_became_active":"2024-03-18T20:27:21.637209+0000","last_became_peered":"2024-03-18T20:27:21.637209+0000","last_unstale":"2024-03-18T20:27:22.053127+0000","last_undegraded":"2024-03-18T20:27:22.053127+0000","last_fullsized":"2024-03-18T20:27:22.053127+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:01.555443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.658253+0000","last_change":"2024-03-18T20:27:22.064512+0000","last_active":"2024-03-18T20:27:24.658253+0000","last_peered":"2024-03-18T20:27:24.658253+0000","last_clean":"2024-03-18T20:27:24.658253+0000","last_became_active":"2024-03-18T20:27:22.064274+0000","last_became_peered":"2024-03-18T20:27:22.064274+0000","last_unstale":"2024-03-18T20:27:24.658253+0000","last_undegraded":"2024-03-18T20:27:24.658253+0000","last_fullsized":"2024-03-18T20:27:24.658253+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:29:27.205683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"273'2","reported_seq":13,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.077849+0000","last_change":"2024-03-18T20:27:21.637736+0000","last_active":"2024-03-18T20:27:22.077849+0000","last_peered":"2024-03-18T20:27:22.077849+0000","last_clean":"2024-03-18T20:27:22.077849+0000","last_became_active":"2024-03-18T20:27:21.637006+0000","last_became_peered":"2024-03-18T20:27:21.637006+0000","last_unstale":"2024-03-18T20:27:22.077849+0000","last_undegraded":"2024-03-18T20:27:22.077849+0000","last_fullsized":"2024-03-18T20:27:22.077849+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:52.857794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.658218+0000","last_change":"2024-03-18T20:27:22.071619+0000","last_active":"2024-03-18T20:27:24.658218+0000","last_peered":"2024-03-18T20:27:24.658218+0000","last_clean":"2024-03-18T20:27:24.658218+0000","last_became_active":"2024-03-18T20:27:22.071311+0000","last_became_peered":"2024-03-18T20:27:22.071311+0000","last_unstale":"2024-03-18T20:27:24.658218+0000","last_undegraded":"2024-03-18T20:27:24.658218+0000","last_fullsized":"2024-03-18T20:27:24.658218+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:09:41.096480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.065049+0000","last_change":"2024-03-18T20:27:21.643025+0000","last_active":"2024-03-18T20:27:22.065049+0000","last_peered":"2024-03-18T20:27:22.065049+0000","last_clean":"2024-03-18T20:27:22.065049+0000","last_became_active":"2024-03-18T20:27:21.642207+0000","last_became_peered":"2024-03-18T20:27:21.642207+0000","last_unstale":"2024-03-18T20:27:22.065049+0000","last_undegraded":"2024-03-18T20:27:22.065049+0000","last_fullsized":"2024-03-18T20:27:22.065049+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:13:18.842477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.e","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:25.159564+0000","last_change":"2024-03-18T20:27:22.070308+0000","last_active":"2024-03-18T20:27:25.159564+0000","last_peered":"2024-03-18T20:27:25.159564+0000","last_clean":"2024-03-18T20:27:25.159564+0000","last_became_active":"2024-03-18T20:27:22.070029+0000","last_became_peered":"2024-03-18T20:27:22.070029+0000","last_unstale":"2024-03-18T20:27:25.159564+0000","last_undegraded":"2024-03-18T20:27:25.159564+0000","last_fullsized":"2024-03-18T20:27:25.159564+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:41:35.684678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.059436+0000","last_change":"2024-03-18T20:27:21.639815+0000","last_active":"2024-03-18T20:27:22.059436+0000","last_peered":"2024-03-18T20:27:22.059436+0000","last_clean":"2024-03-18T20:27:22.059436+0000","last_became_active":"2024-03-18T20:27:21.639527+0000","last_became_peered":"2024-03-18T20:27:21.639527+0000","last_unstale":"2024-03-18T20:27:22.059436+0000","last_undegraded":"2024-03-18T20:27:22.059436+0000","last_fullsized":"2024-03-18T20:27:22.059436+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:11:16.795077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.f","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218061+0000","last_change":"2024-03-18T20:27:22.062557+0000","last_active":"2024-03-18T20:27:24.218061+0000","last_peered":"2024-03-18T20:27:24.218061+0000","last_clean":"2024-03-18T20:27:24.218061+0000","last_became_active":"2024-03-18T20:27:22.062283+0000","last_became_peered":"2024-03-18T20:27:22.062283+0000","last_unstale":"2024-03-18T20:27:24.218061+0000","last_undegraded":"2024-03-18T20:27:24.218061+0000","last_fullsized":"2024-03-18T20:27:24.218061+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:05:24.597193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.053002+0000","last_change":"2024-03-18T20:27:21.637405+0000","last_active":"2024-03-18T20:27:22.053002+0000","last_peered":"2024-03-18T20:27:22.053002+0000","last_clean":"2024-03-18T20:27:22.053002+0000","last_became_active":"2024-03-18T20:27:21.636926+0000","last_became_peered":"2024-03-18T20:27:21.636926+0000","last_unstale":"2024-03-18T20:27:22.053002+0000","last_undegraded":"2024-03-18T20:27:22.053002+0000","last_fullsized":"2024-03-18T20:27:22.053002+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:41:34.677506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.060172+0000","last_change":"2024-03-18T20:27:22.077988+0000","last_active":"2024-03-18T20:27:23.060172+0000","last_peered":"2024-03-18T20:27:23.060172+0000","last_clean":"2024-03-18T20:27:23.060172+0000","last_became_active":"2024-03-18T20:27:22.067431+0000","last_became_peered":"2024-03-18T20:27:22.067431+0000","last_unstale":"2024-03-18T20:27:23.060172+0000","last_undegraded":"2024-03-18T20:27:23.060172+0000","last_fullsized":"2024-03-18T20:27:23.060172+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:46.171545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.7","version":"273'2","reported_seq":13,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.077737+0000","last_change":"2024-03-18T20:27:21.636416+0000","last_active":"2024-03-18T20:27:22.077737+0000","last_peered":"2024-03-18T20:27:22.077737+0000","last_clean":"2024-03-18T20:27:22.077737+0000","last_became_active":"2024-03-18T20:27:21.636100+0000","last_became_peered":"2024-03-18T20:27:21.636100+0000","last_unstale":"2024-03-18T20:27:22.077737+0000","last_undegraded":"2024-03-18T20:27:22.077737+0000","last_fullsized":"2024-03-18T20:27:22.077737+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:45.164373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:25.159358+0000","last_change":"2024-03-18T20:27:22.065249+0000","last_active":"2024-03-18T20:27:25.159358+0000","last_peered":"2024-03-18T20:27:25.159358+0000","last_clean":"2024-03-18T20:27:25.159358+0000","last_became_active":"2024-03-18T20:27:22.063953+0000","last_became_peered":"2024-03-18T20:27:22.063953+0000","last_unstale":"2024-03-18T20:27:25.159358+0000","last_undegraded":"2024-03-18T20:27:25.159358+0000","last_fullsized":"2024-03-18T20:27:25.159358+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:53.864966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.057876+0000","last_change":"2024-03-18T20:27:21.633390+0000","last_active":"2024-03-18T20:27:22.057876+0000","last_peered":"2024-03-18T20:27:22.057876+0000","last_clean":"2024-03-18T20:27:22.057876+0000","last_became_active":"2024-03-18T20:27:21.633097+0000","last_became_peered":"2024-03-18T20:27:21.633097+0000","last_unstale":"2024-03-18T20:27:22.057876+0000","last_undegraded":"2024-03-18T20:27:22.057876+0000","last_fullsized":"2024-03-18T20:27:22.057876+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:52.857794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.060087+0000","last_change":"2024-03-18T20:27:22.080000+0000","last_active":"2024-03-18T20:27:23.060087+0000","last_peered":"2024-03-18T20:27:23.060087+0000","last_clean":"2024-03-18T20:27:23.060087+0000","last_became_active":"2024-03-18T20:27:22.074619+0000","last_became_peered":"2024-03-18T20:27:22.074619+0000","last_unstale":"2024-03-18T20:27:23.060087+0000","last_undegraded":"2024-03-18T20:27:23.060087+0000","last_fullsized":"2024-03-18T20:27:23.060087+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:21:41.361502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.059490+0000","last_change":"2024-03-18T20:27:21.640958+0000","last_active":"2024-03-18T20:27:22.059490+0000","last_peered":"2024-03-18T20:27:22.059490+0000","last_clean":"2024-03-18T20:27:22.059490+0000","last_became_active":"2024-03-18T20:27:21.640694+0000","last_became_peered":"2024-03-18T20:27:21.640694+0000","last_unstale":"2024-03-18T20:27:22.059490+0000","last_undegraded":"2024-03-18T20:27:22.059490+0000","last_fullsized":"2024-03-18T20:27:22.059490+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:23:55.315193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.540156+0000","last_change":"2024-03-18T20:27:22.070127+0000","last_active":"2024-03-18T20:27:23.540156+0000","last_peered":"2024-03-18T20:27:23.540156+0000","last_clean":"2024-03-18T20:27:23.540156+0000","last_became_active":"2024-03-18T20:27:22.069839+0000","last_became_peered":"2024-03-18T20:27:22.069839+0000","last_unstale":"2024-03-18T20:27:23.540156+0000","last_undegraded":"2024-03-18T20:27:23.540156+0000","last_fullsized":"2024-03-18T20:27:23.540156+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:32:46.349269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.058566+0000","last_change":"2024-03-18T20:27:21.630579+0000","last_active":"2024-03-18T20:27:22.058566+0000","last_peered":"2024-03-18T20:27:22.058566+0000","last_clean":"2024-03-18T20:27:22.058566+0000","last_became_active":"2024-03-18T20:27:21.630311+0000","last_became_peered":"2024-03-18T20:27:21.630311+0000","last_unstale":"2024-03-18T20:27:22.058566+0000","last_undegraded":"2024-03-18T20:27:22.058566+0000","last_fullsized":"2024-03-18T20:27:22.058566+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:30:01.555443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.540075+0000","last_change":"2024-03-18T20:27:22.069816+0000","last_active":"2024-03-18T20:27:23.540075+0000","last_peered":"2024-03-18T20:27:23.540075+0000","last_clean":"2024-03-18T20:27:23.540075+0000","last_became_active":"2024-03-18T20:27:22.069541+0000","last_became_peered":"2024-03-18T20:27:22.069541+0000","last_unstale":"2024-03-18T20:27:23.540075+0000","last_undegraded":"2024-03-18T20:27:23.540075+0000","last_fullsized":"2024-03-18T20:27:23.540075+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:17:26.761944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.053302+0000","last_change":"2024-03-18T20:27:21.641737+0000","last_active":"2024-03-18T20:27:22.053302+0000","last_peered":"2024-03-18T20:27:22.053302+0000","last_clean":"2024-03-18T20:27:22.053302+0000","last_became_active":"2024-03-18T20:27:21.641478+0000","last_became_peered":"2024-03-18T20:27:21.641478+0000","last_unstale":"2024-03-18T20:27:22.053302+0000","last_undegraded":"2024-03-18T20:27:22.053302+0000","last_fullsized":"2024-03-18T20:27:22.053302+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:23:24.136007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.060471+0000","last_change":"2024-03-18T20:27:22.079159+0000","last_active":"2024-03-18T20:27:23.060471+0000","last_peered":"2024-03-18T20:27:23.060471+0000","last_clean":"2024-03-18T20:27:23.060471+0000","last_became_active":"2024-03-18T20:27:22.074194+0000","last_became_peered":"2024-03-18T20:27:22.074194+0000","last_unstale":"2024-03-18T20:27:23.060471+0000","last_undegraded":"2024-03-18T20:27:23.060471+0000","last_fullsized":"2024-03-18T20:27:23.060471+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:26:41.614115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.065153+0000","last_change":"2024-03-18T20:27:21.643173+0000","last_active":"2024-03-18T20:27:22.065153+0000","last_peered":"2024-03-18T20:27:22.065153+0000","last_clean":"2024-03-18T20:27:22.065153+0000","last_became_active":"2024-03-18T20:27:21.642919+0000","last_became_peered":"2024-03-18T20:27:21.642919+0000","last_unstale":"2024-03-18T20:27:22.065153+0000","last_undegraded":"2024-03-18T20:27:22.065153+0000","last_fullsized":"2024-03-18T20:27:22.065153+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:54:45.115313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.218125+0000","last_change":"2024-03-18T20:27:22.069073+0000","last_active":"2024-03-18T20:27:24.218125+0000","last_peered":"2024-03-18T20:27:24.218125+0000","last_clean":"2024-03-18T20:27:24.218125+0000","last_became_active":"2024-03-18T20:27:22.068744+0000","last_became_peered":"2024-03-18T20:27:22.068744+0000","last_unstale":"2024-03-18T20:27:24.218125+0000","last_undegraded":"2024-03-18T20:27:24.218125+0000","last_fullsized":"2024-03-18T20:27:24.218125+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:44.884087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.061084+0000","last_change":"2024-03-18T20:27:21.639854+0000","last_active":"2024-03-18T20:27:22.061084+0000","last_peered":"2024-03-18T20:27:22.061084+0000","last_clean":"2024-03-18T20:27:22.061084+0000","last_became_active":"2024-03-18T20:27:21.639563+0000","last_became_peered":"2024-03-18T20:27:21.639563+0000","last_unstale":"2024-03-18T20:27:22.061084+0000","last_undegraded":"2024-03-18T20:27:22.061084+0000","last_fullsized":"2024-03-18T20:27:22.061084+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:28.989185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.7","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:23.060464+0000","last_change":"2024-03-18T20:27:22.082032+0000","last_active":"2024-03-18T20:27:23.060464+0000","last_peered":"2024-03-18T20:27:23.060464+0000","last_clean":"2024-03-18T20:27:23.060464+0000","last_became_active":"2024-03-18T20:27:22.081714+0000","last_became_peered":"2024-03-18T20:27:22.081714+0000","last_unstale":"2024-03-18T20:27:23.060464+0000","last_undegraded":"2024-03-18T20:27:23.060464+0000","last_fullsized":"2024-03-18T20:27:23.060464+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:30:51.011555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":11,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.053150+0000","last_change":"2024-03-18T20:27:21.637491+0000","last_active":"2024-03-18T20:27:22.053150+0000","last_peered":"2024-03-18T20:27:22.053150+0000","last_clean":"2024-03-18T20:27:22.053150+0000","last_became_active":"2024-03-18T20:27:21.637016+0000","last_became_peered":"2024-03-18T20:27:21.637016+0000","last_unstale":"2024-03-18T20:27:22.053150+0000","last_undegraded":"2024-03-18T20:27:22.053150+0000","last_fullsized":"2024-03-18T20:27:22.053150+0000","mapping_epoch":271,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:20.607510+0000","last_clean_scrub_stamp":"2024-03-18T20:27:20.607510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:43.006246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.25","version":"0'0","reported_seq":10,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-18T20:27:24.337281+0000","last_change":"2024-03-18T20:27:22.054757+0000","last_active":"2024-03-18T20:27:24.337281+0000","last_peered":"2024-03-18T20:27:24.337281+0000","last_clean":"2024-03-18T20:27:24.337281+0000","last_became_active":"2024-03-18T20:27:22.054451+0000","last_became_peered":"2024-03-18T20:27:22.054451+0000","last_unstale":"2024-03-18T20:27:24.337281+0000","last_undegraded":"2024-03-18T20:27:24.337281+0000","last_fullsized":"2024-03-18T20:27:24.337281+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:27:21.614682+0000","last_clean_scrub_stamp":"2024-03-18T20:27:21.614682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:30:51.011555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"259'381","reported_seq":1355,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-03-18T20:27:22.055601+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:27:22.055601+0000","last_peered":"2024-03-18T20:27:22.055601+0000","last_clean":"2024-03-18T20:27:22.055601+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:27:22.055601+0000","last_undegraded":"2024-03-18T20:27:22.055601+0000","last_fullsized":"2024-03-18T20:27:22.055601+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":36,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576002,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53156,"kb_used_data":3980,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318684,"statfs":{"total":96636764160,"available":96582332416,"internally_reserved":0,"allocated":4075520,"data_stored":2386750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54900000000000004,"15min":0.58999999999999997},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.72599999999999998,"5min":0.80900000000000005,"15min":1.1339999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.56599999999999995,"15min":0.60199999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":0.73599999999999999,"5min":0.84499999999999997,"15min":1.093},"last":0.48899999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.55500000000000005,"15min":0.60199999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.748,"5min":0.83099999999999996,"15min":1.4299999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.57099999999999995,"15min":0.60799999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73599999999999999,"5min":0.82399999999999995,"15min":1.2629999999999999},"last":0.54600000000000004}]},{"osd":2,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.56100000000000005,"15min":0.60099999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67600000000000005,"5min":0.85499999999999998,"15min":1.216},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57799999999999996,"15min":0.61599999999999999},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75,"5min":1.1279999999999999,"15min":1.2490000000000001},"last":0.64000000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":0.59799999999999998},"min":{"1min":0.38700000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71999999999999997,"5min":0.84699999999999998,"15min":1.4450000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.60699999999999998,"15min":0.64100000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73099999999999998,"5min":0.94599999999999995,"15min":1.5029999999999999},"last":0.66700000000000004}]},{"osd":4,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47099999999999997,"15min":0.47799999999999998},"min":{"1min":0.35999999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.68899999999999995,"5min":0.79100000000000004,"15min":0.86099999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48899999999999999,"15min":0.499},"min":{"1min":0.34100000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.66200000000000003,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.46300000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.51300000000000001,"15min":0.51900000000000002},"min":{"1min":0.373,"5min":0.314,"15min":0.314},"max":{"1min":0.70399999999999996,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52800000000000002,"15min":0.54600000000000004},"min":{"1min":0.38300000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73099999999999998,"5min":0.877,"15min":1.042},"last":0.59299999999999997}]},{"osd":6,"last update":"Mon Mar 18 20:26:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55800000000000005,"15min":0.57699999999999996},"min":{"1min":0.435,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77300000000000002,"5min":0.93500000000000005,"15min":1.129},"last":0.626},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.54700000000000004,"15min":0.56899999999999995},"min":{"1min":0.47399999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76200000000000001,"5min":0.91900000000000004,"15min":1.0609999999999999},"last":0.61599999999999999}]}]},{"osd":6,"up_from":13,"seq":55834576002,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60740,"kb_used_data":3380,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94311100,"statfs":{"total":96636764160,"available":96574566400,"internally_reserved":0,"allocated":3461120,"data_stored":1794166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53300000000000003,"15min":0.58499999999999996},"min":{"1min":0.35399999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68799999999999994,"5min":0.79800000000000004,"15min":1.1519999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.503,"5min":0.54600000000000004,"15min":0.59499999999999997},"min":{"1min":0.33200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.73499999999999999,"5min":1.119,"15min":1.119},"last":0.41299999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53800000000000003,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73199999999999998,"5min":0.79100000000000004,"15min":1.244},"last":0.439},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":0.59499999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":0.70099999999999996,"5min":0.89200000000000002,"15min":1.286},"last":0.59099999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56399999999999995,"15min":0.61699999999999999},"min":{"1min":0.38700000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.69599999999999995,"5min":0.79600000000000004,"15min":1.276},"last":0.629},{"interface":"front","average":{"1min":0.496,"5min":0.55400000000000005,"15min":0.59899999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.65200000000000002,"5min":1.089,"15min":1.2669999999999999},"last":0.47399999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55500000000000005,"15min":0.61099999999999999},"min":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71599999999999997,"5min":1.163,"15min":1.163},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57099999999999995,"15min":0.61099999999999999},"min":{"1min":0.47499999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78100000000000003,"5min":0.82399999999999995,"15min":1.337},"last":0.56100000000000005}]},{"osd":4,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.35299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80200000000000005,"5min":0.85799999999999998,"15min":0.88300000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.50600000000000001,"15min":0.51800000000000002},"min":{"1min":0.32500000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83399999999999996,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.61799999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.50900000000000001,"15min":0.51200000000000001},"min":{"1min":0.42599999999999999,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":0.65000000000000002,"5min":1.008,"15min":1.008},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55900000000000005,"15min":0.55300000000000005},"min":{"1min":0.41799999999999998,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.755,"5min":0.92600000000000005,"15min":0.94199999999999995},"last":0.66800000000000004}]},{"osd":7,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56699999999999995,"15min":0.56299999999999994},"min":{"1min":0.40999999999999998,"5min":0.38,"15min":0.34499999999999997},"max":{"1min":0.91200000000000003,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.34599999999999997},"max":{"1min":0.86699999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.71199999999999997}]}]},{"osd":1,"up_from":13,"seq":55834576002,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52548,"kb_used_data":3384,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319292,"statfs":{"total":96636764160,"available":96582955008,"internally_reserved":0,"allocated":3465216,"data_stored":1794256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.38700000000000001,"15min":0.39300000000000002},"min":{"1min":0.255,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.46400000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39500000000000002,"15min":0.41999999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.623,"5min":0.623,"15min":0.69699999999999995},"last":0.39900000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.40899999999999997,"15min":0.42899999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53100000000000003,"5min":0.61199999999999999,"15min":0.68100000000000005},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.45200000000000001,"15min":0.47099999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.69799999999999995,"5min":0.70099999999999996,"15min":0.80500000000000005},"last":0.34799999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.46500000000000002,"15min":0.48399999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.76100000000000001},"last":0.44},{"interface":"front","average":{"1min":0.372,"5min":0.41899999999999998,"15min":0.433},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.27000000000000002},"max":{"1min":0.54700000000000004,"5min":0.68200000000000005,"15min":0.71699999999999997},"last":0.32400000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.64000000000000001,"15min":0.66500000000000004},"min":{"1min":0.43099999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76800000000000002,"5min":1.0680000000000001,"15min":1.2909999999999999},"last":0.629},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.64600000000000002,"15min":0.67100000000000004},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.88500000000000001,"5min":1.1160000000000001,"15min":1.746},"last":0.55300000000000005}]},{"osd":5,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.63400000000000001,"15min":0.66100000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79300000000000004,"5min":1.0389999999999999,"15min":1.6910000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.67800000000000005,"15min":0.70499999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.85199999999999998,"5min":1.109,"15min":1.2549999999999999},"last":0.59199999999999997}]},{"osd":6,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.65300000000000002,"15min":0.67700000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.70799999999999996,"5min":1.0289999999999999,"15min":1.637},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.67800000000000005,"15min":0.70199999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82099999999999995,"5min":1.014,"15min":1.722},"last":0.52400000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.66000000000000003,"15min":0.69899999999999995},"min":{"1min":0.46100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91100000000000003,"5min":0.99199999999999999,"15min":1.2729999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.67000000000000004,"15min":0.69699999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.80700000000000005,"5min":1.0329999999999999,"15min":1.3380000000000001},"last":0.66900000000000004}]}]},{"osd":0,"up_from":13,"seq":55834576003,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49036,"kb_used_data":3968,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322804,"statfs":{"total":96636764160,"available":96586551296,"internally_reserved":0,"allocated":4063232,"data_stored":2385150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.36399999999999999,"15min":0.38600000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.46100000000000002,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.39600000000000002,"15min":0.41999999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.58799999999999997,"5min":0.60099999999999998,"15min":1.1850000000000001},"last":0.38200000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.41299999999999998,"15min":0.433},"min":{"1min":0.28699999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.47199999999999998,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.41799999999999998,"15min":0.44600000000000001},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.54500000000000004,"5min":0.64800000000000002,"15min":0.70999999999999996},"last":0.45800000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.39600000000000002,"15min":0.42399999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.433,"5min":0.58499999999999996,"15min":0.628},"last":0.44},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.45400000000000001,"15min":0.48699999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.52600000000000002,"5min":0.73399999999999999,"15min":0.91500000000000004},"last":0.47399999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.61299999999999999,"15min":0.64400000000000002},"min":{"1min":0.41299999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.752,"5min":0.91500000000000004,"15min":1.0029999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.64800000000000002,"15min":0.65900000000000003},"min":{"1min":0.51600000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.92300000000000004,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.71599999999999997}]},{"osd":5,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.64400000000000002,"15min":0.66600000000000004},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83299999999999996,"5min":0.92300000000000004,"15min":1.0129999999999999},"last":0.878},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.65300000000000002,"15min":0.67500000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.73199999999999998,"5min":0.96199999999999997,"15min":1.518},"last":0.76100000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.64600000000000002,"15min":0.65700000000000003},"min":{"1min":0.45800000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.73599999999999999,"5min":1.0669999999999999,"15min":1.1279999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.68200000000000005,"15min":0.70299999999999996},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.871,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.92900000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.67200000000000004,"15min":0.69899999999999995},"min":{"1min":0.45300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77500000000000002,"5min":1.008,"15min":1.464},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.67300000000000004,"15min":0.69799999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88900000000000001,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.80800000000000005}]}]},{"osd":2,"up_from":13,"seq":55834576003,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52556,"kb_used_data":3380,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319284,"statfs":{"total":96636764160,"available":96582946816,"internally_reserved":0,"allocated":3461120,"data_stored":1794166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.34699999999999998,"15min":0.371},"min":{"1min":0.20799999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.53800000000000003,"5min":0.54800000000000004,"15min":0.57499999999999996},"last":0.184},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.44700000000000001,"15min":0.47799999999999998},"min":{"1min":0.26100000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60499999999999998,"5min":0.71499999999999997,"15min":0.753},"last":0.32400000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.44400000000000001,"15min":0.47399999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57399999999999995,"5min":0.64300000000000002,"15min":0.749},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.38900000000000001,"15min":0.40899999999999997},"min":{"1min":0.246,"5min":0.224,"15min":0.224},"max":{"1min":0.501,"5min":0.55900000000000005,"15min":0.59999999999999998},"last":0.24199999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.41199999999999998,"15min":0.42899999999999999},"min":{"1min":0.27800000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.56100000000000005,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.47399999999999998,"15min":0.50700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.63200000000000001,"5min":0.68100000000000005,"15min":0.752},"last":0.34499999999999997}]},{"osd":4,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63900000000000001,"15min":0.65300000000000002},"min":{"1min":0.41799999999999998,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":1.2090000000000001,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.622,"5min":0.65700000000000003,"15min":0.66200000000000003},"min":{"1min":0.45100000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.268,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.50700000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.64900000000000002,"15min":0.67500000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79900000000000004,"5min":0.97299999999999998,"15min":1.028},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.63900000000000001,"15min":0.67000000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.77100000000000002,"5min":0.88300000000000001,"15min":1.0629999999999999},"last":0.58499999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.65500000000000003,"15min":0.68100000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.81899999999999995,"5min":1.014,"15min":1.2050000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.70699999999999996,"15min":0.72699999999999998},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.3220000000000001,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.68700000000000006}]},{"osd":7,"last update":"Mon Mar 18 20:26:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.65500000000000003,"15min":0.68899999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.42199999999999999},"max":{"1min":0.73699999999999999,"5min":0.99399999999999999,"15min":1.0269999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68600000000000005,"15min":0.70099999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.2330000000000001,"5min":1.371,"15min":1.371},"last":0.66300000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576003,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52552,"kb_used_data":3388,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319288,"statfs":{"total":96636764160,"available":96582950912,"internally_reserved":0,"allocated":3469312,"data_stored":1794258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.36099999999999999,"15min":0.39000000000000001},"min":{"1min":0.247,"5min":0.223,"15min":0.223},"max":{"1min":0.46400000000000002,"5min":0.66500000000000004,"15min":0.98099999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.373,"15min":0.40400000000000003},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.45600000000000002,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.497}]},{"osd":1,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.35999999999999999,"15min":0.39700000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.44500000000000001,"5min":0.63200000000000001,"15min":0.64000000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.42199999999999999,"15min":0.45600000000000002},"min":{"1min":0.33400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.501,"5min":0.65500000000000003,"15min":0.73099999999999998},"last":0.46100000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.41499999999999998,"15min":0.45200000000000001},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.54900000000000004,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.437,"15min":0.48099999999999998},"min":{"1min":0.32600000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.52300000000000002,"5min":0.64400000000000002,"15min":0.71199999999999997},"last":0.51600000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59499999999999997,"15min":0.64000000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.526,"5min":1.526,"15min":1.526},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61399999999999999,"15min":0.66100000000000003},"min":{"1min":0.39200000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.65800000000000003,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.78900000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.65100000000000002,"15min":0.68700000000000006},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.66500000000000004,"15min":0.71499999999999997},"min":{"1min":0.45300000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78000000000000003,"5min":1.1120000000000001,"15min":1.171},"last":0.80500000000000005}]},{"osd":6,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65800000000000003,"15min":0.69699999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.71999999999999997},"min":{"1min":0.48699999999999999,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.88100000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.64900000000000002,"15min":0.70299999999999996},"min":{"1min":0.43099999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79900000000000004,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64300000000000002,"15min":0.68899999999999995},"min":{"1min":0.51500000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.81999999999999995}]}]},{"osd":4,"up_from":13,"seq":55834576003,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48464,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323376,"statfs":{"total":96636764160,"available":96587137024,"internally_reserved":0,"allocated":3477504,"data_stored":1795820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57799999999999996,"15min":0.58899999999999997},"min":{"1min":0.39600000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56799999999999995,"15min":0.59499999999999997},"min":{"1min":0.40100000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.84199999999999997},"last":0.51500000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.59199999999999997,"15min":0.61099999999999999},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.79300000000000004,"5min":0.81899999999999995,"15min":0.88300000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56399999999999995,"15min":0.57999999999999996},"min":{"1min":0.34899999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.96499999999999997},"last":0.49399999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58699999999999997,"15min":0.59899999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59799999999999998,"15min":0.61399999999999999},"min":{"1min":0.42899999999999999,"5min":0.41299999999999998,"15min":0.38500000000000001},"max":{"1min":0.70799999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.56799999999999995}]},{"osd":3,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.59299999999999997,"15min":0.60199999999999998},"min":{"1min":0.433,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78100000000000003,"5min":1.026,"15min":1.026},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.41399999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.44600000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.48399999999999999,"15min":0.48199999999999998},"min":{"1min":0.318,"5min":0.309,"15min":0.309},"max":{"1min":0.76700000000000002,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.496,"15min":0.505},"min":{"1min":0.38300000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.81499999999999995,"5min":0.85499999999999998,"15min":0.90600000000000003},"last":0.54300000000000004}]},{"osd":6,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.502,"15min":0.50600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.754,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.62},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.371,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.82799999999999996,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.60399999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:26:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54700000000000004,"15min":0.54800000000000004},"min":{"1min":0.40600000000000003,"5min":0.38800000000000001,"15min":0.35699999999999998},"max":{"1min":0.80300000000000005,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.38500000000000001,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.84099999999999997,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.64600000000000002}]}]},{"osd":5,"up_from":13,"seq":55834576002,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66504,"kb_used_data":3388,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305336,"statfs":{"total":96636764160,"available":96568664064,"internally_reserved":0,"allocated":3469312,"data_stored":1794222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.55400000000000005,"15min":0.59099999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.31900000000000001},"max":{"1min":0.76300000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55600000000000005,"15min":0.59499999999999997},"min":{"1min":0.42099999999999999,"5min":0.35599999999999998,"15min":0.33100000000000002},"max":{"1min":0.81100000000000005,"5min":0.82599999999999996,"15min":1.3100000000000001},"last":0.50900000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.56100000000000005,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71499999999999997,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.55400000000000005,"15min":0.59199999999999997},"min":{"1min":0.372,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.69399999999999995,"5min":0.94299999999999995,"15min":1.018},"last":0.59499999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.59399999999999997},"min":{"1min":0.40300000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.75600000000000001,"5min":0.85299999999999998,"15min":0.99299999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.55400000000000005,"15min":0.60099999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77000000000000002,"5min":1.599,"15min":1.599},"last":0.68300000000000005}]},{"osd":3,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.57499999999999996,"15min":0.60399999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72899999999999998,"5min":0.98699999999999999,"15min":1.2430000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59299999999999997,"15min":0.63700000000000001},"min":{"1min":0.42099999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":0.84399999999999997,"15min":1.161},"last":0.70099999999999996}]},{"osd":4,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48999999999999999,"15min":0.48299999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.625,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.35699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.76100000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.503,"15min":0.50900000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.32800000000000001},"max":{"1min":0.64800000000000002,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.746},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54300000000000004,"15min":0.54200000000000004},"min":{"1min":0.35199999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.64600000000000002,"5min":1.643,"15min":1.643},"last":0.71999999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:26:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.40200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.71299999999999997,"5min":1.028,"15min":1.028},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57099999999999995,"15min":0.57799999999999996},"min":{"1min":0.42599999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74299999999999999,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.85199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:27:31.299 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T20:27:31.300 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-18T20:27:31.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.463+0000 7f511e790700 1 Processor -- start 2024-03-18T20:27:31.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.463+0000 7f511e790700 1 -- start start 2024-03-18T20:27:31.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.464+0000 7f511e790700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f511815d3b0 0x7f511806cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:31.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.464+0000 7f511e790700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f511806d430 0x7f5118070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:31.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.464+0000 7f511e790700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51180b2b80 con 0x7f51180b2650 2024-03-18T20:27:31.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.464+0000 7f511e790700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51180b3800 con 0x7f511806d430 2024-03-18T20:27:31.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.464+0000 7f511e790700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51180b2e60 con 0x7f511815d3b0 2024-03-18T20:27:31.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.464+0000 7f51177fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f511806d430 0x7f5118070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:31.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.464+0000 7f51177fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f511806d430 0x7f5118070e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36610/0 (socket says 172.21.15.1:36610) 2024-03-18T20:27:31.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.464+0000 7f51177fe700 1 -- 172.21.15.1:0/1274586257 learned_addr learned my addr 172.21.15.1:0/1274586257 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:31.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.465+0000 7f5117fff700 1 --2- 172.21.15.1:0/1274586257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f511815d3b0 0x7f511806cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:31.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.465+0000 7f51177fe700 1 -- 172.21.15.1:0/1274586257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f511815d3b0 msgr2=0x7f511806cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:31.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.465+0000 7f51177fe700 1 --2- 172.21.15.1:0/1274586257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f511815d3b0 0x7f511806cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.465+0000 7f51177fe700 1 -- 172.21.15.1:0/1274586257 >> v1:172.21.15.165:6789/0 conn(0x7f51180b2650 legacy=0x7f51180b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:31.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.465+0000 7f51177fe700 1 -- 172.21.15.1:0/1274586257 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51180b3140 con 0x7f511806d430 2024-03-18T20:27:31.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.465+0000 7f5117fff700 1 --2- 172.21.15.1:0/1274586257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f511815d3b0 0x7f511806cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:31.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.465+0000 7f51177fe700 1 --2- 172.21.15.1:0/1274586257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511806d430 0x7f5118070e40 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7f51080098f0 tx=0x7f5108009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb92d3795aab1b37 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:31.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.465+0000 7f5116ffd700 1 -- 172.21.15.1:0/1274586257 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5108004230 con 0x7f511806d430 2024-03-18T20:27:31.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.465+0000 7f5116ffd700 1 -- 172.21.15.1:0/1274586257 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5108004390 con 0x7f511806d430 2024-03-18T20:27:31.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.465+0000 7f5116ffd700 1 -- 172.21.15.1:0/1274586257 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51080036a0 con 0x7f511806d430 2024-03-18T20:27:31.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.466+0000 7f511e790700 1 -- 172.21.15.1:0/1274586257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511806d430 msgr2=0x7f5118070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:31.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.466+0000 7f511e790700 1 --2- 172.21.15.1:0/1274586257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511806d430 0x7f5118070e40 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7f51080098f0 tx=0x7f5108009c00 comp rx=0 tx=0).stop 2024-03-18T20:27:31.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.466+0000 7f511e790700 1 -- 172.21.15.1:0/1274586257 shutdown_connections 2024-03-18T20:27:31.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.466+0000 7f511e790700 1 --2- 172.21.15.1:0/1274586257 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f511815d3b0 0x7f511806cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.466+0000 7f511e790700 1 --2- 172.21.15.1:0/1274586257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511806d430 0x7f5118070e40 unknown :-1 s=CLOSED pgs=3089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.466+0000 7f511e790700 1 -- 172.21.15.1:0/1274586257 >> 172.21.15.1:0/1274586257 conn(0x7f51180b1350 msgr2=0x7f51180b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:31.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.466+0000 7f511e790700 1 -- 172.21.15.1:0/1274586257 shutdown_connections 2024-03-18T20:27:31.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.466+0000 7f511e790700 1 -- 172.21.15.1:0/1274586257 wait complete. 2024-03-18T20:27:31.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.467+0000 7f511e790700 1 Processor -- start 2024-03-18T20:27:31.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.467+0000 7f511e790700 1 -- start start 2024-03-18T20:27:31.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.468+0000 7f511e790700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f511806d430 0x7f51181451a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:31.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.468+0000 7f511e790700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51180b2650 0x7f5118070e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:31.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.468+0000 7f511e790700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511815d3b0 0x7f5118071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:31.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.468+0000 7f511e790700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5118159720 con 0x7f511815d3b0 2024-03-18T20:27:31.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.468+0000 7f511e790700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51181540d0 con 0x7f511806d430 2024-03-18T20:27:31.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.468+0000 7f511e790700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51181456e0 con 0x7f51180b2650 2024-03-18T20:27:31.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.468+0000 7f511cd2d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511815d3b0 0x7f5118071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:31.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.468+0000 7f511cd2d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511815d3b0 0x7f5118071380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36612/0 (socket says 172.21.15.1:36612) 2024-03-18T20:27:31.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.468+0000 7f5117fff700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f511806d430 0x7f51181451a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:31.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.468+0000 7f511cd2d700 1 -- 172.21.15.1:0/255615965 learned_addr learned my addr 172.21.15.1:0/255615965 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:31.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.468+0000 7f51177fe700 1 --2- 172.21.15.1:0/255615965 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51180b2650 0x7f5118070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:31.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.468+0000 7f511cd2d700 1 -- 172.21.15.1:0/255615965 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51180b2650 msgr2=0x7f5118070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:31.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.468+0000 7f511cd2d700 1 --2- 172.21.15.1:0/255615965 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51180b2650 0x7f5118070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.468+0000 7f511cd2d700 1 -- 172.21.15.1:0/255615965 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f511806d430 msgr2=0x7f51181451a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:31.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.468+0000 7f511cd2d700 1 --2- 172.21.15.1:0/255615965 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f511806d430 0x7f51181451a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.468+0000 7f511cd2d700 1 -- 172.21.15.1:0/255615965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5108009550 con 0x7f511815d3b0 2024-03-18T20:27:31.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.468+0000 7f511cd2d700 1 --2- 172.21.15.1:0/255615965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511815d3b0 0x7f5118071380 secure :-1 s=READY pgs=3090 cs=0 l=1 rev1=1 crypto rx=0x7f511000cc10 tx=0x7f511000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:31.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.468+0000 7f5114ff9700 1 -- 172.21.15.1:0/255615965 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511000aa60 con 0x7f511815d3b0 2024-03-18T20:27:31.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.469+0000 7f5114ff9700 1 -- 172.21.15.1:0/255615965 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f511000abc0 con 0x7f511815d3b0 2024-03-18T20:27:31.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.469+0000 7f511e790700 1 -- 172.21.15.1:0/255615965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5118154320 con 0x7f511815d3b0 2024-03-18T20:27:31.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.469+0000 7f5114ff9700 1 -- 172.21.15.1:0/255615965 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5110012420 con 0x7f511815d3b0 2024-03-18T20:27:31.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.469+0000 7f511e790700 1 -- 172.21.15.1:0/255615965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51181548e0 con 0x7f511815d3b0 2024-03-18T20:27:31.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.469+0000 7f511e790700 1 -- 172.21.15.1:0/255615965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51180676a0 con 0x7f511815d3b0 2024-03-18T20:27:31.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.470+0000 7f5114ff9700 1 -- 172.21.15.1:0/255615965 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f5110012580 con 0x7f511815d3b0 2024-03-18T20:27:31.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.470+0000 7f5114ff9700 1 --2- 172.21.15.1:0/255615965 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f50f806aed0 0x7f50f806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:31.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.471+0000 7f5117fff700 1 --2- 172.21.15.1:0/255615965 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f50f806aed0 0x7f50f806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:31.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.471+0000 7f5114ff9700 1 -- 172.21.15.1:0/255615965 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f5110089b30 con 0x7f511815d3b0 2024-03-18T20:27:31.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.471+0000 7f5117fff700 1 --2- 172.21.15.1:0/255615965 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f50f806aed0 0x7f50f806d390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5100006fd0 tx=0x7f5100008040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:31.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.475+0000 7f5114ff9700 1 -- 172.21.15.1:0/255615965 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5110054890 con 0x7f511815d3b0 2024-03-18T20:27:31.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.767+0000 7f511e790700 1 -- 172.21.15.1:0/255615965 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f511806f4a0 con 0x7f511815d3b0 2024-03-18T20:27:31.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.771+0000 7f5114ff9700 1 -- 172.21.15.1:0/255615965 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v53) v1 ==== 190+0+0 (secure 0 0 0) 0x7f5110057eb0 con 0x7f511815d3b0 2024-03-18T20:27:31.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.773+0000 7f511e790700 1 -- 172.21.15.1:0/255615965 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f50f806aed0 msgr2=0x7f50f806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:31.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.773+0000 7f511e790700 1 --2- 172.21.15.1:0/255615965 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f50f806aed0 0x7f50f806d390 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5100006fd0 tx=0x7f5100008040 comp rx=0 tx=0).stop 2024-03-18T20:27:31.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.773+0000 7f511e790700 1 -- 172.21.15.1:0/255615965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511815d3b0 msgr2=0x7f5118071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:31.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.773+0000 7f511e790700 1 --2- 172.21.15.1:0/255615965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511815d3b0 0x7f5118071380 secure :-1 s=READY pgs=3090 cs=0 l=1 rev1=1 crypto rx=0x7f511000cc10 tx=0x7f511000cf20 comp rx=0 tx=0).stop 2024-03-18T20:27:31.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.773+0000 7f511e790700 1 -- 172.21.15.1:0/255615965 shutdown_connections 2024-03-18T20:27:31.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.773+0000 7f511e790700 1 --2- 172.21.15.1:0/255615965 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f50f806aed0 0x7f50f806d390 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.773+0000 7f511e790700 1 --2- 172.21.15.1:0/255615965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511815d3b0 0x7f5118071380 unknown :-1 s=CLOSED pgs=3090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.773+0000 7f511e790700 1 --2- 172.21.15.1:0/255615965 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f511806d430 0x7f51181451a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.773+0000 7f511e790700 1 --2- 172.21.15.1:0/255615965 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51180b2650 0x7f5118070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.773+0000 7f511e790700 1 -- 172.21.15.1:0/255615965 >> 172.21.15.1:0/255615965 conn(0x7f51180b1350 msgr2=0x7f511815c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:31.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.773+0000 7f511e790700 1 -- 172.21.15.1:0/255615965 shutdown_connections 2024-03-18T20:27:31.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.773+0000 7f511e790700 1 -- 172.21.15.1:0/255615965 wait complete. 2024-03-18T20:27:31.780 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-18T20:27:31.791 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:27:31.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.910+0000 7f0f044d1700 1 Processor -- start 2024-03-18T20:27:31.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.910+0000 7f0f044d1700 1 -- start start 2024-03-18T20:27:31.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.911+0000 7f0f044d1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0efc06a750 0x7f0efc06ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:31.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.911+0000 7f0f044d1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0efc15d6b0 0x7f0efc160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:31.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.911+0000 7f0f044d1700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0efc0acfb0 con 0x7f0efc0ad790 2024-03-18T20:27:31.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.911+0000 7f0f044d1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0efc0af7a0 con 0x7f0efc06a750 2024-03-18T20:27:31.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.911+0000 7f0f044d1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc06b070 con 0x7f0efc15d6b0 2024-03-18T20:27:31.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.911+0000 7f0f0226d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0efc06a750 0x7f0efc06ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:31.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.911+0000 7f0f0226d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0efc06a750 0x7f0efc06ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36628/0 (socket says 172.21.15.1:36628) 2024-03-18T20:27:31.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.911+0000 7f0f0226d700 1 -- 172.21.15.1:0/334137461 learned_addr learned my addr 172.21.15.1:0/334137461 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:31.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.912+0000 7f0f01a6c700 1 --2- 172.21.15.1:0/334137461 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0efc15d6b0 0x7f0efc160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:31.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.912+0000 7f0f0226d700 1 -- 172.21.15.1:0/334137461 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0efc15d6b0 msgr2=0x7f0efc160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:31.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.912+0000 7f0f0226d700 1 --2- 172.21.15.1:0/334137461 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0efc15d6b0 0x7f0efc160ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.912+0000 7f0f0226d700 1 -- 172.21.15.1:0/334137461 >> v1:172.21.15.165:6789/0 conn(0x7f0efc0ad790 legacy=0x7f0efc161530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:31.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.912+0000 7f0f0226d700 1 -- 172.21.15.1:0/334137461 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0efc0b50a0 con 0x7f0efc06a750 2024-03-18T20:27:31.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.912+0000 7f0f0226d700 1 --2- 172.21.15.1:0/334137461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0efc06a750 0x7f0efc06ab30 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0098f0 tx=0x7f0eec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=910867adb97e20f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:31.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.912+0000 7f0f0126b700 1 -- 172.21.15.1:0/334137461 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec004230 con 0x7f0efc06a750 2024-03-18T20:27:31.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.912+0000 7f0f0126b700 1 -- 172.21.15.1:0/334137461 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eec004390 con 0x7f0efc06a750 2024-03-18T20:27:31.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.912+0000 7f0f0126b700 1 -- 172.21.15.1:0/334137461 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec0036a0 con 0x7f0efc06a750 2024-03-18T20:27:31.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.913+0000 7f0f044d1700 1 -- 172.21.15.1:0/334137461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0efc06a750 msgr2=0x7f0efc06ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:31.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.913+0000 7f0f044d1700 1 --2- 172.21.15.1:0/334137461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0efc06a750 0x7f0efc06ab30 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0098f0 tx=0x7f0eec009c00 comp rx=0 tx=0).stop 2024-03-18T20:27:31.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.913+0000 7f0f044d1700 1 -- 172.21.15.1:0/334137461 shutdown_connections 2024-03-18T20:27:31.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.913+0000 7f0f044d1700 1 --2- 172.21.15.1:0/334137461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0efc06a750 0x7f0efc06ab30 unknown :-1 s=CLOSED pgs=3091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.913+0000 7f0f044d1700 1 --2- 172.21.15.1:0/334137461 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0efc15d6b0 0x7f0efc160ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.913+0000 7f0f044d1700 1 -- 172.21.15.1:0/334137461 >> 172.21.15.1:0/334137461 conn(0x7f0efc06b860 msgr2=0x7f0efc06bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:31.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.913+0000 7f0f044d1700 1 -- 172.21.15.1:0/334137461 shutdown_connections 2024-03-18T20:27:31.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.913+0000 7f0f044d1700 1 -- 172.21.15.1:0/334137461 wait complete. 2024-03-18T20:27:31.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.914+0000 7f0f044d1700 1 Processor -- start 2024-03-18T20:27:31.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.914+0000 7f0f044d1700 1 -- start start 2024-03-18T20:27:31.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f044d1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0efc06a750 0x7f0efc15f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:31.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f044d1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0efc0ad790 0x7f0efc14c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:31.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f044d1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0efc15d6b0 0x7f0efc14cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:31.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f044d1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc0ce0b0 con 0x7f0efc0ad790 2024-03-18T20:27:31.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f044d1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc0ce390 con 0x7f0efc15d6b0 2024-03-18T20:27:31.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f044d1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc0ce670 con 0x7f0efc06a750 2024-03-18T20:27:31.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f01a6c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0efc0ad790 0x7f0efc14c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:31.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f01a6c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0efc0ad790 0x7f0efc14c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36642/0 (socket says 172.21.15.1:36642) 2024-03-18T20:27:31.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f01a6c700 1 -- 172.21.15.1:0/162967064 learned_addr learned my addr 172.21.15.1:0/162967064 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:31.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f0226d700 1 --2- 172.21.15.1:0/162967064 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0efc06a750 0x7f0efc15f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:31.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f02a6e700 1 --2- 172.21.15.1:0/162967064 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0efc15d6b0 0x7f0efc14cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:31.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f01a6c700 1 -- 172.21.15.1:0/162967064 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0efc06a750 msgr2=0x7f0efc15f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:31.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f01a6c700 1 --2- 172.21.15.1:0/162967064 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0efc06a750 0x7f0efc15f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f01a6c700 1 -- 172.21.15.1:0/162967064 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0efc15d6b0 msgr2=0x7f0efc14cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:31.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f01a6c700 1 --2- 172.21.15.1:0/162967064 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0efc15d6b0 0x7f0efc14cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:31.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f01a6c700 1 -- 172.21.15.1:0/162967064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eec009550 con 0x7f0efc0ad790 2024-03-18T20:27:31.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f0226d700 1 --2- 172.21.15.1:0/162967064 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0efc06a750 0x7f0efc15f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:31.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f01a6c700 1 --2- 172.21.15.1:0/162967064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0efc0ad790 0x7f0efc14c830 secure :-1 s=READY pgs=3092 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800d800 tx=0x7f0ef800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:31.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0ef2ffd700 1 -- 172.21.15.1:0/162967064 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef80125f0 con 0x7f0efc0ad790 2024-03-18T20:27:31.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0ef2ffd700 1 -- 172.21.15.1:0/162967064 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef8012750 con 0x7f0efc0ad790 2024-03-18T20:27:31.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0ef2ffd700 1 -- 172.21.15.1:0/162967064 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8012b00 con 0x7f0efc0ad790 2024-03-18T20:27:31.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f044d1700 1 -- 172.21.15.1:0/162967064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0efc0ce950 con 0x7f0efc0ad790 2024-03-18T20:27:31.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.915+0000 7f0f044d1700 1 -- 172.21.15.1:0/162967064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0efc15f890 con 0x7f0efc0ad790 2024-03-18T20:27:31.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.916+0000 7f0f044d1700 1 -- 172.21.15.1:0/162967064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0efc0ad420 con 0x7f0efc0ad790 2024-03-18T20:27:31.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.917+0000 7f0ef2ffd700 1 -- 172.21.15.1:0/162967064 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f0ef8012c60 con 0x7f0efc0ad790 2024-03-18T20:27:31.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.917+0000 7f0ef2ffd700 1 --2- 172.21.15.1:0/162967064 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0ee006aed0 0x7f0ee006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:31.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.917+0000 7f0ef2ffd700 1 -- 172.21.15.1:0/162967064 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f0ef8089ca0 con 0x7f0efc0ad790 2024-03-18T20:27:31.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.917+0000 7f0f0226d700 1 --2- 172.21.15.1:0/162967064 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0ee006aed0 0x7f0ee006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:31.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.918+0000 7f0f0226d700 1 --2- 172.21.15.1:0/162967064 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0ee006aed0 0x7f0ee006d390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0eec009130 tx=0x7f0eec008030 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:31.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:31.922+0000 7f0ef2ffd700 1 -- 172.21.15.1:0/162967064 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0efc0ad420 con 0x7f0efc0ad790 2024-03-18T20:27:32.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.170+0000 7f0f044d1700 1 -- 172.21.15.1:0/162967064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0efc0ad420 con 0x7f0efc0ad790 2024-03-18T20:27:32.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.171+0000 7f0ef2ffd700 1 -- 172.21.15.1:0/162967064 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+4749 (secure 0 0 0) 0x7f0efc0ad420 con 0x7f0efc0ad790 2024-03-18T20:27:32.176 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:32.176 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":136,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9938,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/1320583552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1320583552},{"type":"v1","addr":"172.21.15.165:6833","nonce":1320583552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131},{"gid":11566,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2669900315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2669900315},{"type":"v1","addr":"172.21.15.165:6835","nonce":2669900315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131},{"gid":13170,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2726723271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2726723271},{"type":"v1","addr":"172.21.15.1:6835","nonce":2726723271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:27:22.047616+0000","modified":"2024-03-18T20:27:25.113850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13173},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13173":{"gid":13173,"name":"c","rank":0,"incarnation":132,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/3909787366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":3909787366},{"type":"v1","addr":"172.21.15.1:6837","nonce":3909787366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-03-18T20:27:32.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.173+0000 7f0f044d1700 1 -- 172.21.15.1:0/162967064 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0ee006aed0 msgr2=0x7f0ee006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:32.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.173+0000 7f0f044d1700 1 --2- 172.21.15.1:0/162967064 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0ee006aed0 0x7f0ee006d390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0eec009130 tx=0x7f0eec008030 comp rx=0 tx=0).stop 2024-03-18T20:27:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.173+0000 7f0f044d1700 1 -- 172.21.15.1:0/162967064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0efc0ad790 msgr2=0x7f0efc14c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.173+0000 7f0f044d1700 1 --2- 172.21.15.1:0/162967064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0efc0ad790 0x7f0efc14c830 secure :-1 s=READY pgs=3092 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800d800 tx=0x7f0ef800dbc0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.173+0000 7f0f044d1700 1 -- 172.21.15.1:0/162967064 shutdown_connections 2024-03-18T20:27:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.173+0000 7f0f044d1700 1 --2- 172.21.15.1:0/162967064 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0ee006aed0 0x7f0ee006d390 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.173+0000 7f0f044d1700 1 --2- 172.21.15.1:0/162967064 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0efc06a750 0x7f0efc15f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.173+0000 7f0f044d1700 1 --2- 172.21.15.1:0/162967064 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0efc15d6b0 0x7f0efc14cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.173+0000 7f0f044d1700 1 --2- 172.21.15.1:0/162967064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0efc0ad790 0x7f0efc14c830 unknown :-1 s=CLOSED pgs=3092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.173+0000 7f0f044d1700 1 -- 172.21.15.1:0/162967064 >> 172.21.15.1:0/162967064 conn(0x7f0efc06b860 msgr2=0x7f0efc0b2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.173+0000 7f0f044d1700 1 -- 172.21.15.1:0/162967064 shutdown_connections 2024-03-18T20:27:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.173+0000 7f0f044d1700 1 -- 172.21.15.1:0/162967064 wait complete. 2024-03-18T20:27:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 136 2024-03-18T20:27:32.592 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 136, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T20:27:22.047616+0000', 'modified': '2024-03-18T20:27:25.113850+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 13173}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_13173': {'gid': 13173, 'name': 'c', 'rank': 0, 'incarnation': 132, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6837/3909787366', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6836', 'nonce': 3909787366}, {'type': 'v1', 'addr': '172.21.15.1:6837', 'nonce': 3909787366}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [36], 'metadata_pool': 35, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T20:27:32.592 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T20:27:32.593 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T20:27:32.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.728+0000 7f0f8a4eb700 1 Processor -- start 2024-03-18T20:27:32.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.728+0000 7f0f8a4eb700 1 -- start start 2024-03-18T20:27:32.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.729+0000 7f0f8a4eb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f8406ae60 0x7f0f84168150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.729+0000 7f0f8a4eb700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f840cf600 con 0x7f0f840cfa90 2024-03-18T20:27:32.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.729+0000 7f0f8a4eb700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f840ce720 con 0x7f0f8415d5a0 2024-03-18T20:27:32.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.729+0000 7f0f8a4eb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f840ce9f0 con 0x7f0f8406ae60 2024-03-18T20:27:32.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.729+0000 7f0f7bfff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0f840cfa90 0x7f0f8406a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37468/0 (socket says 172.21.15.1:37468) 2024-03-18T20:27:32.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.729+0000 7f0f7bfff700 1 -- 172.21.15.1:0/4217953171 learned_addr learned my addr 172.21.15.1:0/4217953171 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:32.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.729+0000 7f0f88a88700 1 --2- 172.21.15.1:0/4217953171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f8406ae60 0x7f0f84168150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.729+0000 7f0f837fe700 1 -- 172.21.15.1:0/4217953171 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908666812 0 0) 0x7f0f840cf600 con 0x7f0f840cfa90 2024-03-18T20:27:32.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.729+0000 7f0f837fe700 1 -- 172.21.15.1:0/4217953171 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f700035b0 con 0x7f0f840cfa90 2024-03-18T20:27:32.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f837fe700 1 -- 172.21.15.1:0/4217953171 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4051442598 0 0) 0x7f0f700035b0 con 0x7f0f840cfa90 2024-03-18T20:27:32.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f837fe700 1 -- 172.21.15.1:0/4217953171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f8406ae60 msgr2=0x7f0f84168150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:32.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f837fe700 1 --2- 172.21.15.1:0/4217953171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f8406ae60 0x7f0f84168150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f837fe700 1 -- 172.21.15.1:0/4217953171 >> v1:172.21.15.165:6789/0 conn(0x7f0f8415d5a0 legacy=0x7f0f840ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:32.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f837fe700 1 -- 172.21.15.1:0/4217953171 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f840cecd0 con 0x7f0f840cfa90 2024-03-18T20:27:32.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f837fe700 1 -- 172.21.15.1:0/4217953171 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f0f6c001670 con 0x7f0f840cfa90 2024-03-18T20:27:32.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f837fe700 1 -- 172.21.15.1:0/4217953171 >> v1:172.21.15.1:6789/0 conn(0x7f0f840cfa90 legacy=0x7f0f8406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:32.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f837fe700 1 --2- 172.21.15.1:0/4217953171 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f70003a30 0x7f0f70003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f837fe700 1 --2- 172.21.15.1:0/4217953171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f700043e0 0x7f0f700048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f837fe700 1 --2- 172.21.15.1:0/4217953171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f8406ae60 0x7f0f70005110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f837fe700 1 -- 172.21.15.1:0/4217953171 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f840ce9f0 con 0x7f0f700043e0 2024-03-18T20:27:32.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f837fe700 1 -- 172.21.15.1:0/4217953171 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7000ba70 con 0x7f0f70003a30 2024-03-18T20:27:32.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f837fe700 1 -- 172.21.15.1:0/4217953171 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7000bc30 con 0x7f0f8406ae60 2024-03-18T20:27:32.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f7bfff700 1 --2- 172.21.15.1:0/4217953171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f700043e0 0x7f0f700048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f83fff700 1 --2- 172.21.15.1:0/4217953171 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f70003a30 0x7f0f70003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f88a88700 1 --2- 172.21.15.1:0/4217953171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f8406ae60 0x7f0f70005110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f7bfff700 1 -- 172.21.15.1:0/4217953171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f8406ae60 msgr2=0x7f0f70005110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:32.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f7bfff700 1 --2- 172.21.15.1:0/4217953171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f8406ae60 0x7f0f70005110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f7bfff700 1 -- 172.21.15.1:0/4217953171 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f70003a30 msgr2=0x7f0f70003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:32.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f7bfff700 1 --2- 172.21.15.1:0/4217953171 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f70003a30 0x7f0f70003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f7bfff700 1 -- 172.21.15.1:0/4217953171 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f700035b0 con 0x7f0f700043e0 2024-03-18T20:27:32.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f88a88700 1 --2- 172.21.15.1:0/4217953171 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f8406ae60 0x7f0f70005110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:32.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f83fff700 1 --2- 172.21.15.1:0/4217953171 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f70003a30 0x7f0f70003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:32.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.730+0000 7f0f7bfff700 1 --2- 172.21.15.1:0/4217953171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f700043e0 0x7f0f700048b0 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c001eb0 tx=0x7f0f70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b808e6fbc32095 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:32.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.731+0000 7f0f837fe700 1 -- 172.21.15.1:0/4217953171 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c00ce00 con 0x7f0f700043e0 2024-03-18T20:27:32.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.731+0000 7f0f837fe700 1 -- 172.21.15.1:0/4217953171 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6c005960 con 0x7f0f700043e0 2024-03-18T20:27:32.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.731+0000 7f0f837fe700 1 -- 172.21.15.1:0/4217953171 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c005d10 con 0x7f0f700043e0 2024-03-18T20:27:32.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.731+0000 7f0f8a4eb700 1 -- 172.21.15.1:0/4217953171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f700043e0 msgr2=0x7f0f700048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:32.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.731+0000 7f0f8a4eb700 1 --2- 172.21.15.1:0/4217953171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f700043e0 0x7f0f700048b0 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c001eb0 tx=0x7f0f70005880 comp rx=0 tx=0).stop 2024-03-18T20:27:32.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.731+0000 7f0f88a88700 1 -- 172.21.15.1:0/4217953171 reap_dead start 2024-03-18T20:27:32.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.731+0000 7f0f8a4eb700 1 -- 172.21.15.1:0/4217953171 shutdown_connections 2024-03-18T20:27:32.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.731+0000 7f0f8a4eb700 1 -- 172.21.15.1:0/4217953171 >> 172.21.15.1:0/4217953171 conn(0x7f0f840cc9b0 msgr2=0x7f0f840ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:32.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.731+0000 7f0f8a4eb700 1 -- 172.21.15.1:0/4217953171 shutdown_connections 2024-03-18T20:27:32.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.731+0000 7f0f8a4eb700 1 -- 172.21.15.1:0/4217953171 wait complete. 2024-03-18T20:27:32.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.733+0000 7f0f8a4eb700 1 Processor -- start 2024-03-18T20:27:32.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.733+0000 7f0f8a4eb700 1 -- start start 2024-03-18T20:27:32.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.734+0000 7f0f8a4eb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f700043e0 0x7f0f840c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.734+0000 7f0f8a4eb700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f840aecc0 0x7f0f840c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.734+0000 7f0f8a4eb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f840c9af0 0x7f0f841577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.734+0000 7f0f8a4eb700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f840c6df0 con 0x7f0f840c9af0 2024-03-18T20:27:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.734+0000 7f0f8a4eb700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f840af250 con 0x7f0f840aecc0 2024-03-18T20:27:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.734+0000 7f0f8a4eb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f840af530 con 0x7f0f700043e0 2024-03-18T20:27:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.734+0000 7f0f88a88700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f840c9af0 0x7f0f841577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.734+0000 7f0f88a88700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f840c9af0 0x7f0f841577b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36652/0 (socket says 172.21.15.1:36652) 2024-03-18T20:27:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.734+0000 7f0f88a88700 1 -- 172.21.15.1:0/1260850763 learned_addr learned my addr 172.21.15.1:0/1260850763 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.734+0000 7f0f83fff700 1 --2- 172.21.15.1:0/1260850763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f700043e0 0x7f0f840c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.734+0000 7f0f7bfff700 1 --2- 172.21.15.1:0/1260850763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f840aecc0 0x7f0f840c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.734+0000 7f0f88a88700 1 -- 172.21.15.1:0/1260850763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f700043e0 msgr2=0x7f0f840c68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.734+0000 7f0f88a88700 1 --2- 172.21.15.1:0/1260850763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f700043e0 0x7f0f840c68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.734+0000 7f0f88a88700 1 -- 172.21.15.1:0/1260850763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f840aecc0 msgr2=0x7f0f840c95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.734+0000 7f0f88a88700 1 --2- 172.21.15.1:0/1260850763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f840aecc0 0x7f0f840c95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.734+0000 7f0f88a88700 1 -- 172.21.15.1:0/1260850763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f840af810 con 0x7f0f840c9af0 2024-03-18T20:27:32.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.734+0000 7f0f88a88700 1 --2- 172.21.15.1:0/1260850763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f840c9af0 0x7f0f841577b0 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 crypto rx=0x7f0f8409e000 tx=0x7f0f7c00f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:32.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.735+0000 7f0f817fa700 1 -- 172.21.15.1:0/1260850763 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c00fcb0 con 0x7f0f840c9af0 2024-03-18T20:27:32.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.735+0000 7f0f817fa700 1 -- 172.21.15.1:0/1260850763 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7c011040 con 0x7f0f840c9af0 2024-03-18T20:27:32.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.735+0000 7f0f8a4eb700 1 -- 172.21.15.1:0/1260850763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f840afaf0 con 0x7f0f840c9af0 2024-03-18T20:27:32.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.735+0000 7f0f817fa700 1 -- 172.21.15.1:0/1260850763 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c017400 con 0x7f0f840c9af0 2024-03-18T20:27:32.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.735+0000 7f0f8a4eb700 1 -- 172.21.15.1:0/1260850763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f84157cf0 con 0x7f0f840c9af0 2024-03-18T20:27:32.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.736+0000 7f0f817fa700 1 -- 172.21.15.1:0/1260850763 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f0f7c0203b0 con 0x7f0f840c9af0 2024-03-18T20:27:32.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.737+0000 7f0f817fa700 1 --2- 172.21.15.1:0/1260850763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0f6406aed0 0x7f0f6406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.737+0000 7f0f817fa700 1 -- 172.21.15.1:0/1260850763 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f0f7c0899c0 con 0x7f0f840c9af0 2024-03-18T20:27:32.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.737+0000 7f0f83fff700 1 --2- 172.21.15.1:0/1260850763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0f6406aed0 0x7f0f6406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.737+0000 7f0f83fff700 1 --2- 172.21.15.1:0/1260850763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0f6406aed0 0x7f0f6406d390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0f74007fd0 tx=0x7f0f74009040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.738+0000 7f0f96694b80 1 Processor -- start 2024-03-18T20:27:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.739+0000 7f0f96694b80 1 -- start start 2024-03-18T20:27:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.739+0000 7f0f96694b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fc1c10 0x564a69fc1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.739+0000 7f0f96694b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 0x564a69fde580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.739+0000 7f0f96694b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fd3630 0x564a69f96160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.739+0000 7f0f96694b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a69f0cf20 con 0x564a69fd3630 2024-03-18T20:27:32.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.739+0000 7f0f96694b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a69eeec30 con 0x564a69f930a0 2024-03-18T20:27:32.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.739+0000 7f0f96694b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x564a69eec430 con 0x564a69fc1c10 2024-03-18T20:27:32.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.739+0000 7f0f88a88700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fd3630 0x564a69f96160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.739+0000 7f0f88a88700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fd3630 0x564a69f96160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36658/0 (socket says 172.21.15.1:36658) 2024-03-18T20:27:32.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.739+0000 7f0f88a88700 1 -- 172.21.15.1:0/240177278 learned_addr learned my addr 172.21.15.1:0/240177278 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:32.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.740+0000 7f0f83fff700 1 --2- 172.21.15.1:0/240177278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fc1c10 0x564a69fc1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.740+0000 7f0f7bfff700 1 --2- 172.21.15.1:0/240177278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 0x564a69fde580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.740+0000 7f0f88a88700 1 -- 172.21.15.1:0/240177278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fc1c10 msgr2=0x564a69fc1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:32.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.740+0000 7f0f88a88700 1 --2- 172.21.15.1:0/240177278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fc1c10 0x564a69fc1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.740+0000 7f0f88a88700 1 -- 172.21.15.1:0/240177278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 msgr2=0x564a69fde580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:32.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.740+0000 7f0f88a88700 1 --2- 172.21.15.1:0/240177278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 0x564a69fde580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.740+0000 7f0f88a88700 1 -- 172.21.15.1:0/240177278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a69cdb790 con 0x564a69fd3630 2024-03-18T20:27:32.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.740+0000 7f0f88a88700 1 --2- 172.21.15.1:0/240177278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fd3630 0x564a69f96160 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c0581d0 tx=0x7f0f7c057eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bf722b58c35547d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.740+0000 7f0f79ffb700 1 -- 172.21.15.1:0/240177278 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c060630 con 0x564a69fd3630 2024-03-18T20:27:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.740+0000 7f0f79ffb700 1 -- 172.21.15.1:0/240177278 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7c0584a0 con 0x564a69fd3630 2024-03-18T20:27:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.740+0000 7f0f79ffb700 1 -- 172.21.15.1:0/240177278 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c057730 con 0x564a69fd3630 2024-03-18T20:27:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.741+0000 7f0f96694b80 1 -- 172.21.15.1:0/240177278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fd3630 msgr2=0x564a69f96160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.741+0000 7f0f96694b80 1 --2- 172.21.15.1:0/240177278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fd3630 0x564a69f96160 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c0581d0 tx=0x7f0f7c057eb0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.741+0000 7f0f96694b80 1 -- 172.21.15.1:0/240177278 shutdown_connections 2024-03-18T20:27:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.741+0000 7f0f96694b80 1 --2- 172.21.15.1:0/240177278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fc1c10 0x564a69fc1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.741+0000 7f0f96694b80 1 --2- 172.21.15.1:0/240177278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fd3630 0x564a69f96160 unknown :-1 s=CLOSED pgs=3096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.741+0000 7f0f96694b80 1 --2- 172.21.15.1:0/240177278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 0x564a69fde580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.741+0000 7f0f96694b80 1 -- 172.21.15.1:0/240177278 >> 172.21.15.1:0/240177278 conn(0x564a69ef5cf0 msgr2=0x564a69f95a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.741+0000 7f0f96694b80 1 -- 172.21.15.1:0/240177278 shutdown_connections 2024-03-18T20:27:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.741+0000 7f0f96694b80 1 -- 172.21.15.1:0/240177278 wait complete. 2024-03-18T20:27:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.741+0000 7f0f96694b80 1 -- start start 2024-03-18T20:27:32.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.742+0000 7f0f96694b80 1 Processor -- start 2024-03-18T20:27:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.742+0000 7f0f96694b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 0x564a69fd6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.742+0000 7f0f96694b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fc1c10 0x564a69fb59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.742+0000 7f0f96694b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fd3630 0x564a69fbdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.742+0000 7f0f96694b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a69ee7d20 con 0x564a69fc1c10 2024-03-18T20:27:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.742+0000 7f0f96694b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a69e83770 con 0x564a69f930a0 2024-03-18T20:27:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.742+0000 7f0f96694b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x564a69ea66f0 con 0x564a69fd3630 2024-03-18T20:27:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.742+0000 7f0f96694b80 10 client.0 _ll_register_callbacks cb 0x564a69cef5a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:27:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.742+0000 7f0f7bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fc1c10 0x564a69fb59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.743+0000 7f0f7bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fc1c10 0x564a69fb59c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36666/0 (socket says 172.21.15.1:36666) 2024-03-18T20:27:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.743+0000 7f0f7bfff700 1 -- 172.21.15.1:0/2133490942 learned_addr learned my addr 172.21.15.1:0/2133490942 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.743+0000 7f0f88a88700 1 --2- 172.21.15.1:0/2133490942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fd3630 0x564a69fbdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.743+0000 7f0f83fff700 1 --2- 172.21.15.1:0/2133490942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 0x564a69fd6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.743+0000 7f0f7bfff700 1 -- 172.21.15.1:0/2133490942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fd3630 msgr2=0x564a69fbdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.743+0000 7f0f7bfff700 1 --2- 172.21.15.1:0/2133490942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fd3630 0x564a69fbdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.743+0000 7f0f7bfff700 1 -- 172.21.15.1:0/2133490942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 msgr2=0x564a69fd6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.743+0000 7f0f7bfff700 1 --2- 172.21.15.1:0/2133490942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 0x564a69fd6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.743+0000 7f0f7bfff700 1 -- 172.21.15.1:0/2133490942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f7c089040 con 0x564a69fc1c10 2024-03-18T20:27:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.743+0000 7f0f83fff700 1 --2- 172.21.15.1:0/2133490942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 0x564a69fd6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.743+0000 7f0f7bfff700 1 --2- 172.21.15.1:0/2133490942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fc1c10 0x564a69fb59c0 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c011000 tx=0x7f0f6c009050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26084a74bc7fcd7c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.743+0000 7f0f62ffd700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T20:27:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.743+0000 7f0f62ffd700 1 -- 172.21.15.1:0/2133490942 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c0017e0 con 0x564a69fc1c10 2024-03-18T20:27:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.743+0000 7f0f62ffd700 1 -- 172.21.15.1:0/2133490942 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f0f6c005ce0 con 0x564a69fc1c10 2024-03-18T20:27:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.743+0000 7f0f96694b80 1 -- 172.21.15.1:0/2133490942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564a69f89e90 con 0x564a69fc1c10 2024-03-18T20:27:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.743+0000 7f0f62ffd700 1 -- 172.21.15.1:0/2133490942 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=136) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f6c005ea0 con 0x564a69fc1c10 2024-03-18T20:27:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.743+0000 7f0f96694b80 10 client.13347 fetch_fsmap learned FSMap version 136 2024-03-18T20:27:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.743+0000 7f0f96694b80 1 -- 172.21.15.1:0/2133490942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=136,monmap=2+}) v3 -- 0x564a69dc80a0 con 0x564a69fc1c10 2024-03-18T20:27:32.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.743+0000 7f0f62ffd700 1 -- 172.21.15.1:0/2133490942 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564a69f89e90 con 0x564a69fc1c10 2024-03-18T20:27:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.744+0000 7f0f62ffd700 1 -- 172.21.15.1:0/2133490942 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 136) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0f6c019030 con 0x564a69fc1c10 2024-03-18T20:27:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.744+0000 7f0f96694b80 10 client.13347 fetch_fsmap finished waiting for FSMap version 136 2024-03-18T20:27:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.744+0000 7f0f96694b80 10 client.13347 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T20:27:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.744+0000 7f0f96694b80 20 client.13347 populate_metadata read hostname 'smithi001' 2024-03-18T20:27:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.744+0000 7f0f96694b80 1 --2- 172.21.15.1:0/2133490942 >> [v2:172.21.15.1:6836/3909787366,v1:172.21.15.1:6837/3909787366] conn(0x564a69fb7f10 0x564a69fb82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.744+0000 7f0f96694b80 4 client.13347 mds_command: new command op to 13173 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T20:27:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.744+0000 7f0f96694b80 1 -- 172.21.15.1:0/2133490942 --> [v2:172.21.15.1:6836/3909787366,v1:172.21.15.1:6837/3909787366] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564a69f67010 con 0x564a69fb7f10 2024-03-18T20:27:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.744+0000 7f0f83fff700 1 --2- 172.21.15.1:0/2133490942 >> [v2:172.21.15.1:6836/3909787366,v1:172.21.15.1:6837/3909787366] conn(0x564a69fb7f10 0x564a69fb82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.744+0000 7f0f62ffd700 10 client.13347 ms_handle_connect on v2:172.21.15.1:6836/3909787366 2024-03-18T20:27:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.744+0000 7f0f83fff700 1 --2- 172.21.15.1:0/2133490942 >> [v2:172.21.15.1:6836/3909787366,v1:172.21.15.1:6837/3909787366] conn(0x564a69fb7f10 0x564a69fb82f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb62615ba245580f server_cookie=4ad51e64b02989e8 in_seq=0 out_seq=0 2024-03-18T20:27:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.745+0000 7f0f62ffd700 1 -- 172.21.15.1:0/2133490942 <== mds.0 v2:172.21.15.1:6836/3909787366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564a69f67010 con 0x564a69fb7f10 2024-03-18T20:27:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.745+0000 7f0f62ffd700 10 client.13347 handle_command_reply: tid=0 2024-03-18T20:27:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.745+0000 7f0f96694b80 1 client.13347 shutdown 2024-03-18T20:27:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.745+0000 7f0f96694b80 1 -- 172.21.15.1:0/2133490942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fc1c10 msgr2=0x564a69fb59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:32.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.745+0000 7f0f96694b80 1 --2- 172.21.15.1:0/2133490942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fc1c10 0x564a69fb59c0 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c011000 tx=0x7f0f6c009050 comp rx=0 tx=0).stop 2024-03-18T20:27:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.745+0000 7f0f96694b80 1 -- 172.21.15.1:0/2133490942 shutdown_connections 2024-03-18T20:27:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.745+0000 7f0f96694b80 1 --2- 172.21.15.1:0/2133490942 >> [v2:172.21.15.1:6836/3909787366,v1:172.21.15.1:6837/3909787366] conn(0x564a69fb7f10 0x564a69fb82f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.745+0000 7f0f96694b80 1 --2- 172.21.15.1:0/2133490942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fd3630 0x564a69fbdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.745+0000 7f0f96694b80 1 --2- 172.21.15.1:0/2133490942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 0x564a69fd6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.745+0000 7f0f96694b80 1 --2- 172.21.15.1:0/2133490942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fc1c10 0x564a69fb59c0 unknown :-1 s=CLOSED pgs=3097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.745+0000 7f0f62ffd700 0 client.13347 ms_handle_reset on v2:172.21.15.1:6836/3909787366 2024-03-18T20:27:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.745+0000 7f0f96694b80 1 -- 172.21.15.1:0/2133490942 >> 172.21.15.1:0/2133490942 conn(0x564a69ef5cf0 msgr2=0x564a69f91a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.745+0000 7f0f96694b80 1 -- 172.21.15.1:0/2133490942 shutdown_connections 2024-03-18T20:27:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.745+0000 7f0f96694b80 1 -- 172.21.15.1:0/2133490942 wait complete. 2024-03-18T20:27:32.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.745+0000 7f0f96694b80 20 client.13347 trim_cache size 0 max 16384 2024-03-18T20:27:32.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.763+0000 7f0f96694b80 1 Processor -- start 2024-03-18T20:27:32.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.763+0000 7f0f96694b80 1 -- start start 2024-03-18T20:27:32.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.763+0000 7f0f96694b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 0x564a69fb47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.763+0000 7f0f96694b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fc1c10 0x564a69fbda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.763+0000 7f0f96694b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fd3630 0x564a69fbf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.763+0000 7f0f96694b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a69cdc390 con 0x564a69fd3630 2024-03-18T20:27:32.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.763+0000 7f0f96694b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a69fd68d0 con 0x564a69f930a0 2024-03-18T20:27:32.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.763+0000 7f0f96694b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x564a69fd6360 con 0x564a69fc1c10 2024-03-18T20:27:32.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.764+0000 7f0f88a88700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fd3630 0x564a69fbf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.764+0000 7f0f88a88700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fd3630 0x564a69fbf470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36680/0 (socket says 172.21.15.1:36680) 2024-03-18T20:27:32.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.764+0000 7f0f88a88700 1 -- 172.21.15.1:0/2258337694 learned_addr learned my addr 172.21.15.1:0/2258337694 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:32.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.764+0000 7f0f7bfff700 1 --2- 172.21.15.1:0/2258337694 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fc1c10 0x564a69fbda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.764+0000 7f0f83fff700 1 --2- 172.21.15.1:0/2258337694 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 0x564a69fb47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.764+0000 7f0f88a88700 1 -- 172.21.15.1:0/2258337694 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fc1c10 msgr2=0x564a69fbda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:32.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.764+0000 7f0f88a88700 1 --2- 172.21.15.1:0/2258337694 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fc1c10 0x564a69fbda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.764+0000 7f0f88a88700 1 -- 172.21.15.1:0/2258337694 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 msgr2=0x564a69fb47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:32.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.764+0000 7f0f88a88700 1 --2- 172.21.15.1:0/2258337694 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 0x564a69fb47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.764+0000 7f0f88a88700 1 -- 172.21.15.1:0/2258337694 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a69eae010 con 0x564a69fd3630 2024-03-18T20:27:32.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.764+0000 7f0f83fff700 1 --2- 172.21.15.1:0/2258337694 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 0x564a69fb47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:32.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.764+0000 7f0f88a88700 1 --2- 172.21.15.1:0/2258337694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fd3630 0x564a69fbf470 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c0581d0 tx=0x7f0f7c058ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97446228b801c18f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:32.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.764+0000 7f0f7a7fc700 1 -- 172.21.15.1:0/2258337694 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c087040 con 0x564a69fd3630 2024-03-18T20:27:32.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.764+0000 7f0f7a7fc700 1 -- 172.21.15.1:0/2258337694 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7c060630 con 0x564a69fd3630 2024-03-18T20:27:32.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.764+0000 7f0f7a7fc700 1 -- 172.21.15.1:0/2258337694 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c06f040 con 0x564a69fd3630 2024-03-18T20:27:32.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.765+0000 7f0f96694b80 1 -- 172.21.15.1:0/2258337694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fd3630 msgr2=0x564a69fbf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:32.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.765+0000 7f0f96694b80 1 --2- 172.21.15.1:0/2258337694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fd3630 0x564a69fbf470 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c0581d0 tx=0x7f0f7c058ab0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.765+0000 7f0f96694b80 1 -- 172.21.15.1:0/2258337694 shutdown_connections 2024-03-18T20:27:32.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.765+0000 7f0f96694b80 1 --2- 172.21.15.1:0/2258337694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fd3630 0x564a69fbf470 unknown :-1 s=CLOSED pgs=3098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.765+0000 7f0f96694b80 1 --2- 172.21.15.1:0/2258337694 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 0x564a69fb47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.765+0000 7f0f96694b80 1 --2- 172.21.15.1:0/2258337694 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fc1c10 0x564a69fbda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.765+0000 7f0f96694b80 1 -- 172.21.15.1:0/2258337694 >> 172.21.15.1:0/2258337694 conn(0x564a69ef5cf0 msgr2=0x564a69ffeaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:32.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.765+0000 7f0f96694b80 1 -- 172.21.15.1:0/2258337694 shutdown_connections 2024-03-18T20:27:32.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.765+0000 7f0f96694b80 1 -- 172.21.15.1:0/2258337694 wait complete. 2024-03-18T20:27:32.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.765+0000 7f0f96694b80 1 -- start start 2024-03-18T20:27:32.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.766+0000 7f0f96694b80 1 Processor -- start 2024-03-18T20:27:32.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.766+0000 7f0f96694b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 0x564a6a0435a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.766+0000 7f0f96694b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fc1c10 0x564a69fb42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.766+0000 7f0f96694b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fd3630 0x564a69fb4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.766+0000 7f0f96694b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a6a045100 con 0x564a69fc1c10 2024-03-18T20:27:32.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.766+0000 7f0f96694b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a69fb3ad0 con 0x564a69f930a0 2024-03-18T20:27:32.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.766+0000 7f0f96694b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x564a69fe0610 con 0x564a69fd3630 2024-03-18T20:27:32.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.766+0000 7f0f96694b80 10 client.0 _ll_register_callbacks cb 0x7f0f6c015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:27:32.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.766+0000 7f0f7bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fc1c10 0x564a69fb42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.766+0000 7f0f7bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fc1c10 0x564a69fb42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36688/0 (socket says 172.21.15.1:36688) 2024-03-18T20:27:32.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.766+0000 7f0f7bfff700 1 -- 172.21.15.1:0/3817897488 learned_addr learned my addr 172.21.15.1:0/3817897488 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:32.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.766+0000 7f0f83fff700 1 --2- 172.21.15.1:0/3817897488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 0x564a6a0435a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f88a88700 1 --2- 172.21.15.1:0/3817897488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fd3630 0x564a69fb4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f7bfff700 1 -- 172.21.15.1:0/3817897488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fd3630 msgr2=0x564a69fb4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:32.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f7bfff700 1 --2- 172.21.15.1:0/3817897488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fd3630 0x564a69fb4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f7bfff700 1 -- 172.21.15.1:0/3817897488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 msgr2=0x564a6a0435a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:32.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f7bfff700 1 --2- 172.21.15.1:0/3817897488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 0x564a6a0435a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f7bfff700 1 -- 172.21.15.1:0/3817897488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564a69cdb790 con 0x564a69fc1c10 2024-03-18T20:27:32.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f88a88700 1 --2- 172.21.15.1:0/3817897488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fd3630 0x564a69fb4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:32.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f7bfff700 1 --2- 172.21.15.1:0/3817897488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fc1c10 0x564a69fb42d0 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c011000 tx=0x7f0f6c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26f71479b2e6c2e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:32.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f62ffd700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T20:27:32.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f62ffd700 1 -- 172.21.15.1:0/3817897488 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c0072e0 con 0x564a69fc1c10 2024-03-18T20:27:32.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f62ffd700 1 -- 172.21.15.1:0/3817897488 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f0f6c007440 con 0x564a69fc1c10 2024-03-18T20:27:32.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f96694b80 1 -- 172.21.15.1:0/3817897488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0f7c0584a0 con 0x564a69fc1c10 2024-03-18T20:27:32.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f62ffd700 1 -- 172.21.15.1:0/3817897488 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=136) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f6c001ac0 con 0x564a69fc1c10 2024-03-18T20:27:32.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f96694b80 10 client.13353 fetch_fsmap learned FSMap version 136 2024-03-18T20:27:32.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f96694b80 1 -- 172.21.15.1:0/3817897488 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=136,monmap=2+}) v3 -- 0x564a69d3c1d0 con 0x564a69fc1c10 2024-03-18T20:27:32.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f62ffd700 1 -- 172.21.15.1:0/3817897488 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7c0584a0 con 0x564a69fc1c10 2024-03-18T20:27:32.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f62ffd700 1 -- 172.21.15.1:0/3817897488 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 136) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0f6c019070 con 0x564a69fc1c10 2024-03-18T20:27:32.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f96694b80 10 client.13353 fetch_fsmap finished waiting for FSMap version 136 2024-03-18T20:27:32.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f96694b80 10 client.13353 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T20:27:32.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f96694b80 20 client.13353 populate_metadata read hostname 'smithi001' 2024-03-18T20:27:32.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f96694b80 1 --2- 172.21.15.1:0/3817897488 >> [v2:172.21.15.1:6836/3909787366,v1:172.21.15.1:6837/3909787366] conn(0x564a6a043e10 0x564a6a0441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f96694b80 4 client.13353 mds_command: new command op to 13173 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T20:27:32.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.767+0000 7f0f96694b80 1 -- 172.21.15.1:0/3817897488 --> [v2:172.21.15.1:6836/3909787366,v1:172.21.15.1:6837/3909787366] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564a69d21040 con 0x564a6a043e10 2024-03-18T20:27:32.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.768+0000 7f0f83fff700 1 --2- 172.21.15.1:0/3817897488 >> [v2:172.21.15.1:6836/3909787366,v1:172.21.15.1:6837/3909787366] conn(0x564a6a043e10 0x564a6a0441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.768+0000 7f0f62ffd700 10 client.13353 ms_handle_connect on v2:172.21.15.1:6836/3909787366 2024-03-18T20:27:32.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.768+0000 7f0f83fff700 1 --2- 172.21.15.1:0/3817897488 >> [v2:172.21.15.1:6836/3909787366,v1:172.21.15.1:6837/3909787366] conn(0x564a6a043e10 0x564a6a0441f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e93a9ec06a49be3 server_cookie=42e45b6c30cbfc6e in_seq=0 out_seq=0 2024-03-18T20:27:32.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.768+0000 7f0f62ffd700 1 -- 172.21.15.1:0/3817897488 <== mds.0 v2:172.21.15.1:6836/3909787366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x564a69d21040 con 0x564a6a043e10 2024-03-18T20:27:32.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.768+0000 7f0f62ffd700 10 client.13353 handle_command_reply: tid=0 2024-03-18T20:27:32.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.768+0000 7f0f96694b80 1 client.13353 shutdown 2024-03-18T20:27:32.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.768+0000 7f0f96694b80 1 -- 172.21.15.1:0/3817897488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fc1c10 msgr2=0x564a69fb42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:32.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.768+0000 7f0f96694b80 1 --2- 172.21.15.1:0/3817897488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fc1c10 0x564a69fb42d0 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c011000 tx=0x7f0f6c010000 comp rx=0 tx=0).stop 2024-03-18T20:27:32.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.768+0000 7f0f96694b80 1 -- 172.21.15.1:0/3817897488 shutdown_connections 2024-03-18T20:27:32.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.768+0000 7f0f96694b80 1 --2- 172.21.15.1:0/3817897488 >> [v2:172.21.15.1:6836/3909787366,v1:172.21.15.1:6837/3909787366] conn(0x564a6a043e10 0x564a6a0441f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.768+0000 7f0f96694b80 1 --2- 172.21.15.1:0/3817897488 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x564a69fd3630 0x564a69fb4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.768+0000 7f0f96694b80 1 --2- 172.21.15.1:0/3817897488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564a69f930a0 0x564a6a0435a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.768+0000 7f0f96694b80 1 --2- 172.21.15.1:0/3817897488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a69fc1c10 0x564a69fb42d0 unknown :-1 s=CLOSED pgs=3099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.768+0000 7f0f62ffd700 0 client.13353 ms_handle_reset on v2:172.21.15.1:6836/3909787366 2024-03-18T20:27:32.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.768+0000 7f0f96694b80 1 -- 172.21.15.1:0/3817897488 >> 172.21.15.1:0/3817897488 conn(0x564a69ef5cf0 msgr2=0x564a69fbdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:32.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.768+0000 7f0f96694b80 1 -- 172.21.15.1:0/3817897488 shutdown_connections 2024-03-18T20:27:32.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.768+0000 7f0f96694b80 1 -- 172.21.15.1:0/3817897488 wait complete. 2024-03-18T20:27:32.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.769+0000 7f0f96694b80 20 client.13353 trim_cache size 0 max 16384 2024-03-18T20:27:32.780 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:27:32.780 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "9c1c697f-8655-4ef3-8a13-da568d2cbe86", 2024-03-18T20:27:32.780 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-18T20:27:32.780 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 13173, 2024-03-18T20:27:32.780 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-18T20:27:32.780 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-18T20:27:32.780 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-18T20:27:32.780 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 10.720777571999999, 2024-03-18T20:27:32.781 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 136, 2024-03-18T20:27:32.781 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 274, 2024-03-18T20:27:32.781 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 274, 2024-03-18T20:27:32.781 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 16.039863478000001 2024-03-18T20:27:32.781 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:27:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.769+0000 7f0f8a4eb700 1 -- 172.21.15.1:0/1260850763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0f6406aed0 msgr2=0x7f0f6406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.769+0000 7f0f8a4eb700 1 --2- 172.21.15.1:0/1260850763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0f6406aed0 0x7f0f6406d390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0f74007fd0 tx=0x7f0f74009040 comp rx=0 tx=0).stop 2024-03-18T20:27:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.769+0000 7f0f8a4eb700 1 -- 172.21.15.1:0/1260850763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f840c9af0 msgr2=0x7f0f841577b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.769+0000 7f0f8a4eb700 1 --2- 172.21.15.1:0/1260850763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f840c9af0 0x7f0f841577b0 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 crypto rx=0x7f0f8409e000 tx=0x7f0f7c00f3b0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.769+0000 7f0f8a4eb700 1 -- 172.21.15.1:0/1260850763 shutdown_connections 2024-03-18T20:27:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.769+0000 7f0f8a4eb700 1 --2- 172.21.15.1:0/1260850763 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0f6406aed0 0x7f0f6406d390 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.769+0000 7f0f8a4eb700 1 --2- 172.21.15.1:0/1260850763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0f700043e0 0x7f0f840c68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.769+0000 7f0f8a4eb700 1 --2- 172.21.15.1:0/1260850763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f840c9af0 0x7f0f841577b0 unknown :-1 s=CLOSED pgs=3095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.769+0000 7f0f8a4eb700 1 --2- 172.21.15.1:0/1260850763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f840aecc0 0x7f0f840c95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.769+0000 7f0f8a4eb700 1 -- 172.21.15.1:0/1260850763 >> 172.21.15.1:0/1260850763 conn(0x7f0f840cc9b0 msgr2=0x7f0f8415e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.769+0000 7f0f8a4eb700 1 -- 172.21.15.1:0/1260850763 shutdown_connections 2024-03-18T20:27:32.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.769+0000 7f0f8a4eb700 1 -- 172.21.15.1:0/1260850763 wait complete. 2024-03-18T20:27:32.787 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T20:27:32.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.907+0000 7fbd905dc700 1 Processor -- start 2024-03-18T20:27:32.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.907+0000 7fbd905dc700 1 -- start start 2024-03-18T20:27:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.908+0000 7fbd905dc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbd8806ae60 0x7fbd88166270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.908+0000 7fbd905dc700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd88072720 con 0x7fbd8815b120 2024-03-18T20:27:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.908+0000 7fbd905dc700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8806ffc0 con 0x7fbd8815b500 2024-03-18T20:27:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.908+0000 7fbd905dc700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd8806b520 con 0x7fbd8806ae60 2024-03-18T20:27:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.908+0000 7fbd8eb79700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbd8806ae60 0x7fbd88166270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.908+0000 7fbd8e378700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbd8815b120 0x7fbd88072010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37474/0 (socket says 172.21.15.1:37474) 2024-03-18T20:27:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.908+0000 7fbd8e378700 1 -- 172.21.15.1:0/2108496960 learned_addr learned my addr 172.21.15.1:0/2108496960 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.908+0000 7fbd8eb79700 1 -- 172.21.15.1:0/2108496960 >> v1:172.21.15.165:6789/0 conn(0x7fbd8815b500 legacy=0x7fbd8806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.908+0000 7fbd8eb79700 1 -- 172.21.15.1:0/2108496960 >> v1:172.21.15.1:6789/0 conn(0x7fbd8815b120 legacy=0x7fbd88072010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.908+0000 7fbd8eb79700 1 -- 172.21.15.1:0/2108496960 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd8806ef20 con 0x7fbd8806ae60 2024-03-18T20:27:32.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.909+0000 7fbd8eb79700 1 --2- 172.21.15.1:0/2108496960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8806ae60 0x7fbd88166270 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 crypto rx=0x7fbd80009d90 tx=0x7fbd8000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0bf8b37bbc8c77 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:32.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.909+0000 7fbd8d376700 1 -- 172.21.15.1:0/2108496960 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8000e070 con 0x7fbd8806ae60 2024-03-18T20:27:32.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.909+0000 7fbd8d376700 1 -- 172.21.15.1:0/2108496960 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8000bb90 con 0x7fbd8806ae60 2024-03-18T20:27:32.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.909+0000 7fbd8d376700 1 -- 172.21.15.1:0/2108496960 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd80003a20 con 0x7fbd8806ae60 2024-03-18T20:27:32.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.909+0000 7fbd905dc700 1 -- 172.21.15.1:0/2108496960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8806ae60 msgr2=0x7fbd88166270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:32.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.909+0000 7fbd905dc700 1 --2- 172.21.15.1:0/2108496960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8806ae60 0x7fbd88166270 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 crypto rx=0x7fbd80009d90 tx=0x7fbd8000b330 comp rx=0 tx=0).stop 2024-03-18T20:27:32.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.909+0000 7fbd905dc700 1 -- 172.21.15.1:0/2108496960 shutdown_connections 2024-03-18T20:27:32.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.909+0000 7fbd905dc700 1 --2- 172.21.15.1:0/2108496960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8806ae60 0x7fbd88166270 unknown :-1 s=CLOSED pgs=3101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.909+0000 7fbd905dc700 1 -- 172.21.15.1:0/2108496960 >> 172.21.15.1:0/2108496960 conn(0x7fbd880ccab0 msgr2=0x7fbd880ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:32.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.909+0000 7fbd905dc700 1 -- 172.21.15.1:0/2108496960 shutdown_connections 2024-03-18T20:27:32.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.909+0000 7fbd905dc700 1 -- 172.21.15.1:0/2108496960 wait complete. 2024-03-18T20:27:32.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.911+0000 7fbd905dc700 1 Processor -- start 2024-03-18T20:27:32.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.911+0000 7fbd905dc700 1 -- start start 2024-03-18T20:27:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.911+0000 7fbd905dc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8806ae60 0x7fbd880c39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.911+0000 7fbd905dc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd8815b120 0x7fbd880c5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.911+0000 7fbd905dc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd8815b500 0x7fbd880c64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.911+0000 7fbd905dc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd880c4210 con 0x7fbd8806ae60 2024-03-18T20:27:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.911+0000 7fbd905dc700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd880c44f0 con 0x7fbd8815b120 2024-03-18T20:27:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.911+0000 7fbd905dc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd880c47d0 con 0x7fbd8815b500 2024-03-18T20:27:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.911+0000 7fbd8e378700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8806ae60 0x7fbd880c39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.911+0000 7fbd8e378700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8806ae60 0x7fbd880c39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36706/0 (socket says 172.21.15.1:36706) 2024-03-18T20:27:32.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.911+0000 7fbd8e378700 1 -- 172.21.15.1:0/493005502 learned_addr learned my addr 172.21.15.1:0/493005502 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.912+0000 7fbd8eb79700 1 --2- 172.21.15.1:0/493005502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd8815b500 0x7fbd880c64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.912+0000 7fbd8db77700 1 --2- 172.21.15.1:0/493005502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd8815b120 0x7fbd880c5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.912+0000 7fbd8e378700 1 -- 172.21.15.1:0/493005502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd8815b500 msgr2=0x7fbd880c64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.912+0000 7fbd8e378700 1 --2- 172.21.15.1:0/493005502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd8815b500 0x7fbd880c64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.912+0000 7fbd8e378700 1 -- 172.21.15.1:0/493005502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd8815b120 msgr2=0x7fbd880c5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.912+0000 7fbd8e378700 1 --2- 172.21.15.1:0/493005502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd8815b120 0x7fbd880c5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.912+0000 7fbd8e378700 1 -- 172.21.15.1:0/493005502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd80009a70 con 0x7fbd8806ae60 2024-03-18T20:27:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.912+0000 7fbd8db77700 1 --2- 172.21.15.1:0/493005502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd8815b120 0x7fbd880c5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.912+0000 7fbd8e378700 1 --2- 172.21.15.1:0/493005502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8806ae60 0x7fbd880c39c0 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800c490 tx=0x7fbd7800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.912+0000 7fbd7effd700 1 -- 172.21.15.1:0/493005502 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd78005d70 con 0x7fbd8806ae60 2024-03-18T20:27:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.912+0000 7fbd7effd700 1 -- 172.21.15.1:0/493005502 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd780070d0 con 0x7fbd8806ae60 2024-03-18T20:27:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.912+0000 7fbd905dc700 1 -- 172.21.15.1:0/493005502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd880c4ab0 con 0x7fbd8806ae60 2024-03-18T20:27:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.912+0000 7fbd7effd700 1 -- 172.21.15.1:0/493005502 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd78007460 con 0x7fbd8806ae60 2024-03-18T20:27:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.912+0000 7fbd905dc700 1 -- 172.21.15.1:0/493005502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd880c3f00 con 0x7fbd8806ae60 2024-03-18T20:27:32.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.913+0000 7fbd905dc700 1 -- 172.21.15.1:0/493005502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd880676a0 con 0x7fbd8806ae60 2024-03-18T20:27:32.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.914+0000 7fbd7effd700 1 -- 172.21.15.1:0/493005502 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7fbd780075c0 con 0x7fbd8806ae60 2024-03-18T20:27:32.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.914+0000 7fbd7effd700 1 --2- 172.21.15.1:0/493005502 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbd6c06aed0 0x7fbd6c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:32.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.914+0000 7fbd8db77700 1 --2- 172.21.15.1:0/493005502 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbd6c06aed0 0x7fbd6c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:32.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.914+0000 7fbd7effd700 1 -- 172.21.15.1:0/493005502 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fbd78088ba0 con 0x7fbd8806ae60 2024-03-18T20:27:32.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.914+0000 7fbd8db77700 1 --2- 172.21.15.1:0/493005502 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbd6c06aed0 0x7fbd6c06d390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbd88071290 tx=0x7fbd84008040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:32.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:32.918+0000 7fbd7effd700 1 -- 172.21.15.1:0/493005502 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbd78053900 con 0x7fbd8806ae60 2024-03-18T20:27:33.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:33.166+0000 7fbd905dc700 1 -- 172.21.15.1:0/493005502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbd88072720 con 0x7fbd8806ae60 2024-03-18T20:27:33.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:33.166+0000 7fbd7effd700 1 -- 172.21.15.1:0/493005502 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v136) v1 ==== 53+0+73 (secure 0 0 0) 0x7fbd88072720 con 0x7fbd8806ae60 2024-03-18T20:27:33.172 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T20:27:33.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:33.169+0000 7fbd905dc700 1 -- 172.21.15.1:0/493005502 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbd6c06aed0 msgr2=0x7fbd6c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:33.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:33.169+0000 7fbd905dc700 1 --2- 172.21.15.1:0/493005502 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbd6c06aed0 0x7fbd6c06d390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbd88071290 tx=0x7fbd84008040 comp rx=0 tx=0).stop 2024-03-18T20:27:33.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:33.169+0000 7fbd905dc700 1 -- 172.21.15.1:0/493005502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8806ae60 msgr2=0x7fbd880c39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:33.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:33.169+0000 7fbd905dc700 1 --2- 172.21.15.1:0/493005502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8806ae60 0x7fbd880c39c0 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800c490 tx=0x7fbd7800c850 comp rx=0 tx=0).stop 2024-03-18T20:27:33.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:33.169+0000 7fbd905dc700 1 -- 172.21.15.1:0/493005502 shutdown_connections 2024-03-18T20:27:33.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:33.169+0000 7fbd905dc700 1 --2- 172.21.15.1:0/493005502 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbd6c06aed0 0x7fbd6c06d390 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:33.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:33.169+0000 7fbd905dc700 1 --2- 172.21.15.1:0/493005502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbd8815b500 0x7fbd880c64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:33.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:33.169+0000 7fbd905dc700 1 --2- 172.21.15.1:0/493005502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8806ae60 0x7fbd880c39c0 unknown :-1 s=CLOSED pgs=3102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:33.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:33.169+0000 7fbd905dc700 1 --2- 172.21.15.1:0/493005502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd8815b120 0x7fbd880c5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:33.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:33.169+0000 7fbd905dc700 1 -- 172.21.15.1:0/493005502 >> 172.21.15.1:0/493005502 conn(0x7fbd880ccab0 msgr2=0x7fbd8809e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:33.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:33.169+0000 7fbd905dc700 1 -- 172.21.15.1:0/493005502 shutdown_connections 2024-03-18T20:27:33.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:33.169+0000 7fbd905dc700 1 -- 172.21.15.1:0/493005502 wait complete. 2024-03-18T20:27:33.186 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T20:27:33.186 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T20:27:33.186 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-18T20:27:33.186 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T20:27:33.186 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T20:27:33.186 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T20:27:33.186 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T20:27:33.186 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T20:27:33.186 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:27:33.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:33.187 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-18T20:27:33.205 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T20:27:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T20:27:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T20:27:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:27:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T20:27:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:27:33.206 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:27:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-18T20:27:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-18T20:27:33.206 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:27:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-18T20:27:33.206 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-18T20:27:33.207 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T20:27:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:27:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-18T20:27:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T20:27:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 36716sec preferred_lft 36716sec 2024-03-18T20:27:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link 2024-03-18T20:27:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:27:33.207 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:27:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:27:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-18T20:27:33.207 INFO:teuthology.orchestra.run.smithi001.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T20:27:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:27:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T20:27:33.207 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:27:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::4ca9:f5ff:feb3:79a3/64 scope link 2024-03-18T20:27:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:27:33.208 INFO:teuthology.orchestra.run.smithi001.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T20:27:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T20:27:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::282f:a4ff:fe1a:9908/64 scope link 2024-03-18T20:27:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:27:33.208 INFO:teuthology.orchestra.run.smithi001.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T20:27:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2e:5c:60:78:b2:8c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T20:27:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::2c5c:60ff:fe78:b28c/64 scope link 2024-03-18T20:27:33.208 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:27:33.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:33.209 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T20:27:33.272 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-18T20:27:33.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:33.274 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-18T20:27:33.334 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-18T20:27:33.334 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T20:27:33.334 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T20:27:33.334 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T20:27:33.334 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T20:27:33.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:33.335 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T20:27:33.335 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T20:27:33.335 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T20:27:33.335 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T20:27:33.335 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T20:27:33.335 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T20:27:33.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:33.441 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T20:27:33.458 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T20:27:33.459 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T20:27:33.459 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:27:33.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:33.459 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-18T20:27:33.526 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T20:27:33.526 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:27:33.527 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T20:27:33.585 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:27:33.585 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:27:33.585 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T20:27:33.643 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T20:27:33.711 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:27:33.711 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:27:33.778 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:27:33.779 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:27:33.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:33.780 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T20:27:33.837 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T20:27:33.838 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T20:27:33.880 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T20:27:33.907 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:27:33.907 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:27:33.967 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-18T20:27:33.962+0000 7f4b981185c0 -1 init, newargv = 0x5585b6db5a90 newargc=15 2024-03-18T20:27:33.967 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[170560]: starting ceph client 2024-03-18T20:27:33.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[170560]: starting fuse 2024-03-18T20:27:33.975 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:27:33.977 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:27:33.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:33.977 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T20:27:34.036 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T20:27:34.036 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-03-18T20:27:34.036 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T20:27:34.036 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-18T20:27:34.036 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-18T20:27:34.036 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-18T20:27:34.036 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-18T20:27:34.037 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:27:34.037 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-18T20:27:34.037 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T20:27:34.037 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-18T20:27:34.037 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T20:27:34.037 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:27:34.037 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-18T20:27:34.037 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-18T20:27:34.037 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-18T20:27:34.037 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-18T20:27:34.037 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:27:34.037 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-18T20:27:34.037 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T20:27:34.038 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T20:27:34.038 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T20:27:34.038 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-18T20:27:34.038 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-18T20:27:34.038 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-18T20:27:34.038 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T20:27:34.038 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:27:34.038 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-18T20:27:34.038 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-18T20:27:34.160 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.170560.asok 2024-03-18T20:27:34.342 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.170560.asok 2024-03-18T20:27:34.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:34.343 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.170560.asok status 2024-03-18T20:27:34.504 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:27:34.504 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-18T20:27:34.504 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T20:27:34.504 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T20:27:34.504 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-18T20:27:34.504 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-18T20:27:34.504 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T20:27:34.504 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "170560", 2024-03-18T20:27:34.505 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-18T20:27:34.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:34.505 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-18T20:27:34.505 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-18T20:27:34.505 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 13371, 2024-03-18T20:27:34.505 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-18T20:27:34.505 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-18T20:27:34.505 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-18T20:27:34.505 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 13371 2024-03-18T20:27:34.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:34.505 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T20:27:34.505 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T20:27:34.505 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:27:34.506 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4192433683 2024-03-18T20:27:34.506 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:34.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:34.506 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T20:27:34.506 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T20:27:34.506 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:27:34.506 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4192433683 2024-03-18T20:27:34.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:34.506 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.13371 192.168.0.1:0/4192433683", 2024-03-18T20:27:34.506 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/4192433683", 2024-03-18T20:27:34.506 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-18T20:27:34.507 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 136, 2024-03-18T20:27:34.507 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 274, 2024-03-18T20:27:34.507 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-18T20:27:34.507 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-18T20:27:34.507 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-18T20:27:34.507 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:27:34.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:34.514 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-18T20:27:34.514 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T20:27:34.534 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-18T20:27:34.534 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T20:27:34.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:34.535 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T20:27:34.602 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-18T20:27:34.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.764+0000 7f47240aa700 1 Processor -- start 2024-03-18T20:27:34.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.765+0000 7f47240aa700 1 -- start start 2024-03-18T20:27:34.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.766+0000 7f47240aa700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f471c0aede0 0x7f471c159e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:34.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.766+0000 7f47240aa700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f471c0b3320 0x7f471c0b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:34.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.766+0000 7f47240aa700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471c0b3c90 con 0x7f471c06d950 2024-03-18T20:27:34.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.766+0000 7f47240aa700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f471c15e3c0 con 0x7f471c0aede0 2024-03-18T20:27:34.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.766+0000 7f47240aa700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471c0af1c0 con 0x7f471c0b3320 2024-03-18T20:27:34.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.766+0000 7f4721645700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f471c06d950 0x7f471c15dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50428/0 (socket says 172.21.15.1:50428) 2024-03-18T20:27:34.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.766+0000 7f4721645700 1 -- 172.21.15.1:0/3074291857 learned_addr learned my addr 172.21.15.1:0/3074291857 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:34.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.766+0000 7f4722647700 1 --2- 172.21.15.1:0/3074291857 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f471c0b3320 0x7f471c0b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:34.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.766+0000 7f4721e46700 1 --2- 172.21.15.1:0/3074291857 >> v2:172.21.15.165:3300/0 conn(0x7f471c0aede0 0x7f471c159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:34.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.766+0000 7f4720e44700 1 -- 172.21.15.1:0/3074291857 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17031751 0 0) 0x7f471c0b3c90 con 0x7f471c06d950 2024-03-18T20:27:34.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.766+0000 7f4720e44700 1 -- 172.21.15.1:0/3074291857 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47080035b0 con 0x7f471c06d950 2024-03-18T20:27:34.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.766+0000 7f4722647700 1 -- 172.21.15.1:0/3074291857 >> v2:172.21.15.165:3300/0 conn(0x7f471c0aede0 msgr2=0x7f471c159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:34.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.766+0000 7f4722647700 1 --2- 172.21.15.1:0/3074291857 >> v2:172.21.15.165:3300/0 conn(0x7f471c0aede0 0x7f471c159e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:34.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.766+0000 7f4722647700 1 -- 172.21.15.1:0/3074291857 >> v1:172.21.15.1:6789/0 conn(0x7f471c06d950 legacy=0x7f471c15dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:34.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.766+0000 7f4722647700 1 -- 172.21.15.1:0/3074291857 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c15d4d0 con 0x7f471c0b3320 2024-03-18T20:27:34.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.766+0000 7f4721e46700 1 --2- 172.21.15.1:0/3074291857 >> v2:172.21.15.165:3300/0 conn(0x7f471c0aede0 0x7f471c159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:27:34.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.767+0000 7f4722647700 1 --2- 172.21.15.1:0/3074291857 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f471c0b3320 0x7f471c0b3750 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f471400aea0 tx=0x7f471400c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=833b4916458bb483 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:34.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.767+0000 7f4720e44700 1 -- 172.21.15.1:0/3074291857 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471400f070 con 0x7f471c0b3320 2024-03-18T20:27:34.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.767+0000 7f4720e44700 1 -- 172.21.15.1:0/3074291857 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f471400a2e0 con 0x7f471c0b3320 2024-03-18T20:27:34.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.767+0000 7f4720e44700 1 -- 172.21.15.1:0/3074291857 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471400cb10 con 0x7f471c0b3320 2024-03-18T20:27:34.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.767+0000 7f47240aa700 1 -- 172.21.15.1:0/3074291857 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f471c0b3320 msgr2=0x7f471c0b3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:34.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.767+0000 7f47240aa700 1 --2- 172.21.15.1:0/3074291857 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f471c0b3320 0x7f471c0b3750 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f471400aea0 tx=0x7f471400c330 comp rx=0 tx=0).stop 2024-03-18T20:27:34.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.767+0000 7f47240aa700 1 -- 172.21.15.1:0/3074291857 shutdown_connections 2024-03-18T20:27:34.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.767+0000 7f47240aa700 1 --2- 172.21.15.1:0/3074291857 >> v2:172.21.15.165:3300/0 conn(0x7f471c0aede0 0x7f471c159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:34.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.767+0000 7f47240aa700 1 --2- 172.21.15.1:0/3074291857 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f471c0b3320 0x7f471c0b3750 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:34.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.767+0000 7f47240aa700 1 -- 172.21.15.1:0/3074291857 >> 172.21.15.1:0/3074291857 conn(0x7f471c066b70 msgr2=0x7f471c06a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:34.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.767+0000 7f47240aa700 1 -- 172.21.15.1:0/3074291857 shutdown_connections 2024-03-18T20:27:34.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.767+0000 7f47240aa700 1 -- 172.21.15.1:0/3074291857 wait complete. 2024-03-18T20:27:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.769+0000 7f47240aa700 1 Processor -- start 2024-03-18T20:27:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.769+0000 7f47240aa700 1 -- start start 2024-03-18T20:27:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.769+0000 7f47240aa700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f471c06d950 0x7f471c158390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.769+0000 7f47240aa700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f471c0aede0 0x7f471c154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.769+0000 7f47240aa700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f471c0b3320 0x7f471c148d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.769+0000 7f47240aa700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c1555f0 con 0x7f471c06d950 2024-03-18T20:27:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.769+0000 7f47240aa700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c1558d0 con 0x7f471c0aede0 2024-03-18T20:27:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.769+0000 7f47240aa700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471c155bb0 con 0x7f471c0b3320 2024-03-18T20:27:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.769+0000 7f4721e46700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f471c06d950 0x7f471c158390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.769+0000 7f4721e46700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f471c06d950 0x7f471c158390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60104/0 (socket says 172.21.15.1:60104) 2024-03-18T20:27:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.769+0000 7f4721e46700 1 -- 172.21.15.1:0/2704563977 learned_addr learned my addr 172.21.15.1:0/2704563977 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.770+0000 7f4721645700 1 --2- 172.21.15.1:0/2704563977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f471c0aede0 0x7f471c154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.770+0000 7f4722647700 1 --2- 172.21.15.1:0/2704563977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f471c0b3320 0x7f471c148d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.770+0000 7f4721e46700 1 -- 172.21.15.1:0/2704563977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f471c0b3320 msgr2=0x7f471c148d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.770+0000 7f4721e46700 1 --2- 172.21.15.1:0/2704563977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f471c0b3320 0x7f471c148d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.770+0000 7f4721e46700 1 -- 172.21.15.1:0/2704563977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f471c0aede0 msgr2=0x7f471c154c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.770+0000 7f4721e46700 1 --2- 172.21.15.1:0/2704563977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f471c0aede0 0x7f471c154c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.770+0000 7f4721e46700 1 -- 172.21.15.1:0/2704563977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470c0094a0 con 0x7f471c06d950 2024-03-18T20:27:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.770+0000 7f4721e46700 1 --2- 172.21.15.1:0/2704563977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f471c06d950 0x7f471c158390 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7f470c00ebe0 tx=0x7f470c009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:34.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.770+0000 7f47127fc700 1 -- 172.21.15.1:0/2704563977 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c00c8b0 con 0x7f471c06d950 2024-03-18T20:27:34.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.770+0000 7f47127fc700 1 -- 172.21.15.1:0/2704563977 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470c00ca10 con 0x7f471c06d950 2024-03-18T20:27:34.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.770+0000 7f47240aa700 1 -- 172.21.15.1:0/2704563977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f471400ab80 con 0x7f471c06d950 2024-03-18T20:27:34.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.770+0000 7f47127fc700 1 -- 172.21.15.1:0/2704563977 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c00cda0 con 0x7f471c06d950 2024-03-18T20:27:34.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.770+0000 7f47240aa700 1 -- 172.21.15.1:0/2704563977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471c156170 con 0x7f471c06d950 2024-03-18T20:27:34.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.771+0000 7f47240aa700 1 -- 172.21.15.1:0/2704563977 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471c156390 con 0x7f471c06d950 2024-03-18T20:27:34.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.772+0000 7f47127fc700 1 -- 172.21.15.1:0/2704563977 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f470c013470 con 0x7f471c06d950 2024-03-18T20:27:34.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.772+0000 7f47127fc700 1 --2- 172.21.15.1:0/2704563977 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f470006aed0 0x7f470006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:34.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.772+0000 7f47127fc700 1 -- 172.21.15.1:0/2704563977 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f470c0899b0 con 0x7f471c06d950 2024-03-18T20:27:34.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.772+0000 7f4721645700 1 --2- 172.21.15.1:0/2704563977 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f470006aed0 0x7f470006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:34.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.772+0000 7f4721645700 1 --2- 172.21.15.1:0/2704563977 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f470006aed0 0x7f470006d390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f471c068090 tx=0x7f4718009040 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:34.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:34.777+0000 7f47127fc700 1 -- 172.21.15.1:0/2704563977 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f470c054710 con 0x7f471c06d950 2024-03-18T20:27:35.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.007+0000 7f47240aa700 1 -- 172.21.15.1:0/2704563977 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f471c15acf0 con 0x7f470006aed0 2024-03-18T20:27:35.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.009+0000 7f47127fc700 1 -- 172.21.15.1:0/2704563977 <== mgr.12813 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f4718005960 con 0x7f470006aed0 2024-03-18T20:27:35.013 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T20:27:35.013 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:27:35.013 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephfs" 2024-03-18T20:27:35.013 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:35.013 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T20:27:35.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.010+0000 7f47240aa700 1 -- 172.21.15.1:0/2704563977 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f470006aed0 msgr2=0x7f470006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:35.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.010+0000 7f47240aa700 1 --2- 172.21.15.1:0/2704563977 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f470006aed0 0x7f470006d390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f471c068090 tx=0x7f4718009040 comp rx=0 tx=0).stop 2024-03-18T20:27:35.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.010+0000 7f47240aa700 1 -- 172.21.15.1:0/2704563977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f471c06d950 msgr2=0x7f471c158390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:35.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.010+0000 7f47240aa700 1 --2- 172.21.15.1:0/2704563977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f471c06d950 0x7f471c158390 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7f470c00ebe0 tx=0x7f470c009e20 comp rx=0 tx=0).stop 2024-03-18T20:27:35.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.011+0000 7f47240aa700 1 -- 172.21.15.1:0/2704563977 shutdown_connections 2024-03-18T20:27:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.011+0000 7f47240aa700 1 --2- 172.21.15.1:0/2704563977 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f470006aed0 0x7f470006d390 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.011+0000 7f47240aa700 1 --2- 172.21.15.1:0/2704563977 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f471c0b3320 0x7f471c148d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.011+0000 7f47240aa700 1 --2- 172.21.15.1:0/2704563977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f471c06d950 0x7f471c158390 unknown :-1 s=CLOSED pgs=3107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.011+0000 7f47240aa700 1 --2- 172.21.15.1:0/2704563977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f471c0aede0 0x7f471c154c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.011+0000 7f47240aa700 1 -- 172.21.15.1:0/2704563977 >> 172.21.15.1:0/2704563977 conn(0x7f471c066b70 msgr2=0x7f471c06a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.011+0000 7f47240aa700 1 -- 172.21.15.1:0/2704563977 shutdown_connections 2024-03-18T20:27:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.011+0000 7f47240aa700 1 -- 172.21.15.1:0/2704563977 wait complete. 2024-03-18T20:27:35.026 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-18T20:27:35.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.149+0000 7f7d7fff2700 1 Processor -- start 2024-03-18T20:27:35.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.149+0000 7f7d7fff2700 1 -- start start 2024-03-18T20:27:35.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.150+0000 7f7d7fff2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d7806eda0 0x7f7d78161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.150+0000 7f7d7fff2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d7806f180 0x7f7d780ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.150+0000 7f7d7fff2700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d780caac0 con 0x7f7d7806a750 2024-03-18T20:27:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.150+0000 7f7d7fff2700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d780cb320 con 0x7f7d7806eda0 2024-03-18T20:27:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.150+0000 7f7d7fff2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d780ce010 con 0x7f7d7806f180 2024-03-18T20:27:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.150+0000 7f7d7dd8e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d7806eda0 0x7f7d78161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.150+0000 7f7d7dd8e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d7806eda0 0x7f7d78161c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60106/0 (socket says 172.21.15.1:60106) 2024-03-18T20:27:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.150+0000 7f7d7e58f700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7d7806a750 0x7f7d780715b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50432/0 (socket says 172.21.15.1:50432) 2024-03-18T20:27:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.150+0000 7f7d7dd8e700 1 -- 172.21.15.1:0/2949797601 learned_addr learned my addr 172.21.15.1:0/2949797601 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.150+0000 7f7d7d58d700 1 --2- 172.21.15.1:0/2949797601 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d7806f180 0x7f7d780ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.150+0000 7f7d7dd8e700 1 -- 172.21.15.1:0/2949797601 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d7806f180 msgr2=0x7f7d780ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.150+0000 7f7d7dd8e700 1 --2- 172.21.15.1:0/2949797601 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d7806f180 0x7f7d780ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.150+0000 7f7d7dd8e700 1 -- 172.21.15.1:0/2949797601 >> v1:172.21.15.1:6789/0 conn(0x7f7d7806a750 legacy=0x7f7d780715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.150+0000 7f7d7cd8c700 1 -- 172.21.15.1:0/2949797601 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78076969 0 0) 0x7f7d780caac0 con 0x7f7d7806a750 2024-03-18T20:27:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.150+0000 7f7d7dd8e700 1 -- 172.21.15.1:0/2949797601 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d780cad60 con 0x7f7d7806eda0 2024-03-18T20:27:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.151+0000 7f7d7dd8e700 1 --2- 172.21.15.1:0/2949797601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d7806eda0 0x7f7d78161c60 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 crypto rx=0x7f7d680098f0 tx=0x7f7d68009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f7ec6a1fa4cbff9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.151+0000 7f7d7cd8c700 1 -- 172.21.15.1:0/2949797601 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d68011830 con 0x7f7d7806eda0 2024-03-18T20:27:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.151+0000 7f7d7cd8c700 1 -- 172.21.15.1:0/2949797601 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d68011990 con 0x7f7d7806eda0 2024-03-18T20:27:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.151+0000 7f7d7cd8c700 1 -- 172.21.15.1:0/2949797601 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d68011d20 con 0x7f7d7806eda0 2024-03-18T20:27:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.151+0000 7f7d7fff2700 1 -- 172.21.15.1:0/2949797601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d7806eda0 msgr2=0x7f7d78161c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.151+0000 7f7d7fff2700 1 --2- 172.21.15.1:0/2949797601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d7806eda0 0x7f7d78161c60 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 crypto rx=0x7f7d680098f0 tx=0x7f7d68009c00 comp rx=0 tx=0).stop 2024-03-18T20:27:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.151+0000 7f7d7fff2700 1 -- 172.21.15.1:0/2949797601 shutdown_connections 2024-03-18T20:27:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.151+0000 7f7d7fff2700 1 --2- 172.21.15.1:0/2949797601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d7806eda0 0x7f7d78161c60 unknown :-1 s=CLOSED pgs=3109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.151+0000 7f7d7fff2700 1 --2- 172.21.15.1:0/2949797601 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d7806f180 0x7f7d780ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.151+0000 7f7d7fff2700 1 -- 172.21.15.1:0/2949797601 >> 172.21.15.1:0/2949797601 conn(0x7f7d780cff30 msgr2=0x7f7d780d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.151+0000 7f7d7fff2700 1 -- 172.21.15.1:0/2949797601 shutdown_connections 2024-03-18T20:27:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.151+0000 7f7d7fff2700 1 -- 172.21.15.1:0/2949797601 wait complete. 2024-03-18T20:27:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.153+0000 7f7d7fff2700 1 Processor -- start 2024-03-18T20:27:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.153+0000 7f7d7fff2700 1 -- start start 2024-03-18T20:27:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.153+0000 7f7d7fff2700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d7806a750 0x7f7d780af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.153+0000 7f7d7fff2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d7806eda0 0x7f7d780afe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.153+0000 7f7d7fff2700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d7806f180 0x7f7d781f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.153+0000 7f7d7fff2700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d781f73f0 con 0x7f7d7806eda0 2024-03-18T20:27:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.153+0000 7f7d7fff2700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d781f3e70 con 0x7f7d7806a750 2024-03-18T20:27:35.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.153+0000 7f7d7fff2700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d781f6630 con 0x7f7d7806f180 2024-03-18T20:27:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.153+0000 7f7d7d58d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d7806eda0 0x7f7d780afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.153+0000 7f7d7d58d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d7806eda0 0x7f7d780afe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60110/0 (socket says 172.21.15.1:60110) 2024-03-18T20:27:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.153+0000 7f7d7d58d700 1 -- 172.21.15.1:0/2962799942 learned_addr learned my addr 172.21.15.1:0/2962799942 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.154+0000 7f7d7e58f700 1 --2- 172.21.15.1:0/2962799942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d7806f180 0x7f7d781f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.154+0000 7f7d7dd8e700 1 --2- 172.21.15.1:0/2962799942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d7806a750 0x7f7d780af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.154+0000 7f7d7d58d700 1 -- 172.21.15.1:0/2962799942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d7806f180 msgr2=0x7f7d781f3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.154+0000 7f7d7d58d700 1 --2- 172.21.15.1:0/2962799942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d7806f180 0x7f7d781f3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.154+0000 7f7d7d58d700 1 -- 172.21.15.1:0/2962799942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d7806a750 msgr2=0x7f7d780af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.154+0000 7f7d7d58d700 1 --2- 172.21.15.1:0/2962799942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d7806a750 0x7f7d780af940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.154+0000 7f7d7d58d700 1 -- 172.21.15.1:0/2962799942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d68009550 con 0x7f7d7806eda0 2024-03-18T20:27:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.154+0000 7f7d7dd8e700 1 --2- 172.21.15.1:0/2962799942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d7806a750 0x7f7d780af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.154+0000 7f7d7e58f700 1 --2- 172.21.15.1:0/2962799942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d7806f180 0x7f7d781f3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.154+0000 7f7d7d58d700 1 --2- 172.21.15.1:0/2962799942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d7806eda0 0x7f7d780afe80 secure :-1 s=READY pgs=3110 cs=0 l=1 rev1=1 crypto rx=0x7f7d7400e870 tx=0x7f7d7400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.154+0000 7f7d6e7fc700 1 -- 172.21.15.1:0/2962799942 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d74009590 con 0x7f7d7806eda0 2024-03-18T20:27:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.154+0000 7f7d6e7fc700 1 -- 172.21.15.1:0/2962799942 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7400ee70 con 0x7f7d7806eda0 2024-03-18T20:27:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.154+0000 7f7d7fff2700 1 -- 172.21.15.1:0/2962799942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d781f6910 con 0x7f7d7806eda0 2024-03-18T20:27:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.154+0000 7f7d6e7fc700 1 -- 172.21.15.1:0/2962799942 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7400ca50 con 0x7f7d7806eda0 2024-03-18T20:27:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.154+0000 7f7d7fff2700 1 -- 172.21.15.1:0/2962799942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d781f6ed0 con 0x7f7d7806eda0 2024-03-18T20:27:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.155+0000 7f7d7fff2700 1 -- 172.21.15.1:0/2962799942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d780cba30 con 0x7f7d7806eda0 2024-03-18T20:27:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.155+0000 7f7d6e7fc700 1 -- 172.21.15.1:0/2962799942 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 156) v1 ==== 89341+0+0 (secure 0 0 0) 0x7f7d7400cbb0 con 0x7f7d7806eda0 2024-03-18T20:27:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.156+0000 7f7d6e7fc700 1 --2- 172.21.15.1:0/2962799942 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7d5c06aed0 0x7f7d5c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.156+0000 7f7d6e7fc700 1 -- 172.21.15.1:0/2962799942 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f7d74089820 con 0x7f7d7806eda0 2024-03-18T20:27:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.156+0000 7f7d7dd8e700 1 --2- 172.21.15.1:0/2962799942 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7d5c06aed0 0x7f7d5c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.156+0000 7f7d7dd8e700 1 --2- 172.21.15.1:0/2962799942 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7d5c06aed0 0x7f7d5c06d390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7d780b5d30 tx=0x7f7d68008030 comp rx=0 tx=0).ready entity=mgr.12813 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:35.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.160+0000 7f7d6e7fc700 1 -- 172.21.15.1:0/2962799942 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7d74054580 con 0x7f7d7806eda0 2024-03-18T20:27:35.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:35.423+0000 7f7d7fff2700 1 -- 172.21.15.1:0/2962799942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f7d780676a0 con 0x7f7d7806eda0 2024-03-18T20:27:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.386+0000 7f7d6e7fc700 1 -- 172.21.15.1:0/2962799942 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 157) v1 ==== 89358+0+0 (secure 0 0 0) 0x7f7d74054950 con 0x7f7d7806eda0 2024-03-18T20:27:36.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.387+0000 7f7d6e7fc700 1 -- 172.21.15.1:0/2962799942 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v157) v1 ==== 92+0+0 (secure 0 0 0) 0x7f7d74054580 con 0x7f7d7806eda0 2024-03-18T20:27:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.389+0000 7f7d7fff2700 1 -- 172.21.15.1:0/2962799942 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7d5c06aed0 msgr2=0x7f7d5c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.389+0000 7f7d7fff2700 1 --2- 172.21.15.1:0/2962799942 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7d5c06aed0 0x7f7d5c06d390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7d780b5d30 tx=0x7f7d68008030 comp rx=0 tx=0).stop 2024-03-18T20:27:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.389+0000 7f7d7fff2700 1 -- 172.21.15.1:0/2962799942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d7806eda0 msgr2=0x7f7d780afe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.389+0000 7f7d7fff2700 1 --2- 172.21.15.1:0/2962799942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d7806eda0 0x7f7d780afe80 secure :-1 s=READY pgs=3110 cs=0 l=1 rev1=1 crypto rx=0x7f7d7400e870 tx=0x7f7d7400ec30 comp rx=0 tx=0).stop 2024-03-18T20:27:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.390+0000 7f7d7fff2700 1 -- 172.21.15.1:0/2962799942 shutdown_connections 2024-03-18T20:27:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.390+0000 7f7d7fff2700 1 --2- 172.21.15.1:0/2962799942 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7d5c06aed0 0x7f7d5c06d390 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.390+0000 7f7d7fff2700 1 --2- 172.21.15.1:0/2962799942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7d7806f180 0x7f7d781f3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.390+0000 7f7d7fff2700 1 --2- 172.21.15.1:0/2962799942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d7806a750 0x7f7d780af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.390+0000 7f7d7fff2700 1 --2- 172.21.15.1:0/2962799942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d7806eda0 0x7f7d780afe80 unknown :-1 s=CLOSED pgs=3110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:36.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.390+0000 7f7d7fff2700 1 -- 172.21.15.1:0/2962799942 >> 172.21.15.1:0/2962799942 conn(0x7f7d780cff30 msgr2=0x7f7d780b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.390+0000 7f7d7fff2700 1 -- 172.21.15.1:0/2962799942 shutdown_connections 2024-03-18T20:27:36.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.390+0000 7f7d7fff2700 1 -- 172.21.15.1:0/2962799942 wait complete. 2024-03-18T20:27:36.407 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-18T20:27:36.531 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:36.526+0000 7fb8bdc7b240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:27:36.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.537+0000 7fb2a7bf5700 1 Processor -- start 2024-03-18T20:27:36.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.538+0000 7fb2a7bf5700 1 -- start start 2024-03-18T20:27:36.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.539+0000 7fb2a7bf5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2a00cd430 0x7fb2a00b2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:36.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.539+0000 7fb2a7bf5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2a00c87d0 0x7fb2a00cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:36.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.539+0000 7fb2a7bf5700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a00d01f0 con 0x7fb2a015b6c0 2024-03-18T20:27:36.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.539+0000 7fb2a7bf5700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2a00ccfa0 con 0x7fb2a00c87d0 2024-03-18T20:27:36.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.539+0000 7fb2a7bf5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a00d1db0 con 0x7fb2a00cd430 2024-03-18T20:27:36.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.539+0000 7fb2a6192700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2a00c87d0 0x7fb2a00cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:36.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.539+0000 7fb2a6192700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2a00c87d0 0x7fb2a00cd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60126/0 (socket says 172.21.15.1:60126) 2024-03-18T20:27:36.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.539+0000 7fb2a6192700 1 -- 172.21.15.1:0/1071581423 learned_addr learned my addr 172.21.15.1:0/1071581423 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:36.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.539+0000 7fb2a5190700 1 --2- 172.21.15.1:0/1071581423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2a00cd430 0x7fb2a00b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:36.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.539+0000 7fb2a6192700 1 -- 172.21.15.1:0/1071581423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2a00cd430 msgr2=0x7fb2a00b2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:36.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.539+0000 7fb2a6192700 1 --2- 172.21.15.1:0/1071581423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2a00cd430 0x7fb2a00b2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:36.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.539+0000 7fb2a6192700 1 -- 172.21.15.1:0/1071581423 >> v1:172.21.15.165:6789/0 conn(0x7fb2a015b6c0 legacy=0x7fb2a00c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:36.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.539+0000 7fb2a6192700 1 -- 172.21.15.1:0/1071581423 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a00cff10 con 0x7fb2a00c87d0 2024-03-18T20:27:36.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.539+0000 7fb2a498f700 1 -- 172.21.15.1:0/1071581423 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427329874 0 0) 0x7fb2a00d01f0 con 0x7fb2a015b6c0 2024-03-18T20:27:36.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.539+0000 7fb2a6192700 1 --2- 172.21.15.1:0/1071581423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2a00c87d0 0x7fb2a00cd9e0 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 crypto rx=0x7fb298009e80 tx=0x7fb29800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c7b77b469f2568b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:36.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.539+0000 7fb2a498f700 1 -- 172.21.15.1:0/1071581423 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29800e070 con 0x7fb2a00c87d0 2024-03-18T20:27:36.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.539+0000 7fb2a498f700 1 -- 172.21.15.1:0/1071581423 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb29800b8d0 con 0x7fb2a00c87d0 2024-03-18T20:27:36.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.539+0000 7fb2a498f700 1 -- 172.21.15.1:0/1071581423 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29800bc80 con 0x7fb2a00c87d0 2024-03-18T20:27:36.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.540+0000 7fb2a7bf5700 1 -- 172.21.15.1:0/1071581423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2a00c87d0 msgr2=0x7fb2a00cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:36.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.540+0000 7fb2a7bf5700 1 --2- 172.21.15.1:0/1071581423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2a00c87d0 0x7fb2a00cd9e0 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 crypto rx=0x7fb298009e80 tx=0x7fb29800b330 comp rx=0 tx=0).stop 2024-03-18T20:27:36.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.540+0000 7fb2a7bf5700 1 -- 172.21.15.1:0/1071581423 shutdown_connections 2024-03-18T20:27:36.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.540+0000 7fb2a7bf5700 1 --2- 172.21.15.1:0/1071581423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2a00c87d0 0x7fb2a00cd9e0 unknown :-1 s=CLOSED pgs=3115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:36.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.540+0000 7fb2a7bf5700 1 --2- 172.21.15.1:0/1071581423 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2a00cd430 0x7fb2a00b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:36.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.540+0000 7fb2a7bf5700 1 -- 172.21.15.1:0/1071581423 >> 172.21.15.1:0/1071581423 conn(0x7fb2a00d0ff0 msgr2=0x7fb2a00d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:36.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.540+0000 7fb2a7bf5700 1 -- 172.21.15.1:0/1071581423 shutdown_connections 2024-03-18T20:27:36.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.540+0000 7fb2a7bf5700 1 -- 172.21.15.1:0/1071581423 wait complete. 2024-03-18T20:27:36.546 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:36.541+0000 7fe555cbe240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:27:36.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.542+0000 7fb2a7bf5700 1 Processor -- start 2024-03-18T20:27:36.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.542+0000 7fb2a7bf5700 1 -- start start 2024-03-18T20:27:36.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.543+0000 7fb2a7bf5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2a00c87d0 0x7fb2a00b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:36.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.543+0000 7fb2a7bf5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2a00cd430 0x7fb2a0159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:36.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.543+0000 7fb2a7bf5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2a015b6c0 0x7fb2a009e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:36.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.543+0000 7fb2a7bf5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a009ed30 con 0x7fb2a00cd430 2024-03-18T20:27:36.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.543+0000 7fb2a7bf5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a009ae60 con 0x7fb2a00c87d0 2024-03-18T20:27:36.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.543+0000 7fb2a7bf5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a009b140 con 0x7fb2a015b6c0 2024-03-18T20:27:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.543+0000 7fb2a5190700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2a00cd430 0x7fb2a0159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.543+0000 7fb2a5190700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2a00cd430 0x7fb2a0159ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60130/0 (socket says 172.21.15.1:60130) 2024-03-18T20:27:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.543+0000 7fb2a5190700 1 -- 172.21.15.1:0/155625235 learned_addr learned my addr 172.21.15.1:0/155625235 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.543+0000 7fb2a5991700 1 --2- 172.21.15.1:0/155625235 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2a00c87d0 0x7fb2a00b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.543+0000 7fb2a6192700 1 --2- 172.21.15.1:0/155625235 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2a015b6c0 0x7fb2a009e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.543+0000 7fb2a5190700 1 -- 172.21.15.1:0/155625235 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2a015b6c0 msgr2=0x7fb2a009e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.543+0000 7fb2a5190700 1 --2- 172.21.15.1:0/155625235 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2a015b6c0 0x7fb2a009e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.543+0000 7fb2a5190700 1 -- 172.21.15.1:0/155625235 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2a00c87d0 msgr2=0x7fb2a00b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.543+0000 7fb2a5190700 1 --2- 172.21.15.1:0/155625235 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2a00c87d0 0x7fb2a00b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.543+0000 7fb2a5190700 1 -- 172.21.15.1:0/155625235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb298009b30 con 0x7fb2a00cd430 2024-03-18T20:27:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.543+0000 7fb2a5190700 1 --2- 172.21.15.1:0/155625235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2a00cd430 0x7fb2a0159ef0 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 crypto rx=0x7fb29000e870 tx=0x7fb29000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:36.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.544+0000 7fb2967fc700 1 -- 172.21.15.1:0/155625235 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb290009590 con 0x7fb2a00cd430 2024-03-18T20:27:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.544+0000 7fb2967fc700 1 -- 172.21.15.1:0/155625235 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb290013680 con 0x7fb2a00cd430 2024-03-18T20:27:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.544+0000 7fb2a7bf5700 1 -- 172.21.15.1:0/155625235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a009b420 con 0x7fb2a00cd430 2024-03-18T20:27:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.544+0000 7fb2967fc700 1 -- 172.21.15.1:0/155625235 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2900139f0 con 0x7fb2a00cd430 2024-03-18T20:27:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.544+0000 7fb2a7bf5700 1 -- 172.21.15.1:0/155625235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a00b3000 con 0x7fb2a00cd430 2024-03-18T20:27:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.544+0000 7fb2a7bf5700 1 -- 172.21.15.1:0/155625235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a009b940 con 0x7fb2a00cd430 2024-03-18T20:27:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.545+0000 7fb2967fc700 1 -- 172.21.15.1:0/155625235 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 157) v1 ==== 89358+0+0 (secure 0 0 0) 0x7fb290013b50 con 0x7fb2a00cd430 2024-03-18T20:27:36.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.546+0000 7fb2967fc700 1 --2- 172.21.15.1:0/155625235 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb28406af20 0x7fb28406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:36.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.546+0000 7fb2a5991700 1 -- 172.21.15.1:0/155625235 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb28406af20 msgr2=0x7fb28406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:36.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.546+0000 7fb2967fc700 1 -- 172.21.15.1:0/155625235 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fb290089720 con 0x7fb2a00cd430 2024-03-18T20:27:36.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.546+0000 7fb2a5991700 1 --2- 172.21.15.1:0/155625235 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb28406af20 0x7fb28406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:27:36.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.554+0000 7fb2967fc700 1 -- 172.21.15.1:0/155625235 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb2a009b940 con 0x7fb2a00cd430 2024-03-18T20:27:36.678 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:36.674+0000 7fb8bdc7b240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:27:36.716 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:36.713+0000 7fe555cbe240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:27:36.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.747+0000 7fb2a5991700 1 -- 172.21.15.1:0/155625235 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb28406af20 msgr2=0x7fb28406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:36.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.747+0000 7fb2a5991700 1 --2- 172.21.15.1:0/155625235 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb28406af20 0x7fb28406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:27:36.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.804+0000 7fb2a7bf5700 1 -- 172.21.15.1:0/155625235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fb2a009b940 con 0x7fb2a00cd430 2024-03-18T20:27:36.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.808+0000 7fb2967fc700 1 -- 172.21.15.1:0/155625235 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v89)=0 v89) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb2a009b940 con 0x7fb2a00cd430 2024-03-18T20:27:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.811+0000 7fb2a7bf5700 1 -- 172.21.15.1:0/155625235 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb28406af20 msgr2=0x7fb28406d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:27:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.811+0000 7fb2a7bf5700 1 --2- 172.21.15.1:0/155625235 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb28406af20 0x7fb28406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.811+0000 7fb2a7bf5700 1 -- 172.21.15.1:0/155625235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2a00cd430 msgr2=0x7fb2a0159ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.811+0000 7fb2a7bf5700 1 --2- 172.21.15.1:0/155625235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2a00cd430 0x7fb2a0159ef0 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 crypto rx=0x7fb29000e870 tx=0x7fb29000eb80 comp rx=0 tx=0).stop 2024-03-18T20:27:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.811+0000 7fb2a7bf5700 1 -- 172.21.15.1:0/155625235 shutdown_connections 2024-03-18T20:27:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.811+0000 7fb2a7bf5700 1 --2- 172.21.15.1:0/155625235 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb28406af20 0x7fb28406d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.811+0000 7fb2a7bf5700 1 --2- 172.21.15.1:0/155625235 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb2a015b6c0 0x7fb2a009e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.811+0000 7fb2a7bf5700 1 --2- 172.21.15.1:0/155625235 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb2a00c87d0 0x7fb2a00b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:36.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.811+0000 7fb2a7bf5700 1 --2- 172.21.15.1:0/155625235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2a00cd430 0x7fb2a0159ef0 unknown :-1 s=CLOSED pgs=3116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:36.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.811+0000 7fb2a7bf5700 1 -- 172.21.15.1:0/155625235 >> 172.21.15.1:0/155625235 conn(0x7fb2a00d0ff0 msgr2=0x7fb2a015de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:36.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.811+0000 7fb2a7bf5700 1 -- 172.21.15.1:0/155625235 shutdown_connections 2024-03-18T20:27:36.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.811+0000 7fb2a7bf5700 1 -- 172.21.15.1:0/155625235 wait complete. 2024-03-18T20:27:36.828 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-18T20:27:36.945 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:36.941+0000 7fb8bdc7b240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:27:36.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.953+0000 7f51c702e700 1 Processor -- start 2024-03-18T20:27:36.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.953+0000 7f51c702e700 1 -- start start 2024-03-18T20:27:36.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.954+0000 7f51c702e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f51c00cff70 0x7f51c00cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:36.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.954+0000 7f51c702e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c00d0df0 con 0x7f51c00b1e20 2024-03-18T20:27:36.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.954+0000 7f51c702e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c00d0550 con 0x7f51c00cd870 2024-03-18T20:27:36.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.954+0000 7f51c702e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51c00d0830 con 0x7f51c00cff70 2024-03-18T20:27:36.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.954+0000 7f51bffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f51c00b1e20 0x7f51c00cbce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50454/0 (socket says 172.21.15.1:50454) 2024-03-18T20:27:36.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.954+0000 7f51bffff700 1 -- 172.21.15.1:0/1275334877 learned_addr learned my addr 172.21.15.1:0/1275334877 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:36.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.954+0000 7f51c55cb700 1 --2- 172.21.15.1:0/1275334877 >> v2:172.21.15.165:3300/0 conn(0x7f51c00cff70 0x7f51c00cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:36.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.954+0000 7f51bf7fe700 1 -- 172.21.15.1:0/1275334877 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157848355 0 0) 0x7f51c00d0df0 con 0x7f51c00b1e20 2024-03-18T20:27:36.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.954+0000 7f51bf7fe700 1 -- 172.21.15.1:0/1275334877 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51b00035b0 con 0x7f51c00b1e20 2024-03-18T20:27:36.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.954+0000 7f51bf7fe700 1 -- 172.21.15.1:0/1275334877 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 603111413 0 0) 0x7f51b00035b0 con 0x7f51c00b1e20 2024-03-18T20:27:36.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.954+0000 7f51c55cb700 1 -- 172.21.15.1:0/1275334877 >> v1:172.21.15.165:6789/0 conn(0x7f51c00cd870 legacy=0x7f51c00cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:36.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.955+0000 7f51c55cb700 1 -- 172.21.15.1:0/1275334877 >> v1:172.21.15.1:6789/0 conn(0x7f51c00b1e20 legacy=0x7f51c00cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:36.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.955+0000 7f51c55cb700 1 -- 172.21.15.1:0/1275334877 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c00d0b10 con 0x7f51c00cff70 2024-03-18T20:27:36.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.955+0000 7f51c55cb700 1 --2- 172.21.15.1:0/1275334877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51c00cff70 0x7f51c00cc3f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f51b8009d90 tx=0x7f51b800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bf18d99cf4a31d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:36.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.955+0000 7f51bf7fe700 1 -- 172.21.15.1:0/1275334877 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b800e070 con 0x7f51c00cff70 2024-03-18T20:27:36.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.955+0000 7f51bf7fe700 1 -- 172.21.15.1:0/1275334877 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b8003a20 con 0x7f51c00cff70 2024-03-18T20:27:36.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.955+0000 7f51bf7fe700 1 -- 172.21.15.1:0/1275334877 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ac001670 con 0x7f51c00cff70 2024-03-18T20:27:36.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.955+0000 7f51c702e700 1 -- 172.21.15.1:0/1275334877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51c00cff70 msgr2=0x7f51c00cc3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:36.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.955+0000 7f51c702e700 1 --2- 172.21.15.1:0/1275334877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51c00cff70 0x7f51c00cc3f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f51b8009d90 tx=0x7f51b800b330 comp rx=0 tx=0).stop 2024-03-18T20:27:36.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.956+0000 7f51c702e700 1 -- 172.21.15.1:0/1275334877 shutdown_connections 2024-03-18T20:27:36.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.956+0000 7f51c702e700 1 --2- 172.21.15.1:0/1275334877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51c00cff70 0x7f51c00cc3f0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:36.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.956+0000 7f51c702e700 1 -- 172.21.15.1:0/1275334877 >> 172.21.15.1:0/1275334877 conn(0x7f51c00d0ff0 msgr2=0x7f51c00d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:36.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.956+0000 7f51c702e700 1 -- 172.21.15.1:0/1275334877 shutdown_connections 2024-03-18T20:27:36.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.956+0000 7f51c702e700 1 -- 172.21.15.1:0/1275334877 wait complete. 2024-03-18T20:27:36.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.957+0000 7f51c702e700 1 Processor -- start 2024-03-18T20:27:36.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.957+0000 7f51c702e700 1 -- start start 2024-03-18T20:27:36.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51c702e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c00b1e20 0x7f51c00ca930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:36.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51c702e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51c00cd870 0x7f51c0153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:36.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51c702e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51c00cff70 0x7f51c0154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:36.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51c702e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c009ec60 con 0x7f51c00b1e20 2024-03-18T20:27:36.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51c702e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c009e6a0 con 0x7f51c00cff70 2024-03-18T20:27:36.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51c702e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c009e980 con 0x7f51c00cd870 2024-03-18T20:27:36.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51c4dca700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c00b1e20 0x7f51c00ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:36.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51c4dca700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c00b1e20 0x7f51c00ca930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60146/0 (socket says 172.21.15.1:60146) 2024-03-18T20:27:36.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51c4dca700 1 -- 172.21.15.1:0/136737942 learned_addr learned my addr 172.21.15.1:0/136737942 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:36.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51c55cb700 1 --2- 172.21.15.1:0/136737942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51c00cff70 0x7f51c0154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:36.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51bffff700 1 --2- 172.21.15.1:0/136737942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51c00cd870 0x7f51c0153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:36.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51c4dca700 1 -- 172.21.15.1:0/136737942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51c00cd870 msgr2=0x7f51c0153b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:36.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51c4dca700 1 --2- 172.21.15.1:0/136737942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51c00cd870 0x7f51c0153b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:36.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51c4dca700 1 -- 172.21.15.1:0/136737942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51c00cff70 msgr2=0x7f51c0154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:36.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51c4dca700 1 --2- 172.21.15.1:0/136737942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51c00cff70 0x7f51c0154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:36.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51c4dca700 1 -- 172.21.15.1:0/136737942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b8009a70 con 0x7f51c00b1e20 2024-03-18T20:27:36.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51bffff700 1 --2- 172.21.15.1:0/136737942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51c00cd870 0x7f51c0153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:36.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51c4dca700 1 --2- 172.21.15.1:0/136737942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c00b1e20 0x7f51c00ca930 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 crypto rx=0x7f51b400c780 tx=0x7f51b400ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:36.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51bd7fa700 1 -- 172.21.15.1:0/136737942 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b40054e0 con 0x7f51c00b1e20 2024-03-18T20:27:36.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51bd7fa700 1 -- 172.21.15.1:0/136737942 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b4003680 con 0x7f51c00b1e20 2024-03-18T20:27:36.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51bd7fa700 1 -- 172.21.15.1:0/136737942 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b4003a30 con 0x7f51c00b1e20 2024-03-18T20:27:36.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.958+0000 7f51c702e700 1 -- 172.21.15.1:0/136737942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c0157d60 con 0x7f51c00b1e20 2024-03-18T20:27:36.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.959+0000 7f51c702e700 1 -- 172.21.15.1:0/136737942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c00cae70 con 0x7f51c00b1e20 2024-03-18T20:27:36.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.959+0000 7f51c702e700 1 -- 172.21.15.1:0/136737942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c00cb030 con 0x7f51c00b1e20 2024-03-18T20:27:36.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.960+0000 7f51bd7fa700 1 -- 172.21.15.1:0/136737942 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 157) v1 ==== 89358+0+0 (secure 0 0 0) 0x7f51b4003b90 con 0x7f51c00b1e20 2024-03-18T20:27:36.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.960+0000 7f51bd7fa700 1 --2- 172.21.15.1:0/136737942 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f51a406af20 0x7f51a406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:36.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.961+0000 7f51bffff700 1 -- 172.21.15.1:0/136737942 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f51a406af20 msgr2=0x7f51a406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:36.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.961+0000 7f51bd7fa700 1 -- 172.21.15.1:0/136737942 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f51b4089310 con 0x7f51c00b1e20 2024-03-18T20:27:36.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.961+0000 7f51bffff700 1 --2- 172.21.15.1:0/136737942 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f51a406af20 0x7f51a406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:27:36.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:36.969+0000 7f51bd7fa700 1 -- 172.21.15.1:0/136737942 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f51b4054050 con 0x7f51c00b1e20 2024-03-18T20:27:37.004 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:37.000+0000 7fe555cbe240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:27:37.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.162+0000 7f51bffff700 1 -- 172.21.15.1:0/136737942 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f51a406af20 msgr2=0x7f51a406d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:37.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.162+0000 7f51bffff700 1 --2- 172.21.15.1:0/136737942 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f51a406af20 0x7f51a406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:27:37.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.174+0000 7f51c702e700 1 -- 172.21.15.1:0/136737942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f51c00676a0 con 0x7f51c00b1e20 2024-03-18T20:27:37.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.178+0000 7f51bd7fa700 1 -- 172.21.15.1:0/136737942 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v90)=0 v90) v1 ==== 133+0+0 (secure 0 0 0) 0x7f51b4017020 con 0x7f51c00b1e20 2024-03-18T20:27:37.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.180+0000 7f51c702e700 1 -- 172.21.15.1:0/136737942 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f51a406af20 msgr2=0x7f51a406d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:27:37.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.180+0000 7f51c702e700 1 --2- 172.21.15.1:0/136737942 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f51a406af20 0x7f51a406d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:37.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.180+0000 7f51c702e700 1 -- 172.21.15.1:0/136737942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c00b1e20 msgr2=0x7f51c00ca930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:37.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.180+0000 7f51c702e700 1 --2- 172.21.15.1:0/136737942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c00b1e20 0x7f51c00ca930 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 crypto rx=0x7f51b400c780 tx=0x7f51b400ca90 comp rx=0 tx=0).stop 2024-03-18T20:27:37.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.180+0000 7f51c702e700 1 -- 172.21.15.1:0/136737942 shutdown_connections 2024-03-18T20:27:37.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.180+0000 7f51c702e700 1 --2- 172.21.15.1:0/136737942 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f51a406af20 0x7f51a406d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:37.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.180+0000 7f51c702e700 1 --2- 172.21.15.1:0/136737942 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51c00cff70 0x7f51c0154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:37.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.180+0000 7f51c702e700 1 --2- 172.21.15.1:0/136737942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c00b1e20 0x7f51c00ca930 unknown :-1 s=CLOSED pgs=3118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:37.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.180+0000 7f51c702e700 1 --2- 172.21.15.1:0/136737942 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f51c00cd870 0x7f51c0153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:37.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.180+0000 7f51c702e700 1 -- 172.21.15.1:0/136737942 >> 172.21.15.1:0/136737942 conn(0x7f51c00d0ff0 msgr2=0x7f51c00c82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:37.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.180+0000 7f51c702e700 1 -- 172.21.15.1:0/136737942 shutdown_connections 2024-03-18T20:27:37.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.180+0000 7f51c702e700 1 -- 172.21.15.1:0/136737942 wait complete. 2024-03-18T20:27:37.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:37.197 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-03-18T20:27:37.220 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-03-18T20:27:37.374 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:37.370+0000 7fb8bdc7b240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:27:37.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.388+0000 7ff7af57d700 1 Processor -- start 2024-03-18T20:27:37.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.388+0000 7ff7af57d700 1 -- start start 2024-03-18T20:27:37.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.389+0000 7ff7af57d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff7a815d5a0 0x7ff7a80ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:37.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.389+0000 7ff7af57d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff7a806ae60 0x7ff7a8167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:37.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.389+0000 7ff7af57d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7a80cf600 con 0x7ff7a80ce550 2024-03-18T20:27:37.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.389+0000 7ff7af57d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7a80ce9f0 con 0x7ff7a806ae60 2024-03-18T20:27:37.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.389+0000 7ff7af57d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a80cecd0 con 0x7ff7a815d5a0 2024-03-18T20:27:37.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.389+0000 7ff7acb18700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff7a80ce550 0x7ff7a806a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50468/0 (socket says 172.21.15.1:50468) 2024-03-18T20:27:37.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.389+0000 7ff7acb18700 1 -- 172.21.15.1:0/45888978 learned_addr learned my addr 172.21.15.1:0/45888978 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:37.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.389+0000 7ff7adb1a700 1 --2- 172.21.15.1:0/45888978 >> v2:172.21.15.165:3300/0 conn(0x7ff7a806ae60 0x7ff7a8167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:37.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.389+0000 7ff7ad319700 1 --2- 172.21.15.1:0/45888978 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff7a815d5a0 0x7ff7a80ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:37.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.390+0000 7ff79ffff700 1 -- 172.21.15.1:0/45888978 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252954295 0 0) 0x7ff7a80cf600 con 0x7ff7a80ce550 2024-03-18T20:27:37.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.390+0000 7ff79ffff700 1 -- 172.21.15.1:0/45888978 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7940035b0 con 0x7ff7a80ce550 2024-03-18T20:27:37.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.390+0000 7ff7adb1a700 1 -- 172.21.15.1:0/45888978 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff7a815d5a0 msgr2=0x7ff7a80ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:37.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.390+0000 7ff7adb1a700 1 --2- 172.21.15.1:0/45888978 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff7a815d5a0 0x7ff7a80ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:37.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.390+0000 7ff7adb1a700 1 -- 172.21.15.1:0/45888978 >> v1:172.21.15.1:6789/0 conn(0x7ff7a80ce550 legacy=0x7ff7a806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:37.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.390+0000 7ff7adb1a700 1 -- 172.21.15.1:0/45888978 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a80cefb0 con 0x7ff7a806ae60 2024-03-18T20:27:37.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.390+0000 7ff7ad319700 1 --2- 172.21.15.1:0/45888978 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff7a815d5a0 0x7ff7a80ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:27:37.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.390+0000 7ff7adb1a700 1 --2- 172.21.15.1:0/45888978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7a806ae60 0x7ff7a8167140 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0009d90 tx=0x7ff7a000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f19bfe6c2a4c992f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:37.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.390+0000 7ff79ffff700 1 -- 172.21.15.1:0/45888978 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a000e070 con 0x7ff7a806ae60 2024-03-18T20:27:37.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.390+0000 7ff79ffff700 1 -- 172.21.15.1:0/45888978 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a000bb20 con 0x7ff7a806ae60 2024-03-18T20:27:37.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.390+0000 7ff79ffff700 1 -- 172.21.15.1:0/45888978 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0012430 con 0x7ff7a806ae60 2024-03-18T20:27:37.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.391+0000 7ff7af57d700 1 -- 172.21.15.1:0/45888978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7a806ae60 msgr2=0x7ff7a8167140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:37.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.391+0000 7ff7af57d700 1 --2- 172.21.15.1:0/45888978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7a806ae60 0x7ff7a8167140 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0009d90 tx=0x7ff7a000b330 comp rx=0 tx=0).stop 2024-03-18T20:27:37.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.391+0000 7ff7af57d700 1 -- 172.21.15.1:0/45888978 shutdown_connections 2024-03-18T20:27:37.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.391+0000 7ff7af57d700 1 --2- 172.21.15.1:0/45888978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7a806ae60 0x7ff7a8167140 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:37.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.391+0000 7ff7af57d700 1 --2- 172.21.15.1:0/45888978 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff7a815d5a0 0x7ff7a80ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:37.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.391+0000 7ff7af57d700 1 -- 172.21.15.1:0/45888978 >> 172.21.15.1:0/45888978 conn(0x7ff7a80cc9b0 msgr2=0x7ff7a80ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:37.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.391+0000 7ff7af57d700 1 -- 172.21.15.1:0/45888978 shutdown_connections 2024-03-18T20:27:37.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.391+0000 7ff7af57d700 1 -- 172.21.15.1:0/45888978 wait complete. 2024-03-18T20:27:37.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.392+0000 7ff7af57d700 1 Processor -- start 2024-03-18T20:27:37.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.392+0000 7ff7af57d700 1 -- start start 2024-03-18T20:27:37.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.393+0000 7ff7af57d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7a806ae60 0x7ff7a8156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:37.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.393+0000 7ff7af57d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff7a80ce550 0x7ff7a815ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:37.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.393+0000 7ff7af57d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7a815d5a0 0x7ff7a8160480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:37.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.393+0000 7ff7af57d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a8152f40 con 0x7ff7a806ae60 2024-03-18T20:27:37.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.393+0000 7ff7af57d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a8153220 con 0x7ff7a815d5a0 2024-03-18T20:27:37.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.393+0000 7ff7af57d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a8153500 con 0x7ff7a80ce550 2024-03-18T20:27:37.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.393+0000 7ff7ad319700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7a806ae60 0x7ff7a8156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:37.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.393+0000 7ff7ad319700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7a806ae60 0x7ff7a8156320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60148/0 (socket says 172.21.15.1:60148) 2024-03-18T20:27:37.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.393+0000 7ff7ad319700 1 -- 172.21.15.1:0/920016958 learned_addr learned my addr 172.21.15.1:0/920016958 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:37.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.393+0000 7ff7adb1a700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7a815d5a0 0x7ff7a8160480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:37.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.393+0000 7ff7acb18700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff7a80ce550 0x7ff7a815ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:37.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.393+0000 7ff7ad319700 1 -- 172.21.15.1:0/920016958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff7a80ce550 msgr2=0x7ff7a815ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:37.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.393+0000 7ff7ad319700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff7a80ce550 0x7ff7a815ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:37.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.393+0000 7ff7ad319700 1 -- 172.21.15.1:0/920016958 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7a815d5a0 msgr2=0x7ff7a8160480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:37.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.393+0000 7ff7ad319700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7a815d5a0 0x7ff7a8160480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:37.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.393+0000 7ff7ad319700 1 -- 172.21.15.1:0/920016958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a4009430 con 0x7ff7a806ae60 2024-03-18T20:27:37.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.393+0000 7ff7ad319700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7a806ae60 0x7ff7a8156320 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 crypto rx=0x7ff7a400ebe0 tx=0x7ff7a400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:37.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.394+0000 7ff79dffb700 1 -- 172.21.15.1:0/920016958 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4005a20 con 0x7ff7a806ae60 2024-03-18T20:27:37.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.394+0000 7ff79dffb700 1 -- 172.21.15.1:0/920016958 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a4005b80 con 0x7ff7a806ae60 2024-03-18T20:27:37.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.394+0000 7ff7af57d700 1 -- 172.21.15.1:0/920016958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a0009a70 con 0x7ff7a806ae60 2024-03-18T20:27:37.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.394+0000 7ff79dffb700 1 -- 172.21.15.1:0/920016958 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a400c850 con 0x7ff7a806ae60 2024-03-18T20:27:37.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.394+0000 7ff7af57d700 1 -- 172.21.15.1:0/920016958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a8153ac0 con 0x7ff7a806ae60 2024-03-18T20:27:37.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.395+0000 7ff7af57d700 1 -- 172.21.15.1:0/920016958 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7a8153e30 con 0x7ff7a806ae60 2024-03-18T20:27:37.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.395+0000 7ff79dffb700 1 -- 172.21.15.1:0/920016958 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 157) v1 ==== 89358+0+0 (secure 0 0 0) 0x7ff7a400c9b0 con 0x7ff7a806ae60 2024-03-18T20:27:37.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.396+0000 7ff79dffb700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff78c06af20 0x7ff78c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:37.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.396+0000 7ff7acb18700 1 -- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff78c06af20 msgr2=0x7ff78c06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:37.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.396+0000 7ff7acb18700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff78c06af20 0x7ff78c06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:27:37.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.396+0000 7ff79dffb700 1 -- 172.21.15.1:0/920016958 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6129+0+0 (secure 0 0 0) 0x7ff7a4058bd0 con 0x7ff7a806ae60 2024-03-18T20:27:37.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.401+0000 7ff79dffb700 1 -- 172.21.15.1:0/920016958 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff7a4054810 con 0x7ff7a806ae60 2024-03-18T20:27:37.465 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:37.461+0000 7fe555cbe240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:27:37.586 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:37.582+0000 7fb8bdc7b240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:27:37.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.597+0000 7ff7acb18700 1 -- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff78c06af20 msgr2=0x7ff78c06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:37.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.597+0000 7ff7acb18700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff78c06af20 0x7ff78c06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:27:37.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.633+0000 7ff7af57d700 1 -- 172.21.15.1:0/920016958 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7a8153c50 con 0x7ff78c06af20 2024-03-18T20:27:37.704 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:37.681+0000 7fe555cbe240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:27:37.959 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:37.954+0000 7fb8bdc7b240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:27:38.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.998+0000 7ff7acb18700 1 -- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff78c06af20 msgr2=0x7ff78c06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:38.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:37.998+0000 7ff7acb18700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff78c06af20 0x7ff78c06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T20:27:38.075 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:38.071+0000 7fe555cbe240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:27:38.548 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:38.544+0000 7fb8bdc7b240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:27:38.620 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:38.616+0000 7fb8bdc7b240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:27:38.696 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:38.692+0000 7fe555cbe240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:27:38.746 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:38.742+0000 7fb8bdc7b240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:27:38.771 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:38.768+0000 7fe555cbe240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:27:38.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:38.799+0000 7ff7acb18700 1 -- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff78c06af20 msgr2=0x7ff78c06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:38.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:38.799+0000 7ff7acb18700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff78c06af20 0x7ff78c06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T20:27:38.906 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:38.902+0000 7fe555cbe240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:27:38.928 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:38.924+0000 7fb8bdc7b240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:27:39.000 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:38.996+0000 7fb8bdc7b240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:27:39.078 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:39.074+0000 7fb8bdc7b240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:27:39.092 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:39.088+0000 7fe555cbe240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:27:39.155 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:39.151+0000 7fb8bdc7b240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:27:39.168 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:39.164+0000 7fe555cbe240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:27:39.233 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:39.229+0000 7fb8bdc7b240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:27:39.250 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:39.247+0000 7fe555cbe240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:27:39.332 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:39.328+0000 7fe555cbe240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:27:39.403 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:39.399+0000 7fb8bdc7b240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:27:39.414 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:39.410+0000 7fe555cbe240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:27:39.594 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:39.590+0000 7fe555cbe240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:27:39.617 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:39.613+0000 7fb8bdc7b240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:27:39.769 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:39.765+0000 7fb8bdc7b240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:27:39.817 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:39.813+0000 7fe555cbe240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:27:39.842 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:39.838+0000 7fb8bdc7b240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:27:39.977 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:39.973+0000 7fe555cbe240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:27:40.055 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:40.051+0000 7fe555cbe240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:27:40.289 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:40.285+0000 7fb8bdc7b240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:27:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:40.400+0000 7ff7acb18700 1 -- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff78c06af20 msgr2=0x7ff78c06d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:40.400+0000 7ff7acb18700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff78c06af20 0x7ff78c06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T20:27:40.532 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:40.528+0000 7fe555cbe240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:27:40.626 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:40.622+0000 7fb8bdc7b240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:27:40.790 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:40.786+0000 7fb8bdc7b240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:27:40.886 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:40.882+0000 7fe555cbe240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:27:41.060 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:41.056+0000 7fe555cbe240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:27:41.178 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:41.174+0000 7fb8bdc7b240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:27:41.343 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:41.339+0000 7fb8bdc7b240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:27:41.474 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:27:41.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:27:41.478 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:41.474+0000 7fe555cbe240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:27:41.655 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:41.651+0000 7fe555cbe240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:27:41.709 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:41.705+0000 7fb8bdc7b240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:27:41.797 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:41.793+0000 7fb8bdc7b240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:27:41.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:41.863+0000 7ff79dffb700 1 -- 172.21.15.1:0/920016958 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 158) v1 ==== 89358+0+0 (secure 0 0 0) 0x7ff7a4054be0 con 0x7ff7a806ae60 2024-03-18T20:27:42.039 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:42.036+0000 7fe555cbe240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:27:42.134 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:42.130+0000 7fe555cbe240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:27:42.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:42.149+0000 7ff79dffb700 1 -- 172.21.15.1:0/920016958 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 159) v1 ==== 88865+0+0 (secure 0 0 0) 0x7ff7a4058730 con 0x7ff7a806ae60 2024-03-18T20:27:42.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:42.149+0000 7ff79dffb700 1 -- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff78c06af20 msgr2=0x7ff78c06d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:27:42.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:42.149+0000 7ff79dffb700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff78c06af20 0x7ff78c06d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:42.293 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:42.289+0000 7fe4cb8e1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:42.294 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:42.289+0000 7fe4cb8e1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:42.294 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:42.289+0000 7fe4cb8e1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:42.294 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:42.289+0000 7fe4cb8e1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:42.294 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:42.289+0000 7fe4cb8e1700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:42.294 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:42.290+0000 7fe4c88db700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:42.294 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:42.290+0000 7fe4c88db700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:42.294 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:42.290+0000 7fe4c88db700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:42.294 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:42.290+0000 7fe4c88db700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:42.294 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:42.290+0000 7fe4c88db700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:43.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.153+0000 7ff79dffb700 1 -- 172.21.15.1:0/920016958 <== mon.0 v2:172.21.15.1:3300/0 9 ==== mgrmap(e 160) v1 ==== 88992+0+0 (secure 0 0 0) 0x7ff7a40554b0 con 0x7ff7a806ae60 2024-03-18T20:27:43.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.154+0000 7ff79dffb700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff78c06af20 0x7ff78c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:43.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.154+0000 7ff79dffb700 1 -- 172.21.15.1:0/920016958 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7a8153c50 con 0x7ff78c06af20 2024-03-18T20:27:43.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.156+0000 7ff7acb18700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff78c06af20 0x7ff78c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:43.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.159+0000 7ff7acb18700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff78c06af20 0x7ff78c06d3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8152b30 tx=0x7ff798007dc0 comp rx=0 tx=0).ready entity=mgr.13401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:43.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.218+0000 7ff79dffb700 1 -- 172.21.15.1:0/920016958 <== mgr.13401 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff7a8153c50 con 0x7ff78c06af20 2024-03-18T20:27:43.222 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path snap_test_dir1 2024-03-18T20:27:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.220+0000 7ff7af57d700 1 -- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff78c06af20 msgr2=0x7ff78c06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.220+0000 7ff7af57d700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff78c06af20 0x7ff78c06d3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8152b30 tx=0x7ff798007dc0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.220+0000 7ff7af57d700 1 -- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7a806ae60 msgr2=0x7ff7a8156320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.220+0000 7ff7af57d700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7a806ae60 0x7ff7a8156320 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 crypto rx=0x7ff7a400ebe0 tx=0x7ff7a400eef0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.220+0000 7ff7af57d700 1 -- 172.21.15.1:0/920016958 shutdown_connections 2024-03-18T20:27:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.220+0000 7ff7af57d700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff78c06af20 0x7ff78c06d3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.220+0000 7ff7af57d700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7a815d5a0 0x7ff7a8160480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.220+0000 7ff7af57d700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7a806ae60 0x7ff7a8156320 unknown :-1 s=CLOSED pgs=3120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.221+0000 7ff7af57d700 1 --2- 172.21.15.1:0/920016958 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff7a80ce550 0x7ff7a815ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.221+0000 7ff7af57d700 1 -- 172.21.15.1:0/920016958 >> 172.21.15.1:0/920016958 conn(0x7ff7a80cc9b0 msgr2=0x7ff7a81e65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:43.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.221+0000 7ff7af57d700 1 -- 172.21.15.1:0/920016958 shutdown_connections 2024-03-18T20:27:43.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.221+0000 7ff7af57d700 1 -- 172.21.15.1:0/920016958 wait complete. 2024-03-18T20:27:43.237 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-03-18T20:27:43.238 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2024-03-18T20:27:43.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.365+0000 7f73033d8700 1 Processor -- start 2024-03-18T20:27:43.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.365+0000 7f73033d8700 1 -- start start 2024-03-18T20:27:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.366+0000 7f73033d8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72fc06f7d0 0x7f72fc15dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.366+0000 7f73033d8700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc0b41a0 con 0x7f72fc15a160 2024-03-18T20:27:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.366+0000 7f73033d8700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc0b4d10 con 0x7f72fc06fc50 2024-03-18T20:27:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.366+0000 7f73033d8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc15ec10 con 0x7f72fc06f7d0 2024-03-18T20:27:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.366+0000 7f7301174700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f72fc15a160 0x7f72fc0b3a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50480/0 (socket says 172.21.15.1:50480) 2024-03-18T20:27:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.366+0000 7f7301174700 1 -- 172.21.15.1:0/2062989508 learned_addr learned my addr 172.21.15.1:0/2062989508 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.366+0000 7f7300973700 1 --2- 172.21.15.1:0/2062989508 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72fc06f7d0 0x7f72fc15dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.366+0000 7f72f3fff700 1 -- 172.21.15.1:0/2062989508 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162058306 0 0) 0x7f72fc0b41a0 con 0x7f72fc15a160 2024-03-18T20:27:43.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.366+0000 7f72f3fff700 1 -- 172.21.15.1:0/2062989508 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e80035b0 con 0x7f72fc15a160 2024-03-18T20:27:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.366+0000 7f72f3fff700 1 -- 172.21.15.1:0/2062989508 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740287454 0 0) 0x7f72fc0b4d10 con 0x7f72fc06fc50 2024-03-18T20:27:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.366+0000 7f72f3fff700 1 -- 172.21.15.1:0/2062989508 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72fc0b41a0 con 0x7f72fc06fc50 2024-03-18T20:27:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.366+0000 7f7300973700 1 -- 172.21.15.1:0/2062989508 >> v1:172.21.15.165:6789/0 conn(0x7f72fc06fc50 legacy=0x7f72fc0717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.366+0000 7f7300973700 1 -- 172.21.15.1:0/2062989508 >> v1:172.21.15.1:6789/0 conn(0x7f72fc15a160 legacy=0x7f72fc0b3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.366+0000 7f7300973700 1 -- 172.21.15.1:0/2062989508 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc0b4a30 con 0x7f72fc06f7d0 2024-03-18T20:27:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.367+0000 7f7300973700 1 --2- 172.21.15.1:0/2062989508 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72fc06f7d0 0x7f72fc15dd50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f72f8009870 tx=0x7f72f8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcf3247c49f7b49e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.367+0000 7f72f3fff700 1 -- 172.21.15.1:0/2062989508 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f8011830 con 0x7f72fc06f7d0 2024-03-18T20:27:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.367+0000 7f72f3fff700 1 -- 172.21.15.1:0/2062989508 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f8011990 con 0x7f72fc06f7d0 2024-03-18T20:27:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.367+0000 7f72f3fff700 1 -- 172.21.15.1:0/2062989508 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f8011d00 con 0x7f72fc06f7d0 2024-03-18T20:27:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.367+0000 7f73033d8700 1 -- 172.21.15.1:0/2062989508 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72fc06f7d0 msgr2=0x7f72fc15dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.367+0000 7f73033d8700 1 --2- 172.21.15.1:0/2062989508 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72fc06f7d0 0x7f72fc15dd50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f72f8009870 tx=0x7f72f8009b80 comp rx=0 tx=0).stop 2024-03-18T20:27:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.367+0000 7f73033d8700 1 -- 172.21.15.1:0/2062989508 shutdown_connections 2024-03-18T20:27:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.367+0000 7f73033d8700 1 --2- 172.21.15.1:0/2062989508 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72fc06f7d0 0x7f72fc15dd50 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.367+0000 7f73033d8700 1 -- 172.21.15.1:0/2062989508 >> 172.21.15.1:0/2062989508 conn(0x7f72fc0d00c0 msgr2=0x7f72fc0d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:43.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.367+0000 7f73033d8700 1 -- 172.21.15.1:0/2062989508 shutdown_connections 2024-03-18T20:27:43.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.368+0000 7f73033d8700 1 -- 172.21.15.1:0/2062989508 wait complete. 2024-03-18T20:27:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.369+0000 7f73033d8700 1 Processor -- start 2024-03-18T20:27:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.369+0000 7f73033d8700 1 -- start start 2024-03-18T20:27:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.369+0000 7f73033d8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72fc06f7d0 0x7f72fc157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.369+0000 7f73033d8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72fc06fc50 0x7f72fc0c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.369+0000 7f73033d8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72fc15a160 0x7f72fc152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:43.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.369+0000 7f73033d8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc151d10 con 0x7f72fc06f7d0 2024-03-18T20:27:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.369+0000 7f73033d8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc157e50 con 0x7f72fc15a160 2024-03-18T20:27:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.369+0000 7f73033d8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc158130 con 0x7f72fc06fc50 2024-03-18T20:27:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.370+0000 7f7301174700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72fc06f7d0 0x7f72fc157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.370+0000 7f7301174700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72fc06f7d0 0x7f72fc157750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60332/0 (socket says 172.21.15.1:60332) 2024-03-18T20:27:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.370+0000 7f7301174700 1 -- 172.21.15.1:0/3362217518 learned_addr learned my addr 172.21.15.1:0/3362217518 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.370+0000 7f7301174700 1 -- 172.21.15.1:0/3362217518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72fc06fc50 msgr2=0x7f72fc0c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.370+0000 7f7301174700 1 --2- 172.21.15.1:0/3362217518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72fc06fc50 0x7f72fc0c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.370+0000 7f7301174700 1 -- 172.21.15.1:0/3362217518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72fc15a160 msgr2=0x7f72fc152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.370+0000 7f7301174700 1 --2- 172.21.15.1:0/3362217518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72fc15a160 0x7f72fc152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.370+0000 7f7301174700 1 -- 172.21.15.1:0/3362217518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f80094a0 con 0x7f72fc06f7d0 2024-03-18T20:27:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.370+0000 7f7301174700 1 --2- 172.21.15.1:0/3362217518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72fc06f7d0 0x7f72fc157750 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 crypto rx=0x7f72ec00c700 tx=0x7f72ec00cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.370+0000 7f72f1ffb700 1 -- 172.21.15.1:0/3362217518 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec005d70 con 0x7f72fc06f7d0 2024-03-18T20:27:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.370+0000 7f72f1ffb700 1 -- 172.21.15.1:0/3362217518 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ec0070d0 con 0x7f72fc06f7d0 2024-03-18T20:27:43.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.370+0000 7f73033d8700 1 -- 172.21.15.1:0/3362217518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72fc158410 con 0x7f72fc06f7d0 2024-03-18T20:27:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.370+0000 7f72f1ffb700 1 -- 172.21.15.1:0/3362217518 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec007480 con 0x7f72fc06f7d0 2024-03-18T20:27:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.370+0000 7f73033d8700 1 -- 172.21.15.1:0/3362217518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72fc1589d0 con 0x7f72fc06f7d0 2024-03-18T20:27:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.371+0000 7f73033d8700 1 -- 172.21.15.1:0/3362217518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72fc0676a0 con 0x7f72fc06f7d0 2024-03-18T20:27:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.372+0000 7f72f1ffb700 1 -- 172.21.15.1:0/3362217518 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 160) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f72ec0075e0 con 0x7f72fc06f7d0 2024-03-18T20:27:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.372+0000 7f72f1ffb700 1 --2- 172.21.15.1:0/3362217518 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f72e006ab30 0x7f72e006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.372+0000 7f7300973700 1 --2- 172.21.15.1:0/3362217518 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f72e006ab30 0x7f72e006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.372+0000 7f72f1ffb700 1 -- 172.21.15.1:0/3362217518 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f72ec088950 con 0x7f72fc06f7d0 2024-03-18T20:27:43.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.373+0000 7f7300973700 1 --2- 172.21.15.1:0/3362217518 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f72e006ab30 0x7f72e006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f72fc1f6590 tx=0x7f72f8010000 comp rx=0 tx=0).ready entity=mgr.13401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:43.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.377+0000 7f72f1ffb700 1 -- 172.21.15.1:0/3362217518 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f72ec053aa0 con 0x7f72fc06f7d0 2024-03-18T20:27:43.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.605+0000 7f73033d8700 1 -- 172.21.15.1:0/3362217518 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f72fc06f2b0 con 0x7f72e006ab30 2024-03-18T20:27:43.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.657+0000 7f72f1ffb700 1 -- 172.21.15.1:0/3362217518 <== mgr.13401 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7f72f801e070 con 0x7f72e006ab30 2024-03-18T20:27:43.661 INFO:teuthology.orchestra.run.smithi001.stdout:Retention added to path snap_test_dir1 2024-03-18T20:27:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.659+0000 7f73033d8700 1 -- 172.21.15.1:0/3362217518 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f72e006ab30 msgr2=0x7f72e006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:43.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.659+0000 7f73033d8700 1 --2- 172.21.15.1:0/3362217518 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f72e006ab30 0x7f72e006cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f72fc1f6590 tx=0x7f72f8010000 comp rx=0 tx=0).stop 2024-03-18T20:27:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.659+0000 7f73033d8700 1 -- 172.21.15.1:0/3362217518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72fc06f7d0 msgr2=0x7f72fc157750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.659+0000 7f73033d8700 1 --2- 172.21.15.1:0/3362217518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72fc06f7d0 0x7f72fc157750 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 crypto rx=0x7f72ec00c700 tx=0x7f72ec00cac0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.659+0000 7f73033d8700 1 -- 172.21.15.1:0/3362217518 shutdown_connections 2024-03-18T20:27:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.659+0000 7f73033d8700 1 --2- 172.21.15.1:0/3362217518 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f72e006ab30 0x7f72e006cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.659+0000 7f73033d8700 1 --2- 172.21.15.1:0/3362217518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72fc15a160 0x7f72fc152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.659+0000 7f73033d8700 1 --2- 172.21.15.1:0/3362217518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72fc06f7d0 0x7f72fc157750 unknown :-1 s=CLOSED pgs=3135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.659+0000 7f73033d8700 1 --2- 172.21.15.1:0/3362217518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f72fc06fc50 0x7f72fc0c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.659+0000 7f73033d8700 1 -- 172.21.15.1:0/3362217518 >> 172.21.15.1:0/3362217518 conn(0x7f72fc0d00c0 msgr2=0x7f72fc072180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.659+0000 7f73033d8700 1 -- 172.21.15.1:0/3362217518 shutdown_connections 2024-03-18T20:27:43.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.659+0000 7f73033d8700 1 -- 172.21.15.1:0/3362217518 wait complete. 2024-03-18T20:27:43.675 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2024-03-18T20:27:43.680 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-18T20:27:43.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-18T20:27:43.727 INFO:teuthology.orchestra.run.smithi001.stdout:[global] 2024-03-18T20:27:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: chdir = "" 2024-03-18T20:27:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-18T20:27:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: auth supported = cephx 2024-03-18T20:27:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: filestore xattr use omap = true 2024-03-18T20:27:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: mon clock drift allowed = 1.000 2024-03-18T20:27:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: osd crush chooseleaf type = 0 2024-03-18T20:27:43.727 INFO:teuthology.orchestra.run.smithi001.stdout: auth debug = true 2024-03-18T20:27:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on old message = true 2024-03-18T20:27:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on bug = true 2024-03-18T20:27:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-18T20:27:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: mon pg warn max object skew = 0 2024-03-18T20:27:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: # disable pg_autoscaler by default for new pools 2024-03-18T20:27:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-18T20:27:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default size = 2 2024-03-18T20:27:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow primary affinity = true 2024-03-18T20:27:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow pg remap = true 2024-03-18T20:27:43.728 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on legacy crush tunables = false 2024-03-18T20:27:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on crush straw calc version zero = false 2024-03-18T20:27:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on no sortbitwise = false 2024-03-18T20:27:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on osd down out interval zero = false 2024-03-18T20:27:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on too few osds = false 2024-03-18T20:27:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-18T20:27:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-18T20:27:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: mon_allow_pool_size_one = true 2024-03-18T20:27:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-18T20:27:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: osd default data pool replay window = 5 2024-03-18T20:27:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.729 INFO:teuthology.orchestra.run.smithi001.stdout: mon allow pool delete = true 2024-03-18T20:27:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: mon cluster log file level = debug 2024-03-18T20:27:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: debug asserts on shutdown = true 2024-03-18T20:27:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: mon health detail to clog = false 2024-03-18T20:27:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: mon host = "172.21.15.1,172.21.15.165,[v2:172.21.15.165:3301,v1:172.21.15.165:6790]" 2024-03-18T20:27:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: fsid = 9c1c697f-8655-4ef3-8a13-da568d2cbe86 2024-03-18T20:27:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.730 INFO:teuthology.orchestra.run.smithi001.stdout:[osd] 2024-03-18T20:27:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: osd journal size = 100 2024-03-18T20:27:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub load threshold = 5.0 2024-03-18T20:27:43.730 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub max interval = 600 2024-03-18T20:27:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: osd mclock profile = high_recovery_ops 2024-03-18T20:27:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: osd recover clone overlap = true 2024-03-18T20:27:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: osd recovery max chunk = 1048576 2024-03-18T20:27:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug shutdown = true 2024-03-18T20:27:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug op order = true 2024-03-18T20:27:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify stray on activate = true 2024-03-18T20:27:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: osd open classes on start = true 2024-03-18T20:27:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug pg log writeout = true 2024-03-18T20:27:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: osd deep scrub update digest min age = 30 2024-03-18T20:27:43.731 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: osd map max advance = 10 2024-03-18T20:27:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: journal zero on create = true 2024-03-18T20:27:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: filestore ondisk finisher threads = 3 2024-03-18T20:27:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: filestore apply finisher threads = 3 2024-03-18T20:27:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: bdev debug aio = true 2024-03-18T20:27:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug misdirected ops = true 2024-03-18T20:27:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore block size = 96636764160 2024-03-18T20:27:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore fsck on mount = True 2024-03-18T20:27:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluefs = 20 2024-03-18T20:27:43.732 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluestore = 20 2024-03-18T20:27:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-18T20:27:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: debug osd = 20 2024-03-18T20:27:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: debug rocksdb = 10 2024-03-18T20:27:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd backfillfull_ratio = 0.85 2024-03-18T20:27:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd full ratio = 0.9 2024-03-18T20:27:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd nearfull ratio = 0.8 2024-03-18T20:27:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: osd failsafe full ratio = 0.95 2024-03-18T20:27:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: osd objectstore = bluestore 2024-03-18T20:27:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: osd op complaint time = 180 2024-03-18T20:27:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.733 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr] 2024-03-18T20:27:43.733 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-18T20:27:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: debug mgr = 20 2024-03-18T20:27:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-03-18T20:27:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-03-18T20:27:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-03-18T20:27:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-03-18T20:27:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: mgr/telemetry/nag = false 2024-03-18T20:27:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: debug client = 20 2024-03-18T20:27:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: debug finisher = 20 2024-03-18T20:27:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.734 INFO:teuthology.orchestra.run.smithi001.stdout:[mon] 2024-03-18T20:27:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-18T20:27:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-03-18T20:27:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: debug paxos = 20 2024-03-18T20:27:43.734 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-03-18T20:27:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: mon data avail warn = 5 2024-03-18T20:27:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: mon mgr mkfs grace = 240 2024-03-18T20:27:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-03-18T20:27:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd reporter subtree level = osd 2024-03-18T20:27:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd prime pg temp = true 2024-03-18T20:27:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-03-18T20:27:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-18T20:27:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: auth mon ticket ttl = 660 # 11m 2024-03-18T20:27:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: auth service ticket ttl = 240 # 4m 2024-03-18T20:27:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: # don't complain about insecure global_id in the test suite 2024-03-18T20:27:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-18T20:27:43.735 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-18T20:27:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: # 1m isn't quite enough 2024-03-18T20:27:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: mon_down_mkfs_grace = 2m 2024-03-18T20:27:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_filestore_osds = false 2024-03-18T20:27:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: mon op complaint time = 120 2024-03-18T20:27:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:43.736 INFO:teuthology.orchestra.run.smithi001.stdout:[client] 2024-03-18T20:27:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: rgw cache enabled = true 2024-03-18T20:27:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable ops log = true 2024-03-18T20:27:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable usage log = true 2024-03-18T20:27:43.736 INFO:teuthology.orchestra.run.smithi001.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-18T20:27:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-18T20:27:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: client mount timeout = 600 2024-03-18T20:27:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: debug client = 20 2024-03-18T20:27:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-18T20:27:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = "" 2024-03-18T20:27:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: rados mon op timeout = 900 2024-03-18T20:27:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: rados osd op timeout = 900 2024-03-18T20:27:43.737 INFO:teuthology.orchestra.run.smithi001.stdout: client snapdir = .snap 2024-03-18T20:27:43.738 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.a] 2024-03-18T20:27:43.738 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.b] 2024-03-18T20:27:43.738 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.c] 2024-03-18T20:27:43.738 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.a] 2024-03-18T20:27:43.738 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.c] 2024-03-18T20:27:43.739 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.b] 2024-03-18T20:27:43.739 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.d] 2024-03-18T20:27:43.739 INFO:teuthology.orchestra.run.smithi001.stdout:[mds] 2024-03-18T20:27:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: debug mds = 20 2024-03-18T20:27:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: debug mds balancer = 20 2024-03-18T20:27:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-03-18T20:27:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: mds debug frag = True 2024-03-18T20:27:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: mds debug scatterstat = True 2024-03-18T20:27:43.739 INFO:teuthology.orchestra.run.smithi001.stdout: mds op complaint time = 180 2024-03-18T20:27:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: mds verify scatter = True 2024-03-18T20:27:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: osd op complaint time = 180 2024-03-18T20:27:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: rados mon op timeout = 900 2024-03-18T20:27:43.740 INFO:teuthology.orchestra.run.smithi001.stdout: rados osd op timeout = 900 2024-03-18T20:27:43.740 INFO:teuthology.orchestra.run.smithi165.stdout:[global] 2024-03-18T20:27:43.740 INFO:teuthology.orchestra.run.smithi165.stdout: chdir = "" 2024-03-18T20:27:43.740 INFO:teuthology.orchestra.run.smithi165.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-18T20:27:43.741 INFO:teuthology.orchestra.run.smithi165.stdout: auth supported = cephx 2024-03-18T20:27:43.741 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.741 INFO:teuthology.orchestra.run.smithi165.stdout: filestore xattr use omap = true 2024-03-18T20:27:43.741 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.741 INFO:teuthology.orchestra.run.smithi165.stdout: mon clock drift allowed = 1.000 2024-03-18T20:27:43.741 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.741 INFO:teuthology.orchestra.run.smithi165.stdout: osd crush chooseleaf type = 0 2024-03-18T20:27:43.741 INFO:teuthology.orchestra.run.smithi165.stdout: auth debug = true 2024-03-18T20:27:43.741 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.742 INFO:teuthology.orchestra.run.smithi165.stdout: ms die on old message = true 2024-03-18T20:27:43.742 INFO:teuthology.orchestra.run.smithi165.stdout: ms die on bug = true 2024-03-18T20:27:43.742 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.742 INFO:teuthology.orchestra.run.smithi165.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-18T20:27:43.742 INFO:teuthology.orchestra.run.smithi165.stdout: mon pg warn max object skew = 0 2024-03-18T20:27:43.742 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.742 INFO:teuthology.orchestra.run.smithi165.stdout: # disable pg_autoscaler by default for new pools 2024-03-18T20:27:43.742 INFO:teuthology.orchestra.run.smithi165.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-18T20:27:43.742 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.742 INFO:teuthology.orchestra.run.smithi165.stdout: osd pool default size = 2 2024-03-18T20:27:43.742 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.742 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd allow primary affinity = true 2024-03-18T20:27:43.742 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd allow pg remap = true 2024-03-18T20:27:43.743 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on legacy crush tunables = false 2024-03-18T20:27:43.743 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on crush straw calc version zero = false 2024-03-18T20:27:43.743 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on no sortbitwise = false 2024-03-18T20:27:43.743 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on osd down out interval zero = false 2024-03-18T20:27:43.743 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on too few osds = false 2024-03-18T20:27:43.743 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-18T20:27:43.743 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-18T20:27:43.743 INFO:teuthology.orchestra.run.smithi165.stdout: mon_allow_pool_size_one = true 2024-03-18T20:27:43.743 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.743 INFO:teuthology.orchestra.run.smithi165.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-18T20:27:43.743 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.743 INFO:teuthology.orchestra.run.smithi165.stdout: osd default data pool replay window = 5 2024-03-18T20:27:43.743 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.743 INFO:teuthology.orchestra.run.smithi165.stdout: mon allow pool delete = true 2024-03-18T20:27:43.743 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.744 INFO:teuthology.orchestra.run.smithi165.stdout: mon cluster log file level = debug 2024-03-18T20:27:43.744 INFO:teuthology.orchestra.run.smithi165.stdout: debug asserts on shutdown = true 2024-03-18T20:27:43.744 INFO:teuthology.orchestra.run.smithi165.stdout: mon health detail to clog = false 2024-03-18T20:27:43.744 INFO:teuthology.orchestra.run.smithi165.stdout: mon host = "172.21.15.1,172.21.15.165,[v2:172.21.15.165:3301,v1:172.21.15.165:6790]" 2024-03-18T20:27:43.744 INFO:teuthology.orchestra.run.smithi165.stdout: fsid = 9c1c697f-8655-4ef3-8a13-da568d2cbe86 2024-03-18T20:27:43.744 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.744 INFO:teuthology.orchestra.run.smithi165.stdout:[osd] 2024-03-18T20:27:43.744 INFO:teuthology.orchestra.run.smithi165.stdout: osd journal size = 100 2024-03-18T20:27:43.744 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.744 INFO:teuthology.orchestra.run.smithi165.stdout: osd scrub load threshold = 5.0 2024-03-18T20:27:43.744 INFO:teuthology.orchestra.run.smithi165.stdout: osd scrub max interval = 600 2024-03-18T20:27:43.744 INFO:teuthology.orchestra.run.smithi165.stdout: osd mclock profile = high_recovery_ops 2024-03-18T20:27:43.744 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.744 INFO:teuthology.orchestra.run.smithi165.stdout: osd recover clone overlap = true 2024-03-18T20:27:43.744 INFO:teuthology.orchestra.run.smithi165.stdout: osd recovery max chunk = 1048576 2024-03-18T20:27:43.745 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.745 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug shutdown = true 2024-03-18T20:27:43.745 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug op order = true 2024-03-18T20:27:43.745 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug verify stray on activate = true 2024-03-18T20:27:43.745 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.745 INFO:teuthology.orchestra.run.smithi165.stdout: osd open classes on start = true 2024-03-18T20:27:43.745 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug pg log writeout = true 2024-03-18T20:27:43.745 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.745 INFO:teuthology.orchestra.run.smithi165.stdout: osd deep scrub update digest min age = 30 2024-03-18T20:27:43.745 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.745 INFO:teuthology.orchestra.run.smithi165.stdout: osd map max advance = 10 2024-03-18T20:27:43.745 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.745 INFO:teuthology.orchestra.run.smithi165.stdout: journal zero on create = true 2024-03-18T20:27:43.745 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.745 INFO:teuthology.orchestra.run.smithi165.stdout: filestore ondisk finisher threads = 3 2024-03-18T20:27:43.746 INFO:teuthology.orchestra.run.smithi165.stdout: filestore apply finisher threads = 3 2024-03-18T20:27:43.746 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.746 INFO:teuthology.orchestra.run.smithi165.stdout: bdev debug aio = true 2024-03-18T20:27:43.746 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug misdirected ops = true 2024-03-18T20:27:43.746 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore block size = 96636764160 2024-03-18T20:27:43.746 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore fsck on mount = True 2024-03-18T20:27:43.746 INFO:teuthology.orchestra.run.smithi165.stdout: debug bluefs = 20 2024-03-18T20:27:43.746 INFO:teuthology.orchestra.run.smithi165.stdout: debug bluestore = 20 2024-03-18T20:27:43.746 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-03-18T20:27:43.746 INFO:teuthology.orchestra.run.smithi165.stdout: debug osd = 20 2024-03-18T20:27:43.746 INFO:teuthology.orchestra.run.smithi165.stdout: debug rocksdb = 10 2024-03-18T20:27:43.746 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd backfillfull_ratio = 0.85 2024-03-18T20:27:43.746 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd full ratio = 0.9 2024-03-18T20:27:43.746 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd nearfull ratio = 0.8 2024-03-18T20:27:43.747 INFO:teuthology.orchestra.run.smithi165.stdout: osd failsafe full ratio = 0.95 2024-03-18T20:27:43.747 INFO:teuthology.orchestra.run.smithi165.stdout: osd objectstore = bluestore 2024-03-18T20:27:43.749 INFO:teuthology.orchestra.run.smithi165.stdout: osd op complaint time = 180 2024-03-18T20:27:43.749 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.749 INFO:teuthology.orchestra.run.smithi165.stdout:[mgr] 2024-03-18T20:27:43.750 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-03-18T20:27:43.750 INFO:teuthology.orchestra.run.smithi165.stdout: debug mgr = 20 2024-03-18T20:27:43.750 INFO:teuthology.orchestra.run.smithi165.stdout: debug mon = 20 2024-03-18T20:27:43.750 INFO:teuthology.orchestra.run.smithi165.stdout: debug auth = 20 2024-03-18T20:27:43.750 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min pgs per osd = 4 2024-03-18T20:27:43.750 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min bytes per osd = 10 2024-03-18T20:27:43.750 INFO:teuthology.orchestra.run.smithi165.stdout: mgr/telemetry/nag = false 2024-03-18T20:27:43.750 INFO:teuthology.orchestra.run.smithi165.stdout: debug client = 20 2024-03-18T20:27:43.750 INFO:teuthology.orchestra.run.smithi165.stdout: debug finisher = 20 2024-03-18T20:27:43.750 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.750 INFO:teuthology.orchestra.run.smithi165.stdout:[mon] 2024-03-18T20:27:43.750 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-03-18T20:27:43.750 INFO:teuthology.orchestra.run.smithi165.stdout: debug mon = 20 2024-03-18T20:27:43.751 INFO:teuthology.orchestra.run.smithi165.stdout: debug paxos = 20 2024-03-18T20:27:43.751 INFO:teuthology.orchestra.run.smithi165.stdout: debug auth = 20 2024-03-18T20:27:43.751 INFO:teuthology.orchestra.run.smithi165.stdout: mon data avail warn = 5 2024-03-18T20:27:43.751 INFO:teuthology.orchestra.run.smithi165.stdout: mon mgr mkfs grace = 240 2024-03-18T20:27:43.751 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min pgs per osd = 4 2024-03-18T20:27:43.751 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd reporter subtree level = osd 2024-03-18T20:27:43.751 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd prime pg temp = true 2024-03-18T20:27:43.751 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min bytes per osd = 10 2024-03-18T20:27:43.751 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.751 INFO:teuthology.orchestra.run.smithi165.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-18T20:27:43.751 INFO:teuthology.orchestra.run.smithi165.stdout: auth mon ticket ttl = 660 # 11m 2024-03-18T20:27:43.751 INFO:teuthology.orchestra.run.smithi165.stdout: auth service ticket ttl = 240 # 4m 2024-03-18T20:27:43.751 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.751 INFO:teuthology.orchestra.run.smithi165.stdout: # don't complain about insecure global_id in the test suite 2024-03-18T20:27:43.751 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-18T20:27:43.751 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-18T20:27:43.752 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.752 INFO:teuthology.orchestra.run.smithi165.stdout: # 1m isn't quite enough 2024-03-18T20:27:43.752 INFO:teuthology.orchestra.run.smithi165.stdout: mon_down_mkfs_grace = 2m 2024-03-18T20:27:43.752 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.752 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_filestore_osds = false 2024-03-18T20:27:43.752 INFO:teuthology.orchestra.run.smithi165.stdout: mon op complaint time = 120 2024-03-18T20:27:43.752 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-18T20:27:43.752 INFO:teuthology.orchestra.run.smithi165.stdout:[client] 2024-03-18T20:27:43.752 INFO:teuthology.orchestra.run.smithi165.stdout: rgw cache enabled = true 2024-03-18T20:27:43.752 INFO:teuthology.orchestra.run.smithi165.stdout: rgw enable ops log = true 2024-03-18T20:27:43.752 INFO:teuthology.orchestra.run.smithi165.stdout: rgw enable usage log = true 2024-03-18T20:27:43.752 INFO:teuthology.orchestra.run.smithi165.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-18T20:27:43.752 INFO:teuthology.orchestra.run.smithi165.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-18T20:27:43.752 INFO:teuthology.orchestra.run.smithi165.stdout: client mount timeout = 600 2024-03-18T20:27:43.752 INFO:teuthology.orchestra.run.smithi165.stdout: debug client = 20 2024-03-18T20:27:43.753 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-03-18T20:27:43.753 INFO:teuthology.orchestra.run.smithi165.stdout: pid file = "" 2024-03-18T20:27:43.753 INFO:teuthology.orchestra.run.smithi165.stdout: rados mon op timeout = 900 2024-03-18T20:27:43.753 INFO:teuthology.orchestra.run.smithi165.stdout: rados osd op timeout = 900 2024-03-18T20:27:43.753 INFO:teuthology.orchestra.run.smithi165.stdout: client snapdir = .snap 2024-03-18T20:27:43.753 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.a] 2024-03-18T20:27:43.753 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.b] 2024-03-18T20:27:43.753 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.c] 2024-03-18T20:27:43.753 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.a] 2024-03-18T20:27:43.753 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.c] 2024-03-18T20:27:43.753 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.b] 2024-03-18T20:27:43.753 INFO:teuthology.orchestra.run.smithi165.stdout:[mds.d] 2024-03-18T20:27:43.753 INFO:teuthology.orchestra.run.smithi165.stdout:[mds] 2024-03-18T20:27:43.753 INFO:teuthology.orchestra.run.smithi165.stdout: debug mds = 20 2024-03-18T20:27:43.753 INFO:teuthology.orchestra.run.smithi165.stdout: debug mds balancer = 20 2024-03-18T20:27:43.754 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-03-18T20:27:43.754 INFO:teuthology.orchestra.run.smithi165.stdout: mds debug frag = True 2024-03-18T20:27:43.754 INFO:teuthology.orchestra.run.smithi165.stdout: mds debug scatterstat = True 2024-03-18T20:27:43.754 INFO:teuthology.orchestra.run.smithi165.stdout: mds op complaint time = 180 2024-03-18T20:27:43.754 INFO:teuthology.orchestra.run.smithi165.stdout: mds verify scatter = True 2024-03-18T20:27:43.754 INFO:teuthology.orchestra.run.smithi165.stdout: osd op complaint time = 180 2024-03-18T20:27:43.754 INFO:teuthology.orchestra.run.smithi165.stdout: rados mon op timeout = 900 2024-03-18T20:27:43.754 INFO:teuthology.orchestra.run.smithi165.stdout: rados osd op timeout = 900 2024-03-18T20:27:43.756 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:27:43.756 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:27:43.815 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:27:43.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.976+0000 7faf753c4700 1 Processor -- start 2024-03-18T20:27:43.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.976+0000 7faf753c4700 1 -- start start 2024-03-18T20:27:43.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.977+0000 7faf753c4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7faf7015d8f0 0x7faf700b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:43.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.977+0000 7faf753c4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf700b35e0 0x7faf700b4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.977+0000 7faf753c4700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf700b1960 con 0x7faf70072050 2024-03-18T20:27:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.977+0000 7faf753c4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf700b1bb0 con 0x7faf7015d8f0 2024-03-18T20:27:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.977+0000 7faf753c4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf700adb70 con 0x7faf700b35e0 2024-03-18T20:27:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.977+0000 7faf6f7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7faf70072050 0x7faf7006a7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:50496/0 (socket says 172.21.15.1:50496) 2024-03-18T20:27:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.977+0000 7faf6f7fe700 1 -- 172.21.15.1:0/3538962703 learned_addr learned my addr 172.21.15.1:0/3538962703 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.977+0000 7faf6e7fc700 1 --2- 172.21.15.1:0/3538962703 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf700b35e0 0x7faf700b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.977+0000 7faf6effd700 1 --2- 172.21.15.1:0/3538962703 >> v2:172.21.15.165:3300/0 conn(0x7faf7015d8f0 0x7faf700b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.977+0000 7faf6dffb700 1 -- 172.21.15.1:0/3538962703 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931607825 0 0) 0x7faf700b1960 con 0x7faf70072050 2024-03-18T20:27:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6dffb700 1 -- 172.21.15.1:0/3538962703 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf580035b0 con 0x7faf70072050 2024-03-18T20:27:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6dffb700 1 -- 172.21.15.1:0/3538962703 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1047200833 0 0) 0x7faf580035b0 con 0x7faf70072050 2024-03-18T20:27:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6dffb700 1 -- 172.21.15.1:0/3538962703 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf700b35e0 msgr2=0x7faf700b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6dffb700 1 --2- 172.21.15.1:0/3538962703 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf700b35e0 0x7faf700b4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6dffb700 1 -- 172.21.15.1:0/3538962703 >> v2:172.21.15.165:3300/0 conn(0x7faf7015d8f0 msgr2=0x7faf700b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:43.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6dffb700 1 --2- 172.21.15.1:0/3538962703 >> v2:172.21.15.165:3300/0 conn(0x7faf7015d8f0 0x7faf700b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6dffb700 1 -- 172.21.15.1:0/3538962703 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf700ade50 con 0x7faf70072050 2024-03-18T20:27:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6dffb700 1 -- 172.21.15.1:0/3538962703 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7faf680019f0 con 0x7faf70072050 2024-03-18T20:27:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6dffb700 1 -- 172.21.15.1:0/3538962703 >> v1:172.21.15.1:6789/0 conn(0x7faf70072050 legacy=0x7faf7006a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6dffb700 1 --2- 172.21.15.1:0/3538962703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf58003a00 0x7faf58003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6dffb700 1 --2- 172.21.15.1:0/3538962703 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf58004850 0x7faf58004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6dffb700 1 --2- 172.21.15.1:0/3538962703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf580051d0 0x7faf5800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6dffb700 1 -- 172.21.15.1:0/3538962703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf700b1bb0 con 0x7faf58003a00 2024-03-18T20:27:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6dffb700 1 -- 172.21.15.1:0/3538962703 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf700adb70 con 0x7faf580051d0 2024-03-18T20:27:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6dffb700 1 -- 172.21.15.1:0/3538962703 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf58004650 con 0x7faf58004850 2024-03-18T20:27:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6effd700 1 --2- 172.21.15.1:0/3538962703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf58003a00 0x7faf58003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6e7fc700 1 --2- 172.21.15.1:0/3538962703 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf58004850 0x7faf58004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:43.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6f7fe700 1 --2- 172.21.15.1:0/3538962703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf580051d0 0x7faf5800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6effd700 1 -- 172.21.15.1:0/3538962703 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf58004850 msgr2=0x7faf58004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6effd700 1 --2- 172.21.15.1:0/3538962703 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf58004850 0x7faf58004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6effd700 1 -- 172.21.15.1:0/3538962703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf580051d0 msgr2=0x7faf5800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6effd700 1 --2- 172.21.15.1:0/3538962703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf580051d0 0x7faf5800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6effd700 1 -- 172.21.15.1:0/3538962703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf580035b0 con 0x7faf58003a00 2024-03-18T20:27:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6e7fc700 1 --2- 172.21.15.1:0/3538962703 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf58004850 0x7faf58004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6effd700 1 --2- 172.21.15.1:0/3538962703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf58003a00 0x7faf58003e20 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7faf64009b60 tx=0x7faf58005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=777749509301258e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6dffb700 1 -- 172.21.15.1:0/3538962703 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6400e6c0 con 0x7faf58003a00 2024-03-18T20:27:43.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6dffb700 1 -- 172.21.15.1:0/3538962703 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6400e820 con 0x7faf58003a00 2024-03-18T20:27:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.978+0000 7faf6dffb700 1 -- 172.21.15.1:0/3538962703 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6400eb90 con 0x7faf58003a00 2024-03-18T20:27:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.979+0000 7faf753c4700 1 -- 172.21.15.1:0/3538962703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf58003a00 msgr2=0x7faf58003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.979+0000 7faf753c4700 1 --2- 172.21.15.1:0/3538962703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf58003a00 0x7faf58003e20 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7faf64009b60 tx=0x7faf58005880 comp rx=0 tx=0).stop 2024-03-18T20:27:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.979+0000 7faf6f7fe700 1 -- 172.21.15.1:0/3538962703 reap_dead start 2024-03-18T20:27:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.979+0000 7faf753c4700 1 -- 172.21.15.1:0/3538962703 shutdown_connections 2024-03-18T20:27:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.979+0000 7faf753c4700 1 -- 172.21.15.1:0/3538962703 >> 172.21.15.1:0/3538962703 conn(0x7faf700b5a10 msgr2=0x7faf700b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.979+0000 7faf753c4700 1 -- 172.21.15.1:0/3538962703 shutdown_connections 2024-03-18T20:27:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.979+0000 7faf753c4700 1 -- 172.21.15.1:0/3538962703 wait complete. 2024-03-18T20:27:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.980+0000 7faf753c4700 1 Processor -- start 2024-03-18T20:27:43.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.980+0000 7faf753c4700 1 -- start start 2024-03-18T20:27:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.981+0000 7faf753c4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf58003a00 0x7faf701515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.981+0000 7faf753c4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf70155c60 0x7faf701560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.981+0000 7faf753c4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf701f68f0 0x7faf701f6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.981+0000 7faf753c4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf70151b30 con 0x7faf70155c60 2024-03-18T20:27:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.981+0000 7faf753c4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf701f72b0 con 0x7faf58003a00 2024-03-18T20:27:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.981+0000 7faf753c4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf70156710 con 0x7faf701f68f0 2024-03-18T20:27:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.981+0000 7faf6e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf70155c60 0x7faf701560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.981+0000 7faf6e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf70155c60 0x7faf701560e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60344/0 (socket says 172.21.15.1:60344) 2024-03-18T20:27:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.981+0000 7faf6e7fc700 1 -- 172.21.15.1:0/4086422476 learned_addr learned my addr 172.21.15.1:0/4086422476 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.981+0000 7faf6effd700 1 --2- 172.21.15.1:0/4086422476 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf58003a00 0x7faf701515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.981+0000 7faf6f7fe700 1 --2- 172.21.15.1:0/4086422476 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf701f68f0 0x7faf701f6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.982+0000 7faf6e7fc700 1 -- 172.21.15.1:0/4086422476 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf701f68f0 msgr2=0x7faf701f6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:43.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.982+0000 7faf6e7fc700 1 --2- 172.21.15.1:0/4086422476 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf701f68f0 0x7faf701f6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.982+0000 7faf6e7fc700 1 -- 172.21.15.1:0/4086422476 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf58003a00 msgr2=0x7faf701515f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.982+0000 7faf6e7fc700 1 --2- 172.21.15.1:0/4086422476 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf58003a00 0x7faf701515f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.982+0000 7faf6e7fc700 1 -- 172.21.15.1:0/4086422476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf701569f0 con 0x7faf70155c60 2024-03-18T20:27:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.982+0000 7faf6f7fe700 1 --2- 172.21.15.1:0/4086422476 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf701f68f0 0x7faf701f6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.982+0000 7faf6effd700 1 --2- 172.21.15.1:0/4086422476 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf58003a00 0x7faf701515f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.982+0000 7faf6e7fc700 1 --2- 172.21.15.1:0/4086422476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf70155c60 0x7faf701560e0 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 crypto rx=0x7faf600118d0 tx=0x7faf60011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.982+0000 7faf577fe700 1 -- 172.21.15.1:0/4086422476 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf60004220 con 0x7faf70155c60 2024-03-18T20:27:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.982+0000 7faf577fe700 1 -- 172.21.15.1:0/4086422476 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf60004380 con 0x7faf70155c60 2024-03-18T20:27:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.982+0000 7faf753c4700 1 -- 172.21.15.1:0/4086422476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf70158980 con 0x7faf70155c60 2024-03-18T20:27:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.982+0000 7faf577fe700 1 -- 172.21.15.1:0/4086422476 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf600036a0 con 0x7faf70155c60 2024-03-18T20:27:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.982+0000 7faf753c4700 1 -- 172.21.15.1:0/4086422476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf701585c0 con 0x7faf70155c60 2024-03-18T20:27:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.983+0000 7faf753c4700 1 -- 172.21.15.1:0/4086422476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf70158fd0 con 0x7faf70155c60 2024-03-18T20:27:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.984+0000 7faf577fe700 1 -- 172.21.15.1:0/4086422476 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 160) v1 ==== 88992+0+0 (secure 0 0 0) 0x7faf60003800 con 0x7faf70155c60 2024-03-18T20:27:43.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.984+0000 7faf577fe700 1 --2- 172.21.15.1:0/4086422476 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf5c06ab30 0x7faf5c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:43.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.984+0000 7faf577fe700 1 -- 172.21.15.1:0/4086422476 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7faf60089b60 con 0x7faf70155c60 2024-03-18T20:27:43.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.984+0000 7faf6effd700 1 --2- 172.21.15.1:0/4086422476 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf5c06ab30 0x7faf5c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:43.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.985+0000 7faf6effd700 1 --2- 172.21.15.1:0/4086422476 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf5c06ab30 0x7faf5c06cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7faf6400a010 tx=0x7faf64008030 comp rx=0 tx=0).ready entity=mgr.13401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:43.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:43.988+0000 7faf577fe700 1 -- 172.21.15.1:0/4086422476 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faf60054df0 con 0x7faf70155c60 2024-03-18T20:27:44.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.197+0000 7faf753c4700 1 -- 172.21.15.1:0/4086422476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf701587e0 con 0x7faf70155c60 2024-03-18T20:27:44.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.198+0000 7faf577fe700 1 -- 172.21.15.1:0/4086422476 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v275) v1 ==== 74+0+16970 (secure 0 0 0) 0x7faf701587e0 con 0x7faf70155c60 2024-03-18T20:27:44.202 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:44.203 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":275,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:27:42.143916+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:27:19.891286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"273","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-03-18T20:27:21.027315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"274","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3635558645":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2293377213":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/1796830610":"2024-03-19T20:27:42.143876+0000","172.21.15.1:6833/148356":"2024-03-19T20:27:42.143876+0000","172.21.15.1:6832/148356":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2068313472":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2668486132":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2560935779":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/82444607":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/1446581052":"2024-03-19T20:27:42.143876+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:27:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.200+0000 7faf753c4700 1 -- 172.21.15.1:0/4086422476 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf5c06ab30 msgr2=0x7faf5c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.200+0000 7faf753c4700 1 --2- 172.21.15.1:0/4086422476 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf5c06ab30 0x7faf5c06cff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7faf6400a010 tx=0x7faf64008030 comp rx=0 tx=0).stop 2024-03-18T20:27:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.200+0000 7faf753c4700 1 -- 172.21.15.1:0/4086422476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf70155c60 msgr2=0x7faf701560e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.200+0000 7faf753c4700 1 --2- 172.21.15.1:0/4086422476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf70155c60 0x7faf701560e0 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 crypto rx=0x7faf600118d0 tx=0x7faf60011c90 comp rx=0 tx=0).stop 2024-03-18T20:27:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.201+0000 7faf753c4700 1 -- 172.21.15.1:0/4086422476 shutdown_connections 2024-03-18T20:27:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.201+0000 7faf753c4700 1 --2- 172.21.15.1:0/4086422476 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf5c06ab30 0x7faf5c06cff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.201+0000 7faf753c4700 1 --2- 172.21.15.1:0/4086422476 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf701f68f0 0x7faf701f6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.201+0000 7faf753c4700 1 --2- 172.21.15.1:0/4086422476 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf58003a00 0x7faf701515f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.201+0000 7faf753c4700 1 --2- 172.21.15.1:0/4086422476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf70155c60 0x7faf701560e0 unknown :-1 s=CLOSED pgs=3138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.201+0000 7faf753c4700 1 -- 172.21.15.1:0/4086422476 >> 172.21.15.1:0/4086422476 conn(0x7faf700b5a10 msgr2=0x7faf700b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.201+0000 7faf753c4700 1 -- 172.21.15.1:0/4086422476 shutdown_connections 2024-03-18T20:27:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.201+0000 7faf753c4700 1 -- 172.21.15.1:0/4086422476 wait complete. 2024-03-18T20:27:44.219 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:27:19.891286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '273', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:27:21.027315+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '274', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:27:44.219 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:27:44.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.343+0000 7f30cc7a1700 1 Processor -- start 2024-03-18T20:27:44.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.343+0000 7f30cc7a1700 1 -- start start 2024-03-18T20:27:44.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.344+0000 7f30cc7a1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30c406cef0 0x7f30c40b4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:44.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.344+0000 7f30cc7a1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f30c41467a0 0x7f30c40b4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:44.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.344+0000 7f30cc7a1700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c406ca60 con 0x7f30c4146320 2024-03-18T20:27:44.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.344+0000 7f30cc7a1700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c406be50 con 0x7f30c41467a0 2024-03-18T20:27:44.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.344+0000 7f30cc7a1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c406c130 con 0x7f30c406cef0 2024-03-18T20:27:44.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.345+0000 7f30cad3e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f30c41467a0 0x7f30c40b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:44.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.345+0000 7f30c9d3c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30c406cef0 0x7f30c40b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:44.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.345+0000 7f30ca53d700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f30c4146320 0x7f30c4161a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:55608/0 (socket says 172.21.15.1:55608) 2024-03-18T20:27:44.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.345+0000 7f30cad3e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f30c41467a0 0x7f30c40b4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:44686/0 (socket says 172.21.15.1:44686) 2024-03-18T20:27:44.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.345+0000 7f30ca53d700 1 -- 172.21.15.1:0/3807455238 learned_addr learned my addr 172.21.15.1:0/3807455238 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:44.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.345+0000 7f30cad3e700 1 -- 172.21.15.1:0/3807455238 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30c406cef0 msgr2=0x7f30c40b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:44.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.345+0000 7f30cad3e700 1 --2- 172.21.15.1:0/3807455238 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30c406cef0 0x7f30c40b4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.345+0000 7f30cad3e700 1 -- 172.21.15.1:0/3807455238 >> v1:172.21.15.165:6789/0 conn(0x7f30c4146320 legacy=0x7f30c4161a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:44.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.345+0000 7f30cad3e700 1 -- 172.21.15.1:0/3807455238 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c406c410 con 0x7f30c41467a0 2024-03-18T20:27:44.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.345+0000 7f30c9d3c700 1 --2- 172.21.15.1:0/3807455238 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30c406cef0 0x7f30c40b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:27:44.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.345+0000 7f30cad3e700 1 --2- 172.21.15.1:0/3807455238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30c41467a0 0x7f30c40b4980 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f30bc009d90 tx=0x7f30bc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1b102a27186adee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:44.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.346+0000 7f30c953b700 1 -- 172.21.15.1:0/3807455238 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc00e070 con 0x7f30c41467a0 2024-03-18T20:27:44.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.346+0000 7f30c953b700 1 -- 172.21.15.1:0/3807455238 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30bc00bb20 con 0x7f30c41467a0 2024-03-18T20:27:44.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.346+0000 7f30c953b700 1 -- 172.21.15.1:0/3807455238 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc012430 con 0x7f30c41467a0 2024-03-18T20:27:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.346+0000 7f30cc7a1700 1 -- 172.21.15.1:0/3807455238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30c41467a0 msgr2=0x7f30c40b4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.346+0000 7f30cc7a1700 1 --2- 172.21.15.1:0/3807455238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30c41467a0 0x7f30c40b4980 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f30bc009d90 tx=0x7f30bc00b330 comp rx=0 tx=0).stop 2024-03-18T20:27:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.346+0000 7f30cc7a1700 1 -- 172.21.15.1:0/3807455238 shutdown_connections 2024-03-18T20:27:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.346+0000 7f30cc7a1700 1 --2- 172.21.15.1:0/3807455238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30c41467a0 0x7f30c40b4980 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.346+0000 7f30cc7a1700 1 --2- 172.21.15.1:0/3807455238 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30c406cef0 0x7f30c40b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.346+0000 7f30cc7a1700 1 -- 172.21.15.1:0/3807455238 >> 172.21.15.1:0/3807455238 conn(0x7f30c40b2e30 msgr2=0x7f30c40b3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.346+0000 7f30cc7a1700 1 -- 172.21.15.1:0/3807455238 shutdown_connections 2024-03-18T20:27:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.346+0000 7f30cc7a1700 1 -- 172.21.15.1:0/3807455238 wait complete. 2024-03-18T20:27:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.347+0000 7f30cc7a1700 1 Processor -- start 2024-03-18T20:27:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.347+0000 7f30cc7a1700 1 -- start start 2024-03-18T20:27:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.348+0000 7f30cc7a1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30c406cef0 0x7f30c4139900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:44.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.348+0000 7f30cc7a1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30c4146320 0x7f30c415fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:44.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.348+0000 7f30cc7a1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30c41467a0 0x7f30c415c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:44.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.348+0000 7f30cc7a1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c41f3400 con 0x7f30c41467a0 2024-03-18T20:27:44.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.348+0000 7f30cc7a1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c4140180 con 0x7f30c4146320 2024-03-18T20:27:44.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.348+0000 7f30cc7a1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c4139f90 con 0x7f30c406cef0 2024-03-18T20:27:44.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.348+0000 7f30cad3e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30c41467a0 0x7f30c415c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:44.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.348+0000 7f30cad3e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30c41467a0 0x7f30c415c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60376/0 (socket says 172.21.15.1:60376) 2024-03-18T20:27:44.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.348+0000 7f30cad3e700 1 -- 172.21.15.1:0/1845362031 learned_addr learned my addr 172.21.15.1:0/1845362031 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:44.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.348+0000 7f30ca53d700 1 --2- 172.21.15.1:0/1845362031 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30c406cef0 0x7f30c4139900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:44.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.348+0000 7f30c9d3c700 1 --2- 172.21.15.1:0/1845362031 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30c4146320 0x7f30c415fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:44.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.349+0000 7f30cad3e700 1 -- 172.21.15.1:0/1845362031 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30c406cef0 msgr2=0x7f30c4139900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:44.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.349+0000 7f30cad3e700 1 --2- 172.21.15.1:0/1845362031 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30c406cef0 0x7f30c4139900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.349+0000 7f30cad3e700 1 -- 172.21.15.1:0/1845362031 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30c4146320 msgr2=0x7f30c415fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:44.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.349+0000 7f30cad3e700 1 --2- 172.21.15.1:0/1845362031 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30c4146320 0x7f30c415fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.349+0000 7f30cad3e700 1 -- 172.21.15.1:0/1845362031 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b80094a0 con 0x7f30c41467a0 2024-03-18T20:27:44.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.349+0000 7f30ca53d700 1 --2- 172.21.15.1:0/1845362031 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30c406cef0 0x7f30c4139900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:44.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.349+0000 7f30c9d3c700 1 --2- 172.21.15.1:0/1845362031 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30c4146320 0x7f30c415fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:44.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.349+0000 7f30cad3e700 1 --2- 172.21.15.1:0/1845362031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30c41467a0 0x7f30c415c680 secure :-1 s=READY pgs=3141 cs=0 l=1 rev1=1 crypto rx=0x7f30bc014040 tx=0x7f30bc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:44.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.349+0000 7f30b6ffd700 1 -- 172.21.15.1:0/1845362031 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc012430 con 0x7f30c41467a0 2024-03-18T20:27:44.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.349+0000 7f30b6ffd700 1 -- 172.21.15.1:0/1845362031 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30bc020070 con 0x7f30c41467a0 2024-03-18T20:27:44.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.349+0000 7f30cc7a1700 1 -- 172.21.15.1:0/1845362031 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30bc009a70 con 0x7f30c41467a0 2024-03-18T20:27:44.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.349+0000 7f30b6ffd700 1 -- 172.21.15.1:0/1845362031 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc003ea0 con 0x7f30c41467a0 2024-03-18T20:27:44.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.349+0000 7f30cc7a1700 1 -- 172.21.15.1:0/1845362031 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c413a550 con 0x7f30c41467a0 2024-03-18T20:27:44.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.350+0000 7f30cc7a1700 1 -- 172.21.15.1:0/1845362031 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c4067dc0 con 0x7f30c41467a0 2024-03-18T20:27:44.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.350+0000 7f30b6ffd700 1 -- 172.21.15.1:0/1845362031 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 160) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f30bc004280 con 0x7f30c41467a0 2024-03-18T20:27:44.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.351+0000 7f30b6ffd700 1 --2- 172.21.15.1:0/1845362031 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30a806ab30 0x7f30a806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:44.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.351+0000 7f30b6ffd700 1 -- 172.21.15.1:0/1845362031 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f30bc022070 con 0x7f30c41467a0 2024-03-18T20:27:44.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.351+0000 7f30ca53d700 1 --2- 172.21.15.1:0/1845362031 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30a806ab30 0x7f30a806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:44.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.351+0000 7f30ca53d700 1 --2- 172.21.15.1:0/1845362031 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30a806ab30 0x7f30a806cff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f30c0007fd0 tx=0x7f30c0009040 comp rx=0 tx=0).ready entity=mgr.13401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:44.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.356+0000 7f30b6ffd700 1 -- 172.21.15.1:0/1845362031 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f30bc056670 con 0x7f30c41467a0 2024-03-18T20:27:44.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.557+0000 7f30cc7a1700 1 -- 172.21.15.1:0/1845362031 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f30c40d1f60 con 0x7f30c41467a0 2024-03-18T20:27:44.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.558+0000 7f30b6ffd700 1 -- 172.21.15.1:0/1845362031 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v275) v1 ==== 93+0+10 (secure 0 0 0) 0x7f30bc059c90 con 0x7f30c41467a0 2024-03-18T20:27:44.562 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:27:44.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.560+0000 7f30cc7a1700 1 -- 172.21.15.1:0/1845362031 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30a806ab30 msgr2=0x7f30a806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:44.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.560+0000 7f30cc7a1700 1 --2- 172.21.15.1:0/1845362031 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30a806ab30 0x7f30a806cff0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f30c0007fd0 tx=0x7f30c0009040 comp rx=0 tx=0).stop 2024-03-18T20:27:44.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.560+0000 7f30cc7a1700 1 -- 172.21.15.1:0/1845362031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30c41467a0 msgr2=0x7f30c415c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:44.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.560+0000 7f30cc7a1700 1 --2- 172.21.15.1:0/1845362031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30c41467a0 0x7f30c415c680 secure :-1 s=READY pgs=3141 cs=0 l=1 rev1=1 crypto rx=0x7f30bc014040 tx=0x7f30bc004ad0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.560+0000 7f30cc7a1700 1 -- 172.21.15.1:0/1845362031 shutdown_connections 2024-03-18T20:27:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.560+0000 7f30cc7a1700 1 --2- 172.21.15.1:0/1845362031 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30a806ab30 0x7f30a806cff0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.560+0000 7f30cc7a1700 1 --2- 172.21.15.1:0/1845362031 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30c406cef0 0x7f30c4139900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.560+0000 7f30cc7a1700 1 --2- 172.21.15.1:0/1845362031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30c41467a0 0x7f30c415c680 unknown :-1 s=CLOSED pgs=3141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.560+0000 7f30cc7a1700 1 --2- 172.21.15.1:0/1845362031 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30c4146320 0x7f30c415fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.560+0000 7f30cc7a1700 1 -- 172.21.15.1:0/1845362031 >> 172.21.15.1:0/1845362031 conn(0x7f30c40b2e30 msgr2=0x7f30c4141eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.560+0000 7f30cc7a1700 1 -- 172.21.15.1:0/1845362031 shutdown_connections 2024-03-18T20:27:44.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.560+0000 7f30cc7a1700 1 -- 172.21.15.1:0/1845362031 wait complete. 2024-03-18T20:27:44.576 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:27:44.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.700+0000 7fc4bc9cb700 1 Processor -- start 2024-03-18T20:27:44.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.700+0000 7fc4bc9cb700 1 -- start start 2024-03-18T20:27:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.701+0000 7fc4bc9cb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4b80b0590 0x7fc4b80b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.701+0000 7fc4bc9cb700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc4b80b5130 0x7fc4b80aef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.701+0000 7fc4bc9cb700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b815ed10 con 0x7fc4b815e930 2024-03-18T20:27:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.701+0000 7fc4bc9cb700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4b80b55c0 con 0x7fc4b80b5130 2024-03-18T20:27:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.701+0000 7fc4bc9cb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b809ad80 con 0x7fc4b80b0590 2024-03-18T20:27:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.701+0000 7fc4b6d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc4b80b5130 0x7fc4b80aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.701+0000 7fc4b6d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc4b80b5130 0x7fc4b80aef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54764/0 (socket says 172.21.15.1:54764) 2024-03-18T20:27:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.701+0000 7fc4b6d9d700 1 -- 172.21.15.1:0/710198898 learned_addr learned my addr 172.21.15.1:0/710198898 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:44.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.702+0000 7fc4b5d9b700 1 --2- 172.21.15.1:0/710198898 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4b80b0590 0x7fc4b80b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.702+0000 7fc4b6d9d700 1 -- 172.21.15.1:0/710198898 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4b80b0590 msgr2=0x7fc4b80b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.702+0000 7fc4b6d9d700 1 --2- 172.21.15.1:0/710198898 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4b80b0590 0x7fc4b80b4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.702+0000 7fc4b6d9d700 1 -- 172.21.15.1:0/710198898 >> v1:172.21.15.165:6789/0 conn(0x7fc4b815e930 legacy=0x7fc4b80b40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.702+0000 7fc4b6d9d700 1 -- 172.21.15.1:0/710198898 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b80b47f0 con 0x7fc4b80b5130 2024-03-18T20:27:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.702+0000 7fc4b5d9b700 1 --2- 172.21.15.1:0/710198898 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4b80b0590 0x7fc4b80b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.702+0000 7fc4b6d9d700 1 --2- 172.21.15.1:0/710198898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4b80b5130 0x7fc4b80aef90 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0009e50 tx=0x7fc4b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f35f0635fb8eaa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.702+0000 7fc4b559a700 1 -- 172.21.15.1:0/710198898 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b000e070 con 0x7fc4b80b5130 2024-03-18T20:27:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.702+0000 7fc4b559a700 1 -- 172.21.15.1:0/710198898 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b0003a20 con 0x7fc4b80b5130 2024-03-18T20:27:44.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.702+0000 7fc4b559a700 1 -- 172.21.15.1:0/710198898 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b0003dd0 con 0x7fc4b80b5130 2024-03-18T20:27:44.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.702+0000 7fc4bc9cb700 1 -- 172.21.15.1:0/710198898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4b80b5130 msgr2=0x7fc4b80aef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:44.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.702+0000 7fc4bc9cb700 1 --2- 172.21.15.1:0/710198898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4b80b5130 0x7fc4b80aef90 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0009e50 tx=0x7fc4b000b330 comp rx=0 tx=0).stop 2024-03-18T20:27:44.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.703+0000 7fc4bc9cb700 1 -- 172.21.15.1:0/710198898 shutdown_connections 2024-03-18T20:27:44.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.703+0000 7fc4bc9cb700 1 --2- 172.21.15.1:0/710198898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4b80b5130 0x7fc4b80aef90 unknown :-1 s=CLOSED pgs=3142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.703+0000 7fc4bc9cb700 1 --2- 172.21.15.1:0/710198898 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4b80b0590 0x7fc4b80b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.703+0000 7fc4bc9cb700 1 -- 172.21.15.1:0/710198898 >> 172.21.15.1:0/710198898 conn(0x7fc4b80b2de0 msgr2=0x7fc4b80b31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:44.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.703+0000 7fc4bc9cb700 1 -- 172.21.15.1:0/710198898 shutdown_connections 2024-03-18T20:27:44.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.703+0000 7fc4bc9cb700 1 -- 172.21.15.1:0/710198898 wait complete. 2024-03-18T20:27:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.704+0000 7fc4bc9cb700 1 Processor -- start 2024-03-18T20:27:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.704+0000 7fc4bc9cb700 1 -- start start 2024-03-18T20:27:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4bc9cb700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4b80b0590 0x7fc4b809dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4bc9cb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4b80b5130 0x7fc4b8153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4bc9cb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4b815e930 0x7fc4b809b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4bc9cb700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b809c0a0 con 0x7fc4b815e930 2024-03-18T20:27:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4bc9cb700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b809e4f0 con 0x7fc4b80b0590 2024-03-18T20:27:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4bc9cb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b809e7a0 con 0x7fc4b80b5130 2024-03-18T20:27:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4b6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4b815e930 0x7fc4b809b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4b6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4b815e930 0x7fc4b809b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54776/0 (socket says 172.21.15.1:54776) 2024-03-18T20:27:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4b6d9d700 1 -- 172.21.15.1:0/1314723012 learned_addr learned my addr 172.21.15.1:0/1314723012 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4b5d9b700 1 --2- 172.21.15.1:0/1314723012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4b80b5130 0x7fc4b8153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:44.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4b659c700 1 --2- 172.21.15.1:0/1314723012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4b80b0590 0x7fc4b809dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4b6d9d700 1 -- 172.21.15.1:0/1314723012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4b80b5130 msgr2=0x7fc4b8153c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4b6d9d700 1 --2- 172.21.15.1:0/1314723012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4b80b5130 0x7fc4b8153c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4b6d9d700 1 -- 172.21.15.1:0/1314723012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4b80b0590 msgr2=0x7fc4b809dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4b6d9d700 1 --2- 172.21.15.1:0/1314723012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4b80b0590 0x7fc4b809dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4b6d9d700 1 -- 172.21.15.1:0/1314723012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b0009b30 con 0x7fc4b815e930 2024-03-18T20:27:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4b5d9b700 1 --2- 172.21.15.1:0/1314723012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4b80b5130 0x7fc4b8153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4b659c700 1 --2- 172.21.15.1:0/1314723012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4b80b0590 0x7fc4b809dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4b6d9d700 1 --2- 172.21.15.1:0/1314723012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4b815e930 0x7fc4b809b240 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8072470 tx=0x7fc4b0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4a6ffd700 1 -- 172.21.15.1:0/1314723012 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b0004440 con 0x7fc4b815e930 2024-03-18T20:27:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4a6ffd700 1 -- 172.21.15.1:0/1314723012 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b00045a0 con 0x7fc4b815e930 2024-03-18T20:27:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4bc9cb700 1 -- 172.21.15.1:0/1314723012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b809ea80 con 0x7fc4b815e930 2024-03-18T20:27:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.705+0000 7fc4a6ffd700 1 -- 172.21.15.1:0/1314723012 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b0004950 con 0x7fc4b815e930 2024-03-18T20:27:44.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.706+0000 7fc4bc9cb700 1 -- 172.21.15.1:0/1314723012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b8158b40 con 0x7fc4b815e930 2024-03-18T20:27:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.706+0000 7fc4bc9cb700 1 -- 172.21.15.1:0/1314723012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b81589a0 con 0x7fc4b815e930 2024-03-18T20:27:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.707+0000 7fc4a6ffd700 1 -- 172.21.15.1:0/1314723012 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 160) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fc4b0012430 con 0x7fc4b815e930 2024-03-18T20:27:44.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.707+0000 7fc4a6ffd700 1 --2- 172.21.15.1:0/1314723012 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc49806ab30 0x7fc49806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.707+0000 7fc4a6ffd700 1 -- 172.21.15.1:0/1314723012 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7fc4b008a880 con 0x7fc4b815e930 2024-03-18T20:27:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.707+0000 7fc4b659c700 1 --2- 172.21.15.1:0/1314723012 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc49806ab30 0x7fc49806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:44.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.708+0000 7fc4b659c700 1 --2- 172.21.15.1:0/1314723012 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc49806ab30 0x7fc49806cff0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009930 tx=0x7fc4ac008040 comp rx=0 tx=0).ready entity=mgr.13401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:44.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.712+0000 7fc4a6ffd700 1 -- 172.21.15.1:0/1314723012 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc4b81589a0 con 0x7fc4b815e930 2024-03-18T20:27:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.913+0000 7fc4bc9cb700 1 -- 172.21.15.1:0/1314723012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc4b815ed10 con 0x7fc4b815e930 2024-03-18T20:27:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.913+0000 7fc4a6ffd700 1 -- 172.21.15.1:0/1314723012 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v275) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc4b815ed10 con 0x7fc4b815e930 2024-03-18T20:27:44.917 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:27:44.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.915+0000 7fc4bc9cb700 1 -- 172.21.15.1:0/1314723012 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc49806ab30 msgr2=0x7fc49806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.915+0000 7fc4bc9cb700 1 --2- 172.21.15.1:0/1314723012 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc49806ab30 0x7fc49806cff0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009930 tx=0x7fc4ac008040 comp rx=0 tx=0).stop 2024-03-18T20:27:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.915+0000 7fc4bc9cb700 1 -- 172.21.15.1:0/1314723012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4b815e930 msgr2=0x7fc4b809b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.915+0000 7fc4bc9cb700 1 --2- 172.21.15.1:0/1314723012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4b815e930 0x7fc4b809b240 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8072470 tx=0x7fc4b0004ad0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.915+0000 7fc4bc9cb700 1 -- 172.21.15.1:0/1314723012 shutdown_connections 2024-03-18T20:27:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.915+0000 7fc4bc9cb700 1 --2- 172.21.15.1:0/1314723012 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc49806ab30 0x7fc49806cff0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.915+0000 7fc4bc9cb700 1 --2- 172.21.15.1:0/1314723012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4b815e930 0x7fc4b809b240 unknown :-1 s=CLOSED pgs=3143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.915+0000 7fc4bc9cb700 1 --2- 172.21.15.1:0/1314723012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4b80b0590 0x7fc4b809dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.915+0000 7fc4bc9cb700 1 --2- 172.21.15.1:0/1314723012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc4b80b5130 0x7fc4b8153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.915+0000 7fc4bc9cb700 1 -- 172.21.15.1:0/1314723012 >> 172.21.15.1:0/1314723012 conn(0x7fc4b80b2de0 msgr2=0x7fc4b80af8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.915+0000 7fc4bc9cb700 1 -- 172.21.15.1:0/1314723012 shutdown_connections 2024-03-18T20:27:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:44.915+0000 7fc4bc9cb700 1 -- 172.21.15.1:0/1314723012 wait complete. 2024-03-18T20:27:44.931 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:27:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.056+0000 7f0029737700 1 Processor -- start 2024-03-18T20:27:45.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.056+0000 7f0029737700 1 -- start start 2024-03-18T20:27:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.056+0000 7f0029737700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f00240cf200 0x7f002415c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.056+0000 7f0029737700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f002415b6c0 0x7f00240cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.056+0000 7f0029737700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002415ca40 con 0x7f00240cbc90 2024-03-18T20:27:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.056+0000 7f0029737700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00240cce30 con 0x7f00240cf200 2024-03-18T20:27:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.056+0000 7f0029737700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00240cfb20 con 0x7f002415b6c0 2024-03-18T20:27:45.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.057+0000 7f0022ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f00240cf200 0x7f002415c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.057+0000 7f0022ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f00240cf200 0x7f002415c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54784/0 (socket says 172.21.15.1:54784) 2024-03-18T20:27:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.057+0000 7f0022ffd700 1 -- 172.21.15.1:0/1341371100 learned_addr learned my addr 172.21.15.1:0/1341371100 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.057+0000 7f00227fc700 1 --2- 172.21.15.1:0/1341371100 >> v2:172.21.15.165:3300/0 conn(0x7f002415b6c0 0x7f00240cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.057+0000 7f0022ffd700 1 -- 172.21.15.1:0/1341371100 >> v2:172.21.15.165:3300/0 conn(0x7f002415b6c0 msgr2=0x7f00240cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.057+0000 7f0022ffd700 1 --2- 172.21.15.1:0/1341371100 >> v2:172.21.15.165:3300/0 conn(0x7f002415b6c0 0x7f00240cf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.057+0000 7f0022ffd700 1 -- 172.21.15.1:0/1341371100 >> v1:172.21.15.165:6789/0 conn(0x7f00240cbc90 legacy=0x7f002415eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.057+0000 7f0022ffd700 1 -- 172.21.15.1:0/1341371100 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00240cc220 con 0x7f00240cf200 2024-03-18T20:27:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.057+0000 7f0022ffd700 1 --2- 172.21.15.1:0/1341371100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00240cf200 0x7f002415c500 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7f0024072ac0 tx=0x7f0014009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=164f58af8d51c9c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.057+0000 7f0021ffb700 1 -- 172.21.15.1:0/1341371100 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0014011830 con 0x7f00240cf200 2024-03-18T20:27:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.057+0000 7f0021ffb700 1 -- 172.21.15.1:0/1341371100 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0014011990 con 0x7f00240cf200 2024-03-18T20:27:45.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.057+0000 7f0021ffb700 1 -- 172.21.15.1:0/1341371100 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0014011d20 con 0x7f00240cf200 2024-03-18T20:27:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.058+0000 7f0029737700 1 -- 172.21.15.1:0/1341371100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00240cf200 msgr2=0x7f002415c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.058+0000 7f0029737700 1 --2- 172.21.15.1:0/1341371100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00240cf200 0x7f002415c500 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7f0024072ac0 tx=0x7f0014009bd0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.058+0000 7f0029737700 1 -- 172.21.15.1:0/1341371100 shutdown_connections 2024-03-18T20:27:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.058+0000 7f0029737700 1 --2- 172.21.15.1:0/1341371100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00240cf200 0x7f002415c500 unknown :-1 s=CLOSED pgs=3144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.058+0000 7f0029737700 1 --2- 172.21.15.1:0/1341371100 >> v2:172.21.15.165:3300/0 conn(0x7f002415b6c0 0x7f00240cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.058+0000 7f0029737700 1 -- 172.21.15.1:0/1341371100 >> 172.21.15.1:0/1341371100 conn(0x7f00240d1180 msgr2=0x7f00240d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.058+0000 7f0029737700 1 -- 172.21.15.1:0/1341371100 shutdown_connections 2024-03-18T20:27:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.058+0000 7f0029737700 1 -- 172.21.15.1:0/1341371100 wait complete. 2024-03-18T20:27:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.059+0000 7f0029737700 1 Processor -- start 2024-03-18T20:27:45.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.059+0000 7f0029737700 1 -- start start 2024-03-18T20:27:45.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.060+0000 7f0029737700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00240cbc90 0x7f0024150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:45.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.060+0000 7f0029737700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00240cf200 0x7f00240c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:45.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.060+0000 7f0029737700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f002415b6c0 0x7f00240b2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:45.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.060+0000 7f0029737700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00240c95f0 con 0x7f00240cbc90 2024-03-18T20:27:45.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.060+0000 7f0029737700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00240c5cf0 con 0x7f00240cf200 2024-03-18T20:27:45.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.060+0000 7f0029737700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00240c5fd0 con 0x7f002415b6c0 2024-03-18T20:27:45.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.060+0000 7f0022ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00240cbc90 0x7f0024150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:45.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.060+0000 7f0022ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00240cbc90 0x7f0024150230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54798/0 (socket says 172.21.15.1:54798) 2024-03-18T20:27:45.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.060+0000 7f0022ffd700 1 -- 172.21.15.1:0/1331722531 learned_addr learned my addr 172.21.15.1:0/1331722531 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:45.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.060+0000 7f00237fe700 1 --2- 172.21.15.1:0/1331722531 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f002415b6c0 0x7f00240b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:45.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.060+0000 7f00227fc700 1 --2- 172.21.15.1:0/1331722531 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00240cf200 0x7f00240c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:45.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.060+0000 7f0022ffd700 1 -- 172.21.15.1:0/1331722531 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f002415b6c0 msgr2=0x7f00240b2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:45.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.060+0000 7f0022ffd700 1 --2- 172.21.15.1:0/1331722531 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f002415b6c0 0x7f00240b2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.060+0000 7f0022ffd700 1 -- 172.21.15.1:0/1331722531 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00240cf200 msgr2=0x7f00240c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:45.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.060+0000 7f0022ffd700 1 --2- 172.21.15.1:0/1331722531 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00240cf200 0x7f00240c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.060+0000 7f0022ffd700 1 -- 172.21.15.1:0/1331722531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0014009550 con 0x7f00240cbc90 2024-03-18T20:27:45.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.060+0000 7f00237fe700 1 --2- 172.21.15.1:0/1331722531 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f002415b6c0 0x7f00240b2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:45.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.060+0000 7f0022ffd700 1 --2- 172.21.15.1:0/1331722531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00240cbc90 0x7f0024150230 secure :-1 s=READY pgs=3145 cs=0 l=1 rev1=1 crypto rx=0x7f0014009c10 tx=0x7f0014004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:45.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.061+0000 7f000b7fe700 1 -- 172.21.15.1:0/1331722531 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00140036a0 con 0x7f00240cbc90 2024-03-18T20:27:45.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.061+0000 7f000b7fe700 1 -- 172.21.15.1:0/1331722531 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001401e070 con 0x7f00240cbc90 2024-03-18T20:27:45.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.061+0000 7f0029737700 1 -- 172.21.15.1:0/1331722531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00240c62b0 con 0x7f00240cbc90 2024-03-18T20:27:45.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.061+0000 7f000b7fe700 1 -- 172.21.15.1:0/1331722531 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0014003a60 con 0x7f00240cbc90 2024-03-18T20:27:45.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.061+0000 7f0029737700 1 -- 172.21.15.1:0/1331722531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00241f14e0 con 0x7f00240cbc90 2024-03-18T20:27:45.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.062+0000 7f0029737700 1 -- 172.21.15.1:0/1331722531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00240d0ca0 con 0x7f00240cbc90 2024-03-18T20:27:45.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.062+0000 7f000b7fe700 1 -- 172.21.15.1:0/1331722531 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 160) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f0014003bc0 con 0x7f00240cbc90 2024-03-18T20:27:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.062+0000 7f000b7fe700 1 --2- 172.21.15.1:0/1331722531 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f001006ab30 0x7f001006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.063+0000 7f000b7fe700 1 -- 172.21.15.1:0/1331722531 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f001408a2f0 con 0x7f00240cbc90 2024-03-18T20:27:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.063+0000 7f00227fc700 1 --2- 172.21.15.1:0/1331722531 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f001006ab30 0x7f001006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:45.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.063+0000 7f00227fc700 1 --2- 172.21.15.1:0/1331722531 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f001006ab30 0x7f001006cff0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f00240d10d0 tx=0x7f0018008040 comp rx=0 tx=0).ready entity=mgr.13401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:45.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.067+0000 7f000b7fe700 1 -- 172.21.15.1:0/1331722531 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f00140553f0 con 0x7f00240cbc90 2024-03-18T20:27:45.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.159+0000 7f000b7fe700 1 -- 172.21.15.1:0/1331722531 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 161) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f00140557c0 con 0x7f00240cbc90 2024-03-18T20:27:45.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.272+0000 7f0029737700 1 -- 172.21.15.1:0/1331722531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0024067dc0 con 0x7f00240cbc90 2024-03-18T20:27:45.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.273+0000 7f000b7fe700 1 -- 172.21.15.1:0/1331722531 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v275) v1 ==== 100+0+11 (secure 0 0 0) 0x7f001406ea20 con 0x7f00240cbc90 2024-03-18T20:27:45.277 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:27:45.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.275+0000 7f0029737700 1 -- 172.21.15.1:0/1331722531 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f001006ab30 msgr2=0x7f001006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:45.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.275+0000 7f0029737700 1 --2- 172.21.15.1:0/1331722531 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f001006ab30 0x7f001006cff0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f00240d10d0 tx=0x7f0018008040 comp rx=0 tx=0).stop 2024-03-18T20:27:45.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.275+0000 7f0029737700 1 -- 172.21.15.1:0/1331722531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00240cbc90 msgr2=0x7f0024150230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:45.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.275+0000 7f0029737700 1 --2- 172.21.15.1:0/1331722531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00240cbc90 0x7f0024150230 secure :-1 s=READY pgs=3145 cs=0 l=1 rev1=1 crypto rx=0x7f0014009c10 tx=0x7f0014004750 comp rx=0 tx=0).stop 2024-03-18T20:27:45.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.275+0000 7f0029737700 1 -- 172.21.15.1:0/1331722531 shutdown_connections 2024-03-18T20:27:45.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.275+0000 7f0029737700 1 --2- 172.21.15.1:0/1331722531 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f001006ab30 0x7f001006cff0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.275+0000 7f0029737700 1 --2- 172.21.15.1:0/1331722531 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f002415b6c0 0x7f00240b2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.275+0000 7f0029737700 1 --2- 172.21.15.1:0/1331722531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00240cbc90 0x7f0024150230 unknown :-1 s=CLOSED pgs=3145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.275+0000 7f0029737700 1 --2- 172.21.15.1:0/1331722531 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f00240cf200 0x7f00240c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.275+0000 7f0029737700 1 -- 172.21.15.1:0/1331722531 >> 172.21.15.1:0/1331722531 conn(0x7f00240d1180 msgr2=0x7f00240b15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:45.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.275+0000 7f0029737700 1 -- 172.21.15.1:0/1331722531 shutdown_connections 2024-03-18T20:27:45.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.275+0000 7f0029737700 1 -- 172.21.15.1:0/1331722531 wait complete. 2024-03-18T20:27:45.291 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T20:27:45.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.413+0000 7fba3bcfd700 1 Processor -- start 2024-03-18T20:27:45.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.414+0000 7fba3bcfd700 1 -- start start 2024-03-18T20:27:45.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.414+0000 7fba3bcfd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fba34159fe0 0x7fba3406e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:45.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.414+0000 7fba3bcfd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fba3415a3c0 0x7fba3406ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:45.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.414+0000 7fba3bcfd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba340b3e50 0x7fba340b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:45.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.414+0000 7fba3bcfd700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba3406faa0 con 0x7fba3415a3c0 2024-03-18T20:27:45.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.414+0000 7fba3bcfd700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba340b4820 con 0x7fba34159fe0 2024-03-18T20:27:45.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.414+0000 7fba3bcfd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba340723d0 con 0x7fba340b3e50 2024-03-18T20:27:45.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.415+0000 7fba39298700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fba3415a3c0 0x7fba3406ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:45.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.415+0000 7fba39298700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fba3415a3c0 0x7fba3406ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54800/0 (socket says 172.21.15.1:54800) 2024-03-18T20:27:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.415+0000 7fba39298700 1 -- 172.21.15.1:0/1929605329 learned_addr learned my addr 172.21.15.1:0/1929605329 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.415+0000 7fba39a99700 1 --2- 172.21.15.1:0/1929605329 >> v2:172.21.15.165:3300/0 conn(0x7fba34159fe0 0x7fba3406e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.415+0000 7fba39298700 1 -- 172.21.15.1:0/1929605329 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba340b3e50 msgr2=0x7fba340b42e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T20:27:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.415+0000 7fba39298700 1 --2- 172.21.15.1:0/1929605329 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba340b3e50 0x7fba340b42e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.415+0000 7fba39298700 1 -- 172.21.15.1:0/1929605329 >> v2:172.21.15.165:3300/0 conn(0x7fba34159fe0 msgr2=0x7fba3406e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.415+0000 7fba39298700 1 --2- 172.21.15.1:0/1929605329 >> v2:172.21.15.165:3300/0 conn(0x7fba34159fe0 0x7fba3406e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.415+0000 7fba39298700 1 -- 172.21.15.1:0/1929605329 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3406f200 con 0x7fba3415a3c0 2024-03-18T20:27:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.415+0000 7fba39298700 1 --2- 172.21.15.1:0/1929605329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3415a3c0 0x7fba3406ebd0 secure :-1 s=READY pgs=3146 cs=0 l=1 rev1=1 crypto rx=0x7fba300098f0 tx=0x7fba30009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c929edb49ac57282 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.415+0000 7fba38a97700 1 -- 172.21.15.1:0/1929605329 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3000b860 con 0x7fba3415a3c0 2024-03-18T20:27:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.415+0000 7fba38a97700 1 -- 172.21.15.1:0/1929605329 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3000b9c0 con 0x7fba3415a3c0 2024-03-18T20:27:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.415+0000 7fba38a97700 1 -- 172.21.15.1:0/1929605329 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3000bd50 con 0x7fba3415a3c0 2024-03-18T20:27:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.416+0000 7fba3bcfd700 1 -- 172.21.15.1:0/1929605329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3415a3c0 msgr2=0x7fba3406ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:45.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.416+0000 7fba3bcfd700 1 --2- 172.21.15.1:0/1929605329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3415a3c0 0x7fba3406ebd0 secure :-1 s=READY pgs=3146 cs=0 l=1 rev1=1 crypto rx=0x7fba300098f0 tx=0x7fba30009c00 comp rx=0 tx=0).stop 2024-03-18T20:27:45.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.416+0000 7fba3bcfd700 1 -- 172.21.15.1:0/1929605329 shutdown_connections 2024-03-18T20:27:45.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.416+0000 7fba3bcfd700 1 --2- 172.21.15.1:0/1929605329 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba340b3e50 0x7fba340b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.416+0000 7fba3bcfd700 1 --2- 172.21.15.1:0/1929605329 >> v2:172.21.15.165:3300/0 conn(0x7fba34159fe0 0x7fba3406e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.416+0000 7fba3bcfd700 1 --2- 172.21.15.1:0/1929605329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3415a3c0 0x7fba3406ebd0 unknown :-1 s=CLOSED pgs=3146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.416+0000 7fba3bcfd700 1 -- 172.21.15.1:0/1929605329 >> 172.21.15.1:0/1929605329 conn(0x7fba3406c360 msgr2=0x7fba34099a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:45.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.416+0000 7fba3bcfd700 1 -- 172.21.15.1:0/1929605329 shutdown_connections 2024-03-18T20:27:45.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.416+0000 7fba3bcfd700 1 -- 172.21.15.1:0/1929605329 wait complete. 2024-03-18T20:27:45.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.417+0000 7fba3bcfd700 1 Processor -- start 2024-03-18T20:27:45.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.417+0000 7fba3bcfd700 1 -- start start 2024-03-18T20:27:45.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.418+0000 7fba3bcfd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba340b3e50 0x7fba340b1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:45.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.418+0000 7fba3bcfd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba34159fe0 0x7fba3413c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:45.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.418+0000 7fba3bcfd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3415a3c0 0x7fba3413cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:45.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.418+0000 7fba3bcfd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba341f1450 con 0x7fba3415a3c0 2024-03-18T20:27:45.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.418+0000 7fba3bcfd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3413d450 con 0x7fba34159fe0 2024-03-18T20:27:45.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.418+0000 7fba3bcfd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3414fd50 con 0x7fba340b3e50 2024-03-18T20:27:45.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.418+0000 7fba3a29a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3415a3c0 0x7fba3413cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:45.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.418+0000 7fba3a29a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3415a3c0 0x7fba3413cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54806/0 (socket says 172.21.15.1:54806) 2024-03-18T20:27:45.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.418+0000 7fba3a29a700 1 -- 172.21.15.1:0/2279515521 learned_addr learned my addr 172.21.15.1:0/2279515521 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:45.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.418+0000 7fba39298700 1 --2- 172.21.15.1:0/2279515521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba34159fe0 0x7fba3413c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:45.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.418+0000 7fba39a99700 1 --2- 172.21.15.1:0/2279515521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba340b3e50 0x7fba340b1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:45.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.418+0000 7fba3a29a700 1 -- 172.21.15.1:0/2279515521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba340b3e50 msgr2=0x7fba340b1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:45.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.418+0000 7fba3a29a700 1 --2- 172.21.15.1:0/2279515521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba340b3e50 0x7fba340b1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.418+0000 7fba3a29a700 1 -- 172.21.15.1:0/2279515521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba34159fe0 msgr2=0x7fba3413c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:45.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.418+0000 7fba3a29a700 1 --2- 172.21.15.1:0/2279515521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba34159fe0 0x7fba3413c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.418+0000 7fba3a29a700 1 -- 172.21.15.1:0/2279515521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba30009550 con 0x7fba3415a3c0 2024-03-18T20:27:45.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.418+0000 7fba39a99700 1 --2- 172.21.15.1:0/2279515521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba340b3e50 0x7fba340b1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:45.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.418+0000 7fba3a29a700 1 --2- 172.21.15.1:0/2279515521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3415a3c0 0x7fba3413cf10 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00b780 tx=0x7fba2c00ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:45.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.418+0000 7fba2a7fc700 1 -- 172.21.15.1:0/2279515521 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c010860 con 0x7fba3415a3c0 2024-03-18T20:27:45.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.419+0000 7fba2a7fc700 1 -- 172.21.15.1:0/2279515521 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2c0109c0 con 0x7fba3415a3c0 2024-03-18T20:27:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.419+0000 7fba3bcfd700 1 -- 172.21.15.1:0/2279515521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba341f0ab0 con 0x7fba3415a3c0 2024-03-18T20:27:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.419+0000 7fba2a7fc700 1 -- 172.21.15.1:0/2279515521 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c010d70 con 0x7fba3415a3c0 2024-03-18T20:27:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.419+0000 7fba3bcfd700 1 -- 172.21.15.1:0/2279515521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba341f1070 con 0x7fba3415a3c0 2024-03-18T20:27:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.419+0000 7fba3bcfd700 1 -- 172.21.15.1:0/2279515521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba34067dc0 con 0x7fba3415a3c0 2024-03-18T20:27:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.420+0000 7fba2a7fc700 1 -- 172.21.15.1:0/2279515521 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 161) v1 ==== 89257+0+0 (secure 0 0 0) 0x7fba2c00f3b0 con 0x7fba3415a3c0 2024-03-18T20:27:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.420+0000 7fba2a7fc700 1 --2- 172.21.15.1:0/2279515521 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fba1806ae00 0x7fba1806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.420+0000 7fba2a7fc700 1 -- 172.21.15.1:0/2279515521 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7fba2c089c70 con 0x7fba3415a3c0 2024-03-18T20:27:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.420+0000 7fba39a99700 1 --2- 172.21.15.1:0/2279515521 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fba1806ae00 0x7fba1806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.421+0000 7fba39a99700 1 --2- 172.21.15.1:0/2279515521 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fba1806ae00 0x7fba1806d2c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fba3414ffa0 tx=0x7fba24008040 comp rx=0 tx=0).ready entity=mgr.13401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:45.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.425+0000 7fba2a7fc700 1 -- 172.21.15.1:0/2279515521 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fba2c0548f0 con 0x7fba3415a3c0 2024-03-18T20:27:45.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.644+0000 7fba3bcfd700 1 -- 172.21.15.1:0/2279515521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fba340727f0 con 0x7fba3415a3c0 2024-03-18T20:27:45.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.644+0000 7fba2a7fc700 1 -- 172.21.15.1:0/2279515521 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v275) v1 ==== 81+0+566 (secure 0 0 0) 0x7fba2c057f10 con 0x7fba3415a3c0 2024-03-18T20:27:45.649 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3635558645 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:45.649 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2293377213 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:45.649 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1796830610 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:45.649 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6833/148356 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:45.649 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6832/148356 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:45.649 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2068313472 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:45.649 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2668486132 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:45.649 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2560935779 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:45.649 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/82444607 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:45.649 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1446581052 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:45.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.647+0000 7fba3bcfd700 1 -- 172.21.15.1:0/2279515521 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fba1806ae00 msgr2=0x7fba1806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:45.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.647+0000 7fba3bcfd700 1 --2- 172.21.15.1:0/2279515521 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fba1806ae00 0x7fba1806d2c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fba3414ffa0 tx=0x7fba24008040 comp rx=0 tx=0).stop 2024-03-18T20:27:45.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.647+0000 7fba3bcfd700 1 -- 172.21.15.1:0/2279515521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3415a3c0 msgr2=0x7fba3413cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:45.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.647+0000 7fba3bcfd700 1 --2- 172.21.15.1:0/2279515521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3415a3c0 0x7fba3413cf10 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00b780 tx=0x7fba2c00ba90 comp rx=0 tx=0).stop 2024-03-18T20:27:45.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.647+0000 7fba3bcfd700 1 -- 172.21.15.1:0/2279515521 shutdown_connections 2024-03-18T20:27:45.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.647+0000 7fba3bcfd700 1 --2- 172.21.15.1:0/2279515521 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fba1806ae00 0x7fba1806d2c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.647+0000 7fba3bcfd700 1 --2- 172.21.15.1:0/2279515521 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fba340b3e50 0x7fba340b1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.647+0000 7fba3bcfd700 1 --2- 172.21.15.1:0/2279515521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba3415a3c0 0x7fba3413cf10 unknown :-1 s=CLOSED pgs=3147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.647+0000 7fba3bcfd700 1 --2- 172.21.15.1:0/2279515521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba34159fe0 0x7fba3413c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.647+0000 7fba3bcfd700 1 -- 172.21.15.1:0/2279515521 >> 172.21.15.1:0/2279515521 conn(0x7fba3406c360 msgr2=0x7fba34159210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:45.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.647+0000 7fba3bcfd700 1 -- 172.21.15.1:0/2279515521 shutdown_connections 2024-03-18T20:27:45.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.647+0000 7fba3bcfd700 1 -- 172.21.15.1:0/2279515521 wait complete. 2024-03-18T20:27:45.652 INFO:teuthology.orchestra.run.smithi001.stderr:listed 10 entries 2024-03-18T20:27:45.663 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:27:45.663 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:27:45.682 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:27:45.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.843+0000 7f33399a6700 1 Processor -- start 2024-03-18T20:27:45.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.843+0000 7f33399a6700 1 -- start start 2024-03-18T20:27:45.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.844+0000 7f33399a6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f333415d8c0 0x7f33340b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:45.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.844+0000 7f33399a6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f33340d3820 0x7f33340b0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:45.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.844+0000 7f33399a6700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33340b00a0 con 0x7f33340b3c30 2024-03-18T20:27:45.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.844+0000 7f33399a6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33340d1f40 con 0x7f333415d8c0 2024-03-18T20:27:45.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.844+0000 7f33399a6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33340b4260 con 0x7f33340d3820 2024-03-18T20:27:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.844+0000 7f3332ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f333415d8c0 0x7f33340b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.844+0000 7f3332ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f333415d8c0 0x7f33340b36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54816/0 (socket says 172.21.15.1:54816) 2024-03-18T20:27:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.844+0000 7f3332ffd700 1 -- 172.21.15.1:0/665411250 learned_addr learned my addr 172.21.15.1:0/665411250 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.844+0000 7f3332ffd700 1 -- 172.21.15.1:0/665411250 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f33340d3820 msgr2=0x7f33340b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.844+0000 7f3332ffd700 1 --2- 172.21.15.1:0/665411250 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f33340d3820 0x7f33340b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.844+0000 7f3332ffd700 1 -- 172.21.15.1:0/665411250 >> v1:172.21.15.165:6789/0 conn(0x7f33340b3c30 legacy=0x7f33340d3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.844+0000 7f3332ffd700 1 -- 172.21.15.1:0/665411250 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33340b4540 con 0x7f333415d8c0 2024-03-18T20:27:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.844+0000 7f3332ffd700 1 --2- 172.21.15.1:0/665411250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f333415d8c0 0x7f33340b36f0 secure :-1 s=READY pgs=3148 cs=0 l=1 rev1=1 crypto rx=0x7f33280098f0 tx=0x7f3328009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c3a9f9c88a6b52 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.845+0000 7f3331ffb700 1 -- 172.21.15.1:0/665411250 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3328004230 con 0x7f333415d8c0 2024-03-18T20:27:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.845+0000 7f3331ffb700 1 -- 172.21.15.1:0/665411250 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3328004390 con 0x7f333415d8c0 2024-03-18T20:27:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.845+0000 7f3331ffb700 1 -- 172.21.15.1:0/665411250 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33280036a0 con 0x7f333415d8c0 2024-03-18T20:27:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.845+0000 7f33399a6700 1 -- 172.21.15.1:0/665411250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f333415d8c0 msgr2=0x7f33340b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.845+0000 7f33399a6700 1 --2- 172.21.15.1:0/665411250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f333415d8c0 0x7f33340b36f0 secure :-1 s=READY pgs=3148 cs=0 l=1 rev1=1 crypto rx=0x7f33280098f0 tx=0x7f3328009c00 comp rx=0 tx=0).stop 2024-03-18T20:27:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.845+0000 7f33399a6700 1 -- 172.21.15.1:0/665411250 shutdown_connections 2024-03-18T20:27:45.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.845+0000 7f33399a6700 1 --2- 172.21.15.1:0/665411250 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f33340d3820 0x7f33340b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.845+0000 7f33399a6700 1 --2- 172.21.15.1:0/665411250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f333415d8c0 0x7f33340b36f0 unknown :-1 s=CLOSED pgs=3148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.845+0000 7f33399a6700 1 -- 172.21.15.1:0/665411250 >> 172.21.15.1:0/665411250 conn(0x7f333408afc0 msgr2=0x7f333409c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:45.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.845+0000 7f33399a6700 1 -- 172.21.15.1:0/665411250 shutdown_connections 2024-03-18T20:27:45.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.845+0000 7f33399a6700 1 -- 172.21.15.1:0/665411250 wait complete. 2024-03-18T20:27:45.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.846+0000 7f33399a6700 1 Processor -- start 2024-03-18T20:27:45.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.847+0000 7f33399a6700 1 -- start start 2024-03-18T20:27:45.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.847+0000 7f33399a6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33340b3c30 0x7f33340cef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:45.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.847+0000 7f33399a6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33340d3820 0x7f33340cb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:45.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.847+0000 7f33399a6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f333415d8c0 0x7f333415de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:45.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.847+0000 7f33399a6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3334159110 con 0x7f33340d3820 2024-03-18T20:27:45.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.847+0000 7f33399a6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3334158da0 con 0x7f33340b3c30 2024-03-18T20:27:45.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.847+0000 7f33399a6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33340cbe50 con 0x7f333415d8c0 2024-03-18T20:27:45.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.847+0000 7f33327fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33340d3820 0x7f33340cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:45.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.847+0000 7f33327fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33340d3820 0x7f33340cb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54828/0 (socket says 172.21.15.1:54828) 2024-03-18T20:27:45.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.847+0000 7f33327fc700 1 -- 172.21.15.1:0/938963954 learned_addr learned my addr 172.21.15.1:0/938963954 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:45.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.848+0000 7f3332ffd700 1 --2- 172.21.15.1:0/938963954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33340b3c30 0x7f33340cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:45.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.848+0000 7f33337fe700 1 --2- 172.21.15.1:0/938963954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f333415d8c0 0x7f333415de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:45.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.848+0000 7f33327fc700 1 -- 172.21.15.1:0/938963954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f333415d8c0 msgr2=0x7f333415de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:45.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.848+0000 7f33327fc700 1 --2- 172.21.15.1:0/938963954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f333415d8c0 0x7f333415de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.848+0000 7f33327fc700 1 -- 172.21.15.1:0/938963954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33340b3c30 msgr2=0x7f33340cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:45.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.848+0000 7f33327fc700 1 --2- 172.21.15.1:0/938963954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33340b3c30 0x7f33340cef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.848+0000 7f33327fc700 1 -- 172.21.15.1:0/938963954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3328009550 con 0x7f33340d3820 2024-03-18T20:27:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.848+0000 7f33327fc700 1 --2- 172.21.15.1:0/938963954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33340d3820 0x7f33340cb390 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 crypto rx=0x7f332400c4a0 tx=0x7f332400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.848+0000 7f331bfff700 1 -- 172.21.15.1:0/938963954 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33240054e0 con 0x7f33340d3820 2024-03-18T20:27:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.848+0000 7f331bfff700 1 -- 172.21.15.1:0/938963954 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3324003680 con 0x7f33340d3820 2024-03-18T20:27:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.848+0000 7f33399a6700 1 -- 172.21.15.1:0/938963954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33340cc130 con 0x7f33340d3820 2024-03-18T20:27:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.848+0000 7f331bfff700 1 -- 172.21.15.1:0/938963954 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3324003a30 con 0x7f33340d3820 2024-03-18T20:27:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.848+0000 7f33399a6700 1 -- 172.21.15.1:0/938963954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33340ca6a0 con 0x7f33340d3820 2024-03-18T20:27:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.849+0000 7f33399a6700 1 -- 172.21.15.1:0/938963954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3334159250 con 0x7f33340d3820 2024-03-18T20:27:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.850+0000 7f331bfff700 1 -- 172.21.15.1:0/938963954 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 161) v1 ==== 89257+0+0 (secure 0 0 0) 0x7f3324003b90 con 0x7f33340d3820 2024-03-18T20:27:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.850+0000 7f331bfff700 1 --2- 172.21.15.1:0/938963954 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f332006ae00 0x7f332006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.850+0000 7f331bfff700 1 -- 172.21.15.1:0/938963954 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f3324088bc0 con 0x7f33340d3820 2024-03-18T20:27:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.850+0000 7f3332ffd700 1 --2- 172.21.15.1:0/938963954 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f332006ae00 0x7f332006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:45.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.851+0000 7f3332ffd700 1 --2- 172.21.15.1:0/938963954 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f332006ae00 0x7f332006d2c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3328009130 tx=0x7f3328008030 comp rx=0 tx=0).ready entity=mgr.13401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:45.855+0000 7f331bfff700 1 -- 172.21.15.1:0/938963954 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3334159250 con 0x7f33340d3820 2024-03-18T20:27:46.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.058+0000 7f33399a6700 1 -- 172.21.15.1:0/938963954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3334159250 con 0x7f33340d3820 2024-03-18T20:27:46.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.060+0000 7f331bfff700 1 -- 172.21.15.1:0/938963954 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v275) v1 ==== 74+0+16970 (secure 0 0 0) 0x7f3334159250 con 0x7f33340d3820 2024-03-18T20:27:46.064 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:46.064 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":275,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:27:42.143916+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:27:19.891286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"273","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-03-18T20:27:21.027315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"274","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3635558645":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2293377213":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/1796830610":"2024-03-19T20:27:42.143876+0000","172.21.15.1:6833/148356":"2024-03-19T20:27:42.143876+0000","172.21.15.1:6832/148356":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2068313472":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2668486132":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2560935779":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/82444607":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/1446581052":"2024-03-19T20:27:42.143876+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:27:46.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.062+0000 7f33399a6700 1 -- 172.21.15.1:0/938963954 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f332006ae00 msgr2=0x7f332006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:46.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.062+0000 7f33399a6700 1 --2- 172.21.15.1:0/938963954 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f332006ae00 0x7f332006d2c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3328009130 tx=0x7f3328008030 comp rx=0 tx=0).stop 2024-03-18T20:27:46.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.062+0000 7f33399a6700 1 -- 172.21.15.1:0/938963954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33340d3820 msgr2=0x7f33340cb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:46.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.062+0000 7f33399a6700 1 --2- 172.21.15.1:0/938963954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33340d3820 0x7f33340cb390 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 crypto rx=0x7f332400c4a0 tx=0x7f332400c7b0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.062+0000 7f33399a6700 1 -- 172.21.15.1:0/938963954 shutdown_connections 2024-03-18T20:27:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.062+0000 7f33399a6700 1 --2- 172.21.15.1:0/938963954 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f332006ae00 0x7f332006d2c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.062+0000 7f33399a6700 1 --2- 172.21.15.1:0/938963954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f333415d8c0 0x7f333415de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.062+0000 7f33399a6700 1 --2- 172.21.15.1:0/938963954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33340b3c30 0x7f33340cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.062+0000 7f33399a6700 1 --2- 172.21.15.1:0/938963954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33340d3820 0x7f33340cb390 unknown :-1 s=CLOSED pgs=3149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.062+0000 7f33399a6700 1 -- 172.21.15.1:0/938963954 >> 172.21.15.1:0/938963954 conn(0x7f333408afc0 msgr2=0x7f333406cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.062+0000 7f33399a6700 1 -- 172.21.15.1:0/938963954 shutdown_connections 2024-03-18T20:27:46.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.062+0000 7f33399a6700 1 -- 172.21.15.1:0/938963954 wait complete. 2024-03-18T20:27:46.079 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:27:19.891286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '273', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:27:21.027315+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '274', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:27:46.080 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:27:46.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.208+0000 7fd93598a700 1 Processor -- start 2024-03-18T20:27:46.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.208+0000 7fd93598a700 1 -- start start 2024-03-18T20:27:46.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.209+0000 7fd93598a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd93015d8c0 0x7fd9300b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:46.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.209+0000 7fd93598a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd9300d3820 0x7fd9300b0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:46.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.209+0000 7fd93598a700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9300b00a0 con 0x7fd9300b3c30 2024-03-18T20:27:46.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.209+0000 7fd93598a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9300d1f40 con 0x7fd9300d3820 2024-03-18T20:27:46.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.209+0000 7fd93598a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9300b4260 con 0x7fd93015d8c0 2024-03-18T20:27:46.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.209+0000 7fd92f7fe700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd9300d3820 0x7fd9300b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:46.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.209+0000 7fd92e7fc700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fd9300b3c30 0x7fd9300d3110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:60592/0 (socket says 172.21.15.1:60592) 2024-03-18T20:27:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.209+0000 7fd92effd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd93015d8c0 0x7fd9300b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.209+0000 7fd92e7fc700 1 -- 172.21.15.1:0/3013434908 learned_addr learned my addr 172.21.15.1:0/3013434908 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.210+0000 7fd92dffb700 1 -- 172.21.15.1:0/3013434908 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632277183 0 0) 0x7fd9300b00a0 con 0x7fd9300b3c30 2024-03-18T20:27:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.210+0000 7fd92dffb700 1 -- 172.21.15.1:0/3013434908 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9180035b0 con 0x7fd9300b3c30 2024-03-18T20:27:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.210+0000 7fd92f7fe700 1 -- 172.21.15.1:0/3013434908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd93015d8c0 msgr2=0x7fd9300b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.210+0000 7fd92f7fe700 1 --2- 172.21.15.1:0/3013434908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd93015d8c0 0x7fd9300b36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.210+0000 7fd92f7fe700 1 -- 172.21.15.1:0/3013434908 >> v1:172.21.15.165:6789/0 conn(0x7fd9300b3c30 legacy=0x7fd9300d3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.210+0000 7fd92f7fe700 1 -- 172.21.15.1:0/3013434908 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9300b4540 con 0x7fd9300d3820 2024-03-18T20:27:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.210+0000 7fd92effd700 1 --2- 172.21.15.1:0/3013434908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd93015d8c0 0x7fd9300b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:27:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.210+0000 7fd92f7fe700 1 --2- 172.21.15.1:0/3013434908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9300d3820 0x7fd9300b0530 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fd928009d90 tx=0x7fd92800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a140a7e5329d69ca server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.210+0000 7fd92dffb700 1 -- 172.21.15.1:0/3013434908 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92800e070 con 0x7fd9300d3820 2024-03-18T20:27:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.210+0000 7fd92dffb700 1 -- 172.21.15.1:0/3013434908 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92800bb20 con 0x7fd9300d3820 2024-03-18T20:27:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.210+0000 7fd92dffb700 1 -- 172.21.15.1:0/3013434908 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd928012430 con 0x7fd9300d3820 2024-03-18T20:27:46.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.211+0000 7fd93598a700 1 -- 172.21.15.1:0/3013434908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9300d3820 msgr2=0x7fd9300b0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:46.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.211+0000 7fd93598a700 1 --2- 172.21.15.1:0/3013434908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9300d3820 0x7fd9300b0530 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fd928009d90 tx=0x7fd92800b330 comp rx=0 tx=0).stop 2024-03-18T20:27:46.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.211+0000 7fd93598a700 1 -- 172.21.15.1:0/3013434908 shutdown_connections 2024-03-18T20:27:46.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.211+0000 7fd93598a700 1 --2- 172.21.15.1:0/3013434908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9300d3820 0x7fd9300b0530 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.211+0000 7fd93598a700 1 --2- 172.21.15.1:0/3013434908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd93015d8c0 0x7fd9300b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.211+0000 7fd93598a700 1 -- 172.21.15.1:0/3013434908 >> 172.21.15.1:0/3013434908 conn(0x7fd93008afc0 msgr2=0x7fd93009c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:46.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.211+0000 7fd93598a700 1 -- 172.21.15.1:0/3013434908 shutdown_connections 2024-03-18T20:27:46.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.211+0000 7fd93598a700 1 -- 172.21.15.1:0/3013434908 wait complete. 2024-03-18T20:27:46.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.212+0000 7fd93598a700 1 Processor -- start 2024-03-18T20:27:46.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.212+0000 7fd93598a700 1 -- start start 2024-03-18T20:27:46.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.213+0000 7fd93598a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9300b3c30 0x7fd93015b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:46.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.213+0000 7fd93598a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9300d3820 0x7fd9300ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:46.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.213+0000 7fd93598a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd93015d8c0 0x7fd9301e6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:46.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.213+0000 7fd93598a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9300b6140 con 0x7fd93015d8c0 2024-03-18T20:27:46.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.213+0000 7fd93598a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93015b7c0 con 0x7fd9300d3820 2024-03-18T20:27:46.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.213+0000 7fd93598a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9300b63e0 con 0x7fd9300b3c30 2024-03-18T20:27:46.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.213+0000 7fd92f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd93015d8c0 0x7fd9301e6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:46.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.213+0000 7fd92f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd93015d8c0 0x7fd9301e6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54834/0 (socket says 172.21.15.1:54834) 2024-03-18T20:27:46.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.213+0000 7fd92f7fe700 1 -- 172.21.15.1:0/2287168343 learned_addr learned my addr 172.21.15.1:0/2287168343 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:46.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.213+0000 7fd92e7fc700 1 --2- 172.21.15.1:0/2287168343 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9300d3820 0x7fd9300ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:46.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.213+0000 7fd92effd700 1 --2- 172.21.15.1:0/2287168343 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9300b3c30 0x7fd93015b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:46.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.213+0000 7fd92f7fe700 1 -- 172.21.15.1:0/2287168343 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9300b3c30 msgr2=0x7fd93015b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:46.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.213+0000 7fd92f7fe700 1 --2- 172.21.15.1:0/2287168343 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9300b3c30 0x7fd93015b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.213+0000 7fd92f7fe700 1 -- 172.21.15.1:0/2287168343 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9300d3820 msgr2=0x7fd9300ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:46.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.213+0000 7fd92f7fe700 1 --2- 172.21.15.1:0/2287168343 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9300d3820 0x7fd9300ccd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.213+0000 7fd92f7fe700 1 -- 172.21.15.1:0/2287168343 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9240094a0 con 0x7fd93015d8c0 2024-03-18T20:27:46.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.214+0000 7fd92f7fe700 1 --2- 172.21.15.1:0/2287168343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd93015d8c0 0x7fd9301e6cd0 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 crypto rx=0x7fd92800b370 tx=0x7fd928004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:46.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.214+0000 7fd917fff700 1 -- 172.21.15.1:0/2287168343 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd928012430 con 0x7fd93015d8c0 2024-03-18T20:27:46.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.214+0000 7fd917fff700 1 -- 172.21.15.1:0/2287168343 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd928020070 con 0x7fd93015d8c0 2024-03-18T20:27:46.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.214+0000 7fd93598a700 1 -- 172.21.15.1:0/2287168343 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd928009a70 con 0x7fd93015d8c0 2024-03-18T20:27:46.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.214+0000 7fd917fff700 1 -- 172.21.15.1:0/2287168343 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9280098f0 con 0x7fd93015d8c0 2024-03-18T20:27:46.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.214+0000 7fd93598a700 1 -- 172.21.15.1:0/2287168343 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9300b69a0 con 0x7fd93015d8c0 2024-03-18T20:27:46.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.215+0000 7fd93598a700 1 -- 172.21.15.1:0/2287168343 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9300676b0 con 0x7fd93015d8c0 2024-03-18T20:27:46.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.215+0000 7fd917fff700 1 -- 172.21.15.1:0/2287168343 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 162) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fd928004040 con 0x7fd93015d8c0 2024-03-18T20:27:46.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.216+0000 7fd917fff700 1 --2- 172.21.15.1:0/2287168343 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd91c06ae90 0x7fd91c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:46.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.216+0000 7fd917fff700 1 -- 172.21.15.1:0/2287168343 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7fd928022070 con 0x7fd93015d8c0 2024-03-18T20:27:46.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.216+0000 7fd92effd700 1 --2- 172.21.15.1:0/2287168343 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd91c06ae90 0x7fd91c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:46.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.216+0000 7fd92effd700 1 --2- 172.21.15.1:0/2287168343 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd91c06ae90 0x7fd91c06d350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd924009cd0 tx=0x7fd924009210 comp rx=0 tx=0).ready entity=mgr.13401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:46.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.220+0000 7fd917fff700 1 -- 172.21.15.1:0/2287168343 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd9280561d0 con 0x7fd93015d8c0 2024-03-18T20:27:46.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.420+0000 7fd93598a700 1 -- 172.21.15.1:0/2287168343 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd93008d4a0 con 0x7fd93015d8c0 2024-03-18T20:27:46.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.421+0000 7fd917fff700 1 -- 172.21.15.1:0/2287168343 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v275) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd9280597f0 con 0x7fd93015d8c0 2024-03-18T20:27:46.425 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:27:46.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.423+0000 7fd93598a700 1 -- 172.21.15.1:0/2287168343 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd91c06ae90 msgr2=0x7fd91c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:46.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.423+0000 7fd93598a700 1 --2- 172.21.15.1:0/2287168343 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd91c06ae90 0x7fd91c06d350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd924009cd0 tx=0x7fd924009210 comp rx=0 tx=0).stop 2024-03-18T20:27:46.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.423+0000 7fd93598a700 1 -- 172.21.15.1:0/2287168343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd93015d8c0 msgr2=0x7fd9301e6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:46.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.423+0000 7fd93598a700 1 --2- 172.21.15.1:0/2287168343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd93015d8c0 0x7fd9301e6cd0 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 crypto rx=0x7fd92800b370 tx=0x7fd928004ad0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.423+0000 7fd93598a700 1 -- 172.21.15.1:0/2287168343 shutdown_connections 2024-03-18T20:27:46.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.423+0000 7fd93598a700 1 --2- 172.21.15.1:0/2287168343 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd91c06ae90 0x7fd91c06d350 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.423+0000 7fd93598a700 1 --2- 172.21.15.1:0/2287168343 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd9300b3c30 0x7fd93015b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.423+0000 7fd93598a700 1 --2- 172.21.15.1:0/2287168343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd93015d8c0 0x7fd9301e6cd0 unknown :-1 s=CLOSED pgs=3150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.423+0000 7fd93598a700 1 --2- 172.21.15.1:0/2287168343 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9300d3820 0x7fd9300ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.423+0000 7fd93598a700 1 -- 172.21.15.1:0/2287168343 >> 172.21.15.1:0/2287168343 conn(0x7fd93008afc0 msgr2=0x7fd9300d12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.423+0000 7fd93598a700 1 -- 172.21.15.1:0/2287168343 shutdown_connections 2024-03-18T20:27:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.423+0000 7fd93598a700 1 -- 172.21.15.1:0/2287168343 wait complete. 2024-03-18T20:27:46.439 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:27:46.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.564+0000 7f56bff20700 1 Processor -- start 2024-03-18T20:27:46.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.564+0000 7f56bff20700 1 -- start start 2024-03-18T20:27:46.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.565+0000 7f56bff20700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f56b80cf200 0x7f56b815c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:46.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.565+0000 7f56bff20700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f56b815b6c0 0x7f56b80cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:46.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.565+0000 7f56bff20700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b815ca40 con 0x7f56b80cbc90 2024-03-18T20:27:46.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.565+0000 7f56bff20700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b80cce30 con 0x7f56b80cf200 2024-03-18T20:27:46.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.565+0000 7f56bff20700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b80cfb20 con 0x7f56b815b6c0 2024-03-18T20:27:46.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.565+0000 7f56bdcbc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f56b80cf200 0x7f56b815c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:46.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.565+0000 7f56bdcbc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f56b80cf200 0x7f56b815c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54846/0 (socket says 172.21.15.1:54846) 2024-03-18T20:27:46.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.565+0000 7f56bdcbc700 1 -- 172.21.15.1:0/3856350265 learned_addr learned my addr 172.21.15.1:0/3856350265 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:46.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.565+0000 7f56bdcbc700 1 -- 172.21.15.1:0/3856350265 >> v2:172.21.15.165:3300/0 conn(0x7f56b815b6c0 msgr2=0x7f56b80cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:46.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.565+0000 7f56bdcbc700 1 --2- 172.21.15.1:0/3856350265 >> v2:172.21.15.165:3300/0 conn(0x7f56b815b6c0 0x7f56b80cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.565+0000 7f56bdcbc700 1 -- 172.21.15.1:0/3856350265 >> v1:172.21.15.165:6789/0 conn(0x7f56b80cbc90 legacy=0x7f56b815eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.565+0000 7f56bdcbc700 1 -- 172.21.15.1:0/3856350265 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b80cc220 con 0x7f56b80cf200 2024-03-18T20:27:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.565+0000 7f56bdcbc700 1 --2- 172.21.15.1:0/3856350265 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56b80cf200 0x7f56b815c500 secure :-1 s=READY pgs=3151 cs=0 l=1 rev1=1 crypto rx=0x7f56b8072ac0 tx=0x7f56a8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d377b3356155702 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.566+0000 7f56bccba700 1 -- 172.21.15.1:0/3856350265 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a8004230 con 0x7f56b80cf200 2024-03-18T20:27:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.566+0000 7f56bccba700 1 -- 172.21.15.1:0/3856350265 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a8004390 con 0x7f56b80cf200 2024-03-18T20:27:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.566+0000 7f56bccba700 1 -- 172.21.15.1:0/3856350265 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a80036a0 con 0x7f56b80cf200 2024-03-18T20:27:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.566+0000 7f56bff20700 1 -- 172.21.15.1:0/3856350265 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56b80cf200 msgr2=0x7f56b815c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.566+0000 7f56bff20700 1 --2- 172.21.15.1:0/3856350265 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56b80cf200 0x7f56b815c500 secure :-1 s=READY pgs=3151 cs=0 l=1 rev1=1 crypto rx=0x7f56b8072ac0 tx=0x7f56a8009bd0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.566+0000 7f56bff20700 1 -- 172.21.15.1:0/3856350265 shutdown_connections 2024-03-18T20:27:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.566+0000 7f56bff20700 1 --2- 172.21.15.1:0/3856350265 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56b80cf200 0x7f56b815c500 unknown :-1 s=CLOSED pgs=3151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.566+0000 7f56bff20700 1 --2- 172.21.15.1:0/3856350265 >> v2:172.21.15.165:3300/0 conn(0x7f56b815b6c0 0x7f56b80cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.566+0000 7f56bff20700 1 -- 172.21.15.1:0/3856350265 >> 172.21.15.1:0/3856350265 conn(0x7f56b80d1180 msgr2=0x7f56b80d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:46.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.566+0000 7f56bff20700 1 -- 172.21.15.1:0/3856350265 shutdown_connections 2024-03-18T20:27:46.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.566+0000 7f56bff20700 1 -- 172.21.15.1:0/3856350265 wait complete. 2024-03-18T20:27:46.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.567+0000 7f56bff20700 1 Processor -- start 2024-03-18T20:27:46.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.567+0000 7f56bff20700 1 -- start start 2024-03-18T20:27:46.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.568+0000 7f56bff20700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56b80cbc90 0x7f56b80c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:46.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.568+0000 7f56bff20700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56b80cf200 0x7f56b80aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:46.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.568+0000 7f56bff20700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56b815b6c0 0x7f56b81534d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:46.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.568+0000 7f56bff20700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b80cad50 con 0x7f56b80cbc90 2024-03-18T20:27:46.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.568+0000 7f56bff20700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b80ca180 con 0x7f56b80cf200 2024-03-18T20:27:46.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.568+0000 7f56bff20700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b80b3e10 con 0x7f56b815b6c0 2024-03-18T20:27:46.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.568+0000 7f56bdcbc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56b80cbc90 0x7f56b80c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:46.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.568+0000 7f56bdcbc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56b80cbc90 0x7f56b80c96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54854/0 (socket says 172.21.15.1:54854) 2024-03-18T20:27:46.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.568+0000 7f56bdcbc700 1 -- 172.21.15.1:0/4184575549 learned_addr learned my addr 172.21.15.1:0/4184575549 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:46.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.568+0000 7f56bd4bb700 1 --2- 172.21.15.1:0/4184575549 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56b80cf200 0x7f56b80aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:46.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.568+0000 7f56be4bd700 1 --2- 172.21.15.1:0/4184575549 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56b815b6c0 0x7f56b81534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:46.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.569+0000 7f56bdcbc700 1 -- 172.21.15.1:0/4184575549 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56b815b6c0 msgr2=0x7f56b81534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:46.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.569+0000 7f56bdcbc700 1 --2- 172.21.15.1:0/4184575549 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56b815b6c0 0x7f56b81534d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.569+0000 7f56bdcbc700 1 -- 172.21.15.1:0/4184575549 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56b80cf200 msgr2=0x7f56b80aea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:46.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.569+0000 7f56bdcbc700 1 --2- 172.21.15.1:0/4184575549 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56b80cf200 0x7f56b80aea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.569+0000 7f56bdcbc700 1 -- 172.21.15.1:0/4184575549 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a8009550 con 0x7f56b80cbc90 2024-03-18T20:27:46.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.569+0000 7f56bd4bb700 1 --2- 172.21.15.1:0/4184575549 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56b80cf200 0x7f56b80aea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:46.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.569+0000 7f56bdcbc700 1 --2- 172.21.15.1:0/4184575549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56b80cbc90 0x7f56b80c96c0 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 crypto rx=0x7f56a8009130 tx=0x7f56a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:46.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.569+0000 7f56ae7fc700 1 -- 172.21.15.1:0/4184575549 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a8011830 con 0x7f56b80cbc90 2024-03-18T20:27:46.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.569+0000 7f56ae7fc700 1 -- 172.21.15.1:0/4184575549 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a8011990 con 0x7f56b80cbc90 2024-03-18T20:27:46.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.569+0000 7f56bff20700 1 -- 172.21.15.1:0/4184575549 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b80ca420 con 0x7f56b80cbc90 2024-03-18T20:27:46.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.569+0000 7f56ae7fc700 1 -- 172.21.15.1:0/4184575549 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a8011d40 con 0x7f56b80cbc90 2024-03-18T20:27:46.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.569+0000 7f56bff20700 1 -- 172.21.15.1:0/4184575549 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b80ca9e0 con 0x7f56b80cbc90 2024-03-18T20:27:46.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.570+0000 7f56bff20700 1 -- 172.21.15.1:0/4184575549 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b80d0ca0 con 0x7f56b80cbc90 2024-03-18T20:27:46.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.570+0000 7f56ae7fc700 1 -- 172.21.15.1:0/4184575549 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 162) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f56a8024460 con 0x7f56b80cbc90 2024-03-18T20:27:46.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.571+0000 7f56ae7fc700 1 --2- 172.21.15.1:0/4184575549 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f569c06ae90 0x7f569c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:46.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.571+0000 7f56ae7fc700 1 -- 172.21.15.1:0/4184575549 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f56a808b310 con 0x7f56b80cbc90 2024-03-18T20:27:46.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.571+0000 7f56bd4bb700 1 --2- 172.21.15.1:0/4184575549 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f569c06ae90 0x7f569c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:46.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.571+0000 7f56bd4bb700 1 --2- 172.21.15.1:0/4184575549 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f569c06ae90 0x7f569c06d350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f56b81565a0 tx=0x7f56b4008040 comp rx=0 tx=0).ready entity=mgr.13401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:46.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.575+0000 7f56ae7fc700 1 -- 172.21.15.1:0/4184575549 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f56a8055e90 con 0x7f56b80cbc90 2024-03-18T20:27:46.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.777+0000 7f56bff20700 1 -- 172.21.15.1:0/4184575549 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f56b815ca40 con 0x7f56b80cbc90 2024-03-18T20:27:46.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.777+0000 7f56ae7fc700 1 -- 172.21.15.1:0/4184575549 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v275) v1 ==== 104+0+11 (secure 0 0 0) 0x7f56b815ca40 con 0x7f56b80cbc90 2024-03-18T20:27:46.781 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:27:46.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.779+0000 7f56bff20700 1 -- 172.21.15.1:0/4184575549 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f569c06ae90 msgr2=0x7f569c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:46.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.779+0000 7f56bff20700 1 --2- 172.21.15.1:0/4184575549 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f569c06ae90 0x7f569c06d350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f56b81565a0 tx=0x7f56b4008040 comp rx=0 tx=0).stop 2024-03-18T20:27:46.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.779+0000 7f56bff20700 1 -- 172.21.15.1:0/4184575549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56b80cbc90 msgr2=0x7f56b80c96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:46.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.779+0000 7f56bff20700 1 --2- 172.21.15.1:0/4184575549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56b80cbc90 0x7f56b80c96c0 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 crypto rx=0x7f56a8009130 tx=0x7f56a8004750 comp rx=0 tx=0).stop 2024-03-18T20:27:46.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.779+0000 7f56bff20700 1 -- 172.21.15.1:0/4184575549 shutdown_connections 2024-03-18T20:27:46.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.779+0000 7f56bff20700 1 --2- 172.21.15.1:0/4184575549 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f569c06ae90 0x7f569c06d350 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.779+0000 7f56bff20700 1 --2- 172.21.15.1:0/4184575549 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f56b815b6c0 0x7f56b81534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.779+0000 7f56bff20700 1 --2- 172.21.15.1:0/4184575549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56b80cbc90 0x7f56b80c96c0 unknown :-1 s=CLOSED pgs=3152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.779+0000 7f56bff20700 1 --2- 172.21.15.1:0/4184575549 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f56b80cf200 0x7f56b80aea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.779+0000 7f56bff20700 1 -- 172.21.15.1:0/4184575549 >> 172.21.15.1:0/4184575549 conn(0x7f56b80d1180 msgr2=0x7f56b80b24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:46.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.779+0000 7f56bff20700 1 -- 172.21.15.1:0/4184575549 shutdown_connections 2024-03-18T20:27:46.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.779+0000 7f56bff20700 1 -- 172.21.15.1:0/4184575549 wait complete. 2024-03-18T20:27:46.795 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:27:46.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.920+0000 7f764ec7e700 1 Processor -- start 2024-03-18T20:27:46.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.920+0000 7f764ec7e700 1 -- start start 2024-03-18T20:27:46.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.920+0000 7f764ec7e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f764815b640 0x7f76480b27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.920+0000 7f764ec7e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f76480cf180 0x7f76480c6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.920+0000 7f764ec7e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76480d1d40 con 0x7f764809a440 2024-03-18T20:27:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.920+0000 7f764ec7e700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76480c60c0 con 0x7f76480cf180 2024-03-18T20:27:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.921+0000 7f764ec7e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76480c7250 con 0x7f764815b640 2024-03-18T20:27:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.921+0000 7f7647fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f76480cf180 0x7f76480c6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.921+0000 7f764d21b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f764809a440 0x7f764815efa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58772/0 (socket says 172.21.15.1:58772) 2024-03-18T20:27:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.921+0000 7f7647fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f76480cf180 0x7f76480c6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54870/0 (socket says 172.21.15.1:54870) 2024-03-18T20:27:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.921+0000 7f764d21b700 1 -- 172.21.15.1:0/4260017796 learned_addr learned my addr 172.21.15.1:0/4260017796 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.921+0000 7f764ca1a700 1 --2- 172.21.15.1:0/4260017796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f764815b640 0x7f76480b27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.921+0000 7f7647fff700 1 -- 172.21.15.1:0/4260017796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f764815b640 msgr2=0x7f76480b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.921+0000 7f7647fff700 1 --2- 172.21.15.1:0/4260017796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f764815b640 0x7f76480b27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.921+0000 7f7647fff700 1 -- 172.21.15.1:0/4260017796 >> v1:172.21.15.1:6789/0 conn(0x7f764809a440 legacy=0x7f764815efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:46.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.921+0000 7f7647fff700 1 -- 172.21.15.1:0/4260017796 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76480d1f90 con 0x7f76480cf180 2024-03-18T20:27:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.921+0000 7f76477fe700 1 -- 172.21.15.1:0/4260017796 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15383239 0 0) 0x7f76480d1d40 con 0x7f764809a440 2024-03-18T20:27:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.921+0000 7f764ca1a700 1 --2- 172.21.15.1:0/4260017796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f764815b640 0x7f76480b27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.921+0000 7f7647fff700 1 --2- 172.21.15.1:0/4260017796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76480cf180 0x7f76480c6d10 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 crypto rx=0x7f7634009840 tx=0x7f7634009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a8bfc7c6eac6706 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.921+0000 7f76477fe700 1 -- 172.21.15.1:0/4260017796 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7634011830 con 0x7f76480cf180 2024-03-18T20:27:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.921+0000 7f76477fe700 1 -- 172.21.15.1:0/4260017796 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7634011990 con 0x7f76480cf180 2024-03-18T20:27:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.921+0000 7f76477fe700 1 -- 172.21.15.1:0/4260017796 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7634011d20 con 0x7f76480cf180 2024-03-18T20:27:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.922+0000 7f764ec7e700 1 -- 172.21.15.1:0/4260017796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76480cf180 msgr2=0x7f76480c6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.922+0000 7f764ec7e700 1 --2- 172.21.15.1:0/4260017796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76480cf180 0x7f76480c6d10 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 crypto rx=0x7f7634009840 tx=0x7f7634009b50 comp rx=0 tx=0).stop 2024-03-18T20:27:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.922+0000 7f764ec7e700 1 -- 172.21.15.1:0/4260017796 shutdown_connections 2024-03-18T20:27:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.922+0000 7f764ec7e700 1 --2- 172.21.15.1:0/4260017796 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f764815b640 0x7f76480b27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.922+0000 7f764ec7e700 1 --2- 172.21.15.1:0/4260017796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76480cf180 0x7f76480c6d10 unknown :-1 s=CLOSED pgs=3154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.922+0000 7f764ec7e700 1 -- 172.21.15.1:0/4260017796 >> 172.21.15.1:0/4260017796 conn(0x7f76480d1220 msgr2=0x7f76480d1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.922+0000 7f764ec7e700 1 -- 172.21.15.1:0/4260017796 shutdown_connections 2024-03-18T20:27:46.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.922+0000 7f764ec7e700 1 -- 172.21.15.1:0/4260017796 wait complete. 2024-03-18T20:27:46.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.923+0000 7f764ec7e700 1 Processor -- start 2024-03-18T20:27:46.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.923+0000 7f764ec7e700 1 -- start start 2024-03-18T20:27:46.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.924+0000 7f764ec7e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f764809a440 0x7f76480b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.924+0000 7f764ec7e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76480cf180 0x7f76480ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.924+0000 7f764ec7e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76481f77e0 0x7f764806fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.924+0000 7f764ec7e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76480ae720 con 0x7f764809a440 2024-03-18T20:27:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.924+0000 7f764ec7e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764806c060 con 0x7f76480cf180 2024-03-18T20:27:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.924+0000 7f764ec7e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7648152340 con 0x7f76481f77e0 2024-03-18T20:27:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.924+0000 7f764ca1a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f764809a440 0x7f76480b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.924+0000 7f764ca1a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f764809a440 0x7f76480b02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54886/0 (socket says 172.21.15.1:54886) 2024-03-18T20:27:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.924+0000 7f764ca1a700 1 -- 172.21.15.1:0/4246266813 learned_addr learned my addr 172.21.15.1:0/4246266813 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.924+0000 7f7647fff700 1 --2- 172.21.15.1:0/4246266813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76480cf180 0x7f76480ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.924+0000 7f764d21b700 1 --2- 172.21.15.1:0/4246266813 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76481f77e0 0x7f764806fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.924+0000 7f764ca1a700 1 -- 172.21.15.1:0/4246266813 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76481f77e0 msgr2=0x7f764806fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.924+0000 7f764ca1a700 1 --2- 172.21.15.1:0/4246266813 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76481f77e0 0x7f764806fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.924+0000 7f764ca1a700 1 -- 172.21.15.1:0/4246266813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76480cf180 msgr2=0x7f76480ae1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:46.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.924+0000 7f764ca1a700 1 --2- 172.21.15.1:0/4246266813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76480cf180 0x7f76480ae1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.924+0000 7f764ca1a700 1 -- 172.21.15.1:0/4246266813 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76340094a0 con 0x7f764809a440 2024-03-18T20:27:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.925+0000 7f764ca1a700 1 --2- 172.21.15.1:0/4246266813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f764809a440 0x7f76480b02f0 secure :-1 s=READY pgs=3155 cs=0 l=1 rev1=1 crypto rx=0x7f763c00e810 tx=0x7f763c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.925+0000 7f76457fa700 1 -- 172.21.15.1:0/4246266813 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c004230 con 0x7f764809a440 2024-03-18T20:27:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.925+0000 7f76457fa700 1 -- 172.21.15.1:0/4246266813 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763c004390 con 0x7f764809a440 2024-03-18T20:27:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.925+0000 7f764ec7e700 1 -- 172.21.15.1:0/4246266813 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f764806ba10 con 0x7f764809a440 2024-03-18T20:27:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.925+0000 7f76457fa700 1 -- 172.21.15.1:0/4246266813 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c0036a0 con 0x7f764809a440 2024-03-18T20:27:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.925+0000 7f764ec7e700 1 -- 172.21.15.1:0/4246266813 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f764806d7b0 con 0x7f764809a440 2024-03-18T20:27:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.926+0000 7f764ec7e700 1 -- 172.21.15.1:0/4246266813 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7648067dc0 con 0x7f764809a440 2024-03-18T20:27:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.926+0000 7f76457fa700 1 -- 172.21.15.1:0/4246266813 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 162) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f763c003800 con 0x7f764809a440 2024-03-18T20:27:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.927+0000 7f76457fa700 1 --2- 172.21.15.1:0/4246266813 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f762c06ae90 0x7f762c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.927+0000 7f76457fa700 1 -- 172.21.15.1:0/4246266813 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f763c089600 con 0x7f764809a440 2024-03-18T20:27:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.927+0000 7f7647fff700 1 --2- 172.21.15.1:0/4246266813 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f762c06ae90 0x7f762c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:46.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.927+0000 7f7647fff700 1 --2- 172.21.15.1:0/4246266813 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f762c06ae90 0x7f762c06d350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f76480cf580 tx=0x7f7634008030 comp rx=0 tx=0).ready entity=mgr.13401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:46.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:46.931+0000 7f76457fa700 1 -- 172.21.15.1:0/4246266813 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f763c054100 con 0x7f764809a440 2024-03-18T20:27:47.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.132+0000 7f764ec7e700 1 -- 172.21.15.1:0/4246266813 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f76480b7190 con 0x7f764809a440 2024-03-18T20:27:47.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.132+0000 7f76457fa700 1 -- 172.21.15.1:0/4246266813 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v275) v1 ==== 100+0+11 (secure 0 0 0) 0x7f763c057720 con 0x7f764809a440 2024-03-18T20:27:47.136 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:27:47.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.134+0000 7f764ec7e700 1 -- 172.21.15.1:0/4246266813 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f762c06ae90 msgr2=0x7f762c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:47.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.134+0000 7f764ec7e700 1 --2- 172.21.15.1:0/4246266813 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f762c06ae90 0x7f762c06d350 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f76480cf580 tx=0x7f7634008030 comp rx=0 tx=0).stop 2024-03-18T20:27:47.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.134+0000 7f764ec7e700 1 -- 172.21.15.1:0/4246266813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f764809a440 msgr2=0x7f76480b02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:47.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.134+0000 7f764ec7e700 1 --2- 172.21.15.1:0/4246266813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f764809a440 0x7f76480b02f0 secure :-1 s=READY pgs=3155 cs=0 l=1 rev1=1 crypto rx=0x7f763c00e810 tx=0x7f763c00ebd0 comp rx=0 tx=0).stop 2024-03-18T20:27:47.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.134+0000 7f764ec7e700 1 -- 172.21.15.1:0/4246266813 shutdown_connections 2024-03-18T20:27:47.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.134+0000 7f764ec7e700 1 --2- 172.21.15.1:0/4246266813 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f762c06ae90 0x7f762c06d350 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:47.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.134+0000 7f764ec7e700 1 --2- 172.21.15.1:0/4246266813 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f76481f77e0 0x7f764806fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:47.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.134+0000 7f764ec7e700 1 --2- 172.21.15.1:0/4246266813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f764809a440 0x7f76480b02f0 unknown :-1 s=CLOSED pgs=3155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:47.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.134+0000 7f764ec7e700 1 --2- 172.21.15.1:0/4246266813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f76480cf180 0x7f76480ae1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:47.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.134+0000 7f764ec7e700 1 -- 172.21.15.1:0/4246266813 >> 172.21.15.1:0/4246266813 conn(0x7f76480d1220 msgr2=0x7f7648153ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:47.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.134+0000 7f764ec7e700 1 -- 172.21.15.1:0/4246266813 shutdown_connections 2024-03-18T20:27:47.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.134+0000 7f764ec7e700 1 -- 172.21.15.1:0/4246266813 wait complete. 2024-03-18T20:27:47.151 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T20:27:47.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.273+0000 7f6b11b50700 1 Processor -- start 2024-03-18T20:27:47.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.274+0000 7f6b11b50700 1 -- start start 2024-03-18T20:27:47.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.274+0000 7f6b11b50700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6b0c099a30 0x7f6b0c099e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:47.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.274+0000 7f6b11b50700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6b0c15ef60 0x7f6b0c15cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:47.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.274+0000 7f6b11b50700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b0c06ec60 con 0x7f6b0c15e8e0 2024-03-18T20:27:47.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.274+0000 7f6b11b50700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b0c070100 con 0x7f6b0c15ef60 2024-03-18T20:27:47.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.274+0000 7f6b11b50700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c09a350 con 0x7f6b0c099a30 2024-03-18T20:27:47.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.274+0000 7f6b0bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6b0c15ef60 0x7f6b0c15cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:47.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.274+0000 7f6b0bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6b0c15ef60 0x7f6b0c15cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54896/0 (socket says 172.21.15.1:54896) 2024-03-18T20:27:47.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.275+0000 7f6b0bfff700 1 -- 172.21.15.1:0/4052405017 learned_addr learned my addr 172.21.15.1:0/4052405017 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:47.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.275+0000 7f6b0b7fe700 1 --2- 172.21.15.1:0/4052405017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6b0c099a30 0x7f6b0c099e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:47.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.275+0000 7f6b0bfff700 1 -- 172.21.15.1:0/4052405017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6b0c099a30 msgr2=0x7f6b0c099e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:47.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.275+0000 7f6b0bfff700 1 --2- 172.21.15.1:0/4052405017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6b0c099a30 0x7f6b0c099e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:47.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.275+0000 7f6b0bfff700 1 -- 172.21.15.1:0/4052405017 >> v1:172.21.15.165:6789/0 conn(0x7f6b0c15e8e0 legacy=0x7f6b0c0cf380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:47.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.275+0000 7f6b0bfff700 1 -- 172.21.15.1:0/4052405017 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0c0cfd80 con 0x7f6b0c15ef60 2024-03-18T20:27:47.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.275+0000 7f6b0bfff700 1 --2- 172.21.15.1:0/4052405017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b0c15ef60 0x7f6b0c15cdf0 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 crypto rx=0x7f6b04009e50 tx=0x7f6b0400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db4a1ac64f45e43 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:47.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.275+0000 7f6b0a7fc700 1 -- 172.21.15.1:0/4052405017 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0400e070 con 0x7f6b0c15ef60 2024-03-18T20:27:47.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.275+0000 7f6b0a7fc700 1 -- 172.21.15.1:0/4052405017 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0400b9c0 con 0x7f6b0c15ef60 2024-03-18T20:27:47.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.275+0000 7f6b0a7fc700 1 -- 172.21.15.1:0/4052405017 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0400bd70 con 0x7f6b0c15ef60 2024-03-18T20:27:47.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.276+0000 7f6b11b50700 1 -- 172.21.15.1:0/4052405017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b0c15ef60 msgr2=0x7f6b0c15cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:47.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.276+0000 7f6b11b50700 1 --2- 172.21.15.1:0/4052405017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b0c15ef60 0x7f6b0c15cdf0 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 crypto rx=0x7f6b04009e50 tx=0x7f6b0400b330 comp rx=0 tx=0).stop 2024-03-18T20:27:47.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.276+0000 7f6b11b50700 1 -- 172.21.15.1:0/4052405017 shutdown_connections 2024-03-18T20:27:47.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.276+0000 7f6b11b50700 1 --2- 172.21.15.1:0/4052405017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b0c15ef60 0x7f6b0c15cdf0 unknown :-1 s=CLOSED pgs=3156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:47.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.276+0000 7f6b11b50700 1 --2- 172.21.15.1:0/4052405017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6b0c099a30 0x7f6b0c099e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:47.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.276+0000 7f6b11b50700 1 -- 172.21.15.1:0/4052405017 >> 172.21.15.1:0/4052405017 conn(0x7f6b0c0d1180 msgr2=0x7f6b0c0d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:47.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.276+0000 7f6b11b50700 1 -- 172.21.15.1:0/4052405017 shutdown_connections 2024-03-18T20:27:47.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.276+0000 7f6b11b50700 1 -- 172.21.15.1:0/4052405017 wait complete. 2024-03-18T20:27:47.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.277+0000 7f6b11b50700 1 Processor -- start 2024-03-18T20:27:47.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.277+0000 7f6b11b50700 1 -- start start 2024-03-18T20:27:47.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.278+0000 7f6b11b50700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b0c099a30 0x7f6b0c0c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:47.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.278+0000 7f6b11b50700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6b0c15ef60 0x7f6b0c1f3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:47.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.278+0000 7f6b11b50700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6b0c0cb770 0x7f6b0c0af190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:47.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.278+0000 7f6b11b50700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c0cc820 con 0x7f6b0c099a30 2024-03-18T20:27:47.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.278+0000 7f6b11b50700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c154140 con 0x7f6b0c0cb770 2024-03-18T20:27:47.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.278+0000 7f6b11b50700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c0cbca0 con 0x7f6b0c15ef60 2024-03-18T20:27:47.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.278+0000 7f6b0b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b0c099a30 0x7f6b0c0c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:47.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.278+0000 7f6b0b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b0c099a30 0x7f6b0c0c9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54912/0 (socket says 172.21.15.1:54912) 2024-03-18T20:27:47.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.278+0000 7f6b0b7fe700 1 -- 172.21.15.1:0/3546001668 learned_addr learned my addr 172.21.15.1:0/3546001668 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:47.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.278+0000 7f6b0bfff700 1 --2- 172.21.15.1:0/3546001668 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6b0c0cb770 0x7f6b0c0af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:47.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.278+0000 7f6b0affd700 1 --2- 172.21.15.1:0/3546001668 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6b0c15ef60 0x7f6b0c1f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:47.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.278+0000 7f6b0b7fe700 1 -- 172.21.15.1:0/3546001668 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6b0c15ef60 msgr2=0x7f6b0c1f3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:47.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.278+0000 7f6b0b7fe700 1 --2- 172.21.15.1:0/3546001668 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6b0c15ef60 0x7f6b0c1f3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:47.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.278+0000 7f6b0b7fe700 1 -- 172.21.15.1:0/3546001668 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6b0c0cb770 msgr2=0x7f6b0c0af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:47.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.278+0000 7f6b0b7fe700 1 --2- 172.21.15.1:0/3546001668 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6b0c0cb770 0x7f6b0c0af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:47.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.278+0000 7f6b0b7fe700 1 -- 172.21.15.1:0/3546001668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b04009b30 con 0x7f6b0c099a30 2024-03-18T20:27:47.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.278+0000 7f6b0b7fe700 1 --2- 172.21.15.1:0/3546001668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b0c099a30 0x7f6b0c0c9eb0 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7f6afc00e810 tx=0x7f6afc00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:47.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.279+0000 7f6b03fff700 1 -- 172.21.15.1:0/3546001668 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc00c070 con 0x7f6b0c099a30 2024-03-18T20:27:47.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.279+0000 7f6b03fff700 1 -- 172.21.15.1:0/3546001668 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6afc013610 con 0x7f6b0c099a30 2024-03-18T20:27:47.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.279+0000 7f6b11b50700 1 -- 172.21.15.1:0/3546001668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b0c0cbf80 con 0x7f6b0c099a30 2024-03-18T20:27:47.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.279+0000 7f6b03fff700 1 -- 172.21.15.1:0/3546001668 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc0139c0 con 0x7f6b0c099a30 2024-03-18T20:27:47.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.279+0000 7f6b11b50700 1 -- 172.21.15.1:0/3546001668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b0c0cc540 con 0x7f6b0c099a30 2024-03-18T20:27:47.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.280+0000 7f6b11b50700 1 -- 172.21.15.1:0/3546001668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0c09ada0 con 0x7f6b0c099a30 2024-03-18T20:27:47.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.280+0000 7f6b03fff700 1 -- 172.21.15.1:0/3546001668 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 162) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f6afc013b20 con 0x7f6b0c099a30 2024-03-18T20:27:47.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.281+0000 7f6b03fff700 1 --2- 172.21.15.1:0/3546001668 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6aec06ae90 0x7f6aec06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:47.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.281+0000 7f6b0affd700 1 --2- 172.21.15.1:0/3546001668 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6aec06ae90 0x7f6aec06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:47.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.281+0000 7f6b03fff700 1 -- 172.21.15.1:0/3546001668 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f6afc0897c0 con 0x7f6b0c099a30 2024-03-18T20:27:47.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.281+0000 7f6b0affd700 1 --2- 172.21.15.1:0/3546001668 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6aec06ae90 0x7f6aec06d350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c0b13a0 tx=0x7f6af4009040 comp rx=0 tx=0).ready entity=mgr.13401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:47.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.285+0000 7f6b03fff700 1 -- 172.21.15.1:0/3546001668 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6afc0542d0 con 0x7f6b0c099a30 2024-03-18T20:27:47.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.504+0000 7f6b11b50700 1 -- 172.21.15.1:0/3546001668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6b0c067dc0 con 0x7f6b0c099a30 2024-03-18T20:27:47.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.504+0000 7f6b03fff700 1 -- 172.21.15.1:0/3546001668 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v275) v1 ==== 81+0+566 (secure 0 0 0) 0x7f6afc0578f0 con 0x7f6b0c099a30 2024-03-18T20:27:47.508 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3635558645 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:47.508 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2293377213 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:47.508 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1796830610 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:47.508 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6833/148356 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:47.509 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6832/148356 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:47.509 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2068313472 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:47.509 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2668486132 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:47.509 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2560935779 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:47.509 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/82444607 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:47.509 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1446581052 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:47.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.506+0000 7f6b11b50700 1 -- 172.21.15.1:0/3546001668 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6aec06ae90 msgr2=0x7f6aec06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:47.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.506+0000 7f6b11b50700 1 --2- 172.21.15.1:0/3546001668 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6aec06ae90 0x7f6aec06d350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c0b13a0 tx=0x7f6af4009040 comp rx=0 tx=0).stop 2024-03-18T20:27:47.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.506+0000 7f6b11b50700 1 -- 172.21.15.1:0/3546001668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b0c099a30 msgr2=0x7f6b0c0c9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:47.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.506+0000 7f6b11b50700 1 --2- 172.21.15.1:0/3546001668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b0c099a30 0x7f6b0c0c9eb0 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7f6afc00e810 tx=0x7f6afc00eb20 comp rx=0 tx=0).stop 2024-03-18T20:27:47.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.506+0000 7f6b11b50700 1 -- 172.21.15.1:0/3546001668 shutdown_connections 2024-03-18T20:27:47.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.506+0000 7f6b11b50700 1 --2- 172.21.15.1:0/3546001668 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6aec06ae90 0x7f6aec06d350 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:47.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.506+0000 7f6b11b50700 1 --2- 172.21.15.1:0/3546001668 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6b0c0cb770 0x7f6b0c0af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:47.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.506+0000 7f6b11b50700 1 --2- 172.21.15.1:0/3546001668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b0c099a30 0x7f6b0c0c9eb0 unknown :-1 s=CLOSED pgs=3157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:47.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.506+0000 7f6b11b50700 1 --2- 172.21.15.1:0/3546001668 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6b0c15ef60 0x7f6b0c1f3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:47.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.506+0000 7f6b11b50700 1 -- 172.21.15.1:0/3546001668 >> 172.21.15.1:0/3546001668 conn(0x7f6b0c0d1180 msgr2=0x7f6b0c15da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:47.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.506+0000 7f6b11b50700 1 -- 172.21.15.1:0/3546001668 shutdown_connections 2024-03-18T20:27:47.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.506+0000 7f6b11b50700 1 -- 172.21.15.1:0/3546001668 wait complete. 2024-03-18T20:27:47.512 INFO:teuthology.orchestra.run.smithi001.stderr:listed 10 entries 2024-03-18T20:27:47.522 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-18T20:27:47.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:47.522 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T20:27:47.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[170560]: fuse finished with error 0 and tester_r 0 2024-03-18T20:27:47.559 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:27:47.571 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:27:47.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:47.571 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T20:27:47.630 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:27:47.630 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:27:47.688 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:27:47.688 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T20:27:47.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.848+0000 7f120fcf4700 1 Processor -- start 2024-03-18T20:27:47.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.848+0000 7f120fcf4700 1 -- start start 2024-03-18T20:27:47.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.849+0000 7f120fcf4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f120815d910 0x7f12080b16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:47.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.849+0000 7f120fcf4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f12080af4e0 0x7f120806ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:47.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.849+0000 7f120fcf4700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12080ad7c0 con 0x7f120806a7d0 2024-03-18T20:27:47.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.849+0000 7f120fcf4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f120806b180 con 0x7f120815d910 2024-03-18T20:27:47.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.849+0000 7f120fcf4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12080b4910 con 0x7f12080af4e0 2024-03-18T20:27:47.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.849+0000 7f120da90700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f120815d910 0x7f12080b16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:47.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.849+0000 7f120d28f700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f120806a7d0 0x7f12080afdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58786/0 (socket says 172.21.15.1:58786) 2024-03-18T20:27:47.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.849+0000 7f120d28f700 1 -- 172.21.15.1:0/2139731715 learned_addr learned my addr 172.21.15.1:0/2139731715 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:47.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.850+0000 7f120e291700 1 --2- 172.21.15.1:0/2139731715 >> v2:172.21.15.165:3300/0 conn(0x7f12080af4e0 0x7f120806ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:47.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.850+0000 7f120da90700 1 -- 172.21.15.1:0/2139731715 >> v2:172.21.15.165:3300/0 conn(0x7f12080af4e0 msgr2=0x7f120806ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:47.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.850+0000 7f120da90700 1 --2- 172.21.15.1:0/2139731715 >> v2:172.21.15.165:3300/0 conn(0x7f12080af4e0 0x7f120806ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:47.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.850+0000 7f120da90700 1 -- 172.21.15.1:0/2139731715 >> v1:172.21.15.1:6789/0 conn(0x7f120806a7d0 legacy=0x7f12080afdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:47.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.850+0000 7f120da90700 1 -- 172.21.15.1:0/2139731715 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12080b39b0 con 0x7f120815d910 2024-03-18T20:27:47.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.850+0000 7f120ca8e700 1 -- 172.21.15.1:0/2139731715 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851027905 0 0) 0x7f12080ad7c0 con 0x7f120806a7d0 2024-03-18T20:27:47.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.850+0000 7f120da90700 1 --2- 172.21.15.1:0/2139731715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f120815d910 0x7f12080b16d0 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 crypto rx=0x7f12080b56d0 tx=0x7f11f8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ccde5898ffb0ba0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:47.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.850+0000 7f120ca8e700 1 -- 172.21.15.1:0/2139731715 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f8011830 con 0x7f120815d910 2024-03-18T20:27:47.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.850+0000 7f120ca8e700 1 -- 172.21.15.1:0/2139731715 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f8011990 con 0x7f120815d910 2024-03-18T20:27:47.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.850+0000 7f120ca8e700 1 -- 172.21.15.1:0/2139731715 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f8011d20 con 0x7f120815d910 2024-03-18T20:27:47.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.851+0000 7f120fcf4700 1 -- 172.21.15.1:0/2139731715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f120815d910 msgr2=0x7f12080b16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:47.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.851+0000 7f120fcf4700 1 --2- 172.21.15.1:0/2139731715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f120815d910 0x7f12080b16d0 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 crypto rx=0x7f12080b56d0 tx=0x7f11f8009b20 comp rx=0 tx=0).stop 2024-03-18T20:27:47.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.851+0000 7f120fcf4700 1 -- 172.21.15.1:0/2139731715 shutdown_connections 2024-03-18T20:27:47.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.851+0000 7f120fcf4700 1 --2- 172.21.15.1:0/2139731715 >> v2:172.21.15.165:3300/0 conn(0x7f12080af4e0 0x7f120806ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:47.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.851+0000 7f120fcf4700 1 --2- 172.21.15.1:0/2139731715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f120815d910 0x7f12080b16d0 unknown :-1 s=CLOSED pgs=3159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:47.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.851+0000 7f120fcf4700 1 -- 172.21.15.1:0/2139731715 >> 172.21.15.1:0/2139731715 conn(0x7f12080b5780 msgr2=0x7f12080b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:47.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.851+0000 7f120fcf4700 1 -- 172.21.15.1:0/2139731715 shutdown_connections 2024-03-18T20:27:47.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.851+0000 7f120fcf4700 1 -- 172.21.15.1:0/2139731715 wait complete. 2024-03-18T20:27:47.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.852+0000 7f120fcf4700 1 Processor -- start 2024-03-18T20:27:47.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.852+0000 7f120fcf4700 1 -- start start 2024-03-18T20:27:47.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.853+0000 7f120fcf4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f120806a7d0 0x7f12081549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:47.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.853+0000 7f120fcf4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12080af4e0 0x7f1208160040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:47.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.853+0000 7f120fcf4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f120815d910 0x7f1208160580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:47.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.853+0000 7f120fcf4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1208156430 con 0x7f120806a7d0 2024-03-18T20:27:47.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.853+0000 7f120fcf4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120809c1b0 con 0x7f120815d910 2024-03-18T20:27:47.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.853+0000 7f120fcf4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f120809c490 con 0x7f12080af4e0 2024-03-18T20:27:47.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.853+0000 7f120da90700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f120806a7d0 0x7f12081549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:47.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.853+0000 7f120da90700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f120806a7d0 0x7f12081549a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54936/0 (socket says 172.21.15.1:54936) 2024-03-18T20:27:47.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.853+0000 7f120da90700 1 -- 172.21.15.1:0/3557714791 learned_addr learned my addr 172.21.15.1:0/3557714791 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:47.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.853+0000 7f120da90700 1 -- 172.21.15.1:0/3557714791 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12080af4e0 msgr2=0x7f1208160040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:47.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.853+0000 7f120da90700 1 --2- 172.21.15.1:0/3557714791 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12080af4e0 0x7f1208160040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:47.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.853+0000 7f120da90700 1 -- 172.21.15.1:0/3557714791 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f120815d910 msgr2=0x7f1208160580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:47.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.853+0000 7f120da90700 1 --2- 172.21.15.1:0/3557714791 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f120815d910 0x7f1208160580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:47.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.853+0000 7f120da90700 1 -- 172.21.15.1:0/3557714791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f80094a0 con 0x7f120806a7d0 2024-03-18T20:27:47.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.853+0000 7f120da90700 1 --2- 172.21.15.1:0/3557714791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f120806a7d0 0x7f12081549a0 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 crypto rx=0x7f11f8014040 tx=0x7f11f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:47.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.853+0000 7f11fe7fc700 1 -- 172.21.15.1:0/3557714791 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f80038a0 con 0x7f120806a7d0 2024-03-18T20:27:47.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.853+0000 7f11fe7fc700 1 -- 172.21.15.1:0/3557714791 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f801e070 con 0x7f120806a7d0 2024-03-18T20:27:47.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.853+0000 7f11fe7fc700 1 -- 172.21.15.1:0/3557714791 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f8003c20 con 0x7f120806a7d0 2024-03-18T20:27:47.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.853+0000 7f120fcf4700 1 -- 172.21.15.1:0/3557714791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f120809c770 con 0x7f120806a7d0 2024-03-18T20:27:47.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.854+0000 7f120fcf4700 1 -- 172.21.15.1:0/3557714791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1208154ee0 con 0x7f120806a7d0 2024-03-18T20:27:47.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.854+0000 7f120fcf4700 1 -- 172.21.15.1:0/3557714791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1208067dc0 con 0x7f120806a7d0 2024-03-18T20:27:47.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.855+0000 7f11fe7fc700 1 -- 172.21.15.1:0/3557714791 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 162) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f11f802d450 con 0x7f120806a7d0 2024-03-18T20:27:47.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.855+0000 7f11fe7fc700 1 --2- 172.21.15.1:0/3557714791 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f11ec06ae90 0x7f11ec06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:47.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.855+0000 7f11fe7fc700 1 -- 172.21.15.1:0/3557714791 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f11f808ab30 con 0x7f120806a7d0 2024-03-18T20:27:47.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.855+0000 7f120d28f700 1 --2- 172.21.15.1:0/3557714791 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f11ec06ae90 0x7f11ec06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:47.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.856+0000 7f120d28f700 1 --2- 172.21.15.1:0/3557714791 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f11ec06ae90 0x7f11ec06d350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f12040011e0 tx=0x7f12040079b0 comp rx=0 tx=0).ready entity=mgr.13401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:47.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:47.860+0000 7f11fe7fc700 1 -- 172.21.15.1:0/3557714791 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f11f80556a0 con 0x7f120806a7d0 2024-03-18T20:27:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:48.113+0000 7f120fcf4700 1 -- 172.21.15.1:0/3557714791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f12080b6540 con 0x7f120806a7d0 2024-03-18T20:27:48.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:48.113+0000 7f11fe7fc700 1 -- 172.21.15.1:0/3557714791 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v136) v1 ==== 53+0+73 (secure 0 0 0) 0x7f11f8015070 con 0x7f120806a7d0 2024-03-18T20:27:48.117 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T20:27:48.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:48.116+0000 7f120fcf4700 1 -- 172.21.15.1:0/3557714791 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f11ec06ae90 msgr2=0x7f11ec06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:48.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:48.116+0000 7f120fcf4700 1 --2- 172.21.15.1:0/3557714791 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f11ec06ae90 0x7f11ec06d350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f12040011e0 tx=0x7f12040079b0 comp rx=0 tx=0).stop 2024-03-18T20:27:48.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:48.116+0000 7f120fcf4700 1 -- 172.21.15.1:0/3557714791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f120806a7d0 msgr2=0x7f12081549a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:48.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:48.116+0000 7f120fcf4700 1 --2- 172.21.15.1:0/3557714791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f120806a7d0 0x7f12081549a0 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 crypto rx=0x7f11f8014040 tx=0x7f11f8004750 comp rx=0 tx=0).stop 2024-03-18T20:27:48.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:48.116+0000 7f120fcf4700 1 -- 172.21.15.1:0/3557714791 shutdown_connections 2024-03-18T20:27:48.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:48.116+0000 7f120fcf4700 1 --2- 172.21.15.1:0/3557714791 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f11ec06ae90 0x7f11ec06d350 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:48.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:48.116+0000 7f120fcf4700 1 --2- 172.21.15.1:0/3557714791 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f120815d910 0x7f1208160580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:48.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:48.116+0000 7f120fcf4700 1 --2- 172.21.15.1:0/3557714791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f120806a7d0 0x7f12081549a0 unknown :-1 s=CLOSED pgs=3160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:48.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:48.116+0000 7f120fcf4700 1 --2- 172.21.15.1:0/3557714791 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f12080af4e0 0x7f1208160040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:48.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:48.116+0000 7f120fcf4700 1 -- 172.21.15.1:0/3557714791 >> 172.21.15.1:0/3557714791 conn(0x7f12080b5780 msgr2=0x7f12080b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:48.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:48.116+0000 7f120fcf4700 1 -- 172.21.15.1:0/3557714791 shutdown_connections 2024-03-18T20:27:48.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:48.116+0000 7f120fcf4700 1 -- 172.21.15.1:0/3557714791 wait complete. 2024-03-18T20:27:48.132 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T20:27:48.132 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T20:27:48.132 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-18T20:27:48.133 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T20:27:48.133 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T20:27:48.133 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T20:27:48.133 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T20:27:48.133 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T20:27:48.133 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:27:48.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:48.133 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-18T20:27:48.152 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T20:27:48.152 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T20:27:48.152 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T20:27:48.152 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:27:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T20:27:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:27:48.153 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:27:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-18T20:27:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-18T20:27:48.153 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:27:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-18T20:27:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-18T20:27:48.153 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T20:27:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:27:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-18T20:27:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T20:27:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 36701sec preferred_lft 36701sec 2024-03-18T20:27:48.153 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link 2024-03-18T20:27:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:27:48.154 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:27:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:27:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-18T20:27:48.154 INFO:teuthology.orchestra.run.smithi001.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T20:27:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:27:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T20:27:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:27:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::4ca9:f5ff:feb3:79a3/64 scope link 2024-03-18T20:27:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:27:48.154 INFO:teuthology.orchestra.run.smithi001.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T20:27:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T20:27:48.154 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::282f:a4ff:fe1a:9908/64 scope link 2024-03-18T20:27:48.155 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:27:48.155 INFO:teuthology.orchestra.run.smithi001.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T20:27:48.155 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2e:5c:60:78:b2:8c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T20:27:48.155 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::2c5c:60ff:fe78:b28c/64 scope link 2024-03-18T20:27:48.155 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:27:48.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:48.155 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T20:27:48.219 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-18T20:27:48.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:48.221 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-18T20:27:48.281 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-18T20:27:48.281 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T20:27:48.281 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T20:27:48.281 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T20:27:48.281 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T20:27:48.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:48.281 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T20:27:48.281 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T20:27:48.282 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T20:27:48.282 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T20:27:48.282 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T20:27:48.282 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T20:27:48.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:48.387 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T20:27:48.405 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T20:27:48.406 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T20:27:48.406 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:27:48.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:48.406 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-18T20:27:48.473 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T20:27:48.474 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:27:48.474 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T20:27:48.532 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:27:48.532 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:27:48.532 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T20:27:48.589 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T20:27:48.658 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:27:48.658 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:27:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:27:48.726 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:27:48.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:48.726 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T20:27:48.785 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T20:27:48.785 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T20:27:48.828 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T20:27:48.855 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:27:48.855 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:27:48.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-18T20:27:48.913+0000 7f3c6b4bb5c0 -1 init, newargv = 0x558659832f70 newargc=15 2024-03-18T20:27:48.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[172313]: starting ceph client 2024-03-18T20:27:48.922 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[172313]: starting fuse 2024-03-18T20:27:48.924 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:27:48.925 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:27:48.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:48.926 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T20:27:48.984 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T20:27:48.985 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-03-18T20:27:48.985 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T20:27:48.985 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-18T20:27:48.985 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-18T20:27:48.985 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-18T20:27:48.985 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-18T20:27:48.985 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:27:48.985 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-18T20:27:48.985 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T20:27:48.985 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-18T20:27:48.985 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T20:27:48.985 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:27:48.985 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-18T20:27:48.985 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-18T20:27:48.986 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-18T20:27:48.986 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-18T20:27:48.986 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:27:48.986 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-18T20:27:48.986 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T20:27:48.986 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T20:27:48.986 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T20:27:48.986 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-18T20:27:48.986 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-18T20:27:48.986 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-18T20:27:48.986 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T20:27:48.986 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:27:48.986 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-18T20:27:48.986 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-18T20:27:49.109 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.172313.asok 2024-03-18T20:27:49.291 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.172313.asok 2024-03-18T20:27:49.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:49.291 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.172313.asok status 2024-03-18T20:27:49.453 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:27:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-18T20:27:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T20:27:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T20:27:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-18T20:27:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-18T20:27:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T20:27:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "172313", 2024-03-18T20:27:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-18T20:27:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-18T20:27:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-18T20:27:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 13539, 2024-03-18T20:27:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-18T20:27:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-18T20:27:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-18T20:27:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 13539 2024-03-18T20:27:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T20:27:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T20:27:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:27:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 566312026 2024-03-18T20:27:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:27:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T20:27:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T20:27:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:27:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 566312026 2024-03-18T20:27:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:27:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.13539 192.168.0.1:0/566312026", 2024-03-18T20:27:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/566312026", 2024-03-18T20:27:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-18T20:27:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 136, 2024-03-18T20:27:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 275, 2024-03-18T20:27:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-18T20:27:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-18T20:27:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-18T20:27:49.455 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:27:49.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:49.463 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-18T20:27:49.463 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T20:27:49.481 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-18T20:27:49.481 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T20:27:49.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:49.482 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T20:27:49.549 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-03-18-20_28_00 in ~17s... 2024-03-18T20:27:49.549 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2024-03-18T20:27:49.549 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-03-18T20:27:49.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.712+0000 7fcecf54d700 1 Processor -- start 2024-03-18T20:27:49.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.713+0000 7fcecf54d700 1 -- start start 2024-03-18T20:27:49.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.714+0000 7fcecf54d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcec815a350 0x7fcec80b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:49.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.714+0000 7fcecf54d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcec80b26e0 0x7fcec80b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:49.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.714+0000 7fcecf54d700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec806fb70 con 0x7fcec80b48f0 2024-03-18T20:27:49.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.714+0000 7fcecf54d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcec80aeb10 con 0x7fcec815a350 2024-03-18T20:27:49.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.714+0000 7fcecf54d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcec80b2c50 con 0x7fcec80b26e0 2024-03-18T20:27:49.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.714+0000 7fcecd2e9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcec815a350 0x7fcec80b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:49.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.714+0000 7fcecd2e9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcec815a350 0x7fcec80b32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54946/0 (socket says 172.21.15.1:54946) 2024-03-18T20:27:49.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.714+0000 7fcecd2e9700 1 -- 172.21.15.1:0/3429741402 learned_addr learned my addr 172.21.15.1:0/3429741402 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:49.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.714+0000 7fceccae8700 1 --2- 172.21.15.1:0/3429741402 >> v2:172.21.15.165:3300/0 conn(0x7fcec80b26e0 0x7fcec80b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:49.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.714+0000 7fcecd2e9700 1 -- 172.21.15.1:0/3429741402 >> v2:172.21.15.165:3300/0 conn(0x7fcec80b26e0 msgr2=0x7fcec80b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:49.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.714+0000 7fcecd2e9700 1 --2- 172.21.15.1:0/3429741402 >> v2:172.21.15.165:3300/0 conn(0x7fcec80b26e0 0x7fcec80b4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:49.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.714+0000 7fcecd2e9700 1 -- 172.21.15.1:0/3429741402 >> v1:172.21.15.165:6789/0 conn(0x7fcec80b48f0 legacy=0x7fcec80aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:49.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.714+0000 7fcecd2e9700 1 -- 172.21.15.1:0/3429741402 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec806f5b0 con 0x7fcec815a350 2024-03-18T20:27:49.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.714+0000 7fceccae8700 1 --2- 172.21.15.1:0/3429741402 >> v2:172.21.15.165:3300/0 conn(0x7fcec80b26e0 0x7fcec80b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:49.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.714+0000 7fcecd2e9700 1 --2- 172.21.15.1:0/3429741402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec815a350 0x7fcec80b32d0 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 crypto rx=0x7fcec8071350 tx=0x7fcec4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c397c0f7319f8584 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:49.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.714+0000 7fcebbfff700 1 -- 172.21.15.1:0/3429741402 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec400b870 con 0x7fcec815a350 2024-03-18T20:27:49.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.714+0000 7fcebbfff700 1 -- 172.21.15.1:0/3429741402 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec400b9d0 con 0x7fcec815a350 2024-03-18T20:27:49.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.714+0000 7fcebbfff700 1 -- 172.21.15.1:0/3429741402 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec400bd60 con 0x7fcec815a350 2024-03-18T20:27:49.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.715+0000 7fcecf54d700 1 -- 172.21.15.1:0/3429741402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec815a350 msgr2=0x7fcec80b32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:49.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.715+0000 7fcecf54d700 1 --2- 172.21.15.1:0/3429741402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec815a350 0x7fcec80b32d0 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 crypto rx=0x7fcec8071350 tx=0x7fcec4009bd0 comp rx=0 tx=0).stop 2024-03-18T20:27:49.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.715+0000 7fcecf54d700 1 -- 172.21.15.1:0/3429741402 shutdown_connections 2024-03-18T20:27:49.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.715+0000 7fcecf54d700 1 --2- 172.21.15.1:0/3429741402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec815a350 0x7fcec80b32d0 unknown :-1 s=CLOSED pgs=3164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:49.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.715+0000 7fcecf54d700 1 --2- 172.21.15.1:0/3429741402 >> v2:172.21.15.165:3300/0 conn(0x7fcec80b26e0 0x7fcec80b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:49.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.715+0000 7fcecf54d700 1 -- 172.21.15.1:0/3429741402 >> 172.21.15.1:0/3429741402 conn(0x7fcec80b5a10 msgr2=0x7fcec80b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:49.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.715+0000 7fcecf54d700 1 -- 172.21.15.1:0/3429741402 shutdown_connections 2024-03-18T20:27:49.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.715+0000 7fcecf54d700 1 -- 172.21.15.1:0/3429741402 wait complete. 2024-03-18T20:27:49.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.716+0000 7fcecf54d700 1 Processor -- start 2024-03-18T20:27:49.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.716+0000 7fcecf54d700 1 -- start start 2024-03-18T20:27:49.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.717+0000 7fcecf54d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec80b26e0 0x7fcec81664c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:49.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.717+0000 7fcecf54d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcec80b48f0 0x7fcec8163a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:49.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.717+0000 7fcecf54d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcec815a350 0x7fcec813fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:49.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.717+0000 7fcecf54d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec8163f70 con 0x7fcec80b26e0 2024-03-18T20:27:49.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.717+0000 7fcecf54d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec8164a20 con 0x7fcec815a350 2024-03-18T20:27:49.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.717+0000 7fcecf54d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcec814ee30 con 0x7fcec80b48f0 2024-03-18T20:27:49.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.717+0000 7fcecd2e9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec80b26e0 0x7fcec81664c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:49.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.717+0000 7fcecd2e9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec80b26e0 0x7fcec81664c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54948/0 (socket says 172.21.15.1:54948) 2024-03-18T20:27:49.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.717+0000 7fcecd2e9700 1 -- 172.21.15.1:0/3844364148 learned_addr learned my addr 172.21.15.1:0/3844364148 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:49.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.717+0000 7fceccae8700 1 --2- 172.21.15.1:0/3844364148 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcec80b48f0 0x7fcec8163a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:49.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.717+0000 7fcecdaea700 1 --2- 172.21.15.1:0/3844364148 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcec815a350 0x7fcec813fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:49.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.717+0000 7fcecd2e9700 1 -- 172.21.15.1:0/3844364148 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcec80b48f0 msgr2=0x7fcec8163a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:49.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.717+0000 7fcecd2e9700 1 --2- 172.21.15.1:0/3844364148 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcec80b48f0 0x7fcec8163a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:49.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.717+0000 7fcecd2e9700 1 -- 172.21.15.1:0/3844364148 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcec815a350 msgr2=0x7fcec813fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:49.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.717+0000 7fcecd2e9700 1 --2- 172.21.15.1:0/3844364148 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcec815a350 0x7fcec813fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:49.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.717+0000 7fcecd2e9700 1 -- 172.21.15.1:0/3844364148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec4009550 con 0x7fcec80b26e0 2024-03-18T20:27:49.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.717+0000 7fcecdaea700 1 --2- 172.21.15.1:0/3844364148 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcec815a350 0x7fcec813fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:49.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.717+0000 7fceccae8700 1 --2- 172.21.15.1:0/3844364148 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcec80b48f0 0x7fcec8163a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:49.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.718+0000 7fcecd2e9700 1 --2- 172.21.15.1:0/3844364148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec80b26e0 0x7fcec81664c0 secure :-1 s=READY pgs=3165 cs=0 l=1 rev1=1 crypto rx=0x7fcec4014040 tx=0x7fcec4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:49.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.718+0000 7fceb9ffb700 1 -- 172.21.15.1:0/3844364148 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec4004520 con 0x7fcec80b26e0 2024-03-18T20:27:49.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.718+0000 7fceb9ffb700 1 -- 172.21.15.1:0/3844364148 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec401e070 con 0x7fcec80b26e0 2024-03-18T20:27:49.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.718+0000 7fceb9ffb700 1 -- 172.21.15.1:0/3844364148 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec40038a0 con 0x7fcec80b26e0 2024-03-18T20:27:49.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.718+0000 7fcecf54d700 1 -- 172.21.15.1:0/3844364148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec8164180 con 0x7fcec80b26e0 2024-03-18T20:27:49.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.718+0000 7fcecf54d700 1 -- 172.21.15.1:0/3844364148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec8164740 con 0x7fcec80b26e0 2024-03-18T20:27:49.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.719+0000 7fcecf54d700 1 -- 172.21.15.1:0/3844364148 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcec8067dc0 con 0x7fcec80b26e0 2024-03-18T20:27:49.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.719+0000 7fceb9ffb700 1 -- 172.21.15.1:0/3844364148 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 162) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fcec4003a00 con 0x7fcec80b26e0 2024-03-18T20:27:49.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.720+0000 7fceb9ffb700 1 --2- 172.21.15.1:0/3844364148 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fceac06ae90 0x7fceac06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:49.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.720+0000 7fceb9ffb700 1 -- 172.21.15.1:0/3844364148 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7fcec4020070 con 0x7fcec80b26e0 2024-03-18T20:27:49.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.720+0000 7fceccae8700 1 --2- 172.21.15.1:0/3844364148 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fceac06ae90 0x7fceac06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:49.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.720+0000 7fceccae8700 1 --2- 172.21.15.1:0/3844364148 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fceac06ae90 0x7fceac06d350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcec8166470 tx=0x7fcebc008040 comp rx=0 tx=0).ready entity=mgr.13401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:49.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.725+0000 7fceb9ffb700 1 -- 172.21.15.1:0/3844364148 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcec4055c10 con 0x7fcec80b26e0 2024-03-18T20:27:49.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.951+0000 7fcecf54d700 1 -- 172.21.15.1:0/3844364148 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcec8099bf0 con 0x7fceac06ae90 2024-03-18T20:27:49.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.959+0000 7fceb9ffb700 1 -- 172.21.15.1:0/3844364148 <== mgr.13401 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7fcebc009e90 con 0x7fceac06ae90 2024-03-18T20:27:49.963 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:49.963 INFO:teuthology.orchestra.run.smithi001.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-03-18T20:27:49.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.961+0000 7fcecf54d700 1 -- 172.21.15.1:0/3844364148 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fceac06ae90 msgr2=0x7fceac06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:49.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.961+0000 7fcecf54d700 1 --2- 172.21.15.1:0/3844364148 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fceac06ae90 0x7fceac06d350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcec8166470 tx=0x7fcebc008040 comp rx=0 tx=0).stop 2024-03-18T20:27:49.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.961+0000 7fcecf54d700 1 -- 172.21.15.1:0/3844364148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec80b26e0 msgr2=0x7fcec81664c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:49.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.961+0000 7fcecf54d700 1 --2- 172.21.15.1:0/3844364148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec80b26e0 0x7fcec81664c0 secure :-1 s=READY pgs=3165 cs=0 l=1 rev1=1 crypto rx=0x7fcec4014040 tx=0x7fcec4004750 comp rx=0 tx=0).stop 2024-03-18T20:27:49.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.961+0000 7fcecf54d700 1 -- 172.21.15.1:0/3844364148 shutdown_connections 2024-03-18T20:27:49.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.961+0000 7fcecf54d700 1 --2- 172.21.15.1:0/3844364148 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fceac06ae90 0x7fceac06d350 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:49.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.961+0000 7fcecf54d700 1 --2- 172.21.15.1:0/3844364148 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcec815a350 0x7fcec813fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:49.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.961+0000 7fcecf54d700 1 --2- 172.21.15.1:0/3844364148 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec80b26e0 0x7fcec81664c0 unknown :-1 s=CLOSED pgs=3165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:49.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.961+0000 7fcecf54d700 1 --2- 172.21.15.1:0/3844364148 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcec80b48f0 0x7fcec8163a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:49.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.961+0000 7fcecf54d700 1 -- 172.21.15.1:0/3844364148 >> 172.21.15.1:0/3844364148 conn(0x7fcec80b5a10 msgr2=0x7fcec80b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:49.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.961+0000 7fcecf54d700 1 -- 172.21.15.1:0/3844364148 shutdown_connections 2024-03-18T20:27:49.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:49.961+0000 7fcecf54d700 1 -- 172.21.15.1:0/3844364148 wait complete. 2024-03-18T20:27:49.977 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-03-18T20:27:49.977 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2024-03-18T20:27:49.977 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-03-18T20:27:50.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.100+0000 7fb34b228700 1 Processor -- start 2024-03-18T20:27:50.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.101+0000 7fb34b228700 1 -- start start 2024-03-18T20:27:50.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.101+0000 7fb34b228700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb3440ae240 0x7fb3440b0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.101+0000 7fb34b228700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3440b5e90 con 0x7fb34406aee0 2024-03-18T20:27:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.101+0000 7fb34b228700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3440ccbd0 con 0x7fb34415b5a0 2024-03-18T20:27:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.101+0000 7fb34b228700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb34415b980 con 0x7fb3440ae240 2024-03-18T20:27:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb343fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb34406aee0 0x7fb3440af930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58800/0 (socket says 172.21.15.1:58800) 2024-03-18T20:27:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb343fff700 1 -- 172.21.15.1:0/3266750960 learned_addr learned my addr 172.21.15.1:0/3266750960 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb3497c5700 1 --2- 172.21.15.1:0/3266750960 >> v2:172.21.15.165:3300/0 conn(0x7fb3440ae240 0x7fb3440b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb3437fe700 1 -- 172.21.15.1:0/3266750960 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443374328 0 0) 0x7fb3440b5e90 con 0x7fb34406aee0 2024-03-18T20:27:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb3437fe700 1 -- 172.21.15.1:0/3266750960 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3340035b0 con 0x7fb34406aee0 2024-03-18T20:27:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb3437fe700 1 -- 172.21.15.1:0/3266750960 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499144137 0 0) 0x7fb3440ccbd0 con 0x7fb34415b5a0 2024-03-18T20:27:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb3437fe700 1 -- 172.21.15.1:0/3266750960 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3440b5e90 con 0x7fb34415b5a0 2024-03-18T20:27:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb3437fe700 1 -- 172.21.15.1:0/3266750960 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1725590764 0 0) 0x7fb3340035b0 con 0x7fb34406aee0 2024-03-18T20:27:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb3437fe700 1 -- 172.21.15.1:0/3266750960 >> v2:172.21.15.165:3300/0 conn(0x7fb3440ae240 msgr2=0x7fb3440b0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb3437fe700 1 --2- 172.21.15.1:0/3266750960 >> v2:172.21.15.165:3300/0 conn(0x7fb3440ae240 0x7fb3440b0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb3437fe700 1 -- 172.21.15.1:0/3266750960 >> v1:172.21.15.165:6789/0 conn(0x7fb34415b5a0 legacy=0x7fb34406a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb3437fe700 1 -- 172.21.15.1:0/3266750960 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3440b1190 con 0x7fb34406aee0 2024-03-18T20:27:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb3437fe700 1 -- 172.21.15.1:0/3266750960 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fb330001670 con 0x7fb34406aee0 2024-03-18T20:27:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb3437fe700 1 -- 172.21.15.1:0/3266750960 >> v1:172.21.15.1:6789/0 conn(0x7fb34406aee0 legacy=0x7fb3440af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb3437fe700 1 --2- 172.21.15.1:0/3266750960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb334003b90 0x7fb334003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb3437fe700 1 --2- 172.21.15.1:0/3266750960 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb334004850 0x7fb334004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb3437fe700 1 --2- 172.21.15.1:0/3266750960 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb334005210 0x7fb33400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb3437fe700 1 -- 172.21.15.1:0/3266750960 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34415b980 con 0x7fb334003b90 2024-03-18T20:27:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb3437fe700 1 -- 172.21.15.1:0/3266750960 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33400c090 con 0x7fb334004850 2024-03-18T20:27:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb3437fe700 1 -- 172.21.15.1:0/3266750960 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33400c250 con 0x7fb334005210 2024-03-18T20:27:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb348fc4700 1 --2- 172.21.15.1:0/3266750960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb334003b90 0x7fb334003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.102+0000 7fb343fff700 1 --2- 172.21.15.1:0/3266750960 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb334004850 0x7fb334004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.103+0000 7fb3497c5700 1 --2- 172.21.15.1:0/3266750960 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb334005210 0x7fb33400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.103+0000 7fb348fc4700 1 -- 172.21.15.1:0/3266750960 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb334005210 msgr2=0x7fb33400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.103+0000 7fb348fc4700 1 --2- 172.21.15.1:0/3266750960 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb334005210 0x7fb33400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.103+0000 7fb348fc4700 1 -- 172.21.15.1:0/3266750960 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb334004850 msgr2=0x7fb334004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.103+0000 7fb348fc4700 1 --2- 172.21.15.1:0/3266750960 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb334004850 0x7fb334004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.103+0000 7fb348fc4700 1 -- 172.21.15.1:0/3266750960 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3340035b0 con 0x7fb334003b90 2024-03-18T20:27:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.103+0000 7fb3497c5700 1 -- 172.21.15.1:0/3266750960 reap_dead start 2024-03-18T20:27:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.103+0000 7fb343fff700 1 --2- 172.21.15.1:0/3266750960 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb334004850 0x7fb334004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.103+0000 7fb348fc4700 1 --2- 172.21.15.1:0/3266750960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb334003b90 0x7fb334003f90 secure :-1 s=READY pgs=3167 cs=0 l=1 rev1=1 crypto rx=0x7fb338001940 tx=0x7fb334005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad43c58d43b100ac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.103+0000 7fb3437fe700 1 -- 172.21.15.1:0/3266750960 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338001430 con 0x7fb334003b90 2024-03-18T20:27:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.103+0000 7fb3437fe700 1 -- 172.21.15.1:0/3266750960 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb338001d70 con 0x7fb334003b90 2024-03-18T20:27:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.103+0000 7fb3437fe700 1 -- 172.21.15.1:0/3266750960 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338005510 con 0x7fb334003b90 2024-03-18T20:27:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.103+0000 7fb34b228700 1 -- 172.21.15.1:0/3266750960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb334003b90 msgr2=0x7fb334003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.103+0000 7fb34b228700 1 --2- 172.21.15.1:0/3266750960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb334003b90 0x7fb334003f90 secure :-1 s=READY pgs=3167 cs=0 l=1 rev1=1 crypto rx=0x7fb338001940 tx=0x7fb334005880 comp rx=0 tx=0).stop 2024-03-18T20:27:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.103+0000 7fb34b228700 1 -- 172.21.15.1:0/3266750960 shutdown_connections 2024-03-18T20:27:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.103+0000 7fb34b228700 1 --2- 172.21.15.1:0/3266750960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb334003b90 0x7fb334003f90 unknown :-1 s=CLOSED pgs=3167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.103+0000 7fb34b228700 1 -- 172.21.15.1:0/3266750960 >> 172.21.15.1:0/3266750960 conn(0x7fb344099a30 msgr2=0x7fb3440b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:50.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.103+0000 7fb34b228700 1 -- 172.21.15.1:0/3266750960 shutdown_connections 2024-03-18T20:27:50.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.104+0000 7fb34b228700 1 -- 172.21.15.1:0/3266750960 wait complete. 2024-03-18T20:27:50.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.105+0000 7fb34b228700 1 Processor -- start 2024-03-18T20:27:50.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.105+0000 7fb34b228700 1 -- start start 2024-03-18T20:27:50.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb34b228700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb334003b90 0x7fb3440cf510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:50.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb34b228700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb34413d3e0 0x7fb344151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:50.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb34b228700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3441520d0 0x7fb34414f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:50.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb34b228700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3440cf0f0 con 0x7fb3441520d0 2024-03-18T20:27:50.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb34b228700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3440cfa50 con 0x7fb34413d3e0 2024-03-18T20:27:50.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb34b228700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3440ce470 con 0x7fb334003b90 2024-03-18T20:27:50.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb3497c5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3441520d0 0x7fb34414f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:50.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb3497c5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3441520d0 0x7fb34414f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54966/0 (socket says 172.21.15.1:54966) 2024-03-18T20:27:50.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb3497c5700 1 -- 172.21.15.1:0/2029542449 learned_addr learned my addr 172.21.15.1:0/2029542449 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:50.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb348fc4700 1 --2- 172.21.15.1:0/2029542449 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb334003b90 0x7fb3440cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:50.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb343fff700 1 --2- 172.21.15.1:0/2029542449 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb34413d3e0 0x7fb344151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:50.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb3497c5700 1 -- 172.21.15.1:0/2029542449 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb334003b90 msgr2=0x7fb3440cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:50.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb3497c5700 1 --2- 172.21.15.1:0/2029542449 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb334003b90 0x7fb3440cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:50.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb3497c5700 1 -- 172.21.15.1:0/2029542449 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb34413d3e0 msgr2=0x7fb344151b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:50.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb3497c5700 1 --2- 172.21.15.1:0/2029542449 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb34413d3e0 0x7fb344151b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:50.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb3497c5700 1 -- 172.21.15.1:0/2029542449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3440ce750 con 0x7fb3441520d0 2024-03-18T20:27:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb3497c5700 1 --2- 172.21.15.1:0/2029542449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3441520d0 0x7fb34414f0e0 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 crypto rx=0x7fb3340047d0 tx=0x7fb33c0114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb3417fa700 1 -- 172.21.15.1:0/2029542449 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c00e070 con 0x7fb3441520d0 2024-03-18T20:27:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb3417fa700 1 -- 172.21.15.1:0/2029542449 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33c003c30 con 0x7fb3441520d0 2024-03-18T20:27:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb34b228700 1 -- 172.21.15.1:0/2029542449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3440cea30 con 0x7fb3441520d0 2024-03-18T20:27:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb3417fa700 1 -- 172.21.15.1:0/2029542449 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c003de0 con 0x7fb3441520d0 2024-03-18T20:27:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.106+0000 7fb34b228700 1 -- 172.21.15.1:0/2029542449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34413d860 con 0x7fb3441520d0 2024-03-18T20:27:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.107+0000 7fb34b228700 1 -- 172.21.15.1:0/2029542449 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3440b1d60 con 0x7fb3441520d0 2024-03-18T20:27:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.108+0000 7fb3417fa700 1 -- 172.21.15.1:0/2029542449 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 162) v1 ==== 89313+0+0 (secure 0 0 0) 0x7fb33c011a10 con 0x7fb3441520d0 2024-03-18T20:27:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.108+0000 7fb3417fa700 1 --2- 172.21.15.1:0/2029542449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb32806ae90 0x7fb32806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.108+0000 7fb348fc4700 1 --2- 172.21.15.1:0/2029542449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb32806ae90 0x7fb32806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.108+0000 7fb3417fa700 1 -- 172.21.15.1:0/2029542449 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7fb33c089f40 con 0x7fb3441520d0 2024-03-18T20:27:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.109+0000 7fb348fc4700 1 --2- 172.21.15.1:0/2029542449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb32806ae90 0x7fb32806d350 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb338005040 tx=0x7fb33800a000 comp rx=0 tx=0).ready entity=mgr.13401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:50.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.113+0000 7fb3417fa700 1 -- 172.21.15.1:0/2029542449 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb33c054ad0 con 0x7fb3441520d0 2024-03-18T20:27:50.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.342+0000 7fb34b228700 1 -- 172.21.15.1:0/2029542449 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3440b60e0 con 0x7fb32806ae90 2024-03-18T20:27:50.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.400+0000 7fb3417fa700 1 -- 172.21.15.1:0/2029542449 <== mgr.13401 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fb3380051d0 con 0x7fb32806ae90 2024-03-18T20:27:50.404 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule removed for path snap_test_dir1 2024-03-18T20:27:50.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.402+0000 7fb34b228700 1 -- 172.21.15.1:0/2029542449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb32806ae90 msgr2=0x7fb32806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:50.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.402+0000 7fb34b228700 1 --2- 172.21.15.1:0/2029542449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb32806ae90 0x7fb32806d350 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb338005040 tx=0x7fb33800a000 comp rx=0 tx=0).stop 2024-03-18T20:27:50.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.402+0000 7fb34b228700 1 -- 172.21.15.1:0/2029542449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3441520d0 msgr2=0x7fb34414f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:50.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.402+0000 7fb34b228700 1 --2- 172.21.15.1:0/2029542449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3441520d0 0x7fb34414f0e0 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 crypto rx=0x7fb3340047d0 tx=0x7fb33c0114b0 comp rx=0 tx=0).stop 2024-03-18T20:27:50.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.402+0000 7fb34b228700 1 -- 172.21.15.1:0/2029542449 shutdown_connections 2024-03-18T20:27:50.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.402+0000 7fb34b228700 1 --2- 172.21.15.1:0/2029542449 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb32806ae90 0x7fb32806d350 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:50.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.402+0000 7fb34b228700 1 --2- 172.21.15.1:0/2029542449 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb334003b90 0x7fb3440cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:50.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.402+0000 7fb34b228700 1 --2- 172.21.15.1:0/2029542449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3441520d0 0x7fb34414f0e0 unknown :-1 s=CLOSED pgs=3168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:50.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.402+0000 7fb34b228700 1 --2- 172.21.15.1:0/2029542449 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb34413d3e0 0x7fb344151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:50.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.402+0000 7fb34b228700 1 -- 172.21.15.1:0/2029542449 >> 172.21.15.1:0/2029542449 conn(0x7fb344099a30 msgr2=0x7fb3440b5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:50.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.402+0000 7fb34b228700 1 -- 172.21.15.1:0/2029542449 shutdown_connections 2024-03-18T20:27:50.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.402+0000 7fb34b228700 1 -- 172.21.15.1:0/2029542449 wait complete. 2024-03-18T20:27:50.418 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-03-18T20:27:50.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:50.418 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-03-18T20:27:50.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:50.441 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-03-18T20:27:50.505 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-18T20:27:50.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.669+0000 7f6fda853700 1 Processor -- start 2024-03-18T20:27:50.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.669+0000 7f6fda853700 1 -- start start 2024-03-18T20:27:50.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.670+0000 7f6fda853700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fd40b0590 0x7f6fd40b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:50.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.670+0000 7f6fda853700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6fd40b5130 0x7f6fd40aef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:50.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.670+0000 7f6fda853700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd415ed10 con 0x7f6fd415e930 2024-03-18T20:27:50.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.670+0000 7f6fda853700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd40b55c0 con 0x7f6fd40b5130 2024-03-18T20:27:50.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.670+0000 7f6fda853700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd409ad80 con 0x7f6fd40b0590 2024-03-18T20:27:50.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.670+0000 7f6fd3fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6fd415e930 0x7f6fd40b40e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58806/0 (socket says 172.21.15.1:58806) 2024-03-18T20:27:50.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.670+0000 7f6fd3fff700 1 -- 172.21.15.1:0/3047226299 learned_addr learned my addr 172.21.15.1:0/3047226299 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:50.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd37fe700 1 --2- 172.21.15.1:0/3047226299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fd40b0590 0x7f6fd40b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:50.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd8df0700 1 --2- 172.21.15.1:0/3047226299 >> v2:172.21.15.165:3300/0 conn(0x7f6fd40b5130 0x7f6fd40aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:50.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd2ffd700 1 -- 172.21.15.1:0/3047226299 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177236926 0 0) 0x7f6fc4001110 con 0x7f6fd415e930 2024-03-18T20:27:50.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd2ffd700 1 -- 172.21.15.1:0/3047226299 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fc00035b0 con 0x7f6fd415e930 2024-03-18T20:27:50.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd2ffd700 1 -- 172.21.15.1:0/3047226299 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2919279819 0 0) 0x7f6fc00035b0 con 0x7f6fd415e930 2024-03-18T20:27:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd2ffd700 1 -- 172.21.15.1:0/3047226299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fd40b0590 msgr2=0x7f6fd40b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd2ffd700 1 --2- 172.21.15.1:0/3047226299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fd40b0590 0x7f6fd40b4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd2ffd700 1 -- 172.21.15.1:0/3047226299 >> v2:172.21.15.165:3300/0 conn(0x7f6fd40b5130 msgr2=0x7f6fd40aef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd2ffd700 1 --2- 172.21.15.1:0/3047226299 >> v2:172.21.15.165:3300/0 conn(0x7f6fd40b5130 0x7f6fd40aef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd2ffd700 1 -- 172.21.15.1:0/3047226299 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd40b47f0 con 0x7f6fd415e930 2024-03-18T20:27:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd2ffd700 1 -- 172.21.15.1:0/3047226299 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f6fc4001800 con 0x7f6fd415e930 2024-03-18T20:27:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd2ffd700 1 -- 172.21.15.1:0/3047226299 >> v1:172.21.15.1:6789/0 conn(0x7f6fd415e930 legacy=0x7f6fd40b40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd2ffd700 1 --2- 172.21.15.1:0/3047226299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fd40b0590 0x7f6fc0003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd2ffd700 1 --2- 172.21.15.1:0/3047226299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fc0003f80 0x7f6fc0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd2ffd700 1 --2- 172.21.15.1:0/3047226299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fc0004d90 0x7f6fc0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd2ffd700 1 -- 172.21.15.1:0/3047226299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40b55c0 con 0x7f6fc0003f80 2024-03-18T20:27:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd2ffd700 1 -- 172.21.15.1:0/3047226299 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd409ad80 con 0x7f6fc0004d90 2024-03-18T20:27:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd2ffd700 1 -- 172.21.15.1:0/3047226299 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc000ba10 con 0x7f6fd40b0590 2024-03-18T20:27:50.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd37fe700 1 --2- 172.21.15.1:0/3047226299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fc0003f80 0x7f6fc0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:50.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd3fff700 1 --2- 172.21.15.1:0/3047226299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fd40b0590 0x7f6fc0003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:50.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd8df0700 1 --2- 172.21.15.1:0/3047226299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fc0004d90 0x7f6fc0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:50.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd37fe700 1 -- 172.21.15.1:0/3047226299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fd40b0590 msgr2=0x7f6fc0003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:50.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd37fe700 1 --2- 172.21.15.1:0/3047226299 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fd40b0590 0x7f6fc0003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:50.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd37fe700 1 -- 172.21.15.1:0/3047226299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fc0004d90 msgr2=0x7f6fc0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:50.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd37fe700 1 --2- 172.21.15.1:0/3047226299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fc0004d90 0x7f6fc0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:50.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.671+0000 7f6fd37fe700 1 -- 172.21.15.1:0/3047226299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc00035b0 con 0x7f6fc0003f80 2024-03-18T20:27:50.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.672+0000 7f6fd37fe700 1 --2- 172.21.15.1:0/3047226299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fc0003f80 0x7f6fc0004850 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc009b60 tx=0x7f6fc0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ee46e235264621a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:50.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.672+0000 7f6fd2ffd700 1 -- 172.21.15.1:0/3047226299 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc00e660 con 0x7f6fc0003f80 2024-03-18T20:27:50.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.672+0000 7f6fd2ffd700 1 -- 172.21.15.1:0/3047226299 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc00e7c0 con 0x7f6fc0003f80 2024-03-18T20:27:50.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.672+0000 7f6fd2ffd700 1 -- 172.21.15.1:0/3047226299 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc00eb30 con 0x7f6fc0003f80 2024-03-18T20:27:50.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.672+0000 7f6fda853700 1 -- 172.21.15.1:0/3047226299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fc0003f80 msgr2=0x7f6fc0004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:50.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.672+0000 7f6fda853700 1 --2- 172.21.15.1:0/3047226299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fc0003f80 0x7f6fc0004850 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc009b60 tx=0x7f6fc0005880 comp rx=0 tx=0).stop 2024-03-18T20:27:50.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.672+0000 7f6fd8df0700 1 -- 172.21.15.1:0/3047226299 reap_dead start 2024-03-18T20:27:50.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.672+0000 7f6fda853700 1 -- 172.21.15.1:0/3047226299 shutdown_connections 2024-03-18T20:27:50.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.672+0000 7f6fda853700 1 -- 172.21.15.1:0/3047226299 >> 172.21.15.1:0/3047226299 conn(0x7f6fd40b2de0 msgr2=0x7f6fd40b31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:50.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.672+0000 7f6fda853700 1 -- 172.21.15.1:0/3047226299 shutdown_connections 2024-03-18T20:27:50.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.672+0000 7f6fda853700 1 -- 172.21.15.1:0/3047226299 wait complete. 2024-03-18T20:27:50.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.674+0000 7f6fda853700 1 Processor -- start 2024-03-18T20:27:50.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.674+0000 7f6fda853700 1 -- start start 2024-03-18T20:27:50.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.674+0000 7f6fda853700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fc0003f80 0x7f6fd41f1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:50.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.674+0000 7f6fda853700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fd4142d00 0x7f6fd414f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.674+0000 7f6fda853700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fd415bf20 0x7f6fd40cf880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fda853700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd4143180 con 0x7f6fd415bf20 2024-03-18T20:27:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fda853700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40cf3f0 con 0x7f6fc0003f80 2024-03-18T20:27:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fda853700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40cd380 con 0x7f6fd4142d00 2024-03-18T20:27:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fd8df0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fd415bf20 0x7f6fd40cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fd8df0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fd415bf20 0x7f6fd40cf880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54988/0 (socket says 172.21.15.1:54988) 2024-03-18T20:27:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fd8df0700 1 -- 172.21.15.1:0/1886777301 learned_addr learned my addr 172.21.15.1:0/1886777301 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fd37fe700 1 --2- 172.21.15.1:0/1886777301 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fd4142d00 0x7f6fd414f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fd8df0700 1 -- 172.21.15.1:0/1886777301 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fd4142d00 msgr2=0x7f6fd414f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fd8df0700 1 --2- 172.21.15.1:0/1886777301 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fd4142d00 0x7f6fd414f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fd3fff700 1 --2- 172.21.15.1:0/1886777301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fc0003f80 0x7f6fd41f1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fd8df0700 1 -- 172.21.15.1:0/1886777301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fc0003f80 msgr2=0x7f6fd41f1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:50.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fd8df0700 1 --2- 172.21.15.1:0/1886777301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fc0003f80 0x7f6fd41f1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fd8df0700 1 -- 172.21.15.1:0/1886777301 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd40cd620 con 0x7f6fd415bf20 2024-03-18T20:27:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fd3fff700 1 --2- 172.21.15.1:0/1886777301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fc0003f80 0x7f6fd41f1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:27:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fd8df0700 1 --2- 172.21.15.1:0/1886777301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fd415bf20 0x7f6fd40cf880 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 crypto rx=0x7f6fd40b65b0 tx=0x7f6fcc0114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fd0ff9700 1 -- 172.21.15.1:0/1886777301 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc00e070 con 0x7f6fd415bf20 2024-03-18T20:27:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fd0ff9700 1 -- 172.21.15.1:0/1886777301 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fcc003c30 con 0x7f6fd415bf20 2024-03-18T20:27:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fda853700 1 -- 172.21.15.1:0/1886777301 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd40cf080 con 0x7f6fd415bf20 2024-03-18T20:27:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fd0ff9700 1 -- 172.21.15.1:0/1886777301 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc003de0 con 0x7f6fd415bf20 2024-03-18T20:27:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.675+0000 7f6fda853700 1 -- 172.21.15.1:0/1886777301 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd40cd900 con 0x7f6fd415bf20 2024-03-18T20:27:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.676+0000 7f6fda853700 1 -- 172.21.15.1:0/1886777301 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd40ceee0 con 0x7f6fd415bf20 2024-03-18T20:27:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.677+0000 7f6fd0ff9700 1 -- 172.21.15.1:0/1886777301 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 162) v1 ==== 89313+0+0 (secure 0 0 0) 0x7f6fcc011a10 con 0x7f6fd415bf20 2024-03-18T20:27:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.677+0000 7f6fd0ff9700 1 --2- 172.21.15.1:0/1886777301 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6fb406aec0 0x7f6fb406d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.677+0000 7f6fd0ff9700 1 -- 172.21.15.1:0/1886777301 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f6fcc089f50 con 0x7f6fd415bf20 2024-03-18T20:27:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.677+0000 7f6fd3fff700 1 --2- 172.21.15.1:0/1886777301 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6fb406aec0 0x7f6fb406d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:50.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.678+0000 7f6fd3fff700 1 --2- 172.21.15.1:0/1886777301 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6fb406aec0 0x7f6fb406d380 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4000fb0 tx=0x7f6fc4009040 comp rx=0 tx=0).ready entity=mgr.13401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:50.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.682+0000 7f6fd0ff9700 1 -- 172.21.15.1:0/1886777301 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6fd40ceee0 con 0x7f6fd415bf20 2024-03-18T20:27:50.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:50.944+0000 7f6fda853700 1 -- 172.21.15.1:0/1886777301 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f6fd40ceee0 con 0x7f6fd415bf20 2024-03-18T20:27:51.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.191+0000 7f6fd0ff9700 1 -- 172.21.15.1:0/1886777301 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 163) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f6fcc054e20 con 0x7f6fd415bf20 2024-03-18T20:27:51.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.192+0000 7f6fd0ff9700 1 -- 172.21.15.1:0/1886777301 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v163) v1 ==== 93+0+0 (secure 0 0 0) 0x7f6fcc0860c0 con 0x7f6fd415bf20 2024-03-18T20:27:51.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.194+0000 7f6fda853700 1 -- 172.21.15.1:0/1886777301 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6fb406aec0 msgr2=0x7f6fb406d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:51.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.194+0000 7f6fda853700 1 --2- 172.21.15.1:0/1886777301 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6fb406aec0 0x7f6fb406d380 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4000fb0 tx=0x7f6fc4009040 comp rx=0 tx=0).stop 2024-03-18T20:27:51.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.194+0000 7f6fda853700 1 -- 172.21.15.1:0/1886777301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fd415bf20 msgr2=0x7f6fd40cf880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:51.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.194+0000 7f6fda853700 1 --2- 172.21.15.1:0/1886777301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fd415bf20 0x7f6fd40cf880 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 crypto rx=0x7f6fd40b65b0 tx=0x7f6fcc0114b0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.194+0000 7f6fda853700 1 -- 172.21.15.1:0/1886777301 shutdown_connections 2024-03-18T20:27:51.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.194+0000 7f6fda853700 1 --2- 172.21.15.1:0/1886777301 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6fb406aec0 0x7f6fb406d380 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.195+0000 7f6fda853700 1 --2- 172.21.15.1:0/1886777301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fd415bf20 0x7f6fd40cf880 unknown :-1 s=CLOSED pgs=3171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.195+0000 7f6fda853700 1 --2- 172.21.15.1:0/1886777301 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fc0003f80 0x7f6fd41f1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.195+0000 7f6fda853700 1 --2- 172.21.15.1:0/1886777301 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fd4142d00 0x7f6fd414f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.195+0000 7f6fda853700 1 -- 172.21.15.1:0/1886777301 >> 172.21.15.1:0/1886777301 conn(0x7f6fd40b2de0 msgr2=0x7f6fd415f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:51.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.195+0000 7f6fda853700 1 -- 172.21.15.1:0/1886777301 shutdown_connections 2024-03-18T20:27:51.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.195+0000 7f6fda853700 1 -- 172.21.15.1:0/1886777301 wait complete. 2024-03-18T20:27:51.212 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T20:27:51.213 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:27:51.259 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:27:51.297 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T20:27:51.297 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:27:51.297 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:27:51.317 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:27:51.336 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:51.332+0000 7f4e60685240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:27:51.343 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:51.339+0000 7efcc9740240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:27:51.482 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:51.478+0000 7f4e60685240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:27:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.507+0000 7ff153493700 1 Processor -- start 2024-03-18T20:27:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.508+0000 7ff153493700 1 -- start start 2024-03-18T20:27:51.513 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:51.509+0000 7efcc9740240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:27:51.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.510+0000 7ff153493700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff14c0b59b0 0x7ff14c15aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:51.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.510+0000 7ff153493700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff14c0b3460 0x7ff14c06d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:51.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.510+0000 7ff153493700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c0b3c00 con 0x7ff14c153ed0 2024-03-18T20:27:51.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.510+0000 7ff153493700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff14c0b3ee0 con 0x7ff14c0b3460 2024-03-18T20:27:51.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.510+0000 7ff153493700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0b41c0 con 0x7ff14c0b59b0 2024-03-18T20:27:51.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.510+0000 7ff151a30700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff14c0b3460 0x7ff14c06d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:51.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.510+0000 7ff151a30700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff14c0b3460 0x7ff14c06d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55008/0 (socket says 172.21.15.1:55008) 2024-03-18T20:27:51.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.510+0000 7ff151a30700 1 -- 172.21.15.1:0/2105968651 learned_addr learned my addr 172.21.15.1:0/2105968651 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:51.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.510+0000 7ff150a2e700 1 --2- 172.21.15.1:0/2105968651 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff14c0b59b0 0x7ff14c15aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:51.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.510+0000 7ff151a30700 1 -- 172.21.15.1:0/2105968651 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff14c0b59b0 msgr2=0x7ff14c15aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:51.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.510+0000 7ff151a30700 1 --2- 172.21.15.1:0/2105968651 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff14c0b59b0 0x7ff14c15aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.510+0000 7ff151a30700 1 -- 172.21.15.1:0/2105968651 >> v1:172.21.15.165:6789/0 conn(0x7ff14c153ed0 legacy=0x7ff14c0b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.510+0000 7ff151a30700 1 -- 172.21.15.1:0/2105968651 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c0b44a0 con 0x7ff14c0b3460 2024-03-18T20:27:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.510+0000 7ff151a30700 1 --2- 172.21.15.1:0/2105968651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff14c0b3460 0x7ff14c06d740 secure :-1 s=READY pgs=3174 cs=0 l=1 rev1=1 crypto rx=0x7ff144009e50 tx=0x7ff14400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbbf99fe51328cd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.510+0000 7ff143fff700 1 -- 172.21.15.1:0/2105968651 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14400e070 con 0x7ff14c0b3460 2024-03-18T20:27:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.510+0000 7ff143fff700 1 -- 172.21.15.1:0/2105968651 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14400b910 con 0x7ff14c0b3460 2024-03-18T20:27:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.511+0000 7ff143fff700 1 -- 172.21.15.1:0/2105968651 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14400bcc0 con 0x7ff14c0b3460 2024-03-18T20:27:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.511+0000 7ff153493700 1 -- 172.21.15.1:0/2105968651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff14c0b3460 msgr2=0x7ff14c06d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.511+0000 7ff153493700 1 --2- 172.21.15.1:0/2105968651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff14c0b3460 0x7ff14c06d740 secure :-1 s=READY pgs=3174 cs=0 l=1 rev1=1 crypto rx=0x7ff144009e50 tx=0x7ff14400b330 comp rx=0 tx=0).stop 2024-03-18T20:27:51.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.511+0000 7ff153493700 1 -- 172.21.15.1:0/2105968651 shutdown_connections 2024-03-18T20:27:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.511+0000 7ff153493700 1 --2- 172.21.15.1:0/2105968651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff14c0b3460 0x7ff14c06d740 unknown :-1 s=CLOSED pgs=3174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.511+0000 7ff153493700 1 --2- 172.21.15.1:0/2105968651 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff14c0b59b0 0x7ff14c15aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.511+0000 7ff153493700 1 -- 172.21.15.1:0/2105968651 >> 172.21.15.1:0/2105968651 conn(0x7ff14c072010 msgr2=0x7ff14c06a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.511+0000 7ff153493700 1 -- 172.21.15.1:0/2105968651 shutdown_connections 2024-03-18T20:27:51.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.511+0000 7ff153493700 1 -- 172.21.15.1:0/2105968651 wait complete. 2024-03-18T20:27:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.512+0000 7ff153493700 1 Processor -- start 2024-03-18T20:27:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.512+0000 7ff153493700 1 -- start start 2024-03-18T20:27:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.513+0000 7ff153493700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff14c0b3460 0x7ff14c14a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.513+0000 7ff153493700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff14c0b59b0 0x7ff14c1548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.513+0000 7ff153493700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff14c153ed0 0x7ff14c154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.513+0000 7ff153493700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c14b050 con 0x7ff14c153ed0 2024-03-18T20:27:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.513+0000 7ff153493700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c150be0 con 0x7ff14c0b59b0 2024-03-18T20:27:51.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.513+0000 7ff153493700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c150ec0 con 0x7ff14c0b3460 2024-03-18T20:27:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.513+0000 7ff151a30700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff14c153ed0 0x7ff14c154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.513+0000 7ff151a30700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff14c153ed0 0x7ff14c154df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55016/0 (socket says 172.21.15.1:55016) 2024-03-18T20:27:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.513+0000 7ff151a30700 1 -- 172.21.15.1:0/3827368270 learned_addr learned my addr 172.21.15.1:0/3827368270 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.513+0000 7ff15122f700 1 --2- 172.21.15.1:0/3827368270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff14c0b3460 0x7ff14c14a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.513+0000 7ff150a2e700 1 --2- 172.21.15.1:0/3827368270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff14c0b59b0 0x7ff14c1548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.514+0000 7ff151a30700 1 -- 172.21.15.1:0/3827368270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff14c0b3460 msgr2=0x7ff14c14a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.514+0000 7ff151a30700 1 --2- 172.21.15.1:0/3827368270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff14c0b3460 0x7ff14c14a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.514+0000 7ff151a30700 1 -- 172.21.15.1:0/3827368270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff14c0b59b0 msgr2=0x7ff14c1548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.514+0000 7ff151a30700 1 --2- 172.21.15.1:0/3827368270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff14c0b59b0 0x7ff14c1548b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.514+0000 7ff151a30700 1 -- 172.21.15.1:0/3827368270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff144009b30 con 0x7ff14c153ed0 2024-03-18T20:27:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.514+0000 7ff151a30700 1 --2- 172.21.15.1:0/3827368270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff14c153ed0 0x7ff14c154df0 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 crypto rx=0x7ff14c06c270 tx=0x7ff144004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:51.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.514+0000 7ff141ffb700 1 -- 172.21.15.1:0/3827368270 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff144004800 con 0x7ff14c153ed0 2024-03-18T20:27:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.514+0000 7ff141ffb700 1 -- 172.21.15.1:0/3827368270 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff144004960 con 0x7ff14c153ed0 2024-03-18T20:27:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.514+0000 7ff141ffb700 1 -- 172.21.15.1:0/3827368270 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14401c680 con 0x7ff14c153ed0 2024-03-18T20:27:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.514+0000 7ff153493700 1 -- 172.21.15.1:0/3827368270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14c1511a0 con 0x7ff14c153ed0 2024-03-18T20:27:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.514+0000 7ff153493700 1 -- 172.21.15.1:0/3827368270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14c151760 con 0x7ff14c153ed0 2024-03-18T20:27:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.515+0000 7ff153493700 1 -- 172.21.15.1:0/3827368270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14c151980 con 0x7ff14c153ed0 2024-03-18T20:27:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.516+0000 7ff141ffb700 1 -- 172.21.15.1:0/3827368270 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 163) v1 ==== 89296+0+0 (secure 0 0 0) 0x7ff14401c7e0 con 0x7ff14c153ed0 2024-03-18T20:27:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.517+0000 7ff141ffb700 1 --2- 172.21.15.1:0/3827368270 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff13006ae40 0x7ff13006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.517+0000 7ff141ffb700 1 -- 172.21.15.1:0/3827368270 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7ff14408b5d0 con 0x7ff14c153ed0 2024-03-18T20:27:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.517+0000 7ff15122f700 1 -- 172.21.15.1:0/3827368270 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff13006ae40 msgr2=0x7ff13006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:51.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.517+0000 7ff15122f700 1 --2- 172.21.15.1:0/3827368270 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff13006ae40 0x7ff13006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:27:51.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.521+0000 7ff141ffb700 1 -- 172.21.15.1:0/3827368270 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff144056160 con 0x7ff14c153ed0 2024-03-18T20:27:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.718+0000 7ff15122f700 1 -- 172.21.15.1:0/3827368270 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff13006ae40 msgr2=0x7ff13006d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:51.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.718+0000 7ff15122f700 1 --2- 172.21.15.1:0/3827368270 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff13006ae40 0x7ff13006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:27:51.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.727+0000 7ff153493700 1 -- 172.21.15.1:0/3827368270 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff14c067dc0 con 0x7ff14c153ed0 2024-03-18T20:27:51.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.728+0000 7ff141ffb700 1 -- 172.21.15.1:0/3827368270 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v275) v1 ==== 74+0+16970 (secure 0 0 0) 0x7ff144059780 con 0x7ff14c153ed0 2024-03-18T20:27:51.732 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:51.732 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":275,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:27:42.143916+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:27:19.891286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"273","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-03-18T20:27:21.027315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"274","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3635558645":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2293377213":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/1796830610":"2024-03-19T20:27:42.143876+0000","172.21.15.1:6833/148356":"2024-03-19T20:27:42.143876+0000","172.21.15.1:6832/148356":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2068313472":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2668486132":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2560935779":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/82444607":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/1446581052":"2024-03-19T20:27:42.143876+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:27:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.731+0000 7ff153493700 1 -- 172.21.15.1:0/3827368270 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff13006ae40 msgr2=0x7ff13006d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:27:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.731+0000 7ff153493700 1 --2- 172.21.15.1:0/3827368270 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff13006ae40 0x7ff13006d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.731+0000 7ff153493700 1 -- 172.21.15.1:0/3827368270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff14c153ed0 msgr2=0x7ff14c154df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.731+0000 7ff153493700 1 --2- 172.21.15.1:0/3827368270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff14c153ed0 0x7ff14c154df0 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 crypto rx=0x7ff14c06c270 tx=0x7ff144004ad0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.731+0000 7ff153493700 1 -- 172.21.15.1:0/3827368270 shutdown_connections 2024-03-18T20:27:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.731+0000 7ff153493700 1 --2- 172.21.15.1:0/3827368270 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ff13006ae40 0x7ff13006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.731+0000 7ff153493700 1 --2- 172.21.15.1:0/3827368270 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ff14c0b3460 0x7ff14c14a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.731+0000 7ff153493700 1 --2- 172.21.15.1:0/3827368270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff14c153ed0 0x7ff14c154df0 unknown :-1 s=CLOSED pgs=3175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.731+0000 7ff153493700 1 --2- 172.21.15.1:0/3827368270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff14c0b59b0 0x7ff14c1548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.731+0000 7ff153493700 1 -- 172.21.15.1:0/3827368270 >> 172.21.15.1:0/3827368270 conn(0x7ff14c072010 msgr2=0x7ff14c06a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:51.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.731+0000 7ff153493700 1 -- 172.21.15.1:0/3827368270 shutdown_connections 2024-03-18T20:27:51.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.731+0000 7ff153493700 1 -- 172.21.15.1:0/3827368270 wait complete. 2024-03-18T20:27:51.748 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:27:19.891286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '273', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:27:21.027315+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '274', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:27:51.749 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:27:51.749 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:51.744+0000 7f4e60685240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:27:51.797 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:51.794+0000 7efcc9740240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:27:51.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.875+0000 7fbf8c86b700 1 Processor -- start 2024-03-18T20:27:51.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.875+0000 7fbf8c86b700 1 -- start start 2024-03-18T20:27:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.876+0000 7fbf8c86b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbf8815a350 0x7fbf880b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.876+0000 7fbf8c86b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf880b26e0 0x7fbf880b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.876+0000 7fbf8c86b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf880b48f0 0x7fbf880b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.876+0000 7fbf8c86b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf8806fb70 con 0x7fbf880b48f0 2024-03-18T20:27:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.876+0000 7fbf8c86b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf880aeb10 con 0x7fbf8815a350 2024-03-18T20:27:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.876+0000 7fbf8c86b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf880b2c50 con 0x7fbf880b26e0 2024-03-18T20:27:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.876+0000 7fbf86d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf880b48f0 0x7fbf880b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.876+0000 7fbf86d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbf880b48f0 0x7fbf880b53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55028/0 (socket says 172.21.15.1:55028) 2024-03-18T20:27:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.876+0000 7fbf86d9d700 1 -- 172.21.15.1:0/862365730 learned_addr learned my addr 172.21.15.1:0/862365730 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:51.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.876+0000 7fbf8659c700 1 --2- 172.21.15.1:0/862365730 >> v2:172.21.15.165:3300/0 conn(0x7fbf8815a350 0x7fbf880b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:51.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.876+0000 7fbf85d9b700 1 --2- 172.21.15.1:0/862365730 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf880b26e0 0x7fbf880b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:51.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.877+0000 7fbf86d9d700 1 -- 172.21.15.1:0/862365730 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf880b26e0 msgr2=0x7fbf880b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:51.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.877+0000 7fbf86d9d700 1 --2- 172.21.15.1:0/862365730 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf880b26e0 0x7fbf880b4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.877+0000 7fbf86d9d700 1 -- 172.21.15.1:0/862365730 >> v2:172.21.15.165:3300/0 conn(0x7fbf8815a350 msgr2=0x7fbf880b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:51.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.877+0000 7fbf86d9d700 1 --2- 172.21.15.1:0/862365730 >> v2:172.21.15.165:3300/0 conn(0x7fbf8815a350 0x7fbf880b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.877+0000 7fbf86d9d700 1 -- 172.21.15.1:0/862365730 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8806f5b0 con 0x7fbf880b48f0 2024-03-18T20:27:51.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.877+0000 7fbf86d9d700 1 --2- 172.21.15.1:0/862365730 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf880b48f0 0x7fbf880b53a0 secure :-1 s=READY pgs=3176 cs=0 l=1 rev1=1 crypto rx=0x7fbf880729e0 tx=0x7fbf8000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90c1d89789e08252 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:51.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.877+0000 7fbf8559a700 1 -- 172.21.15.1:0/862365730 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8000e070 con 0x7fbf880b48f0 2024-03-18T20:27:51.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.877+0000 7fbf8559a700 1 -- 172.21.15.1:0/862365730 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8000b910 con 0x7fbf880b48f0 2024-03-18T20:27:51.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.877+0000 7fbf8559a700 1 -- 172.21.15.1:0/862365730 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8000bcc0 con 0x7fbf880b48f0 2024-03-18T20:27:51.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.878+0000 7fbf8c86b700 1 -- 172.21.15.1:0/862365730 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf880b48f0 msgr2=0x7fbf880b53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:51.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.878+0000 7fbf8c86b700 1 --2- 172.21.15.1:0/862365730 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf880b48f0 0x7fbf880b53a0 secure :-1 s=READY pgs=3176 cs=0 l=1 rev1=1 crypto rx=0x7fbf880729e0 tx=0x7fbf8000b330 comp rx=0 tx=0).stop 2024-03-18T20:27:51.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.878+0000 7fbf8c86b700 1 -- 172.21.15.1:0/862365730 shutdown_connections 2024-03-18T20:27:51.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.878+0000 7fbf8c86b700 1 --2- 172.21.15.1:0/862365730 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf880b48f0 0x7fbf880b53a0 unknown :-1 s=CLOSED pgs=3176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.878+0000 7fbf8c86b700 1 --2- 172.21.15.1:0/862365730 >> v2:172.21.15.165:3300/0 conn(0x7fbf8815a350 0x7fbf880b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.878+0000 7fbf8c86b700 1 --2- 172.21.15.1:0/862365730 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf880b26e0 0x7fbf880b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.878+0000 7fbf8c86b700 1 -- 172.21.15.1:0/862365730 >> 172.21.15.1:0/862365730 conn(0x7fbf880b5a10 msgr2=0x7fbf880b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:51.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.878+0000 7fbf8c86b700 1 -- 172.21.15.1:0/862365730 shutdown_connections 2024-03-18T20:27:51.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.878+0000 7fbf8c86b700 1 -- 172.21.15.1:0/862365730 wait complete. 2024-03-18T20:27:51.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.879+0000 7fbf8c86b700 1 Processor -- start 2024-03-18T20:27:51.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.879+0000 7fbf8c86b700 1 -- start start 2024-03-18T20:27:51.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.880+0000 7fbf8c86b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf880b26e0 0x7fbf88146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:51.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.880+0000 7fbf8c86b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf880b48f0 0x7fbf881e0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:51.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.880+0000 7fbf8c86b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf8815a350 0x7fbf881e5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:51.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.880+0000 7fbf8c86b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf88147110 con 0x7fbf880b48f0 2024-03-18T20:27:51.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.880+0000 7fbf8c86b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf881e17a0 con 0x7fbf8815a350 2024-03-18T20:27:51.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.880+0000 7fbf8c86b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf881e1a80 con 0x7fbf880b26e0 2024-03-18T20:27:51.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.880+0000 7fbf85d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf880b48f0 0x7fbf881e0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:51.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.880+0000 7fbf85d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf880b48f0 0x7fbf881e0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55032/0 (socket says 172.21.15.1:55032) 2024-03-18T20:27:51.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.880+0000 7fbf85d9b700 1 -- 172.21.15.1:0/952915468 learned_addr learned my addr 172.21.15.1:0/952915468 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:51.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.880+0000 7fbf8659c700 1 --2- 172.21.15.1:0/952915468 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf880b26e0 0x7fbf88146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:51.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.880+0000 7fbf86d9d700 1 --2- 172.21.15.1:0/952915468 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf8815a350 0x7fbf881e5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:51.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.881+0000 7fbf85d9b700 1 -- 172.21.15.1:0/952915468 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf880b26e0 msgr2=0x7fbf88146bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:51.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.881+0000 7fbf85d9b700 1 --2- 172.21.15.1:0/952915468 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf880b26e0 0x7fbf88146bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.881+0000 7fbf85d9b700 1 -- 172.21.15.1:0/952915468 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf8815a350 msgr2=0x7fbf881e5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:51.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.881+0000 7fbf85d9b700 1 --2- 172.21.15.1:0/952915468 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf8815a350 0x7fbf881e5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:51.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.881+0000 7fbf85d9b700 1 -- 172.21.15.1:0/952915468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf80009b30 con 0x7fbf880b48f0 2024-03-18T20:27:51.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.881+0000 7fbf8659c700 1 --2- 172.21.15.1:0/952915468 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf880b26e0 0x7fbf88146bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:27:51.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.881+0000 7fbf86d9d700 1 --2- 172.21.15.1:0/952915468 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf8815a350 0x7fbf881e5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:27:51.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.881+0000 7fbf85d9b700 1 --2- 172.21.15.1:0/952915468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf880b48f0 0x7fbf881e0de0 secure :-1 s=READY pgs=3177 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00e930 tx=0x7fbf7c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:51.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.881+0000 7fbf76ffd700 1 -- 172.21.15.1:0/952915468 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c013680 con 0x7fbf880b48f0 2024-03-18T20:27:51.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.881+0000 7fbf76ffd700 1 -- 172.21.15.1:0/952915468 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7c0137e0 con 0x7fbf880b48f0 2024-03-18T20:27:51.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.881+0000 7fbf8c86b700 1 -- 172.21.15.1:0/952915468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf881e1d60 con 0x7fbf880b48f0 2024-03-18T20:27:51.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.881+0000 7fbf76ffd700 1 -- 172.21.15.1:0/952915468 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c013b70 con 0x7fbf880b48f0 2024-03-18T20:27:51.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.881+0000 7fbf8c86b700 1 -- 172.21.15.1:0/952915468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf881e2320 con 0x7fbf880b48f0 2024-03-18T20:27:51.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.882+0000 7fbf8c86b700 1 -- 172.21.15.1:0/952915468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf88067dc0 con 0x7fbf880b48f0 2024-03-18T20:27:51.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.883+0000 7fbf76ffd700 1 -- 172.21.15.1:0/952915468 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 163) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fbf7c013cd0 con 0x7fbf880b48f0 2024-03-18T20:27:51.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.883+0000 7fbf76ffd700 1 --2- 172.21.15.1:0/952915468 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf6806ae40 0x7fbf6806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:51.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.883+0000 7fbf8659c700 1 -- 172.21.15.1:0/952915468 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf6806ae40 msgr2=0x7fbf6806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:51.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.883+0000 7fbf76ffd700 1 -- 172.21.15.1:0/952915468 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7fbf7c0898c0 con 0x7fbf880b48f0 2024-03-18T20:27:51.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.883+0000 7fbf8659c700 1 --2- 172.21.15.1:0/952915468 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf6806ae40 0x7fbf6806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:27:51.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:51.887+0000 7fbf76ffd700 1 -- 172.21.15.1:0/952915468 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbf7c0544d0 con 0x7fbf880b48f0 2024-03-18T20:27:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.084+0000 7fbf8659c700 1 -- 172.21.15.1:0/952915468 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf6806ae40 msgr2=0x7fbf6806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:52.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.084+0000 7fbf8659c700 1 --2- 172.21.15.1:0/952915468 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf6806ae40 0x7fbf6806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:27:52.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.120+0000 7fbf8c86b700 1 -- 172.21.15.1:0/952915468 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbf8809b9a0 con 0x7fbf880b48f0 2024-03-18T20:27:52.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.121+0000 7fbf76ffd700 1 -- 172.21.15.1:0/952915468 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v275) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbf7c057af0 con 0x7fbf880b48f0 2024-03-18T20:27:52.125 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:27:52.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.123+0000 7fbf8c86b700 1 -- 172.21.15.1:0/952915468 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf6806ae40 msgr2=0x7fbf6806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:27:52.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.123+0000 7fbf8c86b700 1 --2- 172.21.15.1:0/952915468 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf6806ae40 0x7fbf6806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.123+0000 7fbf8c86b700 1 -- 172.21.15.1:0/952915468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf880b48f0 msgr2=0x7fbf881e0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:52.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.123+0000 7fbf8c86b700 1 --2- 172.21.15.1:0/952915468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf880b48f0 0x7fbf881e0de0 secure :-1 s=READY pgs=3177 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00e930 tx=0x7fbf7c00ec40 comp rx=0 tx=0).stop 2024-03-18T20:27:52.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.123+0000 7fbf8c86b700 1 -- 172.21.15.1:0/952915468 shutdown_connections 2024-03-18T20:27:52.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.123+0000 7fbf8c86b700 1 --2- 172.21.15.1:0/952915468 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf6806ae40 0x7fbf6806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.123+0000 7fbf8c86b700 1 --2- 172.21.15.1:0/952915468 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbf880b26e0 0x7fbf88146bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.123+0000 7fbf8c86b700 1 --2- 172.21.15.1:0/952915468 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf8815a350 0x7fbf881e5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.123+0000 7fbf8c86b700 1 --2- 172.21.15.1:0/952915468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbf880b48f0 0x7fbf881e0de0 unknown :-1 s=CLOSED pgs=3177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.123+0000 7fbf8c86b700 1 -- 172.21.15.1:0/952915468 >> 172.21.15.1:0/952915468 conn(0x7fbf880b5a10 msgr2=0x7fbf88159650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:52.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.123+0000 7fbf8c86b700 1 -- 172.21.15.1:0/952915468 shutdown_connections 2024-03-18T20:27:52.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.123+0000 7fbf8c86b700 1 -- 172.21.15.1:0/952915468 wait complete. 2024-03-18T20:27:52.140 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:27:52.173 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:52.169+0000 7f4e60685240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:27:52.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.267+0000 7fb4fc92e700 1 Processor -- start 2024-03-18T20:27:52.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.267+0000 7fb4fc92e700 1 -- start start 2024-03-18T20:27:52.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.269+0000 7fb4fc92e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f80d3f80 0x7fb4f815d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:52.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.269+0000 7fb4fc92e700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb4f815df00 0x7fb4f80cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:52.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.269+0000 7fb4fc92e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f80cfe30 con 0x7fb4f815d5e0 2024-03-18T20:27:52.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.269+0000 7fb4fc92e700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f80d24e0 con 0x7fb4f815df00 2024-03-18T20:27:52.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.269+0000 7fb4fc92e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f80d0110 con 0x7fb4f80d3f80 2024-03-18T20:27:52.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.269+0000 7fb4f5d9b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f80d3f80 0x7fb4f815d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:52.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.269+0000 7fb4f6d9d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb4f815df00 0x7fb4f80cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:52.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.269+0000 7fb4f659c700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fb4f815d5e0 0x7fb4f80d1dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:60652/0 (socket says 172.21.15.1:60652) 2024-03-18T20:27:52.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.269+0000 7fb4f659c700 1 -- 172.21.15.1:0/1788902831 learned_addr learned my addr 172.21.15.1:0/1788902831 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:52.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.269+0000 7fb4f6d9d700 1 -- 172.21.15.1:0/1788902831 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f80d3f80 msgr2=0x7fb4f815d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:52.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.269+0000 7fb4f6d9d700 1 --2- 172.21.15.1:0/1788902831 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f80d3f80 0x7fb4f815d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.269+0000 7fb4f6d9d700 1 -- 172.21.15.1:0/1788902831 >> v1:172.21.15.165:6789/0 conn(0x7fb4f815d5e0 legacy=0x7fb4f80d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:52.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.269+0000 7fb4f559a700 1 -- 172.21.15.1:0/1788902831 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1796512397 0 0) 0x7fb4f80cfe30 con 0x7fb4f815d5e0 2024-03-18T20:27:52.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.269+0000 7fb4f6d9d700 1 -- 172.21.15.1:0/1788902831 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f80d03f0 con 0x7fb4f815df00 2024-03-18T20:27:52.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.269+0000 7fb4f5d9b700 1 --2- 172.21.15.1:0/1788902831 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f80d3f80 0x7fb4f815d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:27:52.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.270+0000 7fb4f6d9d700 1 --2- 172.21.15.1:0/1788902831 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4f815df00 0x7fb4f80cdf90 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009d90 tx=0x7fb4f000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6735c474d9141949 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.270+0000 7fb4f559a700 1 -- 172.21.15.1:0/1788902831 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f000e070 con 0x7fb4f815df00 2024-03-18T20:27:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.270+0000 7fb4f559a700 1 -- 172.21.15.1:0/1788902831 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f000bb20 con 0x7fb4f815df00 2024-03-18T20:27:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.270+0000 7fb4f559a700 1 -- 172.21.15.1:0/1788902831 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0012430 con 0x7fb4f815df00 2024-03-18T20:27:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.270+0000 7fb4fc92e700 1 -- 172.21.15.1:0/1788902831 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4f815df00 msgr2=0x7fb4f80cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.270+0000 7fb4fc92e700 1 --2- 172.21.15.1:0/1788902831 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4f815df00 0x7fb4f80cdf90 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009d90 tx=0x7fb4f000b330 comp rx=0 tx=0).stop 2024-03-18T20:27:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.270+0000 7fb4fc92e700 1 -- 172.21.15.1:0/1788902831 shutdown_connections 2024-03-18T20:27:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.270+0000 7fb4fc92e700 1 --2- 172.21.15.1:0/1788902831 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4f815df00 0x7fb4f80cdf90 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.271+0000 7fb4fc92e700 1 --2- 172.21.15.1:0/1788902831 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f80d3f80 0x7fb4f815d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.271+0000 7fb4fc92e700 1 -- 172.21.15.1:0/1788902831 >> 172.21.15.1:0/1788902831 conn(0x7fb4f80d3380 msgr2=0x7fb4f80d3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.271+0000 7fb4fc92e700 1 -- 172.21.15.1:0/1788902831 shutdown_connections 2024-03-18T20:27:52.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.271+0000 7fb4fc92e700 1 -- 172.21.15.1:0/1788902831 wait complete. 2024-03-18T20:27:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.272+0000 7fb4fc92e700 1 Processor -- start 2024-03-18T20:27:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.272+0000 7fb4fc92e700 1 -- start start 2024-03-18T20:27:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.273+0000 7fb4fc92e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4f80d3f80 0x7fb4f815a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.273+0000 7fb4fc92e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4f815d5e0 0x7fb4f81e7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.273+0000 7fb4fc92e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f815df00 0x7fb4f81608d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.273+0000 7fb4fc92e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f80cb840 con 0x7fb4f815d5e0 2024-03-18T20:27:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.273+0000 7fb4fc92e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f81e7ec0 con 0x7fb4f80d3f80 2024-03-18T20:27:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.273+0000 7fb4fc92e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f815f0a0 con 0x7fb4f815df00 2024-03-18T20:27:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.273+0000 7fb4f5d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4f815d5e0 0x7fb4f81e7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.273+0000 7fb4f659c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4f80d3f80 0x7fb4f815a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:52.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.273+0000 7fb4f5d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4f815d5e0 0x7fb4f81e7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55046/0 (socket says 172.21.15.1:55046) 2024-03-18T20:27:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.273+0000 7fb4f5d9b700 1 -- 172.21.15.1:0/3707729548 learned_addr learned my addr 172.21.15.1:0/3707729548 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.273+0000 7fb4f6d9d700 1 --2- 172.21.15.1:0/3707729548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f815df00 0x7fb4f81608d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.273+0000 7fb4f659c700 1 -- 172.21.15.1:0/3707729548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f815df00 msgr2=0x7fb4f81608d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.273+0000 7fb4f659c700 1 --2- 172.21.15.1:0/3707729548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f815df00 0x7fb4f81608d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.273+0000 7fb4f659c700 1 -- 172.21.15.1:0/3707729548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4f815d5e0 msgr2=0x7fb4f81e7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.273+0000 7fb4f659c700 1 --2- 172.21.15.1:0/3707729548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4f815d5e0 0x7fb4f81e7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.273+0000 7fb4f659c700 1 -- 172.21.15.1:0/3707729548 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec0094a0 con 0x7fb4f80d3f80 2024-03-18T20:27:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.273+0000 7fb4f659c700 1 --2- 172.21.15.1:0/3707729548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4f80d3f80 0x7fb4f815a440 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb4e800c4b0 tx=0x7fb4e800c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.274+0000 7fb4e6ffd700 1 -- 172.21.15.1:0/3707729548 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8004c80 con 0x7fb4f80d3f80 2024-03-18T20:27:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.274+0000 7fb4e6ffd700 1 -- 172.21.15.1:0/3707729548 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e8004e00 con 0x7fb4f80d3f80 2024-03-18T20:27:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.274+0000 7fb4e6ffd700 1 -- 172.21.15.1:0/3707729548 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8007310 con 0x7fb4f80d3f80 2024-03-18T20:27:52.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.274+0000 7fb4fc92e700 1 -- 172.21.15.1:0/3707729548 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f0009a70 con 0x7fb4f80d3f80 2024-03-18T20:27:52.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.274+0000 7fb4fc92e700 1 -- 172.21.15.1:0/3707729548 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f80cb4d0 con 0x7fb4f80d3f80 2024-03-18T20:27:52.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.275+0000 7fb4fc92e700 1 -- 172.21.15.1:0/3707729548 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f80cbb20 con 0x7fb4f80d3f80 2024-03-18T20:27:52.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.275+0000 7fb4e6ffd700 1 -- 172.21.15.1:0/3707729548 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 163) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fb4e8007470 con 0x7fb4f80d3f80 2024-03-18T20:27:52.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.275+0000 7fb4e6ffd700 1 --2- 172.21.15.1:0/3707729548 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb4d806ae40 0x7fb4d806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:52.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.275+0000 7fb4f5d9b700 1 -- 172.21.15.1:0/3707729548 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb4d806ae40 msgr2=0x7fb4d806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:52.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.276+0000 7fb4e6ffd700 1 -- 172.21.15.1:0/3707729548 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7fb4e8089570 con 0x7fb4f80d3f80 2024-03-18T20:27:52.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.276+0000 7fb4f5d9b700 1 --2- 172.21.15.1:0/3707729548 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb4d806ae40 0x7fb4d806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:27:52.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.280+0000 7fb4e6ffd700 1 -- 172.21.15.1:0/3707729548 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb4e8054240 con 0x7fb4f80d3f80 2024-03-18T20:27:52.286 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:52.282+0000 7efcc9740240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:27:52.383 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:52.379+0000 7f4e60685240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:27:52.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.477+0000 7fb4f5d9b700 1 -- 172.21.15.1:0/3707729548 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb4d806ae40 msgr2=0x7fb4d806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:52.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.477+0000 7fb4f5d9b700 1 --2- 172.21.15.1:0/3707729548 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb4d806ae40 0x7fb4d806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:27:52.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.485+0000 7fb4fc92e700 1 -- 172.21.15.1:0/3707729548 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb4f8067dc0 con 0x7fb4f80d3f80 2024-03-18T20:27:52.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.485+0000 7fb4e6ffd700 1 -- 172.21.15.1:0/3707729548 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v275) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb4e8057860 con 0x7fb4f80d3f80 2024-03-18T20:27:52.490 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:27:52.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.487+0000 7fb4fc92e700 1 -- 172.21.15.1:0/3707729548 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb4d806ae40 msgr2=0x7fb4d806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:27:52.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.487+0000 7fb4fc92e700 1 --2- 172.21.15.1:0/3707729548 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb4d806ae40 0x7fb4d806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.487+0000 7fb4fc92e700 1 -- 172.21.15.1:0/3707729548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4f80d3f80 msgr2=0x7fb4f815a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:52.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.487+0000 7fb4fc92e700 1 --2- 172.21.15.1:0/3707729548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4f80d3f80 0x7fb4f815a440 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb4e800c4b0 tx=0x7fb4e800c7c0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.487+0000 7fb4fc92e700 1 -- 172.21.15.1:0/3707729548 shutdown_connections 2024-03-18T20:27:52.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.487+0000 7fb4fc92e700 1 --2- 172.21.15.1:0/3707729548 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fb4d806ae40 0x7fb4d806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.487+0000 7fb4fc92e700 1 --2- 172.21.15.1:0/3707729548 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fb4f815df00 0x7fb4f81608d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.487+0000 7fb4fc92e700 1 --2- 172.21.15.1:0/3707729548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4f80d3f80 0x7fb4f815a440 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.487+0000 7fb4fc92e700 1 --2- 172.21.15.1:0/3707729548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4f815d5e0 0x7fb4f81e7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.487+0000 7fb4fc92e700 1 -- 172.21.15.1:0/3707729548 >> 172.21.15.1:0/3707729548 conn(0x7fb4f80d3380 msgr2=0x7fb4f80b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:52.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.487+0000 7fb4fc92e700 1 -- 172.21.15.1:0/3707729548 shutdown_connections 2024-03-18T20:27:52.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.487+0000 7fb4fc92e700 1 -- 172.21.15.1:0/3707729548 wait complete. 2024-03-18T20:27:52.504 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:27:52.507 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:52.504+0000 7efcc9740240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:27:52.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.630+0000 7f5f72855700 1 Processor -- start 2024-03-18T20:27:52.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.631+0000 7f5f72855700 1 -- start start 2024-03-18T20:27:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.631+0000 7f5f72855700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5f6c158b80 0x7f5f6c15b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.631+0000 7f5f72855700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5f6c158f60 0x7f5f6c0cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.631+0000 7f5f72855700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f6c06b5e0 con 0x7f5f6c0cbf60 2024-03-18T20:27:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.631+0000 7f5f72855700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f6c15bca0 con 0x7f5f6c158b80 2024-03-18T20:27:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.631+0000 7f5f72855700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c15bf40 con 0x7f5f6c158f60 2024-03-18T20:27:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.632+0000 7f5f6bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5f6c158b80 0x7f5f6c15b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.632+0000 7f5f6bfff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5f6c158b80 0x7f5f6c15b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55058/0 (socket says 172.21.15.1:55058) 2024-03-18T20:27:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.632+0000 7f5f6bfff700 1 -- 172.21.15.1:0/848159566 learned_addr learned my addr 172.21.15.1:0/848159566 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.632+0000 7f5f6b7fe700 1 --2- 172.21.15.1:0/848159566 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5f6c158f60 0x7f5f6c0cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:52.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.632+0000 7f5f6bfff700 1 -- 172.21.15.1:0/848159566 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5f6c158f60 msgr2=0x7f5f6c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.632+0000 7f5f6bfff700 1 --2- 172.21.15.1:0/848159566 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5f6c158f60 0x7f5f6c0cba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.632+0000 7f5f6bfff700 1 -- 172.21.15.1:0/848159566 >> v1:172.21.15.165:6789/0 conn(0x7f5f6c0cbf60 legacy=0x7f5f6c0b1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.632+0000 7f5f6bfff700 1 -- 172.21.15.1:0/848159566 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f6c15c220 con 0x7f5f6c158b80 2024-03-18T20:27:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.632+0000 7f5f6bfff700 1 --2- 172.21.15.1:0/848159566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f6c158b80 0x7f5f6c15b760 secure :-1 s=READY pgs=3178 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c0098f0 tx=0x7f5f5c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27fd14ed90de4178 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.632+0000 7f5f6affd700 1 -- 172.21.15.1:0/848159566 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c011830 con 0x7f5f6c158b80 2024-03-18T20:27:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.632+0000 7f5f6affd700 1 -- 172.21.15.1:0/848159566 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5c011990 con 0x7f5f6c158b80 2024-03-18T20:27:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.632+0000 7f5f6affd700 1 -- 172.21.15.1:0/848159566 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c011d20 con 0x7f5f6c158b80 2024-03-18T20:27:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.633+0000 7f5f72855700 1 -- 172.21.15.1:0/848159566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f6c158b80 msgr2=0x7f5f6c15b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.633+0000 7f5f72855700 1 --2- 172.21.15.1:0/848159566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f6c158b80 0x7f5f6c15b760 secure :-1 s=READY pgs=3178 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c0098f0 tx=0x7f5f5c009c00 comp rx=0 tx=0).stop 2024-03-18T20:27:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.633+0000 7f5f72855700 1 -- 172.21.15.1:0/848159566 shutdown_connections 2024-03-18T20:27:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.633+0000 7f5f72855700 1 --2- 172.21.15.1:0/848159566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f6c158b80 0x7f5f6c15b760 unknown :-1 s=CLOSED pgs=3178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.633+0000 7f5f72855700 1 --2- 172.21.15.1:0/848159566 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5f6c158f60 0x7f5f6c0cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.633+0000 7f5f72855700 1 -- 172.21.15.1:0/848159566 >> 172.21.15.1:0/848159566 conn(0x7f5f6c06f030 msgr2=0x7f5f6c06a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.633+0000 7f5f72855700 1 -- 172.21.15.1:0/848159566 shutdown_connections 2024-03-18T20:27:52.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.633+0000 7f5f72855700 1 -- 172.21.15.1:0/848159566 wait complete. 2024-03-18T20:27:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.635+0000 7f5f72855700 1 Processor -- start 2024-03-18T20:27:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.635+0000 7f5f72855700 1 -- start start 2024-03-18T20:27:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.636+0000 7f5f72855700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5f6c158b80 0x7f5f6c1e5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.636+0000 7f5f72855700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f6c158f60 0x7f5f6c15ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.636+0000 7f5f72855700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f6c1463f0 0x7f5f6c15f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.636+0000 7f5f72855700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c0ac480 con 0x7f5f6c1463f0 2024-03-18T20:27:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.636+0000 7f5f72855700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c0a85c0 con 0x7f5f6c158f60 2024-03-18T20:27:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.636+0000 7f5f72855700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c1e6d80 con 0x7f5f6c158b80 2024-03-18T20:27:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.636+0000 7f5f70df2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f6c1463f0 0x7f5f6c15f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.636+0000 7f5f6b7fe700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f6c158f60 0x7f5f6c15ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.636+0000 7f5f70df2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f6c1463f0 0x7f5f6c15f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55070/0 (socket says 172.21.15.1:55070) 2024-03-18T20:27:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.636+0000 7f5f70df2700 1 -- 172.21.15.1:0/3450142119 learned_addr learned my addr 172.21.15.1:0/3450142119 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.636+0000 7f5f6bfff700 1 --2- 172.21.15.1:0/3450142119 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5f6c158b80 0x7f5f6c1e5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.636+0000 7f5f6bfff700 1 -- 172.21.15.1:0/3450142119 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f6c158f60 msgr2=0x7f5f6c15ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.636+0000 7f5f6bfff700 1 --2- 172.21.15.1:0/3450142119 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f6c158f60 0x7f5f6c15ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.636+0000 7f5f6bfff700 1 -- 172.21.15.1:0/3450142119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f6c1463f0 msgr2=0x7f5f6c15f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.636+0000 7f5f6bfff700 1 --2- 172.21.15.1:0/3450142119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f6c1463f0 0x7f5f6c15f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.636+0000 7f5f6bfff700 1 -- 172.21.15.1:0/3450142119 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5c009550 con 0x7f5f6c158b80 2024-03-18T20:27:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.636+0000 7f5f70df2700 1 --2- 172.21.15.1:0/3450142119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f6c1463f0 0x7f5f6c15f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:27:52.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.636+0000 7f5f6bfff700 1 --2- 172.21.15.1:0/3450142119 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5f6c158b80 0x7f5f6c1e5be0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c016000 tx=0x7f5f5c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:52.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.637+0000 7f5f68ff9700 1 -- 172.21.15.1:0/3450142119 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c004400 con 0x7f5f6c158b80 2024-03-18T20:27:52.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.637+0000 7f5f68ff9700 1 -- 172.21.15.1:0/3450142119 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5c01f070 con 0x7f5f6c158b80 2024-03-18T20:27:52.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.637+0000 7f5f72855700 1 -- 172.21.15.1:0/3450142119 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f6c1e64e0 con 0x7f5f6c158b80 2024-03-18T20:27:52.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.637+0000 7f5f68ff9700 1 -- 172.21.15.1:0/3450142119 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0038e0 con 0x7f5f6c158b80 2024-03-18T20:27:52.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.637+0000 7f5f72855700 1 -- 172.21.15.1:0/3450142119 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f6c1e6aa0 con 0x7f5f6c158b80 2024-03-18T20:27:52.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.638+0000 7f5f72855700 1 -- 172.21.15.1:0/3450142119 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f6c0b4da0 con 0x7f5f6c158b80 2024-03-18T20:27:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.638+0000 7f5f68ff9700 1 -- 172.21.15.1:0/3450142119 <== mon.2 v2:172.21.15.165:3301/0 4 ==== mgrmap(e 163) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f5f5c021030 con 0x7f5f6c158b80 2024-03-18T20:27:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.639+0000 7f5f68ff9700 1 --2- 172.21.15.1:0/3450142119 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5f5806ae40 0x7f5f5806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.639+0000 7f5f6b7fe700 1 -- 172.21.15.1:0/3450142119 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5f5806ae40 msgr2=0x7f5f5806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.639+0000 7f5f68ff9700 1 -- 172.21.15.1:0/3450142119 <== mon.2 v2:172.21.15.165:3301/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f5f5c08ae40 con 0x7f5f6c158b80 2024-03-18T20:27:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.639+0000 7f5f6b7fe700 1 --2- 172.21.15.1:0/3450142119 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5f5806ae40 0x7f5f5806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:27:52.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.644+0000 7f5f68ff9700 1 -- 172.21.15.1:0/3450142119 <== mon.2 v2:172.21.15.165:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5f5c055b60 con 0x7f5f6c158b80 2024-03-18T20:27:52.756 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:52.752+0000 7f4e60685240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:27:52.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.840+0000 7f5f6b7fe700 1 -- 172.21.15.1:0/3450142119 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5f5806ae40 msgr2=0x7f5f5806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:52.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.840+0000 7f5f6b7fe700 1 --2- 172.21.15.1:0/3450142119 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5f5806ae40 0x7f5f5806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:27:52.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.849+0000 7f5f72855700 1 -- 172.21.15.1:0/3450142119 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5f6c0cdf20 con 0x7f5f6c158b80 2024-03-18T20:27:52.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.850+0000 7f5f68ff9700 1 -- 172.21.15.1:0/3450142119 <== mon.2 v2:172.21.15.165:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v275) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5f5c059180 con 0x7f5f6c158b80 2024-03-18T20:27:52.854 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:27:52.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.852+0000 7f5f72855700 1 -- 172.21.15.1:0/3450142119 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5f5806ae40 msgr2=0x7f5f5806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:27:52.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.852+0000 7f5f72855700 1 --2- 172.21.15.1:0/3450142119 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5f5806ae40 0x7f5f5806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.852+0000 7f5f72855700 1 -- 172.21.15.1:0/3450142119 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5f6c158b80 msgr2=0x7f5f6c1e5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:52.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.852+0000 7f5f72855700 1 --2- 172.21.15.1:0/3450142119 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5f6c158b80 0x7f5f6c1e5be0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c016000 tx=0x7f5f5c004750 comp rx=0 tx=0).stop 2024-03-18T20:27:52.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.852+0000 7f5f72855700 1 -- 172.21.15.1:0/3450142119 shutdown_connections 2024-03-18T20:27:52.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.852+0000 7f5f72855700 1 --2- 172.21.15.1:0/3450142119 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5f5806ae40 0x7f5f5806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.852+0000 7f5f72855700 1 --2- 172.21.15.1:0/3450142119 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5f6c158b80 0x7f5f6c1e5be0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.852+0000 7f5f72855700 1 --2- 172.21.15.1:0/3450142119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f6c1463f0 0x7f5f6c15f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.852+0000 7f5f72855700 1 --2- 172.21.15.1:0/3450142119 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f6c158f60 0x7f5f6c15ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:52.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.852+0000 7f5f72855700 1 -- 172.21.15.1:0/3450142119 >> 172.21.15.1:0/3450142119 conn(0x7f5f6c06f030 msgr2=0x7f5f6c0cd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:52.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.852+0000 7f5f72855700 1 -- 172.21.15.1:0/3450142119 shutdown_connections 2024-03-18T20:27:52.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:52.852+0000 7f5f72855700 1 -- 172.21.15.1:0/3450142119 wait complete. 2024-03-18T20:27:52.869 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T20:27:52.906 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:52.902+0000 7efcc9740240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:27:53.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.029+0000 7f988e3c4700 1 Processor -- start 2024-03-18T20:27:53.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.029+0000 7f988e3c4700 1 -- start start 2024-03-18T20:27:53.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.030+0000 7f988e3c4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f98880cf200 0x7f988815c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:53.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.030+0000 7f988e3c4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f988815b6c0 0x7f98880cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:53.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.031+0000 7f988e3c4700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988815ca40 con 0x7f98880cbc90 2024-03-18T20:27:53.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.031+0000 7f988e3c4700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98880cce30 con 0x7f98880cf200 2024-03-18T20:27:53.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.031+0000 7f988e3c4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98880cfb20 con 0x7f988815b6c0 2024-03-18T20:27:53.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.031+0000 7f988c961700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f98880cbc90 0x7f988815eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58824/0 (socket says 172.21.15.1:58824) 2024-03-18T20:27:53.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.031+0000 7f9887fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f98880cf200 0x7f988815c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:53.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.031+0000 7f988c961700 1 -- 172.21.15.1:0/1162153142 learned_addr learned my addr 172.21.15.1:0/1162153142 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:53.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.031+0000 7f98877fe700 1 --2- 172.21.15.1:0/1162153142 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f988815b6c0 0x7f98880cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:53.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.031+0000 7f9887fff700 1 -- 172.21.15.1:0/1162153142 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f988815b6c0 msgr2=0x7f98880cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:53.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.031+0000 7f9887fff700 1 --2- 172.21.15.1:0/1162153142 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f988815b6c0 0x7f98880cf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:53.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.031+0000 7f9887fff700 1 -- 172.21.15.1:0/1162153142 >> v1:172.21.15.1:6789/0 conn(0x7f98880cbc90 legacy=0x7f988815eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:53.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.031+0000 7f9887fff700 1 -- 172.21.15.1:0/1162153142 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98880cc220 con 0x7f98880cf200 2024-03-18T20:27:53.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.031+0000 7f9886ffd700 1 -- 172.21.15.1:0/1162153142 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888599631 0 0) 0x7f9880001490 con 0x7f98880cbc90 2024-03-18T20:27:53.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.031+0000 7f9887fff700 1 --2- 172.21.15.1:0/1162153142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98880cf200 0x7f988815c500 secure :-1 s=READY pgs=3180 cs=0 l=1 rev1=1 crypto rx=0x7f9878009840 tx=0x7f9878009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8876731d0f2774e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:53.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.031+0000 7f9886ffd700 1 -- 172.21.15.1:0/1162153142 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9878011830 con 0x7f98880cf200 2024-03-18T20:27:53.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.031+0000 7f9886ffd700 1 -- 172.21.15.1:0/1162153142 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9878011990 con 0x7f98880cf200 2024-03-18T20:27:53.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.031+0000 7f9886ffd700 1 -- 172.21.15.1:0/1162153142 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9878011d20 con 0x7f98880cf200 2024-03-18T20:27:53.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.032+0000 7f988e3c4700 1 -- 172.21.15.1:0/1162153142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98880cf200 msgr2=0x7f988815c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:53.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.032+0000 7f988e3c4700 1 --2- 172.21.15.1:0/1162153142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98880cf200 0x7f988815c500 secure :-1 s=READY pgs=3180 cs=0 l=1 rev1=1 crypto rx=0x7f9878009840 tx=0x7f9878009b50 comp rx=0 tx=0).stop 2024-03-18T20:27:53.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.032+0000 7f988e3c4700 1 -- 172.21.15.1:0/1162153142 shutdown_connections 2024-03-18T20:27:53.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.032+0000 7f988e3c4700 1 --2- 172.21.15.1:0/1162153142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98880cf200 0x7f988815c500 unknown :-1 s=CLOSED pgs=3180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:53.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.032+0000 7f988e3c4700 1 --2- 172.21.15.1:0/1162153142 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f988815b6c0 0x7f98880cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:53.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.032+0000 7f988e3c4700 1 -- 172.21.15.1:0/1162153142 >> 172.21.15.1:0/1162153142 conn(0x7f98880d1180 msgr2=0x7f98880d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:53.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.032+0000 7f988e3c4700 1 -- 172.21.15.1:0/1162153142 shutdown_connections 2024-03-18T20:27:53.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.032+0000 7f988e3c4700 1 -- 172.21.15.1:0/1162153142 wait complete. 2024-03-18T20:27:53.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.034+0000 7f988e3c4700 1 Processor -- start 2024-03-18T20:27:53.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.034+0000 7f988e3c4700 1 -- start start 2024-03-18T20:27:53.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.035+0000 7f988e3c4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98880cbc90 0x7f98881f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:53.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.035+0000 7f988e3c4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98880cf200 0x7f988806c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:53.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.035+0000 7f988e3c4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f988815b6c0 0x7f98880700f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:53.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.035+0000 7f988e3c4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98880c5810 con 0x7f98880cbc90 2024-03-18T20:27:53.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.035+0000 7f988e3c4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98880c4c00 con 0x7f98880cf200 2024-03-18T20:27:53.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.035+0000 7f988e3c4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98880c4ee0 con 0x7f988815b6c0 2024-03-18T20:27:53.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.035+0000 7f9887fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98880cbc90 0x7f98881f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:53.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.035+0000 7f9887fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98880cbc90 0x7f98881f32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55084/0 (socket says 172.21.15.1:55084) 2024-03-18T20:27:53.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.035+0000 7f9887fff700 1 -- 172.21.15.1:0/394797150 learned_addr learned my addr 172.21.15.1:0/394797150 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:53.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.035+0000 7f988c961700 1 --2- 172.21.15.1:0/394797150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f988815b6c0 0x7f98880700f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:53.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.035+0000 7f98877fe700 1 --2- 172.21.15.1:0/394797150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98880cf200 0x7f988806c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:53.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.035+0000 7f9887fff700 1 -- 172.21.15.1:0/394797150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f988815b6c0 msgr2=0x7f98880700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:53.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.035+0000 7f9887fff700 1 --2- 172.21.15.1:0/394797150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f988815b6c0 0x7f98880700f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:53.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.035+0000 7f9887fff700 1 -- 172.21.15.1:0/394797150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98880cf200 msgr2=0x7f988806c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:53.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.035+0000 7f9887fff700 1 --2- 172.21.15.1:0/394797150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98880cf200 0x7f988806c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:53.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.035+0000 7f9887fff700 1 -- 172.21.15.1:0/394797150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98780094a0 con 0x7f98880cbc90 2024-03-18T20:27:53.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.036+0000 7f9887fff700 1 --2- 172.21.15.1:0/394797150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98880cbc90 0x7f98881f32b0 secure :-1 s=READY pgs=3181 cs=0 l=1 rev1=1 crypto rx=0x7f9878009b90 tx=0x7f9878004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:53.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.036+0000 7f9884ff9700 1 -- 172.21.15.1:0/394797150 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98780036a0 con 0x7f98880cbc90 2024-03-18T20:27:53.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.036+0000 7f9884ff9700 1 -- 172.21.15.1:0/394797150 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987801e070 con 0x7f98880cbc90 2024-03-18T20:27:53.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.036+0000 7f988e3c4700 1 -- 172.21.15.1:0/394797150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98880c51c0 con 0x7f98880cbc90 2024-03-18T20:27:53.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.036+0000 7f9884ff9700 1 -- 172.21.15.1:0/394797150 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9878003a60 con 0x7f98880cbc90 2024-03-18T20:27:53.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.036+0000 7f988e3c4700 1 -- 172.21.15.1:0/394797150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98881f37f0 con 0x7f98880cbc90 2024-03-18T20:27:53.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.037+0000 7f988e3c4700 1 -- 172.21.15.1:0/394797150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98880d0ca0 con 0x7f98880cbc90 2024-03-18T20:27:53.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.037+0000 7f9884ff9700 1 -- 172.21.15.1:0/394797150 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 163) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f9878003bc0 con 0x7f98880cbc90 2024-03-18T20:27:53.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.038+0000 7f9884ff9700 1 --2- 172.21.15.1:0/394797150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f987406ae40 0x7f987406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:53.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.038+0000 7f98877fe700 1 -- 172.21.15.1:0/394797150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f987406ae40 msgr2=0x7f987406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:53.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.038+0000 7f9884ff9700 1 -- 172.21.15.1:0/394797150 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f987808a7e0 con 0x7f98880cbc90 2024-03-18T20:27:53.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.038+0000 7f98877fe700 1 --2- 172.21.15.1:0/394797150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f987406ae40 0x7f987406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:27:53.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.042+0000 7f9884ff9700 1 -- 172.21.15.1:0/394797150 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9878055480 con 0x7f98880cbc90 2024-03-18T20:27:53.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.239+0000 7f98877fe700 1 -- 172.21.15.1:0/394797150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f987406ae40 msgr2=0x7f987406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:53.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.239+0000 7f98877fe700 1 --2- 172.21.15.1:0/394797150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f987406ae40 0x7f987406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:27:53.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.292+0000 7f988e3c4700 1 -- 172.21.15.1:0/394797150 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9888067dc0 con 0x7f98880cbc90 2024-03-18T20:27:53.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.293+0000 7f9884ff9700 1 -- 172.21.15.1:0/394797150 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v275) v1 ==== 81+0+566 (secure 0 0 0) 0x7f9878020250 con 0x7f98880cbc90 2024-03-18T20:27:53.297 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3635558645 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:53.297 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2293377213 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:53.297 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1796830610 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:53.297 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6833/148356 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:53.297 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6832/148356 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:53.297 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2068313472 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:53.298 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2668486132 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:53.298 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2560935779 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:53.298 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/82444607 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:53.298 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1446581052 2024-03-19T20:27:42.143876+0000 2024-03-18T20:27:53.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.297+0000 7f988e3c4700 1 -- 172.21.15.1:0/394797150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f987406ae40 msgr2=0x7f987406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:27:53.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.297+0000 7f988e3c4700 1 --2- 172.21.15.1:0/394797150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f987406ae40 0x7f987406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:53.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.297+0000 7f988e3c4700 1 -- 172.21.15.1:0/394797150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98880cbc90 msgr2=0x7f98881f32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:53.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.297+0000 7f988e3c4700 1 --2- 172.21.15.1:0/394797150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98880cbc90 0x7f98881f32b0 secure :-1 s=READY pgs=3181 cs=0 l=1 rev1=1 crypto rx=0x7f9878009b90 tx=0x7f9878004750 comp rx=0 tx=0).stop 2024-03-18T20:27:53.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.297+0000 7f988e3c4700 1 -- 172.21.15.1:0/394797150 shutdown_connections 2024-03-18T20:27:53.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.297+0000 7f988e3c4700 1 --2- 172.21.15.1:0/394797150 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f987406ae40 0x7f987406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:53.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.297+0000 7f988e3c4700 1 --2- 172.21.15.1:0/394797150 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f988815b6c0 0x7f98880700f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:53.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.297+0000 7f988e3c4700 1 --2- 172.21.15.1:0/394797150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98880cbc90 0x7f98881f32b0 unknown :-1 s=CLOSED pgs=3181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:53.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.297+0000 7f988e3c4700 1 --2- 172.21.15.1:0/394797150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98880cf200 0x7f988806c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:53.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.297+0000 7f988e3c4700 1 -- 172.21.15.1:0/394797150 >> 172.21.15.1:0/394797150 conn(0x7f98880d1180 msgr2=0x7f98880b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:53.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.297+0000 7f988e3c4700 1 -- 172.21.15.1:0/394797150 shutdown_connections 2024-03-18T20:27:53.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.297+0000 7f988e3c4700 1 -- 172.21.15.1:0/394797150 wait complete. 2024-03-18T20:27:53.302 INFO:teuthology.orchestra.run.smithi001.stderr:listed 10 entries 2024-03-18T20:27:53.314 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-18T20:27:53.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:53.315 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T20:27:53.345 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:53.341+0000 7f4e60685240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:27:53.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[172313]: fuse finished with error 0 and tester_r 0 2024-03-18T20:27:53.356 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:27:53.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:53.356 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T20:27:53.418 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:53.414+0000 7f4e60685240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:27:53.509 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:27:53.509 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:27:53.529 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:27:53.529 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:27:53.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:53.530 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-18T20:27:53.547 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:53.543+0000 7f4e60685240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:27:53.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:27:53.565 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-18T20:27:53.565 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:27:53.565 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:27:53.567 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:53.563+0000 7efcc9740240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:27:53.584 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:27:53.584 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:27:53.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:53.584 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-18T20:27:53.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:27:53.642 INFO:teuthology.orchestra.run.smithi001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-18T20:27:53.643 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:27:53.643 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:27:53.644 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:53.640+0000 7efcc9740240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:27:53.662 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:27:53.662 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:27:53.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:27:53.663 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-18T20:27:53.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:27:53.721 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-18T20:27:53.721 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:27:53.735 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:53.731+0000 7f4e60685240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:27:53.781 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:53.777+0000 7efcc9740240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:27:53.808 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:53.804+0000 7f4e60685240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:27:53.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.846+0000 7f2a99cd8700 1 Processor -- start 2024-03-18T20:27:53.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.847+0000 7f2a99cd8700 1 -- start start 2024-03-18T20:27:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.848+0000 7f2a99cd8700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2a9415a520 0x7f2a940b0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.848+0000 7f2a99cd8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2a940b11d0 0x7f2a940b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.848+0000 7f2a99cd8700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a940cc960 con 0x7f2a9415dfe0 2024-03-18T20:27:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.848+0000 7f2a99cd8700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a940cdc50 con 0x7f2a940b11d0 2024-03-18T20:27:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.848+0000 7f2a99cd8700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a9415a900 con 0x7f2a9415a520 2024-03-18T20:27:53.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.848+0000 7f2a92ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2a940b11d0 0x7f2a940b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.848+0000 7f2a93fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2a9415dfe0 0x7f2a9415e450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58834/0 (socket says 172.21.15.1:58834) 2024-03-18T20:27:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.848+0000 7f2a93fff700 1 -- 172.21.15.1:0/3792223223 learned_addr learned my addr 172.21.15.1:0/3792223223 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.848+0000 7f2a937fe700 1 --2- 172.21.15.1:0/3792223223 >> v2:172.21.15.165:3300/0 conn(0x7f2a9415a520 0x7f2a940b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.848+0000 7f2a927fc700 1 -- 172.21.15.1:0/3792223223 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456034309 0 0) 0x7f2a940cc960 con 0x7f2a9415dfe0 2024-03-18T20:27:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.848+0000 7f2a92ffd700 1 -- 172.21.15.1:0/3792223223 >> v2:172.21.15.165:3300/0 conn(0x7f2a9415a520 msgr2=0x7f2a940b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.848+0000 7f2a92ffd700 1 --2- 172.21.15.1:0/3792223223 >> v2:172.21.15.165:3300/0 conn(0x7f2a9415a520 0x7f2a940b0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.848+0000 7f2a92ffd700 1 -- 172.21.15.1:0/3792223223 >> v1:172.21.15.1:6789/0 conn(0x7f2a9415dfe0 legacy=0x7f2a9415e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.848+0000 7f2a92ffd700 1 -- 172.21.15.1:0/3792223223 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a940cd7c0 con 0x7f2a940b11d0 2024-03-18T20:27:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.849+0000 7f2a92ffd700 1 --2- 172.21.15.1:0/3792223223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a940b11d0 0x7f2a940b1660 secure :-1 s=READY pgs=3183 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009840 tx=0x7f2a7c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=633e2f5e8238badc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.849+0000 7f2a927fc700 1 -- 172.21.15.1:0/3792223223 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c011830 con 0x7f2a940b11d0 2024-03-18T20:27:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.849+0000 7f2a927fc700 1 -- 172.21.15.1:0/3792223223 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7c011990 con 0x7f2a940b11d0 2024-03-18T20:27:53.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.849+0000 7f2a927fc700 1 -- 172.21.15.1:0/3792223223 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c011d20 con 0x7f2a940b11d0 2024-03-18T20:27:53.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.850+0000 7f2a99cd8700 1 -- 172.21.15.1:0/3792223223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a940b11d0 msgr2=0x7f2a940b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:53.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.850+0000 7f2a99cd8700 1 --2- 172.21.15.1:0/3792223223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a940b11d0 0x7f2a940b1660 secure :-1 s=READY pgs=3183 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009840 tx=0x7f2a7c009b50 comp rx=0 tx=0).stop 2024-03-18T20:27:53.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.850+0000 7f2a99cd8700 1 -- 172.21.15.1:0/3792223223 shutdown_connections 2024-03-18T20:27:53.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.850+0000 7f2a99cd8700 1 --2- 172.21.15.1:0/3792223223 >> v2:172.21.15.165:3300/0 conn(0x7f2a9415a520 0x7f2a940b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:53.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.850+0000 7f2a99cd8700 1 --2- 172.21.15.1:0/3792223223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a940b11d0 0x7f2a940b1660 unknown :-1 s=CLOSED pgs=3183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:53.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.850+0000 7f2a99cd8700 1 -- 172.21.15.1:0/3792223223 >> 172.21.15.1:0/3792223223 conn(0x7f2a94072120 msgr2=0x7f2a94099a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:53.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.850+0000 7f2a99cd8700 1 -- 172.21.15.1:0/3792223223 shutdown_connections 2024-03-18T20:27:53.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.850+0000 7f2a99cd8700 1 -- 172.21.15.1:0/3792223223 wait complete. 2024-03-18T20:27:53.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.852+0000 7f2a99cd8700 1 Processor -- start 2024-03-18T20:27:53.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.852+0000 7f2a99cd8700 1 -- start start 2024-03-18T20:27:53.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.853+0000 7f2a99cd8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2a940b11d0 0x7f2a941572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:53.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.853+0000 7f2a99cd8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2a9415a520 0x7f2a94159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:53.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.853+0000 7f2a99cd8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a9415dfe0 0x7f2a940adc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:53.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.853+0000 7f2a99cd8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a94158090 con 0x7f2a9415dfe0 2024-03-18T20:27:53.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.853+0000 7f2a99cd8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a94158370 con 0x7f2a9415a520 2024-03-18T20:27:53.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.853+0000 7f2a99cd8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a94158650 con 0x7f2a940b11d0 2024-03-18T20:27:53.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.853+0000 7f2a93fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a9415dfe0 0x7f2a940adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:53.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.853+0000 7f2a93fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a9415dfe0 0x7f2a940adc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55090/0 (socket says 172.21.15.1:55090) 2024-03-18T20:27:53.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.853+0000 7f2a93fff700 1 -- 172.21.15.1:0/2015936959 learned_addr learned my addr 172.21.15.1:0/2015936959 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:53.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.853+0000 7f2a92ffd700 1 --2- 172.21.15.1:0/2015936959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2a9415a520 0x7f2a94159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:53.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.853+0000 7f2a937fe700 1 --2- 172.21.15.1:0/2015936959 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2a940b11d0 0x7f2a941572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:53.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.854+0000 7f2a93fff700 1 -- 172.21.15.1:0/2015936959 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2a940b11d0 msgr2=0x7f2a941572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:53.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.854+0000 7f2a93fff700 1 --2- 172.21.15.1:0/2015936959 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2a940b11d0 0x7f2a941572e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:53.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.854+0000 7f2a93fff700 1 -- 172.21.15.1:0/2015936959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2a9415a520 msgr2=0x7f2a94159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:53.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.854+0000 7f2a93fff700 1 --2- 172.21.15.1:0/2015936959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2a9415a520 0x7f2a94159130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:53.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.854+0000 7f2a93fff700 1 -- 172.21.15.1:0/2015936959 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7c0094a0 con 0x7f2a9415dfe0 2024-03-18T20:27:53.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.854+0000 7f2a93fff700 1 --2- 172.21.15.1:0/2015936959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a9415dfe0 0x7f2a940adc10 secure :-1 s=READY pgs=3184 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00c480 tx=0x7f2a8c00c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:53.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.854+0000 7f2a8bfff700 1 -- 172.21.15.1:0/2015936959 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c004c80 con 0x7f2a9415dfe0 2024-03-18T20:27:53.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.854+0000 7f2a8bfff700 1 -- 172.21.15.1:0/2015936959 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8c004e00 con 0x7f2a9415dfe0 2024-03-18T20:27:53.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.854+0000 7f2a99cd8700 1 -- 172.21.15.1:0/2015936959 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a94158930 con 0x7f2a9415dfe0 2024-03-18T20:27:53.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.854+0000 7f2a8bfff700 1 -- 172.21.15.1:0/2015936959 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c007310 con 0x7f2a9415dfe0 2024-03-18T20:27:53.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.854+0000 7f2a99cd8700 1 -- 172.21.15.1:0/2015936959 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a94157820 con 0x7f2a9415dfe0 2024-03-18T20:27:53.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.855+0000 7f2a99cd8700 1 -- 172.21.15.1:0/2015936959 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a94158e50 con 0x7f2a9415dfe0 2024-03-18T20:27:53.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.855+0000 7f2a8bfff700 1 -- 172.21.15.1:0/2015936959 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 163) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f2a8c007470 con 0x7f2a9415dfe0 2024-03-18T20:27:53.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.856+0000 7f2a8bfff700 1 --2- 172.21.15.1:0/2015936959 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2a7406ae40 0x7f2a7406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:53.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.856+0000 7f2a937fe700 1 -- 172.21.15.1:0/2015936959 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2a7406ae40 msgr2=0x7f2a7406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:53.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.856+0000 7f2a8bfff700 1 -- 172.21.15.1:0/2015936959 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f2a8c089500 con 0x7f2a9415dfe0 2024-03-18T20:27:53.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.856+0000 7f2a937fe700 1 --2- 172.21.15.1:0/2015936959 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2a7406ae40 0x7f2a7406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:27:53.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:53.861+0000 7f2a8bfff700 1 -- 172.21.15.1:0/2015936959 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2a94158e50 con 0x7f2a9415dfe0 2024-03-18T20:27:53.886 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:53.882+0000 7f4e60685240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:27:53.964 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:53.960+0000 7f4e60685240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:27:53.977 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:53.973+0000 7efcc9740240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:27:54.042 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:54.038+0000 7f4e60685240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:27:54.054 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:54.050+0000 7efcc9740240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:27:54.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.057+0000 7f2a937fe700 1 -- 172.21.15.1:0/2015936959 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2a7406ae40 msgr2=0x7f2a7406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:54.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.057+0000 7f2a937fe700 1 --2- 172.21.15.1:0/2015936959 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2a7406ae40 0x7f2a7406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:27:54.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.113+0000 7f2a99cd8700 1 -- 172.21.15.1:0/2015936959 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a94158e50 con 0x7f2a9415dfe0 2024-03-18T20:27:54.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.114+0000 7f2a8bfff700 1 -- 172.21.15.1:0/2015936959 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+4749 (secure 0 0 0) 0x7f2a94158e50 con 0x7f2a9415dfe0 2024-03-18T20:27:54.118 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:54.118 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":136,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9938,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/1320583552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1320583552},{"type":"v1","addr":"172.21.15.165:6833","nonce":1320583552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131},{"gid":11566,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2669900315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2669900315},{"type":"v1","addr":"172.21.15.165:6835","nonce":2669900315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131},{"gid":13170,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2726723271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2726723271},{"type":"v1","addr":"172.21.15.1:6835","nonce":2726723271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:27:22.047616+0000","modified":"2024-03-18T20:27:25.113850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13173},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13173":{"gid":13173,"name":"c","rank":0,"incarnation":132,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/3909787366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":3909787366},{"type":"v1","addr":"172.21.15.1:6837","nonce":3909787366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-03-18T20:27:54.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.116+0000 7f2a99cd8700 1 -- 172.21.15.1:0/2015936959 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2a7406ae40 msgr2=0x7f2a7406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:27:54.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.116+0000 7f2a99cd8700 1 --2- 172.21.15.1:0/2015936959 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2a7406ae40 0x7f2a7406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.116+0000 7f2a99cd8700 1 -- 172.21.15.1:0/2015936959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a9415dfe0 msgr2=0x7f2a940adc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:54.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.116+0000 7f2a99cd8700 1 --2- 172.21.15.1:0/2015936959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a9415dfe0 0x7f2a940adc10 secure :-1 s=READY pgs=3184 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00c480 tx=0x7f2a8c00c790 comp rx=0 tx=0).stop 2024-03-18T20:27:54.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.117+0000 7f2a99cd8700 1 -- 172.21.15.1:0/2015936959 shutdown_connections 2024-03-18T20:27:54.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.117+0000 7f2a99cd8700 1 --2- 172.21.15.1:0/2015936959 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2a7406ae40 0x7f2a7406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.117+0000 7f2a99cd8700 1 --2- 172.21.15.1:0/2015936959 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2a940b11d0 0x7f2a941572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.117+0000 7f2a99cd8700 1 --2- 172.21.15.1:0/2015936959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a9415dfe0 0x7f2a940adc10 unknown :-1 s=CLOSED pgs=3184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.117+0000 7f2a99cd8700 1 --2- 172.21.15.1:0/2015936959 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2a9415a520 0x7f2a94159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.117+0000 7f2a99cd8700 1 -- 172.21.15.1:0/2015936959 >> 172.21.15.1:0/2015936959 conn(0x7f2a94072120 msgr2=0x7f2a940b4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:54.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.117+0000 7f2a99cd8700 1 -- 172.21.15.1:0/2015936959 shutdown_connections 2024-03-18T20:27:54.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.117+0000 7f2a99cd8700 1 -- 172.21.15.1:0/2015936959 wait complete. 2024-03-18T20:27:54.122 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 136 2024-03-18T20:27:54.133 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:27:54.141 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:54.138+0000 7efcc9740240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:27:54.216 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:54.212+0000 7f4e60685240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:27:54.228 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:54.224+0000 7efcc9740240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:27:54.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.257+0000 7fce11dc8700 1 Processor -- start 2024-03-18T20:27:54.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.257+0000 7fce11dc8700 1 -- start start 2024-03-18T20:27:54.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.258+0000 7fce11dc8700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fce0c0cf200 0x7fce0c15c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:54.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.258+0000 7fce11dc8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fce0c15b6c0 0x7fce0c0cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:54.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.258+0000 7fce11dc8700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0c15ca40 con 0x7fce0c0cbc90 2024-03-18T20:27:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.258+0000 7fce11dc8700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce0c0cce30 con 0x7fce0c15b6c0 2024-03-18T20:27:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.258+0000 7fce11dc8700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce0c0cfb20 con 0x7fce0c0cf200 2024-03-18T20:27:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.258+0000 7fce0affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fce0c15b6c0 0x7fce0c0cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.258+0000 7fce0affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fce0c15b6c0 0x7fce0c0cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55104/0 (socket says 172.21.15.1:55104) 2024-03-18T20:27:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.258+0000 7fce0affd700 1 -- 172.21.15.1:0/2324062401 learned_addr learned my addr 172.21.15.1:0/2324062401 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.259+0000 7fce0b7fe700 1 --2- 172.21.15.1:0/2324062401 >> v2:172.21.15.165:3300/0 conn(0x7fce0c0cf200 0x7fce0c15c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.259+0000 7fce0affd700 1 -- 172.21.15.1:0/2324062401 >> v2:172.21.15.165:3300/0 conn(0x7fce0c0cf200 msgr2=0x7fce0c15c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.259+0000 7fce0affd700 1 --2- 172.21.15.1:0/2324062401 >> v2:172.21.15.165:3300/0 conn(0x7fce0c0cf200 0x7fce0c15c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.259+0000 7fce0affd700 1 -- 172.21.15.1:0/2324062401 >> v1:172.21.15.165:6789/0 conn(0x7fce0c0cbc90 legacy=0x7fce0c15eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.259+0000 7fce0affd700 1 -- 172.21.15.1:0/2324062401 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c0cc220 con 0x7fce0c15b6c0 2024-03-18T20:27:54.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.259+0000 7fce0affd700 1 --2- 172.21.15.1:0/2324062401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0c15b6c0 0x7fce0c0cf5e0 secure :-1 s=READY pgs=3185 cs=0 l=1 rev1=1 crypto rx=0x7fce0c06ea00 tx=0x7fce00009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=712778f7e8519c00 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:54.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.259+0000 7fce0a7fc700 1 -- 172.21.15.1:0/2324062401 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0000b870 con 0x7fce0c15b6c0 2024-03-18T20:27:54.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.259+0000 7fce0a7fc700 1 -- 172.21.15.1:0/2324062401 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0000b9d0 con 0x7fce0c15b6c0 2024-03-18T20:27:54.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.259+0000 7fce0a7fc700 1 -- 172.21.15.1:0/2324062401 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0000bd60 con 0x7fce0c15b6c0 2024-03-18T20:27:54.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.260+0000 7fce11dc8700 1 -- 172.21.15.1:0/2324062401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0c15b6c0 msgr2=0x7fce0c0cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:54.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.260+0000 7fce11dc8700 1 --2- 172.21.15.1:0/2324062401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0c15b6c0 0x7fce0c0cf5e0 secure :-1 s=READY pgs=3185 cs=0 l=1 rev1=1 crypto rx=0x7fce0c06ea00 tx=0x7fce00009bd0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.260+0000 7fce11dc8700 1 -- 172.21.15.1:0/2324062401 shutdown_connections 2024-03-18T20:27:54.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.260+0000 7fce11dc8700 1 --2- 172.21.15.1:0/2324062401 >> v2:172.21.15.165:3300/0 conn(0x7fce0c0cf200 0x7fce0c15c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.260+0000 7fce11dc8700 1 --2- 172.21.15.1:0/2324062401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0c15b6c0 0x7fce0c0cf5e0 unknown :-1 s=CLOSED pgs=3185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.260+0000 7fce11dc8700 1 -- 172.21.15.1:0/2324062401 >> 172.21.15.1:0/2324062401 conn(0x7fce0c0d1180 msgr2=0x7fce0c0d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:54.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.260+0000 7fce11dc8700 1 -- 172.21.15.1:0/2324062401 shutdown_connections 2024-03-18T20:27:54.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.260+0000 7fce11dc8700 1 -- 172.21.15.1:0/2324062401 wait complete. 2024-03-18T20:27:54.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.261+0000 7fce11dc8700 1 Processor -- start 2024-03-18T20:27:54.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.262+0000 7fce11dc8700 1 -- start start 2024-03-18T20:27:54.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.262+0000 7fce11dc8700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0c0cbc90 0x7fce0c0c4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:54.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.262+0000 7fce11dc8700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce0c0cf200 0x7fce0c147ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:54.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.262+0000 7fce11dc8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce0c15b6c0 0x7fce0c0b5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:54.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.262+0000 7fce11dc8700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c0c7c90 con 0x7fce0c0cbc90 2024-03-18T20:27:54.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.262+0000 7fce11dc8700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c0c6bf0 con 0x7fce0c15b6c0 2024-03-18T20:27:54.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.262+0000 7fce11dc8700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c0c6ed0 con 0x7fce0c0cf200 2024-03-18T20:27:54.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.262+0000 7fce0b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0c0cbc90 0x7fce0c0c4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:54.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.262+0000 7fce0b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0c0cbc90 0x7fce0c0c4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55110/0 (socket says 172.21.15.1:55110) 2024-03-18T20:27:54.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.262+0000 7fce0b7fe700 1 -- 172.21.15.1:0/470905278 learned_addr learned my addr 172.21.15.1:0/470905278 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:54.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.263+0000 7fce0bfff700 1 --2- 172.21.15.1:0/470905278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce0c15b6c0 0x7fce0c0b5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:54.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.263+0000 7fce0affd700 1 --2- 172.21.15.1:0/470905278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce0c0cf200 0x7fce0c147ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:54.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.263+0000 7fce0b7fe700 1 -- 172.21.15.1:0/470905278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce0c0cf200 msgr2=0x7fce0c147ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:54.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.263+0000 7fce0b7fe700 1 --2- 172.21.15.1:0/470905278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce0c0cf200 0x7fce0c147ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.263+0000 7fce0b7fe700 1 -- 172.21.15.1:0/470905278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce0c15b6c0 msgr2=0x7fce0c0b5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:54.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.263+0000 7fce0b7fe700 1 --2- 172.21.15.1:0/470905278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce0c15b6c0 0x7fce0c0b5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.263+0000 7fce0b7fe700 1 -- 172.21.15.1:0/470905278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce00009550 con 0x7fce0c0cbc90 2024-03-18T20:27:54.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.263+0000 7fce0b7fe700 1 --2- 172.21.15.1:0/470905278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0c0cbc90 0x7fce0c0c4e00 secure :-1 s=READY pgs=3186 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00d800 tx=0x7fcdfc00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:54.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.263+0000 7fcdf3fff700 1 -- 172.21.15.1:0/470905278 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc0125f0 con 0x7fce0c0cbc90 2024-03-18T20:27:54.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.263+0000 7fcdf3fff700 1 -- 172.21.15.1:0/470905278 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdfc012750 con 0x7fce0c0cbc90 2024-03-18T20:27:54.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.263+0000 7fce11dc8700 1 -- 172.21.15.1:0/470905278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce0c0c71b0 con 0x7fce0c0cbc90 2024-03-18T20:27:54.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.263+0000 7fcdf3fff700 1 -- 172.21.15.1:0/470905278 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc012b00 con 0x7fce0c0cbc90 2024-03-18T20:27:54.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.263+0000 7fce11dc8700 1 -- 172.21.15.1:0/470905278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce0c0c7540 con 0x7fce0c0cbc90 2024-03-18T20:27:54.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.264+0000 7fce11dc8700 1 -- 172.21.15.1:0/470905278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0c0d0ca0 con 0x7fce0c0cbc90 2024-03-18T20:27:54.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.265+0000 7fcdf3fff700 1 -- 172.21.15.1:0/470905278 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 163) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fcdfc012c60 con 0x7fce0c0cbc90 2024-03-18T20:27:54.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.265+0000 7fcdf3fff700 1 --2- 172.21.15.1:0/470905278 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcdf806ae40 0x7fcdf806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:54.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.265+0000 7fce0affd700 1 -- 172.21.15.1:0/470905278 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcdf806ae40 msgr2=0x7fcdf806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:54.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.265+0000 7fcdf3fff700 1 -- 172.21.15.1:0/470905278 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7fcdfc089940 con 0x7fce0c0cbc90 2024-03-18T20:27:54.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.265+0000 7fce0affd700 1 --2- 172.21.15.1:0/470905278 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcdf806ae40 0x7fcdf806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:27:54.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.274+0000 7fcdf3fff700 1 -- 172.21.15.1:0/470905278 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcdfc054550 con 0x7fce0c0cbc90 2024-03-18T20:27:54.318 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:54.314+0000 7efcc9740240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:27:54.433 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:54.429+0000 7f4e60685240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:27:54.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.466+0000 7fce0affd700 1 -- 172.21.15.1:0/470905278 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcdf806ae40 msgr2=0x7fcdf806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:54.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.466+0000 7fce0affd700 1 --2- 172.21.15.1:0/470905278 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcdf806ae40 0x7fcdf806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:27:54.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.478+0000 7fce11dc8700 1 -- 172.21.15.1:0/470905278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce0c0c7340 con 0x7fce0c0cbc90 2024-03-18T20:27:54.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.479+0000 7fcdf3fff700 1 -- 172.21.15.1:0/470905278 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v275) v1 ==== 74+0+16970 (secure 0 0 0) 0x7fcdfc057b70 con 0x7fce0c0cbc90 2024-03-18T20:27:54.483 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:54.483 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":275,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:27:42.143916+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:27:19.891286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"273","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-03-18T20:27:21.027315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"274","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3635558645":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2293377213":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/1796830610":"2024-03-19T20:27:42.143876+0000","172.21.15.1:6833/148356":"2024-03-19T20:27:42.143876+0000","172.21.15.1:6832/148356":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2068313472":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2668486132":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2560935779":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/82444607":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/1446581052":"2024-03-19T20:27:42.143876+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:27:54.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.481+0000 7fce11dc8700 1 -- 172.21.15.1:0/470905278 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcdf806ae40 msgr2=0x7fcdf806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:27:54.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.481+0000 7fce11dc8700 1 --2- 172.21.15.1:0/470905278 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcdf806ae40 0x7fcdf806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.481+0000 7fce11dc8700 1 -- 172.21.15.1:0/470905278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0c0cbc90 msgr2=0x7fce0c0c4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:54.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.481+0000 7fce11dc8700 1 --2- 172.21.15.1:0/470905278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0c0cbc90 0x7fce0c0c4e00 secure :-1 s=READY pgs=3186 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00d800 tx=0x7fcdfc00db10 comp rx=0 tx=0).stop 2024-03-18T20:27:54.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.481+0000 7fce11dc8700 1 -- 172.21.15.1:0/470905278 shutdown_connections 2024-03-18T20:27:54.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.481+0000 7fce11dc8700 1 --2- 172.21.15.1:0/470905278 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcdf806ae40 0x7fcdf806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.481+0000 7fce11dc8700 1 --2- 172.21.15.1:0/470905278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce0c15b6c0 0x7fce0c0b5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.481+0000 7fce11dc8700 1 --2- 172.21.15.1:0/470905278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0c0cbc90 0x7fce0c0c4e00 unknown :-1 s=CLOSED pgs=3186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.481+0000 7fce11dc8700 1 --2- 172.21.15.1:0/470905278 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce0c0cf200 0x7fce0c147ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.481+0000 7fce11dc8700 1 -- 172.21.15.1:0/470905278 >> 172.21.15.1:0/470905278 conn(0x7fce0c0d1180 msgr2=0x7fce0c0b09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:54.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.481+0000 7fce11dc8700 1 -- 172.21.15.1:0/470905278 shutdown_connections 2024-03-18T20:27:54.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.481+0000 7fce11dc8700 1 -- 172.21.15.1:0/470905278 wait complete. 2024-03-18T20:27:54.498 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:27:19.891286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '273', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:27:21.027315+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '274', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:27:54.498 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:27:54.501 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:54.497+0000 7efcc9740240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:27:54.583 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:54.579+0000 7f4e60685240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:27:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.629+0000 7f7bbe7dc700 1 Processor -- start 2024-03-18T20:27:54.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.629+0000 7f7bbe7dc700 1 -- start start 2024-03-18T20:27:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.630+0000 7f7bbe7dc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7bb8099a30 0x7f7bb80ad800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.630+0000 7f7bbe7dc700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7bb806f510 0x7f7bb80add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.630+0000 7f7bbe7dc700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb815dab0 con 0x7f7bb815d6d0 2024-03-18T20:27:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.630+0000 7f7bbe7dc700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bb806fad0 con 0x7f7bb8099a30 2024-03-18T20:27:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.630+0000 7f7bbe7dc700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bb80b3780 con 0x7f7bb806f510 2024-03-18T20:27:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.630+0000 7f7bb77fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7bb8099a30 0x7f7bb80ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.630+0000 7f7bb77fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7bb8099a30 0x7f7bb80ad800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57860/0 (socket says 172.21.15.1:57860) 2024-03-18T20:27:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.630+0000 7f7bb77fe700 1 -- 172.21.15.1:0/768387677 learned_addr learned my addr 172.21.15.1:0/768387677 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.630+0000 7f7bbcd79700 1 --2- 172.21.15.1:0/768387677 >> v2:172.21.15.165:3300/0 conn(0x7f7bb806f510 0x7f7bb80add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.630+0000 7f7bb77fe700 1 -- 172.21.15.1:0/768387677 >> v2:172.21.15.165:3300/0 conn(0x7f7bb806f510 msgr2=0x7f7bb80add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.630+0000 7f7bb77fe700 1 --2- 172.21.15.1:0/768387677 >> v2:172.21.15.165:3300/0 conn(0x7f7bb806f510 0x7f7bb80add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.630+0000 7f7bb77fe700 1 -- 172.21.15.1:0/768387677 >> v1:172.21.15.165:6789/0 conn(0x7f7bb815d6d0 legacy=0x7f7bb806ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:54.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.630+0000 7f7bb77fe700 1 -- 172.21.15.1:0/768387677 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb80b3a60 con 0x7f7bb8099a30 2024-03-18T20:27:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.631+0000 7f7bb77fe700 1 --2- 172.21.15.1:0/768387677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb8099a30 0x7f7bb80ad800 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 crypto rx=0x7f7bb80ccbd0 tx=0x7f7ba0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eefb537a30615689 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.631+0000 7f7bb6ffd700 1 -- 172.21.15.1:0/768387677 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0011830 con 0x7f7bb8099a30 2024-03-18T20:27:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.631+0000 7f7bb6ffd700 1 -- 172.21.15.1:0/768387677 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba0011990 con 0x7f7bb8099a30 2024-03-18T20:27:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.631+0000 7f7bb6ffd700 1 -- 172.21.15.1:0/768387677 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0011d20 con 0x7f7bb8099a30 2024-03-18T20:27:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.631+0000 7f7bbe7dc700 1 -- 172.21.15.1:0/768387677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb8099a30 msgr2=0x7f7bb80ad800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.631+0000 7f7bbe7dc700 1 --2- 172.21.15.1:0/768387677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb8099a30 0x7f7bb80ad800 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 crypto rx=0x7f7bb80ccbd0 tx=0x7f7ba0009bd0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.631+0000 7f7bbe7dc700 1 -- 172.21.15.1:0/768387677 shutdown_connections 2024-03-18T20:27:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.631+0000 7f7bbe7dc700 1 --2- 172.21.15.1:0/768387677 >> v2:172.21.15.165:3300/0 conn(0x7f7bb806f510 0x7f7bb80add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.631+0000 7f7bbe7dc700 1 --2- 172.21.15.1:0/768387677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb8099a30 0x7f7bb80ad800 unknown :-1 s=CLOSED pgs=3187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.631+0000 7f7bbe7dc700 1 -- 172.21.15.1:0/768387677 >> 172.21.15.1:0/768387677 conn(0x7f7bb80b5a10 msgr2=0x7f7bb80b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.632+0000 7f7bbe7dc700 1 -- 172.21.15.1:0/768387677 shutdown_connections 2024-03-18T20:27:54.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.632+0000 7f7bbe7dc700 1 -- 172.21.15.1:0/768387677 wait complete. 2024-03-18T20:27:54.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.633+0000 7f7bbe7dc700 1 Processor -- start 2024-03-18T20:27:54.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.633+0000 7f7bbe7dc700 1 -- start start 2024-03-18T20:27:54.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.634+0000 7f7bbe7dc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb806f510 0x7f7bb80cec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:54.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.634+0000 7f7bbe7dc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7bb8099a30 0x7f7bb80cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:54.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.634+0000 7f7bbe7dc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7bb815d6d0 0x7f7bb81567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:54.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.634+0000 7f7bbe7dc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb80cc8e0 con 0x7f7bb806f510 2024-03-18T20:27:54.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.634+0000 7f7bbe7dc700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb80cff40 con 0x7f7bb815d6d0 2024-03-18T20:27:54.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.634+0000 7f7bbe7dc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb80cf330 con 0x7f7bb8099a30 2024-03-18T20:27:54.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.634+0000 7f7bb7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb806f510 0x7f7bb80cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:54.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.634+0000 7f7bb7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb806f510 0x7f7bb80cec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57876/0 (socket says 172.21.15.1:57876) 2024-03-18T20:27:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.634+0000 7f7bb7fff700 1 -- 172.21.15.1:0/4211303200 learned_addr learned my addr 172.21.15.1:0/4211303200 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.634+0000 7f7bbcd79700 1 --2- 172.21.15.1:0/4211303200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7bb815d6d0 0x7f7bb81567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.634+0000 7f7bb77fe700 1 --2- 172.21.15.1:0/4211303200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7bb8099a30 0x7f7bb80cc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.634+0000 7f7bb7fff700 1 -- 172.21.15.1:0/4211303200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7bb8099a30 msgr2=0x7f7bb80cc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.634+0000 7f7bb7fff700 1 --2- 172.21.15.1:0/4211303200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7bb8099a30 0x7f7bb80cc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.634+0000 7f7bb7fff700 1 -- 172.21.15.1:0/4211303200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7bb815d6d0 msgr2=0x7f7bb81567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.634+0000 7f7bb7fff700 1 --2- 172.21.15.1:0/4211303200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7bb815d6d0 0x7f7bb81567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.634+0000 7f7bb7fff700 1 -- 172.21.15.1:0/4211303200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba0009550 con 0x7f7bb806f510 2024-03-18T20:27:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.634+0000 7f7bbcd79700 1 --2- 172.21.15.1:0/4211303200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7bb815d6d0 0x7f7bb81567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:27:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.634+0000 7f7bb7fff700 1 --2- 172.21.15.1:0/4211303200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb806f510 0x7f7bb80cec20 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 crypto rx=0x7f7ba800c450 tx=0x7f7ba800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.635+0000 7f7bb4ff9700 1 -- 172.21.15.1:0/4211303200 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba80052d0 con 0x7f7bb806f510 2024-03-18T20:27:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.635+0000 7f7bb4ff9700 1 -- 172.21.15.1:0/4211303200 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba8005430 con 0x7f7bb806f510 2024-03-18T20:27:54.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.635+0000 7f7bbe7dc700 1 -- 172.21.15.1:0/4211303200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb80cf610 con 0x7f7bb806f510 2024-03-18T20:27:54.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.635+0000 7f7bb4ff9700 1 -- 172.21.15.1:0/4211303200 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba80038c0 con 0x7f7bb806f510 2024-03-18T20:27:54.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.635+0000 7f7bbe7dc700 1 -- 172.21.15.1:0/4211303200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb80cfbd0 con 0x7f7bb806f510 2024-03-18T20:27:54.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.636+0000 7f7bbe7dc700 1 -- 172.21.15.1:0/4211303200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb80cfd60 con 0x7f7bb806f510 2024-03-18T20:27:54.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.637+0000 7f7bb4ff9700 1 -- 172.21.15.1:0/4211303200 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 163) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f7ba8003a20 con 0x7f7bb806f510 2024-03-18T20:27:54.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.637+0000 7f7bb4ff9700 1 --2- 172.21.15.1:0/4211303200 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7b9806ae40 0x7f7b9806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:54.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.637+0000 7f7bb77fe700 1 -- 172.21.15.1:0/4211303200 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7b9806ae40 msgr2=0x7f7b9806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:54.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.637+0000 7f7bb4ff9700 1 -- 172.21.15.1:0/4211303200 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f7ba8088f90 con 0x7f7bb806f510 2024-03-18T20:27:54.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.637+0000 7f7bb77fe700 1 --2- 172.21.15.1:0/4211303200 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7b9806ae40 0x7f7b9806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:27:54.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.641+0000 7f7bb4ff9700 1 -- 172.21.15.1:0/4211303200 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7ba8053d10 con 0x7f7bb806f510 2024-03-18T20:27:54.657 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:54.653+0000 7f4e60685240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:27:54.736 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:54.732+0000 7efcc9740240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:27:54.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.838+0000 7f7bb77fe700 1 -- 172.21.15.1:0/4211303200 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7b9806ae40 msgr2=0x7f7b9806d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:54.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.838+0000 7f7bb77fe700 1 --2- 172.21.15.1:0/4211303200 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7b9806ae40 0x7f7b9806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:27:54.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.848+0000 7f7bbe7dc700 1 -- 172.21.15.1:0/4211303200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7bb8067dc0 con 0x7f7bb806f510 2024-03-18T20:27:54.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.848+0000 7f7bb4ff9700 1 -- 172.21.15.1:0/4211303200 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v275) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7ba80170a0 con 0x7f7bb806f510 2024-03-18T20:27:54.852 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:27:54.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.850+0000 7f7bbe7dc700 1 -- 172.21.15.1:0/4211303200 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7b9806ae40 msgr2=0x7f7b9806d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:27:54.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.850+0000 7f7bbe7dc700 1 --2- 172.21.15.1:0/4211303200 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7b9806ae40 0x7f7b9806d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.850+0000 7f7bbe7dc700 1 -- 172.21.15.1:0/4211303200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb806f510 msgr2=0x7f7bb80cec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:54.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.850+0000 7f7bbe7dc700 1 --2- 172.21.15.1:0/4211303200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb806f510 0x7f7bb80cec20 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 crypto rx=0x7f7ba800c450 tx=0x7f7ba800c760 comp rx=0 tx=0).stop 2024-03-18T20:27:54.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.850+0000 7f7bbe7dc700 1 -- 172.21.15.1:0/4211303200 shutdown_connections 2024-03-18T20:27:54.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.850+0000 7f7bbe7dc700 1 --2- 172.21.15.1:0/4211303200 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7b9806ae40 0x7f7b9806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.850+0000 7f7bbe7dc700 1 --2- 172.21.15.1:0/4211303200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7bb815d6d0 0x7f7bb81567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.850+0000 7f7bbe7dc700 1 --2- 172.21.15.1:0/4211303200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bb806f510 0x7f7bb80cec20 unknown :-1 s=CLOSED pgs=3188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.850+0000 7f7bbe7dc700 1 --2- 172.21.15.1:0/4211303200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7bb8099a30 0x7f7bb80cc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.850+0000 7f7bbe7dc700 1 -- 172.21.15.1:0/4211303200 >> 172.21.15.1:0/4211303200 conn(0x7f7bb80b5a10 msgr2=0x7f7bb80b1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:54.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.851+0000 7f7bbe7dc700 1 -- 172.21.15.1:0/4211303200 shutdown_connections 2024-03-18T20:27:54.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.851+0000 7f7bbe7dc700 1 -- 172.21.15.1:0/4211303200 wait complete. 2024-03-18T20:27:54.867 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:27:54.901 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:54.897+0000 7efcc9740240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:27:54.982 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:54.978+0000 7efcc9740240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:27:54.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.992+0000 7efe38995700 1 Processor -- start 2024-03-18T20:27:54.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.992+0000 7efe38995700 1 -- start start 2024-03-18T20:27:54.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.993+0000 7efe38995700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7efe3415b6c0 0x7efe3415c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:54.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.993+0000 7efe38995700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efe340cbc90 0x7efe34099ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:54.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.993+0000 7efe38995700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe3415ca40 con 0x7efe340cf200 2024-03-18T20:27:54.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.993+0000 7efe38995700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe340cce30 con 0x7efe340cbc90 2024-03-18T20:27:54.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.993+0000 7efe38995700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe340cc220 con 0x7efe3415b6c0 2024-03-18T20:27:54.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.993+0000 7efe32d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efe340cbc90 0x7efe34099ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:54.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.993+0000 7efe32d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efe340cbc90 0x7efe34099ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57884/0 (socket says 172.21.15.1:57884) 2024-03-18T20:27:54.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.993+0000 7efe32d9d700 1 -- 172.21.15.1:0/751776051 learned_addr learned my addr 172.21.15.1:0/751776051 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:54.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.993+0000 7efe31d9b700 1 --2- 172.21.15.1:0/751776051 >> v2:172.21.15.165:3300/0 conn(0x7efe3415b6c0 0x7efe3415c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:54.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.993+0000 7efe32d9d700 1 -- 172.21.15.1:0/751776051 >> v2:172.21.15.165:3300/0 conn(0x7efe3415b6c0 msgr2=0x7efe3415c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:54.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.993+0000 7efe32d9d700 1 --2- 172.21.15.1:0/751776051 >> v2:172.21.15.165:3300/0 conn(0x7efe3415b6c0 0x7efe3415c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:54.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.993+0000 7efe32d9d700 1 -- 172.21.15.1:0/751776051 >> v1:172.21.15.165:6789/0 conn(0x7efe340cf200 legacy=0x7efe340cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:54.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.993+0000 7efe32d9d700 1 -- 172.21.15.1:0/751776051 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe340cc500 con 0x7efe340cbc90 2024-03-18T20:27:54.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.993+0000 7efe31d9b700 1 --2- 172.21.15.1:0/751776051 >> v2:172.21.15.165:3300/0 conn(0x7efe3415b6c0 0x7efe3415c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:54.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.993+0000 7efe32d9d700 1 --2- 172.21.15.1:0/751776051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe340cbc90 0x7efe34099ac0 secure :-1 s=READY pgs=3189 cs=0 l=1 rev1=1 crypto rx=0x7efe34072200 tx=0x7efe2c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=557293cdd4c281cb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:54.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.994+0000 7efe3159a700 1 -- 172.21.15.1:0/751776051 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2c00e070 con 0x7efe340cbc90 2024-03-18T20:27:54.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.994+0000 7efe3159a700 1 -- 172.21.15.1:0/751776051 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe2c00b9c0 con 0x7efe340cbc90 2024-03-18T20:27:54.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.994+0000 7efe3159a700 1 -- 172.21.15.1:0/751776051 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2c00bd70 con 0x7efe340cbc90 2024-03-18T20:27:55.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.994+0000 7efe38995700 1 -- 172.21.15.1:0/751776051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe340cbc90 msgr2=0x7efe34099ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:55.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.994+0000 7efe38995700 1 --2- 172.21.15.1:0/751776051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe340cbc90 0x7efe34099ac0 secure :-1 s=READY pgs=3189 cs=0 l=1 rev1=1 crypto rx=0x7efe34072200 tx=0x7efe2c00b330 comp rx=0 tx=0).stop 2024-03-18T20:27:55.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.994+0000 7efe38995700 1 -- 172.21.15.1:0/751776051 shutdown_connections 2024-03-18T20:27:55.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.994+0000 7efe38995700 1 --2- 172.21.15.1:0/751776051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe340cbc90 0x7efe34099ac0 unknown :-1 s=CLOSED pgs=3189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.994+0000 7efe38995700 1 --2- 172.21.15.1:0/751776051 >> v2:172.21.15.165:3300/0 conn(0x7efe3415b6c0 0x7efe3415c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.994+0000 7efe38995700 1 -- 172.21.15.1:0/751776051 >> 172.21.15.1:0/751776051 conn(0x7efe340d1180 msgr2=0x7efe340d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:55.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.994+0000 7efe38995700 1 -- 172.21.15.1:0/751776051 shutdown_connections 2024-03-18T20:27:55.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.994+0000 7efe38995700 1 -- 172.21.15.1:0/751776051 wait complete. 2024-03-18T20:27:55.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.996+0000 7efe38995700 1 Processor -- start 2024-03-18T20:27:55.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.997+0000 7efe38995700 1 -- start start 2024-03-18T20:27:55.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.998+0000 7efe38995700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe340cbc90 0x7efe34141f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:55.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.998+0000 7efe38995700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe340cf200 0x7efe340c3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:55.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.998+0000 7efe38995700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe3415b6c0 0x7efe3413f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:55.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.998+0000 7efe38995700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe340c9c40 con 0x7efe340cbc90 2024-03-18T20:27:55.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.998+0000 7efe38995700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe340c7420 con 0x7efe340cf200 2024-03-18T20:27:55.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.998+0000 7efe38995700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe340c76a0 con 0x7efe3415b6c0 2024-03-18T20:27:55.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.998+0000 7efe3259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe340cbc90 0x7efe34141f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:55.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.998+0000 7efe3259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe340cbc90 0x7efe34141f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57890/0 (socket says 172.21.15.1:57890) 2024-03-18T20:27:55.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.998+0000 7efe3259c700 1 -- 172.21.15.1:0/3652295875 learned_addr learned my addr 172.21.15.1:0/3652295875 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:55.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.998+0000 7efe32d9d700 1 --2- 172.21.15.1:0/3652295875 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe3415b6c0 0x7efe3413f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:55.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.998+0000 7efe3259c700 1 -- 172.21.15.1:0/3652295875 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe3415b6c0 msgr2=0x7efe3413f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:55.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.998+0000 7efe3259c700 1 --2- 172.21.15.1:0/3652295875 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe3415b6c0 0x7efe3413f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.998+0000 7efe3259c700 1 -- 172.21.15.1:0/3652295875 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe340cf200 msgr2=0x7efe340c3cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T20:27:55.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.998+0000 7efe3259c700 1 --2- 172.21.15.1:0/3652295875 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe340cf200 0x7efe340c3cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.998+0000 7efe3259c700 1 -- 172.21.15.1:0/3652295875 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2c009b30 con 0x7efe340cbc90 2024-03-18T20:27:55.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.998+0000 7efe3259c700 1 --2- 172.21.15.1:0/3652295875 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe340cbc90 0x7efe34141f40 secure :-1 s=READY pgs=3190 cs=0 l=1 rev1=1 crypto rx=0x7efe2400c4e0 tx=0x7efe2400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:55.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.999+0000 7efe22ffd700 1 -- 172.21.15.1:0/3652295875 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe240052d0 con 0x7efe340cbc90 2024-03-18T20:27:55.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.999+0000 7efe22ffd700 1 -- 172.21.15.1:0/3652295875 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe24005430 con 0x7efe340cbc90 2024-03-18T20:27:55.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.999+0000 7efe38995700 1 -- 172.21.15.1:0/3652295875 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe340c9eb0 con 0x7efe340cbc90 2024-03-18T20:27:55.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.999+0000 7efe22ffd700 1 -- 172.21.15.1:0/3652295875 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe240038c0 con 0x7efe340cbc90 2024-03-18T20:27:55.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.999+0000 7efe38995700 1 -- 172.21.15.1:0/3652295875 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe34142480 con 0x7efe340cbc90 2024-03-18T20:27:55.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:54.999+0000 7efe38995700 1 -- 172.21.15.1:0/3652295875 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe340d0ca0 con 0x7efe340cbc90 2024-03-18T20:27:55.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.001+0000 7efe22ffd700 1 -- 172.21.15.1:0/3652295875 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 163) v1 ==== 89296+0+0 (secure 0 0 0) 0x7efe24003a20 con 0x7efe340cbc90 2024-03-18T20:27:55.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.001+0000 7efe22ffd700 1 --2- 172.21.15.1:0/3652295875 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efe1406ae40 0x7efe1406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:55.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.001+0000 7efe31d9b700 1 -- 172.21.15.1:0/3652295875 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efe1406ae40 msgr2=0x7efe1406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:55.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.001+0000 7efe22ffd700 1 -- 172.21.15.1:0/3652295875 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7efe24088c40 con 0x7efe340cbc90 2024-03-18T20:27:55.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.001+0000 7efe31d9b700 1 --2- 172.21.15.1:0/3652295875 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efe1406ae40 0x7efe1406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:27:55.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.010+0000 7efe22ffd700 1 -- 172.21.15.1:0/3652295875 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efe24053850 con 0x7efe340cbc90 2024-03-18T20:27:55.105 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:55.101+0000 7f4e60685240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:27:55.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.202+0000 7efe31d9b700 1 -- 172.21.15.1:0/3652295875 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efe1406ae40 msgr2=0x7efe1406d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:55.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.202+0000 7efe31d9b700 1 --2- 172.21.15.1:0/3652295875 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efe1406ae40 0x7efe1406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:27:55.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.214+0000 7efe38995700 1 -- 172.21.15.1:0/3652295875 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efe340c9aa0 con 0x7efe340cbc90 2024-03-18T20:27:55.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.215+0000 7efe22ffd700 1 -- 172.21.15.1:0/3652295875 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v275) v1 ==== 104+0+11 (secure 0 0 0) 0x7efe24017020 con 0x7efe340cbc90 2024-03-18T20:27:55.219 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:27:55.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.217+0000 7efe38995700 1 -- 172.21.15.1:0/3652295875 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efe1406ae40 msgr2=0x7efe1406d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:27:55.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.217+0000 7efe38995700 1 --2- 172.21.15.1:0/3652295875 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efe1406ae40 0x7efe1406d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.217+0000 7efe38995700 1 -- 172.21.15.1:0/3652295875 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe340cbc90 msgr2=0x7efe34141f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:55.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.217+0000 7efe38995700 1 --2- 172.21.15.1:0/3652295875 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe340cbc90 0x7efe34141f40 secure :-1 s=READY pgs=3190 cs=0 l=1 rev1=1 crypto rx=0x7efe2400c4e0 tx=0x7efe2400c7f0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.217+0000 7efe38995700 1 -- 172.21.15.1:0/3652295875 shutdown_connections 2024-03-18T20:27:55.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.217+0000 7efe38995700 1 --2- 172.21.15.1:0/3652295875 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efe1406ae40 0x7efe1406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.217+0000 7efe38995700 1 --2- 172.21.15.1:0/3652295875 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efe3415b6c0 0x7efe3413f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.217+0000 7efe38995700 1 --2- 172.21.15.1:0/3652295875 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efe340cbc90 0x7efe34141f40 unknown :-1 s=CLOSED pgs=3190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.217+0000 7efe38995700 1 --2- 172.21.15.1:0/3652295875 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe340cf200 0x7efe340c3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.217+0000 7efe38995700 1 -- 172.21.15.1:0/3652295875 >> 172.21.15.1:0/3652295875 conn(0x7efe340d1180 msgr2=0x7efe340b09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:55.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.217+0000 7efe38995700 1 -- 172.21.15.1:0/3652295875 shutdown_connections 2024-03-18T20:27:55.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.217+0000 7efe38995700 1 -- 172.21.15.1:0/3652295875 wait complete. 2024-03-18T20:27:55.234 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:27:55.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.399+0000 7fce0573f700 1 Processor -- start 2024-03-18T20:27:55.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.399+0000 7fce0573f700 1 -- start start 2024-03-18T20:27:55.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.400+0000 7fce0573f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fce0015d5e0 0x7fce000d1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:55.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.400+0000 7fce0573f700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fce000d3f80 0x7fce0015d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:55.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.400+0000 7fce0573f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce000cfe30 con 0x7fce0015df00 2024-03-18T20:27:55.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.400+0000 7fce0573f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce000d0110 con 0x7fce0015d5e0 2024-03-18T20:27:55.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.400+0000 7fce0573f700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce000d03f0 con 0x7fce000d3f80 2024-03-18T20:27:55.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.401+0000 7fcdfeffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fce0015d5e0 0x7fce000d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:55.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.401+0000 7fcdfeffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fce0015d5e0 0x7fce000d1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57892/0 (socket says 172.21.15.1:57892) 2024-03-18T20:27:55.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.401+0000 7fcdfeffd700 1 -- 172.21.15.1:0/454099608 learned_addr learned my addr 172.21.15.1:0/454099608 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:55.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.401+0000 7fcdfe7fc700 1 --2- 172.21.15.1:0/454099608 >> v2:172.21.15.165:3300/0 conn(0x7fce000d3f80 0x7fce0015d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:55.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.401+0000 7fcdfeffd700 1 -- 172.21.15.1:0/454099608 >> v2:172.21.15.165:3300/0 conn(0x7fce000d3f80 msgr2=0x7fce0015d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:55.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.401+0000 7fcdfeffd700 1 --2- 172.21.15.1:0/454099608 >> v2:172.21.15.165:3300/0 conn(0x7fce000d3f80 0x7fce0015d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.401+0000 7fcdfeffd700 1 -- 172.21.15.1:0/454099608 >> v1:172.21.15.165:6789/0 conn(0x7fce0015df00 legacy=0x7fce00161000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:55.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.401+0000 7fcdfeffd700 1 -- 172.21.15.1:0/454099608 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce000d06d0 con 0x7fce0015d5e0 2024-03-18T20:27:55.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.401+0000 7fcdfeffd700 1 --2- 172.21.15.1:0/454099608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0015d5e0 0x7fce000d1dd0 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 crypto rx=0x7fce000d0ee0 tx=0x7fcdf0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6e214260dac0dc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:55.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.401+0000 7fcdfdffb700 1 -- 172.21.15.1:0/454099608 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf000b870 con 0x7fce0015d5e0 2024-03-18T20:27:55.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.401+0000 7fcdfdffb700 1 -- 172.21.15.1:0/454099608 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf000b9d0 con 0x7fce0015d5e0 2024-03-18T20:27:55.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.401+0000 7fcdfdffb700 1 -- 172.21.15.1:0/454099608 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf000bd60 con 0x7fce0015d5e0 2024-03-18T20:27:55.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.402+0000 7fce0573f700 1 -- 172.21.15.1:0/454099608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0015d5e0 msgr2=0x7fce000d1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:55.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.402+0000 7fce0573f700 1 --2- 172.21.15.1:0/454099608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0015d5e0 0x7fce000d1dd0 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 crypto rx=0x7fce000d0ee0 tx=0x7fcdf0009bd0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.402+0000 7fce0573f700 1 -- 172.21.15.1:0/454099608 shutdown_connections 2024-03-18T20:27:55.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.402+0000 7fce0573f700 1 --2- 172.21.15.1:0/454099608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0015d5e0 0x7fce000d1dd0 unknown :-1 s=CLOSED pgs=3191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.402+0000 7fce0573f700 1 --2- 172.21.15.1:0/454099608 >> v2:172.21.15.165:3300/0 conn(0x7fce000d3f80 0x7fce0015d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.402+0000 7fce0573f700 1 -- 172.21.15.1:0/454099608 >> 172.21.15.1:0/454099608 conn(0x7fce000d3380 msgr2=0x7fce000d3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:55.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.402+0000 7fce0573f700 1 -- 172.21.15.1:0/454099608 shutdown_connections 2024-03-18T20:27:55.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.402+0000 7fce0573f700 1 -- 172.21.15.1:0/454099608 wait complete. 2024-03-18T20:27:55.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.404+0000 7fce0573f700 1 Processor -- start 2024-03-18T20:27:55.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.404+0000 7fce0573f700 1 -- start start 2024-03-18T20:27:55.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.406+0000 7fce0573f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce000d3f80 0x7fce000c78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:55.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.406+0000 7fce0573f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce0015d5e0 0x7fce0015b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:55.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.406+0000 7fce0573f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0015df00 0x7fce0015ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:55.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.406+0000 7fce0573f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce00160030 con 0x7fce0015df00 2024-03-18T20:27:55.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.406+0000 7fce0573f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce00160310 con 0x7fce0015d5e0 2024-03-18T20:27:55.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.406+0000 7fce0573f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce001605f0 con 0x7fce000d3f80 2024-03-18T20:27:55.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.406+0000 7fcdff7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0015df00 0x7fce0015ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:55.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.406+0000 7fcdff7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0015df00 0x7fce0015ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57908/0 (socket says 172.21.15.1:57908) 2024-03-18T20:27:55.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.406+0000 7fcdfe7fc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce0015d5e0 0x7fce0015b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:55.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.406+0000 7fcdff7fe700 1 -- 172.21.15.1:0/1789823849 learned_addr learned my addr 172.21.15.1:0/1789823849 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:55.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.406+0000 7fcdfeffd700 1 --2- 172.21.15.1:0/1789823849 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce000d3f80 0x7fce000c78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:55.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.406+0000 7fcdff7fe700 1 -- 172.21.15.1:0/1789823849 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce000d3f80 msgr2=0x7fce000c78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:55.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.406+0000 7fcdff7fe700 1 --2- 172.21.15.1:0/1789823849 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce000d3f80 0x7fce000c78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.406+0000 7fcdff7fe700 1 -- 172.21.15.1:0/1789823849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce0015d5e0 msgr2=0x7fce0015b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:55.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.406+0000 7fcdff7fe700 1 --2- 172.21.15.1:0/1789823849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce0015d5e0 0x7fce0015b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.406+0000 7fcdff7fe700 1 -- 172.21.15.1:0/1789823849 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf0009550 con 0x7fce0015df00 2024-03-18T20:27:55.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.406+0000 7fcdff7fe700 1 --2- 172.21.15.1:0/1789823849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0015df00 0x7fce0015ba90 secure :-1 s=READY pgs=3192 cs=0 l=1 rev1=1 crypto rx=0x7fcdf800cb50 tx=0x7fcdf800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:55.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.406+0000 7fcde77fe700 1 -- 172.21.15.1:0/1789823849 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8003c00 con 0x7fce0015df00 2024-03-18T20:27:55.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.406+0000 7fcde77fe700 1 -- 172.21.15.1:0/1789823849 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf8003d60 con 0x7fce0015df00 2024-03-18T20:27:55.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.407+0000 7fce0573f700 1 -- 172.21.15.1:0/1789823849 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce000d2490 con 0x7fce0015df00 2024-03-18T20:27:55.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.407+0000 7fcde77fe700 1 -- 172.21.15.1:0/1789823849 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf800aa20 con 0x7fce0015df00 2024-03-18T20:27:55.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.407+0000 7fce0573f700 1 -- 172.21.15.1:0/1789823849 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce000c9010 con 0x7fce0015df00 2024-03-18T20:27:55.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.407+0000 7fce0573f700 1 -- 172.21.15.1:0/1789823849 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce001f75f0 con 0x7fce0015df00 2024-03-18T20:27:55.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.408+0000 7fcde77fe700 1 -- 172.21.15.1:0/1789823849 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 163) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fcdf800ab80 con 0x7fce0015df00 2024-03-18T20:27:55.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.408+0000 7fcde77fe700 1 --2- 172.21.15.1:0/1789823849 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcdec06ae40 0x7fcdec06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:55.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.409+0000 7fcdfeffd700 1 -- 172.21.15.1:0/1789823849 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcdec06ae40 msgr2=0x7fcdec06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:55.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.409+0000 7fcde77fe700 1 -- 172.21.15.1:0/1789823849 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7fcdf8089950 con 0x7fce0015df00 2024-03-18T20:27:55.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.409+0000 7fcdfeffd700 1 --2- 172.21.15.1:0/1789823849 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcdec06ae40 0x7fcdec06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:27:55.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.413+0000 7fcde77fe700 1 -- 172.21.15.1:0/1789823849 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcdf8054560 con 0x7fce0015df00 2024-03-18T20:27:55.443 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:55.439+0000 7f4e60685240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:27:55.490 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:55.486+0000 7efcc9740240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:27:55.607 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:55.603+0000 7f4e60685240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:27:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.610+0000 7fcdfeffd700 1 -- 172.21.15.1:0/1789823849 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcdec06ae40 msgr2=0x7fcdec06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:55.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.610+0000 7fcdfeffd700 1 --2- 172.21.15.1:0/1789823849 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcdec06ae40 0x7fcdec06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:27:55.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.631+0000 7fce0573f700 1 -- 172.21.15.1:0/1789823849 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fce00067dc0 con 0x7fce0015df00 2024-03-18T20:27:55.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.631+0000 7fcde77fe700 1 -- 172.21.15.1:0/1789823849 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v275) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcdf8057b80 con 0x7fce0015df00 2024-03-18T20:27:55.636 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:27:55.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.634+0000 7fce0573f700 1 -- 172.21.15.1:0/1789823849 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcdec06ae40 msgr2=0x7fcdec06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:27:55.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.634+0000 7fce0573f700 1 --2- 172.21.15.1:0/1789823849 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcdec06ae40 0x7fcdec06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.634+0000 7fce0573f700 1 -- 172.21.15.1:0/1789823849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0015df00 msgr2=0x7fce0015ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:55.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.634+0000 7fce0573f700 1 --2- 172.21.15.1:0/1789823849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0015df00 0x7fce0015ba90 secure :-1 s=READY pgs=3192 cs=0 l=1 rev1=1 crypto rx=0x7fcdf800cb50 tx=0x7fcdf800ce60 comp rx=0 tx=0).stop 2024-03-18T20:27:55.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.634+0000 7fce0573f700 1 -- 172.21.15.1:0/1789823849 shutdown_connections 2024-03-18T20:27:55.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.634+0000 7fce0573f700 1 --2- 172.21.15.1:0/1789823849 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcdec06ae40 0x7fcdec06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.634+0000 7fce0573f700 1 --2- 172.21.15.1:0/1789823849 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fce000d3f80 0x7fce000c78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.634+0000 7fce0573f700 1 --2- 172.21.15.1:0/1789823849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce0015df00 0x7fce0015ba90 unknown :-1 s=CLOSED pgs=3192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.634+0000 7fce0573f700 1 --2- 172.21.15.1:0/1789823849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce0015d5e0 0x7fce0015b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.634+0000 7fce0573f700 1 -- 172.21.15.1:0/1789823849 >> 172.21.15.1:0/1789823849 conn(0x7fce000d3380 msgr2=0x7fce000b19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:55.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.634+0000 7fce0573f700 1 -- 172.21.15.1:0/1789823849 shutdown_connections 2024-03-18T20:27:55.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.634+0000 7fce0573f700 1 -- 172.21.15.1:0/1789823849 wait complete. 2024-03-18T20:27:55.651 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:27:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.773+0000 7f8a7fc19700 1 Processor -- start 2024-03-18T20:27:55.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.774+0000 7f8a7fc19700 1 -- start start 2024-03-18T20:27:55.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.774+0000 7f8a7fc19700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8a7815e8e0 0x7f8a780cf380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:55.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.774+0000 7f8a7fc19700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a7815ef60 0x7f8a7815cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:55.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.774+0000 7f8a7fc19700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a7806ec60 con 0x7f8a78099a30 2024-03-18T20:27:55.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.774+0000 7f8a7fc19700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a78070100 con 0x7f8a7815e8e0 2024-03-18T20:27:55.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.774+0000 7f8a7fc19700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a780cfd80 con 0x7f8a7815ef60 2024-03-18T20:27:55.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.775+0000 7f8a7e1b6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a7815ef60 0x7f8a7815cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:55.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.775+0000 7f8a7e1b6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a7815ef60 0x7f8a7815cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:40414/0 (socket says 172.21.15.1:40414) 2024-03-18T20:27:55.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.775+0000 7f8a7d1b4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8a7815e8e0 0x7f8a780cf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:55.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.775+0000 7f8a7d9b5700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f8a78099a30 0x7f8a78099e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:45418/0 (socket says 172.21.15.1:45418) 2024-03-18T20:27:55.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.775+0000 7f8a7e1b6700 1 -- 172.21.15.1:0/3092295639 learned_addr learned my addr 172.21.15.1:0/3092295639 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:55.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.775+0000 7f8a7e1b6700 1 -- 172.21.15.1:0/3092295639 >> v2:172.21.15.165:3300/0 conn(0x7f8a7815e8e0 msgr2=0x7f8a780cf380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:55.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.775+0000 7f8a7e1b6700 1 --2- 172.21.15.1:0/3092295639 >> v2:172.21.15.165:3300/0 conn(0x7f8a7815e8e0 0x7f8a780cf380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.775+0000 7f8a7e1b6700 1 -- 172.21.15.1:0/3092295639 >> v1:172.21.15.165:6789/0 conn(0x7f8a78099a30 legacy=0x7f8a78099e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:55.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.775+0000 7f8a7e1b6700 1 -- 172.21.15.1:0/3092295639 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a780d0060 con 0x7f8a7815ef60 2024-03-18T20:27:55.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.775+0000 7f8a7e1b6700 1 --2- 172.21.15.1:0/3092295639 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a7815ef60 0x7f8a7815cdf0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8a70009e50 tx=0x7f8a7000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70954d97f1def1aa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:55.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.775+0000 7f8a7c9b3700 1 -- 172.21.15.1:0/3092295639 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7000e070 con 0x7f8a7815ef60 2024-03-18T20:27:55.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.775+0000 7f8a7c9b3700 1 -- 172.21.15.1:0/3092295639 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7000b910 con 0x7f8a7815ef60 2024-03-18T20:27:55.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.775+0000 7f8a7c9b3700 1 -- 172.21.15.1:0/3092295639 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7000bcc0 con 0x7f8a7815ef60 2024-03-18T20:27:55.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.776+0000 7f8a7fc19700 1 -- 172.21.15.1:0/3092295639 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a7815ef60 msgr2=0x7f8a7815cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:55.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.776+0000 7f8a7fc19700 1 --2- 172.21.15.1:0/3092295639 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a7815ef60 0x7f8a7815cdf0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8a70009e50 tx=0x7f8a7000b330 comp rx=0 tx=0).stop 2024-03-18T20:27:55.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.776+0000 7f8a7fc19700 1 -- 172.21.15.1:0/3092295639 shutdown_connections 2024-03-18T20:27:55.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.776+0000 7f8a7fc19700 1 --2- 172.21.15.1:0/3092295639 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a7815ef60 0x7f8a7815cdf0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.776+0000 7f8a7fc19700 1 --2- 172.21.15.1:0/3092295639 >> v2:172.21.15.165:3300/0 conn(0x7f8a7815e8e0 0x7f8a780cf380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.776+0000 7f8a7fc19700 1 -- 172.21.15.1:0/3092295639 >> 172.21.15.1:0/3092295639 conn(0x7f8a780d1180 msgr2=0x7f8a780d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:55.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.776+0000 7f8a7fc19700 1 -- 172.21.15.1:0/3092295639 shutdown_connections 2024-03-18T20:27:55.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.776+0000 7f8a7fc19700 1 -- 172.21.15.1:0/3092295639 wait complete. 2024-03-18T20:27:55.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.778+0000 7f8a7fc19700 1 Processor -- start 2024-03-18T20:27:55.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.778+0000 7f8a7fc19700 1 -- start start 2024-03-18T20:27:55.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a7fc19700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a78099a30 0x7f8a7814c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:55.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a7fc19700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a7815ef60 0x7f8a78148e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:55.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a7fc19700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a780b6640 0x7f8a780af3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:55.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a7fc19700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a780c7bc0 con 0x7f8a7815ef60 2024-03-18T20:27:55.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a7fc19700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a781493d0 con 0x7f8a780b6640 2024-03-18T20:27:55.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a7fc19700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a780aef50 con 0x7f8a78099a30 2024-03-18T20:27:55.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a7d1b4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a7815ef60 0x7f8a78148e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:55.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a7d1b4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a7815ef60 0x7f8a78148e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57910/0 (socket says 172.21.15.1:57910) 2024-03-18T20:27:55.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a7d1b4700 1 -- 172.21.15.1:0/1547604670 learned_addr learned my addr 172.21.15.1:0/1547604670 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:55.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a7e1b6700 1 --2- 172.21.15.1:0/1547604670 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a780b6640 0x7f8a780af3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:55.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a7d9b5700 1 --2- 172.21.15.1:0/1547604670 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a78099a30 0x7f8a7814c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:55.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a7d1b4700 1 -- 172.21.15.1:0/1547604670 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a78099a30 msgr2=0x7f8a7814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:55.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a7d1b4700 1 --2- 172.21.15.1:0/1547604670 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a78099a30 0x7f8a7814c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a7d1b4700 1 -- 172.21.15.1:0/1547604670 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a780b6640 msgr2=0x7f8a780af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:55.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a7d1b4700 1 --2- 172.21.15.1:0/1547604670 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a780b6640 0x7f8a780af3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:55.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a7d1b4700 1 -- 172.21.15.1:0/1547604670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a70009b30 con 0x7f8a7815ef60 2024-03-18T20:27:55.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a7d1b4700 1 --2- 172.21.15.1:0/1547604670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a7815ef60 0x7f8a78148e90 secure :-1 s=READY pgs=3193 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800e9e0 tx=0x7f8a6800ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:55.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a6e7fc700 1 -- 172.21.15.1:0/1547604670 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a68009710 con 0x7f8a7815ef60 2024-03-18T20:27:55.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a6e7fc700 1 -- 172.21.15.1:0/1547604670 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6800ee80 con 0x7f8a7815ef60 2024-03-18T20:27:55.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a7fc19700 1 -- 172.21.15.1:0/1547604670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a780c7600 con 0x7f8a7815ef60 2024-03-18T20:27:55.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.779+0000 7f8a6e7fc700 1 -- 172.21.15.1:0/1547604670 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6800ca50 con 0x7f8a7815ef60 2024-03-18T20:27:55.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.780+0000 7f8a7fc19700 1 -- 172.21.15.1:0/1547604670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a780aebe0 con 0x7f8a7815ef60 2024-03-18T20:27:55.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.780+0000 7f8a7fc19700 1 -- 172.21.15.1:0/1547604670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a7809ada0 con 0x7f8a7815ef60 2024-03-18T20:27:55.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.781+0000 7f8a6e7fc700 1 -- 172.21.15.1:0/1547604670 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 163) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f8a6800cbb0 con 0x7f8a7815ef60 2024-03-18T20:27:55.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.782+0000 7f8a6e7fc700 1 --2- 172.21.15.1:0/1547604670 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8a5c06ae40 0x7f8a5c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:55.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.782+0000 7f8a7d9b5700 1 -- 172.21.15.1:0/1547604670 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8a5c06ae40 msgr2=0x7f8a5c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:55.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.782+0000 7f8a6e7fc700 1 -- 172.21.15.1:0/1547604670 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f8a68089980 con 0x7f8a7815ef60 2024-03-18T20:27:55.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.782+0000 7f8a7d9b5700 1 --2- 172.21.15.1:0/1547604670 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8a5c06ae40 0x7f8a5c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:27:55.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.786+0000 7f8a6e7fc700 1 -- 172.21.15.1:0/1547604670 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8a68054510 con 0x7f8a7815ef60 2024-03-18T20:27:55.851 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:55.848+0000 7efcc9740240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:27:55.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.983+0000 7f8a7d9b5700 1 -- 172.21.15.1:0/1547604670 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8a5c06ae40 msgr2=0x7f8a5c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:55.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:55.983+0000 7f8a7d9b5700 1 --2- 172.21.15.1:0/1547604670 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8a5c06ae40 0x7f8a5c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:27:55.999 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:55.995+0000 7f4e60685240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:27:56.027 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:56.023+0000 7efcc9740240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:27:56.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.033+0000 7f8a7fc19700 1 -- 172.21.15.1:0/1547604670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a780af0c0 con 0x7f8a7815ef60 2024-03-18T20:27:56.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.034+0000 7f8a6e7fc700 1 -- 172.21.15.1:0/1547604670 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+4749 (secure 0 0 0) 0x7f8a68057b30 con 0x7f8a7815ef60 2024-03-18T20:27:56.038 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:56.038 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":136,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9938,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/1320583552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1320583552},{"type":"v1","addr":"172.21.15.165:6833","nonce":1320583552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131},{"gid":11566,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2669900315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2669900315},{"type":"v1","addr":"172.21.15.165:6835","nonce":2669900315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131},{"gid":13170,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2726723271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2726723271},{"type":"v1","addr":"172.21.15.1:6835","nonce":2726723271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:27:22.047616+0000","modified":"2024-03-18T20:27:25.113850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13173},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13173":{"gid":13173,"name":"c","rank":0,"incarnation":132,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/3909787366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":3909787366},{"type":"v1","addr":"172.21.15.1:6837","nonce":3909787366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-03-18T20:27:56.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.036+0000 7f8a7fc19700 1 -- 172.21.15.1:0/1547604670 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8a5c06ae40 msgr2=0x7f8a5c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:27:56.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.036+0000 7f8a7fc19700 1 --2- 172.21.15.1:0/1547604670 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8a5c06ae40 0x7f8a5c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.036+0000 7f8a7fc19700 1 -- 172.21.15.1:0/1547604670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a7815ef60 msgr2=0x7f8a78148e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:56.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.036+0000 7f8a7fc19700 1 --2- 172.21.15.1:0/1547604670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a7815ef60 0x7f8a78148e90 secure :-1 s=READY pgs=3193 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800e9e0 tx=0x7f8a6800ecf0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.036+0000 7f8a7fc19700 1 -- 172.21.15.1:0/1547604670 shutdown_connections 2024-03-18T20:27:56.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.036+0000 7f8a7fc19700 1 --2- 172.21.15.1:0/1547604670 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8a5c06ae40 0x7f8a5c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.036+0000 7f8a7fc19700 1 --2- 172.21.15.1:0/1547604670 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a78099a30 0x7f8a7814c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.036+0000 7f8a7fc19700 1 --2- 172.21.15.1:0/1547604670 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a780b6640 0x7f8a780af3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.036+0000 7f8a7fc19700 1 --2- 172.21.15.1:0/1547604670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a7815ef60 0x7f8a78148e90 unknown :-1 s=CLOSED pgs=3193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.036+0000 7f8a7fc19700 1 -- 172.21.15.1:0/1547604670 >> 172.21.15.1:0/1547604670 conn(0x7f8a780d1180 msgr2=0x7f8a7815d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:56.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.036+0000 7f8a7fc19700 1 -- 172.21.15.1:0/1547604670 shutdown_connections 2024-03-18T20:27:56.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.036+0000 7f8a7fc19700 1 -- 172.21.15.1:0/1547604670 wait complete. 2024-03-18T20:27:56.042 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 136 2024-03-18T20:27:56.057 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:27:56.164 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:56.160+0000 7f4e60685240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:27:56.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.180+0000 7f6e2c411700 1 Processor -- start 2024-03-18T20:27:56.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.180+0000 7f6e2c411700 1 -- start start 2024-03-18T20:27:56.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.181+0000 7f6e2c411700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6e240d3f80 0x7f6e2415d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:56.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.181+0000 7f6e2c411700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e240cfe30 con 0x7f6e2415df00 2024-03-18T20:27:56.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.181+0000 7f6e2c411700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e240d0110 con 0x7f6e2415d5e0 2024-03-18T20:27:56.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.181+0000 7f6e2c411700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e240d24e0 con 0x7f6e240d3f80 2024-03-18T20:27:56.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.182+0000 7f6e299ac700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6e240d3f80 0x7f6e2415d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:56.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.182+0000 7f6e2a9ae700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6e2415df00 0x7f6e240cdf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39536/0 (socket says 172.21.15.1:39536) 2024-03-18T20:27:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.182+0000 7f6e2a9ae700 1 -- 172.21.15.1:0/468092167 learned_addr learned my addr 172.21.15.1:0/468092167 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.182+0000 7f6e299ac700 1 -- 172.21.15.1:0/468092167 >> v1:172.21.15.165:6789/0 conn(0x7f6e2415d5e0 legacy=0x7f6e240d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.182+0000 7f6e299ac700 1 -- 172.21.15.1:0/468092167 >> v1:172.21.15.1:6789/0 conn(0x7f6e2415df00 legacy=0x7f6e240cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.182+0000 7f6e291ab700 1 -- 172.21.15.1:0/468092167 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963707041 0 0) 0x7f6e240cfe30 con 0x7f6e2415df00 2024-03-18T20:27:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.182+0000 7f6e299ac700 1 -- 172.21.15.1:0/468092167 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e240d03f0 con 0x7f6e240d3f80 2024-03-18T20:27:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.182+0000 7f6e299ac700 1 --2- 172.21.15.1:0/468092167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e240d3f80 0x7f6e2415d9c0 secure :-1 s=READY pgs=3195 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009840 tx=0x7f6e14009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c6ac79dbb04a102 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.182+0000 7f6e291ab700 1 -- 172.21.15.1:0/468092167 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e14011830 con 0x7f6e240d3f80 2024-03-18T20:27:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.182+0000 7f6e291ab700 1 -- 172.21.15.1:0/468092167 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e14011990 con 0x7f6e240d3f80 2024-03-18T20:27:56.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.182+0000 7f6e291ab700 1 -- 172.21.15.1:0/468092167 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e14011d20 con 0x7f6e240d3f80 2024-03-18T20:27:56.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.183+0000 7f6e2c411700 1 -- 172.21.15.1:0/468092167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e240d3f80 msgr2=0x7f6e2415d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:56.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.183+0000 7f6e2c411700 1 --2- 172.21.15.1:0/468092167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e240d3f80 0x7f6e2415d9c0 secure :-1 s=READY pgs=3195 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009840 tx=0x7f6e14009b50 comp rx=0 tx=0).stop 2024-03-18T20:27:56.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.183+0000 7f6e2c411700 1 -- 172.21.15.1:0/468092167 shutdown_connections 2024-03-18T20:27:56.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.183+0000 7f6e2c411700 1 --2- 172.21.15.1:0/468092167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e240d3f80 0x7f6e2415d9c0 unknown :-1 s=CLOSED pgs=3195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.183+0000 7f6e2c411700 1 -- 172.21.15.1:0/468092167 >> 172.21.15.1:0/468092167 conn(0x7f6e240d3380 msgr2=0x7f6e240d3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:56.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.183+0000 7f6e2c411700 1 -- 172.21.15.1:0/468092167 shutdown_connections 2024-03-18T20:27:56.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.183+0000 7f6e2c411700 1 -- 172.21.15.1:0/468092167 wait complete. 2024-03-18T20:27:56.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.184+0000 7f6e2c411700 1 Processor -- start 2024-03-18T20:27:56.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.184+0000 7f6e2c411700 1 -- start start 2024-03-18T20:27:56.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.185+0000 7f6e2c411700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e240d3f80 0x7f6e241503a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:56.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.185+0000 7f6e2c411700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e2415d5e0 0x7f6e240cbe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:56.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.185+0000 7f6e2c411700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e2415df00 0x7f6e240c6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:56.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.185+0000 7f6e2c411700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e240b6dc0 con 0x7f6e240d3f80 2024-03-18T20:27:56.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.185+0000 7f6e2c411700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e24161f10 con 0x7f6e2415d5e0 2024-03-18T20:27:56.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.185+0000 7f6e2c411700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e240b6a20 con 0x7f6e2415df00 2024-03-18T20:27:56.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.185+0000 7f6e2a1ad700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e240d3f80 0x7f6e241503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:56.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.185+0000 7f6e2a1ad700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e240d3f80 0x7f6e241503a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57920/0 (socket says 172.21.15.1:57920) 2024-03-18T20:27:56.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.185+0000 7f6e2a1ad700 1 -- 172.21.15.1:0/2425312999 learned_addr learned my addr 172.21.15.1:0/2425312999 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:56.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.185+0000 7f6e299ac700 1 --2- 172.21.15.1:0/2425312999 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e2415d5e0 0x7f6e240cbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:56.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.185+0000 7f6e2a9ae700 1 --2- 172.21.15.1:0/2425312999 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e2415df00 0x7f6e240c6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:56.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.186+0000 7f6e2a1ad700 1 -- 172.21.15.1:0/2425312999 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e2415df00 msgr2=0x7f6e240c6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:56.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.186+0000 7f6e2a1ad700 1 --2- 172.21.15.1:0/2425312999 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e2415df00 0x7f6e240c6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.186+0000 7f6e2a1ad700 1 -- 172.21.15.1:0/2425312999 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e2415d5e0 msgr2=0x7f6e240cbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:56.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.186+0000 7f6e2a1ad700 1 --2- 172.21.15.1:0/2425312999 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e2415d5e0 0x7f6e240cbe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.186+0000 7f6e2a1ad700 1 -- 172.21.15.1:0/2425312999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e140094a0 con 0x7f6e240d3f80 2024-03-18T20:27:56.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.186+0000 7f6e2a9ae700 1 --2- 172.21.15.1:0/2425312999 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e2415df00 0x7f6e240c6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:27:56.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.186+0000 7f6e2a1ad700 1 --2- 172.21.15.1:0/2425312999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e240d3f80 0x7f6e241503a0 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 crypto rx=0x7f6e2000c4e0 tx=0x7f6e2000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:56.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.186+0000 7f6e1affd700 1 -- 172.21.15.1:0/2425312999 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e200052d0 con 0x7f6e240d3f80 2024-03-18T20:27:56.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.186+0000 7f6e1affd700 1 -- 172.21.15.1:0/2425312999 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e20005430 con 0x7f6e240d3f80 2024-03-18T20:27:56.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.186+0000 7f6e1affd700 1 -- 172.21.15.1:0/2425312999 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e200038c0 con 0x7f6e240d3f80 2024-03-18T20:27:56.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.186+0000 7f6e2c411700 1 -- 172.21.15.1:0/2425312999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e240cc700 con 0x7f6e240d3f80 2024-03-18T20:27:56.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.186+0000 7f6e2c411700 1 -- 172.21.15.1:0/2425312999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e240cccc0 con 0x7f6e240d3f80 2024-03-18T20:27:56.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.187+0000 7f6e2c411700 1 -- 172.21.15.1:0/2425312999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e24067dc0 con 0x7f6e240d3f80 2024-03-18T20:27:56.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.187+0000 7f6e1affd700 1 -- 172.21.15.1:0/2425312999 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 163) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f6e20003a20 con 0x7f6e240d3f80 2024-03-18T20:27:56.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.188+0000 7f6e1affd700 1 --2- 172.21.15.1:0/2425312999 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6e0806ae70 0x7f6e0806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:56.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.188+0000 7f6e299ac700 1 -- 172.21.15.1:0/2425312999 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6e0806ae70 msgr2=0x7f6e0806d330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:56.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.188+0000 7f6e1affd700 1 -- 172.21.15.1:0/2425312999 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f6e20088e30 con 0x7f6e240d3f80 2024-03-18T20:27:56.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.188+0000 7f6e299ac700 1 --2- 172.21.15.1:0/2425312999 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6e0806ae70 0x7f6e0806d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:27:56.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.193+0000 7f6e1affd700 1 -- 172.21.15.1:0/2425312999 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6e200539b0 con 0x7f6e240d3f80 2024-03-18T20:27:56.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.389+0000 7f6e299ac700 1 -- 172.21.15.1:0/2425312999 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6e0806ae70 msgr2=0x7f6e0806d330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.389+0000 7f6e299ac700 1 --2- 172.21.15.1:0/2425312999 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6e0806ae70 0x7f6e0806d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:27:56.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.396+0000 7f6e2c411700 1 -- 172.21.15.1:0/2425312999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e2408d190 con 0x7f6e240d3f80 2024-03-18T20:27:56.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.398+0000 7f6e1affd700 1 -- 172.21.15.1:0/2425312999 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v275) v1 ==== 74+0+16970 (secure 0 0 0) 0x7f6e20017020 con 0x7f6e240d3f80 2024-03-18T20:27:56.402 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:56.402 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":275,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:27:42.143916+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:27:19.891286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"273","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-03-18T20:27:21.027315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"274","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3635558645":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2293377213":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/1796830610":"2024-03-19T20:27:42.143876+0000","172.21.15.1:6833/148356":"2024-03-19T20:27:42.143876+0000","172.21.15.1:6832/148356":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2068313472":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2668486132":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2560935779":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/82444607":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/1446581052":"2024-03-19T20:27:42.143876+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:27:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.400+0000 7f6e2c411700 1 -- 172.21.15.1:0/2425312999 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6e0806ae70 msgr2=0x7f6e0806d330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:27:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.400+0000 7f6e2c411700 1 --2- 172.21.15.1:0/2425312999 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6e0806ae70 0x7f6e0806d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.400+0000 7f6e2c411700 1 -- 172.21.15.1:0/2425312999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e240d3f80 msgr2=0x7f6e241503a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.400+0000 7f6e2c411700 1 --2- 172.21.15.1:0/2425312999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e240d3f80 0x7f6e241503a0 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 crypto rx=0x7f6e2000c4e0 tx=0x7f6e2000c7f0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.401+0000 7f6e2c411700 1 -- 172.21.15.1:0/2425312999 shutdown_connections 2024-03-18T20:27:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.401+0000 7f6e2c411700 1 --2- 172.21.15.1:0/2425312999 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6e0806ae70 0x7f6e0806d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.401+0000 7f6e2c411700 1 --2- 172.21.15.1:0/2425312999 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6e2415df00 0x7f6e240c6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.401+0000 7f6e2c411700 1 --2- 172.21.15.1:0/2425312999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e240d3f80 0x7f6e241503a0 unknown :-1 s=CLOSED pgs=3196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.401+0000 7f6e2c411700 1 --2- 172.21.15.1:0/2425312999 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e2415d5e0 0x7f6e240cbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.401+0000 7f6e2c411700 1 -- 172.21.15.1:0/2425312999 >> 172.21.15.1:0/2425312999 conn(0x7f6e240d3380 msgr2=0x7f6e240b2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:56.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.401+0000 7f6e2c411700 1 -- 172.21.15.1:0/2425312999 shutdown_connections 2024-03-18T20:27:56.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.401+0000 7f6e2c411700 1 -- 172.21.15.1:0/2425312999 wait complete. 2024-03-18T20:27:56.418 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T20:27:56.418 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:27:56.461 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:56.458+0000 7efcc9740240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:27:56.528 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:56.523+0000 7f4e60685240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:27:56.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.540+0000 7f5dd5459700 1 Processor -- start 2024-03-18T20:27:56.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.540+0000 7f5dd5459700 1 -- start start 2024-03-18T20:27:56.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.541+0000 7f5dd5459700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dd00d3f80 0x7f5dd015d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:56.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.541+0000 7f5dd5459700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5dd015df00 0x7f5dd00cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:56.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.541+0000 7f5dd5459700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd00cfe30 con 0x7f5dd015d5e0 2024-03-18T20:27:56.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.541+0000 7f5dd5459700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd00d24e0 con 0x7f5dd015df00 2024-03-18T20:27:56.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.541+0000 7f5dd5459700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd00d0110 con 0x7f5dd00d3f80 2024-03-18T20:27:56.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.541+0000 7f5dcf7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5dd015df00 0x7f5dd00cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:56.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.541+0000 7f5dcf7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5dd015df00 0x7f5dd00cdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57924/0 (socket says 172.21.15.1:57924) 2024-03-18T20:27:56.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.541+0000 7f5dcf7fe700 1 -- 172.21.15.1:0/1623560371 learned_addr learned my addr 172.21.15.1:0/1623560371 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:56.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.541+0000 7f5dce7fc700 1 --2- 172.21.15.1:0/1623560371 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dd00d3f80 0x7f5dd015d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:56.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.541+0000 7f5dcf7fe700 1 -- 172.21.15.1:0/1623560371 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dd00d3f80 msgr2=0x7f5dd015d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:56.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.541+0000 7f5dcf7fe700 1 --2- 172.21.15.1:0/1623560371 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dd00d3f80 0x7f5dd015d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.541+0000 7f5dcf7fe700 1 -- 172.21.15.1:0/1623560371 >> v1:172.21.15.165:6789/0 conn(0x7f5dd015d5e0 legacy=0x7f5dd00d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:56.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.541+0000 7f5dcf7fe700 1 -- 172.21.15.1:0/1623560371 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd00d03f0 con 0x7f5dd015df00 2024-03-18T20:27:56.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.541+0000 7f5dcdffb700 1 -- 172.21.15.1:0/1623560371 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144229583 0 0) 0x7f5dd00cfe30 con 0x7f5dd015d5e0 2024-03-18T20:27:56.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.541+0000 7f5dcf7fe700 1 --2- 172.21.15.1:0/1623560371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd015df00 0x7f5dd00cdf90 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8009e50 tx=0x7f5dc800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4be3d6c786efd0a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:56.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.541+0000 7f5dcdffb700 1 -- 172.21.15.1:0/1623560371 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc800e070 con 0x7f5dd015df00 2024-03-18T20:27:56.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.542+0000 7f5dcdffb700 1 -- 172.21.15.1:0/1623560371 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc800b910 con 0x7f5dd015df00 2024-03-18T20:27:56.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.542+0000 7f5dcdffb700 1 -- 172.21.15.1:0/1623560371 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc800bcc0 con 0x7f5dd015df00 2024-03-18T20:27:56.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.542+0000 7f5dd5459700 1 -- 172.21.15.1:0/1623560371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd015df00 msgr2=0x7f5dd00cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:56.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.542+0000 7f5dd5459700 1 --2- 172.21.15.1:0/1623560371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd015df00 0x7f5dd00cdf90 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8009e50 tx=0x7f5dc800b330 comp rx=0 tx=0).stop 2024-03-18T20:27:56.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.542+0000 7f5dd5459700 1 -- 172.21.15.1:0/1623560371 shutdown_connections 2024-03-18T20:27:56.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.542+0000 7f5dd5459700 1 --2- 172.21.15.1:0/1623560371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd015df00 0x7f5dd00cdf90 unknown :-1 s=CLOSED pgs=3197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.542+0000 7f5dd5459700 1 --2- 172.21.15.1:0/1623560371 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dd00d3f80 0x7f5dd015d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.542+0000 7f5dd5459700 1 -- 172.21.15.1:0/1623560371 >> 172.21.15.1:0/1623560371 conn(0x7f5dd00d3380 msgr2=0x7f5dd00d3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:56.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.542+0000 7f5dd5459700 1 -- 172.21.15.1:0/1623560371 shutdown_connections 2024-03-18T20:27:56.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.542+0000 7f5dd5459700 1 -- 172.21.15.1:0/1623560371 wait complete. 2024-03-18T20:27:56.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.544+0000 7f5dd5459700 1 Processor -- start 2024-03-18T20:27:56.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.544+0000 7f5dd5459700 1 -- start start 2024-03-18T20:27:56.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.544+0000 7f5dd5459700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd00d3f80 0x7f5dd00c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:56.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.544+0000 7f5dd5459700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dd015d5e0 0x7f5dd01e8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:56.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.544+0000 7f5dd5459700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dd015df00 0x7f5dd014eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:56.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.544+0000 7f5dd5459700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd00c7dc0 con 0x7f5dd00d3f80 2024-03-18T20:27:56.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.544+0000 7f5dd5459700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd00c80a0 con 0x7f5dd015d5e0 2024-03-18T20:27:56.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.544+0000 7f5dd5459700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd00c8380 con 0x7f5dd015df00 2024-03-18T20:27:56.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.544+0000 7f5dceffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd00d3f80 0x7f5dd00c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:56.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.544+0000 7f5dceffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd00d3f80 0x7f5dd00c96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57940/0 (socket says 172.21.15.1:57940) 2024-03-18T20:27:56.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.544+0000 7f5dceffd700 1 -- 172.21.15.1:0/1977583635 learned_addr learned my addr 172.21.15.1:0/1977583635 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:56.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.544+0000 7f5dce7fc700 1 --2- 172.21.15.1:0/1977583635 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dd015d5e0 0x7f5dd01e8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:56.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.545+0000 7f5dcf7fe700 1 --2- 172.21.15.1:0/1977583635 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dd015df00 0x7f5dd014eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:56.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.545+0000 7f5dceffd700 1 -- 172.21.15.1:0/1977583635 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dd015df00 msgr2=0x7f5dd014eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:56.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.545+0000 7f5dceffd700 1 --2- 172.21.15.1:0/1977583635 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dd015df00 0x7f5dd014eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.545+0000 7f5dceffd700 1 -- 172.21.15.1:0/1977583635 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dd015d5e0 msgr2=0x7f5dd01e8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:56.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.545+0000 7f5dceffd700 1 --2- 172.21.15.1:0/1977583635 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dd015d5e0 0x7f5dd01e8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.545+0000 7f5dceffd700 1 -- 172.21.15.1:0/1977583635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc8009b30 con 0x7f5dd00d3f80 2024-03-18T20:27:56.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.545+0000 7f5dceffd700 1 --2- 172.21.15.1:0/1977583635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd00d3f80 0x7f5dd00c96f0 secure :-1 s=READY pgs=3198 cs=0 l=1 rev1=1 crypto rx=0x7f5dc00011e0 tx=0x7f5dc000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:56.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.545+0000 7f5db77fe700 1 -- 172.21.15.1:0/1977583635 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc00054e0 con 0x7f5dd00d3f80 2024-03-18T20:27:56.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.545+0000 7f5db77fe700 1 -- 172.21.15.1:0/1977583635 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc0003680 con 0x7f5dd00d3f80 2024-03-18T20:27:56.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.545+0000 7f5dd5459700 1 -- 172.21.15.1:0/1977583635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd00c8660 con 0x7f5dd00d3f80 2024-03-18T20:27:56.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.545+0000 7f5db77fe700 1 -- 172.21.15.1:0/1977583635 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0003a30 con 0x7f5dd00d3f80 2024-03-18T20:27:56.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.545+0000 7f5dd5459700 1 -- 172.21.15.1:0/1977583635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd00b91a0 con 0x7f5dd00d3f80 2024-03-18T20:27:56.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.546+0000 7f5dd5459700 1 -- 172.21.15.1:0/1977583635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd0067dc0 con 0x7f5dd00d3f80 2024-03-18T20:27:56.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.547+0000 7f5db77fe700 1 -- 172.21.15.1:0/1977583635 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 163) v1 ==== 89296+0+0 (secure 0 0 0) 0x7f5dc0003b90 con 0x7f5dd00d3f80 2024-03-18T20:27:56.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.547+0000 7f5db77fe700 1 --2- 172.21.15.1:0/1977583635 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5dbc06ae40 0x7f5dbc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:56.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.547+0000 7f5dce7fc700 1 -- 172.21.15.1:0/1977583635 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5dbc06ae40 msgr2=0x7f5dbc06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:56.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.547+0000 7f5db77fe700 1 -- 172.21.15.1:0/1977583635 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7f5dc0088d00 con 0x7f5dd00d3f80 2024-03-18T20:27:56.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.547+0000 7f5dce7fc700 1 --2- 172.21.15.1:0/1977583635 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5dbc06ae40 0x7f5dbc06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:27:56.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.551+0000 7f5db77fe700 1 -- 172.21.15.1:0/1977583635 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5dc00539a0 con 0x7f5dd00d3f80 2024-03-18T20:27:56.617 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:27:56.613+0000 7f4e60685240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:27:56.647 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:56.643+0000 7efcc9740240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:27:56.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.748+0000 7f5dce7fc700 1 -- 172.21.15.1:0/1977583635 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5dbc06ae40 msgr2=0x7f5dbc06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:56.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.748+0000 7f5dce7fc700 1 --2- 172.21.15.1:0/1977583635 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5dbc06ae40 0x7f5dbc06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:27:56.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.800+0000 7f5dd5459700 1 -- 172.21.15.1:0/1977583635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5dd00cfb70 con 0x7f5dd00d3f80 2024-03-18T20:27:56.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.800+0000 7f5db77fe700 1 -- 172.21.15.1:0/1977583635 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+4749 (secure 0 0 0) 0x7f5dc0017020 con 0x7f5dd00d3f80 2024-03-18T20:27:56.804 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:56.805 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":136,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9938,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/1320583552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1320583552},{"type":"v1","addr":"172.21.15.165:6833","nonce":1320583552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131},{"gid":11566,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2669900315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2669900315},{"type":"v1","addr":"172.21.15.165:6835","nonce":2669900315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131},{"gid":13170,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2726723271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2726723271},{"type":"v1","addr":"172.21.15.1:6835","nonce":2726723271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:27:22.047616+0000","modified":"2024-03-18T20:27:25.113850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13173},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13173":{"gid":13173,"name":"c","rank":0,"incarnation":132,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/3909787366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":3909787366},{"type":"v1","addr":"172.21.15.1:6837","nonce":3909787366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-03-18T20:27:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.803+0000 7f5dd5459700 1 -- 172.21.15.1:0/1977583635 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5dbc06ae40 msgr2=0x7f5dbc06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:27:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.803+0000 7f5dd5459700 1 --2- 172.21.15.1:0/1977583635 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5dbc06ae40 0x7f5dbc06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.803+0000 7f5dd5459700 1 -- 172.21.15.1:0/1977583635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd00d3f80 msgr2=0x7f5dd00c96f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.803+0000 7f5dd5459700 1 --2- 172.21.15.1:0/1977583635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd00d3f80 0x7f5dd00c96f0 secure :-1 s=READY pgs=3198 cs=0 l=1 rev1=1 crypto rx=0x7f5dc00011e0 tx=0x7f5dc000ca30 comp rx=0 tx=0).stop 2024-03-18T20:27:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.803+0000 7f5dd5459700 1 -- 172.21.15.1:0/1977583635 shutdown_connections 2024-03-18T20:27:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.803+0000 7f5dd5459700 1 --2- 172.21.15.1:0/1977583635 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5dbc06ae40 0x7f5dbc06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.803+0000 7f5dd5459700 1 --2- 172.21.15.1:0/1977583635 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5dd015df00 0x7f5dd014eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.803+0000 7f5dd5459700 1 --2- 172.21.15.1:0/1977583635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5dd00d3f80 0x7f5dd00c96f0 unknown :-1 s=CLOSED pgs=3198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.803+0000 7f5dd5459700 1 --2- 172.21.15.1:0/1977583635 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dd015d5e0 0x7f5dd01e8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.803+0000 7f5dd5459700 1 -- 172.21.15.1:0/1977583635 >> 172.21.15.1:0/1977583635 conn(0x7f5dd00d3380 msgr2=0x7f5dd00b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:56.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.803+0000 7f5dd5459700 1 -- 172.21.15.1:0/1977583635 shutdown_connections 2024-03-18T20:27:56.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.803+0000 7f5dd5459700 1 -- 172.21.15.1:0/1977583635 wait complete. 2024-03-18T20:27:56.808 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 136 2024-03-18T20:27:56.821 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:27:56.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.987+0000 7fa745b72700 1 Processor -- start 2024-03-18T20:27:56.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.988+0000 7fa745b72700 1 -- start start 2024-03-18T20:27:56.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.988+0000 7fa745b72700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7400cf200 0x7fa74015c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:56.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.988+0000 7fa745b72700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa7400cbc90 0x7fa740099ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:56.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.988+0000 7fa745b72700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa74015ca40 con 0x7fa74015b6c0 2024-03-18T20:27:56.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.988+0000 7fa745b72700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7400cce30 con 0x7fa7400cbc90 2024-03-18T20:27:56.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.988+0000 7fa745b72700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7400cc220 con 0x7fa7400cf200 2024-03-18T20:27:56.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.988+0000 7fa73effd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa74015b6c0 0x7fa7400cf5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39546/0 (socket says 172.21.15.1:39546) 2024-03-18T20:27:56.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.988+0000 7fa73effd700 1 -- 172.21.15.1:0/468436161 learned_addr learned my addr 172.21.15.1:0/468436161 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:56.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73ffff700 1 --2- 172.21.15.1:0/468436161 >> v2:172.21.15.165:3300/0 conn(0x7fa7400cbc90 0x7fa740099ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:56.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73f7fe700 1 --2- 172.21.15.1:0/468436161 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7400cf200 0x7fa74015c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:56.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73e7fc700 1 -- 172.21.15.1:0/468436161 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414037449 0 0) 0x7fa734001110 con 0x7fa74015b6c0 2024-03-18T20:27:56.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73e7fc700 1 -- 172.21.15.1:0/468436161 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7280035b0 con 0x7fa74015b6c0 2024-03-18T20:27:56.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73e7fc700 1 -- 172.21.15.1:0/468436161 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 82694044 0 0) 0x7fa7280035b0 con 0x7fa74015b6c0 2024-03-18T20:27:56.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73e7fc700 1 -- 172.21.15.1:0/468436161 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7400cf200 msgr2=0x7fa74015c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:56.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73e7fc700 1 --2- 172.21.15.1:0/468436161 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7400cf200 0x7fa74015c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73e7fc700 1 -- 172.21.15.1:0/468436161 >> v2:172.21.15.165:3300/0 conn(0x7fa7400cbc90 msgr2=0x7fa740099ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:56.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73e7fc700 1 --2- 172.21.15.1:0/468436161 >> v2:172.21.15.165:3300/0 conn(0x7fa7400cbc90 0x7fa740099ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73e7fc700 1 -- 172.21.15.1:0/468436161 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7400cc500 con 0x7fa74015b6c0 2024-03-18T20:27:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73e7fc700 1 -- 172.21.15.1:0/468436161 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fa734001800 con 0x7fa74015b6c0 2024-03-18T20:27:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73e7fc700 1 -- 172.21.15.1:0/468436161 >> v1:172.21.15.1:6789/0 conn(0x7fa74015b6c0 legacy=0x7fa7400cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73e7fc700 1 --2- 172.21.15.1:0/468436161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa728003a00 0x7fa728003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73e7fc700 1 --2- 172.21.15.1:0/468436161 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7400cf200 0x7fa728004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73e7fc700 1 --2- 172.21.15.1:0/468436161 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa728004d90 0x7fa728005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73e7fc700 1 -- 172.21.15.1:0/468436161 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7400cce30 con 0x7fa728003a00 2024-03-18T20:27:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73e7fc700 1 -- 172.21.15.1:0/468436161 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7400cc220 con 0x7fa728004d90 2024-03-18T20:27:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73e7fc700 1 -- 172.21.15.1:0/468436161 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa72800ba10 con 0x7fa7400cf200 2024-03-18T20:27:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73f7fe700 1 --2- 172.21.15.1:0/468436161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa728003a00 0x7fa728003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73effd700 1 --2- 172.21.15.1:0/468436161 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7400cf200 0x7fa728004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.989+0000 7fa73ffff700 1 --2- 172.21.15.1:0/468436161 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa728004d90 0x7fa728005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.990+0000 7fa73f7fe700 1 -- 172.21.15.1:0/468436161 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7400cf200 msgr2=0x7fa728004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.990+0000 7fa73f7fe700 1 --2- 172.21.15.1:0/468436161 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7400cf200 0x7fa728004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.990+0000 7fa73f7fe700 1 -- 172.21.15.1:0/468436161 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa728004d90 msgr2=0x7fa728005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:56.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.990+0000 7fa73f7fe700 1 --2- 172.21.15.1:0/468436161 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa728004d90 0x7fa728005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:56.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.990+0000 7fa73f7fe700 1 -- 172.21.15.1:0/468436161 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7280035b0 con 0x7fa728003a00 2024-03-18T20:27:56.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.990+0000 7fa73f7fe700 1 --2- 172.21.15.1:0/468436161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa728003a00 0x7fa728003e20 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 crypto rx=0x7fa730009b60 tx=0x7fa728005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f8ea8d9bfbf394 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:56.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.990+0000 7fa73e7fc700 1 -- 172.21.15.1:0/468436161 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73000e6c0 con 0x7fa728003a00 2024-03-18T20:27:56.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.990+0000 7fa73e7fc700 1 -- 172.21.15.1:0/468436161 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73000e820 con 0x7fa728003a00 2024-03-18T20:27:56.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.990+0000 7fa73e7fc700 1 -- 172.21.15.1:0/468436161 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73000eb90 con 0x7fa728003a00 2024-03-18T20:27:56.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.991+0000 7fa745b72700 1 -- 172.21.15.1:0/468436161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa728003a00 msgr2=0x7fa728003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:56.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.991+0000 7fa745b72700 1 --2- 172.21.15.1:0/468436161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa728003a00 0x7fa728003e20 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 crypto rx=0x7fa730009b60 tx=0x7fa728005880 comp rx=0 tx=0).stop 2024-03-18T20:27:56.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.991+0000 7fa73ffff700 1 -- 172.21.15.1:0/468436161 reap_dead start 2024-03-18T20:27:56.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.991+0000 7fa745b72700 1 -- 172.21.15.1:0/468436161 shutdown_connections 2024-03-18T20:27:56.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.991+0000 7fa745b72700 1 -- 172.21.15.1:0/468436161 >> 172.21.15.1:0/468436161 conn(0x7fa7400d1180 msgr2=0x7fa7400d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:56.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.991+0000 7fa745b72700 1 -- 172.21.15.1:0/468436161 shutdown_connections 2024-03-18T20:27:56.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.991+0000 7fa745b72700 1 -- 172.21.15.1:0/468436161 wait complete. 2024-03-18T20:27:57.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.992+0000 7fa745b72700 1 Processor -- start 2024-03-18T20:27:57.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.992+0000 7fa745b72700 1 -- start start 2024-03-18T20:27:57.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.993+0000 7fa745b72700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa740154a50 0x7fa7401e5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:57.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.993+0000 7fa745b72700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7401e64c0 0x7fa7400c4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:57.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.993+0000 7fa745b72700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7400c5140 0x7fa7400c6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.993+0000 7fa745b72700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7400c55f0 con 0x7fa740154a50 2024-03-18T20:27:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.993+0000 7fa745b72700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7401e6940 con 0x7fa7401e64c0 2024-03-18T20:27:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.993+0000 7fa745b72700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74006d930 con 0x7fa7400c5140 2024-03-18T20:27:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.993+0000 7fa73f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa740154a50 0x7fa7401e5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.993+0000 7fa73f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa740154a50 0x7fa7401e5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57952/0 (socket says 172.21.15.1:57952) 2024-03-18T20:27:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.993+0000 7fa73f7fe700 1 -- 172.21.15.1:0/1202078807 learned_addr learned my addr 172.21.15.1:0/1202078807 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.993+0000 7fa73effd700 1 --2- 172.21.15.1:0/1202078807 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7401e64c0 0x7fa7400c4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.993+0000 7fa73ffff700 1 --2- 172.21.15.1:0/1202078807 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7400c5140 0x7fa7400c6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.993+0000 7fa73f7fe700 1 -- 172.21.15.1:0/1202078807 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7400c5140 msgr2=0x7fa7400c6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.993+0000 7fa73f7fe700 1 --2- 172.21.15.1:0/1202078807 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7400c5140 0x7fa7400c6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.993+0000 7fa73f7fe700 1 -- 172.21.15.1:0/1202078807 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7401e64c0 msgr2=0x7fa7400c4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.993+0000 7fa73f7fe700 1 --2- 172.21.15.1:0/1202078807 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7401e64c0 0x7fa7400c4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.993+0000 7fa73f7fe700 1 -- 172.21.15.1:0/1202078807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7400c57e0 con 0x7fa740154a50 2024-03-18T20:27:57.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.993+0000 7fa73effd700 1 --2- 172.21.15.1:0/1202078807 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7401e64c0 0x7fa7400c4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:27:57.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.994+0000 7fa73f7fe700 1 --2- 172.21.15.1:0/1202078807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa740154a50 0x7fa7401e5f80 secure :-1 s=READY pgs=3201 cs=0 l=1 rev1=1 crypto rx=0x7fa728004730 tx=0x7fa7300047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:57.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.994+0000 7fa727fff700 1 -- 172.21.15.1:0/1202078807 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73000e6c0 con 0x7fa740154a50 2024-03-18T20:27:57.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.994+0000 7fa727fff700 1 -- 172.21.15.1:0/1202078807 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7300038c0 con 0x7fa740154a50 2024-03-18T20:27:57.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.994+0000 7fa727fff700 1 -- 172.21.15.1:0/1202078807 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa730003c70 con 0x7fa740154a50 2024-03-18T20:27:57.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.994+0000 7fa745b72700 1 -- 172.21.15.1:0/1202078807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7400c5ac0 con 0x7fa740154a50 2024-03-18T20:27:57.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.994+0000 7fa745b72700 1 -- 172.21.15.1:0/1202078807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7400c6080 con 0x7fa740154a50 2024-03-18T20:27:57.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.995+0000 7fa745b72700 1 -- 172.21.15.1:0/1202078807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7400d0ca0 con 0x7fa740154a50 2024-03-18T20:27:57.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.995+0000 7fa727fff700 1 -- 172.21.15.1:0/1202078807 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 163) v1 ==== 89296+0+0 (secure 0 0 0) 0x7fa730013470 con 0x7fa740154a50 2024-03-18T20:27:57.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.996+0000 7fa727fff700 1 --2- 172.21.15.1:0/1202078807 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa72c06ae40 0x7fa72c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:57.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.996+0000 7fa73effd700 1 -- 172.21.15.1:0/1202078807 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa72c06ae40 msgr2=0x7fa72c06d300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:27:57.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.996+0000 7fa727fff700 1 -- 172.21.15.1:0/1202078807 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6559+0+0 (secure 0 0 0) 0x7fa73008b0c0 con 0x7fa740154a50 2024-03-18T20:27:57.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:56.996+0000 7fa73effd700 1 --2- 172.21.15.1:0/1202078807 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa72c06ae40 0x7fa72c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:27:57.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.000+0000 7fa727fff700 1 -- 172.21.15.1:0/1202078807 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa730055d60 con 0x7fa740154a50 2024-03-18T20:27:57.039 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:57.035+0000 7efcc9740240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:27:57.135 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:57.131+0000 7efcc9740240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:27:57.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.150+0000 7fa727fff700 1 -- 172.21.15.1:0/1202078807 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 164) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fa730056130 con 0x7fa740154a50 2024-03-18T20:27:57.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.150+0000 7fa727fff700 1 -- 172.21.15.1:0/1202078807 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa72c06ae40 msgr2=0x7fa72c06d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:27:57.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.150+0000 7fa727fff700 1 --2- 172.21.15.1:0/1202078807 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa72c06ae40 0x7fa72c06d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:57.223 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:57.219+0000 7efc41b68700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:57.223 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:57.219+0000 7efc41b68700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:57.223 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:57.219+0000 7efc41b68700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:57.223 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:57.219+0000 7efc4536f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:57.223 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:57.219+0000 7efc4536f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:57.223 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:57.219+0000 7efc4536f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:57.223 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:57.219+0000 7efc4536f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:57.224 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:57.219+0000 7efc4536f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:57.224 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:57.219+0000 7efc41b68700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:57.224 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:27:57.219+0000 7efc41b68700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:27:57.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.270+0000 7fa745b72700 1 -- 172.21.15.1:0/1202078807 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa740067dc0 con 0x7fa740154a50 2024-03-18T20:27:57.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.270+0000 7fa727fff700 1 -- 172.21.15.1:0/1202078807 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+4749 (secure 0 0 0) 0x7fa730000e60 con 0x7fa740154a50 2024-03-18T20:27:57.275 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:57.275 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":136,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9938,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/1320583552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1320583552},{"type":"v1","addr":"172.21.15.165:6833","nonce":1320583552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131},{"gid":11566,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2669900315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2669900315},{"type":"v1","addr":"172.21.15.165:6835","nonce":2669900315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131},{"gid":13170,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2726723271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2726723271},{"type":"v1","addr":"172.21.15.1:6835","nonce":2726723271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:27:22.047616+0000","modified":"2024-03-18T20:27:25.113850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13173},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13173":{"gid":13173,"name":"c","rank":0,"incarnation":132,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6837/3909787366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":3909787366},{"type":"v1","addr":"172.21.15.1:6837","nonce":3909787366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[36],"metadata_pool":35,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2024-03-18T20:27:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.273+0000 7fa745b72700 1 -- 172.21.15.1:0/1202078807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa740154a50 msgr2=0x7fa7401e5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.273+0000 7fa745b72700 1 --2- 172.21.15.1:0/1202078807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa740154a50 0x7fa7401e5f80 secure :-1 s=READY pgs=3201 cs=0 l=1 rev1=1 crypto rx=0x7fa728004730 tx=0x7fa7300047c0 comp rx=0 tx=0).stop 2024-03-18T20:27:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.273+0000 7fa745b72700 1 -- 172.21.15.1:0/1202078807 shutdown_connections 2024-03-18T20:27:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.273+0000 7fa745b72700 1 --2- 172.21.15.1:0/1202078807 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa72c06ae40 0x7fa72c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.273+0000 7fa745b72700 1 --2- 172.21.15.1:0/1202078807 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa7400c5140 0x7fa7400c6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.273+0000 7fa745b72700 1 --2- 172.21.15.1:0/1202078807 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa740154a50 0x7fa7401e5f80 unknown :-1 s=CLOSED pgs=3201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.273+0000 7fa745b72700 1 --2- 172.21.15.1:0/1202078807 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7401e64c0 0x7fa7400c4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.273+0000 7fa745b72700 1 -- 172.21.15.1:0/1202078807 >> 172.21.15.1:0/1202078807 conn(0x7fa7400d1180 msgr2=0x7fa7400b0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.273+0000 7fa745b72700 1 -- 172.21.15.1:0/1202078807 shutdown_connections 2024-03-18T20:27:57.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.273+0000 7fa745b72700 1 -- 172.21.15.1:0/1202078807 wait complete. 2024-03-18T20:27:57.278 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 136 2024-03-18T20:27:57.290 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:27:57.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.412+0000 7f154f6c1700 1 Processor -- start 2024-03-18T20:27:57.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.413+0000 7f154f6c1700 1 -- start start 2024-03-18T20:27:57.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.413+0000 7f154f6c1700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1548140cb0 0x7f1548142f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:57.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.413+0000 7f154f6c1700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1548142a70 con 0x7f1548143440 2024-03-18T20:27:57.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.413+0000 7f154f6c1700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1548144120 con 0x7f154815b230 2024-03-18T20:27:57.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.413+0000 7f154f6c1700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1548141e60 con 0x7f1548140cb0 2024-03-18T20:27:57.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.413+0000 7f154cc5c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1548143440 0x7f154815ab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39556/0 (socket says 172.21.15.1:39556) 2024-03-18T20:27:57.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.414+0000 7f154cc5c700 1 -- 172.21.15.1:0/1072519640 learned_addr learned my addr 172.21.15.1:0/1072519640 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:57.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.414+0000 7f154d45d700 1 --2- 172.21.15.1:0/1072519640 >> v2:172.21.15.165:3300/0 conn(0x7f1548140cb0 0x7f1548142f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:57.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.414+0000 7f153bfff700 1 -- 172.21.15.1:0/1072519640 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302615489 0 0) 0x7f1548142a70 con 0x7f1548143440 2024-03-18T20:27:57.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.414+0000 7f153bfff700 1 -- 172.21.15.1:0/1072519640 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15340035b0 con 0x7f1548143440 2024-03-18T20:27:57.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.414+0000 7f154d45d700 1 -- 172.21.15.1:0/1072519640 >> v1:172.21.15.165:6789/0 conn(0x7f154815b230 legacy=0x7f15480b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:57.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.414+0000 7f154d45d700 1 -- 172.21.15.1:0/1072519640 >> v1:172.21.15.1:6789/0 conn(0x7f1548143440 legacy=0x7f154815ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:57.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.414+0000 7f154d45d700 1 -- 172.21.15.1:0/1072519640 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1548142140 con 0x7f1548140cb0 2024-03-18T20:27:57.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.414+0000 7f154d45d700 1 --2- 172.21.15.1:0/1072519640 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1548140cb0 0x7f1548142f00 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f1544009840 tx=0x7f1544009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ced08fec81f8672 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:57.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.414+0000 7f153bfff700 1 -- 172.21.15.1:0/1072519640 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1544011830 con 0x7f1548140cb0 2024-03-18T20:27:57.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.414+0000 7f153bfff700 1 -- 172.21.15.1:0/1072519640 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1544011990 con 0x7f1548140cb0 2024-03-18T20:27:57.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.414+0000 7f153bfff700 1 -- 172.21.15.1:0/1072519640 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1544011d20 con 0x7f1548140cb0 2024-03-18T20:27:57.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.415+0000 7f154f6c1700 1 -- 172.21.15.1:0/1072519640 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1548140cb0 msgr2=0x7f1548142f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:57.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.415+0000 7f154f6c1700 1 --2- 172.21.15.1:0/1072519640 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1548140cb0 0x7f1548142f00 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f1544009840 tx=0x7f1544009b50 comp rx=0 tx=0).stop 2024-03-18T20:27:57.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.415+0000 7f154f6c1700 1 -- 172.21.15.1:0/1072519640 shutdown_connections 2024-03-18T20:27:57.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.415+0000 7f154f6c1700 1 --2- 172.21.15.1:0/1072519640 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1548140cb0 0x7f1548142f00 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:57.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.415+0000 7f154f6c1700 1 -- 172.21.15.1:0/1072519640 >> 172.21.15.1:0/1072519640 conn(0x7f15480b5780 msgr2=0x7f15480b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:57.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.415+0000 7f154f6c1700 1 -- 172.21.15.1:0/1072519640 shutdown_connections 2024-03-18T20:27:57.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.415+0000 7f154f6c1700 1 -- 172.21.15.1:0/1072519640 wait complete. 2024-03-18T20:27:57.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.416+0000 7f154f6c1700 1 Processor -- start 2024-03-18T20:27:57.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.416+0000 7f154f6c1700 1 -- start start 2024-03-18T20:27:57.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.417+0000 7f154f6c1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1548140cb0 0x7f1548159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:57.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.417+0000 7f154f6c1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f154815b230 0x7f15480ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:57.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.417+0000 7f154f6c1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f154815d0a0 0x7f154813c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:57.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.417+0000 7f154f6c1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15480caa80 con 0x7f154815b230 2024-03-18T20:27:57.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.417+0000 7f154f6c1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154815a660 con 0x7f1548140cb0 2024-03-18T20:27:57.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.417+0000 7f154f6c1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154815d550 con 0x7f154815d0a0 2024-03-18T20:27:57.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.417+0000 7f154cc5c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f154815b230 0x7f15480ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:57.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.417+0000 7f154cc5c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f154815b230 0x7f15480ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58050/0 (socket says 172.21.15.1:58050) 2024-03-18T20:27:57.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.417+0000 7f154cc5c700 1 -- 172.21.15.1:0/3407300272 learned_addr learned my addr 172.21.15.1:0/3407300272 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:57.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.417+0000 7f154d45d700 1 --2- 172.21.15.1:0/3407300272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1548140cb0 0x7f1548159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.417+0000 7f154dc5e700 1 --2- 172.21.15.1:0/3407300272 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f154815d0a0 0x7f154813c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.417+0000 7f154cc5c700 1 -- 172.21.15.1:0/3407300272 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f154815d0a0 msgr2=0x7f154813c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.417+0000 7f154cc5c700 1 --2- 172.21.15.1:0/3407300272 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f154815d0a0 0x7f154813c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.417+0000 7f154cc5c700 1 -- 172.21.15.1:0/3407300272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1548140cb0 msgr2=0x7f1548159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.417+0000 7f154cc5c700 1 --2- 172.21.15.1:0/3407300272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1548140cb0 0x7f1548159de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.417+0000 7f154cc5c700 1 -- 172.21.15.1:0/3407300272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15440094a0 con 0x7f154815b230 2024-03-18T20:27:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.417+0000 7f154d45d700 1 --2- 172.21.15.1:0/3407300272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1548140cb0 0x7f1548159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:27:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.418+0000 7f154cc5c700 1 --2- 172.21.15.1:0/3407300272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f154815b230 0x7f15480ca540 secure :-1 s=READY pgs=3210 cs=0 l=1 rev1=1 crypto rx=0x7f153c00c730 tx=0x7f153c00ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.418+0000 7f1539ffb700 1 -- 172.21.15.1:0/3407300272 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c005d70 con 0x7f154815b230 2024-03-18T20:27:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.418+0000 7f1539ffb700 1 -- 172.21.15.1:0/3407300272 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153c0070d0 con 0x7f154815b230 2024-03-18T20:27:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.418+0000 7f154f6c1700 1 -- 172.21.15.1:0/3407300272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15480c9780 con 0x7f154815b230 2024-03-18T20:27:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.418+0000 7f1539ffb700 1 -- 172.21.15.1:0/3407300272 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c007440 con 0x7f154815b230 2024-03-18T20:27:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.418+0000 7f154f6c1700 1 -- 172.21.15.1:0/3407300272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15480c9d40 con 0x7f154815b230 2024-03-18T20:27:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.419+0000 7f154f6c1700 1 -- 172.21.15.1:0/3407300272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15480cc480 con 0x7f154815b230 2024-03-18T20:27:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.419+0000 7f1539ffb700 1 -- 172.21.15.1:0/3407300272 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 164) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f153c0075a0 con 0x7f154815b230 2024-03-18T20:27:57.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.420+0000 7f1539ffb700 1 -- 172.21.15.1:0/3407300272 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 6860+0+0 (secure 0 0 0) 0x7f153c088530 con 0x7f154815b230 2024-03-18T20:27:57.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.424+0000 7f1539ffb700 1 -- 172.21.15.1:0/3407300272 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f153c053790 con 0x7f154815b230 2024-03-18T20:27:57.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.627+0000 7f154f6c1700 1 -- 172.21.15.1:0/3407300272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1548067dc0 con 0x7f154815b230 2024-03-18T20:27:57.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.628+0000 7f1539ffb700 1 -- 172.21.15.1:0/3407300272 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v276) v1 ==== 74+0+17396 (secure 0 0 0) 0x7f153c056db0 con 0x7f154815b230 2024-03-18T20:27:57.632 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:27:57.632 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":276,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:27:57.145410+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:27:19.891286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"273","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_data","create_time":"2024-03-18T20:27:21.027315+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"274","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2479917750":"2024-03-19T20:27:57.145342+0000","172.21.15.1:0/3443204009":"2024-03-19T20:27:57.145342+0000","172.21.15.1:0/1591377561":"2024-03-19T20:27:57.145342+0000","172.21.15.1:0/1174893696":"2024-03-19T20:27:57.145342+0000","172.21.15.1:0/883947895":"2024-03-19T20:27:57.145342+0000","172.21.15.1:0/3317322019":"2024-03-19T20:27:57.145342+0000","172.21.15.1:0/3677925121":"2024-03-19T20:27:57.145342+0000","172.21.15.1:0/3635558645":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2293377213":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/1796830610":"2024-03-19T20:27:42.143876+0000","172.21.15.1:6833/148356":"2024-03-19T20:27:42.143876+0000","172.21.15.1:6832/148356":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2068313472":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2668486132":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/2560935779":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/82444607":"2024-03-19T20:27:42.143876+0000","172.21.15.1:0/1446581052":"2024-03-19T20:27:42.143876+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:27:57.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.630+0000 7f154f6c1700 1 -- 172.21.15.1:0/3407300272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f154815b230 msgr2=0x7f15480ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:57.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.630+0000 7f154f6c1700 1 --2- 172.21.15.1:0/3407300272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f154815b230 0x7f15480ca540 secure :-1 s=READY pgs=3210 cs=0 l=1 rev1=1 crypto rx=0x7f153c00c730 tx=0x7f153c00ca40 comp rx=0 tx=0).stop 2024-03-18T20:27:57.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.630+0000 7f154f6c1700 1 -- 172.21.15.1:0/3407300272 shutdown_connections 2024-03-18T20:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.630+0000 7f154f6c1700 1 --2- 172.21.15.1:0/3407300272 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f154815d0a0 0x7f154813c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.630+0000 7f154f6c1700 1 --2- 172.21.15.1:0/3407300272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1548140cb0 0x7f1548159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.630+0000 7f154f6c1700 1 --2- 172.21.15.1:0/3407300272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f154815b230 0x7f15480ca540 unknown :-1 s=CLOSED pgs=3210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.630+0000 7f154f6c1700 1 -- 172.21.15.1:0/3407300272 >> 172.21.15.1:0/3407300272 conn(0x7f15480b5780 msgr2=0x7f15480b4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.630+0000 7f154f6c1700 1 -- 172.21.15.1:0/3407300272 shutdown_connections 2024-03-18T20:27:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.630+0000 7f154f6c1700 1 -- 172.21.15.1:0/3407300272 wait complete. 2024-03-18T20:27:57.647 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T20:27:57.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.769+0000 7f2900eff700 1 Processor -- start 2024-03-18T20:27:57.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.769+0000 7f2900eff700 1 -- start start 2024-03-18T20:27:57.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.770+0000 7f2900eff700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f28fc15d6d0 0x7f28fc06ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:57.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.770+0000 7f2900eff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28fc099a30 0x7f28fc0ad800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:57.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.770+0000 7f2900eff700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28fc15dab0 con 0x7f28fc06b3b0 2024-03-18T20:27:57.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.770+0000 7f2900eff700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28fc06fad0 con 0x7f28fc15d6d0 2024-03-18T20:27:57.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.770+0000 7f2900eff700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc0b3780 con 0x7f28fc099a30 2024-03-18T20:27:57.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.770+0000 7f28fa59c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f28fc15d6d0 0x7f28fc06ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:57.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.770+0000 7f28f9d9b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28fc099a30 0x7f28fc0ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:57.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.770+0000 7f28fad9d700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f28fc06b3b0 0x7f28fc0b3d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:45468/0 (socket says 172.21.15.1:45468) 2024-03-18T20:27:57.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.770+0000 7f28fad9d700 1 -- 172.21.15.1:0/27957847 learned_addr learned my addr 172.21.15.1:0/27957847 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:57.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.771+0000 7f28fa59c700 1 -- 172.21.15.1:0/27957847 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28fc099a30 msgr2=0x7f28fc0ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:57.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.771+0000 7f28fa59c700 1 --2- 172.21.15.1:0/27957847 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28fc099a30 0x7f28fc0ad800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:57.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.771+0000 7f28fa59c700 1 -- 172.21.15.1:0/27957847 >> v1:172.21.15.165:6789/0 conn(0x7f28fc06b3b0 legacy=0x7f28fc0b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:57.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.771+0000 7f28fa59c700 1 -- 172.21.15.1:0/27957847 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28fc0b3a60 con 0x7f28fc15d6d0 2024-03-18T20:27:57.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.771+0000 7f28f959a700 1 -- 172.21.15.1:0/27957847 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238444198 0 0) 0x7f28fc15dab0 con 0x7f28fc06b3b0 2024-03-18T20:27:57.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.771+0000 7f28fa59c700 1 --2- 172.21.15.1:0/27957847 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28fc15d6d0 0x7f28fc06ae70 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f28f0009840 tx=0x7f28f0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3c5236dd82153fb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:57.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.771+0000 7f28f959a700 1 -- 172.21.15.1:0/27957847 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f0011830 con 0x7f28fc15d6d0 2024-03-18T20:27:57.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.771+0000 7f28f959a700 1 -- 172.21.15.1:0/27957847 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f0011990 con 0x7f28fc15d6d0 2024-03-18T20:27:57.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.771+0000 7f28f959a700 1 -- 172.21.15.1:0/27957847 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f0011d20 con 0x7f28fc15d6d0 2024-03-18T20:27:57.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.772+0000 7f2900eff700 1 -- 172.21.15.1:0/27957847 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28fc15d6d0 msgr2=0x7f28fc06ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:57.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.772+0000 7f2900eff700 1 --2- 172.21.15.1:0/27957847 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28fc15d6d0 0x7f28fc06ae70 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f28f0009840 tx=0x7f28f0009b50 comp rx=0 tx=0).stop 2024-03-18T20:27:57.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.772+0000 7f2900eff700 1 -- 172.21.15.1:0/27957847 shutdown_connections 2024-03-18T20:27:57.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.772+0000 7f2900eff700 1 --2- 172.21.15.1:0/27957847 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28fc15d6d0 0x7f28fc06ae70 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:57.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.772+0000 7f2900eff700 1 --2- 172.21.15.1:0/27957847 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28fc099a30 0x7f28fc0ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:57.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.772+0000 7f2900eff700 1 -- 172.21.15.1:0/27957847 >> 172.21.15.1:0/27957847 conn(0x7f28fc0b5a10 msgr2=0x7f28fc0b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:57.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.772+0000 7f2900eff700 1 -- 172.21.15.1:0/27957847 shutdown_connections 2024-03-18T20:27:57.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.772+0000 7f2900eff700 1 -- 172.21.15.1:0/27957847 wait complete. 2024-03-18T20:27:57.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.773+0000 7f2900eff700 1 Processor -- start 2024-03-18T20:27:57.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.773+0000 7f2900eff700 1 -- start start 2024-03-18T20:27:57.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.774+0000 7f2900eff700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28fc06b3b0 0x7f28fc1e60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:57.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.774+0000 7f2900eff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28fc099a30 0x7f28fc13e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:57.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.774+0000 7f2900eff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28fc15d6d0 0x7f28fc13e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:57.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.774+0000 7f2900eff700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc0cac10 con 0x7f28fc099a30 2024-03-18T20:27:57.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.774+0000 7f2900eff700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc0caef0 con 0x7f28fc06b3b0 2024-03-18T20:27:57.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.774+0000 7f2900eff700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc0cb1d0 con 0x7f28fc15d6d0 2024-03-18T20:27:57.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.774+0000 7f28f9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28fc099a30 0x7f28fc13e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:57.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.774+0000 7f28f9d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28fc099a30 0x7f28fc13e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58064/0 (socket says 172.21.15.1:58064) 2024-03-18T20:27:57.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.774+0000 7f28fa59c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28fc06b3b0 0x7f28fc1e60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:57.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.774+0000 7f28f9d9b700 1 -- 172.21.15.1:0/1813866979 learned_addr learned my addr 172.21.15.1:0/1813866979 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:57.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.774+0000 7f28fad9d700 1 --2- 172.21.15.1:0/1813866979 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28fc15d6d0 0x7f28fc13e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:57.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.774+0000 7f28f9d9b700 1 -- 172.21.15.1:0/1813866979 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28fc15d6d0 msgr2=0x7f28fc13e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:57.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.774+0000 7f28f9d9b700 1 --2- 172.21.15.1:0/1813866979 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28fc15d6d0 0x7f28fc13e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:57.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.774+0000 7f28f9d9b700 1 -- 172.21.15.1:0/1813866979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28fc06b3b0 msgr2=0x7f28fc1e60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:57.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.774+0000 7f28f9d9b700 1 --2- 172.21.15.1:0/1813866979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28fc06b3b0 0x7f28fc1e60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:57.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.774+0000 7f28f9d9b700 1 -- 172.21.15.1:0/1813866979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f00094a0 con 0x7f28fc099a30 2024-03-18T20:27:57.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.774+0000 7f28fa59c700 1 --2- 172.21.15.1:0/1813866979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28fc06b3b0 0x7f28fc1e60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:27:57.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.775+0000 7f28f9d9b700 1 --2- 172.21.15.1:0/1813866979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28fc099a30 0x7f28fc13e1f0 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7f28ec00e9b0 tx=0x7f28ec00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:57.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.775+0000 7f28eaffd700 1 -- 172.21.15.1:0/1813866979 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ec009710 con 0x7f28fc099a30 2024-03-18T20:27:57.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.775+0000 7f28eaffd700 1 -- 172.21.15.1:0/1813866979 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ec00ee50 con 0x7f28fc099a30 2024-03-18T20:27:57.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.775+0000 7f2900eff700 1 -- 172.21.15.1:0/1813866979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28fc0cb4b0 con 0x7f28fc099a30 2024-03-18T20:27:57.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.775+0000 7f28eaffd700 1 -- 172.21.15.1:0/1813866979 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ec00ca50 con 0x7f28fc099a30 2024-03-18T20:27:57.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.775+0000 7f2900eff700 1 -- 172.21.15.1:0/1813866979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28fc13ec70 con 0x7f28fc099a30 2024-03-18T20:27:57.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.776+0000 7f2900eff700 1 -- 172.21.15.1:0/1813866979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28fc067dc0 con 0x7f28fc099a30 2024-03-18T20:27:57.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.776+0000 7f28eaffd700 1 -- 172.21.15.1:0/1813866979 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 164) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f28ec00cbb0 con 0x7f28fc099a30 2024-03-18T20:27:57.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.777+0000 7f28eaffd700 1 -- 172.21.15.1:0/1813866979 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 6860+0+0 (secure 0 0 0) 0x7f28ec089190 con 0x7f28fc099a30 2024-03-18T20:27:57.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:57.781+0000 7f28eaffd700 1 -- 172.21.15.1:0/1813866979 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f28ec054340 con 0x7f28fc099a30 2024-03-18T20:27:58.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.035+0000 7f2900eff700 1 -- 172.21.15.1:0/1813866979 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f28fc0adb00 con 0x7f28fc099a30 2024-03-18T20:27:58.039 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-18T20:27:58.035+0000 7fbdbeb46700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T20:27:58.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.047+0000 7f28eaffd700 1 -- 172.21.15.1:0/1813866979 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v137) v1 ==== 161+0+0 (secure 0 0 0) 0x7f28ec00ce60 con 0x7f28fc099a30 2024-03-18T20:27:58.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.049+0000 7f2900eff700 1 -- 172.21.15.1:0/1813866979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28fc099a30 msgr2=0x7f28fc13e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:58.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.049+0000 7f2900eff700 1 --2- 172.21.15.1:0/1813866979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28fc099a30 0x7f28fc13e1f0 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7f28ec00e9b0 tx=0x7f28ec00ecc0 comp rx=0 tx=0).stop 2024-03-18T20:27:58.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.049+0000 7f2900eff700 1 -- 172.21.15.1:0/1813866979 shutdown_connections 2024-03-18T20:27:58.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.049+0000 7f2900eff700 1 --2- 172.21.15.1:0/1813866979 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f28fc15d6d0 0x7f28fc13e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:58.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.049+0000 7f2900eff700 1 --2- 172.21.15.1:0/1813866979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28fc06b3b0 0x7f28fc1e60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:58.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.049+0000 7f2900eff700 1 --2- 172.21.15.1:0/1813866979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28fc099a30 0x7f28fc13e1f0 unknown :-1 s=CLOSED pgs=3211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:58.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.049+0000 7f2900eff700 1 -- 172.21.15.1:0/1813866979 >> 172.21.15.1:0/1813866979 conn(0x7f28fc0b5a10 msgr2=0x7f28fc1e4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:58.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.050+0000 7f2900eff700 1 -- 172.21.15.1:0/1813866979 shutdown_connections 2024-03-18T20:27:58.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.050+0000 7f2900eff700 1 -- 172.21.15.1:0/1813866979 wait complete. 2024-03-18T20:27:58.054 INFO:tasks.ceph.mds.c.smithi001.stderr: -3833> 2024-03-18T20:27:27.606+0000 7f5a17ec2700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T20:27:58.055 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T20:27:58.066 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T20:27:58.083 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T20:27:58.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.196+0000 7f0516049700 1 Processor -- start 2024-03-18T20:27:58.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.196+0000 7f0516049700 1 -- start start 2024-03-18T20:27:58.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.197+0000 7f0516049700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f051015b300 0x7f05100b06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:58.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.197+0000 7f0516049700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05100cedc0 0x7f051006ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:58.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.197+0000 7f0516049700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05100cc1f0 con 0x7f051015b6e0 2024-03-18T20:27:58.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.197+0000 7f0516049700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05100cbc70 con 0x7f051015b300 2024-03-18T20:27:58.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.197+0000 7f0516049700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f051009ace0 con 0x7f05100cedc0 2024-03-18T20:27:58.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.197+0000 7f050f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f051015b300 0x7f05100b06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:58.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.197+0000 7f050ffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f051015b6e0 0x7f05100cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39572/0 (socket says 172.21.15.1:39572) 2024-03-18T20:27:58.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.197+0000 7f050ffff700 1 -- 172.21.15.1:0/3601213482 learned_addr learned my addr 172.21.15.1:0/3601213482 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:58.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.198+0000 7f050effd700 1 --2- 172.21.15.1:0/3601213482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05100cedc0 0x7f051006ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:58.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.198+0000 7f050f7fe700 1 -- 172.21.15.1:0/3601213482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05100cedc0 msgr2=0x7f051006ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:27:58.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.198+0000 7f050f7fe700 1 --2- 172.21.15.1:0/3601213482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05100cedc0 0x7f051006ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:58.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.198+0000 7f050f7fe700 1 -- 172.21.15.1:0/3601213482 >> v1:172.21.15.1:6789/0 conn(0x7f051015b6e0 legacy=0x7f05100cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:58.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.198+0000 7f050f7fe700 1 -- 172.21.15.1:0/3601213482 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051009a440 con 0x7f051015b300 2024-03-18T20:27:58.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.198+0000 7f050e7fc700 1 -- 172.21.15.1:0/3601213482 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039111209 0 0) 0x7f05100cc1f0 con 0x7f051015b6e0 2024-03-18T20:27:58.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.198+0000 7f050f7fe700 1 --2- 172.21.15.1:0/3601213482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f051015b300 0x7f05100b06a0 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 crypto rx=0x7f0500009840 tx=0x7f0500009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bd984c0dbcf8102 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:58.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.198+0000 7f050e7fc700 1 -- 172.21.15.1:0/3601213482 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0500011830 con 0x7f051015b300 2024-03-18T20:27:58.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.198+0000 7f050e7fc700 1 -- 172.21.15.1:0/3601213482 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0500011990 con 0x7f051015b300 2024-03-18T20:27:58.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.198+0000 7f050e7fc700 1 -- 172.21.15.1:0/3601213482 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0500011d20 con 0x7f051015b300 2024-03-18T20:27:58.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.199+0000 7f0516049700 1 -- 172.21.15.1:0/3601213482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f051015b300 msgr2=0x7f05100b06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:58.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.199+0000 7f0516049700 1 --2- 172.21.15.1:0/3601213482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f051015b300 0x7f05100b06a0 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 crypto rx=0x7f0500009840 tx=0x7f0500009b50 comp rx=0 tx=0).stop 2024-03-18T20:27:58.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.199+0000 7f0516049700 1 -- 172.21.15.1:0/3601213482 shutdown_connections 2024-03-18T20:27:58.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.199+0000 7f0516049700 1 --2- 172.21.15.1:0/3601213482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f051015b300 0x7f05100b06a0 unknown :-1 s=CLOSED pgs=3216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:58.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.199+0000 7f0516049700 1 --2- 172.21.15.1:0/3601213482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05100cedc0 0x7f051006ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:58.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.199+0000 7f0516049700 1 -- 172.21.15.1:0/3601213482 >> 172.21.15.1:0/3601213482 conn(0x7f05100d1180 msgr2=0x7f05100d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:58.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.199+0000 7f0516049700 1 -- 172.21.15.1:0/3601213482 shutdown_connections 2024-03-18T20:27:58.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.199+0000 7f0516049700 1 -- 172.21.15.1:0/3601213482 wait complete. 2024-03-18T20:27:58.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.200+0000 7f0516049700 1 Processor -- start 2024-03-18T20:27:58.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.200+0000 7f0516049700 1 -- start start 2024-03-18T20:27:58.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.201+0000 7f0516049700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05100cedc0 0x7f05101e5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:58.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.201+0000 7f0516049700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f051015b300 0x7f051014f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:58.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.201+0000 7f0516049700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f051015b6e0 0x7f051014fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:58.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.201+0000 7f0516049700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051015e9d0 con 0x7f051015b6e0 2024-03-18T20:27:58.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.201+0000 7f0516049700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051015e130 con 0x7f051015b300 2024-03-18T20:27:58.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.201+0000 7f0516049700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f051015e410 con 0x7f05100cedc0 2024-03-18T20:27:58.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.201+0000 7f050ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f051015b6e0 0x7f051014fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:58.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.201+0000 7f050f7fe700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05100cedc0 0x7f05101e5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:58.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.201+0000 7f050ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f051015b6e0 0x7f051014fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58082/0 (socket says 172.21.15.1:58082) 2024-03-18T20:27:58.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.201+0000 7f050ffff700 1 -- 172.21.15.1:0/2775501560 learned_addr learned my addr 172.21.15.1:0/2775501560 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:58.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.201+0000 7f050effd700 1 --2- 172.21.15.1:0/2775501560 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f051015b300 0x7f051014f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:58.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.202+0000 7f050ffff700 1 -- 172.21.15.1:0/2775501560 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05100cedc0 msgr2=0x7f05101e5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:58.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.202+0000 7f050ffff700 1 --2- 172.21.15.1:0/2775501560 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05100cedc0 0x7f05101e5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:58.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.202+0000 7f050ffff700 1 -- 172.21.15.1:0/2775501560 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f051015b300 msgr2=0x7f051014f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:58.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.202+0000 7f050ffff700 1 --2- 172.21.15.1:0/2775501560 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f051015b300 0x7f051014f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:58.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.202+0000 7f050ffff700 1 -- 172.21.15.1:0/2775501560 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05000094a0 con 0x7f051015b6e0 2024-03-18T20:27:58.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.202+0000 7f050ffff700 1 --2- 172.21.15.1:0/2775501560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f051015b6e0 0x7f051014fb80 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 crypto rx=0x7f050800cca0 tx=0x7f050800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:58.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.202+0000 7f0507fff700 1 -- 172.21.15.1:0/2775501560 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050800ab70 con 0x7f051015b6e0 2024-03-18T20:27:58.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.202+0000 7f0507fff700 1 -- 172.21.15.1:0/2775501560 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050800acd0 con 0x7f051015b6e0 2024-03-18T20:27:58.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.202+0000 7f0516049700 1 -- 172.21.15.1:0/2775501560 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f051015e6f0 con 0x7f051015b6e0 2024-03-18T20:27:58.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.202+0000 7f0507fff700 1 -- 172.21.15.1:0/2775501560 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0508007310 con 0x7f051015b6e0 2024-03-18T20:27:58.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.202+0000 7f0516049700 1 -- 172.21.15.1:0/2775501560 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05101e6090 con 0x7f051015b6e0 2024-03-18T20:27:58.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.203+0000 7f0516049700 1 -- 172.21.15.1:0/2775501560 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05101e6280 con 0x7f051015b6e0 2024-03-18T20:27:58.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.203+0000 7f0507fff700 1 -- 172.21.15.1:0/2775501560 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 165) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f0508007470 con 0x7f051015b6e0 2024-03-18T20:27:58.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.204+0000 7f0507fff700 1 --2- 172.21.15.1:0/2775501560 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f04f006aae0 0x7f04f006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:58.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.204+0000 7f050f7fe700 1 --2- 172.21.15.1:0/2775501560 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f04f006aae0 0x7f04f006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:58.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.204+0000 7f0507fff700 1 -- 172.21.15.1:0/2775501560 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(277..277 src has 1..277) v4 ==== 6946+0+0 (secure 0 0 0) 0x7f0508088f80 con 0x7f051015b6e0 2024-03-18T20:27:58.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.204+0000 7f050f7fe700 1 --2- 172.21.15.1:0/2775501560 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f04f006aae0 0x7f04f006cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0500000c00 tx=0x7f050001c040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:58.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.209+0000 7f0507fff700 1 -- 172.21.15.1:0/2775501560 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f05080bc580 con 0x7f051015b6e0 2024-03-18T20:27:58.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:58.459+0000 7f0516049700 1 -- 172.21.15.1:0/2775501560 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f05100c8530 con 0x7f051015b6e0 2024-03-18T20:27:59.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.050+0000 7f0507fff700 1 -- 172.21.15.1:0/2775501560 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v138) v1 ==== 104+0+0 (secure 0 0 0) 0x7f05100c8530 con 0x7f051015b6e0 2024-03-18T20:27:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.053+0000 7f0516049700 1 -- 172.21.15.1:0/2775501560 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f04f006aae0 msgr2=0x7f04f006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.053+0000 7f0516049700 1 --2- 172.21.15.1:0/2775501560 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f04f006aae0 0x7f04f006cfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0500000c00 tx=0x7f050001c040 comp rx=0 tx=0).stop 2024-03-18T20:27:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.053+0000 7f0516049700 1 -- 172.21.15.1:0/2775501560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f051015b6e0 msgr2=0x7f051014fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.053+0000 7f0516049700 1 --2- 172.21.15.1:0/2775501560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f051015b6e0 0x7f051014fb80 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 crypto rx=0x7f050800cca0 tx=0x7f050800a060 comp rx=0 tx=0).stop 2024-03-18T20:27:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.053+0000 7f0516049700 1 -- 172.21.15.1:0/2775501560 shutdown_connections 2024-03-18T20:27:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.053+0000 7f0516049700 1 --2- 172.21.15.1:0/2775501560 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f04f006aae0 0x7f04f006cfa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.053+0000 7f0516049700 1 --2- 172.21.15.1:0/2775501560 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05100cedc0 0x7f05101e5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.053+0000 7f0516049700 1 --2- 172.21.15.1:0/2775501560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f051015b6e0 0x7f051014fb80 unknown :-1 s=CLOSED pgs=3217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.053+0000 7f0516049700 1 --2- 172.21.15.1:0/2775501560 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f051015b300 0x7f051014f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.053+0000 7f0516049700 1 -- 172.21.15.1:0/2775501560 >> 172.21.15.1:0/2775501560 conn(0x7f05100d1180 msgr2=0x7f051006b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.054+0000 7f0516049700 1 -- 172.21.15.1:0/2775501560 shutdown_connections 2024-03-18T20:27:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.054+0000 7f0516049700 1 -- 172.21.15.1:0/2775501560 wait complete. 2024-03-18T20:27:59.073 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T20:27:59.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.260+0000 7fdd19b54700 1 Processor -- start 2024-03-18T20:27:59.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.260+0000 7fdd19b54700 1 -- start start 2024-03-18T20:27:59.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.261+0000 7fdd19b54700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd1409aa40 0x7fdd14099a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:59.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.261+0000 7fdd19b54700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1409a3c0 con 0x7fdd14099f70 2024-03-18T20:27:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.261+0000 7fdd19b54700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd140b56b0 con 0x7fdd1415d4d0 2024-03-18T20:27:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.261+0000 7fdd19b54700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd140b5990 con 0x7fdd1409aa40 2024-03-18T20:27:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.261+0000 7fdd13fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdd14099f70 0x7fdd1415d8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39586/0 (socket says 172.21.15.1:39586) 2024-03-18T20:27:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.261+0000 7fdd13fff700 1 -- 172.21.15.1:0/1807815296 learned_addr learned my addr 172.21.15.1:0/1807815296 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:27:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.261+0000 7fdd12ffd700 1 --2- 172.21.15.1:0/1807815296 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd1409aa40 0x7fdd14099a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.262+0000 7fdd127fc700 1 -- 172.21.15.1:0/1807815296 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240322949 0 0) 0x7fdd0c001490 con 0x7fdd14099f70 2024-03-18T20:27:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.262+0000 7fdd127fc700 1 -- 172.21.15.1:0/1807815296 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd000035b0 con 0x7fdd14099f70 2024-03-18T20:27:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.262+0000 7fdd127fc700 1 -- 172.21.15.1:0/1807815296 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572135784 0 0) 0x7fdd140b56b0 con 0x7fdd1415d4d0 2024-03-18T20:27:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.262+0000 7fdd127fc700 1 -- 172.21.15.1:0/1807815296 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd0c001490 con 0x7fdd1415d4d0 2024-03-18T20:27:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.262+0000 7fdd12ffd700 1 -- 172.21.15.1:0/1807815296 >> v1:172.21.15.165:6789/0 conn(0x7fdd1415d4d0 legacy=0x7fdd140b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.262+0000 7fdd12ffd700 1 -- 172.21.15.1:0/1807815296 >> v1:172.21.15.1:6789/0 conn(0x7fdd14099f70 legacy=0x7fdd1415d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:59.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.262+0000 7fdd12ffd700 1 -- 172.21.15.1:0/1807815296 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd140b5c70 con 0x7fdd1409aa40 2024-03-18T20:27:59.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.262+0000 7fdd12ffd700 1 --2- 172.21.15.1:0/1807815296 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd1409aa40 0x7fdd14099a30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc009870 tx=0x7fdcfc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8cd5ed3c2e75428 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:59.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.262+0000 7fdd127fc700 1 -- 172.21.15.1:0/1807815296 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc011830 con 0x7fdd1409aa40 2024-03-18T20:27:59.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.262+0000 7fdd127fc700 1 -- 172.21.15.1:0/1807815296 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcfc011990 con 0x7fdd1409aa40 2024-03-18T20:27:59.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.262+0000 7fdd127fc700 1 -- 172.21.15.1:0/1807815296 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc011d20 con 0x7fdd1409aa40 2024-03-18T20:27:59.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.263+0000 7fdd19b54700 1 -- 172.21.15.1:0/1807815296 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd1409aa40 msgr2=0x7fdd14099a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:59.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.263+0000 7fdd19b54700 1 --2- 172.21.15.1:0/1807815296 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd1409aa40 0x7fdd14099a30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc009870 tx=0x7fdcfc009b80 comp rx=0 tx=0).stop 2024-03-18T20:27:59.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.263+0000 7fdd19b54700 1 -- 172.21.15.1:0/1807815296 shutdown_connections 2024-03-18T20:27:59.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.263+0000 7fdd19b54700 1 --2- 172.21.15.1:0/1807815296 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd1409aa40 0x7fdd14099a30 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:59.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.263+0000 7fdd19b54700 1 -- 172.21.15.1:0/1807815296 >> 172.21.15.1:0/1807815296 conn(0x7fdd140b6890 msgr2=0x7fdd140b2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:27:59.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.263+0000 7fdd19b54700 1 -- 172.21.15.1:0/1807815296 shutdown_connections 2024-03-18T20:27:59.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.263+0000 7fdd19b54700 1 -- 172.21.15.1:0/1807815296 wait complete. 2024-03-18T20:27:59.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.264+0000 7fdd19b54700 1 Processor -- start 2024-03-18T20:27:59.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.264+0000 7fdd19b54700 1 -- start start 2024-03-18T20:27:59.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.265+0000 7fdd19b54700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd14099f70 0x7fdd141604b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:59.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.265+0000 7fdd19b54700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd1409aa40 0x7fdd1414ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:59.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.265+0000 7fdd19b54700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd1415d4d0 0x7fdd141569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:59.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.265+0000 7fdd19b54700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd140cfe20 con 0x7fdd1415d4d0 2024-03-18T20:27:59.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.265+0000 7fdd19b54700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd140d0100 con 0x7fdd1409aa40 2024-03-18T20:27:59.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.265+0000 7fdd19b54700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd140d03e0 con 0x7fdd14099f70 2024-03-18T20:27:59.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.265+0000 7fdd13fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd1415d4d0 0x7fdd141569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:59.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.265+0000 7fdd13fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd1415d4d0 0x7fdd141569a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58102/0 (socket says 172.21.15.1:58102) 2024-03-18T20:27:59.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.265+0000 7fdd13fff700 1 -- 172.21.15.1:0/2010846572 learned_addr learned my addr 172.21.15.1:0/2010846572 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:27:59.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.265+0000 7fdd12ffd700 1 --2- 172.21.15.1:0/2010846572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd1409aa40 0x7fdd1414ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:59.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.265+0000 7fdd137fe700 1 --2- 172.21.15.1:0/2010846572 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd14099f70 0x7fdd141604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:59.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.265+0000 7fdd13fff700 1 -- 172.21.15.1:0/2010846572 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd14099f70 msgr2=0x7fdd141604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:59.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.265+0000 7fdd13fff700 1 --2- 172.21.15.1:0/2010846572 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd14099f70 0x7fdd141604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:59.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.265+0000 7fdd13fff700 1 -- 172.21.15.1:0/2010846572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd1409aa40 msgr2=0x7fdd1414ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:27:59.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.265+0000 7fdd13fff700 1 --2- 172.21.15.1:0/2010846572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd1409aa40 0x7fdd1414ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:27:59.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.265+0000 7fdd13fff700 1 -- 172.21.15.1:0/2010846572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcfc0094a0 con 0x7fdd1415d4d0 2024-03-18T20:27:59.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.265+0000 7fdd12ffd700 1 --2- 172.21.15.1:0/2010846572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd1409aa40 0x7fdd1414ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:27:59.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.265+0000 7fdd13fff700 1 --2- 172.21.15.1:0/2010846572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd1415d4d0 0x7fdd141569a0 secure :-1 s=READY pgs=3221 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00cc30 tx=0x7fdd0c00cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:59.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.266+0000 7fdd0bfff700 1 -- 172.21.15.1:0/2010846572 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c00ab00 con 0x7fdd1415d4d0 2024-03-18T20:27:59.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.266+0000 7fdd0bfff700 1 -- 172.21.15.1:0/2010846572 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0c00ac60 con 0x7fdd1415d4d0 2024-03-18T20:27:59.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.266+0000 7fdd19b54700 1 -- 172.21.15.1:0/2010846572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd140b5480 con 0x7fdd1415d4d0 2024-03-18T20:27:59.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.266+0000 7fdd0bfff700 1 -- 172.21.15.1:0/2010846572 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c012400 con 0x7fdd1415d4d0 2024-03-18T20:27:59.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.266+0000 7fdd19b54700 1 -- 172.21.15.1:0/2010846572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd140cfad0 con 0x7fdd1415d4d0 2024-03-18T20:27:59.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.267+0000 7fdd19b54700 1 -- 172.21.15.1:0/2010846572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd140d05a0 con 0x7fdd1415d4d0 2024-03-18T20:27:59.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.267+0000 7fdd0bfff700 1 -- 172.21.15.1:0/2010846572 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 165) v1 ==== 88975+0+0 (secure 0 0 0) 0x7fdd0c01b3b0 con 0x7fdd1415d4d0 2024-03-18T20:27:59.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.268+0000 7fdd0bfff700 1 --2- 172.21.15.1:0/2010846572 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdcf406aae0 0x7fdcf406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:27:59.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.268+0000 7fdd137fe700 1 --2- 172.21.15.1:0/2010846572 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdcf406aae0 0x7fdcf406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:27:59.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.268+0000 7fdd0bfff700 1 -- 172.21.15.1:0/2010846572 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(277..277 src has 1..277) v4 ==== 6946+0+0 (secure 0 0 0) 0x7fdd0c0890d0 con 0x7fdd1415d4d0 2024-03-18T20:27:59.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.268+0000 7fdd137fe700 1 --2- 172.21.15.1:0/2010846572 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdcf406aae0 0x7fdcf406cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdd040011e0 tx=0x7fdd04009040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:27:59.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.273+0000 7fdd0bfff700 1 -- 172.21.15.1:0/2010846572 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdd0c054040 con 0x7fdd1415d4d0 2024-03-18T20:27:59.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:27:59.473+0000 7fdd19b54700 1 -- 172.21.15.1:0/2010846572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdd140cff60 con 0x7fdd1415d4d0 2024-03-18T20:28:00.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.166+0000 7fdd0bfff700 1 -- 172.21.15.1:0/2010846572 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 166) v1 ==== 89183+0+0 (secure 0 0 0) 0x7fdd0c054410 con 0x7fdd1415d4d0 2024-03-18T20:28:00.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.206+0000 7fdd0bfff700 1 -- 172.21.15.1:0/2010846572 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v278) v1 ==== 181+0+0 (secure 0 0 0) 0x7fdd0c072d90 con 0x7fdd1415d4d0 2024-03-18T20:28:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.209+0000 7fdd19b54700 1 -- 172.21.15.1:0/2010846572 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdcf406aae0 msgr2=0x7fdcf406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.209+0000 7fdd19b54700 1 --2- 172.21.15.1:0/2010846572 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdcf406aae0 0x7fdcf406cfa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdd040011e0 tx=0x7fdd04009040 comp rx=0 tx=0).stop 2024-03-18T20:28:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.209+0000 7fdd19b54700 1 -- 172.21.15.1:0/2010846572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd1415d4d0 msgr2=0x7fdd141569a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.209+0000 7fdd19b54700 1 --2- 172.21.15.1:0/2010846572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd1415d4d0 0x7fdd141569a0 secure :-1 s=READY pgs=3221 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00cc30 tx=0x7fdd0c00cf40 comp rx=0 tx=0).stop 2024-03-18T20:28:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.210+0000 7fdd19b54700 1 -- 172.21.15.1:0/2010846572 shutdown_connections 2024-03-18T20:28:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.210+0000 7fdd19b54700 1 --2- 172.21.15.1:0/2010846572 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdcf406aae0 0x7fdcf406cfa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.210+0000 7fdd19b54700 1 --2- 172.21.15.1:0/2010846572 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdd14099f70 0x7fdd141604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.210+0000 7fdd19b54700 1 --2- 172.21.15.1:0/2010846572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd1415d4d0 0x7fdd141569a0 unknown :-1 s=CLOSED pgs=3221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.210+0000 7fdd19b54700 1 --2- 172.21.15.1:0/2010846572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd1409aa40 0x7fdd1414ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:00.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.210+0000 7fdd19b54700 1 -- 172.21.15.1:0/2010846572 >> 172.21.15.1:0/2010846572 conn(0x7fdd140b6890 msgr2=0x7fdd140cc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:00.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.210+0000 7fdd19b54700 1 -- 172.21.15.1:0/2010846572 shutdown_connections 2024-03-18T20:28:00.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.210+0000 7fdd19b54700 1 -- 172.21.15.1:0/2010846572 wait complete. 2024-03-18T20:28:00.215 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-18T20:28:00.228 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T20:28:00.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.355+0000 7ffa27d81700 1 Processor -- start 2024-03-18T20:28:00.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.355+0000 7ffa27d81700 1 -- start start 2024-03-18T20:28:00.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.356+0000 7ffa27d81700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffa200b3240 0x7ffa200b4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:00.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.356+0000 7ffa27d81700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ffa200b4f80 0x7ffa2006dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:00.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.356+0000 7ffa27d81700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2015b5c0 con 0x7ffa2015b1e0 2024-03-18T20:28:00.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.356+0000 7ffa27d81700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa2006fae0 con 0x7ffa200b3240 2024-03-18T20:28:00.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.356+0000 7ffa27d81700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa2015e900 con 0x7ffa200b4f80 2024-03-18T20:28:00.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.356+0000 7ffa2531c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffa200b3240 0x7ffa200b4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:00.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.356+0000 7ffa25b1d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ffa2015b1e0 0x7ffa2006f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39600/0 (socket says 172.21.15.1:39600) 2024-03-18T20:28:00.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.356+0000 7ffa25b1d700 1 -- 172.21.15.1:0/3837531350 learned_addr learned my addr 172.21.15.1:0/3837531350 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:00.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.356+0000 7ffa2631e700 1 --2- 172.21.15.1:0/3837531350 >> v2:172.21.15.165:3300/0 conn(0x7ffa200b4f80 0x7ffa2006dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:00.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.356+0000 7ffa2531c700 1 -- 172.21.15.1:0/3837531350 >> v2:172.21.15.165:3300/0 conn(0x7ffa200b4f80 msgr2=0x7ffa2006dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:00.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.356+0000 7ffa2531c700 1 --2- 172.21.15.1:0/3837531350 >> v2:172.21.15.165:3300/0 conn(0x7ffa200b4f80 0x7ffa2006dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:00.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.356+0000 7ffa2531c700 1 -- 172.21.15.1:0/3837531350 >> v1:172.21.15.1:6789/0 conn(0x7ffa2015b1e0 legacy=0x7ffa2006f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:00.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.356+0000 7ffa2531c700 1 -- 172.21.15.1:0/3837531350 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2015dcf0 con 0x7ffa200b3240 2024-03-18T20:28:00.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.356+0000 7ffa24b1b700 1 -- 172.21.15.1:0/3837531350 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3509771504 0 0) 0x7ffa2015b5c0 con 0x7ffa2015b1e0 2024-03-18T20:28:00.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.357+0000 7ffa2531c700 1 --2- 172.21.15.1:0/3837531350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa200b3240 0x7ffa200b4a40 secure :-1 s=READY pgs=3223 cs=0 l=1 rev1=1 crypto rx=0x7ffa2009a060 tx=0x7ffa1c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52ff179e06d40967 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:00.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.357+0000 7ffa24b1b700 1 -- 172.21.15.1:0/3837531350 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c011830 con 0x7ffa200b3240 2024-03-18T20:28:00.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.357+0000 7ffa24b1b700 1 -- 172.21.15.1:0/3837531350 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1c011990 con 0x7ffa200b3240 2024-03-18T20:28:00.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.357+0000 7ffa24b1b700 1 -- 172.21.15.1:0/3837531350 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c011d20 con 0x7ffa200b3240 2024-03-18T20:28:00.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.357+0000 7ffa27d81700 1 -- 172.21.15.1:0/3837531350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa200b3240 msgr2=0x7ffa200b4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:00.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.357+0000 7ffa27d81700 1 --2- 172.21.15.1:0/3837531350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa200b3240 0x7ffa200b4a40 secure :-1 s=READY pgs=3223 cs=0 l=1 rev1=1 crypto rx=0x7ffa2009a060 tx=0x7ffa1c009b20 comp rx=0 tx=0).stop 2024-03-18T20:28:00.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.357+0000 7ffa27d81700 1 -- 172.21.15.1:0/3837531350 shutdown_connections 2024-03-18T20:28:00.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.357+0000 7ffa27d81700 1 --2- 172.21.15.1:0/3837531350 >> v2:172.21.15.165:3300/0 conn(0x7ffa200b4f80 0x7ffa2006dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:00.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.357+0000 7ffa27d81700 1 --2- 172.21.15.1:0/3837531350 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa200b3240 0x7ffa200b4a40 unknown :-1 s=CLOSED pgs=3223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:00.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.357+0000 7ffa27d81700 1 -- 172.21.15.1:0/3837531350 >> 172.21.15.1:0/3837531350 conn(0x7ffa20099a30 msgr2=0x7ffa200b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:00.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.357+0000 7ffa27d81700 1 -- 172.21.15.1:0/3837531350 shutdown_connections 2024-03-18T20:28:00.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.357+0000 7ffa27d81700 1 -- 172.21.15.1:0/3837531350 wait complete. 2024-03-18T20:28:00.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.359+0000 7ffa27d81700 1 Processor -- start 2024-03-18T20:28:00.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.359+0000 7ffa27d81700 1 -- start start 2024-03-18T20:28:00.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.359+0000 7ffa27d81700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa200b3240 0x7ffa20144d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:00.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.359+0000 7ffa27d81700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa200b4f80 0x7ffa2015a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:00.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.359+0000 7ffa27d81700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa2015b1e0 0x7ffa20134010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:00.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.359+0000 7ffa27d81700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2015ad40 con 0x7ffa200b4f80 2024-03-18T20:28:00.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.359+0000 7ffa27d81700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa20145340 con 0x7ffa2015b1e0 2024-03-18T20:28:00.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.359+0000 7ffa27d81700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa20145620 con 0x7ffa200b3240 2024-03-18T20:28:00.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.359+0000 7ffa2531c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa200b4f80 0x7ffa2015a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:00.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.359+0000 7ffa2531c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa200b4f80 0x7ffa2015a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58110/0 (socket says 172.21.15.1:58110) 2024-03-18T20:28:00.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.359+0000 7ffa2531c700 1 -- 172.21.15.1:0/1276115547 learned_addr learned my addr 172.21.15.1:0/1276115547 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:00.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.360+0000 7ffa2631e700 1 --2- 172.21.15.1:0/1276115547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa2015b1e0 0x7ffa20134010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:00.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.360+0000 7ffa25b1d700 1 --2- 172.21.15.1:0/1276115547 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa200b3240 0x7ffa20144d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:00.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.360+0000 7ffa2531c700 1 -- 172.21.15.1:0/1276115547 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa200b3240 msgr2=0x7ffa20144d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:00.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.360+0000 7ffa2531c700 1 --2- 172.21.15.1:0/1276115547 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa200b3240 0x7ffa20144d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:00.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.360+0000 7ffa2531c700 1 -- 172.21.15.1:0/1276115547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa2015b1e0 msgr2=0x7ffa20134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:00.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.360+0000 7ffa2531c700 1 --2- 172.21.15.1:0/1276115547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa2015b1e0 0x7ffa20134010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:00.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.360+0000 7ffa2531c700 1 -- 172.21.15.1:0/1276115547 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1c0094a0 con 0x7ffa200b4f80 2024-03-18T20:28:00.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.360+0000 7ffa2631e700 1 --2- 172.21.15.1:0/1276115547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa2015b1e0 0x7ffa20134010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:28:00.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.360+0000 7ffa25b1d700 1 --2- 172.21.15.1:0/1276115547 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa200b3240 0x7ffa20144d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:28:00.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.360+0000 7ffa2531c700 1 --2- 172.21.15.1:0/1276115547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa200b4f80 0x7ffa2015a800 secure :-1 s=READY pgs=3224 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c014040 tx=0x7ffa1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:00.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.360+0000 7ffa167fc700 1 -- 172.21.15.1:0/1276115547 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c0038a0 con 0x7ffa200b4f80 2024-03-18T20:28:00.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.360+0000 7ffa167fc700 1 -- 172.21.15.1:0/1276115547 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1c01e070 con 0x7ffa200b4f80 2024-03-18T20:28:00.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.360+0000 7ffa27d81700 1 -- 172.21.15.1:0/1276115547 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa20145900 con 0x7ffa200b4f80 2024-03-18T20:28:00.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.360+0000 7ffa167fc700 1 -- 172.21.15.1:0/1276115547 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c003c60 con 0x7ffa200b4f80 2024-03-18T20:28:00.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.360+0000 7ffa27d81700 1 -- 172.21.15.1:0/1276115547 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa20145ec0 con 0x7ffa200b4f80 2024-03-18T20:28:00.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.361+0000 7ffa27d81700 1 -- 172.21.15.1:0/1276115547 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa20067dc0 con 0x7ffa200b4f80 2024-03-18T20:28:00.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.362+0000 7ffa167fc700 1 -- 172.21.15.1:0/1276115547 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 166) v1 ==== 89183+0+0 (secure 0 0 0) 0x7ffa1c02d450 con 0x7ffa200b4f80 2024-03-18T20:28:00.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.362+0000 7ffa167fc700 1 --2- 172.21.15.1:0/1276115547 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffa0406ad20 0x7ffa0406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:00.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.362+0000 7ffa167fc700 1 -- 172.21.15.1:0/1276115547 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(278..278 src has 1..278) v4 ==== 6508+0+0 (secure 0 0 0) 0x7ffa1c08a740 con 0x7ffa200b4f80 2024-03-18T20:28:00.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.362+0000 7ffa25b1d700 1 --2- 172.21.15.1:0/1276115547 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffa0406ad20 0x7ffa0406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:00.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.362+0000 7ffa25b1d700 1 --2- 172.21.15.1:0/1276115547 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffa0406ad20 0x7ffa0406d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ffa10000f60 tx=0x7ffa10009040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:00.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.367+0000 7ffa167fc700 1 -- 172.21.15.1:0/1276115547 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffa1c0556a0 con 0x7ffa200b4f80 2024-03-18T20:28:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:00.569+0000 7ffa27d81700 1 -- 172.21.15.1:0/1276115547 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ffa200afab0 con 0x7ffa200b4f80 2024-03-18T20:28:01.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.220+0000 7ffa167fc700 1 -- 172.21.15.1:0/1276115547 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ffa1c055a70 con 0x7ffa200b4f80 2024-03-18T20:28:01.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.223+0000 7ffa167fc700 1 -- 172.21.15.1:0/1276115547 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v279) v1 ==== 169+0+0 (secure 0 0 0) 0x7ffa1c059130 con 0x7ffa200b4f80 2024-03-18T20:28:01.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.227+0000 7ffa27d81700 1 -- 172.21.15.1:0/1276115547 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffa0406ad20 msgr2=0x7ffa0406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:01.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.227+0000 7ffa27d81700 1 --2- 172.21.15.1:0/1276115547 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffa0406ad20 0x7ffa0406d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ffa10000f60 tx=0x7ffa10009040 comp rx=0 tx=0).stop 2024-03-18T20:28:01.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.227+0000 7ffa27d81700 1 -- 172.21.15.1:0/1276115547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa200b4f80 msgr2=0x7ffa2015a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:01.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.227+0000 7ffa27d81700 1 --2- 172.21.15.1:0/1276115547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa200b4f80 0x7ffa2015a800 secure :-1 s=READY pgs=3224 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c014040 tx=0x7ffa1c004750 comp rx=0 tx=0).stop 2024-03-18T20:28:01.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.227+0000 7ffa27d81700 1 -- 172.21.15.1:0/1276115547 shutdown_connections 2024-03-18T20:28:01.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.227+0000 7ffa27d81700 1 --2- 172.21.15.1:0/1276115547 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffa0406ad20 0x7ffa0406d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.227+0000 7ffa27d81700 1 --2- 172.21.15.1:0/1276115547 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffa200b3240 0x7ffa20144d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.227+0000 7ffa27d81700 1 --2- 172.21.15.1:0/1276115547 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa2015b1e0 0x7ffa20134010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.227+0000 7ffa27d81700 1 --2- 172.21.15.1:0/1276115547 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa200b4f80 0x7ffa2015a800 unknown :-1 s=CLOSED pgs=3224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.227+0000 7ffa27d81700 1 -- 172.21.15.1:0/1276115547 >> 172.21.15.1:0/1276115547 conn(0x7ffa20099a30 msgr2=0x7ffa200aeac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:01.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.228+0000 7ffa27d81700 1 -- 172.21.15.1:0/1276115547 shutdown_connections 2024-03-18T20:28:01.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.228+0000 7ffa27d81700 1 -- 172.21.15.1:0/1276115547 wait complete. 2024-03-18T20:28:01.233 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-18T20:28:01.250 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-18T20:28:01.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.385+0000 7fc536571700 1 Processor -- start 2024-03-18T20:28:01.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.385+0000 7fc536571700 1 -- start start 2024-03-18T20:28:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.385+0000 7fc536571700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5300cf200 0x7fc53015c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc536571700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc53015b6c0 0x7fc5300cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc536571700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5300cbc90 0x7fc53015eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc536571700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc53015ca40 con 0x7fc5300cbc90 2024-03-18T20:28:01.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc536571700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5300cce30 con 0x7fc53015b6c0 2024-03-18T20:28:01.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc536571700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5300cfb20 con 0x7fc5300cf200 2024-03-18T20:28:01.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc534b0e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5300cbc90 0x7fc53015eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:01.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc534b0e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc5300cbc90 0x7fc53015eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58120/0 (socket says 172.21.15.1:58120) 2024-03-18T20:28:01.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc534b0e700 1 -- 172.21.15.1:0/4271272748 learned_addr learned my addr 172.21.15.1:0/4271272748 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:01.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc52ffff700 1 --2- 172.21.15.1:0/4271272748 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5300cf200 0x7fc53015c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:01.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc52f7fe700 1 --2- 172.21.15.1:0/4271272748 >> v2:172.21.15.165:3300/0 conn(0x7fc53015b6c0 0x7fc5300cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:01.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc534b0e700 1 -- 172.21.15.1:0/4271272748 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5300cf200 msgr2=0x7fc53015c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:01.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc534b0e700 1 --2- 172.21.15.1:0/4271272748 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5300cf200 0x7fc53015c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc534b0e700 1 -- 172.21.15.1:0/4271272748 >> v2:172.21.15.165:3300/0 conn(0x7fc53015b6c0 msgr2=0x7fc5300cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:01.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc534b0e700 1 --2- 172.21.15.1:0/4271272748 >> v2:172.21.15.165:3300/0 conn(0x7fc53015b6c0 0x7fc5300cf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc534b0e700 1 -- 172.21.15.1:0/4271272748 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5300cc220 con 0x7fc5300cbc90 2024-03-18T20:28:01.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc52ffff700 1 --2- 172.21.15.1:0/4271272748 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5300cf200 0x7fc53015c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:28:01.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc52f7fe700 1 --2- 172.21.15.1:0/4271272748 >> v2:172.21.15.165:3300/0 conn(0x7fc53015b6c0 0x7fc5300cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:28:01.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc534b0e700 1 --2- 172.21.15.1:0/4271272748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5300cbc90 0x7fc53015eff0 secure :-1 s=READY pgs=3225 cs=0 l=1 rev1=1 crypto rx=0x7fc5300b3e10 tx=0x7fc52800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f292a7e82a540c04 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:01.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc52effd700 1 -- 172.21.15.1:0/4271272748 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52800e070 con 0x7fc5300cbc90 2024-03-18T20:28:01.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc52effd700 1 -- 172.21.15.1:0/4271272748 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52800b9c0 con 0x7fc5300cbc90 2024-03-18T20:28:01.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.386+0000 7fc52effd700 1 -- 172.21.15.1:0/4271272748 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52800bd70 con 0x7fc5300cbc90 2024-03-18T20:28:01.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.387+0000 7fc536571700 1 -- 172.21.15.1:0/4271272748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5300cbc90 msgr2=0x7fc53015eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:01.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.387+0000 7fc536571700 1 --2- 172.21.15.1:0/4271272748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5300cbc90 0x7fc53015eff0 secure :-1 s=READY pgs=3225 cs=0 l=1 rev1=1 crypto rx=0x7fc5300b3e10 tx=0x7fc52800b330 comp rx=0 tx=0).stop 2024-03-18T20:28:01.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.387+0000 7fc536571700 1 -- 172.21.15.1:0/4271272748 shutdown_connections 2024-03-18T20:28:01.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.387+0000 7fc536571700 1 --2- 172.21.15.1:0/4271272748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5300cbc90 0x7fc53015eff0 unknown :-1 s=CLOSED pgs=3225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.387+0000 7fc536571700 1 --2- 172.21.15.1:0/4271272748 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc5300cf200 0x7fc53015c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.387+0000 7fc536571700 1 --2- 172.21.15.1:0/4271272748 >> v2:172.21.15.165:3300/0 conn(0x7fc53015b6c0 0x7fc5300cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.387+0000 7fc536571700 1 -- 172.21.15.1:0/4271272748 >> 172.21.15.1:0/4271272748 conn(0x7fc5300d1180 msgr2=0x7fc5300d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:01.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.387+0000 7fc536571700 1 -- 172.21.15.1:0/4271272748 shutdown_connections 2024-03-18T20:28:01.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.387+0000 7fc536571700 1 -- 172.21.15.1:0/4271272748 wait complete. 2024-03-18T20:28:01.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.388+0000 7fc536571700 1 Processor -- start 2024-03-18T20:28:01.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.388+0000 7fc536571700 1 -- start start 2024-03-18T20:28:01.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.389+0000 7fc536571700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5300cbc90 0x7fc530150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:01.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.389+0000 7fc536571700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5300cf200 0x7fc5301e2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:01.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.389+0000 7fc536571700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc53015b6c0 0x7fc5300b4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:01.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.389+0000 7fc536571700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5301e3960 con 0x7fc5300cf200 2024-03-18T20:28:01.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.389+0000 7fc536571700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5301e2d50 con 0x7fc5300cbc90 2024-03-18T20:28:01.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.389+0000 7fc536571700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5301e3030 con 0x7fc53015b6c0 2024-03-18T20:28:01.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.389+0000 7fc52f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5300cf200 0x7fc5301e2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.389+0000 7fc52f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5300cf200 0x7fc5301e2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58130/0 (socket says 172.21.15.1:58130) 2024-03-18T20:28:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.389+0000 7fc52f7fe700 1 -- 172.21.15.1:0/1435783954 learned_addr learned my addr 172.21.15.1:0/1435783954 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.389+0000 7fc52ffff700 1 --2- 172.21.15.1:0/1435783954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5300cbc90 0x7fc530150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.389+0000 7fc534b0e700 1 --2- 172.21.15.1:0/1435783954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc53015b6c0 0x7fc5300b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.389+0000 7fc52f7fe700 1 -- 172.21.15.1:0/1435783954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc53015b6c0 msgr2=0x7fc5300b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.389+0000 7fc52f7fe700 1 --2- 172.21.15.1:0/1435783954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc53015b6c0 0x7fc5300b4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.390+0000 7fc52f7fe700 1 -- 172.21.15.1:0/1435783954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5300cbc90 msgr2=0x7fc530150bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.390+0000 7fc52f7fe700 1 --2- 172.21.15.1:0/1435783954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5300cbc90 0x7fc530150bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.390+0000 7fc52f7fe700 1 -- 172.21.15.1:0/1435783954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc528009b30 con 0x7fc5300cf200 2024-03-18T20:28:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.390+0000 7fc52f7fe700 1 --2- 172.21.15.1:0/1435783954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5300cf200 0x7fc5301e2300 secure :-1 s=READY pgs=3226 cs=0 l=1 rev1=1 crypto rx=0x7fc52400c7f0 tx=0x7fc52400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.390+0000 7fc52cff9700 1 -- 172.21.15.1:0/1435783954 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc524011800 con 0x7fc5300cf200 2024-03-18T20:28:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.390+0000 7fc52cff9700 1 -- 172.21.15.1:0/1435783954 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc524011960 con 0x7fc5300cf200 2024-03-18T20:28:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.390+0000 7fc52cff9700 1 -- 172.21.15.1:0/1435783954 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc524011d10 con 0x7fc5300cf200 2024-03-18T20:28:01.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.390+0000 7fc536571700 1 -- 172.21.15.1:0/1435783954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5301e3310 con 0x7fc5300cf200 2024-03-18T20:28:01.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.390+0000 7fc536571700 1 -- 172.21.15.1:0/1435783954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5300c6c10 con 0x7fc5300cf200 2024-03-18T20:28:01.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.391+0000 7fc536571700 1 -- 172.21.15.1:0/1435783954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5300d0ca0 con 0x7fc5300cf200 2024-03-18T20:28:01.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.391+0000 7fc52cff9700 1 -- 172.21.15.1:0/1435783954 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc524018360 con 0x7fc5300cf200 2024-03-18T20:28:01.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.392+0000 7fc52cff9700 1 --2- 172.21.15.1:0/1435783954 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc51c06adb0 0x7fc51c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:01.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.392+0000 7fc52cff9700 1 -- 172.21.15.1:0/1435783954 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6094+0+0 (secure 0 0 0) 0x7fc524089570 con 0x7fc5300cf200 2024-03-18T20:28:01.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.392+0000 7fc52ffff700 1 --2- 172.21.15.1:0/1435783954 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc51c06adb0 0x7fc51c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:01.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.392+0000 7fc52ffff700 1 --2- 172.21.15.1:0/1435783954 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc51c06adb0 0x7fc51c06d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc520006fd0 tx=0x7fc520008040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:01.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.396+0000 7fc52cff9700 1 -- 172.21.15.1:0/1435783954 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc524054320 con 0x7fc5300cf200 2024-03-18T20:28:01.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.599+0000 7fc536571700 1 -- 172.21.15.1:0/1435783954 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fc5300cff90 con 0x7fc5300cf200 2024-03-18T20:28:01.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.603+0000 7fc52cff9700 1 -- 172.21.15.1:0/1435783954 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v91) v1 ==== 115+0+0 (secure 0 0 0) 0x7fc524057940 con 0x7fc5300cf200 2024-03-18T20:28:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.605+0000 7fc536571700 1 -- 172.21.15.1:0/1435783954 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc51c06adb0 msgr2=0x7fc51c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.605+0000 7fc536571700 1 --2- 172.21.15.1:0/1435783954 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc51c06adb0 0x7fc51c06d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc520006fd0 tx=0x7fc520008040 comp rx=0 tx=0).stop 2024-03-18T20:28:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.605+0000 7fc536571700 1 -- 172.21.15.1:0/1435783954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5300cf200 msgr2=0x7fc5301e2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.605+0000 7fc536571700 1 --2- 172.21.15.1:0/1435783954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5300cf200 0x7fc5301e2300 secure :-1 s=READY pgs=3226 cs=0 l=1 rev1=1 crypto rx=0x7fc52400c7f0 tx=0x7fc52400cb00 comp rx=0 tx=0).stop 2024-03-18T20:28:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.606+0000 7fc536571700 1 -- 172.21.15.1:0/1435783954 shutdown_connections 2024-03-18T20:28:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.606+0000 7fc536571700 1 --2- 172.21.15.1:0/1435783954 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc51c06adb0 0x7fc51c06d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.606+0000 7fc536571700 1 --2- 172.21.15.1:0/1435783954 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc53015b6c0 0x7fc5300b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.606+0000 7fc536571700 1 --2- 172.21.15.1:0/1435783954 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5300cbc90 0x7fc530150bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.606+0000 7fc536571700 1 --2- 172.21.15.1:0/1435783954 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5300cf200 0x7fc5301e2300 unknown :-1 s=CLOSED pgs=3226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.606+0000 7fc536571700 1 -- 172.21.15.1:0/1435783954 >> 172.21.15.1:0/1435783954 conn(0x7fc5300d1180 msgr2=0x7fc5300b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:01.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.606+0000 7fc536571700 1 -- 172.21.15.1:0/1435783954 shutdown_connections 2024-03-18T20:28:01.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.606+0000 7fc536571700 1 -- 172.21.15.1:0/1435783954 wait complete. 2024-03-18T20:28:01.623 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-18T20:28:01.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.746+0000 7f549cc10700 1 Processor -- start 2024-03-18T20:28:01.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.746+0000 7f549cc10700 1 -- start start 2024-03-18T20:28:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.747+0000 7f549cc10700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f549815d8f0 0x7f54980b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.747+0000 7f549cc10700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f54980b35e0 0x7f54980b4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.747+0000 7f549cc10700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5498072050 0x7f549806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.747+0000 7f549cc10700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54980b1960 con 0x7f5498072050 2024-03-18T20:28:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.747+0000 7f549cc10700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54980b1bb0 con 0x7f54980b35e0 2024-03-18T20:28:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.747+0000 7f549cc10700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54980adb70 con 0x7f549815d8f0 2024-03-18T20:28:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.747+0000 7f5496d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5498072050 0x7f549806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.747+0000 7f5496d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5498072050 0x7f549806a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58144/0 (socket says 172.21.15.1:58144) 2024-03-18T20:28:01.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.747+0000 7f5496d9d700 1 -- 172.21.15.1:0/4143976387 learned_addr learned my addr 172.21.15.1:0/4143976387 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.747+0000 7f5495d9b700 1 --2- 172.21.15.1:0/4143976387 >> v2:172.21.15.165:3300/0 conn(0x7f54980b35e0 0x7f54980b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.747+0000 7f549659c700 1 --2- 172.21.15.1:0/4143976387 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f549815d8f0 0x7f54980b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.747+0000 7f5496d9d700 1 -- 172.21.15.1:0/4143976387 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f549815d8f0 msgr2=0x7f54980b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.747+0000 7f5496d9d700 1 --2- 172.21.15.1:0/4143976387 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f549815d8f0 0x7f54980b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.747+0000 7f5496d9d700 1 -- 172.21.15.1:0/4143976387 >> v2:172.21.15.165:3300/0 conn(0x7f54980b35e0 msgr2=0x7f54980b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.747+0000 7f5496d9d700 1 --2- 172.21.15.1:0/4143976387 >> v2:172.21.15.165:3300/0 conn(0x7f54980b35e0 0x7f54980b4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.747+0000 7f5496d9d700 1 -- 172.21.15.1:0/4143976387 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54980ade50 con 0x7f5498072050 2024-03-18T20:28:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.747+0000 7f5495d9b700 1 --2- 172.21.15.1:0/4143976387 >> v2:172.21.15.165:3300/0 conn(0x7f54980b35e0 0x7f54980b4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:28:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.747+0000 7f5496d9d700 1 --2- 172.21.15.1:0/4143976387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5498072050 0x7f549806a7d0 secure :-1 s=READY pgs=3227 cs=0 l=1 rev1=1 crypto rx=0x7f549806da50 tx=0x7f549000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51b5d8f0ac95216e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.747+0000 7f549559a700 1 -- 172.21.15.1:0/4143976387 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549000e070 con 0x7f5498072050 2024-03-18T20:28:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.748+0000 7f549559a700 1 -- 172.21.15.1:0/4143976387 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549000b9c0 con 0x7f5498072050 2024-03-18T20:28:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.748+0000 7f549559a700 1 -- 172.21.15.1:0/4143976387 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549000bd70 con 0x7f5498072050 2024-03-18T20:28:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.748+0000 7f549cc10700 1 -- 172.21.15.1:0/4143976387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5498072050 msgr2=0x7f549806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:01.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.748+0000 7f549cc10700 1 --2- 172.21.15.1:0/4143976387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5498072050 0x7f549806a7d0 secure :-1 s=READY pgs=3227 cs=0 l=1 rev1=1 crypto rx=0x7f549806da50 tx=0x7f549000b330 comp rx=0 tx=0).stop 2024-03-18T20:28:01.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.748+0000 7f549cc10700 1 -- 172.21.15.1:0/4143976387 shutdown_connections 2024-03-18T20:28:01.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.748+0000 7f549cc10700 1 --2- 172.21.15.1:0/4143976387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5498072050 0x7f549806a7d0 unknown :-1 s=CLOSED pgs=3227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.748+0000 7f549cc10700 1 --2- 172.21.15.1:0/4143976387 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f549815d8f0 0x7f54980b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.748+0000 7f549cc10700 1 --2- 172.21.15.1:0/4143976387 >> v2:172.21.15.165:3300/0 conn(0x7f54980b35e0 0x7f54980b4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.748+0000 7f549cc10700 1 -- 172.21.15.1:0/4143976387 >> 172.21.15.1:0/4143976387 conn(0x7f54980b5a10 msgr2=0x7f54980b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:01.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.748+0000 7f549cc10700 1 -- 172.21.15.1:0/4143976387 shutdown_connections 2024-03-18T20:28:01.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.748+0000 7f549cc10700 1 -- 172.21.15.1:0/4143976387 wait complete. 2024-03-18T20:28:01.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.750+0000 7f549cc10700 1 Processor -- start 2024-03-18T20:28:01.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.750+0000 7f549cc10700 1 -- start start 2024-03-18T20:28:01.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f549cc10700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5498072050 0x7f549809a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:01.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f549cc10700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f549815d8f0 0x7f549809cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:01.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f549cc10700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5498148a50 0x7f54980aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:01.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f549cc10700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549809e0f0 con 0x7f549815d8f0 2024-03-18T20:28:01.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f549cc10700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549809d220 con 0x7f5498072050 2024-03-18T20:28:01.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f549cc10700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5498152330 con 0x7f5498148a50 2024-03-18T20:28:01.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f5495d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f549815d8f0 0x7f549809cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:01.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f5495d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f549815d8f0 0x7f549809cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58160/0 (socket says 172.21.15.1:58160) 2024-03-18T20:28:01.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f5495d9b700 1 -- 172.21.15.1:0/49181944 learned_addr learned my addr 172.21.15.1:0/49181944 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:01.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f5496d9d700 1 --2- 172.21.15.1:0/49181944 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5498148a50 0x7f54980aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:01.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f5495d9b700 1 -- 172.21.15.1:0/49181944 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5498148a50 msgr2=0x7f54980aff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:01.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f5495d9b700 1 --2- 172.21.15.1:0/49181944 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5498148a50 0x7f54980aff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f5495d9b700 1 -- 172.21.15.1:0/49181944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5498072050 msgr2=0x7f549809a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:01.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f549659c700 1 --2- 172.21.15.1:0/49181944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5498072050 0x7f549809a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:01.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f5495d9b700 1 --2- 172.21.15.1:0/49181944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5498072050 0x7f549809a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f5495d9b700 1 -- 172.21.15.1:0/49181944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5490009b30 con 0x7f549815d8f0 2024-03-18T20:28:01.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f5495d9b700 1 --2- 172.21.15.1:0/49181944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f549815d8f0 0x7f549809cce0 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 crypto rx=0x7f548c00c770 tx=0x7f548c00cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:01.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f5486ffd700 1 -- 172.21.15.1:0/49181944 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c011800 con 0x7f549815d8f0 2024-03-18T20:28:01.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f5486ffd700 1 -- 172.21.15.1:0/49181944 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548c011960 con 0x7f549815d8f0 2024-03-18T20:28:01.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f5486ffd700 1 -- 172.21.15.1:0/49181944 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c011cf0 con 0x7f549815d8f0 2024-03-18T20:28:01.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f549cc10700 1 -- 172.21.15.1:0/49181944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f549809d470 con 0x7f549815d8f0 2024-03-18T20:28:01.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.751+0000 7f549cc10700 1 -- 172.21.15.1:0/49181944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f549809da30 con 0x7f549815d8f0 2024-03-18T20:28:01.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.752+0000 7f549cc10700 1 -- 172.21.15.1:0/49181944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f549809e230 con 0x7f549815d8f0 2024-03-18T20:28:01.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.753+0000 7f5486ffd700 1 -- 172.21.15.1:0/49181944 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f548c021350 con 0x7f549815d8f0 2024-03-18T20:28:01.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.753+0000 7f5486ffd700 1 --2- 172.21.15.1:0/49181944 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f547806adb0 0x7f547806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:01.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.753+0000 7f5486ffd700 1 -- 172.21.15.1:0/49181944 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f548c089580 con 0x7f549815d8f0 2024-03-18T20:28:01.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.753+0000 7f549659c700 1 --2- 172.21.15.1:0/49181944 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f547806adb0 0x7f547806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:01.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.754+0000 7f549659c700 1 --2- 172.21.15.1:0/49181944 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f547806adb0 0x7f547806d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f548800b840 tx=0x7f5488008040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:01.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.758+0000 7f5486ffd700 1 -- 172.21.15.1:0/49181944 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f549809e230 con 0x7f549815d8f0 2024-03-18T20:28:01.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.961+0000 7f549cc10700 1 -- 172.21.15.1:0/49181944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f549809e230 con 0x7f549815d8f0 2024-03-18T20:28:01.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.965+0000 7f5486ffd700 1 -- 172.21.15.1:0/49181944 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v92) v1 ==== 120+0+0 (secure 0 0 0) 0x7f549809e230 con 0x7f549815d8f0 2024-03-18T20:28:01.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.967+0000 7f549cc10700 1 -- 172.21.15.1:0/49181944 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f547806adb0 msgr2=0x7f547806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:01.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.967+0000 7f549cc10700 1 --2- 172.21.15.1:0/49181944 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f547806adb0 0x7f547806d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f548800b840 tx=0x7f5488008040 comp rx=0 tx=0).stop 2024-03-18T20:28:01.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.967+0000 7f549cc10700 1 -- 172.21.15.1:0/49181944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f549815d8f0 msgr2=0x7f549809cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:01.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.967+0000 7f549cc10700 1 --2- 172.21.15.1:0/49181944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f549815d8f0 0x7f549809cce0 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 crypto rx=0x7f548c00c770 tx=0x7f548c00cb30 comp rx=0 tx=0).stop 2024-03-18T20:28:01.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.968+0000 7f549cc10700 1 -- 172.21.15.1:0/49181944 shutdown_connections 2024-03-18T20:28:01.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.968+0000 7f549cc10700 1 --2- 172.21.15.1:0/49181944 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f547806adb0 0x7f547806d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.968+0000 7f549cc10700 1 --2- 172.21.15.1:0/49181944 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5498148a50 0x7f54980aff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.968+0000 7f549cc10700 1 --2- 172.21.15.1:0/49181944 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5498072050 0x7f549809a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.968+0000 7f549cc10700 1 --2- 172.21.15.1:0/49181944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f549815d8f0 0x7f549809cce0 unknown :-1 s=CLOSED pgs=3228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:01.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.968+0000 7f549cc10700 1 -- 172.21.15.1:0/49181944 >> 172.21.15.1:0/49181944 conn(0x7f54980b5a10 msgr2=0x7f549815c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:01.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.968+0000 7f549cc10700 1 -- 172.21.15.1:0/49181944 shutdown_connections 2024-03-18T20:28:01.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:01.968+0000 7f549cc10700 1 -- 172.21.15.1:0/49181944 wait complete. 2024-03-18T20:28:01.984 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-18T20:28:02.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.108+0000 7fdaad254700 1 Processor -- start 2024-03-18T20:28:02.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.108+0000 7fdaad254700 1 -- start start 2024-03-18T20:28:02.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.109+0000 7fdaad254700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaa80adc50 0x7fdaa80b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:02.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.109+0000 7fdaad254700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdaa8099b30 0x7fdaa80ae030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:02.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.109+0000 7fdaad254700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa80ae570 con 0x7fdaa815b610 2024-03-18T20:28:02.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.109+0000 7fdaad254700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaa80b2c20 con 0x7fdaa8099b30 2024-03-18T20:28:02.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.109+0000 7fdaad254700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa806f860 con 0x7fdaa80adc50 2024-03-18T20:28:02.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.109+0000 7fdaa659c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdaa8099b30 0x7fdaa80ae030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:02.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.109+0000 7fdaa759e700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdaa815b610 0x7fdaa806a7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39612/0 (socket says 172.21.15.1:39612) 2024-03-18T20:28:02.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.109+0000 7fdaa759e700 1 -- 172.21.15.1:0/1495804379 learned_addr learned my addr 172.21.15.1:0/1495804379 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:02.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.109+0000 7fdaa6d9d700 1 --2- 172.21.15.1:0/1495804379 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaa80adc50 0x7fdaa80b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:02.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.109+0000 7fdaa659c700 1 -- 172.21.15.1:0/1495804379 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaa80adc50 msgr2=0x7fdaa80b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:02.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.109+0000 7fdaa659c700 1 --2- 172.21.15.1:0/1495804379 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaa80adc50 0x7fdaa80b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:02.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.109+0000 7fdaa659c700 1 -- 172.21.15.1:0/1495804379 >> v1:172.21.15.1:6789/0 conn(0x7fdaa815b610 legacy=0x7fdaa806a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:02.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.109+0000 7fdaa659c700 1 -- 172.21.15.1:0/1495804379 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa80ad3c0 con 0x7fdaa8099b30 2024-03-18T20:28:02.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.109+0000 7fdaa5d9b700 1 -- 172.21.15.1:0/1495804379 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748187530 0 0) 0x7fdaa80ae570 con 0x7fdaa815b610 2024-03-18T20:28:02.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.109+0000 7fdaa659c700 1 --2- 172.21.15.1:0/1495804379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaa8099b30 0x7fdaa80ae030 secure :-1 s=READY pgs=3230 cs=0 l=1 rev1=1 crypto rx=0x7fda98009840 tx=0x7fda98009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239ff6cc278862cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:02.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.110+0000 7fdaa5d9b700 1 -- 172.21.15.1:0/1495804379 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda98011830 con 0x7fdaa8099b30 2024-03-18T20:28:02.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.110+0000 7fdaa5d9b700 1 -- 172.21.15.1:0/1495804379 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda98011990 con 0x7fdaa8099b30 2024-03-18T20:28:02.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.110+0000 7fdaa5d9b700 1 -- 172.21.15.1:0/1495804379 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda98011d20 con 0x7fdaa8099b30 2024-03-18T20:28:02.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.110+0000 7fdaad254700 1 -- 172.21.15.1:0/1495804379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaa8099b30 msgr2=0x7fdaa80ae030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:02.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.110+0000 7fdaad254700 1 --2- 172.21.15.1:0/1495804379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaa8099b30 0x7fdaa80ae030 secure :-1 s=READY pgs=3230 cs=0 l=1 rev1=1 crypto rx=0x7fda98009840 tx=0x7fda98009b50 comp rx=0 tx=0).stop 2024-03-18T20:28:02.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.110+0000 7fdaad254700 1 -- 172.21.15.1:0/1495804379 shutdown_connections 2024-03-18T20:28:02.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.110+0000 7fdaad254700 1 --2- 172.21.15.1:0/1495804379 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaa80adc50 0x7fdaa80b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:02.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.110+0000 7fdaad254700 1 --2- 172.21.15.1:0/1495804379 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaa8099b30 0x7fdaa80ae030 unknown :-1 s=CLOSED pgs=3230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:02.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.110+0000 7fdaad254700 1 -- 172.21.15.1:0/1495804379 >> 172.21.15.1:0/1495804379 conn(0x7fdaa80b5a10 msgr2=0x7fdaa80b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:02.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.110+0000 7fdaad254700 1 -- 172.21.15.1:0/1495804379 shutdown_connections 2024-03-18T20:28:02.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.110+0000 7fdaad254700 1 -- 172.21.15.1:0/1495804379 wait complete. 2024-03-18T20:28:02.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.112+0000 7fdaad254700 1 Processor -- start 2024-03-18T20:28:02.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.112+0000 7fdaad254700 1 -- start start 2024-03-18T20:28:02.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.113+0000 7fdaad254700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaa8099b30 0x7fdaa81f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:02.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.113+0000 7fdaad254700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaa80adc50 0x7fdaa80b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:02.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.113+0000 7fdaad254700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaa815b610 0x7fdaa80c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:02.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.113+0000 7fdaad254700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa81f1ec0 con 0x7fdaa8099b30 2024-03-18T20:28:02.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.113+0000 7fdaad254700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa81f1900 con 0x7fdaa815b610 2024-03-18T20:28:02.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.113+0000 7fdaad254700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa81f1be0 con 0x7fdaa80adc50 2024-03-18T20:28:02.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.113+0000 7fdaa6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaa8099b30 0x7fdaa81f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:02.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.113+0000 7fdaa6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaa8099b30 0x7fdaa81f0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58182/0 (socket says 172.21.15.1:58182) 2024-03-18T20:28:02.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.113+0000 7fdaa6d9d700 1 -- 172.21.15.1:0/3969442574 learned_addr learned my addr 172.21.15.1:0/3969442574 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:02.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.113+0000 7fdaa659c700 1 --2- 172.21.15.1:0/3969442574 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaa80adc50 0x7fdaa80b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:02.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.113+0000 7fdaa759e700 1 --2- 172.21.15.1:0/3969442574 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaa815b610 0x7fdaa80c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:02.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.113+0000 7fdaa6d9d700 1 -- 172.21.15.1:0/3969442574 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaa80adc50 msgr2=0x7fdaa80b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:02.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.113+0000 7fdaa6d9d700 1 --2- 172.21.15.1:0/3969442574 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaa80adc50 0x7fdaa80b2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:02.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.113+0000 7fdaa6d9d700 1 -- 172.21.15.1:0/3969442574 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaa815b610 msgr2=0x7fdaa80c5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:02.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.113+0000 7fdaa6d9d700 1 --2- 172.21.15.1:0/3969442574 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaa815b610 0x7fdaa80c5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:02.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.114+0000 7fdaa6d9d700 1 -- 172.21.15.1:0/3969442574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda980094a0 con 0x7fdaa8099b30 2024-03-18T20:28:02.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.114+0000 7fdaa759e700 1 --2- 172.21.15.1:0/3969442574 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaa815b610 0x7fdaa80c5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:28:02.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.114+0000 7fdaa6d9d700 1 --2- 172.21.15.1:0/3969442574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaa8099b30 0x7fdaa81f0fb0 secure :-1 s=READY pgs=3231 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00e810 tx=0x7fda9c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:02.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.114+0000 7fda8f7fe700 1 -- 172.21.15.1:0/3969442574 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9c013610 con 0x7fdaa8099b30 2024-03-18T20:28:02.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.114+0000 7fda8f7fe700 1 -- 172.21.15.1:0/3969442574 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9c013770 con 0x7fdaa8099b30 2024-03-18T20:28:02.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.114+0000 7fdaad254700 1 -- 172.21.15.1:0/3969442574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa80c52e0 con 0x7fdaa8099b30 2024-03-18T20:28:02.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.114+0000 7fda8f7fe700 1 -- 172.21.15.1:0/3969442574 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9c013b20 con 0x7fdaa8099b30 2024-03-18T20:28:02.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.114+0000 7fdaad254700 1 -- 172.21.15.1:0/3969442574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa81f14f0 con 0x7fdaa8099b30 2024-03-18T20:28:02.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.115+0000 7fdaad254700 1 -- 172.21.15.1:0/3969442574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa80c5930 con 0x7fdaa8099b30 2024-03-18T20:28:02.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.115+0000 7fda8f7fe700 1 -- 172.21.15.1:0/3969442574 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fda9c013c80 con 0x7fdaa8099b30 2024-03-18T20:28:02.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.116+0000 7fda8f7fe700 1 --2- 172.21.15.1:0/3969442574 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fda9406adb0 0x7fda9406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:02.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.116+0000 7fda8f7fe700 1 -- 172.21.15.1:0/3969442574 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6094+0+0 (secure 0 0 0) 0x7fda9c089320 con 0x7fdaa8099b30 2024-03-18T20:28:02.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.116+0000 7fdaa659c700 1 --2- 172.21.15.1:0/3969442574 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fda9406adb0 0x7fda9406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:02.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.116+0000 7fdaa659c700 1 --2- 172.21.15.1:0/3969442574 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fda9406adb0 0x7fda9406d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fda980090b0 tx=0x7fda98008030 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:02.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.120+0000 7fda8f7fe700 1 -- 172.21.15.1:0/3969442574 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fda9c054170 con 0x7fdaa8099b30 2024-03-18T20:28:02.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.324+0000 7fdaad254700 1 -- 172.21.15.1:0/3969442574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fdaa8067dc0 con 0x7fdaa8099b30 2024-03-18T20:28:02.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.328+0000 7fda8f7fe700 1 -- 172.21.15.1:0/3969442574 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v93) v1 ==== 104+0+0 (secure 0 0 0) 0x7fda9c057790 con 0x7fdaa8099b30 2024-03-18T20:28:02.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.330+0000 7fdaad254700 1 -- 172.21.15.1:0/3969442574 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fda9406adb0 msgr2=0x7fda9406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:02.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.330+0000 7fdaad254700 1 --2- 172.21.15.1:0/3969442574 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fda9406adb0 0x7fda9406d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fda980090b0 tx=0x7fda98008030 comp rx=0 tx=0).stop 2024-03-18T20:28:02.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.330+0000 7fdaad254700 1 -- 172.21.15.1:0/3969442574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaa8099b30 msgr2=0x7fdaa81f0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:02.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.330+0000 7fdaad254700 1 --2- 172.21.15.1:0/3969442574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaa8099b30 0x7fdaa81f0fb0 secure :-1 s=READY pgs=3231 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00e810 tx=0x7fda9c00eb20 comp rx=0 tx=0).stop 2024-03-18T20:28:02.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.330+0000 7fdaad254700 1 -- 172.21.15.1:0/3969442574 shutdown_connections 2024-03-18T20:28:02.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.330+0000 7fdaad254700 1 --2- 172.21.15.1:0/3969442574 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fda9406adb0 0x7fda9406d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:02.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.330+0000 7fdaad254700 1 --2- 172.21.15.1:0/3969442574 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdaa815b610 0x7fdaa80c5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:02.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.330+0000 7fdaad254700 1 --2- 172.21.15.1:0/3969442574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaa8099b30 0x7fdaa81f0fb0 unknown :-1 s=CLOSED pgs=3231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:02.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.330+0000 7fdaad254700 1 --2- 172.21.15.1:0/3969442574 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fdaa80adc50 0x7fdaa80b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:02.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.330+0000 7fdaad254700 1 -- 172.21.15.1:0/3969442574 >> 172.21.15.1:0/3969442574 conn(0x7fdaa80b5a10 msgr2=0x7fdaa80b0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:02.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.330+0000 7fdaad254700 1 -- 172.21.15.1:0/3969442574 shutdown_connections 2024-03-18T20:28:02.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.330+0000 7fdaad254700 1 -- 172.21.15.1:0/3969442574 wait complete. 2024-03-18T20:28:02.346 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2024-03-18T20:28:02.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.467+0000 7fddbdbbf700 1 Processor -- start 2024-03-18T20:28:02.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.467+0000 7fddbdbbf700 1 -- start start 2024-03-18T20:28:02.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddbdbbf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fddb8158ec0 0x7fddb806b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:02.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddbdbbf700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fddb806c360 0x7fddb80d0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:02.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddbdbbf700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fddb80d07d0 0x7fddb815ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:02.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddbdbbf700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb80ce330 con 0x7fddb806c360 2024-03-18T20:28:02.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddbdbbf700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb80d0be0 con 0x7fddb80d07d0 2024-03-18T20:28:02.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddbdbbf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb815b970 con 0x7fddb8158ec0 2024-03-18T20:28:02.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddb6ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fddb806c360 0x7fddb80d0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:02.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddb6ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fddb806c360 0x7fddb80d0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58188/0 (socket says 172.21.15.1:58188) 2024-03-18T20:28:02.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddb6ffd700 1 -- 172.21.15.1:0/3115821175 learned_addr learned my addr 172.21.15.1:0/3115821175 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:02.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddb77fe700 1 --2- 172.21.15.1:0/3115821175 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fddb8158ec0 0x7fddb806b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:02.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddb7fff700 1 --2- 172.21.15.1:0/3115821175 >> v2:172.21.15.165:3300/0 conn(0x7fddb80d07d0 0x7fddb815ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:02.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddb6ffd700 1 -- 172.21.15.1:0/3115821175 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fddb8158ec0 msgr2=0x7fddb806b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:02.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddb6ffd700 1 --2- 172.21.15.1:0/3115821175 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fddb8158ec0 0x7fddb806b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:02.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddb6ffd700 1 -- 172.21.15.1:0/3115821175 >> v2:172.21.15.165:3300/0 conn(0x7fddb80d07d0 msgr2=0x7fddb815ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:02.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddb6ffd700 1 --2- 172.21.15.1:0/3115821175 >> v2:172.21.15.165:3300/0 conn(0x7fddb80d07d0 0x7fddb815ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:02.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddb6ffd700 1 -- 172.21.15.1:0/3115821175 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb815bc30 con 0x7fddb806c360 2024-03-18T20:28:02.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddb7fff700 1 --2- 172.21.15.1:0/3115821175 >> v2:172.21.15.165:3300/0 conn(0x7fddb80d07d0 0x7fddb815ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:28:02.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddb6ffd700 1 --2- 172.21.15.1:0/3115821175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddb806c360 0x7fddb80d0290 secure :-1 s=READY pgs=3232 cs=0 l=1 rev1=1 crypto rx=0x7fddb80b5980 tx=0x7fddac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b94091e7975feba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:02.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddb67fc700 1 -- 172.21.15.1:0/3115821175 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac00b870 con 0x7fddb806c360 2024-03-18T20:28:02.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddb67fc700 1 -- 172.21.15.1:0/3115821175 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddac00b9d0 con 0x7fddb806c360 2024-03-18T20:28:02.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.468+0000 7fddb67fc700 1 -- 172.21.15.1:0/3115821175 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac00bd60 con 0x7fddb806c360 2024-03-18T20:28:02.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.469+0000 7fddbdbbf700 1 -- 172.21.15.1:0/3115821175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddb806c360 msgr2=0x7fddb80d0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:02.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.469+0000 7fddbdbbf700 1 --2- 172.21.15.1:0/3115821175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddb806c360 0x7fddb80d0290 secure :-1 s=READY pgs=3232 cs=0 l=1 rev1=1 crypto rx=0x7fddb80b5980 tx=0x7fddac009bd0 comp rx=0 tx=0).stop 2024-03-18T20:28:02.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.469+0000 7fddbdbbf700 1 -- 172.21.15.1:0/3115821175 shutdown_connections 2024-03-18T20:28:02.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.469+0000 7fddbdbbf700 1 --2- 172.21.15.1:0/3115821175 >> v2:172.21.15.165:3300/0 conn(0x7fddb80d07d0 0x7fddb815ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:02.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.469+0000 7fddbdbbf700 1 --2- 172.21.15.1:0/3115821175 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fddb8158ec0 0x7fddb806b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:02.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.469+0000 7fddbdbbf700 1 --2- 172.21.15.1:0/3115821175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddb806c360 0x7fddb80d0290 unknown :-1 s=CLOSED pgs=3232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:02.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.469+0000 7fddbdbbf700 1 -- 172.21.15.1:0/3115821175 >> 172.21.15.1:0/3115821175 conn(0x7fddb8072120 msgr2=0x7fddb8099a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:02.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.469+0000 7fddbdbbf700 1 -- 172.21.15.1:0/3115821175 shutdown_connections 2024-03-18T20:28:02.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.469+0000 7fddbdbbf700 1 -- 172.21.15.1:0/3115821175 wait complete. 2024-03-18T20:28:02.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.470+0000 7fddbdbbf700 1 Processor -- start 2024-03-18T20:28:02.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.470+0000 7fddbdbbf700 1 -- start start 2024-03-18T20:28:02.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.471+0000 7fddbdbbf700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddb806c360 0x7fddb81507d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:02.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.471+0000 7fddbdbbf700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fddb80d07d0 0x7fddb81f2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:02.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.471+0000 7fddbdbbf700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fddb8158ec0 0x7fddb8156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:02.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.471+0000 7fddbdbbf700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb81f1e10 con 0x7fddb806c360 2024-03-18T20:28:02.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.471+0000 7fddbdbbf700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb81f2770 con 0x7fddb8158ec0 2024-03-18T20:28:02.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.471+0000 7fddbdbbf700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb8157260 con 0x7fddb80d07d0 2024-03-18T20:28:02.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.471+0000 7fddb77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddb806c360 0x7fddb81507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:02.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.471+0000 7fddb77fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddb806c360 0x7fddb81507d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58202/0 (socket says 172.21.15.1:58202) 2024-03-18T20:28:02.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.471+0000 7fddb77fe700 1 -- 172.21.15.1:0/3754438908 learned_addr learned my addr 172.21.15.1:0/3754438908 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:02.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.471+0000 7fddb7fff700 1 --2- 172.21.15.1:0/3754438908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fddb8158ec0 0x7fddb8156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:02.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.472+0000 7fddb77fe700 1 -- 172.21.15.1:0/3754438908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fddb80d07d0 msgr2=0x7fddb81f2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:02.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.472+0000 7fddb6ffd700 1 --2- 172.21.15.1:0/3754438908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fddb80d07d0 0x7fddb81f2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:02.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.472+0000 7fddb77fe700 1 --2- 172.21.15.1:0/3754438908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fddb80d07d0 0x7fddb81f2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:02.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.472+0000 7fddb77fe700 1 -- 172.21.15.1:0/3754438908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fddb8158ec0 msgr2=0x7fddb8156d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:02.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.472+0000 7fddb77fe700 1 --2- 172.21.15.1:0/3754438908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fddb8158ec0 0x7fddb8156d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:02.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.472+0000 7fddb77fe700 1 -- 172.21.15.1:0/3754438908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddac009550 con 0x7fddb806c360 2024-03-18T20:28:02.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.472+0000 7fddb7fff700 1 --2- 172.21.15.1:0/3754438908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fddb8158ec0 0x7fddb8156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:28:02.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.472+0000 7fddb77fe700 1 --2- 172.21.15.1:0/3754438908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddb806c360 0x7fddb81507d0 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 crypto rx=0x7fdda800c7f0 tx=0x7fdda800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:02.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.472+0000 7fdd9ffff700 1 -- 172.21.15.1:0/3754438908 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda8011800 con 0x7fddb806c360 2024-03-18T20:28:02.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.472+0000 7fdd9ffff700 1 -- 172.21.15.1:0/3754438908 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda8011960 con 0x7fddb806c360 2024-03-18T20:28:02.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.472+0000 7fddbdbbf700 1 -- 172.21.15.1:0/3754438908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb8157500 con 0x7fddb806c360 2024-03-18T20:28:02.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.472+0000 7fdd9ffff700 1 -- 172.21.15.1:0/3754438908 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda8011cf0 con 0x7fddb806c360 2024-03-18T20:28:02.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.472+0000 7fddbdbbf700 1 -- 172.21.15.1:0/3754438908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb81f1a30 con 0x7fddb806c360 2024-03-18T20:28:02.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.473+0000 7fddbdbbf700 1 -- 172.21.15.1:0/3754438908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb80b1160 con 0x7fddb806c360 2024-03-18T20:28:02.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.473+0000 7fdd9ffff700 1 -- 172.21.15.1:0/3754438908 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fdda8018360 con 0x7fddb806c360 2024-03-18T20:28:02.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.474+0000 7fdd9ffff700 1 --2- 172.21.15.1:0/3754438908 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdda406adb0 0x7fdda406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:02.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.474+0000 7fddb6ffd700 1 --2- 172.21.15.1:0/3754438908 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdda406adb0 0x7fdda406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:02.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.474+0000 7fdd9ffff700 1 -- 172.21.15.1:0/3754438908 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6094+0+0 (secure 0 0 0) 0x7fdda80894d0 con 0x7fddb806c360 2024-03-18T20:28:02.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.474+0000 7fddb6ffd700 1 --2- 172.21.15.1:0/3754438908 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdda406adb0 0x7fdda406d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fddb80b0b80 tx=0x7fddac008030 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:02.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.479+0000 7fdd9ffff700 1 -- 172.21.15.1:0/3754438908 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdda80543a0 con 0x7fddb806c360 2024-03-18T20:28:02.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:02.765+0000 7fddbdbbf700 1 -- 172.21.15.1:0/3754438908 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7fddb81f2080 con 0x7fddb806c360 2024-03-18T20:28:03.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.219+0000 7fdd9ffff700 1 -- 172.21.15.1:0/3754438908 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3251) v1 ==== 155+0+0 (secure 0 0 0) 0x7fdda80579c0 con 0x7fddb806c360 2024-03-18T20:28:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.221+0000 7fddbdbbf700 1 -- 172.21.15.1:0/3754438908 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdda406adb0 msgr2=0x7fdda406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.221+0000 7fddbdbbf700 1 --2- 172.21.15.1:0/3754438908 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdda406adb0 0x7fdda406d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fddb80b0b80 tx=0x7fddac008030 comp rx=0 tx=0).stop 2024-03-18T20:28:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.221+0000 7fddbdbbf700 1 -- 172.21.15.1:0/3754438908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddb806c360 msgr2=0x7fddb81507d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.221+0000 7fddbdbbf700 1 --2- 172.21.15.1:0/3754438908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddb806c360 0x7fddb81507d0 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 crypto rx=0x7fdda800c7f0 tx=0x7fdda800cbb0 comp rx=0 tx=0).stop 2024-03-18T20:28:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.221+0000 7fddbdbbf700 1 -- 172.21.15.1:0/3754438908 shutdown_connections 2024-03-18T20:28:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.221+0000 7fddbdbbf700 1 --2- 172.21.15.1:0/3754438908 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fdda406adb0 0x7fdda406d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.221+0000 7fddbdbbf700 1 --2- 172.21.15.1:0/3754438908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fddb8158ec0 0x7fddb8156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.221+0000 7fddbdbbf700 1 --2- 172.21.15.1:0/3754438908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fddb806c360 0x7fddb81507d0 unknown :-1 s=CLOSED pgs=3233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.221+0000 7fddbdbbf700 1 --2- 172.21.15.1:0/3754438908 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fddb80d07d0 0x7fddb81f2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.221+0000 7fddbdbbf700 1 -- 172.21.15.1:0/3754438908 >> 172.21.15.1:0/3754438908 conn(0x7fddb8072120 msgr2=0x7fddb815d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.222+0000 7fddbdbbf700 1 -- 172.21.15.1:0/3754438908 shutdown_connections 2024-03-18T20:28:03.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.222+0000 7fddbdbbf700 1 -- 172.21.15.1:0/3754438908 wait complete. 2024-03-18T20:28:03.240 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2024-03-18T20:28:03.241 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) Test subvol schedule creation succeeds for default subvolgroup. 2024-03-18T20:28:03.241 INFO:tasks.cephfs_test_runner:test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-03-18T20:28:03.241 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01' 2024-03-18T20:28:03.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.369+0000 7f0757022700 1 Processor -- start 2024-03-18T20:28:03.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.369+0000 7f0757022700 1 -- start start 2024-03-18T20:28:03.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.370+0000 7f0757022700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f07500b3260 0x7f07500b4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:03.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.370+0000 7f0757022700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0750142a70 con 0x7f0750140cb0 2024-03-18T20:28:03.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.370+0000 7f0757022700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0750144120 con 0x7f0750143880 2024-03-18T20:28:03.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.370+0000 7f0757022700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0750143610 con 0x7f07500b3260 2024-03-18T20:28:03.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.370+0000 7f074ffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f07500b3260 0x7f07500b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:03.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.370+0000 7f0754dbe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0750140cb0 0x7f0750142f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39626/0 (socket says 172.21.15.1:39626) 2024-03-18T20:28:03.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.370+0000 7f0754dbe700 1 -- 172.21.15.1:0/1722185363 learned_addr learned my addr 172.21.15.1:0/1722185363 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:03.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.370+0000 7f074ffff700 1 -- 172.21.15.1:0/1722185363 >> v1:172.21.15.165:6789/0 conn(0x7f0750143880 legacy=0x7f075015ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:03.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.370+0000 7f074ffff700 1 -- 172.21.15.1:0/1722185363 >> v1:172.21.15.1:6789/0 conn(0x7f0750140cb0 legacy=0x7f0750142f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:03.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.370+0000 7f074ffff700 1 -- 172.21.15.1:0/1722185363 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0750141e60 con 0x7f07500b3260 2024-03-18T20:28:03.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.370+0000 7f074ffff700 1 --2- 172.21.15.1:0/1722185363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07500b3260 0x7f07500b4840 secure :-1 s=READY pgs=3235 cs=0 l=1 rev1=1 crypto rx=0x7f073c009840 tx=0x7f073c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d2b59881d591a88 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:03.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.370+0000 7f074f7fe700 1 -- 172.21.15.1:0/1722185363 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073c004230 con 0x7f07500b3260 2024-03-18T20:28:03.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.371+0000 7f074f7fe700 1 -- 172.21.15.1:0/1722185363 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f073c004390 con 0x7f07500b3260 2024-03-18T20:28:03.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.371+0000 7f074f7fe700 1 -- 172.21.15.1:0/1722185363 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073c0036a0 con 0x7f07500b3260 2024-03-18T20:28:03.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.371+0000 7f0757022700 1 -- 172.21.15.1:0/1722185363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07500b3260 msgr2=0x7f07500b4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:03.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.371+0000 7f0757022700 1 --2- 172.21.15.1:0/1722185363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07500b3260 0x7f07500b4840 secure :-1 s=READY pgs=3235 cs=0 l=1 rev1=1 crypto rx=0x7f073c009840 tx=0x7f073c009b50 comp rx=0 tx=0).stop 2024-03-18T20:28:03.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.371+0000 7f0757022700 1 -- 172.21.15.1:0/1722185363 shutdown_connections 2024-03-18T20:28:03.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.371+0000 7f0757022700 1 --2- 172.21.15.1:0/1722185363 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07500b3260 0x7f07500b4840 unknown :-1 s=CLOSED pgs=3235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:03.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.371+0000 7f0757022700 1 -- 172.21.15.1:0/1722185363 >> 172.21.15.1:0/1722185363 conn(0x7f07500b5780 msgr2=0x7f07500b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:03.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.371+0000 7f0757022700 1 -- 172.21.15.1:0/1722185363 shutdown_connections 2024-03-18T20:28:03.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.371+0000 7f0757022700 1 -- 172.21.15.1:0/1722185363 wait complete. 2024-03-18T20:28:03.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.372+0000 7f0757022700 1 Processor -- start 2024-03-18T20:28:03.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.372+0000 7f0757022700 1 -- start start 2024-03-18T20:28:03.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.373+0000 7f0757022700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07500b3260 0x7f07500ce370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:03.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.373+0000 7f0757022700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0750140cb0 0x7f075009a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:03.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.373+0000 7f0757022700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0750143880 0x7f07501505e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:03.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.373+0000 7f0757022700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0750150b20 con 0x7f0750140cb0 2024-03-18T20:28:03.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.373+0000 7f0757022700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075009ce80 con 0x7f0750143880 2024-03-18T20:28:03.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.373+0000 7f0757022700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075009d100 con 0x7f07500b3260 2024-03-18T20:28:03.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.373+0000 7f074ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0750140cb0 0x7f075009a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:03.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.373+0000 7f074ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0750140cb0 0x7f075009a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58226/0 (socket says 172.21.15.1:58226) 2024-03-18T20:28:03.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.373+0000 7f074ffff700 1 -- 172.21.15.1:0/1717875250 learned_addr learned my addr 172.21.15.1:0/1717875250 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:03.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.373+0000 7f0754dbe700 1 --2- 172.21.15.1:0/1717875250 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07500b3260 0x7f07500ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:03.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.373+0000 7f07555bf700 1 --2- 172.21.15.1:0/1717875250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0750143880 0x7f07501505e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:03.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.374+0000 7f074ffff700 1 -- 172.21.15.1:0/1717875250 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07500b3260 msgr2=0x7f07500ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:03.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.374+0000 7f074ffff700 1 --2- 172.21.15.1:0/1717875250 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07500b3260 0x7f07500ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:03.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.374+0000 7f074ffff700 1 -- 172.21.15.1:0/1717875250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0750143880 msgr2=0x7f07501505e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:03.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.374+0000 7f074ffff700 1 --2- 172.21.15.1:0/1717875250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0750143880 0x7f07501505e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:03.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.374+0000 7f074ffff700 1 -- 172.21.15.1:0/1717875250 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073c0094a0 con 0x7f0750140cb0 2024-03-18T20:28:03.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.374+0000 7f074ffff700 1 --2- 172.21.15.1:0/1717875250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0750140cb0 0x7f075009a730 secure :-1 s=READY pgs=3236 cs=0 l=1 rev1=1 crypto rx=0x7f073c0090b0 tx=0x7f073c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:03.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.374+0000 7f074d7fa700 1 -- 172.21.15.1:0/1717875250 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073c011830 con 0x7f0750140cb0 2024-03-18T20:28:03.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.374+0000 7f074d7fa700 1 -- 172.21.15.1:0/1717875250 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f073c01e070 con 0x7f0750140cb0 2024-03-18T20:28:03.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.374+0000 7f0757022700 1 -- 172.21.15.1:0/1717875250 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0750150d80 con 0x7f0750140cb0 2024-03-18T20:28:03.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.374+0000 7f074d7fa700 1 -- 172.21.15.1:0/1717875250 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073c011bf0 con 0x7f0750140cb0 2024-03-18T20:28:03.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.374+0000 7f0757022700 1 -- 172.21.15.1:0/1717875250 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0750151340 con 0x7f0750140cb0 2024-03-18T20:28:03.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.375+0000 7f0757022700 1 -- 172.21.15.1:0/1717875250 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f075009d2c0 con 0x7f0750140cb0 2024-03-18T20:28:03.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.375+0000 7f074d7fa700 1 -- 172.21.15.1:0/1717875250 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f073c011d50 con 0x7f0750140cb0 2024-03-18T20:28:03.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.376+0000 7f074d7fa700 1 --2- 172.21.15.1:0/1717875250 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f073406adb0 0x7f073406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:03.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.376+0000 7f0754dbe700 1 --2- 172.21.15.1:0/1717875250 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f073406adb0 0x7f073406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:03.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.376+0000 7f074d7fa700 1 -- 172.21.15.1:0/1717875250 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f073c08a950 con 0x7f0750140cb0 2024-03-18T20:28:03.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.376+0000 7f0754dbe700 1 --2- 172.21.15.1:0/1717875250 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f073406adb0 0x7f073406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0744000f60 tx=0x7f0744009040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:03.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.381+0000 7f074d7fa700 1 -- 172.21.15.1:0/1717875250 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f073c055930 con 0x7f0750140cb0 2024-03-18T20:28:03.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:03.668+0000 7f0757022700 1 -- 172.21.15.1:0/1717875250 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]} v 0) v1 -- 0x7f0750067dc0 con 0x7f0750140cb0 2024-03-18T20:28:04.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.219+0000 7f074d7fa700 1 -- 172.21.15.1:0/1717875250 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]}]=0 v3252) v1 ==== 204+0+0 (secure 0 0 0) 0x7f073c020020 con 0x7f0750140cb0 2024-03-18T20:28:04.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.222+0000 7f0757022700 1 -- 172.21.15.1:0/1717875250 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f073406adb0 msgr2=0x7f073406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:04.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.222+0000 7f0757022700 1 --2- 172.21.15.1:0/1717875250 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f073406adb0 0x7f073406d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0744000f60 tx=0x7f0744009040 comp rx=0 tx=0).stop 2024-03-18T20:28:04.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.222+0000 7f0757022700 1 -- 172.21.15.1:0/1717875250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0750140cb0 msgr2=0x7f075009a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:04.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.222+0000 7f0757022700 1 --2- 172.21.15.1:0/1717875250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0750140cb0 0x7f075009a730 secure :-1 s=READY pgs=3236 cs=0 l=1 rev1=1 crypto rx=0x7f073c0090b0 tx=0x7f073c004750 comp rx=0 tx=0).stop 2024-03-18T20:28:04.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.222+0000 7f0757022700 1 -- 172.21.15.1:0/1717875250 shutdown_connections 2024-03-18T20:28:04.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.222+0000 7f0757022700 1 --2- 172.21.15.1:0/1717875250 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f073406adb0 0x7f073406d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:04.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.222+0000 7f0757022700 1 --2- 172.21.15.1:0/1717875250 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f07500b3260 0x7f07500ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:04.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.222+0000 7f0757022700 1 --2- 172.21.15.1:0/1717875250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0750143880 0x7f07501505e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:04.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.222+0000 7f0757022700 1 --2- 172.21.15.1:0/1717875250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0750140cb0 0x7f075009a730 unknown :-1 s=CLOSED pgs=3236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:04.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.222+0000 7f0757022700 1 -- 172.21.15.1:0/1717875250 >> 172.21.15.1:0/1717875250 conn(0x7f07500b5780 msgr2=0x7f075006a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:04.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.222+0000 7f0757022700 1 -- 172.21.15.1:0/1717875250 shutdown_connections 2024-03-18T20:28:04.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.222+0000 7f0757022700 1 -- 172.21.15.1:0/1717875250 wait complete. 2024-03-18T20:28:04.241 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-18T20:28:04.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.372+0000 7f05c1158700 1 Processor -- start 2024-03-18T20:28:04.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.372+0000 7f05c1158700 1 -- start start 2024-03-18T20:28:04.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.373+0000 7f05c1158700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f05bc159fe0 0x7f05bc06e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:04.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.373+0000 7f05c1158700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05bc0b4560 0x7f05bc06ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:04.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.373+0000 7f05c1158700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05bc06faa0 con 0x7f05bc15a3c0 2024-03-18T20:28:04.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.373+0000 7f05c1158700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05bc0723d0 con 0x7f05bc159fe0 2024-03-18T20:28:04.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.373+0000 7f05c1158700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc06f200 con 0x7f05bc0b4560 2024-03-18T20:28:04.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.373+0000 7f05bad9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f05bc159fe0 0x7f05bc06e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:04.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.373+0000 7f05bad9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f05bc159fe0 0x7f05bc06e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54118/0 (socket says 172.21.15.1:54118) 2024-03-18T20:28:04.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.373+0000 7f05bad9d700 1 -- 172.21.15.1:0/910534482 learned_addr learned my addr 172.21.15.1:0/910534482 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:04.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.373+0000 7f05bb59e700 1 --2- 172.21.15.1:0/910534482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05bc0b4560 0x7f05bc06ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:04.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.374+0000 7f05bad9d700 1 -- 172.21.15.1:0/910534482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05bc0b4560 msgr2=0x7f05bc06ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:04.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.374+0000 7f05bad9d700 1 --2- 172.21.15.1:0/910534482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05bc0b4560 0x7f05bc06ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:04.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.374+0000 7f05bad9d700 1 -- 172.21.15.1:0/910534482 >> v1:172.21.15.165:6789/0 conn(0x7f05bc15a3c0 legacy=0x7f05bc0b3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:04.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.374+0000 7f05bad9d700 1 -- 172.21.15.1:0/910534482 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05bc06f4e0 con 0x7f05bc159fe0 2024-03-18T20:28:04.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.374+0000 7f05bad9d700 1 --2- 172.21.15.1:0/910534482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05bc159fe0 0x7f05bc06e690 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 crypto rx=0x7f05b00098f0 tx=0x7f05b0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76078d2a7b6d4f19 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:04.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.374+0000 7f05b9d9b700 1 -- 172.21.15.1:0/910534482 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b000b8a0 con 0x7f05bc159fe0 2024-03-18T20:28:04.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.374+0000 7f05b9d9b700 1 -- 172.21.15.1:0/910534482 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b000ba00 con 0x7f05bc159fe0 2024-03-18T20:28:04.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.374+0000 7f05b9d9b700 1 -- 172.21.15.1:0/910534482 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b000bd90 con 0x7f05bc159fe0 2024-03-18T20:28:04.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.374+0000 7f05c1158700 1 -- 172.21.15.1:0/910534482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05bc159fe0 msgr2=0x7f05bc06e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:04.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.374+0000 7f05c1158700 1 --2- 172.21.15.1:0/910534482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05bc159fe0 0x7f05bc06e690 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 crypto rx=0x7f05b00098f0 tx=0x7f05b0009c00 comp rx=0 tx=0).stop 2024-03-18T20:28:04.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.375+0000 7f05c1158700 1 -- 172.21.15.1:0/910534482 shutdown_connections 2024-03-18T20:28:04.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.375+0000 7f05c1158700 1 --2- 172.21.15.1:0/910534482 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05bc0b4560 0x7f05bc06ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:04.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.375+0000 7f05c1158700 1 --2- 172.21.15.1:0/910534482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05bc159fe0 0x7f05bc06e690 unknown :-1 s=CLOSED pgs=3237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:04.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.375+0000 7f05c1158700 1 -- 172.21.15.1:0/910534482 >> 172.21.15.1:0/910534482 conn(0x7f05bc06c360 msgr2=0x7f05bc099a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:04.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.375+0000 7f05c1158700 1 -- 172.21.15.1:0/910534482 shutdown_connections 2024-03-18T20:28:04.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.375+0000 7f05c1158700 1 -- 172.21.15.1:0/910534482 wait complete. 2024-03-18T20:28:04.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.376+0000 7f05c1158700 1 Processor -- start 2024-03-18T20:28:04.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.376+0000 7f05c1158700 1 -- start start 2024-03-18T20:28:04.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.377+0000 7f05c1158700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05bc0b4560 0x7f05bc1e7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:04.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.377+0000 7f05c1158700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05bc159fe0 0x7f05bc06e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:04.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.377+0000 7f05c1158700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05bc15a3c0 0x7f05bc1377b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:04.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.377+0000 7f05c1158700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc15bb00 con 0x7f05bc0b4560 2024-03-18T20:28:04.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.377+0000 7f05c1158700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc15bde0 con 0x7f05bc15a3c0 2024-03-18T20:28:04.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.377+0000 7f05c1158700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc15c0c0 con 0x7f05bc159fe0 2024-03-18T20:28:04.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.377+0000 7f05bad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05bc0b4560 0x7f05bc1e7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:04.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.377+0000 7f05bad9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05bc0b4560 0x7f05bc1e7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54128/0 (socket says 172.21.15.1:54128) 2024-03-18T20:28:04.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.377+0000 7f05bad9d700 1 -- 172.21.15.1:0/745299451 learned_addr learned my addr 172.21.15.1:0/745299451 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:04.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.377+0000 7f05bb59e700 1 --2- 172.21.15.1:0/745299451 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05bc15a3c0 0x7f05bc1377b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:04.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.377+0000 7f05ba59c700 1 --2- 172.21.15.1:0/745299451 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05bc159fe0 0x7f05bc06e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:04.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.377+0000 7f05bad9d700 1 -- 172.21.15.1:0/745299451 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05bc159fe0 msgr2=0x7f05bc06e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:04.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.377+0000 7f05bad9d700 1 --2- 172.21.15.1:0/745299451 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05bc159fe0 0x7f05bc06e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:04.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.377+0000 7f05bad9d700 1 -- 172.21.15.1:0/745299451 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05bc15a3c0 msgr2=0x7f05bc1377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:04.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.377+0000 7f05bad9d700 1 --2- 172.21.15.1:0/745299451 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05bc15a3c0 0x7f05bc1377b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:04.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.377+0000 7f05bad9d700 1 -- 172.21.15.1:0/745299451 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b0009550 con 0x7f05bc0b4560 2024-03-18T20:28:04.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.377+0000 7f05bad9d700 1 --2- 172.21.15.1:0/745299451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05bc0b4560 0x7f05bc1e7040 secure :-1 s=READY pgs=3238 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009c40 tx=0x7f05b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:04.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.378+0000 7f05a37fe700 1 -- 172.21.15.1:0/745299451 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b00036a0 con 0x7f05bc0b4560 2024-03-18T20:28:04.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.378+0000 7f05a37fe700 1 -- 172.21.15.1:0/745299451 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b001f070 con 0x7f05bc0b4560 2024-03-18T20:28:04.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.378+0000 7f05c1158700 1 -- 172.21.15.1:0/745299451 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05bc15c3a0 con 0x7f05bc0b4560 2024-03-18T20:28:04.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.378+0000 7f05a37fe700 1 -- 172.21.15.1:0/745299451 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b0003a20 con 0x7f05bc0b4560 2024-03-18T20:28:04.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.378+0000 7f05c1158700 1 -- 172.21.15.1:0/745299451 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05bc1e7580 con 0x7f05bc0b4560 2024-03-18T20:28:04.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.379+0000 7f05c1158700 1 -- 172.21.15.1:0/745299451 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05bc1f3350 con 0x7f05bc0b4560 2024-03-18T20:28:04.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.379+0000 7f05a37fe700 1 -- 172.21.15.1:0/745299451 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f05b0003b80 con 0x7f05bc0b4560 2024-03-18T20:28:04.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.379+0000 7f05a37fe700 1 --2- 172.21.15.1:0/745299451 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f05a806adb0 0x7f05a806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:04.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.380+0000 7f05a37fe700 1 -- 172.21.15.1:0/745299451 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f05b0021070 con 0x7f05bc0b4560 2024-03-18T20:28:04.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.380+0000 7f05ba59c700 1 --2- 172.21.15.1:0/745299451 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f05a806adb0 0x7f05a806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:04.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.380+0000 7f05ba59c700 1 --2- 172.21.15.1:0/745299451 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f05a806adb0 0x7f05a806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f05bc0683b0 tx=0x7f05ac009040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:04.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.384+0000 7f05a37fe700 1 -- 172.21.15.1:0/745299451 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f05b0055f10 con 0x7f05bc0b4560 2024-03-18T20:28:04.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.584+0000 7f05c1158700 1 -- 172.21.15.1:0/745299451 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f05bc067dc0 con 0x7f05bc0b4560 2024-03-18T20:28:04.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.587+0000 7f05a37fe700 1 -- 172.21.15.1:0/745299451 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v94)=0 v94) v1 ==== 122+0+0 (secure 0 0 0) 0x7f05b002b020 con 0x7f05bc0b4560 2024-03-18T20:28:04.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.590+0000 7f05c1158700 1 -- 172.21.15.1:0/745299451 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f05a806adb0 msgr2=0x7f05a806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:04.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.590+0000 7f05c1158700 1 --2- 172.21.15.1:0/745299451 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f05a806adb0 0x7f05a806d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f05bc0683b0 tx=0x7f05ac009040 comp rx=0 tx=0).stop 2024-03-18T20:28:04.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.590+0000 7f05c1158700 1 -- 172.21.15.1:0/745299451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05bc0b4560 msgr2=0x7f05bc1e7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:04.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.590+0000 7f05c1158700 1 --2- 172.21.15.1:0/745299451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05bc0b4560 0x7f05bc1e7040 secure :-1 s=READY pgs=3238 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009c40 tx=0x7f05b0004750 comp rx=0 tx=0).stop 2024-03-18T20:28:04.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.590+0000 7f05c1158700 1 -- 172.21.15.1:0/745299451 shutdown_connections 2024-03-18T20:28:04.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.590+0000 7f05c1158700 1 --2- 172.21.15.1:0/745299451 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f05a806adb0 0x7f05a806d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:04.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.590+0000 7f05c1158700 1 --2- 172.21.15.1:0/745299451 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05bc15a3c0 0x7f05bc1377b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:04.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.590+0000 7f05c1158700 1 --2- 172.21.15.1:0/745299451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05bc0b4560 0x7f05bc1e7040 unknown :-1 s=CLOSED pgs=3238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:04.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.590+0000 7f05c1158700 1 --2- 172.21.15.1:0/745299451 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f05bc159fe0 0x7f05bc06e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:04.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.590+0000 7f05c1158700 1 -- 172.21.15.1:0/745299451 >> 172.21.15.1:0/745299451 conn(0x7f05bc06c360 msgr2=0x7f05bc0b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:04.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.590+0000 7f05c1158700 1 -- 172.21.15.1:0/745299451 shutdown_connections 2024-03-18T20:28:04.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.590+0000 7f05c1158700 1 -- 172.21.15.1:0/745299451 wait complete. 2024-03-18T20:28:04.606 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T20:28:04.606 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:28:04.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:28:04.701 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T20:28:04.701 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:28:04.701 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:28:04.720 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:28:04.720 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:28:04.720 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:28:04.741 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:28:04.741 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:28:04.741 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:28:04.778 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:28:04.778 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:28:04.778 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:28:04.802 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:28:04.802 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-18T20:28:04.802 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:28:04.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.938+0000 7fa568fce700 1 Processor -- start 2024-03-18T20:28:04.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.938+0000 7fa568fce700 1 -- start start 2024-03-18T20:28:04.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.939+0000 7fa568fce700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa564140510 0x7fa56415e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:04.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.939+0000 7fa568fce700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa564070ea0 con 0x7fa5640b48b0 2024-03-18T20:28:04.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.939+0000 7fa568fce700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa564070300 con 0x7fa56415dc00 2024-03-18T20:28:04.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.939+0000 7fa568fce700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5641408f0 con 0x7fa564140510 2024-03-18T20:28:04.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.939+0000 7fa562d9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa5640b48b0 0x7fa564140b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54088/0 (socket says 172.21.15.1:54088) 2024-03-18T20:28:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.939+0000 7fa562d9d700 1 -- 172.21.15.1:0/755943385 learned_addr learned my addr 172.21.15.1:0/755943385 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.939+0000 7fa56259c700 1 --2- 172.21.15.1:0/755943385 >> v2:172.21.15.165:3300/0 conn(0x7fa564140510 0x7fa56415e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.939+0000 7fa56159a700 1 -- 172.21.15.1:0/755943385 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254926580 0 0) 0x7fa564070ea0 con 0x7fa5640b48b0 2024-03-18T20:28:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa56159a700 1 -- 172.21.15.1:0/755943385 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa54c0035b0 con 0x7fa5640b48b0 2024-03-18T20:28:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa56159a700 1 -- 172.21.15.1:0/755943385 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1032495592 0 0) 0x7fa54c0035b0 con 0x7fa5640b48b0 2024-03-18T20:28:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa56159a700 1 -- 172.21.15.1:0/755943385 >> v2:172.21.15.165:3300/0 conn(0x7fa564140510 msgr2=0x7fa56415e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa56159a700 1 --2- 172.21.15.1:0/755943385 >> v2:172.21.15.165:3300/0 conn(0x7fa564140510 0x7fa56415e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa56159a700 1 -- 172.21.15.1:0/755943385 >> v1:172.21.15.165:6789/0 conn(0x7fa56415dc00 legacy=0x7fa5640b41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa56159a700 1 -- 172.21.15.1:0/755943385 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa564070570 con 0x7fa5640b48b0 2024-03-18T20:28:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa56159a700 1 -- 172.21.15.1:0/755943385 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fa55c0019f0 con 0x7fa5640b48b0 2024-03-18T20:28:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa56159a700 1 -- 172.21.15.1:0/755943385 >> v1:172.21.15.1:6789/0 conn(0x7fa5640b48b0 legacy=0x7fa564140b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:04.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa56159a700 1 --2- 172.21.15.1:0/755943385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa54c003a00 0x7fa54c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa56159a700 1 --2- 172.21.15.1:0/755943385 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa54c004850 0x7fa54c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa56159a700 1 --2- 172.21.15.1:0/755943385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa54c005210 0x7fa54c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa56159a700 1 -- 172.21.15.1:0/755943385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5641408f0 con 0x7fa54c005210 2024-03-18T20:28:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa56159a700 1 -- 172.21.15.1:0/755943385 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c00c0b0 con 0x7fa54c003a00 2024-03-18T20:28:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa56159a700 1 -- 172.21.15.1:0/755943385 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c00c270 con 0x7fa54c004850 2024-03-18T20:28:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa562d9d700 1 --2- 172.21.15.1:0/755943385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa54c005210 0x7fa54c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa56259c700 1 --2- 172.21.15.1:0/755943385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa54c003a00 0x7fa54c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa561d9b700 1 --2- 172.21.15.1:0/755943385 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa54c004850 0x7fa54c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa562d9d700 1 -- 172.21.15.1:0/755943385 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa54c004850 msgr2=0x7fa54c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa562d9d700 1 --2- 172.21.15.1:0/755943385 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa54c004850 0x7fa54c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa562d9d700 1 -- 172.21.15.1:0/755943385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa54c003a00 msgr2=0x7fa54c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa562d9d700 1 --2- 172.21.15.1:0/755943385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa54c003a00 0x7fa54c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa562d9d700 1 -- 172.21.15.1:0/755943385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54c0035b0 con 0x7fa54c005210 2024-03-18T20:28:04.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa562d9d700 1 -- 172.21.15.1:0/755943385 reap_dead start 2024-03-18T20:28:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.940+0000 7fa562d9d700 1 --2- 172.21.15.1:0/755943385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa54c005210 0x7fa54c00ba10 secure :-1 s=READY pgs=3240 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00a830 tx=0x7fa54c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f882e6c6d1fa4eb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.941+0000 7fa56159a700 1 -- 172.21.15.1:0/755943385 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c00a2f0 con 0x7fa54c005210 2024-03-18T20:28:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.941+0000 7fa56159a700 1 -- 172.21.15.1:0/755943385 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa55c012620 con 0x7fa54c005210 2024-03-18T20:28:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.941+0000 7fa56159a700 1 -- 172.21.15.1:0/755943385 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c0129d0 con 0x7fa54c005210 2024-03-18T20:28:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.941+0000 7fa568fce700 1 -- 172.21.15.1:0/755943385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa54c005210 msgr2=0x7fa54c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.941+0000 7fa568fce700 1 --2- 172.21.15.1:0/755943385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa54c005210 0x7fa54c00ba10 secure :-1 s=READY pgs=3240 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00a830 tx=0x7fa54c005880 comp rx=0 tx=0).stop 2024-03-18T20:28:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.941+0000 7fa568fce700 1 -- 172.21.15.1:0/755943385 shutdown_connections 2024-03-18T20:28:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.941+0000 7fa568fce700 1 --2- 172.21.15.1:0/755943385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa54c005210 0x7fa54c00ba10 unknown :-1 s=CLOSED pgs=3240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.941+0000 7fa568fce700 1 -- 172.21.15.1:0/755943385 >> 172.21.15.1:0/755943385 conn(0x7fa5640b2ea0 msgr2=0x7fa5640b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.941+0000 7fa568fce700 1 -- 172.21.15.1:0/755943385 shutdown_connections 2024-03-18T20:28:04.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.941+0000 7fa568fce700 1 -- 172.21.15.1:0/755943385 wait complete. 2024-03-18T20:28:04.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.943+0000 7fa568fce700 1 Processor -- start 2024-03-18T20:28:04.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.943+0000 7fa568fce700 1 -- start start 2024-03-18T20:28:04.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa568fce700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa54c005210 0x7fa56413f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:04.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa568fce700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5640ccf00 0x7fa5641458b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:04.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa568fce700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa564145df0 0x7fa5641462a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:04.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa568fce700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56415b1a0 con 0x7fa54c005210 2024-03-18T20:28:04.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa568fce700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56413ffa0 con 0x7fa564145df0 2024-03-18T20:28:04.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa568fce700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5641446d0 con 0x7fa5640ccf00 2024-03-18T20:28:04.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa56259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa54c005210 0x7fa56413f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:04.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa56259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa54c005210 0x7fa56413f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54140/0 (socket says 172.21.15.1:54140) 2024-03-18T20:28:04.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa56259c700 1 -- 172.21.15.1:0/967435188 learned_addr learned my addr 172.21.15.1:0/967435188 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:04.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa561d9b700 1 --2- 172.21.15.1:0/967435188 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5640ccf00 0x7fa5641458b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:04.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa562d9d700 1 --2- 172.21.15.1:0/967435188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa564145df0 0x7fa5641462a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:04.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa56259c700 1 -- 172.21.15.1:0/967435188 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5640ccf00 msgr2=0x7fa5641458b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:04.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa56259c700 1 --2- 172.21.15.1:0/967435188 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5640ccf00 0x7fa5641458b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:04.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa56259c700 1 -- 172.21.15.1:0/967435188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa564145df0 msgr2=0x7fa5641462a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:04.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa56259c700 1 --2- 172.21.15.1:0/967435188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa564145df0 0x7fa5641462a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:04.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa56259c700 1 -- 172.21.15.1:0/967435188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56413fa40 con 0x7fa54c005210 2024-03-18T20:28:04.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa56259c700 1 --2- 172.21.15.1:0/967435188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa54c005210 0x7fa56413f500 secure :-1 s=READY pgs=3241 cs=0 l=1 rev1=1 crypto rx=0x7fa5580119d0 tx=0x7fa558011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:04.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa552ffd700 1 -- 172.21.15.1:0/967435188 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55800f6d0 con 0x7fa54c005210 2024-03-18T20:28:04.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa552ffd700 1 -- 172.21.15.1:0/967435188 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa55800f830 con 0x7fa54c005210 2024-03-18T20:28:04.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa568fce700 1 -- 172.21.15.1:0/967435188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5640ca520 con 0x7fa54c005210 2024-03-18T20:28:04.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa552ffd700 1 -- 172.21.15.1:0/967435188 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55800fbe0 con 0x7fa54c005210 2024-03-18T20:28:04.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.944+0000 7fa568fce700 1 -- 172.21.15.1:0/967435188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa564144c00 con 0x7fa54c005210 2024-03-18T20:28:04.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.946+0000 7fa568fce700 1 -- 172.21.15.1:0/967435188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5640676b0 con 0x7fa54c005210 2024-03-18T20:28:04.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.946+0000 7fa552ffd700 1 -- 172.21.15.1:0/967435188 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa55800fd40 con 0x7fa54c005210 2024-03-18T20:28:04.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.946+0000 7fa552ffd700 1 --2- 172.21.15.1:0/967435188 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa54406adb0 0x7fa54406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:04.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.947+0000 7fa561d9b700 1 --2- 172.21.15.1:0/967435188 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa54406adb0 0x7fa54406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:04.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.947+0000 7fa552ffd700 1 -- 172.21.15.1:0/967435188 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(279..279 src has 1..279) v4 ==== 6094+0+0 (secure 0 0 0) 0x7fa558089a70 con 0x7fa54c005210 2024-03-18T20:28:04.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.947+0000 7fa561d9b700 1 --2- 172.21.15.1:0/967435188 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa54406adb0 0x7fa54406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa564068bb0 tx=0x7fa554009040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:04.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:04.951+0000 7fa552ffd700 1 -- 172.21.15.1:0/967435188 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa558054a50 con 0x7fa54c005210 2024-03-18T20:28:05.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.197+0000 7fa568fce700 1 -- 172.21.15.1:0/967435188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa56415d280 con 0x7fa54c005210 2024-03-18T20:28:05.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.198+0000 7fa552ffd700 1 -- 172.21.15.1:0/967435188 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+3537 (secure 0 0 0) 0x7fa558058070 con 0x7fa54c005210 2024-03-18T20:28:05.202 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:28:05.202 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":138,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9938,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/1320583552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1320583552},{"type":"v1","addr":"172.21.15.165:6833","nonce":1320583552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131},{"gid":11566,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2669900315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2669900315},{"type":"v1","addr":"172.21.15.165:6835","nonce":2669900315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131},{"gid":13170,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2726723271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2726723271},{"type":"v1","addr":"172.21.15.1:6835","nonce":2726723271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":131},{"gid":13704,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2173936171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":2173936171},{"type":"v1","addr":"172.21.15.1:6837","nonce":2173936171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138}],"filesystems":[]} 2024-03-18T20:28:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.200+0000 7fa568fce700 1 -- 172.21.15.1:0/967435188 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa54406adb0 msgr2=0x7fa54406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.200+0000 7fa568fce700 1 --2- 172.21.15.1:0/967435188 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa54406adb0 0x7fa54406d270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa564068bb0 tx=0x7fa554009040 comp rx=0 tx=0).stop 2024-03-18T20:28:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.200+0000 7fa568fce700 1 -- 172.21.15.1:0/967435188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa54c005210 msgr2=0x7fa56413f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.200+0000 7fa568fce700 1 --2- 172.21.15.1:0/967435188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa54c005210 0x7fa56413f500 secure :-1 s=READY pgs=3241 cs=0 l=1 rev1=1 crypto rx=0x7fa5580119d0 tx=0x7fa558011ce0 comp rx=0 tx=0).stop 2024-03-18T20:28:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.200+0000 7fa568fce700 1 -- 172.21.15.1:0/967435188 shutdown_connections 2024-03-18T20:28:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.200+0000 7fa568fce700 1 --2- 172.21.15.1:0/967435188 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa54406adb0 0x7fa54406d270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.200+0000 7fa568fce700 1 --2- 172.21.15.1:0/967435188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa564145df0 0x7fa5641462a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.200+0000 7fa568fce700 1 --2- 172.21.15.1:0/967435188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa54c005210 0x7fa56413f500 unknown :-1 s=CLOSED pgs=3241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.200+0000 7fa568fce700 1 --2- 172.21.15.1:0/967435188 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa5640ccf00 0x7fa5641458b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.200+0000 7fa568fce700 1 -- 172.21.15.1:0/967435188 >> 172.21.15.1:0/967435188 conn(0x7fa5640b2ea0 msgr2=0x7fa564160140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.200+0000 7fa568fce700 1 -- 172.21.15.1:0/967435188 shutdown_connections 2024-03-18T20:28:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.200+0000 7fa568fce700 1 -- 172.21.15.1:0/967435188 wait complete. 2024-03-18T20:28:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 138 2024-03-18T20:28:05.217 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T20:28:05.217 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-18T20:28:05.217 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T20:28:05.217 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:28:05.217 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T20:28:05.217 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-18T20:28:05.218 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-18T20:28:05.218 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:28:05.218 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T20:28:05.218 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-18T20:28:05.218 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-18T20:28:05.218 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:28:05.218 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T20:28:05.219 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-18T20:28:05.219 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-18T20:28:05.219 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:28:05.228 INFO:tasks.ceph.mds.a:Stopped 2024-03-18T20:28:05.228 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-18T20:28:05.229 INFO:tasks.ceph.mds.c:Stopped 2024-03-18T20:28:05.229 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-18T20:28:05.231 INFO:tasks.ceph.mds.d:Stopped 2024-03-18T20:28:05.231 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-18T20:28:05.231 INFO:tasks.ceph.mds.b:Stopped 2024-03-18T20:28:05.232 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-18T20:28:05.271 INFO:tasks.ceph.mds.a:Started 2024-03-18T20:28:05.272 DEBUG:teuthology.parallel:result is None 2024-03-18T20:28:05.273 INFO:tasks.ceph.mds.c:Started 2024-03-18T20:28:05.273 DEBUG:teuthology.parallel:result is None 2024-03-18T20:28:05.274 INFO:tasks.ceph.mds.d:Started 2024-03-18T20:28:05.274 DEBUG:teuthology.parallel:result is None 2024-03-18T20:28:05.274 INFO:tasks.ceph.mds.b:Started 2024-03-18T20:28:05.275 DEBUG:teuthology.parallel:result is None 2024-03-18T20:28:05.275 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-18T20:28:05.353 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-03-18T20:28:05.353 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-18T20:28:05.360 INFO:tasks.ceph.mds.d.smithi165.stdout:starting mds.d at 2024-03-18T20:28:05.360 INFO:tasks.ceph.mds.b.smithi165.stdout:starting mds.b at 2024-03-18T20:28:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.404+0000 7f6275249700 1 Processor -- start 2024-03-18T20:28:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.404+0000 7f6275249700 1 -- start start 2024-03-18T20:28:05.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.405+0000 7f6275249700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6270157ce0 0x7f62700729e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:05.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.405+0000 7f6275249700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f627015a790 0x7f627015e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:05.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.405+0000 7f6275249700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f627015afc0 con 0x7f627015e9f0 2024-03-18T20:28:05.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.405+0000 7f6275249700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f627015dd80 con 0x7f6270157ce0 2024-03-18T20:28:05.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.405+0000 7f6275249700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f627015b2a0 con 0x7f627015a790 2024-03-18T20:28:05.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.405+0000 7f626e59c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f627015a790 0x7f627015e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:05.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.405+0000 7f626ed9d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6270157ce0 0x7f62700729e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:05.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.405+0000 7f626e59c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f627015a790 0x7f627015e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:48416/0 (socket says 172.21.15.1:48416) 2024-03-18T20:28:05.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.405+0000 7f626e59c700 1 -- 172.21.15.1:0/1022131607 learned_addr learned my addr 172.21.15.1:0/1022131607 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:05.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.405+0000 7f626dd9b700 1 -- 172.21.15.1:0/1022131607 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934877046 0 0) 0x7f627015afc0 con 0x7f627015e9f0 2024-03-18T20:28:05.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.405+0000 7f626dd9b700 1 -- 172.21.15.1:0/1022131607 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f625c0035b0 con 0x7f627015e9f0 2024-03-18T20:28:05.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.405+0000 7f626e59c700 1 -- 172.21.15.1:0/1022131607 >> v2:172.21.15.165:3300/0 conn(0x7f6270157ce0 msgr2=0x7f62700729e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:05.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.405+0000 7f626e59c700 1 --2- 172.21.15.1:0/1022131607 >> v2:172.21.15.165:3300/0 conn(0x7f6270157ce0 0x7f62700729e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:05.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.405+0000 7f626e59c700 1 -- 172.21.15.1:0/1022131607 >> v1:172.21.15.165:6789/0 conn(0x7f627015e9f0 legacy=0x7f62700b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:05.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.405+0000 7f626e59c700 1 -- 172.21.15.1:0/1022131607 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627015b580 con 0x7f627015a790 2024-03-18T20:28:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.405+0000 7f626ed9d700 1 --2- 172.21.15.1:0/1022131607 >> v2:172.21.15.165:3300/0 conn(0x7f6270157ce0 0x7f62700729e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:28:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.406+0000 7f626e59c700 1 --2- 172.21.15.1:0/1022131607 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f627015a790 0x7f627015e4b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f62580098f0 tx=0x7f6258009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e9c5d2b3ce3124 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.406+0000 7f626dd9b700 1 -- 172.21.15.1:0/1022131607 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6258011830 con 0x7f627015a790 2024-03-18T20:28:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.406+0000 7f626dd9b700 1 -- 172.21.15.1:0/1022131607 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6258011990 con 0x7f627015a790 2024-03-18T20:28:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.406+0000 7f626dd9b700 1 -- 172.21.15.1:0/1022131607 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6258011d20 con 0x7f627015a790 2024-03-18T20:28:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.406+0000 7f6275249700 1 -- 172.21.15.1:0/1022131607 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f627015a790 msgr2=0x7f627015e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.406+0000 7f6275249700 1 --2- 172.21.15.1:0/1022131607 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f627015a790 0x7f627015e4b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f62580098f0 tx=0x7f6258009c00 comp rx=0 tx=0).stop 2024-03-18T20:28:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.406+0000 7f6275249700 1 -- 172.21.15.1:0/1022131607 shutdown_connections 2024-03-18T20:28:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.406+0000 7f6275249700 1 --2- 172.21.15.1:0/1022131607 >> v2:172.21.15.165:3300/0 conn(0x7f6270157ce0 0x7f62700729e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.406+0000 7f6275249700 1 --2- 172.21.15.1:0/1022131607 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f627015a790 0x7f627015e4b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.406+0000 7f6275249700 1 -- 172.21.15.1:0/1022131607 >> 172.21.15.1:0/1022131607 conn(0x7f627006a9b0 msgr2=0x7f627006adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.406+0000 7f6275249700 1 -- 172.21.15.1:0/1022131607 shutdown_connections 2024-03-18T20:28:05.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.406+0000 7f6275249700 1 -- 172.21.15.1:0/1022131607 wait complete. 2024-03-18T20:28:05.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.408+0000 7f6275249700 1 Processor -- start 2024-03-18T20:28:05.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.408+0000 7f6275249700 1 -- start start 2024-03-18T20:28:05.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f6275249700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6270157ce0 0x7f627014c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:05.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f6275249700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f627015a790 0x7f6270156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f6275249700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f627015e9f0 0x7f6270155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f6275249700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62701556e0 con 0x7f627015e9f0 2024-03-18T20:28:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f6275249700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62700b15c0 con 0x7f6270157ce0 2024-03-18T20:28:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f6275249700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62700b18a0 con 0x7f627015a790 2024-03-18T20:28:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f626f59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f627015e9f0 0x7f6270155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f626e59c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f627015a790 0x7f6270156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f626f59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f627015e9f0 0x7f6270155b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54186/0 (socket says 172.21.15.1:54186) 2024-03-18T20:28:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f626f59e700 1 -- 172.21.15.1:0/4222266630 learned_addr learned my addr 172.21.15.1:0/4222266630 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f626ed9d700 1 --2- 172.21.15.1:0/4222266630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6270157ce0 0x7f627014c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f626f59e700 1 -- 172.21.15.1:0/4222266630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f627015a790 msgr2=0x7f6270156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f626f59e700 1 --2- 172.21.15.1:0/4222266630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f627015a790 0x7f6270156d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f626f59e700 1 -- 172.21.15.1:0/4222266630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6270157ce0 msgr2=0x7f627014c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f626f59e700 1 --2- 172.21.15.1:0/4222266630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6270157ce0 0x7f627014c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:05.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f626f59e700 1 -- 172.21.15.1:0/4222266630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6268009a70 con 0x7f627015e9f0 2024-03-18T20:28:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f626f59e700 1 --2- 172.21.15.1:0/4222266630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f627015e9f0 0x7f6270155b70 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7f626000c430 tx=0x7f626000c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f62677fe700 1 -- 172.21.15.1:0/4222266630 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62600050d0 con 0x7f627015e9f0 2024-03-18T20:28:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f6275249700 1 -- 172.21.15.1:0/4222266630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6258009550 con 0x7f627015e9f0 2024-03-18T20:28:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f62677fe700 1 -- 172.21.15.1:0/4222266630 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6260005250 con 0x7f627015e9f0 2024-03-18T20:28:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.409+0000 7f62677fe700 1 -- 172.21.15.1:0/4222266630 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6260003680 con 0x7f627015e9f0 2024-03-18T20:28:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.410+0000 7f6275249700 1 -- 172.21.15.1:0/4222266630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f627015ad90 con 0x7f627015e9f0 2024-03-18T20:28:05.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.411+0000 7f6275249700 1 -- 172.21.15.1:0/4222266630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62700ad120 con 0x7f627015e9f0 2024-03-18T20:28:05.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.412+0000 7f62677fe700 1 -- 172.21.15.1:0/4222266630 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f62600037e0 con 0x7f627015e9f0 2024-03-18T20:28:05.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.412+0000 7f62677fe700 1 --2- 172.21.15.1:0/4222266630 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f625006adb0 0x7f625006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:05.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.412+0000 7f62677fe700 1 -- 172.21.15.1:0/4222266630 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(283..283 src has 1..283) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f6260088dc0 con 0x7f627015e9f0 2024-03-18T20:28:05.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.412+0000 7f626ed9d700 1 --2- 172.21.15.1:0/4222266630 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f625006adb0 0x7f625006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:05.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.412+0000 7f626ed9d700 1 --2- 172.21.15.1:0/4222266630 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f625006adb0 0x7f625006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6268009fd0 tx=0x7f626800b040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:05.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.416+0000 7f62677fe700 1 -- 172.21.15.1:0/4222266630 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6260053d10 con 0x7f627015e9f0 2024-03-18T20:28:05.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:05.634+0000 7f6275249700 1 -- 172.21.15.1:0/4222266630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f62700ad400 con 0x7f627015e9f0 2024-03-18T20:28:06.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.591+0000 7f62677fe700 1 -- 172.21.15.1:0/4222266630 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v284) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6260017080 con 0x7f627015e9f0 2024-03-18T20:28:06.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.594+0000 7f6275249700 1 -- 172.21.15.1:0/4222266630 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f625006adb0 msgr2=0x7f625006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:06.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.594+0000 7f6275249700 1 --2- 172.21.15.1:0/4222266630 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f625006adb0 0x7f625006d270 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6268009fd0 tx=0x7f626800b040 comp rx=0 tx=0).stop 2024-03-18T20:28:06.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.594+0000 7f6275249700 1 -- 172.21.15.1:0/4222266630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f627015e9f0 msgr2=0x7f6270155b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:06.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.594+0000 7f6275249700 1 --2- 172.21.15.1:0/4222266630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f627015e9f0 0x7f6270155b70 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7f626000c430 tx=0x7f626000c740 comp rx=0 tx=0).stop 2024-03-18T20:28:06.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.594+0000 7f6275249700 1 -- 172.21.15.1:0/4222266630 shutdown_connections 2024-03-18T20:28:06.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.594+0000 7f6275249700 1 --2- 172.21.15.1:0/4222266630 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f625006adb0 0x7f625006d270 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:06.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.594+0000 7f6275249700 1 --2- 172.21.15.1:0/4222266630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f627015e9f0 0x7f6270155b70 unknown :-1 s=CLOSED pgs=3248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:06.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.594+0000 7f6275249700 1 --2- 172.21.15.1:0/4222266630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6270157ce0 0x7f627014c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:06.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.594+0000 7f6275249700 1 --2- 172.21.15.1:0/4222266630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f627015a790 0x7f6270156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:06.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.594+0000 7f6275249700 1 -- 172.21.15.1:0/4222266630 >> 172.21.15.1:0/4222266630 conn(0x7f627006a9b0 msgr2=0x7f62700719e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:06.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.594+0000 7f6275249700 1 -- 172.21.15.1:0/4222266630 shutdown_connections 2024-03-18T20:28:06.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.594+0000 7f6275249700 1 -- 172.21.15.1:0/4222266630 wait complete. 2024-03-18T20:28:06.600 INFO:teuthology.orchestra.run.smithi001.stderr: removed all blocklist entries 2024-03-18T20:28:06.614 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-18T20:28:06.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.741+0000 7f17f5144700 1 Processor -- start 2024-03-18T20:28:06.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.741+0000 7f17f5144700 1 -- start start 2024-03-18T20:28:06.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.742+0000 7f17f5144700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f17f015b4f0 0x7f17f00b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:06.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.742+0000 7f17f5144700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f0071fe0 con 0x7f17f00b0f50 2024-03-18T20:28:06.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.742+0000 7f17f5144700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f00af930 con 0x7f17f00ad750 2024-03-18T20:28:06.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.742+0000 7f17f5144700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17f00ad5e0 con 0x7f17f015b4f0 2024-03-18T20:28:06.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.742+0000 7f17ef59e700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f17f00b0f50 0x7f17f00b1380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54090/0 (socket says 172.21.15.1:54090) 2024-03-18T20:28:06.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.742+0000 7f17eed9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f17f015b4f0 0x7f17f00b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:06.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.742+0000 7f17ef59e700 1 -- 172.21.15.1:0/455512776 learned_addr learned my addr 172.21.15.1:0/455512776 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:06.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.743+0000 7f17eed9d700 1 -- 172.21.15.1:0/455512776 >> v1:172.21.15.165:6789/0 conn(0x7f17f00ad750 legacy=0x7f17f00adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:06.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.743+0000 7f17eed9d700 1 -- 172.21.15.1:0/455512776 >> v1:172.21.15.1:6789/0 conn(0x7f17f00b0f50 legacy=0x7f17f00b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:06.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.743+0000 7f17edd9b700 1 -- 172.21.15.1:0/455512776 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066797523 0 0) 0x7f17f0071fe0 con 0x7f17f00b0f50 2024-03-18T20:28:06.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.743+0000 7f17eed9d700 1 -- 172.21.15.1:0/455512776 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f00afbd0 con 0x7f17f015b4f0 2024-03-18T20:28:06.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.743+0000 7f17eed9d700 1 --2- 172.21.15.1:0/455512776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f015b4f0 0x7f17f00b26e0 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009d90 tx=0x7f17e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d483e071e9fca16e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:06.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.743+0000 7f17edd9b700 1 -- 172.21.15.1:0/455512776 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e000e070 con 0x7f17f015b4f0 2024-03-18T20:28:06.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.743+0000 7f17edd9b700 1 -- 172.21.15.1:0/455512776 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e000bb90 con 0x7f17f015b4f0 2024-03-18T20:28:06.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.743+0000 7f17edd9b700 1 -- 172.21.15.1:0/455512776 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0003a20 con 0x7f17f015b4f0 2024-03-18T20:28:06.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.743+0000 7f17f5144700 1 -- 172.21.15.1:0/455512776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f015b4f0 msgr2=0x7f17f00b26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:06.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.743+0000 7f17f5144700 1 --2- 172.21.15.1:0/455512776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f015b4f0 0x7f17f00b26e0 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009d90 tx=0x7f17e000b330 comp rx=0 tx=0).stop 2024-03-18T20:28:06.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.743+0000 7f17f5144700 1 -- 172.21.15.1:0/455512776 shutdown_connections 2024-03-18T20:28:06.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.743+0000 7f17f5144700 1 --2- 172.21.15.1:0/455512776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f015b4f0 0x7f17f00b26e0 unknown :-1 s=CLOSED pgs=3250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:06.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.743+0000 7f17f5144700 1 -- 172.21.15.1:0/455512776 >> 172.21.15.1:0/455512776 conn(0x7f17f00b5a10 msgr2=0x7f17f00b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:06.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.744+0000 7f17f5144700 1 -- 172.21.15.1:0/455512776 shutdown_connections 2024-03-18T20:28:06.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.744+0000 7f17f5144700 1 -- 172.21.15.1:0/455512776 wait complete. 2024-03-18T20:28:06.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.745+0000 7f17f5144700 1 Processor -- start 2024-03-18T20:28:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.745+0000 7f17f5144700 1 -- start start 2024-03-18T20:28:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.746+0000 7f17f5144700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17f00ad750 0x7f17f0150a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.746+0000 7f17f5144700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17f00b0f50 0x7f17f01f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.746+0000 7f17f5144700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f015b4f0 0x7f17f01f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.746+0000 7f17f5144700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f0159060 con 0x7f17f015b4f0 2024-03-18T20:28:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.746+0000 7f17f5144700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f0159340 con 0x7f17f00b0f50 2024-03-18T20:28:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.746+0000 7f17f5144700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f0159620 con 0x7f17f00ad750 2024-03-18T20:28:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.746+0000 7f17ef59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f015b4f0 0x7f17f01f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.746+0000 7f17ef59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f015b4f0 0x7f17f01f4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54192/0 (socket says 172.21.15.1:54192) 2024-03-18T20:28:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.746+0000 7f17ef59e700 1 -- 172.21.15.1:0/681570034 learned_addr learned my addr 172.21.15.1:0/681570034 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.747+0000 7f17eed9d700 1 --2- 172.21.15.1:0/681570034 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17f00ad750 0x7f17f0150a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.747+0000 7f17ef59e700 1 -- 172.21.15.1:0/681570034 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17f00ad750 msgr2=0x7f17f0150a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:06.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.747+0000 7f17ef59e700 1 --2- 172.21.15.1:0/681570034 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17f00ad750 0x7f17f0150a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:06.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.747+0000 7f17ef59e700 1 -- 172.21.15.1:0/681570034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17f00b0f50 msgr2=0x7f17f01f3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:06.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.747+0000 7f17ef59e700 1 --2- 172.21.15.1:0/681570034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17f00b0f50 0x7f17f01f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:06.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.747+0000 7f17ef59e700 1 -- 172.21.15.1:0/681570034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e0009a70 con 0x7f17f015b4f0 2024-03-18T20:28:06.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.747+0000 7f17ef59e700 1 --2- 172.21.15.1:0/681570034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f015b4f0 0x7f17f01f4470 secure :-1 s=READY pgs=3251 cs=0 l=1 rev1=1 crypto rx=0x7f17e800c4a0 tx=0x7f17e800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:06.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.747+0000 7f17e77fe700 1 -- 172.21.15.1:0/681570034 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e8004d40 con 0x7f17f015b4f0 2024-03-18T20:28:06.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.747+0000 7f17e77fe700 1 -- 172.21.15.1:0/681570034 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e80070d0 con 0x7f17f015b4f0 2024-03-18T20:28:06.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.747+0000 7f17f5144700 1 -- 172.21.15.1:0/681570034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f0159900 con 0x7f17f015b4f0 2024-03-18T20:28:06.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.747+0000 7f17e77fe700 1 -- 172.21.15.1:0/681570034 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e8004ea0 con 0x7f17f015b4f0 2024-03-18T20:28:06.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.747+0000 7f17f5144700 1 -- 172.21.15.1:0/681570034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f0150f80 con 0x7f17f015b4f0 2024-03-18T20:28:06.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.748+0000 7f17f5144700 1 -- 172.21.15.1:0/681570034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f0159e20 con 0x7f17f015b4f0 2024-03-18T20:28:06.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.749+0000 7f17e77fe700 1 -- 172.21.15.1:0/681570034 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f17e8007480 con 0x7f17f015b4f0 2024-03-18T20:28:06.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.749+0000 7f17e77fe700 1 --2- 172.21.15.1:0/681570034 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f17d006adb0 0x7f17d006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:06.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.749+0000 7f17e77fe700 1 -- 172.21.15.1:0/681570034 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f17e8088f30 con 0x7f17f015b4f0 2024-03-18T20:28:06.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.749+0000 7f17eed9d700 1 --2- 172.21.15.1:0/681570034 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f17d006adb0 0x7f17d006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:06.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.749+0000 7f17eed9d700 1 --2- 172.21.15.1:0/681570034 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f17d006adb0 0x7f17d006d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f17e0000f80 tx=0x7f17e0011000 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:06.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:06.754+0000 7f17e77fe700 1 -- 172.21.15.1:0/681570034 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f17f0159e20 con 0x7f17f015b4f0 2024-03-18T20:28:07.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.037+0000 7f17f5144700 1 -- 172.21.15.1:0/681570034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f17f0159e20 con 0x7f17f015b4f0 2024-03-18T20:28:07.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.038+0000 7f17e77fe700 1 -- 172.21.15.1:0/681570034 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v53) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f17f0159e20 con 0x7f17f015b4f0 2024-03-18T20:28:07.042 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:28:07.042 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:28:07.042 INFO:teuthology.orchestra.run.smithi001.stdout: "auth_dump": [ 2024-03-18T20:28:07.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:07.042 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.a", 2024-03-18T20:28:07.042 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlR93/CxAAbpe+5n3wHRur9xi8QzdCnQ==", 2024-03-18T20:28:07.042 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:28:07.042 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:28:07.042 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:28:07.043 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:28:07.043 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:28:07.043 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:07.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:07.043 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.b", 2024-03-18T20:28:07.043 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJ3OmGBAA37u227nxrMsh/JHt71HBxQ==", 2024-03-18T20:28:07.043 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:28:07.043 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:28:07.043 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:28:07.043 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:28:07.043 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:28:07.043 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:07.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:07.044 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.c", 2024-03-18T20:28:07.044 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlJZFoExAAmI16oxwNKbf8dKetPfyk+g==", 2024-03-18T20:28:07.044 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:28:07.044 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:28:07.044 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:28:07.044 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:28:07.044 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:28:07.044 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.044 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:07.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:07.044 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mds.d", 2024-03-18T20:28:07.044 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlQrcvIBAA1LMv+EgC7S6wspX6QF49Pg==", 2024-03-18T20:28:07.044 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:28:07.044 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:28:07.045 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:28:07.045 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:28:07.045 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:28:07.045 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:07.045 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:07.045 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.0", 2024-03-18T20:28:07.045 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCUjPhlYOVHChAARFQrksZmMIdTlntd8TgvrQ==", 2024-03-18T20:28:07.045 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:28:07.045 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:28:07.045 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:28:07.045 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:28:07.045 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:07.045 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:07.046 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.1", 2024-03-18T20:28:07.046 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCXjPhl2zSDCBAA6Al1gkWVB79ZI9djGpVChA==", 2024-03-18T20:28:07.046 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:28:07.046 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:28:07.046 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:28:07.046 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:28:07.046 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:07.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:07.046 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.2", 2024-03-18T20:28:07.046 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCajPhlSoG/BhAAR93USDe6DCQW+hREOsnTlw==", 2024-03-18T20:28:07.046 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:28:07.046 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:28:07.046 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:28:07.047 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:28:07.047 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:07.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:07.047 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.3", 2024-03-18T20:28:07.047 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCdjPhlhJg5BRAAWuQhcnxyjR5UOiM13yKdAQ==", 2024-03-18T20:28:07.047 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:28:07.047 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:28:07.047 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:28:07.047 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:28:07.047 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:07.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:07.047 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.4", 2024-03-18T20:28:07.048 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCjjPhlCFnpCRAAKIyOQyVdjXZUTnAS6Nw/ng==", 2024-03-18T20:28:07.048 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:28:07.048 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:28:07.048 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:28:07.048 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:28:07.048 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:07.048 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:07.048 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.5", 2024-03-18T20:28:07.048 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCmjPhlAeFkCBAAJcEFpRk/03lQJxj+l2hEPA==", 2024-03-18T20:28:07.048 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:28:07.048 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:28:07.048 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:28:07.048 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:28:07.048 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:07.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:07.049 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.6", 2024-03-18T20:28:07.049 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCpjPhl/xuuBhAA0LgKfXEvN2O11ZlRQJVOkw==", 2024-03-18T20:28:07.049 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:28:07.049 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:28:07.049 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:28:07.049 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:28:07.049 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:07.049 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:07.049 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "osd.7", 2024-03-18T20:28:07.049 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCsjPhldiR4BRAA571oh6U8PzOv/4LzjYLF/A==", 2024-03-18T20:28:07.049 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:28:07.050 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow profile osd", 2024-03-18T20:28:07.050 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile osd", 2024-03-18T20:28:07.050 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:28:07.050 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.050 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:07.050 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:07.050 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.0", 2024-03-18T20:28:07.050 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlu+2xJBAANnXGGjfeqP6S2FTecsfOsg==", 2024-03-18T20:28:07.050 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:28:07.050 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:28:07.050 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow r", 2024-03-18T20:28:07.050 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-18T20:28:07.050 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.050 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:07.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:07.051 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.1", 2024-03-18T20:28:07.051 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlhJ9+KxAAEnrQrCrgJR6xI+pucwzAXA==", 2024-03-18T20:28:07.051 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:28:07.051 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:28:07.051 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:28:07.051 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:28:07.051 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:28:07.051 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:07.051 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:07.051 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.2", 2024-03-18T20:28:07.051 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlxBUpKBAAIGTwmLWyj+P2Yvl55QVcWA==", 2024-03-18T20:28:07.051 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:28:07.052 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:28:07.052 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:28:07.052 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:28:07.052 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:28:07.052 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.052 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:07.052 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:07.052 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.3", 2024-03-18T20:28:07.052 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlmao3MRAAHeiQ6eyBh9QjjKPCtxkC8g==", 2024-03-18T20:28:07.052 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:28:07.052 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow", 2024-03-18T20:28:07.052 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow r", 2024-03-18T20:28:07.052 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow rw", 2024-03-18T20:28:07.052 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow rwx" 2024-03-18T20:28:07.052 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:07.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:07.053 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "client.admin", 2024-03-18T20:28:07.053 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCRjPhlemOuKRAAVUggmtB7+EiVgyfJZ8TCCA==", 2024-03-18T20:28:07.053 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:28:07.053 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:28:07.053 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": "allow *", 2024-03-18T20:28:07.053 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow *", 2024-03-18T20:28:07.053 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:28:07.053 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:07.053 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:07.053 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.x", 2024-03-18T20:28:07.053 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhleIuOCBAA9CH+AHewQbn+gpVr+6lJyw==", 2024-03-18T20:28:07.053 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:28:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:28:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T20:28:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:28:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: "entity": "mgr.y", 2024-03-18T20:28:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: "key": "AQCSjPhlL0pTBBAAdFiPcdwwz514cuLAvioxYA==", 2024-03-18T20:28:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": { 2024-03-18T20:28:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": "allow *", 2024-03-18T20:28:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": "allow profile mgr", 2024-03-18T20:28:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": "allow *" 2024-03-18T20:28:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.054 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:07.055 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-18T20:28:07.055 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:28:07.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.040+0000 7f17f5144700 1 -- 172.21.15.1:0/681570034 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f17d006adb0 msgr2=0x7f17d006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:07.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.040+0000 7f17f5144700 1 --2- 172.21.15.1:0/681570034 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f17d006adb0 0x7f17d006d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f17e0000f80 tx=0x7f17e0011000 comp rx=0 tx=0).stop 2024-03-18T20:28:07.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.040+0000 7f17f5144700 1 -- 172.21.15.1:0/681570034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f015b4f0 msgr2=0x7f17f01f4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:07.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.040+0000 7f17f5144700 1 --2- 172.21.15.1:0/681570034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f015b4f0 0x7f17f01f4470 secure :-1 s=READY pgs=3251 cs=0 l=1 rev1=1 crypto rx=0x7f17e800c4a0 tx=0x7f17e800c7b0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.040+0000 7f17f5144700 1 -- 172.21.15.1:0/681570034 shutdown_connections 2024-03-18T20:28:07.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.040+0000 7f17f5144700 1 --2- 172.21.15.1:0/681570034 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f17d006adb0 0x7f17d006d270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.040+0000 7f17f5144700 1 --2- 172.21.15.1:0/681570034 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f17f00ad750 0x7f17f0150a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.040+0000 7f17f5144700 1 --2- 172.21.15.1:0/681570034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f015b4f0 0x7f17f01f4470 unknown :-1 s=CLOSED pgs=3251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.040+0000 7f17f5144700 1 --2- 172.21.15.1:0/681570034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17f00b0f50 0x7f17f01f3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.040+0000 7f17f5144700 1 -- 172.21.15.1:0/681570034 >> 172.21.15.1:0/681570034 conn(0x7f17f00b5a10 msgr2=0x7f17f00aef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:07.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.040+0000 7f17f5144700 1 -- 172.21.15.1:0/681570034 shutdown_connections 2024-03-18T20:28:07.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.040+0000 7f17f5144700 1 -- 172.21.15.1:0/681570034 wait complete. 2024-03-18T20:28:07.058 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:28:07.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.179+0000 7fa835f45700 1 Processor -- start 2024-03-18T20:28:07.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.179+0000 7fa835f45700 1 -- start start 2024-03-18T20:28:07.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.179+0000 7fa835f45700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa83006fc60 0x7fa830099a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.180+0000 7fa835f45700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa830099f70 0x7fa8300b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.180+0000 7fa835f45700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa8300b4fc0 0x7fa830161110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.180+0000 7fa835f45700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa83009a3f0 con 0x7fa83006fc60 2024-03-18T20:28:07.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.180+0000 7fa835f45700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8300b5800 con 0x7fa8300b4fc0 2024-03-18T20:28:07.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.180+0000 7fa835f45700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8300b5ae0 con 0x7fa830099f70 2024-03-18T20:28:07.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.180+0000 7fa82f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa83006fc60 0x7fa830099a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.180+0000 7fa82f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa83006fc60 0x7fa830099a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54204/0 (socket says 172.21.15.1:54204) 2024-03-18T20:28:07.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.180+0000 7fa82f7fe700 1 -- 172.21.15.1:0/1445028218 learned_addr learned my addr 172.21.15.1:0/1445028218 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:07.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.180+0000 7fa82ffff700 1 --2- 172.21.15.1:0/1445028218 >> v2:172.21.15.165:3300/0 conn(0x7fa8300b4fc0 0x7fa830161110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.180+0000 7fa82effd700 1 --2- 172.21.15.1:0/1445028218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa830099f70 0x7fa8300b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.180+0000 7fa82f7fe700 1 -- 172.21.15.1:0/1445028218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa830099f70 msgr2=0x7fa8300b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:07.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.180+0000 7fa82f7fe700 1 --2- 172.21.15.1:0/1445028218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa830099f70 0x7fa8300b4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.180+0000 7fa82f7fe700 1 -- 172.21.15.1:0/1445028218 >> v2:172.21.15.165:3300/0 conn(0x7fa8300b4fc0 msgr2=0x7fa830161110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:07.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.180+0000 7fa82f7fe700 1 --2- 172.21.15.1:0/1445028218 >> v2:172.21.15.165:3300/0 conn(0x7fa8300b4fc0 0x7fa830161110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.180+0000 7fa82f7fe700 1 -- 172.21.15.1:0/1445028218 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8300b5dc0 con 0x7fa83006fc60 2024-03-18T20:28:07.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.180+0000 7fa82f7fe700 1 --2- 172.21.15.1:0/1445028218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa83006fc60 0x7fa830099a30 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 crypto rx=0x7fa830070960 tx=0x7fa820009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3651ce4d1037c446 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:07.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.180+0000 7fa82e7fc700 1 -- 172.21.15.1:0/1445028218 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82000b870 con 0x7fa83006fc60 2024-03-18T20:28:07.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.180+0000 7fa82e7fc700 1 -- 172.21.15.1:0/1445028218 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82000b9d0 con 0x7fa83006fc60 2024-03-18T20:28:07.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.180+0000 7fa82e7fc700 1 -- 172.21.15.1:0/1445028218 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82000bd60 con 0x7fa83006fc60 2024-03-18T20:28:07.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.181+0000 7fa835f45700 1 -- 172.21.15.1:0/1445028218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa83006fc60 msgr2=0x7fa830099a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:07.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.181+0000 7fa835f45700 1 --2- 172.21.15.1:0/1445028218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa83006fc60 0x7fa830099a30 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 crypto rx=0x7fa830070960 tx=0x7fa820009bd0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.181+0000 7fa835f45700 1 -- 172.21.15.1:0/1445028218 shutdown_connections 2024-03-18T20:28:07.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.181+0000 7fa835f45700 1 --2- 172.21.15.1:0/1445028218 >> v2:172.21.15.165:3300/0 conn(0x7fa8300b4fc0 0x7fa830161110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.181+0000 7fa835f45700 1 --2- 172.21.15.1:0/1445028218 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa83006fc60 0x7fa830099a30 unknown :-1 s=CLOSED pgs=3252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.181+0000 7fa835f45700 1 --2- 172.21.15.1:0/1445028218 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa830099f70 0x7fa8300b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.181+0000 7fa835f45700 1 -- 172.21.15.1:0/1445028218 >> 172.21.15.1:0/1445028218 conn(0x7fa8300b2de0 msgr2=0x7fa8300b31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:07.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.181+0000 7fa835f45700 1 -- 172.21.15.1:0/1445028218 shutdown_connections 2024-03-18T20:28:07.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.181+0000 7fa835f45700 1 -- 172.21.15.1:0/1445028218 wait complete. 2024-03-18T20:28:07.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.182+0000 7fa835f45700 1 Processor -- start 2024-03-18T20:28:07.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.182+0000 7fa835f45700 1 -- start start 2024-03-18T20:28:07.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.183+0000 7fa835f45700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa83006fc60 0x7fa830143c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.183+0000 7fa835f45700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa830099f70 0x7fa830153f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.183+0000 7fa835f45700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8300b4fc0 0x7fa830152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.183+0000 7fa835f45700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa830152eb0 con 0x7fa830099f70 2024-03-18T20:28:07.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.183+0000 7fa835f45700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa830153190 con 0x7fa83006fc60 2024-03-18T20:28:07.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.183+0000 7fa835f45700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa830153470 con 0x7fa8300b4fc0 2024-03-18T20:28:07.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.183+0000 7fa82effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa830099f70 0x7fa830153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.183+0000 7fa82effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa830099f70 0x7fa830153f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54212/0 (socket says 172.21.15.1:54212) 2024-03-18T20:28:07.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.183+0000 7fa82effd700 1 -- 172.21.15.1:0/1046252739 learned_addr learned my addr 172.21.15.1:0/1046252739 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:07.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.183+0000 7fa82f7fe700 1 --2- 172.21.15.1:0/1046252739 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa83006fc60 0x7fa830143c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.183+0000 7fa82ffff700 1 --2- 172.21.15.1:0/1046252739 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8300b4fc0 0x7fa830152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.183+0000 7fa82effd700 1 -- 172.21.15.1:0/1046252739 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8300b4fc0 msgr2=0x7fa830152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:07.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.183+0000 7fa82effd700 1 --2- 172.21.15.1:0/1046252739 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8300b4fc0 0x7fa830152580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.183+0000 7fa82effd700 1 -- 172.21.15.1:0/1046252739 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa83006fc60 msgr2=0x7fa830143c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:07.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.183+0000 7fa82effd700 1 --2- 172.21.15.1:0/1046252739 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa83006fc60 0x7fa830143c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.184+0000 7fa82effd700 1 -- 172.21.15.1:0/1046252739 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa820009550 con 0x7fa830099f70 2024-03-18T20:28:07.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.184+0000 7fa82f7fe700 1 --2- 172.21.15.1:0/1046252739 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa83006fc60 0x7fa830143c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:28:07.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.184+0000 7fa82effd700 1 --2- 172.21.15.1:0/1046252739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa830099f70 0x7fa830153f50 secure :-1 s=READY pgs=3253 cs=0 l=1 rev1=1 crypto rx=0x7fa82400c7f0 tx=0x7fa82400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:07.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.184+0000 7fa817fff700 1 -- 172.21.15.1:0/1046252739 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa824011800 con 0x7fa830099f70 2024-03-18T20:28:07.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.184+0000 7fa817fff700 1 -- 172.21.15.1:0/1046252739 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa824011960 con 0x7fa830099f70 2024-03-18T20:28:07.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.184+0000 7fa835f45700 1 -- 172.21.15.1:0/1046252739 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8300b55d0 con 0x7fa830099f70 2024-03-18T20:28:07.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.184+0000 7fa817fff700 1 -- 172.21.15.1:0/1046252739 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa824011cf0 con 0x7fa830099f70 2024-03-18T20:28:07.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.184+0000 7fa835f45700 1 -- 172.21.15.1:0/1046252739 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa830153a30 con 0x7fa830099f70 2024-03-18T20:28:07.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.185+0000 7fa835f45700 1 -- 172.21.15.1:0/1046252739 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa830067dc0 con 0x7fa830099f70 2024-03-18T20:28:07.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.185+0000 7fa817fff700 1 -- 172.21.15.1:0/1046252739 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa824018360 con 0x7fa830099f70 2024-03-18T20:28:07.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.186+0000 7fa817fff700 1 --2- 172.21.15.1:0/1046252739 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa81c06adb0 0x7fa81c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.186+0000 7fa817fff700 1 -- 172.21.15.1:0/1046252739 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 5277+0+0 (secure 0 0 0) 0x7fa824089420 con 0x7fa830099f70 2024-03-18T20:28:07.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.186+0000 7fa82f7fe700 1 --2- 172.21.15.1:0/1046252739 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa81c06adb0 0x7fa81c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.186+0000 7fa82f7fe700 1 --2- 172.21.15.1:0/1046252739 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa81c06adb0 0x7fa81c06d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa820000c00 tx=0x7fa820008030 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:07.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.190+0000 7fa817fff700 1 -- 172.21.15.1:0/1046252739 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa8240545a0 con 0x7fa830099f70 2024-03-18T20:28:07.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.394+0000 7fa835f45700 1 -- 172.21.15.1:0/1046252739 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8300b01d0 con 0x7fa830099f70 2024-03-18T20:28:07.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.395+0000 7fa817fff700 1 -- 172.21.15.1:0/1046252739 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v284) v1 ==== 74+0+12717 (secure 0 0 0) 0x7fa824057bc0 con 0x7fa830099f70 2024-03-18T20:28:07.399 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:28:07.399 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":284,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:28:06.587772+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":36,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":272,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:28:07.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.397+0000 7fa835f45700 1 -- 172.21.15.1:0/1046252739 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa81c06adb0 msgr2=0x7fa81c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:07.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.397+0000 7fa835f45700 1 --2- 172.21.15.1:0/1046252739 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa81c06adb0 0x7fa81c06d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa820000c00 tx=0x7fa820008030 comp rx=0 tx=0).stop 2024-03-18T20:28:07.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.397+0000 7fa835f45700 1 -- 172.21.15.1:0/1046252739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa830099f70 msgr2=0x7fa830153f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:07.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.397+0000 7fa835f45700 1 --2- 172.21.15.1:0/1046252739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa830099f70 0x7fa830153f50 secure :-1 s=READY pgs=3253 cs=0 l=1 rev1=1 crypto rx=0x7fa82400c7f0 tx=0x7fa82400cbb0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.397+0000 7fa835f45700 1 -- 172.21.15.1:0/1046252739 shutdown_connections 2024-03-18T20:28:07.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.397+0000 7fa835f45700 1 --2- 172.21.15.1:0/1046252739 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa81c06adb0 0x7fa81c06d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.397+0000 7fa835f45700 1 --2- 172.21.15.1:0/1046252739 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8300b4fc0 0x7fa830152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.397+0000 7fa835f45700 1 --2- 172.21.15.1:0/1046252739 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa83006fc60 0x7fa830143c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.397+0000 7fa835f45700 1 --2- 172.21.15.1:0/1046252739 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa830099f70 0x7fa830153f50 unknown :-1 s=CLOSED pgs=3253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.397+0000 7fa835f45700 1 -- 172.21.15.1:0/1046252739 >> 172.21.15.1:0/1046252739 conn(0x7fa8300b2de0 msgr2=0x7fa8300d1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:07.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.397+0000 7fa835f45700 1 -- 172.21.15.1:0/1046252739 shutdown_connections 2024-03-18T20:28:07.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.397+0000 7fa835f45700 1 -- 172.21.15.1:0/1046252739 wait complete. 2024-03-18T20:28:07.414 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:28:07.414 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:28:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.538+0000 7f342efdd700 1 Processor -- start 2024-03-18T20:28:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.538+0000 7f342efdd700 1 -- start start 2024-03-18T20:28:07.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.539+0000 7f342efdd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34280b0f50 0x7f34280af680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.539+0000 7f342efdd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3428099a30 0x7f34280b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.539+0000 7f342efdd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f34280b1870 0x7f34280b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.539+0000 7f342efdd700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34280b1d00 con 0x7f3428099a30 2024-03-18T20:28:07.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.539+0000 7f342efdd700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34280b2a50 con 0x7f34280b1870 2024-03-18T20:28:07.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.539+0000 7f342efdd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34280aeba0 con 0x7f34280b0f50 2024-03-18T20:28:07.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.539+0000 7f3427fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3428099a30 0x7f34280b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.539+0000 7f3427fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3428099a30 0x7f34280b1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54218/0 (socket says 172.21.15.1:54218) 2024-03-18T20:28:07.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.539+0000 7f3427fff700 1 -- 172.21.15.1:0/1552363503 learned_addr learned my addr 172.21.15.1:0/1552363503 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:07.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.539+0000 7f342d57a700 1 --2- 172.21.15.1:0/1552363503 >> v2:172.21.15.165:3300/0 conn(0x7f34280b1870 0x7f34280b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.539+0000 7f342cd79700 1 --2- 172.21.15.1:0/1552363503 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34280b0f50 0x7f34280af680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.539+0000 7f3427fff700 1 -- 172.21.15.1:0/1552363503 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34280b0f50 msgr2=0x7f34280af680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:07.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.539+0000 7f3427fff700 1 --2- 172.21.15.1:0/1552363503 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34280b0f50 0x7f34280af680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.539+0000 7f3427fff700 1 -- 172.21.15.1:0/1552363503 >> v2:172.21.15.165:3300/0 conn(0x7f34280b1870 msgr2=0x7f34280b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.539+0000 7f3427fff700 1 --2- 172.21.15.1:0/1552363503 >> v2:172.21.15.165:3300/0 conn(0x7f34280b1870 0x7f34280b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.539+0000 7f3427fff700 1 -- 172.21.15.1:0/1552363503 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34280aee80 con 0x7f3428099a30 2024-03-18T20:28:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.540+0000 7f3427fff700 1 --2- 172.21.15.1:0/1552363503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3428099a30 0x7f34280b1330 secure :-1 s=READY pgs=3254 cs=0 l=1 rev1=1 crypto rx=0x7f34280712a0 tx=0x7f3414009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ed4781782e470d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.540+0000 7f34277fe700 1 -- 172.21.15.1:0/1552363503 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3414011830 con 0x7f3428099a30 2024-03-18T20:28:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.540+0000 7f34277fe700 1 -- 172.21.15.1:0/1552363503 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3414011990 con 0x7f3428099a30 2024-03-18T20:28:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.540+0000 7f34277fe700 1 -- 172.21.15.1:0/1552363503 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3414011d20 con 0x7f3428099a30 2024-03-18T20:28:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.540+0000 7f342efdd700 1 -- 172.21.15.1:0/1552363503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3428099a30 msgr2=0x7f34280b1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.540+0000 7f342efdd700 1 --2- 172.21.15.1:0/1552363503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3428099a30 0x7f34280b1330 secure :-1 s=READY pgs=3254 cs=0 l=1 rev1=1 crypto rx=0x7f34280712a0 tx=0x7f3414009bd0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.540+0000 7f342efdd700 1 -- 172.21.15.1:0/1552363503 shutdown_connections 2024-03-18T20:28:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.540+0000 7f342efdd700 1 --2- 172.21.15.1:0/1552363503 >> v2:172.21.15.165:3300/0 conn(0x7f34280b1870 0x7f34280b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.540+0000 7f342efdd700 1 --2- 172.21.15.1:0/1552363503 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34280b0f50 0x7f34280af680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.540+0000 7f342efdd700 1 --2- 172.21.15.1:0/1552363503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3428099a30 0x7f34280b1330 unknown :-1 s=CLOSED pgs=3254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.540+0000 7f342efdd700 1 -- 172.21.15.1:0/1552363503 >> 172.21.15.1:0/1552363503 conn(0x7f34280b5a10 msgr2=0x7f34280b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:07.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.540+0000 7f342efdd700 1 -- 172.21.15.1:0/1552363503 shutdown_connections 2024-03-18T20:28:07.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.540+0000 7f342efdd700 1 -- 172.21.15.1:0/1552363503 wait complete. 2024-03-18T20:28:07.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.542+0000 7f342efdd700 1 Processor -- start 2024-03-18T20:28:07.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.542+0000 7f342efdd700 1 -- start start 2024-03-18T20:28:07.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.543+0000 7f342efdd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3428099a30 0x7f34281e3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.543+0000 7f342efdd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34280b0f50 0x7f34281e4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.543+0000 7f342efdd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34280b1870 0x7f34281576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.543+0000 7f342efdd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34281e44c0 con 0x7f3428099a30 2024-03-18T20:28:07.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.543+0000 7f342efdd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34281e2ee0 con 0x7f34280b0f50 2024-03-18T20:28:07.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.543+0000 7f342efdd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34281e31c0 con 0x7f34280b1870 2024-03-18T20:28:07.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.543+0000 7f342cd79700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3428099a30 0x7f34281e3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.543+0000 7f342cd79700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3428099a30 0x7f34281e3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54220/0 (socket says 172.21.15.1:54220) 2024-03-18T20:28:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.543+0000 7f342cd79700 1 -- 172.21.15.1:0/1103904516 learned_addr learned my addr 172.21.15.1:0/1103904516 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.543+0000 7f3427fff700 1 --2- 172.21.15.1:0/1103904516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34280b0f50 0x7f34281e4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.543+0000 7f342d57a700 1 --2- 172.21.15.1:0/1103904516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34280b1870 0x7f34281576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.543+0000 7f342cd79700 1 -- 172.21.15.1:0/1103904516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34280b1870 msgr2=0x7f34281576e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.543+0000 7f342cd79700 1 --2- 172.21.15.1:0/1103904516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34280b1870 0x7f34281576e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.543+0000 7f342cd79700 1 -- 172.21.15.1:0/1103904516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34280b0f50 msgr2=0x7f34281e4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.543+0000 7f342cd79700 1 --2- 172.21.15.1:0/1103904516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34280b0f50 0x7f34281e4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.543+0000 7f342cd79700 1 -- 172.21.15.1:0/1103904516 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3414009550 con 0x7f3428099a30 2024-03-18T20:28:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.544+0000 7f342cd79700 1 --2- 172.21.15.1:0/1103904516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3428099a30 0x7f34281e3f80 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 crypto rx=0x7f341c00e930 tx=0x7f341c00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.544+0000 7f34257fa700 1 -- 172.21.15.1:0/1103904516 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c009690 con 0x7f3428099a30 2024-03-18T20:28:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.544+0000 7f34257fa700 1 -- 172.21.15.1:0/1103904516 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341c00c850 con 0x7f3428099a30 2024-03-18T20:28:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.544+0000 7f342efdd700 1 -- 172.21.15.1:0/1103904516 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34281e34a0 con 0x7f3428099a30 2024-03-18T20:28:07.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.544+0000 7f34257fa700 1 -- 172.21.15.1:0/1103904516 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c00cbe0 con 0x7f3428099a30 2024-03-18T20:28:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.544+0000 7f342efdd700 1 -- 172.21.15.1:0/1103904516 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34281e3a60 con 0x7f3428099a30 2024-03-18T20:28:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.545+0000 7f342efdd700 1 -- 172.21.15.1:0/1103904516 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3428067dc0 con 0x7f3428099a30 2024-03-18T20:28:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.545+0000 7f34257fa700 1 -- 172.21.15.1:0/1103904516 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f341c00cd40 con 0x7f3428099a30 2024-03-18T20:28:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.546+0000 7f34257fa700 1 --2- 172.21.15.1:0/1103904516 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f340c06adb0 0x7f340c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.546+0000 7f34257fa700 1 -- 172.21.15.1:0/1103904516 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f341c089300 con 0x7f3428099a30 2024-03-18T20:28:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.546+0000 7f3427fff700 1 --2- 172.21.15.1:0/1103904516 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f340c06adb0 0x7f340c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.546+0000 7f3427fff700 1 --2- 172.21.15.1:0/1103904516 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f340c06adb0 0x7f340c06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f34280712a0 tx=0x7f3414008030 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:07.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.550+0000 7f34257fa700 1 -- 172.21.15.1:0/1103904516 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f341c054610 con 0x7f3428099a30 2024-03-18T20:28:07.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.752+0000 7f342efdd700 1 -- 172.21.15.1:0/1103904516 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f34280cc5e0 con 0x7f3428099a30 2024-03-18T20:28:07.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.753+0000 7f34257fa700 1 -- 172.21.15.1:0/1103904516 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v284) v1 ==== 93+0+10 (secure 0 0 0) 0x7f341c057c30 con 0x7f3428099a30 2024-03-18T20:28:07.757 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:28:07.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.755+0000 7f342efdd700 1 -- 172.21.15.1:0/1103904516 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f340c06adb0 msgr2=0x7f340c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:07.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.755+0000 7f342efdd700 1 --2- 172.21.15.1:0/1103904516 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f340c06adb0 0x7f340c06d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f34280712a0 tx=0x7f3414008030 comp rx=0 tx=0).stop 2024-03-18T20:28:07.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.755+0000 7f342efdd700 1 -- 172.21.15.1:0/1103904516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3428099a30 msgr2=0x7f34281e3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:07.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.755+0000 7f342efdd700 1 --2- 172.21.15.1:0/1103904516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3428099a30 0x7f34281e3f80 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 crypto rx=0x7f341c00e930 tx=0x7f341c00ecf0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.755+0000 7f342efdd700 1 -- 172.21.15.1:0/1103904516 shutdown_connections 2024-03-18T20:28:07.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.755+0000 7f342efdd700 1 --2- 172.21.15.1:0/1103904516 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f340c06adb0 0x7f340c06d270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.755+0000 7f342efdd700 1 --2- 172.21.15.1:0/1103904516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f34280b1870 0x7f34281576e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.755+0000 7f342efdd700 1 --2- 172.21.15.1:0/1103904516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3428099a30 0x7f34281e3f80 unknown :-1 s=CLOSED pgs=3255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.755+0000 7f342efdd700 1 --2- 172.21.15.1:0/1103904516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34280b0f50 0x7f34281e4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.755+0000 7f342efdd700 1 -- 172.21.15.1:0/1103904516 >> 172.21.15.1:0/1103904516 conn(0x7f34280b5a10 msgr2=0x7f34280ce350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:07.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.755+0000 7f342efdd700 1 -- 172.21.15.1:0/1103904516 shutdown_connections 2024-03-18T20:28:07.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.755+0000 7f342efdd700 1 -- 172.21.15.1:0/1103904516 wait complete. 2024-03-18T20:28:07.772 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T20:28:07.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.893+0000 7fe34b347700 1 Processor -- start 2024-03-18T20:28:07.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.893+0000 7fe34b347700 1 -- start start 2024-03-18T20:28:07.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.894+0000 7fe34b347700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe344099a30 0x7fe3440b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.894+0000 7fe34b347700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe3440b1870 0x7fe3440b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.894+0000 7fe34b347700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3440afd90 con 0x7fe3440b0f50 2024-03-18T20:28:07.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.894+0000 7fe34b347700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3440b1c80 con 0x7fe3440b1870 2024-03-18T20:28:07.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.894+0000 7fe34b347700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3440b2a50 con 0x7fe344099a30 2024-03-18T20:28:07.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.894+0000 7fe3498e4700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe3440b1870 0x7fe3440b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.894+0000 7fe3490e3700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe3440b0f50 0x7fe3440af680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54106/0 (socket says 172.21.15.1:54106) 2024-03-18T20:28:07.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.894+0000 7fe3490e3700 1 -- 172.21.15.1:0/2352024138 learned_addr learned my addr 172.21.15.1:0/2352024138 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:07.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.895+0000 7fe3488e2700 1 --2- 172.21.15.1:0/2352024138 >> v2:172.21.15.165:3300/0 conn(0x7fe344099a30 0x7fe3440b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.895+0000 7fe3498e4700 1 -- 172.21.15.1:0/2352024138 >> v2:172.21.15.165:3300/0 conn(0x7fe344099a30 msgr2=0x7fe3440b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:07.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.895+0000 7fe3498e4700 1 --2- 172.21.15.1:0/2352024138 >> v2:172.21.15.165:3300/0 conn(0x7fe344099a30 0x7fe3440b1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.895+0000 7fe3498e4700 1 -- 172.21.15.1:0/2352024138 >> v1:172.21.15.1:6789/0 conn(0x7fe3440b0f50 legacy=0x7fe3440af680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:07.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.895+0000 7fe33bfff700 1 -- 172.21.15.1:0/2352024138 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038497105 0 0) 0x7fe3440afd90 con 0x7fe3440b0f50 2024-03-18T20:28:07.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.895+0000 7fe3498e4700 1 -- 172.21.15.1:0/2352024138 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3440aeba0 con 0x7fe3440b1870 2024-03-18T20:28:07.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.895+0000 7fe3498e4700 1 --2- 172.21.15.1:0/2352024138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3440b1870 0x7fe3440b32d0 secure :-1 s=READY pgs=3257 cs=0 l=1 rev1=1 crypto rx=0x7fe344071a30 tx=0x7fe33c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e483f93f96b54764 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:07.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.895+0000 7fe33bfff700 1 -- 172.21.15.1:0/2352024138 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c00e070 con 0x7fe3440b1870 2024-03-18T20:28:07.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.895+0000 7fe33bfff700 1 -- 172.21.15.1:0/2352024138 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33c00bbd0 con 0x7fe3440b1870 2024-03-18T20:28:07.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.895+0000 7fe33bfff700 1 -- 172.21.15.1:0/2352024138 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c012430 con 0x7fe3440b1870 2024-03-18T20:28:07.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.895+0000 7fe34b347700 1 -- 172.21.15.1:0/2352024138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3440b1870 msgr2=0x7fe3440b32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:07.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.895+0000 7fe34b347700 1 --2- 172.21.15.1:0/2352024138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3440b1870 0x7fe3440b32d0 secure :-1 s=READY pgs=3257 cs=0 l=1 rev1=1 crypto rx=0x7fe344071a30 tx=0x7fe33c00b330 comp rx=0 tx=0).stop 2024-03-18T20:28:07.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.895+0000 7fe34b347700 1 -- 172.21.15.1:0/2352024138 shutdown_connections 2024-03-18T20:28:07.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.895+0000 7fe34b347700 1 --2- 172.21.15.1:0/2352024138 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3440b1870 0x7fe3440b32d0 unknown :-1 s=CLOSED pgs=3257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.895+0000 7fe34b347700 1 --2- 172.21.15.1:0/2352024138 >> v2:172.21.15.165:3300/0 conn(0x7fe344099a30 0x7fe3440b1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.896+0000 7fe34b347700 1 -- 172.21.15.1:0/2352024138 >> 172.21.15.1:0/2352024138 conn(0x7fe3440b5a10 msgr2=0x7fe3440b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:07.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.896+0000 7fe34b347700 1 -- 172.21.15.1:0/2352024138 shutdown_connections 2024-03-18T20:28:07.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.896+0000 7fe34b347700 1 -- 172.21.15.1:0/2352024138 wait complete. 2024-03-18T20:28:07.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.897+0000 7fe34b347700 1 Processor -- start 2024-03-18T20:28:07.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.897+0000 7fe34b347700 1 -- start start 2024-03-18T20:28:07.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.897+0000 7fe34b347700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe344099a30 0x7fe3440cea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe34b347700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3440b0f50 0x7fe3440cef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe34b347700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3440b1870 0x7fe34406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe34b347700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3441f4600 con 0x7fe344099a30 2024-03-18T20:28:07.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe34b347700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3441f48e0 con 0x7fe3440b1870 2024-03-18T20:28:07.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe34b347700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3441f4bc0 con 0x7fe3440b0f50 2024-03-18T20:28:07.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe3490e3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe344099a30 0x7fe3440cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe3490e3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe344099a30 0x7fe3440cea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54228/0 (socket says 172.21.15.1:54228) 2024-03-18T20:28:07.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe3490e3700 1 -- 172.21.15.1:0/2823399920 learned_addr learned my addr 172.21.15.1:0/2823399920 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:07.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe3498e4700 1 --2- 172.21.15.1:0/2823399920 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3440b1870 0x7fe34406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe3488e2700 1 --2- 172.21.15.1:0/2823399920 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3440b0f50 0x7fe3440cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe3490e3700 1 -- 172.21.15.1:0/2823399920 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3440b0f50 msgr2=0x7fe3440cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:07.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe3490e3700 1 --2- 172.21.15.1:0/2823399920 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3440b0f50 0x7fe3440cef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe3490e3700 1 -- 172.21.15.1:0/2823399920 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3440b1870 msgr2=0x7fe34406a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:07.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe3490e3700 1 --2- 172.21.15.1:0/2823399920 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3440b1870 0x7fe34406a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:07.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe3490e3700 1 -- 172.21.15.1:0/2823399920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33c009a70 con 0x7fe344099a30 2024-03-18T20:28:07.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe3498e4700 1 --2- 172.21.15.1:0/2823399920 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3440b1870 0x7fe34406a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:28:07.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe3490e3700 1 --2- 172.21.15.1:0/2823399920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe344099a30 0x7fe3440cea10 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 crypto rx=0x7fe33400c550 tx=0x7fe33400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:07.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe339ffb700 1 -- 172.21.15.1:0/2823399920 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe334004c80 con 0x7fe344099a30 2024-03-18T20:28:07.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe339ffb700 1 -- 172.21.15.1:0/2823399920 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe334004e00 con 0x7fe344099a30 2024-03-18T20:28:07.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe34b347700 1 -- 172.21.15.1:0/2823399920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3441f4ea0 con 0x7fe344099a30 2024-03-18T20:28:07.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe339ffb700 1 -- 172.21.15.1:0/2823399920 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe334007310 con 0x7fe344099a30 2024-03-18T20:28:07.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.898+0000 7fe34b347700 1 -- 172.21.15.1:0/2823399920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3441f5460 con 0x7fe344099a30 2024-03-18T20:28:07.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.899+0000 7fe34b347700 1 -- 172.21.15.1:0/2823399920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe344067dc0 con 0x7fe344099a30 2024-03-18T20:28:07.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.900+0000 7fe339ffb700 1 -- 172.21.15.1:0/2823399920 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe334007470 con 0x7fe344099a30 2024-03-18T20:28:07.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.900+0000 7fe339ffb700 1 --2- 172.21.15.1:0/2823399920 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe32806adb0 0x7fe32806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:07.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.900+0000 7fe339ffb700 1 -- 172.21.15.1:0/2823399920 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 5277+0+0 (secure 0 0 0) 0x7fe334089070 con 0x7fe344099a30 2024-03-18T20:28:07.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.900+0000 7fe3488e2700 1 --2- 172.21.15.1:0/2823399920 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe32806adb0 0x7fe32806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:07.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.901+0000 7fe3488e2700 1 --2- 172.21.15.1:0/2823399920 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe32806adb0 0x7fe32806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe3441f4560 tx=0x7fe340008040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:07.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:07.905+0000 7fe339ffb700 1 -- 172.21.15.1:0/2823399920 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe3340542f0 con 0x7fe344099a30 2024-03-18T20:28:08.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.110+0000 7fe34b347700 1 -- 172.21.15.1:0/2823399920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe3440cc5e0 con 0x7fe344099a30 2024-03-18T20:28:08.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.110+0000 7fe339ffb700 1 -- 172.21.15.1:0/2823399920 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v284) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe334057910 con 0x7fe344099a30 2024-03-18T20:28:08.115 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:28:08.115 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T20:28:08.115 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:08.115 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-03-18T20:28:08.115 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-03-18T20:28:08.115 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:08.115 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T20:28:08.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.112+0000 7fe34b347700 1 -- 172.21.15.1:0/2823399920 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe32806adb0 msgr2=0x7fe32806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:08.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.112+0000 7fe34b347700 1 --2- 172.21.15.1:0/2823399920 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe32806adb0 0x7fe32806d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe3441f4560 tx=0x7fe340008040 comp rx=0 tx=0).stop 2024-03-18T20:28:08.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.112+0000 7fe34b347700 1 -- 172.21.15.1:0/2823399920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe344099a30 msgr2=0x7fe3440cea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:08.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.112+0000 7fe34b347700 1 --2- 172.21.15.1:0/2823399920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe344099a30 0x7fe3440cea10 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 crypto rx=0x7fe33400c550 tx=0x7fe33400c910 comp rx=0 tx=0).stop 2024-03-18T20:28:08.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.112+0000 7fe34b347700 1 -- 172.21.15.1:0/2823399920 shutdown_connections 2024-03-18T20:28:08.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.112+0000 7fe34b347700 1 --2- 172.21.15.1:0/2823399920 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe32806adb0 0x7fe32806d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:08.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.112+0000 7fe34b347700 1 --2- 172.21.15.1:0/2823399920 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3440b1870 0x7fe34406a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:08.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.112+0000 7fe34b347700 1 --2- 172.21.15.1:0/2823399920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe344099a30 0x7fe3440cea10 unknown :-1 s=CLOSED pgs=3258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:08.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.112+0000 7fe34b347700 1 --2- 172.21.15.1:0/2823399920 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe3440b0f50 0x7fe3440cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:08.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.112+0000 7fe34b347700 1 -- 172.21.15.1:0/2823399920 >> 172.21.15.1:0/2823399920 conn(0x7fe3440b5a10 msgr2=0x7fe3440b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:08.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.113+0000 7fe34b347700 1 -- 172.21.15.1:0/2823399920 shutdown_connections 2024-03-18T20:28:08.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.113+0000 7fe34b347700 1 -- 172.21.15.1:0/2823399920 wait complete. 2024-03-18T20:28:08.129 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T20:28:08.129 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T20:28:08.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.255+0000 7f8a069b3700 1 Processor -- start 2024-03-18T20:28:08.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.255+0000 7f8a069b3700 1 -- start start 2024-03-18T20:28:08.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.256+0000 7f8a069b3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a000b54a0 0x7f8a000b5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:08.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.256+0000 7f8a069b3700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a000aeef0 con 0x7f8a000acfe0 2024-03-18T20:28:08.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.256+0000 7f8a069b3700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a000ae320 con 0x7f8a000af380 2024-03-18T20:28:08.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.256+0000 7f8a069b3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a000b5e90 con 0x7f8a000b54a0 2024-03-18T20:28:08.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.256+0000 7f89fffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8a000acfe0 0x7f8a0015e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54110/0 (socket says 172.21.15.1:54110) 2024-03-18T20:28:08.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.256+0000 7f89fffff700 1 -- 172.21.15.1:0/99552949 learned_addr learned my addr 172.21.15.1:0/99552949 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:08.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.256+0000 7f89ff7fe700 1 -- 172.21.15.1:0/99552949 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3892530782 0 0) 0x7f8a000aeef0 con 0x7f8a000acfe0 2024-03-18T20:28:08.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.256+0000 7f8a04f50700 1 --2- 172.21.15.1:0/99552949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a000b54a0 0x7f8a000b5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:08.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.256+0000 7f89ff7fe700 1 -- 172.21.15.1:0/99552949 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89ec0035b0 con 0x7f8a000acfe0 2024-03-18T20:28:08.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f89ff7fe700 1 -- 172.21.15.1:0/99552949 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3773523275 0 0) 0x7f89ec0035b0 con 0x7f8a000acfe0 2024-03-18T20:28:08.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f89ff7fe700 1 -- 172.21.15.1:0/99552949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a000b54a0 msgr2=0x7f8a000b5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:08.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f89ff7fe700 1 --2- 172.21.15.1:0/99552949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a000b54a0 0x7f8a000b5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:08.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f89ff7fe700 1 -- 172.21.15.1:0/99552949 >> v1:172.21.15.165:6789/0 conn(0x7f8a000af380 legacy=0x7f8a000ad3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:08.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f89ff7fe700 1 -- 172.21.15.1:0/99552949 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a000ae5c0 con 0x7f8a000acfe0 2024-03-18T20:28:08.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f89ff7fe700 1 -- 172.21.15.1:0/99552949 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f89f0001670 con 0x7f8a000acfe0 2024-03-18T20:28:08.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f89ff7fe700 1 -- 172.21.15.1:0/99552949 >> v1:172.21.15.1:6789/0 conn(0x7f8a000acfe0 legacy=0x7f8a0015e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:08.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f89ff7fe700 1 --2- 172.21.15.1:0/99552949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89ec003a30 0x7f89ec003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:08.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f89ff7fe700 1 --2- 172.21.15.1:0/99552949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a000b54a0 0x7f89ec004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:08.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f89ff7fe700 1 --2- 172.21.15.1:0/99552949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89ec004d90 0x7f89ec005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:08.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f89ff7fe700 1 -- 172.21.15.1:0/99552949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a000b5e90 con 0x7f89ec004d90 2024-03-18T20:28:08.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f89ff7fe700 1 -- 172.21.15.1:0/99552949 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec00ba10 con 0x7f89ec003a30 2024-03-18T20:28:08.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f89ff7fe700 1 -- 172.21.15.1:0/99552949 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec00bbd0 con 0x7f8a000b54a0 2024-03-18T20:28:08.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f8a04f50700 1 --2- 172.21.15.1:0/99552949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89ec004d90 0x7f89ec005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:08.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f89fffff700 1 --2- 172.21.15.1:0/99552949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89ec003a30 0x7f89ec003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:08.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f89f7fff700 1 --2- 172.21.15.1:0/99552949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a000b54a0 0x7f89ec004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:08.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f8a04f50700 1 -- 172.21.15.1:0/99552949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a000b54a0 msgr2=0x7f89ec004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:08.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f8a04f50700 1 --2- 172.21.15.1:0/99552949 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a000b54a0 0x7f89ec004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:08.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f8a04f50700 1 -- 172.21.15.1:0/99552949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89ec003a30 msgr2=0x7f89ec003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:08.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f8a04f50700 1 --2- 172.21.15.1:0/99552949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89ec003a30 0x7f89ec003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:08.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f8a04f50700 1 -- 172.21.15.1:0/99552949 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ec0035b0 con 0x7f89ec004d90 2024-03-18T20:28:08.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.257+0000 7f8a04f50700 1 --2- 172.21.15.1:0/99552949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89ec004d90 0x7f89ec005210 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 crypto rx=0x7f89f8009fc0 tx=0x7f89ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9552fd84a1ebb71 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:08.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.258+0000 7f89ff7fe700 1 -- 172.21.15.1:0/99552949 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f800ec30 con 0x7f89ec004d90 2024-03-18T20:28:08.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.258+0000 7f89ff7fe700 1 -- 172.21.15.1:0/99552949 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f800f040 con 0x7f89ec004d90 2024-03-18T20:28:08.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.258+0000 7f89ff7fe700 1 -- 172.21.15.1:0/99552949 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f8014480 con 0x7f89ec004d90 2024-03-18T20:28:08.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.258+0000 7f8a069b3700 1 -- 172.21.15.1:0/99552949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89ec004d90 msgr2=0x7f89ec005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:08.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.258+0000 7f8a069b3700 1 --2- 172.21.15.1:0/99552949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89ec004d90 0x7f89ec005210 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 crypto rx=0x7f89f8009fc0 tx=0x7f89ec005880 comp rx=0 tx=0).stop 2024-03-18T20:28:08.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.258+0000 7f8a04f50700 1 -- 172.21.15.1:0/99552949 reap_dead start 2024-03-18T20:28:08.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.258+0000 7f8a069b3700 1 -- 172.21.15.1:0/99552949 shutdown_connections 2024-03-18T20:28:08.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.258+0000 7f8a069b3700 1 -- 172.21.15.1:0/99552949 >> 172.21.15.1:0/99552949 conn(0x7f8a0006a7d0 msgr2=0x7f8a0006abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:08.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.258+0000 7f8a069b3700 1 -- 172.21.15.1:0/99552949 shutdown_connections 2024-03-18T20:28:08.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.258+0000 7f8a069b3700 1 -- 172.21.15.1:0/99552949 wait complete. 2024-03-18T20:28:08.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.259+0000 7f8a069b3700 1 Processor -- start 2024-03-18T20:28:08.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.259+0000 7f8a069b3700 1 -- start start 2024-03-18T20:28:08.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.260+0000 7f8a069b3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a001466f0 0x7f8a00156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:08.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.260+0000 7f8a069b3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a000cc6c0 0x7f8a0015ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:08.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.260+0000 7f8a069b3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a00156f60 0x7f8a0015f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:08.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.260+0000 7f8a069b3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a000cd7d0 con 0x7f8a000cc6c0 2024-03-18T20:28:08.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.260+0000 7f8a069b3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a000ccb50 con 0x7f8a00156f60 2024-03-18T20:28:08.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.260+0000 7f8a069b3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a000cce30 con 0x7f8a001466f0 2024-03-18T20:28:08.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.260+0000 7f89f7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a000cc6c0 0x7f8a0015ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:08.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.260+0000 7f89f7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a000cc6c0 0x7f8a0015ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54252/0 (socket says 172.21.15.1:54252) 2024-03-18T20:28:08.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.260+0000 7f89f7fff700 1 -- 172.21.15.1:0/653184673 learned_addr learned my addr 172.21.15.1:0/653184673 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:08.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.260+0000 7f89fffff700 1 --2- 172.21.15.1:0/653184673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a001466f0 0x7f8a00156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:08.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.260+0000 7f8a04f50700 1 --2- 172.21.15.1:0/653184673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a00156f60 0x7f8a0015f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:08.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.261+0000 7f89f7fff700 1 -- 172.21.15.1:0/653184673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a001466f0 msgr2=0x7f8a00156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:08.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.261+0000 7f89f7fff700 1 --2- 172.21.15.1:0/653184673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a001466f0 0x7f8a00156a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:08.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.261+0000 7f89f7fff700 1 -- 172.21.15.1:0/653184673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a00156f60 msgr2=0x7f8a0015f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:08.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.261+0000 7f89f7fff700 1 --2- 172.21.15.1:0/653184673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a00156f60 0x7f8a0015f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:08.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.261+0000 7f89f7fff700 1 -- 172.21.15.1:0/653184673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a000cd110 con 0x7f8a000cc6c0 2024-03-18T20:28:08.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.261+0000 7f89f7fff700 1 --2- 172.21.15.1:0/653184673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a000cc6c0 0x7f8a0015ebe0 secure :-1 s=READY pgs=3261 cs=0 l=1 rev1=1 crypto rx=0x7f89e800f740 tx=0x7f89e800fa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:08.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.261+0000 7f89fd7fa700 1 -- 172.21.15.1:0/653184673 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e80052e0 con 0x7f8a000cc6c0 2024-03-18T20:28:08.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.261+0000 7f89fd7fa700 1 -- 172.21.15.1:0/653184673 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e8005460 con 0x7f8a000cc6c0 2024-03-18T20:28:08.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.261+0000 7f89fd7fa700 1 -- 172.21.15.1:0/653184673 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e80038c0 con 0x7f8a000cc6c0 2024-03-18T20:28:08.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.261+0000 7f8a069b3700 1 -- 172.21.15.1:0/653184673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a000cd3f0 con 0x7f8a000cc6c0 2024-03-18T20:28:08.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.261+0000 7f8a069b3700 1 -- 172.21.15.1:0/653184673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a00146cb0 con 0x7f8a000cc6c0 2024-03-18T20:28:08.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.262+0000 7f8a069b3700 1 -- 172.21.15.1:0/653184673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a00067dc0 con 0x7f8a000cc6c0 2024-03-18T20:28:08.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.262+0000 7f89fd7fa700 1 -- 172.21.15.1:0/653184673 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f89e8003a20 con 0x7f8a000cc6c0 2024-03-18T20:28:08.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.263+0000 7f89fd7fa700 1 --2- 172.21.15.1:0/653184673 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f89e006adb0 0x7f89e006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:08.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.263+0000 7f89fffff700 1 --2- 172.21.15.1:0/653184673 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f89e006adb0 0x7f89e006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:08.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.263+0000 7f89fd7fa700 1 -- 172.21.15.1:0/653184673 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(284..284 src has 1..284) v4 ==== 5277+0+0 (secure 0 0 0) 0x7f89e8088fd0 con 0x7f8a000cc6c0 2024-03-18T20:28:08.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.263+0000 7f89fffff700 1 --2- 172.21.15.1:0/653184673 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f89e006adb0 0x7f89e006d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f89f0001050 tx=0x7f89f0000f80 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:08.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.268+0000 7f89fd7fa700 1 -- 172.21.15.1:0/653184673 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f89e80542e0 con 0x7f8a000cc6c0 2024-03-18T20:28:08.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:08.468+0000 7f8a069b3700 1 -- 172.21.15.1:0/653184673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8a000acad0 con 0x7f8a000cc6c0 2024-03-18T20:28:09.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.234+0000 7f89fd7fa700 1 -- 172.21.15.1:0/653184673 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v285) v1 ==== 138+0+0 (secure 0 0 0) 0x7f89e8057900 con 0x7f8a000cc6c0 2024-03-18T20:28:09.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.238+0000 7f8a069b3700 1 -- 172.21.15.1:0/653184673 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f89e006adb0 msgr2=0x7f89e006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:09.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.238+0000 7f8a069b3700 1 --2- 172.21.15.1:0/653184673 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f89e006adb0 0x7f89e006d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f89f0001050 tx=0x7f89f0000f80 comp rx=0 tx=0).stop 2024-03-18T20:28:09.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.238+0000 7f8a069b3700 1 -- 172.21.15.1:0/653184673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a000cc6c0 msgr2=0x7f8a0015ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:09.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.238+0000 7f8a069b3700 1 --2- 172.21.15.1:0/653184673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a000cc6c0 0x7f8a0015ebe0 secure :-1 s=READY pgs=3261 cs=0 l=1 rev1=1 crypto rx=0x7f89e800f740 tx=0x7f89e800fa50 comp rx=0 tx=0).stop 2024-03-18T20:28:09.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.238+0000 7f8a069b3700 1 -- 172.21.15.1:0/653184673 shutdown_connections 2024-03-18T20:28:09.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.238+0000 7f8a069b3700 1 --2- 172.21.15.1:0/653184673 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f89e006adb0 0x7f89e006d270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:09.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.238+0000 7f8a069b3700 1 --2- 172.21.15.1:0/653184673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8a001466f0 0x7f8a00156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.238+0000 7f8a069b3700 1 --2- 172.21.15.1:0/653184673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a00156f60 0x7f8a0015f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.238+0000 7f8a069b3700 1 --2- 172.21.15.1:0/653184673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a000cc6c0 0x7f8a0015ebe0 unknown :-1 s=CLOSED pgs=3261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.238+0000 7f8a069b3700 1 -- 172.21.15.1:0/653184673 >> 172.21.15.1:0/653184673 conn(0x7f8a0006a7d0 msgr2=0x7f8a0006b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.238+0000 7f8a069b3700 1 -- 172.21.15.1:0/653184673 shutdown_connections 2024-03-18T20:28:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.238+0000 7f8a069b3700 1 -- 172.21.15.1:0/653184673 wait complete. 2024-03-18T20:28:09.243 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-03-18T20:28:09.261 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T20:28:09.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.396+0000 7fc6355f7700 1 Processor -- start 2024-03-18T20:28:09.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.396+0000 7fc6355f7700 1 -- start start 2024-03-18T20:28:09.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.397+0000 7fc6355f7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6300d1bc0 0x7fc6300d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.397+0000 7fc6355f7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc63015e3e0 0x7fc6300cde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.397+0000 7fc6355f7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6301618d0 con 0x7fc6300d1fa0 2024-03-18T20:28:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.397+0000 7fc6355f7700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc63015b670 con 0x7fc63015e3e0 2024-03-18T20:28:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.397+0000 7fc6355f7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc630161bb0 con 0x7fc6300d1bc0 2024-03-18T20:28:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.397+0000 7fc62f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc63015e3e0 0x7fc6300cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.397+0000 7fc62f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc63015e3e0 0x7fc6300cde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54254/0 (socket says 172.21.15.1:54254) 2024-03-18T20:28:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.397+0000 7fc62e7fc700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc6300d1fa0 0x7fc6301610e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54126/0 (socket says 172.21.15.1:54126) 2024-03-18T20:28:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.397+0000 7fc62f7fe700 1 -- 172.21.15.1:0/1892895284 learned_addr learned my addr 172.21.15.1:0/1892895284 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.397+0000 7fc62effd700 1 --2- 172.21.15.1:0/1892895284 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6300d1bc0 0x7fc6300d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.397+0000 7fc62f7fe700 1 -- 172.21.15.1:0/1892895284 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6300d1bc0 msgr2=0x7fc6300d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.397+0000 7fc62f7fe700 1 --2- 172.21.15.1:0/1892895284 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6300d1bc0 0x7fc6300d1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:09.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.398+0000 7fc62f7fe700 1 -- 172.21.15.1:0/1892895284 >> v1:172.21.15.1:6789/0 conn(0x7fc6300d1fa0 legacy=0x7fc6301610e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.398+0000 7fc62f7fe700 1 -- 172.21.15.1:0/1892895284 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc630161e90 con 0x7fc63015e3e0 2024-03-18T20:28:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.398+0000 7fc62dffb700 1 -- 172.21.15.1:0/1892895284 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1151033678 0 0) 0x7fc6301618d0 con 0x7fc6300d1fa0 2024-03-18T20:28:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.398+0000 7fc62effd700 1 --2- 172.21.15.1:0/1892895284 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6300d1bc0 0x7fc6300d1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:28:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.398+0000 7fc62f7fe700 1 --2- 172.21.15.1:0/1892895284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63015e3e0 0x7fc6300cde80 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7fc628009e80 tx=0x7fc62800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=258baf1dd4fd60d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.398+0000 7fc62dffb700 1 -- 172.21.15.1:0/1892895284 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62800e070 con 0x7fc63015e3e0 2024-03-18T20:28:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.398+0000 7fc62dffb700 1 -- 172.21.15.1:0/1892895284 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62800b980 con 0x7fc63015e3e0 2024-03-18T20:28:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.398+0000 7fc62dffb700 1 -- 172.21.15.1:0/1892895284 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62800bd30 con 0x7fc63015e3e0 2024-03-18T20:28:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.398+0000 7fc6355f7700 1 -- 172.21.15.1:0/1892895284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63015e3e0 msgr2=0x7fc6300cde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.398+0000 7fc6355f7700 1 --2- 172.21.15.1:0/1892895284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63015e3e0 0x7fc6300cde80 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7fc628009e80 tx=0x7fc62800b330 comp rx=0 tx=0).stop 2024-03-18T20:28:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.398+0000 7fc6355f7700 1 -- 172.21.15.1:0/1892895284 shutdown_connections 2024-03-18T20:28:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.398+0000 7fc6355f7700 1 --2- 172.21.15.1:0/1892895284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc63015e3e0 0x7fc6300cde80 unknown :-1 s=CLOSED pgs=3263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.398+0000 7fc6355f7700 1 --2- 172.21.15.1:0/1892895284 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6300d1bc0 0x7fc6300d1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.398+0000 7fc6355f7700 1 -- 172.21.15.1:0/1892895284 >> 172.21.15.1:0/1892895284 conn(0x7fc63008afc0 msgr2=0x7fc6300d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.398+0000 7fc6355f7700 1 -- 172.21.15.1:0/1892895284 shutdown_connections 2024-03-18T20:28:09.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.399+0000 7fc6355f7700 1 -- 172.21.15.1:0/1892895284 wait complete. 2024-03-18T20:28:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.400+0000 7fc6355f7700 1 Processor -- start 2024-03-18T20:28:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.400+0000 7fc6355f7700 1 -- start start 2024-03-18T20:28:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc6355f7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300d1bc0 0x7fc6301562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc6355f7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6300d1fa0 0x7fc630150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc6355f7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc63015e3e0 0x7fc630156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc6355f7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6300cba90 con 0x7fc6300d1bc0 2024-03-18T20:28:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc6355f7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6300cbc20 con 0x7fc63015e3e0 2024-03-18T20:28:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc6355f7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6301511e0 con 0x7fc6300d1fa0 2024-03-18T20:28:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc62effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300d1bc0 0x7fc6301562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc62effd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300d1bc0 0x7fc6301562d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54268/0 (socket says 172.21.15.1:54268) 2024-03-18T20:28:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc62effd700 1 -- 172.21.15.1:0/890447634 learned_addr learned my addr 172.21.15.1:0/890447634 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:09.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc62e7fc700 1 --2- 172.21.15.1:0/890447634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6300d1fa0 0x7fc630150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc62f7fe700 1 --2- 172.21.15.1:0/890447634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc63015e3e0 0x7fc630156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc62effd700 1 -- 172.21.15.1:0/890447634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6300d1fa0 msgr2=0x7fc630150ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc62effd700 1 --2- 172.21.15.1:0/890447634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6300d1fa0 0x7fc630150ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc62effd700 1 -- 172.21.15.1:0/890447634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc63015e3e0 msgr2=0x7fc630156aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc62effd700 1 --2- 172.21.15.1:0/890447634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc63015e3e0 0x7fc630156aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc62effd700 1 -- 172.21.15.1:0/890447634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc628009b30 con 0x7fc6300d1bc0 2024-03-18T20:28:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc62f7fe700 1 --2- 172.21.15.1:0/890447634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc63015e3e0 0x7fc630156aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:28:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc62effd700 1 --2- 172.21.15.1:0/890447634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300d1bc0 0x7fc6301562d0 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 crypto rx=0x7fc62400e840 tx=0x7fc62400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc6177fe700 1 -- 172.21.15.1:0/890447634 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc624013610 con 0x7fc6300d1bc0 2024-03-18T20:28:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc6177fe700 1 -- 172.21.15.1:0/890447634 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc624013770 con 0x7fc6300d1bc0 2024-03-18T20:28:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc6355f7700 1 -- 172.21.15.1:0/890447634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc630156fe0 con 0x7fc6300d1bc0 2024-03-18T20:28:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc6177fe700 1 -- 172.21.15.1:0/890447634 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc624013b20 con 0x7fc6300d1bc0 2024-03-18T20:28:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.401+0000 7fc6355f7700 1 -- 172.21.15.1:0/890447634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc630160730 con 0x7fc6300d1bc0 2024-03-18T20:28:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.402+0000 7fc6355f7700 1 -- 172.21.15.1:0/890447634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6300676b0 con 0x7fc6300d1bc0 2024-03-18T20:28:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.403+0000 7fc6177fe700 1 -- 172.21.15.1:0/890447634 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc624013c80 con 0x7fc6300d1bc0 2024-03-18T20:28:09.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.403+0000 7fc6177fe700 1 --2- 172.21.15.1:0/890447634 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc61c06adb0 0x7fc61c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:09.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.404+0000 7fc6177fe700 1 -- 172.21.15.1:0/890447634 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(285..285 src has 1..285) v4 ==== 5647+0+0 (secure 0 0 0) 0x7fc624089570 con 0x7fc6300d1bc0 2024-03-18T20:28:09.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.404+0000 7fc62e7fc700 1 --2- 172.21.15.1:0/890447634 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc61c06adb0 0x7fc61c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:09.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.404+0000 7fc62e7fc700 1 --2- 172.21.15.1:0/890447634 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc61c06adb0 0x7fc61c06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc6300cfbc0 tx=0x7fc620007be0 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:09.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.408+0000 7fc6177fe700 1 -- 172.21.15.1:0/890447634 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc6240546e0 con 0x7fc6300d1bc0 2024-03-18T20:28:09.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:09.610+0000 7fc6355f7700 1 -- 172.21.15.1:0/890447634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc6300d0a90 con 0x7fc6300d1bc0 2024-03-18T20:28:10.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.243+0000 7fc6177fe700 1 -- 172.21.15.1:0/890447634 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v286) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc624057d00 con 0x7fc6300d1bc0 2024-03-18T20:28:10.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.250+0000 7fc6355f7700 1 -- 172.21.15.1:0/890447634 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc61c06adb0 msgr2=0x7fc61c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:10.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.250+0000 7fc6355f7700 1 --2- 172.21.15.1:0/890447634 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc61c06adb0 0x7fc61c06d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc6300cfbc0 tx=0x7fc620007be0 comp rx=0 tx=0).stop 2024-03-18T20:28:10.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.250+0000 7fc6355f7700 1 -- 172.21.15.1:0/890447634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300d1bc0 msgr2=0x7fc6301562d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:10.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.250+0000 7fc6355f7700 1 --2- 172.21.15.1:0/890447634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300d1bc0 0x7fc6301562d0 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 crypto rx=0x7fc62400e840 tx=0x7fc62400ec00 comp rx=0 tx=0).stop 2024-03-18T20:28:10.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.251+0000 7fc6355f7700 1 -- 172.21.15.1:0/890447634 shutdown_connections 2024-03-18T20:28:10.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.251+0000 7fc6355f7700 1 --2- 172.21.15.1:0/890447634 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc61c06adb0 0x7fc61c06d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:10.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.251+0000 7fc6355f7700 1 --2- 172.21.15.1:0/890447634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc63015e3e0 0x7fc630156aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:10.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.251+0000 7fc6355f7700 1 --2- 172.21.15.1:0/890447634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300d1bc0 0x7fc6301562d0 secure :-1 s=CLOSED pgs=3264 cs=0 l=1 rev1=1 crypto rx=0x7fc62400e840 tx=0x7fc62400ec00 comp rx=0 tx=0).stop 2024-03-18T20:28:10.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.251+0000 7fc6355f7700 1 --2- 172.21.15.1:0/890447634 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc6300d1fa0 0x7fc630150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:10.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.251+0000 7fc6355f7700 1 -- 172.21.15.1:0/890447634 >> 172.21.15.1:0/890447634 conn(0x7fc63008afc0 msgr2=0x7fc6300d1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.253+0000 7fc6355f7700 1 -- 172.21.15.1:0/890447634 shutdown_connections 2024-03-18T20:28:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.255+0000 7fc6355f7700 1 -- 172.21.15.1:0/890447634 wait complete. 2024-03-18T20:28:10.261 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-03-18T20:28:10.277 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T20:28:10.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.407+0000 7f45e97eb700 1 Processor -- start 2024-03-18T20:28:10.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.407+0000 7f45e97eb700 1 -- start start 2024-03-18T20:28:10.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.408+0000 7f45e97eb700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f45e40b5300 0x7f45e4166080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:10.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.408+0000 7f45e97eb700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45e415ed10 con 0x7f45e415e930 2024-03-18T20:28:10.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.408+0000 7f45e97eb700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45e409ad80 con 0x7f45e40b0590 2024-03-18T20:28:10.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.408+0000 7f45e97eb700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45e40b47f0 con 0x7f45e40b5300 2024-03-18T20:28:10.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.408+0000 7f45e37fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f45e40b5300 0x7f45e4166080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:10.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.408+0000 7f45e2ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f45e415e930 0x7f45e40b40e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54130/0 (socket says 172.21.15.1:54130) 2024-03-18T20:28:10.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.408+0000 7f45e2ffd700 1 -- 172.21.15.1:0/748185868 learned_addr learned my addr 172.21.15.1:0/748185868 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:10.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.408+0000 7f45e37fe700 1 -- 172.21.15.1:0/748185868 >> v1:172.21.15.165:6789/0 conn(0x7f45e40b0590 legacy=0x7f45e40b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:10.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.408+0000 7f45e37fe700 1 -- 172.21.15.1:0/748185868 >> v1:172.21.15.1:6789/0 conn(0x7f45e415e930 legacy=0x7f45e40b40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:10.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.408+0000 7f45e37fe700 1 -- 172.21.15.1:0/748185868 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e409a4e0 con 0x7f45e40b5300 2024-03-18T20:28:10.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.408+0000 7f45e1ffb700 1 -- 172.21.15.1:0/748185868 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3750290842 0 0) 0x7f45d4001110 con 0x7f45e415e930 2024-03-18T20:28:10.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.408+0000 7f45e37fe700 1 --2- 172.21.15.1:0/748185868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45e40b5300 0x7f45e4166080 secure :-1 s=READY pgs=3266 cs=0 l=1 rev1=1 crypto rx=0x7f45dc009d90 tx=0x7f45dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d11f670984e75bb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:10.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.408+0000 7f45e1ffb700 1 -- 172.21.15.1:0/748185868 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc00e070 con 0x7f45e40b5300 2024-03-18T20:28:10.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.408+0000 7f45e1ffb700 1 -- 172.21.15.1:0/748185868 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45dc00bb90 con 0x7f45e40b5300 2024-03-18T20:28:10.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.408+0000 7f45e1ffb700 1 -- 172.21.15.1:0/748185868 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc003a20 con 0x7f45e40b5300 2024-03-18T20:28:10.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.409+0000 7f45e97eb700 1 -- 172.21.15.1:0/748185868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45e40b5300 msgr2=0x7f45e4166080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:10.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.409+0000 7f45e97eb700 1 --2- 172.21.15.1:0/748185868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45e40b5300 0x7f45e4166080 secure :-1 s=READY pgs=3266 cs=0 l=1 rev1=1 crypto rx=0x7f45dc009d90 tx=0x7f45dc00b330 comp rx=0 tx=0).stop 2024-03-18T20:28:10.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.409+0000 7f45e97eb700 1 -- 172.21.15.1:0/748185868 shutdown_connections 2024-03-18T20:28:10.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.409+0000 7f45e97eb700 1 --2- 172.21.15.1:0/748185868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45e40b5300 0x7f45e4166080 unknown :-1 s=CLOSED pgs=3266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:10.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.409+0000 7f45e97eb700 1 -- 172.21.15.1:0/748185868 >> 172.21.15.1:0/748185868 conn(0x7f45e40b2de0 msgr2=0x7f45e40b31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:10.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.409+0000 7f45e97eb700 1 -- 172.21.15.1:0/748185868 shutdown_connections 2024-03-18T20:28:10.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.409+0000 7f45e97eb700 1 -- 172.21.15.1:0/748185868 wait complete. 2024-03-18T20:28:10.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.410+0000 7f45e97eb700 1 Processor -- start 2024-03-18T20:28:10.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.410+0000 7f45e97eb700 1 -- start start 2024-03-18T20:28:10.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.411+0000 7f45e97eb700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45e40b0590 0x7f45e414dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:10.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.411+0000 7f45e97eb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45e40b5300 0x7f45e40cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:10.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.411+0000 7f45e97eb700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45e415e930 0x7f45e40cb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:10.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.411+0000 7f45e97eb700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e406c300 con 0x7f45e40b5300 2024-03-18T20:28:10.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.411+0000 7f45e97eb700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e406c580 con 0x7f45e415e930 2024-03-18T20:28:10.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.411+0000 7f45e97eb700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45e406c860 con 0x7f45e40b0590 2024-03-18T20:28:10.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.411+0000 7f45e27fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45e40b5300 0x7f45e40cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:10.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.411+0000 7f45e27fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45e40b5300 0x7f45e40cae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54282/0 (socket says 172.21.15.1:54282) 2024-03-18T20:28:10.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.411+0000 7f45e27fc700 1 -- 172.21.15.1:0/3259516755 learned_addr learned my addr 172.21.15.1:0/3259516755 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:10.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.411+0000 7f45e37fe700 1 --2- 172.21.15.1:0/3259516755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45e415e930 0x7f45e40cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:10.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.411+0000 7f45e2ffd700 1 --2- 172.21.15.1:0/3259516755 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45e40b0590 0x7f45e414dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:10.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.411+0000 7f45e27fc700 1 -- 172.21.15.1:0/3259516755 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45e40b0590 msgr2=0x7f45e414dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:10.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.411+0000 7f45e27fc700 1 --2- 172.21.15.1:0/3259516755 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45e40b0590 0x7f45e414dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:10.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.411+0000 7f45e27fc700 1 -- 172.21.15.1:0/3259516755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45e415e930 msgr2=0x7f45e40cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:10.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.411+0000 7f45e27fc700 1 --2- 172.21.15.1:0/3259516755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45e415e930 0x7f45e40cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:10.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.411+0000 7f45e27fc700 1 -- 172.21.15.1:0/3259516755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45dc009a70 con 0x7f45e40b5300 2024-03-18T20:28:10.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.412+0000 7f45e27fc700 1 --2- 172.21.15.1:0/3259516755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45e40b5300 0x7f45e40cae50 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 crypto rx=0x7f45d800c4d0 tx=0x7f45d800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:10.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.412+0000 7f45cb7fe700 1 -- 172.21.15.1:0/3259516755 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d8011790 con 0x7f45e40b5300 2024-03-18T20:28:10.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.412+0000 7f45cb7fe700 1 -- 172.21.15.1:0/3259516755 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d80118f0 con 0x7f45e40b5300 2024-03-18T20:28:10.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.412+0000 7f45e97eb700 1 -- 172.21.15.1:0/3259516755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e406cb40 con 0x7f45e40b5300 2024-03-18T20:28:10.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.412+0000 7f45cb7fe700 1 -- 172.21.15.1:0/3259516755 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d8011ca0 con 0x7f45e40b5300 2024-03-18T20:28:10.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.412+0000 7f45e97eb700 1 -- 172.21.15.1:0/3259516755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e406d100 con 0x7f45e40b5300 2024-03-18T20:28:10.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.413+0000 7f45e97eb700 1 -- 172.21.15.1:0/3259516755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e4073be0 con 0x7f45e40b5300 2024-03-18T20:28:10.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.414+0000 7f45cb7fe700 1 -- 172.21.15.1:0/3259516755 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f45d801c460 con 0x7f45e40b5300 2024-03-18T20:28:10.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.414+0000 7f45cb7fe700 1 --2- 172.21.15.1:0/3259516755 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f45d006adb0 0x7f45d006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:10.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.414+0000 7f45cb7fe700 1 -- 172.21.15.1:0/3259516755 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(286..286 src has 1..286) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f45d8089360 con 0x7f45e40b5300 2024-03-18T20:28:10.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.414+0000 7f45e2ffd700 1 --2- 172.21.15.1:0/3259516755 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f45d006adb0 0x7f45d006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:10.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.415+0000 7f45e2ffd700 1 --2- 172.21.15.1:0/3259516755 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f45d006adb0 0x7f45d006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f45d4000f60 tx=0x7f45d4007e70 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:10.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.419+0000 7f45cb7fe700 1 -- 172.21.15.1:0/3259516755 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f45d8054260 con 0x7f45e40b5300 2024-03-18T20:28:10.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.667+0000 7f45e97eb700 1 -- 172.21.15.1:0/3259516755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f45e4067dc0 con 0x7f45e40b5300 2024-03-18T20:28:10.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.672+0000 7f45cb7fe700 1 -- 172.21.15.1:0/3259516755 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 37 and data pool 38 v140) v1 ==== 174+0+0 (secure 0 0 0) 0x7f45d8057880 con 0x7f45e40b5300 2024-03-18T20:28:10.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.694+0000 7f45e97eb700 1 -- 172.21.15.1:0/3259516755 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f45d006adb0 msgr2=0x7f45d006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:10.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.694+0000 7f45e97eb700 1 --2- 172.21.15.1:0/3259516755 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f45d006adb0 0x7f45d006d270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f45d4000f60 tx=0x7f45d4007e70 comp rx=0 tx=0).stop 2024-03-18T20:28:10.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.694+0000 7f45e97eb700 1 -- 172.21.15.1:0/3259516755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45e40b5300 msgr2=0x7f45e40cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:10.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.694+0000 7f45e97eb700 1 --2- 172.21.15.1:0/3259516755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45e40b5300 0x7f45e40cae50 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 crypto rx=0x7f45d800c4d0 tx=0x7f45d800c7e0 comp rx=0 tx=0).stop 2024-03-18T20:28:10.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.695+0000 7f45e97eb700 1 -- 172.21.15.1:0/3259516755 shutdown_connections 2024-03-18T20:28:10.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.695+0000 7f45e97eb700 1 --2- 172.21.15.1:0/3259516755 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f45d006adb0 0x7f45d006d270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:10.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.695+0000 7f45e97eb700 1 --2- 172.21.15.1:0/3259516755 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f45e40b0590 0x7f45e414dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:10.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.695+0000 7f45e97eb700 1 --2- 172.21.15.1:0/3259516755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45e415e930 0x7f45e40cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:10.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.695+0000 7f45e97eb700 1 --2- 172.21.15.1:0/3259516755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45e40b5300 0x7f45e40cae50 unknown :-1 s=CLOSED pgs=3267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:10.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.695+0000 7f45e97eb700 1 -- 172.21.15.1:0/3259516755 >> 172.21.15.1:0/3259516755 conn(0x7f45e40b2de0 msgr2=0x7f45e40b4bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:10.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.699+0000 7f45e97eb700 1 -- 172.21.15.1:0/3259516755 shutdown_connections 2024-03-18T20:28:10.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.699+0000 7f45e97eb700 1 -- 172.21.15.1:0/3259516755 wait complete. 2024-03-18T20:28:10.704 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 37 and data pool 38 2024-03-18T20:28:10.716 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:28:10.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.838+0000 7fbfc1cdc700 1 Processor -- start 2024-03-18T20:28:10.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.838+0000 7fbfc1cdc700 1 -- start start 2024-03-18T20:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.839+0000 7fbfc1cdc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbfbc15a9b0 0x7fbfbc15ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.839+0000 7fbfc1cdc700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfbc0b3ba0 con 0x7fbfbc0b32d0 2024-03-18T20:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.839+0000 7fbfc1cdc700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfbc0b3e80 con 0x7fbfbc15bb30 2024-03-18T20:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.839+0000 7fbfc1cdc700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfbc0b4160 con 0x7fbfbc15a9b0 2024-03-18T20:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.839+0000 7fbfbb7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbfbc15a9b0 0x7fbfbc15ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.839+0000 7fbfb2ffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbfbc0b32d0 0x7fbfbc15b2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54142/0 (socket says 172.21.15.1:54142) 2024-03-18T20:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.839+0000 7fbfb2ffd700 1 -- 172.21.15.1:0/2027436642 learned_addr learned my addr 172.21.15.1:0/2027436642 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.840+0000 7fbfbb7fe700 1 -- 172.21.15.1:0/2027436642 >> v1:172.21.15.165:6789/0 conn(0x7fbfbc15bb30 legacy=0x7fbfbc15bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.840+0000 7fbfbb7fe700 1 -- 172.21.15.1:0/2027436642 >> v1:172.21.15.1:6789/0 conn(0x7fbfbc0b32d0 legacy=0x7fbfbc15b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.840+0000 7fbfbb7fe700 1 -- 172.21.15.1:0/2027436642 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfbc0b4440 con 0x7fbfbc15a9b0 2024-03-18T20:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.840+0000 7fbfbb7fe700 1 --2- 172.21.15.1:0/2027436642 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfbc15a9b0 0x7fbfbc15ad90 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7fbfac009840 tx=0x7fbfac009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab7ee48181bc095 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.840+0000 7fbfbaffd700 1 -- 172.21.15.1:0/2027436642 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac00f8a0 con 0x7fbfbc15a9b0 2024-03-18T20:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.840+0000 7fbfbaffd700 1 -- 172.21.15.1:0/2027436642 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfac00fa00 con 0x7fbfbc15a9b0 2024-03-18T20:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.840+0000 7fbfbaffd700 1 -- 172.21.15.1:0/2027436642 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac00fd90 con 0x7fbfbc15a9b0 2024-03-18T20:28:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.841+0000 7fbfc1cdc700 1 -- 172.21.15.1:0/2027436642 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfbc15a9b0 msgr2=0x7fbfbc15ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.841+0000 7fbfc1cdc700 1 --2- 172.21.15.1:0/2027436642 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfbc15a9b0 0x7fbfbc15ad90 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7fbfac009840 tx=0x7fbfac009b50 comp rx=0 tx=0).stop 2024-03-18T20:28:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.841+0000 7fbfc1cdc700 1 -- 172.21.15.1:0/2027436642 shutdown_connections 2024-03-18T20:28:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.841+0000 7fbfc1cdc700 1 --2- 172.21.15.1:0/2027436642 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfbc15a9b0 0x7fbfbc15ad90 unknown :-1 s=CLOSED pgs=3269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.841+0000 7fbfc1cdc700 1 -- 172.21.15.1:0/2027436642 >> 172.21.15.1:0/2027436642 conn(0x7fbfbc0b5780 msgr2=0x7fbfbc0b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.841+0000 7fbfc1cdc700 1 -- 172.21.15.1:0/2027436642 shutdown_connections 2024-03-18T20:28:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.841+0000 7fbfc1cdc700 1 -- 172.21.15.1:0/2027436642 wait complete. 2024-03-18T20:28:10.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.842+0000 7fbfc1cdc700 1 Processor -- start 2024-03-18T20:28:10.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.842+0000 7fbfc1cdc700 1 -- start start 2024-03-18T20:28:10.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.843+0000 7fbfc1cdc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfbc0b32d0 0x7fbfbc150bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:10.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.843+0000 7fbfc1cdc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfbc15a9b0 0x7fbfbc0ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:10.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.843+0000 7fbfc1cdc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfbc15bb30 0x7fbfbc0aee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:10.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.843+0000 7fbfc1cdc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfbc15a010 con 0x7fbfbc15a9b0 2024-03-18T20:28:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.843+0000 7fbfc1cdc700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfbc0b14f0 con 0x7fbfbc15bb30 2024-03-18T20:28:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.843+0000 7fbfc1cdc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfbc0ccdc0 con 0x7fbfbc0b32d0 2024-03-18T20:28:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.843+0000 7fbfb2ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfbc15a9b0 0x7fbfbc0ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.843+0000 7fbfb2ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfbc15a9b0 0x7fbfbc0ae900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54288/0 (socket says 172.21.15.1:54288) 2024-03-18T20:28:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.843+0000 7fbfb2ffd700 1 -- 172.21.15.1:0/1138062665 learned_addr learned my addr 172.21.15.1:0/1138062665 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.843+0000 7fbfbbfff700 1 --2- 172.21.15.1:0/1138062665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfbc15bb30 0x7fbfbc0aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.843+0000 7fbfbb7fe700 1 --2- 172.21.15.1:0/1138062665 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfbc0b32d0 0x7fbfbc150bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.843+0000 7fbfb2ffd700 1 -- 172.21.15.1:0/1138062665 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfbc0b32d0 msgr2=0x7fbfbc150bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.843+0000 7fbfb2ffd700 1 --2- 172.21.15.1:0/1138062665 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfbc0b32d0 0x7fbfbc150bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.843+0000 7fbfb2ffd700 1 -- 172.21.15.1:0/1138062665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfbc15bb30 msgr2=0x7fbfbc0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.843+0000 7fbfb2ffd700 1 --2- 172.21.15.1:0/1138062665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfbc15bb30 0x7fbfbc0aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.843+0000 7fbfb2ffd700 1 -- 172.21.15.1:0/1138062665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfac0094a0 con 0x7fbfbc15a9b0 2024-03-18T20:28:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.843+0000 7fbfb2ffd700 1 --2- 172.21.15.1:0/1138062665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfbc15a9b0 0x7fbfbc0ae900 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 crypto rx=0x7fbfa400c500 tx=0x7fbfa400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.844+0000 7fbfb8ff9700 1 -- 172.21.15.1:0/1138062665 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4005d70 con 0x7fbfbc15a9b0 2024-03-18T20:28:10.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.844+0000 7fbfb8ff9700 1 -- 172.21.15.1:0/1138062665 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa40070d0 con 0x7fbfbc15a9b0 2024-03-18T20:28:10.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.844+0000 7fbfc1cdc700 1 -- 172.21.15.1:0/1138062665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfbc0cd0a0 con 0x7fbfbc15a9b0 2024-03-18T20:28:10.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.844+0000 7fbfb8ff9700 1 -- 172.21.15.1:0/1138062665 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4007440 con 0x7fbfbc15a9b0 2024-03-18T20:28:10.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.844+0000 7fbfc1cdc700 1 -- 172.21.15.1:0/1138062665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfbc0b0810 con 0x7fbfbc15a9b0 2024-03-18T20:28:10.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.845+0000 7fbfc1cdc700 1 -- 172.21.15.1:0/1138062665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfbc0ccbc0 con 0x7fbfbc15a9b0 2024-03-18T20:28:10.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.845+0000 7fbfb8ff9700 1 -- 172.21.15.1:0/1138062665 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fbfa40075a0 con 0x7fbfbc15a9b0 2024-03-18T20:28:10.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.845+0000 7fbfb8ff9700 1 --2- 172.21.15.1:0/1138062665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf9c06adb0 0x7fbf9c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:10.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.846+0000 7fbfb8ff9700 1 -- 172.21.15.1:0/1138062665 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fbfa40888e0 con 0x7fbfbc15a9b0 2024-03-18T20:28:10.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.846+0000 7fbfbb7fe700 1 --2- 172.21.15.1:0/1138062665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf9c06adb0 0x7fbf9c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:10.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.846+0000 7fbfbb7fe700 1 --2- 172.21.15.1:0/1138062665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf9c06adb0 0x7fbf9c06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbfac0090b0 tx=0x7fbfac01b040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:10.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:10.850+0000 7fbfb8ff9700 1 -- 172.21.15.1:0/1138062665 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbfa40538b0 con 0x7fbfbc15a9b0 2024-03-18T20:28:11.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.053+0000 7fbfc1cdc700 1 -- 172.21.15.1:0/1138062665 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfbc0b37b0 con 0x7fbfbc15a9b0 2024-03-18T20:28:11.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.055+0000 7fbfb8ff9700 1 -- 172.21.15.1:0/1138062665 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v287) v1 ==== 74+0+16386 (secure 0 0 0) 0x7fbfa4017090 con 0x7fbfbc15a9b0 2024-03-18T20:28:11.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:28:11.059 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":287,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:28:10.668236+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:28:08.503736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"287","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-03-18T20:28:09.644774+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"287","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:28:11.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.057+0000 7fbfc1cdc700 1 -- 172.21.15.1:0/1138062665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf9c06adb0 msgr2=0x7fbf9c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:11.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.057+0000 7fbfc1cdc700 1 --2- 172.21.15.1:0/1138062665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf9c06adb0 0x7fbf9c06d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbfac0090b0 tx=0x7fbfac01b040 comp rx=0 tx=0).stop 2024-03-18T20:28:11.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.057+0000 7fbfc1cdc700 1 -- 172.21.15.1:0/1138062665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfbc15a9b0 msgr2=0x7fbfbc0ae900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:11.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.057+0000 7fbfc1cdc700 1 --2- 172.21.15.1:0/1138062665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfbc15a9b0 0x7fbfbc0ae900 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 crypto rx=0x7fbfa400c500 tx=0x7fbfa400c810 comp rx=0 tx=0).stop 2024-03-18T20:28:11.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.057+0000 7fbfc1cdc700 1 -- 172.21.15.1:0/1138062665 shutdown_connections 2024-03-18T20:28:11.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.057+0000 7fbfc1cdc700 1 --2- 172.21.15.1:0/1138062665 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fbf9c06adb0 0x7fbf9c06d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:11.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.057+0000 7fbfc1cdc700 1 --2- 172.21.15.1:0/1138062665 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fbfbc0b32d0 0x7fbfbc150bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:11.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.057+0000 7fbfc1cdc700 1 --2- 172.21.15.1:0/1138062665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfbc15bb30 0x7fbfbc0aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:11.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.057+0000 7fbfc1cdc700 1 --2- 172.21.15.1:0/1138062665 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfbc15a9b0 0x7fbfbc0ae900 unknown :-1 s=CLOSED pgs=3270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:11.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.057+0000 7fbfc1cdc700 1 -- 172.21.15.1:0/1138062665 >> 172.21.15.1:0/1138062665 conn(0x7fbfbc0b5780 msgr2=0x7fbfbc13d730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:11.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.057+0000 7fbfc1cdc700 1 -- 172.21.15.1:0/1138062665 shutdown_connections 2024-03-18T20:28:11.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.057+0000 7fbfc1cdc700 1 -- 172.21.15.1:0/1138062665 wait complete. 2024-03-18T20:28:11.073 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:28:11.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.195+0000 7f147b0ac700 1 Processor -- start 2024-03-18T20:28:11.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.196+0000 7f147b0ac700 1 -- start start 2024-03-18T20:28:11.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.196+0000 7f147b0ac700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f147415d410 0x7f147406e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.196+0000 7f147b0ac700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f14740ae1a0 0x7f14740ae830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.196+0000 7f147b0ac700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f147406d350 con 0x7f14741610a0 2024-03-18T20:28:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.196+0000 7f147b0ac700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f147406e1b0 con 0x7f14740ae1a0 2024-03-18T20:28:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.196+0000 7f147b0ac700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f147406d5a0 con 0x7f147415d410 2024-03-18T20:28:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f1479649700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f14741610a0 0x7f14740b3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54146/0 (socket says 172.21.15.1:54146) 2024-03-18T20:28:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f1479649700 1 -- 172.21.15.1:0/836007689 learned_addr learned my addr 172.21.15.1:0/836007689 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f1478e48700 1 --2- 172.21.15.1:0/836007689 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f147415d410 0x7f147406e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f1473fff700 1 --2- 172.21.15.1:0/836007689 >> v2:172.21.15.165:3300/0 conn(0x7f14740ae1a0 0x7f14740ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f14737fe700 1 -- 172.21.15.1:0/836007689 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816417729 0 0) 0x7f147406d350 con 0x7f14741610a0 2024-03-18T20:28:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f14737fe700 1 -- 172.21.15.1:0/836007689 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14640035b0 con 0x7f14741610a0 2024-03-18T20:28:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f14737fe700 1 -- 172.21.15.1:0/836007689 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1809720569 0 0) 0x7f14640035b0 con 0x7f14741610a0 2024-03-18T20:28:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f14737fe700 1 -- 172.21.15.1:0/836007689 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f147415d410 msgr2=0x7f147406e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:11.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f14737fe700 1 --2- 172.21.15.1:0/836007689 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f147415d410 0x7f147406e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f14737fe700 1 -- 172.21.15.1:0/836007689 >> v2:172.21.15.165:3300/0 conn(0x7f14740ae1a0 msgr2=0x7f14740ae830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f14737fe700 1 --2- 172.21.15.1:0/836007689 >> v2:172.21.15.165:3300/0 conn(0x7f14740ae1a0 0x7f14740ae830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f14737fe700 1 -- 172.21.15.1:0/836007689 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147406d880 con 0x7f14741610a0 2024-03-18T20:28:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f14737fe700 1 -- 172.21.15.1:0/836007689 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f146c0019f0 con 0x7f14741610a0 2024-03-18T20:28:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f14737fe700 1 -- 172.21.15.1:0/836007689 >> v1:172.21.15.1:6789/0 conn(0x7f14741610a0 legacy=0x7f14740b3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f14737fe700 1 --2- 172.21.15.1:0/836007689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1464003a00 0x7f1464003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f14737fe700 1 --2- 172.21.15.1:0/836007689 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f147415d410 0x7f1464004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f14737fe700 1 --2- 172.21.15.1:0/836007689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1464004d90 0x7f1464005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f14737fe700 1 -- 172.21.15.1:0/836007689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147406e1b0 con 0x7f1464004d90 2024-03-18T20:28:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f14737fe700 1 -- 172.21.15.1:0/836007689 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147406d5a0 con 0x7f1464003a00 2024-03-18T20:28:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f14737fe700 1 -- 172.21.15.1:0/836007689 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f146400ba10 con 0x7f147415d410 2024-03-18T20:28:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f1479649700 1 --2- 172.21.15.1:0/836007689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1464004d90 0x7f1464005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.197+0000 7f1473fff700 1 --2- 172.21.15.1:0/836007689 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f147415d410 0x7f1464004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:11.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.198+0000 7f1478e48700 1 --2- 172.21.15.1:0/836007689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1464003a00 0x7f1464003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.198+0000 7f1479649700 1 -- 172.21.15.1:0/836007689 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f147415d410 msgr2=0x7f1464004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.198+0000 7f1479649700 1 --2- 172.21.15.1:0/836007689 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f147415d410 0x7f1464004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.198+0000 7f1479649700 1 -- 172.21.15.1:0/836007689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1464003a00 msgr2=0x7f1464003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.198+0000 7f1479649700 1 --2- 172.21.15.1:0/836007689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1464003a00 0x7f1464003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.198+0000 7f1479649700 1 -- 172.21.15.1:0/836007689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14640035b0 con 0x7f1464004d90 2024-03-18T20:28:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.198+0000 7f1473fff700 1 --2- 172.21.15.1:0/836007689 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f147415d410 0x7f1464004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:28:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.198+0000 7f1479649700 1 --2- 172.21.15.1:0/836007689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1464004d90 0x7f1464005210 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 crypto rx=0x7f146c00a830 tx=0x7f1464005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94dbc13cfb6a76de server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.198+0000 7f14737fe700 1 -- 172.21.15.1:0/836007689 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c00a2f0 con 0x7f1464004d90 2024-03-18T20:28:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.198+0000 7f14737fe700 1 -- 172.21.15.1:0/836007689 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146c012600 con 0x7f1464004d90 2024-03-18T20:28:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.198+0000 7f14737fe700 1 -- 172.21.15.1:0/836007689 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c0129b0 con 0x7f1464004d90 2024-03-18T20:28:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.198+0000 7f147b0ac700 1 -- 172.21.15.1:0/836007689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1464004d90 msgr2=0x7f1464005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:11.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.198+0000 7f147b0ac700 1 --2- 172.21.15.1:0/836007689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1464004d90 0x7f1464005210 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 crypto rx=0x7f146c00a830 tx=0x7f1464005880 comp rx=0 tx=0).stop 2024-03-18T20:28:11.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.198+0000 7f1479649700 1 -- 172.21.15.1:0/836007689 reap_dead start 2024-03-18T20:28:11.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.198+0000 7f147b0ac700 1 -- 172.21.15.1:0/836007689 shutdown_connections 2024-03-18T20:28:11.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.198+0000 7f147b0ac700 1 -- 172.21.15.1:0/836007689 >> 172.21.15.1:0/836007689 conn(0x7f1474072030 msgr2=0x7f147406a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:11.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.198+0000 7f147b0ac700 1 -- 172.21.15.1:0/836007689 shutdown_connections 2024-03-18T20:28:11.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.199+0000 7f147b0ac700 1 -- 172.21.15.1:0/836007689 wait complete. 2024-03-18T20:28:11.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.200+0000 7f147b0ac700 1 Processor -- start 2024-03-18T20:28:11.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.200+0000 7f147b0ac700 1 -- start start 2024-03-18T20:28:11.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f147b0ac700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1464004d90 0x7f14741477e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:11.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f147b0ac700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14740b4910 0x7f14741f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f147b0ac700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14741f7110 0x7f147414c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f147b0ac700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14741f7590 con 0x7f14740b4910 2024-03-18T20:28:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f147b0ac700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14740b4e70 con 0x7f14741f7110 2024-03-18T20:28:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f147b0ac700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14740b5150 con 0x7f1464004d90 2024-03-18T20:28:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f1473fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14740b4910 0x7f14741f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f1473fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14740b4910 0x7f14741f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54314/0 (socket says 172.21.15.1:54314) 2024-03-18T20:28:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f1473fff700 1 -- 172.21.15.1:0/1392249976 learned_addr learned my addr 172.21.15.1:0/1392249976 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f1479649700 1 --2- 172.21.15.1:0/1392249976 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14741f7110 0x7f147414c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f1478e48700 1 --2- 172.21.15.1:0/1392249976 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1464004d90 0x7f14741477e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f1473fff700 1 -- 172.21.15.1:0/1392249976 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1464004d90 msgr2=0x7f14741477e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f1473fff700 1 --2- 172.21.15.1:0/1392249976 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1464004d90 0x7f14741477e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:11.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f1473fff700 1 -- 172.21.15.1:0/1392249976 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14741f7110 msgr2=0x7f147414c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f1473fff700 1 --2- 172.21.15.1:0/1392249976 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14741f7110 0x7f147414c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f1473fff700 1 -- 172.21.15.1:0/1392249976 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14740b5430 con 0x7f14740b4910 2024-03-18T20:28:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f1473fff700 1 --2- 172.21.15.1:0/1392249976 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14740b4910 0x7f14741f6bd0 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 crypto rx=0x7f14600118d0 tx=0x7f1460011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f14717fa700 1 -- 172.21.15.1:0/1392249976 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1460004220 con 0x7f14740b4910 2024-03-18T20:28:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f14717fa700 1 -- 172.21.15.1:0/1392249976 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1460004380 con 0x7f14740b4910 2024-03-18T20:28:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f147b0ac700 1 -- 172.21.15.1:0/1392249976 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14740b5710 con 0x7f14740b4910 2024-03-18T20:28:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f14717fa700 1 -- 172.21.15.1:0/1392249976 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14600036a0 con 0x7f14740b4910 2024-03-18T20:28:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.201+0000 7f147b0ac700 1 -- 172.21.15.1:0/1392249976 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147414c610 con 0x7f14740b4910 2024-03-18T20:28:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.202+0000 7f147b0ac700 1 -- 172.21.15.1:0/1392249976 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1474067dc0 con 0x7f14740b4910 2024-03-18T20:28:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.203+0000 7f14717fa700 1 -- 172.21.15.1:0/1392249976 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1460003800 con 0x7f14740b4910 2024-03-18T20:28:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.203+0000 7f14717fa700 1 --2- 172.21.15.1:0/1392249976 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f145806adb0 0x7f145806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.203+0000 7f14717fa700 1 -- 172.21.15.1:0/1392249976 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f1460089b40 con 0x7f14740b4910 2024-03-18T20:28:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.203+0000 7f1478e48700 1 --2- 172.21.15.1:0/1392249976 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f145806adb0 0x7f145806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:11.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.204+0000 7f1478e48700 1 --2- 172.21.15.1:0/1392249976 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f145806adb0 0x7f145806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f14680045f0 tx=0x7f1468008040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:11.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.208+0000 7f14717fa700 1 -- 172.21.15.1:0/1392249976 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f14600549e0 con 0x7f14740b4910 2024-03-18T20:28:11.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.410+0000 7f147b0ac700 1 -- 172.21.15.1:0/1392249976 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1474066600 con 0x7f14740b4910 2024-03-18T20:28:11.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.411+0000 7f14717fa700 1 -- 172.21.15.1:0/1392249976 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v287) v1 ==== 74+0+16386 (secure 0 0 0) 0x7f1460058000 con 0x7f14740b4910 2024-03-18T20:28:11.415 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:28:11.416 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":287,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:28:10.668236+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:28:08.503736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"287","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-03-18T20:28:09.644774+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"287","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:28:11.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.413+0000 7f147b0ac700 1 -- 172.21.15.1:0/1392249976 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f145806adb0 msgr2=0x7f145806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:11.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.413+0000 7f147b0ac700 1 --2- 172.21.15.1:0/1392249976 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f145806adb0 0x7f145806d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f14680045f0 tx=0x7f1468008040 comp rx=0 tx=0).stop 2024-03-18T20:28:11.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.413+0000 7f147b0ac700 1 -- 172.21.15.1:0/1392249976 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14740b4910 msgr2=0x7f14741f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:11.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.413+0000 7f147b0ac700 1 --2- 172.21.15.1:0/1392249976 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14740b4910 0x7f14741f6bd0 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 crypto rx=0x7f14600118d0 tx=0x7f1460011c90 comp rx=0 tx=0).stop 2024-03-18T20:28:11.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.413+0000 7f147b0ac700 1 -- 172.21.15.1:0/1392249976 shutdown_connections 2024-03-18T20:28:11.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.413+0000 7f147b0ac700 1 --2- 172.21.15.1:0/1392249976 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f145806adb0 0x7f145806d270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:11.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.413+0000 7f147b0ac700 1 --2- 172.21.15.1:0/1392249976 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1464004d90 0x7f14741477e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:11.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.413+0000 7f147b0ac700 1 --2- 172.21.15.1:0/1392249976 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f14741f7110 0x7f147414c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:11.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.413+0000 7f147b0ac700 1 --2- 172.21.15.1:0/1392249976 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14740b4910 0x7f14741f6bd0 unknown :-1 s=CLOSED pgs=3273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:11.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.413+0000 7f147b0ac700 1 -- 172.21.15.1:0/1392249976 >> 172.21.15.1:0/1392249976 conn(0x7f1474072030 msgr2=0x7f14740b0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:11.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.413+0000 7f147b0ac700 1 -- 172.21.15.1:0/1392249976 shutdown_connections 2024-03-18T20:28:11.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.413+0000 7f147b0ac700 1 -- 172.21.15.1:0/1392249976 wait complete. 2024-03-18T20:28:11.430 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T20:28:11.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.555+0000 7f874b984700 1 Processor -- start 2024-03-18T20:28:11.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.555+0000 7f874b984700 1 -- start start 2024-03-18T20:28:11.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.556+0000 7f874b984700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f874415d8f0 0x7f87440b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:11.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.556+0000 7f874b984700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8744072050 0x7f87440b4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:11.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.556+0000 7f874b984700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87440b1960 con 0x7f87440b35e0 2024-03-18T20:28:11.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.556+0000 7f874b984700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87440b1bb0 con 0x7f8744072050 2024-03-18T20:28:11.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.556+0000 7f874b984700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87440adb70 con 0x7f874415d8f0 2024-03-18T20:28:11.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.556+0000 7f8749f21700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8744072050 0x7f87440b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:11.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.556+0000 7f8748f1f700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f87440b35e0 0x7f874406a7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54152/0 (socket says 172.21.15.1:54152) 2024-03-18T20:28:11.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.556+0000 7f8748f1f700 1 -- 172.21.15.1:0/1330711664 learned_addr learned my addr 172.21.15.1:0/1330711664 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:11.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.556+0000 7f8749720700 1 --2- 172.21.15.1:0/1330711664 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f874415d8f0 0x7f87440b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:11.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.556+0000 7f8749f21700 1 -- 172.21.15.1:0/1330711664 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f874415d8f0 msgr2=0x7f87440b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:11.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.556+0000 7f8749f21700 1 --2- 172.21.15.1:0/1330711664 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f874415d8f0 0x7f87440b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:11.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.556+0000 7f8749f21700 1 -- 172.21.15.1:0/1330711664 >> v1:172.21.15.1:6789/0 conn(0x7f87440b35e0 legacy=0x7f874406a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:11.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.556+0000 7f8749f21700 1 -- 172.21.15.1:0/1330711664 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87440ade50 con 0x7f8744072050 2024-03-18T20:28:11.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.556+0000 7f873bfff700 1 -- 172.21.15.1:0/1330711664 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863220167 0 0) 0x7f87440b1960 con 0x7f87440b35e0 2024-03-18T20:28:11.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.556+0000 7f8749720700 1 --2- 172.21.15.1:0/1330711664 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f874415d8f0 0x7f87440b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:28:11.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.556+0000 7f8749f21700 1 --2- 172.21.15.1:0/1330711664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8744072050 0x7f87440b4320 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7f873c009d90 tx=0x7f873c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f5fe083c19beaff server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:11.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.556+0000 7f873bfff700 1 -- 172.21.15.1:0/1330711664 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c00e070 con 0x7f8744072050 2024-03-18T20:28:11.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.557+0000 7f873bfff700 1 -- 172.21.15.1:0/1330711664 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873c00bbd0 con 0x7f8744072050 2024-03-18T20:28:11.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.557+0000 7f873bfff700 1 -- 172.21.15.1:0/1330711664 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c012430 con 0x7f8744072050 2024-03-18T20:28:11.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.557+0000 7f874b984700 1 -- 172.21.15.1:0/1330711664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8744072050 msgr2=0x7f87440b4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:11.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.557+0000 7f874b984700 1 --2- 172.21.15.1:0/1330711664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8744072050 0x7f87440b4320 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7f873c009d90 tx=0x7f873c00b330 comp rx=0 tx=0).stop 2024-03-18T20:28:11.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.557+0000 7f874b984700 1 -- 172.21.15.1:0/1330711664 shutdown_connections 2024-03-18T20:28:11.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.557+0000 7f874b984700 1 --2- 172.21.15.1:0/1330711664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8744072050 0x7f87440b4320 unknown :-1 s=CLOSED pgs=3275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:11.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.557+0000 7f874b984700 1 --2- 172.21.15.1:0/1330711664 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f874415d8f0 0x7f87440b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:11.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.557+0000 7f874b984700 1 -- 172.21.15.1:0/1330711664 >> 172.21.15.1:0/1330711664 conn(0x7f87440b5a10 msgr2=0x7f87440b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:11.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.557+0000 7f874b984700 1 -- 172.21.15.1:0/1330711664 shutdown_connections 2024-03-18T20:28:11.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.557+0000 7f874b984700 1 -- 172.21.15.1:0/1330711664 wait complete. 2024-03-18T20:28:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.558+0000 7f874b984700 1 Processor -- start 2024-03-18T20:28:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.558+0000 7f874b984700 1 -- start start 2024-03-18T20:28:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.559+0000 7f874b984700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8744072050 0x7f8744152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.559+0000 7f874b984700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f874415d8f0 0x7f87441449a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.559+0000 7f874b984700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f874409cf80 0x7f874406dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.559+0000 7f874b984700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874409e030 con 0x7f874409cf80 2024-03-18T20:28:11.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.559+0000 7f874b984700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874409d420 con 0x7f874415d8f0 2024-03-18T20:28:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.559+0000 7f874b984700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f874409d700 con 0x7f8744072050 2024-03-18T20:28:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.559+0000 7f8749f21700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f874409cf80 0x7f874406dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.559+0000 7f8749f21700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f874409cf80 0x7f874406dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54324/0 (socket says 172.21.15.1:54324) 2024-03-18T20:28:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.559+0000 7f8749f21700 1 -- 172.21.15.1:0/625624199 learned_addr learned my addr 172.21.15.1:0/625624199 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.559+0000 7f8748f1f700 1 --2- 172.21.15.1:0/625624199 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f874415d8f0 0x7f87441449a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.559+0000 7f8749720700 1 --2- 172.21.15.1:0/625624199 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8744072050 0x7f8744152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.559+0000 7f8749f21700 1 -- 172.21.15.1:0/625624199 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8744072050 msgr2=0x7f8744152190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.559+0000 7f8749f21700 1 --2- 172.21.15.1:0/625624199 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8744072050 0x7f8744152190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.559+0000 7f8749f21700 1 -- 172.21.15.1:0/625624199 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f874415d8f0 msgr2=0x7f87441449a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.559+0000 7f8749f21700 1 --2- 172.21.15.1:0/625624199 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f874415d8f0 0x7f87441449a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.559+0000 7f8749f21700 1 -- 172.21.15.1:0/625624199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c009a70 con 0x7f874409cf80 2024-03-18T20:28:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.559+0000 7f8749720700 1 --2- 172.21.15.1:0/625624199 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8744072050 0x7f8744152190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:28:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.560+0000 7f8749f21700 1 --2- 172.21.15.1:0/625624199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f874409cf80 0x7f874406dde0 secure :-1 s=READY pgs=3276 cs=0 l=1 rev1=1 crypto rx=0x7f873c014040 tx=0x7f873c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:11.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.560+0000 7f8739ffb700 1 -- 172.21.15.1:0/625624199 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c012430 con 0x7f874409cf80 2024-03-18T20:28:11.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.560+0000 7f8739ffb700 1 -- 172.21.15.1:0/625624199 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873c020070 con 0x7f874409cf80 2024-03-18T20:28:11.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.560+0000 7f8739ffb700 1 -- 172.21.15.1:0/625624199 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c004280 con 0x7f874409cf80 2024-03-18T20:28:11.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.560+0000 7f874b984700 1 -- 172.21.15.1:0/625624199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f874409d9e0 con 0x7f874409cf80 2024-03-18T20:28:11.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.560+0000 7f874b984700 1 -- 172.21.15.1:0/625624199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8744144ee0 con 0x7f874409cf80 2024-03-18T20:28:11.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.561+0000 7f874b984700 1 -- 172.21.15.1:0/625624199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f874409e1e0 con 0x7f874409cf80 2024-03-18T20:28:11.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.561+0000 7f8739ffb700 1 -- 172.21.15.1:0/625624199 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f873c0043e0 con 0x7f874409cf80 2024-03-18T20:28:11.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.562+0000 7f8739ffb700 1 --2- 172.21.15.1:0/625624199 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f872806adb0 0x7f872806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:11.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.562+0000 7f8739ffb700 1 -- 172.21.15.1:0/625624199 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(287..287 src has 1..287) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f873c022070 con 0x7f874409cf80 2024-03-18T20:28:11.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.562+0000 7f8749720700 1 --2- 172.21.15.1:0/625624199 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f872806adb0 0x7f872806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:11.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.562+0000 7f8749720700 1 --2- 172.21.15.1:0/625624199 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f872806adb0 0x7f872806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f874000aa20 tx=0x7f8740008040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:11.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.567+0000 7f8739ffb700 1 -- 172.21.15.1:0/625624199 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f874409e1e0 con 0x7f874409cf80 2024-03-18T20:28:11.603 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:28:11.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:28:11.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:11.823+0000 7f874b984700 1 -- 172.21.15.1:0/625624199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f874409e3a0 con 0x7f874409cf80 2024-03-18T20:28:12.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.688+0000 7f8739ffb700 1 -- 172.21.15.1:0/625624199 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v143) v1 ==== 118+0+0 (secure 0 0 0) 0x7f873c056400 con 0x7f874409cf80 2024-03-18T20:28:12.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.691+0000 7f874b984700 1 -- 172.21.15.1:0/625624199 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f872806adb0 msgr2=0x7f872806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:12.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.691+0000 7f874b984700 1 --2- 172.21.15.1:0/625624199 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f872806adb0 0x7f872806d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f874000aa20 tx=0x7f8740008040 comp rx=0 tx=0).stop 2024-03-18T20:28:12.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.691+0000 7f874b984700 1 -- 172.21.15.1:0/625624199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f874409cf80 msgr2=0x7f874406dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:12.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.691+0000 7f874b984700 1 --2- 172.21.15.1:0/625624199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f874409cf80 0x7f874406dde0 secure :-1 s=READY pgs=3276 cs=0 l=1 rev1=1 crypto rx=0x7f873c014040 tx=0x7f873c004ad0 comp rx=0 tx=0).stop 2024-03-18T20:28:12.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.691+0000 7f874b984700 1 -- 172.21.15.1:0/625624199 shutdown_connections 2024-03-18T20:28:12.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.691+0000 7f874b984700 1 --2- 172.21.15.1:0/625624199 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f872806adb0 0x7f872806d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:12.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.691+0000 7f874b984700 1 --2- 172.21.15.1:0/625624199 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8744072050 0x7f8744152190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:12.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.691+0000 7f874b984700 1 --2- 172.21.15.1:0/625624199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f874409cf80 0x7f874406dde0 unknown :-1 s=CLOSED pgs=3276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:12.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.691+0000 7f874b984700 1 --2- 172.21.15.1:0/625624199 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f874415d8f0 0x7f87441449a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:12.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.691+0000 7f874b984700 1 -- 172.21.15.1:0/625624199 >> 172.21.15.1:0/625624199 conn(0x7f87440b5a10 msgr2=0x7f87440b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:12.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.691+0000 7f874b984700 1 -- 172.21.15.1:0/625624199 shutdown_connections 2024-03-18T20:28:12.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.692+0000 7f874b984700 1 -- 172.21.15.1:0/625624199 wait complete. 2024-03-18T20:28:12.710 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-18T20:28:12.710 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T20:28:12.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.842+0000 7fd052fb4700 1 Processor -- start 2024-03-18T20:28:12.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.842+0000 7fd052fb4700 1 -- start start 2024-03-18T20:28:12.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.843+0000 7fd052fb4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd04c15a350 0x7fd04c0b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:12.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.843+0000 7fd052fb4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd04c0b26e0 0x7fd04c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.843+0000 7fd052fb4700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd04c06fb70 con 0x7fd04c0b48f0 2024-03-18T20:28:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.843+0000 7fd052fb4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd04c0aeb10 con 0x7fd04c15a350 2024-03-18T20:28:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.843+0000 7fd052fb4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c0b2c50 con 0x7fd04c0b26e0 2024-03-18T20:28:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.843+0000 7fd04bfff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd04c0b26e0 0x7fd04c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.843+0000 7fd050d50700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd04c15a350 0x7fd04c0b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.843+0000 7fd04bfff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd04c0b26e0 0x7fd04c0b4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:48590/0 (socket says 172.21.15.1:48590) 2024-03-18T20:28:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.843+0000 7fd04bfff700 1 -- 172.21.15.1:0/2545955475 learned_addr learned my addr 172.21.15.1:0/2545955475 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.843+0000 7fd051551700 1 --1- 172.21.15.1:0/2545955475 >> v1:172.21.15.165:6789/0 conn(0x7fd04c0b48f0 0x7fd04c0aefa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:44628/0 (socket says 172.21.15.1:44628) 2024-03-18T20:28:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.843+0000 7fd04bfff700 1 -- 172.21.15.1:0/2545955475 >> v2:172.21.15.165:3300/0 conn(0x7fd04c15a350 msgr2=0x7fd04c0b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.843+0000 7fd04bfff700 1 --2- 172.21.15.1:0/2545955475 >> v2:172.21.15.165:3300/0 conn(0x7fd04c15a350 0x7fd04c0b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.843+0000 7fd04bfff700 1 -- 172.21.15.1:0/2545955475 >> v1:172.21.15.165:6789/0 conn(0x7fd04c0b48f0 legacy=0x7fd04c0aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.843+0000 7fd04bfff700 1 -- 172.21.15.1:0/2545955475 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c06f5b0 con 0x7fd04c0b26e0 2024-03-18T20:28:12.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.843+0000 7fd050d50700 1 --2- 172.21.15.1:0/2545955475 >> v2:172.21.15.165:3300/0 conn(0x7fd04c15a350 0x7fd04c0b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:28:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.844+0000 7fd04bfff700 1 --2- 172.21.15.1:0/2545955475 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd04c0b26e0 0x7fd04c0b4e60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd03800bb90 tx=0x7fd03800bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a7398f78cb2a3e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.844+0000 7fd04b7fe700 1 -- 172.21.15.1:0/2545955475 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03800dd30 con 0x7fd04c0b26e0 2024-03-18T20:28:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.844+0000 7fd04b7fe700 1 -- 172.21.15.1:0/2545955475 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd03800de90 con 0x7fd04c0b26e0 2024-03-18T20:28:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.844+0000 7fd04b7fe700 1 -- 172.21.15.1:0/2545955475 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd038010690 con 0x7fd04c0b26e0 2024-03-18T20:28:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.844+0000 7fd052fb4700 1 -- 172.21.15.1:0/2545955475 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd04c0b26e0 msgr2=0x7fd04c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.844+0000 7fd052fb4700 1 --2- 172.21.15.1:0/2545955475 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd04c0b26e0 0x7fd04c0b4e60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd03800bb90 tx=0x7fd03800bea0 comp rx=0 tx=0).stop 2024-03-18T20:28:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.844+0000 7fd052fb4700 1 -- 172.21.15.1:0/2545955475 shutdown_connections 2024-03-18T20:28:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.844+0000 7fd052fb4700 1 --2- 172.21.15.1:0/2545955475 >> v2:172.21.15.165:3300/0 conn(0x7fd04c15a350 0x7fd04c0b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.844+0000 7fd052fb4700 1 --2- 172.21.15.1:0/2545955475 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd04c0b26e0 0x7fd04c0b4e60 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.844+0000 7fd052fb4700 1 -- 172.21.15.1:0/2545955475 >> 172.21.15.1:0/2545955475 conn(0x7fd04c0b5a10 msgr2=0x7fd04c0b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.844+0000 7fd052fb4700 1 -- 172.21.15.1:0/2545955475 shutdown_connections 2024-03-18T20:28:12.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.844+0000 7fd052fb4700 1 -- 172.21.15.1:0/2545955475 wait complete. 2024-03-18T20:28:12.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.846+0000 7fd052fb4700 1 Processor -- start 2024-03-18T20:28:12.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.846+0000 7fd052fb4700 1 -- start start 2024-03-18T20:28:12.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.846+0000 7fd052fb4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd04c0b26e0 0x7fd04c158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:12.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd052fb4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd04c0b48f0 0x7fd04c09d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:12.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd052fb4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd04c15a350 0x7fd04c09d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:12.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd052fb4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c1587a0 con 0x7fd04c0b48f0 2024-03-18T20:28:12.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd052fb4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c157b20 con 0x7fd04c0b26e0 2024-03-18T20:28:12.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd052fb4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c157e00 con 0x7fd04c15a350 2024-03-18T20:28:12.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd04bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd04c0b48f0 0x7fd04c09d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:12.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd04bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd04c0b48f0 0x7fd04c09d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54332/0 (socket says 172.21.15.1:54332) 2024-03-18T20:28:12.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd04bfff700 1 -- 172.21.15.1:0/3586137324 learned_addr learned my addr 172.21.15.1:0/3586137324 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:12.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd050d50700 1 --2- 172.21.15.1:0/3586137324 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd04c0b26e0 0x7fd04c158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:12.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd051551700 1 --2- 172.21.15.1:0/3586137324 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd04c15a350 0x7fd04c09d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:12.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd04bfff700 1 -- 172.21.15.1:0/3586137324 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd04c15a350 msgr2=0x7fd04c09d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:12.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd04bfff700 1 --2- 172.21.15.1:0/3586137324 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd04c15a350 0x7fd04c09d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:12.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd04bfff700 1 -- 172.21.15.1:0/3586137324 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd04c0b26e0 msgr2=0x7fd04c158bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:12.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd04bfff700 1 --2- 172.21.15.1:0/3586137324 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd04c0b26e0 0x7fd04c158bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:12.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd04bfff700 1 -- 172.21.15.1:0/3586137324 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0400094a0 con 0x7fd04c0b48f0 2024-03-18T20:28:12.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd04bfff700 1 --2- 172.21.15.1:0/3586137324 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd04c0b48f0 0x7fd04c09d200 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 crypto rx=0x7fd038000c00 tx=0x7fd038004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:12.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd0497fa700 1 -- 172.21.15.1:0/3586137324 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03801f070 con 0x7fd04c0b48f0 2024-03-18T20:28:12.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd0497fa700 1 -- 172.21.15.1:0/3586137324 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd038003970 con 0x7fd04c0b48f0 2024-03-18T20:28:12.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd052fb4700 1 -- 172.21.15.1:0/3586137324 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03800b7f0 con 0x7fd04c0b48f0 2024-03-18T20:28:12.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd0497fa700 1 -- 172.21.15.1:0/3586137324 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd038003d20 con 0x7fd04c0b48f0 2024-03-18T20:28:12.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.847+0000 7fd052fb4700 1 -- 172.21.15.1:0/3586137324 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd04c1583c0 con 0x7fd04c0b48f0 2024-03-18T20:28:12.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.848+0000 7fd052fb4700 1 -- 172.21.15.1:0/3586137324 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd04c1588e0 con 0x7fd04c0b48f0 2024-03-18T20:28:12.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.849+0000 7fd0497fa700 1 -- 172.21.15.1:0/3586137324 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd038023410 con 0x7fd04c0b48f0 2024-03-18T20:28:12.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.850+0000 7fd0497fa700 1 --2- 172.21.15.1:0/3586137324 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd03006adb0 0x7fd03006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:12.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.850+0000 7fd0497fa700 1 -- 172.21.15.1:0/3586137324 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fd03808cd10 con 0x7fd04c0b48f0 2024-03-18T20:28:12.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.850+0000 7fd050d50700 1 --2- 172.21.15.1:0/3586137324 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd03006adb0 0x7fd03006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:12.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.850+0000 7fd050d50700 1 --2- 172.21.15.1:0/3586137324 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd03006adb0 0x7fd03006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd040009cd0 tx=0x7fd040009210 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:12.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:12.854+0000 7fd0497fa700 1 -- 172.21.15.1:0/3586137324 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd038057db0 con 0x7fd04c0b48f0 2024-03-18T20:28:13.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.100+0000 7fd052fb4700 1 -- 172.21.15.1:0/3586137324 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd04c1588e0 con 0x7fd04c0b48f0 2024-03-18T20:28:13.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.690+0000 7fd0497fa700 1 -- 172.21.15.1:0/3586137324 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v144) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd04c1588e0 con 0x7fd04c0b48f0 2024-03-18T20:28:13.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.693+0000 7fd052fb4700 1 -- 172.21.15.1:0/3586137324 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd03006adb0 msgr2=0x7fd03006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.693+0000 7fd052fb4700 1 --2- 172.21.15.1:0/3586137324 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd03006adb0 0x7fd03006d270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd040009cd0 tx=0x7fd040009210 comp rx=0 tx=0).stop 2024-03-18T20:28:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.693+0000 7fd052fb4700 1 -- 172.21.15.1:0/3586137324 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd04c0b48f0 msgr2=0x7fd04c09d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.693+0000 7fd052fb4700 1 --2- 172.21.15.1:0/3586137324 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd04c0b48f0 0x7fd04c09d200 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 crypto rx=0x7fd038000c00 tx=0x7fd038004750 comp rx=0 tx=0).stop 2024-03-18T20:28:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.693+0000 7fd052fb4700 1 -- 172.21.15.1:0/3586137324 shutdown_connections 2024-03-18T20:28:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.694+0000 7fd052fb4700 1 --2- 172.21.15.1:0/3586137324 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd03006adb0 0x7fd03006d270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.694+0000 7fd052fb4700 1 --2- 172.21.15.1:0/3586137324 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd04c15a350 0x7fd04c09d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.694+0000 7fd052fb4700 1 --2- 172.21.15.1:0/3586137324 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd04c0b26e0 0x7fd04c158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.694+0000 7fd052fb4700 1 --2- 172.21.15.1:0/3586137324 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd04c0b48f0 0x7fd04c09d200 unknown :-1 s=CLOSED pgs=3277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.694+0000 7fd052fb4700 1 -- 172.21.15.1:0/3586137324 >> 172.21.15.1:0/3586137324 conn(0x7fd04c0b5a10 msgr2=0x7fd04c1550a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.694+0000 7fd052fb4700 1 -- 172.21.15.1:0/3586137324 shutdown_connections 2024-03-18T20:28:13.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.694+0000 7fd052fb4700 1 -- 172.21.15.1:0/3586137324 wait complete. 2024-03-18T20:28:13.713 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T20:28:13.713 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:28:13.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.840+0000 7fe9419a9700 1 Processor -- start 2024-03-18T20:28:13.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.841+0000 7fe9419a9700 1 -- start start 2024-03-18T20:28:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.841+0000 7fe9419a9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe93c15d560 0x7fe93c0d1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.841+0000 7fe9419a9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe93c0cf730 0x7fe93c0cde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.841+0000 7fe9419a9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe93c0d23b0 con 0x7fe93c0d3030 2024-03-18T20:28:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.841+0000 7fe9419a9700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe93c0ce3d0 con 0x7fe93c0cf730 2024-03-18T20:28:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.841+0000 7fe9419a9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0ce690 con 0x7fe93c15d560 2024-03-18T20:28:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.841+0000 7fe93b7fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe93c0d3030 0x7fe93c0d3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54164/0 (socket says 172.21.15.1:54164) 2024-03-18T20:28:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.841+0000 7fe93b7fe700 1 -- 172.21.15.1:0/2335435447 learned_addr learned my addr 172.21.15.1:0/2335435447 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe93affd700 1 --2- 172.21.15.1:0/2335435447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe93c15d560 0x7fe93c0d1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe93a7fc700 1 --2- 172.21.15.1:0/2335435447 >> v2:172.21.15.165:3300/0 conn(0x7fe93c0cf730 0x7fe93c0cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe939ffb700 1 -- 172.21.15.1:0/2335435447 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408693143 0 0) 0x7fe934001490 con 0x7fe93c0d3030 2024-03-18T20:28:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe939ffb700 1 -- 172.21.15.1:0/2335435447 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9280035b0 con 0x7fe93c0d3030 2024-03-18T20:28:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe939ffb700 1 -- 172.21.15.1:0/2335435447 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3062810686 0 0) 0x7fe9280035b0 con 0x7fe93c0d3030 2024-03-18T20:28:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe939ffb700 1 -- 172.21.15.1:0/2335435447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe93c15d560 msgr2=0x7fe93c0d1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe939ffb700 1 --2- 172.21.15.1:0/2335435447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe93c15d560 0x7fe93c0d1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe939ffb700 1 -- 172.21.15.1:0/2335435447 >> v2:172.21.15.165:3300/0 conn(0x7fe93c0cf730 msgr2=0x7fe93c0cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe939ffb700 1 --2- 172.21.15.1:0/2335435447 >> v2:172.21.15.165:3300/0 conn(0x7fe93c0cf730 0x7fe93c0cde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe939ffb700 1 -- 172.21.15.1:0/2335435447 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c0ce970 con 0x7fe93c0d3030 2024-03-18T20:28:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe939ffb700 1 -- 172.21.15.1:0/2335435447 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fe934001b80 con 0x7fe93c0d3030 2024-03-18T20:28:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe939ffb700 1 -- 172.21.15.1:0/2335435447 >> v1:172.21.15.1:6789/0 conn(0x7fe93c0d3030 legacy=0x7fe93c0d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe939ffb700 1 --2- 172.21.15.1:0/2335435447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe93c15d560 0x7fe928003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe939ffb700 1 --2- 172.21.15.1:0/2335435447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe928003f80 0x7fe928004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe939ffb700 1 --2- 172.21.15.1:0/2335435447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe928004d90 0x7fe928005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe939ffb700 1 -- 172.21.15.1:0/2335435447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0ce3d0 con 0x7fe928004d90 2024-03-18T20:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe939ffb700 1 -- 172.21.15.1:0/2335435447 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0ce690 con 0x7fe928003f80 2024-03-18T20:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe939ffb700 1 -- 172.21.15.1:0/2335435447 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe92800ba10 con 0x7fe93c15d560 2024-03-18T20:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe93b7fe700 1 --2- 172.21.15.1:0/2335435447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe928004d90 0x7fe928005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe93affd700 1 --2- 172.21.15.1:0/2335435447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe93c15d560 0x7fe928003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.842+0000 7fe93a7fc700 1 --2- 172.21.15.1:0/2335435447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe928003f80 0x7fe928004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.843+0000 7fe93b7fe700 1 -- 172.21.15.1:0/2335435447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe93c15d560 msgr2=0x7fe928003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.843+0000 7fe93b7fe700 1 --2- 172.21.15.1:0/2335435447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe93c15d560 0x7fe928003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.843+0000 7fe93b7fe700 1 -- 172.21.15.1:0/2335435447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe928003f80 msgr2=0x7fe928004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.843+0000 7fe93b7fe700 1 --2- 172.21.15.1:0/2335435447 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe928003f80 0x7fe928004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.843+0000 7fe93b7fe700 1 -- 172.21.15.1:0/2335435447 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9280035b0 con 0x7fe928004d90 2024-03-18T20:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.843+0000 7fe93b7fe700 1 --2- 172.21.15.1:0/2335435447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe928004d90 0x7fe928005210 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 crypto rx=0x7fe93400a830 tx=0x7fe928005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef1453ef02d75775 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.843+0000 7fe939ffb700 1 -- 172.21.15.1:0/2335435447 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93400a4c0 con 0x7fe928004d90 2024-03-18T20:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.843+0000 7fe939ffb700 1 -- 172.21.15.1:0/2335435447 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe934012600 con 0x7fe928004d90 2024-03-18T20:28:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.843+0000 7fe939ffb700 1 -- 172.21.15.1:0/2335435447 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9340129b0 con 0x7fe928004d90 2024-03-18T20:28:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.843+0000 7fe9419a9700 1 -- 172.21.15.1:0/2335435447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe928004d90 msgr2=0x7fe928005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.843+0000 7fe9419a9700 1 --2- 172.21.15.1:0/2335435447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe928004d90 0x7fe928005210 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 crypto rx=0x7fe93400a830 tx=0x7fe928005880 comp rx=0 tx=0).stop 2024-03-18T20:28:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.843+0000 7fe93b7fe700 1 -- 172.21.15.1:0/2335435447 reap_dead start 2024-03-18T20:28:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.843+0000 7fe9419a9700 1 -- 172.21.15.1:0/2335435447 shutdown_connections 2024-03-18T20:28:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.843+0000 7fe9419a9700 1 -- 172.21.15.1:0/2335435447 >> 172.21.15.1:0/2335435447 conn(0x7fe93c09c6e0 msgr2=0x7fe93c08afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.843+0000 7fe9419a9700 1 -- 172.21.15.1:0/2335435447 shutdown_connections 2024-03-18T20:28:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.843+0000 7fe9419a9700 1 -- 172.21.15.1:0/2335435447 wait complete. 2024-03-18T20:28:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.845+0000 7fe9419a9700 1 Processor -- start 2024-03-18T20:28:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.845+0000 7fe9419a9700 1 -- start start 2024-03-18T20:28:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe9419a9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe928004d90 0x7fe93c1469d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe9419a9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe93c0b76d0 0x7fe93c0b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe9419a9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe93c0b89d0 0x7fe93c156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe9419a9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0b8e60 con 0x7fe93c0b89d0 2024-03-18T20:28:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe9419a9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0b6630 con 0x7fe93c0b76d0 2024-03-18T20:28:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe9419a9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0b6910 con 0x7fe928004d90 2024-03-18T20:28:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe93b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe93c0b89d0 0x7fe93c156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe93b7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe93c0b89d0 0x7fe93c156780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54352/0 (socket says 172.21.15.1:54352) 2024-03-18T20:28:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe93b7fe700 1 -- 172.21.15.1:0/277713630 learned_addr learned my addr 172.21.15.1:0/277713630 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:13.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe93a7fc700 1 --2- 172.21.15.1:0/277713630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe93c0b76d0 0x7fe93c0b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe93affd700 1 --2- 172.21.15.1:0/277713630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe928004d90 0x7fe93c1469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe93b7fe700 1 -- 172.21.15.1:0/277713630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe928004d90 msgr2=0x7fe93c1469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe93b7fe700 1 --2- 172.21.15.1:0/277713630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe928004d90 0x7fe93c1469d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe93b7fe700 1 -- 172.21.15.1:0/277713630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe93c0b76d0 msgr2=0x7fe93c0b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe93b7fe700 1 --2- 172.21.15.1:0/277713630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe93c0b76d0 0x7fe93c0b5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe93b7fe700 1 -- 172.21.15.1:0/277713630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c0b6bf0 con 0x7fe93c0b89d0 2024-03-18T20:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe93a7fc700 1 --2- 172.21.15.1:0/277713630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe93c0b76d0 0x7fe93c0b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe93b7fe700 1 --2- 172.21.15.1:0/277713630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe93c0b89d0 0x7fe93c156780 secure :-1 s=READY pgs=3280 cs=0 l=1 rev1=1 crypto rx=0x7fe934012000 tx=0x7fe93c0d3b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe933fff700 1 -- 172.21.15.1:0/277713630 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93400a4c0 con 0x7fe93c0b89d0 2024-03-18T20:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe933fff700 1 -- 172.21.15.1:0/277713630 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe934001490 con 0x7fe93c0b89d0 2024-03-18T20:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe9419a9700 1 -- 172.21.15.1:0/277713630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93c0b6ed0 con 0x7fe93c0b89d0 2024-03-18T20:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.846+0000 7fe933fff700 1 -- 172.21.15.1:0/277713630 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93401e630 con 0x7fe93c0b89d0 2024-03-18T20:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.847+0000 7fe9419a9700 1 -- 172.21.15.1:0/277713630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93c156cc0 con 0x7fe93c0b89d0 2024-03-18T20:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.848+0000 7fe9419a9700 1 -- 172.21.15.1:0/277713630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93c0b73f0 con 0x7fe93c0b89d0 2024-03-18T20:28:13.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.848+0000 7fe933fff700 1 -- 172.21.15.1:0/277713630 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe93401e790 con 0x7fe93c0b89d0 2024-03-18T20:28:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.848+0000 7fe933fff700 1 --2- 172.21.15.1:0/277713630 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe91c06ade0 0x7fe91c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.848+0000 7fe933fff700 1 -- 172.21.15.1:0/277713630 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fe93408aed0 con 0x7fe93c0b89d0 2024-03-18T20:28:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.848+0000 7fe93affd700 1 --2- 172.21.15.1:0/277713630 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe91c06ade0 0x7fe91c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.849+0000 7fe93affd700 1 --2- 172.21.15.1:0/277713630 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe91c06ade0 0x7fe91c06d2a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe92c009ba0 tx=0x7fe92c008040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:13.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:13.853+0000 7fe933fff700 1 -- 172.21.15.1:0/277713630 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe93c0b73f0 con 0x7fe93c0b89d0 2024-03-18T20:28:14.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.102+0000 7fe9419a9700 1 -- 172.21.15.1:0/277713630 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe93c0d23b0 con 0x7fe93c0b89d0 2024-03-18T20:28:14.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.103+0000 7fe933fff700 1 -- 172.21.15.1:0/277713630 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4750 (secure 0 0 0) 0x7fe93c0d23b0 con 0x7fe93c0b89d0 2024-03-18T20:28:14.107 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:28:14.107 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":144,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10322,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/2385318877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":2385318877},{"type":"v1","addr":"172.21.15.165:6834","nonce":2385318877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139},{"gid":12040,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1024465047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1024465047},{"type":"v1","addr":"172.21.15.165:6835","nonce":1024465047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139},{"gid":13791,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2374816325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2374816325},{"type":"v1","addr":"172.21.15.1:6837","nonce":2374816325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:28:10.668549+0000","modified":"2024-03-18T20:28:13.687432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13794},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13794":{"gid":13794,"name":"a","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6836/1111156155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1111156155},{"type":"v1","addr":"172.21.15.1:6836","nonce":1111156155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-03-18T20:28:14.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.105+0000 7fe9419a9700 1 -- 172.21.15.1:0/277713630 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe91c06ade0 msgr2=0x7fe91c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:14.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.105+0000 7fe9419a9700 1 --2- 172.21.15.1:0/277713630 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe91c06ade0 0x7fe91c06d2a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe92c009ba0 tx=0x7fe92c008040 comp rx=0 tx=0).stop 2024-03-18T20:28:14.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.105+0000 7fe9419a9700 1 -- 172.21.15.1:0/277713630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe93c0b89d0 msgr2=0x7fe93c156780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:14.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.105+0000 7fe9419a9700 1 --2- 172.21.15.1:0/277713630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe93c0b89d0 0x7fe93c156780 secure :-1 s=READY pgs=3280 cs=0 l=1 rev1=1 crypto rx=0x7fe934012000 tx=0x7fe93c0d3b90 comp rx=0 tx=0).stop 2024-03-18T20:28:14.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.105+0000 7fe9419a9700 1 -- 172.21.15.1:0/277713630 shutdown_connections 2024-03-18T20:28:14.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.105+0000 7fe9419a9700 1 --2- 172.21.15.1:0/277713630 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe91c06ade0 0x7fe91c06d2a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:14.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.105+0000 7fe9419a9700 1 --2- 172.21.15.1:0/277713630 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe928004d90 0x7fe93c1469d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:14.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.105+0000 7fe9419a9700 1 --2- 172.21.15.1:0/277713630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe93c0b89d0 0x7fe93c156780 unknown :-1 s=CLOSED pgs=3280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:14.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.105+0000 7fe9419a9700 1 --2- 172.21.15.1:0/277713630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe93c0b76d0 0x7fe93c0b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:14.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.105+0000 7fe9419a9700 1 -- 172.21.15.1:0/277713630 >> 172.21.15.1:0/277713630 conn(0x7fe93c09c6e0 msgr2=0x7fe93c0ca7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:14.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.105+0000 7fe9419a9700 1 -- 172.21.15.1:0/277713630 shutdown_connections 2024-03-18T20:28:14.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.105+0000 7fe9419a9700 1 -- 172.21.15.1:0/277713630 wait complete. 2024-03-18T20:28:14.111 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 144 2024-03-18T20:28:14.122 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:28:14.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.243+0000 7f6feb690700 1 Processor -- start 2024-03-18T20:28:14.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.243+0000 7f6feb690700 1 -- start start 2024-03-18T20:28:14.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.244+0000 7f6feb690700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6fe406f310 0x7f6fe406f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:14.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.244+0000 7f6feb690700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe406fca0 con 0x7f6fe40b34d0 2024-03-18T20:28:14.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.244+0000 7f6feb690700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe4147320 con 0x7f6fe4146f40 2024-03-18T20:28:14.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.244+0000 7f6feb690700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe406e370 con 0x7f6fe406f310 2024-03-18T20:28:14.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.244+0000 7f6fe8c2b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6fe40b34d0 0x7f6fe406a7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54172/0 (socket says 172.21.15.1:54172) 2024-03-18T20:28:14.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.244+0000 7f6fe8c2b700 1 -- 172.21.15.1:0/2617739991 learned_addr learned my addr 172.21.15.1:0/2617739991 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:14.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.244+0000 7f6fe9c2d700 1 --2- 172.21.15.1:0/2617739991 >> v2:172.21.15.165:3300/0 conn(0x7f6fe406f310 0x7f6fe406f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:14.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.245+0000 7f6fdbfff700 1 -- 172.21.15.1:0/2617739991 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697464328 0 0) 0x7f6fe0001110 con 0x7f6fe40b34d0 2024-03-18T20:28:14.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.245+0000 7f6fdbfff700 1 -- 172.21.15.1:0/2617739991 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd00035b0 con 0x7f6fe40b34d0 2024-03-18T20:28:14.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.245+0000 7f6fdbfff700 1 -- 172.21.15.1:0/2617739991 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801167973 0 0) 0x7f6fe4147320 con 0x7f6fe4146f40 2024-03-18T20:28:14.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.245+0000 7f6fdbfff700 1 -- 172.21.15.1:0/2617739991 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe0001110 con 0x7f6fe4146f40 2024-03-18T20:28:14.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.245+0000 7f6fe9c2d700 1 -- 172.21.15.1:0/2617739991 >> v1:172.21.15.165:6789/0 conn(0x7f6fe4146f40 legacy=0x7f6fe406e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:14.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.245+0000 7f6fe9c2d700 1 -- 172.21.15.1:0/2617739991 >> v1:172.21.15.1:6789/0 conn(0x7f6fe40b34d0 legacy=0x7f6fe406a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:14.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.245+0000 7f6fe9c2d700 1 -- 172.21.15.1:0/2617739991 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe406b370 con 0x7f6fe406f310 2024-03-18T20:28:14.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.245+0000 7f6fe9c2d700 1 --2- 172.21.15.1:0/2617739991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fe406f310 0x7f6fe406f760 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009d90 tx=0x7f6fdc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9698592f7daf7c3f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:14.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.245+0000 7f6fdbfff700 1 -- 172.21.15.1:0/2617739991 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc00e070 con 0x7f6fe406f310 2024-03-18T20:28:14.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.245+0000 7f6fdbfff700 1 -- 172.21.15.1:0/2617739991 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fdc00bb20 con 0x7f6fe406f310 2024-03-18T20:28:14.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.245+0000 7f6fdbfff700 1 -- 172.21.15.1:0/2617739991 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc00be90 con 0x7f6fe406f310 2024-03-18T20:28:14.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.246+0000 7f6feb690700 1 -- 172.21.15.1:0/2617739991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fe406f310 msgr2=0x7f6fe406f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:14.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.246+0000 7f6feb690700 1 --2- 172.21.15.1:0/2617739991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fe406f310 0x7f6fe406f760 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009d90 tx=0x7f6fdc00b330 comp rx=0 tx=0).stop 2024-03-18T20:28:14.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.246+0000 7f6feb690700 1 -- 172.21.15.1:0/2617739991 shutdown_connections 2024-03-18T20:28:14.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.246+0000 7f6feb690700 1 --2- 172.21.15.1:0/2617739991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fe406f310 0x7f6fe406f760 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:14.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.246+0000 7f6feb690700 1 -- 172.21.15.1:0/2617739991 >> 172.21.15.1:0/2617739991 conn(0x7f6fe406c360 msgr2=0x7f6fe4099a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:14.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.246+0000 7f6feb690700 1 -- 172.21.15.1:0/2617739991 shutdown_connections 2024-03-18T20:28:14.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.246+0000 7f6feb690700 1 -- 172.21.15.1:0/2617739991 wait complete. 2024-03-18T20:28:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.247+0000 7f6feb690700 1 Processor -- start 2024-03-18T20:28:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.247+0000 7f6feb690700 1 -- start start 2024-03-18T20:28:14.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.248+0000 7f6feb690700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fe406f310 0x7f6fe41f3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:14.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.248+0000 7f6feb690700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fe40b34d0 0x7f6fe415e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:14.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.248+0000 7f6feb690700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe4146f40 0x7f6fe4160180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:14.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.248+0000 7f6feb690700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe40b1ce0 con 0x7f6fe4146f40 2024-03-18T20:28:14.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.248+0000 7f6feb690700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe40b1060 con 0x7f6fe406f310 2024-03-18T20:28:14.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.248+0000 7f6feb690700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe40b1340 con 0x7f6fe40b34d0 2024-03-18T20:28:14.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.248+0000 7f6fe9c2d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe4146f40 0x7f6fe4160180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:14.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.248+0000 7f6fe9c2d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe4146f40 0x7f6fe4160180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54354/0 (socket says 172.21.15.1:54354) 2024-03-18T20:28:14.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.248+0000 7f6fe9c2d700 1 -- 172.21.15.1:0/1439082504 learned_addr learned my addr 172.21.15.1:0/1439082504 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:14.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.248+0000 7f6fe8c2b700 1 --2- 172.21.15.1:0/1439082504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fe40b34d0 0x7f6fe415e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:14.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.248+0000 7f6fe942c700 1 --2- 172.21.15.1:0/1439082504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fe406f310 0x7f6fe41f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:14.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.248+0000 7f6fe9c2d700 1 -- 172.21.15.1:0/1439082504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fe40b34d0 msgr2=0x7f6fe415e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:14.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.248+0000 7f6fe9c2d700 1 --2- 172.21.15.1:0/1439082504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fe40b34d0 0x7f6fe415e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:14.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.248+0000 7f6fe9c2d700 1 -- 172.21.15.1:0/1439082504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fe406f310 msgr2=0x7f6fe41f3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:14.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.248+0000 7f6fe9c2d700 1 --2- 172.21.15.1:0/1439082504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fe406f310 0x7f6fe41f3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:14.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.248+0000 7f6fe9c2d700 1 -- 172.21.15.1:0/1439082504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc009a70 con 0x7f6fe4146f40 2024-03-18T20:28:14.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.248+0000 7f6fe942c700 1 --2- 172.21.15.1:0/1439082504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fe406f310 0x7f6fe41f3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:28:14.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.248+0000 7f6fe9c2d700 1 --2- 172.21.15.1:0/1439082504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe4146f40 0x7f6fe4160180 secure :-1 s=READY pgs=3282 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40741c0 tx=0x7f6fdc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:14.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.248+0000 7f6fd9ffb700 1 -- 172.21.15.1:0/1439082504 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc004970 con 0x7f6fe4146f40 2024-03-18T20:28:14.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.249+0000 7f6fd9ffb700 1 -- 172.21.15.1:0/1439082504 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fdc020070 con 0x7f6fe4146f40 2024-03-18T20:28:14.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.249+0000 7f6feb690700 1 -- 172.21.15.1:0/1439082504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe40b1620 con 0x7f6fe4146f40 2024-03-18T20:28:14.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.249+0000 7f6fd9ffb700 1 -- 172.21.15.1:0/1439082504 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc0276a0 con 0x7f6fe4146f40 2024-03-18T20:28:14.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.249+0000 7f6feb690700 1 -- 172.21.15.1:0/1439082504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe415e850 con 0x7f6fe4146f40 2024-03-18T20:28:14.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.250+0000 7f6feb690700 1 -- 172.21.15.1:0/1439082504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe4067dc0 con 0x7f6fe4146f40 2024-03-18T20:28:14.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.250+0000 7f6fd9ffb700 1 -- 172.21.15.1:0/1439082504 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f6fdc012430 con 0x7f6fe4146f40 2024-03-18T20:28:14.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.251+0000 7f6fd9ffb700 1 --2- 172.21.15.1:0/1439082504 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6fc806adb0 0x7f6fc806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:14.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.251+0000 7f6fd9ffb700 1 -- 172.21.15.1:0/1439082504 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f6fdc022070 con 0x7f6fe4146f40 2024-03-18T20:28:14.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.251+0000 7f6fe942c700 1 --2- 172.21.15.1:0/1439082504 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6fc806adb0 0x7f6fc806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:14.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.251+0000 7f6fe942c700 1 --2- 172.21.15.1:0/1439082504 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6fc806adb0 0x7f6fc806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4001790 tx=0x7f6fd4009040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:14.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.255+0000 7f6fd9ffb700 1 -- 172.21.15.1:0/1439082504 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6fdc0561b0 con 0x7f6fe4146f40 2024-03-18T20:28:14.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.459+0000 7f6feb690700 1 -- 172.21.15.1:0/1439082504 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fe40af2e0 con 0x7f6fe4146f40 2024-03-18T20:28:14.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.460+0000 7f6fd9ffb700 1 -- 172.21.15.1:0/1439082504 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v288) v1 ==== 74+0+16373 (secure 0 0 0) 0x7f6fdc0597d0 con 0x7f6fe4146f40 2024-03-18T20:28:14.465 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:28:14.465 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":288,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:28:11.670215+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:28:08.503736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"287","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-03-18T20:28:09.644774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"288","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:28:14.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.463+0000 7f6feb690700 1 -- 172.21.15.1:0/1439082504 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6fc806adb0 msgr2=0x7f6fc806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:14.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.463+0000 7f6feb690700 1 --2- 172.21.15.1:0/1439082504 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6fc806adb0 0x7f6fc806d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4001790 tx=0x7f6fd4009040 comp rx=0 tx=0).stop 2024-03-18T20:28:14.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.463+0000 7f6feb690700 1 -- 172.21.15.1:0/1439082504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe4146f40 msgr2=0x7f6fe4160180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:14.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.463+0000 7f6feb690700 1 --2- 172.21.15.1:0/1439082504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe4146f40 0x7f6fe4160180 secure :-1 s=READY pgs=3282 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40741c0 tx=0x7f6fdc004ad0 comp rx=0 tx=0).stop 2024-03-18T20:28:14.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.463+0000 7f6feb690700 1 -- 172.21.15.1:0/1439082504 shutdown_connections 2024-03-18T20:28:14.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.463+0000 7f6feb690700 1 --2- 172.21.15.1:0/1439082504 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f6fc806adb0 0x7f6fc806d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:14.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.463+0000 7f6feb690700 1 --2- 172.21.15.1:0/1439082504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe4146f40 0x7f6fe4160180 unknown :-1 s=CLOSED pgs=3282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:14.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.463+0000 7f6feb690700 1 --2- 172.21.15.1:0/1439082504 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fe406f310 0x7f6fe41f3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:14.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.463+0000 7f6feb690700 1 --2- 172.21.15.1:0/1439082504 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f6fe40b34d0 0x7f6fe415e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:14.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.463+0000 7f6feb690700 1 -- 172.21.15.1:0/1439082504 >> 172.21.15.1:0/1439082504 conn(0x7f6fe406c360 msgr2=0x7f6fe406e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:14.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.463+0000 7f6feb690700 1 -- 172.21.15.1:0/1439082504 shutdown_connections 2024-03-18T20:28:14.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.463+0000 7f6feb690700 1 -- 172.21.15.1:0/1439082504 wait complete. 2024-03-18T20:28:14.479 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T20:28:14.480 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:28:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.601+0000 7fcd59429700 1 Processor -- start 2024-03-18T20:28:14.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.601+0000 7fcd59429700 1 -- start start 2024-03-18T20:28:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.602+0000 7fcd59429700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcd5415d8c0 0x7fcd540b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.602+0000 7fcd59429700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd540b00a0 con 0x7fcd540d3820 2024-03-18T20:28:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.602+0000 7fcd59429700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd540d1f40 con 0x7fcd540b3c30 2024-03-18T20:28:14.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.602+0000 7fcd59429700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd540b4260 con 0x7fcd5415d8c0 2024-03-18T20:28:14.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.602+0000 7fcd52ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcd5415d8c0 0x7fcd540b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:14.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.602+0000 7fcd537fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcd540d3820 0x7fcd540b0530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40692/0 (socket says 172.21.15.1:40692) 2024-03-18T20:28:14.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.602+0000 7fcd537fe700 1 -- 172.21.15.1:0/1028585965 learned_addr learned my addr 172.21.15.1:0/1028585965 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:14.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.602+0000 7fcd52ffd700 1 -- 172.21.15.1:0/1028585965 >> v1:172.21.15.165:6789/0 conn(0x7fcd540b3c30 legacy=0x7fcd540d3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:14.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.602+0000 7fcd52ffd700 1 -- 172.21.15.1:0/1028585965 >> v1:172.21.15.1:6789/0 conn(0x7fcd540d3820 legacy=0x7fcd540b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:14.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.602+0000 7fcd52ffd700 1 -- 172.21.15.1:0/1028585965 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd540b4540 con 0x7fcd5415d8c0 2024-03-18T20:28:14.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.602+0000 7fcd52ffd700 1 --2- 172.21.15.1:0/1028585965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd5415d8c0 0x7fcd540b36f0 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009840 tx=0x7fcd44009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec79b261a1793962 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:14.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.602+0000 7fcd51ffb700 1 -- 172.21.15.1:0/1028585965 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd44004230 con 0x7fcd5415d8c0 2024-03-18T20:28:14.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.602+0000 7fcd51ffb700 1 -- 172.21.15.1:0/1028585965 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd44004390 con 0x7fcd5415d8c0 2024-03-18T20:28:14.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.602+0000 7fcd51ffb700 1 -- 172.21.15.1:0/1028585965 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd440036a0 con 0x7fcd5415d8c0 2024-03-18T20:28:14.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.603+0000 7fcd59429700 1 -- 172.21.15.1:0/1028585965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd5415d8c0 msgr2=0x7fcd540b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:14.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.603+0000 7fcd59429700 1 --2- 172.21.15.1:0/1028585965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd5415d8c0 0x7fcd540b36f0 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009840 tx=0x7fcd44009b50 comp rx=0 tx=0).stop 2024-03-18T20:28:14.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.603+0000 7fcd59429700 1 -- 172.21.15.1:0/1028585965 shutdown_connections 2024-03-18T20:28:14.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.603+0000 7fcd59429700 1 --2- 172.21.15.1:0/1028585965 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd5415d8c0 0x7fcd540b36f0 unknown :-1 s=CLOSED pgs=3284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:14.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.603+0000 7fcd59429700 1 -- 172.21.15.1:0/1028585965 >> 172.21.15.1:0/1028585965 conn(0x7fcd5408afc0 msgr2=0x7fcd5409c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:14.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.603+0000 7fcd59429700 1 -- 172.21.15.1:0/1028585965 shutdown_connections 2024-03-18T20:28:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.603+0000 7fcd59429700 1 -- 172.21.15.1:0/1028585965 wait complete. 2024-03-18T20:28:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.604+0000 7fcd59429700 1 Processor -- start 2024-03-18T20:28:14.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.604+0000 7fcd59429700 1 -- start start 2024-03-18T20:28:14.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.605+0000 7fcd59429700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd540b3c30 0x7fcd5409e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:14.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.605+0000 7fcd59429700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd540d3820 0x7fcd540b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:14.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.605+0000 7fcd59429700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd5415d8c0 0x7fcd540b80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:14.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.605+0000 7fcd59429700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd540b76e0 con 0x7fcd540d3820 2024-03-18T20:28:14.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.605+0000 7fcd59429700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5409ed70 con 0x7fcd540b3c30 2024-03-18T20:28:14.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.605+0000 7fcd59429700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5409f050 con 0x7fcd5415d8c0 2024-03-18T20:28:14.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.605+0000 7fcd527fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd540d3820 0x7fcd540b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:14.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.605+0000 7fcd527fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd540d3820 0x7fcd540b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57266/0 (socket says 172.21.15.1:57266) 2024-03-18T20:28:14.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.605+0000 7fcd527fc700 1 -- 172.21.15.1:0/2051882493 learned_addr learned my addr 172.21.15.1:0/2051882493 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:14.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.605+0000 7fcd537fe700 1 --2- 172.21.15.1:0/2051882493 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd5415d8c0 0x7fcd540b80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:14.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.605+0000 7fcd52ffd700 1 --2- 172.21.15.1:0/2051882493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd540b3c30 0x7fcd5409e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:14.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.605+0000 7fcd527fc700 1 -- 172.21.15.1:0/2051882493 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd5415d8c0 msgr2=0x7fcd540b80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:14.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.605+0000 7fcd527fc700 1 --2- 172.21.15.1:0/2051882493 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd5415d8c0 0x7fcd540b80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:14.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.605+0000 7fcd527fc700 1 -- 172.21.15.1:0/2051882493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd540b3c30 msgr2=0x7fcd5409e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:14.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.605+0000 7fcd527fc700 1 --2- 172.21.15.1:0/2051882493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd540b3c30 0x7fcd5409e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:14.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.605+0000 7fcd527fc700 1 -- 172.21.15.1:0/2051882493 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd440094a0 con 0x7fcd540d3820 2024-03-18T20:28:14.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.606+0000 7fcd527fc700 1 --2- 172.21.15.1:0/2051882493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd540d3820 0x7fcd540b7b70 secure :-1 s=READY pgs=3285 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800c4d0 tx=0x7fcd4800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:14.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.606+0000 7fcd3b7fe700 1 -- 172.21.15.1:0/2051882493 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd48011790 con 0x7fcd540d3820 2024-03-18T20:28:14.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.606+0000 7fcd59429700 1 -- 172.21.15.1:0/2051882493 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd540b7090 con 0x7fcd540d3820 2024-03-18T20:28:14.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.606+0000 7fcd3b7fe700 1 -- 172.21.15.1:0/2051882493 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd480118f0 con 0x7fcd540d3820 2024-03-18T20:28:14.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.606+0000 7fcd3b7fe700 1 -- 172.21.15.1:0/2051882493 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd48011ca0 con 0x7fcd540d3820 2024-03-18T20:28:14.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.606+0000 7fcd59429700 1 -- 172.21.15.1:0/2051882493 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd5415ce20 con 0x7fcd540d3820 2024-03-18T20:28:14.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.607+0000 7fcd59429700 1 -- 172.21.15.1:0/2051882493 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd540b7890 con 0x7fcd540d3820 2024-03-18T20:28:14.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.607+0000 7fcd3b7fe700 1 -- 172.21.15.1:0/2051882493 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fcd4801c460 con 0x7fcd540d3820 2024-03-18T20:28:14.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.607+0000 7fcd3b7fe700 1 --2- 172.21.15.1:0/2051882493 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd4006adb0 0x7fcd4006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:14.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.608+0000 7fcd3b7fe700 1 -- 172.21.15.1:0/2051882493 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fcd48089470 con 0x7fcd540d3820 2024-03-18T20:28:14.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.608+0000 7fcd52ffd700 1 --2- 172.21.15.1:0/2051882493 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd4006adb0 0x7fcd4006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:14.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.608+0000 7fcd52ffd700 1 --2- 172.21.15.1:0/2051882493 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd4006adb0 0x7fcd4006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009b90 tx=0x7fcd44013040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:14.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.612+0000 7fcd3b7fe700 1 -- 172.21.15.1:0/2051882493 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcd540b7890 con 0x7fcd540d3820 2024-03-18T20:28:14.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.812+0000 7fcd59429700 1 -- 172.21.15.1:0/2051882493 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcd5415dca0 con 0x7fcd4006adb0 2024-03-18T20:28:14.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.821+0000 7fcd3b7fe700 1 -- 172.21.15.1:0/2051882493 <== mgr.11833 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+282121 (secure 0 0 0) 0x7fcd440040b0 con 0x7fcd4006adb0 2024-03-18T20:28:14.829 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:28:14.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.825+0000 7fcd59429700 1 -- 172.21.15.1:0/2051882493 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd4006adb0 msgr2=0x7fcd4006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:14.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.825+0000 7fcd59429700 1 --2- 172.21.15.1:0/2051882493 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd4006adb0 0x7fcd4006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009b90 tx=0x7fcd44013040 comp rx=0 tx=0).stop 2024-03-18T20:28:14.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.825+0000 7fcd59429700 1 -- 172.21.15.1:0/2051882493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd540d3820 msgr2=0x7fcd540b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:14.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.825+0000 7fcd59429700 1 --2- 172.21.15.1:0/2051882493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd540d3820 0x7fcd540b7b70 secure :-1 s=READY pgs=3285 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800c4d0 tx=0x7fcd4800c890 comp rx=0 tx=0).stop 2024-03-18T20:28:14.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.825+0000 7fcd59429700 1 -- 172.21.15.1:0/2051882493 shutdown_connections 2024-03-18T20:28:14.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.825+0000 7fcd59429700 1 --2- 172.21.15.1:0/2051882493 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd4006adb0 0x7fcd4006d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:14.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.825+0000 7fcd59429700 1 --2- 172.21.15.1:0/2051882493 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcd5415d8c0 0x7fcd540b80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:14.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.825+0000 7fcd59429700 1 --2- 172.21.15.1:0/2051882493 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcd540b3c30 0x7fcd5409e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:14.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.825+0000 7fcd59429700 1 --2- 172.21.15.1:0/2051882493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd540d3820 0x7fcd540b7b70 unknown :-1 s=CLOSED pgs=3285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:14.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.825+0000 7fcd59429700 1 -- 172.21.15.1:0/2051882493 >> 172.21.15.1:0/2051882493 conn(0x7fcd5408afc0 msgr2=0x7fcd5406c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:14.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.825+0000 7fcd59429700 1 -- 172.21.15.1:0/2051882493 shutdown_connections 2024-03-18T20:28:14.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:14.825+0000 7fcd59429700 1 -- 172.21.15.1:0/2051882493 wait complete. 2024-03-18T20:28:14.832 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:28:14.841 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-18T20:28:13.164098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590390,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":699,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":421,"ondisk_log_size":421,"up":98,"acting":98,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":99,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":449008,"kb_used_data":29420,"kb_used_omap":12,"kb_used_meta":419443,"kb_avail":754525712,"statfs":{"total":773094113280,"available":772634329088,"internally_reserved":0,"allocated":30126080,"data_stored":16172418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":429510201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000662"},"pg_stats":[{"pgid":"38.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.3d","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245090+0000","last_change":"2024-03-18T20:28:10.691425+0000","last_active":"2024-03-18T20:28:12.245090+0000","last_peered":"2024-03-18T20:28:12.245090+0000","last_clean":"2024-03-18T20:28:12.245090+0000","last_became_active":"2024-03-18T20:28:10.691157+0000","last_became_peered":"2024-03-18T20:28:10.691157+0000","last_unstale":"2024-03-18T20:28:12.245090+0000","last_undegraded":"2024-03-18T20:28:12.245090+0000","last_fullsized":"2024-03-18T20:28:12.245090+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:56.377155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.3c","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245684+0000","last_change":"2024-03-18T20:28:10.696610+0000","last_active":"2024-03-18T20:28:12.245684+0000","last_peered":"2024-03-18T20:28:12.245684+0000","last_clean":"2024-03-18T20:28:12.245684+0000","last_became_active":"2024-03-18T20:28:10.696124+0000","last_became_peered":"2024-03-18T20:28:10.696124+0000","last_unstale":"2024-03-18T20:28:12.245684+0000","last_undegraded":"2024-03-18T20:28:12.245684+0000","last_fullsized":"2024-03-18T20:28:12.245684+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:10:23.902348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.39","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.679934+0000","last_change":"2024-03-18T20:28:10.677642+0000","last_active":"2024-03-18T20:28:11.679934+0000","last_peered":"2024-03-18T20:28:11.679934+0000","last_clean":"2024-03-18T20:28:11.679934+0000","last_became_active":"2024-03-18T20:28:10.677362+0000","last_became_peered":"2024-03-18T20:28:10.677362+0000","last_unstale":"2024-03-18T20:28:11.679934+0000","last_undegraded":"2024-03-18T20:28:11.679934+0000","last_fullsized":"2024-03-18T20:28:11.679934+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:54.278981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.37","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245619+0000","last_change":"2024-03-18T20:28:10.694142+0000","last_active":"2024-03-18T20:28:12.245619+0000","last_peered":"2024-03-18T20:28:12.245619+0000","last_clean":"2024-03-18T20:28:12.245619+0000","last_became_active":"2024-03-18T20:28:10.693695+0000","last_became_peered":"2024-03-18T20:28:10.693695+0000","last_unstale":"2024-03-18T20:28:12.245619+0000","last_undegraded":"2024-03-18T20:28:12.245619+0000","last_fullsized":"2024-03-18T20:28:12.245619+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:36:03.872267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.35","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245238+0000","last_change":"2024-03-18T20:28:10.694666+0000","last_active":"2024-03-18T20:28:12.245238+0000","last_peered":"2024-03-18T20:28:12.245238+0000","last_clean":"2024-03-18T20:28:12.245238+0000","last_became_active":"2024-03-18T20:28:10.694006+0000","last_became_peered":"2024-03-18T20:28:10.694006+0000","last_unstale":"2024-03-18T20:28:12.245238+0000","last_undegraded":"2024-03-18T20:28:12.245238+0000","last_fullsized":"2024-03-18T20:28:12.245238+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:21.765997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.2f","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245652+0000","last_change":"2024-03-18T20:28:10.693944+0000","last_active":"2024-03-18T20:28:12.245652+0000","last_peered":"2024-03-18T20:28:12.245652+0000","last_clean":"2024-03-18T20:28:12.245652+0000","last_became_active":"2024-03-18T20:28:10.693393+0000","last_became_peered":"2024-03-18T20:28:10.693393+0000","last_unstale":"2024-03-18T20:28:12.245652+0000","last_undegraded":"2024-03-18T20:28:12.245652+0000","last_fullsized":"2024-03-18T20:28:12.245652+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:22:02.619215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.2e","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246346+0000","last_change":"2024-03-18T20:28:10.692666+0000","last_active":"2024-03-18T20:28:12.246346+0000","last_peered":"2024-03-18T20:28:12.246346+0000","last_clean":"2024-03-18T20:28:12.246346+0000","last_became_active":"2024-03-18T20:28:10.692222+0000","last_became_peered":"2024-03-18T20:28:10.692222+0000","last_unstale":"2024-03-18T20:28:12.246346+0000","last_undegraded":"2024-03-18T20:28:12.246346+0000","last_fullsized":"2024-03-18T20:28:12.246346+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:28:40.101291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2d","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245149+0000","last_change":"2024-03-18T20:28:10.690869+0000","last_active":"2024-03-18T20:28:12.245149+0000","last_peered":"2024-03-18T20:28:12.245149+0000","last_clean":"2024-03-18T20:28:12.245149+0000","last_became_active":"2024-03-18T20:28:10.689853+0000","last_became_peered":"2024-03-18T20:28:10.689853+0000","last_unstale":"2024-03-18T20:28:12.245149+0000","last_undegraded":"2024-03-18T20:28:12.245149+0000","last_fullsized":"2024-03-18T20:28:12.245149+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:21:24.668231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.2c","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245797+0000","last_change":"2024-03-18T20:28:10.696409+0000","last_active":"2024-03-18T20:28:12.245797+0000","last_peered":"2024-03-18T20:28:12.245797+0000","last_clean":"2024-03-18T20:28:12.245797+0000","last_became_active":"2024-03-18T20:28:10.691837+0000","last_became_peered":"2024-03-18T20:28:10.691837+0000","last_unstale":"2024-03-18T20:28:12.245797+0000","last_undegraded":"2024-03-18T20:28:12.245797+0000","last_fullsized":"2024-03-18T20:28:12.245797+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:42:10.004825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.2a","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246025+0000","last_change":"2024-03-18T20:28:10.691396+0000","last_active":"2024-03-18T20:28:12.246025+0000","last_peered":"2024-03-18T20:28:12.246025+0000","last_clean":"2024-03-18T20:28:12.246025+0000","last_became_active":"2024-03-18T20:28:10.690158+0000","last_became_peered":"2024-03-18T20:28:10.690158+0000","last_unstale":"2024-03-18T20:28:12.246025+0000","last_undegraded":"2024-03-18T20:28:12.246025+0000","last_fullsized":"2024-03-18T20:28:12.246025+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:18.038073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.29","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245933+0000","last_change":"2024-03-18T20:28:10.686763+0000","last_active":"2024-03-18T20:28:12.245933+0000","last_peered":"2024-03-18T20:28:12.245933+0000","last_clean":"2024-03-18T20:28:12.245933+0000","last_became_active":"2024-03-18T20:28:10.686336+0000","last_became_peered":"2024-03-18T20:28:10.686336+0000","last_unstale":"2024-03-18T20:28:12.245933+0000","last_undegraded":"2024-03-18T20:28:12.245933+0000","last_fullsized":"2024-03-18T20:28:12.245933+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:23:20.359399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.28","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.244719+0000","last_change":"2024-03-18T20:28:10.686197+0000","last_active":"2024-03-18T20:28:12.244719+0000","last_peered":"2024-03-18T20:28:12.244719+0000","last_clean":"2024-03-18T20:28:12.244719+0000","last_became_active":"2024-03-18T20:28:10.686012+0000","last_became_peered":"2024-03-18T20:28:10.686012+0000","last_unstale":"2024-03-18T20:28:12.244719+0000","last_undegraded":"2024-03-18T20:28:12.244719+0000","last_fullsized":"2024-03-18T20:28:12.244719+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:13:13.614656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.26","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245611+0000","last_change":"2024-03-18T20:28:10.699682+0000","last_active":"2024-03-18T20:28:12.245611+0000","last_peered":"2024-03-18T20:28:12.245611+0000","last_clean":"2024-03-18T20:28:12.245611+0000","last_became_active":"2024-03-18T20:28:10.699417+0000","last_became_peered":"2024-03-18T20:28:10.699417+0000","last_unstale":"2024-03-18T20:28:12.245611+0000","last_undegraded":"2024-03-18T20:28:12.245611+0000","last_fullsized":"2024-03-18T20:28:12.245611+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:50:29.488405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.25","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246565+0000","last_change":"2024-03-18T20:28:10.688937+0000","last_active":"2024-03-18T20:28:12.246565+0000","last_peered":"2024-03-18T20:28:12.246565+0000","last_clean":"2024-03-18T20:28:12.246565+0000","last_became_active":"2024-03-18T20:28:10.686441+0000","last_became_peered":"2024-03-18T20:28:10.686441+0000","last_unstale":"2024-03-18T20:28:12.246565+0000","last_undegraded":"2024-03-18T20:28:12.246565+0000","last_fullsized":"2024-03-18T20:28:12.246565+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:26:04.526911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.24","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245454+0000","last_change":"2024-03-18T20:28:10.690417+0000","last_active":"2024-03-18T20:28:12.245454+0000","last_peered":"2024-03-18T20:28:12.245454+0000","last_clean":"2024-03-18T20:28:12.245454+0000","last_became_active":"2024-03-18T20:28:10.689963+0000","last_became_peered":"2024-03-18T20:28:10.689963+0000","last_unstale":"2024-03-18T20:28:12.245454+0000","last_undegraded":"2024-03-18T20:28:12.245454+0000","last_fullsized":"2024-03-18T20:28:12.245454+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:05:28.077758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.23","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.680119+0000","last_change":"2024-03-18T20:28:10.682430+0000","last_active":"2024-03-18T20:28:11.680119+0000","last_peered":"2024-03-18T20:28:11.680119+0000","last_clean":"2024-03-18T20:28:11.680119+0000","last_became_active":"2024-03-18T20:28:10.681171+0000","last_became_peered":"2024-03-18T20:28:10.681171+0000","last_unstale":"2024-03-18T20:28:11.680119+0000","last_undegraded":"2024-03-18T20:28:11.680119+0000","last_fullsized":"2024-03-18T20:28:11.680119+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:12:44.829333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.21","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.679925+0000","last_change":"2024-03-18T20:28:10.694653+0000","last_active":"2024-03-18T20:28:11.679925+0000","last_peered":"2024-03-18T20:28:11.679925+0000","last_clean":"2024-03-18T20:28:11.679925+0000","last_became_active":"2024-03-18T20:28:10.694369+0000","last_became_peered":"2024-03-18T20:28:10.694369+0000","last_unstale":"2024-03-18T20:28:11.679925+0000","last_undegraded":"2024-03-18T20:28:11.679925+0000","last_fullsized":"2024-03-18T20:28:11.679925+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:18:14.919929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.20","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245554+0000","last_change":"2024-03-18T20:28:10.696083+0000","last_active":"2024-03-18T20:28:12.245554+0000","last_peered":"2024-03-18T20:28:12.245554+0000","last_clean":"2024-03-18T20:28:12.245554+0000","last_became_active":"2024-03-18T20:28:10.695545+0000","last_became_peered":"2024-03-18T20:28:10.695545+0000","last_unstale":"2024-03-18T20:28:12.245554+0000","last_undegraded":"2024-03-18T20:28:12.245554+0000","last_fullsized":"2024-03-18T20:28:12.245554+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:01:46.023861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1c","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245951+0000","last_change":"2024-03-18T20:28:10.694987+0000","last_active":"2024-03-18T20:28:12.245951+0000","last_peered":"2024-03-18T20:28:12.245951+0000","last_clean":"2024-03-18T20:28:12.245951+0000","last_became_active":"2024-03-18T20:28:10.694716+0000","last_became_peered":"2024-03-18T20:28:10.694716+0000","last_unstale":"2024-03-18T20:28:12.245951+0000","last_undegraded":"2024-03-18T20:28:12.245951+0000","last_fullsized":"2024-03-18T20:28:12.245951+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:08:38.855285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:09.228103+0000","last_change":"2024-03-18T20:28:09.228103+0000","last_active":"2024-03-18T20:28:09.228103+0000","last_peered":"2024-03-18T20:28:09.228103+0000","last_clean":"2024-03-18T20:28:09.228103+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:09.228103+0000","last_undegraded":"2024-03-18T20:28:09.228103+0000","last_fullsized":"2024-03-18T20:28:09.228103+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.1d","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245335+0000","last_change":"2024-03-18T20:28:10.686116+0000","last_active":"2024-03-18T20:28:12.245335+0000","last_peered":"2024-03-18T20:28:12.245335+0000","last_clean":"2024-03-18T20:28:12.245335+0000","last_became_active":"2024-03-18T20:28:10.685838+0000","last_became_peered":"2024-03-18T20:28:10.685838+0000","last_unstale":"2024-03-18T20:28:12.245335+0000","last_undegraded":"2024-03-18T20:28:12.245335+0000","last_fullsized":"2024-03-18T20:28:12.245335+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:13.367186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1e","version":"287'1","reported_seq":12,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246469+0000","last_change":"2024-03-18T20:28:10.252517+0000","last_active":"2024-03-18T20:28:12.246469+0000","last_peered":"2024-03-18T20:28:12.246469+0000","last_clean":"2024-03-18T20:28:12.246469+0000","last_became_active":"2024-03-18T20:28:10.252231+0000","last_became_peered":"2024-03-18T20:28:10.252231+0000","last_unstale":"2024-03-18T20:28:12.246469+0000","last_undegraded":"2024-03-18T20:28:12.246469+0000","last_fullsized":"2024-03-18T20:28:12.246469+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:13:41.714453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"37.1d","version":"287'2","reported_seq":13,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245203+0000","last_change":"2024-03-18T20:28:10.256247+0000","last_active":"2024-03-18T20:28:12.245203+0000","last_peered":"2024-03-18T20:28:12.245203+0000","last_clean":"2024-03-18T20:28:12.245203+0000","last_became_active":"2024-03-18T20:28:10.255731+0000","last_became_peered":"2024-03-18T20:28:10.255731+0000","last_unstale":"2024-03-18T20:28:12.245203+0000","last_undegraded":"2024-03-18T20:28:12.245203+0000","last_fullsized":"2024-03-18T20:28:12.245203+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:58.779282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"37.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:09.228103+0000","last_change":"2024-03-18T20:28:09.228103+0000","last_active":"2024-03-18T20:28:09.228103+0000","last_peered":"2024-03-18T20:28:09.228103+0000","last_clean":"2024-03-18T20:28:09.228103+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:09.228103+0000","last_undegraded":"2024-03-18T20:28:09.228103+0000","last_fullsized":"2024-03-18T20:28:09.228103+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"37.1b","version":"287'1","reported_seq":12,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245333+0000","last_change":"2024-03-18T20:28:10.254958+0000","last_active":"2024-03-18T20:28:12.245333+0000","last_peered":"2024-03-18T20:28:12.245333+0000","last_clean":"2024-03-18T20:28:12.245333+0000","last_became_active":"2024-03-18T20:28:10.254665+0000","last_became_peered":"2024-03-18T20:28:10.254665+0000","last_unstale":"2024-03-18T20:28:12.245333+0000","last_undegraded":"2024-03-18T20:28:12.245333+0000","last_fullsized":"2024-03-18T20:28:12.245333+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:33:42.452709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.19","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245053+0000","last_change":"2024-03-18T20:28:10.698117+0000","last_active":"2024-03-18T20:28:12.245053+0000","last_peered":"2024-03-18T20:28:12.245053+0000","last_clean":"2024-03-18T20:28:12.245053+0000","last_became_active":"2024-03-18T20:28:10.697822+0000","last_became_peered":"2024-03-18T20:28:10.697822+0000","last_unstale":"2024-03-18T20:28:12.245053+0000","last_undegraded":"2024-03-18T20:28:12.245053+0000","last_fullsized":"2024-03-18T20:28:12.245053+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:47:32.592333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1a","version":"287'1","reported_seq":12,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245095+0000","last_change":"2024-03-18T20:28:10.251064+0000","last_active":"2024-03-18T20:28:12.245095+0000","last_peered":"2024-03-18T20:28:12.245095+0000","last_clean":"2024-03-18T20:28:12.245095+0000","last_became_active":"2024-03-18T20:28:10.250767+0000","last_became_peered":"2024-03-18T20:28:10.250767+0000","last_unstale":"2024-03-18T20:28:12.245095+0000","last_undegraded":"2024-03-18T20:28:12.245095+0000","last_fullsized":"2024-03-18T20:28:12.245095+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:28:39.095694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1a","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245816+0000","last_change":"2024-03-18T20:28:10.691283+0000","last_active":"2024-03-18T20:28:12.245816+0000","last_peered":"2024-03-18T20:28:12.245816+0000","last_clean":"2024-03-18T20:28:12.245816+0000","last_became_active":"2024-03-18T20:28:10.689800+0000","last_became_peered":"2024-03-18T20:28:10.689800+0000","last_unstale":"2024-03-18T20:28:12.245816+0000","last_undegraded":"2024-03-18T20:28:12.245816+0000","last_fullsized":"2024-03-18T20:28:12.245816+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:13.897647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:09.228103+0000","last_change":"2024-03-18T20:28:09.228103+0000","last_active":"2024-03-18T20:28:09.228103+0000","last_peered":"2024-03-18T20:28:09.228103+0000","last_clean":"2024-03-18T20:28:09.228103+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:09.228103+0000","last_undegraded":"2024-03-18T20:28:09.228103+0000","last_fullsized":"2024-03-18T20:28:09.228103+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.679797+0000","last_change":"2024-03-18T20:28:10.249588+0000","last_active":"2024-03-18T20:28:11.679797+0000","last_peered":"2024-03-18T20:28:11.679797+0000","last_clean":"2024-03-18T20:28:11.679797+0000","last_became_active":"2024-03-18T20:28:10.249316+0000","last_became_peered":"2024-03-18T20:28:10.249316+0000","last_unstale":"2024-03-18T20:28:11.679797+0000","last_undegraded":"2024-03-18T20:28:11.679797+0000","last_fullsized":"2024-03-18T20:28:11.679797+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:47.380054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.14","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246130+0000","last_change":"2024-03-18T20:28:10.689984+0000","last_active":"2024-03-18T20:28:12.246130+0000","last_peered":"2024-03-18T20:28:12.246130+0000","last_clean":"2024-03-18T20:28:12.246130+0000","last_became_active":"2024-03-18T20:28:10.687856+0000","last_became_peered":"2024-03-18T20:28:10.687856+0000","last_unstale":"2024-03-18T20:28:12.246130+0000","last_undegraded":"2024-03-18T20:28:12.246130+0000","last_fullsized":"2024-03-18T20:28:12.246130+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:17:11.428228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:09.228103+0000","last_change":"2024-03-18T20:28:09.228103+0000","last_active":"2024-03-18T20:28:09.228103+0000","last_peered":"2024-03-18T20:28:09.228103+0000","last_clean":"2024-03-18T20:28:09.228103+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:09.228103+0000","last_undegraded":"2024-03-18T20:28:09.228103+0000","last_fullsized":"2024-03-18T20:28:09.228103+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:09.228103+0000","last_change":"2024-03-18T20:28:09.228103+0000","last_active":"2024-03-18T20:28:09.228103+0000","last_peered":"2024-03-18T20:28:09.228103+0000","last_clean":"2024-03-18T20:28:09.228103+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:09.228103+0000","last_undegraded":"2024-03-18T20:28:09.228103+0000","last_fullsized":"2024-03-18T20:28:09.228103+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"37.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:09.228103+0000","last_change":"2024-03-18T20:28:09.228103+0000","last_active":"2024-03-18T20:28:09.228103+0000","last_peered":"2024-03-18T20:28:09.228103+0000","last_clean":"2024-03-18T20:28:09.228103+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:09.228103+0000","last_undegraded":"2024-03-18T20:28:09.228103+0000","last_fullsized":"2024-03-18T20:28:09.228103+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.17","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245595+0000","last_change":"2024-03-18T20:28:10.688268+0000","last_active":"2024-03-18T20:28:12.245595+0000","last_peered":"2024-03-18T20:28:12.245595+0000","last_clean":"2024-03-18T20:28:12.245595+0000","last_became_active":"2024-03-18T20:28:10.687999+0000","last_became_peered":"2024-03-18T20:28:10.687999+0000","last_unstale":"2024-03-18T20:28:12.245595+0000","last_undegraded":"2024-03-18T20:28:12.245595+0000","last_fullsized":"2024-03-18T20:28:12.245595+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:59.784879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:09.228103+0000","last_change":"2024-03-18T20:28:09.228103+0000","last_active":"2024-03-18T20:28:09.228103+0000","last_peered":"2024-03-18T20:28:09.228103+0000","last_clean":"2024-03-18T20:28:09.228103+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:09.228103+0000","last_undegraded":"2024-03-18T20:28:09.228103+0000","last_fullsized":"2024-03-18T20:28:09.228103+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.10","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.244919+0000","last_change":"2024-03-18T20:28:10.687448+0000","last_active":"2024-03-18T20:28:12.244919+0000","last_peered":"2024-03-18T20:28:12.244919+0000","last_clean":"2024-03-18T20:28:12.244919+0000","last_became_active":"2024-03-18T20:28:10.687056+0000","last_became_peered":"2024-03-18T20:28:10.687056+0000","last_unstale":"2024-03-18T20:28:12.244919+0000","last_undegraded":"2024-03-18T20:28:12.244919+0000","last_fullsized":"2024-03-18T20:28:12.244919+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:30.714033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:09.228103+0000","last_change":"2024-03-18T20:28:09.228103+0000","last_active":"2024-03-18T20:28:09.228103+0000","last_peered":"2024-03-18T20:28:09.228103+0000","last_clean":"2024-03-18T20:28:09.228103+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:09.228103+0000","last_undegraded":"2024-03-18T20:28:09.228103+0000","last_fullsized":"2024-03-18T20:28:09.228103+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:09.228103+0000","last_change":"2024-03-18T20:28:09.228103+0000","last_active":"2024-03-18T20:28:09.228103+0000","last_peered":"2024-03-18T20:28:09.228103+0000","last_clean":"2024-03-18T20:28:09.228103+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:09.228103+0000","last_undegraded":"2024-03-18T20:28:09.228103+0000","last_fullsized":"2024-03-18T20:28:09.228103+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.12","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245936+0000","last_change":"2024-03-18T20:28:10.692634+0000","last_active":"2024-03-18T20:28:12.245936+0000","last_peered":"2024-03-18T20:28:12.245936+0000","last_clean":"2024-03-18T20:28:12.245936+0000","last_became_active":"2024-03-18T20:28:10.692255+0000","last_became_peered":"2024-03-18T20:28:10.692255+0000","last_unstale":"2024-03-18T20:28:12.245936+0000","last_undegraded":"2024-03-18T20:28:12.245936+0000","last_fullsized":"2024-03-18T20:28:12.245936+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:17:18.898257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:09.228103+0000","last_change":"2024-03-18T20:28:09.228103+0000","last_active":"2024-03-18T20:28:09.228103+0000","last_peered":"2024-03-18T20:28:09.228103+0000","last_clean":"2024-03-18T20:28:09.228103+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:09.228103+0000","last_undegraded":"2024-03-18T20:28:09.228103+0000","last_fullsized":"2024-03-18T20:28:09.228103+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"37.10","version":"287'8","reported_seq":19,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245874+0000","last_change":"2024-03-18T20:28:10.254130+0000","last_active":"2024-03-18T20:28:12.245874+0000","last_peered":"2024-03-18T20:28:12.245874+0000","last_clean":"2024-03-18T20:28:12.245874+0000","last_became_active":"2024-03-18T20:28:10.253861+0000","last_became_peered":"2024-03-18T20:28:10.253861+0000","last_unstale":"2024-03-18T20:28:12.245874+0000","last_undegraded":"2024-03-18T20:28:12.245874+0000","last_fullsized":"2024-03-18T20:28:12.245874+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:17:44.771039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"37.f","version":"287'1","reported_seq":12,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246452+0000","last_change":"2024-03-18T20:28:10.256047+0000","last_active":"2024-03-18T20:28:12.246452+0000","last_peered":"2024-03-18T20:28:12.246452+0000","last_clean":"2024-03-18T20:28:12.246452+0000","last_became_active":"2024-03-18T20:28:10.255781+0000","last_became_peered":"2024-03-18T20:28:10.255781+0000","last_unstale":"2024-03-18T20:28:12.246452+0000","last_undegraded":"2024-03-18T20:28:12.246452+0000","last_fullsized":"2024-03-18T20:28:12.246452+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:41:39.131637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"37.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:09.228103+0000","last_change":"2024-03-18T20:28:09.228103+0000","last_active":"2024-03-18T20:28:09.228103+0000","last_peered":"2024-03-18T20:28:09.228103+0000","last_clean":"2024-03-18T20:28:09.228103+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:09.228103+0000","last_undegraded":"2024-03-18T20:28:09.228103+0000","last_fullsized":"2024-03-18T20:28:09.228103+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"37.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:09.228103+0000","last_change":"2024-03-18T20:28:09.228103+0000","last_active":"2024-03-18T20:28:09.228103+0000","last_peered":"2024-03-18T20:28:09.228103+0000","last_clean":"2024-03-18T20:28:09.228103+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:09.228103+0000","last_undegraded":"2024-03-18T20:28:09.228103+0000","last_fullsized":"2024-03-18T20:28:09.228103+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.f","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245318+0000","last_change":"2024-03-18T20:28:10.688969+0000","last_active":"2024-03-18T20:28:12.245318+0000","last_peered":"2024-03-18T20:28:12.245318+0000","last_clean":"2024-03-18T20:28:12.245318+0000","last_became_active":"2024-03-18T20:28:10.688412+0000","last_became_peered":"2024-03-18T20:28:10.688412+0000","last_unstale":"2024-03-18T20:28:12.245318+0000","last_undegraded":"2024-03-18T20:28:12.245318+0000","last_fullsized":"2024-03-18T20:28:12.245318+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:46.507265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:09.228103+0000","last_change":"2024-03-18T20:28:09.228103+0000","last_active":"2024-03-18T20:28:09.228103+0000","last_peered":"2024-03-18T20:28:09.228103+0000","last_clean":"2024-03-18T20:28:09.228103+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:09.228103+0000","last_undegraded":"2024-03-18T20:28:09.228103+0000","last_fullsized":"2024-03-18T20:28:09.228103+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.8","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246094+0000","last_change":"2024-03-18T20:28:10.687864+0000","last_active":"2024-03-18T20:28:12.246094+0000","last_peered":"2024-03-18T20:28:12.246094+0000","last_clean":"2024-03-18T20:28:12.246094+0000","last_became_active":"2024-03-18T20:28:10.686313+0000","last_became_peered":"2024-03-18T20:28:10.686313+0000","last_unstale":"2024-03-18T20:28:12.246094+0000","last_undegraded":"2024-03-18T20:28:12.246094+0000","last_fullsized":"2024-03-18T20:28:12.246094+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:27:14.212049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.b","version":"287'2","reported_seq":13,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245414+0000","last_change":"2024-03-18T20:28:10.251075+0000","last_active":"2024-03-18T20:28:12.245414+0000","last_peered":"2024-03-18T20:28:12.245414+0000","last_clean":"2024-03-18T20:28:12.245414+0000","last_became_active":"2024-03-18T20:28:10.250694+0000","last_became_peered":"2024-03-18T20:28:10.250694+0000","last_unstale":"2024-03-18T20:28:12.245414+0000","last_undegraded":"2024-03-18T20:28:12.245414+0000","last_fullsized":"2024-03-18T20:28:12.245414+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:21:00.221902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.9","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245153+0000","last_change":"2024-03-18T20:28:10.697975+0000","last_active":"2024-03-18T20:28:12.245153+0000","last_peered":"2024-03-18T20:28:12.245153+0000","last_clean":"2024-03-18T20:28:12.245153+0000","last_became_active":"2024-03-18T20:28:10.697579+0000","last_became_peered":"2024-03-18T20:28:10.697579+0000","last_unstale":"2024-03-18T20:28:12.245153+0000","last_undegraded":"2024-03-18T20:28:12.245153+0000","last_fullsized":"2024-03-18T20:28:12.245153+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:23:16.308223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246039+0000","last_change":"2024-03-18T20:28:10.256626+0000","last_active":"2024-03-18T20:28:12.246039+0000","last_peered":"2024-03-18T20:28:12.246039+0000","last_clean":"2024-03-18T20:28:12.246039+0000","last_became_active":"2024-03-18T20:28:10.256326+0000","last_became_peered":"2024-03-18T20:28:10.256326+0000","last_unstale":"2024-03-18T20:28:12.246039+0000","last_undegraded":"2024-03-18T20:28:12.246039+0000","last_fullsized":"2024-03-18T20:28:12.246039+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:27:04.227514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.244850+0000","last_change":"2024-03-18T20:28:10.256291+0000","last_active":"2024-03-18T20:28:12.244850+0000","last_peered":"2024-03-18T20:28:12.244850+0000","last_clean":"2024-03-18T20:28:12.244850+0000","last_became_active":"2024-03-18T20:28:10.256019+0000","last_became_peered":"2024-03-18T20:28:10.256019+0000","last_unstale":"2024-03-18T20:28:12.244850+0000","last_undegraded":"2024-03-18T20:28:12.244850+0000","last_fullsized":"2024-03-18T20:28:12.244850+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:42:08.999227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.244700+0000","last_change":"2024-03-18T20:28:10.254825+0000","last_active":"2024-03-18T20:28:12.244700+0000","last_peered":"2024-03-18T20:28:12.244700+0000","last_clean":"2024-03-18T20:28:12.244700+0000","last_became_active":"2024-03-18T20:28:10.254552+0000","last_became_peered":"2024-03-18T20:28:10.254552+0000","last_unstale":"2024-03-18T20:28:12.244700+0000","last_undegraded":"2024-03-18T20:28:12.244700+0000","last_fullsized":"2024-03-18T20:28:12.244700+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:12.892050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.4","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246246+0000","last_change":"2024-03-18T20:28:10.689459+0000","last_active":"2024-03-18T20:28:12.246246+0000","last_peered":"2024-03-18T20:28:12.246246+0000","last_clean":"2024-03-18T20:28:12.246246+0000","last_became_active":"2024-03-18T20:28:10.688434+0000","last_became_peered":"2024-03-18T20:28:10.688434+0000","last_unstale":"2024-03-18T20:28:12.246246+0000","last_undegraded":"2024-03-18T20:28:12.246246+0000","last_fullsized":"2024-03-18T20:28:12.246246+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:05:30.166747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.7","version":"287'2","reported_seq":13,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.680248+0000","last_change":"2024-03-18T20:28:10.254441+0000","last_active":"2024-03-18T20:28:11.680248+0000","last_peered":"2024-03-18T20:28:11.680248+0000","last_clean":"2024-03-18T20:28:11.680248+0000","last_became_active":"2024-03-18T20:28:10.253399+0000","last_became_peered":"2024-03-18T20:28:10.253399+0000","last_unstale":"2024-03-18T20:28:11.680248+0000","last_undegraded":"2024-03-18T20:28:11.680248+0000","last_fullsized":"2024-03-18T20:28:11.680248+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:07.991604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.5","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.680224+0000","last_change":"2024-03-18T20:28:10.693201+0000","last_active":"2024-03-18T20:28:11.680224+0000","last_peered":"2024-03-18T20:28:11.680224+0000","last_clean":"2024-03-18T20:28:11.680224+0000","last_became_active":"2024-03-18T20:28:10.692937+0000","last_became_peered":"2024-03-18T20:28:10.692937+0000","last_unstale":"2024-03-18T20:28:11.680224+0000","last_undegraded":"2024-03-18T20:28:11.680224+0000","last_fullsized":"2024-03-18T20:28:11.680224+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:25.186657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245587+0000","last_change":"2024-03-18T20:28:10.253371+0000","last_active":"2024-03-18T20:28:12.245587+0000","last_peered":"2024-03-18T20:28:12.245587+0000","last_clean":"2024-03-18T20:28:12.245587+0000","last_became_active":"2024-03-18T20:28:10.250075+0000","last_became_peered":"2024-03-18T20:28:10.250075+0000","last_unstale":"2024-03-18T20:28:12.245587+0000","last_undegraded":"2024-03-18T20:28:12.245587+0000","last_fullsized":"2024-03-18T20:28:12.245587+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:40.729355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:09.228103+0000","last_change":"2024-03-18T20:28:09.228103+0000","last_active":"2024-03-18T20:28:09.228103+0000","last_peered":"2024-03-18T20:28:09.228103+0000","last_clean":"2024-03-18T20:28:09.228103+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:09.228103+0000","last_undegraded":"2024-03-18T20:28:09.228103+0000","last_fullsized":"2024-03-18T20:28:09.228103+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.7","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245677+0000","last_change":"2024-03-18T20:28:10.699609+0000","last_active":"2024-03-18T20:28:12.245677+0000","last_peered":"2024-03-18T20:28:12.245677+0000","last_clean":"2024-03-18T20:28:12.245677+0000","last_became_active":"2024-03-18T20:28:10.699320+0000","last_became_peered":"2024-03-18T20:28:10.699320+0000","last_unstale":"2024-03-18T20:28:12.245677+0000","last_undegraded":"2024-03-18T20:28:12.245677+0000","last_fullsized":"2024-03-18T20:28:12.245677+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:23:42.961689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:09.228103+0000","last_change":"2024-03-18T20:28:09.228103+0000","last_active":"2024-03-18T20:28:09.228103+0000","last_peered":"2024-03-18T20:28:09.228103+0000","last_clean":"2024-03-18T20:28:09.228103+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:09.228103+0000","last_undegraded":"2024-03-18T20:28:09.228103+0000","last_fullsized":"2024-03-18T20:28:09.228103+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:09.228103+0000","last_change":"2024-03-18T20:28:09.228103+0000","last_active":"2024-03-18T20:28:09.228103+0000","last_peered":"2024-03-18T20:28:09.228103+0000","last_clean":"2024-03-18T20:28:09.228103+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:09.228103+0000","last_undegraded":"2024-03-18T20:28:09.228103+0000","last_fullsized":"2024-03-18T20:28:09.228103+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:09.228103+0000","last_change":"2024-03-18T20:28:09.228103+0000","last_active":"2024-03-18T20:28:09.228103+0000","last_peered":"2024-03-18T20:28:09.228103+0000","last_clean":"2024-03-18T20:28:09.228103+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:09.228103+0000","last_undegraded":"2024-03-18T20:28:09.228103+0000","last_fullsized":"2024-03-18T20:28:09.228103+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.2","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245231+0000","last_change":"2024-03-18T20:28:10.699519+0000","last_active":"2024-03-18T20:28:12.245231+0000","last_peered":"2024-03-18T20:28:12.245231+0000","last_clean":"2024-03-18T20:28:12.245231+0000","last_became_active":"2024-03-18T20:28:10.699250+0000","last_became_peered":"2024-03-18T20:28:10.699250+0000","last_unstale":"2024-03-18T20:28:12.245231+0000","last_undegraded":"2024-03-18T20:28:12.245231+0000","last_fullsized":"2024-03-18T20:28:12.245231+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:28:26.050233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:09.228103+0000","last_change":"2024-03-18T20:28:09.228103+0000","last_active":"2024-03-18T20:28:09.228103+0000","last_peered":"2024-03-18T20:28:09.228103+0000","last_clean":"2024-03-18T20:28:09.228103+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:09.228103+0000","last_undegraded":"2024-03-18T20:28:09.228103+0000","last_fullsized":"2024-03-18T20:28:09.228103+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"38.3","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.680248+0000","last_change":"2024-03-18T20:28:10.693857+0000","last_active":"2024-03-18T20:28:11.680248+0000","last_peered":"2024-03-18T20:28:11.680248+0000","last_clean":"2024-03-18T20:28:11.680248+0000","last_became_active":"2024-03-18T20:28:10.687810+0000","last_became_peered":"2024-03-18T20:28:10.687810+0000","last_unstale":"2024-03-18T20:28:11.680248+0000","last_undegraded":"2024-03-18T20:28:11.680248+0000","last_fullsized":"2024-03-18T20:28:11.680248+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:04.311131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.244776+0000","last_change":"2024-03-18T20:28:10.251083+0000","last_active":"2024-03-18T20:28:12.244776+0000","last_peered":"2024-03-18T20:28:12.244776+0000","last_clean":"2024-03-18T20:28:12.244776+0000","last_became_active":"2024-03-18T20:28:10.250787+0000","last_became_peered":"2024-03-18T20:28:10.250787+0000","last_unstale":"2024-03-18T20:28:12.244776+0000","last_undegraded":"2024-03-18T20:28:12.244776+0000","last_fullsized":"2024-03-18T20:28:12.244776+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:34.322198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-18T20:28:10.233700+0000","last_change":"2024-03-18T20:28:10.233700+0000","last_active":"2024-03-18T20:28:10.233700+0000","last_peered":"2024-03-18T20:28:10.233700+0000","last_clean":"2024-03-18T20:28:10.233700+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-18T20:28:10.233700+0000","last_undegraded":"2024-03-18T20:28:10.233700+0000","last_fullsized":"2024-03-18T20:28:10.233700+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"277'403","reported_seq":1444,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-03-18T20:28:10.680155+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:28:10.680155+0000","last_peered":"2024-03-18T20:28:10.680155+0000","last_clean":"2024-03-18T20:28:10.680155+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:28:10.680155+0000","last_undegraded":"2024-03-18T20:28:10.680155+0000","last_fullsized":"2024-03-18T20:28:10.680155+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":38,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":68,"acting":68,"num_store_stats":4},{"poolid":37,"num_pg":32,"stat_sum":{"num_bytes":22,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":28,"acting":28,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576010,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53304,"kb_used_data":4140,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318536,"statfs":{"total":96636764160,"available":96582180864,"internally_reserved":0,"allocated":4239360,"data_stored":2478459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.54400000000000004,"15min":0.58899999999999997},"min":{"1min":0.439,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":1.1339999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55500000000000005,"15min":0.59599999999999997},"min":{"1min":0.35799999999999998,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":1.093},"last":0.45000000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.59799999999999998},"min":{"1min":0.433,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76100000000000001,"5min":0.82999999999999996,"15min":1.4299999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56100000000000005,"15min":0.60399999999999998},"min":{"1min":0.35199999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":1.2629999999999999},"last":0.61899999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54800000000000004,"15min":0.59599999999999997},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":0.77800000000000002,"5min":0.85499999999999998,"15min":1.216},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56999999999999995,"15min":0.61099999999999999},"min":{"1min":0.44700000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.753,"5min":1.1279999999999999,"15min":1.2490000000000001},"last":0.67500000000000004}]},{"osd":3,"last update":"Mon Mar 18 20:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55000000000000004,"15min":0.59599999999999997},"min":{"1min":0.42299999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81299999999999994,"5min":0.84699999999999998,"15min":1.4450000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.58899999999999997,"15min":0.63600000000000001},"min":{"1min":0.378,"5min":0.371,"15min":0.371},"max":{"1min":0.66700000000000004,"5min":0.94599999999999995,"15min":1.5029999999999999},"last":0.63300000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.46999999999999997,"15min":0.47799999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.75900000000000001,"5min":0.79100000000000004,"15min":0.86099999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.49399999999999999,"15min":0.499},"min":{"1min":0.34000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.83799999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.52600000000000002,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.79600000000000004,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54000000000000004,"15min":0.54700000000000004},"min":{"1min":0.36799999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.76800000000000002,"5min":0.877,"15min":1.042},"last":0.66300000000000003}]},{"osd":6,"last update":"Mon Mar 18 20:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.435,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80100000000000005,"5min":0.93500000000000005,"15min":1.129},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.56100000000000005,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.873,"5min":0.91900000000000004,"15min":1.0609999999999999},"last":0.60699999999999998}]}]},{"osd":6,"up_from":13,"seq":55834576010,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65024,"kb_used_data":3564,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306816,"statfs":{"total":96636764160,"available":96570179584,"internally_reserved":0,"allocated":3649536,"data_stored":1888673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":62912967},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53300000000000003,"15min":0.58499999999999996},"min":{"1min":0.35399999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68799999999999994,"5min":0.79800000000000004,"15min":1.1519999999999999},"last":0.378},{"interface":"front","average":{"1min":0.503,"5min":0.54600000000000004,"15min":0.59499999999999997},"min":{"1min":0.33200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.73499999999999999,"5min":1.119,"15min":1.119},"last":0.35899999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53800000000000003,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73199999999999998,"5min":0.79100000000000004,"15min":1.244},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":0.59499999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":0.70099999999999996,"5min":0.89200000000000002,"15min":1.286},"last":0.65300000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56399999999999995,"15min":0.61699999999999999},"min":{"1min":0.38700000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.69599999999999995,"5min":0.79600000000000004,"15min":1.276},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.496,"5min":0.55400000000000005,"15min":0.59899999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.65200000000000002,"5min":1.089,"15min":1.2669999999999999},"last":0.46100000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55500000000000005,"15min":0.61099999999999999},"min":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71599999999999997,"5min":1.163,"15min":1.163},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57099999999999995,"15min":0.61099999999999999},"min":{"1min":0.47499999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78100000000000003,"5min":0.82399999999999995,"15min":1.337},"last":0.496}]},{"osd":4,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.35299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80200000000000005,"5min":0.85799999999999998,"15min":0.88300000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.50600000000000001,"15min":0.51800000000000002},"min":{"1min":0.32500000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83399999999999996,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.623}]},{"osd":5,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.50900000000000001,"15min":0.51200000000000001},"min":{"1min":0.42599999999999999,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":0.65000000000000002,"5min":1.008,"15min":1.008},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55900000000000005,"15min":0.55300000000000005},"min":{"1min":0.41799999999999998,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.755,"5min":0.92600000000000005,"15min":0.94199999999999995},"last":0.72499999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56699999999999995,"15min":0.56299999999999994},"min":{"1min":0.40999999999999998,"5min":0.38,"15min":0.34499999999999997},"max":{"1min":0.91200000000000003,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.34599999999999997},"max":{"1min":0.86699999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.59899999999999998}]}]},{"osd":1,"up_from":13,"seq":55834576010,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52720,"kb_used_data":3556,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319120,"statfs":{"total":96636764160,"available":96582778880,"internally_reserved":0,"allocated":3641344,"data_stored":1888069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.38,"15min":0.39000000000000001},"min":{"1min":0.25900000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.60899999999999999,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.374,"5min":0.38700000000000001,"15min":0.41699999999999998},"min":{"1min":0.25800000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.48399999999999999,"5min":0.623,"15min":0.69699999999999995},"last":0.309}]},{"osd":2,"last update":"Mon Mar 18 20:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.39400000000000002,"15min":0.42299999999999999},"min":{"1min":0.29399999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.46999999999999997,"5min":0.61199999999999999,"15min":0.68100000000000005},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.438,"15min":0.46700000000000003},"min":{"1min":0.314,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64200000000000002,"5min":0.70099999999999996,"15min":0.80500000000000005},"last":0.33900000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.44400000000000001,"15min":0.47899999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58199999999999996,"5min":0.65600000000000003,"15min":0.76100000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.40200000000000002,"15min":0.42799999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54600000000000004,"5min":0.68200000000000005,"15min":0.71699999999999997},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.63100000000000001,"15min":0.66500000000000004},"min":{"1min":0.432,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.288,"5min":1.288,"15min":1.2909999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.64100000000000001,"15min":0.67100000000000004},"min":{"1min":0.436,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.349,"5min":1.349,"15min":1.746},"last":0.58299999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.626,"15min":0.66200000000000003},"min":{"1min":0.44500000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.361,"5min":1.361,"15min":1.6910000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.67600000000000005,"15min":0.70599999999999996},"min":{"1min":0.46700000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.63800000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.63800000000000001,"15min":0.67600000000000005},"min":{"1min":0.42199999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.637},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.65600000000000003,"15min":0.69799999999999995},"min":{"1min":0.44500000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.93500000000000005,"5min":1.014,"15min":1.722},"last":0.57299999999999995}]},{"osd":7,"last update":"Mon Mar 18 20:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.65600000000000003,"15min":0.69799999999999995},"min":{"1min":0.47799999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.66500000000000004,"15min":0.69599999999999995},"min":{"1min":0.47699999999999998,"5min":0.441,"15min":0.441},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.56799999999999995}]}]},{"osd":0,"up_from":13,"seq":55834576010,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49168,"kb_used_data":4088,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322672,"statfs":{"total":96636764160,"available":96586416128,"internally_reserved":0,"allocated":4186112,"data_stored":2451590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.34399999999999997,"15min":0.38},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.40600000000000003,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.38,"15min":0.41499999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.51200000000000001,"5min":0.60099999999999998,"15min":1.1850000000000001},"last":0.45500000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.39200000000000002,"15min":0.42599999999999999},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.501,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.39800000000000002,"15min":0.439},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.48699999999999999,"5min":0.64800000000000002,"15min":0.70799999999999996},"last":0.53300000000000003}]},{"osd":3,"last update":"Mon Mar 18 20:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.38100000000000001,"15min":0.41699999999999998},"min":{"1min":0.28899999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.44800000000000001,"5min":0.58499999999999996,"15min":0.628},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.42899999999999999,"15min":0.47999999999999998},"min":{"1min":0.33300000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.53900000000000003,"5min":0.73399999999999999,"15min":0.91500000000000004},"last":0.60599999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59699999999999998,"15min":0.63900000000000001},"min":{"1min":0.435,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86299999999999999,"5min":0.90200000000000002,"15min":1.0029999999999999},"last":0.625},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.61399999999999999,"15min":0.65000000000000002},"min":{"1min":0.42199999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72099999999999997,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.76900000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61899999999999999,"15min":0.65900000000000003},"min":{"1min":0.41699999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.878,"5min":0.89700000000000002,"15min":1.0129999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.628,"15min":0.66700000000000004},"min":{"1min":0.47899999999999998,"5min":0.432,"15min":0.432},"max":{"1min":0.77800000000000002,"5min":0.96199999999999997,"15min":1.518},"last":0.69299999999999995}]},{"osd":6,"last update":"Mon Mar 18 20:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.61599999999999999,"15min":0.65000000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.78700000000000003,"5min":1.0669999999999999,"15min":1.1279999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.65100000000000002,"15min":0.69399999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92900000000000005,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.73399999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.65600000000000003,"15min":0.69199999999999995},"min":{"1min":0.496,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.81699999999999995,"5min":1.008,"15min":1.464},"last":0.752},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.64100000000000001,"15min":0.69099999999999995},"min":{"1min":0.46400000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.80800000000000005,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.72199999999999998}]}]},{"osd":2,"up_from":13,"seq":55834576010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52680,"kb_used_data":3508,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319160,"statfs":{"total":96636764160,"available":96582819840,"internally_reserved":0,"allocated":3592192,"data_stored":1861222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.33300000000000002,"15min":0.36599999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.34799999999999998,"5min":0.54800000000000004,"15min":0.57499999999999996},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.42999999999999999,"15min":0.47199999999999998},"min":{"1min":0.30499999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.51300000000000001,"5min":0.71499999999999997,"15min":0.753},"last":0.48299999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.434,"15min":0.46899999999999997},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.47599999999999998,"5min":0.64300000000000002,"15min":0.749},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.317,"5min":0.37,"15min":0.40400000000000003},"min":{"1min":0.22800000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.41299999999999998,"5min":0.55000000000000004,"15min":0.59999999999999998},"last":0.372}]},{"osd":3,"last update":"Mon Mar 18 20:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.39600000000000002,"15min":0.42499999999999999},"min":{"1min":0.26500000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.45400000000000001,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.308},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.45000000000000001,"15min":0.5},"min":{"1min":0.33700000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.48999999999999999,"5min":0.68100000000000005,"15min":0.752},"last":0.56299999999999994}]},{"osd":4,"last update":"Mon Mar 18 20:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.622,"15min":0.65100000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.93999999999999995,"5min":1.2090000000000001,"15min":1.4079999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.63300000000000001,"15min":0.65900000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.95099999999999996,"5min":1.268,"15min":1.2729999999999999},"last":0.53000000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.628,"15min":0.66900000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.873,"5min":0.97299999999999998,"15min":1.028},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.626,"15min":0.66600000000000004},"min":{"1min":0.437,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80800000000000005,"5min":0.88300000000000001,"15min":1.0629999999999999},"last":0.626}]},{"osd":6,"last update":"Mon Mar 18 20:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.63500000000000001,"15min":0.67500000000000004},"min":{"1min":0.42299999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76400000000000001,"5min":1.014,"15min":1.2050000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.69299999999999995,"15min":0.72299999999999998},"min":{"1min":0.505,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0109999999999999,"5min":1.3220000000000001,"15min":1.3859999999999999},"last":0.84699999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.63700000000000001,"15min":0.68300000000000005},"min":{"1min":0.434,"5min":0.432,"15min":0.42199999999999999},"max":{"1min":0.72499999999999998,"5min":0.99399999999999999,"15min":1.0269999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.66500000000000004,"15min":0.69699999999999995},"min":{"1min":0.46200000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.97499999999999998,"5min":1.2330000000000001,"15min":1.371},"last":0.67300000000000004}]}]},{"osd":3,"up_from":13,"seq":55834576011,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56820,"kb_used_data":3560,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315020,"statfs":{"total":96636764160,"available":96578580480,"internally_reserved":0,"allocated":3645440,"data_stored":1888103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.36099999999999999,"15min":0.39000000000000001},"min":{"1min":0.247,"5min":0.223,"15min":0.223},"max":{"1min":0.46400000000000002,"5min":0.66500000000000004,"15min":0.98099999999999998},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.373,"15min":0.40400000000000003},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.45600000000000002,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.247}]},{"osd":1,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.35999999999999999,"15min":0.39700000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.44500000000000001,"5min":0.63200000000000001,"15min":0.64000000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.42199999999999999,"15min":0.45600000000000002},"min":{"1min":0.33400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.501,"5min":0.65500000000000003,"15min":0.73099999999999998},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.41499999999999998,"15min":0.45200000000000001},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.54900000000000004,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.437,"15min":0.48099999999999998},"min":{"1min":0.32600000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.52300000000000002,"5min":0.64400000000000002,"15min":0.71199999999999997},"last":0.433}]},{"osd":4,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59499999999999997,"15min":0.64000000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.526,"5min":1.526,"15min":1.526},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61399999999999999,"15min":0.66100000000000003},"min":{"1min":0.39200000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.65800000000000003,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.56799999999999995}]},{"osd":5,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.65100000000000002,"15min":0.68700000000000006},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.66500000000000004,"15min":0.71499999999999997},"min":{"1min":0.45300000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78000000000000003,"5min":1.1120000000000001,"15min":1.171},"last":0.50700000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65800000000000003,"15min":0.69699999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.71999999999999997},"min":{"1min":0.48699999999999999,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.53100000000000003}]},{"osd":7,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.64900000000000002,"15min":0.70299999999999996},"min":{"1min":0.43099999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79900000000000004,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64300000000000002,"15min":0.68899999999999995},"min":{"1min":0.51500000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.47899999999999998}]}]},{"osd":4,"up_from":13,"seq":55834576010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48564,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323276,"statfs":{"total":96636764160,"available":96587034624,"internally_reserved":0,"allocated":3579904,"data_stored":1855080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56200000000000006,"15min":0.58399999999999996},"min":{"1min":0.373,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.81699999999999995,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55500000000000005,"15min":0.58999999999999997},"min":{"1min":0.41399999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78300000000000003,"5min":0.80600000000000005,"15min":0.84199999999999997},"last":0.64000000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.60599999999999998},"min":{"1min":0.38200000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.88300000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54900000000000004,"15min":0.57499999999999996},"min":{"1min":0.372,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.66600000000000004,"5min":0.77800000000000002,"15min":0.96499999999999997},"last":0.48599999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.57699999999999996,"15min":0.59499999999999997},"min":{"1min":0.374,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85299999999999998,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58499999999999996,"15min":0.60799999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38500000000000001},"max":{"1min":0.81100000000000005,"5min":0.81399999999999995,"15min":0.84099999999999997},"last":0.58299999999999996}]},{"osd":3,"last update":"Mon Mar 18 20:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.57199999999999995,"15min":0.59499999999999997},"min":{"1min":0.38400000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79400000000000004,"5min":1.026,"15min":1.026},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59999999999999998,"15min":0.61499999999999999},"min":{"1min":0.42299999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88600000000000001,"5min":1.028,"15min":1.028},"last":0.61199999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.48199999999999998},"min":{"1min":0.33200000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.747,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.497,"15min":0.502},"min":{"1min":0.31900000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.69199999999999995,"5min":0.85499999999999998,"15min":0.90600000000000003},"last":0.70099999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.50600000000000001,"15min":0.504},"min":{"1min":0.33900000000000002,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.63300000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.35199999999999998,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.72999999999999998,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.66800000000000004}]},{"osd":7,"last update":"Mon Mar 18 20:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55500000000000005,"15min":0.54800000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.35699999999999998},"max":{"1min":0.749,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58099999999999996,"15min":0.58199999999999996},"min":{"1min":0.38900000000000001,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.76800000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.68400000000000005}]}]},{"osd":5,"up_from":13,"seq":55834576009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70728,"kb_used_data":3508,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94301112,"statfs":{"total":96636764160,"available":96564338688,"internally_reserved":0,"allocated":3592192,"data_stored":1861222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52900000000000003,"15min":0.58399999999999996},"min":{"1min":0.38400000000000001,"5min":0.33500000000000002,"15min":0.31900000000000001},"max":{"1min":0.67500000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.53400000000000003,"15min":0.58899999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":0.66600000000000004,"5min":0.82399999999999995,"15min":1.3100000000000001},"last":0.43099999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:27:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.53900000000000003,"15min":0.59299999999999997},"min":{"1min":0.34899999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73699999999999999,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.53300000000000003,"15min":0.58599999999999997},"min":{"1min":0.376,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.74199999999999999,"5min":0.94299999999999995,"15min":1.018},"last":0.378}]},{"osd":2,"last update":"Mon Mar 18 20:27:54 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.53400000000000003,"15min":0.58699999999999997},"min":{"1min":0.34599999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.63100000000000001,"5min":0.85299999999999998,"15min":0.99299999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.53600000000000003,"15min":0.59599999999999997},"min":{"1min":0.38800000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.68300000000000005,"5min":1.599,"15min":1.599},"last":0.44900000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:27:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56000000000000005,"15min":0.60099999999999998},"min":{"1min":0.40500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.72999999999999998,"5min":0.98699999999999999,"15min":1.2430000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.63500000000000001},"min":{"1min":0.45900000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.75800000000000001,"5min":0.84399999999999997,"15min":1.161},"last":0.54100000000000004}]},{"osd":4,"last update":"Mon Mar 18 20:27:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.48399999999999999},"min":{"1min":0.32300000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70999999999999996,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.53500000000000003,"15min":0.52100000000000002},"min":{"1min":0.39900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78100000000000003,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.60999999999999999}]},{"osd":6,"last update":"Mon Mar 18 20:27:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.36699999999999999,"5min":0.33300000000000002,"15min":0.32800000000000001},"max":{"1min":0.751,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.32800000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.76300000000000001,"5min":1.643,"15min":1.643},"last":0.59599999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:27:54 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56799999999999995,"15min":0.56000000000000005},"min":{"1min":0.378,"5min":0.317,"15min":0.317},"max":{"1min":0.80700000000000005,"5min":1.028,"15min":1.028},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57999999999999996,"15min":0.58099999999999996},"min":{"1min":0.442,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.85199999999999998,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.627}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:28:14.846 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:28:15.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.004+0000 7efff15b5700 1 Processor -- start 2024-03-18T20:28:15.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.004+0000 7efff15b5700 1 -- start start 2024-03-18T20:28:15.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.004+0000 7efff15b5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effec160c00 0x7effec0b3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.004+0000 7efff15b5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7effec160fe0 0x7effec0b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.004+0000 7efff15b5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7effec0af200 0x7effec0b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.004+0000 7efff15b5700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effec0b37a0 con 0x7effec0af200 2024-03-18T20:28:15.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.004+0000 7efff15b5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effec0b0920 con 0x7effec160fe0 2024-03-18T20:28:15.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.005+0000 7efff15b5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effec0b0c00 con 0x7effec160c00 2024-03-18T20:28:15.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.005+0000 7effeb7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7effec0af200 0x7effec0b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.005+0000 7effeb7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7effec0af200 0x7effec0b40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57270/0 (socket says 172.21.15.1:57270) 2024-03-18T20:28:15.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.005+0000 7effeb7fe700 1 -- 172.21.15.1:0/2902402477 learned_addr learned my addr 172.21.15.1:0/2902402477 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:15.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.005+0000 7effeaffd700 1 --2- 172.21.15.1:0/2902402477 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effec160c00 0x7effec0b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.005+0000 7effea7fc700 1 --2- 172.21.15.1:0/2902402477 >> v2:172.21.15.165:3300/0 conn(0x7effec160fe0 0x7effec0b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.005+0000 7effeb7fe700 1 -- 172.21.15.1:0/2902402477 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effec160c00 msgr2=0x7effec0b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:15.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.005+0000 7effeb7fe700 1 --2- 172.21.15.1:0/2902402477 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effec160c00 0x7effec0b3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.005+0000 7effeb7fe700 1 -- 172.21.15.1:0/2902402477 >> v2:172.21.15.165:3300/0 conn(0x7effec160fe0 msgr2=0x7effec0b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:15.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.005+0000 7effeb7fe700 1 --2- 172.21.15.1:0/2902402477 >> v2:172.21.15.165:3300/0 conn(0x7effec160fe0 0x7effec0b3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.005+0000 7effeb7fe700 1 -- 172.21.15.1:0/2902402477 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effec0b15d0 con 0x7effec0af200 2024-03-18T20:28:15.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.005+0000 7effeb7fe700 1 --2- 172.21.15.1:0/2902402477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effec0af200 0x7effec0b40b0 secure :-1 s=READY pgs=3286 cs=0 l=1 rev1=1 crypto rx=0x7effec0711b0 tx=0x7effe400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c290d7955c32e822 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:15.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.005+0000 7effe9ffb700 1 -- 172.21.15.1:0/2902402477 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe400e070 con 0x7effec0af200 2024-03-18T20:28:15.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.005+0000 7effe9ffb700 1 -- 172.21.15.1:0/2902402477 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe400b9c0 con 0x7effec0af200 2024-03-18T20:28:15.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.005+0000 7effe9ffb700 1 -- 172.21.15.1:0/2902402477 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe400bd70 con 0x7effec0af200 2024-03-18T20:28:15.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.006+0000 7efff15b5700 1 -- 172.21.15.1:0/2902402477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effec0af200 msgr2=0x7effec0b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:15.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.006+0000 7efff15b5700 1 --2- 172.21.15.1:0/2902402477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effec0af200 0x7effec0b40b0 secure :-1 s=READY pgs=3286 cs=0 l=1 rev1=1 crypto rx=0x7effec0711b0 tx=0x7effe400b330 comp rx=0 tx=0).stop 2024-03-18T20:28:15.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.006+0000 7efff15b5700 1 -- 172.21.15.1:0/2902402477 shutdown_connections 2024-03-18T20:28:15.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.006+0000 7efff15b5700 1 --2- 172.21.15.1:0/2902402477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effec0af200 0x7effec0b40b0 unknown :-1 s=CLOSED pgs=3286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.006+0000 7efff15b5700 1 --2- 172.21.15.1:0/2902402477 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effec160c00 0x7effec0b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.006+0000 7efff15b5700 1 --2- 172.21.15.1:0/2902402477 >> v2:172.21.15.165:3300/0 conn(0x7effec160fe0 0x7effec0b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.006+0000 7efff15b5700 1 -- 172.21.15.1:0/2902402477 >> 172.21.15.1:0/2902402477 conn(0x7effec0b5780 msgr2=0x7effec0b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:15.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.006+0000 7efff15b5700 1 -- 172.21.15.1:0/2902402477 shutdown_connections 2024-03-18T20:28:15.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.006+0000 7efff15b5700 1 -- 172.21.15.1:0/2902402477 wait complete. 2024-03-18T20:28:15.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.007+0000 7efff15b5700 1 Processor -- start 2024-03-18T20:28:15.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.007+0000 7efff15b5700 1 -- start start 2024-03-18T20:28:15.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.008+0000 7efff15b5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effec0af200 0x7effec153f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.008+0000 7efff15b5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effec160c00 0x7effec147b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.008+0000 7efff15b5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effec160fe0 0x7effec15bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.008+0000 7efff15b5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effec15ef00 con 0x7effec160c00 2024-03-18T20:28:15.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.008+0000 7efff15b5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effec15e030 con 0x7effec160fe0 2024-03-18T20:28:15.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.008+0000 7efff15b5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effec15e2f0 con 0x7effec0af200 2024-03-18T20:28:15.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.008+0000 7effea7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effec160c00 0x7effec147b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.008+0000 7effea7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effec160c00 0x7effec147b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57272/0 (socket says 172.21.15.1:57272) 2024-03-18T20:28:15.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.008+0000 7effea7fc700 1 -- 172.21.15.1:0/3584337918 learned_addr learned my addr 172.21.15.1:0/3584337918 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:15.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.008+0000 7effeb7fe700 1 --2- 172.21.15.1:0/3584337918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effec160fe0 0x7effec15bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.008+0000 7effeaffd700 1 --2- 172.21.15.1:0/3584337918 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effec0af200 0x7effec153f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.008+0000 7effea7fc700 1 -- 172.21.15.1:0/3584337918 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effec0af200 msgr2=0x7effec153f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:15.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.008+0000 7effea7fc700 1 --2- 172.21.15.1:0/3584337918 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effec0af200 0x7effec153f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.008+0000 7effea7fc700 1 -- 172.21.15.1:0/3584337918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effec160fe0 msgr2=0x7effec15bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:15.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.008+0000 7effea7fc700 1 --2- 172.21.15.1:0/3584337918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effec160fe0 0x7effec15bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.008+0000 7effea7fc700 1 -- 172.21.15.1:0/3584337918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe4009b30 con 0x7effec160c00 2024-03-18T20:28:15.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.008+0000 7effeaffd700 1 --2- 172.21.15.1:0/3584337918 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effec0af200 0x7effec153f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:28:15.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.009+0000 7effea7fc700 1 --2- 172.21.15.1:0/3584337918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effec160c00 0x7effec147b00 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 crypto rx=0x7effe000c7f0 tx=0x7effe000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:15.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.009+0000 7effd37fe700 1 -- 172.21.15.1:0/3584337918 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe0011800 con 0x7effec160c00 2024-03-18T20:28:15.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.009+0000 7effd37fe700 1 -- 172.21.15.1:0/3584337918 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe0011960 con 0x7effec160c00 2024-03-18T20:28:15.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.009+0000 7efff15b5700 1 -- 172.21.15.1:0/3584337918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effec15e5d0 con 0x7effec160c00 2024-03-18T20:28:15.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.009+0000 7effd37fe700 1 -- 172.21.15.1:0/3584337918 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe0011cd0 con 0x7effec160c00 2024-03-18T20:28:15.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.009+0000 7efff15b5700 1 -- 172.21.15.1:0/3584337918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effec15eb90 con 0x7effec160c00 2024-03-18T20:28:15.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.010+0000 7efff15b5700 1 -- 172.21.15.1:0/3584337918 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effec067dc0 con 0x7effec160c00 2024-03-18T20:28:15.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.010+0000 7effd37fe700 1 -- 172.21.15.1:0/3584337918 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7effe0010350 con 0x7effec160c00 2024-03-18T20:28:15.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.011+0000 7effd37fe700 1 --2- 172.21.15.1:0/3584337918 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7effd806ade0 0x7effd806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.011+0000 7effd37fe700 1 -- 172.21.15.1:0/3584337918 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6129+0+0 (secure 0 0 0) 0x7effe0089270 con 0x7effec160c00 2024-03-18T20:28:15.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.011+0000 7effeaffd700 1 --2- 172.21.15.1:0/3584337918 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7effd806ade0 0x7effd806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.011+0000 7effeaffd700 1 --2- 172.21.15.1:0/3584337918 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7effd806ade0 0x7effd806d2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7effdc00aa90 tx=0x7effdc008040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:15.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.015+0000 7effd37fe700 1 -- 172.21.15.1:0/3584337918 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7effe0054000 con 0x7effec160c00 2024-03-18T20:28:15.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.215+0000 7efff15b5700 1 -- 172.21.15.1:0/3584337918 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7effec0b0610 con 0x7effd806ade0 2024-03-18T20:28:15.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.223+0000 7effd37fe700 1 -- 172.21.15.1:0/3584337918 <== mgr.11833 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289814 (secure 0 0 0) 0x7effdc00a810 con 0x7effd806ade0 2024-03-18T20:28:15.230 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:28:15.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.226+0000 7efff15b5700 1 -- 172.21.15.1:0/3584337918 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7effd806ade0 msgr2=0x7effd806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:15.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.226+0000 7efff15b5700 1 --2- 172.21.15.1:0/3584337918 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7effd806ade0 0x7effd806d2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7effdc00aa90 tx=0x7effdc008040 comp rx=0 tx=0).stop 2024-03-18T20:28:15.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.226+0000 7efff15b5700 1 -- 172.21.15.1:0/3584337918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effec160c00 msgr2=0x7effec147b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:15.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.226+0000 7efff15b5700 1 --2- 172.21.15.1:0/3584337918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effec160c00 0x7effec147b00 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 crypto rx=0x7effe000c7f0 tx=0x7effe000cb00 comp rx=0 tx=0).stop 2024-03-18T20:28:15.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.226+0000 7efff15b5700 1 -- 172.21.15.1:0/3584337918 shutdown_connections 2024-03-18T20:28:15.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.226+0000 7efff15b5700 1 --2- 172.21.15.1:0/3584337918 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7effd806ade0 0x7effd806d2a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.226+0000 7efff15b5700 1 --2- 172.21.15.1:0/3584337918 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7effec0af200 0x7effec153f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.226+0000 7efff15b5700 1 --2- 172.21.15.1:0/3584337918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effec160fe0 0x7effec15bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.226+0000 7efff15b5700 1 --2- 172.21.15.1:0/3584337918 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effec160c00 0x7effec147b00 unknown :-1 s=CLOSED pgs=3287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.226+0000 7efff15b5700 1 -- 172.21.15.1:0/3584337918 >> 172.21.15.1:0/3584337918 conn(0x7effec0b5780 msgr2=0x7effec06ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:15.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.226+0000 7efff15b5700 1 -- 172.21.15.1:0/3584337918 shutdown_connections 2024-03-18T20:28:15.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.226+0000 7efff15b5700 1 -- 172.21.15.1:0/3584337918 wait complete. 2024-03-18T20:28:15.233 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:28:15.242 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-18T20:28:15.164485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":723,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":441,"ondisk_log_size":441,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":449232,"kb_used_data":29672,"kb_used_omap":12,"kb_used_meta":419443,"kb_avail":754525488,"statfs":{"total":773094113280,"available":772634099712,"internally_reserved":0,"allocated":30384128,"data_stored":16290012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":429510201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.930461"},"pg_stats":[{"pgid":"38.3f","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.683766+0000","last_change":"2024-03-18T20:28:10.693223+0000","last_active":"2024-03-18T20:28:11.683766+0000","last_peered":"2024-03-18T20:28:11.683766+0000","last_clean":"2024-03-18T20:28:11.683766+0000","last_became_active":"2024-03-18T20:28:10.692995+0000","last_became_peered":"2024-03-18T20:28:10.692995+0000","last_unstale":"2024-03-18T20:28:11.683766+0000","last_undegraded":"2024-03-18T20:28:11.683766+0000","last_fullsized":"2024-03-18T20:28:11.683766+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:41:40.137234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.3e","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.864339+0000","last_change":"2024-03-18T20:28:10.694140+0000","last_active":"2024-03-18T20:28:11.864339+0000","last_peered":"2024-03-18T20:28:11.864339+0000","last_clean":"2024-03-18T20:28:11.864339+0000","last_became_active":"2024-03-18T20:28:10.691020+0000","last_became_peered":"2024-03-18T20:28:10.691020+0000","last_unstale":"2024-03-18T20:28:11.864339+0000","last_undegraded":"2024-03-18T20:28:11.864339+0000","last_fullsized":"2024-03-18T20:28:11.864339+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:25:01.428802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.3d","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245090+0000","last_change":"2024-03-18T20:28:10.691425+0000","last_active":"2024-03-18T20:28:12.245090+0000","last_peered":"2024-03-18T20:28:12.245090+0000","last_clean":"2024-03-18T20:28:12.245090+0000","last_became_active":"2024-03-18T20:28:10.691157+0000","last_became_peered":"2024-03-18T20:28:10.691157+0000","last_unstale":"2024-03-18T20:28:12.245090+0000","last_undegraded":"2024-03-18T20:28:12.245090+0000","last_fullsized":"2024-03-18T20:28:12.245090+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:56.377155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.3c","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245684+0000","last_change":"2024-03-18T20:28:10.696610+0000","last_active":"2024-03-18T20:28:12.245684+0000","last_peered":"2024-03-18T20:28:12.245684+0000","last_clean":"2024-03-18T20:28:12.245684+0000","last_became_active":"2024-03-18T20:28:10.696124+0000","last_became_peered":"2024-03-18T20:28:10.696124+0000","last_unstale":"2024-03-18T20:28:12.245684+0000","last_undegraded":"2024-03-18T20:28:12.245684+0000","last_fullsized":"2024-03-18T20:28:12.245684+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:10:23.902348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.3b","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246171+0000","last_change":"2024-03-18T20:28:10.688275+0000","last_active":"2024-03-18T20:28:12.246171+0000","last_peered":"2024-03-18T20:28:12.246171+0000","last_clean":"2024-03-18T20:28:12.246171+0000","last_became_active":"2024-03-18T20:28:10.687978+0000","last_became_peered":"2024-03-18T20:28:10.687978+0000","last_unstale":"2024-03-18T20:28:12.246171+0000","last_undegraded":"2024-03-18T20:28:12.246171+0000","last_fullsized":"2024-03-18T20:28:12.246171+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:25.186657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.3a","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246778+0000","last_change":"2024-03-18T20:28:10.695671+0000","last_active":"2024-03-18T20:28:12.246778+0000","last_peered":"2024-03-18T20:28:12.246778+0000","last_clean":"2024-03-18T20:28:12.246778+0000","last_became_active":"2024-03-18T20:28:10.695386+0000","last_became_peered":"2024-03-18T20:28:10.695386+0000","last_unstale":"2024-03-18T20:28:12.246778+0000","last_undegraded":"2024-03-18T20:28:12.246778+0000","last_fullsized":"2024-03-18T20:28:12.246778+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:04.311131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.39","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.679934+0000","last_change":"2024-03-18T20:28:10.677642+0000","last_active":"2024-03-18T20:28:11.679934+0000","last_peered":"2024-03-18T20:28:11.679934+0000","last_clean":"2024-03-18T20:28:11.679934+0000","last_became_active":"2024-03-18T20:28:10.677362+0000","last_became_peered":"2024-03-18T20:28:10.677362+0000","last_unstale":"2024-03-18T20:28:11.679934+0000","last_undegraded":"2024-03-18T20:28:11.679934+0000","last_fullsized":"2024-03-18T20:28:11.679934+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:54.278981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.38","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.683600+0000","last_change":"2024-03-18T20:28:10.688417+0000","last_active":"2024-03-18T20:28:11.683600+0000","last_peered":"2024-03-18T20:28:11.683600+0000","last_clean":"2024-03-18T20:28:11.683600+0000","last_became_active":"2024-03-18T20:28:10.688159+0000","last_became_peered":"2024-03-18T20:28:10.688159+0000","last_unstale":"2024-03-18T20:28:11.683600+0000","last_undegraded":"2024-03-18T20:28:11.683600+0000","last_fullsized":"2024-03-18T20:28:11.683600+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:13:42.720051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.37","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245619+0000","last_change":"2024-03-18T20:28:10.694142+0000","last_active":"2024-03-18T20:28:12.245619+0000","last_peered":"2024-03-18T20:28:12.245619+0000","last_clean":"2024-03-18T20:28:12.245619+0000","last_became_active":"2024-03-18T20:28:10.693695+0000","last_became_peered":"2024-03-18T20:28:10.693695+0000","last_unstale":"2024-03-18T20:28:12.245619+0000","last_undegraded":"2024-03-18T20:28:12.245619+0000","last_fullsized":"2024-03-18T20:28:12.245619+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:36:03.872267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.36","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246976+0000","last_change":"2024-03-18T20:28:10.688744+0000","last_active":"2024-03-18T20:28:12.246976+0000","last_peered":"2024-03-18T20:28:12.246976+0000","last_clean":"2024-03-18T20:28:12.246976+0000","last_became_active":"2024-03-18T20:28:10.686945+0000","last_became_peered":"2024-03-18T20:28:10.686945+0000","last_unstale":"2024-03-18T20:28:12.246976+0000","last_undegraded":"2024-03-18T20:28:12.246976+0000","last_fullsized":"2024-03-18T20:28:12.246976+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:17:45.776637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.35","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245238+0000","last_change":"2024-03-18T20:28:10.694666+0000","last_active":"2024-03-18T20:28:12.245238+0000","last_peered":"2024-03-18T20:28:12.245238+0000","last_clean":"2024-03-18T20:28:12.245238+0000","last_became_active":"2024-03-18T20:28:10.694006+0000","last_became_peered":"2024-03-18T20:28:10.694006+0000","last_unstale":"2024-03-18T20:28:12.245238+0000","last_undegraded":"2024-03-18T20:28:12.245238+0000","last_fullsized":"2024-03-18T20:28:12.245238+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:12:21.765997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.34","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.247350+0000","last_change":"2024-03-18T20:28:10.691626+0000","last_active":"2024-03-18T20:28:12.247350+0000","last_peered":"2024-03-18T20:28:12.247350+0000","last_clean":"2024-03-18T20:28:12.247350+0000","last_became_active":"2024-03-18T20:28:10.691361+0000","last_became_peered":"2024-03-18T20:28:10.691361+0000","last_unstale":"2024-03-18T20:28:12.247350+0000","last_undegraded":"2024-03-18T20:28:12.247350+0000","last_fullsized":"2024-03-18T20:28:12.247350+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:18:14.919929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.33","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246066+0000","last_change":"2024-03-18T20:28:10.694889+0000","last_active":"2024-03-18T20:28:12.246066+0000","last_peered":"2024-03-18T20:28:12.246066+0000","last_clean":"2024-03-18T20:28:12.246066+0000","last_became_active":"2024-03-18T20:28:10.694615+0000","last_became_peered":"2024-03-18T20:28:10.694615+0000","last_unstale":"2024-03-18T20:28:12.246066+0000","last_undegraded":"2024-03-18T20:28:12.246066+0000","last_fullsized":"2024-03-18T20:28:12.246066+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:04.311131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.32","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.863825+0000","last_change":"2024-03-18T20:28:10.697831+0000","last_active":"2024-03-18T20:28:11.863825+0000","last_peered":"2024-03-18T20:28:11.863825+0000","last_clean":"2024-03-18T20:28:11.863825+0000","last_became_active":"2024-03-18T20:28:10.697567+0000","last_became_peered":"2024-03-18T20:28:10.697567+0000","last_unstale":"2024-03-18T20:28:11.863825+0000","last_undegraded":"2024-03-18T20:28:11.863825+0000","last_fullsized":"2024-03-18T20:28:11.863825+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:58:37.579079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.31","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.863835+0000","last_change":"2024-03-18T20:28:10.691063+0000","last_active":"2024-03-18T20:28:11.863835+0000","last_peered":"2024-03-18T20:28:11.863835+0000","last_clean":"2024-03-18T20:28:11.863835+0000","last_became_active":"2024-03-18T20:28:10.690834+0000","last_became_peered":"2024-03-18T20:28:10.690834+0000","last_unstale":"2024-03-18T20:28:11.863835+0000","last_undegraded":"2024-03-18T20:28:11.863835+0000","last_fullsized":"2024-03-18T20:28:11.863835+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:11:29.538645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.30","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246367+0000","last_change":"2024-03-18T20:28:10.689351+0000","last_active":"2024-03-18T20:28:12.246367+0000","last_peered":"2024-03-18T20:28:12.246367+0000","last_clean":"2024-03-18T20:28:12.246367+0000","last_became_active":"2024-03-18T20:28:10.687209+0000","last_became_peered":"2024-03-18T20:28:10.687209+0000","last_unstale":"2024-03-18T20:28:12.246367+0000","last_undegraded":"2024-03-18T20:28:12.246367+0000","last_fullsized":"2024-03-18T20:28:12.246367+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:13:42.720051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.2f","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245652+0000","last_change":"2024-03-18T20:28:10.693944+0000","last_active":"2024-03-18T20:28:12.245652+0000","last_peered":"2024-03-18T20:28:12.245652+0000","last_clean":"2024-03-18T20:28:12.245652+0000","last_became_active":"2024-03-18T20:28:10.693393+0000","last_became_peered":"2024-03-18T20:28:10.693393+0000","last_unstale":"2024-03-18T20:28:12.245652+0000","last_undegraded":"2024-03-18T20:28:12.245652+0000","last_fullsized":"2024-03-18T20:28:12.245652+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:22:02.619215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.2e","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246346+0000","last_change":"2024-03-18T20:28:10.692666+0000","last_active":"2024-03-18T20:28:12.246346+0000","last_peered":"2024-03-18T20:28:12.246346+0000","last_clean":"2024-03-18T20:28:12.246346+0000","last_became_active":"2024-03-18T20:28:10.692222+0000","last_became_peered":"2024-03-18T20:28:10.692222+0000","last_unstale":"2024-03-18T20:28:12.246346+0000","last_undegraded":"2024-03-18T20:28:12.246346+0000","last_fullsized":"2024-03-18T20:28:12.246346+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:28:40.101291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2d","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245149+0000","last_change":"2024-03-18T20:28:10.690869+0000","last_active":"2024-03-18T20:28:12.245149+0000","last_peered":"2024-03-18T20:28:12.245149+0000","last_clean":"2024-03-18T20:28:12.245149+0000","last_became_active":"2024-03-18T20:28:10.689853+0000","last_became_peered":"2024-03-18T20:28:10.689853+0000","last_unstale":"2024-03-18T20:28:12.245149+0000","last_undegraded":"2024-03-18T20:28:12.245149+0000","last_fullsized":"2024-03-18T20:28:12.245149+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:21:24.668231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.2c","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245797+0000","last_change":"2024-03-18T20:28:10.696409+0000","last_active":"2024-03-18T20:28:12.245797+0000","last_peered":"2024-03-18T20:28:12.245797+0000","last_clean":"2024-03-18T20:28:12.245797+0000","last_became_active":"2024-03-18T20:28:10.691837+0000","last_became_peered":"2024-03-18T20:28:10.691837+0000","last_unstale":"2024-03-18T20:28:12.245797+0000","last_undegraded":"2024-03-18T20:28:12.245797+0000","last_fullsized":"2024-03-18T20:28:12.245797+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:42:10.004825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.2b","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.863919+0000","last_change":"2024-03-18T20:28:10.689584+0000","last_active":"2024-03-18T20:28:11.863919+0000","last_peered":"2024-03-18T20:28:11.863919+0000","last_clean":"2024-03-18T20:28:11.863919+0000","last_became_active":"2024-03-18T20:28:10.689365+0000","last_became_peered":"2024-03-18T20:28:10.689365+0000","last_unstale":"2024-03-18T20:28:11.863919+0000","last_undegraded":"2024-03-18T20:28:11.863919+0000","last_fullsized":"2024-03-18T20:28:11.863919+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:52:14.893372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.2a","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246025+0000","last_change":"2024-03-18T20:28:10.691396+0000","last_active":"2024-03-18T20:28:12.246025+0000","last_peered":"2024-03-18T20:28:12.246025+0000","last_clean":"2024-03-18T20:28:12.246025+0000","last_became_active":"2024-03-18T20:28:10.690158+0000","last_became_peered":"2024-03-18T20:28:10.690158+0000","last_unstale":"2024-03-18T20:28:12.246025+0000","last_undegraded":"2024-03-18T20:28:12.246025+0000","last_fullsized":"2024-03-18T20:28:12.246025+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:18.038073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.29","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245933+0000","last_change":"2024-03-18T20:28:10.686763+0000","last_active":"2024-03-18T20:28:12.245933+0000","last_peered":"2024-03-18T20:28:12.245933+0000","last_clean":"2024-03-18T20:28:12.245933+0000","last_became_active":"2024-03-18T20:28:10.686336+0000","last_became_peered":"2024-03-18T20:28:10.686336+0000","last_unstale":"2024-03-18T20:28:12.245933+0000","last_undegraded":"2024-03-18T20:28:12.245933+0000","last_fullsized":"2024-03-18T20:28:12.245933+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:23:20.359399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.28","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.244719+0000","last_change":"2024-03-18T20:28:10.686197+0000","last_active":"2024-03-18T20:28:12.244719+0000","last_peered":"2024-03-18T20:28:12.244719+0000","last_clean":"2024-03-18T20:28:12.244719+0000","last_became_active":"2024-03-18T20:28:10.686012+0000","last_became_peered":"2024-03-18T20:28:10.686012+0000","last_unstale":"2024-03-18T20:28:12.244719+0000","last_undegraded":"2024-03-18T20:28:12.244719+0000","last_fullsized":"2024-03-18T20:28:12.244719+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:13:13.614656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.26","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245611+0000","last_change":"2024-03-18T20:28:10.699682+0000","last_active":"2024-03-18T20:28:12.245611+0000","last_peered":"2024-03-18T20:28:12.245611+0000","last_clean":"2024-03-18T20:28:12.245611+0000","last_became_active":"2024-03-18T20:28:10.699417+0000","last_became_peered":"2024-03-18T20:28:10.699417+0000","last_unstale":"2024-03-18T20:28:12.245611+0000","last_undegraded":"2024-03-18T20:28:12.245611+0000","last_fullsized":"2024-03-18T20:28:12.245611+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:50:29.488405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.25","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246565+0000","last_change":"2024-03-18T20:28:10.688937+0000","last_active":"2024-03-18T20:28:12.246565+0000","last_peered":"2024-03-18T20:28:12.246565+0000","last_clean":"2024-03-18T20:28:12.246565+0000","last_became_active":"2024-03-18T20:28:10.686441+0000","last_became_peered":"2024-03-18T20:28:10.686441+0000","last_unstale":"2024-03-18T20:28:12.246565+0000","last_undegraded":"2024-03-18T20:28:12.246565+0000","last_fullsized":"2024-03-18T20:28:12.246565+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:26:04.526911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.24","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245454+0000","last_change":"2024-03-18T20:28:10.690417+0000","last_active":"2024-03-18T20:28:12.245454+0000","last_peered":"2024-03-18T20:28:12.245454+0000","last_clean":"2024-03-18T20:28:12.245454+0000","last_became_active":"2024-03-18T20:28:10.689963+0000","last_became_peered":"2024-03-18T20:28:10.689963+0000","last_unstale":"2024-03-18T20:28:12.245454+0000","last_undegraded":"2024-03-18T20:28:12.245454+0000","last_fullsized":"2024-03-18T20:28:12.245454+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:05:28.077758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.23","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.680119+0000","last_change":"2024-03-18T20:28:10.682430+0000","last_active":"2024-03-18T20:28:11.680119+0000","last_peered":"2024-03-18T20:28:11.680119+0000","last_clean":"2024-03-18T20:28:11.680119+0000","last_became_active":"2024-03-18T20:28:10.681171+0000","last_became_peered":"2024-03-18T20:28:10.681171+0000","last_unstale":"2024-03-18T20:28:11.680119+0000","last_undegraded":"2024-03-18T20:28:11.680119+0000","last_fullsized":"2024-03-18T20:28:11.680119+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:12:44.829333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.22","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.863922+0000","last_change":"2024-03-18T20:28:10.690635+0000","last_active":"2024-03-18T20:28:11.863922+0000","last_peered":"2024-03-18T20:28:11.863922+0000","last_clean":"2024-03-18T20:28:11.863922+0000","last_became_active":"2024-03-18T20:28:10.690386+0000","last_became_peered":"2024-03-18T20:28:10.690386+0000","last_unstale":"2024-03-18T20:28:11.863922+0000","last_undegraded":"2024-03-18T20:28:11.863922+0000","last_fullsized":"2024-03-18T20:28:11.863922+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:47:49.100278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.21","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.679925+0000","last_change":"2024-03-18T20:28:10.694653+0000","last_active":"2024-03-18T20:28:11.679925+0000","last_peered":"2024-03-18T20:28:11.679925+0000","last_clean":"2024-03-18T20:28:11.679925+0000","last_became_active":"2024-03-18T20:28:10.694369+0000","last_became_peered":"2024-03-18T20:28:10.694369+0000","last_unstale":"2024-03-18T20:28:11.679925+0000","last_undegraded":"2024-03-18T20:28:11.679925+0000","last_fullsized":"2024-03-18T20:28:11.679925+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:18:14.919929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.20","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245554+0000","last_change":"2024-03-18T20:28:10.696083+0000","last_active":"2024-03-18T20:28:12.245554+0000","last_peered":"2024-03-18T20:28:12.245554+0000","last_clean":"2024-03-18T20:28:12.245554+0000","last_became_active":"2024-03-18T20:28:10.695545+0000","last_became_peered":"2024-03-18T20:28:10.695545+0000","last_unstale":"2024-03-18T20:28:12.245554+0000","last_undegraded":"2024-03-18T20:28:12.245554+0000","last_fullsized":"2024-03-18T20:28:12.245554+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:01:46.023861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1c","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245951+0000","last_change":"2024-03-18T20:28:10.694987+0000","last_active":"2024-03-18T20:28:12.245951+0000","last_peered":"2024-03-18T20:28:12.245951+0000","last_clean":"2024-03-18T20:28:12.245951+0000","last_became_active":"2024-03-18T20:28:10.694716+0000","last_became_peered":"2024-03-18T20:28:10.694716+0000","last_unstale":"2024-03-18T20:28:12.245951+0000","last_undegraded":"2024-03-18T20:28:12.245951+0000","last_fullsized":"2024-03-18T20:28:12.245951+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:08:38.855285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1f","version":"287'4","reported_seq":15,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246166+0000","last_change":"2024-03-18T20:28:10.255502+0000","last_active":"2024-03-18T20:28:12.246166+0000","last_peered":"2024-03-18T20:28:12.246166+0000","last_clean":"2024-03-18T20:28:12.246166+0000","last_became_active":"2024-03-18T20:28:10.255242+0000","last_became_peered":"2024-03-18T20:28:10.255242+0000","last_unstale":"2024-03-18T20:28:12.246166+0000","last_undegraded":"2024-03-18T20:28:12.246166+0000","last_fullsized":"2024-03-18T20:28:12.246166+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:47.380054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.1d","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245335+0000","last_change":"2024-03-18T20:28:10.686116+0000","last_active":"2024-03-18T20:28:12.245335+0000","last_peered":"2024-03-18T20:28:12.245335+0000","last_clean":"2024-03-18T20:28:12.245335+0000","last_became_active":"2024-03-18T20:28:10.685838+0000","last_became_peered":"2024-03-18T20:28:10.685838+0000","last_unstale":"2024-03-18T20:28:12.245335+0000","last_undegraded":"2024-03-18T20:28:12.245335+0000","last_fullsized":"2024-03-18T20:28:12.245335+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:13:13.367186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1e","version":"287'1","reported_seq":12,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246469+0000","last_change":"2024-03-18T20:28:10.252517+0000","last_active":"2024-03-18T20:28:12.246469+0000","last_peered":"2024-03-18T20:28:12.246469+0000","last_clean":"2024-03-18T20:28:12.246469+0000","last_became_active":"2024-03-18T20:28:10.252231+0000","last_became_peered":"2024-03-18T20:28:10.252231+0000","last_unstale":"2024-03-18T20:28:12.246469+0000","last_undegraded":"2024-03-18T20:28:12.246469+0000","last_fullsized":"2024-03-18T20:28:12.246469+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:13:41.714453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1e","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.864223+0000","last_change":"2024-03-18T20:28:10.694080+0000","last_active":"2024-03-18T20:28:11.864223+0000","last_peered":"2024-03-18T20:28:11.864223+0000","last_clean":"2024-03-18T20:28:11.864223+0000","last_became_active":"2024-03-18T20:28:10.690709+0000","last_became_peered":"2024-03-18T20:28:10.690709+0000","last_unstale":"2024-03-18T20:28:11.864223+0000","last_undegraded":"2024-03-18T20:28:11.864223+0000","last_fullsized":"2024-03-18T20:28:11.864223+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:04:49.202387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"287'2","reported_seq":13,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245203+0000","last_change":"2024-03-18T20:28:10.256247+0000","last_active":"2024-03-18T20:28:12.245203+0000","last_peered":"2024-03-18T20:28:12.245203+0000","last_clean":"2024-03-18T20:28:12.245203+0000","last_became_active":"2024-03-18T20:28:10.255731+0000","last_became_peered":"2024-03-18T20:28:10.255731+0000","last_unstale":"2024-03-18T20:28:12.245203+0000","last_undegraded":"2024-03-18T20:28:12.245203+0000","last_fullsized":"2024-03-18T20:28:12.245203+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:58.779282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1f","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.864073+0000","last_change":"2024-03-18T20:28:10.695782+0000","last_active":"2024-03-18T20:28:11.864073+0000","last_peered":"2024-03-18T20:28:11.864073+0000","last_clean":"2024-03-18T20:28:11.864073+0000","last_became_active":"2024-03-18T20:28:10.691224+0000","last_became_peered":"2024-03-18T20:28:10.691224+0000","last_unstale":"2024-03-18T20:28:11.864073+0000","last_undegraded":"2024-03-18T20:28:11.864073+0000","last_fullsized":"2024-03-18T20:28:11.864073+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:19.563671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"287'2","reported_seq":13,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.864060+0000","last_change":"2024-03-18T20:28:10.257430+0000","last_active":"2024-03-18T20:28:11.864060+0000","last_peered":"2024-03-18T20:28:11.864060+0000","last_clean":"2024-03-18T20:28:11.864060+0000","last_became_active":"2024-03-18T20:28:10.254957+0000","last_became_peered":"2024-03-18T20:28:10.254957+0000","last_unstale":"2024-03-18T20:28:11.864060+0000","last_undegraded":"2024-03-18T20:28:11.864060+0000","last_fullsized":"2024-03-18T20:28:11.864060+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:55.371558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.18","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.683630+0000","last_change":"2024-03-18T20:28:10.687909+0000","last_active":"2024-03-18T20:28:11.683630+0000","last_peered":"2024-03-18T20:28:11.683630+0000","last_clean":"2024-03-18T20:28:11.683630+0000","last_became_active":"2024-03-18T20:28:10.687626+0000","last_became_peered":"2024-03-18T20:28:10.687626+0000","last_unstale":"2024-03-18T20:28:11.683630+0000","last_undegraded":"2024-03-18T20:28:11.683630+0000","last_fullsized":"2024-03-18T20:28:11.683630+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:21:13.398524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1b","version":"287'1","reported_seq":12,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245333+0000","last_change":"2024-03-18T20:28:10.254958+0000","last_active":"2024-03-18T20:28:12.245333+0000","last_peered":"2024-03-18T20:28:12.245333+0000","last_clean":"2024-03-18T20:28:12.245333+0000","last_became_active":"2024-03-18T20:28:10.254665+0000","last_became_peered":"2024-03-18T20:28:10.254665+0000","last_unstale":"2024-03-18T20:28:12.245333+0000","last_undegraded":"2024-03-18T20:28:12.245333+0000","last_fullsized":"2024-03-18T20:28:12.245333+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:33:42.452709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.19","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245053+0000","last_change":"2024-03-18T20:28:10.698117+0000","last_active":"2024-03-18T20:28:12.245053+0000","last_peered":"2024-03-18T20:28:12.245053+0000","last_clean":"2024-03-18T20:28:12.245053+0000","last_became_active":"2024-03-18T20:28:10.697822+0000","last_became_peered":"2024-03-18T20:28:10.697822+0000","last_unstale":"2024-03-18T20:28:12.245053+0000","last_undegraded":"2024-03-18T20:28:12.245053+0000","last_fullsized":"2024-03-18T20:28:12.245053+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:47:32.592333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1a","version":"287'1","reported_seq":12,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245095+0000","last_change":"2024-03-18T20:28:10.251064+0000","last_active":"2024-03-18T20:28:12.245095+0000","last_peered":"2024-03-18T20:28:12.245095+0000","last_clean":"2024-03-18T20:28:12.245095+0000","last_became_active":"2024-03-18T20:28:10.250767+0000","last_became_peered":"2024-03-18T20:28:10.250767+0000","last_unstale":"2024-03-18T20:28:12.245095+0000","last_undegraded":"2024-03-18T20:28:12.245095+0000","last_fullsized":"2024-03-18T20:28:12.245095+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:28:39.095694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1a","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245816+0000","last_change":"2024-03-18T20:28:10.691283+0000","last_active":"2024-03-18T20:28:12.245816+0000","last_peered":"2024-03-18T20:28:12.245816+0000","last_clean":"2024-03-18T20:28:12.245816+0000","last_became_active":"2024-03-18T20:28:10.689800+0000","last_became_peered":"2024-03-18T20:28:10.689800+0000","last_unstale":"2024-03-18T20:28:12.245816+0000","last_undegraded":"2024-03-18T20:28:12.245816+0000","last_fullsized":"2024-03-18T20:28:12.245816+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:13.897647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.863931+0000","last_change":"2024-03-18T20:28:10.258284+0000","last_active":"2024-03-18T20:28:11.863931+0000","last_peered":"2024-03-18T20:28:11.863931+0000","last_clean":"2024-03-18T20:28:11.863931+0000","last_became_active":"2024-03-18T20:28:10.257890+0000","last_became_peered":"2024-03-18T20:28:10.257890+0000","last_unstale":"2024-03-18T20:28:11.863931+0000","last_undegraded":"2024-03-18T20:28:11.863931+0000","last_fullsized":"2024-03-18T20:28:11.863931+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:13:12.609059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.1b","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.863969+0000","last_change":"2024-03-18T20:28:10.696848+0000","last_active":"2024-03-18T20:28:11.863969+0000","last_peered":"2024-03-18T20:28:11.863969+0000","last_clean":"2024-03-18T20:28:11.863969+0000","last_became_active":"2024-03-18T20:28:10.696600+0000","last_became_peered":"2024-03-18T20:28:10.696600+0000","last_unstale":"2024-03-18T20:28:11.863969+0000","last_undegraded":"2024-03-18T20:28:11.863969+0000","last_fullsized":"2024-03-18T20:28:11.863969+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:49.431262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.679797+0000","last_change":"2024-03-18T20:28:10.249588+0000","last_active":"2024-03-18T20:28:11.679797+0000","last_peered":"2024-03-18T20:28:11.679797+0000","last_clean":"2024-03-18T20:28:11.679797+0000","last_became_active":"2024-03-18T20:28:10.249316+0000","last_became_peered":"2024-03-18T20:28:10.249316+0000","last_unstale":"2024-03-18T20:28:11.679797+0000","last_undegraded":"2024-03-18T20:28:11.679797+0000","last_fullsized":"2024-03-18T20:28:11.679797+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:37:47.380054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.14","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246130+0000","last_change":"2024-03-18T20:28:10.689984+0000","last_active":"2024-03-18T20:28:12.246130+0000","last_peered":"2024-03-18T20:28:12.246130+0000","last_clean":"2024-03-18T20:28:12.246130+0000","last_became_active":"2024-03-18T20:28:10.687856+0000","last_became_peered":"2024-03-18T20:28:10.687856+0000","last_unstale":"2024-03-18T20:28:12.246130+0000","last_undegraded":"2024-03-18T20:28:12.246130+0000","last_fullsized":"2024-03-18T20:28:12.246130+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:17:11.428228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246271+0000","last_change":"2024-03-18T20:28:10.253423+0000","last_active":"2024-03-18T20:28:12.246271+0000","last_peered":"2024-03-18T20:28:12.246271+0000","last_clean":"2024-03-18T20:28:12.246271+0000","last_became_active":"2024-03-18T20:28:10.253093+0000","last_became_peered":"2024-03-18T20:28:10.253093+0000","last_unstale":"2024-03-18T20:28:12.246271+0000","last_undegraded":"2024-03-18T20:28:12.246271+0000","last_fullsized":"2024-03-18T20:28:12.246271+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:30:59.318354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.15","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246855+0000","last_change":"2024-03-18T20:28:10.696197+0000","last_active":"2024-03-18T20:28:12.246855+0000","last_peered":"2024-03-18T20:28:12.246855+0000","last_clean":"2024-03-18T20:28:12.246855+0000","last_became_active":"2024-03-18T20:28:10.694929+0000","last_became_peered":"2024-03-18T20:28:10.694929+0000","last_unstale":"2024-03-18T20:28:12.246855+0000","last_undegraded":"2024-03-18T20:28:12.246855+0000","last_fullsized":"2024-03-18T20:28:12.246855+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:12:44.829333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.864295+0000","last_change":"2024-03-18T20:28:10.259209+0000","last_active":"2024-03-18T20:28:11.864295+0000","last_peered":"2024-03-18T20:28:11.864295+0000","last_clean":"2024-03-18T20:28:11.864295+0000","last_became_active":"2024-03-18T20:28:10.258910+0000","last_became_peered":"2024-03-18T20:28:10.258910+0000","last_unstale":"2024-03-18T20:28:11.864295+0000","last_undegraded":"2024-03-18T20:28:11.864295+0000","last_fullsized":"2024-03-18T20:28:11.864295+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:50:43.887460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.16","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.864179+0000","last_change":"2024-03-18T20:28:10.694263+0000","last_active":"2024-03-18T20:28:11.864179+0000","last_peered":"2024-03-18T20:28:11.864179+0000","last_clean":"2024-03-18T20:28:11.864179+0000","last_became_active":"2024-03-18T20:28:10.693734+0000","last_became_peered":"2024-03-18T20:28:10.693734+0000","last_unstale":"2024-03-18T20:28:11.864179+0000","last_undegraded":"2024-03-18T20:28:11.864179+0000","last_fullsized":"2024-03-18T20:28:11.864179+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:07:04.444530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.15","version":"287'3","reported_seq":14,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.864120+0000","last_change":"2024-03-18T20:28:10.257889+0000","last_active":"2024-03-18T20:28:11.864120+0000","last_peered":"2024-03-18T20:28:11.864120+0000","last_clean":"2024-03-18T20:28:11.864120+0000","last_became_active":"2024-03-18T20:28:10.257584+0000","last_became_peered":"2024-03-18T20:28:10.257584+0000","last_unstale":"2024-03-18T20:28:11.864120+0000","last_undegraded":"2024-03-18T20:28:11.864120+0000","last_fullsized":"2024-03-18T20:28:11.864120+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:16:18.403057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.17","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245595+0000","last_change":"2024-03-18T20:28:10.688268+0000","last_active":"2024-03-18T20:28:12.245595+0000","last_peered":"2024-03-18T20:28:12.245595+0000","last_clean":"2024-03-18T20:28:12.245595+0000","last_became_active":"2024-03-18T20:28:10.687999+0000","last_became_peered":"2024-03-18T20:28:10.687999+0000","last_unstale":"2024-03-18T20:28:12.245595+0000","last_undegraded":"2024-03-18T20:28:12.245595+0000","last_fullsized":"2024-03-18T20:28:12.245595+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:59.784879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.14","version":"287'6","reported_seq":17,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.683823+0000","last_change":"2024-03-18T20:28:10.251818+0000","last_active":"2024-03-18T20:28:11.683823+0000","last_peered":"2024-03-18T20:28:11.683823+0000","last_clean":"2024-03-18T20:28:11.683823+0000","last_became_active":"2024-03-18T20:28:10.251254+0000","last_became_peered":"2024-03-18T20:28:10.251254+0000","last_unstale":"2024-03-18T20:28:11.683823+0000","last_undegraded":"2024-03-18T20:28:11.683823+0000","last_fullsized":"2024-03-18T20:28:11.683823+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:15:12.771032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.10","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.244919+0000","last_change":"2024-03-18T20:28:10.687448+0000","last_active":"2024-03-18T20:28:12.244919+0000","last_peered":"2024-03-18T20:28:12.244919+0000","last_clean":"2024-03-18T20:28:12.244919+0000","last_became_active":"2024-03-18T20:28:10.687056+0000","last_became_peered":"2024-03-18T20:28:10.687056+0000","last_unstale":"2024-03-18T20:28:12.244919+0000","last_undegraded":"2024-03-18T20:28:12.244919+0000","last_fullsized":"2024-03-18T20:28:12.244919+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:30.714033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.13","version":"287'2","reported_seq":13,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.247087+0000","last_change":"2024-03-18T20:28:10.250593+0000","last_active":"2024-03-18T20:28:12.247087+0000","last_peered":"2024-03-18T20:28:12.247087+0000","last_clean":"2024-03-18T20:28:12.247087+0000","last_became_active":"2024-03-18T20:28:10.250337+0000","last_became_peered":"2024-03-18T20:28:10.250337+0000","last_unstale":"2024-03-18T20:28:12.247087+0000","last_undegraded":"2024-03-18T20:28:12.247087+0000","last_fullsized":"2024-03-18T20:28:12.247087+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:01:40.106560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.11","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245860+0000","last_change":"2024-03-18T20:28:10.689489+0000","last_active":"2024-03-18T20:28:12.245860+0000","last_peered":"2024-03-18T20:28:12.245860+0000","last_clean":"2024-03-18T20:28:12.245860+0000","last_became_active":"2024-03-18T20:28:10.689208+0000","last_became_peered":"2024-03-18T20:28:10.689208+0000","last_unstale":"2024-03-18T20:28:12.245860+0000","last_undegraded":"2024-03-18T20:28:12.245860+0000","last_fullsized":"2024-03-18T20:28:12.245860+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:12:44.829333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246463+0000","last_change":"2024-03-18T20:28:10.258636+0000","last_active":"2024-03-18T20:28:12.246463+0000","last_peered":"2024-03-18T20:28:12.246463+0000","last_clean":"2024-03-18T20:28:12.246463+0000","last_became_active":"2024-03-18T20:28:10.258378+0000","last_became_peered":"2024-03-18T20:28:10.258378+0000","last_unstale":"2024-03-18T20:28:12.246463+0000","last_undegraded":"2024-03-18T20:28:12.246463+0000","last_fullsized":"2024-03-18T20:28:12.246463+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:57:15.765777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.12","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245936+0000","last_change":"2024-03-18T20:28:10.692634+0000","last_active":"2024-03-18T20:28:12.245936+0000","last_peered":"2024-03-18T20:28:12.245936+0000","last_clean":"2024-03-18T20:28:12.245936+0000","last_became_active":"2024-03-18T20:28:10.692255+0000","last_became_peered":"2024-03-18T20:28:10.692255+0000","last_unstale":"2024-03-18T20:28:12.245936+0000","last_undegraded":"2024-03-18T20:28:12.245936+0000","last_fullsized":"2024-03-18T20:28:12.245936+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:17:18.898257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246165+0000","last_change":"2024-03-18T20:28:10.260584+0000","last_active":"2024-03-18T20:28:12.246165+0000","last_peered":"2024-03-18T20:28:12.246165+0000","last_clean":"2024-03-18T20:28:12.246165+0000","last_became_active":"2024-03-18T20:28:10.260349+0000","last_became_peered":"2024-03-18T20:28:10.260349+0000","last_unstale":"2024-03-18T20:28:12.246165+0000","last_undegraded":"2024-03-18T20:28:12.246165+0000","last_fullsized":"2024-03-18T20:28:12.246165+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:59:44.397544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.13","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246782+0000","last_change":"2024-03-18T20:28:10.694255+0000","last_active":"2024-03-18T20:28:12.246782+0000","last_peered":"2024-03-18T20:28:12.246782+0000","last_clean":"2024-03-18T20:28:12.246782+0000","last_became_active":"2024-03-18T20:28:10.694013+0000","last_became_peered":"2024-03-18T20:28:10.694013+0000","last_unstale":"2024-03-18T20:28:12.246782+0000","last_undegraded":"2024-03-18T20:28:12.246782+0000","last_fullsized":"2024-03-18T20:28:12.246782+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:41:40.137234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.10","version":"287'8","reported_seq":19,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245874+0000","last_change":"2024-03-18T20:28:10.254130+0000","last_active":"2024-03-18T20:28:12.245874+0000","last_peered":"2024-03-18T20:28:12.245874+0000","last_clean":"2024-03-18T20:28:12.245874+0000","last_became_active":"2024-03-18T20:28:10.253861+0000","last_became_peered":"2024-03-18T20:28:10.253861+0000","last_unstale":"2024-03-18T20:28:12.245874+0000","last_undegraded":"2024-03-18T20:28:12.245874+0000","last_fullsized":"2024-03-18T20:28:12.245874+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:17:44.771039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.c","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.683917+0000","last_change":"2024-03-18T20:28:10.696978+0000","last_active":"2024-03-18T20:28:11.683917+0000","last_peered":"2024-03-18T20:28:11.683917+0000","last_clean":"2024-03-18T20:28:11.683917+0000","last_became_active":"2024-03-18T20:28:10.696718+0000","last_became_peered":"2024-03-18T20:28:10.696718+0000","last_unstale":"2024-03-18T20:28:11.683917+0000","last_undegraded":"2024-03-18T20:28:11.683917+0000","last_fullsized":"2024-03-18T20:28:11.683917+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:17:45.776637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.f","version":"287'1","reported_seq":12,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246452+0000","last_change":"2024-03-18T20:28:10.256047+0000","last_active":"2024-03-18T20:28:12.246452+0000","last_peered":"2024-03-18T20:28:12.246452+0000","last_clean":"2024-03-18T20:28:12.246452+0000","last_became_active":"2024-03-18T20:28:10.255781+0000","last_became_peered":"2024-03-18T20:28:10.255781+0000","last_unstale":"2024-03-18T20:28:12.246452+0000","last_undegraded":"2024-03-18T20:28:12.246452+0000","last_fullsized":"2024-03-18T20:28:12.246452+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:41:39.131637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.d","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245605+0000","last_change":"2024-03-18T20:28:10.690584+0000","last_active":"2024-03-18T20:28:12.245605+0000","last_peered":"2024-03-18T20:28:12.245605+0000","last_clean":"2024-03-18T20:28:12.245605+0000","last_became_active":"2024-03-18T20:28:10.690314+0000","last_became_peered":"2024-03-18T20:28:10.690314+0000","last_unstale":"2024-03-18T20:28:12.245605+0000","last_undegraded":"2024-03-18T20:28:12.245605+0000","last_fullsized":"2024-03-18T20:28:12.245605+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:27:05.233111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.e","version":"287'2","reported_seq":13,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.683820+0000","last_change":"2024-03-18T20:28:10.254831+0000","last_active":"2024-03-18T20:28:11.683820+0000","last_peered":"2024-03-18T20:28:11.683820+0000","last_clean":"2024-03-18T20:28:11.683820+0000","last_became_active":"2024-03-18T20:28:10.254472+0000","last_became_peered":"2024-03-18T20:28:10.254472+0000","last_unstale":"2024-03-18T20:28:11.683820+0000","last_undegraded":"2024-03-18T20:28:11.683820+0000","last_fullsized":"2024-03-18T20:28:11.683820+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:49:53.273384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.e","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.247006+0000","last_change":"2024-03-18T20:28:10.695498+0000","last_active":"2024-03-18T20:28:12.247006+0000","last_peered":"2024-03-18T20:28:12.247006+0000","last_clean":"2024-03-18T20:28:12.247006+0000","last_became_active":"2024-03-18T20:28:10.695219+0000","last_became_peered":"2024-03-18T20:28:10.695219+0000","last_unstale":"2024-03-18T20:28:12.247006+0000","last_undegraded":"2024-03-18T20:28:12.247006+0000","last_fullsized":"2024-03-18T20:28:12.247006+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:21:13.398524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.d","version":"287'1","reported_seq":12,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245704+0000","last_change":"2024-03-18T20:28:10.252068+0000","last_active":"2024-03-18T20:28:12.245704+0000","last_peered":"2024-03-18T20:28:12.245704+0000","last_clean":"2024-03-18T20:28:12.245704+0000","last_became_active":"2024-03-18T20:28:10.251309+0000","last_became_peered":"2024-03-18T20:28:10.251309+0000","last_unstale":"2024-03-18T20:28:12.245704+0000","last_undegraded":"2024-03-18T20:28:12.245704+0000","last_fullsized":"2024-03-18T20:28:12.245704+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:01:40.106560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.f","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245318+0000","last_change":"2024-03-18T20:28:10.688969+0000","last_active":"2024-03-18T20:28:12.245318+0000","last_peered":"2024-03-18T20:28:12.245318+0000","last_clean":"2024-03-18T20:28:12.245318+0000","last_became_active":"2024-03-18T20:28:10.688412+0000","last_became_peered":"2024-03-18T20:28:10.688412+0000","last_unstale":"2024-03-18T20:28:12.245318+0000","last_undegraded":"2024-03-18T20:28:12.245318+0000","last_fullsized":"2024-03-18T20:28:12.245318+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:46.507265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.864098+0000","last_change":"2024-03-18T20:28:10.253290+0000","last_active":"2024-03-18T20:28:11.864098+0000","last_peered":"2024-03-18T20:28:11.864098+0000","last_clean":"2024-03-18T20:28:11.864098+0000","last_became_active":"2024-03-18T20:28:10.252950+0000","last_became_peered":"2024-03-18T20:28:10.252950+0000","last_unstale":"2024-03-18T20:28:11.864098+0000","last_undegraded":"2024-03-18T20:28:11.864098+0000","last_fullsized":"2024-03-18T20:28:11.864098+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:21:23.662634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.8","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246094+0000","last_change":"2024-03-18T20:28:10.687864+0000","last_active":"2024-03-18T20:28:12.246094+0000","last_peered":"2024-03-18T20:28:12.246094+0000","last_clean":"2024-03-18T20:28:12.246094+0000","last_became_active":"2024-03-18T20:28:10.686313+0000","last_became_peered":"2024-03-18T20:28:10.686313+0000","last_unstale":"2024-03-18T20:28:12.246094+0000","last_undegraded":"2024-03-18T20:28:12.246094+0000","last_fullsized":"2024-03-18T20:28:12.246094+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:27:14.212049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.b","version":"287'2","reported_seq":13,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245414+0000","last_change":"2024-03-18T20:28:10.251075+0000","last_active":"2024-03-18T20:28:12.245414+0000","last_peered":"2024-03-18T20:28:12.245414+0000","last_clean":"2024-03-18T20:28:12.245414+0000","last_became_active":"2024-03-18T20:28:10.250694+0000","last_became_peered":"2024-03-18T20:28:10.250694+0000","last_unstale":"2024-03-18T20:28:12.245414+0000","last_undegraded":"2024-03-18T20:28:12.245414+0000","last_fullsized":"2024-03-18T20:28:12.245414+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:21:00.221902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.9","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245153+0000","last_change":"2024-03-18T20:28:10.697975+0000","last_active":"2024-03-18T20:28:12.245153+0000","last_peered":"2024-03-18T20:28:12.245153+0000","last_clean":"2024-03-18T20:28:12.245153+0000","last_became_active":"2024-03-18T20:28:10.697579+0000","last_became_peered":"2024-03-18T20:28:10.697579+0000","last_unstale":"2024-03-18T20:28:12.245153+0000","last_undegraded":"2024-03-18T20:28:12.245153+0000","last_fullsized":"2024-03-18T20:28:12.245153+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:23:16.308223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246039+0000","last_change":"2024-03-18T20:28:10.256626+0000","last_active":"2024-03-18T20:28:12.246039+0000","last_peered":"2024-03-18T20:28:12.246039+0000","last_clean":"2024-03-18T20:28:12.246039+0000","last_became_active":"2024-03-18T20:28:10.256326+0000","last_became_peered":"2024-03-18T20:28:10.256326+0000","last_unstale":"2024-03-18T20:28:12.246039+0000","last_undegraded":"2024-03-18T20:28:12.246039+0000","last_fullsized":"2024-03-18T20:28:12.246039+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:27:04.227514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.a","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246358+0000","last_change":"2024-03-18T20:28:10.685092+0000","last_active":"2024-03-18T20:28:12.246358+0000","last_peered":"2024-03-18T20:28:12.246358+0000","last_clean":"2024-03-18T20:28:12.246358+0000","last_became_active":"2024-03-18T20:28:10.684811+0000","last_became_peered":"2024-03-18T20:28:10.684811+0000","last_unstale":"2024-03-18T20:28:12.246358+0000","last_undegraded":"2024-03-18T20:28:12.246358+0000","last_fullsized":"2024-03-18T20:28:12.246358+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:02:31.163314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.244850+0000","last_change":"2024-03-18T20:28:10.256291+0000","last_active":"2024-03-18T20:28:12.244850+0000","last_peered":"2024-03-18T20:28:12.244850+0000","last_clean":"2024-03-18T20:28:12.244850+0000","last_became_active":"2024-03-18T20:28:10.256019+0000","last_became_peered":"2024-03-18T20:28:10.256019+0000","last_unstale":"2024-03-18T20:28:12.244850+0000","last_undegraded":"2024-03-18T20:28:12.244850+0000","last_fullsized":"2024-03-18T20:28:12.244850+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:42:08.999227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.b","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246958+0000","last_change":"2024-03-18T20:28:10.683515+0000","last_active":"2024-03-18T20:28:12.246958+0000","last_peered":"2024-03-18T20:28:12.246958+0000","last_clean":"2024-03-18T20:28:12.246958+0000","last_became_active":"2024-03-18T20:28:10.682638+0000","last_became_peered":"2024-03-18T20:28:10.682638+0000","last_unstale":"2024-03-18T20:28:12.246958+0000","last_undegraded":"2024-03-18T20:28:12.246958+0000","last_fullsized":"2024-03-18T20:28:12.246958+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:15:13.776629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.244700+0000","last_change":"2024-03-18T20:28:10.254825+0000","last_active":"2024-03-18T20:28:12.244700+0000","last_peered":"2024-03-18T20:28:12.244700+0000","last_clean":"2024-03-18T20:28:12.244700+0000","last_became_active":"2024-03-18T20:28:10.254552+0000","last_became_peered":"2024-03-18T20:28:10.254552+0000","last_unstale":"2024-03-18T20:28:12.244700+0000","last_undegraded":"2024-03-18T20:28:12.244700+0000","last_fullsized":"2024-03-18T20:28:12.244700+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:12.892050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.4","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246246+0000","last_change":"2024-03-18T20:28:10.689459+0000","last_active":"2024-03-18T20:28:12.246246+0000","last_peered":"2024-03-18T20:28:12.246246+0000","last_clean":"2024-03-18T20:28:12.246246+0000","last_became_active":"2024-03-18T20:28:10.688434+0000","last_became_peered":"2024-03-18T20:28:10.688434+0000","last_unstale":"2024-03-18T20:28:12.246246+0000","last_undegraded":"2024-03-18T20:28:12.246246+0000","last_fullsized":"2024-03-18T20:28:12.246246+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:05:30.166747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.7","version":"287'2","reported_seq":13,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.680248+0000","last_change":"2024-03-18T20:28:10.254441+0000","last_active":"2024-03-18T20:28:11.680248+0000","last_peered":"2024-03-18T20:28:11.680248+0000","last_clean":"2024-03-18T20:28:11.680248+0000","last_became_active":"2024-03-18T20:28:10.253399+0000","last_became_peered":"2024-03-18T20:28:10.253399+0000","last_unstale":"2024-03-18T20:28:11.680248+0000","last_undegraded":"2024-03-18T20:28:11.680248+0000","last_fullsized":"2024-03-18T20:28:11.680248+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:07.991604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.5","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.680224+0000","last_change":"2024-03-18T20:28:10.693201+0000","last_active":"2024-03-18T20:28:11.680224+0000","last_peered":"2024-03-18T20:28:11.680224+0000","last_clean":"2024-03-18T20:28:11.680224+0000","last_became_active":"2024-03-18T20:28:10.692937+0000","last_became_peered":"2024-03-18T20:28:10.692937+0000","last_unstale":"2024-03-18T20:28:11.680224+0000","last_undegraded":"2024-03-18T20:28:11.680224+0000","last_fullsized":"2024-03-18T20:28:11.680224+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:25.186657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245587+0000","last_change":"2024-03-18T20:28:10.253371+0000","last_active":"2024-03-18T20:28:12.245587+0000","last_peered":"2024-03-18T20:28:12.245587+0000","last_clean":"2024-03-18T20:28:12.245587+0000","last_became_active":"2024-03-18T20:28:10.250075+0000","last_became_peered":"2024-03-18T20:28:10.250075+0000","last_unstale":"2024-03-18T20:28:12.245587+0000","last_undegraded":"2024-03-18T20:28:12.245587+0000","last_fullsized":"2024-03-18T20:28:12.245587+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:40.729355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.6","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.864128+0000","last_change":"2024-03-18T20:28:10.694461+0000","last_active":"2024-03-18T20:28:11.864128+0000","last_peered":"2024-03-18T20:28:11.864128+0000","last_clean":"2024-03-18T20:28:11.864128+0000","last_became_active":"2024-03-18T20:28:10.693902+0000","last_became_peered":"2024-03-18T20:28:10.693902+0000","last_unstale":"2024-03-18T20:28:11.864128+0000","last_undegraded":"2024-03-18T20:28:11.864128+0000","last_fullsized":"2024-03-18T20:28:11.864128+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:36:57.764835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246946+0000","last_change":"2024-03-18T20:28:10.251501+0000","last_active":"2024-03-18T20:28:12.246946+0000","last_peered":"2024-03-18T20:28:12.246946+0000","last_clean":"2024-03-18T20:28:12.246946+0000","last_became_active":"2024-03-18T20:28:10.251222+0000","last_became_peered":"2024-03-18T20:28:10.251222+0000","last_unstale":"2024-03-18T20:28:12.246946+0000","last_undegraded":"2024-03-18T20:28:12.246946+0000","last_fullsized":"2024-03-18T20:28:12.246946+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:14.544406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.7","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245677+0000","last_change":"2024-03-18T20:28:10.699609+0000","last_active":"2024-03-18T20:28:12.245677+0000","last_peered":"2024-03-18T20:28:12.245677+0000","last_clean":"2024-03-18T20:28:12.245677+0000","last_became_active":"2024-03-18T20:28:10.699320+0000","last_became_peered":"2024-03-18T20:28:10.699320+0000","last_unstale":"2024-03-18T20:28:12.245677+0000","last_undegraded":"2024-03-18T20:28:12.245677+0000","last_fullsized":"2024-03-18T20:28:12.245677+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:23:42.961689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245681+0000","last_change":"2024-03-18T20:28:10.250745+0000","last_active":"2024-03-18T20:28:12.245681+0000","last_peered":"2024-03-18T20:28:12.245681+0000","last_clean":"2024-03-18T20:28:12.245681+0000","last_became_active":"2024-03-18T20:28:10.250472+0000","last_became_peered":"2024-03-18T20:28:10.250472+0000","last_unstale":"2024-03-18T20:28:12.245681+0000","last_undegraded":"2024-03-18T20:28:12.245681+0000","last_fullsized":"2024-03-18T20:28:12.245681+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:50:07.991604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.0","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245514+0000","last_change":"2024-03-18T20:28:10.685947+0000","last_active":"2024-03-18T20:28:12.245514+0000","last_peered":"2024-03-18T20:28:12.245514+0000","last_clean":"2024-03-18T20:28:12.245514+0000","last_became_active":"2024-03-18T20:28:10.685678+0000","last_became_peered":"2024-03-18T20:28:10.685678+0000","last_unstale":"2024-03-18T20:28:12.245514+0000","last_undegraded":"2024-03-18T20:28:12.245514+0000","last_fullsized":"2024-03-18T20:28:12.245514+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:41:40.137234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.683706+0000","last_change":"2024-03-18T20:28:10.262339+0000","last_active":"2024-03-18T20:28:11.683706+0000","last_peered":"2024-03-18T20:28:11.683706+0000","last_clean":"2024-03-18T20:28:11.683706+0000","last_became_active":"2024-03-18T20:28:10.262058+0000","last_became_peered":"2024-03-18T20:28:10.262058+0000","last_unstale":"2024-03-18T20:28:11.683706+0000","last_undegraded":"2024-03-18T20:28:11.683706+0000","last_fullsized":"2024-03-18T20:28:11.683706+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:12:43.823736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.1","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.863845+0000","last_change":"2024-03-18T20:28:10.691426+0000","last_active":"2024-03-18T20:28:11.863845+0000","last_peered":"2024-03-18T20:28:11.863845+0000","last_clean":"2024-03-18T20:28:11.863845+0000","last_became_active":"2024-03-18T20:28:10.691155+0000","last_became_peered":"2024-03-18T20:28:10.691155+0000","last_unstale":"2024-03-18T20:28:11.863845+0000","last_undegraded":"2024-03-18T20:28:11.863845+0000","last_fullsized":"2024-03-18T20:28:11.863845+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:16:43.773511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.863935+0000","last_change":"2024-03-18T20:28:10.261044+0000","last_active":"2024-03-18T20:28:11.863935+0000","last_peered":"2024-03-18T20:28:11.863935+0000","last_clean":"2024-03-18T20:28:11.863935+0000","last_became_active":"2024-03-18T20:28:10.260707+0000","last_became_peered":"2024-03-18T20:28:10.260707+0000","last_unstale":"2024-03-18T20:28:11.863935+0000","last_undegraded":"2024-03-18T20:28:11.863935+0000","last_fullsized":"2024-03-18T20:28:11.863935+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:45.501668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.2","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.245231+0000","last_change":"2024-03-18T20:28:10.699519+0000","last_active":"2024-03-18T20:28:12.245231+0000","last_peered":"2024-03-18T20:28:12.245231+0000","last_clean":"2024-03-18T20:28:12.245231+0000","last_became_active":"2024-03-18T20:28:10.699250+0000","last_became_peered":"2024-03-18T20:28:10.699250+0000","last_unstale":"2024-03-18T20:28:12.245231+0000","last_undegraded":"2024-03-18T20:28:12.245231+0000","last_fullsized":"2024-03-18T20:28:12.245231+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:28:26.050233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.246268+0000","last_change":"2024-03-18T20:28:10.260130+0000","last_active":"2024-03-18T20:28:12.246268+0000","last_peered":"2024-03-18T20:28:12.246268+0000","last_clean":"2024-03-18T20:28:12.246268+0000","last_became_active":"2024-03-18T20:28:10.259783+0000","last_became_peered":"2024-03-18T20:28:10.259783+0000","last_unstale":"2024-03-18T20:28:12.246268+0000","last_undegraded":"2024-03-18T20:28:12.246268+0000","last_fullsized":"2024-03-18T20:28:12.246268+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:06:16.837176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.3","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:11.680248+0000","last_change":"2024-03-18T20:28:10.693857+0000","last_active":"2024-03-18T20:28:11.680248+0000","last_peered":"2024-03-18T20:28:11.680248+0000","last_clean":"2024-03-18T20:28:11.680248+0000","last_became_active":"2024-03-18T20:28:10.687810+0000","last_became_peered":"2024-03-18T20:28:10.687810+0000","last_unstale":"2024-03-18T20:28:11.680248+0000","last_undegraded":"2024-03-18T20:28:11.680248+0000","last_fullsized":"2024-03-18T20:28:11.680248+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:04.311131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":11,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.244776+0000","last_change":"2024-03-18T20:28:10.251083+0000","last_active":"2024-03-18T20:28:12.244776+0000","last_peered":"2024-03-18T20:28:12.244776+0000","last_clean":"2024-03-18T20:28:12.244776+0000","last_became_active":"2024-03-18T20:28:10.250787+0000","last_became_peered":"2024-03-18T20:28:10.250787+0000","last_unstale":"2024-03-18T20:28:12.244776+0000","last_undegraded":"2024-03-18T20:28:12.244776+0000","last_fullsized":"2024-03-18T20:28:12.244776+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:09.228103+0000","last_clean_scrub_stamp":"2024-03-18T20:28:09.228103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:32:34.322198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.27","version":"0'0","reported_seq":10,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-18T20:28:12.247073+0000","last_change":"2024-03-18T20:28:10.693599+0000","last_active":"2024-03-18T20:28:12.247073+0000","last_peered":"2024-03-18T20:28:12.247073+0000","last_clean":"2024-03-18T20:28:12.247073+0000","last_became_active":"2024-03-18T20:28:10.693365+0000","last_became_peered":"2024-03-18T20:28:10.693365+0000","last_unstale":"2024-03-18T20:28:12.247073+0000","last_undegraded":"2024-03-18T20:28:12.247073+0000","last_fullsized":"2024-03-18T20:28:12.247073+0000","mapping_epoch":286,"log_start":"0'0","ondisk_log_start":"0'0","created":286,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T20:28:10.233700+0000","last_clean_scrub_stamp":"2024-03-18T20:28:10.233700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:27:05.233111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"277'403","reported_seq":1444,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-03-18T20:28:10.680155+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:28:10.680155+0000","last_peered":"2024-03-18T20:28:10.680155+0000","last_clean":"2024-03-18T20:28:10.680155+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:28:10.680155+0000","last_undegraded":"2024-03-18T20:28:10.680155+0000","last_fullsized":"2024-03-18T20:28:10.680155+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":38,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":37,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576010,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53304,"kb_used_data":4140,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94318536,"statfs":{"total":96636764160,"available":96582180864,"internally_reserved":0,"allocated":4239360,"data_stored":2478459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.54400000000000004,"15min":0.58899999999999997},"min":{"1min":0.439,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":1.1339999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55500000000000005,"15min":0.59599999999999997},"min":{"1min":0.35799999999999998,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":1.093},"last":0.45000000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54900000000000004,"15min":0.59799999999999998},"min":{"1min":0.433,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76100000000000001,"5min":0.82999999999999996,"15min":1.4299999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56100000000000005,"15min":0.60399999999999998},"min":{"1min":0.35199999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":1.2629999999999999},"last":0.61899999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54800000000000004,"15min":0.59599999999999997},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":0.77800000000000002,"5min":0.85499999999999998,"15min":1.216},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56999999999999995,"15min":0.61099999999999999},"min":{"1min":0.44700000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.753,"5min":1.1279999999999999,"15min":1.2490000000000001},"last":0.67500000000000004}]},{"osd":3,"last update":"Mon Mar 18 20:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.55000000000000004,"15min":0.59599999999999997},"min":{"1min":0.42299999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81299999999999994,"5min":0.84699999999999998,"15min":1.4450000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.58899999999999997,"15min":0.63600000000000001},"min":{"1min":0.378,"5min":0.371,"15min":0.371},"max":{"1min":0.66700000000000004,"5min":0.94599999999999995,"15min":1.5029999999999999},"last":0.63300000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.46999999999999997,"15min":0.47799999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.75900000000000001,"5min":0.79100000000000004,"15min":0.86099999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.49399999999999999,"15min":0.499},"min":{"1min":0.34000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.83799999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.52600000000000002,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.79600000000000004,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54000000000000004,"15min":0.54700000000000004},"min":{"1min":0.36799999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.76800000000000002,"5min":0.877,"15min":1.042},"last":0.66300000000000003}]},{"osd":6,"last update":"Mon Mar 18 20:27:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.435,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80100000000000005,"5min":0.93500000000000005,"15min":1.129},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.56100000000000005,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.873,"5min":0.91900000000000004,"15min":1.0609999999999999},"last":0.60699999999999998}]}]},{"osd":6,"up_from":13,"seq":55834576010,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65024,"kb_used_data":3564,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306816,"statfs":{"total":96636764160,"available":96570179584,"internally_reserved":0,"allocated":3649536,"data_stored":1888673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":62912967},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53300000000000003,"15min":0.58499999999999996},"min":{"1min":0.35399999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68799999999999994,"5min":0.79800000000000004,"15min":1.1519999999999999},"last":0.378},{"interface":"front","average":{"1min":0.503,"5min":0.54600000000000004,"15min":0.59499999999999997},"min":{"1min":0.33200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.73499999999999999,"5min":1.119,"15min":1.119},"last":0.35899999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53800000000000003,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73199999999999998,"5min":0.79100000000000004,"15min":1.244},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":0.59499999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.33300000000000002},"max":{"1min":0.70099999999999996,"5min":0.89200000000000002,"15min":1.286},"last":0.65300000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56399999999999995,"15min":0.61699999999999999},"min":{"1min":0.38700000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.69599999999999995,"5min":0.79600000000000004,"15min":1.276},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.496,"5min":0.55400000000000005,"15min":0.59899999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.65200000000000002,"5min":1.089,"15min":1.2669999999999999},"last":0.46100000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55500000000000005,"15min":0.61099999999999999},"min":{"1min":0.39200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71599999999999997,"5min":1.163,"15min":1.163},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57099999999999995,"15min":0.61099999999999999},"min":{"1min":0.47499999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78100000000000003,"5min":0.82399999999999995,"15min":1.337},"last":0.496}]},{"osd":4,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.35299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80200000000000005,"5min":0.85799999999999998,"15min":0.88300000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.50600000000000001,"15min":0.51800000000000002},"min":{"1min":0.32500000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83399999999999996,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.623}]},{"osd":5,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.50900000000000001,"15min":0.51200000000000001},"min":{"1min":0.42599999999999999,"5min":0.32900000000000001,"15min":0.32500000000000001},"max":{"1min":0.65000000000000002,"5min":1.008,"15min":1.008},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55900000000000005,"15min":0.55300000000000005},"min":{"1min":0.41799999999999998,"5min":0.33100000000000002,"15min":0.32500000000000001},"max":{"1min":0.755,"5min":0.92600000000000005,"15min":0.94199999999999995},"last":0.72499999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56699999999999995,"15min":0.56299999999999994},"min":{"1min":0.40999999999999998,"5min":0.38,"15min":0.34499999999999997},"max":{"1min":0.91200000000000003,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.34599999999999997},"max":{"1min":0.86699999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.59899999999999998}]}]},{"osd":1,"up_from":13,"seq":55834576010,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52720,"kb_used_data":3556,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319120,"statfs":{"total":96636764160,"available":96582778880,"internally_reserved":0,"allocated":3641344,"data_stored":1888069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.38,"15min":0.39000000000000001},"min":{"1min":0.25900000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.60899999999999999,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.374,"5min":0.38700000000000001,"15min":0.41699999999999998},"min":{"1min":0.25800000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.48399999999999999,"5min":0.623,"15min":0.69699999999999995},"last":0.309}]},{"osd":2,"last update":"Mon Mar 18 20:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.39400000000000002,"15min":0.42299999999999999},"min":{"1min":0.29399999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.46999999999999997,"5min":0.61199999999999999,"15min":0.68100000000000005},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.438,"15min":0.46700000000000003},"min":{"1min":0.314,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64200000000000002,"5min":0.70099999999999996,"15min":0.80500000000000005},"last":0.33900000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.44400000000000001,"15min":0.47899999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58199999999999996,"5min":0.65600000000000003,"15min":0.76100000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.40200000000000002,"15min":0.42799999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54600000000000004,"5min":0.68200000000000005,"15min":0.71699999999999997},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.63100000000000001,"15min":0.66500000000000004},"min":{"1min":0.432,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.288,"5min":1.288,"15min":1.2909999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.64100000000000001,"15min":0.67100000000000004},"min":{"1min":0.436,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.349,"5min":1.349,"15min":1.746},"last":0.58299999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.626,"15min":0.66200000000000003},"min":{"1min":0.44500000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.361,"5min":1.361,"15min":1.6910000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.67600000000000005,"15min":0.70599999999999996},"min":{"1min":0.46700000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.63800000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.63800000000000001,"15min":0.67600000000000005},"min":{"1min":0.42199999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.637},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.65600000000000003,"15min":0.69799999999999995},"min":{"1min":0.44500000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.93500000000000005,"5min":1.014,"15min":1.722},"last":0.57299999999999995}]},{"osd":7,"last update":"Mon Mar 18 20:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.65600000000000003,"15min":0.69799999999999995},"min":{"1min":0.47799999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.66500000000000004,"15min":0.69599999999999995},"min":{"1min":0.47699999999999998,"5min":0.441,"15min":0.441},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.56799999999999995}]}]},{"osd":0,"up_from":13,"seq":55834576011,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49208,"kb_used_data":4140,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322632,"statfs":{"total":96636764160,"available":96586375168,"internally_reserved":0,"allocated":4239360,"data_stored":2478527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.34399999999999997,"15min":0.38},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.40600000000000003,"5min":0.60399999999999998,"15min":0.61199999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.38,"15min":0.41499999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.51200000000000001,"5min":0.60099999999999998,"15min":1.1850000000000001},"last":0.47299999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.39200000000000002,"15min":0.42599999999999999},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.501,"5min":0.69399999999999995,"15min":0.79400000000000004},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.372,"5min":0.39800000000000002,"15min":0.439},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.48699999999999999,"5min":0.64800000000000002,"15min":0.70799999999999996},"last":0.58499999999999996}]},{"osd":3,"last update":"Mon Mar 18 20:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.38100000000000001,"15min":0.41699999999999998},"min":{"1min":0.28899999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.44800000000000001,"5min":0.58499999999999996,"15min":0.628},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.42899999999999999,"15min":0.47999999999999998},"min":{"1min":0.33300000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.53900000000000003,"5min":0.73399999999999999,"15min":0.91500000000000004},"last":0.57699999999999996}]},{"osd":4,"last update":"Mon Mar 18 20:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59699999999999998,"15min":0.63900000000000001},"min":{"1min":0.435,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86299999999999999,"5min":0.90200000000000002,"15min":1.0029999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.61399999999999999,"15min":0.65000000000000002},"min":{"1min":0.42199999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72099999999999997,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.77400000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61899999999999999,"15min":0.65900000000000003},"min":{"1min":0.41699999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.878,"5min":0.89700000000000002,"15min":1.0129999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.628,"15min":0.66700000000000004},"min":{"1min":0.47899999999999998,"5min":0.432,"15min":0.432},"max":{"1min":0.77800000000000002,"5min":0.96199999999999997,"15min":1.518},"last":0.78300000000000003}]},{"osd":6,"last update":"Mon Mar 18 20:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.61599999999999999,"15min":0.65000000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.78700000000000003,"5min":1.0669999999999999,"15min":1.1279999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.65100000000000002,"15min":0.69399999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92900000000000005,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.93200000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.65600000000000003,"15min":0.69199999999999995},"min":{"1min":0.496,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.81699999999999995,"5min":1.008,"15min":1.464},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.64100000000000001,"15min":0.69099999999999995},"min":{"1min":0.46400000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.80800000000000005,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.82499999999999996}]}]},{"osd":2,"up_from":13,"seq":55834576011,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52736,"kb_used_data":3572,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":3657728,"data_stored":1889703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.33300000000000002,"15min":0.36599999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.34799999999999998,"5min":0.54800000000000004,"15min":0.57499999999999996},"last":0.317},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.42999999999999999,"15min":0.47199999999999998},"min":{"1min":0.30499999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.51300000000000001,"5min":0.71499999999999997,"15min":0.753},"last":0.37}]},{"osd":1,"last update":"Mon Mar 18 20:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.434,"15min":0.46899999999999997},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.47599999999999998,"5min":0.64300000000000002,"15min":0.749},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.317,"5min":0.37,"15min":0.40400000000000003},"min":{"1min":0.22800000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.41299999999999998,"5min":0.55000000000000004,"15min":0.59999999999999998},"last":0.33300000000000002}]},{"osd":3,"last update":"Mon Mar 18 20:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.39600000000000002,"15min":0.42499999999999999},"min":{"1min":0.26500000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.45400000000000001,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.45000000000000001,"15min":0.5},"min":{"1min":0.33700000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.48999999999999999,"5min":0.68100000000000005,"15min":0.752},"last":0.47299999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.622,"15min":0.65100000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.93999999999999995,"5min":1.2090000000000001,"15min":1.4079999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.63300000000000001,"15min":0.65900000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.95099999999999996,"5min":1.268,"15min":1.2729999999999999},"last":0.499}]},{"osd":5,"last update":"Mon Mar 18 20:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.628,"15min":0.66900000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.873,"5min":0.97299999999999998,"15min":1.028},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.626,"15min":0.66600000000000004},"min":{"1min":0.437,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80800000000000005,"5min":0.88300000000000001,"15min":1.0629999999999999},"last":0.77400000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.63500000000000001,"15min":0.67500000000000004},"min":{"1min":0.42299999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76400000000000001,"5min":1.014,"15min":1.2050000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.69299999999999995,"15min":0.72299999999999998},"min":{"1min":0.505,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0109999999999999,"5min":1.3220000000000001,"15min":1.3859999999999999},"last":0.83999999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.63700000000000001,"15min":0.68300000000000005},"min":{"1min":0.434,"5min":0.432,"15min":0.42199999999999999},"max":{"1min":0.72499999999999998,"5min":0.99399999999999999,"15min":1.0269999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.66500000000000004,"15min":0.69699999999999995},"min":{"1min":0.46200000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.97499999999999998,"5min":1.2330000000000001,"15min":1.371},"last":0.84999999999999998}]}]},{"osd":3,"up_from":13,"seq":55834576011,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56820,"kb_used_data":3560,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94315020,"statfs":{"total":96636764160,"available":96578580480,"internally_reserved":0,"allocated":3645440,"data_stored":1888103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.36099999999999999,"15min":0.39000000000000001},"min":{"1min":0.247,"5min":0.223,"15min":0.223},"max":{"1min":0.46400000000000002,"5min":0.66500000000000004,"15min":0.98099999999999998},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.373,"15min":0.40400000000000003},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.45600000000000002,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.247}]},{"osd":1,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.35999999999999999,"15min":0.39700000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.44500000000000001,"5min":0.63200000000000001,"15min":0.64000000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.42199999999999999,"15min":0.45600000000000002},"min":{"1min":0.33400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.501,"5min":0.65500000000000003,"15min":0.73099999999999998},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.41499999999999998,"15min":0.45200000000000001},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.54900000000000004,"5min":0.67700000000000005,"15min":0.72999999999999998},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.437,"15min":0.48099999999999998},"min":{"1min":0.32600000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.52300000000000002,"5min":0.64400000000000002,"15min":0.71199999999999997},"last":0.433}]},{"osd":4,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59499999999999997,"15min":0.64000000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.526,"5min":1.526,"15min":1.526},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61399999999999999,"15min":0.66100000000000003},"min":{"1min":0.39200000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.65800000000000003,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.56799999999999995}]},{"osd":5,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.65100000000000002,"15min":0.68700000000000006},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.66500000000000004,"15min":0.71499999999999997},"min":{"1min":0.45300000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78000000000000003,"5min":1.1120000000000001,"15min":1.171},"last":0.50700000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65800000000000003,"15min":0.69699999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.71999999999999997},"min":{"1min":0.48699999999999999,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.53100000000000003}]},{"osd":7,"last update":"Mon Mar 18 20:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.64900000000000002,"15min":0.70299999999999996},"min":{"1min":0.43099999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79900000000000004,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.64300000000000002,"15min":0.68899999999999995},"min":{"1min":0.51500000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.47899999999999998}]}]},{"osd":4,"up_from":13,"seq":55834576011,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48648,"kb_used_data":3580,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323192,"statfs":{"total":96636764160,"available":96586948608,"internally_reserved":0,"allocated":3665920,"data_stored":1890375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56200000000000006,"15min":0.58399999999999996},"min":{"1min":0.373,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.81699999999999995,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55500000000000005,"15min":0.58999999999999997},"min":{"1min":0.41399999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78300000000000003,"5min":0.80600000000000005,"15min":0.84199999999999997},"last":0.61699999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.60599999999999998},"min":{"1min":0.38200000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.88300000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54900000000000004,"15min":0.57499999999999996},"min":{"1min":0.372,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.66600000000000004,"5min":0.77800000000000002,"15min":0.96499999999999997},"last":0.47899999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.57699999999999996,"15min":0.59499999999999997},"min":{"1min":0.374,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85299999999999998,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58499999999999996,"15min":0.60799999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38500000000000001},"max":{"1min":0.81100000000000005,"5min":0.81399999999999995,"15min":0.84099999999999997},"last":0.56200000000000006}]},{"osd":3,"last update":"Mon Mar 18 20:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.57199999999999995,"15min":0.59499999999999997},"min":{"1min":0.38400000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79400000000000004,"5min":1.026,"15min":1.026},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59999999999999998,"15min":0.61499999999999999},"min":{"1min":0.42299999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88600000000000001,"5min":1.028,"15min":1.028},"last":0.63100000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.48199999999999998},"min":{"1min":0.33200000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.747,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.497,"15min":0.502},"min":{"1min":0.31900000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.69199999999999995,"5min":0.85499999999999998,"15min":0.90600000000000003},"last":0.64800000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.50600000000000001,"15min":0.504},"min":{"1min":0.33900000000000002,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.63300000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.35199999999999998,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.72999999999999998,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.66500000000000004}]},{"osd":7,"last update":"Mon Mar 18 20:27:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55500000000000005,"15min":0.54800000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.35699999999999998},"max":{"1min":0.749,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58099999999999996,"15min":0.58199999999999996},"min":{"1min":0.38900000000000001,"5min":0.35499999999999998,"15min":0.33700000000000002},"max":{"1min":0.76800000000000002,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.70599999999999996}]}]},{"osd":5,"up_from":13,"seq":55834576010,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70772,"kb_used_data":3560,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94301068,"statfs":{"total":96636764160,"available":96564293632,"internally_reserved":0,"allocated":3645440,"data_stored":1888103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:27:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52900000000000003,"15min":0.58399999999999996},"min":{"1min":0.38400000000000001,"5min":0.33500000000000002,"15min":0.31900000000000001},"max":{"1min":0.67500000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.53400000000000003,"15min":0.58899999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":0.66600000000000004,"5min":0.82399999999999995,"15min":1.3100000000000001},"last":0.65900000000000003}]},{"osd":1,"last update":"Mon Mar 18 20:27:54 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.53900000000000003,"15min":0.59299999999999997},"min":{"1min":0.34899999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73699999999999999,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.53300000000000003,"15min":0.58599999999999997},"min":{"1min":0.376,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.74199999999999999,"5min":0.94299999999999995,"15min":1.018},"last":0.67300000000000004}]},{"osd":2,"last update":"Mon Mar 18 20:27:54 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.53400000000000003,"15min":0.58699999999999997},"min":{"1min":0.34599999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.63100000000000001,"5min":0.85299999999999998,"15min":0.99299999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.53600000000000003,"15min":0.59599999999999997},"min":{"1min":0.38800000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.68300000000000005,"5min":1.599,"15min":1.599},"last":0.63100000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:27:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56000000000000005,"15min":0.60099999999999998},"min":{"1min":0.40500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.72999999999999998,"5min":0.98699999999999999,"15min":1.2430000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.63500000000000001},"min":{"1min":0.45900000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.75800000000000001,"5min":0.84399999999999997,"15min":1.161},"last":0.76100000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:27:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.48399999999999999},"min":{"1min":0.32300000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.70999999999999996,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.53500000000000003,"15min":0.52100000000000002},"min":{"1min":0.39900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78100000000000003,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.56699999999999995}]},{"osd":6,"last update":"Mon Mar 18 20:27:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.36699999999999999,"5min":0.33300000000000002,"15min":0.32800000000000001},"max":{"1min":0.751,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.32800000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.76300000000000001,"5min":1.643,"15min":1.643},"last":0.51300000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:27:54 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56799999999999995,"15min":0.56000000000000005},"min":{"1min":0.378,"5min":0.317,"15min":0.317},"max":{"1min":0.80700000000000005,"5min":1.028,"15min":1.028},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57999999999999996,"15min":0.58099999999999996},"min":{"1min":0.442,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.85199999999999998,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.61399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:28:15.247 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T20:28:15.247 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-18T20:28:15.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.412+0000 7fd8da5f4700 1 Processor -- start 2024-03-18T20:28:15.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.412+0000 7fd8da5f4700 1 -- start start 2024-03-18T20:28:15.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.413+0000 7fd8da5f4700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd8d40cfc20 0x7fd8d40cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.413+0000 7fd8da5f4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8d4099a30 0x7fd8d4099ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.413+0000 7fd8da5f4700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d40cd080 con 0x7fd8d415eb30 2024-03-18T20:28:15.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.413+0000 7fd8da5f4700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8d409a400 con 0x7fd8d40cfc20 2024-03-18T20:28:15.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.413+0000 7fd8da5f4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d40d0080 con 0x7fd8d4099a30 2024-03-18T20:28:15.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.413+0000 7fd8d3fff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd8d415eb30 0x7fd8d40cd510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40696/0 (socket says 172.21.15.1:40696) 2024-03-18T20:28:15.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.413+0000 7fd8d3fff700 1 -- 172.21.15.1:0/2195623624 learned_addr learned my addr 172.21.15.1:0/2195623624 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:15.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.413+0000 7fd8d8b91700 1 --2- 172.21.15.1:0/2195623624 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8d4099a30 0x7fd8d4099ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.413+0000 7fd8d37fe700 1 --2- 172.21.15.1:0/2195623624 >> v2:172.21.15.165:3300/0 conn(0x7fd8d40cfc20 0x7fd8d40cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.413+0000 7fd8d2ffd700 1 -- 172.21.15.1:0/2195623624 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595845503 0 0) 0x7fd8d40cd080 con 0x7fd8d415eb30 2024-03-18T20:28:15.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.413+0000 7fd8d2ffd700 1 -- 172.21.15.1:0/2195623624 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8bc0035b0 con 0x7fd8d415eb30 2024-03-18T20:28:15.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d2ffd700 1 -- 172.21.15.1:0/2195623624 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1525697639 0 0) 0x7fd8bc0035b0 con 0x7fd8d415eb30 2024-03-18T20:28:15.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d2ffd700 1 -- 172.21.15.1:0/2195623624 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8d4099a30 msgr2=0x7fd8d4099ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:15.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d2ffd700 1 --2- 172.21.15.1:0/2195623624 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8d4099a30 0x7fd8d4099ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d2ffd700 1 -- 172.21.15.1:0/2195623624 >> v2:172.21.15.165:3300/0 conn(0x7fd8d40cfc20 msgr2=0x7fd8d40cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:15.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d2ffd700 1 --2- 172.21.15.1:0/2195623624 >> v2:172.21.15.165:3300/0 conn(0x7fd8d40cfc20 0x7fd8d40cbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d2ffd700 1 -- 172.21.15.1:0/2195623624 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d415ef10 con 0x7fd8d415eb30 2024-03-18T20:28:15.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d2ffd700 1 -- 172.21.15.1:0/2195623624 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fd8c4001670 con 0x7fd8d415eb30 2024-03-18T20:28:15.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d2ffd700 1 -- 172.21.15.1:0/2195623624 >> v1:172.21.15.1:6789/0 conn(0x7fd8d415eb30 legacy=0x7fd8d40cd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:15.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d2ffd700 1 --2- 172.21.15.1:0/2195623624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8bc003a00 0x7fd8bc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d2ffd700 1 --2- 172.21.15.1:0/2195623624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc004850 0x7fd8bc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d2ffd700 1 --2- 172.21.15.1:0/2195623624 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8d4099a30 0x7fd8bc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d2ffd700 1 -- 172.21.15.1:0/2195623624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d409a400 con 0x7fd8bc004850 2024-03-18T20:28:15.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d2ffd700 1 -- 172.21.15.1:0/2195623624 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d40d0080 con 0x7fd8bc003a00 2024-03-18T20:28:15.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d2ffd700 1 -- 172.21.15.1:0/2195623624 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc00ba10 con 0x7fd8d4099a30 2024-03-18T20:28:15.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d37fe700 1 --2- 172.21.15.1:0/2195623624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc004850 0x7fd8bc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d3fff700 1 --2- 172.21.15.1:0/2195623624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8bc003a00 0x7fd8bc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d8b91700 1 --2- 172.21.15.1:0/2195623624 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8d4099a30 0x7fd8bc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d37fe700 1 -- 172.21.15.1:0/2195623624 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8d4099a30 msgr2=0x7fd8bc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:15.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d37fe700 1 --2- 172.21.15.1:0/2195623624 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8d4099a30 0x7fd8bc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d37fe700 1 -- 172.21.15.1:0/2195623624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8bc003a00 msgr2=0x7fd8bc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:15.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d37fe700 1 --2- 172.21.15.1:0/2195623624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8bc003a00 0x7fd8bc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d37fe700 1 -- 172.21.15.1:0/2195623624 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc0035b0 con 0x7fd8bc004850 2024-03-18T20:28:15.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d3fff700 1 --2- 172.21.15.1:0/2195623624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8bc003a00 0x7fd8bc003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:28:15.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d37fe700 1 --2- 172.21.15.1:0/2195623624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc004850 0x7fd8bc004cd0 secure :-1 s=READY pgs=3289 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8009b90 tx=0x7fd8bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef7a70b4e114fb1a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:15.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d2ffd700 1 -- 172.21.15.1:0/2195623624 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c800e6c0 con 0x7fd8bc004850 2024-03-18T20:28:15.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d2ffd700 1 -- 172.21.15.1:0/2195623624 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c800e820 con 0x7fd8bc004850 2024-03-18T20:28:15.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.414+0000 7fd8d2ffd700 1 -- 172.21.15.1:0/2195623624 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c800eb90 con 0x7fd8bc004850 2024-03-18T20:28:15.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.415+0000 7fd8da5f4700 1 -- 172.21.15.1:0/2195623624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc004850 msgr2=0x7fd8bc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:15.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.415+0000 7fd8da5f4700 1 --2- 172.21.15.1:0/2195623624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc004850 0x7fd8bc004cd0 secure :-1 s=READY pgs=3289 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8009b90 tx=0x7fd8bc005880 comp rx=0 tx=0).stop 2024-03-18T20:28:15.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.415+0000 7fd8d8b91700 1 -- 172.21.15.1:0/2195623624 reap_dead start 2024-03-18T20:28:15.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.415+0000 7fd8da5f4700 1 -- 172.21.15.1:0/2195623624 shutdown_connections 2024-03-18T20:28:15.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.415+0000 7fd8da5f4700 1 -- 172.21.15.1:0/2195623624 >> 172.21.15.1:0/2195623624 conn(0x7fd8d40d1210 msgr2=0x7fd8d40d1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:15.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.415+0000 7fd8da5f4700 1 -- 172.21.15.1:0/2195623624 shutdown_connections 2024-03-18T20:28:15.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.415+0000 7fd8da5f4700 1 -- 172.21.15.1:0/2195623624 wait complete. 2024-03-18T20:28:15.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.416+0000 7fd8da5f4700 1 Processor -- start 2024-03-18T20:28:15.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.416+0000 7fd8da5f4700 1 -- start start 2024-03-18T20:28:15.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.417+0000 7fd8da5f4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc004850 0x7fd8d40c9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.417+0000 7fd8da5f4700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8d4150640 0x7fd8d41512e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.417+0000 7fd8da5f4700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8d40b65d0 0x7fd8d40b6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.417+0000 7fd8da5f4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d4150ec0 con 0x7fd8bc004850 2024-03-18T20:28:15.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.417+0000 7fd8da5f4700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d40b6f90 con 0x7fd8d4150640 2024-03-18T20:28:15.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.417+0000 7fd8da5f4700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d40ca280 con 0x7fd8d40b65d0 2024-03-18T20:28:15.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.417+0000 7fd8d3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc004850 0x7fd8d40c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.417+0000 7fd8d3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc004850 0x7fd8d40c9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57286/0 (socket says 172.21.15.1:57286) 2024-03-18T20:28:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.417+0000 7fd8d3fff700 1 -- 172.21.15.1:0/1068301511 learned_addr learned my addr 172.21.15.1:0/1068301511 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.417+0000 7fd8d37fe700 1 --2- 172.21.15.1:0/1068301511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8d4150640 0x7fd8d41512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.417+0000 7fd8d8b91700 1 --2- 172.21.15.1:0/1068301511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8d40b65d0 0x7fd8d40b6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.417+0000 7fd8d3fff700 1 -- 172.21.15.1:0/1068301511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8d40b65d0 msgr2=0x7fd8d40b6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.417+0000 7fd8d3fff700 1 --2- 172.21.15.1:0/1068301511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8d40b65d0 0x7fd8d40b6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.418+0000 7fd8d3fff700 1 -- 172.21.15.1:0/1068301511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8d4150640 msgr2=0x7fd8d41512e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.418+0000 7fd8d3fff700 1 --2- 172.21.15.1:0/1068301511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8d4150640 0x7fd8d41512e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.418+0000 7fd8d3fff700 1 -- 172.21.15.1:0/1068301511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d40ca4b0 con 0x7fd8bc004850 2024-03-18T20:28:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.418+0000 7fd8d37fe700 1 --2- 172.21.15.1:0/1068301511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8d4150640 0x7fd8d41512e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:28:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.418+0000 7fd8d3fff700 1 --2- 172.21.15.1:0/1068301511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc004850 0x7fd8d40c9d40 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 crypto rx=0x7fd8c400eaf0 tx=0x7fd8c400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.418+0000 7fd8d0ff9700 1 -- 172.21.15.1:0/1068301511 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c4005e00 con 0x7fd8bc004850 2024-03-18T20:28:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.418+0000 7fd8d0ff9700 1 -- 172.21.15.1:0/1068301511 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c400c750 con 0x7fd8bc004850 2024-03-18T20:28:15.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.418+0000 7fd8da5f4700 1 -- 172.21.15.1:0/1068301511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d40ca790 con 0x7fd8bc004850 2024-03-18T20:28:15.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.418+0000 7fd8d0ff9700 1 -- 172.21.15.1:0/1068301511 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c400cae0 con 0x7fd8bc004850 2024-03-18T20:28:15.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.418+0000 7fd8da5f4700 1 -- 172.21.15.1:0/1068301511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d40b4890 con 0x7fd8bc004850 2024-03-18T20:28:15.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.419+0000 7fd8da5f4700 1 -- 172.21.15.1:0/1068301511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d4067dc0 con 0x7fd8bc004850 2024-03-18T20:28:15.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.419+0000 7fd8d0ff9700 1 -- 172.21.15.1:0/1068301511 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fd8c400cc40 con 0x7fd8bc004850 2024-03-18T20:28:15.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.420+0000 7fd8d0ff9700 1 --2- 172.21.15.1:0/1068301511 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd8c006adb0 0x7fd8c006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.420+0000 7fd8d0ff9700 1 -- 172.21.15.1:0/1068301511 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fd8c4089530 con 0x7fd8bc004850 2024-03-18T20:28:15.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.420+0000 7fd8d37fe700 1 --2- 172.21.15.1:0/1068301511 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd8c006adb0 0x7fd8c006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.420+0000 7fd8d37fe700 1 --2- 172.21.15.1:0/1068301511 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd8c006adb0 0x7fd8c006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd8c80090b0 tx=0x7fd8c8008030 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:15.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.424+0000 7fd8d0ff9700 1 -- 172.21.15.1:0/1068301511 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd8c40543b0 con 0x7fd8bc004850 2024-03-18T20:28:15.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.708+0000 7fd8da5f4700 1 -- 172.21.15.1:0/1068301511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd8d40b0760 con 0x7fd8bc004850 2024-03-18T20:28:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.711+0000 7fd8d0ff9700 1 -- 172.21.15.1:0/1068301511 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v54) v1 ==== 190+0+0 (secure 0 0 0) 0x7fd8c40579d0 con 0x7fd8bc004850 2024-03-18T20:28:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.714+0000 7fd8da5f4700 1 -- 172.21.15.1:0/1068301511 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd8c006adb0 msgr2=0x7fd8c006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.714+0000 7fd8da5f4700 1 --2- 172.21.15.1:0/1068301511 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd8c006adb0 0x7fd8c006d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd8c80090b0 tx=0x7fd8c8008030 comp rx=0 tx=0).stop 2024-03-18T20:28:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.714+0000 7fd8da5f4700 1 -- 172.21.15.1:0/1068301511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc004850 msgr2=0x7fd8d40c9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.714+0000 7fd8da5f4700 1 --2- 172.21.15.1:0/1068301511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc004850 0x7fd8d40c9d40 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 crypto rx=0x7fd8c400eaf0 tx=0x7fd8c400eeb0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.714+0000 7fd8da5f4700 1 -- 172.21.15.1:0/1068301511 shutdown_connections 2024-03-18T20:28:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.714+0000 7fd8da5f4700 1 --2- 172.21.15.1:0/1068301511 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fd8c006adb0 0x7fd8c006d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.714+0000 7fd8da5f4700 1 --2- 172.21.15.1:0/1068301511 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fd8d40b65d0 0x7fd8d40b6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.714+0000 7fd8da5f4700 1 --2- 172.21.15.1:0/1068301511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8bc004850 0x7fd8d40c9d40 unknown :-1 s=CLOSED pgs=3290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.714+0000 7fd8da5f4700 1 --2- 172.21.15.1:0/1068301511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8d4150640 0x7fd8d41512e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.714+0000 7fd8da5f4700 1 -- 172.21.15.1:0/1068301511 >> 172.21.15.1:0/1068301511 conn(0x7fd8d40d1210 msgr2=0x7fd8d415e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:15.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.714+0000 7fd8da5f4700 1 -- 172.21.15.1:0/1068301511 shutdown_connections 2024-03-18T20:28:15.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.714+0000 7fd8da5f4700 1 -- 172.21.15.1:0/1068301511 wait complete. 2024-03-18T20:28:15.719 INFO:teuthology.orchestra.run.smithi001.stderr:updated caps for client.0 2024-03-18T20:28:15.730 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:28:15.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.851+0000 7f0277405700 1 Processor -- start 2024-03-18T20:28:15.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.851+0000 7f0277405700 1 -- start start 2024-03-18T20:28:15.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.852+0000 7f0277405700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0270140cb0 0x7f0270142f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.852+0000 7f0277405700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0270143440 0x7f027015ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.852+0000 7f0277405700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0270142a70 con 0x7f02700b3260 2024-03-18T20:28:15.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.852+0000 7f0277405700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0270144120 con 0x7f0270143440 2024-03-18T20:28:15.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.852+0000 7f0277405700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0270141e60 con 0x7f0270140cb0 2024-03-18T20:28:15.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.852+0000 7f02749a0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0270143440 0x7f027015ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.852+0000 7f02759a2700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f02700b3260 0x7f02700b3830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40712/0 (socket says 172.21.15.1:40712) 2024-03-18T20:28:15.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.852+0000 7f02759a2700 1 -- 172.21.15.1:0/2658884589 learned_addr learned my addr 172.21.15.1:0/2658884589 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:15.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.853+0000 7f02749a0700 1 -- 172.21.15.1:0/2658884589 >> v2:172.21.15.165:3300/0 conn(0x7f0270140cb0 msgr2=0x7f0270142f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:15.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.853+0000 7f02749a0700 1 --2- 172.21.15.1:0/2658884589 >> v2:172.21.15.165:3300/0 conn(0x7f0270140cb0 0x7f0270142f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.853+0000 7f02749a0700 1 -- 172.21.15.1:0/2658884589 >> v1:172.21.15.1:6789/0 conn(0x7f02700b3260 legacy=0x7f02700b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:15.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.853+0000 7f0263fff700 1 -- 172.21.15.1:0/2658884589 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655044472 0 0) 0x7f0270142a70 con 0x7f02700b3260 2024-03-18T20:28:15.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.853+0000 7f02749a0700 1 -- 172.21.15.1:0/2658884589 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0270142140 con 0x7f0270143440 2024-03-18T20:28:15.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.853+0000 7f02749a0700 1 --2- 172.21.15.1:0/2658884589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0270143440 0x7f027015ab20 secure :-1 s=READY pgs=3292 cs=0 l=1 rev1=1 crypto rx=0x7f02700b2fb0 tx=0x7f026400bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c53590376a4b2fac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:15.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.853+0000 7f0263fff700 1 -- 172.21.15.1:0/2658884589 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026400dc70 con 0x7f0270143440 2024-03-18T20:28:15.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.853+0000 7f0263fff700 1 -- 172.21.15.1:0/2658884589 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026400ddd0 con 0x7f0270143440 2024-03-18T20:28:15.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.853+0000 7f0263fff700 1 -- 172.21.15.1:0/2658884589 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0264010690 con 0x7f0270143440 2024-03-18T20:28:15.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.854+0000 7f0277405700 1 -- 172.21.15.1:0/2658884589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0270143440 msgr2=0x7f027015ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:15.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.854+0000 7f0277405700 1 --2- 172.21.15.1:0/2658884589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0270143440 0x7f027015ab20 secure :-1 s=READY pgs=3292 cs=0 l=1 rev1=1 crypto rx=0x7f02700b2fb0 tx=0x7f026400bdc0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.854+0000 7f0277405700 1 -- 172.21.15.1:0/2658884589 shutdown_connections 2024-03-18T20:28:15.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.854+0000 7f0277405700 1 --2- 172.21.15.1:0/2658884589 >> v2:172.21.15.165:3300/0 conn(0x7f0270140cb0 0x7f0270142f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.854+0000 7f0277405700 1 --2- 172.21.15.1:0/2658884589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0270143440 0x7f027015ab20 unknown :-1 s=CLOSED pgs=3292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.854+0000 7f0277405700 1 -- 172.21.15.1:0/2658884589 >> 172.21.15.1:0/2658884589 conn(0x7f02700b5780 msgr2=0x7f02700b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:15.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.854+0000 7f0277405700 1 -- 172.21.15.1:0/2658884589 shutdown_connections 2024-03-18T20:28:15.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.854+0000 7f0277405700 1 -- 172.21.15.1:0/2658884589 wait complete. 2024-03-18T20:28:15.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.855+0000 7f0277405700 1 Processor -- start 2024-03-18T20:28:15.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.855+0000 7f0277405700 1 -- start start 2024-03-18T20:28:15.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f0277405700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02700b3260 0x7f02701f22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f0277405700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0270140cb0 0x7f0270158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f0277405700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0270158db0 0x7f0270153ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f0277405700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0270154ae0 con 0x7f0270140cb0 2024-03-18T20:28:15.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f0277405700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0270154d50 con 0x7f0270158db0 2024-03-18T20:28:15.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f0277405700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0270155030 con 0x7f02700b3260 2024-03-18T20:28:15.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f02749a0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0270140cb0 0x7f0270158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f02749a0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0270140cb0 0x7f0270158870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57302/0 (socket says 172.21.15.1:57302) 2024-03-18T20:28:15.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f02749a0700 1 -- 172.21.15.1:0/3180464670 learned_addr learned my addr 172.21.15.1:0/3180464670 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:15.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f02751a1700 1 --2- 172.21.15.1:0/3180464670 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02700b3260 0x7f02701f22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f02759a2700 1 --2- 172.21.15.1:0/3180464670 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0270158db0 0x7f0270153ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f02749a0700 1 -- 172.21.15.1:0/3180464670 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02700b3260 msgr2=0x7f02701f22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:15.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f02749a0700 1 --2- 172.21.15.1:0/3180464670 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02700b3260 0x7f02701f22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f02749a0700 1 -- 172.21.15.1:0/3180464670 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0270158db0 msgr2=0x7f0270153ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:15.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f02749a0700 1 --2- 172.21.15.1:0/3180464670 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0270158db0 0x7f0270153ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:15.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f02749a0700 1 -- 172.21.15.1:0/3180464670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026400b740 con 0x7f0270140cb0 2024-03-18T20:28:15.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f02749a0700 1 --2- 172.21.15.1:0/3180464670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0270140cb0 0x7f0270158870 secure :-1 s=READY pgs=3293 cs=0 l=1 rev1=1 crypto rx=0x7f0264000c00 tx=0x7f0264004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:15.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f0261ffb700 1 -- 172.21.15.1:0/3180464670 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026401f070 con 0x7f0270140cb0 2024-03-18T20:28:15.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f0261ffb700 1 -- 172.21.15.1:0/3180464670 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0264003b20 con 0x7f0270140cb0 2024-03-18T20:28:15.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f0261ffb700 1 -- 172.21.15.1:0/3180464670 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0264024470 con 0x7f0270140cb0 2024-03-18T20:28:15.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.856+0000 7f0277405700 1 -- 172.21.15.1:0/3180464670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0270155310 con 0x7f0270140cb0 2024-03-18T20:28:15.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.857+0000 7f0277405700 1 -- 172.21.15.1:0/3180464670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02701561d0 con 0x7f0270140cb0 2024-03-18T20:28:15.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.858+0000 7f0277405700 1 -- 172.21.15.1:0/3180464670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02700b6550 con 0x7f0270140cb0 2024-03-18T20:28:15.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.858+0000 7f0261ffb700 1 -- 172.21.15.1:0/3180464670 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f026402d430 con 0x7f0270140cb0 2024-03-18T20:28:15.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.859+0000 7f0261ffb700 1 --2- 172.21.15.1:0/3180464670 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f025406adb0 0x7f025406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:15.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.859+0000 7f02751a1700 1 --2- 172.21.15.1:0/3180464670 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f025406adb0 0x7f025406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:15.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.859+0000 7f0261ffb700 1 -- 172.21.15.1:0/3180464670 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f026408cc60 con 0x7f0270140cb0 2024-03-18T20:28:15.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.859+0000 7f02751a1700 1 --2- 172.21.15.1:0/3180464670 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f025406adb0 0x7f025406d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f026c006fd0 tx=0x7f026c008040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:15.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:15.863+0000 7f0261ffb700 1 -- 172.21.15.1:0/3180464670 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0264057a60 con 0x7f0270140cb0 2024-03-18T20:28:16.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.111+0000 7f0277405700 1 -- 172.21.15.1:0/3180464670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0270155520 con 0x7f0270140cb0 2024-03-18T20:28:16.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.112+0000 7f0261ffb700 1 -- 172.21.15.1:0/3180464670 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4750 (secure 0 0 0) 0x7f026405b080 con 0x7f0270140cb0 2024-03-18T20:28:16.116 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:28:16.116 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":144,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10322,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/2385318877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":2385318877},{"type":"v1","addr":"172.21.15.165:6834","nonce":2385318877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139},{"gid":12040,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1024465047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1024465047},{"type":"v1","addr":"172.21.15.165:6835","nonce":1024465047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139},{"gid":13791,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2374816325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2374816325},{"type":"v1","addr":"172.21.15.1:6837","nonce":2374816325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:28:10.668549+0000","modified":"2024-03-18T20:28:13.687432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13794},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13794":{"gid":13794,"name":"a","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6836/1111156155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1111156155},{"type":"v1","addr":"172.21.15.1:6836","nonce":1111156155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-03-18T20:28:16.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.114+0000 7f0277405700 1 -- 172.21.15.1:0/3180464670 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f025406adb0 msgr2=0x7f025406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:16.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.114+0000 7f0277405700 1 --2- 172.21.15.1:0/3180464670 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f025406adb0 0x7f025406d270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f026c006fd0 tx=0x7f026c008040 comp rx=0 tx=0).stop 2024-03-18T20:28:16.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.114+0000 7f0277405700 1 -- 172.21.15.1:0/3180464670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0270140cb0 msgr2=0x7f0270158870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:16.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.114+0000 7f0277405700 1 --2- 172.21.15.1:0/3180464670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0270140cb0 0x7f0270158870 secure :-1 s=READY pgs=3293 cs=0 l=1 rev1=1 crypto rx=0x7f0264000c00 tx=0x7f0264004750 comp rx=0 tx=0).stop 2024-03-18T20:28:16.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.114+0000 7f0277405700 1 -- 172.21.15.1:0/3180464670 shutdown_connections 2024-03-18T20:28:16.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.114+0000 7f0277405700 1 --2- 172.21.15.1:0/3180464670 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f025406adb0 0x7f025406d270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.114+0000 7f0277405700 1 --2- 172.21.15.1:0/3180464670 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f02700b3260 0x7f02701f22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.114+0000 7f0277405700 1 --2- 172.21.15.1:0/3180464670 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0270158db0 0x7f0270153ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.114+0000 7f0277405700 1 --2- 172.21.15.1:0/3180464670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0270140cb0 0x7f0270158870 unknown :-1 s=CLOSED pgs=3293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.114+0000 7f0277405700 1 -- 172.21.15.1:0/3180464670 >> 172.21.15.1:0/3180464670 conn(0x7f02700b5780 msgr2=0x7f0270141090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:16.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.114+0000 7f0277405700 1 -- 172.21.15.1:0/3180464670 shutdown_connections 2024-03-18T20:28:16.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.114+0000 7f0277405700 1 -- 172.21.15.1:0/3180464670 wait complete. 2024-03-18T20:28:16.121 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 144 2024-03-18T20:28:16.235 INFO:tasks.ceph.mds.a.smithi001.stderr:2024-03-18T20:28:16.232+0000 7ff684585700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T20:28:16.534 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 144, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T20:28:10.668549+0000', 'modified': '2024-03-18T20:28:13.687432+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 13794}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_13794': {'gid': 13794, 'name': 'a', 'rank': 0, 'incarnation': 140, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.1:6836/1111156155', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6835', 'nonce': 1111156155}, {'type': 'v1', 'addr': '172.21.15.1:6836', 'nonce': 1111156155}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [38], 'metadata_pool': 37, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-18T20:28:16.534 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T20:28:16.534 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-18T20:28:16.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.660+0000 7fa8dd0c0700 1 Processor -- start 2024-03-18T20:28:16.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.660+0000 7fa8dd0c0700 1 -- start start 2024-03-18T20:28:16.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8dd0c0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8d815d800 0x7fa8d80b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8dd0c0700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d80cde40 con 0x7fa8d80cd910 2024-03-18T20:28:16.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8dd0c0700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d80b0910 con 0x7fa8d8161200 2024-03-18T20:28:16.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8dd0c0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d81620e0 con 0x7fa8d815d800 2024-03-18T20:28:16.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8d659c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa8d80cd910 0x7fa8d80d0c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40720/0 (socket says 172.21.15.1:40720) 2024-03-18T20:28:16.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8d659c700 1 -- 172.21.15.1:0/2853096518 learned_addr learned my addr 172.21.15.1:0/2853096518 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:16.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8d5d9b700 1 -- 172.21.15.1:0/2853096518 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208332127 0 0) 0x7fa8d80cde40 con 0x7fa8d80cd910 2024-03-18T20:28:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8d5d9b700 1 -- 172.21.15.1:0/2853096518 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8c00035b0 con 0x7fa8d80cd910 2024-03-18T20:28:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8d6d9d700 1 --2- 172.21.15.1:0/2853096518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8d815d800 0x7fa8d80b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8d5d9b700 1 -- 172.21.15.1:0/2853096518 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 630071339 0 0) 0x7fa8c00035b0 con 0x7fa8d80cd910 2024-03-18T20:28:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8d5d9b700 1 -- 172.21.15.1:0/2853096518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8d815d800 msgr2=0x7fa8d80b32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8d5d9b700 1 --2- 172.21.15.1:0/2853096518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8d815d800 0x7fa8d80b32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8d5d9b700 1 -- 172.21.15.1:0/2853096518 >> v1:172.21.15.165:6789/0 conn(0x7fa8d8161200 legacy=0x7fa8d80cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8d5d9b700 1 -- 172.21.15.1:0/2853096518 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d80b0bf0 con 0x7fa8d80cd910 2024-03-18T20:28:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8d5d9b700 1 -- 172.21.15.1:0/2853096518 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fa8cc001670 con 0x7fa8d80cd910 2024-03-18T20:28:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8d5d9b700 1 -- 172.21.15.1:0/2853096518 >> v1:172.21.15.1:6789/0 conn(0x7fa8d80cd910 legacy=0x7fa8d80d0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8d5d9b700 1 --2- 172.21.15.1:0/2853096518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8c0003a30 0x7fa8c0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8d5d9b700 1 --2- 172.21.15.1:0/2853096518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8d815d800 0x7fa8c0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8d5d9b700 1 --2- 172.21.15.1:0/2853096518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8c0004d90 0x7fa8c0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8d5d9b700 1 -- 172.21.15.1:0/2853096518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d81620e0 con 0x7fa8c0004d90 2024-03-18T20:28:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8d5d9b700 1 -- 172.21.15.1:0/2853096518 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c000bac0 con 0x7fa8c0003a30 2024-03-18T20:28:16.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.661+0000 7fa8d5d9b700 1 -- 172.21.15.1:0/2853096518 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c000bc80 con 0x7fa8d815d800 2024-03-18T20:28:16.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.662+0000 7fa8d759e700 1 --2- 172.21.15.1:0/2853096518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8c0004d90 0x7fa8c0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.662+0000 7fa8d6d9d700 1 --2- 172.21.15.1:0/2853096518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8c0003a30 0x7fa8c0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.662+0000 7fa8d659c700 1 --2- 172.21.15.1:0/2853096518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8d815d800 0x7fa8c0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.662+0000 7fa8d759e700 1 -- 172.21.15.1:0/2853096518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8d815d800 msgr2=0x7fa8c0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:16.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.662+0000 7fa8d759e700 1 --2- 172.21.15.1:0/2853096518 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8d815d800 0x7fa8c0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.662+0000 7fa8d759e700 1 -- 172.21.15.1:0/2853096518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8c0003a30 msgr2=0x7fa8c0003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:16.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.662+0000 7fa8d759e700 1 --2- 172.21.15.1:0/2853096518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8c0003a30 0x7fa8c0003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.662+0000 7fa8d759e700 1 -- 172.21.15.1:0/2853096518 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c00035b0 con 0x7fa8c0004d90 2024-03-18T20:28:16.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.662+0000 7fa8d759e700 1 --2- 172.21.15.1:0/2853096518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8c0004d90 0x7fa8c0005210 secure :-1 s=READY pgs=3295 cs=0 l=1 rev1=1 crypto rx=0x7fa8d000c160 tx=0x7fa8c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c8e3bfda9d7ad5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:16.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.662+0000 7fa8d5d9b700 1 -- 172.21.15.1:0/2853096518 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0001db0 con 0x7fa8c0004d90 2024-03-18T20:28:16.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.662+0000 7fa8d5d9b700 1 -- 172.21.15.1:0/2853096518 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d000cca0 con 0x7fa8c0004d90 2024-03-18T20:28:16.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.662+0000 7fa8d5d9b700 1 -- 172.21.15.1:0/2853096518 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0003c40 con 0x7fa8c0004d90 2024-03-18T20:28:16.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.663+0000 7fa8dd0c0700 1 -- 172.21.15.1:0/2853096518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8c0004d90 msgr2=0x7fa8c0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:16.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.663+0000 7fa8dd0c0700 1 --2- 172.21.15.1:0/2853096518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8c0004d90 0x7fa8c0005210 secure :-1 s=READY pgs=3295 cs=0 l=1 rev1=1 crypto rx=0x7fa8d000c160 tx=0x7fa8c0005880 comp rx=0 tx=0).stop 2024-03-18T20:28:16.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.663+0000 7fa8d759e700 1 -- 172.21.15.1:0/2853096518 reap_dead start 2024-03-18T20:28:16.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.663+0000 7fa8dd0c0700 1 -- 172.21.15.1:0/2853096518 shutdown_connections 2024-03-18T20:28:16.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.663+0000 7fa8dd0c0700 1 -- 172.21.15.1:0/2853096518 >> 172.21.15.1:0/2853096518 conn(0x7fa8d8072120 msgr2=0x7fa8d8099a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:16.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.663+0000 7fa8dd0c0700 1 -- 172.21.15.1:0/2853096518 shutdown_connections 2024-03-18T20:28:16.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.663+0000 7fa8dd0c0700 1 -- 172.21.15.1:0/2853096518 wait complete. 2024-03-18T20:28:16.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.664+0000 7fa8dd0c0700 1 Processor -- start 2024-03-18T20:28:16.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.664+0000 7fa8dd0c0700 1 -- start start 2024-03-18T20:28:16.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.665+0000 7fa8dd0c0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8d8144e10 0x7fa8d8152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.665+0000 7fa8dd0c0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8d81532d0 0x7fa8d8154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.665+0000 7fa8dd0c0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8d8155370 0x7fa8d815e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.665+0000 7fa8dd0c0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d8156950 con 0x7fa8d8155370 2024-03-18T20:28:16.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.665+0000 7fa8dd0c0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d81574d0 con 0x7fa8d81532d0 2024-03-18T20:28:16.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.665+0000 7fa8dd0c0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d8156ba0 con 0x7fa8d8144e10 2024-03-18T20:28:16.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.665+0000 7fa8d759e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8d8155370 0x7fa8d815e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.665+0000 7fa8d759e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8d8155370 0x7fa8d815e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57310/0 (socket says 172.21.15.1:57310) 2024-03-18T20:28:16.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.665+0000 7fa8d759e700 1 -- 172.21.15.1:0/961385834 learned_addr learned my addr 172.21.15.1:0/961385834 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:16.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.665+0000 7fa8d659c700 1 --2- 172.21.15.1:0/961385834 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8d81532d0 0x7fa8d8154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.665+0000 7fa8d6d9d700 1 --2- 172.21.15.1:0/961385834 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8d8144e10 0x7fa8d8152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.665+0000 7fa8d759e700 1 -- 172.21.15.1:0/961385834 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8d8144e10 msgr2=0x7fa8d8152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:16.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.665+0000 7fa8d759e700 1 --2- 172.21.15.1:0/961385834 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8d8144e10 0x7fa8d8152bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.665+0000 7fa8d759e700 1 -- 172.21.15.1:0/961385834 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8d81532d0 msgr2=0x7fa8d8154e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:16.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.665+0000 7fa8d759e700 1 --2- 172.21.15.1:0/961385834 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8d81532d0 0x7fa8d8154e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.665+0000 7fa8d759e700 1 -- 172.21.15.1:0/961385834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d8156e80 con 0x7fa8d8155370 2024-03-18T20:28:16.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.665+0000 7fa8d659c700 1 --2- 172.21.15.1:0/961385834 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8d81532d0 0x7fa8d8154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:28:16.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.666+0000 7fa8d759e700 1 --2- 172.21.15.1:0/961385834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8d8155370 0x7fa8d815e800 secure :-1 s=READY pgs=3296 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0015040 tx=0x7fa8d00014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:16.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.666+0000 7fa8bf7fe700 1 -- 172.21.15.1:0/961385834 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0003e10 con 0x7fa8d8155370 2024-03-18T20:28:16.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.666+0000 7fa8bf7fe700 1 -- 172.21.15.1:0/961385834 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d00050d0 con 0x7fa8d8155370 2024-03-18T20:28:16.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.666+0000 7fa8dd0c0700 1 -- 172.21.15.1:0/961385834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d8157160 con 0x7fa8d8155370 2024-03-18T20:28:16.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.666+0000 7fa8bf7fe700 1 -- 172.21.15.1:0/961385834 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0005460 con 0x7fa8d8155370 2024-03-18T20:28:16.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.666+0000 7fa8dd0c0700 1 -- 172.21.15.1:0/961385834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d8153750 con 0x7fa8d8155370 2024-03-18T20:28:16.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.667+0000 7fa8bf7fe700 1 -- 172.21.15.1:0/961385834 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa8d0019040 con 0x7fa8d8155370 2024-03-18T20:28:16.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.668+0000 7fa8bf7fe700 1 --2- 172.21.15.1:0/961385834 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8c406adb0 0x7fa8c406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.668+0000 7fa8bf7fe700 1 -- 172.21.15.1:0/961385834 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fa8d008a480 con 0x7fa8d8155370 2024-03-18T20:28:16.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.668+0000 7fa8d6d9d700 1 --2- 172.21.15.1:0/961385834 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8c406adb0 0x7fa8c406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.668+0000 7fa8d6d9d700 1 --2- 172.21.15.1:0/961385834 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8c406adb0 0x7fa8c406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8009aa0 tx=0x7fa8c8008040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:16.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.669+0000 7fa8e9269b80 1 Processor -- start 2024-03-18T20:28:16.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.670+0000 7fa8e9269b80 1 -- start start 2024-03-18T20:28:16.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.670+0000 7fa8e9269b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d28b8c10 0x55b9d28b8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.670+0000 7fa8e9269b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b9d288a0a0 0x55b9d28d5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.670+0000 7fa8e9269b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b9d28ca630 0x55b9d288d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.670+0000 7fa8e9269b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9d2803f20 con 0x55b9d28b8c10 2024-03-18T20:28:16.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.670+0000 7fa8e9269b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9d27e5c30 con 0x55b9d28ca630 2024-03-18T20:28:16.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.670+0000 7fa8e9269b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b9d27e3430 con 0x55b9d288a0a0 2024-03-18T20:28:16.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.671+0000 7fa8d6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d28b8c10 0x55b9d28b8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.671+0000 7fa8d6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d28b8c10 0x55b9d28b8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57314/0 (socket says 172.21.15.1:57314) 2024-03-18T20:28:16.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.671+0000 7fa8d6d9d700 1 -- 172.21.15.1:0/3574069901 learned_addr learned my addr 172.21.15.1:0/3574069901 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:16.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.671+0000 7fa8d6d9d700 1 -- 172.21.15.1:0/3574069901 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b9d288a0a0 msgr2=0x55b9d28d5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:16.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.671+0000 7fa8d6d9d700 1 --2- 172.21.15.1:0/3574069901 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b9d288a0a0 0x55b9d28d5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.671+0000 7fa8d6d9d700 1 -- 172.21.15.1:0/3574069901 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b9d28ca630 msgr2=0x55b9d288d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:16.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.671+0000 7fa8d6d9d700 1 --2- 172.21.15.1:0/3574069901 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b9d28ca630 0x55b9d288d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.671+0000 7fa8d6d9d700 1 -- 172.21.15.1:0/3574069901 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9d25d2790 con 0x55b9d28b8c10 2024-03-18T20:28:16.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.671+0000 7fa8d6d9d700 1 --2- 172.21.15.1:0/3574069901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d28b8c10 0x55b9d28b8ff0 secure :-1 s=READY pgs=3297 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8013c30 tx=0x7fa8c8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=938d481472bbb652 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:16.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.671+0000 7fa8bcff9700 1 -- 172.21.15.1:0/3574069901 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8004030 con 0x55b9d28b8c10 2024-03-18T20:28:16.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.671+0000 7fa8bcff9700 1 -- 172.21.15.1:0/3574069901 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c8004190 con 0x55b9d28b8c10 2024-03-18T20:28:16.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.671+0000 7fa8bcff9700 1 -- 172.21.15.1:0/3574069901 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8004500 con 0x55b9d28b8c10 2024-03-18T20:28:16.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.672+0000 7fa8e9269b80 1 -- 172.21.15.1:0/3574069901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d28b8c10 msgr2=0x55b9d28b8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:16.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.672+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/3574069901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d28b8c10 0x55b9d28b8ff0 secure :-1 s=READY pgs=3297 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8013c30 tx=0x7fa8c8016000 comp rx=0 tx=0).stop 2024-03-18T20:28:16.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.672+0000 7fa8e9269b80 1 -- 172.21.15.1:0/3574069901 shutdown_connections 2024-03-18T20:28:16.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.672+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/3574069901 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b9d28ca630 0x55b9d288d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.672+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/3574069901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d28b8c10 0x55b9d28b8ff0 unknown :-1 s=CLOSED pgs=3297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.672+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/3574069901 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b9d288a0a0 0x55b9d28d5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.672+0000 7fa8e9269b80 1 -- 172.21.15.1:0/3574069901 >> 172.21.15.1:0/3574069901 conn(0x55b9d27eccf0 msgr2=0x55b9d288ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:16.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.672+0000 7fa8e9269b80 1 -- 172.21.15.1:0/3574069901 shutdown_connections 2024-03-18T20:28:16.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.672+0000 7fa8e9269b80 1 -- 172.21.15.1:0/3574069901 wait complete. 2024-03-18T20:28:16.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.673+0000 7fa8e9269b80 1 -- start start 2024-03-18T20:28:16.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.673+0000 7fa8e9269b80 1 Processor -- start 2024-03-18T20:28:16.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.673+0000 7fa8e9269b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d288a0a0 0x55b9d28b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.673+0000 7fa8e9269b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b9d28b8c10 0x55b9d28b2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.673+0000 7fa8e9269b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b9d28ca630 0x55b9d288d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.673+0000 7fa8e9269b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9d27ded20 con 0x55b9d288a0a0 2024-03-18T20:28:16.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.673+0000 7fa8e9269b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9d277a770 con 0x55b9d28b8c10 2024-03-18T20:28:16.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.673+0000 7fa8e9269b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b9d279d6f0 con 0x55b9d28ca630 2024-03-18T20:28:16.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.673+0000 7fa8e9269b80 10 client.0 _ll_register_callbacks cb 0x55b9d25e65a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:28:16.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.674+0000 7fa8d6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d288a0a0 0x55b9d28b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.674+0000 7fa8d6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d288a0a0 0x55b9d28b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57318/0 (socket says 172.21.15.1:57318) 2024-03-18T20:28:16.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.674+0000 7fa8d6d9d700 1 -- 172.21.15.1:0/1152387600 learned_addr learned my addr 172.21.15.1:0/1152387600 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:16.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.674+0000 7fa8d659c700 1 --2- 172.21.15.1:0/1152387600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b9d28b8c10 0x55b9d28b2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.674+0000 7fa8d759e700 1 --2- 172.21.15.1:0/1152387600 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b9d28ca630 0x55b9d288d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.674+0000 7fa8d6d9d700 1 -- 172.21.15.1:0/1152387600 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b9d28ca630 msgr2=0x55b9d288d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:16.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.674+0000 7fa8d6d9d700 1 --2- 172.21.15.1:0/1152387600 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b9d28ca630 0x55b9d288d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.674+0000 7fa8d6d9d700 1 -- 172.21.15.1:0/1152387600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b9d28b8c10 msgr2=0x55b9d28b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:16.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.674+0000 7fa8d6d9d700 1 --2- 172.21.15.1:0/1152387600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b9d28b8c10 0x55b9d28b2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.674+0000 7fa8d6d9d700 1 -- 172.21.15.1:0/1152387600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c8013910 con 0x55b9d288a0a0 2024-03-18T20:28:16.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.674+0000 7fa8d6d9d700 1 --2- 172.21.15.1:0/1152387600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d288a0a0 0x55b9d28b2590 secure :-1 s=READY pgs=3298 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8026040 tx=0x7fa8c80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a391519ea8124d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:16.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.674+0000 7fa8b5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T20:28:16.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.674+0000 7fa8b5ffb700 1 -- 172.21.15.1:0/1152387600 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8016b00 con 0x55b9d288a0a0 2024-03-18T20:28:16.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.674+0000 7fa8b5ffb700 1 -- 172.21.15.1:0/1152387600 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(288..288 src has 1..288) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fa8c8016c60 con 0x55b9d288a0a0 2024-03-18T20:28:16.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.674+0000 7fa8e9269b80 1 -- 172.21.15.1:0/1152387600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b9d2880e90 con 0x55b9d288a0a0 2024-03-18T20:28:16.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.674+0000 7fa8b5ffb700 1 -- 172.21.15.1:0/1152387600 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8c8016e20 con 0x55b9d288a0a0 2024-03-18T20:28:16.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.674+0000 7fa8e9269b80 10 client.13944 fetch_fsmap learned FSMap version 144 2024-03-18T20:28:16.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.674+0000 7fa8e9269b80 1 -- 172.21.15.1:0/1152387600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=2+}) v3 -- 0x55b9d26bf0a0 con 0x55b9d288a0a0 2024-03-18T20:28:16.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.675+0000 7fa8b5ffb700 1 -- 172.21.15.1:0/1152387600 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b9d2880e90 con 0x55b9d288a0a0 2024-03-18T20:28:16.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.675+0000 7fa8b5ffb700 1 -- 172.21.15.1:0/1152387600 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 144) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa8c80036a0 con 0x55b9d288a0a0 2024-03-18T20:28:16.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.675+0000 7fa8e9269b80 10 client.13944 fetch_fsmap finished waiting for FSMap version 144 2024-03-18T20:28:16.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.675+0000 7fa8e9269b80 10 client.13944 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-18T20:28:16.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.675+0000 7fa8e9269b80 20 client.13944 populate_metadata read hostname 'smithi001' 2024-03-18T20:28:16.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.675+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/1152387600 >> [v2:172.21.15.1:6835/1111156155,v1:172.21.15.1:6836/1111156155] conn(0x55b9d288db10 0x55b9d288def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.675+0000 7fa8e9269b80 4 client.13944 mds_command: new command op to 13794 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T20:28:16.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.675+0000 7fa8e9269b80 1 -- 172.21.15.1:0/1152387600 --> [v2:172.21.15.1:6835/1111156155,v1:172.21.15.1:6836/1111156155] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b9d285e010 con 0x55b9d288db10 2024-03-18T20:28:16.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.675+0000 7fa8d659c700 1 --2- 172.21.15.1:0/1152387600 >> [v2:172.21.15.1:6835/1111156155,v1:172.21.15.1:6836/1111156155] conn(0x55b9d288db10 0x55b9d288def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.675+0000 7fa8d659c700 1 --2- 172.21.15.1:0/1152387600 >> [v2:172.21.15.1:6835/1111156155,v1:172.21.15.1:6836/1111156155] conn(0x55b9d288db10 0x55b9d288def0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfbc192dad44a329 server_cookie=112264560d495616 in_seq=0 out_seq=0 2024-03-18T20:28:16.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.675+0000 7fa8b5ffb700 10 client.13944 ms_handle_connect on v2:172.21.15.1:6835/1111156155 2024-03-18T20:28:16.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.676+0000 7fa8b5ffb700 1 -- 172.21.15.1:0/1152387600 <== mds.0 v2:172.21.15.1:6835/1111156155 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b9d285e010 con 0x55b9d288db10 2024-03-18T20:28:16.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.676+0000 7fa8b5ffb700 10 client.13944 handle_command_reply: tid=0 2024-03-18T20:28:16.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.676+0000 7fa8e9269b80 1 client.13944 shutdown 2024-03-18T20:28:16.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.676+0000 7fa8e9269b80 1 -- 172.21.15.1:0/1152387600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d288a0a0 msgr2=0x55b9d28b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:16.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.676+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/1152387600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d288a0a0 0x55b9d28b2590 secure :-1 s=READY pgs=3298 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8026040 tx=0x7fa8c80047c0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.676+0000 7fa8e9269b80 1 -- 172.21.15.1:0/1152387600 shutdown_connections 2024-03-18T20:28:16.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.676+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/1152387600 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b9d28ca630 0x55b9d288d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.676+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/1152387600 >> [v2:172.21.15.1:6835/1111156155,v1:172.21.15.1:6836/1111156155] conn(0x55b9d288db10 0x55b9d288def0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.676+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/1152387600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d288a0a0 0x55b9d28b2590 unknown :-1 s=CLOSED pgs=3298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.676+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/1152387600 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b9d28b8c10 0x55b9d28b2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.676+0000 7fa8b5ffb700 0 client.13944 ms_handle_reset on v2:172.21.15.1:6835/1111156155 2024-03-18T20:28:16.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.676+0000 7fa8e9269b80 1 -- 172.21.15.1:0/1152387600 >> 172.21.15.1:0/1152387600 conn(0x55b9d27eccf0 msgr2=0x55b9d28b6510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:16.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.676+0000 7fa8e9269b80 1 -- 172.21.15.1:0/1152387600 shutdown_connections 2024-03-18T20:28:16.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.676+0000 7fa8e9269b80 1 -- 172.21.15.1:0/1152387600 wait complete. 2024-03-18T20:28:16.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.676+0000 7fa8e9269b80 20 client.13944 trim_cache size 0 max 16384 2024-03-18T20:28:16.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.694+0000 7fa8e9269b80 1 Processor -- start 2024-03-18T20:28:16.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.694+0000 7fa8e9269b80 1 -- start start 2024-03-18T20:28:16.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8e9269b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d288a0a0 0x55b9d28b0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8e9269b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b9d28b8c10 0x55b9d288f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8e9269b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b9d28ca630 0x55b9d28d3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8e9269b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9d25d3390 con 0x55b9d288a0a0 2024-03-18T20:28:16.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8e9269b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9d28b96b0 con 0x55b9d28b8c10 2024-03-18T20:28:16.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8e9269b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b9d28b6a40 con 0x55b9d28ca630 2024-03-18T20:28:16.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8d6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d288a0a0 0x55b9d28b0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8d6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d288a0a0 0x55b9d28b0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57324/0 (socket says 172.21.15.1:57324) 2024-03-18T20:28:16.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8d6d9d700 1 -- 172.21.15.1:0/2605083569 learned_addr learned my addr 172.21.15.1:0/2605083569 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:16.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8d659c700 1 --2- 172.21.15.1:0/2605083569 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b9d28b8c10 0x55b9d288f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8d759e700 1 --2- 172.21.15.1:0/2605083569 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b9d28ca630 0x55b9d28d3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8d6d9d700 1 -- 172.21.15.1:0/2605083569 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b9d28ca630 msgr2=0x55b9d28d3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:16.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8d6d9d700 1 --2- 172.21.15.1:0/2605083569 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b9d28ca630 0x55b9d28d3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8d6d9d700 1 -- 172.21.15.1:0/2605083569 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b9d28b8c10 msgr2=0x55b9d288f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:16.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8d6d9d700 1 --2- 172.21.15.1:0/2605083569 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b9d28b8c10 0x55b9d288f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8d6d9d700 1 -- 172.21.15.1:0/2605083569 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9d27a5010 con 0x55b9d288a0a0 2024-03-18T20:28:16.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8d6d9d700 1 --2- 172.21.15.1:0/2605083569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d288a0a0 0x55b9d28b0af0 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 crypto rx=0x7fa8c80167f0 tx=0x7fa8c802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3129f8eeb084f7e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:16.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8bd7fa700 1 -- 172.21.15.1:0/2605083569 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8047070 con 0x55b9d288a0a0 2024-03-18T20:28:16.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8bd7fa700 1 -- 172.21.15.1:0/2605083569 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c800f040 con 0x55b9d288a0a0 2024-03-18T20:28:16.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.695+0000 7fa8bd7fa700 1 -- 172.21.15.1:0/2605083569 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8004510 con 0x55b9d288a0a0 2024-03-18T20:28:16.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.696+0000 7fa8e9269b80 1 -- 172.21.15.1:0/2605083569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d288a0a0 msgr2=0x55b9d28b0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:16.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.696+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/2605083569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d288a0a0 0x55b9d28b0af0 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 crypto rx=0x7fa8c80167f0 tx=0x7fa8c802f000 comp rx=0 tx=0).stop 2024-03-18T20:28:16.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.696+0000 7fa8e9269b80 1 -- 172.21.15.1:0/2605083569 shutdown_connections 2024-03-18T20:28:16.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.696+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/2605083569 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b9d28ca630 0x55b9d28d3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.696+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/2605083569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d288a0a0 0x55b9d28b0af0 unknown :-1 s=CLOSED pgs=3299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.696+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/2605083569 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b9d28b8c10 0x55b9d288f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.696+0000 7fa8e9269b80 1 -- 172.21.15.1:0/2605083569 >> 172.21.15.1:0/2605083569 conn(0x55b9d27eccf0 msgr2=0x55b9d28cac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:16.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.696+0000 7fa8e9269b80 1 -- 172.21.15.1:0/2605083569 shutdown_connections 2024-03-18T20:28:16.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.696+0000 7fa8e9269b80 1 -- 172.21.15.1:0/2605083569 wait complete. 2024-03-18T20:28:16.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.696+0000 7fa8e9269b80 1 -- start start 2024-03-18T20:28:16.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.696+0000 7fa8e9269b80 1 Processor -- start 2024-03-18T20:28:16.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.697+0000 7fa8e9269b80 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b9d288a0a0 0x55b9d2924610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.697+0000 7fa8e9269b80 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b9d28b8c10 0x55b9d288f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.697+0000 7fa8e9269b80 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d28ca630 0x55b9d288bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.697+0000 7fa8e9269b80 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9d2888930 con 0x55b9d28ca630 2024-03-18T20:28:16.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.697+0000 7fa8e9269b80 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9d28cd0a0 con 0x55b9d288a0a0 2024-03-18T20:28:16.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.697+0000 7fa8e9269b80 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b9d288fd30 con 0x55b9d28b8c10 2024-03-18T20:28:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.697+0000 7fa8e9269b80 10 client.0 _ll_register_callbacks cb 0x7fa8c8013b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T20:28:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.697+0000 7fa8d759e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d28ca630 0x55b9d288bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.697+0000 7fa8d759e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d28ca630 0x55b9d288bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57332/0 (socket says 172.21.15.1:57332) 2024-03-18T20:28:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.697+0000 7fa8d759e700 1 -- 172.21.15.1:0/2502145394 learned_addr learned my addr 172.21.15.1:0/2502145394 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.697+0000 7fa8d659c700 1 --2- 172.21.15.1:0/2502145394 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b9d28b8c10 0x55b9d288f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8d759e700 1 -- 172.21.15.1:0/2502145394 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b9d28b8c10 msgr2=0x55b9d288f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8d759e700 1 --2- 172.21.15.1:0/2502145394 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b9d28b8c10 0x55b9d288f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8d759e700 1 -- 172.21.15.1:0/2502145394 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b9d288a0a0 msgr2=0x55b9d2924610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T20:28:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8d759e700 1 --2- 172.21.15.1:0/2502145394 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b9d288a0a0 0x55b9d2924610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8d759e700 1 -- 172.21.15.1:0/2502145394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b9d26bf0a0 con 0x55b9d28ca630 2024-03-18T20:28:16.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8d759e700 1 --2- 172.21.15.1:0/2502145394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d28ca630 0x55b9d288bc00 secure :-1 s=READY pgs=3300 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0058e70 tx=0x7fa8d0058b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6641554cfdc2662 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8b5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-03-18T20:28:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8b5ffb700 1 -- 172.21.15.1:0/2502145394 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0059870 con 0x55b9d28ca630 2024-03-18T20:28:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8b5ffb700 1 -- 172.21.15.1:0/2502145394 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(288..288 src has 1..288) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fa8d001be10 con 0x55b9d28ca630 2024-03-18T20:28:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8e9269b80 1 -- 172.21.15.1:0/2502145394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa8c8004190 con 0x55b9d28ca630 2024-03-18T20:28:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8b5ffb700 1 -- 172.21.15.1:0/2502145394 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8d0059170 con 0x55b9d28ca630 2024-03-18T20:28:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8e9269b80 10 client.13950 fetch_fsmap learned FSMap version 144 2024-03-18T20:28:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8e9269b80 1 -- 172.21.15.1:0/2502145394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=2+}) v3 -- 0x55b9d26331d0 con 0x55b9d28ca630 2024-03-18T20:28:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8b5ffb700 1 -- 172.21.15.1:0/2502145394 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c8004190 con 0x55b9d28ca630 2024-03-18T20:28:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8b5ffb700 1 -- 172.21.15.1:0/2502145394 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 144) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa8d008a0b0 con 0x55b9d28ca630 2024-03-18T20:28:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8e9269b80 10 client.13950 fetch_fsmap finished waiting for FSMap version 144 2024-03-18T20:28:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8e9269b80 10 client.13950 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-18T20:28:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8e9269b80 20 client.13950 populate_metadata read hostname 'smithi001' 2024-03-18T20:28:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/2502145394 >> [v2:172.21.15.1:6835/1111156155,v1:172.21.15.1:6836/1111156155] conn(0x55b9d28ec730 0x55b9d28ecb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8e9269b80 4 client.13950 mds_command: new command op to 13794 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T20:28:16.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.698+0000 7fa8e9269b80 1 -- 172.21.15.1:0/2502145394 --> [v2:172.21.15.1:6835/1111156155,v1:172.21.15.1:6836/1111156155] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b9d2618040 con 0x55b9d28ec730 2024-03-18T20:28:16.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.699+0000 7fa8d6d9d700 1 --2- 172.21.15.1:0/2502145394 >> [v2:172.21.15.1:6835/1111156155,v1:172.21.15.1:6836/1111156155] conn(0x55b9d28ec730 0x55b9d28ecb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.699+0000 7fa8d6d9d700 1 --2- 172.21.15.1:0/2502145394 >> [v2:172.21.15.1:6835/1111156155,v1:172.21.15.1:6836/1111156155] conn(0x55b9d28ec730 0x55b9d28ecb10 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf88b4a1f1f68232 server_cookie=62b7f268aa3bc9c in_seq=0 out_seq=0 2024-03-18T20:28:16.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.699+0000 7fa8b5ffb700 10 client.13950 ms_handle_connect on v2:172.21.15.1:6835/1111156155 2024-03-18T20:28:16.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.699+0000 7fa8b5ffb700 1 -- 172.21.15.1:0/2502145394 <== mds.0 v2:172.21.15.1:6835/1111156155 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55b9d2618040 con 0x55b9d28ec730 2024-03-18T20:28:16.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.699+0000 7fa8b5ffb700 10 client.13950 handle_command_reply: tid=0 2024-03-18T20:28:16.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.699+0000 7fa8e9269b80 1 client.13950 shutdown 2024-03-18T20:28:16.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.699+0000 7fa8e9269b80 1 -- 172.21.15.1:0/2502145394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d28ca630 msgr2=0x55b9d288bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:16.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.699+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/2502145394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d28ca630 0x55b9d288bc00 secure :-1 s=READY pgs=3300 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0058e70 tx=0x7fa8d0058b80 comp rx=0 tx=0).stop 2024-03-18T20:28:16.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.699+0000 7fa8e9269b80 1 -- 172.21.15.1:0/2502145394 shutdown_connections 2024-03-18T20:28:16.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.699+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/2502145394 >> [v2:172.21.15.1:6835/1111156155,v1:172.21.15.1:6836/1111156155] conn(0x55b9d28ec730 0x55b9d28ecb10 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.699+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/2502145394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9d28ca630 0x55b9d288bc00 unknown :-1 s=CLOSED pgs=3300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.699+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/2502145394 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b9d288a0a0 0x55b9d2924610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.699+0000 7fa8b5ffb700 0 client.13950 ms_handle_reset on v2:172.21.15.1:6835/1111156155 2024-03-18T20:28:16.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.699+0000 7fa8e9269b80 1 --2- 172.21.15.1:0/2502145394 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x55b9d28b8c10 0x55b9d288f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.699+0000 7fa8e9269b80 1 -- 172.21.15.1:0/2502145394 >> 172.21.15.1:0/2502145394 conn(0x55b9d27eccf0 msgr2=0x55b9d28b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:16.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.699+0000 7fa8e9269b80 1 -- 172.21.15.1:0/2502145394 shutdown_connections 2024-03-18T20:28:16.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.699+0000 7fa8e9269b80 1 -- 172.21.15.1:0/2502145394 wait complete. 2024-03-18T20:28:16.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.700+0000 7fa8e9269b80 20 client.13950 trim_cache size 0 max 16384 2024-03-18T20:28:16.711 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:28:16.711 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "9c1c697f-8655-4ef3-8a13-da568d2cbe86", 2024-03-18T20:28:16.711 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-03-18T20:28:16.711 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 13794, 2024-03-18T20:28:16.712 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-03-18T20:28:16.712 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-03-18T20:28:16.712 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-03-18T20:28:16.712 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 6.0297112329999996, 2024-03-18T20:28:16.712 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 144, 2024-03-18T20:28:16.712 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 288, 2024-03-18T20:28:16.712 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 288, 2024-03-18T20:28:16.712 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 11.348496987000001 2024-03-18T20:28:16.712 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:28:16.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.700+0000 7fa8dd0c0700 1 -- 172.21.15.1:0/961385834 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8c406adb0 msgr2=0x7fa8c406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:16.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.700+0000 7fa8dd0c0700 1 --2- 172.21.15.1:0/961385834 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8c406adb0 0x7fa8c406d270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8009aa0 tx=0x7fa8c8008040 comp rx=0 tx=0).stop 2024-03-18T20:28:16.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.700+0000 7fa8dd0c0700 1 -- 172.21.15.1:0/961385834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8d8155370 msgr2=0x7fa8d815e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:16.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.700+0000 7fa8dd0c0700 1 --2- 172.21.15.1:0/961385834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8d8155370 0x7fa8d815e800 secure :-1 s=READY pgs=3296 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0015040 tx=0x7fa8d00014a0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.700+0000 7fa8dd0c0700 1 -- 172.21.15.1:0/961385834 shutdown_connections 2024-03-18T20:28:16.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.700+0000 7fa8dd0c0700 1 --2- 172.21.15.1:0/961385834 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa8c406adb0 0x7fa8c406d270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.700+0000 7fa8dd0c0700 1 --2- 172.21.15.1:0/961385834 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa8d8144e10 0x7fa8d8152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.700+0000 7fa8dd0c0700 1 --2- 172.21.15.1:0/961385834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8d8155370 0x7fa8d815e800 unknown :-1 s=CLOSED pgs=3296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.700+0000 7fa8dd0c0700 1 --2- 172.21.15.1:0/961385834 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8d81532d0 0x7fa8d8154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.700+0000 7fa8dd0c0700 1 -- 172.21.15.1:0/961385834 >> 172.21.15.1:0/961385834 conn(0x7fa8d8072120 msgr2=0x7fa8d8158b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:16.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.700+0000 7fa8dd0c0700 1 -- 172.21.15.1:0/961385834 shutdown_connections 2024-03-18T20:28:16.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.700+0000 7fa8dd0c0700 1 -- 172.21.15.1:0/961385834 wait complete. 2024-03-18T20:28:16.717 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T20:28:16.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.838+0000 7faccf259700 1 Processor -- start 2024-03-18T20:28:16.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.838+0000 7faccf259700 1 -- start start 2024-03-18T20:28:16.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.839+0000 7faccf259700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7facc815d5e0 0x7facc80d1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.839+0000 7faccf259700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facc80d3f80 0x7facc815d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.839+0000 7faccf259700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc80cfe30 con 0x7facc815df00 2024-03-18T20:28:16.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.839+0000 7faccf259700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc80d0110 con 0x7facc815d5e0 2024-03-18T20:28:16.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.839+0000 7faccf259700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc80d03f0 con 0x7facc80d3f80 2024-03-18T20:28:16.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.839+0000 7facccff5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7facc815d5e0 0x7facc80d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.839+0000 7facccff5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7facc815d5e0 0x7facc80d1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57334/0 (socket says 172.21.15.1:57334) 2024-03-18T20:28:16.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.839+0000 7facccff5700 1 -- 172.21.15.1:0/3828051643 learned_addr learned my addr 172.21.15.1:0/3828051643 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:16.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.839+0000 7facc7fff700 1 --2- 172.21.15.1:0/3828051643 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facc80d3f80 0x7facc815d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.839+0000 7facccff5700 1 -- 172.21.15.1:0/3828051643 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facc80d3f80 msgr2=0x7facc815d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:16.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.839+0000 7facccff5700 1 --2- 172.21.15.1:0/3828051643 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facc80d3f80 0x7facc815d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.839+0000 7facccff5700 1 -- 172.21.15.1:0/3828051643 >> v1:172.21.15.165:6789/0 conn(0x7facc815df00 legacy=0x7facc8161000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:16.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.839+0000 7facccff5700 1 -- 172.21.15.1:0/3828051643 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc80d06d0 con 0x7facc815d5e0 2024-03-18T20:28:16.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.839+0000 7facc7fff700 1 --2- 172.21.15.1:0/3828051643 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facc80d3f80 0x7facc815d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:28:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.839+0000 7facccff5700 1 --2- 172.21.15.1:0/3828051643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facc815d5e0 0x7facc80d1dd0 secure :-1 s=READY pgs=3301 cs=0 l=1 rev1=1 crypto rx=0x7facbc0098f0 tx=0x7facbc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be36e82514ad1ba5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.839+0000 7facc77fe700 1 -- 172.21.15.1:0/3828051643 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc00b8a0 con 0x7facc815d5e0 2024-03-18T20:28:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.839+0000 7facc77fe700 1 -- 172.21.15.1:0/3828051643 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facbc00ba00 con 0x7facc815d5e0 2024-03-18T20:28:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.839+0000 7facc77fe700 1 -- 172.21.15.1:0/3828051643 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc00bd90 con 0x7facc815d5e0 2024-03-18T20:28:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.840+0000 7faccf259700 1 -- 172.21.15.1:0/3828051643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facc815d5e0 msgr2=0x7facc80d1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:16.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.840+0000 7faccf259700 1 --2- 172.21.15.1:0/3828051643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facc815d5e0 0x7facc80d1dd0 secure :-1 s=READY pgs=3301 cs=0 l=1 rev1=1 crypto rx=0x7facbc0098f0 tx=0x7facbc009c00 comp rx=0 tx=0).stop 2024-03-18T20:28:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.840+0000 7faccf259700 1 -- 172.21.15.1:0/3828051643 shutdown_connections 2024-03-18T20:28:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.840+0000 7faccf259700 1 --2- 172.21.15.1:0/3828051643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facc815d5e0 0x7facc80d1dd0 unknown :-1 s=CLOSED pgs=3301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.840+0000 7faccf259700 1 --2- 172.21.15.1:0/3828051643 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facc80d3f80 0x7facc815d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.840+0000 7faccf259700 1 -- 172.21.15.1:0/3828051643 >> 172.21.15.1:0/3828051643 conn(0x7facc80d3380 msgr2=0x7facc80d3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.840+0000 7faccf259700 1 -- 172.21.15.1:0/3828051643 shutdown_connections 2024-03-18T20:28:16.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.840+0000 7faccf259700 1 -- 172.21.15.1:0/3828051643 wait complete. 2024-03-18T20:28:16.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.841+0000 7faccf259700 1 Processor -- start 2024-03-18T20:28:16.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.841+0000 7faccf259700 1 -- start start 2024-03-18T20:28:16.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.842+0000 7faccf259700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facc80d3f80 0x7facc8158590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.842+0000 7faccf259700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facc815d5e0 0x7facc80cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.842+0000 7faccf259700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facc815df00 0x7facc80b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.842+0000 7faccf259700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc80cc160 con 0x7facc815d5e0 2024-03-18T20:28:16.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.842+0000 7faccf259700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc80cc440 con 0x7facc80d3f80 2024-03-18T20:28:16.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.842+0000 7faccf259700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc80cc720 con 0x7facc815df00 2024-03-18T20:28:16.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.842+0000 7facc7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facc815d5e0 0x7facc80cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.842+0000 7facc7fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facc815d5e0 0x7facc80cb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57348/0 (socket says 172.21.15.1:57348) 2024-03-18T20:28:16.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.842+0000 7facc7fff700 1 -- 172.21.15.1:0/3547183091 learned_addr learned my addr 172.21.15.1:0/3547183091 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:16.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.842+0000 7facccff5700 1 --2- 172.21.15.1:0/3547183091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facc80d3f80 0x7facc8158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.842+0000 7faccd7f6700 1 --2- 172.21.15.1:0/3547183091 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facc815df00 0x7facc80b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.843+0000 7facc7fff700 1 -- 172.21.15.1:0/3547183091 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facc815df00 msgr2=0x7facc80b40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:16.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.843+0000 7facc7fff700 1 --2- 172.21.15.1:0/3547183091 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facc815df00 0x7facc80b40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.843+0000 7facc7fff700 1 -- 172.21.15.1:0/3547183091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facc80d3f80 msgr2=0x7facc8158590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.843+0000 7facc7fff700 1 --2- 172.21.15.1:0/3547183091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facc80d3f80 0x7facc8158590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.843+0000 7facc7fff700 1 -- 172.21.15.1:0/3547183091 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc009550 con 0x7facc815d5e0 2024-03-18T20:28:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.843+0000 7facc7fff700 1 --2- 172.21.15.1:0/3547183091 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facc815d5e0 0x7facc80cb8a0 secure :-1 s=READY pgs=3302 cs=0 l=1 rev1=1 crypto rx=0x7facb400e810 tx=0x7facb400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.843+0000 7facc57fa700 1 -- 172.21.15.1:0/3547183091 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb4013610 con 0x7facc815d5e0 2024-03-18T20:28:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.843+0000 7facc57fa700 1 -- 172.21.15.1:0/3547183091 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb4013770 con 0x7facc815d5e0 2024-03-18T20:28:16.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.843+0000 7faccf259700 1 -- 172.21.15.1:0/3547183091 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc80d2490 con 0x7facc815d5e0 2024-03-18T20:28:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.843+0000 7facc57fa700 1 -- 172.21.15.1:0/3547183091 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb4013b20 con 0x7facc815d5e0 2024-03-18T20:28:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.843+0000 7faccf259700 1 -- 172.21.15.1:0/3547183091 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc80ccce0 con 0x7facc815d5e0 2024-03-18T20:28:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.844+0000 7faccf259700 1 -- 172.21.15.1:0/3547183091 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc8067dc0 con 0x7facc815d5e0 2024-03-18T20:28:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.845+0000 7facc57fa700 1 -- 172.21.15.1:0/3547183091 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7facb4013c80 con 0x7facc815d5e0 2024-03-18T20:28:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.845+0000 7facc57fa700 1 --2- 172.21.15.1:0/3547183091 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facac06adb0 0x7facac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:16.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.845+0000 7facc57fa700 1 -- 172.21.15.1:0/3547183091 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6129+0+0 (secure 0 0 0) 0x7facb4089330 con 0x7facc815d5e0 2024-03-18T20:28:16.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.845+0000 7facccff5700 1 --2- 172.21.15.1:0/3547183091 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facac06adb0 0x7facac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:16.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.845+0000 7facccff5700 1 --2- 172.21.15.1:0/3547183091 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facac06adb0 0x7facac06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7facbc009c40 tx=0x7facbc008030 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:16.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:16.849+0000 7facc57fa700 1 -- 172.21.15.1:0/3547183091 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7facb4054140 con 0x7facc815d5e0 2024-03-18T20:28:17.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:17.099+0000 7faccf259700 1 -- 172.21.15.1:0/3547183091 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7facc80cfb70 con 0x7facc815d5e0 2024-03-18T20:28:17.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:17.099+0000 7facc57fa700 1 -- 172.21.15.1:0/3547183091 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v144) v1 ==== 53+0+73 (secure 0 0 0) 0x7facb4057760 con 0x7facc815d5e0 2024-03-18T20:28:17.103 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T20:28:17.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:17.101+0000 7faccf259700 1 -- 172.21.15.1:0/3547183091 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facac06adb0 msgr2=0x7facac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:17.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:17.101+0000 7faccf259700 1 --2- 172.21.15.1:0/3547183091 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facac06adb0 0x7facac06d270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7facbc009c40 tx=0x7facbc008030 comp rx=0 tx=0).stop 2024-03-18T20:28:17.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:17.101+0000 7faccf259700 1 -- 172.21.15.1:0/3547183091 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facc815d5e0 msgr2=0x7facc80cb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:17.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:17.101+0000 7faccf259700 1 --2- 172.21.15.1:0/3547183091 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facc815d5e0 0x7facc80cb8a0 secure :-1 s=READY pgs=3302 cs=0 l=1 rev1=1 crypto rx=0x7facb400e810 tx=0x7facb400eb20 comp rx=0 tx=0).stop 2024-03-18T20:28:17.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:17.101+0000 7faccf259700 1 -- 172.21.15.1:0/3547183091 shutdown_connections 2024-03-18T20:28:17.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:17.101+0000 7faccf259700 1 --2- 172.21.15.1:0/3547183091 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facac06adb0 0x7facac06d270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:17.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:17.101+0000 7faccf259700 1 --2- 172.21.15.1:0/3547183091 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facc815df00 0x7facc80b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:17.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:17.101+0000 7faccf259700 1 --2- 172.21.15.1:0/3547183091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facc80d3f80 0x7facc8158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:17.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:17.101+0000 7faccf259700 1 --2- 172.21.15.1:0/3547183091 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facc815d5e0 0x7facc80cb8a0 unknown :-1 s=CLOSED pgs=3302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:17.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:17.101+0000 7faccf259700 1 -- 172.21.15.1:0/3547183091 >> 172.21.15.1:0/3547183091 conn(0x7facc80d3380 msgr2=0x7facc80b11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:17.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:17.101+0000 7faccf259700 1 -- 172.21.15.1:0/3547183091 shutdown_connections 2024-03-18T20:28:17.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:17.101+0000 7faccf259700 1 -- 172.21.15.1:0/3547183091 wait complete. 2024-03-18T20:28:17.118 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T20:28:17.118 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T20:28:17.118 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-03-18T20:28:17.118 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T20:28:17.118 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T20:28:17.118 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T20:28:17.118 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T20:28:17.118 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T20:28:17.118 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:28:17.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:28:17.119 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-03-18T20:28:17.138 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T20:28:17.138 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T20:28:17.138 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T20:28:17.138 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:28:17.138 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-03-18T20:28:17.138 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:28:17.138 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:28:17.138 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-03-18T20:28:17.138 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-03-18T20:28:17.138 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:28:17.139 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-03-18T20:28:17.139 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-03-18T20:28:17.139 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T20:28:17.139 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:28:17.139 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-03-18T20:28:17.139 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T20:28:17.139 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 36672sec preferred_lft 36672sec 2024-03-18T20:28:17.139 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link 2024-03-18T20:28:17.139 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:28:17.139 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T20:28:17.139 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:28:17.139 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-03-18T20:28:17.139 INFO:teuthology.orchestra.run.smithi001.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-18T20:28:17.139 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff 2024-03-18T20:28:17.139 INFO:teuthology.orchestra.run.smithi001.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-18T20:28:17.140 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:28:17.140 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::4ca9:f5ff:feb3:79a3/64 scope link 2024-03-18T20:28:17.140 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:28:17.140 INFO:teuthology.orchestra.run.smithi001.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T20:28:17.140 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2a:2f:a4:1a:99:08 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T20:28:17.140 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::282f:a4ff:fe1a:9908/64 scope link 2024-03-18T20:28:17.140 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:28:17.140 INFO:teuthology.orchestra.run.smithi001.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-18T20:28:17.140 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 2e:5c:60:78:b2:8c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T20:28:17.140 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::2c5c:60ff:fe78:b28c/64 scope link 2024-03-18T20:28:17.140 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-03-18T20:28:17.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:28:17.141 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T20:28:17.204 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-18T20:28:17.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:28:17.206 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-03-18T20:28:17.265 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-03-18T20:28:17.266 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T20:28:17.266 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T20:28:17.266 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T20:28:17.266 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T20:28:17.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:28:17.267 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec bash -c ' 2024-03-18T20:28:17.267 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-03-18T20:28:17.267 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T20:28:17.267 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T20:28:17.267 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T20:28:17.267 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-03-18T20:28:17.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:28:17.372 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T20:28:17.390 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T20:28:17.390 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T20:28:17.391 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:28:17.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:28:17.391 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-03-18T20:28:17.459 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-18T20:28:17.459 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:28:17.459 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T20:28:17.517 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T20:28:17.517 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T20:28:17.517 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T20:28:17.575 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T20:28:17.643 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:28:17.644 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:28:17.711 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:28:17.712 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:28:17.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:28:17.713 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T20:28:17.771 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T20:28:17.771 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-18T20:28:17.813 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-03-18T20:28:17.841 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-18T20:28:17.842 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-18T20:28:17.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-03-18T20:28:17.898+0000 7f9e353295c0 -1 init, newargv = 0x564814d4f120 newargc=15 2024-03-18T20:28:17.902 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[175864]: starting ceph client 2024-03-18T20:28:17.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[175864]: starting fuse 2024-03-18T20:28:17.912 INFO:teuthology.orchestra.run.smithi001.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-18T20:28:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-18T20:28:17.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:28:17.913 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-03-18T20:28:17.972 INFO:teuthology.orchestra.run.smithi001.stdout:47 2024-03-18T20:28:17.972 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2024-03-18T20:28:17.972 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-18T20:28:17.972 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-03-18T20:28:17.972 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-03-18T20:28:17.973 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-03-18T20:28:17.973 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-03-18T20:28:17.973 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:28:17.973 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-03-18T20:28:17.973 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T20:28:17.973 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-03-18T20:28:17.973 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T20:28:17.973 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:28:17.973 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-03-18T20:28:17.973 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-03-18T20:28:17.973 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-03-18T20:28:17.973 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-03-18T20:28:17.973 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:28:17.973 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-03-18T20:28:17.973 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T20:28:17.974 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T20:28:17.974 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T20:28:17.974 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-03-18T20:28:17.974 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-03-18T20:28:17.974 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-03-18T20:28:17.974 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T20:28:17.974 DEBUG:teuthology.orchestra.run.smithi001:> 2024-03-18T20:28:17.974 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-03-18T20:28:17.974 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-03-18T20:28:18.097 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.175864.asok 2024-03-18T20:28:18.279 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.175864.asok 2024-03-18T20:28:18.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:28:18.280 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.175864.asok status 2024-03-18T20:28:18.440 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:28:18.441 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-03-18T20:28:18.441 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "49bdc51142759622a281d3e5327eebd07cbb8bb3", 2024-03-18T20:28:18.441 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 18.2.2-914-g49bdc511 (49bdc51142759622a281d3e5327eebd07cbb8bb3) reef (stable)", 2024-03-18T20:28:18.441 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-03-18T20:28:18.441 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-03-18T20:28:18.441 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T20:28:18.441 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "175864", 2024-03-18T20:28:18.441 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-03-18T20:28:18.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:18.441 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-03-18T20:28:18.441 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-03-18T20:28:18.441 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 13962, 2024-03-18T20:28:18.441 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-03-18T20:28:18.442 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-03-18T20:28:18.442 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-03-18T20:28:18.442 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 13962 2024-03-18T20:28:18.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:18.442 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T20:28:18.442 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T20:28:18.442 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:28:18.442 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1634978370 2024-03-18T20:28:18.442 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:18.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:18.442 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-03-18T20:28:18.442 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-03-18T20:28:18.442 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-03-18T20:28:18.442 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1634978370 2024-03-18T20:28:18.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-03-18T20:28:18.443 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.13962 192.168.0.1:0/1634978370", 2024-03-18T20:28:18.443 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/1634978370", 2024-03-18T20:28:18.443 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-03-18T20:28:18.443 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 144, 2024-03-18T20:28:18.443 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 288, 2024-03-18T20:28:18.443 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-03-18T20:28:18.443 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-03-18T20:28:18.443 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-03-18T20:28:18.443 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:28:18.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:28:18.451 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-03-18T20:28:18.451 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T20:28:18.469 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-03-18T20:28:18.470 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T20:28:18.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:28:18.470 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T20:28:18.538 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-03-18T20:28:18.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.702+0000 7fedfc299700 1 Processor -- start 2024-03-18T20:28:18.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.702+0000 7fedfc299700 1 -- start start 2024-03-18T20:28:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.703+0000 7fedfc299700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fedf4099a30 0x7fedf4099e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.703+0000 7fedfc299700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fedf415e8e0 0x7fedf40cf380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.703+0000 7fedfc299700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf406ec60 con 0x7fedf415ef60 2024-03-18T20:28:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.703+0000 7fedfc299700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf4070100 con 0x7fedf415e8e0 2024-03-18T20:28:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.703+0000 7fedfc299700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf409a350 con 0x7fedf4099a30 2024-03-18T20:28:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.703+0000 7fedf9834700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fedf415e8e0 0x7fedf40cf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.703+0000 7fedf9834700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fedf415e8e0 0x7fedf40cf380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57354/0 (socket says 172.21.15.1:57354) 2024-03-18T20:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.703+0000 7fedf9834700 1 -- 172.21.15.1:0/1055857832 learned_addr learned my addr 172.21.15.1:0/1055857832 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.703+0000 7fedfa035700 1 --2- 172.21.15.1:0/1055857832 >> v2:172.21.15.165:3300/0 conn(0x7fedf4099a30 0x7fedf4099e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.703+0000 7fedf9834700 1 -- 172.21.15.1:0/1055857832 >> v2:172.21.15.165:3300/0 conn(0x7fedf4099a30 msgr2=0x7fedf4099e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.703+0000 7fedf9834700 1 --2- 172.21.15.1:0/1055857832 >> v2:172.21.15.165:3300/0 conn(0x7fedf4099a30 0x7fedf4099e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.703+0000 7fedf9834700 1 -- 172.21.15.1:0/1055857832 >> v1:172.21.15.165:6789/0 conn(0x7fedf415ef60 legacy=0x7fedf415cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.703+0000 7fedf9834700 1 -- 172.21.15.1:0/1055857832 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf40cfd80 con 0x7fedf415e8e0 2024-03-18T20:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.703+0000 7fedf9834700 1 --2- 172.21.15.1:0/1055857832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedf415e8e0 0x7fedf40cf380 secure :-1 s=READY pgs=3305 cs=0 l=1 rev1=1 crypto rx=0x7fedf4071e10 tx=0x7fede4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e81ef757f18947ac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.703+0000 7fedf9033700 1 -- 172.21.15.1:0/1055857832 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede4004230 con 0x7fedf415e8e0 2024-03-18T20:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.703+0000 7fedf9033700 1 -- 172.21.15.1:0/1055857832 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede4004390 con 0x7fedf415e8e0 2024-03-18T20:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.703+0000 7fedf9033700 1 -- 172.21.15.1:0/1055857832 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede40036a0 con 0x7fedf415e8e0 2024-03-18T20:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.704+0000 7fedfc299700 1 -- 172.21.15.1:0/1055857832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedf415e8e0 msgr2=0x7fedf40cf380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.704+0000 7fedfc299700 1 --2- 172.21.15.1:0/1055857832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedf415e8e0 0x7fedf40cf380 secure :-1 s=READY pgs=3305 cs=0 l=1 rev1=1 crypto rx=0x7fedf4071e10 tx=0x7fede4009bd0 comp rx=0 tx=0).stop 2024-03-18T20:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.704+0000 7fedfc299700 1 -- 172.21.15.1:0/1055857832 shutdown_connections 2024-03-18T20:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.704+0000 7fedfc299700 1 --2- 172.21.15.1:0/1055857832 >> v2:172.21.15.165:3300/0 conn(0x7fedf4099a30 0x7fedf4099e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.704+0000 7fedfc299700 1 --2- 172.21.15.1:0/1055857832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedf415e8e0 0x7fedf40cf380 unknown :-1 s=CLOSED pgs=3305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.704+0000 7fedfc299700 1 -- 172.21.15.1:0/1055857832 >> 172.21.15.1:0/1055857832 conn(0x7fedf40d1180 msgr2=0x7fedf40d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:18.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.704+0000 7fedfc299700 1 -- 172.21.15.1:0/1055857832 shutdown_connections 2024-03-18T20:28:18.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.704+0000 7fedfc299700 1 -- 172.21.15.1:0/1055857832 wait complete. 2024-03-18T20:28:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.706+0000 7fedfc299700 1 Processor -- start 2024-03-18T20:28:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.706+0000 7fedfc299700 1 -- start start 2024-03-18T20:28:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedfc299700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fedf4099a30 0x7fedf40b5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedfc299700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fedf415ef60 0x7fedf40b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedfc299700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedf40b6900 0x7fedf40cc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedfc299700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf40b59f0 con 0x7fedf40b6900 2024-03-18T20:28:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedfc299700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf40b6d90 con 0x7fedf415ef60 2024-03-18T20:28:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedfc299700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf40cdc60 con 0x7fedf4099a30 2024-03-18T20:28:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedfa836700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedf40b6900 0x7fedf40cc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedfa836700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedf40b6900 0x7fedf40cc7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57356/0 (socket says 172.21.15.1:57356) 2024-03-18T20:28:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedfa836700 1 -- 172.21.15.1:0/3403555312 learned_addr learned my addr 172.21.15.1:0/3403555312 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedfa035700 1 --2- 172.21.15.1:0/3403555312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fedf4099a30 0x7fedf40b5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedf9834700 1 --2- 172.21.15.1:0/3403555312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fedf415ef60 0x7fedf40b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:18.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedfa836700 1 -- 172.21.15.1:0/3403555312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fedf4099a30 msgr2=0x7fedf40b5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedfa836700 1 --2- 172.21.15.1:0/3403555312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fedf4099a30 0x7fedf40b5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedfa836700 1 -- 172.21.15.1:0/3403555312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fedf415ef60 msgr2=0x7fedf40b63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedfa836700 1 --2- 172.21.15.1:0/3403555312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fedf415ef60 0x7fedf40b63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedfa836700 1 -- 172.21.15.1:0/3403555312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede4009550 con 0x7fedf40b6900 2024-03-18T20:28:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedfa836700 1 --2- 172.21.15.1:0/3403555312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedf40b6900 0x7fedf40cc7c0 secure :-1 s=READY pgs=3306 cs=0 l=1 rev1=1 crypto rx=0x7fedec00cc10 tx=0x7fedec00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedeaffd700 1 -- 172.21.15.1:0/3403555312 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec00aa60 con 0x7fedf40b6900 2024-03-18T20:28:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedeaffd700 1 -- 172.21.15.1:0/3403555312 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedec00abc0 con 0x7fedf40b6900 2024-03-18T20:28:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedeaffd700 1 -- 172.21.15.1:0/3403555312 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec012420 con 0x7fedf40b6900 2024-03-18T20:28:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.707+0000 7fedfc299700 1 -- 172.21.15.1:0/3403555312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf40cdf40 con 0x7fedf40b6900 2024-03-18T20:28:18.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.708+0000 7fedfc299700 1 -- 172.21.15.1:0/3403555312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf40b5330 con 0x7fedf40b6900 2024-03-18T20:28:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.708+0000 7fedfc299700 1 -- 172.21.15.1:0/3403555312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf409ada0 con 0x7fedf40b6900 2024-03-18T20:28:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.709+0000 7fedeaffd700 1 -- 172.21.15.1:0/3403555312 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fedec012580 con 0x7fedf40b6900 2024-03-18T20:28:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.709+0000 7fedeaffd700 1 --2- 172.21.15.1:0/3403555312 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fedd806adb0 0x7fedd806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.709+0000 7fedeaffd700 1 -- 172.21.15.1:0/3403555312 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fedec089970 con 0x7fedf40b6900 2024-03-18T20:28:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.709+0000 7fedfa035700 1 --2- 172.21.15.1:0/3403555312 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fedd806adb0 0x7fedd806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:18.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.710+0000 7fedfa035700 1 --2- 172.21.15.1:0/3403555312 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fedd806adb0 0x7fedd806d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fedf0006b20 tx=0x7fedf0008040 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:18.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.714+0000 7fedeaffd700 1 -- 172.21.15.1:0/3403555312 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fedec054890 con 0x7fedf40b6900 2024-03-18T20:28:18.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.950+0000 7fedfc299700 1 -- 172.21.15.1:0/3403555312 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fedf40d0620 con 0x7fedd806adb0 2024-03-18T20:28:18.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.951+0000 7fedeaffd700 1 -- 172.21.15.1:0/3403555312 <== mgr.11833 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fedf00076b0 con 0x7fedd806adb0 2024-03-18T20:28:18.955 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-03-18T20:28:18.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-03-18T20:28:18.955 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephfs" 2024-03-18T20:28:18.956 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-03-18T20:28:18.956 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-03-18T20:28:18.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.953+0000 7fedfc299700 1 -- 172.21.15.1:0/3403555312 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fedd806adb0 msgr2=0x7fedd806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.953+0000 7fedfc299700 1 --2- 172.21.15.1:0/3403555312 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fedd806adb0 0x7fedd806d270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fedf0006b20 tx=0x7fedf0008040 comp rx=0 tx=0).stop 2024-03-18T20:28:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.953+0000 7fedfc299700 1 -- 172.21.15.1:0/3403555312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedf40b6900 msgr2=0x7fedf40cc7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.953+0000 7fedfc299700 1 --2- 172.21.15.1:0/3403555312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedf40b6900 0x7fedf40cc7c0 secure :-1 s=READY pgs=3306 cs=0 l=1 rev1=1 crypto rx=0x7fedec00cc10 tx=0x7fedec00cf20 comp rx=0 tx=0).stop 2024-03-18T20:28:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.953+0000 7fedfc299700 1 -- 172.21.15.1:0/3403555312 shutdown_connections 2024-03-18T20:28:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.953+0000 7fedfc299700 1 --2- 172.21.15.1:0/3403555312 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fedd806adb0 0x7fedd806d270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.953+0000 7fedfc299700 1 --2- 172.21.15.1:0/3403555312 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fedf4099a30 0x7fedf40b5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.953+0000 7fedfc299700 1 --2- 172.21.15.1:0/3403555312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedf40b6900 0x7fedf40cc7c0 unknown :-1 s=CLOSED pgs=3306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.953+0000 7fedfc299700 1 --2- 172.21.15.1:0/3403555312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fedf415ef60 0x7fedf40b63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.953+0000 7fedfc299700 1 -- 172.21.15.1:0/3403555312 >> 172.21.15.1:0/3403555312 conn(0x7fedf40d1180 msgr2=0x7fedf40b0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.953+0000 7fedfc299700 1 -- 172.21.15.1:0/3403555312 shutdown_connections 2024-03-18T20:28:18.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:18.954+0000 7fedfc299700 1 -- 172.21.15.1:0/3403555312 wait complete. 2024-03-18T20:28:18.969 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-03-18T20:28:19.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.093+0000 7f2d01d8b700 1 Processor -- start 2024-03-18T20:28:19.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.093+0000 7f2d01d8b700 1 -- start start 2024-03-18T20:28:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.093+0000 7f2d01d8b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2cfc15d8f0 0x7f2cfc0b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.093+0000 7f2d01d8b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2cfc072050 0x7f2cfc0b4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.093+0000 7f2d01d8b700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cfc0b1960 con 0x7f2cfc0b35e0 2024-03-18T20:28:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.093+0000 7f2d01d8b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cfc0b1bb0 con 0x7f2cfc15d8f0 2024-03-18T20:28:19.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.093+0000 7f2d01d8b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cfc0adb70 con 0x7f2cfc072050 2024-03-18T20:28:19.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.094+0000 7f2cfb7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2cfc15d8f0 0x7f2cfc0b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:19.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.094+0000 7f2cfb7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2cfc15d8f0 0x7f2cfc0b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57364/0 (socket says 172.21.15.1:57364) 2024-03-18T20:28:19.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.094+0000 7f2cfb7fe700 1 -- 172.21.15.1:0/3995178682 learned_addr learned my addr 172.21.15.1:0/3995178682 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:19.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.094+0000 7f2cfb7fe700 1 -- 172.21.15.1:0/3995178682 >> v2:172.21.15.165:3300/0 conn(0x7f2cfc072050 msgr2=0x7f2cfc0b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:19.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.094+0000 7f2cfb7fe700 1 --2- 172.21.15.1:0/3995178682 >> v2:172.21.15.165:3300/0 conn(0x7f2cfc072050 0x7f2cfc0b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:19.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.094+0000 7f2cfb7fe700 1 -- 172.21.15.1:0/3995178682 >> v1:172.21.15.165:6789/0 conn(0x7f2cfc0b35e0 legacy=0x7f2cfc06a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:19.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.094+0000 7f2cfb7fe700 1 -- 172.21.15.1:0/3995178682 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc0ade50 con 0x7f2cfc15d8f0 2024-03-18T20:28:19.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.094+0000 7f2cfb7fe700 1 --2- 172.21.15.1:0/3995178682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cfc15d8f0 0x7f2cfc0b3de0 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc0b1e90 tx=0x7f2cec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbf5bdcd952c158a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:19.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.094+0000 7f2cfa7fc700 1 -- 172.21.15.1:0/3995178682 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec004230 con 0x7f2cfc15d8f0 2024-03-18T20:28:19.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.094+0000 7f2cfa7fc700 1 -- 172.21.15.1:0/3995178682 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cec004390 con 0x7f2cfc15d8f0 2024-03-18T20:28:19.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.094+0000 7f2cfa7fc700 1 -- 172.21.15.1:0/3995178682 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec0036a0 con 0x7f2cfc15d8f0 2024-03-18T20:28:19.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.095+0000 7f2d01d8b700 1 -- 172.21.15.1:0/3995178682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cfc15d8f0 msgr2=0x7f2cfc0b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:19.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.095+0000 7f2d01d8b700 1 --2- 172.21.15.1:0/3995178682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cfc15d8f0 0x7f2cfc0b3de0 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc0b1e90 tx=0x7f2cec009bd0 comp rx=0 tx=0).stop 2024-03-18T20:28:19.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.095+0000 7f2d01d8b700 1 -- 172.21.15.1:0/3995178682 shutdown_connections 2024-03-18T20:28:19.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.095+0000 7f2d01d8b700 1 --2- 172.21.15.1:0/3995178682 >> v2:172.21.15.165:3300/0 conn(0x7f2cfc072050 0x7f2cfc0b4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:19.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.095+0000 7f2d01d8b700 1 --2- 172.21.15.1:0/3995178682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cfc15d8f0 0x7f2cfc0b3de0 unknown :-1 s=CLOSED pgs=3307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:19.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.095+0000 7f2d01d8b700 1 -- 172.21.15.1:0/3995178682 >> 172.21.15.1:0/3995178682 conn(0x7f2cfc0b5a10 msgr2=0x7f2cfc0b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:19.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.095+0000 7f2d01d8b700 1 -- 172.21.15.1:0/3995178682 shutdown_connections 2024-03-18T20:28:19.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.095+0000 7f2d01d8b700 1 -- 172.21.15.1:0/3995178682 wait complete. 2024-03-18T20:28:19.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.096+0000 7f2d01d8b700 1 Processor -- start 2024-03-18T20:28:19.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.096+0000 7f2d01d8b700 1 -- start start 2024-03-18T20:28:19.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.097+0000 7f2d01d8b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2cfc072050 0x7f2cfc14b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.097+0000 7f2d01d8b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cfc15d8f0 0x7f2cfc15c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.097+0000 7f2d01d8b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cfc15cdd0 0x7f2cfc150d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.097+0000 7f2d01d8b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc1e5f60 con 0x7f2cfc15cdd0 2024-03-18T20:28:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.097+0000 7f2d01d8b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc1e6240 con 0x7f2cfc15d8f0 2024-03-18T20:28:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.097+0000 7f2d01d8b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc1e6520 con 0x7f2cfc072050 2024-03-18T20:28:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.097+0000 7f2cfbfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cfc15cdd0 0x7f2cfc150d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.097+0000 7f2cfbfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cfc15cdd0 0x7f2cfc150d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57374/0 (socket says 172.21.15.1:57374) 2024-03-18T20:28:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.097+0000 7f2cfbfff700 1 -- 172.21.15.1:0/2565159808 learned_addr learned my addr 172.21.15.1:0/2565159808 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.097+0000 7f2cfaffd700 1 --2- 172.21.15.1:0/2565159808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cfc15d8f0 0x7f2cfc15c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.097+0000 7f2cfb7fe700 1 --2- 172.21.15.1:0/2565159808 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2cfc072050 0x7f2cfc14b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.097+0000 7f2cfbfff700 1 -- 172.21.15.1:0/2565159808 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2cfc072050 msgr2=0x7f2cfc14b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.097+0000 7f2cfbfff700 1 --2- 172.21.15.1:0/2565159808 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2cfc072050 0x7f2cfc14b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.097+0000 7f2cfbfff700 1 -- 172.21.15.1:0/2565159808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cfc15d8f0 msgr2=0x7f2cfc15c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.097+0000 7f2cfbfff700 1 --2- 172.21.15.1:0/2565159808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cfc15d8f0 0x7f2cfc15c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:19.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.097+0000 7f2cfbfff700 1 -- 172.21.15.1:0/2565159808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cec009550 con 0x7f2cfc15cdd0 2024-03-18T20:28:19.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.097+0000 7f2cfbfff700 1 --2- 172.21.15.1:0/2565159808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cfc15cdd0 0x7f2cfc150d50 secure :-1 s=READY pgs=3308 cs=0 l=1 rev1=1 crypto rx=0x7f2cf400b9e0 tx=0x7f2cf400bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:19.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.098+0000 7f2ce3fff700 1 -- 172.21.15.1:0/2565159808 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf400d8e0 con 0x7f2cfc15cdd0 2024-03-18T20:28:19.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.098+0000 7f2ce3fff700 1 -- 172.21.15.1:0/2565159808 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf400da60 con 0x7f2cfc15cdd0 2024-03-18T20:28:19.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.098+0000 7f2d01d8b700 1 -- 172.21.15.1:0/2565159808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cfc1e6800 con 0x7f2cfc15cdd0 2024-03-18T20:28:19.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.098+0000 7f2ce3fff700 1 -- 172.21.15.1:0/2565159808 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf400de10 con 0x7f2cfc15cdd0 2024-03-18T20:28:19.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.098+0000 7f2d01d8b700 1 -- 172.21.15.1:0/2565159808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cfc1e6dc0 con 0x7f2cfc15cdd0 2024-03-18T20:28:19.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.099+0000 7f2d01d8b700 1 -- 172.21.15.1:0/2565159808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cfc067dc0 con 0x7f2cfc15cdd0 2024-03-18T20:28:19.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.099+0000 7f2ce3fff700 1 -- 172.21.15.1:0/2565159808 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 167) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2cf4015420 con 0x7f2cfc15cdd0 2024-03-18T20:28:19.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.099+0000 7f2ce3fff700 1 --2- 172.21.15.1:0/2565159808 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2ce806adb0 0x7f2ce806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:19.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.100+0000 7f2ce3fff700 1 -- 172.21.15.1:0/2565159808 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f2cf4011070 con 0x7f2cfc15cdd0 2024-03-18T20:28:19.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.100+0000 7f2cfb7fe700 1 --2- 172.21.15.1:0/2565159808 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2ce806adb0 0x7f2ce806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:19.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.100+0000 7f2cfb7fe700 1 --2- 172.21.15.1:0/2565159808 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2ce806adb0 0x7f2ce806d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2cec009130 tx=0x7f2cec008030 comp rx=0 tx=0).ready entity=mgr.11833 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:19.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.104+0000 7f2ce3fff700 1 -- 172.21.15.1:0/2565159808 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2cf404f580 con 0x7f2cfc15cdd0 2024-03-18T20:28:19.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:19.368+0000 7f2d01d8b700 1 -- 172.21.15.1:0/2565159808 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f2cfc0b3bc0 con 0x7f2cfc15cdd0 2024-03-18T20:28:20.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.225+0000 7f2ce3fff700 1 -- 172.21.15.1:0/2565159808 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 168) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f2cf405e0c0 con 0x7f2cfc15cdd0 2024-03-18T20:28:20.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.227+0000 7f2ce3fff700 1 -- 172.21.15.1:0/2565159808 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v168) v1 ==== 92+0+0 (secure 0 0 0) 0x7f2cf40727b0 con 0x7f2cfc15cdd0 2024-03-18T20:28:20.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.230+0000 7f2d01d8b700 1 -- 172.21.15.1:0/2565159808 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2ce806adb0 msgr2=0x7f2ce806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:20.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.230+0000 7f2d01d8b700 1 --2- 172.21.15.1:0/2565159808 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2ce806adb0 0x7f2ce806d270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2cec009130 tx=0x7f2cec008030 comp rx=0 tx=0).stop 2024-03-18T20:28:20.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.230+0000 7f2d01d8b700 1 -- 172.21.15.1:0/2565159808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cfc15cdd0 msgr2=0x7f2cfc150d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:20.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.230+0000 7f2d01d8b700 1 --2- 172.21.15.1:0/2565159808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cfc15cdd0 0x7f2cfc150d50 secure :-1 s=READY pgs=3308 cs=0 l=1 rev1=1 crypto rx=0x7f2cf400b9e0 tx=0x7f2cf400bcf0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.230+0000 7f2d01d8b700 1 -- 172.21.15.1:0/2565159808 shutdown_connections 2024-03-18T20:28:20.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.230+0000 7f2d01d8b700 1 --2- 172.21.15.1:0/2565159808 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f2ce806adb0 0x7f2ce806d270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.230+0000 7f2d01d8b700 1 --2- 172.21.15.1:0/2565159808 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2cfc072050 0x7f2cfc14b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.230+0000 7f2d01d8b700 1 --2- 172.21.15.1:0/2565159808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cfc15cdd0 0x7f2cfc150d50 unknown :-1 s=CLOSED pgs=3308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.230+0000 7f2d01d8b700 1 --2- 172.21.15.1:0/2565159808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cfc15d8f0 0x7f2cfc15c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.230+0000 7f2d01d8b700 1 -- 172.21.15.1:0/2565159808 >> 172.21.15.1:0/2565159808 conn(0x7f2cfc0b5a10 msgr2=0x7f2cfc0b0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:20.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.230+0000 7f2d01d8b700 1 -- 172.21.15.1:0/2565159808 shutdown_connections 2024-03-18T20:28:20.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.231+0000 7f2d01d8b700 1 -- 172.21.15.1:0/2565159808 wait complete. 2024-03-18T20:28:20.251 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-03-18T20:28:20.373 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:20.369+0000 7f493cd31240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:28:20.388 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:20.384+0000 7f5a40fa9240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:28:20.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.385+0000 7f9cdcb20700 1 Processor -- start 2024-03-18T20:28:20.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.386+0000 7f9cdcb20700 1 -- start start 2024-03-18T20:28:20.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.387+0000 7f9cdcb20700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9cd80b35e0 0x7f9cd80ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:20.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.387+0000 7f9cdcb20700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cd8072050 0x7f9cd80ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:20.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.387+0000 7f9cdcb20700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd80b1960 con 0x7f9cd815d8f0 2024-03-18T20:28:20.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.387+0000 7f9cdcb20700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cd80b1bb0 con 0x7f9cd80b35e0 2024-03-18T20:28:20.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.387+0000 7f9cdcb20700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd80adb70 con 0x7f9cd8072050 2024-03-18T20:28:20.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.387+0000 7f9cd5d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9cd80b35e0 0x7f9cd80ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:20.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.387+0000 7f9cd5d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9cd80b35e0 0x7f9cd80ae370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57410/0 (socket says 172.21.15.1:57410) 2024-03-18T20:28:20.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.387+0000 7f9cd5d9b700 1 -- 172.21.15.1:0/2605257915 learned_addr learned my addr 172.21.15.1:0/2605257915 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:20.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.387+0000 7f9cd6d9d700 1 --2- 172.21.15.1:0/2605257915 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cd8072050 0x7f9cd80ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:20.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.387+0000 7f9cd5d9b700 1 -- 172.21.15.1:0/2605257915 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cd8072050 msgr2=0x7f9cd80ae8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:20.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.387+0000 7f9cd5d9b700 1 --2- 172.21.15.1:0/2605257915 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cd8072050 0x7f9cd80ae8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.387+0000 7f9cd5d9b700 1 -- 172.21.15.1:0/2605257915 >> v1:172.21.15.165:6789/0 conn(0x7f9cd815d8f0 legacy=0x7f9cd80b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:20.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.387+0000 7f9cd5d9b700 1 -- 172.21.15.1:0/2605257915 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd80ade50 con 0x7f9cd80b35e0 2024-03-18T20:28:20.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.387+0000 7f9cd6d9d700 1 --2- 172.21.15.1:0/2605257915 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cd8072050 0x7f9cd80ae8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:28:20.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.387+0000 7f9cd5d9b700 1 --2- 172.21.15.1:0/2605257915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cd80b35e0 0x7f9cd80ae370 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc0098f0 tx=0x7f9ccc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43d284a3d4bb4ded server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:20.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.387+0000 7f9cd559a700 1 -- 172.21.15.1:0/2605257915 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc00b8a0 con 0x7f9cd80b35e0 2024-03-18T20:28:20.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.387+0000 7f9cd559a700 1 -- 172.21.15.1:0/2605257915 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ccc00ba00 con 0x7f9cd80b35e0 2024-03-18T20:28:20.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.387+0000 7f9cd559a700 1 -- 172.21.15.1:0/2605257915 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc00bd90 con 0x7f9cd80b35e0 2024-03-18T20:28:20.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.388+0000 7f9cdcb20700 1 -- 172.21.15.1:0/2605257915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cd80b35e0 msgr2=0x7f9cd80ae370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:20.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.388+0000 7f9cdcb20700 1 --2- 172.21.15.1:0/2605257915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cd80b35e0 0x7f9cd80ae370 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc0098f0 tx=0x7f9ccc009c00 comp rx=0 tx=0).stop 2024-03-18T20:28:20.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.388+0000 7f9cdcb20700 1 -- 172.21.15.1:0/2605257915 shutdown_connections 2024-03-18T20:28:20.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.388+0000 7f9cdcb20700 1 --2- 172.21.15.1:0/2605257915 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cd8072050 0x7f9cd80ae8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.388+0000 7f9cdcb20700 1 --2- 172.21.15.1:0/2605257915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cd80b35e0 0x7f9cd80ae370 unknown :-1 s=CLOSED pgs=3313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.388+0000 7f9cdcb20700 1 -- 172.21.15.1:0/2605257915 >> 172.21.15.1:0/2605257915 conn(0x7f9cd80b5a10 msgr2=0x7f9cd80b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:20.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.388+0000 7f9cdcb20700 1 -- 172.21.15.1:0/2605257915 shutdown_connections 2024-03-18T20:28:20.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.388+0000 7f9cdcb20700 1 -- 172.21.15.1:0/2605257915 wait complete. 2024-03-18T20:28:20.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.389+0000 7f9cdcb20700 1 Processor -- start 2024-03-18T20:28:20.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.390+0000 7f9cdcb20700 1 -- start start 2024-03-18T20:28:20.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.390+0000 7f9cdcb20700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9cd8072050 0x7f9cd8157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:20.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.390+0000 7f9cdcb20700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cd815d8f0 0x7f9cd814c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:20.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.390+0000 7f9cdcb20700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cd814cad0 0x7f9cd8159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:20.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.390+0000 7f9cdcb20700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd815dcd0 con 0x7f9cd814cad0 2024-03-18T20:28:20.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.390+0000 7f9cdcb20700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd8157ee0 con 0x7f9cd8072050 2024-03-18T20:28:20.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.390+0000 7f9cdcb20700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd80ae310 con 0x7f9cd815d8f0 2024-03-18T20:28:20.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.390+0000 7f9cd6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cd814cad0 0x7f9cd8159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:20.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.390+0000 7f9cd6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cd814cad0 0x7f9cd8159840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57422/0 (socket says 172.21.15.1:57422) 2024-03-18T20:28:20.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.390+0000 7f9cd6d9d700 1 -- 172.21.15.1:0/3428376429 learned_addr learned my addr 172.21.15.1:0/3428376429 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:20.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.390+0000 7f9cd5d9b700 1 --2- 172.21.15.1:0/3428376429 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cd815d8f0 0x7f9cd814c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:20.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.390+0000 7f9cd659c700 1 --2- 172.21.15.1:0/3428376429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9cd8072050 0x7f9cd8157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:20.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.391+0000 7f9cd6d9d700 1 -- 172.21.15.1:0/3428376429 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cd815d8f0 msgr2=0x7f9cd814c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:20.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.391+0000 7f9cd6d9d700 1 --2- 172.21.15.1:0/3428376429 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cd815d8f0 0x7f9cd814c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.391+0000 7f9cd6d9d700 1 -- 172.21.15.1:0/3428376429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9cd8072050 msgr2=0x7f9cd8157600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.391+0000 7f9cd6d9d700 1 --2- 172.21.15.1:0/3428376429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9cd8072050 0x7f9cd8157600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.391+0000 7f9cd6d9d700 1 -- 172.21.15.1:0/3428376429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ccc009550 con 0x7f9cd814cad0 2024-03-18T20:28:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.391+0000 7f9cd6d9d700 1 --2- 172.21.15.1:0/3428376429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cd814cad0 0x7f9cd8159840 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0007e30 tx=0x7f9cd000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.391+0000 7f9cc6ffd700 1 -- 172.21.15.1:0/3428376429 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd000dd20 con 0x7f9cd814cad0 2024-03-18T20:28:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.391+0000 7f9cc6ffd700 1 -- 172.21.15.1:0/3428376429 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd000f040 con 0x7f9cd814cad0 2024-03-18T20:28:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.391+0000 7f9cdcb20700 1 -- 172.21.15.1:0/3428376429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd80ae5f0 con 0x7f9cd814cad0 2024-03-18T20:28:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.391+0000 7f9cc6ffd700 1 -- 172.21.15.1:0/3428376429 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0003c60 con 0x7f9cd814cad0 2024-03-18T20:28:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.391+0000 7f9cdcb20700 1 -- 172.21.15.1:0/3428376429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd81581c0 con 0x7f9cd814cad0 2024-03-18T20:28:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.392+0000 7f9cdcb20700 1 -- 172.21.15.1:0/3428376429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd80aeb10 con 0x7f9cd814cad0 2024-03-18T20:28:20.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.393+0000 7f9cc6ffd700 1 -- 172.21.15.1:0/3428376429 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 168) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f9cd0004030 con 0x7f9cd814cad0 2024-03-18T20:28:20.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.393+0000 7f9cc6ffd700 1 --2- 172.21.15.1:0/3428376429 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9cb806ae00 0x7f9cb806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:20.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.393+0000 7f9cd659c700 1 -- 172.21.15.1:0/3428376429 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9cb806ae00 msgr2=0x7f9cb806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:28:20.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.393+0000 7f9cc6ffd700 1 -- 172.21.15.1:0/3428376429 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f9cd0089780 con 0x7f9cd814cad0 2024-03-18T20:28:20.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.393+0000 7f9cd659c700 1 --2- 172.21.15.1:0/3428376429 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9cb806ae00 0x7f9cb806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:28:20.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.398+0000 7f9cc6ffd700 1 -- 172.21.15.1:0/3428376429 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9cd0054620 con 0x7f9cd814cad0 2024-03-18T20:28:20.521 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:20.516+0000 7f493cd31240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:28:20.557 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:20.553+0000 7f5a40fa9240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:28:20.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.594+0000 7f9cd659c700 1 -- 172.21.15.1:0/3428376429 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9cb806ae00 msgr2=0x7f9cb806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:28:20.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.594+0000 7f9cd659c700 1 --2- 172.21.15.1:0/3428376429 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9cb806ae00 0x7f9cb806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:28:20.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.648+0000 7f9cdcb20700 1 -- 172.21.15.1:0/3428376429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f9cd8067dc0 con 0x7f9cd814cad0 2024-03-18T20:28:20.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.652+0000 7f9cc6ffd700 1 -- 172.21.15.1:0/3428376429 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v95)=0 v95) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9cd0057c40 con 0x7f9cd814cad0 2024-03-18T20:28:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.654+0000 7f9cdcb20700 1 -- 172.21.15.1:0/3428376429 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9cb806ae00 msgr2=0x7f9cb806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:28:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.654+0000 7f9cdcb20700 1 --2- 172.21.15.1:0/3428376429 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9cb806ae00 0x7f9cb806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.654+0000 7f9cdcb20700 1 -- 172.21.15.1:0/3428376429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cd814cad0 msgr2=0x7f9cd8159840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.654+0000 7f9cdcb20700 1 --2- 172.21.15.1:0/3428376429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cd814cad0 0x7f9cd8159840 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0007e30 tx=0x7f9cd000d3b0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.654+0000 7f9cdcb20700 1 -- 172.21.15.1:0/3428376429 shutdown_connections 2024-03-18T20:28:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.654+0000 7f9cdcb20700 1 --2- 172.21.15.1:0/3428376429 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9cb806ae00 0x7f9cb806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.654+0000 7f9cdcb20700 1 --2- 172.21.15.1:0/3428376429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cd814cad0 0x7f9cd8159840 unknown :-1 s=CLOSED pgs=3314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.654+0000 7f9cdcb20700 1 --2- 172.21.15.1:0/3428376429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9cd8072050 0x7f9cd8157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.654+0000 7f9cdcb20700 1 --2- 172.21.15.1:0/3428376429 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9cd815d8f0 0x7f9cd814c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.654+0000 7f9cdcb20700 1 -- 172.21.15.1:0/3428376429 >> 172.21.15.1:0/3428376429 conn(0x7f9cd80b5a10 msgr2=0x7f9cd80b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.654+0000 7f9cdcb20700 1 -- 172.21.15.1:0/3428376429 shutdown_connections 2024-03-18T20:28:20.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.654+0000 7f9cdcb20700 1 -- 172.21.15.1:0/3428376429 wait complete. 2024-03-18T20:28:20.671 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-03-18T20:28:20.787 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:20.783+0000 7f493cd31240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:28:20.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.796+0000 7f504cea7700 1 Processor -- start 2024-03-18T20:28:20.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.796+0000 7f504cea7700 1 -- start start 2024-03-18T20:28:20.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.797+0000 7f504cea7700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f50480cf200 0x7f504815c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:20.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.797+0000 7f504cea7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f504815b6c0 0x7f50480cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:20.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.797+0000 7f504cea7700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504815ca40 con 0x7f50480cbc90 2024-03-18T20:28:20.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.797+0000 7f504cea7700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50480cce30 con 0x7f50480cf200 2024-03-18T20:28:20.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.797+0000 7f504cea7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50480cfb20 con 0x7f504815b6c0 2024-03-18T20:28:20.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.797+0000 7f504659c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f50480cf200 0x7f504815c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:20.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.797+0000 7f5046d9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f50480cbc90 0x7f504815eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40742/0 (socket says 172.21.15.1:40742) 2024-03-18T20:28:20.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.797+0000 7f5046d9d700 1 -- 172.21.15.1:0/2218825012 learned_addr learned my addr 172.21.15.1:0/2218825012 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:20.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.797+0000 7f5045d9b700 1 --2- 172.21.15.1:0/2218825012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f504815b6c0 0x7f50480cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:20.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.798+0000 7f504659c700 1 -- 172.21.15.1:0/2218825012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f504815b6c0 msgr2=0x7f50480cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:20.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.798+0000 7f504659c700 1 --2- 172.21.15.1:0/2218825012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f504815b6c0 0x7f50480cf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.798+0000 7f504659c700 1 -- 172.21.15.1:0/2218825012 >> v1:172.21.15.1:6789/0 conn(0x7f50480cbc90 legacy=0x7f504815eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:20.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.798+0000 7f504659c700 1 -- 172.21.15.1:0/2218825012 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50480cc220 con 0x7f50480cf200 2024-03-18T20:28:20.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.798+0000 7f504559a700 1 -- 172.21.15.1:0/2218825012 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577104796 0 0) 0x7f5040001490 con 0x7f50480cbc90 2024-03-18T20:28:20.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.798+0000 7f504659c700 1 --2- 172.21.15.1:0/2218825012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50480cf200 0x7f504815c500 secure :-1 s=READY pgs=3316 cs=0 l=1 rev1=1 crypto rx=0x7f5038009840 tx=0x7f5038009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=573e59f2200e360a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:20.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.798+0000 7f504559a700 1 -- 172.21.15.1:0/2218825012 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5038011830 con 0x7f50480cf200 2024-03-18T20:28:20.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.798+0000 7f504559a700 1 -- 172.21.15.1:0/2218825012 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5038011990 con 0x7f50480cf200 2024-03-18T20:28:20.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.798+0000 7f504559a700 1 -- 172.21.15.1:0/2218825012 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5038011d20 con 0x7f50480cf200 2024-03-18T20:28:20.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.799+0000 7f504cea7700 1 -- 172.21.15.1:0/2218825012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50480cf200 msgr2=0x7f504815c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:20.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.799+0000 7f504cea7700 1 --2- 172.21.15.1:0/2218825012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50480cf200 0x7f504815c500 secure :-1 s=READY pgs=3316 cs=0 l=1 rev1=1 crypto rx=0x7f5038009840 tx=0x7f5038009b50 comp rx=0 tx=0).stop 2024-03-18T20:28:20.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.799+0000 7f504cea7700 1 -- 172.21.15.1:0/2218825012 shutdown_connections 2024-03-18T20:28:20.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.799+0000 7f504cea7700 1 --2- 172.21.15.1:0/2218825012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50480cf200 0x7f504815c500 unknown :-1 s=CLOSED pgs=3316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.799+0000 7f504cea7700 1 --2- 172.21.15.1:0/2218825012 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f504815b6c0 0x7f50480cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.799+0000 7f504cea7700 1 -- 172.21.15.1:0/2218825012 >> 172.21.15.1:0/2218825012 conn(0x7f50480d1180 msgr2=0x7f50480d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:20.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.799+0000 7f504cea7700 1 -- 172.21.15.1:0/2218825012 shutdown_connections 2024-03-18T20:28:20.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.799+0000 7f504cea7700 1 -- 172.21.15.1:0/2218825012 wait complete. 2024-03-18T20:28:20.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.800+0000 7f504cea7700 1 Processor -- start 2024-03-18T20:28:20.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.800+0000 7f504cea7700 1 -- start start 2024-03-18T20:28:20.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.801+0000 7f504cea7700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50480cbc90 0x7f50481528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:20.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.801+0000 7f504cea7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50480cf200 0x7f5048099ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:20.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.801+0000 7f504cea7700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f504815b6c0 0x7f504809a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:20.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.801+0000 7f504cea7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50481f4840 con 0x7f50480cf200 2024-03-18T20:28:20.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.801+0000 7f504cea7700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504806c840 con 0x7f504815b6c0 2024-03-18T20:28:20.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.801+0000 7f504cea7700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5048152e00 con 0x7f50480cbc90 2024-03-18T20:28:20.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.801+0000 7f5045d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50480cf200 0x7f5048099ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:20.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.801+0000 7f5045d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50480cf200 0x7f5048099ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57448/0 (socket says 172.21.15.1:57448) 2024-03-18T20:28:20.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.801+0000 7f5045d9b700 1 -- 172.21.15.1:0/3908082016 learned_addr learned my addr 172.21.15.1:0/3908082016 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:20.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.801+0000 7f504659c700 1 --2- 172.21.15.1:0/3908082016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50480cbc90 0x7f50481528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:20.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.801+0000 7f5046d9d700 1 --2- 172.21.15.1:0/3908082016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f504815b6c0 0x7f504809a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:20.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.801+0000 7f5045d9b700 1 -- 172.21.15.1:0/3908082016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50480cbc90 msgr2=0x7f50481528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:20.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.801+0000 7f5045d9b700 1 --2- 172.21.15.1:0/3908082016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50480cbc90 0x7f50481528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.801+0000 7f5045d9b700 1 -- 172.21.15.1:0/3908082016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f504815b6c0 msgr2=0x7f504809a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:20.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.801+0000 7f5045d9b700 1 --2- 172.21.15.1:0/3908082016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f504815b6c0 0x7f504809a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:20.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.801+0000 7f5045d9b700 1 -- 172.21.15.1:0/3908082016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50380094a0 con 0x7f50480cf200 2024-03-18T20:28:20.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.802+0000 7f5045d9b700 1 --2- 172.21.15.1:0/3908082016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50480cf200 0x7f5048099ac0 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7f503c00d830 tx=0x7f503c00db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:20.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.802+0000 7f5036ffd700 1 -- 172.21.15.1:0/3908082016 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c0125f0 con 0x7f50480cf200 2024-03-18T20:28:20.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.802+0000 7f5036ffd700 1 -- 172.21.15.1:0/3908082016 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503c012750 con 0x7f50480cf200 2024-03-18T20:28:20.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.802+0000 7f504cea7700 1 -- 172.21.15.1:0/3908082016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f504806ba80 con 0x7f50480cf200 2024-03-18T20:28:20.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.802+0000 7f5036ffd700 1 -- 172.21.15.1:0/3908082016 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c012ae0 con 0x7f50480cf200 2024-03-18T20:28:20.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.802+0000 7f504cea7700 1 -- 172.21.15.1:0/3908082016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f504806c040 con 0x7f50480cf200 2024-03-18T20:28:20.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.803+0000 7f504cea7700 1 -- 172.21.15.1:0/3908082016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50480d0ca0 con 0x7f50480cf200 2024-03-18T20:28:20.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.803+0000 7f5036ffd700 1 -- 172.21.15.1:0/3908082016 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 168) v1 ==== 89256+0+0 (secure 0 0 0) 0x7f503c012c40 con 0x7f50480cf200 2024-03-18T20:28:20.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.804+0000 7f5036ffd700 1 --2- 172.21.15.1:0/3908082016 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f502806ae00 0x7f502806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:20.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.804+0000 7f504659c700 1 -- 172.21.15.1:0/3908082016 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f502806ae00 msgr2=0x7f502806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:28:20.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.804+0000 7f5036ffd700 1 -- 172.21.15.1:0/3908082016 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f503c089730 con 0x7f50480cf200 2024-03-18T20:28:20.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.804+0000 7f504659c700 1 --2- 172.21.15.1:0/3908082016 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f502806ae00 0x7f502806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:28:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:20.812+0000 7f5036ffd700 1 -- 172.21.15.1:0/3908082016 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f503c054650 con 0x7f50480cf200 2024-03-18T20:28:20.847 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:20.843+0000 7f5a40fa9240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:28:21.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.005+0000 7f504659c700 1 -- 172.21.15.1:0/3908082016 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f502806ae00 msgr2=0x7f502806d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:28:21.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.005+0000 7f504659c700 1 --2- 172.21.15.1:0/3908082016 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f502806ae00 0x7f502806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:28:21.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.016+0000 7f504cea7700 1 -- 172.21.15.1:0/3908082016 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f5048067dc0 con 0x7f50480cf200 2024-03-18T20:28:21.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.020+0000 7f5036ffd700 1 -- 172.21.15.1:0/3908082016 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v96)=0 v96) v1 ==== 133+0+0 (secure 0 0 0) 0x7f503c057c70 con 0x7f50480cf200 2024-03-18T20:28:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.022+0000 7f504cea7700 1 -- 172.21.15.1:0/3908082016 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f502806ae00 msgr2=0x7f502806d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:28:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.022+0000 7f504cea7700 1 --2- 172.21.15.1:0/3908082016 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f502806ae00 0x7f502806d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.022+0000 7f504cea7700 1 -- 172.21.15.1:0/3908082016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50480cf200 msgr2=0x7f5048099ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:21.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.022+0000 7f504cea7700 1 --2- 172.21.15.1:0/3908082016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50480cf200 0x7f5048099ac0 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7f503c00d830 tx=0x7f503c00db40 comp rx=0 tx=0).stop 2024-03-18T20:28:21.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.022+0000 7f504cea7700 1 -- 172.21.15.1:0/3908082016 shutdown_connections 2024-03-18T20:28:21.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.022+0000 7f504cea7700 1 --2- 172.21.15.1:0/3908082016 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f502806ae00 0x7f502806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:21.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.022+0000 7f504cea7700 1 --2- 172.21.15.1:0/3908082016 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f50480cbc90 0x7f50481528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:21.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.022+0000 7f504cea7700 1 --2- 172.21.15.1:0/3908082016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f504815b6c0 0x7f504809a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:21.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.022+0000 7f504cea7700 1 --2- 172.21.15.1:0/3908082016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50480cf200 0x7f5048099ac0 unknown :-1 s=CLOSED pgs=3317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:21.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.022+0000 7f504cea7700 1 -- 172.21.15.1:0/3908082016 >> 172.21.15.1:0/3908082016 conn(0x7f50480d1180 msgr2=0x7f50480b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:21.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.022+0000 7f504cea7700 1 -- 172.21.15.1:0/3908082016 shutdown_connections 2024-03-18T20:28:21.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.022+0000 7f504cea7700 1 -- 172.21.15.1:0/3908082016 wait complete. 2024-03-18T20:28:21.039 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv01 2024-03-18T20:28:21.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.166+0000 7fcdb222d700 1 Processor -- start 2024-03-18T20:28:21.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.166+0000 7fcdb222d700 1 -- start start 2024-03-18T20:28:21.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.167+0000 7fcdb222d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdac160c00 0x7fcdac0b3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:21.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.167+0000 7fcdb222d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcdac0b4280 0x7fcdac0ad870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:21.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.167+0000 7fcdb222d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdac0b37a0 con 0x7fcdac160fe0 2024-03-18T20:28:21.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.167+0000 7fcdb222d700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdac0b0920 con 0x7fcdac0b4280 2024-03-18T20:28:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.167+0000 7fcdb222d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac0b0c00 con 0x7fcdac160c00 2024-03-18T20:28:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.167+0000 7fcdaaffd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcdac160fe0 0x7fcdac0b3b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40754/0 (socket says 172.21.15.1:40754) 2024-03-18T20:28:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.167+0000 7fcdaaffd700 1 -- 172.21.15.1:0/2203037447 learned_addr learned my addr 172.21.15.1:0/2203037447 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.168+0000 7fcdabfff700 1 --2- 172.21.15.1:0/2203037447 >> v2:172.21.15.165:3300/0 conn(0x7fcdac0b4280 0x7fcdac0ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.168+0000 7fcdab7fe700 1 --2- 172.21.15.1:0/2203037447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdac160c00 0x7fcdac0b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.168+0000 7fcdaa7fc700 1 -- 172.21.15.1:0/2203037447 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197555894 0 0) 0x7fcdac0b37a0 con 0x7fcdac160fe0 2024-03-18T20:28:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.168+0000 7fcdaa7fc700 1 -- 172.21.15.1:0/2203037447 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd980035b0 con 0x7fcdac160fe0 2024-03-18T20:28:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.168+0000 7fcdab7fe700 1 -- 172.21.15.1:0/2203037447 >> v2:172.21.15.165:3300/0 conn(0x7fcdac0b4280 msgr2=0x7fcdac0ad870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.168+0000 7fcdab7fe700 1 --2- 172.21.15.1:0/2203037447 >> v2:172.21.15.165:3300/0 conn(0x7fcdac0b4280 0x7fcdac0ad870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.168+0000 7fcdab7fe700 1 -- 172.21.15.1:0/2203037447 >> v1:172.21.15.1:6789/0 conn(0x7fcdac160fe0 legacy=0x7fcdac0b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.168+0000 7fcdab7fe700 1 -- 172.21.15.1:0/2203037447 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac0b15d0 con 0x7fcdac160c00 2024-03-18T20:28:21.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.168+0000 7fcdabfff700 1 --2- 172.21.15.1:0/2203037447 >> v2:172.21.15.165:3300/0 conn(0x7fcdac0b4280 0x7fcdac0ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:28:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.168+0000 7fcdab7fe700 1 --2- 172.21.15.1:0/2203037447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdac160c00 0x7fcdac0b3260 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c009d90 tx=0x7fcd9c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a00734f5f23b1dd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.168+0000 7fcdaa7fc700 1 -- 172.21.15.1:0/2203037447 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c00e070 con 0x7fcdac160c00 2024-03-18T20:28:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.168+0000 7fcdaa7fc700 1 -- 172.21.15.1:0/2203037447 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9c00bb20 con 0x7fcdac160c00 2024-03-18T20:28:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.168+0000 7fcdaa7fc700 1 -- 172.21.15.1:0/2203037447 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c012430 con 0x7fcdac160c00 2024-03-18T20:28:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.169+0000 7fcdb222d700 1 -- 172.21.15.1:0/2203037447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdac160c00 msgr2=0x7fcdac0b3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.169+0000 7fcdb222d700 1 --2- 172.21.15.1:0/2203037447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdac160c00 0x7fcdac0b3260 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c009d90 tx=0x7fcd9c00b330 comp rx=0 tx=0).stop 2024-03-18T20:28:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.169+0000 7fcdb222d700 1 -- 172.21.15.1:0/2203037447 shutdown_connections 2024-03-18T20:28:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.169+0000 7fcdb222d700 1 --2- 172.21.15.1:0/2203037447 >> v2:172.21.15.165:3300/0 conn(0x7fcdac0b4280 0x7fcdac0ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.169+0000 7fcdb222d700 1 --2- 172.21.15.1:0/2203037447 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdac160c00 0x7fcdac0b3260 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.169+0000 7fcdb222d700 1 -- 172.21.15.1:0/2203037447 >> 172.21.15.1:0/2203037447 conn(0x7fcdac0b5780 msgr2=0x7fcdac0b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.169+0000 7fcdb222d700 1 -- 172.21.15.1:0/2203037447 shutdown_connections 2024-03-18T20:28:21.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.169+0000 7fcdb222d700 1 -- 172.21.15.1:0/2203037447 wait complete. 2024-03-18T20:28:21.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.171+0000 7fcdb222d700 1 Processor -- start 2024-03-18T20:28:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.171+0000 7fcdb222d700 1 -- start start 2024-03-18T20:28:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.172+0000 7fcdb222d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdac0b4280 0x7fcdac13d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.172+0000 7fcdb222d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdac160c00 0x7fcdac1e6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.172+0000 7fcdb222d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdac160fe0 0x7fcdac1e6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.172+0000 7fcdb222d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac14f770 con 0x7fcdac0b4280 2024-03-18T20:28:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.172+0000 7fcdb222d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac1e7900 con 0x7fcdac160c00 2024-03-18T20:28:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.172+0000 7fcdb222d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac1e6cf0 con 0x7fcdac160fe0 2024-03-18T20:28:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.172+0000 7fcdab7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdac0b4280 0x7fcdac13d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.172+0000 7fcdab7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdac0b4280 0x7fcdac13d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57454/0 (socket says 172.21.15.1:57454) 2024-03-18T20:28:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.172+0000 7fcdab7fe700 1 -- 172.21.15.1:0/1366873117 learned_addr learned my addr 172.21.15.1:0/1366873117 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.172+0000 7fcdaaffd700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdac160c00 0x7fcdac1e6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.172+0000 7fcdabfff700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdac160fe0 0x7fcdac1e6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.172+0000 7fcdab7fe700 1 -- 172.21.15.1:0/1366873117 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdac160fe0 msgr2=0x7fcdac1e6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:21.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.172+0000 7fcdab7fe700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdac160fe0 0x7fcdac1e6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.172+0000 7fcdab7fe700 1 -- 172.21.15.1:0/1366873117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdac160c00 msgr2=0x7fcdac1e6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.172+0000 7fcdab7fe700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdac160c00 0x7fcdac1e6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.172+0000 7fcdab7fe700 1 -- 172.21.15.1:0/1366873117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda40094a0 con 0x7fcdac0b4280 2024-03-18T20:28:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.172+0000 7fcdab7fe700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdac0b4280 0x7fcdac13d930 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c014040 tx=0x7fcd9c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.173+0000 7fcda3fff700 1 -- 172.21.15.1:0/1366873117 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c012430 con 0x7fcdac0b4280 2024-03-18T20:28:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.173+0000 7fcda3fff700 1 -- 172.21.15.1:0/1366873117 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9c020070 con 0x7fcdac0b4280 2024-03-18T20:28:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.173+0000 7fcdb222d700 1 -- 172.21.15.1:0/1366873117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9c009a70 con 0x7fcdac0b4280 2024-03-18T20:28:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.173+0000 7fcda3fff700 1 -- 172.21.15.1:0/1366873117 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c004280 con 0x7fcdac0b4280 2024-03-18T20:28:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.173+0000 7fcdb222d700 1 -- 172.21.15.1:0/1366873117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdac1e72b0 con 0x7fcdac0b4280 2024-03-18T20:28:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.174+0000 7fcdb222d700 1 -- 172.21.15.1:0/1366873117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdac067dc0 con 0x7fcdac0b4280 2024-03-18T20:28:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.174+0000 7fcda3fff700 1 -- 172.21.15.1:0/1366873117 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 168) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fcd9c0043e0 con 0x7fcdac0b4280 2024-03-18T20:28:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.175+0000 7fcda3fff700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd8c06ae00 0x7fcd8c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.175+0000 7fcdaaffd700 1 -- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd8c06ae00 msgr2=0x7fcd8c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:28:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.175+0000 7fcda3fff700 1 -- 172.21.15.1:0/1366873117 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(288..288 src has 1..288) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fcd9c022070 con 0x7fcdac0b4280 2024-03-18T20:28:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.175+0000 7fcdaaffd700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd8c06ae00 0x7fcd8c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:28:21.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.183+0000 7fcda3fff700 1 -- 172.21.15.1:0/1366873117 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcd9c056190 con 0x7fcdac0b4280 2024-03-18T20:28:21.216 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:21.212+0000 7f493cd31240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:28:21.313 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:21.309+0000 7f5a40fa9240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:28:21.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.376+0000 7fcdaaffd700 1 -- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd8c06ae00 msgr2=0x7fcd8c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:28:21.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.376+0000 7fcdaaffd700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd8c06ae00 0x7fcd8c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:28:21.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.422+0000 7fcdb222d700 1 -- 172.21.15.1:0/1366873117 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdac0b0610 con 0x7fcd8c06ae00 2024-03-18T20:28:21.428 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:21.424+0000 7f493cd31240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:28:21.534 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:21.530+0000 7f5a40fa9240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:28:21.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.777+0000 7fcdaaffd700 1 -- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd8c06ae00 msgr2=0x7fcd8c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:28:21.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:21.777+0000 7fcdaaffd700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd8c06ae00 0x7fcd8c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T20:28:21.800 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:21.795+0000 7f493cd31240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:28:21.925 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:21.921+0000 7f5a40fa9240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:28:22.384 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:22.380+0000 7f493cd31240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:28:22.456 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:22.452+0000 7f493cd31240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:28:22.556 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:22.552+0000 7f5a40fa9240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:28:22.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:22.578+0000 7fcdaaffd700 1 -- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd8c06ae00 msgr2=0x7fcd8c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:28:22.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:22.578+0000 7fcdaaffd700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd8c06ae00 0x7fcd8c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T20:28:22.583 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:22.579+0000 7f493cd31240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:28:22.633 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:22.629+0000 7f5a40fa9240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:28:22.762 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:22.758+0000 7f493cd31240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:28:22.768 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:22.765+0000 7f5a40fa9240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:28:22.834 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:22.830+0000 7f493cd31240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:28:22.912 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:22.908+0000 7f493cd31240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:28:22.956 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:22.952+0000 7f5a40fa9240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:28:22.990 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:22.985+0000 7f493cd31240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:28:23.032 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:23.028+0000 7f5a40fa9240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:28:23.067 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:23.063+0000 7f493cd31240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:28:23.115 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:23.112+0000 7f5a40fa9240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:28:23.199 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:23.195+0000 7f5a40fa9240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:28:23.236 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:23.232+0000 7f493cd31240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:28:23.282 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:23.278+0000 7f5a40fa9240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:28:23.452 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:23.448+0000 7f493cd31240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:28:23.464 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:23.460+0000 7f5a40fa9240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:28:23.603 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:23.598+0000 7f493cd31240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:28:23.675 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:23.671+0000 7f493cd31240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:28:23.688 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:23.684+0000 7f5a40fa9240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:28:23.848 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:23.845+0000 7f5a40fa9240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:28:23.926 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:23.922+0000 7f5a40fa9240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:28:24.123 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:24.119+0000 7f493cd31240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:28:24.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:24.179+0000 7fcdaaffd700 1 -- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd8c06ae00 msgr2=0x7fcd8c06d2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:28:24.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:24.179+0000 7fcdaaffd700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd8c06ae00 0x7fcd8c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T20:28:24.403 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:24.400+0000 7f5a40fa9240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:28:24.459 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:24.455+0000 7f493cd31240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:28:24.622 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:24.618+0000 7f493cd31240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:28:24.759 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:24.755+0000 7f5a40fa9240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:28:24.934 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:24.930+0000 7f5a40fa9240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:28:25.011 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:25.007+0000 7f493cd31240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:28:25.176 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:25.172+0000 7f493cd31240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:28:25.350 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:25.347+0000 7f5a40fa9240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:28:25.528 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:25.524+0000 7f5a40fa9240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:28:25.539 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:25.535+0000 7f493cd31240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:28:25.628 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:28:25.624+0000 7f493cd31240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:28:25.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:25.693+0000 7fcda3fff700 1 -- 172.21.15.1:0/1366873117 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 169) v1 ==== 89256+0+0 (secure 0 0 0) 0x7fcd9c056560 con 0x7fcdac0b4280 2024-03-18T20:28:25.914 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:25.910+0000 7f5a40fa9240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:28:26.009 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:26.005+0000 7f5a40fa9240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:28:26.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:26.025+0000 7fcda3fff700 1 -- 172.21.15.1:0/1366873117 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 170) v1 ==== 88865+0+0 (secure 0 0 0) 0x7fcd9c05d8e0 con 0x7fcdac0b4280 2024-03-18T20:28:26.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:26.025+0000 7fcda3fff700 1 -- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd8c06ae00 msgr2=0x7fcd8c06d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:28:26.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:26.025+0000 7fcda3fff700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd8c06ae00 0x7fcd8c06d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:26.169 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:26.165+0000 7f59b6bcc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:28:26.170 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:26.165+0000 7f59b6bcc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:28:26.170 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:26.165+0000 7f59b6bcc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:28:26.170 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:26.165+0000 7f59b6bcc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:28:26.170 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:26.165+0000 7f59b6bcc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:28:26.170 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:26.166+0000 7f59b3bc6700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:28:26.170 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:26.166+0000 7f59b3bc6700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:28:26.170 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:26.166+0000 7f59b3bc6700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:28:26.171 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:26.166+0000 7f59b3bc6700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:28:26.171 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:26.166+0000 7f59b3bc6700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:28:27.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.033+0000 7fcda3fff700 1 -- 172.21.15.1:0/1366873117 <== mon.0 v2:172.21.15.1:3300/0 9 ==== mgrmap(e 171) v1 ==== 88992+0+0 (secure 0 0 0) 0x7fcd9c056e30 con 0x7fcdac0b4280 2024-03-18T20:28:27.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.033+0000 7fcda3fff700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd8c06ae00 0x7fcd8c06d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:27.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.033+0000 7fcda3fff700 1 -- 172.21.15.1:0/1366873117 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd8c01d270 con 0x7fcd8c06ae00 2024-03-18T20:28:27.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.035+0000 7fcdaaffd700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd8c06ae00 0x7fcd8c06d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:27.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.037+0000 7fcdaaffd700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd8c06ae00 0x7fcd8c06d240 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fcdac1f6390 tx=0x7fcd94007dc0 comp rx=0 tx=0).ready entity=mgr.13986 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:27.048 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:27.044+0000 7f59b0bc0700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:28:27.048 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:27.044+0000 7f59b0bc0700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:28:27.048 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:27.044+0000 7f59b0bc0700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:28:27.048 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:27.044+0000 7f59b0bc0700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:28:27.048 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:28:27.044+0000 7f59b0bc0700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:28:27.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.077+0000 7fcda3fff700 1 -- 172.21.15.1:0/1366873117 <== mgr.13986 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcd8c01d270 con 0x7fcd8c06ae00 2024-03-18T20:28:27.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.079+0000 7fcdb222d700 1 -- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd8c06ae00 msgr2=0x7fcd8c06d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:27.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.079+0000 7fcdb222d700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd8c06ae00 0x7fcd8c06d240 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fcdac1f6390 tx=0x7fcd94007dc0 comp rx=0 tx=0).stop 2024-03-18T20:28:27.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.079+0000 7fcdb222d700 1 -- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdac0b4280 msgr2=0x7fcdac13d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:27.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.079+0000 7fcdb222d700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdac0b4280 0x7fcdac13d930 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c014040 tx=0x7fcd9c004ad0 comp rx=0 tx=0).stop 2024-03-18T20:28:27.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.079+0000 7fcdb222d700 1 -- 172.21.15.1:0/1366873117 shutdown_connections 2024-03-18T20:28:27.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.079+0000 7fcdb222d700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcd8c06ae00 0x7fcd8c06d240 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:27.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.079+0000 7fcdb222d700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcdac160fe0 0x7fcdac1e6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:27.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.079+0000 7fcdb222d700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdac0b4280 0x7fcdac13d930 unknown :-1 s=CLOSED pgs=3319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:27.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.079+0000 7fcdb222d700 1 --2- 172.21.15.1:0/1366873117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdac160c00 0x7fcdac1e6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:27.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.079+0000 7fcdb222d700 1 -- 172.21.15.1:0/1366873117 >> 172.21.15.1:0/1366873117 conn(0x7fcdac0b5780 msgr2=0x7fcdac06e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:27.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.080+0000 7fcdb222d700 1 -- 172.21.15.1:0/1366873117 shutdown_connections 2024-03-18T20:28:27.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.080+0000 7fcdb222d700 1 -- 172.21.15.1:0/1366873117 wait complete. 2024-03-18T20:28:27.098 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --subvol sv01 . 1m 2024-03-18T20:28:27.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.226+0000 7f8955b2a700 1 Processor -- start 2024-03-18T20:28:27.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.226+0000 7f8955b2a700 1 -- start start 2024-03-18T20:28:27.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.227+0000 7f8955b2a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f895015d680 0x7f89500b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:27.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.227+0000 7f8955b2a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f89500b5030 0x7f8950099a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:27.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.227+0000 7f8955b2a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89500b54f0 con 0x7f895009a900 2024-03-18T20:28:27.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.227+0000 7f8955b2a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89500b57d0 con 0x7f89500b5030 2024-03-18T20:28:27.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.227+0000 7f8955b2a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89500b5ab0 con 0x7f895015d680 2024-03-18T20:28:27.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.227+0000 7f894effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f89500b5030 0x7f8950099a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:27.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.227+0000 7f894ffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f895009a900 0x7f8950161210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46212/0 (socket says 172.21.15.1:46212) 2024-03-18T20:28:27.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.227+0000 7f894ffff700 1 -- 172.21.15.1:0/2857527983 learned_addr learned my addr 172.21.15.1:0/2857527983 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:28:27.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.227+0000 7f894f7fe700 1 --2- 172.21.15.1:0/2857527983 >> v2:172.21.15.165:3300/0 conn(0x7f895015d680 0x7f89500b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:27.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.227+0000 7f894effd700 1 -- 172.21.15.1:0/2857527983 >> v2:172.21.15.165:3300/0 conn(0x7f895015d680 msgr2=0x7f89500b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:28:27.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.227+0000 7f894effd700 1 --2- 172.21.15.1:0/2857527983 >> v2:172.21.15.165:3300/0 conn(0x7f895015d680 0x7f89500b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:27.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.227+0000 7f894effd700 1 -- 172.21.15.1:0/2857527983 >> v1:172.21.15.1:6789/0 conn(0x7f895009a900 legacy=0x7f8950161210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:27.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.227+0000 7f894e7fc700 1 -- 172.21.15.1:0/2857527983 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459785687 0 0) 0x7f89500b54f0 con 0x7f895009a900 2024-03-18T20:28:27.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.227+0000 7f894effd700 1 -- 172.21.15.1:0/2857527983 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89500b5d90 con 0x7f89500b5030 2024-03-18T20:28:27.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.227+0000 7f894effd700 1 --2- 172.21.15.1:0/2857527983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89500b5030 0x7f8950099a30 secure :-1 s=READY pgs=3333 cs=0 l=1 rev1=1 crypto rx=0x7f8950072200 tx=0x7f8944009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0871c6b0a9ba9c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:27.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.228+0000 7f894e7fc700 1 -- 172.21.15.1:0/2857527983 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8944011830 con 0x7f89500b5030 2024-03-18T20:28:27.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.228+0000 7f894e7fc700 1 -- 172.21.15.1:0/2857527983 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8944011990 con 0x7f89500b5030 2024-03-18T20:28:27.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.228+0000 7f894e7fc700 1 -- 172.21.15.1:0/2857527983 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8944011d20 con 0x7f89500b5030 2024-03-18T20:28:27.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.228+0000 7f8955b2a700 1 -- 172.21.15.1:0/2857527983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89500b5030 msgr2=0x7f8950099a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:27.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.228+0000 7f8955b2a700 1 --2- 172.21.15.1:0/2857527983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89500b5030 0x7f8950099a30 secure :-1 s=READY pgs=3333 cs=0 l=1 rev1=1 crypto rx=0x7f8950072200 tx=0x7f8944009b20 comp rx=0 tx=0).stop 2024-03-18T20:28:27.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.228+0000 7f8955b2a700 1 -- 172.21.15.1:0/2857527983 shutdown_connections 2024-03-18T20:28:27.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.228+0000 7f8955b2a700 1 --2- 172.21.15.1:0/2857527983 >> v2:172.21.15.165:3300/0 conn(0x7f895015d680 0x7f89500b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:27.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.228+0000 7f8955b2a700 1 --2- 172.21.15.1:0/2857527983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89500b5030 0x7f8950099a30 unknown :-1 s=CLOSED pgs=3333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:27.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.228+0000 7f8955b2a700 1 -- 172.21.15.1:0/2857527983 >> 172.21.15.1:0/2857527983 conn(0x7f89500b2ec0 msgr2=0x7f89500b32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:27.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.228+0000 7f8955b2a700 1 -- 172.21.15.1:0/2857527983 shutdown_connections 2024-03-18T20:28:27.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.228+0000 7f8955b2a700 1 -- 172.21.15.1:0/2857527983 wait complete. 2024-03-18T20:28:27.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.230+0000 7f8955b2a700 1 Processor -- start 2024-03-18T20:28:27.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.230+0000 7f8955b2a700 1 -- start start 2024-03-18T20:28:27.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.231+0000 7f8955b2a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f895009a900 0x7f8950144710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:27.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.231+0000 7f8955b2a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89500b5030 0x7f89501499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:27.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.231+0000 7f8955b2a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f895015d680 0x7f8950149ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:27.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.231+0000 7f8955b2a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89500ae230 con 0x7f895009a900 2024-03-18T20:28:27.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.231+0000 7f8955b2a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8950144f50 con 0x7f895015d680 2024-03-18T20:28:27.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.231+0000 7f8955b2a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8950145230 con 0x7f89500b5030 2024-03-18T20:28:27.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.231+0000 7f894f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f895009a900 0x7f8950144710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:27.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.231+0000 7f894f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f895009a900 0x7f8950144710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52994/0 (socket says 172.21.15.1:52994) 2024-03-18T20:28:27.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.231+0000 7f894f7fe700 1 -- 172.21.15.1:0/3645490769 learned_addr learned my addr 172.21.15.1:0/3645490769 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:28:27.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.231+0000 7f894effd700 1 --2- 172.21.15.1:0/3645490769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89500b5030 0x7f89501499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:27.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.231+0000 7f894ffff700 1 --2- 172.21.15.1:0/3645490769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f895015d680 0x7f8950149ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:27.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.231+0000 7f894f7fe700 1 -- 172.21.15.1:0/3645490769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89500b5030 msgr2=0x7f89501499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:27.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.231+0000 7f894f7fe700 1 --2- 172.21.15.1:0/3645490769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89500b5030 0x7f89501499a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:27.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.231+0000 7f894f7fe700 1 -- 172.21.15.1:0/3645490769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f895015d680 msgr2=0x7f8950149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:27.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.231+0000 7f894f7fe700 1 --2- 172.21.15.1:0/3645490769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f895015d680 0x7f8950149ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:27.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.231+0000 7f894f7fe700 1 -- 172.21.15.1:0/3645490769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89440094a0 con 0x7f895009a900 2024-03-18T20:28:27.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.231+0000 7f894ffff700 1 --2- 172.21.15.1:0/3645490769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f895015d680 0x7f8950149ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:28:27.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.231+0000 7f894f7fe700 1 --2- 172.21.15.1:0/3645490769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f895009a900 0x7f8950144710 secure :-1 s=READY pgs=3334 cs=0 l=1 rev1=1 crypto rx=0x7f894000e850 tx=0x7f894000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:27.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.232+0000 7f8937fff700 1 -- 172.21.15.1:0/3645490769 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89400095b0 con 0x7f895009a900 2024-03-18T20:28:27.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.232+0000 7f8937fff700 1 -- 172.21.15.1:0/3645490769 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f894000ee50 con 0x7f895009a900 2024-03-18T20:28:27.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.232+0000 7f8955b2a700 1 -- 172.21.15.1:0/3645490769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8950145510 con 0x7f895009a900 2024-03-18T20:28:27.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.232+0000 7f8937fff700 1 -- 172.21.15.1:0/3645490769 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894000ca70 con 0x7f895009a900 2024-03-18T20:28:27.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.232+0000 7f8955b2a700 1 -- 172.21.15.1:0/3645490769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8950144c50 con 0x7f895009a900 2024-03-18T20:28:27.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.233+0000 7f8955b2a700 1 -- 172.21.15.1:0/3645490769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8950067dc0 con 0x7f895009a900 2024-03-18T20:28:27.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.233+0000 7f8937fff700 1 -- 172.21.15.1:0/3645490769 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 171) v1 ==== 88992+0+0 (secure 0 0 0) 0x7f894000cbd0 con 0x7f895009a900 2024-03-18T20:28:27.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.233+0000 7f8937fff700 1 --2- 172.21.15.1:0/3645490769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f893c06ab30 0x7f893c06cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:28:27.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.234+0000 7f8937fff700 1 -- 172.21.15.1:0/3645490769 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(289..289 src has 1..289) v4 ==== 6473+0+0 (secure 0 0 0) 0x7f8940089270 con 0x7f895009a900 2024-03-18T20:28:27.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.234+0000 7f894effd700 1 --2- 172.21.15.1:0/3645490769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f893c06ab30 0x7f893c06cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:28:27.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.234+0000 7f894effd700 1 --2- 172.21.15.1:0/3645490769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f893c06ab30 0x7f893c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8950072200 tx=0x7f8944008030 comp rx=0 tx=0).ready entity=mgr.13986 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:28:27.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.238+0000 7f8937fff700 1 -- 172.21.15.1:0/3645490769 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8940054490 con 0x7f895009a900 2024-03-18T20:28:27.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.465+0000 7f8955b2a700 1 -- 172.21.15.1:0/3645490769 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "subvol": "sv01", "path": ".", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f89500b3d90 con 0x7f893c06ab30 2024-03-18T20:28:27.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.527+0000 7f8937fff700 1 -- 172.21.15.1:0/3645490769 <== mgr.13986 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+85 (secure 0 0 0) 0x7f89500b3d90 con 0x7f893c06ab30 2024-03-18T20:28:27.533 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path /volumes/_nogroup/sv01/41b35cb8-cb4a-45c2-a643-c335f1be04d4/.. 2024-03-18T20:28:27.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.529+0000 7f8955b2a700 1 -- 172.21.15.1:0/3645490769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f893c06ab30 msgr2=0x7f893c06cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:27.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.529+0000 7f8955b2a700 1 --2- 172.21.15.1:0/3645490769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f893c06ab30 0x7f893c06cff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8950072200 tx=0x7f8944008030 comp rx=0 tx=0).stop 2024-03-18T20:28:27.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.529+0000 7f8955b2a700 1 -- 172.21.15.1:0/3645490769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f895009a900 msgr2=0x7f8950144710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:28:27.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.529+0000 7f8955b2a700 1 --2- 172.21.15.1:0/3645490769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f895009a900 0x7f8950144710 secure :-1 s=READY pgs=3334 cs=0 l=1 rev1=1 crypto rx=0x7f894000e850 tx=0x7f894000ec10 comp rx=0 tx=0).stop 2024-03-18T20:28:27.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.529+0000 7f8955b2a700 1 -- 172.21.15.1:0/3645490769 shutdown_connections 2024-03-18T20:28:27.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.529+0000 7f8955b2a700 1 --2- 172.21.15.1:0/3645490769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f893c06ab30 0x7f893c06cff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:27.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.529+0000 7f8955b2a700 1 --2- 172.21.15.1:0/3645490769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f895015d680 0x7f8950149ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:27.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.529+0000 7f8955b2a700 1 --2- 172.21.15.1:0/3645490769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f895009a900 0x7f8950144710 unknown :-1 s=CLOSED pgs=3334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:27.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.529+0000 7f8955b2a700 1 --2- 172.21.15.1:0/3645490769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f89500b5030 0x7f89501499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:28:27.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.529+0000 7f8955b2a700 1 -- 172.21.15.1:0/3645490769 >> 172.21.15.1:0/3645490769 conn(0x7f89500b2ec0 msgr2=0x7f8950156e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:28:27.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.530+0000 7f8955b2a700 1 -- 172.21.15.1:0/3645490769 shutdown_connections 2024-03-18T20:28:27.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:28:27.530+0000 7f8955b2a700 1 -- 172.21.15.1:0/3645490769 wait complete. 2024-03-18T20:28:27.547 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /volumes/_nogroup/sv01/41b35cb8-cb4a-45c2-a643-c335f1be04d4/.. 2024-03-18T20:28:41.784 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:28:41.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:29:00.544 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:00.540+0000 7f59a1b62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:29:00.544 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:00.540+0000 7f59a1b62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:29:00.544 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:00.540+0000 7f59a1b62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:29:00.544 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:00.540+0000 7f59a1b62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:29:00.544 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:00.540+0000 7f59a1b62700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:29:11.858 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:29:11.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:29:41.925 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:29:41.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:29:42.548 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-03-18T20:29:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.684+0000 7f5a44e45700 1 Processor -- start 2024-03-18T20:29:42.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.684+0000 7f5a44e45700 1 -- start start 2024-03-18T20:29:42.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.685+0000 7f5a44e45700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5a400adc50 0x7f5a400b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:42.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.685+0000 7f5a44e45700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4006f860 con 0x7f5a40099b30 2024-03-18T20:29:42.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.685+0000 7f5a44e45700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a400ad3c0 con 0x7f5a4015b610 2024-03-18T20:29:42.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.685+0000 7f5a44e45700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a400b2c20 con 0x7f5a400adc50 2024-03-18T20:29:42.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.685+0000 7f5a3e59c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5a400adc50 0x7f5a400b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:42.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.685+0000 7f5a3dd9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5a40099b30 0x7f5a400ae030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47936/0 (socket says 172.21.15.1:47936) 2024-03-18T20:29:42.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.685+0000 7f5a3dd9b700 1 -- 172.21.15.1:0/2520773402 learned_addr learned my addr 172.21.15.1:0/2520773402 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:29:42.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.686+0000 7f5a3e59c700 1 -- 172.21.15.1:0/2520773402 >> v1:172.21.15.165:6789/0 conn(0x7f5a4015b610 legacy=0x7f5a4015bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:42.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.686+0000 7f5a3e59c700 1 -- 172.21.15.1:0/2520773402 >> v1:172.21.15.1:6789/0 conn(0x7f5a40099b30 legacy=0x7f5a400ae030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:42.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.686+0000 7f5a3e59c700 1 -- 172.21.15.1:0/2520773402 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a400ad6a0 con 0x7f5a400adc50 2024-03-18T20:29:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.686+0000 7f5a3d59a700 1 -- 172.21.15.1:0/2520773402 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607145552 0 0) 0x7f5a4006f860 con 0x7f5a40099b30 2024-03-18T20:29:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.686+0000 7f5a3e59c700 1 --2- 172.21.15.1:0/2520773402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a400adc50 0x7f5a400b26e0 secure :-1 s=READY pgs=3340 cs=0 l=1 rev1=1 crypto rx=0x7f5a34009840 tx=0x7f5a34009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b473516aac01ce92 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.686+0000 7f5a3d59a700 1 -- 172.21.15.1:0/2520773402 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a34004230 con 0x7f5a400adc50 2024-03-18T20:29:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.686+0000 7f5a3d59a700 1 -- 172.21.15.1:0/2520773402 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a34004390 con 0x7f5a400adc50 2024-03-18T20:29:42.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.686+0000 7f5a3d59a700 1 -- 172.21.15.1:0/2520773402 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a340036a0 con 0x7f5a400adc50 2024-03-18T20:29:42.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.687+0000 7f5a44e45700 1 -- 172.21.15.1:0/2520773402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a400adc50 msgr2=0x7f5a400b26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:42.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.687+0000 7f5a44e45700 1 --2- 172.21.15.1:0/2520773402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a400adc50 0x7f5a400b26e0 secure :-1 s=READY pgs=3340 cs=0 l=1 rev1=1 crypto rx=0x7f5a34009840 tx=0x7f5a34009b50 comp rx=0 tx=0).stop 2024-03-18T20:29:42.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.687+0000 7f5a44e45700 1 -- 172.21.15.1:0/2520773402 shutdown_connections 2024-03-18T20:29:42.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.687+0000 7f5a44e45700 1 --2- 172.21.15.1:0/2520773402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a400adc50 0x7f5a400b26e0 unknown :-1 s=CLOSED pgs=3340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:42.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.687+0000 7f5a44e45700 1 -- 172.21.15.1:0/2520773402 >> 172.21.15.1:0/2520773402 conn(0x7f5a400b5a10 msgr2=0x7f5a400b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:42.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.687+0000 7f5a44e45700 1 -- 172.21.15.1:0/2520773402 shutdown_connections 2024-03-18T20:29:42.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.687+0000 7f5a44e45700 1 -- 172.21.15.1:0/2520773402 wait complete. 2024-03-18T20:29:42.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.688+0000 7f5a44e45700 1 Processor -- start 2024-03-18T20:29:42.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.688+0000 7f5a44e45700 1 -- start start 2024-03-18T20:29:42.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.689+0000 7f5a44e45700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a40099b30 0x7f5a4014ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:42.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.689+0000 7f5a44e45700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a400adc50 0x7f5a40153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:42.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.689+0000 7f5a44e45700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a4015b610 0x7f5a4014b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:42.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.689+0000 7f5a44e45700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4014b270 con 0x7f5a400adc50 2024-03-18T20:29:42.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.689+0000 7f5a44e45700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a40154010 con 0x7f5a4015b610 2024-03-18T20:29:42.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.689+0000 7f5a44e45700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a40155b30 con 0x7f5a40099b30 2024-03-18T20:29:42.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.689+0000 7f5a3dd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a400adc50 0x7f5a40153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:42.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.689+0000 7f5a3dd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a400adc50 0x7f5a40153600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44994/0 (socket says 172.21.15.1:44994) 2024-03-18T20:29:42.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.689+0000 7f5a3dd9b700 1 -- 172.21.15.1:0/91969841 learned_addr learned my addr 172.21.15.1:0/91969841 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.689+0000 7f5a3ed9d700 1 --2- 172.21.15.1:0/91969841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a4015b610 0x7f5a4014b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.689+0000 7f5a3e59c700 1 --2- 172.21.15.1:0/91969841 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a40099b30 0x7f5a4014ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.689+0000 7f5a3dd9b700 1 -- 172.21.15.1:0/91969841 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a40099b30 msgr2=0x7f5a4014ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.689+0000 7f5a3dd9b700 1 --2- 172.21.15.1:0/91969841 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a40099b30 0x7f5a4014ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.689+0000 7f5a3dd9b700 1 -- 172.21.15.1:0/91969841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a4015b610 msgr2=0x7f5a4014b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.689+0000 7f5a3dd9b700 1 --2- 172.21.15.1:0/91969841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a4015b610 0x7f5a4014b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.689+0000 7f5a3dd9b700 1 -- 172.21.15.1:0/91969841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a340094a0 con 0x7f5a400adc50 2024-03-18T20:29:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.689+0000 7f5a3e59c700 1 --2- 172.21.15.1:0/91969841 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a40099b30 0x7f5a4014ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:29:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.690+0000 7f5a3dd9b700 1 --2- 172.21.15.1:0/91969841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a400adc50 0x7f5a40153600 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7f5a300011e0 tx=0x7f5a3000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.690+0000 7f5a2effd700 1 -- 172.21.15.1:0/91969841 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a30004c80 con 0x7f5a400adc50 2024-03-18T20:29:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.690+0000 7f5a2effd700 1 -- 172.21.15.1:0/91969841 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a30004e00 con 0x7f5a400adc50 2024-03-18T20:29:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.690+0000 7f5a44e45700 1 -- 172.21.15.1:0/91969841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a40155e10 con 0x7f5a400adc50 2024-03-18T20:29:42.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.690+0000 7f5a2effd700 1 -- 172.21.15.1:0/91969841 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a30007310 con 0x7f5a400adc50 2024-03-18T20:29:42.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.690+0000 7f5a44e45700 1 -- 172.21.15.1:0/91969841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a401563d0 con 0x7f5a400adc50 2024-03-18T20:29:42.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.691+0000 7f5a44e45700 1 -- 172.21.15.1:0/91969841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a401565f0 con 0x7f5a400adc50 2024-03-18T20:29:42.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.691+0000 7f5a2effd700 1 -- 172.21.15.1:0/91969841 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 174) v1 ==== 89421+0+0 (secure 0 0 0) 0x7f5a30007470 con 0x7f5a400adc50 2024-03-18T20:29:42.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.692+0000 7f5a2effd700 1 --2- 172.21.15.1:0/91969841 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5a2006afb0 0x7f5a2006d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:42.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.692+0000 7f5a3e59c700 1 --2- 172.21.15.1:0/91969841 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5a2006afb0 0x7f5a2006d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:42.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.692+0000 7f5a2effd700 1 -- 172.21.15.1:0/91969841 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(291..291 src has 1..291) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f5a30089880 con 0x7f5a400adc50 2024-03-18T20:29:42.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.692+0000 7f5a3e59c700 1 --2- 172.21.15.1:0/91969841 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5a2006afb0 0x7f5a2006d470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5a34000c00 tx=0x7f5a34008030 comp rx=0 tx=0).ready entity=mgr.13986 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:42.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.697+0000 7f5a2effd700 1 -- 172.21.15.1:0/91969841 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5a30054450 con 0x7f5a400adc50 2024-03-18T20:29:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:42.958+0000 7f5a44e45700 1 -- 172.21.15.1:0/91969841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f5a40067dc0 con 0x7f5a400adc50 2024-03-18T20:29:43.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.152+0000 7f5a2effd700 1 -- 172.21.15.1:0/91969841 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 175) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f5a30054820 con 0x7f5a400adc50 2024-03-18T20:29:43.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.153+0000 7f5a2effd700 1 -- 172.21.15.1:0/91969841 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v175) v1 ==== 93+0+0 (secure 0 0 0) 0x7f5a30073c30 con 0x7f5a400adc50 2024-03-18T20:29:43.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.156+0000 7f5a44e45700 1 -- 172.21.15.1:0/91969841 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5a2006afb0 msgr2=0x7f5a2006d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:43.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.156+0000 7f5a44e45700 1 --2- 172.21.15.1:0/91969841 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5a2006afb0 0x7f5a2006d470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5a34000c00 tx=0x7f5a34008030 comp rx=0 tx=0).stop 2024-03-18T20:29:43.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.156+0000 7f5a44e45700 1 -- 172.21.15.1:0/91969841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a400adc50 msgr2=0x7f5a40153600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:43.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.156+0000 7f5a44e45700 1 --2- 172.21.15.1:0/91969841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a400adc50 0x7f5a40153600 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7f5a300011e0 tx=0x7f5a3000cae0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.156+0000 7f5a44e45700 1 -- 172.21.15.1:0/91969841 shutdown_connections 2024-03-18T20:29:43.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.156+0000 7f5a44e45700 1 --2- 172.21.15.1:0/91969841 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5a2006afb0 0x7f5a2006d470 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.156+0000 7f5a44e45700 1 --2- 172.21.15.1:0/91969841 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5a40099b30 0x7f5a4014ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.156+0000 7f5a44e45700 1 --2- 172.21.15.1:0/91969841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a4015b610 0x7f5a4014b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.156+0000 7f5a44e45700 1 --2- 172.21.15.1:0/91969841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a400adc50 0x7f5a40153600 unknown :-1 s=CLOSED pgs=3341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.156+0000 7f5a44e45700 1 -- 172.21.15.1:0/91969841 >> 172.21.15.1:0/91969841 conn(0x7f5a400b5a10 msgr2=0x7f5a4006daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:43.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.157+0000 7f5a44e45700 1 -- 172.21.15.1:0/91969841 shutdown_connections 2024-03-18T20:29:43.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.157+0000 7f5a44e45700 1 -- 172.21.15.1:0/91969841 wait complete. 2024-03-18T20:29:43.175 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-18T20:29:43.175 DEBUG:teuthology.orchestra.run.smithi001:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:29:43.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-18T20:29:43.260 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-18T20:29:43.261 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:29:43.261 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:29:43.279 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:29:43.305 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:43.301+0000 7f972cced240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:29:43.307 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:43.302+0000 7f97503aa240 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T20:29:43.451 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:43.447+0000 7f972cced240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:29:43.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.472+0000 7fde4f841700 1 Processor -- start 2024-03-18T20:29:43.477 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:43.472+0000 7f97503aa240 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T20:29:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.472+0000 7fde4f841700 1 -- start start 2024-03-18T20:29:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.473+0000 7fde4f841700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde481599c0 0x7fde480b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.473+0000 7fde4f841700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fde480ad520 0x7fde4806d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.473+0000 7fde4f841700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fde480ad980 0x7fde4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.473+0000 7fde4f841700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde480aea60 con 0x7fde480ad980 2024-03-18T20:29:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.473+0000 7fde4f841700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde480ade50 con 0x7fde480ad520 2024-03-18T20:29:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.473+0000 7fde4f841700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde480ae130 con 0x7fde481599c0 2024-03-18T20:29:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.473+0000 7fde4ddde700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fde480ad980 0x7fde4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:43.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.473+0000 7fde4ddde700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fde480ad980 0x7fde4806d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45018/0 (socket says 172.21.15.1:45018) 2024-03-18T20:29:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.473+0000 7fde4ddde700 1 -- 172.21.15.1:0/471747763 learned_addr learned my addr 172.21.15.1:0/471747763 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.473+0000 7fde4cddc700 1 --2- 172.21.15.1:0/471747763 >> v2:172.21.15.165:3300/0 conn(0x7fde480ad520 0x7fde4806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.473+0000 7fde4d5dd700 1 --2- 172.21.15.1:0/471747763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde481599c0 0x7fde480b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.474+0000 7fde4ddde700 1 -- 172.21.15.1:0/471747763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde481599c0 msgr2=0x7fde480b3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.474+0000 7fde4ddde700 1 --2- 172.21.15.1:0/471747763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde481599c0 0x7fde480b3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.474+0000 7fde4ddde700 1 -- 172.21.15.1:0/471747763 >> v2:172.21.15.165:3300/0 conn(0x7fde480ad520 msgr2=0x7fde4806d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.474+0000 7fde4ddde700 1 --2- 172.21.15.1:0/471747763 >> v2:172.21.15.165:3300/0 conn(0x7fde480ad520 0x7fde4806d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.474+0000 7fde4ddde700 1 -- 172.21.15.1:0/471747763 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde480ae410 con 0x7fde480ad980 2024-03-18T20:29:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.474+0000 7fde4cddc700 1 --2- 172.21.15.1:0/471747763 >> v2:172.21.15.165:3300/0 conn(0x7fde480ad520 0x7fde4806d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:29:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.474+0000 7fde4ddde700 1 --2- 172.21.15.1:0/471747763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde480ad980 0x7fde4806d9c0 secure :-1 s=READY pgs=3345 cs=0 l=1 rev1=1 crypto rx=0x7fde4806eb10 tx=0x7fde4000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe552e2e1b44fbf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.474+0000 7fde3ffff700 1 -- 172.21.15.1:0/471747763 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4000e070 con 0x7fde480ad980 2024-03-18T20:29:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.474+0000 7fde3ffff700 1 -- 172.21.15.1:0/471747763 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4000b910 con 0x7fde480ad980 2024-03-18T20:29:43.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.474+0000 7fde3ffff700 1 -- 172.21.15.1:0/471747763 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4000bcc0 con 0x7fde480ad980 2024-03-18T20:29:43.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.475+0000 7fde4f841700 1 -- 172.21.15.1:0/471747763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde480ad980 msgr2=0x7fde4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:43.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.475+0000 7fde4f841700 1 --2- 172.21.15.1:0/471747763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde480ad980 0x7fde4806d9c0 secure :-1 s=READY pgs=3345 cs=0 l=1 rev1=1 crypto rx=0x7fde4806eb10 tx=0x7fde4000b330 comp rx=0 tx=0).stop 2024-03-18T20:29:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.475+0000 7fde4f841700 1 -- 172.21.15.1:0/471747763 shutdown_connections 2024-03-18T20:29:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.475+0000 7fde4f841700 1 --2- 172.21.15.1:0/471747763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde480ad980 0x7fde4806d9c0 unknown :-1 s=CLOSED pgs=3345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.475+0000 7fde4f841700 1 --2- 172.21.15.1:0/471747763 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde481599c0 0x7fde480b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.475+0000 7fde4f841700 1 --2- 172.21.15.1:0/471747763 >> v2:172.21.15.165:3300/0 conn(0x7fde480ad520 0x7fde4806d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.475+0000 7fde4f841700 1 -- 172.21.15.1:0/471747763 >> 172.21.15.1:0/471747763 conn(0x7fde4806a9b0 msgr2=0x7fde4806adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.475+0000 7fde4f841700 1 -- 172.21.15.1:0/471747763 shutdown_connections 2024-03-18T20:29:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.475+0000 7fde4f841700 1 -- 172.21.15.1:0/471747763 wait complete. 2024-03-18T20:29:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.476+0000 7fde4f841700 1 Processor -- start 2024-03-18T20:29:43.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.476+0000 7fde4f841700 1 -- start start 2024-03-18T20:29:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.478+0000 7fde4f841700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde480ad520 0x7fde481fa3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.478+0000 7fde4f841700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde480ad980 0x7fde48071b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.478+0000 7fde4f841700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde481599c0 0x7fde480720c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.478+0000 7fde4f841700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde480ccf40 con 0x7fde481599c0 2024-03-18T20:29:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.478+0000 7fde4f841700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4809c610 con 0x7fde480ad520 2024-03-18T20:29:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.478+0000 7fde4f841700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde480cbfc0 con 0x7fde480ad980 2024-03-18T20:29:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.478+0000 7fde4ddde700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde481599c0 0x7fde480720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:43.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.478+0000 7fde4ddde700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde481599c0 0x7fde480720c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45028/0 (socket says 172.21.15.1:45028) 2024-03-18T20:29:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.478+0000 7fde4ddde700 1 -- 172.21.15.1:0/4177264862 learned_addr learned my addr 172.21.15.1:0/4177264862 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.478+0000 7fde4d5dd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde480ad520 0x7fde481fa3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.478+0000 7fde4cddc700 1 --2- 172.21.15.1:0/4177264862 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde480ad980 0x7fde48071b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.478+0000 7fde4ddde700 1 -- 172.21.15.1:0/4177264862 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde480ad980 msgr2=0x7fde48071b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.478+0000 7fde4ddde700 1 --2- 172.21.15.1:0/4177264862 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde480ad980 0x7fde48071b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.478+0000 7fde4ddde700 1 -- 172.21.15.1:0/4177264862 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde480ad520 msgr2=0x7fde481fa3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.478+0000 7fde4ddde700 1 --2- 172.21.15.1:0/4177264862 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde480ad520 0x7fde481fa3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.478+0000 7fde4ddde700 1 -- 172.21.15.1:0/4177264862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde3800a550 con 0x7fde481599c0 2024-03-18T20:29:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.478+0000 7fde4d5dd700 1 --2- 172.21.15.1:0/4177264862 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde480ad520 0x7fde481fa3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:29:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.478+0000 7fde4ddde700 1 --2- 172.21.15.1:0/4177264862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde481599c0 0x7fde480720c0 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 crypto rx=0x7fde40012000 tx=0x7fde40004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.479+0000 7fde3dffb700 1 -- 172.21.15.1:0/4177264862 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde40004880 con 0x7fde481599c0 2024-03-18T20:29:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.479+0000 7fde3dffb700 1 -- 172.21.15.1:0/4177264862 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde400092e0 con 0x7fde481599c0 2024-03-18T20:29:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.479+0000 7fde3dffb700 1 -- 172.21.15.1:0/4177264862 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4001c660 con 0x7fde481599c0 2024-03-18T20:29:43.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.479+0000 7fde4f841700 1 -- 172.21.15.1:0/4177264862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde40009b30 con 0x7fde481599c0 2024-03-18T20:29:43.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.479+0000 7fde4f841700 1 -- 172.21.15.1:0/4177264862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde480ccb60 con 0x7fde481599c0 2024-03-18T20:29:43.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.480+0000 7fde4f841700 1 -- 172.21.15.1:0/4177264862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde480cd080 con 0x7fde481599c0 2024-03-18T20:29:43.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.482+0000 7fde3dffb700 1 -- 172.21.15.1:0/4177264862 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 175) v1 ==== 89404+0+0 (secure 0 0 0) 0x7fde4001c7c0 con 0x7fde481599c0 2024-03-18T20:29:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.482+0000 7fde3dffb700 1 --2- 172.21.15.1:0/4177264862 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fde2c06af60 0x7fde2c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.482+0000 7fde4d5dd700 1 -- 172.21.15.1:0/4177264862 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fde2c06af60 msgr2=0x7fde2c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.482+0000 7fde3dffb700 1 -- 172.21.15.1:0/4177264862 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(291..291 src has 1..291) v4 ==== 6560+0+0 (secure 0 0 0) 0x7fde4008b570 con 0x7fde481599c0 2024-03-18T20:29:43.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.482+0000 7fde4d5dd700 1 --2- 172.21.15.1:0/4177264862 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fde2c06af60 0x7fde2c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:29:43.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.485+0000 7fde3dffb700 1 -- 172.21.15.1:0/4177264862 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fde40056190 con 0x7fde481599c0 2024-03-18T20:29:43.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.683+0000 7fde4d5dd700 1 -- 172.21.15.1:0/4177264862 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fde2c06af60 msgr2=0x7fde2c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:43.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.683+0000 7fde4d5dd700 1 --2- 172.21.15.1:0/4177264862 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fde2c06af60 0x7fde2c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:29:43.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.689+0000 7fde4f841700 1 -- 172.21.15.1:0/4177264862 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde480cd080 con 0x7fde481599c0 2024-03-18T20:29:43.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.690+0000 7fde3dffb700 1 -- 172.21.15.1:0/4177264862 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v291) v1 ==== 74+0+16996 (secure 0 0 0) 0x7fde480cd080 con 0x7fde481599c0 2024-03-18T20:29:43.695 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:29:43.695 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":291,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:29:27.236798+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:28:08.503736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"287","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-03-18T20:28:09.644774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"288","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":290,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":290,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"38.2","mappings":[{"from":7,"to":2}]},{"pgid":"38.8","mappings":[{"from":7,"to":4}]},{"pgid":"38.12","mappings":[{"from":7,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/1650741459":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/2451665032":"2024-03-19T20:28:26.019720+0000","172.21.15.1:6833/148356":"2024-03-19T20:28:26.019720+0000","172.21.15.1:6832/148356":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/1408878463":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/582749307":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/997400106":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/2561447959":"2024-03-19T20:28:26.019720+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:29:43.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.692+0000 7fde4f841700 1 -- 172.21.15.1:0/4177264862 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fde2c06af60 msgr2=0x7fde2c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:29:43.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.692+0000 7fde4f841700 1 --2- 172.21.15.1:0/4177264862 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fde2c06af60 0x7fde2c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.692+0000 7fde4f841700 1 -- 172.21.15.1:0/4177264862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde481599c0 msgr2=0x7fde480720c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:43.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.692+0000 7fde4f841700 1 --2- 172.21.15.1:0/4177264862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde481599c0 0x7fde480720c0 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 crypto rx=0x7fde40012000 tx=0x7fde40004ad0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.692+0000 7fde4f841700 1 -- 172.21.15.1:0/4177264862 shutdown_connections 2024-03-18T20:29:43.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.692+0000 7fde4f841700 1 --2- 172.21.15.1:0/4177264862 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fde2c06af60 0x7fde2c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.692+0000 7fde4f841700 1 --2- 172.21.15.1:0/4177264862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde481599c0 0x7fde480720c0 unknown :-1 s=CLOSED pgs=3346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.692+0000 7fde4f841700 1 --2- 172.21.15.1:0/4177264862 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde480ad520 0x7fde481fa3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.692+0000 7fde4f841700 1 --2- 172.21.15.1:0/4177264862 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde480ad980 0x7fde48071b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.692+0000 7fde4f841700 1 -- 172.21.15.1:0/4177264862 >> 172.21.15.1:0/4177264862 conn(0x7fde4806a9b0 msgr2=0x7fde480b2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:43.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.692+0000 7fde4f841700 1 -- 172.21.15.1:0/4177264862 shutdown_connections 2024-03-18T20:29:43.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.693+0000 7fde4f841700 1 -- 172.21.15.1:0/4177264862 wait complete. 2024-03-18T20:29:43.710 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:28:08.503736+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '287', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 38, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:28:09.644774+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '288', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:29:43.711 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:29:43.717 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:43.713+0000 7f972cced240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:29:43.759 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:43.754+0000 7f97503aa240 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T20:29:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.837+0000 7fde7692f700 1 Processor -- start 2024-03-18T20:29:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.837+0000 7fde7692f700 1 -- start start 2024-03-18T20:29:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.838+0000 7fde7692f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fde7009a140 0x7fde700b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.838+0000 7fde7692f700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde700b5800 con 0x7fde7006fc60 2024-03-18T20:29:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.838+0000 7fde7692f700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde700b5ae0 con 0x7fde700b4fc0 2024-03-18T20:29:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.838+0000 7fde7692f700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde700b5dc0 con 0x7fde7009a140 2024-03-18T20:29:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.838+0000 7fde6f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fde7009a140 0x7fde700b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.838+0000 7fde6ffff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fde7006fc60 0x7fde70099a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47942/0 (socket says 172.21.15.1:47942) 2024-03-18T20:29:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.838+0000 7fde6f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fde7009a140 0x7fde700b4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45032/0 (socket says 172.21.15.1:45032) 2024-03-18T20:29:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.838+0000 7fde6ffff700 1 -- 172.21.15.1:0/1207907044 learned_addr learned my addr 172.21.15.1:0/1207907044 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:29:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.839+0000 7fde6f7fe700 1 -- 172.21.15.1:0/1207907044 >> v1:172.21.15.165:6789/0 conn(0x7fde700b4fc0 legacy=0x7fde7006bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.839+0000 7fde6f7fe700 1 -- 172.21.15.1:0/1207907044 >> v1:172.21.15.1:6789/0 conn(0x7fde7006fc60 legacy=0x7fde70099a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.839+0000 7fde6effd700 1 -- 172.21.15.1:0/1207907044 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871752586 0 0) 0x7fde700b5800 con 0x7fde7006fc60 2024-03-18T20:29:43.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.839+0000 7fde6f7fe700 1 -- 172.21.15.1:0/1207907044 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde700b60a0 con 0x7fde7009a140 2024-03-18T20:29:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.839+0000 7fde6f7fe700 1 --2- 172.21.15.1:0/1207907044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde7009a140 0x7fde700b4a80 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7fde64009840 tx=0x7fde64009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f654522bda701387 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.839+0000 7fde6effd700 1 -- 172.21.15.1:0/1207907044 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde64011830 con 0x7fde7009a140 2024-03-18T20:29:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.839+0000 7fde6effd700 1 -- 172.21.15.1:0/1207907044 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde64011990 con 0x7fde7009a140 2024-03-18T20:29:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.839+0000 7fde6effd700 1 -- 172.21.15.1:0/1207907044 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde64011d20 con 0x7fde7009a140 2024-03-18T20:29:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.840+0000 7fde7692f700 1 -- 172.21.15.1:0/1207907044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde7009a140 msgr2=0x7fde700b4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.840+0000 7fde7692f700 1 --2- 172.21.15.1:0/1207907044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde7009a140 0x7fde700b4a80 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7fde64009840 tx=0x7fde64009b50 comp rx=0 tx=0).stop 2024-03-18T20:29:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.840+0000 7fde7692f700 1 -- 172.21.15.1:0/1207907044 shutdown_connections 2024-03-18T20:29:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.840+0000 7fde7692f700 1 --2- 172.21.15.1:0/1207907044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde7009a140 0x7fde700b4a80 unknown :-1 s=CLOSED pgs=3348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.840+0000 7fde7692f700 1 -- 172.21.15.1:0/1207907044 >> 172.21.15.1:0/1207907044 conn(0x7fde700b2de0 msgr2=0x7fde700b31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.840+0000 7fde7692f700 1 -- 172.21.15.1:0/1207907044 shutdown_connections 2024-03-18T20:29:43.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.840+0000 7fde7692f700 1 -- 172.21.15.1:0/1207907044 wait complete. 2024-03-18T20:29:43.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.841+0000 7fde7692f700 1 Processor -- start 2024-03-18T20:29:43.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.841+0000 7fde7692f700 1 -- start start 2024-03-18T20:29:43.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.842+0000 7fde7692f700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde7006fc60 0x7fde7015db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:43.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.842+0000 7fde7692f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde7009a140 0x7fde7015ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:43.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.842+0000 7fde7692f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde700b4fc0 0x7fde700ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:43.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.842+0000 7fde7692f700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde70147080 con 0x7fde7009a140 2024-03-18T20:29:43.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.842+0000 7fde7692f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde700cc650 con 0x7fde700b4fc0 2024-03-18T20:29:43.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.842+0000 7fde7692f700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde700cba40 con 0x7fde7006fc60 2024-03-18T20:29:43.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.842+0000 7fde6f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde7009a140 0x7fde7015ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:43.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.842+0000 7fde6f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde7009a140 0x7fde7015ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45044/0 (socket says 172.21.15.1:45044) 2024-03-18T20:29:43.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.842+0000 7fde6f7fe700 1 -- 172.21.15.1:0/4072604410 learned_addr learned my addr 172.21.15.1:0/4072604410 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:43.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.842+0000 7fde74ecc700 1 --2- 172.21.15.1:0/4072604410 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde700b4fc0 0x7fde700ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:43.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.842+0000 7fde6ffff700 1 --2- 172.21.15.1:0/4072604410 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde7006fc60 0x7fde7015db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:43.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.842+0000 7fde6f7fe700 1 -- 172.21.15.1:0/4072604410 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde7006fc60 msgr2=0x7fde7015db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:43.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.842+0000 7fde6f7fe700 1 --2- 172.21.15.1:0/4072604410 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde7006fc60 0x7fde7015db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.842+0000 7fde6f7fe700 1 -- 172.21.15.1:0/4072604410 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde700b4fc0 msgr2=0x7fde700ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:43.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.842+0000 7fde6f7fe700 1 --2- 172.21.15.1:0/4072604410 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde700b4fc0 0x7fde700ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:43.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.842+0000 7fde6f7fe700 1 -- 172.21.15.1:0/4072604410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde640094a0 con 0x7fde7009a140 2024-03-18T20:29:43.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.842+0000 7fde6f7fe700 1 --2- 172.21.15.1:0/4072604410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde7009a140 0x7fde7015ef40 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7fde640090b0 tx=0x7fde64004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:43.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.843+0000 7fde6cff9700 1 -- 172.21.15.1:0/4072604410 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde640045d0 con 0x7fde7009a140 2024-03-18T20:29:43.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.843+0000 7fde6cff9700 1 -- 172.21.15.1:0/4072604410 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6401f070 con 0x7fde7009a140 2024-03-18T20:29:43.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.843+0000 7fde7692f700 1 -- 172.21.15.1:0/4072604410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde700cbd20 con 0x7fde7009a140 2024-03-18T20:29:43.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.843+0000 7fde6cff9700 1 -- 172.21.15.1:0/4072604410 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde640038a0 con 0x7fde7009a140 2024-03-18T20:29:43.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.843+0000 7fde7692f700 1 -- 172.21.15.1:0/4072604410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde700cc2e0 con 0x7fde7009a140 2024-03-18T20:29:43.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.844+0000 7fde7692f700 1 -- 172.21.15.1:0/4072604410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde70073be0 con 0x7fde7009a140 2024-03-18T20:29:43.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.845+0000 7fde6cff9700 1 -- 172.21.15.1:0/4072604410 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 175) v1 ==== 89404+0+0 (secure 0 0 0) 0x7fde64003a00 con 0x7fde7009a140 2024-03-18T20:29:43.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.845+0000 7fde6cff9700 1 --2- 172.21.15.1:0/4072604410 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fde5c06af60 0x7fde5c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:43.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.845+0000 7fde6ffff700 1 -- 172.21.15.1:0/4072604410 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fde5c06af60 msgr2=0x7fde5c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:43.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.845+0000 7fde6cff9700 1 -- 172.21.15.1:0/4072604410 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(291..291 src has 1..291) v4 ==== 6560+0+0 (secure 0 0 0) 0x7fde64021070 con 0x7fde7009a140 2024-03-18T20:29:43.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.845+0000 7fde6ffff700 1 --2- 172.21.15.1:0/4072604410 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fde5c06af60 0x7fde5c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:29:43.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:43.849+0000 7fde6cff9700 1 -- 172.21.15.1:0/4072604410 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fde64056190 con 0x7fde7009a140 2024-03-18T20:29:44.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.046+0000 7fde6ffff700 1 -- 172.21.15.1:0/4072604410 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fde5c06af60 msgr2=0x7fde5c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:44.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.046+0000 7fde6ffff700 1 --2- 172.21.15.1:0/4072604410 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fde5c06af60 0x7fde5c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:29:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.050+0000 7fde7692f700 1 -- 172.21.15.1:0/4072604410 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fde70067dc0 con 0x7fde7009a140 2024-03-18T20:29:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.051+0000 7fde6cff9700 1 -- 172.21.15.1:0/4072604410 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v291) v1 ==== 93+0+10 (secure 0 0 0) 0x7fde640150a0 con 0x7fde7009a140 2024-03-18T20:29:44.056 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:29:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.053+0000 7fde7692f700 1 -- 172.21.15.1:0/4072604410 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fde5c06af60 msgr2=0x7fde5c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:29:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.053+0000 7fde7692f700 1 --2- 172.21.15.1:0/4072604410 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fde5c06af60 0x7fde5c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.053+0000 7fde7692f700 1 -- 172.21.15.1:0/4072604410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde7009a140 msgr2=0x7fde7015ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.053+0000 7fde7692f700 1 --2- 172.21.15.1:0/4072604410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde7009a140 0x7fde7015ef40 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7fde640090b0 tx=0x7fde64004750 comp rx=0 tx=0).stop 2024-03-18T20:29:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.053+0000 7fde7692f700 1 -- 172.21.15.1:0/4072604410 shutdown_connections 2024-03-18T20:29:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.053+0000 7fde7692f700 1 --2- 172.21.15.1:0/4072604410 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fde5c06af60 0x7fde5c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.053+0000 7fde7692f700 1 --2- 172.21.15.1:0/4072604410 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fde7006fc60 0x7fde7015db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.053+0000 7fde7692f700 1 --2- 172.21.15.1:0/4072604410 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde700b4fc0 0x7fde700ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.053+0000 7fde7692f700 1 --2- 172.21.15.1:0/4072604410 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde7009a140 0x7fde7015ef40 unknown :-1 s=CLOSED pgs=3349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.053+0000 7fde7692f700 1 -- 172.21.15.1:0/4072604410 >> 172.21.15.1:0/4072604410 conn(0x7fde700b2de0 msgr2=0x7fde700af500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.053+0000 7fde7692f700 1 -- 172.21.15.1:0/4072604410 shutdown_connections 2024-03-18T20:29:44.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.053+0000 7fde7692f700 1 -- 172.21.15.1:0/4072604410 wait complete. 2024-03-18T20:29:44.071 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:29:44.142 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:44.138+0000 7f972cced240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:29:44.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.197+0000 7fa35f622700 1 Processor -- start 2024-03-18T20:29:44.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.197+0000 7fa35f622700 1 -- start start 2024-03-18T20:29:44.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.198+0000 7fa35f622700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3580d3f80 0x7fa35815d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:44.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.198+0000 7fa35f622700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3580cfe30 con 0x7fa35815df00 2024-03-18T20:29:44.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.198+0000 7fa35f622700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3580d0110 con 0x7fa35815d5e0 2024-03-18T20:29:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.198+0000 7fa35f622700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3580d24e0 con 0x7fa3580d3f80 2024-03-18T20:29:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.198+0000 7fa35dbbf700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa35815df00 0x7fa3580cdf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47950/0 (socket says 172.21.15.1:47950) 2024-03-18T20:29:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.198+0000 7fa35dbbf700 1 -- 172.21.15.1:0/2788846220 learned_addr learned my addr 172.21.15.1:0/2788846220 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:29:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.198+0000 7fa35cbbd700 1 --2- 172.21.15.1:0/2788846220 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3580d3f80 0x7fa35815d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.198+0000 7fa34ffff700 1 -- 172.21.15.1:0/2788846220 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583603052 0 0) 0x7fa3580cfe30 con 0x7fa35815df00 2024-03-18T20:29:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.198+0000 7fa34ffff700 1 -- 172.21.15.1:0/2788846220 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3440035b0 con 0x7fa35815df00 2024-03-18T20:29:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.198+0000 7fa34ffff700 1 -- 172.21.15.1:0/2788846220 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464845745 0 0) 0x7fa3580d0110 con 0x7fa35815d5e0 2024-03-18T20:29:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.198+0000 7fa34ffff700 1 -- 172.21.15.1:0/2788846220 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3580cfe30 con 0x7fa35815d5e0 2024-03-18T20:29:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.198+0000 7fa35cbbd700 1 -- 172.21.15.1:0/2788846220 >> v1:172.21.15.165:6789/0 conn(0x7fa35815d5e0 legacy=0x7fa3580d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.198+0000 7fa35cbbd700 1 -- 172.21.15.1:0/2788846220 >> v1:172.21.15.1:6789/0 conn(0x7fa35815df00 legacy=0x7fa3580cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.198+0000 7fa35cbbd700 1 -- 172.21.15.1:0/2788846220 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3580d03f0 con 0x7fa3580d3f80 2024-03-18T20:29:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.198+0000 7fa35cbbd700 1 --2- 172.21.15.1:0/2788846220 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3580d3f80 0x7fa35815d9c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa348009870 tx=0x7fa348009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66326466d3f32a33 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:44.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.199+0000 7fa34ffff700 1 -- 172.21.15.1:0/2788846220 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa348011830 con 0x7fa3580d3f80 2024-03-18T20:29:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.199+0000 7fa34ffff700 1 -- 172.21.15.1:0/2788846220 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa348011990 con 0x7fa3580d3f80 2024-03-18T20:29:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.199+0000 7fa34ffff700 1 -- 172.21.15.1:0/2788846220 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa348011d20 con 0x7fa3580d3f80 2024-03-18T20:29:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.199+0000 7fa35f622700 1 -- 172.21.15.1:0/2788846220 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3580d3f80 msgr2=0x7fa35815d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.199+0000 7fa35f622700 1 --2- 172.21.15.1:0/2788846220 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3580d3f80 0x7fa35815d9c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa348009870 tx=0x7fa348009b80 comp rx=0 tx=0).stop 2024-03-18T20:29:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.199+0000 7fa35f622700 1 -- 172.21.15.1:0/2788846220 shutdown_connections 2024-03-18T20:29:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.199+0000 7fa35f622700 1 --2- 172.21.15.1:0/2788846220 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3580d3f80 0x7fa35815d9c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.199+0000 7fa35f622700 1 -- 172.21.15.1:0/2788846220 >> 172.21.15.1:0/2788846220 conn(0x7fa3580d3380 msgr2=0x7fa3580d3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.199+0000 7fa35f622700 1 -- 172.21.15.1:0/2788846220 shutdown_connections 2024-03-18T20:29:44.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.199+0000 7fa35f622700 1 -- 172.21.15.1:0/2788846220 wait complete. 2024-03-18T20:29:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.201+0000 7fa35f622700 1 Processor -- start 2024-03-18T20:29:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.201+0000 7fa35f622700 1 -- start start 2024-03-18T20:29:44.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.201+0000 7fa35f622700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3580d3f80 0x7fa3580cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.201+0000 7fa35f622700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa35815d5e0 0x7fa3580c6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.201+0000 7fa35f622700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa35815df00 0x7fa3580c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.201+0000 7fa35f622700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3580cbb30 con 0x7fa3580d3f80 2024-03-18T20:29:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.201+0000 7fa35f622700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3580cbe10 con 0x7fa35815d5e0 2024-03-18T20:29:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.201+0000 7fa35f622700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3580cc0f0 con 0x7fa35815df00 2024-03-18T20:29:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.201+0000 7fa35d3be700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3580d3f80 0x7fa3580cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.202+0000 7fa35d3be700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3580d3f80 0x7fa3580cb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45060/0 (socket says 172.21.15.1:45060) 2024-03-18T20:29:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.202+0000 7fa35d3be700 1 -- 172.21.15.1:0/2275145750 learned_addr learned my addr 172.21.15.1:0/2275145750 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.202+0000 7fa35dbbf700 1 --2- 172.21.15.1:0/2275145750 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa35815df00 0x7fa3580c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.202+0000 7fa35cbbd700 1 --2- 172.21.15.1:0/2275145750 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa35815d5e0 0x7fa3580c6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.202+0000 7fa35d3be700 1 -- 172.21.15.1:0/2275145750 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa35815df00 msgr2=0x7fa3580c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.202+0000 7fa35d3be700 1 --2- 172.21.15.1:0/2275145750 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa35815df00 0x7fa3580c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.202+0000 7fa35d3be700 1 -- 172.21.15.1:0/2275145750 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa35815d5e0 msgr2=0x7fa3580c6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.202+0000 7fa35d3be700 1 --2- 172.21.15.1:0/2275145750 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa35815d5e0 0x7fa3580c6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.202+0000 7fa35d3be700 1 -- 172.21.15.1:0/2275145750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3480094a0 con 0x7fa3580d3f80 2024-03-18T20:29:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.202+0000 7fa35d3be700 1 --2- 172.21.15.1:0/2275145750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3580d3f80 0x7fa3580cb4a0 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 crypto rx=0x7fa35400c520 tx=0x7fa35400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.202+0000 7fa34dffb700 1 -- 172.21.15.1:0/2275145750 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3540052d0 con 0x7fa3580d3f80 2024-03-18T20:29:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.202+0000 7fa34dffb700 1 -- 172.21.15.1:0/2275145750 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa354005430 con 0x7fa3580d3f80 2024-03-18T20:29:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.202+0000 7fa35f622700 1 -- 172.21.15.1:0/2275145750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3580cc3d0 con 0x7fa3580d3f80 2024-03-18T20:29:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.202+0000 7fa34dffb700 1 -- 172.21.15.1:0/2275145750 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3540038c0 con 0x7fa3580d3f80 2024-03-18T20:29:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.202+0000 7fa35f622700 1 -- 172.21.15.1:0/2275145750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3580ccbd0 con 0x7fa3580d3f80 2024-03-18T20:29:44.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.203+0000 7fa35f622700 1 -- 172.21.15.1:0/2275145750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3580cc8f0 con 0x7fa3580d3f80 2024-03-18T20:29:44.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.205+0000 7fa34dffb700 1 -- 172.21.15.1:0/2275145750 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 175) v1 ==== 89404+0+0 (secure 0 0 0) 0x7fa354003a20 con 0x7fa3580d3f80 2024-03-18T20:29:44.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.205+0000 7fa34dffb700 1 --2- 172.21.15.1:0/2275145750 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa33c06af60 0x7fa33c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:44.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.205+0000 7fa35cbbd700 1 -- 172.21.15.1:0/2275145750 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa33c06af60 msgr2=0x7fa33c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:44.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.205+0000 7fa34dffb700 1 -- 172.21.15.1:0/2275145750 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(291..291 src has 1..291) v4 ==== 6560+0+0 (secure 0 0 0) 0x7fa3540890c0 con 0x7fa3580d3f80 2024-03-18T20:29:44.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.205+0000 7fa35cbbd700 1 --2- 172.21.15.1:0/2275145750 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa33c06af60 0x7fa33c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:29:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.209+0000 7fa34dffb700 1 -- 172.21.15.1:0/2275145750 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa354053d00 con 0x7fa3580d3f80 2024-03-18T20:29:44.222 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:44.218+0000 7f97503aa240 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T20:29:44.353 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:44.349+0000 7f972cced240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:29:44.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.406+0000 7fa35cbbd700 1 -- 172.21.15.1:0/2275145750 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa33c06af60 msgr2=0x7fa33c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:44.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.406+0000 7fa35cbbd700 1 --2- 172.21.15.1:0/2275145750 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa33c06af60 0x7fa33c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:29:44.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.438+0000 7fa35f622700 1 -- 172.21.15.1:0/2275145750 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa3580cc8f0 con 0x7fa3580d3f80 2024-03-18T20:29:44.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.438+0000 7fa34dffb700 1 -- 172.21.15.1:0/2275145750 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v291) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa3580cc8f0 con 0x7fa3580d3f80 2024-03-18T20:29:44.443 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:29:44.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.441+0000 7fa35f622700 1 -- 172.21.15.1:0/2275145750 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa33c06af60 msgr2=0x7fa33c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:29:44.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.441+0000 7fa35f622700 1 --2- 172.21.15.1:0/2275145750 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa33c06af60 0x7fa33c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.441+0000 7fa35f622700 1 -- 172.21.15.1:0/2275145750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3580d3f80 msgr2=0x7fa3580cb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:44.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.441+0000 7fa35f622700 1 --2- 172.21.15.1:0/2275145750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3580d3f80 0x7fa3580cb4a0 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 crypto rx=0x7fa35400c520 tx=0x7fa35400c830 comp rx=0 tx=0).stop 2024-03-18T20:29:44.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.441+0000 7fa35f622700 1 -- 172.21.15.1:0/2275145750 shutdown_connections 2024-03-18T20:29:44.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.441+0000 7fa35f622700 1 --2- 172.21.15.1:0/2275145750 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa33c06af60 0x7fa33c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.441+0000 7fa35f622700 1 --2- 172.21.15.1:0/2275145750 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa35815df00 0x7fa3580c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.441+0000 7fa35f622700 1 --2- 172.21.15.1:0/2275145750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa3580d3f80 0x7fa3580cb4a0 unknown :-1 s=CLOSED pgs=3351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.441+0000 7fa35f622700 1 --2- 172.21.15.1:0/2275145750 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa35815d5e0 0x7fa3580c6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.441+0000 7fa35f622700 1 -- 172.21.15.1:0/2275145750 >> 172.21.15.1:0/2275145750 conn(0x7fa3580d3380 msgr2=0x7fa3580b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:44.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.441+0000 7fa35f622700 1 -- 172.21.15.1:0/2275145750 shutdown_connections 2024-03-18T20:29:44.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.441+0000 7fa35f622700 1 -- 172.21.15.1:0/2275145750 wait complete. 2024-03-18T20:29:44.458 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:29:44.459 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:44.454+0000 7f97503aa240 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T20:29:44.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.583+0000 7f3ecb550700 1 Processor -- start 2024-03-18T20:29:44.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.583+0000 7f3ecb550700 1 -- start start 2024-03-18T20:29:44.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.584+0000 7f3ecb550700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3ec40cf200 0x7f3ec415c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:44.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.584+0000 7f3ecb550700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3ec415b6c0 0x7f3ec40cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:44.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.584+0000 7f3ecb550700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec415ca40 con 0x7f3ec40cbc90 2024-03-18T20:29:44.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.584+0000 7f3ecb550700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec40cce30 con 0x7f3ec415b6c0 2024-03-18T20:29:44.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.584+0000 7f3ecb550700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec40cfb20 con 0x7f3ec40cf200 2024-03-18T20:29:44.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.584+0000 7f3ec8aeb700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3ec415b6c0 0x7f3ec40cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:44.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.584+0000 7f3ec9aed700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3ec40cbc90 0x7f3ec415eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43940/0 (socket says 172.21.15.1:43940) 2024-03-18T20:29:44.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.584+0000 7f3ec9aed700 1 -- 172.21.15.1:0/1634363523 learned_addr learned my addr 172.21.15.1:0/1634363523 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:29:44.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.584+0000 7f3ec92ec700 1 --2- 172.21.15.1:0/1634363523 >> v2:172.21.15.165:3300/0 conn(0x7f3ec40cf200 0x7f3ec415c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:44.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.585+0000 7f3ebbfff700 1 -- 172.21.15.1:0/1634363523 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103879895 0 0) 0x7f3ebc001490 con 0x7f3ec40cbc90 2024-03-18T20:29:44.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.585+0000 7f3ec8aeb700 1 -- 172.21.15.1:0/1634363523 >> v2:172.21.15.165:3300/0 conn(0x7f3ec40cf200 msgr2=0x7f3ec415c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:44.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.585+0000 7f3ec8aeb700 1 --2- 172.21.15.1:0/1634363523 >> v2:172.21.15.165:3300/0 conn(0x7f3ec40cf200 0x7f3ec415c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.585+0000 7f3ec8aeb700 1 -- 172.21.15.1:0/1634363523 >> v1:172.21.15.1:6789/0 conn(0x7f3ec40cbc90 legacy=0x7f3ec415eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:44.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.585+0000 7f3ec8aeb700 1 -- 172.21.15.1:0/1634363523 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec40cc220 con 0x7f3ec415b6c0 2024-03-18T20:29:44.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.585+0000 7f3ec8aeb700 1 --2- 172.21.15.1:0/1634363523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec415b6c0 0x7f3ec40cf5e0 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7f3ec406ea00 tx=0x7f3eb4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a9bff7ea4fcbc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:44.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.585+0000 7f3ebbfff700 1 -- 172.21.15.1:0/1634363523 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4011830 con 0x7f3ec415b6c0 2024-03-18T20:29:44.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.585+0000 7f3ebbfff700 1 -- 172.21.15.1:0/1634363523 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb4011990 con 0x7f3ec415b6c0 2024-03-18T20:29:44.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.585+0000 7f3ebbfff700 1 -- 172.21.15.1:0/1634363523 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4011d20 con 0x7f3ec415b6c0 2024-03-18T20:29:44.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.586+0000 7f3ecb550700 1 -- 172.21.15.1:0/1634363523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec415b6c0 msgr2=0x7f3ec40cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:44.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.586+0000 7f3ecb550700 1 --2- 172.21.15.1:0/1634363523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec415b6c0 0x7f3ec40cf5e0 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7f3ec406ea00 tx=0x7f3eb4009b20 comp rx=0 tx=0).stop 2024-03-18T20:29:44.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.586+0000 7f3ecb550700 1 -- 172.21.15.1:0/1634363523 shutdown_connections 2024-03-18T20:29:44.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.586+0000 7f3ecb550700 1 --2- 172.21.15.1:0/1634363523 >> v2:172.21.15.165:3300/0 conn(0x7f3ec40cf200 0x7f3ec415c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.586+0000 7f3ecb550700 1 --2- 172.21.15.1:0/1634363523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec415b6c0 0x7f3ec40cf5e0 unknown :-1 s=CLOSED pgs=3353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.586+0000 7f3ecb550700 1 -- 172.21.15.1:0/1634363523 >> 172.21.15.1:0/1634363523 conn(0x7f3ec40d1180 msgr2=0x7f3ec40d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:44.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.586+0000 7f3ecb550700 1 -- 172.21.15.1:0/1634363523 shutdown_connections 2024-03-18T20:29:44.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.586+0000 7f3ecb550700 1 -- 172.21.15.1:0/1634363523 wait complete. 2024-03-18T20:29:44.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.587+0000 7f3ecb550700 1 Processor -- start 2024-03-18T20:29:44.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.587+0000 7f3ecb550700 1 -- start start 2024-03-18T20:29:44.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.588+0000 7f3ecb550700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ec40cbc90 0x7f3ec40c8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:44.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.588+0000 7f3ecb550700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec40cf200 0x7f3ec41569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:44.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.588+0000 7f3ecb550700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec415b6c0 0x7f3ec4156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:44.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.588+0000 7f3ecb550700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec40c8ba0 con 0x7f3ec40cf200 2024-03-18T20:29:44.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.588+0000 7f3ecb550700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec40b5870 con 0x7f3ec415b6c0 2024-03-18T20:29:44.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.588+0000 7f3ecb550700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec40c7bb0 con 0x7f3ec40cbc90 2024-03-18T20:29:44.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.588+0000 7f3ec8aeb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec40cf200 0x7f3ec41569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:44.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.588+0000 7f3ec8aeb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec40cf200 0x7f3ec41569f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59988/0 (socket says 172.21.15.1:59988) 2024-03-18T20:29:44.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.588+0000 7f3ec8aeb700 1 -- 172.21.15.1:0/243543191 learned_addr learned my addr 172.21.15.1:0/243543191 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:44.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.588+0000 7f3ec92ec700 1 --2- 172.21.15.1:0/243543191 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ec40cbc90 0x7f3ec40c8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:44.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.588+0000 7f3ec9aed700 1 --2- 172.21.15.1:0/243543191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec415b6c0 0x7f3ec4156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:44.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.588+0000 7f3ec8aeb700 1 -- 172.21.15.1:0/243543191 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ec40cbc90 msgr2=0x7f3ec40c8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:44.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.588+0000 7f3ec8aeb700 1 --2- 172.21.15.1:0/243543191 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ec40cbc90 0x7f3ec40c8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.588+0000 7f3ec8aeb700 1 -- 172.21.15.1:0/243543191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec415b6c0 msgr2=0x7f3ec4156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:44.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.588+0000 7f3ec8aeb700 1 --2- 172.21.15.1:0/243543191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec415b6c0 0x7f3ec4156170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.588+0000 7f3ec8aeb700 1 -- 172.21.15.1:0/243543191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb40094a0 con 0x7f3ec40cf200 2024-03-18T20:29:44.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.588+0000 7f3ec8aeb700 1 --2- 172.21.15.1:0/243543191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec40cf200 0x7f3ec41569f0 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4013000 tx=0x7f3eb4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:44.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.589+0000 7f3eb9ffb700 1 -- 172.21.15.1:0/243543191 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb40036a0 con 0x7f3ec40cf200 2024-03-18T20:29:44.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.589+0000 7f3eb9ffb700 1 -- 172.21.15.1:0/243543191 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb401f070 con 0x7f3ec40cf200 2024-03-18T20:29:44.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.589+0000 7f3ecb550700 1 -- 172.21.15.1:0/243543191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec4156f30 con 0x7f3ec40cf200 2024-03-18T20:29:44.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.589+0000 7f3eb9ffb700 1 -- 172.21.15.1:0/243543191 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4003a60 con 0x7f3ec40cf200 2024-03-18T20:29:44.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.589+0000 7f3ecb550700 1 -- 172.21.15.1:0/243543191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec40b5a80 con 0x7f3ec40cf200 2024-03-18T20:29:44.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.590+0000 7f3ecb550700 1 -- 172.21.15.1:0/243543191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec40d0ca0 con 0x7f3ec40cf200 2024-03-18T20:29:44.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.590+0000 7f3eb9ffb700 1 -- 172.21.15.1:0/243543191 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 175) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f3eb4003bc0 con 0x7f3ec40cf200 2024-03-18T20:29:44.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.591+0000 7f3eb9ffb700 1 --2- 172.21.15.1:0/243543191 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3ea806af60 0x7f3ea806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:44.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.591+0000 7f3ec92ec700 1 -- 172.21.15.1:0/243543191 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3ea806af60 msgr2=0x7f3ea806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:44.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.591+0000 7f3eb9ffb700 1 -- 172.21.15.1:0/243543191 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(291..291 src has 1..291) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f3eb4021070 con 0x7f3ec40cf200 2024-03-18T20:29:44.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.591+0000 7f3ec92ec700 1 --2- 172.21.15.1:0/243543191 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3ea806af60 0x7f3ea806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:29:44.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.595+0000 7f3eb9ffb700 1 -- 172.21.15.1:0/243543191 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3eb4055d50 con 0x7f3ec40cf200 2024-03-18T20:29:44.725 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:44.721+0000 7f972cced240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:29:44.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.792+0000 7f3ec92ec700 1 -- 172.21.15.1:0/243543191 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3ea806af60 msgr2=0x7f3ea806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:44.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.792+0000 7f3ec92ec700 1 --2- 172.21.15.1:0/243543191 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3ea806af60 0x7f3ea806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:29:44.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.799+0000 7f3ecb550700 1 -- 172.21.15.1:0/243543191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3ec4067dc0 con 0x7f3ec40cf200 2024-03-18T20:29:44.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.800+0000 7f3eb9ffb700 1 -- 172.21.15.1:0/243543191 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v291) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3eb4059370 con 0x7f3ec40cf200 2024-03-18T20:29:44.805 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:29:44.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.802+0000 7f3ecb550700 1 -- 172.21.15.1:0/243543191 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3ea806af60 msgr2=0x7f3ea806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:29:44.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.802+0000 7f3ecb550700 1 --2- 172.21.15.1:0/243543191 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3ea806af60 0x7f3ea806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.802+0000 7f3ecb550700 1 -- 172.21.15.1:0/243543191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec40cf200 msgr2=0x7f3ec41569f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:44.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.802+0000 7f3ecb550700 1 --2- 172.21.15.1:0/243543191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec40cf200 0x7f3ec41569f0 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4013000 tx=0x7f3eb4004750 comp rx=0 tx=0).stop 2024-03-18T20:29:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.802+0000 7f3ecb550700 1 -- 172.21.15.1:0/243543191 shutdown_connections 2024-03-18T20:29:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.802+0000 7f3ecb550700 1 --2- 172.21.15.1:0/243543191 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f3ea806af60 0x7f3ea806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.802+0000 7f3ecb550700 1 --2- 172.21.15.1:0/243543191 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f3ec40cbc90 0x7f3ec40c8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.802+0000 7f3ecb550700 1 --2- 172.21.15.1:0/243543191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec415b6c0 0x7f3ec4156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.802+0000 7f3ecb550700 1 --2- 172.21.15.1:0/243543191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ec40cf200 0x7f3ec41569f0 unknown :-1 s=CLOSED pgs=3354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.802+0000 7f3ecb550700 1 -- 172.21.15.1:0/243543191 >> 172.21.15.1:0/243543191 conn(0x7f3ec40d1180 msgr2=0x7f3ec40b1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.802+0000 7f3ecb550700 1 -- 172.21.15.1:0/243543191 shutdown_connections 2024-03-18T20:29:44.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.802+0000 7f3ecb550700 1 -- 172.21.15.1:0/243543191 wait complete. 2024-03-18T20:29:44.820 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T20:29:44.859 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:44.854+0000 7f97503aa240 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T20:29:44.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.942+0000 7f10aa486700 1 Processor -- start 2024-03-18T20:29:44.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.942+0000 7f10aa486700 1 -- start start 2024-03-18T20:29:44.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.943+0000 7f10aa486700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f10a40cf200 0x7f10a415c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:44.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.943+0000 7f10aa486700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10a40cbc90 0x7f10a4099ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:44.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.943+0000 7f10aa486700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a415ca40 con 0x7f10a415b6c0 2024-03-18T20:29:44.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.943+0000 7f10aa486700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a40cce30 con 0x7f10a40cf200 2024-03-18T20:29:44.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.943+0000 7f10aa486700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a40cc220 con 0x7f10a40cbc90 2024-03-18T20:29:44.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.943+0000 7f10a3fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f10a40cf200 0x7f10a415c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:44.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.943+0000 7f10a37fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f10a415b6c0 0x7f10a40cf5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43952/0 (socket says 172.21.15.1:43952) 2024-03-18T20:29:44.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.943+0000 7f10a37fe700 1 -- 172.21.15.1:0/1159108526 learned_addr learned my addr 172.21.15.1:0/1159108526 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:29:44.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.943+0000 7f10a8a23700 1 --2- 172.21.15.1:0/1159108526 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10a40cbc90 0x7f10a4099ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:44.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.943+0000 7f10a3fff700 1 -- 172.21.15.1:0/1159108526 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10a40cbc90 msgr2=0x7f10a4099ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.943+0000 7f10a3fff700 1 --2- 172.21.15.1:0/1159108526 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10a40cbc90 0x7f10a4099ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.943+0000 7f10a3fff700 1 -- 172.21.15.1:0/1159108526 >> v1:172.21.15.1:6789/0 conn(0x7f10a415b6c0 legacy=0x7f10a40cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.944+0000 7f10a3fff700 1 -- 172.21.15.1:0/1159108526 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a40cc500 con 0x7f10a40cf200 2024-03-18T20:29:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.944+0000 7f10a2ffd700 1 -- 172.21.15.1:0/1159108526 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198890405 0 0) 0x7f1098001110 con 0x7f10a415b6c0 2024-03-18T20:29:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.944+0000 7f10a3fff700 1 --2- 172.21.15.1:0/1159108526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10a40cf200 0x7f10a415c500 secure :-1 s=READY pgs=3356 cs=0 l=1 rev1=1 crypto rx=0x7f1094009840 tx=0x7f1094009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daf5f796a8cf47bd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.944+0000 7f10a2ffd700 1 -- 172.21.15.1:0/1159108526 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1094011830 con 0x7f10a40cf200 2024-03-18T20:29:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.944+0000 7f10a2ffd700 1 -- 172.21.15.1:0/1159108526 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1094011990 con 0x7f10a40cf200 2024-03-18T20:29:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.944+0000 7f10a2ffd700 1 -- 172.21.15.1:0/1159108526 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1094011d20 con 0x7f10a40cf200 2024-03-18T20:29:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.944+0000 7f10aa486700 1 -- 172.21.15.1:0/1159108526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10a40cf200 msgr2=0x7f10a415c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.944+0000 7f10aa486700 1 --2- 172.21.15.1:0/1159108526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10a40cf200 0x7f10a415c500 secure :-1 s=READY pgs=3356 cs=0 l=1 rev1=1 crypto rx=0x7f1094009840 tx=0x7f1094009b50 comp rx=0 tx=0).stop 2024-03-18T20:29:44.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.944+0000 7f10aa486700 1 -- 172.21.15.1:0/1159108526 shutdown_connections 2024-03-18T20:29:44.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.944+0000 7f10aa486700 1 --2- 172.21.15.1:0/1159108526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10a40cf200 0x7f10a415c500 unknown :-1 s=CLOSED pgs=3356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.944+0000 7f10aa486700 1 --2- 172.21.15.1:0/1159108526 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10a40cbc90 0x7f10a4099ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.944+0000 7f10aa486700 1 -- 172.21.15.1:0/1159108526 >> 172.21.15.1:0/1159108526 conn(0x7f10a40d1180 msgr2=0x7f10a40d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:44.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.945+0000 7f10aa486700 1 -- 172.21.15.1:0/1159108526 shutdown_connections 2024-03-18T20:29:44.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.945+0000 7f10aa486700 1 -- 172.21.15.1:0/1159108526 wait complete. 2024-03-18T20:29:44.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.946+0000 7f10aa486700 1 Processor -- start 2024-03-18T20:29:44.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.946+0000 7f10aa486700 1 -- start start 2024-03-18T20:29:44.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10aa486700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10a40cbc90 0x7f10a4152610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:44.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10aa486700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10a40cf200 0x7f10a40af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:44.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10aa486700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10a415b6c0 0x7f10a40af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:44.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10aa486700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a40b6070 con 0x7f10a40cbc90 2024-03-18T20:29:44.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10aa486700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a4157f90 con 0x7f10a40cf200 2024-03-18T20:29:44.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10aa486700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a4157380 con 0x7f10a415b6c0 2024-03-18T20:29:44.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10a3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10a40cbc90 0x7f10a4152610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:44.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10a3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10a40cbc90 0x7f10a4152610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60008/0 (socket says 172.21.15.1:60008) 2024-03-18T20:29:44.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10a3fff700 1 -- 172.21.15.1:0/1532461769 learned_addr learned my addr 172.21.15.1:0/1532461769 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:44.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10a8a23700 1 --2- 172.21.15.1:0/1532461769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10a415b6c0 0x7f10a40af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:44.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10a37fe700 1 --2- 172.21.15.1:0/1532461769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10a40cf200 0x7f10a40af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:44.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10a3fff700 1 -- 172.21.15.1:0/1532461769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10a415b6c0 msgr2=0x7f10a40af5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:44.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10a3fff700 1 --2- 172.21.15.1:0/1532461769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10a415b6c0 0x7f10a40af5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10a3fff700 1 -- 172.21.15.1:0/1532461769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10a40cf200 msgr2=0x7f10a40af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10a3fff700 1 --2- 172.21.15.1:0/1532461769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10a40cf200 0x7f10a40af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10a3fff700 1 -- 172.21.15.1:0/1532461769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10940094a0 con 0x7f10a40cbc90 2024-03-18T20:29:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10a3fff700 1 --2- 172.21.15.1:0/1532461769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10a40cbc90 0x7f10a4152610 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7f1094000c00 tx=0x7f1094004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10a0ff9700 1 -- 172.21.15.1:0/1532461769 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10940045d0 con 0x7f10a40cbc90 2024-03-18T20:29:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10a0ff9700 1 -- 172.21.15.1:0/1532461769 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109401f070 con 0x7f10a40cbc90 2024-03-18T20:29:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10aa486700 1 -- 172.21.15.1:0/1532461769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a4157660 con 0x7f10a40cbc90 2024-03-18T20:29:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10a0ff9700 1 -- 172.21.15.1:0/1532461769 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10940038e0 con 0x7f10a40cbc90 2024-03-18T20:29:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.947+0000 7f10aa486700 1 -- 172.21.15.1:0/1532461769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a4157c20 con 0x7f10a40cbc90 2024-03-18T20:29:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.948+0000 7f10aa486700 1 -- 172.21.15.1:0/1532461769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a40d0ca0 con 0x7f10a40cbc90 2024-03-18T20:29:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.949+0000 7f10a0ff9700 1 -- 172.21.15.1:0/1532461769 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 175) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f1094003a40 con 0x7f10a40cbc90 2024-03-18T20:29:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.949+0000 7f10a0ff9700 1 --2- 172.21.15.1:0/1532461769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f109006af60 0x7f109006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.949+0000 7f10a37fe700 1 -- 172.21.15.1:0/1532461769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f109006af60 msgr2=0x7f109006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.949+0000 7f10a0ff9700 1 -- 172.21.15.1:0/1532461769 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(291..291 src has 1..291) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f1094021070 con 0x7f10a40cbc90 2024-03-18T20:29:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.949+0000 7f10a37fe700 1 --2- 172.21.15.1:0/1532461769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f109006af60 0x7f109006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:29:44.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:44.954+0000 7f10a0ff9700 1 -- 172.21.15.1:0/1532461769 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1094055f10 con 0x7f10a40cbc90 2024-03-18T20:29:45.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.150+0000 7f10a37fe700 1 -- 172.21.15.1:0/1532461769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f109006af60 msgr2=0x7f109006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:45.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.150+0000 7f10a37fe700 1 --2- 172.21.15.1:0/1532461769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f109006af60 0x7f109006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:29:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.210+0000 7f10aa486700 1 -- 172.21.15.1:0/1532461769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f10a40cff40 con 0x7f10a40cbc90 2024-03-18T20:29:45.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.210+0000 7f10a0ff9700 1 -- 172.21.15.1:0/1532461769 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v291) v1 ==== 80+0+452 (secure 0 0 0) 0x7f1094059530 con 0x7f10a40cbc90 2024-03-18T20:29:45.215 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1650741459 2024-03-19T20:28:26.019720+0000 2024-03-18T20:29:45.215 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2451665032 2024-03-19T20:28:26.019720+0000 2024-03-18T20:29:45.215 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6833/148356 2024-03-19T20:28:26.019720+0000 2024-03-18T20:29:45.216 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6832/148356 2024-03-19T20:28:26.019720+0000 2024-03-18T20:29:45.216 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1408878463 2024-03-19T20:28:26.019720+0000 2024-03-18T20:29:45.216 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/582749307 2024-03-19T20:28:26.019720+0000 2024-03-18T20:29:45.216 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/997400106 2024-03-19T20:28:26.019720+0000 2024-03-18T20:29:45.216 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2561447959 2024-03-19T20:28:26.019720+0000 2024-03-18T20:29:45.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.212+0000 7f10aa486700 1 -- 172.21.15.1:0/1532461769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f109006af60 msgr2=0x7f109006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:29:45.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.212+0000 7f10aa486700 1 --2- 172.21.15.1:0/1532461769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f109006af60 0x7f109006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:45.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.212+0000 7f10aa486700 1 -- 172.21.15.1:0/1532461769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10a40cbc90 msgr2=0x7f10a4152610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:45.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.212+0000 7f10aa486700 1 --2- 172.21.15.1:0/1532461769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10a40cbc90 0x7f10a4152610 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7f1094000c00 tx=0x7f1094004750 comp rx=0 tx=0).stop 2024-03-18T20:29:45.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.212+0000 7f10aa486700 1 -- 172.21.15.1:0/1532461769 shutdown_connections 2024-03-18T20:29:45.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.212+0000 7f10aa486700 1 --2- 172.21.15.1:0/1532461769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f109006af60 0x7f109006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:45.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.212+0000 7f10aa486700 1 --2- 172.21.15.1:0/1532461769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f10a415b6c0 0x7f10a40af5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:45.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.212+0000 7f10aa486700 1 --2- 172.21.15.1:0/1532461769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10a40cbc90 0x7f10a4152610 unknown :-1 s=CLOSED pgs=3357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:45.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.212+0000 7f10aa486700 1 --2- 172.21.15.1:0/1532461769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10a40cf200 0x7f10a40af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:45.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.212+0000 7f10aa486700 1 -- 172.21.15.1:0/1532461769 >> 172.21.15.1:0/1532461769 conn(0x7f10a40d1180 msgr2=0x7f10a4156ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:45.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.213+0000 7f10aa486700 1 -- 172.21.15.1:0/1532461769 shutdown_connections 2024-03-18T20:29:45.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.213+0000 7f10aa486700 1 -- 172.21.15.1:0/1532461769 wait complete. 2024-03-18T20:29:45.219 INFO:teuthology.orchestra.run.smithi001.stderr:listed 8 entries 2024-03-18T20:29:45.230 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-03-18T20:29:45.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:29:45.230 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T20:29:45.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[175864]: fuse finished with error 0 and tester_r 0 2024-03-18T20:29:45.312 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:45.308+0000 7f972cced240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:29:45.313 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:29:45.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:29:45.313 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T20:29:45.384 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:45.381+0000 7f972cced240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:29:45.409 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:29:45.409 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:29:45.428 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:29:45.428 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:29:45.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:29:45.428 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-18T20:29:45.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:29:45.486 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-18T20:29:45.486 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:29:45.486 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:29:45.505 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:29:45.505 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:29:45.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:29:45.505 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-18T20:29:45.520 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:45.516+0000 7f972cced240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:29:45.520 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:45.515+0000 7f97503aa240 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T20:29:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:29:45.538 INFO:teuthology.orchestra.run.smithi001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-18T20:29:45.538 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:29:45.538 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:29:45.557 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:29:45.557 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:29:45.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:29:45.557 DEBUG:teuthology.orchestra.run.smithi165:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-18T20:29:45.598 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:45.593+0000 7f97503aa240 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T20:29:45.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:29:45.616 INFO:teuthology.orchestra.run.smithi165.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-18T20:29:45.616 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:29:45.705 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:45.701+0000 7f972cced240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:29:45.738 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:45.733+0000 7f97503aa240 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T20:29:45.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.738+0000 7f24ac845700 1 Processor -- start 2024-03-18T20:29:45.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.738+0000 7f24ac845700 1 -- start start 2024-03-18T20:29:45.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.739+0000 7f24ac845700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f24a80cf200 0x7f24a815c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:45.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.739+0000 7f24ac845700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f24a815b6c0 0x7f24a80cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:45.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.739+0000 7f24ac845700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a815ca40 con 0x7f24a80cbc90 2024-03-18T20:29:45.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.739+0000 7f24ac845700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a80cce30 con 0x7f24a815b6c0 2024-03-18T20:29:45.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.739+0000 7f24ac845700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a80cfb20 con 0x7f24a80cf200 2024-03-18T20:29:45.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.739+0000 7f24a5d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f24a815b6c0 0x7f24a80cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:45.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.739+0000 7f24a5d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f24a815b6c0 0x7f24a80cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60014/0 (socket says 172.21.15.1:60014) 2024-03-18T20:29:45.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.739+0000 7f24a5d9b700 1 -- 172.21.15.1:0/3502492356 learned_addr learned my addr 172.21.15.1:0/3502492356 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:45.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.739+0000 7f24a659c700 1 --2- 172.21.15.1:0/3502492356 >> v2:172.21.15.165:3300/0 conn(0x7f24a80cf200 0x7f24a815c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:45.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.739+0000 7f24a5d9b700 1 -- 172.21.15.1:0/3502492356 >> v2:172.21.15.165:3300/0 conn(0x7f24a80cf200 msgr2=0x7f24a815c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:45.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.739+0000 7f24a5d9b700 1 --2- 172.21.15.1:0/3502492356 >> v2:172.21.15.165:3300/0 conn(0x7f24a80cf200 0x7f24a815c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:45.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.739+0000 7f24a5d9b700 1 -- 172.21.15.1:0/3502492356 >> v1:172.21.15.165:6789/0 conn(0x7f24a80cbc90 legacy=0x7f24a815eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:45.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.739+0000 7f24a5d9b700 1 -- 172.21.15.1:0/3502492356 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a80cc220 con 0x7f24a815b6c0 2024-03-18T20:29:45.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.739+0000 7f24a5d9b700 1 --2- 172.21.15.1:0/3502492356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a815b6c0 0x7f24a80cf5e0 secure :-1 s=READY pgs=3358 cs=0 l=1 rev1=1 crypto rx=0x7f24a806ea00 tx=0x7f249c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee63fdce1405412 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:45.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.739+0000 7f24a559a700 1 -- 172.21.15.1:0/3502492356 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c011830 con 0x7f24a815b6c0 2024-03-18T20:29:45.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.739+0000 7f24a559a700 1 -- 172.21.15.1:0/3502492356 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249c011990 con 0x7f24a815b6c0 2024-03-18T20:29:45.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.739+0000 7f24a559a700 1 -- 172.21.15.1:0/3502492356 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c011d20 con 0x7f24a815b6c0 2024-03-18T20:29:45.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.740+0000 7f24ac845700 1 -- 172.21.15.1:0/3502492356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a815b6c0 msgr2=0x7f24a80cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:45.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.740+0000 7f24ac845700 1 --2- 172.21.15.1:0/3502492356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a815b6c0 0x7f24a80cf5e0 secure :-1 s=READY pgs=3358 cs=0 l=1 rev1=1 crypto rx=0x7f24a806ea00 tx=0x7f249c009bd0 comp rx=0 tx=0).stop 2024-03-18T20:29:45.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.740+0000 7f24ac845700 1 -- 172.21.15.1:0/3502492356 shutdown_connections 2024-03-18T20:29:45.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.740+0000 7f24ac845700 1 --2- 172.21.15.1:0/3502492356 >> v2:172.21.15.165:3300/0 conn(0x7f24a80cf200 0x7f24a815c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:45.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.740+0000 7f24ac845700 1 --2- 172.21.15.1:0/3502492356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a815b6c0 0x7f24a80cf5e0 unknown :-1 s=CLOSED pgs=3358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:45.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.740+0000 7f24ac845700 1 -- 172.21.15.1:0/3502492356 >> 172.21.15.1:0/3502492356 conn(0x7f24a80d1180 msgr2=0x7f24a80d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:45.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.740+0000 7f24ac845700 1 -- 172.21.15.1:0/3502492356 shutdown_connections 2024-03-18T20:29:45.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.740+0000 7f24ac845700 1 -- 172.21.15.1:0/3502492356 wait complete. 2024-03-18T20:29:45.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.741+0000 7f24ac845700 1 Processor -- start 2024-03-18T20:29:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.741+0000 7f24ac845700 1 -- start start 2024-03-18T20:29:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.742+0000 7f24ac845700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24a80cbc90 0x7f24a8151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.742+0000 7f24ac845700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24a80cf200 0x7f24a80b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.742+0000 7f24ac845700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a815b6c0 0x7f24a80af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.742+0000 7f24ac845700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a80b6c70 con 0x7f24a815b6c0 2024-03-18T20:29:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.742+0000 7f24ac845700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a80cb170 con 0x7f24a80cf200 2024-03-18T20:29:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.742+0000 7f24ac845700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a80cb450 con 0x7f24a80cbc90 2024-03-18T20:29:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.742+0000 7f24a6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a815b6c0 0x7f24a80af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.742+0000 7f24a6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a815b6c0 0x7f24a80af080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60028/0 (socket says 172.21.15.1:60028) 2024-03-18T20:29:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.742+0000 7f24a6d9d700 1 -- 172.21.15.1:0/1732893017 learned_addr learned my addr 172.21.15.1:0/1732893017 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:45.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.742+0000 7f24a5d9b700 1 --2- 172.21.15.1:0/1732893017 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24a80cf200 0x7f24a80b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.742+0000 7f24a659c700 1 --2- 172.21.15.1:0/1732893017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24a80cbc90 0x7f24a8151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.743+0000 7f24a6d9d700 1 -- 172.21.15.1:0/1732893017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24a80cbc90 msgr2=0x7f24a8151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.743+0000 7f24a6d9d700 1 --2- 172.21.15.1:0/1732893017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24a80cbc90 0x7f24a8151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.743+0000 7f24a6d9d700 1 -- 172.21.15.1:0/1732893017 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24a80cf200 msgr2=0x7f24a80b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.743+0000 7f24a6d9d700 1 --2- 172.21.15.1:0/1732893017 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24a80cf200 0x7f24a80b01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.743+0000 7f24a6d9d700 1 -- 172.21.15.1:0/1732893017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249c009550 con 0x7f24a815b6c0 2024-03-18T20:29:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.743+0000 7f24a6d9d700 1 --2- 172.21.15.1:0/1732893017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a815b6c0 0x7f24a80af080 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7f24a000cb50 tx=0x7f24a000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.743+0000 7f2496ffd700 1 -- 172.21.15.1:0/1732893017 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a0003c00 con 0x7f24a815b6c0 2024-03-18T20:29:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.743+0000 7f2496ffd700 1 -- 172.21.15.1:0/1732893017 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a0003d60 con 0x7f24a815b6c0 2024-03-18T20:29:45.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.743+0000 7f24ac845700 1 -- 172.21.15.1:0/1732893017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a80b3d80 con 0x7f24a815b6c0 2024-03-18T20:29:45.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.743+0000 7f2496ffd700 1 -- 172.21.15.1:0/1732893017 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a000aa20 con 0x7f24a815b6c0 2024-03-18T20:29:45.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.743+0000 7f24ac845700 1 -- 172.21.15.1:0/1732893017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a80b4340 con 0x7f24a815b6c0 2024-03-18T20:29:45.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.744+0000 7f24ac845700 1 -- 172.21.15.1:0/1732893017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a80d0ca0 con 0x7f24a815b6c0 2024-03-18T20:29:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.745+0000 7f2496ffd700 1 -- 172.21.15.1:0/1732893017 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 175) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f24a000ab80 con 0x7f24a815b6c0 2024-03-18T20:29:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.746+0000 7f2496ffd700 1 --2- 172.21.15.1:0/1732893017 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f248806af60 0x7f248806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.746+0000 7f24a659c700 1 -- 172.21.15.1:0/1732893017 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f248806af60 msgr2=0x7f248806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.746+0000 7f2496ffd700 1 -- 172.21.15.1:0/1732893017 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(291..291 src has 1..291) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f24a008a400 con 0x7f24a815b6c0 2024-03-18T20:29:45.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.746+0000 7f24a659c700 1 --2- 172.21.15.1:0/1732893017 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f248806af60 0x7f248806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:29:45.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.749+0000 7f2496ffd700 1 -- 172.21.15.1:0/1732893017 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f24a0058a10 con 0x7f24a815b6c0 2024-03-18T20:29:45.779 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:45.774+0000 7f972cced240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:29:45.856 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:45.853+0000 7f972cced240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:29:45.928 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:45.923+0000 7f97503aa240 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T20:29:45.933 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:45.929+0000 7f972cced240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:29:45.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.947+0000 7f24a659c700 1 -- 172.21.15.1:0/1732893017 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f248806af60 msgr2=0x7f248806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:45.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.947+0000 7f24a659c700 1 --2- 172.21.15.1:0/1732893017 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f248806af60 0x7f248806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:29:46.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.997+0000 7f24ac845700 1 -- 172.21.15.1:0/1732893017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24a815ca40 con 0x7f24a815b6c0 2024-03-18T20:29:46.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:45.998+0000 7f2496ffd700 1 -- 172.21.15.1:0/1732893017 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4750 (secure 0 0 0) 0x7f24a815ca40 con 0x7f24a815b6c0 2024-03-18T20:29:46.003 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:29:46.004 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":144,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10322,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/2385318877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":2385318877},{"type":"v1","addr":"172.21.15.165:6834","nonce":2385318877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139},{"gid":12040,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1024465047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1024465047},{"type":"v1","addr":"172.21.15.165:6835","nonce":1024465047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139},{"gid":13791,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2374816325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2374816325},{"type":"v1","addr":"172.21.15.1:6837","nonce":2374816325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:28:10.668549+0000","modified":"2024-03-18T20:28:13.687432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13794},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13794":{"gid":13794,"name":"a","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6836/1111156155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1111156155},{"type":"v1","addr":"172.21.15.1:6836","nonce":1111156155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-03-18T20:29:46.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.000+0000 7f24ac845700 1 -- 172.21.15.1:0/1732893017 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f248806af60 msgr2=0x7f248806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:29:46.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.000+0000 7f24ac845700 1 --2- 172.21.15.1:0/1732893017 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f248806af60 0x7f248806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.000+0000 7f24ac845700 1 -- 172.21.15.1:0/1732893017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a815b6c0 msgr2=0x7f24a80af080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:46.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.000+0000 7f24ac845700 1 --2- 172.21.15.1:0/1732893017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a815b6c0 0x7f24a80af080 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7f24a000cb50 tx=0x7f24a000ce60 comp rx=0 tx=0).stop 2024-03-18T20:29:46.005 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:46.000+0000 7f97503aa240 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T20:29:46.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.000+0000 7f24ac845700 1 -- 172.21.15.1:0/1732893017 shutdown_connections 2024-03-18T20:29:46.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.000+0000 7f24ac845700 1 --2- 172.21.15.1:0/1732893017 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f248806af60 0x7f248806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.000+0000 7f24ac845700 1 --2- 172.21.15.1:0/1732893017 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f24a80cbc90 0x7f24a8151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.000+0000 7f24ac845700 1 --2- 172.21.15.1:0/1732893017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24a815b6c0 0x7f24a80af080 unknown :-1 s=CLOSED pgs=3359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.000+0000 7f24ac845700 1 --2- 172.21.15.1:0/1732893017 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24a80cf200 0x7f24a80b01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.000+0000 7f24ac845700 1 -- 172.21.15.1:0/1732893017 >> 172.21.15.1:0/1732893017 conn(0x7f24a80d1180 msgr2=0x7f24a815baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:46.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.000+0000 7f24ac845700 1 -- 172.21.15.1:0/1732893017 shutdown_connections 2024-03-18T20:29:46.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.000+0000 7f24ac845700 1 -- 172.21.15.1:0/1732893017 wait complete. 2024-03-18T20:29:46.007 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 144 2024-03-18T20:29:46.011 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:46.007+0000 7f972cced240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:29:46.018 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:29:46.094 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:46.090+0000 7f97503aa240 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T20:29:46.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.141+0000 7fc97a605700 1 Processor -- start 2024-03-18T20:29:46.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.141+0000 7fc97a605700 1 -- start start 2024-03-18T20:29:46.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.142+0000 7fc97a605700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc97415d5b0 0x7fc9740b3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:46.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.142+0000 7fc97a605700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc97406e4a0 0x7fc97406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:46.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.142+0000 7fc97a605700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9740b4b20 con 0x7fc9740aefa0 2024-03-18T20:29:46.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.142+0000 7fc97a605700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9740b50f0 con 0x7fc97415d5b0 2024-03-18T20:29:46.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.142+0000 7fc97a605700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97406f570 con 0x7fc97406e4a0 2024-03-18T20:29:46.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.142+0000 7fc973fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc97415d5b0 0x7fc9740b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:46.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.142+0000 7fc9737fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc9740aefa0 0x7fc9740ad030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43966/0 (socket says 172.21.15.1:43966) 2024-03-18T20:29:46.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.142+0000 7fc9737fe700 1 -- 172.21.15.1:0/3435995970 learned_addr learned my addr 172.21.15.1:0/3435995970 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:29:46.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.142+0000 7fc978ba2700 1 --2- 172.21.15.1:0/3435995970 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc97406e4a0 0x7fc97406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:46.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.142+0000 7fc973fff700 1 -- 172.21.15.1:0/3435995970 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc97406e4a0 msgr2=0x7fc97406e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:46.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.142+0000 7fc973fff700 1 --2- 172.21.15.1:0/3435995970 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc97406e4a0 0x7fc97406e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.142+0000 7fc973fff700 1 -- 172.21.15.1:0/3435995970 >> v1:172.21.15.1:6789/0 conn(0x7fc9740aefa0 legacy=0x7fc9740ad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:46.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.142+0000 7fc972ffd700 1 -- 172.21.15.1:0/3435995970 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540557926 0 0) 0x7fc9740b4b20 con 0x7fc9740aefa0 2024-03-18T20:29:46.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.142+0000 7fc973fff700 1 -- 172.21.15.1:0/3435995970 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9740b4d50 con 0x7fc97415d5b0 2024-03-18T20:29:46.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.142+0000 7fc973fff700 1 --2- 172.21.15.1:0/3435995970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc97415d5b0 0x7fc9740b3050 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7fc95c009840 tx=0x7fc95c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17953c122ee12eb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:46.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.142+0000 7fc972ffd700 1 -- 172.21.15.1:0/3435995970 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c011830 con 0x7fc97415d5b0 2024-03-18T20:29:46.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.143+0000 7fc972ffd700 1 -- 172.21.15.1:0/3435995970 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95c011990 con 0x7fc97415d5b0 2024-03-18T20:29:46.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.143+0000 7fc972ffd700 1 -- 172.21.15.1:0/3435995970 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c011d20 con 0x7fc97415d5b0 2024-03-18T20:29:46.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.143+0000 7fc97a605700 1 -- 172.21.15.1:0/3435995970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc97415d5b0 msgr2=0x7fc9740b3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:46.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.143+0000 7fc97a605700 1 --2- 172.21.15.1:0/3435995970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc97415d5b0 0x7fc9740b3050 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7fc95c009840 tx=0x7fc95c009b50 comp rx=0 tx=0).stop 2024-03-18T20:29:46.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.143+0000 7fc97a605700 1 -- 172.21.15.1:0/3435995970 shutdown_connections 2024-03-18T20:29:46.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.143+0000 7fc97a605700 1 --2- 172.21.15.1:0/3435995970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc97415d5b0 0x7fc9740b3050 unknown :-1 s=CLOSED pgs=3361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.143+0000 7fc97a605700 1 --2- 172.21.15.1:0/3435995970 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc97406e4a0 0x7fc97406e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.143+0000 7fc97a605700 1 -- 172.21.15.1:0/3435995970 >> 172.21.15.1:0/3435995970 conn(0x7fc9740b5570 msgr2=0x7fc9740b5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:46.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.143+0000 7fc97a605700 1 -- 172.21.15.1:0/3435995970 shutdown_connections 2024-03-18T20:29:46.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.143+0000 7fc97a605700 1 -- 172.21.15.1:0/3435995970 wait complete. 2024-03-18T20:29:46.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.145+0000 7fc97a605700 1 Processor -- start 2024-03-18T20:29:46.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.145+0000 7fc97a605700 1 -- start start 2024-03-18T20:29:46.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc97a605700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc97406e4a0 0x7fc97414ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:46.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc97a605700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9740aefa0 0x7fc974152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:46.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc97a605700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc97415d5b0 0x7fc97414f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:46.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc97a605700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc974156fb0 con 0x7fc9740aefa0 2024-03-18T20:29:46.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc97a605700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9741563a0 con 0x7fc97415d5b0 2024-03-18T20:29:46.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc97a605700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc974156680 con 0x7fc97406e4a0 2024-03-18T20:29:46.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc9737fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9740aefa0 0x7fc974152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:46.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc9737fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9740aefa0 0x7fc974152060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60044/0 (socket says 172.21.15.1:60044) 2024-03-18T20:29:46.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc9737fe700 1 -- 172.21.15.1:0/3615178069 learned_addr learned my addr 172.21.15.1:0/3615178069 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:46.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc973fff700 1 --2- 172.21.15.1:0/3615178069 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc97406e4a0 0x7fc97414ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:46.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc978ba2700 1 --2- 172.21.15.1:0/3615178069 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc97415d5b0 0x7fc97414f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:46.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc9737fe700 1 -- 172.21.15.1:0/3615178069 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc97406e4a0 msgr2=0x7fc97414ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:46.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc9737fe700 1 --2- 172.21.15.1:0/3615178069 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc97406e4a0 0x7fc97414ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc9737fe700 1 -- 172.21.15.1:0/3615178069 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc97415d5b0 msgr2=0x7fc97414f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:46.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc9737fe700 1 --2- 172.21.15.1:0/3615178069 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc97415d5b0 0x7fc97414f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc9737fe700 1 -- 172.21.15.1:0/3615178069 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95c0094a0 con 0x7fc9740aefa0 2024-03-18T20:29:46.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc973fff700 1 --2- 172.21.15.1:0/3615178069 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc97406e4a0 0x7fc97414ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:29:46.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc9737fe700 1 --2- 172.21.15.1:0/3615178069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9740aefa0 0x7fc974152060 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7fc96400c550 tx=0x7fc96400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:46.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc970ff9700 1 -- 172.21.15.1:0/3615178069 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc964004c80 con 0x7fc9740aefa0 2024-03-18T20:29:46.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.146+0000 7fc970ff9700 1 -- 172.21.15.1:0/3615178069 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc964004e00 con 0x7fc9740aefa0 2024-03-18T20:29:46.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.147+0000 7fc97a605700 1 -- 172.21.15.1:0/3615178069 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc974156960 con 0x7fc9740aefa0 2024-03-18T20:29:46.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.147+0000 7fc970ff9700 1 -- 172.21.15.1:0/3615178069 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc964007310 con 0x7fc9740aefa0 2024-03-18T20:29:46.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.147+0000 7fc97a605700 1 -- 172.21.15.1:0/3615178069 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97414f1a0 con 0x7fc9740aefa0 2024-03-18T20:29:46.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.148+0000 7fc970ff9700 1 -- 172.21.15.1:0/3615178069 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 175) v1 ==== 89404+0+0 (secure 0 0 0) 0x7fc964007470 con 0x7fc9740aefa0 2024-03-18T20:29:46.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.148+0000 7fc970ff9700 1 --2- 172.21.15.1:0/3615178069 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc95406af60 0x7fc95406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:46.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.148+0000 7fc973fff700 1 -- 172.21.15.1:0/3615178069 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc95406af60 msgr2=0x7fc95406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:46.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.149+0000 7fc970ff9700 1 -- 172.21.15.1:0/3615178069 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(291..291 src has 1..291) v4 ==== 6560+0+0 (secure 0 0 0) 0x7fc964089660 con 0x7fc9740aefa0 2024-03-18T20:29:46.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.149+0000 7fc973fff700 1 --2- 172.21.15.1:0/3615178069 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc95406af60 0x7fc95406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:29:46.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.152+0000 7fc97a605700 1 -- 172.21.15.1:0/3615178069 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc974157120 con 0x7fc9740aefa0 2024-03-18T20:29:46.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.157+0000 7fc970ff9700 1 -- 172.21.15.1:0/3615178069 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc9640541a0 con 0x7fc9740aefa0 2024-03-18T20:29:46.181 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:46.177+0000 7f972cced240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:29:46.182 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:46.177+0000 7f97503aa240 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T20:29:46.267 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:46.263+0000 7f97503aa240 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T20:29:46.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.350+0000 7fc973fff700 1 -- 172.21.15.1:0/3615178069 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc95406af60 msgr2=0x7fc95406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:46.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.350+0000 7fc973fff700 1 --2- 172.21.15.1:0/3615178069 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc95406af60 0x7fc95406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:29:46.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.364+0000 7fc97a605700 1 -- 172.21.15.1:0/3615178069 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc974067dc0 con 0x7fc9740aefa0 2024-03-18T20:29:46.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.366+0000 7fc970ff9700 1 -- 172.21.15.1:0/3615178069 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v291) v1 ==== 74+0+16996 (secure 0 0 0) 0x7fc9640577c0 con 0x7fc9740aefa0 2024-03-18T20:29:46.371 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:29:46.371 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":291,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:29:27.236798+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:28:08.503736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"287","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-03-18T20:28:09.644774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"288","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":290,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":290,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"38.2","mappings":[{"from":7,"to":2}]},{"pgid":"38.8","mappings":[{"from":7,"to":4}]},{"pgid":"38.12","mappings":[{"from":7,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/1650741459":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/2451665032":"2024-03-19T20:28:26.019720+0000","172.21.15.1:6833/148356":"2024-03-19T20:28:26.019720+0000","172.21.15.1:6832/148356":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/1408878463":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/582749307":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/997400106":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/2561447959":"2024-03-19T20:28:26.019720+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:29:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.368+0000 7fc97a605700 1 -- 172.21.15.1:0/3615178069 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc95406af60 msgr2=0x7fc95406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:29:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.368+0000 7fc97a605700 1 --2- 172.21.15.1:0/3615178069 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc95406af60 0x7fc95406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.368+0000 7fc97a605700 1 -- 172.21.15.1:0/3615178069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9740aefa0 msgr2=0x7fc974152060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.368+0000 7fc97a605700 1 --2- 172.21.15.1:0/3615178069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9740aefa0 0x7fc974152060 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7fc96400c550 tx=0x7fc96400c860 comp rx=0 tx=0).stop 2024-03-18T20:29:46.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.368+0000 7fc97a605700 1 -- 172.21.15.1:0/3615178069 shutdown_connections 2024-03-18T20:29:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.368+0000 7fc97a605700 1 --2- 172.21.15.1:0/3615178069 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc95406af60 0x7fc95406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.368+0000 7fc97a605700 1 --2- 172.21.15.1:0/3615178069 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc97406e4a0 0x7fc97414ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.368+0000 7fc97a605700 1 --2- 172.21.15.1:0/3615178069 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc97415d5b0 0x7fc97414f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.368+0000 7fc97a605700 1 --2- 172.21.15.1:0/3615178069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc9740aefa0 0x7fc974152060 unknown :-1 s=CLOSED pgs=3362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.368+0000 7fc97a605700 1 -- 172.21.15.1:0/3615178069 >> 172.21.15.1:0/3615178069 conn(0x7fc9740b5570 msgr2=0x7fc97406e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.368+0000 7fc97a605700 1 -- 172.21.15.1:0/3615178069 shutdown_connections 2024-03-18T20:29:46.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.368+0000 7fc97a605700 1 -- 172.21.15.1:0/3615178069 wait complete. 2024-03-18T20:29:46.386 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T18:49:39.443363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T20:28:08.503736+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '287', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 38, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T20:28:09.644774+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '288', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T20:29:46.386 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T20:29:46.398 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:46.394+0000 7f972cced240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:29:46.451 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:46.447+0000 7f97503aa240 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T20:29:46.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.512+0000 7f31130cd700 1 Processor -- start 2024-03-18T20:29:46.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.512+0000 7f31130cd700 1 -- start start 2024-03-18T20:29:46.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.512+0000 7f31130cd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f310c15e8e0 0x7f310c0cf380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:46.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f31130cd700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310c06ec60 con 0x7f310c099a30 2024-03-18T20:29:46.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f31130cd700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310c0cfd80 con 0x7f310c15ef60 2024-03-18T20:29:46.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f31130cd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310c070100 con 0x7f310c15e8e0 2024-03-18T20:29:46.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f3110e69700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f310c099a30 0x7f310c099e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43970/0 (socket says 172.21.15.1:43970) 2024-03-18T20:29:46.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f3110e69700 1 -- 172.21.15.1:0/3463320516 learned_addr learned my addr 172.21.15.1:0/3463320516 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:29:46.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f310bfff700 1 --2- 172.21.15.1:0/3463320516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f310c15e8e0 0x7f310c0cf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:46.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f310b7fe700 1 -- 172.21.15.1:0/3463320516 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240875347 0 0) 0x7f310c06ec60 con 0x7f310c099a30 2024-03-18T20:29:46.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f310b7fe700 1 -- 172.21.15.1:0/3463320516 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30fc0035b0 con 0x7f310c099a30 2024-03-18T20:29:46.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f310b7fe700 1 -- 172.21.15.1:0/3463320516 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592630533 0 0) 0x7f310c0cfd80 con 0x7f310c15ef60 2024-03-18T20:29:46.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f310b7fe700 1 -- 172.21.15.1:0/3463320516 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f310c06ec60 con 0x7f310c15ef60 2024-03-18T20:29:46.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f310b7fe700 1 -- 172.21.15.1:0/3463320516 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1494738175 0 0) 0x7f30fc0035b0 con 0x7f310c099a30 2024-03-18T20:29:46.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f310b7fe700 1 -- 172.21.15.1:0/3463320516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f310c15e8e0 msgr2=0x7f310c0cf380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:46.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f310b7fe700 1 --2- 172.21.15.1:0/3463320516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f310c15e8e0 0x7f310c0cf380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f310b7fe700 1 -- 172.21.15.1:0/3463320516 >> v1:172.21.15.165:6789/0 conn(0x7f310c15ef60 legacy=0x7f310c15cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:46.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f310b7fe700 1 -- 172.21.15.1:0/3463320516 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c0d0060 con 0x7f310c099a30 2024-03-18T20:29:46.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f310b7fe700 1 -- 172.21.15.1:0/3463320516 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f3100001670 con 0x7f310c099a30 2024-03-18T20:29:46.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f310b7fe700 1 -- 172.21.15.1:0/3463320516 >> v1:172.21.15.1:6789/0 conn(0x7f310c099a30 legacy=0x7f310c099e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:46.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f310b7fe700 1 --2- 172.21.15.1:0/3463320516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30fc003bc0 0x7f30fc003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:46.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f310b7fe700 1 --2- 172.21.15.1:0/3463320516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30fc004850 0x7f30fc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:46.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f310b7fe700 1 --2- 172.21.15.1:0/3463320516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30fc005210 0x7f30fc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:46.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f310b7fe700 1 -- 172.21.15.1:0/3463320516 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc00c000 con 0x7f30fc003bc0 2024-03-18T20:29:46.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f310b7fe700 1 -- 172.21.15.1:0/3463320516 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc00c140 con 0x7f30fc005210 2024-03-18T20:29:46.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f310b7fe700 1 -- 172.21.15.1:0/3463320516 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc00c300 con 0x7f30fc004850 2024-03-18T20:29:46.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f310bfff700 1 --2- 172.21.15.1:0/3463320516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f310c15e8e0 0x7f310c0cf380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:29:46.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.513+0000 7f3110e69700 1 --2- 172.21.15.1:0/3463320516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30fc003bc0 0x7f30fc003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:46.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.514+0000 7f311166a700 1 --2- 172.21.15.1:0/3463320516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30fc005210 0x7f30fc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:46.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.514+0000 7f310bfff700 1 --2- 172.21.15.1:0/3463320516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30fc004850 0x7f30fc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:46.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.514+0000 7f3110e69700 1 -- 172.21.15.1:0/3463320516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30fc004850 msgr2=0x7f30fc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:46.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.514+0000 7f3110e69700 1 --2- 172.21.15.1:0/3463320516 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f30fc004850 0x7f30fc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.514+0000 7f3110e69700 1 -- 172.21.15.1:0/3463320516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30fc005210 msgr2=0x7f30fc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:46.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.514+0000 7f3110e69700 1 --2- 172.21.15.1:0/3463320516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30fc005210 0x7f30fc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.514+0000 7f3110e69700 1 -- 172.21.15.1:0/3463320516 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30fc0035b0 con 0x7f30fc003bc0 2024-03-18T20:29:46.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.514+0000 7f3110e69700 1 --2- 172.21.15.1:0/3463320516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30fc003bc0 0x7f30fc003fc0 secure :-1 s=READY pgs=3364 cs=0 l=1 rev1=1 crypto rx=0x7f310000c210 tx=0x7f30fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb970d3d0573ded6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:46.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.514+0000 7f310b7fe700 1 -- 172.21.15.1:0/3463320516 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310000cdb0 con 0x7f30fc003bc0 2024-03-18T20:29:46.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.514+0000 7f310b7fe700 1 -- 172.21.15.1:0/3463320516 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3100001af0 con 0x7f30fc003bc0 2024-03-18T20:29:46.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.514+0000 7f310b7fe700 1 -- 172.21.15.1:0/3463320516 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3100005ba0 con 0x7f30fc003bc0 2024-03-18T20:29:46.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.515+0000 7f31130cd700 1 -- 172.21.15.1:0/3463320516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30fc003bc0 msgr2=0x7f30fc003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:46.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.515+0000 7f31130cd700 1 --2- 172.21.15.1:0/3463320516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30fc003bc0 0x7f30fc003fc0 secure :-1 s=READY pgs=3364 cs=0 l=1 rev1=1 crypto rx=0x7f310000c210 tx=0x7f30fc005880 comp rx=0 tx=0).stop 2024-03-18T20:29:46.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.515+0000 7f311166a700 1 -- 172.21.15.1:0/3463320516 reap_dead start 2024-03-18T20:29:46.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.515+0000 7f31130cd700 1 -- 172.21.15.1:0/3463320516 shutdown_connections 2024-03-18T20:29:46.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.515+0000 7f31130cd700 1 -- 172.21.15.1:0/3463320516 >> 172.21.15.1:0/3463320516 conn(0x7f310c0d1180 msgr2=0x7f310c0d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:46.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.515+0000 7f31130cd700 1 -- 172.21.15.1:0/3463320516 shutdown_connections 2024-03-18T20:29:46.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.515+0000 7f31130cd700 1 -- 172.21.15.1:0/3463320516 wait complete. 2024-03-18T20:29:46.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.516+0000 7f31130cd700 1 Processor -- start 2024-03-18T20:29:46.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.516+0000 7f31130cd700 1 -- start start 2024-03-18T20:29:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.517+0000 7f31130cd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30fc003bc0 0x7f310c0cdf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.517+0000 7f31130cd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f310c1f7e70 0x7f310c0c7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.517+0000 7f31130cd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f310c1f72d0 0x7f310c1f7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.517+0000 7f31130cd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310c0b4f90 con 0x7f310c1f72d0 2024-03-18T20:29:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.517+0000 7f31130cd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310c1f7c90 con 0x7f30fc003bc0 2024-03-18T20:29:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.517+0000 7f31130cd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310c0cb910 con 0x7f310c1f7e70 2024-03-18T20:29:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.517+0000 7f311166a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f310c1f72d0 0x7f310c1f7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.517+0000 7f311166a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f310c1f72d0 0x7f310c1f7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60068/0 (socket says 172.21.15.1:60068) 2024-03-18T20:29:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.517+0000 7f311166a700 1 -- 172.21.15.1:0/3758659450 learned_addr learned my addr 172.21.15.1:0/3758659450 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.517+0000 7f310bfff700 1 --2- 172.21.15.1:0/3758659450 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f310c1f7e70 0x7f310c0c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:46.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.517+0000 7f3110e69700 1 --2- 172.21.15.1:0/3758659450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30fc003bc0 0x7f310c0cdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.517+0000 7f311166a700 1 -- 172.21.15.1:0/3758659450 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f310c1f7e70 msgr2=0x7f310c0c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.517+0000 7f311166a700 1 --2- 172.21.15.1:0/3758659450 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f310c1f7e70 0x7f310c0c7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.517+0000 7f311166a700 1 -- 172.21.15.1:0/3758659450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30fc003bc0 msgr2=0x7f310c0cdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.517+0000 7f311166a700 1 --2- 172.21.15.1:0/3758659450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30fc003bc0 0x7f310c0cdf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.517+0000 7f311166a700 1 -- 172.21.15.1:0/3758659450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c0cbbf0 con 0x7f310c1f72d0 2024-03-18T20:29:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.518+0000 7f311166a700 1 --2- 172.21.15.1:0/3758659450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f310c1f72d0 0x7f310c1f7750 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7f310c0b0c90 tx=0x7f310400d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.518+0000 7f31097fa700 1 -- 172.21.15.1:0/3758659450 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104003c10 con 0x7f310c1f72d0 2024-03-18T20:29:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.518+0000 7f31097fa700 1 -- 172.21.15.1:0/3758659450 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3104003d70 con 0x7f310c1f72d0 2024-03-18T20:29:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.518+0000 7f31130cd700 1 -- 172.21.15.1:0/3758659450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f310c0ce4c0 con 0x7f310c1f72d0 2024-03-18T20:29:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.518+0000 7f31097fa700 1 -- 172.21.15.1:0/3758659450 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310400db20 con 0x7f310c1f72d0 2024-03-18T20:29:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.518+0000 7f31130cd700 1 -- 172.21.15.1:0/3758659450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f310c0cbed0 con 0x7f310c1f72d0 2024-03-18T20:29:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.519+0000 7f31130cd700 1 -- 172.21.15.1:0/3758659450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f310c067dc0 con 0x7f310c1f72d0 2024-03-18T20:29:46.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.520+0000 7f31097fa700 1 -- 172.21.15.1:0/3758659450 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 175) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f310400dc80 con 0x7f310c1f72d0 2024-03-18T20:29:46.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.520+0000 7f31097fa700 1 --2- 172.21.15.1:0/3758659450 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30f006af60 0x7f30f006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:46.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.520+0000 7f3110e69700 1 -- 172.21.15.1:0/3758659450 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30f006af60 msgr2=0x7f30f006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:46.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.520+0000 7f31097fa700 1 -- 172.21.15.1:0/3758659450 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(291..291 src has 1..291) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f3104089d40 con 0x7f310c1f72d0 2024-03-18T20:29:46.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.520+0000 7f3110e69700 1 --2- 172.21.15.1:0/3758659450 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30f006af60 0x7f30f006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:29:46.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.528+0000 7f31097fa700 1 -- 172.21.15.1:0/3758659450 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3104054880 con 0x7f310c1f72d0 2024-03-18T20:29:46.550 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:46.546+0000 7f972cced240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:29:46.622 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:46.618+0000 7f972cced240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:29:46.679 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:46.674+0000 7f97503aa240 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T20:29:46.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.721+0000 7f3110e69700 1 -- 172.21.15.1:0/3758659450 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30f006af60 msgr2=0x7f30f006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:46.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.721+0000 7f3110e69700 1 --2- 172.21.15.1:0/3758659450 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30f006af60 0x7f30f006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:29:46.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.735+0000 7f31130cd700 1 -- 172.21.15.1:0/3758659450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f310c0b3bf0 con 0x7f310c1f72d0 2024-03-18T20:29:46.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.735+0000 7f31097fa700 1 -- 172.21.15.1:0/3758659450 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v291) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3104057ea0 con 0x7f310c1f72d0 2024-03-18T20:29:46.740 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-18T20:29:46.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.737+0000 7f31130cd700 1 -- 172.21.15.1:0/3758659450 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30f006af60 msgr2=0x7f30f006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:29:46.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.737+0000 7f31130cd700 1 --2- 172.21.15.1:0/3758659450 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30f006af60 0x7f30f006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.737+0000 7f31130cd700 1 -- 172.21.15.1:0/3758659450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f310c1f72d0 msgr2=0x7f310c1f7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:46.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.737+0000 7f31130cd700 1 --2- 172.21.15.1:0/3758659450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f310c1f72d0 0x7f310c1f7750 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7f310c0b0c90 tx=0x7f310400d430 comp rx=0 tx=0).stop 2024-03-18T20:29:46.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.737+0000 7f31130cd700 1 -- 172.21.15.1:0/3758659450 shutdown_connections 2024-03-18T20:29:46.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.737+0000 7f31130cd700 1 --2- 172.21.15.1:0/3758659450 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f30f006af60 0x7f30f006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.737+0000 7f31130cd700 1 --2- 172.21.15.1:0/3758659450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f310c1f72d0 0x7f310c1f7750 unknown :-1 s=CLOSED pgs=3365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.737+0000 7f31130cd700 1 --2- 172.21.15.1:0/3758659450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30fc003bc0 0x7f310c0cdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.737+0000 7f31130cd700 1 --2- 172.21.15.1:0/3758659450 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f310c1f7e70 0x7f310c0c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.737+0000 7f31130cd700 1 -- 172.21.15.1:0/3758659450 >> 172.21.15.1:0/3758659450 conn(0x7f310c0d1180 msgr2=0x7f310c15d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:46.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.737+0000 7f31130cd700 1 -- 172.21.15.1:0/3758659450 shutdown_connections 2024-03-18T20:29:46.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.737+0000 7f31130cd700 1 -- 172.21.15.1:0/3758659450 wait complete. 2024-03-18T20:29:46.755 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T20:29:46.846 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:46.841+0000 7f97503aa240 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T20:29:46.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.880+0000 7fef5d882700 1 Processor -- start 2024-03-18T20:29:46.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.880+0000 7fef5d882700 1 -- start start 2024-03-18T20:29:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.881+0000 7fef5d882700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef58160ff0 0x7fef580b2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.881+0000 7fef5d882700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef580b37e0 con 0x7fef580b30c0 2024-03-18T20:29:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.881+0000 7fef5d882700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef580b1590 con 0x7fef58160c10 2024-03-18T20:29:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.881+0000 7fef5d882700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef580b3550 con 0x7fef58160ff0 2024-03-18T20:29:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.882+0000 7fef577fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fef580b30c0 0x7fef580b0030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43978/0 (socket says 172.21.15.1:43978) 2024-03-18T20:29:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.882+0000 7fef577fe700 1 -- 172.21.15.1:0/2773408573 learned_addr learned my addr 172.21.15.1:0/2773408573 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:29:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.882+0000 7fef567fc700 1 --2- 172.21.15.1:0/2773408573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef58160ff0 0x7fef580b2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.882+0000 7fef55ffb700 1 -- 172.21.15.1:0/2773408573 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404410790 0 0) 0x7fef580b37e0 con 0x7fef580b30c0 2024-03-18T20:29:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.882+0000 7fef55ffb700 1 -- 172.21.15.1:0/2773408573 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef400035b0 con 0x7fef580b30c0 2024-03-18T20:29:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.882+0000 7fef55ffb700 1 -- 172.21.15.1:0/2773408573 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231544949 0 0) 0x7fef580b1590 con 0x7fef58160c10 2024-03-18T20:29:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.882+0000 7fef55ffb700 1 -- 172.21.15.1:0/2773408573 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef580b37e0 con 0x7fef58160c10 2024-03-18T20:29:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.882+0000 7fef567fc700 1 -- 172.21.15.1:0/2773408573 >> v1:172.21.15.165:6789/0 conn(0x7fef58160c10 legacy=0x7fef580b2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.882+0000 7fef567fc700 1 -- 172.21.15.1:0/2773408573 >> v1:172.21.15.1:6789/0 conn(0x7fef580b30c0 legacy=0x7fef580b0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.882+0000 7fef567fc700 1 -- 172.21.15.1:0/2773408573 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef580b1870 con 0x7fef58160ff0 2024-03-18T20:29:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.882+0000 7fef567fc700 1 --2- 172.21.15.1:0/2773408573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef58160ff0 0x7fef580b2b80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fef4c009870 tx=0x7fef4c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71edbd8116b38765 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.882+0000 7fef55ffb700 1 -- 172.21.15.1:0/2773408573 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c011830 con 0x7fef58160ff0 2024-03-18T20:29:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.882+0000 7fef55ffb700 1 -- 172.21.15.1:0/2773408573 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4c011990 con 0x7fef58160ff0 2024-03-18T20:29:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.882+0000 7fef55ffb700 1 -- 172.21.15.1:0/2773408573 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c011d20 con 0x7fef58160ff0 2024-03-18T20:29:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.883+0000 7fef5d882700 1 -- 172.21.15.1:0/2773408573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef58160ff0 msgr2=0x7fef580b2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.883+0000 7fef5d882700 1 --2- 172.21.15.1:0/2773408573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef58160ff0 0x7fef580b2b80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fef4c009870 tx=0x7fef4c009b80 comp rx=0 tx=0).stop 2024-03-18T20:29:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.883+0000 7fef5d882700 1 -- 172.21.15.1:0/2773408573 shutdown_connections 2024-03-18T20:29:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.883+0000 7fef5d882700 1 --2- 172.21.15.1:0/2773408573 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef58160ff0 0x7fef580b2b80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.883+0000 7fef5d882700 1 -- 172.21.15.1:0/2773408573 >> 172.21.15.1:0/2773408573 conn(0x7fef5808afc0 msgr2=0x7fef580d3400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.883+0000 7fef5d882700 1 -- 172.21.15.1:0/2773408573 shutdown_connections 2024-03-18T20:29:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.883+0000 7fef5d882700 1 -- 172.21.15.1:0/2773408573 wait complete. 2024-03-18T20:29:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.885+0000 7fef5d882700 1 Processor -- start 2024-03-18T20:29:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.885+0000 7fef5d882700 1 -- start start 2024-03-18T20:29:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.886+0000 7fef5d882700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef580b30c0 0x7fef58146520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.886+0000 7fef5d882700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef58160c10 0x7fef580cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.886+0000 7fef5d882700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef58160ff0 0x7fef580b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.886+0000 7fef5d882700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef580b8050 con 0x7fef58160ff0 2024-03-18T20:29:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.886+0000 7fef5d882700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef58146b00 con 0x7fef580b30c0 2024-03-18T20:29:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.886+0000 7fef5d882700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef58146de0 con 0x7fef58160c10 2024-03-18T20:29:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.886+0000 7fef577fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef58160ff0 0x7fef580b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.886+0000 7fef56ffd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef580b30c0 0x7fef58146520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.886+0000 7fef56ffd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef580b30c0 0x7fef58146520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:47128/0 (socket says 172.21.15.1:47128) 2024-03-18T20:29:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.886+0000 7fef56ffd700 1 -- 172.21.15.1:0/4013478769 learned_addr learned my addr 172.21.15.1:0/4013478769 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.886+0000 7fef577fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef58160ff0 0x7fef580b61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60084/0 (socket says 172.21.15.1:60084) 2024-03-18T20:29:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.886+0000 7fef567fc700 1 --2- 172.21.15.1:0/4013478769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef58160c10 0x7fef580cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.886+0000 7fef56ffd700 1 -- 172.21.15.1:0/4013478769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef58160c10 msgr2=0x7fef580cac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.886+0000 7fef56ffd700 1 --2- 172.21.15.1:0/4013478769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef58160c10 0x7fef580cac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.886+0000 7fef56ffd700 1 -- 172.21.15.1:0/4013478769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef58160ff0 msgr2=0x7fef580b61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.886+0000 7fef56ffd700 1 --2- 172.21.15.1:0/4013478769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef58160ff0 0x7fef580b61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.886+0000 7fef56ffd700 1 -- 172.21.15.1:0/4013478769 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4c0094a0 con 0x7fef580b30c0 2024-03-18T20:29:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.886+0000 7fef577fe700 1 --2- 172.21.15.1:0/4013478769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef58160ff0 0x7fef580b61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:29:46.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.887+0000 7fef56ffd700 1 --2- 172.21.15.1:0/4013478769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef580b30c0 0x7fef58146520 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fef4800c520 tx=0x7fef4800c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.887+0000 7fef3ffff700 1 -- 172.21.15.1:0/4013478769 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef480052d0 con 0x7fef580b30c0 2024-03-18T20:29:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.887+0000 7fef3ffff700 1 -- 172.21.15.1:0/4013478769 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef48005430 con 0x7fef580b30c0 2024-03-18T20:29:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.887+0000 7fef5d882700 1 -- 172.21.15.1:0/4013478769 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef581470c0 con 0x7fef580b30c0 2024-03-18T20:29:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.887+0000 7fef3ffff700 1 -- 172.21.15.1:0/4013478769 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef480038c0 con 0x7fef580b30c0 2024-03-18T20:29:46.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.887+0000 7fef5d882700 1 -- 172.21.15.1:0/4013478769 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef581473a0 con 0x7fef580b30c0 2024-03-18T20:29:46.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.888+0000 7fef5d882700 1 -- 172.21.15.1:0/4013478769 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef580b7e90 con 0x7fef580b30c0 2024-03-18T20:29:46.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.888+0000 7fef3ffff700 1 -- 172.21.15.1:0/4013478769 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 175) v1 ==== 89404+0+0 (secure 0 0 0) 0x7fef48003a20 con 0x7fef580b30c0 2024-03-18T20:29:46.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.889+0000 7fef3ffff700 1 --2- 172.21.15.1:0/4013478769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fef4406af60 0x7fef4406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:46.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.889+0000 7fef567fc700 1 -- 172.21.15.1:0/4013478769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fef4406af60 msgr2=0x7fef4406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:46.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.889+0000 7fef3ffff700 1 -- 172.21.15.1:0/4013478769 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(291..291 src has 1..291) v4 ==== 6560+0+0 (secure 0 0 0) 0x7fef48088fb0 con 0x7fef580b30c0 2024-03-18T20:29:46.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.889+0000 7fef567fc700 1 --2- 172.21.15.1:0/4013478769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fef4406af60 0x7fef4406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:29:46.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:46.893+0000 7fef3ffff700 1 -- 172.21.15.1:0/4013478769 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fef580b7e90 con 0x7fef580b30c0 2024-03-18T20:29:46.926 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:46.921+0000 7f97503aa240 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T20:29:47.071 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:47.068+0000 7f972cced240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:29:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.090+0000 7fef567fc700 1 -- 172.21.15.1:0/4013478769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fef4406af60 msgr2=0x7fef4406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:47.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.090+0000 7fef567fc700 1 --2- 172.21.15.1:0/4013478769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fef4406af60 0x7fef4406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:29:47.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.096+0000 7fef5d882700 1 -- 172.21.15.1:0/4013478769 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fef580b7e90 con 0x7fef580b30c0 2024-03-18T20:29:47.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.097+0000 7fef3ffff700 1 -- 172.21.15.1:0/4013478769 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v291) v1 ==== 104+0+11 (secure 0 0 0) 0x7fef580b7e90 con 0x7fef580b30c0 2024-03-18T20:29:47.102 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-03-18T20:29:47.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.099+0000 7fef5d882700 1 -- 172.21.15.1:0/4013478769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fef4406af60 msgr2=0x7fef4406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:29:47.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.099+0000 7fef5d882700 1 --2- 172.21.15.1:0/4013478769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fef4406af60 0x7fef4406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.099+0000 7fef5d882700 1 -- 172.21.15.1:0/4013478769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef580b30c0 msgr2=0x7fef58146520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:47.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.099+0000 7fef5d882700 1 --2- 172.21.15.1:0/4013478769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef580b30c0 0x7fef58146520 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fef4800c520 tx=0x7fef4800c830 comp rx=0 tx=0).stop 2024-03-18T20:29:47.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.100+0000 7fef5d882700 1 -- 172.21.15.1:0/4013478769 shutdown_connections 2024-03-18T20:29:47.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.100+0000 7fef5d882700 1 --2- 172.21.15.1:0/4013478769 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fef4406af60 0x7fef4406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.100+0000 7fef5d882700 1 --2- 172.21.15.1:0/4013478769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef58160ff0 0x7fef580b61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.100+0000 7fef5d882700 1 --2- 172.21.15.1:0/4013478769 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef580b30c0 0x7fef58146520 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.100+0000 7fef5d882700 1 --2- 172.21.15.1:0/4013478769 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fef58160c10 0x7fef580cac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.100+0000 7fef5d882700 1 -- 172.21.15.1:0/4013478769 >> 172.21.15.1:0/4013478769 conn(0x7fef5808afc0 msgr2=0x7fef580b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:47.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.100+0000 7fef5d882700 1 -- 172.21.15.1:0/4013478769 shutdown_connections 2024-03-18T20:29:47.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.100+0000 7fef5d882700 1 -- 172.21.15.1:0/4013478769 wait complete. 2024-03-18T20:29:47.117 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T20:29:47.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.243+0000 7f5355fdc700 1 Processor -- start 2024-03-18T20:29:47.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.243+0000 7f5355fdc700 1 -- start start 2024-03-18T20:29:47.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.243+0000 7f5355fdc700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f535015d5e0 0x7f53500d1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.243+0000 7f5355fdc700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53500cfe30 con 0x7f53500d3f80 2024-03-18T20:29:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.243+0000 7f5355fdc700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53500d0110 con 0x7f535015e0d0 2024-03-18T20:29:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.243+0000 7f5355fdc700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53500d03f0 con 0x7f535015d5e0 2024-03-18T20:29:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.244+0000 7f534effd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f53500d3f80 0x7f535015d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43984/0 (socket says 172.21.15.1:43984) 2024-03-18T20:29:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.244+0000 7f534effd700 1 -- 172.21.15.1:0/271819408 learned_addr learned my addr 172.21.15.1:0/271819408 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:29:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.244+0000 7f534f7fe700 1 --2- 172.21.15.1:0/271819408 >> v2:172.21.15.165:3300/0 conn(0x7f535015d5e0 0x7f53500d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.244+0000 7f534e7fc700 1 -- 172.21.15.1:0/271819408 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905815358 0 0) 0x7f53500cfe30 con 0x7f53500d3f80 2024-03-18T20:29:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.244+0000 7f534e7fc700 1 -- 172.21.15.1:0/271819408 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53380035b0 con 0x7f53500d3f80 2024-03-18T20:29:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.244+0000 7f534e7fc700 1 -- 172.21.15.1:0/271819408 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431294966 0 0) 0x7f53500d0110 con 0x7f535015e0d0 2024-03-18T20:29:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.244+0000 7f534e7fc700 1 -- 172.21.15.1:0/271819408 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53500cfe30 con 0x7f535015e0d0 2024-03-18T20:29:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.244+0000 7f534f7fe700 1 -- 172.21.15.1:0/271819408 >> v1:172.21.15.165:6789/0 conn(0x7f535015e0d0 legacy=0x7f53500cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.244+0000 7f534f7fe700 1 -- 172.21.15.1:0/271819408 >> v1:172.21.15.1:6789/0 conn(0x7f53500d3f80 legacy=0x7f535015d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.244+0000 7f534f7fe700 1 -- 172.21.15.1:0/271819408 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53500d06d0 con 0x7f535015d5e0 2024-03-18T20:29:47.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.244+0000 7f534f7fe700 1 --2- 172.21.15.1:0/271819408 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f535015d5e0 0x7f53500d1dd0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f5340009840 tx=0x7f5340009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34348a45f9f3067 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:47.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.244+0000 7f534e7fc700 1 -- 172.21.15.1:0/271819408 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5340011830 con 0x7f535015d5e0 2024-03-18T20:29:47.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.245+0000 7f534e7fc700 1 -- 172.21.15.1:0/271819408 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5340011990 con 0x7f535015d5e0 2024-03-18T20:29:47.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.245+0000 7f534e7fc700 1 -- 172.21.15.1:0/271819408 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5340011d20 con 0x7f535015d5e0 2024-03-18T20:29:47.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.245+0000 7f5355fdc700 1 -- 172.21.15.1:0/271819408 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f535015d5e0 msgr2=0x7f53500d1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:47.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.245+0000 7f5355fdc700 1 --2- 172.21.15.1:0/271819408 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f535015d5e0 0x7f53500d1dd0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f5340009840 tx=0x7f5340009b50 comp rx=0 tx=0).stop 2024-03-18T20:29:47.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.245+0000 7f5355fdc700 1 -- 172.21.15.1:0/271819408 shutdown_connections 2024-03-18T20:29:47.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.245+0000 7f5355fdc700 1 --2- 172.21.15.1:0/271819408 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f535015d5e0 0x7f53500d1dd0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.245+0000 7f5355fdc700 1 -- 172.21.15.1:0/271819408 >> 172.21.15.1:0/271819408 conn(0x7f53500d3380 msgr2=0x7f53500d3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:47.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.245+0000 7f5355fdc700 1 -- 172.21.15.1:0/271819408 shutdown_connections 2024-03-18T20:29:47.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.245+0000 7f5355fdc700 1 -- 172.21.15.1:0/271819408 wait complete. 2024-03-18T20:29:47.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.246+0000 7f5355fdc700 1 Processor -- start 2024-03-18T20:29:47.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.247+0000 7f5355fdc700 1 -- start start 2024-03-18T20:29:47.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.247+0000 7f5355fdc700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53500d3f80 0x7f535015bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:47.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.247+0000 7f5355fdc700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f535015d5e0 0x7f53500c9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:47.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.247+0000 7f5355fdc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f535015e0d0 0x7f53501608c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:47.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.247+0000 7f5355fdc700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53501f79d0 con 0x7f535015e0d0 2024-03-18T20:29:47.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.247+0000 7f5355fdc700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53501f6e50 con 0x7f53500d3f80 2024-03-18T20:29:47.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.247+0000 7f5355fdc700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53501f7130 con 0x7f535015d5e0 2024-03-18T20:29:47.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.248+0000 7f534f7fe700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53500d3f80 0x7f535015bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:47.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.248+0000 7f534ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f535015e0d0 0x7f53501608c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:47.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.248+0000 7f534f7fe700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53500d3f80 0x7f535015bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:47132/0 (socket says 172.21.15.1:47132) 2024-03-18T20:29:47.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.248+0000 7f534f7fe700 1 -- 172.21.15.1:0/3875456311 learned_addr learned my addr 172.21.15.1:0/3875456311 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:47.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.248+0000 7f534ffff700 1 --2- 172.21.15.1:0/3875456311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f535015e0d0 0x7f53501608c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60088/0 (socket says 172.21.15.1:60088) 2024-03-18T20:29:47.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.248+0000 7f534effd700 1 --2- 172.21.15.1:0/3875456311 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f535015d5e0 0x7f53500c9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:47.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.248+0000 7f534f7fe700 1 -- 172.21.15.1:0/3875456311 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f535015d5e0 msgr2=0x7f53500c9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:47.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.248+0000 7f534f7fe700 1 --2- 172.21.15.1:0/3875456311 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f535015d5e0 0x7f53500c9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.248+0000 7f534f7fe700 1 -- 172.21.15.1:0/3875456311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f535015e0d0 msgr2=0x7f53501608c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:47.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.248+0000 7f534f7fe700 1 --2- 172.21.15.1:0/3875456311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f535015e0d0 0x7f53501608c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.248+0000 7f534f7fe700 1 -- 172.21.15.1:0/3875456311 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53400094a0 con 0x7f53500d3f80 2024-03-18T20:29:47.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.248+0000 7f534ffff700 1 --2- 172.21.15.1:0/3875456311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f535015e0d0 0x7f53501608c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:29:47.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.248+0000 7f534f7fe700 1 --2- 172.21.15.1:0/3875456311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53500d3f80 0x7f535015bd80 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f53400090b0 tx=0x7f5340004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:47.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.248+0000 7f5337fff700 1 -- 172.21.15.1:0/3875456311 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534001a6e0 con 0x7f53500d3f80 2024-03-18T20:29:47.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.248+0000 7f5337fff700 1 -- 172.21.15.1:0/3875456311 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534001a840 con 0x7f53500d3f80 2024-03-18T20:29:47.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.248+0000 7f5355fdc700 1 -- 172.21.15.1:0/3875456311 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53500d2490 con 0x7f53500d3f80 2024-03-18T20:29:47.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.248+0000 7f5337fff700 1 -- 172.21.15.1:0/3875456311 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534001abb0 con 0x7f53500d3f80 2024-03-18T20:29:47.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.248+0000 7f5355fdc700 1 -- 172.21.15.1:0/3875456311 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53501f76f0 con 0x7f53500d3f80 2024-03-18T20:29:47.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.249+0000 7f5355fdc700 1 -- 172.21.15.1:0/3875456311 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5350067dc0 con 0x7f53500d3f80 2024-03-18T20:29:47.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.250+0000 7f5337fff700 1 -- 172.21.15.1:0/3875456311 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 175) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f5340022070 con 0x7f53500d3f80 2024-03-18T20:29:47.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.250+0000 7f5337fff700 1 --2- 172.21.15.1:0/3875456311 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f533c06af60 0x7f533c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:47.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.250+0000 7f534effd700 1 -- 172.21.15.1:0/3875456311 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f533c06af60 msgr2=0x7f533c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:47.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.250+0000 7f5337fff700 1 -- 172.21.15.1:0/3875456311 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(291..291 src has 1..291) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f534008a670 con 0x7f53500d3f80 2024-03-18T20:29:47.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.250+0000 7f534effd700 1 --2- 172.21.15.1:0/3875456311 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f533c06af60 0x7f533c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:29:47.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.255+0000 7f5337fff700 1 -- 172.21.15.1:0/3875456311 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5340055230 con 0x7f53500d3f80 2024-03-18T20:29:47.412 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:47.408+0000 7f972cced240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:29:47.413 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:47.409+0000 7f97503aa240 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T20:29:47.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.451+0000 7f534effd700 1 -- 172.21.15.1:0/3875456311 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f533c06af60 msgr2=0x7f533c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:47.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.451+0000 7f534effd700 1 --2- 172.21.15.1:0/3875456311 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f533c06af60 0x7f533c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:29:47.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.457+0000 7f5355fdc700 1 -- 172.21.15.1:0/3875456311 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f53500cfb70 con 0x7f53500d3f80 2024-03-18T20:29:47.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.457+0000 7f5337fff700 1 -- 172.21.15.1:0/3875456311 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v291) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5340058850 con 0x7f53500d3f80 2024-03-18T20:29:47.462 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-03-18T20:29:47.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.460+0000 7f5355fdc700 1 -- 172.21.15.1:0/3875456311 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f533c06af60 msgr2=0x7f533c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:29:47.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.460+0000 7f5355fdc700 1 --2- 172.21.15.1:0/3875456311 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f533c06af60 0x7f533c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.460+0000 7f5355fdc700 1 -- 172.21.15.1:0/3875456311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53500d3f80 msgr2=0x7f535015bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:47.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.460+0000 7f5355fdc700 1 --2- 172.21.15.1:0/3875456311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53500d3f80 0x7f535015bd80 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f53400090b0 tx=0x7f5340004750 comp rx=0 tx=0).stop 2024-03-18T20:29:47.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.460+0000 7f5355fdc700 1 -- 172.21.15.1:0/3875456311 shutdown_connections 2024-03-18T20:29:47.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.460+0000 7f5355fdc700 1 --2- 172.21.15.1:0/3875456311 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f533c06af60 0x7f533c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.460+0000 7f5355fdc700 1 --2- 172.21.15.1:0/3875456311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f535015e0d0 0x7f53501608c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.460+0000 7f5355fdc700 1 --2- 172.21.15.1:0/3875456311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53500d3f80 0x7f535015bd80 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.460+0000 7f5355fdc700 1 --2- 172.21.15.1:0/3875456311 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f535015d5e0 0x7f53500c9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.460+0000 7f5355fdc700 1 -- 172.21.15.1:0/3875456311 >> 172.21.15.1:0/3875456311 conn(0x7f53500d3380 msgr2=0x7f53500b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:47.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.460+0000 7f5355fdc700 1 -- 172.21.15.1:0/3875456311 shutdown_connections 2024-03-18T20:29:47.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.460+0000 7f5355fdc700 1 -- 172.21.15.1:0/3875456311 wait complete. 2024-03-18T20:29:47.479 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:29:47.579 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:47.575+0000 7f972cced240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:29:47.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.603+0000 7fcc91080700 1 Processor -- start 2024-03-18T20:29:47.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.603+0000 7fcc91080700 1 -- start start 2024-03-18T20:29:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.604+0000 7fcc91080700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc8c15b6c0 0x7fcc8c15c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.604+0000 7fcc91080700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcc8c0cbc90 0x7fcc8c099ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.604+0000 7fcc91080700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc8c15ca40 con 0x7fcc8c0cf200 2024-03-18T20:29:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.604+0000 7fcc91080700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc8c0cce30 con 0x7fcc8c0cbc90 2024-03-18T20:29:47.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.604+0000 7fcc91080700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8c0cc220 con 0x7fcc8c15b6c0 2024-03-18T20:29:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.604+0000 7fcc8b59e700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcc8c0cbc90 0x7fcc8c099ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.604+0000 7fcc8ad9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcc8c0cf200 0x7fcc8c0cf5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43986/0 (socket says 172.21.15.1:43986) 2024-03-18T20:29:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.604+0000 7fcc8ad9d700 1 -- 172.21.15.1:0/2937928814 learned_addr learned my addr 172.21.15.1:0/2937928814 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:29:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.605+0000 7fcc8a59c700 1 --2- 172.21.15.1:0/2937928814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc8c15b6c0 0x7fcc8c15c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.605+0000 7fcc8b59e700 1 -- 172.21.15.1:0/2937928814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc8c15b6c0 msgr2=0x7fcc8c15c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.605+0000 7fcc8b59e700 1 --2- 172.21.15.1:0/2937928814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc8c15b6c0 0x7fcc8c15c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.605+0000 7fcc8b59e700 1 -- 172.21.15.1:0/2937928814 >> v1:172.21.15.1:6789/0 conn(0x7fcc8c0cf200 legacy=0x7fcc8c0cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.605+0000 7fcc89d9b700 1 -- 172.21.15.1:0/2937928814 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527558833 0 0) 0x7fcc7c001110 con 0x7fcc8c0cf200 2024-03-18T20:29:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.605+0000 7fcc8b59e700 1 -- 172.21.15.1:0/2937928814 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8c0cc500 con 0x7fcc8c0cbc90 2024-03-18T20:29:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.605+0000 7fcc8b59e700 1 --2- 172.21.15.1:0/2937928814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc8c0cbc90 0x7fcc8c099ac0 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7fcc84009d90 tx=0x7fcc8400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b4651705e01803e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.605+0000 7fcc89d9b700 1 -- 172.21.15.1:0/2937928814 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8400e070 con 0x7fcc8c0cbc90 2024-03-18T20:29:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.605+0000 7fcc89d9b700 1 -- 172.21.15.1:0/2937928814 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc8400bb20 con 0x7fcc8c0cbc90 2024-03-18T20:29:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.605+0000 7fcc89d9b700 1 -- 172.21.15.1:0/2937928814 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc84012430 con 0x7fcc8c0cbc90 2024-03-18T20:29:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.606+0000 7fcc91080700 1 -- 172.21.15.1:0/2937928814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc8c0cbc90 msgr2=0x7fcc8c099ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:47.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.606+0000 7fcc91080700 1 --2- 172.21.15.1:0/2937928814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc8c0cbc90 0x7fcc8c099ac0 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7fcc84009d90 tx=0x7fcc8400b330 comp rx=0 tx=0).stop 2024-03-18T20:29:47.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.606+0000 7fcc91080700 1 -- 172.21.15.1:0/2937928814 shutdown_connections 2024-03-18T20:29:47.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.606+0000 7fcc91080700 1 --2- 172.21.15.1:0/2937928814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc8c0cbc90 0x7fcc8c099ac0 unknown :-1 s=CLOSED pgs=3369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.606+0000 7fcc91080700 1 --2- 172.21.15.1:0/2937928814 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc8c15b6c0 0x7fcc8c15c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.606+0000 7fcc91080700 1 -- 172.21.15.1:0/2937928814 >> 172.21.15.1:0/2937928814 conn(0x7fcc8c0d1180 msgr2=0x7fcc8c0d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:47.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.606+0000 7fcc91080700 1 -- 172.21.15.1:0/2937928814 shutdown_connections 2024-03-18T20:29:47.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.606+0000 7fcc91080700 1 -- 172.21.15.1:0/2937928814 wait complete. 2024-03-18T20:29:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.607+0000 7fcc91080700 1 Processor -- start 2024-03-18T20:29:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.607+0000 7fcc91080700 1 -- start start 2024-03-18T20:29:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.608+0000 7fcc91080700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc8c0cbc90 0x7fcc8c0c52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.608+0000 7fcc91080700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc8c0cf200 0x7fcc8c0c6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.608+0000 7fcc91080700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc8c15b6c0 0x7fcc8c0b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:47.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.608+0000 7fcc91080700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8c1f7800 con 0x7fcc8c15b6c0 2024-03-18T20:29:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.608+0000 7fcc91080700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8c0c7ad0 con 0x7fcc8c0cf200 2024-03-18T20:29:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.608+0000 7fcc91080700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8c0c77f0 con 0x7fcc8c0cbc90 2024-03-18T20:29:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.608+0000 7fcc8b59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc8c15b6c0 0x7fcc8c0b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.608+0000 7fcc8b59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc8c15b6c0 0x7fcc8c0b0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60118/0 (socket says 172.21.15.1:60118) 2024-03-18T20:29:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.608+0000 7fcc8b59e700 1 -- 172.21.15.1:0/2193177762 learned_addr learned my addr 172.21.15.1:0/2193177762 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.608+0000 7fcc8ad9d700 1 --2- 172.21.15.1:0/2193177762 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc8c0cbc90 0x7fcc8c0c52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.608+0000 7fcc8a59c700 1 --2- 172.21.15.1:0/2193177762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc8c0cf200 0x7fcc8c0c6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.608+0000 7fcc8b59e700 1 -- 172.21.15.1:0/2193177762 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc8c0cbc90 msgr2=0x7fcc8c0c52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.608+0000 7fcc8b59e700 1 --2- 172.21.15.1:0/2193177762 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc8c0cbc90 0x7fcc8c0c52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.608+0000 7fcc8b59e700 1 -- 172.21.15.1:0/2193177762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc8c0cf200 msgr2=0x7fcc8c0c6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.608+0000 7fcc8b59e700 1 --2- 172.21.15.1:0/2193177762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc8c0cf200 0x7fcc8c0c6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.609+0000 7fcc8b59e700 1 -- 172.21.15.1:0/2193177762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc84009a70 con 0x7fcc8c15b6c0 2024-03-18T20:29:47.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.609+0000 7fcc8b59e700 1 --2- 172.21.15.1:0/2193177762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc8c15b6c0 0x7fcc8c0b0e50 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c072540 tx=0x7fcc84004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:47.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.609+0000 7fcc737fe700 1 -- 172.21.15.1:0/2193177762 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc84012430 con 0x7fcc8c15b6c0 2024-03-18T20:29:47.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.609+0000 7fcc737fe700 1 -- 172.21.15.1:0/2193177762 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc84020070 con 0x7fcc8c15b6c0 2024-03-18T20:29:47.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.609+0000 7fcc91080700 1 -- 172.21.15.1:0/2193177762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc8c1f7140 con 0x7fcc8c15b6c0 2024-03-18T20:29:47.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.609+0000 7fcc737fe700 1 -- 172.21.15.1:0/2193177762 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc84004280 con 0x7fcc8c15b6c0 2024-03-18T20:29:47.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.609+0000 7fcc91080700 1 -- 172.21.15.1:0/2193177762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc8c0c5810 con 0x7fcc8c15b6c0 2024-03-18T20:29:47.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.610+0000 7fcc91080700 1 -- 172.21.15.1:0/2193177762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc8c1f7940 con 0x7fcc8c15b6c0 2024-03-18T20:29:47.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.610+0000 7fcc737fe700 1 -- 172.21.15.1:0/2193177762 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 175) v1 ==== 89404+0+0 (secure 0 0 0) 0x7fcc840043e0 con 0x7fcc8c15b6c0 2024-03-18T20:29:47.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.611+0000 7fcc737fe700 1 --2- 172.21.15.1:0/2193177762 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcc7806af60 0x7fcc7806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:47.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.611+0000 7fcc8ad9d700 1 -- 172.21.15.1:0/2193177762 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcc7806af60 msgr2=0x7fcc7806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:47.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.611+0000 7fcc737fe700 1 -- 172.21.15.1:0/2193177762 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(291..291 src has 1..291) v4 ==== 6560+0+0 (secure 0 0 0) 0x7fcc84022070 con 0x7fcc8c15b6c0 2024-03-18T20:29:47.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.611+0000 7fcc8ad9d700 1 --2- 172.21.15.1:0/2193177762 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcc7806af60 0x7fcc7806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:29:47.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.616+0000 7fcc737fe700 1 -- 172.21.15.1:0/2193177762 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcc8c1f7940 con 0x7fcc8c15b6c0 2024-03-18T20:29:47.775 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:47.770+0000 7f97503aa240 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T20:29:47.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.812+0000 7fcc8ad9d700 1 -- 172.21.15.1:0/2193177762 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcc7806af60 msgr2=0x7fcc7806d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:47.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.812+0000 7fcc8ad9d700 1 --2- 172.21.15.1:0/2193177762 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcc7806af60 0x7fcc7806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:29:47.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.865+0000 7fcc91080700 1 -- 172.21.15.1:0/2193177762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc8c1f7940 con 0x7fcc8c15b6c0 2024-03-18T20:29:47.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.865+0000 7fcc737fe700 1 -- 172.21.15.1:0/2193177762 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4750 (secure 0 0 0) 0x7fcc8c1f7940 con 0x7fcc8c15b6c0 2024-03-18T20:29:47.870 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:29:47.870 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":144,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10322,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/2385318877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":2385318877},{"type":"v1","addr":"172.21.15.165:6834","nonce":2385318877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139},{"gid":12040,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1024465047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1024465047},{"type":"v1","addr":"172.21.15.165:6835","nonce":1024465047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139},{"gid":13791,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2374816325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2374816325},{"type":"v1","addr":"172.21.15.1:6837","nonce":2374816325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:28:10.668549+0000","modified":"2024-03-18T20:28:13.687432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13794},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13794":{"gid":13794,"name":"a","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6836/1111156155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1111156155},{"type":"v1","addr":"172.21.15.1:6836","nonce":1111156155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-03-18T20:29:47.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.868+0000 7fcc91080700 1 -- 172.21.15.1:0/2193177762 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcc7806af60 msgr2=0x7fcc7806d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:29:47.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.868+0000 7fcc91080700 1 --2- 172.21.15.1:0/2193177762 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcc7806af60 0x7fcc7806d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.868+0000 7fcc91080700 1 -- 172.21.15.1:0/2193177762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc8c15b6c0 msgr2=0x7fcc8c0b0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:47.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.868+0000 7fcc91080700 1 --2- 172.21.15.1:0/2193177762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc8c15b6c0 0x7fcc8c0b0e50 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c072540 tx=0x7fcc84004ad0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.868+0000 7fcc91080700 1 -- 172.21.15.1:0/2193177762 shutdown_connections 2024-03-18T20:29:47.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.868+0000 7fcc91080700 1 --2- 172.21.15.1:0/2193177762 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fcc7806af60 0x7fcc7806d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.868+0000 7fcc91080700 1 --2- 172.21.15.1:0/2193177762 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fcc8c0cbc90 0x7fcc8c0c52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.868+0000 7fcc91080700 1 --2- 172.21.15.1:0/2193177762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcc8c15b6c0 0x7fcc8c0b0e50 unknown :-1 s=CLOSED pgs=3370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.868+0000 7fcc91080700 1 --2- 172.21.15.1:0/2193177762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc8c0cf200 0x7fcc8c0c6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:47.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.868+0000 7fcc91080700 1 -- 172.21.15.1:0/2193177762 >> 172.21.15.1:0/2193177762 conn(0x7fcc8c0d1180 msgr2=0x7fcc8c0b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:47.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.868+0000 7fcc91080700 1 -- 172.21.15.1:0/2193177762 shutdown_connections 2024-03-18T20:29:47.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:47.868+0000 7fcc91080700 1 -- 172.21.15.1:0/2193177762 wait complete. 2024-03-18T20:29:47.874 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 144 2024-03-18T20:29:47.885 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:29:47.954 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:47.950+0000 7f97503aa240 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:29:47.967 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:47.964+0000 7f972cced240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:29:48.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.008+0000 7f554fe07700 1 Processor -- start 2024-03-18T20:29:48.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.008+0000 7f554fe07700 1 -- start start 2024-03-18T20:29:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.009+0000 7f554fe07700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f55480adc50 0x7f55480b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.009+0000 7f554fe07700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5548099b30 0x7f55480ae030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.009+0000 7f554fe07700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f554815b610 0x7f554806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.009+0000 7f554fe07700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55480b2c20 con 0x7f5548099b30 2024-03-18T20:29:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.009+0000 7f554fe07700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55480ae570 con 0x7f55480adc50 2024-03-18T20:29:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.009+0000 7f554fe07700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554806f860 con 0x7f554815b610 2024-03-18T20:29:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.009+0000 7f554d3a2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5548099b30 0x7f55480ae030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.009+0000 7f554d3a2700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5548099b30 0x7f55480ae030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60128/0 (socket says 172.21.15.1:60128) 2024-03-18T20:29:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.009+0000 7f554d3a2700 1 -- 172.21.15.1:0/1852510307 learned_addr learned my addr 172.21.15.1:0/1852510307 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.009+0000 7f554dba3700 1 --2- 172.21.15.1:0/1852510307 >> v2:172.21.15.165:3300/0 conn(0x7f55480adc50 0x7f55480b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.009+0000 7f554e3a4700 1 --2- 172.21.15.1:0/1852510307 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f554815b610 0x7f554806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.010+0000 7f554d3a2700 1 -- 172.21.15.1:0/1852510307 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f554815b610 msgr2=0x7f554806a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:48.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.010+0000 7f554d3a2700 1 --2- 172.21.15.1:0/1852510307 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f554815b610 0x7f554806a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.010+0000 7f554d3a2700 1 -- 172.21.15.1:0/1852510307 >> v2:172.21.15.165:3300/0 conn(0x7f55480adc50 msgr2=0x7f55480b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:48.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.010+0000 7f554d3a2700 1 --2- 172.21.15.1:0/1852510307 >> v2:172.21.15.165:3300/0 conn(0x7f55480adc50 0x7f55480b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.010+0000 7f554d3a2700 1 -- 172.21.15.1:0/1852510307 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55480ad3c0 con 0x7f5548099b30 2024-03-18T20:29:48.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.010+0000 7f554d3a2700 1 --2- 172.21.15.1:0/1852510307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5548099b30 0x7f55480ae030 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7f5548071320 tx=0x7f5538009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb5622a53ab8474 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:48.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.010+0000 7f554cba1700 1 -- 172.21.15.1:0/1852510307 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5538011830 con 0x7f5548099b30 2024-03-18T20:29:48.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.010+0000 7f554cba1700 1 -- 172.21.15.1:0/1852510307 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5538011990 con 0x7f5548099b30 2024-03-18T20:29:48.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.010+0000 7f554cba1700 1 -- 172.21.15.1:0/1852510307 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5538011d20 con 0x7f5548099b30 2024-03-18T20:29:48.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.011+0000 7f554fe07700 1 -- 172.21.15.1:0/1852510307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5548099b30 msgr2=0x7f55480ae030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:48.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.011+0000 7f554fe07700 1 --2- 172.21.15.1:0/1852510307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5548099b30 0x7f55480ae030 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7f5548071320 tx=0x7f5538009bd0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.011+0000 7f554fe07700 1 -- 172.21.15.1:0/1852510307 shutdown_connections 2024-03-18T20:29:48.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.011+0000 7f554fe07700 1 --2- 172.21.15.1:0/1852510307 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f554815b610 0x7f554806a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.011+0000 7f554fe07700 1 --2- 172.21.15.1:0/1852510307 >> v2:172.21.15.165:3300/0 conn(0x7f55480adc50 0x7f55480b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.011+0000 7f554fe07700 1 --2- 172.21.15.1:0/1852510307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5548099b30 0x7f55480ae030 unknown :-1 s=CLOSED pgs=3371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.011+0000 7f554fe07700 1 -- 172.21.15.1:0/1852510307 >> 172.21.15.1:0/1852510307 conn(0x7f55480b5a10 msgr2=0x7f55480b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:48.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.011+0000 7f554fe07700 1 -- 172.21.15.1:0/1852510307 shutdown_connections 2024-03-18T20:29:48.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.011+0000 7f554fe07700 1 -- 172.21.15.1:0/1852510307 wait complete. 2024-03-18T20:29:48.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.012+0000 7f554fe07700 1 Processor -- start 2024-03-18T20:29:48.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.012+0000 7f554fe07700 1 -- start start 2024-03-18T20:29:48.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.013+0000 7f554fe07700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5548099b30 0x7f5548151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:48.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.013+0000 7f554fe07700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55480adc50 0x7f55481e3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:48.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.013+0000 7f554fe07700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f554815b610 0x7f554809a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:48.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.013+0000 7f554fe07700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55480b3ef0 con 0x7f55480adc50 2024-03-18T20:29:48.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.013+0000 7f554fe07700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5548152500 con 0x7f554815b610 2024-03-18T20:29:48.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.013+0000 7f554fe07700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55480b3550 con 0x7f5548099b30 2024-03-18T20:29:48.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.013+0000 7f554d3a2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55480adc50 0x7f55481e3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.013+0000 7f554d3a2700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55480adc50 0x7f55481e3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60140/0 (socket says 172.21.15.1:60140) 2024-03-18T20:29:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.013+0000 7f554d3a2700 1 -- 172.21.15.1:0/265297880 learned_addr learned my addr 172.21.15.1:0/265297880 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.013+0000 7f554dba3700 1 --2- 172.21.15.1:0/265297880 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5548099b30 0x7f5548151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.013+0000 7f554e3a4700 1 --2- 172.21.15.1:0/265297880 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f554815b610 0x7f554809a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.013+0000 7f554d3a2700 1 -- 172.21.15.1:0/265297880 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5548099b30 msgr2=0x7f5548151fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.013+0000 7f554d3a2700 1 --2- 172.21.15.1:0/265297880 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5548099b30 0x7f5548151fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.013+0000 7f554d3a2700 1 -- 172.21.15.1:0/265297880 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f554815b610 msgr2=0x7f554809a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.013+0000 7f554d3a2700 1 --2- 172.21.15.1:0/265297880 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f554815b610 0x7f554809a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.013+0000 7f554d3a2700 1 -- 172.21.15.1:0/265297880 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5538009550 con 0x7f55480adc50 2024-03-18T20:29:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.013+0000 7f554d3a2700 1 --2- 172.21.15.1:0/265297880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55480adc50 0x7f55481e3840 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 crypto rx=0x7f5538000c00 tx=0x7f5538004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.014+0000 7f553e7fc700 1 -- 172.21.15.1:0/265297880 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5538004240 con 0x7f55480adc50 2024-03-18T20:29:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.014+0000 7f553e7fc700 1 -- 172.21.15.1:0/265297880 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553801f070 con 0x7f55480adc50 2024-03-18T20:29:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.014+0000 7f554fe07700 1 -- 172.21.15.1:0/265297880 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55480b3830 con 0x7f55480adc50 2024-03-18T20:29:48.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.014+0000 7f553e7fc700 1 -- 172.21.15.1:0/265297880 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55380036a0 con 0x7f55480adc50 2024-03-18T20:29:48.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.014+0000 7f554fe07700 1 -- 172.21.15.1:0/265297880 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55481e3d80 con 0x7f55480adc50 2024-03-18T20:29:48.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.015+0000 7f554fe07700 1 -- 172.21.15.1:0/265297880 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55480b4030 con 0x7f55480adc50 2024-03-18T20:29:48.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.015+0000 7f553e7fc700 1 -- 172.21.15.1:0/265297880 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 175) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f5538021030 con 0x7f55480adc50 2024-03-18T20:29:48.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.016+0000 7f553e7fc700 1 --2- 172.21.15.1:0/265297880 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f552c06af60 0x7f552c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:48.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.016+0000 7f554dba3700 1 -- 172.21.15.1:0/265297880 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f552c06af60 msgr2=0x7f552c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:48.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.016+0000 7f553e7fc700 1 -- 172.21.15.1:0/265297880 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(291..291 src has 1..291) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f553808adb0 con 0x7f55480adc50 2024-03-18T20:29:48.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.016+0000 7f554dba3700 1 --2- 172.21.15.1:0/265297880 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f552c06af60 0x7f552c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:29:48.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.020+0000 7f553e7fc700 1 -- 172.21.15.1:0/265297880 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5538055a80 con 0x7f55480adc50 2024-03-18T20:29:48.134 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:48.130+0000 7f972cced240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:29:48.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.217+0000 7f554dba3700 1 -- 172.21.15.1:0/265297880 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f552c06af60 msgr2=0x7f552c06d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:48.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.217+0000 7f554dba3700 1 --2- 172.21.15.1:0/265297880 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f552c06af60 0x7f552c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:29:48.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.222+0000 7f554fe07700 1 -- 172.21.15.1:0/265297880 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55480b4030 con 0x7f55480adc50 2024-03-18T20:29:48.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.223+0000 7f553e7fc700 1 -- 172.21.15.1:0/265297880 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v291) v1 ==== 74+0+16996 (secure 0 0 0) 0x7f55480b4030 con 0x7f55480adc50 2024-03-18T20:29:48.229 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:29:48.229 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":291,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:29:27.236798+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:28:08.503736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"287","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-03-18T20:28:09.644774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"288","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":290,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":290,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"38.2","mappings":[{"from":7,"to":2}]},{"pgid":"38.8","mappings":[{"from":7,"to":4}]},{"pgid":"38.12","mappings":[{"from":7,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/1650741459":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/2451665032":"2024-03-19T20:28:26.019720+0000","172.21.15.1:6833/148356":"2024-03-19T20:28:26.019720+0000","172.21.15.1:6832/148356":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/1408878463":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/582749307":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/997400106":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/2561447959":"2024-03-19T20:28:26.019720+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:29:48.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.226+0000 7f554fe07700 1 -- 172.21.15.1:0/265297880 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f552c06af60 msgr2=0x7f552c06d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:29:48.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.226+0000 7f554fe07700 1 --2- 172.21.15.1:0/265297880 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f552c06af60 0x7f552c06d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.226+0000 7f554fe07700 1 -- 172.21.15.1:0/265297880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55480adc50 msgr2=0x7f55481e3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:48.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.226+0000 7f554fe07700 1 --2- 172.21.15.1:0/265297880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55480adc50 0x7f55481e3840 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 crypto rx=0x7f5538000c00 tx=0x7f5538004750 comp rx=0 tx=0).stop 2024-03-18T20:29:48.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.226+0000 7f554fe07700 1 -- 172.21.15.1:0/265297880 shutdown_connections 2024-03-18T20:29:48.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.226+0000 7f554fe07700 1 --2- 172.21.15.1:0/265297880 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f552c06af60 0x7f552c06d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.226+0000 7f554fe07700 1 --2- 172.21.15.1:0/265297880 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5548099b30 0x7f5548151fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.226+0000 7f554fe07700 1 --2- 172.21.15.1:0/265297880 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f554815b610 0x7f554809a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.226+0000 7f554fe07700 1 --2- 172.21.15.1:0/265297880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55480adc50 0x7f55481e3840 unknown :-1 s=CLOSED pgs=3372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.226+0000 7f554fe07700 1 -- 172.21.15.1:0/265297880 >> 172.21.15.1:0/265297880 conn(0x7f55480b5a10 msgr2=0x7f5548154df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.226+0000 7f554fe07700 1 -- 172.21.15.1:0/265297880 shutdown_connections 2024-03-18T20:29:48.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.226+0000 7f554fe07700 1 -- 172.21.15.1:0/265297880 wait complete. 2024-03-18T20:29:48.244 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T20:29:48.244 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:29:48.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.367+0000 7f7f588e1700 1 Processor -- start 2024-03-18T20:29:48.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.367+0000 7f7f588e1700 1 -- start start 2024-03-18T20:29:48.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.368+0000 7f7f588e1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f540d3110 0x7f7f540d3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:48.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.368+0000 7f7f588e1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7f540b0530 0x7f7f540b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:48.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.368+0000 7f7f588e1700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f540b00a0 con 0x7f7f5415d8c0 2024-03-18T20:29:48.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.368+0000 7f7f588e1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f540d3ad0 con 0x7f7f540b0530 2024-03-18T20:29:48.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.368+0000 7f7f588e1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f540d1f40 con 0x7f7f540d3110 2024-03-18T20:29:48.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.368+0000 7f7f52d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7f540b0530 0x7f7f540b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:48.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.368+0000 7f7f52d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7f540b0530 0x7f7f540b09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60152/0 (socket says 172.21.15.1:60152) 2024-03-18T20:29:48.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.368+0000 7f7f52d9d700 1 -- 172.21.15.1:0/3303475139 learned_addr learned my addr 172.21.15.1:0/3303475139 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:48.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.368+0000 7f7f51d9b700 1 --2- 172.21.15.1:0/3303475139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f540d3110 0x7f7f540d3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:48.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.368+0000 7f7f52d9d700 1 -- 172.21.15.1:0/3303475139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f540d3110 msgr2=0x7f7f540d3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:48.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.369+0000 7f7f52d9d700 1 --2- 172.21.15.1:0/3303475139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f540d3110 0x7f7f540d3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.369+0000 7f7f52d9d700 1 -- 172.21.15.1:0/3303475139 >> v1:172.21.15.165:6789/0 conn(0x7f7f5415d8c0 legacy=0x7f7f540b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:48.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.369+0000 7f7f52d9d700 1 -- 172.21.15.1:0/3303475139 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f540b4260 con 0x7f7f540b0530 2024-03-18T20:29:48.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.369+0000 7f7f52d9d700 1 --2- 172.21.15.1:0/3303475139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f540b0530 0x7f7f540b09c0 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c009e50 tx=0x7f7f4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d2f9d79ad627dd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:48.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.369+0000 7f7f5159a700 1 -- 172.21.15.1:0/3303475139 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c00e070 con 0x7f7f540b0530 2024-03-18T20:29:48.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.369+0000 7f7f5159a700 1 -- 172.21.15.1:0/3303475139 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4c00b910 con 0x7f7f540b0530 2024-03-18T20:29:48.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.369+0000 7f7f5159a700 1 -- 172.21.15.1:0/3303475139 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c00bcc0 con 0x7f7f540b0530 2024-03-18T20:29:48.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.369+0000 7f7f588e1700 1 -- 172.21.15.1:0/3303475139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f540b0530 msgr2=0x7f7f540b09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:48.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.369+0000 7f7f588e1700 1 --2- 172.21.15.1:0/3303475139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f540b0530 0x7f7f540b09c0 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c009e50 tx=0x7f7f4c00b330 comp rx=0 tx=0).stop 2024-03-18T20:29:48.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.370+0000 7f7f588e1700 1 -- 172.21.15.1:0/3303475139 shutdown_connections 2024-03-18T20:29:48.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.370+0000 7f7f588e1700 1 --2- 172.21.15.1:0/3303475139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f540b0530 0x7f7f540b09c0 unknown :-1 s=CLOSED pgs=3373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.370+0000 7f7f588e1700 1 --2- 172.21.15.1:0/3303475139 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f540d3110 0x7f7f540d3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.370+0000 7f7f588e1700 1 -- 172.21.15.1:0/3303475139 >> 172.21.15.1:0/3303475139 conn(0x7f7f5408afc0 msgr2=0x7f7f5409c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:48.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.370+0000 7f7f588e1700 1 -- 172.21.15.1:0/3303475139 shutdown_connections 2024-03-18T20:29:48.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.370+0000 7f7f588e1700 1 -- 172.21.15.1:0/3303475139 wait complete. 2024-03-18T20:29:48.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.371+0000 7f7f588e1700 1 Processor -- start 2024-03-18T20:29:48.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.371+0000 7f7f588e1700 1 -- start start 2024-03-18T20:29:48.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.372+0000 7f7f588e1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f540b0530 0x7f7f5415ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:48.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.372+0000 7f7f588e1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f540d3110 0x7f7f5415e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:48.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.372+0000 7f7f588e1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f5415d8c0 0x7f7f540ca640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:48.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.372+0000 7f7f588e1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5415f7a0 con 0x7f7f540b0530 2024-03-18T20:29:48.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.372+0000 7f7f588e1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5415fc30 con 0x7f7f540d3110 2024-03-18T20:29:48.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.372+0000 7f7f588e1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5415eb90 con 0x7f7f5415d8c0 2024-03-18T20:29:48.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.372+0000 7f7f5259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f540b0530 0x7f7f5415ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:48.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.372+0000 7f7f5259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f540b0530 0x7f7f5415ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60154/0 (socket says 172.21.15.1:60154) 2024-03-18T20:29:48.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.372+0000 7f7f5259c700 1 -- 172.21.15.1:0/2553149045 learned_addr learned my addr 172.21.15.1:0/2553149045 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:48.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.372+0000 7f7f52d9d700 1 --2- 172.21.15.1:0/2553149045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f5415d8c0 0x7f7f540ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:48.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.372+0000 7f7f51d9b700 1 --2- 172.21.15.1:0/2553149045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f540d3110 0x7f7f5415e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:48.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.372+0000 7f7f5259c700 1 -- 172.21.15.1:0/2553149045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f5415d8c0 msgr2=0x7f7f540ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:48.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.372+0000 7f7f5259c700 1 --2- 172.21.15.1:0/2553149045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f5415d8c0 0x7f7f540ca640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.372+0000 7f7f5259c700 1 -- 172.21.15.1:0/2553149045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f540d3110 msgr2=0x7f7f5415e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:48.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.372+0000 7f7f5259c700 1 --2- 172.21.15.1:0/2553149045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f540d3110 0x7f7f5415e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.372+0000 7f7f5259c700 1 -- 172.21.15.1:0/2553149045 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c009b30 con 0x7f7f540b0530 2024-03-18T20:29:48.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.372+0000 7f7f5259c700 1 --2- 172.21.15.1:0/2553149045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f540b0530 0x7f7f5415ba40 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 crypto rx=0x7f7f4400c450 tx=0x7f7f4400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:48.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.373+0000 7f7f42ffd700 1 -- 172.21.15.1:0/2553149045 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f440052d0 con 0x7f7f540b0530 2024-03-18T20:29:48.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.373+0000 7f7f42ffd700 1 -- 172.21.15.1:0/2553149045 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f44005430 con 0x7f7f540b0530 2024-03-18T20:29:48.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.373+0000 7f7f588e1700 1 -- 172.21.15.1:0/2553149045 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f5415ee70 con 0x7f7f540b0530 2024-03-18T20:29:48.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.373+0000 7f7f42ffd700 1 -- 172.21.15.1:0/2553149045 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f440038c0 con 0x7f7f540b0530 2024-03-18T20:29:48.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.373+0000 7f7f588e1700 1 -- 172.21.15.1:0/2553149045 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f5415f430 con 0x7f7f540b0530 2024-03-18T20:29:48.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.374+0000 7f7f588e1700 1 -- 172.21.15.1:0/2553149045 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f540676b0 con 0x7f7f540b0530 2024-03-18T20:29:48.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.374+0000 7f7f42ffd700 1 -- 172.21.15.1:0/2553149045 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 175) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f7f44003a20 con 0x7f7f540b0530 2024-03-18T20:29:48.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.375+0000 7f7f42ffd700 1 --2- 172.21.15.1:0/2553149045 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7f3406af60 0x7f7f3406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:48.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.375+0000 7f7f51d9b700 1 -- 172.21.15.1:0/2553149045 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7f3406af60 msgr2=0x7f7f3406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:48.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.375+0000 7f7f42ffd700 1 -- 172.21.15.1:0/2553149045 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(291..291 src has 1..291) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f7f44088f00 con 0x7f7f540b0530 2024-03-18T20:29:48.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.375+0000 7f7f51d9b700 1 --2- 172.21.15.1:0/2553149045 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7f3406af60 0x7f7f3406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:29:48.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.379+0000 7f7f42ffd700 1 -- 172.21.15.1:0/2553149045 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7f44053b40 con 0x7f7f540b0530 2024-03-18T20:29:48.386 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:48.382+0000 7f97503aa240 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T20:29:48.499 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:48.495+0000 7f972cced240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:29:48.566 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:48.562+0000 7f97503aa240 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T20:29:48.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.576+0000 7f7f51d9b700 1 -- 172.21.15.1:0/2553149045 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7f3406af60 msgr2=0x7f7f3406d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:48.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.576+0000 7f7f51d9b700 1 --2- 172.21.15.1:0/2553149045 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7f3406af60 0x7f7f3406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:29:48.588 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-03-18T20:29:48.584+0000 7f972cced240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:29:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.626+0000 7f7f588e1700 1 -- 172.21.15.1:0/2553149045 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f540c7eb0 con 0x7f7f540b0530 2024-03-18T20:29:48.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.627+0000 7f7f42ffd700 1 -- 172.21.15.1:0/2553149045 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4750 (secure 0 0 0) 0x7f7f4400ce60 con 0x7f7f540b0530 2024-03-18T20:29:48.632 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:29:48.632 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":144,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10322,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/2385318877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":2385318877},{"type":"v1","addr":"172.21.15.165:6834","nonce":2385318877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139},{"gid":12040,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1024465047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1024465047},{"type":"v1","addr":"172.21.15.165:6835","nonce":1024465047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139},{"gid":13791,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2374816325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2374816325},{"type":"v1","addr":"172.21.15.1:6837","nonce":2374816325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:28:10.668549+0000","modified":"2024-03-18T20:28:13.687432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13794},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13794":{"gid":13794,"name":"a","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6836/1111156155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1111156155},{"type":"v1","addr":"172.21.15.1:6836","nonce":1111156155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-03-18T20:29:48.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.629+0000 7f7f588e1700 1 -- 172.21.15.1:0/2553149045 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7f3406af60 msgr2=0x7f7f3406d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:29:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.629+0000 7f7f588e1700 1 --2- 172.21.15.1:0/2553149045 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7f3406af60 0x7f7f3406d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.629+0000 7f7f588e1700 1 -- 172.21.15.1:0/2553149045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f540b0530 msgr2=0x7f7f5415ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.629+0000 7f7f588e1700 1 --2- 172.21.15.1:0/2553149045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f540b0530 0x7f7f5415ba40 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 crypto rx=0x7f7f4400c450 tx=0x7f7f4400c760 comp rx=0 tx=0).stop 2024-03-18T20:29:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.629+0000 7f7f588e1700 1 -- 172.21.15.1:0/2553149045 shutdown_connections 2024-03-18T20:29:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.629+0000 7f7f588e1700 1 --2- 172.21.15.1:0/2553149045 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7f3406af60 0x7f7f3406d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.629+0000 7f7f588e1700 1 --2- 172.21.15.1:0/2553149045 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7f5415d8c0 0x7f7f540ca640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.629+0000 7f7f588e1700 1 --2- 172.21.15.1:0/2553149045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f540b0530 0x7f7f5415ba40 unknown :-1 s=CLOSED pgs=3374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.629+0000 7f7f588e1700 1 --2- 172.21.15.1:0/2553149045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f540d3110 0x7f7f5415e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.629+0000 7f7f588e1700 1 -- 172.21.15.1:0/2553149045 >> 172.21.15.1:0/2553149045 conn(0x7f7f5408afc0 msgr2=0x7f7f540c6f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.629+0000 7f7f588e1700 1 -- 172.21.15.1:0/2553149045 shutdown_connections 2024-03-18T20:29:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.629+0000 7f7f588e1700 1 -- 172.21.15.1:0/2553149045 wait complete. 2024-03-18T20:29:48.635 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 144 2024-03-18T20:29:48.647 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:29:48.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.772+0000 7f5764a09700 1 Processor -- start 2024-03-18T20:29:48.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.772+0000 7f5764a09700 1 -- start start 2024-03-18T20:29:48.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.773+0000 7f5764a09700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57600cbc90 0x7f576015c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:48.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.773+0000 7f5764a09700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576015ca40 con 0x7f576015b6c0 2024-03-18T20:29:48.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.773+0000 7f5764a09700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57600cce30 con 0x7f57600cf200 2024-03-18T20:29:48.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.773+0000 7f5764a09700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57600cc220 con 0x7f57600cbc90 2024-03-18T20:29:48.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.773+0000 7f575dd9b700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f576015b6c0 0x7f5760099ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:43992/0 (socket says 172.21.15.1:43992) 2024-03-18T20:29:48.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.773+0000 7f575dd9b700 1 -- 172.21.15.1:0/1142670502 learned_addr learned my addr 172.21.15.1:0/1142670502 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:29:48.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.773+0000 7f575ed9d700 1 --2- 172.21.15.1:0/1142670502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57600cbc90 0x7f576015c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:48.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.773+0000 7f575d59a700 1 -- 172.21.15.1:0/1142670502 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010263535 0 0) 0x7f5754001110 con 0x7f576015b6c0 2024-03-18T20:29:48.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.774+0000 7f575d59a700 1 -- 172.21.15.1:0/1142670502 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57480035b0 con 0x7f576015b6c0 2024-03-18T20:29:48.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.774+0000 7f575d59a700 1 -- 172.21.15.1:0/1142670502 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062448808 0 0) 0x7f57600cce30 con 0x7f57600cf200 2024-03-18T20:29:48.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.774+0000 7f575d59a700 1 -- 172.21.15.1:0/1142670502 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5754001110 con 0x7f57600cf200 2024-03-18T20:29:48.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.774+0000 7f575ed9d700 1 -- 172.21.15.1:0/1142670502 >> v1:172.21.15.165:6789/0 conn(0x7f57600cf200 legacy=0x7f57600cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:48.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.774+0000 7f575ed9d700 1 -- 172.21.15.1:0/1142670502 >> v1:172.21.15.1:6789/0 conn(0x7f576015b6c0 legacy=0x7f5760099ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:48.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.774+0000 7f575d59a700 1 -- 172.21.15.1:0/1142670502 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4273312338 0 0) 0x7f57480035b0 con 0x7f576015b6c0 2024-03-18T20:29:48.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.774+0000 7f575ed9d700 1 -- 172.21.15.1:0/1142670502 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57600cc500 con 0x7f57600cbc90 2024-03-18T20:29:48.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.774+0000 7f575ed9d700 1 --2- 172.21.15.1:0/1142670502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57600cbc90 0x7f576015c500 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5758009dc0 tx=0x7f575800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28ac968312e937e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:48.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.774+0000 7f575d59a700 1 -- 172.21.15.1:0/1142670502 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575800e070 con 0x7f57600cbc90 2024-03-18T20:29:48.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.774+0000 7f575d59a700 1 -- 172.21.15.1:0/1142670502 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5758003a20 con 0x7f57600cbc90 2024-03-18T20:29:48.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.774+0000 7f575d59a700 1 -- 172.21.15.1:0/1142670502 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5754001800 con 0x7f57600cbc90 2024-03-18T20:29:48.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.775+0000 7f5764a09700 1 -- 172.21.15.1:0/1142670502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57600cbc90 msgr2=0x7f576015c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:48.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.775+0000 7f5764a09700 1 --2- 172.21.15.1:0/1142670502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57600cbc90 0x7f576015c500 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5758009dc0 tx=0x7f575800b330 comp rx=0 tx=0).stop 2024-03-18T20:29:48.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.775+0000 7f5764a09700 1 -- 172.21.15.1:0/1142670502 shutdown_connections 2024-03-18T20:29:48.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.775+0000 7f5764a09700 1 --2- 172.21.15.1:0/1142670502 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57600cbc90 0x7f576015c500 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.775+0000 7f5764a09700 1 -- 172.21.15.1:0/1142670502 >> 172.21.15.1:0/1142670502 conn(0x7f57600d1180 msgr2=0x7f57600d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:48.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.775+0000 7f5764a09700 1 -- 172.21.15.1:0/1142670502 shutdown_connections 2024-03-18T20:29:48.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.775+0000 7f5764a09700 1 -- 172.21.15.1:0/1142670502 wait complete. 2024-03-18T20:29:48.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.777+0000 7f5764a09700 1 Processor -- start 2024-03-18T20:29:48.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.777+0000 7f5764a09700 1 -- start start 2024-03-18T20:29:48.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.778+0000 7f5764a09700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57600cbc90 0x7f57600b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:48.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.778+0000 7f5764a09700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57600cf200 0x7f57600c92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:48.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.778+0000 7f5764a09700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f576015b6c0 0x7f57601538d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:48.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.778+0000 7f5764a09700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57600c9ad0 con 0x7f57600cbc90 2024-03-18T20:29:48.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.778+0000 7f5764a09700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57600c9db0 con 0x7f576015b6c0 2024-03-18T20:29:48.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.778+0000 7f5764a09700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57600ca090 con 0x7f57600cf200 2024-03-18T20:29:48.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.778+0000 7f575e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57600cbc90 0x7f57600b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:48.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.778+0000 7f575e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57600cbc90 0x7f57600b4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60162/0 (socket says 172.21.15.1:60162) 2024-03-18T20:29:48.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.778+0000 7f575e59c700 1 -- 172.21.15.1:0/580716382 learned_addr learned my addr 172.21.15.1:0/580716382 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:48.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.778+0000 7f575ed9d700 1 --2- 172.21.15.1:0/580716382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f576015b6c0 0x7f57601538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:48.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.778+0000 7f575dd9b700 1 --2- 172.21.15.1:0/580716382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57600cf200 0x7f57600c92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:48.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.779+0000 7f575e59c700 1 -- 172.21.15.1:0/580716382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57600cf200 msgr2=0x7f57600c92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:48.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.779+0000 7f575e59c700 1 --2- 172.21.15.1:0/580716382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57600cf200 0x7f57600c92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.779+0000 7f575e59c700 1 -- 172.21.15.1:0/580716382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f576015b6c0 msgr2=0x7f57601538d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:48.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.779+0000 7f575e59c700 1 --2- 172.21.15.1:0/580716382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f576015b6c0 0x7f57601538d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:48.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.779+0000 7f575e59c700 1 -- 172.21.15.1:0/580716382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5758009a70 con 0x7f57600cbc90 2024-03-18T20:29:48.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.779+0000 7f575e59c700 1 --2- 172.21.15.1:0/580716382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57600cbc90 0x7f57600b4080 secure :-1 s=READY pgs=3376 cs=0 l=1 rev1=1 crypto rx=0x7f57500011e0 tx=0x7f575000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:48.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.779+0000 7f574effd700 1 -- 172.21.15.1:0/580716382 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57500054e0 con 0x7f57600cbc90 2024-03-18T20:29:48.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.779+0000 7f574effd700 1 -- 172.21.15.1:0/580716382 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5750003680 con 0x7f57600cbc90 2024-03-18T20:29:48.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.779+0000 7f5764a09700 1 -- 172.21.15.1:0/580716382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57600ca370 con 0x7f57600cbc90 2024-03-18T20:29:48.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.779+0000 7f574effd700 1 -- 172.21.15.1:0/580716382 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5750003a30 con 0x7f57600cbc90 2024-03-18T20:29:48.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.779+0000 7f5764a09700 1 -- 172.21.15.1:0/580716382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57600c97e0 con 0x7f57600cbc90 2024-03-18T20:29:48.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.780+0000 7f5764a09700 1 -- 172.21.15.1:0/580716382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57600d0ca0 con 0x7f57600cbc90 2024-03-18T20:29:48.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.781+0000 7f574effd700 1 -- 172.21.15.1:0/580716382 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 175) v1 ==== 89404+0+0 (secure 0 0 0) 0x7f5750003b90 con 0x7f57600cbc90 2024-03-18T20:29:48.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.781+0000 7f574effd700 1 --2- 172.21.15.1:0/580716382 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f574006af60 0x7f574006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:48.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.781+0000 7f575dd9b700 1 -- 172.21.15.1:0/580716382 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f574006af60 msgr2=0x7f574006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:48.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.781+0000 7f574effd700 1 -- 172.21.15.1:0/580716382 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(291..291 src has 1..291) v4 ==== 6560+0+0 (secure 0 0 0) 0x7f5750088f80 con 0x7f57600cbc90 2024-03-18T20:29:48.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.781+0000 7f575dd9b700 1 --2- 172.21.15.1:0/580716382 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f574006af60 0x7f574006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T20:29:48.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.789+0000 7f574effd700 1 -- 172.21.15.1:0/580716382 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5750053a40 con 0x7f57600cbc90 2024-03-18T20:29:48.956 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:48.951+0000 7f97503aa240 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T20:29:48.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.982+0000 7f575dd9b700 1 -- 172.21.15.1:0/580716382 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f574006af60 msgr2=0x7f574006d420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6832/148356 2024-03-18T20:29:48.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:48.982+0000 7f575dd9b700 1 --2- 172.21.15.1:0/580716382 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f574006af60 0x7f574006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T20:29:49.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.037+0000 7f5764a09700 1 -- 172.21.15.1:0/580716382 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57600ca890 con 0x7f57600cbc90 2024-03-18T20:29:49.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.038+0000 7f574effd700 1 -- 172.21.15.1:0/580716382 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4750 (secure 0 0 0) 0x7f5750057060 con 0x7f57600cbc90 2024-03-18T20:29:49.042 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:29:49.043 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":144,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10322,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/2385318877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":2385318877},{"type":"v1","addr":"172.21.15.165:6834","nonce":2385318877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139},{"gid":12040,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1024465047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1024465047},{"type":"v1","addr":"172.21.15.165:6835","nonce":1024465047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139},{"gid":13791,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2374816325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2374816325},{"type":"v1","addr":"172.21.15.1:6837","nonce":2374816325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T20:28:10.668549+0000","modified":"2024-03-18T20:28:13.687432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":13794},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13794":{"gid":13794,"name":"a","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.1:6836/1111156155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":1111156155},{"type":"v1","addr":"172.21.15.1:6836","nonce":1111156155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[38],"metadata_pool":37,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":18}]} 2024-03-18T20:29:49.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.040+0000 7f5764a09700 1 -- 172.21.15.1:0/580716382 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f574006af60 msgr2=0x7f574006d420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T20:29:49.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.040+0000 7f5764a09700 1 --2- 172.21.15.1:0/580716382 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f574006af60 0x7f574006d420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.040+0000 7f5764a09700 1 -- 172.21.15.1:0/580716382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57600cbc90 msgr2=0x7f57600b4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:49.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.040+0000 7f5764a09700 1 --2- 172.21.15.1:0/580716382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57600cbc90 0x7f57600b4080 secure :-1 s=READY pgs=3376 cs=0 l=1 rev1=1 crypto rx=0x7f57500011e0 tx=0x7f575000ca30 comp rx=0 tx=0).stop 2024-03-18T20:29:49.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.040+0000 7f5764a09700 1 -- 172.21.15.1:0/580716382 shutdown_connections 2024-03-18T20:29:49.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.040+0000 7f5764a09700 1 --2- 172.21.15.1:0/580716382 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f574006af60 0x7f574006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.040+0000 7f5764a09700 1 --2- 172.21.15.1:0/580716382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f576015b6c0 0x7f57601538d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.040+0000 7f5764a09700 1 --2- 172.21.15.1:0/580716382 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57600cbc90 0x7f57600b4080 unknown :-1 s=CLOSED pgs=3376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.040+0000 7f5764a09700 1 --2- 172.21.15.1:0/580716382 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f57600cf200 0x7f57600c92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.040+0000 7f5764a09700 1 -- 172.21.15.1:0/580716382 >> 172.21.15.1:0/580716382 conn(0x7f57600d1180 msgr2=0x7f5760099ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:49.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.040+0000 7f5764a09700 1 -- 172.21.15.1:0/580716382 shutdown_connections 2024-03-18T20:29:49.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.040+0000 7f5764a09700 1 -- 172.21.15.1:0/580716382 wait complete. 2024-03-18T20:29:49.046 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 144 2024-03-18T20:29:49.053 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:49.047+0000 7f97503aa240 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T20:29:49.057 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:29:49.167 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:49.160+0000 7f96c87d2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:29:49.167 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:49.160+0000 7f96c87d2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:29:49.167 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:49.160+0000 7f96c87d2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:29:49.167 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:49.160+0000 7f96c87d2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:29:49.167 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:49.160+0000 7f96c87d2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:29:49.167 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:49.160+0000 7f96cb7d8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:29:49.167 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:49.161+0000 7f96cb7d8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:29:49.167 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:49.161+0000 7f96cb7d8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:29:49.168 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:49.161+0000 7f96cb7d8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:29:49.168 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-03-18T20:29:49.161+0000 7f96cb7d8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T20:29:49.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.212+0000 7f895e026700 1 Processor -- start 2024-03-18T20:29:49.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.212+0000 7f895e026700 1 -- start start 2024-03-18T20:29:49.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.213+0000 7f895e026700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f89580adc50 0x7f89580b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:49.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.213+0000 7f895e026700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8958099b30 0x7f89580ae030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:49.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.213+0000 7f895e026700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f895815b610 0x7f895806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:49.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.213+0000 7f895e026700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89580b2c20 con 0x7f8958099b30 2024-03-18T20:29:49.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.213+0000 7f895e026700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89580ae570 con 0x7f89580adc50 2024-03-18T20:29:49.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.213+0000 7f895e026700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f895806f860 con 0x7f895815b610 2024-03-18T20:29:49.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.213+0000 7f8956ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8958099b30 0x7f89580ae030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:49.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.213+0000 7f8956ffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8958099b30 0x7f89580ae030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60286/0 (socket says 172.21.15.1:60286) 2024-03-18T20:29:49.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.213+0000 7f8956ffd700 1 -- 172.21.15.1:0/1390083132 learned_addr learned my addr 172.21.15.1:0/1390083132 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:49.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.213+0000 7f8957fff700 1 --2- 172.21.15.1:0/1390083132 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f895815b610 0x7f895806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:49.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.213+0000 7f89577fe700 1 --2- 172.21.15.1:0/1390083132 >> v2:172.21.15.165:3300/0 conn(0x7f89580adc50 0x7f89580b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:49.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.213+0000 7f8956ffd700 1 -- 172.21.15.1:0/1390083132 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f895815b610 msgr2=0x7f895806a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:49.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.213+0000 7f8956ffd700 1 --2- 172.21.15.1:0/1390083132 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f895815b610 0x7f895806a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.213+0000 7f8956ffd700 1 -- 172.21.15.1:0/1390083132 >> v2:172.21.15.165:3300/0 conn(0x7f89580adc50 msgr2=0x7f89580b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:49.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.213+0000 7f8956ffd700 1 --2- 172.21.15.1:0/1390083132 >> v2:172.21.15.165:3300/0 conn(0x7f89580adc50 0x7f89580b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.213+0000 7f8956ffd700 1 -- 172.21.15.1:0/1390083132 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89580ad3c0 con 0x7f8958099b30 2024-03-18T20:29:49.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.214+0000 7f8956ffd700 1 --2- 172.21.15.1:0/1390083132 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8958099b30 0x7f89580ae030 secure :-1 s=READY pgs=3382 cs=0 l=1 rev1=1 crypto rx=0x7f8958071320 tx=0x7f8948009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d02eb10d921c7c6f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:49.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.214+0000 7f89567fc700 1 -- 172.21.15.1:0/1390083132 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8948011830 con 0x7f8958099b30 2024-03-18T20:29:49.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.214+0000 7f89567fc700 1 -- 172.21.15.1:0/1390083132 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8948011990 con 0x7f8958099b30 2024-03-18T20:29:49.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.214+0000 7f89567fc700 1 -- 172.21.15.1:0/1390083132 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8948011d20 con 0x7f8958099b30 2024-03-18T20:29:49.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.214+0000 7f895e026700 1 -- 172.21.15.1:0/1390083132 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8958099b30 msgr2=0x7f89580ae030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:49.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.214+0000 7f895e026700 1 --2- 172.21.15.1:0/1390083132 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8958099b30 0x7f89580ae030 secure :-1 s=READY pgs=3382 cs=0 l=1 rev1=1 crypto rx=0x7f8958071320 tx=0x7f8948009bd0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.214+0000 7f895e026700 1 -- 172.21.15.1:0/1390083132 shutdown_connections 2024-03-18T20:29:49.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.214+0000 7f895e026700 1 --2- 172.21.15.1:0/1390083132 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f895815b610 0x7f895806a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.214+0000 7f895e026700 1 --2- 172.21.15.1:0/1390083132 >> v2:172.21.15.165:3300/0 conn(0x7f89580adc50 0x7f89580b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.214+0000 7f895e026700 1 --2- 172.21.15.1:0/1390083132 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8958099b30 0x7f89580ae030 unknown :-1 s=CLOSED pgs=3382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.214+0000 7f895e026700 1 -- 172.21.15.1:0/1390083132 >> 172.21.15.1:0/1390083132 conn(0x7f89580b5a10 msgr2=0x7f89580b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:49.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.214+0000 7f895e026700 1 -- 172.21.15.1:0/1390083132 shutdown_connections 2024-03-18T20:29:49.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.214+0000 7f895e026700 1 -- 172.21.15.1:0/1390083132 wait complete. 2024-03-18T20:29:49.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.215+0000 7f895e026700 1 Processor -- start 2024-03-18T20:29:49.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.216+0000 7f895e026700 1 -- start start 2024-03-18T20:29:49.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.216+0000 7f895e026700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8958099b30 0x7f89581e0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:49.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.216+0000 7f895e026700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89580adc50 0x7f895809ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:49.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.216+0000 7f895e026700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f895815b610 0x7f895813d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:49.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.216+0000 7f895e026700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895813eba0 con 0x7f89580adc50 2024-03-18T20:29:49.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.216+0000 7f895e026700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895809b3a0 con 0x7f895815b610 2024-03-18T20:29:49.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.216+0000 7f895e026700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f895813e270 con 0x7f8958099b30 2024-03-18T20:29:49.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.217+0000 7f8956ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89580adc50 0x7f895809ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:49.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.217+0000 7f8956ffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89580adc50 0x7f895809ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60302/0 (socket says 172.21.15.1:60302) 2024-03-18T20:29:49.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.217+0000 7f8956ffd700 1 -- 172.21.15.1:0/2168682996 learned_addr learned my addr 172.21.15.1:0/2168682996 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:49.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.217+0000 7f8957fff700 1 --2- 172.21.15.1:0/2168682996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f895815b610 0x7f895813d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:49.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.217+0000 7f89577fe700 1 --2- 172.21.15.1:0/2168682996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8958099b30 0x7f89581e0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:49.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.217+0000 7f8956ffd700 1 -- 172.21.15.1:0/2168682996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8958099b30 msgr2=0x7f89581e0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:49.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.217+0000 7f8956ffd700 1 --2- 172.21.15.1:0/2168682996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8958099b30 0x7f89581e0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.217+0000 7f8956ffd700 1 -- 172.21.15.1:0/2168682996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f895815b610 msgr2=0x7f895813d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.217+0000 7f8956ffd700 1 --2- 172.21.15.1:0/2168682996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f895815b610 0x7f895813d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.217+0000 7f8956ffd700 1 -- 172.21.15.1:0/2168682996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8948009550 con 0x7f89580adc50 2024-03-18T20:29:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.217+0000 7f8956ffd700 1 --2- 172.21.15.1:0/2168682996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89580adc50 0x7f895809ae60 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 crypto rx=0x7f8948000c00 tx=0x7f8948004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.217+0000 7f893ffff700 1 -- 172.21.15.1:0/2168682996 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8948004240 con 0x7f89580adc50 2024-03-18T20:29:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.217+0000 7f893ffff700 1 -- 172.21.15.1:0/2168682996 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f894801f070 con 0x7f89580adc50 2024-03-18T20:29:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.217+0000 7f895e026700 1 -- 172.21.15.1:0/2168682996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f895813e550 con 0x7f89580adc50 2024-03-18T20:29:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.217+0000 7f893ffff700 1 -- 172.21.15.1:0/2168682996 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89480036a0 con 0x7f89580adc50 2024-03-18T20:29:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.217+0000 7f895e026700 1 -- 172.21.15.1:0/2168682996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89581e10f0 con 0x7f89580adc50 2024-03-18T20:29:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.218+0000 7f895e026700 1 -- 172.21.15.1:0/2168682996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89580ae840 con 0x7f89580adc50 2024-03-18T20:29:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.219+0000 7f893ffff700 1 -- 172.21.15.1:0/2168682996 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 176) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f8948021030 con 0x7f89580adc50 2024-03-18T20:29:49.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.219+0000 7f893ffff700 1 -- 172.21.15.1:0/2168682996 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 6947+0+0 (secure 0 0 0) 0x7f894808a3e0 con 0x7f89580adc50 2024-03-18T20:29:49.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.224+0000 7f893ffff700 1 -- 172.21.15.1:0/2168682996 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8948055530 con 0x7f89580adc50 2024-03-18T20:29:49.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.428+0000 7f895e026700 1 -- 172.21.15.1:0/2168682996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8958067dc0 con 0x7f89580adc50 2024-03-18T20:29:49.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.429+0000 7f893ffff700 1 -- 172.21.15.1:0/2168682996 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v292) v1 ==== 74+0+17543 (secure 0 0 0) 0x7f8948058b50 con 0x7f89580adc50 2024-03-18T20:29:49.434 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:29:49.434 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":292,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:29:49.061700+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_metadata","create_time":"2024-03-18T20:28:08.503736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"287","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_data","create_time":"2024-03-18T20:28:09.644774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"288","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":290,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":290,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"38.2","mappings":[{"from":7,"to":2}]},{"pgid":"38.8","mappings":[{"from":7,"to":4}]},{"pgid":"38.12","mappings":[{"from":7,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/1196516798":"2024-03-19T20:29:49.061624+0000","172.21.15.1:0/497050575":"2024-03-19T20:29:49.061624+0000","172.21.15.1:0/4210456696":"2024-03-19T20:29:49.061624+0000","172.21.15.1:0/1141901935":"2024-03-19T20:29:49.061624+0000","172.21.15.1:0/4226319191":"2024-03-19T20:29:49.061624+0000","172.21.15.1:0/598999338":"2024-03-19T20:29:49.061624+0000","172.21.15.1:0/1687261463":"2024-03-19T20:29:49.061624+0000","172.21.15.1:0/1650741459":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/2451665032":"2024-03-19T20:28:26.019720+0000","172.21.15.1:6833/148356":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/2778202311":"2024-03-19T20:29:49.061624+0000","172.21.15.1:6832/148356":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/1931029093":"2024-03-19T20:29:49.061624+0000","172.21.15.1:0/1408878463":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/582749307":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/997400106":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/2561447959":"2024-03-19T20:28:26.019720+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:29:49.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.431+0000 7f895e026700 1 -- 172.21.15.1:0/2168682996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89580adc50 msgr2=0x7f895809ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:49.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.431+0000 7f895e026700 1 --2- 172.21.15.1:0/2168682996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89580adc50 0x7f895809ae60 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 crypto rx=0x7f8948000c00 tx=0x7f8948004750 comp rx=0 tx=0).stop 2024-03-18T20:29:49.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.431+0000 7f895e026700 1 -- 172.21.15.1:0/2168682996 shutdown_connections 2024-03-18T20:29:49.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.431+0000 7f895e026700 1 --2- 172.21.15.1:0/2168682996 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8958099b30 0x7f89581e0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.431+0000 7f895e026700 1 --2- 172.21.15.1:0/2168682996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f895815b610 0x7f895813d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.431+0000 7f895e026700 1 --2- 172.21.15.1:0/2168682996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89580adc50 0x7f895809ae60 unknown :-1 s=CLOSED pgs=3383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.431+0000 7f895e026700 1 -- 172.21.15.1:0/2168682996 >> 172.21.15.1:0/2168682996 conn(0x7f89580b5a10 msgr2=0x7f89581518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:49.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.431+0000 7f895e026700 1 -- 172.21.15.1:0/2168682996 shutdown_connections 2024-03-18T20:29:49.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.431+0000 7f895e026700 1 -- 172.21.15.1:0/2168682996 wait complete. 2024-03-18T20:29:49.448 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T20:29:49.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.569+0000 7f0012fa3700 1 Processor -- start 2024-03-18T20:29:49.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.569+0000 7f0012fa3700 1 -- start start 2024-03-18T20:29:49.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.570+0000 7f0012fa3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f000c15d530 0x7f000c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:49.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.570+0000 7f0012fa3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000c0acfb0 0x7f000c0b3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:49.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.570+0000 7f0012fa3700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f000c15d910 con 0x7f000c0ad440 2024-03-18T20:29:49.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.570+0000 7f0012fa3700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f000c06ed10 con 0x7f000c15d530 2024-03-18T20:29:49.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.570+0000 7f0012fa3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000c06d730 con 0x7f000c0acfb0 2024-03-18T20:29:49.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.570+0000 7f0010d3f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f000c15d530 0x7f000c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:49.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.570+0000 7f0010d3f700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f000c15d530 0x7f000c06e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60304/0 (socket says 172.21.15.1:60304) 2024-03-18T20:29:49.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.570+0000 7f0010d3f700 1 -- 172.21.15.1:0/3490536871 learned_addr learned my addr 172.21.15.1:0/3490536871 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:49.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.570+0000 7f0011540700 1 --2- 172.21.15.1:0/3490536871 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000c0acfb0 0x7f000c0b3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:49.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.571+0000 7f0010d3f700 1 -- 172.21.15.1:0/3490536871 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000c0acfb0 msgr2=0x7f000c0b3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:49.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.571+0000 7f0010d3f700 1 --2- 172.21.15.1:0/3490536871 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000c0acfb0 0x7f000c0b3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.571+0000 7f0010d3f700 1 -- 172.21.15.1:0/3490536871 >> v1:172.21.15.165:6789/0 conn(0x7f000c0ad440 legacy=0x7f000c0ac8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:49.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.571+0000 7f0010d3f700 1 -- 172.21.15.1:0/3490536871 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000c06da10 con 0x7f000c15d530 2024-03-18T20:29:49.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.571+0000 7f0011540700 1 --2- 172.21.15.1:0/3490536871 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000c0acfb0 0x7f000c0b3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:29:49.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.571+0000 7f0010d3f700 1 --2- 172.21.15.1:0/3490536871 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c15d530 0x7f000c06e7d0 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 crypto rx=0x7f0004009e50 tx=0x7f000400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47ae4ea96aa4c99f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:49.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.571+0000 7f000b7fe700 1 -- 172.21.15.1:0/3490536871 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000400e070 con 0x7f000c15d530 2024-03-18T20:29:49.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.571+0000 7f000b7fe700 1 -- 172.21.15.1:0/3490536871 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000400b9c0 con 0x7f000c15d530 2024-03-18T20:29:49.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.571+0000 7f000b7fe700 1 -- 172.21.15.1:0/3490536871 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000400bd70 con 0x7f000c15d530 2024-03-18T20:29:49.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.571+0000 7f0012fa3700 1 -- 172.21.15.1:0/3490536871 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c15d530 msgr2=0x7f000c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:49.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.571+0000 7f0012fa3700 1 --2- 172.21.15.1:0/3490536871 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c15d530 0x7f000c06e7d0 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 crypto rx=0x7f0004009e50 tx=0x7f000400b330 comp rx=0 tx=0).stop 2024-03-18T20:29:49.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.572+0000 7f0012fa3700 1 -- 172.21.15.1:0/3490536871 shutdown_connections 2024-03-18T20:29:49.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.572+0000 7f0012fa3700 1 --2- 172.21.15.1:0/3490536871 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000c0acfb0 0x7f000c0b3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.572+0000 7f0012fa3700 1 --2- 172.21.15.1:0/3490536871 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c15d530 0x7f000c06e7d0 unknown :-1 s=CLOSED pgs=3384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.572+0000 7f0012fa3700 1 -- 172.21.15.1:0/3490536871 >> 172.21.15.1:0/3490536871 conn(0x7f000c072080 msgr2=0x7f000c06a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:49.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.572+0000 7f0012fa3700 1 -- 172.21.15.1:0/3490536871 shutdown_connections 2024-03-18T20:29:49.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.572+0000 7f0012fa3700 1 -- 172.21.15.1:0/3490536871 wait complete. 2024-03-18T20:29:49.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.573+0000 7f0012fa3700 1 Processor -- start 2024-03-18T20:29:49.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.573+0000 7f0012fa3700 1 -- start start 2024-03-18T20:29:49.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f0012fa3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000c0acfb0 0x7f000c1435c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:49.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f0012fa3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f000c0ad440 0x7f000c15fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:49.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f0012fa3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c15d530 0x7f000c160250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:49.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f0012fa3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000c143b00 con 0x7f000c15d530 2024-03-18T20:29:49.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f0012fa3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000c1e7c80 con 0x7f000c0ad440 2024-03-18T20:29:49.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f0012fa3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000c1e6be0 con 0x7f000c0acfb0 2024-03-18T20:29:49.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f0011540700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c15d530 0x7f000c160250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:49.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f0011540700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c15d530 0x7f000c160250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60308/0 (socket says 172.21.15.1:60308) 2024-03-18T20:29:49.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f0011540700 1 -- 172.21.15.1:0/3771756269 learned_addr learned my addr 172.21.15.1:0/3771756269 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:49.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f000bfff700 1 --2- 172.21.15.1:0/3771756269 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f000c0ad440 0x7f000c15fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:49.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f0010d3f700 1 --2- 172.21.15.1:0/3771756269 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000c0acfb0 0x7f000c1435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:49.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f0011540700 1 -- 172.21.15.1:0/3771756269 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000c0acfb0 msgr2=0x7f000c1435c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:49.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f0011540700 1 --2- 172.21.15.1:0/3771756269 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000c0acfb0 0x7f000c1435c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f0011540700 1 -- 172.21.15.1:0/3771756269 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f000c0ad440 msgr2=0x7f000c15fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:49.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f0011540700 1 --2- 172.21.15.1:0/3771756269 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f000c0ad440 0x7f000c15fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f0011540700 1 -- 172.21.15.1:0/3771756269 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0004009b30 con 0x7f000c15d530 2024-03-18T20:29:49.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f0011540700 1 --2- 172.21.15.1:0/3771756269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c15d530 0x7f000c160250 secure :-1 s=READY pgs=3385 cs=0 l=1 rev1=1 crypto rx=0x7efffc00e7e0 tx=0x7efffc00eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:49.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f00097fa700 1 -- 172.21.15.1:0/3771756269 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc00c070 con 0x7f000c15d530 2024-03-18T20:29:49.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f00097fa700 1 -- 172.21.15.1:0/3771756269 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efffc0136d0 con 0x7f000c15d530 2024-03-18T20:29:49.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f0012fa3700 1 -- 172.21.15.1:0/3771756269 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f000c1e6ec0 con 0x7f000c15d530 2024-03-18T20:29:49.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.574+0000 7f00097fa700 1 -- 172.21.15.1:0/3771756269 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc013a60 con 0x7f000c15d530 2024-03-18T20:29:49.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.575+0000 7f0012fa3700 1 -- 172.21.15.1:0/3771756269 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f000c1e7480 con 0x7f000c15d530 2024-03-18T20:29:49.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.575+0000 7f0012fa3700 1 -- 172.21.15.1:0/3771756269 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f000c06bb10 con 0x7f000c15d530 2024-03-18T20:29:49.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.576+0000 7f00097fa700 1 -- 172.21.15.1:0/3771756269 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 176) v1 ==== 88848+0+0 (secure 0 0 0) 0x7efffc013bc0 con 0x7f000c15d530 2024-03-18T20:29:49.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.577+0000 7f00097fa700 1 -- 172.21.15.1:0/3771756269 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(292..292 src has 1..292) v4 ==== 6947+0+0 (secure 0 0 0) 0x7efffc089050 con 0x7f000c15d530 2024-03-18T20:29:49.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.581+0000 7f00097fa700 1 -- 172.21.15.1:0/3771756269 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efffc054120 con 0x7f000c15d530 2024-03-18T20:29:49.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.828+0000 7f0012fa3700 1 -- 172.21.15.1:0/3771756269 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f000c067dc0 con 0x7f000c15d530 2024-03-18T20:29:49.833 INFO:tasks.ceph.mon.a.smithi001.stderr:2024-03-18T20:29:49.829+0000 7fbdbeb46700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T20:29:49.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.835+0000 7f00097fa700 1 -- 172.21.15.1:0/3771756269 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v145) v1 ==== 161+0+0 (secure 0 0 0) 0x7efffc057740 con 0x7f000c15d530 2024-03-18T20:29:49.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.837+0000 7f0012fa3700 1 -- 172.21.15.1:0/3771756269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c15d530 msgr2=0x7f000c160250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:49.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.837+0000 7f0012fa3700 1 --2- 172.21.15.1:0/3771756269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c15d530 0x7f000c160250 secure :-1 s=READY pgs=3385 cs=0 l=1 rev1=1 crypto rx=0x7efffc00e7e0 tx=0x7efffc00eaf0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.838+0000 7f0012fa3700 1 -- 172.21.15.1:0/3771756269 shutdown_connections 2024-03-18T20:29:49.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.838+0000 7f0012fa3700 1 --2- 172.21.15.1:0/3771756269 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f000c0acfb0 0x7f000c1435c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.838+0000 7f0012fa3700 1 --2- 172.21.15.1:0/3771756269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f000c15d530 0x7f000c160250 unknown :-1 s=CLOSED pgs=3385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.838+0000 7f0012fa3700 1 --2- 172.21.15.1:0/3771756269 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f000c0ad440 0x7f000c15fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.838+0000 7f0012fa3700 1 -- 172.21.15.1:0/3771756269 >> 172.21.15.1:0/3771756269 conn(0x7f000c072080 msgr2=0x7f000c06b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:49.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.838+0000 7f0012fa3700 1 -- 172.21.15.1:0/3771756269 shutdown_connections 2024-03-18T20:29:49.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.838+0000 7f0012fa3700 1 -- 172.21.15.1:0/3771756269 wait complete. 2024-03-18T20:29:49.844 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T20:29:49.855 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T20:29:49.877 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-03-18T20:29:49.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.978+0000 7f8c8830d700 1 Processor -- start 2024-03-18T20:29:49.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.978+0000 7f8c8830d700 1 -- start start 2024-03-18T20:29:49.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.979+0000 7f8c8830d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c8015da50 0x7f8c800b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:49.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.979+0000 7f8c8830d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8c800cca50 0x7f8c800cd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:49.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.979+0000 7f8c8830d700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c8006a950 con 0x7f8c800cd7c0 2024-03-18T20:29:49.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.979+0000 7f8c8830d700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c800cdc50 con 0x7f8c800cca50 2024-03-18T20:29:49.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.979+0000 7f8c8830d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8006ac30 con 0x7f8c8015da50 2024-03-18T20:29:49.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.979+0000 7f8c868aa700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8c800cd7c0 0x7f8c800ae390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44004/0 (socket says 172.21.15.1:44004) 2024-03-18T20:29:49.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.979+0000 7f8c858a8700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8c800cca50 0x7f8c800cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:49.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.979+0000 7f8c868aa700 1 -- 172.21.15.1:0/3733568112 learned_addr learned my addr 172.21.15.1:0/3733568112 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:29:49.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.979+0000 7f8c860a9700 1 --2- 172.21.15.1:0/3733568112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c8015da50 0x7f8c800b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:49.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.979+0000 7f8c858a8700 1 -- 172.21.15.1:0/3733568112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c8015da50 msgr2=0x7f8c800b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:49.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.979+0000 7f8c858a8700 1 --2- 172.21.15.1:0/3733568112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c8015da50 0x7f8c800b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.979+0000 7f8c858a8700 1 -- 172.21.15.1:0/3733568112 >> v1:172.21.15.1:6789/0 conn(0x7f8c800cd7c0 legacy=0x7f8c800ae390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:49.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.979+0000 7f8c858a8700 1 -- 172.21.15.1:0/3733568112 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8006af10 con 0x7f8c800cca50 2024-03-18T20:29:49.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.979+0000 7f8c850a7700 1 -- 172.21.15.1:0/3733568112 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574997309 0 0) 0x7f8c8006a950 con 0x7f8c800cd7c0 2024-03-18T20:29:49.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.980+0000 7f8c858a8700 1 --2- 172.21.15.1:0/3733568112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c800cca50 0x7f8c800cd280 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c009840 tx=0x7f8c7c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17bae2687de145b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:49.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.980+0000 7f8c850a7700 1 -- 172.21.15.1:0/3733568112 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c011830 con 0x7f8c800cca50 2024-03-18T20:29:49.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.980+0000 7f8c850a7700 1 -- 172.21.15.1:0/3733568112 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7c011990 con 0x7f8c800cca50 2024-03-18T20:29:49.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.980+0000 7f8c850a7700 1 -- 172.21.15.1:0/3733568112 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c011d20 con 0x7f8c800cca50 2024-03-18T20:29:49.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.980+0000 7f8c8830d700 1 -- 172.21.15.1:0/3733568112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c800cca50 msgr2=0x7f8c800cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:49.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.980+0000 7f8c8830d700 1 --2- 172.21.15.1:0/3733568112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c800cca50 0x7f8c800cd280 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c009840 tx=0x7f8c7c009b50 comp rx=0 tx=0).stop 2024-03-18T20:29:49.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.980+0000 7f8c8830d700 1 -- 172.21.15.1:0/3733568112 shutdown_connections 2024-03-18T20:29:49.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.980+0000 7f8c8830d700 1 --2- 172.21.15.1:0/3733568112 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c8015da50 0x7f8c800b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.980+0000 7f8c8830d700 1 --2- 172.21.15.1:0/3733568112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c800cca50 0x7f8c800cd280 unknown :-1 s=CLOSED pgs=3390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.980+0000 7f8c8830d700 1 -- 172.21.15.1:0/3733568112 >> 172.21.15.1:0/3733568112 conn(0x7f8c800b55e0 msgr2=0x7f8c800b59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:49.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.980+0000 7f8c8830d700 1 -- 172.21.15.1:0/3733568112 shutdown_connections 2024-03-18T20:29:49.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.980+0000 7f8c8830d700 1 -- 172.21.15.1:0/3733568112 wait complete. 2024-03-18T20:29:49.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.981+0000 7f8c8830d700 1 Processor -- start 2024-03-18T20:29:49.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.982+0000 7f8c8830d700 1 -- start start 2024-03-18T20:29:49.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.982+0000 7f8c8830d700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c800cca50 0x7f8c8014e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:49.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.982+0000 7f8c8830d700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c800cd7c0 0x7f8c801535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:49.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.982+0000 7f8c8830d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c8015da50 0x7f8c8006e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:49.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.982+0000 7f8c8830d700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c801531d0 con 0x7f8c8015da50 2024-03-18T20:29:49.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.982+0000 7f8c8830d700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c801528c0 con 0x7f8c800cd7c0 2024-03-18T20:29:49.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.982+0000 7f8c8830d700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c801549c0 con 0x7f8c800cca50 2024-03-18T20:29:49.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.982+0000 7f8c868aa700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c8015da50 0x7f8c8006e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:49.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.982+0000 7f8c868aa700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c8015da50 0x7f8c8006e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60364/0 (socket says 172.21.15.1:60364) 2024-03-18T20:29:49.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.982+0000 7f8c868aa700 1 -- 172.21.15.1:0/2394823841 learned_addr learned my addr 172.21.15.1:0/2394823841 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:49.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.983+0000 7f8c860a9700 1 --2- 172.21.15.1:0/2394823841 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c800cca50 0x7f8c8014e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:49.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.983+0000 7f8c858a8700 1 --2- 172.21.15.1:0/2394823841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c800cd7c0 0x7f8c801535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:49.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.983+0000 7f8c868aa700 1 -- 172.21.15.1:0/2394823841 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c800cca50 msgr2=0x7f8c8014e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:49.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.983+0000 7f8c868aa700 1 --2- 172.21.15.1:0/2394823841 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c800cca50 0x7f8c8014e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.983+0000 7f8c868aa700 1 -- 172.21.15.1:0/2394823841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c800cd7c0 msgr2=0x7f8c801535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:49.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.983+0000 7f8c868aa700 1 --2- 172.21.15.1:0/2394823841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c800cd7c0 0x7f8c801535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:49.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.983+0000 7f8c868aa700 1 -- 172.21.15.1:0/2394823841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c0094a0 con 0x7f8c8015da50 2024-03-18T20:29:49.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.983+0000 7f8c858a8700 1 --2- 172.21.15.1:0/2394823841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c800cd7c0 0x7f8c801535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:29:49.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.983+0000 7f8c860a9700 1 --2- 172.21.15.1:0/2394823841 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c800cca50 0x7f8c8014e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:29:49.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.983+0000 7f8c868aa700 1 --2- 172.21.15.1:0/2394823841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c8015da50 0x7f8c8006e2e0 secure :-1 s=READY pgs=3391 cs=0 l=1 rev1=1 crypto rx=0x7f8c7800cc70 tx=0x7f8c7800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:49.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.983+0000 7f8c76ffd700 1 -- 172.21.15.1:0/2394823841 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7800ada0 con 0x7f8c8015da50 2024-03-18T20:29:49.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.983+0000 7f8c76ffd700 1 -- 172.21.15.1:0/2394823841 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7800ce70 con 0x7f8c8015da50 2024-03-18T20:29:49.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.983+0000 7f8c76ffd700 1 -- 172.21.15.1:0/2394823841 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c780072f0 con 0x7f8c8015da50 2024-03-18T20:29:49.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.983+0000 7f8c8830d700 1 -- 172.21.15.1:0/2394823841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c80152b10 con 0x7f8c8015da50 2024-03-18T20:29:49.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.983+0000 7f8c8830d700 1 -- 172.21.15.1:0/2394823841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8014ef70 con 0x7f8c8015da50 2024-03-18T20:29:49.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.984+0000 7f8c8830d700 1 -- 172.21.15.1:0/2394823841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c80153310 con 0x7f8c8015da50 2024-03-18T20:29:49.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.985+0000 7f8c76ffd700 1 -- 172.21.15.1:0/2394823841 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 176) v1 ==== 88848+0+0 (secure 0 0 0) 0x7f8c78007450 con 0x7f8c8015da50 2024-03-18T20:29:49.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.985+0000 7f8c76ffd700 1 -- 172.21.15.1:0/2394823841 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(293..293 src has 1..293) v4 ==== 7033+0+0 (secure 0 0 0) 0x7f8c78088e70 con 0x7f8c8015da50 2024-03-18T20:29:49.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:49.989+0000 7f8c76ffd700 1 -- 172.21.15.1:0/2394823841 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8c80153310 con 0x7f8c8015da50 2024-03-18T20:29:50.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:50.072+0000 7f8c76ffd700 1 -- 172.21.15.1:0/2394823841 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 177) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f8c78053a20 con 0x7f8c8015da50 2024-03-18T20:29:50.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:50.072+0000 7f8c76ffd700 1 --2- 172.21.15.1:0/2394823841 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8c64036150 0x7f8c64072da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:50.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:50.073+0000 7f8c860a9700 1 --2- 172.21.15.1:0/2394823841 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8c64036150 0x7f8c64072da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:50.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:50.074+0000 7f8c860a9700 1 --2- 172.21.15.1:0/2394823841 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8c64036150 0x7f8c64072da0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8c700099f0 tx=0x7f8c70008040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:50.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:50.245+0000 7f8c8830d700 1 -- 172.21.15.1:0/2394823841 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8c80153310 con 0x7f8c8015da50 2024-03-18T20:29:50.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:50.838+0000 7f8c76ffd700 1 -- 172.21.15.1:0/2394823841 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v146) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8c80153310 con 0x7f8c8015da50 2024-03-18T20:29:50.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:50.841+0000 7f8c8830d700 1 -- 172.21.15.1:0/2394823841 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8c64036150 msgr2=0x7f8c64072da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:50.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:50.841+0000 7f8c8830d700 1 --2- 172.21.15.1:0/2394823841 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8c64036150 0x7f8c64072da0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8c700099f0 tx=0x7f8c70008040 comp rx=0 tx=0).stop 2024-03-18T20:29:50.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:50.841+0000 7f8c8830d700 1 -- 172.21.15.1:0/2394823841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c8015da50 msgr2=0x7f8c8006e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:50.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:50.841+0000 7f8c8830d700 1 --2- 172.21.15.1:0/2394823841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c8015da50 0x7f8c8006e2e0 secure :-1 s=READY pgs=3391 cs=0 l=1 rev1=1 crypto rx=0x7f8c7800cc70 tx=0x7f8c7800a340 comp rx=0 tx=0).stop 2024-03-18T20:29:50.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:50.841+0000 7f8c8830d700 1 -- 172.21.15.1:0/2394823841 shutdown_connections 2024-03-18T20:29:50.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:50.841+0000 7f8c8830d700 1 --2- 172.21.15.1:0/2394823841 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f8c64036150 0x7f8c64072da0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:50.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:50.841+0000 7f8c8830d700 1 --2- 172.21.15.1:0/2394823841 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f8c800cca50 0x7f8c8014e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:50.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:50.841+0000 7f8c8830d700 1 --2- 172.21.15.1:0/2394823841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c8015da50 0x7f8c8006e2e0 unknown :-1 s=CLOSED pgs=3391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:50.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:50.841+0000 7f8c8830d700 1 --2- 172.21.15.1:0/2394823841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c800cd7c0 0x7f8c801535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:50.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:50.841+0000 7f8c8830d700 1 -- 172.21.15.1:0/2394823841 >> 172.21.15.1:0/2394823841 conn(0x7f8c800b55e0 msgr2=0x7f8c800cced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:50.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:50.841+0000 7f8c8830d700 1 -- 172.21.15.1:0/2394823841 shutdown_connections 2024-03-18T20:29:50.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:50.841+0000 7f8c8830d700 1 -- 172.21.15.1:0/2394823841 wait complete. 2024-03-18T20:29:50.861 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T20:29:51.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.035+0000 7f86482a3700 1 Processor -- start 2024-03-18T20:29:51.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.035+0000 7f86482a3700 1 -- start start 2024-03-18T20:29:51.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.036+0000 7f86482a3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f864015b710 0x7f86400af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:51.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.036+0000 7f86482a3700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8640066c10 con 0x7f864015f6e0 2024-03-18T20:29:51.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.036+0000 7f86482a3700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8640066930 con 0x7f864015b330 2024-03-18T20:29:51.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.036+0000 7f86482a3700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86400cc780 con 0x7f864015b710 2024-03-18T20:29:51.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.036+0000 7f8646840700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f864015b710 0x7f86400af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:51.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.036+0000 7f864583e700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f864015f6e0 0x7f86400cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44012/0 (socket says 172.21.15.1:44012) 2024-03-18T20:29:51.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.036+0000 7f8646840700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f864015b710 0x7f86400af290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60376/0 (socket says 172.21.15.1:60376) 2024-03-18T20:29:51.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.036+0000 7f864583e700 1 -- 172.21.15.1:0/1674080629 learned_addr learned my addr 172.21.15.1:0/1674080629 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:29:51.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.036+0000 7f8646840700 1 -- 172.21.15.1:0/1674080629 >> v1:172.21.15.165:6789/0 conn(0x7f864015b330 legacy=0x7f864015efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:51.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.036+0000 7f8646840700 1 -- 172.21.15.1:0/1674080629 >> v1:172.21.15.1:6789/0 conn(0x7f864015f6e0 legacy=0x7f86400cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:51.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.036+0000 7f8646840700 1 -- 172.21.15.1:0/1674080629 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86400cca60 con 0x7f864015b710 2024-03-18T20:29:51.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.036+0000 7f8646840700 1 --2- 172.21.15.1:0/1674080629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f864015b710 0x7f86400af290 secure :-1 s=READY pgs=3393 cs=0 l=1 rev1=1 crypto rx=0x7f8638009d90 tx=0x7f863800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a4480ef05c8e4dd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:51.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.037+0000 7f864503d700 1 -- 172.21.15.1:0/1674080629 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863800e070 con 0x7f864015b710 2024-03-18T20:29:51.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.037+0000 7f864503d700 1 -- 172.21.15.1:0/1674080629 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863800bb90 con 0x7f864015b710 2024-03-18T20:29:51.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.037+0000 7f864503d700 1 -- 172.21.15.1:0/1674080629 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8638003a20 con 0x7f864015b710 2024-03-18T20:29:51.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.037+0000 7f86482a3700 1 -- 172.21.15.1:0/1674080629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f864015b710 msgr2=0x7f86400af290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:51.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.037+0000 7f86482a3700 1 --2- 172.21.15.1:0/1674080629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f864015b710 0x7f86400af290 secure :-1 s=READY pgs=3393 cs=0 l=1 rev1=1 crypto rx=0x7f8638009d90 tx=0x7f863800b330 comp rx=0 tx=0).stop 2024-03-18T20:29:51.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.037+0000 7f86482a3700 1 -- 172.21.15.1:0/1674080629 shutdown_connections 2024-03-18T20:29:51.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.037+0000 7f86482a3700 1 --2- 172.21.15.1:0/1674080629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f864015b710 0x7f86400af290 unknown :-1 s=CLOSED pgs=3393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:51.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.037+0000 7f86482a3700 1 -- 172.21.15.1:0/1674080629 >> 172.21.15.1:0/1674080629 conn(0x7f8640072110 msgr2=0x7f864006a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:51.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.037+0000 7f86482a3700 1 -- 172.21.15.1:0/1674080629 shutdown_connections 2024-03-18T20:29:51.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.037+0000 7f86482a3700 1 -- 172.21.15.1:0/1674080629 wait complete. 2024-03-18T20:29:51.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.038+0000 7f86482a3700 1 Processor -- start 2024-03-18T20:29:51.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.038+0000 7f86482a3700 1 -- start start 2024-03-18T20:29:51.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.039+0000 7f86482a3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f864015b330 0x7f864015c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:51.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.039+0000 7f86482a3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f864015b710 0x7f86400cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:51.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.039+0000 7f86482a3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f864015f6e0 0x7f8640151060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:51.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.039+0000 7f86482a3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8640153130 con 0x7f864015b330 2024-03-18T20:29:51.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.039+0000 7f86482a3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864015c9a0 con 0x7f864015b710 2024-03-18T20:29:51.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.039+0000 7f86482a3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86400afa10 con 0x7f864015f6e0 2024-03-18T20:29:51.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.039+0000 7f864603f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f864015b330 0x7f864015c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:51.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.039+0000 7f864603f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f864015b330 0x7f864015c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60386/0 (socket says 172.21.15.1:60386) 2024-03-18T20:29:51.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.039+0000 7f864603f700 1 -- 172.21.15.1:0/3695718765 learned_addr learned my addr 172.21.15.1:0/3695718765 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:51.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.039+0000 7f864583e700 1 --2- 172.21.15.1:0/3695718765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f864015b710 0x7f86400cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:51.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.039+0000 7f864603f700 1 -- 172.21.15.1:0/3695718765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f864015f6e0 msgr2=0x7f8640151060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:51.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.040+0000 7f864603f700 1 --2- 172.21.15.1:0/3695718765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f864015f6e0 0x7f8640151060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:51.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.040+0000 7f864603f700 1 -- 172.21.15.1:0/3695718765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f864015b710 msgr2=0x7f86400cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:51.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.040+0000 7f864603f700 1 --2- 172.21.15.1:0/3695718765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f864015b710 0x7f86400cd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:51.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.040+0000 7f864603f700 1 -- 172.21.15.1:0/3695718765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8638009a70 con 0x7f864015b330 2024-03-18T20:29:51.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.040+0000 7f864603f700 1 --2- 172.21.15.1:0/3695718765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f864015b330 0x7f864015c460 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 crypto rx=0x7f863000c4d0 tx=0x7f863000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:51.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.040+0000 7f8636ffd700 1 -- 172.21.15.1:0/3695718765 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8630011790 con 0x7f864015b330 2024-03-18T20:29:51.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.040+0000 7f8636ffd700 1 -- 172.21.15.1:0/3695718765 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86300118f0 con 0x7f864015b330 2024-03-18T20:29:51.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.040+0000 7f86482a3700 1 -- 172.21.15.1:0/3695718765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86400afcf0 con 0x7f864015b330 2024-03-18T20:29:51.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.040+0000 7f8636ffd700 1 -- 172.21.15.1:0/3695718765 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8630011ca0 con 0x7f864015b330 2024-03-18T20:29:51.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.040+0000 7f86482a3700 1 -- 172.21.15.1:0/3695718765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86400b02b0 con 0x7f864015b330 2024-03-18T20:29:51.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.041+0000 7f86482a3700 1 -- 172.21.15.1:0/3695718765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8640067dc0 con 0x7f864015b330 2024-03-18T20:29:51.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.041+0000 7f8636ffd700 1 -- 172.21.15.1:0/3695718765 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 177) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f863001c460 con 0x7f864015b330 2024-03-18T20:29:51.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.042+0000 7f8636ffd700 1 --2- 172.21.15.1:0/3695718765 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f862406aae0 0x7f862406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:51.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.042+0000 7f8636ffd700 1 -- 172.21.15.1:0/3695718765 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(293..293 src has 1..293) v4 ==== 7033+0+0 (secure 0 0 0) 0x7f86300892a0 con 0x7f864015b330 2024-03-18T20:29:51.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.042+0000 7f864583e700 1 --2- 172.21.15.1:0/3695718765 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f862406aae0 0x7f862406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:51.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.042+0000 7f864583e700 1 --2- 172.21.15.1:0/3695718765 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f862406aae0 0x7f862406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f86400cdd60 tx=0x7f863c009040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:51.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.046+0000 7f8636ffd700 1 -- 172.21.15.1:0/3695718765 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8630054290 con 0x7f864015b330 2024-03-18T20:29:51.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:51.254+0000 7f86482a3700 1 -- 172.21.15.1:0/3695718765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f864006e7e0 con 0x7f864015b330 2024-03-18T20:29:52.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.082+0000 7f8636ffd700 1 -- 172.21.15.1:0/3695718765 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 178) v1 ==== 89183+0+0 (secure 0 0 0) 0x7f8630054660 con 0x7f864015b330 2024-03-18T20:29:52.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.170+0000 7f8636ffd700 1 -- 172.21.15.1:0/3695718765 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v294) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8630070cb0 con 0x7f864015b330 2024-03-18T20:29:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.174+0000 7f86482a3700 1 -- 172.21.15.1:0/3695718765 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f862406aae0 msgr2=0x7f862406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.174+0000 7f86482a3700 1 --2- 172.21.15.1:0/3695718765 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f862406aae0 0x7f862406cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f86400cdd60 tx=0x7f863c009040 comp rx=0 tx=0).stop 2024-03-18T20:29:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.174+0000 7f86482a3700 1 -- 172.21.15.1:0/3695718765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f864015b330 msgr2=0x7f864015c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.174+0000 7f86482a3700 1 --2- 172.21.15.1:0/3695718765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f864015b330 0x7f864015c460 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 crypto rx=0x7f863000c4d0 tx=0x7f863000c890 comp rx=0 tx=0).stop 2024-03-18T20:29:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.174+0000 7f86482a3700 1 -- 172.21.15.1:0/3695718765 shutdown_connections 2024-03-18T20:29:52.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.174+0000 7f86482a3700 1 --2- 172.21.15.1:0/3695718765 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f862406aae0 0x7f862406cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.174+0000 7f86482a3700 1 --2- 172.21.15.1:0/3695718765 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f864015f6e0 0x7f8640151060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.174+0000 7f86482a3700 1 --2- 172.21.15.1:0/3695718765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f864015b330 0x7f864015c460 unknown :-1 s=CLOSED pgs=3394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.174+0000 7f86482a3700 1 --2- 172.21.15.1:0/3695718765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f864015b710 0x7f86400cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.174+0000 7f86482a3700 1 -- 172.21.15.1:0/3695718765 >> 172.21.15.1:0/3695718765 conn(0x7f8640072110 msgr2=0x7f86400ac930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.174+0000 7f86482a3700 1 -- 172.21.15.1:0/3695718765 shutdown_connections 2024-03-18T20:29:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.174+0000 7f86482a3700 1 -- 172.21.15.1:0/3695718765 wait complete. 2024-03-18T20:29:52.180 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-03-18T20:29:52.195 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T20:29:52.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.327+0000 7faf23531700 1 Processor -- start 2024-03-18T20:29:52.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.327+0000 7faf23531700 1 -- start start 2024-03-18T20:29:52.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.328+0000 7faf23531700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf1c09a900 0x7faf1c099a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:52.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.328+0000 7faf23531700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faf1c099f70 0x7faf1c15da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:52.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.328+0000 7faf23531700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf1c0b54f0 con 0x7faf1c15d680 2024-03-18T20:29:52.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.328+0000 7faf23531700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf1c09a3e0 con 0x7faf1c099f70 2024-03-18T20:29:52.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.328+0000 7faf23531700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c0b57d0 con 0x7faf1c09a900 2024-03-18T20:29:52.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.328+0000 7faf21ace700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faf1c099f70 0x7faf1c15da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:52.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.328+0000 7faf21ace700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faf1c099f70 0x7faf1c15da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60422/0 (socket says 172.21.15.1:60422) 2024-03-18T20:29:52.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.328+0000 7faf21ace700 1 -- 172.21.15.1:0/3280764737 learned_addr learned my addr 172.21.15.1:0/3280764737 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:52.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.329+0000 7faf21ace700 1 -- 172.21.15.1:0/3280764737 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf1c09a900 msgr2=0x7faf1c099a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:52.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.329+0000 7faf20acc700 1 --2- 172.21.15.1:0/3280764737 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf1c09a900 0x7faf1c099a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:52.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.329+0000 7faf21ace700 1 --2- 172.21.15.1:0/3280764737 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf1c09a900 0x7faf1c099a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:52.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.329+0000 7faf21ace700 1 -- 172.21.15.1:0/3280764737 >> v1:172.21.15.165:6789/0 conn(0x7faf1c15d680 legacy=0x7faf1c0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:52.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.329+0000 7faf21ace700 1 -- 172.21.15.1:0/3280764737 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf1c0b5ab0 con 0x7faf1c099f70 2024-03-18T20:29:52.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.329+0000 7faf21ace700 1 --2- 172.21.15.1:0/3280764737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf1c099f70 0x7faf1c15da60 secure :-1 s=READY pgs=3397 cs=0 l=1 rev1=1 crypto rx=0x7faf14009e50 tx=0x7faf1400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5359edc1204cb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:52.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.329+0000 7faf13fff700 1 -- 172.21.15.1:0/3280764737 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1400e070 con 0x7faf1c099f70 2024-03-18T20:29:52.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.329+0000 7faf13fff700 1 -- 172.21.15.1:0/3280764737 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf1400b9c0 con 0x7faf1c099f70 2024-03-18T20:29:52.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.329+0000 7faf13fff700 1 -- 172.21.15.1:0/3280764737 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1400bd70 con 0x7faf1c099f70 2024-03-18T20:29:52.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.330+0000 7faf23531700 1 -- 172.21.15.1:0/3280764737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf1c099f70 msgr2=0x7faf1c15da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.330+0000 7faf23531700 1 --2- 172.21.15.1:0/3280764737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf1c099f70 0x7faf1c15da60 secure :-1 s=READY pgs=3397 cs=0 l=1 rev1=1 crypto rx=0x7faf14009e50 tx=0x7faf1400b330 comp rx=0 tx=0).stop 2024-03-18T20:29:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.330+0000 7faf23531700 1 -- 172.21.15.1:0/3280764737 shutdown_connections 2024-03-18T20:29:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.330+0000 7faf23531700 1 --2- 172.21.15.1:0/3280764737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf1c099f70 0x7faf1c15da60 unknown :-1 s=CLOSED pgs=3397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.330+0000 7faf23531700 1 --2- 172.21.15.1:0/3280764737 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf1c09a900 0x7faf1c099a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.330+0000 7faf23531700 1 -- 172.21.15.1:0/3280764737 >> 172.21.15.1:0/3280764737 conn(0x7faf1c0b2ec0 msgr2=0x7faf1c0b32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.330+0000 7faf23531700 1 -- 172.21.15.1:0/3280764737 shutdown_connections 2024-03-18T20:29:52.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.330+0000 7faf23531700 1 -- 172.21.15.1:0/3280764737 wait complete. 2024-03-18T20:29:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.331+0000 7faf23531700 1 Processor -- start 2024-03-18T20:29:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.331+0000 7faf23531700 1 -- start start 2024-03-18T20:29:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.332+0000 7faf23531700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf1c099f70 0x7faf1c140d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.332+0000 7faf23531700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf1c09a900 0x7faf1c151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.332+0000 7faf23531700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf1c15d680 0x7faf1c15ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.332+0000 7faf23531700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c141290 con 0x7faf1c15d680 2024-03-18T20:29:52.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.332+0000 7faf23531700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c153350 con 0x7faf1c09a900 2024-03-18T20:29:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.332+0000 7faf23531700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c152740 con 0x7faf1c099f70 2024-03-18T20:29:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.332+0000 7faf21ace700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf1c15d680 0x7faf1c15ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.332+0000 7faf21ace700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf1c15d680 0x7faf1c15ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60426/0 (socket says 172.21.15.1:60426) 2024-03-18T20:29:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.332+0000 7faf21ace700 1 -- 172.21.15.1:0/3305499939 learned_addr learned my addr 172.21.15.1:0/3305499939 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.333+0000 7faf20acc700 1 --2- 172.21.15.1:0/3305499939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf1c09a900 0x7faf1c151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.333+0000 7faf212cd700 1 --2- 172.21.15.1:0/3305499939 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf1c099f70 0x7faf1c140d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.333+0000 7faf21ace700 1 -- 172.21.15.1:0/3305499939 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf1c099f70 msgr2=0x7faf1c140d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.333+0000 7faf21ace700 1 --2- 172.21.15.1:0/3305499939 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf1c099f70 0x7faf1c140d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.333+0000 7faf21ace700 1 -- 172.21.15.1:0/3305499939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf1c09a900 msgr2=0x7faf1c151e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.333+0000 7faf21ace700 1 --2- 172.21.15.1:0/3305499939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf1c09a900 0x7faf1c151e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.333+0000 7faf21ace700 1 -- 172.21.15.1:0/3305499939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf14009b30 con 0x7faf1c15d680 2024-03-18T20:29:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.333+0000 7faf212cd700 1 --2- 172.21.15.1:0/3305499939 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf1c099f70 0x7faf1c140d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:29:52.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.333+0000 7faf20acc700 1 --2- 172.21.15.1:0/3305499939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf1c09a900 0x7faf1c151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:29:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.333+0000 7faf21ace700 1 --2- 172.21.15.1:0/3305499939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf1c15d680 0x7faf1c15ac70 secure :-1 s=READY pgs=3398 cs=0 l=1 rev1=1 crypto rx=0x7faf1c0723c0 tx=0x7faf14004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.333+0000 7faf11ffb700 1 -- 172.21.15.1:0/3305499939 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf140092e0 con 0x7faf1c15d680 2024-03-18T20:29:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.333+0000 7faf11ffb700 1 -- 172.21.15.1:0/3305499939 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf14004960 con 0x7faf1c15d680 2024-03-18T20:29:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.333+0000 7faf11ffb700 1 -- 172.21.15.1:0/3305499939 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1401c680 con 0x7faf1c15d680 2024-03-18T20:29:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.333+0000 7faf23531700 1 -- 172.21.15.1:0/3305499939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf1c152a20 con 0x7faf1c15d680 2024-03-18T20:29:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.333+0000 7faf23531700 1 -- 172.21.15.1:0/3305499939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf1c152fe0 con 0x7faf1c15d680 2024-03-18T20:29:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.334+0000 7faf23531700 1 -- 172.21.15.1:0/3305499939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1c1531a0 con 0x7faf1c15d680 2024-03-18T20:29:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.335+0000 7faf11ffb700 1 -- 172.21.15.1:0/3305499939 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 178) v1 ==== 89183+0+0 (secure 0 0 0) 0x7faf1401c7e0 con 0x7faf1c15d680 2024-03-18T20:29:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.335+0000 7faf11ffb700 1 --2- 172.21.15.1:0/3305499939 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf0006ad20 0x7faf0006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.335+0000 7faf11ffb700 1 -- 172.21.15.1:0/3305499939 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(294..294 src has 1..294) v4 ==== 6595+0+0 (secure 0 0 0) 0x7faf1408b580 con 0x7faf1c15d680 2024-03-18T20:29:52.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.335+0000 7faf212cd700 1 --2- 172.21.15.1:0/3305499939 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf0006ad20 0x7faf0006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:52.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.336+0000 7faf212cd700 1 --2- 172.21.15.1:0/3305499939 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf0006ad20 0x7faf0006d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7faf0c000f60 tx=0x7faf0c009040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:52.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.340+0000 7faf11ffb700 1 -- 172.21.15.1:0/3305499939 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faf14056370 con 0x7faf1c15d680 2024-03-18T20:29:52.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:52.543+0000 7faf23531700 1 -- 172.21.15.1:0/3305499939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faf1c067dc0 con 0x7faf1c15d680 2024-03-18T20:29:53.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.187+0000 7faf11ffb700 1 -- 172.21.15.1:0/3305499939 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7faf14056740 con 0x7faf1c15d680 2024-03-18T20:29:53.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.188+0000 7faf11ffb700 1 -- 172.21.15.1:0/3305499939 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v295) v1 ==== 169+0+0 (secure 0 0 0) 0x7faf1406fab0 con 0x7faf1c15d680 2024-03-18T20:29:53.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.193+0000 7faf23531700 1 -- 172.21.15.1:0/3305499939 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf0006ad20 msgr2=0x7faf0006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:53.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.193+0000 7faf23531700 1 --2- 172.21.15.1:0/3305499939 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf0006ad20 0x7faf0006d1e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7faf0c000f60 tx=0x7faf0c009040 comp rx=0 tx=0).stop 2024-03-18T20:29:53.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.193+0000 7faf23531700 1 -- 172.21.15.1:0/3305499939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf1c15d680 msgr2=0x7faf1c15ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:53.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.193+0000 7faf23531700 1 --2- 172.21.15.1:0/3305499939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf1c15d680 0x7faf1c15ac70 secure :-1 s=READY pgs=3398 cs=0 l=1 rev1=1 crypto rx=0x7faf1c0723c0 tx=0x7faf14004ad0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.193+0000 7faf23531700 1 -- 172.21.15.1:0/3305499939 shutdown_connections 2024-03-18T20:29:53.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.193+0000 7faf23531700 1 --2- 172.21.15.1:0/3305499939 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7faf0006ad20 0x7faf0006d1e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.193+0000 7faf23531700 1 --2- 172.21.15.1:0/3305499939 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faf1c099f70 0x7faf1c140d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.193+0000 7faf23531700 1 --2- 172.21.15.1:0/3305499939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf1c15d680 0x7faf1c15ac70 unknown :-1 s=CLOSED pgs=3398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.193+0000 7faf23531700 1 --2- 172.21.15.1:0/3305499939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf1c09a900 0x7faf1c151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.193+0000 7faf23531700 1 -- 172.21.15.1:0/3305499939 >> 172.21.15.1:0/3305499939 conn(0x7faf1c0b2ec0 msgr2=0x7faf1c15b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:53.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.194+0000 7faf23531700 1 -- 172.21.15.1:0/3305499939 shutdown_connections 2024-03-18T20:29:53.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.194+0000 7faf23531700 1 -- 172.21.15.1:0/3305499939 wait complete. 2024-03-18T20:29:53.200 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-03-18T20:29:53.214 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2024-03-18T20:29:53.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.342+0000 7f19512f9700 1 Processor -- start 2024-03-18T20:29:53.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.342+0000 7f19512f9700 1 -- start start 2024-03-18T20:29:53.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.343+0000 7f19512f9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f194c15e0d0 0x7f194c0cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:53.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.343+0000 7f19512f9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f194c0cfe30 con 0x7f194c15d5e0 2024-03-18T20:29:53.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.343+0000 7f19512f9700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f194c0d0110 con 0x7f194c0d3f80 2024-03-18T20:29:53.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.343+0000 7f19512f9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f194c0d24e0 con 0x7f194c15e0d0 2024-03-18T20:29:53.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.343+0000 7f194affd700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f194c15d5e0 0x7f194c0d1dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44014/0 (socket says 172.21.15.1:44014) 2024-03-18T20:29:53.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.343+0000 7f194affd700 1 -- 172.21.15.1:0/3676671194 learned_addr learned my addr 172.21.15.1:0/3676671194 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:29:53.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.343+0000 7f194b7fe700 1 --2- 172.21.15.1:0/3676671194 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f194c15e0d0 0x7f194c0cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:53.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.343+0000 7f1949ffb700 1 -- 172.21.15.1:0/3676671194 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650434128 0 0) 0x7f194c0cfe30 con 0x7f194c15d5e0 2024-03-18T20:29:53.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.343+0000 7f1949ffb700 1 -- 172.21.15.1:0/3676671194 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19340035b0 con 0x7f194c15d5e0 2024-03-18T20:29:53.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f1949ffb700 1 -- 172.21.15.1:0/3676671194 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 533972929 0 0) 0x7f19340035b0 con 0x7f194c15d5e0 2024-03-18T20:29:53.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f1949ffb700 1 -- 172.21.15.1:0/3676671194 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f194c15e0d0 msgr2=0x7f194c0cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:53.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f1949ffb700 1 --2- 172.21.15.1:0/3676671194 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f194c15e0d0 0x7f194c0cefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f1949ffb700 1 -- 172.21.15.1:0/3676671194 >> v1:172.21.15.165:6789/0 conn(0x7f194c0d3f80 legacy=0x7f194c15d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:53.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f1949ffb700 1 -- 172.21.15.1:0/3676671194 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194c0d03f0 con 0x7f194c15d5e0 2024-03-18T20:29:53.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f1949ffb700 1 -- 172.21.15.1:0/3676671194 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f1940001670 con 0x7f194c15d5e0 2024-03-18T20:29:53.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f1949ffb700 1 -- 172.21.15.1:0/3676671194 >> v1:172.21.15.1:6789/0 conn(0x7f194c15d5e0 legacy=0x7f194c0d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f1949ffb700 1 --2- 172.21.15.1:0/3676671194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1934003a30 0x7f1934003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f1949ffb700 1 --2- 172.21.15.1:0/3676671194 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1934004850 0x7f1934004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f1949ffb700 1 --2- 172.21.15.1:0/3676671194 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f194c15e0d0 0x7f1934005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f1949ffb700 1 -- 172.21.15.1:0/3676671194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c0d24e0 con 0x7f1934003a30 2024-03-18T20:29:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f1949ffb700 1 -- 172.21.15.1:0/3676671194 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193400bac0 con 0x7f1934004850 2024-03-18T20:29:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f1949ffb700 1 -- 172.21.15.1:0/3676671194 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193400bc50 con 0x7f194c15e0d0 2024-03-18T20:29:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f194affd700 1 --2- 172.21.15.1:0/3676671194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1934003a30 0x7f1934003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f194a7fc700 1 --2- 172.21.15.1:0/3676671194 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1934004850 0x7f1934004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f194affd700 1 -- 172.21.15.1:0/3676671194 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f194c15e0d0 msgr2=0x7f1934005210 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T20:29:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f194affd700 1 --2- 172.21.15.1:0/3676671194 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f194c15e0d0 0x7f1934005210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f194affd700 1 -- 172.21.15.1:0/3676671194 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1934004850 msgr2=0x7f1934004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f194affd700 1 --2- 172.21.15.1:0/3676671194 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1934004850 0x7f1934004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f194affd700 1 -- 172.21.15.1:0/3676671194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19340035b0 con 0x7f1934003a30 2024-03-18T20:29:53.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f194affd700 1 --2- 172.21.15.1:0/3676671194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1934003a30 0x7f1934003e30 secure :-1 s=READY pgs=3400 cs=0 l=1 rev1=1 crypto rx=0x7f194000c2f0 tx=0x7f1934005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1be89084132df215 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:53.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f1949ffb700 1 -- 172.21.15.1:0/3676671194 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194000cdf0 con 0x7f1934003a30 2024-03-18T20:29:53.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f1949ffb700 1 -- 172.21.15.1:0/3676671194 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1940005960 con 0x7f1934003a30 2024-03-18T20:29:53.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.344+0000 7f1949ffb700 1 -- 172.21.15.1:0/3676671194 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1940005d10 con 0x7f1934003a30 2024-03-18T20:29:53.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.345+0000 7f19512f9700 1 -- 172.21.15.1:0/3676671194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1934003a30 msgr2=0x7f1934003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:53.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.345+0000 7f19512f9700 1 --2- 172.21.15.1:0/3676671194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1934003a30 0x7f1934003e30 secure :-1 s=READY pgs=3400 cs=0 l=1 rev1=1 crypto rx=0x7f194000c2f0 tx=0x7f1934005880 comp rx=0 tx=0).stop 2024-03-18T20:29:53.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.345+0000 7f194b7fe700 1 -- 172.21.15.1:0/3676671194 reap_dead start 2024-03-18T20:29:53.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.345+0000 7f19512f9700 1 -- 172.21.15.1:0/3676671194 shutdown_connections 2024-03-18T20:29:53.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.345+0000 7f19512f9700 1 -- 172.21.15.1:0/3676671194 >> 172.21.15.1:0/3676671194 conn(0x7f194c0d3380 msgr2=0x7f194c0d3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:53.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.345+0000 7f19512f9700 1 -- 172.21.15.1:0/3676671194 shutdown_connections 2024-03-18T20:29:53.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.345+0000 7f19512f9700 1 -- 172.21.15.1:0/3676671194 wait complete. 2024-03-18T20:29:53.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.347+0000 7f19512f9700 1 Processor -- start 2024-03-18T20:29:53.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.347+0000 7f19512f9700 1 -- start start 2024-03-18T20:29:53.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f19512f9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1934003a30 0x7f194c0b8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:53.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f19512f9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f194c0b6bf0 0x7f194c15e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:53.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f19512f9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f194c147550 0x7f194c1479d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:53.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f19512f9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c15aa00 con 0x7f1934003a30 2024-03-18T20:29:53.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f19512f9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c15eec0 con 0x7f194c147550 2024-03-18T20:29:53.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f19512f9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f194c0b71b0 con 0x7f194c0b6bf0 2024-03-18T20:29:53.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f194affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1934003a30 0x7f194c0b8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:53.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f194affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1934003a30 0x7f194c0b8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60436/0 (socket says 172.21.15.1:60436) 2024-03-18T20:29:53.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f194affd700 1 -- 172.21.15.1:0/220600140 learned_addr learned my addr 172.21.15.1:0/220600140 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:53.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f194affd700 1 -- 172.21.15.1:0/220600140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f194c0b6bf0 msgr2=0x7f194c15e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:53.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f194affd700 1 --2- 172.21.15.1:0/220600140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f194c0b6bf0 0x7f194c15e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f194affd700 1 -- 172.21.15.1:0/220600140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f194c147550 msgr2=0x7f194c1479d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:53.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f194affd700 1 --2- 172.21.15.1:0/220600140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f194c147550 0x7f194c1479d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f194affd700 1 -- 172.21.15.1:0/220600140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194c0b7490 con 0x7f1934003a30 2024-03-18T20:29:53.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f194affd700 1 --2- 172.21.15.1:0/220600140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1934003a30 0x7f194c0b8250 secure :-1 s=READY pgs=3401 cs=0 l=1 rev1=1 crypto rx=0x7f1940008fd0 tx=0x7f1940001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:53.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f19337fe700 1 -- 172.21.15.1:0/220600140 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1940004a80 con 0x7f1934003a30 2024-03-18T20:29:53.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f19337fe700 1 -- 172.21.15.1:0/220600140 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1940004be0 con 0x7f1934003a30 2024-03-18T20:29:53.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f19512f9700 1 -- 172.21.15.1:0/220600140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f194c0b7770 con 0x7f1934003a30 2024-03-18T20:29:53.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f19337fe700 1 -- 172.21.15.1:0/220600140 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19400070d0 con 0x7f1934003a30 2024-03-18T20:29:53.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.348+0000 7f19512f9700 1 -- 172.21.15.1:0/220600140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f194c0b7d30 con 0x7f1934003a30 2024-03-18T20:29:53.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.349+0000 7f19512f9700 1 -- 172.21.15.1:0/220600140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f194c0b7ef0 con 0x7f1934003a30 2024-03-18T20:29:53.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.350+0000 7f19337fe700 1 -- 172.21.15.1:0/220600140 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1940007230 con 0x7f1934003a30 2024-03-18T20:29:53.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.351+0000 7f19337fe700 1 --2- 172.21.15.1:0/220600140 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f193806adb0 0x7f193806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:53.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.351+0000 7f19337fe700 1 -- 172.21.15.1:0/220600140 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f194008b580 con 0x7f1934003a30 2024-03-18T20:29:53.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.351+0000 7f194a7fc700 1 --2- 172.21.15.1:0/220600140 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f193806adb0 0x7f193806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:53.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.351+0000 7f194a7fc700 1 --2- 172.21.15.1:0/220600140 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f193806adb0 0x7f193806d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f194c0b5d10 tx=0x7f193c009040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:53.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.355+0000 7f19337fe700 1 -- 172.21.15.1:0/220600140 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1940056600 con 0x7f1934003a30 2024-03-18T20:29:53.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.554+0000 7f19512f9700 1 -- 172.21.15.1:0/220600140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f194c0b80a0 con 0x7f1934003a30 2024-03-18T20:29:53.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.558+0000 7f19337fe700 1 -- 172.21.15.1:0/220600140 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v97) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1940007da0 con 0x7f1934003a30 2024-03-18T20:29:53.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.560+0000 7f19512f9700 1 -- 172.21.15.1:0/220600140 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f193806adb0 msgr2=0x7f193806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:53.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.560+0000 7f19512f9700 1 --2- 172.21.15.1:0/220600140 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f193806adb0 0x7f193806d270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f194c0b5d10 tx=0x7f193c009040 comp rx=0 tx=0).stop 2024-03-18T20:29:53.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.560+0000 7f19512f9700 1 -- 172.21.15.1:0/220600140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1934003a30 msgr2=0x7f194c0b8250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:53.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.560+0000 7f19512f9700 1 --2- 172.21.15.1:0/220600140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1934003a30 0x7f194c0b8250 secure :-1 s=READY pgs=3401 cs=0 l=1 rev1=1 crypto rx=0x7f1940008fd0 tx=0x7f1940001a00 comp rx=0 tx=0).stop 2024-03-18T20:29:53.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.560+0000 7f19512f9700 1 -- 172.21.15.1:0/220600140 shutdown_connections 2024-03-18T20:29:53.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.560+0000 7f19512f9700 1 --2- 172.21.15.1:0/220600140 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f193806adb0 0x7f193806d270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.560+0000 7f19512f9700 1 --2- 172.21.15.1:0/220600140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f194c147550 0x7f194c1479d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.560+0000 7f19512f9700 1 --2- 172.21.15.1:0/220600140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1934003a30 0x7f194c0b8250 unknown :-1 s=CLOSED pgs=3401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.560+0000 7f19512f9700 1 --2- 172.21.15.1:0/220600140 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f194c0b6bf0 0x7f194c15e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.560+0000 7f19512f9700 1 -- 172.21.15.1:0/220600140 >> 172.21.15.1:0/220600140 conn(0x7f194c0d3380 msgr2=0x7f194c15b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:53.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.560+0000 7f19512f9700 1 -- 172.21.15.1:0/220600140 shutdown_connections 2024-03-18T20:29:53.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.560+0000 7f19512f9700 1 -- 172.21.15.1:0/220600140 wait complete. 2024-03-18T20:29:53.578 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2024-03-18T20:29:53.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.700+0000 7f0daf386700 1 Processor -- start 2024-03-18T20:29:53.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.700+0000 7f0daf386700 1 -- start start 2024-03-18T20:29:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.701+0000 7f0daf386700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0da815d8f0 0x7f0da80b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.701+0000 7f0daf386700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0da8072050 0x7f0da80b4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.701+0000 7f0daf386700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0da80b1960 con 0x7f0da80b35e0 2024-03-18T20:29:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.701+0000 7f0daf386700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0da80b1bb0 con 0x7f0da815d8f0 2024-03-18T20:29:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.701+0000 7f0daf386700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0da80adb70 con 0x7f0da8072050 2024-03-18T20:29:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.701+0000 7f0dad923700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0da8072050 0x7f0da80b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.701+0000 7f0dad122700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0da815d8f0 0x7f0da80b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.701+0000 7f0dac921700 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f0da80b35e0 0x7f0da806a7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:51674/0 (socket says 172.21.15.1:51674) 2024-03-18T20:29:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.701+0000 7f0dac921700 1 -- 172.21.15.1:0/1633245819 learned_addr learned my addr 172.21.15.1:0/1633245819 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:29:53.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.702+0000 7f0dad122700 1 -- 172.21.15.1:0/1633245819 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0da8072050 msgr2=0x7f0da80b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.702+0000 7f0dad122700 1 --2- 172.21.15.1:0/1633245819 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0da8072050 0x7f0da80b4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.702+0000 7f0dad122700 1 -- 172.21.15.1:0/1633245819 >> v1:172.21.15.165:6789/0 conn(0x7f0da80b35e0 legacy=0x7f0da806a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.702+0000 7f0dad122700 1 -- 172.21.15.1:0/1633245819 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da80ade50 con 0x7f0da815d8f0 2024-03-18T20:29:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.702+0000 7f0dad923700 1 --2- 172.21.15.1:0/1633245819 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0da8072050 0x7f0da80b4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:29:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.702+0000 7f0dad122700 1 --2- 172.21.15.1:0/1633245819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0da815d8f0 0x7f0da80b3de0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f0d98009840 tx=0x7f0d98009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=511f8d5f1877d445 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.702+0000 7f0d9ffff700 1 -- 172.21.15.1:0/1633245819 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d98011830 con 0x7f0da815d8f0 2024-03-18T20:29:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.702+0000 7f0d9ffff700 1 -- 172.21.15.1:0/1633245819 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d98011990 con 0x7f0da815d8f0 2024-03-18T20:29:53.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.702+0000 7f0d9ffff700 1 -- 172.21.15.1:0/1633245819 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d98011d20 con 0x7f0da815d8f0 2024-03-18T20:29:53.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.703+0000 7f0daf386700 1 -- 172.21.15.1:0/1633245819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0da815d8f0 msgr2=0x7f0da80b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:53.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.703+0000 7f0daf386700 1 --2- 172.21.15.1:0/1633245819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0da815d8f0 0x7f0da80b3de0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f0d98009840 tx=0x7f0d98009b50 comp rx=0 tx=0).stop 2024-03-18T20:29:53.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.703+0000 7f0daf386700 1 -- 172.21.15.1:0/1633245819 shutdown_connections 2024-03-18T20:29:53.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.703+0000 7f0daf386700 1 --2- 172.21.15.1:0/1633245819 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0da8072050 0x7f0da80b4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.703+0000 7f0daf386700 1 --2- 172.21.15.1:0/1633245819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0da815d8f0 0x7f0da80b3de0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.703+0000 7f0daf386700 1 -- 172.21.15.1:0/1633245819 >> 172.21.15.1:0/1633245819 conn(0x7f0da80b5a10 msgr2=0x7f0da80b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:53.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.703+0000 7f0daf386700 1 -- 172.21.15.1:0/1633245819 shutdown_connections 2024-03-18T20:29:53.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.703+0000 7f0daf386700 1 -- 172.21.15.1:0/1633245819 wait complete. 2024-03-18T20:29:53.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.704+0000 7f0daf386700 1 Processor -- start 2024-03-18T20:29:53.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.704+0000 7f0daf386700 1 -- start start 2024-03-18T20:29:53.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.705+0000 7f0daf386700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0da8072050 0x7f0da8141a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:53.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.705+0000 7f0daf386700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0da815d8f0 0x7f0da814f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:53.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.705+0000 7f0daf386700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0da814c7f0 0x7f0da814d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:53.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.705+0000 7f0daf386700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da814e240 con 0x7f0da814c7f0 2024-03-18T20:29:53.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.705+0000 7f0daf386700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da815dd60 con 0x7f0da8072050 2024-03-18T20:29:53.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.705+0000 7f0daf386700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0da8141f50 con 0x7f0da815d8f0 2024-03-18T20:29:53.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.705+0000 7f0dad923700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0da814c7f0 0x7f0da814d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:53.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.705+0000 7f0dad923700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0da814c7f0 0x7f0da814d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60442/0 (socket says 172.21.15.1:60442) 2024-03-18T20:29:53.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.705+0000 7f0dad923700 1 -- 172.21.15.1:0/588915401 learned_addr learned my addr 172.21.15.1:0/588915401 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:53.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.705+0000 7f0dad122700 1 --2- 172.21.15.1:0/588915401 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0da8072050 0x7f0da8141a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:53.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.705+0000 7f0dac921700 1 --2- 172.21.15.1:0/588915401 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0da815d8f0 0x7f0da814f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:53.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.705+0000 7f0dad923700 1 -- 172.21.15.1:0/588915401 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0da815d8f0 msgr2=0x7f0da814f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:53.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.705+0000 7f0dad923700 1 --2- 172.21.15.1:0/588915401 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0da815d8f0 0x7f0da814f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.705+0000 7f0dad923700 1 -- 172.21.15.1:0/588915401 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0da8072050 msgr2=0x7f0da8141a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:53.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.705+0000 7f0dad923700 1 --2- 172.21.15.1:0/588915401 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0da8072050 0x7f0da8141a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.705+0000 7f0dad923700 1 -- 172.21.15.1:0/588915401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da0009a70 con 0x7f0da814c7f0 2024-03-18T20:29:53.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.705+0000 7f0dad923700 1 --2- 172.21.15.1:0/588915401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0da814c7f0 0x7f0da814d5f0 secure :-1 s=READY pgs=3402 cs=0 l=1 rev1=1 crypto rx=0x7f0da000efc0 tx=0x7f0da000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:53.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.705+0000 7f0d9dffb700 1 -- 172.21.15.1:0/588915401 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da000cc00 con 0x7f0da814c7f0 2024-03-18T20:29:53.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.705+0000 7f0d9dffb700 1 -- 172.21.15.1:0/588915401 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da000f040 con 0x7f0da814c7f0 2024-03-18T20:29:53.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.706+0000 7f0daf386700 1 -- 172.21.15.1:0/588915401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d980094a0 con 0x7f0da814c7f0 2024-03-18T20:29:53.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.706+0000 7f0d9dffb700 1 -- 172.21.15.1:0/588915401 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da0003a20 con 0x7f0da814c7f0 2024-03-18T20:29:53.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.706+0000 7f0daf386700 1 -- 172.21.15.1:0/588915401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0da815dfb0 con 0x7f0da814c7f0 2024-03-18T20:29:53.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.707+0000 7f0daf386700 1 -- 172.21.15.1:0/588915401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0da80b5530 con 0x7f0da814c7f0 2024-03-18T20:29:53.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.707+0000 7f0d9dffb700 1 -- 172.21.15.1:0/588915401 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f0da0014480 con 0x7f0da814c7f0 2024-03-18T20:29:53.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.707+0000 7f0d9dffb700 1 --2- 172.21.15.1:0/588915401 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0d8c06adb0 0x7f0d8c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:53.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.708+0000 7f0d9dffb700 1 -- 172.21.15.1:0/588915401 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f0da008a190 con 0x7f0da814c7f0 2024-03-18T20:29:53.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.708+0000 7f0dad122700 1 --2- 172.21.15.1:0/588915401 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0d8c06adb0 0x7f0d8c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:53.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.708+0000 7f0dad122700 1 --2- 172.21.15.1:0/588915401 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0d8c06adb0 0x7f0d8c06d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0d98000c00 tx=0x7f0d9801c040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:53.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.712+0000 7f0d9dffb700 1 -- 172.21.15.1:0/588915401 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0da0054f30 con 0x7f0da814c7f0 2024-03-18T20:29:53.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.917+0000 7f0daf386700 1 -- 172.21.15.1:0/588915401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f0da8067dc0 con 0x7f0da814c7f0 2024-03-18T20:29:53.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.921+0000 7f0d9dffb700 1 -- 172.21.15.1:0/588915401 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v98) v1 ==== 120+0+0 (secure 0 0 0) 0x7f0da0058550 con 0x7f0da814c7f0 2024-03-18T20:29:53.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.923+0000 7f0daf386700 1 -- 172.21.15.1:0/588915401 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0d8c06adb0 msgr2=0x7f0d8c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:53.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.923+0000 7f0daf386700 1 --2- 172.21.15.1:0/588915401 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0d8c06adb0 0x7f0d8c06d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0d98000c00 tx=0x7f0d9801c040 comp rx=0 tx=0).stop 2024-03-18T20:29:53.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.923+0000 7f0daf386700 1 -- 172.21.15.1:0/588915401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0da814c7f0 msgr2=0x7f0da814d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:53.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.923+0000 7f0daf386700 1 --2- 172.21.15.1:0/588915401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0da814c7f0 0x7f0da814d5f0 secure :-1 s=READY pgs=3402 cs=0 l=1 rev1=1 crypto rx=0x7f0da000efc0 tx=0x7f0da000c3a0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.923+0000 7f0daf386700 1 -- 172.21.15.1:0/588915401 shutdown_connections 2024-03-18T20:29:53.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.923+0000 7f0daf386700 1 --2- 172.21.15.1:0/588915401 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f0d8c06adb0 0x7f0d8c06d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.923+0000 7f0daf386700 1 --2- 172.21.15.1:0/588915401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0da814c7f0 0x7f0da814d5f0 unknown :-1 s=CLOSED pgs=3402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.923+0000 7f0daf386700 1 --2- 172.21.15.1:0/588915401 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0da8072050 0x7f0da8141a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.923+0000 7f0daf386700 1 --2- 172.21.15.1:0/588915401 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f0da815d8f0 0x7f0da814f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:53.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.923+0000 7f0daf386700 1 -- 172.21.15.1:0/588915401 >> 172.21.15.1:0/588915401 conn(0x7f0da80b5a10 msgr2=0x7f0da80b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:53.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.923+0000 7f0daf386700 1 -- 172.21.15.1:0/588915401 shutdown_connections 2024-03-18T20:29:53.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:53.923+0000 7f0daf386700 1 -- 172.21.15.1:0/588915401 wait complete. 2024-03-18T20:29:53.940 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-18T20:29:54.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.069+0000 7efc41f48700 1 Processor -- start 2024-03-18T20:29:54.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.069+0000 7efc41f48700 1 -- start start 2024-03-18T20:29:54.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.070+0000 7efc41f48700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7efc3c0afd70 0x7efc3c0adbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:54.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.070+0000 7efc41f48700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc3c0b20b0 0x7efc3c1610f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:54.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.070+0000 7efc41f48700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc3c0b3c80 con 0x7efc3c0ad0f0 2024-03-18T20:29:54.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.070+0000 7efc41f48700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc3c160f30 con 0x7efc3c0afd70 2024-03-18T20:29:54.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.070+0000 7efc41f48700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c0ae120 con 0x7efc3c0b20b0 2024-03-18T20:29:54.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.071+0000 7efc3bfff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc3c0b20b0 0x7efc3c1610f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:54.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.071+0000 7efc3bfff700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc3c0b20b0 0x7efc3c1610f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:57112/0 (socket says 172.21.15.1:57112) 2024-03-18T20:29:54.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.071+0000 7efc3bfff700 1 -- 172.21.15.1:0/2760841877 learned_addr learned my addr 172.21.15.1:0/2760841877 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:54.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.071+0000 7efc3affd700 1 --2- 172.21.15.1:0/2760841877 >> v2:172.21.15.165:3300/0 conn(0x7efc3c0afd70 0x7efc3c0adbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:54.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.071+0000 7efc3bfff700 1 -- 172.21.15.1:0/2760841877 >> v2:172.21.15.165:3300/0 conn(0x7efc3c0afd70 msgr2=0x7efc3c0adbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:54.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.071+0000 7efc3bfff700 1 --2- 172.21.15.1:0/2760841877 >> v2:172.21.15.165:3300/0 conn(0x7efc3c0afd70 0x7efc3c0adbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:54.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.071+0000 7efc3bfff700 1 -- 172.21.15.1:0/2760841877 >> v1:172.21.15.1:6789/0 conn(0x7efc3c0ad0f0 legacy=0x7efc3c0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:54.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.071+0000 7efc3bfff700 1 -- 172.21.15.1:0/2760841877 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3c0b3070 con 0x7efc3c0b20b0 2024-03-18T20:29:54.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.072+0000 7efc3bfff700 1 --2- 172.21.15.1:0/2760841877 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc3c0b20b0 0x7efc3c1610f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7efc34009e50 tx=0x7efc3400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9494872ef448514 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:54.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.072+0000 7efc3a7fc700 1 -- 172.21.15.1:0/2760841877 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3400e070 con 0x7efc3c0b20b0 2024-03-18T20:29:54.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.072+0000 7efc3a7fc700 1 -- 172.21.15.1:0/2760841877 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc34003a20 con 0x7efc3c0b20b0 2024-03-18T20:29:54.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.072+0000 7efc3a7fc700 1 -- 172.21.15.1:0/2760841877 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc34003dd0 con 0x7efc3c0b20b0 2024-03-18T20:29:54.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.073+0000 7efc41f48700 1 -- 172.21.15.1:0/2760841877 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc3c0b20b0 msgr2=0x7efc3c1610f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:54.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.073+0000 7efc41f48700 1 --2- 172.21.15.1:0/2760841877 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc3c0b20b0 0x7efc3c1610f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7efc34009e50 tx=0x7efc3400b330 comp rx=0 tx=0).stop 2024-03-18T20:29:54.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.074+0000 7efc41f48700 1 -- 172.21.15.1:0/2760841877 shutdown_connections 2024-03-18T20:29:54.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.074+0000 7efc41f48700 1 --2- 172.21.15.1:0/2760841877 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc3c0b20b0 0x7efc3c1610f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:54.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.074+0000 7efc41f48700 1 --2- 172.21.15.1:0/2760841877 >> v2:172.21.15.165:3300/0 conn(0x7efc3c0afd70 0x7efc3c0adbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:54.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.074+0000 7efc41f48700 1 -- 172.21.15.1:0/2760841877 >> 172.21.15.1:0/2760841877 conn(0x7efc3c0b0db0 msgr2=0x7efc3c0b11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:54.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.074+0000 7efc41f48700 1 -- 172.21.15.1:0/2760841877 shutdown_connections 2024-03-18T20:29:54.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.074+0000 7efc41f48700 1 -- 172.21.15.1:0/2760841877 wait complete. 2024-03-18T20:29:54.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.075+0000 7efc41f48700 1 Processor -- start 2024-03-18T20:29:54.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.076+0000 7efc41f48700 1 -- start start 2024-03-18T20:29:54.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.076+0000 7efc41f48700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc3c0ad0f0 0x7efc3c0cded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:54.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.076+0000 7efc41f48700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc3c0afd70 0x7efc3c150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:54.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.076+0000 7efc41f48700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc3c0b20b0 0x7efc3c150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:54.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.076+0000 7efc41f48700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c14da40 con 0x7efc3c0b20b0 2024-03-18T20:29:54.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.076+0000 7efc41f48700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c14f7e0 con 0x7efc3c0ad0f0 2024-03-18T20:29:54.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.076+0000 7efc41f48700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c0cbb10 con 0x7efc3c0afd70 2024-03-18T20:29:54.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.077+0000 7efc3bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc3c0b20b0 0x7efc3c150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:54.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.077+0000 7efc3bfff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc3c0b20b0 0x7efc3c150760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60448/0 (socket says 172.21.15.1:60448) 2024-03-18T20:29:54.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.077+0000 7efc3bfff700 1 -- 172.21.15.1:0/4198937676 learned_addr learned my addr 172.21.15.1:0/4198937676 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:54.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.077+0000 7efc3b7fe700 1 --2- 172.21.15.1:0/4198937676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc3c0ad0f0 0x7efc3c0cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:54.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.077+0000 7efc3affd700 1 --2- 172.21.15.1:0/4198937676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc3c0afd70 0x7efc3c150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:54.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.077+0000 7efc3bfff700 1 -- 172.21.15.1:0/4198937676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc3c0afd70 msgr2=0x7efc3c150220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:54.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.077+0000 7efc3bfff700 1 --2- 172.21.15.1:0/4198937676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc3c0afd70 0x7efc3c150220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:54.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.077+0000 7efc3bfff700 1 -- 172.21.15.1:0/4198937676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc3c0ad0f0 msgr2=0x7efc3c0cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:54.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.077+0000 7efc3bfff700 1 --2- 172.21.15.1:0/4198937676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc3c0ad0f0 0x7efc3c0cded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:54.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.077+0000 7efc3bfff700 1 -- 172.21.15.1:0/4198937676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc34009b30 con 0x7efc3c0b20b0 2024-03-18T20:29:54.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.077+0000 7efc3bfff700 1 --2- 172.21.15.1:0/4198937676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc3c0b20b0 0x7efc3c150760 secure :-1 s=READY pgs=3404 cs=0 l=1 rev1=1 crypto rx=0x7efc3400b680 tx=0x7efc34004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:54.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.077+0000 7efc23fff700 1 -- 172.21.15.1:0/4198937676 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc34004440 con 0x7efc3c0b20b0 2024-03-18T20:29:54.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.077+0000 7efc23fff700 1 -- 172.21.15.1:0/4198937676 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc340045c0 con 0x7efc3c0b20b0 2024-03-18T20:29:54.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.077+0000 7efc23fff700 1 -- 172.21.15.1:0/4198937676 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc34004970 con 0x7efc3c0b20b0 2024-03-18T20:29:54.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.077+0000 7efc41f48700 1 -- 172.21.15.1:0/4198937676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc3c14f220 con 0x7efc3c0b20b0 2024-03-18T20:29:54.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.078+0000 7efc41f48700 1 -- 172.21.15.1:0/4198937676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc3c0cb7a0 con 0x7efc3c0b20b0 2024-03-18T20:29:54.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.079+0000 7efc41f48700 1 -- 172.21.15.1:0/4198937676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc3c0676b0 con 0x7efc3c0b20b0 2024-03-18T20:29:54.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.079+0000 7efc23fff700 1 -- 172.21.15.1:0/4198937676 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7efc34019050 con 0x7efc3c0b20b0 2024-03-18T20:29:54.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.080+0000 7efc23fff700 1 --2- 172.21.15.1:0/4198937676 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efc2806adb0 0x7efc2806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:54.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.080+0000 7efc3b7fe700 1 --2- 172.21.15.1:0/4198937676 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efc2806adb0 0x7efc2806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:54.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.080+0000 7efc23fff700 1 -- 172.21.15.1:0/4198937676 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7efc3408aa50 con 0x7efc3c0b20b0 2024-03-18T20:29:54.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.080+0000 7efc3b7fe700 1 --2- 172.21.15.1:0/4198937676 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efc2806adb0 0x7efc2806d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efc2c007fd0 tx=0x7efc2c009040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:54.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.084+0000 7efc23fff700 1 -- 172.21.15.1:0/4198937676 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efc34055880 con 0x7efc3c0b20b0 2024-03-18T20:29:54.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.287+0000 7efc41f48700 1 -- 172.21.15.1:0/4198937676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7efc3c0cbc50 con 0x7efc3c0b20b0 2024-03-18T20:29:54.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.291+0000 7efc23fff700 1 -- 172.21.15.1:0/4198937676 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v99) v1 ==== 104+0+0 (secure 0 0 0) 0x7efc340170a0 con 0x7efc3c0b20b0 2024-03-18T20:29:54.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.293+0000 7efc41f48700 1 -- 172.21.15.1:0/4198937676 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efc2806adb0 msgr2=0x7efc2806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:54.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.293+0000 7efc41f48700 1 --2- 172.21.15.1:0/4198937676 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efc2806adb0 0x7efc2806d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efc2c007fd0 tx=0x7efc2c009040 comp rx=0 tx=0).stop 2024-03-18T20:29:54.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.293+0000 7efc41f48700 1 -- 172.21.15.1:0/4198937676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc3c0b20b0 msgr2=0x7efc3c150760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:54.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.293+0000 7efc41f48700 1 --2- 172.21.15.1:0/4198937676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc3c0b20b0 0x7efc3c150760 secure :-1 s=READY pgs=3404 cs=0 l=1 rev1=1 crypto rx=0x7efc3400b680 tx=0x7efc34004ad0 comp rx=0 tx=0).stop 2024-03-18T20:29:54.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.293+0000 7efc41f48700 1 -- 172.21.15.1:0/4198937676 shutdown_connections 2024-03-18T20:29:54.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.293+0000 7efc41f48700 1 --2- 172.21.15.1:0/4198937676 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7efc2806adb0 0x7efc2806d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:54.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.293+0000 7efc41f48700 1 --2- 172.21.15.1:0/4198937676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc3c0b20b0 0x7efc3c150760 unknown :-1 s=CLOSED pgs=3404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:54.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.293+0000 7efc41f48700 1 --2- 172.21.15.1:0/4198937676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc3c0ad0f0 0x7efc3c0cded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:54.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.293+0000 7efc41f48700 1 --2- 172.21.15.1:0/4198937676 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7efc3c0afd70 0x7efc3c150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:54.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.293+0000 7efc41f48700 1 -- 172.21.15.1:0/4198937676 >> 172.21.15.1:0/4198937676 conn(0x7efc3c0b0db0 msgr2=0x7efc3c15e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:54.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.293+0000 7efc41f48700 1 -- 172.21.15.1:0/4198937676 shutdown_connections 2024-03-18T20:29:54.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.293+0000 7efc41f48700 1 -- 172.21.15.1:0/4198937676 wait complete. 2024-03-18T20:29:54.310 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01' 2024-03-18T20:29:54.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.430+0000 7fabaaba9700 1 Processor -- start 2024-03-18T20:29:54.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.431+0000 7fabaaba9700 1 -- start start 2024-03-18T20:29:54.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.431+0000 7fabaaba9700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7faba415d8f0 0x7faba40b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:54.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.431+0000 7fabaaba9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faba40b35e0 0x7faba40b4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:54.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.431+0000 7fabaaba9700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba40b1960 con 0x7faba4072050 2024-03-18T20:29:54.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.431+0000 7fabaaba9700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba40b1bb0 con 0x7faba40b35e0 2024-03-18T20:29:54.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.431+0000 7fabaaba9700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba40adb70 con 0x7faba415d8f0 2024-03-18T20:29:54.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.431+0000 7faba3fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faba40b35e0 0x7faba40b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:54.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.432+0000 7faba3fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faba40b35e0 0x7faba40b4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49986/0 (socket says 172.21.15.1:49986) 2024-03-18T20:29:54.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.432+0000 7faba3fff700 1 -- 172.21.15.1:0/1550008902 learned_addr learned my addr 172.21.15.1:0/1550008902 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:54.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.432+0000 7faba8945700 1 --2- 172.21.15.1:0/1550008902 >> v2:172.21.15.165:3300/0 conn(0x7faba415d8f0 0x7faba40b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:54.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.432+0000 7faba3fff700 1 -- 172.21.15.1:0/1550008902 >> v2:172.21.15.165:3300/0 conn(0x7faba415d8f0 msgr2=0x7faba40b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:54.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.432+0000 7faba3fff700 1 --2- 172.21.15.1:0/1550008902 >> v2:172.21.15.165:3300/0 conn(0x7faba415d8f0 0x7faba40b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:54.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.432+0000 7faba3fff700 1 -- 172.21.15.1:0/1550008902 >> v1:172.21.15.165:6789/0 conn(0x7faba4072050 legacy=0x7faba406a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:54.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.432+0000 7faba3fff700 1 -- 172.21.15.1:0/1550008902 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba40ade50 con 0x7faba40b35e0 2024-03-18T20:29:54.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.432+0000 7faba3fff700 1 --2- 172.21.15.1:0/1550008902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faba40b35e0 0x7faba40b4320 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 crypto rx=0x7faba40b5f10 tx=0x7fab90009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b2c25b2f20fe974 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:54.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.432+0000 7faba37fe700 1 -- 172.21.15.1:0/1550008902 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9000b870 con 0x7faba40b35e0 2024-03-18T20:29:54.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.432+0000 7faba37fe700 1 -- 172.21.15.1:0/1550008902 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9000b9d0 con 0x7faba40b35e0 2024-03-18T20:29:54.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.432+0000 7faba37fe700 1 -- 172.21.15.1:0/1550008902 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9000bd60 con 0x7faba40b35e0 2024-03-18T20:29:54.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.433+0000 7fabaaba9700 1 -- 172.21.15.1:0/1550008902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faba40b35e0 msgr2=0x7faba40b4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:54.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.433+0000 7fabaaba9700 1 --2- 172.21.15.1:0/1550008902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faba40b35e0 0x7faba40b4320 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 crypto rx=0x7faba40b5f10 tx=0x7fab90009bd0 comp rx=0 tx=0).stop 2024-03-18T20:29:54.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.433+0000 7fabaaba9700 1 -- 172.21.15.1:0/1550008902 shutdown_connections 2024-03-18T20:29:54.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.433+0000 7fabaaba9700 1 --2- 172.21.15.1:0/1550008902 >> v2:172.21.15.165:3300/0 conn(0x7faba415d8f0 0x7faba40b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:54.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.433+0000 7fabaaba9700 1 --2- 172.21.15.1:0/1550008902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faba40b35e0 0x7faba40b4320 unknown :-1 s=CLOSED pgs=3405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:54.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.433+0000 7fabaaba9700 1 -- 172.21.15.1:0/1550008902 >> 172.21.15.1:0/1550008902 conn(0x7faba40b5a10 msgr2=0x7faba40b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:54.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.433+0000 7fabaaba9700 1 -- 172.21.15.1:0/1550008902 shutdown_connections 2024-03-18T20:29:54.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.433+0000 7fabaaba9700 1 -- 172.21.15.1:0/1550008902 wait complete. 2024-03-18T20:29:54.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.434+0000 7fabaaba9700 1 Processor -- start 2024-03-18T20:29:54.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.434+0000 7fabaaba9700 1 -- start start 2024-03-18T20:29:54.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.435+0000 7fabaaba9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faba4072050 0x7faba415b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:54.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.435+0000 7fabaaba9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faba415d8f0 0x7faba4152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.435+0000 7fabaaba9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faba4153d70 0x7faba40b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.435+0000 7fabaaba9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba4153240 con 0x7faba415d8f0 2024-03-18T20:29:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.435+0000 7fabaaba9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba41538e0 con 0x7faba4072050 2024-03-18T20:29:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.435+0000 7fabaaba9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba41e9830 con 0x7faba4153d70 2024-03-18T20:29:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.435+0000 7faba3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faba415d8f0 0x7faba4152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.435+0000 7faba3fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faba415d8f0 0x7faba4152d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50000/0 (socket says 172.21.15.1:50000) 2024-03-18T20:29:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.435+0000 7faba3fff700 1 -- 172.21.15.1:0/3246227396 learned_addr learned my addr 172.21.15.1:0/3246227396 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.435+0000 7faba8945700 1 --2- 172.21.15.1:0/3246227396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faba4072050 0x7faba415b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.435+0000 7faba9146700 1 --2- 172.21.15.1:0/3246227396 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faba4153d70 0x7faba40b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.435+0000 7faba3fff700 1 -- 172.21.15.1:0/3246227396 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faba4153d70 msgr2=0x7faba40b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.435+0000 7faba3fff700 1 --2- 172.21.15.1:0/3246227396 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faba4153d70 0x7faba40b1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.435+0000 7faba3fff700 1 -- 172.21.15.1:0/3246227396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faba4072050 msgr2=0x7faba415b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.435+0000 7faba3fff700 1 --2- 172.21.15.1:0/3246227396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faba4072050 0x7faba415b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:54.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.435+0000 7faba3fff700 1 -- 172.21.15.1:0/3246227396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab90009550 con 0x7faba415d8f0 2024-03-18T20:29:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.435+0000 7faba9146700 1 --2- 172.21.15.1:0/3246227396 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faba4153d70 0x7faba40b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:29:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.435+0000 7faba3fff700 1 --2- 172.21.15.1:0/3246227396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faba415d8f0 0x7faba4152d00 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7faba4071390 tx=0x7fab90004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.436+0000 7faba17fa700 1 -- 172.21.15.1:0/3246227396 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab90004540 con 0x7faba415d8f0 2024-03-18T20:29:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.436+0000 7faba17fa700 1 -- 172.21.15.1:0/3246227396 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab900036a0 con 0x7faba415d8f0 2024-03-18T20:29:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.436+0000 7fabaaba9700 1 -- 172.21.15.1:0/3246227396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba41548d0 con 0x7faba415d8f0 2024-03-18T20:29:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.436+0000 7faba17fa700 1 -- 172.21.15.1:0/3246227396 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab90003a30 con 0x7faba415d8f0 2024-03-18T20:29:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.436+0000 7fabaaba9700 1 -- 172.21.15.1:0/3246227396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba4154e90 con 0x7faba415d8f0 2024-03-18T20:29:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.437+0000 7fabaaba9700 1 -- 172.21.15.1:0/3246227396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba4067dc0 con 0x7faba415d8f0 2024-03-18T20:29:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.437+0000 7faba17fa700 1 -- 172.21.15.1:0/3246227396 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fab90021070 con 0x7faba415d8f0 2024-03-18T20:29:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.438+0000 7faba17fa700 1 --2- 172.21.15.1:0/3246227396 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fab8806adb0 0x7fab8806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.438+0000 7faba17fa700 1 -- 172.21.15.1:0/3246227396 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7fab9008abe0 con 0x7faba415d8f0 2024-03-18T20:29:54.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.438+0000 7faba8945700 1 --2- 172.21.15.1:0/3246227396 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fab8806adb0 0x7fab8806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:54.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.438+0000 7faba8945700 1 --2- 172.21.15.1:0/3246227396 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fab8806adb0 0x7fab8806d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fab98006b20 tx=0x7fab98008040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:54.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.442+0000 7faba17fa700 1 -- 172.21.15.1:0/3246227396 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fab90055c70 con 0x7faba415d8f0 2024-03-18T20:29:54.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:54.730+0000 7fabaaba9700 1 -- 172.21.15.1:0/3246227396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]} v 0) v1 -- 0x7faba4153a20 con 0x7faba415d8f0 2024-03-18T20:29:55.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:55.186+0000 7faba17fa700 1 -- 172.21.15.1:0/3246227396 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]}]=0 v3323) v1 ==== 201+0+0 (secure 0 0 0) 0x7fab90029110 con 0x7faba415d8f0 2024-03-18T20:29:55.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:55.189+0000 7fabaaba9700 1 -- 172.21.15.1:0/3246227396 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fab8806adb0 msgr2=0x7fab8806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:55.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:55.189+0000 7fabaaba9700 1 --2- 172.21.15.1:0/3246227396 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fab8806adb0 0x7fab8806d270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fab98006b20 tx=0x7fab98008040 comp rx=0 tx=0).stop 2024-03-18T20:29:55.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:55.189+0000 7fabaaba9700 1 -- 172.21.15.1:0/3246227396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faba415d8f0 msgr2=0x7faba4152d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:55.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:55.189+0000 7fabaaba9700 1 --2- 172.21.15.1:0/3246227396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faba415d8f0 0x7faba4152d00 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7faba4071390 tx=0x7fab90004750 comp rx=0 tx=0).stop 2024-03-18T20:29:55.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:55.189+0000 7fabaaba9700 1 -- 172.21.15.1:0/3246227396 shutdown_connections 2024-03-18T20:29:55.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:55.189+0000 7fabaaba9700 1 --2- 172.21.15.1:0/3246227396 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fab8806adb0 0x7fab8806d270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:55.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:55.189+0000 7fabaaba9700 1 --2- 172.21.15.1:0/3246227396 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7faba4153d70 0x7faba40b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:55.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:55.189+0000 7fabaaba9700 1 --2- 172.21.15.1:0/3246227396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faba4072050 0x7faba415b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:55.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:55.189+0000 7fabaaba9700 1 --2- 172.21.15.1:0/3246227396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faba415d8f0 0x7faba4152d00 unknown :-1 s=CLOSED pgs=3406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:55.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:55.189+0000 7fabaaba9700 1 -- 172.21.15.1:0/3246227396 >> 172.21.15.1:0/3246227396 conn(0x7faba40b5a10 msgr2=0x7faba40afcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:55.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:55.189+0000 7fabaaba9700 1 -- 172.21.15.1:0/3246227396 shutdown_connections 2024-03-18T20:29:55.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:55.189+0000 7fabaaba9700 1 -- 172.21.15.1:0/3246227396 wait complete. 2024-03-18T20:29:55.213 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. ... ERROR 2024-03-18T20:29:55.213 INFO:tasks.cephfs_test_runner: 2024-03-18T20:29:55.214 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-18T20:29:55.214 INFO:tasks.cephfs_test_runner:ERROR: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-03-18T20:29:55.214 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. 2024-03-18T20:29:55.214 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-18T20:29:55.214 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-18T20:29:55.214 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_49bdc51142759622a281d3e5327eebd07cbb8bb3/qa/tasks/cephfs/test_snap_schedules.py", line 678, in test_snap_schedule_subvol_and_group_arguments_01 2024-03-18T20:29:55.214 INFO:tasks.cephfs_test_runner: self._verify_snap_schedule(self.CREATE_VERSION, 'sv01', None) 2024-03-18T20:29:55.214 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_49bdc51142759622a281d3e5327eebd07cbb8bb3/qa/tasks/cephfs/test_snap_schedules.py", line 660, in _verify_snap_schedule 2024-03-18T20:29:55.214 INFO:tasks.cephfs_test_runner: path = self._get_subvol_snapdir_path(version, subvol, group) 2024-03-18T20:29:55.214 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_49bdc51142759622a281d3e5327eebd07cbb8bb3/qa/tasks/cephfs/test_snap_schedules.py", line 653, in _get_subvol_snapdir_path 2024-03-18T20:29:55.214 INFO:tasks.cephfs_test_runner: path = self.get_ceph_cmd_stdout("fs", *args).rstrip() 2024-03-18T20:29:55.214 INFO:tasks.cephfs_test_runner:AttributeError: 'TestSnapSchedulesSubvolAndGroupArguments' object has no attribute 'get_ceph_cmd_stdout' 2024-03-18T20:29:55.214 INFO:tasks.cephfs_test_runner: 2024-03-18T20:29:55.215 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-18T20:29:55.215 INFO:tasks.cephfs_test_runner:Ran 15 tests in 5957.823s 2024-03-18T20:29:55.215 INFO:tasks.cephfs_test_runner: 2024-03-18T20:29:55.215 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-18T20:29:55.215 INFO:tasks.cephfs_test_runner: 2024-03-18T20:29:55.215 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-18T20:29:55.215 INFO:tasks.cephfs_test_runner:ERROR: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-03-18T20:29:55.215 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. 2024-03-18T20:29:55.215 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-18T20:29:55.215 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-18T20:29:55.215 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_49bdc51142759622a281d3e5327eebd07cbb8bb3/qa/tasks/cephfs/test_snap_schedules.py", line 678, in test_snap_schedule_subvol_and_group_arguments_01 2024-03-18T20:29:55.215 INFO:tasks.cephfs_test_runner: self._verify_snap_schedule(self.CREATE_VERSION, 'sv01', None) 2024-03-18T20:29:55.215 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_49bdc51142759622a281d3e5327eebd07cbb8bb3/qa/tasks/cephfs/test_snap_schedules.py", line 660, in _verify_snap_schedule 2024-03-18T20:29:55.216 INFO:tasks.cephfs_test_runner: path = self._get_subvol_snapdir_path(version, subvol, group) 2024-03-18T20:29:55.216 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_49bdc51142759622a281d3e5327eebd07cbb8bb3/qa/tasks/cephfs/test_snap_schedules.py", line 653, in _get_subvol_snapdir_path 2024-03-18T20:29:55.216 INFO:tasks.cephfs_test_runner: path = self.get_ceph_cmd_stdout("fs", *args).rstrip() 2024-03-18T20:29:55.216 INFO:tasks.cephfs_test_runner:AttributeError: 'TestSnapSchedulesSubvolAndGroupArguments' object has no attribute 'get_ceph_cmd_stdout' 2024-03-18T20:29:55.216 INFO:tasks.cephfs_test_runner: 2024-03-18T20:29:55.216 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_49bdc51142759622a281d3e5327eebd07cbb8bb3/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-03-18T20:29:55.418 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8ebaf6ac9229413f92893d0158b1a87a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_49bdc51142759622a281d3e5327eebd07cbb8bb3/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-03-18T20:29:55.420 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-18T20:29:55.430 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-18T20:29:55.471 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-18T20:29:55.472 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:29:55.472 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:29:55.499 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:29:55.499 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-18T20:29:55.499 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:29:55.562 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16254224k,nr_inodes=4063556,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5550 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3254384k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2024-03-18T20:29:55.562 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:29:55.562 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:29:55.584 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:29:55.584 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-18T20:29:55.584 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T20:29:55.647 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16251156k,nr_inodes=4062789,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3726 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3253764k,mode=700,uid=1000,gid=1285 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2024-03-18T20:29:55.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:29:55.648 DEBUG:teuthology.orchestra.run.smithi165:> ip netns list 2024-03-18T20:29:55.708 INFO:teuthology.orchestra.run.smithi165.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-18T20:29:55.709 INFO:teuthology.orchestra.run.smithi165.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-18T20:29:55.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:29:55.709 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-18T20:29:55.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:29:55.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-18T20:29:55.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:29:55.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo ip link delete ceph-brx 2024-03-18T20:29:55.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:29:55.970 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-03-18T20:29:55.993 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-18T20:29:55.993 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T20:29:55.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:29:55.993 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-18T20:29:56.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:29:56.078 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T20:29:56.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T20:29:56.155 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link delete ceph-brx 2024-03-18T20:29:56.252 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-18T20:29:56.262 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-18T20:29:56.262 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:29:56.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.385+0000 7fc8a154a700 1 Processor -- start 2024-03-18T20:29:56.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.385+0000 7fc8a154a700 1 -- start start 2024-03-18T20:29:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.386+0000 7fc8a154a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc89c15d8c0 0x7fc89c0b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.386+0000 7fc8a154a700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc89c0d3820 0x7fc89c0b0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.386+0000 7fc8a154a700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc89c0b00a0 con 0x7fc89c0b3c30 2024-03-18T20:29:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.386+0000 7fc8a154a700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc89c0d1f40 con 0x7fc89c15d8c0 2024-03-18T20:29:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.386+0000 7fc8a154a700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc89c0b4260 con 0x7fc89c0d3820 2024-03-18T20:29:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.386+0000 7fc89affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc89c15d8c0 0x7fc89c0b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:56.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.386+0000 7fc89affd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc89c15d8c0 0x7fc89c0b36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50008/0 (socket says 172.21.15.1:50008) 2024-03-18T20:29:56.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.386+0000 7fc89affd700 1 -- 172.21.15.1:0/2583153609 learned_addr learned my addr 172.21.15.1:0/2583153609 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:56.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.386+0000 7fc89b7fe700 1 --2- 172.21.15.1:0/2583153609 >> v2:172.21.15.165:3300/0 conn(0x7fc89c0d3820 0x7fc89c0b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:56.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.386+0000 7fc89affd700 1 -- 172.21.15.1:0/2583153609 >> v2:172.21.15.165:3300/0 conn(0x7fc89c0d3820 msgr2=0x7fc89c0b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:56.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.386+0000 7fc89affd700 1 --2- 172.21.15.1:0/2583153609 >> v2:172.21.15.165:3300/0 conn(0x7fc89c0d3820 0x7fc89c0b0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:56.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.386+0000 7fc89affd700 1 -- 172.21.15.1:0/2583153609 >> v1:172.21.15.1:6789/0 conn(0x7fc89c0b3c30 legacy=0x7fc89c0d3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:56.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.386+0000 7fc89affd700 1 -- 172.21.15.1:0/2583153609 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89c0b4540 con 0x7fc89c15d8c0 2024-03-18T20:29:56.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.386+0000 7fc89b7fe700 1 --2- 172.21.15.1:0/2583153609 >> v2:172.21.15.165:3300/0 conn(0x7fc89c0d3820 0x7fc89c0b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:29:56.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.386+0000 7fc89affd700 1 --2- 172.21.15.1:0/2583153609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc89c15d8c0 0x7fc89c0b36f0 secure :-1 s=READY pgs=3408 cs=0 l=1 rev1=1 crypto rx=0x7fc89c08d5c0 tx=0x7fc890009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e77c38897ada73d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:56.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.387+0000 7fc899ffb700 1 -- 172.21.15.1:0/2583153609 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89000b870 con 0x7fc89c15d8c0 2024-03-18T20:29:56.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.387+0000 7fc899ffb700 1 -- 172.21.15.1:0/2583153609 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc89000b9d0 con 0x7fc89c15d8c0 2024-03-18T20:29:56.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.387+0000 7fc899ffb700 1 -- 172.21.15.1:0/2583153609 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89000bd60 con 0x7fc89c15d8c0 2024-03-18T20:29:56.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.387+0000 7fc8a154a700 1 -- 172.21.15.1:0/2583153609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc89c15d8c0 msgr2=0x7fc89c0b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:56.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.387+0000 7fc8a154a700 1 --2- 172.21.15.1:0/2583153609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc89c15d8c0 0x7fc89c0b36f0 secure :-1 s=READY pgs=3408 cs=0 l=1 rev1=1 crypto rx=0x7fc89c08d5c0 tx=0x7fc890009bd0 comp rx=0 tx=0).stop 2024-03-18T20:29:56.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.387+0000 7fc8a154a700 1 -- 172.21.15.1:0/2583153609 shutdown_connections 2024-03-18T20:29:56.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.387+0000 7fc8a154a700 1 --2- 172.21.15.1:0/2583153609 >> v2:172.21.15.165:3300/0 conn(0x7fc89c0d3820 0x7fc89c0b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:56.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.387+0000 7fc8a154a700 1 --2- 172.21.15.1:0/2583153609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc89c15d8c0 0x7fc89c0b36f0 unknown :-1 s=CLOSED pgs=3408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:56.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.387+0000 7fc8a154a700 1 -- 172.21.15.1:0/2583153609 >> 172.21.15.1:0/2583153609 conn(0x7fc89c08afc0 msgr2=0x7fc89c09c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:56.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.387+0000 7fc8a154a700 1 -- 172.21.15.1:0/2583153609 shutdown_connections 2024-03-18T20:29:56.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.387+0000 7fc8a154a700 1 -- 172.21.15.1:0/2583153609 wait complete. 2024-03-18T20:29:56.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.389+0000 7fc8a154a700 1 Processor -- start 2024-03-18T20:29:56.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.389+0000 7fc8a154a700 1 -- start start 2024-03-18T20:29:56.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.389+0000 7fc8a154a700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc89c0b3c30 0x7fc89c0cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:56.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.389+0000 7fc8a154a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc89c0d3820 0x7fc89c0cbd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:56.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc8a154a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc89c15d8c0 0x7fc89c15bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:56.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc8a154a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c15c410 con 0x7fc89c0d3820 2024-03-18T20:29:56.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc8a154a700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c09e5c0 con 0x7fc89c15d8c0 2024-03-18T20:29:56.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc8a154a700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c09e2e0 con 0x7fc89c0b3c30 2024-03-18T20:29:56.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc89a7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc89c0d3820 0x7fc89c0cbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:56.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc89a7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc89c0d3820 0x7fc89c0cbd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50020/0 (socket says 172.21.15.1:50020) 2024-03-18T20:29:56.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc89a7fc700 1 -- 172.21.15.1:0/1494754587 learned_addr learned my addr 172.21.15.1:0/1494754587 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:56.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc89b7fe700 1 --2- 172.21.15.1:0/1494754587 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc89c15d8c0 0x7fc89c15bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:56.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc89affd700 1 --2- 172.21.15.1:0/1494754587 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc89c0b3c30 0x7fc89c0cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:56.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc89a7fc700 1 -- 172.21.15.1:0/1494754587 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc89c0b3c30 msgr2=0x7fc89c0cd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:56.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc89a7fc700 1 --2- 172.21.15.1:0/1494754587 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc89c0b3c30 0x7fc89c0cd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:56.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc89a7fc700 1 -- 172.21.15.1:0/1494754587 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc89c15d8c0 msgr2=0x7fc89c15bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:56.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc89a7fc700 1 --2- 172.21.15.1:0/1494754587 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc89c15d8c0 0x7fc89c15bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:56.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc89a7fc700 1 -- 172.21.15.1:0/1494754587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc890009550 con 0x7fc89c0d3820 2024-03-18T20:29:56.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc89b7fe700 1 --2- 172.21.15.1:0/1494754587 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc89c15d8c0 0x7fc89c15bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:29:56.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc89a7fc700 1 --2- 172.21.15.1:0/1494754587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc89c0d3820 0x7fc89c0cbd30 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 crypto rx=0x7fc88c00c4e0 tx=0x7fc88c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:56.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc8837fe700 1 -- 172.21.15.1:0/1494754587 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88c0052d0 con 0x7fc89c0d3820 2024-03-18T20:29:56.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc8837fe700 1 -- 172.21.15.1:0/1494754587 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc88c005430 con 0x7fc89c0d3820 2024-03-18T20:29:56.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc8837fe700 1 -- 172.21.15.1:0/1494754587 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88c0038c0 con 0x7fc89c0d3820 2024-03-18T20:29:56.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc8a154a700 1 -- 172.21.15.1:0/1494754587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc89c0cb250 con 0x7fc89c0d3820 2024-03-18T20:29:56.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.390+0000 7fc8a154a700 1 -- 172.21.15.1:0/1494754587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc89c0cb810 con 0x7fc89c0d3820 2024-03-18T20:29:56.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.391+0000 7fc8a154a700 1 -- 172.21.15.1:0/1494754587 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc89c0676b0 con 0x7fc89c0d3820 2024-03-18T20:29:56.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.392+0000 7fc8837fe700 1 -- 172.21.15.1:0/1494754587 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc88c003a20 con 0x7fc89c0d3820 2024-03-18T20:29:56.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.392+0000 7fc8837fe700 1 --2- 172.21.15.1:0/1494754587 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc88806adb0 0x7fc88806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:56.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.392+0000 7fc8837fe700 1 -- 172.21.15.1:0/1494754587 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7fc88c088a50 con 0x7fc89c0d3820 2024-03-18T20:29:56.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.392+0000 7fc89affd700 1 --2- 172.21.15.1:0/1494754587 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc88806adb0 0x7fc88806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:56.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.393+0000 7fc89affd700 1 --2- 172.21.15.1:0/1494754587 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc88806adb0 0x7fc88806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc890014040 tx=0x7fc890008030 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:56.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.397+0000 7fc8837fe700 1 -- 172.21.15.1:0/1494754587 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc88c053a30 con 0x7fc89c0d3820 2024-03-18T20:29:56.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.596+0000 7fc8a154a700 1 -- 172.21.15.1:0/1494754587 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc89c15dca0 con 0x7fc88806adb0 2024-03-18T20:29:56.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.598+0000 7fc8837fe700 1 -- 172.21.15.1:0/1494754587 <== mgr.14091 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50601 (secure 0 0 0) 0x7fc89000bb40 con 0x7fc88806adb0 2024-03-18T20:29:56.604 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:29:56.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.601+0000 7fc8a154a700 1 -- 172.21.15.1:0/1494754587 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc88806adb0 msgr2=0x7fc88806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:56.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.601+0000 7fc8a154a700 1 --2- 172.21.15.1:0/1494754587 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc88806adb0 0x7fc88806d270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc890014040 tx=0x7fc890008030 comp rx=0 tx=0).stop 2024-03-18T20:29:56.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.601+0000 7fc8a154a700 1 -- 172.21.15.1:0/1494754587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc89c0d3820 msgr2=0x7fc89c0cbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:56.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.601+0000 7fc8a154a700 1 --2- 172.21.15.1:0/1494754587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc89c0d3820 0x7fc89c0cbd30 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 crypto rx=0x7fc88c00c4e0 tx=0x7fc88c00c8a0 comp rx=0 tx=0).stop 2024-03-18T20:29:56.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.601+0000 7fc8a154a700 1 -- 172.21.15.1:0/1494754587 shutdown_connections 2024-03-18T20:29:56.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.601+0000 7fc8a154a700 1 --2- 172.21.15.1:0/1494754587 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc88806adb0 0x7fc88806d270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:56.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.601+0000 7fc8a154a700 1 --2- 172.21.15.1:0/1494754587 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc89c0b3c30 0x7fc89c0cd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:56.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.601+0000 7fc8a154a700 1 --2- 172.21.15.1:0/1494754587 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc89c15d8c0 0x7fc89c15bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:56.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.601+0000 7fc8a154a700 1 --2- 172.21.15.1:0/1494754587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc89c0d3820 0x7fc89c0cbd30 unknown :-1 s=CLOSED pgs=3409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:56.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.601+0000 7fc8a154a700 1 -- 172.21.15.1:0/1494754587 >> 172.21.15.1:0/1494754587 conn(0x7fc89c08afc0 msgr2=0x7fc89c06ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:56.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.601+0000 7fc8a154a700 1 -- 172.21.15.1:0/1494754587 shutdown_connections 2024-03-18T20:29:56.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.601+0000 7fc8a154a700 1 -- 172.21.15.1:0/1494754587 wait complete. 2024-03-18T20:29:56.610 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:29:56.617 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-03-18T20:29:55.079564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":104,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":454308,"kb_used_data":30648,"kb_used_omap":12,"kb_used_meta":423539,"kb_avail":754520412,"statfs":{"total":773094113280,"available":772628901888,"internally_reserved":0,"allocated":31383552,"data_stored":17032614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":433704505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-766079,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-200,"num_read_kb":-56,"num_write":-289,"num_write_kb":-2297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.998364"},"pg_stats":[{"pgid":"1.0","version":"293'425","reported_seq":1530,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-03-18T20:29:52.603160+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:29:52.603160+0000","last_peered":"2024-03-18T20:29:52.603160+0000","last_clean":"2024-03-18T20:29:52.603160+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:29:52.603160+0000","last_undegraded":"2024-03-18T20:29:52.603160+0000","last_fullsized":"2024-03-18T20:29:52.603160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576029,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57488,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314352,"statfs":{"total":96636764160,"available":96577896448,"internally_reserved":0,"allocated":4329472,"data_stored":2528024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56100000000000005,"15min":0.58999999999999997},"min":{"1min":0.38200000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.82899999999999996,"5min":0.88400000000000001,"15min":1.1339999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54500000000000004,"15min":0.59099999999999997},"min":{"1min":0.439,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":0.70699999999999996,"5min":0.92200000000000004,"15min":1.093},"last":0.70999999999999996}]},{"osd":1,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.55400000000000005,"15min":0.59799999999999998},"min":{"1min":0.44700000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.749,"5min":0.82999999999999996,"15min":1.4299999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.56999999999999995,"15min":0.60199999999999998},"min":{"1min":0.39100000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.77400000000000002,"5min":0.89100000000000001,"15min":1.2629999999999999},"last":0.73599999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55100000000000005,"15min":0.59399999999999997},"min":{"1min":0.33600000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.80300000000000005,"5min":0.85499999999999998,"15min":1.216},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57899999999999996,"15min":0.60999999999999999},"min":{"1min":0.42799999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79300000000000004,"5min":1.1279999999999999,"15min":1.2490000000000001},"last":0.61299999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.55800000000000005,"15min":0.59299999999999997},"min":{"1min":0.40799999999999997,"5min":0.374,"15min":0.35199999999999998},"max":{"1min":0.751,"5min":0.84699999999999998,"15min":1.4450000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.58599999999999997,"15min":0.63500000000000001},"min":{"1min":0.42799999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.75800000000000001,"5min":0.94599999999999995,"15min":1.5029999999999999},"last":0.64300000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.35199999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.86099999999999999},"last":0.624},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.498,"15min":0.5},"min":{"1min":0.34100000000000003,"5min":0.34000000000000002,"15min":0.30499999999999999},"max":{"1min":0.74299999999999999,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.505}]},{"osd":5,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53000000000000003,"15min":0.52500000000000002},"min":{"1min":0.373,"5min":0.314,"15min":0.314},"max":{"1min":0.86899999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.41699999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.81599999999999995,"5min":0.877,"15min":1.042},"last":0.76100000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.42299999999999999,"5min":0.39500000000000002,"15min":0.36199999999999999},"max":{"1min":0.85599999999999998,"5min":0.93500000000000005,"15min":1.129},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.57299999999999995},"min":{"1min":0.44,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83799999999999997,"5min":0.91900000000000004,"15min":1.0609999999999999},"last":0.65700000000000003}]}]},{"osd":6,"up_from":13,"seq":55834576029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65100,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306740,"statfs":{"total":96636764160,"available":96570101760,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":62912967},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.52200000000000002,"15min":0.57499999999999996},"min":{"1min":0.39600000000000002,"5min":0.32500000000000001,"15min":0.28799999999999998},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":1.1519999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53700000000000003,"15min":0.58399999999999996},"min":{"1min":0.44,"5min":0.317,"15min":0.317},"max":{"1min":0.73599999999999999,"5min":1.119,"15min":1.119},"last":0.50600000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.53300000000000003,"15min":0.57799999999999996},"min":{"1min":0.42599999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":1.244},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54700000000000004,"15min":0.58799999999999997},"min":{"1min":0.34999999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":1.286},"last":0.56000000000000005}]},{"osd":2,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.56799999999999995,"15min":0.60999999999999999},"min":{"1min":0.42099999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":1.276},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55100000000000005,"15min":0.59199999999999997},"min":{"1min":0.376,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83599999999999997,"5min":1.089,"15min":1.2669999999999999},"last":0.36099999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.56499999999999995,"15min":0.60399999999999998},"min":{"1min":0.39900000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80400000000000005,"5min":1.163,"15min":1.163},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56899999999999995,"15min":0.60599999999999998},"min":{"1min":0.40600000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79200000000000004,"5min":0.82399999999999995,"15min":1.337},"last":0.61199999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.51700000000000002,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.308,"15min":0.30399999999999999},"max":{"1min":0.746,"5min":0.80200000000000005,"15min":0.88300000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52800000000000002,"15min":0.52000000000000002},"min":{"1min":0.32000000000000001,"5min":0.31,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":0.83399999999999996,"15min":1.0309999999999999},"last":0.65600000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.51500000000000001,"15min":0.51000000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.68300000000000005,"5min":1.008,"15min":1.008},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.57399999999999995,"15min":0.55400000000000005},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.77800000000000002,"5min":0.874,"15min":0.94199999999999995},"last":0.67300000000000004}]},{"osd":7,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57999999999999996,"15min":0.56399999999999995},"min":{"1min":0.38100000000000001,"5min":0.35899999999999999,"15min":0.34499999999999997},"max":{"1min":0.76500000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58699999999999997,"15min":0.57599999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.74299999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.71099999999999997}]}]},{"osd":1,"up_from":13,"seq":55834576029,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52820,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319020,"statfs":{"total":96636764160,"available":96582676480,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.373,"15min":0.38900000000000001},"min":{"1min":0.246,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.50800000000000001,"5min":0.60899999999999999,"15min":0.70199999999999996},"last":0.313},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.39000000000000001,"15min":0.41499999999999998},"min":{"1min":0.28999999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.69699999999999995},"last":0.39100000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.39300000000000002,"15min":0.42099999999999999},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.626,"5min":0.626,"15min":0.68100000000000005},"last":0.253},{"interface":"front","average":{"1min":0.438,"5min":0.436,"15min":0.46300000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61599999999999999,"5min":0.70099999999999996,"15min":0.80500000000000005},"last":0.36099999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.441,"15min":0.47599999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65200000000000002,"5min":0.65600000000000003,"15min":0.76100000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.39800000000000002,"15min":0.42699999999999999},"min":{"1min":0.24399999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.58899999999999997,"5min":0.68200000000000005,"15min":0.71699999999999997},"last":0.34100000000000003}]},{"osd":4,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63200000000000001,"15min":0.66100000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.94599999999999995,"5min":1.288,"15min":1.2909999999999999},"last":0.63},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63400000000000001,"15min":0.66600000000000004},"min":{"1min":0.44700000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.88900000000000001,"5min":1.349,"15min":1.746},"last":0.48999999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.623,"15min":0.65900000000000003},"min":{"1min":0.40100000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.92700000000000005,"5min":1.361,"15min":1.6910000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.66500000000000004,"15min":0.69999999999999996},"min":{"1min":0.48199999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.97099999999999997,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.58799999999999997}]},{"osd":6,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63,"15min":0.67100000000000004},"min":{"1min":0.39400000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":1.3120000000000001,"15min":1.637},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.65300000000000002,"15min":0.69299999999999995},"min":{"1min":0.44800000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.91100000000000003,"5min":0.93500000000000005,"15min":1.722},"last":0.51000000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.65400000000000003,"15min":0.69599999999999995},"min":{"1min":0.42899999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.89100000000000001,"5min":1.375,"15min":1.375},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.66100000000000003,"15min":0.69099999999999995},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.98899999999999999,"5min":1.431,"15min":1.431},"last":0.55800000000000005}]}]},{"osd":0,"up_from":13,"seq":55834576030,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49296,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322544,"statfs":{"total":96636764160,"available":96586285056,"internally_reserved":0,"allocated":4329472,"data_stored":2528024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.35899999999999999,"15min":0.38},"min":{"1min":0.25,"5min":0.214,"15min":0.214},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.38900000000000001,"15min":0.41599999999999998},"min":{"1min":0.30599999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":1.1850000000000001},"last":0.32500000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.41199999999999998,"15min":0.42899999999999999},"min":{"1min":0.34899999999999998,"5min":0.27100000000000002,"15min":0.24099999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.41099999999999998,"15min":0.439},"min":{"1min":0.28799999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.70799999999999996},"last":0.34899999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.38700000000000001,"15min":0.41699999999999998},"min":{"1min":0.312,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.57499999999999996,"5min":0.58499999999999996,"15min":0.628},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.45200000000000001,"15min":0.47899999999999998},"min":{"1min":0.309,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.91500000000000004},"last":0.79900000000000004}]},{"osd":4,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.61299999999999999,"15min":0.64000000000000001},"min":{"1min":0.40100000000000002,"5min":0.40000000000000002,"15min":0.36799999999999999},"max":{"1min":0.91200000000000003,"5min":1.506,"15min":1.506},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.443,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.01,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.51200000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.65000000000000002,"15min":0.66400000000000003},"min":{"1min":0.439,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.002,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.64500000000000002,"15min":0.67300000000000004},"min":{"1min":0.53000000000000003,"5min":0.432,"15min":0.432},"max":{"1min":1.006,"5min":1.468,"15min":1.518},"last":0.75900000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.63400000000000001,"15min":0.65600000000000003},"min":{"1min":0.52700000000000002,"5min":0.436,"15min":0.436},"max":{"1min":0.92000000000000004,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.67400000000000004,"15min":0.69699999999999995},"min":{"1min":0.56200000000000006,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0860000000000001,"5min":1.5649999999999999,"15min":1.5649999999999999},"last":0.83399999999999996}]},{"osd":7,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.67100000000000004,"15min":0.69499999999999995},"min":{"1min":0.504,"5min":0.45300000000000001,"15min":0.41899999999999998},"max":{"1min":1.016,"5min":1.55,"15min":1.55},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.65200000000000002,"15min":0.68700000000000006},"min":{"1min":0.57299999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0269999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.57299999999999995}]}]},{"osd":2,"up_from":13,"seq":55834576030,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52812,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319028,"statfs":{"total":96636764160,"available":96582684672,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.33900000000000002,"15min":0.36399999999999999},"min":{"1min":0.28299999999999997,"5min":0.184,"15min":0.184},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.42899999999999999,"15min":0.46600000000000003},"min":{"1min":0.34300000000000003,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.753},"last":0.33300000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.433,"15min":0.46400000000000002},"min":{"1min":0.40500000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.749},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.36399999999999999,"15min":0.39900000000000002},"min":{"1min":0.307,"5min":0.224,"15min":0.224},"max":{"1min":0.54000000000000004,"5min":0.54600000000000004,"15min":0.59999999999999998},"last":0.378}]},{"osd":3,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.38600000000000001,"15min":0.41899999999999998},"min":{"1min":0.32800000000000001,"5min":0.26500000000000001,"15min":0.22800000000000001},"max":{"1min":0.623,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.44700000000000001,"15min":0.49399999999999999},"min":{"1min":0.372,"5min":0.315,"15min":0.315},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.752},"last":0.42899999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.63300000000000001,"15min":0.64900000000000002},"min":{"1min":0.505,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.093,"5min":1.2090000000000001,"15min":1.4079999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.62,"15min":0.65500000000000003},"min":{"1min":0.39900000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98799999999999999,"5min":1.268,"15min":1.2729999999999999},"last":0.57999999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.624,"15min":0.66300000000000003},"min":{"1min":0.53400000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1,"5min":1,"15min":1.028},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.63200000000000001,"15min":0.66400000000000003},"min":{"1min":0.56599999999999995,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.64000000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.626,"15min":0.67000000000000004},"min":{"1min":0.52900000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.628},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.69599999999999995,"15min":0.72099999999999997},"min":{"1min":0.499,"5min":0.439,"15min":0.439},"max":{"1min":1.1299999999999999,"5min":1.3220000000000001,"15min":1.3859999999999999},"last":0.61399999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.63500000000000001,"15min":0.67900000000000005},"min":{"1min":0.497,"5min":0.432,"15min":0.42199999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0269999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.746,"5min":0.67000000000000004,"15min":0.69299999999999995},"min":{"1min":0.50600000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1739999999999999,"5min":1.2330000000000001,"15min":1.371},"last":0.66300000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576029,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57252,"kb_used_data":3992,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314588,"statfs":{"total":96636764160,"available":96578138112,"internally_reserved":0,"allocated":4087808,"data_stored":2309579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.34200000000000003,"15min":0.378},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.66500000000000004},"last":0.372},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.35999999999999999,"15min":0.39900000000000002},"min":{"1min":0.27200000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.61699999999999999},"last":0.35599999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.35499999999999998,"15min":0.39000000000000001},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.214},"max":{"1min":0.55100000000000005,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.41099999999999998,"15min":0.44700000000000001},"min":{"1min":0.32800000000000001,"5min":0.29099999999999998,"15min":0.25800000000000001},"max":{"1min":0.60399999999999998,"5min":0.63500000000000001,"15min":0.72599999999999998},"last":0.40899999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.40799999999999997,"15min":0.44400000000000001},"min":{"1min":0.32600000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.68400000000000005},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.433,"15min":0.47299999999999998},"min":{"1min":0.36299999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.71199999999999997},"last":0.436}]},{"osd":4,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.439,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98499999999999999,"5min":1.526,"15min":1.526},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.622,"15min":0.65600000000000003},"min":{"1min":0.45500000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97899999999999998,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.64800000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.65600000000000003,"15min":0.67700000000000005},"min":{"1min":0.41299999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94399999999999995,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.66100000000000003,"15min":0.70799999999999996},"min":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.99099999999999999,"5min":1.1120000000000001,"15min":1.171},"last":0.69699999999999995}]},{"osd":6,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.66000000000000003,"15min":0.68899999999999995},"min":{"1min":0.42499999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.878,"5min":1.573,"15min":1.573},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.67300000000000004,"15min":0.70999999999999996},"min":{"1min":0.48999999999999999,"5min":0.47399999999999998,"15min":0.45100000000000001},"max":{"1min":1.02,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.60099999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63700000000000001,"15min":0.68999999999999995},"min":{"1min":0.42799999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.032,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.624},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.65300000000000002,"15min":0.68200000000000005},"min":{"1min":0.439,"5min":0.41099999999999998,"15min":0.39200000000000002},"max":{"1min":0.94699999999999995,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.66800000000000004}]}]},{"osd":4,"up_from":13,"seq":55834576030,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48720,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.55600000000000005,"15min":0.57999999999999996},"min":{"1min":0.44,"5min":0.34300000000000003,"15min":0.29499999999999998},"max":{"1min":0.77600000000000002,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56599999999999995,"15min":0.58399999999999996},"min":{"1min":0.46700000000000003,"5min":0.40100000000000002,"15min":0.36299999999999999},"max":{"1min":0.79200000000000004,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.63300000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.58199999999999996,"15min":0.60299999999999998},"min":{"1min":0.45500000000000002,"5min":0.35799999999999998,"15min":0.318},"max":{"1min":0.81000000000000005,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.55500000000000005,"15min":0.57299999999999995},"min":{"1min":0.496,"5min":0.34899999999999998,"15min":0.34300000000000003},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.96499999999999997},"last":0.64800000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.58699999999999997,"15min":0.59199999999999997},"min":{"1min":0.52800000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.82699999999999996,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.58899999999999997,"15min":0.60499999999999998},"min":{"1min":0.40200000000000002,"5min":0.38900000000000001,"15min":0.38500000000000001},"max":{"1min":0.76700000000000002,"5min":0.83099999999999996,"15min":0.84099999999999997},"last":0.67600000000000005}]},{"osd":3,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58099999999999996,"15min":0.59199999999999997},"min":{"1min":0.5,"5min":0.374,"15min":0.33900000000000002},"max":{"1min":0.82799999999999996,"5min":1.026,"15min":1.026},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.60499999999999998,"15min":0.61099999999999999},"min":{"1min":0.503,"5min":0.36699999999999999,"15min":0.34999999999999998},"max":{"1min":0.85499999999999998,"5min":1.028,"15min":1.028},"last":0.69799999999999995}]},{"osd":5,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.499,"15min":0.48199999999999998},"min":{"1min":0.34799999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.67400000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.52400000000000002,"15min":0.505},"min":{"1min":0.40799999999999997,"5min":0.31900000000000001,"15min":0.29999999999999999},"max":{"1min":0.68100000000000005,"5min":0.85499999999999998,"15min":0.90600000000000003},"last":0.54400000000000004}]},{"osd":6,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52700000000000002,"15min":0.50600000000000001},"min":{"1min":0.36799999999999999,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.748,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.56999999999999995,"15min":0.55300000000000005},"min":{"1min":0.435,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.77900000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.57299999999999995}]},{"osd":7,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56599999999999995,"15min":0.54800000000000004},"min":{"1min":0.38700000000000001,"5min":0.36899999999999999,"15min":0.35699999999999998},"max":{"1min":0.70499999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60699999999999998,"15min":0.58399999999999996},"min":{"1min":0.45900000000000002,"5min":0.36599999999999999,"15min":0.33700000000000002},"max":{"1min":0.79200000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.61499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834576028,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70820,"kb_used_data":3608,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94301020,"statfs":{"total":96636764160,"available":96564244480,"internally_reserved":0,"allocated":3694592,"data_stored":1916363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54100000000000004,"15min":0.58399999999999996},"min":{"1min":0.36599999999999999,"5min":0.33500000000000002,"15min":0.31900000000000001},"max":{"1min":0.73599999999999999,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53700000000000003,"15min":0.58599999999999997},"min":{"1min":0.41799999999999998,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":0.80600000000000005,"5min":0.82399999999999995,"15min":1.3100000000000001},"last":0.44400000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55100000000000005,"15min":0.59199999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.31900000000000001},"max":{"1min":0.77600000000000002,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53900000000000003,"15min":0.58399999999999996},"min":{"1min":0.378,"5min":0.372,"15min":0.35199999999999998},"max":{"1min":0.81699999999999995,"5min":0.94299999999999995,"15min":1.018},"last":0.41399999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.54200000000000004,"15min":0.58499999999999996},"min":{"1min":0.35499999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.99299999999999999},"last":0.377},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54900000000000004,"15min":0.59599999999999997},"min":{"1min":0.44900000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80000000000000004,"5min":1.599,"15min":1.599},"last":0.38700000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56299999999999994,"15min":0.59799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76800000000000002,"5min":0.98699999999999999,"15min":1.2430000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58799999999999997,"15min":0.63200000000000001},"min":{"1min":0.42999999999999999,"5min":0.40600000000000003,"15min":0.36699999999999999},"max":{"1min":0.76200000000000001,"5min":0.84399999999999997,"15min":1.161},"last":0.42499999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.48799999999999999,"15min":0.48299999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57799999999999996,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53200000000000003,"15min":0.52000000000000002},"min":{"1min":0.28199999999999997,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.70599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.433}]},{"osd":6,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.50700000000000001,"15min":0.51200000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.65200000000000002,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.54400000000000004,"15min":0.54000000000000004},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.68500000000000005,"5min":1.643,"15min":1.643},"last":0.35299999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56599999999999995,"15min":0.56000000000000005},"min":{"1min":0.40300000000000002,"5min":0.32800000000000001,"15min":0.317},"max":{"1min":0.77600000000000002,"5min":1.028,"15min":1.028},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.58299999999999996,"15min":0.57899999999999996},"min":{"1min":0.39200000000000002,"5min":0.34300000000000003,"15min":0.29799999999999999},"max":{"1min":0.67800000000000005,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.40300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:29:56.620 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:29:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.741+0000 7f9e5e2f9700 1 Processor -- start 2024-03-18T20:29:56.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.741+0000 7f9e5e2f9700 1 -- start start 2024-03-18T20:29:56.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.741+0000 7f9e5e2f9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9e580b32d0 0x7f9e5815b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:56.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.741+0000 7f9e5e2f9700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e580b3ba0 con 0x7f9e5815bb30 2024-03-18T20:29:56.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.741+0000 7f9e5e2f9700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e580b3e80 con 0x7f9e5815a9b0 2024-03-18T20:29:56.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.741+0000 7f9e5e2f9700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e580b4160 con 0x7f9e580b32d0 2024-03-18T20:29:56.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.742+0000 7f9e577fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9e580b32d0 0x7f9e5815b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:56.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.742+0000 7f9e577fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9e580b32d0 0x7f9e5815b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50032/0 (socket says 172.21.15.1:50032) 2024-03-18T20:29:56.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.742+0000 7f9e577fe700 1 -- 172.21.15.1:0/848420246 learned_addr learned my addr 172.21.15.1:0/848420246 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:56.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.742+0000 7f9e577fe700 1 -- 172.21.15.1:0/848420246 >> v1:172.21.15.165:6789/0 conn(0x7f9e5815a9b0 legacy=0x7f9e5815ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:56.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.742+0000 7f9e577fe700 1 -- 172.21.15.1:0/848420246 >> v1:172.21.15.1:6789/0 conn(0x7f9e5815bb30 legacy=0x7f9e5815bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:56.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.742+0000 7f9e577fe700 1 -- 172.21.15.1:0/848420246 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e580b4440 con 0x7f9e580b32d0 2024-03-18T20:29:56.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.742+0000 7f9e577fe700 1 --2- 172.21.15.1:0/848420246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e580b32d0 0x7f9e5815b4a0 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c0098f0 tx=0x7f9e4c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e9e738eef1cd5e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:56.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.742+0000 7f9e56ffd700 1 -- 172.21.15.1:0/848420246 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c00f8a0 con 0x7f9e580b32d0 2024-03-18T20:29:56.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.742+0000 7f9e56ffd700 1 -- 172.21.15.1:0/848420246 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4c00fa00 con 0x7f9e580b32d0 2024-03-18T20:29:56.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.742+0000 7f9e56ffd700 1 -- 172.21.15.1:0/848420246 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c00fd90 con 0x7f9e580b32d0 2024-03-18T20:29:56.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.743+0000 7f9e5e2f9700 1 -- 172.21.15.1:0/848420246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e580b32d0 msgr2=0x7f9e5815b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:56.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.743+0000 7f9e5e2f9700 1 --2- 172.21.15.1:0/848420246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e580b32d0 0x7f9e5815b4a0 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c0098f0 tx=0x7f9e4c009c00 comp rx=0 tx=0).stop 2024-03-18T20:29:56.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.743+0000 7f9e5e2f9700 1 -- 172.21.15.1:0/848420246 shutdown_connections 2024-03-18T20:29:56.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.743+0000 7f9e5e2f9700 1 --2- 172.21.15.1:0/848420246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e580b32d0 0x7f9e5815b4a0 unknown :-1 s=CLOSED pgs=3411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:56.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.743+0000 7f9e5e2f9700 1 -- 172.21.15.1:0/848420246 >> 172.21.15.1:0/848420246 conn(0x7f9e580b5780 msgr2=0x7f9e580b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:56.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.743+0000 7f9e5e2f9700 1 -- 172.21.15.1:0/848420246 shutdown_connections 2024-03-18T20:29:56.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.743+0000 7f9e5e2f9700 1 -- 172.21.15.1:0/848420246 wait complete. 2024-03-18T20:29:56.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.744+0000 7f9e5e2f9700 1 Processor -- start 2024-03-18T20:29:56.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.744+0000 7f9e5e2f9700 1 -- start start 2024-03-18T20:29:56.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.745+0000 7f9e5e2f9700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e580b32d0 0x7f9e580cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:56.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.745+0000 7f9e5e2f9700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e5815a9b0 0x7f9e58159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:56.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.745+0000 7f9e5e2f9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5815bb30 0x7f9e5813d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:56.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.745+0000 7f9e5e2f9700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e580cb710 con 0x7f9e5815bb30 2024-03-18T20:29:56.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.745+0000 7f9e5e2f9700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e580caa90 con 0x7f9e580b32d0 2024-03-18T20:29:56.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.745+0000 7f9e5e2f9700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e580cad70 con 0x7f9e5815a9b0 2024-03-18T20:29:56.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.745+0000 7f9e5c896700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5815bb30 0x7f9e5813d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:56.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.745+0000 7f9e5c896700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5815bb30 0x7f9e5813d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50038/0 (socket says 172.21.15.1:50038) 2024-03-18T20:29:56.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.745+0000 7f9e5c896700 1 -- 172.21.15.1:0/1197421291 learned_addr learned my addr 172.21.15.1:0/1197421291 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:56.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.745+0000 7f9e57fff700 1 --2- 172.21.15.1:0/1197421291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e580b32d0 0x7f9e580cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:56.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.745+0000 7f9e577fe700 1 --2- 172.21.15.1:0/1197421291 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e5815a9b0 0x7f9e58159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:56.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.745+0000 7f9e5c896700 1 -- 172.21.15.1:0/1197421291 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e5815a9b0 msgr2=0x7f9e58159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:56.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.745+0000 7f9e5c896700 1 --2- 172.21.15.1:0/1197421291 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e5815a9b0 0x7f9e58159de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:56.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.745+0000 7f9e5c896700 1 -- 172.21.15.1:0/1197421291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e580b32d0 msgr2=0x7f9e580cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:56.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.745+0000 7f9e5c896700 1 --2- 172.21.15.1:0/1197421291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e580b32d0 0x7f9e580cbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:56.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.745+0000 7f9e5c896700 1 -- 172.21.15.1:0/1197421291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e4c009550 con 0x7f9e5815bb30 2024-03-18T20:29:56.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.745+0000 7f9e577fe700 1 --2- 172.21.15.1:0/1197421291 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e5815a9b0 0x7f9e58159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:29:56.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.746+0000 7f9e5c896700 1 --2- 172.21.15.1:0/1197421291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5815bb30 0x7f9e5813d8c0 secure :-1 s=READY pgs=3412 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000cb50 tx=0x7f9e5000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:56.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.746+0000 7f9e54ff9700 1 -- 172.21.15.1:0/1197421291 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e50003c00 con 0x7f9e5815bb30 2024-03-18T20:29:56.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.746+0000 7f9e54ff9700 1 -- 172.21.15.1:0/1197421291 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e50003d60 con 0x7f9e5815bb30 2024-03-18T20:29:56.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.746+0000 7f9e5e2f9700 1 -- 172.21.15.1:0/1197421291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e580cb050 con 0x7f9e5815bb30 2024-03-18T20:29:56.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.746+0000 7f9e54ff9700 1 -- 172.21.15.1:0/1197421291 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5000aac0 con 0x7f9e5815bb30 2024-03-18T20:29:56.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.746+0000 7f9e5e2f9700 1 -- 172.21.15.1:0/1197421291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e580cc070 con 0x7f9e5815bb30 2024-03-18T20:29:56.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.747+0000 7f9e5e2f9700 1 -- 172.21.15.1:0/1197421291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e580cb850 con 0x7f9e5815bb30 2024-03-18T20:29:56.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.747+0000 7f9e54ff9700 1 -- 172.21.15.1:0/1197421291 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9e5000ac20 con 0x7f9e5815bb30 2024-03-18T20:29:56.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.748+0000 7f9e54ff9700 1 --2- 172.21.15.1:0/1197421291 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e4406adb0 0x7f9e4406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:56.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.748+0000 7f9e54ff9700 1 -- 172.21.15.1:0/1197421291 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f9e50089570 con 0x7f9e5815bb30 2024-03-18T20:29:56.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.748+0000 7f9e57fff700 1 --2- 172.21.15.1:0/1197421291 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e4406adb0 0x7f9e4406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:56.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.748+0000 7f9e57fff700 1 --2- 172.21.15.1:0/1197421291 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e4406adb0 0x7f9e4406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9e48007fd0 tx=0x7f9e48009040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:56.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.753+0000 7f9e54ff9700 1 -- 172.21.15.1:0/1197421291 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9e580cb850 con 0x7f9e5815bb30 2024-03-18T20:29:56.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.955+0000 7f9e5e2f9700 1 -- 172.21.15.1:0/1197421291 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9e580cb850 con 0x7f9e4406adb0 2024-03-18T20:29:56.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.957+0000 7f9e54ff9700 1 -- 172.21.15.1:0/1197421291 <== mgr.14091 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50601 (secure 0 0 0) 0x7f9e580cb850 con 0x7f9e4406adb0 2024-03-18T20:29:56.962 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:29:56.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.959+0000 7f9e5e2f9700 1 -- 172.21.15.1:0/1197421291 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e4406adb0 msgr2=0x7f9e4406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:56.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.959+0000 7f9e5e2f9700 1 --2- 172.21.15.1:0/1197421291 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e4406adb0 0x7f9e4406d270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9e48007fd0 tx=0x7f9e48009040 comp rx=0 tx=0).stop 2024-03-18T20:29:56.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.959+0000 7f9e5e2f9700 1 -- 172.21.15.1:0/1197421291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5815bb30 msgr2=0x7f9e5813d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:56.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.959+0000 7f9e5e2f9700 1 --2- 172.21.15.1:0/1197421291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5815bb30 0x7f9e5813d8c0 secure :-1 s=READY pgs=3412 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000cb50 tx=0x7f9e5000ce60 comp rx=0 tx=0).stop 2024-03-18T20:29:56.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.959+0000 7f9e5e2f9700 1 -- 172.21.15.1:0/1197421291 shutdown_connections 2024-03-18T20:29:56.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.959+0000 7f9e5e2f9700 1 --2- 172.21.15.1:0/1197421291 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9e4406adb0 0x7f9e4406d270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:56.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.959+0000 7f9e5e2f9700 1 --2- 172.21.15.1:0/1197421291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e5815bb30 0x7f9e5813d8c0 unknown :-1 s=CLOSED pgs=3412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:56.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.959+0000 7f9e5e2f9700 1 --2- 172.21.15.1:0/1197421291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9e580b32d0 0x7f9e580cbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:56.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.959+0000 7f9e5e2f9700 1 --2- 172.21.15.1:0/1197421291 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9e5815a9b0 0x7f9e58159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:56.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.959+0000 7f9e5e2f9700 1 -- 172.21.15.1:0/1197421291 >> 172.21.15.1:0/1197421291 conn(0x7f9e580b5780 msgr2=0x7f9e5815c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:56.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.959+0000 7f9e5e2f9700 1 -- 172.21.15.1:0/1197421291 shutdown_connections 2024-03-18T20:29:56.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:56.959+0000 7f9e5e2f9700 1 -- 172.21.15.1:0/1197421291 wait complete. 2024-03-18T20:29:56.966 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:29:56.976 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-03-18T20:29:55.079564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":104,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":454308,"kb_used_data":30648,"kb_used_omap":12,"kb_used_meta":423539,"kb_avail":754520412,"statfs":{"total":773094113280,"available":772628901888,"internally_reserved":0,"allocated":31383552,"data_stored":17032614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":433704505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-766079,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-200,"num_read_kb":-56,"num_write":-289,"num_write_kb":-2297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.998364"},"pg_stats":[{"pgid":"1.0","version":"293'425","reported_seq":1530,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-03-18T20:29:52.603160+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:29:52.603160+0000","last_peered":"2024-03-18T20:29:52.603160+0000","last_clean":"2024-03-18T20:29:52.603160+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:29:52.603160+0000","last_undegraded":"2024-03-18T20:29:52.603160+0000","last_fullsized":"2024-03-18T20:29:52.603160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576029,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57488,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314352,"statfs":{"total":96636764160,"available":96577896448,"internally_reserved":0,"allocated":4329472,"data_stored":2528024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56100000000000005,"15min":0.58999999999999997},"min":{"1min":0.38200000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.82899999999999996,"5min":0.88400000000000001,"15min":1.1339999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54500000000000004,"15min":0.59099999999999997},"min":{"1min":0.439,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":0.70699999999999996,"5min":0.92200000000000004,"15min":1.093},"last":0.70999999999999996}]},{"osd":1,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.55400000000000005,"15min":0.59799999999999998},"min":{"1min":0.44700000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.749,"5min":0.82999999999999996,"15min":1.4299999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.56999999999999995,"15min":0.60199999999999998},"min":{"1min":0.39100000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.77400000000000002,"5min":0.89100000000000001,"15min":1.2629999999999999},"last":0.73599999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55100000000000005,"15min":0.59399999999999997},"min":{"1min":0.33600000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.80300000000000005,"5min":0.85499999999999998,"15min":1.216},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57899999999999996,"15min":0.60999999999999999},"min":{"1min":0.42799999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79300000000000004,"5min":1.1279999999999999,"15min":1.2490000000000001},"last":0.61299999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.55800000000000005,"15min":0.59299999999999997},"min":{"1min":0.40799999999999997,"5min":0.374,"15min":0.35199999999999998},"max":{"1min":0.751,"5min":0.84699999999999998,"15min":1.4450000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.58599999999999997,"15min":0.63500000000000001},"min":{"1min":0.42799999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.75800000000000001,"5min":0.94599999999999995,"15min":1.5029999999999999},"last":0.64300000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.35199999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.86099999999999999},"last":0.624},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.498,"15min":0.5},"min":{"1min":0.34100000000000003,"5min":0.34000000000000002,"15min":0.30499999999999999},"max":{"1min":0.74299999999999999,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.505}]},{"osd":5,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53000000000000003,"15min":0.52500000000000002},"min":{"1min":0.373,"5min":0.314,"15min":0.314},"max":{"1min":0.86899999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.41699999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.81599999999999995,"5min":0.877,"15min":1.042},"last":0.76100000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.42299999999999999,"5min":0.39500000000000002,"15min":0.36199999999999999},"max":{"1min":0.85599999999999998,"5min":0.93500000000000005,"15min":1.129},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.57299999999999995},"min":{"1min":0.44,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83799999999999997,"5min":0.91900000000000004,"15min":1.0609999999999999},"last":0.65700000000000003}]}]},{"osd":6,"up_from":13,"seq":55834576029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65100,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306740,"statfs":{"total":96636764160,"available":96570101760,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":62912967},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.52200000000000002,"15min":0.57499999999999996},"min":{"1min":0.39600000000000002,"5min":0.32500000000000001,"15min":0.28799999999999998},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":1.1519999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53700000000000003,"15min":0.58399999999999996},"min":{"1min":0.44,"5min":0.317,"15min":0.317},"max":{"1min":0.73599999999999999,"5min":1.119,"15min":1.119},"last":0.50600000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.53300000000000003,"15min":0.57799999999999996},"min":{"1min":0.42599999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":1.244},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54700000000000004,"15min":0.58799999999999997},"min":{"1min":0.34999999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":1.286},"last":0.56000000000000005}]},{"osd":2,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.56799999999999995,"15min":0.60999999999999999},"min":{"1min":0.42099999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":1.276},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55100000000000005,"15min":0.59199999999999997},"min":{"1min":0.376,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83599999999999997,"5min":1.089,"15min":1.2669999999999999},"last":0.36099999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.56499999999999995,"15min":0.60399999999999998},"min":{"1min":0.39900000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80400000000000005,"5min":1.163,"15min":1.163},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56899999999999995,"15min":0.60599999999999998},"min":{"1min":0.40600000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79200000000000004,"5min":0.82399999999999995,"15min":1.337},"last":0.61199999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.51700000000000002,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.308,"15min":0.30399999999999999},"max":{"1min":0.746,"5min":0.80200000000000005,"15min":0.88300000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52800000000000002,"15min":0.52000000000000002},"min":{"1min":0.32000000000000001,"5min":0.31,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":0.83399999999999996,"15min":1.0309999999999999},"last":0.65600000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.51500000000000001,"15min":0.51000000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.68300000000000005,"5min":1.008,"15min":1.008},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.57399999999999995,"15min":0.55400000000000005},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.77800000000000002,"5min":0.874,"15min":0.94199999999999995},"last":0.67300000000000004}]},{"osd":7,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57999999999999996,"15min":0.56399999999999995},"min":{"1min":0.38100000000000001,"5min":0.35899999999999999,"15min":0.34499999999999997},"max":{"1min":0.76500000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58699999999999997,"15min":0.57599999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.74299999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.71099999999999997}]}]},{"osd":1,"up_from":13,"seq":55834576029,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52820,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319020,"statfs":{"total":96636764160,"available":96582676480,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.373,"15min":0.38900000000000001},"min":{"1min":0.246,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.50800000000000001,"5min":0.60899999999999999,"15min":0.70199999999999996},"last":0.313},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.39000000000000001,"15min":0.41499999999999998},"min":{"1min":0.28999999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.69699999999999995},"last":0.39100000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.39300000000000002,"15min":0.42099999999999999},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.626,"5min":0.626,"15min":0.68100000000000005},"last":0.253},{"interface":"front","average":{"1min":0.438,"5min":0.436,"15min":0.46300000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61599999999999999,"5min":0.70099999999999996,"15min":0.80500000000000005},"last":0.36099999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.441,"15min":0.47599999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65200000000000002,"5min":0.65600000000000003,"15min":0.76100000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.39800000000000002,"15min":0.42699999999999999},"min":{"1min":0.24399999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.58899999999999997,"5min":0.68200000000000005,"15min":0.71699999999999997},"last":0.34100000000000003}]},{"osd":4,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63200000000000001,"15min":0.66100000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.94599999999999995,"5min":1.288,"15min":1.2909999999999999},"last":0.63},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63400000000000001,"15min":0.66600000000000004},"min":{"1min":0.44700000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.88900000000000001,"5min":1.349,"15min":1.746},"last":0.48999999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.623,"15min":0.65900000000000003},"min":{"1min":0.40100000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.92700000000000005,"5min":1.361,"15min":1.6910000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.66500000000000004,"15min":0.69999999999999996},"min":{"1min":0.48199999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.97099999999999997,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.58799999999999997}]},{"osd":6,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63,"15min":0.67100000000000004},"min":{"1min":0.39400000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":1.3120000000000001,"15min":1.637},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.65300000000000002,"15min":0.69299999999999995},"min":{"1min":0.44800000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.91100000000000003,"5min":0.93500000000000005,"15min":1.722},"last":0.51000000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.65400000000000003,"15min":0.69599999999999995},"min":{"1min":0.42899999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.89100000000000001,"5min":1.375,"15min":1.375},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.66100000000000003,"15min":0.69099999999999995},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.98899999999999999,"5min":1.431,"15min":1.431},"last":0.55800000000000005}]}]},{"osd":0,"up_from":13,"seq":55834576030,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49296,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322544,"statfs":{"total":96636764160,"available":96586285056,"internally_reserved":0,"allocated":4329472,"data_stored":2528024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.35899999999999999,"15min":0.38},"min":{"1min":0.25,"5min":0.214,"15min":0.214},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.38900000000000001,"15min":0.41599999999999998},"min":{"1min":0.30599999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":1.1850000000000001},"last":0.32500000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.41199999999999998,"15min":0.42899999999999999},"min":{"1min":0.34899999999999998,"5min":0.27100000000000002,"15min":0.24099999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.41099999999999998,"15min":0.439},"min":{"1min":0.28799999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.70799999999999996},"last":0.34899999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.38700000000000001,"15min":0.41699999999999998},"min":{"1min":0.312,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.57499999999999996,"5min":0.58499999999999996,"15min":0.628},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.45200000000000001,"15min":0.47899999999999998},"min":{"1min":0.309,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.91500000000000004},"last":0.79900000000000004}]},{"osd":4,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.61299999999999999,"15min":0.64000000000000001},"min":{"1min":0.40100000000000002,"5min":0.40000000000000002,"15min":0.36799999999999999},"max":{"1min":0.91200000000000003,"5min":1.506,"15min":1.506},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.443,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.01,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.51200000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.65000000000000002,"15min":0.66400000000000003},"min":{"1min":0.439,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.002,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.64500000000000002,"15min":0.67300000000000004},"min":{"1min":0.53000000000000003,"5min":0.432,"15min":0.432},"max":{"1min":1.006,"5min":1.468,"15min":1.518},"last":0.75900000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.63400000000000001,"15min":0.65600000000000003},"min":{"1min":0.52700000000000002,"5min":0.436,"15min":0.436},"max":{"1min":0.92000000000000004,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.67400000000000004,"15min":0.69699999999999995},"min":{"1min":0.56200000000000006,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0860000000000001,"5min":1.5649999999999999,"15min":1.5649999999999999},"last":0.83399999999999996}]},{"osd":7,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.67100000000000004,"15min":0.69499999999999995},"min":{"1min":0.504,"5min":0.45300000000000001,"15min":0.41899999999999998},"max":{"1min":1.016,"5min":1.55,"15min":1.55},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.65200000000000002,"15min":0.68700000000000006},"min":{"1min":0.57299999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0269999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.57299999999999995}]}]},{"osd":2,"up_from":13,"seq":55834576030,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52812,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319028,"statfs":{"total":96636764160,"available":96582684672,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.33900000000000002,"15min":0.36399999999999999},"min":{"1min":0.28299999999999997,"5min":0.184,"15min":0.184},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.42899999999999999,"15min":0.46600000000000003},"min":{"1min":0.34300000000000003,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.753},"last":0.33300000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.433,"15min":0.46400000000000002},"min":{"1min":0.40500000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.749},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.36399999999999999,"15min":0.39900000000000002},"min":{"1min":0.307,"5min":0.224,"15min":0.224},"max":{"1min":0.54000000000000004,"5min":0.54600000000000004,"15min":0.59999999999999998},"last":0.378}]},{"osd":3,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.38600000000000001,"15min":0.41899999999999998},"min":{"1min":0.32800000000000001,"5min":0.26500000000000001,"15min":0.22800000000000001},"max":{"1min":0.623,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.44700000000000001,"15min":0.49399999999999999},"min":{"1min":0.372,"5min":0.315,"15min":0.315},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.752},"last":0.42899999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.63300000000000001,"15min":0.64900000000000002},"min":{"1min":0.505,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.093,"5min":1.2090000000000001,"15min":1.4079999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.62,"15min":0.65500000000000003},"min":{"1min":0.39900000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98799999999999999,"5min":1.268,"15min":1.2729999999999999},"last":0.57999999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.624,"15min":0.66300000000000003},"min":{"1min":0.53400000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1,"5min":1,"15min":1.028},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.63200000000000001,"15min":0.66400000000000003},"min":{"1min":0.56599999999999995,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.64000000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.626,"15min":0.67000000000000004},"min":{"1min":0.52900000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.628},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.69599999999999995,"15min":0.72099999999999997},"min":{"1min":0.499,"5min":0.439,"15min":0.439},"max":{"1min":1.1299999999999999,"5min":1.3220000000000001,"15min":1.3859999999999999},"last":0.61399999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.63500000000000001,"15min":0.67900000000000005},"min":{"1min":0.497,"5min":0.432,"15min":0.42199999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0269999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.746,"5min":0.67000000000000004,"15min":0.69299999999999995},"min":{"1min":0.50600000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1739999999999999,"5min":1.2330000000000001,"15min":1.371},"last":0.66300000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576029,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57252,"kb_used_data":3992,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314588,"statfs":{"total":96636764160,"available":96578138112,"internally_reserved":0,"allocated":4087808,"data_stored":2309579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.34200000000000003,"15min":0.378},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.66500000000000004},"last":0.372},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.35999999999999999,"15min":0.39900000000000002},"min":{"1min":0.27200000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.61699999999999999},"last":0.35599999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.35499999999999998,"15min":0.39000000000000001},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.214},"max":{"1min":0.55100000000000005,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.41099999999999998,"15min":0.44700000000000001},"min":{"1min":0.32800000000000001,"5min":0.29099999999999998,"15min":0.25800000000000001},"max":{"1min":0.60399999999999998,"5min":0.63500000000000001,"15min":0.72599999999999998},"last":0.40899999999999997}]},{"osd":2,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.40799999999999997,"15min":0.44400000000000001},"min":{"1min":0.32600000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.68400000000000005},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.433,"15min":0.47299999999999998},"min":{"1min":0.36299999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.71199999999999997},"last":0.436}]},{"osd":4,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.439,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98499999999999999,"5min":1.526,"15min":1.526},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.622,"15min":0.65600000000000003},"min":{"1min":0.45500000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97899999999999998,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.64800000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.65600000000000003,"15min":0.67700000000000005},"min":{"1min":0.41299999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94399999999999995,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.66100000000000003,"15min":0.70799999999999996},"min":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.99099999999999999,"5min":1.1120000000000001,"15min":1.171},"last":0.69699999999999995}]},{"osd":6,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.66000000000000003,"15min":0.68899999999999995},"min":{"1min":0.42499999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.878,"5min":1.573,"15min":1.573},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.67300000000000004,"15min":0.70999999999999996},"min":{"1min":0.48999999999999999,"5min":0.47399999999999998,"15min":0.45100000000000001},"max":{"1min":1.02,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.60099999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63700000000000001,"15min":0.68999999999999995},"min":{"1min":0.42799999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.032,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.624},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.65300000000000002,"15min":0.68200000000000005},"min":{"1min":0.439,"5min":0.41099999999999998,"15min":0.39200000000000002},"max":{"1min":0.94699999999999995,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.66800000000000004}]}]},{"osd":4,"up_from":13,"seq":55834576030,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48720,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.55600000000000005,"15min":0.57999999999999996},"min":{"1min":0.44,"5min":0.34300000000000003,"15min":0.29499999999999998},"max":{"1min":0.77600000000000002,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56599999999999995,"15min":0.58399999999999996},"min":{"1min":0.46700000000000003,"5min":0.40100000000000002,"15min":0.36299999999999999},"max":{"1min":0.79200000000000004,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.63300000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.58199999999999996,"15min":0.60299999999999998},"min":{"1min":0.45500000000000002,"5min":0.35799999999999998,"15min":0.318},"max":{"1min":0.81000000000000005,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.55500000000000005,"15min":0.57299999999999995},"min":{"1min":0.496,"5min":0.34899999999999998,"15min":0.34300000000000003},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.96499999999999997},"last":0.64800000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.58699999999999997,"15min":0.59199999999999997},"min":{"1min":0.52800000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.82699999999999996,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.58899999999999997,"15min":0.60499999999999998},"min":{"1min":0.40200000000000002,"5min":0.38900000000000001,"15min":0.38500000000000001},"max":{"1min":0.76700000000000002,"5min":0.83099999999999996,"15min":0.84099999999999997},"last":0.67600000000000005}]},{"osd":3,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58099999999999996,"15min":0.59199999999999997},"min":{"1min":0.5,"5min":0.374,"15min":0.33900000000000002},"max":{"1min":0.82799999999999996,"5min":1.026,"15min":1.026},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.60499999999999998,"15min":0.61099999999999999},"min":{"1min":0.503,"5min":0.36699999999999999,"15min":0.34999999999999998},"max":{"1min":0.85499999999999998,"5min":1.028,"15min":1.028},"last":0.69799999999999995}]},{"osd":5,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.499,"15min":0.48199999999999998},"min":{"1min":0.34799999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.67400000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.52400000000000002,"15min":0.505},"min":{"1min":0.40799999999999997,"5min":0.31900000000000001,"15min":0.29999999999999999},"max":{"1min":0.68100000000000005,"5min":0.85499999999999998,"15min":0.90600000000000003},"last":0.54400000000000004}]},{"osd":6,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52700000000000002,"15min":0.50600000000000001},"min":{"1min":0.36799999999999999,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.748,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.56999999999999995,"15min":0.55300000000000005},"min":{"1min":0.435,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.77900000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.57299999999999995}]},{"osd":7,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56599999999999995,"15min":0.54800000000000004},"min":{"1min":0.38700000000000001,"5min":0.36899999999999999,"15min":0.35699999999999998},"max":{"1min":0.70499999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60699999999999998,"15min":0.58399999999999996},"min":{"1min":0.45900000000000002,"5min":0.36599999999999999,"15min":0.33700000000000002},"max":{"1min":0.79200000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.61499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834576028,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70820,"kb_used_data":3608,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94301020,"statfs":{"total":96636764160,"available":96564244480,"internally_reserved":0,"allocated":3694592,"data_stored":1916363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54100000000000004,"15min":0.58399999999999996},"min":{"1min":0.36599999999999999,"5min":0.33500000000000002,"15min":0.31900000000000001},"max":{"1min":0.73599999999999999,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53700000000000003,"15min":0.58599999999999997},"min":{"1min":0.41799999999999998,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":0.80600000000000005,"5min":0.82399999999999995,"15min":1.3100000000000001},"last":0.44400000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55100000000000005,"15min":0.59199999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.31900000000000001},"max":{"1min":0.77600000000000002,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53900000000000003,"15min":0.58399999999999996},"min":{"1min":0.378,"5min":0.372,"15min":0.35199999999999998},"max":{"1min":0.81699999999999995,"5min":0.94299999999999995,"15min":1.018},"last":0.41399999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.54200000000000004,"15min":0.58499999999999996},"min":{"1min":0.35499999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.99299999999999999},"last":0.377},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54900000000000004,"15min":0.59599999999999997},"min":{"1min":0.44900000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80000000000000004,"5min":1.599,"15min":1.599},"last":0.38700000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56299999999999994,"15min":0.59799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76800000000000002,"5min":0.98699999999999999,"15min":1.2430000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58799999999999997,"15min":0.63200000000000001},"min":{"1min":0.42999999999999999,"5min":0.40600000000000003,"15min":0.36699999999999999},"max":{"1min":0.76200000000000001,"5min":0.84399999999999997,"15min":1.161},"last":0.42499999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.48799999999999999,"15min":0.48299999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57799999999999996,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53200000000000003,"15min":0.52000000000000002},"min":{"1min":0.28199999999999997,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.70599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.433}]},{"osd":6,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.50700000000000001,"15min":0.51200000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.65200000000000002,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.54400000000000004,"15min":0.54000000000000004},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.68500000000000005,"5min":1.643,"15min":1.643},"last":0.35299999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56599999999999995,"15min":0.56000000000000005},"min":{"1min":0.40300000000000002,"5min":0.32800000000000001,"15min":0.317},"max":{"1min":0.77600000000000002,"5min":1.028,"15min":1.028},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.58299999999999996,"15min":0.57899999999999996},"min":{"1min":0.39200000000000002,"5min":0.34300000000000003,"15min":0.29799999999999999},"max":{"1min":0.67800000000000005,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.40300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:29:56.978 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-18T20:29:56.978 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:29:57.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.099+0000 7f9ba51a0700 1 Processor -- start 2024-03-18T20:29:57.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.099+0000 7f9ba51a0700 1 -- start start 2024-03-18T20:29:57.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.099+0000 7f9ba51a0700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9ba0070ee0 0x7f9ba00712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:57.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.099+0000 7f9ba51a0700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9ba015d990 0x7f9ba015de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:57.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.099+0000 7f9ba51a0700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba0071800 con 0x7f9ba0160f80 2024-03-18T20:29:57.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.099+0000 7f9ba51a0700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ba00adda0 con 0x7f9ba015d990 2024-03-18T20:29:57.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.100+0000 7f9ba51a0700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ba015e9d0 con 0x7f9ba0070ee0 2024-03-18T20:29:57.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.100+0000 7f9b9f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9ba015d990 0x7f9ba015de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:57.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.100+0000 7f9b9f7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9ba015d990 0x7f9ba015de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50050/0 (socket says 172.21.15.1:50050) 2024-03-18T20:29:57.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.100+0000 7f9b9f7fe700 1 -- 172.21.15.1:0/165121805 learned_addr learned my addr 172.21.15.1:0/165121805 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:57.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.100+0000 7f9b9ffff700 1 --2- 172.21.15.1:0/165121805 >> v2:172.21.15.165:3300/0 conn(0x7f9ba0070ee0 0x7f9ba00712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:57.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.100+0000 7f9b9f7fe700 1 -- 172.21.15.1:0/165121805 >> v2:172.21.15.165:3300/0 conn(0x7f9ba0070ee0 msgr2=0x7f9ba00712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:57.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.100+0000 7f9b9f7fe700 1 --2- 172.21.15.1:0/165121805 >> v2:172.21.15.165:3300/0 conn(0x7f9ba0070ee0 0x7f9ba00712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.100+0000 7f9b9f7fe700 1 -- 172.21.15.1:0/165121805 >> v1:172.21.15.1:6789/0 conn(0x7f9ba0160f80 legacy=0x7f9ba0161410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:57.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.100+0000 7f9b9f7fe700 1 -- 172.21.15.1:0/165121805 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba015e6f0 con 0x7f9ba015d990 2024-03-18T20:29:57.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.100+0000 7f9b9ffff700 1 --2- 172.21.15.1:0/165121805 >> v2:172.21.15.165:3300/0 conn(0x7f9ba0070ee0 0x7f9ba00712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:29:57.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.100+0000 7f9b9f7fe700 1 --2- 172.21.15.1:0/165121805 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ba015d990 0x7f9ba015de40 secure :-1 s=READY pgs=3414 cs=0 l=1 rev1=1 crypto rx=0x7f9ba00b2370 tx=0x7f9b90009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f105f5e3baff6e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:57.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.100+0000 7f9b9effd700 1 -- 172.21.15.1:0/165121805 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9000b860 con 0x7f9ba015d990 2024-03-18T20:29:57.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.100+0000 7f9b9effd700 1 -- 172.21.15.1:0/165121805 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9000b9c0 con 0x7f9ba015d990 2024-03-18T20:29:57.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.100+0000 7f9b9effd700 1 -- 172.21.15.1:0/165121805 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9000bd50 con 0x7f9ba015d990 2024-03-18T20:29:57.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.101+0000 7f9ba51a0700 1 -- 172.21.15.1:0/165121805 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ba015d990 msgr2=0x7f9ba015de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:57.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.101+0000 7f9ba51a0700 1 --2- 172.21.15.1:0/165121805 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ba015d990 0x7f9ba015de40 secure :-1 s=READY pgs=3414 cs=0 l=1 rev1=1 crypto rx=0x7f9ba00b2370 tx=0x7f9b90009c00 comp rx=0 tx=0).stop 2024-03-18T20:29:57.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.101+0000 7f9ba51a0700 1 -- 172.21.15.1:0/165121805 shutdown_connections 2024-03-18T20:29:57.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.101+0000 7f9ba51a0700 1 --2- 172.21.15.1:0/165121805 >> v2:172.21.15.165:3300/0 conn(0x7f9ba0070ee0 0x7f9ba00712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.101+0000 7f9ba51a0700 1 --2- 172.21.15.1:0/165121805 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ba015d990 0x7f9ba015de40 unknown :-1 s=CLOSED pgs=3414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.101+0000 7f9ba51a0700 1 -- 172.21.15.1:0/165121805 >> 172.21.15.1:0/165121805 conn(0x7f9ba00b13e0 msgr2=0x7f9ba00b17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:57.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.101+0000 7f9ba51a0700 1 -- 172.21.15.1:0/165121805 shutdown_connections 2024-03-18T20:29:57.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.101+0000 7f9ba51a0700 1 -- 172.21.15.1:0/165121805 wait complete. 2024-03-18T20:29:57.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.102+0000 7f9ba51a0700 1 Processor -- start 2024-03-18T20:29:57.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.102+0000 7f9ba51a0700 1 -- start start 2024-03-18T20:29:57.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.103+0000 7f9ba51a0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ba0070ee0 0x7f9ba013df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:57.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.103+0000 7f9ba51a0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ba015d990 0x7f9ba01568a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:57.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.103+0000 7f9ba51a0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ba0160f80 0x7f9ba0160100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:57.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.103+0000 7f9ba51a0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba01e73d0 con 0x7f9ba0070ee0 2024-03-18T20:29:57.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.103+0000 7f9ba51a0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba01e6da0 con 0x7f9ba015d990 2024-03-18T20:29:57.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.103+0000 7f9ba51a0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba01572f0 con 0x7f9ba0160f80 2024-03-18T20:29:57.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.103+0000 7f9b9ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ba0070ee0 0x7f9ba013df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:57.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.103+0000 7f9b9ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ba0070ee0 0x7f9ba013df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50056/0 (socket says 172.21.15.1:50056) 2024-03-18T20:29:57.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.103+0000 7f9b9ffff700 1 -- 172.21.15.1:0/2485153558 learned_addr learned my addr 172.21.15.1:0/2485153558 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:57.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.103+0000 7f9b9f7fe700 1 --2- 172.21.15.1:0/2485153558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ba015d990 0x7f9ba01568a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:57.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.103+0000 7f9ba499f700 1 --2- 172.21.15.1:0/2485153558 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ba0160f80 0x7f9ba0160100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:57.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.103+0000 7f9b9ffff700 1 -- 172.21.15.1:0/2485153558 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ba0160f80 msgr2=0x7f9ba0160100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:57.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.103+0000 7f9b9ffff700 1 --2- 172.21.15.1:0/2485153558 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ba0160f80 0x7f9ba0160100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.103+0000 7f9b9ffff700 1 -- 172.21.15.1:0/2485153558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ba015d990 msgr2=0x7f9ba01568a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:57.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.103+0000 7f9b9ffff700 1 --2- 172.21.15.1:0/2485153558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ba015d990 0x7f9ba01568a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.103+0000 7f9b9ffff700 1 -- 172.21.15.1:0/2485153558 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b90009550 con 0x7f9ba0070ee0 2024-03-18T20:29:57.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.104+0000 7f9b9ffff700 1 --2- 172.21.15.1:0/2485153558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ba0070ee0 0x7f9ba013df90 secure :-1 s=READY pgs=3415 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400c7f0 tx=0x7f9b9400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:57.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.104+0000 7f9b9cff9700 1 -- 172.21.15.1:0/2485153558 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94011800 con 0x7f9ba0070ee0 2024-03-18T20:29:57.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.104+0000 7f9b9cff9700 1 -- 172.21.15.1:0/2485153558 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b94011960 con 0x7f9ba0070ee0 2024-03-18T20:29:57.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.104+0000 7f9ba51a0700 1 -- 172.21.15.1:0/2485153558 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba01575d0 con 0x7f9ba0070ee0 2024-03-18T20:29:57.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.104+0000 7f9b9cff9700 1 -- 172.21.15.1:0/2485153558 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94011d10 con 0x7f9ba0070ee0 2024-03-18T20:29:57.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.104+0000 7f9ba51a0700 1 -- 172.21.15.1:0/2485153558 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba013e4d0 con 0x7f9ba0070ee0 2024-03-18T20:29:57.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.105+0000 7f9ba51a0700 1 -- 172.21.15.1:0/2485153558 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba0067b90 con 0x7f9ba0070ee0 2024-03-18T20:29:57.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.105+0000 7f9b9cff9700 1 -- 172.21.15.1:0/2485153558 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9b94018360 con 0x7f9ba0070ee0 2024-03-18T20:29:57.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.106+0000 7f9b9cff9700 1 --2- 172.21.15.1:0/2485153558 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9b8c06ade0 0x7f9b8c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:57.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.106+0000 7f9b9cff9700 1 -- 172.21.15.1:0/2485153558 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f9b94089740 con 0x7f9ba0070ee0 2024-03-18T20:29:57.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.106+0000 7f9b9f7fe700 1 --2- 172.21.15.1:0/2485153558 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9b8c06ade0 0x7f9b8c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:57.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.106+0000 7f9b9f7fe700 1 --2- 172.21.15.1:0/2485153558 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9b8c06ade0 0x7f9b8c06d2a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9b90009870 tx=0x7f9b90008030 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:57.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.111+0000 7f9b9cff9700 1 -- 172.21.15.1:0/2485153558 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9b940544e0 con 0x7f9ba0070ee0 2024-03-18T20:29:57.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.311+0000 7f9ba51a0700 1 -- 172.21.15.1:0/2485153558 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9ba00b2e40 con 0x7f9b8c06ade0 2024-03-18T20:29:57.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.314+0000 7f9b9cff9700 1 -- 172.21.15.1:0/2485153558 <== mgr.14091 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50614 (secure 0 0 0) 0x7f9b9000bbb0 con 0x7f9b8c06ade0 2024-03-18T20:29:57.318 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:29:57.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.316+0000 7f9ba51a0700 1 -- 172.21.15.1:0/2485153558 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9b8c06ade0 msgr2=0x7f9b8c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:57.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.316+0000 7f9ba51a0700 1 --2- 172.21.15.1:0/2485153558 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9b8c06ade0 0x7f9b8c06d2a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9b90009870 tx=0x7f9b90008030 comp rx=0 tx=0).stop 2024-03-18T20:29:57.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.316+0000 7f9ba51a0700 1 -- 172.21.15.1:0/2485153558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ba0070ee0 msgr2=0x7f9ba013df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:57.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.316+0000 7f9ba51a0700 1 --2- 172.21.15.1:0/2485153558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ba0070ee0 0x7f9ba013df90 secure :-1 s=READY pgs=3415 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400c7f0 tx=0x7f9b9400cbb0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.316+0000 7f9ba51a0700 1 -- 172.21.15.1:0/2485153558 shutdown_connections 2024-03-18T20:29:57.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.316+0000 7f9ba51a0700 1 --2- 172.21.15.1:0/2485153558 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f9b8c06ade0 0x7f9b8c06d2a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.316+0000 7f9ba51a0700 1 --2- 172.21.15.1:0/2485153558 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f9ba0160f80 0x7f9ba0160100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.316+0000 7f9ba51a0700 1 --2- 172.21.15.1:0/2485153558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ba0070ee0 0x7f9ba013df90 unknown :-1 s=CLOSED pgs=3415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.316+0000 7f9ba51a0700 1 --2- 172.21.15.1:0/2485153558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ba015d990 0x7f9ba01568a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.316+0000 7f9ba51a0700 1 -- 172.21.15.1:0/2485153558 >> 172.21.15.1:0/2485153558 conn(0x7f9ba00b13e0 msgr2=0x7f9ba00afe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:57.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.316+0000 7f9ba51a0700 1 -- 172.21.15.1:0/2485153558 shutdown_connections 2024-03-18T20:29:57.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.316+0000 7f9ba51a0700 1 -- 172.21.15.1:0/2485153558 wait complete. 2024-03-18T20:29:57.322 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:29:57.331 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-03-18T20:29:57.079806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":74,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":454004,"kb_used_data":30344,"kb_used_omap":12,"kb_used_meta":423539,"kb_avail":754520716,"statfs":{"total":773094113280,"available":772629213184,"internally_reserved":0,"allocated":31072256,"data_stored":16681984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":433704505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-766079,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-200,"num_read_kb":-56,"num_write":-289,"num_write_kb":-2297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.998606"},"pg_stats":[{"pgid":"1.0","version":"293'425","reported_seq":1530,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-03-18T20:29:52.603160+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:29:52.603160+0000","last_peered":"2024-03-18T20:29:52.603160+0000","last_clean":"2024-03-18T20:29:52.603160+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:29:52.603160+0000","last_undegraded":"2024-03-18T20:29:52.603160+0000","last_fullsized":"2024-03-18T20:29:52.603160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576029,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57488,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314352,"statfs":{"total":96636764160,"available":96577896448,"internally_reserved":0,"allocated":4329472,"data_stored":2528024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56100000000000005,"15min":0.58999999999999997},"min":{"1min":0.38200000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.82899999999999996,"5min":0.88400000000000001,"15min":1.1339999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54500000000000004,"15min":0.59099999999999997},"min":{"1min":0.439,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":0.70699999999999996,"5min":0.92200000000000004,"15min":1.093},"last":0.70999999999999996}]},{"osd":1,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.55400000000000005,"15min":0.59799999999999998},"min":{"1min":0.44700000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.749,"5min":0.82999999999999996,"15min":1.4299999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.56999999999999995,"15min":0.60199999999999998},"min":{"1min":0.39100000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.77400000000000002,"5min":0.89100000000000001,"15min":1.2629999999999999},"last":0.73599999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55100000000000005,"15min":0.59399999999999997},"min":{"1min":0.33600000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.80300000000000005,"5min":0.85499999999999998,"15min":1.216},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57899999999999996,"15min":0.60999999999999999},"min":{"1min":0.42799999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79300000000000004,"5min":1.1279999999999999,"15min":1.2490000000000001},"last":0.61299999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.55800000000000005,"15min":0.59299999999999997},"min":{"1min":0.40799999999999997,"5min":0.374,"15min":0.35199999999999998},"max":{"1min":0.751,"5min":0.84699999999999998,"15min":1.4450000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.58599999999999997,"15min":0.63500000000000001},"min":{"1min":0.42799999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.75800000000000001,"5min":0.94599999999999995,"15min":1.5029999999999999},"last":0.64300000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.35199999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.86099999999999999},"last":0.624},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.498,"15min":0.5},"min":{"1min":0.34100000000000003,"5min":0.34000000000000002,"15min":0.30499999999999999},"max":{"1min":0.74299999999999999,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.505}]},{"osd":5,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53000000000000003,"15min":0.52500000000000002},"min":{"1min":0.373,"5min":0.314,"15min":0.314},"max":{"1min":0.86899999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.41699999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.81599999999999995,"5min":0.877,"15min":1.042},"last":0.76100000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.42299999999999999,"5min":0.39500000000000002,"15min":0.36199999999999999},"max":{"1min":0.85599999999999998,"5min":0.93500000000000005,"15min":1.129},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.57299999999999995},"min":{"1min":0.44,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83799999999999997,"5min":0.91900000000000004,"15min":1.0609999999999999},"last":0.65700000000000003}]}]},{"osd":6,"up_from":13,"seq":55834576029,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65100,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306740,"statfs":{"total":96636764160,"available":96570101760,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":62912967},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.52200000000000002,"15min":0.57499999999999996},"min":{"1min":0.39600000000000002,"5min":0.32500000000000001,"15min":0.28799999999999998},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":1.1519999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53700000000000003,"15min":0.58399999999999996},"min":{"1min":0.44,"5min":0.317,"15min":0.317},"max":{"1min":0.73599999999999999,"5min":1.119,"15min":1.119},"last":0.50600000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.53300000000000003,"15min":0.57799999999999996},"min":{"1min":0.42599999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":1.244},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54700000000000004,"15min":0.58799999999999997},"min":{"1min":0.34999999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":1.286},"last":0.56000000000000005}]},{"osd":2,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.56799999999999995,"15min":0.60999999999999999},"min":{"1min":0.42099999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":1.276},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55100000000000005,"15min":0.59199999999999997},"min":{"1min":0.376,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83599999999999997,"5min":1.089,"15min":1.2669999999999999},"last":0.36099999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.56499999999999995,"15min":0.60399999999999998},"min":{"1min":0.39900000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80400000000000005,"5min":1.163,"15min":1.163},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56899999999999995,"15min":0.60599999999999998},"min":{"1min":0.40600000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79200000000000004,"5min":0.82399999999999995,"15min":1.337},"last":0.61199999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.51700000000000002,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.308,"15min":0.30399999999999999},"max":{"1min":0.746,"5min":0.80200000000000005,"15min":0.88300000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52800000000000002,"15min":0.52000000000000002},"min":{"1min":0.32000000000000001,"5min":0.31,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":0.83399999999999996,"15min":1.0309999999999999},"last":0.65600000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.51500000000000001,"15min":0.51000000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.68300000000000005,"5min":1.008,"15min":1.008},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.57399999999999995,"15min":0.55400000000000005},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.77800000000000002,"5min":0.874,"15min":0.94199999999999995},"last":0.67300000000000004}]},{"osd":7,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57999999999999996,"15min":0.56399999999999995},"min":{"1min":0.38100000000000001,"5min":0.35899999999999999,"15min":0.34499999999999997},"max":{"1min":0.76500000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58699999999999997,"15min":0.57599999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.74299999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.71099999999999997}]}]},{"osd":1,"up_from":13,"seq":55834576029,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52820,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319020,"statfs":{"total":96636764160,"available":96582676480,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.373,"15min":0.38900000000000001},"min":{"1min":0.246,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.50800000000000001,"5min":0.60899999999999999,"15min":0.70199999999999996},"last":0.313},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.39000000000000001,"15min":0.41499999999999998},"min":{"1min":0.28999999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.69699999999999995},"last":0.39100000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.39300000000000002,"15min":0.42099999999999999},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.626,"5min":0.626,"15min":0.68100000000000005},"last":0.253},{"interface":"front","average":{"1min":0.438,"5min":0.436,"15min":0.46300000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61599999999999999,"5min":0.70099999999999996,"15min":0.80500000000000005},"last":0.36099999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.441,"15min":0.47599999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65200000000000002,"5min":0.65600000000000003,"15min":0.76100000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.39800000000000002,"15min":0.42699999999999999},"min":{"1min":0.24399999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.58899999999999997,"5min":0.68200000000000005,"15min":0.71699999999999997},"last":0.34100000000000003}]},{"osd":4,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63200000000000001,"15min":0.66100000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.94599999999999995,"5min":1.288,"15min":1.2909999999999999},"last":0.63},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63400000000000001,"15min":0.66600000000000004},"min":{"1min":0.44700000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.88900000000000001,"5min":1.349,"15min":1.746},"last":0.48999999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.623,"15min":0.65900000000000003},"min":{"1min":0.40100000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.92700000000000005,"5min":1.361,"15min":1.6910000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.66500000000000004,"15min":0.69999999999999996},"min":{"1min":0.48199999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.97099999999999997,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.58799999999999997}]},{"osd":6,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63,"15min":0.67100000000000004},"min":{"1min":0.39400000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":1.3120000000000001,"15min":1.637},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.65300000000000002,"15min":0.69299999999999995},"min":{"1min":0.44800000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.91100000000000003,"5min":0.93500000000000005,"15min":1.722},"last":0.51000000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.65400000000000003,"15min":0.69599999999999995},"min":{"1min":0.42899999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.89100000000000001,"5min":1.375,"15min":1.375},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.66100000000000003,"15min":0.69099999999999995},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.98899999999999999,"5min":1.431,"15min":1.431},"last":0.55800000000000005}]}]},{"osd":0,"up_from":13,"seq":55834576030,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49296,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322544,"statfs":{"total":96636764160,"available":96586285056,"internally_reserved":0,"allocated":4329472,"data_stored":2528024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.35899999999999999,"15min":0.38},"min":{"1min":0.25,"5min":0.214,"15min":0.214},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.38900000000000001,"15min":0.41599999999999998},"min":{"1min":0.30599999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":1.1850000000000001},"last":0.32500000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.41199999999999998,"15min":0.42899999999999999},"min":{"1min":0.34899999999999998,"5min":0.27100000000000002,"15min":0.24099999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.41099999999999998,"15min":0.439},"min":{"1min":0.28799999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.70799999999999996},"last":0.34899999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.38700000000000001,"15min":0.41699999999999998},"min":{"1min":0.312,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.57499999999999996,"5min":0.58499999999999996,"15min":0.628},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.45200000000000001,"15min":0.47899999999999998},"min":{"1min":0.309,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.91500000000000004},"last":0.79900000000000004}]},{"osd":4,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.61299999999999999,"15min":0.64000000000000001},"min":{"1min":0.40100000000000002,"5min":0.40000000000000002,"15min":0.36799999999999999},"max":{"1min":0.91200000000000003,"5min":1.506,"15min":1.506},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.443,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.01,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.51200000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.65000000000000002,"15min":0.66400000000000003},"min":{"1min":0.439,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.002,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.64500000000000002,"15min":0.67300000000000004},"min":{"1min":0.53000000000000003,"5min":0.432,"15min":0.432},"max":{"1min":1.006,"5min":1.468,"15min":1.518},"last":0.75900000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.63400000000000001,"15min":0.65600000000000003},"min":{"1min":0.52700000000000002,"5min":0.436,"15min":0.436},"max":{"1min":0.92000000000000004,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.67400000000000004,"15min":0.69699999999999995},"min":{"1min":0.56200000000000006,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0860000000000001,"5min":1.5649999999999999,"15min":1.5649999999999999},"last":0.83399999999999996}]},{"osd":7,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.67100000000000004,"15min":0.69499999999999995},"min":{"1min":0.504,"5min":0.45300000000000001,"15min":0.41899999999999998},"max":{"1min":1.016,"5min":1.55,"15min":1.55},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.65200000000000002,"15min":0.68700000000000006},"min":{"1min":0.57299999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0269999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.57299999999999995}]}]},{"osd":2,"up_from":13,"seq":55834576030,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52812,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319028,"statfs":{"total":96636764160,"available":96582684672,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.33900000000000002,"15min":0.36399999999999999},"min":{"1min":0.28299999999999997,"5min":0.184,"15min":0.184},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.42899999999999999,"15min":0.46600000000000003},"min":{"1min":0.34300000000000003,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.753},"last":0.33300000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.433,"15min":0.46400000000000002},"min":{"1min":0.40500000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.749},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.36399999999999999,"15min":0.39900000000000002},"min":{"1min":0.307,"5min":0.224,"15min":0.224},"max":{"1min":0.54000000000000004,"5min":0.54600000000000004,"15min":0.59999999999999998},"last":0.378}]},{"osd":3,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.38600000000000001,"15min":0.41899999999999998},"min":{"1min":0.32800000000000001,"5min":0.26500000000000001,"15min":0.22800000000000001},"max":{"1min":0.623,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.44700000000000001,"15min":0.49399999999999999},"min":{"1min":0.372,"5min":0.315,"15min":0.315},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.752},"last":0.42899999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.63300000000000001,"15min":0.64900000000000002},"min":{"1min":0.505,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.093,"5min":1.2090000000000001,"15min":1.4079999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.62,"15min":0.65500000000000003},"min":{"1min":0.39900000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98799999999999999,"5min":1.268,"15min":1.2729999999999999},"last":0.57999999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.624,"15min":0.66300000000000003},"min":{"1min":0.53400000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1,"5min":1,"15min":1.028},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.63200000000000001,"15min":0.66400000000000003},"min":{"1min":0.56599999999999995,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.64000000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.626,"15min":0.67000000000000004},"min":{"1min":0.52900000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.628},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.69599999999999995,"15min":0.72099999999999997},"min":{"1min":0.499,"5min":0.439,"15min":0.439},"max":{"1min":1.1299999999999999,"5min":1.3220000000000001,"15min":1.3859999999999999},"last":0.61399999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.63500000000000001,"15min":0.67900000000000005},"min":{"1min":0.497,"5min":0.432,"15min":0.42199999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0269999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.746,"5min":0.67000000000000004,"15min":0.69299999999999995},"min":{"1min":0.50600000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1739999999999999,"5min":1.2330000000000001,"15min":1.371},"last":0.66300000000000003}]}]},{"osd":3,"up_from":13,"seq":55834576030,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56908,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314932,"statfs":{"total":96636764160,"available":96578490368,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.34200000000000003,"15min":0.378},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.66500000000000004},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.35999999999999999,"15min":0.39900000000000002},"min":{"1min":0.27200000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.61699999999999999},"last":0.312}]},{"osd":1,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.35499999999999998,"15min":0.39000000000000001},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.214},"max":{"1min":0.55100000000000005,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.41099999999999998,"15min":0.44700000000000001},"min":{"1min":0.32800000000000001,"5min":0.29099999999999998,"15min":0.25800000000000001},"max":{"1min":0.60399999999999998,"5min":0.63500000000000001,"15min":0.72599999999999998},"last":0.27600000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.40799999999999997,"15min":0.44400000000000001},"min":{"1min":0.32600000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.68400000000000005},"last":0.374},{"interface":"front","average":{"1min":0.502,"5min":0.433,"15min":0.47299999999999998},"min":{"1min":0.36299999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.71199999999999997},"last":0.34799999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.439,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98499999999999999,"5min":1.526,"15min":1.526},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.622,"15min":0.65600000000000003},"min":{"1min":0.45500000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97899999999999998,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.44700000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.65600000000000003,"15min":0.67700000000000005},"min":{"1min":0.41299999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94399999999999995,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.66100000000000003,"15min":0.70799999999999996},"min":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.99099999999999999,"5min":1.1120000000000001,"15min":1.171},"last":0.52300000000000002}]},{"osd":6,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.66000000000000003,"15min":0.68899999999999995},"min":{"1min":0.42499999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.878,"5min":1.573,"15min":1.573},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.67300000000000004,"15min":0.70999999999999996},"min":{"1min":0.48999999999999999,"5min":0.47399999999999998,"15min":0.45100000000000001},"max":{"1min":1.02,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.51100000000000001}]},{"osd":7,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63700000000000001,"15min":0.68999999999999995},"min":{"1min":0.42799999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.032,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.65300000000000002,"15min":0.68200000000000005},"min":{"1min":0.439,"5min":0.41099999999999998,"15min":0.39200000000000002},"max":{"1min":0.94699999999999995,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.61799999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576030,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48720,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.55600000000000005,"15min":0.57999999999999996},"min":{"1min":0.44,"5min":0.34300000000000003,"15min":0.29499999999999998},"max":{"1min":0.77600000000000002,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56599999999999995,"15min":0.58399999999999996},"min":{"1min":0.46700000000000003,"5min":0.40100000000000002,"15min":0.36299999999999999},"max":{"1min":0.79200000000000004,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.63300000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.58199999999999996,"15min":0.60299999999999998},"min":{"1min":0.45500000000000002,"5min":0.35799999999999998,"15min":0.318},"max":{"1min":0.81000000000000005,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.55500000000000005,"15min":0.57299999999999995},"min":{"1min":0.496,"5min":0.34899999999999998,"15min":0.34300000000000003},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.96499999999999997},"last":0.64800000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.58699999999999997,"15min":0.59199999999999997},"min":{"1min":0.52800000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.82699999999999996,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.58899999999999997,"15min":0.60499999999999998},"min":{"1min":0.40200000000000002,"5min":0.38900000000000001,"15min":0.38500000000000001},"max":{"1min":0.76700000000000002,"5min":0.83099999999999996,"15min":0.84099999999999997},"last":0.67600000000000005}]},{"osd":3,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58099999999999996,"15min":0.59199999999999997},"min":{"1min":0.5,"5min":0.374,"15min":0.33900000000000002},"max":{"1min":0.82799999999999996,"5min":1.026,"15min":1.026},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.60499999999999998,"15min":0.61099999999999999},"min":{"1min":0.503,"5min":0.36699999999999999,"15min":0.34999999999999998},"max":{"1min":0.85499999999999998,"5min":1.028,"15min":1.028},"last":0.69799999999999995}]},{"osd":5,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.499,"15min":0.48199999999999998},"min":{"1min":0.34799999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.67400000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.52400000000000002,"15min":0.505},"min":{"1min":0.40799999999999997,"5min":0.31900000000000001,"15min":0.29999999999999999},"max":{"1min":0.68100000000000005,"5min":0.85499999999999998,"15min":0.90600000000000003},"last":0.54400000000000004}]},{"osd":6,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52700000000000002,"15min":0.50600000000000001},"min":{"1min":0.36799999999999999,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.748,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.56999999999999995,"15min":0.55300000000000005},"min":{"1min":0.435,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.77900000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.57299999999999995}]},{"osd":7,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56599999999999995,"15min":0.54800000000000004},"min":{"1min":0.38700000000000001,"5min":0.36899999999999999,"15min":0.35699999999999998},"max":{"1min":0.70499999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60699999999999998,"15min":0.58399999999999996},"min":{"1min":0.45900000000000002,"5min":0.36599999999999999,"15min":0.33700000000000002},"max":{"1min":0.79200000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.61499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834576029,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70860,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94300980,"statfs":{"total":96636764160,"available":96564203520,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54100000000000004,"15min":0.58399999999999996},"min":{"1min":0.36599999999999999,"5min":0.33500000000000002,"15min":0.31900000000000001},"max":{"1min":0.73599999999999999,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.371},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53700000000000003,"15min":0.58599999999999997},"min":{"1min":0.41799999999999998,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":0.80600000000000005,"5min":0.82399999999999995,"15min":1.3100000000000001},"last":0.56899999999999995}]},{"osd":1,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55100000000000005,"15min":0.59199999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.31900000000000001},"max":{"1min":0.77600000000000002,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53900000000000003,"15min":0.58399999999999996},"min":{"1min":0.378,"5min":0.372,"15min":0.35199999999999998},"max":{"1min":0.81699999999999995,"5min":0.94299999999999995,"15min":1.018},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.54200000000000004,"15min":0.58499999999999996},"min":{"1min":0.35499999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.99299999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54900000000000004,"15min":0.59599999999999997},"min":{"1min":0.44900000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80000000000000004,"5min":1.599,"15min":1.599},"last":0.501}]},{"osd":3,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56299999999999994,"15min":0.59799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76800000000000002,"5min":0.98699999999999999,"15min":1.2430000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58799999999999997,"15min":0.63200000000000001},"min":{"1min":0.42999999999999999,"5min":0.40600000000000003,"15min":0.36699999999999999},"max":{"1min":0.76200000000000001,"5min":0.84399999999999997,"15min":1.161},"last":0.47599999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.48799999999999999,"15min":0.48299999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57799999999999996,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53200000000000003,"15min":0.52000000000000002},"min":{"1min":0.28199999999999997,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.70599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.57999999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.50700000000000001,"15min":0.51200000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.65200000000000002,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.54400000000000004,"15min":0.54000000000000004},"min":{"1min":0.33400000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.68500000000000005,"5min":1.643,"15min":1.643},"last":0.59299999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56599999999999995,"15min":0.56000000000000005},"min":{"1min":0.40300000000000002,"5min":0.32800000000000001,"15min":0.317},"max":{"1min":0.77600000000000002,"5min":1.028,"15min":1.028},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.58299999999999996,"15min":0.57899999999999996},"min":{"1min":0.39200000000000002,"5min":0.34300000000000003,"15min":0.29799999999999999},"max":{"1min":0.67800000000000005,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.54200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:29:57.333 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T20:29:57.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.455+0000 7ffb45833700 1 Processor -- start 2024-03-18T20:29:57.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.455+0000 7ffb45833700 1 -- start start 2024-03-18T20:29:57.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.455+0000 7ffb45833700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffb400ae1a0 0x7ffb400ae830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:57.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.455+0000 7ffb45833700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ffb401610a0 0x7ffb400aed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:57.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.455+0000 7ffb45833700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4006d350 con 0x7ffb4015d410 2024-03-18T20:29:57.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.455+0000 7ffb45833700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb4006ed50 con 0x7ffb400ae1a0 2024-03-18T20:29:57.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.455+0000 7ffb45833700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb4006e1b0 con 0x7ffb401610a0 2024-03-18T20:29:57.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.456+0000 7ffb3e7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffb400ae1a0 0x7ffb400ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:57.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.456+0000 7ffb3e7fc700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffb400ae1a0 0x7ffb400ae830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50070/0 (socket says 172.21.15.1:50070) 2024-03-18T20:29:57.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.456+0000 7ffb3e7fc700 1 -- 172.21.15.1:0/2471145322 learned_addr learned my addr 172.21.15.1:0/2471145322 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:57.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.456+0000 7ffb3effd700 1 --1- 172.21.15.1:0/2471145322 >> v1:172.21.15.1:6789/0 conn(0x7ffb4015d410 0x7ffb4006e640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59254/0 (socket says 172.21.15.1:59254) 2024-03-18T20:29:57.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.456+0000 7ffb3f7fe700 1 --2- 172.21.15.1:0/2471145322 >> v2:172.21.15.165:3300/0 conn(0x7ffb401610a0 0x7ffb400aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:57.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.456+0000 7ffb3e7fc700 1 -- 172.21.15.1:0/2471145322 >> v2:172.21.15.165:3300/0 conn(0x7ffb401610a0 msgr2=0x7ffb400aed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:57.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.456+0000 7ffb3e7fc700 1 --2- 172.21.15.1:0/2471145322 >> v2:172.21.15.165:3300/0 conn(0x7ffb401610a0 0x7ffb400aed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.456+0000 7ffb3e7fc700 1 -- 172.21.15.1:0/2471145322 >> v1:172.21.15.1:6789/0 conn(0x7ffb4015d410 legacy=0x7ffb4006e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:57.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.456+0000 7ffb3e7fc700 1 -- 172.21.15.1:0/2471145322 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4006d5a0 con 0x7ffb400ae1a0 2024-03-18T20:29:57.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.456+0000 7ffb3e7fc700 1 --2- 172.21.15.1:0/2471145322 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb400ae1a0 0x7ffb400ae830 secure :-1 s=READY pgs=3417 cs=0 l=1 rev1=1 crypto rx=0x7ffb40066d00 tx=0x7ffb34009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d859b3da73d3cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:57.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.456+0000 7ffb3dffb700 1 -- 172.21.15.1:0/2471145322 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3400b870 con 0x7ffb400ae1a0 2024-03-18T20:29:57.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.456+0000 7ffb3dffb700 1 -- 172.21.15.1:0/2471145322 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3400b9d0 con 0x7ffb400ae1a0 2024-03-18T20:29:57.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.456+0000 7ffb3dffb700 1 -- 172.21.15.1:0/2471145322 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3400bd60 con 0x7ffb400ae1a0 2024-03-18T20:29:57.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.457+0000 7ffb45833700 1 -- 172.21.15.1:0/2471145322 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb400ae1a0 msgr2=0x7ffb400ae830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:57.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.457+0000 7ffb45833700 1 --2- 172.21.15.1:0/2471145322 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb400ae1a0 0x7ffb400ae830 secure :-1 s=READY pgs=3417 cs=0 l=1 rev1=1 crypto rx=0x7ffb40066d00 tx=0x7ffb34009bd0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.457+0000 7ffb45833700 1 -- 172.21.15.1:0/2471145322 shutdown_connections 2024-03-18T20:29:57.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.457+0000 7ffb45833700 1 --2- 172.21.15.1:0/2471145322 >> v2:172.21.15.165:3300/0 conn(0x7ffb401610a0 0x7ffb400aed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.457+0000 7ffb45833700 1 --2- 172.21.15.1:0/2471145322 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb400ae1a0 0x7ffb400ae830 unknown :-1 s=CLOSED pgs=3417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.457+0000 7ffb45833700 1 -- 172.21.15.1:0/2471145322 >> 172.21.15.1:0/2471145322 conn(0x7ffb40072030 msgr2=0x7ffb4006a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:57.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.457+0000 7ffb45833700 1 -- 172.21.15.1:0/2471145322 shutdown_connections 2024-03-18T20:29:57.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.457+0000 7ffb45833700 1 -- 172.21.15.1:0/2471145322 wait complete. 2024-03-18T20:29:57.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.458+0000 7ffb45833700 1 Processor -- start 2024-03-18T20:29:57.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.458+0000 7ffb45833700 1 -- start start 2024-03-18T20:29:57.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.459+0000 7ffb45833700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb400ae1a0 0x7ffb40155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:57.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.459+0000 7ffb45833700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4015d410 0x7ffb4014d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:57.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.459+0000 7ffb45833700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb401610a0 0x7ffb40153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:57.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.459+0000 7ffb45833700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4014dcc0 con 0x7ffb4015d410 2024-03-18T20:29:57.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.459+0000 7ffb45833700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4015eb90 con 0x7ffb400ae1a0 2024-03-18T20:29:57.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.459+0000 7ffb45833700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb40153b40 con 0x7ffb401610a0 2024-03-18T20:29:57.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.459+0000 7ffb3e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4015d410 0x7ffb4014d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:57.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.459+0000 7ffb3e7fc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4015d410 0x7ffb4014d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50086/0 (socket says 172.21.15.1:50086) 2024-03-18T20:29:57.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.459+0000 7ffb3e7fc700 1 -- 172.21.15.1:0/3670797937 learned_addr learned my addr 172.21.15.1:0/3670797937 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:57.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.459+0000 7ffb3effd700 1 --2- 172.21.15.1:0/3670797937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb400ae1a0 0x7ffb40155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:57.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.459+0000 7ffb3f7fe700 1 --2- 172.21.15.1:0/3670797937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb401610a0 0x7ffb40153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:57.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.459+0000 7ffb3e7fc700 1 -- 172.21.15.1:0/3670797937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb401610a0 msgr2=0x7ffb40153600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:57.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.459+0000 7ffb3e7fc700 1 --2- 172.21.15.1:0/3670797937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb401610a0 0x7ffb40153600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.459+0000 7ffb3e7fc700 1 -- 172.21.15.1:0/3670797937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb400ae1a0 msgr2=0x7ffb40155ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:57.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.459+0000 7ffb3e7fc700 1 --2- 172.21.15.1:0/3670797937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb400ae1a0 0x7ffb40155ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.459+0000 7ffb3e7fc700 1 -- 172.21.15.1:0/3670797937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb34009550 con 0x7ffb4015d410 2024-03-18T20:29:57.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.460+0000 7ffb3e7fc700 1 --2- 172.21.15.1:0/3670797937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4015d410 0x7ffb4014d780 secure :-1 s=READY pgs=3418 cs=0 l=1 rev1=1 crypto rx=0x7ffb34000c00 tx=0x7ffb34004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:57.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.460+0000 7ffb27fff700 1 -- 172.21.15.1:0/3670797937 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb34004540 con 0x7ffb4015d410 2024-03-18T20:29:57.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.460+0000 7ffb27fff700 1 -- 172.21.15.1:0/3670797937 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb340036a0 con 0x7ffb4015d410 2024-03-18T20:29:57.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.460+0000 7ffb45833700 1 -- 172.21.15.1:0/3670797937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4015e260 con 0x7ffb4015d410 2024-03-18T20:29:57.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.460+0000 7ffb27fff700 1 -- 172.21.15.1:0/3670797937 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb34003a30 con 0x7ffb4015d410 2024-03-18T20:29:57.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.460+0000 7ffb45833700 1 -- 172.21.15.1:0/3670797937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb4015e820 con 0x7ffb4015d410 2024-03-18T20:29:57.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.461+0000 7ffb45833700 1 -- 172.21.15.1:0/3670797937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb40067dc0 con 0x7ffb4015d410 2024-03-18T20:29:57.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.461+0000 7ffb27fff700 1 -- 172.21.15.1:0/3670797937 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7ffb34021070 con 0x7ffb4015d410 2024-03-18T20:29:57.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.462+0000 7ffb27fff700 1 --2- 172.21.15.1:0/3670797937 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffb2c06ade0 0x7ffb2c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:57.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.462+0000 7ffb27fff700 1 -- 172.21.15.1:0/3670797937 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7ffb3408ae00 con 0x7ffb4015d410 2024-03-18T20:29:57.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.462+0000 7ffb3effd700 1 --2- 172.21.15.1:0/3670797937 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffb2c06ade0 0x7ffb2c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:57.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.462+0000 7ffb3effd700 1 --2- 172.21.15.1:0/3670797937 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffb2c06ade0 0x7ffb2c06d2a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ffb30000f60 tx=0x7ffb30007d10 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:57.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.466+0000 7ffb27fff700 1 -- 172.21.15.1:0/3670797937 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffb34055e90 con 0x7ffb4015d410 2024-03-18T20:29:57.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.669+0000 7ffb45833700 1 -- 172.21.15.1:0/3670797937 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb4015e9b0 con 0x7ffb4015d410 2024-03-18T20:29:57.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.670+0000 7ffb27fff700 1 -- 172.21.15.1:0/3670797937 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v295) v1 ==== 74+0+13875 (secure 0 0 0) 0x7ffb34015020 con 0x7ffb4015d410 2024-03-18T20:29:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:29:57.674 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":295,"fsid":"9c1c697f-8655-4ef3-8a13-da568d2cbe86","created":"2024-03-18T18:49:27.125279+0000","modified":"2024-03-18T20:29:53.178007+0000","last_up_change":"2024-03-18T18:49:37.757484+0000","last_in_change":"2024-03-18T18:49:29.672898+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":38,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T18:49:39.443363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"380d0a40-8340-4654-9353-97df2416c043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":77296},{"type":"v1","addr":"172.21.15.1:6825","nonce":77296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":77296},{"type":"v1","addr":"172.21.15.1:6827","nonce":77296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":77296},{"type":"v1","addr":"172.21.15.1:6831","nonce":77296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":77296},{"type":"v1","addr":"172.21.15.1:6829","nonce":77296}]},"public_addr":"172.21.15.1:6825/77296","cluster_addr":"172.21.15.1:6827/77296","heartbeat_back_addr":"172.21.15.1:6831/77296","heartbeat_front_addr":"172.21.15.1:6829/77296","state":["exists","up"]},{"osd":1,"uuid":"9c851da3-1c5d-4839-aba9-2d35a12ac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":77295},{"type":"v1","addr":"172.21.15.1:6801","nonce":77295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":77295},{"type":"v1","addr":"172.21.15.1:6803","nonce":77295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":77295},{"type":"v1","addr":"172.21.15.1:6807","nonce":77295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":77295},{"type":"v1","addr":"172.21.15.1:6805","nonce":77295}]},"public_addr":"172.21.15.1:6801/77295","cluster_addr":"172.21.15.1:6803/77295","heartbeat_back_addr":"172.21.15.1:6807/77295","heartbeat_front_addr":"172.21.15.1:6805/77295","state":["exists","up"]},{"osd":2,"uuid":"36be270f-cbee-4db9-a511-9bbb18e21a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":77305},{"type":"v1","addr":"172.21.15.1:6809","nonce":77305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":77305},{"type":"v1","addr":"172.21.15.1:6811","nonce":77305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":77305},{"type":"v1","addr":"172.21.15.1:6815","nonce":77305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":77305},{"type":"v1","addr":"172.21.15.1:6813","nonce":77305}]},"public_addr":"172.21.15.1:6809/77305","cluster_addr":"172.21.15.1:6811/77305","heartbeat_back_addr":"172.21.15.1:6815/77305","heartbeat_front_addr":"172.21.15.1:6813/77305","state":["exists","up"]},{"osd":3,"uuid":"428759f1-b3c5-4d09-9ec1-d39c92985a96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":290,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":77310},{"type":"v1","addr":"172.21.15.1:6817","nonce":77310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":77310},{"type":"v1","addr":"172.21.15.1:6819","nonce":77310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":77310},{"type":"v1","addr":"172.21.15.1:6823","nonce":77310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":77310},{"type":"v1","addr":"172.21.15.1:6821","nonce":77310}]},"public_addr":"172.21.15.1:6817/77310","cluster_addr":"172.21.15.1:6819/77310","heartbeat_back_addr":"172.21.15.1:6823/77310","heartbeat_front_addr":"172.21.15.1:6821/77310","state":["exists","up"]},{"osd":4,"uuid":"3543b3ec-2d8d-48b9-9a55-f7911b347e90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":290,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":77474},{"type":"v1","addr":"172.21.15.165:6809","nonce":77474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":77474},{"type":"v1","addr":"172.21.15.165:6811","nonce":77474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":77474},{"type":"v1","addr":"172.21.15.165:6815","nonce":77474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":77474},{"type":"v1","addr":"172.21.15.165:6813","nonce":77474}]},"public_addr":"172.21.15.165:6809/77474","cluster_addr":"172.21.15.165:6811/77474","heartbeat_back_addr":"172.21.15.165:6815/77474","heartbeat_front_addr":"172.21.15.165:6813/77474","state":["exists","up"]},{"osd":5,"uuid":"0a8b4e7c-f079-4266-9142-75791dbe3b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":77475},{"type":"v1","addr":"172.21.15.165:6801","nonce":77475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":77475},{"type":"v1","addr":"172.21.15.165:6803","nonce":77475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":77475},{"type":"v1","addr":"172.21.15.165:6807","nonce":77475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":77475},{"type":"v1","addr":"172.21.15.165:6805","nonce":77475}]},"public_addr":"172.21.15.165:6801/77475","cluster_addr":"172.21.15.165:6803/77475","heartbeat_back_addr":"172.21.15.165:6807/77475","heartbeat_front_addr":"172.21.15.165:6805/77475","state":["exists","up"]},{"osd":6,"uuid":"e7af56bf-343c-469e-9f54-fb2f0cb3d51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6817","nonce":77477},{"type":"v1","addr":"172.21.15.165:6819","nonce":77477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":77477},{"type":"v1","addr":"172.21.15.165:6824","nonce":77477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":77477},{"type":"v1","addr":"172.21.15.165:6831","nonce":77477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6827","nonce":77477},{"type":"v1","addr":"172.21.15.165:6829","nonce":77477}]},"public_addr":"172.21.15.165:6819/77477","cluster_addr":"172.21.15.165:6824/77477","heartbeat_back_addr":"172.21.15.165:6831/77477","heartbeat_front_addr":"172.21.15.165:6829/77477","state":["exists","up"]},{"osd":7,"uuid":"2ca85776-3586-4679-b571-dc8d294d9dbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":286,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":77476},{"type":"v1","addr":"172.21.15.165:6818","nonce":77476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":77476},{"type":"v1","addr":"172.21.15.165:6821","nonce":77476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":77476},{"type":"v1","addr":"172.21.15.165:6828","nonce":77476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6823","nonce":77476},{"type":"v1","addr":"172.21.15.165:6825","nonce":77476}]},"public_addr":"172.21.15.165:6818/77476","cluster_addr":"172.21.15.165:6821/77476","heartbeat_back_addr":"172.21.15.165:6828/77476","heartbeat_front_addr":"172.21.15.165:6825/77476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.772413+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.830534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.873264+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.786637+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.872317+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.859323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T18:49:34.840227+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:6836/1111156155":"2024-03-19T20:29:49.830260+0000","172.21.15.1:6835/1111156155":"2024-03-19T20:29:49.830260+0000","172.21.15.1:0/1196516798":"2024-03-19T20:29:49.061624+0000","172.21.15.1:0/497050575":"2024-03-19T20:29:49.061624+0000","172.21.15.1:0/4210456696":"2024-03-19T20:29:49.061624+0000","172.21.15.1:0/1141901935":"2024-03-19T20:29:49.061624+0000","172.21.15.1:0/4226319191":"2024-03-19T20:29:49.061624+0000","172.21.15.1:0/598999338":"2024-03-19T20:29:49.061624+0000","172.21.15.1:0/1687261463":"2024-03-19T20:29:49.061624+0000","172.21.15.1:0/1650741459":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/2451665032":"2024-03-19T20:28:26.019720+0000","172.21.15.1:6833/148356":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/2778202311":"2024-03-19T20:29:49.061624+0000","172.21.15.1:6832/148356":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/1931029093":"2024-03-19T20:29:49.061624+0000","172.21.15.1:0/1408878463":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/582749307":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/997400106":"2024-03-19T20:28:26.019720+0000","172.21.15.1:0/2561447959":"2024-03-19T20:28:26.019720+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":2,"length":2}]},{"pool":20,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]},{"pool":24,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T20:29:57.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.672+0000 7ffb45833700 1 -- 172.21.15.1:0/3670797937 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffb2c06ade0 msgr2=0x7ffb2c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:57.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.672+0000 7ffb45833700 1 --2- 172.21.15.1:0/3670797937 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffb2c06ade0 0x7ffb2c06d2a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ffb30000f60 tx=0x7ffb30007d10 comp rx=0 tx=0).stop 2024-03-18T20:29:57.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.672+0000 7ffb45833700 1 -- 172.21.15.1:0/3670797937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4015d410 msgr2=0x7ffb4014d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:57.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.672+0000 7ffb45833700 1 --2- 172.21.15.1:0/3670797937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4015d410 0x7ffb4014d780 secure :-1 s=READY pgs=3418 cs=0 l=1 rev1=1 crypto rx=0x7ffb34000c00 tx=0x7ffb34004750 comp rx=0 tx=0).stop 2024-03-18T20:29:57.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.672+0000 7ffb45833700 1 -- 172.21.15.1:0/3670797937 shutdown_connections 2024-03-18T20:29:57.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.672+0000 7ffb45833700 1 --2- 172.21.15.1:0/3670797937 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7ffb2c06ade0 0x7ffb2c06d2a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.672+0000 7ffb45833700 1 --2- 172.21.15.1:0/3670797937 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7ffb401610a0 0x7ffb40153600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.672+0000 7ffb45833700 1 --2- 172.21.15.1:0/3670797937 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb400ae1a0 0x7ffb40155ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.672+0000 7ffb45833700 1 --2- 172.21.15.1:0/3670797937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb4015d410 0x7ffb4014d780 unknown :-1 s=CLOSED pgs=3418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:57.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.672+0000 7ffb45833700 1 -- 172.21.15.1:0/3670797937 >> 172.21.15.1:0/3670797937 conn(0x7ffb40072030 msgr2=0x7ffb400ac8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:57.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.672+0000 7ffb45833700 1 -- 172.21.15.1:0/3670797937 shutdown_connections 2024-03-18T20:29:57.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:57.672+0000 7ffb45833700 1 -- 172.21.15.1:0/3670797937 wait complete. 2024-03-18T20:29:58.689 INFO:tasks.ceph:Scrubbing osd.0 2024-03-18T20:29:58.689 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-18T20:29:58.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.829+0000 7f7fcff1c700 1 Processor -- start 2024-03-18T20:29:58.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.830+0000 7f7fcff1c700 1 -- start start 2024-03-18T20:29:58.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.830+0000 7f7fcff1c700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7fc806f420 0x7f7fc815b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:58.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.830+0000 7f7fcff1c700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7fc80b3be0 0x7f7fc8099a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:58.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.830+0000 7f7fcff1c700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc806df80 con 0x7f7fc815a890 2024-03-18T20:29:58.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.830+0000 7f7fcff1c700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc8099f70 con 0x7f7fc80b3be0 2024-03-18T20:29:58.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.830+0000 7f7fcff1c700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc80b41a0 con 0x7f7fc806f420 2024-03-18T20:29:58.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.830+0000 7f7fce4b9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7fc80b3be0 0x7f7fc8099a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:58.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.830+0000 7f7fce4b9700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7fc80b3be0 0x7f7fc8099a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50100/0 (socket says 172.21.15.1:50100) 2024-03-18T20:29:58.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.831+0000 7f7fce4b9700 1 -- 172.21.15.1:0/706143691 learned_addr learned my addr 172.21.15.1:0/706143691 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:58.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.831+0000 7f7fcdcb8700 1 --1- 172.21.15.1:0/706143691 >> v1:172.21.15.1:6789/0 conn(0x7f7fc815a890 0x7f7fc815ac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59268/0 (socket says 172.21.15.1:59268) 2024-03-18T20:29:58.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.831+0000 7f7fcd4b7700 1 --2- 172.21.15.1:0/706143691 >> v2:172.21.15.165:3300/0 conn(0x7f7fc806f420 0x7f7fc815b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:58.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.831+0000 7f7fce4b9700 1 -- 172.21.15.1:0/706143691 >> v2:172.21.15.165:3300/0 conn(0x7f7fc806f420 msgr2=0x7f7fc815b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:58.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.831+0000 7f7fce4b9700 1 --2- 172.21.15.1:0/706143691 >> v2:172.21.15.165:3300/0 conn(0x7f7fc806f420 0x7f7fc815b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:58.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.831+0000 7f7fce4b9700 1 -- 172.21.15.1:0/706143691 >> v1:172.21.15.1:6789/0 conn(0x7f7fc815a890 legacy=0x7f7fc815ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:58.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.831+0000 7f7fce4b9700 1 -- 172.21.15.1:0/706143691 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc80b4480 con 0x7f7fc80b3be0 2024-03-18T20:29:58.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.831+0000 7f7fce4b9700 1 --2- 172.21.15.1:0/706143691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fc80b3be0 0x7f7fc8099a30 secure :-1 s=READY pgs=3420 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8067310 tx=0x7f7fc000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e10ec6f39e1752a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:58.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.831+0000 7f7fcccb6700 1 -- 172.21.15.1:0/706143691 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc000e070 con 0x7f7fc80b3be0 2024-03-18T20:29:58.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.831+0000 7f7fcccb6700 1 -- 172.21.15.1:0/706143691 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc000b9c0 con 0x7f7fc80b3be0 2024-03-18T20:29:58.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.831+0000 7f7fcccb6700 1 -- 172.21.15.1:0/706143691 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc000bd70 con 0x7f7fc80b3be0 2024-03-18T20:29:58.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.832+0000 7f7fcff1c700 1 -- 172.21.15.1:0/706143691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fc80b3be0 msgr2=0x7f7fc8099a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:58.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.832+0000 7f7fcff1c700 1 --2- 172.21.15.1:0/706143691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fc80b3be0 0x7f7fc8099a30 secure :-1 s=READY pgs=3420 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8067310 tx=0x7f7fc000b330 comp rx=0 tx=0).stop 2024-03-18T20:29:58.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.832+0000 7f7fcff1c700 1 -- 172.21.15.1:0/706143691 shutdown_connections 2024-03-18T20:29:58.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.832+0000 7f7fcff1c700 1 --2- 172.21.15.1:0/706143691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fc80b3be0 0x7f7fc8099a30 unknown :-1 s=CLOSED pgs=3420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:58.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.832+0000 7f7fcff1c700 1 --2- 172.21.15.1:0/706143691 >> v2:172.21.15.165:3300/0 conn(0x7f7fc806f420 0x7f7fc815b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:58.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.832+0000 7f7fcff1c700 1 -- 172.21.15.1:0/706143691 >> 172.21.15.1:0/706143691 conn(0x7f7fc80b5910 msgr2=0x7f7fc80b1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:58.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.832+0000 7f7fcff1c700 1 -- 172.21.15.1:0/706143691 shutdown_connections 2024-03-18T20:29:58.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.832+0000 7f7fcff1c700 1 -- 172.21.15.1:0/706143691 wait complete. 2024-03-18T20:29:58.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.833+0000 7f7fcff1c700 1 Processor -- start 2024-03-18T20:29:58.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.833+0000 7f7fcff1c700 1 -- start start 2024-03-18T20:29:58.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fcff1c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fc806f420 0x7f7fc8149c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:58.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fcff1c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fc80b3be0 0x7f7fc81470a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:58.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fcff1c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fc815a890 0x7f7fc8152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:58.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fcff1c700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc814a150 con 0x7f7fc815a890 2024-03-18T20:29:58.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fcff1c700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc8148b70 con 0x7f7fc806f420 2024-03-18T20:29:58.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fcff1c700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc8148e50 con 0x7f7fc80b3be0 2024-03-18T20:29:58.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fce4b9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fc815a890 0x7f7fc8152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:58.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fce4b9700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fc815a890 0x7f7fc8152f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50104/0 (socket says 172.21.15.1:50104) 2024-03-18T20:29:58.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fce4b9700 1 -- 172.21.15.1:0/1236028186 learned_addr learned my addr 172.21.15.1:0/1236028186 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:58.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fcdcb8700 1 --2- 172.21.15.1:0/1236028186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fc806f420 0x7f7fc8149c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:58.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fcd4b7700 1 --2- 172.21.15.1:0/1236028186 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fc80b3be0 0x7f7fc81470a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:58.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fce4b9700 1 -- 172.21.15.1:0/1236028186 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fc80b3be0 msgr2=0x7f7fc81470a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:58.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fce4b9700 1 --2- 172.21.15.1:0/1236028186 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f7fc80b3be0 0x7f7fc81470a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:58.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fce4b9700 1 -- 172.21.15.1:0/1236028186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fc806f420 msgr2=0x7f7fc8149c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:58.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fce4b9700 1 --2- 172.21.15.1:0/1236028186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fc806f420 0x7f7fc8149c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:58.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fce4b9700 1 -- 172.21.15.1:0/1236028186 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc0009b30 con 0x7f7fc815a890 2024-03-18T20:29:58.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fcdcb8700 1 --2- 172.21.15.1:0/1236028186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7fc806f420 0x7f7fc8149c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:29:58.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fce4b9700 1 --2- 172.21.15.1:0/1236028186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fc815a890 0x7f7fc8152f60 secure :-1 s=READY pgs=3421 cs=0 l=1 rev1=1 crypto rx=0x7f7fc000b730 tx=0x7f7fc0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:58.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fbe7fc700 1 -- 172.21.15.1:0/1236028186 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc00099b0 con 0x7f7fc815a890 2024-03-18T20:29:58.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fbe7fc700 1 -- 172.21.15.1:0/1236028186 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc00092e0 con 0x7f7fc815a890 2024-03-18T20:29:58.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fcff1c700 1 -- 172.21.15.1:0/1236028186 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc8149130 con 0x7f7fc815a890 2024-03-18T20:29:58.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.834+0000 7f7fbe7fc700 1 -- 172.21.15.1:0/1236028186 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc001c660 con 0x7f7fc815a890 2024-03-18T20:29:58.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.835+0000 7f7fcff1c700 1 -- 172.21.15.1:0/1236028186 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc81496f0 con 0x7f7fc815a890 2024-03-18T20:29:58.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.835+0000 7f7fcff1c700 1 -- 172.21.15.1:0/1236028186 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7fc806dc10 con 0x7f7fc815a890 2024-03-18T20:29:58.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.836+0000 7f7fbe7fc700 1 -- 172.21.15.1:0/1236028186 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f7fc001c7c0 con 0x7f7fc815a890 2024-03-18T20:29:58.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.836+0000 7f7fbe7fc700 1 --2- 172.21.15.1:0/1236028186 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fac06adb0 0x7f7fac06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:58.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.836+0000 7f7fbe7fc700 1 -- 172.21.15.1:0/1236028186 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f7fc008b740 con 0x7f7fc815a890 2024-03-18T20:29:58.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.836+0000 7f7fcdcb8700 1 --2- 172.21.15.1:0/1236028186 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fac06adb0 0x7f7fac06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:58.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.837+0000 7f7fbe7fc700 1 --2- 172.21.15.1:0/1236028186 >> [v2:172.21.15.1:6824/77296,v1:172.21.15.1:6825/77296] conn(0x7f7fac071750 0x7f7fac073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:58.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.837+0000 7f7fbe7fc700 1 -- 172.21.15.1:0/1236028186 --> [v2:172.21.15.1:6824/77296,v1:172.21.15.1:6825/77296] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7fac0742a0 con 0x7f7fac071750 2024-03-18T20:29:58.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.837+0000 7f7fbe7fc700 1 -- 172.21.15.1:0/1236028186 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=295) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7fc008bb90 con 0x7f7fc815a890 2024-03-18T20:29:58.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.837+0000 7f7fcd4b7700 1 --2- 172.21.15.1:0/1236028186 >> [v2:172.21.15.1:6824/77296,v1:172.21.15.1:6825/77296] conn(0x7f7fac071750 0x7f7fac073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:58.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.837+0000 7f7fcdcb8700 1 --2- 172.21.15.1:0/1236028186 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fac06adb0 0x7f7fac06d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8000f60 tx=0x7f7fb8007df0 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:58.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.837+0000 7f7fcd4b7700 1 --2- 172.21.15.1:0/1236028186 >> [v2:172.21.15.1:6824/77296,v1:172.21.15.1:6825/77296] conn(0x7f7fac071750 0x7f7fac073bd0 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:58.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.838+0000 7f7fbe7fc700 1 -- 172.21.15.1:0/1236028186 <== osd.0 v2:172.21.15.1:6824/77296 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7fac0742a0 con 0x7f7fac071750 2024-03-18T20:29:58.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.852+0000 7f7fcff1c700 1 -- 172.21.15.1:0/1236028186 --> [v2:172.21.15.1:6824/77296,v1:172.21.15.1:6825/77296] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7fc80d0e00 con 0x7f7fac071750 2024-03-18T20:29:58.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.852+0000 7f7fbe7fc700 1 -- 172.21.15.1:0/1236028186 <== osd.0 v2:172.21.15.1:6824/77296 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f7fc80d0e00 con 0x7f7fac071750 2024-03-18T20:29:58.856 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:29:58.856 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T20:29:58.857 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:29:58.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.852+0000 7f7fcff1c700 1 -- 172.21.15.1:0/1236028186 >> [v2:172.21.15.1:6824/77296,v1:172.21.15.1:6825/77296] conn(0x7f7fac071750 msgr2=0x7f7fac073bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.852+0000 7f7fcff1c700 1 --2- 172.21.15.1:0/1236028186 >> [v2:172.21.15.1:6824/77296,v1:172.21.15.1:6825/77296] conn(0x7f7fac071750 0x7f7fac073bd0 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.853+0000 7f7fcff1c700 1 -- 172.21.15.1:0/1236028186 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fac06adb0 msgr2=0x7f7fac06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.853+0000 7f7fcff1c700 1 --2- 172.21.15.1:0/1236028186 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f7fac06adb0 0x7f7fac06d270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8000f60 tx=0x7f7fb8007df0 comp rx=0 tx=0).stop 2024-03-18T20:29:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.853+0000 7f7fcff1c700 1 -- 172.21.15.1:0/1236028186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fc815a890 msgr2=0x7f7fc8152f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.853+0000 7f7fcff1c700 1 --2- 172.21.15.1:0/1236028186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fc815a890 0x7f7fc8152f60 secure :-1 s=READY pgs=3421 cs=0 l=1 rev1=1 crypto rx=0x7f7fc000b730 tx=0x7f7fc0004ad0 comp rx=0 tx=0).stop 2024-03-18T20:29:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.853+0000 7f7fce4b9700 1 -- 172.21.15.1:0/1236028186 reap_dead start 2024-03-18T20:29:58.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.853+0000 7f7fcff1c700 1 -- 172.21.15.1:0/1236028186 shutdown_connections 2024-03-18T20:29:58.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.853+0000 7f7fcff1c700 1 -- 172.21.15.1:0/1236028186 >> 172.21.15.1:0/1236028186 conn(0x7f7fc80b5910 msgr2=0x7f7fc815cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:58.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.853+0000 7f7fcff1c700 1 -- 172.21.15.1:0/1236028186 shutdown_connections 2024-03-18T20:29:58.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.853+0000 7f7fcff1c700 1 -- 172.21.15.1:0/1236028186 wait complete. 2024-03-18T20:29:58.868 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-18T20:29:58.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.990+0000 7fa30b420700 1 Processor -- start 2024-03-18T20:29:58.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.990+0000 7fa30b420700 1 -- start start 2024-03-18T20:29:58.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.991+0000 7fa30b420700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa3040b0f50 0x7fa3040af680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:58.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.991+0000 7fa30b420700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3040b1a40 0x7fa3040b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:58.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.991+0000 7fa30b420700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3040b2a50 con 0x7fa304099a30 2024-03-18T20:29:58.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.991+0000 7fa30b420700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3040aeba0 con 0x7fa3040b0f50 2024-03-18T20:29:58.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.991+0000 7fa30b420700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3040aee80 con 0x7fa3040b1a40 2024-03-18T20:29:58.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.991+0000 7fa3099bd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3040b1a40 0x7fa3040b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:58.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.991+0000 7fa3091bc700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa3040b0f50 0x7fa3040af680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:58.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.991+0000 7fa3099bd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3040b1a40 0x7fa3040b32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:60074/0 (socket says 172.21.15.1:60074) 2024-03-18T20:29:58.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.991+0000 7fa3091bc700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa3040b0f50 0x7fa3040af680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.1:59332/0 (socket says 172.21.15.1:59332) 2024-03-18T20:29:58.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.991+0000 7fa3099bd700 1 -- 172.21.15.1:0/3713847323 learned_addr learned my addr 172.21.15.1:0/3713847323 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:58.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.991+0000 7fa3089bb700 1 --1- 172.21.15.1:0/3713847323 >> v1:172.21.15.165:6789/0 conn(0x7fa304099a30 0x7fa3040b1330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.1:37192/0 (socket says 172.21.15.1:37192) 2024-03-18T20:29:58.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.992+0000 7fa3099bd700 1 -- 172.21.15.1:0/3713847323 >> v2:172.21.15.165:3300/0 conn(0x7fa3040b0f50 msgr2=0x7fa3040af680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:58.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.992+0000 7fa3099bd700 1 --2- 172.21.15.1:0/3713847323 >> v2:172.21.15.165:3300/0 conn(0x7fa3040b0f50 0x7fa3040af680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:58.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.992+0000 7fa3099bd700 1 -- 172.21.15.1:0/3713847323 >> v1:172.21.15.165:6789/0 conn(0x7fa304099a30 legacy=0x7fa3040b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:58.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.992+0000 7fa3099bd700 1 -- 172.21.15.1:0/3713847323 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3040af160 con 0x7fa3040b1a40 2024-03-18T20:29:58.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.992+0000 7fa3099bd700 1 --2- 172.21.15.1:0/3713847323 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3040b1a40 0x7fa3040b32d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc009e50 tx=0x7fa2fc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fbc187ecce96a19 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:58.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.992+0000 7fa2fbfff700 1 -- 172.21.15.1:0/3713847323 <== mon.2 v2:172.21.15.165:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc00e070 con 0x7fa3040b1a40 2024-03-18T20:29:58.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.992+0000 7fa2fbfff700 1 -- 172.21.15.1:0/3713847323 <== mon.2 v2:172.21.15.165:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2fc00b910 con 0x7fa3040b1a40 2024-03-18T20:29:58.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.992+0000 7fa2fbfff700 1 -- 172.21.15.1:0/3713847323 <== mon.2 v2:172.21.15.165:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc00bcc0 con 0x7fa3040b1a40 2024-03-18T20:29:59.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.993+0000 7fa30b420700 1 -- 172.21.15.1:0/3713847323 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3040b1a40 msgr2=0x7fa3040b32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.993+0000 7fa30b420700 1 --2- 172.21.15.1:0/3713847323 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3040b1a40 0x7fa3040b32d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc009e50 tx=0x7fa2fc00b330 comp rx=0 tx=0).stop 2024-03-18T20:29:59.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.993+0000 7fa30b420700 1 -- 172.21.15.1:0/3713847323 shutdown_connections 2024-03-18T20:29:59.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.993+0000 7fa30b420700 1 --2- 172.21.15.1:0/3713847323 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3040b1a40 0x7fa3040b32d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.993+0000 7fa30b420700 1 --2- 172.21.15.1:0/3713847323 >> v2:172.21.15.165:3300/0 conn(0x7fa3040b0f50 0x7fa3040af680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.993+0000 7fa30b420700 1 -- 172.21.15.1:0/3713847323 >> 172.21.15.1:0/3713847323 conn(0x7fa3040b5a10 msgr2=0x7fa3040b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:59.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.993+0000 7fa30b420700 1 -- 172.21.15.1:0/3713847323 shutdown_connections 2024-03-18T20:29:59.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.993+0000 7fa30b420700 1 -- 172.21.15.1:0/3713847323 wait complete. 2024-03-18T20:29:59.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.994+0000 7fa30b420700 1 Processor -- start 2024-03-18T20:29:59.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.994+0000 7fa30b420700 1 -- start start 2024-03-18T20:29:59.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.995+0000 7fa30b420700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa304099a30 0x7fa304150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.995+0000 7fa30b420700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3040b0f50 0x7fa30413e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.995+0000 7fa30b420700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa3040b1a40 0x7fa30413ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.995+0000 7fa30b420700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3040ceb70 con 0x7fa304099a30 2024-03-18T20:29:59.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.995+0000 7fa30b420700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3040cee50 con 0x7fa3040b1a40 2024-03-18T20:29:59.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.995+0000 7fa30b420700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3040cf130 con 0x7fa3040b0f50 2024-03-18T20:29:59.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.995+0000 7fa3091bc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa304099a30 0x7fa304150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.995+0000 7fa3091bc700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa304099a30 0x7fa304150230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50112/0 (socket says 172.21.15.1:50112) 2024-03-18T20:29:59.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.995+0000 7fa3091bc700 1 -- 172.21.15.1:0/72443345 learned_addr learned my addr 172.21.15.1:0/72443345 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:59.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.995+0000 7fa3099bd700 1 --2- 172.21.15.1:0/72443345 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa3040b1a40 0x7fa30413ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.995+0000 7fa3089bb700 1 --2- 172.21.15.1:0/72443345 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3040b0f50 0x7fa30413e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.995+0000 7fa3091bc700 1 -- 172.21.15.1:0/72443345 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3040b0f50 msgr2=0x7fa30413e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.995+0000 7fa3091bc700 1 --2- 172.21.15.1:0/72443345 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3040b0f50 0x7fa30413e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.995+0000 7fa3091bc700 1 -- 172.21.15.1:0/72443345 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa3040b1a40 msgr2=0x7fa30413ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.995+0000 7fa3091bc700 1 --2- 172.21.15.1:0/72443345 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa3040b1a40 0x7fa30413ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.995+0000 7fa3091bc700 1 -- 172.21.15.1:0/72443345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2fc009b30 con 0x7fa304099a30 2024-03-18T20:29:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.995+0000 7fa3091bc700 1 --2- 172.21.15.1:0/72443345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa304099a30 0x7fa304150230 secure :-1 s=READY pgs=3422 cs=0 l=1 rev1=1 crypto rx=0x7fa30000e9b0 tx=0x7fa30000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.995+0000 7fa2f9ffb700 1 -- 172.21.15.1:0/72443345 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa300009710 con 0x7fa304099a30 2024-03-18T20:29:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.996+0000 7fa2f9ffb700 1 -- 172.21.15.1:0/72443345 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30000ee50 con 0x7fa304099a30 2024-03-18T20:29:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.996+0000 7fa30b420700 1 -- 172.21.15.1:0/72443345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3040cf410 con 0x7fa304099a30 2024-03-18T20:29:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.996+0000 7fa2f9ffb700 1 -- 172.21.15.1:0/72443345 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30000ca50 con 0x7fa304099a30 2024-03-18T20:29:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.996+0000 7fa30b420700 1 -- 172.21.15.1:0/72443345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3040ce800 con 0x7fa304099a30 2024-03-18T20:29:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.997+0000 7fa30b420700 1 -- 172.21.15.1:0/72443345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa30415b020 con 0x7fa304099a30 2024-03-18T20:29:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.997+0000 7fa2f9ffb700 1 -- 172.21.15.1:0/72443345 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa30000cbb0 con 0x7fa304099a30 2024-03-18T20:29:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.998+0000 7fa2f9ffb700 1 --2- 172.21.15.1:0/72443345 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa2e806adb0 0x7fa2e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.998+0000 7fa2f9ffb700 1 -- 172.21.15.1:0/72443345 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7fa3000895c0 con 0x7fa304099a30 2024-03-18T20:29:59.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.998+0000 7fa3089bb700 1 --2- 172.21.15.1:0/72443345 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa2e806adb0 0x7fa2e806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:58.998+0000 7fa3089bb700 1 --2- 172.21.15.1:0/72443345 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa2e806adb0 0x7fa2e806d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa3041f4380 tx=0x7fa2f4007ed0 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:59.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.003+0000 7fa2f9ffb700 1 -- 172.21.15.1:0/72443345 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa30415b020 con 0x7fa304099a30 2024-03-18T20:29:59.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.212+0000 7fa30b420700 1 -- 172.21.15.1:0/72443345 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3040b26e0 con 0x7fa2e806adb0 2024-03-18T20:29:59.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.212+0000 7fa2f9ffb700 1 -- 172.21.15.1:0/72443345 <== mgr.14091 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa2f40133f0 con 0x7fa2e806adb0 2024-03-18T20:29:59.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.215+0000 7fa30b420700 1 -- 172.21.15.1:0/72443345 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa2e806adb0 msgr2=0x7fa2e806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.215+0000 7fa30b420700 1 --2- 172.21.15.1:0/72443345 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa2e806adb0 0x7fa2e806d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa3041f4380 tx=0x7fa2f4007ed0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.215+0000 7fa30b420700 1 -- 172.21.15.1:0/72443345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa304099a30 msgr2=0x7fa304150230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.215+0000 7fa30b420700 1 --2- 172.21.15.1:0/72443345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa304099a30 0x7fa304150230 secure :-1 s=READY pgs=3422 cs=0 l=1 rev1=1 crypto rx=0x7fa30000e9b0 tx=0x7fa30000ecc0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.215+0000 7fa30b420700 1 -- 172.21.15.1:0/72443345 shutdown_connections 2024-03-18T20:29:59.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.215+0000 7fa30b420700 1 --2- 172.21.15.1:0/72443345 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa2e806adb0 0x7fa2e806d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.215+0000 7fa30b420700 1 --2- 172.21.15.1:0/72443345 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa3040b1a40 0x7fa30413ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.215+0000 7fa30b420700 1 --2- 172.21.15.1:0/72443345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa304099a30 0x7fa304150230 unknown :-1 s=CLOSED pgs=3422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.215+0000 7fa30b420700 1 --2- 172.21.15.1:0/72443345 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa3040b0f50 0x7fa30413e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.215+0000 7fa30b420700 1 -- 172.21.15.1:0/72443345 >> 172.21.15.1:0/72443345 conn(0x7fa3040b5a10 msgr2=0x7fa3040b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:59.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.215+0000 7fa30b420700 1 -- 172.21.15.1:0/72443345 shutdown_connections 2024-03-18T20:29:59.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.215+0000 7fa30b420700 1 -- 172.21.15.1:0/72443345 wait complete. 2024-03-18T20:29:59.221 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 0 to deep-scrub 2024-03-18T20:29:59.231 INFO:tasks.ceph:Scrubbing osd.1 2024-03-18T20:29:59.231 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-18T20:29:59.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.357+0000 7f9135efa700 1 Processor -- start 2024-03-18T20:29:59.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.357+0000 7f9135efa700 1 -- start start 2024-03-18T20:29:59.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.358+0000 7f9135efa700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91301441d0 0x7f91300d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.358+0000 7f9135efa700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f91300b4b10 0x7f91300b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.358+0000 7f9135efa700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f913015efb0 0x7f913015f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.358+0000 7f9135efa700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91300d0fe0 con 0x7f913015efb0 2024-03-18T20:29:59.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.358+0000 7f9135efa700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91300b54d0 con 0x7f91300b4b10 2024-03-18T20:29:59.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.358+0000 7f9135efa700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913006b0e0 con 0x7f91301441d0 2024-03-18T20:29:59.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.358+0000 7f912ffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f913015efb0 0x7f913015f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.358+0000 7f912ffff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f913015efb0 0x7f913015f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50120/0 (socket says 172.21.15.1:50120) 2024-03-18T20:29:59.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.358+0000 7f912ffff700 1 -- 172.21.15.1:0/3990542702 learned_addr learned my addr 172.21.15.1:0/3990542702 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:59.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.358+0000 7f912f7fe700 1 --2- 172.21.15.1:0/3990542702 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91301441d0 0x7f91300d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.358+0000 7f912ffff700 1 -- 172.21.15.1:0/3990542702 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91301441d0 msgr2=0x7f91300d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:59.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.358+0000 7f912effd700 1 --2- 172.21.15.1:0/3990542702 >> v2:172.21.15.165:3300/0 conn(0x7f91300b4b10 0x7f91300b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.358+0000 7f912ffff700 1 --2- 172.21.15.1:0/3990542702 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91301441d0 0x7f91300d0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.358+0000 7f912ffff700 1 -- 172.21.15.1:0/3990542702 >> v2:172.21.15.165:3300/0 conn(0x7f91300b4b10 msgr2=0x7f91300b4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:59.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.358+0000 7f912ffff700 1 --2- 172.21.15.1:0/3990542702 >> v2:172.21.15.165:3300/0 conn(0x7f91300b4b10 0x7f91300b4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.358+0000 7f912ffff700 1 -- 172.21.15.1:0/3990542702 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91301445b0 con 0x7f913015efb0 2024-03-18T20:29:59.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.358+0000 7f912f7fe700 1 --2- 172.21.15.1:0/3990542702 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91301441d0 0x7f91300d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:29:59.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.358+0000 7f912effd700 1 --2- 172.21.15.1:0/3990542702 >> v2:172.21.15.165:3300/0 conn(0x7f91300b4b10 0x7f91300b4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:29:59.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.358+0000 7f912ffff700 1 --2- 172.21.15.1:0/3990542702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f913015efb0 0x7f913015f420 secure :-1 s=READY pgs=3423 cs=0 l=1 rev1=1 crypto rx=0x7f91300d1200 tx=0x7f912800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=422ce4e01a1a133a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:59.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.359+0000 7f912e7fc700 1 -- 172.21.15.1:0/3990542702 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912800e070 con 0x7f913015efb0 2024-03-18T20:29:59.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.359+0000 7f912e7fc700 1 -- 172.21.15.1:0/3990542702 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912800b9c0 con 0x7f913015efb0 2024-03-18T20:29:59.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.359+0000 7f912e7fc700 1 -- 172.21.15.1:0/3990542702 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912800bd70 con 0x7f913015efb0 2024-03-18T20:29:59.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.359+0000 7f9135efa700 1 -- 172.21.15.1:0/3990542702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f913015efb0 msgr2=0x7f913015f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.359+0000 7f9135efa700 1 --2- 172.21.15.1:0/3990542702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f913015efb0 0x7f913015f420 secure :-1 s=READY pgs=3423 cs=0 l=1 rev1=1 crypto rx=0x7f91300d1200 tx=0x7f912800b330 comp rx=0 tx=0).stop 2024-03-18T20:29:59.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.359+0000 7f9135efa700 1 -- 172.21.15.1:0/3990542702 shutdown_connections 2024-03-18T20:29:59.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.359+0000 7f9135efa700 1 --2- 172.21.15.1:0/3990542702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f913015efb0 0x7f913015f420 unknown :-1 s=CLOSED pgs=3423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.359+0000 7f9135efa700 1 --2- 172.21.15.1:0/3990542702 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f91301441d0 0x7f91300d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.359+0000 7f9135efa700 1 --2- 172.21.15.1:0/3990542702 >> v2:172.21.15.165:3300/0 conn(0x7f91300b4b10 0x7f91300b4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.359+0000 7f9135efa700 1 -- 172.21.15.1:0/3990542702 >> 172.21.15.1:0/3990542702 conn(0x7f9130099a30 msgr2=0x7f91300b5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:59.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.359+0000 7f9135efa700 1 -- 172.21.15.1:0/3990542702 shutdown_connections 2024-03-18T20:29:59.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.359+0000 7f9135efa700 1 -- 172.21.15.1:0/3990542702 wait complete. 2024-03-18T20:29:59.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.361+0000 7f9135efa700 1 Processor -- start 2024-03-18T20:29:59.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.361+0000 7f9135efa700 1 -- start start 2024-03-18T20:29:59.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.362+0000 7f9135efa700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91300b4b10 0x7f913009d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.362+0000 7f9135efa700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91301441d0 0x7f91301420f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.362+0000 7f9135efa700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f913015efb0 0x7f9130142630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.362+0000 7f9135efa700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9130141970 con 0x7f91300b4b10 2024-03-18T20:29:59.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.362+0000 7f9135efa700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91300b2cf0 con 0x7f91301441d0 2024-03-18T20:29:59.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.362+0000 7f9135efa700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913009ded0 con 0x7f913015efb0 2024-03-18T20:29:59.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.362+0000 7f912f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91300b4b10 0x7f913009d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.362+0000 7f912f7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91300b4b10 0x7f913009d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50132/0 (socket says 172.21.15.1:50132) 2024-03-18T20:29:59.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.362+0000 7f912f7fe700 1 -- 172.21.15.1:0/3744499890 learned_addr learned my addr 172.21.15.1:0/3744499890 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:59.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.362+0000 7f912effd700 1 --2- 172.21.15.1:0/3744499890 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91301441d0 0x7f91301420f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.362+0000 7f912ffff700 1 --2- 172.21.15.1:0/3744499890 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f913015efb0 0x7f9130142630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.362+0000 7f912f7fe700 1 -- 172.21.15.1:0/3744499890 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f913015efb0 msgr2=0x7f9130142630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.362+0000 7f912f7fe700 1 --2- 172.21.15.1:0/3744499890 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f913015efb0 0x7f9130142630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.362+0000 7f912f7fe700 1 -- 172.21.15.1:0/3744499890 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91301441d0 msgr2=0x7f91301420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.362+0000 7f912f7fe700 1 --2- 172.21.15.1:0/3744499890 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91301441d0 0x7f91301420f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.362+0000 7f912f7fe700 1 -- 172.21.15.1:0/3744499890 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9128009b30 con 0x7f91300b4b10 2024-03-18T20:29:59.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.362+0000 7f912effd700 1 --2- 172.21.15.1:0/3744499890 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91301441d0 0x7f91301420f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:29:59.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.363+0000 7f912f7fe700 1 --2- 172.21.15.1:0/3744499890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91300b4b10 0x7f913009d990 secure :-1 s=READY pgs=3424 cs=0 l=1 rev1=1 crypto rx=0x7f912000c750 tx=0x7f912000cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:59.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.363+0000 7f9127fff700 1 -- 172.21.15.1:0/3744499890 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120011800 con 0x7f91300b4b10 2024-03-18T20:29:59.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.363+0000 7f9127fff700 1 -- 172.21.15.1:0/3744499890 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9120011960 con 0x7f91300b4b10 2024-03-18T20:29:59.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.363+0000 7f9135efa700 1 -- 172.21.15.1:0/3744499890 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91301410d0 con 0x7f91300b4b10 2024-03-18T20:29:59.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.363+0000 7f9127fff700 1 -- 172.21.15.1:0/3744499890 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120011cd0 con 0x7f91300b4b10 2024-03-18T20:29:59.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.363+0000 7f9135efa700 1 -- 172.21.15.1:0/3744499890 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9130141690 con 0x7f91300b4b10 2024-03-18T20:29:59.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.364+0000 7f9135efa700 1 -- 172.21.15.1:0/3744499890 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f913006aa90 con 0x7f91300b4b10 2024-03-18T20:29:59.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.364+0000 7f9127fff700 1 -- 172.21.15.1:0/3744499890 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f9120021350 con 0x7f91300b4b10 2024-03-18T20:29:59.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.365+0000 7f9127fff700 1 --2- 172.21.15.1:0/3744499890 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f911006adb0 0x7f911006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.365+0000 7f9127fff700 1 -- 172.21.15.1:0/3744499890 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f9120089650 con 0x7f91300b4b10 2024-03-18T20:29:59.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.365+0000 7f912effd700 1 --2- 172.21.15.1:0/3744499890 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f911006adb0 0x7f911006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.365+0000 7f9127fff700 1 --2- 172.21.15.1:0/3744499890 >> [v2:172.21.15.1:6800/77295,v1:172.21.15.1:6801/77295] conn(0x7f9110071750 0x7f9110073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.365+0000 7f9127fff700 1 -- 172.21.15.1:0/3744499890 --> [v2:172.21.15.1:6800/77295,v1:172.21.15.1:6801/77295] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91100742a0 con 0x7f9110071750 2024-03-18T20:29:59.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.365+0000 7f9127fff700 1 -- 172.21.15.1:0/3744499890 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=295) v2 ==== 24+0+0 (secure 0 0 0) 0x7f912008bd10 con 0x7f91300b4b10 2024-03-18T20:29:59.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.365+0000 7f912ffff700 1 --2- 172.21.15.1:0/3744499890 >> [v2:172.21.15.1:6800/77295,v1:172.21.15.1:6801/77295] conn(0x7f9110071750 0x7f9110073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.365+0000 7f912effd700 1 --2- 172.21.15.1:0/3744499890 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f911006adb0 0x7f911006d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f913009ae40 tx=0x7f9118008040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:59.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.365+0000 7f912ffff700 1 --2- 172.21.15.1:0/3744499890 >> [v2:172.21.15.1:6800/77295,v1:172.21.15.1:6801/77295] conn(0x7f9110071750 0x7f9110073bd0 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:59.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.366+0000 7f9127fff700 1 -- 172.21.15.1:0/3744499890 <== osd.1 v2:172.21.15.1:6800/77295 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f91100742a0 con 0x7f9110071750 2024-03-18T20:29:59.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.380+0000 7f9135efa700 1 -- 172.21.15.1:0/3744499890 --> [v2:172.21.15.1:6800/77295,v1:172.21.15.1:6801/77295] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f91300b6150 con 0x7f9110071750 2024-03-18T20:29:59.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.380+0000 7f9127fff700 1 -- 172.21.15.1:0/3744499890 <== osd.1 v2:172.21.15.1:6800/77295 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f9128003a20 con 0x7f9110071750 2024-03-18T20:29:59.384 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:29:59.385 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T20:29:59.385 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:29:59.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.380+0000 7f9135efa700 1 -- 172.21.15.1:0/3744499890 >> [v2:172.21.15.1:6800/77295,v1:172.21.15.1:6801/77295] conn(0x7f9110071750 msgr2=0x7f9110073bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.380+0000 7f9135efa700 1 --2- 172.21.15.1:0/3744499890 >> [v2:172.21.15.1:6800/77295,v1:172.21.15.1:6801/77295] conn(0x7f9110071750 0x7f9110073bd0 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.381+0000 7f9135efa700 1 -- 172.21.15.1:0/3744499890 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f911006adb0 msgr2=0x7f911006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.381+0000 7f9135efa700 1 --2- 172.21.15.1:0/3744499890 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f911006adb0 0x7f911006d270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f913009ae40 tx=0x7f9118008040 comp rx=0 tx=0).stop 2024-03-18T20:29:59.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.381+0000 7f9135efa700 1 -- 172.21.15.1:0/3744499890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91300b4b10 msgr2=0x7f913009d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.381+0000 7f9135efa700 1 --2- 172.21.15.1:0/3744499890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91300b4b10 0x7f913009d990 secure :-1 s=READY pgs=3424 cs=0 l=1 rev1=1 crypto rx=0x7f912000c750 tx=0x7f912000cb10 comp rx=0 tx=0).stop 2024-03-18T20:29:59.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.381+0000 7f912ffff700 1 -- 172.21.15.1:0/3744499890 reap_dead start 2024-03-18T20:29:59.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.381+0000 7f9135efa700 1 -- 172.21.15.1:0/3744499890 shutdown_connections 2024-03-18T20:29:59.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.381+0000 7f9135efa700 1 -- 172.21.15.1:0/3744499890 >> 172.21.15.1:0/3744499890 conn(0x7f9130099a30 msgr2=0x7f913015d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:59.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.381+0000 7f9135efa700 1 -- 172.21.15.1:0/3744499890 shutdown_connections 2024-03-18T20:29:59.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.381+0000 7f9135efa700 1 -- 172.21.15.1:0/3744499890 wait complete. 2024-03-18T20:29:59.396 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-18T20:29:59.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.518+0000 7fe785ec3700 1 Processor -- start 2024-03-18T20:29:59.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.518+0000 7fe785ec3700 1 -- start start 2024-03-18T20:29:59.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.519+0000 7fe785ec3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7800b0f50 0x7fe7800af680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.519+0000 7fe785ec3700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe780099a30 0x7fe7800b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.519+0000 7fe785ec3700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe7800b1870 0x7fe7800b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.519+0000 7fe785ec3700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7800b1d00 con 0x7fe780099a30 2024-03-18T20:29:59.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.519+0000 7fe785ec3700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7800b2a50 con 0x7fe7800b1870 2024-03-18T20:29:59.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.519+0000 7fe785ec3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7800aeba0 con 0x7fe7800b0f50 2024-03-18T20:29:59.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.519+0000 7fe77effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe780099a30 0x7fe7800b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.519+0000 7fe77effd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe780099a30 0x7fe7800b1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50140/0 (socket says 172.21.15.1:50140) 2024-03-18T20:29:59.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.519+0000 7fe77effd700 1 -- 172.21.15.1:0/1137769738 learned_addr learned my addr 172.21.15.1:0/1137769738 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:59.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.519+0000 7fe77f7fe700 1 --2- 172.21.15.1:0/1137769738 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7800b0f50 0x7fe7800af680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.519+0000 7fe77ffff700 1 --2- 172.21.15.1:0/1137769738 >> v2:172.21.15.165:3300/0 conn(0x7fe7800b1870 0x7fe7800b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.520+0000 7fe77effd700 1 -- 172.21.15.1:0/1137769738 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7800b0f50 msgr2=0x7fe7800af680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:59.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.520+0000 7fe77effd700 1 --2- 172.21.15.1:0/1137769738 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7800b0f50 0x7fe7800af680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.520+0000 7fe77effd700 1 -- 172.21.15.1:0/1137769738 >> v2:172.21.15.165:3300/0 conn(0x7fe7800b1870 msgr2=0x7fe7800b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:59.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.520+0000 7fe77effd700 1 --2- 172.21.15.1:0/1137769738 >> v2:172.21.15.165:3300/0 conn(0x7fe7800b1870 0x7fe7800b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.520+0000 7fe77effd700 1 -- 172.21.15.1:0/1137769738 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7800aee80 con 0x7fe780099a30 2024-03-18T20:29:59.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.520+0000 7fe77ffff700 1 --2- 172.21.15.1:0/1137769738 >> v2:172.21.15.165:3300/0 conn(0x7fe7800b1870 0x7fe7800b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:29:59.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.520+0000 7fe77effd700 1 --2- 172.21.15.1:0/1137769738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe780099a30 0x7fe7800b1330 secure :-1 s=READY pgs=3425 cs=0 l=1 rev1=1 crypto rx=0x7fe7800712a0 tx=0x7fe770009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5101897c33cd8b88 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:59.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.520+0000 7fe77e7fc700 1 -- 172.21.15.1:0/1137769738 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77000b870 con 0x7fe780099a30 2024-03-18T20:29:59.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.520+0000 7fe77e7fc700 1 -- 172.21.15.1:0/1137769738 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe77000b9d0 con 0x7fe780099a30 2024-03-18T20:29:59.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.520+0000 7fe77e7fc700 1 -- 172.21.15.1:0/1137769738 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77000bd60 con 0x7fe780099a30 2024-03-18T20:29:59.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.521+0000 7fe785ec3700 1 -- 172.21.15.1:0/1137769738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe780099a30 msgr2=0x7fe7800b1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.521+0000 7fe785ec3700 1 --2- 172.21.15.1:0/1137769738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe780099a30 0x7fe7800b1330 secure :-1 s=READY pgs=3425 cs=0 l=1 rev1=1 crypto rx=0x7fe7800712a0 tx=0x7fe770009bd0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.521+0000 7fe785ec3700 1 -- 172.21.15.1:0/1137769738 shutdown_connections 2024-03-18T20:29:59.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.521+0000 7fe785ec3700 1 --2- 172.21.15.1:0/1137769738 >> v2:172.21.15.165:3300/0 conn(0x7fe7800b1870 0x7fe7800b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.521+0000 7fe785ec3700 1 --2- 172.21.15.1:0/1137769738 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7800b0f50 0x7fe7800af680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.521+0000 7fe785ec3700 1 --2- 172.21.15.1:0/1137769738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe780099a30 0x7fe7800b1330 unknown :-1 s=CLOSED pgs=3425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.521+0000 7fe785ec3700 1 -- 172.21.15.1:0/1137769738 >> 172.21.15.1:0/1137769738 conn(0x7fe7800b5a10 msgr2=0x7fe7800b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:59.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.521+0000 7fe785ec3700 1 -- 172.21.15.1:0/1137769738 shutdown_connections 2024-03-18T20:29:59.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.521+0000 7fe785ec3700 1 -- 172.21.15.1:0/1137769738 wait complete. 2024-03-18T20:29:59.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.522+0000 7fe785ec3700 1 Processor -- start 2024-03-18T20:29:59.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.522+0000 7fe785ec3700 1 -- start start 2024-03-18T20:29:59.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.523+0000 7fe785ec3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe780099a30 0x7fe78014dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.523+0000 7fe785ec3700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7800b0f50 0x7fe7801e4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.523+0000 7fe785ec3700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7800b1870 0x7fe7800ce9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.523+0000 7fe785ec3700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe780150ce0 con 0x7fe7800b1870 2024-03-18T20:29:59.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.523+0000 7fe785ec3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78014e530 con 0x7fe780099a30 2024-03-18T20:29:59.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.523+0000 7fe785ec3700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe780150060 con 0x7fe7800b0f50 2024-03-18T20:29:59.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.523+0000 7fe77ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7800b1870 0x7fe7800ce9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.523+0000 7fe77ffff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7800b1870 0x7fe7800ce9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50142/0 (socket says 172.21.15.1:50142) 2024-03-18T20:29:59.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.524+0000 7fe77ffff700 1 -- 172.21.15.1:0/3623281508 learned_addr learned my addr 172.21.15.1:0/3623281508 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:59.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.524+0000 7fe77f7fe700 1 --2- 172.21.15.1:0/3623281508 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe780099a30 0x7fe78014dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.524+0000 7fe77effd700 1 --2- 172.21.15.1:0/3623281508 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7800b0f50 0x7fe7801e4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.524+0000 7fe77ffff700 1 -- 172.21.15.1:0/3623281508 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7800b0f50 msgr2=0x7fe7801e4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.524+0000 7fe77ffff700 1 --2- 172.21.15.1:0/3623281508 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7800b0f50 0x7fe7801e4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.524+0000 7fe77ffff700 1 -- 172.21.15.1:0/3623281508 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe780099a30 msgr2=0x7fe78014dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.524+0000 7fe77ffff700 1 --2- 172.21.15.1:0/3623281508 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe780099a30 0x7fe78014dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.524+0000 7fe77ffff700 1 -- 172.21.15.1:0/3623281508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe770009550 con 0x7fe7800b1870 2024-03-18T20:29:59.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.524+0000 7fe77effd700 1 --2- 172.21.15.1:0/3623281508 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7800b0f50 0x7fe7801e4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:29:59.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.524+0000 7fe77ffff700 1 --2- 172.21.15.1:0/3623281508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7800b1870 0x7fe7800ce9d0 secure :-1 s=READY pgs=3426 cs=0 l=1 rev1=1 crypto rx=0x7fe778007ef0 tx=0x7fe77800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:59.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.524+0000 7fe777fff700 1 -- 172.21.15.1:0/3623281508 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77800de00 con 0x7fe7800b1870 2024-03-18T20:29:59.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.524+0000 7fe777fff700 1 -- 172.21.15.1:0/3623281508 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe77800f040 con 0x7fe7800b1870 2024-03-18T20:29:59.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.524+0000 7fe785ec3700 1 -- 172.21.15.1:0/3623281508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe780150340 con 0x7fe7800b1870 2024-03-18T20:29:59.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.524+0000 7fe777fff700 1 -- 172.21.15.1:0/3623281508 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe778003c60 con 0x7fe7800b1870 2024-03-18T20:29:59.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.524+0000 7fe785ec3700 1 -- 172.21.15.1:0/3623281508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe780150900 con 0x7fe7800b1870 2024-03-18T20:29:59.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.525+0000 7fe785ec3700 1 -- 172.21.15.1:0/3623281508 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe780150e20 con 0x7fe7800b1870 2024-03-18T20:29:59.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.526+0000 7fe777fff700 1 -- 172.21.15.1:0/3623281508 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe778004030 con 0x7fe7800b1870 2024-03-18T20:29:59.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.526+0000 7fe777fff700 1 --2- 172.21.15.1:0/3623281508 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe76006adb0 0x7fe76006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.526+0000 7fe777fff700 1 -- 172.21.15.1:0/3623281508 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7fe778089890 con 0x7fe7800b1870 2024-03-18T20:29:59.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.526+0000 7fe77f7fe700 1 --2- 172.21.15.1:0/3623281508 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe76006adb0 0x7fe76006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.527+0000 7fe77f7fe700 1 --2- 172.21.15.1:0/3623281508 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe76006adb0 0x7fe76006d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe76800c900 tx=0x7fe768008040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:59.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.531+0000 7fe777fff700 1 -- 172.21.15.1:0/3623281508 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe780150e20 con 0x7fe7800b1870 2024-03-18T20:29:59.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.730+0000 7fe785ec3700 1 -- 172.21.15.1:0/3623281508 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7800b26e0 con 0x7fe76006adb0 2024-03-18T20:29:59.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.730+0000 7fe777fff700 1 -- 172.21.15.1:0/3623281508 <== mgr.14091 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe76800cdd0 con 0x7fe76006adb0 2024-03-18T20:29:59.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.732+0000 7fe785ec3700 1 -- 172.21.15.1:0/3623281508 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe76006adb0 msgr2=0x7fe76006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.732+0000 7fe785ec3700 1 --2- 172.21.15.1:0/3623281508 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe76006adb0 0x7fe76006d270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe76800c900 tx=0x7fe768008040 comp rx=0 tx=0).stop 2024-03-18T20:29:59.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.732+0000 7fe785ec3700 1 -- 172.21.15.1:0/3623281508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7800b1870 msgr2=0x7fe7800ce9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.732+0000 7fe785ec3700 1 --2- 172.21.15.1:0/3623281508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7800b1870 0x7fe7800ce9d0 secure :-1 s=READY pgs=3426 cs=0 l=1 rev1=1 crypto rx=0x7fe778007ef0 tx=0x7fe77800d3b0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.732+0000 7fe785ec3700 1 -- 172.21.15.1:0/3623281508 shutdown_connections 2024-03-18T20:29:59.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.732+0000 7fe785ec3700 1 --2- 172.21.15.1:0/3623281508 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe76006adb0 0x7fe76006d270 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.732+0000 7fe785ec3700 1 --2- 172.21.15.1:0/3623281508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7800b1870 0x7fe7800ce9d0 unknown :-1 s=CLOSED pgs=3426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.732+0000 7fe785ec3700 1 --2- 172.21.15.1:0/3623281508 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe780099a30 0x7fe78014dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.732+0000 7fe785ec3700 1 --2- 172.21.15.1:0/3623281508 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe7800b0f50 0x7fe7801e4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.732+0000 7fe785ec3700 1 -- 172.21.15.1:0/3623281508 >> 172.21.15.1:0/3623281508 conn(0x7fe7800b5a10 msgr2=0x7fe78006db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:59.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.733+0000 7fe785ec3700 1 -- 172.21.15.1:0/3623281508 shutdown_connections 2024-03-18T20:29:59.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.733+0000 7fe785ec3700 1 -- 172.21.15.1:0/3623281508 wait complete. 2024-03-18T20:29:59.738 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 1 to deep-scrub 2024-03-18T20:29:59.748 INFO:tasks.ceph:Scrubbing osd.2 2024-03-18T20:29:59.749 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-18T20:29:59.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.874+0000 7f419cce6700 1 Processor -- start 2024-03-18T20:29:59.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.874+0000 7f419cce6700 1 -- start start 2024-03-18T20:29:59.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.875+0000 7f419cce6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f41980cd000 0x7f419806e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.875+0000 7f419cce6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41980cd5c0 0x7f41980cda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.875+0000 7f419cce6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f41980d0140 0x7f41980d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.875+0000 7f419cce6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f419815ed80 con 0x7f41980d0140 2024-03-18T20:29:59.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.875+0000 7f419cce6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41980d0b10 con 0x7f41980cd000 2024-03-18T20:29:59.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.875+0000 7f419cce6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41980cd3e0 con 0x7f41980cd5c0 2024-03-18T20:29:59.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.875+0000 7f4196d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f41980d0140 0x7f41980d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.875+0000 7f4196d9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f41980d0140 0x7f41980d05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50144/0 (socket says 172.21.15.1:50144) 2024-03-18T20:29:59.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.875+0000 7f4196d9d700 1 -- 172.21.15.1:0/1020794559 learned_addr learned my addr 172.21.15.1:0/1020794559 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:59.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.875+0000 7f419659c700 1 --2- 172.21.15.1:0/1020794559 >> v2:172.21.15.165:3300/0 conn(0x7f41980cd000 0x7f419806e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.875+0000 7f4195d9b700 1 --2- 172.21.15.1:0/1020794559 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41980cd5c0 0x7f41980cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.875+0000 7f4196d9d700 1 -- 172.21.15.1:0/1020794559 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41980cd5c0 msgr2=0x7f41980cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:59.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.875+0000 7f4196d9d700 1 --2- 172.21.15.1:0/1020794559 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41980cd5c0 0x7f41980cda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.875+0000 7f4196d9d700 1 -- 172.21.15.1:0/1020794559 >> v2:172.21.15.165:3300/0 conn(0x7f41980cd000 msgr2=0x7f419806e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:29:59.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.875+0000 7f4196d9d700 1 --2- 172.21.15.1:0/1020794559 >> v2:172.21.15.165:3300/0 conn(0x7f41980cd000 0x7f419806e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.875+0000 7f4196d9d700 1 -- 172.21.15.1:0/1020794559 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419815e170 con 0x7f41980d0140 2024-03-18T20:29:59.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.875+0000 7f419659c700 1 --2- 172.21.15.1:0/1020794559 >> v2:172.21.15.165:3300/0 conn(0x7f41980cd000 0x7f419806e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:29:59.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.875+0000 7f4196d9d700 1 --2- 172.21.15.1:0/1020794559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41980d0140 0x7f41980d05d0 secure :-1 s=READY pgs=3427 cs=0 l=1 rev1=1 crypto rx=0x7f4198071e30 tx=0x7f419000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2623860533d9f6dd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:59.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.875+0000 7f419559a700 1 -- 172.21.15.1:0/1020794559 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419000e070 con 0x7f41980d0140 2024-03-18T20:29:59.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.876+0000 7f419559a700 1 -- 172.21.15.1:0/1020794559 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419000b9c0 con 0x7f41980d0140 2024-03-18T20:29:59.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.876+0000 7f419559a700 1 -- 172.21.15.1:0/1020794559 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419000bd70 con 0x7f41980d0140 2024-03-18T20:29:59.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.876+0000 7f419cce6700 1 -- 172.21.15.1:0/1020794559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41980d0140 msgr2=0x7f41980d05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.876+0000 7f419cce6700 1 --2- 172.21.15.1:0/1020794559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41980d0140 0x7f41980d05d0 secure :-1 s=READY pgs=3427 cs=0 l=1 rev1=1 crypto rx=0x7f4198071e30 tx=0x7f419000b330 comp rx=0 tx=0).stop 2024-03-18T20:29:59.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.876+0000 7f419cce6700 1 -- 172.21.15.1:0/1020794559 shutdown_connections 2024-03-18T20:29:59.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.876+0000 7f419cce6700 1 --2- 172.21.15.1:0/1020794559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41980d0140 0x7f41980d05d0 unknown :-1 s=CLOSED pgs=3427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.876+0000 7f419cce6700 1 --2- 172.21.15.1:0/1020794559 >> v2:172.21.15.165:3300/0 conn(0x7f41980cd000 0x7f419806e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.876+0000 7f419cce6700 1 --2- 172.21.15.1:0/1020794559 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41980cd5c0 0x7f41980cda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.876+0000 7f419cce6700 1 -- 172.21.15.1:0/1020794559 >> 172.21.15.1:0/1020794559 conn(0x7f4198099a30 msgr2=0x7f41980b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:59.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.876+0000 7f419cce6700 1 -- 172.21.15.1:0/1020794559 shutdown_connections 2024-03-18T20:29:59.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.876+0000 7f419cce6700 1 -- 172.21.15.1:0/1020794559 wait complete. 2024-03-18T20:29:59.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.877+0000 7f419cce6700 1 Processor -- start 2024-03-18T20:29:59.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.878+0000 7f419cce6700 1 -- start start 2024-03-18T20:29:59.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.878+0000 7f419cce6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41980cd000 0x7f4198159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.878+0000 7f419cce6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41980cd5c0 0x7f41980ca720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.878+0000 7f419cce6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41980d0140 0x7f419809b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.878+0000 7f419cce6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41980ca290 con 0x7f41980cd5c0 2024-03-18T20:29:59.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.878+0000 7f419cce6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41980c9680 con 0x7f41980d0140 2024-03-18T20:29:59.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.878+0000 7f419cce6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41980c9960 con 0x7f41980cd000 2024-03-18T20:29:59.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.878+0000 7f4195d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41980cd5c0 0x7f41980ca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.878+0000 7f4195d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41980cd5c0 0x7f41980ca720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50146/0 (socket says 172.21.15.1:50146) 2024-03-18T20:29:59.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.878+0000 7f4195d9b700 1 -- 172.21.15.1:0/2393370371 learned_addr learned my addr 172.21.15.1:0/2393370371 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:29:59.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.879+0000 7f4196d9d700 1 --2- 172.21.15.1:0/2393370371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41980d0140 0x7f419809b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.879+0000 7f419659c700 1 --2- 172.21.15.1:0/2393370371 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41980cd000 0x7f4198159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.879+0000 7f4195d9b700 1 -- 172.21.15.1:0/2393370371 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41980cd000 msgr2=0x7f4198159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.879+0000 7f4195d9b700 1 --2- 172.21.15.1:0/2393370371 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41980cd000 0x7f4198159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.879+0000 7f4195d9b700 1 -- 172.21.15.1:0/2393370371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41980d0140 msgr2=0x7f419809b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.879+0000 7f4195d9b700 1 --2- 172.21.15.1:0/2393370371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41980d0140 0x7f419809b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.879+0000 7f4195d9b700 1 -- 172.21.15.1:0/2393370371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4190009b30 con 0x7f41980cd5c0 2024-03-18T20:29:59.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.879+0000 7f419659c700 1 --2- 172.21.15.1:0/2393370371 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41980cd000 0x7f4198159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:29:59.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.879+0000 7f4195d9b700 1 --2- 172.21.15.1:0/2393370371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41980cd5c0 0x7f41980ca720 secure :-1 s=READY pgs=3428 cs=0 l=1 rev1=1 crypto rx=0x7f418c00d800 tx=0x7f418c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:59.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.879+0000 7f4186ffd700 1 -- 172.21.15.1:0/2393370371 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c0125f0 con 0x7f41980cd5c0 2024-03-18T20:29:59.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.879+0000 7f4186ffd700 1 -- 172.21.15.1:0/2393370371 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418c012750 con 0x7f41980cd5c0 2024-03-18T20:29:59.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.879+0000 7f419cce6700 1 -- 172.21.15.1:0/2393370371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41980c9c40 con 0x7f41980cd5c0 2024-03-18T20:29:59.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.879+0000 7f4186ffd700 1 -- 172.21.15.1:0/2393370371 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c012b00 con 0x7f41980cd5c0 2024-03-18T20:29:59.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.879+0000 7f419cce6700 1 -- 172.21.15.1:0/2393370371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f419809a5f0 con 0x7f41980cd5c0 2024-03-18T20:29:59.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.880+0000 7f419cce6700 1 -- 172.21.15.1:0/2393370371 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f419815ea10 con 0x7f41980cd5c0 2024-03-18T20:29:59.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.880+0000 7f4186ffd700 1 -- 172.21.15.1:0/2393370371 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f418c012c60 con 0x7f41980cd5c0 2024-03-18T20:29:59.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.881+0000 7f4186ffd700 1 --2- 172.21.15.1:0/2393370371 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f417806adb0 0x7f417806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.881+0000 7f4186ffd700 1 -- 172.21.15.1:0/2393370371 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f418c089980 con 0x7f41980cd5c0 2024-03-18T20:29:59.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.881+0000 7f419659c700 1 --2- 172.21.15.1:0/2393370371 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f417806adb0 0x7f417806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.881+0000 7f4186ffd700 1 --2- 172.21.15.1:0/2393370371 >> [v2:172.21.15.1:6808/77305,v1:172.21.15.1:6809/77305] conn(0x7f4178071750 0x7f4178073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:29:59.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.881+0000 7f4186ffd700 1 -- 172.21.15.1:0/2393370371 --> [v2:172.21.15.1:6808/77305,v1:172.21.15.1:6809/77305] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f41780742a0 con 0x7f4178071750 2024-03-18T20:29:59.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.881+0000 7f4186ffd700 1 -- 172.21.15.1:0/2393370371 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=295) v2 ==== 24+0+0 (secure 0 0 0) 0x7f418c08e2c0 con 0x7f41980cd5c0 2024-03-18T20:29:59.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.881+0000 7f4196d9d700 1 --2- 172.21.15.1:0/2393370371 >> [v2:172.21.15.1:6808/77305,v1:172.21.15.1:6809/77305] conn(0x7f4178071750 0x7f4178073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:29:59.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.881+0000 7f419659c700 1 --2- 172.21.15.1:0/2393370371 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f417806adb0 0x7f417806d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f41980cff20 tx=0x7f4188008040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:59.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.881+0000 7f4196d9d700 1 --2- 172.21.15.1:0/2393370371 >> [v2:172.21.15.1:6808/77305,v1:172.21.15.1:6809/77305] conn(0x7f4178071750 0x7f4178073bd0 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:29:59.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.882+0000 7f4186ffd700 1 -- 172.21.15.1:0/2393370371 <== osd.2 v2:172.21.15.1:6808/77305 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f41780742a0 con 0x7f4178071750 2024-03-18T20:29:59.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.896+0000 7f419cce6700 1 -- 172.21.15.1:0/2393370371 --> [v2:172.21.15.1:6808/77305,v1:172.21.15.1:6809/77305] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f419815e730 con 0x7f4178071750 2024-03-18T20:29:59.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.897+0000 7f4186ffd700 1 -- 172.21.15.1:0/2393370371 <== osd.2 v2:172.21.15.1:6808/77305 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f41900099b0 con 0x7f4178071750 2024-03-18T20:29:59.902 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:29:59.902 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T20:29:59.902 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:29:59.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.897+0000 7f419cce6700 1 -- 172.21.15.1:0/2393370371 >> [v2:172.21.15.1:6808/77305,v1:172.21.15.1:6809/77305] conn(0x7f4178071750 msgr2=0x7f4178073bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.897+0000 7f419cce6700 1 --2- 172.21.15.1:0/2393370371 >> [v2:172.21.15.1:6808/77305,v1:172.21.15.1:6809/77305] conn(0x7f4178071750 0x7f4178073bd0 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:29:59.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.897+0000 7f419cce6700 1 -- 172.21.15.1:0/2393370371 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f417806adb0 msgr2=0x7f417806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.897+0000 7f419cce6700 1 --2- 172.21.15.1:0/2393370371 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f417806adb0 0x7f417806d270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f41980cff20 tx=0x7f4188008040 comp rx=0 tx=0).stop 2024-03-18T20:29:59.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.897+0000 7f419cce6700 1 -- 172.21.15.1:0/2393370371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41980cd5c0 msgr2=0x7f41980ca720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:29:59.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.897+0000 7f419cce6700 1 --2- 172.21.15.1:0/2393370371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41980cd5c0 0x7f41980ca720 secure :-1 s=READY pgs=3428 cs=0 l=1 rev1=1 crypto rx=0x7f418c00d800 tx=0x7f418c00db10 comp rx=0 tx=0).stop 2024-03-18T20:29:59.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.897+0000 7f4196d9d700 1 -- 172.21.15.1:0/2393370371 reap_dead start 2024-03-18T20:29:59.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.897+0000 7f419cce6700 1 -- 172.21.15.1:0/2393370371 shutdown_connections 2024-03-18T20:29:59.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.897+0000 7f419cce6700 1 -- 172.21.15.1:0/2393370371 >> 172.21.15.1:0/2393370371 conn(0x7f4198099a30 msgr2=0x7f4198160860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:29:59.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.897+0000 7f419cce6700 1 -- 172.21.15.1:0/2393370371 shutdown_connections 2024-03-18T20:29:59.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:29:59.897+0000 7f419cce6700 1 -- 172.21.15.1:0/2393370371 wait complete. 2024-03-18T20:29:59.912 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-18T20:30:00.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.035+0000 7f5c620ba700 1 Processor -- start 2024-03-18T20:30:00.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.035+0000 7f5c620ba700 1 -- start start 2024-03-18T20:30:00.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.036+0000 7f5c620ba700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c5c06fc60 0x7f5c5c099a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.036+0000 7f5c620ba700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c5c0b5190 0x7f5c5c06bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.036+0000 7f5c620ba700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5c0b5800 con 0x7f5c5c099f70 2024-03-18T20:30:00.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.036+0000 7f5c620ba700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c5c09a420 con 0x7f5c5c06fc60 2024-03-18T20:30:00.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.036+0000 7f5c620ba700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c0b5620 con 0x7f5c5c0b5190 2024-03-18T20:30:00.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.036+0000 7f5c5b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c5c06fc60 0x7f5c5c099a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.036+0000 7f5c5b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5c5c06fc60 0x7f5c5c099a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50162/0 (socket says 172.21.15.1:50162) 2024-03-18T20:30:00.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.036+0000 7f5c5b7fe700 1 -- 172.21.15.1:0/1815128226 learned_addr learned my addr 172.21.15.1:0/1815128226 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:00.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.037+0000 7f5c5bfff700 1 --2- 172.21.15.1:0/1815128226 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c5c0b5190 0x7f5c5c06bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.037+0000 7f5c5b7fe700 1 -- 172.21.15.1:0/1815128226 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c5c0b5190 msgr2=0x7f5c5c06bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:00.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.037+0000 7f5c5b7fe700 1 --2- 172.21.15.1:0/1815128226 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c5c0b5190 0x7f5c5c06bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.037+0000 7f5c5b7fe700 1 -- 172.21.15.1:0/1815128226 >> v1:172.21.15.165:6789/0 conn(0x7f5c5c099f70 legacy=0x7f5c5c0b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.037+0000 7f5c5b7fe700 1 -- 172.21.15.1:0/1815128226 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5c0b5ae0 con 0x7f5c5c06fc60 2024-03-18T20:30:00.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.037+0000 7f5c5bfff700 1 --2- 172.21.15.1:0/1815128226 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c5c0b5190 0x7f5c5c06bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:30:00.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.037+0000 7f5c5b7fe700 1 --2- 172.21.15.1:0/1815128226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c5c06fc60 0x7f5c5c099a30 secure :-1 s=READY pgs=3429 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c0098f0 tx=0x7f5c4c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce1e11e377dbfa2d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:00.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.037+0000 7f5c5a7fc700 1 -- 172.21.15.1:0/1815128226 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c00b8a0 con 0x7f5c5c06fc60 2024-03-18T20:30:00.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.037+0000 7f5c5a7fc700 1 -- 172.21.15.1:0/1815128226 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4c00ba00 con 0x7f5c5c06fc60 2024-03-18T20:30:00.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.037+0000 7f5c5a7fc700 1 -- 172.21.15.1:0/1815128226 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c00bd90 con 0x7f5c5c06fc60 2024-03-18T20:30:00.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.037+0000 7f5c620ba700 1 -- 172.21.15.1:0/1815128226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c5c06fc60 msgr2=0x7f5c5c099a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.038+0000 7f5c620ba700 1 --2- 172.21.15.1:0/1815128226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c5c06fc60 0x7f5c5c099a30 secure :-1 s=READY pgs=3429 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c0098f0 tx=0x7f5c4c009c00 comp rx=0 tx=0).stop 2024-03-18T20:30:00.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.038+0000 7f5c620ba700 1 -- 172.21.15.1:0/1815128226 shutdown_connections 2024-03-18T20:30:00.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.038+0000 7f5c620ba700 1 --2- 172.21.15.1:0/1815128226 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c5c0b5190 0x7f5c5c06bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.038+0000 7f5c620ba700 1 --2- 172.21.15.1:0/1815128226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c5c06fc60 0x7f5c5c099a30 unknown :-1 s=CLOSED pgs=3429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.038+0000 7f5c620ba700 1 -- 172.21.15.1:0/1815128226 >> 172.21.15.1:0/1815128226 conn(0x7f5c5c0b2de0 msgr2=0x7f5c5c0b31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:00.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.038+0000 7f5c620ba700 1 -- 172.21.15.1:0/1815128226 shutdown_connections 2024-03-18T20:30:00.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.038+0000 7f5c620ba700 1 -- 172.21.15.1:0/1815128226 wait complete. 2024-03-18T20:30:00.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.039+0000 7f5c620ba700 1 Processor -- start 2024-03-18T20:30:00.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.039+0000 7f5c620ba700 1 -- start start 2024-03-18T20:30:00.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c620ba700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c5c06fc60 0x7f5c5c1407c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c620ba700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c5c099f70 0x7f5c5c159710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c620ba700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c5c0b5190 0x7f5c5c159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c620ba700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c1550d0 con 0x7f5c5c099f70 2024-03-18T20:30:00.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c620ba700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c155690 con 0x7f5c5c0b5190 2024-03-18T20:30:00.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c620ba700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c160620 con 0x7f5c5c06fc60 2024-03-18T20:30:00.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c5affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c5c099f70 0x7f5c5c159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c5affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c5c099f70 0x7f5c5c159710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50166/0 (socket says 172.21.15.1:50166) 2024-03-18T20:30:00.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c5affd700 1 -- 172.21.15.1:0/1123913466 learned_addr learned my addr 172.21.15.1:0/1123913466 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:00.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c5b7fe700 1 --2- 172.21.15.1:0/1123913466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c5c06fc60 0x7f5c5c1407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c5bfff700 1 --2- 172.21.15.1:0/1123913466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c5c0b5190 0x7f5c5c159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c5affd700 1 -- 172.21.15.1:0/1123913466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c5c06fc60 msgr2=0x7f5c5c1407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c5affd700 1 --2- 172.21.15.1:0/1123913466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c5c06fc60 0x7f5c5c1407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c5affd700 1 -- 172.21.15.1:0/1123913466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c5c0b5190 msgr2=0x7f5c5c159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c5affd700 1 --2- 172.21.15.1:0/1123913466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c5c0b5190 0x7f5c5c159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c5affd700 1 -- 172.21.15.1:0/1123913466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c009550 con 0x7f5c5c099f70 2024-03-18T20:30:00.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c5bfff700 1 --2- 172.21.15.1:0/1123913466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c5c0b5190 0x7f5c5c159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:30:00.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c5b7fe700 1 --2- 172.21.15.1:0/1123913466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c5c06fc60 0x7f5c5c1407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:30:00.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c5affd700 1 --2- 172.21.15.1:0/1123913466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c5c099f70 0x7f5c5c159710 secure :-1 s=READY pgs=3430 cs=0 l=1 rev1=1 crypto rx=0x7f5c4400c4e0 tx=0x7f5c4400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:00.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c53fff700 1 -- 172.21.15.1:0/1123913466 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c440052d0 con 0x7f5c5c099f70 2024-03-18T20:30:00.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c53fff700 1 -- 172.21.15.1:0/1123913466 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c44005430 con 0x7f5c5c099f70 2024-03-18T20:30:00.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c620ba700 1 -- 172.21.15.1:0/1123913466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5c160900 con 0x7f5c5c099f70 2024-03-18T20:30:00.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.040+0000 7f5c53fff700 1 -- 172.21.15.1:0/1123913466 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c440038c0 con 0x7f5c5c099f70 2024-03-18T20:30:00.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.041+0000 7f5c620ba700 1 -- 172.21.15.1:0/1123913466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5c15f850 con 0x7f5c5c099f70 2024-03-18T20:30:00.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.042+0000 7f5c620ba700 1 -- 172.21.15.1:0/1123913466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c5c160af0 con 0x7f5c5c099f70 2024-03-18T20:30:00.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.042+0000 7f5c53fff700 1 -- 172.21.15.1:0/1123913466 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5c44003a20 con 0x7f5c5c099f70 2024-03-18T20:30:00.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.042+0000 7f5c53fff700 1 --2- 172.21.15.1:0/1123913466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5c3c06adb0 0x7f5c3c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.042+0000 7f5c53fff700 1 -- 172.21.15.1:0/1123913466 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f5c44088ad0 con 0x7f5c5c099f70 2024-03-18T20:30:00.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.042+0000 7f5c5b7fe700 1 --2- 172.21.15.1:0/1123913466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5c3c06adb0 0x7f5c3c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.043+0000 7f5c5b7fe700 1 --2- 172.21.15.1:0/1123913466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5c3c06adb0 0x7f5c3c06d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009c40 tx=0x7f5c4c008030 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:00.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.047+0000 7f5c53fff700 1 -- 172.21.15.1:0/1123913466 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c44053a20 con 0x7f5c5c099f70 2024-03-18T20:30:00.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.251+0000 7f5c620ba700 1 -- 172.21.15.1:0/1123913466 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c5c0b6380 con 0x7f5c3c06adb0 2024-03-18T20:30:00.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.251+0000 7f5c53fff700 1 -- 172.21.15.1:0/1123913466 <== mgr.14091 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5c4c00bc70 con 0x7f5c3c06adb0 2024-03-18T20:30:00.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.254+0000 7f5c620ba700 1 -- 172.21.15.1:0/1123913466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5c3c06adb0 msgr2=0x7f5c3c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.254+0000 7f5c620ba700 1 --2- 172.21.15.1:0/1123913466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5c3c06adb0 0x7f5c3c06d270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009c40 tx=0x7f5c4c008030 comp rx=0 tx=0).stop 2024-03-18T20:30:00.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.254+0000 7f5c620ba700 1 -- 172.21.15.1:0/1123913466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c5c099f70 msgr2=0x7f5c5c159710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.254+0000 7f5c620ba700 1 --2- 172.21.15.1:0/1123913466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c5c099f70 0x7f5c5c159710 secure :-1 s=READY pgs=3430 cs=0 l=1 rev1=1 crypto rx=0x7f5c4400c4e0 tx=0x7f5c4400c8a0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.254+0000 7f5c620ba700 1 -- 172.21.15.1:0/1123913466 shutdown_connections 2024-03-18T20:30:00.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.254+0000 7f5c620ba700 1 --2- 172.21.15.1:0/1123913466 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5c3c06adb0 0x7f5c3c06d270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.254+0000 7f5c620ba700 1 --2- 172.21.15.1:0/1123913466 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5c5c06fc60 0x7f5c5c1407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.254+0000 7f5c620ba700 1 --2- 172.21.15.1:0/1123913466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c5c0b5190 0x7f5c5c159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.254+0000 7f5c620ba700 1 --2- 172.21.15.1:0/1123913466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c5c099f70 0x7f5c5c159710 unknown :-1 s=CLOSED pgs=3430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.254+0000 7f5c620ba700 1 -- 172.21.15.1:0/1123913466 >> 172.21.15.1:0/1123913466 conn(0x7f5c5c0b2de0 msgr2=0x7f5c5c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:00.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.254+0000 7f5c620ba700 1 -- 172.21.15.1:0/1123913466 shutdown_connections 2024-03-18T20:30:00.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.254+0000 7f5c620ba700 1 -- 172.21.15.1:0/1123913466 wait complete. 2024-03-18T20:30:00.259 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 2 to deep-scrub 2024-03-18T20:30:00.270 INFO:tasks.ceph:Scrubbing osd.3 2024-03-18T20:30:00.270 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-18T20:30:00.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.397+0000 7f2388fb0700 1 Processor -- start 2024-03-18T20:30:00.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.397+0000 7f2388fb0700 1 -- start start 2024-03-18T20:30:00.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.398+0000 7f2388fb0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f238415a220 0x7f2384072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.398+0000 7f2388fb0700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f23840ac1a0 0x7f23840ade10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.398+0000 7f2388fb0700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23840ac910 con 0x7f2384143600 2024-03-18T20:30:00.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.398+0000 7f2388fb0700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f238415a600 con 0x7f23840ac1a0 2024-03-18T20:30:00.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.398+0000 7f2388fb0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23840ac630 con 0x7f238415a220 2024-03-18T20:30:00.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.398+0000 7f238259c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f238415a220 0x7f2384072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.398+0000 7f2382d9d700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f23840ac1a0 0x7f23840ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.398+0000 7f238259c700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f238415a220 0x7f2384072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3301/0 says I am v2:172.21.15.1:60164/0 (socket says 172.21.15.1:60164) 2024-03-18T20:30:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.398+0000 7f238259c700 1 -- 172.21.15.1:0/1097369994 learned_addr learned my addr 172.21.15.1:0/1097369994 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.399+0000 7f2382d9d700 1 -- 172.21.15.1:0/1097369994 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f238415a220 msgr2=0x7f2384072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.399+0000 7f2382d9d700 1 --2- 172.21.15.1:0/1097369994 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f238415a220 0x7f2384072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.399+0000 7f2382d9d700 1 -- 172.21.15.1:0/1097369994 >> v1:172.21.15.165:6789/0 conn(0x7f2384143600 legacy=0x7f23840b37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.399+0000 7f2382d9d700 1 -- 172.21.15.1:0/1097369994 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23840b2d00 con 0x7f23840ac1a0 2024-03-18T20:30:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.399+0000 7f238159a700 1 -- 172.21.15.1:0/1097369994 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752517394 0 0) 0x7f23840ac910 con 0x7f2384143600 2024-03-18T20:30:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.399+0000 7f238259c700 1 --2- 172.21.15.1:0/1097369994 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f238415a220 0x7f2384072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:30:00.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.399+0000 7f2382d9d700 1 --2- 172.21.15.1:0/1097369994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23840ac1a0 0x7f23840ade10 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f237c009840 tx=0x7f237c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d26154389e56837 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:00.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.399+0000 7f238159a700 1 -- 172.21.15.1:0/1097369994 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c011830 con 0x7f23840ac1a0 2024-03-18T20:30:00.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.399+0000 7f238159a700 1 -- 172.21.15.1:0/1097369994 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237c011990 con 0x7f23840ac1a0 2024-03-18T20:30:00.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.399+0000 7f238159a700 1 -- 172.21.15.1:0/1097369994 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c011d20 con 0x7f23840ac1a0 2024-03-18T20:30:00.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.400+0000 7f2388fb0700 1 -- 172.21.15.1:0/1097369994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23840ac1a0 msgr2=0x7f23840ade10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.400+0000 7f2388fb0700 1 --2- 172.21.15.1:0/1097369994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23840ac1a0 0x7f23840ade10 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f237c009840 tx=0x7f237c009b50 comp rx=0 tx=0).stop 2024-03-18T20:30:00.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.400+0000 7f2388fb0700 1 -- 172.21.15.1:0/1097369994 shutdown_connections 2024-03-18T20:30:00.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.400+0000 7f2388fb0700 1 --2- 172.21.15.1:0/1097369994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23840ac1a0 0x7f23840ade10 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.400+0000 7f2388fb0700 1 --2- 172.21.15.1:0/1097369994 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f238415a220 0x7f2384072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.400+0000 7f2388fb0700 1 -- 172.21.15.1:0/1097369994 >> 172.21.15.1:0/1097369994 conn(0x7f238406abf0 msgr2=0x7f238406b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:00.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.400+0000 7f2388fb0700 1 -- 172.21.15.1:0/1097369994 shutdown_connections 2024-03-18T20:30:00.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.400+0000 7f2388fb0700 1 -- 172.21.15.1:0/1097369994 wait complete. 2024-03-18T20:30:00.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.401+0000 7f2388fb0700 1 Processor -- start 2024-03-18T20:30:00.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.401+0000 7f2388fb0700 1 -- start start 2024-03-18T20:30:00.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.402+0000 7f2388fb0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2384143600 0x7f2384098230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.402+0000 7f2388fb0700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f238415a220 0x7f238415cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.402+0000 7f2388fb0700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2384098a30 0x7f238415d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.402+0000 7f2388fb0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23840b06d0 con 0x7f2384143600 2024-03-18T20:30:00.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.402+0000 7f2388fb0700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238415d7d0 con 0x7f238415a220 2024-03-18T20:30:00.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.402+0000 7f2388fb0700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238415b8d0 con 0x7f2384098a30 2024-03-18T20:30:00.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.402+0000 7f238259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2384143600 0x7f2384098230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.402+0000 7f238259c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2384143600 0x7f2384098230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50174/0 (socket says 172.21.15.1:50174) 2024-03-18T20:30:00.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.402+0000 7f238259c700 1 -- 172.21.15.1:0/3375571869 learned_addr learned my addr 172.21.15.1:0/3375571869 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:00.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.402+0000 7f2381d9b700 1 --2- 172.21.15.1:0/3375571869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f238415a220 0x7f238415cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.402+0000 7f2382d9d700 1 --2- 172.21.15.1:0/3375571869 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2384098a30 0x7f238415d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.402+0000 7f238259c700 1 -- 172.21.15.1:0/3375571869 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2384098a30 msgr2=0x7f238415d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.402+0000 7f238259c700 1 --2- 172.21.15.1:0/3375571869 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f2384098a30 0x7f238415d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.402+0000 7f238259c700 1 -- 172.21.15.1:0/3375571869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f238415a220 msgr2=0x7f238415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.402+0000 7f238259c700 1 --2- 172.21.15.1:0/3375571869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f238415a220 0x7f238415cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.402+0000 7f238259c700 1 -- 172.21.15.1:0/3375571869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237400bd40 con 0x7f2384143600 2024-03-18T20:30:00.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.402+0000 7f2381d9b700 1 --2- 172.21.15.1:0/3375571869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f238415a220 0x7f238415cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:30:00.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.403+0000 7f238259c700 1 --2- 172.21.15.1:0/3375571869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2384143600 0x7f2384098230 secure :-1 s=READY pgs=3431 cs=0 l=1 rev1=1 crypto rx=0x7f237400e0f0 tx=0x7f237400e400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:00.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.403+0000 7f2372ffd700 1 -- 172.21.15.1:0/3375571869 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237400ea80 con 0x7f2384143600 2024-03-18T20:30:00.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.403+0000 7f2372ffd700 1 -- 172.21.15.1:0/3375571869 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2374010030 con 0x7f2384143600 2024-03-18T20:30:00.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.403+0000 7f2388fb0700 1 -- 172.21.15.1:0/3375571869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237c0094a0 con 0x7f2384143600 2024-03-18T20:30:00.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.403+0000 7f2372ffd700 1 -- 172.21.15.1:0/3375571869 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237400ede0 con 0x7f2384143600 2024-03-18T20:30:00.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.403+0000 7f2388fb0700 1 -- 172.21.15.1:0/3375571869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f238415b5f0 con 0x7f2384143600 2024-03-18T20:30:00.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.404+0000 7f2388fb0700 1 -- 172.21.15.1:0/3375571869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f23840b32c0 con 0x7f2384143600 2024-03-18T20:30:00.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.404+0000 7f2372ffd700 1 -- 172.21.15.1:0/3375571869 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f2374009600 con 0x7f2384143600 2024-03-18T20:30:00.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.405+0000 7f2372ffd700 1 --2- 172.21.15.1:0/3375571869 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f236406adb0 0x7f236406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.405+0000 7f2372ffd700 1 -- 172.21.15.1:0/3375571869 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f237408bed0 con 0x7f2384143600 2024-03-18T20:30:00.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.405+0000 7f2381d9b700 1 --2- 172.21.15.1:0/3375571869 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f236406adb0 0x7f236406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.405+0000 7f2372ffd700 1 --2- 172.21.15.1:0/3375571869 >> [v2:172.21.15.1:6816/77310,v1:172.21.15.1:6817/77310] conn(0x7f2364071750 0x7f2364073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.405+0000 7f2372ffd700 1 -- 172.21.15.1:0/3375571869 --> [v2:172.21.15.1:6816/77310,v1:172.21.15.1:6817/77310] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f23640742a0 con 0x7f2364071750 2024-03-18T20:30:00.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.405+0000 7f2372ffd700 1 -- 172.21.15.1:0/3375571869 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=295) v2 ==== 24+0+0 (secure 0 0 0) 0x7f237408e590 con 0x7f2384143600 2024-03-18T20:30:00.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.405+0000 7f2382d9d700 1 --2- 172.21.15.1:0/3375571869 >> [v2:172.21.15.1:6816/77310,v1:172.21.15.1:6817/77310] conn(0x7f2364071750 0x7f2364073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.405+0000 7f2381d9b700 1 --2- 172.21.15.1:0/3375571869 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f236406adb0 0x7f236406d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f238414a640 tx=0x7f2378009040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:00.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.405+0000 7f2382d9d700 1 --2- 172.21.15.1:0/3375571869 >> [v2:172.21.15.1:6816/77310,v1:172.21.15.1:6817/77310] conn(0x7f2364071750 0x7f2364073bd0 crc :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:00.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.406+0000 7f2372ffd700 1 -- 172.21.15.1:0/3375571869 <== osd.3 v2:172.21.15.1:6816/77310 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f23640742a0 con 0x7f2364071750 2024-03-18T20:30:00.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.420+0000 7f2388fb0700 1 -- 172.21.15.1:0/3375571869 --> [v2:172.21.15.1:6816/77310,v1:172.21.15.1:6817/77310] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f23840676b0 con 0x7f2364071750 2024-03-18T20:30:00.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.420+0000 7f2372ffd700 1 -- 172.21.15.1:0/3375571869 <== osd.3 v2:172.21.15.1:6816/77310 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f23840676b0 con 0x7f2364071750 2024-03-18T20:30:00.425 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:30:00.425 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T20:30:00.425 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:30:00.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.421+0000 7f2388fb0700 1 -- 172.21.15.1:0/3375571869 >> [v2:172.21.15.1:6816/77310,v1:172.21.15.1:6817/77310] conn(0x7f2364071750 msgr2=0x7f2364073bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.421+0000 7f2388fb0700 1 --2- 172.21.15.1:0/3375571869 >> [v2:172.21.15.1:6816/77310,v1:172.21.15.1:6817/77310] conn(0x7f2364071750 0x7f2364073bd0 crc :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.421+0000 7f2388fb0700 1 -- 172.21.15.1:0/3375571869 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f236406adb0 msgr2=0x7f236406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.421+0000 7f2388fb0700 1 --2- 172.21.15.1:0/3375571869 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f236406adb0 0x7f236406d270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f238414a640 tx=0x7f2378009040 comp rx=0 tx=0).stop 2024-03-18T20:30:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.421+0000 7f2388fb0700 1 -- 172.21.15.1:0/3375571869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2384143600 msgr2=0x7f2384098230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.421+0000 7f2388fb0700 1 --2- 172.21.15.1:0/3375571869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2384143600 0x7f2384098230 secure :-1 s=READY pgs=3431 cs=0 l=1 rev1=1 crypto rx=0x7f237400e0f0 tx=0x7f237400e400 comp rx=0 tx=0).stop 2024-03-18T20:30:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.421+0000 7f2382d9d700 1 -- 172.21.15.1:0/3375571869 reap_dead start 2024-03-18T20:30:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.421+0000 7f2388fb0700 1 -- 172.21.15.1:0/3375571869 shutdown_connections 2024-03-18T20:30:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.421+0000 7f2388fb0700 1 -- 172.21.15.1:0/3375571869 >> 172.21.15.1:0/3375571869 conn(0x7f238406abf0 msgr2=0x7f23840b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.421+0000 7f2388fb0700 1 -- 172.21.15.1:0/3375571869 shutdown_connections 2024-03-18T20:30:00.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.421+0000 7f2388fb0700 1 -- 172.21.15.1:0/3375571869 wait complete. 2024-03-18T20:30:00.437 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-18T20:30:00.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.559+0000 7f1f80c5b700 1 Processor -- start 2024-03-18T20:30:00.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.559+0000 7f1f80c5b700 1 -- start start 2024-03-18T20:30:00.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.560+0000 7f1f80c5b700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1f7c0b0f50 0x7f1f7c0af680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.560+0000 7f1f80c5b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1f7c099a30 0x7f1f7c0b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.560+0000 7f1f80c5b700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7c0b1c80 con 0x7f1f7c0b1870 2024-03-18T20:30:00.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.560+0000 7f1f80c5b700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f7c0b2a50 con 0x7f1f7c099a30 2024-03-18T20:30:00.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.560+0000 7f1f80c5b700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f7c0aeba0 con 0x7f1f7c0b0f50 2024-03-18T20:30:00.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.560+0000 7f1f79d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1f7c099a30 0x7f1f7c0b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.560+0000 7f1f79d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1f7c099a30 0x7f1f7c0b1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50184/0 (socket says 172.21.15.1:50184) 2024-03-18T20:30:00.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.560+0000 7f1f79d9b700 1 -- 172.21.15.1:0/397544696 learned_addr learned my addr 172.21.15.1:0/397544696 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:00.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.560+0000 7f1f7ad9d700 1 --1- 172.21.15.1:0/397544696 >> v1:172.21.15.1:6789/0 conn(0x7f1f7c0b1870 0x7f1f7c0b32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59272/0 (socket says 172.21.15.1:59272) 2024-03-18T20:30:00.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.560+0000 7f1f7a59c700 1 --2- 172.21.15.1:0/397544696 >> v2:172.21.15.165:3300/0 conn(0x7f1f7c0b0f50 0x7f1f7c0af680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.560+0000 7f1f79d9b700 1 -- 172.21.15.1:0/397544696 >> v2:172.21.15.165:3300/0 conn(0x7f1f7c0b0f50 msgr2=0x7f1f7c0af680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:00.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.560+0000 7f1f79d9b700 1 --2- 172.21.15.1:0/397544696 >> v2:172.21.15.165:3300/0 conn(0x7f1f7c0b0f50 0x7f1f7c0af680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.560+0000 7f1f79d9b700 1 -- 172.21.15.1:0/397544696 >> v1:172.21.15.1:6789/0 conn(0x7f1f7c0b1870 legacy=0x7f1f7c0b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.560+0000 7f1f79d9b700 1 -- 172.21.15.1:0/397544696 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7c0aee80 con 0x7f1f7c099a30 2024-03-18T20:30:00.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.560+0000 7f1f7a59c700 1 --2- 172.21.15.1:0/397544696 >> v2:172.21.15.165:3300/0 conn(0x7f1f7c0b0f50 0x7f1f7c0af680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:30:00.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.560+0000 7f1f79d9b700 1 --2- 172.21.15.1:0/397544696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f7c099a30 0x7f1f7c0b1330 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0b59c0 tx=0x7f1f6c00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad3b91058e707af9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.561+0000 7f1f7959a700 1 -- 172.21.15.1:0/397544696 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c00dd30 con 0x7f1f7c099a30 2024-03-18T20:30:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.561+0000 7f1f7959a700 1 -- 172.21.15.1:0/397544696 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6c00de90 con 0x7f1f7c099a30 2024-03-18T20:30:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.561+0000 7f1f7959a700 1 -- 172.21.15.1:0/397544696 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c010690 con 0x7f1f7c099a30 2024-03-18T20:30:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.561+0000 7f1f80c5b700 1 -- 172.21.15.1:0/397544696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f7c099a30 msgr2=0x7f1f7c0b1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.561+0000 7f1f80c5b700 1 --2- 172.21.15.1:0/397544696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f7c099a30 0x7f1f7c0b1330 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0b59c0 tx=0x7f1f6c00be70 comp rx=0 tx=0).stop 2024-03-18T20:30:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.561+0000 7f1f80c5b700 1 -- 172.21.15.1:0/397544696 shutdown_connections 2024-03-18T20:30:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.561+0000 7f1f80c5b700 1 --2- 172.21.15.1:0/397544696 >> v2:172.21.15.165:3300/0 conn(0x7f1f7c0b0f50 0x7f1f7c0af680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.561+0000 7f1f80c5b700 1 --2- 172.21.15.1:0/397544696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f7c099a30 0x7f1f7c0b1330 unknown :-1 s=CLOSED pgs=3433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.561+0000 7f1f80c5b700 1 -- 172.21.15.1:0/397544696 >> 172.21.15.1:0/397544696 conn(0x7f1f7c0b5a10 msgr2=0x7f1f7c0b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.561+0000 7f1f80c5b700 1 -- 172.21.15.1:0/397544696 shutdown_connections 2024-03-18T20:30:00.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.561+0000 7f1f80c5b700 1 -- 172.21.15.1:0/397544696 wait complete. 2024-03-18T20:30:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.563+0000 7f1f80c5b700 1 Processor -- start 2024-03-18T20:30:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.563+0000 7f1f80c5b700 1 -- start start 2024-03-18T20:30:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.563+0000 7f1f80c5b700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f7c099a30 0x7f1f7c1428f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.563+0000 7f1f80c5b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f7c0b0f50 0x7f1f7c0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.563+0000 7f1f80c5b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f7c0b1870 0x7f1f7c0cf430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.563+0000 7f1f80c5b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c1eed20 con 0x7f1f7c0b0f50 2024-03-18T20:30:00.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.563+0000 7f1f80c5b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c0ce3c0 con 0x7f1f7c0b1870 2024-03-18T20:30:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.563+0000 7f1f80c5b700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c0cd840 con 0x7f1f7c099a30 2024-03-18T20:30:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.563+0000 7f1f79d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f7c0b0f50 0x7f1f7c0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.564+0000 7f1f79d9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f7c0b0f50 0x7f1f7c0ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50200/0 (socket says 172.21.15.1:50200) 2024-03-18T20:30:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.564+0000 7f1f79d9b700 1 -- 172.21.15.1:0/3809478598 learned_addr learned my addr 172.21.15.1:0/3809478598 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.564+0000 7f1f7ad9d700 1 --2- 172.21.15.1:0/3809478598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f7c0b1870 0x7f1f7c0cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.564+0000 7f1f7a59c700 1 --2- 172.21.15.1:0/3809478598 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f7c099a30 0x7f1f7c1428f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.564+0000 7f1f79d9b700 1 -- 172.21.15.1:0/3809478598 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f7c099a30 msgr2=0x7f1f7c1428f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.564+0000 7f1f79d9b700 1 --2- 172.21.15.1:0/3809478598 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f7c099a30 0x7f1f7c1428f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.564+0000 7f1f79d9b700 1 -- 172.21.15.1:0/3809478598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f7c0b1870 msgr2=0x7f1f7c0cf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.564+0000 7f1f79d9b700 1 --2- 172.21.15.1:0/3809478598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f7c0b1870 0x7f1f7c0cf430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.564+0000 7f1f79d9b700 1 -- 172.21.15.1:0/3809478598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c00b7f0 con 0x7f1f7c0b0f50 2024-03-18T20:30:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.564+0000 7f1f7ad9d700 1 --2- 172.21.15.1:0/3809478598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f7c0b1870 0x7f1f7c0cf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:30:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.564+0000 7f1f79d9b700 1 --2- 172.21.15.1:0/3809478598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f7c0b0f50 0x7f1f7c0ceef0 secure :-1 s=READY pgs=3434 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c000c00 tx=0x7f1f6c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.564+0000 7f1f6affd700 1 -- 172.21.15.1:0/3809478598 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c01f070 con 0x7f1f7c0b0f50 2024-03-18T20:30:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.564+0000 7f1f6affd700 1 -- 172.21.15.1:0/3809478598 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6c003ba0 con 0x7f1f7c0b0f50 2024-03-18T20:30:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.564+0000 7f1f80c5b700 1 -- 172.21.15.1:0/3809478598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7c0cdb20 con 0x7f1f7c0b0f50 2024-03-18T20:30:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.564+0000 7f1f6affd700 1 -- 172.21.15.1:0/3809478598 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c010470 con 0x7f1f7c0b0f50 2024-03-18T20:30:00.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.564+0000 7f1f80c5b700 1 -- 172.21.15.1:0/3809478598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f7c0ce0e0 con 0x7f1f7c0b0f50 2024-03-18T20:30:00.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.565+0000 7f1f80c5b700 1 -- 172.21.15.1:0/3809478598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f7c1eee60 con 0x7f1f7c0b0f50 2024-03-18T20:30:00.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.566+0000 7f1f6affd700 1 -- 172.21.15.1:0/3809478598 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f1f6c019400 con 0x7f1f7c0b0f50 2024-03-18T20:30:00.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.566+0000 7f1f6affd700 1 --2- 172.21.15.1:0/3809478598 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1f5c06adb0 0x7f1f5c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.566+0000 7f1f7a59c700 1 --2- 172.21.15.1:0/3809478598 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1f5c06adb0 0x7f1f5c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.566+0000 7f1f6affd700 1 -- 172.21.15.1:0/3809478598 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f1f6c08c940 con 0x7f1f7c0b0f50 2024-03-18T20:30:00.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.567+0000 7f1f7a59c700 1 --2- 172.21.15.1:0/3809478598 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1f5c06adb0 0x7f1f5c06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1f70009930 tx=0x7f1f70008040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:00.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.571+0000 7f1f6affd700 1 -- 172.21.15.1:0/3809478598 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1f6c057810 con 0x7f1f7c0b0f50 2024-03-18T20:30:00.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.777+0000 7f1f80c5b700 1 -- 172.21.15.1:0/3809478598 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f7c0b26e0 con 0x7f1f5c06adb0 2024-03-18T20:30:00.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.777+0000 7f1f6affd700 1 -- 172.21.15.1:0/3809478598 <== mgr.14091 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1f70009e00 con 0x7f1f5c06adb0 2024-03-18T20:30:00.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.779+0000 7f1f80c5b700 1 -- 172.21.15.1:0/3809478598 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1f5c06adb0 msgr2=0x7f1f5c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.779+0000 7f1f80c5b700 1 --2- 172.21.15.1:0/3809478598 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1f5c06adb0 0x7f1f5c06d270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1f70009930 tx=0x7f1f70008040 comp rx=0 tx=0).stop 2024-03-18T20:30:00.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.779+0000 7f1f80c5b700 1 -- 172.21.15.1:0/3809478598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f7c0b0f50 msgr2=0x7f1f7c0ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.779+0000 7f1f80c5b700 1 --2- 172.21.15.1:0/3809478598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f7c0b0f50 0x7f1f7c0ceef0 secure :-1 s=READY pgs=3434 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c000c00 tx=0x7f1f6c004750 comp rx=0 tx=0).stop 2024-03-18T20:30:00.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.779+0000 7f1f80c5b700 1 -- 172.21.15.1:0/3809478598 shutdown_connections 2024-03-18T20:30:00.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.779+0000 7f1f80c5b700 1 --2- 172.21.15.1:0/3809478598 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f1f5c06adb0 0x7f1f5c06d270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.779+0000 7f1f80c5b700 1 --2- 172.21.15.1:0/3809478598 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f1f7c099a30 0x7f1f7c1428f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.779+0000 7f1f80c5b700 1 --2- 172.21.15.1:0/3809478598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f7c0b1870 0x7f1f7c0cf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.779+0000 7f1f80c5b700 1 --2- 172.21.15.1:0/3809478598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f7c0b0f50 0x7f1f7c0ceef0 unknown :-1 s=CLOSED pgs=3434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.779+0000 7f1f80c5b700 1 -- 172.21.15.1:0/3809478598 >> 172.21.15.1:0/3809478598 conn(0x7f1f7c0b5a10 msgr2=0x7f1f7c0b3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:00.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.779+0000 7f1f80c5b700 1 -- 172.21.15.1:0/3809478598 shutdown_connections 2024-03-18T20:30:00.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.779+0000 7f1f80c5b700 1 -- 172.21.15.1:0/3809478598 wait complete. 2024-03-18T20:30:00.788 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 3 to deep-scrub 2024-03-18T20:30:00.795 INFO:tasks.ceph:Scrubbing osd.4 2024-03-18T20:30:00.796 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-18T20:30:00.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.921+0000 7f5c027c5700 1 Processor -- start 2024-03-18T20:30:00.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.921+0000 7f5c027c5700 1 -- start start 2024-03-18T20:30:00.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.922+0000 7f5c027c5700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5bfc09a900 0x7f5bfc099a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.922+0000 7f5c027c5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bfc099f70 0x7f5bfc15da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.922+0000 7f5c027c5700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bfc09a370 con 0x7f5bfc15d680 2024-03-18T20:30:00.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.922+0000 7f5c027c5700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bfc0b54f0 con 0x7f5bfc09a900 2024-03-18T20:30:00.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.922+0000 7f5c027c5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0b57d0 con 0x7f5bfc099f70 2024-03-18T20:30:00.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.922+0000 7f5bfbfff700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5bfc15d680 0x7f5bfc0b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59276/0 (socket says 172.21.15.1:59276) 2024-03-18T20:30:00.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.922+0000 7f5bfbfff700 1 -- 172.21.15.1:0/2274948 learned_addr learned my addr 172.21.15.1:0/2274948 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:30:00.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.922+0000 7f5bfb7fe700 1 --2- 172.21.15.1:0/2274948 >> v2:172.21.15.165:3300/0 conn(0x7f5bfc09a900 0x7f5bfc099a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.922+0000 7f5c00d62700 1 --2- 172.21.15.1:0/2274948 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bfc099f70 0x7f5bfc15da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.922+0000 7f5bfaffd700 1 -- 172.21.15.1:0/2274948 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783077289 0 0) 0x7f5bfc09a370 con 0x7f5bfc15d680 2024-03-18T20:30:00.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.922+0000 7f5bfaffd700 1 -- 172.21.15.1:0/2274948 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5be80035b0 con 0x7f5bfc15d680 2024-03-18T20:30:00.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.922+0000 7f5bfaffd700 1 -- 172.21.15.1:0/2274948 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3701249180 0 0) 0x7f5be80035b0 con 0x7f5bfc15d680 2024-03-18T20:30:00.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.922+0000 7f5bfaffd700 1 -- 172.21.15.1:0/2274948 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bfc099f70 msgr2=0x7f5bfc15da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:00.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.922+0000 7f5bfaffd700 1 --2- 172.21.15.1:0/2274948 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bfc099f70 0x7f5bfc15da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.922+0000 7f5bfaffd700 1 -- 172.21.15.1:0/2274948 >> v2:172.21.15.165:3300/0 conn(0x7f5bfc09a900 msgr2=0x7f5bfc099a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:00.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.922+0000 7f5bfaffd700 1 --2- 172.21.15.1:0/2274948 >> v2:172.21.15.165:3300/0 conn(0x7f5bfc09a900 0x7f5bfc099a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.922+0000 7f5bfaffd700 1 -- 172.21.15.1:0/2274948 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc0b5ab0 con 0x7f5bfc15d680 2024-03-18T20:30:00.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5bfaffd700 1 -- 172.21.15.1:0/2274948 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f5be4001670 con 0x7f5bfc15d680 2024-03-18T20:30:00.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5bfaffd700 1 -- 172.21.15.1:0/2274948 >> v1:172.21.15.1:6789/0 conn(0x7f5bfc15d680 legacy=0x7f5bfc0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5bfaffd700 1 --2- 172.21.15.1:0/2274948 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5be8003a00 0x7f5be8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5bfaffd700 1 --2- 172.21.15.1:0/2274948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be8004850 0x7f5be8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5bfaffd700 1 --2- 172.21.15.1:0/2274948 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bfc099f70 0x7f5be8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5bfaffd700 1 -- 172.21.15.1:0/2274948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0b54f0 con 0x7f5be8004850 2024-03-18T20:30:00.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5bfaffd700 1 -- 172.21.15.1:0/2274948 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0b57d0 con 0x7f5be8003a00 2024-03-18T20:30:00.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5bfaffd700 1 -- 172.21.15.1:0/2274948 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be800ba90 con 0x7f5bfc099f70 2024-03-18T20:30:00.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5bfb7fe700 1 --2- 172.21.15.1:0/2274948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be8004850 0x7f5be8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5bfbfff700 1 --2- 172.21.15.1:0/2274948 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5be8003a00 0x7f5be8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5c00d62700 1 --2- 172.21.15.1:0/2274948 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bfc099f70 0x7f5be8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5bfb7fe700 1 -- 172.21.15.1:0/2274948 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bfc099f70 msgr2=0x7f5be8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:00.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5bfb7fe700 1 --2- 172.21.15.1:0/2274948 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bfc099f70 0x7f5be8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5bfb7fe700 1 -- 172.21.15.1:0/2274948 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5be8003a00 msgr2=0x7f5be8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:00.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5bfb7fe700 1 --2- 172.21.15.1:0/2274948 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5be8003a00 0x7f5be8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5bfb7fe700 1 -- 172.21.15.1:0/2274948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be80035b0 con 0x7f5be8004850 2024-03-18T20:30:00.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5c00d62700 1 --2- 172.21.15.1:0/2274948 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bfc099f70 0x7f5be8005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:30:00.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5bfb7fe700 1 --2- 172.21.15.1:0/2274948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be8004850 0x7f5be8004cd0 secure :-1 s=READY pgs=3436 cs=0 l=1 rev1=1 crypto rx=0x7f5bec009b60 tx=0x7f5be8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498b039e650a795a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:00.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5bfaffd700 1 -- 172.21.15.1:0/2274948 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec00e6c0 con 0x7f5be8004850 2024-03-18T20:30:00.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5bfaffd700 1 -- 172.21.15.1:0/2274948 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bec00e820 con 0x7f5be8004850 2024-03-18T20:30:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.923+0000 7f5bfaffd700 1 -- 172.21.15.1:0/2274948 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec00eb90 con 0x7f5be8004850 2024-03-18T20:30:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.924+0000 7f5c027c5700 1 -- 172.21.15.1:0/2274948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be8004850 msgr2=0x7f5be8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.924+0000 7f5c027c5700 1 --2- 172.21.15.1:0/2274948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5be8004850 0x7f5be8004cd0 secure :-1 s=READY pgs=3436 cs=0 l=1 rev1=1 crypto rx=0x7f5bec009b60 tx=0x7f5be8005880 comp rx=0 tx=0).stop 2024-03-18T20:30:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.924+0000 7f5c00d62700 1 -- 172.21.15.1:0/2274948 reap_dead start 2024-03-18T20:30:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.924+0000 7f5c027c5700 1 -- 172.21.15.1:0/2274948 shutdown_connections 2024-03-18T20:30:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.924+0000 7f5c027c5700 1 -- 172.21.15.1:0/2274948 >> 172.21.15.1:0/2274948 conn(0x7f5bfc0b2ec0 msgr2=0x7f5bfc0b32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.924+0000 7f5c027c5700 1 -- 172.21.15.1:0/2274948 shutdown_connections 2024-03-18T20:30:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.924+0000 7f5c027c5700 1 -- 172.21.15.1:0/2274948 wait complete. 2024-03-18T20:30:00.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.925+0000 7f5c027c5700 1 Processor -- start 2024-03-18T20:30:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.925+0000 7f5c027c5700 1 -- start start 2024-03-18T20:30:00.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.926+0000 7f5c027c5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5be8004850 0x7f5bfc155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.926+0000 7f5c027c5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc1577a0 0x7f5bfc157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.926+0000 7f5c027c5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bfc09dc80 0x7f5bfc09e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.926+0000 7f5c027c5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc158160 con 0x7f5bfc1577a0 2024-03-18T20:30:00.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.926+0000 7f5c027c5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0cfaf0 con 0x7f5be8004850 2024-03-18T20:30:00.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.926+0000 7f5c027c5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc09f270 con 0x7f5bfc09dc80 2024-03-18T20:30:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.926+0000 7f5bfb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc1577a0 0x7f5bfc157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.926+0000 7f5bfb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc1577a0 0x7f5bfc157c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50226/0 (socket says 172.21.15.1:50226) 2024-03-18T20:30:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.926+0000 7f5bfb7fe700 1 -- 172.21.15.1:0/2754974591 learned_addr learned my addr 172.21.15.1:0/2754974591 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.926+0000 7f5bfbfff700 1 --2- 172.21.15.1:0/2754974591 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5be8004850 0x7f5bfc155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.926+0000 7f5c00d62700 1 --2- 172.21.15.1:0/2754974591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bfc09dc80 0x7f5bfc09e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.926+0000 7f5bfb7fe700 1 -- 172.21.15.1:0/2754974591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bfc09dc80 msgr2=0x7f5bfc09e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.927+0000 7f5bfb7fe700 1 --2- 172.21.15.1:0/2754974591 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f5bfc09dc80 0x7f5bfc09e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.927+0000 7f5bfb7fe700 1 -- 172.21.15.1:0/2754974591 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5be8004850 msgr2=0x7f5bfc155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.927+0000 7f5bfb7fe700 1 --2- 172.21.15.1:0/2754974591 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5be8004850 0x7f5bfc155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.927+0000 7f5bfb7fe700 1 -- 172.21.15.1:0/2754974591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc09e9d0 con 0x7f5bfc1577a0 2024-03-18T20:30:00.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.927+0000 7f5bfbfff700 1 --2- 172.21.15.1:0/2754974591 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5be8004850 0x7f5bfc155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:30:00.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.927+0000 7f5bfb7fe700 1 --2- 172.21.15.1:0/2754974591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc1577a0 0x7f5bfc157c20 secure :-1 s=READY pgs=3437 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0042d0 tx=0x7f5bec0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:00.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.927+0000 7f5bf8ff9700 1 -- 172.21.15.1:0/2754974591 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec00e6c0 con 0x7f5bfc1577a0 2024-03-18T20:30:00.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.927+0000 7f5bf8ff9700 1 -- 172.21.15.1:0/2754974591 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bec003ab0 con 0x7f5bfc1577a0 2024-03-18T20:30:00.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.927+0000 7f5bf8ff9700 1 -- 172.21.15.1:0/2754974591 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec003e60 con 0x7f5bfc1577a0 2024-03-18T20:30:00.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.927+0000 7f5c027c5700 1 -- 172.21.15.1:0/2754974591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bfc09ecb0 con 0x7f5bfc1577a0 2024-03-18T20:30:00.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.927+0000 7f5c027c5700 1 -- 172.21.15.1:0/2754974591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bfc09e670 con 0x7f5bfc1577a0 2024-03-18T20:30:00.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.928+0000 7f5c027c5700 1 -- 172.21.15.1:0/2754974591 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5bfc0b5200 con 0x7f5bfc1577a0 2024-03-18T20:30:00.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.928+0000 7f5bf8ff9700 1 -- 172.21.15.1:0/2754974591 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f5bec013470 con 0x7f5bfc1577a0 2024-03-18T20:30:00.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.929+0000 7f5bf8ff9700 1 --2- 172.21.15.1:0/2754974591 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5bdc06adb0 0x7f5bdc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.929+0000 7f5bf8ff9700 1 -- 172.21.15.1:0/2754974591 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f5bec08afb0 con 0x7f5bfc1577a0 2024-03-18T20:30:00.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.929+0000 7f5bfbfff700 1 --2- 172.21.15.1:0/2754974591 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5bdc06adb0 0x7f5bdc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.929+0000 7f5bf8ff9700 1 --2- 172.21.15.1:0/2754974591 >> [v2:172.21.15.165:6808/77474,v1:172.21.15.165:6809/77474] conn(0x7f5bdc071750 0x7f5bdc073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:00.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.929+0000 7f5bf8ff9700 1 -- 172.21.15.1:0/2754974591 --> [v2:172.21.15.165:6808/77474,v1:172.21.15.165:6809/77474] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5bdc0742a0 con 0x7f5bdc071750 2024-03-18T20:30:00.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.929+0000 7f5bf8ff9700 1 -- 172.21.15.1:0/2754974591 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=295) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5bec08d530 con 0x7f5bfc1577a0 2024-03-18T20:30:00.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.929+0000 7f5bfbfff700 1 --2- 172.21.15.1:0/2754974591 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5bdc06adb0 0x7f5bdc06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5be4007fd0 tx=0x7f5be4009040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:00.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.930+0000 7f5c00d62700 1 --2- 172.21.15.1:0/2754974591 >> [v2:172.21.15.165:6808/77474,v1:172.21.15.165:6809/77474] conn(0x7f5bdc071750 0x7f5bdc073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:00.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.931+0000 7f5c00d62700 1 --2- 172.21.15.1:0/2754974591 >> [v2:172.21.15.165:6808/77474,v1:172.21.15.165:6809/77474] conn(0x7f5bdc071750 0x7f5bdc073bd0 crc :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:00.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.932+0000 7f5bf8ff9700 1 -- 172.21.15.1:0/2754974591 <== osd.4 v2:172.21.15.165:6808/77474 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f5bdc0742a0 con 0x7f5bdc071750 2024-03-18T20:30:00.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.946+0000 7f5c027c5700 1 -- 172.21.15.1:0/2754974591 --> [v2:172.21.15.165:6808/77474,v1:172.21.15.165:6809/77474] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5bfc158160 con 0x7f5bdc071750 2024-03-18T20:30:00.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.946+0000 7f5bf8ff9700 1 -- 172.21.15.1:0/2754974591 <== osd.4 v2:172.21.15.165:6808/77474 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f5bfc158160 con 0x7f5bdc071750 2024-03-18T20:30:00.950 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:30:00.950 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T20:30:00.950 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:30:00.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.946+0000 7f5c027c5700 1 -- 172.21.15.1:0/2754974591 >> [v2:172.21.15.165:6808/77474,v1:172.21.15.165:6809/77474] conn(0x7f5bdc071750 msgr2=0x7f5bdc073bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.946+0000 7f5c027c5700 1 --2- 172.21.15.1:0/2754974591 >> [v2:172.21.15.165:6808/77474,v1:172.21.15.165:6809/77474] conn(0x7f5bdc071750 0x7f5bdc073bd0 crc :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.946+0000 7f5c027c5700 1 -- 172.21.15.1:0/2754974591 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5bdc06adb0 msgr2=0x7f5bdc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.946+0000 7f5c027c5700 1 --2- 172.21.15.1:0/2754974591 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f5bdc06adb0 0x7f5bdc06d270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5be4007fd0 tx=0x7f5be4009040 comp rx=0 tx=0).stop 2024-03-18T20:30:00.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.946+0000 7f5c027c5700 1 -- 172.21.15.1:0/2754974591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc1577a0 msgr2=0x7f5bfc157c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:00.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.946+0000 7f5c027c5700 1 --2- 172.21.15.1:0/2754974591 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc1577a0 0x7f5bfc157c20 secure :-1 s=READY pgs=3437 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0042d0 tx=0x7f5bec0047c0 comp rx=0 tx=0).stop 2024-03-18T20:30:00.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.946+0000 7f5c00d62700 1 -- 172.21.15.1:0/2754974591 reap_dead start 2024-03-18T20:30:00.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.947+0000 7f5c027c5700 1 -- 172.21.15.1:0/2754974591 shutdown_connections 2024-03-18T20:30:00.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.947+0000 7f5c027c5700 1 -- 172.21.15.1:0/2754974591 >> 172.21.15.1:0/2754974591 conn(0x7f5bfc0b2ec0 msgr2=0x7f5bfc0cbdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:00.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.947+0000 7f5c027c5700 1 -- 172.21.15.1:0/2754974591 shutdown_connections 2024-03-18T20:30:00.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:00.947+0000 7f5c027c5700 1 -- 172.21.15.1:0/2754974591 wait complete. 2024-03-18T20:30:00.962 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-18T20:30:01.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.083+0000 7f4bbc978700 1 Processor -- start 2024-03-18T20:30:01.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.084+0000 7f4bbc978700 1 -- start start 2024-03-18T20:30:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.084+0000 7f4bbc978700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bb815ef60 0x7f4bb815cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.084+0000 7f4bbc978700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb806ec60 con 0x7f4bb8099a30 2024-03-18T20:30:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.084+0000 7f4bbc978700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb80cfd80 con 0x7f4bb815e8e0 2024-03-18T20:30:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.084+0000 7f4bbc978700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb8070100 con 0x7f4bb815ef60 2024-03-18T20:30:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb659c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4bb8099a30 0x7f4bb8099e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59282/0 (socket says 172.21.15.1:59282) 2024-03-18T20:30:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb659c700 1 -- 172.21.15.1:0/3354980307 learned_addr learned my addr 172.21.15.1:0/3354980307 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:30:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb559a700 1 -- 172.21.15.1:0/3354980307 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513194167 0 0) 0x7f4bb806ec60 con 0x7f4bb8099a30 2024-03-18T20:30:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb559a700 1 -- 172.21.15.1:0/3354980307 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba40035b0 con 0x7f4bb8099a30 2024-03-18T20:30:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb6d9d700 1 --2- 172.21.15.1:0/3354980307 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bb815ef60 0x7f4bb815cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb559a700 1 -- 172.21.15.1:0/3354980307 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2415491407 0 0) 0x7f4ba40035b0 con 0x7f4bb8099a30 2024-03-18T20:30:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb559a700 1 -- 172.21.15.1:0/3354980307 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bb815ef60 msgr2=0x7f4bb815cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb559a700 1 --2- 172.21.15.1:0/3354980307 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bb815ef60 0x7f4bb815cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb559a700 1 -- 172.21.15.1:0/3354980307 >> v1:172.21.15.165:6789/0 conn(0x7f4bb815e8e0 legacy=0x7f4bb80cf380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb559a700 1 -- 172.21.15.1:0/3354980307 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb80d0060 con 0x7f4bb8099a30 2024-03-18T20:30:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb559a700 1 -- 172.21.15.1:0/3354980307 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f4ba8001670 con 0x7f4bb8099a30 2024-03-18T20:30:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb559a700 1 -- 172.21.15.1:0/3354980307 >> v1:172.21.15.1:6789/0 conn(0x7f4bb8099a30 legacy=0x7f4bb8099e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb559a700 1 --2- 172.21.15.1:0/3354980307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ba4003a30 0x7f4ba4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb559a700 1 --2- 172.21.15.1:0/3354980307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ba4004850 0x7f4ba4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb559a700 1 --2- 172.21.15.1:0/3354980307 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bb815ef60 0x7f4ba4005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb559a700 1 -- 172.21.15.1:0/3354980307 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba400bb10 con 0x7f4ba4004850 2024-03-18T20:30:01.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb559a700 1 -- 172.21.15.1:0/3354980307 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba400bc80 con 0x7f4ba4003a30 2024-03-18T20:30:01.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb559a700 1 -- 172.21.15.1:0/3354980307 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba400be40 con 0x7f4bb815ef60 2024-03-18T20:30:01.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb5d9b700 1 --2- 172.21.15.1:0/3354980307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ba4004850 0x7f4ba4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.085+0000 7f4bb659c700 1 --2- 172.21.15.1:0/3354980307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ba4003a30 0x7f4ba4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.086+0000 7f4bb6d9d700 1 --2- 172.21.15.1:0/3354980307 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bb815ef60 0x7f4ba4005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.086+0000 7f4bb5d9b700 1 -- 172.21.15.1:0/3354980307 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bb815ef60 msgr2=0x7f4ba4005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:01.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.086+0000 7f4bb5d9b700 1 --2- 172.21.15.1:0/3354980307 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bb815ef60 0x7f4ba4005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.086+0000 7f4bb5d9b700 1 -- 172.21.15.1:0/3354980307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ba4003a30 msgr2=0x7f4ba4003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.086+0000 7f4bb5d9b700 1 --2- 172.21.15.1:0/3354980307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ba4003a30 0x7f4ba4003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.086+0000 7f4bb5d9b700 1 -- 172.21.15.1:0/3354980307 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba40035b0 con 0x7f4ba4004850 2024-03-18T20:30:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.086+0000 7f4bb5d9b700 1 --2- 172.21.15.1:0/3354980307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ba4004850 0x7f4ba4004cd0 secure :-1 s=READY pgs=3439 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0001800 tx=0x7f4ba4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c1863aaba0ec5e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.086+0000 7f4bb559a700 1 -- 172.21.15.1:0/3354980307 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba00012f0 con 0x7f4ba4004850 2024-03-18T20:30:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.086+0000 7f4bb559a700 1 -- 172.21.15.1:0/3354980307 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba00052f0 con 0x7f4ba4004850 2024-03-18T20:30:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.086+0000 7f4bb559a700 1 -- 172.21.15.1:0/3354980307 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0006860 con 0x7f4ba4004850 2024-03-18T20:30:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.087+0000 7f4bbc978700 1 -- 172.21.15.1:0/3354980307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ba4004850 msgr2=0x7f4ba4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.087+0000 7f4bbc978700 1 --2- 172.21.15.1:0/3354980307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ba4004850 0x7f4ba4004cd0 secure :-1 s=READY pgs=3439 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0001800 tx=0x7f4ba4005880 comp rx=0 tx=0).stop 2024-03-18T20:30:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.087+0000 7f4bb6d9d700 1 -- 172.21.15.1:0/3354980307 reap_dead start 2024-03-18T20:30:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.087+0000 7f4bbc978700 1 -- 172.21.15.1:0/3354980307 shutdown_connections 2024-03-18T20:30:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.087+0000 7f4bbc978700 1 -- 172.21.15.1:0/3354980307 >> 172.21.15.1:0/3354980307 conn(0x7f4bb80d1180 msgr2=0x7f4bb80d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.087+0000 7f4bbc978700 1 -- 172.21.15.1:0/3354980307 shutdown_connections 2024-03-18T20:30:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.087+0000 7f4bbc978700 1 -- 172.21.15.1:0/3354980307 wait complete. 2024-03-18T20:30:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.088+0000 7f4bbc978700 1 Processor -- start 2024-03-18T20:30:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.088+0000 7f4bbc978700 1 -- start start 2024-03-18T20:30:01.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.089+0000 7f4bbc978700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ba4004850 0x7f4bb81f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.089+0000 7f4bbc978700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bb81f6830 0x7f4bb81f6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.089+0000 7f4bbc978700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bb80c4c00 0x7f4bb80c5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.089+0000 7f4bbc978700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb81f5790 con 0x7f4bb80c4c00 2024-03-18T20:30:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.089+0000 7f4bbc978700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb81f5a70 con 0x7f4ba4004850 2024-03-18T20:30:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.089+0000 7f4bbc978700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb81f5d50 con 0x7f4bb81f6830 2024-03-18T20:30:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.089+0000 7f4bb6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bb80c4c00 0x7f4bb80c5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.089+0000 7f4bb6d9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bb80c4c00 0x7f4bb80c5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50236/0 (socket says 172.21.15.1:50236) 2024-03-18T20:30:01.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.089+0000 7f4bb6d9d700 1 -- 172.21.15.1:0/3354177319 learned_addr learned my addr 172.21.15.1:0/3354177319 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:01.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.089+0000 7f4bb659c700 1 --2- 172.21.15.1:0/3354177319 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ba4004850 0x7f4bb81f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.089+0000 7f4bb5d9b700 1 --2- 172.21.15.1:0/3354177319 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bb81f6830 0x7f4bb81f6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.089+0000 7f4bb6d9d700 1 -- 172.21.15.1:0/3354177319 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bb81f6830 msgr2=0x7f4bb81f6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.089+0000 7f4bb6d9d700 1 --2- 172.21.15.1:0/3354177319 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bb81f6830 0x7f4bb81f6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.089+0000 7f4bb6d9d700 1 -- 172.21.15.1:0/3354177319 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ba4004850 msgr2=0x7f4bb81f51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.089+0000 7f4bb6d9d700 1 --2- 172.21.15.1:0/3354177319 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ba4004850 0x7f4bb81f51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.089+0000 7f4bb6d9d700 1 -- 172.21.15.1:0/3354177319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb81f6030 con 0x7f4bb80c4c00 2024-03-18T20:30:01.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.089+0000 7f4bb5d9b700 1 --2- 172.21.15.1:0/3354177319 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bb81f6830 0x7f4bb81f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:30:01.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.089+0000 7f4bb659c700 1 --2- 172.21.15.1:0/3354177319 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ba4004850 0x7f4bb81f51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:30:01.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.089+0000 7f4bb6d9d700 1 --2- 172.21.15.1:0/3354177319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bb80c4c00 0x7f4bb80c5080 secure :-1 s=READY pgs=3440 cs=0 l=1 rev1=1 crypto rx=0x7f4bb80b0c90 tx=0x7f4bb0010440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:01.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.090+0000 7f4baeffd700 1 -- 172.21.15.1:0/3354177319 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0010b70 con 0x7f4bb80c4c00 2024-03-18T20:30:01.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.090+0000 7f4baeffd700 1 -- 172.21.15.1:0/3354177319 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb0010cd0 con 0x7f4bb80c4c00 2024-03-18T20:30:01.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.090+0000 7f4bbc978700 1 -- 172.21.15.1:0/3354177319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb81f6310 con 0x7f4bb80c4c00 2024-03-18T20:30:01.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.090+0000 7f4baeffd700 1 -- 172.21.15.1:0/3354177319 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0017430 con 0x7f4bb80c4c00 2024-03-18T20:30:01.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.090+0000 7f4bbc978700 1 -- 172.21.15.1:0/3354177319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb815c310 con 0x7f4bb80c4c00 2024-03-18T20:30:01.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.090+0000 7f4bbc978700 1 -- 172.21.15.1:0/3354177319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bb8067dc0 con 0x7f4bb80c4c00 2024-03-18T20:30:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.091+0000 7f4baeffd700 1 -- 172.21.15.1:0/3354177319 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f4bb0017590 con 0x7f4bb80c4c00 2024-03-18T20:30:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.091+0000 7f4baeffd700 1 --2- 172.21.15.1:0/3354177319 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4b9806adb0 0x7f4b9806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.092+0000 7f4baeffd700 1 -- 172.21.15.1:0/3354177319 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f4bb0089ff0 con 0x7f4bb80c4c00 2024-03-18T20:30:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.092+0000 7f4bb659c700 1 --2- 172.21.15.1:0/3354177319 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4b9806adb0 0x7f4b9806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.092+0000 7f4bb659c700 1 --2- 172.21.15.1:0/3354177319 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4b9806adb0 0x7f4b9806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8007dd0 tx=0x7f4ba8009040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.096+0000 7f4baeffd700 1 -- 172.21.15.1:0/3354177319 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4bb0054f40 con 0x7f4bb80c4c00 2024-03-18T20:30:01.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.301+0000 7f4bbc978700 1 -- 172.21.15.1:0/3354177319 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f4bb81f5be0 con 0x7f4b9806adb0 2024-03-18T20:30:01.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.302+0000 7f4baeffd700 1 -- 172.21.15.1:0/3354177319 <== mgr.14091 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4bb81f5be0 con 0x7f4b9806adb0 2024-03-18T20:30:01.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.304+0000 7f4bbc978700 1 -- 172.21.15.1:0/3354177319 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4b9806adb0 msgr2=0x7f4b9806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.304+0000 7f4bbc978700 1 --2- 172.21.15.1:0/3354177319 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4b9806adb0 0x7f4b9806d270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8007dd0 tx=0x7f4ba8009040 comp rx=0 tx=0).stop 2024-03-18T20:30:01.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.304+0000 7f4bbc978700 1 -- 172.21.15.1:0/3354177319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bb80c4c00 msgr2=0x7f4bb80c5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.304+0000 7f4bbc978700 1 --2- 172.21.15.1:0/3354177319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bb80c4c00 0x7f4bb80c5080 secure :-1 s=READY pgs=3440 cs=0 l=1 rev1=1 crypto rx=0x7f4bb80b0c90 tx=0x7f4bb0010440 comp rx=0 tx=0).stop 2024-03-18T20:30:01.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.304+0000 7f4bbc978700 1 -- 172.21.15.1:0/3354177319 shutdown_connections 2024-03-18T20:30:01.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.304+0000 7f4bbc978700 1 --2- 172.21.15.1:0/3354177319 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f4b9806adb0 0x7f4b9806d270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.304+0000 7f4bbc978700 1 --2- 172.21.15.1:0/3354177319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4bb80c4c00 0x7f4bb80c5080 unknown :-1 s=CLOSED pgs=3440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.304+0000 7f4bbc978700 1 --2- 172.21.15.1:0/3354177319 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ba4004850 0x7f4bb81f51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.304+0000 7f4bbc978700 1 --2- 172.21.15.1:0/3354177319 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f4bb81f6830 0x7f4bb81f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.304+0000 7f4bbc978700 1 -- 172.21.15.1:0/3354177319 >> 172.21.15.1:0/3354177319 conn(0x7f4bb80d1180 msgr2=0x7f4bb80b0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:01.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.304+0000 7f4bbc978700 1 -- 172.21.15.1:0/3354177319 shutdown_connections 2024-03-18T20:30:01.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.304+0000 7f4bbc978700 1 -- 172.21.15.1:0/3354177319 wait complete. 2024-03-18T20:30:01.310 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 4 to deep-scrub 2024-03-18T20:30:01.320 INFO:tasks.ceph:Scrubbing osd.5 2024-03-18T20:30:01.321 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-18T20:30:01.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.448+0000 7fca4becd700 1 Processor -- start 2024-03-18T20:30:01.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.448+0000 7fca4becd700 1 -- start start 2024-03-18T20:30:01.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca4becd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fca440cd000 0x7fca4406e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca4becd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca440d0850 0x7fca440cda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca4becd700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca440cd3e0 con 0x7fca440cd5c0 2024-03-18T20:30:01.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca4becd700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca4415ed80 con 0x7fca440cd000 2024-03-18T20:30:01.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca4becd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca4415e170 con 0x7fca440d0850 2024-03-18T20:30:01.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca49468700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fca440cd5c0 0x7fca440d0140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59294/0 (socket says 172.21.15.1:59294) 2024-03-18T20:30:01.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca49468700 1 -- 172.21.15.1:0/3989018383 learned_addr learned my addr 172.21.15.1:0/3989018383 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:30:01.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca49c69700 1 --2- 172.21.15.1:0/3989018383 >> v2:172.21.15.165:3300/0 conn(0x7fca440cd000 0x7fca4406e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca4a46a700 1 --2- 172.21.15.1:0/3989018383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca440d0850 0x7fca440cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca48c67700 1 -- 172.21.15.1:0/3989018383 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275839819 0 0) 0x7fca440cd3e0 con 0x7fca440cd5c0 2024-03-18T20:30:01.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca48c67700 1 -- 172.21.15.1:0/3989018383 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca300035b0 con 0x7fca440cd5c0 2024-03-18T20:30:01.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca48c67700 1 -- 172.21.15.1:0/3989018383 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3290973298 0 0) 0x7fca300035b0 con 0x7fca440cd5c0 2024-03-18T20:30:01.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca48c67700 1 -- 172.21.15.1:0/3989018383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca440d0850 msgr2=0x7fca440cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:01.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca48c67700 1 --2- 172.21.15.1:0/3989018383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca440d0850 0x7fca440cda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca48c67700 1 -- 172.21.15.1:0/3989018383 >> v2:172.21.15.165:3300/0 conn(0x7fca440cd000 msgr2=0x7fca4406e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:01.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca48c67700 1 --2- 172.21.15.1:0/3989018383 >> v2:172.21.15.165:3300/0 conn(0x7fca440cd000 0x7fca4406e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca48c67700 1 -- 172.21.15.1:0/3989018383 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca4415e450 con 0x7fca440cd5c0 2024-03-18T20:30:01.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca48c67700 1 -- 172.21.15.1:0/3989018383 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fca40001670 con 0x7fca440cd5c0 2024-03-18T20:30:01.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca48c67700 1 -- 172.21.15.1:0/3989018383 >> v1:172.21.15.1:6789/0 conn(0x7fca440cd5c0 legacy=0x7fca440d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca48c67700 1 --2- 172.21.15.1:0/3989018383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca30003a00 0x7fca30003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca48c67700 1 --2- 172.21.15.1:0/3989018383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca30004850 0x7fca30004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca48c67700 1 --2- 172.21.15.1:0/3989018383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca440d0850 0x7fca30005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.449+0000 7fca48c67700 1 -- 172.21.15.1:0/3989018383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4415ed80 con 0x7fca30004850 2024-03-18T20:30:01.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.450+0000 7fca48c67700 1 -- 172.21.15.1:0/3989018383 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4415e170 con 0x7fca30003a00 2024-03-18T20:30:01.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.450+0000 7fca48c67700 1 -- 172.21.15.1:0/3989018383 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3000ba10 con 0x7fca440d0850 2024-03-18T20:30:01.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.450+0000 7fca49468700 1 --2- 172.21.15.1:0/3989018383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca30004850 0x7fca30004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.450+0000 7fca49c69700 1 --2- 172.21.15.1:0/3989018383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca30003a00 0x7fca30003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.450+0000 7fca4a46a700 1 --2- 172.21.15.1:0/3989018383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca440d0850 0x7fca30005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.450+0000 7fca49468700 1 -- 172.21.15.1:0/3989018383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca440d0850 msgr2=0x7fca30005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:01.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.450+0000 7fca49468700 1 --2- 172.21.15.1:0/3989018383 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca440d0850 0x7fca30005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.450+0000 7fca49468700 1 -- 172.21.15.1:0/3989018383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca30003a00 msgr2=0x7fca30003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:01.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.450+0000 7fca49468700 1 --2- 172.21.15.1:0/3989018383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca30003a00 0x7fca30003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.450+0000 7fca49468700 1 -- 172.21.15.1:0/3989018383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca300035b0 con 0x7fca30004850 2024-03-18T20:30:01.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.450+0000 7fca49c69700 1 --2- 172.21.15.1:0/3989018383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca30003a00 0x7fca30003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T20:30:01.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.450+0000 7fca49468700 1 --2- 172.21.15.1:0/3989018383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca30004850 0x7fca30004cd0 secure :-1 s=READY pgs=3442 cs=0 l=1 rev1=1 crypto rx=0x7fca4000c370 tx=0x7fca30005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff53bd4fa852341 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:01.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.450+0000 7fca48c67700 1 -- 172.21.15.1:0/3989018383 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca40007c60 con 0x7fca30004850 2024-03-18T20:30:01.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.450+0000 7fca48c67700 1 -- 172.21.15.1:0/3989018383 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca40005960 con 0x7fca30004850 2024-03-18T20:30:01.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.450+0000 7fca48c67700 1 -- 172.21.15.1:0/3989018383 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca40005d10 con 0x7fca30004850 2024-03-18T20:30:01.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.451+0000 7fca4becd700 1 -- 172.21.15.1:0/3989018383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca30004850 msgr2=0x7fca30004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.451+0000 7fca4becd700 1 --2- 172.21.15.1:0/3989018383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca30004850 0x7fca30004cd0 secure :-1 s=READY pgs=3442 cs=0 l=1 rev1=1 crypto rx=0x7fca4000c370 tx=0x7fca30005880 comp rx=0 tx=0).stop 2024-03-18T20:30:01.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.451+0000 7fca4a46a700 1 -- 172.21.15.1:0/3989018383 reap_dead start 2024-03-18T20:30:01.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.451+0000 7fca4becd700 1 -- 172.21.15.1:0/3989018383 shutdown_connections 2024-03-18T20:30:01.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.451+0000 7fca4becd700 1 -- 172.21.15.1:0/3989018383 >> 172.21.15.1:0/3989018383 conn(0x7fca44099a30 msgr2=0x7fca440b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:01.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.451+0000 7fca4becd700 1 -- 172.21.15.1:0/3989018383 shutdown_connections 2024-03-18T20:30:01.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.451+0000 7fca4becd700 1 -- 172.21.15.1:0/3989018383 wait complete. 2024-03-18T20:30:01.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.452+0000 7fca4becd700 1 Processor -- start 2024-03-18T20:30:01.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.452+0000 7fca4becd700 1 -- start start 2024-03-18T20:30:01.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.453+0000 7fca4becd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca30004850 0x7fca44157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.453+0000 7fca4becd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca440cbfe0 0x7fca441f2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.453+0000 7fca4becd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca441f6c00 0x7fca44158be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.453+0000 7fca4becd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca441f6420 con 0x7fca441f6c00 2024-03-18T20:30:01.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.453+0000 7fca4becd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca441f3710 con 0x7fca30004850 2024-03-18T20:30:01.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.453+0000 7fca4becd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca441f39f0 con 0x7fca440cbfe0 2024-03-18T20:30:01.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.453+0000 7fca4a46a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca441f6c00 0x7fca44158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.453+0000 7fca4a46a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca441f6c00 0x7fca44158be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50250/0 (socket says 172.21.15.1:50250) 2024-03-18T20:30:01.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.453+0000 7fca4a46a700 1 -- 172.21.15.1:0/2613416545 learned_addr learned my addr 172.21.15.1:0/2613416545 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:01.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.453+0000 7fca49c69700 1 --2- 172.21.15.1:0/2613416545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca30004850 0x7fca44157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.453+0000 7fca49468700 1 --2- 172.21.15.1:0/2613416545 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca440cbfe0 0x7fca441f2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.453+0000 7fca4a46a700 1 -- 172.21.15.1:0/2613416545 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca440cbfe0 msgr2=0x7fca441f2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.453+0000 7fca4a46a700 1 --2- 172.21.15.1:0/2613416545 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fca440cbfe0 0x7fca441f2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.453+0000 7fca4a46a700 1 -- 172.21.15.1:0/2613416545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca30004850 msgr2=0x7fca44157140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.453+0000 7fca4a46a700 1 --2- 172.21.15.1:0/2613416545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca30004850 0x7fca44157140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.453+0000 7fca4a46a700 1 -- 172.21.15.1:0/2613416545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca441f3cd0 con 0x7fca441f6c00 2024-03-18T20:30:01.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.453+0000 7fca4a46a700 1 --2- 172.21.15.1:0/2613416545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca441f6c00 0x7fca44158be0 secure :-1 s=READY pgs=3443 cs=0 l=1 rev1=1 crypto rx=0x7fca440ce620 tx=0x7fca3c0114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:01.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.453+0000 7fca3a7fc700 1 -- 172.21.15.1:0/2613416545 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c00e070 con 0x7fca441f6c00 2024-03-18T20:30:01.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.454+0000 7fca3a7fc700 1 -- 172.21.15.1:0/2613416545 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3c003c30 con 0x7fca441f6c00 2024-03-18T20:30:01.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.454+0000 7fca4becd700 1 -- 172.21.15.1:0/2613416545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca441f66e0 con 0x7fca441f6c00 2024-03-18T20:30:01.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.454+0000 7fca3a7fc700 1 -- 172.21.15.1:0/2613416545 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c003de0 con 0x7fca441f6c00 2024-03-18T20:30:01.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.454+0000 7fca4becd700 1 -- 172.21.15.1:0/2613416545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca441f70b0 con 0x7fca441f6c00 2024-03-18T20:30:01.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.454+0000 7fca4becd700 1 -- 172.21.15.1:0/2613416545 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fca4406e760 con 0x7fca441f6c00 2024-03-18T20:30:01.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.455+0000 7fca3a7fc700 1 -- 172.21.15.1:0/2613416545 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fca3c011a10 con 0x7fca441f6c00 2024-03-18T20:30:01.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.455+0000 7fca3a7fc700 1 --2- 172.21.15.1:0/2613416545 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fca2806ade0 0x7fca2806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.456+0000 7fca3a7fc700 1 -- 172.21.15.1:0/2613416545 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7fca3c08a610 con 0x7fca441f6c00 2024-03-18T20:30:01.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.456+0000 7fca49c69700 1 --2- 172.21.15.1:0/2613416545 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fca2806ade0 0x7fca2806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.456+0000 7fca3a7fc700 1 --2- 172.21.15.1:0/2613416545 >> [v2:172.21.15.165:6800/77475,v1:172.21.15.165:6801/77475] conn(0x7fca28071780 0x7fca28073c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.456+0000 7fca3a7fc700 1 -- 172.21.15.1:0/2613416545 --> [v2:172.21.15.165:6800/77475,v1:172.21.15.165:6801/77475] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fca280742d0 con 0x7fca28071780 2024-03-18T20:30:01.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.456+0000 7fca3a7fc700 1 -- 172.21.15.1:0/2613416545 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=295) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca3c08aa60 con 0x7fca441f6c00 2024-03-18T20:30:01.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.456+0000 7fca49c69700 1 --2- 172.21.15.1:0/2613416545 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fca2806ade0 0x7fca2806d2a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fca34009be0 tx=0x7fca34009210 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:01.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.456+0000 7fca49468700 1 --2- 172.21.15.1:0/2613416545 >> [v2:172.21.15.165:6800/77475,v1:172.21.15.165:6801/77475] conn(0x7fca28071780 0x7fca28073c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.456+0000 7fca49468700 1 --2- 172.21.15.1:0/2613416545 >> [v2:172.21.15.165:6800/77475,v1:172.21.15.165:6801/77475] conn(0x7fca28071780 0x7fca28073c00 crc :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:01.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.457+0000 7fca3a7fc700 1 -- 172.21.15.1:0/2613416545 <== osd.5 v2:172.21.15.165:6800/77475 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fca280742d0 con 0x7fca28071780 2024-03-18T20:30:01.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.472+0000 7fca4becd700 1 -- 172.21.15.1:0/2613416545 --> [v2:172.21.15.165:6800/77475,v1:172.21.15.165:6801/77475] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fca4415ea10 con 0x7fca28071780 2024-03-18T20:30:01.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.472+0000 7fca3a7fc700 1 -- 172.21.15.1:0/2613416545 <== osd.5 v2:172.21.15.165:6800/77475 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fca4000ce30 con 0x7fca28071780 2024-03-18T20:30:01.478 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:30:01.478 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T20:30:01.479 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:30:01.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.473+0000 7fca4becd700 1 -- 172.21.15.1:0/2613416545 >> [v2:172.21.15.165:6800/77475,v1:172.21.15.165:6801/77475] conn(0x7fca28071780 msgr2=0x7fca28073c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.473+0000 7fca4becd700 1 --2- 172.21.15.1:0/2613416545 >> [v2:172.21.15.165:6800/77475,v1:172.21.15.165:6801/77475] conn(0x7fca28071780 0x7fca28073c00 crc :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.473+0000 7fca4becd700 1 -- 172.21.15.1:0/2613416545 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fca2806ade0 msgr2=0x7fca2806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.473+0000 7fca4becd700 1 --2- 172.21.15.1:0/2613416545 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fca2806ade0 0x7fca2806d2a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fca34009be0 tx=0x7fca34009210 comp rx=0 tx=0).stop 2024-03-18T20:30:01.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.473+0000 7fca4becd700 1 -- 172.21.15.1:0/2613416545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca441f6c00 msgr2=0x7fca44158be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.473+0000 7fca4becd700 1 --2- 172.21.15.1:0/2613416545 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca441f6c00 0x7fca44158be0 secure :-1 s=READY pgs=3443 cs=0 l=1 rev1=1 crypto rx=0x7fca440ce620 tx=0x7fca3c0114b0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.473+0000 7fca4a46a700 1 -- 172.21.15.1:0/2613416545 reap_dead start 2024-03-18T20:30:01.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.473+0000 7fca4becd700 1 -- 172.21.15.1:0/2613416545 shutdown_connections 2024-03-18T20:30:01.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.473+0000 7fca4becd700 1 -- 172.21.15.1:0/2613416545 >> 172.21.15.1:0/2613416545 conn(0x7fca44099a30 msgr2=0x7fca440cfa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:01.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.473+0000 7fca4becd700 1 -- 172.21.15.1:0/2613416545 shutdown_connections 2024-03-18T20:30:01.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.473+0000 7fca4becd700 1 -- 172.21.15.1:0/2613416545 wait complete. 2024-03-18T20:30:01.488 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-18T20:30:01.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.610+0000 7fad9dc39700 1 Processor -- start 2024-03-18T20:30:01.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.611+0000 7fad9dc39700 1 -- start start 2024-03-18T20:30:01.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.611+0000 7fad9dc39700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fad980b1a40 0x7fad980b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.611+0000 7fad9dc39700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad980afd90 con 0x7fad98099a30 2024-03-18T20:30:01.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.611+0000 7fad9dc39700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad980b2a50 con 0x7fad980b0f50 2024-03-18T20:30:01.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.611+0000 7fad9dc39700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad980aeba0 con 0x7fad980b1a40 2024-03-18T20:30:01.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.611+0000 7fad97fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fad980b1a40 0x7fad980b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.612+0000 7fad97fff700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fad980b1a40 0x7fad980b32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50264/0 (socket says 172.21.15.1:50264) 2024-03-18T20:30:01.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.612+0000 7fad97fff700 1 -- 172.21.15.1:0/1262052477 learned_addr learned my addr 172.21.15.1:0/1262052477 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:01.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.612+0000 7fad96ffd700 1 --1- 172.21.15.1:0/1262052477 >> v1:172.21.15.1:6789/0 conn(0x7fad98099a30 0x7fad980b1330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59308/0 (socket says 172.21.15.1:59308) 2024-03-18T20:30:01.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.612+0000 7fad97fff700 1 -- 172.21.15.1:0/1262052477 >> v1:172.21.15.165:6789/0 conn(0x7fad980b0f50 legacy=0x7fad980af680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:01.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.612+0000 7fad97fff700 1 -- 172.21.15.1:0/1262052477 >> v1:172.21.15.1:6789/0 conn(0x7fad98099a30 legacy=0x7fad980b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.612+0000 7fad97fff700 1 -- 172.21.15.1:0/1262052477 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad980aee80 con 0x7fad980b1a40 2024-03-18T20:30:01.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.612+0000 7fad97fff700 1 --2- 172.21.15.1:0/1262052477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad980b1a40 0x7fad980b32d0 secure :-1 s=READY pgs=3445 cs=0 l=1 rev1=1 crypto rx=0x7fad90009e50 tx=0x7fad9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=306a8ab403c6e8fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:01.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.612+0000 7fad967fc700 1 -- 172.21.15.1:0/1262052477 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9000e070 con 0x7fad980b1a40 2024-03-18T20:30:01.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.612+0000 7fad967fc700 1 -- 172.21.15.1:0/1262052477 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad90003a20 con 0x7fad980b1a40 2024-03-18T20:30:01.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.612+0000 7fad967fc700 1 -- 172.21.15.1:0/1262052477 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90003dd0 con 0x7fad980b1a40 2024-03-18T20:30:01.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.613+0000 7fad9dc39700 1 -- 172.21.15.1:0/1262052477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad980b1a40 msgr2=0x7fad980b32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.613+0000 7fad9dc39700 1 --2- 172.21.15.1:0/1262052477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad980b1a40 0x7fad980b32d0 secure :-1 s=READY pgs=3445 cs=0 l=1 rev1=1 crypto rx=0x7fad90009e50 tx=0x7fad9000b330 comp rx=0 tx=0).stop 2024-03-18T20:30:01.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.613+0000 7fad9dc39700 1 -- 172.21.15.1:0/1262052477 shutdown_connections 2024-03-18T20:30:01.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.613+0000 7fad9dc39700 1 --2- 172.21.15.1:0/1262052477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad980b1a40 0x7fad980b32d0 unknown :-1 s=CLOSED pgs=3445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.613+0000 7fad9dc39700 1 -- 172.21.15.1:0/1262052477 >> 172.21.15.1:0/1262052477 conn(0x7fad980b5a10 msgr2=0x7fad980b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:01.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.613+0000 7fad9dc39700 1 -- 172.21.15.1:0/1262052477 shutdown_connections 2024-03-18T20:30:01.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.613+0000 7fad9dc39700 1 -- 172.21.15.1:0/1262052477 wait complete. 2024-03-18T20:30:01.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.614+0000 7fad9dc39700 1 Processor -- start 2024-03-18T20:30:01.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.614+0000 7fad9dc39700 1 -- start start 2024-03-18T20:30:01.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.615+0000 7fad9dc39700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad98099a30 0x7fad981e7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.615+0000 7fad9dc39700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fad980b0f50 0x7fad98145470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.615+0000 7fad9dc39700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad980b1a40 0x7fad98159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.615+0000 7fad9dc39700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad981582b0 con 0x7fad980b1a40 2024-03-18T20:30:01.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.615+0000 7fad9dc39700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad98158550 con 0x7fad98099a30 2024-03-18T20:30:01.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.615+0000 7fad9dc39700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad98158830 con 0x7fad980b0f50 2024-03-18T20:30:01.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.615+0000 7fad97fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad980b1a40 0x7fad98159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.615+0000 7fad97fff700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad980b1a40 0x7fad98159310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50280/0 (socket says 172.21.15.1:50280) 2024-03-18T20:30:01.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.615+0000 7fad97fff700 1 -- 172.21.15.1:0/2537754260 learned_addr learned my addr 172.21.15.1:0/2537754260 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:01.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.615+0000 7fad977fe700 1 --2- 172.21.15.1:0/2537754260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad98099a30 0x7fad981e7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.615+0000 7fad96ffd700 1 --2- 172.21.15.1:0/2537754260 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fad980b0f50 0x7fad98145470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.615+0000 7fad97fff700 1 -- 172.21.15.1:0/2537754260 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fad980b0f50 msgr2=0x7fad98145470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.615+0000 7fad97fff700 1 --2- 172.21.15.1:0/2537754260 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fad980b0f50 0x7fad98145470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.615+0000 7fad97fff700 1 -- 172.21.15.1:0/2537754260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad98099a30 msgr2=0x7fad981e7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.615+0000 7fad97fff700 1 --2- 172.21.15.1:0/2537754260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad98099a30 0x7fad981e7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.615+0000 7fad97fff700 1 -- 172.21.15.1:0/2537754260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad90009b30 con 0x7fad980b1a40 2024-03-18T20:30:01.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.615+0000 7fad97fff700 1 --2- 172.21.15.1:0/2537754260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad980b1a40 0x7fad98159310 secure :-1 s=READY pgs=3446 cs=0 l=1 rev1=1 crypto rx=0x7fad90015040 tx=0x7fad90004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:01.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.616+0000 7fad7ffff700 1 -- 172.21.15.1:0/2537754260 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90004240 con 0x7fad980b1a40 2024-03-18T20:30:01.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.616+0000 7fad7ffff700 1 -- 172.21.15.1:0/2537754260 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad900043a0 con 0x7fad980b1a40 2024-03-18T20:30:01.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.616+0000 7fad9dc39700 1 -- 172.21.15.1:0/2537754260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad98158b10 con 0x7fad980b1a40 2024-03-18T20:30:01.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.616+0000 7fad7ffff700 1 -- 172.21.15.1:0/2537754260 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90004730 con 0x7fad980b1a40 2024-03-18T20:30:01.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.616+0000 7fad9dc39700 1 -- 172.21.15.1:0/2537754260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad981e76a0 con 0x7fad980b1a40 2024-03-18T20:30:01.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.616+0000 7fad9dc39700 1 -- 172.21.15.1:0/2537754260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad98159030 con 0x7fad980b1a40 2024-03-18T20:30:01.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.617+0000 7fad7ffff700 1 -- 172.21.15.1:0/2537754260 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fad90019070 con 0x7fad980b1a40 2024-03-18T20:30:01.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.617+0000 7fad7ffff700 1 --2- 172.21.15.1:0/2537754260 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fad8406adb0 0x7fad8406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.618+0000 7fad7ffff700 1 -- 172.21.15.1:0/2537754260 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7fad9008a6d0 con 0x7fad980b1a40 2024-03-18T20:30:01.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.618+0000 7fad977fe700 1 --2- 172.21.15.1:0/2537754260 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fad8406adb0 0x7fad8406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.618+0000 7fad977fe700 1 --2- 172.21.15.1:0/2537754260 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fad8406adb0 0x7fad8406d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fad88009870 tx=0x7fad88008040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:01.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.622+0000 7fad7ffff700 1 -- 172.21.15.1:0/2537754260 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fad98159030 con 0x7fad980b1a40 2024-03-18T20:30:01.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.826+0000 7fad9dc39700 1 -- 172.21.15.1:0/2537754260 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fad980b26e0 con 0x7fad8406adb0 2024-03-18T20:30:01.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.826+0000 7fad7ffff700 1 -- 172.21.15.1:0/2537754260 <== mgr.14091 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fad88014430 con 0x7fad8406adb0 2024-03-18T20:30:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.828+0000 7fad9dc39700 1 -- 172.21.15.1:0/2537754260 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fad8406adb0 msgr2=0x7fad8406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.828+0000 7fad9dc39700 1 --2- 172.21.15.1:0/2537754260 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fad8406adb0 0x7fad8406d270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fad88009870 tx=0x7fad88008040 comp rx=0 tx=0).stop 2024-03-18T20:30:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.828+0000 7fad9dc39700 1 -- 172.21.15.1:0/2537754260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad980b1a40 msgr2=0x7fad98159310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.828+0000 7fad9dc39700 1 --2- 172.21.15.1:0/2537754260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad980b1a40 0x7fad98159310 secure :-1 s=READY pgs=3446 cs=0 l=1 rev1=1 crypto rx=0x7fad90015040 tx=0x7fad90004ad0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.828+0000 7fad9dc39700 1 -- 172.21.15.1:0/2537754260 shutdown_connections 2024-03-18T20:30:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.828+0000 7fad9dc39700 1 --2- 172.21.15.1:0/2537754260 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fad8406adb0 0x7fad8406d270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.828+0000 7fad9dc39700 1 --2- 172.21.15.1:0/2537754260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad980b1a40 0x7fad98159310 unknown :-1 s=CLOSED pgs=3446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.828+0000 7fad9dc39700 1 --2- 172.21.15.1:0/2537754260 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad98099a30 0x7fad981e7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.828+0000 7fad9dc39700 1 --2- 172.21.15.1:0/2537754260 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fad980b0f50 0x7fad98145470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.828+0000 7fad9dc39700 1 -- 172.21.15.1:0/2537754260 >> 172.21.15.1:0/2537754260 conn(0x7fad980b5a10 msgr2=0x7fad980ce810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.829+0000 7fad9dc39700 1 -- 172.21.15.1:0/2537754260 shutdown_connections 2024-03-18T20:30:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.829+0000 7fad9dc39700 1 -- 172.21.15.1:0/2537754260 wait complete. 2024-03-18T20:30:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 5 to deep-scrub 2024-03-18T20:30:01.844 INFO:tasks.ceph:Scrubbing osd.6 2024-03-18T20:30:01.845 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-18T20:30:01.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.970+0000 7f9405078700 1 Processor -- start 2024-03-18T20:30:01.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.970+0000 7f9405078700 1 -- start start 2024-03-18T20:30:01.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.970+0000 7f9405078700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f94000ac1a0 0x7f9400072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.970+0000 7f9405078700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94000ac910 con 0x7f9400143600 2024-03-18T20:30:01.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f9405078700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f940015a600 con 0x7f940015a220 2024-03-18T20:30:01.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f9405078700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94000ac630 con 0x7f94000ac1a0 2024-03-18T20:30:01.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93fe59c700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9400143600 0x7f94000ade10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59316/0 (socket says 172.21.15.1:59316) 2024-03-18T20:30:01.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93fe59c700 1 -- 172.21.15.1:0/2911879413 learned_addr learned my addr 172.21.15.1:0/2911879413 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:30:01.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93ff59e700 1 --2- 172.21.15.1:0/2911879413 >> v2:172.21.15.165:3300/0 conn(0x7f94000ac1a0 0x7f9400072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93fdd9b700 1 -- 172.21.15.1:0/2911879413 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729598618 0 0) 0x7f94000ac910 con 0x7f9400143600 2024-03-18T20:30:01.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93fdd9b700 1 -- 172.21.15.1:0/2911879413 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93e80035b0 con 0x7f9400143600 2024-03-18T20:30:01.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93fdd9b700 1 -- 172.21.15.1:0/2911879413 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3538545407 0 0) 0x7f93e80035b0 con 0x7f9400143600 2024-03-18T20:30:01.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93fdd9b700 1 -- 172.21.15.1:0/2911879413 >> v2:172.21.15.165:3300/0 conn(0x7f94000ac1a0 msgr2=0x7f9400072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:01.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93fdd9b700 1 --2- 172.21.15.1:0/2911879413 >> v2:172.21.15.165:3300/0 conn(0x7f94000ac1a0 0x7f9400072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93fdd9b700 1 -- 172.21.15.1:0/2911879413 >> v1:172.21.15.165:6789/0 conn(0x7f940015a220 legacy=0x7f94000b37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93fdd9b700 1 -- 172.21.15.1:0/2911879413 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94000b2d00 con 0x7f9400143600 2024-03-18T20:30:01.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93fdd9b700 1 -- 172.21.15.1:0/2911879413 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f93f4001670 con 0x7f9400143600 2024-03-18T20:30:01.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93fdd9b700 1 -- 172.21.15.1:0/2911879413 >> v1:172.21.15.1:6789/0 conn(0x7f9400143600 legacy=0x7f94000ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93fdd9b700 1 --2- 172.21.15.1:0/2911879413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93e8003a00 0x7f93e8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93fdd9b700 1 --2- 172.21.15.1:0/2911879413 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93e8004850 0x7f93e8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93fdd9b700 1 --2- 172.21.15.1:0/2911879413 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93e8005210 0x7f93e800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93fdd9b700 1 -- 172.21.15.1:0/2911879413 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94000ac630 con 0x7f93e8003a00 2024-03-18T20:30:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93fdd9b700 1 -- 172.21.15.1:0/2911879413 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e800c1b0 con 0x7f93e8004850 2024-03-18T20:30:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93fdd9b700 1 -- 172.21.15.1:0/2911879413 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e800c370 con 0x7f93e8005210 2024-03-18T20:30:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93fed9d700 1 --2- 172.21.15.1:0/2911879413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93e8003a00 0x7f93e8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.971+0000 7f93fe59c700 1 --2- 172.21.15.1:0/2911879413 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93e8004850 0x7f93e8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.972+0000 7f93ff59e700 1 --2- 172.21.15.1:0/2911879413 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93e8005210 0x7f93e800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.972+0000 7f93fed9d700 1 -- 172.21.15.1:0/2911879413 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93e8005210 msgr2=0x7f93e800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.972+0000 7f93fed9d700 1 --2- 172.21.15.1:0/2911879413 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93e8005210 0x7f93e800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.972+0000 7f93fed9d700 1 -- 172.21.15.1:0/2911879413 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93e8004850 msgr2=0x7f93e8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.972+0000 7f93fed9d700 1 --2- 172.21.15.1:0/2911879413 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93e8004850 0x7f93e8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.972+0000 7f93fed9d700 1 -- 172.21.15.1:0/2911879413 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e80035b0 con 0x7f93e8003a00 2024-03-18T20:30:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.972+0000 7f93ff59e700 1 -- 172.21.15.1:0/2911879413 reap_dead start 2024-03-18T20:30:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.972+0000 7f93fed9d700 1 --2- 172.21.15.1:0/2911879413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93e8003a00 0x7f93e8003e20 secure :-1 s=READY pgs=3448 cs=0 l=1 rev1=1 crypto rx=0x7f93f0001800 tx=0x7f93e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1613a0fa6d0fa0c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.972+0000 7f93fdd9b700 1 -- 172.21.15.1:0/2911879413 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f00012f0 con 0x7f93e8003a00 2024-03-18T20:30:01.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.972+0000 7f93fdd9b700 1 -- 172.21.15.1:0/2911879413 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f00052f0 con 0x7f93e8003a00 2024-03-18T20:30:01.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.972+0000 7f93fdd9b700 1 -- 172.21.15.1:0/2911879413 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f0006860 con 0x7f93e8003a00 2024-03-18T20:30:01.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.972+0000 7f9405078700 1 -- 172.21.15.1:0/2911879413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93e8003a00 msgr2=0x7f93e8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.972+0000 7f9405078700 1 --2- 172.21.15.1:0/2911879413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93e8003a00 0x7f93e8003e20 secure :-1 s=READY pgs=3448 cs=0 l=1 rev1=1 crypto rx=0x7f93f0001800 tx=0x7f93e8005880 comp rx=0 tx=0).stop 2024-03-18T20:30:01.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.972+0000 7f9405078700 1 -- 172.21.15.1:0/2911879413 shutdown_connections 2024-03-18T20:30:01.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.972+0000 7f9405078700 1 --2- 172.21.15.1:0/2911879413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93e8003a00 0x7f93e8003e20 unknown :-1 s=CLOSED pgs=3448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.972+0000 7f9405078700 1 -- 172.21.15.1:0/2911879413 >> 172.21.15.1:0/2911879413 conn(0x7f940006abf0 msgr2=0x7f940006b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:01.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.973+0000 7f9405078700 1 -- 172.21.15.1:0/2911879413 shutdown_connections 2024-03-18T20:30:01.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.973+0000 7f9405078700 1 -- 172.21.15.1:0/2911879413 wait complete. 2024-03-18T20:30:01.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.974+0000 7f9405078700 1 Processor -- start 2024-03-18T20:30:01.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.974+0000 7f9405078700 1 -- start start 2024-03-18T20:30:01.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.975+0000 7f9405078700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93e8003a00 0x7f94000720e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.975+0000 7f9405078700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94001f4120 0x7f9400070e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.975+0000 7f9405078700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9400071360 0x7f94001f35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.975+0000 7f9405078700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9400156f10 con 0x7f9400071360 2024-03-18T20:30:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.975+0000 7f9405078700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9400156290 con 0x7f94001f4120 2024-03-18T20:30:01.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.975+0000 7f9405078700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9400156570 con 0x7f93e8003a00 2024-03-18T20:30:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.975+0000 7f93ff59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9400071360 0x7f94001f35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.975+0000 7f93ff59e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9400071360 0x7f94001f35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50296/0 (socket says 172.21.15.1:50296) 2024-03-18T20:30:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.975+0000 7f93ff59e700 1 -- 172.21.15.1:0/2116743200 learned_addr learned my addr 172.21.15.1:0/2116743200 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:01.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.975+0000 7f93fe59c700 1 --2- 172.21.15.1:0/2116743200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94001f4120 0x7f9400070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.975+0000 7f93fed9d700 1 --2- 172.21.15.1:0/2116743200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93e8003a00 0x7f94000720e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.975+0000 7f93ff59e700 1 -- 172.21.15.1:0/2116743200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93e8003a00 msgr2=0x7f94000720e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.975+0000 7f93ff59e700 1 --2- 172.21.15.1:0/2116743200 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f93e8003a00 0x7f94000720e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.975+0000 7f93ff59e700 1 -- 172.21.15.1:0/2116743200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94001f4120 msgr2=0x7f9400070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:01.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.975+0000 7f93ff59e700 1 --2- 172.21.15.1:0/2116743200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94001f4120 0x7f9400070e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:01.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.975+0000 7f93ff59e700 1 -- 172.21.15.1:0/2116743200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9400156850 con 0x7f9400071360 2024-03-18T20:30:01.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.975+0000 7f93ff59e700 1 --2- 172.21.15.1:0/2116743200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9400071360 0x7f94001f35d0 secure :-1 s=READY pgs=3449 cs=0 l=1 rev1=1 crypto rx=0x7f9400079e50 tx=0x7f93f80111d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.975+0000 7f93e77fe700 1 -- 172.21.15.1:0/2116743200 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f800e070 con 0x7f9400071360 2024-03-18T20:30:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.976+0000 7f93e77fe700 1 -- 172.21.15.1:0/2116743200 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f8003cf0 con 0x7f9400071360 2024-03-18T20:30:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.976+0000 7f93e77fe700 1 -- 172.21.15.1:0/2116743200 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f8003ea0 con 0x7f9400071360 2024-03-18T20:30:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.976+0000 7f9405078700 1 -- 172.21.15.1:0/2116743200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9400156b30 con 0x7f9400071360 2024-03-18T20:30:01.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.976+0000 7f9405078700 1 -- 172.21.15.1:0/2116743200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9400072620 con 0x7f9400071360 2024-03-18T20:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.976+0000 7f9405078700 1 -- 172.21.15.1:0/2116743200 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f94000b32c0 con 0x7f9400071360 2024-03-18T20:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.977+0000 7f93e77fe700 1 -- 172.21.15.1:0/2116743200 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f93f8011740 con 0x7f9400071360 2024-03-18T20:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.977+0000 7f93e77fe700 1 --2- 172.21.15.1:0/2116743200 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93ec06adb0 0x7f93ec06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.978+0000 7f93e77fe700 1 -- 172.21.15.1:0/2116743200 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f93f808a520 con 0x7f9400071360 2024-03-18T20:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.978+0000 7f93fed9d700 1 --2- 172.21.15.1:0/2116743200 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93ec06adb0 0x7f93ec06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.978+0000 7f93e77fe700 1 --2- 172.21.15.1:0/2116743200 >> [v2:172.21.15.165:6817/77477,v1:172.21.15.165:6819/77477] conn(0x7f93ec071750 0x7f93ec073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.978+0000 7f93e77fe700 1 -- 172.21.15.1:0/2116743200 --> [v2:172.21.15.165:6817/77477,v1:172.21.15.165:6819/77477] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f93ec0742a0 con 0x7f93ec071750 2024-03-18T20:30:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.978+0000 7f93e77fe700 1 -- 172.21.15.1:0/2116743200 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=295) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93f808a970 con 0x7f9400071360 2024-03-18T20:30:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.978+0000 7f93fed9d700 1 --2- 172.21.15.1:0/2116743200 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93ec06adb0 0x7f93ec06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f93f0000c00 tx=0x7f93f0009030 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.978+0000 7f93fe59c700 1 --2- 172.21.15.1:0/2116743200 >> [v2:172.21.15.165:6817/77477,v1:172.21.15.165:6819/77477] conn(0x7f93ec071750 0x7f93ec073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.978+0000 7f93fe59c700 1 --2- 172.21.15.1:0/2116743200 >> [v2:172.21.15.165:6817/77477,v1:172.21.15.165:6819/77477] conn(0x7f93ec071750 0x7f93ec073bd0 crc :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:01.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.979+0000 7f93e77fe700 1 -- 172.21.15.1:0/2116743200 <== osd.6 v2:172.21.15.165:6817/77477 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f93ec0742a0 con 0x7f93ec071750 2024-03-18T20:30:01.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.993+0000 7f9405078700 1 -- 172.21.15.1:0/2116743200 --> [v2:172.21.15.165:6817/77477,v1:172.21.15.165:6819/77477] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9400157050 con 0x7f93ec071750 2024-03-18T20:30:01.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.994+0000 7f93e77fe700 1 -- 172.21.15.1:0/2116743200 <== osd.6 v2:172.21.15.165:6817/77477 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f9400157050 con 0x7f93ec071750 2024-03-18T20:30:01.998 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:30:01.998 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T20:30:01.999 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:30:02.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.994+0000 7f9405078700 1 -- 172.21.15.1:0/2116743200 >> [v2:172.21.15.165:6817/77477,v1:172.21.15.165:6819/77477] conn(0x7f93ec071750 msgr2=0x7f93ec073bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.994+0000 7f9405078700 1 --2- 172.21.15.1:0/2116743200 >> [v2:172.21.15.165:6817/77477,v1:172.21.15.165:6819/77477] conn(0x7f93ec071750 0x7f93ec073bd0 crc :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.995+0000 7f9405078700 1 -- 172.21.15.1:0/2116743200 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93ec06adb0 msgr2=0x7f93ec06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.995+0000 7f9405078700 1 --2- 172.21.15.1:0/2116743200 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f93ec06adb0 0x7f93ec06d270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f93f0000c00 tx=0x7f93f0009030 comp rx=0 tx=0).stop 2024-03-18T20:30:02.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.995+0000 7f9405078700 1 -- 172.21.15.1:0/2116743200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9400071360 msgr2=0x7f94001f35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.995+0000 7f9405078700 1 --2- 172.21.15.1:0/2116743200 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9400071360 0x7f94001f35d0 secure :-1 s=READY pgs=3449 cs=0 l=1 rev1=1 crypto rx=0x7f9400079e50 tx=0x7f93f80111d0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.995+0000 7f93ff59e700 1 -- 172.21.15.1:0/2116743200 reap_dead start 2024-03-18T20:30:02.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.995+0000 7f9405078700 1 -- 172.21.15.1:0/2116743200 shutdown_connections 2024-03-18T20:30:02.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.995+0000 7f9405078700 1 -- 172.21.15.1:0/2116743200 >> 172.21.15.1:0/2116743200 conn(0x7f940006abf0 msgr2=0x7f9400142be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:02.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.995+0000 7f9405078700 1 -- 172.21.15.1:0/2116743200 shutdown_connections 2024-03-18T20:30:02.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:01.995+0000 7f9405078700 1 -- 172.21.15.1:0/2116743200 wait complete. 2024-03-18T20:30:02.010 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-18T20:30:02.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.132+0000 7fe204f0e700 1 Processor -- start 2024-03-18T20:30:02.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.133+0000 7fe204f0e700 1 -- start start 2024-03-18T20:30:02.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe204f0e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2000b35e0 0x7fe2000ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe204f0e700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2000b1960 con 0x7fe200072050 2024-03-18T20:30:02.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe204f0e700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2000b1bb0 con 0x7fe20015d8f0 2024-03-18T20:30:02.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe204f0e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2000adb70 con 0x7fe2000b35e0 2024-03-18T20:30:02.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe1fed9d700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe200072050 0x7fe2000ae8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59324/0 (socket says 172.21.15.1:59324) 2024-03-18T20:30:02.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe1fed9d700 1 -- 172.21.15.1:0/781087886 learned_addr learned my addr 172.21.15.1:0/781087886 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:30:02.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe1fdd9b700 1 --2- 172.21.15.1:0/781087886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2000b35e0 0x7fe2000ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe1fd59a700 1 -- 172.21.15.1:0/781087886 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121982399 0 0) 0x7fe2000b1960 con 0x7fe200072050 2024-03-18T20:30:02.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe1fd59a700 1 -- 172.21.15.1:0/781087886 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1ec0035b0 con 0x7fe200072050 2024-03-18T20:30:02.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe1fd59a700 1 -- 172.21.15.1:0/781087886 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3552427262 0 0) 0x7fe1ec0035b0 con 0x7fe200072050 2024-03-18T20:30:02.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe1fd59a700 1 -- 172.21.15.1:0/781087886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2000b35e0 msgr2=0x7fe2000ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:02.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe1fd59a700 1 --2- 172.21.15.1:0/781087886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe2000b35e0 0x7fe2000ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe1fd59a700 1 -- 172.21.15.1:0/781087886 >> v1:172.21.15.165:6789/0 conn(0x7fe20015d8f0 legacy=0x7fe2000b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe1fd59a700 1 -- 172.21.15.1:0/781087886 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2000ade50 con 0x7fe200072050 2024-03-18T20:30:02.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe1fd59a700 1 -- 172.21.15.1:0/781087886 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7fe1f80019f0 con 0x7fe200072050 2024-03-18T20:30:02.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe1fd59a700 1 -- 172.21.15.1:0/781087886 >> v1:172.21.15.1:6789/0 conn(0x7fe200072050 legacy=0x7fe2000ae8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe1fd59a700 1 --2- 172.21.15.1:0/781087886 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1ec003a30 0x7fe1ec003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe1fd59a700 1 --2- 172.21.15.1:0/781087886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1ec0043e0 0x7fe1ec004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe1fd59a700 1 --2- 172.21.15.1:0/781087886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1ec004fd0 0x7fe1ec00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe1fd59a700 1 -- 172.21.15.1:0/781087886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2000adb70 con 0x7fe1ec004fd0 2024-03-18T20:30:02.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe1fd59a700 1 -- 172.21.15.1:0/781087886 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec0056f0 con 0x7fe1ec003a30 2024-03-18T20:30:02.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe1fd59a700 1 -- 172.21.15.1:0/781087886 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec00bfa0 con 0x7fe1ec0043e0 2024-03-18T20:30:02.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.134+0000 7fe1fed9d700 1 --2- 172.21.15.1:0/781087886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1ec004fd0 0x7fe1ec00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.135+0000 7fe1fdd9b700 1 --2- 172.21.15.1:0/781087886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1ec0043e0 0x7fe1ec004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.135+0000 7fe1fe59c700 1 --2- 172.21.15.1:0/781087886 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1ec003a30 0x7fe1ec003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.135+0000 7fe1fed9d700 1 -- 172.21.15.1:0/781087886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1ec0043e0 msgr2=0x7fe1ec004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:02.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.135+0000 7fe1fed9d700 1 --2- 172.21.15.1:0/781087886 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe1ec0043e0 0x7fe1ec004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.135+0000 7fe1fed9d700 1 -- 172.21.15.1:0/781087886 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1ec003a30 msgr2=0x7fe1ec003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:02.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.135+0000 7fe1fed9d700 1 --2- 172.21.15.1:0/781087886 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1ec003a30 0x7fe1ec003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.135+0000 7fe1fed9d700 1 -- 172.21.15.1:0/781087886 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ec0035b0 con 0x7fe1ec004fd0 2024-03-18T20:30:02.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.135+0000 7fe1fed9d700 1 --2- 172.21.15.1:0/781087886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1ec004fd0 0x7fe1ec00ba10 secure :-1 s=READY pgs=3451 cs=0 l=1 rev1=1 crypto rx=0x7fe2000b1eb0 tx=0x7fe1ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3ecd20291eeb18f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:02.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.135+0000 7fe1fd59a700 1 -- 172.21.15.1:0/781087886 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8012600 con 0x7fe1ec004fd0 2024-03-18T20:30:02.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.135+0000 7fe1fd59a700 1 -- 172.21.15.1:0/781087886 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f8012780 con 0x7fe1ec004fd0 2024-03-18T20:30:02.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.135+0000 7fe1fd59a700 1 -- 172.21.15.1:0/781087886 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8012b30 con 0x7fe1ec004fd0 2024-03-18T20:30:02.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.136+0000 7fe204f0e700 1 -- 172.21.15.1:0/781087886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1ec004fd0 msgr2=0x7fe1ec00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.136+0000 7fe204f0e700 1 --2- 172.21.15.1:0/781087886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1ec004fd0 0x7fe1ec00ba10 secure :-1 s=READY pgs=3451 cs=0 l=1 rev1=1 crypto rx=0x7fe2000b1eb0 tx=0x7fe1ec005880 comp rx=0 tx=0).stop 2024-03-18T20:30:02.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.136+0000 7fe1fed9d700 1 -- 172.21.15.1:0/781087886 reap_dead start 2024-03-18T20:30:02.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.136+0000 7fe204f0e700 1 -- 172.21.15.1:0/781087886 shutdown_connections 2024-03-18T20:30:02.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.136+0000 7fe204f0e700 1 -- 172.21.15.1:0/781087886 >> 172.21.15.1:0/781087886 conn(0x7fe2000b5a10 msgr2=0x7fe2000b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:02.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.136+0000 7fe204f0e700 1 -- 172.21.15.1:0/781087886 shutdown_connections 2024-03-18T20:30:02.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.136+0000 7fe204f0e700 1 -- 172.21.15.1:0/781087886 wait complete. 2024-03-18T20:30:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.137+0000 7fe204f0e700 1 Processor -- start 2024-03-18T20:30:02.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.137+0000 7fe204f0e700 1 -- start start 2024-03-18T20:30:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.138+0000 7fe204f0e700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1ec004fd0 0x7fe200158ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.138+0000 7fe204f0e700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe200152ed0 0x7fe200159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.138+0000 7fe204f0e700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe200159c60 0x7fe2001f33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.138+0000 7fe204f0e700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20015a110 con 0x7fe200152ed0 2024-03-18T20:30:02.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.138+0000 7fe204f0e700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe200158a10 con 0x7fe1ec004fd0 2024-03-18T20:30:02.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.138+0000 7fe204f0e700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2001f4160 con 0x7fe200159c60 2024-03-18T20:30:02.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.138+0000 7fe1fdd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe200152ed0 0x7fe200159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.138+0000 7fe1fdd9b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe200152ed0 0x7fe200159720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50306/0 (socket says 172.21.15.1:50306) 2024-03-18T20:30:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.138+0000 7fe1fdd9b700 1 -- 172.21.15.1:0/493997369 learned_addr learned my addr 172.21.15.1:0/493997369 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.138+0000 7fe1fe59c700 1 --2- 172.21.15.1:0/493997369 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1ec004fd0 0x7fe200158ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.138+0000 7fe1fed9d700 1 --2- 172.21.15.1:0/493997369 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe200159c60 0x7fe2001f33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.138+0000 7fe1fdd9b700 1 -- 172.21.15.1:0/493997369 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe200159c60 msgr2=0x7fe2001f33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.138+0000 7fe1fdd9b700 1 --2- 172.21.15.1:0/493997369 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe200159c60 0x7fe2001f33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.138+0000 7fe1fdd9b700 1 -- 172.21.15.1:0/493997369 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1ec004fd0 msgr2=0x7fe200158ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.138+0000 7fe1fdd9b700 1 --2- 172.21.15.1:0/493997369 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1ec004fd0 0x7fe200158ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.138+0000 7fe1fdd9b700 1 -- 172.21.15.1:0/493997369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2001580e0 con 0x7fe200152ed0 2024-03-18T20:30:02.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.138+0000 7fe1fed9d700 1 --2- 172.21.15.1:0/493997369 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe200159c60 0x7fe2001f33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:30:02.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.138+0000 7fe1fdd9b700 1 --2- 172.21.15.1:0/493997369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe200152ed0 0x7fe200159720 secure :-1 s=READY pgs=3452 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8011aa0 tx=0x7fe1e8011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:02.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.139+0000 7fe1f6ffd700 1 -- 172.21.15.1:0/493997369 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e800f940 con 0x7fe200152ed0 2024-03-18T20:30:02.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.139+0000 7fe1f6ffd700 1 -- 172.21.15.1:0/493997369 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e800faa0 con 0x7fe200152ed0 2024-03-18T20:30:02.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.139+0000 7fe204f0e700 1 -- 172.21.15.1:0/493997369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2001583c0 con 0x7fe200152ed0 2024-03-18T20:30:02.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.139+0000 7fe1f6ffd700 1 -- 172.21.15.1:0/493997369 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e800fe50 con 0x7fe200152ed0 2024-03-18T20:30:02.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.139+0000 7fe204f0e700 1 -- 172.21.15.1:0/493997369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe200153350 con 0x7fe200152ed0 2024-03-18T20:30:02.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.139+0000 7fe204f0e700 1 -- 172.21.15.1:0/493997369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe200067dc0 con 0x7fe200152ed0 2024-03-18T20:30:02.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.140+0000 7fe1f6ffd700 1 -- 172.21.15.1:0/493997369 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fe1e8014440 con 0x7fe200152ed0 2024-03-18T20:30:02.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.140+0000 7fe1f6ffd700 1 --2- 172.21.15.1:0/493997369 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe1e006adb0 0x7fe1e006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.141+0000 7fe1f6ffd700 1 -- 172.21.15.1:0/493997369 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7fe1e8089da0 con 0x7fe200152ed0 2024-03-18T20:30:02.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.141+0000 7fe1fe59c700 1 --2- 172.21.15.1:0/493997369 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe1e006adb0 0x7fe1e006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.141+0000 7fe1fe59c700 1 --2- 172.21.15.1:0/493997369 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe1e006adb0 0x7fe1e006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe2001f6750 tx=0x7fe1f0009040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:02.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.145+0000 7fe1f6ffd700 1 -- 172.21.15.1:0/493997369 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe1e8054e60 con 0x7fe200152ed0 2024-03-18T20:30:02.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.349+0000 7fe204f0e700 1 -- 172.21.15.1:0/493997369 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fe200099a30 con 0x7fe1e006adb0 2024-03-18T20:30:02.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.349+0000 7fe1f6ffd700 1 -- 172.21.15.1:0/493997369 <== mgr.14091 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe1f0006860 con 0x7fe1e006adb0 2024-03-18T20:30:02.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.351+0000 7fe204f0e700 1 -- 172.21.15.1:0/493997369 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe1e006adb0 msgr2=0x7fe1e006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.351+0000 7fe204f0e700 1 --2- 172.21.15.1:0/493997369 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe1e006adb0 0x7fe1e006d270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe2001f6750 tx=0x7fe1f0009040 comp rx=0 tx=0).stop 2024-03-18T20:30:02.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.351+0000 7fe204f0e700 1 -- 172.21.15.1:0/493997369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe200152ed0 msgr2=0x7fe200159720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.351+0000 7fe204f0e700 1 --2- 172.21.15.1:0/493997369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe200152ed0 0x7fe200159720 secure :-1 s=READY pgs=3452 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8011aa0 tx=0x7fe1e8011db0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.351+0000 7fe204f0e700 1 -- 172.21.15.1:0/493997369 shutdown_connections 2024-03-18T20:30:02.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.351+0000 7fe204f0e700 1 --2- 172.21.15.1:0/493997369 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fe1e006adb0 0x7fe1e006d270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.351+0000 7fe204f0e700 1 --2- 172.21.15.1:0/493997369 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fe200159c60 0x7fe2001f33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.351+0000 7fe204f0e700 1 --2- 172.21.15.1:0/493997369 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1ec004fd0 0x7fe200158ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.351+0000 7fe204f0e700 1 --2- 172.21.15.1:0/493997369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe200152ed0 0x7fe200159720 unknown :-1 s=CLOSED pgs=3452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.351+0000 7fe204f0e700 1 -- 172.21.15.1:0/493997369 >> 172.21.15.1:0/493997369 conn(0x7fe2000b5a10 msgr2=0x7fe2000ae350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:02.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.351+0000 7fe204f0e700 1 -- 172.21.15.1:0/493997369 shutdown_connections 2024-03-18T20:30:02.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.351+0000 7fe204f0e700 1 -- 172.21.15.1:0/493997369 wait complete. 2024-03-18T20:30:02.358 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 6 to deep-scrub 2024-03-18T20:30:02.367 INFO:tasks.ceph:Scrubbing osd.7 2024-03-18T20:30:02.367 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-18T20:30:02.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.492+0000 7fc890ff5700 1 Processor -- start 2024-03-18T20:30:02.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.492+0000 7fc890ff5700 1 -- start start 2024-03-18T20:30:02.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.493+0000 7fc890ff5700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc88c0cd5c0 0x7fc88c0cda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.493+0000 7fc890ff5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc88c0d0140 0x7fc88c0d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.493+0000 7fc890ff5700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc88c0cd3e0 con 0x7fc88c0cd000 2024-03-18T20:30:02.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.493+0000 7fc890ff5700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc88c15ed80 con 0x7fc88c0cd5c0 2024-03-18T20:30:02.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.493+0000 7fc890ff5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c0d0b10 con 0x7fc88c0d0140 2024-03-18T20:30:02.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.493+0000 7fc889d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc88c0cd5c0 0x7fc88c0cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.493+0000 7fc889d9b700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc88c0cd5c0 0x7fc88c0cda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50320/0 (socket says 172.21.15.1:50320) 2024-03-18T20:30:02.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.493+0000 7fc889d9b700 1 -- 172.21.15.1:0/1030730890 learned_addr learned my addr 172.21.15.1:0/1030730890 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:02.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.493+0000 7fc88ad9d700 1 --2- 172.21.15.1:0/1030730890 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc88c0d0140 0x7fc88c0d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.493+0000 7fc889d9b700 1 -- 172.21.15.1:0/1030730890 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc88c0d0140 msgr2=0x7fc88c0d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:02.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.493+0000 7fc889d9b700 1 --2- 172.21.15.1:0/1030730890 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc88c0d0140 0x7fc88c0d05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.493+0000 7fc889d9b700 1 -- 172.21.15.1:0/1030730890 >> v1:172.21.15.1:6789/0 conn(0x7fc88c0cd000 legacy=0x7fc88c06e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.493+0000 7fc889d9b700 1 -- 172.21.15.1:0/1030730890 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88c15e170 con 0x7fc88c0cd5c0 2024-03-18T20:30:02.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.494+0000 7fc889d9b700 1 --2- 172.21.15.1:0/1030730890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc88c0cd5c0 0x7fc88c0cda40 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 crypto rx=0x7fc87c0098f0 tx=0x7fc87c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc7b2d90c42d5d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:02.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.494+0000 7fc88959a700 1 -- 172.21.15.1:0/1030730890 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c00b8a0 con 0x7fc88c0cd5c0 2024-03-18T20:30:02.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.494+0000 7fc88959a700 1 -- 172.21.15.1:0/1030730890 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87c00ba00 con 0x7fc88c0cd5c0 2024-03-18T20:30:02.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.494+0000 7fc88959a700 1 -- 172.21.15.1:0/1030730890 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c00bd90 con 0x7fc88c0cd5c0 2024-03-18T20:30:02.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.494+0000 7fc890ff5700 1 -- 172.21.15.1:0/1030730890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc88c0cd5c0 msgr2=0x7fc88c0cda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.494+0000 7fc890ff5700 1 --2- 172.21.15.1:0/1030730890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc88c0cd5c0 0x7fc88c0cda40 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 crypto rx=0x7fc87c0098f0 tx=0x7fc87c009c00 comp rx=0 tx=0).stop 2024-03-18T20:30:02.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.494+0000 7fc890ff5700 1 -- 172.21.15.1:0/1030730890 shutdown_connections 2024-03-18T20:30:02.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.494+0000 7fc890ff5700 1 --2- 172.21.15.1:0/1030730890 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc88c0d0140 0x7fc88c0d05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.494+0000 7fc890ff5700 1 --2- 172.21.15.1:0/1030730890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc88c0cd5c0 0x7fc88c0cda40 unknown :-1 s=CLOSED pgs=3454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.494+0000 7fc890ff5700 1 -- 172.21.15.1:0/1030730890 >> 172.21.15.1:0/1030730890 conn(0x7fc88c099a30 msgr2=0x7fc88c0b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:02.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.494+0000 7fc890ff5700 1 -- 172.21.15.1:0/1030730890 shutdown_connections 2024-03-18T20:30:02.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.494+0000 7fc890ff5700 1 -- 172.21.15.1:0/1030730890 wait complete. 2024-03-18T20:30:02.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.496+0000 7fc890ff5700 1 Processor -- start 2024-03-18T20:30:02.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.496+0000 7fc890ff5700 1 -- start start 2024-03-18T20:30:02.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc890ff5700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc88c0cd000 0x7fc88c09dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc890ff5700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc88c0cd5c0 0x7fc88c157f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc890ff5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc88c0d0140 0x7fc88c0cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc890ff5700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c0caa10 con 0x7fc88c0cd000 2024-03-18T20:30:02.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc890ff5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c1584d0 con 0x7fc88c0d0140 2024-03-18T20:30:02.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc890ff5700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c0cbca0 con 0x7fc88c0cd5c0 2024-03-18T20:30:02.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc88a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc88c0cd000 0x7fc88c09dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc88a59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc88c0cd000 0x7fc88c09dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50330/0 (socket says 172.21.15.1:50330) 2024-03-18T20:30:02.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc88a59c700 1 -- 172.21.15.1:0/3176596544 learned_addr learned my addr 172.21.15.1:0/3176596544 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:02.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc88ad9d700 1 --2- 172.21.15.1:0/3176596544 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc88c0d0140 0x7fc88c0cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc889d9b700 1 --2- 172.21.15.1:0/3176596544 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc88c0cd5c0 0x7fc88c157f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc88a59c700 1 -- 172.21.15.1:0/3176596544 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc88c0cd5c0 msgr2=0x7fc88c157f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc88a59c700 1 --2- 172.21.15.1:0/3176596544 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc88c0cd5c0 0x7fc88c157f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc88a59c700 1 -- 172.21.15.1:0/3176596544 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc88c0d0140 msgr2=0x7fc88c0cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc88a59c700 1 --2- 172.21.15.1:0/3176596544 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc88c0d0140 0x7fc88c0cb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc88a59c700 1 -- 172.21.15.1:0/3176596544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c009550 con 0x7fc88c0cd000 2024-03-18T20:30:02.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc889d9b700 1 --2- 172.21.15.1:0/3176596544 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fc88c0cd5c0 0x7fc88c157f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:30:02.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc88ad9d700 1 --2- 172.21.15.1:0/3176596544 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc88c0d0140 0x7fc88c0cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:30:02.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc88a59c700 1 --2- 172.21.15.1:0/3176596544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc88c0cd000 0x7fc88c09dd00 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 crypto rx=0x7fc88000c450 tx=0x7fc88000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:02.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc87affd700 1 -- 172.21.15.1:0/3176596544 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8800052d0 con 0x7fc88c0cd000 2024-03-18T20:30:02.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc87affd700 1 -- 172.21.15.1:0/3176596544 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc880005430 con 0x7fc88c0cd000 2024-03-18T20:30:02.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc890ff5700 1 -- 172.21.15.1:0/3176596544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc88c0cbf20 con 0x7fc88c0cd000 2024-03-18T20:30:02.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc87affd700 1 -- 172.21.15.1:0/3176596544 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8800038c0 con 0x7fc88c0cd000 2024-03-18T20:30:02.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.497+0000 7fc890ff5700 1 -- 172.21.15.1:0/3176596544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc88c0cc4e0 con 0x7fc88c0cd000 2024-03-18T20:30:02.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.498+0000 7fc890ff5700 1 -- 172.21.15.1:0/3176596544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc88c15ea10 con 0x7fc88c0cd000 2024-03-18T20:30:02.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.499+0000 7fc87affd700 1 -- 172.21.15.1:0/3176596544 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fc880003a20 con 0x7fc88c0cd000 2024-03-18T20:30:02.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.499+0000 7fc87affd700 1 --2- 172.21.15.1:0/3176596544 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc86c06ade0 0x7fc86c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.499+0000 7fc87affd700 1 -- 172.21.15.1:0/3176596544 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7fc880088c20 con 0x7fc88c0cd000 2024-03-18T20:30:02.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.499+0000 7fc889d9b700 1 --2- 172.21.15.1:0/3176596544 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc86c06ade0 0x7fc86c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.500+0000 7fc87affd700 1 --2- 172.21.15.1:0/3176596544 >> [v2:172.21.15.165:6816/77476,v1:172.21.15.165:6818/77476] conn(0x7fc86c071780 0x7fc86c073c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.500+0000 7fc87affd700 1 -- 172.21.15.1:0/3176596544 --> [v2:172.21.15.165:6816/77476,v1:172.21.15.165:6818/77476] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc86c0742d0 con 0x7fc86c071780 2024-03-18T20:30:02.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.500+0000 7fc87affd700 1 -- 172.21.15.1:0/3176596544 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=295) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc880089070 con 0x7fc88c0cd000 2024-03-18T20:30:02.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.500+0000 7fc88ad9d700 1 --2- 172.21.15.1:0/3176596544 >> [v2:172.21.15.165:6816/77476,v1:172.21.15.165:6818/77476] conn(0x7fc86c071780 0x7fc86c073c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.500+0000 7fc889d9b700 1 --2- 172.21.15.1:0/3176596544 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc86c06ade0 0x7fc86c06d2a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc87c014040 tx=0x7fc87c008030 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:02.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.501+0000 7fc88ad9d700 1 --2- 172.21.15.1:0/3176596544 >> [v2:172.21.15.165:6816/77476,v1:172.21.15.165:6818/77476] conn(0x7fc86c071780 0x7fc86c073c00 crc :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:02.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.501+0000 7fc87affd700 1 -- 172.21.15.1:0/3176596544 <== osd.7 v2:172.21.15.165:6816/77476 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fc86c0742d0 con 0x7fc86c071780 2024-03-18T20:30:02.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.516+0000 7fc890ff5700 1 -- 172.21.15.1:0/3176596544 --> [v2:172.21.15.165:6816/77476,v1:172.21.15.165:6818/77476] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc88c15e730 con 0x7fc86c071780 2024-03-18T20:30:02.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.516+0000 7fc87affd700 1 -- 172.21.15.1:0/3176596544 <== osd.7 v2:172.21.15.165:6816/77476 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fc884009c30 con 0x7fc86c071780 2024-03-18T20:30:02.520 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-18T20:30:02.521 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T20:30:02.521 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-18T20:30:02.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.516+0000 7fc890ff5700 1 -- 172.21.15.1:0/3176596544 >> [v2:172.21.15.165:6816/77476,v1:172.21.15.165:6818/77476] conn(0x7fc86c071780 msgr2=0x7fc86c073c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.516+0000 7fc890ff5700 1 --2- 172.21.15.1:0/3176596544 >> [v2:172.21.15.165:6816/77476,v1:172.21.15.165:6818/77476] conn(0x7fc86c071780 0x7fc86c073c00 crc :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.517+0000 7fc890ff5700 1 -- 172.21.15.1:0/3176596544 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc86c06ade0 msgr2=0x7fc86c06d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.517+0000 7fc890ff5700 1 --2- 172.21.15.1:0/3176596544 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fc86c06ade0 0x7fc86c06d2a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc87c014040 tx=0x7fc87c008030 comp rx=0 tx=0).stop 2024-03-18T20:30:02.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.517+0000 7fc890ff5700 1 -- 172.21.15.1:0/3176596544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc88c0cd000 msgr2=0x7fc88c09dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.517+0000 7fc890ff5700 1 --2- 172.21.15.1:0/3176596544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc88c0cd000 0x7fc88c09dd00 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 crypto rx=0x7fc88000c450 tx=0x7fc88000c810 comp rx=0 tx=0).stop 2024-03-18T20:30:02.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.517+0000 7fc88ad9d700 1 -- 172.21.15.1:0/3176596544 reap_dead start 2024-03-18T20:30:02.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.517+0000 7fc890ff5700 1 -- 172.21.15.1:0/3176596544 shutdown_connections 2024-03-18T20:30:02.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.517+0000 7fc890ff5700 1 -- 172.21.15.1:0/3176596544 >> 172.21.15.1:0/3176596544 conn(0x7fc88c099a30 msgr2=0x7fc88c15eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:02.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.517+0000 7fc890ff5700 1 -- 172.21.15.1:0/3176596544 shutdown_connections 2024-03-18T20:30:02.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.517+0000 7fc890ff5700 1 -- 172.21.15.1:0/3176596544 wait complete. 2024-03-18T20:30:02.532 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-18T20:30:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.654+0000 7f420e472700 1 Processor -- start 2024-03-18T20:30:02.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.654+0000 7f420e472700 1 -- start start 2024-03-18T20:30:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.655+0000 7f420e472700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f420815f660 0x7f42081662e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.655+0000 7f420e472700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4208070ae0 con 0x7f4208070220 2024-03-18T20:30:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.655+0000 7f420e472700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4208070dc0 con 0x7f4208141020 2024-03-18T20:30:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.655+0000 7f420e472700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42080710a0 con 0x7f420815f660 2024-03-18T20:30:02.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.655+0000 7f42077fe700 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4208070220 0x7f420815ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59350/0 (socket says 172.21.15.1:59350) 2024-03-18T20:30:02.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.655+0000 7f42077fe700 1 -- 172.21.15.1:0/1664978006 learned_addr learned my addr 172.21.15.1:0/1664978006 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-03-18T20:30:02.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f4206ffd700 1 -- 172.21.15.1:0/1664978006 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078294388 0 0) 0x7f4208070ae0 con 0x7f4208070220 2024-03-18T20:30:02.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f420ca0f700 1 --2- 172.21.15.1:0/1664978006 >> v2:172.21.15.165:3300/0 conn(0x7f420815f660 0x7f42081662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f4206ffd700 1 -- 172.21.15.1:0/1664978006 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41f40035b0 con 0x7f4208070220 2024-03-18T20:30:02.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f4206ffd700 1 -- 172.21.15.1:0/1664978006 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1878007740 0 0) 0x7f41f40035b0 con 0x7f4208070220 2024-03-18T20:30:02.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f4206ffd700 1 -- 172.21.15.1:0/1664978006 >> v2:172.21.15.165:3300/0 conn(0x7f420815f660 msgr2=0x7f42081662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:02.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f4206ffd700 1 --2- 172.21.15.1:0/1664978006 >> v2:172.21.15.165:3300/0 conn(0x7f420815f660 0x7f42081662e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f4206ffd700 1 -- 172.21.15.1:0/1664978006 >> v1:172.21.15.165:6789/0 conn(0x7f4208141020 legacy=0x7f4208141400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f4206ffd700 1 -- 172.21.15.1:0/1664978006 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420813ead0 con 0x7f4208070220 2024-03-18T20:30:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f4206ffd700 1 -- 172.21.15.1:0/1664978006 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 129181672 0 0) 0x7f41f0001670 con 0x7f4208070220 2024-03-18T20:30:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f4206ffd700 1 -- 172.21.15.1:0/1664978006 >> v1:172.21.15.1:6789/0 conn(0x7f4208070220 legacy=0x7f420815ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f4206ffd700 1 --2- 172.21.15.1:0/1664978006 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41f4003a00 0x7f41f4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f4206ffd700 1 --2- 172.21.15.1:0/1664978006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41f4004850 0x7f41f4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f4206ffd700 1 --2- 172.21.15.1:0/1664978006 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41f4005210 0x7f41f400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f4206ffd700 1 -- 172.21.15.1:0/1664978006 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42080710a0 con 0x7f41f4004850 2024-03-18T20:30:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f4206ffd700 1 -- 172.21.15.1:0/1664978006 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f400c0b0 con 0x7f41f4005210 2024-03-18T20:30:02.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f4206ffd700 1 -- 172.21.15.1:0/1664978006 --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f400c270 con 0x7f41f4003a00 2024-03-18T20:30:02.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f42077fe700 1 --2- 172.21.15.1:0/1664978006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41f4004850 0x7f41f4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f4207fff700 1 --2- 172.21.15.1:0/1664978006 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41f4003a00 0x7f41f4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f420ca0f700 1 --2- 172.21.15.1:0/1664978006 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41f4005210 0x7f41f400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f42077fe700 1 -- 172.21.15.1:0/1664978006 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41f4003a00 msgr2=0x7f41f4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:02.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f42077fe700 1 --2- 172.21.15.1:0/1664978006 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f41f4003a00 0x7f41f4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f42077fe700 1 -- 172.21.15.1:0/1664978006 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41f4005210 msgr2=0x7f41f400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:02.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f42077fe700 1 --2- 172.21.15.1:0/1664978006 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41f4005210 0x7f41f400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f42077fe700 1 -- 172.21.15.1:0/1664978006 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f40035b0 con 0x7f41f4004850 2024-03-18T20:30:02.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.656+0000 7f420ca0f700 1 -- 172.21.15.1:0/1664978006 reap_dead start 2024-03-18T20:30:02.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.657+0000 7f42077fe700 1 --2- 172.21.15.1:0/1664978006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41f4004850 0x7f41f4004cd0 secure :-1 s=READY pgs=3457 cs=0 l=1 rev1=1 crypto rx=0x7f41f000c240 tx=0x7f41f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69bdb257a31a0880 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:02.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.657+0000 7f4206ffd700 1 -- 172.21.15.1:0/1664978006 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f0007c10 con 0x7f41f4004850 2024-03-18T20:30:02.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.657+0000 7f4206ffd700 1 -- 172.21.15.1:0/1664978006 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f000ce00 con 0x7f41f4004850 2024-03-18T20:30:02.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.657+0000 7f4206ffd700 1 -- 172.21.15.1:0/1664978006 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f0005ba0 con 0x7f41f4004850 2024-03-18T20:30:02.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.657+0000 7f420e472700 1 -- 172.21.15.1:0/1664978006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41f4004850 msgr2=0x7f41f4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.657+0000 7f420e472700 1 --2- 172.21.15.1:0/1664978006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41f4004850 0x7f41f4004cd0 secure :-1 s=READY pgs=3457 cs=0 l=1 rev1=1 crypto rx=0x7f41f000c240 tx=0x7f41f4005880 comp rx=0 tx=0).stop 2024-03-18T20:30:02.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.657+0000 7f420e472700 1 -- 172.21.15.1:0/1664978006 shutdown_connections 2024-03-18T20:30:02.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.657+0000 7f420e472700 1 --2- 172.21.15.1:0/1664978006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41f4004850 0x7f41f4004cd0 unknown :-1 s=CLOSED pgs=3457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.657+0000 7f420e472700 1 -- 172.21.15.1:0/1664978006 >> 172.21.15.1:0/1664978006 conn(0x7f42080b2e30 msgr2=0x7f42080b3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:02.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.657+0000 7f420e472700 1 -- 172.21.15.1:0/1664978006 shutdown_connections 2024-03-18T20:30:02.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.657+0000 7f420e472700 1 -- 172.21.15.1:0/1664978006 wait complete. 2024-03-18T20:30:02.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.658+0000 7f420e472700 1 Processor -- start 2024-03-18T20:30:02.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.659+0000 7f420e472700 1 -- start start 2024-03-18T20:30:02.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.659+0000 7f420e472700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41f4004850 0x7f4208147c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.659+0000 7f420e472700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42081529f0 0x7f42081536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.659+0000 7f420e472700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4208151d00 0x7f4208153be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.659+0000 7f420e472700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42081545b0 con 0x7f4208151d00 2024-03-18T20:30:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.659+0000 7f420e472700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42081f1150 con 0x7f41f4004850 2024-03-18T20:30:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.659+0000 7f420e472700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420813adf0 con 0x7f42081529f0 2024-03-18T20:30:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.659+0000 7f420ca0f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4208151d00 0x7f4208153be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.660+0000 7f420ca0f700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4208151d00 0x7f4208153be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50352/0 (socket says 172.21.15.1:50352) 2024-03-18T20:30:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.660+0000 7f420ca0f700 1 -- 172.21.15.1:0/724118893 learned_addr learned my addr 172.21.15.1:0/724118893 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:02.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.660+0000 7f4207fff700 1 --2- 172.21.15.1:0/724118893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41f4004850 0x7f4208147c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.660+0000 7f420ca0f700 1 -- 172.21.15.1:0/724118893 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42081529f0 msgr2=0x7f42081536a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.660+0000 7f420ca0f700 1 --2- 172.21.15.1:0/724118893 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42081529f0 0x7f42081536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.660+0000 7f420ca0f700 1 -- 172.21.15.1:0/724118893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41f4004850 msgr2=0x7f4208147c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.660+0000 7f420ca0f700 1 --2- 172.21.15.1:0/724118893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41f4004850 0x7f4208147c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.660+0000 7f420ca0f700 1 -- 172.21.15.1:0/724118893 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420813b0d0 con 0x7f4208151d00 2024-03-18T20:30:02.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.660+0000 7f420ca0f700 1 --2- 172.21.15.1:0/724118893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4208151d00 0x7f4208153be0 secure :-1 s=READY pgs=3458 cs=0 l=1 rev1=1 crypto rx=0x7f420815ff90 tx=0x7f4200011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:02.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.660+0000 7f4204ff9700 1 -- 172.21.15.1:0/724118893 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420000e040 con 0x7f4208151d00 2024-03-18T20:30:02.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.660+0000 7f4204ff9700 1 -- 172.21.15.1:0/724118893 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f420000f040 con 0x7f4208151d00 2024-03-18T20:30:02.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.660+0000 7f420e472700 1 -- 172.21.15.1:0/724118893 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f420813b3b0 con 0x7f4208151d00 2024-03-18T20:30:02.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.660+0000 7f4204ff9700 1 -- 172.21.15.1:0/724118893 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420000f8f0 con 0x7f4208151d00 2024-03-18T20:30:02.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.660+0000 7f420e472700 1 -- 172.21.15.1:0/724118893 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f420813bbb0 con 0x7f4208151d00 2024-03-18T20:30:02.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.661+0000 7f420e472700 1 -- 172.21.15.1:0/724118893 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f420813b8d0 con 0x7f4208151d00 2024-03-18T20:30:02.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.662+0000 7f4204ff9700 1 -- 172.21.15.1:0/724118893 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f420000fa50 con 0x7f4208151d00 2024-03-18T20:30:02.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.662+0000 7f4204ff9700 1 --2- 172.21.15.1:0/724118893 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f41e806adb0 0x7f41e806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:02.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.662+0000 7f4204ff9700 1 -- 172.21.15.1:0/724118893 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f4200089e00 con 0x7f4208151d00 2024-03-18T20:30:02.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.662+0000 7f4207fff700 1 --2- 172.21.15.1:0/724118893 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f41e806adb0 0x7f41e806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:02.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.662+0000 7f4207fff700 1 --2- 172.21.15.1:0/724118893 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f41e806adb0 0x7f41e806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f41f8000c00 tx=0x7f41f8009040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:02.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.667+0000 7f4204ff9700 1 -- 172.21.15.1:0/724118893 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f420813b8d0 con 0x7f4208151d00 2024-03-18T20:30:02.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.866+0000 7f420e472700 1 -- 172.21.15.1:0/724118893 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f420815ef50 con 0x7f41e806adb0 2024-03-18T20:30:02.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.866+0000 7f4204ff9700 1 -- 172.21.15.1:0/724118893 <== mgr.14091 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f41f8005270 con 0x7f41e806adb0 2024-03-18T20:30:02.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.868+0000 7f420e472700 1 -- 172.21.15.1:0/724118893 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f41e806adb0 msgr2=0x7f41e806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.868+0000 7f420e472700 1 --2- 172.21.15.1:0/724118893 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f41e806adb0 0x7f41e806d270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f41f8000c00 tx=0x7f41f8009040 comp rx=0 tx=0).stop 2024-03-18T20:30:02.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.868+0000 7f420e472700 1 -- 172.21.15.1:0/724118893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4208151d00 msgr2=0x7f4208153be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:02.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.868+0000 7f420e472700 1 --2- 172.21.15.1:0/724118893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4208151d00 0x7f4208153be0 secure :-1 s=READY pgs=3458 cs=0 l=1 rev1=1 crypto rx=0x7f420815ff90 tx=0x7f4200011ed0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.868+0000 7f420e472700 1 -- 172.21.15.1:0/724118893 shutdown_connections 2024-03-18T20:30:02.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.868+0000 7f420e472700 1 --2- 172.21.15.1:0/724118893 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f41e806adb0 0x7f41e806d270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.868+0000 7f420e472700 1 --2- 172.21.15.1:0/724118893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4208151d00 0x7f4208153be0 unknown :-1 s=CLOSED pgs=3458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.868+0000 7f420e472700 1 --2- 172.21.15.1:0/724118893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41f4004850 0x7f4208147c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.868+0000 7f420e472700 1 --2- 172.21.15.1:0/724118893 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f42081529f0 0x7f42081536a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:02.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.868+0000 7f420e472700 1 -- 172.21.15.1:0/724118893 >> 172.21.15.1:0/724118893 conn(0x7f42080b2e30 msgr2=0x7f420813c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:02.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.868+0000 7f420e472700 1 -- 172.21.15.1:0/724118893 shutdown_connections 2024-03-18T20:30:02.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:02.868+0000 7f420e472700 1 -- 172.21.15.1:0/724118893 wait complete. 2024-03-18T20:30:02.874 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 7 to deep-scrub 2024-03-18T20:30:02.884 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:30:03.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.005+0000 7f1305e38700 1 Processor -- start 2024-03-18T20:30:03.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.005+0000 7f1305e38700 1 -- start start 2024-03-18T20:30:03.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.006+0000 7f1305e38700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f130015a520 0x7f13000b0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:03.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.006+0000 7f1305e38700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f13000b11d0 0x7f13000b1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:03.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.006+0000 7f1305e38700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13000cc960 con 0x7f130015dfe0 2024-03-18T20:30:03.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.006+0000 7f1305e38700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13000cdc50 con 0x7f13000b11d0 2024-03-18T20:30:03.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.006+0000 7f1305e38700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130015a900 con 0x7f130015a520 2024-03-18T20:30:03.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.006+0000 7f12feffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f13000b11d0 0x7f13000b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:03.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.006+0000 7f12feffd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f13000b11d0 0x7f13000b1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50358/0 (socket says 172.21.15.1:50358) 2024-03-18T20:30:03.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.006+0000 7f12feffd700 1 -- 172.21.15.1:0/1100587199 learned_addr learned my addr 172.21.15.1:0/1100587199 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:03.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.006+0000 7f12ff7fe700 1 --2- 172.21.15.1:0/1100587199 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f130015a520 0x7f13000b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:03.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.006+0000 7f12feffd700 1 -- 172.21.15.1:0/1100587199 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f130015a520 msgr2=0x7f13000b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:03.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.006+0000 7f12feffd700 1 --2- 172.21.15.1:0/1100587199 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f130015a520 0x7f13000b0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:03.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.006+0000 7f12feffd700 1 -- 172.21.15.1:0/1100587199 >> v1:172.21.15.165:6789/0 conn(0x7f130015dfe0 legacy=0x7f130015e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:03.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.006+0000 7f12feffd700 1 -- 172.21.15.1:0/1100587199 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13000cd7c0 con 0x7f13000b11d0 2024-03-18T20:30:03.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.006+0000 7f12feffd700 1 --2- 172.21.15.1:0/1100587199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13000b11d0 0x7f13000b1650 secure :-1 s=READY pgs=3459 cs=0 l=1 rev1=1 crypto rx=0x7f12f00098f0 tx=0x7f12f0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=271996aee581c6fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:03.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.007+0000 7f12fe7fc700 1 -- 172.21.15.1:0/1100587199 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f000b8a0 con 0x7f13000b11d0 2024-03-18T20:30:03.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.007+0000 7f12fe7fc700 1 -- 172.21.15.1:0/1100587199 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f000ba00 con 0x7f13000b11d0 2024-03-18T20:30:03.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.007+0000 7f12fe7fc700 1 -- 172.21.15.1:0/1100587199 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f000bd90 con 0x7f13000b11d0 2024-03-18T20:30:03.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.007+0000 7f1305e38700 1 -- 172.21.15.1:0/1100587199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13000b11d0 msgr2=0x7f13000b1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:03.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.007+0000 7f1305e38700 1 --2- 172.21.15.1:0/1100587199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13000b11d0 0x7f13000b1650 secure :-1 s=READY pgs=3459 cs=0 l=1 rev1=1 crypto rx=0x7f12f00098f0 tx=0x7f12f0009c00 comp rx=0 tx=0).stop 2024-03-18T20:30:03.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.007+0000 7f1305e38700 1 -- 172.21.15.1:0/1100587199 shutdown_connections 2024-03-18T20:30:03.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.007+0000 7f1305e38700 1 --2- 172.21.15.1:0/1100587199 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f130015a520 0x7f13000b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:03.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.007+0000 7f1305e38700 1 --2- 172.21.15.1:0/1100587199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13000b11d0 0x7f13000b1650 unknown :-1 s=CLOSED pgs=3459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:03.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.007+0000 7f1305e38700 1 -- 172.21.15.1:0/1100587199 >> 172.21.15.1:0/1100587199 conn(0x7f1300072120 msgr2=0x7f1300099a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:03.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.007+0000 7f1305e38700 1 -- 172.21.15.1:0/1100587199 shutdown_connections 2024-03-18T20:30:03.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.007+0000 7f1305e38700 1 -- 172.21.15.1:0/1100587199 wait complete. 2024-03-18T20:30:03.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.008+0000 7f1305e38700 1 Processor -- start 2024-03-18T20:30:03.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.009+0000 7f1305e38700 1 -- start start 2024-03-18T20:30:03.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.009+0000 7f1305e38700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f13000b11d0 0x7f1300149e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:03.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.009+0000 7f1305e38700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f130015a520 0x7f13000ae4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:03.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.009+0000 7f1305e38700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f130015dfe0 0x7f13000aea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:03.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.009+0000 7f1305e38700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13000ad220 con 0x7f130015a520 2024-03-18T20:30:03.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.009+0000 7f1305e38700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13000ad430 con 0x7f130015dfe0 2024-03-18T20:30:03.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.009+0000 7f1305e38700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13000ad710 con 0x7f13000b11d0 2024-03-18T20:30:03.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.009+0000 7f12feffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f130015a520 0x7f13000ae4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:03.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.009+0000 7f12feffd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f130015a520 0x7f13000ae4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50362/0 (socket says 172.21.15.1:50362) 2024-03-18T20:30:03.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.009+0000 7f12feffd700 1 -- 172.21.15.1:0/2327197985 learned_addr learned my addr 172.21.15.1:0/2327197985 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:03.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.010+0000 7f12fffff700 1 --2- 172.21.15.1:0/2327197985 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f130015dfe0 0x7f13000aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:03.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.010+0000 7f12ff7fe700 1 --2- 172.21.15.1:0/2327197985 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f13000b11d0 0x7f1300149e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:03.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.010+0000 7f12feffd700 1 -- 172.21.15.1:0/2327197985 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f13000b11d0 msgr2=0x7f1300149e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:03.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.010+0000 7f12feffd700 1 --2- 172.21.15.1:0/2327197985 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f13000b11d0 0x7f1300149e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:03.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.010+0000 7f12feffd700 1 -- 172.21.15.1:0/2327197985 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f130015dfe0 msgr2=0x7f13000aea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:03.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.010+0000 7f12feffd700 1 --2- 172.21.15.1:0/2327197985 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f130015dfe0 0x7f13000aea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:03.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.010+0000 7f12feffd700 1 -- 172.21.15.1:0/2327197985 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f0009550 con 0x7f130015a520 2024-03-18T20:30:03.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.010+0000 7f12ff7fe700 1 --2- 172.21.15.1:0/2327197985 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f13000b11d0 0x7f1300149e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T20:30:03.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.010+0000 7f12feffd700 1 --2- 172.21.15.1:0/2327197985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f130015a520 0x7f13000ae4d0 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7f12f0000c00 tx=0x7f12f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:03.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.010+0000 7f12f7fff700 1 -- 172.21.15.1:0/2327197985 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f0004480 con 0x7f130015a520 2024-03-18T20:30:03.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.010+0000 7f12f7fff700 1 -- 172.21.15.1:0/2327197985 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f001f070 con 0x7f130015a520 2024-03-18T20:30:03.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.010+0000 7f12f7fff700 1 -- 172.21.15.1:0/2327197985 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f00038c0 con 0x7f130015a520 2024-03-18T20:30:03.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.010+0000 7f1305e38700 1 -- 172.21.15.1:0/2327197985 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13000ad9f0 con 0x7f130015a520 2024-03-18T20:30:03.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.010+0000 7f1305e38700 1 -- 172.21.15.1:0/2327197985 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13000adfb0 con 0x7f130015a520 2024-03-18T20:30:03.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.011+0000 7f1305e38700 1 -- 172.21.15.1:0/2327197985 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1300067dc0 con 0x7f130015a520 2024-03-18T20:30:03.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.012+0000 7f12f7fff700 1 -- 172.21.15.1:0/2327197985 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f12f0021030 con 0x7f130015a520 2024-03-18T20:30:03.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.012+0000 7f12f7fff700 1 --2- 172.21.15.1:0/2327197985 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f12e006adb0 0x7f12e006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:03.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.012+0000 7f12f7fff700 1 -- 172.21.15.1:0/2327197985 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f12f008ac70 con 0x7f130015a520 2024-03-18T20:30:03.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.012+0000 7f12ff7fe700 1 --2- 172.21.15.1:0/2327197985 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f12e006adb0 0x7f12e006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:03.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.012+0000 7f12ff7fe700 1 --2- 172.21.15.1:0/2327197985 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f12e006adb0 0x7f12e006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f12e8009a00 tx=0x7f12e8008040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:03.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.016+0000 7f12f7fff700 1 -- 172.21.15.1:0/2327197985 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f12f0055d70 con 0x7f130015a520 2024-03-18T20:30:03.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.216+0000 7f1305e38700 1 -- 172.21.15.1:0/2327197985 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f13000cd170 con 0x7f12e006adb0 2024-03-18T20:30:03.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.218+0000 7f12f7fff700 1 -- 172.21.15.1:0/2327197985 <== mgr.14091 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50425 (secure 0 0 0) 0x7f12e8009610 con 0x7f12e006adb0 2024-03-18T20:30:03.224 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:30:03.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.220+0000 7f1305e38700 1 -- 172.21.15.1:0/2327197985 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f12e006adb0 msgr2=0x7f12e006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:03.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.220+0000 7f1305e38700 1 --2- 172.21.15.1:0/2327197985 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f12e006adb0 0x7f12e006d270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f12e8009a00 tx=0x7f12e8008040 comp rx=0 tx=0).stop 2024-03-18T20:30:03.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.221+0000 7f1305e38700 1 -- 172.21.15.1:0/2327197985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f130015a520 msgr2=0x7f13000ae4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.221+0000 7f1305e38700 1 --2- 172.21.15.1:0/2327197985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f130015a520 0x7f13000ae4d0 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7f12f0000c00 tx=0x7f12f0004750 comp rx=0 tx=0).stop 2024-03-18T20:30:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.221+0000 7f1305e38700 1 -- 172.21.15.1:0/2327197985 shutdown_connections 2024-03-18T20:30:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.221+0000 7f1305e38700 1 --2- 172.21.15.1:0/2327197985 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f12e006adb0 0x7f12e006d270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.221+0000 7f1305e38700 1 --2- 172.21.15.1:0/2327197985 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f13000b11d0 0x7f1300149e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.221+0000 7f1305e38700 1 --2- 172.21.15.1:0/2327197985 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f130015dfe0 0x7f13000aea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:03.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.221+0000 7f1305e38700 1 --2- 172.21.15.1:0/2327197985 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f130015a520 0x7f13000ae4d0 unknown :-1 s=CLOSED pgs=3460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:03.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.221+0000 7f1305e38700 1 -- 172.21.15.1:0/2327197985 >> 172.21.15.1:0/2327197985 conn(0x7f1300072120 msgr2=0x7f130006f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:03.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.221+0000 7f1305e38700 1 -- 172.21.15.1:0/2327197985 shutdown_connections 2024-03-18T20:30:03.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:03.221+0000 7f1305e38700 1 -- 172.21.15.1:0/2327197985 wait complete. 2024-03-18T20:30:03.227 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:30:03.237 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-03-18T20:30:03.080682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":454004,"kb_used_data":30344,"kb_used_omap":12,"kb_used_meta":423539,"kb_avail":754520716,"statfs":{"total":773094113280,"available":772629213184,"internally_reserved":0,"allocated":31072256,"data_stored":16681984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":433704505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-123,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":19,"num_write_kb":-1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001584"},"pg_stats":[{"pgid":"1.0","version":"293'425","reported_seq":1530,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-03-18T20:29:52.603160+0000","last_change":"2024-03-18T18:49:41.768501+0000","last_active":"2024-03-18T20:29:52.603160+0000","last_peered":"2024-03-18T20:29:52.603160+0000","last_clean":"2024-03-18T20:29:52.603160+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:29:52.603160+0000","last_undegraded":"2024-03-18T20:29:52.603160+0000","last_fullsized":"2024-03-18T20:29:52.603160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T18:49:40.240662+0000","last_clean_scrub_stamp":"2024-03-18T18:49:40.240662+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:36.655111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576030,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57488,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314352,"statfs":{"total":96636764160,"available":96577896448,"internally_reserved":0,"allocated":4329472,"data_stored":2528024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56100000000000005,"15min":0.58999999999999997},"min":{"1min":0.38200000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.82899999999999996,"5min":0.88400000000000001,"15min":1.1339999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54500000000000004,"15min":0.59099999999999997},"min":{"1min":0.439,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":0.70699999999999996,"5min":0.92200000000000004,"15min":1.093},"last":0.52500000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.55400000000000005,"15min":0.59799999999999998},"min":{"1min":0.44700000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.749,"5min":0.82999999999999996,"15min":1.4299999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.56999999999999995,"15min":0.60199999999999998},"min":{"1min":0.39100000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.77400000000000002,"5min":0.89100000000000001,"15min":1.2629999999999999},"last":0.65600000000000003}]},{"osd":2,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55100000000000005,"15min":0.59399999999999997},"min":{"1min":0.33600000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.80300000000000005,"5min":0.85499999999999998,"15min":1.216},"last":0.504},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57899999999999996,"15min":0.60999999999999999},"min":{"1min":0.42799999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79300000000000004,"5min":1.1279999999999999,"15min":1.2490000000000001},"last":0.63200000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.55800000000000005,"15min":0.59299999999999997},"min":{"1min":0.40799999999999997,"5min":0.374,"15min":0.35199999999999998},"max":{"1min":0.751,"5min":0.84699999999999998,"15min":1.4450000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.58599999999999997,"15min":0.63500000000000001},"min":{"1min":0.42799999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.75800000000000001,"5min":0.94599999999999995,"15min":1.5029999999999999},"last":0.60299999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.47299999999999998,"15min":0.47899999999999998},"min":{"1min":0.35199999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.86099999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.498,"15min":0.5},"min":{"1min":0.34100000000000003,"5min":0.34000000000000002,"15min":0.30499999999999999},"max":{"1min":0.74299999999999999,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.64100000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53000000000000003,"15min":0.52500000000000002},"min":{"1min":0.373,"5min":0.314,"15min":0.314},"max":{"1min":0.86899999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.442},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.41699999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.81599999999999995,"5min":0.877,"15min":1.042},"last":0.57099999999999995}]},{"osd":6,"last update":"Mon Mar 18 20:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.42299999999999999,"5min":0.39500000000000002,"15min":0.36199999999999999},"max":{"1min":0.85599999999999998,"5min":0.93500000000000005,"15min":1.129},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.57299999999999995},"min":{"1min":0.44,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83799999999999997,"5min":0.91900000000000004,"15min":1.0609999999999999},"last":0.59399999999999997}]}]},{"osd":6,"up_from":13,"seq":55834576030,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65100,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306740,"statfs":{"total":96636764160,"available":96570101760,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":62912967},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.52200000000000002,"15min":0.57499999999999996},"min":{"1min":0.39600000000000002,"5min":0.32500000000000001,"15min":0.28799999999999998},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":1.1519999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53700000000000003,"15min":0.58399999999999996},"min":{"1min":0.44,"5min":0.317,"15min":0.317},"max":{"1min":0.73599999999999999,"5min":1.119,"15min":1.119},"last":0.46000000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.53300000000000003,"15min":0.57799999999999996},"min":{"1min":0.42599999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":1.244},"last":0.37},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54700000000000004,"15min":0.58799999999999997},"min":{"1min":0.34999999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":1.286},"last":0.441}]},{"osd":2,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.56799999999999995,"15min":0.60999999999999999},"min":{"1min":0.42099999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":1.276},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55100000000000005,"15min":0.59199999999999997},"min":{"1min":0.376,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83599999999999997,"5min":1.089,"15min":1.2669999999999999},"last":0.43099999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.56499999999999995,"15min":0.60399999999999998},"min":{"1min":0.39900000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80400000000000005,"5min":1.163,"15min":1.163},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56899999999999995,"15min":0.60599999999999998},"min":{"1min":0.40600000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79200000000000004,"5min":0.82399999999999995,"15min":1.337},"last":0.39700000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.51700000000000002,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.308,"15min":0.30399999999999999},"max":{"1min":0.746,"5min":0.80200000000000005,"15min":0.88300000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52800000000000002,"15min":0.52000000000000002},"min":{"1min":0.32000000000000001,"5min":0.31,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":0.83399999999999996,"15min":1.0309999999999999},"last":0.32600000000000001}]},{"osd":5,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.51500000000000001,"15min":0.51000000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.68300000000000005,"5min":1.008,"15min":1.008},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.57399999999999995,"15min":0.55400000000000005},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.77800000000000002,"5min":0.874,"15min":0.94199999999999995},"last":0.35299999999999998}]},{"osd":7,"last update":"Mon Mar 18 20:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57999999999999996,"15min":0.56399999999999995},"min":{"1min":0.38100000000000001,"5min":0.35899999999999999,"15min":0.34499999999999997},"max":{"1min":0.76500000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58699999999999997,"15min":0.57599999999999996},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.74299999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.29799999999999999}]}]},{"osd":1,"up_from":13,"seq":55834576030,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52820,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319020,"statfs":{"total":96636764160,"available":96582676480,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36899999999999999,"15min":0.38700000000000001},"min":{"1min":0.26600000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.45800000000000002,"5min":0.60899999999999999,"15min":0.70199999999999996},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.434,"5min":0.39800000000000002,"15min":0.41799999999999998},"min":{"1min":0.29699999999999999,"5min":0.255,"15min":0.255},"max":{"1min":0.57499999999999996,"5min":0.64900000000000002,"15min":0.69699999999999995},"last":0.42399999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:29:57 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.38700000000000001,"15min":0.41999999999999998},"min":{"1min":0.253,"5min":0.215,"15min":0.215},"max":{"1min":0.497,"5min":0.626,"15min":0.68100000000000005},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.433,"15min":0.46300000000000002},"min":{"1min":0.308,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61699999999999999,"5min":0.69799999999999995,"15min":0.80500000000000005},"last":0.308}]},{"osd":3,"last update":"Mon Mar 18 20:29:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.437,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.75900000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38900000000000001,"15min":0.42499999999999999},"min":{"1min":0.245,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.55200000000000005,"5min":0.58899999999999997,"15min":0.71699999999999997},"last":0.35399999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:29:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64400000000000002,"15min":0.66600000000000004},"min":{"1min":0.54100000000000004,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.94699999999999995,"5min":1.288,"15min":1.2909999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.63800000000000001,"15min":0.66600000000000004},"min":{"1min":0.46800000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.85199999999999998,"5min":1.349,"15min":1.746},"last":0.46800000000000003}]},{"osd":5,"last update":"Mon Mar 18 20:29:57 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.629,"15min":0.66100000000000003},"min":{"1min":0.47599999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.97799999999999998,"5min":1.361,"15min":1.6910000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.67200000000000004,"15min":0.70099999999999996},"min":{"1min":0.504,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86799999999999999,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.68600000000000005}]},{"osd":6,"last update":"Mon Mar 18 20:29:57 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.63400000000000001,"15min":0.67100000000000004},"min":{"1min":0.46000000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.879,"5min":1.3120000000000001,"15min":1.637},"last":0.499},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.66000000000000003,"15min":0.69599999999999995},"min":{"1min":0.51000000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.91200000000000003,"5min":0.93500000000000005,"15min":1.722},"last":0.71499999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:29:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.66100000000000003,"15min":0.69499999999999995},"min":{"1min":0.53000000000000003,"5min":0.42899999999999999,"15min":0.39500000000000002},"max":{"1min":0.96099999999999997,"5min":1.375,"15min":1.375},"last":0.628},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.66500000000000004,"15min":0.69299999999999995},"min":{"1min":0.48899999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.89700000000000002,"5min":1.431,"15min":1.431},"last":0.59499999999999997}]}]},{"osd":0,"up_from":13,"seq":55834576031,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49296,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322544,"statfs":{"total":96636764160,"available":96586285056,"internally_reserved":0,"allocated":4329472,"data_stored":2528024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.35899999999999999,"15min":0.38},"min":{"1min":0.25,"5min":0.214,"15min":0.214},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.38900000000000001,"15min":0.41599999999999998},"min":{"1min":0.30599999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":1.1850000000000001},"last":0.28799999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.41199999999999998,"15min":0.42899999999999999},"min":{"1min":0.34899999999999998,"5min":0.27100000000000002,"15min":0.24099999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.41099999999999998,"15min":0.439},"min":{"1min":0.28799999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.70799999999999996},"last":0.376}]},{"osd":3,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.38700000000000001,"15min":0.41699999999999998},"min":{"1min":0.312,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.57499999999999996,"5min":0.58499999999999996,"15min":0.628},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.45200000000000001,"15min":0.47899999999999998},"min":{"1min":0.309,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.91500000000000004},"last":0.32200000000000001}]},{"osd":4,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.61299999999999999,"15min":0.64000000000000001},"min":{"1min":0.40100000000000002,"5min":0.40000000000000002,"15min":0.36799999999999999},"max":{"1min":0.91200000000000003,"5min":1.506,"15min":1.506},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.443,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.01,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.52100000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.65000000000000002,"15min":0.66400000000000003},"min":{"1min":0.439,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.002,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.64500000000000002,"15min":0.67300000000000004},"min":{"1min":0.53000000000000003,"5min":0.432,"15min":0.432},"max":{"1min":1.006,"5min":1.468,"15min":1.518},"last":0.50900000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.63400000000000001,"15min":0.65600000000000003},"min":{"1min":0.52700000000000002,"5min":0.436,"15min":0.436},"max":{"1min":0.92000000000000004,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.67400000000000004,"15min":0.69699999999999995},"min":{"1min":0.56200000000000006,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0860000000000001,"5min":1.5649999999999999,"15min":1.5649999999999999},"last":0.53300000000000003}]},{"osd":7,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.67100000000000004,"15min":0.69499999999999995},"min":{"1min":0.504,"5min":0.45300000000000001,"15min":0.41899999999999998},"max":{"1min":1.016,"5min":1.55,"15min":1.55},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.65200000000000002,"15min":0.68700000000000006},"min":{"1min":0.57299999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0269999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.626}]}]},{"osd":2,"up_from":13,"seq":55834576031,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52812,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319028,"statfs":{"total":96636764160,"available":96582684672,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.33900000000000002,"15min":0.36399999999999999},"min":{"1min":0.28299999999999997,"5min":0.184,"15min":0.184},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.42899999999999999,"15min":0.46600000000000003},"min":{"1min":0.34300000000000003,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.753},"last":0.438}]},{"osd":1,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.433,"15min":0.46400000000000002},"min":{"1min":0.40500000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.749},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.36399999999999999,"15min":0.39900000000000002},"min":{"1min":0.307,"5min":0.224,"15min":0.224},"max":{"1min":0.54000000000000004,"5min":0.54600000000000004,"15min":0.59999999999999998},"last":1.2669999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.38600000000000001,"15min":0.41899999999999998},"min":{"1min":0.32800000000000001,"5min":0.26500000000000001,"15min":0.22800000000000001},"max":{"1min":0.623,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.44700000000000001,"15min":0.49399999999999999},"min":{"1min":0.372,"5min":0.315,"15min":0.315},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.752},"last":0.47599999999999998}]},{"osd":4,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.63300000000000001,"15min":0.64900000000000002},"min":{"1min":0.505,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.093,"5min":1.2090000000000001,"15min":1.4079999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.62,"15min":0.65500000000000003},"min":{"1min":0.39900000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98799999999999999,"5min":1.268,"15min":1.2729999999999999},"last":0.58099999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.624,"15min":0.66300000000000003},"min":{"1min":0.53400000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1,"5min":1,"15min":1.028},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.63200000000000001,"15min":0.66400000000000003},"min":{"1min":0.56599999999999995,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.72999999999999998}]},{"osd":6,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.626,"15min":0.67000000000000004},"min":{"1min":0.52900000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.69599999999999995,"15min":0.72099999999999997},"min":{"1min":0.499,"5min":0.439,"15min":0.439},"max":{"1min":1.1299999999999999,"5min":1.3220000000000001,"15min":1.3859999999999999},"last":0.80200000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.63500000000000001,"15min":0.67900000000000005},"min":{"1min":0.497,"5min":0.432,"15min":0.42199999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0269999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.746,"5min":0.67000000000000004,"15min":0.69299999999999995},"min":{"1min":0.50600000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1739999999999999,"5min":1.2330000000000001,"15min":1.371},"last":0.73899999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576031,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56908,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314932,"statfs":{"total":96636764160,"available":96578490368,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.34200000000000003,"15min":0.378},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.66500000000000004},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.35999999999999999,"15min":0.39900000000000002},"min":{"1min":0.27200000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.61699999999999999},"last":0.312}]},{"osd":1,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.35499999999999998,"15min":0.39000000000000001},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.214},"max":{"1min":0.55100000000000005,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.41099999999999998,"15min":0.44700000000000001},"min":{"1min":0.32800000000000001,"5min":0.29099999999999998,"15min":0.25800000000000001},"max":{"1min":0.60399999999999998,"5min":0.63500000000000001,"15min":0.72599999999999998},"last":0.36799999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.40799999999999997,"15min":0.44400000000000001},"min":{"1min":0.32600000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.68400000000000005},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.433,"15min":0.47299999999999998},"min":{"1min":0.36299999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.71199999999999997},"last":0.42899999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.439,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98499999999999999,"5min":1.526,"15min":1.526},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.622,"15min":0.65600000000000003},"min":{"1min":0.45500000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97899999999999998,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.59899999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.65600000000000003,"15min":0.67700000000000005},"min":{"1min":0.41299999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94399999999999995,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.66100000000000003,"15min":0.70799999999999996},"min":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.99099999999999999,"5min":1.1120000000000001,"15min":1.171},"last":0.63200000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.66000000000000003,"15min":0.68899999999999995},"min":{"1min":0.42499999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.878,"5min":1.573,"15min":1.573},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.67300000000000004,"15min":0.70999999999999996},"min":{"1min":0.48999999999999999,"5min":0.47399999999999998,"15min":0.45100000000000001},"max":{"1min":1.02,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.68500000000000005}]},{"osd":7,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63700000000000001,"15min":0.68999999999999995},"min":{"1min":0.42799999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.032,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.65300000000000002,"15min":0.68200000000000005},"min":{"1min":0.439,"5min":0.41099999999999998,"15min":0.39200000000000002},"max":{"1min":0.94699999999999995,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.63900000000000001}]}]},{"osd":4,"up_from":13,"seq":55834576031,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48720,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.55600000000000005,"15min":0.57999999999999996},"min":{"1min":0.44,"5min":0.34300000000000003,"15min":0.29499999999999998},"max":{"1min":0.77600000000000002,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56599999999999995,"15min":0.58399999999999996},"min":{"1min":0.46700000000000003,"5min":0.40100000000000002,"15min":0.36299999999999999},"max":{"1min":0.79200000000000004,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.55100000000000005}]},{"osd":1,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.58199999999999996,"15min":0.60299999999999998},"min":{"1min":0.45500000000000002,"5min":0.35799999999999998,"15min":0.318},"max":{"1min":0.81000000000000005,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.55500000000000005,"15min":0.57299999999999995},"min":{"1min":0.496,"5min":0.34899999999999998,"15min":0.34300000000000003},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.96499999999999997},"last":0.51300000000000001}]},{"osd":2,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.58699999999999997,"15min":0.59199999999999997},"min":{"1min":0.52800000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.82699999999999996,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.63,"5min":0.58899999999999997,"15min":0.60499999999999998},"min":{"1min":0.40200000000000002,"5min":0.38900000000000001,"15min":0.38500000000000001},"max":{"1min":0.76700000000000002,"5min":0.83099999999999996,"15min":0.84099999999999997},"last":0.63600000000000001}]},{"osd":3,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58099999999999996,"15min":0.59199999999999997},"min":{"1min":0.5,"5min":0.374,"15min":0.33900000000000002},"max":{"1min":0.82799999999999996,"5min":1.026,"15min":1.026},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.60499999999999998,"15min":0.61099999999999999},"min":{"1min":0.503,"5min":0.36699999999999999,"15min":0.34999999999999998},"max":{"1min":0.85499999999999998,"5min":1.028,"15min":1.028},"last":0.65000000000000002}]},{"osd":5,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.499,"15min":0.48199999999999998},"min":{"1min":0.34799999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.67400000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.52400000000000002,"15min":0.505},"min":{"1min":0.40799999999999997,"5min":0.31900000000000001,"15min":0.29999999999999999},"max":{"1min":0.68100000000000005,"5min":0.85499999999999998,"15min":0.90600000000000003},"last":0.56299999999999994}]},{"osd":6,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52700000000000002,"15min":0.50600000000000001},"min":{"1min":0.36799999999999999,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.748,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.56999999999999995,"15min":0.55300000000000005},"min":{"1min":0.435,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.77900000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.71399999999999997}]},{"osd":7,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56599999999999995,"15min":0.54800000000000004},"min":{"1min":0.38700000000000001,"5min":0.36899999999999999,"15min":0.35699999999999998},"max":{"1min":0.70499999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60699999999999998,"15min":0.58399999999999996},"min":{"1min":0.45900000000000002,"5min":0.36599999999999999,"15min":0.33700000000000002},"max":{"1min":0.79200000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.73099999999999998}]}]},{"osd":5,"up_from":13,"seq":55834576030,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70860,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94300980,"statfs":{"total":96636764160,"available":96564203520,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.53900000000000003,"15min":0.58199999999999996},"min":{"1min":0.36399999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76500000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.441},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.53600000000000003,"15min":0.58099999999999996},"min":{"1min":0.36899999999999999,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":0.73899999999999999,"5min":0.82399999999999995,"15min":1.3100000000000001},"last":0.36899999999999999}]},{"osd":1,"last update":"Mon Mar 18 20:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55000000000000004,"15min":0.58999999999999997},"min":{"1min":0.32900000000000001,"5min":0.32600000000000001,"15min":0.31900000000000001},"max":{"1min":0.75,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.53700000000000003,"15min":0.57899999999999996},"min":{"1min":0.41399999999999998,"5min":0.372,"15min":0.35199999999999998},"max":{"1min":0.72499999999999998,"5min":0.94299999999999995,"15min":1.018},"last":0.53100000000000003}]},{"osd":2,"last update":"Mon Mar 18 20:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.53600000000000003,"15min":0.58199999999999996},"min":{"1min":0.377,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.70099999999999996,"5min":0.85399999999999998,"15min":0.99299999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55100000000000005,"15min":0.59199999999999997},"min":{"1min":0.38700000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73299999999999998,"5min":1.599,"15min":1.599},"last":0.49399999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55500000000000005,"15min":0.59599999999999997},"min":{"1min":0.39600000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.80500000000000005,"5min":0.98699999999999999,"15min":1.2430000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58299999999999996,"15min":0.628},"min":{"1min":0.42499999999999999,"5min":0.40600000000000003,"15min":0.36699999999999999},"max":{"1min":0.78000000000000003,"5min":0.84399999999999997,"15min":1.161},"last":0.46400000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.48799999999999999,"15min":0.48199999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.58499999999999996,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53200000000000003,"15min":0.51900000000000002},"min":{"1min":0.33800000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.77600000000000002,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.57999999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.308,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64400000000000002,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.495,"5min":0.53300000000000003,"15min":0.53800000000000003},"min":{"1min":0.35299999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.68500000000000005,"5min":1.643,"15min":1.643},"last":0.52200000000000002}]},{"osd":7,"last update":"Mon Mar 18 20:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.747,"5min":1.028,"15min":1.028},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.57299999999999995,"15min":0.57599999999999996},"min":{"1min":0.36099999999999999,"5min":0.34300000000000003,"15min":0.29799999999999999},"max":{"1min":0.75600000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.56799999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:30:03.240 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-18T18:49:40.240662+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=18, tm_min=49, tm_sec=40, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=20, tm_min=29, tm_sec=57, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T20:30:03.240 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-18T20:30:11.987 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:30:11.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:30:23.242 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T20:30:23.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.379+0000 7fa2b4ae1700 1 Processor -- start 2024-03-18T20:30:23.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.379+0000 7fa2b4ae1700 1 -- start start 2024-03-18T20:30:23.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.380+0000 7fa2b4ae1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2b00b3260 0x7fa2b00b4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:23.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.380+0000 7fa2b4ae1700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2b0143880 0x7fa2b00b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:23.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.380+0000 7fa2b4ae1700 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b0142a70 con 0x7fa2b0140cb0 2024-03-18T20:30:23.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.380+0000 7fa2b4ae1700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b0144120 con 0x7fa2b0143880 2024-03-18T20:30:23.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.380+0000 7fa2b4ae1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b0143610 con 0x7fa2b00b3260 2024-03-18T20:30:23.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.380+0000 7fa2aed9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2b0143880 0x7fa2b00b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:23.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.380+0000 7fa2aed9d700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2b0143880 0x7fa2b00b4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49372/0 (socket says 172.21.15.1:49372) 2024-03-18T20:30:23.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.380+0000 7fa2aed9d700 1 -- 172.21.15.1:0/2141274838 learned_addr learned my addr 172.21.15.1:0/2141274838 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:23.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.380+0000 7fa2aed9d700 1 -- 172.21.15.1:0/2141274838 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2b00b3260 msgr2=0x7fa2b00b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:23.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.380+0000 7fa2aed9d700 1 --2- 172.21.15.1:0/2141274838 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2b00b3260 0x7fa2b00b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.380+0000 7fa2aed9d700 1 -- 172.21.15.1:0/2141274838 >> v1:172.21.15.1:6789/0 conn(0x7fa2b0140cb0 legacy=0x7fa2b0142f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:23.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.380+0000 7fa2aed9d700 1 -- 172.21.15.1:0/2141274838 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b0141e60 con 0x7fa2b0143880 2024-03-18T20:30:23.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.380+0000 7fa2aed9d700 1 --2- 172.21.15.1:0/2141274838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2b0143880 0x7fa2b00b4d80 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8009e50 tx=0x7fa2a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d751f0f92c1a180 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:23.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.380+0000 7fa2ad59a700 1 -- 172.21.15.1:0/2141274838 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a800bbc0 con 0x7fa2b0143880 2024-03-18T20:30:23.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.380+0000 7fa2ad59a700 1 -- 172.21.15.1:0/2141274838 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a800bd20 con 0x7fa2b0143880 2024-03-18T20:30:23.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.380+0000 7fa2ad59a700 1 -- 172.21.15.1:0/2141274838 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a80116a0 con 0x7fa2b0143880 2024-03-18T20:30:23.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.381+0000 7fa2b4ae1700 1 -- 172.21.15.1:0/2141274838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2b0143880 msgr2=0x7fa2b00b4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:23.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.381+0000 7fa2b4ae1700 1 --2- 172.21.15.1:0/2141274838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2b0143880 0x7fa2b00b4d80 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8009e50 tx=0x7fa2a800b330 comp rx=0 tx=0).stop 2024-03-18T20:30:23.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.381+0000 7fa2b4ae1700 1 -- 172.21.15.1:0/2141274838 shutdown_connections 2024-03-18T20:30:23.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.381+0000 7fa2b4ae1700 1 --2- 172.21.15.1:0/2141274838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2b0143880 0x7fa2b00b4d80 unknown :-1 s=CLOSED pgs=3462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.381+0000 7fa2b4ae1700 1 --2- 172.21.15.1:0/2141274838 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2b00b3260 0x7fa2b00b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.381+0000 7fa2b4ae1700 1 -- 172.21.15.1:0/2141274838 >> 172.21.15.1:0/2141274838 conn(0x7fa2b00b5780 msgr2=0x7fa2b00b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:23.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.381+0000 7fa2b4ae1700 1 -- 172.21.15.1:0/2141274838 shutdown_connections 2024-03-18T20:30:23.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.381+0000 7fa2b4ae1700 1 -- 172.21.15.1:0/2141274838 wait complete. 2024-03-18T20:30:23.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.382+0000 7fa2b4ae1700 1 Processor -- start 2024-03-18T20:30:23.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.382+0000 7fa2b4ae1700 1 -- start start 2024-03-18T20:30:23.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.383+0000 7fa2b4ae1700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2b00b3260 0x7fa2b0159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:23.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.383+0000 7fa2b4ae1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2b0140cb0 0x7fa2b006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:23.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.383+0000 7fa2b4ae1700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2b0143880 0x7fa2b006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:23.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.383+0000 7fa2b4ae1700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b00af440 con 0x7fa2b0143880 2024-03-18T20:30:23.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.383+0000 7fa2b4ae1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b00ae850 con 0x7fa2b0140cb0 2024-03-18T20:30:23.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.383+0000 7fa2b4ae1700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b015a270 con 0x7fa2b00b3260 2024-03-18T20:30:23.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.383+0000 7fa2aed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2b0143880 0x7fa2b006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:23.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.383+0000 7fa2aed9d700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2b0143880 0x7fa2b006ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49374/0 (socket says 172.21.15.1:49374) 2024-03-18T20:30:23.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.383+0000 7fa2aed9d700 1 -- 172.21.15.1:0/2832647673 learned_addr learned my addr 172.21.15.1:0/2832647673 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.384+0000 7fa2add9b700 1 --2- 172.21.15.1:0/2832647673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2b0140cb0 0x7fa2b006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.384+0000 7fa2aed9d700 1 -- 172.21.15.1:0/2832647673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2b00b3260 msgr2=0x7fa2b0159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.384+0000 7fa2aed9d700 1 --2- 172.21.15.1:0/2832647673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2b00b3260 0x7fa2b0159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.384+0000 7fa2aed9d700 1 -- 172.21.15.1:0/2832647673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2b0140cb0 msgr2=0x7fa2b006ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.384+0000 7fa2aed9d700 1 --2- 172.21.15.1:0/2832647673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2b0140cb0 0x7fa2b006ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.384+0000 7fa2aed9d700 1 -- 172.21.15.1:0/2832647673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a8009b30 con 0x7fa2b0143880 2024-03-18T20:30:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.384+0000 7fa2aed9d700 1 --2- 172.21.15.1:0/2832647673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2b0143880 0x7fa2b006ef50 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00716e0 tx=0x7fa2a8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.384+0000 7fa29effd700 1 -- 172.21.15.1:0/2832647673 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8021070 con 0x7fa2b0143880 2024-03-18T20:30:23.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.384+0000 7fa29effd700 1 -- 172.21.15.1:0/2832647673 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a80099b0 con 0x7fa2b0143880 2024-03-18T20:30:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.384+0000 7fa2b4ae1700 1 -- 172.21.15.1:0/2832647673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b00aeaa0 con 0x7fa2b0143880 2024-03-18T20:30:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.384+0000 7fa29effd700 1 -- 172.21.15.1:0/2832647673 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8011a70 con 0x7fa2b0143880 2024-03-18T20:30:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.384+0000 7fa2b4ae1700 1 -- 172.21.15.1:0/2832647673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b00af060 con 0x7fa2b0143880 2024-03-18T20:30:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.385+0000 7fa2b4ae1700 1 -- 172.21.15.1:0/2832647673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b00af580 con 0x7fa2b0143880 2024-03-18T20:30:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.385+0000 7fa29effd700 1 -- 172.21.15.1:0/2832647673 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7fa2a8011bd0 con 0x7fa2b0143880 2024-03-18T20:30:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.386+0000 7fa29effd700 1 --2- 172.21.15.1:0/2832647673 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa29006adb0 0x7fa29006d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.386+0000 7fa29effd700 1 -- 172.21.15.1:0/2832647673 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7fa2a808cfa0 con 0x7fa2b0143880 2024-03-18T20:30:23.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.386+0000 7fa2ae59c700 1 --2- 172.21.15.1:0/2832647673 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa29006adb0 0x7fa29006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.386+0000 7fa2ae59c700 1 --2- 172.21.15.1:0/2832647673 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa29006adb0 0x7fa29006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0000f60 tx=0x7fa2a0007a40 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:23.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.391+0000 7fa29effd700 1 -- 172.21.15.1:0/2832647673 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa2a80c0580 con 0x7fa2b0143880 2024-03-18T20:30:23.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.591+0000 7fa2b4ae1700 1 -- 172.21.15.1:0/2832647673 --> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa2b0142700 con 0x7fa29006adb0 2024-03-18T20:30:23.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.593+0000 7fa29effd700 1 -- 172.21.15.1:0/2832647673 <== mgr.14091 v2:172.21.15.1:6832/148356 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50518 (secure 0 0 0) 0x7fa2a0005080 con 0x7fa29006adb0 2024-03-18T20:30:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:30:23.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.595+0000 7fa2b4ae1700 1 -- 172.21.15.1:0/2832647673 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa29006adb0 msgr2=0x7fa29006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:23.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.595+0000 7fa2b4ae1700 1 --2- 172.21.15.1:0/2832647673 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa29006adb0 0x7fa29006d270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0000f60 tx=0x7fa2a0007a40 comp rx=0 tx=0).stop 2024-03-18T20:30:23.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.595+0000 7fa2b4ae1700 1 -- 172.21.15.1:0/2832647673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2b0143880 msgr2=0x7fa2b006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:23.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.595+0000 7fa2b4ae1700 1 --2- 172.21.15.1:0/2832647673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2b0143880 0x7fa2b006ef50 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00716e0 tx=0x7fa2a8004ad0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.595+0000 7fa2b4ae1700 1 -- 172.21.15.1:0/2832647673 shutdown_connections 2024-03-18T20:30:23.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.595+0000 7fa2b4ae1700 1 --2- 172.21.15.1:0/2832647673 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7fa29006adb0 0x7fa29006d270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.595+0000 7fa2b4ae1700 1 --2- 172.21.15.1:0/2832647673 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7fa2b00b3260 0x7fa2b0159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.595+0000 7fa2b4ae1700 1 --2- 172.21.15.1:0/2832647673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2b0143880 0x7fa2b006ef50 unknown :-1 s=CLOSED pgs=3463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.595+0000 7fa2b4ae1700 1 --2- 172.21.15.1:0/2832647673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2b0140cb0 0x7fa2b006ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.595+0000 7fa2b4ae1700 1 -- 172.21.15.1:0/2832647673 >> 172.21.15.1:0/2832647673 conn(0x7fa2b00b5780 msgr2=0x7fa2b006ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:23.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.595+0000 7fa2b4ae1700 1 -- 172.21.15.1:0/2832647673 shutdown_connections 2024-03-18T20:30:23.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.595+0000 7fa2b4ae1700 1 -- 172.21.15.1:0/2832647673 wait complete. 2024-03-18T20:30:23.601 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-18T20:30:23.611 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-18T20:30:23.084434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":426,"ondisk_log_size":426,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":453996,"kb_used_data":30344,"kb_used_omap":12,"kb_used_meta":423539,"kb_avail":754520724,"statfs":{"total":773094113280,"available":772629221376,"internally_reserved":0,"allocated":31072256,"data_stored":16681984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":433704505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002281"},"pg_stats":[{"pgid":"1.0","version":"295'426","reported_seq":1544,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-18T20:30:06.974247+0000","last_change":"2024-03-18T20:30:06.974247+0000","last_active":"2024-03-18T20:30:06.974247+0000","last_peered":"2024-03-18T20:30:06.974247+0000","last_clean":"2024-03-18T20:30:06.974247+0000","last_became_active":"2024-03-18T18:49:41.767575+0000","last_became_peered":"2024-03-18T18:49:41.767575+0000","last_unstale":"2024-03-18T20:30:06.974247+0000","last_undegraded":"2024-03-18T20:30:06.974247+0000","last_fullsized":"2024-03-18T20:30:06.974247+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'426","last_scrub_stamp":"2024-03-18T20:30:06.974103+0000","last_deep_scrub":"295'426","last_deep_scrub_stamp":"2024-03-18T20:30:06.974103+0000","last_clean_scrub_stamp":"2024-03-18T20:30:06.974103+0000","objects_scrubbed":2,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:31:43.443385+0000","scrub_duration":0.0053380490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2804,"num_read_kb":5868,"num_write":719,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":426,"ondisk_log_size":426,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576034,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57484,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314356,"statfs":{"total":96636764160,"available":96577900544,"internally_reserved":0,"allocated":4329472,"data_stored":2528024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:30:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54500000000000004,"15min":0.58899999999999997},"min":{"1min":0.41299999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.72799999999999998,"5min":0.88400000000000001,"15min":1.1339999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.54100000000000004,"15min":0.59199999999999997},"min":{"1min":0.43099999999999999,"5min":0.33800000000000002,"15min":0.33300000000000002},"max":{"1min":0.70999999999999996,"5min":0.92200000000000004,"15min":1.093},"last":0.45700000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:30:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.54300000000000004,"15min":0.59599999999999997},"min":{"1min":0.36799999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68300000000000005,"5min":0.76100000000000001,"15min":1.4299999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.56999999999999995,"15min":0.60399999999999998},"min":{"1min":0.41799999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.77200000000000002,"5min":0.89100000000000001,"15min":1.2629999999999999},"last":0.504}]},{"osd":2,"last update":"Mon Mar 18 20:30:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.54100000000000004,"15min":0.59399999999999997},"min":{"1min":0.432,"5min":0.316,"15min":0.316},"max":{"1min":0.70799999999999996,"5min":0.85499999999999998,"15min":1.216},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56699999999999995,"15min":0.60799999999999998},"min":{"1min":0.39600000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.70799999999999996,"5min":1.1279999999999999,"15min":1.2490000000000001},"last":0.61199999999999999}]},{"osd":3,"last update":"Mon Mar 18 20:30:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.55400000000000005,"15min":0.59399999999999997},"min":{"1min":0.40899999999999997,"5min":0.374,"15min":0.35199999999999998},"max":{"1min":0.76200000000000001,"5min":0.83899999999999997,"15min":1.4450000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.57699999999999996,"15min":0.63300000000000001},"min":{"1min":0.48799999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.73899999999999999,"5min":0.94599999999999995,"15min":1.5029999999999999},"last":0.57499999999999996}]},{"osd":4,"last update":"Mon Mar 18 20:30:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.47899999999999998,"15min":0.47799999999999998},"min":{"1min":0.34799999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.72699999999999998,"5min":0.77300000000000002,"15min":0.79100000000000004},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.496,"15min":0.497},"min":{"1min":0.36299999999999999,"5min":0.34000000000000002,"15min":0.30499999999999999},"max":{"1min":0.64100000000000001,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.40999999999999998}]},{"osd":5,"last update":"Mon Mar 18 20:30:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.52800000000000002,"15min":0.52100000000000002},"min":{"1min":0.33900000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.73899999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.439},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.41599999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.81699999999999995,"5min":0.877,"15min":0.877},"last":0.55400000000000005}]},{"osd":6,"last update":"Mon Mar 18 20:30:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.44600000000000001,"5min":0.39500000000000002,"15min":0.36199999999999999},"max":{"1min":0.78000000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.626},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56799999999999995,"15min":0.57099999999999995},"min":{"1min":0.39600000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.80400000000000005,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.54400000000000004}]}]},{"osd":6,"up_from":13,"seq":55834576034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65100,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94306740,"statfs":{"total":96636764160,"available":96570101760,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":62912967},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:30:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.52500000000000002,"15min":0.56999999999999995},"min":{"1min":0.34200000000000003,"5min":0.32500000000000001,"15min":0.28799999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":1.1519999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.52900000000000003,"15min":0.57799999999999996},"min":{"1min":0.38100000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.68700000000000006,"5min":1.119,"15min":1.119},"last":0.72499999999999998}]},{"osd":1,"last update":"Mon Mar 18 20:30:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52800000000000002,"15min":0.57199999999999995},"min":{"1min":0.36599999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.73699999999999999,"5min":0.80000000000000004,"15min":1.244},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54800000000000004,"15min":0.58299999999999996},"min":{"1min":0.40400000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72999999999999998,"5min":0.84899999999999998,"15min":1.286},"last":0.73899999999999999}]},{"osd":2,"last update":"Mon Mar 18 20:30:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56799999999999995,"15min":0.60699999999999998},"min":{"1min":0.41599999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.81399999999999995,"5min":0.81699999999999995,"15min":1.276},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54400000000000004,"15min":0.58599999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.754,"5min":1.089,"15min":1.2669999999999999},"last":0.68999999999999995}]},{"osd":3,"last update":"Mon Mar 18 20:30:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56799999999999995,"15min":0.59899999999999998},"min":{"1min":0.35499999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.81499999999999995,"5min":1.163,"15min":1.163},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56399999999999995,"15min":0.60099999999999998},"min":{"1min":0.39700000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":1.337},"last":0.754}]},{"osd":4,"last update":"Mon Mar 18 20:30:16 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.51300000000000001,"15min":0.49399999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76500000000000001,"5min":0.80200000000000005,"15min":0.86499999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52400000000000002,"15min":0.52000000000000002},"min":{"1min":0.32600000000000001,"5min":0.31,"15min":0.29799999999999999},"max":{"1min":0.78000000000000003,"5min":0.83399999999999996,"15min":1.0309999999999999},"last":0.626}]},{"osd":5,"last update":"Mon Mar 18 20:30:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51500000000000001,"15min":0.50800000000000001},"min":{"1min":0.32600000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73799999999999999,"5min":1.008,"15min":1.008},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57199999999999995,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79800000000000004,"5min":0.874,"15min":0.92600000000000005},"last":0.65500000000000003}]},{"osd":7,"last update":"Mon Mar 18 20:30:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57899999999999996,"15min":0.56399999999999995},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.97499999999999998,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.57899999999999996,"15min":0.57299999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.871,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.67500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834576034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52820,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319020,"statfs":{"total":96636764160,"available":96582676480,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36899999999999999,"15min":0.38700000000000001},"min":{"1min":0.26600000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.45800000000000002,"5min":0.60899999999999999,"15min":0.70199999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.434,"5min":0.39800000000000002,"15min":0.41799999999999998},"min":{"1min":0.29699999999999999,"5min":0.255,"15min":0.255},"max":{"1min":0.57499999999999996,"5min":0.64900000000000002,"15min":0.69699999999999995},"last":0.46400000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:29:57 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.38700000000000001,"15min":0.41999999999999998},"min":{"1min":0.253,"5min":0.215,"15min":0.215},"max":{"1min":0.497,"5min":0.626,"15min":0.68100000000000005},"last":0.441},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.433,"15min":0.46300000000000002},"min":{"1min":0.308,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61699999999999999,"5min":0.69799999999999995,"15min":0.80500000000000005},"last":0.56299999999999994}]},{"osd":3,"last update":"Mon Mar 18 20:29:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.437,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.75900000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38900000000000001,"15min":0.42499999999999999},"min":{"1min":0.245,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.55200000000000005,"5min":0.58899999999999997,"15min":0.71699999999999997},"last":0.53300000000000003}]},{"osd":4,"last update":"Mon Mar 18 20:29:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64400000000000002,"15min":0.66600000000000004},"min":{"1min":0.54100000000000004,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.94699999999999995,"5min":1.288,"15min":1.2909999999999999},"last":0.87},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.63800000000000001,"15min":0.66600000000000004},"min":{"1min":0.46800000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.85199999999999998,"5min":1.349,"15min":1.746},"last":0.69099999999999995}]},{"osd":5,"last update":"Mon Mar 18 20:29:57 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.629,"15min":0.66100000000000003},"min":{"1min":0.47599999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.97799999999999998,"5min":1.361,"15min":1.6910000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.67200000000000004,"15min":0.70099999999999996},"min":{"1min":0.504,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86799999999999999,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.83299999999999996}]},{"osd":6,"last update":"Mon Mar 18 20:29:57 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.63400000000000001,"15min":0.67100000000000004},"min":{"1min":0.46000000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.879,"5min":1.3120000000000001,"15min":1.637},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.66000000000000003,"15min":0.69599999999999995},"min":{"1min":0.51000000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.91200000000000003,"5min":0.93500000000000005,"15min":1.722},"last":0.73899999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:29:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.66100000000000003,"15min":0.69499999999999995},"min":{"1min":0.53000000000000003,"5min":0.42899999999999999,"15min":0.39500000000000002},"max":{"1min":0.96099999999999997,"5min":1.375,"15min":1.375},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.66500000000000004,"15min":0.69299999999999995},"min":{"1min":0.48899999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.89700000000000002,"5min":1.431,"15min":1.431},"last":0.79000000000000004}]}]},{"osd":0,"up_from":13,"seq":55834576035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49292,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94322548,"statfs":{"total":96636764160,"available":96586289152,"internally_reserved":0,"allocated":4329472,"data_stored":2528024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.35899999999999999,"15min":0.38},"min":{"1min":0.25,"5min":0.214,"15min":0.214},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.38900000000000001,"15min":0.41599999999999998},"min":{"1min":0.30599999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":1.1850000000000001},"last":0.39900000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.41199999999999998,"15min":0.42899999999999999},"min":{"1min":0.34899999999999998,"5min":0.27100000000000002,"15min":0.24099999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.41099999999999998,"15min":0.439},"min":{"1min":0.28799999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.70799999999999996},"last":0.56799999999999995}]},{"osd":3,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.38700000000000001,"15min":0.41699999999999998},"min":{"1min":0.312,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.57499999999999996,"5min":0.58499999999999996,"15min":0.628},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.45200000000000001,"15min":0.47899999999999998},"min":{"1min":0.309,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.91500000000000004},"last":0.52800000000000002}]},{"osd":4,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.61299999999999999,"15min":0.64000000000000001},"min":{"1min":0.40100000000000002,"5min":0.40000000000000002,"15min":0.36799999999999999},"max":{"1min":0.91200000000000003,"5min":1.506,"15min":1.506},"last":0.626},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.443,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.01,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.58999999999999997}]},{"osd":5,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.65000000000000002,"15min":0.66400000000000003},"min":{"1min":0.439,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.002,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.64500000000000002,"15min":0.67300000000000004},"min":{"1min":0.53000000000000003,"5min":0.432,"15min":0.432},"max":{"1min":1.006,"5min":1.468,"15min":1.518},"last":0.71999999999999997}]},{"osd":6,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.63400000000000001,"15min":0.65600000000000003},"min":{"1min":0.52700000000000002,"5min":0.436,"15min":0.436},"max":{"1min":0.92000000000000004,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.67400000000000004,"15min":0.69699999999999995},"min":{"1min":0.56200000000000006,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0860000000000001,"5min":1.5649999999999999,"15min":1.5649999999999999},"last":0.73999999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:29:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.67100000000000004,"15min":0.69499999999999995},"min":{"1min":0.504,"5min":0.45300000000000001,"15min":0.41899999999999998},"max":{"1min":1.016,"5min":1.55,"15min":1.55},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.65200000000000002,"15min":0.68700000000000006},"min":{"1min":0.57299999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0269999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.70799999999999996}]}]},{"osd":2,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52812,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319028,"statfs":{"total":96636764160,"available":96582684672,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.33900000000000002,"15min":0.36399999999999999},"min":{"1min":0.28299999999999997,"5min":0.184,"15min":0.184},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.439},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.42899999999999999,"15min":0.46600000000000003},"min":{"1min":0.34300000000000003,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.753},"last":0.51700000000000002}]},{"osd":1,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.433,"15min":0.46400000000000002},"min":{"1min":0.40500000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.749},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.36399999999999999,"15min":0.39900000000000002},"min":{"1min":0.307,"5min":0.224,"15min":0.224},"max":{"1min":0.54000000000000004,"5min":0.54600000000000004,"15min":0.59999999999999998},"last":0.41199999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.38600000000000001,"15min":0.41899999999999998},"min":{"1min":0.32800000000000001,"5min":0.26500000000000001,"15min":0.22800000000000001},"max":{"1min":0.623,"5min":0.64200000000000002,"15min":0.66300000000000003},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.44700000000000001,"15min":0.49399999999999999},"min":{"1min":0.372,"5min":0.315,"15min":0.315},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.752},"last":0.48499999999999999}]},{"osd":4,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.63300000000000001,"15min":0.64900000000000002},"min":{"1min":0.505,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.093,"5min":1.2090000000000001,"15min":1.4079999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.62,"15min":0.65500000000000003},"min":{"1min":0.39900000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98799999999999999,"5min":1.268,"15min":1.2729999999999999},"last":0.60999999999999999}]},{"osd":5,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.624,"15min":0.66300000000000003},"min":{"1min":0.53400000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1,"5min":1,"15min":1.028},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.63200000000000001,"15min":0.66400000000000003},"min":{"1min":0.56599999999999995,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.65800000000000003}]},{"osd":6,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.626,"15min":0.67000000000000004},"min":{"1min":0.52900000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.69599999999999995,"15min":0.72099999999999997},"min":{"1min":0.499,"5min":0.439,"15min":0.439},"max":{"1min":1.1299999999999999,"5min":1.3220000000000001,"15min":1.3859999999999999},"last":0.625}]},{"osd":7,"last update":"Mon Mar 18 20:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.63500000000000001,"15min":0.67900000000000005},"min":{"1min":0.497,"5min":0.432,"15min":0.42199999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0269999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.746,"5min":0.67000000000000004,"15min":0.69299999999999995},"min":{"1min":0.50600000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1739999999999999,"5min":1.2330000000000001,"15min":1.371},"last":0.68100000000000005}]}]},{"osd":3,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56908,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94314932,"statfs":{"total":96636764160,"available":96578490368,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.34200000000000003,"15min":0.378},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.66500000000000004},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.35999999999999999,"15min":0.39900000000000002},"min":{"1min":0.27200000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.61699999999999999},"last":0.38900000000000001}]},{"osd":1,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.35499999999999998,"15min":0.39000000000000001},"min":{"1min":0.28100000000000003,"5min":0.214,"15min":0.214},"max":{"1min":0.55100000000000005,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.41099999999999998,"15min":0.44700000000000001},"min":{"1min":0.32800000000000001,"5min":0.29099999999999998,"15min":0.25800000000000001},"max":{"1min":0.60399999999999998,"5min":0.63500000000000001,"15min":0.72599999999999998},"last":0.41799999999999998}]},{"osd":2,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.40799999999999997,"15min":0.44400000000000001},"min":{"1min":0.32600000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.68400000000000005},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.502,"5min":0.433,"15min":0.47299999999999998},"min":{"1min":0.36299999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.71199999999999997},"last":0.59299999999999997}]},{"osd":4,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.439,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98499999999999999,"5min":1.526,"15min":1.526},"last":0.503},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.622,"15min":0.65600000000000003},"min":{"1min":0.45500000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97899999999999998,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.65600000000000003,"15min":0.67700000000000005},"min":{"1min":0.41299999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94399999999999995,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.66100000000000003,"15min":0.70799999999999996},"min":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.99099999999999999,"5min":1.1120000000000001,"15min":1.171},"last":0.67300000000000004}]},{"osd":6,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.66000000000000003,"15min":0.68899999999999995},"min":{"1min":0.42499999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.878,"5min":1.573,"15min":1.573},"last":0.621},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.67300000000000004,"15min":0.70999999999999996},"min":{"1min":0.48999999999999999,"5min":0.47399999999999998,"15min":0.45100000000000001},"max":{"1min":1.02,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.60899999999999999}]},{"osd":7,"last update":"Mon Mar 18 20:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63700000000000001,"15min":0.68999999999999995},"min":{"1min":0.42799999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.032,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.65300000000000002,"15min":0.68200000000000005},"min":{"1min":0.439,"5min":0.41099999999999998,"15min":0.39200000000000002},"max":{"1min":0.94699999999999995,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.57599999999999996}]}]},{"osd":4,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48720,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.55600000000000005,"15min":0.57999999999999996},"min":{"1min":0.44,"5min":0.34300000000000003,"15min":0.29499999999999998},"max":{"1min":0.77600000000000002,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56599999999999995,"15min":0.58399999999999996},"min":{"1min":0.46700000000000003,"5min":0.40100000000000002,"15min":0.36299999999999999},"max":{"1min":0.79200000000000004,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.67800000000000005}]},{"osd":1,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.58199999999999996,"15min":0.60299999999999998},"min":{"1min":0.45500000000000002,"5min":0.35799999999999998,"15min":0.318},"max":{"1min":0.81000000000000005,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.55500000000000005,"15min":0.57299999999999995},"min":{"1min":0.496,"5min":0.34899999999999998,"15min":0.34300000000000003},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.96499999999999997},"last":0.64900000000000002}]},{"osd":2,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.58699999999999997,"15min":0.59199999999999997},"min":{"1min":0.52800000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.82699999999999996,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.63,"5min":0.58899999999999997,"15min":0.60499999999999998},"min":{"1min":0.40200000000000002,"5min":0.38900000000000001,"15min":0.38500000000000001},"max":{"1min":0.76700000000000002,"5min":0.83099999999999996,"15min":0.84099999999999997},"last":0.60399999999999998}]},{"osd":3,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58099999999999996,"15min":0.59199999999999997},"min":{"1min":0.5,"5min":0.374,"15min":0.33900000000000002},"max":{"1min":0.82799999999999996,"5min":1.026,"15min":1.026},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.60499999999999998,"15min":0.61099999999999999},"min":{"1min":0.503,"5min":0.36699999999999999,"15min":0.34999999999999998},"max":{"1min":0.85499999999999998,"5min":1.028,"15min":1.028},"last":0.57499999999999996}]},{"osd":5,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.499,"15min":0.48199999999999998},"min":{"1min":0.34799999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.67400000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.52400000000000002,"15min":0.505},"min":{"1min":0.40799999999999997,"5min":0.31900000000000001,"15min":0.29999999999999999},"max":{"1min":0.68100000000000005,"5min":0.85499999999999998,"15min":0.90600000000000003},"last":0.58599999999999997}]},{"osd":6,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52700000000000002,"15min":0.50600000000000001},"min":{"1min":0.36799999999999999,"5min":0.32200000000000001,"15min":0.32000000000000001},"max":{"1min":0.748,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.56999999999999995,"15min":0.55300000000000005},"min":{"1min":0.435,"5min":0.34499999999999997,"15min":0.33500000000000002},"max":{"1min":0.77900000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.54600000000000004}]},{"osd":7,"last update":"Mon Mar 18 20:29:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56599999999999995,"15min":0.54800000000000004},"min":{"1min":0.38700000000000001,"5min":0.36899999999999999,"15min":0.35699999999999998},"max":{"1min":0.70499999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.495},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60699999999999998,"15min":0.58399999999999996},"min":{"1min":0.45900000000000002,"5min":0.36599999999999999,"15min":0.33700000000000002},"max":{"1min":0.79200000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.61899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834576034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70860,"kb_used_data":3648,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94300980,"statfs":{"total":96636764160,"available":96564203520,"internally_reserved":0,"allocated":3735552,"data_stored":1937656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 18 20:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.53900000000000003,"15min":0.58199999999999996},"min":{"1min":0.36399999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76500000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.53600000000000003,"15min":0.58099999999999996},"min":{"1min":0.36899999999999999,"5min":0.33300000000000002,"15min":0.33100000000000002},"max":{"1min":0.73899999999999999,"5min":0.82399999999999995,"15min":1.3100000000000001},"last":0.59399999999999997}]},{"osd":1,"last update":"Mon Mar 18 20:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55000000000000004,"15min":0.58999999999999997},"min":{"1min":0.32900000000000001,"5min":0.32600000000000001,"15min":0.31900000000000001},"max":{"1min":0.75,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.53700000000000003,"15min":0.57899999999999996},"min":{"1min":0.41399999999999998,"5min":0.372,"15min":0.35199999999999998},"max":{"1min":0.72499999999999998,"5min":0.94299999999999995,"15min":1.018},"last":0.54900000000000004}]},{"osd":2,"last update":"Mon Mar 18 20:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.53600000000000003,"15min":0.58199999999999996},"min":{"1min":0.377,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.70099999999999996,"5min":0.85399999999999998,"15min":0.99299999999999999},"last":0.626},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55100000000000005,"15min":0.59199999999999997},"min":{"1min":0.38700000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73299999999999998,"5min":1.599,"15min":1.599},"last":0.68899999999999995}]},{"osd":3,"last update":"Mon Mar 18 20:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55500000000000005,"15min":0.59599999999999997},"min":{"1min":0.39600000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.80500000000000005,"5min":0.98699999999999999,"15min":1.2430000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58299999999999996,"15min":0.628},"min":{"1min":0.42499999999999999,"5min":0.40600000000000003,"15min":0.36699999999999999},"max":{"1min":0.78000000000000003,"5min":0.84399999999999997,"15min":1.161},"last":0.67600000000000005}]},{"osd":4,"last update":"Mon Mar 18 20:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.48799999999999999,"15min":0.48199999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.58499999999999996,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.497},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53200000000000003,"15min":0.51900000000000002},"min":{"1min":0.33800000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.77600000000000002,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.45400000000000001}]},{"osd":6,"last update":"Mon Mar 18 20:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.308,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64400000000000002,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.53300000000000003,"15min":0.53800000000000003},"min":{"1min":0.35299999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.68500000000000005,"5min":1.643,"15min":1.643},"last":0.57199999999999995}]},{"osd":7,"last update":"Mon Mar 18 20:29:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.747,"5min":1.028,"15min":1.028},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.57299999999999995,"15min":0.57599999999999996},"min":{"1min":0.36099999999999999,"5min":0.34300000000000003,"15min":0.29799999999999999},"max":{"1min":0.75600000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.65000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T20:30:23.614 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-18T20:30:23.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.735+0000 7f6811cf6700 1 Processor -- start 2024-03-18T20:30:23.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.735+0000 7f6811cf6700 1 -- start start 2024-03-18T20:30:23.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.735+0000 7f6811cf6700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f680c140820 0x7f680c06e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:23.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.736+0000 7f6811cf6700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f680c06e9a0 0x7f680c0cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:23.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.736+0000 7f6811cf6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f680c0ccbe0 0x7f680c15dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:23.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.736+0000 7f6811cf6700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f680c0ce8a0 con 0x7f680c140820 2024-03-18T20:30:23.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.736+0000 7f6811cf6700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f680c06ee20 con 0x7f680c06e9a0 2024-03-18T20:30:23.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.736+0000 7f6811cf6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f680c0ceab0 con 0x7f680c0ccbe0 2024-03-18T20:30:23.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.736+0000 7f680b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f680c140820 0x7f680c06e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:23.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.736+0000 7f680b7fe700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f680c140820 0x7f680c06e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49382/0 (socket says 172.21.15.1:49382) 2024-03-18T20:30:23.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.736+0000 7f680b7fe700 1 -- 172.21.15.1:0/580832525 learned_addr learned my addr 172.21.15.1:0/580832525 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:23.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.736+0000 7f680affd700 1 --2- 172.21.15.1:0/580832525 >> v2:172.21.15.165:3300/0 conn(0x7f680c06e9a0 0x7f680c0cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:23.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.736+0000 7f680bfff700 1 --2- 172.21.15.1:0/580832525 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f680c0ccbe0 0x7f680c15dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:23.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.736+0000 7f680b7fe700 1 -- 172.21.15.1:0/580832525 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f680c0ccbe0 msgr2=0x7f680c15dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:23.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.736+0000 7f680b7fe700 1 --2- 172.21.15.1:0/580832525 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f680c0ccbe0 0x7f680c15dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.736+0000 7f680b7fe700 1 -- 172.21.15.1:0/580832525 >> v2:172.21.15.165:3300/0 conn(0x7f680c06e9a0 msgr2=0x7f680c0cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:23.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.736+0000 7f680b7fe700 1 --2- 172.21.15.1:0/580832525 >> v2:172.21.15.165:3300/0 conn(0x7f680c06e9a0 0x7f680c0cc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.736+0000 7f680b7fe700 1 -- 172.21.15.1:0/580832525 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680c0ced90 con 0x7f680c140820 2024-03-18T20:30:23.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.736+0000 7f680b7fe700 1 --2- 172.21.15.1:0/580832525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f680c140820 0x7f680c06e460 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7f680c0679c0 tx=0x7f6800009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83705ee1ff0b65b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:23.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.736+0000 7f680a7fc700 1 -- 172.21.15.1:0/580832525 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6800011830 con 0x7f680c140820 2024-03-18T20:30:23.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.736+0000 7f680a7fc700 1 -- 172.21.15.1:0/580832525 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6800011990 con 0x7f680c140820 2024-03-18T20:30:23.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.736+0000 7f680a7fc700 1 -- 172.21.15.1:0/580832525 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6800011d20 con 0x7f680c140820 2024-03-18T20:30:23.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.737+0000 7f6811cf6700 1 -- 172.21.15.1:0/580832525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f680c140820 msgr2=0x7f680c06e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:23.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.737+0000 7f6811cf6700 1 --2- 172.21.15.1:0/580832525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f680c140820 0x7f680c06e460 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7f680c0679c0 tx=0x7f6800009bd0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.737+0000 7f6811cf6700 1 -- 172.21.15.1:0/580832525 shutdown_connections 2024-03-18T20:30:23.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.737+0000 7f6811cf6700 1 --2- 172.21.15.1:0/580832525 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f680c0ccbe0 0x7f680c15dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.737+0000 7f6811cf6700 1 --2- 172.21.15.1:0/580832525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f680c140820 0x7f680c06e460 unknown :-1 s=CLOSED pgs=3464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.737+0000 7f6811cf6700 1 --2- 172.21.15.1:0/580832525 >> v2:172.21.15.165:3300/0 conn(0x7f680c06e9a0 0x7f680c0cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.737+0000 7f6811cf6700 1 -- 172.21.15.1:0/580832525 >> 172.21.15.1:0/580832525 conn(0x7f680c072080 msgr2=0x7f680c06a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:23.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.737+0000 7f6811cf6700 1 -- 172.21.15.1:0/580832525 shutdown_connections 2024-03-18T20:30:23.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.737+0000 7f6811cf6700 1 -- 172.21.15.1:0/580832525 wait complete. 2024-03-18T20:30:23.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.738+0000 7f6811cf6700 1 Processor -- start 2024-03-18T20:30:23.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.738+0000 7f6811cf6700 1 -- start start 2024-03-18T20:30:23.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.739+0000 7f6811cf6700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f680c06e9a0 0x7f680c1f3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:23.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.739+0000 7f6811cf6700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f680c0ccbe0 0x7f680c139470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:23.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.739+0000 7f6811cf6700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f680c140820 0x7f680c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:23.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.739+0000 7f6811cf6700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c1f43d0 con 0x7f680c0ccbe0 2024-03-18T20:30:23.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.739+0000 7f6811cf6700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c139d30 con 0x7f680c06e9a0 2024-03-18T20:30:23.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.739+0000 7f6811cf6700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f680c13a010 con 0x7f680c140820 2024-03-18T20:30:23.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.739+0000 7f680affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f680c0ccbe0 0x7f680c139470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:23.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.739+0000 7f680affd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f680c0ccbe0 0x7f680c139470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49388/0 (socket says 172.21.15.1:49388) 2024-03-18T20:30:23.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.739+0000 7f680affd700 1 -- 172.21.15.1:0/941401800 learned_addr learned my addr 172.21.15.1:0/941401800 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:23.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.739+0000 7f680b7fe700 1 --2- 172.21.15.1:0/941401800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f680c06e9a0 0x7f680c1f3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:23.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.739+0000 7f680bfff700 1 --2- 172.21.15.1:0/941401800 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f680c140820 0x7f680c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:23.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.739+0000 7f680affd700 1 -- 172.21.15.1:0/941401800 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f680c140820 msgr2=0x7f680c06d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:23.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.739+0000 7f680affd700 1 --2- 172.21.15.1:0/941401800 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f680c140820 0x7f680c06d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.739+0000 7f680affd700 1 -- 172.21.15.1:0/941401800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f680c06e9a0 msgr2=0x7f680c1f3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:23.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.739+0000 7f680affd700 1 --2- 172.21.15.1:0/941401800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f680c06e9a0 0x7f680c1f3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.739+0000 7f680affd700 1 -- 172.21.15.1:0/941401800 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6800009550 con 0x7f680c0ccbe0 2024-03-18T20:30:23.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.739+0000 7f680b7fe700 1 --2- 172.21.15.1:0/941401800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f680c06e9a0 0x7f680c1f3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T20:30:23.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.740+0000 7f680affd700 1 --2- 172.21.15.1:0/941401800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f680c0ccbe0 0x7f680c139470 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00e870 tx=0x7f67fc00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:23.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.740+0000 7f67f3fff700 1 -- 172.21.15.1:0/941401800 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc009590 con 0x7f680c0ccbe0 2024-03-18T20:30:23.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.740+0000 7f67f3fff700 1 -- 172.21.15.1:0/941401800 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67fc00ee70 con 0x7f680c0ccbe0 2024-03-18T20:30:23.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.740+0000 7f6811cf6700 1 -- 172.21.15.1:0/941401800 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680c13a2f0 con 0x7f680c0ccbe0 2024-03-18T20:30:23.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.740+0000 7f67f3fff700 1 -- 172.21.15.1:0/941401800 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc00ca50 con 0x7f680c0ccbe0 2024-03-18T20:30:23.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.740+0000 7f6811cf6700 1 -- 172.21.15.1:0/941401800 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680c13a8b0 con 0x7f680c0ccbe0 2024-03-18T20:30:23.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.741+0000 7f6811cf6700 1 -- 172.21.15.1:0/941401800 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f680c067dc0 con 0x7f680c0ccbe0 2024-03-18T20:30:23.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.741+0000 7f67f3fff700 1 -- 172.21.15.1:0/941401800 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7f67fc00cbb0 con 0x7f680c0ccbe0 2024-03-18T20:30:23.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.742+0000 7f67f3fff700 1 --2- 172.21.15.1:0/941401800 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f67f806adb0 0x7f67f806d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:23.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.742+0000 7f67f3fff700 1 -- 172.21.15.1:0/941401800 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f67fc089540 con 0x7f680c0ccbe0 2024-03-18T20:30:23.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.742+0000 7f680b7fe700 1 --2- 172.21.15.1:0/941401800 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f67f806adb0 0x7f67f806d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:23.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.742+0000 7f680b7fe700 1 --2- 172.21.15.1:0/941401800 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f67f806adb0 0x7f67f806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6800009840 tx=0x7f6800008030 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:23.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.746+0000 7f67f3fff700 1 -- 172.21.15.1:0/941401800 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f67fc054520 con 0x7f680c0ccbe0 2024-03-18T20:30:23.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.950+0000 7f6811cf6700 1 -- 172.21.15.1:0/941401800 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f680c066720 con 0x7f680c0ccbe0 2024-03-18T20:30:23.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.953+0000 7f67f3fff700 1 -- 172.21.15.1:0/941401800 <== mon.0 v2:172.21.15.1:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f67fc054520 con 0x7f680c0ccbe0 2024-03-18T20:30:23.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.953+0000 7f67f3fff700 1 -- 172.21.15.1:0/941401800 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v100)=0 v100) v1 ==== 123+0+0 (secure 0 0 0) 0x7f67fc00ce60 con 0x7f680c0ccbe0 2024-03-18T20:30:23.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.956+0000 7f6811cf6700 1 -- 172.21.15.1:0/941401800 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f67f806adb0 msgr2=0x7f67f806d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:23.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.956+0000 7f6811cf6700 1 --2- 172.21.15.1:0/941401800 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f67f806adb0 0x7f67f806d270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6800009840 tx=0x7f6800008030 comp rx=0 tx=0).stop 2024-03-18T20:30:23.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.956+0000 7f6811cf6700 1 -- 172.21.15.1:0/941401800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f680c0ccbe0 msgr2=0x7f680c139470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:23.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.956+0000 7f6811cf6700 1 --2- 172.21.15.1:0/941401800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f680c0ccbe0 0x7f680c139470 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00e870 tx=0x7f67fc00ec30 comp rx=0 tx=0).stop 2024-03-18T20:30:23.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.956+0000 7f6811cf6700 1 -- 172.21.15.1:0/941401800 shutdown_connections 2024-03-18T20:30:23.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.956+0000 7f6811cf6700 1 --2- 172.21.15.1:0/941401800 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7f67f806adb0 0x7f67f806d270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.956+0000 7f6811cf6700 1 --2- 172.21.15.1:0/941401800 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7f680c140820 0x7f680c06d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.956+0000 7f6811cf6700 1 --2- 172.21.15.1:0/941401800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f680c06e9a0 0x7f680c1f3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.956+0000 7f6811cf6700 1 --2- 172.21.15.1:0/941401800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f680c0ccbe0 0x7f680c139470 unknown :-1 s=CLOSED pgs=3465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:23.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.956+0000 7f6811cf6700 1 -- 172.21.15.1:0/941401800 >> 172.21.15.1:0/941401800 conn(0x7f680c072080 msgr2=0x7f680c13cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:23.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.956+0000 7f6811cf6700 1 -- 172.21.15.1:0/941401800 shutdown_connections 2024-03-18T20:30:23.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:23.956+0000 7f6811cf6700 1 -- 172.21.15.1:0/941401800 wait complete. 2024-03-18T20:30:23.973 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T20:30:23.973 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T20:30:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.094+0000 7face27bd700 1 Processor -- start 2024-03-18T20:30:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.094+0000 7face27bd700 1 -- start start 2024-03-18T20:30:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.095+0000 7face27bd700 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7facdc140760 0x7facdc0cf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.095+0000 7face27bd700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7facdc06e3f0 0x7facdc06e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.095+0000 7face27bd700 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facdc0cf960 con 0x7facdc140380 2024-03-18T20:30:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.095+0000 7face27bd700 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facdc0cd230 con 0x7facdc06e3f0 2024-03-18T20:30:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.095+0000 7face27bd700 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facdc06eda0 con 0x7facdc140760 2024-03-18T20:30:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.095+0000 7face0d5a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7facdc06e3f0 0x7facdc06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.095+0000 7face0d5a700 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7facdc06e3f0 0x7facdc06e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49404/0 (socket says 172.21.15.1:49404) 2024-03-18T20:30:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.095+0000 7face0d5a700 1 -- 172.21.15.1:0/2533394490 learned_addr learned my addr 172.21.15.1:0/2533394490 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.095+0000 7face0d5a700 1 -- 172.21.15.1:0/2533394490 >> v2:172.21.15.165:3300/0 conn(0x7facdc140760 msgr2=0x7facdc0cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.095+0000 7facdb7fe700 1 --2- 172.21.15.1:0/2533394490 >> v2:172.21.15.165:3300/0 conn(0x7facdc140760 0x7facdc0cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.095+0000 7face0d5a700 1 --2- 172.21.15.1:0/2533394490 >> v2:172.21.15.165:3300/0 conn(0x7facdc140760 0x7facdc0cf420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.095+0000 7face0d5a700 1 -- 172.21.15.1:0/2533394490 >> v1:172.21.15.165:6789/0 conn(0x7facdc140380 legacy=0x7facdc0cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T20:30:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.095+0000 7face0d5a700 1 -- 172.21.15.1:0/2533394490 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc0ce6c0 con 0x7facdc06e3f0 2024-03-18T20:30:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.095+0000 7face0d5a700 1 --2- 172.21.15.1:0/2533394490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facdc06e3f0 0x7facdc06e860 secure :-1 s=READY pgs=3466 cs=0 l=1 rev1=1 crypto rx=0x7facdc074410 tx=0x7facd400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85c2e67f66872f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.096+0000 7facdaffd700 1 -- 172.21.15.1:0/2533394490 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd400e070 con 0x7facdc06e3f0 2024-03-18T20:30:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.096+0000 7facdaffd700 1 -- 172.21.15.1:0/2533394490 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7facd4003a20 con 0x7facdc06e3f0 2024-03-18T20:30:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.096+0000 7facdaffd700 1 -- 172.21.15.1:0/2533394490 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd4003df0 con 0x7facdc06e3f0 2024-03-18T20:30:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.096+0000 7face27bd700 1 -- 172.21.15.1:0/2533394490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facdc06e3f0 msgr2=0x7facdc06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.096+0000 7face27bd700 1 --2- 172.21.15.1:0/2533394490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facdc06e3f0 0x7facdc06e860 secure :-1 s=READY pgs=3466 cs=0 l=1 rev1=1 crypto rx=0x7facdc074410 tx=0x7facd400b330 comp rx=0 tx=0).stop 2024-03-18T20:30:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.096+0000 7face27bd700 1 -- 172.21.15.1:0/2533394490 shutdown_connections 2024-03-18T20:30:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.096+0000 7face27bd700 1 --2- 172.21.15.1:0/2533394490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facdc06e3f0 0x7facdc06e860 unknown :-1 s=CLOSED pgs=3466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.096+0000 7face27bd700 1 --2- 172.21.15.1:0/2533394490 >> v2:172.21.15.165:3300/0 conn(0x7facdc140760 0x7facdc0cf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.096+0000 7face27bd700 1 -- 172.21.15.1:0/2533394490 >> 172.21.15.1:0/2533394490 conn(0x7facdc0720c0 msgr2=0x7facdc06a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.096+0000 7face27bd700 1 -- 172.21.15.1:0/2533394490 shutdown_connections 2024-03-18T20:30:24.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.096+0000 7face27bd700 1 -- 172.21.15.1:0/2533394490 wait complete. 2024-03-18T20:30:24.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.098+0000 7face27bd700 1 Processor -- start 2024-03-18T20:30:24.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.098+0000 7face27bd700 1 -- start start 2024-03-18T20:30:24.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.099+0000 7face27bd700 1 --2- >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facdc06e3f0 0x7facdc137cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:24.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.099+0000 7face27bd700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facdc140380 0x7facdc1508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:24.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.099+0000 7face27bd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facdc140760 0x7facdc06d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:24.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.099+0000 7face27bd700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facdc150e00 con 0x7facdc140380 2024-03-18T20:30:24.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.099+0000 7face27bd700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facdc159c70 con 0x7facdc140760 2024-03-18T20:30:24.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.099+0000 7face27bd700 1 -- --> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facdc136f10 con 0x7facdc06e3f0 2024-03-18T20:30:24.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.099+0000 7facdb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facdc140380 0x7facdc1508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:24.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.099+0000 7facdb7fe700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facdc140380 0x7facdc1508c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49420/0 (socket says 172.21.15.1:49420) 2024-03-18T20:30:24.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.099+0000 7facdb7fe700 1 -- 172.21.15.1:0/1233374234 learned_addr learned my addr 172.21.15.1:0/1233374234 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-03-18T20:30:24.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.099+0000 7face0d5a700 1 --2- 172.21.15.1:0/1233374234 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facdc140760 0x7facdc06d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:24.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.099+0000 7facdbfff700 1 --2- 172.21.15.1:0/1233374234 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facdc06e3f0 0x7facdc137cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:24.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.099+0000 7facdb7fe700 1 -- 172.21.15.1:0/1233374234 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facdc06e3f0 msgr2=0x7facdc137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:24.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.099+0000 7facdb7fe700 1 --2- 172.21.15.1:0/1233374234 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facdc06e3f0 0x7facdc137cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:24.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.099+0000 7facdb7fe700 1 -- 172.21.15.1:0/1233374234 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facdc140760 msgr2=0x7facdc06d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:24.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.099+0000 7facdb7fe700 1 --2- 172.21.15.1:0/1233374234 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facdc140760 0x7facdc06d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:24.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.099+0000 7facdb7fe700 1 -- 172.21.15.1:0/1233374234 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd4009b30 con 0x7facdc140380 2024-03-18T20:30:24.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.099+0000 7facdb7fe700 1 --2- 172.21.15.1:0/1233374234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facdc140380 0x7facdc1508c0 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7facc400c7f0 tx=0x7facc400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:24.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.100+0000 7facd8ff9700 1 -- 172.21.15.1:0/1233374234 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc4011800 con 0x7facdc140380 2024-03-18T20:30:24.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.100+0000 7facd8ff9700 1 -- 172.21.15.1:0/1233374234 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7facc4011960 con 0x7facdc140380 2024-03-18T20:30:24.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.100+0000 7face27bd700 1 -- 172.21.15.1:0/1233374234 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facdc1371f0 con 0x7facdc140380 2024-03-18T20:30:24.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.100+0000 7facd8ff9700 1 -- 172.21.15.1:0/1233374234 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc4011d10 con 0x7facdc140380 2024-03-18T20:30:24.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.100+0000 7face27bd700 1 -- 172.21.15.1:0/1233374234 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facdc1377b0 con 0x7facdc140380 2024-03-18T20:30:24.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.101+0000 7face27bd700 1 -- 172.21.15.1:0/1233374234 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facdc06ca20 con 0x7facdc140380 2024-03-18T20:30:24.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.101+0000 7facd8ff9700 1 -- 172.21.15.1:0/1233374234 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 179) v1 ==== 89239+0+0 (secure 0 0 0) 0x7facc4018360 con 0x7facdc140380 2024-03-18T20:30:24.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.101+0000 7facd8ff9700 1 --2- 172.21.15.1:0/1233374234 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facbc06adb0 0x7facbc06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T20:30:24.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.102+0000 7facd8ff9700 1 -- 172.21.15.1:0/1233374234 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(295..295 src has 1..295) v4 ==== 6094+0+0 (secure 0 0 0) 0x7facc4089780 con 0x7facdc140380 2024-03-18T20:30:24.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.102+0000 7facdbfff700 1 --2- 172.21.15.1:0/1233374234 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facbc06adb0 0x7facbc06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T20:30:24.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.102+0000 7facdbfff700 1 --2- 172.21.15.1:0/1233374234 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facbc06adb0 0x7facbc06d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faccc001730 tx=0x7faccc009040 comp rx=0 tx=0).ready entity=mgr.14091 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T20:30:24.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.106+0000 7facd8ff9700 1 -- 172.21.15.1:0/1233374234 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7facc4054530 con 0x7facdc140380 2024-03-18T20:30:24.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.358+0000 7face27bd700 1 -- 172.21.15.1:0/1233374234 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facdc067dc0 con 0x7facdc140380 2024-03-18T20:30:24.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.359+0000 7facd8ff9700 1 -- 172.21.15.1:0/1233374234 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+3538 (secure 0 0 0) 0x7facc401fdf0 con 0x7facdc140380 2024-03-18T20:30:24.363 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-18T20:30:24.363 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":146,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10322,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6834/2385318877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":2385318877},{"type":"v1","addr":"172.21.15.165:6834","nonce":2385318877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139},{"gid":12040,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/1024465047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6833","nonce":1024465047},{"type":"v1","addr":"172.21.15.165:6835","nonce":1024465047}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139},{"gid":13791,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/2374816325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2374816325},{"type":"v1","addr":"172.21.15.1:6837","nonce":2374816325}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":139},{"gid":14235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6836/4203557131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6835","nonce":4203557131},{"type":"v1","addr":"172.21.15.1:6836","nonce":4203557131}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146}],"filesystems":[]} 2024-03-18T20:30:24.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.361+0000 7face27bd700 1 -- 172.21.15.1:0/1233374234 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facbc06adb0 msgr2=0x7facbc06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:24.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.361+0000 7face27bd700 1 --2- 172.21.15.1:0/1233374234 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facbc06adb0 0x7facbc06d270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faccc001730 tx=0x7faccc009040 comp rx=0 tx=0).stop 2024-03-18T20:30:24.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.361+0000 7face27bd700 1 -- 172.21.15.1:0/1233374234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facdc140380 msgr2=0x7facdc1508c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T20:30:24.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.361+0000 7face27bd700 1 --2- 172.21.15.1:0/1233374234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facdc140380 0x7facdc1508c0 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7facc400c7f0 tx=0x7facc400cbb0 comp rx=0 tx=0).stop 2024-03-18T20:30:24.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.361+0000 7face27bd700 1 -- 172.21.15.1:0/1233374234 shutdown_connections 2024-03-18T20:30:24.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.361+0000 7face27bd700 1 --2- 172.21.15.1:0/1233374234 >> [v2:172.21.15.1:6832/148356,v1:172.21.15.1:6833/148356] conn(0x7facbc06adb0 0x7facbc06d270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:24.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.361+0000 7face27bd700 1 --2- 172.21.15.1:0/1233374234 >> [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] conn(0x7facdc06e3f0 0x7facdc137cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:24.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.361+0000 7face27bd700 1 --2- 172.21.15.1:0/1233374234 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facdc140760 0x7facdc06d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:24.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.361+0000 7face27bd700 1 --2- 172.21.15.1:0/1233374234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facdc140380 0x7facdc1508c0 unknown :-1 s=CLOSED pgs=3467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T20:30:24.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.361+0000 7face27bd700 1 -- 172.21.15.1:0/1233374234 >> 172.21.15.1:0/1233374234 conn(0x7facdc0720c0 msgr2=0x7facdc13e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T20:30:24.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.361+0000 7face27bd700 1 -- 172.21.15.1:0/1233374234 shutdown_connections 2024-03-18T20:30:24.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-03-18T20:30:24.361+0000 7face27bd700 1 -- 172.21.15.1:0/1233374234 wait complete. 2024-03-18T20:30:24.367 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 146 2024-03-18T20:30:24.377 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-18T20:30:24.377 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-18T20:30:24.378 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T20:30:24.378 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:30:24.386 INFO:tasks.ceph.mds.a:Stopped 2024-03-18T20:30:24.386 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-18T20:30:24.386 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:30:24.435 INFO:tasks.ceph.mds.c:Stopped 2024-03-18T20:30:24.436 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-18T20:30:24.436 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:30:24.447 INFO:tasks.ceph.mds.b:Stopped 2024-03-18T20:30:24.447 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-18T20:30:24.447 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:30:24.498 INFO:tasks.ceph.mds.d:Stopped 2024-03-18T20:30:24.498 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-18T20:30:24.498 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-18T20:30:24.498 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:30:24.530 INFO:tasks.ceph.osd.0:Stopped 2024-03-18T20:30:24.531 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-18T20:30:24.531 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:30:24.602 INFO:tasks.ceph.osd.1:Stopped 2024-03-18T20:30:24.603 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-18T20:30:24.603 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:30:24.677 INFO:tasks.ceph.osd.2:Stopped 2024-03-18T20:30:24.678 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-18T20:30:24.678 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:30:24.744 INFO:tasks.ceph.osd.3:Stopped 2024-03-18T20:30:24.744 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-18T20:30:24.744 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:30:24.786 INFO:tasks.ceph.osd.4:Stopped 2024-03-18T20:30:24.786 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-18T20:30:24.786 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:30:24.824 INFO:tasks.ceph.osd.5:Stopped 2024-03-18T20:30:24.824 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-18T20:30:24.825 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:30:24.894 INFO:tasks.ceph.osd.6:Stopped 2024-03-18T20:30:24.894 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-18T20:30:24.894 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:30:24.963 INFO:tasks.ceph.osd.7:Stopped 2024-03-18T20:30:24.963 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-18T20:30:24.964 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-18T20:30:24.964 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:30:24.995 INFO:tasks.ceph.mgr.y:Stopped 2024-03-18T20:30:24.995 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-18T20:30:24.995 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:30:25.034 INFO:tasks.ceph.mgr.x:Stopped 2024-03-18T20:30:25.035 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-18T20:30:25.035 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-18T20:30:25.035 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:30:25.051 INFO:tasks.ceph.mon.a:Stopped 2024-03-18T20:30:25.051 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-18T20:30:25.051 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:30:25.090 INFO:tasks.ceph.mon.b:Stopped 2024-03-18T20:30:25.090 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-18T20:30:25.090 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T20:30:25.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-03-18T20:30:25.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-03-18T20:30:25.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-03-18T20:30:25.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-03-18T20:30:25.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-03-18T20:30:25.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-03-18T20:30:25.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-03-18T20:30:25.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-03-18T20:30:25.147 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-03-18T20:30:25.147 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-03-18T20:30:25.147 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-03-18T20:30:25.147 INFO:tasks.ceph.mon.c:Stopped 2024-03-18T20:30:25.147 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-18T20:30:25.148 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-18T20:30:25.182 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:30:25.183 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-18T20:30:25.408 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:30:25.409 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-18T20:30:25.499 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:30:25.499 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-18T20:30:25.580 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:30:25.581 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-18T20:30:25.651 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:30:25.651 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-18T20:30:25.819 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:30:25.820 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-18T20:30:25.903 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:30:25.903 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-18T20:30:25.982 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:30:25.982 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-18T20:30:26.051 INFO:tasks.ceph:Archiving mon data... 2024-03-18T20:30:26.053 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606246/data/mon.a.tgz 2024-03-18T20:30:26.053 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-03-18T20:30:26.072 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.IBIajsoSxv 2024-03-18T20:30:26.073 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.IBIajsoSxv 2024-03-18T20:30:26.814 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.IBIajsoSxv 2024-03-18T20:30:26.903 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.IBIajsoSxv is 4MB 2024-03-18T20:30:27.077 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.IBIajsoSxv 2024-03-18T20:30:27.095 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606246/data/mon.b.tgz 2024-03-18T20:30:27.096 DEBUG:teuthology.orchestra.run.smithi165:> mktemp 2024-03-18T20:30:27.118 INFO:teuthology.orchestra.run.smithi165.stdout:/tmp/tmp.nnU9zgNd4y 2024-03-18T20:30:27.119 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.nnU9zgNd4y 2024-03-18T20:30:28.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0666 /tmp/tmp.nnU9zgNd4y 2024-03-18T20:30:28.204 DEBUG:teuthology.orchestra.remote:smithi165:/tmp/tmp.nnU9zgNd4y is 4MB 2024-03-18T20:30:28.374 DEBUG:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.nnU9zgNd4y 2024-03-18T20:30:28.392 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606246/data/mon.c.tgz 2024-03-18T20:30:28.392 DEBUG:teuthology.orchestra.run.smithi165:> mktemp 2024-03-18T20:30:28.449 INFO:teuthology.orchestra.run.smithi165.stdout:/tmp/tmp.6p2HTooi3H 2024-03-18T20:30:28.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.6p2HTooi3H 2024-03-18T20:30:29.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0666 /tmp/tmp.6p2HTooi3H 2024-03-18T20:30:29.236 DEBUG:teuthology.orchestra.remote:smithi165:/tmp/tmp.6p2HTooi3H is 4MB 2024-03-18T20:30:29.404 DEBUG:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.6p2HTooi3H 2024-03-18T20:30:29.422 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-18T20:30:29.422 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-18T20:30:29.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-18T20:30:29.489 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-03-18T20:30:29.489 DEBUG:teuthology.orchestra.run.smithi001:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-03-18T20:30:29.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-03-18T20:30:29.530 INFO:teuthology.orchestra.run.smithi001.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-03-18T20:30:29.556 INFO:teuthology.orchestra.run.smithi165.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-03-18T20:30:29.562 INFO:tasks.ceph:Archiving crash dumps... 2024-03-18T20:30:29.564 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606246/remote/smithi001/crash 2024-03-18T20:30:29.565 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-18T20:30:29.609 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606246/remote/smithi165/crash 2024-03-18T20:30:29.610 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-18T20:30:29.637 INFO:tasks.ceph:Shutting down logrotate 2024-03-18T20:30:29.638 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:30:29.655 DEBUG:teuthology.orchestra.run.smithi165:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T20:30:29.732 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-18T20:30:29.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-18T20:30:29.791 INFO:tasks.ceph:Compressing logs... 2024-03-18T20:30:29.791 DEBUG:teuthology.orchestra.run.smithi001:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T20:30:29.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T20:30:30.148 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-03-18T20:30:30.148 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-03-18T20:30:30.148 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-03-18T20:30:30.148 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-03-18T20:30:30.148 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-03-18T20:30:30.148 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-03-18T20:30:30.148 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-03-18T20:30:30.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-03-18T20:30:30.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-03-18T20:30:30.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-03-18T20:30:30.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-03-18T20:31:01.780 INFO:tasks.ceph:Archiving logs... 2024-03-18T20:31:01.780 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606246/remote/smithi001/log 2024-03-18T20:31:01.782 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T20:31:06.343 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606246/remote/smithi165/log 2024-03-18T20:31:06.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T20:31:09.254 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-18T20:31:09.264 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-18T20:31:09.264 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T20:31:09.267 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T20:31:09.359 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-18T20:31:09.368 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-18T20:31:09.368 DEBUG:teuthology.orchestra.run.smithi001:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T20:31:09.371 DEBUG:teuthology.orchestra.run.smithi165:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T20:31:09.389 INFO:teuthology.orchestra.run.smithi001.stderr:bash: ntpq: command not found 2024-03-18T20:31:09.390 INFO:teuthology.orchestra.run.smithi165.stderr:bash: ntpq: command not found 2024-03-18T20:31:09.394 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T20:31:09.395 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-03-18T20:31:09.395 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 24 +5184us[+5238us] +/- 109ms 2024-03-18T20:31:09.395 INFO:teuthology.orchestra.run.smithi001.stdout:^* hv02.front.sepia.ceph.com 3 6 377 18 -2655us[-2602us] +/- 74ms 2024-03-18T20:31:09.395 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 -828us[ -775us] +/- 80ms 2024-03-18T20:31:09.395 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T20:31:09.396 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T20:31:09.397 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-03-18T20:31:09.397 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 24 +5519us[+5466us] +/- 109ms 2024-03-18T20:31:09.397 INFO:teuthology.orchestra.run.smithi165.stdout:^* hv02.front.sepia.ceph.com 3 8 377 23 -2279us[-2331us] +/- 74ms 2024-03-18T20:31:09.397 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 22 -503us[ -503us] +/- 80ms 2024-03-18T20:31:09.397 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T20:31:09.397 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-18T20:31:09.406 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-18T20:31:09.407 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-18T20:31:09.457 DEBUG:teuthology.orchestra.run.smithi001:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T20:31:09.576 DEBUG:teuthology.orchestra.run.smithi165:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T20:31:09.652 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-18T20:31:09.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:31:09.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-18T20:31:09.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:31:09.721 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-18T20:31:09.730 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-18T20:31:09.781 INFO:teuthology.task.internal:Duration was 6660.782151 seconds 2024-03-18T20:31:09.782 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-18T20:31:09.822 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-18T20:31:09.822 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T20:31:09.824 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T20:31:09.872 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T20:31:09.873 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T20:31:10.248 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-18T20:31:10.248 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi001.front.sepia.ceph.com 2024-03-18T20:31:10.249 DEBUG:teuthology.orchestra.run.smithi001:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T20:31:10.274 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi165.front.sepia.ceph.com 2024-03-18T20:31:10.274 DEBUG:teuthology.orchestra.run.smithi165:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T20:31:10.298 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-18T20:31:10.299 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T20:31:10.317 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T20:31:10.384 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-18T20:31:10.384 DEBUG:teuthology.orchestra.run.smithi001:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T20:31:10.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T20:31:10.491 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-18T20:31:10.502 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-18T20:31:10.502 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T20:31:10.530 DEBUG:teuthology.orchestra.run.smithi165:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T20:31:10.557 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-18T20:31:10.567 DEBUG:teuthology.orchestra.run.smithi001:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T20:31:10.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T20:31:10.598 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = core 2024-03-18T20:31:10.624 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = core 2024-03-18T20:31:10.637 DEBUG:teuthology.orchestra.run.smithi001:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T20:31:10.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:31:10.669 DEBUG:teuthology.orchestra.run.smithi165:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T20:31:10.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T20:31:10.694 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-18T20:31:10.704 INFO:teuthology.task.internal:Transferring archived files... 2024-03-18T20:31:10.704 DEBUG:teuthology.misc:Transferring archived files from smithi001:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606246/remote/smithi001 2024-03-18T20:31:10.705 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T20:31:10.757 DEBUG:teuthology.misc:Transferring archived files from smithi165:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606246/remote/smithi165 2024-03-18T20:31:10.757 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T20:31:10.799 INFO:teuthology.task.internal:Removing archive directory... 2024-03-18T20:31:10.799 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T20:31:10.802 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T20:31:10.851 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-18T20:31:10.863 INFO:teuthology.task.internal:Not uploading archives. 2024-03-18T20:31:10.864 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-18T20:31:10.894 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-18T20:31:10.894 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T20:31:10.897 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T20:31:10.912 INFO:teuthology.orchestra.run.smithi001.stdout: 525356 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 20:31 /home/ubuntu/cephtest 2024-03-18T20:31:10.914 INFO:teuthology.orchestra.run.smithi165.stdout: 525356 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 20:31 /home/ubuntu/cephtest 2024-03-18T20:31:10.915 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-18T20:31:10.932 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-18T20:31:11.036 INFO:teuthology.nuke:Checking targets against current locks 2024-03-18T20:31:11.064 DEBUG:teuthology.nuke:shortname: smithi001 2024-03-18T20:31:11.064 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T20:31:11.088 DEBUG:teuthology.nuke:shortname: smithi165 2024-03-18T20:31:11.088 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T20:31:11.092 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606246', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 18:06:01.370357', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAG5rtJ0ZhNMR5UGSUXb7UXuV0i2c8I2qy7VHzgQ9LhnisB7m9uAD+tqlNDM5PLIXclL2I2FcSts8ljJ4fNGHP4='} 2024-03-18T20:31:11.117 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606246', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 18:06:01.368621', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKlrV/zKFmHgntFhZLft2rs+fc929sATSRWOM5Wdio8Ows/4qHxwsjOObNBEEGsHqIpnPNQ3+d0ferKQgPJGLZ4='} 2024-03-18T20:31:11.151 INFO:teuthology.orchestra.console.smithi001:Power off 2024-03-18T20:31:11.151 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T20:31:11.170 INFO:teuthology.orchestra.console.smithi165:Power off 2024-03-18T20:31:11.170 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T20:31:11.180 DEBUG:teuthology.orchestra.console.smithi001:power off output: Chassis Power Control: Down/Off 2024-03-18T20:31:11.181 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T20:31:11.198 DEBUG:teuthology.orchestra.console.smithi165:power off output: Chassis Power Control: Down/Off 2024-03-18T20:31:11.198 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T20:31:11.207 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is on 2024-03-18T20:31:11.220 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-03-18T20:31:15.208 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T20:31:15.220 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T20:31:15.319 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is on 2024-03-18T20:31:15.331 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-03-18T20:31:19.322 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T20:31:19.331 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T20:31:19.433 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is off 2024-03-18T20:31:19.442 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-03-18T20:31:19.535 INFO:teuthology.orchestra.console.smithi001:Power off completed 2024-03-18T20:31:23.446 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T20:31:23.556 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is off 2024-03-18T20:31:23.657 INFO:teuthology.orchestra.console.smithi165:Power off completed 2024-03-18T20:31:23.762 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} duration: 6660.782151222229 failure_reason: 'Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments)' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=8ebaf6ac9229413f92893d0158b1a87a status: fail success: false 2024-03-18T20:31:23.762 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T20:31:23.860 INFO:teuthology.run:FAIL